commit e2f33166c9f9163d5d58777911b4c837830ff29b Author: kuraeow Date: Sun Oct 25 12:26:18 2026 +0300 UltraTears 0.1.0 An unofficial native PC port of The Legend of Zelda: Tears of the Kingdom (Switch 1.4.3) by static recompilation, built on the player's PC from their own dump. See README.md. Co-Authored-By: Claude Opus 5.5 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..b4c1c97 --- /dev/null +++ b/.gitignore @@ -0,0 +1,47 @@ +# The source folder holds no game data. Everything made from the game lives in the game folder (a builder package +# unpacked elsewhere, tools/deploy.py); the patterns below are a safety net for what must never be committed. + +# The user's own files from their console: never commit +Game/ +Keys/ +Firmware/ +*.nsp +*.nsz +*.xci +*.xcz +*.nca +*.nso +*.nro +*.keys +*.tik +*.bfttf + +# Development only, not in the repository or a release: local tools, reference checkouts, .NET caches, builds of +# development tools, release packages +/dev-local/ + +# A game folder's own (tools/setup.py), should one ever be made inside the source folder +/game/ +/data/ +/build/ +/userdata/ +/diag/ +/deps/ +/ultratears.cfg +/UltraTears.exe +/deploy-files.txt + +# Former development layout (kept ignored until removed) +/work/ +/refs/ +/user-data/ +.ninja_* + +# Local environments and build state +.venv/ +__pycache__/ +third_party/maxwell_shader/src/*/bin/ +third_party/maxwell_shader/src/*/obj/ +third_party/shader_translator/bin/ +third_party/shader_translator/obj/ +third_party/totk_xlate/ diff --git a/BUILDING.md b/BUILDING.md new file mode 100644 index 0000000..5bb0179 --- /dev/null +++ b/BUILDING.md @@ -0,0 +1,92 @@ +# Building UltraTears from the source + +The builder package (README.md) is this source plus the tools below, so it needs nothing installed. From a clone you +install the tools yourself; the build itself is the same command. + +## What you need on your PC + +| Tool | Version used | Notes | +|---|---|---| +| Windows 10/11 x64 | | | +| Python | **3.14** or newer (the build uses `compression.zstd`) | with the packages `lz4 capstone unicorn`: `python -m pip install lz4 capstone unicorn` (a venv is fine) | +| MSYS2, environment CLANG64 | clang/LLVM 23 | packages: `pacman -S mingw-w64-clang-x86_64-{clang,lld,ninja,SDL3,opus,libvpx,zstd,libiconv,vulkan-headers}`. Another folder with the same layout works too: set `ULTRS_CLANG64` to its `bin` | +| The shader translator `ultrs-shaders.exe` | | either `python tools/build_xlate.py` (needs the .NET 10 SDK and network access the first time; everything it fetches stays inside the clone) or copy `deps\shaders\` from the builder package | +| git | | for the clone | + +The tools are looked up in this order (`tools/toolchain.py`): `deps\` in the clone (the builder package's layout), +then `ULTRS_CLANG64`, then `C:\msys64\clang64\bin`. + +## Build + +``` +git clone UltraTears +cd UltraTears +python tools/setup.py +``` + +The dump is `game\` in the clone (`game\exefs`, `game\romfs`; README.md, "What you need"), or the folder given with +`--dump ` (remembered, so later runs need only `python tools/setup.py`). `--cpus N` limits the logical processors the build uses (default: all, at low +priority so the PC stays usable). `--only ` runs one step. + +The steps, what they write and how long they take on an 8-core PC: + +| Step | Writes | Time | Runs again when | +|---|---|---|---| +| `check` | nothing: checks the tools, the SHA-256 of `exefs\main`, `subsdk0`, `sdk` (1.4.3) and refuses encrypted packages | seconds | always | +| `romfs` | `data\romfs\romfs.bin` + index: the RomFS folder in one file, read with 16 threads; its content hash is checked against 1.4.3 | ~4 min | the image is missing | +| `lift` | `data\exefs` (a copy of the code modules), `data\generated` (the game's code as C, ~0.9 GB) | ~2 min | the lifter changed | +| `compile` | `UltraTears.exe` (~0.4 GB) and `build\` (objects ~1.8 GB, debug symbols) | ~15 min (seconds when little changed) | sources changed (ninja) | +| `shaders` | `data\shaders` (SPIR-V of the 28,769 shaders) | ~5 min | the finder or the translator changed; new shaders only otherwise | +| `textures` | `data\textures` (BC7 / BC1 of every ASTC texture, ~7.4 GB) | ~7 min | textures not yet in the cache | +| `finish` | `ultratears.cfg` | seconds | always | + +`data\`, `build\`, `userdata\`, `ultratears.cfg` and `UltraTears.exe` hold material made from your copy of the game: +they are in `.gitignore` and must never be shared. + +## Keeping the game apart from the source (developers) + +`python tools/setup.py` makes the clone itself the game folder. To keep the source folder free of anything made from +the game, unpack a builder package into another folder once (it brings the tools) and build there from your working +tree instead: + +``` +python tools/deploy.py --home D:\Games\UltraTears +``` + +It copies the source files (uncommitted edits too) into that folder and runs its `setup.py`; only what changed is +rebuilt. Run the game from there; development output (profiles, audits, frame dumps) goes to its `diag\`. + +## Where the game looks for its data + +`UltraTears.exe` reads `ultratears.cfg` next to it: the ExeFS (`exefs=`) and the `TOTK_*` variables for the RomFS, +shaders, texture cache and saves. Without that file it uses the development layout (`work\` of the developer's +machine). The RomFS may also be a plain folder (`TOTK_ROMFS=`): it is indexed when the game +starts, but loading reads many small files, which is slower; the packed image is the default. + +## Troubleshooting + +- `clang not found`: install MSYS2 CLANG64 with the packages above, or set `ULTRS_CLANG64`. +- `exefs\main is not the one of Tears of the Kingdom 1.4.3`: extract the ExeFS again with update 1.4.3 applied. +- `the RomFS is not the one of ... 1.4.3`: extract the RomFS again with the update applied, without mods. +- `encrypted packages are not accepted`: give the extracted folders, not an `.nsp` / `.xci` / `.nca`. +- The game starts with a black window or a Vulkan error: update the graphics driver (Vulkan 1.3 with + `VK_EXT_shader_object` is needed). + +## Making the builder package (maintainers) + +``` +python tools/build_xlate.py +git tag v +python tools/deploy.py +python tools/make_package.py +python tools/public_tree.py --publish +``` + +The version has one source, the tag `v` on the release commit (`tools/version.py`): the package, the public +commit and the exe take it from there. The exe writes it as the first line of its log and shows it in the file's +properties; between releases it reads like `0.1.0-3-gabc1234` (`git describe`). + +It needs a complete build in the game folder (`tools/deploy.py`, default `..\release-test`) and a clean working tree, +and writes `dev-local\dist\UltraTears-builder--windows-x86_64.zip`: the public tree of HEAD, the part of MSYS2 CLANG64 the build +reads (measured from ninja's dependency records and the linker's trace), Python 3.14 with the packages above, +`ultrs-shaders.exe`, `setup.bat`, and the licenses of all of it. diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..3877ae0 --- /dev/null +++ b/LICENSE @@ -0,0 +1,674 @@ + GNU GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU General Public License is a free, copyleft license for +software and other kinds of works. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +the GNU General Public License is intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. We, the Free Software Foundation, use the +GNU General Public License for most of our software; it applies also to +any other work released this way by its authors. You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + To protect your rights, we need to prevent others from denying you +these rights or asking you to surrender the rights. Therefore, you have +certain responsibilities if you distribute copies of the software, or if +you modify it: responsibilities to respect the freedom of others. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must pass on to the recipients the same +freedoms that you received. You must make sure that they, too, receive +or can get the source code. And you must show them these terms so they +know their rights. + + Developers that use the GNU GPL protect your rights with two steps: +(1) assert copyright on the software, and (2) offer you this License +giving you legal permission to copy, distribute and/or modify it. + + For the developers' and authors' protection, the GPL clearly explains +that there is no warranty for this free software. For both users' and +authors' sake, the GPL requires that modified versions be marked as +changed, so that their problems will not be attributed erroneously to +authors of previous versions. + + Some devices are designed to deny users access to install or run +modified versions of the software inside them, although the manufacturer +can do so. This is fundamentally incompatible with the aim of +protecting users' freedom to change the software. The systematic +pattern of such abuse occurs in the area of products for individuals to +use, which is precisely where it is most unacceptable. Therefore, we +have designed this version of the GPL to prohibit the practice for those +products. If such problems arise substantially in other domains, we +stand ready to extend this provision to those domains in future versions +of the GPL, as needed to protect the freedom of users. + + Finally, every program is threatened constantly by software patents. +States should not allow patents to restrict development and use of +software on general-purpose computers, but in those that do, we wish to +avoid the special danger that patents applied to a free program could +make it effectively proprietary. To prevent this, the GPL assures that +patents cannot be used to render the program non-free. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Use with the GNU Affero General Public License. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU Affero General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the special requirements of the GNU Affero General Public License, +section 13, concerning interaction through a network will apply to the +combination as such. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . + +Also add information on how to contact you by electronic and paper mail. + + If the program does terminal interaction, make it output a short +notice like this when it starts in an interactive mode: + + Copyright (C) + This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, your program's commands +might be different; for a GUI interface, you would use an "about box". + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU GPL, see +. + + The GNU General Public License does not permit incorporating your program +into proprietary programs. If your program is a subroutine library, you +may consider it more useful to permit linking proprietary applications with +the library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. But first, please read +. diff --git a/README.md b/README.md new file mode 100644 index 0000000..615307e --- /dev/null +++ b/README.md @@ -0,0 +1,110 @@ +# UltraTears + +**UltraTears** is an unofficial native PC port of *The Legend of Zelda: Tears of the Kingdom* (Nintendo Switch, +version 1.4.3) for Windows, and Steam Deck through Proton. It is not an emulator: the game's program code is translated +ahead of time (static recompilation, ARM64 → C → x86-64), and the Switch system and its NVN graphics API are replaced +by the port's own implementations on Windows and Vulkan. Shaders and textures are converted for PC graphics cards when +the game is built, not while you play. + +> **This repository contains no game code, no game assets, no Nintendo SDK code, no firmware and no keys, and it does +> not tell you where to get any of them.** It does nothing without a copy of the game you dumped yourself from your own +> console. Everything made from the game (the translated code, the compiled game, shaders, textures) is made on your +> PC from your files and must stay there: do not share it, and do not open issues asking for it. +> +> UltraTears is not affiliated with or endorsed by Nintendo. *The Legend of Zelda* and *Tears of the Kingdom* are +> trademarks of Nintendo; the name is used only to say which game this works with. + +Русская версия: [README.ru.md](README.ru.md). + +## Status + +Early. Measured on the developer's PC (Windows 10, Ryzen, 16 threads, GeForce RTX 3060): + +| What | Result | +|---|---| +| Title screen, new game, prologue, Great Sky Island | plays | +| Frame rate | the game's own 30 fps, with drops while new shaders compile (being fixed: shaders compiled ahead) | +| Steam Deck | through Proton, being tested | +| Other PCs and graphics cards | not tested yet | + +Known issues: hitches the first time an effect or place is seen (shader compilation); the controller layout follows the +Switch (A is the right face button); no settings menu yet (window size, resolution, upscaling come later). + +## The game is never downloaded + +There is no prebuilt game to download, and there never will be: the game is translated and compiled **on your PC, from +your files**. What you download is this source code, alone or as the **builder package**: the same source plus the +free tools the build needs, so you install nothing. + +## What you need + +- **Your own extracted copy of Tears of the Kingdom with update 1.4.3 applied**: the ExeFS and the RomFS as plain + files, which go into the `game` folder of UltraTears: + ``` + UltraTears\game\ + exefs\ main, subsdk0, sdk, main.npdm, ... + romfs\ Actor\, Pack\, Shader\, ... (about 304,000 files, 16 GB) + ``` + You make them yourself, from your own console and your own copy of the game, with the update installed; this + project does not provide or point to tools for that. The build checks that it is exactly version 1.4.3. Encrypted packages (`.nsp`, + `.xci`, `.nca`, ...) are refused: this project decrypts nothing and reads no keys. +- Windows 10/11 x64; a processor with AVX2 (Intel Haswell / AMD Ryzen or newer). +- A graphics card with a current Vulkan 1.3 driver that has `VK_EXT_shader_object` (recent NVIDIA, AMD and Intel + drivers; Steam Deck's Mesa). +- About **32 GB** free where you unzip the builder (the build makes about 30 GB: the packed RomFS 16 GB, textures + 7.4 GB, shaders 2.4 GB, the compiled game and its intermediate files 4 GB), besides the 16 GB of your game in `game\`, + whose `romfs` you may delete after the build. 16 GB of RAM. + +## Getting started (builder package) + +1. **Download** `UltraTears-builder--windows-x86_64.zip` from the **Releases** page and unzip it into a folder + of its own, for example `D:\Games\UltraTears`. +2. **Put your game in**: copy your extracted `exefs` and `romfs` folders into `game\` there (a dump kept elsewhere + also works: drag its folder onto `setup.bat`). +3. **Build**: run **`setup.bat`**. It checks the dump, then builds, showing each step: + + | Step | What it does | Time (8-core PC) | + |---|---|---| + | check | the version of your dump | seconds | + | romfs | the RomFS packed into one file (`data\romfs`; reading 304,000 small files while playing would stall) | ~4 min | + | lift | the game's code translated to C | ~2 min | + | compile | the C compiled into `UltraTears.exe` | ~15 min | + | shaders | every shader of the game translated for Vulkan | ~5 min | + | textures | the textures re-encoded for PC graphics cards (BC7 / BC1) | ~7 min | + + If it stops (a power cut, you closed it), run `setup.bat` again: it goes on where it stopped. +4. **Play**: start **`UltraTears.exe`** in that folder. The first start takes longer; saves are in `userdata\`. + +After the build `game\romfs` is no longer needed (the game reads its packed copy in `data\romfs`): delete it to free +16 GB, or keep it. Do not move or delete `data\`. + +Keyboard (until a settings menu exists): WASD move, arrow keys camera, J = A, K / Space = B, I = X, U = Y, +Q / E = L / R, left Alt / Shift = ZL / ZR, Enter = +, Tab / Esc = − (map), T F G H = d-pad, Z / C = stick buttons, +F11 or Alt+Enter = full screen. Controllers (Xbox, DualSense, Switch Pro, ...) work through SDL3. + +**Updating the port:** unzip the new builder package over the old one (or into a new folder and copy `data\` and +`userdata\` there) and run `setup.bat`: it rebuilds only what changed. + +**Steam Deck (Proton):** build on a PC, copy the whole UltraTears folder to the Deck, add `UltraTears.exe` to Steam as a +non-Steam game and force Proton in its properties. + +## Building from the source + +See [BUILDING.md](BUILDING.md): the same `tools/setup.py`, with the tools installed yourself. + +## Legal + +- Nothing of the game is in this repository or in a release: no code (also none translated from the game), no assets, + shaders, textures, fonts, keys or firmware. Everything derived from the game is made on the player's PC from the + player's own dump and stays there. +- UltraTears decrypts nothing and reads no keys: it takes the ExeFS and RomFS the player has already extracted. +- No Nintendo SDK code or headers are used; the system and graphics layers are the port's own implementations. +- Only the facts needed to run are in the source: addresses, hashes, short instruction replacements. +- The icon (`assets/icon.ico`) is the author's own drawing, not Nintendo artwork. +- UltraTears is not affiliated with or endorsed by Nintendo; you need your own purchased copy of the game. + +## License + +UltraTears is free software under the **GNU General Public License v3.0 or later** ([LICENSE](LICENSE)). Third-party +code keeps its own license: [third_party/README.md](third_party/README.md); the builder package adds +`THIRD-PARTY-NOTICES.md` and `licenses\` for the tools it carries. diff --git a/README.ru.md b/README.ru.md new file mode 100644 index 0000000..94e6eeb --- /dev/null +++ b/README.ru.md @@ -0,0 +1,109 @@ +# UltraTears + +**UltraTears** — неофициальный нативный порт *The Legend of Zelda: Tears of the Kingdom* (Nintendo Switch, версия +1.4.3) на Windows и на Steam Deck через Proton. Это не эмулятор: код игры переводится заранее (статическая +рекомпиляция ARM64 → C → x86-64), а система Switch и графика NVN заменены своими реализациями на Windows и Vulkan. +Шейдеры и текстуры переводятся для видеокарт ПК при сборке, а не во время игры. + +> **В репозитории нет ни кода игры, ни её ресурсов, ни кода Nintendo SDK, ни прошивки, ни ключей, и здесь не сказано, +> где их взять.** Без копии игры, снятой вами со своей консоли, порт ничего не делает. Всё, что получается из игры +> (переведённый код, собранная игра, шейдеры, текстуры), делается на вашем ПК из ваших файлов и должно там остаться: +> не распространяйте это и не просите в issues. +> +> UltraTears не связан с Nintendo и не одобрен ею. *The Legend of Zelda* и *Tears of the Kingdom* — товарные знаки +> Nintendo; название используется только для указания, с какой игрой работает порт. + +English: [README.md](README.md). + +## Состояние + +Ранняя версия. Замеры на ПК разработчика (Windows 10, 16 потоков, GeForce RTX 3060): + +| Что | Результат | +|---|---| +| Титульный экран, новая игра, пролог, Великий Небесный остров | играется | +| Частота кадров | родные 30 к/с игры; просадки, пока компилируются новые шейдеры (исправляется: шейдеры заранее) | +| Steam Deck | через Proton, проверяется | +| Другие ПК и видеокарты | ещё не проверялись | + +Известное: подтормаживания, когда эффект или место встречается впервые (компиляция шейдеров); раскладка геймпада как +на Switch (A — правая кнопка); меню настроек пока нет (размер окна, разрешение, апскейл — позже). + +## Игра не скачивается + +Готовой игры для скачивания нет и не будет: игра переводится и собирается **на вашем ПК из ваших файлов**. Скачивается +этот исходный код — отдельно или как **пакет сборщика**: те же исходники плюс бесплатные инструменты для сборки, чтобы +ничего не устанавливать. + +## Что нужно + +- **Своя извлечённая копия Tears of the Kingdom с обновлением 1.4.3**: ExeFS и RomFS обычными файлами — они кладутся в + папку `game` UltraTears: + ``` + UltraTears\game\ + exefs\ main, subsdk0, sdk, main.npdm, ... + romfs\ Actor\, Pack\, Shader\, ... (около 304 000 файлов, 16 ГБ) + ``` + Их вы делаете сами — со своей консоли и из своей копии игры с установленным обновлением; проект не даёт для этого + инструментов и не указывает на них. Сборка проверяет, что это точно версия 1.4.3. Зашифрованные пакеты (`.nsp`, `.xci`, + `.nca`, ...) не принимаются: проект ничего не расшифровывает и ключей не читает. +- Windows 10/11 x64; процессор с AVX2 (Intel Haswell / AMD Ryzen и новее). +- Видеокарта со свежим драйвером Vulkan 1.3 с `VK_EXT_shader_object` (современные драйверы NVIDIA, AMD, Intel; Mesa на + Steam Deck). +- Около **32 ГБ** свободного места там, куда распакован сборщик (сборка делает около 30 ГБ: упакованная RomFS 16 ГБ, + текстуры 7,4 ГБ, шейдеры 2,4 ГБ, собранная игра с промежуточными файлами 4 ГБ), плюс 16 ГБ самой игры в `game\` — + её `romfs` после сборки можно удалить. 16 ГБ памяти. + +## Как начать (пакет сборщика) + +1. **Скачать** `UltraTears-builder-<версия>-windows-x86_64.zip` со страницы **Releases** и распаковать в отдельную + папку, например `D:\Games\UltraTears`. +2. **Положить игру**: скопировать извлечённые папки `exefs` и `romfs` в `game\` (дамп в другом месте тоже подойдёт: + перетащить его папку на `setup.bat`). +3. **Собрать**: запустить **`setup.bat`**. Сборщик проверит дамп и соберёт игру, показывая шаги: + + | Шаг | Что делает | Время (8 ядер) | + |---|---|---| + | check | проверяет версию дампа | секунды | + | romfs | упаковывает RomFS в один файл (`data\romfs`; читать 304 000 мелких файлов во время игры — фризы) | ~4 мин | + | lift | переводит код игры в C | ~2 мин | + | compile | компилирует C в `UltraTears.exe` | ~15 мин | + | shaders | переводит все шейдеры игры для Vulkan | ~5 мин | + | textures | перекодирует текстуры для видеокарт ПК (BC7 / BC1) | ~7 мин | + + Если сборка прервалась (свет выключили, окно закрыли) — запустить `setup.bat` ещё раз: продолжит с того же места. +4. **Играть**: запустить **`UltraTears.exe`** в этой папке. Первый запуск дольше; сохранения — в `userdata\`. + +После сборки `game\romfs` больше не нужна (игра читает упакованную копию в `data\romfs`): её можно удалить и +освободить 16 ГБ. Папку `data\` не трогать. + +Клавиатура (пока нет меню настроек): WASD — движение, стрелки — камера, J = A, K / пробел = B, I = X, U = Y, +Q / E = L / R, левый Alt / Shift = ZL / ZR, Enter = +, Tab / Esc = − (карта), T F G H — крестовина, Z / C — нажатие +стиков, F11 или Alt+Enter — полный экран. Геймпады (Xbox, DualSense, Switch Pro и др.) работают через SDL3. + +**Обновление порта:** распаковать новый пакет поверх старого (или в новую папку и перенести туда `data\` и +`userdata\`) и запустить `setup.bat`: пересоберётся только изменившееся. + +**Steam Deck (Proton):** собрать на ПК, скопировать всю папку UltraTears на Deck, добавить `UltraTears.exe` в Steam как +стороннюю игру и в свойствах включить Proton. + +## Сборка из исходников + +См. [BUILDING.md](BUILDING.md): тот же `tools/setup.py`, инструменты ставятся самостоятельно. + +## Правовое + +- Ничего от игры нет ни в репозитории, ни в релизе: ни кода (в том числе переведённого из игры), ни ресурсов, шейдеров, + текстур, шрифтов, ключей, прошивки. Всё производное от игры делается на ПК игрока из его собственного дампа и там + остаётся. +- UltraTears ничего не расшифровывает и не читает ключи: он берёт уже извлечённые игроком ExeFS и RomFS. +- Код и заголовки Nintendo SDK не используются; системный и графический слои — собственные реализации порта. +- В исходниках только факты, нужные для работы: адреса, хеши, короткие замены инструкций. +- Иконка (`assets/icon.ico`) — собственный рисунок автора, не арт Nintendo. +- UltraTears не связан с Nintendo и не одобрен ею; нужна своя купленная копия игры. + +## Лицензия + +UltraTears — свободная программа под **GNU General Public License v3.0 или новее** ([LICENSE](LICENSE)). Сторонний +код сохраняет свои лицензии: [third_party/README.md](third_party/README.md); в пакете сборщика — `THIRD-PARTY-NOTICES.md` +и `licenses\` для инструментов, которые он содержит. diff --git a/assets/icon.ico b/assets/icon.ico new file mode 100644 index 0000000..c4d87d8 Binary files /dev/null and b/assets/icon.ico differ diff --git a/src/gfx/backend_null.c b/src/gfx/backend_null.c new file mode 100644 index 0000000..ec3c8bb --- /dev/null +++ b/src/gfx/backend_null.c @@ -0,0 +1,103 @@ +// Milestone 3.1: the NVN front end (MacNativeGames hle_nvn.c) without a GPU. Every command buffer is "executed" at +// once by doing nothing, fences complete immediately, presents only count. Plus the stand-ins for MacNativeGames' +// diagnostics (life log, timeline), MetalFX, virtual pools (not ported yet: stop with their name) and host glue. +// The Vulkan replayer replaces the mtl_* part (milestone 3.2/3.3). +#include +#include +#include + +#include "compat/gfx_compat.h" +#include "host/hle.h" +#include "nvn/nvn_obj.h" +#include "nvn/totk_vpool.h" + +// ---------------- mtl_*: execution ---------------- +int mtl_warm_wait(int timeout_ms) { (void)timeout_ms; return 1; } +void mtl_prog_release(prog_t *p) { (void)p; } +void mtl_dead_forget(const uint64_t *serials, size_t n) { (void)serials; (void)n; } +void mtl_pin_refs(void (*mark)(const void *)) { (void)mark; } +void mtl_probe_pin_rt(const tex_t *t) { (void)t; } +void mtl_xv_window_tex(uint64_t serial) { (void)serial; } +void mtl_present_wait(uint64_t ticket) { (void)ticket; } +void mtl_shot_at_present(const tex_t *t) { (void)t; } +int mtl_has_window(void) { return 0; } +int mtl_display_hz(void) { return 60; } +double mtl_presented_time(uint64_t n) { (void)n; return 0; } +void mtl_set_frame(uint64_t f) { (void)f; } +void mtl_note_world(void) {} +void mtl_ub_present(double iv_ms) { (void)iv_ms; } +uint32_t mtl_present_qos(void) { return 0; } +int mtl_link_active(void) { return 0; } +int mtl_link_wait_tick(uint64_t timeout_ns, unsigned frame_ticks) { (void)timeout_ns; (void)frame_ticks; return 0; } +double mtl_gpu_ms(uint64_t f) { (void)f; return 0; } +int mtl_gpu_span(uint64_t f, uint64_t *t0, uint64_t *t1) { (void)f; *t0 = *t1 = 0; return 0; } + +// ---------------- mtl_*: statistics (all zero) ---------------- +void mtl_alloc_stats(uint64_t *a, uint64_t *b, int *c) { *a = *b = 0; *c = 0; } +void mtl_args_stats(uint64_t *enc, uint64_t *reuse) { *enc = *reuse = 0; } +void mtl_blit_stats(uint64_t v[6]) { memset(v, 0, 6 * sizeof *v); } +void mtl_bw(uint64_t *load, uint64_t *store, uint64_t *passes) { *load = *store = *passes = 0; } +uint64_t mtl_chunk_commits(void) { return 0; } +void mtl_create_ns(uint64_t *lib_ns, uint64_t *pso_ns) { *lib_ns = *pso_ns = 0; } +void mtl_create_stats(uint64_t *lib, uint64_t *pso) { *lib = *pso = 0; } +uint64_t mtl_drawable_ns_take(void) { return 0; } +uint64_t mtl_ds_omit(void) { return 0; } +void mtl_keepalive_stats(uint64_t v[3]) { memset(v, 0, 3 * sizeof *v); } +void mtl_layer_stats(uint64_t v[4]) { memset(v, 0, 4 * sizeof *v); } +uint64_t mtl_pkg_miss(void) { return 0; } +void mtl_pkg_stats(uint64_t *uploads, uint64_t *reuploads) { *uploads = *reuploads = 0; } +void mtl_pool_res_stats(uint64_t v[4]) { memset(v, 0, 4 * sizeof *v); } +void mtl_ren_stats(uint64_t v[6]) { memset(v, 0, 6 * sizeof *v); } +void mtl_stats(uint64_t *passes, uint64_t *draws) { *passes = *draws = 0; } +uint64_t mtl_udc_saved_bytes(void) { return 0; } +uint64_t mtl_upload_bytes(void) { return 0; } +uint64_t mtl_upload_gpu_ns_take(void) { return 0; } +uint64_t mtl_upload_ns_take(void) { return 0; } +void mtl_upload_wait_stats(uint64_t *n, uint64_t *ns) { *n = *ns = 0; } +void mtl_use_stats(uint64_t *call, uint64_t *skip) { *call = *skip = 0; } + +// ---------------- MacNativeGames diagnostics: off ---------------- +int totk_life_on(void) { return 0; } +void totk_life_log(const char *fmt, ...) { (void)fmt; } +void totk_life_reg(int id, const tex_t *t) { (void)id; (void)t; } +uint64_t totk_life_reg_serial(int id) { (void)id; return 0; } +int totk_life_match(const tex_t *t) { (void)t; return 0; } +void totk_life_cb_mark(const void *cb) { (void)cb; } +int totk_life_cb_take(const void *cb) { (void)cb; return 0; } +void totk_life_handle_add(uint64_t h) { (void)h; } +int totk_life_handle_has(uint64_t h) { (void)h; return 0; } +int thread_selfcounts(int type, void *buf, size_t nbytes) { (void)type; (void)buf; (void)nbytes; return -1; } +void runtime_watch_set(uintptr_t addr, size_t len) { (void)addr; (void)len; } +_Atomic int runtime_survey_pending; // the world survey is not ported: never pending +void runtime_survey_tick(uint64_t frame) { (void)frame; } +const int32_t runtime_survey_sync[2] = {0, -1}; + +// ---------------- MetalFX (no equivalent here: off) ---------------- +int mfx_enabled(void) { return 0; } +int mfx_latch(void) { return 0; } +void mfx_set(int mode) { (void)mode; } + +// ---------------- host glue (MacNativeGames d2r_gfx/host, runtime/loader.c) ---------------- +uint64_t g_t0_ns; +uint64_t g_draw_skip[16]; +__thread sigjmp_buf *runtime_fault_guard; +uint64_t now_ns(void) { return runtime_now_ns(); } +uint64_t guest_ns(void) { return runtime_now_ns(); } +int dm_detclock_enabled(void) { return 0; } +void dm_guest_frame_tick(void) {} +uintptr_t runtime_frame_controller_return_pc(void) { return 0; } +int runtime_module_offset(uint64_t pc, const char **name, uint64_t *off) { + for (int i = 0; i < g_nmodules; i++) + if (pc >= g_modules[i].base && pc < g_modules[i].base + g_modules[i].size) { + *name = g_modules[i].name; *off = pc - g_modules[i].base; return 1; + } + return 0; +} +void runtime_describe_address(uint64_t a, char *out, size_t n) { + uint64_t off = 0; + const char *s = loader_symbol_at(a, &off); + if (s) snprintf(out, n, "%s+%#llx", s, (unsigned long long)off); + else snprintf(out, n, "%#llx", (unsigned long long)a); +} +void nvn_unimpl_entry(void) { runtime_fail("NVN: an unimplemented entry was reached"); } +void nvn_prof_entry(void) { runtime_fail("NVN: the profiling entry is not ported"); } diff --git a/src/gfx/backend_vk.c b/src/gfx/backend_vk.c new file mode 100644 index 0000000..3fd1c77 --- /dev/null +++ b/src/gfx/backend_vk.c @@ -0,0 +1,1957 @@ +// The NVN replayer on Vulkan (milestone 3.3): the mtl_* execution layer that MacNativeGames implements on Metal +// (nvn_mtl.m), written anew for Vulkan against the same interface (nvn_mtl.h, nvn_obj.h). Step 1: memory pools, +// submissions and their completion; the recorded command stream is walked, and the commands with CPU-visible effects +// (events, counters, buffer copies and updates) run; drawing comes in the next steps. +// - memory pools: the game's memory is imported as VkBuffers (VK_EXT_external_memory_host), no copies; a GPU address +// is the CPU address, and mtl_addr finds the buffer and offset for it; +// - a submission is one command buffer; its fence is the value it signals on one timeline semaphore; +// - visible effects of completion (event values, counter records) are written by the retire thread after the GPU has +// finished the submission, in submission order, and only then is the fence complete for mtl_wait (as on Metal). +// Without a usable Vulkan device everything completes at once (the null device's behaviour, events still written). +#include +#include +#include +#include + +#include "compat/gfx_compat.h" +#include "host/hle.h" +#include "nvn/nvn_dirty.h" +#include "nvn/nvn_obj.h" +#include "../vk/present.h" +#include "vkb.h" +#include "vkb_shader.h" +#include "vkb_audit.h" +#include "vkb_lock.h" +#include "nvn/totk_timeline.h" + +#define VLOG(...) LOG("[vk] " __VA_ARGS__) + +static _Atomic int g_ok = -1; // -1 not initialised, 0 no Vulkan (null behaviour), 1 ready +static VkSemaphore g_timeline; +static uint64_t g_fence; // last submitted (submit thread; read under g_mu by others) +static uint64_t g_done; // last retired (under g_mu) +static pthread_mutex_t g_mu = PTHREAD_MUTEX_INITIALIZER; +static pthread_cond_t g_cv = PTHREAD_COND_INITIALIZER; + +void mtl_finish(void); +static void pass_end(void); +static void pass_end_why(int why); +int mtl_wait(uint64_t fence, uint64_t timeout_ns); +static uint64_t gpu_ticks(void) { return runtime_now_ns() * 12 / 625; } // the Switch's 19.2 MHz counter + +// ---------------- memory pools ---------------- +// sh..defer_pass: the pool's copy in device memory (below, "copies of the pools in device memory") +typedef struct { uintptr_t lo, hi, base; VkBuffer buf; VkDeviceMemory mem; const void *key; VkDeviceSize bytes; + VkBuffer sbuf; VkDeviceSize soff; uint32_t sh; uint8_t no_shadow, filled; uint64_t stamp, sync_sub, used_sub, defer_pass, wb_sub; } pool_ent; +static pool_ent *g_pools; +static int g_npools, g_capools; +static vkb_rw g_pool_mu; // recording threads look pools up for every draw (read); changes write +static uint64_t g_pool_bytes; + +// vkb_lock.h: Windows slim reader-writer locks (zero-initialised = unlocked), with the wait measured as vkb_lock does +void vkb_rw_read(vkb_rw *l, int kind) { + if (TryAcquireSRWLockShared((PSRWLOCK)l)) return; + uint64_t t0 = runtime_now_ns(); + AcquireSRWLockShared((PSRWLOCK)l); + atomic_fetch_add_explicit(&vkb_lock_wait_ns[kind], runtime_now_ns() - t0, memory_order_relaxed); + atomic_fetch_add_explicit(&vkb_lock_waits[kind], 1, memory_order_relaxed); +} +void vkb_rw_read_end(vkb_rw *l) { ReleaseSRWLockShared((PSRWLOCK)l); } +void vkb_rw_write(vkb_rw *l, int kind) { + if (TryAcquireSRWLockExclusive((PSRWLOCK)l)) return; + uint64_t t0 = runtime_now_ns(); + AcquireSRWLockExclusive((PSRWLOCK)l); + atomic_fetch_add_explicit(&vkb_lock_wait_ns[kind], runtime_now_ns() - t0, memory_order_relaxed); + atomic_fetch_add_explicit(&vkb_lock_waits[kind], 1, memory_order_relaxed); +} +void vkb_rw_write_end(vkb_rw *l) { ReleaseSRWLockExclusive((PSRWLOCK)l); } + +// ---- per recording thread (vkb_mt.c design, implemented below in "several recording threads") +static __thread int t_worker; // vkb_worker(): 0 the translation thread (and any other), 1.. the recording threads +static __thread int t_recording; // recording a chunk (or the prologue) of a submission +static __thread VkCommandBuffer t_cb; // the command buffer it records (VK_NULL_HANDLE: the submission's own) +static __thread uint64_t t_seq; // the position of the command being replayed +int vkb_worker(void) { return t_worker; } +int vkb_mt_recording(void) { return t_recording; } +uint64_t vkb_seq(void) { return t_seq; } +void vkb_seq_set(uint64_t seq) { t_seq = seq; } +void vkb_cmd_set(VkCommandBuffer cb) { t_cb = cb; } + +static void wb_forget(VkBuffer host); +static uint32_t memory_type(uint32_t bits) { + for (uint32_t i = 0; i < vkc.memory.memoryTypeCount; i++) if (bits & (1u << i)) return i; + return UINT32_MAX; +} + +// Commands recorded per frame (the present log line every 300 presents): barriers with their image layout changes and +// render passes begun. Counted by wrapping the two functions once the device exists. +static PFN_vkCmdPipelineBarrier2 g_real_barrier; +static PFN_vkCmdBeginRendering g_real_begin; +static vkb_cnt g_n_barriers, g_n_img_barriers, g_n_passes; +static uint64_t g_parts; +_Atomic uint64_t vkb_lock_wait_ns[LK_COUNT], vkb_lock_waits[LK_COUNT], vkb_step_ns[PK_COUNT], vkb_step_n[PK_COUNT]; +static void VKAPI_CALL count_barrier(VkCommandBuffer cb, const VkDependencyInfo *d) { + vkb_cnt_add(g_n_barriers, 1); vkb_cnt_add(g_n_img_barriers, d->imageMemoryBarrierCount); + g_real_barrier(cb, d); +} +static void VKAPI_CALL count_begin(VkCommandBuffer cb, const VkRenderingInfo *r) { vkb_cnt_add(g_n_passes, 1); g_real_begin(cb, r); } + +static void backend_init(void) { + if (g_ok >= 0) return; + static pthread_mutex_t mu = PTHREAD_MUTEX_INITIALIZER; + pthread_mutex_lock(&mu); + if (g_ok < 0) { + int ok = vk_init(NULL); // already done by the window when there is one + if (ok) { + VkSemaphoreTypeCreateInfo ti = {VK_STRUCTURE_TYPE_SEMAPHORE_TYPE_CREATE_INFO, .semaphoreType = VK_SEMAPHORE_TYPE_TIMELINE}; + VkSemaphoreCreateInfo si = {VK_STRUCTURE_TYPE_SEMAPHORE_CREATE_INFO, &ti}; + ok = vk_check(vk.vkCreateSemaphore(vkc.device, &si, NULL, &g_timeline), "timeline semaphore"); + } + void retire_start(void); + if (ok) retire_start(); + if (ok) { + g_real_barrier = vk.vkCmdPipelineBarrier2; vk.vkCmdPipelineBarrier2 = count_barrier; + g_real_begin = vk.vkCmdBeginRendering; vk.vkCmdBeginRendering = count_begin; + } + vkb_state_reset(); + VLOG("replayer %s", ok ? "on Vulkan" : "without a GPU: submissions complete at once"); + atomic_store(&g_ok, ok); + } + pthread_mutex_unlock(&mu); +} + +void mtl_pool_add(const void *mem, uint64_t size) { + backend_init(); + if (!g_ok || !mem || !size) return; + uintptr_t align = vkc.host_pointer_alignment ? vkc.host_pointer_alignment : 4096; + uintptr_t lo = (uintptr_t)mem, hi = lo + size, a = lo & ~(align - 1), b = (hi + align - 1) & ~(align - 1); + // The import needs committed memory: guest pages are committed on first touch, so the pool is committed now. + // A virtual pool is an inaccessible address range (totk_vpool_win.c): no memory to import (sparse binding, later). + MEMORY_BASIC_INFORMATION mi; + if (a >= GUEST_LO && b <= GUEST_HI) plat_commit(a, b - a); + else if (!VirtualQuery((void *)a, &mi, sizeof mi) || mi.State != MEM_COMMIT || mi.Protect == PAGE_NOACCESS) return; + VkMemoryHostPointerPropertiesEXT hp = {VK_STRUCTURE_TYPE_MEMORY_HOST_POINTER_PROPERTIES_EXT}; + if (!vk_check(vk.vkGetMemoryHostPointerPropertiesEXT(vkc.device, VK_EXTERNAL_MEMORY_HANDLE_TYPE_HOST_ALLOCATION_BIT_EXT, + (void *)a, &hp), "host pointer properties")) return; + VkExternalMemoryBufferCreateInfo eb = {VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_BUFFER_CREATE_INFO, + .handleTypes = VK_EXTERNAL_MEMORY_HANDLE_TYPE_HOST_ALLOCATION_BIT_EXT}; + VkBufferCreateInfo bi = {VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, &eb, .size = b - a, + .usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT | VK_BUFFER_USAGE_UNIFORM_BUFFER_BIT | + VK_BUFFER_USAGE_STORAGE_BUFFER_BIT | VK_BUFFER_USAGE_INDEX_BUFFER_BIT | VK_BUFFER_USAGE_VERTEX_BUFFER_BIT | + VK_BUFFER_USAGE_INDIRECT_BUFFER_BIT | VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT, + .sharingMode = VK_SHARING_MODE_EXCLUSIVE}; + VkBuffer buf; + if (!vk_check(vk.vkCreateBuffer(vkc.device, &bi, NULL, &buf), "pool buffer")) return; + VkMemoryRequirements req; + vk.vkGetBufferMemoryRequirements(vkc.device, buf, &req); + uint32_t type = memory_type(req.memoryTypeBits & hp.memoryTypeBits); + VkImportMemoryHostPointerInfoEXT imp = {VK_STRUCTURE_TYPE_IMPORT_MEMORY_HOST_POINTER_INFO_EXT, + .handleType = VK_EXTERNAL_MEMORY_HANDLE_TYPE_HOST_ALLOCATION_BIT_EXT, .pHostPointer = (void *)a}; + VkMemoryAllocateFlagsInfo fl = {VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_FLAGS_INFO, &imp, .flags = VK_MEMORY_ALLOCATE_DEVICE_ADDRESS_BIT}; + VkMemoryAllocateInfo ai = {VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO, &fl, .allocationSize = b - a, .memoryTypeIndex = type}; + VkDeviceMemory dm; + if (type == UINT32_MAX || !vk_check(vk.vkAllocateMemory(vkc.device, &ai, NULL, &dm), "pool import")) { + VLOG("memory pool %p+%#llx: cannot import (memory types %#x / %#x)", mem, (unsigned long long)size, req.memoryTypeBits, hp.memoryTypeBits); + vk.vkDestroyBuffer(vkc.device, buf, NULL); + return; + } + vk.vkBindBufferMemory(vkc.device, buf, dm, 0); + vkb_rw_write(&g_pool_mu, LK_POOLS); + if (g_npools == g_capools) { + g_capools = g_capools ? g_capools * 2 : 64; + g_pools = realloc(g_pools, (size_t)g_capools * sizeof *g_pools); + if (!g_pools) die("pool table"); + } + int i = g_npools; + while (i > 0 && g_pools[i - 1].lo > lo) { g_pools[i] = g_pools[i - 1]; i--; } + g_pools[i] = (pool_ent){lo, hi, a, buf, dm, mem, b - a}; + g_npools++; + for (int j = i - 4; j <= i + 4; j++) // overlapping pools: one copy each would diverge, so neither is copied + if (j >= 0 && j < g_npools && j != i && g_pools[j].lo < hi && lo < g_pools[j].hi) { + static int said; + if (said++ < 8) VLOG("memory pool %p+%#llx overlaps pool %#llx..%#llx: neither is copied to device memory", mem, + (unsigned long long)size, (unsigned long long)g_pools[j].lo, (unsigned long long)g_pools[j].hi); + g_pools[i].no_shadow = g_pools[j].no_shadow = 1; + if (g_pools[j].sh) { vkb_shadow_free(g_pools[j].sh); g_pools[j].sh = 0; } + } + g_pool_bytes += hi - lo; + static unsigned said; + if (said++ < 4) VLOG("memory pool %p+%#llx imported (%d pools, %.1f MiB)", mem, (unsigned long long)size, g_npools, g_pool_bytes / 1048576.0); + vkb_rw_write_end(&g_pool_mu); +} + +void mtl_pool_del(const void *mem) { + if (g_ok != 1) return; + pool_ent e = {0}; + vkb_rw_write(&g_pool_mu, LK_POOLS); + for (int i = 0; i < g_npools; i++) + if (g_pools[i].key == mem) { + e = g_pools[i]; + memmove(&g_pools[i], &g_pools[i + 1], (size_t)(g_npools - i - 1) * sizeof *g_pools); + g_npools--; + g_pool_bytes -= e.hi - e.lo; + break; + } + if (e.sh) vkb_shadow_free(e.sh); + if (e.buf) wb_forget(e.buf); + vkb_rw_write_end(&g_pool_mu); + if (!e.buf) return; + if (vkc.has_checkpoints) VLOG("memory pool %p %#llx..%#llx deleted (present %llu)", mem, (unsigned long long)e.lo, (unsigned long long)e.hi, (unsigned long long)vkb_frame()); + // The frontend drains the queue before it deletes a pool (hle_nvn.c x_drain); the GPU may still be finishing. + mtl_finish(); + vk.vkDestroyBuffer(vkc.device, e.buf, NULL); + vk.vkFreeMemory(vkc.device, e.mem, NULL); +} + +void mtl_pool_invalidate(const pool_t *p, uint64_t off, uint64_t n) { (void)p; (void)off; (void)n; } // zero-copy: nothing cached + +static __thread int g_pool_last; // the entry this thread found last (draws use a few pools over and over) +static const pool_ent *pool_find(uintptr_t addr, uint64_t len) { // g_pool_mu held + if (g_pool_last < g_npools && addr >= g_pools[g_pool_last].lo && addr + len <= g_pools[g_pool_last].hi) return &g_pools[g_pool_last]; + int lo = 0, hi = g_npools - 1, found = -1; + while (lo <= hi) { + int m = (lo + hi) / 2; + if (g_pools[m].lo <= addr) { found = m; lo = m + 1; } else hi = m - 1; + } + for (int i = found; i >= 0 && i > found - 4; i--) // nested / adjacent pools: the closest few that start below + if (addr >= g_pools[i].lo && addr + len <= g_pools[i].hi) { g_pool_last = i; return &g_pools[i]; } + return NULL; +} + +// ---------------- copies of the pools in device memory (discrete GPUs; memory: vkb_shadow.c) ---------------- +// Game memory is host memory imported into Vulkan, so on a discrete GPU every vertex, index and constant a draw reads +// crosses PCIe: on the Great Sky Island 28 of the frame's 34 ms of GPU time were spent inside render passes, 7.6 ms with +// the pools copied to device memory (experiment, 2026-10-24). A pool the GPU uses gets a copy in device memory; in a +// submission, once the copy is current, every GPU access to the pool goes to it (vkb_pool_addr). Keeping it current: +// - CPU writes: pages written since the copy was last brought up to date (the write watch, nvn_dirty_runs) are copied +// in when the pool is first used in a submission. In a render pass (no copies there) a pool with written pages stays +// in game memory until the pass ends, unless a shader writes it (then the pass ends); a clean one switches at once. +// At the first pass of a submission, the pools used in the two before are brought up to date together. +// - GPU writes (shader stores, buffer copies, uniform updates, texture-to-buffer copies) go to the copy and are copied +// back to game memory at the end of the submission, through host memory and the retire thread (wb_merge), which +// leaves alone the pages the CPU wrote meanwhile (the CPU reads the results after the game's wait). +// [temporary] Page granularity: GPU writes into a page the CPU also wrote are lost (counted in the log line). +// Pages the copy back writes are not CPU writes; were the write watch to count them, they would only be copied in again. +// The game writes memory for a submission before it is submitted, and does not overwrite what the GPU still reads (it +// waits for the GPU first), so bringing a copy up to date early (ahead, or for the whole pool) does not change results. +// Integrated GPUs (the Steam Deck) read host memory at full speed: no copies there (vkb_shadow_on). +typedef struct { VkBuffer src, dst; VkDeviceSize soff, doff, n; uint64_t base, stamp; } wb_t; // base: the pool's import address; stamp: its copy's +static wb_t *g_wb; // GPU-written ranges of copies in this submission: back to game memory at its end +static int g_nwb, g_capwb; +static VkBufferCopy *g_runs; // copies being gathered (sync_gather / sync_flush) +static int g_nruns, g_caruns; +typedef struct { VkBuffer src, dst; int first, n; } run_grp; +static run_grp *g_grp; +static int g_ngrp, g_capgrp; +static uint64_t g_shadow_budget, g_presync_sub; +static uint64_t g_sh_in, g_sh_in_bytes, g_sh_out_bytes, g_sh_deferred, g_sh_scan_ns; // the per-frame log line + +static void shadow_barrier(VkCommandBuffer cb, VkPipelineStageFlags2 src, VkAccessFlags2 sa, VkPipelineStageFlags2 dst, VkAccessFlags2 da) { + VkMemoryBarrier2 mb = {VK_STRUCTURE_TYPE_MEMORY_BARRIER_2, .srcStageMask = src, .srcAccessMask = sa, .dstStageMask = dst, .dstAccessMask = da}; + VkDependencyInfo dep = {VK_STRUCTURE_TYPE_DEPENDENCY_INFO, .memoryBarrierCount = 1, .pMemoryBarriers = &mb}; + vk.vkCmdPipelineBarrier2(cb, &dep); +} +static void run_put(VkBuffer src, VkBuffer dst, VkDeviceSize soff, VkDeviceSize doff, VkDeviceSize n) { + if (g_nruns == g_caruns) { + g_caruns = g_caruns ? g_caruns * 2 : 256; + if (!(g_runs = realloc(g_runs, sizeof *g_runs * (size_t)g_caruns))) die("pool copy runs"); + } + if (!g_ngrp || g_grp[g_ngrp - 1].src != src || g_grp[g_ngrp - 1].dst != dst) { + if (g_ngrp == g_capgrp) { + g_capgrp = g_capgrp ? g_capgrp * 2 : 64; + if (!(g_grp = realloc(g_grp, sizeof *g_grp * (size_t)g_capgrp))) die("pool copy groups"); + } + g_grp[g_ngrp++] = (run_grp){src, dst, g_nruns, 0}; + } + g_runs[g_nruns++] = (VkBufferCopy){soff, doff, n}; + g_grp[g_ngrp - 1].n++; + g_sh_in_bytes += n; +} +static void run_written(void *ctx, size_t off, size_t n) { // nvn_dirty_runs: off from the pool's start + const pool_ent *p = ctx; + VkDeviceSize o = (p->lo - p->base) + off; + run_put(p->buf, p->sbuf, o, p->soff + o, n); +} +static void sync_flush(void) { // the gathered copies, outside a pass (threads recording: in the prologue) + if (!g_nruns) return; + int mt = t_recording; + if (mt) vkb_pro_begin(); + else pass_end_why(PE_SHADOW); + VkCommandBuffer cb = vkb_cmd(); + vkb_checkpoint("pool copies in", NULL, NULL); + shadow_barrier(cb, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT, + VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT); + for (int g = 0; g < g_ngrp; g++) vk.vkCmdCopyBuffer(cb, g_grp[g].src, g_grp[g].dst, (uint32_t)g_grp[g].n, &g_runs[g_grp[g].first]); + shadow_barrier(cb, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT); + g_sh_in += (uint64_t)g_nruns; + g_nruns = g_ngrp = 0; + if (mt) vkb_pro_end(); +} +// Gathers the copies that bring p's copy up to date for submission sub (g_pool_mu held). 1: current once they are +// recorded (sync_flush); 0: p stays in game memory for now (a pass is open, copies are needed and may_end is 0) or +// for good (no_shadow). +static int sync_gather(pool_ent *p, uint64_t sub, int may_end) { + if (p->no_shadow) return 0; + // GPU writes of an earlier submission not yet in game memory (wb_merge): the copy is the only current one + if (p->wb_sub && p->wb_sub <= g_fence && !mtl_wait(p->wb_sub, 0)) may_end = 1; + if (!may_end && vkb_in_pass() && p->defer_pass == vkb_pass_serial()) return 0; // already found written in this pass + if (!p->sh) { + if (!g_shadow_budget) { // half of the largest device-local heap + for (uint32_t h = 0; h < vkc.memory.memoryHeapCount; h++) + if ((vkc.memory.memoryHeaps[h].flags & VK_MEMORY_HEAP_DEVICE_LOCAL_BIT) && vkc.memory.memoryHeaps[h].size / 2 > g_shadow_budget) + g_shadow_budget = vkc.memory.memoryHeaps[h].size / 2; + VLOG("copies of memory pools: up to %llu MiB of device memory", (unsigned long long)(g_shadow_budget >> 20)); + } + const char *why = !nvn_dirty_tracked((const void *)p->lo, p->hi - p->lo) ? "its writes are not tracked" + : vkb_shadow_bytes() + p->bytes > g_shadow_budget ? "over the device memory budget" + : !(p->sh = vkb_shadow_alloc(p->bytes, &p->sbuf, &p->soff)) ? "no device memory" : NULL; + if (why) { + p->no_shadow = 1; + static int said; + if (said++ < 16) VLOG("[temporary] memory pool %#llx+%#llx stays in game memory: %s", (unsigned long long)p->lo, (unsigned long long)(p->hi - p->lo), why); + return 0; + } + p->filled = 0; + } + uint64_t t0 = runtime_now_ns(); + int first = g_nruns, grp = g_ngrp; + uint64_t bytes0 = g_sh_in_bytes; + if (!p->filled) { + nvn_dirty_since((const void *)p->lo, p->hi - p->lo, UINT64_MAX); // writes so far are in the whole copy + run_put(p->buf, p->sbuf, 0, p->soff, p->bytes); + } else if (nvn_dirty_runs((const void *)p->lo, p->hi - p->lo, p->stamp, 65536, run_written, p) < 0) { + p->no_shadow = 1; // its writes can no longer be told + vkb_shadow_free(p->sh); p->sh = 0; + return 0; + } + uint64_t stamp = nvn_dirty_now(); + g_sh_scan_ns += runtime_now_ns() - t0; + if (g_nruns > first && vkb_in_pass() && !may_end) { // copies need the pass to end: game memory for the rest of it + g_sh_in_bytes = bytes0; + g_nruns = first; g_ngrp = grp; + if (grp) g_grp[grp - 1].n = first - g_grp[grp - 1].first; + p->defer_pass = vkb_pass_serial(); + g_sh_deferred++; + return 0; + } + p->filled = 1; + p->stamp = stamp; + p->sync_sub = sub; + return 1; +} +static void wb_add(pool_ent *p, uint64_t addr, uint64_t len) { // a GPU write to p's copy + p->wb_sub = g_fence + 1; + VkDeviceSize o = addr - p->base; + if (g_nwb && g_wb[g_nwb - 1].dst == p->buf && o <= g_wb[g_nwb - 1].doff + g_wb[g_nwb - 1].n && g_wb[g_nwb - 1].doff <= o + len) { + wb_t *w = &g_wb[g_nwb - 1]; // overlaps or touches the last one: merged + VkDeviceSize lo = w->doff < o ? w->doff : o, hi = w->doff + w->n > o + len ? w->doff + w->n : o + len; + w->soff -= w->doff - lo; w->doff = lo; w->n = hi - lo; + return; + } + if (g_nwb == g_capwb) { + g_capwb = g_capwb ? g_capwb * 2 : 256; + if (!(g_wb = realloc(g_wb, sizeof *g_wb * (size_t)g_capwb))) die("pool copies back"); + } + g_wb[g_nwb++] = (wb_t){p->sbuf, p->buf, p->soff + o, o, len, p->base, p->stamp}; +} +static void wb_forget(VkBuffer host) { // a pool deleted (g_pool_mu held): its writes have nowhere to go + for (int k = 0; k < g_nwb;) if (g_wb[k].dst == host) g_wb[k] = g_wb[--g_nwb]; else k++; +} +static int wb_cmp(const void *a, const void *b) { + const wb_t *x = a, *y = b; + if (x->dst != y->dst) return (uintptr_t)x->dst < (uintptr_t)y->dst ? -1 : 1; + return x->doff < y->doff ? -1 : x->doff > y->doff; +} +// The copy back goes through host memory of the submission's slot (wbst_t): the GPU copies the written ranges there, +// and when the submission has completed the retire thread copies them into game memory, page by page, except pages +// the CPU wrote since the copy was brought up to date (wb_merge). Copied straight into game memory, the whole range +// overwrote what the game's CPU wrote there meanwhile: about 45% of the ranges on the Great Sky Island (2026-10-24). +// The game's own writes are newer than the GPU's there (it writes a buffer for its next use after the GPU's), so the +// CPU's pages win. A slot is reused only after its submission has retired, so its host memory is free by then. +typedef struct { VkBuffer buf; VkDeviceMemory mem; uint8_t *map; VkDeviceSize cap; } wbst_t; +static wbst_t g_wbst[VKB_SLOTS]; +static uint64_t *g_wb_ranges; // per range of the last shadow_writeback: guest address, bytes, stamp, offset in the slot's memory +static int g_wb_nranges; +static uint64_t g_wb_kept_pages, g_wb_pages; // pages left to the CPU's writes / all pages copied back (the log line) +static uint8_t *wbst_get(int slot, VkDeviceSize need, VkBuffer *buf) { // the slot's host memory, at least need bytes + wbst_t *s = &g_wbst[slot]; + if (s->cap < need) { + if (s->buf) { vk.vkDestroyBuffer(vkc.device, s->buf, NULL); vk.vkFreeMemory(vkc.device, s->mem, NULL); } + *s = (wbst_t){0}; + VkDeviceSize cap = 1u << 20; + while (cap < need) cap *= 2; + VkBufferCreateInfo bi = {VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .size = cap, .usage = VK_BUFFER_USAGE_TRANSFER_DST_BIT}; + if (!vk_check(vk.vkCreateBuffer(vkc.device, &bi, NULL, &s->buf), "copy back buffer")) die("copy back buffer"); + VkMemoryRequirements req; + vk.vkGetBufferMemoryRequirements(vkc.device, s->buf, &req); + uint32_t type = UINT32_MAX, want = VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT; + for (int pass = 0; pass < 2 && type == UINT32_MAX; pass++) // cached first: the CPU reads it + for (uint32_t i = 0; i < vkc.memory.memoryTypeCount && type == UINT32_MAX; i++) { + VkMemoryPropertyFlags f = vkc.memory.memoryTypes[i].propertyFlags; + if ((req.memoryTypeBits & (1u << i)) && (f & want) == want && (pass || (f & VK_MEMORY_PROPERTY_HOST_CACHED_BIT))) type = i; + } + VkMemoryAllocateInfo ai = {VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO, .allocationSize = req.size, .memoryTypeIndex = type}; + if (type == UINT32_MAX || !vk_check(vk.vkAllocateMemory(vkc.device, &ai, NULL, &s->mem), "copy back memory")) die("copy back memory"); + vk.vkBindBufferMemory(vkc.device, s->buf, s->mem, 0); + void *p; + if (!vk_check(vk.vkMapMemory(vkc.device, s->mem, 0, VK_WHOLE_SIZE, 0, &p), "copy back map")) die("copy back map"); + s->map = p; s->cap = cap; + } + *buf = s->buf; + return s->map; +} +// Retire thread, the submission complete: its copied-back ranges into game memory, but for pages the CPU wrote since. +// These writes are the GPU's, not the CPU's (nvn_dirty_gpu_write_*): counted as the CPU's, a later submission's copy +// back to the same range would find them newer than its copy and keep them in place of its own GPU writes. +static void wb_put(uint64_t a, const uint8_t *src, uint64_t n) { + void *tok = nvn_dirty_gpu_write_begin((void *)(uintptr_t)a, n); + memcpy((void *)(uintptr_t)a, src, n); + nvn_dirty_gpu_write_end(tok); +} +static void wb_merge(const uint64_t *r, int n, const uint8_t *src) { + for (int k = 0; k < n; k++, r += 4) { + uint64_t a = r[0], len = r[1], stamp = r[2]; + if (!nvn_dirty_since((const void *)(uintptr_t)a, len, stamp)) { wb_put(a, src + r[3], len); g_wb_pages += (len + 4095) / 4096; continue; } + for (uint64_t p = a; p < a + len;) { // page by page + uint64_t e = ((p >> 12) + 1) << 12; + if (e > a + len) e = a + len; + g_wb_pages++; + if (nvn_dirty_since((const void *)(uintptr_t)p, e - p, stamp)) g_wb_kept_pages++; + else wb_put(p, src + r[3] + (p - a), e - p); + p = e; + } + } +} +static void shadow_writeback(VkCommandBuffer cb) { // end of a submission: GPU writes to the slot's host memory + vkb_rw_write(&g_pool_mu, LK_POOLS); + g_wb_ranges = NULL; g_wb_nranges = 0; + if (g_nwb) { + qsort(g_wb, (size_t)g_nwb, sizeof *g_wb, wb_cmp); + int m = 0; // overlapping ranges of one pool merged + for (int k = 1; k < g_nwb; k++) { + wb_t *w = &g_wb[m]; + if (g_wb[k].dst == w->dst && g_wb[k].doff <= w->doff + w->n) { + VkDeviceSize hi = g_wb[k].doff + g_wb[k].n; + if (hi > w->doff + w->n) w->n = hi - w->doff; + if (g_wb[k].stamp < w->stamp) w->stamp = g_wb[k].stamp; + } else g_wb[++m] = g_wb[k]; + } + g_nwb = m + 1; + g_wb_ranges = malloc(sizeof *g_wb_ranges * 4 * (size_t)g_nwb); + if (!g_wb_ranges) die("copy back ranges"); + VkDeviceSize total = 0; + for (int k = 0; k < g_nwb; k++) { + uint64_t *r = &g_wb_ranges[4 * k]; + r[0] = g_wb[k].base + g_wb[k].doff; r[1] = g_wb[k].n; r[2] = g_wb[k].stamp; r[3] = total; + total += (g_wb[k].n + 15) & ~(VkDeviceSize)15; + } + g_wb_nranges = g_nwb; + VkBuffer st; + wbst_get(vkb_slot(), total, &st); + vkb_checkpoint("pool copies out", NULL, NULL); + shadow_barrier(cb, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_WRITE_BIT, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_READ_BIT); + for (int k = 0; k < g_nwb;) { + int e = k; + VkBufferCopy r[64]; + int n = 0; + while (e < g_nwb && g_wb[e].src == g_wb[k].src && n < 64) { + r[n++] = (VkBufferCopy){g_wb[e].soff, g_wb_ranges[4 * e + 3], g_wb[e].n}; + g_sh_out_bytes += g_wb[e].n; + e++; + } + vk.vkCmdCopyBuffer(cb, g_wb[k].src, st, (uint32_t)n, r); + k = e; + } + shadow_barrier(cb, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT | VK_PIPELINE_STAGE_2_HOST_BIT, + VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT | VK_ACCESS_2_HOST_READ_BIT); + g_nwb = 0; + } + vkb_rw_write_end(&g_pool_mu); +} +// The first pass of a submission: the pools used in the two before are brought up to date together (one pair of +// barriers), so that draws in passes find them current. +static void shadow_presync(void) { + uint64_t sub = g_fence + 1; + if (!vkb_shadow_on() || __atomic_load_n(&g_presync_sub, __ATOMIC_ACQUIRE) == sub) return; + vkb_rw_write(&g_pool_mu, LK_POOLS); + if (g_presync_sub == sub) { vkb_rw_write_end(&g_pool_mu); return; } // another thread did it meanwhile + __atomic_store_n(&g_presync_sub, sub, __ATOMIC_RELEASE); + for (int i = 0; i < g_npools; i++) + if (g_pools[i].sh && g_pools[i].sync_sub != sub && g_pools[i].used_sub + 2 >= sub) sync_gather(&g_pools[i], sub, 1); + sync_flush(); + vkb_rw_write_end(&g_pool_mu); +} +static pool_ent *pool_find_mut(uintptr_t addr, uint64_t len); +void vkb_pool_use(uint64_t addr, uint64_t len, int write) { + if (g_ok != 1 || !addr || !len || !vkb_shadow_on()) return; + uint64_t sub = g_fence + 1; + if (!write) { // the common case: a read of a pool already current in this submission + vkb_rw_read(&g_pool_mu, LK_POOLS); + pool_ent *p = pool_find_mut((uintptr_t)addr, len); + int done = !p || p->no_shadow || p->sync_sub == sub; + if (p) __atomic_store_n(&p->used_sub, sub, __ATOMIC_RELAXED); + vkb_rw_read_end(&g_pool_mu); + if (done) return; + } + vkb_rw_write(&g_pool_mu, LK_POOLS); + pool_ent *p = pool_find_mut((uintptr_t)addr, len); + if (p) { + p->used_sub = sub; + // threads recording: the copies go into the prologue, before every chunk (no pass to end) + int ok = p->sync_sub == sub || sync_gather(p, sub, write || t_recording); + sync_flush(); + if (ok && write) wb_add(p, addr, len); + } + vkb_rw_write_end(&g_pool_mu); +} +int vkb_pool_addr(uint64_t addr, uint64_t len, VkBuffer *buf, VkDeviceSize *off) { + if (g_ok != 1) return 0; + vkb_rw_read(&g_pool_mu, LK_POOLS); + pool_ent *p = pool_find_mut((uintptr_t)addr, len); + if (p) { + uint64_t sub = g_fence + 1; + __atomic_store_n(&p->used_sub, sub, __ATOMIC_RELAXED); + if (p->sh && p->sync_sub == sub) { *buf = p->sbuf; *off = p->soff + (addr - p->base); } + else { *buf = p->buf; *off = addr - p->base; } + } + vkb_rw_read_end(&g_pool_mu); + return p != NULL; +} +static pool_ent *pool_find_mut(uintptr_t addr, uint64_t len) { return (pool_ent *)pool_find(addr, len); } + +int mtl_addr(uint64_t addr, uint64_t len, void **buf, uint64_t *off) { + if (g_ok != 1) { *buf = NULL; *off = 0; return 0; } + vkb_rw_read(&g_pool_mu, LK_POOLS); + const pool_ent *p = pool_find((uintptr_t)addr, len); + if (p) { *buf = (void *)p->buf; *off = addr - p->base; } + vkb_rw_read_end(&g_pool_mu); + return p != NULL; +} + +// ---------------- submissions ---------------- +typedef struct { int kind; volatile uint32_t *val; uint32_t mode, v; uint64_t addr, type; } post_op; // 1 event, 2 counter +typedef struct { VkCommandPool pool; VkCommandBuffer cb; uint64_t fence; } cmd_slot; +#define SLOTS VKB_SLOTS +static cmd_slot g_slots[SLOTS]; +static int g_nslots, g_cur = -1; +static post_op *g_post; +static int g_npost, g_capost; +static uint64_t g_op_count[OP_COUNT_]; +static int g_op_said[OP_COUNT_]; + +typedef struct job { uint64_t fence; post_op *ops; int n; int slot; uint64_t *wb; int nwb; struct job *next; } job; +static job *g_jobs, **g_jobs_tail = &g_jobs; + +static void run_posts(const post_op *ops, int n) { + for (int i = 0; i < n; i++) { + if (ops[i].kind == 1) { + if (ops[i].mode == 1) *ops[i].val += ops[i].v; + else *ops[i].val = ops[i].v; + } else { + // NVNcounterData {counter, timestamp}. Samples passed: [temporary] reported as visible until draws are replayed + // (with 0 the game culls every queried object); timestamps and other types report counter 0, as on Metal. + uint64_t rec[2] = {ops[i].type == NVN_COUNTER_TYPE_SAMPLES_PASSED ? 1u << 20 : 0, gpu_ticks()}; + guest_prepare_write((void *)(uintptr_t)ops[i].addr, sizeof rec); + memcpy((void *)(uintptr_t)ops[i].addr, rec, sizeof rec); + } + } +} + +static void cpend_done(uint64_t fence); // conditional rendering (below): the submission's counter reports are in memory +static void complete(uint64_t fence) { + pthread_mutex_lock(&g_mu); + if (fence > g_done) g_done = fence; + pthread_cond_broadcast(&g_cv); + pthread_mutex_unlock(&g_mu); +} + +// ---- TOTK_VK_CHECKPOINTS=1 (development): a marker before each draw, dispatch and copy names it; when the device is +// lost, the markers the GPU last reached (VK_NV_device_diagnostic_checkpoints) are reported with what they name. +enum { CKPT_RING = 1 << 16 }; +typedef struct { uint64_t frame; uint32_t n; const char *what; char k1[32], k2[32]; } ckpt_t; +static ckpt_t *g_ckpt; +static _Atomic uint32_t g_ckpt_n; +void vkb_checkpoint(const char *what, const char *k1, const char *k2) { + if (!vkc.has_checkpoints) return; + if (!g_ckpt && !(g_ckpt = calloc(CKPT_RING, sizeof *g_ckpt))) return; + uint32_t n = atomic_fetch_add(&g_ckpt_n, 1), i = n % CKPT_RING; + ckpt_t *c = &g_ckpt[i]; + c->frame = vkb_frame(); c->n = n; c->what = what; + snprintf(c->k1, sizeof c->k1, "%s", k1 ? k1 : "-"); + snprintf(c->k2, sizeof c->k2, "%s", k2 ? k2 : "-"); + vk.vkCmdSetCheckpointNV(vkb_cmd(), (void *)(uintptr_t)(i + 1)); +} +static void checkpoint_report(void) { + if (!vkc.has_checkpoints || !g_ckpt) return; + VkCheckpointDataNV d[16]; + uint32_t n = 16; + for (uint32_t i = 0; i < n; i++) d[i] = (VkCheckpointDataNV){VK_STRUCTURE_TYPE_CHECKPOINT_DATA_NV}; + vk_queue_lock(); + vk.vkGetQueueCheckpointDataNV(vkc.queue, &n, d); + vk_queue_unlock(); + VLOG("device lost: %u checkpoints reached (last %u recorded)", n, (unsigned)atomic_load(&g_ckpt_n)); + for (uint32_t i = 0; i < n; i++) { + uintptr_t m = (uintptr_t)d[i].pCheckpointMarker; + if (!m || m > CKPT_RING) continue; + const ckpt_t *c = &g_ckpt[m - 1]; + VLOG(" stage %#llx: #%u present %llu %s %s %s", (unsigned long long)d[i].stage, c->n, (unsigned long long)c->frame, c->what, c->k1, c->k2); + unsigned long long a = strtoull(c->k1, NULL, 16); + if (a) { // an address: what the memory and the pool table hold now + MEMORY_BASIC_INFORMATION mi = {0}; + VirtualQuery((void *)(uintptr_t)a, &mi, sizeof mi); + vkb_rw_read(&g_pool_mu, LK_POOLS); + const pool_ent *p = pool_find((uintptr_t)a, 1); + VLOG(" memory %#llx: state %#lx protect %#lx region %p+%#llx; pool %s %#llx..%#llx", a, mi.State, mi.Protect, mi.BaseAddress, + (unsigned long long)mi.RegionSize, p ? "registered" : "none", p ? (unsigned long long)p->lo : 0ull, p ? (unsigned long long)p->hi : 0ull); + vkb_rw_read_end(&g_pool_mu); + } + } +} + +// ---- deferred destruction: objects given up while a submission is recorded may be used by its earlier commands (a +// texture view replaced in the middle of a frame, a texture the game finalizes). Destroying them after waiting for the +// last submitted work (as before) freed what the open submission still referred to: the GPU read a destroyed view and +// the device was lost (TOTK's item-get screen, 2026-10-24). Now each waits for the first submission after it was given up. +typedef struct dead { uint64_t fence; VkImageView view; VkImage img; VkDeviceMemory mem; struct dead *next; } dead; +static dead *g_dead_new, *g_dead; // given up since the last submission (no fence yet) / waiting for their fence +static pthread_mutex_t g_dead_mu = PTHREAD_MUTEX_INITIALIZER; +void vkb_destroy_later(VkImageView view, VkImage img, VkDeviceMemory mem) { + dead *d = malloc(sizeof *d); + if (!d) die("deferred destruction"); + *d = (dead){0, view, img, mem, NULL}; + pthread_mutex_lock(&g_dead_mu); + d->next = g_dead_new; g_dead_new = d; + pthread_mutex_unlock(&g_dead_mu); +} +static void dead_tag(uint64_t fence) { // a submission: what was given up so far waits for it + pthread_mutex_lock(&g_dead_mu); + while (g_dead_new) { dead *d = g_dead_new; g_dead_new = d->next; d->fence = fence; d->next = g_dead; g_dead = d; } + pthread_mutex_unlock(&g_dead_mu); +} +static void dead_reap(uint64_t done) { // the GPU finished up to `done` + dead *free_list = NULL; + pthread_mutex_lock(&g_dead_mu); + for (dead **p = &g_dead; *p;) { + if ((*p)->fence <= done) { dead *d = *p; *p = d->next; d->next = free_list; free_list = d; } else p = &(*p)->next; + } + pthread_mutex_unlock(&g_dead_mu); + while (free_list) { + dead *d = free_list; free_list = d->next; + if (d->view) vk.vkDestroyImageView(vkc.device, d->view, NULL); + if (d->img) vk.vkDestroyImage(vkc.device, d->img, NULL); + if (d->mem) vk.vkFreeMemory(vkc.device, d->mem, NULL); + free(d); + } +} + +static void gpu_span(int slot, uint64_t fence); +// Retires submissions in order: waits for the GPU, writes their events and counters, completes their fence. +static void *retire_thread(void *unused) { + (void)unused; + pthread_setname_np("totk-vk-retire"); + for (;;) { + pthread_mutex_lock(&g_mu); + while (!g_jobs) pthread_cond_wait(&g_cv, &g_mu); + job *j = g_jobs; + g_jobs = j->next; + if (!g_jobs) g_jobs_tail = &g_jobs; + pthread_mutex_unlock(&g_mu); + VkSemaphoreWaitInfo wi = {VK_STRUCTURE_TYPE_SEMAPHORE_WAIT_INFO, .semaphoreCount = 1, .pSemaphores = &g_timeline, + .pValues = &j->fence}; + VkResult r = vk.vkWaitSemaphores(vkc.device, &wi, UINT64_MAX); + if (r != VK_SUCCESS) { vk_check(r, "vkWaitSemaphores"); checkpoint_report(); die("GPU submission %llu lost", (unsigned long long)j->fence); } + if (totk_tl_on()) gpu_span(j->slot, j->fence); // TOTK_TIMELINE: when the GPU ran this command buffer + if (j->wb) { wb_merge(j->wb, j->nwb, g_wbst[j->slot].map); free(j->wb); } // GPU writes into game memory + run_posts(j->ops, j->n); + cpend_done(j->fence); + complete(j->fence); + dead_reap(j->fence); + free(j->ops); + free(j); + } + return NULL; +} +void retire_start(void) { + pthread_t t; + if (pthread_create(&t, NULL, retire_thread, NULL)) die("retire thread"); +} + +// GPU time of every submission (a timestamp at its start and end), summed per frame for the per-frame log line, and the +// time the recording thread waits for a free command slot (the GPU is behind). +static VkQueryPool g_tsq; +static int g_ts_used[SLOTS]; +static uint64_t g_gpu_ns, g_slot_wait_ns; +static void gpu_time_begin(int slot) { // into the slot's own command buffer (the submission's first) + if (!g_tsq) { + VkQueryPoolCreateInfo ci = {VK_STRUCTURE_TYPE_QUERY_POOL_CREATE_INFO, .queryType = VK_QUERY_TYPE_TIMESTAMP, .queryCount = 2 * SLOTS}; + if (!vk_check(vk.vkCreateQueryPool(vkc.device, &ci, NULL, &g_tsq), "timestamp pool")) return; + vk.vkResetQueryPool(vkc.device, g_tsq, 0, 2 * SLOTS); + } + if (g_ts_used[slot]) { // the slot's previous submission is complete + uint64_t t[2]; + if (vk.vkGetQueryPoolResults(vkc.device, g_tsq, (uint32_t)(2 * slot), 2, sizeof t, t, sizeof t[0], VK_QUERY_RESULT_64_BIT) == VK_SUCCESS && t[1] > t[0]) + g_gpu_ns += (uint64_t)((double)(t[1] - t[0]) * vkc.props.limits.timestampPeriod); + g_ts_used[slot] = 0; + } + vk.vkCmdResetQueryPool(g_slots[slot].cb, g_tsq, (uint32_t)(2 * slot), 2); + vk.vkCmdWriteTimestamp2(g_slots[slot].cb, VK_PIPELINE_STAGE_2_TOP_OF_PIPE_BIT, g_tsq, (uint32_t)(2 * slot)); +} +// The GPU span of a completed command buffer on the host clock: it ended about when the retire thread saw its fence +// (the thread waits on fences in order), and lasted what its two timestamps say. +static void gpu_span(int slot, uint64_t fence) { + uint64_t end = runtime_now_ns(), t[2]; + if (!g_tsq || slot < 0) return; + if (vk.vkGetQueryPoolResults(vkc.device, g_tsq, (uint32_t)(2 * slot), 2, sizeof t, t, sizeof t[0], VK_QUERY_RESULT_64_BIT) != VK_SUCCESS || t[1] <= t[0]) return; + uint64_t dur = (uint64_t)((double)(t[1] - t[0]) * vkc.props.limits.timestampPeriod); + totk_tl_at(TL_GPU, end > dur ? end - dur : 0, fence, end); +} +static void gpu_time_end(int slot, VkCommandBuffer cb) { // into the submission's last command buffer + if (!g_tsq) return; + vk.vkCmdWriteTimestamp2(cb, VK_PIPELINE_STAGE_2_BOTTOM_OF_PIPE_BIT, g_tsq, (uint32_t)(2 * slot + 1)); + g_ts_used[slot] = 1; +} + +// A submission whose recording took over 100 ms: where the time went (log line "long submission"). The game waits for +// its recording at the present: these are the hitches when new parts of the world arrive. +extern uint64_t vkb_astc_ns, vkb_shader_load_ns; +static uint64_t g_sub_t0, g_sub_snap[6]; +static void sub_snap(uint64_t v[6]) { + v[0] = vkb_step_ns[PK_CREATE]; v[1] = vkb_step_ns[PK_UPLOAD]; v[2] = vkb_astc_ns; v[3] = vkb_shader_load_ns; + v[4] = g_sh_scan_ns; v[5] = g_sh_in_bytes; +} +static void sub_long_check(uint64_t fence) { + uint64_t now = runtime_now_ns(), v[6]; + if (!g_sub_t0 || now - g_sub_t0 < 100000000ull) return; + sub_snap(v); + static int said; + if (said++ < 64) + VLOG("long submission %llu: recorded in %.0f ms: new images %.0f ms, uploads %.0f ms (ASTC %.0f ms), shaders made %.0f ms, " + "pool copy checks %.0f ms, pool copies in %.1f MiB; draws %llu", (unsigned long long)fence, (now - g_sub_t0) / 1e6, + (v[0] - g_sub_snap[0]) / 1e6, (v[1] - g_sub_snap[1]) / 1e6, (v[2] - g_sub_snap[2]) / 1e6, (v[3] - g_sub_snap[3]) / 1e6, + (v[4] - g_sub_snap[4]) / 1e6, (v[5] - g_sub_snap[5]) / 1048576.0, (unsigned long long)vkb_draw_count()); +} + +void mtl_sub_begin(void) { + backend_init(); + g_sub_t0 = runtime_now_ns(); + sub_snap(g_sub_snap); + g_npost = 0; + if (g_ok != 1) return; + pthread_mutex_lock(&g_mu); + uint64_t done = g_done; + pthread_mutex_unlock(&g_mu); + int pick = -1; + for (int i = 0; i < g_nslots; i++) if (g_slots[i].fence <= done) { pick = i; break; } + if (pick < 0 && g_nslots < SLOTS) { + VkCommandPoolCreateInfo pi = {VK_STRUCTURE_TYPE_COMMAND_POOL_CREATE_INFO, .flags = VK_COMMAND_POOL_CREATE_TRANSIENT_BIT, + .queueFamilyIndex = vkc.family}; + cmd_slot *s = &g_slots[g_nslots]; + if (!vk_check(vk.vkCreateCommandPool(vkc.device, &pi, NULL, &s->pool), "command pool")) die("command pool"); + VkCommandBufferAllocateInfo ai = {VK_STRUCTURE_TYPE_COMMAND_BUFFER_ALLOCATE_INFO, .commandPool = s->pool, + .level = VK_COMMAND_BUFFER_LEVEL_PRIMARY, .commandBufferCount = 1}; + if (!vk_check(vk.vkAllocateCommandBuffers(vkc.device, &ai, &s->cb), "command buffer")) die("command buffer"); + pick = g_nslots++; + } + if (pick < 0) { // all in flight: the oldest + pick = 0; + for (int i = 1; i < g_nslots; i++) if (g_slots[i].fence < g_slots[pick].fence) pick = i; + uint64_t t0 = runtime_now_ns(); + mtl_wait(g_slots[pick].fence, UINT64_MAX); + g_slot_wait_ns += runtime_now_ns() - t0; + } + g_cur = pick; + nvn_dirty_epoch_begin(); // write-watch scans: once per range in this submission (nvn_dirty_win.c) + vkb_slot_reset(pick); // its previous submission is complete: upload memory and descriptors are free + vkb_mt_slot_reset(pick); + vk.vkResetCommandPool(vkc.device, g_slots[pick].pool, 0); + VkCommandBufferBeginInfo bi = {VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO, .flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT}; + vk.vkBeginCommandBuffer(g_slots[pick].cb, &bi); + gpu_time_begin(pick); +} + +static void post(post_op o) { + if (g_npost == g_capost) { + g_capost = g_capost ? g_capost * 2 : 64; + g_post = realloc(g_post, (size_t)g_capost * sizeof *g_post); + if (!g_post) die("post ops"); + } + g_post[g_npost++] = o; +} +void mtl_sub_event(volatile uint32_t *val, uint32_t mode, uint32_t v) { post((post_op){1, val, mode, v, 0, 0}); } +static uint64_t g_counter_types[32]; // reports by NVN counter type (the per-frame log line) +// ---------- Conditional rendering (NVN SetRenderEnableConditional), evaluated in GPU order ---------- +// Ported from MacNativeGames nvn_mtl.m (cpend_*, cond_eval; MIT). TOTK gates its ClearColor commands with it: the CPU +// writes counter=1 to request a clear and the TIMESTAMP_TOP report replayed right after the clear resets it to 0, so the +// clear runs once. Counter reports replayed earlier whose memory write has not happened yet (run_posts writes it when +// the submission retires) are kept here with the value they will write: 0 for timestamps and other types, unknown for +// samples passed. A conditional looks there first, then at game memory - the counter values the GPU would see. +// Replaying every gated clear (as before 2026-10-24) cleared light and history targets on frames the game had not +// asked for: menu and world lighting went black for a few frames after camera cuts. +typedef struct { uint64_t addr, fence, val; int known; } cpend_ent; +#define CPEND_N 4096 +static cpend_ent g_cpend[CPEND_N]; +static unsigned g_cpend_head, g_cpend_tail; +static pthread_mutex_t g_cpend_mu = PTHREAD_MUTEX_INITIALIZER; // replay thread pushes, retire thread clears +static void cpend_push(uint64_t addr, uint64_t fence, uint64_t val, int known) { + pthread_mutex_lock(&g_cpend_mu); + g_cpend[g_cpend_head % CPEND_N] = (cpend_ent){addr, fence, val, known}; + g_cpend_head++; + pthread_mutex_unlock(&g_cpend_mu); +} +static void cpend_done(uint64_t fence) { // the reports of this submission are in memory now + pthread_mutex_lock(&g_cpend_mu); + if (g_cpend_head - g_cpend_tail > CPEND_N) g_cpend_tail = g_cpend_head - CPEND_N; + for (unsigned i = g_cpend_tail; i != g_cpend_head; i++) + if (g_cpend[i % CPEND_N].addr && g_cpend[i % CPEND_N].fence <= fence) g_cpend[i % CPEND_N].addr = 0; + while (g_cpend_tail != g_cpend_head && !g_cpend[g_cpend_tail % CPEND_N].addr) g_cpend_tail++; + pthread_mutex_unlock(&g_cpend_mu); +} +static int cpend_lookup(uint64_t addr, uint64_t *val) { // 1 known (in *val), 0 pending but unknown, -1 none (read memory) + int r = -1; + pthread_mutex_lock(&g_cpend_mu); + unsigned n = g_cpend_head - g_cpend_tail; + if (n > CPEND_N) n = CPEND_N; + for (unsigned k = 0; k < n; k++) { // the latest report to the address + const cpend_ent *e = &g_cpend[(g_cpend_head - 1 - k) % CPEND_N]; + if (e->addr == addr) { r = e->known; *val = e->val; break; } + } + pthread_mutex_unlock(&g_cpend_mu); + return r; +} +static vkb_cnt g_cond_skipped; // draws and clears not replayed (log line) +static uint64_t g_cond_unknown; // conditionals taken as enabled (log line; the walk / one thread) +static int cond_eval(const p_render_cond *c) { // 1: rendering enabled + uint64_t x = 0, y = 0; + int kx = cpend_lookup(c->addr, &x), ky = cpend_lookup(c->addr + 16, &y); + void *bf; uint64_t off; + if (kx == 0 || ky == 0 || !mtl_addr(c->addr, 32, &bf, &off)) { g_cond_unknown++; return 1; } + if (kx < 0) memcpy(&x, (const void *)(uintptr_t)c->addr, 8); + if (ky < 0) memcpy(&y, (const void *)(uintptr_t)(c->addr + 16), 8); + return c->mode == 0 ? x == y : x != y; +} +void mtl_sub_counter(uint64_t addr, uint32_t type) { + g_counter_types[type & 31]++; + cpend_push(addr, g_fence + 1, 0, type != NVN_COUNTER_TYPE_SAMPLES_PASSED); // the counter word run_posts will write + post((post_op){2, NULL, 0, 0, addr, type}); +} + +// Everything before is visible to everything after (step 1: transfers only; passes get precise barriers later). +static void full_barrier(VkCommandBuffer cb) { + VkMemoryBarrier2 mb = {VK_STRUCTURE_TYPE_MEMORY_BARRIER_2, .srcStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + .srcAccessMask = VK_ACCESS_2_MEMORY_WRITE_BIT, .dstStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + .dstAccessMask = VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT}; + VkDependencyInfo di = {VK_STRUCTURE_TYPE_DEPENDENCY_INFO, .memoryBarrierCount = 1, .pMemoryBarriers = &mb}; + vk.vkCmdPipelineBarrier2(cb, &di); +} + +// TOTK_WATCH=
:: GPU-side writes into that range (buffer copies, inline uniform updates) are logged too. +static int watched(uint64_t dst, uint64_t n) { + static uint64_t lo, hi; + static int init; + if (!init) { + init = 1; + const char *e = getenv("TOTK_WATCH"); + if (e) { char *end; lo = strtoull(e, &end, 0); hi = *end == ':' ? lo + strtoull(end + 1, NULL, 0) : lo; } + } + return hi > lo && dst < hi && lo < dst + n; +} + +void mtl_sub_copy(uint64_t src, uint64_t dst, uint64_t n) { + if (watched(dst, n)) { + static int said; + if (said++ < 32) VLOG("[watch] present %llu: GPU copy %#llx -> %#llx (%llu bytes)", (unsigned long long)vkb_frame(), + (unsigned long long)src, (unsigned long long)dst, (unsigned long long)n); + } + if (g_ok != 1 || !n) { if (g_ok != 1 && n) memmove((void *)(uintptr_t)dst, (const void *)(uintptr_t)src, n); return; } + pass_end_why(PE_COPY); + vkb_mirrors_evict(src, n); // ranges that straddle a mirror: game memory first + vkb_mirrors_evict(dst, n); + vkb_pool_use(src, n, 0); + vkb_pool_use(dst, n, 1); + vkb_ucap_drop(dst, n); + VkBuffer sb, db; VkDeviceSize so, dof; + int from_cap = vkb_ucap_range(src, n, &sb, &so); // the source as of the submit call + if (watched(dst, n)) { + float cv[4] = {0}, mv[4] = {0}; + vkb_ucap_read(src + 0x30, cv, 16); + memcpy(mv, (const void *)(uintptr_t)(src + 0x30), 16); + static int said; + if (vkb_frame() >= 330 && said++ < 64) VLOG("[watch] present %llu: copy %#llx -> %#llx from the capture %d (why %d); source +0x30 captured %g %g %g %g, memory now %g %g %g %g", + (unsigned long long)vkb_frame(), (unsigned long long)src, (unsigned long long)dst, from_cap, vkb_ucap_why(src, n), cv[0], cv[1], cv[2], cv[3], mv[0], mv[1], mv[2], mv[3]); + } + if ((!from_cap && !vkb_gpu_addr(src, n, &sb, &so)) || !vkb_gpu_addr(dst, n, &db, &dof)) { + vkb_audit(A_COPY_SKIPPED, NULL); + static int said; if (said++ < 8) VLOG("copy %#llx -> %#llx (%llu bytes) outside the memory pools: skipped", + (unsigned long long)src, (unsigned long long)dst, (unsigned long long)n); + return; + } + pass_end_why(PE_COPY); + VkCommandBuffer cb = vkb_cmd(); + full_barrier(cb); + VkBufferCopy r = {so, dof, n}; + vkb_checkpoint("copy buffer", NULL, NULL); + vk.vkCmdCopyBuffer(cb, sb, db, 1, &r); + full_barrier(cb); +} + +static void update_buffer(uint64_t dst, uint64_t n, const uint8_t *data) { + if (watched(dst, n)) { + static int said; + float f[4] = {0}; + memcpy(f, data, n < 16 ? n : 16); + if (said++ < 32) VLOG("[watch] present %llu: uniform update %#llx+%llu from command memory %p: %g %g %g %g", + (unsigned long long)vkb_frame(), (unsigned long long)dst, (unsigned long long)n, + (const void *)data, f[0], f[1], f[2], f[3]); + } + if (g_ok != 1) { memcpy((void *)(uintptr_t)dst, data, n); return; } + if (!n) return; + pass_end_why(PE_UPDATE); + vkb_mirrors_evict(dst, n); + vkb_pool_use(dst, n, 1); + vkb_ucap_drop(dst, n); // an inline uniform update: later draws read the updated memory + VkBuffer db; VkDeviceSize off; + if (!vkb_gpu_addr(dst, n, &db, &off)) return; + VkCommandBuffer cb = vkb_cmd(); + full_barrier(cb); + // vkCmdUpdateBuffer: 4-byte aligned, at most 65536 bytes per call + if (!(off & 3) && !(n & 3)) { + for (uint64_t k = 0; k < n; k += 65536) vk.vkCmdUpdateBuffer(cb, db, off + k, n - k < 65536 ? n - k : 65536, data + k); + } else { + vkb_audit(A_UPDATE_SKIPPED, NULL); + static int said; if (said++ < 4) VLOG("[temporary] unaligned uniform update %#llx+%llu: skipped", (unsigned long long)dst, (unsigned long long)n); + } + full_barrier(cb); +} + +// ---------------- render passes (milestone 3.3 step 3) ---------------- +// SetRenderTargets only records the targets; the pass (dynamic rendering) begins at the first command that renders and +// ends before anything that is not a draw or a clear (copies, other passes, the end of the submission). +static __thread p_set_rt g_rt; // current render targets (per recording thread) +static _Atomic uint64_t g_ticket; // presents so far +static __thread int g_in_pass; +static __thread uint32_t g_pass_fb; // the current pass's feedback targets (vkb.h) +static __thread VkExtent2D g_pass_extent; + +VkCommandBuffer vkb_cmd(void) { return t_cb ? t_cb : g_cur >= 0 ? g_slots[g_cur].cb : VK_NULL_HANDLE; } +int vkb_slot(void) { return g_cur; } +int vkb_in_pass(void) { return g_in_pass; } +const p_set_rt *vkb_rt(void) { return &g_rt; } +uint64_t vkb_frame(void) { return atomic_load(&g_ticket); } +int vkb_rt_count(void) { return g_rt.n < 0 ? 0 : g_rt.n > 8 ? 8 : g_rt.n; } +VkExtent2D vkb_rt_extent(void) { return g_pass_extent; } +int vkb_ready(void) { return g_ok == 1; } + +// ---- TOTK_GPU_PASSPROF=: GPU time of every render pass of that frame (a timestamp at its start and end; work +// between passes - copies, dispatches, barriers - as gaps). One timestamp pair per pass, not per draw: per-draw +// timestamps serialise the GPU and inflate small draws. The report names each pass by its targets, draws and first +// fragment shader, sorted by time (development). +enum { PP_MAX = 4096 }; +typedef struct { uint32_t w, h, ncolor, fmt0, depth, draws; uint64_t draws0; char key[16]; } pp_info; +static VkQueryPool g_ppq; +static pp_info g_pp[PP_MAX]; +static int g_ppn, g_pp_done, g_pp_open = -1; +static long long pp_frame(void) { + static long long at = -1; + if (at < 0) { const char *e = getenv("TOTK_GPU_PASSPROF"); at = e ? atoll(e) : 0; } + return at; +} +static int pp_on(void) { return pp_frame() > 0 && !g_pp_done && vkb_frame() == (uint64_t)pp_frame(); } +void vkb_passprof_note(const char *key) { // vkb_draw.c: the first fragment shader of the open pass + if (g_pp_open >= 0 && !g_pp[g_pp_open].key[0] && key) snprintf(g_pp[g_pp_open].key, sizeof g_pp[0].key, "%.12s", key); +} +static void pp_begin(VkCommandBuffer cb, VkExtent2D ext) { + if (!pp_on() || g_ppn >= PP_MAX / 2) return; + if (!g_ppq) { + VkQueryPoolCreateInfo ci = {VK_STRUCTURE_TYPE_QUERY_POOL_CREATE_INFO, .queryType = VK_QUERY_TYPE_TIMESTAMP, .queryCount = PP_MAX}; + if (!vk_check(vk.vkCreateQueryPool(vkc.device, &ci, NULL, &g_ppq), "pass timestamp pool")) { g_pp_done = 1; return; } + vk.vkResetQueryPool(vkc.device, g_ppq, 0, PP_MAX); + } + pp_info *p = &g_pp[g_ppn]; + memset(p, 0, sizeof *p); + p->w = ext.width; p->h = ext.height; + p->ncolor = (uint32_t)(g_rt.n < 0 ? 0 : g_rt.n); + p->fmt0 = g_rt.n > 0 && g_rt.color[0] ? (uint32_t)g_rt.color[0]->d.format : 0; + p->depth = g_rt.depth ? (uint32_t)g_rt.depth->d.format : 0; + p->draws0 = vkb_draw_count(); + vk.vkCmdWriteTimestamp2(cb, VK_PIPELINE_STAGE_2_TOP_OF_PIPE_BIT, g_ppq, (uint32_t)(2 * g_ppn)); + g_pp_open = g_ppn++; +} +static void pp_end(VkCommandBuffer cb) { + if (g_pp_open < 0) return; + g_pp[g_pp_open].draws = (uint32_t)(vkb_draw_count() - g_pp[g_pp_open].draws0); + vk.vkCmdWriteTimestamp2(cb, VK_PIPELINE_STAGE_2_BOTTOM_OF_PIPE_BIT, g_ppq, (uint32_t)(2 * g_pp_open + 1)); + g_pp_open = -1; +} +static int pp_cmp(const void *a, const void *b) { + double x = ((const double *)a)[0], y = ((const double *)b)[0]; + return x < y ? 1 : x > y ? -1 : 0; +} +static void pp_report(void) { // at a present after the frame: wait for the GPU, read, print + if (!g_ppn || g_pp_done || pp_frame() <= 0 || vkb_frame() <= (uint64_t)pp_frame()) return; + g_pp_done = 1; + mtl_finish(); + uint64_t *ts = malloc(sizeof *ts * 2 * (size_t)g_ppn); + double (*row)[2] = malloc(sizeof *row * (size_t)g_ppn); + if (!ts || !row) return; + VkResult r = vk.vkGetQueryPoolResults(vkc.device, g_ppq, 0, (uint32_t)(2 * g_ppn), sizeof *ts * 2 * (size_t)g_ppn, ts, sizeof *ts, + VK_QUERY_RESULT_64_BIT); + if (r != VK_SUCCESS) { VLOG("pass profile: results %d", (int)r); return; } + double per = vkc.props.limits.timestampPeriod / 1e6, inpass = 0, gaps = 0; + for (int i = 0; i < g_ppn; i++) { + row[i][0] = ts[2 * i + 1] > ts[2 * i] ? (double)(ts[2 * i + 1] - ts[2 * i]) * per : 0; + row[i][1] = i; + inpass += row[i][0]; + if (i && ts[2 * i] > ts[2 * i - 1]) gaps += (double)(ts[2 * i] - ts[2 * i - 1]) * per; + } + double span = (double)(ts[2 * g_ppn - 1] - ts[0]) * per; + VLOG("pass profile of present %lld: %d passes, %.2f ms in passes, %.2f ms between them, %.2f ms first to last", + pp_frame(), g_ppn, inpass, gaps, span); + qsort(row, (size_t)g_ppn, sizeof *row, pp_cmp); + for (int k = 0; k < g_ppn && k < 30; k++) { + const pp_info *p = &g_pp[(int)row[k][1]]; + VLOG(" %7.3f ms pass %4d %ux%u %u colour (fmt %#x) depth %#x %u draws fs %s", row[k][0], (int)row[k][1], p->w, p->h, + p->ncolor, p->fmt0, p->depth, p->draws, p->key[0] ? p->key : "-"); + } + free(ts); free(row); +} + +static void pass_end(void) { + if (!g_in_pass) return; + vk.vkCmdEndRendering(vkb_cmd()); + pp_end(vkb_cmd()); + g_in_pass = 0; +} + +static uint32_t level_size(uint32_t v, const view_t *view) { + uint32_t base = view && view->set_levels ? view->base_level : 0, s = v >> base; + return s ? s : 1; +} + +// 0 when the targets cannot be rendered to (no image yet for a format). Without any target the pass has no attachments +// and covers `bare` (the draw's viewport): TOTK draws with no targets whose fragment shaders write storage buffers (the +// ambient light from the environment cube map goes to the scene buffer that the lighting reads). +// fb: targets that the pass's draws also sample (vkb.h): TOTK's G-buffer decals read the G-buffer they draw into and +// effects read the depth they test against, as on NVN where a bound target may be sampled. +static __thread uint64_t g_pass_serial; // passes begun on this thread (vkb_pass_serial: per-pass state of the draws) +uint64_t vkb_pass_serial(void) { return g_pass_serial; } +static void shadow_presync(void); +static int pass_begin(VkExtent2D bare, uint32_t fb) { + if (g_in_pass) return 1; + shadow_presync(); + g_pass_serial++; + if (!vkc.has_feedback_loop) fb = 0; + g_pass_fb = fb; + VkCommandBuffer cb = vkb_cmd(); + if (vkb_stores_take()) { // storage writes of earlier draws are visible to this pass (vkb_draw.c g_stores_pending) + VkMemoryBarrier2 mb = {VK_STRUCTURE_TYPE_MEMORY_BARRIER_2, + .srcStageMask = VK_PIPELINE_STAGE_2_VERTEX_SHADER_BIT | VK_PIPELINE_STAGE_2_FRAGMENT_SHADER_BIT, + .srcAccessMask = VK_ACCESS_2_SHADER_STORAGE_WRITE_BIT, + .dstStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + .dstAccessMask = VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT}; + VkDependencyInfo dep = {VK_STRUCTURE_TYPE_DEPENDENCY_INFO, .memoryBarrierCount = 1, .pMemoryBarriers = &mb}; + vk.vkCmdPipelineBarrier2(cb, &dep); + } + VkRenderingAttachmentInfo color[8] = {0}, depth = {0}, stencil = {0}; + uint32_t w = UINT32_MAX, h = UINT32_MAX, layers = 1; + int ncolor = g_rt.n < 0 ? 0 : g_rt.n > 8 ? 8 : g_rt.n, any = 0; + for (int i = 0; i < ncolor; i++) { + color[i].sType = VK_STRUCTURE_TYPE_RENDERING_ATTACHMENT_INFO; + vkb_image *im = g_rt.color[i] ? vkb_image_of(g_rt.color[i]) : NULL; + const view_t *v = g_rt.has_cview[i] ? &g_rt.cview[i] : NULL; + if (im && im->refresh_from) vkb_tex_refresh(cb, im); // newer contents at this memory in another size (not in a pass yet) + if (!im || im->aspect != VK_IMAGE_ASPECT_COLOR_BIT || !vkb_view(im, v, 1, -1)) { + if (im) vkb_audit(A_RT_NOT_ATTACHABLE, NULL); + if (im) { static int said; if (said++ < 4) VLOG("[temporary] texture %llu (format %#x) as a render target: not attachable (compressed)", + (unsigned long long)im->serial, im->d.format); } + continue; + } + int loop = (fb >> i) & 1; + VkImageLayout layout = loop ? VK_IMAGE_LAYOUT_ATTACHMENT_FEEDBACK_LOOP_OPTIMAL_EXT : VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL; + vkb_image_use(cb, im, layout, VK_PIPELINE_STAGE_2_COLOR_ATTACHMENT_OUTPUT_BIT | (loop ? VK_PIPELINE_STAGE_2_ALL_GRAPHICS_BIT : 0), + VK_ACCESS_2_COLOR_ATTACHMENT_READ_BIT | VK_ACCESS_2_COLOR_ATTACHMENT_WRITE_BIT | (loop ? VK_ACCESS_2_SHADER_SAMPLED_READ_BIT : 0)); + color[i].imageView = vkb_view(im, v, 1, -1); + vkb_image_written(im); + color[i].imageLayout = layout; + color[i].loadOp = VK_ATTACHMENT_LOAD_OP_LOAD; + color[i].storeOp = VK_ATTACHMENT_STORE_OP_STORE; + uint32_t lw = level_size(im->width, v), lh = level_size(im->height, v); + if (lw < w) w = lw; + if (lh < h) h = lh; + any = 1; + } + vkb_image *dim = g_rt.depth ? vkb_image_of(g_rt.depth) : NULL; + if (dim && dim->refresh_from) vkb_tex_refresh(cb, dim); + if (dim && (dim->aspect & (VK_IMAGE_ASPECT_DEPTH_BIT | VK_IMAGE_ASPECT_STENCIL_BIT))) { + const view_t *v = g_rt.has_dview ? &g_rt.dview : NULL; + int loop = (fb & VKB_FB_DEPTH) != 0; + VkImageLayout layout = loop ? VK_IMAGE_LAYOUT_ATTACHMENT_FEEDBACK_LOOP_OPTIMAL_EXT : VK_IMAGE_LAYOUT_DEPTH_STENCIL_ATTACHMENT_OPTIMAL; + vkb_image_use(cb, dim, layout, + VK_PIPELINE_STAGE_2_EARLY_FRAGMENT_TESTS_BIT | VK_PIPELINE_STAGE_2_LATE_FRAGMENT_TESTS_BIT | (loop ? VK_PIPELINE_STAGE_2_ALL_GRAPHICS_BIT : 0), + VK_ACCESS_2_DEPTH_STENCIL_ATTACHMENT_READ_BIT | VK_ACCESS_2_DEPTH_STENCIL_ATTACHMENT_WRITE_BIT | (loop ? VK_ACCESS_2_SHADER_SAMPLED_READ_BIT : 0)); + VkRenderingAttachmentInfo a = {VK_STRUCTURE_TYPE_RENDERING_ATTACHMENT_INFO, .imageView = vkb_view(dim, v, 1, -1), + .imageLayout = layout, .loadOp = VK_ATTACHMENT_LOAD_OP_LOAD, + .storeOp = VK_ATTACHMENT_STORE_OP_STORE}; + vkb_image_written(dim); + if (dim->aspect & VK_IMAGE_ASPECT_DEPTH_BIT) depth = a; + if (dim->aspect & VK_IMAGE_ASPECT_STENCIL_BIT) stencil = a; + uint32_t lw = level_size(dim->width, v), lh = level_size(dim->height, v); + if (lw < w) w = lw; + if (lh < h) h = lh; + any = 1; + } + if (!any) { + if (ncolor || !bare.width || !bare.height) return 0; // targets that cannot be attached: not drawn + w = bare.width < vkc.props.limits.maxFramebufferWidth ? bare.width : vkc.props.limits.maxFramebufferWidth; + h = bare.height < vkc.props.limits.maxFramebufferHeight ? bare.height : vkc.props.limits.maxFramebufferHeight; + } + g_pass_extent = (VkExtent2D){w, h}; + VkRenderingInfo ri = {VK_STRUCTURE_TYPE_RENDERING_INFO, .renderArea = {{0, 0}, {w, h}}, .layerCount = layers, + .colorAttachmentCount = (uint32_t)ncolor, .pColorAttachments = color, + .pDepthAttachment = depth.imageView ? &depth : NULL, .pStencilAttachment = stencil.imageView ? &stencil : NULL}; + vkb_checkpoint("begin pass", NULL, NULL); + pp_begin(cb, g_pass_extent); + vk.vkCmdBeginRendering(cb, &ri); + g_in_pass = 1; + return 1; +} + +// TOTK_VK_TRACE=[@]: the first n render-target and clear commands (from that present on), with their +// textures (development). +static long trace_left(void) { + static long left = -1; + static unsigned long long from; + if (left < 0) { + const char *e = getenv("TOTK_VK_TRACE"); + left = e ? atol(e) : 0; + from = e && strchr(e, '@') ? strtoull(strchr(e, '@') + 1, NULL, 10) : 0; + } + if (atomic_load(&g_ticket) < from) return 0; + return left > 0 ? left-- : 0; +} +static void trace_tex(const char *what, int i, const tex_t *t, const view_t *v) { + if (t) VLOG(" %s %d: texture %llu format %#x target %d %ux%ux%u levels %u%s layers %u+%u level %u format %#x @%p", what, i, + (unsigned long long)t->serial, t->d.format, t->d.target, t->d.width, t->d.height, t->d.depth, t->d.levels, + v ? " | view" : "", v && v->set_layers ? v->min_layer : 0, v && v->set_layers ? v->nlayers : 0, + v && v->set_levels ? v->base_level : 0, v && v->set_format ? v->format : 0u, (const void *)t->mem); +} +static void trace_rt(void) { + if (!trace_left()) return; + VLOG("[trace] SetRenderTargets %d colour + %s", g_rt.n, g_rt.depth ? "depth" : "no depth"); + for (int i = 0; i < g_rt.n && i < 8; i++) trace_tex("colour", i, g_rt.color[i], g_rt.has_cview[i] ? &g_rt.cview[i] : NULL); + trace_tex("depth", 0, g_rt.depth, g_rt.has_dview ? &g_rt.dview : NULL); +} + +int vkb_pass_begin(VkExtent2D bare, uint32_t fb) { return pass_begin(bare, fb); } +uint32_t vkb_pass_fb(void) { return g_in_pass ? g_pass_fb : 0; } +vkb_cnt vkb_pass_end_n[PE_COUNT]; // passes ended, by reason (the per-frame log line) +static void pass_end_why(int why) { if (g_in_pass) vkb_cnt_add(vkb_pass_end_n[why], 1); pass_end(); } +void vkb_pass_end_r(int why) { pass_end_why(why); } +void vkb_pass_end(void) { pass_end_why(PE_OTHER); } + +static void clear_color(const p_clear_color *c) { + if (trace_left()) { + VkRect2D s = {0}; + int has = vkb_scissor(&s); + float v[4] = {0}; + int hv = vkb_viewport0(v); + VLOG("[trace] ClearColor index %d mask %#x rgba %g %g %g %g scissor %s %d,%d %ux%u viewport %s %g,%g %gx%g", c->index, c->mask, c->rgba[0], + c->rgba[1], c->rgba[2], c->rgba[3], has ? "set" : "none", s.offset.x, s.offset.y, s.extent.width, s.extent.height, hv ? "set" : "none", + v[0], v[1], v[2], v[3]); + } + if (c->index < 0 || c->index >= 8 || c->index >= g_rt.n || !g_rt.color[c->index]) return; + static int skip = -2; // TOTK_VK_SKIP_CLEAR=: colour clears of that target index are not replayed (development A/B) + if (skip == -2) { const char *e = getenv("TOTK_VK_SKIP_CLEAR"); skip = e ? atoi(e) : -1; } + if (c->index == skip) return; + if ((c->mask & 0xF) != 0xF) { // [temporary] per-channel masked clears need a draw (step 5) + vkb_audit(A_CLEAR_MASKED, NULL); + static int said; if (!said++) VLOG("[temporary] clear with channel mask %#x: not replayed yet", c->mask); + if (!(c->mask & 0xF)) return; + } + if (!pass_begin((VkExtent2D){0, 0}, 0)) return; + VkClearAttachment a = {VK_IMAGE_ASPECT_COLOR_BIT, (uint32_t)c->index, {.color = {{c->rgba[0], c->rgba[1], c->rgba[2], c->rgba[3]}}}}; + vkb_image *im = vkb_image_of(g_rt.color[c->index]); + const nvn_texdesc *d = im ? &im->d : NULL; + if (d && (d->format == 0x27 || d->format == 0x28 || d->format == 0x2C || d->format == 0x2D || d->format == 0x2F || d->format == 0x30)) + for (int k = 0; k < 4; k++) a.clearValue.color.uint32[k] = (uint32_t)c->rgba[k]; // integer targets (NVN takes float values) + VkClearRect r = {{{0, 0}, g_pass_extent}, 0, 1}; + vkb_checkpoint("clear attachment", NULL, NULL); + vk.vkCmdClearAttachments(vkb_cmd(), 1, &a, 1, &r); +} + +static void clear_ds(const p_clear_ds *c) { + if (!g_rt.depth || !pass_begin((VkExtent2D){0, 0}, 0)) return; + vkb_image *im = vkb_image_of(g_rt.depth); + if (!im) return; + VkImageAspectFlags asp = 0; + if ((im->aspect & VK_IMAGE_ASPECT_DEPTH_BIT) && c->depth_mask) asp |= VK_IMAGE_ASPECT_DEPTH_BIT; + if ((im->aspect & VK_IMAGE_ASPECT_STENCIL_BIT) && (c->stencil_mask & 0xFF)) asp |= VK_IMAGE_ASPECT_STENCIL_BIT; + if (!asp) return; + if ((asp & VK_IMAGE_ASPECT_STENCIL_BIT) && (c->stencil_mask & 0xFF) != 0xFF) { + static int said; if (!said++) VLOG("[temporary] stencil clear with write mask %#x: cleared fully", c->stencil_mask); + } + VkClearAttachment a = {asp, 0, {.depthStencil = {c->depth, (uint32_t)c->stencil}}}; + VkClearRect r = {{{0, 0}, g_pass_extent}, 0, 1}; + vkb_checkpoint("clear attachment", NULL, NULL); + vk.vkCmdClearAttachments(vkb_cmd(), 1, &a, 1, &r); +} + +// ClearTexture: a region of one texture outside a pass +static void clear_tex(const p_clear_tex *c) { + pass_end_why(PE_CLEAR_TEX); + vkb_image *im = vkb_image_of(c->tex); + if (!im) return; + VkCommandBuffer cb = vkb_cmd(); + const view_t *v = c->has_view ? &c->view : NULL; + uint32_t level = v && v->set_levels ? v->base_level : 0, layer = v && v->set_layers ? v->min_layer : 0; + uint32_t lw = im->width >> level, lh = im->height >> level; + if (!lw) lw = 1; + if (!lh) lh = 1; + int full = c->r.xoffset == 0 && c->r.yoffset == 0 && (uint32_t)c->r.width >= lw && (uint32_t)c->r.height >= lh; + if (!full || (c->mask & 0xF) != 0xF) { // [temporary] partial clears need a pass with a scissor (step 5) + static int said; if (said++ < 4) VLOG("[temporary] partial ClearTexture %dx%d of %ux%u mask %#x: not replayed yet", + c->r.width, c->r.height, lw, lh, c->mask); + return; + } + vkb_image_use(cb, im, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, VK_PIPELINE_STAGE_2_CLEAR_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT); + vkb_image_written(im); + uint32_t nlayers = c->r.depth > 0 ? (uint32_t)c->r.depth : 1; + VkImageSubresourceRange range = {im->aspect, level, 1, layer + (uint32_t)c->r.zoffset, nlayers}; + if (im->aspect == VK_IMAGE_ASPECT_COLOR_BIT) { + VkClearColorValue cv; + memcpy(&cv, &c->c, sizeof cv); + vkb_checkpoint("clear texture", NULL, NULL); + vk.vkCmdClearColorImage(cb, im->img, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, &cv, 1, &range); + } else { + VkClearDepthStencilValue dv = {c->c.f[0], c->c.u[1]}; + vkb_checkpoint("clear depth texture", NULL, NULL); + vk.vkCmdClearDepthStencilImage(cb, im->img, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, &dv, 1, &range); + } +} + +void mtl_tex_release(tex_t *t) { vkb_tex_release(t); } + +// ---------------- presentation ---------------- + +// TOTK_SHOT=:: saves that presented frame (development check of the picture). +static VkBuffer g_shot_buf; +static VkDeviceMemory g_shot_mem; +static VkBuffer shot_record(VkCommandBuffer cb, uint64_t n, VkImage out, uint32_t w, uint32_t h) { + static long long at = -1; + if (at == -1) { const char *e = getenv("TOTK_SHOT"); at = e ? atoll(e) : 0; } + if (at <= 0 || (uint64_t)at != n) return VK_NULL_HANDLE; + VkBufferCreateInfo bi = {VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .size = (VkDeviceSize)w * h * 4, + .usage = VK_BUFFER_USAGE_TRANSFER_DST_BIT}; + if (!vk_check(vk.vkCreateBuffer(vkc.device, &bi, NULL, &g_shot_buf), "shot buffer")) return VK_NULL_HANDLE; + VkMemoryRequirements req; + vk.vkGetBufferMemoryRequirements(vkc.device, g_shot_buf, &req); + uint32_t type = UINT32_MAX; + for (uint32_t i = 0; i < vkc.memory.memoryTypeCount && type == UINT32_MAX; i++) + if ((req.memoryTypeBits & (1u << i)) && (vkc.memory.memoryTypes[i].propertyFlags & + (VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT)) == (VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT)) + type = i; + VkMemoryAllocateInfo ai = {VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO, .allocationSize = req.size, .memoryTypeIndex = type}; + if (type == UINT32_MAX || !vk_check(vk.vkAllocateMemory(vkc.device, &ai, NULL, &g_shot_mem), "shot memory")) return VK_NULL_HANDLE; + vk.vkBindBufferMemory(vkc.device, g_shot_buf, g_shot_mem, 0); + VkBufferImageCopy r = {.imageSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1}, .imageExtent = {w, h, 1}}; + vk.vkCmdCopyImageToBuffer(cb, out, VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, g_shot_buf, 1, &r); + return g_shot_buf; +} +static void shot_write(uint64_t fence, uint32_t w, uint32_t h) { + const char *e = getenv("TOTK_SHOT"), *path = e ? strchr(e, ':') : NULL; + mtl_wait(fence, UINT64_MAX); + void *px = NULL; + if (path && vk.vkMapMemory(vkc.device, g_shot_mem, 0, VK_WHOLE_SIZE, 0, &px) == VK_SUCCESS) { + FILE *f = fopen(path + 1, "wb"); + if (f) { // 32-bit BGRA, top-down + uint32_t size = w * h * 4; + uint8_t hdr[54] = {'B', 'M'}; + uint32_t v; + v = 54 + size; memcpy(hdr + 2, &v, 4); + v = 54; memcpy(hdr + 10, &v, 4); + v = 40; memcpy(hdr + 14, &v, 4); + memcpy(hdr + 18, &w, 4); + int32_t nh = -(int32_t)h; memcpy(hdr + 22, &nh, 4); + hdr[26] = 1; hdr[28] = 32; + memcpy(hdr + 34, &size, 4); + fwrite(hdr, 1, sizeof hdr, f); + fwrite(px, 1, size, f); + fclose(f); + VLOG("frame saved to %s (%ux%u)", path + 1, w, h); + } + vk.vkUnmapMemory(vkc.device, g_shot_mem); + } + vk.vkDestroyBuffer(vkc.device, g_shot_buf, NULL); + vk.vkFreeMemory(vkc.device, g_shot_mem, NULL); + g_shot_buf = VK_NULL_HANDLE; +} +// TOTK_DUMP=:: the frame's render targets (development); TOTK_DUMP_EXIT=1 then exits +static void dump_check(uint64_t n) { + static long long dump_at = -1; + static const char *dump_dir; + if (dump_at == -1) { const char *e = getenv("TOTK_DUMP"); dump_at = e ? atoll(e) : 0; dump_dir = e && strchr(e, ':') ? strchr(e, ':') + 1 : NULL; } + if (dump_at > 0 && n >= (uint64_t)dump_at && dump_dir) { // present numbers can skip: the first at or after it + dump_at = -2; + vkb_tex_dump(dump_dir); + if (getenv("TOTK_DUMP_EXIT")) { fflush(NULL); _Exit(0); } // short development runs: done after the dump + } +} + +uint64_t mtl_present(const tex_t *t, double period_s, const int crop[4]) { + (void)period_s; + uint64_t n = atomic_fetch_add(&g_ticket, 1) + 1; + if (n == 1 || n % 300 == 0) { + uint64_t sl, sm; + vkb_shader_stats(&sl, &sm); + uint64_t dn, dsk; + vkb_draw_stats(&dn, &dsk); + VLOG("present %llu: shaders loaded %llu, without translation %llu; draws %llu, skipped %llu", (unsigned long long)n, + (unsigned long long)sl, (unsigned long long)sm, (unsigned long long)dn, (unsigned long long)dsk); + static uint64_t pb, pi, pp, pd, pn; + if (n > pn) { + double f = (double)(n - pn); + static uint64_t pparts, pgpu, pwait; + VLOG("memory pools: %d imported, %.1f MiB", g_npools, (double)g_pool_bytes / 1048576.0); + { + static uint64_t pct[32]; + char cl[256]; int cn = 0; + for (int k = 0; k < 32; k++) if (g_counter_types[k] > pct[k]) { cn += snprintf(cl + cn, sizeof cl - (size_t)cn, " type %d: %.1f", k, (double)(g_counter_types[k] - pct[k]) / f); pct[k] = g_counter_types[k]; } + if (cn) VLOG("per frame: counter reports by type:%s", cl); + } + { + static uint64_t pc, pb, pn, ph, pd; + uint64_t c, b, ns, h, d; + vkb_ucap_stats(&c, &b, &ns, &h, &d); + VLOG("per frame: uniforms copied at submit %.2f MiB in %.0f submissions (%.2f ms on the game thread), bindings from the copy %.0f, dropped by GPU writes %.1f", + (double)(b - pb) / 1048576.0 / f, (double)(c - pc) / f, (double)(ns - pn) / 1e6 / f, (double)(h - ph) / f, (double)(d - pd) / f); + pc = c; pb = b; pn = ns; ph = h; pd = d; + } + if (vkb_shadow_on()) { + static uint64_t pin, pib, pob, pdf, psc, pwp, pwk; + VLOG("per frame: pool copies %.1f MiB of device memory; in %.0f copies %.2f MiB, back %.2f MiB, " + "left in game memory in a pass %.0f, write watch %.2f ms; pages copied back %.0f, left to CPU writes %.1f", (double)vkb_shadow_bytes() / 1048576.0, + (double)(g_sh_in - pin) / f, (double)(g_sh_in_bytes - pib) / 1048576.0 / f, (double)(g_sh_out_bytes - pob) / 1048576.0 / f, + (double)(g_sh_deferred - pdf) / f, (double)(g_sh_scan_ns - psc) / 1e6 / f, + (double)(g_wb_pages - pwp) / f, (double)(g_wb_kept_pages - pwk) / f); + pwp = g_wb_pages; pwk = g_wb_kept_pages; + pin = g_sh_in; pib = g_sh_in_bytes; pob = g_sh_out_bytes; pdf = g_sh_deferred; psc = g_sh_scan_ns; + } + VLOG("per frame: GPU %.1f ms (whole submissions), recording waited %.1f ms for a free command slot", + (double)(g_gpu_ns - pgpu) / 1e6 / f, (double)(g_slot_wait_ns - pwait) / 1e6 / f); + pgpu = g_gpu_ns; pwait = g_slot_wait_ns; + static uint64_t pl[LK_COUNT], pn[LK_COUNT]; + uint64_t l0 = atomic_load(&vkb_lock_wait_ns[0]), l1 = atomic_load(&vkb_lock_wait_ns[1]), l2 = atomic_load(&vkb_lock_wait_ns[2]); + VLOG("per frame: lock waits (all threads) texture registry %.2f ms / %.0f, images %.2f ms / %.0f, pools %.2f ms / %.0f", + (double)(l0 - pl[0]) / 1e6 / f, (double)(atomic_load(&vkb_lock_waits[0]) - pn[0]) / f, + (double)(l1 - pl[1]) / 1e6 / f, (double)(atomic_load(&vkb_lock_waits[1]) - pn[1]) / f, + (double)(l2 - pl[2]) / 1e6 / f, (double)(atomic_load(&vkb_lock_waits[2]) - pn[2]) / f); + for (int k = 0; k < LK_COUNT; k++) { pl[k] = atomic_load(&vkb_lock_wait_ns[k]); pn[k] = atomic_load(&vkb_lock_waits[k]); } + static uint64_t ps[PK_COUNT], psn[PK_COUNT]; + VLOG("per frame: new images %.2f ms / %.1f, uploads %.2f ms / %.1f, queue submits %.2f ms / %.1f", + (double)(vkb_step_ns[0] - ps[0]) / 1e6 / f, (double)(vkb_step_n[0] - psn[0]) / f, (double)(vkb_step_ns[1] - ps[1]) / 1e6 / f, + (double)(vkb_step_n[1] - psn[1]) / f, (double)(vkb_step_ns[2] - ps[2]) / 1e6 / f, (double)(vkb_step_n[2] - psn[2]) / f); + for (int k = 0; k < PK_COUNT; k++) { ps[k] = vkb_step_ns[k]; psn[k] = vkb_step_n[k]; } + { // ASTC as BC7 (vkb_bc7.c) + static uint64_t ph, pmi, pe, pi, pens; + uint64_t h, mi, en, dr, ens; size_t q; + vkb_bc7_stats(&h, &mi, &en, &dr, &ens, &q); + uint64_t vkb_bc7_images(void), bi = vkb_bc7_images(); + VLOG("per frame: ASTC as BC7: %.1f images, %.1f surfaces from the cache, %.1f not there, %.1f encoded in the background " + "(%.1f ms each), %.1f MiB queued, %llu dropped", (double)(bi - pi) / f, (double)(h - ph) / f, (double)(mi - pmi) / f, + (double)(en - pe) / f, en > pe ? (double)(ens - pens) / 1e6 / (double)(en - pe) : 0.0, q / 1048576.0, (unsigned long long)dr); + ph = h; pmi = mi; pe = en; pi = bi; pens = ens; + } + static uint64_t pm, psh, ptb; + uint64_t im_made, im_shared, im_back; + vkb_tex_image_stats(&im_made, &im_shared, &im_back); + VLOG("per frame: images made %.1f, shared with a live texture %.1f, taken back from the side %.1f", + (double)(im_made - pm) / f, (double)(im_shared - psh) / f, (double)(im_back - ptb) / f); + pm = im_made; psh = im_shared; ptb = im_back; + VLOG("per frame: %.0f draws, %.0f render passes, %.0f barriers (%.0f image layout changes), %.1f extra GPU submissions", (double)(dn - pd) / f, + (double)(vkb_cnt_sum(g_n_passes) - pp) / f, (double)(vkb_cnt_sum(g_n_barriers) - pb) / f, + (double)(vkb_cnt_sum(g_n_img_barriers) - pi) / f, (double)(g_parts - pparts) / f); + pparts = g_parts; + static const char *const why[PE_COUNT] = {"other", "set targets", "game barrier", "buffer copy", "uniform update", "texture clear", + "end of submission", "after a storing draw", "snapshot", "sampled targets changed", "texture to prepare", "mirror", "dispatch", "texture copy", "pool copies"}; + static uint64_t prev[PE_COUNT]; + char line[512]; int at = 0; + for (int k = 0; k < PE_COUNT; k++) { + uint64_t ended = vkb_cnt_sum(vkb_pass_end_n[k]); + if (ended > prev[k] && at < (int)sizeof line - 48) + at += snprintf(line + at, sizeof line - (size_t)at, " %s %.0f,", why[k], (double)(ended - prev[k]) / f); + prev[k] = ended; + } + if (at) VLOG("passes ended per frame by:%s", line); + } + pb = vkb_cnt_sum(g_n_barriers); pi = vkb_cnt_sum(g_n_img_barriers); pp = vkb_cnt_sum(g_n_passes); pd = dn; pn = n; + } + vkb_tex_snap_flush(); + vkb_tex_lumlog(t, n); + vkb_audit_present(n); + dump_check(n); + vkb_gprof_report(); + pp_report(); + static int shot_wanted = -1; + if (shot_wanted < 0) shot_wanted = getenv("TOTK_SHOT") || getenv("TOTK_DUMP"); + vkb_image *im = g_ok == 1 && (vkp_active() || shot_wanted) ? vkb_image_of(t) : NULL; // headless: only for TOTK_SHOT + if (!im) { vkp_frame(); return n; } + VkImage out; + int slot = vkp_out_acquire(im->width, im->height, &out); + if (slot < 0) return n; // the window is behind: this frame is dropped + mtl_sub_begin(); + VkCommandBuffer cb = vkb_cmd(); + vkb_image_use(cb, im, VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, VK_PIPELINE_STAGE_2_BLIT_BIT, VK_ACCESS_2_TRANSFER_READ_BIT); + VkImageMemoryBarrier2 b = {VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2, .srcStageMask = VK_PIPELINE_STAGE_2_NONE, + .dstStageMask = VK_PIPELINE_STAGE_2_BLIT_BIT, .dstAccessMask = VK_ACCESS_2_TRANSFER_WRITE_BIT, + .oldLayout = VK_IMAGE_LAYOUT_UNDEFINED, .newLayout = VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, + .srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, .image = out, + .subresourceRange = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 1, 0, 1}}; + VkDependencyInfo di = {VK_STRUCTURE_TYPE_DEPENDENCY_INFO, .imageMemoryBarrierCount = 1, .pImageMemoryBarriers = &b}; + vk.vkCmdPipelineBarrier2(cb, &di); + VkImageBlit r = {{VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1}, {{0, 0, 0}, {(int32_t)im->width, (int32_t)im->height, 1}}, + {VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1}, {{0, 0, 0}, {(int32_t)im->width, (int32_t)im->height, 1}}}; + vk.vkCmdBlitImage(cb, im->img, VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, out, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, 1, &r, VK_FILTER_NEAREST); + b.srcStageMask = VK_PIPELINE_STAGE_2_BLIT_BIT; b.srcAccessMask = VK_ACCESS_2_TRANSFER_WRITE_BIT; + b.dstStageMask = VK_PIPELINE_STAGE_2_BLIT_BIT; b.dstAccessMask = VK_ACCESS_2_TRANSFER_READ_BIT; + b.oldLayout = VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL; b.newLayout = VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL; + vk.vkCmdPipelineBarrier2(cb, &di); + VkBuffer shot = shot_record(cb, n, out, im->width, im->height); + uint64_t fence = mtl_sub_commit(); + if (shot) shot_write(fence, im->width, im->height); + vkp_out_ready(slot, g_timeline, fence, crop); + return n; +} + +// A long submission goes to the GPU in parts, split where the render targets change (the pass ends there anyway): the +// GPU starts on the first part while the rest is recorded. One part per NVN submission made the GPU wait for the whole +// recording: on the Great Sky Island ~20 ms of recording then ~26 ms of GPU time, one after the other (2026-10-24; as +// SavageWind found for BotW). Events and counters recorded in a part are signalled when that part completes. +// TOTK_VK_SPLIT= (default 0: one part per submission). Measured on the Great Sky Island without frame pacing: 45.6 ms +// a frame with one part, 49.4 ms with parts of 1000 draws (2026-10-24); with pool copies (GPU 12 ms, recording ~30 ms) 36.9 ms +// with parts of 1500 draws against 36-41 ms with one: no gain - off. +static uint64_t g_part_draws; // vkb_draw_count() when the current part began +static void split_if_due(void) { + static long long every = -1; + if (every < 0) { const char *e = getenv("TOTK_VK_SPLIT"); every = e ? atoll(e) : 0; } + if (g_ok != 1 || every <= 0 || g_cur < 0) return; + uint64_t n = vkb_draw_count(); + if (n < g_part_draws) g_part_draws = n; + if (n - g_part_draws < (uint64_t)every) return; + mtl_sub_commit(); + mtl_sub_begin(); + g_part_draws = n; + g_parts++; +} + +static __thread int g_render_off; // SetRenderEnable(false), or a conditional that evaluated to disabled +int vkb_render_suppressed(void) { return g_render_off; } + +static const uint8_t *const *g_rest; +static int g_nrest; +void mtl_exec_rest(const uint64_t *h, int n) { g_rest = (const uint8_t *const *)h; g_nrest = n; } +void mtl_sub_index(const uint64_t *h, int n) { (void)h; (void)n; } + +// The commands of a handle, in one of three ways: one thread replaying everything (EX_ONE), the walk of a submission +// before threads record it (EX_WALK: state, events and counters, conditionals, captured uniforms voided; cuts chunks), +// and a chunk on a recording thread (EX_CHUNK: up to the chunk's end, without events and counters). The position +// (t_seq) advances by one per command read, the same in all three. +enum { EX_ONE, EX_WALK, EX_CHUNK }; +typedef struct mt_chunk mt_chunk; +static int walk_op(const uint8_t *p, int hidx); // EX_WALK: a command at depth 0 (may cut a chunk before it) +static int exec_ops(const uint8_t *p, int depth, int mode, int hidx, const mt_chunk *c); +void mtl_exec(const uint8_t *p) { + if (trace_left()) VLOG("[trace] command handle %p (submission %llu, present %llu)", (const void *)p, (unsigned long long)(g_fence + 1), + (unsigned long long)vkb_frame()); + exec_ops(p, 0, EX_ONE, 0, NULL); +} +static int chunk_end_at(const mt_chunk *c, int hidx, const uint8_t *p); +static void walk_command(uint32_t op, const uint8_t *a); +static int exec_ops(const uint8_t *p, int depth, int mode, int hidx, const mt_chunk *c) { // 1: stopped at the chunk's end + for (;;) { + op_hdr h; + memcpy(&h, p, sizeof h); + if (h.size < 8 || (h.size & 7) || h.op >= OP_COUNT_) die("corrupt command record at %p: op %u size %u", (const void *)p, h.op, h.size); + if (mode == EX_CHUNK && !depth && chunk_end_at(c, hidx, p)) return 1; + if (mode == EX_WALK && !depth) walk_op(p, hidx); + t_seq++; + const uint8_t *a = p + 8; + if (mode != EX_CHUNK) g_op_count[h.op]++; + if (mode == EX_WALK) { + if (h.op == OP_END) return 0; + if (h.op == OP_JUMP) { const uint8_t *t; memcpy(&t, a, 8); p = t; continue; } + if (h.op == OP_CALL) { const uint8_t *t; memcpy(&t, a, 8); exec_ops(t, depth + 1, mode, hidx, c); } + else walk_command(h.op, a); + p += h.size; + continue; + } + switch (h.op) { + case OP_END: return 0; + case OP_BEGIN: break; + case OP_JUMP: { const uint8_t *t; memcpy(&t, a, 8); p = t; continue; } + case OP_CALL: { const uint8_t *t; memcpy(&t, a, 8); exec_ops(t, depth + 1, mode, hidx, c); break; } + case OP_SIGNAL_EVENT: { // a chunk: the walk posted it + if (mode == EX_CHUNK) break; + volatile uint32_t *val; uint32_t m, v; + memcpy(&val, a, 8); memcpy(&m, a + 8, 4); memcpy(&v, a + 12, 4); + mtl_sub_event(val, m, v); + break; + } + case OP_REPORT_COUNTER: { if (mode == EX_CHUNK) break; uint64_t t, ad; memcpy(&t, a, 8); memcpy(&ad, a + 8, 8); mtl_sub_counter(ad, (uint32_t)t); break; } + case OP_RESET_COUNTER: break; + case OP_COPY_B2B: { + uint64_t s, d, n; + memcpy(&s, a, 8); memcpy(&d, a + 8, 8); memcpy(&n, a + 16, 8); + mtl_sub_copy(s, d, n); + break; + } + case OP_SET_RT: pass_end_why(PE_SET_RT); if (mode == EX_ONE) split_if_due(); memcpy(&g_rt, a, sizeof g_rt); trace_rt(); break; + case OP_CLEAR_COLOR: if (g_render_off) vkb_cnt_add(g_cond_skipped, 1); else if (g_ok == 1) clear_color((const p_clear_color *)a); break; + case OP_CLEAR_DS: if (g_render_off) vkb_cnt_add(g_cond_skipped, 1); else if (g_ok == 1) clear_ds((const p_clear_ds *)a); break; + case OP_CLEAR_TEX: if (g_ok == 1) clear_tex((const p_clear_tex *)a); break; + case OP_COPY_STRIDE: vkb_copy_stride(((const p_copy_stride *)a)->stride); break; + case OP_COPY_B2T: if (g_ok == 1) vkb_copy_buf_tex((const p_copy_b2t *)a, 1); break; + case OP_COPY_T2B: if (g_ok == 1) vkb_copy_buf_tex((const p_copy_b2t *)a, 0); break; + case OP_COPY_T2T: if (g_ok == 1) vkb_copy_tex_tex((const p_copy_t2t *)a); break; + case OP_BIND_PROGRAM: case OP_BIND_STATE: case OP_VIEWPORTS: case OP_SCISSOR: case OP_DEPTH_RANGE: case OP_STENCIL: + case OP_POLY_OFFSET: case OP_BLEND_COLOR: case OP_BIND_VATTR: case OP_BIND_VSTREAM: case OP_BIND_VBUF: case OP_BIND_UBO: + case OP_BIND_SSBO: case OP_BIND_TEX: case OP_BIND_IMG: case OP_DEPTH_CLAMP: case OP_RASTER_DISCARD: + vkb_state(h.op, a); + break; + case OP_DRAW: case OP_DRAW_IDX: + if (g_render_off) { vkb_cnt_add(g_cond_skipped, 1); break; } + if (g_ok == 1) vkb_draw(a, h.op == OP_DRAW_IDX); + break; + case OP_DISPATCH: if (g_ok == 1) vkb_dispatch(a); break; + case OP_BARRIER: // [temporary] everything before is visible to everything after + if (g_ok == 1) { pass_end_why(PE_BARRIER); full_barrier(vkb_cmd()); } + break; + case OP_SET_TEX_POOL: case OP_SET_SAMP_POOL: case OP_BIND_SAMP: break; + // rendering enable / conditional rendering: draws and clears are skipped while off (dispatches are not) + case OP_RENDER_ENABLE: g_render_off = !((const p_flag *)a)->value; break; + case OP_RENDER_COND: // a chunk: as the walk evaluated it, in order (in the command's pad: 1 enabled, 2 disabled) + g_render_off = mode == EX_CHUNK ? ((const p_render_cond *)a)->pad == 2 : !cond_eval((const p_render_cond *)a); + break; + case OP_UPDATE_UB: { uint64_t d, n; memcpy(&d, a, 8); memcpy(&n, a + 8, 8); update_buffer(d, n, a + 16); break; } + default: + vkb_audit(A_OP_NOT_REPLAYED, NULL); + if (!g_op_said[h.op]++) VLOG("command %u not replayed yet (steps 2-5 of milestone 3.3)", h.op); + break; + } + p += h.size; + } +} + +// Threads recording (vkb_mt_submit): the submission's command buffers after the slot's own, the last one for the work at +// its end (copies back, the GPU end time). +enum { MT_MAXCH = 96 }; +static VkCommandBuffer g_mt_list[2 * MT_MAXCH + 2]; +static int g_mt_nlist; +uint64_t mtl_sub_commit(void) { + uint64_t fence = ++g_fence; + post_op *ops = NULL; + if (g_npost) { + ops = malloc((size_t)g_npost * sizeof *ops); + if (!ops) die("post ops"); + memcpy(ops, g_post, (size_t)g_npost * sizeof *ops); + } + int n = g_npost; + g_npost = 0; + if (g_ok != 1) { run_posts(ops, n); free(ops); complete(fence); return fence; } + pass_end_why(PE_SUBMIT); + sub_long_check(fence); + VkCommandBuffer last = g_mt_nlist ? g_mt_list[g_mt_nlist - 1] : g_slots[g_cur].cb; + vkb_mirrors_submit(last); // written ranges back to game memory before the fence + shadow_writeback(last); + dead_tag(fence); + gpu_time_end(g_cur, last); + cmd_slot *s = &g_slots[g_cur]; + s->fence = fence; + vk.vkEndCommandBuffer(s->cb); + if (g_mt_nlist) vk.vkEndCommandBuffer(last); + VkCommandBufferSubmitInfo cbi[2 * MT_MAXCH + 3]; + int ncbi = 0; + cbi[ncbi++] = (VkCommandBufferSubmitInfo){VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = s->cb}; + for (int i = 0; i < g_mt_nlist; i++) + cbi[ncbi++] = (VkCommandBufferSubmitInfo){VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = g_mt_list[i]}; + VkSemaphoreSubmitInfo sig = {VK_STRUCTURE_TYPE_SEMAPHORE_SUBMIT_INFO, .semaphore = g_timeline, .value = fence, + .stageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT}; + VkSubmitInfo2 si = {VK_STRUCTURE_TYPE_SUBMIT_INFO_2, .commandBufferInfoCount = (uint32_t)ncbi, .pCommandBufferInfos = cbi, + .signalSemaphoreInfoCount = 1, .pSignalSemaphoreInfos = &sig}; + uint64_t ts0 = runtime_now_ns(); + vk_queue_lock(); + VkResult r = vk.vkQueueSubmit2(vkc.queue, 1, &si, VK_NULL_HANDLE); + vk_queue_unlock(); + vkb_step_add(PK_SUBMIT, ts0); + if (r != VK_SUCCESS) { vk_check(r, "vkQueueSubmit2"); die("GPU submission failed"); } + job *j = malloc(sizeof *j); + if (!j) die("retire job"); + *j = (job){fence, ops, n, g_cur, g_wb_ranges, g_wb_nranges, NULL}; // g_wb_ranges: in the slot's copy back memory + g_wb_ranges = NULL; + totk_tl(TL_COMMIT, fence, 1); + pthread_mutex_lock(&g_mu); + *g_jobs_tail = j; + g_jobs_tail = &j->next; + pthread_cond_broadcast(&g_cv); + pthread_mutex_unlock(&g_mu); + g_cur = -1; + static uint64_t said_at; + if (fence == 1 || fence >= said_at + 3000) { + said_at = fence; + VLOG("submission %llu: commands so far draw %llu, indexed %llu, dispatch %llu, clear %llu, set targets %llu, " + "render enable %llu, conditional render %llu (draws and clears skipped %llu, conditionals unknown %llu)", + (unsigned long long)fence, (unsigned long long)g_op_count[OP_DRAW], (unsigned long long)g_op_count[OP_DRAW_IDX], + (unsigned long long)g_op_count[OP_DISPATCH], (unsigned long long)g_op_count[OP_CLEAR_COLOR], + (unsigned long long)g_op_count[OP_SET_RT], (unsigned long long)g_op_count[OP_RENDER_ENABLE], + (unsigned long long)g_op_count[OP_RENDER_COND], (unsigned long long)vkb_cnt_sum(g_cond_skipped), (unsigned long long)g_cond_unknown); + } + return fence; +} + +// ---------------- several recording threads (vkb.h, "vkb_mt.c") ---------------- +// One submission of the Great Sky Island is one handle of 5000-7000 draws; recording it took ~25 ms of one thread +// (2026-10-24), the most of the 33 ms frame. Here the translation thread walks it once in order (exec_ops EX_WALK: +// render state, events and counters, conditional rendering, the GPU writes that void captured uniforms) and cuts it into +// chunks of about MT_DRAWS draws at render target changes, each with the state at its start. Recording threads take the +// chunks as they are cut (the translation thread too, after the walk) and record each into a command buffer of their +// own, with their own caches, upload ring and descriptor memory. Work that has to come before every chunk goes into the +// prologue (texture uploads and pool copies: the CPU wrote those bytes before the submission). Images: each chunk keeps +// the layout every image is first needed in and the one it is left in (vkb_trk); a small command buffer before each +// chunk waits for all earlier work and moves its images there. The submission is then one vkQueueSubmit2 of: the slot's +// command buffer (GPU start time, captured uniforms, the move into the prologue's layouts), the prologue, and for each +// chunk its move and itself, then the end (copies back). +// [temporary] Mirrors of written storage buffers are not made (one thread only); TOTK_VK_RECORD_THREADS=1 records as before. +enum { MT_DRAWS = 400 }; +struct mt_chunk { + int h0, h1; const uint8_t *p0, *p1; // its first command, and the one after its last (h1 == the handle count: the end) + uint64_t seq; // the position before its first command + void *state; p_set_rt rt; int render_off; int64_t stride; // the state it starts in + VkCommandBuffer cb; + vkb_trk trk; +}; +static mt_chunk g_ch[MT_MAXCH]; +static int g_nch; +static const uint64_t *g_mt_h; +static int g_mt_n, g_mt_threads; +static pthread_mutex_t g_mt_mu = PTHREAD_MUTEX_INITIALIZER; +static pthread_cond_t g_mt_cv = PTHREAD_COND_INITIALIZER, g_mt_done_cv = PTHREAD_COND_INITIALIZER; +static int g_mt_avail, g_mt_next, g_mt_done; // chunks cut / taken / recorded +static uint64_t g_mt_draws; // draws in the chunk being cut +static void *g_walk_state; // where the walk ended: the next one starts there +static p_set_rt g_walk_rt; +static int g_walk_off, g_walk_have; +static int64_t g_walk_stride; +// the prologue: any recording thread, one at a time +static pthread_mutex_t g_pro_mu = PTHREAD_MUTEX_INITIALIZER; +static VkCommandBuffer g_pro_cb; +static vkb_trk g_pro_trk; +static __thread VkCommandBuffer t_pro_cb; +static __thread vkb_trk *t_pro_trk; +static __thread int t_pro_depth; +// command buffers per slot and recording thread; [VKB_MAXREC]: the prologue's, the moves' and the end's +typedef struct { VkCommandPool pool; VkCommandBuffer *cb; int n, used; } wpool_t; +static wpool_t g_wp[SLOTS][VKB_MAXREC + 1]; + +int vkb_mt_on(void) { + static int n = -1; + if (n < 0) { + SYSTEM_INFO si; + GetSystemInfo(&si); + int cpus = (int)si.dwNumberOfProcessors; + const char *e = getenv("TOTK_VK_RECORD_THREADS"); + n = e && *e ? atoi(e) : cpus >= 8 ? 4 : cpus >= 4 ? 2 : 1; + if (n > VKB_MAXREC) n = VKB_MAXREC; + if (n < 1) n = 1; + static const char *const diag[] = {"TOTK_VK_DRAWTRACE", "TOTK_VK_TRACE", "TOTK_SNAP", "TOTK_GPU_PROF", "TOTK_GPU_PASSPROF", + "TOTK_VK_CHECKPOINTS", "TOTK_AUDIT", "TOTK_VK_SPLIT", "TOTK_VK_STORES_LOG", "TOTK_WATCH"}; + const char *why = NULL; // development tools that follow the recording in order: one thread + for (unsigned i = 0; i < sizeof diag / sizeof *diag; i++) if (getenv(diag[i])) why = diag[i]; + if (n > 1 && why) n = 1; + if (n > 1 && g_ok != 1) n = 1; + g_mt_threads = n; + VLOG("recording threads: %d%s%s", n, why ? " (one: " : "", why ? why : ""); + if (why) VLOG(" (one thread while %s is set)", why); + } + return n > 1; +} + +static VkCommandBuffer wp_get(int slot, int w) { // a command buffer of that thread's pool for the slot, begun + wpool_t *p = &g_wp[slot][w]; + if (!p->pool) { + VkCommandPoolCreateInfo pi = {VK_STRUCTURE_TYPE_COMMAND_POOL_CREATE_INFO, .flags = VK_COMMAND_POOL_CREATE_TRANSIENT_BIT, + .queueFamilyIndex = vkc.family}; + if (!vk_check(vk.vkCreateCommandPool(vkc.device, &pi, NULL, &p->pool), "recording command pool")) die("command pool"); + } + if (p->used == p->n) { + int add = 16; + if (!(p->cb = realloc(p->cb, sizeof *p->cb * (size_t)(p->n + add)))) die("command buffers"); + VkCommandBufferAllocateInfo ai = {VK_STRUCTURE_TYPE_COMMAND_BUFFER_ALLOCATE_INFO, .commandPool = p->pool, + .level = VK_COMMAND_BUFFER_LEVEL_PRIMARY, .commandBufferCount = (uint32_t)add}; + if (!vk_check(vk.vkAllocateCommandBuffers(vkc.device, &ai, p->cb + p->n), "command buffers")) die("command buffers"); + p->n += add; + } + VkCommandBuffer cb = p->cb[p->used++]; + VkCommandBufferBeginInfo bi = {VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO, .flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT}; + vk.vkBeginCommandBuffer(cb, &bi); + return cb; +} +void vkb_mt_slot_reset(int slot) { // the slot's previous submission has completed (no thread is recording) + for (int w = 0; w <= VKB_MAXREC; w++) + if (g_wp[slot][w].used) { vk.vkResetCommandPool(vkc.device, g_wp[slot][w].pool, 0); g_wp[slot][w].used = 0; } +} + +void vkb_pro_begin(void) { + if (!t_recording || t_pro_depth++) return; + pthread_mutex_lock(&g_pro_mu); + t_pro_cb = t_cb; t_pro_trk = vkb_trk_get(); + t_cb = g_pro_cb; + vkb_trk_set(&g_pro_trk); +} +void vkb_pro_end(void) { + if (!t_recording || --t_pro_depth) return; + t_cb = t_pro_cb; + vkb_trk_set(t_pro_trk); + pthread_mutex_unlock(&g_pro_mu); +} + +static int chunk_end_at(const mt_chunk *c, int hidx, const uint8_t *p) { return hidx == c->h1 && p == c->p1; } +static vkb_cnt g_mt_rec_ns; // chunk recording time, all threads (the log line) +static void mt_record(mt_chunk *c) { + uint64_t t0 = runtime_now_ns(); + c->cb = wp_get(g_cur, t_worker); + t_cb = c->cb; + t_recording = 1; + vkb_trk_reset(&c->trk); + vkb_trk_set(&c->trk); + vkb_rec_begin(); + vkb_state_load(c->state); + g_rt = c->rt; + g_render_off = c->render_off; + vkb_copy_stride(c->stride); + g_in_pass = 0; + vkb_stores_take(); + t_seq = c->seq; + for (int i = c->h0; i < g_mt_n; i++) { + const uint8_t *p = i == c->h0 ? c->p0 : (const uint8_t *)(uintptr_t)g_mt_h[i]; + if (chunk_end_at(c, i, p) || exec_ops(p, 0, EX_CHUNK, i, c)) break; + } + pass_end_why(PE_SUBMIT); + vk.vkEndCommandBuffer(c->cb); + vkb_trk_set(NULL); + t_recording = 0; + t_cb = VK_NULL_HANDLE; + vkb_cnt_add(g_mt_rec_ns, runtime_now_ns() - t0); +} +static int mt_take(void) { // the next chunk cut and not taken, or -1 + pthread_mutex_lock(&g_mt_mu); + int k = g_mt_next < g_mt_avail ? g_mt_next++ : -1; + pthread_mutex_unlock(&g_mt_mu); + return k; +} +static void mt_recorded(void) { + pthread_mutex_lock(&g_mt_mu); + g_mt_done++; + pthread_cond_broadcast(&g_mt_done_cv); + pthread_mutex_unlock(&g_mt_mu); +} +static void *mt_thread(void *arg) { + t_worker = (int)(intptr_t)arg; + char name[32]; + snprintf(name, sizeof name, "totk-record-%d", t_worker); + pthread_setname_np(name); + { void plat_thread_boost(void); plat_thread_boost(); } // the frame waits for these, as for the translation thread + for (;;) { + pthread_mutex_lock(&g_mt_mu); + while (g_mt_next >= g_mt_avail) pthread_cond_wait(&g_mt_cv, &g_mt_mu); + int k = g_mt_next++; + pthread_mutex_unlock(&g_mt_mu); + mt_record(&g_ch[k]); + mt_recorded(); + } + return NULL; +} + +// ---- the walk +static void chunk_open(int hidx, const uint8_t *p) { // a chunk starts at p, in the state the walk is in now + mt_chunk *c = &g_ch[g_nch]; + c->h0 = hidx; c->p0 = p; c->seq = t_seq; + if (!c->state && !(c->state = malloc(vkb_state_size()))) die("chunk state"); + vkb_state_save(c->state); + c->rt = g_rt; + c->render_off = g_render_off; + c->stride = vkb_copy_stride_get(); + g_mt_draws = 0; +} +static void chunk_close(int hidx, const uint8_t *p) { // the open chunk ends before p: recording threads may take it + mt_chunk *c = &g_ch[g_nch]; + c->h1 = hidx; c->p1 = p; + pthread_mutex_lock(&g_mt_mu); + g_mt_avail = ++g_nch; + pthread_cond_broadcast(&g_mt_cv); + pthread_mutex_unlock(&g_mt_mu); +} +static int walk_op(const uint8_t *p, int hidx) { // before a command at depth 0: a new chunk at a render target change + op_hdr h; + memcpy(&h, p, sizeof h); + if (h.op == OP_SET_RT && g_mt_draws >= MT_DRAWS && g_nch < MT_MAXCH - 1) { + chunk_close(hidx, p); + chunk_open(hidx, p); + } + return 0; +} +static void walk_command(uint32_t op, const uint8_t *a) { + switch (op) { + case OP_SIGNAL_EVENT: { + volatile uint32_t *val; uint32_t m, v; + memcpy(&val, a, 8); memcpy(&m, a + 8, 4); memcpy(&v, a + 12, 4); + mtl_sub_event(val, m, v); + break; + } + case OP_REPORT_COUNTER: { uint64_t t, ad; memcpy(&t, a, 8); memcpy(&ad, a + 8, 8); mtl_sub_counter(ad, (uint32_t)t); break; } + case OP_COPY_B2B: { uint64_t d, n; memcpy(&d, a + 8, 8); memcpy(&n, a + 16, 8); if (n) vkb_ucap_drop(d, n); break; } // GPU writes + case OP_UPDATE_UB: { uint64_t d, n; memcpy(&d, a, 8); memcpy(&n, a + 8, 8); if (n) vkb_ucap_drop(d, n); break; } + case OP_COPY_T2B: { uint64_t ad, len; if (vkb_copy_t2b_range((const p_copy_b2t *)a, &ad, &len)) vkb_ucap_drop(ad, len); break; } + case OP_COPY_STRIDE: vkb_copy_stride(((const p_copy_stride *)a)->stride); break; + case OP_SET_RT: memcpy(&g_rt, a, sizeof g_rt); break; + case OP_BIND_PROGRAM: case OP_BIND_STATE: case OP_VIEWPORTS: case OP_SCISSOR: case OP_DEPTH_RANGE: case OP_STENCIL: + case OP_POLY_OFFSET: case OP_BLEND_COLOR: case OP_BIND_VATTR: case OP_BIND_VSTREAM: case OP_BIND_VBUF: case OP_BIND_UBO: + case OP_BIND_SSBO: case OP_BIND_TEX: case OP_BIND_IMG: case OP_DEPTH_CLAMP: case OP_RASTER_DISCARD: + vkb_state(op, a); + break; + case OP_DRAW: case OP_DRAW_IDX: + if (!g_render_off) { g_mt_draws++; vkb_draw_prescan(a, op == OP_DRAW_IDX, 0); } + break; + case OP_DISPATCH: g_mt_draws++; vkb_draw_prescan(a, 0, 1); break; + case OP_RENDER_ENABLE: g_render_off = !((const p_flag *)a)->value; break; + case OP_RENDER_COND: { // evaluated here, in order; the chunk reads the answer from the command + p_render_cond *r = (p_render_cond *)(uintptr_t)a; + g_render_off = !cond_eval(r); + r->pad = g_render_off ? 2 : 1; + break; + } + default: break; + } +} + +// Before a chunk: everything earlier is done, and its images move into the layouts it needs first. +static void glue(VkCommandBuffer cb, vkb_trk *t) { + VkMemoryBarrier2 mb = {VK_STRUCTURE_TYPE_MEMORY_BARRIER_2, .srcStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + .srcAccessMask = VK_ACCESS_2_MEMORY_WRITE_BIT, .dstStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + .dstAccessMask = VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT}; + VkImageMemoryBarrier2 ib[64]; + int ni = 0, first = 1; + for (int k = 0; k <= t->n; k++) { + if (k < t->n) { + vkb_trk_e *e = &t->e[k]; + if (e->im->layout == e->first) continue; + ib[ni++] = (VkImageMemoryBarrier2){VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2, .srcStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + .srcAccessMask = VK_ACCESS_2_MEMORY_WRITE_BIT, .dstStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + .dstAccessMask = VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT, .oldLayout = e->im->layout, .newLayout = e->first, + .srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, .image = e->im->img, + .subresourceRange = {e->im->aspect, 0, VK_REMAINING_MIP_LEVELS, 0, VK_REMAINING_ARRAY_LAYERS}}; + if (ni < 64) continue; + } + if (ni || first) { + VkDependencyInfo dep = {VK_STRUCTURE_TYPE_DEPENDENCY_INFO, .memoryBarrierCount = first ? 1u : 0u, .pMemoryBarriers = &mb, + .imageMemoryBarrierCount = (uint32_t)ni, .pImageMemoryBarriers = ib}; + vk.vkCmdPipelineBarrier2(cb, &dep); + ni = 0; first = 0; + } + } + for (int k = 0; k < t->n; k++) t->e[k].im->layout = t->e[k].cur; +} + +void vkb_mt_submit(const uint64_t *h, int n) { // after mtl_sub_begin and the uniform capture; commits + uint64_t t0 = runtime_now_ns(); + static uint64_t walk_ns, wall_ns, subs, chunks, said_at, glue_images; + static int started; + if (!started) { + started = 1; + for (int w = 1; w < g_mt_threads; w++) { + pthread_t t; + if (pthread_create(&t, NULL, mt_thread, (void *)(intptr_t)w) == 0) pthread_detach(t); + else die("recording thread"); + } + } + g_mt_h = h; g_mt_n = n; + pthread_mutex_lock(&g_mt_mu); + g_mt_avail = g_mt_next = g_mt_done = 0; + g_nch = 0; + pthread_mutex_unlock(&g_mt_mu); + // the prologue: objects made on first use, the pool copies of the last submissions + g_pro_cb = wp_get(g_cur, VKB_MAXREC); + vkb_trk_reset(&g_pro_trk); + t_recording = 1; + vkb_pro_begin(); + vkb_draw_mt_init(); + vkb_pro_end(); + shadow_presync(); + t_recording = 0; + uint64_t tw = runtime_now_ns(); + // the walk, from where the last one ended + if (g_walk_have) { vkb_state_load(g_walk_state); g_rt = g_walk_rt; g_render_off = g_walk_off; vkb_copy_stride(g_walk_stride); } + g_in_pass = 0; + chunk_open(0, n ? (const uint8_t *)(uintptr_t)h[0] : NULL); + for (int i = 0; i < n; i++) exec_ops((const uint8_t *)(uintptr_t)h[i], 0, EX_WALK, i, NULL); + chunk_close(n, NULL); + if (!g_walk_state && !(g_walk_state = malloc(vkb_state_size()))) die("walk state"); + vkb_state_save(g_walk_state); + g_walk_rt = g_rt; g_walk_off = g_render_off; g_walk_stride = vkb_copy_stride_get(); g_walk_have = 1; + walk_ns += runtime_now_ns() - tw; + // this thread records chunks too, then waits for the others + for (int k; (k = mt_take()) >= 0;) { mt_record(&g_ch[k]); mt_recorded(); } + pthread_mutex_lock(&g_mt_mu); + while (g_mt_done < g_nch) pthread_cond_wait(&g_mt_done_cv, &g_mt_mu); + pthread_mutex_unlock(&g_mt_mu); + // in order: the slot's command buffer moves images into the prologue's layouts, then each chunk's move and the chunk + vk.vkEndCommandBuffer(g_pro_cb); + glue(g_slots[g_cur].cb, &g_pro_trk); + g_mt_nlist = 0; + g_mt_list[g_mt_nlist++] = g_pro_cb; + for (int k = 0; k < g_nch; k++) { + VkCommandBuffer m = wp_get(g_cur, VKB_MAXREC); + glue_images += (uint64_t)g_ch[k].trk.n; + glue(m, &g_ch[k].trk); + vk.vkEndCommandBuffer(m); + g_mt_list[g_mt_nlist++] = m; + g_mt_list[g_mt_nlist++] = g_ch[k].cb; + } + g_mt_list[g_mt_nlist++] = wp_get(g_cur, VKB_MAXREC); // the end + mtl_sub_commit(); + g_mt_nlist = 0; + wall_ns += runtime_now_ns() - t0; + subs++; chunks += (uint64_t)g_nch; + if (subs >= said_at + 600) { // the log line: where a submission's recording time goes + said_at = subs; + static uint64_t prev_rec; + uint64_t rec = vkb_cnt_sum(g_mt_rec_ns); + VLOG("recording threads: per submission %.1f ms until submitted (walk %.1f ms), chunks %.1f, chunk recording %.1f ms in all " + "threads, images moved between chunks %.0f", wall_ns / 1e6 / 600, walk_ns / 1e6 / 600, chunks / 600.0, + (rec - prev_rec) / 1e6 / 600, glue_images / 600.0); + prev_rec = rec; wall_ns = walk_ns = chunks = glue_images = 0; + } +} + +uint64_t mtl_last_fence(void) { return g_fence; } +uint64_t mtl_done(void) { pthread_mutex_lock(&g_mu); uint64_t d = g_done; pthread_mutex_unlock(&g_mu); return d; } // completed by the GPU + +int mtl_wait(uint64_t fence, uint64_t timeout_ns) { + pthread_mutex_lock(&g_mu); + if (g_done >= fence || !timeout_ns) { int ok = g_done >= fence; pthread_mutex_unlock(&g_mu); return ok; } + uint64_t end = timeout_ns == UINT64_MAX ? UINT64_MAX : runtime_now_ns() + timeout_ns; + while (g_done < fence) { + uint64_t now = runtime_now_ns(); + if (now >= end) break; + struct timespec ts; + clock_gettime(CLOCK_REALTIME, &ts); + uint64_t wait = end - now > 100000000 ? 100000000 : end - now; // re-check at least every 100 ms + ts.tv_nsec += (long)(wait % 1000000000); + ts.tv_sec += (time_t)(wait / 1000000000) + ts.tv_nsec / 1000000000; + ts.tv_nsec %= 1000000000; + pthread_cond_timedwait(&g_cv, &g_mu, &ts); + } + int ok = g_done >= fence; + pthread_mutex_unlock(&g_mu); + return ok; +} +void mtl_finish(void) { mtl_wait(g_fence, UINT64_MAX); } +void mtl_drain(void) { mtl_finish(); } diff --git a/src/gfx/compat/TargetConditionals.h b/src/gfx/compat/TargetConditionals.h new file mode 100644 index 0000000..b752731 --- /dev/null +++ b/src/gfx/compat/TargetConditionals.h @@ -0,0 +1,3 @@ +// Stand-in for the macOS header (MacNativeGames code on Windows): definitions live in gfx_compat.h. +#pragma once +#include "gfx_compat.h" diff --git a/src/gfx/compat/gfx_compat.h b/src/gfx/compat/gfx_compat.h new file mode 100644 index 0000000..f8706a5 --- /dev/null +++ b/src/gfx/compat/gfx_compat.h @@ -0,0 +1,114 @@ +// Windows stand-ins for the macOS APIs MacNativeGames' graphics front end (d2r_gfx/gfx/nvn) uses. +// - Mach time: mach_absolute_time() is the runtime's monotonic nanoseconds (timebase 1/1), mach_wait_until() sleeps +// precisely until such a time. +// - Mach thread/task statistics and real-time thread policies: unavailable (KERN_FAILURE); MacNativeGames only logs +// them, and every caller checks the result. +// - QoS classes: no-ops (Windows thread priorities are set where the port needs them). +// - sigsetjmp guards: plain setjmp (guest pages are committed on first touch, so guarded guest reads do not fault). +// - malloc_size: _msize. +#pragma once +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +uint64_t plat_now_ns(void); +void plat_sleep_until_ns(uint64_t t); + +// ---- Mach time +typedef struct { uint32_t numer, denom; } mach_timebase_info_data_t; +static inline int mach_timebase_info(mach_timebase_info_data_t *tb) { tb->numer = 1; tb->denom = 1; return 0; } +static inline uint64_t mach_absolute_time(void) { return plat_now_ns(); } +static inline uint64_t mach_continuous_time(void) { return plat_now_ns(); } +static inline int mach_wait_until(uint64_t t) { plat_sleep_until_ns(t); return 0; } +#ifndef CLOCK_UPTIME_RAW +#define CLOCK_UPTIME_RAW CLOCK_MONOTONIC +#endif + +// ---- Mach kernel calls MacNativeGames uses for statistics: always "failed" +typedef int kern_return_t; +typedef int boolean_t; +typedef unsigned mach_port_t; +typedef unsigned mach_msg_type_number_t; +typedef uintptr_t vm_address_t; +typedef size_t vm_size_t; +typedef int *thread_info_t, *task_info_t, *thread_policy_t; +#define KERN_SUCCESS 0 +#define KERN_FAILURE 5 +typedef struct { int seconds, microseconds; } time_value_t; +typedef struct { time_value_t user_time, system_time; int cpu_usage, policy, run_state, flags, suspend_count, sleep_time; } thread_basic_info_data_t; +typedef struct { uint64_t pth_user_time, pth_system_time; int32_t pth_cpu_usage, pth_policy, pth_run_state, pth_flags, pth_sleep_time, pth_curpri, pth_priority, pth_maxpriority; char pth_name[64]; } thread_extended_info_data_t; +typedef struct { uint32_t period, computation, constraint; boolean_t preemptible; } thread_time_constraint_policy_data_t; +typedef struct { uint64_t virtual_size, resident_size, phys_footprint, compressed; } task_vm_info_data_t; +#define THREAD_BASIC_INFO 3 +#define THREAD_BASIC_INFO_COUNT 10 +#define THREAD_EXTENDED_INFO 5 +#define THREAD_EXTENDED_INFO_COUNT 28 +#define THREAD_TIME_CONSTRAINT_POLICY 2 +#define THREAD_TIME_CONSTRAINT_POLICY_COUNT 4 +#define TASK_VM_INFO 22 +#define TASK_VM_INFO_COUNT 80 +static inline mach_port_t mach_thread_self(void) { return 0; } +static inline mach_port_t mach_task_self(void) { return 0; } +static inline mach_port_t pthread_mach_thread_np(pthread_t t) { (void)t; return 0; } +static inline kern_return_t mach_port_deallocate(mach_port_t t, mach_port_t p) { (void)t; (void)p; return KERN_SUCCESS; } +static inline kern_return_t thread_info(mach_port_t t, int f, thread_info_t i, mach_msg_type_number_t *n) { (void)t; (void)f; (void)i; (void)n; return KERN_FAILURE; } +static inline kern_return_t task_info(mach_port_t t, int f, task_info_t i, mach_msg_type_number_t *n) { (void)t; (void)f; (void)i; (void)n; return KERN_FAILURE; } +static inline kern_return_t thread_policy_set(mach_port_t t, int f, thread_policy_t p, mach_msg_type_number_t n) { (void)t; (void)f; (void)p; (void)n; return KERN_FAILURE; } +static inline kern_return_t thread_policy_get(mach_port_t t, int f, thread_policy_t p, mach_msg_type_number_t *n, boolean_t *d) { (void)t; (void)f; (void)p; (void)n; (void)d; return KERN_FAILURE; } +static inline kern_return_t vm_read_overwrite(mach_port_t t, vm_address_t a, vm_size_t n, vm_address_t d, vm_size_t *got) { (void)t; (void)a; (void)n; (void)d; *got = 0; return KERN_FAILURE; } + +// ---- QoS +typedef int qos_class_t; +#define QOS_CLASS_UNSPECIFIED 0 +#define QOS_CLASS_USER_INTERACTIVE 0x21 +#define QOS_CLASS_USER_INITIATED 0x19 +static inline int pthread_attr_set_qos_class_np(pthread_attr_t *a, qos_class_t q, int r) { (void)a; (void)q; (void)r; return 0; } +static inline int pthread_set_qos_class_self_np(qos_class_t q, int r) { (void)q; (void)r; return 0; } +static inline int pthread_get_qos_class_np(pthread_t t, qos_class_t *q, int *r) { (void)t; *q = QOS_CLASS_UNSPECIFIED; *r = 0; return 0; } + +// ---- signals and guarded reads +#ifndef SIGBUS +#define SIGBUS 10 +#endif +typedef jmp_buf sigjmp_buf; +#define sigsetjmp(b, m) setjmp(b) +#define siglongjmp(b, v) longjmp(b, v) + +// ---- resource usage (statistics only) +struct rusage { struct { long tv_sec, tv_usec; } ru_utime, ru_stime; long ru_maxrss, ru_minflt, ru_majflt, ru_nvcsw, ru_nivcsw; }; +#define RUSAGE_SELF 0 +static inline int getrusage(int who, struct rusage *r) { (void)who; memset(r, 0, sizeof *r); return -1; } + +// ---- memory +#define malloc_size(p) _msize(p) +#define PROT_NONE 0 +#define PROT_READ 1 +#define PROT_WRITE 2 +#define MAP_PRIVATE 2 +#define MAP_ANON 0x20 +#define MAP_FAILED ((void *)-1) +static inline void *mmap(void *a, size_t n, int p, int f, int fd, long o) { (void)a; (void)n; (void)p; (void)f; (void)fd; (void)o; return MAP_FAILED; } +static inline int munmap(void *a, size_t n) { (void)a; (void)n; return -1; } +static inline int mprotect(void *a, size_t n, int p) { (void)a; (void)n; (void)p; return -1; } +static inline int getpagesize(void) { return 4096; } + +// ---- threads: macOS names only the calling thread +static inline int gfx_setname(const char *n) { return pthread_setname_np(pthread_self(), n); } +#define pthread_setname_np(n) gfx_setname(n) + +// ---- more Darwin calls: the thread id and per-clock nanoseconds +unsigned long __stdcall GetCurrentThreadId(void); +static inline int pthread_threadid_np(void *t, uint64_t *id) { (void)t; *id = GetCurrentThreadId(); return 0; } +static inline uint64_t clock_gettime_nsec_np(clockid_t clk) { + struct timespec ts; + if (clock_gettime(clk, &ts)) return 0; + return (uint64_t)ts.tv_sec * 1000000000ull + (uint64_t)ts.tv_nsec; +} diff --git a/src/gfx/compat/mach-o/getsect.h b/src/gfx/compat/mach-o/getsect.h new file mode 100644 index 0000000..174682c --- /dev/null +++ b/src/gfx/compat/mach-o/getsect.h @@ -0,0 +1,3 @@ +// Stand-in for the macOS header (MacNativeGames code on Windows): definitions live in gfx_compat.h. +#pragma once +#include "gfx_compat.h" diff --git a/src/gfx/compat/mach-o/ldsyms.h b/src/gfx/compat/mach-o/ldsyms.h new file mode 100644 index 0000000..72e5d5f --- /dev/null +++ b/src/gfx/compat/mach-o/ldsyms.h @@ -0,0 +1,3 @@ +// Stand-in for the macOS header (MacNativeGames code on Windows): definitions live in gfx_compat.h. +#pragma once +#include "gfx_compat.h" diff --git a/src/gfx/compat/mach/mach.h b/src/gfx/compat/mach/mach.h new file mode 100644 index 0000000..aea3260 --- /dev/null +++ b/src/gfx/compat/mach/mach.h @@ -0,0 +1,3 @@ +// Stand-in for the macOS header (MacNativeGames code on Windows): definitions live in gfx_compat.h. +#pragma once +#include "gfx_compat.h" diff --git a/src/gfx/compat/mach/mach_time.h b/src/gfx/compat/mach/mach_time.h new file mode 100644 index 0000000..53d8b20 --- /dev/null +++ b/src/gfx/compat/mach/mach_time.h @@ -0,0 +1,3 @@ +// Stand-in for the macOS header (MacNativeGames code on Windows): definitions live in gfx_compat.h. +#pragma once +#include "gfx_compat.h" diff --git a/src/gfx/compat/mach/thread_policy.h b/src/gfx/compat/mach/thread_policy.h new file mode 100644 index 0000000..2351210 --- /dev/null +++ b/src/gfx/compat/mach/thread_policy.h @@ -0,0 +1,3 @@ +// Stand-in for the macOS header (MacNativeGames code on Windows): definitions live in gfx_compat.h. +#pragma once +#include "gfx_compat.h" diff --git a/src/gfx/compat/mach_vm_compat.h b/src/gfx/compat/mach_vm_compat.h new file mode 100644 index 0000000..dc613da --- /dev/null +++ b/src/gfx/compat/mach_vm_compat.h @@ -0,0 +1,3 @@ +// Stand-in for the macOS header (MacNativeGames code on Windows): definitions live in gfx_compat.h. +#pragma once +#include "gfx_compat.h" diff --git a/src/gfx/compat/malloc/malloc.h b/src/gfx/compat/malloc/malloc.h new file mode 100644 index 0000000..0758822 --- /dev/null +++ b/src/gfx/compat/malloc/malloc.h @@ -0,0 +1,3 @@ +// Stand-in for the macOS header (MacNativeGames code on Windows): definitions live in gfx_compat.h. +#pragma once +#include "gfx_compat.h" diff --git a/src/gfx/compat/pthread/qos.h b/src/gfx/compat/pthread/qos.h new file mode 100644 index 0000000..71b6069 --- /dev/null +++ b/src/gfx/compat/pthread/qos.h @@ -0,0 +1,3 @@ +// Stand-in for the macOS header (MacNativeGames code on Windows): definitions live in gfx_compat.h. +#pragma once +#include "gfx_compat.h" diff --git a/src/gfx/compat/sys/mman.h b/src/gfx/compat/sys/mman.h new file mode 100644 index 0000000..17d7c1e --- /dev/null +++ b/src/gfx/compat/sys/mman.h @@ -0,0 +1,3 @@ +// Stand-in for the macOS header (MacNativeGames code on Windows): definitions live in gfx_compat.h. +#pragma once +#include "gfx_compat.h" diff --git a/src/gfx/compat/sys/resource.h b/src/gfx/compat/sys/resource.h new file mode 100644 index 0000000..ebd4f52 --- /dev/null +++ b/src/gfx/compat/sys/resource.h @@ -0,0 +1,3 @@ +// Stand-in for the macOS header (MacNativeGames code on Windows): definitions live in gfx_compat.h. +#pragma once +#include "gfx_compat.h" diff --git a/src/gfx/compat/sys/ucontext.h b/src/gfx/compat/sys/ucontext.h new file mode 100644 index 0000000..937a5fd --- /dev/null +++ b/src/gfx/compat/sys/ucontext.h @@ -0,0 +1,3 @@ +// Stand-in for the macOS header (MacNativeGames code on Windows): definitions live in gfx_compat.h. +#pragma once +#include "gfx_compat.h" diff --git a/src/gfx/front_win.c b/src/gfx/front_win.c new file mode 100644 index 0000000..63f2bfa --- /dev/null +++ b/src/gfx/front_win.c @@ -0,0 +1,58 @@ +// nn::vi (display, layer, native window, vsync event) for the recompiled game. +// Ported from MacNativeGames games/totk/runtime/frontend.m @b75be8c (MIT): the display and layer are opaque host +// identities; the native window is the host window object the game hands back to NVN (nvnWindowBuilderSetNativeWindow). +// The vsync event is signalled by a host thread at the display rate. Milestone 3.1: no window yet, a 60 Hz clock. +#include + +#include "compat/gfx_compat.h" +#include "host/hle.h" + +static int vi_ready; +static unsigned display_identity, layer_identity, window_identity; +static pthread_mutex_t vsync_lock = PTHREAD_MUTEX_INITIALIZER; +static void *vsync_events[16]; +static unsigned vsync_count; + +void *front_metal_layer(void) { return &window_identity; } // milestone 3.1: the Vulkan surface comes in 3.2 +void front_frame_presented(void) {} +uint64_t front_game_res(int docked) { (void)docked; return (720ULL << 32) | 1280; } + +static void *vsync_thread(void *unused) { + (void)unused; + pthread_setname_np("totk-vsync"); // gfx_compat.h: the Darwin form + const uint64_t period = 16666667; // 60 Hz until the window reports its display + uint64_t next = runtime_now_ns() + period; + for (;;) { + plat_sleep_until_ns(next); + next += period; + pthread_mutex_lock(&vsync_lock); + for (unsigned i = 0; i < vsync_count; i++) hle_signal_system_event(vsync_events[i]); + pthread_mutex_unlock(&vsync_lock); + } + return NULL; +} + +static void vi_initialize(void) { + vi_ready = 1; + pthread_t t; + if (pthread_create(&t, NULL, vsync_thread, NULL)) runtime_fail("vsync thread"); + fprintf(stderr, "[totk] VI display initialized (no window yet, 60 Hz vsync clock)\n"); +} +static uint32_t vi_open(void **out) { if (!vi_ready || !out) return 0xf601; *out = &display_identity; return 0; } +static uint32_t vi_layer(void **out, void *display) { if (!out || display != &display_identity) return 0xf601; *out = &layer_identity; return 0; } +static uint32_t vi_window(void **out, void *layer) { if (!out || layer != &layer_identity) return 0xf601; *out = front_metal_layer(); return 0; } +static uint32_t vi_vsync(void *ev, void *display) { + if (!ev || display != &display_identity) return 0xf601; + hle_bind_system_event(ev, 1); // on the calling guest thread: creates the SDK's own system event + pthread_mutex_lock(&vsync_lock); + if (vsync_count >= 16) runtime_fail("VI display event limit"); + vsync_events[vsync_count++] = ev; + pthread_mutex_unlock(&vsync_lock); + return 0; +} + +HLE_NATIVE("_ZN2nn2vi10InitializeEv", vi_initialize, 'v') +HLE_NATIVE("_ZN2nn2vi18OpenDefaultDisplayEPPNS0_7DisplayE", vi_open, 'w') +HLE_NATIVE("_ZN2nn2vi11CreateLayerEPPNS0_5LayerEPNS0_7DisplayE", vi_layer, 'w') +HLE_NATIVE("_ZN2nn2vi15GetNativeWindowEPPvPNS0_5LayerE", vi_window, 'w') +HLE_NATIVE("_ZN2nn2vi20GetDisplayVsyncEventEPNS_2os15SystemEventTypeEPNS0_7DisplayE", vi_vsync, 'w') diff --git a/src/gfx/gfx_bridge.c b/src/gfx/gfx_bridge.c new file mode 100644 index 0000000..b5ecda8 --- /dev/null +++ b/src/gfx/gfx_bridge.c @@ -0,0 +1,22 @@ +// The game's entry into graphics: nv::SetGraphicsAllocator / nv::SetGraphicsDevtoolsAllocator / nv::InitializeGraphics. +// Ported from MacNativeGames games/totk/runtime/gfx_bridge.c @b75be8c (MIT) without its diagnostic watchpoint. +// The allocator callbacks are guest functions: kept as guest addresses (host code calls them through GCALL). +#include "compat/gfx_compat.h" +#include "host/hle.h" + +typedef struct { uint64_t alloc, free, realloc, user; } GuestAllocator; +GuestAllocator g_nv_alloc, g_nv_devtools_alloc; +extern uint64_t g_t0_ns; + +static void graphics_allocator(uint64_t a, uint64_t f, uint64_t r, uint64_t u) { g_nv_alloc = (GuestAllocator){a, f, r, u}; } +static void graphics_dev_allocator(uint64_t a, uint64_t f, uint64_t r, uint64_t u) { g_nv_devtools_alloc = (GuestAllocator){a, f, r, u}; } +static void graphics_initialize(void *mem, uint64_t size) { + if (!mem || !size) runtime_fail("invalid graphics work memory"); + g_t0_ns = runtime_now_ns(); + fprintf(stderr, "[totk] graphics initialized: %llu KiB work memory at %p (backend: none yet)\n", + (unsigned long long)(size >> 10), mem); +} + +HLE_NATIVE("_ZN2nv18InitializeGraphicsEPvm", graphics_initialize, 'v') +HLE_NATIVE("_ZN2nv20SetGraphicsAllocatorEPFPvmmS0_EPFvS0_S0_EPFS0_S0_mS0_ES0_", graphics_allocator, 'v') +HLE_NATIVE("_ZN2nv28SetGraphicsDevtoolsAllocatorEPFPvmmS0_EPFvS0_S0_EPFS0_S0_mS0_ES0_", graphics_dev_allocator, 'v') diff --git a/src/gfx/host/dm.h b/src/gfx/host/dm.h new file mode 100644 index 0000000..15aeb32 --- /dev/null +++ b/src/gfx/host/dm.h @@ -0,0 +1,20 @@ +#pragma once +#include +#include +#include +#include +#include +#include +#define LOG(...) do {fprintf(stderr,"[totk-gfx] " __VA_ARGS__);fputc('\n',stderr);}while(0) +#define die(...) do {LOG(__VA_ARGS__);exit(2);}while(0) +typedef struct {const char *name;uint8_t *base;size_t size;} module_t; +module_t *module_of(const void *p); +uint64_t now_ns(void);uint64_t guest_ns(void); +int dm_detclock_enabled(void); +void dm_fault_hook_register(int (*fn)(void *,int,void *)); +void dm_guest_write_prep_register(void (*fn)(void *,size_t)); +void dm_res_report_register(void (*fn)(uint32_t,const char *)); +void dm_guest_write_prepare(void *addr,size_t n); +void crash_set_wait_dump(void (*fn)(void)); +void dm_guest_frame_tick(void); +void dm_shot_written(const char *path,int ok); diff --git a/src/gfx/host/front.h b/src/gfx/host/front.h new file mode 100644 index 0000000..8c01089 --- /dev/null +++ b/src/gfx/host/front.h @@ -0,0 +1,5 @@ +#pragma once +#include +void *front_metal_layer(void); +void front_frame_presented(void); +uint64_t front_game_res(int docked); diff --git a/src/gfx/host/hle.h b/src/gfx/host/hle.h new file mode 100644 index 0000000..1af995d --- /dev/null +++ b/src/gfx/host/hle.h @@ -0,0 +1,7 @@ +// [Windows port] Stands in for MacNativeGames d2r_gfx/host/hle.h: the graphics front end's host services (dm.h) plus the +// runtime's guest-symbol binding (HLE_FUNC / HLE_NATIVE, hle_register) and the Horizon host (runtime_fail ...). +#pragma once +#include "../../runtime/hle.h" +#include "../../host/host.h" +#undef LOG +#include "dm.h" diff --git a/src/gfx/nvn/LICENSE-MacNativeGames b/src/gfx/nvn/LICENSE-MacNativeGames new file mode 100644 index 0000000..6102617 --- /dev/null +++ b/src/gfx/nvn/LICENSE-MacNativeGames @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 MacNativeGames contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/src/gfx/nvn/hle_nvn.c b/src/gfx/nvn/hle_nvn.c new file mode 100644 index 0000000..07714df --- /dev/null +++ b/src/gfx/nvn/hle_nvn.c @@ -0,0 +1,2824 @@ +/* Windows port of MacNativeGames games/totk/third_party/d2r_gfx/gfx/nvn/hle_nvn.c @b75be8c (MIT). Changes are marked + [port]: implementations registered in PE sections (nvn_registry.c); the game gets fake guest addresses of the + generated register thunks (nvn_thunks.gen.c) instead of native function pointers; Mach APIs through + src/gfx/compat/gfx_compat.h. */ +#include +// NVN 接口层：接管 main 导入的 nvnBootstrapLoader，sdk 里的 NVN 驱动整块不执行。 +// 523 个 NVN 名字一律返回计数跳板（nvn_thunks.S），跳板再转到这里的实现： +// · 有语义的（设备 / 内存池 / 贴图与采样器 / 队列 / 窗口 / 同步 / 事件 / 命令缓冲内存）按 NVN 语义实现，对象是真的； +// · 只改 GPU 状态、没有 CPU 可见结果的（状态对象、绑定、绘制、清除…）录成命令，提交时回放； +// · 其余（main 只加载不调用的 307 个 + 头文件无原型的）接 nvn_unimpl_entry：被调到即报红打印名字。 +// GPU 地址 = CPU 地址（内存池用游戏给的真内存，同时建成零拷贝 MTLBuffer）。命令录进游戏给的命令内存，提交时回放成 +// Metal 命令缓冲（nvn_mtl.m）：缓冲拷贝 / uniform 更新走 blit（GPU 时序），事件写值 / 计数器在 GPU 完成后写； +// 同步对象 = 提交栅栏值，等待只在 GPU 完成后放行。 +// 每个实现都按 nvn_protos.h 的原型做编译期签名检查（NVNboolean 参数按 uint32_t 收、只看低 8 位）。 +// 环境开关：DM_NVN_FRAMES=N 前 N 帧逐帧打印调用表（默认 3）；DM_NVN_TSV=路径 每帧调用表与计时写 TSV（DM_NVN_TSV_CALLS=0 只写计时）；DM_SLOWFRAME=1 只把掉帧的逐帧分项写进日志；DM_NVN_LOG=1 记对象创建。 +// NVN interface layer: takes over the nvnBootstrapLoader imported by main; the NVN driver in sdk is not executed at all. +// All 523 NVN names return counting thunks (nvn_thunks.S), and the thunks forward to the implementations here: +// - those with semantics (device / memory pools / textures and samplers / queue / window / sync / events / command buffer memory) are implemented per NVN semantics, and the objects are real; +// - those that only change GPU state and have no CPU-visible result (state objects, bindings, draws, clears...) are recorded as commands and replayed at submit; +// - the rest (the 307 that main only loads but never calls + those without a prototype in the header) go to nvn_unimpl_entry: reports a failure and prints the name when called. +// GPU address = CPU address (memory pools use the real memory given by the game, also built as zero-copy MTLBuffer). Commands are recorded into the command memory given by the game and replayed at submit into a +// Metal command buffer (nvn_mtl.m): buffer copies / uniform updates go through blit (GPU ordering), event value writes / counters are written after the GPU completes; +// sync object = submit fence value, and waits are released only after the GPU completes. +// Every implementation gets a compile-time signature check against the prototypes in nvn_protos.h (NVNboolean parameters are received as uint32_t, only the low 8 bits are looked at). +// Environment switches: DM_NVN_FRAMES=N prints the call table per frame for the first N frames (default 3); DM_NVN_TSV=path writes the per-frame call table and timing to a TSV (DM_NVN_TSV_CALLS=0 writes timing only); DM_SLOWFRAME=1 writes only the per-frame breakdown of dropped frames to the log; DM_NVN_LOG=1 logs object creation. +#include "../host/hle.h" +#include "nvn_protos.h" +#include "nvn_count.h" +#include "nvn_tex.h" +#include "nvn_hmap.h" +#include "nvn_mtl.h" +#include "nvn_pace.h" +#include "nvn_diag_range.h" +#include "../host/front.h" +#include "nvn_dirty.h" +#include "totk_vpool.h" +#include "totk_lifelog.h" +#include +#include +#include +#include +#include +#include +#include +#if TARGET_OS_OSX +#define DM_MEM_CAP_DEFAULT_GB 24 +#else +#define DM_MEM_CAP_DEFAULT_GB 0 +#endif +#include +#include +#include +#include +#include +#include +#include + +// ---------- 槽表、跳板、未实现桩 ---------- +// ---------- Slot table, thunks, unimplemented stubs ---------- +typedef struct { _Atomic uint64_t cnt; void *fn; } nvn_slot_t; +nvn_slot_t nvn_slot[NVN_NFUNC]; // 跳板按 i*16 访问，布局不可改 | The thunk accesses it by i*16; the layout must not change +_Static_assert(sizeof(nvn_slot_t) == 16, "跳板按 16 字节步长寻址"); +extern void *nvn_thunk[NVN_NFUNC]; +extern const char *nvn_names[NVN_NFUNC]; +extern void nvn_unimpl_entry(void); +extern void nvn_prof_entry(void); +void *nvn_real[NVN_NFUNC]; // DM_NVN_PROF：计时包装调用的真实现 | DM_NVN_PROF: the real implementation called by the timing wrapper +_Atomic uint64_t nvn_prof_ticks[NVN_NFUNC]; // DM_NVN_PROF：各函数累计耗时（cntvct 计数） | DM_NVN_PROF: cumulative time of each function (cntvct counts) +static uint8_t g_unimpl_seen[NVN_NFUNC]; + +uint64_t nvn_unimpl_c(nvn_slot_t *slot, uint64_t x0, uint64_t x1, uint64_t x2) { + unsigned i = (unsigned)(slot - nvn_slot); + if (!g_unimpl_seen[i]++) LOG("【红】NVN 未实现被调用：%s (x0=%#llx x1=%#llx x2=%#llx)", nvn_names[i], x0, x1, x2); + die("unimplemented NVN call %s",nvn_names[i]); +} + +#include "nvn_impl.h" + +static int g_log; +uint64_t nvn_new_serial(void) { static _Atomic uint64_t n; return ++n; } +static unsigned g_nsem; // 语义实现个数（nvn_init_slots 填） | Number of semantic implementations (filled by nvn_init_slots) +#define OLOG(...) do { if (g_log) LOG("[nvn] " __VA_ARGS__); } while (0) + +static void nvn_init_slots(void) { + static int done; if (done) return; done = 1; + g_log = getenv("DM_NVN_LOG") != NULL; + unsigned nim = 0, hit = 0; + const nvn_impl *im = nvn_registry_impls(&nim); /* [port] PE section instead of getsectiondata */ + for (unsigned i = 0; i < NVN_NFUNC; i++) nvn_slot[i].fn = (void *)nvn_unimpl_entry; + for (unsigned k = 0; k < nim; k++) { + unsigned i = 0; + while (i < NVN_NFUNC && strcmp(nvn_names[i], im[k].name)) i++; + if (i == NVN_NFUNC) die("NVN 实现 %s 不在 main 的名字表里", im[k].name); + nvn_slot[i].fn = im[k].fn; hit++; + } + g_nsem = hit; +} + +// TOTK: resolve an implemented entry by name (for NVN_EXTRA fallbacks that forward to regular entries). +void *nvn_slot_fn(const char *name) { + nvn_init_slots(); + for (unsigned i = 0; i < NVN_NFUNC; i++) if (!strcmp(nvn_names[i], name)) return nvn_slot[i].fn; + die("nvn_slot_fn: unknown %s", name); +} +void *runtime_hle_wrap(void *fn); // runtime/hle_stack.c: run on the calling guest thread's host stack +void *runtime_hle_wrap_named(void *fn, const char *name); // the same, with the NVN name for diagnostics (hletrace) +/* [port] The game receives the fake guest address of the name's register thunk (nvn_thunks.gen.c, hle_register): its + recompiled code calls it through rt_dispatch. A name without an implementation gets a host stub that stops the game + with the name when it is called (as MacNativeGames' nvn_unimpl_entry does). */ +static void *nvn_lookup(const char *name) { + nvn_init_slots(); + if (!name) return NULL; + uint64_t a = nvn_registry_guest_address(name); + if (!a) LOG("【红】NVN 取址未知名字：%s（未实现：被调用时报错停止）", name); + return (void *)(uintptr_t)a; +} + +// nvnBootstrapLoader 由 main 导入（sdk 定义）；强制接管后 sdk 的 NVN 驱动不再被引用 +// nvnBootstrapLoader is imported by main (defined in sdk); after it is forcibly taken over, the NVN driver in sdk is no longer referenced +static void *h_nvnBootstrapLoader(const char *name) { return nvn_lookup(name); } +HLE_NATIVE("nvnBootstrapLoader", h_nvnBootstrapLoader, 'x') /* [port] the SDK's NVN driver is never run */ + +// ---------- 布尔参数：Switch 调用方不扩展，只看低 8 位 ---------- +// ---------- Boolean parameters: Switch callers do not extend them, only look at the low 8 bits ---------- +#define B(x) ((x) & 0xff) + +#include "nvn_obj.h" +#include "totk_timeline.h" +void mtl_args_stats(uint64_t *enc, uint64_t *reuse); + +// ---------- 设备 ---------- +// ---------- Device ---------- +// TOTK translation thread (see "translation thread" below) +static uint64_t fence_now(int *token); +static int fence_wait(uint64_t k, uint64_t timeout_ns); +static void x_drain(void); +static int async_replay(void); +static int call_stats_on(void); +static void call_stats_end(const void *cb, const uint8_t *copy); +NVN(nvnDeviceBuilderSetDefaults, void, NVNdeviceBuilder *b) { memset(b, 0, sizeof(dev_builder_t)); } +NVN(nvnDeviceBuilderSetFlags, void, NVNdeviceBuilder *b, int f) { ((dev_builder_t *)b)->flags = f; } +NVN(nvnDeviceInitialize, NVNboolean, NVNdevice *d, const NVNdeviceBuilder *b) { + dev_t_ *o = (dev_t_ *)d; memset(o, 0, sizeof *o); + o->magic = MAGIC('D', 'E', 'V', 0); o->flags = ((const dev_builder_t *)b)->flags; + OLOG("DeviceInitialize %p flags=%#x", (void *)d, o->flags); + return 1; +} +NVN(nvnDeviceFinalize, void, NVNdevice *d) { ((dev_t_ *)d)->magic = 0; } +NVN(nvnDeviceGetProcAddress, PFNNVNGENERICFUNCPTRPROC, const NVNdevice *d, const char *name) { + (void)d; return (PFNNVNGENERICFUNCPTRPROC)nvn_lookup(name); +} + +// 设备属性：Tegra X1 实测值（nnheaders nvn_DeviceConstantsNX.h，在 SMO 上逐项调 nvnDeviceGetInteger 得到） +// Device properties: values measured on Tegra X1 (nnheaders nvn_DeviceConstantsNX.h, obtained by calling nvnDeviceGetInteger item by item on SMO) +static const int g_devinfo[] = { + [0x00] = 55, [0x01] = 15, // Experimental ABI target: TOTK main+e9d614 requires major 55, minor >= 15. Unsupported functions stop. + [0x02] = 14, [0x03] = 65536, [0x04] = 256, [0x05] = 8, [0x06] = 16, [0x07] = 4, [0x08] = 16, [0x09] = 32, + [0x0A] = 16, [0x0B] = 4, [0x0C] = 4, [0x0D] = 4, [0x0E] = 16, [0x0F] = 32, [0x10] = 32, [0x11] = 256, [0x12] = 256, + [0x13] = 4, [0x14] = 8, [0x15] = 512, [0x16] = 256, [0x17] = 2, [0x18] = 256, [0x19] = 4096, [0x1A] = 131072, + [0x1B] = 16, [0x1C] = 1536, [0x1D] = 1024, [0x1E] = 64, [0x1F] = 1536, [0x20] = 65535, [0x21] = 65535, [0x22] = 65535, + [0x23] = 8, [0x24] = 1048576, [0x25] = 4096, [0x26] = 16, [0x27] = 4096, [0x28] = 1, [0x29] = 1, [0x2A] = 1, + [0x2B] = 262144, [0x2C] = 15, [0x2D] = 2048, [0x2E] = 1, [0x2F] = 1, [0x30] = 14, [0x31] = 5, [0x32] = 1, [0x33] = 8, + [0x34] = 8, [0x35] = 4, [0x36] = 32, [0x37] = 128, [0x38] = 32, [0x39] = 128, [0x3A] = 65536, [0x3B] = 1, [0x3C] = 4, + [0x3D] = 16384, [0x3E] = 134217728, [0x3F] = 2048, [0x40] = 16384, [0x41] = 16384, [0x42] = 1, [0x43] = 1, [0x44] = 1, + [0x45] = 1, [0x46] = 5, [0x47] = 1, [0x48] = 1, [0x49] = 1, [0x4A] = 1, [0x4B] = 1, [0x4C] = 1, [0x4D] = 16, + [0x4E] = 1024, [0x4F] = 32, [0x50] = 4096, [0x51] = 65536, [0x52] = 65536, [0x53] = 256, [0x54] = 16384, + [0x55] = 262144, [0x56] = 4096, [0x57] = 1, [0x58] = 4, [0x59] = 2, [0x5A] = 0, [0x5B] = 4096, [0x5C] = 16384, + [0x5D] = 4096, [0x5E] = 128, [0x5F] = 4096, +}; +NVN(nvnDeviceGetInteger, void, const NVNdevice *d, NVNdeviceInfo what, int *out) { + (void)d; + unsigned w = (unsigned)what; + // 0x5F 分离采样器绑定数、0x60/0x61 在 SMO 实测表里没有：0x5F 按采样器池上限给 4096 未核，0x60/0x61 报红给 0 + // 0x5F separate sampler binding count and 0x60/0x61 are not in the table measured on SMO: 0x5F is given 4096 per the sampler pool limit, unverified; 0x60/0x61 report a failure and return 0 + if (w >= sizeof g_devinfo / sizeof *g_devinfo) { die("unsupported NVN device property %#x", w); } + *out = g_devinfo[w]; + OLOG("DeviceGetInteger %#x -> %d", w, *out); +} +/* Metal chooses fast-clear encodings itself; this backend has no programmable + * NVN fast-clear registry. Report unavailable so the game retains normal clears. */ +NVN(nvnDeviceRegisterFastClearColor, NVNboolean, NVNdevice *d,const float *color,NVNformat format){(void)d;(void)color;(void)format;return 0;} +// TOTK: fast-clear registration is an optimisation hint; "not registered" keeps the normal clear path. +NVN(nvnDeviceRegisterFastClearColori, NVNboolean, NVNdevice *d, const int *c, NVNformat f) { (void)d; (void)c; (void)f; return 0; } +NVN(nvnDeviceRegisterFastClearColorui, NVNboolean, NVNdevice *d, const uint32_t *c, NVNformat f) { (void)d; (void)c; (void)f; return 0; } +NVN_NOPROTO(nvnDeviceRegisterFastClearDepth, NVNboolean, NVNdevice *d) { (void)d; return 0; } +NVN(nvnDeviceGetImageHandle, NVNimageHandle, const NVNdevice *d, int id) { (void)d; return (NVNimageHandle)id; } +// 分离贴图 / 采样器句柄：值 = 描述符号（采样器号放 20 位以上，与组合句柄的位布局一致） +// Separate texture / sampler handles: value = descriptor index (the sampler index goes in bit 20 and above, consistent with the bit layout of combined handles) +// TOTK: combined handle = texture id (20 bits) | sampler id << 20, the encoding the replay layer decodes. +NVN(nvnDeviceGetTextureHandle, NVNtextureHandle, const NVNdevice *d, int tid, int sid) { + (void)d; if (tid < 0 || tid >= (1 << 20) || sid < 0 || sid >= (1 << 12)) die("texture handle ids out of range %d/%d", tid, sid); + return (NVNtextureHandle)((uint64_t)tid | (uint64_t)sid << 20); +} +NVN(nvnDeviceGetTexelFetchHandle, NVNtextureHandle, const NVNdevice *d, int tid) { + (void)d; if (tid < 0 || tid >= (1 << 20)) die("texel fetch id out of range %d", tid); return (NVNtextureHandle)tid; +} +NVN(nvnDeviceGetSeparateTextureHandle, NVNseparateTextureHandle, const NVNdevice *d, int id) { (void)d; return (NVNseparateTextureHandle){(uint64_t)id}; } +NVN(nvnDeviceGetSeparateSamplerHandle, NVNseparateSamplerHandle, const NVNdevice *d, int id) { (void)d; return (NVNseparateSamplerHandle){(uint64_t)id << 20}; } +NVN(nvnDeviceInstallDebugCallback, void, NVNdevice *d, const PFNNVNDEBUGCALLBACKPROC cb, void *data, uint32_t enable) { + dev_t_ *o = (dev_t_ *)d; o->dbg_cb = B(enable) ? (void *)cb : NULL; o->dbg_data = data; +} +NVN(nvnDeviceSetWindowOriginMode, void, NVNdevice *d, NVNwindowOriginMode m) { ((dev_t_ *)d)->origin = m; LOG("[nvn] 窗口原点 %s", m ? "左上（同 D3D / Metal）" : "左下（GL）"); } +static _Atomic int g_depth_near_minus_w; +int totk_depth_near_minus_w(void) { return atomic_load(&g_depth_near_minus_w); } +NVN(nvnDeviceSetDepthMode, void, NVNdevice *d, NVNdepthMode m) { ((dev_t_ *)d)->depth_mode = m; atomic_store(&g_depth_near_minus_w, m == NVN_DEPTH_MODE_NEAR_IS_MINUS_W); LOG("[nvn] 深度模式 %s", m ? "近=0（同 D3D / Metal）" : "近=-w（GL）：顶点着色器翻译时做 z=(z+w)/2 转换"); } +NVN(nvnDeviceFinalizeCommandHandle, void, NVNdevice *d, NVNcommandHandle h) { (void)d; (void)h; } + +// ---------- 内存池 ---------- +// ---------- Memory pools ---------- +NVN(nvnMemoryPoolBuilderSetDefaults, void, NVNmemoryPoolBuilder *b) { memset(b, 0, sizeof(pool_builder_t)); } +NVN(nvnMemoryPoolBuilderSetDevice, void, NVNmemoryPoolBuilder *b, NVNdevice *d) { ((pool_builder_t *)b)->dev = d; } +NVN(nvnMemoryPoolBuilderSetStorage, void, NVNmemoryPoolBuilder *b, void *m, size_t n) { ((pool_builder_t *)b)->mem = m; ((pool_builder_t *)b)->size = n; } +NVN(nvnMemoryPoolBuilderSetFlags, void, NVNmemoryPoolBuilder *b, int f) { ((pool_builder_t *)b)->flags = f; } +// TOTK: logging switches checked on per-call paths (command memory, recording, submits) are read once. +static int cmdlog_env(void) { static int on = -1; if (on < 0) on = getenv("TOTK_CMD_LOG") != NULL; return on; } +static int poollog_env(void) { static int on = -1; if (on < 0) on = getenv("TOTK_POOL_LOG") != NULL; return on; } +NVN(nvnMemoryPoolInitialize, NVNboolean, NVNmemoryPool *p, const NVNmemoryPoolBuilder *b) { + const pool_builder_t *s = (const pool_builder_t *)b; pool_t *o = (pool_t *)p; + memset(o, 0, sizeof *o); + o->magic = MAGIC('P', 'O', 'O', 'L'); o->flags = s->flags; o->mem = s->mem; o->size = s->size; + if ((o->flags & (NVN_MEMORY_POOL_FLAGS_PHYSICAL | NVN_MEMORY_POOL_FLAGS_VIRTUAL)) && !totk_vpool_init(o)) { o->magic = 0; return 0; } + int gpu_visible = !(o->flags & NVN_MEMORY_POOL_FLAGS_GPU_NO_ACCESS); + if (o->mem && gpu_visible) { mtl_pool_add(o->mem, o->size); nvn_dirty_add_range(o->mem, o->size); } + if (!o->mem) LOG("【红】MemoryPoolInitialize 池没有存储 flags=%#x size=%#llx", o->flags, o->size); + OLOG("MemoryPool %p mem=%p size=%#llx flags=%#x", (void *)p, (void *)o->mem, o->size, o->flags); + if (poollog_env()) LOG("[pool] %p mem=%p..%p size=%#llx flags=%#x game=%p", (void *)p, (void *)o->mem, (void *)(o->mem + o->size), o->size, o->flags, (void *)o->game_mem); + return o->mem != NULL; +} +NVN(nvnMemoryPoolFinalize, void, NVNmemoryPool *p) { + pool_t *o = (pool_t *)p; + if (o->mem && !(o->flags & NVN_MEMORY_POOL_FLAGS_GPU_NO_ACCESS)) { x_drain(); nvn_dirty_remove_range(o->mem); mtl_pool_del(o->mem); } + totk_vpool_fini(o); o->magic = 0; +} +NVN(nvnMemoryPoolFlushMappedRange, void, const NVNmemoryPool *p, ptrdiff_t off, size_t n) { (void)p; (void)off; (void)n; } +NVN(nvnMemoryPoolGetFlags, NVNmemoryPoolFlags, const NVNmemoryPool *p){return (NVNmemoryPoolFlags)((const pool_t *)p)->flags;} +NVN(nvnMemoryPoolMap, void *, const NVNmemoryPool *p){const pool_t *v=(const void *)p;if(v->magic!=MAGIC('P','O','O','L'))return NULL;return (v->flags&NVN_MEMORY_POOL_FLAGS_CPU_NO_ACCESS)?NULL:v->mem;} +NVN(nvnMemoryPoolMapVirtual, NVNboolean, NVNmemoryPool *p, int n, const NVNmappingRequest *r) { + return (NVNboolean)totk_vpool_map((pool_t *)p, n, (const totk_mapping_request *)(const void *)r); +} +NVN(nvnMemoryPoolGetBufferAddress, NVNbufferAddress, const NVNmemoryPool *p) { return (NVNbufferAddress)((const pool_t *)p)->mem; } + +// Buffer objects reference their pool; they do not own the backing allocation. +typedef struct{NVNdevice *dev;pool_t *pool;int64_t off;uint64_t size;} totk_buffer_builder; +typedef struct{uint32_t magic;pool_t *pool;int64_t off;uint64_t size;} totk_buffer; +FITS(totk_buffer_builder,NVNbufferBuilder);FITS(totk_buffer,NVNbuffer); +NVN(nvnBufferBuilderSetDevice, void, NVNbufferBuilder *b,NVNdevice *d){((totk_buffer_builder *)b)->dev=d;} +NVN(nvnBufferBuilderSetDefaults, void, NVNbufferBuilder *b){NVNdevice *d=((totk_buffer_builder *)b)->dev;memset(b,0,sizeof(totk_buffer_builder));((totk_buffer_builder *)b)->dev=d;} +NVN(nvnBufferBuilderSetStorage, void, NVNbufferBuilder *b,NVNmemoryPool *p,ptrdiff_t off,size_t size){totk_buffer_builder *v=(void *)b;v->pool=(void *)p;v->off=off;v->size=size;} +NVN(nvnBufferInitialize, NVNboolean, NVNbuffer *b,const NVNbufferBuilder *builder){const totk_buffer_builder *v=(const void *)builder;if(!v->pool||v->pool->magic!=MAGIC('P','O','O','L')||v->off<0||(uint64_t)v->off>v->pool->size||v->size>v->pool->size-(uint64_t)v->off)return 0;*(totk_buffer *)b=(totk_buffer){MAGIC('B','U','F',0),v->pool,v->off,v->size};return 1;} +NVN(nvnBufferFinalize, void, NVNbuffer *b){memset(b,0,sizeof(totk_buffer));} +NVN(nvnBufferMap, void *, const NVNbuffer *b){const totk_buffer *v=(const void *)b;return v->pool&&v->pool->mem?v->pool->mem+v->off:NULL;} +NVN(nvnBufferGetAddress, NVNbufferAddress, const NVNbuffer *b){return (uintptr_t)h_nvnBufferMap(b);} +NVN(nvnBufferGetMemoryPool, NVNmemoryPool *, const NVNbuffer *b){return (void *)((const totk_buffer *)b)->pool;} +NVN(nvnBufferGetMemoryOffset, ptrdiff_t, const NVNbuffer *b){return ((const totk_buffer *)b)->off;} +NVN(nvnBufferGetSize, size_t, const NVNbuffer *b){return ((const totk_buffer *)b)->size;} +NVN(nvnBufferFlushMappedRange, void, const NVNbuffer *b,ptrdiff_t off,size_t size){const totk_buffer *v=(const void *)b;if(off<0||(uint64_t)off>v->size||size>v->size-(uint64_t)off)die("invalid NVN buffer flush range");dm_guest_write_prepare((uint8_t *)h_nvnBufferMap(b)+off,size);} + +// ---------- 描述符池 ---------- +// ---------- Descriptor pools ---------- +NVN(nvnTexturePoolInitialize, NVNboolean, NVNtexturePool *tp, const NVNmemoryPool *p, ptrdiff_t off, int n) { + *(dpool_t *)tp = (dpool_t){(const pool_t *)p, off, n}; OLOG("TexturePool n=%d", n); mtl_set_tex_pool_n(n); return 1; +} +NVN(nvnTexturePoolFinalize, void, NVNtexturePool *tp) { (void)tp; } +extern uint64_t g_frame_pub; +NVN(nvnTexturePoolRegisterTexture, void, const NVNtexturePool *tp, int id, const NVNtexture *t, const NVNtextureView *v) { (void)tp; + { static long want = -2; if (want == -2) { const char *e = getenv("DM_NVN_REGLOG"); want = e ? atol(e) : -1; } // TOTK diagnostic: DM_NVN_REGLOG= logs every registration of that texture id + static long rf0 = -2, rf1; if (rf0 == -2) { const char *e = getenv("DM_NVN_REGLOG_FRAMES"); rf0 = e ? atol(e) : -1; rf1 = e && strchr(e, '-') ? atol(strchr(e, '-') + 1) : rf0; } // DM_NVN_REGLOG_FRAMES=a-b: log every registration in those frames + if ((want >= 0 && id == want) || (rf0 >= 0 && (long)g_frame_pub >= rf0 && (long)g_frame_pub <= rf1)) { const tex_t *x = (const tex_t *)t; const view_t *vv = (const view_t *)v; + LOG("[reglog] F%llu id=%d ser=%llu fmt=%#x %ux%u lv%u flags=%#x mem=%p pool=%p off=%lld view=%s vfmt=%#x vlv=%d+%d", (unsigned long long)g_frame_pub, id, (unsigned long long)x->serial, x->d.format, x->d.width, x->d.height, x->d.levels, x->d.flags, (void *)x->mem, (void *)x->pool, (long long)x->off, + vv ? "yes" : "no", vv && vv->set_format ? vv->format : 0, vv && vv->set_levels ? vv->base_level : -1, vv && vv->set_levels ? vv->nlevels : 0); } } + totk_life_reg(id, (const tex_t *)t); + mtl_reg_tex(id, (const tex_t *)t, (const view_t *)v); } + +NVN(nvnSamplerPoolInitialize, NVNboolean, NVNsamplerPool *sp, const NVNmemoryPool *p, ptrdiff_t off, int n) { + *(dpool_t *)sp = (dpool_t){(const pool_t *)p, off, n}; OLOG("SamplerPool n=%d", n); return 1; +} +NVN(nvnSamplerPoolFinalize, void, NVNsamplerPool *sp) { (void)sp; } +NVN(nvnSamplerPoolRegisterSampler, void, const NVNsamplerPool *sp, int id, const NVNsampler *s) { (void)sp; mtl_reg_samp(id, &((const samp_t *)s)->p); } + +// ---------- 贴图 ---------- +// ---------- Textures ---------- +#define TB(b) ((tex_builder_t *)(b)) +NVN(nvnTextureBuilderSetDefaults, void, NVNtextureBuilder *b) { + NVNdevice *dev = TB(b)->dev; memset(b, 0, sizeof(tex_builder_t)); TB(b)->dev = dev; + TB(b)->d.swizzle_r=2;TB(b)->d.swizzle_g=3;TB(b)->d.swizzle_b=4;TB(b)->d.swizzle_a=5; + TB(b)->d.target = NVN_TEXTURE_TARGET_2D; TB(b)->d.width = TB(b)->d.height = TB(b)->d.depth = TB(b)->d.levels = 1; +} +NVN(nvnTextureBuilderSetDevice, void, NVNtextureBuilder *b, NVNdevice *d) { TB(b)->dev = d; } +NVN(nvnTextureBuilderSetFlags, void, NVNtextureBuilder *b, int f) { TB(b)->d.flags = f; } +NVN(nvnTextureBuilderSetTarget, void, NVNtextureBuilder *b, NVNtextureTarget t) { TB(b)->d.target = t; } +NVN(nvnTextureBuilderSetWidth, void, NVNtextureBuilder *b, int v) { TB(b)->d.width = v; } +static void size_chain(int w, int h); +NVN(nvnTextureBuilderSetHeight, void, NVNtextureBuilder *b, int v) { TB(b)->d.height = v; size_chain(TB(b)->d.width, v); } +NVN(nvnTextureBuilderSetDepth, void, NVNtextureBuilder *b, int v) { TB(b)->d.depth = v; } +NVN(nvnTextureBuilderSetSize1D, void, NVNtextureBuilder *b, int w) { TB(b)->d.width = w; TB(b)->d.height = TB(b)->d.depth = 1; } +static void guest_chain(const char *what); +NVN(nvnTextureBuilderSetSize2D, void, NVNtextureBuilder *b, int w, int h) { TB(b)->d.width=w;TB(b)->d.height=h;TB(b)->d.depth=1; + /* TOTK_SIZE_CHAIN=x (diagnostic): the game call chain of the first 60 textures given this size (where render target sizes come from) */ + size_chain(w, h); } +static void size_chain(int w, int h) { + static int sw = -1, sh, n; if (sw < 0) { const char *e = getenv("TOTK_SIZE_CHAIN"); if (!e || sscanf(e, "%dx%d", &sw, &sh) != 2) sw = 0; } + if (sw && w == sw && h == sh && n++ < 60) { char m[64]; snprintf(m, sizeof m, "texture size %dx%d", w, h); guest_chain(m); } +} +NVN(nvnTextureBuilderSetSize3D, void, NVNtextureBuilder *b, int w, int h, int d) { TB(b)->d.width=w;TB(b)->d.height=h;TB(b)->d.depth=d; } +NVN(nvnTextureBuilderSetLevels, void, NVNtextureBuilder *b, int v) { TB(b)->d.levels = v; } +NVN(nvnTextureBuilderSetFormat, void, NVNtextureBuilder *b, NVNformat f) { TB(b)->d.format = f; } +// This adapter currently samples the depth plane; stencil-only sampling stops. +NVN(nvnTextureBuilderSetDepthStencilMode, void, NVNtextureBuilder *b,NVNtextureDepthStencilMode mode){(void)b;if(mode!=NVN_TEXTURE_DEPTH_STENCIL_MODE_DEPTH)die("stencil-plane texture sampling is not implemented");} +NVN(nvnTextureBuilderSetSwizzle, void, NVNtextureBuilder *b,NVNtextureSwizzle r,NVNtextureSwizzle g,NVNtextureSwizzle blue,NVNtextureSwizzle a){if((unsigned)r>5||(unsigned)g>5||(unsigned)blue>5||(unsigned)a>5)die("invalid NVN texture swizzle");TB(b)->d.swizzle_r=r;TB(b)->d.swizzle_g=g;TB(b)->d.swizzle_b=blue;TB(b)->d.swizzle_a=a;} +NVN(nvnTextureBuilderSetSamples, void, NVNtextureBuilder *b, int s) { TB(b)->d.samples = s; } +NVN(nvnTextureBuilderSetStride, void, NVNtextureBuilder *b, ptrdiff_t s) { TB(b)->d.stride = (uint32_t)s; } +NVN(nvnTextureBuilderSetStorage, void, NVNtextureBuilder *b, NVNmemoryPool *p, ptrdiff_t off) { TB(b)->pool = (pool_t *)p; TB(b)->off = off; } +NVN(nvnTextureBuilderSetPackagedTextureLayout, void, NVNtextureBuilder *b, const NVNpackagedTextureLayout *l) { + TB(b)->has_layout = l != NULL; if (l) memcpy(TB(b)->layout, l->layout, 8); + TB(b)->d.pkg_block_h = l ? l->layout[0] + 1 : 0; +} +// TOTK: packaged (already GPU-laid-out) data copied into the texture's pool storage at Initialize; the copy is +// a CPU write into the pool, so dirty tracking re-uploads it like any other packaged texture content. +NVN(nvnTextureBuilderSetPackagedTextureData, void, NVNtextureBuilder *b, const void *data) { TB(b)->pkg_data = data; } +NVN(nvnTextureBuilderGetPackagedTextureData, const void *, const NVNtextureBuilder *b) { return ((const tex_builder_t *)b)->pkg_data; } +/* Storage classes are opaque compatibility identifiers for this backend. + * All encoded texture storage lives in byte-addressed host pools and is decoded + * per texture descriptor; class 1 therefore shares that backing convention. + * Buffer textures use class 0. This is not a Tegra memory-kind number. + */ +static NVNstorageClass totk_storage_class(const nvn_texdesc *d){return d->target==NVN_TEXTURE_TARGET_BUFFER?0:1;} +NVN(nvnTextureBuilderGetStorageClass, NVNstorageClass, const NVNtextureBuilder *b){return totk_storage_class(&((const tex_builder_t *)b)->d);} +NVN(nvnTextureBuilderGetFlags, NVNtextureFlags, const NVNtextureBuilder *b) { return (NVNtextureFlags)((const tex_builder_t *)b)->d.flags; } +static uint64_t g_layout_logged; +static void log_layout(const tex_builder_t *b, const nvn_texlayout *L) { + if (b->has_layout && g_layout_logged++ < 16) + LOG("[nvn] 预排布贴图 %ux%ux%u 级%u 格式%#x 目标%d 布局字节 %02x%02x%02x%02x%02x%02x%02x%02x → 块高 log2=%u（取布局第 0 字节）尺寸 %#llx", + b->d.width, b->d.height, b->d.depth, b->d.levels, b->d.format, b->d.target, b->layout[0], b->layout[1], b->layout[2], + b->layout[3], b->layout[4], b->layout[5], b->layout[6], b->layout[7], L->tileH, L->size); +} +NVN(nvnTextureBuilderGetStorageSize, size_t, const NVNtextureBuilder *b) { + nvn_texlayout L; nvn_tex_layout(&((const tex_builder_t *)b)->d, &L); + if (!L.bpb) LOG("【红】TextureBuilderGetStorageSize 未知格式 %#x", ((const tex_builder_t *)b)->d.format); + log_layout((const tex_builder_t *)b, &L); + { static int sizelog = -1; if (sizelog < 0) sizelog = getenv("DM_NVN_SIZELOG") != NULL; + if (sizelog) { const nvn_texdesc *d = &((const tex_builder_t *)b)->d; + fprintf(stderr, "[nvn-size] t=%d f=%#x fl=%#x s=%d w=%u h=%u d=%u lv=%u st=%u pbh=%d size=%#llx tileH=%u tileD=%u layer=%#llx\n", + d->target, d->format, d->flags, d->samples, d->width, d->height, d->depth, d->levels, d->stride, d->pkg_block_h, + (unsigned long long)L.size, L.tileH, L.tileD, (unsigned long long)L.layer_size); } } + return L.size; +} +NVN(nvnTextureBuilderGetStorageAlignment, size_t, const NVNtextureBuilder *b) { + nvn_texlayout L; nvn_tex_layout(&((const tex_builder_t *)b)->d, &L); return L.align; +} +#define TX(t) ((tex_t *)(t)) +#define CTX(t) ((const tex_t *)(t)) +static void live_put(const tex_t *o); +NVN(nvnTextureInitialize, NVNboolean, NVNtexture *t, const NVNtextureBuilder *b) { + const tex_builder_t *s = (const tex_builder_t *)b; tex_t *o = TX(t); + if (o->magic == MAGIC('T', 'E', 'X', 0) && o->serial) { // re-initialised without Finalize: release the previous serial's host texture, else it leaks + static _Atomic unsigned n; if (atomic_fetch_add(&n, 1) < 3) LOG("[nvn] texture %p re-initialised without Finalize (serial %llu released)", (void *)t, (unsigned long long)o->serial); + mtl_tex_release(o); + } + memset(o, 0, sizeof *o); + o->magic = MAGIC('T', 'E', 'X', 0); o->d = s->d; o->pool = s->pool; o->off = s->off; + o->has_layout = s->has_layout; memcpy(o->layout, s->layout, 8); + nvn_tex_layout(&o->d, &o->L); + o->mem = s->pool && s->pool->mem ? s->pool->mem + s->off : NULL; + if (s->pkg_data) { + if (!o->mem) die("packaged texture data without CPU-visible storage"); + dm_guest_write_prepare(o->mem, (size_t)o->L.size); + memcpy(o->mem, s->pkg_data, (size_t)o->L.size); + } + o->serial = nvn_new_serial(); + live_put(o); + if (totk_life_match(o)) totk_life_log("init tex=%p ser=%llu mem=%p", (void *)t, (unsigned long long)o->serial, (void *)o->mem); + OLOG("Texture %p %ux%ux%u lv%u fmt=%#x tgt=%d flags=%#x size=%#llx", (void *)t, o->d.width, o->d.height, o->d.depth, + o->d.levels, o->d.format, o->d.target, o->d.flags, o->L.size); + return 1; +} +/* TOTK: live texture by storage address. The game re-initialises transient textures (G-buffer etc.) every frame at the + same pool memory without re-registering the texture-pool descriptor (an NVN descriptor refers to the memory, so it + stays valid on hardware). A registration whose serial has since been finalised is redirected by the replay to the + texture currently living at that address (nvn_tex_live_at). */ +static nvn_hmap g_live_mem = {.klen = 8}; // storage address -> malloc'd tex_t copy of the live texture +static pthread_mutex_t g_live_mu = PTHREAD_MUTEX_INITIALIZER; +static void live_put(const tex_t *o) { + if (!o->mem) return; uint64_t k = (uint64_t)(uintptr_t)o->mem; tex_t *c = malloc(sizeof *c); if (!c) die("live texture copy"); *c = *o; + pthread_mutex_lock(&g_live_mu); tex_t *old = nvn_hm_put(&g_live_mem, &k, c); pthread_mutex_unlock(&g_live_mu); free(old); +} +static void live_del(const tex_t *o); +void live_del_x(const tex_t *o) { live_del(o); } +static void live_del(const tex_t *o) { + if (!o->mem) return; uint64_t k = (uint64_t)(uintptr_t)o->mem; + pthread_mutex_lock(&g_live_mu); tex_t *cur = nvn_hm_get(&g_live_mem, &k); if (cur && cur->serial == o->serial) { nvn_hm_del(&g_live_mem, &k); free(cur); } pthread_mutex_unlock(&g_live_mu); +} +int nvn_tex_live_at(const void *mem, tex_t *out) { + if (!mem) return 0; uint64_t k = (uint64_t)(uintptr_t)mem; + pthread_mutex_lock(&g_live_mu); tex_t *cur = nvn_hm_get(&g_live_mem, &k); if (cur) *out = *cur; pthread_mutex_unlock(&g_live_mu); + return cur != NULL; +} +// TOTK translation thread: the release runs in queue order on a copy (only the serial and storage address are used), so +// a texture finalized right after a submit stays usable by that submit's replay, without waiting for the queue. +NVN(nvnTextureFinalize, void, NVNtexture *t) { + if (totk_life_match(TX(t))) totk_life_log("fin tex=%p ser=%llu mem=%p", (void *)t, (unsigned long long)TX(t)->serial, (void *)TX(t)->mem); + if (async_replay()) { void x_texrel(const tex_t *o); x_texrel(TX(t)); TX(t)->magic = 0; return; } + live_del(TX(t)); mtl_tex_release(TX(t)); TX(t)->magic = 0; +} +NVN(nvnTextureGetStorageClass, NVNstorageClass, const NVNtexture *t){return totk_storage_class(&CTX(t)->d);} +NVN(nvnTextureGetFlags, NVNtextureFlags, const NVNtexture *t) { return (NVNtextureFlags)CTX(t)->d.flags; } +NVN(nvnTextureGetWidth, int, const NVNtexture *t) { return (int)CTX(t)->d.width; } +NVN(nvnTextureGetHeight, int, const NVNtexture *t) { return (int)CTX(t)->d.height; } +NVN(nvnTextureGetDepth, int, const NVNtexture *t) { return (int)CTX(t)->d.depth; } +NVN(nvnTextureGetStorageSize, int, const NVNtexture *t) { return (int)CTX(t)->L.size; } +NVN(nvnTextureCompare, NVNboolean, const NVNtexture *a, const NVNtexture *b) { + const tex_t *x = CTX(a), *y = CTX(b); + return !memcmp(&x->d, &y->d, sizeof x->d) && x->mem == y->mem; +} +// 头文件无原型；main 0x174899c 一类调用点把返回值与 1 比，不等则置某状态位 0x100。含义未核，先给 1 +// No prototype in the header; call sites such as main 0x174899c compare the return value with 1 and set some status bit 0x100 if not equal. Meaning unverified; return 1 for now +NVN_NOPROTO(nvnTextureGetRawStorageClass, int, const NVNtexture *t) { (void)t; return 1; } +#define VW(v) ((view_t *)(v)) +NVN(nvnTextureViewSetDefaults, void, NVNtextureView *v) { memset(v, 0, sizeof(view_t)); } +NVN(nvnTextureViewSetLevels, void, NVNtextureView *v, int base, int n) { if ((unsigned)base > 255 || (unsigned)n > 255) die("texture view levels %d/%d", base, n); VW(v)->set_levels = 1; VW(v)->base_level = base; VW(v)->nlevels = n; } +NVN(nvnTextureViewSetLayers, void, NVNtextureView *v, int min, int n) { if ((unsigned)min > 65535 || (unsigned)n > 65535) die("texture view layers %d/%d", min, n); VW(v)->set_layers = 1; VW(v)->min_layer = min; VW(v)->nlayers = n; } +NVN(nvnTextureViewSetFormat, void, NVNtextureView *v, NVNformat f) { if ((unsigned)f > 65535) die("texture view format %d", (int)f); VW(v)->set_format = 1; VW(v)->format = f; } +NVN(nvnTextureViewSetTarget, void, NVNtextureView *v, NVNtextureTarget t) { if ((unsigned)t > 255) die("texture view target %d", (int)t); VW(v)->set_target = 1; VW(v)->target = t; } +NVN(nvnTextureViewGetLayers, NVNboolean, const NVNtextureView *v, int *min, int *n) { + const view_t *o = (const view_t *)v; *min = o->min_layer; *n = o->nlayers; return o->set_layers; +} +NVN(nvnTextureGetViewOffset, ptrdiff_t, const NVNtexture *t, const NVNtextureView *v) { + const tex_t *o = CTX(t); const view_t *w = (const view_t *)v; + if (!w) return 0; + uint64_t off = w->set_levels ? nvn_tex_level_offset(&o->d, (unsigned)w->base_level) : 0; + if (w->set_layers) off += (uint64_t)w->min_layer * o->L.layer_size; + return (ptrdiff_t)off; +} +// CPU 读贴图（NVN ReadTexels）：预排布贴图以池内存为真值（解排布直读），GPU 写出的贴图从 MTLTexture 读回（nvn_mtl.m） +// CPU texture read (NVN ReadTexels): for packaged textures the pool memory is the ground truth (de-layout and read directly); GPU-written textures are read back from the MTLTexture (nvn_mtl.m) +static void read_texels(const tex_t *o, const view_t *v, const NVNcopyRegion *r, uint8_t *dst, ptrdiff_t row, ptrdiff_t img) { + int bw, bh, bpb = nvn_fmt_bpb(o->d.format, &bw, &bh); + uint32_t wb = (uint32_t)((r->width + bw - 1) / bw) * bpb, hb = (uint32_t)(r->height + bh - 1) / bh; + if (!row) row = wb; + if (!img) img = row * hb; + unsigned level = v && v->set_levels ? (unsigned)v->base_level : 0, layer0 = v && v->set_layers ? (unsigned)v->min_layer : 0; + x_drain(); mtl_read_texels(o, level, layer0, r, dst, (size_t)row, (size_t)img); +} +NVN(nvnTextureReadTexels, void, const NVNtexture *t, const NVNtextureView *v, const NVNcopyRegion *r, void *dst) { + read_texels(CTX(t), (const view_t *)v, r, dst, 0, 0); +} +NVN(nvnTextureReadTexelsStrided, void, const NVNtexture *t, const NVNtextureView *v, const NVNcopyRegion *r, void *dst, ptrdiff_t row, ptrdiff_t img) { + read_texels(CTX(t), (const view_t *)v, r, dst, row, img); +} +static void write_texels(const tex_t *o, const view_t *v, const NVNcopyRegion *r, const uint8_t *src, ptrdiff_t row, ptrdiff_t img) { + int bw, bh, bpb = nvn_fmt_bpb(o->d.format, &bw, &bh); + uint32_t wb = (uint32_t)((r->width + bw - 1) / bw) * bpb, hb = (uint32_t)(r->height + bh - 1) / bh; + if (!row) row = wb; + if (!img) img = row * hb; + if (row < (ptrdiff_t)wb || img < row * (ptrdiff_t)hb) die("WriteTexels: stride smaller than region"); + unsigned level = v && v->set_levels ? (unsigned)v->base_level : 0, layer0 = v && v->set_layers ? (unsigned)v->min_layer : 0; + if (v && v->set_format && v->format != o->d.format) die("WriteTexels: format-reinterpreting view not supported"); + x_drain(); mtl_write_texels(o, level, layer0, r, src, (size_t)row, (size_t)img); +} +NVN(nvnTextureWriteTexels, void, const NVNtexture *t, const NVNtextureView *v, const NVNcopyRegion *r, const void *src) { + write_texels(CTX(t), (const view_t *)v, r, src, 0, 0); +} +NVN(nvnTextureWriteTexelsStrided, void, const NVNtexture *t, const NVNtextureView *v, const NVNcopyRegion *r, const void *src, ptrdiff_t row, ptrdiff_t img) { + write_texels(CTX(t), (const view_t *)v, r, src, row, img); +} +NVN(nvnTextureFlushTexels, void, const NVNtexture *t, const NVNtextureView *v, const NVNcopyRegion *r) { (void)t; (void)v; (void)r; } +NVN(nvnTextureInvalidateTexels, void, const NVNtexture *t, const NVNtextureView *v, const NVNcopyRegion *r) { (void)t; (void)v; (void)r; } + +// ---------- 采样器 ---------- +// ---------- Samplers ---------- +#define SB(b) (&((samp_builder_t *)(b))->p) +NVN(nvnSamplerBuilderSetDefaults, void, NVNsamplerBuilder *b) { + NVNdevice *dev = ((samp_builder_t *)b)->dev; memset(b, 0, sizeof(samp_builder_t)); ((samp_builder_t *)b)->dev = dev; + samp_par *o = SB(b); + o->minf = NVN_MIN_FILTER_NEAREST_MIPMAP_LINEAR; o->magf = NVN_MAG_FILTER_LINEAR; // GL 默认 | GL defaults + o->ws = o->wt = o->wr = NVN_WRAP_MODE_REPEAT; o->cmp_func = NVN_COMPARE_FUNC_LESS; + o->lod_min = -1000.f; o->lod_max = 1000.f; o->aniso = 1.f; +} +NVN(nvnSamplerBuilderSetDevice, void, NVNsamplerBuilder *b, NVNdevice *d) { ((samp_builder_t *)b)->dev = d; } +NVN(nvnSamplerBuilderSetMinMagFilter, void, NVNsamplerBuilder *b, NVNminFilter mi, NVNmagFilter ma) { SB(b)->minf = mi; SB(b)->magf = ma; } +NVN(nvnSamplerBuilderSetWrapMode, void, NVNsamplerBuilder *b, NVNwrapMode s, NVNwrapMode t, NVNwrapMode r) { SB(b)->ws = s; SB(b)->wt = t; SB(b)->wr = r; } +NVN(nvnSamplerBuilderSetLodClamp, void, NVNsamplerBuilder *b, float lo, float hi) { SB(b)->lod_min = lo; SB(b)->lod_max = hi; } +NVN(nvnSamplerBuilderSetLodBias, void, NVNsamplerBuilder *b, float v) { SB(b)->lod_bias = v; } +NVN(nvnSamplerBuilderSetCompare, void, NVNsamplerBuilder *b, NVNcompareMode m, NVNcompareFunc f) { SB(b)->cmp_mode = m; SB(b)->cmp_func = f; } +NVN(nvnSamplerBuilderSetBorderColor, void, NVNsamplerBuilder *b, const float *c) { memcpy(SB(b)->border, c, 16); } +NVN(nvnSamplerBuilderSetMaxAnisotropy, void, NVNsamplerBuilder *b, float v) { SB(b)->aniso = v; } +NVN(nvnSamplerBuilderSetLodSnap, void, NVNsamplerBuilder *b, float v) { SB(b)->lod_snap = v; } // Metal has no LOD snapping; stored for queries // Metal 无对应，忽略 | No Metal equivalent, ignored +NVN(nvnSamplerBuilderGetMinMagFilter, void, const NVNsamplerBuilder *b, NVNminFilter *mi, NVNmagFilter *ma) { + const samp_par *o = &((const samp_builder_t *)b)->p; *mi = (NVNminFilter)o->minf; *ma = (NVNmagFilter)o->magf; +} +NVN(nvnSamplerBuilderGetWrapMode, void, const NVNsamplerBuilder *b, NVNwrapMode *s, NVNwrapMode *t, NVNwrapMode *r) { + const samp_par *o = &((const samp_builder_t *)b)->p; *s = (NVNwrapMode)o->ws; *t = (NVNwrapMode)o->wt; *r = (NVNwrapMode)o->wr; +} +NVN(nvnSamplerBuilderSetReductionFilter, void, NVNsamplerBuilder *b, NVNsamplerReduction r) { if ((unsigned)r > 2) die("invalid sampler reduction %d", (int)r); SB(b)->reduction = (uint8_t)r; } +NVN(nvnSamplerBuilderGetReductionFilter, NVNsamplerReduction, const NVNsamplerBuilder *b) { return (NVNsamplerReduction)((const samp_builder_t *)b)->p.reduction; } +NVN(nvnSamplerGetReductionFilter, NVNsamplerReduction, const NVNsampler *s) { return (NVNsamplerReduction)((const samp_t *)s)->p.reduction; } +NVN(nvnSamplerInitialize, NVNboolean, NVNsampler *s, const NVNsamplerBuilder *b) { + samp_t *o = (samp_t *)s; o->magic = MAGIC('S', 'M', 'P', 0); o->p = ((const samp_builder_t *)b)->p; + { static int samplog = -1; if (samplog < 0) samplog = getenv("DM_NVN_SAMPLOG") != NULL; + if (samplog) { const samp_par *q = &o->p; + fprintf(stderr, "[nvn-samp] s=%p min=%u mag=%u wrap=%u/%u/%u cmp=%u/%u red=%u lod=%.6g..%.6g bias=%.6g aniso=%.6g border=%.4g,%.4g,%.4g,%.4g snap=%.6g\n", + (void *)s, q->minf, q->magf, q->ws, q->wt, q->wr, q->cmp_mode, q->cmp_func, q->reduction, q->lod_min, q->lod_max, q->lod_bias, q->aniso, + q->border[0], q->border[1], q->border[2], q->border[3], q->lod_snap); } } + return 1; +} +NVN(nvnSamplerFinalize, void, NVNsampler *s) { ((samp_t *)s)->magic = 0; } + +// ---------- 程序 ---------- +// ---------- Programs ---------- +NVN(nvnProgramInitialize, NVNboolean, NVNprogram *p, NVNdevice *d) { + (void)d; memset(p, 0, sizeof(prog_t)); ((prog_t *)p)->magic = MAGIC('P', 'R', 'O', 'G'); return 1; +} +NVN(nvnProgramFinalize, void, NVNprogram *p) { x_drain(); mtl_prog_release((prog_t *)p); ((prog_t *)p)->magic = 0; } +NVN(nvnProgramSetShaders, NVNboolean, NVNprogram *p, int n, const NVNshaderData *sd) { + NVN_BLOCKING(); + prog_t *o = (prog_t *)p; + if (n < 0 || n > 6) { LOG("【红】ProgramSetShaders 阶段数 %d", n); return 0; } + mtl_prog_release(o); o->nstage = n; memcpy(o->sd, sd, n * sizeof *sd); o->serial = nvn_new_serial(); + OLOG("ProgramSetShaders %p n=%d data0=%#llx", (void *)p, n, n ? (unsigned long long)sd[0].data : 0ull); + return 1; +} + +// TOTK: NVNshaderDataExt = {data, control, reserved}, 0x18 bytes (runtime dump of main's call; reserved observed 0). +typedef struct { NVNbufferAddress data; const void *control; uint64_t reserved; } totk_shader_data_ext; +_Static_assert(sizeof(totk_shader_data_ext) == 0x18, "NVNshaderDataExt stride"); +NVN_NOPROTO(nvnProgramSetShadersExt, NVNboolean, NVNprogram *p, int n, const totk_shader_data_ext *sd) { + if (n < 0 || n > 6) { LOG("【红】ProgramSetShadersExt 阶段数 %d", n); return 0; } + NVNshaderData plain[6]; + for (int i = 0; i < n; i++) { + if (sd[i].reserved) die("ProgramSetShadersExt: unknown non-zero field %#llx in stage %d", (unsigned long long)sd[i].reserved, i); + plain[i] = (NVNshaderData){sd[i].data, sd[i].control}; + } + return h_nvnProgramSetShaders(p, n, plain); +} + +// ---------- 同步 / 事件 ---------- +// ---------- Sync / events ---------- +NVN(nvnSyncInitialize, NVNboolean, NVNsync *s, NVNdevice *d) { (void)d; *(sync_t *)s = (sync_t){MAGIC('S', 'Y', 'N', 'C'), 0, 0}; return 1; } +NVN(nvnSyncFinalize, void, NVNsync *s) { ((sync_t *)s)->magic = 0; } +// 围栏 = 「此前提交到队列的命令全部完成」：记下最近一次提交的栅栏值 +// Fence = "all commands previously submitted to the queue have completed": record the fence value of the most recent submit +// TOTK: with the translation thread a fence is a submit count (bit 2 of fenced), see fence_now / fence_wait. +/* TOTK_SYNC_CHAIN=[:] (diagnostic): guest call chains of the first n fence placements and of the first n waits that block, + to find the game code that makes frame N+1 wait for frame N on the GPU (see the 2026-10-15 timeline). */ +static void guest_chain(const char *what); +static uint64_t g_frame; // tentative; defined with the frame bookkeeping below +static uint64_t g_game_sys; +extern _Atomic int runtime_survey_pending; +extern void runtime_survey_tick(uint64_t frame); +extern const int32_t runtime_survey_sync[2]; +/* TOTK_SURVEY_DEBUG=1: while a command is pending, the first 6 waited syncs, their thread and guest call chain, and where the + system object holds them (pointer at sys+o, or inside / pointed to from the object at *(sys+o)). */ +static int vm_read8(uint64_t a, uint64_t *out); +static void survey_wait_debug(const NVNsync *s) { + static int on = -1; static _Atomic int left = 6; + if (on < 0) { const char *e = getenv("TOTK_SURVEY_DEBUG"); on = e && !strcmp(e, "1"); } + if (!on || atomic_fetch_sub(&left, 1) <= 0) return; + uint64_t sv = (uintptr_t)s, tid = 0; pthread_threadid_np(NULL, &tid); + LOG("[survey] wait sync=%p frame=%llu thread=%llu", (const void *)s, (unsigned long long)g_frame, (unsigned long long)tid); + for (uint64_t o = 0; g_game_sys && o < 0x800; o += 8) { + uint64_t q; + if (!vm_read8(g_game_sys + o, &q) || !q) continue; + if (q == sv) LOG("[survey] = *(sys+%#llx)", (unsigned long long)o); + if (sv >= q && sv - q < 0x800) LOG("[survey] = *(sys+%#llx)+%#llx", (unsigned long long)o, (unsigned long long)(sv - q)); + for (uint64_t o2 = 0; o2 < 0x800; o2 += 8) { uint64_t r; if (vm_read8(q + o2, &r) && r == sv) LOG("[survey] = *(*(sys+%#llx)+%#llx)", (unsigned long long)o, (unsigned long long)o2); } + } + guest_chain("survey wait"); +} +static void survey_after_wait(const NVNsync *s) { + if (__builtin_expect(atomic_load_explicit(&runtime_survey_pending, memory_order_relaxed), 0)) { + survey_wait_debug(s); + uintptr_t at = g_game_sys ? *(const uintptr_t *)(uintptr_t)(g_game_sys + runtime_survey_sync[0]) : 0; + if (at && runtime_survey_sync[1] >= 0) at = *(const uintptr_t *)(at + runtime_survey_sync[1]); + if (at && (uintptr_t)s == at) runtime_survey_tick(g_frame); + } +} +/* TOTK_SYNC_CHAIN=[:] */ +static int sync_chain_left(int k) { + static int lim = -2; static uint64_t from; static _Atomic int used[2]; + if (lim == -2) { const char *e = getenv("TOTK_SYNC_CHAIN"); lim = e ? atoi(e) : 0; from = e && strchr(e, ':') ? strtoull(strchr(e, ':') + 1, NULL, 10) : 0; } + return lim > 0 && g_frame >= from && atomic_fetch_add(&used[k], 1) < lim; +} +NVN(nvnQueueFenceSync, void, NVNqueue *q, NVNsync *s, NVNsyncCondition c, int f) { (void)q; (void)c; (void)f; int tok; uint64_t v = fence_now(&tok); ((sync_t *)s)->fenced = tok ? 5 : 1; ((sync_t *)s)->fence = v; + if (sync_chain_left(0)) { char w[96]; snprintf(w, sizeof w, "FenceSync sync=%p fence=%llu cond=%d flags=%d", (void *)s, (unsigned long long)v, (int)c, f); guest_chain(w); } } +// 没围栏过的同步对象按真机行为等到超时；已围栏的只在 GPU 真正完成到该栅栏值后放行 +// A sync object that was never fenced waits until timeout, as on a real device; a fenced one is released only after the GPU has actually completed up to that fence value +// TOTK_SYNC_LOG=1: CPU waits by kind (0 fence sync, 1 window-acquire sync): calls, calls that had to block, blocked time, +// and calls whose fence was the newest committed one (a full drain of the queue). +_Atomic uint64_t g_sw_calls[2], g_sw_blocked[2], g_sw_ns[2], g_sw_newest[2]; +static int synclog_env(void) { static int on = -1; if (on < 0) on = getenv("TOTK_SYNC_LOG") != NULL; return on; } +NVN(nvnSyncWait, NVNsyncWaitResult, const NVNsync *s, uint64_t timeout) { + NVN_BLOCKING(); + const sync_t *o = (const sync_t *)s; + if (!o->fenced) return NVN_SYNC_WAIT_RESULT_TIMEOUT_EXPIRED; + int k = (o->fenced & 2) != 0, lg = synclog_env(); + if (lg) { atomic_fetch_add(&g_sw_calls[k], 1); if (o->fence == mtl_last_fence()) atomic_fetch_add(&g_sw_newest[k], 1); } + int tok = (o->fenced & 4) != 0; + if (tok ? fence_wait(o->fence, 0) : mtl_wait(o->fence, 0)) { survey_after_wait(s); return NVN_SYNC_WAIT_RESULT_ALREADY_SIGNALED; } + uint64_t t0 = lg ? now_ns() : 0; + totk_tl(TL_WAIT_IN, o->fence, o->fence == mtl_last_fence()); + if (sync_chain_left(1)) { char w[96]; snprintf(w, sizeof w, "SyncWait sync=%p fence=%llu newest=%d", (const void *)s, (unsigned long long)o->fence, o->fence == mtl_last_fence()); guest_chain(w); } + int ok = tok ? fence_wait(o->fence, timeout) : mtl_wait(o->fence, timeout); + totk_tl(TL_WAIT_OUT, o->fence, ok); + if (lg) { atomic_fetch_add(&g_sw_blocked[k], 1); atomic_fetch_add(&g_sw_ns[k], now_ns() - t0); } + if (ok) survey_after_wait(s); + return ok ? NVN_SYNC_WAIT_RESULT_CONDITION_SATISFIED : NVN_SYNC_WAIT_RESULT_TIMEOUT_EXPIRED; +} +NVN(nvnEventBuilderSetDefaults, void, NVNeventBuilder *b) { memset(b, 0, sizeof(ev_builder_t)); } +NVN(nvnEventBuilderSetStorage, void, NVNeventBuilder *b, const NVNmemoryPool *p, int64_t off) { *(ev_builder_t *)b = (ev_builder_t){(const pool_t *)p, off}; } +NVN(nvnEventInitialize, NVNboolean, NVNevent *e, const NVNeventBuilder *b) { + const ev_builder_t *s = (const ev_builder_t *)b; + if (!s->pool || !s->pool->mem) { LOG("【红】EventInitialize 没有存储"); return 0; } + *(ev_t *)e = (ev_t){MAGIC('E', 'V', 'N', 'T'), (volatile uint32_t *)(s->pool->mem + s->off)}; return 1; +} +NVN(nvnEventFinalize, void, NVNevent *e) { ((ev_t *)e)->magic = 0; } +NVN(nvnEventGetValue, uint32_t, const NVNevent *e) { return *((const ev_t *)e)->val; } +static void ev_signal(const ev_t *e, int mode, uint32_t v) { if (mode == NVN_EVENT_SIGNAL_MODE_ADD) *e->val += v; else *e->val = v; } +NVN(nvnEventSignal, void, NVNevent *e, NVNeventSignalMode m, uint32_t v) { ev_signal((const ev_t *)e, m, v); } + +// ---------- 队列 ---------- +// ---------- Queue ---------- +#define QB(b) ((queue_builder_t *)(b)) +NVN(nvnQueueBuilderSetDefaults, void, NVNqueueBuilder *b) { + NVNdevice *dev = QB(b)->dev; memset(b, 0, sizeof(queue_builder_t)); QB(b)->dev = dev; + QB(b)->cmd_sz = 65536; QB(b)->comp_sz = 262144; QB(b)->ctl_sz = 16384; QB(b)->flush = 4096; // 设备默认值 | Device defaults +} +NVN(nvnQueueBuilderSetDevice, void, NVNqueueBuilder *b, NVNdevice *d) { QB(b)->dev = d; } +NVN(nvnQueueBuilderSetFlags, void, NVNqueueBuilder *b, int f) { QB(b)->flags = f; } +NVN(nvnQueueBuilderSetCommandMemorySize, void, NVNqueueBuilder *b, size_t n) { QB(b)->cmd_sz = n; } +NVN(nvnQueueBuilderSetComputeMemorySize, void, NVNqueueBuilder *b, size_t n) { QB(b)->comp_sz = n; } +NVN(nvnQueueBuilderSetControlMemorySize, void, NVNqueueBuilder *b, size_t n) { QB(b)->ctl_sz = n; } +NVN(nvnQueueBuilderSetCommandFlushThreshold, void, NVNqueueBuilder *b, size_t n) { QB(b)->flush = n; } +// 队列内存：命令 + 计算 + 控制三块之和按 4 KiB 取整（宿主不使用，只需游戏按此大小分配） +// Queue memory: the sum of the command + compute + control blocks rounded to 4 KiB (the host does not use it; the game only needs to allocate this size) +NVN(nvnQueueBuilderGetQueueMemorySize, size_t, const NVNqueueBuilder *b) { + const queue_builder_t *o = (const queue_builder_t *)b; + return (o->cmd_sz + o->comp_sz + o->ctl_sz + 4095) & ~(size_t)4095; +} +NVN(nvnQueueBuilderSetQueueMemory, void, NVNqueueBuilder *b, void *m, size_t n) { QB(b)->mem = m; QB(b)->mem_sz = n; } +NVN(nvnQueueInitialize, NVNboolean, NVNqueue *q, const NVNqueueBuilder *b) { + *(queue_t *)q = (queue_t){MAGIC('Q', 'U', 'E', 'U'), ((const queue_builder_t *)b)->dev, 0}; + OLOG("QueueInitialize %p", (void *)q); return 1; +} +NVN(nvnQueueFinalize, void, NVNqueue *q) { ((queue_t *)q)->magic = 0; } +NVN(nvnQueueGetError, NVNqueueGetErrorResult, NVNqueue *q, NVNqueueErrorInfo *info) { + (void)q; if (info) memset(info, 0, sizeof *info); return NVN_QUEUE_GET_ERROR_RESULT_GPU_NO_ERROR; +} +NVN(nvnQueueFlush, void, NVNqueue *q) { (void)q; } +NVN(nvnQueueFinish, void, NVNqueue *q) { NVN_BLOCKING(); (void)q; totk_tl(TL_FINISH_IN, 0, 0); x_drain(); mtl_finish(); totk_tl(TL_FINISH_OUT, 0, 0); } + +// ---------- 命令缓冲：录进游戏的命令内存，提交时执行 CPU 可见副作用 ---------- +// ---------- Command buffer: record into the game's command memory, perform CPU-visible side effects at submit ---------- +#define CB(c) ((cmd_t *)(c)) +#define OP_RESERVE 24 // 段尾永远留够一条 JUMP / END | Always keep enough room at the segment end for one JUMP / END + +NVN(nvnCommandBufferInitialize, NVNboolean, NVNcommandBuffer *c, NVNdevice *d) { + memset(c, 0, sizeof(cmd_t)); CB(c)->magic = MAGIC('C', 'M', 'D', 'B'); CB(c)->dev = d; CB(c)->inline_cur = ~0ull; CB(c)->inline_end = 0; return 1; +} +NVN(nvnCommandBufferFinalize, void, NVNcommandBuffer *c) { free(CB(c)->seg); CB(c)->seg = CB(c)->cur = CB(c)->end = NULL; CB(c)->magic = 0; } +NVN(nvnCommandBufferSetMemoryCallback, void, NVNcommandBuffer *c, PFNNVNCOMMANDBUFFERMEMORYCALLBACKPROC cb) { CB(c)->cb = cb; } +NVN(nvnCommandBufferSetMemoryCallbackData, void, NVNcommandBuffer *c, void *d) { CB(c)->cb_data = d; } +static uint8_t *align8(uint8_t *p) { return (uint8_t *)(((uintptr_t)p + 7) & ~(uintptr_t)7); } +// 诊断 DM_NVN_CMDSTAT=1：每 300 帧记一行命令录制的用量——每帧录了多少字节 / 多少条、游戏每次给多大一段命令内存、每帧补内存回调几次、宿主补段几次、 +// 各类命令的条数与字节（找「我们的记录比真机命令大多少」）。只计数，不改录制 / 回放 +// Diagnostic DM_NVN_CMDSTAT=1: every 300 frames log one line of command recording usage -- how many bytes / records were recorded per frame, how large a command memory segment the game gives each time, how many out-of-memory callbacks per frame, how many host-supplied segments, +// and the count and bytes of each command type (to find "how much larger our records are than real-device commands"). Counts only; does not change recording / replay +static struct { int on; uint64_t bytes, recs, grants, grant_bytes, grant_min, grant_max, cbs, hostseg, op_n[64], op_b[64], frame0, waste; } g_cs = { .on = -1 }; +static pthread_mutex_t g_cs_mu = PTHREAD_MUTEX_INITIALIZER; +uint64_t dm_nvn_frame_count(void); +static int cs_on(void) { if (g_cs.on < 0) g_cs.on = getenv("DM_NVN_CMDSTAT") != NULL; return g_cs.on; } +static void cs_op(uint32_t op, uint32_t size) { if (!cs_on() || op >= 64) return; pthread_mutex_lock(&g_cs_mu); g_cs.op_n[op]++; g_cs.op_b[op] += size; pthread_mutex_unlock(&g_cs_mu); } +static void cs_dump(void) { // 每帧调一次（出图处） | Called once per frame (at present) + if (!cs_on()) return; uint64_t f = dm_nvn_frame_count(); if (f - g_cs.frame0 < 300) return; + pthread_mutex_lock(&g_cs_mu); double n = (double)(f - g_cs.frame0); char top[700]; int k = 0; uint8_t used[64] = {0}; + for (int r = 0; r < 8; r++) { int bi = -1; for (int i = 0; i < 64; i++) if (!used[i] && g_cs.op_b[i] && (bi < 0 || g_cs.op_b[i] > g_cs.op_b[bi])) bi = i; if (bi < 0) break; used[bi] = 1; + k += snprintf(top + k, sizeof top - k, " 命令%d：%.0f 条×%.0f 字节=%.1f KB；", bi, g_cs.op_n[bi] / n, (double)g_cs.op_b[bi] / (double)g_cs.op_n[bi], g_cs.op_b[bi] / n / 1024.0); } + LOG("[nvn] 命令内存 至帧 %llu：每帧录 %.1f KB / %.0f 条；游戏给段 每帧 %.2f 次，每段 均 %.1f KB（最小 %.1f / 最大 %.1f）；补内存回调 每帧 %.2f 次；宿主补段 每帧 %.3f 次；换段丢弃的段尾 每帧 %.1f KB｜每帧%s", + (unsigned long long)f, g_cs.bytes / n / 1024.0, g_cs.recs / n, g_cs.grants / n, g_cs.grants ? g_cs.grant_bytes / (double)g_cs.grants / 1024.0 : 0, g_cs.grant_min / 1024.0, g_cs.grant_max / 1024.0, g_cs.cbs / n, g_cs.hostseg / n, g_cs.waste / n / 1024.0, top); + int on = g_cs.on; memset(&g_cs, 0, sizeof g_cs); g_cs.on = on; g_cs.frame0 = f; pthread_mutex_unlock(&g_cs_mu); +} +static __thread uint32_t t_rec_hist[OP_COUNT_]; static __thread uint32_t t_rec_n; +static void put_hdr(uint8_t *p, uint32_t op, uint32_t size) { op_hdr h = {op, size}; memcpy(p, &h, sizeof h); if (op) { cs_op(op, size); if (op < OP_COUNT_) t_rec_hist[op]++; } } +// TOTK: records are written into a growable host buffer per command buffer, never into game command memory. +// Games hand the same command memory to several command buffers recording concurrently (budgeted for real NVN +// encodings); our records are larger and would overwrite each other there. The game's memory is accounted only. +static void cmd_reserve(cmd_t *o, size_t need) { + if (o->seg && o->cur + need + OP_RESERVE <= o->end) return; + size_t used = o->seg ? (size_t)(o->cur - o->seg) : 0; + size_t cap = o->seg ? (size_t)(o->end - o->seg) : 0; if (!cap) cap = 64 << 10; + while (used + need + OP_RESERVE > cap) cap *= 2; + uint8_t *b = realloc(o->seg, cap); if (!b) die("command recording buffer allocation (%zu bytes)", cap); + o->seg = b; o->cur = b + used; o->end = b + cap; +} + +// TOTK diagnostics: guest return chain of the current NVN call (frames link through runtime_hle_switch). +void runtime_describe_address(uint64_t a, char *out, size_t n); +static void guest_chain(const char *what) { + char line[1024]; size_t o = 0; uint64_t *fp = (uint64_t *)__builtin_frame_address(0); + for (int i = 0; i < 14 && fp && !((uintptr_t)fp & 7); i++) { char w[96]; runtime_describe_address(fp[1], w, sizeof w); if (o < sizeof line - 100) o += snprintf(line + o, sizeof line - o, " <- %s", w); if ((uint64_t *)fp[0] == fp) break; fp = (uint64_t *)fp[0]; } + LOG("[chain] %s%s", what, line); +} +/* Offset in main of the first guest return address on the frame chain (0 = none): which game code issued the call. */ +static uint64_t guest_main_caller(void) { + extern int runtime_module_offset(uint64_t pc, const char **name, uint64_t *off); + uint64_t *fp = (uint64_t *)__builtin_frame_address(0); + for (int i = 0; i < 16 && fp && !((uintptr_t)fp & 7); i++) { + const char *nm; uint64_t off; + if (runtime_module_offset(fp[1], &nm, &off) && !strcmp(nm, "main")) return off; + if ((uint64_t *)fp[0] == fp) break; fp = (uint64_t *)fp[0]; + } + return 0; +} +static void release_copies(uintptr_t lo, uintptr_t hi); +NVN(nvnCommandBufferAddCommandMemory, void, NVNcommandBuffer *c, const NVNmemoryPool *p, ptrdiff_t off, size_t n) { + if (cmdlog_env()) LOG("[cmd] AddCommandMemory cb=%p size=%#zx (accounting only)", (void *)c, n); + /* Command memory can be shared by independent recordings with distinct control memory. */ + uintptr_t base = (uintptr_t)(p ? ((const pool_t *)p)->mem : NULL) + (uintptr_t)off; + CB(c)->cmd_base = base; CB(c)->game_size = n; CB(c)->cmd_used = 0; CB(c)->hw_used = 0; +} +// The SDK encodes handles in control memory (sdk+0x444530). Reusing that storage retires its copies. +NVN(nvnCommandBufferAddControlMemory, void, NVNcommandBuffer *c, void *m, size_t n) { + if (cmdlog_env()) LOG("[cmd] AddControlMemory cb=%p mem=%p size=%#zx", (void *)c, m, n); + release_copies((uintptr_t)m, (uintptr_t)m + n); + CB(c)->ctl = m; CB(c)->ctl_size = n; CB(c)->ctl_used = 0; +} +NVN(nvnCommandBufferGetControlMemorySize, size_t, const NVNcommandBuffer *c) { return ((const cmd_t *)c)->ctl_size; } +/* Usage is reported (records live in host memory, but the game decides from it whether a recording is empty); + it is bounded by the memory the game provided, so the game never sees more used than it gave. */ +NVN(nvnCommandBufferGetControlMemoryUsed, size_t, const NVNcommandBuffer *c) { const cmd_t *o = (const cmd_t *)c; return o->ctl_used < o->ctl_size ? o->ctl_used : o->ctl_size; } +NVN(nvnCommandBufferGetControlMemoryFree, size_t, const NVNcommandBuffer *c) { const cmd_t *o = (const cmd_t *)c; return o->ctl_size - (o->ctl_used < o->ctl_size ? o->ctl_used : o->ctl_size); } +NVN(nvnCommandBufferGetCommandMemorySize, size_t, const NVNcommandBuffer *c) { return ((const cmd_t *)c)->game_size; } +NVN(nvnCommandBufferGetCommandMemoryUsed, size_t, const NVNcommandBuffer *c) { const cmd_t *o = (const cmd_t *)c; return o->hw_used < o->game_size ? o->hw_used : o->game_size; } +NVN(nvnCommandBufferGetCommandMemoryFree, size_t, const NVNcommandBuffer *c) { const cmd_t *o = (const cmd_t *)c; return o->game_size - (o->hw_used < o->game_size ? o->hw_used : o->game_size); } +NVN(nvnCommandBufferGetMemoryCallback, PFNNVNCOMMANDBUFFERMEMORYCALLBACKPROC, const NVNcommandBuffer *c) { return ((const cmd_t *)c)->cb; } +NVN_NOPROTO(nvnCommandBufferGetMemoryCallbackData, void *, const NVNcommandBuffer *c) { return ((const cmd_t *)c)->cb_data; } +// 取一条记录的空间；不够则按 NVN 约定回调游戏补命令内存 +// Get space for one record; if not enough, call back into the game per the NVN convention to add command memory +static uint8_t *cmd_alloc(cmd_t *o, uint32_t size) { + size = (size + 7) & ~7u; + cmd_reserve(o, size); + if (cs_on()) { pthread_mutex_lock(&g_cs_mu); g_cs.bytes += size; g_cs.recs++; pthread_mutex_unlock(&g_cs_mu); } + uint8_t *p = o->cur; o->cur += size; o->cmd_used += size; o->hw_used += size; o->last_rec = size; put_hdr(p, 0, size); return p; +} +/* TOTK: the game sums nvnCommandBufferGetCommandMemoryUsed over recordings to size resource heaps (RSTB), so usage must be what + the real NVN pushbuffer encoding takes, not our record size. Known encodings (Maxwell methods, from a real recording): + DrawElements 12 words (index format, index address, 6-word draw macro, 1-word macro), DrawElementsBaseVertex 11 words, + BindVertexAttribState 1+n words, BindVertexStreamState 16+n words, BindVertexBuffer 6 words each (address + limit). */ +static void hw_fix(cmd_t *o, size_t bytes) { o->hw_used += bytes; o->hw_used -= o->last_rec; o->last_rec = bytes; } +static int reclog_limit(void) { static int v = -1; if (v < 0) { const char *e = getenv("DM_NVN_RECLOG"); v = e ? atoi(e) : 0; } return v; } +NVN(nvnCommandBufferBeginRecording, void, NVNcommandBuffer *c) { + cmd_t *o = CB(c); + if (o->recording) LOG("【红】BeginRecording 重入"); + o->recording = 1; o->cur = o->seg; /* the previous recording was copied at EndRecording */ + o->rec_start = o->cmd_used; + uint8_t *p = cmd_alloc(o, 8); put_hdr(p, OP_BEGIN, 8); o->hw_used -= 8; /* BEGIN/END records have no hardware equivalent */ + o->hw_start = o->hw_used; memset(t_rec_hist, 0, sizeof t_rec_hist); +} +// TOTK: an NVN command handle lives in the command buffer's control memory; games reuse command memory +// for later recordings while still calling earlier handles (observed: temporary command buffers sharing one +// 4 MiB command region, each with its own control memory). The finished recording is therefore copied into +// host memory (following segment jumps), and the copy lives until the same control memory is added again. +typedef struct { uintptr_t ctl_lo, ctl_hi; uint8_t *copy; uint64_t stale_at; uint32_t token; } cmd_copy; // stale_at: submit count when its memory was re-added (0 = live); token: its mark is in the control memory (copy_token_write) +static uint64_t g_submits; +/* Recently freed handles, to report what a later use of a freed handle looked like. */ +#define FREED_RING 65536 +static struct { uintptr_t h; uint64_t freed_at, stale_at, used_at, made_at; uintptr_t lo; } g_freed[FREED_RING]; static unsigned g_nfreed; +static size_t g_ncopies; +/* TOTK: copies indexed by the 4 KiB page of their control-memory position (a copy covers one byte, ctl_lo). Releasing a + re-added control region looks at its pages only: it used to scan all ~30,000 live copies under one lock on every + AddControlMemory (about 65 per frame, from three recording threads), ~19 ms of game-thread time per frame on iPhone 14. */ +#define COPY_PAGE_SHIFT 12 +typedef struct { uintptr_t page; cmd_copy *v; uint32_t n, cap; uint16_t used, empty; } copy_bucket; // empty: all its copies are gone (cbk_compact drops it) +static copy_bucket *g_cbk; static size_t g_cbk_cap, g_cbk_used, g_cbk_empty; // open addressing, power-of-two capacity; emptied buckets go at a sweep window start +static uint64_t g_cbk_compactions; +static size_t cbk_slot(uintptr_t page) { uint64_t h = (uint64_t)page * 0x9E3779B97F4A7C15ull; return (size_t)(h >> 20) & (g_cbk_cap - 1); } +static copy_bucket *cbk_get(uintptr_t page, int create) { // g_copy_mu held + if (create && (g_cbk_used + 1) * 2 > g_cbk_cap) { + size_t oc = g_cbk_cap; copy_bucket *ob = g_cbk; g_cbk_cap = oc ? oc * 2 : 1024; g_cbk = calloc(g_cbk_cap, sizeof *g_cbk); if (!g_cbk) die("command copy index"); + for (size_t i = 0; i < oc; i++) if (ob[i].used) { size_t k = cbk_slot(ob[i].page); while (g_cbk[k].used) k = (k + 1) & (g_cbk_cap - 1); g_cbk[k] = ob[i]; } + free(ob); + } + if (!g_cbk_cap) return NULL; + for (size_t k = cbk_slot(page);; k = (k + 1) & (g_cbk_cap - 1)) { + if (g_cbk[k].used && g_cbk[k].page == page) return &g_cbk[k]; + if (!g_cbk[k].used) { if (!create) return NULL; g_cbk[k] = (copy_bucket){page, NULL, 0, 0, 1, 0}; g_cbk_used++; return &g_cbk[k]; } + } +} +static pthread_mutex_t g_copy_mu = PTHREAD_MUTEX_INITIALIZER; +/* The opaque handle points past this host-only validation and accounting prefix. */ +#define COPY_MAGIC 0x59504f43444d43ull /* "CMDCOPY" */ +typedef struct { uint64_t magic; _Atomic uint64_t used_at; uint64_t made_at; uint32_t body_bytes, last_size; } copy_prefix; +static _Atomic uint64_t g_copy_bytes; /* live recording-copy bytes (logged with the copy count) */ +static copy_prefix *copy_of(NVNcommandHandle h) { copy_prefix *c = (copy_prefix *)(uintptr_t)h - 1; return c; } +static uint8_t *flatten_recording(const uint8_t *p) { + size_t n = sizeof(copy_prefix), cap = 4096; uint8_t *out = malloc(cap); + int bulk_ok = 1; uint32_t last_size = 0; + if (!out) die("command copy allocation"); + for (;;) { + op_hdr h; memcpy(&h, p, sizeof h); + if (h.size < 8 || (h.size & 7)) die("corrupt recording at %p (op %u size %u)", (const void *)p, h.op, h.size); + if (h.op == OP_JUMP) { memcpy(&p, p + 8, 8); continue; } + if (h.op >= OP_COUNT_ || h.size > (64u << 20)) bulk_ok = 0; + if (n == sizeof(copy_prefix)) { if (h.op != OP_BEGIN || h.size != 8) bulk_ok = 0; } + else if (h.op == OP_BEGIN) bulk_ok = 0; + else if (h.op != OP_END) last_size = h.size; + if (n + h.size > cap) { while (n + h.size > cap) cap *= 2; out = realloc(out, cap); if (!out) die("command copy allocation"); } + memcpy(out + n, p, h.size); n += h.size; + if (h.op == OP_END) { + uint8_t *trimmed = realloc(out, n); if (trimmed) out = trimmed; + copy_prefix *c = (copy_prefix *)out; c->magic = COPY_MAGIC; c->used_at = 0; c->made_at = g_submits; + size_t body = n >= sizeof(copy_prefix) + 16 ? n - sizeof(copy_prefix) - 16 : 0; + c->body_bytes = bulk_ok && h.size == 8 && body <= UINT32_MAX ? (uint32_t)body : 0; + c->last_size = c->body_bytes ? last_size : 0; + g_copy_bytes += malloc_size(out); return out + sizeof(copy_prefix); + } + p += h.size; + } +} +/* Track handle use for diagnostics; lifetime depends on control-memory reuse, never elapsed frames. */ +static void copy_touch(NVNcommandHandle h, uint64_t now) { + copy_prefix *c = copy_of(h); + if (c->magic != COPY_MAGIC) { + for (unsigned i = 0; i < FREED_RING && i < g_nfreed; i++) if (g_freed[i].h == (uintptr_t)h) + LOG("【红】handle %#llx made at submit %llu was freed at submit %llu (memory start %#lx re-added at submit %llu, last use at submit %llu); now submit %llu", + (unsigned long long)h, (unsigned long long)g_freed[i].made_at, (unsigned long long)g_freed[i].freed_at, (unsigned long)g_freed[i].lo, + (unsigned long long)g_freed[i].stale_at, (unsigned long long)g_freed[i].used_at, (unsigned long long)now); + die("command handle %#llx is not a live recording (freed or never recorded)", (unsigned long long)h); + } + c->used_at = now; +} + +static uint64_t g_copy_drop_readd, g_copy_drop_token, g_copy_mark_faults; /* why copies were released: their control memory was added again / their mark there is gone (of those: unmapped) */ +static void copy_drop(copy_bucket *b, uint32_t i, int by_token) { /* g_copy_mu held: release copy i of bucket b */ + cmd_copy *c = &b->v[i]; + c->stale_at = g_submits + 1; + copy_prefix *pf = (copy_prefix *)(c->copy - sizeof(copy_prefix)); + if (cmdlog_env()) LOG("[cmd] free copy=%p%s", (void *)c->copy, by_token ? " (its mark in the control memory is gone)" : ""); + g_freed[g_nfreed++ % FREED_RING] = (typeof(g_freed[0])){(uintptr_t)c->copy, g_submits, c->stale_at, pf->used_at, pf->made_at, c->ctl_lo}; + pf->magic = 0; g_copy_bytes -= malloc_size(pf); free(pf); b->v[i] = b->v[--b->n]; g_ncopies--; + if (by_token) g_copy_drop_token++; else g_copy_drop_readd++; + if (!b->n && !b->empty) { b->empty = 1; g_cbk_empty++; } +} +static void release_copies(uintptr_t lo, uintptr_t hi) { + if (hi <= lo) return; + pthread_mutex_lock(&g_copy_mu); + for (uintptr_t pg = lo >> COPY_PAGE_SHIFT; pg <= (hi - 1) >> COPY_PAGE_SHIFT; pg++) { + copy_bucket *b = cbk_get(pg, 0); if (!b) continue; + for (uint32_t i = 0; i < b->n;) { + cmd_copy *c = &b->v[i]; + if (!(c->ctl_lo < hi && lo < c->ctl_hi)) { i++; continue; } + copy_drop(b, i, 0); + } + } + pthread_mutex_unlock(&g_copy_mu); +} +/* TOTK (2026-10-17): a recording copy also goes when the game reuses its control memory for something else. The original + driver's handle is the address of a descriptor it writes into the recorder's control memory (sdk+0x444530), so a game + that overwrites or unmaps that memory has no valid handle left. EndRecording writes a mark (this magic and the copy's + address) into the 64 bytes of control memory it reports as used; the sweep's walk (pin_scan) releases a copy whose mark + is gone. Without it, copies were released only when the same control memory was added again, and a game that frees + a command buffer's control memory to its heap (teleporting between towns: new areas, new command buffers) left every + copy recorded there behind: 88,000 -> 483,000 live copies (29 -> 146 MiB) in a 27-minute session (analysis/long19), + and the sweep walks them all. TOTK_COPY_TOKEN=0: no marks (the old behaviour). */ +#define COPY_TOKEN_MAGIC 0x4e454b4f54594d43ull +extern __thread sigjmp_buf *runtime_fault_guard; // runtime/loader.c: a guarded read of guest memory survives a fault +static int async_replay(void); /* below: TOTK_ASYNC_REPLAY */ +/* With the translation thread (TOTK_ASYNC_REPLAY=1, iPhone default) the game may reuse a submitted command buffer's control + memory as soon as the submit call returns, before the queued replay used the copy: marks are not used there yet (a copy + last submitted at or after the submission being replayed would have to be kept). */ +static int copy_token_on(void) { static int v = -1; if (v < 0) { const char *e = getenv("TOTK_COPY_TOKEN"); v = !(e && !strcmp(e, "0")) && !async_replay(); + if (!v) LOG("[cmd] 录制副本记号：关（%s）：副本只在控制内存被重新加入时释放", e && !strcmp(e, "0") ? "TOTK_COPY_TOKEN=0" : "翻译线程模式还不支持"); } return v; } +static void copy_token_write(uintptr_t at, const uint8_t *copy) { uint64_t v[2] = {COPY_TOKEN_MAGIC, (uint64_t)(uintptr_t)copy}; memcpy((void *)at, v, sizeof v); } +static int copy_token_negative(void) { static int v = -1; if (v < 0) { const char *e = getenv("TOTK_PROBE_NEGATIVE"); v = e && atoi(e) == 22; } return v; } /* probe negative 22: marks never checked */ +/* Reading a mark (pin_scan): the fault guard is armed only around the 16-byte read; the jump buffer is set once per walked + slice with the signal mask saved (sigsetjmp(…, 1)) -- the crash handler runs with SIGSEGV / SIGBUS blocked, and a jump out + of it that does not restore the mask leaves them blocked for good: the thread's next fault (a dirty-page write) then + faults forever at 100 % CPU (2026-10-17 analysis/long20, first version with sigsetjmp(…, 0) per read: hung while loading). */ +static sigjmp_buf *g_mark_jb; /* only the replay thread walks copies (g_copy_mu held) */ +static int copy_token_intact(const cmd_copy *c) { + if (copy_token_negative()) return 1; + uint64_t v[2]; + runtime_fault_guard = g_mark_jb; + memcpy(v, (const void *)c->ctl_lo, sizeof v); + runtime_fault_guard = NULL; + return v[0] == COPY_TOKEN_MAGIC && v[1] == (uint64_t)(uintptr_t)c->copy; +} +/* Drop the buckets whose copies are all gone (g_copy_mu held, between sweep windows: the walk keeps bucket positions). */ +static void cbk_compact(void) { + if (g_cbk_empty < 4096 || g_cbk_empty * 2 < g_cbk_used) return; + size_t oc = g_cbk_cap, live = g_cbk_used - g_cbk_empty, cap = 1024; copy_bucket *ob = g_cbk; + while (cap < live * 2 + 2) cap *= 2; + g_cbk = calloc(cap, sizeof *g_cbk); if (!g_cbk) die("command copy index"); g_cbk_cap = cap; + for (size_t i = 0; i < oc; i++) if (ob[i].used) { + if (ob[i].empty) { free(ob[i].v); continue; } + size_t k = cbk_slot(ob[i].page); while (g_cbk[k].used) k = (k + 1) & (g_cbk_cap - 1); g_cbk[k] = ob[i]; + } + free(ob); g_cbk_used = live; g_cbk_empty = 0; g_cbk_compactions++; +} +NVN(nvnCommandBufferEndRecording, NVNcommandHandle, NVNcommandBuffer *c) { + cmd_t *o = CB(c); + size_t hw_before = o->hw_used; uint8_t *p = cmd_alloc(o, 8); put_hdr(p, OP_END, 8); o->hw_used -= 8; + o->recording = 0; + if (reclog_limit() && (int)t_rec_n < reclog_limit()) { char line[1024]; size_t n = 0; t_rec_n++; + for (unsigned i = 1; i < OP_COUNT_ && n < sizeof line - 24; i++) if (t_rec_hist[i]) n += snprintf(line + n, sizeof line - n, " op%u×%u", i, t_rec_hist[i]); + LOG("[rec] #%u cb=%p ours=%zu hw=%zu%s", t_rec_n, (void *)c, o->cmd_used - o->rec_start - 16, hw_before - o->hw_start, line); } + uint8_t *copy = flatten_recording(o->seg); + if (call_stats_on()) call_stats_end(c, copy); + /* Finalizing the recorder does not retire its handle; only reusing the backing control region does. */ + uintptr_t lo = (uintptr_t)o->ctl + (o->ctl_used < o->ctl_size ? o->ctl_used : 0); + uintptr_t hi = o->ctl ? lo + 1 : 0; + int token = o->ctl && o->ctl_used + 64 <= o->ctl_size && copy_token_on(); /* only inside the control memory the game gave */ + o->ctl_used += 64; + if (token) copy_token_write(lo, copy); + pthread_mutex_lock(&g_copy_mu); + { copy_bucket *b = cbk_get(lo >> COPY_PAGE_SHIFT, 1); + if (b->empty) { b->empty = 0; g_cbk_empty--; } + if (b->n == b->cap) { b->cap = b->cap ? b->cap * 2 : 8; b->v = realloc(b->v, b->cap * sizeof *b->v); if (!b->v) die("command copy table"); } + b->v[b->n++] = (cmd_copy){lo, hi, copy, 0, (uint32_t)token}; g_ncopies++; } + { static uint64_t made; if (++made % 20000 == 0) + LOG("[cmd] recording copies: %llu made, %zu live, %.1f MiB, submit %llu; released %llu at a re-added control memory, %llu whose mark there was gone (%llu unmapped); index %zu buckets (%zu empty) of %zu, compacted %llu times", + (unsigned long long)made, g_ncopies, g_copy_bytes / 1048576.0, (unsigned long long)g_submits, (unsigned long long)g_copy_drop_readd, (unsigned long long)g_copy_drop_token, (unsigned long long)g_copy_mark_faults, + g_cbk_used, g_cbk_empty, g_cbk_cap, (unsigned long long)g_cbk_compactions); } + pthread_mutex_unlock(&g_copy_mu); + if (cmdlog_env()) LOG("[cmd] EndRecording cb=%p recorded=%p copy=%p ctl=%p", (void *)c, (void *)o->seg, (void *)copy, o->ctl); + if (totk_life_on() && totk_life_cb_take(c)) { totk_life_log("end cb=%p handle=%p", (void *)c, (void *)copy); totk_life_handle_add((uint64_t)(uintptr_t)copy); } + if (getenv("TOTK_CHAIN_LOG")) { static int n; if (n++ < 200) guest_chain("EndRecording"); } + return (NVNcommandHandle)copy; +} +NVN(nvnCommandBufferIsRecording, NVNboolean, const NVNcommandBuffer *c) { return ((const cmd_t *)c)->recording != 0; } +// TOTK: the referenced recording is copied into the caller at this point (it is complete and valid now). The game +// recycles command memory in small steps, so a handle may be gone by the time the caller is submitted; inlining +// keeps the caller self-contained. Copying at the call position is the observable behaviour of both entry points. +/* TOTK_CALL_STATS=1: how much of a frame comes from called command lists and how much of that repeats unchanged + (same handle, same bytes as the previous frame): the measure of what caching translated lists could save. Printed + every 300 frames by nvnQueuePresentTexture. */ +static uint64_t g_frame; // defined with the frame bookkeeping below +typedef struct { uint64_t handle, hash, frame; } call_seen; +#define CALL_SEEN 8192 +static call_seen g_call_seen[CALL_SEEN]; +static uint64_t g_cs_calls, g_cs_draws, g_cs_same_calls, g_cs_same_draws, g_cs_bytes, g_cs_same_bytes; +static int call_stats_on(void) { static int v = -1; if (v < 0) v = getenv("TOTK_CALL_STATS") != NULL; return v; } +// Nested lists are inlined into their caller, so a draw would be counted at every level: each finished list keeps its +// own draws only (total minus what was inlined into it while recording), in host-side direct-mapped tables. +#define CS_TAB 16384 +static struct { const void *cb; uint64_t inl; } g_cs_inl[CS_TAB]; +static struct { uint64_t handle, own; } g_cs_own[CS_TAB]; +static uint64_t list_draws(const uint8_t *q) { uint64_t d = 0; for (;;) { op_hdr x; memcpy(&x, q, sizeof x); if (x.op == OP_END || x.size < 8) break; if (x.op == OP_DRAW || x.op == OP_DRAW_IDX) d++; q += x.size; } return d; } +static void call_stats_end(const void *cb, const uint8_t *copy) { + unsigned i = (unsigned)(((uintptr_t)cb >> 4) % CS_TAB); uint64_t inl = g_cs_inl[i].cb == cb ? g_cs_inl[i].inl : 0; g_cs_inl[i].cb = NULL; g_cs_inl[i].inl = 0; + uint64_t all = list_draws(copy); unsigned j = (unsigned)(((uintptr_t)copy >> 4) % CS_TAB); + g_cs_own[j].handle = (uint64_t)(uintptr_t)copy; g_cs_own[j].own = all > inl ? all - inl : 0; +} +static void call_stats_note(const void *cb, uint64_t hd, const uint8_t *p) { + uint64_t h = 1469598103934665603ull, draws = 0, alld = 0, bytes = 0; + for (const uint8_t *q = p;;) { op_hdr x; memcpy(&x, q, sizeof x); if (x.op == OP_END || x.size < 8) break; + if (x.op == OP_DRAW || x.op == OP_DRAW_IDX) alld++; + for (uint32_t i = 0; i < x.size; i += 8) { uint64_t w; memcpy(&w, q + i, 8); h = (h ^ w) * 1099511628211ull; } + bytes += x.size; q += x.size; } + { unsigned j = (unsigned)((hd >> 4) % CS_TAB); draws = g_cs_own[j].handle == hd ? g_cs_own[j].own : alld; } + { unsigned i = (unsigned)(((uintptr_t)cb >> 4) % CS_TAB); if (g_cs_inl[i].cb != cb) { g_cs_inl[i].cb = cb; g_cs_inl[i].inl = 0; } g_cs_inl[i].inl += alld; } + call_seen *e = &g_call_seen[(hd >> 4) % CALL_SEEN]; + int same = e->handle == hd && e->hash == h && e->frame + 1 >= g_frame; + g_cs_calls++; g_cs_draws += draws; g_cs_bytes += bytes; + if (same) { g_cs_same_calls++; g_cs_same_draws += draws; g_cs_same_bytes += bytes; } + *e = (call_seen){hd, h, g_frame}; +} +static _Atomic int g_inline_bulk_on = -1; +static int g_inline_bulk_negative; +static int inline_bulk_on(void) { + int v = atomic_load_explicit(&g_inline_bulk_on, memory_order_relaxed); + if (v < 0) { + const char *e = getenv("TOTK_INLINE_BULK"); int expected = -1; + atomic_compare_exchange_strong(&g_inline_bulk_on, &expected, e && !strcmp(e, "1")); + v = atomic_load_explicit(&g_inline_bulk_on, memory_order_relaxed); + } + return v; +} +void dm_nvn_inline_bulk_set(int v) { + atomic_store_explicit(&g_inline_bulk_on, v != 0, memory_order_relaxed); + LOG("[flag] inline_bulk = %d", v != 0); +} +static void inline_recording(NVNcommandBuffer *c, NVNcommandHandle hd) { + const uint8_t *p = (const uint8_t *)(uintptr_t)hd; + if (!p) die("CallCommands/CopyCommands with a null handle"); + copy_touch(hd, g_submits); + if (call_stats_on()) call_stats_note(c, hd, p); + /* EndRecording already flattened and checked this contiguous body. Copy + its bytes now, at exactly the original call position. Deferred bindings + remain deferred; source handle retirement cannot affect the caller. + CMDSTAT retains the record-by-record path and its exact counters. */ + const copy_prefix *meta = copy_of(hd); + if (inline_bulk_on() && !cs_on() && meta->body_bytes && meta->last_size) { + cmd_t *o = CB(c); + uint8_t *dst = cmd_alloc(o, meta->body_bytes); + memcpy(dst, p + 8, meta->body_bytes); + o->last_rec = meta->last_size; + if (g_inline_bulk_negative) dst[8] ^= 1; + return; + } + for (;;) { + op_hdr h; memcpy(&h, p, sizeof h); + if (h.size < 8 || (h.size & 7) || h.size > (64u << 20) || h.op >= OP_COUNT_) die("corrupt recording in called handle %#llx (op %u size %u)", (unsigned long long)hd, h.op, h.size); + if (h.op == OP_END) return; + if (h.op != OP_BEGIN) { uint8_t *d = cmd_alloc(CB(c), h.size); memcpy(d, p, h.size); } + p += h.size; + } +} +static void life_call(NVNcommandBuffer *c, int n, const NVNcommandHandle *h, const char *what) { // DM_NVN_LIFELOG + if (totk_life_on()) for (int i = 0; i < n; i++) if (totk_life_handle_has(h[i])) { totk_life_log("%s cb=%p handle=%#llx", what, (void *)c, (unsigned long long)h[i]); totk_life_cb_mark(c); } +} +NVN(nvnCommandBufferCopyCommands, void, NVNcommandBuffer *c, int n, const NVNcommandHandle *h) { + life_call(c, n, h, "copy"); + for (int i = 0; i < n; i++) inline_recording(c, h[i]); +} +NVN(nvnCommandBufferCallCommands, void, NVNcommandBuffer *c, int n, const NVNcommandHandle *h) { + if (cmdlog_env()) for (int i = 0; i < n; i++) LOG("[cmd] CallCommands cb=%p handle=%#llx", (void *)c, (unsigned long long)h[i]); + life_call(c, n, h, "call"); + for (int i = 0; i < n; i++) inline_recording(c, h[i]); +} +NVN(nvnCommandBufferSignalEvent, void, NVNcommandBuffer *c, const NVNevent *e, NVNeventSignalMode m, NVNeventSignalLocation loc, int flags, uint32_t v) { + (void)loc; (void)flags; + uint8_t *p = cmd_alloc(CB(c), 24); put_hdr(p, OP_SIGNAL_EVENT, 24); + const ev_t *ev = (const ev_t *)e; volatile uint32_t *val = ev->val; uint32_t mm = m; + memcpy(p + 8, &val, 8); memcpy(p + 16, &mm, 4); memcpy(p + 20, &v, 4); +} +NVN(nvnCommandBufferResetCounter, void, NVNcommandBuffer *c, NVNcounterType t) { + uint8_t *p = cmd_alloc(CB(c), 16); put_hdr(p, OP_RESET_COUNTER, 16); uint64_t tt = t; memcpy(p + 8, &tt, 8); +} +// TOTK: TOTK_CMD_LOG=1 logs selected recorded commands; TOTK_CMD_LOG_FRAMES=a-b restricts it to a frame window (g_frame_pub) +static int cmdlog_on(void) { + static int on = -1; static long f0 = -1, f1 = -1; + if (on < 0) { on = getenv("TOTK_CMD_LOG") != NULL; const char *e = getenv("TOTK_CMD_LOG_FRAMES"); if (e) { f0 = atol(e); f1 = strchr(e, '-') ? atol(strchr(e, '-') + 1) : f0; } } + if (!on) return 0; if (f0 < 0) return 1; extern uint64_t g_frame_pub; return (long)g_frame_pub >= f0 && (long)g_frame_pub <= f1; +} +NVN(nvnCommandBufferReportCounter, void, NVNcommandBuffer *c, NVNcounterType t, NVNbufferAddress a) { + if (cmdlog_on()) LOG("[cmd] F%llu ReportCounter type=%d addr=%#llx", (unsigned long long)g_frame_pub, (int)t, (unsigned long long)a); + uint8_t *p = cmd_alloc(CB(c), 24); put_hdr(p, OP_REPORT_COUNTER, 24); uint64_t tt = t; memcpy(p + 8, &tt, 8); memcpy(p + 16, &a, 8); +} +NVN(nvnCommandBufferCopyBufferToBuffer, void, NVNcommandBuffer *c, NVNbufferAddress src, NVNbufferAddress dst, size_t n, int flags) { + (void)flags; + uint8_t *p = cmd_alloc(CB(c), 32); put_hdr(p, OP_COPY_B2B, 32); uint64_t nn = n; + memcpy(p + 8, &src, 8); memcpy(p + 16, &dst, 8); memcpy(p + 24, &nn, 8); +} +// uniform 更新：数据在录制时拷进命令内存（NVN 语义：录制后调用方可改原数据），提交时写进缓冲 +// uniform update: the data is copied into command memory at record time (NVN semantics: the caller may modify the original data after recording) and written into the buffer at submit +NVN(nvnCommandBufferUpdateUniformBuffer, void, NVNcommandBuffer *c, NVNbufferAddress buf, size_t bufsz, ptrdiff_t off, size_t n, const void *data) { + (void)bufsz; + uint32_t sz = (uint32_t)(24 + n); + uint8_t *p = cmd_alloc(CB(c), sz); put_hdr(p, OP_UPDATE_UB, (sz + 7) & ~7u); + uint64_t dst = buf + off, nn = n; memcpy(p + 8, &dst, 8); memcpy(p + 16, &nn, 8); memcpy(p + 24, data, n); +} + +// ---------- 渲染命令：参数按 nvn_obj.h 的载荷格式录进命令内存，回放见 nvn_mtl.m ---------- +// ---------- Render commands: parameters are recorded into command memory in the payload format of nvn_obj.h; for replay see nvn_mtl.m ---------- +static void *rec(NVNcommandBuffer *c, uint32_t op, size_t payload) { + uint32_t sz = (uint32_t)((8 + payload + 7) & ~(size_t)7); + uint8_t *p = cmd_alloc(CB(c), sz); put_hdr(p, op, sz); + memset(p + 8, 0, sz - 8); + return p + 8; +} +#define REC(c, OP, T) ((T *)rec((c), (OP), sizeof(T))) +// 带数组的绑定记录按实际个数分配：前缀字段 + n 个元素（回放端只读前 n 个）。原先按最大个数定长记（绑顶点缓冲 272 字节、绑贴图 / 采样器 280 字节， +// 实际常只有 1–2 个），空尾占每帧命令字节的四到五成；88 级法师的高级属性页每帧录 12.5 MB，把游戏自己的命令内存池（约每帧 105 段 × 128 KB）用空 +// Binding records with arrays are allocated by the actual count: prefix fields + n elements (the replay side reads only the first n). Previously they were recorded at fixed length for the maximum count (bind vertex buffers 272 bytes, bind textures / samplers 280 bytes, +// while there are usually only 1–2 in practice), and the empty tail took 40–50% of the command bytes per frame; the advanced stats page of the level 88 sorceress recorded 12.5 MB per frame, using up the game's own command memory pool (about 105 segments x 128 KB per frame) +#define RECN(c, OP, T, ARR, N) ((T *)rec((c), (OP), offsetof(T, ARR) + sizeof(((T *)0)->ARR[0]) * (size_t)(N))) +static void copy_view(view_t *dst, uint8_t *has, const NVNtextureView *v) { *has = v != NULL; if (v) *dst = *(const view_t *)v; } +// TOTK: recordings keep texture objects by pointer, but the game may finalize or reuse an NVNtexture (e.g. a +// temporary object) after recording and before submit; NVN itself only captured the GPU state. Records therefore +// point at an immutable host copy per texture initialization (serial). +/* TOTK (2026-10-17): those copies used to live for the whole process. The game initialises its transient render targets + again every frame (a new serial each time), so a long session piled them up: about 38 a frame, 3.16 million copies and + 405 MB of 128-byte blocks plus a 64 MB index after 27 minutes at the towns (analysis/long18: heap --showSizes), the + bulk of the long-session memory growth. Now a mark-and-sweep on the replay thread, before a submit's replay, in windows + of PIN_SWEEP_SUBMITS submits (the live recordings, 39,000 copies / 14 MiB, walked in PIN_SCAN_SLICES steps, one per + submit: a whole walk at once took 3-6 ms): a copy is marked if a live recording copy refers to it (the ops that hold one: set render + targets, copy buffer <-> texture, clear texture, copy texture -> texture), if the replay's current render targets do + (they carry over to the next submit, rt_same reads them), or if it was pinned in the last PIN_SWEEP_SUBMITS submits (a + recording in progress). It is freed after two windows in a row without a mark: a recording in progress that inlined a + list released since then, or a copy made after its bucket was walked, is walked in the next one. Marking goes by pointer through a set of the live copies, so a + stale pointer in an op is never dereferenced. TOTK_PIN_SWEEP=0 keeps every copy (the old behaviour). */ +#define PIN_SWEEP_SUBMITS 600 /* one marking window */ +#define PIN_SCAN_SLICES 120 /* the live recordings are walked in this many steps, one per submit, early in a window */ +typedef struct { tex_t t; uint64_t last_pin, alive_sweep; } pin_t; /* t first: ops hold (const tex_t *) pins */ +_Static_assert(offsetof(pin_t, t) == 0, "ops hold pins as tex_t pointers"); +typedef struct { pin_t **v; size_t cap, n; } pin_set; /* the live copies by address, for marking without dereferencing */ +static pthread_mutex_t g_pin_mu = PTHREAD_MUTEX_INITIALIZER; /* lock order: g_copy_mu, then g_pin_mu */ +static pin_t **g_pin_tab; static size_t g_pin_cap, g_pin_n; +static pin_set g_pin_ptrs; +static uint64_t g_pin_sweep_id = 1, g_pin_freed, g_pin_marks; +static size_t pin_slot(uint64_t serial, size_t cap) { return (size_t)((serial * 0x9E3779B97F4A7C15ull) & (cap - 1)); } +static size_t ptr_slot(const void *p, size_t cap) { return (size_t)((((uintptr_t)p >> 4) * 0x9E3779B97F4A7C15ull) >> 16) & (cap - 1); } +static void ptrs_put(pin_set *s, pin_t *p) { size_t j = ptr_slot(p, s->cap); while (s->v[j]) j = (j + 1) & (s->cap - 1); s->v[j] = p; s->n++; } +static void pin_rebuild(size_t cap) { /* g_pin_mu held: the serial table and the address set from the live copies (entries set to NULL are gone) */ + pin_t **t2 = calloc(cap, sizeof *t2); pin_set s2 = {calloc(cap, sizeof(pin_t *)), cap, 0}; + if (!t2 || !s2.v) die("texture pin table"); + for (size_t i = 0; i < g_pin_cap; i++) if (g_pin_tab[i]) { + size_t j = pin_slot(g_pin_tab[i]->t.serial, cap); while (t2[j]) j = (j + 1) & (cap - 1); t2[j] = g_pin_tab[i]; + ptrs_put(&s2, g_pin_tab[i]); + } + free(g_pin_tab); g_pin_tab = t2; g_pin_cap = cap; + free(g_pin_ptrs.v); g_pin_ptrs = s2; +} +static const tex_t *tex_pin(const NVNtexture *nt) { + const tex_t *t = (const tex_t *)nt; + if (!t) return NULL; + if (!t->serial) die("recording references an uninitialized texture object %p", (const void *)t); + pthread_mutex_lock(&g_pin_mu); + if ((g_pin_n + 1) * 2 > g_pin_cap) pin_rebuild(g_pin_cap ? g_pin_cap * 2 : 4096); + size_t j = pin_slot(t->serial, g_pin_cap); + while (g_pin_tab[j] && g_pin_tab[j]->t.serial != t->serial) j = (j + 1) & (g_pin_cap - 1); + if (!g_pin_tab[j]) { + pin_t *c = malloc(sizeof *c); if (!c) die("texture pin copy"); + c->t = *t; c->alive_sweep = g_pin_sweep_id; g_pin_tab[j] = c; g_pin_n++; + ptrs_put(&g_pin_ptrs, c); /* same capacity as the serial table, so it has room */ + } + g_pin_tab[j]->last_pin = __atomic_load_n(&g_submits, __ATOMIC_RELAXED); + const tex_t *r = &g_pin_tab[j]->t; + pthread_mutex_unlock(&g_pin_mu); + return r; +} +static void pin_mark(const void *p) { /* g_pin_mu held */ + if (!p || !g_pin_ptrs.cap) return; + for (size_t j = ptr_slot(p, g_pin_ptrs.cap); g_pin_ptrs.v[j]; j = (j + 1) & (g_pin_ptrs.cap - 1)) + if ((const void *)g_pin_ptrs.v[j] == p) { g_pin_ptrs.v[j]->alive_sweep = g_pin_sweep_id; g_pin_marks++; return; } +} +static int pin_negative(void) { static int v = -1; if (v < 0) { const char *e = getenv("TOTK_PROBE_NEGATIVE"); v = e && atoi(e) == 21; } return v; } +static void pin_mark_ops(const uint8_t *p) { /* one recording copy: OP_BEGIN ... OP_END */ + for (;;) { + op_hdr h; memcpy(&h, p, sizeof h); + if (h.op == OP_END || h.size < 8) return; + const uint8_t *a = p + 8; + switch (h.op) { + case OP_SET_RT: { if (pin_negative()) break; /* probe negative 21: render targets of recordings not marked */ + const p_set_rt *r = (const void *)a; int n = r->n < 0 ? 0 : r->n > 8 ? 8 : r->n; + for (int i = 0; i < n; i++) pin_mark(r->color[i]); pin_mark(r->depth); break; } + case OP_COPY_B2T: case OP_COPY_T2B: pin_mark(((const p_copy_b2t *)(const void *)a)->tex); break; + case OP_CLEAR_TEX: pin_mark(((const p_clear_tex *)(const void *)a)->tex); break; + case OP_COPY_T2T: { const p_copy_t2t *c = (const void *)a; pin_mark(c->src); pin_mark(c->dst); break; } + default: break; + } + p += h.size; + } +} +void mtl_pin_refs(void (*mark)(const void *)); /* nvn_mtl.m: the replay state that outlives a submit */ +static int pin_sweep_on(void) { static int v = -1; if (v < 0) { const char *e = getenv("TOTK_PIN_SWEEP"); v = !(e && !strcmp(e, "0")); if (!v) LOG("[nvn] TOTK_PIN_SWEEP=0：录制里的贴图副本一直保留（旧行为，长局内存会涨）"); } return v; } +/* One window: the copy buckets are walked in slices (pin_scan), then pin_decide frees what no recording, no replay state + and no recent recording refers to in this window and the one before. A bucket-table resize during the walk makes the + window incomplete: no decision for it (a skipped bucket must not cost a reference). */ +static size_t g_pin_scan_pos, g_pin_scan_cap; static int g_pin_scan_done = 1, g_pin_scan_ok; static uint64_t g_pin_win_end, g_pin_scanned; +static void pin_window_start(uint64_t now) { /* g_copy_mu held */ + cbk_compact(); + g_pin_win_end = now + PIN_SWEEP_SUBMITS; g_pin_scan_pos = 0; g_pin_scan_cap = g_cbk_cap; g_pin_scan_done = 0; g_pin_scan_ok = 1; g_pin_scanned = 0; +} +static void pin_scan(size_t buckets) { /* g_copy_mu and g_pin_mu held */ + if (g_pin_scan_done) return; + if (g_cbk_cap != g_pin_scan_cap) { g_pin_scan_ok = 0; g_pin_scan_done = 1; return; } + size_t end = g_pin_scan_pos + buckets < g_pin_scan_cap ? g_pin_scan_pos + buckets : g_pin_scan_cap; + volatile size_t k = g_pin_scan_pos; volatile uint32_t i = 0; + sigjmp_buf jb; + if (sigsetjmp(jb, 1)) { runtime_fault_guard = NULL; copy_drop(&g_cbk[k], i, 1); g_copy_mark_faults++; } /* reading copy (k, i)'s mark faulted: its control memory is unmapped */ + g_mark_jb = &jb; + for (; k < end; k++, i = 0) { + copy_bucket *b = &g_cbk[k]; if (!b->used) continue; + while (i < b->n) { + if (b->v[i].token && !copy_token_intact(&b->v[i])) { copy_drop(b, i, 1); continue; } /* its control memory was reused: the copy goes */ + pin_mark_ops(b->v[i].copy); g_pin_scanned++; i++; + } + } + g_mark_jb = NULL; + g_pin_scan_pos = end; if (end >= g_pin_scan_cap) g_pin_scan_done = 1; +} +/* Linear probing without tombstones: take the entry at slot i out and move each later entry of the run back if its home + slot does not lie in the gap (Knuth's algorithm R). home(): the slot an entry hashes to. */ +static void probe_delete(pin_t **tab, size_t cap, size_t i, size_t (*home)(const pin_t *, size_t)) { + for (size_t j = i;;) { + tab[i] = NULL; + for (;;) { + j = (j + 1) & (cap - 1); + if (!tab[j]) return; + size_t k = home(tab[j], cap); + if (i <= j ? (i < k && k <= j) : (i < k || k <= j)) continue; /* home inside (i, j]: it may stay */ + break; + } + tab[i] = tab[j]; i = j; + } +} +static size_t home_serial(const pin_t *p, size_t cap) { return pin_slot(p->t.serial, cap); } +/* Serials of the copies freed by the last decision: nvn_mtl.m forgets them as finalized serials (mtl_dead_forget, called + without the pin locks), since only a recording's texture copy can bring a finalized serial back to the replay. */ +static uint64_t *g_pin_gone; static size_t g_pin_gone_n, g_pin_gone_cap; +void mtl_dead_forget(const uint64_t *serials, size_t n); /* nvn_mtl.m */ +static void pin_gone_flush(void) { /* g_pin_mu not held */ + pthread_mutex_lock(&g_pin_mu); uint64_t *l = g_pin_gone; size_t n = g_pin_gone_n; g_pin_gone = NULL; g_pin_gone_n = g_pin_gone_cap = 0; pthread_mutex_unlock(&g_pin_mu); + if (n) mtl_dead_forget(l, n); + free(l); +} +int dm_nvn_pin_has_serial(uint64_t serial) { /* nvn_mtl.m (texture finalize): a recording may still bring this serial to the replay */ + pthread_mutex_lock(&g_pin_mu); + int has = 0; + if (g_pin_cap) for (size_t j = pin_slot(serial, g_pin_cap); g_pin_tab[j]; j = (j + 1) & (g_pin_cap - 1)) if (g_pin_tab[j]->t.serial == serial) { has = 1; break; } + pthread_mutex_unlock(&g_pin_mu); + return has; +} +static size_t home_ptr(const pin_t *p, size_t cap) { return ptr_slot(p, cap); } +static size_t pin_decide(uint64_t now, size_t *recent_out) { /* g_pin_mu held; the window's walk is complete */ + mtl_pin_refs(pin_mark); + size_t recent = 0, nf = 0; + static pin_t **gone; static size_t gone_cap; + for (size_t i = 0; i < g_pin_cap; i++) { + pin_t *p = g_pin_tab[i]; if (!p) continue; + if (p->last_pin + PIN_SWEEP_SUBMITS > now) { p->alive_sweep = g_pin_sweep_id; recent++; } + if (g_pin_sweep_id - p->alive_sweep >= 2) { + if (nf == gone_cap) { gone_cap = gone_cap ? gone_cap * 2 : 4096; gone = realloc(gone, gone_cap * sizeof *gone); if (!gone) die("texture pin sweep list"); } + gone[nf++] = p; + } + } + for (size_t f = 0; f < nf; f++) { /* out of both tables in place (a rebuild of both took ~2 ms at 46,000 copies), then freed */ + pin_t *p = gone[f]; size_t j; + for (j = pin_slot(p->t.serial, g_pin_cap); g_pin_tab[j] != p; j = (j + 1) & (g_pin_cap - 1)) if (!g_pin_tab[j]) die("texture pin table lost an entry"); + probe_delete(g_pin_tab, g_pin_cap, j, home_serial); + for (j = ptr_slot(p, g_pin_ptrs.cap); g_pin_ptrs.v[j] != p; j = (j + 1) & (g_pin_ptrs.cap - 1)) if (!g_pin_ptrs.v[j]) die("texture pin set lost an entry"); + probe_delete(g_pin_ptrs.v, g_pin_ptrs.cap, j, home_ptr); + if (g_pin_gone_n == g_pin_gone_cap) { g_pin_gone_cap = g_pin_gone_cap ? g_pin_gone_cap * 2 : 4096; g_pin_gone = realloc(g_pin_gone, g_pin_gone_cap * sizeof *g_pin_gone); if (!g_pin_gone) die("texture pin sweep list"); } + g_pin_gone[g_pin_gone_n++] = p->t.serial; + g_pin_ptrs.n--; g_pin_n--; free(p); + } + g_pin_freed += nf; g_pin_sweep_id++; *recent_out = recent; + return nf; +} +/* The replay thread, before a submit's replay (no replay state is being built). */ +static void pin_step(uint64_t now) { + static uint64_t windows, skipped, ns_total, ns_max, ns_decide_max; + if (!pin_sweep_on()) return; + uint64_t t0 = now_ns(); + pthread_mutex_lock(&g_copy_mu); pthread_mutex_lock(&g_pin_mu); + if (!g_pin_win_end) pin_window_start(now); + pin_scan((g_pin_scan_cap + PIN_SCAN_SLICES - 1) / PIN_SCAN_SLICES); + size_t freed = 0, recent = 0; int decided = 0; + if (now >= g_pin_win_end && g_pin_scan_done) { + uint64_t td = now_ns(); + if (g_pin_scan_ok) { freed = pin_decide(now, &recent); decided = 1; } else skipped++; + uint64_t dd = now_ns() - td; if (dd > ns_decide_max) ns_decide_max = dd; + windows++; pin_window_start(now); + } + size_t live = g_pin_n; uint64_t marks = g_pin_marks; + pthread_mutex_unlock(&g_pin_mu); pthread_mutex_unlock(&g_copy_mu); + if (decided) pin_gone_flush(); + uint64_t dt = now_ns() - t0; ns_total += dt; if (dt > ns_max) ns_max = dt; + if (decided && (windows == 1 || windows % 10 == 0)) + LOG("[nvn] 贴图副本回收 至提交 %llu：在册 %zu 个，本周期释放 %zu 个（累计 %llu，周期 %llu 个、扫描没走完不决定 %llu 个），近 %d 次提交录过 %zu 个，累计标记 %llu 次；每次提交平均 %.3f ms、最长 %.2f ms（决定那次最长 %.2f ms）", + (unsigned long long)now, live, freed, (unsigned long long)g_pin_freed, (unsigned long long)windows, (unsigned long long)skipped, PIN_SWEEP_SUBMITS, recent, + (unsigned long long)marks, ns_total / 1e6 / (double)now, ns_max / 1e6, ns_decide_max / 1e6); +} +/* The probe's whole sweep in one call: walk every bucket, then decide (now: the submit count it decides at). */ +static void pin_sweep(uint64_t now, int force) { + (void)force; + pthread_mutex_lock(&g_copy_mu); pthread_mutex_lock(&g_pin_mu); + pin_window_start(now); pin_scan(g_cbk_cap); size_t recent; if (g_pin_scan_ok) pin_decide(now, &recent); + g_pin_win_end = 0; /* the next pin_step starts a fresh window */ + pthread_mutex_unlock(&g_pin_mu); pthread_mutex_unlock(&g_copy_mu); + pin_gone_flush(); +} +size_t dm_nvn_pin_count(void) { pthread_mutex_lock(&g_pin_mu); size_t n = g_pin_n; pthread_mutex_unlock(&g_pin_mu); return n; } +NVN(nvnCommandBufferSetRenderTargets, void, NVNcommandBuffer *c, int n, const NVNtexture *const *ct, const NVNtextureView *const *cv, + const NVNtexture *dt, const NVNtextureView *dv) { + if (n < 0 || n > 8) { LOG("【红】SetRenderTargets 数量 %d", n); n = n < 0 ? 0 : 8; } + p_set_rt *r = REC(c, OP_SET_RT, p_set_rt); r->n = n; + for (int i = 0; i < n; i++) { r->color[i] = ct ? tex_pin(ct[i]) : NULL; copy_view(&r->cview[i], &r->has_cview[i], cv ? cv[i] : NULL); } + r->depth = tex_pin(dt); copy_view(&r->dview, &r->has_dview, dv); + if (totk_life_on()) for (int i = 0; i <= n; i++) { // DM_NVN_LIFELOG: render targets of the followed size (i == n: depth) + const tex_t *x = i < n ? (ct ? (const tex_t *)ct[i] : NULL) : (const tex_t *)dt; + if (totk_life_match(x)) { totk_life_log("rt cb=%p %s%d ser=%llu mem=%p", (void *)c, i < n ? "c" : "d", i < n ? i : 0, (unsigned long long)x->serial, (void *)x->mem); totk_life_cb_mark(c); } + } +} + +NVN(nvnCommandBufferClearColor, void, NVNcommandBuffer *c, int index, const float *rgba, int mask) { + p_clear_color *r = REC(c, OP_CLEAR_COLOR, p_clear_color); r->index = index; r->mask = mask; memcpy(r->rgba, rgba, 16); + if (cmdlog_on()) LOG("[cmd] F%llu ClearColor %d rgba=%g,%g,%g,%g mask=%#x", (unsigned long long)g_frame_pub, index, rgba[0], rgba[1], rgba[2], rgba[3], mask); +} +NVN(nvnCommandBufferClearDepthStencil, void, NVNcommandBuffer *c, float depth, uint32_t dmask, int stencil, int smask) { + if (cmdlog_on()) LOG("[cmd] F%llu ClearDepthStencil depth=%g dmask=%u stencil=%d smask=%#x", (unsigned long long)g_frame_pub, depth, dmask, stencil, smask); + p_clear_ds *r = REC(c, OP_CLEAR_DS, p_clear_ds); *r = (p_clear_ds){depth, (int)B(dmask), stencil, smask}; +} +NVN(nvnCommandBufferSetViewports, void, NVNcommandBuffer *c, int first, int n, const float *v) { + if (n < 0 || first + n > 16) { LOG("【红】SetViewports first=%d n=%d", first, n); return; } + p_viewports *r = REC(c, OP_VIEWPORTS, p_viewports); r->first = first; r->n = n; + for (int i = 0; i < n; i++) memcpy(r->v[i], v + 4 * i, 16); +} +NVN(nvnCommandBufferSetScissor, void, NVNcommandBuffer *c, int x, int y, int w, int h) { *REC(c, OP_SCISSOR, p_scissor) = (p_scissor){x, y, w, h}; } + +/* CPU-only regression: independently recorded handles can share command storage. */ +void dm_nvn_probe_commands(void) { + NVNcommandBuffer a, b, caller; + uint8_t control[3][256] = {{0}}, commands[4096] = {0}; + pool_t pool = {.mem = commands}; + NVNcommandBuffer *buffers[] = {&a, &b, &caller}; + size_t before = g_ncopies; uint64_t bytes_before = g_copy_bytes; + for (unsigned i = 0; i < 3; i++) { + h_nvnCommandBufferInitialize(buffers[i], NULL); + h_nvnCommandBufferAddControlMemory(buffers[i], control[i], sizeof control[i]); + h_nvnCommandBufferAddCommandMemory(buffers[i], (NVNmemoryPool *)&pool, 0, sizeof commands); + } + h_nvnCommandBufferBeginRecording(&a); + uint8_t *end = CB(&a)->end; + ((uint8_t *)&a)[0x10] = 1; /* Actual inline NVN dirty-flag write. */ + if (CB(&a)->end != end) die("command probe: guest flag corrupted allocator"); + h_nvnCommandBufferSetScissor(&a, 11, 22, 33, 44); + NVNcommandHandle ha = h_nvnCommandBufferEndRecording(&a); + h_nvnCommandBufferFinalize(&a); /* Handles outlive their recorder. */ + h_nvnCommandBufferAddCommandMemory(&b, (NVNmemoryPool *)&pool, 0, sizeof commands); + if (g_ncopies != before + 1) die("command probe: shared command memory retired a handle"); + h_nvnCommandBufferBeginRecording(&b); + h_nvnCommandBufferSetScissor(&b, 55, 66, 77, 88); + NVNcommandHandle hb = h_nvnCommandBufferEndRecording(&b); + h_nvnCommandBufferBeginRecording(&caller); + h_nvnCommandBufferCallCommands(&caller, 1, &ha); + h_nvnCommandBufferCopyCommands(&caller, 1, &hb); + NVNcommandHandle hc = h_nvnCommandBufferEndRecording(&caller); + h_nvnCommandBufferAddControlMemory(&b, control[0], sizeof control[0]); + if (g_ncopies != before + 2) die("command probe: control reuse retired wrong recordings"); + copy_touch(hb, g_submits); copy_touch(hc, g_submits); + const uint8_t *p = (const uint8_t *)(uintptr_t)hc; + const p_scissor expected[] = {{11,22,33,44}, {55,66,77,88}}; + unsigned n = 0; + for (;;) { + op_hdr h; memcpy(&h, p, sizeof h); + if (h.op == OP_END) break; + if (h.op == OP_SCISSOR) { + if (n >= 2 || memcmp(p + 8, &expected[n++], sizeof(p_scissor))) + die("command probe: cached payload changed"); + } + p += h.size; + } + if (n != 2) die("command probe: cached commands lost"); + h_nvnCommandBufferAddControlMemory(&b, control[1], sizeof control[1]); + h_nvnCommandBufferAddControlMemory(&caller, control[2], sizeof control[2]); + h_nvnCommandBufferFinalize(&b); h_nvnCommandBufferFinalize(&caller); + if (g_ncopies != before || g_copy_bytes != bytes_before) die("command probe: copy leak"); +} +/* TOTK (2026-10-17): a recording copy goes when its control memory is reused for other data (its mark there is gone) or + unmapped (reading the mark faults), and not otherwise: a copy whose mark is intact, or one recorded without room for a + mark, stays. Negative 22 (marks never checked) must fail here. */ +void dm_nvn_probe_copy_marks(void) { + static uint8_t ctl[3][256], cmds[4096]; memset(ctl, 0, sizeof ctl); + pool_t pool = {.mem = cmds}; + size_t pg = (size_t)getpagesize(); uint8_t *gone = mmap(NULL, pg, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (gone == MAP_FAILED) die("copy mark probe: mmap"); + NVNcommandBuffer a, b, c, d; NVNcommandBuffer *all[] = {&a, &b, &c, &d}; uint8_t *mem[] = {ctl[0], ctl[1], ctl[2], gone}; + size_t size[] = {sizeof ctl[0], sizeof ctl[1], 48 /* no room for a mark */, pg}; + size_t before = g_ncopies; uint64_t by_mark0 = g_copy_drop_token; + NVNcommandHandle h[4]; + for (int i = 0; i < 4; i++) { + h_nvnCommandBufferInitialize(all[i], NULL); + h_nvnCommandBufferAddControlMemory(all[i], mem[i], size[i]); + h_nvnCommandBufferAddCommandMemory(all[i], (NVNmemoryPool *)&pool, 0, sizeof cmds); + h_nvnCommandBufferBeginRecording(all[i]); h_nvnCommandBufferSetScissor(all[i], i, i, 1, 1); h[i] = h_nvnCommandBufferEndRecording(all[i]); + } + uint64_t m[2]; memcpy(m, ctl[0], sizeof m); + if (m[0] != COPY_TOKEN_MAGIC || m[1] != (uint64_t)h[0]) die("copy mark probe: EndRecording did not write its mark into the control memory"); + memcpy(m, ctl[2], sizeof m); if (m[0] == COPY_TOKEN_MAGIC) die("copy mark probe: a mark was written past the control memory the game gave"); + memset(ctl[0], 0xa5, 16); memset(ctl[2], 0xa5, 16); /* the game reuses a's and c's control memory for its own data */ + if (mprotect(gone, pg, PROT_NONE)) die("copy mark probe: mprotect"); /* and d's is gone */ + pin_sweep(g_submits, 1); + if (0) { /* [port] no signal masks on Windows */ die("copy mark probe: the fault guard left SIGSEGV / SIGBUS blocked"); } + if (g_ncopies == before + 4) die("copy mark probe: a recording copy whose control memory was reused was not released"); + if (g_ncopies != before + 2 || g_copy_drop_token != by_mark0 + 2) + die("copy mark probe: %zu of 4 copies left (expected 2: the intact mark and the one without a mark)", g_ncopies - before); + copy_touch(h[1], g_submits); copy_touch(h[2], g_submits); /* the two kept ones are still live recordings */ + munmap(gone, pg); + h_nvnCommandBufferAddControlMemory(&b, ctl[1], sizeof ctl[1]); h_nvnCommandBufferAddControlMemory(&c, ctl[2], 48); + for (int i = 0; i < 4; i++) h_nvnCommandBufferFinalize(all[i]); + if (g_ncopies != before) die("copy mark probe: copy leak"); +} +/* TOTK (2026-10-17): texture copies held by recordings go after two windows without a reference (the window a copy is + made in counts as referenced), and not earlier: not while a live recording, the replay's current render targets or a + recent recording (pinned in the last window) still refers to them. Negative 21 (render targets of recordings not + marked) must fail here. */ +void mtl_probe_pin_rt(const tex_t *t); /* nvn_mtl.m: the replay's current colour target (NULL: none) */ +void dm_nvn_probe_texture_pins(void) { + { /* in-place deletion keeps every remaining entry findable: 3,000 entries in 8,192 slots (clusters wrap around), half deleted at random */ + enum { N = 3000, CAP = 8192 }; static pin_t pins[N]; static pin_t *tab[CAP]; static uint8_t gone[N]; + memset(tab, 0, sizeof tab); memset(gone, 0, sizeof gone); uint64_t x = 0x9E3779B97F4A7C15ull; + for (int i = 0; i < N; i++) { pins[i].t.serial = 0xfeed100000000000ull + (uint64_t)i * 977u; size_t j = pin_slot(pins[i].t.serial, CAP); while (tab[j]) j = (j + 1) & (CAP - 1); tab[j] = &pins[i]; } + for (int r = 0; r < N / 2; r++) { + x ^= x << 13; x ^= x >> 7; x ^= x << 17; int i = (int)(x % N); if (gone[i]) continue; + size_t j = pin_slot(pins[i].t.serial, CAP); while (tab[j] != &pins[i]) { if (!tab[j]) die("pin probe: entry lost before deletion"); j = (j + 1) & (CAP - 1); } + probe_delete(tab, CAP, j, home_serial); gone[i] = 1; + } + size_t left = 0; for (int k = 0; k < CAP; k++) left += tab[k] != NULL; + for (int i = 0; i < N; i++) { + size_t j = pin_slot(pins[i].t.serial, CAP); int found = 0; + for (; tab[j]; j = (j + 1) & (CAP - 1)) if (tab[j] == &pins[i]) { found = 1; break; } + if (found == gone[i]) die("pin probe: in-place deletion %s entry %d", gone[i] ? "kept a deleted" : "lost a remaining", i); + left -= !gone[i]; + } + if (left) die("pin probe: in-place deletion left stray entries"); + } + const size_t pins0 = dm_nvn_pin_count(); + tex_t t[3]; memset(t, 0, sizeof t); for (int i = 0; i < 3; i++) t[i].serial = 0xfeed0000ull + (uint64_t)i; /* serials no real texture has */ + NVNcommandBuffer ra, rb; uint8_t ctl_a[256] = {0}, ctl_b[256] = {0}, cmds[4096] = {0}; pool_t pool = {.mem = cmds}; + NVNcommandBuffer *both[] = {&ra, &rb}; uint8_t *ctl[] = {ctl_a, ctl_b}; + for (int i = 0; i < 2; i++) { h_nvnCommandBufferInitialize(both[i], NULL); h_nvnCommandBufferAddControlMemory(both[i], ctl[i], 256); h_nvnCommandBufferAddCommandMemory(both[i], (NVNmemoryPool *)&pool, 0, sizeof cmds); } + const NVNtexture *c0 = (const NVNtexture *)&t[0], *c2 = (const NVNtexture *)&t[2]; + h_nvnCommandBufferBeginRecording(&ra); h_nvnCommandBufferSetRenderTargets(&ra, 1, &c0, NULL, (const NVNtexture *)&t[1], NULL); NVNcommandHandle ha = h_nvnCommandBufferEndRecording(&ra); + h_nvnCommandBufferBeginRecording(&rb); h_nvnCommandBufferSetRenderTargets(&rb, 1, &c2, NULL, NULL, NULL); (void)h_nvnCommandBufferEndRecording(&rb); + if (dm_nvn_pin_count() != pins0 + 3) die("pin probe: three copies expected, %zu", dm_nvn_pin_count() - pins0); + h_nvnCommandBufferAddControlMemory(&rb, ctl_b, 256); /* re-adding its control memory retires rb's recording: t[2]'s copy has no reference now */ + const uint64_t far = g_submits + 10 * PIN_SWEEP_SUBMITS; /* long after the recordings: none counts as recent */ + /* the window a copy is made in counts as referenced; it goes after two more windows without a reference */ + pin_sweep(far, 1); if (dm_nvn_pin_count() != pins0 + 3) die("pin probe: a copy went in the window it was made in"); + pin_sweep(far + 1, 1); if (dm_nvn_pin_count() != pins0 + 3) die("pin probe: a copy went after one window without a reference"); + pin_sweep(far + 2, 1); + if (dm_nvn_pin_count() < pins0 + 2) die("pin probe: copies a live recording refers to were freed"); + if (dm_nvn_pin_count() > pins0 + 2) die("pin probe: the unreferenced copy was not freed after two windows without a reference"); + pin_sweep(far + 3, 1); pin_sweep(far + 4, 1); if (dm_nvn_pin_count() != pins0 + 2) die("pin probe: copies a live recording refers to were freed"); + const uint8_t *p = (const uint8_t *)(uintptr_t)ha; const p_set_rt *rt = NULL; + for (;;) { op_hdr h; memcpy(&h, p, sizeof h); if (h.op == OP_END) break; if (h.op == OP_SET_RT) rt = (const void *)(p + 8); p += h.size; } + if (!rt || !rt->color[0] || rt->color[0]->serial != t[0].serial || !rt->depth || rt->depth->serial != t[1].serial) die("pin probe: the live recording lost its targets"); + /* the replay's current render target keeps its copy after its recording is gone */ + mtl_probe_pin_rt(rt->color[0]); + h_nvnCommandBufferAddControlMemory(&ra, ctl_a, 256); /* retires ha */ + pin_sweep(far + 5, 1); pin_sweep(far + 6, 1); pin_sweep(far + 7, 1); + if (dm_nvn_pin_count() != pins0 + 1) die("pin probe: %zu copies left, expected only the replay's render target", dm_nvn_pin_count() - pins0); + mtl_probe_pin_rt(NULL); + /* a copy pinned in the last window stays (a recording in progress may hold it) */ + h_nvnCommandBufferBeginRecording(&rb); h_nvnCommandBufferSetRenderTargets(&rb, 1, &c2, NULL, NULL, NULL); (void)h_nvnCommandBufferEndRecording(&rb); + h_nvnCommandBufferAddControlMemory(&rb, ctl_b, 256); + const uint64_t now = g_submits; pin_sweep(now, 1); pin_sweep(now, 1); pin_sweep(now, 1); + if (dm_nvn_pin_count() != pins0 + 2) die("pin probe: a recently pinned copy was freed (%zu left)", dm_nvn_pin_count() - pins0); + pin_sweep(far + 8, 1); pin_sweep(far + 9, 1); + if (dm_nvn_pin_count() != pins0) die("pin probe: %zu copies stayed after nothing referred to them", dm_nvn_pin_count() - pins0); + h_nvnCommandBufferFinalize(&ra); h_nvnCommandBufferFinalize(&rb); +} +/* Compare the complete production recording and allocator accounting against + the original per-record copier, including growth, nesting and source reuse. */ +void dm_nvn_probe_inline_bulk(void) { + int saved = atomic_load(&g_inline_bulk_on); unsigned checks = 0; + const unsigned lengths[] = {0, 1, 7, 3000}; + size_t copies_before = g_ncopies; uint64_t bytes_before = g_copy_bytes; + for (unsigned scenario = 0; scenario < sizeof lengths / sizeof *lengths; scenario++) { + NVNcommandBuffer cb[4]; uint8_t control[4][256] = {{0}}, commands[4096] = {0}; + pool_t pool = {.mem = commands}; + for (unsigned i = 0; i < 4; i++) { + h_nvnCommandBufferInitialize(&cb[i], NULL); + h_nvnCommandBufferAddControlMemory(&cb[i], control[i], sizeof control[i]); + h_nvnCommandBufferAddCommandMemory(&cb[i], (NVNmemoryPool *)&pool, 0, sizeof commands); + } + h_nvnCommandBufferBeginRecording(&cb[0]); + uint64_t payload[17]; for (unsigned j = 0; j < 17; j++) payload[j] = 0x12340000u + j; + for (unsigned j = 0; j < lengths[scenario]; j++) { + h_nvnCommandBufferSetScissor(&cb[0], j, j + 1, j + 2, j + 3); + if (j % 7 == 0) h_nvnCommandBufferUpdateUniformBuffer(&cb[0], 0x1000, sizeof payload, 0, (j % 17 + 1) * 8, payload); + } + NVNcommandHandle source = h_nvnCommandBufferEndRecording(&cb[0]); + for (unsigned mode = 0; mode < 2; mode++) { + atomic_store(&g_inline_bulk_on, mode); + g_inline_bulk_negative = mode && scenario && getenv("TOTK_PROBE_NEGATIVE") && atoi(getenv("TOTK_PROBE_NEGATIVE")) == 16; + h_nvnCommandBufferBeginRecording(&cb[mode + 1]); + h_nvnCommandBufferSetScissor(&cb[mode + 1], 91, 92, 93, 94); + h_nvnCommandBufferCallCommands(&cb[mode + 1], 1, &source); + h_nvnCommandBufferCopyCommands(&cb[mode + 1], 1, &source); + } + cmd_t *a = CB(&cb[1]), *b = CB(&cb[2]); + size_t used = a->cur - a->seg; + if ((size_t)(b->cur - b->seg) != used || memcmp(a->seg, b->seg, used)) die("inline bulk probe: copied bytes differ"); checks++; + if (a->cmd_used != b->cmd_used || a->hw_used != b->hw_used || a->last_rec != b->last_rec) die("inline bulk probe: accounting differs"); checks++; + // An operation after the call must still fix up its own hardware size. + hw_fix(a, 12); hw_fix(b, 12); + if (a->hw_used != b->hw_used || a->last_rec != b->last_rec) die("inline bulk probe: hardware fix differs"); checks++; + NVNcommandHandle reference = h_nvnCommandBufferEndRecording(&cb[1]); + NVNcommandHandle fast = h_nvnCommandBufferEndRecording(&cb[2]); + h_nvnCommandBufferBeginRecording(&cb[3]); + h_nvnCommandBufferCallCommands(&cb[3], 1, &fast); + NVNcommandHandle nested = h_nvnCommandBufferEndRecording(&cb[3]); + h_nvnCommandBufferAddControlMemory(&cb[0], control[0], sizeof control[0]); + h_nvnCommandBufferAddControlMemory(&cb[2], control[2], sizeof control[2]); + // source and intermediate fast handle are gone; nested bytes must survive. + size_t full_size = copy_of(reference)->body_bytes + 16; + if (memcmp((const void *)(uintptr_t)reference, (const void *)(uintptr_t)nested, full_size)) die("inline bulk probe: nested copy changed after source retirement"); checks++; + for (unsigned i = 0; i < 4; i++) { + h_nvnCommandBufferAddControlMemory(&cb[i], control[i], sizeof control[i]); + h_nvnCommandBufferFinalize(&cb[i]); + } + } + if (g_ncopies != copies_before || g_copy_bytes != bytes_before) die("inline bulk probe: copy leak"); checks++; + g_inline_bulk_negative = 0; atomic_store(&g_inline_bulk_on, saved); + LOG("inline bulk probe: %u byte/accounting/growth/nesting/lifetime checks passed", checks); +} +NVN(nvnCommandBufferSetDepthRange, void, NVNcommandBuffer *c, float n, float f) { *REC(c, OP_DEPTH_RANGE, p_depth_range) = (p_depth_range){n, f}; } +// TOTK: the replay keeps one scissor / depth range (index 0); other indices only matter to shaders that +// select a viewport index, so differing values there are reported instead of silently used. +NVN(nvnCommandBufferSetViewport, void, NVNcommandBuffer *c, int x, int y, int w, int h) { float v[4] = {(float)x, (float)y, (float)w, (float)h}; h_nvnCommandBufferSetViewports(c, 0, 1, v); } +NVN(nvnCommandBufferSetScissors, void, NVNcommandBuffer *c, int first, int n, const int *r) { + if (first < 0 || n < 0 || first + n > 16) die("SetScissors first=%d n=%d", first, n); + for (int i = 1; i < n; i++) if (memcmp(r + 4 * i, r, 16)) { static int once; if (!once++) LOG("【红】SetScissors: differing scissor %d ignored (single-scissor replay)", first + i); } + if (first == 0 && n > 0) h_nvnCommandBufferSetScissor(c, r[0], r[1], r[2], r[3]); +} +NVN(nvnCommandBufferSetDepthRanges, void, NVNcommandBuffer *c, int first, int n, const float *r) { + if (first < 0 || n < 0 || first + n > 16) die("SetDepthRanges first=%d n=%d", first, n); + for (int i = 1; i < n; i++) if (memcmp(r + 2 * i, r, 8)) { static int once; if (!once++) LOG("【红】SetDepthRanges: differing range %d ignored (single-range replay)", first + i); } + if (first == 0 && n > 0) h_nvnCommandBufferSetDepthRange(c, r[0], r[1]); +} +NVN(nvnCommandBufferSetDepthClamp, void, NVNcommandBuffer *c, uint32_t e) { REC(c, OP_DEPTH_CLAMP, p_flag)->value = B(e) != 0; } +NVN(nvnCommandBufferSetRenderEnable, void, NVNcommandBuffer *c, uint32_t e) { + if (cmdlog_on()) LOG("[cmd] F%llu SetRenderEnable %d", (unsigned long long)g_frame_pub, B(e) != 0); + REC(c, OP_RENDER_ENABLE, p_flag)->value = B(e) != 0; } +// Conditional rendering (NVN SetRenderEnableConditional): rendering is enabled when the two NVNcounterData.counter values at +// addr / addr+16 compare equal (mode 0) or not equal (mode 1). TOTK gates most of its ClearColor commands with it: the CPU +// writes counter=1 to request a clear and the TIMESTAMP_TOP report issued right after the clear resets it to 0, so the +// clear runs exactly once. The replay evaluates it in GPU order (see cond_eval in nvn_mtl.m). +NVN(nvnCommandBufferSetRenderEnableConditional, void, NVNcommandBuffer *c, NVNconditionalRenderMode m, NVNbufferAddress a) { + if (cmdlog_on()) { void *bf; uint64_t off; uint64_t x = 0, xt = 0, y = 0, yt = 0; int ok = mtl_addr(a, 32, &bf, &off); + if (ok) { memcpy(&x, (const void *)(uintptr_t)a, 8); memcpy(&xt, (const void *)(uintptr_t)(a + 8), 8); memcpy(&y, (const void *)(uintptr_t)(a + 16), 8); memcpy(&yt, (const void *)(uintptr_t)(a + 24), 8); } + LOG("[cmd] F%llu SetRenderEnableConditional mode=%d addr=%#llx mapped=%d x=%llu/%llu y=%llu/%llu", (unsigned long long)g_frame_pub, (int)m, (unsigned long long)a, ok, + (unsigned long long)x, (unsigned long long)xt, (unsigned long long)y, (unsigned long long)yt); } + if ((int)m != 0 && (int)m != 1) die("SetRenderEnableConditional: unknown mode %d", (int)m); + p_render_cond *r = REC(c, OP_RENDER_COND, p_render_cond); r->mode = (int32_t)m; r->pad = 0; r->addr = a; +} +// Rasterizer discard runs vertex work without rasterising; with no transform feedback that is the same as not drawing. +NVN(nvnCommandBufferSetRasterizerDiscard, void, NVNcommandBuffer *c, uint32_t e) { REC(c, OP_RASTER_DISCARD, p_flag)->value = B(e) != 0; } +/* TOTK diagnostic: TOTK_BINDPROG_LOG=- prints every BindProgram of those frames with the program object, its + vertex-stage shader data address and the guest call chain (module+offset), so a program choice can be traced to game code. */ +static uint64_t g_frame; +int runtime_module_offset(uint64_t pc, const char **name, uint64_t *off); +static _Atomic int g_diag_env_once = -1; +static const char *bindprog_env(void) { return getenv("TOTK_BINDPROG_LOG"); } +static int diag_env_once_on(void) { + int v = atomic_load_explicit(&g_diag_env_once, memory_order_relaxed); + if (v < 0) { + const char *e = getenv("TOTK_DIAG_ENV_ONCE"); int initial = -1; + // [port] default on: off re-reads TOTK_BINDPROG_LOG on every nvnCommandBufferBindProgram while it is unset, + // and the Windows CRT getenv (a lock and a scan of the environment) took 7-13% of the game threads' time. + int desired = !(e && !strcmp(e, "0")); + atomic_compare_exchange_strong(&g_diag_env_once, &initial, desired); + v = atomic_load_explicit(&g_diag_env_once, memory_order_relaxed); + } + return v; +} +void nvn_diag_env_once_set(int v) { + atomic_store_explicit(&g_diag_env_once, v != 0, memory_order_relaxed); + LOG("[flag] diag_env_once = %d", v != 0); +} +static void bindprog_log(const prog_t *o, int stages) { + static nvn_diag_range_cache range; + static _Atomic unsigned lines; + nvn_diag_range_value v = nvn_diag_range_get(&range, diag_env_once_on(), bindprog_env); + if (v.lo < 0 || (long)g_frame < v.lo || (long)g_frame > v.hi) return; + if (atomic_load_explicit(&lines, memory_order_relaxed) >= 20000) return; + if (atomic_fetch_add_explicit(&lines, 1, memory_order_relaxed) >= 20000) return; + char b[700]; int n = snprintf(b, sizeof b, "[bindprog] frame=%llu mask=%#x prog=%p serial=%llu nstage=%u data0=%#llx ctl0=%p chain:", (unsigned long long)g_frame, stages, (const void *)o, (unsigned long long)(o ? o->serial : 0), o ? o->nstage : 0, o && o->nstage ? (unsigned long long)o->sd[0].data : 0ull, o && o->nstage ? o->sd[0].control : NULL); + uint64_t *fp = __builtin_frame_address(0); uint64_t *sw = fp ? (uint64_t *)fp[0] : NULL; /* our frame -> runtime_hle_switch frame record {guest fp, guest lr} */ + uint64_t gfp = sw ? sw[0] : 0, glr = sw ? sw[1] : 0; + for (int d = 0; d < 8 && n < (int)sizeof b - 48; d++) { + const char *nm; uint64_t off; + if (runtime_module_offset(glr, &nm, &off)) n += snprintf(b + n, sizeof b - n, " %s+%#llx", nm, (unsigned long long)off); else n += snprintf(b + n, sizeof b - n, " %#llx", (unsigned long long)glr); + if (!gfp || (gfp & 7)) break; uint64_t *f = (uint64_t *)(uintptr_t)gfp; glr = f[1]; gfp = f[0]; if (!glr) break; + } + LOG("%s", b); +} +NVN(nvnCommandBufferBindProgram, void, NVNcommandBuffer *c, const NVNprogram *p, int stages) { + *REC(c, OP_BIND_PROGRAM, p_bind_program) = (p_bind_program){(const prog_t *)p, stages}; + bindprog_log((const prog_t *)p, stages); +} +static void bind_state(NVNcommandBuffer *c, int kind, const void *st, size_t n) { p_bind_state *r = REC(c, OP_BIND_STATE, p_bind_state); r->kind = kind; memcpy(r->bytes, st, n); } +NVN(nvnCommandBufferBindBlendState, void, NVNcommandBuffer *c, const NVNblendState *st) { bind_state(c, 0, st, sizeof(blend_st)); } +NVN(nvnCommandBufferBindColorState, void, NVNcommandBuffer *c, const NVNcolorState *st) { bind_state(c, 1, st, sizeof(color_st)); } +NVN(nvnCommandBufferBindChannelMaskState, void, NVNcommandBuffer *c, const NVNchannelMaskState *st) { bind_state(c, 2, st, sizeof(chmask_st)); } +NVN(nvnCommandBufferBindMultisampleState, void, NVNcommandBuffer *c, const NVNmultisampleState *st) { bind_state(c, 3, st, sizeof(ms_st)); } +NVN(nvnCommandBufferBindPolygonState, void, NVNcommandBuffer *c, const NVNpolygonState *st) { bind_state(c, 4, st, sizeof(poly_st)); } +NVN(nvnCommandBufferBindDepthStencilState, void, NVNcommandBuffer *c, const NVNdepthStencilState *st) { bind_state(c, 5, st, sizeof(ds_st)); } +NVN(nvnCommandBufferBindVertexAttribState, void, NVNcommandBuffer *c, int n, const NVNvertexAttribState *st) { + if (n < 0 || n > 16) { LOG("【红】BindVertexAttribState n=%d", n); return; } + p_bind_vattr *r = RECN(c, OP_BIND_VATTR, p_bind_vattr, st, n); r->first = 0; r->n = n; memcpy(r->st, st, 4 * n); hw_fix(CB(c), 68); /* the driver always writes all 16 attribute formats (unused = disabled): 17 words */ +} +NVN(nvnCommandBufferBindVertexStreamState, void, NVNcommandBuffer *c, int n, const NVNvertexStreamState *st) { + if (n < 0 || n > 16) { LOG("【红】BindVertexStreamState n=%d", n); return; } + p_bind_vstream *r = RECN(c, OP_BIND_VSTREAM, p_bind_vstream, st, n); r->first = 0; r->n = n; memcpy(r->st, st, 8 * n); hw_fix(CB(c), 4 * (16 + (size_t)n)); +} +NVN(nvnCommandBufferBindVertexBuffers, void, NVNcommandBuffer *c, int first, int n, const NVNbufferRange *rg) { + if (n < 0 || first + n > 16) { LOG("【红】BindVertexBuffers first=%d n=%d", first, n); return; } + p_bind_vbuf *r = RECN(c, OP_BIND_VBUF, p_bind_vbuf, r, n); r->first = first; r->n = n; memcpy(r->r, rg, sizeof *rg * n); hw_fix(CB(c), 24 * (size_t)n); +} +NVN(nvnCommandBufferBindUniformBuffer, void, NVNcommandBuffer *c, NVNshaderStage st, int i, NVNbufferAddress a, size_t n) { + *REC(c, OP_BIND_UBO, p_bind_buf) = (p_bind_buf){st, i, a, n}; hw_fix(CB(c), 20); /* CB_SIZE/ADDRESS (4 words) + CB_BIND (1) */ +} +NVN(nvnCommandBufferBindUniformBuffers, void, NVNcommandBuffer *c, NVNshaderStage st, int first, int n, const NVNbufferRange *rg) { + for (int i = 0; i < n; i++) { *REC(c, OP_BIND_UBO, p_bind_buf) = (p_bind_buf){st, first + i, rg[i].address, rg[i].size}; hw_fix(CB(c), 20); } +} +NVN(nvnCommandBufferBindStorageBuffer, void, NVNcommandBuffer *c, NVNshaderStage st, int i, NVNbufferAddress a, size_t n) { + *REC(c, OP_BIND_SSBO, p_bind_buf) = (p_bind_buf){st, i, a, n}; hw_fix(CB(c), 20); /* macro 16 (1 word) + CB_DATA address/size (4) */ +} +NVN(nvnCommandBufferBindStorageBuffers, void, NVNcommandBuffer *c, NVNshaderStage st, int first, int n, const NVNbufferRange *rg) { + for (int i = 0; i < n; i++) { *REC(c, OP_BIND_SSBO, p_bind_buf) = (p_bind_buf){st, first + i, rg[i].address, rg[i].size}; hw_fix(CB(c), 20); } +} +/* DM_NVN_LIFELOG (totk_lifelog.h): one bound handle of a followed texture; for a deferred bind the logged id is the + value at record time, the replay's "use" lines show the one resolved at submit. */ +static void life_bind(NVNcommandBuffer *c, int op, int st, int slot, const uint64_t *hp, int deferred) { + int id = (int)(*hp & 0xFFFFF); uint64_t ser = totk_life_reg_serial(id); + if (!ser) return; + uint64_t caller = guest_main_caller(); + totk_life_log("bind cb=%p %s%s st=%d slot=%d id=%d ser=%llu hp=%p caller=main+%#llx", (void *)c, op == OP_BIND_TEX ? "tex" : "img", deferred ? " deferred" : "", + st, slot, id, (unsigned long long)ser, (const void *)hp, (unsigned long long)caller); + totk_life_cb_mark(c); + /* DM_NVN_LIFELOG_WATCH=
: write-watch the handle that caller binds first (who stores it, when) */ + { static long want = -2; static int armed; if (want == -2) { const char *e = getenv("DM_NVN_LIFELOG_WATCH"); want = e ? strtol(e, NULL, 0) : -1; } + if (want > 0 && !armed && caller == (uint64_t)want) { armed = 1; extern void runtime_watch_set(uintptr_t addr, size_t len); + totk_life_log("watching the handle at %p bound by main+%#lx (id=%d)", (const void *)hp, want, id); runtime_watch_set((uintptr_t)hp, 8); } } + /* the game's call chain of the first bind per (command buffer, slot): which code picks the handle */ + static const void *chained[32][2]; static int nchained; static pthread_mutex_t chain_mu = PTHREAD_MUTEX_INITIALIZER; int fresh = 0; + pthread_mutex_lock(&chain_mu); + { int j = 0; while (j < nchained && !(chained[j][0] == (const void *)c && chained[j][1] == (const void *)(intptr_t)slot)) j++; + if (j == nchained && nchained < 32) { chained[nchained][0] = c; chained[nchained][1] = (const void *)(intptr_t)slot; nchained++; fresh = 1; } } + pthread_mutex_unlock(&chain_mu); + if (fresh) { char w[64]; snprintf(w, sizeof w, "[life] bind chain cb=%p slot=%d", (void *)c, slot); guest_chain(w); } +} +static void bind_handles_rec(NVNcommandBuffer *c, int op, int st, int first, int n, const uint64_t *h, int deferred) { + if (n < 0 || n > 32) { LOG("【红】绑定句柄数 %d", n); return; } + p_bind_handles *r = RECN(c, op, p_bind_handles, h, n); r->stage = st; r->first = first; r->n = n; r->pad_ = 0; + r->src = deferred ? (uint64_t)(uintptr_t)h : 0; memcpy(r->h, h, 8 * n); + if (totk_life_on() && (op == OP_BIND_TEX || op == OP_BIND_IMG)) for (int k = 0; k < n; k++) life_bind(c, op, st, first + k, h + k, deferred); + hw_fix(CB(c), 16 + 4 * (size_t)n); /* macro 7 + method 0x1a2c + CB_POS and one 32-bit handle word per binding (64-bit handles: callers override with 16 + 8n) */ +} +static void bind_handles(NVNcommandBuffer *c, int op, int st, int first, int n, const uint64_t *h) { bind_handles_rec(c, op, st, first, n, h, 0); } +/* NVN 55 *Deferred binds (totk_nvn_ext.c): the original driver (sdk+0x46cc00 BindTexturesDeferred) only writes a control + token {0x17, stage, first, count} and the pointer to the game's handle array; the handles are read when the command + buffer is submitted. The game relies on that: material command buffers are recorded first and their handle tables + are filled before the submit. Reading the handles at record time bound the textures registered two frames earlier; + when the game moved a render target (the linear depth) to other memory, materials, lighting and fog read the old + memory for two frames: objects flickered and the ground showed through while the camera moved (2026-10-15). + hw_bytes: command bytes reported for the plain variant (resource heaps are sized from them, keep them unchanged). */ +void nvn_bind_handles_deferred(NVNcommandBuffer *c, int op, int st, int first, int n, const uint64_t *h, size_t hw_bytes) { + bind_handles_rec(c, op, st, first, n, h, 1); + if (hw_bytes && n >= 0 && n <= 32) hw_fix(CB(c), hw_bytes); +} +static _Atomic int g_def_on = -1; // TOTK_DEFERRED_BINDS=0 / runtime flag "deferred 0": keep the record-time handles (A/B) +static uint64_t g_def_n, g_def_changed, g_def_logged, g_def_ns; +void nvn_set_deferred_binds(int on) { atomic_store(&g_def_on, on); LOG("[flag] deferred = %d", on); } +static void resolve_deferred(const uint8_t *p, int depth) { // read the handles of every deferred bind in a submitted recording + for (;;) { + op_hdr h; memcpy(&h, p, sizeof h); + if (h.size < 8 || (h.size & 7) || h.op >= OP_COUNT_) die("corrupt recording at %p while resolving deferred binds (op %u size %u)", (const void *)p, h.op, h.size); + if (h.op == OP_END) return; + if (h.op == OP_JUMP) { memcpy(&p, p + 8, 8); continue; } + if (h.op == OP_CALL) { uint64_t t; memcpy(&t, p + 8, 8); if (depth >= 16) die("deferred bind resolution: call depth over 16"); resolve_deferred((const uint8_t *)(uintptr_t)t, depth + 1); } + else if (h.op == OP_BIND_TEX || h.op == OP_BIND_IMG || h.op == OP_BIND_SAMP) { + p_bind_handles *b = (p_bind_handles *)(uintptr_t)(p + 8); + if (b->src && b->n > 0 && b->n <= 32) { + uint64_t cur[32]; memcpy(cur, (const void *)(uintptr_t)b->src, 8 * (size_t)b->n); g_def_n++; + if (memcmp(cur, b->h, 8 * (size_t)b->n)) { + if (g_def_logged++ < 6) LOG("[nvn] 延迟绑定在提交时读到新句柄：阶段 %d 槽 %d 起，录制时 %#llx → 提交时 %#llx", b->stage, b->first, (unsigned long long)b->h[0], (unsigned long long)cur[0]); + g_def_changed++; memcpy(b->h, cur, 8 * (size_t)b->n); + } + } + } + p += h.size; + } +} +// 图像（可写贴图）：句柄 = 贴图池号（GetImageHandle 原样返回；游戏不调 RegisterImage，图像与贴图共用登记） +// Image (writable texture): handle = texture pool index (GetImageHandle returns it as-is; the game does not call RegisterImage, images and textures share registration) +NVN(nvnCommandBufferBindImages, void, NVNcommandBuffer *c, NVNshaderStage st, int first, int n, const NVNimageHandle *h) { + bind_handles(c, OP_BIND_IMG, st, first, n, (const uint64_t *)h); if (n >= 0 && n <= 32) hw_fix(CB(c), 16 + 8 * (size_t)n); +} +NVN(nvnCommandBufferBindSeparateTextures, void, NVNcommandBuffer *c, NVNshaderStage st, int first, int n, const NVNseparateTextureHandle *h) { + bind_handles(c, OP_BIND_TEX, st, first, n, (const uint64_t *)h); if (n >= 0 && n <= 32) hw_fix(CB(c), 16 + 8 * (size_t)n); +} +NVN(nvnCommandBufferBindSeparateSamplers, void, NVNcommandBuffer *c, NVNshaderStage st, int first, int n, const NVNseparateSamplerHandle *h) { + bind_handles(c, OP_BIND_SAMP, st, first, n, (const uint64_t *)h); if (n >= 0 && n <= 32) hw_fix(CB(c), 16 + 8 * (size_t)n); +} +NVN(nvnCommandBufferBindSeparateSampler, void, NVNcommandBuffer *c, NVNshaderStage st, int i, NVNseparateSamplerHandle h) { + bind_handles(c, OP_BIND_SAMP, st, i, 1, &h.value); hw_fix(CB(c), 24); +} +// TOTK: combined texture handles share the texture slots with separate textures (sampler id in bits 20+). +NVN(nvnCommandBufferBindTexture, void, NVNcommandBuffer *c, NVNshaderStage st, int i, NVNtextureHandle h) { uint64_t v = h; bind_handles(c, OP_BIND_TEX, st, i, 1, &v); } +NVN(nvnCommandBufferBindTextures, void, NVNcommandBuffer *c, NVNshaderStage st, int first, int n, const NVNtextureHandle *h) { bind_handles(c, OP_BIND_TEX, st, first, n, (const uint64_t *)h); } +NVN(nvnCommandBufferBindImage, void, NVNcommandBuffer *c, NVNshaderStage st, int i, NVNimageHandle h) { uint64_t v = h; bind_handles(c, OP_BIND_IMG, st, i, 1, &v); hw_fix(CB(c), 24); } +NVN(nvnCommandBufferBindSeparateTexture, void, NVNcommandBuffer *c, NVNshaderStage st, int i, NVNseparateTextureHandle h) { bind_handles(c, OP_BIND_TEX, st, i, 1, &h.value); hw_fix(CB(c), 24); } +NVN(nvnCommandBufferBindVertexBuffer, void, NVNcommandBuffer *c, int i, NVNbufferAddress a, size_t n) { NVNbufferRange r = {a, n}; h_nvnCommandBufferBindVertexBuffers(c, i, 1, &r); } +/* CPU-only regression: deferred binds take the handles the game's array holds at submit, also through CallCommands; + immediate binds keep the recorded value. TOTK_PROBE_NEGATIVE=1 skips the submit-time read and must fail. */ +void dm_nvn_probe_deferred_binds(void) { + typedef void (*bind_fn)(NVNcommandBuffer *, NVNshaderStage, int, int, const NVNtextureHandle *); + bind_fn deferred = (bind_fn)nvn_lookup("nvnCommandBufferBindTexturesDeferred"); + if (!deferred) die("deferred probe: nvnCommandBufferBindTexturesDeferred not exported"); + NVNcommandBuffer cb, caller; + uint8_t control[2][256] = {{0}}, commands[4096] = {0}; + pool_t pool = {.mem = commands}; + NVNcommandBuffer *buffers[] = {&cb, &caller}; + for (unsigned i = 0; i < 2; i++) { + h_nvnCommandBufferInitialize(buffers[i], NULL); + h_nvnCommandBufferAddControlMemory(buffers[i], control[i], sizeof control[i]); + h_nvnCommandBufferAddCommandMemory(buffers[i], (NVNmemoryPool *)&pool, 0, sizeof commands); + } + NVNtextureHandle table[2] = {0x10200001, 0x10200002}, immediate = 0x10200003; + h_nvnCommandBufferBeginRecording(&cb); + deferred(&cb, NVN_SHADER_STAGE_FRAGMENT, 4, 2, table); /* the game fills the table later */ + h_nvnCommandBufferBindTextures(&cb, NVN_SHADER_STAGE_FRAGMENT, 6, 1, &immediate); + NVNcommandHandle hd = h_nvnCommandBufferEndRecording(&cb); + h_nvnCommandBufferBeginRecording(&caller); + h_nvnCommandBufferCallCommands(&caller, 1, &hd); + NVNcommandHandle hc = h_nvnCommandBufferEndRecording(&caller); + table[0] = 0x10200011; table[1] = 0x10200012; immediate = 0x10200013; /* written after recording, before submit */ + int negative = getenv("TOTK_PROBE_NEGATIVE") && atoi(getenv("TOTK_PROBE_NEGATIVE")) == 1; /* other numbers are later checks of probe-gfx */ + for (unsigned k = 0; k < 2; k++) { + const uint8_t *p = (const uint8_t *)(uintptr_t)(k ? hc : hd); + if (!negative) resolve_deferred(p, 0); + unsigned seen = 0; + for (;;) { + op_hdr h; memcpy(&h, p, sizeof h); + if (h.op == OP_END) break; + if (h.op == OP_BIND_TEX) { + const p_bind_handles *b = (const p_bind_handles *)(p + 8); + if (b->first == 4 && (b->n != 2 || b->h[0] != 0x10200011 || b->h[1] != 0x10200012)) die("deferred probe: %s bind kept the record-time handles %#llx %#llx", k ? "called" : "direct", (unsigned long long)b->h[0], (unsigned long long)b->h[1]); + if (b->first == 6 && b->h[0] != 0x10200003) die("deferred probe: immediate bind changed after recording"); + seen++; + } + p += h.size; + } + if (seen != 2) die("deferred probe: %u binds in the %s recording, expected 2", seen, k ? "called" : "direct"); + } + h_nvnCommandBufferAddControlMemory(&cb, control[1], sizeof control[1]); + h_nvnCommandBufferAddControlMemory(&caller, control[0], sizeof control[0]); + h_nvnCommandBufferFinalize(&cb); h_nvnCommandBufferFinalize(&caller); +} +NVN(nvnCommandBufferSetStencilValueMask, void, NVNcommandBuffer *c, NVNface f, int m) { *REC(c, OP_STENCIL, p_stencil) = (p_stencil){f, m, 0, 0, 0}; } +NVN(nvnCommandBufferSetStencilMask, void, NVNcommandBuffer *c, NVNface f, int m) { *REC(c, OP_STENCIL, p_stencil) = (p_stencil){f, 0, m, 0, 1}; } +NVN(nvnCommandBufferSetStencilRef, void, NVNcommandBuffer *c, NVNface f, int v) { *REC(c, OP_STENCIL, p_stencil) = (p_stencil){f, 0, 0, v, 2}; } +NVN(nvnCommandBufferSetPolygonOffsetClamp, void, NVNcommandBuffer *c, float fa, float un, float cl) { + *REC(c, OP_POLY_OFFSET, p_poly_offset) = (p_poly_offset){fa, un, cl}; +} +NVN(nvnCommandBufferDrawArrays, void, NVNcommandBuffer *c, NVNdrawPrimitive pr, int first, int n) { *REC(c, OP_DRAW, p_draw) = (p_draw){pr, first, n, 0, 1}; } +NVN(nvnCommandBufferDrawArraysInstanced, void, NVNcommandBuffer *c, NVNdrawPrimitive pr, int first, int n, int bi, int inst) { + if (inst <= 0) return; // NVN: zero instances draw nothing + *REC(c, OP_DRAW, p_draw) = (p_draw){pr, first, n, bi, inst}; +} +NVN(nvnCommandBufferDrawElements, void, NVNcommandBuffer *c, NVNdrawPrimitive pr, NVNindexType it, int n, NVNbufferAddress ib) { + *REC(c, OP_DRAW_IDX, p_draw_idx) = (p_draw_idx){pr, it, n, 0, 0, 1, ib}; hw_fix(CB(c), 48); +} +NVN(nvnCommandBufferDrawElementsBaseVertex, void, NVNcommandBuffer *c, NVNdrawPrimitive pr, NVNindexType it, int n, NVNbufferAddress ib, int bv) { + *REC(c, OP_DRAW_IDX, p_draw_idx) = (p_draw_idx){pr, it, n, bv, 0, 1, ib}; hw_fix(CB(c), 44); +} +NVN(nvnCommandBufferDrawElementsInstanced, void, NVNcommandBuffer *c, NVNdrawPrimitive pr, NVNindexType it, int n, NVNbufferAddress ib, + int bv, int bi, int inst) { + *REC(c, OP_DRAW_IDX, p_draw_idx) = (p_draw_idx){pr, it, n, bv, bi, inst, ib}; hw_fix(CB(c), 44); /* assumed: same 6-parameter draw macro as the base-vertex form (unverified) */ +} +NVN(nvnCommandBufferDispatchCompute, void, NVNcommandBuffer *c, int x, int y, int z) { + // 跳板是尾跳（br），这里的返回地址就是游戏里的调用点 + // The thunk is a tail jump (br), so the return address here is the call site in the game + *REC(c, OP_DISPATCH, p_dispatch) = (p_dispatch){x, y, z, (uint64_t)__builtin_return_address(0)}; +} +NVN(nvnCommandBufferBarrier, void, NVNcommandBuffer *c, int bits) { *(int32_t *)rec(c, OP_BARRIER, 8) = bits; } +NVN(nvnCommandBufferSetTexturePool, void, NVNcommandBuffer *c, const NVNtexturePool *p) { REC(c, OP_SET_TEX_POOL, p_set_pool)->pool = (const dpool_t *)p; } +NVN(nvnCommandBufferSetSamplerPool, void, NVNcommandBuffer *c, const NVNsamplerPool *p) { REC(c, OP_SET_SAMP_POOL, p_set_pool)->pool = (const dpool_t *)p; } + +NVN(nvnCommandBufferCopyBufferToTexture, void, NVNcommandBuffer *c, NVNbufferAddress src, const NVNtexture *t, const NVNtextureView *v, + const NVNcopyRegion *r, int flags) { + p_copy_b2t *o = REC(c, OP_COPY_B2T, p_copy_b2t); o->src = src; o->tex = tex_pin(t); copy_view(&o->view, &o->has_view, v); o->r = *r; o->flags = flags; +} +NVN(nvnCommandBufferCopyTextureToBuffer, void, NVNcommandBuffer *c, const NVNtexture *t, const NVNtextureView *v, const NVNcopyRegion *r, + NVNbufferAddress dst, int flags) { + p_copy_b2t *o = REC(c, OP_COPY_T2B, p_copy_b2t); o->src = dst; o->tex = tex_pin(t); copy_view(&o->view, &o->has_view, v); o->r = *r; o->flags = flags; +} +// 用给定值清贴图区域（浮点 / 无符号整数两版；mask = 通道写掩码） +// Clear a texture region with the given value (float / unsigned integer versions; mask = channel write mask) +NVN(nvnCommandBufferClearTexture, void, NVNcommandBuffer *c, const NVNtexture *t, const NVNtextureView *v, const NVNcopyRegion *r, const float *rgba, int mask) { + p_clear_tex *o = REC(c, OP_CLEAR_TEX, p_clear_tex); o->tex = tex_pin(t); copy_view(&o->view, &o->has_view, v); o->r = *r; o->mask = mask; o->is_uint = 0; memcpy(o->c.f, rgba, 16); +} +NVN(nvnCommandBufferClearTextureui, void, NVNcommandBuffer *c, const NVNtexture *t, const NVNtextureView *v, const NVNcopyRegion *r, const uint32_t *rgba, int mask) { + p_clear_tex *o = REC(c, OP_CLEAR_TEX, p_clear_tex); o->tex = tex_pin(t); copy_view(&o->view, &o->has_view, v); o->r = *r; o->mask = mask; o->is_uint = 1; memcpy(o->c.u, rgba, 16); +} +NVN(nvnCommandBufferCopyTextureToTexture, void, NVNcommandBuffer *c, const NVNtexture *s, const NVNtextureView *sv, const NVNcopyRegion *sr, + const NVNtexture *d, const NVNtextureView *dv, const NVNcopyRegion *dr, int flags) { + p_copy_t2t *o = REC(c, OP_COPY_T2T, p_copy_t2t); o->src = tex_pin(s); o->dst = tex_pin(d); + copy_view(&o->sv, &o->has_sv, sv); copy_view(&o->dv, &o->has_dv, dv); o->sr = *sr; o->dr = *dr; o->flags = flags; +} +NVN(nvnCommandBufferSetCopyRowStride, void, NVNcommandBuffer *c, ptrdiff_t st) { REC(c, OP_COPY_STRIDE, p_copy_stride)->stride = st; } +NVN(nvnCommandBufferSetBlendColor, void, NVNcommandBuffer *c, const float *v) { memcpy(REC(c, OP_BLEND_COLOR, p_blend_color)->rgba, v, 16); } + +// ---------- 状态对象：就地打包（nvn_obj.h） ---------- +// ---------- State objects: packed in place (nvn_obj.h) ---------- +#define ST(T, x) ((T *)(x)) +NVN(nvnBlendStateSetDefaults, void, NVNblendState *s) { *ST(blend_st, s) = (blend_st){0, NVN_BLEND_FUNC_ONE, NVN_BLEND_FUNC_ZERO, NVN_BLEND_FUNC_ONE, NVN_BLEND_FUNC_ZERO, NVN_BLEND_EQUATION_ADD, NVN_BLEND_EQUATION_ADD}; } +NVN(nvnBlendStateSetBlendTarget, void, NVNblendState *s, int t) { ST(blend_st, s)->target = t; } +NVN(nvnBlendStateSetBlendFunc, void, NVNblendState *s, NVNblendFunc sr, NVNblendFunc dr, NVNblendFunc sa, NVNblendFunc da) { + blend_st *o = ST(blend_st, s); o->src_rgb = sr; o->dst_rgb = dr; o->src_a = sa; o->dst_a = da; +} +NVN(nvnBlendStateSetBlendEquation, void, NVNblendState *s, NVNblendEquation er, NVNblendEquation ea) { ST(blend_st, s)->eq_rgb = er; ST(blend_st, s)->eq_a = ea; } +NVN(nvnColorStateSetDefaults, void, NVNcolorState *s) { *ST(color_st, s) = (color_st){0, NVN_LOGIC_OP_COPY, NVN_ALPHA_FUNC_ALWAYS}; } +NVN(nvnColorStateSetBlendEnable, void, NVNcolorState *s, int i, uint32_t en) { + color_st *o = ST(color_st, s); if (i < 0 || i > 7) return; + o->blend_en = B(en) ? (o->blend_en | 1u << i) : (o->blend_en & ~(1u << i)); +} +NVN(nvnChannelMaskStateSetDefaults, void, NVNchannelMaskState *s) { ST(chmask_st, s)->mask = 0xffffffffu; } +NVN(nvnChannelMaskStateSetChannelMask, void, NVNchannelMaskState *s, int i, uint32_t r, uint32_t g, uint32_t b, uint32_t a) { + if (i < 0 || i > 7) return; + uint32_t m = (B(r) ? 1 : 0) | (B(g) ? 2 : 0) | (B(b) ? 4 : 0) | (B(a) ? 8 : 0); + chmask_st *o = ST(chmask_st, s); o->mask = (o->mask & ~(0xfu << (4 * i))) | m << (4 * i); +} +NVN(nvnMultisampleStateSetDefaults, void, NVNmultisampleState *s) { memset(s, 0, sizeof(ms_st)); ST(ms_st, s)->enable = 1; } +NVN(nvnMultisampleStateSetMultisampleEnable, void, NVNmultisampleState *s, uint32_t e) { ST(ms_st, s)->enable = B(e) != 0; } +NVN(nvnMultisampleStateSetSamples, void, NVNmultisampleState *s, int n) { ST(ms_st, s)->samples = (uint8_t)n; } +NVN(nvnMultisampleStateSetAlphaToCoverageEnable, void, NVNmultisampleState *s, uint32_t e) { ST(ms_st, s)->a2c = B(e) != 0; } +NVN(nvnMultisampleStateSetSampleLocationsEnable, void, NVNmultisampleState *s, uint32_t e) { ST(ms_st, s)->sl_en = B(e) != 0; } +NVN(nvnMultisampleStateSetSampleLocations, void, NVNmultisampleState *s, int start, int n, const float *v) { (void)s; (void)start; (void)n; (void)v; } +NVN(nvnPolygonStateSetDefaults, void, NVNpolygonState *s) { *ST(poly_st, s) = (poly_st){NVN_FACE_NONE, NVN_FRONT_FACE_CCW, NVN_POLYGON_MODE_FILL, 0}; } +NVN(nvnPolygonStateSetCullFace, void, NVNpolygonState *s, NVNface f) { ST(poly_st, s)->cull = f; } +NVN(nvnPolygonStateSetFrontFace, void, NVNpolygonState *s, NVNfrontFace f) { ST(poly_st, s)->front = f; } +NVN(nvnPolygonStateSetPolygonMode, void, NVNpolygonState *s, NVNpolygonMode m) { ST(poly_st, s)->mode = m; } +NVN(nvnPolygonStateSetPolygonOffsetEnables, void, NVNpolygonState *s, int e) { ST(poly_st, s)->offset_en = e; } +NVN(nvnDepthStencilStateSetDefaults, void, NVNdepthStencilState *s) { + *ST(ds_st, s) = (ds_st){0, 1, NVN_DEPTH_FUNC_LESS, 0, NVN_STENCIL_FUNC_ALWAYS, NVN_STENCIL_FUNC_ALWAYS, + NVN_STENCIL_OP_KEEP, NVN_STENCIL_OP_KEEP, NVN_STENCIL_OP_KEEP, NVN_STENCIL_OP_KEEP, NVN_STENCIL_OP_KEEP, NVN_STENCIL_OP_KEEP}; +} +NVN(nvnDepthStencilStateSetDepthTestEnable, void, NVNdepthStencilState *s, uint32_t e) { ST(ds_st, s)->dt = B(e) != 0; } +NVN(nvnDepthStencilStateSetDepthWriteEnable, void, NVNdepthStencilState *s, uint32_t e) { ST(ds_st, s)->dw = B(e) != 0; } +NVN(nvnDepthStencilStateSetDepthFunc, void, NVNdepthStencilState *s, NVNdepthFunc f) { ST(ds_st, s)->df = f; } +NVN(nvnDepthStencilStateSetStencilTestEnable, void, NVNdepthStencilState *s, uint32_t e) { ST(ds_st, s)->st = B(e) != 0; } +NVN(nvnDepthStencilStateSetStencilFunc, void, NVNdepthStencilState *s, NVNface f, NVNstencilFunc fn) { + ds_st *o = ST(ds_st, s); if (f & NVN_FACE_FRONT) o->sf_f = fn; if (f & NVN_FACE_BACK) o->sf_b = fn; +} +NVN(nvnDepthStencilStateSetStencilOp, void, NVNdepthStencilState *s, NVNface f, NVNstencilOp fail, NVNstencilOp zfail, NVNstencilOp zpass) { + ds_st *o = ST(ds_st, s); + if (f & NVN_FACE_FRONT) { o->fail_f = fail; o->zfail_f = zfail; o->zpass_f = zpass; } + if (f & NVN_FACE_BACK) { o->fail_b = fail; o->zfail_b = zfail; o->zpass_b = zpass; } +} +NVN(nvnVertexAttribStateSetDefaults, void, NVNvertexAttribState *s) { *ST(va_st, s) = (va_st){0, 0, 0}; } +NVN(nvnVertexAttribStateSetFormat, void, NVNvertexAttribState *s, NVNformat f, ptrdiff_t off) { + if (off < 0 || off >= (1 << 19)) LOG("【红】顶点属性偏移 %td 超出打包范围", off); + ST(va_st, s)->fmt = f; ST(va_st, s)->offset = (uint32_t)off; +} +NVN(nvnVertexAttribStateSetStreamIndex, void, NVNvertexAttribState *s, int i) { ST(va_st, s)->stream = i; } +NVN(nvnVertexStreamStateSetDefaults, void, NVNvertexStreamState *s) { *ST(vs_st, s) = (vs_st){0, 0}; } +NVN(nvnVertexStreamStateSetStride, void, NVNvertexStreamState *s, ptrdiff_t st) { ST(vs_st, s)->stride = (uint32_t)st; } +NVN(nvnVertexStreamStateSetDivisor, void, NVNvertexStreamState *s, int d) { ST(vs_st, s)->divisor = d; } + +static _Atomic uint64_t g_sub_wall_ns, g_sub_cpu_ns, g_sub_instr; // 提交（不含出图）的墙钟 / 本线程 CPU 时间 / 本线程退休指令数（逐帧取走） | Wall clock / this thread's CPU time / this thread's retired instruction count for submit (excluding present) (taken away per frame) +static _Atomic uint64_t g_cum_sub_instr, g_cum_sub_cpu_ns; // 同上累计（每 300 帧汇总「翻译」行取差，真机经日志口读：每次绘制指令数 / 每帧提交 CPU） | Cumulative versions of the above (the 「翻译」(translation) summary row every 300 frames takes the difference; on a real device read through the log port: instructions per draw / submit CPU per frame) +extern int thread_selfcounts(int type, void *buf, size_t nbytes); // 本线程累计 {指令, 周期}（libSystem 私有接口，Mac / iOS 都有） | This thread's cumulative {instructions, cycles} (libSystem private interface, present on both Mac and iOS) +static uint64_t thr_instr(void) { uint64_t c[2] = {0, 0}; thread_selfcounts(1, c, sizeof c); return c[0]; } +// 提交路径的补充读数（真机 Metal 4 指令数 −18% 而提交 CPU 时间 +49%，要分清是内核态、执行效率还是被降频）： +// 本线程周期数（同一接口的第二项）、内核态 CPU 时间（THREAD_BASIC_INFO）、墙钟 +// Supplementary readings for the submit path (on a real device Metal 4 instruction count is -18% while submit CPU time is +49%; need to tell apart kernel mode, execution efficiency, or frequency throttling): +// this thread's cycle count (the second item of the same interface), kernel-mode CPU time (THREAD_BASIC_INFO), wall clock +static void thr_counts(uint64_t *instr, uint64_t *cyc) { uint64_t c[2] = {0, 0}; thread_selfcounts(1, c, sizeof c); *instr = c[0]; *cyc = c[1]; } +static uint64_t thr_sys_ns(void) { + thread_basic_info_data_t bi; mach_msg_type_number_t n = THREAD_BASIC_INFO_COUNT; mach_port_t me = mach_thread_self(); uint64_t r = 0; + if (thread_info(me, THREAD_BASIC_INFO, (thread_info_t)&bi, &n) == KERN_SUCCESS) r = (uint64_t)bi.system_time.seconds * 1000000000ull + (uint64_t)bi.system_time.microseconds * 1000ull; + mach_port_deallocate(mach_task_self(), me); return r; +} +static _Atomic uint64_t g_cum_sub_cyc, g_cum_sub_sys_ns, g_cum_sub_wall_ns; +static int frame_rec(void); +static _Atomic uint64_t g_nvn_ns; // 本帧在提交 / 出图里花的宿主时间（回放编码 + 出图），计时埋点用 | Host time spent in submit / present this frame (replay encoding + present), used by the timing instrumentation + +// ---------- 生命周期：dm_gfx_suspend（iOS 进后台前调用；等效 Switch 挂起） ---------- +// on=1：之后的提交 / 出图阻塞到 on=0；返回前等正在进行的提交 / 出图调用退出、并等所有在途 GPU 工作完成（iOS 要求后台无 GPU 工作）。 +// ---------- Lifecycle: dm_gfx_suspend (called before iOS goes to the background; equivalent to Switch suspend) ---------- +// on=1: subsequent submits / presents block until on=0; before returning, wait for in-progress submit / present calls to exit and for all in-flight GPU work to complete (iOS requires no GPU work in the background). +static pthread_mutex_t g_susp_mu = PTHREAD_MUTEX_INITIALIZER; +static pthread_cond_t g_susp_cv = PTHREAD_COND_INITIALIZER; +static int g_susp, g_inflight_api; +static void gfx_enter(void) { pthread_mutex_lock(&g_susp_mu); while (g_susp) pthread_cond_wait(&g_susp_cv, &g_susp_mu); g_inflight_api++; pthread_mutex_unlock(&g_susp_mu); } +static void gfx_leave(void) { pthread_mutex_lock(&g_susp_mu); g_inflight_api--; pthread_cond_broadcast(&g_susp_cv); pthread_mutex_unlock(&g_susp_mu); } +void dm_gfx_suspend(int on) { + pthread_mutex_lock(&g_susp_mu); + g_susp = on != 0; + if (on) while (g_inflight_api) pthread_cond_wait(&g_susp_cv, &g_susp_mu); + else pthread_cond_broadcast(&g_susp_cv); + pthread_mutex_unlock(&g_susp_mu); + if (on) { x_drain(); mtl_drain(); LOG("[nvn] 图形挂起：在途 GPU 工作已清空"); } else LOG("[nvn] 图形恢复"); +} +_Atomic int dm_submit_busy; // TOTK diagnostic: submissions being replayed (runtime/gfx_bridge.c write watch) +// ---------- TOTK: translation thread (TOTK_ASYNC_REPLAY=1) ---------- +// Submits and presents are queued in order to one translation thread; the game thread records them and returns, so game +// logic of the next frame overlaps the replay + Metal encoding of this one (the design of D2R Mobile b94670c, adapted). +// - Fences: FenceSync records the number of submits enqueued (bit 2 of `fenced` marks such a token). SyncWait first waits +// until the translation thread has run that submit, then for the Metal fence that submit ended with. Waiting on a fence +// therefore means "translated and GPU done", so when the game reuses command / control memory or rewrites objects after +// a fence, the translation thread has long finished reading them. +// - Presents are jobs too (after the submit that drew the frame); the acquire of a swapchain image first waits for the +// job of that image's last present, then for its present ticket as before. The replay frame number travels in the job. +// - Operations that destroy or read Metal state (pool / texture / program finalize, ReadTexels / WriteTexels, QueueFinish, +// suspend) wait until the queue is empty first (x_drain), so they never overtake a queued replay that still uses them. +// TOTK_ASYNC_REPLAY=0 runs everything on the calling thread as before. [Windows port] On by default since 2026-10-24: with +// the Vulkan replay on the game's thread the Great Sky Island ran at ~95 ms a frame, with this thread at ~58 ms. +enum { XJ_SUBMIT = 1, XJ_PRESENT, XJ_TEXREL }; +typedef struct { int kind, n; uint64_t *h; void *ucap; win_t *win; int idx; double period; int crop[4]; uint64_t frame, sub; tex_t tex; } xjob; // tex: copy of a finalized texture (XJ_TEXREL); ucap: uniforms captured at the submit call +#define XQ 64 +#define XF 4096 +static xjob g_xq[XQ]; +static uint64_t g_xq_in, g_xq_done, g_xsub_in, g_xsub_done, g_xsub_fence[XF]; +static pthread_mutex_t g_xq_mu = PTHREAD_MUTEX_INITIALIZER; +static pthread_cond_t g_xq_cv = PTHREAD_COND_INITIALIZER; +static win_t *g_xp_win; static uint64_t g_xp_job[4]; // job number of each swapchain image's last present (TOTK has one window) +static _Atomic uint64_t g_x_drains, g_x_drain_ns; +static int async_replay(void) { + static int v = -1; + if (v < 0) { const char *e = getenv("TOTK_ASYNC_REPLAY"); v = !(e && atoi(e) <= 0); if (v) LOG("[nvn] 翻译线程：提交与出图排队交给独立线程回放（TOTK_ASYNC_REPLAY=0 关）"); } + return v; +} +static void submit_exec(int n, const uint64_t *h, void *ucap); +static void present_exec(win_t *o, int idx, double period, const int crop[4], uint64_t frame); +static void *x_thread(void *unused) { + (void)unused; pthread_setname_np("totk-translate"); + { void plat_thread_boost(void); plat_thread_boost(); } // [Windows port] the frame waits for this thread + for (;;) { + pthread_mutex_lock(&g_xq_mu); + while (g_xq_done == g_xq_in) pthread_cond_wait(&g_xq_cv, &g_xq_mu); + xjob j = g_xq[g_xq_done % XQ]; + pthread_mutex_unlock(&g_xq_mu); + if (j.kind == XJ_SUBMIT) { submit_exec(j.n, j.h, j.ucap); free(j.h); } + else if (j.kind == XJ_TEXREL) { void live_del_x(const tex_t *o); live_del_x(&j.tex); mtl_tex_release(&j.tex); } + else { + // Real-time like the game's render thread that used to replay (set at acquire): otherwise the game's ~50 + // worker threads preempt the replay (measured: replay 6.6 -> 13 ms, 10 fps). Half the period is declared. + static uint64_t rt; uint64_t per = (uint64_t)(j.period * 1e9); + if (per && rt != per) { rt = per; void set_time_constraint_comp_x(uint64_t, uint64_t); set_time_constraint_comp_x(per, per / 2); } + present_exec(j.win, j.idx, j.period, j.crop, j.frame); + } + pthread_mutex_lock(&g_xq_mu); + if (j.kind == XJ_SUBMIT) { g_xsub_fence[j.sub % XF] = mtl_last_fence(); g_xsub_done = j.sub; } + g_xq_done++; + pthread_cond_broadcast(&g_xq_cv); + pthread_mutex_unlock(&g_xq_mu); + } + return NULL; +} +static uint64_t x_push(xjob *j) { // returns the job number (1-based) + static pthread_once_t once = PTHREAD_ONCE_INIT; + void x_start(void); + pthread_once(&once, x_start); + pthread_mutex_lock(&g_xq_mu); + while (g_xq_in - g_xq_done >= XQ) pthread_cond_wait(&g_xq_cv, &g_xq_mu); + if (j->kind == XJ_SUBMIT) j->sub = ++g_xsub_in; + g_xq[g_xq_in % XQ] = *j; + uint64_t no = ++g_xq_in; + pthread_cond_broadcast(&g_xq_cv); + pthread_mutex_unlock(&g_xq_mu); + return no; +} +void x_texrel(const tex_t *o) { xjob j = {.kind = XJ_TEXREL}; j.tex = *o; x_push(&j); } +void x_start(void) { + pthread_t t; pthread_attr_t a; pthread_attr_init(&a); pthread_attr_setstacksize(&a, 8u << 20); + pthread_attr_set_qos_class_np(&a, QOS_CLASS_USER_INTERACTIVE, 0); + if (pthread_create(&t, &a, x_thread, NULL)) die("translation thread"); + pthread_attr_destroy(&a); pthread_detach(t); +} +static void x_wait_job(uint64_t job) { + if (!job) return; + pthread_mutex_lock(&g_xq_mu); while (g_xq_done < job) pthread_cond_wait(&g_xq_cv, &g_xq_mu); pthread_mutex_unlock(&g_xq_mu); +} +static void x_drain(void) { // everything queued so far has run (no-op when synchronous or idle) + if (!async_replay()) return; + NVN_BLOCKING(); // a guest thread gives its virtual core back while it waits (TOTK has three) + pthread_mutex_lock(&g_xq_mu); uint64_t want = g_xq_in, have = g_xq_done; pthread_mutex_unlock(&g_xq_mu); + if (have >= want) return; + uint64_t t0 = now_ns(); x_wait_job(want); + atomic_fetch_add(&g_x_drains, 1); atomic_fetch_add(&g_x_drain_ns, now_ns() - t0); +} +void nvn_x_drain(void) { x_drain(); } // for the graphics layer (pool invalidate write-back) +// Fence token for FenceSync: submits enqueued so far (async) or the Metal fence (synchronous). +static uint64_t fence_now(int *token) { + if (!async_replay()) { *token = 0; return mtl_last_fence(); } + pthread_mutex_lock(&g_xq_mu); uint64_t k = g_xsub_in; pthread_mutex_unlock(&g_xq_mu); *token = 1; return k; +} +// Waits for a token: the submit has been translated, then the Metal fence it ended with. 1 = reached. +static int fence_wait(uint64_t k, uint64_t timeout_ns) { + uint64_t t0 = now_ns(); + pthread_mutex_lock(&g_xq_mu); + if (g_xsub_done < k) { + struct timespec dl; clock_gettime(CLOCK_REALTIME, &dl); + uint64_t ns = (uint64_t)dl.tv_nsec + (timeout_ns > 3600000000000ull ? 3600000000000ull : timeout_ns); + dl.tv_sec += ns / 1000000000ull; dl.tv_nsec = ns % 1000000000ull; + while (g_xsub_done < k) if (pthread_cond_timedwait(&g_xq_cv, &g_xq_mu, &dl)) break; + } + int ok = g_xsub_done >= k; + uint64_t f = !k ? 0 : g_xsub_done - k < XF ? g_xsub_fence[k % XF] : mtl_last_fence(); // older entries were overwritten: a later fence is conservative + pthread_mutex_unlock(&g_xq_mu); + if (!ok) return 0; + uint64_t used = now_ns() - t0; + return mtl_wait(f, timeout_ns > used ? timeout_ns - used : 0); +} + +NVN(nvnQueueSubmitCommands, void, NVNqueue *q, int n, const NVNcommandHandle *h) { + NVN_BLOCKING(); + totk_tl(TL_SUB_IN, (uint64_t)n, 0); + { uint64_t now = __atomic_add_fetch(&g_submits, 1, __ATOMIC_RELAXED); for (int i = 0; i < n; i++) copy_touch(h[i], now); } + if (totk_life_on()) { int k = 0; for (int i = 0; i < n; i++) k += totk_life_handle_has(h[i]); totk_life_log("submit n=%d followed=%d", n, k); } + ((queue_t *)q)->submits++; + /* TOTK: what the console reads at the submit call is read here, on the game thread, before the submission may be handed + to the translation thread: the handles of deferred binds (the game fills its handle tables for the next frame as soon + as this call returns) and, with the translation thread, the bound uniform bytes (other game threads rebuild the + constant pools right after the submit; see mtl_sub_capture_ubos). */ + if (atomic_load(&g_def_on) < 0) { const char *e = getenv("TOTK_DEFERRED_BINDS"); atomic_store(&g_def_on, !(e && !strcmp(e, "0"))); LOG("[nvn] 延迟绑定：%s", atomic_load(&g_def_on) ? "提交时读句柄" : "录制时读句柄（TOTK_DEFERRED_BINDS=0，旧行为）"); } + if (atomic_load(&g_def_on)) { + uint64_t r0 = now_ns(); + for (int i = 0; i < n; i++) resolve_deferred((const uint8_t *)h[i], 0); + g_def_ns += now_ns() - r0; + if (((queue_t *)q)->submits % 600 == 0) LOG("[nvn] 延迟绑定 至提交 %llu：提交时读句柄 %llu 次，其中与录制时不同 %llu 次；每次提交读句柄平均 %.3f ms", (unsigned long long)((queue_t *)q)->submits, (unsigned long long)g_def_n, (unsigned long long)g_def_changed, g_def_ns / 1e6 / (double)((queue_t *)q)->submits); + } + if (async_replay()) { + xjob j = {.kind = XJ_SUBMIT, .n = n, .h = malloc(sizeof(uint64_t) * (n > 0 ? n : 1))}; + if (!j.h) die("submit job"); + memcpy(j.h, h, sizeof(uint64_t) * (size_t)n); + j.ucap = mtl_sub_capture_ubos((const uint64_t *)h, n); + x_push(&j); + totk_tl(TL_SUB_OUT, 1, 0); + return; + } + submit_exec(n, (const uint64_t *)h, NULL); + totk_tl(TL_SUB_OUT, 0, 0); +} +static void submit_exec(int n, const uint64_t *h, void *ucap) { + gfx_enter(); + totk_tl(TL_REP_BEGIN, 0, 0); + pin_step(__atomic_load_n(&g_submits, __ATOMIC_RELAXED)); /* TOTK: free texture copies no recording refers to any more */ + if (cmdlog_env()) for (int i = 0; i < n; i++) { + unsigned ops = 0, draws = 0, calls = 0; const uint8_t *p = (const uint8_t *)h[i]; + for (;;) { op_hdr hh; memcpy(&hh, p, sizeof hh); ops++; if (hh.op == OP_DRAW || hh.op == OP_DRAW_IDX) draws++; if (hh.op == OP_CALL) calls++; if (hh.op == OP_END || hh.size < 8 || ops > 1000000) break; p += hh.size; } + LOG("[cmd] Submit %d/%d handle=%#llx ops=%u draws=%u calls=%u", i, n, (unsigned long long)h[i], ops, draws, calls); } + { static int shape = -1; // TOTK_SUBMIT_SHAPE=1 (development): command handles per big submission and their draws + if (shape < 0) shape = getenv("TOTK_SUBMIT_SHAPE") != NULL; + static unsigned said; + if (shape) { + unsigned tot = 0, big = 0, mx = 0, d[64], calld[256], ncall = 0; int m = n < 64 ? n : 64; + for (int i = 0; i < n; i++) { // draws in each handle, through its calls (one level) + unsigned draws = 0; const uint8_t *p = (const uint8_t *)h[i]; + for (unsigned ops = 0; ops < 2000000; ops++) { + op_hdr hh; memcpy(&hh, p, sizeof hh); + if (hh.op == OP_DRAW || hh.op == OP_DRAW_IDX || hh.op == OP_DISPATCH) draws++; + if (hh.op == OP_CALL) { const uint8_t *t; memcpy(&t, p + 8, 8); unsigned cd = 0; + for (unsigned k = 0; k < 2000000; k++) { op_hdr h2; memcpy(&h2, t, sizeof h2); + if (h2.op == OP_DRAW || h2.op == OP_DRAW_IDX || h2.op == OP_DISPATCH) { draws++; cd++; } + if (h2.op == OP_END || h2.size < 8) { if (ncall < 256) calld[ncall] = cd; ncall++; break; } + if (h2.op == OP_JUMP) { memcpy(&t, t + 8, 8); continue; } + t += h2.size; } } + if (hh.op == OP_END || hh.size < 8) break; + if (hh.op == OP_JUMP) { memcpy(&p, p + 8, 8); continue; } + p += hh.size; + } + if (i < 64) d[i] = draws; + tot += draws; if (draws > mx) mx = draws; if (draws >= 100) big++; + } + if (tot > 1000 && said++ % 150 == 0) { + char line[700]; int ln = 0; + for (int i = 0; i < m && ln < 600; i++) ln += snprintf(line + ln, sizeof line - (size_t)ln, " %u", d[i]); + LOG("[shape] submission: %d handles, draws+dispatches %u, largest %u, handles with 100+: %u; per handle:%s", n, tot, mx, big, line); + unsigned inc = 0, cbig = 0, cmx = 0; ln = 0; + for (unsigned i = 0; i < ncall && i < 256; i++) { inc += calld[i]; if (calld[i] > cmx) cmx = calld[i]; if (calld[i] >= 100) cbig++; + if (ln < 600) ln += snprintf(line + ln, sizeof line - (size_t)ln, " %u", calld[i]); } + LOG("[shape] calls %u: draws in calls %u (outside %u), largest %u, calls with 100+: %u; per call:%s", ncall, inc, tot - inc, cmx, cbig, line); + } + } } + if (getenv("TOTK_CHAIN_LOG")) { static int k; if (k++ < 20) guest_chain("Submit"); } + uint64_t t0 = now_ns(), c0 = clock_gettime_nsec_np(CLOCK_THREAD_CPUTIME_ID), i0, y0, s0 = thr_sys_ns(); thr_counts(&i0, &y0); + /* TOTK diagnostic (TOTK_WATCH_ADDR): submissions in progress, and their start / end times, to line up guest writes with replay */ + atomic_fetch_add(&dm_submit_busy, 1); + { static int wl = -1; if (wl < 0) wl = getenv("TOTK_WATCH_ADDR") != NULL; if (wl) LOG("[sub] F%llu start t_us=%llu", (unsigned long long)g_frame_pub, (unsigned long long)(clock_gettime_nsec_np(CLOCK_UPTIME_RAW) / 1000)); } + mtl_sub_begin(); + { void vkb_seq_set(uint64_t seq); vkb_seq_set(0); } // [Windows port] positions in this submission (captured uniforms) + mtl_sub_snapshot_ubos_cap((const uint64_t *)h, n, ucap); // TOTK: uniforms as of the submit call (ucap: bytes captured there) + mtl_sub_index((const uint64_t *)h, n); + { // [Windows port] several recording threads (src/gfx/backend_vk.c vkb_mt_submit), else this one + int vkb_mt_on(void); void vkb_mt_submit(const uint64_t *h, int n); + if (vkb_mt_on()) vkb_mt_submit((const uint64_t *)h, n); + else { + for (int i = 0; i < n; i++) { mtl_exec_rest((const uint64_t *)h + i + 1, n - i - 1); mtl_exec((const uint8_t *)h[i]); } + mtl_exec_rest(NULL, 0); + mtl_sub_commit(); + } + } + atomic_fetch_sub(&dm_submit_busy, 1); + { static int wl = -1; if (wl < 0) wl = getenv("TOTK_WATCH_ADDR") != NULL; if (wl) LOG("[sub] F%llu end t_us=%llu", (unsigned long long)g_frame_pub, (unsigned long long)(clock_gettime_nsec_np(CLOCK_UPTIME_RAW) / 1000)); } + uint64_t dw = now_ns() - t0; + atomic_fetch_add(&g_nvn_ns, dw); atomic_fetch_add(&g_sub_wall_ns, dw); + uint64_t i1, y1; thr_counts(&i1, &y1); uint64_t dc = clock_gettime_nsec_np(CLOCK_THREAD_CPUTIME_ID) - c0, di = i1 - i0; + atomic_fetch_add(&g_cum_sub_cyc, y1 - y0); atomic_fetch_add(&g_cum_sub_sys_ns, thr_sys_ns() - s0); atomic_fetch_add(&g_cum_sub_wall_ns, dw); + atomic_fetch_add(&g_sub_cpu_ns, dc); atomic_fetch_add(&g_cum_sub_cpu_ns, dc); + atomic_fetch_add(&g_sub_instr, di); atomic_fetch_add(&g_cum_sub_instr, di); + totk_tl(TL_REP_END, 0, 0); + gfx_leave(); +} + +// 只影响 GPU 状态 / 只在 GPU 上执行的命令：空设备只计数（跳板已计） +// Commands that only affect GPU state / only execute on the GPU: the null device only counts them (the thunk has already counted) +static void nvn_nop(void) {} +// TOTK: entries that would silently drop GPU work (indirect draws/dispatch, downsample, depth bounds test) were removed +// so a real call stops at the unimplemented stub. WaitSync/WaitEvent rely on in-order replay on the single Metal queue. +static const char *const g_nops[] = { + "nvnCommandBufferSetPatchSize", + "nvnCommandBufferWaitSync", + "nvnCommandBufferSetShaderScratchMemory", + "nvnCommandBufferWaitEvent", + "nvnWindowFinalize", +}; +// [port] The registry hands these names a guest address whose host function does nothing (nvn_registry.c). +int nvn_nop_name(const char *name) { + for (unsigned k = 0; k < sizeof g_nops / sizeof *g_nops; k++) if (!strcmp(g_nops[k], name)) return 1; + return 0; +} +__attribute__((constructor)) static void nvn_nops_register(void) { + // 空实现在槽表初始化后覆盖未实现桩；名字写错（不在 main 名字表里）启动即报红中止 + // Empty implementations overwrite the unimplemented stubs after the slot table is initialized; a misspelled name (not in main's name table) reports a failure and aborts at startup + nvn_init_slots(); + for (unsigned k = 0; k < sizeof g_nops / sizeof *g_nops; k++) { + unsigned i = 0; + while (i < NVN_NFUNC && strcmp(nvn_names[i], g_nops[k])) i++; + if (i == NVN_NFUNC) die("NVN 空实现名 %s 不在 main 名字表里", g_nops[k]); + if (nvn_slot[i].fn != (void *)nvn_unimpl_entry) die("NVN %s 既有实现又列为空实现", g_nops[k]); + nvn_slot[i].fn = (void *)nvn_nop; + } + unsigned nnop = sizeof g_nops / sizeof *g_nops; + // DM_NVN_PROF=1：已实现的函数（未实现桩与出图除外：出图含节拍等待，单独计）套计时包装，用于拆「翻译层录制期」耗时 + // DM_NVN_PROF=1: implemented functions (except unimplemented stubs and present: present includes the display-link wait and is counted separately) get a timing wrapper, used to break down the time spent in the "translation layer recording phase" + if (getenv("DM_NVN_PROF")) + for (unsigned i = 0; i < NVN_NFUNC; i++) + if (nvn_slot[i].fn != (void *)nvn_unimpl_entry && strcmp(nvn_names[i], "nvnQueuePresentTexture")) { nvn_real[i] = nvn_slot[i].fn; nvn_slot[i].fn = (void *)nvn_prof_entry; } + LOG("[nvn] 空设备：名字 %u，已实现 %u（语义 %u + 空实现 %u），其余 %u 个接报红桩", + NVN_NFUNC, g_nsem + nnop, g_nsem, nnop, NVN_NFUNC - g_nsem - nnop); +} + +// ---------- 窗口 / 出图 ---------- +// ---------- Window / present ---------- +#define WB(b) ((win_builder_t *)(b)) +NVN(nvnWindowBuilderSetDefaults, void, NVNwindowBuilder *b) { NVNdevice *dev = WB(b)->dev; memset(b, 0, sizeof(win_builder_t)); WB(b)->dev = dev; WB(b)->interval = 1; } +NVN(nvnWindowBuilderSetDevice, void, NVNwindowBuilder *b, NVNdevice *d) { WB(b)->dev = d; } +NVN(nvnWindowBuilderSetNativeWindow, void, NVNwindowBuilder *b, NVNnativeWindow w) { WB(b)->native = w; } +NVN(nvnWindowBuilderSetTextures, void, NVNwindowBuilder *b, int n, NVNtexture *const *t) { + if (n < 0 || n > 4) { LOG("【红】WindowBuilderSetTextures 数量 %d", n); n = n < 0 ? 0 : 4; } + WB(b)->ntex = n; for (int i = 0; i < n; i++) WB(b)->tex[i] = t[i]; +} +// TOTK sets two active images. Preserve the requested acquisition limit. +NVN_NOPROTO(nvnWindowBuilderSetNumActiveTextures, void, NVNwindowBuilder *b, int n) { if(n<2||n>4)die("unsupported active texture count %d",n);WB(b)->active=n; } +NVN_NOPROTO(nvnWindowBuilderGetNumActiveTextures, int, const NVNwindowBuilder *b) { const win_builder_t *v=(const win_builder_t *)b;return v->active?v->active:v->ntex; } +NVN_NOPROTO(nvnWindowSetNumActiveTextures, void, NVNwindow *w, int n) { win_t *v=(win_t *)w;if(n<2||n>v->ntex)die("invalid active window texture count %d",n);v->active=n; } +NVN_NOPROTO(nvnWindowGetNumActiveTextures, int, const NVNwindow *w) { const win_t *v=(const win_t *)w;return v->active?v->active:v->ntex; } +NVN(nvnWindowBuilderSetPresentInterval, void, NVNwindowBuilder *b, int v) { WB(b)->interval = v; } +NVN(nvnWindowInitialize, NVNboolean, NVNwindow *w, const NVNwindowBuilder *b) { + const win_builder_t *s = (const win_builder_t *)b; win_t *o = (win_t *)w; + memset(o, 0, sizeof *o); o->magic = MAGIC('W', 'I', 'N', 'D'); o->native = s->native; o->ntex = s->ntex; o->interval = s->interval; o->active=s->active?s->active:s->ntex; + if(o->ntex<2||o->ntex>4||o->active<2||o->active>o->ntex)return 0; + memcpy(o->tex, s->tex, sizeof o->tex); + for (int i = 0; i < o->ntex; i++) if (o->tex[i]) mtl_xv_window_tex(CTX(o->tex[i])->serial); + LOG("[nvn] WindowInitialize 交换链 %d 张 间隔 %d 首张 %dx%d 格式 %#x", o->ntex, o->interval, + o->ntex ? (int)CTX(o->tex[0])->d.width : 0, o->ntex ? (int)CTX(o->tex[0])->d.height : 0, o->ntex ? CTX(o->tex[0])->d.format : 0); + return 1; +} +/* TOTK: the game picks its present interval (main+0x8f6bec) from its own frame work time: system object +0x118 = ticks + (19.2 MHz) from the end of the previous frame function to after its update / draw calls (main+0x8f6a54), present not + included; interval = clamp(trunc(work * 60 + bias) + 1, min, max), raised after +0xc frames over, lowered only after a + cooldown. The controller calls nvnWindowSetPresentInterval with the system object saved in its frame (caller x19 at + x29 + 0x38), so it is picked up here, and every frame's +0x118 is reported with the controller state (game_work_tick). */ +static int dcmp(const void *a, const void *b); +static int vm_read8(uint64_t a, uint64_t *out) { vm_size_t got = 0; return a && !(a & 7) && vm_read_overwrite(mach_task_self(), (vm_address_t)a, 8, (vm_address_t)out, &got) == KERN_SUCCESS && got == 8; } +extern uintptr_t runtime_frame_controller_return_pc(void); +static void find_game_sys(void) { + uintptr_t controller_return = runtime_frame_controller_return_pc(); + uint64_t fp = (uint64_t)(uintptr_t)__builtin_frame_address(0); + for (int i = 0; i < 24 && fp && !(fp & 7); i++) { + uint64_t f0, f1; if (!vm_read8(fp, &f0) || !vm_read8(fp + 8, &f1)) return; + if (f1 == controller_return) { uint64_t sys = 0; if (vm_read8(f0 + 0x38, &sys) && sys) { g_game_sys = sys; LOG("[nvn] 游戏系统对象 %#llx（出图间隔控制器的帧里取得）", (unsigned long long)sys); } return; } + if (f0 == fp) return; fp = f0; + } +} +static void game_work_tick(uint64_t frame) { + if (!g_game_sys) return; + static double win[300]; static int n; static uint64_t over33; + int64_t t = *(volatile int64_t *)(uintptr_t)(g_game_sys + 0x118); double ms = t / 19200.0; + if (ms >= 0 && ms < 10000 && n < 300) win[n++] = ms; // loading frames leave the frame start unset (absolute tick) + if (ms > 33.333 && ms < 10000) over33++; + if (frame % 300) return; + if (n < 30) { LOG("[nvn] 游戏帧工作时间（+0x118）至帧 %llu：有效值不足（%d 个），原始 %lld", (unsigned long long)frame, n, (long long)t); n = 0; return; } + double v[300]; memcpy(v, win, sizeof(double) * n); qsort(v, n, sizeof *v, dcmp); + const int32_t *c = (const int32_t *)(uintptr_t)(g_game_sys + 0x248); + extern uint64_t nvn_frame_ctl(void); + if (nvn_frame_ctl()) { // [Windows port] 1.4.3: the controller's own state (nvnWindowSetPresentInterval) + const int32_t *k = (const int32_t *)(uintptr_t)nvn_frame_ctl(); + LOG("[nvn] 游戏帧工作时间（+0x118）至帧 %llu：p50 %.2f / p90 %.2f / 最大 %.2f ms（%d 帧，超 33.3 ms 累计 %llu 帧）；控制器 区间 %d..%d 当前 %d", + (unsigned long long)frame, v[n / 2], v[n * 9 / 10], v[n - 1], n, (unsigned long long)over33, k[0], k[1], k[0x40 / 4]); + n = 0; + return; + } + LOG("[nvn] 游戏帧工作时间（+0x118）至帧 %llu：p50 %.2f / p90 %.2f / 最大 %.2f ms（%d 帧，超 33.3 ms 累计 %llu 帧）；控制器 区间 %d..%d 偏置 %.3f 升档需 %d 帧 冷却 %d 当前 %d 降档目标 %d 升计数 %d 冷却计数 %d", + (unsigned long long)frame, v[n / 2], v[n * 9 / 10], v[n - 1], n, (unsigned long long)over33, c[0], c[1], *(const float *)&c[2], c[3], c[4], c[5], c[6], c[7], c[8]); + { const uint8_t *b = (const uint8_t *)(uintptr_t)g_game_sys; // TOTK: frame-end mode (main+0xa91f98): +0x223 1 = wait for the last frame's GPU fence + LOG("[nvn] 帧结束方式 至帧 %llu：+0x222=%#x +0x223=%u（1 每帧等上一帧 GPU 做完）+0x224=%u +0x225=%#x +0xa6=%u +0x50=%d +0x88=%u", (unsigned long long)frame, + b[0x222], b[0x223], b[0x224], b[0x225], b[0xa6], *(const int32_t *)(b + 0x50), *(const uint32_t *)(b + 0x88)); } + n = 0; +} +/* TOTK diagnostic "flag framewait 0|1": the game's frame-end mode request (system object +0x224: 2 = no per-frame wait for the + GPU, 1 = wait), applied by the game at the start of its next frame (main+0x8f69e0). Measurement only. */ +void nvn_set_framewait(int on) { + if (!g_game_sys) { LOG("【红】[flag] framewait：游戏系统对象还没找到"); return; } + *(volatile uint8_t *)(uintptr_t)(g_game_sys + 0x224) = on ? 1 : 2; + LOG("[flag] framewait = %d（写 +0x224 = %d）", on, on ? 1 : 2); +} +/* [Windows port] TOTK 1.4.3's present-interval controller is main+0x2db174 (f_8002db174; MacNativeGames' main+0x8f6bec + of 1.2.1): x0 / x20 its state {+0 minimum 2, +4 maximum 3, +0x40 current}, x1 / x19 the system object (+0x118 frame + work ticks). It calls nvnWindowSetPresentInterval only when the interval changes (return address main+0x2db2ec) and + after the call stores the chosen interval (x21) to system object +0x88, from which the game steps its time. + TOTK_MAX_INTERVAL= (default 2; 0 = the game's own maximum): the highest interval the controller may pick, as + SavageWind does for BotW (SAVAGEWIND_MAX_INTERVAL). The game never drops to 20 fps (interval 3) when a frame misses + its tick; it keeps stepping its time by the interval it chose, so its speed stays right (a slow frame slows the game + down instead). A request for more than the maximum at that call is lowered together with the controller's current + interval and x21, so the time step matches what is presented. */ +/* The game's dynamic scene resolution (tools/lifter/patches.py, drs_off): off unless TOTK_DRS=1. */ +int totk_drs_enabled(void) { + static int on = -1; + if (on < 0) { const char *e = getenv("TOTK_DRS"); on = e && atoi(e) > 0; LOG("[nvn] dynamic resolution %s (TOTK_DRS)", on ? "on" : "off: fixed scene size"); } + return on; +} + +#define FRAME_CONTROLLER_LR 0x8002db2ecull +static uint64_t g_frame_ctl; // the controller's state (guest address) +uint64_t nvn_frame_ctl(void) { return g_frame_ctl; } +NVN(nvnWindowSetPresentInterval, void, NVNwindow *w, int v) { + Cpu *c = guest_cpu(); + if (c && c->x[30] == FRAME_CONTROLLER_LR) { + static int max_iv = -1; + if (max_iv < 0) { const char *e = getenv("TOTK_MAX_INTERVAL"); max_iv = e && *e ? atoi(e) : 2; LOG("[nvn] 出图间隔上限 %d（TOTK_MAX_INTERVAL；0 = 游戏自己的上限）", max_iv); } + if (!g_frame_ctl) { g_frame_ctl = c->x[20]; if (!g_game_sys) { g_game_sys = c->x[19]; LOG("[nvn] 游戏系统对象 %#llx，间隔控制器 %#llx", (unsigned long long)g_game_sys, (unsigned long long)g_frame_ctl); } } + uint32_t *ctl = (uint32_t *)(uintptr_t)c->x[20]; + if (max_iv > 0 && ctl[1] > (uint32_t)max_iv) ctl[1] = (uint32_t)max_iv; + if (max_iv > 0 && v > max_iv) { v = max_iv; c->x[21] = (uint32_t)v; ctl[0x40 / 4] = (uint32_t)v; } + } + if (((win_t *)w)->interval != v) LOG("[nvn] 游戏设出图间隔 %d → %d", ((win_t *)w)->interval, v); + ((win_t *)w)->interval = v; + if (!g_game_sys) find_game_sys(); +} +NVN(nvnWindowSetCrop, void, NVNwindow *w, int x, int y, int ww, int hh) { + win_t *o = (win_t *)w; int v[4] = {x, y, ww, hh}; + if (memcmp(o->crop, v, sizeof v)) { LOG("[nvn] WindowCrop %d,%d %dx%d", x, y, ww, hh); if (getenv("TOTK_SIZE_CHAIN")) guest_chain("window crop"); } + memcpy(o->crop, v, sizeof v); +} +static void trow(uint64_t f, const char *name, double v); +static void set_time_constraint(uint64_t period_ns); +static void rt_comp_tick(uint64_t period_ns); +static int g_uncapped; // TOTK_UNCAPPED configured, see present_cfg; suspended while frame interpolation paces the game (uncapped_now) +static int g_paced_hz; // the display-following rate, used instead of UNCAPPED_HZ while interpolation paces the game +#include "totk_mfx_mode.h" /* totk_mfx.m: frame interpolation mode of this game frame (TOTK_MFX, key 2) */ +/* TOTK: frame interpolation at 30 fps or at half the refresh rate paces the game itself, so it suspends the uncapped mode; + the uncapped interpolation mode keeps it. Key 2 cycles the modes at run time (2026-10-16 night). */ +static int uncapped_now(void) { int m = mfx_enabled(); return g_uncapped && (m == MFX_OFF || m == MFX_MAX); } +/* 30+30 on a display whose rate is not a multiple of 30 (the user's 100 Hz): the game is paced by our own clock at exactly + 30 fps instead of by the display link (3 refreshes would be 33 fps); the display link still shows each picture at a refresh. */ +static int mfx_own_clock(void) { return mfx_enabled() == MFX_30 && (g_paced_hz <= 0 || g_paced_hz % 30); } +/* Opt-in until live scene/clock validation. Only applies to the existing Mac + real-time uncapped path; original timing, MetalFX and deterministic tests + retain the guest interval. GPU/texture lifetime waits are unchanged. */ +static _Atomic int g_uncapped_interval = -1; +void nvn_set_uncapped_interval(int on) { + atomic_store(&g_uncapped_interval, on != 0); + LOG("[flag] uncapped_interval = %d", on != 0); +} +static int uncapped_interval_on(void) { + int v = atomic_load(&g_uncapped_interval); + if (v < 0) { const char *e = getenv("TOTK_UNCAPPED_INTERVAL"); v = e && atoi(e); atomic_store(&g_uncapped_interval, v); } + return v; +} +static double g_rt_comp_ms; // 当前声明的实时计算量（ms；0 = 未声明） | Currently declared real-time computation amount (ms; 0 = not declared) +static double g_rt_period_ms; +static void sleep_until_ns(uint64_t dl); +static uint64_t present_period(const win_t *o); +static unsigned present_ticks(const win_t *o); +extern void runtime_frame_step_update(uint64_t now_ns) __attribute__((weak_import)); +extern void movie_host_tick(void) __attribute__((weak_import)); /* TOTK runtime/movie_host.c: pre-rendered cutscenes at their own frame rate */ +__attribute__((weak)) void dm_main_yield_stats(uint64_t *n, uint64_t *ns) { *n = 0; *ns = 0; } // 宿主 hle_thread.c 提供真值；图形单测里为 0 | The host's hle_thread.c provides the ground truth; 0 in graphics unit tests +__attribute__((weak)) uint32_t dm_guest_work_qos(unsigned *count) { if (count) *count = 0; return QOS_CLASS_UNSPECIFIED; } +extern uint32_t mtl_present_qos(void); +uint64_t g_frame_pub; +static uint64_t g_acq_wait_ns, g_pq_wait_ns; // 取图里节拍等待 / 等出图队列归还的累计，出图时按帧记（一帧内可有多次取图） | Accumulated display-link wait / wait for the present queue to return inside acquire, recorded per frame at present (there can be several acquires within one frame) +uint64_t dm_nvn_frame_count(void) { return g_frame; } +extern uint64_t g_t0_ns; // 宿主进程起点（main.c；UIMSG 时间戳同基） | Host process start point (main.c; same base as the UIMSG timestamps) +NVN(nvnWindowAcquireTexture, NVNwindowAcquireTextureResult, NVNwindow *w, NVNsync *s, int *idx) { + NVN_BLOCKING(); + totk_tl(TL_ACQ_IN, 0, 0); + win_t *o = (win_t *)w; + *idx = o->active ? o->next++ % o->active : 0; + // 垂直同步阻塞在这里（真机 NVN：交换链里的贴图都在等显示时，等下一张被释放），不在出图里——游戏常持锁调出图 + // Vsync blocks here (real-device NVN: when all textures in the swapchain are waiting for display, wait for the next one to be released), not in present -- the game often calls present while holding a lock + uint64_t at = *idx < 4 ? o->avail[*idx] : 0, now = dm_detclock_enabled() ? guest_ns() : now_ns(), wait = at > now ? at - now : 0; + if (!dm_detclock_enabled()) { // 取图线程登记为时间约束线程（不再只在「来早了」时才登记：热态一直落后时也要声明） | Register the acquire thread as a time-constraint thread (no longer registered only when it "arrived early": it must also declare itself when it is constantly behind in the hot state) + static uint64_t rt_period; uint64_t period = present_period(o); + if (period && rt_period != period) { rt_period = period; set_time_constraint(period); } + rt_comp_tick(period); + if (mtl_link_active() && !uncapped_now() && !mfx_own_clock()) { if (!mtl_link_wait_tick(period ? 2 * period : 33333333ull, present_ticks(o)) && wait) sleep_until_ns(at); } // DM_PRESENT_LINK：节拍完全跟显示节拍（跑得快就等下一拍，落后立刻走）；自己的 60 Hz 时钟只作等不到节拍时的兜底 | DM_PRESENT_LINK: pacing fully follows the display link (wait for the next display-link tick when running fast, go immediately when behind); our own 60 Hz clock is only the fallback when no display-link tick arrives + else if (wait) sleep_until_ns(at); + } else { // DM_DETCLOCK_PACE=1（画面闸门用）：客体时钟按帧走的同时，每帧墙钟不短于一个步长——帧数与游戏内时间确定，后台加载线程每帧至少有一个步长的墙钟可用； | DM_DETCLOCK_PACE=1 (used by the screenshot gate): while the guest clock advances per frame, each frame's wall clock is no shorter than one step -- frame count and in-game time are deterministic, and the background loading thread has at least one step of wall clock available per frame; + // 机器满载时帧只会更慢、后台得到的墙钟只会更多，所以「消息后第 N 帧」截到的画面与负载无关 + // When the machine is fully loaded frames only get slower and the background only gets more wall clock, so the picture captured at "the Nth frame after the message" is independent of load + static int pace = -1; static uint64_t step, last; + if (pace < 0) { const char *e = getenv("DM_DETCLOCK_PACE"), *d = getenv("DM_DETCLOCK"); step = d ? strtoull(d, 0, 0) : 0; pace = e && atoi(e) && step; } + if (pace) { if (last && now_ns() < last + step) sleep_until_ns(last + step); last = now_ns(); } + } + g_acq_wait_ns += wait; trow(g_frame, "@late_ms", wait ? (now_ns() - at) / 1e6 : 0); + // 该张上一次出图还没被出图线程处理完（显示跟不上、落后一整轮交换链）：等它归还，再交给游戏画 + // The previous present of this texture has not yet been processed by the present thread (display cannot keep up, a whole swapchain round behind): wait for it to be returned, then hand it to the game to draw + if (async_replay() && *idx < 4 && g_xp_win == o) x_wait_job(g_xp_job[*idx]); // TOTK: that image's last present has run (its ticket is set) + uint64_t tk = *idx < 4 ? o->ticket[*idx] : 0; + if (tk) { uint64_t w0 = now_ns(); mtl_present_wait(tk); g_pq_wait_ns += now_ns() - w0; } + /* TOTK: the acquire sync means "this swapchain texture is no longer read by the display path". mtl_present_wait above + already waited until the GPU finished the present copy that last read this texture, and every later use of it is + ordered after that copy on the one Metal queue. Fencing the sync to the newest submission instead (the D2R behaviour, + TOTK_ACQ_SYNC=old) made the game's CPU wait for the whole previous frame here, so replay and GPU of consecutive + frames never overlapped and the GPU idled between frames. Bit 1 of fenced marks acquire syncs for TOTK_SYNC_LOG. */ + if (s) { static int old_sync = -1; if (old_sync < 0) { const char *e = getenv("TOTK_ACQ_SYNC"); old_sync = e && !strcmp(e, "old"); } + int tok = 0; uint64_t v = old_sync ? fence_now(&tok) : 0; + ((sync_t *)s)->fenced = tok ? 7 : 3; ((sync_t *)s)->fence = v; } + totk_tl(TL_ACQ_OUT, (uint64_t)*idx, 0); + return NVN_WINDOW_ACQUIRE_TEXTURE_RESULT_SUCCESS; +} + +// 帧边界：每次出图打印 / 落盘本帧各 NVN 函数调用次数 +// Frame boundary: on each present, print / write to disk the call count of each NVN function for this frame +static uint64_t g_prev[NVN_NFUNC]; +static FILE *g_tsv; +static void dump_frame(void) { + static int inited, nframes = 3, calls = 1; + if (!inited) { + inited = 1; + const char *e = getenv("DM_NVN_FRAMES"); if (e) nframes = atoi(e); + const char *c = getenv("DM_NVN_TSV_CALLS"); if (c) calls = atoi(c); // 0 = 只写 @ 计时行（真机拉文件有大小上限，调用计数行占约 2/3） | 0 = write only the @ timing rows (pulling files from a real device has a size limit, and the call-count rows take about 2/3) + const char *t = getenv("DM_NVN_TSV"); if (t && !(g_tsv = fopen(t, "w"))) LOG("【红】打不开 %s", t); + if (g_tsv) fprintf(g_tsv, "# kind: nvn-per-frame name 为 NVN 函数名时值 = 调用次数；@ 开头为计时埋点（ms / 个）\nframe\tname\tvalue\n"); + } + uint64_t total = 0; unsigned nfn = 0; + char line[8192]; size_t o = 0; line[0] = 0; + for (unsigned i = 0; i < NVN_NFUNC; i++) { + uint64_t c = atomic_load(&nvn_slot[i].cnt), d = c - g_prev[i]; + g_prev[i] = c; + if (!d) continue; + total += d; nfn++; + if (g_tsv && calls) fprintf(g_tsv, "%llu\t%s\t%llu\n", g_frame, nvn_names[i], d); + if (g_frame <= (uint64_t)nframes && o < sizeof line - 96) o += snprintf(line + o, sizeof line - o, " %s=%llu", nvn_names[i] + 3, d); + } + if (g_tsv) fflush(g_tsv); + if (g_frame <= (uint64_t)nframes) LOG("[nvn] %s %llu：%u 个函数 %llu 次调用 |%s", g_frame ? "帧" : "初始化", g_frame, nfn, total, line); + else if (g_frame % 300 == 0) LOG("[nvn] 帧 %llu：%u 个函数 %llu 次调用", g_frame, nfn, total); + g_frame++; g_frame_pub = g_frame; cs_dump(); + { void runtime_watch_tick(uint64_t); runtime_watch_tick(g_frame); } // TOTK diagnostic write watchpoint (TOTK_WATCH) + { void runtime_input_script_tick(uint64_t); runtime_input_script_tick(g_frame); } // TOTK_INPUT_SCRIPT (development) + // 每 300 帧读一次常驻内存，交图形层打内存读数行（mtl_mem_stat，低频封顶）。内存看门狗：常驻超过上限就报红自行退出——只在 Mac 宿主上缺省开（24 GB：宿主内存失控 + // 会把整机拖进内核崩溃，同机别的程序两次如此），或测试时用 DM_MEM_CAP_GB=<整数> 指定；玩家的 iOS 包不设上限（被系统杀之前自己退没有意义，反而多一种闪退） + // Every 300 frames read resident memory and hand it to the graphics layer to print the memory readout line (mtl_mem_stat, low frequency, capped). Memory watchdog: when resident memory exceeds the cap, log a failure and exit on its own. On by default only on the Mac host (24 GB: runaway host memory + // drags the whole machine into a kernel panic, which happened twice with other programs on the same machine), or set for tests with DM_MEM_CAP_GB=; the player's iOS app has no cap (exiting before the system kills it is pointless and only adds another kind of crash) + if (g_frame % 300 == 0) { task_vm_info_data_t vi; mach_msg_type_number_t cnt = TASK_VM_INFO_COUNT; + if (task_info(mach_task_self(), TASK_VM_INFO, (task_info_t)&vi, &cnt) == KERN_SUCCESS) { unsigned long long mb = (unsigned long long)(vi.phys_footprint >> 20); void mtl_mem_stat(unsigned long long, unsigned long long); mtl_mem_stat(g_frame, mb); + static long cap = -1; if (cap < 0) { const char *e = getenv("DM_MEM_CAP_GB"); cap = e ? atol(e) : DM_MEM_CAP_DEFAULT_GB; } + if (cap > 0 && mb > (unsigned long long)cap * 1024) { LOG("【红】内存看门狗：常驻 %llu MB 超过上限 %ld GB（帧 %llu），自行退出", mb, cap, (unsigned long long)g_frame); _exit(86); } } } + dm_guest_frame_tick(); // 确定性客体时钟（DM_DETCLOCK，宿主 hle_os.c）：每次出图推进固定步长；未开为空操作 | Deterministic guest clock (DM_DETCLOCK, host hle_os.c): each present advances a fixed step; a no-op when not enabled +} +// ---------- 计时埋点（逐帧行写进 DM_NVN_TSV：frame / @指标 / 值；每 300 帧汇总 p50 / p95 / 最大） ---------- +// ---------- Timing instrumentation (per-frame rows written to DM_NVN_TSV: frame / @metric / value; every 300 frames a summary of p50 / p95 / max) ---------- +#define TWIN 300 +static double g_tw[6][TWIN]; static unsigned g_tn[6]; +static double g_pres_iv[TWIN]; static unsigned g_presn; // 上屏间隔（ms；−1 = 被系统丢弃，0 = 无读数） | Present interval (ms; -1 = dropped by the system, 0 = no reading) +__attribute__((weak)) double dm_brightness_now(void) { return -1; } +__attribute__((weak)) void dm_perf_present(double iv_ms) { (void)iv_ms; } // 缺省（单测）；宿主 diag.c 的强定义：常开性能记录 perf.txt | Default (unit tests); strong definition in the host's diag.c: always-on performance record perf.txt +uint64_t dm_nvn_sub_cpu_ns(void) { return atomic_load(&g_cum_sub_cpu_ns); } // 提交（不含出图）累计 CPU 纳秒 | Cumulative CPU nanoseconds of submit (excluding present) +__attribute__((weak)) int dm_thermal_now(void) { return -1; } // 缺省（Mac）；iOS front_ios.m 覆盖 // 缺省（Mac）；iOS 前端 front_ios.m 的强定义覆盖它 | Default (Mac); overridden by iOS front_ios.m // default (Mac); the strong definition in the iOS front end front_ios.m overrides it +// 长帧日志的逐帧环（GPU 时间滞后 8 帧才有，所以先存后判） +// Per-frame ring for the long-frame log (GPU time only arrives 8 frames late, so store first and judge later) +#define LFN 16 +typedef struct { uint64_t f; double fr, nv, t_s, make_ms, up_mb, up_ms; unsigned pso, lib, up, rup; int ls; } lf_rec; // ls：这一帧出图时读取画面是否盖着 | ls: whether the loading screen was covering the picture when this frame was presented +static lf_rec g_lf[LFN]; +__attribute__((weak)) void dm_longframe_note(const char *what) { LOG("[长帧] %s", what); } // 缺省（图形单测）；宿主 diag.c 的强定义补上场景上下文并写 perf.txt | Default (graphics unit tests); the strong definition in the host's diag.c adds scene context and writes perf.txt +extern void mtl_create_ns(uint64_t *lib_ns, uint64_t *pso_ns); extern uint64_t mtl_upload_bytes(void); extern void mtl_pkg_stats(uint64_t *uploads, uint64_t *reuploads); +static const char *const g_tname[6] = {"帧时间", "出帧工作", "NVN 提交+出图", "GPU", "翻译层录制期", "翻译层提交期"}; +static void tpush(int k, double v) { g_tw[k][g_tn[k]++ % TWIN] = v; } +// 发热归因（iOS DM_HEAT）：每帧 GPU 毫秒另存一圈，取「上次取数以来」的均值 / p95（最多最近 HRN 帧），与计时汇总的 300 帧窗口互不干扰 +// Thermal attribution (iOS DM_HEAT): per-frame GPU milliseconds are kept in a separate ring, taking the mean / p95 "since the last read" (at most the last HRN frames), without interfering with the timing summary's 300-frame window +#define HRN 1024 +static double g_hr[HRN]; static uint64_t g_hn, g_hn_read; +static int dcmp(const void *a, const void *b); +void nvn_gpu_recent(double *avg, double *p95, unsigned *n) { + uint64_t hi = g_hn, lo = g_hn_read > hi ? hi : g_hn_read; if (hi - lo > HRN) lo = hi - HRN; g_hn_read = hi; + unsigned c = (unsigned)(hi - lo); static double v[HRN]; double s = 0; + for (unsigned i = 0; i < c; i++) { v[i] = g_hr[(lo + i) % HRN]; s += v[i]; } + qsort(v, c, sizeof *v, dcmp); *n = c; *avg = c ? s / c : -1; *p95 = c ? v[(c * 95) / 100 < c ? (c * 95) / 100 : c - 1] : -1; +} +static int dcmp(const void *a, const void *b) { double x = *(const double *)a, y = *(const double *)b; return (x > y) - (x < y); } +static void tsummary(uint64_t frame) { + char line[1024]; size_t o = 0; + for (int k = 0; k < 6; k++) { + unsigned n = g_tn[k] < TWIN ? g_tn[k] : TWIN; if (!n) continue; + double v[TWIN]; memcpy(v, g_tw[k], n * sizeof *v); qsort(v, n, sizeof *v, dcmp); + o += snprintf(line + o, sizeof line - o, " %s p50 %.2f / p95 %.2f / 最大 %.2f ms（%u 帧）；", g_tname[k], v[n / 2], v[(n * 95) / 100 < n ? (n * 95) / 100 : n - 1], v[n - 1], n); + } + LOG("[nvn] 计时 至帧 %llu：%s", frame, line); + { unsigned n = g_tn[0] < TWIN ? g_tn[0] : TWIN, over = 0; double v[TWIN]; // 帧间隔另起一行（M1 基线）：本窗口 > 16.7 ms 帧数与 p99 | frame interval gets its own line (M1 baseline): number of frames > 16.7 ms in this window and p99 + if (n) { memcpy(v, g_tw[0], n * sizeof *v); qsort(v, n, sizeof *v, dcmp); for (unsigned i = 0; i < n; i++) over += v[i] > 16.7; + LOG("[nvn] 帧间隔 至帧 %llu：>16.7ms %u / %u 帧，p99 %.2f ms", frame, over, n, v[(n * 99) / 100 < n ? (n * 99) / 100 : n - 1]); } } + { unsigned n = g_presn < TWIN ? g_presn : TWIN, k = 0, drop = 0, discard = 0; double v[TWIN]; // 上屏：掉帧 = 间隔 > 16.7 × 1.5 ms 或被丢弃 | on-screen present: dropped frame = interval > 16.7 × 1.5 ms or discarded + for (unsigned i = 0; i < n; i++) { double x = g_pres_iv[i]; if (x < 0) { drop++; discard++; } else if (x > 0) { v[k++] = x; drop += x > 25.0; } } + task_vm_info_data_t vi; mach_msg_type_number_t vc = TASK_VM_INFO_COUNT; double fp = task_info(mach_task_self(), TASK_VM_INFO, (task_info_t)&vi, &vc) == KERN_SUCCESS ? vi.phys_footprint / 1048576.0 : -1; + double br = dm_brightness_now(); // footprint / 亮度随上屏行带出：不依赖拍入口观测门（营地局也有） | footprint / brightness are carried on the on-screen present row: does not depend on the tick-entry observation gate (camp runs have it too) + if (k) { qsort(v, k, sizeof *v, dcmp); LOG("[nvn] 上屏 至帧 %llu：%u 帧，掉帧 %u（丢弃 %u），p99 %.2f ms；footprint %.0f MiB；亮度 %.2f；温控 %d", frame, k + discard, drop, discard, v[(k * 99) / 100 < k ? (k * 99) / 100 : k - 1], fp, br, dm_thermal_now()); } + g_presn = 0; } + { static uint64_t pi, pc, pd, pf; uint64_t ci = atomic_load(&g_cum_sub_instr), cc = atomic_load(&g_cum_sub_cpu_ns), ps_, dr_; mtl_stats(&ps_, &dr_); // 翻译层 | translation layer + if (pf && frame > pf && dr_ > pd) LOG("[nvn] 翻译 至帧 %llu：每次绘制 %.0f 条指令，每帧提交 %.2f M 条 / CPU %.3f ms，每帧绘制 %.0f", frame, + (double)(ci - pi) / (double)(dr_ - pd), (ci - pi) / 1e6 / (double)(frame - pf), (cc - pc) / 1e6 / (double)(frame - pf), (double)(dr_ - pd) / (double)(frame - pf)); + { static uint64_t pae, par; uint64_t ae, ar; mtl_args_stats(&ae, &ar); // TOTK: argument blocks encoded vs rebound unchanged + if (pf && frame > pf) LOG("[nvn] 参数块 至帧 %llu：每帧新编 %.0f 个，复用 %.0f 个（%.0f%%）", frame, (double)(ae - pae) / (double)(frame - pf), (double)(ar - par) / (double)(frame - pf), + ae - pae + ar - par ? 100.0 * (double)(ar - par) / (double)(ae - pae + ar - par) : 0.0); + pae = ae; par = ar; } + { extern uint64_t g_draw_skip[16]; static uint64_t prev[16]; char b[512]; size_t o = 0; b[0] = 0; + static const char *why[] = {"no-program", "zero-source", "defer-take", "render-disabled", "raster-discard", "cull-both", "scissor-empty", "deferred", "no-encoder", "no-pipeline", "viewport-empty", "primitive", + "clear-render-disabled", "cond-disabled", "cond-unknown", "shader-missing"}; // TOTK: 12-14 conditional rendering (see cond_eval); 15 a shader variant the iPhone does not have yet + for (int i = 0; i < 16; i++) { uint64_t d = g_draw_skip[i] - prev[i]; prev[i] = g_draw_skip[i]; if (d && o < sizeof b - 48) o += snprintf(b + o, sizeof b - o, " %s=%llu", why[i], (unsigned long long)d); } + if (o) LOG("[nvn] TOTK 回放丢弃的绘制 至帧 %llu:%s", frame, b); } + { static uint64_t py, pk, pw; uint64_t cy = atomic_load(&g_cum_sub_cyc), ck = atomic_load(&g_cum_sub_sys_ns), cw = atomic_load(&g_cum_sub_wall_ns); + if (pf && frame > pf && ci > pi && cy > py) { double nf = (double)(frame - pf), cyc = (double)(cy - py), cpu = (double)(cc - pc); + qos_class_t qc = QOS_CLASS_UNSPECIFIED; int rp = 0; pthread_get_qos_class_np(pthread_self(), &qc, &rp); + uint64_t aw = 0, ag = 0; int an = 0; void mtl_alloc_stats(uint64_t *, uint64_t *, int *); mtl_alloc_stats(&aw, &ag, &an); + static uint64_t pyn, pyns; uint64_t yn = 0, yns = 0; dm_main_yield_stats(&yn, &yns); + LOG("[nvn] 翻译细项 至帧 %llu：每帧提交 周期 %.2f M（每条指令 %.2f 周期）、CPU %.3f ms 其中内核态 %.3f ms、墙钟 %.3f ms；平均主频 %.2f GHz（周期 / CPU 时间）；本线程 QoS %#x；分配器池 %d 个、累计等 GPU %llu 次；实时声明计算 %.2f ms；主线程让出 每帧 %.0f 次 / %.3f ms", frame, + cyc / 1e6 / nf, cyc / (double)(ci - pi), cpu / 1e6 / nf, (double)(ck - pk) / 1e6 / nf, (double)(cw - pw) / 1e6 / nf, cpu > 0 ? cyc / cpu : 0, (unsigned)qc, an, (unsigned long long)aw, g_rt_comp_ms, (double)(yn - pyn) / nf, (double)(yns - pyns) / 1e6 / nf); pyn = yn; pyns = yns; } + { static uint64_t pun, puns; uint64_t un = 0, uns = 0; void mtl_upload_wait_stats(uint64_t *, uint64_t *); mtl_upload_wait_stats(&un, &uns); + if (pf && frame > pf) LOG("[nvn] 上传槽等待 至帧 %llu：每帧 %.2f 次 / %.3f ms", frame, (double)(un - pun) / (double)(frame - pf), (double)(uns - puns) / 1e6 / (double)(frame - pf)); + pun = un; puns = uns; } + { static uint64_t pcc; uint64_t mtl_chunk_commits(void), cc = mtl_chunk_commits(); + if (pf && frame > pf) LOG("[nvn] 分段提交 至帧 %llu：每帧额外 %.2f 段", frame, (double)(cc - pcc) / (double)(frame - pf)); + pcc = cc; } + if (synclog_env() && pf && frame > pf) { static uint64_t pc[2], pb[2], pn[2], pw[2]; double nf = (double)(frame - pf); + for (int k = 0; k < 2; k++) { uint64_t c = atomic_load(&g_sw_calls[k]), b = atomic_load(&g_sw_blocked[k]), n = atomic_load(&g_sw_ns[k]), w = atomic_load(&g_sw_newest[k]); + LOG("[nvn] CPU 等同步 至帧 %llu：%s 每帧 %.2f 次，其中真的等 %.2f 次 / %.3f ms，等的是最近一次提交 %.2f 次", frame, k ? "取图同步" : "栅栏同步", (c - pc[k]) / nf, (b - pb[k]) / nf, (n - pn[k]) / 1e6 / nf, (w - pw[k]) / nf); + pc[k] = c; pb[k] = b; pn[k] = n; pw[k] = w; } } + { static uint64_t pv[6]; uint64_t v[6]; void mtl_blit_stats(uint64_t[6]); mtl_blit_stats(v); + if (pf && frame > pf) { double nf = (double)(frame - pf); + LOG("[nvn] 拷贝编码器 至帧 %llu：每帧 %.1f 个（其中打断渲染通道 %.1f）；uniform 更新 %.1f 次（打断 %.1f），缓冲拷贝 %.1f 次（打断 %.1f）", frame, + (v[0] - pv[0]) / nf, (v[1] - pv[1]) / nf, (v[2] - pv[2]) / nf, (v[3] - pv[3]) / nf, (v[4] - pv[4]) / nf, (v[5] - pv[5]) / nf); } + memcpy(pv, v, sizeof pv); } + { static uint64_t pp[4]; uint64_t v[4]; void mtl_pool_res_stats(uint64_t[4]); mtl_pool_res_stats(v); // TOTK: memory pools resident on first GPU use (nvn_mtl.m pool_res) + if (memcmp(v, pp, sizeof pp)) LOG("[nvn] 内存池 至帧 %llu：在册 %llu 个 / %llu MB，其中已放进驻留集 %llu 个 / %llu MB", frame, + (unsigned long long)v[0], (unsigned long long)(v[1] >> 20), (unsigned long long)v[2], (unsigned long long)(v[3] >> 20)); + memcpy(pp, v, sizeof pp); } + { static uint64_t pk[2]; uint64_t v[3]; void mtl_keepalive_stats(uint64_t[3]); mtl_keepalive_stats(v); // TOTK: GPU keep-alive spin kernels (nvn_mtl.m keepalive_start) + if (pf && frame > pf && v[0] > pk[0]) LOG("[nvn] GPU 保活 至帧 %llu：每帧 %.2f 个自旋核，平均 %.2f ms，最长 %.2f ms", frame, (double)(v[0] - pk[0]) / (double)(frame - pf), + (double)(v[1] - pk[1]) / 1e3 / (double)(v[0] - pk[0]), (double)v[2] / 1e3); + pk[0] = v[0]; pk[1] = v[1]; } + { static uint64_t pl[4]; uint64_t v[4]; void mtl_layer_stats(uint64_t[4]); mtl_layer_stats(v); // TOTK: layered passes (nvn_mtl.m g_lay) + if (pf && frame > pf) { double nf = (double)(frame - pf); + LOG("[nvn] 渲染通道 至帧 %llu：每帧 %.1f 个，其中分层通道 %.2f 个，并入的设目标 %.2f 次，因绘制不能写层号退回 %.2f 次", frame, + (v[0] - pl[0]) / nf, (v[1] - pl[1]) / nf, (v[2] - pl[2]) / nf, (v[3] - pl[3]) / nf); } + memcpy(pl, v, sizeof pl); } + { static uint64_t pr[6]; uint64_t v[6]; void mtl_ren_stats(uint64_t[6]); mtl_ren_stats(v); + if (pf && frame > pf) { double nf = (double)(frame - pf); + LOG("[nvn] uniform 换副本 至帧 %llu：每帧更新 %.1f 次，新副本 %.1f 个 / %.1f KB，提前写回 %.2f 次，开编码器后才发现重叠 %.2f 次，表满回退 %.2f 次", frame, + (v[0] - pr[0]) / nf, (v[1] - pr[1]) / nf, (v[2] - pr[2]) / nf / 1024, (v[3] - pr[3]) / nf, (v[4] - pr[4]) / nf, (v[5] - pr[5]) / nf); } + memcpy(pr, v, sizeof pr); } + if (frame % 300 == 0) { unsigned wn = 0; uint32_t wq = dm_guest_work_qos(&wn); thread_extended_info_data_t xi; mach_msg_type_number_t xc = THREAD_EXTENDED_INFO_COUNT; int okx = thread_info(pthread_mach_thread_np(pthread_self()), THREAD_EXTENDED_INFO, (thread_info_t)&xi, &xc) == KERN_SUCCESS; + thread_time_constraint_policy_data_t tp; mach_msg_type_number_t tc = THREAD_TIME_CONSTRAINT_POLICY_COUNT; boolean_t def = 0; int rt_live = thread_policy_get(pthread_mach_thread_np(pthread_self()), THREAD_TIME_CONSTRAINT_POLICY, (thread_policy_t)&tp, &tc, &def) == KERN_SUCCESS && !def; // def = 1：实时策略没在生效（没设上 / 被撤） | def = 1: the real-time policy is not in effect (never set / revoked) + LOG("[nvn] 线程策略：主线程 实时 计算 %.2f ms / 周期 %.2f ms，实时策略%s，当前优先级 %d（基础 %d，调度类 %d）；工作线程 %u 条 QoS %#x；出图 QoS %#x", g_rt_comp_ms, g_rt_period_ms, rt_live ? "在生效" : "未生效", okx ? xi.pth_curpri : -1, okx ? xi.pth_priority : -1, okx ? xi.pth_policy : -1, wn, wq, mtl_present_qos()); } + py = cy; pk = ck; pw = cw; } + pi = ci; pc = cc; pd = dr_; pf = frame; } + uint64_t up, re, fa, pr; mtl_pkg_stats(&up, &re); nvn_dirty_stats(&fa, &pr); + LOG("[nvn] 预排布贴图 首传 %llu 张、池内容变化重传 %llu 次；写保护缺页 %llu 次、宿主写前解保护 %llu 页；漏检 %llu（累计）", up, re, fa, pr, mtl_pkg_miss()); + memset(g_tn, 0, sizeof g_tn); +} +// DM_SLOWFRAME=1（测试开关，真机经日志端口读）：逐帧计时先存近 16 帧环；滞后 8 帧（GPU / 上屏时刻已回）判掉帧， +// 只对掉帧写一行「[slowframe]\t帧号\t@名=值…」进日志，另每 300 帧一行分母。不写逐帧文件：日志 tee 每行 fsync，逐行写会扰动读数 +// DM_SLOWFRAME=1 (test switch, read over the log port on a real device): per-frame timings are first stored in a ring of the last 16 frames; dropped frames are judged 8 frames late (GPU / on-screen present times are back by then), +// and only for dropped frames is one line "[slowframe]\t\t@=..." written to the log, plus one denominator line every 300 frames. No per-frame file is written: the log tee fsyncs every line, and writing line by line would disturb the readings +#define SRN 16 +static struct { uint64_t f; int n; const char *k[64]; double v[64]; } g_sr[SRN]; +static int g_slow = -1; +static int slow_on(void) { if (g_slow < 0) g_slow = getenv("DM_SLOWFRAME") != NULL; return g_slow; } +static int frame_rec(void) { return g_tsv || slow_on(); } +static void trow(uint64_t f, const char *name, double v) { + if (g_tsv) fprintf(g_tsv, "%llu\t%s\t%.3f\n", f, name, v); + if (slow_on()) { __typeof__(g_sr[0]) *r = &g_sr[f % SRN]; if (r->f != f) { r->f = f; r->n = 0; } if (r->n < 64) { r->k[r->n] = name; r->v[r->n++] = v; } } +} +static double srow(uint64_t f, const char *name, double dflt) { + __typeof__(g_sr[0]) *r = &g_sr[f % SRN]; + if (r->f == f) for (int i = 0; i < r->n; i++) if (!strcmp(r->k[i], name)) return r->v[i]; + return dflt; +} +// 掉帧口径：上屏间隔 > 16.7×1.5 ms 或 −1；无窗口（无上屏时刻）退回出图调用间隔 +// Dropped-frame definition: on-screen present interval > 16.7×1.5 ms or −1; without a window (no on-screen present time) fall back to the present call interval +static void slow_emit(uint64_t f) { + static uint64_t judged, slow; + double pm = srow(f, "@presented_ms", 0), fr = srow(f, "@frame_ms", 0), lim = 1000.0 / 60 * 1.5; + int has = mtl_has_window(), bad = has ? (pm > lim || pm < 0) : fr > lim; + if (has ? pm != 0 : fr != 0) judged++; + if (bad) { + slow++; + char b[4096]; size_t o = snprintf(b, sizeof b, "[slowframe]\t%llu", f); + __typeof__(g_sr[0]) *r = &g_sr[f % SRN]; + for (int i = 0; i < r->n && o < sizeof b - 64; i++) o += snprintf(b + o, sizeof b - o, "\t%s=%.3f", r->k[i], r->v[i]); + double m = 0; for (uint64_t p = f >= 3 ? f - 3 : 0; p < f; p++) { double x = srow(p, "@frame_ms", 0); if (x > m) m = x; } + snprintf(b + o, sizeof b - o, "\t@prev3_max_frame_ms=%.3f", m); + fprintf(stderr, "%s\n", b); + } + if (f % 300 == 0) fprintf(stderr, "[slowframe]\t#\t至帧 %llu\tjudged=%llu\tslow=%llu\t判据=%s\n", f, judged, slow, has ? "上屏间隔" : "出图调用间隔（无窗口）"); +} +// 时间约束（实时）线程的声明：周期 = 出图周期，约束 = 周期，计算 = 这条线程每帧实际要用的 CPU 时间。 +// 原先计算量写死 2 ms，而游戏主线程（游戏逻辑 + 录制 + 提交回放）热态每帧约 10 ms——声明与实际差 5 倍： +// 实时线程超出自己声明的计算量会被系统降级处理，性能控制器也按声明定频（系统就是靠这份声明得知帧期限和工作量的）。 +// DM_RT_COMP=<毫秒> 固定声明；缺省 auto = 每 120 帧取本线程每帧 CPU 时间的 p90，与已声明值差 > 15% 就重新声明（上限 0.9 × 周期）。 +// Declaration for the time-constraint (real-time) thread: period = present period, constraint = period, computation = the CPU time this thread actually needs per frame. +// The computation used to be hard-coded at 2 ms, while the game main thread (game logic + recording + submit replay) takes about 10 ms per frame when hot - a 5× gap between declared and actual: +// a real-time thread that exceeds its declared computation gets demoted by the system, and the performance controller also sets frequency from the declaration (this declaration is how the system learns the frame deadline and the workload). +// DM_RT_COMP= fixes the declaration; default auto = every 120 frames take the p90 of this thread's per-frame CPU time and re-declare if it differs from the declared value by > 15% (capped at 0.9 × period). +static void set_time_constraint_comp(uint64_t period_ns, uint64_t comp_ns) { + static mach_timebase_info_data_t tb; if (!tb.denom) mach_timebase_info(&tb); + if (comp_ns < 1000000ull) comp_ns = 1000000ull; if (comp_ns > period_ns * 9 / 10) comp_ns = period_ns * 9 / 10; + thread_time_constraint_policy_data_t p = { + .period = (uint32_t)(period_ns * tb.denom / tb.numer), .computation = (uint32_t)(comp_ns * tb.denom / tb.numer), + .constraint = (uint32_t)(period_ns * tb.denom / tb.numer), .preemptible = 1}; + kern_return_t kr = thread_policy_set(pthread_mach_thread_np(pthread_self()), THREAD_TIME_CONSTRAINT_POLICY, (thread_policy_t)&p, THREAD_TIME_CONSTRAINT_POLICY_COUNT); + if (kr == KERN_SUCCESS) { g_rt_comp_ms = comp_ns / 1e6; g_rt_period_ms = period_ns / 1e6; } + LOG("[nvn] 出图线程时间约束策略（周期 %.3f ms、计算 %.3f ms）%s", period_ns / 1e6, comp_ns / 1e6, kr == KERN_SUCCESS ? "已设" : "设置失败"); +} +// TOTK translation thread: the same policy for the calling thread without touching the render thread's declaration state. +void set_time_constraint_comp_x(uint64_t period_ns, uint64_t comp_ns) { + mach_timebase_info_data_t tb; mach_timebase_info(&tb); + thread_time_constraint_policy_data_t p = { + .period = (uint32_t)(period_ns * tb.denom / tb.numer), .computation = (uint32_t)(comp_ns * tb.denom / tb.numer), + .constraint = (uint32_t)(period_ns * tb.denom / tb.numer), .preemptible = 1}; + kern_return_t kr = thread_policy_set(pthread_mach_thread_np(pthread_self()), THREAD_TIME_CONSTRAINT_POLICY, (thread_policy_t)&p, THREAD_TIME_CONSTRAINT_POLICY_COUNT); + LOG("[nvn] 翻译线程时间约束策略（周期 %.3f ms、计算 %.3f ms）%s", period_ns / 1e6, comp_ns / 1e6, kr == KERN_SUCCESS ? "已设" : "设置失败"); +} +static int rt_comp_fixed_ms(void) { static int v = -2; if (v == -2) { const char *e = getenv("DM_RT_COMP"); v = e && *e && strcmp(e, "auto") ? atoi(e) : -1; } return v; } // −1 = auto +static void set_time_constraint(uint64_t period_ns) { int f = rt_comp_fixed_ms(); set_time_constraint_comp(period_ns, f > 0 ? (uint64_t)f * 1000000ull : 2000000ull); } // 首次：固定值，或 auto 的起始值 2 ms | first time: the fixed value, or auto's starting value of 2 ms +// 每帧在取图处调：记本线程自上次以来的 CPU 时间；auto 模式下按 p90 重新声明 +// Called every frame at acquire: records this thread's CPU time since the last call; in auto mode re-declares based on p90 +static void rt_comp_tick(uint64_t period_ns) { + if (rt_comp_fixed_ms() > 0 || !period_ns || !g_rt_comp_ms) return; + static uint64_t last; static double win[120]; static int n; + uint64_t c = clock_gettime_nsec_np(CLOCK_THREAD_CPUTIME_ID); if (!last) { last = c; return; } + win[n++] = (double)(c - last) / 1e6; last = c; + if (n < 120) return; + double v[120]; memcpy(v, win, sizeof v); qsort(v, 120, sizeof *v, dcmp); n = 0; + double p90 = v[107], comp = p90 * 1.25, cap = period_ns * 0.9 / 1e6; if (comp > cap) comp = cap; if (comp < 1) comp = 1; + if (comp > g_rt_comp_ms * 1.15 || comp < g_rt_comp_ms * 0.85) set_time_constraint_comp(period_ns, (uint64_t)(comp * 1e6)); +} +static void sleep_until_ns(uint64_t dl) { // 单调时钟上的绝对时刻 | absolute time on the monotonic clock + static mach_timebase_info_data_t tb; if (!tb.denom) mach_timebase_info(&tb); + uint64_t now = now_ns(); if (dl <= now) return; + mach_wait_until(mach_absolute_time() + (dl - now) * tb.denom / tb.numer); +} +// 出图固定 60 Hz：自适应机制在 24/24 份真机日志第 60 帧退档且不再回升，本期先以 60 帧为硬底。 +// DM_PRESENT_HZ=0（不限帧率）只作显式的性能剖析开关：游戏部分界面逻辑按帧计数（同存档不限帧率时 +// SaveIndicatorPanel 4181 条、限 60 帧时 0 条，与真值一致），不限帧率在语义上是错的跑法，不能作判据 / 对拍 / 缺省。单测用 nvn_present_config 直接设 +// Present is fixed at 60 Hz: the adaptive mechanism stepped down at frame 60 in 24/24 real-device logs and never came back up, so for this phase 60 fps is the hard floor. +// DM_PRESENT_HZ=0 (uncapped frame rate) serves only as an explicit performance profiling switch: some of the game's UI logic counts frames (with the same save, uncapped gives +// 4181 SaveIndicatorPanel entries and capped at 60 fps gives 0, which matches the ground truth), so uncapped is a semantically wrong way to run and must not be used as a criterion / cross-check / default. Unit tests set it directly with nvn_present_config +static int g_hz = -1, g_display_mode; +/* TOTK_UNCAPPED (Mac and iPhone, default on; =0 off): like vsync off in other games. The game no longer waits for the display: it is + paced by our own clock at UNCAPPED_HZ, and the display link shows the newest finished frame at every refresh (older ones + are merged away, as when the game runs ahead). The ceiling is 120 because the real-time frame step (game_patches.c) + clamps a frame to at least 1/120 s: above that the game would run fast. On a display of 120 Hz or more it changes nothing. */ +#define UNCAPPED_HZ 120 +/* TOTK_UNCAPPED_HZ: the uncapped clock's ceiling, 30..120 (default 120), or "display" for the display's refresh rate: the + iPhone app's default since 2026-10-17 (user: at most 60 on the iPhone 14's 60 Hz screen, where frames above it are never + shown and only heat the phone). The game still does not wait for the display, so a scene slower than the ceiling is not + held to 60 / 30 steps. */ +static int uncapped_cap(int scr) { + const char *e = getenv("TOTK_UNCAPPED_HZ"); + int v = !e ? UNCAPPED_HZ : !strcmp(e, "display") ? (scr > 0 ? scr : 60) : atoi(e); + return v < 30 ? 30 : v > UNCAPPED_HZ ? UNCAPPED_HZ : v; +} +void nvn_present_config(int hz, int display_mode) { g_hz = g_paced_hz = hz; g_display_mode = display_mode; } +static void present_cfg(const win_t *o) { + if (g_hz >= 0) return; + // 有窗口仍缺省 display 间隔模式；无窗口缺省 game 模式。环境变量可覆盖固定 60 Hz。 + // With a window the default is still display interval mode; without a window the default is game mode. Environment variables can override the fixed 60 Hz. + int scr = mtl_display_hz(); + const char *e = getenv("DM_PRESENT_HZ"), *rt = getenv("TOTK_REALTIME"); + int realtime = !(rt && !strcmp(rt, "0")); + g_hz = g_paced_hz = e ? atoi(e) : (realtime && scr > 0 ? scr : 60); + { const char *u = getenv("TOTK_UNCAPPED"), *f60 = getenv("TOTK_FPS60"); // the original 30-frame mode needs fixed pacing; MetalFX interpolation suspends it while on (uncapped_now) + // [port] Uncapped needs MacNativeGames' 60 FPS game patches and real-time frame step (game_patches_v143.c), not + // ported yet: without them the engine steps game time per frame as if at its chosen + // interval, and ~60 presents a second ran the game fast. Off unless TOTK_UNCAPPED=1. + g_uncapped = !e && realtime && u && !strcmp(u, "1") && !(f60 && !strcmp(f60, "0")) && g_hz < UNCAPPED_HZ; } + if (g_uncapped) { g_hz = uncapped_cap(scr); LOG("[nvn] 不限帧率（TOTK_UNCAPPED，默认开；=0 关）：不等显示器刷新，自己的时钟上限 %d 帧（TOTK_UNCAPPED_HZ）；显示器 %d Hz 每次刷新显示最新一帧%s", g_hz, scr, + mfx_enabled() ? "（补帧开着时暂停：每两次刷新一帧）" : "（补帧打开时暂停，2 键）"); } + if (!g_hz) LOG("[nvn] 【警告】DM_PRESENT_HZ=0：不限帧率，帧节奏非真机，界面逻辑可能偏离（仅供性能剖析，不能用于判据 / 对拍）"); + // 剪口（有意偏离原版行为）：DM_PRESENT_INTERVAL=display 时不按游戏设的出图间隔，每个节拍出一帧， + // 固定 60 帧硬底；逻辑 25 拍不变，且客户端按帧计数的界面逻辑不受影响—— + // 同存档同输入下两模式 UI 消息流逐条一致。证明（选角前）：r9 game / display 60 ×3 / display 120 ×2 共 6 组 uicmp 一致 + // （存档指示按时序竞态豁免），关键消息时刻差在 game 自身噪声带内；进局后的证明尚缺，待动作脚本覆盖 + // Cut (intentional deviation from the original game's behavior): with DM_PRESENT_INTERVAL=display the present interval set by the game is not followed and one frame is presented per display-link tick, + // with a fixed hard floor of 60 fps; the logic stays at 25 ticks, and the client's frame-counted UI logic is unaffected - + // with the same save and the same input, the UI message streams of the two modes match message by message. Proof (before character select): r9 game / display 60 ×3 / display 120 ×2, 6 groups in total, uicmp consistent + // (the save indicator is exempted as a timing race), and the timing differences of key messages are within game mode's own noise band; proof after entering a run is still missing, pending action script coverage + /* TOTK: game mode is the default. The engine's variable-frame-rate step (main+0x7a83e4) advances game time by the + present interval it chose for every frame, so presenting on every display tick (D2R's display mode) made menus + and light scenes run up to twice as fast. DM_PRESENT_INTERVAL=display keeps the old behaviour for comparison. */ + const char *m = getenv("DM_PRESENT_INTERVAL"); g_display_mode = m && !strcmp(m, "display"); + LOG("[nvn] 出图节拍 %d Hz，间隔模式 %s（游戏设的出图间隔 %d；%s）", g_hz, g_display_mode ? "display（每节拍一帧，剪口）" : "game（原版语义）", o->interval, + e ? "显式指定" : g_uncapped ? "不限帧率" : realtime && scr > 0 ? "跟随显示器" : "固定 60 Hz"); +} +static unsigned present_ticks(const win_t *o) { /* display ticks per game frame */ + present_cfg(o); + /* TOTK_MFX / key 2: MetalFX makes the frame between two game frames. Half the refresh rate: the game presents every + second display tick (50 + 50 on a 100 Hz display); 30+30: hz / 30 ticks (2 at 60 Hz; at 100 Hz the own clock paces, + mfx_own_clock). Held here and not only by the game's controller, because the code carries the 60 FPS patch (the + controller may choose 1); the real-time frame step keeps game speed right. Uncapped interpolation paces as uncapped. */ + int m = mfx_enabled(); + if (m == MFX_HALF) return 2; + if (m == MFX_30) { int hz = g_paced_hz > 0 ? g_paced_hz : 60; return (unsigned)(hz >= 45 ? (hz + 15) / 30 : 1); } + return g_display_mode || (uncapped_now() && !dm_detclock_enabled() && uncapped_interval_on()) || o->interval <= 0 ? 1u : (unsigned)o->interval; +} +static uint64_t present_period(const win_t *o) { + unsigned itv = present_ticks(o); int m = mfx_enabled(), hz = g_uncapped && !uncapped_now() ? g_paced_hz : g_hz; + { static int was = -1; if (was >= 0 && m != was) + LOG("[nvn] 出图节拍切换：%s", m == MFX_30 ? (mfx_own_clock() ? "补帧 30+30：游戏按自己的时钟每秒 30 帧" : "补帧 30+30：跟随显示器，每秒 30 帧") : + m == MFX_HALF ? "补帧 刷新率一半：跟随显示器、每两次刷新一帧" : + m == MFX_MAX ? (g_uncapped ? "补帧 不限帧：不限帧率，每帧补一帧" : "补帧 不限帧：按游戏的出图间隔，每帧补一帧") : + g_uncapped ? "补帧关：不限帧率" : "补帧关：按游戏的出图间隔"); was = m; } + if (m == MFX_30) return 1000000000ull / 30; /* exactly 30 fps, by display ticks or by our own clock */ + return hz > 0 ? (uint64_t)itv * 1000000000ull / (uint64_t)hz : 0; +} +/* Exercise production policy, including the shrine's interval=2 and the + legacy 30 FPS path. No sleep, present or GPU allocation is involved. */ +void dm_nvn_probe_present_interval(void) { + int saved_hz = g_hz, saved_display = g_display_mode, saved_uncapped = g_uncapped; + int saved_flag = atomic_load(&g_uncapped_interval), checked = 0; + const char *neg = getenv("TOTK_PROBE_NEGATIVE"); + for (int hz = 60; hz <= 120; hz += 60) for (int uncapped = 0; uncapped < 2; uncapped++) + for (int display = 0; display < 2; display++) for (int on = 0; on < 2; on++) + for (int interval = -1; interval <= 3; interval++) { + win_t w = {.interval = interval}; g_hz = hz; g_display_mode = display; g_uncapped = uncapped; + atomic_store(&g_uncapped_interval, neg && atoi(neg) == 11 ? 0 : on); + unsigned expected = display || (uncapped && on && !dm_detclock_enabled()) || interval <= 0 ? 1u : (unsigned)interval; + if (present_ticks(&w) != expected || present_period(&w) != (uint64_t)expected * 1000000000ull / hz) + die("present interval probe: effective pacing differs from expected"); + checked++; + } + /* Frame interpolation modes (key 2 / flag mfx_interp, taken over at the frame boundary). Half the refresh rate: two + display ticks per game frame at the display rate, also when the uncapped mode is configured (suspended). 30+30: + exactly 1/30 s, by display ticks where the rate is a multiple of 30 and by our own clock otherwise (100 Hz). + Uncapped interpolation: the same pacing as without interpolation. Negative control 20: the request is never taken + over, so the uncapped pacing stays and the half-rate check fails. */ + extern void mfx_set(int mode); extern int mfx_latch(void); + int saved_paced = g_paced_hz, saved_mfx = mfx_enabled(); + mfx_set(MFX_HALF); if (!(neg && atoi(neg) == 20)) mfx_latch(); + for (int uncapped = 0; uncapped < 2; uncapped++) for (int interval = -1; interval <= 3; interval++) { + win_t w = {.interval = interval}; g_paced_hz = 100; g_hz = uncapped ? UNCAPPED_HZ : 100; g_display_mode = 0; g_uncapped = uncapped; + if (present_ticks(&w) != 2 || present_period(&w) != 2 * 1000000000ull / 100 || uncapped_now() || mfx_own_clock()) + die("present interval probe: frame interpolation must pace two display ticks at the display rate"); + checked++; + } + mfx_set(MFX_30); mfx_latch(); + const int rates[] = {60, 100, 120, 144}; const unsigned ticks30[] = {2, 3, 4, 5}; + for (int r = 0; r < 4; r++) for (int uncapped = 0; uncapped < 2; uncapped++) { + win_t w = {.interval = 1}; g_paced_hz = rates[r]; g_hz = uncapped ? UNCAPPED_HZ : rates[r]; g_display_mode = 0; g_uncapped = uncapped; + if (present_period(&w) != 1000000000ull / 30 || present_ticks(&w) != ticks30[r] || uncapped_now() || mfx_own_clock() != (rates[r] % 30 != 0)) + die("present interval probe: 30+30 must pace exactly 30 fps (display ticks at 60 / 120 Hz, our own clock at 100 / 144 Hz)"); + checked++; + } + mfx_set(MFX_MAX); mfx_latch(); + for (int uncapped = 0; uncapped < 2; uncapped++) for (int interval = -1; interval <= 3; interval++) { + win_t w = {.interval = interval}; g_paced_hz = 100; g_hz = uncapped ? UNCAPPED_HZ : 100; g_display_mode = 0; g_uncapped = uncapped; + atomic_store(&g_uncapped_interval, 0); + unsigned expected = interval <= 0 ? 1u : (unsigned)interval; + if (present_ticks(&w) != expected || present_period(&w) != (uint64_t)expected * 1000000000ull / (uncapped ? UNCAPPED_HZ : 100) || uncapped_now() != uncapped || mfx_own_clock()) + die("present interval probe: uncapped interpolation must keep the pacing of the mode without interpolation"); + checked++; + } + mfx_set(saved_mfx); mfx_latch(); + g_hz = saved_hz; g_paced_hz = saved_paced; g_display_mode = saved_display; g_uncapped = saved_uncapped; + atomic_store(&g_uncapped_interval, saved_flag); + LOG("[present-interval] %d pacing cases passed", checked); +} +int (*g_nvn_loadscreen_up)(void); // 宿主登记：读取画面此刻是否开着（diag.c 订阅界面消息时填；图形单测里为空 = 不设闸） | host registration: whether the loading screen is up right now (filled in when diag.c subscribes to UI messages; null in graphics unit tests = no gate) +// The present itself: screenshot hook, then the present queue (returns at once). frame != 0 (translation thread): also the +// replay frame number, which the synchronous path sets at the end of nvnQueuePresentTexture. +static void present_exec(win_t *o, int idx, double period, const int crop[4], uint64_t frame) { + gfx_enter(); + { void mtl_shot_at_present(const tex_t *t); if (idx >= 0 && idx < o->ntex && idx < 4) mtl_shot_at_present(CTX(o->tex[idx])); } + if (idx >= 0 && idx < o->ntex && idx < 4) o->ticket[idx] = mtl_present(CTX(o->tex[idx]), period, (int *)crop); // 只入队，立即返回 | enqueue only, return immediately + if (frame) mtl_set_frame(frame); + gfx_leave(); +} +NVN(nvnQueuePresentTexture, void, NVNqueue *q, NVNwindow *w, int idx) { + NVN_BLOCKING(); + (void)q; win_t *o = (win_t *)w; o->presents++; + totk_tl_frame(o->presents); totk_tl(TL_PRES_IN, (uint64_t)idx, 0); + if (runtime_frame_step_update) runtime_frame_step_update(now_ns()); + if (movie_host_tick) movie_host_tick(); + game_work_tick(o->presents); + static uint64_t prev_in, prev_out, prev_pass, prev_draw; + // D2R installs g_nvn_loadscreen_up from its UI-message observer. The TOTK + // native host has no such observer, so also recognize the first world frame + // from its draw count. The title transition briefly hit 1473 draws in a + // measured run; the first world frames exceeded 2000, so >1000 would + // mistakenly stall the title rather than the save-loading screen. + // Hold that first frame behind the loading screen if cache warm-up is still + // active. This does not claim that an undiscovered variant cannot appear. + { static int gate; static uint64_t last_draws; + uint64_t passes, draws; mtl_stats(&passes, &draws); + uint64_t this_draws = draws >= last_draws ? draws - last_draws : 0; + last_draws = draws; + if (!gate && ((g_nvn_loadscreen_up && g_nvn_loadscreen_up()) || this_draws > 2000)) { + gate = 1; extern void mtl_note_world(void); mtl_note_world(); int ms = mtl_warm_wait(15000); + if (ms) LOG("[nvn] 首次进局等管线预热 %d ms（本帧绘制 %llu）", ms, (unsigned long long)this_draws); + } } + present_cfg(o); + uint64_t t0 = now_ns(); + if (async_replay()) { // TOTK: the present (and the replay frame number below) run on the translation thread, after this frame's submits + xjob j = {.kind = XJ_PRESENT, .win = o, .idx = idx, .period = present_period(o) / 1e9}; + memcpy(j.crop, o->crop, sizeof j.crop); + j.frame = g_frame + 1; // dump_frame below advances g_frame before mtl_set_frame + uint64_t no = x_push(&j); + if (idx >= 0 && idx < 4) { g_xp_win = o; g_xp_job[idx] = no; } + } else present_exec(o, idx, present_period(o) / 1e9, o->crop, 0); + atomic_fetch_add(&g_nvn_ns, now_ns() - t0); + uint64_t f = g_frame, ps, dr; mtl_stats(&ps, &dr); + trow(f, "@present_guest_ticks", o->interval); + trow(f, "@present_effective_ticks", present_ticks(o)); + if (nvn_real[0] || nvn_real[1]) { // DM_NVN_PROF：录制期 = 除提交外全部已实现函数耗时；提交期 = QueueSubmitCommands | DM_NVN_PROF: record phase = time of all implemented functions except submit; submit phase = QueueSubmitCommands + static unsigned isub = ~0u; static mach_timebase_info_data_t tb; if (!tb.denom) mach_timebase_info(&tb); + if (isub == ~0u) for (isub = 0; isub < NVN_NFUNC && strcmp(nvn_names[isub], "nvnQueueSubmitCommands"); isub++) {} + uint64_t rec = 0, sub = 0; + static uint64_t acc[NVN_NFUNC]; + for (unsigned i = 0; i < NVN_NFUNC; i++) { uint64_t t = atomic_exchange(&nvn_prof_ticks[i], 0); acc[i] += t; if (i == isub) sub = t; else rec += t; } + if ((g_frame + 1) % TWIN == 0) { // 每 300 帧列出耗时最多的 5 个函数 | every 300 frames list the 5 most time-consuming functions + char line[1024]; size_t o = 0; + for (int k = 0; k < 5; k++) { unsigned best = 0; for (unsigned i = 1; i < NVN_NFUNC; i++) if (acc[i] > acc[best]) best = i; + if (!acc[best]) break; o += snprintf(line + o, sizeof line - o, " %s %.2f ms；", nvn_names[best] + 3, acc[best] * tb.numer / tb.denom / 1e6); acc[best] = 0; } + LOG("[nvn] 翻译层耗时前 5（近 300 帧合计）：%s", line); memset(acc, 0, sizeof acc); + } + trow(g_frame, "@rec_ms", rec * tb.numer / tb.denom / 1e6); trow(g_frame, "@submit_ms", sub * tb.numer / tb.denom / 1e6); + tpush(4, rec * tb.numer / tb.denom / 1e6); tpush(5, sub * tb.numer / tb.denom / 1e6); + } + if (prev_in) { + double fr = (t0 - prev_in) / 1e6, wk = (t0 - prev_out) / 1e6, nv = atomic_exchange(&g_nvn_ns, 0) / 1e6; + tpush(0, fr); tpush(1, wk); tpush(2, nv); + trow(f, "@frame_ms", fr); trow(f, "@work_ms", wk); trow(f, "@nvn_ms", nv); + // 慢帧归因（只在开 DM_NVN_TSV 时取）：本帧出图线程 CPU 时间（换线程记 −1）、进程全部线程 CPU、取图等待按帧累计、出图调用时刻（与 GPU / 上屏时刻同基：进程起点 ms） + // Slow-frame attribution (taken only when DM_NVN_TSV is on): this frame's present-thread CPU time (−1 if the thread changed), CPU of all threads in the process, acquire wait accumulated per frame, present call time (same base as the GPU / on-screen present times: ms since process start) + trow(f, "@wait_ms", g_acq_wait_ns / 1e6); trow(f, "@pq_wait_ms", g_pq_wait_ns / 1e6); + if (frame_rec()) { + static pthread_t pth; static uint64_t pcpu, pproc; + uint64_t c = clock_gettime_nsec_np(CLOCK_THREAD_CPUTIME_ID); pthread_t th = pthread_self(); + trow(f, "@cpu_ms", pcpu && pthread_equal(th, pth) ? (c - pcpu) / 1e6 : -1); pth = th; pcpu = c; + struct rusage ru; getrusage(RUSAGE_SELF, &ru); + uint64_t pc = (uint64_t)(ru.ru_utime.tv_sec + ru.ru_stime.tv_sec) * 1000000000ull + (uint64_t)(ru.ru_utime.tv_usec + ru.ru_stime.tv_usec) * 1000ull; + if (pproc) trow(f, "@proc_cpu_ms", (pc - pproc) / 1e6); pproc = pc; + trow(f, "@call_ms", (t0 - g_t0_ns) / 1e6); + } + trow(f, "@passes", (double)(ps - prev_pass)); trow(f, "@draws", (double)(dr - prev_draw)); + static uint64_t pl, pst, pf, pudc; uint64_t l, st, fu; mtl_bw(&l, &st, &fu); + trow(f, "@rt_load_mb", (l - pl) / 1048576.0); trow(f, "@rt_store_mb", (st - pst) / 1048576.0); trow(f, "@pass_full_rw", (double)(fu - pf)); + uint64_t udc = mtl_udc_saved_bytes(); trow(f, "@unwritten_dc_mb", (udc - pudc) / 1048576.0); + pl = l; pst = st; pf = fu; pudc = udc; + static uint64_t pae, par; uint64_t ae_, ar_; mtl_args_stats(&ae_, &ar_); + trow(f, "@args_enc", (double)(ae_ - pae)); trow(f, "@args_reuse", (double)(ar_ - par)); pae = ae_; par = ar_; + static uint64_t puc, pus; uint64_t uc_, us_; mtl_use_stats(&uc_, &us_); + static uint64_t pdo; uint64_t do_ = mtl_ds_omit(); trow(f, "@ds_omit", (double)(do_ - pdo)); pdo = do_; + trow(f, "@use_call", (double)(uc_ - puc)); trow(f, "@use_skip", (double)(us_ - pus)); puc = uc_; pus = us_; + static uint64_t pfa, pre_; uint64_t fa_, pr_, up_, re_; nvn_dirty_stats(&fa_, &pr_); mtl_pkg_stats(&up_, &re_); + trow(f, "@faults", (double)(fa_ - pfa)); trow(f, "@reuploads", (double)(re_ - pre_)); pfa = fa_; pre_ = re_; + static uint64_t pmc, pmp, pmn; uint64_t mc_, mpg_, mn_; nvn_dirty_mp_stats(&mc_, &mpg_, &mn_); + trow(f, "@mprotect_calls", (double)(mc_ - pmc)); trow(f, "@mprotect_pages", (double)(mpg_ - pmp)); trow(f, "@mprotect_ms", (mn_ - pmn) / 1e6); pmc = mc_; pmp = mpg_; pmn = mn_; + double up_ms = mtl_upload_ns_take() / 1e6; trow(f, "@upload_ms", up_ms); + trow(f, "@upload_gpu_ms", mtl_upload_gpu_ns_take() / 1e6); + // 分账：本帧提交线程同步创建数 + 与 UIMSG 同一时钟的时刻（离线按读取画面区间把帧分成 前端 / 读取 / 局内） + // Accounting: number of synchronous creations on the submit thread this frame + a timestamp on the same clock as UIMSG (offline, frames are split into front end / loading / in-run by the loading screen intervals) + static uint64_t plib, ppso; uint64_t lib_, pso_; mtl_create_stats(&lib_, &pso_); + trow(f, "@lib_new", (double)(lib_ - plib)); trow(f, "@pso_new", (double)(pso_ - ppso)); + { // 长帧日志：本帧各项先记进环，等 8 帧后 GPU 时间到了再决定打不打（见下） | long-frame log: this frame's items are first recorded into the ring; whether to print is decided 8 frames later, once the GPU time has arrived (see below) + static uint64_t plns, ppns, pup, prup, pub; uint64_t lns, pns, up, rup, ub; mtl_create_ns(&lns, &pns); mtl_pkg_stats(&up, &rup); ub = mtl_upload_bytes(); + lf_rec *r = &g_lf[f % LFN]; r->f = f; r->fr = fr; r->nv = nv; r->ls = g_nvn_loadscreen_up ? g_nvn_loadscreen_up() : 0; r->t_s = (now_ns() - g_t0_ns) / 1e9; r->pso = (unsigned)(pso_ - ppso); r->lib = (unsigned)(lib_ - plib); + r->make_ms = ((lns - plns) + (pns - ppns)) / 1e6; r->up = (unsigned)(up - pup); r->rup = (unsigned)(rup - prup); r->up_mb = (ub - pub) / 1048576.0; r->up_ms = up_ms; + plns = lns; ppns = pns; pup = up; prup = rup; pub = ub; + } + plib = lib_; ppso = pso_; + trow(f, "@t_s", (now_ns() - g_t0_ns) / 1e9); + trow(f, "@drawable_ms", mtl_drawable_ns_take() / 1e6); + trow(f, "@sub_wall_ms", atomic_exchange(&g_sub_wall_ns, 0) / 1e6); trow(f, "@sub_cpu_ms", atomic_exchange(&g_sub_cpu_ns, 0) / 1e6); trow(f, "@sub_instr", (double)atomic_exchange(&g_sub_instr, 0)); + } else atomic_store(&g_nvn_ns, 0); + g_acq_wait_ns = g_pq_wait_ns = 0; + if (f >= 8) { double g = mtl_gpu_ms(f - 8); if (g >= 0) { tpush(3, g); trow(f - 8, "@gpu_ms", g); g_hr[g_hn++ % HRN] = g; } // GPU 时间滞后 8 帧取（已完成） | GPU time is taken 8 frames late (already completed) + { lf_rec *r = &g_lf[(f - 8) % LFN]; // 长帧日志（缺省开，DM_LONGFRAME=0 关）：整帧 > 20 ms 或 GPU > 16.7 ms、且明显高于近况，才打一行 | long-frame log (on by default, DM_LONGFRAME=0 turns it off): a line is printed only when the whole frame > 20 ms or GPU > 16.7 ms, and it is clearly above recent conditions + static int on = -1; if (on < 0) { const char *e = getenv("DM_LONGFRAME"); on = !(e && !strcmp(e, "0")); } + // 「长」是相对近况说的：稳定跑在 30 帧（菜单 / 无窗口宿主）或持续过载时每帧都 > 20 ms，那不是长帧，不刷屏——要比近 16 帧的滑动均值高出四成才算 + // "Long" is relative to recent conditions: when running steadily at 30 fps (menus / windowless host) or under sustained overload, every frame is > 20 ms; those are not long frames and must not flood the log - a frame has to be 40% above the sliding mean of the last 16 frames to count + static double ef = 16.7, eg = 8; int lf = r->fr > 20.0 && r->fr > 1.4 * ef, lg = g > 16.7 && g > 1.3 * eg; + if (r->f == f - 8) { ef += ((r->fr < 3 * ef ? r->fr : 3 * ef) - ef) / 16; eg += ((g < 3 * eg ? g : 3 * eg) - eg) / 16; } + if (on && r->f == f - 8 && f - 8 > 0 && (lf || lg)) { char b[512]; + snprintf(b, sizeof b, "帧 %llu（%.2f s）整帧 %.1f ms，提交+出图 %.1f ms，GPU %.1f ms；新建管线 %u 条 + 着色器库 %u 个共 %.1f ms；上传贴图 首传 %u 张 + 重传 %u 次 %.1f MB / %.1f ms", + (unsigned long long)r->f, r->t_s, r->fr, r->nv, g, r->pso, r->lib, r->make_ms, r->up, r->rup, r->up_mb, r->up_ms); + // 可见性：长帧「晚出来」的那段时间屏幕上停着的是上一帧；上一帧若是读取画面（或本帧自己就是），玩家看到的只是读取画面多停了一下，没有运动被打断 + // Visibility: during the time a long frame "comes out late", what stays on screen is the previous frame; if the previous frame was a loading screen (or this frame itself is one), all the player sees is the loading screen staying a bit longer, with no motion interrupted + const lf_rec *pv = &g_lf[(f - 9) % LFN]; int hidden = r->ls || (f >= 9 && pv->f == f - 9 && pv->ls); + if (hidden) strlcat(b, "；接读取画面，不可见", sizeof b); + dm_longframe_note(b); } } + uint64_t g0, g1; if (frame_rec() && mtl_gpu_span(f - 8, &g0, &g1)) { trow(f - 8, "@gpu_beg_ms", (g0 - g_t0_ns) / 1e6); trow(f - 8, "@gpu_end_ms", (g1 - g_t0_ns) / 1e6); } } + prev_in = t0; prev_pass = ps; prev_draw = dr; + dump_frame(); + if (g_frame % TWIN == 0) tsummary(g_frame); + if (call_stats_on() && g_frame % 300 == 0) { + static uint64_t pd; uint64_t ps2, dr2; mtl_stats(&ps2, &dr2); uint64_t all = dr2 - pd; pd = dr2; + LOG("[nvn] 调用列表 近 300 帧：每帧绘制 %.0f，其中来自被调用列表 %.0f（%.0f%%），与上一帧完全相同的列表里 %.0f（%.0f%%）；每帧调用 %.0f 次 / %.0f KB，相同的 %.0f 次 / %.0f KB", + all / 300.0, g_cs_draws / 300.0, all ? 100.0 * g_cs_draws / all : 0, g_cs_same_draws / 300.0, all ? 100.0 * g_cs_same_draws / all : 0, + g_cs_calls / 300.0, g_cs_bytes / 300.0 / 1024, g_cs_same_calls / 300.0, g_cs_same_bytes / 300.0 / 1024); + g_cs_calls = g_cs_draws = g_cs_same_calls = g_cs_same_draws = g_cs_bytes = g_cs_same_bytes = 0; + } + if (!async_replay()) mtl_set_frame(g_frame); + else if (g_frame % 300 == 0) { uint64_t d = atomic_exchange(&g_x_drains, 0), dn = atomic_exchange(&g_x_drain_ns, 0); + pthread_mutex_lock(&g_xq_mu); uint64_t q = g_xq_in - g_xq_done; pthread_mutex_unlock(&g_xq_mu); + LOG("[nvn] 翻译线程 至帧 %llu：排空等待 %llu 次 / %.1f ms（近 300 帧），队列中 %llu 项", (unsigned long long)g_frame, (unsigned long long)d, dn / 1e6, (unsigned long long)q); } + // 真机语义：出图只入队、立即返回；本帧显示时刻按节拍器排，该贴图到「显示时刻 + 一个周期」才可再获取（阻塞在 AcquireTexture） + // Real-device semantics: present only enqueues and returns immediately; this frame's display time is scheduled by the pacer, and the texture can be acquired again only at "display time + one period" (blocking in AcquireTexture) + uint64_t period = present_period(o), now = dm_detclock_enabled() ? guest_ns() : now_ns(); + nvn_pacer pc = {o->pace}; + uint64_t slot = now + nvn_pace(&pc, now, period); + o->pace = pc.target; + if (idx >= 0 && idx < 4) o->avail[idx] = period ? slot + period : 0; + static uint64_t prev_slot; if (prev_slot && period) trow(f, "@display_ms", (slot - prev_slot) / 1e6); prev_slot = slot; // 屏幕上相邻两帧的显示间隔（掉帧看这个） | display interval between two adjacent frames on screen (this is the one to look at for dropped frames) + // 实际呈现间隔（窗口模式）：滞后 8 次出图取（呈现回调已到），第 n 次与第 n−1 次之差；−1 = 被丢弃 + // Actual presentation interval (windowed mode): taken 8 presents late (the presented callback has arrived by then), as the difference between the n-th and the (n−1)-th; −1 = discarded + if (o->presents > 9) { uint64_t n = o->presents - 8; double a_ = mtl_presented_time(n), b_ = mtl_presented_time(n - 1); + if (a_ < 0) trow(f - 8, "@presented_ms", -1); else if (a_ > 0 && b_ > 0) trow(f - 8, "@presented_ms", (a_ - b_) * 1e3); + g_pres_iv[g_presn++ % TWIN] = a_ < 0 ? -1 : (a_ > 0 && b_ > 0 ? (a_ - b_) * 1e3 : 0); + { double iv_ = a_ < 0 ? -1 : (a_ > 0 && b_ > 0 ? (a_ - b_) * 1e3 : 0); mtl_ub_present(iv_); } // 同进程交替测量（DM_NVN_UB）按相统计上屏间隔；没开时只是一次原子读 | same-process alternating measurement (DM_NVN_UB) tallies on-screen present intervals per phase; when off it is just one atomic read + dm_perf_present(a_ < 0 ? -1 : (a_ > 0 && b_ > 0 ? (a_ - b_) * 1e3 : 0)); // 上屏间隔另存一圈，tsummary 每 300 帧报掉帧（M1 判据） | on-screen present intervals are also kept in a separate ring; tsummary reports dropped frames every 300 frames (M1 criterion) + if (a_ > 0) trow(f - 8, "@pres_at_ms", a_ * 1e3 - g_t0_ns / 1e6); } // 上屏时刻（进程起点 ms；presentedTime 与 mach_absolute_time 同基） | on-screen present time (ms since process start; presentedTime shares a time base with mach_absolute_time) + if (f >= 8 && slow_on()) slow_emit(f - 8); + prev_out = now_ns(); + totk_tl(TL_PRES_OUT, 0, 0); +} diff --git a/src/gfx/nvn/include/nvn.h b/src/gfx/nvn/include/nvn.h new file mode 100644 index 0000000..57bdef0 --- /dev/null +++ b/src/gfx/nvn/include/nvn.h @@ -0,0 +1,1757 @@ +/* + * Copyright (c) 2020 h1k421 + * + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the "Software"), to deal + * in the Software without restriction, including without limitation the rights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +// NVN types reconstructed from Darksiders II DWARF information. +// NOTE: This only includes enums and structs used for NVN 53.105 but might contain more elements +// than expected (as Darksiders II uses a newer version of NVN). +// TODO: Include the extra information from the version used on Darksiders II for completeness. + +#pragma once + +#ifdef __cplusplus +extern "C" { +#endif + +#include +#include + +typedef uint8_t NVNboolean; +typedef uint64_t NVNtextureHandle; +typedef uint64_t NVNimageHandle; +typedef uint64_t NVNcommandHandle; +typedef uint64_t NVNbufferAddress; +typedef uint64_t NVNtextureAddress; +typedef int NVNdebugDomainId; +typedef int NVNstorageClass; +typedef void* NVNsubroutineLinkageMapPtr; +typedef void* NVNnativeWindow; + +typedef struct { + char reserved[0x3000]; +} NVNdevice; + +typedef struct { + char reserved[0x40]; +} NVNdeviceBuilder; + +typedef struct { + // TODO: reverse and define this as it looks like it's not an opaque type. + char dummy[1]; +} NVNtextureSparseTileLayout; + +typedef struct { + // TODO: reverse and define this as it looks like it's not an opaque type. + char dummy[1]; +} NVNdrawTextureRegion; + +typedef struct { + char reserved[0x18]; +} NVNmultisampleState; + +typedef struct { + char reserved[0x8]; +} NVNdepthStencilState; + +typedef struct { + char reserved[0x4]; +} NVNchannelMaskState; + +typedef struct { + char reserved[0x4]; +} NVNpolygonState; + +typedef struct { + NVNbufferAddress address; + uint64_t size; +} NVNbufferRange; + +typedef struct { + unsigned char reserved[4]; +} NVNcolorState; + +typedef struct { + uint64_t counter; + uint64_t timestamp; +} NVNcounterData; + +typedef struct { + uint64_t value; +} NVNseparateTextureHandle; + +typedef struct { + uint64_t value; +} NVNseparateSamplerHandle; + +typedef struct { + char reserved[0x2000]; +} NVNqueue; + +typedef struct { + // TODO: reverse and define this as it looks like it's not an opaque type. + char dummy[1]; +} NVNqueueErrorInfo; + +typedef struct { + char reserved[0x40]; +} NVNqueueBuilder; + +typedef struct { + char reserved[384]; +} NVNwindow; + +typedef struct { + char reserved[0x80]; /* TOTK v0 main+214c964 allocates 128 bytes. */ +} NVNtexture; + +typedef struct { + char reserved[0x10]; /* TOTK NVN 55: 16 bytes (game stack layout at main+0x823398) */ +} NVNtextureView; + +typedef struct { + char reserved[0xA0]; +} NVNcommandBuffer; + +typedef struct { + char reserved[0x40]; +} NVNsync; + +typedef struct { + int x; + int y; + int width; + int height; +} NVNrectangle; + +typedef struct { + char reserved[0xC0]; +} NVNprogram; + +typedef struct { + NVNbufferAddress data; + const void* control; +} NVNshaderData; + +typedef struct { + char reserved[0x100]; +} NVNmemoryPool; + +typedef struct { + char reserved[0x20]; +} NVNtexturePool; + +typedef struct { + char reserved[0x20]; +} NVNsamplerPool; + +typedef struct { + char reserved[0x8]; +} NVNblendState; + +typedef struct { + char reserved[0x4]; +} NVNvertexAttribState; + +typedef struct { + char reserved[0x8]; +} NVNvertexStreamState; + +typedef struct { + char reserved[0x40]; +} NVNmemoryPoolBuilder; + +typedef struct { + // TODO: reverse and define this as it looks like it's not an opaque type. + char dummy[1]; +} NVNmappingRequest; + +typedef struct { + char reserved[0x60]; +} NVNsamplerBuilder; + +typedef struct { + char reserved[0x30]; +} NVNsampler; + +typedef struct { + char reserved[0x40]; +} NVNbufferBuilder; + +typedef struct { + char reserved[0x30]; +} NVNbuffer; + +typedef struct { + char reserved[0x40]; +} NVNevent; + +typedef struct { + char reserved[0x20]; +} NVNeventBuilder; + +typedef struct { + int xoffset; + int yoffset; + int zoffset; + int width; + int height; + int depth; +} NVNcopyRegion; + +typedef struct { + char reserved[0x40]; +} NVNwindowBuilder; + +typedef struct { + char reserved[0x80]; +} NVNtextureBuilder; + +typedef struct { + unsigned char layout[8]; +} NVNpackagedTextureLayout; + +typedef enum { + NVN_DEVICE_INFO_API_MAJOR_VERSION = 0x0, + NVN_DEVICE_INFO_API_MINOR_VERSION = 0x1, + NVN_DEVICE_INFO_UNIFORM_BUFFER_BINDINGS_PER_STAGE = 0x2, + NVN_DEVICE_INFO_MAX_UNIFORM_BUFFER_SIZE = 0x3, + NVN_DEVICE_INFO_UNIFORM_BUFFER_ALIGNMENT = 0x4, + NVN_DEVICE_INFO_COLOR_BUFFER_BINDINGS = 0x5, + NVN_DEVICE_INFO_VERTEX_BUFFER_BINDINGS = 0x6, + NVN_DEVICE_INFO_TRANSFORM_FEEDBACK_BUFFER_BINDINGS = 0x7, + NVN_DEVICE_INFO_SHADER_STORAGE_BUFFER_BINDINGS_PER_STAGE = 0x8, + NVN_DEVICE_INFO_TEXTURE_BINDINGS_PER_STAGE = 0x9, + NVN_DEVICE_INFO_COUNTER_ALIGNMENT = 0xA, + NVN_DEVICE_INFO_TRANSFORM_FEEDBACK_BUFFER_ALIGNMENT = 0xB, + NVN_DEVICE_INFO_TRANSFORM_FEEDBACK_CONTROL_ALIGNMENT = 0xC, + NVN_DEVICE_INFO_INDIRECT_DRAW_ALIGNMENT = 0xD, + NVN_DEVICE_INFO_VERTEX_ATTRIBUTES = 0xE, + NVN_DEVICE_INFO_TEXTURE_DESCRIPTOR_SIZE = 0xF, + NVN_DEVICE_INFO_SAMPLER_DESCRIPTOR_SIZE = 0x10, + NVN_DEVICE_INFO_RESERVED_TEXTURE_DESCRIPTORS = 0x11, + NVN_DEVICE_INFO_RESERVED_SAMPLER_DESCRIPTORS = 0x12, + NVN_DEVICE_INFO_COMMAND_BUFFER_COMMAND_ALIGNMENT = 0x13, + NVN_DEVICE_INFO_COMMAND_BUFFER_CONTROL_ALIGNMENT = 0x14, + NVN_DEVICE_INFO_COMMAND_BUFFER_MIN_COMMAND_SIZE = 0x15, + NVN_DEVICE_INFO_COMMAND_BUFFER_MIN_CONTROL_SIZE = 0x16, + NVN_DEVICE_INFO_SHADER_SCRATCH_MEMORY_SCALE_FACTOR_MINIMUM = 0x17, + NVN_DEVICE_INFO_SHADER_SCRATCH_MEMORY_SCALE_FACTOR_RECOMMENDED = 0x18, + NVN_DEVICE_INFO_SHADER_SCRATCH_MEMORY_ALIGNMENT = 0x19, + NVN_DEVICE_INFO_SHADER_SCRATCH_MEMORY_GRANULARITY = 0x1A, + NVN_DEVICE_INFO_MAX_TEXTURE_ANISOTROPY = 0x1B, + NVN_DEVICE_INFO_MAX_COMPUTE_WORK_GROUP_SIZE_X = 0x1C, + NVN_DEVICE_INFO_MAX_COMPUTE_WORK_GROUP_SIZE_Y = 0x1D, + NVN_DEVICE_INFO_MAX_COMPUTE_WORK_GROUP_SIZE_Z = 0x1E, + NVN_DEVICE_INFO_MAX_COMPUTE_WORK_GROUP_SIZE_THREADS = 0x1F, + NVN_DEVICE_INFO_MAX_COMPUTE_DISPATCH_WORK_GROUPS_X = 0x20, + NVN_DEVICE_INFO_MAX_COMPUTE_DISPATCH_WORK_GROUPS_Y = 0x21, + NVN_DEVICE_INFO_MAX_COMPUTE_DISPATCH_WORK_GROUPS_Z = 0x22, + NVN_DEVICE_INFO_IMAGE_BINDINGS_PER_STAGE = 0x23, + NVN_DEVICE_INFO_MAX_TEXTURE_POOL_SIZE = 0x24, + NVN_DEVICE_INFO_MAX_SAMPLER_POOL_SIZE = 0x25, + NVN_DEVICE_INFO_MAX_VIEWPORTS = 0x26, + NVN_DEVICE_INFO_MEMPOOL_TEXTURE_OBJECT_PAGE_ALIGNMENT = 0x27, + NVN_DEVICE_INFO_SUPPORTS_MIN_MAX_FILTERING = 0x28, + NVN_DEVICE_INFO_SUPPORTS_STENCIL8_FORMAT = 0x29, + NVN_DEVICE_INFO_SUPPORTS_ASTC_FORMATS = 0x2A, + NVN_DEVICE_INFO_L2_SIZE = 0x2B, + NVN_DEVICE_INFO_MAX_TEXTURE_LEVELS = 0x2C, + NVN_DEVICE_INFO_MAX_TEXTURE_LAYERS = 0x2D, + NVN_DEVICE_INFO_GLSLC_MAX_SUPPORTED_GPU_CODE_MAJOR_VERSION = 0x2E, + NVN_DEVICE_INFO_GLSLC_MIN_SUPPORTED_GPU_CODE_MAJOR_VERSION = 0x2F, + NVN_DEVICE_INFO_GLSLC_MAX_SUPPORTED_GPU_CODE_MINOR_VERSION = 0x30, + NVN_DEVICE_INFO_GLSLC_MIN_SUPPORTED_GPU_CODE_MINOR_VERSION = 0x31, + NVN_DEVICE_INFO_SUPPORTS_CONSERVATIVE_RASTER = 0x32, + NVN_DEVICE_INFO_SUBPIXEL_BITS = 0x33, + NVN_DEVICE_INFO_MAX_SUBPIXEL_BIAS_BITS = 0x34, + NVN_DEVICE_INFO_INDIRECT_DISPATCH_ALIGNMENT = 0x35, + NVN_DEVICE_INFO_ZCULL_SAVE_RESTORE_ALIGNMENT = 0x36, + NVN_DEVICE_INFO_SHADER_SCRATCH_MEMORY_COMPUTE_SCALE_FACTOR_MINIMUM = 0x37, + NVN_DEVICE_INFO_LINEAR_TEXTURE_STRIDE_ALIGNMENT = 0x38, + NVN_DEVICE_INFO_LINEAR_RENDER_TARGET_STRIDE_ALIGNMENT = 0x39, + NVN_DEVICE_INFO_MEMORY_POOL_PAGE_SIZE = 0x3A, + NVN_DEVICE_INFO_SUPPORTS_ZERO_FROM_UNMAPPED_VIRTUAL_POOL_PAGES = 0x3B, + NVN_DEVICE_INFO_UNIFORM_BUFFER_UPDATE_ALIGNMENT = 0x3C, + NVN_DEVICE_INFO_MAX_TEXTURE_SIZE = 0x3D, + NVN_DEVICE_INFO_MAX_BUFFER_TEXTURE_SIZE = 0x3E, + NVN_DEVICE_INFO_MAX_3D_TEXTURE_SIZE = 0x3F, + NVN_DEVICE_INFO_MAX_CUBE_MAP_TEXTURE_SIZE = 0x40, + NVN_DEVICE_INFO_MAX_RECTANGLE_TEXTURE_SIZE = 0x41, + NVN_DEVICE_INFO_SUPPORTS_PASSTHROUGH_GEOMETRY_SHADERS = 0x42, + NVN_DEVICE_INFO_SUPPORTS_VIEWPORT_SWIZZLE = 0x43, + NVN_DEVICE_INFO_SUPPORTS_SPARSE_TILED_PACKAGED_TEXTURES = 0x44, + NVN_DEVICE_INFO_SUPPORTS_ADVANCED_BLEND_MODES = 0x45, + NVN_DEVICE_INFO_MAX_PRESENT_INTERVAL = 0x46, + NVN_DEVICE_INFO_SUPPORTS_DRAW_TEXTURE = 0x47, + NVN_DEVICE_INFO_SUPPORTS_TARGET_INDEPENDENT_RASTERIZATION = 0x48, + NVN_DEVICE_INFO_SUPPORTS_FRAGMENT_COVERAGE_TO_COLOR = 0x49, + NVN_DEVICE_INFO_SUPPORTS_POST_DEPTH_COVERAGE = 0x4A, + NVN_DEVICE_INFO_SUPPORTS_IMAGES_USING_TEXTURE_HANDLES = 0x4B, + NVN_DEVICE_INFO_SUPPORTS_SAMPLE_LOCATIONS = 0x4C, + NVN_DEVICE_INFO_MAX_SAMPLE_LOCATION_TABLE_ENTRIES = 0x4D, + NVN_DEVICE_INFO_SHADER_CODE_MEMORY_POOL_PADDING_SIZE = 0x4E, + NVN_DEVICE_INFO_MAX_PATCH_SIZE = 0x4F, + NVN_DEVICE_INFO_QUEUE_COMMAND_MEMORY_GRANULARITY = 0x50, + NVN_DEVICE_INFO_QUEUE_COMMAND_MEMORY_MIN_SIZE = 0x51, + NVN_DEVICE_INFO_QUEUE_COMMAND_MEMORY_DEFAULT_SIZE = 0x52, + NVN_DEVICE_INFO_QUEUE_COMPUTE_MEMORY_GRANULARITY = 0x53, + NVN_DEVICE_INFO_QUEUE_COMPUTE_MEMORY_MIN_SIZE = 0x54, + NVN_DEVICE_INFO_QUEUE_COMPUTE_MEMORY_DEFAULT_SIZE = 0x55, + NVN_DEVICE_INFO_QUEUE_COMMAND_MEMORY_MIN_FLUSH_THRESHOLD = 0x56, + NVN_DEVICE_INFO_SUPPORTS_FRAGMENT_SHADER_INTERLOCK = 0x57, + NVN_DEVICE_INFO_MAX_TEXTURES_PER_WINDOW = 0x58, + NVN_DEVICE_INFO_MIN_TEXTURES_PER_WINDOW = 0x59, + NVN_DEVICE_INFO_SUPPORTS_DEBUG_LAYER = 0x5A, + NVN_DEVICE_INFO_QUEUE_CONTROL_MEMORY_MIN_SIZE = 0x5B, + NVN_DEVICE_INFO_QUEUE_CONTROL_MEMORY_DEFAULT_SIZE = 0x5C, + NVN_DEVICE_INFO_QUEUE_CONTROL_MEMORY_GRANULARITY = 0x5D, + NVN_DEVICE_INFO_SEPARATE_TEXTURE_BINDINGS_PER_STAGE = 0x5E, + NVN_DEVICE_INFO_SEPARATE_SAMPLER_BINDINGS_PER_STAGE = 0x5F, + NVN_DEVICE_INFO_DEBUG_GROUPS_MAX_DOMAIN_ID = 0x60, + NVN_DEVICE_INFO_EVENTS_SUPPORT_REDUCTION_OPERATIONS = 0x61, + + NVN_DEVICE_INFO_LARGE = 0x7FFFFFFF +} NVNdeviceInfo; + +typedef enum { + NVN_DEVICE_FLAG_DEBUG_ENABLE = 0x1, + NVN_DEVICE_FLAG_DEBUG_SKIP_CALLS_ON_ERROR = 0x2, + NVN_DEVICE_FLAG_DEBUG_DRAW_VALIDATION = 0x4, + NVN_DEVICE_FLAG_DEFERRED_FINALIZE = 0x8, + NVN_DEVICE_FLAG_DEBUG_DRAW_VALIDATION_HEAVY = 0x10, + NVN_DEVICE_FLAG_DEBUG_ENABLE_LEVEL_0 = 0x20, + NVN_DEVICE_FLAG_DEBUG_ENABLE_LEVEL_1 = 0x40, + NVN_DEVICE_FLAG_DEBUG_ENABLE_LEVEL_2 = 0x1, + NVN_DEVICE_FLAG_DEBUG_ENABLE_LEVEL_3 = 0x4, + NVN_DEVICE_FLAG_DEBUG_ENABLE_LEVEL_4 = 0x10, + NVN_DEVICE_FLAG_DEFERRED_FIRMWARE_MEMORY_RECLAIM = 0x80, + NVN_DEVICE_FLAG_ENABLE_SEPARATE_SAMPLER_TEXTURE_SUPPORT = 0x100, + + NVN_DEVICE_FLAG_LARGE = 0x7FFFFFFF +} NVNdeviceFlagBits; + +enum NVNclearColorMask { + NVN_CLEAR_COLOR_MASK_R_BIT = 0x1, + NVN_CLEAR_COLOR_MASK_G_BIT = 0x2, + NVN_CLEAR_COLOR_MASK_B_BIT = 0x4, + NVN_CLEAR_COLOR_MASK_A_BIT = 0x8, + NVN_CLEAR_COLOR_MASK_RGBA = 0xF, + NVN_CLEAR_COLOR_MASK_LARGE = 0x7FFFFFFF +}; + +typedef enum { + NVN_DEBUG_CALLBACK_SOURCE_API = 0x0, + + NVN_DEBUG_CALLBACK_SOURCE_LARGE = 0x7FFFFFFF +} NVNdebugCallbackSource; + +typedef enum { + NVN_DEBUG_CALLBACK_TYPE_API_ERROR = 0x0, + NVN_DEBUG_CALLBACK_TYPE_API_WARNING = 0x1, + + NVN_DEBUG_CALLBACK_TYPE_LARGE = 0x7FFFFFFF +} NVNdebugCallbackType; + +typedef enum { + NVN_DEBUG_CALLBACK_SEVERITY_HIGH = 0x0, + NVN_DEBUG_CALLBACK_SEVERITY_MEDIUM = 0x1, + NVN_DEBUG_CALLBACK_SEVERITY_LOW = 0x2, + NVN_DEBUG_CALLBACK_SEVERITY_NOTIFICATION = 0x3, + + NVN_DEBUG_CALLBACK_SEVERITY_LARGE = 0x7FFFFFFF +} NVNdebugCallbackSeverity; + +typedef enum { + NVN_WINDOW_ORIGIN_MODE_LOWER_LEFT = 0x0, + NVN_WINDOW_ORIGIN_MODE_UPPER_LEFT = 0x1, + + NVN_WINDOW_ORIGIN_MODE_LARGE = 0x7FFFFFFF +} NVNwindowOriginMode; + +typedef enum { + NVN_DEPTH_MODE_NEAR_IS_MINUS_W = 0x0, + NVN_DEPTH_MODE_NEAR_IS_ZERO = 0x1, + + NVN_DEPTH_MODE_LARGE = 0x7FFFFFFF +} NVNdepthMode; + +typedef enum { + NVN_FORMAT_NONE = 0x0, + NVN_FORMAT_R8 = 0x1, + NVN_FORMAT_R8SN = 0x2, + NVN_FORMAT_R8UI = 0x3, + NVN_FORMAT_R8I = 0x4, + NVN_FORMAT_R16F = 0x5, + NVN_FORMAT_R16 = 0x6, + NVN_FORMAT_R16SN = 0x7, + NVN_FORMAT_R16UI = 0x8, + NVN_FORMAT_R16I = 0x9, + NVN_FORMAT_R32F = 0xA, + NVN_FORMAT_R32UI = 0xB, + NVN_FORMAT_R32I = 0xC, + NVN_FORMAT_RG8 = 0xD, + NVN_FORMAT_RG8SN = 0xE, + NVN_FORMAT_RG8UI = 0xF, + NVN_FORMAT_RG8I = 0x10, + NVN_FORMAT_RG16F = 0x11, + NVN_FORMAT_RG16 = 0x12, + NVN_FORMAT_RG16SN = 0x13, + NVN_FORMAT_RG16UI = 0x14, + NVN_FORMAT_RG16I = 0x15, + NVN_FORMAT_RG32F = 0x16, + NVN_FORMAT_RG32UI = 0x17, + NVN_FORMAT_RG32I = 0x18, + NVN_FORMAT_RGB8 = 0x19, + NVN_FORMAT_RGB8SN = 0x1A, + NVN_FORMAT_RGB8UI = 0x1B, + NVN_FORMAT_RGB8I = 0x1C, + NVN_FORMAT_RGB16F = 0x1D, + NVN_FORMAT_RGB16 = 0x1E, + NVN_FORMAT_RGB16SN = 0x1F, + NVN_FORMAT_RGB16UI = 0x20, + NVN_FORMAT_RGB16I = 0x21, + NVN_FORMAT_RGB32F = 0x22, + NVN_FORMAT_RGB32UI = 0x23, + NVN_FORMAT_RGB32I = 0x24, + NVN_FORMAT_RGBA8 = 0x25, + NVN_FORMAT_RGBA8SN = 0x26, + NVN_FORMAT_RGBA8UI = 0x27, + NVN_FORMAT_RGBA8I = 0x28, + NVN_FORMAT_RGBA16F = 0x29, + NVN_FORMAT_RGBA16 = 0x2A, + NVN_FORMAT_RGBA16SN = 0x2B, + NVN_FORMAT_RGBA16UI = 0x2C, + NVN_FORMAT_RGBA16I = 0x2D, + NVN_FORMAT_RGBA32F = 0x2E, + NVN_FORMAT_RGBA32UI = 0x2F, + NVN_FORMAT_RGBA32I = 0x30, + NVN_FORMAT_STENCIL8 = 0x31, + NVN_FORMAT_DEPTH16 = 0x32, + NVN_FORMAT_DEPTH24 = 0x33, + NVN_FORMAT_DEPTH32F = 0x34, + NVN_FORMAT_DEPTH24_STENCIL8 = 0x35, + NVN_FORMAT_DEPTH32F_STENCIL8 = 0x36, + NVN_FORMAT_RGBX8_SRGB = 0x37, + NVN_FORMAT_RGBA8_SRGB = 0x38, + NVN_FORMAT_RGBA4 = 0x39, + NVN_FORMAT_RGB5 = 0x3A, + NVN_FORMAT_RGB5A1 = 0x3B, + NVN_FORMAT_RGB565 = 0x3C, + NVN_FORMAT_RGB10A2 = 0x3D, + NVN_FORMAT_RGB10A2UI = 0x3E, + NVN_FORMAT_R11G11B10F = 0x3F, + NVN_FORMAT_RGB9E5F = 0x40, + NVN_FORMAT_RGB_DXT1 = 0x41, + NVN_FORMAT_RGBA_DXT1 = 0x42, + NVN_FORMAT_RGBA_DXT3 = 0x43, + NVN_FORMAT_RGBA_DXT5 = 0x44, + NVN_FORMAT_RGB_DXT1_SRGB = 0x45, + NVN_FORMAT_RGBA_DXT1_SRGB = 0x46, + NVN_FORMAT_RGBA_DXT3_SRGB = 0x47, + NVN_FORMAT_RGBA_DXT5_SRGB = 0x48, + NVN_FORMAT_RGTC1_UNORM = 0x49, + NVN_FORMAT_RGTC1_SNORM = 0x4A, + NVN_FORMAT_RGTC2_UNORM = 0x4B, + NVN_FORMAT_RGTC2_SNORM = 0x4C, + NVN_FORMAT_BPTC_UNORM = 0x4D, + NVN_FORMAT_BPTC_UNORM_SRGB = 0x4E, + NVN_FORMAT_BPTC_SFLOAT = 0x4F, + NVN_FORMAT_BPTC_UFLOAT = 0x50, + NVN_FORMAT_R8_UI2F = 0x51, + NVN_FORMAT_R8_I2F = 0x52, + NVN_FORMAT_R16_UI2F = 0x53, + NVN_FORMAT_R16_I2F = 0x54, + NVN_FORMAT_R32_UI2F = 0x55, + NVN_FORMAT_R32_I2F = 0x56, + NVN_FORMAT_RG8_UI2F = 0x57, + NVN_FORMAT_RG8_I2F = 0x58, + NVN_FORMAT_RG16_UI2F = 0x59, + NVN_FORMAT_RG16_I2F = 0x5A, + NVN_FORMAT_RG32_UI2F = 0x5B, + NVN_FORMAT_RG32_I2F = 0x5C, + NVN_FORMAT_RGB8_UI2F = 0x5D, + NVN_FORMAT_RGB8_I2F = 0x5E, + NVN_FORMAT_RGB16_UI2F = 0x5F, + NVN_FORMAT_RGB16_I2F = 0x60, + NVN_FORMAT_RGB32_UI2F = 0x61, + NVN_FORMAT_RGB32_I2F = 0x62, + NVN_FORMAT_RGBA8_UI2F = 0x63, + NVN_FORMAT_RGBA8_I2F = 0x64, + NVN_FORMAT_RGBA16_UI2F = 0x65, + NVN_FORMAT_RGBA16_I2F = 0x66, + NVN_FORMAT_RGBA32_UI2F = 0x67, + NVN_FORMAT_RGBA32_I2F = 0x68, + NVN_FORMAT_RGB10A2SN = 0x69, + NVN_FORMAT_RGB10A2I = 0x6A, + NVN_FORMAT_RGB10A2_UI2F = 0x6B, + NVN_FORMAT_RGB10A2_I2F = 0x6C, + NVN_FORMAT_RGBX8 = 0x6D, + NVN_FORMAT_RGBX8SN = 0x6E, + NVN_FORMAT_RGBX8UI = 0x6F, + NVN_FORMAT_RGBX8I = 0x70, + NVN_FORMAT_RGBX16F = 0x71, + NVN_FORMAT_RGBX16 = 0x72, + NVN_FORMAT_RGBX16SN = 0x73, + NVN_FORMAT_RGBX16UI = 0x74, + NVN_FORMAT_RGBX16I = 0x75, + NVN_FORMAT_RGBX32F = 0x76, + NVN_FORMAT_RGBX32UI = 0x77, + NVN_FORMAT_RGBX32I = 0x78, + NVN_FORMAT_RGBA_ASTC_4x4 = 0x79, + NVN_FORMAT_RGBA_ASTC_5x4 = 0x7A, + NVN_FORMAT_RGBA_ASTC_5x5 = 0x7B, + NVN_FORMAT_RGBA_ASTC_6x5 = 0x7C, + NVN_FORMAT_RGBA_ASTC_6x6 = 0x7D, + NVN_FORMAT_RGBA_ASTC_8x5 = 0x7E, + NVN_FORMAT_RGBA_ASTC_8x6 = 0x7F, + NVN_FORMAT_RGBA_ASTC_8x8 = 0x80, + NVN_FORMAT_RGBA_ASTC_10x5 = 0x81, + NVN_FORMAT_RGBA_ASTC_10x6 = 0x82, + NVN_FORMAT_RGBA_ASTC_10x8 = 0x83, + NVN_FORMAT_RGBA_ASTC_10x10 = 0x84, + NVN_FORMAT_RGBA_ASTC_12x10 = 0x85, + NVN_FORMAT_RGBA_ASTC_12x12 = 0x86, + NVN_FORMAT_RGBA_ASTC_4x4_SRGB = 0x87, + NVN_FORMAT_RGBA_ASTC_5x4_SRGB = 0x88, + NVN_FORMAT_RGBA_ASTC_5x5_SRGB = 0x89, + NVN_FORMAT_RGBA_ASTC_6x5_SRGB = 0x8A, + NVN_FORMAT_RGBA_ASTC_6x6_SRGB = 0x8B, + NVN_FORMAT_RGBA_ASTC_8x5_SRGB = 0x8C, + NVN_FORMAT_RGBA_ASTC_8x6_SRGB = 0x8D, + NVN_FORMAT_RGBA_ASTC_8x8_SRGB = 0x8E, + NVN_FORMAT_RGBA_ASTC_10x5_SRGB = 0x8F, + NVN_FORMAT_RGBA_ASTC_10x6_SRGB = 0x90, + NVN_FORMAT_RGBA_ASTC_10x8_SRGB = 0x91, + NVN_FORMAT_RGBA_ASTC_10x10_SRGB = 0x92, + NVN_FORMAT_RGBA_ASTC_12x10_SRGB = 0x93, + NVN_FORMAT_RGBA_ASTC_12x12_SRGB = 0x94, + NVN_FORMAT_BGR565 = 0x95, + NVN_FORMAT_BGR5 = 0x96, + NVN_FORMAT_BGR5A1 = 0x97, + NVN_FORMAT_A1BGR5 = 0x98, + NVN_FORMAT_BGRX8 = 0x99, + NVN_FORMAT_BGRA8 = 0x9A, + NVN_FORMAT_BGRX8_SRGB = 0x9B, + NVN_FORMAT_BGRA8_SRGB = 0x9C, + + NVN_FORMAT_LARGE = 0x7FFFFFFF +} NVNformat; + +typedef enum { + NVN_DEBUG_OBJECT_TYPE_WINDOW = 0x0, + NVN_DEBUG_OBJECT_TYPE_QUEUE = 0x1, + NVN_DEBUG_OBJECT_TYPE_COMMAND_BUFFER = 0x2, + NVN_DEBUG_OBJECT_TYPE_MEMORY_POOL = 0x3, + NVN_DEBUG_OBJECT_TYPE_SYNC = 0x7, + NVN_DEBUG_OBJECT_TYPE_PROGRAM = 0x9, + NVN_DEBUG_OBJECT_TYPE_TEXTURE_POOL = 0xA, + NVN_DEBUG_OBJECT_TYPE_SAMPLER_POOL = 0xB, + + NVN_DEBUG_OBJECT_TYPE_LARGE = 0x7FFFFFFF +} NVNdebugObjectType; + +typedef enum { + NVN_QUEUE_GET_ERROR_RESULT_GPU_NO_ERROR = 0x0, + NVN_QUEUE_GET_ERROR_RESULT_GPU_ERROR_UNKNOWN = 0x1, + NVN_QUEUE_GET_ERROR_RESULT_GPU_ERROR_MMU_FAULT = 0x2, + NVN_QUEUE_GET_ERROR_RESULT_GPU_ERROR_PBDMA_EXCEPTION = 0x3, + NVN_QUEUE_GET_ERROR_RESULT_GPU_ERROR_ENGINE_EXCEPTION = 0x4, + NVN_QUEUE_GET_ERROR_RESULT_GPU_ERROR_TIMEOUT = 0x5, + + NVN_QUEUE_GET_ERROR_RESULT_LARGE = 0x7FFFFFFF +} NVNqueueGetErrorResult; + +typedef enum { + NVN_QUEUE_BUILDER_FLAGS_NONE = 0x0, + NVN_QUEUE_BUILDER_FLAGS_NO_FRAGMENT_INTERLOCK = 0x1, + NVN_QUEUE_BUILDER_FLAGS_NO_ZCULL = 0x2, + + NVN_QUEUE_BUILDER_FLAGS_LARGE = 0x7FFFFFFF +} NVNqueueBuilderFlags; + +typedef enum { + NVN_QUEUE_ACQUIRE_TEXTURE_RESULT_SUCCESS = 0x0, + NVN_QUEUE_ACQUIRE_TEXTURE_RESULT_NATIVE_ERROR = 0x1, + + NVN_QUEUE_ACQUIRE_TEXTURE_RESULT_LARGE = 0x7FFFFFFF +} NVNqueueAcquireTextureResult; + +typedef enum { + NVN_WINDOW_ACQUIRE_TEXTURE_RESULT_SUCCESS = 0x0, + NVN_WINDOW_ACQUIRE_TEXTURE_RESULT_NATIVE_ERROR = 0x1, + + NVN_WINDOW_ACQUIRE_TEXTURE_RESULT_LARGE = 0x7FFFFFFF +} NVNwindowAcquireTextureResult; + +typedef enum { + NVN_MEMORY_POOL_FLAGS_CPU_NO_ACCESS = 0x1, + NVN_MEMORY_POOL_FLAGS_CPU_UNCACHED = 0x2, + NVN_MEMORY_POOL_FLAGS_CPU_CACHED = 0x4, + NVN_MEMORY_POOL_FLAGS_GPU_NO_ACCESS = 0x8, + NVN_MEMORY_POOL_FLAGS_GPU_UNCACHED = 0x10, + NVN_MEMORY_POOL_FLAGS_GPU_CACHED = 0x20, + NVN_MEMORY_POOL_FLAGS_SHADER_CODE = 0x40, + NVN_MEMORY_POOL_FLAGS_COMPRESSIBLE = 0x80, + NVN_MEMORY_POOL_FLAGS_PHYSICAL = 0x100, + NVN_MEMORY_POOL_FLAGS_VIRTUAL = 0x200, + + NVN_MEMORY_POOL_FLAGS_LARGE = 0x7FFFFFFF +} NVNmemoryPoolFlags; + +typedef enum { + NVN_TEXTURE_SWIZZLE_ZERO = 0x0, + NVN_TEXTURE_SWIZZLE_ONE = 0x1, + NVN_TEXTURE_SWIZZLE_R = 0x2, + NVN_TEXTURE_SWIZZLE_G = 0x3, + NVN_TEXTURE_SWIZZLE_B = 0x4, + NVN_TEXTURE_SWIZZLE_A = 0x5, + + NVN_TEXTURE_SWIZZLE_LARGE = 0x7FFFFFFF +} NVNtextureSwizzle; + +typedef enum { + NVN_SHADER_STAGE_VERTEX = 0x0, + NVN_SHADER_STAGE_FRAGMENT = 0x1, + NVN_SHADER_STAGE_GEOMETRY = 0x2, + NVN_SHADER_STAGE_TESS_CONTROL = 0x3, + NVN_SHADER_STAGE_TESS_EVALUATION = 0x4, + NVN_SHADER_STAGE_COMPUTE = 0x5, + + NVN_SHADER_STAGE_LARGE = 0x7FFFFFFF +} NVNshaderStage; + +typedef enum { + NVN_SHADER_STAGE_VERTEX_BIT = 0x1, + NVN_SHADER_STAGE_FRAGMENT_BIT = 0x2, + NVN_SHADER_STAGE_GEOMETRY_BIT = 0x4, + NVN_SHADER_STAGE_TESS_CONTROL_BIT = 0x8, + NVN_SHADER_STAGE_TESS_EVALUATION_BIT = 0x10, + NVN_SHADER_STAGE_COMPUTE_BIT = 0x20, + NVN_SHADER_STAGE_ALL_GRAPHICS_BITS = 31, + + NVN_SHADER_STAGE_BITS_LARGE = 0x7FFFFFFF +} NVNshaderStageBits; + +typedef enum { + NVN_EVENT_WAIT_MODE_EQUAL = 0x0, + NVN_EVENT_WAIT_MODE_GEQUAL_WRAP = 0x1, + + NVN_EVENT_WAIT_MODE_LARGE = 0x7FFFFFFF +} NVNeventWaitMode; + +typedef enum { + NVN_EVENT_SIGNAL_MODE_WRITE = 0x0, + NVN_EVENT_SIGNAL_MODE_ADD = 0x1, + + NVN_EVENT_SIGNAL_MODE_LARGE = 0x7FFFFFFF +} NVNeventSignalMode; + +typedef enum { + NVN_EVENT_SIGNAL_LOCATION_TOP = 0x0, + NVN_EVENT_SIGNAL_LOCATION_VERTEX_PIPE = 0x1, + NVN_EVENT_SIGNAL_LOCATION_BOTTOM = 0x2, + + NVN_EVENT_SIGNAL_LOCATION_LARGE = 0x7FFFFFFF +} NVNeventSignalLocation; + +typedef enum { + NVN_SYNC_CONDITION_ALL_GPU_COMMANDS_COMPLETE = 0x0, + NVN_SYNC_CONDITION_GRAPHICS_WORLD_SPACE_COMPLETE = 0x1, + + NVN_SYNC_CONDITION_LARGE = 0x7FFFFFFF +} NVNsyncCondition; + +typedef enum { + NVN_COUNTER_TYPE_TIMESTAMP = 0x0, + NVN_COUNTER_TYPE_SAMPLES_PASSED = 0x1, + NVN_COUNTER_TYPE_INPUT_VERTICES = 0x2, + NVN_COUNTER_TYPE_INPUT_PRIMITIVES = 0x3, + NVN_COUNTER_TYPE_VERTEX_SHADER_INVOCATIONS = 0x4, + NVN_COUNTER_TYPE_TESS_CONTROL_SHADER_INVOCATIONS = 0x5, + NVN_COUNTER_TYPE_TESS_EVALUATION_SHADER_INVOCATIONS = 0x6, + NVN_COUNTER_TYPE_GEOMETRY_SHADER_INVOCATIONS = 0x7, + NVN_COUNTER_TYPE_FRAGMENT_SHADER_INVOCATIONS = 0x8, + NVN_COUNTER_TYPE_TESS_EVALUATION_SHADER_PRIMITIVES = 0x9, + NVN_COUNTER_TYPE_GEOMETRY_SHADER_PRIMITIVES = 0xA, + NVN_COUNTER_TYPE_CLIPPER_INPUT_PRIMITIVES = 0xB, + NVN_COUNTER_TYPE_CLIPPER_OUTPUT_PRIMITIVES = 0xC, + NVN_COUNTER_TYPE_PRIMITIVES_GENERATED = 0xD, + NVN_COUNTER_TYPE_TRANSFORM_FEEDBACK_PRIMITIVES_WRITTEN = 0xE, + NVN_COUNTER_TYPE_ZCULL_STATS = 0xF, + NVN_COUNTER_TYPE_TIMESTAMP_TOP = 0x10, + + NVN_COUNTER_TYPE_LARGE = 0x7FFFFFFF +} NVNcounterType; + +typedef enum { + NVN_FACE_NONE = 0x0, + NVN_FACE_FRONT = 0x1, + NVN_FACE_BACK = 0x2, + NVN_FACE_FRONT_AND_BACK = 0x3, + + NVN_FACE_LARGE = 0x7FFFFFFF +} NVNface; + +typedef enum { + NVN_COMMAND_BUFFER_MEMORY_EVENT_OUT_OF_COMMAND_MEMORY = 0x0, + NVN_COMMAND_BUFFER_MEMORY_EVENT_OUT_OF_CONTROL_MEMORY = 0x1, + + NVN_COMMAND_BUFFER_MEMORY_EVENT_LARGE = 0x7FFFFFFF +} NVNcommandBufferMemoryEvent; + +typedef enum { + NVN_DRAW_PRIMITIVE_POINTS = 0x0, + NVN_DRAW_PRIMITIVE_LINES = 0x1, + NVN_DRAW_PRIMITIVE_LINE_LOOP = 0x2, + NVN_DRAW_PRIMITIVE_LINE_STRIP = 0x3, + NVN_DRAW_PRIMITIVE_TRIANGLES = 0x4, + NVN_DRAW_PRIMITIVE_TRIANGLE_STRIP = 0x5, + NVN_DRAW_PRIMITIVE_TRIANGLE_FAN = 0x6, + NVN_DRAW_PRIMITIVE_QUADS = 0x7, + NVN_DRAW_PRIMITIVE_QUAD_STRIP = 0x8, + NVN_DRAW_PRIMITIVE_POLYGON = 0x9, + NVN_DRAW_PRIMITIVE_LINES_ADJACENCY = 0xA, + NVN_DRAW_PRIMITIVE_LINE_STRIP_ADJACENCY = 0xB, + NVN_DRAW_PRIMITIVE_TRIANGLES_ADJACENCY = 0xC, + NVN_DRAW_PRIMITIVE_TRIANGLE_STRIP_ADJACENCY = 0xD, + NVN_DRAW_PRIMITIVE_PATCHES = 0xE, + + NVN_DRAW_PRIMITIVE_LARGE = 0x7FFFFFFF +} NVNdrawPrimitive; + +typedef enum { + NVN_INDEX_TYPE_UNSIGNED_BYTE = 0x0, + NVN_INDEX_TYPE_UNSIGNED_SHORT = 0x1, + NVN_INDEX_TYPE_UNSIGNED_INT = 0x2, + + NVN_INDEX_TYPE_LARGE = 0x7FFFFFFF +} NVNindexType; + +typedef enum { + NVN_TEXTURE_FLAGS_DISPLAY = 0x1, + NVN_TEXTURE_FLAGS_VIDEO_DECODE = 0x2, + NVN_TEXTURE_FLAGS_IMAGE = 0x4, + NVN_TEXTURE_FLAGS_COMPRESSIBLE = 0x8, + NVN_TEXTURE_FLAGS_LINEAR = 0x10, + NVN_TEXTURE_FLAGS_SPARSE = 0x20, + NVN_TEXTURE_FLAGS_LINEAR_RENDER_TARGET = 0x40, + NVN_TEXTURE_FLAGS_ADAPTIVE_ZCULL = 0x80, + NVN_TEXTURE_FLAGS_DEPTH16_PREFER_FAST_CLEAR = 0x100, + NVN_TEXTURE_FLAGS_MINIMAL_LAYOUT = 0x200, + NVN_TEXTURE_FLAGS_ZCULL_SUPPORT_STENCIL = 0x400, + + NVN_TEXTURE_FLAGS_LARGE = 0x7FFFFFFF +} NVNtextureFlags; + +typedef enum { + NVN_BARRIER_ORDER_PRIMITIVES_BIT = 0x1, + NVN_BARRIER_ORDER_FRAGMENTS_BIT = 0x2, + NVN_BARRIER_ORDER_FRAGMENTS_TILED_BIT = 0x4, + NVN_BARRIER_ORDER_INDIRECT_DATA_BIT = 0x8, + NVN_BARRIER_INVALIDATE_TEXTURE_BIT = 0x10, + NVN_BARRIER_INVALIDATE_SHADER_BIT = 0x20, + NVN_BARRIER_INVALIDATE_TEXTURE_DESCRIPTOR_BIT = 0x40, + NVN_BARRIER_INVALIDATE_ZCULL_BIT = 0x80, + NVN_BARRIER_BITS_LARGE = 0x7FFFFFFF +} NVNbarrierBits; + +typedef enum { + NVN_TEXTURE_TARGET_1D = 0x0, + NVN_TEXTURE_TARGET_2D = 0x1, + NVN_TEXTURE_TARGET_3D = 0x2, + NVN_TEXTURE_TARGET_1D_ARRAY = 0x3, + NVN_TEXTURE_TARGET_2D_ARRAY = 0x4, + NVN_TEXTURE_TARGET_2D_MULTISAMPLE = 0x5, + NVN_TEXTURE_TARGET_2D_MULTISAMPLE_ARRAY = 0x6, + NVN_TEXTURE_TARGET_RECTANGLE = 0x7, + NVN_TEXTURE_TARGET_CUBEMAP = 0x8, + NVN_TEXTURE_TARGET_CUBEMAP_ARRAY = 0x9, + NVN_TEXTURE_TARGET_BUFFER = 0xA, + + NVN_TEXTURE_TARGET_LARGE = 0x7FFFFFFF +} NVNtextureTarget; + +typedef enum { + NVN_TEXTURE_DEPTH_STENCIL_MODE_DEPTH = 0x0, + NVN_TEXTURE_DEPTH_STENCIL_MODE_STENCIL = 0x1, + + NVN_TEXTURE_DEPTH_STENCIL_MODE_LARGE = 0x7FFFFFFF +} NVNtextureDepthStencilMode; + +typedef enum { + NVN_LOGIC_OP_CLEAR = 0x0, + NVN_LOGIC_OP_AND = 0x1, + NVN_LOGIC_OP_AND_REVERSE = 0x2, + NVN_LOGIC_OP_COPY = 0x3, + NVN_LOGIC_OP_AND_INVERTED = 0x4, + NVN_LOGIC_OP_NOOP = 0x5, + NVN_LOGIC_OP_XOR = 0x6, + NVN_LOGIC_OP_OR = 0x7, + NVN_LOGIC_OP_NOR = 0x8, + NVN_LOGIC_OP_EQUIV = 0x9, + NVN_LOGIC_OP_INVERT = 0xA, + NVN_LOGIC_OP_OR_REVERSE = 0xB, + NVN_LOGIC_OP_COPY_INVERTED = 0xC, + NVN_LOGIC_OP_OR_INVERTED = 0xD, + NVN_LOGIC_OP_NAND = 0xE, + NVN_LOGIC_OP_SET = 0xF, + + NVN_LOGIC_OP_LARGE = 0x7FFFFFFF +} NVNlogicOp; + +typedef enum { + NVN_ALPHA_FUNC_NEVER = 0x1, + NVN_ALPHA_FUNC_LESS = 0x2, + NVN_ALPHA_FUNC_EQUAL = 0x3, + NVN_ALPHA_FUNC_LEQUAL = 0x4, + NVN_ALPHA_FUNC_GREATER = 0x5, + NVN_ALPHA_FUNC_NOTEQUAL = 0x6, + NVN_ALPHA_FUNC_GEQUAL = 0x7, + NVN_ALPHA_FUNC_ALWAYS = 0x8, + + NVN_ALPHA_FUNC_LARGE = 0x7FFFFFFF +} NVNalphaFunc; + +typedef enum { + NVN_BLEND_ADVANCED_MODE_NONE = 0x0, + NVN_BLEND_ADVANCED_MODE_ZERO = 0x5, + NVN_BLEND_ADVANCED_MODE_SRC = 0x6, + NVN_BLEND_ADVANCED_MODE_DST = 0x7, + NVN_BLEND_ADVANCED_MODE_SRC_OVER = 0x8, + NVN_BLEND_ADVANCED_MODE_DST_OVER = 0x9, + NVN_BLEND_ADVANCED_MODE_SRC_IN = 0xA, + NVN_BLEND_ADVANCED_MODE_DST_IN = 0xB, + NVN_BLEND_ADVANCED_MODE_SRC_OUT = 0xC, + NVN_BLEND_ADVANCED_MODE_DST_OUT = 0xD, + NVN_BLEND_ADVANCED_MODE_SRC_ATOP = 0xE, + NVN_BLEND_ADVANCED_MODE_DST_ATOP = 0xF, + NVN_BLEND_ADVANCED_MODE_XOR = 0x10, + NVN_BLEND_ADVANCED_MODE_PLUS = 0x11, + NVN_BLEND_ADVANCED_MODE_PLUS_CLAMPED = 0x12, + NVN_BLEND_ADVANCED_MODE_PLUS_CLAMPED_ALPHA = 0x13, + NVN_BLEND_ADVANCED_MODE_PLUS_DARKER = 0x14, + NVN_BLEND_ADVANCED_MODE_MULTIPLY = 0x15, + NVN_BLEND_ADVANCED_MODE_SCREEN = 0x16, + NVN_BLEND_ADVANCED_MODE_OVERLAY = 0x17, + NVN_BLEND_ADVANCED_MODE_DARKEN = 0x18, + NVN_BLEND_ADVANCED_MODE_LIGHTEN = 0x19, + NVN_BLEND_ADVANCED_MODE_COLORDODGE = 0x1A, + NVN_BLEND_ADVANCED_MODE_COLORBURN = 0x1B, + NVN_BLEND_ADVANCED_MODE_HARDLIGHT = 0x1C, + NVN_BLEND_ADVANCED_MODE_SOFTLIGHT = 0x1D, + NVN_BLEND_ADVANCED_MODE_DIFFERENCE = 0x1E, + NVN_BLEND_ADVANCED_MODE_MINUS = 0x1F, + NVN_BLEND_ADVANCED_MODE_MINUS_CLAMPED = 0x20, + NVN_BLEND_ADVANCED_MODE_EXCLUSION = 0x21, + NVN_BLEND_ADVANCED_MODE_CONTRAST = 0x22, + NVN_BLEND_ADVANCED_MODE_INVERT = 0x23, + NVN_BLEND_ADVANCED_MODE_INVERT_RGB = 0x24, + NVN_BLEND_ADVANCED_MODE_INVERT_OVG = 0x25, + NVN_BLEND_ADVANCED_MODE_LINEARDODGE = 0x26, + NVN_BLEND_ADVANCED_MODE_LINEARBURN = 0x27, + NVN_BLEND_ADVANCED_MODE_VIVIDLIGHT = 0x28, + NVN_BLEND_ADVANCED_MODE_LINEARLIGHT = 0x29, + NVN_BLEND_ADVANCED_MODE_PINLIGHT = 0x2A, + NVN_BLEND_ADVANCED_MODE_HARDMIX = 0x2B, + NVN_BLEND_ADVANCED_MODE_RED = 0x2C, + NVN_BLEND_ADVANCED_MODE_GREEN = 0x2D, + NVN_BLEND_ADVANCED_MODE_BLUE = 0x2E, + NVN_BLEND_ADVANCED_MODE_HSL_HUE = 0x2F, + NVN_BLEND_ADVANCED_MODE_HSL_SATURATION = 0x30, + NVN_BLEND_ADVANCED_MODE_HSL_COLOR = 0x31, + NVN_BLEND_ADVANCED_MODE_HSL_LUMINOSITY = 0x32, + + NVN_BLEND_ADVANCED_MODE_LARGE = 0x7FFFFFFF +} NVNblendAdvancedMode; + +typedef enum { + NVN_MIN_FILTER_NEAREST = 0x0, + NVN_MIN_FILTER_LINEAR = 0x1, + NVN_MIN_FILTER_NEAREST_MIPMAP_NEAREST = 0x2, + NVN_MIN_FILTER_LINEAR_MIPMAP_NEAREST = 0x3, + NVN_MIN_FILTER_NEAREST_MIPMAP_LINEAR = 0x4, + NVN_MIN_FILTER_LINEAR_MIPMAP_LINEAR = 0x5, + + NVN_MIN_FILTER_LARGE = 0x7FFFFFFF +} NVNminFilter; + +typedef enum { + NVN_MAG_FILTER_NEAREST = 0x0, + NVN_MAG_FILTER_LINEAR = 0x1, + + NVN_MAG_FILTER_LARGE = 0x7FFFFFFF +} NVNmagFilter; + +typedef enum { + NVN_WRAP_MODE_CLAMP = 0x0, + NVN_WRAP_MODE_REPEAT = 0x1, + NVN_WRAP_MODE_MIRROR_CLAMP = 0x2, + NVN_WRAP_MODE_MIRROR_CLAMP_TO_EDGE = 0x3, + NVN_WRAP_MODE_MIRROR_CLAMP_TO_BORDER = 0x4, + NVN_WRAP_MODE_CLAMP_TO_BORDER = 0x5, + NVN_WRAP_MODE_MIRRORED_REPEAT = 0x6, + NVN_WRAP_MODE_CLAMP_TO_EDGE = 0x7, + + NVN_WRAP_MODE_LARGE = 0x7FFFFFFF +} NVNwrapMode; + +typedef enum { + NVN_COMPARE_MODE_NONE = 0x0, + NVN_COMPARE_MODE_COMPARE_R_TO_TEXTURE = 0x1, + + NVN_COMPARE_MODE_LARGE = 0x7FFFFFFF +} NVNcompareMode; + +typedef enum { + NVN_COMPARE_FUNC_NEVER = 0x1, + NVN_COMPARE_FUNC_LESS = 0x2, + NVN_COMPARE_FUNC_EQUAL = 0x3, + NVN_COMPARE_FUNC_LEQUAL = 0x4, + NVN_COMPARE_FUNC_GREATER = 0x5, + NVN_COMPARE_FUNC_NOTEQUAL = 0x6, + NVN_COMPARE_FUNC_GEQUAL = 0x7, + NVN_COMPARE_FUNC_ALWAYS = 0x8, + + NVN_COMPARE_FUNC_LARGE = 0x7FFFFFFF +} NVNcompareFunc; + +typedef enum { + NVN_SAMPLER_REDUCTION_AVERAGE = 0x0, + NVN_SAMPLER_REDUCTION_MIN = 0x1, + NVN_SAMPLER_REDUCTION_MAX = 0x2, + + NVN_SAMPLER_REDUCTION_LARGE = 0x7FFFFFFF +} NVNsamplerReduction; + +typedef enum { + NVN_BLEND_FUNC_ZERO = 0x1, + NVN_BLEND_FUNC_ONE = 0x2, + NVN_BLEND_FUNC_SRC_COLOR = 0x3, + NVN_BLEND_FUNC_ONE_MINUS_SRC_COLOR = 0x4, + NVN_BLEND_FUNC_SRC_ALPHA = 0x5, + NVN_BLEND_FUNC_ONE_MINUS_SRC_ALPHA = 0x6, + NVN_BLEND_FUNC_DST_ALPHA = 0x7, + NVN_BLEND_FUNC_ONE_MINUS_DST_ALPHA = 0x8, + NVN_BLEND_FUNC_DST_COLOR = 0x9, + NVN_BLEND_FUNC_ONE_MINUS_DST_COLOR = 0xA, + NVN_BLEND_FUNC_SRC_ALPHA_SATURATE = 0xB, + NVN_BLEND_FUNC_SRC1_COLOR = 0x10, + NVN_BLEND_FUNC_ONE_MINUS_SRC1_COLOR = 0x11, + NVN_BLEND_FUNC_SRC1_ALPHA = 0x12, + NVN_BLEND_FUNC_ONE_MINUS_SRC1_ALPHA = 0x13, + NVN_BLEND_FUNC_CONSTANT_COLOR = 0x61, + NVN_BLEND_FUNC_ONE_MINUS_CONSTANT_COLOR = 0x62, + NVN_BLEND_FUNC_CONSTANT_ALPHA = 0x63, + NVN_BLEND_FUNC_ONE_MINUS_CONSTANT_ALPHA = 0x64, + + NVN_BLEND_FUNC_LARGE = 0x7FFFFFFF +} NVNblendFunc; + +typedef enum { + NVN_BLEND_EQUATION_ADD = 0x1, + NVN_BLEND_EQUATION_SUB = 0x2, + NVN_BLEND_EQUATION_REVERSE_SUB = 0x3, + NVN_BLEND_EQUATION_MIN = 0x4, + NVN_BLEND_EQUATION_MAX = 0x5, + + NVN_BLEND_EQUATION_LARGE = 0x7FFFFFFF +} NVNblendEquation; + +typedef enum { + NVN_BLEND_ADVANCED_OVERLAP_UNCORRELATED = 0x0, + NVN_BLEND_ADVANCED_OVERLAP_DISJOINT = 0x1, + NVN_BLEND_ADVANCED_OVERLAP_CONJOINT = 0x2, + + NVN_BLEND_ADVANCED_OVERLAP_LARGE = 0x7FFFFFFF +} NVNblendAdvancedOverlap; + +typedef enum { + NVN_COVERAGE_MODULATION_MODE_NONE = 0x0, + NVN_COVERAGE_MODULATION_MODE_RGB = 0x1, + NVN_COVERAGE_MODULATION_MODE_ALPHA = 0x2, + NVN_COVERAGE_MODULATION_MODE_RGBA = 0x3, + + NVN_COVERAGE_MODULATION_MODE_LARGE = 0x7FFFFFFF +} NVNcoverageModulationMode; + +typedef enum { + NVN_DEPTH_FUNC_NEVER = 0x1, + NVN_DEPTH_FUNC_LESS = 0x2, + NVN_DEPTH_FUNC_EQUAL = 0x3, + NVN_DEPTH_FUNC_LEQUAL = 0x4, + NVN_DEPTH_FUNC_GREATER = 0x5, + NVN_DEPTH_FUNC_NOTEQUAL = 0x6, + NVN_DEPTH_FUNC_GEQUAL = 0x7, + NVN_DEPTH_FUNC_ALWAYS = 0x8, + + NVN_DEPTH_FUNC_LARGE = 0x7FFFFFFF +} NVNdepthFunc; + +typedef enum { + NVN_STENCIL_FUNC_NEVER = 0x1, + NVN_STENCIL_FUNC_LESS = 0x2, + NVN_STENCIL_FUNC_EQUAL = 0x3, + NVN_STENCIL_FUNC_LEQUAL = 0x4, + NVN_STENCIL_FUNC_GREATER = 0x5, + NVN_STENCIL_FUNC_NOTEQUAL = 0x6, + NVN_STENCIL_FUNC_GEQUAL = 0x7, + NVN_STENCIL_FUNC_ALWAYS = 0x8, + + NVN_STENCIL_FUNC_LARGE = 0x7FFFFFFF +} NVNstencilFunc; + +typedef enum { + NVN_STENCIL_OP_KEEP = 0x1, + NVN_STENCIL_OP_ZERO = 0x2, + NVN_STENCIL_OP_REPLACE = 0x3, + NVN_STENCIL_OP_INCR = 0x4, + NVN_STENCIL_OP_DECR = 0x5, + NVN_STENCIL_OP_INVERT = 0x6, + NVN_STENCIL_OP_INCR_WRAP = 0x7, + NVN_STENCIL_OP_DECR_WRAP = 0x8, + + NVN_STENCIL_OP_LARGE = 0x7FFFFFFF +} NVNstencilOp; + +typedef enum { + NVN_FRONT_FACE_CW = 0x0, + NVN_FRONT_FACE_CCW = 0x1, + + NVN_FRONT_FACE_LARGE = 0x7FFFFFFF +} NVNfrontFace; + +typedef enum { + NVN_SYNC_WAIT_RESULT_ALREADY_SIGNALED = 0x0, + NVN_SYNC_WAIT_RESULT_CONDITION_SATISFIED = 0x1, + NVN_SYNC_WAIT_RESULT_TIMEOUT_EXPIRED = 0x2, + NVN_SYNC_WAIT_RESULT_FAILED = 0x3, + + NVN_SYNC_WAIT_RESULT_LARGE = 0x7FFFFFFF +} NVNsyncWaitResult; + +typedef enum { + NVN_POLYGON_OFFSET_ENABLE_NONE = 0x0, + NVN_POLYGON_OFFSET_ENABLE_POINT = 0x1, + NVN_POLYGON_OFFSET_ENABLE_LINE = 0x2, + NVN_POLYGON_OFFSET_ENABLE_FILL = 0x4, + + NVN_POLYGON_OFFSET_ENABLE_LARGE = 0x7FFFFFFF +} NVNpolygonOffsetEnable; + +typedef enum { + NVN_POLYGON_MODE_POINT = 0x0, + NVN_POLYGON_MODE_LINE = 0x1, + NVN_POLYGON_MODE_FILL = 0x2, + + NVN_POLYGON_MODE_LARGE = 0x7FFFFFFF +} NVNpolygonMode; + +typedef enum { + NVN_VIEWPORT_SWIZZLE_POSITIVE_X = 0x0, + NVN_VIEWPORT_SWIZZLE_NEGATIVE_X = 0x1, + NVN_VIEWPORT_SWIZZLE_POSITIVE_Y = 0x2, + NVN_VIEWPORT_SWIZZLE_NEGATIVE_Y = 0x3, + NVN_VIEWPORT_SWIZZLE_POSITIVE_Z = 0x4, + NVN_VIEWPORT_SWIZZLE_NEGATIVE_Z = 0x5, + NVN_VIEWPORT_SWIZZLE_POSITIVE_W = 0x6, + NVN_VIEWPORT_SWIZZLE_NEGATIVE_W = 0x7, + + NVN_VIEWPORT_SWIZZLE_LARGE = 0x7FFFFFFF +} NVNviewportSwizzle; + +typedef enum { + NVN_TILED_CACHE_ACTION_ENABLE = 0x1, + NVN_TILED_CACHE_ACTION_DISABLE = 0x2, + NVN_TILED_CACHE_ACTION_FLUSH = 0x3, + NVN_TILED_CACHE_ACTION_FLUSH_NO_TILING = 0x4, + NVN_TILED_CACHE_ACTION_ENABLE_RENDER_TARGET_BINNING = 0x5, + NVN_TILED_CACHE_ACTION_DISABLE_RENDER_TARGET_BINNING = 0x6, + + NVN_TILED_CACHE_ACTION_LARGE = 0x7FFFFFFF +} NVNtiledCacheAction; + +typedef enum { + NVN_CONDITIONAL_RENDER_MODE_RENDER_IF_EQUAL = 0x0, + NVN_CONDITIONAL_RENDER_MODE_RENDER_IF_NOT_EQUAL = 0x1, + + NVN_CONDITIONAL_RENDER_MODE_LARGE = 0x7FFFFFFF +} NVNconditionalRenderMode; + +typedef enum { + NVN_COPY_FLAGS_NONE = 0x0, + NVN_COPY_FLAGS_LINEAR_FILTER_BIT = 0x1, + NVN_COPY_FLAGS_ENGINE_2D_BIT = 0x2, + NVN_COPY_FLAGS_MIRROR_X_BIT = 0x4, + NVN_COPY_FLAGS_MIRROR_Y_BIT = 0x8, + NVN_COPY_FLAGS_MIRROR_Z_BIT = 0x10, + NVN_COPY_FLAGS_LARGE = 0x7FFFFFFF +} NVNcopyFlags; + +#ifdef __cplusplus +} +#endif + +typedef void (*PFNNVNGENERICFUNCPTRPROC)(void); +typedef PFNNVNGENERICFUNCPTRPROC (*PFNNVNBOOTSTRAPLOADERPROC)(const char*); +typedef void (*PFNNVNCOMMANDBUFFERMEMORYCALLBACKPROC)(NVNcommandBuffer*, + NVNcommandBufferMemoryEvent, size_t, void*); +typedef void (*PFNNVNDEBUGCALLBACKPROC)(NVNdebugCallbackSource source, NVNdebugCallbackType type, + int id, NVNdebugCallbackSeverity severity, + const unsigned char* message, void* user); +typedef void (*PFNNVNWALKDEBUGDATABASECALLBACKPROC)(void*, void*); + +typedef void (*PFNNVNDEVICEBUILDERSETDEFAULTSPROC)(NVNdeviceBuilder*); +typedef void (*PFNNVNDEVICEBUILDERSETFLAGSPROC)(NVNdeviceBuilder*, int); +typedef NVNboolean (*PFNNVNDEVICEINITIALIZEPROC)(NVNdevice*, const NVNdeviceBuilder*); +typedef void (*PFNNVNDEVICEFINALIZEPROC)(NVNdevice*); +typedef void (*PFNNVNDEVICESETDEBUGLABELPROC)(NVNdevice*, const char*); +typedef PFNNVNGENERICFUNCPTRPROC (*PFNNVNDEVICEGETPROCADDRESSPROC)(const NVNdevice*, const char*); +typedef void (*PFNNVNDEVICEGETINTEGERPROC)(const NVNdevice*, NVNdeviceInfo, int*); +typedef uint64_t (*PFNNVNDEVICEGETCURRENTTIMESTAMPINNANOSECONDSPROC)(const NVNdevice*); +typedef void (*PFNNVNDEVICESETINTERMEDIATESHADERCACHEPROC)(NVNdevice*, int); +typedef NVNtextureHandle (*PFNNVNDEVICEGETTEXTUREHANDLEPROC)(const NVNdevice*, int, int); +typedef NVNtextureHandle (*PFNNVNDEVICEGETTEXELFETCHHANDLEPROC)(const NVNdevice*, int); +typedef NVNimageHandle (*PFNNVNDEVICEGETIMAGEHANDLEPROC)(const NVNdevice*, int); +typedef void (*PFNNVNDEVICEINSTALLDEBUGCALLBACKPROC)(NVNdevice*, const PFNNVNDEBUGCALLBACKPROC, + void*, NVNboolean); +typedef NVNdebugDomainId (*PFNNVNDEVICEGENERATEDEBUGDOMAINIDPROC)(const NVNdevice*, const char*); +typedef void (*PFNNVNDEVICESETWINDOWORIGINMODEPROC)(NVNdevice*, NVNwindowOriginMode); +typedef void (*PFNNVNDEVICESETDEPTHMODEPROC)(NVNdevice*, NVNdepthMode); +typedef NVNboolean (*PFNNVNDEVICEREGISTERFASTCLEARCOLORPROC)(NVNdevice*, const float*, NVNformat); +typedef NVNboolean (*PFNNVNDEVICEREGISTERFASTCLEARCOLORIPROC)(NVNdevice*, const int*, NVNformat); +typedef NVNboolean (*PFNNVNDEVICEREGISTERFASTCLEARCOLORUIPROC)(NVNdevice*, const uint32_t*, + NVNformat); +typedef NVNboolean (*PFNNVNDEVICEREGISTERFASTCLEARDEPTHPROC)(NVNdevice*, float); +typedef NVNwindowOriginMode (*PFNNVNDEVICEGETWINDOWORIGINMODEPROC)(const NVNdevice*); +typedef NVNdepthMode (*PFNNVNDEVICEGETDEPTHMODEPROC)(const NVNdevice*); +typedef uint64_t (*PFNNVNDEVICEGETTIMESTAMPINNANOSECONDSPROC)(const NVNdevice*, + const NVNcounterData*); +typedef void (*PFNNVNDEVICEAPPLYDEFERREDFINALIZESPROC)(NVNdevice*, int); +typedef void (*PFNNVNDEVICEFINALIZECOMMANDHANDLEPROC)(NVNdevice*, NVNcommandHandle); +typedef void (*PFNNVNDEVICEWALKDEBUGDATABASEPROC)(const NVNdevice*, NVNdebugObjectType, + PFNNVNWALKDEBUGDATABASECALLBACKPROC, void*); +typedef NVNseparateTextureHandle (*PFNNVNDEVICEGETSEPARATETEXTUREHANDLEPROC)(const NVNdevice*, int); +typedef NVNseparateSamplerHandle (*PFNNVNDEVICEGETSEPARATESAMPLERHANDLEPROC)(const NVNdevice*, int); +typedef NVNboolean (*PFNNVNDEVICEISEXTERNALDEBUGGERATTACHEDPROC)(const NVNdevice*); +typedef NVNqueueGetErrorResult (*PFNNVNQUEUEGETERRORPROC)(NVNqueue*, NVNqueueErrorInfo*); +typedef size_t (*PFNNVNQUEUEGETTOTALCOMMANDMEMORYUSEDPROC)(NVNqueue*); +typedef size_t (*PFNNVNQUEUEGETTOTALCONTROLMEMORYUSEDPROC)(NVNqueue*); +typedef size_t (*PFNNVNQUEUEGETTOTALCOMPUTEMEMORYUSEDPROC)(NVNqueue*); +typedef void (*PFNNVNQUEUERESETMEMORYUSAGECOUNTSPROC)(NVNqueue*); +typedef void (*PFNNVNQUEUEBUILDERSETDEVICEPROC)(NVNqueueBuilder*, NVNdevice*); +typedef void (*PFNNVNQUEUEBUILDERSETDEFAULTSPROC)(NVNqueueBuilder*); +typedef void (*PFNNVNQUEUEBUILDERSETFLAGSPROC)(NVNqueueBuilder*, int); +typedef void (*PFNNVNQUEUEBUILDERSETCOMMANDMEMORYSIZEPROC)(NVNqueueBuilder*, size_t); +typedef void (*PFNNVNQUEUEBUILDERSETCOMPUTEMEMORYSIZEPROC)(NVNqueueBuilder*, size_t); +typedef void (*PFNNVNQUEUEBUILDERSETCONTROLMEMORYSIZEPROC)(NVNqueueBuilder*, size_t); +typedef size_t (*PFNNVNQUEUEBUILDERGETQUEUEMEMORYSIZEPROC)(const NVNqueueBuilder*); +typedef void (*PFNNVNQUEUEBUILDERSETQUEUEMEMORYPROC)(NVNqueueBuilder*, void*, size_t); +typedef void (*PFNNVNQUEUEBUILDERSETCOMMANDFLUSHTHRESHOLDPROC)(NVNqueueBuilder*, size_t); +typedef NVNboolean (*PFNNVNQUEUEINITIALIZEPROC)(NVNqueue*, const NVNqueueBuilder*); +typedef void (*PFNNVNQUEUEFINALIZEPROC)(NVNqueue*); +typedef void (*PFNNVNQUEUESETDEBUGLABELPROC)(NVNqueue*, const char*); +typedef void (*PFNNVNQUEUESUBMITCOMMANDSPROC)(NVNqueue*, int, const NVNcommandHandle*); +typedef void (*PFNNVNQUEUEFLUSHPROC)(NVNqueue*); +typedef void (*PFNNVNQUEUEFINISHPROC)(NVNqueue*); +typedef void (*PFNNVNQUEUEPRESENTTEXTUREPROC)(NVNqueue*, NVNwindow*, int); +typedef NVNqueueAcquireTextureResult (*PFNNVNQUEUEACQUIRETEXTUREPROC)(NVNqueue*, NVNwindow*, int*); +typedef void (*PFNNVNWINDOWBUILDERSETDEVICEPROC)(NVNwindowBuilder*, NVNdevice*); +typedef void (*PFNNVNWINDOWBUILDERSETDEFAULTSPROC)(NVNwindowBuilder*); +typedef void (*PFNNVNWINDOWBUILDERSETNATIVEWINDOWPROC)(NVNwindowBuilder*, NVNnativeWindow); +typedef void (*PFNNVNWINDOWBUILDERSETTEXTURESPROC)(NVNwindowBuilder*, int, NVNtexture* const*); +typedef void (*PFNNVNWINDOWBUILDERSETPRESENTINTERVALPROC)(NVNwindowBuilder*, int); +typedef NVNnativeWindow (*PFNNVNWINDOWBUILDERGETNATIVEWINDOWPROC)(const NVNwindowBuilder*); +typedef int (*PFNNVNWINDOWBUILDERGETPRESENTINTERVALPROC)(const NVNwindowBuilder*); +typedef NVNboolean (*PFNNVNWINDOWINITIALIZEPROC)(NVNwindow*, const NVNwindowBuilder*); +typedef void (*PFNNVNWINDOWFINALIZEPROC)(NVNwindow*); +typedef void (*PFNNVNWINDOWSETDEBUGLABELPROC)(NVNwindow*, const char*); +typedef NVNwindowAcquireTextureResult (*PFNNVNWINDOWACQUIRETEXTUREPROC)(NVNwindow*, NVNsync*, int*); +typedef NVNnativeWindow (*PFNNVNWINDOWGETNATIVEWINDOWPROC)(const NVNwindow*); +typedef int (*PFNNVNWINDOWGETPRESENTINTERVALPROC)(const NVNwindow*); +typedef void (*PFNNVNWINDOWSETPRESENTINTERVALPROC)(NVNwindow*, int); +typedef void (*PFNNVNWINDOWSETCROPPROC)(NVNwindow*, int, int, int, int); +typedef void (*PFNNVNWINDOWGETCROPPROC)(const NVNwindow*, NVNrectangle*); +typedef NVNboolean (*PFNNVNPROGRAMINITIALIZEPROC)(NVNprogram*, NVNdevice*); +typedef void (*PFNNVNPROGRAMFINALIZEPROC)(NVNprogram*); +typedef void (*PFNNVNPROGRAMSETDEBUGLABELPROC)(NVNprogram*, const char*); +typedef NVNboolean (*PFNNVNPROGRAMSETSHADERSPROC)(NVNprogram*, int, const NVNshaderData*); +typedef void (*PFNNVNMEMORYPOOLBUILDERSETDEVICEPROC)(NVNmemoryPoolBuilder*, NVNdevice*); +typedef void (*PFNNVNMEMORYPOOLBUILDERSETDEFAULTSPROC)(NVNmemoryPoolBuilder*); +typedef void (*PFNNVNMEMORYPOOLBUILDERSETSTORAGEPROC)(NVNmemoryPoolBuilder*, void*, size_t); +typedef void (*PFNNVNMEMORYPOOLBUILDERSETFLAGSPROC)(NVNmemoryPoolBuilder*, int); +typedef void (*PFNNVNMEMORYPOOLBUILDERGETMEMORYPROC)(const NVNmemoryPoolBuilder*); +typedef size_t (*PFNNVNMEMORYPOOLBUILDERGETSIZEPROC)(const NVNmemoryPoolBuilder*); +typedef NVNmemoryPoolFlags (*PFNNVNMEMORYPOOLBUILDERGETFLAGSPROC)(const NVNmemoryPoolBuilder*); +typedef NVNboolean (*PFNNVNMEMORYPOOLINITIALIZEPROC)(NVNmemoryPool*, const NVNmemoryPoolBuilder*); +typedef void (*PFNNVNMEMORYPOOLSETDEBUGLABELPROC)(NVNmemoryPool*, const char*); +typedef void (*PFNNVNMEMORYPOOLFINALIZEPROC)(NVNmemoryPool*); +typedef void* (*PFNNVNMEMORYPOOLMAPPROC)(const NVNmemoryPool*); +typedef void (*PFNNVNMEMORYPOOLFLUSHMAPPEDRANGEPROC)(const NVNmemoryPool*, ptrdiff_t, size_t); +typedef void (*PFNNVNMEMORYPOOLINVALIDATEMAPPEDRANGEPROC)(const NVNmemoryPool*, ptrdiff_t, size_t); +typedef NVNbufferAddress (*PFNNVNMEMORYPOOLGETBUFFERADDRESSPROC)(const NVNmemoryPool*); +typedef NVNboolean (*PFNNVNMEMORYPOOLMAPVIRTUALPROC)(NVNmemoryPool*, int, const NVNmappingRequest*); +typedef size_t (*PFNNVNMEMORYPOOLGETSIZEPROC)(const NVNmemoryPool*); +typedef NVNmemoryPoolFlags (*PFNNVNMEMORYPOOLGETFLAGSPROC)(const NVNmemoryPool*); +typedef NVNboolean (*PFNNVNTEXTUREPOOLINITIALIZEPROC)(NVNtexturePool*, const NVNmemoryPool*, + ptrdiff_t, int); +typedef void (*PFNNVNTEXTUREPOOLSETDEBUGLABELPROC)(NVNtexturePool*, const char*); +typedef void (*PFNNVNTEXTUREPOOLFINALIZEPROC)(NVNtexturePool*); +typedef void (*PFNNVNTEXTUREPOOLREGISTERTEXTUREPROC)(const NVNtexturePool*, int, const NVNtexture*, + const NVNtextureView*); +typedef void (*PFNNVNTEXTUREPOOLREGISTERIMAGEPROC)(const NVNtexturePool*, int, const NVNtexture*, + const NVNtextureView*); +typedef const NVNmemoryPool* (*PFNNVNTEXTUREPOOLGETMEMORYPOOLPROC)(const NVNtexturePool*); +typedef ptrdiff_t (*PFNNVNTEXTUREPOOLGETMEMORYOFFSETPROC)(const NVNtexturePool*); +typedef int (*PFNNVNTEXTUREPOOLGETSIZEPROC)(const NVNtexturePool*); +typedef NVNboolean (*PFNNVNSAMPLERPOOLINITIALIZEPROC)(NVNsamplerPool*, const NVNmemoryPool*, + ptrdiff_t, int); +typedef void (*PFNNVNSAMPLERPOOLSETDEBUGLABELPROC)(NVNsamplerPool*, const char*); +typedef void (*PFNNVNSAMPLERPOOLFINALIZEPROC)(NVNsamplerPool*); +typedef void (*PFNNVNSAMPLERPOOLREGISTERSAMPLERPROC)(const NVNsamplerPool*, int, const NVNsampler*); +typedef void (*PFNNVNSAMPLERPOOLREGISTERSAMPLERBUILDERPROC)(const NVNsamplerPool*, int, + const NVNsamplerBuilder*); +typedef const NVNmemoryPool* (*PFNNVNSAMPLERPOOLGETMEMORYPOOLPROC)(const NVNsamplerPool*); +typedef ptrdiff_t (*PFNNVNSAMPLERPOOLGETMEMORYOFFSETPROC)(const NVNsamplerPool*); +typedef int (*PFNNVNSAMPLERPOOLGETSIZEPROC)(const NVNsamplerPool*); +typedef void (*PFNNVNBUFFERBUILDERSETDEVICEPROC)(NVNbufferBuilder*, NVNdevice*); +typedef void (*PFNNVNBUFFERBUILDERSETDEFAULTSPROC)(NVNbufferBuilder*); +typedef void (*PFNNVNBUFFERBUILDERSETSTORAGEPROC)(NVNbufferBuilder*, NVNmemoryPool*, ptrdiff_t, + size_t); +typedef NVNmemoryPool (*PFNNVNBUFFERBUILDERGETMEMORYPOOLPROC)(const NVNbufferBuilder*); +typedef ptrdiff_t (*PFNNVNBUFFERBUILDERGETMEMORYOFFSETPROC)(const NVNbufferBuilder*); +typedef size_t (*PFNNVNBUFFERBUILDERGETSIZEPROC)(const NVNbufferBuilder*); +typedef NVNboolean (*PFNNVNBUFFERINITIALIZEPROC)(NVNbuffer*, const NVNbufferBuilder*); +typedef void (*PFNNVNBUFFERSETDEBUGLABELPROC)(NVNbuffer*, const char*); +typedef void (*PFNNVNBUFFERFINALIZEPROC)(NVNbuffer*); +typedef void* (*PFNNVNBUFFERMAPPROC)(const NVNbuffer*); +typedef NVNbufferAddress (*PFNNVNBUFFERGETADDRESSPROC)(const NVNbuffer*); +typedef void (*PFNNVNBUFFERFLUSHMAPPEDRANGEPROC)(const NVNbuffer*, ptrdiff_t, size_t); +typedef void (*PFNNVNBUFFERINVALIDATEMAPPEDRANGEPROC)(const NVNbuffer*, ptrdiff_t, size_t); +typedef NVNmemoryPool* (*PFNNVNBUFFERGETMEMORYPOOLPROC)(const NVNbuffer*); +typedef ptrdiff_t (*PFNNVNBUFFERGETMEMORYOFFSETPROC)(const NVNbuffer*); +typedef size_t (*PFNNVNBUFFERGETSIZEPROC)(const NVNbuffer*); +typedef uint64_t (*PFNNVNBUFFERGETDEBUGIDPROC)(const NVNbuffer*); +typedef void (*PFNNVNTEXTUREBUILDERSETDEVICEPROC)(NVNtextureBuilder*, NVNdevice*); +typedef void (*PFNNVNTEXTUREBUILDERSETDEFAULTSPROC)(NVNtextureBuilder*); +typedef void (*PFNNVNTEXTUREBUILDERSETFLAGSPROC)(NVNtextureBuilder*, int); +typedef void (*PFNNVNTEXTUREBUILDERSETTARGETPROC)(NVNtextureBuilder*, NVNtextureTarget); +typedef void (*PFNNVNTEXTUREBUILDERSETWIDTHPROC)(NVNtextureBuilder*, int); +typedef void (*PFNNVNTEXTUREBUILDERSETHEIGHTPROC)(NVNtextureBuilder*, int); +typedef void (*PFNNVNTEXTUREBUILDERSETDEPTHPROC)(NVNtextureBuilder*, int); +typedef void (*PFNNVNTEXTUREBUILDERSETSIZE1DPROC)(NVNtextureBuilder*, int); +typedef void (*PFNNVNTEXTUREBUILDERSETSIZE2DPROC)(NVNtextureBuilder*, int, int); +typedef void (*PFNNVNTEXTUREBUILDERSETSIZE3DPROC)(NVNtextureBuilder*, int, int, int); +typedef void (*PFNNVNTEXTUREBUILDERSETLEVELSPROC)(NVNtextureBuilder*, int); +typedef void (*PFNNVNTEXTUREBUILDERSETFORMATPROC)(NVNtextureBuilder*, NVNformat); +typedef void (*PFNNVNTEXTUREBUILDERSETSAMPLESPROC)(NVNtextureBuilder*, int); +typedef void (*PFNNVNTEXTUREBUILDERSETSWIZZLEPROC)(NVNtextureBuilder*, NVNtextureSwizzle, + NVNtextureSwizzle, NVNtextureSwizzle, + NVNtextureSwizzle); +typedef void (*PFNNVNTEXTUREBUILDERSETDEPTHSTENCILMODEPROC)(NVNtextureBuilder*, + NVNtextureDepthStencilMode); +typedef size_t (*PFNNVNTEXTUREBUILDERGETSTORAGESIZEPROC)(const NVNtextureBuilder*); +typedef size_t (*PFNNVNTEXTUREBUILDERGETSTORAGEALIGNMENTPROC)(const NVNtextureBuilder*); +typedef void (*PFNNVNTEXTUREBUILDERSETSTORAGEPROC)(NVNtextureBuilder*, NVNmemoryPool*, ptrdiff_t); +typedef void (*PFNNVNTEXTUREBUILDERSETPACKAGEDTEXTUREDATAPROC)(NVNtextureBuilder*, const void*); +typedef void (*PFNNVNTEXTUREBUILDERSETPACKAGEDTEXTURELAYOUTPROC)(NVNtextureBuilder*, + const NVNpackagedTextureLayout*); +typedef void (*PFNNVNTEXTUREBUILDERSETSTRIDEPROC)(NVNtextureBuilder*, ptrdiff_t); +typedef void (*PFNNVNTEXTUREBUILDERSETGLTEXTURENAMEPROC)(NVNtextureBuilder*, uint32_t); +typedef NVNstorageClass (*PFNNVNTEXTUREBUILDERGETSTORAGECLASSPROC)(const NVNtextureBuilder*); +typedef NVNtextureFlags (*PFNNVNTEXTUREBUILDERGETFLAGSPROC)(const NVNtextureBuilder*); +typedef NVNtextureTarget (*PFNNVNTEXTUREBUILDERGETTARGETPROC)(const NVNtextureBuilder*); +typedef int (*PFNNVNTEXTUREBUILDERGETWIDTHPROC)(const NVNtextureBuilder*); +typedef int (*PFNNVNTEXTUREBUILDERGETHEIGHTPROC)(const NVNtextureBuilder*); +typedef int (*PFNNVNTEXTUREBUILDERGETDEPTHPROC)(const NVNtextureBuilder*); +typedef int (*PFNNVNTEXTUREBUILDERGETLEVELSPROC)(const NVNtextureBuilder*); +typedef NVNformat (*PFNNVNTEXTUREBUILDERGETFORMATPROC)(const NVNtextureBuilder*); +typedef int (*PFNNVNTEXTUREBUILDERGETSAMPLESPROC)(const NVNtextureBuilder*); +typedef void (*PFNNVNTEXTUREBUILDERGETSWIZZLEPROC)(const NVNtextureBuilder*, NVNtextureSwizzle*, + NVNtextureSwizzle*, NVNtextureSwizzle*, + NVNtextureSwizzle*); +typedef NVNtextureDepthStencilMode (*PFNNVNTEXTUREBUILDERGETDEPTHSTENCILMODEPROC)( + const NVNtextureBuilder*); +typedef const void* (*PFNNVNTEXTUREBUILDERGETPACKAGEDTEXTUREDATAPROC)(const NVNtextureBuilder*); +typedef ptrdiff_t (*PFNNVNTEXTUREBUILDERGETSTRIDEPROC)(const NVNtextureBuilder*); +typedef void (*PFNNVNTEXTUREBUILDERGETSPARSETILELAYOUTPROC)(const NVNtextureBuilder*, + NVNtextureSparseTileLayout*); +typedef uint32_t (*PFNNVNTEXTUREBUILDERGETGLTEXTURENAMEPROC)(const NVNtextureBuilder*); +typedef size_t (*PFNNVNTEXTUREBUILDERGETZCULLSTORAGESIZEPROC)(const NVNtextureBuilder*); +typedef NVNmemoryPool (*PFNNVNTEXTUREBUILDERGETMEMORYPOOLPROC)(const NVNtextureBuilder*); +typedef ptrdiff_t (*PFNNVNTEXTUREBUILDERGETMEMORYOFFSETPROC)(const NVNtextureBuilder*); +typedef void (*PFNNVNTEXTUREVIEWSETDEFAULTSPROC)(NVNtextureView*); +typedef void (*PFNNVNTEXTUREVIEWSETLEVELSPROC)(NVNtextureView*, int, int); +typedef void (*PFNNVNTEXTUREVIEWSETLAYERSPROC)(NVNtextureView*, int, int); +typedef void (*PFNNVNTEXTUREVIEWSETFORMATPROC)(NVNtextureView*, NVNformat); +typedef void (*PFNNVNTEXTUREVIEWSETSWIZZLEPROC)(NVNtextureView*, NVNtextureSwizzle, + NVNtextureSwizzle, NVNtextureSwizzle, + NVNtextureSwizzle); +typedef void (*PFNNVNTEXTUREVIEWSETDEPTHSTENCILMODEPROC)(NVNtextureView*, + NVNtextureDepthStencilMode); +typedef void (*PFNNVNTEXTUREVIEWSETTARGETPROC)(NVNtextureView*, NVNtextureTarget); +typedef NVNboolean (*PFNNVNTEXTUREVIEWGETLEVELSPROC)(const NVNtextureView*, int*, int*); +typedef NVNboolean (*PFNNVNTEXTUREVIEWGETLAYERSPROC)(const NVNtextureView*, int*, int*); +typedef NVNboolean (*PFNNVNTEXTUREVIEWGETFORMATPROC)(const NVNtextureView*, NVNformat*); +typedef NVNboolean (*PFNNVNTEXTUREVIEWGETSWIZZLEPROC)(const NVNtextureView*, NVNtextureSwizzle*, + NVNtextureSwizzle*, NVNtextureSwizzle*, + NVNtextureSwizzle*); +typedef NVNboolean (*PFNNVNTEXTUREVIEWGETDEPTHSTENCILMODEPROC)(const NVNtextureView*, + NVNtextureDepthStencilMode*); +typedef NVNboolean (*PFNNVNTEXTUREVIEWGETTARGETPROC)(const NVNtextureView*, NVNtextureTarget*); +typedef NVNboolean (*PFNNVNTEXTUREVIEWCOMPAREPROC)(const NVNtextureView*, const NVNtextureView*); +typedef NVNboolean (*PFNNVNTEXTUREINITIALIZEPROC)(NVNtexture*, const NVNtextureBuilder*); +typedef size_t (*PFNNVNTEXTUREGETZCULLSTORAGESIZEPROC)(const NVNtexture*); +typedef void (*PFNNVNTEXTUREFINALIZEPROC)(NVNtexture*); +typedef void (*PFNNVNTEXTURESETDEBUGLABELPROC)(NVNtexture*, const char*); +typedef NVNstorageClass (*PFNNVNTEXTUREGETSTORAGECLASSPROC)(const NVNtexture*); +typedef ptrdiff_t (*PFNNVNTEXTUREGETVIEWOFFSETPROC)(const NVNtexture*, const NVNtextureView*); +typedef NVNtextureFlags (*PFNNVNTEXTUREGETFLAGSPROC)(const NVNtexture*); +typedef NVNtextureTarget (*PFNNVNTEXTUREGETTARGETPROC)(const NVNtexture*); +typedef int (*PFNNVNTEXTUREGETWIDTHPROC)(const NVNtexture*); +typedef int (*PFNNVNTEXTUREGETHEIGHTPROC)(const NVNtexture*); +typedef int (*PFNNVNTEXTUREGETDEPTHPROC)(const NVNtexture*); +typedef int (*PFNNVNTEXTUREGETLEVELSPROC)(const NVNtexture*); +typedef NVNformat (*PFNNVNTEXTUREGETFORMATPROC)(const NVNtexture*); +typedef int (*PFNNVNTEXTUREGETSAMPLESPROC)(const NVNtexture*); +typedef void (*PFNNVNTEXTUREGETSWIZZLEPROC)(const NVNtexture*, NVNtextureSwizzle*, + NVNtextureSwizzle*, NVNtextureSwizzle*, + NVNtextureSwizzle*); +typedef NVNtextureDepthStencilMode (*PFNNVNTEXTUREGETDEPTHSTENCILMODEPROC)(const NVNtexture*); +typedef ptrdiff_t (*PFNNVNTEXTUREGETSTRIDEPROC)(const NVNtexture*); +typedef NVNtextureAddress (*PFNNVNTEXTUREGETTEXTUREADDRESSPROC)(const NVNtexture*); +typedef void (*PFNNVNTEXTUREGETSPARSETILELAYOUTPROC)(const NVNtexture*, + NVNtextureSparseTileLayout*); +typedef void (*PFNNVNTEXTUREWRITETEXELSPROC)(const NVNtexture*, const NVNtextureView*, + const NVNcopyRegion*, const void*); +typedef void (*PFNNVNTEXTUREWRITETEXELSSTRIDEDPROC)(const NVNtexture*, const NVNtextureView*, + const NVNcopyRegion*, const void*, ptrdiff_t, + ptrdiff_t); +typedef void (*PFNNVNTEXTUREREADTEXELSPROC)(const NVNtexture*, const NVNtextureView*, + const NVNcopyRegion*, void*); +typedef void (*PFNNVNTEXTUREREADTEXELSSTRIDEDPROC)(const NVNtexture*, const NVNtextureView*, + const NVNcopyRegion*, void*, ptrdiff_t, + ptrdiff_t); +typedef void (*PFNNVNTEXTUREFLUSHTEXELSPROC)(const NVNtexture*, const NVNtextureView*, + const NVNcopyRegion*); +typedef void (*PFNNVNTEXTUREINVALIDATETEXELSPROC)(const NVNtexture*, const NVNtextureView*, + const NVNcopyRegion*); +typedef NVNmemoryPool (*PFNNVNTEXTUREGETMEMORYPOOLPROC)(const NVNtexture*); +typedef ptrdiff_t (*PFNNVNTEXTUREGETMEMORYOFFSETPROC)(const NVNtexture*); +typedef int (*PFNNVNTEXTUREGETSTORAGESIZEPROC)(const NVNtexture*); +typedef NVNboolean (*PFNNVNTEXTURECOMPAREPROC)(const NVNtexture*, const NVNtexture*); +typedef uint64_t (*PFNNVNTEXTUREGETDEBUGIDPROC)(const NVNtexture*); +typedef void (*PFNNVNSAMPLERBUILDERSETDEVICEPROC)(NVNsamplerBuilder*, NVNdevice*); +typedef void (*PFNNVNSAMPLERBUILDERSETDEFAULTSPROC)(NVNsamplerBuilder*); +typedef void (*PFNNVNSAMPLERBUILDERSETMINMAGFILTERPROC)(NVNsamplerBuilder*, NVNminFilter, + NVNmagFilter); +typedef void (*PFNNVNSAMPLERBUILDERSETWRAPMODEPROC)(NVNsamplerBuilder*, NVNwrapMode, NVNwrapMode, + NVNwrapMode); +typedef void (*PFNNVNSAMPLERBUILDERSETLODCLAMPPROC)(NVNsamplerBuilder*, float, float); +typedef void (*PFNNVNSAMPLERBUILDERSETLODBIASPROC)(NVNsamplerBuilder*, float); +typedef void (*PFNNVNSAMPLERBUILDERSETCOMPAREPROC)(NVNsamplerBuilder*, NVNcompareMode, + NVNcompareFunc); +typedef void (*PFNNVNSAMPLERBUILDERSETBORDERCOLORPROC)(NVNsamplerBuilder*, const float*); +typedef void (*PFNNVNSAMPLERBUILDERSETBORDERCOLORIPROC)(NVNsamplerBuilder*, const int*); +typedef void (*PFNNVNSAMPLERBUILDERSETBORDERCOLORUIPROC)(NVNsamplerBuilder*, const uint32_t*); +typedef void (*PFNNVNSAMPLERBUILDERSETMAXANISOTROPYPROC)(NVNsamplerBuilder*, float); +typedef void (*PFNNVNSAMPLERBUILDERSETREDUCTIONFILTERPROC)(NVNsamplerBuilder*, NVNsamplerReduction); +typedef void (*PFNNVNSAMPLERBUILDERSETLODSNAPPROC)(NVNsamplerBuilder*, float); +typedef void (*PFNNVNSAMPLERBUILDERGETMINMAGFILTERPROC)(const NVNsamplerBuilder*, NVNminFilter*, + NVNmagFilter*); +typedef void (*PFNNVNSAMPLERBUILDERGETWRAPMODEPROC)(const NVNsamplerBuilder*, NVNwrapMode*, + NVNwrapMode*, NVNwrapMode*); +typedef void (*PFNNVNSAMPLERBUILDERGETLODCLAMPPROC)(const NVNsamplerBuilder*, float*, float*); +typedef float (*PFNNVNSAMPLERBUILDERGETLODBIASPROC)(const NVNsamplerBuilder*); +typedef void (*PFNNVNSAMPLERBUILDERGETCOMPAREPROC)(const NVNsamplerBuilder*, NVNcompareMode*, + NVNcompareFunc*); +typedef void (*PFNNVNSAMPLERBUILDERGETBORDERCOLORPROC)(const NVNsamplerBuilder*, float*); +typedef void (*PFNNVNSAMPLERBUILDERGETBORDERCOLORIPROC)(const NVNsamplerBuilder*, int*); +typedef void (*PFNNVNSAMPLERBUILDERGETBORDERCOLORUIPROC)(const NVNsamplerBuilder*, uint32_t*); +typedef float (*PFNNVNSAMPLERBUILDERGETMAXANISOTROPYPROC)(const NVNsamplerBuilder*); +typedef NVNsamplerReduction (*PFNNVNSAMPLERBUILDERGETREDUCTIONFILTERPROC)(const NVNsamplerBuilder*); +typedef float (*PFNNVNSAMPLERBUILDERGETLODSNAPPROC)(const NVNsamplerBuilder*); +typedef NVNboolean (*PFNNVNSAMPLERINITIALIZEPROC)(NVNsampler*, const NVNsamplerBuilder*); +typedef void (*PFNNVNSAMPLERFINALIZEPROC)(NVNsampler*); +typedef void (*PFNNVNSAMPLERSETDEBUGLABELPROC)(NVNsampler*, const char*); +typedef void (*PFNNVNSAMPLERGETMINMAGFILTERPROC)(const NVNsampler*, NVNminFilter*, NVNmagFilter*); +typedef void (*PFNNVNSAMPLERGETWRAPMODEPROC)(const NVNsampler*, NVNwrapMode*, NVNwrapMode*, + NVNwrapMode*); +typedef void (*PFNNVNSAMPLERGETLODCLAMPPROC)(const NVNsampler*, float*, float*); +typedef float (*PFNNVNSAMPLERGETLODBIASPROC)(const NVNsampler*); +typedef void (*PFNNVNSAMPLERGETCOMPAREPROC)(const NVNsampler*, NVNcompareMode*, NVNcompareFunc*); +typedef void (*PFNNVNSAMPLERGETBORDERCOLORPROC)(const NVNsampler*, float*); +typedef void (*PFNNVNSAMPLERGETBORDERCOLORIPROC)(const NVNsampler*, int*); +typedef void (*PFNNVNSAMPLERGETBORDERCOLORUIPROC)(const NVNsampler*, uint32_t*); +typedef float (*PFNNVNSAMPLERGETMAXANISOTROPYPROC)(const NVNsampler*); +typedef NVNsamplerReduction (*PFNNVNSAMPLERGETREDUCTIONFILTERPROC)(const NVNsampler*); +typedef NVNboolean (*PFNNVNSAMPLERCOMPAREPROC)(const NVNsampler*, const NVNsampler*); +typedef uint64_t (*PFNNVNSAMPLERGETDEBUGIDPROC)(const NVNsampler*); +typedef void (*PFNNVNBLENDSTATESETDEFAULTSPROC)(NVNblendState*); +typedef void (*PFNNVNBLENDSTATESETBLENDTARGETPROC)(NVNblendState*, int); +typedef void (*PFNNVNBLENDSTATESETBLENDFUNCPROC)(NVNblendState*, NVNblendFunc, NVNblendFunc, + NVNblendFunc, NVNblendFunc); +typedef void (*PFNNVNBLENDSTATESETBLENDEQUATIONPROC)(NVNblendState*, NVNblendEquation, + NVNblendEquation); +typedef void (*PFNNVNBLENDSTATESETADVANCEDMODEPROC)(NVNblendState*, NVNblendAdvancedMode); +typedef void (*PFNNVNBLENDSTATESETADVANCEDOVERLAPPROC)(NVNblendState*, NVNblendAdvancedOverlap); +typedef void (*PFNNVNBLENDSTATESETADVANCEDPREMULTIPLIEDSRCPROC)(NVNblendState*, NVNboolean); +typedef void (*PFNNVNBLENDSTATESETADVANCEDNORMALIZEDDSTPROC)(NVNblendState*, NVNboolean); +typedef int (*PFNNVNBLENDSTATEGETBLENDTARGETPROC)(const NVNblendState*); +typedef void (*PFNNVNBLENDSTATEGETBLENDFUNCPROC)(const NVNblendState*, NVNblendFunc*, NVNblendFunc*, + NVNblendFunc*, NVNblendFunc*); +typedef void (*PFNNVNBLENDSTATEGETBLENDEQUATIONPROC)(const NVNblendState*, NVNblendEquation*, + NVNblendEquation*); +typedef NVNblendAdvancedMode (*PFNNVNBLENDSTATEGETADVANCEDMODEPROC)(const NVNblendState*); +typedef NVNblendAdvancedOverlap (*PFNNVNBLENDSTATEGETADVANCEDOVERLAPPROC)(const NVNblendState*); +typedef NVNboolean (*PFNNVNBLENDSTATEGETADVANCEDPREMULTIPLIEDSRCPROC)(const NVNblendState*); +typedef NVNboolean (*PFNNVNBLENDSTATEGETADVANCEDNORMALIZEDDSTPROC)(const NVNblendState*); +typedef void (*PFNNVNCOLORSTATESETDEFAULTSPROC)(NVNcolorState*); +typedef void (*PFNNVNCOLORSTATESETBLENDENABLEPROC)(NVNcolorState*, int, NVNboolean); +typedef void (*PFNNVNCOLORSTATESETLOGICOPPROC)(NVNcolorState*, NVNlogicOp); +typedef void (*PFNNVNCOLORSTATESETALPHATESTPROC)(NVNcolorState*, NVNalphaFunc); +typedef NVNboolean (*PFNNVNCOLORSTATEGETBLENDENABLEPROC)(const NVNcolorState*, int); +typedef NVNlogicOp (*PFNNVNCOLORSTATEGETLOGICOPPROC)(const NVNcolorState*); +typedef NVNalphaFunc (*PFNNVNCOLORSTATEGETALPHATESTPROC)(const NVNcolorState*); +typedef void (*PFNNVNCHANNELMASKSTATESETDEFAULTSPROC)(NVNchannelMaskState*); +typedef void (*PFNNVNCHANNELMASKSTATESETCHANNELMASKPROC)(NVNchannelMaskState*, int, NVNboolean, + NVNboolean, NVNboolean, NVNboolean); +typedef void (*PFNNVNCHANNELMASKSTATEGETCHANNELMASKPROC)(const NVNchannelMaskState*, int, + NVNboolean*, NVNboolean*, NVNboolean*, + NVNboolean*); +typedef void (*PFNNVNMULTISAMPLESTATESETDEFAULTSPROC)(NVNmultisampleState*); +typedef void (*PFNNVNMULTISAMPLESTATESETMULTISAMPLEENABLEPROC)(NVNmultisampleState*, NVNboolean); +typedef void (*PFNNVNMULTISAMPLESTATESETSAMPLESPROC)(NVNmultisampleState*, int); +typedef void (*PFNNVNMULTISAMPLESTATESETALPHATOCOVERAGEENABLEPROC)(NVNmultisampleState*, + NVNboolean); +typedef void (*PFNNVNMULTISAMPLESTATESETALPHATOCOVERAGEDITHERPROC)(NVNmultisampleState*, + NVNboolean); +typedef NVNboolean (*PFNNVNMULTISAMPLESTATEGETMULTISAMPLEENABLEPROC)(const NVNmultisampleState*); +typedef int (*PFNNVNMULTISAMPLESTATEGETSAMPLESPROC)(const NVNmultisampleState*); +typedef NVNboolean (*PFNNVNMULTISAMPLESTATEGETALPHATOCOVERAGEENABLEPROC)( + const NVNmultisampleState*); +typedef NVNboolean (*PFNNVNMULTISAMPLESTATEGETALPHATOCOVERAGEDITHERPROC)( + const NVNmultisampleState*); +typedef void (*PFNNVNMULTISAMPLESTATESETRASTERSAMPLESPROC)(NVNmultisampleState*, int); +typedef int (*PFNNVNMULTISAMPLESTATEGETRASTERSAMPLESPROC)(NVNmultisampleState*); +typedef void (*PFNNVNMULTISAMPLESTATESETCOVERAGEMODULATIONMODEPROC)(NVNmultisampleState*, + NVNcoverageModulationMode); +typedef NVNcoverageModulationMode (*PFNNVNMULTISAMPLESTATEGETCOVERAGEMODULATIONMODEPROC)( + const NVNmultisampleState*); +typedef void (*PFNNVNMULTISAMPLESTATESETCOVERAGETOCOLORENABLEPROC)(NVNmultisampleState*, + NVNboolean); +typedef NVNboolean (*PFNNVNMULTISAMPLESTATEGETCOVERAGETOCOLORENABLEPROC)( + const NVNmultisampleState*); +typedef void (*PFNNVNMULTISAMPLESTATESETCOVERAGETOCOLOROUTPUTPROC)(NVNmultisampleState*, int); +typedef int (*PFNNVNMULTISAMPLESTATEGETCOVERAGETOCOLOROUTPUTPROC)(const NVNmultisampleState*); +typedef void (*PFNNVNMULTISAMPLESTATESETSAMPLELOCATIONSENABLEPROC)(NVNmultisampleState*, + NVNboolean); +typedef NVNboolean (*PFNNVNMULTISAMPLESTATEGETSAMPLELOCATIONSENABLEPROC)( + const NVNmultisampleState*); +typedef void (*PFNNVNMULTISAMPLESTATEGETSAMPLELOCATIONSGRIDPROC)(NVNmultisampleState*, int*, int*); +typedef void (*PFNNVNMULTISAMPLESTATESETSAMPLELOCATIONSGRIDENABLEPROC)(NVNmultisampleState*, + NVNboolean); +typedef NVNboolean (*PFNNVNMULTISAMPLESTATEGETSAMPLELOCATIONSGRIDENABLEPROC)( + const NVNmultisampleState*); +typedef void (*PFNNVNMULTISAMPLESTATESETSAMPLELOCATIONSPROC)(NVNmultisampleState*, int, int, + const float*); +typedef void (*PFNNVNPOLYGONSTATESETDEFAULTSPROC)(NVNpolygonState*); +typedef void (*PFNNVNPOLYGONSTATESETCULLFACEPROC)(NVNpolygonState*, NVNface); +typedef void (*PFNNVNPOLYGONSTATESETFRONTFACEPROC)(NVNpolygonState*, NVNfrontFace); +typedef void (*PFNNVNPOLYGONSTATESETPOLYGONMODEPROC)(NVNpolygonState*, NVNpolygonMode); +typedef void (*PFNNVNPOLYGONSTATESETPOLYGONOFFSETENABLESPROC)(NVNpolygonState*, int); +typedef NVNface (*PFNNVNPOLYGONSTATEGETCULLFACEPROC)(const NVNpolygonState*); +typedef NVNfrontFace (*PFNNVNPOLYGONSTATEGETFRONTFACEPROC)(const NVNpolygonState*); +typedef NVNpolygonMode (*PFNNVNPOLYGONSTATEGETPOLYGONMODEPROC)(const NVNpolygonState*); +typedef NVNpolygonOffsetEnable (*PFNNVNPOLYGONSTATEGETPOLYGONOFFSETENABLESPROC)( + const NVNpolygonState*); +typedef void (*PFNNVNDEPTHSTENCILSTATESETDEFAULTSPROC)(NVNdepthStencilState*); +typedef void (*PFNNVNDEPTHSTENCILSTATESETDEPTHTESTENABLEPROC)(NVNdepthStencilState*, NVNboolean); +typedef void (*PFNNVNDEPTHSTENCILSTATESETDEPTHWRITEENABLEPROC)(NVNdepthStencilState*, NVNboolean); +typedef void (*PFNNVNDEPTHSTENCILSTATESETDEPTHFUNCPROC)(NVNdepthStencilState*, NVNdepthFunc); +typedef void (*PFNNVNDEPTHSTENCILSTATESETSTENCILTESTENABLEPROC)(NVNdepthStencilState*, NVNboolean); +typedef void (*PFNNVNDEPTHSTENCILSTATESETSTENCILFUNCPROC)(NVNdepthStencilState*, NVNface, + NVNstencilFunc); +typedef void (*PFNNVNDEPTHSTENCILSTATESETSTENCILOPPROC)(NVNdepthStencilState*, NVNface, + NVNstencilOp, NVNstencilOp, NVNstencilOp); +typedef NVNboolean (*PFNNVNDEPTHSTENCILSTATEGETDEPTHTESTENABLEPROC)(const NVNdepthStencilState*); +typedef NVNboolean (*PFNNVNDEPTHSTENCILSTATEGETDEPTHWRITEENABLEPROC)(const NVNdepthStencilState*); +typedef NVNdepthFunc (*PFNNVNDEPTHSTENCILSTATEGETDEPTHFUNCPROC)(const NVNdepthStencilState*); +typedef NVNboolean (*PFNNVNDEPTHSTENCILSTATEGETSTENCILTESTENABLEPROC)(const NVNdepthStencilState*); +typedef NVNstencilFunc (*PFNNVNDEPTHSTENCILSTATEGETSTENCILFUNCPROC)(const NVNdepthStencilState*, + NVNface); +typedef void (*PFNNVNDEPTHSTENCILSTATEGETSTENCILOPPROC)(const NVNdepthStencilState*, NVNface, + NVNstencilOp*, NVNstencilOp*, + NVNstencilOp*); +typedef void (*PFNNVNVERTEXATTRIBSTATESETDEFAULTSPROC)(NVNvertexAttribState*); +typedef void (*PFNNVNVERTEXATTRIBSTATESETFORMATPROC)(NVNvertexAttribState*, NVNformat, ptrdiff_t); +typedef void (*PFNNVNVERTEXATTRIBSTATESETSTREAMINDEXPROC)(NVNvertexAttribState*, int); +typedef void (*PFNNVNVERTEXATTRIBSTATEGETFORMATPROC)(const NVNvertexAttribState*, NVNformat*, + ptrdiff_t*); +typedef int (*PFNNVNVERTEXATTRIBSTATEGETSTREAMINDEXPROC)(const NVNvertexAttribState*); +typedef void (*PFNNVNVERTEXSTREAMSTATESETDEFAULTSPROC)(NVNvertexStreamState*); +typedef void (*PFNNVNVERTEXSTREAMSTATESETSTRIDEPROC)(NVNvertexStreamState*, ptrdiff_t); +typedef void (*PFNNVNVERTEXSTREAMSTATESETDIVISORPROC)(NVNvertexStreamState*, int); +typedef ptrdiff_t (*PFNNVNVERTEXSTREAMSTATEGETSTRIDEPROC)(const NVNvertexStreamState*); +typedef int (*PFNNVNVERTEXSTREAMSTATEGETDIVISORPROC)(const NVNvertexStreamState*); +typedef NVNboolean (*PFNNVNCOMMANDBUFFERINITIALIZEPROC)(NVNcommandBuffer*, NVNdevice*); +typedef void (*PFNNVNCOMMANDBUFFERFINALIZEPROC)(NVNcommandBuffer*); +typedef void (*PFNNVNCOMMANDBUFFERSETDEBUGLABELPROC)(NVNcommandBuffer*, const char*); +typedef void (*PFNNVNCOMMANDBUFFERSETMEMORYCALLBACKPROC)(NVNcommandBuffer*, + PFNNVNCOMMANDBUFFERMEMORYCALLBACKPROC); +typedef void (*PFNNVNCOMMANDBUFFERSETMEMORYCALLBACKDATAPROC)(NVNcommandBuffer*, void*); +typedef void (*PFNNVNCOMMANDBUFFERADDCOMMANDMEMORYPROC)(NVNcommandBuffer*, const NVNmemoryPool*, + ptrdiff_t, size_t); +typedef void (*PFNNVNCOMMANDBUFFERADDCONTROLMEMORYPROC)(NVNcommandBuffer*, void*, size_t); +typedef size_t (*PFNNVNCOMMANDBUFFERGETCOMMANDMEMORYSIZEPROC)(const NVNcommandBuffer*); +typedef size_t (*PFNNVNCOMMANDBUFFERGETCOMMANDMEMORYUSEDPROC)(const NVNcommandBuffer*); +typedef size_t (*PFNNVNCOMMANDBUFFERGETCOMMANDMEMORYFREEPROC)(const NVNcommandBuffer*); +typedef size_t (*PFNNVNCOMMANDBUFFERGETCONTROLMEMORYSIZEPROC)(const NVNcommandBuffer*); +typedef size_t (*PFNNVNCOMMANDBUFFERGETCONTROLMEMORYUSEDPROC)(const NVNcommandBuffer*); +typedef size_t (*PFNNVNCOMMANDBUFFERGETCONTROLMEMORYFREEPROC)(const NVNcommandBuffer*); +typedef void (*PFNNVNCOMMANDBUFFERBEGINRECORDINGPROC)(NVNcommandBuffer*); +typedef NVNcommandHandle (*PFNNVNCOMMANDBUFFERENDRECORDINGPROC)(NVNcommandBuffer*); +typedef void (*PFNNVNCOMMANDBUFFERCALLCOMMANDSPROC)(NVNcommandBuffer*, int, + const NVNcommandHandle*); +typedef void (*PFNNVNCOMMANDBUFFERCOPYCOMMANDSPROC)(NVNcommandBuffer*, int, + const NVNcommandHandle*); +typedef void (*PFNNVNCOMMANDBUFFERBINDBLENDSTATEPROC)(NVNcommandBuffer*, const NVNblendState*); +typedef void (*PFNNVNCOMMANDBUFFERBINDCHANNELMASKSTATEPROC)(NVNcommandBuffer*, + const NVNchannelMaskState*); +typedef void (*PFNNVNCOMMANDBUFFERBINDCOLORSTATEPROC)(NVNcommandBuffer*, const NVNcolorState*); +typedef void (*PFNNVNCOMMANDBUFFERBINDMULTISAMPLESTATEPROC)(NVNcommandBuffer*, + const NVNmultisampleState*); +typedef void (*PFNNVNCOMMANDBUFFERBINDPOLYGONSTATEPROC)(NVNcommandBuffer*, const NVNpolygonState*); +typedef void (*PFNNVNCOMMANDBUFFERBINDDEPTHSTENCILSTATEPROC)(NVNcommandBuffer*, + const NVNdepthStencilState*); +typedef void (*PFNNVNCOMMANDBUFFERBINDVERTEXATTRIBSTATEPROC)(NVNcommandBuffer*, int, + const NVNvertexAttribState*); +typedef void (*PFNNVNCOMMANDBUFFERBINDVERTEXSTREAMSTATEPROC)(NVNcommandBuffer*, int, + const NVNvertexStreamState*); +typedef void (*PFNNVNCOMMANDBUFFERBINDPROGRAMPROC)(NVNcommandBuffer*, const NVNprogram*, int); +typedef void (*PFNNVNCOMMANDBUFFERBINDVERTEXBUFFERPROC)(NVNcommandBuffer*, int, NVNbufferAddress, + size_t); +typedef void (*PFNNVNCOMMANDBUFFERBINDVERTEXBUFFERSPROC)(NVNcommandBuffer*, int, int, + const NVNbufferRange*); +typedef void (*PFNNVNCOMMANDBUFFERBINDUNIFORMBUFFERPROC)(NVNcommandBuffer*, NVNshaderStage, int, + NVNbufferAddress, size_t); +typedef void (*PFNNVNCOMMANDBUFFERBINDUNIFORMBUFFERSPROC)(NVNcommandBuffer*, NVNshaderStage, int, + int, const NVNbufferRange*); +typedef void (*PFNNVNCOMMANDBUFFERBINDTRANSFORMFEEDBACKBUFFERPROC)(NVNcommandBuffer*, int, + NVNbufferAddress, size_t); +typedef void (*PFNNVNCOMMANDBUFFERBINDTRANSFORMFEEDBACKBUFFERSPROC)(NVNcommandBuffer*, int, int, + const NVNbufferRange*); +typedef void (*PFNNVNCOMMANDBUFFERBINDSTORAGEBUFFERPROC)(NVNcommandBuffer*, NVNshaderStage, int, + NVNbufferAddress, size_t); +typedef void (*PFNNVNCOMMANDBUFFERBINDSTORAGEBUFFERSPROC)(NVNcommandBuffer*, NVNshaderStage, int, + int, const NVNbufferRange*); +typedef void (*PFNNVNCOMMANDBUFFERBINDTEXTUREPROC)(NVNcommandBuffer*, NVNshaderStage, int, + NVNtextureHandle); +typedef void (*PFNNVNCOMMANDBUFFERBINDTEXTURESPROC)(NVNcommandBuffer*, NVNshaderStage, int, int, + const NVNtextureHandle*); +typedef void (*PFNNVNCOMMANDBUFFERBINDIMAGEPROC)(NVNcommandBuffer*, NVNshaderStage, int, + NVNimageHandle); +typedef void (*PFNNVNCOMMANDBUFFERBINDIMAGESPROC)(NVNcommandBuffer*, NVNshaderStage, int, int, + const NVNimageHandle*); +typedef void (*PFNNVNCOMMANDBUFFERSETPATCHSIZEPROC)(NVNcommandBuffer*, int); +typedef void (*PFNNVNCOMMANDBUFFERSETINNERTESSELLATIONLEVELSPROC)(NVNcommandBuffer*, const float*); +typedef void (*PFNNVNCOMMANDBUFFERSETOUTERTESSELLATIONLEVELSPROC)(NVNcommandBuffer*, const float*); +typedef void (*PFNNVNCOMMANDBUFFERSETPRIMITIVERESTARTPROC)(NVNcommandBuffer*, NVNboolean, int); +typedef void (*PFNNVNCOMMANDBUFFERBEGINTRANSFORMFEEDBACKPROC)(NVNcommandBuffer*, NVNbufferAddress); +typedef void (*PFNNVNCOMMANDBUFFERENDTRANSFORMFEEDBACKPROC)(NVNcommandBuffer*, NVNbufferAddress); +typedef void (*PFNNVNCOMMANDBUFFERPAUSETRANSFORMFEEDBACKPROC)(NVNcommandBuffer*, NVNbufferAddress); +typedef void (*PFNNVNCOMMANDBUFFERRESUMETRANSFORMFEEDBACKPROC)(NVNcommandBuffer*, NVNbufferAddress); +typedef void (*PFNNVNCOMMANDBUFFERDRAWTRANSFORMFEEDBACKPROC)(NVNcommandBuffer*, NVNdrawPrimitive, + NVNbufferAddress); +typedef void (*PFNNVNCOMMANDBUFFERDRAWARRAYSPROC)(NVNcommandBuffer*, NVNdrawPrimitive, int, int); +typedef void (*PFNNVNCOMMANDBUFFERDRAWELEMENTSPROC)(NVNcommandBuffer*, NVNdrawPrimitive, + NVNindexType, int, NVNbufferAddress); +typedef void (*PFNNVNCOMMANDBUFFERDRAWELEMENTSBASEVERTEXPROC)(NVNcommandBuffer*, NVNdrawPrimitive, + NVNindexType, int, NVNbufferAddress, + int); +typedef void (*PFNNVNCOMMANDBUFFERDRAWARRAYSINSTANCEDPROC)(NVNcommandBuffer*, NVNdrawPrimitive, int, + int, int, int); +typedef void (*PFNNVNCOMMANDBUFFERDRAWELEMENTSINSTANCEDPROC)(NVNcommandBuffer*, NVNdrawPrimitive, + NVNindexType, int, NVNbufferAddress, + int, int, int); +typedef void (*PFNNVNCOMMANDBUFFERDRAWARRAYSINDIRECTPROC)(NVNcommandBuffer*, NVNdrawPrimitive, + NVNbufferAddress); +typedef void (*PFNNVNCOMMANDBUFFERDRAWELEMENTSINDIRECTPROC)(NVNcommandBuffer*, NVNdrawPrimitive, + NVNindexType, NVNbufferAddress, + NVNbufferAddress); +typedef void (*PFNNVNCOMMANDBUFFERMULTIDRAWARRAYSINDIRECTCOUNTPROC)( + NVNcommandBuffer*, NVNdrawPrimitive, NVNbufferAddress, NVNbufferAddress, int, ptrdiff_t); +typedef void (*PFNNVNCOMMANDBUFFERMULTIDRAWELEMENTSINDIRECTCOUNTPROC)( + NVNcommandBuffer*, NVNdrawPrimitive, NVNindexType, NVNbufferAddress, NVNbufferAddress, + NVNbufferAddress, int, ptrdiff_t); +typedef void (*PFNNVNCOMMANDBUFFERCLEARCOLORPROC)(NVNcommandBuffer*, int, const float*, int); +typedef void (*PFNNVNCOMMANDBUFFERCLEARCOLORIPROC)(NVNcommandBuffer*, int, const int*, int); +typedef void (*PFNNVNCOMMANDBUFFERCLEARCOLORUIPROC)(NVNcommandBuffer*, int, const uint32_t*, int); +typedef void (*PFNNVNCOMMANDBUFFERCLEARDEPTHSTENCILPROC)(NVNcommandBuffer*, float, NVNboolean, int, + int); +typedef void (*PFNNVNCOMMANDBUFFERDISPATCHCOMPUTEPROC)(NVNcommandBuffer*, int, int, int); +typedef void (*PFNNVNCOMMANDBUFFERDISPATCHCOMPUTEINDIRECTPROC)(NVNcommandBuffer*, NVNbufferAddress); +typedef void (*PFNNVNCOMMANDBUFFERSETVIEWPORTPROC)(NVNcommandBuffer*, int, int, int, int); +typedef void (*PFNNVNCOMMANDBUFFERSETVIEWPORTSPROC)(NVNcommandBuffer*, int, int, const float*); +typedef void (*PFNNVNCOMMANDBUFFERSETVIEWPORTSWIZZLESPROC)(NVNcommandBuffer*, int, int, + const NVNviewportSwizzle*); +typedef void (*PFNNVNCOMMANDBUFFERSETSCISSORPROC)(NVNcommandBuffer*, int, int, int, int); +typedef void (*PFNNVNCOMMANDBUFFERSETSCISSORSPROC)(NVNcommandBuffer*, int, int, const int*); +typedef void (*PFNNVNCOMMANDBUFFERSETDEPTHRANGEPROC)(NVNcommandBuffer*, float, float); +typedef void (*PFNNVNCOMMANDBUFFERSETDEPTHBOUNDSPROC)(NVNcommandBuffer*, NVNboolean, float, float); +typedef void (*PFNNVNCOMMANDBUFFERSETDEPTHRANGESPROC)(NVNcommandBuffer*, int, int, const float*); +typedef void (*PFNNVNCOMMANDBUFFERSETTILEDCACHEACTIONPROC)(NVNcommandBuffer*, NVNtiledCacheAction); +typedef void (*PFNNVNCOMMANDBUFFERSETTILEDCACHETILESIZEPROC)(NVNcommandBuffer*, int, int); +typedef void (*PFNNVNCOMMANDBUFFERBINDSEPARATETEXTUREPROC)(NVNcommandBuffer*, NVNshaderStage, int, + NVNseparateTextureHandle); +typedef void (*PFNNVNCOMMANDBUFFERBINDSEPARATESAMPLERPROC)(NVNcommandBuffer*, NVNshaderStage, int, + NVNseparateSamplerHandle); +typedef void (*PFNNVNCOMMANDBUFFERBINDSEPARATETEXTURESPROC)(NVNcommandBuffer*, NVNshaderStage, int, + int, const NVNseparateTextureHandle*); +typedef void (*PFNNVNCOMMANDBUFFERBINDSEPARATESAMPLERSPROC)(NVNcommandBuffer*, NVNshaderStage, int, + int, const NVNseparateSamplerHandle*); +typedef void (*PFNNVNCOMMANDBUFFERSETSTENCILVALUEMASKPROC)(NVNcommandBuffer*, NVNface, int); +typedef void (*PFNNVNCOMMANDBUFFERSETSTENCILMASKPROC)(NVNcommandBuffer*, NVNface, int); +typedef void (*PFNNVNCOMMANDBUFFERSETSTENCILREFPROC)(NVNcommandBuffer*, NVNface, int); +typedef void (*PFNNVNCOMMANDBUFFERSETBLENDCOLORPROC)(NVNcommandBuffer*, const float*); +typedef void (*PFNNVNCOMMANDBUFFERSETPOINTSIZEPROC)(NVNcommandBuffer*, float); +typedef void (*PFNNVNCOMMANDBUFFERSETLINEWIDTHPROC)(NVNcommandBuffer*, float); +typedef void (*PFNNVNCOMMANDBUFFERSETPOLYGONOFFSETCLAMPPROC)(NVNcommandBuffer*, float, float, + float); +typedef void (*PFNNVNCOMMANDBUFFERSETALPHAREFPROC)(NVNcommandBuffer*, float); +typedef void (*PFNNVNCOMMANDBUFFERSETSAMPLEMASKPROC)(NVNcommandBuffer*, int); +typedef void (*PFNNVNCOMMANDBUFFERSETRASTERIZERDISCARDPROC)(NVNcommandBuffer*, NVNboolean); +typedef void (*PFNNVNCOMMANDBUFFERSETDEPTHCLAMPPROC)(NVNcommandBuffer*, NVNboolean); +typedef void (*PFNNVNCOMMANDBUFFERSETCONSERVATIVERASTERENABLEPROC)(NVNcommandBuffer*, NVNboolean); +typedef void (*PFNNVNCOMMANDBUFFERSETCONSERVATIVERASTERDILATEPROC)(NVNcommandBuffer*, float); +typedef void (*PFNNVNCOMMANDBUFFERSETSUBPIXELPRECISIONBIASPROC)(NVNcommandBuffer*, int, int); +typedef void (*PFNNVNCOMMANDBUFFERCOPYBUFFERTOTEXTUREPROC)(NVNcommandBuffer*, NVNbufferAddress, + const NVNtexture*, const NVNtextureView*, + const NVNcopyRegion*, int); +typedef void (*PFNNVNCOMMANDBUFFERCOPYTEXTURETOBUFFERPROC)(NVNcommandBuffer*, const NVNtexture*, + const NVNtextureView*, + const NVNcopyRegion*, NVNbufferAddress, + int); +typedef void (*PFNNVNCOMMANDBUFFERCOPYTEXTURETOTEXTUREPROC)(NVNcommandBuffer*, const NVNtexture*, + const NVNtextureView*, + const NVNcopyRegion*, const NVNtexture*, + const NVNtextureView*, + const NVNcopyRegion*, int); +typedef void (*PFNNVNCOMMANDBUFFERCOPYBUFFERTOBUFFERPROC)(NVNcommandBuffer*, NVNbufferAddress, + NVNbufferAddress, size_t, int); +typedef void (*PFNNVNCOMMANDBUFFERCLEARBUFFERPROC)(NVNcommandBuffer*, NVNbufferAddress, size_t, + uint32_t); +typedef void (*PFNNVNCOMMANDBUFFERCLEARTEXTUREPROC)(NVNcommandBuffer*, const NVNtexture*, + const NVNtextureView*, const NVNcopyRegion*, + const float*, int); +typedef void (*PFNNVNCOMMANDBUFFERCLEARTEXTUREIPROC)(NVNcommandBuffer*, const NVNtexture*, + const NVNtextureView*, const NVNcopyRegion*, + const int*, int); +typedef void (*PFNNVNCOMMANDBUFFERCLEARTEXTUREUIPROC)(NVNcommandBuffer*, const NVNtexture*, + const NVNtextureView*, const NVNcopyRegion*, + const uint32_t*, int); +typedef void (*PFNNVNCOMMANDBUFFERUPDATEUNIFORMBUFFERPROC)(NVNcommandBuffer*, NVNbufferAddress, + size_t, ptrdiff_t, size_t, const void*); +typedef void (*PFNNVNCOMMANDBUFFERREPORTCOUNTERPROC)(NVNcommandBuffer*, NVNcounterType, + NVNbufferAddress); +typedef void (*PFNNVNCOMMANDBUFFERRESETCOUNTERPROC)(NVNcommandBuffer*, NVNcounterType); +typedef void (*PFNNVNCOMMANDBUFFERREPORTVALUEPROC)(NVNcommandBuffer*, uint32_t, NVNbufferAddress); +typedef void (*PFNNVNCOMMANDBUFFERSETRENDERENABLEPROC)(NVNcommandBuffer*, NVNboolean); +typedef void (*PFNNVNCOMMANDBUFFERSETRENDERENABLECONDITIONALPROC)(NVNcommandBuffer*, + NVNconditionalRenderMode, + NVNbufferAddress); +typedef void (*PFNNVNCOMMANDBUFFERSETRENDERTARGETSPROC)(NVNcommandBuffer*, int, + const NVNtexture* const*, + const NVNtextureView* const*, + const NVNtexture*, const NVNtextureView*); +typedef void (*PFNNVNCOMMANDBUFFERDISCARDCOLORPROC)(NVNcommandBuffer*, int); +typedef void (*PFNNVNCOMMANDBUFFERDISCARDDEPTHSTENCILPROC)(NVNcommandBuffer*); +typedef void (*PFNNVNCOMMANDBUFFERDOWNSAMPLEPROC)(NVNcommandBuffer*, const NVNtexture*, + const NVNtexture*); +typedef void (*PFNNVNCOMMANDBUFFERTILEDDOWNSAMPLEPROC)(NVNcommandBuffer*, const NVNtexture*, + const NVNtexture*); +typedef void (*PFNNVNCOMMANDBUFFERDOWNSAMPLETEXTUREVIEWPROC)(NVNcommandBuffer*, const NVNtexture*, + const NVNtextureView*, + const NVNtexture*, + const NVNtextureView*); +typedef void (*PFNNVNCOMMANDBUFFERTILEDDOWNSAMPLETEXTUREVIEWPROC)(NVNcommandBuffer*, + const NVNtexture*, + const NVNtextureView*, + const NVNtexture*, + const NVNtextureView*); +typedef void (*PFNNVNCOMMANDBUFFERBARRIERPROC)(NVNcommandBuffer*, int); +typedef void (*PFNNVNCOMMANDBUFFERWAITSYNCPROC)(NVNcommandBuffer*, const NVNsync*); +typedef void (*PFNNVNCOMMANDBUFFERFENCESYNCPROC)(NVNcommandBuffer*, NVNsync*, NVNsyncCondition, + int); +typedef void (*PFNNVNCOMMANDBUFFERSETTEXTUREPOOLPROC)(NVNcommandBuffer*, const NVNtexturePool*); +typedef void (*PFNNVNCOMMANDBUFFERSETSAMPLERPOOLPROC)(NVNcommandBuffer*, const NVNsamplerPool*); +typedef void (*PFNNVNCOMMANDBUFFERSETSHADERSCRATCHMEMORYPROC)(NVNcommandBuffer*, + const NVNmemoryPool*, ptrdiff_t, + size_t); +typedef void (*PFNNVNCOMMANDBUFFERSAVEZCULLDATAPROC)(NVNcommandBuffer*, NVNbufferAddress, size_t); +typedef void (*PFNNVNCOMMANDBUFFERRESTOREZCULLDATAPROC)(NVNcommandBuffer*, NVNbufferAddress, + size_t); +typedef void (*PFNNVNCOMMANDBUFFERSETCOPYROWSTRIDEPROC)(NVNcommandBuffer*, ptrdiff_t); +typedef void (*PFNNVNCOMMANDBUFFERSETCOPYIMAGESTRIDEPROC)(NVNcommandBuffer*, ptrdiff_t); +typedef ptrdiff_t (*PFNNVNCOMMANDBUFFERGETCOPYROWSTRIDEPROC)(const NVNcommandBuffer*); +typedef ptrdiff_t (*PFNNVNCOMMANDBUFFERGETCOPYIMAGESTRIDEPROC)(const NVNcommandBuffer*); +typedef void (*PFNNVNCOMMANDBUFFERDRAWTEXTUREPROC)(NVNcommandBuffer*, NVNtextureHandle, + const NVNdrawTextureRegion*, + const NVNdrawTextureRegion*); +typedef NVNboolean (*PFNNVNPROGRAMSETSUBROUTINELINKAGEPROC)(NVNprogram*, int, + const NVNsubroutineLinkageMapPtr*); +typedef void (*PFNNVNCOMMANDBUFFERSETPROGRAMSUBROUTINESPROC)(NVNcommandBuffer*, NVNprogram*, + NVNshaderStage, const int, const int, + const int*); +typedef void (*PFNNVNCOMMANDBUFFERBINDCOVERAGEMODULATIONTABLEPROC)(NVNcommandBuffer*, const float*); +typedef void (*PFNNVNCOMMANDBUFFERRESOLVEDEPTHBUFFERPROC)(NVNcommandBuffer*); +typedef void (*PFNNVNCOMMANDBUFFERPUSHDEBUGGROUPSTATICPROC)(NVNcommandBuffer*, uint32_t, + const char*); +typedef void (*PFNNVNCOMMANDBUFFERPUSHDEBUGGROUPDYNAMICPROC)(NVNcommandBuffer*, uint32_t, + const char*); +typedef void (*PFNNVNCOMMANDBUFFERPUSHDEBUGGROUPPROC)(NVNcommandBuffer*, uint32_t, const char*); +typedef void (*PFNNVNCOMMANDBUFFERPOPDEBUGGROUPPROC)(NVNcommandBuffer*); +typedef void (*PFNNVNCOMMANDBUFFERPOPDEBUGGROUPIDPROC)(NVNcommandBuffer*, uint32_t); +typedef void (*PFNNVNCOMMANDBUFFERINSERTDEBUGMARKERSTATICPROC)(NVNcommandBuffer*, uint32_t, + const char*); +typedef void (*PFNNVNCOMMANDBUFFERINSERTDEBUGMARKERDYNAMICPROC)(NVNcommandBuffer*, uint32_t, + const char*); +typedef void (*PFNNVNCOMMANDBUFFERINSERTDEBUGMARKERPROC)(NVNcommandBuffer*, const char*); +typedef PFNNVNCOMMANDBUFFERMEMORYCALLBACKPROC (*PFNNVNCOMMANDBUFFERGETMEMORYCALLBACKPROC)( + const NVNcommandBuffer*); +typedef void (*PFNNVNCOMMANDBUFFERGETMEMORYCALLBACKDATAPROC)(const NVNcommandBuffer*); +typedef NVNboolean (*PFNNVNCOMMANDBUFFERISRECORDINGPROC)(const NVNcommandBuffer*); +typedef NVNboolean (*PFNNVNSYNCINITIALIZEPROC)(NVNsync*, NVNdevice*); +typedef void (*PFNNVNSYNCFINALIZEPROC)(NVNsync*); +typedef void (*PFNNVNSYNCSETDEBUGLABELPROC)(NVNsync*, const char*); +typedef void (*PFNNVNQUEUEFENCESYNCPROC)(NVNqueue*, NVNsync*, NVNsyncCondition, int); +typedef NVNsyncWaitResult (*PFNNVNSYNCWAITPROC)(const NVNsync*, uint64_t); +typedef NVNboolean (*PFNNVNQUEUEWAITSYNCPROC)(NVNqueue*, const NVNsync*); +typedef void (*PFNNVNEVENTBUILDERSETDEFAULTSPROC)(NVNeventBuilder*); +typedef void (*PFNNVNEVENTBUILDERSETSTORAGEPROC)(NVNeventBuilder*, const NVNmemoryPool*, int64_t); +typedef NVNboolean (*PFNNVNEVENTINITIALIZEPROC)(NVNevent*, const NVNeventBuilder*); +typedef void (*PFNNVNEVENTFINALIZEPROC)(NVNevent*); +typedef uint32_t (*PFNNVNEVENTGETVALUEPROC)(const NVNevent*); +typedef void (*PFNNVNEVENTSIGNALPROC)(NVNevent*, NVNeventSignalMode, uint32_t); +typedef void (*PFNNVNCOMMANDBUFFERWAITEVENTPROC)(NVNcommandBuffer*, const NVNevent*, + NVNeventWaitMode, uint32_t); +typedef void (*PFNNVNCOMMANDBUFFERSIGNALEVENTPROC)(NVNcommandBuffer*, const NVNevent*, + NVNeventSignalMode, NVNeventSignalLocation, int, + uint32_t); \ No newline at end of file diff --git a/src/gfx/nvn/names.txt b/src/gfx/nvn/names.txt new file mode 100644 index 0000000..4937715 --- /dev/null +++ b/src/gfx/nvn/names.txt @@ -0,0 +1,523 @@ +nvnDeviceBuilderSetDefaults +nvnDeviceBuilderSetFlags +nvnDeviceBuilderGetFlags +nvnDeviceInitialize +nvnDeviceFinalize +nvnDeviceSetDebugLabel +nvnDeviceGetProcAddress +nvnDeviceGetInteger +nvnDeviceGetCurrentTimestampInNanoseconds +nvnDeviceSetIntermediateShaderCache +nvnDeviceGetTextureHandle +nvnDeviceGetTexelFetchHandle +nvnDeviceGetImageHandle +nvnDeviceInstallDebugCallback +nvnDeviceGenerateDebugDomainId +nvnDeviceSetWindowOriginMode +nvnDeviceSetDepthMode +nvnDeviceRegisterFastClearColor +nvnDeviceRegisterFastClearColori +nvnDeviceRegisterFastClearColorui +nvnDeviceRegisterFastClearDepth +nvnDeviceGetWindowOriginMode +nvnDeviceGetDepthMode +nvnDeviceGetTimestampInNanoseconds +nvnDeviceApplyDeferredFinalizes +nvnDeviceFinalizeCommandHandle +nvnDeviceWalkDebugDatabase +nvnDeviceGetSeparateTextureHandle +nvnDeviceGetSeparateSamplerHandle +nvnDeviceIsExternalDebuggerAttached +nvnQueueGetError +nvnQueueGetTotalCommandMemoryUsed +nvnQueueGetTotalControlMemoryUsed +nvnQueueGetTotalComputeMemoryUsed +nvnQueueResetMemoryUsageCounts +nvnQueueBuilderSetDevice +nvnQueueBuilderSetDefaults +nvnQueueBuilderSetFlags +nvnQueueBuilderSetCommandMemorySize +nvnQueueBuilderSetComputeMemorySize +nvnQueueBuilderSetControlMemorySize +nvnQueueBuilderGetQueueMemorySize +nvnQueueBuilderSetQueueMemory +nvnQueueBuilderSetCommandFlushThreshold +nvnQueueBuilderSetQueuePriority +nvnQueueBuilderGetQueuePriority +nvnQueueBuilderGetDevice +nvnQueueBuilderGetFlags +nvnQueueBuilderGetCommandMemorySize +nvnQueueBuilderGetComputeMemorySize +nvnQueueBuilderGetControlMemorySize +nvnQueueBuilderGetCommandFlushThreshold +nvnQueueBuilderGetMemorySize +nvnQueueBuilderGetMemory +nvnQueueInitialize +nvnQueueFinalize +nvnQueueSetDebugLabel +nvnQueueSubmitCommands +nvnQueueFlush +nvnQueueFinish +nvnQueuePresentTexture +nvnQueueAcquireTexture +nvnWindowBuilderSetDevice +nvnWindowBuilderSetDefaults +nvnWindowBuilderSetNativeWindow +nvnWindowBuilderSetTextures +nvnWindowBuilderSetPresentInterval +nvnWindowBuilderSetNumActiveTextures +nvnWindowBuilderGetDevice +nvnWindowBuilderGetNumTextures +nvnWindowBuilderGetTexture +nvnWindowBuilderGetNativeWindow +nvnWindowBuilderGetPresentInterval +nvnWindowBuilderGetNumActiveTextures +nvnWindowInitialize +nvnWindowFinalize +nvnWindowSetDebugLabel +nvnWindowAcquireTexture +nvnWindowGetNativeWindow +nvnWindowGetPresentInterval +nvnWindowSetPresentInterval +nvnWindowSetCrop +nvnWindowGetCrop +nvnWindowSetNumActiveTextures +nvnWindowGetNumActiveTextures +nvnWindowGetNumTextures +nvnProgramInitialize +nvnProgramFinalize +nvnProgramSetDebugLabel +nvnProgramSetShaders +nvnProgramSetShadersExt +nvnProgramSetSampleShading +nvnMemoryPoolBuilderSetDevice +nvnMemoryPoolBuilderSetDefaults +nvnMemoryPoolBuilderSetStorage +nvnMemoryPoolBuilderSetFlags +nvnMemoryPoolBuilderGetDevice +nvnMemoryPoolBuilderGetMemory +nvnMemoryPoolBuilderGetSize +nvnMemoryPoolBuilderGetFlags +nvnMemoryPoolInitialize +nvnMemoryPoolSetDebugLabel +nvnMemoryPoolFinalize +nvnMemoryPoolMap +nvnMemoryPoolFlushMappedRange +nvnMemoryPoolInvalidateMappedRange +nvnMemoryPoolGetBufferAddress +nvnMemoryPoolMapVirtual +nvnMemoryPoolGetSize +nvnMemoryPoolGetFlags +nvnTexturePoolInitialize +nvnTexturePoolSetDebugLabel +nvnTexturePoolFinalize +nvnTexturePoolRegisterTexture +nvnTexturePoolRegisterImage +nvnTexturePoolGetMemoryPool +nvnTexturePoolGetMemoryOffset +nvnTexturePoolGetSize +nvnSamplerPoolInitialize +nvnSamplerPoolSetDebugLabel +nvnSamplerPoolFinalize +nvnSamplerPoolRegisterSampler +nvnSamplerPoolRegisterSamplerBuilder +nvnSamplerPoolGetMemoryPool +nvnSamplerPoolGetMemoryOffset +nvnSamplerPoolGetSize +nvnBufferBuilderSetDevice +nvnBufferBuilderSetDefaults +nvnBufferBuilderSetStorage +nvnBufferBuilderGetDevice +nvnBufferBuilderGetMemoryPool +nvnBufferBuilderGetMemoryOffset +nvnBufferBuilderGetSize +nvnBufferInitialize +nvnBufferSetDebugLabel +nvnBufferFinalize +nvnBufferMap +nvnBufferGetAddress +nvnBufferFlushMappedRange +nvnBufferInvalidateMappedRange +nvnBufferGetMemoryPool +nvnBufferGetMemoryOffset +nvnBufferGetSize +nvnBufferGetDebugID +nvnTextureBuilderSetDevice +nvnTextureBuilderSetDefaults +nvnTextureBuilderSetFlags +nvnTextureBuilderSetTarget +nvnTextureBuilderSetWidth +nvnTextureBuilderSetHeight +nvnTextureBuilderSetDepth +nvnTextureBuilderSetSize1D +nvnTextureBuilderSetSize2D +nvnTextureBuilderSetSize3D +nvnTextureBuilderSetLevels +nvnTextureBuilderSetFormat +nvnTextureBuilderSetSamples +nvnTextureBuilderSetSwizzle +nvnTextureBuilderSetDepthStencilMode +nvnTextureBuilderGetStorageSize +nvnTextureBuilderGetStorageAlignment +nvnTextureBuilderSetStorage +nvnTextureBuilderSetPackagedTextureData +nvnTextureBuilderSetPackagedTextureLayout +nvnTextureBuilderSetStride +nvnTextureBuilderSetGLTextureName +nvnTextureBuilderGetStorageClass +nvnTextureBuilderGetDevice +nvnTextureBuilderGetFlags +nvnTextureBuilderGetTarget +nvnTextureBuilderGetWidth +nvnTextureBuilderGetHeight +nvnTextureBuilderGetDepth +nvnTextureBuilderGetLevels +nvnTextureBuilderGetFormat +nvnTextureBuilderGetSamples +nvnTextureBuilderGetSwizzle +nvnTextureBuilderGetDepthStencilMode +nvnTextureBuilderGetPackagedTextureData +nvnTextureBuilderGetPackagedTextureLayout +nvnTextureBuilderGetStride +nvnTextureBuilderGetSparseTileLayout +nvnTextureBuilderGetGLTextureName +nvnTextureBuilderGetZCullStorageSize +nvnTextureBuilderGetMemoryPool +nvnTextureBuilderGetMemoryOffset +nvnTextureBuilderGetRawStorageClass +nvnTextureViewSetDefaults +nvnTextureViewSetLevels +nvnTextureViewSetLayers +nvnTextureViewSetFormat +nvnTextureViewSetSwizzle +nvnTextureViewSetDepthStencilMode +nvnTextureViewSetTarget +nvnTextureViewGetLevels +nvnTextureViewGetLayers +nvnTextureViewGetFormat +nvnTextureViewGetSwizzle +nvnTextureViewGetDepthStencilMode +nvnTextureViewGetTarget +nvnTextureViewCompare +nvnTextureInitialize +nvnTextureGetZCullStorageSize +nvnTextureFinalize +nvnTextureSetDebugLabel +nvnTextureGetStorageClass +nvnTextureGetViewOffset +nvnTextureGetFlags +nvnTextureGetTarget +nvnTextureGetWidth +nvnTextureGetHeight +nvnTextureGetDepth +nvnTextureGetLevels +nvnTextureGetFormat +nvnTextureGetSamples +nvnTextureGetSwizzle +nvnTextureGetDepthStencilMode +nvnTextureGetStride +nvnTextureGetTextureAddress +nvnTextureGetSparseTileLayout +nvnTextureWriteTexels +nvnTextureWriteTexelsStrided +nvnTextureReadTexels +nvnTextureReadTexelsStrided +nvnTextureFlushTexels +nvnTextureInvalidateTexels +nvnTextureGetMemoryPool +nvnTextureGetMemoryOffset +nvnTextureGetStorageSize +nvnTextureCompare +nvnTextureGetDebugID +nvnTextureGetRawStorageClass +nvnSamplerBuilderSetDevice +nvnSamplerBuilderSetDefaults +nvnSamplerBuilderSetMinMagFilter +nvnSamplerBuilderSetWrapMode +nvnSamplerBuilderSetLodClamp +nvnSamplerBuilderSetLodBias +nvnSamplerBuilderSetCompare +nvnSamplerBuilderSetBorderColor +nvnSamplerBuilderSetBorderColori +nvnSamplerBuilderSetBorderColorui +nvnSamplerBuilderSetMaxAnisotropy +nvnSamplerBuilderSetReductionFilter +nvnSamplerBuilderSetLodSnap +nvnSamplerBuilderGetDevice +nvnSamplerBuilderGetMinMagFilter +nvnSamplerBuilderGetWrapMode +nvnSamplerBuilderGetLodClamp +nvnSamplerBuilderGetLodBias +nvnSamplerBuilderGetCompare +nvnSamplerBuilderGetBorderColor +nvnSamplerBuilderGetBorderColori +nvnSamplerBuilderGetBorderColorui +nvnSamplerBuilderGetMaxAnisotropy +nvnSamplerBuilderGetReductionFilter +nvnSamplerBuilderGetLodSnap +nvnSamplerInitialize +nvnSamplerFinalize +nvnSamplerSetDebugLabel +nvnSamplerGetMinMagFilter +nvnSamplerGetWrapMode +nvnSamplerGetLodClamp +nvnSamplerGetLodBias +nvnSamplerGetCompare +nvnSamplerGetBorderColor +nvnSamplerGetBorderColori +nvnSamplerGetBorderColorui +nvnSamplerGetMaxAnisotropy +nvnSamplerGetReductionFilter +nvnSamplerCompare +nvnSamplerGetDebugID +nvnBlendStateSetDefaults +nvnBlendStateSetBlendTarget +nvnBlendStateSetBlendFunc +nvnBlendStateSetBlendEquation +nvnBlendStateSetAdvancedMode +nvnBlendStateSetAdvancedOverlap +nvnBlendStateSetAdvancedPremultipliedSrc +nvnBlendStateSetAdvancedNormalizedDst +nvnBlendStateGetBlendTarget +nvnBlendStateGetBlendFunc +nvnBlendStateGetBlendEquation +nvnBlendStateGetAdvancedMode +nvnBlendStateGetAdvancedOverlap +nvnBlendStateGetAdvancedPremultipliedSrc +nvnBlendStateGetAdvancedNormalizedDst +nvnColorStateSetDefaults +nvnColorStateSetBlendEnable +nvnColorStateSetLogicOp +nvnColorStateSetAlphaTest +nvnColorStateGetBlendEnable +nvnColorStateGetLogicOp +nvnColorStateGetAlphaTest +nvnChannelMaskStateSetDefaults +nvnChannelMaskStateSetChannelMask +nvnChannelMaskStateGetChannelMask +nvnMultisampleStateSetDefaults +nvnMultisampleStateSetMultisampleEnable +nvnMultisampleStateSetSamples +nvnMultisampleStateSetAlphaToCoverageEnable +nvnMultisampleStateSetAlphaToCoverageDither +nvnMultisampleStateGetMultisampleEnable +nvnMultisampleStateGetSamples +nvnMultisampleStateGetAlphaToCoverageEnable +nvnMultisampleStateGetAlphaToCoverageDither +nvnMultisampleStateSetRasterSamples +nvnMultisampleStateGetRasterSamples +nvnMultisampleStateSetCoverageModulationMode +nvnMultisampleStateGetCoverageModulationMode +nvnMultisampleStateSetCoverageToColorEnable +nvnMultisampleStateGetCoverageToColorEnable +nvnMultisampleStateSetCoverageToColorOutput +nvnMultisampleStateGetCoverageToColorOutput +nvnMultisampleStateSetSampleLocationsEnable +nvnMultisampleStateGetSampleLocationsEnable +nvnMultisampleStateGetSampleLocationsGrid +nvnMultisampleStateSetSampleLocationsGridEnable +nvnMultisampleStateGetSampleLocationsGridEnable +nvnMultisampleStateSetSampleLocations +nvnPolygonStateSetDefaults +nvnPolygonStateSetCullFace +nvnPolygonStateSetFrontFace +nvnPolygonStateSetPolygonMode +nvnPolygonStateSetPolygonOffsetEnables +nvnPolygonStateGetCullFace +nvnPolygonStateGetFrontFace +nvnPolygonStateGetPolygonMode +nvnPolygonStateGetPolygonOffsetEnables +nvnDepthStencilStateSetDefaults +nvnDepthStencilStateSetDepthTestEnable +nvnDepthStencilStateSetDepthWriteEnable +nvnDepthStencilStateSetDepthFunc +nvnDepthStencilStateSetStencilTestEnable +nvnDepthStencilStateSetStencilFunc +nvnDepthStencilStateSetStencilOp +nvnDepthStencilStateGetDepthTestEnable +nvnDepthStencilStateGetDepthWriteEnable +nvnDepthStencilStateGetDepthFunc +nvnDepthStencilStateGetStencilTestEnable +nvnDepthStencilStateGetStencilFunc +nvnDepthStencilStateGetStencilOp +nvnVertexAttribStateSetDefaults +nvnVertexAttribStateSetFormat +nvnVertexAttribStateSetStreamIndex +nvnVertexAttribStateGetFormat +nvnVertexAttribStateGetStreamIndex +nvnVertexStreamStateSetDefaults +nvnVertexStreamStateSetStride +nvnVertexStreamStateSetDivisor +nvnVertexStreamStateGetStride +nvnVertexStreamStateGetDivisor +nvnCommandBufferInitialize +nvnCommandBufferFinalize +nvnCommandBufferSetDebugLabel +nvnCommandBufferSetMemoryCallback +nvnCommandBufferSetMemoryCallbackData +nvnCommandBufferSetCommandMemoryCallbackEnabled +nvnCommandBufferAddCommandMemory +nvnCommandBufferAddControlMemory +nvnCommandBufferGetCommandMemorySize +nvnCommandBufferGetCommandMemoryUsed +nvnCommandBufferGetCommandMemoryFree +nvnCommandBufferGetControlMemorySize +nvnCommandBufferGetControlMemoryUsed +nvnCommandBufferGetControlMemoryFree +nvnCommandBufferBeginRecording +nvnCommandBufferEndRecording +nvnCommandBufferCallCommands +nvnCommandBufferCopyCommands +nvnCommandBufferBindBlendState +nvnCommandBufferBindChannelMaskState +nvnCommandBufferBindColorState +nvnCommandBufferBindMultisampleState +nvnCommandBufferBindPolygonState +nvnCommandBufferBindDepthStencilState +nvnCommandBufferBindVertexAttribState +nvnCommandBufferBindVertexStreamState +nvnCommandBufferBindProgram +nvnCommandBufferBindVertexBuffer +nvnCommandBufferBindVertexBuffers +nvnCommandBufferBindUniformBuffer +nvnCommandBufferBindUniformBuffers +nvnCommandBufferBindTransformFeedbackBuffer +nvnCommandBufferBindTransformFeedbackBuffers +nvnCommandBufferBindStorageBuffer +nvnCommandBufferBindStorageBuffers +nvnCommandBufferBindTexture +nvnCommandBufferBindTextures +nvnCommandBufferBindImage +nvnCommandBufferBindImages +nvnCommandBufferSetPatchSize +nvnCommandBufferSetInnerTessellationLevels +nvnCommandBufferSetOuterTessellationLevels +nvnCommandBufferSetPrimitiveRestart +nvnCommandBufferBeginTransformFeedback +nvnCommandBufferEndTransformFeedback +nvnCommandBufferPauseTransformFeedback +nvnCommandBufferResumeTransformFeedback +nvnCommandBufferDrawTransformFeedback +nvnCommandBufferDrawArrays +nvnCommandBufferDrawElements +nvnCommandBufferDrawElementsBaseVertex +nvnCommandBufferDrawArraysInstanced +nvnCommandBufferDrawElementsInstanced +nvnCommandBufferDrawArraysIndirect +nvnCommandBufferDrawElementsIndirect +nvnCommandBufferMultiDrawArraysIndirectCount +nvnCommandBufferMultiDrawElementsIndirectCount +nvnCommandBufferClearColor +nvnCommandBufferClearColori +nvnCommandBufferClearColorui +nvnCommandBufferClearDepthStencil +nvnCommandBufferDispatchCompute +nvnCommandBufferDispatchComputeIndirect +nvnCommandBufferSetViewport +nvnCommandBufferSetViewports +nvnCommandBufferSetViewportSwizzles +nvnCommandBufferSetScissor +nvnCommandBufferSetScissors +nvnCommandBufferSetDepthRange +nvnCommandBufferSetDepthBounds +nvnCommandBufferSetDepthRanges +nvnCommandBufferSetTiledCacheAction +nvnCommandBufferSetTiledCacheTileSize +nvnCommandBufferBindSeparateTexture +nvnCommandBufferBindSeparateSampler +nvnCommandBufferBindSeparateTextures +nvnCommandBufferBindSeparateSamplers +nvnCommandBufferSetStencilValueMask +nvnCommandBufferSetStencilMask +nvnCommandBufferSetStencilRef +nvnCommandBufferSetBlendColor +nvnCommandBufferSetPointSize +nvnCommandBufferSetLineWidth +nvnCommandBufferSetPolygonOffsetClamp +nvnCommandBufferSetAlphaRef +nvnCommandBufferSetSampleMask +nvnCommandBufferSetRasterizerDiscard +nvnCommandBufferSetDepthClamp +nvnCommandBufferSetConservativeRasterEnable +nvnCommandBufferSetConservativeRasterDilate +nvnCommandBufferSetSubpixelPrecisionBias +nvnCommandBufferCopyBufferToTexture +nvnCommandBufferCopyTextureToBuffer +nvnCommandBufferCopyTextureToTexture +nvnCommandBufferCopyBufferToBuffer +nvnCommandBufferClearBuffer +nvnCommandBufferClearTexture +nvnCommandBufferClearTexturei +nvnCommandBufferClearTextureui +nvnCommandBufferUpdateUniformBuffer +nvnCommandBufferReportCounter +nvnCommandBufferResetCounter +nvnCommandBufferReportValue +nvnCommandBufferSetRenderEnable +nvnCommandBufferSetRenderEnableConditional +nvnCommandBufferSetRenderTargets +nvnCommandBufferDiscardColor +nvnCommandBufferDiscardDepthStencil +nvnCommandBufferDownsample +nvnCommandBufferTiledDownsample +nvnCommandBufferDownsampleTextureView +nvnCommandBufferTiledDownsampleTextureView +nvnCommandBufferBarrier +nvnCommandBufferWaitSync +nvnCommandBufferFenceSync +nvnCommandBufferSetTexturePool +nvnCommandBufferSetSamplerPool +nvnCommandBufferSetShaderScratchMemory +nvnCommandBufferSaveZCullData +nvnCommandBufferRestoreZCullData +nvnCommandBufferSetCopyRowStride +nvnCommandBufferSetCopyImageStride +nvnCommandBufferGetCopyRowStride +nvnCommandBufferGetCopyImageStride +nvnCommandBufferDrawTexture +nvnProgramSetSubroutineLinkage +nvnCommandBufferSetProgramSubroutines +nvnCommandBufferBindCoverageModulationTable +nvnCommandBufferResolveDepthBuffer +nvnCommandBufferSetColorReductionEnable +nvnCommandBufferSetColorReductionThresholds +nvnCommandBufferPushDebugGroupStatic +nvnCommandBufferPushDebugGroupDynamic +nvnCommandBufferPushDebugGroup +nvnCommandBufferPopDebugGroup +nvnCommandBufferPopDebugGroupId +nvnCommandBufferInsertDebugMarkerStatic +nvnCommandBufferInsertDebugMarkerDynamic +nvnCommandBufferInsertDebugMarker +nvnCommandBufferGetMemoryCallback +nvnCommandBufferGetMemoryCallbackData +nvnCommandBufferIsRecording +nvnSyncInitialize +nvnSyncFinalize +nvnSyncSetDebugLabel +nvnQueueFenceSync +nvnSyncWait +nvnQueueWaitSync +nvnSyncInitializeFromFencedGLSync +nvnSyncCreateGLSync +nvnEventBuilderSetDefaults +nvnEventBuilderSetStorage +nvnEventBuilderGetStorage +nvnEventBuilderGetMemoryPool +nvnEventBuilderGetMemoryOffset +nvnEventInitialize +nvnEventFinalize +nvnEventGetValue +nvnEventSignal +nvnEventGetMemoryPool +nvnEventGetMemoryOffset +nvnCommandBufferWaitEvent +nvnCommandBufferSignalEvent +nvnCommandBufferSetStencilCullCriteria +nvnCommandBufferInsertCheckpointStatic +nvnCommandBufferInsertCheckpointDynamic +nvnDeviceSetAutomaticCheckpointsEnable +nvnDeviceSetAutomaticCheckpointsSamplingInterval +nvnDeviceSetAutomaticCheckpointsByDebugGroup +nvnDeviceSetCheckpointCallStacksEnable +nvnDeviceSetCheckpointMaxCallStacksDepth diff --git a/src/gfx/nvn/nvn_count.h b/src/gfx/nvn/nvn_count.h new file mode 100644 index 0000000..81c1dd3 --- /dev/null +++ b/src/gfx/nvn/nvn_count.h @@ -0,0 +1,4 @@ +// 由 gen_protos.py 生成 +// Generated by gen_protos.py +#pragma once +#define NVN_NFUNC 523 diff --git a/src/gfx/nvn/nvn_diag_range.h b/src/gfx/nvn/nvn_diag_range.h new file mode 100644 index 0000000..d3c0611 --- /dev/null +++ b/src/gfx/nvn/nvn_diag_range.h @@ -0,0 +1,33 @@ +#ifndef NVN_DIAG_RANGE_H +#define NVN_DIAG_RANGE_H +#include +#include +#include + +typedef struct { long lo, hi; } nvn_diag_range_value; +typedef struct { _Atomic int ready; nvn_diag_range_value value; } nvn_diag_range_cache; + +static inline nvn_diag_range_value nvn_diag_range_parse(const char *text) { + nvn_diag_range_value v = {-2, -2}; + if (!text || sscanf(text, "%ld-%ld", &v.lo, &v.hi) != 2) v.lo = v.hi = -2; + return v; +} + +/* The immutable environment range is published once to all recording threads. + Legacy mode intentionally repeats the disabled-range lookup for in-process A/B. + The old lo < 0 initialization test also matched its initialized -2 sentinel. */ +static inline nvn_diag_range_value nvn_diag_range_get(nvn_diag_range_cache *c, + int cache_disabled, const char *(*read_env)(void)) { + if (atomic_load_explicit(&c->ready, memory_order_acquire) != 2) { + int initial = 0; + if (atomic_compare_exchange_strong_explicit(&c->ready, &initial, 1, + memory_order_acq_rel, memory_order_acquire)) { + c->value = nvn_diag_range_parse(read_env()); + atomic_store_explicit(&c->ready, 2, memory_order_release); + } else { + while (atomic_load_explicit(&c->ready, memory_order_acquire) != 2) sched_yield(); + } + } + return !cache_disabled && c->value.lo < 0 ? nvn_diag_range_parse(read_env()) : c->value; +} +#endif diff --git a/src/gfx/nvn/nvn_dirty.h b/src/gfx/nvn/nvn_dirty.h new file mode 100644 index 0000000..a5a56f4 --- /dev/null +++ b/src/gfx/nvn/nvn_dirty.h @@ -0,0 +1,23 @@ +// 客体内存写保护脏页跟踪（nvn_dirty.c） +// Write-protect dirty-page tracking for guest memory (nvn_dirty.c) +#pragma once +#include +#include + +void nvn_dirty_add_range(const void *mem, size_t size); // 登记一个内存池为跟踪区间（首次调用时注册缺页钩子与宿主写前钩子） | Register a memory pool as a tracked range (the first call registers the page-fault hook and the host pre-write hook) +void nvn_dirty_stat(int *live, int *peak, int *cap); // 在册的内存池数、峰值、跟踪表容量（内存读数行用） | number of registered memory pools, peak, tracking table capacity (for the memory readout line) +void nvn_dirty_remove_range(const void *mem); // 池销毁：受保护页还原可写并标失效 | Pool destruction: restore protected pages to writable and mark invalid +int nvn_dirty_tracked(const void *addr, size_t len); +void nvn_dirty_protect(const void *addr, size_t len); // 覆盖页设只读 | Set the covered pages read-only +uint64_t nvn_dirty_now(void); // 当前代号：上传前取，作为该份内容的戳 | Current generation: taken before upload, as the stamp of that content +int nvn_dirty_since(const void *addr, size_t len, uint64_t stamp); +// Runs of pages written since stamp (merged when closer than gap bytes): fn(ctx, offset, bytes); -1 = untracked +void nvn_dirty_epoch_begin(void); // [Windows port] a new submission: one write-watch scan per range on this thread until the next +int nvn_dirty_runs(const void *addr, size_t len, uint64_t stamp, size_t gap, void (*fn)(void *ctx, size_t off, size_t n), void *ctx); +void *nvn_dirty_gpu_write_begin(void *addr, size_t len); // write guest memory on behalf of the GPU: not counted as a CPU write +void nvn_dirty_gpu_write_end(void *tok); +typedef struct { uintptr_t page; uint64_t pc, lr; uint32_t src; } nvn_dirty_rec; // src 0: guest write (fault), 1: host write +int nvn_dirty_recent(uintptr_t lo, uintptr_t hi, nvn_dirty_rec *out, int max); // latest unprotections in [lo, hi) (DM_NVN_TILELOG) // 覆盖页在戳之后被写过 → 1 | Covered pages were written after the stamp -> 1 +void nvn_dirty_stats(uint64_t *faults, uint64_t *preps); // 累计：游戏写触发的缺页次数、宿主写前解保护的页数 | Cumulative: number of page faults triggered by game writes, number of pages unprotected before host writes +void nvn_dirty_mp_stats(uint64_t *calls, uint64_t *pages, uint64_t *ns); // 累计：本模块 mprotect 调用次数、覆盖页数、耗时 | Cumulative: this module's mprotect call count, pages covered, time spent +void nvn_dirty_slot_stats(int *live, int *used_slots, uint64_t *adds_total); // 当前存活槽、槽表高水位、登记总次数 | Currently live slots, slot table high-water mark, total number of registrations diff --git a/src/gfx/nvn/nvn_dirty_win.c b/src/gfx/nvn/nvn_dirty_win.c new file mode 100644 index 0000000..5b7fea2 --- /dev/null +++ b/src/gfx/nvn/nvn_dirty_win.c @@ -0,0 +1,184 @@ +// [Windows port] Dirty-page tracking of guest memory pools (the API of MacNativeGames' nvn_dirty.h) through the guest +// window's write watch (MEM_WRITE_WATCH, src/runtime/platform/windows.c plat_written_pages) instead of write-protecting +// pages and catching the faults. The OS marks every page anything writes (the game, host code, file reads by the kernel), +// so there are no protection changes, no fault handler and no host pre-write hook. +// As in MacNativeGames, each page of a tracked pool has a generation: the generation of its last observed write. +// nvn_dirty_since() collects the pages written since the last look (resetting their watch bits) and gives them a new +// generation, then compares the range's generations with the caller's stamp. +#include +#include +#include + +#include "../host/dm.h" +#include "nvn_dirty.h" + +int plat_written_pages(uint64_t addr, uint64_t size, uint64_t *pages, int max, uint64_t *page); + +#define MAX_RANGE 16384 // MacNativeGames: about 3,900 live pools in TOTK's world +typedef struct { uintptr_t base, end; _Atomic uint64_t *gen; _Atomic int live, watched; _Atomic uint64_t epoch; } range_t; // watched: the OS reports its writes +static range_t g_r[MAX_RANGE]; +static _Atomic int g_nr, g_live, g_live_peak; +static _Atomic uint64_t g_now = 1, g_adds_total, g_scans, g_scan_pages; +static pthread_mutex_t g_slot_mu = PTHREAD_MUTEX_INITIALIZER; +static int g_watch = -1; // write watch available (the window was reserved with it) + +static _Thread_local int t_last; // the range this thread found last (most lookups repeat it) +static range_t *find(uintptr_t a) { + int n = atomic_load(&g_nr), k = t_last; + if (k < n && atomic_load_explicit(&g_r[k].live, memory_order_acquire) && a >= g_r[k].base && a < g_r[k].end) return &g_r[k]; + for (int i = 0; i < n; i++) + if (atomic_load_explicit(&g_r[i].live, memory_order_acquire) && a >= g_r[i].base && a < g_r[i].end) { t_last = i; return &g_r[i]; } + return NULL; +} + +// Scans once per epoch (nvn_dirty_epoch_begin, a GPU submission being recorded) on the thread that began it: the first +// look at a range scans all of it, later looks in the epoch use the generations gathered then. The game writes what a +// submission uses before submitting it, so one look per submission sees those writes; per-texture and per-pool scans +// were hundreds of GetWriteWatch calls a submission (~9% of the recording thread, Great Sky Island, 2026-10-24). +// Other threads (the retire thread's copy back of GPU writes) scan every time. +static _Atomic uint64_t g_epoch; +static _Thread_local int t_epoch_thread; +void nvn_dirty_epoch_begin(void) { t_epoch_thread = 1; atomic_fetch_add(&g_epoch, 1); } + +// Pages of [a, a+len) written since the last scan get a new generation; their watch bits are reset. +static void scan_pages(range_t *r, uintptr_t a, size_t len, int count); +static void scan(range_t *r, uintptr_t a, size_t len, int count) { + if (count && t_epoch_thread) { + uint64_t e = atomic_load(&g_epoch); + if (atomic_load(&r->epoch) == e) return; + atomic_store(&r->epoch, e); + a = r->base; len = r->end - r->base; + } + scan_pages(r, a, len, count); +} +static void scan_pages(range_t *r, uintptr_t a, size_t len, int count) { + uint64_t pages[512], pg = 4096; + uintptr_t lo = a & ~(uintptr_t)0xfff, hi = (a + len + 0xfff) & ~(uintptr_t)0xfff; + if (lo < r->base) lo = r->base; + if (hi > r->end) hi = r->end; + while (lo < hi) { + int n = plat_written_pages(lo, hi - lo, pages, 512, &pg); + if (n < 0) { // this range only (it used to switch tracking off for every pool, silently) + if (atomic_exchange(&r->watched, 0)) { + static _Atomic int said; + if (atomic_fetch_add(&said, 1) < 8) LOG("dirty tracking: no write watch for pool %p..%p: treated as unchanged", (void *)r->base, (void *)r->end); + } + return; + } + atomic_fetch_add(&g_scans, 1); + uint64_t g = n && count ? atomic_fetch_add(&g_now, 1) + 1 : 0; + for (int i = 0; i < n; i++) { + uintptr_t p = (pages[i] - r->base) / 4096; + if (count) atomic_store(&r->gen[p], g); + } + atomic_fetch_add(&g_scan_pages, (uint64_t)n); + if (n < 512) break; // all of the range reported; with 512 there may be more: scan again (bits were reset) + } +} + +void nvn_dirty_add_range(const void *mem, size_t size) { + if (g_watch < 0) { // probe once on a page inside the guest window (the main module's base) + uint64_t p, pg; + g_watch = plat_written_pages(0x0800000000ull, 4096, &p, 1, &pg) >= 0; + if (!g_watch) LOG("dirty tracking: the guest window has no write watch; every use re-uploads"); + } + uintptr_t base = (uintptr_t)mem & ~(uintptr_t)0xfff, end = ((uintptr_t)mem + size + 0xfff) & ~(uintptr_t)0xfff; + size_t npages = (end - base) / 4096; + _Atomic uint64_t *gen = calloc(npages ? npages : 1, sizeof *gen); + if (!gen) die("dirty tracking: allocation"); + pthread_mutex_lock(&g_slot_mu); + int slot = -1, n = atomic_load(&g_nr); + for (int i = 0; i < n; i++) if (!atomic_load(&g_r[i].live)) { slot = i; break; } + if (slot < 0) { + if (n == MAX_RANGE) { pthread_mutex_unlock(&g_slot_mu); free(gen); LOG("dirty tracking: more than %d pools", MAX_RANGE); return; } + slot = n; atomic_store(&g_nr, n + 1); + } + range_t *r = &g_r[slot]; + free(r->gen); + r->base = base; r->end = end; r->gen = gen; + atomic_store(&r->watched, 1); + atomic_store(&r->epoch, 0); + atomic_store_explicit(&r->live, 1, memory_order_release); + pthread_mutex_unlock(&g_slot_mu); + scan(r, base, end - base, 0); // forget writes from before the pool existed + atomic_fetch_add(&g_adds_total, 1); + int live = atomic_fetch_add(&g_live, 1) + 1, peak = atomic_load(&g_live_peak); + while (live > peak && !atomic_compare_exchange_weak(&g_live_peak, &peak, live)) {} +} + +void nvn_dirty_remove_range(const void *mem) { + pthread_mutex_lock(&g_slot_mu); + range_t *r = find((uintptr_t)mem & ~(uintptr_t)0xfff); + if (r) { atomic_store(&r->live, 0); atomic_fetch_sub(&g_live, 1); } + pthread_mutex_unlock(&g_slot_mu); +} + +int nvn_dirty_tracked(const void *addr, size_t len) { + range_t *r = find((uintptr_t)addr); + return r && g_watch > 0 && atomic_load(&r->watched) && (uintptr_t)addr + len <= r->end; +} + +void nvn_dirty_protect(const void *addr, size_t len) { + range_t *r = find((uintptr_t)addr); + if (r) scan(r, (uintptr_t)addr, len, 1); // writes up to now belong to the content being uploaded +} + +uint64_t nvn_dirty_now(void) { return atomic_load(&g_now); } + +int nvn_dirty_since(const void *addr, size_t len, uint64_t stamp) { + range_t *r = find((uintptr_t)addr); + if (!r || g_watch <= 0 || !atomic_load(&r->watched)) return 1; // untracked: always treat as changed + scan(r, (uintptr_t)addr, len, 1); + uintptr_t lo = ((uintptr_t)addr - r->base) / 4096, hi = ((uintptr_t)addr + len - r->base + 4095) / 4096; + for (uintptr_t p = lo; p < hi; p++) if (atomic_load(&r->gen[p]) > stamp) return 1; + return 0; +} + +// Runs of pages of [addr, addr+len) written since `stamp`: fn(ctx, offset from addr, bytes), in order; runs less than +// `gap` bytes apart are merged (fewer, larger copies). -1: not tracked (its writes cannot be told). +int nvn_dirty_runs(const void *addr, size_t len, uint64_t stamp, size_t gap, void (*fn)(void *ctx, size_t off, size_t n), void *ctx) { + range_t *r = find((uintptr_t)addr); + if (!r || g_watch <= 0 || !atomic_load(&r->watched) || (uintptr_t)addr + len > r->end) return -1; + scan(r, (uintptr_t)addr, len, 1); + uintptr_t a = (uintptr_t)addr, first = (a - r->base) / 4096, last = (a + len - r->base + 4095) / 4096; + uintptr_t run_lo = 0, run_hi = 0; // pages, relative to the range base; run_hi 0: none open + for (uintptr_t p = first; p < last; p++) { + if (atomic_load(&r->gen[p]) <= stamp) continue; + if (run_hi && (p - run_hi) * 4096 <= gap) { run_hi = p + 1; continue; } + if (run_hi) { + uintptr_t lo = r->base + run_lo * 4096 < a ? a : r->base + run_lo * 4096, hi = r->base + run_hi * 4096 > a + len ? a + len : r->base + run_hi * 4096; + fn(ctx, lo - a, hi - lo); + } + run_lo = p; run_hi = p + 1; + } + if (run_hi) { + uintptr_t lo = r->base + run_lo * 4096 < a ? a : r->base + run_lo * 4096, hi = r->base + run_hi * 4096 > a + len ? a + len : r->base + run_hi * 4096; + fn(ctx, lo - a, hi - lo); + } + return 0; +} + +// The GPU writes guest memory (readbacks): those writes are not the CPU's. The pages are scanned without a new +// generation afterwards (a CPU write racing with it would be lost: accepted, as the readback overwrites them anyway). +typedef struct { void *addr; size_t len; } gpu_tok; +void *nvn_dirty_gpu_write_begin(void *addr, size_t len) { + gpu_tok *t = malloc(sizeof *t); + if (t) *t = (gpu_tok){addr, len}; + return t; +} +void nvn_dirty_gpu_write_end(void *tok) { + gpu_tok *t = tok; + if (!t) return; + range_t *r = find((uintptr_t)t->addr); + if (r) scan(r, (uintptr_t)t->addr, t->len, 0); + free(t); +} + +int nvn_dirty_recent(uintptr_t lo, uintptr_t hi, nvn_dirty_rec *out, int max) { (void)lo; (void)hi; (void)out; (void)max; return 0; } +void nvn_dirty_stat(int *live, int *peak, int *cap) { *live = atomic_load(&g_live); *peak = atomic_load(&g_live_peak); *cap = MAX_RANGE; } +void nvn_dirty_stats(uint64_t *faults, uint64_t *preps) { *faults = atomic_load(&g_scan_pages); *preps = 0; } +void nvn_dirty_mp_stats(uint64_t *calls, uint64_t *pages, uint64_t *ns) { *calls = atomic_load(&g_scans); *pages = atomic_load(&g_scan_pages); *ns = 0; } +void nvn_dirty_slot_stats(int *live, int *used_slots, uint64_t *adds_total) { + *live = atomic_load(&g_live); *used_slots = atomic_load(&g_nr); *adds_total = atomic_load(&g_adds_total); +} +void dm_guest_write_prepare(void *addr, size_t n) { (void)addr; (void)n; } // the OS tracks host writes as well diff --git a/src/gfx/nvn/nvn_hmap.h b/src/gfx/nvn/nvn_hmap.h new file mode 100644 index 0000000..ab83d53 --- /dev/null +++ b/src/gfx/nvn/nvn_hmap.h @@ -0,0 +1,76 @@ +// 定长字节键 → 指针 的开放寻址哈希表（线性探测 + 删除时回移，无墓碑）。 +// 回放热路径用它替代 NSDictionary：键不装箱、不分配，查找只做一次哈希 + memcmp。 +// 不加锁：调用方按各自的锁保护。值的所有权归调用方（put / del 返回被替换 / 删除的旧值）。单测 tests/t_hmap.c。 +// Open-addressing hash table from fixed-length byte keys -> pointers (linear probing + backward shift on deletion, no tombstones). +// The replay hot path uses it instead of NSDictionary: keys are not boxed, no allocation, a lookup does only one hash + memcmp. +// No locking: callers protect it with their own locks. Ownership of values belongs to the caller (put / del return the replaced / deleted old value). Unit test tests/t_hmap.c. +#pragma once +#include +#include +#include + +typedef struct { uint8_t *keys; void **vals; uint32_t klen, cap, n; } nvn_hmap; // vals[i] == NULL 即空位 | vals[i] == NULL means an empty slot + +static inline uint64_t nvn_hm_hash(const void *k, uint32_t len) { + const uint8_t *p = k; uint64_t h = 0x9e3779b97f4a7c15ull ^ len; + for (; len >= 8; p += 8, len -= 8) { uint64_t w; memcpy(&w, p, 8); h = (h ^ w) * 0xff51afd7ed558ccdull; h ^= h >> 32; } + if (len) { uint64_t w = 0; memcpy(&w, p, len); h = (h ^ w) * 0xff51afd7ed558ccdull; } + h ^= h >> 33; h *= 0xc4ceb9fe1a85ec53ull; h ^= h >> 33; + return h; +} +static inline void nvn_hm_init(nvn_hmap *m, uint32_t klen) { memset(m, 0, sizeof *m); m->klen = klen; } +static inline void *nvn_hm_get(const nvn_hmap *m, const void *k) { + if (!m->n) return NULL; + for (uint32_t i = (uint32_t)nvn_hm_hash(k, m->klen) & (m->cap - 1);; i = (i + 1) & (m->cap - 1)) { + if (!m->vals[i]) return NULL; + if (!memcmp(m->keys + (size_t)i * m->klen, k, m->klen)) return m->vals[i]; + } +} +static inline void *nvn_hm_put(nvn_hmap *m, const void *k, void *v); +static inline void nvn_hm_grow(nvn_hmap *m) { + nvn_hmap o = *m; + m->cap = o.cap ? o.cap * 2 : 64; m->n = 0; + m->keys = malloc((size_t)m->cap * m->klen); m->vals = calloc(m->cap, sizeof *m->vals); + if (!m->keys || !m->vals) abort(); + for (uint32_t i = 0; i < o.cap; i++) if (o.vals[i]) nvn_hm_put(m, o.keys + (size_t)i * o.klen, o.vals[i]); + free(o.keys); free(o.vals); +} +// v 不得为 NULL；返回同键的旧值（无则 NULL） +// v must not be NULL; returns the old value for the same key (NULL if none) +static inline void *nvn_hm_put(nvn_hmap *m, const void *k, void *v) { + if ((m->n + 1) * 4 > m->cap * 3) nvn_hm_grow(m); // 装载率 ≤ 3/4 | Load factor <= 3/4 + for (uint32_t i = (uint32_t)nvn_hm_hash(k, m->klen) & (m->cap - 1);; i = (i + 1) & (m->cap - 1)) { + uint8_t *ki = m->keys + (size_t)i * m->klen; + if (!m->vals[i]) { memcpy(ki, k, m->klen); m->vals[i] = v; m->n++; return NULL; } + if (!memcmp(ki, k, m->klen)) { void *old = m->vals[i]; m->vals[i] = v; return old; } + } +} +// 删除后把同一探测链上后面的项回移补洞，保证查找遇空位即可停 +// After deletion, shift back later entries on the same probe chain to fill the hole, so that a lookup can stop at the first empty slot +static inline void *nvn_hm_del(nvn_hmap *m, const void *k) { + if (!m->n) return NULL; + uint32_t mask = m->cap - 1, i = (uint32_t)nvn_hm_hash(k, m->klen) & mask; + for (;; i = (i + 1) & mask) { + if (!m->vals[i]) return NULL; + if (!memcmp(m->keys + (size_t)i * m->klen, k, m->klen)) break; + } + void *old = m->vals[i]; + for (uint32_t j = i;;) { + m->vals[i] = NULL; + for (;;) { + j = (j + 1) & mask; + if (!m->vals[j]) { m->n--; return old; } + uint32_t h = (uint32_t)nvn_hm_hash(m->keys + (size_t)j * m->klen, m->klen) & mask; + if (i <= j ? (i < h && h <= j) : (i < h || h <= j)) continue; // 理想位在 (i, j] 内：不能前移 | Ideal position is within (i, j]: cannot move forward + break; + } + memcpy(m->keys + (size_t)i * m->klen, m->keys + (size_t)j * m->klen, m->klen); m->vals[i] = m->vals[j]; + i = j; + } +} +// 清空；rel 非空时对每个值调用一次（释放所有权） +// Clear; when rel is non-null, call it once for each value (releasing ownership) +static inline void nvn_hm_clear(nvn_hmap *m, void (*rel)(void *)) { + for (uint32_t i = 0; i < m->cap; i++) if (m->vals[i]) { if (rel) rel(m->vals[i]); m->vals[i] = NULL; } + m->n = 0; +} diff --git a/src/gfx/nvn/nvn_impl.h b/src/gfx/nvn/nvn_impl.h new file mode 100644 index 0000000..b5e665d --- /dev/null +++ b/src/gfx/nvn/nvn_impl.h @@ -0,0 +1,37 @@ +// NVN implementation registration macros shared by hle_nvn.c and TOTK extension files. +// [Windows port] Registered in PE sections .nvni$m / .nvnx$m between the markers of nvn_registry.c. +#pragma once +#include "nvn_protos.h" + +// TOTK: NVN entry points that wait for the GPU, the display or other threads give up the caller's virtual core for +// their duration (runtime/threads.c), as the Horizon thread would block there. +int runtime_core_away(void); +void runtime_core_back(int *away); +#define NVN_BLOCKING() __attribute__((cleanup(runtime_core_back))) int totk_core_away_ = runtime_core_away() + +typedef struct { const char *name; void *fn; } nvn_impl; +#define NVN(name, ret, ...) \ + static ret h_##name(__VA_ARGS__); \ + static const NVNT_##name chk_##name __attribute__((unused)) = h_##name; \ + __attribute__((used, section(".nvni$m"))) static const nvn_impl e_##name = {#name, (void *)h_##name}; \ + static ret h_##name(__VA_ARGS__) +// 无原型的新版接口：不做签名检查 +// Newer interfaces without prototypes: no signature check +#define NVN_NOPROTO(name, ret, ...) \ + static ret h_##name(__VA_ARGS__); \ + __attribute__((used, section(".nvni$m"))) static const nvn_impl e_##name = {#name, (void *)h_##name}; \ + static ret h_##name(__VA_ARGS__) + + +// TOTK: entry points the game asks nvnDeviceGetProcAddress / nvnBootstrapLoader for that are not in main's +// name table (no counting thunk). They are returned directly by nvn_lookup. +#define NVN_EXTRA(name, ret, ...) \ + static ret x_##name(__VA_ARGS__); \ + __attribute__((used, section(".nvnx$m"))) static const nvn_impl ex_##name = {#name, (void *)x_##name}; \ + static ret x_##name(__VA_ARGS__) + +// [Windows port] nvn_registry.c +const nvn_impl *nvn_registry_impls(unsigned *n); +const nvn_impl *nvn_registry_extras(unsigned *n); +void *nvn_impl_fn(const char *name, int extra); +uint64_t nvn_registry_guest_address(const char *name); diff --git a/src/gfx/nvn/nvn_mtl.h b/src/gfx/nvn/nvn_mtl.h new file mode 100644 index 0000000..a668c2f --- /dev/null +++ b/src/gfx/nvn/nvn_mtl.h @@ -0,0 +1,76 @@ +// NVN 空设备的 Metal 执行层（C 接口）：内存池 = 零拷贝 MTLBuffer，提交 = MTLCommandBuffer，同步 = MTLSharedEvent。 +// 时序约定（防 x86 路线 GO_MAP_SHARE 花屏的覆辙）：一切「GPU 完成」的可见效果（同步对象就绪、事件值、计数器） +// 只在 Metal 命令缓冲真正完成之后才发生；CPU 侧等待只看 MTLSharedEvent 的已完成值。 +// Metal execution layer of the NVN null device (C interface): memory pool = zero-copy MTLBuffer, submit = MTLCommandBuffer, sync = MTLSharedEvent. +// Ordering convention (to avoid repeating the GO_MAP_SHARE corrupted-picture mistake of the x86 route): every visible effect of "GPU completion" (sync object ready, event values, counters) +// happens only after the Metal command buffer has actually completed; CPU-side waits look only at the completed value of the MTLSharedEvent. +#pragma once +#include +#include + +void mtl_init(void); +void bf_want_note(const char *key); + +// 内存池：[mem, mem+size) 外扩到页边界建零拷贝缓冲；按 GPU 地址（= CPU 地址）反查缓冲与偏移 +// Memory pool: [mem, mem+size) is expanded outward to page boundaries to build a zero-copy buffer; look up the buffer and offset by GPU address (= CPU address) +void mtl_pool_add(const void *mem, uint64_t size); +void mtl_pool_del(const void *mem); +int mtl_addr(uint64_t addr, uint64_t len, void **buf, uint64_t *off); // 成功 1；buf 是 (__bridge) id | 1 on success; buf is a (__bridge) id + +typedef struct { + uint32_t pso, cps, shaders, dss; + uint32_t view_nonnull, view_slots, live_tex, sview_nonnull; + uint32_t pools; + uint64_t pool_adds; + uint32_t c1_bufs; uint64_t c1_alloc, c1_data; // 着色器 c1 常数段：缓冲个数（4 MiB 块 + 独占 64 KiB）、分配字节、实际字节 | Shader c1 constant segment: buffer count (4 MiB blocks + dedicated 64 KiB), allocated bytes, actual bytes +} mtl_res_count; +void mtl_res_counts(uint32_t tick, const char *why); // 打一行资源计数（宿主按拍 / 事件调用，见 dm.h dm_res_report_register） | Print one line of resource counts (the host calls it per tick / event, see dm_res_report_register in dm.h) +void mtl_res_snapshot(mtl_res_count *out); + +// 一次提交：begin → 若干操作 → commit（返回本次提交完成时的栅栏值） +// One submit: begin -> several operations -> commit (returns the fence value at which this submit completes) +void mtl_sub_begin(void); +/* TOTK: uniform contents as of the submit call. capture runs in nvnQueueSubmitCommands (game thread) when the replay is + handed to the translation thread; snapshot_cap builds the submission's copies at the start of its replay (cap may be + NULL: copy from memory now) and frees cap. */ +void *mtl_sub_capture_ubos(const uint64_t *h, int n); +void mtl_sub_snapshot_ubos_cap(const uint64_t *h, int n, void *cap); +void mtl_sub_copy(uint64_t src, uint64_t dst, uint64_t n); // GPU 时序上的缓冲拷贝 | Buffer copy in GPU ordering +void mtl_sub_event(volatile uint32_t *val, uint32_t mode, uint32_t v); // 完成后写事件值（0 写 / 1 加） | Write the event value after completion (0 write / 1 add) +void mtl_sub_counter(uint64_t addr, uint32_t type); // 完成后写计数器记录 | Write the counter record after completion +uint64_t mtl_sub_commit(void); +int mtl_probe_vertex_streams(void); +int mtl_probe_arg_epoch(void); +int mtl_probe_tex_slots(void); +int mtl_probe_pool_retire(void); +int mtl_probe_anomaly(void); +int mtl_probe_widen(void); // TOTK: standalone probes of isolated GPU paths (die on mismatch, return checked count) +int mtl_probe_blockrt(void); +int mtl_probe_midclear(void); +int mtl_probe_srgb_view(void); +int mtl_probe_storage_view(void); +int mtl_probe_compressed_alias(void); +int mtl_probe_linear_rt_upload(void); +int mtl_probe_texture_retire(void); +int mtl_probe_ds_clear(void); +int mtl_probe_ub_rename(void); +int mtl_probe_ubo_snapshot(void); +int mtl_probe_drv_c0(void); +int mtl_probe_dispatch_chain(void); +int mtl_probe_copy_alias(void); +int mtl_probe_layer_merge(void); +int mtl_probe_cube_merge(void); +void mtl_cube_merge_stats(uint64_t v[4]); +int mtl_probe_winding(void); +int mtl_probe_depth_output(void); +int mtl_probe_tile(void); + +uint64_t mtl_last_fence(void); // 最近一次提交的栅栏值（0 = 从未提交） | Fence value of the most recent submit (0 = never submitted) +int mtl_wait(uint64_t fence, uint64_t timeout_ns); // GPU 完成到 fence 返回 1，超时 0 | Returns 1 when the GPU has completed up to fence, 0 on timeout +void mtl_finish(void); // 等所有已提交的命令完成 | Wait for all submitted commands to complete +void mtl_drain(void); // 等队列上全部命令（含出图）完成：提交空命令缓冲并等它 | Wait for all commands on the queue (including presents) to complete: submit an empty command buffer and wait for it +void mtl_xv_window_tex(uint64_t serial); // 登记窗口（交换链）贴图：跨提交有效性账本永不收它 | Register a window (swapchain) texture: the cross-submit validity ledger never collects it +void mtl_ui_ghost(int on, float px, float py, const float grid[4], float cx, float cy); // 拖货跟手图：on=1 时把盖住屏幕点 (px,py)、压在货架格网 grid（左上角 x,y、一格宽高）上的那件货的图标在界面趟末尾平移到中心 (cx,cy) 再画一遍（归一化，左上原点）；任意线程调 | Drag-follow image for shop items: when on=1, the icon of the item that covers screen point (px,py) and sits on the shop shelf grid `grid` (top-left x,y, width and height of one cell) is translated to center (cx,cy) and drawn once more at the end of the UI pass (normalized, top-left origin); callable from any thread +int mtl_ui_ghost_rect(float r[4]); // 上面认到的那件货的屏幕矩形（归一化）；还没认到返回 0 | Screen rectangle (normalized) of the item recognized above; returns 0 if not recognized yet + +int mtl_probe_upload_batch(void); diff --git a/src/gfx/nvn/nvn_names.gen.c b/src/gfx/nvn/nvn_names.gen.c new file mode 100644 index 0000000..4423079 --- /dev/null +++ b/src/gfx/nvn/nvn_names.gen.c @@ -0,0 +1,528 @@ +// Generated from MacNativeGames d2r_gfx/gfx/nvn/names.txt (the NVN names main asks for). Do not edit. +#include "nvn_count.h" +_Static_assert(NVN_NFUNC == 523, "names.txt and nvn_count.h disagree"); +const char *nvn_names[NVN_NFUNC] = { + "nvnDeviceBuilderSetDefaults", + "nvnDeviceBuilderSetFlags", + "nvnDeviceBuilderGetFlags", + "nvnDeviceInitialize", + "nvnDeviceFinalize", + "nvnDeviceSetDebugLabel", + "nvnDeviceGetProcAddress", + "nvnDeviceGetInteger", + "nvnDeviceGetCurrentTimestampInNanoseconds", + "nvnDeviceSetIntermediateShaderCache", + "nvnDeviceGetTextureHandle", + "nvnDeviceGetTexelFetchHandle", + "nvnDeviceGetImageHandle", + "nvnDeviceInstallDebugCallback", + "nvnDeviceGenerateDebugDomainId", + "nvnDeviceSetWindowOriginMode", + "nvnDeviceSetDepthMode", + "nvnDeviceRegisterFastClearColor", + "nvnDeviceRegisterFastClearColori", + "nvnDeviceRegisterFastClearColorui", + "nvnDeviceRegisterFastClearDepth", + "nvnDeviceGetWindowOriginMode", + "nvnDeviceGetDepthMode", + "nvnDeviceGetTimestampInNanoseconds", + "nvnDeviceApplyDeferredFinalizes", + "nvnDeviceFinalizeCommandHandle", + "nvnDeviceWalkDebugDatabase", + "nvnDeviceGetSeparateTextureHandle", + "nvnDeviceGetSeparateSamplerHandle", + "nvnDeviceIsExternalDebuggerAttached", + "nvnQueueGetError", + "nvnQueueGetTotalCommandMemoryUsed", + "nvnQueueGetTotalControlMemoryUsed", + "nvnQueueGetTotalComputeMemoryUsed", + "nvnQueueResetMemoryUsageCounts", + "nvnQueueBuilderSetDevice", + "nvnQueueBuilderSetDefaults", + "nvnQueueBuilderSetFlags", + "nvnQueueBuilderSetCommandMemorySize", + "nvnQueueBuilderSetComputeMemorySize", + "nvnQueueBuilderSetControlMemorySize", + "nvnQueueBuilderGetQueueMemorySize", + "nvnQueueBuilderSetQueueMemory", + "nvnQueueBuilderSetCommandFlushThreshold", + "nvnQueueBuilderSetQueuePriority", + "nvnQueueBuilderGetQueuePriority", + "nvnQueueBuilderGetDevice", + "nvnQueueBuilderGetFlags", + "nvnQueueBuilderGetCommandMemorySize", + "nvnQueueBuilderGetComputeMemorySize", + "nvnQueueBuilderGetControlMemorySize", + "nvnQueueBuilderGetCommandFlushThreshold", + "nvnQueueBuilderGetMemorySize", + "nvnQueueBuilderGetMemory", + "nvnQueueInitialize", + "nvnQueueFinalize", + "nvnQueueSetDebugLabel", + "nvnQueueSubmitCommands", + "nvnQueueFlush", + "nvnQueueFinish", + "nvnQueuePresentTexture", + "nvnQueueAcquireTexture", + "nvnWindowBuilderSetDevice", + "nvnWindowBuilderSetDefaults", + "nvnWindowBuilderSetNativeWindow", + "nvnWindowBuilderSetTextures", + "nvnWindowBuilderSetPresentInterval", + "nvnWindowBuilderSetNumActiveTextures", + "nvnWindowBuilderGetDevice", + "nvnWindowBuilderGetNumTextures", + "nvnWindowBuilderGetTexture", + "nvnWindowBuilderGetNativeWindow", + "nvnWindowBuilderGetPresentInterval", + "nvnWindowBuilderGetNumActiveTextures", + "nvnWindowInitialize", + "nvnWindowFinalize", + "nvnWindowSetDebugLabel", + "nvnWindowAcquireTexture", + "nvnWindowGetNativeWindow", + "nvnWindowGetPresentInterval", + "nvnWindowSetPresentInterval", + "nvnWindowSetCrop", + "nvnWindowGetCrop", + "nvnWindowSetNumActiveTextures", + "nvnWindowGetNumActiveTextures", + "nvnWindowGetNumTextures", + "nvnProgramInitialize", + "nvnProgramFinalize", + "nvnProgramSetDebugLabel", + "nvnProgramSetShaders", + "nvnProgramSetShadersExt", + "nvnProgramSetSampleShading", + "nvnMemoryPoolBuilderSetDevice", + "nvnMemoryPoolBuilderSetDefaults", + "nvnMemoryPoolBuilderSetStorage", + "nvnMemoryPoolBuilderSetFlags", + "nvnMemoryPoolBuilderGetDevice", + "nvnMemoryPoolBuilderGetMemory", + "nvnMemoryPoolBuilderGetSize", + "nvnMemoryPoolBuilderGetFlags", + "nvnMemoryPoolInitialize", + "nvnMemoryPoolSetDebugLabel", + "nvnMemoryPoolFinalize", + "nvnMemoryPoolMap", + "nvnMemoryPoolFlushMappedRange", + "nvnMemoryPoolInvalidateMappedRange", + "nvnMemoryPoolGetBufferAddress", + "nvnMemoryPoolMapVirtual", + "nvnMemoryPoolGetSize", + "nvnMemoryPoolGetFlags", + "nvnTexturePoolInitialize", + "nvnTexturePoolSetDebugLabel", + "nvnTexturePoolFinalize", + "nvnTexturePoolRegisterTexture", + "nvnTexturePoolRegisterImage", + "nvnTexturePoolGetMemoryPool", + "nvnTexturePoolGetMemoryOffset", + "nvnTexturePoolGetSize", + "nvnSamplerPoolInitialize", + "nvnSamplerPoolSetDebugLabel", + "nvnSamplerPoolFinalize", + "nvnSamplerPoolRegisterSampler", + "nvnSamplerPoolRegisterSamplerBuilder", + "nvnSamplerPoolGetMemoryPool", + "nvnSamplerPoolGetMemoryOffset", + "nvnSamplerPoolGetSize", + "nvnBufferBuilderSetDevice", + "nvnBufferBuilderSetDefaults", + "nvnBufferBuilderSetStorage", + "nvnBufferBuilderGetDevice", + "nvnBufferBuilderGetMemoryPool", + "nvnBufferBuilderGetMemoryOffset", + "nvnBufferBuilderGetSize", + "nvnBufferInitialize", + "nvnBufferSetDebugLabel", + "nvnBufferFinalize", + "nvnBufferMap", + "nvnBufferGetAddress", + "nvnBufferFlushMappedRange", + "nvnBufferInvalidateMappedRange", + "nvnBufferGetMemoryPool", + "nvnBufferGetMemoryOffset", + "nvnBufferGetSize", + "nvnBufferGetDebugID", + "nvnTextureBuilderSetDevice", + "nvnTextureBuilderSetDefaults", + "nvnTextureBuilderSetFlags", + "nvnTextureBuilderSetTarget", + "nvnTextureBuilderSetWidth", + "nvnTextureBuilderSetHeight", + "nvnTextureBuilderSetDepth", + "nvnTextureBuilderSetSize1D", + "nvnTextureBuilderSetSize2D", + "nvnTextureBuilderSetSize3D", + "nvnTextureBuilderSetLevels", + "nvnTextureBuilderSetFormat", + "nvnTextureBuilderSetSamples", + "nvnTextureBuilderSetSwizzle", + "nvnTextureBuilderSetDepthStencilMode", + "nvnTextureBuilderGetStorageSize", + "nvnTextureBuilderGetStorageAlignment", + "nvnTextureBuilderSetStorage", + "nvnTextureBuilderSetPackagedTextureData", + "nvnTextureBuilderSetPackagedTextureLayout", + "nvnTextureBuilderSetStride", + "nvnTextureBuilderSetGLTextureName", + "nvnTextureBuilderGetStorageClass", + "nvnTextureBuilderGetDevice", + "nvnTextureBuilderGetFlags", + "nvnTextureBuilderGetTarget", + "nvnTextureBuilderGetWidth", + "nvnTextureBuilderGetHeight", + "nvnTextureBuilderGetDepth", + "nvnTextureBuilderGetLevels", + "nvnTextureBuilderGetFormat", + "nvnTextureBuilderGetSamples", + "nvnTextureBuilderGetSwizzle", + "nvnTextureBuilderGetDepthStencilMode", + "nvnTextureBuilderGetPackagedTextureData", + "nvnTextureBuilderGetPackagedTextureLayout", + "nvnTextureBuilderGetStride", + "nvnTextureBuilderGetSparseTileLayout", + "nvnTextureBuilderGetGLTextureName", + "nvnTextureBuilderGetZCullStorageSize", + "nvnTextureBuilderGetMemoryPool", + "nvnTextureBuilderGetMemoryOffset", + "nvnTextureBuilderGetRawStorageClass", + "nvnTextureViewSetDefaults", + "nvnTextureViewSetLevels", + "nvnTextureViewSetLayers", + "nvnTextureViewSetFormat", + "nvnTextureViewSetSwizzle", + "nvnTextureViewSetDepthStencilMode", + "nvnTextureViewSetTarget", + "nvnTextureViewGetLevels", + "nvnTextureViewGetLayers", + "nvnTextureViewGetFormat", + "nvnTextureViewGetSwizzle", + "nvnTextureViewGetDepthStencilMode", + "nvnTextureViewGetTarget", + "nvnTextureViewCompare", + "nvnTextureInitialize", + "nvnTextureGetZCullStorageSize", + "nvnTextureFinalize", + "nvnTextureSetDebugLabel", + "nvnTextureGetStorageClass", + "nvnTextureGetViewOffset", + "nvnTextureGetFlags", + "nvnTextureGetTarget", + "nvnTextureGetWidth", + "nvnTextureGetHeight", + "nvnTextureGetDepth", + "nvnTextureGetLevels", + "nvnTextureGetFormat", + "nvnTextureGetSamples", + "nvnTextureGetSwizzle", + "nvnTextureGetDepthStencilMode", + "nvnTextureGetStride", + "nvnTextureGetTextureAddress", + "nvnTextureGetSparseTileLayout", + "nvnTextureWriteTexels", + "nvnTextureWriteTexelsStrided", + "nvnTextureReadTexels", + "nvnTextureReadTexelsStrided", + "nvnTextureFlushTexels", + "nvnTextureInvalidateTexels", + "nvnTextureGetMemoryPool", + "nvnTextureGetMemoryOffset", + "nvnTextureGetStorageSize", + "nvnTextureCompare", + "nvnTextureGetDebugID", + "nvnTextureGetRawStorageClass", + "nvnSamplerBuilderSetDevice", + "nvnSamplerBuilderSetDefaults", + "nvnSamplerBuilderSetMinMagFilter", + "nvnSamplerBuilderSetWrapMode", + "nvnSamplerBuilderSetLodClamp", + "nvnSamplerBuilderSetLodBias", + "nvnSamplerBuilderSetCompare", + "nvnSamplerBuilderSetBorderColor", + "nvnSamplerBuilderSetBorderColori", + "nvnSamplerBuilderSetBorderColorui", + "nvnSamplerBuilderSetMaxAnisotropy", + "nvnSamplerBuilderSetReductionFilter", + "nvnSamplerBuilderSetLodSnap", + "nvnSamplerBuilderGetDevice", + "nvnSamplerBuilderGetMinMagFilter", + "nvnSamplerBuilderGetWrapMode", + "nvnSamplerBuilderGetLodClamp", + "nvnSamplerBuilderGetLodBias", + "nvnSamplerBuilderGetCompare", + "nvnSamplerBuilderGetBorderColor", + "nvnSamplerBuilderGetBorderColori", + "nvnSamplerBuilderGetBorderColorui", + "nvnSamplerBuilderGetMaxAnisotropy", + "nvnSamplerBuilderGetReductionFilter", + "nvnSamplerBuilderGetLodSnap", + "nvnSamplerInitialize", + "nvnSamplerFinalize", + "nvnSamplerSetDebugLabel", + "nvnSamplerGetMinMagFilter", + "nvnSamplerGetWrapMode", + "nvnSamplerGetLodClamp", + "nvnSamplerGetLodBias", + "nvnSamplerGetCompare", + "nvnSamplerGetBorderColor", + "nvnSamplerGetBorderColori", + "nvnSamplerGetBorderColorui", + "nvnSamplerGetMaxAnisotropy", + "nvnSamplerGetReductionFilter", + "nvnSamplerCompare", + "nvnSamplerGetDebugID", + "nvnBlendStateSetDefaults", + "nvnBlendStateSetBlendTarget", + "nvnBlendStateSetBlendFunc", + "nvnBlendStateSetBlendEquation", + "nvnBlendStateSetAdvancedMode", + "nvnBlendStateSetAdvancedOverlap", + "nvnBlendStateSetAdvancedPremultipliedSrc", + "nvnBlendStateSetAdvancedNormalizedDst", + "nvnBlendStateGetBlendTarget", + "nvnBlendStateGetBlendFunc", + "nvnBlendStateGetBlendEquation", + "nvnBlendStateGetAdvancedMode", + "nvnBlendStateGetAdvancedOverlap", + "nvnBlendStateGetAdvancedPremultipliedSrc", + "nvnBlendStateGetAdvancedNormalizedDst", + "nvnColorStateSetDefaults", + "nvnColorStateSetBlendEnable", + "nvnColorStateSetLogicOp", + "nvnColorStateSetAlphaTest", + "nvnColorStateGetBlendEnable", + "nvnColorStateGetLogicOp", + "nvnColorStateGetAlphaTest", + "nvnChannelMaskStateSetDefaults", + "nvnChannelMaskStateSetChannelMask", + "nvnChannelMaskStateGetChannelMask", + "nvnMultisampleStateSetDefaults", + "nvnMultisampleStateSetMultisampleEnable", + "nvnMultisampleStateSetSamples", + "nvnMultisampleStateSetAlphaToCoverageEnable", + "nvnMultisampleStateSetAlphaToCoverageDither", + "nvnMultisampleStateGetMultisampleEnable", + "nvnMultisampleStateGetSamples", + "nvnMultisampleStateGetAlphaToCoverageEnable", + "nvnMultisampleStateGetAlphaToCoverageDither", + "nvnMultisampleStateSetRasterSamples", + "nvnMultisampleStateGetRasterSamples", + "nvnMultisampleStateSetCoverageModulationMode", + "nvnMultisampleStateGetCoverageModulationMode", + "nvnMultisampleStateSetCoverageToColorEnable", + "nvnMultisampleStateGetCoverageToColorEnable", + "nvnMultisampleStateSetCoverageToColorOutput", + "nvnMultisampleStateGetCoverageToColorOutput", + "nvnMultisampleStateSetSampleLocationsEnable", + "nvnMultisampleStateGetSampleLocationsEnable", + "nvnMultisampleStateGetSampleLocationsGrid", + "nvnMultisampleStateSetSampleLocationsGridEnable", + "nvnMultisampleStateGetSampleLocationsGridEnable", + "nvnMultisampleStateSetSampleLocations", + "nvnPolygonStateSetDefaults", + "nvnPolygonStateSetCullFace", + "nvnPolygonStateSetFrontFace", + "nvnPolygonStateSetPolygonMode", + "nvnPolygonStateSetPolygonOffsetEnables", + "nvnPolygonStateGetCullFace", + "nvnPolygonStateGetFrontFace", + "nvnPolygonStateGetPolygonMode", + "nvnPolygonStateGetPolygonOffsetEnables", + "nvnDepthStencilStateSetDefaults", + "nvnDepthStencilStateSetDepthTestEnable", + "nvnDepthStencilStateSetDepthWriteEnable", + "nvnDepthStencilStateSetDepthFunc", + "nvnDepthStencilStateSetStencilTestEnable", + "nvnDepthStencilStateSetStencilFunc", + "nvnDepthStencilStateSetStencilOp", + "nvnDepthStencilStateGetDepthTestEnable", + "nvnDepthStencilStateGetDepthWriteEnable", + "nvnDepthStencilStateGetDepthFunc", + "nvnDepthStencilStateGetStencilTestEnable", + "nvnDepthStencilStateGetStencilFunc", + "nvnDepthStencilStateGetStencilOp", + "nvnVertexAttribStateSetDefaults", + "nvnVertexAttribStateSetFormat", + "nvnVertexAttribStateSetStreamIndex", + "nvnVertexAttribStateGetFormat", + "nvnVertexAttribStateGetStreamIndex", + "nvnVertexStreamStateSetDefaults", + "nvnVertexStreamStateSetStride", + "nvnVertexStreamStateSetDivisor", + "nvnVertexStreamStateGetStride", + "nvnVertexStreamStateGetDivisor", + "nvnCommandBufferInitialize", + "nvnCommandBufferFinalize", + "nvnCommandBufferSetDebugLabel", + "nvnCommandBufferSetMemoryCallback", + "nvnCommandBufferSetMemoryCallbackData", + "nvnCommandBufferSetCommandMemoryCallbackEnabled", + "nvnCommandBufferAddCommandMemory", + "nvnCommandBufferAddControlMemory", + "nvnCommandBufferGetCommandMemorySize", + "nvnCommandBufferGetCommandMemoryUsed", + "nvnCommandBufferGetCommandMemoryFree", + "nvnCommandBufferGetControlMemorySize", + "nvnCommandBufferGetControlMemoryUsed", + "nvnCommandBufferGetControlMemoryFree", + "nvnCommandBufferBeginRecording", + "nvnCommandBufferEndRecording", + "nvnCommandBufferCallCommands", + "nvnCommandBufferCopyCommands", + "nvnCommandBufferBindBlendState", + "nvnCommandBufferBindChannelMaskState", + "nvnCommandBufferBindColorState", + "nvnCommandBufferBindMultisampleState", + "nvnCommandBufferBindPolygonState", + "nvnCommandBufferBindDepthStencilState", + "nvnCommandBufferBindVertexAttribState", + "nvnCommandBufferBindVertexStreamState", + "nvnCommandBufferBindProgram", + "nvnCommandBufferBindVertexBuffer", + "nvnCommandBufferBindVertexBuffers", + "nvnCommandBufferBindUniformBuffer", + "nvnCommandBufferBindUniformBuffers", + "nvnCommandBufferBindTransformFeedbackBuffer", + "nvnCommandBufferBindTransformFeedbackBuffers", + "nvnCommandBufferBindStorageBuffer", + "nvnCommandBufferBindStorageBuffers", + "nvnCommandBufferBindTexture", + "nvnCommandBufferBindTextures", + "nvnCommandBufferBindImage", + "nvnCommandBufferBindImages", + "nvnCommandBufferSetPatchSize", + "nvnCommandBufferSetInnerTessellationLevels", + "nvnCommandBufferSetOuterTessellationLevels", + "nvnCommandBufferSetPrimitiveRestart", + "nvnCommandBufferBeginTransformFeedback", + "nvnCommandBufferEndTransformFeedback", + "nvnCommandBufferPauseTransformFeedback", + "nvnCommandBufferResumeTransformFeedback", + "nvnCommandBufferDrawTransformFeedback", + "nvnCommandBufferDrawArrays", + "nvnCommandBufferDrawElements", + "nvnCommandBufferDrawElementsBaseVertex", + "nvnCommandBufferDrawArraysInstanced", + "nvnCommandBufferDrawElementsInstanced", + "nvnCommandBufferDrawArraysIndirect", + "nvnCommandBufferDrawElementsIndirect", + "nvnCommandBufferMultiDrawArraysIndirectCount", + "nvnCommandBufferMultiDrawElementsIndirectCount", + "nvnCommandBufferClearColor", + "nvnCommandBufferClearColori", + "nvnCommandBufferClearColorui", + "nvnCommandBufferClearDepthStencil", + "nvnCommandBufferDispatchCompute", + "nvnCommandBufferDispatchComputeIndirect", + "nvnCommandBufferSetViewport", + "nvnCommandBufferSetViewports", + "nvnCommandBufferSetViewportSwizzles", + "nvnCommandBufferSetScissor", + "nvnCommandBufferSetScissors", + "nvnCommandBufferSetDepthRange", + "nvnCommandBufferSetDepthBounds", + "nvnCommandBufferSetDepthRanges", + "nvnCommandBufferSetTiledCacheAction", + "nvnCommandBufferSetTiledCacheTileSize", + "nvnCommandBufferBindSeparateTexture", + "nvnCommandBufferBindSeparateSampler", + "nvnCommandBufferBindSeparateTextures", + "nvnCommandBufferBindSeparateSamplers", + "nvnCommandBufferSetStencilValueMask", + "nvnCommandBufferSetStencilMask", + "nvnCommandBufferSetStencilRef", + "nvnCommandBufferSetBlendColor", + "nvnCommandBufferSetPointSize", + "nvnCommandBufferSetLineWidth", + "nvnCommandBufferSetPolygonOffsetClamp", + "nvnCommandBufferSetAlphaRef", + "nvnCommandBufferSetSampleMask", + "nvnCommandBufferSetRasterizerDiscard", + "nvnCommandBufferSetDepthClamp", + "nvnCommandBufferSetConservativeRasterEnable", + "nvnCommandBufferSetConservativeRasterDilate", + "nvnCommandBufferSetSubpixelPrecisionBias", + "nvnCommandBufferCopyBufferToTexture", + "nvnCommandBufferCopyTextureToBuffer", + "nvnCommandBufferCopyTextureToTexture", + "nvnCommandBufferCopyBufferToBuffer", + "nvnCommandBufferClearBuffer", + "nvnCommandBufferClearTexture", + "nvnCommandBufferClearTexturei", + "nvnCommandBufferClearTextureui", + "nvnCommandBufferUpdateUniformBuffer", + "nvnCommandBufferReportCounter", + "nvnCommandBufferResetCounter", + "nvnCommandBufferReportValue", + "nvnCommandBufferSetRenderEnable", + "nvnCommandBufferSetRenderEnableConditional", + "nvnCommandBufferSetRenderTargets", + "nvnCommandBufferDiscardColor", + "nvnCommandBufferDiscardDepthStencil", + "nvnCommandBufferDownsample", + "nvnCommandBufferTiledDownsample", + "nvnCommandBufferDownsampleTextureView", + "nvnCommandBufferTiledDownsampleTextureView", + "nvnCommandBufferBarrier", + "nvnCommandBufferWaitSync", + "nvnCommandBufferFenceSync", + "nvnCommandBufferSetTexturePool", + "nvnCommandBufferSetSamplerPool", + "nvnCommandBufferSetShaderScratchMemory", + "nvnCommandBufferSaveZCullData", + "nvnCommandBufferRestoreZCullData", + "nvnCommandBufferSetCopyRowStride", + "nvnCommandBufferSetCopyImageStride", + "nvnCommandBufferGetCopyRowStride", + "nvnCommandBufferGetCopyImageStride", + "nvnCommandBufferDrawTexture", + "nvnProgramSetSubroutineLinkage", + "nvnCommandBufferSetProgramSubroutines", + "nvnCommandBufferBindCoverageModulationTable", + "nvnCommandBufferResolveDepthBuffer", + "nvnCommandBufferSetColorReductionEnable", + "nvnCommandBufferSetColorReductionThresholds", + "nvnCommandBufferPushDebugGroupStatic", + "nvnCommandBufferPushDebugGroupDynamic", + "nvnCommandBufferPushDebugGroup", + "nvnCommandBufferPopDebugGroup", + "nvnCommandBufferPopDebugGroupId", + "nvnCommandBufferInsertDebugMarkerStatic", + "nvnCommandBufferInsertDebugMarkerDynamic", + "nvnCommandBufferInsertDebugMarker", + "nvnCommandBufferGetMemoryCallback", + "nvnCommandBufferGetMemoryCallbackData", + "nvnCommandBufferIsRecording", + "nvnSyncInitialize", + "nvnSyncFinalize", + "nvnSyncSetDebugLabel", + "nvnQueueFenceSync", + "nvnSyncWait", + "nvnQueueWaitSync", + "nvnSyncInitializeFromFencedGLSync", + "nvnSyncCreateGLSync", + "nvnEventBuilderSetDefaults", + "nvnEventBuilderSetStorage", + "nvnEventBuilderGetStorage", + "nvnEventBuilderGetMemoryPool", + "nvnEventBuilderGetMemoryOffset", + "nvnEventInitialize", + "nvnEventFinalize", + "nvnEventGetValue", + "nvnEventSignal", + "nvnEventGetMemoryPool", + "nvnEventGetMemoryOffset", + "nvnCommandBufferWaitEvent", + "nvnCommandBufferSignalEvent", + "nvnCommandBufferSetStencilCullCriteria", + "nvnCommandBufferInsertCheckpointStatic", + "nvnCommandBufferInsertCheckpointDynamic", + "nvnDeviceSetAutomaticCheckpointsEnable", + "nvnDeviceSetAutomaticCheckpointsSamplingInterval", + "nvnDeviceSetAutomaticCheckpointsByDebugGroup", + "nvnDeviceSetCheckpointCallStacksEnable", + "nvnDeviceSetCheckpointMaxCallStacksDepth", +}; diff --git a/src/gfx/nvn/nvn_obj.h b/src/gfx/nvn/nvn_obj.h new file mode 100644 index 0000000..f08a4a8 --- /dev/null +++ b/src/gfx/nvn/nvn_obj.h @@ -0,0 +1,203 @@ +// NVN 宿主侧对象布局与命令记录格式：hle_nvn.c（接口层，写）与 nvn_mtl.m（Metal 回放层，读）共用。 +// 对象就地存在游戏分配的 NVN 对象内存里，大小不超过 nvn.h 的保留大小（FITS 静态断言）。 +// 状态对象（Blend / Color / … 4–0x18 字节）是游戏按值拷贝的小结构，布局用位域就地打包。 +// NVN host-side object layout and command record format: shared by hle_nvn.c (interface layer, writes) and nvn_mtl.m (Metal replay layer, reads). +// Objects are stored in place in the NVN object memory allocated by the game, with size not exceeding the reserved size in nvn.h (FITS static assertion). +// State objects (Blend / Color / ... 4–0x18 bytes) are small structs the game copies by value; their layout is packed in place with bit-fields. +#pragma once +#include +#include +#include "nvn.h" +#include "nvn_tex.h" + +#define MAGIC(a, b, c, d) ((uint32_t)(a) | (uint32_t)(b) << 8 | (uint32_t)(c) << 16 | (uint32_t)(d) << 24) +#define FITS(T, H) _Static_assert(sizeof(T) <= sizeof(H), #T " 超出 " #H " 的保留大小") + +typedef struct { uint32_t flags; } dev_builder_t; FITS(dev_builder_t, NVNdeviceBuilder); +typedef struct { uint32_t magic, flags, origin, depth_mode; void *dbg_cb, *dbg_data; } dev_t_; FITS(dev_t_, NVNdevice); + +typedef struct { NVNdevice *dev; uint32_t flags; uint64_t cmd_sz, comp_sz, ctl_sz, flush; void *mem; uint64_t mem_sz; } queue_builder_t; +FITS(queue_builder_t, NVNqueueBuilder); +typedef struct { uint32_t magic; NVNdevice *dev; uint64_t submits; } queue_t; FITS(queue_t, NVNqueue); + +typedef struct { NVNdevice *dev; void *mem; uint64_t size; uint32_t flags; } pool_builder_t; FITS(pool_builder_t, NVNmemoryPoolBuilder); +// game_mem/host/span: TOTK host-owned VM storage for physical/virtual pools (totk_vpool.c); host is NULL for game-backed pools. +typedef struct { uint32_t magic, flags; uint8_t *mem; uint64_t size; uint8_t *game_mem, *host; uint64_t span; } pool_t; FITS(pool_t, NVNmemoryPool); + +typedef struct { const pool_t *pool; int64_t off; int32_t n; } dpool_t; // 贴图 / 采样器描述符池 | Texture / sampler descriptor pool +FITS(dpool_t, NVNtexturePool); FITS(dpool_t, NVNsamplerPool); + +typedef struct { + NVNdevice *dev; nvn_texdesc d; pool_t *pool; int64_t off; + uint8_t has_layout, layout[8]; + const void *pkg_data; // TOTK: SetPackagedTextureData source, copied into storage at Initialize +} tex_builder_t; FITS(tex_builder_t, NVNtextureBuilder); +typedef struct { + uint32_t magic; nvn_texdesc d; uint8_t has_layout, layout[8]; + nvn_texlayout L; uint8_t *mem; pool_t *pool; int64_t off; + uint64_t serial; // 初始化序号：回放层按它在宿主表里找 MTLTexture（游戏可能按值拷贝对象，不能存对象指针） | Initialization serial: the replay layer uses it to find the MTLTexture in the host table (the game may copy the object by value, so object pointers cannot be stored) +} tex_t; FITS(tex_t, NVNtexture); +// TOTK: the game reserves 16 bytes for NVNtextureView on its stack (main+0x823398 SetDefaults at sp+0x30, its own +// data at sp+0x40), so the view is packed into 16 bytes. swz = NVN view swizzle selectors. +typedef struct { + uint8_t set_levels : 1, set_layers : 1, set_format : 1, set_target : 1, set_swizzle : 1, pad0 : 3; + uint8_t base_level, nlevels, target; uint16_t min_layer, nlayers; uint16_t format; uint8_t swz[4]; uint8_t pad1[2]; +} view_t; +_Static_assert(sizeof(view_t) == 16, "NVNtextureView is 16 bytes in TOTK's NVN"); +FITS(view_t, NVNtextureView); + +// 采样器参数（builder 与对象共用一份） +// Sampler parameters (one copy shared by builder and object) +typedef struct { uint8_t minf, magf, ws, wt, wr, cmp_mode, cmp_func, reduction; float lod_min, lod_max, lod_bias, aniso, border[4], lod_snap; } samp_par; // TOTK: lod_snap stored for queries (Metal has no equivalent) // 枚举值都 < 256 | Enum values are all < 256 +typedef struct { NVNdevice *dev; samp_par p; } samp_builder_t; FITS(samp_builder_t, NVNsamplerBuilder); +typedef struct { uint32_t magic; samp_par p; } samp_t; FITS(samp_t, NVNsampler); +_Static_assert(sizeof(samp_par) == 44, "TOTK sampler parameters"); + +typedef struct { uint32_t magic, nstage; NVNshaderData sd[6]; uint64_t serial; } prog_t; FITS(prog_t, NVNprogram); // serial：回放层解析结果的表键 | serial: table key for the replay layer's resolved result + +typedef struct { uint32_t magic, fenced; uint64_t fence; } sync_t; FITS(sync_t, NVNsync); +typedef struct { const pool_t *pool; int64_t off; } ev_builder_t; FITS(ev_builder_t, NVNeventBuilder); +typedef struct { uint32_t magic; volatile uint32_t *val; } ev_t; FITS(ev_t, NVNevent); + +typedef struct { NVNdevice *dev; void *native; int32_t ntex, interval, active; NVNtexture *tex[4]; } win_builder_t; +FITS(win_builder_t, NVNwindowBuilder); +// avail[i]：交换链第 i 张可再次获取的时刻（ns，单调时钟）= 它的显示时刻 + 一个出图周期；pace：本窗口的出图节拍器目标时刻 +// avail[i]: time at which swapchain image i can be acquired again (ns, monotonic clock) = its display time + one present period; pace: target time of this window's present pacer +typedef struct { uint32_t magic; void *native; int32_t ntex, interval, next, active, crop[4]; NVNtexture *tex[4]; uint64_t presents, avail[4], pace, ticket[4]; } win_t; // ticket[i]：第 i 张最近一次出图的票号 | ticket[i]: ticket of the most recent present of image i +FITS(win_t, NVNwindow); + +// Command records live in host memory; finished copies are owned by guest control memory. +typedef struct { + uint32_t magic, recording; + uint8_t *cur; + uint64_t guest_flags; // +0x10: NVN 55 inline code writes the dirty flag here + uint8_t *end; + // TOTK: NVN 55 headers inline fast paths that read the command write pointer at +0x20 and its end at +0x28 + // (main+0x70644c) and, when 0x30 bytes fit, emit raw GPU methods themselves. ~0 / 0 never fit, so every + // call takes the API path (the *Deferred entry points). + uint64_t inline_cur, inline_end; + PFNNVNCOMMANDBUFFERMEMORYCALLBACKPROC cb; void *cb_data; + NVNdevice *dev; + uint8_t *seg; // TOTK: start of the host recording buffer (records never go into game memory) + void *ctl; size_t ctl_size; // TOTK: control memory given by the game; recording copies live as long as it + size_t game_size; // TOTK: command memory the game added (accounting only; not used for storage) + uintptr_t cmd_base; // TOTK: pool storage address + offset of the command memory the game added + size_t cmd_used, ctl_used, rec_start; // rec_start: cmd_used when the current recording began + size_t hw_used, last_rec, hw_start; // TOTK: command memory the real NVN driver would have consumed (what GetCommandMemoryUsed reports; the game sizes resource heaps from it), size of the last record // TOTK: usage reported to the game since the memory was added (main+0x711e4c submits only non-empty buffers) +} cmd_t; FITS(cmd_t, NVNcommandBuffer); +_Static_assert(__builtin_offsetof(cmd_t, guest_flags) == 0x10, "NVN inline dirty flag"); +_Static_assert(__builtin_offsetof(cmd_t, inline_cur) == 0x20 && __builtin_offsetof(cmd_t, inline_end) == 0x28, "NVN inline fast-path fields"); + +// ---------- 状态对象（默认值按 NVN / GL 默认） ---------- +// ---------- State objects (defaults follow the NVN / GL defaults) ---------- +typedef struct { uint32_t blend_en : 8, logic_op : 4, alpha_func : 4; } color_st; FITS(color_st, NVNcolorState); +typedef struct { uint32_t mask; } chmask_st; /* 每目标 4 位：r=1 g=2 b=4 a=8，置位 = 写 | 4 bits per target: r=1 g=2 b=4 a=8, bit set = write */ FITS(chmask_st, NVNchannelMaskState); +typedef struct { uint64_t target : 3, src_rgb : 7, dst_rgb : 7, src_a : 7, dst_a : 7, eq_rgb : 3, eq_a : 3; } blend_st; +FITS(blend_st, NVNblendState); +typedef struct { uint8_t enable, samples, a2c, a2c_dither, raster_samples, cov_mod, cov2color_en, cov2color_out, sl_en, sl_grid; } ms_st; +FITS(ms_st, NVNmultisampleState); +typedef struct { uint32_t cull : 2, front : 1, mode : 2, offset_en : 3; } poly_st; FITS(poly_st, NVNpolygonState); +typedef struct { + uint64_t dt : 1, dw : 1, df : 4, st : 1, sf_f : 4, sf_b : 4, + fail_f : 4, zfail_f : 4, zpass_f : 4, fail_b : 4, zfail_b : 4, zpass_b : 4; +} ds_st; FITS(ds_st, NVNdepthStencilState); +typedef struct { uint32_t fmt : 8, offset : 19, stream : 5; } va_st; FITS(va_st, NVNvertexAttribState); +typedef struct { uint32_t stride, divisor; } vs_st; FITS(vs_st, NVNvertexStreamState); + +// ---------- 命令记录：{op, size（含头，8 对齐）} + 载荷 ---------- +// ---------- Command records: {op, size (including header, aligned to 8)} + payload ---------- +typedef struct { uint32_t op, size; } op_hdr; +enum { + OP_END, OP_JUMP, OP_BEGIN, OP_CALL, + OP_SIGNAL_EVENT, OP_REPORT_COUNTER, OP_COPY_B2B, OP_UPDATE_UB, // CPU 可见副作用 | CPU-visible side effects + OP_SET_RT, OP_CLEAR_COLOR, OP_CLEAR_DS, OP_VIEWPORTS, OP_SCISSOR, OP_DEPTH_RANGE, + OP_BIND_PROGRAM, OP_BIND_STATE, OP_BIND_VATTR, OP_BIND_VSTREAM, OP_BIND_VBUF, + OP_BIND_UBO, OP_BIND_SSBO, OP_BIND_TEX, OP_BIND_SAMP, + OP_STENCIL, OP_POLY_OFFSET, OP_DRAW, OP_DRAW_IDX, OP_DISPATCH, OP_BARRIER, + OP_SET_TEX_POOL, OP_SET_SAMP_POOL, OP_COPY_B2T, OP_COPY_STRIDE, OP_BLEND_COLOR, OP_COPY_T2B, OP_BIND_IMG, OP_COPY_T2T, OP_RESET_COUNTER, OP_CLEAR_TEX, OP_DEPTH_CLAMP, OP_RENDER_ENABLE, OP_RASTER_DISCARD, OP_RENDER_COND, OP_COUNT_ +}; +// 载荷（紧跟 op_hdr） +// Payload (immediately follows op_hdr) +typedef struct { int32_t n; const tex_t *color[8]; view_t cview[8]; uint8_t has_cview[8]; const tex_t *depth; view_t dview; uint8_t has_dview; } p_set_rt; +typedef struct { int32_t index, mask; float rgba[4]; } p_clear_color; +typedef struct { float depth; int32_t depth_mask, stencil, stencil_mask; } p_clear_ds; +typedef struct { int32_t first, n; float v[16][6]; } p_viewports; // x, y, w, h, 近, 远（近远由 DEPTH_RANGE 覆盖） | x, y, w, h, near, far (near and far are overridden by DEPTH_RANGE) +typedef struct { int32_t x, y, w, h; } p_scissor; +typedef struct { int32_t value; } p_flag; // TOTK: single boolean render state (OP_DEPTH_CLAMP) +typedef struct { int32_t mode; int32_t pad; uint64_t addr; } p_render_cond; // TOTK: SetRenderEnableConditional: mode 0 = render if the two NVNcounterData.counter values at addr / addr+16 are equal, 1 = if not equal +typedef struct { float n, f; } p_depth_range; +typedef struct { const prog_t *prog; int32_t stages; } p_bind_program; +typedef struct { int32_t kind; uint8_t bytes[0x18]; } p_bind_state; // kind：0 blend 1 color 2 chmask 3 ms 4 poly 5 ds +typedef struct { int32_t first, n; uint32_t st[16]; } p_bind_vattr; +typedef struct { int32_t first, n; vs_st st[16]; } p_bind_vstream; +typedef struct { int32_t first, n; NVNbufferRange r[16]; } p_bind_vbuf; +typedef struct { int32_t stage, index; uint64_t addr, size; } p_bind_buf; // UBO / SSBO +// src: NVN 55 *Deferred binds keep the game's handle array pointer (read at submit into h, see hle_nvn.c resolve_deferred); 0 = immediate +typedef struct { int32_t stage, first, n, pad_; uint64_t src; uint64_t h[32]; } p_bind_handles; // 分离贴图 / 采样器句柄 | Separate texture / sampler handles +typedef struct { int32_t face, value_mask, mask, ref, which; } p_stencil; // which：0 值掩码 1 写掩码 2 参考值 | which: 0 value mask 1 write mask 2 reference value +typedef struct { float factor, units, clamp; } p_poly_offset; +typedef struct { int32_t prim, first, count, base_instance, instances; } p_draw; // TOTK: instances 0 = one instance (DrawArrays) +typedef struct { int32_t prim, itype, count, base_vertex, base_instance, instances; uint64_t ibuf; } p_draw_idx; +typedef struct { int32_t x, y, z; uint64_t caller; } p_dispatch; // caller：游戏调用点返回地址（定位派发者） | caller: return address of the game call site (to locate the dispatcher) +typedef struct { const dpool_t *pool; } p_set_pool; +typedef struct { uint64_t src; const tex_t *tex; view_t view; uint8_t has_view; NVNcopyRegion r; int32_t flags; } p_copy_b2t; +typedef struct { const tex_t *src, *dst; view_t sv, dv; uint8_t has_sv, has_dv; NVNcopyRegion sr, dr; int32_t flags; } p_copy_t2t; // T2B 复用：src 存目标缓冲地址 | Reused for T2B: src stores the destination buffer address +typedef struct { const tex_t *tex; view_t view; uint8_t has_view, is_uint; NVNcopyRegion r; int32_t mask; union { float f[4]; uint32_t u[4]; } c; } p_clear_tex; +typedef struct { int64_t stride; } p_copy_stride; +typedef struct { float rgba[4]; } p_blend_color; + +// Metal 回放层入口（nvn_mtl.m）：把一组命令句柄回放进当前提交 +// Entry point of the Metal replay layer (nvn_mtl.m): replays a set of command handles into the current submission +void mtl_exec(const uint8_t *handle); +void mtl_exec_rest(const uint64_t *handles, int n); // 告知本次提交里当前句柄之后还有哪些句柄（通道前瞻跨句柄用） | Tells which handles in this submission still follow the current handle (used by the pass lookahead across handles) +// 出图：把交换链贴图画到前端 CAMetalLayer（无窗口时只计数） +// Present: draw the swapchain texture to the front-end CAMetalLayer (only counts when there is no window) +uint64_t mtl_present(const tex_t *t, double period_s, const int crop[4]); // 出图入队、立即返回票号；period_s：该张至少显示的时长（= 出图周期，0 = 不限） | Enqueue a present and return the ticket immediately; period_s: minimum time this image is displayed (= present period, 0 = unlimited) +void mtl_present_wait(uint64_t ticket); // 等该票号的出图处理完（blit 已提交到 GPU 队列） | Wait until the present with this ticket has been processed (blit committed to the GPU queue) +void mtl_present_flush(void); // 等已入队的出图全部处理完 | Wait until all enqueued presents have been processed +void mtl_present_test_stall(uint64_t ns); // 单测：出图线程每次出图前额外停顿 ns | Unit test: extra pause in ns before each present on the present thread +int mtl_display_hz(void); // 有窗口时屏幕最高刷新率（Hz），无窗口 0 | Maximum refresh rate of the screen (Hz) when there is a window, 0 without a window +void mtl_set_tex_pool_n(int n); // 贴图描述符池容量：池外的贴图号按「空槽」处理（读出 0） | Texture descriptor pool capacity: texture IDs outside the pool are treated as 'empty slots' (read 0) +// 计时埋点（Mac / iOS 共用）：提交按帧号打标；GPU 时间 = 该帧全部命令缓冲 GPUEnd−GPUStart 之和，全部完成后才可读（否则 −1） +// Timing instrumentation (shared by Mac / iOS): submissions are tagged with the frame number; GPU time = sum of GPUEnd−GPUStart over all of that frame's command buffers, readable only after all complete (otherwise −1) +void mtl_set_frame(uint64_t f); +void mtl_ub_present(double iv_ms); // 同进程交替测量（DM_NVN_UB）：出图线程每帧报实际上屏间隔 | Same-process alternating measurement (DM_NVN_UB): the present thread reports the actual present interval every frame +double mtl_gpu_ms(uint64_t f); +void mtl_sub_index(const uint64_t *h, int n); // 提交开始：按回放顺序建写回前瞻索引（nvn_mtl.m） | Start of submission: build the store lookahead index in replay order (nvn_mtl.m) +void xv_test_counts(int *dc, int *viol); +int mtl_gpu_span(uint64_t f, uint64_t *t0, uint64_t *t1); // 该帧最早 GPUStart / 最晚 GPUEnd（mach 绝对时间 ns）；未全部完成或无 GPU 时间返回 0 | The frame's earliest GPUStart / latest GPUEnd (mach absolute time, ns); returns 0 if not all complete or there is no GPU time +void mtl_stats(uint64_t *passes, uint64_t *draws); +void mtl_args_stats(uint64_t *enc, uint64_t *reuse); +void mtl_use_stats(uint64_t *call, uint64_t *skip); +uint64_t mtl_ds_omit(void); // 前瞻证明不用深度模板、未挂深度附件的通道累计数 | Cumulative number of passes with no depth attachment because the lookahead proved depth-stencil unused +uint64_t mtl_drawable_ns_take(void); // 取走并清零：出图里等 nextDrawable 的累计 ns // useResource 实发 / 同编码器去重跳过累计次数 // 参数缓冲新编 / 复用累计次数 | Take and zero: cumulative ns spent waiting for nextDrawable in present // Cumulative useResource calls actually issued / skipped by same-encoder dedup // Cumulative argument buffer new-encode / reuse counts +void mtl_bw(uint64_t *load, uint64_t *store, uint64_t *full_rw_passes); // 附件读入 / 写出累计字节、全量读写通道累计数 | Cumulative attachment load / store bytes, cumulative number of full load-store passes +uint64_t mtl_udc_saved_bytes(void); // DM_NVN_UNWRITTEN_DC 实际省下的写回字节 | Write-back bytes actually saved by DM_NVN_UNWRITTEN_DC +int mtl_has_window(void); +int mtl_warm_wait(int timeout_ms); +int mtl_link_active(void); +/* frame_ticks: 60 Hz ticks one game frame occupies (the game's present interval; 1 in display mode). */ +int mtl_link_wait_tick(uint64_t timeout_ns, unsigned frame_ticks); +double mtl_presented_time(uint64_t n); + // 第 n 次出图（从 1 计）的实际呈现时刻（秒）；0 = 未就绪，−1 = 被丢弃 + // Actual presentation time (seconds) of the n-th present (counted from 1); 0 = not ready, −1 = dropped +// 贴图释放（Finalize 时） +// Texture release (at Finalize) +void mtl_tex_release(tex_t *t); +void mtl_pkg_stats(uint64_t *uploads, uint64_t *reuploads); +uint64_t mtl_pkg_miss(void); +uint64_t mtl_upload_ns_take(void); // 取走并清零：上次以来提交线程上预排布贴图上传耗时 // DM_NVN_DIRTYCHECK 诊断：漏检次数 // 预排布贴图：首传次数、因池内容变化而重传次数 | Take and zero: time spent uploading packaged textures on the submit thread since last time // DM_NVN_DIRTYCHECK diagnostic: number of missed detections // Packaged textures: number of first uploads, number of re-uploads due to pool content changes +uint64_t mtl_upload_gpu_ns_take(void); // 取走并清零：上传命令缓冲的显卡执行耗时（GPUEndTime − GPUStartTime 累加） | Take and zero: GPU execution time of upload command buffers (sum of GPUEndTime − GPUStartTime) +void mtl_create_stats(uint64_t *lib, uint64_t *pso); // 累计：提交线程上同步载入的着色器库数、同步新建的渲染 + 计算管线数（预热领用的不算） | Cumulative: number of shader libraries loaded synchronously on the submit thread, number of render + compute pipelines created synchronously (those taken from warm-up do not count) +// CPU 读贴图：level / layer0 起，按 row / img 字节步长写进 dst +// CPU texture read: starting at level / layer0, written into dst with row / img byte strides +void mtl_read_texels(const tex_t *t, unsigned level, unsigned layer0, const NVNcopyRegion *r, uint8_t *dst, size_t row, size_t img); +void mtl_write_texels(const tex_t *t, unsigned level, unsigned layer0, const NVNcopyRegion *r, const uint8_t *src, size_t row, size_t img); // TOTK: CPU texel write (WriteTexels) +// 描述符池登记（贴图号 / 采样器号 → 对象快照）、程序释放 +// Descriptor pool registration (texture ID / sampler ID → object snapshot), program release +void mtl_reg_tex(int id, const tex_t *t, const view_t *v); +void mtl_reg_samp(int id, const samp_par *p); +void mtl_prog_release(prog_t *p); +// 新序号（> 0，全局递增） +// New serial (> 0, globally increasing) +uint64_t nvn_new_serial(void); diff --git a/src/gfx/nvn/nvn_pace.h b/src/gfx/nvn/nvn_pace.h new file mode 100644 index 0000000..fd915d5 --- /dev/null +++ b/src/gfx/nvn/nvn_pace.h @@ -0,0 +1,20 @@ +// 出图节拍器（纯函数，Mac / iOS 共用；单测 tests/t_pace.c）。 +// 语义同垂直同步：节拍点 = 上一节拍点 + 周期。早到 → 等到节拍点；迟到不足一个周期 → 立即出图、保持原相位； +// 落后一个周期以上（卡顿）→ 从当前时刻重新起拍，不连发追帧。周期 0 = 不等（节拍点跟随当前时刻）。 +// Present pacer (pure function, shared by Mac / iOS; unit test tests/t_pace.c). +// Semantics as vsync: present slot = previous present slot + period. Early → wait until the present slot; late by less than one period → present immediately and keep the original phase; +// behind by one period or more (stutter) → restart the cadence from the current time, no burst of presents to catch up. Period 0 = no wait (the present slot follows the current time). +#pragma once +#include + +typedef struct { uint64_t target; } nvn_pacer; + +// now：单调时钟（ns）；period：本帧周期（ns，0 = 不等）。返回需要等待的 ns（0 = 立即出图），并更新内部目标时刻。 +// now: monotonic clock (ns); period: this frame's period (ns, 0 = no wait). Returns the ns to wait (0 = present immediately) and updates the internal target time. +static inline uint64_t nvn_pace(nvn_pacer *p, uint64_t now, uint64_t period) { + if (!period || !p->target) { p->target = now; return 0; } + uint64_t next = p->target + period; + if (now <= next) { p->target = next; return next - now; } + p->target = now - next < period ? next : now; + return 0; +} diff --git a/src/gfx/nvn/nvn_protos.h b/src/gfx/nvn/nvn_protos.h new file mode 100644 index 0000000..4177399 --- /dev/null +++ b/src/gfx/nvn/nvn_protos.h @@ -0,0 +1,577 @@ +// 由 dm/gfx/tools/gen_protos.py 生成，勿手改。NVNboolean 参数已换成 uint32_t。 +// Generated by dm/gfx/tools/gen_protos.py, do not edit by hand. NVNboolean parameters have been replaced with uint32_t. +#pragma once +#include "nvn.h" + +typedef void (*NVNT_nvnDeviceBuilderSetDefaults)(NVNdeviceBuilder*); +typedef void (*NVNT_nvnDeviceBuilderSetFlags)(NVNdeviceBuilder*, int); +typedef NVNboolean (*NVNT_nvnDeviceInitialize)(NVNdevice*, const NVNdeviceBuilder*); +typedef void (*NVNT_nvnDeviceFinalize)(NVNdevice*); +typedef void (*NVNT_nvnDeviceSetDebugLabel)(NVNdevice*, const char*); +typedef PFNNVNGENERICFUNCPTRPROC (*NVNT_nvnDeviceGetProcAddress)(const NVNdevice*, const char*); +typedef void (*NVNT_nvnDeviceGetInteger)(const NVNdevice*, NVNdeviceInfo, int*); +typedef uint64_t (*NVNT_nvnDeviceGetCurrentTimestampInNanoseconds)(const NVNdevice*); +typedef void (*NVNT_nvnDeviceSetIntermediateShaderCache)(NVNdevice*, int); +typedef NVNtextureHandle (*NVNT_nvnDeviceGetTextureHandle)(const NVNdevice*, int, int); +typedef NVNtextureHandle (*NVNT_nvnDeviceGetTexelFetchHandle)(const NVNdevice*, int); +typedef NVNimageHandle (*NVNT_nvnDeviceGetImageHandle)(const NVNdevice*, int); +typedef void (*NVNT_nvnDeviceInstallDebugCallback)(NVNdevice*, const PFNNVNDEBUGCALLBACKPROC, void*, uint32_t); +typedef NVNdebugDomainId (*NVNT_nvnDeviceGenerateDebugDomainId)(const NVNdevice*, const char*); +typedef void (*NVNT_nvnDeviceSetWindowOriginMode)(NVNdevice*, NVNwindowOriginMode); +typedef void (*NVNT_nvnDeviceSetDepthMode)(NVNdevice*, NVNdepthMode); +typedef NVNboolean (*NVNT_nvnDeviceRegisterFastClearColor)(NVNdevice*, const float*, NVNformat); +typedef NVNboolean (*NVNT_nvnDeviceRegisterFastClearColori)(NVNdevice*, const int*, NVNformat); +typedef NVNboolean (*NVNT_nvnDeviceRegisterFastClearColorui)(NVNdevice*, const uint32_t*, NVNformat); +typedef NVNboolean (*NVNT_nvnDeviceRegisterFastClearDepth)(NVNdevice*, float); +typedef NVNwindowOriginMode (*NVNT_nvnDeviceGetWindowOriginMode)(const NVNdevice*); +typedef NVNdepthMode (*NVNT_nvnDeviceGetDepthMode)(const NVNdevice*); +typedef uint64_t (*NVNT_nvnDeviceGetTimestampInNanoseconds)(const NVNdevice*, const NVNcounterData*); +typedef void (*NVNT_nvnDeviceApplyDeferredFinalizes)(NVNdevice*, int); +typedef void (*NVNT_nvnDeviceFinalizeCommandHandle)(NVNdevice*, NVNcommandHandle); +typedef void (*NVNT_nvnDeviceWalkDebugDatabase)(const NVNdevice*, NVNdebugObjectType, PFNNVNWALKDEBUGDATABASECALLBACKPROC, void*); +typedef NVNseparateTextureHandle (*NVNT_nvnDeviceGetSeparateTextureHandle)(const NVNdevice*, int); +typedef NVNseparateSamplerHandle (*NVNT_nvnDeviceGetSeparateSamplerHandle)(const NVNdevice*, int); +typedef NVNboolean (*NVNT_nvnDeviceIsExternalDebuggerAttached)(const NVNdevice*); +typedef NVNqueueGetErrorResult (*NVNT_nvnQueueGetError)(NVNqueue*, NVNqueueErrorInfo*); +typedef size_t (*NVNT_nvnQueueGetTotalCommandMemoryUsed)(NVNqueue*); +typedef size_t (*NVNT_nvnQueueGetTotalControlMemoryUsed)(NVNqueue*); +typedef size_t (*NVNT_nvnQueueGetTotalComputeMemoryUsed)(NVNqueue*); +typedef void (*NVNT_nvnQueueResetMemoryUsageCounts)(NVNqueue*); +typedef void (*NVNT_nvnQueueBuilderSetDevice)(NVNqueueBuilder*, NVNdevice*); +typedef void (*NVNT_nvnQueueBuilderSetDefaults)(NVNqueueBuilder*); +typedef void (*NVNT_nvnQueueBuilderSetFlags)(NVNqueueBuilder*, int); +typedef void (*NVNT_nvnQueueBuilderSetCommandMemorySize)(NVNqueueBuilder*, size_t); +typedef void (*NVNT_nvnQueueBuilderSetComputeMemorySize)(NVNqueueBuilder*, size_t); +typedef void (*NVNT_nvnQueueBuilderSetControlMemorySize)(NVNqueueBuilder*, size_t); +typedef size_t (*NVNT_nvnQueueBuilderGetQueueMemorySize)(const NVNqueueBuilder*); +typedef void (*NVNT_nvnQueueBuilderSetQueueMemory)(NVNqueueBuilder*, void*, size_t); +typedef void (*NVNT_nvnQueueBuilderSetCommandFlushThreshold)(NVNqueueBuilder*, size_t); +typedef NVNboolean (*NVNT_nvnQueueInitialize)(NVNqueue*, const NVNqueueBuilder*); +typedef void (*NVNT_nvnQueueFinalize)(NVNqueue*); +typedef void (*NVNT_nvnQueueSetDebugLabel)(NVNqueue*, const char*); +typedef void (*NVNT_nvnQueueSubmitCommands)(NVNqueue*, int, const NVNcommandHandle*); +typedef void (*NVNT_nvnQueueFlush)(NVNqueue*); +typedef void (*NVNT_nvnQueueFinish)(NVNqueue*); +typedef void (*NVNT_nvnQueuePresentTexture)(NVNqueue*, NVNwindow*, int); +typedef NVNqueueAcquireTextureResult (*NVNT_nvnQueueAcquireTexture)(NVNqueue*, NVNwindow*, int*); +typedef void (*NVNT_nvnWindowBuilderSetDevice)(NVNwindowBuilder*, NVNdevice*); +typedef void (*NVNT_nvnWindowBuilderSetDefaults)(NVNwindowBuilder*); +typedef void (*NVNT_nvnWindowBuilderSetNativeWindow)(NVNwindowBuilder*, NVNnativeWindow); +typedef void (*NVNT_nvnWindowBuilderSetTextures)(NVNwindowBuilder*, int, NVNtexture* const*); +typedef void (*NVNT_nvnWindowBuilderSetPresentInterval)(NVNwindowBuilder*, int); +typedef NVNnativeWindow (*NVNT_nvnWindowBuilderGetNativeWindow)(const NVNwindowBuilder*); +typedef int (*NVNT_nvnWindowBuilderGetPresentInterval)(const NVNwindowBuilder*); +typedef NVNboolean (*NVNT_nvnWindowInitialize)(NVNwindow*, const NVNwindowBuilder*); +typedef void (*NVNT_nvnWindowFinalize)(NVNwindow*); +typedef void (*NVNT_nvnWindowSetDebugLabel)(NVNwindow*, const char*); +typedef NVNwindowAcquireTextureResult (*NVNT_nvnWindowAcquireTexture)(NVNwindow*, NVNsync*, int*); +typedef NVNnativeWindow (*NVNT_nvnWindowGetNativeWindow)(const NVNwindow*); +typedef int (*NVNT_nvnWindowGetPresentInterval)(const NVNwindow*); +typedef void (*NVNT_nvnWindowSetPresentInterval)(NVNwindow*, int); +typedef void (*NVNT_nvnWindowSetCrop)(NVNwindow*, int, int, int, int); +typedef void (*NVNT_nvnWindowGetCrop)(const NVNwindow*, NVNrectangle*); +typedef NVNboolean (*NVNT_nvnProgramInitialize)(NVNprogram*, NVNdevice*); +typedef void (*NVNT_nvnProgramFinalize)(NVNprogram*); +typedef void (*NVNT_nvnProgramSetDebugLabel)(NVNprogram*, const char*); +typedef NVNboolean (*NVNT_nvnProgramSetShaders)(NVNprogram*, int, const NVNshaderData*); +typedef void (*NVNT_nvnMemoryPoolBuilderSetDevice)(NVNmemoryPoolBuilder*, NVNdevice*); +typedef void (*NVNT_nvnMemoryPoolBuilderSetDefaults)(NVNmemoryPoolBuilder*); +typedef void (*NVNT_nvnMemoryPoolBuilderSetStorage)(NVNmemoryPoolBuilder*, void*, size_t); +typedef void (*NVNT_nvnMemoryPoolBuilderSetFlags)(NVNmemoryPoolBuilder*, int); +typedef void (*NVNT_nvnMemoryPoolBuilderGetMemory)(const NVNmemoryPoolBuilder*); +typedef size_t (*NVNT_nvnMemoryPoolBuilderGetSize)(const NVNmemoryPoolBuilder*); +typedef NVNmemoryPoolFlags (*NVNT_nvnMemoryPoolBuilderGetFlags)(const NVNmemoryPoolBuilder*); +typedef NVNboolean (*NVNT_nvnMemoryPoolInitialize)(NVNmemoryPool*, const NVNmemoryPoolBuilder*); +typedef void (*NVNT_nvnMemoryPoolSetDebugLabel)(NVNmemoryPool*, const char*); +typedef void (*NVNT_nvnMemoryPoolFinalize)(NVNmemoryPool*); +typedef void* (*NVNT_nvnMemoryPoolMap)(const NVNmemoryPool*); +typedef void (*NVNT_nvnMemoryPoolFlushMappedRange)(const NVNmemoryPool*, ptrdiff_t, size_t); +typedef void (*NVNT_nvnMemoryPoolInvalidateMappedRange)(const NVNmemoryPool*, ptrdiff_t, size_t); +typedef NVNbufferAddress (*NVNT_nvnMemoryPoolGetBufferAddress)(const NVNmemoryPool*); +typedef NVNboolean (*NVNT_nvnMemoryPoolMapVirtual)(NVNmemoryPool*, int, const NVNmappingRequest*); +typedef size_t (*NVNT_nvnMemoryPoolGetSize)(const NVNmemoryPool*); +typedef NVNmemoryPoolFlags (*NVNT_nvnMemoryPoolGetFlags)(const NVNmemoryPool*); +typedef NVNboolean (*NVNT_nvnTexturePoolInitialize)(NVNtexturePool*, const NVNmemoryPool*, ptrdiff_t, int); +typedef void (*NVNT_nvnTexturePoolSetDebugLabel)(NVNtexturePool*, const char*); +typedef void (*NVNT_nvnTexturePoolFinalize)(NVNtexturePool*); +typedef void (*NVNT_nvnTexturePoolRegisterTexture)(const NVNtexturePool*, int, const NVNtexture*, const NVNtextureView*); +typedef void (*NVNT_nvnTexturePoolRegisterImage)(const NVNtexturePool*, int, const NVNtexture*, const NVNtextureView*); +typedef const NVNmemoryPool* (*NVNT_nvnTexturePoolGetMemoryPool)(const NVNtexturePool*); +typedef ptrdiff_t (*NVNT_nvnTexturePoolGetMemoryOffset)(const NVNtexturePool*); +typedef int (*NVNT_nvnTexturePoolGetSize)(const NVNtexturePool*); +typedef NVNboolean (*NVNT_nvnSamplerPoolInitialize)(NVNsamplerPool*, const NVNmemoryPool*, ptrdiff_t, int); +typedef void (*NVNT_nvnSamplerPoolSetDebugLabel)(NVNsamplerPool*, const char*); +typedef void (*NVNT_nvnSamplerPoolFinalize)(NVNsamplerPool*); +typedef void (*NVNT_nvnSamplerPoolRegisterSampler)(const NVNsamplerPool*, int, const NVNsampler*); +typedef void (*NVNT_nvnSamplerPoolRegisterSamplerBuilder)(const NVNsamplerPool*, int, const NVNsamplerBuilder*); +typedef const NVNmemoryPool* (*NVNT_nvnSamplerPoolGetMemoryPool)(const NVNsamplerPool*); +typedef ptrdiff_t (*NVNT_nvnSamplerPoolGetMemoryOffset)(const NVNsamplerPool*); +typedef int (*NVNT_nvnSamplerPoolGetSize)(const NVNsamplerPool*); +typedef void (*NVNT_nvnBufferBuilderSetDevice)(NVNbufferBuilder*, NVNdevice*); +typedef void (*NVNT_nvnBufferBuilderSetDefaults)(NVNbufferBuilder*); +typedef void (*NVNT_nvnBufferBuilderSetStorage)(NVNbufferBuilder*, NVNmemoryPool*, ptrdiff_t, size_t); +typedef NVNmemoryPool (*NVNT_nvnBufferBuilderGetMemoryPool)(const NVNbufferBuilder*); +typedef ptrdiff_t (*NVNT_nvnBufferBuilderGetMemoryOffset)(const NVNbufferBuilder*); +typedef size_t (*NVNT_nvnBufferBuilderGetSize)(const NVNbufferBuilder*); +typedef NVNboolean (*NVNT_nvnBufferInitialize)(NVNbuffer*, const NVNbufferBuilder*); +typedef void (*NVNT_nvnBufferSetDebugLabel)(NVNbuffer*, const char*); +typedef void (*NVNT_nvnBufferFinalize)(NVNbuffer*); +typedef void* (*NVNT_nvnBufferMap)(const NVNbuffer*); +typedef NVNbufferAddress (*NVNT_nvnBufferGetAddress)(const NVNbuffer*); +typedef void (*NVNT_nvnBufferFlushMappedRange)(const NVNbuffer*, ptrdiff_t, size_t); +typedef void (*NVNT_nvnBufferInvalidateMappedRange)(const NVNbuffer*, ptrdiff_t, size_t); +typedef NVNmemoryPool* (*NVNT_nvnBufferGetMemoryPool)(const NVNbuffer*); +typedef ptrdiff_t (*NVNT_nvnBufferGetMemoryOffset)(const NVNbuffer*); +typedef size_t (*NVNT_nvnBufferGetSize)(const NVNbuffer*); +typedef uint64_t (*NVNT_nvnBufferGetDebugID)(const NVNbuffer*); +typedef void (*NVNT_nvnTextureBuilderSetDevice)(NVNtextureBuilder*, NVNdevice*); +typedef void (*NVNT_nvnTextureBuilderSetDefaults)(NVNtextureBuilder*); +typedef void (*NVNT_nvnTextureBuilderSetFlags)(NVNtextureBuilder*, int); +typedef void (*NVNT_nvnTextureBuilderSetTarget)(NVNtextureBuilder*, NVNtextureTarget); +typedef void (*NVNT_nvnTextureBuilderSetWidth)(NVNtextureBuilder*, int); +typedef void (*NVNT_nvnTextureBuilderSetHeight)(NVNtextureBuilder*, int); +typedef void (*NVNT_nvnTextureBuilderSetDepth)(NVNtextureBuilder*, int); +typedef void (*NVNT_nvnTextureBuilderSetSize1D)(NVNtextureBuilder*, int); +typedef void (*NVNT_nvnTextureBuilderSetSize2D)(NVNtextureBuilder*, int, int); +typedef void (*NVNT_nvnTextureBuilderSetSize3D)(NVNtextureBuilder*, int, int, int); +typedef void (*NVNT_nvnTextureBuilderSetLevels)(NVNtextureBuilder*, int); +typedef void (*NVNT_nvnTextureBuilderSetFormat)(NVNtextureBuilder*, NVNformat); +typedef void (*NVNT_nvnTextureBuilderSetSamples)(NVNtextureBuilder*, int); +typedef void (*NVNT_nvnTextureBuilderSetSwizzle)(NVNtextureBuilder*, NVNtextureSwizzle, NVNtextureSwizzle, NVNtextureSwizzle, NVNtextureSwizzle); +typedef void (*NVNT_nvnTextureBuilderSetDepthStencilMode)(NVNtextureBuilder*, NVNtextureDepthStencilMode); +typedef size_t (*NVNT_nvnTextureBuilderGetStorageSize)(const NVNtextureBuilder*); +typedef size_t (*NVNT_nvnTextureBuilderGetStorageAlignment)(const NVNtextureBuilder*); +typedef void (*NVNT_nvnTextureBuilderSetStorage)(NVNtextureBuilder*, NVNmemoryPool*, ptrdiff_t); +typedef void (*NVNT_nvnTextureBuilderSetPackagedTextureData)(NVNtextureBuilder*, const void*); +typedef void (*NVNT_nvnTextureBuilderSetPackagedTextureLayout)(NVNtextureBuilder*, const NVNpackagedTextureLayout*); +typedef void (*NVNT_nvnTextureBuilderSetStride)(NVNtextureBuilder*, ptrdiff_t); +typedef void (*NVNT_nvnTextureBuilderSetGLTextureName)(NVNtextureBuilder*, uint32_t); +typedef NVNstorageClass (*NVNT_nvnTextureBuilderGetStorageClass)(const NVNtextureBuilder*); +typedef NVNtextureFlags (*NVNT_nvnTextureBuilderGetFlags)(const NVNtextureBuilder*); +typedef NVNtextureTarget (*NVNT_nvnTextureBuilderGetTarget)(const NVNtextureBuilder*); +typedef int (*NVNT_nvnTextureBuilderGetWidth)(const NVNtextureBuilder*); +typedef int (*NVNT_nvnTextureBuilderGetHeight)(const NVNtextureBuilder*); +typedef int (*NVNT_nvnTextureBuilderGetDepth)(const NVNtextureBuilder*); +typedef int (*NVNT_nvnTextureBuilderGetLevels)(const NVNtextureBuilder*); +typedef NVNformat (*NVNT_nvnTextureBuilderGetFormat)(const NVNtextureBuilder*); +typedef int (*NVNT_nvnTextureBuilderGetSamples)(const NVNtextureBuilder*); +typedef void (*NVNT_nvnTextureBuilderGetSwizzle)(const NVNtextureBuilder*, NVNtextureSwizzle*, NVNtextureSwizzle*, NVNtextureSwizzle*, NVNtextureSwizzle*); +typedef NVNtextureDepthStencilMode (*NVNT_nvnTextureBuilderGetDepthStencilMode)(const NVNtextureBuilder*); +typedef const void* (*NVNT_nvnTextureBuilderGetPackagedTextureData)(const NVNtextureBuilder*); +typedef ptrdiff_t (*NVNT_nvnTextureBuilderGetStride)(const NVNtextureBuilder*); +typedef void (*NVNT_nvnTextureBuilderGetSparseTileLayout)(const NVNtextureBuilder*, NVNtextureSparseTileLayout*); +typedef uint32_t (*NVNT_nvnTextureBuilderGetGLTextureName)(const NVNtextureBuilder*); +typedef size_t (*NVNT_nvnTextureBuilderGetZCullStorageSize)(const NVNtextureBuilder*); +typedef NVNmemoryPool (*NVNT_nvnTextureBuilderGetMemoryPool)(const NVNtextureBuilder*); +typedef ptrdiff_t (*NVNT_nvnTextureBuilderGetMemoryOffset)(const NVNtextureBuilder*); +typedef void (*NVNT_nvnTextureViewSetDefaults)(NVNtextureView*); +typedef void (*NVNT_nvnTextureViewSetLevels)(NVNtextureView*, int, int); +typedef void (*NVNT_nvnTextureViewSetLayers)(NVNtextureView*, int, int); +typedef void (*NVNT_nvnTextureViewSetFormat)(NVNtextureView*, NVNformat); +typedef void (*NVNT_nvnTextureViewSetSwizzle)(NVNtextureView*, NVNtextureSwizzle, NVNtextureSwizzle, NVNtextureSwizzle, NVNtextureSwizzle); +typedef void (*NVNT_nvnTextureViewSetDepthStencilMode)(NVNtextureView*, NVNtextureDepthStencilMode); +typedef void (*NVNT_nvnTextureViewSetTarget)(NVNtextureView*, NVNtextureTarget); +typedef NVNboolean (*NVNT_nvnTextureViewGetLevels)(const NVNtextureView*, int*, int*); +typedef NVNboolean (*NVNT_nvnTextureViewGetLayers)(const NVNtextureView*, int*, int*); +typedef NVNboolean (*NVNT_nvnTextureViewGetFormat)(const NVNtextureView*, NVNformat*); +typedef NVNboolean (*NVNT_nvnTextureViewGetSwizzle)(const NVNtextureView*, NVNtextureSwizzle*, NVNtextureSwizzle*, NVNtextureSwizzle*, NVNtextureSwizzle*); +typedef NVNboolean (*NVNT_nvnTextureViewGetDepthStencilMode)(const NVNtextureView*, NVNtextureDepthStencilMode*); +typedef NVNboolean (*NVNT_nvnTextureViewGetTarget)(const NVNtextureView*, NVNtextureTarget*); +typedef NVNboolean (*NVNT_nvnTextureViewCompare)(const NVNtextureView*, const NVNtextureView*); +typedef NVNboolean (*NVNT_nvnTextureInitialize)(NVNtexture*, const NVNtextureBuilder*); +typedef size_t (*NVNT_nvnTextureGetZCullStorageSize)(const NVNtexture*); +typedef void (*NVNT_nvnTextureFinalize)(NVNtexture*); +typedef void (*NVNT_nvnTextureSetDebugLabel)(NVNtexture*, const char*); +typedef NVNstorageClass (*NVNT_nvnTextureGetStorageClass)(const NVNtexture*); +typedef ptrdiff_t (*NVNT_nvnTextureGetViewOffset)(const NVNtexture*, const NVNtextureView*); +typedef NVNtextureFlags (*NVNT_nvnTextureGetFlags)(const NVNtexture*); +typedef NVNtextureTarget (*NVNT_nvnTextureGetTarget)(const NVNtexture*); +typedef int (*NVNT_nvnTextureGetWidth)(const NVNtexture*); +typedef int (*NVNT_nvnTextureGetHeight)(const NVNtexture*); +typedef int (*NVNT_nvnTextureGetDepth)(const NVNtexture*); +typedef int (*NVNT_nvnTextureGetLevels)(const NVNtexture*); +typedef NVNformat (*NVNT_nvnTextureGetFormat)(const NVNtexture*); +typedef int (*NVNT_nvnTextureGetSamples)(const NVNtexture*); +typedef void (*NVNT_nvnTextureGetSwizzle)(const NVNtexture*, NVNtextureSwizzle*, NVNtextureSwizzle*, NVNtextureSwizzle*, NVNtextureSwizzle*); +typedef NVNtextureDepthStencilMode (*NVNT_nvnTextureGetDepthStencilMode)(const NVNtexture*); +typedef ptrdiff_t (*NVNT_nvnTextureGetStride)(const NVNtexture*); +typedef NVNtextureAddress (*NVNT_nvnTextureGetTextureAddress)(const NVNtexture*); +typedef void (*NVNT_nvnTextureGetSparseTileLayout)(const NVNtexture*, NVNtextureSparseTileLayout*); +typedef void (*NVNT_nvnTextureWriteTexels)(const NVNtexture*, const NVNtextureView*, const NVNcopyRegion*, const void*); +typedef void (*NVNT_nvnTextureWriteTexelsStrided)(const NVNtexture*, const NVNtextureView*, const NVNcopyRegion*, const void*, ptrdiff_t, ptrdiff_t); +typedef void (*NVNT_nvnTextureReadTexels)(const NVNtexture*, const NVNtextureView*, const NVNcopyRegion*, void*); +typedef void (*NVNT_nvnTextureReadTexelsStrided)(const NVNtexture*, const NVNtextureView*, const NVNcopyRegion*, void*, ptrdiff_t, ptrdiff_t); +typedef void (*NVNT_nvnTextureFlushTexels)(const NVNtexture*, const NVNtextureView*, const NVNcopyRegion*); +typedef void (*NVNT_nvnTextureInvalidateTexels)(const NVNtexture*, const NVNtextureView*, const NVNcopyRegion*); +typedef NVNmemoryPool (*NVNT_nvnTextureGetMemoryPool)(const NVNtexture*); +typedef ptrdiff_t (*NVNT_nvnTextureGetMemoryOffset)(const NVNtexture*); +typedef int (*NVNT_nvnTextureGetStorageSize)(const NVNtexture*); +typedef NVNboolean (*NVNT_nvnTextureCompare)(const NVNtexture*, const NVNtexture*); +typedef uint64_t (*NVNT_nvnTextureGetDebugID)(const NVNtexture*); +typedef void (*NVNT_nvnSamplerBuilderSetDevice)(NVNsamplerBuilder*, NVNdevice*); +typedef void (*NVNT_nvnSamplerBuilderSetDefaults)(NVNsamplerBuilder*); +typedef void (*NVNT_nvnSamplerBuilderSetMinMagFilter)(NVNsamplerBuilder*, NVNminFilter, NVNmagFilter); +typedef void (*NVNT_nvnSamplerBuilderSetWrapMode)(NVNsamplerBuilder*, NVNwrapMode, NVNwrapMode, NVNwrapMode); +typedef void (*NVNT_nvnSamplerBuilderSetLodClamp)(NVNsamplerBuilder*, float, float); +typedef void (*NVNT_nvnSamplerBuilderSetLodBias)(NVNsamplerBuilder*, float); +typedef void (*NVNT_nvnSamplerBuilderSetCompare)(NVNsamplerBuilder*, NVNcompareMode, NVNcompareFunc); +typedef void (*NVNT_nvnSamplerBuilderSetBorderColor)(NVNsamplerBuilder*, const float*); +typedef void (*NVNT_nvnSamplerBuilderSetBorderColori)(NVNsamplerBuilder*, const int*); +typedef void (*NVNT_nvnSamplerBuilderSetBorderColorui)(NVNsamplerBuilder*, const uint32_t*); +typedef void (*NVNT_nvnSamplerBuilderSetMaxAnisotropy)(NVNsamplerBuilder*, float); +typedef void (*NVNT_nvnSamplerBuilderSetReductionFilter)(NVNsamplerBuilder*, NVNsamplerReduction); +typedef void (*NVNT_nvnSamplerBuilderSetLodSnap)(NVNsamplerBuilder*, float); +typedef void (*NVNT_nvnSamplerBuilderGetMinMagFilter)(const NVNsamplerBuilder*, NVNminFilter*, NVNmagFilter*); +typedef void (*NVNT_nvnSamplerBuilderGetWrapMode)(const NVNsamplerBuilder*, NVNwrapMode*, NVNwrapMode*, NVNwrapMode*); +typedef void (*NVNT_nvnSamplerBuilderGetLodClamp)(const NVNsamplerBuilder*, float*, float*); +typedef float (*NVNT_nvnSamplerBuilderGetLodBias)(const NVNsamplerBuilder*); +typedef void (*NVNT_nvnSamplerBuilderGetCompare)(const NVNsamplerBuilder*, NVNcompareMode*, NVNcompareFunc*); +typedef void (*NVNT_nvnSamplerBuilderGetBorderColor)(const NVNsamplerBuilder*, float*); +typedef void (*NVNT_nvnSamplerBuilderGetBorderColori)(const NVNsamplerBuilder*, int*); +typedef void (*NVNT_nvnSamplerBuilderGetBorderColorui)(const NVNsamplerBuilder*, uint32_t*); +typedef float (*NVNT_nvnSamplerBuilderGetMaxAnisotropy)(const NVNsamplerBuilder*); +typedef NVNsamplerReduction (*NVNT_nvnSamplerBuilderGetReductionFilter)(const NVNsamplerBuilder*); +typedef float (*NVNT_nvnSamplerBuilderGetLodSnap)(const NVNsamplerBuilder*); +typedef NVNboolean (*NVNT_nvnSamplerInitialize)(NVNsampler*, const NVNsamplerBuilder*); +typedef void (*NVNT_nvnSamplerFinalize)(NVNsampler*); +typedef void (*NVNT_nvnSamplerSetDebugLabel)(NVNsampler*, const char*); +typedef void (*NVNT_nvnSamplerGetMinMagFilter)(const NVNsampler*, NVNminFilter*, NVNmagFilter*); +typedef void (*NVNT_nvnSamplerGetWrapMode)(const NVNsampler*, NVNwrapMode*, NVNwrapMode*, NVNwrapMode*); +typedef void (*NVNT_nvnSamplerGetLodClamp)(const NVNsampler*, float*, float*); +typedef float (*NVNT_nvnSamplerGetLodBias)(const NVNsampler*); +typedef void (*NVNT_nvnSamplerGetCompare)(const NVNsampler*, NVNcompareMode*, NVNcompareFunc*); +typedef void (*NVNT_nvnSamplerGetBorderColor)(const NVNsampler*, float*); +typedef void (*NVNT_nvnSamplerGetBorderColori)(const NVNsampler*, int*); +typedef void (*NVNT_nvnSamplerGetBorderColorui)(const NVNsampler*, uint32_t*); +typedef float (*NVNT_nvnSamplerGetMaxAnisotropy)(const NVNsampler*); +typedef NVNsamplerReduction (*NVNT_nvnSamplerGetReductionFilter)(const NVNsampler*); +typedef NVNboolean (*NVNT_nvnSamplerCompare)(const NVNsampler*, const NVNsampler*); +typedef uint64_t (*NVNT_nvnSamplerGetDebugID)(const NVNsampler*); +typedef void (*NVNT_nvnBlendStateSetDefaults)(NVNblendState*); +typedef void (*NVNT_nvnBlendStateSetBlendTarget)(NVNblendState*, int); +typedef void (*NVNT_nvnBlendStateSetBlendFunc)(NVNblendState*, NVNblendFunc, NVNblendFunc, NVNblendFunc, NVNblendFunc); +typedef void (*NVNT_nvnBlendStateSetBlendEquation)(NVNblendState*, NVNblendEquation, NVNblendEquation); +typedef void (*NVNT_nvnBlendStateSetAdvancedMode)(NVNblendState*, NVNblendAdvancedMode); +typedef void (*NVNT_nvnBlendStateSetAdvancedOverlap)(NVNblendState*, NVNblendAdvancedOverlap); +typedef void (*NVNT_nvnBlendStateSetAdvancedPremultipliedSrc)(NVNblendState*, uint32_t); +typedef void (*NVNT_nvnBlendStateSetAdvancedNormalizedDst)(NVNblendState*, uint32_t); +typedef int (*NVNT_nvnBlendStateGetBlendTarget)(const NVNblendState*); +typedef void (*NVNT_nvnBlendStateGetBlendFunc)(const NVNblendState*, NVNblendFunc*, NVNblendFunc*, NVNblendFunc*, NVNblendFunc*); +typedef void (*NVNT_nvnBlendStateGetBlendEquation)(const NVNblendState*, NVNblendEquation*, NVNblendEquation*); +typedef NVNblendAdvancedMode (*NVNT_nvnBlendStateGetAdvancedMode)(const NVNblendState*); +typedef NVNblendAdvancedOverlap (*NVNT_nvnBlendStateGetAdvancedOverlap)(const NVNblendState*); +typedef NVNboolean (*NVNT_nvnBlendStateGetAdvancedPremultipliedSrc)(const NVNblendState*); +typedef NVNboolean (*NVNT_nvnBlendStateGetAdvancedNormalizedDst)(const NVNblendState*); +typedef void (*NVNT_nvnColorStateSetDefaults)(NVNcolorState*); +typedef void (*NVNT_nvnColorStateSetBlendEnable)(NVNcolorState*, int, uint32_t); +typedef void (*NVNT_nvnColorStateSetLogicOp)(NVNcolorState*, NVNlogicOp); +typedef void (*NVNT_nvnColorStateSetAlphaTest)(NVNcolorState*, NVNalphaFunc); +typedef NVNboolean (*NVNT_nvnColorStateGetBlendEnable)(const NVNcolorState*, int); +typedef NVNlogicOp (*NVNT_nvnColorStateGetLogicOp)(const NVNcolorState*); +typedef NVNalphaFunc (*NVNT_nvnColorStateGetAlphaTest)(const NVNcolorState*); +typedef void (*NVNT_nvnChannelMaskStateSetDefaults)(NVNchannelMaskState*); +typedef void (*NVNT_nvnChannelMaskStateSetChannelMask)(NVNchannelMaskState*, int, uint32_t, uint32_t, uint32_t, uint32_t); +typedef void (*NVNT_nvnChannelMaskStateGetChannelMask)(const NVNchannelMaskState*, int, NVNboolean*, NVNboolean*, NVNboolean*, NVNboolean*); +typedef void (*NVNT_nvnMultisampleStateSetDefaults)(NVNmultisampleState*); +typedef void (*NVNT_nvnMultisampleStateSetMultisampleEnable)(NVNmultisampleState*, uint32_t); +typedef void (*NVNT_nvnMultisampleStateSetSamples)(NVNmultisampleState*, int); +typedef void (*NVNT_nvnMultisampleStateSetAlphaToCoverageEnable)(NVNmultisampleState*, uint32_t); +typedef void (*NVNT_nvnMultisampleStateSetAlphaToCoverageDither)(NVNmultisampleState*, uint32_t); +typedef NVNboolean (*NVNT_nvnMultisampleStateGetMultisampleEnable)(const NVNmultisampleState*); +typedef int (*NVNT_nvnMultisampleStateGetSamples)(const NVNmultisampleState*); +typedef NVNboolean (*NVNT_nvnMultisampleStateGetAlphaToCoverageEnable)(const NVNmultisampleState*); +typedef NVNboolean (*NVNT_nvnMultisampleStateGetAlphaToCoverageDither)(const NVNmultisampleState*); +typedef void (*NVNT_nvnMultisampleStateSetRasterSamples)(NVNmultisampleState*, int); +typedef int (*NVNT_nvnMultisampleStateGetRasterSamples)(NVNmultisampleState*); +typedef void (*NVNT_nvnMultisampleStateSetCoverageModulationMode)(NVNmultisampleState*, NVNcoverageModulationMode); +typedef NVNcoverageModulationMode (*NVNT_nvnMultisampleStateGetCoverageModulationMode)(const NVNmultisampleState*); +typedef void (*NVNT_nvnMultisampleStateSetCoverageToColorEnable)(NVNmultisampleState*, uint32_t); +typedef NVNboolean (*NVNT_nvnMultisampleStateGetCoverageToColorEnable)(const NVNmultisampleState*); +typedef void (*NVNT_nvnMultisampleStateSetCoverageToColorOutput)(NVNmultisampleState*, int); +typedef int (*NVNT_nvnMultisampleStateGetCoverageToColorOutput)(const NVNmultisampleState*); +typedef void (*NVNT_nvnMultisampleStateSetSampleLocationsEnable)(NVNmultisampleState*, uint32_t); +typedef NVNboolean (*NVNT_nvnMultisampleStateGetSampleLocationsEnable)(const NVNmultisampleState*); +typedef void (*NVNT_nvnMultisampleStateGetSampleLocationsGrid)(NVNmultisampleState*, int*, int*); +typedef void (*NVNT_nvnMultisampleStateSetSampleLocationsGridEnable)(NVNmultisampleState*, uint32_t); +typedef NVNboolean (*NVNT_nvnMultisampleStateGetSampleLocationsGridEnable)(const NVNmultisampleState*); +typedef void (*NVNT_nvnMultisampleStateSetSampleLocations)(NVNmultisampleState*, int, int, const float*); +typedef void (*NVNT_nvnPolygonStateSetDefaults)(NVNpolygonState*); +typedef void (*NVNT_nvnPolygonStateSetCullFace)(NVNpolygonState*, NVNface); +typedef void (*NVNT_nvnPolygonStateSetFrontFace)(NVNpolygonState*, NVNfrontFace); +typedef void (*NVNT_nvnPolygonStateSetPolygonMode)(NVNpolygonState*, NVNpolygonMode); +typedef void (*NVNT_nvnPolygonStateSetPolygonOffsetEnables)(NVNpolygonState*, int); +typedef NVNface (*NVNT_nvnPolygonStateGetCullFace)(const NVNpolygonState*); +typedef NVNfrontFace (*NVNT_nvnPolygonStateGetFrontFace)(const NVNpolygonState*); +typedef NVNpolygonMode (*NVNT_nvnPolygonStateGetPolygonMode)(const NVNpolygonState*); +typedef NVNpolygonOffsetEnable (*NVNT_nvnPolygonStateGetPolygonOffsetEnables)(const NVNpolygonState*); +typedef void (*NVNT_nvnDepthStencilStateSetDefaults)(NVNdepthStencilState*); +typedef void (*NVNT_nvnDepthStencilStateSetDepthTestEnable)(NVNdepthStencilState*, uint32_t); +typedef void (*NVNT_nvnDepthStencilStateSetDepthWriteEnable)(NVNdepthStencilState*, uint32_t); +typedef void (*NVNT_nvnDepthStencilStateSetDepthFunc)(NVNdepthStencilState*, NVNdepthFunc); +typedef void (*NVNT_nvnDepthStencilStateSetStencilTestEnable)(NVNdepthStencilState*, uint32_t); +typedef void (*NVNT_nvnDepthStencilStateSetStencilFunc)(NVNdepthStencilState*, NVNface, NVNstencilFunc); +typedef void (*NVNT_nvnDepthStencilStateSetStencilOp)(NVNdepthStencilState*, NVNface, NVNstencilOp, NVNstencilOp, NVNstencilOp); +typedef NVNboolean (*NVNT_nvnDepthStencilStateGetDepthTestEnable)(const NVNdepthStencilState*); +typedef NVNboolean (*NVNT_nvnDepthStencilStateGetDepthWriteEnable)(const NVNdepthStencilState*); +typedef NVNdepthFunc (*NVNT_nvnDepthStencilStateGetDepthFunc)(const NVNdepthStencilState*); +typedef NVNboolean (*NVNT_nvnDepthStencilStateGetStencilTestEnable)(const NVNdepthStencilState*); +typedef NVNstencilFunc (*NVNT_nvnDepthStencilStateGetStencilFunc)(const NVNdepthStencilState*, NVNface); +typedef void (*NVNT_nvnDepthStencilStateGetStencilOp)(const NVNdepthStencilState*, NVNface, NVNstencilOp*, NVNstencilOp*, NVNstencilOp*); +typedef void (*NVNT_nvnVertexAttribStateSetDefaults)(NVNvertexAttribState*); +typedef void (*NVNT_nvnVertexAttribStateSetFormat)(NVNvertexAttribState*, NVNformat, ptrdiff_t); +typedef void (*NVNT_nvnVertexAttribStateSetStreamIndex)(NVNvertexAttribState*, int); +typedef void (*NVNT_nvnVertexAttribStateGetFormat)(const NVNvertexAttribState*, NVNformat*, ptrdiff_t*); +typedef int (*NVNT_nvnVertexAttribStateGetStreamIndex)(const NVNvertexAttribState*); +typedef void (*NVNT_nvnVertexStreamStateSetDefaults)(NVNvertexStreamState*); +typedef void (*NVNT_nvnVertexStreamStateSetStride)(NVNvertexStreamState*, ptrdiff_t); +typedef void (*NVNT_nvnVertexStreamStateSetDivisor)(NVNvertexStreamState*, int); +typedef ptrdiff_t (*NVNT_nvnVertexStreamStateGetStride)(const NVNvertexStreamState*); +typedef int (*NVNT_nvnVertexStreamStateGetDivisor)(const NVNvertexStreamState*); +typedef NVNboolean (*NVNT_nvnCommandBufferInitialize)(NVNcommandBuffer*, NVNdevice*); +typedef void (*NVNT_nvnCommandBufferFinalize)(NVNcommandBuffer*); +typedef void (*NVNT_nvnCommandBufferSetDebugLabel)(NVNcommandBuffer*, const char*); +typedef void (*NVNT_nvnCommandBufferSetMemoryCallback)(NVNcommandBuffer*, PFNNVNCOMMANDBUFFERMEMORYCALLBACKPROC); +typedef void (*NVNT_nvnCommandBufferSetMemoryCallbackData)(NVNcommandBuffer*, void*); +typedef void (*NVNT_nvnCommandBufferAddCommandMemory)(NVNcommandBuffer*, const NVNmemoryPool*, ptrdiff_t, size_t); +typedef void (*NVNT_nvnCommandBufferAddControlMemory)(NVNcommandBuffer*, void*, size_t); +typedef size_t (*NVNT_nvnCommandBufferGetCommandMemorySize)(const NVNcommandBuffer*); +typedef size_t (*NVNT_nvnCommandBufferGetCommandMemoryUsed)(const NVNcommandBuffer*); +typedef size_t (*NVNT_nvnCommandBufferGetCommandMemoryFree)(const NVNcommandBuffer*); +typedef size_t (*NVNT_nvnCommandBufferGetControlMemorySize)(const NVNcommandBuffer*); +typedef size_t (*NVNT_nvnCommandBufferGetControlMemoryUsed)(const NVNcommandBuffer*); +typedef size_t (*NVNT_nvnCommandBufferGetControlMemoryFree)(const NVNcommandBuffer*); +typedef void (*NVNT_nvnCommandBufferBeginRecording)(NVNcommandBuffer*); +typedef NVNcommandHandle (*NVNT_nvnCommandBufferEndRecording)(NVNcommandBuffer*); +typedef void (*NVNT_nvnCommandBufferCallCommands)(NVNcommandBuffer*, int, const NVNcommandHandle*); +typedef void (*NVNT_nvnCommandBufferCopyCommands)(NVNcommandBuffer*, int, const NVNcommandHandle*); +typedef void (*NVNT_nvnCommandBufferBindBlendState)(NVNcommandBuffer*, const NVNblendState*); +typedef void (*NVNT_nvnCommandBufferBindChannelMaskState)(NVNcommandBuffer*, const NVNchannelMaskState*); +typedef void (*NVNT_nvnCommandBufferBindColorState)(NVNcommandBuffer*, const NVNcolorState*); +typedef void (*NVNT_nvnCommandBufferBindMultisampleState)(NVNcommandBuffer*, const NVNmultisampleState*); +typedef void (*NVNT_nvnCommandBufferBindPolygonState)(NVNcommandBuffer*, const NVNpolygonState*); +typedef void (*NVNT_nvnCommandBufferBindDepthStencilState)(NVNcommandBuffer*, const NVNdepthStencilState*); +typedef void (*NVNT_nvnCommandBufferBindVertexAttribState)(NVNcommandBuffer*, int, const NVNvertexAttribState*); +typedef void (*NVNT_nvnCommandBufferBindVertexStreamState)(NVNcommandBuffer*, int, const NVNvertexStreamState*); +typedef void (*NVNT_nvnCommandBufferBindProgram)(NVNcommandBuffer*, const NVNprogram*, int); +typedef void (*NVNT_nvnCommandBufferBindVertexBuffer)(NVNcommandBuffer*, int, NVNbufferAddress, size_t); +typedef void (*NVNT_nvnCommandBufferBindVertexBuffers)(NVNcommandBuffer*, int, int, const NVNbufferRange*); +typedef void (*NVNT_nvnCommandBufferBindUniformBuffer)(NVNcommandBuffer*, NVNshaderStage, int, NVNbufferAddress, size_t); +typedef void (*NVNT_nvnCommandBufferBindUniformBuffers)(NVNcommandBuffer*, NVNshaderStage, int, int, const NVNbufferRange*); +typedef void (*NVNT_nvnCommandBufferBindTransformFeedbackBuffer)(NVNcommandBuffer*, int, NVNbufferAddress, size_t); +typedef void (*NVNT_nvnCommandBufferBindTransformFeedbackBuffers)(NVNcommandBuffer*, int, int, const NVNbufferRange*); +typedef void (*NVNT_nvnCommandBufferBindStorageBuffer)(NVNcommandBuffer*, NVNshaderStage, int, NVNbufferAddress, size_t); +typedef void (*NVNT_nvnCommandBufferBindStorageBuffers)(NVNcommandBuffer*, NVNshaderStage, int, int, const NVNbufferRange*); +typedef void (*NVNT_nvnCommandBufferBindTexture)(NVNcommandBuffer*, NVNshaderStage, int, NVNtextureHandle); +typedef void (*NVNT_nvnCommandBufferBindTextures)(NVNcommandBuffer*, NVNshaderStage, int, int, const NVNtextureHandle*); +typedef void (*NVNT_nvnCommandBufferBindImage)(NVNcommandBuffer*, NVNshaderStage, int, NVNimageHandle); +typedef void (*NVNT_nvnCommandBufferBindImages)(NVNcommandBuffer*, NVNshaderStage, int, int, const NVNimageHandle*); +typedef void (*NVNT_nvnCommandBufferSetPatchSize)(NVNcommandBuffer*, int); +typedef void (*NVNT_nvnCommandBufferSetInnerTessellationLevels)(NVNcommandBuffer*, const float*); +typedef void (*NVNT_nvnCommandBufferSetOuterTessellationLevels)(NVNcommandBuffer*, const float*); +typedef void (*NVNT_nvnCommandBufferSetPrimitiveRestart)(NVNcommandBuffer*, uint32_t, int); +typedef void (*NVNT_nvnCommandBufferBeginTransformFeedback)(NVNcommandBuffer*, NVNbufferAddress); +typedef void (*NVNT_nvnCommandBufferEndTransformFeedback)(NVNcommandBuffer*, NVNbufferAddress); +typedef void (*NVNT_nvnCommandBufferPauseTransformFeedback)(NVNcommandBuffer*, NVNbufferAddress); +typedef void (*NVNT_nvnCommandBufferResumeTransformFeedback)(NVNcommandBuffer*, NVNbufferAddress); +typedef void (*NVNT_nvnCommandBufferDrawTransformFeedback)(NVNcommandBuffer*, NVNdrawPrimitive, NVNbufferAddress); +typedef void (*NVNT_nvnCommandBufferDrawArrays)(NVNcommandBuffer*, NVNdrawPrimitive, int, int); +typedef void (*NVNT_nvnCommandBufferDrawElements)(NVNcommandBuffer*, NVNdrawPrimitive, NVNindexType, int, NVNbufferAddress); +typedef void (*NVNT_nvnCommandBufferDrawElementsBaseVertex)(NVNcommandBuffer*, NVNdrawPrimitive, NVNindexType, int, NVNbufferAddress, int); +typedef void (*NVNT_nvnCommandBufferDrawArraysInstanced)(NVNcommandBuffer*, NVNdrawPrimitive, int, int, int, int); +typedef void (*NVNT_nvnCommandBufferDrawElementsInstanced)(NVNcommandBuffer*, NVNdrawPrimitive, NVNindexType, int, NVNbufferAddress, int, int, int); +typedef void (*NVNT_nvnCommandBufferDrawArraysIndirect)(NVNcommandBuffer*, NVNdrawPrimitive, NVNbufferAddress); +typedef void (*NVNT_nvnCommandBufferDrawElementsIndirect)(NVNcommandBuffer*, NVNdrawPrimitive, NVNindexType, NVNbufferAddress, NVNbufferAddress); +typedef void (*NVNT_nvnCommandBufferMultiDrawArraysIndirectCount)(NVNcommandBuffer*, NVNdrawPrimitive, NVNbufferAddress, NVNbufferAddress, int, ptrdiff_t); +typedef void (*NVNT_nvnCommandBufferMultiDrawElementsIndirectCount)(NVNcommandBuffer*, NVNdrawPrimitive, NVNindexType, NVNbufferAddress, NVNbufferAddress, NVNbufferAddress, int, ptrdiff_t); +typedef void (*NVNT_nvnCommandBufferClearColor)(NVNcommandBuffer*, int, const float*, int); +typedef void (*NVNT_nvnCommandBufferClearColori)(NVNcommandBuffer*, int, const int*, int); +typedef void (*NVNT_nvnCommandBufferClearColorui)(NVNcommandBuffer*, int, const uint32_t*, int); +typedef void (*NVNT_nvnCommandBufferClearDepthStencil)(NVNcommandBuffer*, float, uint32_t, int, int); +typedef void (*NVNT_nvnCommandBufferDispatchCompute)(NVNcommandBuffer*, int, int, int); +typedef void (*NVNT_nvnCommandBufferDispatchComputeIndirect)(NVNcommandBuffer*, NVNbufferAddress); +typedef void (*NVNT_nvnCommandBufferSetViewport)(NVNcommandBuffer*, int, int, int, int); +typedef void (*NVNT_nvnCommandBufferSetViewports)(NVNcommandBuffer*, int, int, const float*); +typedef void (*NVNT_nvnCommandBufferSetViewportSwizzles)(NVNcommandBuffer*, int, int, const NVNviewportSwizzle*); +typedef void (*NVNT_nvnCommandBufferSetScissor)(NVNcommandBuffer*, int, int, int, int); +typedef void (*NVNT_nvnCommandBufferSetScissors)(NVNcommandBuffer*, int, int, const int*); +typedef void (*NVNT_nvnCommandBufferSetDepthRange)(NVNcommandBuffer*, float, float); +typedef void (*NVNT_nvnCommandBufferSetDepthBounds)(NVNcommandBuffer*, uint32_t, float, float); +typedef void (*NVNT_nvnCommandBufferSetDepthRanges)(NVNcommandBuffer*, int, int, const float*); +typedef void (*NVNT_nvnCommandBufferSetTiledCacheAction)(NVNcommandBuffer*, NVNtiledCacheAction); +typedef void (*NVNT_nvnCommandBufferSetTiledCacheTileSize)(NVNcommandBuffer*, int, int); +typedef void (*NVNT_nvnCommandBufferBindSeparateTexture)(NVNcommandBuffer*, NVNshaderStage, int, NVNseparateTextureHandle); +typedef void (*NVNT_nvnCommandBufferBindSeparateSampler)(NVNcommandBuffer*, NVNshaderStage, int, NVNseparateSamplerHandle); +typedef void (*NVNT_nvnCommandBufferBindSeparateTextures)(NVNcommandBuffer*, NVNshaderStage, int, int, const NVNseparateTextureHandle*); +typedef void (*NVNT_nvnCommandBufferBindSeparateSamplers)(NVNcommandBuffer*, NVNshaderStage, int, int, const NVNseparateSamplerHandle*); +typedef void (*NVNT_nvnCommandBufferSetStencilValueMask)(NVNcommandBuffer*, NVNface, int); +typedef void (*NVNT_nvnCommandBufferSetStencilMask)(NVNcommandBuffer*, NVNface, int); +typedef void (*NVNT_nvnCommandBufferSetStencilRef)(NVNcommandBuffer*, NVNface, int); +typedef void (*NVNT_nvnCommandBufferSetBlendColor)(NVNcommandBuffer*, const float*); +typedef void (*NVNT_nvnCommandBufferSetPointSize)(NVNcommandBuffer*, float); +typedef void (*NVNT_nvnCommandBufferSetLineWidth)(NVNcommandBuffer*, float); +typedef void (*NVNT_nvnCommandBufferSetPolygonOffsetClamp)(NVNcommandBuffer*, float, float, float); +typedef void (*NVNT_nvnCommandBufferSetAlphaRef)(NVNcommandBuffer*, float); +typedef void (*NVNT_nvnCommandBufferSetSampleMask)(NVNcommandBuffer*, int); +typedef void (*NVNT_nvnCommandBufferSetRasterizerDiscard)(NVNcommandBuffer*, uint32_t); +typedef void (*NVNT_nvnCommandBufferSetDepthClamp)(NVNcommandBuffer*, uint32_t); +typedef void (*NVNT_nvnCommandBufferSetConservativeRasterEnable)(NVNcommandBuffer*, uint32_t); +typedef void (*NVNT_nvnCommandBufferSetConservativeRasterDilate)(NVNcommandBuffer*, float); +typedef void (*NVNT_nvnCommandBufferSetSubpixelPrecisionBias)(NVNcommandBuffer*, int, int); +typedef void (*NVNT_nvnCommandBufferCopyBufferToTexture)(NVNcommandBuffer*, NVNbufferAddress, const NVNtexture*, const NVNtextureView*, const NVNcopyRegion*, int); +typedef void (*NVNT_nvnCommandBufferCopyTextureToBuffer)(NVNcommandBuffer*, const NVNtexture*, const NVNtextureView*, const NVNcopyRegion*, NVNbufferAddress, int); +typedef void (*NVNT_nvnCommandBufferCopyTextureToTexture)(NVNcommandBuffer*, const NVNtexture*, const NVNtextureView*, const NVNcopyRegion*, const NVNtexture*, const NVNtextureView*, const NVNcopyRegion*, int); +typedef void (*NVNT_nvnCommandBufferCopyBufferToBuffer)(NVNcommandBuffer*, NVNbufferAddress, NVNbufferAddress, size_t, int); +typedef void (*NVNT_nvnCommandBufferClearBuffer)(NVNcommandBuffer*, NVNbufferAddress, size_t, uint32_t); +typedef void (*NVNT_nvnCommandBufferClearTexture)(NVNcommandBuffer*, const NVNtexture*, const NVNtextureView*, const NVNcopyRegion*, const float*, int); +typedef void (*NVNT_nvnCommandBufferClearTexturei)(NVNcommandBuffer*, const NVNtexture*, const NVNtextureView*, const NVNcopyRegion*, const int*, int); +typedef void (*NVNT_nvnCommandBufferClearTextureui)(NVNcommandBuffer*, const NVNtexture*, const NVNtextureView*, const NVNcopyRegion*, const uint32_t*, int); +typedef void (*NVNT_nvnCommandBufferUpdateUniformBuffer)(NVNcommandBuffer*, NVNbufferAddress, size_t, ptrdiff_t, size_t, const void*); +typedef void (*NVNT_nvnCommandBufferReportCounter)(NVNcommandBuffer*, NVNcounterType, NVNbufferAddress); +typedef void (*NVNT_nvnCommandBufferResetCounter)(NVNcommandBuffer*, NVNcounterType); +typedef void (*NVNT_nvnCommandBufferReportValue)(NVNcommandBuffer*, uint32_t, NVNbufferAddress); +typedef void (*NVNT_nvnCommandBufferSetRenderEnable)(NVNcommandBuffer*, uint32_t); +typedef void (*NVNT_nvnCommandBufferSetRenderEnableConditional)(NVNcommandBuffer*, NVNconditionalRenderMode, NVNbufferAddress); +typedef void (*NVNT_nvnCommandBufferSetRenderTargets)(NVNcommandBuffer*, int, const NVNtexture* const*, const NVNtextureView* const*, const NVNtexture*, const NVNtextureView*); +typedef void (*NVNT_nvnCommandBufferDiscardColor)(NVNcommandBuffer*, int); +typedef void (*NVNT_nvnCommandBufferDiscardDepthStencil)(NVNcommandBuffer*); +typedef void (*NVNT_nvnCommandBufferDownsample)(NVNcommandBuffer*, const NVNtexture*, const NVNtexture*); +typedef void (*NVNT_nvnCommandBufferTiledDownsample)(NVNcommandBuffer*, const NVNtexture*, const NVNtexture*); +typedef void (*NVNT_nvnCommandBufferDownsampleTextureView)(NVNcommandBuffer*, const NVNtexture*, const NVNtextureView*, const NVNtexture*, const NVNtextureView*); +typedef void (*NVNT_nvnCommandBufferTiledDownsampleTextureView)(NVNcommandBuffer*, const NVNtexture*, const NVNtextureView*, const NVNtexture*, const NVNtextureView*); +typedef void (*NVNT_nvnCommandBufferBarrier)(NVNcommandBuffer*, int); +typedef void (*NVNT_nvnCommandBufferWaitSync)(NVNcommandBuffer*, const NVNsync*); +typedef void (*NVNT_nvnCommandBufferFenceSync)(NVNcommandBuffer*, NVNsync*, NVNsyncCondition, int); +typedef void (*NVNT_nvnCommandBufferSetTexturePool)(NVNcommandBuffer*, const NVNtexturePool*); +typedef void (*NVNT_nvnCommandBufferSetSamplerPool)(NVNcommandBuffer*, const NVNsamplerPool*); +typedef void (*NVNT_nvnCommandBufferSetShaderScratchMemory)(NVNcommandBuffer*, const NVNmemoryPool*, ptrdiff_t, size_t); +typedef void (*NVNT_nvnCommandBufferSaveZCullData)(NVNcommandBuffer*, NVNbufferAddress, size_t); +typedef void (*NVNT_nvnCommandBufferRestoreZCullData)(NVNcommandBuffer*, NVNbufferAddress, size_t); +typedef void (*NVNT_nvnCommandBufferSetCopyRowStride)(NVNcommandBuffer*, ptrdiff_t); +typedef void (*NVNT_nvnCommandBufferSetCopyImageStride)(NVNcommandBuffer*, ptrdiff_t); +typedef ptrdiff_t (*NVNT_nvnCommandBufferGetCopyRowStride)(const NVNcommandBuffer*); +typedef ptrdiff_t (*NVNT_nvnCommandBufferGetCopyImageStride)(const NVNcommandBuffer*); +typedef void (*NVNT_nvnCommandBufferDrawTexture)(NVNcommandBuffer*, NVNtextureHandle, const NVNdrawTextureRegion*, const NVNdrawTextureRegion*); +typedef NVNboolean (*NVNT_nvnProgramSetSubroutineLinkage)(NVNprogram*, int, const NVNsubroutineLinkageMapPtr*); +typedef void (*NVNT_nvnCommandBufferSetProgramSubroutines)(NVNcommandBuffer*, NVNprogram*, NVNshaderStage, const int, const int, const int*); +typedef void (*NVNT_nvnCommandBufferBindCoverageModulationTable)(NVNcommandBuffer*, const float*); +typedef void (*NVNT_nvnCommandBufferResolveDepthBuffer)(NVNcommandBuffer*); +typedef void (*NVNT_nvnCommandBufferPushDebugGroupStatic)(NVNcommandBuffer*, uint32_t, const char*); +typedef void (*NVNT_nvnCommandBufferPushDebugGroupDynamic)(NVNcommandBuffer*, uint32_t, const char*); +typedef void (*NVNT_nvnCommandBufferPushDebugGroup)(NVNcommandBuffer*, uint32_t, const char*); +typedef void (*NVNT_nvnCommandBufferPopDebugGroup)(NVNcommandBuffer*); +typedef void (*NVNT_nvnCommandBufferPopDebugGroupId)(NVNcommandBuffer*, uint32_t); +typedef void (*NVNT_nvnCommandBufferInsertDebugMarkerStatic)(NVNcommandBuffer*, uint32_t, const char*); +typedef void (*NVNT_nvnCommandBufferInsertDebugMarkerDynamic)(NVNcommandBuffer*, uint32_t, const char*); +typedef void (*NVNT_nvnCommandBufferInsertDebugMarker)(NVNcommandBuffer*, const char*); +typedef PFNNVNCOMMANDBUFFERMEMORYCALLBACKPROC (*NVNT_nvnCommandBufferGetMemoryCallback)(const NVNcommandBuffer*); +typedef void (*NVNT_nvnCommandBufferGetMemoryCallbackData)(const NVNcommandBuffer*); +typedef NVNboolean (*NVNT_nvnCommandBufferIsRecording)(const NVNcommandBuffer*); +typedef NVNboolean (*NVNT_nvnSyncInitialize)(NVNsync*, NVNdevice*); +typedef void (*NVNT_nvnSyncFinalize)(NVNsync*); +typedef void (*NVNT_nvnSyncSetDebugLabel)(NVNsync*, const char*); +typedef void (*NVNT_nvnQueueFenceSync)(NVNqueue*, NVNsync*, NVNsyncCondition, int); +typedef NVNsyncWaitResult (*NVNT_nvnSyncWait)(const NVNsync*, uint64_t); +typedef NVNboolean (*NVNT_nvnQueueWaitSync)(NVNqueue*, const NVNsync*); +typedef void (*NVNT_nvnEventBuilderSetDefaults)(NVNeventBuilder*); +typedef void (*NVNT_nvnEventBuilderSetStorage)(NVNeventBuilder*, const NVNmemoryPool*, int64_t); +typedef NVNboolean (*NVNT_nvnEventInitialize)(NVNevent*, const NVNeventBuilder*); +typedef void (*NVNT_nvnEventFinalize)(NVNevent*); +typedef uint32_t (*NVNT_nvnEventGetValue)(const NVNevent*); +typedef void (*NVNT_nvnEventSignal)(NVNevent*, NVNeventSignalMode, uint32_t); +typedef void (*NVNT_nvnCommandBufferWaitEvent)(NVNcommandBuffer*, const NVNevent*, NVNeventWaitMode, uint32_t); +typedef void (*NVNT_nvnCommandBufferSignalEvent)(NVNcommandBuffer*, const NVNevent*, NVNeventSignalMode, NVNeventSignalLocation, int, uint32_t); + +// 头文件里没有原型的名字（新版接口）：只能接记名桩 +// nvnDeviceBuilderGetFlags +// nvnQueueBuilderSetQueuePriority +// nvnQueueBuilderGetQueuePriority +// nvnQueueBuilderGetDevice +// nvnQueueBuilderGetFlags +// nvnQueueBuilderGetCommandMemorySize +// nvnQueueBuilderGetComputeMemorySize +// nvnQueueBuilderGetControlMemorySize +// nvnQueueBuilderGetCommandFlushThreshold +// nvnQueueBuilderGetMemorySize +// nvnQueueBuilderGetMemory +// nvnWindowBuilderSetNumActiveTextures +// nvnWindowBuilderGetDevice +// nvnWindowBuilderGetNumTextures +// nvnWindowBuilderGetTexture +// nvnWindowBuilderGetNumActiveTextures +// nvnWindowSetNumActiveTextures +// nvnWindowGetNumActiveTextures +// nvnWindowGetNumTextures +// nvnProgramSetShadersExt +// nvnProgramSetSampleShading +// nvnMemoryPoolBuilderGetDevice +// nvnBufferBuilderGetDevice +// nvnTextureBuilderGetDevice +// nvnTextureBuilderGetPackagedTextureLayout +// nvnTextureBuilderGetRawStorageClass +// nvnTextureGetRawStorageClass +// nvnSamplerBuilderGetDevice +// nvnCommandBufferSetCommandMemoryCallbackEnabled +// nvnCommandBufferSetColorReductionEnable +// nvnCommandBufferSetColorReductionThresholds +// nvnSyncInitializeFromFencedGLSync +// nvnSyncCreateGLSync +// nvnEventBuilderGetStorage +// nvnEventBuilderGetMemoryPool +// nvnEventBuilderGetMemoryOffset +// nvnEventGetMemoryPool +// nvnEventGetMemoryOffset +// nvnCommandBufferSetStencilCullCriteria +// nvnCommandBufferInsertCheckpointStatic +// nvnCommandBufferInsertCheckpointDynamic +// nvnDeviceSetAutomaticCheckpointsEnable +// nvnDeviceSetAutomaticCheckpointsSamplingInterval +// nvnDeviceSetAutomaticCheckpointsByDebugGroup +// nvnDeviceSetCheckpointCallStacksEnable +// nvnDeviceSetCheckpointMaxCallStacksDepth +// Names with no prototype in the header (newer interface): can only be wired to named stubs +// nvnDeviceBuilderGetFlags +// nvnQueueBuilderSetQueuePriority +// nvnQueueBuilderGetQueuePriority +// nvnQueueBuilderGetDevice +// nvnQueueBuilderGetFlags +// nvnQueueBuilderGetCommandMemorySize +// nvnQueueBuilderGetComputeMemorySize +// nvnQueueBuilderGetControlMemorySize +// nvnQueueBuilderGetCommandFlushThreshold +// nvnQueueBuilderGetMemorySize +// nvnQueueBuilderGetMemory +// nvnWindowBuilderSetNumActiveTextures +// nvnWindowBuilderGetDevice +// nvnWindowBuilderGetNumTextures +// nvnWindowBuilderGetTexture +// nvnWindowBuilderGetNumActiveTextures +// nvnWindowSetNumActiveTextures +// nvnWindowGetNumActiveTextures +// nvnWindowGetNumTextures +// nvnProgramSetShadersExt +// nvnProgramSetSampleShading +// nvnMemoryPoolBuilderGetDevice +// nvnBufferBuilderGetDevice +// nvnTextureBuilderGetDevice +// nvnTextureBuilderGetPackagedTextureLayout +// nvnTextureBuilderGetRawStorageClass +// nvnTextureGetRawStorageClass +// nvnSamplerBuilderGetDevice +// nvnCommandBufferSetCommandMemoryCallbackEnabled +// nvnCommandBufferSetColorReductionEnable +// nvnCommandBufferSetColorReductionThresholds +// nvnSyncInitializeFromFencedGLSync +// nvnSyncCreateGLSync +// nvnEventBuilderGetStorage +// nvnEventBuilderGetMemoryPool +// nvnEventBuilderGetMemoryOffset +// nvnEventGetMemoryPool +// nvnEventGetMemoryOffset +// nvnCommandBufferSetStencilCullCriteria +// nvnCommandBufferInsertCheckpointStatic +// nvnCommandBufferInsertCheckpointDynamic +// nvnDeviceSetAutomaticCheckpointsEnable +// nvnDeviceSetAutomaticCheckpointsSamplingInterval +// nvnDeviceSetAutomaticCheckpointsByDebugGroup +// nvnDeviceSetCheckpointCallStacksEnable +// nvnDeviceSetCheckpointMaxCallStacksDepth diff --git a/src/gfx/nvn/nvn_registry.c b/src/gfx/nvn/nvn_registry.c new file mode 100644 index 0000000..a40f3d5 --- /dev/null +++ b/src/gfx/nvn/nvn_registry.c @@ -0,0 +1,46 @@ +// [Windows port] The registry of NVN implementations (MacNativeGames' Mach-O sections swnvn / totknvnx) as PE sections, +// and the guest addresses the game receives for NVN names. +#include + +#include "../host/hle.h" +#include "nvn_impl.h" + +// The linker sorts .nvni$a < .nvni$m (NVN / NVN_NOPROTO entries) < .nvni$z by the text after '$'; likewise .nvnx. +__attribute__((section(".nvni$a"), used)) static const nvn_impl impls_start[1]; +__attribute__((section(".nvni$z"), used)) static const nvn_impl impls_stop[1]; +__attribute__((section(".nvnx$a"), used)) static const nvn_impl extras_start[1]; +__attribute__((section(".nvnx$z"), used)) static const nvn_impl extras_stop[1]; + +static const nvn_impl *range(const nvn_impl *a, const nvn_impl *b, unsigned *n) { + __asm__("" : "+r"(a), "+r"(b)); // separate objects: keep the compiler from reasoning about them + const nvn_impl *first = a + 1; + while (first < b && !first->name) first++; // (section padding) + *n = (unsigned)(b - first); + return first; +} +const nvn_impl *nvn_registry_impls(unsigned *n) { return range(impls_start, impls_stop, n); } +const nvn_impl *nvn_registry_extras(unsigned *n) { return range(extras_start, extras_stop, n); } + +// The native implementation of an NVN name (the thunks call it). +void *nvn_impl_fn(const char *name, int extra) { + unsigned n; + const nvn_impl *t = extra ? nvn_registry_extras(&n) : nvn_registry_impls(&n); + for (unsigned i = 0; i < n; i++) if (t[i].name && !strcmp(t[i].name, name)) return t[i].fn; + LOG("NVN implementation %s missing from the registry", name); + runtime_fail("NVN registry"); +} + +extern const struct { const char *name; void (*fn)(Cpu *); } nvn_thunks[]; // nvn_thunks.gen.c +extern const unsigned nvn_thunk_count; + +int nvn_nop_name(const char *name); // hle_nvn.c g_nops +static void nvn_nop_thunk(Cpu *c) { (void)c; } + +// The guest address for an NVN name: its thunk's, an empty function for the GPU-only commands the null device ignores +// (hle_nvn.c g_nops), or a host stub that stops the game naming it when it is called. +uint64_t nvn_registry_guest_address(const char *name) { + for (unsigned i = 0; i < nvn_thunk_count; i++) + if (!strcmp(nvn_thunks[i].name, name)) return hle_register(nvn_thunks[i].name, nvn_thunks[i].fn); + if (nvn_nop_name(name)) return hle_register(name, nvn_nop_thunk); + return hle_register(name, NULL); +} diff --git a/src/gfx/nvn/nvn_tex.c b/src/gfx/nvn/nvn_tex.c new file mode 100644 index 0000000..952fd8d --- /dev/null +++ b/src/gfx/nvn/nvn_tex.c @@ -0,0 +1,164 @@ +// NVN 贴图的存储尺寸 / 对齐 / 各级偏移（Tegra X1 块线性布局）。 +// 基于 deko3d source/dk_image.cpp（zlib 许可，Copyright (C) 2018-2020 fincs），自动块高按 NVN SDK 的整除规则修正： +// 一个 GOB = 64 字节 × 8 行 = 512 字节；块高（tileH，单位 GOB）按「高度×1.5」挑；各级按块对齐累加。 +// 游戏的预排布贴图（SetPackagedTextureLayout）带 8 字节布局参数：第 0 字节 = 块高 log2（真游戏 16 例对照得出， +// 其中 4 例与自动算法不同），有它时以它为准。 +// Storage size / alignment / per-level offsets of NVN textures (Tegra X1 block-linear layout). +// Based on deko3d source/dk_image.cpp (zlib license, Copyright (C) 2018-2020 fincs), with NVN SDK integer rounding for automatic tile height: +// one GOB = 64 bytes × 8 rows = 512 bytes; the block height (tileH, in GOBs) is picked by 'height×1.5'; levels are accumulated with block alignment. +// The game's packaged textures (SetPackagedTextureLayout) carry 8 bytes of layout parameters: byte 0 = log2 of the block height (derived by comparing 16 cases from the real game, +// 4 of which differ from the automatic algorithm); when present it takes precedence. +#include "nvn_tex.h" +#include +#include +#include + +typedef struct { uint8_t bpb, bw, bh; } fmt_t; // 每块字节、块宽、块高（非压缩格式块宽高 = 1） | Bytes per block, block width, block height (for uncompressed formats block width and height = 1) + +static fmt_t fmt_of(int f) { + static const uint8_t astc[14][2] = {{4,4},{5,4},{5,5},{6,5},{6,6},{8,5},{8,6},{8,8},{10,5},{10,6},{10,8},{10,10},{12,10},{12,12}}; + if (f >= 0x79 && f <= 0x94) { int k = (f - 0x79) % 14; return (fmt_t){16, astc[k][0], astc[k][1]}; } + if (f >= 0x41 && f <= 0x50) { // DXT / RGTC / BPTC：4×4 块 | DXT / RGTC / BPTC: 4×4 blocks + int b8 = (f <= 0x42) || (f == 0x45 || f == 0x46) || (f == 0x49 || f == 0x4A); + return (fmt_t){b8 ? 8 : 16, 4, 4}; + } + static const struct { uint8_t lo, hi, bpb; } r[] = { + {0x01,0x04,1},{0x05,0x09,2},{0x0A,0x0C,4},{0x0D,0x10,2},{0x11,0x15,4},{0x16,0x18,8}, + {0x19,0x1C,3},{0x1D,0x21,6},{0x22,0x24,12},{0x25,0x28,4},{0x29,0x2D,8},{0x2E,0x30,16}, + {0x31,0x31,1},{0x32,0x32,2},{0x33,0x35,4},{0x36,0x36,8},{0x37,0x38,4},{0x39,0x3C,2},{0x3D,0x40,4}, + {0x51,0x52,1},{0x53,0x54,2},{0x55,0x56,4},{0x57,0x58,2},{0x59,0x5A,4},{0x5B,0x5C,8}, + {0x5D,0x5E,3},{0x5F,0x60,6},{0x61,0x62,12},{0x63,0x64,4},{0x65,0x66,8},{0x67,0x68,16}, + {0x69,0x6C,4},{0x6D,0x70,4},{0x71,0x75,8},{0x76,0x78,16},{0x95,0x98,2},{0x99,0x9C,4}, + }; + for (unsigned i = 0; i < sizeof r / sizeof *r; i++) + if (f >= r[i].lo && f <= r[i].hi) return (fmt_t){r[i].bpb, 1, 1}; + return (fmt_t){0, 1, 1}; +} + +int nvn_fmt_bpb(int f, int *bw, int *bh) { fmt_t t = fmt_of(f); if (bw) *bw = t.bw; if (bh) *bh = t.bh; return t.bpb; } + +static uint32_t mip(uint32_t s, unsigned l) { s >>= l; return s ? s : 1; } +static uint32_t blk(uint32_t s, uint32_t b) { return (s + b - 1) / b; } +static unsigned pick_tile(uint32_t gobs) { return gobs >= 16 ? 4 : gobs >= 8 ? 3 : gobs >= 4 ? 2 : gobs >= 2 ? 1 : 0; } +static unsigned adj_tile(unsigned shift, uint32_t unit, uint32_t dim) { + if (!shift) return 0; + uint32_t x = unit << (shift - 1); + if (x >= dim) while (--shift) { x >>= 1; if (x < dim) break; } + return shift; +} + +// target：NVNtextureTarget；dims 维数、是否分层由 target 决定 +// target: NVNtextureTarget; the number of dims and whether it is layered are determined by target +static void shape(const nvn_texdesc *d, unsigned *dpl, int *layered, uint32_t *w, uint32_t *h, uint32_t *z) { + *w = d->width; *h = d->height; *z = d->depth; *layered = 0; + switch (d->target) { + case 0: *dpl = 1; *h = 1; *z = 1; break; // 1D + case 3: *dpl = 1; *z = d->height; *h = 1; *layered = 1; break; // 1D_ARRAY（层数在 height） | 1D_ARRAY (layer count is in height) + case 2: *dpl = 3; break; // 3D + case 4: case 6: *dpl = 2; *layered = 1; break; // 2D_ARRAY / MS_ARRAY + case 8: *dpl = 2; *z = 6; *layered = 1; break; // CUBEMAP + case 9: *dpl = 2; *z = d->depth * 6; *layered = 1; break; // CUBEMAP_ARRAY + default: *dpl = 2; *z = 1; break; // 2D / MS / RECTANGLE + } + if (*z == 0) *z = 1; +} + +static void samples_xy(int s, uint32_t *sx, uint32_t *sy) { + *sx = *sy = 1; + if (s == 2) *sx = 2; else if (s == 4) *sx = *sy = 2; else if (s == 8) { *sx = 4; *sy = 2; } +} + +static uint64_t level_offset(const nvn_texdesc *d, unsigned level, unsigned dpl, uint32_t w, uint32_t h, uint32_t z, + unsigned tileH, unsigned tileD, fmt_t t) { + uint32_t sx, sy; samples_xy(d->samples, &sx, &sy); + uint64_t off = 0; + for (unsigned i = 0; i < level; i++) { + uint32_t lw = blk(mip(w * sx, i), t.bw); + uint32_t lh = dpl >= 2 ? blk(mip(h * sy, i), t.bh) : 1; + uint32_t ld = dpl >= 3 ? mip(z, i) : 1; + uint32_t lwb = lw * t.bpb; + unsigned th = adj_tile(tileH, 8, lh), td = adj_tile(tileD, 1, ld); + uint32_t wg = (lwb + 63) / 64, hg = (lh + 7) / 8; + uint32_t wt = wg, ht = (hg + (1u << th) - 1) >> th, dt = (ld + (1u << td) - 1) >> td; + off += (uint64_t)wt * ht * dt << (9 + th + td); + } + return off; +} + +/* NVN v55 sdk+0x4429c8 compares the default footprint against one-GOB tiles + for MINIMAL_LAYOUT. Keep the default if equal; otherwise use tile H=D=0. + Packaged layouts override this choice. Layout cannot be switched on live + resources: the startup-only knob is a negative control for fresh sessions. */ +static int minimal_layout_on(void) { + static _Atomic int enabled = -1; + int value = atomic_load(&enabled); + if (value < 0) { const char *e = getenv("TOTK_MINIMAL_LAYOUT"); value = !(e && !strcmp(e, "0")); atomic_store(&enabled, value); } + return value; +} +static uint64_t layer_bytes(const nvn_texdesc *d, unsigned dpl, int layered, + uint32_t w, uint32_t h, uint32_t z, unsigned tileH, unsigned tileD, fmt_t t) { + uint64_t ls = level_offset(d, d->levels ? d->levels : 1, dpl, w, h, z, tileH, tileD, t); + if (layered) { + uint32_t hh = blk(h, t.bh); unsigned th = tileH, td = tileD; + while (th && hh <= (8u << (th - 1))) th--; + while (td && z <= (1u << (td - 1))) td--; + uint64_t align = 1ull << (th + td + 9); + ls = (ls + align - 1) & ~(align - 1); + } + return ls; +} + +void nvn_tex_layout(const nvn_texdesc *d, nvn_texlayout *o) { + fmt_t t = fmt_of(d->format); + unsigned dpl; int layered; uint32_t w, h, z; + shape(d, &dpl, &layered, &w, &h, &z); + uint32_t sx, sy; samples_xy(d->samples, &sx, &sy); + o->bpb = t.bpb; o->tileH = o->tileD = 0; o->layers = layered ? z : 1; + if (!t.bpb) { o->size = 0; o->align = 512; o->layer_size = 0; return; } + if (d->target == 10) { // BUFFER + o->layer_size = o->size = (uint64_t)t.bpb * d->width; + o->align = t.bpb == 12 ? 4 : t.bpb; return; + } + if (nvn_tex_is_linear(d)) { // Pitch storage: explicit stride, otherwise 32-byte (LINEAR) / 128-byte (LINEAR_RENDER_TARGET) alignment + o->align = (d->flags & 0x40) ? 128 : 32; + uint32_t stride = nvn_tex_linear_stride(d); + o->layer_size = o->size = (uint64_t)stride * blk(h, t.bh); + return; + } + unsigned tileH = 0, tileD = 0; + /* NVN truncates height*1.5 to whole GOB rows before choosing the power of + two. Rounding up here changes e.g. 40 BC rows from H2 to H3 even when + the total allocation size is unchanged, so padding becomes black texels. */ + if (dpl == 2) { uint32_t ah = blk(h * sy, t.bh); tileH = d->pkg_block_h ? (unsigned)d->pkg_block_h - 1 : pick_tile((ah + ah / 2) / 8); } + else if (dpl == 3) tileD = pick_tile(z + z / 2); + uint64_t ls = layer_bytes(d, dpl, layered, w, h, z, tileH, tileD, t); + if ((d->flags & 0x200) && !d->pkg_block_h && minimal_layout_on()) { + uint64_t minimal = layer_bytes(d, dpl, layered, w, h, z, 0, 0, t); + if (minimal != ls) { tileH = tileD = 0; ls = minimal; } + } + o->tileH = tileH; o->tileD = tileD; + o->layer_size = ls; + o->size = layered ? ls * z : ls; + o->align = 512; +} + +uint64_t nvn_tex_level_offset(const nvn_texdesc *d, unsigned level) { + fmt_t t = fmt_of(d->format); + unsigned dpl; int layered; uint32_t w, h, z; + shape(d, &dpl, &layered, &w, &h, &z); + if (!t.bpb || nvn_tex_is_linear(d) || d->target == 10) return 0; + nvn_texlayout L; nvn_tex_layout(d, &L); + return level_offset(d, level, dpl, w, h, z, L.tileH, L.tileD, t); +} + +void nvn_tex_level_tiles(const nvn_texdesc *d, unsigned level, unsigned *tileH, unsigned *tileD) { + fmt_t t = fmt_of(d->format); + unsigned dpl; int layered; uint32_t w, h, z; + shape(d, &dpl, &layered, &w, &h, &z); + uint32_t sx, sy; samples_xy(d->samples, &sx, &sy); + nvn_texlayout L; nvn_tex_layout(d, &L); + uint32_t lh = dpl >= 2 ? blk(mip(h * sy, level), t.bh) : 1; + uint32_t ld = dpl >= 3 ? mip(z, level) : 1; + *tileH = adj_tile(L.tileH, 8, lh); + *tileD = adj_tile(L.tileD, 1, ld); +} diff --git a/src/gfx/nvn/nvn_tex.h b/src/gfx/nvn/nvn_tex.h new file mode 100644 index 0000000..8952f9e --- /dev/null +++ b/src/gfx/nvn/nvn_tex.h @@ -0,0 +1,33 @@ +// NVN 贴图存储布局（Tegra X1 块线性），见 nvn_tex.c。 +// NVN texture storage layout (Tegra X1 block-linear), see nvn_tex.c. +#pragma once +#include + +typedef struct { + int target, format, flags, samples; // NVNtextureTarget / NVNformat / NVNtextureFlags / 采样数（0 或 1 = 单采样） | NVNtextureTarget / NVNformat / NVNtextureFlags / sample count (0 or 1 = single-sampled) + uint32_t width, height, depth, levels, stride; + signed int pkg_block_h:8; unsigned int swizzle_r:3,swizzle_g:3,swizzle_b:3,swizzle_a:3,swizzle_pad:12; // 预排布贴图布局参数第 0 字节（块高 log2）+ 1；0 = 按自动算法 | Byte 0 of the packaged texture layout parameters (log2 of block height) + 1; 0 = use the automatic algorithm +} nvn_texdesc; + +typedef struct { + uint64_t size, layer_size; // 总存储字节、每层字节 | Total storage bytes, bytes per layer + uint32_t align, layers, bpb; // 对齐、层数、每块字节 | Alignment, layer count, bytes per block + unsigned tileH, tileD; // 块高 / 块深（log2 GOB 数） | Block height / block depth (log2 of the GOB count) +} nvn_texlayout; + +void nvn_tex_layout(const nvn_texdesc *d, nvn_texlayout *o); +uint64_t nvn_tex_level_offset(const nvn_texdesc *d, unsigned level); +int nvn_fmt_bpb(int format, int *bw, int *bh); // 每块字节（0 = 未知格式），块宽高 | Bytes per block (0 = unknown format), block width and height +// 第 level 级实际使用的块高 / 块深（log2 GOB 数）：贴图级块高按该级行数（块行）收缩，解排布逐级用它 +// Block height / block depth actually used by level `level` (log2 of the GOB count): the per-level block height shrinks with that level's row count (block rows), and deswizzling uses it level by level +void nvn_tex_level_tiles(const nvn_texdesc *d, unsigned level, unsigned *tileH, unsigned *tileD); + +/* LINEAR_RENDER_TARGET (0x40) uses pitch storage even without LINEAR (0x10). + TOTK's CPU-loaded water height/mask textures use that flag alone. */ +static inline int nvn_tex_is_linear(const nvn_texdesc *d) { return (d->flags & 0x50) != 0; } +static inline uint32_t nvn_tex_linear_stride(const nvn_texdesc *d) { + if (d->stride) return d->stride; + int bw, bh; int bpb = nvn_fmt_bpb(d->format, &bw, &bh); + uint32_t align = (d->flags & 0x40) ? 128 : 32; + return (((d->width + bw - 1) / bw) * bpb + align - 1) & ~(align - 1); +} diff --git a/src/gfx/nvn/nvn_thunks.gen.c b/src/gfx/nvn/nvn_thunks.gen.c new file mode 100644 index 0000000..0ffd907 --- /dev/null +++ b/src/gfx/nvn/nvn_thunks.gen.c @@ -0,0 +1,2114 @@ +// Generated by tools/gen_nvn_thunks.py from the NVN implementations' declarations. Do not edit. +#include +#include "include/nvn.h" +#include "../../runtime/cpu.h" +void *nvn_impl_fn(const char *name, int extra); + +static void T_nvnBlendStateGetBlendEquation(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBlendStateGetBlendEquation", 0); + ((void (*)(void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnBlendStateGetBlendFunc(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBlendStateGetBlendFunc", 0); + ((void (*)(void *, void *, void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2], (void *)(uintptr_t)c->x[3], (void *)(uintptr_t)c->x[4]); +} +static void T_nvnBlendStateGetBlendTarget(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBlendStateGetBlendTarget", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnBlendStateSetAdvancedMode(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBlendStateSetAdvancedMode", 0); + ((void (*)(void *, NVNblendAdvancedMode))f)((void *)(uintptr_t)c->x[0], (NVNblendAdvancedMode)c->x[1]); +} +static void T_nvnBlendStateSetAdvancedNormalizedDst(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBlendStateSetAdvancedNormalizedDst", 0); + ((void (*)(void *, uint32_t))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1]); +} +static void T_nvnBlendStateSetAdvancedOverlap(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBlendStateSetAdvancedOverlap", 0); + ((void (*)(void *, NVNblendAdvancedOverlap))f)((void *)(uintptr_t)c->x[0], (NVNblendAdvancedOverlap)c->x[1]); +} +static void T_nvnBlendStateSetAdvancedPremultipliedSrc(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBlendStateSetAdvancedPremultipliedSrc", 0); + ((void (*)(void *, uint32_t))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1]); +} +static void T_nvnBlendStateSetBlendEquation(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBlendStateSetBlendEquation", 0); + ((void (*)(void *, NVNblendEquation, NVNblendEquation))f)((void *)(uintptr_t)c->x[0], (NVNblendEquation)c->x[1], (NVNblendEquation)c->x[2]); +} +static void T_nvnBlendStateSetBlendFunc(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBlendStateSetBlendFunc", 0); + ((void (*)(void *, NVNblendFunc, NVNblendFunc, NVNblendFunc, NVNblendFunc))f)((void *)(uintptr_t)c->x[0], (NVNblendFunc)c->x[1], (NVNblendFunc)c->x[2], (NVNblendFunc)c->x[3], (NVNblendFunc)c->x[4]); +} +static void T_nvnBlendStateSetBlendTarget(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBlendStateSetBlendTarget", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnBlendStateSetDefaults(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBlendStateSetDefaults", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnBufferBuilderSetDefaults(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBufferBuilderSetDefaults", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnBufferBuilderSetDevice(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBufferBuilderSetDevice", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnBufferBuilderSetStorage(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBufferBuilderSetStorage", 0); + ((void (*)(void *, void *, ptrdiff_t, size_t))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (ptrdiff_t)c->x[2], (size_t)c->x[3]); +} +static void T_nvnBufferFinalize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBufferFinalize", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnBufferFlushMappedRange(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBufferFlushMappedRange", 0); + ((void (*)(void *, ptrdiff_t, size_t))f)((void *)(uintptr_t)c->x[0], (ptrdiff_t)c->x[1], (size_t)c->x[2]); +} +static void T_nvnBufferGetAddress(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBufferGetAddress", 0); + c->x[0] = (uint64_t)((NVNbufferAddress (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnBufferGetMemoryOffset(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBufferGetMemoryOffset", 0); + c->x[0] = (uint64_t)((ptrdiff_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnBufferGetMemoryPool(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBufferGetMemoryPool", 0); + c->x[0] = (uint64_t)(uintptr_t)((void * (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnBufferGetSize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBufferGetSize", 0); + c->x[0] = (uint64_t)((size_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnBufferInitialize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBufferInitialize", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnBufferInvalidateMappedRange(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBufferInvalidateMappedRange", 0); + ((void (*)(void *, ptrdiff_t, size_t))f)((void *)(uintptr_t)c->x[0], (ptrdiff_t)c->x[1], (size_t)c->x[2]); +} +static void T_nvnBufferMap(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBufferMap", 0); + c->x[0] = (uint64_t)(uintptr_t)((void * (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnBufferSetDebugLabel(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnBufferSetDebugLabel", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnChannelMaskStateGetChannelMask(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnChannelMaskStateGetChannelMask", 0); + ((void (*)(void *, int, void *, void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (void *)(uintptr_t)c->x[2], (void *)(uintptr_t)c->x[3], (void *)(uintptr_t)c->x[4], (void *)(uintptr_t)c->x[5]); +} +static void T_nvnChannelMaskStateSetChannelMask(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnChannelMaskStateSetChannelMask", 0); + ((void (*)(void *, int, uint32_t, uint32_t, uint32_t, uint32_t))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (uint32_t)c->x[2], (uint32_t)c->x[3], (uint32_t)c->x[4], (uint32_t)c->x[5]); +} +static void T_nvnChannelMaskStateSetDefaults(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnChannelMaskStateSetDefaults", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnColorStateGetAlphaTest(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnColorStateGetAlphaTest", 0); + c->x[0] = (uint64_t)((NVNalphaFunc (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnColorStateGetBlendEnable(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnColorStateGetBlendEnable", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnColorStateGetLogicOp(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnColorStateGetLogicOp", 0); + c->x[0] = (uint64_t)((NVNlogicOp (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnColorStateSetAlphaTest(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnColorStateSetAlphaTest", 0); + ((void (*)(void *, NVNalphaFunc))f)((void *)(uintptr_t)c->x[0], (NVNalphaFunc)c->x[1]); +} +static void T_nvnColorStateSetBlendEnable(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnColorStateSetBlendEnable", 0); + ((void (*)(void *, int, uint32_t))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (uint32_t)c->x[2]); +} +static void T_nvnColorStateSetDefaults(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnColorStateSetDefaults", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnColorStateSetLogicOp(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnColorStateSetLogicOp", 0); + ((void (*)(void *, NVNlogicOp))f)((void *)(uintptr_t)c->x[0], (NVNlogicOp)c->x[1]); +} +static void T_nvnCommandBufferAddCommandMemory(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferAddCommandMemory", 0); + ((void (*)(void *, void *, ptrdiff_t, size_t))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (ptrdiff_t)c->x[2], (size_t)c->x[3]); +} +static void T_nvnCommandBufferAddControlMemory(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferAddControlMemory", 0); + ((void (*)(void *, void *, size_t))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (size_t)c->x[2]); +} +static void T_nvnCommandBufferBarrier(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBarrier", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnCommandBufferBeginRecording(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBeginRecording", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnCommandBufferBindBlendState(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindBlendState", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnCommandBufferBindChannelMaskState(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindChannelMaskState", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnCommandBufferBindColorState(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindColorState", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnCommandBufferBindDepthStencilState(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindDepthStencilState", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnCommandBufferBindImage(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindImage", 0); + ((void (*)(void *, NVNshaderStage, int, NVNimageHandle))f)((void *)(uintptr_t)c->x[0], (NVNshaderStage)c->x[1], (int)c->x[2], (NVNimageHandle)c->x[3]); +} +static void T_nvnCommandBufferBindImages(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindImages", 0); + ((void (*)(void *, NVNshaderStage, int, int, void *))f)((void *)(uintptr_t)c->x[0], (NVNshaderStage)c->x[1], (int)c->x[2], (int)c->x[3], (void *)(uintptr_t)c->x[4]); +} +static void T_nvnCommandBufferBindImagesDeferred(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindImagesDeferred", 1); + ((void (*)(void *, NVNshaderStage, int, int, void *))f)((void *)(uintptr_t)c->x[0], (NVNshaderStage)c->x[1], (int)c->x[2], (int)c->x[3], (void *)(uintptr_t)c->x[4]); +} +static void T_nvnCommandBufferBindMultisampleState(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindMultisampleState", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnCommandBufferBindPolygonState(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindPolygonState", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnCommandBufferBindProgram(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindProgram", 0); + ((void (*)(void *, void *, int))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (int)c->x[2]); +} +static void T_nvnCommandBufferBindSeparateSampler(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindSeparateSampler", 0); + ((void (*)(void *, NVNshaderStage, int, NVNseparateSamplerHandle))f)((void *)(uintptr_t)c->x[0], (NVNshaderStage)c->x[1], (int)c->x[2], ({ NVNseparateSamplerHandle v_; memcpy(&v_, &c->x[3], sizeof v_); v_; })); +} +static void T_nvnCommandBufferBindSeparateSamplers(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindSeparateSamplers", 0); + ((void (*)(void *, NVNshaderStage, int, int, void *))f)((void *)(uintptr_t)c->x[0], (NVNshaderStage)c->x[1], (int)c->x[2], (int)c->x[3], (void *)(uintptr_t)c->x[4]); +} +static void T_nvnCommandBufferBindSeparateSamplersDeferred(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindSeparateSamplersDeferred", 1); + ((void (*)(void *, NVNshaderStage, int, int, void *))f)((void *)(uintptr_t)c->x[0], (NVNshaderStage)c->x[1], (int)c->x[2], (int)c->x[3], (void *)(uintptr_t)c->x[4]); +} +static void T_nvnCommandBufferBindSeparateTexture(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindSeparateTexture", 0); + ((void (*)(void *, NVNshaderStage, int, NVNseparateTextureHandle))f)((void *)(uintptr_t)c->x[0], (NVNshaderStage)c->x[1], (int)c->x[2], ({ NVNseparateTextureHandle v_; memcpy(&v_, &c->x[3], sizeof v_); v_; })); +} +static void T_nvnCommandBufferBindSeparateTextures(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindSeparateTextures", 0); + ((void (*)(void *, NVNshaderStage, int, int, void *))f)((void *)(uintptr_t)c->x[0], (NVNshaderStage)c->x[1], (int)c->x[2], (int)c->x[3], (void *)(uintptr_t)c->x[4]); +} +static void T_nvnCommandBufferBindSeparateTexturesDeferred(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindSeparateTexturesDeferred", 1); + ((void (*)(void *, NVNshaderStage, int, int, void *))f)((void *)(uintptr_t)c->x[0], (NVNshaderStage)c->x[1], (int)c->x[2], (int)c->x[3], (void *)(uintptr_t)c->x[4]); +} +static void T_nvnCommandBufferBindStorageBuffer(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindStorageBuffer", 0); + ((void (*)(void *, NVNshaderStage, int, NVNbufferAddress, size_t))f)((void *)(uintptr_t)c->x[0], (NVNshaderStage)c->x[1], (int)c->x[2], (NVNbufferAddress)c->x[3], (size_t)c->x[4]); +} +static void T_nvnCommandBufferBindStorageBuffers(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindStorageBuffers", 0); + ((void (*)(void *, NVNshaderStage, int, int, void *))f)((void *)(uintptr_t)c->x[0], (NVNshaderStage)c->x[1], (int)c->x[2], (int)c->x[3], (void *)(uintptr_t)c->x[4]); +} +static void T_nvnCommandBufferBindStorageBuffersDeferred(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindStorageBuffersDeferred", 1); + ((void (*)(void *, NVNshaderStage, int, int, void *))f)((void *)(uintptr_t)c->x[0], (NVNshaderStage)c->x[1], (int)c->x[2], (int)c->x[3], (void *)(uintptr_t)c->x[4]); +} +static void T_nvnCommandBufferBindTexture(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindTexture", 0); + ((void (*)(void *, NVNshaderStage, int, NVNtextureHandle))f)((void *)(uintptr_t)c->x[0], (NVNshaderStage)c->x[1], (int)c->x[2], (NVNtextureHandle)c->x[3]); +} +static void T_nvnCommandBufferBindTextures(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindTextures", 0); + ((void (*)(void *, NVNshaderStage, int, int, void *))f)((void *)(uintptr_t)c->x[0], (NVNshaderStage)c->x[1], (int)c->x[2], (int)c->x[3], (void *)(uintptr_t)c->x[4]); +} +static void T_nvnCommandBufferBindTexturesDeferred(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindTexturesDeferred", 1); + ((void (*)(void *, NVNshaderStage, int, int, void *))f)((void *)(uintptr_t)c->x[0], (NVNshaderStage)c->x[1], (int)c->x[2], (int)c->x[3], (void *)(uintptr_t)c->x[4]); +} +static void T_nvnCommandBufferBindUniformBuffer(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindUniformBuffer", 0); + ((void (*)(void *, NVNshaderStage, int, NVNbufferAddress, size_t))f)((void *)(uintptr_t)c->x[0], (NVNshaderStage)c->x[1], (int)c->x[2], (NVNbufferAddress)c->x[3], (size_t)c->x[4]); +} +static void T_nvnCommandBufferBindUniformBuffers(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindUniformBuffers", 0); + ((void (*)(void *, NVNshaderStage, int, int, void *))f)((void *)(uintptr_t)c->x[0], (NVNshaderStage)c->x[1], (int)c->x[2], (int)c->x[3], (void *)(uintptr_t)c->x[4]); +} +static void T_nvnCommandBufferBindUniformBuffersDeferred(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindUniformBuffersDeferred", 1); + ((void (*)(void *, NVNshaderStage, int, int, void *))f)((void *)(uintptr_t)c->x[0], (NVNshaderStage)c->x[1], (int)c->x[2], (int)c->x[3], (void *)(uintptr_t)c->x[4]); +} +static void T_nvnCommandBufferBindVertexAttribState(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindVertexAttribState", 0); + ((void (*)(void *, int, void *))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnCommandBufferBindVertexBuffer(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindVertexBuffer", 0); + ((void (*)(void *, int, NVNbufferAddress, size_t))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (NVNbufferAddress)c->x[2], (size_t)c->x[3]); +} +static void T_nvnCommandBufferBindVertexBuffers(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindVertexBuffers", 0); + ((void (*)(void *, int, int, void *))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (int)c->x[2], (void *)(uintptr_t)c->x[3]); +} +static void T_nvnCommandBufferBindVertexStreamState(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferBindVertexStreamState", 0); + ((void (*)(void *, int, void *))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnCommandBufferCallCommands(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferCallCommands", 0); + ((void (*)(void *, int, void *))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnCommandBufferClearColor(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferClearColor", 0); + ((void (*)(void *, int, void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (void *)(uintptr_t)c->x[2], (int)c->x[3]); +} +static void T_nvnCommandBufferClearDepthStencil(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferClearDepthStencil", 0); + ((void (*)(void *, float, uint32_t, int, int))f)((void *)(uintptr_t)c->x[0], ({ float v_; memcpy(&v_, &c->v[0], sizeof v_); v_; }), (uint32_t)c->x[1], (int)c->x[2], (int)c->x[3]); +} +static void T_nvnCommandBufferClearTexture(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferClearTexture", 0); + ((void (*)(void *, void *, void *, void *, void *, int))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2], (void *)(uintptr_t)c->x[3], (void *)(uintptr_t)c->x[4], (int)c->x[5]); +} +static void T_nvnCommandBufferClearTextureui(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferClearTextureui", 0); + ((void (*)(void *, void *, void *, void *, void *, int))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2], (void *)(uintptr_t)c->x[3], (void *)(uintptr_t)c->x[4], (int)c->x[5]); +} +static void T_nvnCommandBufferCopyBufferToBuffer(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferCopyBufferToBuffer", 0); + ((void (*)(void *, NVNbufferAddress, NVNbufferAddress, size_t, int))f)((void *)(uintptr_t)c->x[0], (NVNbufferAddress)c->x[1], (NVNbufferAddress)c->x[2], (size_t)c->x[3], (int)c->x[4]); +} +static void T_nvnCommandBufferCopyBufferToTexture(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferCopyBufferToTexture", 0); + ((void (*)(void *, NVNbufferAddress, void *, void *, void *, int))f)((void *)(uintptr_t)c->x[0], (NVNbufferAddress)c->x[1], (void *)(uintptr_t)c->x[2], (void *)(uintptr_t)c->x[3], (void *)(uintptr_t)c->x[4], (int)c->x[5]); +} +static void T_nvnCommandBufferCopyCommands(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferCopyCommands", 0); + ((void (*)(void *, int, void *))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnCommandBufferCopyTextureToBuffer(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferCopyTextureToBuffer", 0); + ((void (*)(void *, void *, void *, void *, NVNbufferAddress, int))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2], (void *)(uintptr_t)c->x[3], (NVNbufferAddress)c->x[4], (int)c->x[5]); +} +static void T_nvnCommandBufferCopyTextureToTexture(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferCopyTextureToTexture", 0); + ((void (*)(void *, void *, void *, void *, void *, void *, void *, int))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2], (void *)(uintptr_t)c->x[3], (void *)(uintptr_t)c->x[4], (void *)(uintptr_t)c->x[5], (void *)(uintptr_t)c->x[6], (int)c->x[7]); +} +static void T_nvnCommandBufferDiscardColor(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferDiscardColor", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnCommandBufferDiscardDepthStencil(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferDiscardDepthStencil", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnCommandBufferDispatchCompute(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferDispatchCompute", 0); + ((void (*)(void *, int, int, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (int)c->x[2], (int)c->x[3]); +} +static void T_nvnCommandBufferDrawArrays(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferDrawArrays", 0); + ((void (*)(void *, NVNdrawPrimitive, int, int))f)((void *)(uintptr_t)c->x[0], (NVNdrawPrimitive)c->x[1], (int)c->x[2], (int)c->x[3]); +} +static void T_nvnCommandBufferDrawArraysInstanced(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferDrawArraysInstanced", 0); + ((void (*)(void *, NVNdrawPrimitive, int, int, int, int))f)((void *)(uintptr_t)c->x[0], (NVNdrawPrimitive)c->x[1], (int)c->x[2], (int)c->x[3], (int)c->x[4], (int)c->x[5]); +} +static void T_nvnCommandBufferDrawElements(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferDrawElements", 0); + ((void (*)(void *, NVNdrawPrimitive, NVNindexType, int, NVNbufferAddress))f)((void *)(uintptr_t)c->x[0], (NVNdrawPrimitive)c->x[1], (NVNindexType)c->x[2], (int)c->x[3], (NVNbufferAddress)c->x[4]); +} +static void T_nvnCommandBufferDrawElementsBaseVertex(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferDrawElementsBaseVertex", 0); + ((void (*)(void *, NVNdrawPrimitive, NVNindexType, int, NVNbufferAddress, int))f)((void *)(uintptr_t)c->x[0], (NVNdrawPrimitive)c->x[1], (NVNindexType)c->x[2], (int)c->x[3], (NVNbufferAddress)c->x[4], (int)c->x[5]); +} +static void T_nvnCommandBufferDrawElementsInstanced(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferDrawElementsInstanced", 0); + ((void (*)(void *, NVNdrawPrimitive, NVNindexType, int, NVNbufferAddress, int, int, int))f)((void *)(uintptr_t)c->x[0], (NVNdrawPrimitive)c->x[1], (NVNindexType)c->x[2], (int)c->x[3], (NVNbufferAddress)c->x[4], (int)c->x[5], (int)c->x[6], (int)c->x[7]); +} +static void T_nvnCommandBufferEndRecording(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferEndRecording", 0); + c->x[0] = (uint64_t)((NVNcommandHandle (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnCommandBufferFinalize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferFinalize", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnCommandBufferGetCommandMemoryFree(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferGetCommandMemoryFree", 0); + c->x[0] = (uint64_t)((size_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnCommandBufferGetCommandMemorySize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferGetCommandMemorySize", 0); + c->x[0] = (uint64_t)((size_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnCommandBufferGetCommandMemoryUsed(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferGetCommandMemoryUsed", 0); + c->x[0] = (uint64_t)((size_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnCommandBufferGetControlMemoryFree(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferGetControlMemoryFree", 0); + c->x[0] = (uint64_t)((size_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnCommandBufferGetControlMemorySize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferGetControlMemorySize", 0); + c->x[0] = (uint64_t)((size_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnCommandBufferGetControlMemoryUsed(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferGetControlMemoryUsed", 0); + c->x[0] = (uint64_t)((size_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnCommandBufferGetMemoryCallback(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferGetMemoryCallback", 0); + c->x[0] = (uint64_t)((PFNNVNCOMMANDBUFFERMEMORYCALLBACKPROC (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnCommandBufferGetMemoryCallbackData(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferGetMemoryCallbackData", 0); + c->x[0] = (uint64_t)(uintptr_t)((void * (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnCommandBufferInitialize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferInitialize", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnCommandBufferInsertCheckpointDynamic(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferInsertCheckpointDynamic", 0); + ((void (*)(void *, uint32_t, void *))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnCommandBufferInsertCheckpointStatic(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferInsertCheckpointStatic", 0); + ((void (*)(void *, uint32_t, void *))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnCommandBufferInsertDebugMarker(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferInsertDebugMarker", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnCommandBufferInsertDebugMarkerDynamic(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferInsertDebugMarkerDynamic", 0); + ((void (*)(void *, uint32_t, void *))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnCommandBufferInsertDebugMarkerStatic(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferInsertDebugMarkerStatic", 0); + ((void (*)(void *, uint32_t, void *))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnCommandBufferIsRecording(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferIsRecording", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnCommandBufferOverrideSubtileSize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferOverrideSubtileSize", 1); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnCommandBufferPopDebugGroup(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferPopDebugGroup", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnCommandBufferPopDebugGroupId(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferPopDebugGroupId", 0); + ((void (*)(void *, uint32_t))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1]); +} +static void T_nvnCommandBufferPushDebugGroup(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferPushDebugGroup", 0); + ((void (*)(void *, uint32_t, void *))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnCommandBufferPushDebugGroupDynamic(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferPushDebugGroupDynamic", 0); + ((void (*)(void *, uint32_t, void *))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnCommandBufferPushDebugGroupStatic(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferPushDebugGroupStatic", 0); + ((void (*)(void *, uint32_t, void *))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnCommandBufferReportCounter(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferReportCounter", 0); + ((void (*)(void *, NVNcounterType, NVNbufferAddress))f)((void *)(uintptr_t)c->x[0], (NVNcounterType)c->x[1], (NVNbufferAddress)c->x[2]); +} +static void T_nvnCommandBufferResetCounter(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferResetCounter", 0); + ((void (*)(void *, NVNcounterType))f)((void *)(uintptr_t)c->x[0], (NVNcounterType)c->x[1]); +} +static void T_nvnCommandBufferRestoreZCullData(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferRestoreZCullData", 0); + ((void (*)(void *, NVNbufferAddress, size_t))f)((void *)(uintptr_t)c->x[0], (NVNbufferAddress)c->x[1], (size_t)c->x[2]); +} +static void T_nvnCommandBufferSaveZCullData(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSaveZCullData", 0); + ((void (*)(void *, NVNbufferAddress, size_t))f)((void *)(uintptr_t)c->x[0], (NVNbufferAddress)c->x[1], (size_t)c->x[2]); +} +static void T_nvnCommandBufferSetAlphaRef(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetAlphaRef", 0); + ((void (*)(void *, float))f)((void *)(uintptr_t)c->x[0], ({ float v_; memcpy(&v_, &c->v[0], sizeof v_); v_; })); +} +static void T_nvnCommandBufferSetBlendColor(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetBlendColor", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnCommandBufferSetCommandMemoryCallbackEnabled(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetCommandMemoryCallbackEnabled", 0); + ((void (*)(void *, NVNboolean))f)((void *)(uintptr_t)c->x[0], (NVNboolean)c->x[1]); +} +static void T_nvnCommandBufferSetConservativeRasterDilate(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetConservativeRasterDilate", 0); + ((void (*)(void *, float))f)((void *)(uintptr_t)c->x[0], ({ float v_; memcpy(&v_, &c->v[0], sizeof v_); v_; })); +} +static void T_nvnCommandBufferSetConservativeRasterEnable(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetConservativeRasterEnable", 0); + ((void (*)(void *, uint32_t))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1]); +} +static void T_nvnCommandBufferSetControlMemoryCallbackEnabled(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetControlMemoryCallbackEnabled", 1); + ((void (*)(void *, NVNboolean))f)((void *)(uintptr_t)c->x[0], (NVNboolean)c->x[1]); +} +static void T_nvnCommandBufferSetCopyRowStride(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetCopyRowStride", 0); + ((void (*)(void *, ptrdiff_t))f)((void *)(uintptr_t)c->x[0], (ptrdiff_t)c->x[1]); +} +static void T_nvnCommandBufferSetDebugLabel(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetDebugLabel", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnCommandBufferSetDepthBounds(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetDepthBounds", 0); + ((void (*)(void *, uint32_t, float, float))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1], ({ float v_; memcpy(&v_, &c->v[0], sizeof v_); v_; }), ({ float v_; memcpy(&v_, &c->v[1], sizeof v_); v_; })); +} +static void T_nvnCommandBufferSetDepthClamp(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetDepthClamp", 0); + ((void (*)(void *, uint32_t))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1]); +} +static void T_nvnCommandBufferSetDepthRange(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetDepthRange", 0); + ((void (*)(void *, float, float))f)((void *)(uintptr_t)c->x[0], ({ float v_; memcpy(&v_, &c->v[0], sizeof v_); v_; }), ({ float v_; memcpy(&v_, &c->v[1], sizeof v_); v_; })); +} +static void T_nvnCommandBufferSetDepthRanges(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetDepthRanges", 0); + ((void (*)(void *, int, int, void *))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (int)c->x[2], (void *)(uintptr_t)c->x[3]); +} +static void T_nvnCommandBufferSetLineWidth(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetLineWidth", 0); + ((void (*)(void *, float))f)((void *)(uintptr_t)c->x[0], ({ float v_; memcpy(&v_, &c->v[0], sizeof v_); v_; })); +} +static void T_nvnCommandBufferSetMemoryCallback(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetMemoryCallback", 0); + ((void (*)(void *, PFNNVNCOMMANDBUFFERMEMORYCALLBACKPROC))f)((void *)(uintptr_t)c->x[0], (PFNNVNCOMMANDBUFFERMEMORYCALLBACKPROC)c->x[1]); +} +static void T_nvnCommandBufferSetMemoryCallbackData(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetMemoryCallbackData", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnCommandBufferSetPointSize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetPointSize", 0); + ((void (*)(void *, float))f)((void *)(uintptr_t)c->x[0], ({ float v_; memcpy(&v_, &c->v[0], sizeof v_); v_; })); +} +static void T_nvnCommandBufferSetPolygonOffsetClamp(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetPolygonOffsetClamp", 0); + ((void (*)(void *, float, float, float))f)((void *)(uintptr_t)c->x[0], ({ float v_; memcpy(&v_, &c->v[0], sizeof v_); v_; }), ({ float v_; memcpy(&v_, &c->v[1], sizeof v_); v_; }), ({ float v_; memcpy(&v_, &c->v[2], sizeof v_); v_; })); +} +static void T_nvnCommandBufferSetPrimitiveRestart(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetPrimitiveRestart", 0); + ((void (*)(void *, uint32_t, int))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1], (int)c->x[2]); +} +static void T_nvnCommandBufferSetProvokingVertexMode(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetProvokingVertexMode", 1); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnCommandBufferSetRasterizerDiscard(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetRasterizerDiscard", 0); + ((void (*)(void *, uint32_t))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1]); +} +static void T_nvnCommandBufferSetRenderEnable(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetRenderEnable", 0); + ((void (*)(void *, uint32_t))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1]); +} +static void T_nvnCommandBufferSetRenderEnableConditional(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetRenderEnableConditional", 0); + ((void (*)(void *, NVNconditionalRenderMode, NVNbufferAddress))f)((void *)(uintptr_t)c->x[0], (NVNconditionalRenderMode)c->x[1], (NVNbufferAddress)c->x[2]); +} +static void T_nvnCommandBufferSetRenderTargets(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetRenderTargets", 0); + ((void (*)(void *, int, void *, void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (void *)(uintptr_t)c->x[2], (void *)(uintptr_t)c->x[3], (void *)(uintptr_t)c->x[4], (void *)(uintptr_t)c->x[5]); +} +static void T_nvnCommandBufferSetSampleMask(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetSampleMask", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnCommandBufferSetSamplerPool(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetSamplerPool", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnCommandBufferSetScissor(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetScissor", 0); + ((void (*)(void *, int, int, int, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (int)c->x[2], (int)c->x[3], (int)c->x[4]); +} +static void T_nvnCommandBufferSetScissors(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetScissors", 0); + ((void (*)(void *, int, int, void *))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (int)c->x[2], (void *)(uintptr_t)c->x[3]); +} +static void T_nvnCommandBufferSetStencilMask(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetStencilMask", 0); + ((void (*)(void *, NVNface, int))f)((void *)(uintptr_t)c->x[0], (NVNface)c->x[1], (int)c->x[2]); +} +static void T_nvnCommandBufferSetStencilRef(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetStencilRef", 0); + ((void (*)(void *, NVNface, int))f)((void *)(uintptr_t)c->x[0], (NVNface)c->x[1], (int)c->x[2]); +} +static void T_nvnCommandBufferSetStencilValueMask(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetStencilValueMask", 0); + ((void (*)(void *, NVNface, int))f)((void *)(uintptr_t)c->x[0], (NVNface)c->x[1], (int)c->x[2]); +} +static void T_nvnCommandBufferSetSubpixelPrecisionBias(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetSubpixelPrecisionBias", 0); + ((void (*)(void *, int, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (int)c->x[2]); +} +static void T_nvnCommandBufferSetTexturePool(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetTexturePool", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnCommandBufferSetTiledCacheAction(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetTiledCacheAction", 0); + ((void (*)(void *, NVNtiledCacheAction))f)((void *)(uintptr_t)c->x[0], (NVNtiledCacheAction)c->x[1]); +} +static void T_nvnCommandBufferSetTiledCacheTileSize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetTiledCacheTileSize", 0); + ((void (*)(void *, int, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (int)c->x[2]); +} +static void T_nvnCommandBufferSetViewport(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetViewport", 0); + ((void (*)(void *, int, int, int, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (int)c->x[2], (int)c->x[3], (int)c->x[4]); +} +static void T_nvnCommandBufferSetViewports(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetViewports", 0); + ((void (*)(void *, int, int, void *))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (int)c->x[2], (void *)(uintptr_t)c->x[3]); +} +static void T_nvnCommandBufferSetZCullZF32CompressionEnable(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSetZCullZF32CompressionEnable", 1); + ((void (*)(void *, NVNboolean))f)((void *)(uintptr_t)c->x[0], (NVNboolean)c->x[1]); +} +static void T_nvnCommandBufferSignalEvent(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferSignalEvent", 0); + ((void (*)(void *, void *, NVNeventSignalMode, NVNeventSignalLocation, int, uint32_t))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (NVNeventSignalMode)c->x[2], (NVNeventSignalLocation)c->x[3], (int)c->x[4], (uint32_t)c->x[5]); +} +static void T_nvnCommandBufferUpdateUniformBuffer(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnCommandBufferUpdateUniformBuffer", 0); + ((void (*)(void *, NVNbufferAddress, size_t, ptrdiff_t, size_t, void *))f)((void *)(uintptr_t)c->x[0], (NVNbufferAddress)c->x[1], (size_t)c->x[2], (ptrdiff_t)c->x[3], (size_t)c->x[4], (void *)(uintptr_t)c->x[5]); +} +static void T_nvnDepthStencilStateGetDepthFunc(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDepthStencilStateGetDepthFunc", 0); + c->x[0] = (uint64_t)((NVNdepthFunc (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnDepthStencilStateGetDepthTestEnable(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDepthStencilStateGetDepthTestEnable", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnDepthStencilStateGetDepthWriteEnable(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDepthStencilStateGetDepthWriteEnable", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnDepthStencilStateGetStencilFunc(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDepthStencilStateGetStencilFunc", 0); + c->x[0] = (uint64_t)((NVNstencilFunc (*)(void *, NVNface))f)((void *)(uintptr_t)c->x[0], (NVNface)c->x[1]); +} +static void T_nvnDepthStencilStateGetStencilOp(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDepthStencilStateGetStencilOp", 0); + ((void (*)(void *, NVNface, void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (NVNface)c->x[1], (void *)(uintptr_t)c->x[2], (void *)(uintptr_t)c->x[3], (void *)(uintptr_t)c->x[4]); +} +static void T_nvnDepthStencilStateGetStencilTestEnable(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDepthStencilStateGetStencilTestEnable", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnDepthStencilStateSetDefaults(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDepthStencilStateSetDefaults", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnDepthStencilStateSetDepthFunc(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDepthStencilStateSetDepthFunc", 0); + ((void (*)(void *, NVNdepthFunc))f)((void *)(uintptr_t)c->x[0], (NVNdepthFunc)c->x[1]); +} +static void T_nvnDepthStencilStateSetDepthTestEnable(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDepthStencilStateSetDepthTestEnable", 0); + ((void (*)(void *, uint32_t))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1]); +} +static void T_nvnDepthStencilStateSetDepthWriteEnable(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDepthStencilStateSetDepthWriteEnable", 0); + ((void (*)(void *, uint32_t))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1]); +} +static void T_nvnDepthStencilStateSetStencilFunc(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDepthStencilStateSetStencilFunc", 0); + ((void (*)(void *, NVNface, NVNstencilFunc))f)((void *)(uintptr_t)c->x[0], (NVNface)c->x[1], (NVNstencilFunc)c->x[2]); +} +static void T_nvnDepthStencilStateSetStencilOp(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDepthStencilStateSetStencilOp", 0); + ((void (*)(void *, NVNface, NVNstencilOp, NVNstencilOp, NVNstencilOp))f)((void *)(uintptr_t)c->x[0], (NVNface)c->x[1], (NVNstencilOp)c->x[2], (NVNstencilOp)c->x[3], (NVNstencilOp)c->x[4]); +} +static void T_nvnDepthStencilStateSetStencilTestEnable(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDepthStencilStateSetStencilTestEnable", 0); + ((void (*)(void *, uint32_t))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1]); +} +static void T_nvnDeviceBuilderSetDefaults(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceBuilderSetDefaults", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnDeviceBuilderSetFlags(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceBuilderSetFlags", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnDeviceFinalize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceFinalize", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnDeviceFinalizeCommandHandle(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceFinalizeCommandHandle", 0); + ((void (*)(void *, NVNcommandHandle))f)((void *)(uintptr_t)c->x[0], (NVNcommandHandle)c->x[1]); +} +static void T_nvnDeviceGetCurrentTimestampInNanoseconds(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceGetCurrentTimestampInNanoseconds", 0); + c->x[0] = (uint64_t)((uint64_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnDeviceGetImageHandle(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceGetImageHandle", 0); + c->x[0] = (uint64_t)((NVNimageHandle (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnDeviceGetInteger(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceGetInteger", 0); + ((void (*)(void *, NVNdeviceInfo, void *))f)((void *)(uintptr_t)c->x[0], (NVNdeviceInfo)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnDeviceGetProcAddress(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceGetProcAddress", 0); + c->x[0] = (uint64_t)((PFNNVNGENERICFUNCPTRPROC (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnDeviceGetSeparateSamplerHandle(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceGetSeparateSamplerHandle", 0); + NVNseparateSamplerHandle r_ = ((NVNseparateSamplerHandle (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); uint64_t v_ = 0; memcpy(&v_, &r_, sizeof r_); c->x[0] = v_; +} +static void T_nvnDeviceGetSeparateTextureHandle(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceGetSeparateTextureHandle", 0); + NVNseparateTextureHandle r_ = ((NVNseparateTextureHandle (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); uint64_t v_ = 0; memcpy(&v_, &r_, sizeof r_); c->x[0] = v_; +} +static void T_nvnDeviceGetTexelFetchHandle(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceGetTexelFetchHandle", 0); + c->x[0] = (uint64_t)((NVNtextureHandle (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnDeviceGetTextureHandle(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceGetTextureHandle", 0); + c->x[0] = (uint64_t)((NVNtextureHandle (*)(void *, int, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (int)c->x[2]); +} +static void T_nvnDeviceGetTimestampInNanoseconds(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceGetTimestampInNanoseconds", 0); + c->x[0] = (uint64_t)((uint64_t (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnDeviceInitialize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceInitialize", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnDeviceInstallDebugCallback(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceInstallDebugCallback", 0); + ((void (*)(void *, const PFNNVNDEBUGCALLBACKPROC, void *, uint32_t))f)((void *)(uintptr_t)c->x[0], (const PFNNVNDEBUGCALLBACKPROC)c->x[1], (void *)(uintptr_t)c->x[2], (uint32_t)c->x[3]); +} +static void T_nvnDeviceIsExternalDebuggerAttached(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceIsExternalDebuggerAttached", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnDeviceRegisterFastClearColor(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceRegisterFastClearColor", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *, NVNformat))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (NVNformat)c->x[2]); +} +static void T_nvnDeviceRegisterFastClearColori(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceRegisterFastClearColori", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *, NVNformat))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (NVNformat)c->x[2]); +} +static void T_nvnDeviceRegisterFastClearColorui(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceRegisterFastClearColorui", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *, NVNformat))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (NVNformat)c->x[2]); +} +static void T_nvnDeviceRegisterFastClearDepth(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceRegisterFastClearDepth", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnDeviceSetAutomaticCheckpointsMask(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceSetAutomaticCheckpointsMask", 1); + ((void (*)(void *, uint32_t))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1]); +} +static void T_nvnDeviceSetDebugLabel(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceSetDebugLabel", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnDeviceSetDepthMode(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceSetDepthMode", 0); + ((void (*)(void *, NVNdepthMode))f)((void *)(uintptr_t)c->x[0], (NVNdepthMode)c->x[1]); +} +static void T_nvnDeviceSetWindowOriginMode(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnDeviceSetWindowOriginMode", 0); + ((void (*)(void *, NVNwindowOriginMode))f)((void *)(uintptr_t)c->x[0], (NVNwindowOriginMode)c->x[1]); +} +static void T_nvnEventBuilderSetDefaults(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnEventBuilderSetDefaults", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnEventBuilderSetStorage(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnEventBuilderSetStorage", 0); + ((void (*)(void *, void *, int64_t))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (int64_t)c->x[2]); +} +static void T_nvnEventFinalize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnEventFinalize", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnEventGetValue(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnEventGetValue", 0); + c->x[0] = (uint64_t)((uint32_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnEventInitialize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnEventInitialize", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnEventSignal(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnEventSignal", 0); + ((void (*)(void *, NVNeventSignalMode, uint32_t))f)((void *)(uintptr_t)c->x[0], (NVNeventSignalMode)c->x[1], (uint32_t)c->x[2]); +} +static void T_nvnMemoryPoolBuilderSetDefaults(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMemoryPoolBuilderSetDefaults", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnMemoryPoolBuilderSetDevice(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMemoryPoolBuilderSetDevice", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnMemoryPoolBuilderSetFlags(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMemoryPoolBuilderSetFlags", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnMemoryPoolBuilderSetStorage(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMemoryPoolBuilderSetStorage", 0); + ((void (*)(void *, void *, size_t))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (size_t)c->x[2]); +} +static void T_nvnMemoryPoolFinalize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMemoryPoolFinalize", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnMemoryPoolFlushMappedRange(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMemoryPoolFlushMappedRange", 0); + ((void (*)(void *, ptrdiff_t, size_t))f)((void *)(uintptr_t)c->x[0], (ptrdiff_t)c->x[1], (size_t)c->x[2]); +} +static void T_nvnMemoryPoolGetBufferAddress(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMemoryPoolGetBufferAddress", 0); + c->x[0] = (uint64_t)((NVNbufferAddress (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnMemoryPoolGetFlags(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMemoryPoolGetFlags", 0); + c->x[0] = (uint64_t)((NVNmemoryPoolFlags (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnMemoryPoolGetSize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMemoryPoolGetSize", 0); + c->x[0] = (uint64_t)((size_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnMemoryPoolInitialize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMemoryPoolInitialize", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnMemoryPoolInvalidateMappedRange(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMemoryPoolInvalidateMappedRange", 0); + ((void (*)(void *, ptrdiff_t, size_t))f)((void *)(uintptr_t)c->x[0], (ptrdiff_t)c->x[1], (size_t)c->x[2]); +} +static void T_nvnMemoryPoolMap(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMemoryPoolMap", 0); + c->x[0] = (uint64_t)(uintptr_t)((void * (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnMemoryPoolMapVirtual(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMemoryPoolMapVirtual", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, int, void *))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnMemoryPoolSetDebugLabel(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMemoryPoolSetDebugLabel", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnMultisampleStateGetAlphaToCoverageEnable(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMultisampleStateGetAlphaToCoverageEnable", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnMultisampleStateGetMultisampleEnable(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMultisampleStateGetMultisampleEnable", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnMultisampleStateGetRasterSamples(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMultisampleStateGetRasterSamples", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnMultisampleStateGetSamples(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMultisampleStateGetSamples", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnMultisampleStateSetAlphaToCoverageDither(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMultisampleStateSetAlphaToCoverageDither", 0); + ((void (*)(void *, uint32_t))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1]); +} +static void T_nvnMultisampleStateSetAlphaToCoverageEnable(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMultisampleStateSetAlphaToCoverageEnable", 0); + ((void (*)(void *, uint32_t))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1]); +} +static void T_nvnMultisampleStateSetDefaults(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMultisampleStateSetDefaults", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnMultisampleStateSetMultisampleEnable(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMultisampleStateSetMultisampleEnable", 0); + ((void (*)(void *, uint32_t))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1]); +} +static void T_nvnMultisampleStateSetRasterSamples(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMultisampleStateSetRasterSamples", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnMultisampleStateSetSampleLocations(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMultisampleStateSetSampleLocations", 0); + ((void (*)(void *, int, int, void *))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (int)c->x[2], (void *)(uintptr_t)c->x[3]); +} +static void T_nvnMultisampleStateSetSampleLocationsEnable(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMultisampleStateSetSampleLocationsEnable", 0); + ((void (*)(void *, uint32_t))f)((void *)(uintptr_t)c->x[0], (uint32_t)c->x[1]); +} +static void T_nvnMultisampleStateSetSamples(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnMultisampleStateSetSamples", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnPolygonStateGetCullFace(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnPolygonStateGetCullFace", 0); + c->x[0] = (uint64_t)((NVNface (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnPolygonStateGetFrontFace(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnPolygonStateGetFrontFace", 0); + c->x[0] = (uint64_t)((NVNfrontFace (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnPolygonStateGetPolygonMode(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnPolygonStateGetPolygonMode", 0); + c->x[0] = (uint64_t)((NVNpolygonMode (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnPolygonStateGetPolygonOffsetEnables(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnPolygonStateGetPolygonOffsetEnables", 0); + c->x[0] = (uint64_t)((NVNpolygonOffsetEnable (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnPolygonStateSetCullFace(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnPolygonStateSetCullFace", 0); + ((void (*)(void *, NVNface))f)((void *)(uintptr_t)c->x[0], (NVNface)c->x[1]); +} +static void T_nvnPolygonStateSetDefaults(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnPolygonStateSetDefaults", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnPolygonStateSetFrontFace(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnPolygonStateSetFrontFace", 0); + ((void (*)(void *, NVNfrontFace))f)((void *)(uintptr_t)c->x[0], (NVNfrontFace)c->x[1]); +} +static void T_nvnPolygonStateSetPolygonMode(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnPolygonStateSetPolygonMode", 0); + ((void (*)(void *, NVNpolygonMode))f)((void *)(uintptr_t)c->x[0], (NVNpolygonMode)c->x[1]); +} +static void T_nvnPolygonStateSetPolygonOffsetEnables(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnPolygonStateSetPolygonOffsetEnables", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnProgramFinalize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnProgramFinalize", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnProgramGetSubtileSize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnProgramGetSubtileSize", 1); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnProgramInitialize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnProgramInitialize", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnProgramSetDebugLabel(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnProgramSetDebugLabel", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnProgramSetShaders(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnProgramSetShaders", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, int, void *))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnProgramSetShadersExt(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnProgramSetShadersExt", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, int, void *))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnQueueBuilderGetQueueMemorySize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnQueueBuilderGetQueueMemorySize", 0); + c->x[0] = (uint64_t)((size_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnQueueBuilderSetCommandFlushThreshold(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnQueueBuilderSetCommandFlushThreshold", 0); + ((void (*)(void *, size_t))f)((void *)(uintptr_t)c->x[0], (size_t)c->x[1]); +} +static void T_nvnQueueBuilderSetCommandMemorySize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnQueueBuilderSetCommandMemorySize", 0); + ((void (*)(void *, size_t))f)((void *)(uintptr_t)c->x[0], (size_t)c->x[1]); +} +static void T_nvnQueueBuilderSetComputeMemorySize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnQueueBuilderSetComputeMemorySize", 0); + ((void (*)(void *, size_t))f)((void *)(uintptr_t)c->x[0], (size_t)c->x[1]); +} +static void T_nvnQueueBuilderSetControlMemorySize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnQueueBuilderSetControlMemorySize", 0); + ((void (*)(void *, size_t))f)((void *)(uintptr_t)c->x[0], (size_t)c->x[1]); +} +static void T_nvnQueueBuilderSetDefaults(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnQueueBuilderSetDefaults", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnQueueBuilderSetDevice(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnQueueBuilderSetDevice", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnQueueBuilderSetFlags(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnQueueBuilderSetFlags", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnQueueBuilderSetQueueMemory(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnQueueBuilderSetQueueMemory", 0); + ((void (*)(void *, void *, size_t))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (size_t)c->x[2]); +} +static void T_nvnQueueFenceSync(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnQueueFenceSync", 0); + ((void (*)(void *, void *, NVNsyncCondition, int))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (NVNsyncCondition)c->x[2], (int)c->x[3]); +} +static void T_nvnQueueFinalize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnQueueFinalize", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnQueueFinish(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnQueueFinish", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnQueueFlush(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnQueueFlush", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnQueueGetError(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnQueueGetError", 0); + c->x[0] = (uint64_t)((NVNqueueGetErrorResult (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnQueueInitialize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnQueueInitialize", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnQueuePresentTexture(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnQueuePresentTexture", 0); + ((void (*)(void *, void *, int))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (int)c->x[2]); +} +static void T_nvnQueueSetDebugLabel(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnQueueSetDebugLabel", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnQueueSubmitCommands(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnQueueSubmitCommands", 0); + ((void (*)(void *, int, void *))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnSamplerBuilderGetBorderColor(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderGetBorderColor", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnSamplerBuilderGetBorderColori(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderGetBorderColori", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnSamplerBuilderGetBorderColorui(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderGetBorderColorui", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnSamplerBuilderGetCompare(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderGetCompare", 0); + ((void (*)(void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnSamplerBuilderGetDevice(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderGetDevice", 0); + c->x[0] = (uint64_t)(uintptr_t)((void * (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnSamplerBuilderGetLodBias(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderGetLodBias", 0); + float r_ = ((float (*)(void *))f)((void *)(uintptr_t)c->x[0]); memset(&c->v[0], 0, sizeof c->v[0]); memcpy(&c->v[0], &r_, sizeof r_); +} +static void T_nvnSamplerBuilderGetLodClamp(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderGetLodClamp", 0); + ((void (*)(void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnSamplerBuilderGetLodSnap(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderGetLodSnap", 0); + float r_ = ((float (*)(void *))f)((void *)(uintptr_t)c->x[0]); memset(&c->v[0], 0, sizeof c->v[0]); memcpy(&c->v[0], &r_, sizeof r_); +} +static void T_nvnSamplerBuilderGetMaxAnisotropy(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderGetMaxAnisotropy", 0); + float r_ = ((float (*)(void *))f)((void *)(uintptr_t)c->x[0]); memset(&c->v[0], 0, sizeof c->v[0]); memcpy(&c->v[0], &r_, sizeof r_); +} +static void T_nvnSamplerBuilderGetMinMagFilter(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderGetMinMagFilter", 0); + ((void (*)(void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnSamplerBuilderGetReductionFilter(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderGetReductionFilter", 0); + c->x[0] = (uint64_t)((NVNsamplerReduction (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnSamplerBuilderGetWrapMode(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderGetWrapMode", 0); + ((void (*)(void *, void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2], (void *)(uintptr_t)c->x[3]); +} +static void T_nvnSamplerBuilderSetBorderColor(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderSetBorderColor", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnSamplerBuilderSetBorderColori(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderSetBorderColori", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnSamplerBuilderSetBorderColorui(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderSetBorderColorui", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnSamplerBuilderSetCompare(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderSetCompare", 0); + ((void (*)(void *, NVNcompareMode, NVNcompareFunc))f)((void *)(uintptr_t)c->x[0], (NVNcompareMode)c->x[1], (NVNcompareFunc)c->x[2]); +} +static void T_nvnSamplerBuilderSetDefaults(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderSetDefaults", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnSamplerBuilderSetDevice(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderSetDevice", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnSamplerBuilderSetLodBias(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderSetLodBias", 0); + ((void (*)(void *, float))f)((void *)(uintptr_t)c->x[0], ({ float v_; memcpy(&v_, &c->v[0], sizeof v_); v_; })); +} +static void T_nvnSamplerBuilderSetLodClamp(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderSetLodClamp", 0); + ((void (*)(void *, float, float))f)((void *)(uintptr_t)c->x[0], ({ float v_; memcpy(&v_, &c->v[0], sizeof v_); v_; }), ({ float v_; memcpy(&v_, &c->v[1], sizeof v_); v_; })); +} +static void T_nvnSamplerBuilderSetLodSnap(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderSetLodSnap", 0); + ((void (*)(void *, float))f)((void *)(uintptr_t)c->x[0], ({ float v_; memcpy(&v_, &c->v[0], sizeof v_); v_; })); +} +static void T_nvnSamplerBuilderSetMaxAnisotropy(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderSetMaxAnisotropy", 0); + ((void (*)(void *, float))f)((void *)(uintptr_t)c->x[0], ({ float v_; memcpy(&v_, &c->v[0], sizeof v_); v_; })); +} +static void T_nvnSamplerBuilderSetMinMagFilter(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderSetMinMagFilter", 0); + ((void (*)(void *, NVNminFilter, NVNmagFilter))f)((void *)(uintptr_t)c->x[0], (NVNminFilter)c->x[1], (NVNmagFilter)c->x[2]); +} +static void T_nvnSamplerBuilderSetReductionFilter(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderSetReductionFilter", 0); + ((void (*)(void *, NVNsamplerReduction))f)((void *)(uintptr_t)c->x[0], (NVNsamplerReduction)c->x[1]); +} +static void T_nvnSamplerBuilderSetWrapMode(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerBuilderSetWrapMode", 0); + ((void (*)(void *, NVNwrapMode, NVNwrapMode, NVNwrapMode))f)((void *)(uintptr_t)c->x[0], (NVNwrapMode)c->x[1], (NVNwrapMode)c->x[2], (NVNwrapMode)c->x[3]); +} +static void T_nvnSamplerCompare(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerCompare", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnSamplerFinalize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerFinalize", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnSamplerGetBorderColor(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerGetBorderColor", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnSamplerGetBorderColori(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerGetBorderColori", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnSamplerGetBorderColorui(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerGetBorderColorui", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnSamplerGetCompare(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerGetCompare", 0); + ((void (*)(void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnSamplerGetDebugID(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerGetDebugID", 0); + c->x[0] = (uint64_t)((uint64_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnSamplerGetLodBias(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerGetLodBias", 0); + float r_ = ((float (*)(void *))f)((void *)(uintptr_t)c->x[0]); memset(&c->v[0], 0, sizeof c->v[0]); memcpy(&c->v[0], &r_, sizeof r_); +} +static void T_nvnSamplerGetLodClamp(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerGetLodClamp", 0); + ((void (*)(void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnSamplerGetMaxAnisotropy(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerGetMaxAnisotropy", 0); + float r_ = ((float (*)(void *))f)((void *)(uintptr_t)c->x[0]); memset(&c->v[0], 0, sizeof c->v[0]); memcpy(&c->v[0], &r_, sizeof r_); +} +static void T_nvnSamplerGetMinMagFilter(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerGetMinMagFilter", 0); + ((void (*)(void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnSamplerGetReductionFilter(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerGetReductionFilter", 0); + c->x[0] = (uint64_t)((NVNsamplerReduction (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnSamplerGetWrapMode(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerGetWrapMode", 0); + ((void (*)(void *, void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2], (void *)(uintptr_t)c->x[3]); +} +static void T_nvnSamplerInitialize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerInitialize", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnSamplerPoolFinalize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerPoolFinalize", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnSamplerPoolInitialize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerPoolInitialize", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *, ptrdiff_t, int))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (ptrdiff_t)c->x[2], (int)c->x[3]); +} +static void T_nvnSamplerPoolRegisterSampler(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerPoolRegisterSampler", 0); + ((void (*)(void *, int, void *))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnSamplerPoolSetDebugLabel(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerPoolSetDebugLabel", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnSamplerSetDebugLabel(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSamplerSetDebugLabel", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnSyncFinalize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSyncFinalize", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnSyncInitialize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSyncInitialize", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnSyncSetDebugLabel(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSyncSetDebugLabel", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnSyncWait(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnSyncWait", 0); + c->x[0] = (uint64_t)((NVNsyncWaitResult (*)(void *, uint64_t))f)((void *)(uintptr_t)c->x[0], (uint64_t)c->x[1]); +} +static void T_nvnTextureBuilderGetDepth(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderGetDepth", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureBuilderGetDepthStencilMode(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderGetDepthStencilMode", 0); + c->x[0] = (uint64_t)((NVNtextureDepthStencilMode (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureBuilderGetDevice(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderGetDevice", 0); + c->x[0] = (uint64_t)(uintptr_t)((void * (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureBuilderGetFlags(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderGetFlags", 0); + c->x[0] = (uint64_t)((NVNtextureFlags (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureBuilderGetFormat(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderGetFormat", 0); + c->x[0] = (uint64_t)((NVNformat (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureBuilderGetHeight(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderGetHeight", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureBuilderGetLevels(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderGetLevels", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureBuilderGetMemoryOffset(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderGetMemoryOffset", 0); + c->x[0] = (uint64_t)((ptrdiff_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureBuilderGetMemoryPool(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderGetMemoryPool", 0); + c->x[0] = (uint64_t)(uintptr_t)((void * (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureBuilderGetPackagedTextureData(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderGetPackagedTextureData", 0); + c->x[0] = (uint64_t)(uintptr_t)((void * (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureBuilderGetSamples(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderGetSamples", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureBuilderGetStorageAlignment(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderGetStorageAlignment", 0); + c->x[0] = (uint64_t)((size_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureBuilderGetStorageClass(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderGetStorageClass", 0); + c->x[0] = (uint64_t)((NVNstorageClass (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureBuilderGetStorageSize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderGetStorageSize", 0); + c->x[0] = (uint64_t)((size_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureBuilderGetStride(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderGetStride", 0); + c->x[0] = (uint64_t)((ptrdiff_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureBuilderGetSwizzle(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderGetSwizzle", 0); + ((void (*)(void *, void *, void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2], (void *)(uintptr_t)c->x[3], (void *)(uintptr_t)c->x[4]); +} +static void T_nvnTextureBuilderGetTarget(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderGetTarget", 0); + c->x[0] = (uint64_t)((NVNtextureTarget (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureBuilderGetViewOffset(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderGetViewOffset", 1); + c->x[0] = (uint64_t)((ptrdiff_t (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnTextureBuilderGetWidth(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderGetWidth", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureBuilderGetZCullStorageSize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderGetZCullStorageSize", 0); + c->x[0] = (uint64_t)((size_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureBuilderSetDefaults(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderSetDefaults", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureBuilderSetDepth(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderSetDepth", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnTextureBuilderSetDepthStencilMode(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderSetDepthStencilMode", 0); + ((void (*)(void *, NVNtextureDepthStencilMode))f)((void *)(uintptr_t)c->x[0], (NVNtextureDepthStencilMode)c->x[1]); +} +static void T_nvnTextureBuilderSetDevice(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderSetDevice", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnTextureBuilderSetFlags(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderSetFlags", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnTextureBuilderSetFormat(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderSetFormat", 0); + ((void (*)(void *, NVNformat))f)((void *)(uintptr_t)c->x[0], (NVNformat)c->x[1]); +} +static void T_nvnTextureBuilderSetHeight(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderSetHeight", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnTextureBuilderSetLevels(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderSetLevels", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnTextureBuilderSetPackagedTextureData(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderSetPackagedTextureData", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnTextureBuilderSetPackagedTextureLayout(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderSetPackagedTextureLayout", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnTextureBuilderSetSamples(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderSetSamples", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnTextureBuilderSetSize1D(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderSetSize1D", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnTextureBuilderSetSize2D(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderSetSize2D", 0); + ((void (*)(void *, int, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (int)c->x[2]); +} +static void T_nvnTextureBuilderSetSize3D(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderSetSize3D", 0); + ((void (*)(void *, int, int, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (int)c->x[2], (int)c->x[3]); +} +static void T_nvnTextureBuilderSetStorage(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderSetStorage", 0); + ((void (*)(void *, void *, ptrdiff_t))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (ptrdiff_t)c->x[2]); +} +static void T_nvnTextureBuilderSetStride(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderSetStride", 0); + ((void (*)(void *, ptrdiff_t))f)((void *)(uintptr_t)c->x[0], (ptrdiff_t)c->x[1]); +} +static void T_nvnTextureBuilderSetSwizzle(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderSetSwizzle", 0); + ((void (*)(void *, NVNtextureSwizzle, NVNtextureSwizzle, NVNtextureSwizzle, NVNtextureSwizzle))f)((void *)(uintptr_t)c->x[0], (NVNtextureSwizzle)c->x[1], (NVNtextureSwizzle)c->x[2], (NVNtextureSwizzle)c->x[3], (NVNtextureSwizzle)c->x[4]); +} +static void T_nvnTextureBuilderSetTarget(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderSetTarget", 0); + ((void (*)(void *, NVNtextureTarget))f)((void *)(uintptr_t)c->x[0], (NVNtextureTarget)c->x[1]); +} +static void T_nvnTextureBuilderSetWidth(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureBuilderSetWidth", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnTextureCompare(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureCompare", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnTextureFinalize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureFinalize", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureFlushTexels(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureFlushTexels", 0); + ((void (*)(void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnTextureGetDebugID(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureGetDebugID", 0); + c->x[0] = (uint64_t)((uint64_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureGetDepth(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureGetDepth", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureGetDepthStencilMode(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureGetDepthStencilMode", 0); + c->x[0] = (uint64_t)((NVNtextureDepthStencilMode (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureGetFlags(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureGetFlags", 0); + c->x[0] = (uint64_t)((NVNtextureFlags (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureGetFormat(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureGetFormat", 0); + c->x[0] = (uint64_t)((NVNformat (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureGetHeight(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureGetHeight", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureGetLevels(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureGetLevels", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureGetMemoryOffset(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureGetMemoryOffset", 0); + c->x[0] = (uint64_t)((ptrdiff_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureGetMemoryPool(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureGetMemoryPool", 0); + c->x[0] = (uint64_t)(uintptr_t)((void * (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureGetRawStorageClass(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureGetRawStorageClass", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureGetSamples(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureGetSamples", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureGetStorageClass(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureGetStorageClass", 0); + c->x[0] = (uint64_t)((NVNstorageClass (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureGetStorageSize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureGetStorageSize", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureGetStride(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureGetStride", 0); + c->x[0] = (uint64_t)((ptrdiff_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureGetSwizzle(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureGetSwizzle", 0); + ((void (*)(void *, void *, void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2], (void *)(uintptr_t)c->x[3], (void *)(uintptr_t)c->x[4]); +} +static void T_nvnTextureGetTarget(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureGetTarget", 0); + c->x[0] = (uint64_t)((NVNtextureTarget (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureGetTextureAddress(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureGetTextureAddress", 0); + c->x[0] = (uint64_t)((NVNtextureAddress (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureGetViewOffset(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureGetViewOffset", 0); + c->x[0] = (uint64_t)((ptrdiff_t (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnTextureGetWidth(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureGetWidth", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureGetZCullStorageSize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureGetZCullStorageSize", 0); + c->x[0] = (uint64_t)((size_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureInitialize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureInitialize", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnTextureInvalidateTexels(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureInvalidateTexels", 0); + ((void (*)(void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnTexturePoolFinalize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTexturePoolFinalize", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTexturePoolInitialize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTexturePoolInitialize", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *, ptrdiff_t, int))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (ptrdiff_t)c->x[2], (int)c->x[3]); +} +static void T_nvnTexturePoolRegisterTexture(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTexturePoolRegisterTexture", 0); + ((void (*)(void *, int, void *, void *))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (void *)(uintptr_t)c->x[2], (void *)(uintptr_t)c->x[3]); +} +static void T_nvnTexturePoolSetDebugLabel(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTexturePoolSetDebugLabel", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnTextureReadTexels(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureReadTexels", 0); + ((void (*)(void *, void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2], (void *)(uintptr_t)c->x[3]); +} +static void T_nvnTextureReadTexelsStrided(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureReadTexelsStrided", 0); + ((void (*)(void *, void *, void *, void *, ptrdiff_t, ptrdiff_t))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2], (void *)(uintptr_t)c->x[3], (ptrdiff_t)c->x[4], (ptrdiff_t)c->x[5]); +} +static void T_nvnTextureSetDebugLabel(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureSetDebugLabel", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnTextureViewCompare(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureViewCompare", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnTextureViewGetDepthStencilMode(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureViewGetDepthStencilMode", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnTextureViewGetFormat(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureViewGetFormat", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnTextureViewGetLayers(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureViewGetLayers", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnTextureViewGetLevels(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureViewGetLevels", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnTextureViewGetSwizzle(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureViewGetSwizzle", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *, void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2], (void *)(uintptr_t)c->x[3], (void *)(uintptr_t)c->x[4]); +} +static void T_nvnTextureViewGetTarget(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureViewGetTarget", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnTextureViewSetDefaults(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureViewSetDefaults", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnTextureViewSetDepthStencilMode(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureViewSetDepthStencilMode", 0); + ((void (*)(void *, NVNtextureDepthStencilMode))f)((void *)(uintptr_t)c->x[0], (NVNtextureDepthStencilMode)c->x[1]); +} +static void T_nvnTextureViewSetFormat(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureViewSetFormat", 0); + ((void (*)(void *, NVNformat))f)((void *)(uintptr_t)c->x[0], (NVNformat)c->x[1]); +} +static void T_nvnTextureViewSetLayers(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureViewSetLayers", 0); + ((void (*)(void *, int, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (int)c->x[2]); +} +static void T_nvnTextureViewSetLevels(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureViewSetLevels", 0); + ((void (*)(void *, int, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (int)c->x[2]); +} +static void T_nvnTextureViewSetSwizzle(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureViewSetSwizzle", 0); + ((void (*)(void *, NVNtextureSwizzle, NVNtextureSwizzle, NVNtextureSwizzle, NVNtextureSwizzle))f)((void *)(uintptr_t)c->x[0], (NVNtextureSwizzle)c->x[1], (NVNtextureSwizzle)c->x[2], (NVNtextureSwizzle)c->x[3], (NVNtextureSwizzle)c->x[4]); +} +static void T_nvnTextureViewSetTarget(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureViewSetTarget", 0); + ((void (*)(void *, NVNtextureTarget))f)((void *)(uintptr_t)c->x[0], (NVNtextureTarget)c->x[1]); +} +static void T_nvnTextureWriteTexels(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureWriteTexels", 0); + ((void (*)(void *, void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2], (void *)(uintptr_t)c->x[3]); +} +static void T_nvnTextureWriteTexelsStrided(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnTextureWriteTexelsStrided", 0); + ((void (*)(void *, void *, void *, void *, ptrdiff_t, ptrdiff_t))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2], (void *)(uintptr_t)c->x[3], (ptrdiff_t)c->x[4], (ptrdiff_t)c->x[5]); +} +static void T_nvnVertexAttribStateGetFormat(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnVertexAttribStateGetFormat", 0); + ((void (*)(void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnVertexAttribStateGetStreamIndex(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnVertexAttribStateGetStreamIndex", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnVertexAttribStateSetDefaults(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnVertexAttribStateSetDefaults", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnVertexAttribStateSetFormat(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnVertexAttribStateSetFormat", 0); + ((void (*)(void *, NVNformat, ptrdiff_t))f)((void *)(uintptr_t)c->x[0], (NVNformat)c->x[1], (ptrdiff_t)c->x[2]); +} +static void T_nvnVertexAttribStateSetStreamIndex(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnVertexAttribStateSetStreamIndex", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnVertexStreamStateGetDivisor(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnVertexStreamStateGetDivisor", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnVertexStreamStateGetStride(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnVertexStreamStateGetStride", 0); + c->x[0] = (uint64_t)((ptrdiff_t (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnVertexStreamStateSetDefaults(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnVertexStreamStateSetDefaults", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnVertexStreamStateSetDivisor(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnVertexStreamStateSetDivisor", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnVertexStreamStateSetStride(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnVertexStreamStateSetStride", 0); + ((void (*)(void *, ptrdiff_t))f)((void *)(uintptr_t)c->x[0], (ptrdiff_t)c->x[1]); +} +static void T_nvnWindowAcquireTexture(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowAcquireTexture", 0); + c->x[0] = (uint64_t)((NVNwindowAcquireTextureResult (*)(void *, void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnWindowBuilderGetDevice(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowBuilderGetDevice", 0); + c->x[0] = (uint64_t)(uintptr_t)((void * (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnWindowBuilderGetNativeWindow(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowBuilderGetNativeWindow", 0); + c->x[0] = (uint64_t)((NVNnativeWindow (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnWindowBuilderGetNumActiveTextures(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowBuilderGetNumActiveTextures", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnWindowBuilderGetNumTextures(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowBuilderGetNumTextures", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnWindowBuilderGetPresentInterval(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowBuilderGetPresentInterval", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnWindowBuilderGetTexture(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowBuilderGetTexture", 0); + c->x[0] = (uint64_t)(uintptr_t)((void * (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnWindowBuilderSetDefaults(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowBuilderSetDefaults", 0); + ((void (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnWindowBuilderSetDevice(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowBuilderSetDevice", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnWindowBuilderSetNativeWindow(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowBuilderSetNativeWindow", 0); + ((void (*)(void *, NVNnativeWindow))f)((void *)(uintptr_t)c->x[0], (NVNnativeWindow)c->x[1]); +} +static void T_nvnWindowBuilderSetNumActiveTextures(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowBuilderSetNumActiveTextures", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnWindowBuilderSetPresentInterval(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowBuilderSetPresentInterval", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnWindowBuilderSetTextures(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowBuilderSetTextures", 0); + ((void (*)(void *, int, void *))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (void *)(uintptr_t)c->x[2]); +} +static void T_nvnWindowGetCrop(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowGetCrop", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnWindowGetNativeWindow(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowGetNativeWindow", 0); + c->x[0] = (uint64_t)((NVNnativeWindow (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnWindowGetNumActiveTextures(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowGetNumActiveTextures", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnWindowGetNumTextures(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowGetNumTextures", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnWindowGetPresentInterval(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowGetPresentInterval", 0); + c->x[0] = (uint64_t)((int (*)(void *))f)((void *)(uintptr_t)c->x[0]); +} +static void T_nvnWindowInitialize(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowInitialize", 0); + c->x[0] = (uint64_t)((NVNboolean (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnWindowSetCrop(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowSetCrop", 0); + ((void (*)(void *, int, int, int, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1], (int)c->x[2], (int)c->x[3], (int)c->x[4]); +} +static void T_nvnWindowSetDebugLabel(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowSetDebugLabel", 0); + ((void (*)(void *, void *))f)((void *)(uintptr_t)c->x[0], (void *)(uintptr_t)c->x[1]); +} +static void T_nvnWindowSetNumActiveTextures(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowSetNumActiveTextures", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +static void T_nvnWindowSetPresentInterval(Cpu *c) { + static void *f; if (!f) f = nvn_impl_fn("nvnWindowSetPresentInterval", 0); + ((void (*)(void *, int))f)((void *)(uintptr_t)c->x[0], (int)c->x[1]); +} +const struct { const char *name; void (*fn)(Cpu *); } nvn_thunks[] = { + {"nvnBlendStateGetBlendEquation", T_nvnBlendStateGetBlendEquation}, + {"nvnBlendStateGetBlendFunc", T_nvnBlendStateGetBlendFunc}, + {"nvnBlendStateGetBlendTarget", T_nvnBlendStateGetBlendTarget}, + {"nvnBlendStateSetAdvancedMode", T_nvnBlendStateSetAdvancedMode}, + {"nvnBlendStateSetAdvancedNormalizedDst", T_nvnBlendStateSetAdvancedNormalizedDst}, + {"nvnBlendStateSetAdvancedOverlap", T_nvnBlendStateSetAdvancedOverlap}, + {"nvnBlendStateSetAdvancedPremultipliedSrc", T_nvnBlendStateSetAdvancedPremultipliedSrc}, + {"nvnBlendStateSetBlendEquation", T_nvnBlendStateSetBlendEquation}, + {"nvnBlendStateSetBlendFunc", T_nvnBlendStateSetBlendFunc}, + {"nvnBlendStateSetBlendTarget", T_nvnBlendStateSetBlendTarget}, + {"nvnBlendStateSetDefaults", T_nvnBlendStateSetDefaults}, + {"nvnBufferBuilderSetDefaults", T_nvnBufferBuilderSetDefaults}, + {"nvnBufferBuilderSetDevice", T_nvnBufferBuilderSetDevice}, + {"nvnBufferBuilderSetStorage", T_nvnBufferBuilderSetStorage}, + {"nvnBufferFinalize", T_nvnBufferFinalize}, + {"nvnBufferFlushMappedRange", T_nvnBufferFlushMappedRange}, + {"nvnBufferGetAddress", T_nvnBufferGetAddress}, + {"nvnBufferGetMemoryOffset", T_nvnBufferGetMemoryOffset}, + {"nvnBufferGetMemoryPool", T_nvnBufferGetMemoryPool}, + {"nvnBufferGetSize", T_nvnBufferGetSize}, + {"nvnBufferInitialize", T_nvnBufferInitialize}, + {"nvnBufferInvalidateMappedRange", T_nvnBufferInvalidateMappedRange}, + {"nvnBufferMap", T_nvnBufferMap}, + {"nvnBufferSetDebugLabel", T_nvnBufferSetDebugLabel}, + {"nvnChannelMaskStateGetChannelMask", T_nvnChannelMaskStateGetChannelMask}, + {"nvnChannelMaskStateSetChannelMask", T_nvnChannelMaskStateSetChannelMask}, + {"nvnChannelMaskStateSetDefaults", T_nvnChannelMaskStateSetDefaults}, + {"nvnColorStateGetAlphaTest", T_nvnColorStateGetAlphaTest}, + {"nvnColorStateGetBlendEnable", T_nvnColorStateGetBlendEnable}, + {"nvnColorStateGetLogicOp", T_nvnColorStateGetLogicOp}, + {"nvnColorStateSetAlphaTest", T_nvnColorStateSetAlphaTest}, + {"nvnColorStateSetBlendEnable", T_nvnColorStateSetBlendEnable}, + {"nvnColorStateSetDefaults", T_nvnColorStateSetDefaults}, + {"nvnColorStateSetLogicOp", T_nvnColorStateSetLogicOp}, + {"nvnCommandBufferAddCommandMemory", T_nvnCommandBufferAddCommandMemory}, + {"nvnCommandBufferAddControlMemory", T_nvnCommandBufferAddControlMemory}, + {"nvnCommandBufferBarrier", T_nvnCommandBufferBarrier}, + {"nvnCommandBufferBeginRecording", T_nvnCommandBufferBeginRecording}, + {"nvnCommandBufferBindBlendState", T_nvnCommandBufferBindBlendState}, + {"nvnCommandBufferBindChannelMaskState", T_nvnCommandBufferBindChannelMaskState}, + {"nvnCommandBufferBindColorState", T_nvnCommandBufferBindColorState}, + {"nvnCommandBufferBindDepthStencilState", T_nvnCommandBufferBindDepthStencilState}, + {"nvnCommandBufferBindImage", T_nvnCommandBufferBindImage}, + {"nvnCommandBufferBindImages", T_nvnCommandBufferBindImages}, + {"nvnCommandBufferBindImagesDeferred", T_nvnCommandBufferBindImagesDeferred}, + {"nvnCommandBufferBindMultisampleState", T_nvnCommandBufferBindMultisampleState}, + {"nvnCommandBufferBindPolygonState", T_nvnCommandBufferBindPolygonState}, + {"nvnCommandBufferBindProgram", T_nvnCommandBufferBindProgram}, + {"nvnCommandBufferBindSeparateSampler", T_nvnCommandBufferBindSeparateSampler}, + {"nvnCommandBufferBindSeparateSamplers", T_nvnCommandBufferBindSeparateSamplers}, + {"nvnCommandBufferBindSeparateSamplersDeferred", T_nvnCommandBufferBindSeparateSamplersDeferred}, + {"nvnCommandBufferBindSeparateTexture", T_nvnCommandBufferBindSeparateTexture}, + {"nvnCommandBufferBindSeparateTextures", T_nvnCommandBufferBindSeparateTextures}, + {"nvnCommandBufferBindSeparateTexturesDeferred", T_nvnCommandBufferBindSeparateTexturesDeferred}, + {"nvnCommandBufferBindStorageBuffer", T_nvnCommandBufferBindStorageBuffer}, + {"nvnCommandBufferBindStorageBuffers", T_nvnCommandBufferBindStorageBuffers}, + {"nvnCommandBufferBindStorageBuffersDeferred", T_nvnCommandBufferBindStorageBuffersDeferred}, + {"nvnCommandBufferBindTexture", T_nvnCommandBufferBindTexture}, + {"nvnCommandBufferBindTextures", T_nvnCommandBufferBindTextures}, + {"nvnCommandBufferBindTexturesDeferred", T_nvnCommandBufferBindTexturesDeferred}, + {"nvnCommandBufferBindUniformBuffer", T_nvnCommandBufferBindUniformBuffer}, + {"nvnCommandBufferBindUniformBuffers", T_nvnCommandBufferBindUniformBuffers}, + {"nvnCommandBufferBindUniformBuffersDeferred", T_nvnCommandBufferBindUniformBuffersDeferred}, + {"nvnCommandBufferBindVertexAttribState", T_nvnCommandBufferBindVertexAttribState}, + {"nvnCommandBufferBindVertexBuffer", T_nvnCommandBufferBindVertexBuffer}, + {"nvnCommandBufferBindVertexBuffers", T_nvnCommandBufferBindVertexBuffers}, + {"nvnCommandBufferBindVertexStreamState", T_nvnCommandBufferBindVertexStreamState}, + {"nvnCommandBufferCallCommands", T_nvnCommandBufferCallCommands}, + {"nvnCommandBufferClearColor", T_nvnCommandBufferClearColor}, + {"nvnCommandBufferClearDepthStencil", T_nvnCommandBufferClearDepthStencil}, + {"nvnCommandBufferClearTexture", T_nvnCommandBufferClearTexture}, + {"nvnCommandBufferClearTextureui", T_nvnCommandBufferClearTextureui}, + {"nvnCommandBufferCopyBufferToBuffer", T_nvnCommandBufferCopyBufferToBuffer}, + {"nvnCommandBufferCopyBufferToTexture", T_nvnCommandBufferCopyBufferToTexture}, + {"nvnCommandBufferCopyCommands", T_nvnCommandBufferCopyCommands}, + {"nvnCommandBufferCopyTextureToBuffer", T_nvnCommandBufferCopyTextureToBuffer}, + {"nvnCommandBufferCopyTextureToTexture", T_nvnCommandBufferCopyTextureToTexture}, + {"nvnCommandBufferDiscardColor", T_nvnCommandBufferDiscardColor}, + {"nvnCommandBufferDiscardDepthStencil", T_nvnCommandBufferDiscardDepthStencil}, + {"nvnCommandBufferDispatchCompute", T_nvnCommandBufferDispatchCompute}, + {"nvnCommandBufferDrawArrays", T_nvnCommandBufferDrawArrays}, + {"nvnCommandBufferDrawArraysInstanced", T_nvnCommandBufferDrawArraysInstanced}, + {"nvnCommandBufferDrawElements", T_nvnCommandBufferDrawElements}, + {"nvnCommandBufferDrawElementsBaseVertex", T_nvnCommandBufferDrawElementsBaseVertex}, + {"nvnCommandBufferDrawElementsInstanced", T_nvnCommandBufferDrawElementsInstanced}, + {"nvnCommandBufferEndRecording", T_nvnCommandBufferEndRecording}, + {"nvnCommandBufferFinalize", T_nvnCommandBufferFinalize}, + {"nvnCommandBufferGetCommandMemoryFree", T_nvnCommandBufferGetCommandMemoryFree}, + {"nvnCommandBufferGetCommandMemorySize", T_nvnCommandBufferGetCommandMemorySize}, + {"nvnCommandBufferGetCommandMemoryUsed", T_nvnCommandBufferGetCommandMemoryUsed}, + {"nvnCommandBufferGetControlMemoryFree", T_nvnCommandBufferGetControlMemoryFree}, + {"nvnCommandBufferGetControlMemorySize", T_nvnCommandBufferGetControlMemorySize}, + {"nvnCommandBufferGetControlMemoryUsed", T_nvnCommandBufferGetControlMemoryUsed}, + {"nvnCommandBufferGetMemoryCallback", T_nvnCommandBufferGetMemoryCallback}, + {"nvnCommandBufferGetMemoryCallbackData", T_nvnCommandBufferGetMemoryCallbackData}, + {"nvnCommandBufferInitialize", T_nvnCommandBufferInitialize}, + {"nvnCommandBufferInsertCheckpointDynamic", T_nvnCommandBufferInsertCheckpointDynamic}, + {"nvnCommandBufferInsertCheckpointStatic", T_nvnCommandBufferInsertCheckpointStatic}, + {"nvnCommandBufferInsertDebugMarker", T_nvnCommandBufferInsertDebugMarker}, + {"nvnCommandBufferInsertDebugMarkerDynamic", T_nvnCommandBufferInsertDebugMarkerDynamic}, + {"nvnCommandBufferInsertDebugMarkerStatic", T_nvnCommandBufferInsertDebugMarkerStatic}, + {"nvnCommandBufferIsRecording", T_nvnCommandBufferIsRecording}, + {"nvnCommandBufferOverrideSubtileSize", T_nvnCommandBufferOverrideSubtileSize}, + {"nvnCommandBufferPopDebugGroup", T_nvnCommandBufferPopDebugGroup}, + {"nvnCommandBufferPopDebugGroupId", T_nvnCommandBufferPopDebugGroupId}, + {"nvnCommandBufferPushDebugGroup", T_nvnCommandBufferPushDebugGroup}, + {"nvnCommandBufferPushDebugGroupDynamic", T_nvnCommandBufferPushDebugGroupDynamic}, + {"nvnCommandBufferPushDebugGroupStatic", T_nvnCommandBufferPushDebugGroupStatic}, + {"nvnCommandBufferReportCounter", T_nvnCommandBufferReportCounter}, + {"nvnCommandBufferResetCounter", T_nvnCommandBufferResetCounter}, + {"nvnCommandBufferRestoreZCullData", T_nvnCommandBufferRestoreZCullData}, + {"nvnCommandBufferSaveZCullData", T_nvnCommandBufferSaveZCullData}, + {"nvnCommandBufferSetAlphaRef", T_nvnCommandBufferSetAlphaRef}, + {"nvnCommandBufferSetBlendColor", T_nvnCommandBufferSetBlendColor}, + {"nvnCommandBufferSetCommandMemoryCallbackEnabled", T_nvnCommandBufferSetCommandMemoryCallbackEnabled}, + {"nvnCommandBufferSetConservativeRasterDilate", T_nvnCommandBufferSetConservativeRasterDilate}, + {"nvnCommandBufferSetConservativeRasterEnable", T_nvnCommandBufferSetConservativeRasterEnable}, + {"nvnCommandBufferSetControlMemoryCallbackEnabled", T_nvnCommandBufferSetControlMemoryCallbackEnabled}, + {"nvnCommandBufferSetCopyRowStride", T_nvnCommandBufferSetCopyRowStride}, + {"nvnCommandBufferSetDebugLabel", T_nvnCommandBufferSetDebugLabel}, + {"nvnCommandBufferSetDepthBounds", T_nvnCommandBufferSetDepthBounds}, + {"nvnCommandBufferSetDepthClamp", T_nvnCommandBufferSetDepthClamp}, + {"nvnCommandBufferSetDepthRange", T_nvnCommandBufferSetDepthRange}, + {"nvnCommandBufferSetDepthRanges", T_nvnCommandBufferSetDepthRanges}, + {"nvnCommandBufferSetLineWidth", T_nvnCommandBufferSetLineWidth}, + {"nvnCommandBufferSetMemoryCallback", T_nvnCommandBufferSetMemoryCallback}, + {"nvnCommandBufferSetMemoryCallbackData", T_nvnCommandBufferSetMemoryCallbackData}, + {"nvnCommandBufferSetPointSize", T_nvnCommandBufferSetPointSize}, + {"nvnCommandBufferSetPolygonOffsetClamp", T_nvnCommandBufferSetPolygonOffsetClamp}, + {"nvnCommandBufferSetPrimitiveRestart", T_nvnCommandBufferSetPrimitiveRestart}, + {"nvnCommandBufferSetProvokingVertexMode", T_nvnCommandBufferSetProvokingVertexMode}, + {"nvnCommandBufferSetRasterizerDiscard", T_nvnCommandBufferSetRasterizerDiscard}, + {"nvnCommandBufferSetRenderEnable", T_nvnCommandBufferSetRenderEnable}, + {"nvnCommandBufferSetRenderEnableConditional", T_nvnCommandBufferSetRenderEnableConditional}, + {"nvnCommandBufferSetRenderTargets", T_nvnCommandBufferSetRenderTargets}, + {"nvnCommandBufferSetSampleMask", T_nvnCommandBufferSetSampleMask}, + {"nvnCommandBufferSetSamplerPool", T_nvnCommandBufferSetSamplerPool}, + {"nvnCommandBufferSetScissor", T_nvnCommandBufferSetScissor}, + {"nvnCommandBufferSetScissors", T_nvnCommandBufferSetScissors}, + {"nvnCommandBufferSetStencilMask", T_nvnCommandBufferSetStencilMask}, + {"nvnCommandBufferSetStencilRef", T_nvnCommandBufferSetStencilRef}, + {"nvnCommandBufferSetStencilValueMask", T_nvnCommandBufferSetStencilValueMask}, + {"nvnCommandBufferSetSubpixelPrecisionBias", T_nvnCommandBufferSetSubpixelPrecisionBias}, + {"nvnCommandBufferSetTexturePool", T_nvnCommandBufferSetTexturePool}, + {"nvnCommandBufferSetTiledCacheAction", T_nvnCommandBufferSetTiledCacheAction}, + {"nvnCommandBufferSetTiledCacheTileSize", T_nvnCommandBufferSetTiledCacheTileSize}, + {"nvnCommandBufferSetViewport", T_nvnCommandBufferSetViewport}, + {"nvnCommandBufferSetViewports", T_nvnCommandBufferSetViewports}, + {"nvnCommandBufferSetZCullZF32CompressionEnable", T_nvnCommandBufferSetZCullZF32CompressionEnable}, + {"nvnCommandBufferSignalEvent", T_nvnCommandBufferSignalEvent}, + {"nvnCommandBufferUpdateUniformBuffer", T_nvnCommandBufferUpdateUniformBuffer}, + {"nvnDepthStencilStateGetDepthFunc", T_nvnDepthStencilStateGetDepthFunc}, + {"nvnDepthStencilStateGetDepthTestEnable", T_nvnDepthStencilStateGetDepthTestEnable}, + {"nvnDepthStencilStateGetDepthWriteEnable", T_nvnDepthStencilStateGetDepthWriteEnable}, + {"nvnDepthStencilStateGetStencilFunc", T_nvnDepthStencilStateGetStencilFunc}, + {"nvnDepthStencilStateGetStencilOp", T_nvnDepthStencilStateGetStencilOp}, + {"nvnDepthStencilStateGetStencilTestEnable", T_nvnDepthStencilStateGetStencilTestEnable}, + {"nvnDepthStencilStateSetDefaults", T_nvnDepthStencilStateSetDefaults}, + {"nvnDepthStencilStateSetDepthFunc", T_nvnDepthStencilStateSetDepthFunc}, + {"nvnDepthStencilStateSetDepthTestEnable", T_nvnDepthStencilStateSetDepthTestEnable}, + {"nvnDepthStencilStateSetDepthWriteEnable", T_nvnDepthStencilStateSetDepthWriteEnable}, + {"nvnDepthStencilStateSetStencilFunc", T_nvnDepthStencilStateSetStencilFunc}, + {"nvnDepthStencilStateSetStencilOp", T_nvnDepthStencilStateSetStencilOp}, + {"nvnDepthStencilStateSetStencilTestEnable", T_nvnDepthStencilStateSetStencilTestEnable}, + {"nvnDeviceBuilderSetDefaults", T_nvnDeviceBuilderSetDefaults}, + {"nvnDeviceBuilderSetFlags", T_nvnDeviceBuilderSetFlags}, + {"nvnDeviceFinalize", T_nvnDeviceFinalize}, + {"nvnDeviceFinalizeCommandHandle", T_nvnDeviceFinalizeCommandHandle}, + {"nvnDeviceGetCurrentTimestampInNanoseconds", T_nvnDeviceGetCurrentTimestampInNanoseconds}, + {"nvnDeviceGetImageHandle", T_nvnDeviceGetImageHandle}, + {"nvnDeviceGetInteger", T_nvnDeviceGetInteger}, + {"nvnDeviceGetProcAddress", T_nvnDeviceGetProcAddress}, + {"nvnDeviceGetSeparateSamplerHandle", T_nvnDeviceGetSeparateSamplerHandle}, + {"nvnDeviceGetSeparateTextureHandle", T_nvnDeviceGetSeparateTextureHandle}, + {"nvnDeviceGetTexelFetchHandle", T_nvnDeviceGetTexelFetchHandle}, + {"nvnDeviceGetTextureHandle", T_nvnDeviceGetTextureHandle}, + {"nvnDeviceGetTimestampInNanoseconds", T_nvnDeviceGetTimestampInNanoseconds}, + {"nvnDeviceInitialize", T_nvnDeviceInitialize}, + {"nvnDeviceInstallDebugCallback", T_nvnDeviceInstallDebugCallback}, + {"nvnDeviceIsExternalDebuggerAttached", T_nvnDeviceIsExternalDebuggerAttached}, + {"nvnDeviceRegisterFastClearColor", T_nvnDeviceRegisterFastClearColor}, + {"nvnDeviceRegisterFastClearColori", T_nvnDeviceRegisterFastClearColori}, + {"nvnDeviceRegisterFastClearColorui", T_nvnDeviceRegisterFastClearColorui}, + {"nvnDeviceRegisterFastClearDepth", T_nvnDeviceRegisterFastClearDepth}, + {"nvnDeviceSetAutomaticCheckpointsMask", T_nvnDeviceSetAutomaticCheckpointsMask}, + {"nvnDeviceSetDebugLabel", T_nvnDeviceSetDebugLabel}, + {"nvnDeviceSetDepthMode", T_nvnDeviceSetDepthMode}, + {"nvnDeviceSetWindowOriginMode", T_nvnDeviceSetWindowOriginMode}, + {"nvnEventBuilderSetDefaults", T_nvnEventBuilderSetDefaults}, + {"nvnEventBuilderSetStorage", T_nvnEventBuilderSetStorage}, + {"nvnEventFinalize", T_nvnEventFinalize}, + {"nvnEventGetValue", T_nvnEventGetValue}, + {"nvnEventInitialize", T_nvnEventInitialize}, + {"nvnEventSignal", T_nvnEventSignal}, + {"nvnMemoryPoolBuilderSetDefaults", T_nvnMemoryPoolBuilderSetDefaults}, + {"nvnMemoryPoolBuilderSetDevice", T_nvnMemoryPoolBuilderSetDevice}, + {"nvnMemoryPoolBuilderSetFlags", T_nvnMemoryPoolBuilderSetFlags}, + {"nvnMemoryPoolBuilderSetStorage", T_nvnMemoryPoolBuilderSetStorage}, + {"nvnMemoryPoolFinalize", T_nvnMemoryPoolFinalize}, + {"nvnMemoryPoolFlushMappedRange", T_nvnMemoryPoolFlushMappedRange}, + {"nvnMemoryPoolGetBufferAddress", T_nvnMemoryPoolGetBufferAddress}, + {"nvnMemoryPoolGetFlags", T_nvnMemoryPoolGetFlags}, + {"nvnMemoryPoolGetSize", T_nvnMemoryPoolGetSize}, + {"nvnMemoryPoolInitialize", T_nvnMemoryPoolInitialize}, + {"nvnMemoryPoolInvalidateMappedRange", T_nvnMemoryPoolInvalidateMappedRange}, + {"nvnMemoryPoolMap", T_nvnMemoryPoolMap}, + {"nvnMemoryPoolMapVirtual", T_nvnMemoryPoolMapVirtual}, + {"nvnMemoryPoolSetDebugLabel", T_nvnMemoryPoolSetDebugLabel}, + {"nvnMultisampleStateGetAlphaToCoverageEnable", T_nvnMultisampleStateGetAlphaToCoverageEnable}, + {"nvnMultisampleStateGetMultisampleEnable", T_nvnMultisampleStateGetMultisampleEnable}, + {"nvnMultisampleStateGetRasterSamples", T_nvnMultisampleStateGetRasterSamples}, + {"nvnMultisampleStateGetSamples", T_nvnMultisampleStateGetSamples}, + {"nvnMultisampleStateSetAlphaToCoverageDither", T_nvnMultisampleStateSetAlphaToCoverageDither}, + {"nvnMultisampleStateSetAlphaToCoverageEnable", T_nvnMultisampleStateSetAlphaToCoverageEnable}, + {"nvnMultisampleStateSetDefaults", T_nvnMultisampleStateSetDefaults}, + {"nvnMultisampleStateSetMultisampleEnable", T_nvnMultisampleStateSetMultisampleEnable}, + {"nvnMultisampleStateSetRasterSamples", T_nvnMultisampleStateSetRasterSamples}, + {"nvnMultisampleStateSetSampleLocations", T_nvnMultisampleStateSetSampleLocations}, + {"nvnMultisampleStateSetSampleLocationsEnable", T_nvnMultisampleStateSetSampleLocationsEnable}, + {"nvnMultisampleStateSetSamples", T_nvnMultisampleStateSetSamples}, + {"nvnPolygonStateGetCullFace", T_nvnPolygonStateGetCullFace}, + {"nvnPolygonStateGetFrontFace", T_nvnPolygonStateGetFrontFace}, + {"nvnPolygonStateGetPolygonMode", T_nvnPolygonStateGetPolygonMode}, + {"nvnPolygonStateGetPolygonOffsetEnables", T_nvnPolygonStateGetPolygonOffsetEnables}, + {"nvnPolygonStateSetCullFace", T_nvnPolygonStateSetCullFace}, + {"nvnPolygonStateSetDefaults", T_nvnPolygonStateSetDefaults}, + {"nvnPolygonStateSetFrontFace", T_nvnPolygonStateSetFrontFace}, + {"nvnPolygonStateSetPolygonMode", T_nvnPolygonStateSetPolygonMode}, + {"nvnPolygonStateSetPolygonOffsetEnables", T_nvnPolygonStateSetPolygonOffsetEnables}, + {"nvnProgramFinalize", T_nvnProgramFinalize}, + {"nvnProgramGetSubtileSize", T_nvnProgramGetSubtileSize}, + {"nvnProgramInitialize", T_nvnProgramInitialize}, + {"nvnProgramSetDebugLabel", T_nvnProgramSetDebugLabel}, + {"nvnProgramSetShaders", T_nvnProgramSetShaders}, + {"nvnProgramSetShadersExt", T_nvnProgramSetShadersExt}, + {"nvnQueueBuilderGetQueueMemorySize", T_nvnQueueBuilderGetQueueMemorySize}, + {"nvnQueueBuilderSetCommandFlushThreshold", T_nvnQueueBuilderSetCommandFlushThreshold}, + {"nvnQueueBuilderSetCommandMemorySize", T_nvnQueueBuilderSetCommandMemorySize}, + {"nvnQueueBuilderSetComputeMemorySize", T_nvnQueueBuilderSetComputeMemorySize}, + {"nvnQueueBuilderSetControlMemorySize", T_nvnQueueBuilderSetControlMemorySize}, + {"nvnQueueBuilderSetDefaults", T_nvnQueueBuilderSetDefaults}, + {"nvnQueueBuilderSetDevice", T_nvnQueueBuilderSetDevice}, + {"nvnQueueBuilderSetFlags", T_nvnQueueBuilderSetFlags}, + {"nvnQueueBuilderSetQueueMemory", T_nvnQueueBuilderSetQueueMemory}, + {"nvnQueueFenceSync", T_nvnQueueFenceSync}, + {"nvnQueueFinalize", T_nvnQueueFinalize}, + {"nvnQueueFinish", T_nvnQueueFinish}, + {"nvnQueueFlush", T_nvnQueueFlush}, + {"nvnQueueGetError", T_nvnQueueGetError}, + {"nvnQueueInitialize", T_nvnQueueInitialize}, + {"nvnQueuePresentTexture", T_nvnQueuePresentTexture}, + {"nvnQueueSetDebugLabel", T_nvnQueueSetDebugLabel}, + {"nvnQueueSubmitCommands", T_nvnQueueSubmitCommands}, + {"nvnSamplerBuilderGetBorderColor", T_nvnSamplerBuilderGetBorderColor}, + {"nvnSamplerBuilderGetBorderColori", T_nvnSamplerBuilderGetBorderColori}, + {"nvnSamplerBuilderGetBorderColorui", T_nvnSamplerBuilderGetBorderColorui}, + {"nvnSamplerBuilderGetCompare", T_nvnSamplerBuilderGetCompare}, + {"nvnSamplerBuilderGetDevice", T_nvnSamplerBuilderGetDevice}, + {"nvnSamplerBuilderGetLodBias", T_nvnSamplerBuilderGetLodBias}, + {"nvnSamplerBuilderGetLodClamp", T_nvnSamplerBuilderGetLodClamp}, + {"nvnSamplerBuilderGetLodSnap", T_nvnSamplerBuilderGetLodSnap}, + {"nvnSamplerBuilderGetMaxAnisotropy", T_nvnSamplerBuilderGetMaxAnisotropy}, + {"nvnSamplerBuilderGetMinMagFilter", T_nvnSamplerBuilderGetMinMagFilter}, + {"nvnSamplerBuilderGetReductionFilter", T_nvnSamplerBuilderGetReductionFilter}, + {"nvnSamplerBuilderGetWrapMode", T_nvnSamplerBuilderGetWrapMode}, + {"nvnSamplerBuilderSetBorderColor", T_nvnSamplerBuilderSetBorderColor}, + {"nvnSamplerBuilderSetBorderColori", T_nvnSamplerBuilderSetBorderColori}, + {"nvnSamplerBuilderSetBorderColorui", T_nvnSamplerBuilderSetBorderColorui}, + {"nvnSamplerBuilderSetCompare", T_nvnSamplerBuilderSetCompare}, + {"nvnSamplerBuilderSetDefaults", T_nvnSamplerBuilderSetDefaults}, + {"nvnSamplerBuilderSetDevice", T_nvnSamplerBuilderSetDevice}, + {"nvnSamplerBuilderSetLodBias", T_nvnSamplerBuilderSetLodBias}, + {"nvnSamplerBuilderSetLodClamp", T_nvnSamplerBuilderSetLodClamp}, + {"nvnSamplerBuilderSetLodSnap", T_nvnSamplerBuilderSetLodSnap}, + {"nvnSamplerBuilderSetMaxAnisotropy", T_nvnSamplerBuilderSetMaxAnisotropy}, + {"nvnSamplerBuilderSetMinMagFilter", T_nvnSamplerBuilderSetMinMagFilter}, + {"nvnSamplerBuilderSetReductionFilter", T_nvnSamplerBuilderSetReductionFilter}, + {"nvnSamplerBuilderSetWrapMode", T_nvnSamplerBuilderSetWrapMode}, + {"nvnSamplerCompare", T_nvnSamplerCompare}, + {"nvnSamplerFinalize", T_nvnSamplerFinalize}, + {"nvnSamplerGetBorderColor", T_nvnSamplerGetBorderColor}, + {"nvnSamplerGetBorderColori", T_nvnSamplerGetBorderColori}, + {"nvnSamplerGetBorderColorui", T_nvnSamplerGetBorderColorui}, + {"nvnSamplerGetCompare", T_nvnSamplerGetCompare}, + {"nvnSamplerGetDebugID", T_nvnSamplerGetDebugID}, + {"nvnSamplerGetLodBias", T_nvnSamplerGetLodBias}, + {"nvnSamplerGetLodClamp", T_nvnSamplerGetLodClamp}, + {"nvnSamplerGetMaxAnisotropy", T_nvnSamplerGetMaxAnisotropy}, + {"nvnSamplerGetMinMagFilter", T_nvnSamplerGetMinMagFilter}, + {"nvnSamplerGetReductionFilter", T_nvnSamplerGetReductionFilter}, + {"nvnSamplerGetWrapMode", T_nvnSamplerGetWrapMode}, + {"nvnSamplerInitialize", T_nvnSamplerInitialize}, + {"nvnSamplerPoolFinalize", T_nvnSamplerPoolFinalize}, + {"nvnSamplerPoolInitialize", T_nvnSamplerPoolInitialize}, + {"nvnSamplerPoolRegisterSampler", T_nvnSamplerPoolRegisterSampler}, + {"nvnSamplerPoolSetDebugLabel", T_nvnSamplerPoolSetDebugLabel}, + {"nvnSamplerSetDebugLabel", T_nvnSamplerSetDebugLabel}, + {"nvnSyncFinalize", T_nvnSyncFinalize}, + {"nvnSyncInitialize", T_nvnSyncInitialize}, + {"nvnSyncSetDebugLabel", T_nvnSyncSetDebugLabel}, + {"nvnSyncWait", T_nvnSyncWait}, + {"nvnTextureBuilderGetDepth", T_nvnTextureBuilderGetDepth}, + {"nvnTextureBuilderGetDepthStencilMode", T_nvnTextureBuilderGetDepthStencilMode}, + {"nvnTextureBuilderGetDevice", T_nvnTextureBuilderGetDevice}, + {"nvnTextureBuilderGetFlags", T_nvnTextureBuilderGetFlags}, + {"nvnTextureBuilderGetFormat", T_nvnTextureBuilderGetFormat}, + {"nvnTextureBuilderGetHeight", T_nvnTextureBuilderGetHeight}, + {"nvnTextureBuilderGetLevels", T_nvnTextureBuilderGetLevels}, + {"nvnTextureBuilderGetMemoryOffset", T_nvnTextureBuilderGetMemoryOffset}, + {"nvnTextureBuilderGetMemoryPool", T_nvnTextureBuilderGetMemoryPool}, + {"nvnTextureBuilderGetPackagedTextureData", T_nvnTextureBuilderGetPackagedTextureData}, + {"nvnTextureBuilderGetSamples", T_nvnTextureBuilderGetSamples}, + {"nvnTextureBuilderGetStorageAlignment", T_nvnTextureBuilderGetStorageAlignment}, + {"nvnTextureBuilderGetStorageClass", T_nvnTextureBuilderGetStorageClass}, + {"nvnTextureBuilderGetStorageSize", T_nvnTextureBuilderGetStorageSize}, + {"nvnTextureBuilderGetStride", T_nvnTextureBuilderGetStride}, + {"nvnTextureBuilderGetSwizzle", T_nvnTextureBuilderGetSwizzle}, + {"nvnTextureBuilderGetTarget", T_nvnTextureBuilderGetTarget}, + {"nvnTextureBuilderGetViewOffset", T_nvnTextureBuilderGetViewOffset}, + {"nvnTextureBuilderGetWidth", T_nvnTextureBuilderGetWidth}, + {"nvnTextureBuilderGetZCullStorageSize", T_nvnTextureBuilderGetZCullStorageSize}, + {"nvnTextureBuilderSetDefaults", T_nvnTextureBuilderSetDefaults}, + {"nvnTextureBuilderSetDepth", T_nvnTextureBuilderSetDepth}, + {"nvnTextureBuilderSetDepthStencilMode", T_nvnTextureBuilderSetDepthStencilMode}, + {"nvnTextureBuilderSetDevice", T_nvnTextureBuilderSetDevice}, + {"nvnTextureBuilderSetFlags", T_nvnTextureBuilderSetFlags}, + {"nvnTextureBuilderSetFormat", T_nvnTextureBuilderSetFormat}, + {"nvnTextureBuilderSetHeight", T_nvnTextureBuilderSetHeight}, + {"nvnTextureBuilderSetLevels", T_nvnTextureBuilderSetLevels}, + {"nvnTextureBuilderSetPackagedTextureData", T_nvnTextureBuilderSetPackagedTextureData}, + {"nvnTextureBuilderSetPackagedTextureLayout", T_nvnTextureBuilderSetPackagedTextureLayout}, + {"nvnTextureBuilderSetSamples", T_nvnTextureBuilderSetSamples}, + {"nvnTextureBuilderSetSize1D", T_nvnTextureBuilderSetSize1D}, + {"nvnTextureBuilderSetSize2D", T_nvnTextureBuilderSetSize2D}, + {"nvnTextureBuilderSetSize3D", T_nvnTextureBuilderSetSize3D}, + {"nvnTextureBuilderSetStorage", T_nvnTextureBuilderSetStorage}, + {"nvnTextureBuilderSetStride", T_nvnTextureBuilderSetStride}, + {"nvnTextureBuilderSetSwizzle", T_nvnTextureBuilderSetSwizzle}, + {"nvnTextureBuilderSetTarget", T_nvnTextureBuilderSetTarget}, + {"nvnTextureBuilderSetWidth", T_nvnTextureBuilderSetWidth}, + {"nvnTextureCompare", T_nvnTextureCompare}, + {"nvnTextureFinalize", T_nvnTextureFinalize}, + {"nvnTextureFlushTexels", T_nvnTextureFlushTexels}, + {"nvnTextureGetDebugID", T_nvnTextureGetDebugID}, + {"nvnTextureGetDepth", T_nvnTextureGetDepth}, + {"nvnTextureGetDepthStencilMode", T_nvnTextureGetDepthStencilMode}, + {"nvnTextureGetFlags", T_nvnTextureGetFlags}, + {"nvnTextureGetFormat", T_nvnTextureGetFormat}, + {"nvnTextureGetHeight", T_nvnTextureGetHeight}, + {"nvnTextureGetLevels", T_nvnTextureGetLevels}, + {"nvnTextureGetMemoryOffset", T_nvnTextureGetMemoryOffset}, + {"nvnTextureGetMemoryPool", T_nvnTextureGetMemoryPool}, + {"nvnTextureGetRawStorageClass", T_nvnTextureGetRawStorageClass}, + {"nvnTextureGetSamples", T_nvnTextureGetSamples}, + {"nvnTextureGetStorageClass", T_nvnTextureGetStorageClass}, + {"nvnTextureGetStorageSize", T_nvnTextureGetStorageSize}, + {"nvnTextureGetStride", T_nvnTextureGetStride}, + {"nvnTextureGetSwizzle", T_nvnTextureGetSwizzle}, + {"nvnTextureGetTarget", T_nvnTextureGetTarget}, + {"nvnTextureGetTextureAddress", T_nvnTextureGetTextureAddress}, + {"nvnTextureGetViewOffset", T_nvnTextureGetViewOffset}, + {"nvnTextureGetWidth", T_nvnTextureGetWidth}, + {"nvnTextureGetZCullStorageSize", T_nvnTextureGetZCullStorageSize}, + {"nvnTextureInitialize", T_nvnTextureInitialize}, + {"nvnTextureInvalidateTexels", T_nvnTextureInvalidateTexels}, + {"nvnTexturePoolFinalize", T_nvnTexturePoolFinalize}, + {"nvnTexturePoolInitialize", T_nvnTexturePoolInitialize}, + {"nvnTexturePoolRegisterTexture", T_nvnTexturePoolRegisterTexture}, + {"nvnTexturePoolSetDebugLabel", T_nvnTexturePoolSetDebugLabel}, + {"nvnTextureReadTexels", T_nvnTextureReadTexels}, + {"nvnTextureReadTexelsStrided", T_nvnTextureReadTexelsStrided}, + {"nvnTextureSetDebugLabel", T_nvnTextureSetDebugLabel}, + {"nvnTextureViewCompare", T_nvnTextureViewCompare}, + {"nvnTextureViewGetDepthStencilMode", T_nvnTextureViewGetDepthStencilMode}, + {"nvnTextureViewGetFormat", T_nvnTextureViewGetFormat}, + {"nvnTextureViewGetLayers", T_nvnTextureViewGetLayers}, + {"nvnTextureViewGetLevels", T_nvnTextureViewGetLevels}, + {"nvnTextureViewGetSwizzle", T_nvnTextureViewGetSwizzle}, + {"nvnTextureViewGetTarget", T_nvnTextureViewGetTarget}, + {"nvnTextureViewSetDefaults", T_nvnTextureViewSetDefaults}, + {"nvnTextureViewSetDepthStencilMode", T_nvnTextureViewSetDepthStencilMode}, + {"nvnTextureViewSetFormat", T_nvnTextureViewSetFormat}, + {"nvnTextureViewSetLayers", T_nvnTextureViewSetLayers}, + {"nvnTextureViewSetLevels", T_nvnTextureViewSetLevels}, + {"nvnTextureViewSetSwizzle", T_nvnTextureViewSetSwizzle}, + {"nvnTextureViewSetTarget", T_nvnTextureViewSetTarget}, + {"nvnTextureWriteTexels", T_nvnTextureWriteTexels}, + {"nvnTextureWriteTexelsStrided", T_nvnTextureWriteTexelsStrided}, + {"nvnVertexAttribStateGetFormat", T_nvnVertexAttribStateGetFormat}, + {"nvnVertexAttribStateGetStreamIndex", T_nvnVertexAttribStateGetStreamIndex}, + {"nvnVertexAttribStateSetDefaults", T_nvnVertexAttribStateSetDefaults}, + {"nvnVertexAttribStateSetFormat", T_nvnVertexAttribStateSetFormat}, + {"nvnVertexAttribStateSetStreamIndex", T_nvnVertexAttribStateSetStreamIndex}, + {"nvnVertexStreamStateGetDivisor", T_nvnVertexStreamStateGetDivisor}, + {"nvnVertexStreamStateGetStride", T_nvnVertexStreamStateGetStride}, + {"nvnVertexStreamStateSetDefaults", T_nvnVertexStreamStateSetDefaults}, + {"nvnVertexStreamStateSetDivisor", T_nvnVertexStreamStateSetDivisor}, + {"nvnVertexStreamStateSetStride", T_nvnVertexStreamStateSetStride}, + {"nvnWindowAcquireTexture", T_nvnWindowAcquireTexture}, + {"nvnWindowBuilderGetDevice", T_nvnWindowBuilderGetDevice}, + {"nvnWindowBuilderGetNativeWindow", T_nvnWindowBuilderGetNativeWindow}, + {"nvnWindowBuilderGetNumActiveTextures", T_nvnWindowBuilderGetNumActiveTextures}, + {"nvnWindowBuilderGetNumTextures", T_nvnWindowBuilderGetNumTextures}, + {"nvnWindowBuilderGetPresentInterval", T_nvnWindowBuilderGetPresentInterval}, + {"nvnWindowBuilderGetTexture", T_nvnWindowBuilderGetTexture}, + {"nvnWindowBuilderSetDefaults", T_nvnWindowBuilderSetDefaults}, + {"nvnWindowBuilderSetDevice", T_nvnWindowBuilderSetDevice}, + {"nvnWindowBuilderSetNativeWindow", T_nvnWindowBuilderSetNativeWindow}, + {"nvnWindowBuilderSetNumActiveTextures", T_nvnWindowBuilderSetNumActiveTextures}, + {"nvnWindowBuilderSetPresentInterval", T_nvnWindowBuilderSetPresentInterval}, + {"nvnWindowBuilderSetTextures", T_nvnWindowBuilderSetTextures}, + {"nvnWindowGetCrop", T_nvnWindowGetCrop}, + {"nvnWindowGetNativeWindow", T_nvnWindowGetNativeWindow}, + {"nvnWindowGetNumActiveTextures", T_nvnWindowGetNumActiveTextures}, + {"nvnWindowGetNumTextures", T_nvnWindowGetNumTextures}, + {"nvnWindowGetPresentInterval", T_nvnWindowGetPresentInterval}, + {"nvnWindowInitialize", T_nvnWindowInitialize}, + {"nvnWindowSetCrop", T_nvnWindowSetCrop}, + {"nvnWindowSetDebugLabel", T_nvnWindowSetDebugLabel}, + {"nvnWindowSetNumActiveTextures", T_nvnWindowSetNumActiveTextures}, + {"nvnWindowSetPresentInterval", T_nvnWindowSetPresentInterval}, +}; +const unsigned nvn_thunk_count = 421; diff --git a/src/gfx/nvn/totk_lifelog.h b/src/gfx/nvn/totk_lifelog.h new file mode 100644 index 0000000..82a3300 --- /dev/null +++ b/src/gfx/nvn/totk_lifelog.h @@ -0,0 +1,21 @@ +// TOTK diagnostic DM_NVN_LIFELOG=WxH[:nvnfmt] (e.g. 1600x900:0xa): CPU-side timeline of the textures of that size +// (and format), to follow one resource through the game's frame pipeline. Logged: Initialize / Finalize, texture-pool +// registrations, render targets and texture handles recorded into command buffers, EndRecording / CallCommands of the +// command buffers that recorded them, every queue submission, and (replay side) the texture each draw resolves for a +// handle. Lines carry the time, the thread and the CPU frame (presents so far), so the order across the game's threads +// and submissions can be read off. Off by default; costs one branch per hooked call when off. +#pragma once +#include +#include "nvn_obj.h" + +int totk_life_on(void); +int totk_life_match(const tex_t *t); +void totk_life_log(const char *fmt, ...) __attribute__((format(printf, 1, 2))); +// texture-pool shadow: the serial of the latest matching registration of an id (0 = none or not matching) +void totk_life_reg(int id, const tex_t *t); +uint64_t totk_life_reg_serial(int id); +// command buffers / handles that carried a matching texture, so their EndRecording / CallCommands are logged +void totk_life_cb_mark(const void *cb); +int totk_life_cb_take(const void *cb); +void totk_life_handle_add(uint64_t h); +int totk_life_handle_has(uint64_t h); diff --git a/src/gfx/nvn/totk_mfx_mode.h b/src/gfx/nvn/totk_mfx_mode.h new file mode 100644 index 0000000..e7b70ee --- /dev/null +++ b/src/gfx/nvn/totk_mfx_mode.h @@ -0,0 +1,5 @@ +// TOTK: MetalFX frame interpolation modes, shared by totk_mfx.m (Objective-C) and the pacing in hle_nvn.c (C). +// Key 2 cycles off -> 30+30 -> half the refresh rate (50+50 at 100 Hz) -> uncapped -> off (user, 2026-10-16 night). +#pragma once +enum { MFX_OFF, MFX_30, MFX_HALF, MFX_MAX, MFX_NMODES }; +int mfx_enabled(void); // the mode of this game frame (0 = off) diff --git a/src/gfx/nvn/totk_nvn_ext.c b/src/gfx/nvn/totk_nvn_ext.c new file mode 100644 index 0000000..265071f --- /dev/null +++ b/src/gfx/nvn/totk_nvn_ext.c @@ -0,0 +1,275 @@ +#include +// TOTK: NVN entry points beyond the donor implementation that only report stored object state, +// attach debugging metadata, or convert timestamps. None of them changes rendering results. +#include "../host/hle.h" +#include "nvn_impl.h" +#include "nvn_obj.h" +#include "nvn_tex.h" +#include +#include + +// ---------- Debug labels, markers, groups, checkpoints: meaningful only to NVN debugging tools ---------- +NVN(nvnDeviceSetDebugLabel, void, NVNdevice *o, const char *s) { (void)o; (void)s; } +NVN(nvnQueueSetDebugLabel, void, NVNqueue *o, const char *s) { (void)o; (void)s; } +NVN(nvnWindowSetDebugLabel, void, NVNwindow *o, const char *s) { (void)o; (void)s; } +NVN(nvnProgramSetDebugLabel, void, NVNprogram *o, const char *s) { (void)o; (void)s; } +NVN(nvnMemoryPoolSetDebugLabel, void, NVNmemoryPool *o, const char *s) { (void)o; (void)s; } +NVN(nvnTexturePoolSetDebugLabel, void, NVNtexturePool *o, const char *s) { (void)o; (void)s; } +NVN(nvnSamplerPoolSetDebugLabel, void, NVNsamplerPool *o, const char *s) { (void)o; (void)s; } +NVN(nvnBufferSetDebugLabel, void, NVNbuffer *o, const char *s) { (void)o; (void)s; } +NVN(nvnTextureSetDebugLabel, void, NVNtexture *o, const char *s) { (void)o; (void)s; } +NVN(nvnSamplerSetDebugLabel, void, NVNsampler *o, const char *s) { (void)o; (void)s; } +NVN(nvnCommandBufferSetDebugLabel, void, NVNcommandBuffer *o, const char *s) { (void)o; (void)s; } +NVN(nvnSyncSetDebugLabel, void, NVNsync *o, const char *s) { (void)o; (void)s; } +NVN(nvnCommandBufferPushDebugGroupStatic, void, NVNcommandBuffer *c, uint32_t id, const char *s) { (void)c; (void)id; (void)s; } +NVN(nvnCommandBufferPushDebugGroupDynamic, void, NVNcommandBuffer *c, uint32_t id, const char *s) { (void)c; (void)id; (void)s; } +NVN(nvnCommandBufferPushDebugGroup, void, NVNcommandBuffer *c, uint32_t id, const char *s) { (void)c; (void)id; (void)s; } +NVN(nvnCommandBufferPopDebugGroup, void, NVNcommandBuffer *c) { (void)c; } +NVN(nvnCommandBufferPopDebugGroupId, void, NVNcommandBuffer *c, uint32_t id) { (void)c; (void)id; } +NVN(nvnCommandBufferInsertDebugMarkerStatic, void, NVNcommandBuffer *c, uint32_t id, const char *s) { (void)c; (void)id; (void)s; } +NVN(nvnCommandBufferInsertDebugMarkerDynamic, void, NVNcommandBuffer *c, uint32_t id, const char *s) { (void)c; (void)id; (void)s; } +NVN(nvnCommandBufferInsertDebugMarker, void, NVNcommandBuffer *c, const char *s) { (void)c; (void)s; } +NVN_NOPROTO(nvnCommandBufferInsertCheckpointStatic, void, NVNcommandBuffer *c, uint32_t id, const char *s) { (void)c; (void)id; (void)s; } +NVN_NOPROTO(nvnCommandBufferInsertCheckpointDynamic, void, NVNcommandBuffer *c, uint32_t id, const char *s) { (void)c; (void)id; (void)s; } +NVN(nvnDeviceIsExternalDebuggerAttached, NVNboolean, const NVNdevice *d) { (void)d; return 0; } + +// ---------- Timestamps: counters are written in Tegra 19.2 MHz ticks (gpu_ticks in nvn_mtl.m) ---------- +NVN(nvnDeviceGetCurrentTimestampInNanoseconds, uint64_t, const NVNdevice *d) { + (void)d; mach_timebase_info_data_t tb; mach_timebase_info(&tb); return (uint64_t)((__uint128_t)mach_absolute_time() * tb.numer / tb.denom); // TOTK: GetSystemTick's base (mach_absolute_time, no sleep), see gpu_ticks +} +NVN(nvnDeviceGetTimestampInNanoseconds, uint64_t, const NVNdevice *d, const NVNcounterData *c) { + (void)d; return c->timestamp / 12 * 625 + c->timestamp % 12 * 625 / 12; +} + +// ---------- Memory pools and buffers: unified memory, CPU and GPU views are coherent ---------- +NVN(nvnMemoryPoolGetSize, size_t, const NVNmemoryPool *p) { return (size_t)((const pool_t *)p)->size; } +// The CPU is about to read what the GPU wrote: GPU renderings into this pool's textures are written back (nvn_mtl.m). +void mtl_pool_invalidate(const pool_t *p, uint64_t off, uint64_t n); +NVN(nvnMemoryPoolInvalidateMappedRange, void, const NVNmemoryPool *p, ptrdiff_t off, size_t n) { if (off >= 0) mtl_pool_invalidate((const pool_t *)p, (uint64_t)off, n); } +NVN(nvnBufferInvalidateMappedRange, void, const NVNbuffer *b, ptrdiff_t off, size_t n) { (void)b; (void)off; (void)n; } + +// ---------- Texture builder and texture queries ---------- +#define CTB(b) ((const tex_builder_t *)(b)) +#define CTX(t) ((const tex_t *)(t)) +static void get_swizzle(const nvn_texdesc *d, NVNtextureSwizzle *r, NVNtextureSwizzle *g, NVNtextureSwizzle *b, NVNtextureSwizzle *a) { + *r = (NVNtextureSwizzle)d->swizzle_r; *g = (NVNtextureSwizzle)d->swizzle_g; *b = (NVNtextureSwizzle)d->swizzle_b; *a = (NVNtextureSwizzle)d->swizzle_a; +} +NVN_NOPROTO(nvnTextureBuilderGetDevice, NVNdevice *, const NVNtextureBuilder *b) { return CTB(b)->dev; } +NVN(nvnTextureBuilderGetTarget, NVNtextureTarget, const NVNtextureBuilder *b) { return (NVNtextureTarget)CTB(b)->d.target; } +NVN(nvnTextureBuilderGetWidth, int, const NVNtextureBuilder *b) { return (int)CTB(b)->d.width; } +NVN(nvnTextureBuilderGetHeight, int, const NVNtextureBuilder *b) { return (int)CTB(b)->d.height; } +NVN(nvnTextureBuilderGetDepth, int, const NVNtextureBuilder *b) { return (int)CTB(b)->d.depth; } +NVN(nvnTextureBuilderGetLevels, int, const NVNtextureBuilder *b) { return (int)CTB(b)->d.levels; } +NVN(nvnTextureBuilderGetFormat, NVNformat, const NVNtextureBuilder *b) { return (NVNformat)CTB(b)->d.format; } +NVN(nvnTextureBuilderGetSamples, int, const NVNtextureBuilder *b) { return CTB(b)->d.samples; } +NVN(nvnTextureBuilderGetSwizzle, void, const NVNtextureBuilder *b, NVNtextureSwizzle *r, NVNtextureSwizzle *g, NVNtextureSwizzle *bl, NVNtextureSwizzle *a) { get_swizzle(&CTB(b)->d, r, g, bl, a); } +NVN(nvnTextureBuilderGetDepthStencilMode, NVNtextureDepthStencilMode, const NVNtextureBuilder *b) { (void)b; return NVN_TEXTURE_DEPTH_STENCIL_MODE_DEPTH; } +NVN(nvnTextureBuilderGetStride, ptrdiff_t, const NVNtextureBuilder *b) { return (ptrdiff_t)CTB(b)->d.stride; } +NVN_NOPROTO(nvnTextureBuilderGetMemoryPool, const NVNmemoryPool *, const NVNtextureBuilder *b) { return (const NVNmemoryPool *)(const void *)CTB(b)->pool; } +NVN(nvnTextureBuilderGetMemoryOffset, ptrdiff_t, const NVNtextureBuilder *b) { return (ptrdiff_t)CTB(b)->off; } +NVN(nvnTextureGetTarget, NVNtextureTarget, const NVNtexture *t) { return (NVNtextureTarget)CTX(t)->d.target; } +NVN(nvnTextureGetLevels, int, const NVNtexture *t) { return (int)CTX(t)->d.levels; } +NVN(nvnTextureGetFormat, NVNformat, const NVNtexture *t) { return (NVNformat)CTX(t)->d.format; } +NVN(nvnTextureGetSamples, int, const NVNtexture *t) { return CTX(t)->d.samples; } +NVN(nvnTextureGetSwizzle, void, const NVNtexture *t, NVNtextureSwizzle *r, NVNtextureSwizzle *g, NVNtextureSwizzle *b, NVNtextureSwizzle *a) { get_swizzle(&CTX(t)->d, r, g, b, a); } +NVN(nvnTextureGetDepthStencilMode, NVNtextureDepthStencilMode, const NVNtexture *t) { (void)t; return NVN_TEXTURE_DEPTH_STENCIL_MODE_DEPTH; } +NVN(nvnTextureGetStride, ptrdiff_t, const NVNtexture *t) { return (ptrdiff_t)CTX(t)->d.stride; } +NVN(nvnTextureGetTextureAddress, NVNtextureAddress, const NVNtexture *t) { return (NVNtextureAddress)(uintptr_t)CTX(t)->mem; } +NVN_NOPROTO(nvnTextureGetMemoryPool, const NVNmemoryPool *, const NVNtexture *t) { return (const NVNmemoryPool *)(const void *)CTX(t)->pool; } +NVN(nvnTextureGetMemoryOffset, ptrdiff_t, const NVNtexture *t) { return (ptrdiff_t)CTX(t)->off; } +NVN(nvnTextureGetDebugID, uint64_t, const NVNtexture *t) { return CTX(t)->serial; } + +// ---------- Window queries ---------- +#define CWIN(w) ((const win_t *)(w)) +#define CWB(b) ((const win_builder_t *)(b)) +NVN_NOPROTO(nvnWindowBuilderGetDevice, NVNdevice *, const NVNwindowBuilder *b) { return CWB(b)->dev; } +NVN(nvnWindowBuilderGetNativeWindow, NVNnativeWindow, const NVNwindowBuilder *b) { return (NVNnativeWindow)CWB(b)->native; } +NVN(nvnWindowBuilderGetPresentInterval, int, const NVNwindowBuilder *b) { return CWB(b)->interval; } +NVN_NOPROTO(nvnWindowBuilderGetNumTextures, int, const NVNwindowBuilder *b) { return CWB(b)->ntex; } +NVN_NOPROTO(nvnWindowBuilderGetTexture, NVNtexture *, const NVNwindowBuilder *b, int i) { return i >= 0 && i < CWB(b)->ntex ? CWB(b)->tex[i] : NULL; } +NVN(nvnWindowGetNativeWindow, NVNnativeWindow, const NVNwindow *w) { return (NVNnativeWindow)CWIN(w)->native; } +NVN(nvnWindowGetPresentInterval, int, const NVNwindow *w) { return CWIN(w)->interval; } +NVN_NOPROTO(nvnWindowGetNumTextures, int, const NVNwindow *w) { return CWIN(w)->ntex; } +NVN(nvnWindowGetCrop, void, const NVNwindow *w, NVNrectangle *r) { + const win_t *o = CWIN(w); + if (!o->crop[2] && !o->crop[3] && o->ntex) { *r = (NVNrectangle){0, 0, (int)CTX(o->tex[0])->d.width, (int)CTX(o->tex[0])->d.height}; return; } // default crop: whole image + r->x = o->crop[0]; r->y = o->crop[1]; r->width = o->crop[2]; r->height = o->crop[3]; +} + +// ---------- Texture views ---------- +#define VW(v) ((view_t *)(v)) +#define CVW(v) ((const view_t *)(v)) +NVN(nvnTextureViewSetSwizzle, void, NVNtextureView *v, NVNtextureSwizzle r, NVNtextureSwizzle g, NVNtextureSwizzle b, NVNtextureSwizzle a) { + if ((unsigned)r > 5 || (unsigned)g > 5 || (unsigned)b > 5 || (unsigned)a > 5) die("invalid NVN texture view swizzle"); + VW(v)->set_swizzle = 1; VW(v)->swz[0] = (uint8_t)r; VW(v)->swz[1] = (uint8_t)g; VW(v)->swz[2] = (uint8_t)b; VW(v)->swz[3] = (uint8_t)a; +} +// Only depth-plane sampling is implemented (same restriction as the texture builder). +NVN(nvnTextureViewSetDepthStencilMode, void, NVNtextureView *v, NVNtextureDepthStencilMode m) { (void)v; if (m != NVN_TEXTURE_DEPTH_STENCIL_MODE_DEPTH) die("stencil-plane texture view sampling is not implemented"); } +NVN(nvnTextureViewGetLevels, NVNboolean, const NVNtextureView *v, int *base, int *n) { *base = CVW(v)->base_level; *n = CVW(v)->nlevels; return CVW(v)->set_levels; } +NVN(nvnTextureViewGetFormat, NVNboolean, const NVNtextureView *v, NVNformat *f) { *f = (NVNformat)CVW(v)->format; return CVW(v)->set_format; } +NVN(nvnTextureViewGetTarget, NVNboolean, const NVNtextureView *v, NVNtextureTarget *t) { *t = (NVNtextureTarget)CVW(v)->target; return CVW(v)->set_target; } +NVN(nvnTextureViewGetSwizzle, NVNboolean, const NVNtextureView *v, NVNtextureSwizzle *r, NVNtextureSwizzle *g, NVNtextureSwizzle *b, NVNtextureSwizzle *a) { + const view_t *o = CVW(v); + *r = (NVNtextureSwizzle)o->swz[0]; *g = (NVNtextureSwizzle)o->swz[1]; *b = (NVNtextureSwizzle)o->swz[2]; *a = (NVNtextureSwizzle)o->swz[3]; + return o->set_swizzle; +} +NVN(nvnTextureViewGetDepthStencilMode, NVNboolean, const NVNtextureView *v, NVNtextureDepthStencilMode *m) { (void)v; *m = NVN_TEXTURE_DEPTH_STENCIL_MODE_DEPTH; return 0; } +NVN(nvnTextureViewCompare, NVNboolean, const NVNtextureView *a, const NVNtextureView *b) { + const view_t *x = CVW(a), *y = CVW(b); + if (x->set_levels != y->set_levels || x->set_layers != y->set_layers || x->set_format != y->set_format || x->set_target != y->set_target || x->set_swizzle != y->set_swizzle) return 0; + if (x->set_levels && (x->base_level != y->base_level || x->nlevels != y->nlevels)) return 0; + if (x->set_layers && (x->min_layer != y->min_layer || x->nlayers != y->nlayers)) return 0; + if (x->set_format && x->format != y->format) return 0; + if (x->set_target && x->target != y->target) return 0; + return !x->set_swizzle || !memcmp(x->swz, y->swz, 4); +} + + +// ---------- Sampler queries ---------- +#define CSB(b) (&((const samp_builder_t *)(b))->p) +#define CSP(s) (&((const samp_t *)(s))->p) +#define MSB(b) (&((samp_builder_t *)(b))->p) +static void border_from_int(samp_par *p, const int32_t *c, int is_signed) { for (int i = 0; i < 4; i++) p->border[i] = is_signed ? (float)c[i] : (float)(uint32_t)c[i]; } +NVN(nvnSamplerBuilderSetBorderColori, void, NVNsamplerBuilder *b, const int *c) { border_from_int(MSB(b), c, 1); } +NVN(nvnSamplerBuilderSetBorderColorui, void, NVNsamplerBuilder *b, const uint32_t *c) { border_from_int(MSB(b), (const int32_t *)c, 0); } +NVN_NOPROTO(nvnSamplerBuilderGetDevice, NVNdevice *, const NVNsamplerBuilder *b) { return ((const samp_builder_t *)b)->dev; } +static void lod_clamp(const samp_par *p, float *lo, float *hi) { *lo = p->lod_min; *hi = p->lod_max; } +static void compare(const samp_par *p, NVNcompareMode *m, NVNcompareFunc *f) { *m = (NVNcompareMode)p->cmp_mode; *f = (NVNcompareFunc)p->cmp_func; } +static void border_i(const samp_par *p, int *c) { for (int i = 0; i < 4; i++) c[i] = (int)p->border[i]; } +static void border_ui(const samp_par *p, uint32_t *c) { for (int i = 0; i < 4; i++) c[i] = (uint32_t)p->border[i]; } +NVN(nvnSamplerBuilderGetLodClamp, void, const NVNsamplerBuilder *b, float *lo, float *hi) { lod_clamp(CSB(b), lo, hi); } +NVN(nvnSamplerBuilderGetLodBias, float, const NVNsamplerBuilder *b) { return CSB(b)->lod_bias; } +NVN(nvnSamplerBuilderGetCompare, void, const NVNsamplerBuilder *b, NVNcompareMode *m, NVNcompareFunc *f) { compare(CSB(b), m, f); } +NVN(nvnSamplerBuilderGetBorderColor, void, const NVNsamplerBuilder *b, float *c) { memcpy(c, CSB(b)->border, 16); } +NVN(nvnSamplerBuilderGetBorderColori, void, const NVNsamplerBuilder *b, int *c) { border_i(CSB(b), c); } +NVN(nvnSamplerBuilderGetBorderColorui, void, const NVNsamplerBuilder *b, uint32_t *c) { border_ui(CSB(b), c); } +NVN(nvnSamplerBuilderGetMaxAnisotropy, float, const NVNsamplerBuilder *b) { return CSB(b)->aniso; } +NVN(nvnSamplerBuilderGetLodSnap, float, const NVNsamplerBuilder *b) { return CSB(b)->lod_snap; } +NVN(nvnSamplerGetMinMagFilter, void, const NVNsampler *s, NVNminFilter *mi, NVNmagFilter *ma) { *mi = (NVNminFilter)CSP(s)->minf; *ma = (NVNmagFilter)CSP(s)->magf; } +NVN(nvnSamplerGetWrapMode, void, const NVNsampler *s, NVNwrapMode *a, NVNwrapMode *b, NVNwrapMode *c) { *a = (NVNwrapMode)CSP(s)->ws; *b = (NVNwrapMode)CSP(s)->wt; *c = (NVNwrapMode)CSP(s)->wr; } +NVN(nvnSamplerGetLodClamp, void, const NVNsampler *s, float *lo, float *hi) { lod_clamp(CSP(s), lo, hi); } +NVN(nvnSamplerGetLodBias, float, const NVNsampler *s) { return CSP(s)->lod_bias; } +NVN(nvnSamplerGetCompare, void, const NVNsampler *s, NVNcompareMode *m, NVNcompareFunc *f) { compare(CSP(s), m, f); } +NVN(nvnSamplerGetBorderColor, void, const NVNsampler *s, float *c) { memcpy(c, CSP(s)->border, 16); } +NVN(nvnSamplerGetBorderColori, void, const NVNsampler *s, int *c) { border_i(CSP(s), c); } +NVN(nvnSamplerGetBorderColorui, void, const NVNsampler *s, uint32_t *c) { border_ui(CSP(s), c); } +NVN(nvnSamplerGetMaxAnisotropy, float, const NVNsampler *s) { return CSP(s)->aniso; } +NVN(nvnSamplerCompare, NVNboolean, const NVNsampler *a, const NVNsampler *b) { return !memcmp(CSP(a), CSP(b), sizeof(samp_par)); } +NVN(nvnSamplerGetDebugID, uint64_t, const NVNsampler *s) { return (uint64_t)(uintptr_t)s; } + +// ---------- Fixed-function state objects: setters without a Metal equivalent store the value, and the +// replay reports any non-default use (nvn_mtl.m draw); getters return what was set. ---------- +#define CS(T, s) ((const T *)(const void *)(s)) +#define MS(T, s) ((T *)(void *)(s)) +NVN(nvnColorStateSetLogicOp, void, NVNcolorState *s, NVNlogicOp op) { if ((unsigned)op > 15) die("logic op %d", (int)op); MS(color_st, s)->logic_op = op; } +NVN(nvnColorStateSetAlphaTest, void, NVNcolorState *s, NVNalphaFunc f) { if ((unsigned)f > 15) die("alpha func %d", (int)f); MS(color_st, s)->alpha_func = f; } +NVN(nvnColorStateGetBlendEnable, NVNboolean, const NVNcolorState *s, int i) { return i >= 0 && i < 8 && (CS(color_st, s)->blend_en >> i & 1); } +NVN(nvnColorStateGetLogicOp, NVNlogicOp, const NVNcolorState *s) { return (NVNlogicOp)CS(color_st, s)->logic_op; } +NVN(nvnColorStateGetAlphaTest, NVNalphaFunc, const NVNcolorState *s) { return (NVNalphaFunc)CS(color_st, s)->alpha_func; } +NVN(nvnChannelMaskStateGetChannelMask, void, const NVNchannelMaskState *s, int i, NVNboolean *r, NVNboolean *g, NVNboolean *b, NVNboolean *a) { + uint32_t m = i >= 0 && i < 8 ? CS(chmask_st, s)->mask >> (4 * i) : 0; *r = m & 1; *g = m >> 1 & 1; *b = m >> 2 & 1; *a = m >> 3 & 1; +} +NVN(nvnBlendStateGetBlendTarget, int, const NVNblendState *s) { return (int)CS(blend_st, s)->target; } +NVN(nvnBlendStateGetBlendFunc, void, const NVNblendState *s, NVNblendFunc *sr, NVNblendFunc *dr, NVNblendFunc *sa, NVNblendFunc *da) { + const blend_st *b = CS(blend_st, s); *sr = (NVNblendFunc)b->src_rgb; *dr = (NVNblendFunc)b->dst_rgb; *sa = (NVNblendFunc)b->src_a; *da = (NVNblendFunc)b->dst_a; +} +NVN(nvnBlendStateGetBlendEquation, void, const NVNblendState *s, NVNblendEquation *rgb, NVNblendEquation *a) { *rgb = (NVNblendEquation)CS(blend_st, s)->eq_rgb; *a = (NVNblendEquation)CS(blend_st, s)->eq_a; } +NVN(nvnBlendStateSetAdvancedMode, void, NVNblendState *s, NVNblendAdvancedMode m) { (void)s; if (m) die("advanced blend mode %d not implemented", (int)m); } +NVN(nvnBlendStateSetAdvancedOverlap, void, NVNblendState *s, NVNblendAdvancedOverlap o) { (void)s; (void)o; } /* only meaningful with an advanced mode */ +NVN(nvnBlendStateSetAdvancedPremultipliedSrc, void, NVNblendState *s, uint32_t v) { (void)s; (void)v; } +NVN(nvnBlendStateSetAdvancedNormalizedDst, void, NVNblendState *s, uint32_t v) { (void)s; (void)v; } +NVN(nvnMultisampleStateGetMultisampleEnable, NVNboolean, const NVNmultisampleState *s) { return CS(ms_st, s)->enable; } +NVN(nvnMultisampleStateGetSamples, int, const NVNmultisampleState *s) { return CS(ms_st, s)->samples; } +NVN(nvnMultisampleStateGetAlphaToCoverageEnable, NVNboolean, const NVNmultisampleState *s) { return CS(ms_st, s)->a2c; } +NVN(nvnMultisampleStateGetRasterSamples, int, NVNmultisampleState *s) { return CS(ms_st, s)->raster_samples; } +NVN(nvnMultisampleStateSetRasterSamples, void, NVNmultisampleState *s, int n) { if (n < 0 || n > 16) die("raster samples %d", n); MS(ms_st, s)->raster_samples = (uint8_t)n; } +NVN(nvnMultisampleStateSetAlphaToCoverageDither, void, NVNmultisampleState *s, uint32_t e) { MS(ms_st, s)->a2c_dither = (e & 0xff) != 0; } +NVN(nvnPolygonStateGetCullFace, NVNface, const NVNpolygonState *s) { return (NVNface)CS(poly_st, s)->cull; } +NVN(nvnPolygonStateGetFrontFace, NVNfrontFace, const NVNpolygonState *s) { return (NVNfrontFace)CS(poly_st, s)->front; } +NVN(nvnPolygonStateGetPolygonMode, NVNpolygonMode, const NVNpolygonState *s) { return (NVNpolygonMode)CS(poly_st, s)->mode; } +NVN(nvnPolygonStateGetPolygonOffsetEnables, NVNpolygonOffsetEnable, const NVNpolygonState *s) { return (NVNpolygonOffsetEnable)CS(poly_st, s)->offset_en; } +NVN(nvnDepthStencilStateGetDepthTestEnable, NVNboolean, const NVNdepthStencilState *s) { return CS(ds_st, s)->dt; } +NVN(nvnDepthStencilStateGetDepthWriteEnable, NVNboolean, const NVNdepthStencilState *s) { return CS(ds_st, s)->dw; } +NVN(nvnDepthStencilStateGetDepthFunc, NVNdepthFunc, const NVNdepthStencilState *s) { return (NVNdepthFunc)CS(ds_st, s)->df; } +NVN(nvnDepthStencilStateGetStencilTestEnable, NVNboolean, const NVNdepthStencilState *s) { return CS(ds_st, s)->st; } +NVN(nvnDepthStencilStateGetStencilFunc, NVNstencilFunc, const NVNdepthStencilState *s, NVNface f) { return (NVNstencilFunc)(f == NVN_FACE_BACK ? CS(ds_st, s)->sf_b : CS(ds_st, s)->sf_f); } +NVN(nvnDepthStencilStateGetStencilOp, void, const NVNdepthStencilState *s, NVNface f, NVNstencilOp *fail, NVNstencilOp *zf, NVNstencilOp *zp) { + const ds_st *d = CS(ds_st, s); int back = f == NVN_FACE_BACK; + *fail = (NVNstencilOp)(back ? d->fail_b : d->fail_f); *zf = (NVNstencilOp)(back ? d->zfail_b : d->zfail_f); *zp = (NVNstencilOp)(back ? d->zpass_b : d->zpass_f); +} +NVN(nvnVertexAttribStateGetFormat, void, const NVNvertexAttribState *s, NVNformat *f, ptrdiff_t *off) { *f = (NVNformat)CS(va_st, s)->fmt; *off = (ptrdiff_t)CS(va_st, s)->offset; } +NVN(nvnVertexAttribStateGetStreamIndex, int, const NVNvertexAttribState *s) { return (int)CS(va_st, s)->stream; } +NVN(nvnVertexStreamStateGetStride, ptrdiff_t, const NVNvertexStreamState *s) { return (ptrdiff_t)CS(vs_st, s)->stride; } +NVN(nvnVertexStreamStateGetDivisor, int, const NVNvertexStreamState *s) { return (int)CS(vs_st, s)->divisor; } + +// ---------- Rasterisation state without a Metal equivalent: lines are 1 pixel wide and point size comes from +// the vertex shader on Metal. Non-default values are reported once (the drawing proceeds without them). ---------- +NVN(nvnCommandBufferSetLineWidth, void, NVNcommandBuffer *c, float w) { + (void)c; if (w != 1.0f) { static int once; if (!once++) LOG("【红】SetLineWidth %g: Metal draws 1-pixel lines", w); } +} +NVN(nvnCommandBufferSetPointSize, void, NVNcommandBuffer *c, float sz) { + (void)c; if (sz != 1.0f) { static int once; if (!once++) LOG("【红】SetPointSize %g: Metal takes point size from the vertex shader", sz); } +} + +// ---------- Entry points outside main's name table (NVN 55 inline fast-path fallbacks and newer queries) ---------- +// The *Deferred binds are what the inline fast paths call when the command write pointer has no room +// (always, see cmd_t.inline_cur). Texture / image / separate handle binds read the game's handle array at submit, +// as the original driver does (hle_nvn.c nvn_bind_handles_deferred); buffer ranges still bind at record time. +#define CALL_SLOT(name) ((NVNT_##name)nvn_slot_fn(#name)) +void *nvn_slot_fn(const char *name); +void nvn_bind_handles_deferred(NVNcommandBuffer *c, int op, int st, int first, int n, const uint64_t *h, size_t hw_bytes); +NVN_EXTRA(nvnCommandBufferBindTexturesDeferred, void, NVNcommandBuffer *c, NVNshaderStage st, int first, int n, const NVNtextureHandle *h) { nvn_bind_handles_deferred(c, OP_BIND_TEX, st, first, n, (const uint64_t *)h, 0); } +NVN_EXTRA(nvnCommandBufferBindUniformBuffersDeferred, void, NVNcommandBuffer *c, NVNshaderStage st, int first, int n, const NVNbufferRange *r) { static int once; if (!once++) LOG("[nvn] BindUniformBuffersDeferred 首次调用（缓冲范围仍按录制时绑定）"); CALL_SLOT(nvnCommandBufferBindUniformBuffers)(c, st, first, n, r); } +NVN_EXTRA(nvnCommandBufferBindStorageBuffersDeferred, void, NVNcommandBuffer *c, NVNshaderStage st, int first, int n, const NVNbufferRange *r) { static int once; if (!once++) LOG("[nvn] BindStorageBuffersDeferred 首次调用（缓冲范围仍按录制时绑定）"); CALL_SLOT(nvnCommandBufferBindStorageBuffers)(c, st, first, n, r); } +NVN_EXTRA(nvnCommandBufferBindImagesDeferred, void, NVNcommandBuffer *c, NVNshaderStage st, int first, int n, const NVNimageHandle *h) { nvn_bind_handles_deferred(c, OP_BIND_IMG, st, first, n, (const uint64_t *)h, 16 + 8 * (size_t)n); } +NVN_EXTRA(nvnCommandBufferBindSeparateTexturesDeferred, void, NVNcommandBuffer *c, NVNshaderStage st, int first, int n, const NVNseparateTextureHandle *h) { nvn_bind_handles_deferred(c, OP_BIND_TEX, st, first, n, (const uint64_t *)h, 16 + 8 * (size_t)n); } +NVN_EXTRA(nvnCommandBufferBindSeparateSamplersDeferred, void, NVNcommandBuffer *c, NVNshaderStage st, int first, int n, const NVNseparateSamplerHandle *h) { nvn_bind_handles_deferred(c, OP_BIND_SAMP, st, first, n, (const uint64_t *)h, 16 + 8 * (size_t)n); } +/* Memory callbacks are never needed: records live in host memory (hle_nvn.c cmd_reserve). */ +NVN_EXTRA(nvnCommandBufferSetControlMemoryCallbackEnabled, void, NVNcommandBuffer *c, NVNboolean e) { (void)c; (void)e; } +NVN_NOPROTO(nvnCommandBufferSetCommandMemoryCallbackEnabled, void, NVNcommandBuffer *c, NVNboolean e) { (void)c; (void)e; } +/* Tiled-cache subtile size, ZCull compression and debug checkpoints are hardware tuning / debugging hints. */ +NVN_EXTRA(nvnCommandBufferOverrideSubtileSize, void, NVNcommandBuffer *c) { (void)c; } +NVN_EXTRA(nvnCommandBufferSetZCullZF32CompressionEnable, void, NVNcommandBuffer *c, NVNboolean e) { (void)c; (void)e; } +NVN_EXTRA(nvnDeviceSetAutomaticCheckpointsMask, void, NVNdevice *d, uint32_t m) { (void)d; (void)m; } +/* NVN default provoking vertex is the last vertex (GL convention); Metal uses the first. Only the default is accepted. */ +NVN_EXTRA(nvnCommandBufferSetProvokingVertexMode, void, NVNcommandBuffer *c, int mode) { + (void)c; if (mode != 0) { static int once; if (!once++) LOG("【红】SetProvokingVertexMode %d: provoking-vertex conversion not implemented", mode); } +} +NVN_EXTRA(nvnTextureBuilderGetViewOffset, ptrdiff_t, const NVNtextureBuilder *b, const NVNtextureView *v) { + const tex_builder_t *o = CTB(b); const view_t *w = CVW(v); + if (!w) return 0; + nvn_texlayout L; nvn_tex_layout(&o->d, &L); + uint64_t off = w->set_levels ? nvn_tex_level_offset(&o->d, w->base_level) : 0; + if (w->set_layers) off += (uint64_t)w->min_layer * L.layer_size; + return (ptrdiff_t)off; +} +NVN_EXTRA(nvnProgramGetSubtileSize, void, const NVNprogram *p) { (void)p; die("nvnProgramGetSubtileSize: signature not identified"); } + +// ---------- ZCull (Tegra depth-cull cache): no Metal equivalent; zero storage is needed and save/restore have nothing to move ---------- +NVN(nvnTextureBuilderGetZCullStorageSize, size_t, const NVNtextureBuilder *b) { (void)b; return 0; } +NVN(nvnTextureGetZCullStorageSize, size_t, const NVNtexture *t) { (void)t; return 0; } +NVN(nvnCommandBufferSaveZCullData, void, NVNcommandBuffer *c, NVNbufferAddress a, size_t n) { (void)c; (void)a; (void)n; } +NVN(nvnCommandBufferRestoreZCullData, void, NVNcommandBuffer *c, NVNbufferAddress a, size_t n) { (void)c; (void)a; (void)n; } + +// ---------- Tegra tiled-cache and rasterisation hints ---------- +NVN(nvnCommandBufferSetTiledCacheAction, void, NVNcommandBuffer *c, NVNtiledCacheAction a) { (void)c; (void)a; } /* tiled-cache flush/enable hint */ +NVN(nvnCommandBufferSetTiledCacheTileSize, void, NVNcommandBuffer *c, int w, int h) { (void)c; (void)w; (void)h; } +NVN(nvnCommandBufferSetSubpixelPrecisionBias, void, NVNcommandBuffer *c, int x, int y) { (void)c; (void)x; (void)y; } +NVN(nvnCommandBufferDiscardColor, void, NVNcommandBuffer *c, int i) { (void)c; (void)i; } /* contents become undefined: keeping them is valid */ +NVN(nvnCommandBufferDiscardDepthStencil, void, NVNcommandBuffer *c) { (void)c; } +NVN(nvnCommandBufferSetConservativeRasterEnable, void, NVNcommandBuffer *c, uint32_t e) { + (void)c; if (e & 0xff) { static int once; if (!once++) LOG("【红】conservative rasterisation requested: not available on Metal (drawn normally)"); } +} +NVN(nvnCommandBufferSetConservativeRasterDilate, void, NVNcommandBuffer *c, float d) { (void)c; (void)d; } +NVN(nvnCommandBufferSetSampleMask, void, NVNcommandBuffer *c, int m) { + (void)c; if ((m & 0xffff) != 0xffff) { static int once; if (!once++) LOG("【红】SetSampleMask %#x: fixed-function sample mask not applied", m); } +} +NVN(nvnCommandBufferSetAlphaRef, void, NVNcommandBuffer *c, float r) { (void)c; (void)r; } /* only used with the alpha test, reported in the replay */ +/* Metal always restarts strips at the all-ones index; other restart indices are reported. */ +NVN(nvnCommandBufferSetPrimitiveRestart, void, NVNcommandBuffer *c, uint32_t e, int index) { + (void)c; if ((e & 0xff) && index != -1 && index != 0xffff) { static int once; if (!once++) LOG("【红】SetPrimitiveRestart index %#x: Metal restarts only at the all-ones index", index); } +} +/* Metal has no depth-bounds test. Disabling is exact; an enabled range is drawn without the test and reported. */ +NVN(nvnCommandBufferSetDepthBounds, void, NVNcommandBuffer *c, uint32_t e, float lo, float hi) { + (void)c; if ((e & 0xff) && (lo > 0.0f || hi < 1.0f)) { static int once; if (!once++) LOG("【红】SetDepthBounds [%g, %g]: depth-bounds test not available on Metal (fragments outside the range are kept)", lo, hi); } +} diff --git a/src/gfx/nvn/totk_timeline.h b/src/gfx/nvn/totk_timeline.h new file mode 100644 index 0000000..b7d2907 --- /dev/null +++ b/src/gfx/nvn/totk_timeline.h @@ -0,0 +1,27 @@ +/* TOTK_TIMELINE=:: (diagnostic, off by default): one timeline of a run of frames on one clock, + to see which link of the chain "game logic -> submit -> replay -> GPU -> the game waiting for that frame" sets the frame + time. Host events are CLOCK_UPTIME_RAW nanoseconds; GPU start / end of each committed command buffer come from Metal's + commit feedback (seconds on the same mach clock). Frames are counted by the game's presents (nvnQueuePresentTexture). + The events go to a preallocated buffer and are written to once, a few frames after the window + (scripts/diag/timeline.py reads it). */ +#pragma once +#include + +enum { + TL_SUB_IN = 1, TL_SUB_OUT, /* nvnQueueSubmitCommands enter / return (a = draws so far is not known: 0) */ + TL_REP_BEGIN, TL_REP_END, /* replay of a submission (submit_exec), on the game thread or the translation thread */ + TL_COMMIT, /* a command buffer committed: a = commit id, b = 1 last of its submission, 0 mid-submission */ + TL_GPU, /* GPU span of commit a: t = start, b = end (ns) */ + TL_WAIT_IN, TL_WAIT_OUT, /* nvnSyncWait that blocks: a = fence value waited for, b = 1 fence of the newest commit */ + TL_ACQ_IN, TL_ACQ_OUT, /* nvnWindowAcquireTexture */ + TL_PRES_IN, TL_PRES_OUT, /* nvnQueuePresentTexture */ + TL_FINISH_IN, TL_FINISH_OUT, /* nvnQueueFinish */ + TL_FENCE, /* fence value of commit a: b */ + TL_RS_JOIN, /* the replay thread waited a ns for the encode stage (b = join count) */ +}; + +int totk_tl_on(void); /* the window is open (cheap after the first call) */ +void totk_tl(int kind, uint64_t a, uint64_t b); /* event now, on this thread */ +void totk_tl_at(int kind, uint64_t t_ns, uint64_t a, uint64_t b); +void totk_tl_frame(uint64_t presents); /* the game presented: counts frames, opens / closes the window, writes the file */ +uint64_t totk_tl_commit_id(void); /* id for a command buffer about to be committed */ diff --git a/src/gfx/nvn/totk_timeline_win.c b/src/gfx/nvn/totk_timeline_win.c new file mode 100644 index 0000000..d605714 --- /dev/null +++ b/src/gfx/nvn/totk_timeline_win.c @@ -0,0 +1,80 @@ +// [Windows port] One-frame-chain timeline (totk_timeline.h), from MacNativeGames' totk_timeline.c (MIT). Changes: the +// host clock is runtime_now_ns (QueryPerformanceCounter), thread ids from GetCurrentThreadId; GPU spans come from the +// Vulkan back end (a timestamp at the start and end of each command buffer, placed so that it ends when the retire thread +// saw its fence). TOTK_TIMELINE=::; tools/dev_timeline.py reads the file. +#include +#include +#include +#include +#include + +#include "totk_timeline.h" + +uint64_t runtime_now_ns(void); + +typedef struct { uint64_t t, a, b; uint32_t kind, tid; uint64_t frame; } tl_ev; +enum { TL_CAP = 1 << 20 }; +static tl_ev *g_ev; +static _Atomic uint64_t g_n, g_commit, g_frame; +static _Atomic int g_state = -1; // -1 unread, 0 off, 1 waiting for the window, 2 recording, 3 written +static uint64_t g_first, g_count; +static char g_path[1024]; + +static void tl_init(void) { + const char *e = getenv("TOTK_TIMELINE"); + int st = 0; + if (e && *e) { // ::; the file may hold a drive letter's colon + const char *c1 = strrchr(e, ':'); + const char *c0 = c1 ? c1 - 1 : NULL; + while (c0 && c0 > e && *c0 != ':') c0--; + if (c1 && c0 && c0 > e && *c0 == ':' && (size_t)(c0 - e) < sizeof g_path) { + memcpy(g_path, e, (size_t)(c0 - e)); g_path[c0 - e] = 0; + g_first = strtoull(c0 + 1, NULL, 10); g_count = strtoull(c1 + 1, NULL, 10); + g_ev = calloc(TL_CAP, sizeof *g_ev); + if (g_ev && g_count) st = 1; + } + if (!st) fprintf(stderr, "[timeline] TOTK_TIMELINE=%s: want ::; not recording\n", e); + else fprintf(stderr, "[timeline] recording %llu frames from frame %llu into %s\n", (unsigned long long)g_count, (unsigned long long)g_first, g_path); + } + int want = -1; atomic_compare_exchange_strong(&g_state, &want, st); +} +int totk_tl_on(void) { + int s = atomic_load_explicit(&g_state, memory_order_relaxed); + if (s < 0) { tl_init(); s = atomic_load(&g_state); } + return s == 2; +} +void totk_tl_at(int kind, uint64_t t_ns, uint64_t a, uint64_t b) { + if (!totk_tl_on()) return; + uint64_t i = atomic_fetch_add_explicit(&g_n, 1, memory_order_relaxed); + if (i >= TL_CAP) return; + g_ev[i] = (tl_ev){t_ns, a, b, (uint32_t)kind, (uint32_t)GetCurrentThreadId(), atomic_load_explicit(&g_frame, memory_order_relaxed)}; +} +void totk_tl(int kind, uint64_t a, uint64_t b) { + if (!totk_tl_on()) return; + totk_tl_at(kind, runtime_now_ns(), a, b); +} +uint64_t totk_tl_commit_id(void) { return atomic_fetch_add(&g_commit, 1) + 1; } + +static void tl_write(void) { + FILE *f = fopen(g_path, "w"); + if (!f) { fprintf(stderr, "[timeline] cannot write %s\n", g_path); return; } + uint64_t n = atomic_load(&g_n); if (n > TL_CAP) n = TL_CAP; + fprintf(f, "# t_ns\tkind\ttid\tframe\ta\tb\n"); + static const char *names[] = {"?", "sub_in", "sub_out", "rep_begin", "rep_end", "commit", "gpu", "wait_in", "wait_out", + "acq_in", "acq_out", "pres_in", "pres_out", "finish_in", "finish_out", "fence", "rs_join"}; + for (uint64_t i = 0; i < n; i++) { + const tl_ev *e = &g_ev[i]; + fprintf(f, "%llu\t%s\t%u\t%llu\t%llu\t%llu\n", (unsigned long long)e->t, e->kind < sizeof names / sizeof *names ? names[e->kind] : "?", + e->tid, (unsigned long long)e->frame, (unsigned long long)e->a, (unsigned long long)e->b); + } + fclose(f); + fprintf(stderr, "[timeline] wrote %llu events to %s\n", (unsigned long long)n, g_path); +} +void totk_tl_frame(uint64_t presents) { + totk_tl_on(); + atomic_store(&g_frame, presents); + int s = atomic_load(&g_state); + if (s == 1 && presents >= g_first) atomic_store(&g_state, 2); + // GPU spans of the last frames arrive after their presents: 30 more frames, then the file + else if (s == 2 && presents >= g_first + g_count + 30) { atomic_store(&g_state, 3); tl_write(); } +} diff --git a/src/gfx/nvn/totk_vpool.h b/src/gfx/nvn/totk_vpool.h new file mode 100644 index 0000000..cc11eb7 --- /dev/null +++ b/src/gfx/nvn/totk_vpool.h @@ -0,0 +1,31 @@ +// TOTK: NVN physical / virtual memory pools backed by Darwin VM aliasing. +#pragma once +#include +#include "nvn_obj.h" + +// Runtime-verified layout of NVNmappingRequest (0x28 bytes, main nvnMemoryPoolMapVirtual call). +typedef struct { + const NVNmemoryPool *physical_pool; + int64_t physical_offset; + int64_t virtual_offset; + uint64_t size; + int32_t storage_class; + uint32_t padding; +} totk_mapping_request; +_Static_assert(sizeof(totk_mapping_request) == 0x28, "NVNmappingRequest stride"); + +#define TOTK_VPOOL_PAGE 0x10000ull // NVN virtual mapping granularity (64 KiB) + +// Called by MemoryPoolInitialize for PHYSICAL or VIRTUAL pools. Returns 0 with a logged reason on failure. +int totk_vpool_init(pool_t *pool); +// Called by MemoryPoolFinalize. Returns 1 when it released the pool's host storage. +int totk_vpool_fini(pool_t *pool); +// All requests are validated before any mapping changes; returns 0 and leaves mappings intact on failure. +int totk_vpool_map(pool_t *virtual_pool, int count, const totk_mapping_request *requests); +// TOTK diagnostics: CPU address of the physical backing of a virtual-pool range (1 = mapped, *cpu set). +int totk_vpool_backing(uintptr_t va, uint64_t len, uintptr_t *cpu); +// [va, va+len) of a virtual pool copied through its page table (unmapped pages: zeros); 0 = va is not in a virtual pool. +int totk_vpool_gather(uintptr_t va, uint64_t len, void *dst); +uint64_t totk_vpool_generation(void); // changes with every MapVirtual +// The physical memory behind a virtual-pool range written since stamp (nvn_dirty): 1 / 0; -1 = not in a virtual pool. +int totk_vpool_dirty_since(uintptr_t va, uint64_t len, uint64_t stamp); diff --git a/src/gfx/nvn/totk_vpool_win.c b/src/gfx/nvn/totk_vpool_win.c new file mode 100644 index 0000000..580c9f3 --- /dev/null +++ b/src/gfx/nvn/totk_vpool_win.c @@ -0,0 +1,172 @@ +// [Windows port] NVN physical / virtual memory pools (the API of MacNativeGames' totk_vpool.h). +// MacNativeGames maps a virtual pool's 64 KiB pages onto physical pools' pages a second time (mach_vm_remap). Windows +// cannot alias ordinary private memory, and a virtual pool is a GPU address range: the game reaches the physical memory +// through the physical pool, the GPU through the virtual one. So here a virtual pool is +// - a unique reserved, inaccessible address range (its GPU addresses; a CPU access faults visibly), and +// - a page table: virtual 64 KiB page -> (physical pool, offset), which the GPU back end resolves (later: Vulkan +// sparse binding of the physical pools' memory). +// Physical pools the CPU cannot access get host storage (as in MacNativeGames); CPU-visible ones keep the game storage. +#include +#include +#include +#include + +#include "../host/hle.h" +#include "nvn_dirty.h" +#include "nvn_mtl.h" +#include "totk_vpool.h" + +#define F_CPU_NO_ACCESS NVN_MEMORY_POOL_FLAGS_CPU_NO_ACCESS +#define F_PHYSICAL NVN_MEMORY_POOL_FLAGS_PHYSICAL +#define F_VIRTUAL NVN_MEMORY_POOL_FLAGS_VIRTUAL +#define POOL_MAGIC MAGIC('P', 'O', 'O', 'L') + +typedef struct { const pool_t *phys; uint64_t off; } vpage; // phys NULL: unmapped +typedef struct vtable { const pool_t *pool; uint64_t npages; vpage *pages; struct vtable *next; } vtable; +static vtable *g_tables; +static pthread_mutex_t g_map_mu = PTHREAD_MUTEX_INITIALIZER; +static _Atomic uint64_t g_map_gen = 1; // bumped by every MapVirtual: textures in virtual pools upload again + +static vtable *table_of(const pool_t *v) { + for (vtable *t = g_tables; t; t = t->next) if (t->pool == v) return t; + return NULL; +} + +int totk_vpool_init(pool_t *o) { + int phys = (o->flags & F_PHYSICAL) != 0, virt = (o->flags & F_VIRTUAL) != 0; + o->game_mem = o->mem; o->host = NULL; o->span = 0; + if (phys == virt) { LOG("vpool: pool flags %#x must be exactly one of PHYSICAL/VIRTUAL", o->flags); return 0; } + if (!o->size || o->size % TOTK_VPOOL_PAGE) { LOG("vpool: pool size %#llx not a multiple of 64 KiB", (unsigned long long)o->size); return 0; } + if (virt && o->mem) { LOG("vpool: virtual pool given storage %p", (void *)o->mem); return 0; } + if (phys && !o->mem) { LOG("vpool: physical pool without storage"); return 0; } + if (phys && !(o->flags & F_CPU_NO_ACCESS)) return 1; // the game storage itself + o->span = o->size; + if (phys) { // CPU_NO_ACCESS: our own storage + o->host = VirtualAlloc(NULL, o->span, MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE); + } else { // virtual: addresses only + o->host = VirtualAlloc(NULL, o->span, MEM_RESERVE, PAGE_NOACCESS); + vtable *t = calloc(1, sizeof *t); + if (!t) die("vpool: table allocation"); + t->pool = o; t->npages = o->size / TOTK_VPOOL_PAGE; + t->pages = calloc(t->npages, sizeof *t->pages); + if (!t->pages) die("vpool: page table allocation"); + pthread_mutex_lock(&g_map_mu); t->next = g_tables; g_tables = t; pthread_mutex_unlock(&g_map_mu); + } + if (!o->host) { LOG("vpool: VirtualAlloc(%#llx) failed (%lu)", (unsigned long long)o->span, GetLastError()); return 0; } + o->mem = o->host; + return 1; +} + +int totk_vpool_fini(pool_t *o) { + if (!o->host) return 0; + pthread_mutex_lock(&g_map_mu); + for (vtable **p = &g_tables; *p; p = &(*p)->next) + if ((*p)->pool == o) { vtable *t = *p; *p = t->next; free(t->pages); free(t); break; } + pthread_mutex_unlock(&g_map_mu); + VirtualFree(o->host, 0, MEM_RELEASE); + o->host = NULL; o->mem = o->game_mem; o->span = 0; + return 1; +} + +static int check_request(const pool_t *v, const totk_mapping_request *r, int i) { + uint64_t end = (uint64_t)r->virtual_offset + r->size; + if (r->virtual_offset < 0 || r->physical_offset < 0 || !r->size) { LOG("vpool: request %d negative/empty range", i); return 0; } + if ((r->virtual_offset | r->physical_offset | (int64_t)r->size) % TOTK_VPOOL_PAGE) { LOG("vpool: request %d not 64 KiB aligned", i); return 0; } + if (end < r->size || end > v->size) { LOG("vpool: request %d exceeds virtual pool", i); return 0; } + if (!r->physical_pool) return 1; // unmap + const pool_t *p = (const pool_t *)(const void *)r->physical_pool; + if (p->magic != POOL_MAGIC || !(p->flags & F_PHYSICAL)) { LOG("vpool: request %d source %p is not a physical pool", i, (const void *)p); return 0; } + uint64_t pend = (uint64_t)r->physical_offset + r->size; + if (pend < r->size || pend > p->size) { LOG("vpool: request %d exceeds physical pool", i); return 0; } + return 1; +} + +int totk_vpool_map(pool_t *v, int count, const totk_mapping_request *r) { + if (!v || v->magic != POOL_MAGIC || !(v->flags & F_VIRTUAL) || !v->mem) { LOG("vpool: MapVirtual target %p is not a virtual pool", (void *)v); return 0; } + if (count < 0 || (count && !r)) { LOG("vpool: invalid request count %d", count); return 0; } + for (int i = 0; i < count; i++) if (!check_request(v, &r[i], i)) return 0; + pthread_mutex_lock(&g_map_mu); + vtable *t = table_of(v); + if (!t) { pthread_mutex_unlock(&g_map_mu); die("vpool: virtual pool %p has no page table", (void *)v); } + for (int i = 0; i < count; i++) { + uint64_t first = (uint64_t)r[i].virtual_offset / TOTK_VPOOL_PAGE, n = r[i].size / TOTK_VPOOL_PAGE; + for (uint64_t k = 0; k < n; k++) + t->pages[first + k] = (vpage){(const pool_t *)(const void *)r[i].physical_pool, + r[i].physical_pool ? (uint64_t)r[i].physical_offset + k * TOTK_VPOOL_PAGE : 0}; + } + pthread_mutex_unlock(&g_map_mu); + atomic_fetch_add(&g_map_gen, 1); + return 1; +} + +uint64_t totk_vpool_generation(void) { return atomic_load(&g_map_gen); } + +// [va, va+len) of a virtual pool gathered through its page table into dst (unmapped pages read as zeros). +// 0: va is not in a virtual pool (the caller reads it directly). +int totk_vpool_gather(uintptr_t va, uint64_t len, void *dst) { + int found = 0; + pthread_mutex_lock(&g_map_mu); + for (vtable *t = g_tables; t; t = t->next) { + uintptr_t base = (uintptr_t)t->pool->mem; + if (va < base || va >= base + t->pool->size) continue; + found = 1; + for (uint64_t done = 0; done < len;) { + uint64_t at = va + done - base, page = at / TOTK_VPOOL_PAGE, in = at % TOTK_VPOOL_PAGE; + uint64_t n = TOTK_VPOOL_PAGE - in < len - done ? TOTK_VPOOL_PAGE - in : len - done; + const vpage *p = page < t->npages ? &t->pages[page] : NULL; + if (p && p->phys) memcpy((uint8_t *)dst + done, (const uint8_t *)p->phys->mem + p->off + in, n); + else memset((uint8_t *)dst + done, 0, n); + done += n; + } + break; + } + pthread_mutex_unlock(&g_map_mu); + return found; +} + +// Whether the physical memory behind [va, va+len) of a virtual pool was written since `stamp` (nvn_dirty generations). +// The virtual range itself is never written (no CPU access): the game writes the physical pools' memory, so a texture +// in a virtual pool is checked page by page there (TOTK's pre-rendered cutscenes: the movie frame, every frame). +// Untracked backing (CPU_NO_ACCESS pools: GPU data) counts as unchanged. -1: not in a virtual pool. +int totk_vpool_dirty_since(uintptr_t va, uint64_t len, uint64_t stamp) { + int found = -1; + pthread_mutex_lock(&g_map_mu); + for (vtable *t = g_tables; t; t = t->next) { + uintptr_t base = (uintptr_t)t->pool->mem; + if (va < base || va >= base + t->pool->size) continue; + found = 0; + for (uint64_t done = 0; done < len && !found;) { + uint64_t at = va + done - base, page = at / TOTK_VPOOL_PAGE, in = at % TOTK_VPOOL_PAGE; + uint64_t n = TOTK_VPOOL_PAGE - in < len - done ? TOTK_VPOOL_PAGE - in : len - done; + const vpage *p = page < t->npages ? &t->pages[page] : NULL; + if (p && p->phys) { + const uint8_t *cpu = (const uint8_t *)p->phys->mem + p->off + in; + found = nvn_dirty_tracked(cpu, n) && nvn_dirty_since(cpu, n, stamp); + } + done += n; + } + break; + } + pthread_mutex_unlock(&g_map_mu); + return found; +} + +// The CPU address of the physical backing of [va, va+len) in a virtual pool (1 = mapped and contiguous). +int totk_vpool_backing(uintptr_t va, uint64_t len, uintptr_t *cpu) { + int ok = 0; + pthread_mutex_lock(&g_map_mu); + for (vtable *t = g_tables; t; t = t->next) { + uintptr_t base = (uintptr_t)t->pool->mem; + if (va < base || va + len > base + t->pool->size) continue; + uint64_t page = (va - base) / TOTK_VPOOL_PAGE, last = (va + len - 1 - base) / TOTK_VPOOL_PAGE; + const vpage *p = &t->pages[page]; + if (!p->phys) break; + ok = 1; + for (uint64_t k = page + 1; k <= last && ok; k++) + ok = t->pages[k].phys == p->phys && t->pages[k].off == p->off + (k - page) * TOTK_VPOOL_PAGE; + if (ok) *cpu = (uintptr_t)p->phys->mem + p->off + (va - base) % TOTK_VPOOL_PAGE; + break; + } + pthread_mutex_unlock(&g_map_mu); + return ok; +} diff --git a/src/gfx/tex/LICENSE-LASRecomp b/src/gfx/tex/LICENSE-LASRecomp new file mode 100644 index 0000000..b0977fd --- /dev/null +++ b/src/gfx/tex/LICENSE-LASRecomp @@ -0,0 +1,24 @@ +MIT License + +Copyright (c) 2026 Peiikai + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +This license covers LAS Recomp's own code. It does not cover The Legend of Zelda: Link's Awakening or any other +Nintendo property, nor the third-party components in third_party/, which keep their own licenses. diff --git a/src/gfx/tex/LICENSE-MacNativeGames b/src/gfx/tex/LICENSE-MacNativeGames new file mode 100644 index 0000000..6102617 --- /dev/null +++ b/src/gfx/tex/LICENSE-MacNativeGames @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 MacNativeGames contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/src/gfx/tex/README.md b/src/gfx/tex/README.md new file mode 100644 index 0000000..ca090e1 --- /dev/null +++ b/src/gfx/tex/README.md @@ -0,0 +1,12 @@ +# Texture helpers (copied as is) + +| File | Origin | License | +|---|---|---| +| `nvn_deswizzle.c/.h` | MacNativeGames `games/totk/third_party/d2r_gfx/gfx/tex` @b75be8c | MIT (`LICENSE-MacNativeGames`) | +| `astc.c/.h` | LASRecomp `runtime/nvn` @da35b83 | MIT (`LICENSE-LASRecomp`) | +| `bc7enc/bc7enc.cpp/.h` | richgel999/bc7enc_rdo @b943862 (2026-07-30) | MIT (alternative A of `bc7enc/LICENSE`; Rich Geldreich) | +| `bc7enc/rgbcx.cpp/.h`, `rgbcx_table4*.h` | the same repository and commit (BC1 encoder) | MIT (the same `bc7enc/LICENSE`) | + +`nvn_deswizzle` uses `../nvn/nvn_tex.h` (also from MacNativeGames). ASTC decoding on the CPU is temporary: ASTC is +transcoded to BC7 (`bc7.cpp`, the port's own wrapper; cache: `../vkb_bc7.c`), filled at build time from the RomFS in the +next step. diff --git a/src/gfx/tex/astc.c b/src/gfx/tex/astc.c new file mode 100644 index 0000000..6847a7d --- /dev/null +++ b/src/gfx/tex/astc.c @@ -0,0 +1,530 @@ +// ASTC LDR decoder, 2D only (Khronos Data Format spec, ASTC chapter). Matches ARM astcenc's 8-bit +// decode bit for bit (the decode_unorm8 path its CLI uses for -dl / -ds into 8-bit images): +// endpoints expand to 16 bits (linear: v * 257, sRGB: v << 8 | 0x80, all four channels), +// texel = (e0 * (64 - w) + e1 * w + 32) >> 6, output = top 8 bits. +// Error blocks (illegal encodings, HDR void extents) -> magenta; HDR endpoint modes -> magenta +// endpoints for that partition only (as astcenc and the spec's LDR profile do). +#include "astc.h" + +#include + +// Quantization levels QUANT_2 .. QUANT_256 (index 0..20): bits / trits / quints per level. +static const uint8_t q_bits[21] = {1, 0, 2, 0, 1, 3, 1, 2, 4, 2, 3, 5, 3, 4, 6, 4, 5, 7, 5, 6, 8}; +static const uint8_t q_trits[21] = {0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0}; +static const uint8_t q_quints[21] = {0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0}; +enum { QUANT_6 = 4, QUANT_256 = 20 }; + +// Unquantization tables indexed by the raw ISE value (trit/quint << bits | bits), generated from the +// spec's formulas. Weights: QUANT_2..QUANT_32 -> 0..64. Colours: QUANT_6..QUANT_256 -> 0..255. +static const uint8_t wunq_data[] = { + 0,64, + 0,32,64, + 0,21,43,64, + 0,16,32,48,64, + 0,64,12,52,25,39, + 0,9,18,27,37,46,55,64, + 0,64,7,57,14,50,21,43,28,36, + 0,64,17,47,5,59,23,41,11,53,28,36, + 0,4,8,12,17,21,25,29,35,39,43,47,52,56,60,64, + 0,64,16,48,3,61,19,45,6,58,23,41,9,55,26,38,13,51,29,35, + 0,64,8,56,16,48,24,40,2,62,11,53,19,45,27,37,5,59,13,51,22,42,30,34, + 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64, +}; +static const uint16_t wunq_off[12] = {0,2,5,9,14,20,28,38,50,66,86,110}; +static const uint8_t cunq_data[] = { + /* 6 */ + 0,255,51,204,102,153, + /* 8 */ + 0,36,73,109,146,182,219,255, + /* 10 */ + 0,255,28,227,56,199,84,171,113,142, + /* 12 */ + 0,255,69,186,23,232,92,163,46,209,116,139, + /* 16 */ + 0,17,34,51,68,85,102,119,136,153,170,187,204,221,238,255, + /* 20 */ + 0,255,67,188,13,242,80,175,27,228,94,161,40,215,107,148,54,201,121,134, + /* 24 */ + 0,255,33,222,66,189,99,156,11,244,44,211,77,178,110,145,22,233,55,200,88,167,121,134, + /* 32 */ + 0,8,16,24,33,41,49,57,66,74,82,90,99,107,115,123,132,140,148,156,165,173,181,189,198,206,214,222, + 231,239,247,255, + /* 40 */ + 0,255,32,223,65,190,97,158,6,249,39,216,71,184,104,151,13,242,45,210,78,177,110,145,19,236,52,203, + 84,171,117,138,26,229,58,197,91,164,123,132, + /* 48 */ + 0,255,16,239,32,223,48,207,65,190,81,174,97,158,113,142,5,250,21,234,38,217,54,201,70,185,86,169, + 103,152,119,136,11,244,27,228,43,212,59,196,76,179,92,163,108,147,124,131, + /* 64 */ + 0,4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,65,69,73,77,81,85,89,93,97,101,105,109,113,117,121,125, + 130,134,138,142,146,150,154,158,162,166,170,174,178,182,186,190,195,199,203,207,211,215,219,223,227, + 231,235,239,243,247,251,255, + /* 80 */ + 0,255,16,239,32,223,48,207,64,191,80,175,96,159,112,143,3,252,19,236,35,220,51,204,67,188,83,172, + 100,155,116,139,6,249,22,233,38,217,54,201,71,184,87,168,103,152,119,136,9,246,25,230,42,213,58,197, + 74,181,90,165,106,149,122,133,13,242,29,226,45,210,61,194,77,178,93,162,109,146,125,130, + /* 96 */ + 0,255,8,247,16,239,24,231,32,223,40,215,48,207,56,199,64,191,72,183,80,175,88,167,96,159,104,151, + 112,143,120,135,2,253,10,245,18,237,26,229,35,220,43,212,51,204,59,196,67,188,75,180,83,172,91,164, + 99,156,107,148,115,140,123,132,5,250,13,242,21,234,29,226,37,218,45,210,53,202,61,194,70,185,78,177, + 86,169,94,161,102,153,110,145,118,137,126,129, + /* 128 */ + 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68, + 70,72,74,76,78,80,82,84,86,88,90,92,94,96,98,100,102,104,106,108,110,112,114,116,118,120,122,124, + 126,129,131,133,135,137,139,141,143,145,147,149,151,153,155,157,159,161,163,165,167,169,171,173,175, + 177,179,181,183,185,187,189,191,193,195,197,199,201,203,205,207,209,211,213,215,217,219,221,223,225, + 227,229,231,233,235,237,239,241,243,245,247,249,251,253,255, + /* 160 */ + 0,255,8,247,16,239,24,231,32,223,40,215,48,207,56,199,64,191,72,183,80,175,88,167,96,159,104,151, + 112,143,120,135,1,254,9,246,17,238,25,230,33,222,41,214,49,206,57,198,65,190,73,182,81,174,89,166, + 97,158,105,150,113,142,121,134,3,252,11,244,19,236,27,228,35,220,43,212,51,204,59,196,67,188,75,180, + 83,172,91,164,99,156,107,148,115,140,123,132,4,251,12,243,20,235,28,227,36,219,44,211,52,203,60,195, + 68,187,76,179,84,171,92,163,100,155,108,147,116,139,124,131,6,249,14,241,22,233,30,225,38,217,46, + 209,54,201,62,193,70,185,78,177,86,169,94,161,102,153,110,145,118,137,126,129, + /* 192 */ + 0,255,4,251,8,247,12,243,16,239,20,235,24,231,28,227,32,223,36,219,40,215,44,211,48,207,52,203,56, + 199,60,195,64,191,68,187,72,183,76,179,80,175,84,171,88,167,92,163,96,159,100,155,104,151,108,147, + 112,143,116,139,120,135,124,131,1,254,5,250,9,246,13,242,17,238,21,234,25,230,29,226,33,222,37,218, + 41,214,45,210,49,206,53,202,57,198,61,194,65,190,69,186,73,182,77,178,81,174,85,170,89,166,93,162, + 97,158,101,154,105,150,109,146,113,142,117,138,121,134,125,130,2,253,6,249,10,245,14,241,18,237,22, + 233,26,229,30,225,34,221,38,217,42,213,46,209,50,205,54,201,58,197,62,193,66,189,70,185,74,181,78, + 177,82,173,86,169,90,165,94,161,98,157,102,153,106,149,110,145,114,141,118,137,122,133,126,129, +}; +static const uint16_t cunq_off[16] = {0,6,14,24,36,52,72,96,128,168,216,280,360,456,584,744}; // QUANT_256: identity +_Static_assert(sizeof wunq_data == 142 && sizeof cunq_data == 936, "unquantization table size"); + +// ---- bits ---- + +typedef struct { uint64_t lo, hi; } Bits128; + +static inline uint64_t load64(const uint8_t *p) { + uint64_t v = 0; + for (int i = 7; i >= 0; i--) v = v << 8 | p[i]; + return v; +} + +// n (<= 32) bits at pos; bits past 127 read as zero. +static inline uint32_t rbits(const Bits128 *b, int pos, int n) { + if (n == 0 || pos >= 128) return 0; + uint64_t v; + if (pos >= 64) v = b->hi >> (pos - 64); + else if (pos == 0) v = b->lo; + else v = b->lo >> pos | b->hi << (64 - pos); + return (uint32_t)(v & ((1ull << n) - 1)); +} + +static inline uint64_t bitrev64(uint64_t v) { + v = (v >> 1 & 0x5555555555555555ull) | (v & 0x5555555555555555ull) << 1; + v = (v >> 2 & 0x3333333333333333ull) | (v & 0x3333333333333333ull) << 2; + v = (v >> 4 & 0x0F0F0F0F0F0F0F0Full) | (v & 0x0F0F0F0F0F0F0F0Full) << 4; + v = (v >> 8 & 0x00FF00FF00FF00FFull) | (v & 0x00FF00FF00FF00FFull) << 8; + v = (v >> 16 & 0x0000FFFF0000FFFFull) | (v & 0x0000FFFF0000FFFFull) << 16; + return v >> 32 | v << 32; +} + +// ---- integer sequence encoding ---- + +static int ise_bitcount(int count, int q) { + int n = count * q_bits[q]; + if (q_trits[q]) n += (8 * count + 4) / 5; + if (q_quints[q]) n += (7 * count + 2) / 3; + return n; +} + +static void trits_of(uint32_t T, int t[5]) { + uint32_t C; + if ((T >> 2 & 7) == 7) { + C = (T >> 5 & 7) << 2 | (T & 3); + t[4] = t[3] = 2; + } else { + C = T & 0x1F; + if ((T >> 5 & 3) == 3) { t[4] = 2; t[3] = T >> 7 & 1; } + else { t[4] = T >> 7 & 1; t[3] = T >> 5 & 3; } + } + if ((C & 3) == 3) { + t[2] = 2; t[1] = C >> 4 & 1; + t[0] = (C >> 3 & 1) << 1 | ((C >> 2 & 1) & ~(C >> 3) & 1); + } else if ((C >> 2 & 3) == 3) { + t[2] = 2; t[1] = 2; t[0] = C & 3; + } else { + t[2] = C >> 4 & 1; t[1] = C >> 2 & 3; + t[0] = (C >> 1 & 1) << 1 | ((C & 1) & ~(C >> 1) & 1); + } +} + +static void quints_of(uint32_t Q, int q[3]) { + if ((Q >> 1 & 3) == 3 && (Q >> 5 & 3) == 0) { + uint32_t q0 = Q & 1; + q[2] = (int)(q0 << 2 | ((Q >> 4 & 1) & ~q0 & 1) << 1 | ((Q >> 3 & 1) & ~q0 & 1)); + q[1] = q[0] = 4; + return; + } + uint32_t C; + if ((Q >> 1 & 3) == 3) { q[2] = 4; C = (Q >> 3 & 3) << 3 | (~(Q >> 5) & 3) << 1 | (Q & 1); } + else { q[2] = Q >> 5 & 3; C = Q & 0x1F; } + if ((C & 7) == 5) { q[1] = 4; q[0] = C >> 3 & 3; } + else { q[1] = C >> 3 & 3; q[0] = C & 7; } +} + +// Decodes count raw ISE values (tq << bits | bits) starting at bit pos. +static void ise_decode(const Bits128 *b, int pos, int count, int q, uint8_t *out) { + int nb = q_bits[q]; + if (q_trits[q]) { + static const uint8_t tb[5] = {2, 2, 1, 2, 1}, ts[5] = {0, 2, 4, 5, 7}; + for (int i = 0; i < count; i += 5) { + uint32_t m[5], T = 0; + int k = 0, t[5]; + for (; k < 5 && i + k < count; k++) { + m[k] = rbits(b, pos, nb); pos += nb; + T |= rbits(b, pos, tb[k]) << ts[k]; pos += tb[k]; + } + trits_of(T, t); + for (int j = 0; j < k; j++) out[i + j] = (uint8_t)((uint32_t)t[j] << nb | m[j]); + } + } else if (q_quints[q]) { + static const uint8_t qb[3] = {3, 2, 2}, qs[3] = {0, 3, 5}; + for (int i = 0; i < count; i += 3) { + uint32_t m[3], Q = 0; + int k = 0, v[3]; + for (; k < 3 && i + k < count; k++) { + m[k] = rbits(b, pos, nb); pos += nb; + Q |= rbits(b, pos, qb[k]) << qs[k]; pos += qb[k]; + } + quints_of(Q, v); + for (int j = 0; j < k; j++) out[i + j] = (uint8_t)((uint32_t)v[j] << nb | m[j]); + } + } else { + for (int i = 0; i < count; i++, pos += nb) out[i] = (uint8_t)rbits(b, pos, nb); + } +} + +// ---- block mode ---- + +typedef struct { int wx, wy, dual, quant, wbits; } BlockMode; + +static bool block_mode_2d(uint32_t m, uint32_t bw, uint32_t bh, BlockMode *bm) { + uint32_t r = m >> 4 & 1, h = m >> 9 & 1, d = m >> 10 & 1, a = m >> 5 & 3; + int wx, wy; + if (m & 3) { + r |= (m & 3) << 1; + uint32_t b = m >> 7 & 3; + switch (m >> 2 & 3) { + case 0: wx = b + 4; wy = a + 2; break; + case 1: wx = b + 8; wy = a + 2; break; + case 2: wx = a + 2; wy = b + 8; break; + default: + b &= 1; + if (m & 0x100) { wx = b + 2; wy = a + 2; } + else { wx = a + 2; wy = b + 6; } + break; + } + } else { + r |= (m >> 2 & 3) << 1; + if ((m >> 2 & 3) == 0) return false; + uint32_t b = m >> 9 & 3; + switch (m >> 7 & 3) { + case 0: wx = 12; wy = a + 2; break; + case 1: wx = a + 2; wy = 12; break; + case 2: wx = a + 6; wy = b + 6; d = 0; h = 0; break; + default: + if (a == 0) { wx = 6; wy = 10; } + else if (a == 1) { wx = 10; wy = 6; } + else return false; + break; + } + } + int count = wx * wy * (int)(d + 1); + bm->wx = wx; bm->wy = wy; bm->dual = (int)d; + bm->quant = (int)(r - 2 + 6 * h); + bm->wbits = ise_bitcount(count, bm->quant); + return count <= 64 && bm->wbits >= 24 && bm->wbits <= 96 && wx <= (int)bw && wy <= (int)bh; +} + +// ---- partitions ---- + +static uint32_t hash52(uint32_t p) { + p ^= p >> 15; p -= p << 17; p += p << 7; p += p << 4; + p ^= p >> 5; p += p << 16; p ^= p >> 7; p ^= p >> 3; + p ^= p << 6; p ^= p >> 17; + return p; +} + +typedef struct { int s[8]; uint32_t r[4]; int count; } PartSel; + +static void part_init(PartSel *ps, int seed, int count) { + seed += (count - 1) * 1024; + uint32_t rnum = hash52((uint32_t)seed); + int s[8]; + for (int i = 0; i < 8; i++) { int v = (int)(rnum >> (4 * i) & 0xF); s[i] = v * v; } + int sh1, sh2; + if (seed & 1) { sh1 = seed & 2 ? 4 : 5; sh2 = count == 3 ? 6 : 5; } + else { sh1 = count == 3 ? 6 : 5; sh2 = seed & 2 ? 4 : 5; } + for (int i = 0; i < 8; i++) ps->s[i] = s[i] >> (i & 1 ? sh2 : sh1); + ps->r[0] = rnum >> 14; ps->r[1] = rnum >> 10; ps->r[2] = rnum >> 6; ps->r[3] = rnum >> 2; + ps->count = count; +} + +static inline int part_of(const PartSel *ps, int x, int y) { + // z = 0 for 2D, so seeds 9..12 drop out + int a = (int)((uint32_t)(ps->s[0] * x + ps->s[1] * y) + ps->r[0]) & 0x3F; + int b = (int)((uint32_t)(ps->s[2] * x + ps->s[3] * y) + ps->r[1]) & 0x3F; + int c = ps->count >= 3 ? (int)((uint32_t)(ps->s[4] * x + ps->s[5] * y) + ps->r[2]) & 0x3F : 0; + int d = ps->count >= 4 ? (int)((uint32_t)(ps->s[6] * x + ps->s[7] * y) + ps->r[3]) & 0x3F : 0; + if (a >= b && a >= c && a >= d) return 0; + if (b >= c && b >= d) return 1; + if (c >= d) return 2; + return 3; +} + +// ---- colour endpoints ---- + +static inline int clamp255(int v) { return v < 0 ? 0 : v > 255 ? 255 : v; } + +// e[0..3] endpoint 0, e[4..7] endpoint 1 (8-bit). Returns false for an HDR mode. +static bool unpack_endpoints(int cem, const uint8_t *v, int e0[4], int e1[4]) { + switch (cem) { + case 0: // L direct + e0[0] = e0[1] = e0[2] = v[0]; e1[0] = e1[1] = e1[2] = v[1]; + e0[3] = e1[3] = 255; + return true; + case 1: { // L delta + int l0 = v[0] >> 2 | (v[1] & 0xC0); + int l1 = l0 + (v[1] & 0x3F); + if (l1 > 255) l1 = 255; + e0[0] = e0[1] = e0[2] = l0; e1[0] = e1[1] = e1[2] = l1; + e0[3] = e1[3] = 255; + return true; + } + case 4: // LA direct + e0[0] = e0[1] = e0[2] = v[0]; e1[0] = e1[1] = e1[2] = v[1]; + e0[3] = v[2]; e1[3] = v[3]; + return true; + case 5: { // LA delta + int l0 = v[0], l1 = v[1], a0 = v[2], a1 = v[3]; + l0 |= (l1 & 0x80) << 1; a0 |= (a1 & 0x80) << 1; + l1 &= 0x7F; a1 &= 0x7F; + if (l1 & 0x40) l1 -= 0x80; + if (a1 & 0x40) a1 -= 0x80; + l0 >>= 1; a0 >>= 1; l1 >>= 1; a1 >>= 1; // arithmetic shift of the signed deltas + l1 = clamp255(l1 + l0); a1 = clamp255(a1 + a0); + e0[0] = e0[1] = e0[2] = l0; e1[0] = e1[1] = e1[2] = l1; + e0[3] = a0; e1[3] = a1; + return true; + } + case 6: // RGB scale + case 10: { // RGB scale + two alphas + int s = v[3]; + for (int c = 0; c < 3; c++) { e1[c] = v[c]; e0[c] = v[c] * s >> 8; } + if (cem == 6) e0[3] = e1[3] = 255; + else { e0[3] = v[4]; e1[3] = v[5]; } + return true; + } + case 8: // RGB(A) direct + case 12: { + int a[4] = {v[0], v[2], v[4], cem == 12 ? v[6] : 0}; + int b[4] = {v[1], v[3], v[5], cem == 12 ? v[7] : 0}; + if (a[0] + a[1] + a[2] > b[0] + b[1] + b[2]) { // blue contraction: swap + uncontract + for (int c = 0; c < 4; c++) { e0[c] = b[c]; e1[c] = a[c]; } + e0[0] = (e0[0] + e0[2]) >> 1; e0[1] = (e0[1] + e0[2]) >> 1; + e1[0] = (e1[0] + e1[2]) >> 1; e1[1] = (e1[1] + e1[2]) >> 1; + } else { + for (int c = 0; c < 4; c++) { e0[c] = a[c]; e1[c] = b[c]; } + } + if (cem == 8) e0[3] = e1[3] = 255; + return true; + } + case 9: // RGB(A) base + delta + case 13: { + int n = cem == 13 ? 4 : 3; + int a[4] = {0, 0, 0, 0}, b[4] = {0, 0, 0, 0}; + for (int c = 0; c < n; c++) { + int base = v[2 * c], delta = v[2 * c + 1]; + a[c] = base >> 1 | (delta & 0x80); + b[c] = delta >> 1 & 0x3F; + if (b[c] & 0x20) b[c] -= 0x40; + } + int sum = b[0] + b[1] + b[2]; + for (int c = 0; c < 4; c++) b[c] += a[c]; + if (sum < 0) { // swap + uncontract + for (int c = 0; c < 4; c++) { e0[c] = b[c]; e1[c] = a[c]; } + e0[0] = (e0[0] + e0[2]) >> 1; e0[1] = (e0[1] + e0[2]) >> 1; + e1[0] = (e1[0] + e1[2]) >> 1; e1[1] = (e1[1] + e1[2]) >> 1; + } else { + for (int c = 0; c < 4; c++) { e0[c] = a[c]; e1[c] = b[c]; } + } + for (int c = 0; c < 4; c++) { e0[c] = clamp255(e0[c]); e1[c] = clamp255(e1[c]); } + if (cem == 9) e0[3] = e1[3] = 255; + return true; + } + default: // 2, 3, 7, 11, 14, 15: HDR + return false; + } +} + +// ---- block ---- + +static void fill_error(uint8_t *out, int texels) { + for (int i = 0; i < texels; i++) { + out[4 * i + 0] = 0xFF; out[4 * i + 1] = 0x00; out[4 * i + 2] = 0xFF; out[4 * i + 3] = 0xFF; + } +} + +bool astc_decode_block(const uint8_t block[16], uint32_t bw, uint32_t bh, bool srgb, uint8_t *out) { + if (bw < 4 || bh < 4 || bw > 12 || bh > 12) return false; // not an ASTC 2D footprint + int texels = (int)(bw * bh); + Bits128 b = {load64(block), load64(block + 8)}; + uint32_t mode = (uint32_t)(b.lo & 0x7FF); + + if ((mode & 0x1FF) == 0x1FC) { // void extent + if (mode & 0x200) goto error; // HDR (fp16) constant: error in LDR + if ((b.lo >> 10 & 3) != 3) goto error; // reserved bits + uint32_t s0 = rbits(&b, 12, 13), s1 = rbits(&b, 25, 13); + uint32_t t0 = rbits(&b, 38, 13), t1 = rbits(&b, 51, 13); + bool all_ones = s0 == 0x1FFF && s1 == 0x1FFF && t0 == 0x1FFF && t1 == 0x1FFF; + if ((s0 >= s1 || t0 >= t1) && !all_ones) goto error; + uint8_t c[4] = {block[9], block[11], block[13], block[15]}; // top 8 bits of the UNORM16s + for (int i = 0; i < texels; i++) memcpy(out + 4 * i, c, 4); + return true; + } + + BlockMode bm; + if (!block_mode_2d(mode, bw, bh, &bm)) goto error; + int pcount = (int)(b.lo >> 11 & 3) + 1; + int wcount = bm.wx * bm.wy; + + // Weights, read backwards from the top of the block. + uint8_t raw[64]; + Bits128 rev = {bitrev64(b.hi), bitrev64(b.lo)}; + ise_decode(&rev, 0, wcount << bm.dual, bm.quant, raw); + if (bm.dual && pcount == 4) goto error; + + const uint8_t *wt = wunq_data + wunq_off[bm.quant]; + uint8_t grid[2][80]; + memset(grid, 0, sizeof grid); + if (bm.dual) { + for (int i = 0; i < wcount; i++) { grid[0][i] = wt[raw[2 * i]]; grid[1][i] = wt[raw[2 * i + 1]]; } + } else { + for (int i = 0; i < wcount; i++) grid[0][i] = wt[raw[i]]; + } + + // Colour endpoint modes. + int cem[4]; + int below = 128 - bm.wbits, high = 0; + if (pcount == 1) { + cem[0] = (int)(b.lo >> 13 & 0xF); + } else { + high = 3 * pcount - 4; + below -= high; + uint32_t enc = (uint32_t)(b.lo >> 23 & 0x3F) | rbits(&b, below, high) << 6; + uint32_t base = enc & 3; + if (base == 0) { + for (int i = 0; i < pcount; i++) cem[i] = (int)(enc >> 2 & 0xF); + below += high; + high = 0; + } else { + base--; + for (int i = 0; i < pcount; i++) cem[i] = (int)(((enc >> (2 + i) & 1) + base) << 2); + for (int i = 0; i < pcount; i++) cem[i] |= (int)(enc >> (2 + pcount + 2 * i) & 3); + } + } + int ccount = 0; + for (int i = 0; i < pcount; i++) ccount += ((cem[i] >> 2) + 1) * 2; + if (ccount > 18) goto error; + int cbits = (pcount == 1 ? 111 : 99) - bm.wbits - high - (bm.dual ? 2 : 0); + if (cbits < 0) cbits = 0; + int cq = -1; + for (int q = 20; q >= 0; q--) { + if (ise_bitcount(ccount, q) <= cbits) { cq = q; break; } + } + if (cq < QUANT_6) goto error; + uint8_t cv[18]; + ise_decode(&b, pcount == 1 ? 17 : 29, ccount, cq, cv); + if (cq != QUANT_256) { + const uint8_t *ct = cunq_data + cunq_off[cq - QUANT_6]; + for (int i = 0; i < ccount; i++) cv[i] = ct[cv[i]]; + } + int ccs = bm.dual ? (int)rbits(&b, below - 2, 2) : -1; + + // Endpoints, expanded to 16 bits. + int ep[4][2][4]; + for (int p = 0, k = 0; p < pcount; p++) { + int *e0 = ep[p][0], *e1 = ep[p][1]; + if (!unpack_endpoints(cem[p], cv + k, e0, e1)) { + static const int magenta[4] = {0xFF, 0x00, 0xFF, 0xFF}; + memcpy(e0, magenta, sizeof magenta); + memcpy(e1, magenta, sizeof magenta); + } + k += ((cem[p] >> 2) + 1) * 2; + for (int c = 0; c < 4; c++) { + e0[c] = srgb ? e0[c] << 8 | 0x80 : e0[c] * 257; + e1[c] = srgb ? e1[c] << 8 | 0x80 : e1[c] * 257; + } + } + + // Weight grid -> texel weights (bilinear infill), per axis. + int xi[12], xf[12], yi[12], yf[12]; + for (uint32_t x = 0; x < bw; x++) { + int v = (int)(((1024 + bw / 2) / (bw - 1)) * x * (uint32_t)(bm.wx - 1) + 32) >> 6; + xi[x] = v >> 4; xf[x] = v & 0xF; + } + for (uint32_t y = 0; y < bh; y++) { + int v = (int)(((1024 + bh / 2) / (bh - 1)) * y * (uint32_t)(bm.wy - 1) + 32) >> 6; + yi[y] = (v >> 4) * bm.wx; yf[y] = v & 0xF; + } + + PartSel ps = {0}; + if (pcount > 1) part_init(&ps, (int)(b.lo >> 13 & 0x3FF), pcount); + int small = texels < 32; + + uint8_t *o = out; + for (uint32_t y = 0; y < bh; y++) { + for (uint32_t x = 0; x < bw; x++, o += 4) { + int fs = xf[x], ft = yf[y]; + int w11 = (fs * ft + 8) >> 4, w10 = fs - w11, w01 = ft - w11, w00 = 16 - fs - ft + w11; + int i0 = yi[y] + xi[x]; + int w[2]; + for (int pl = 0; pl <= bm.dual; pl++) { + const uint8_t *g = grid[pl]; + w[pl] = (g[i0] * w00 + g[i0 + 1] * w10 + g[i0 + bm.wx] * w01 + g[i0 + bm.wx + 1] * w11 + 8) >> 4; + } + int p = pcount > 1 ? part_of(&ps, small ? (int)x << 1 : (int)x, small ? (int)y << 1 : (int)y) : 0; + const int *e0 = ep[p][0], *e1 = ep[p][1]; + for (int c = 0; c < 4; c++) { + int wc = c == ccs ? w[1] : w[0]; + o[c] = (uint8_t)((e0[c] * (64 - wc) + e1[c] * wc + 32) >> 14); // >> 6, then top 8 bits + } + } + } + return true; + +error: + fill_error(out, texels); + return false; +} + +void astc_decode_ldr(const uint8_t *blocks, uint32_t width, uint32_t height, uint32_t bw, uint32_t bh, + bool srgb, uint8_t *rgba_out) { + astc_decode_ldr_rows(blocks, width, height, bw, bh, srgb, rgba_out, 0, UINT32_MAX); +} + +void astc_decode_ldr_rows(const uint8_t *blocks, uint32_t width, uint32_t height, uint32_t bw, uint32_t bh, + bool srgb, uint8_t *rgba_out, uint32_t row0, uint32_t row1) { + if (bw < 4 || bh < 4 || bw > 12 || bh > 12) return; + uint32_t bx = (width + bw - 1) / bw, by = (height + bh - 1) / bh; + if (row1 > by) row1 = by; + uint8_t tmp[12 * 12 * 4]; + for (uint32_t j = row0; j < row1; j++) { + for (uint32_t i = 0; i < bx; i++) { + const uint8_t *src = blocks + 16 * ((size_t)j * bx + i); + uint32_t x0 = i * bw, y0 = j * bh; + uint32_t cw = width - x0 < bw ? width - x0 : bw, ch = height - y0 < bh ? height - y0 : bh; + // decode in place when the block's rows are contiguous in the output + uint8_t *dst = width == bw && ch == bh ? rgba_out + (size_t)y0 * width * 4 : tmp; + astc_decode_block(src, bw, bh, srgb, dst); + if (dst == tmp) { + for (uint32_t y = 0; y < ch; y++) + memcpy(rgba_out + ((size_t)(y0 + y) * width + x0) * 4, tmp + y * bw * 4, cw * 4); + } + } + } +} diff --git a/src/gfx/tex/astc.h b/src/gfx/tex/astc.h new file mode 100644 index 0000000..321a0f5 --- /dev/null +++ b/src/gfx/tex/astc.h @@ -0,0 +1,18 @@ +// ASTC LDR decoder (2D blocks) to RGBA8, for GPUs without native ASTC (desktop NVIDIA / AMD). +// Bit-exact with ARM astcenc's 8-bit (decode_unorm8) decode, -dl (srgb = false) / -ds (srgb = true). +#pragma once +#include +#include + +// blocks: row-major 16-byte blocks covering ceil(width/bw) x ceil(height/bh). +// rgba_out: width*height*4 bytes, tightly packed rows. Values are the 8-bit UNORM results (no sRGB +// -> linear conversion; srgb only selects the sRGB endpoint expansion, which changes rounding). +// Error / illegal blocks and HDR content decode to magenta (FF 00 FF FF). +void astc_decode_ldr(const uint8_t *blocks, uint32_t width, uint32_t height, uint32_t bw, uint32_t bh, + bool srgb, uint8_t *rgba_out); + +// One block -> bw*bh*4 bytes (rows of bw texels). Returns false for an error block (output magenta). +// Block rows [row0, row1) only (rows of bh texels; the rest of rgba_out is not touched): for decoding on several threads. +void astc_decode_ldr_rows(const uint8_t *blocks, uint32_t width, uint32_t height, uint32_t bw, uint32_t bh, + bool srgb, uint8_t *rgba_out, uint32_t row0, uint32_t row1); +bool astc_decode_block(const uint8_t block[16], uint32_t bw, uint32_t bh, bool srgb, uint8_t *rgba_out); diff --git a/src/gfx/tex/bc7.cpp b/src/gfx/tex/bc7.cpp new file mode 100644 index 0000000..36ef2f9 --- /dev/null +++ b/src/gfx/tex/bc7.cpp @@ -0,0 +1,61 @@ +// BC7 and BC1 encoding of RGBA8 texels: bc7enc and rgbcx (tex/bc7enc, Rich Geldreich, MIT) block by block, with the +// port's C interface. +#include +#include + +#include "bc7.h" +#include "bc7enc/bc7enc.h" +#include "bc7enc/rgbcx.h" + +static pthread_once_t g_once = PTHREAD_ONCE_INIT; +static void init(void) { bc7enc_compress_block_init(); rgbcx::init(); } // their tables, once (read-only afterwards) + +// The 4x4 texels of block (x, y) of a slice; past the edge, the last row / column repeats. +static void block_px(const uint8_t *slice, uint32_t w, uint32_t h, uint32_t x, uint32_t y, uint8_t px[64]) { + for (uint32_t j = 0; j < 4; j++) { + uint32_t sy = y * 4 + j < h ? y * 4 + j : h - 1; + for (uint32_t i = 0; i < 4; i++) { + uint32_t sx = x * 4 + i < w ? x * 4 + i : w - 1; + memcpy(px + (j * 4 + i) * 4, slice + ((size_t)sy * w + sx) * 4, 4); + } + } +} + +extern "C" size_t bc1_size(uint32_t w, uint32_t h, uint32_t depth) { + return (size_t)((w + 3) / 4) * ((h + 3) / 4) * (depth ? depth : 1) * 8; +} +extern "C" void bc1_encode(const uint8_t *rgba, uint32_t w, uint32_t h, uint32_t depth, int level, uint8_t *out) { + pthread_once(&g_once, init); + uint32_t bx = (w + 3) / 4, by = (h + 3) / 4; + for (uint32_t z = 0; z < (depth ? depth : 1); z++) + for (uint32_t y = 0; y < by; y++) + for (uint32_t x = 0; x < bx; x++) { + uint8_t px[64]; + block_px(rgba + (size_t)w * h * 4 * z, w, h, x, y, px); + rgbcx::encode_bc1((uint32_t)level, out, px, false, false); // 4-colour blocks: opaque, no punch-through + out += 8; + } +} + +extern "C" size_t bc7_size(uint32_t w, uint32_t h, uint32_t depth) { + return (size_t)((w + 3) / 4) * ((h + 3) / 4) * (depth ? depth : 1) * 16; +} + +extern "C" void bc7_encode(const uint8_t *rgba, uint32_t w, uint32_t h, uint32_t depth, int perceptual, int quality, uint8_t *out) { + pthread_once(&g_once, init); + bc7enc_compress_block_params p; + bc7enc_compress_block_params_init(&p); + if (!perceptual) bc7enc_compress_block_params_init_linear_weights(&p); + p.m_uber_level = (uint32_t)(quality < 0 ? 0 : quality > BC7ENC_MAX_UBER_LEVEL ? BC7ENC_MAX_UBER_LEVEL : quality); + uint32_t bx = (w + 3) / 4, by = (h + 3) / 4; + for (uint32_t z = 0; z < (depth ? depth : 1); z++) { + const uint8_t *slice = rgba + (size_t)w * h * 4 * z; + for (uint32_t y = 0; y < by; y++) + for (uint32_t x = 0; x < bx; x++) { + uint8_t px[64]; + block_px(slice, w, h, x, y, px); + bc7enc_compress_block(out, px, &p); + out += 16; + } + } +} diff --git a/src/gfx/tex/bc7.h b/src/gfx/tex/bc7.h new file mode 100644 index 0000000..2a228d4 --- /dev/null +++ b/src/gfx/tex/bc7.h @@ -0,0 +1,22 @@ +// BC7 and BC1 encoding of RGBA8 texels (the port's wrapper of bc7enc and rgbcx, tex/bc7enc: MIT). C interface. +#pragma once +#include +#include + +#ifdef __cplusplus +extern "C" { +#endif + +// Bytes of BC7 blocks for w x h x depth texels: 16 per 4x4 block, rows of blocks, slices one after another. +size_t bc7_size(uint32_t w, uint32_t h, uint32_t depth); +// Encodes w x h texels per slice (rows of w * 4 bytes, `depth` slices one after another) into out (bc7_size bytes). +// perceptual: colour weights (sRGB textures); else every channel weighs the same (normals, masks, other data). +// quality: 0 (fastest) .. 4 (best, slowest); blocks past the edge repeat the last row / column. +void bc7_encode(const uint8_t *rgba, uint32_t w, uint32_t h, uint32_t depth, int perceptual, int quality, uint8_t *out); +// BC1 (opaque: alpha ignored; 8 bytes per 4x4 block, rows of blocks, slices one after another), rgbcx's level 0..18. +size_t bc1_size(uint32_t w, uint32_t h, uint32_t depth); +void bc1_encode(const uint8_t *rgba, uint32_t w, uint32_t h, uint32_t depth, int level, uint8_t *out); + +#ifdef __cplusplus +} +#endif diff --git a/src/gfx/tex/bc7enc/LICENSE b/src/gfx/tex/bc7enc/LICENSE new file mode 100644 index 0000000..5258641 --- /dev/null +++ b/src/gfx/tex/bc7enc/LICENSE @@ -0,0 +1,68 @@ +If you use this software in a product, attribution / credits is requested but not required. + +bc7e.ispc uses the Apache 2.0 license and is Copyright (C) 2018-2021 Binomial LLC. +LodePNG is Copyright (c) 2005-2016 Lode Vandevenne. See LodePNG.cpp for its license. + +All other source code files in this repo are available under 2 licenses -- choose whichever you prefer. + +ALTERNATIVE A - MIT License +Copyright(c) 2020-2021 Richard Geldreich, Jr. +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files(the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and / or sell copies +of the Software, and to permit persons to whom the Software is furnished to do +so, subject to the following conditions : +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. +------------------------------------------------------------------------------ +ALTERNATIVE B - Public Domain(www.unlicense.org) +This is free and unencumbered software released into the public domain. +Anyone is free to copy, modify, publish, use, compile, sell, or distribute this +software, either in source code form or as a compiled binary, for any purpose, +commercial or non - commercial, and by any means. +In jurisdictions that recognize copyright laws, the author or authors of this +software dedicate any and all copyright interest in the software to the public +domain.We make this dedication for the benefit of the public at large and to +the detriment of our heirs and successors.We intend this dedication to be an +overt act of relinquishment in perpetuity of all present and future rights to +this software under copyright law. +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE +AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN +ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +------------------------------------------------------------------------------ + +LodePNG version 20161127 + +Copyright (c) 2005-2016 Lode Vandevenne + +This software is provided 'as-is', without any express or implied +warranty. In no event will the authors be held liable for any damages +arising from the use of this software. + +Permission is granted to anyone to use this software for any purpose, +including commercial applications, and to alter it and redistribute it +freely, subject to the following restrictions: + + 1. The origin of this software must not be misrepresented; you must not + claim that you wrote the original software. If you use this software + in a product, an acknowledgment in the product documentation would be + appreciated but is not required. + + 2. Altered source versions must be plainly marked as such, and must not be + misrepresented as being the original software. + + 3. This notice may not be removed or altered from any source + distribution. + diff --git a/src/gfx/tex/bc7enc/bc7enc.cpp b/src/gfx/tex/bc7enc/bc7enc.cpp new file mode 100644 index 0000000..b9cf067 --- /dev/null +++ b/src/gfx/tex/bc7enc/bc7enc.cpp @@ -0,0 +1,2580 @@ +// File: bc7enc.c - Richard Geldreich, Jr. 3/31/2020 - MIT license or public domain (see end of file) +// Currently supports modes 1, 6 for RGB blocks, and modes 5, 6, 7 for RGBA blocks. +#include "bc7enc.h" +#include +#include +#include +#include +#include + +// Helpers +static inline int32_t clampi(int32_t value, int32_t low, int32_t high) { if (value < low) value = low; else if (value > high) value = high; return value; } +static inline float clampf(float value, float low, float high) { if (value < low) value = low; else if (value > high) value = high; return value; } +static inline float saturate(float value) { return clampf(value, 0, 1.0f); } +static inline uint8_t minimumub(uint8_t a, uint8_t b) { return (a < b) ? a : b; } +static inline int32_t minimumi(int32_t a, int32_t b) { return (a < b) ? a : b; } +static inline uint32_t minimumu(uint32_t a, uint32_t b) { return (a < b) ? a : b; } +static inline float minimumf(float a, float b) { return (a < b) ? a : b; } +static inline uint8_t maximumub(uint8_t a, uint8_t b) { return (a > b) ? a : b; } +static inline uint32_t maximumu(uint32_t a, uint32_t b) { return (a > b) ? a : b; } +static inline int32_t maximumi(int32_t a, int32_t b) { return (a > b) ? a : b; } +static inline float maximumf(float a, float b) { return (a > b) ? a : b; } +static inline int squarei(int i) { return i * i; } +static inline float squaref(float i) { return i * i; } +template inline T0 lerp(T0 a, T0 b, T1 c) { return a + (b - a) * c; } + +static inline int32_t iabs32(int32_t v) { uint32_t msk = v >> 31; return (v ^ msk) - msk; } +static inline void swapub(uint8_t* a, uint8_t* b) { uint8_t t = *a; *a = *b; *b = t; } +static inline void swapu(uint32_t* a, uint32_t* b) { uint32_t t = *a; *a = *b; *b = t; } +static inline void swapf(float* a, float* b) { float t = *a; *a = *b; *b = t; } + +struct vec4F { float m_c[4]; }; + +static inline color_rgba *color_quad_u8_set_clamped(color_rgba *pRes, int32_t r, int32_t g, int32_t b, int32_t a) { pRes->m_c[0] = (uint8_t)clampi(r, 0, 255); pRes->m_c[1] = (uint8_t)clampi(g, 0, 255); pRes->m_c[2] = (uint8_t)clampi(b, 0, 255); pRes->m_c[3] = (uint8_t)clampi(a, 0, 255); return pRes; } +static inline color_rgba *color_quad_u8_set(color_rgba *pRes, int32_t r, int32_t g, int32_t b, int32_t a) { assert((uint32_t)(r | g | b | a) <= 255); pRes->m_c[0] = (uint8_t)r; pRes->m_c[1] = (uint8_t)g; pRes->m_c[2] = (uint8_t)b; pRes->m_c[3] = (uint8_t)a; return pRes; } +static inline bool color_quad_u8_notequals(const color_rgba *pLHS, const color_rgba *pRHS) { return (pLHS->m_c[0] != pRHS->m_c[0]) || (pLHS->m_c[1] != pRHS->m_c[1]) || (pLHS->m_c[2] != pRHS->m_c[2]) || (pLHS->m_c[3] != pRHS->m_c[3]); } +static inline vec4F *vec4F_set_scalar(vec4F *pV, float x) { pV->m_c[0] = x; pV->m_c[1] = x; pV->m_c[2] = x; pV->m_c[3] = x; return pV; } +static inline vec4F *vec4F_set(vec4F *pV, float x, float y, float z, float w) { pV->m_c[0] = x; pV->m_c[1] = y; pV->m_c[2] = z; pV->m_c[3] = w; return pV; } +static inline vec4F *vec4F_saturate_in_place(vec4F *pV) { pV->m_c[0] = saturate(pV->m_c[0]); pV->m_c[1] = saturate(pV->m_c[1]); pV->m_c[2] = saturate(pV->m_c[2]); pV->m_c[3] = saturate(pV->m_c[3]); return pV; } +static inline vec4F vec4F_saturate(const vec4F *pV) { vec4F res; res.m_c[0] = saturate(pV->m_c[0]); res.m_c[1] = saturate(pV->m_c[1]); res.m_c[2] = saturate(pV->m_c[2]); res.m_c[3] = saturate(pV->m_c[3]); return res; } +static inline vec4F vec4F_from_color(const color_rgba *pC) { vec4F res; vec4F_set(&res, pC->m_c[0], pC->m_c[1], pC->m_c[2], pC->m_c[3]); return res; } +static inline vec4F vec4F_add(const vec4F *pLHS, const vec4F *pRHS) { vec4F res; vec4F_set(&res, pLHS->m_c[0] + pRHS->m_c[0], pLHS->m_c[1] + pRHS->m_c[1], pLHS->m_c[2] + pRHS->m_c[2], pLHS->m_c[3] + pRHS->m_c[3]); return res; } +static inline vec4F vec4F_sub(const vec4F *pLHS, const vec4F *pRHS) { vec4F res; vec4F_set(&res, pLHS->m_c[0] - pRHS->m_c[0], pLHS->m_c[1] - pRHS->m_c[1], pLHS->m_c[2] - pRHS->m_c[2], pLHS->m_c[3] - pRHS->m_c[3]); return res; } +static inline float vec4F_dot(const vec4F *pLHS, const vec4F *pRHS) { return pLHS->m_c[0] * pRHS->m_c[0] + pLHS->m_c[1] * pRHS->m_c[1] + pLHS->m_c[2] * pRHS->m_c[2] + pLHS->m_c[3] * pRHS->m_c[3]; } +static inline vec4F vec4F_mul(const vec4F *pLHS, float s) { vec4F res; vec4F_set(&res, pLHS->m_c[0] * s, pLHS->m_c[1] * s, pLHS->m_c[2] * s, pLHS->m_c[3] * s); return res; } +static inline vec4F *vec4F_normalize_in_place(vec4F *pV) { float s = pV->m_c[0] * pV->m_c[0] + pV->m_c[1] * pV->m_c[1] + pV->m_c[2] * pV->m_c[2] + pV->m_c[3] * pV->m_c[3]; if (s != 0.0f) { s = 1.0f / sqrtf(s); pV->m_c[0] *= s; pV->m_c[1] *= s; pV->m_c[2] *= s; pV->m_c[3] *= s; } return pV; } + +// Various BC7 tables +static const uint32_t g_bc7_weights2[4] = { 0, 21, 43, 64 }; +static const uint32_t g_bc7_weights3[8] = { 0, 9, 18, 27, 37, 46, 55, 64 }; +static const uint32_t g_bc7_weights4[16] = { 0, 4, 9, 13, 17, 21, 26, 30, 34, 38, 43, 47, 51, 55, 60, 64 }; +// Precomputed weight constants used during least fit determination. For each entry in g_bc7_weights[]: w * w, (1.0f - w) * w, (1.0f - w) * (1.0f - w), w +static const float g_bc7_weights2x[4 * 4] = { 0.000000f, 0.000000f, 1.000000f, 0.000000f, 0.107666f, 0.220459f, 0.451416f, 0.328125f, 0.451416f, 0.220459f, 0.107666f, 0.671875f, 1.000000f, 0.000000f, 0.000000f, 1.000000f }; +static const float g_bc7_weights3x[8 * 4] = { 0.000000f, 0.000000f, 1.000000f, 0.000000f, 0.019775f, 0.120850f, 0.738525f, 0.140625f, 0.079102f, 0.202148f, 0.516602f, 0.281250f, 0.177979f, 0.243896f, 0.334229f, 0.421875f, 0.334229f, 0.243896f, 0.177979f, 0.578125f, 0.516602f, 0.202148f, + 0.079102f, 0.718750f, 0.738525f, 0.120850f, 0.019775f, 0.859375f, 1.000000f, 0.000000f, 0.000000f, 1.000000f }; +static const float g_bc7_weights4x[16 * 4] = { 0.000000f, 0.000000f, 1.000000f, 0.000000f, 0.003906f, 0.058594f, 0.878906f, 0.062500f, 0.019775f, 0.120850f, 0.738525f, 0.140625f, 0.041260f, 0.161865f, 0.635010f, 0.203125f, 0.070557f, 0.195068f, 0.539307f, 0.265625f, 0.107666f, 0.220459f, + 0.451416f, 0.328125f, 0.165039f, 0.241211f, 0.352539f, 0.406250f, 0.219727f, 0.249023f, 0.282227f, 0.468750f, 0.282227f, 0.249023f, 0.219727f, 0.531250f, 0.352539f, 0.241211f, 0.165039f, 0.593750f, 0.451416f, 0.220459f, 0.107666f, 0.671875f, 0.539307f, 0.195068f, 0.070557f, 0.734375f, + 0.635010f, 0.161865f, 0.041260f, 0.796875f, 0.738525f, 0.120850f, 0.019775f, 0.859375f, 0.878906f, 0.058594f, 0.003906f, 0.937500f, 1.000000f, 0.000000f, 0.000000f, 1.000000f }; + +static const uint8_t g_bc7_partition1[16] = { 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 }; +static const uint8_t g_bc7_partition2[64 * 16] = +{ + 0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1, 0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1, 0,1,1,1,0,1,1,1,0,1,1,1,0,1,1,1, 0,0,0,1,0,0,1,1,0,0,1,1,0,1,1,1, 0,0,0,0,0,0,0,1,0,0,0,1,0,0,1,1, 0,0,1,1,0,1,1,1,0,1,1,1,1,1,1,1, 0,0,0,1,0,0,1,1,0,1,1,1,1,1,1,1, 0,0,0,0,0,0,0,1,0,0,1,1,0,1,1,1, + 0,0,0,0,0,0,0,0,0,0,0,1,0,0,1,1, 0,0,1,1,0,1,1,1,1,1,1,1,1,1,1,1, 0,0,0,0,0,0,0,1,0,1,1,1,1,1,1,1, 0,0,0,0,0,0,0,0,0,0,0,1,0,1,1,1, 0,0,0,1,0,1,1,1,1,1,1,1,1,1,1,1, 0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1, 0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1, 0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1, + 0,0,0,0,1,0,0,0,1,1,1,0,1,1,1,1, 0,1,1,1,0,0,0,1,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,1,0,0,0,1,1,1,0, 0,1,1,1,0,0,1,1,0,0,0,1,0,0,0,0, 0,0,1,1,0,0,0,1,0,0,0,0,0,0,0,0, 0,0,0,0,1,0,0,0,1,1,0,0,1,1,1,0, 0,0,0,0,0,0,0,0,1,0,0,0,1,1,0,0, 0,1,1,1,0,0,1,1,0,0,1,1,0,0,0,1, + 0,0,1,1,0,0,0,1,0,0,0,1,0,0,0,0, 0,0,0,0,1,0,0,0,1,0,0,0,1,1,0,0, 0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0, 0,0,1,1,0,1,1,0,0,1,1,0,1,1,0,0, 0,0,0,1,0,1,1,1,1,1,1,0,1,0,0,0, 0,0,0,0,1,1,1,1,1,1,1,1,0,0,0,0, 0,1,1,1,0,0,0,1,1,0,0,0,1,1,1,0, 0,0,1,1,1,0,0,1,1,0,0,1,1,1,0,0, + 0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1, 0,0,0,0,1,1,1,1,0,0,0,0,1,1,1,1, 0,1,0,1,1,0,1,0,0,1,0,1,1,0,1,0, 0,0,1,1,0,0,1,1,1,1,0,0,1,1,0,0, 0,0,1,1,1,1,0,0,0,0,1,1,1,1,0,0, 0,1,0,1,0,1,0,1,1,0,1,0,1,0,1,0, 0,1,1,0,1,0,0,1,0,1,1,0,1,0,0,1, 0,1,0,1,1,0,1,0,1,0,1,0,0,1,0,1, + 0,1,1,1,0,0,1,1,1,1,0,0,1,1,1,0, 0,0,0,1,0,0,1,1,1,1,0,0,1,0,0,0, 0,0,1,1,0,0,1,0,0,1,0,0,1,1,0,0, 0,0,1,1,1,0,1,1,1,1,0,1,1,1,0,0, 0,1,1,0,1,0,0,1,1,0,0,1,0,1,1,0, 0,0,1,1,1,1,0,0,1,1,0,0,0,0,1,1, 0,1,1,0,0,1,1,0,1,0,0,1,1,0,0,1, 0,0,0,0,0,1,1,0,0,1,1,0,0,0,0,0, + 0,1,0,0,1,1,1,0,0,1,0,0,0,0,0,0, 0,0,1,0,0,1,1,1,0,0,1,0,0,0,0,0, 0,0,0,0,0,0,1,0,0,1,1,1,0,0,1,0, 0,0,0,0,0,1,0,0,1,1,1,0,0,1,0,0, 0,1,1,0,1,1,0,0,1,0,0,1,0,0,1,1, 0,0,1,1,0,1,1,0,1,1,0,0,1,0,0,1, 0,1,1,0,0,0,1,1,1,0,0,1,1,1,0,0, 0,0,1,1,1,0,0,1,1,1,0,0,0,1,1,0, + 0,1,1,0,1,1,0,0,1,1,0,0,1,0,0,1, 0,1,1,0,0,0,1,1,0,0,1,1,1,0,0,1, 0,1,1,1,1,1,1,0,1,0,0,0,0,0,0,1, 0,0,0,1,1,0,0,0,1,1,1,0,0,1,1,1, 0,0,0,0,1,1,1,1,0,0,1,1,0,0,1,1, 0,0,1,1,0,0,1,1,1,1,1,1,0,0,0,0, 0,0,1,0,0,0,1,0,1,1,1,0,1,1,1,0, 0,1,0,0,0,1,0,0,0,1,1,1,0,1,1,1 +}; + +static const uint8_t g_bc7_partition3[64 * 16] = +{ + 0,0,1,1,0,0,1,1,0,2,2,1,2,2,2,2, 0,0,0,1,0,0,1,1,2,2,1,1,2,2,2,1, 0,0,0,0,2,0,0,1,2,2,1,1,2,2,1,1, 0,2,2,2,0,0,2,2,0,0,1,1,0,1,1,1, 0,0,0,0,0,0,0,0,1,1,2,2,1,1,2,2, 0,0,1,1,0,0,1,1,0,0,2,2,0,0,2,2, 0,0,2,2,0,0,2,2,1,1,1,1,1,1,1,1, 0,0,1,1,0,0,1,1,2,2,1,1,2,2,1,1, + 0,0,0,0,0,0,0,0,1,1,1,1,2,2,2,2, 0,0,0,0,1,1,1,1,1,1,1,1,2,2,2,2, 0,0,0,0,1,1,1,1,2,2,2,2,2,2,2,2, 0,0,1,2,0,0,1,2,0,0,1,2,0,0,1,2, 0,1,1,2,0,1,1,2,0,1,1,2,0,1,1,2, 0,1,2,2,0,1,2,2,0,1,2,2,0,1,2,2, 0,0,1,1,0,1,1,2,1,1,2,2,1,2,2,2, 0,0,1,1,2,0,0,1,2,2,0,0,2,2,2,0, + 0,0,0,1,0,0,1,1,0,1,1,2,1,1,2,2, 0,1,1,1,0,0,1,1,2,0,0,1,2,2,0,0, 0,0,0,0,1,1,2,2,1,1,2,2,1,1,2,2, 0,0,2,2,0,0,2,2,0,0,2,2,1,1,1,1, 0,1,1,1,0,1,1,1,0,2,2,2,0,2,2,2, 0,0,0,1,0,0,0,1,2,2,2,1,2,2,2,1, 0,0,0,0,0,0,1,1,0,1,2,2,0,1,2,2, 0,0,0,0,1,1,0,0,2,2,1,0,2,2,1,0, + 0,1,2,2,0,1,2,2,0,0,1,1,0,0,0,0, 0,0,1,2,0,0,1,2,1,1,2,2,2,2,2,2, 0,1,1,0,1,2,2,1,1,2,2,1,0,1,1,0, 0,0,0,0,0,1,1,0,1,2,2,1,1,2,2,1, 0,0,2,2,1,1,0,2,1,1,0,2,0,0,2,2, 0,1,1,0,0,1,1,0,2,0,0,2,2,2,2,2, 0,0,1,1,0,1,2,2,0,1,2,2,0,0,1,1, 0,0,0,0,2,0,0,0,2,2,1,1,2,2,2,1, + 0,0,0,0,0,0,0,2,1,1,2,2,1,2,2,2, 0,2,2,2,0,0,2,2,0,0,1,2,0,0,1,1, 0,0,1,1,0,0,1,2,0,0,2,2,0,2,2,2, 0,1,2,0,0,1,2,0,0,1,2,0,0,1,2,0, 0,0,0,0,1,1,1,1,2,2,2,2,0,0,0,0, 0,1,2,0,1,2,0,1,2,0,1,2,0,1,2,0, 0,1,2,0,2,0,1,2,1,2,0,1,0,1,2,0, 0,0,1,1,2,2,0,0,1,1,2,2,0,0,1,1, + 0,0,1,1,1,1,2,2,2,2,0,0,0,0,1,1, 0,1,0,1,0,1,0,1,2,2,2,2,2,2,2,2, 0,0,0,0,0,0,0,0,2,1,2,1,2,1,2,1, 0,0,2,2,1,1,2,2,0,0,2,2,1,1,2,2, 0,0,2,2,0,0,1,1,0,0,2,2,0,0,1,1, 0,2,2,0,1,2,2,1,0,2,2,0,1,2,2,1, 0,1,0,1,2,2,2,2,2,2,2,2,0,1,0,1, 0,0,0,0,2,1,2,1,2,1,2,1,2,1,2,1, + 0,1,0,1,0,1,0,1,0,1,0,1,2,2,2,2, 0,2,2,2,0,1,1,1,0,2,2,2,0,1,1,1, 0,0,0,2,1,1,1,2,0,0,0,2,1,1,1,2, 0,0,0,0,2,1,1,2,2,1,1,2,2,1,1,2, 0,2,2,2,0,1,1,1,0,1,1,1,0,2,2,2, 0,0,0,2,1,1,1,2,1,1,1,2,0,0,0,2, 0,1,1,0,0,1,1,0,0,1,1,0,2,2,2,2, 0,0,0,0,0,0,0,0,2,1,1,2,2,1,1,2, + 0,1,1,0,0,1,1,0,2,2,2,2,2,2,2,2, 0,0,2,2,0,0,1,1,0,0,1,1,0,0,2,2, 0,0,2,2,1,1,2,2,1,1,2,2,0,0,2,2, 0,0,0,0,0,0,0,0,0,0,0,0,2,1,1,2, 0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,1, 0,2,2,2,1,2,2,2,0,2,2,2,1,2,2,2, 0,1,0,1,2,2,2,2,2,2,2,2,2,2,2,2, 0,1,1,1,2,0,1,1,2,2,0,1,2,2,2,0, +}; + +static const uint8_t g_bc7_table_anchor_index_third_subset_1[64] = +{ + 3, 3,15,15, 8, 3,15,15, 8, 8, 6, 6, 6, 5, 3, 3, 3, 3, 8,15, 3, 3, 6,10, 5, 8, 8, 6, 8, 5,15,15, 8,15, 3, 5, 6,10, 8,15, 15, 3,15, 5,15,15,15,15, 3,15, 5, 5, 5, 8, 5,10, 5,10, 8,13,15,12, 3, 3 +}; + +static const uint8_t g_bc7_table_anchor_index_third_subset_2[64] = +{ + 15, 8, 8, 3,15,15, 3, 8, 15,15,15,15,15,15,15, 8, 15, 8,15, 3,15, 8,15, 8, 3,15, 6,10,15,15,10, 8, 15, 3,15,10,10, 8, 9,10, 6,15, 8,15, 3, 6, 6, 8, 15, 3,15,15,15,15,15,15, 15,15,15,15, 3,15,15, 8 +}; + +static const uint8_t g_bc7_table_anchor_index_second_subset[64] = { 15,15,15,15,15,15,15,15, 15,15,15,15,15,15,15,15, 15, 2, 8, 2, 2, 8, 8,15, 2, 8, 2, 2, 8, 8, 2, 2, 15,15, 6, 8, 2, 8,15,15, 2, 8, 2, 2, 2,15,15, 6, 6, 2, 6, 8,15,15, 2, 2, 15,15,15,15,15, 2, 2,15 }; +static const uint8_t g_bc7_num_subsets[8] = { 3, 2, 3, 2, 1, 1, 1, 2 }; +static const uint8_t g_bc7_partition_bits[8] = { 4, 6, 6, 6, 0, 0, 0, 6 }; +static const uint8_t g_bc7_color_index_bitcount[8] = { 3, 3, 2, 2, 2, 2, 4, 2 }; +static int get_bc7_color_index_size(int mode, int index_selection_bit) { return g_bc7_color_index_bitcount[mode] + index_selection_bit; } +static uint8_t g_bc7_alpha_index_bitcount[8] = { 0, 0, 0, 0, 3, 2, 4, 2 }; +static int get_bc7_alpha_index_size(int mode, int index_selection_bit) { return g_bc7_alpha_index_bitcount[mode] - index_selection_bit; } +static const uint8_t g_bc7_mode_has_p_bits[8] = { 1, 1, 0, 1, 0, 0, 1, 1 }; +static const uint8_t g_bc7_mode_has_shared_p_bits[8] = { 0, 1, 0, 0, 0, 0, 0, 0 }; +static const uint8_t g_bc7_color_precision_table[8] = { 4, 6, 5, 7, 5, 7, 7, 5 }; +static const int8_t g_bc7_alpha_precision_table[8] = { 0, 0, 0, 0, 6, 8, 7, 5 }; +static bool get_bc7_mode_has_seperate_alpha_selectors(int mode) { return (mode == 4) || (mode == 5); } + +typedef struct { uint16_t m_error; uint8_t m_lo; uint8_t m_hi; } endpoint_err; + +static endpoint_err g_bc7_mode_1_optimal_endpoints[256][2]; // [c][pbit] +static const uint32_t BC7ENC_MODE_1_OPTIMAL_INDEX = 2; + +static endpoint_err g_bc7_mode_7_optimal_endpoints[256][2][2]; // [c][pbit][hp][lp] +const uint32_t BC7E_MODE_7_OPTIMAL_INDEX = 1; + +static float g_mode1_rgba_midpoints[64][2]; +static float g_mode5_rgba_midpoints[128]; +static float g_mode7_rgba_midpoints[32][2]; + +static uint8_t g_mode6_reduced_quant[2048][2]; + +static bool g_initialized; + +// Initialize the lookup table used for optimal single color compression in mode 1/7. Must be called before encoding. +void bc7enc_compress_block_init() +{ + if (g_initialized) + return; + + // Mode 7 endpoint midpoints + for (uint32_t p = 0; p < 2; p++) + { + for (uint32_t i = 0; i < 32; i++) + { + uint32_t vl = ((i << 1) | p) << 2; + vl |= (vl >> 6); + float lo = vl / 255.0f; + + uint32_t vh = ((minimumi(31, (i + 1)) << 1) | p) << 2; + vh |= (vh >> 6); + float hi = vh / 255.0f; + + //g_mode7_quant_values[i][p] = lo; + if (i == 31) + g_mode7_rgba_midpoints[i][p] = 1.0f; + else + g_mode7_rgba_midpoints[i][p] = (lo + hi) / 2.0f; + } + } + + // Mode 1 endpoint midpoints + for (uint32_t p = 0; p < 2; p++) + { + for (uint32_t i = 0; i < 64; i++) + { + uint32_t vl = ((i << 1) | p) << 1; + vl |= (vl >> 7); + float lo = vl / 255.0f; + + uint32_t vh = ((minimumi(63, (i + 1)) << 1) | p) << 1; + vh |= (vh >> 7); + float hi = vh / 255.0f; + + //g_mode1_quant_values[i][p] = lo; + if (i == 63) + g_mode1_rgba_midpoints[i][p] = 1.0f; + else + g_mode1_rgba_midpoints[i][p] = (lo + hi) / 2.0f; + } + } + + // Mode 5 endpoint midpoints + for (uint32_t i = 0; i < 128; i++) + { + uint32_t vl = (i << 1); + vl |= (vl >> 7); + float lo = vl / 255.0f; + + uint32_t vh = minimumi(127, i + 1) << 1; + vh |= (vh >> 7); + float hi = vh / 255.0f; + + if (i == 127) + g_mode5_rgba_midpoints[i] = 1.0f; + else + g_mode5_rgba_midpoints[i] = (lo + hi) / 2.0f; + } + + for (uint32_t p = 0; p < 2; p++) + { + for (uint32_t i = 0; i < 2048; i++) + { + float f = i / 2047.0f; + + float best_err = 1e+9f; + int best_index = 0; + for (int j = 0; j < 64; j++) + { + int ik = (j * 127 + 31) / 63; + float k = ((ik << 1) + p) / 255.0f; + + float e = fabsf(k - f); + if (e < best_err) + { + best_err = e; + best_index = ik; + } + } + + g_mode6_reduced_quant[i][p] = (uint8_t)best_index; + } + } // p + + // Mode 1 + for (int c = 0; c < 256; c++) + { + for (uint32_t lp = 0; lp < 2; lp++) + { + endpoint_err best; + best.m_error = (uint16_t)UINT16_MAX; + for (uint32_t l = 0; l < 64; l++) + { + uint32_t low = ((l << 1) | lp) << 1; + low |= (low >> 7); + for (uint32_t h = 0; h < 64; h++) + { + uint32_t high = ((h << 1) | lp) << 1; + high |= (high >> 7); + const int k = (low * (64 - g_bc7_weights3[BC7ENC_MODE_1_OPTIMAL_INDEX]) + high * g_bc7_weights3[BC7ENC_MODE_1_OPTIMAL_INDEX] + 32) >> 6; + const int err = (k - c) * (k - c); + if (err < best.m_error) + { + best.m_error = (uint16_t)err; + best.m_lo = (uint8_t)l; + best.m_hi = (uint8_t)h; + } + } // h + } // l + g_bc7_mode_1_optimal_endpoints[c][lp] = best; + } // lp + } // c + + // Mode 7: 555.1 2-bit indices + for (int c = 0; c < 256; c++) + { + for (uint32_t hp = 0; hp < 2; hp++) + { + for (uint32_t lp = 0; lp < 2; lp++) + { + endpoint_err best; + best.m_error = (uint16_t)UINT16_MAX; + best.m_lo = 0; + best.m_hi = 0; + + for (uint32_t l = 0; l < 32; l++) + { + uint32_t low = ((l << 1) | lp) << 2; + low |= (low >> 6); + + for (uint32_t h = 0; h < 32; h++) + { + uint32_t high = ((h << 1) | hp) << 2; + high |= (high >> 6); + + const int k = (low * (64 - g_bc7_weights2[BC7E_MODE_7_OPTIMAL_INDEX]) + high * g_bc7_weights2[BC7E_MODE_7_OPTIMAL_INDEX] + 32) >> 6; + + const int err = (k - c) * (k - c); + if (err < best.m_error) + { + best.m_error = (uint16_t)err; + best.m_lo = (uint8_t)l; + best.m_hi = (uint8_t)h; + } + } // h + } // l + + g_bc7_mode_7_optimal_endpoints[c][hp][lp] = best; + + } // hp + + } // lp + + } // c + + g_initialized = true; +} + +static void compute_least_squares_endpoints_rgba(uint32_t N, const uint8_t *pSelectors, const vec4F *pSelector_weights, vec4F *pXl, vec4F *pXh, const color_rgba *pColors) +{ + // Least squares using normal equations: http://www.cs.cornell.edu/~bindel/class/cs3220-s12/notes/lec10.pdf + // I did this in matrix form first, expanded out all the ops, then optimized it a bit. + float z00 = 0.0f, z01 = 0.0f, z10 = 0.0f, z11 = 0.0f; + float q00_r = 0.0f, q10_r = 0.0f, t_r = 0.0f; + float q00_g = 0.0f, q10_g = 0.0f, t_g = 0.0f; + float q00_b = 0.0f, q10_b = 0.0f, t_b = 0.0f; + float q00_a = 0.0f, q10_a = 0.0f, t_a = 0.0f; + for (uint32_t i = 0; i < N; i++) + { + const uint32_t sel = pSelectors[i]; + z00 += pSelector_weights[sel].m_c[0]; + z10 += pSelector_weights[sel].m_c[1]; + z11 += pSelector_weights[sel].m_c[2]; + float w = pSelector_weights[sel].m_c[3]; + q00_r += w * pColors[i].m_c[0]; t_r += pColors[i].m_c[0]; + q00_g += w * pColors[i].m_c[1]; t_g += pColors[i].m_c[1]; + q00_b += w * pColors[i].m_c[2]; t_b += pColors[i].m_c[2]; + q00_a += w * pColors[i].m_c[3]; t_a += pColors[i].m_c[3]; + } + + q10_r = t_r - q00_r; + q10_g = t_g - q00_g; + q10_b = t_b - q00_b; + q10_a = t_a - q00_a; + + z01 = z10; + + float det = z00 * z11 - z01 * z10; + if (det != 0.0f) + det = 1.0f / det; + + float iz00, iz01, iz10, iz11; + iz00 = z11 * det; + iz01 = -z01 * det; + iz10 = -z10 * det; + iz11 = z00 * det; + + pXl->m_c[0] = (float)(iz00 * q00_r + iz01 * q10_r); pXh->m_c[0] = (float)(iz10 * q00_r + iz11 * q10_r); + pXl->m_c[1] = (float)(iz00 * q00_g + iz01 * q10_g); pXh->m_c[1] = (float)(iz10 * q00_g + iz11 * q10_g); + pXl->m_c[2] = (float)(iz00 * q00_b + iz01 * q10_b); pXh->m_c[2] = (float)(iz10 * q00_b + iz11 * q10_b); + pXl->m_c[3] = (float)(iz00 * q00_a + iz01 * q10_a); pXh->m_c[3] = (float)(iz10 * q00_a + iz11 * q10_a); + + for (uint32_t c = 0; c < 4; c++) + { + if ((pXl->m_c[c] < 0.0f) || (pXh->m_c[c] > 255.0f)) + { + uint32_t lo_v = UINT32_MAX, hi_v = 0; + for (uint32_t i = 0; i < N; i++) + { + lo_v = minimumu(lo_v, pColors[i].m_c[c]); + hi_v = maximumu(hi_v, pColors[i].m_c[c]); + } + + if (lo_v == hi_v) + { + pXl->m_c[c] = (float)lo_v; + pXh->m_c[c] = (float)hi_v; + } + } + } +} + +static void compute_least_squares_endpoints_rgb(uint32_t N, const uint8_t *pSelectors, const vec4F *pSelector_weights, vec4F *pXl, vec4F *pXh, const color_rgba*pColors) +{ + float z00 = 0.0f, z01 = 0.0f, z10 = 0.0f, z11 = 0.0f; + float q00_r = 0.0f, q10_r = 0.0f, t_r = 0.0f; + float q00_g = 0.0f, q10_g = 0.0f, t_g = 0.0f; + float q00_b = 0.0f, q10_b = 0.0f, t_b = 0.0f; + for (uint32_t i = 0; i < N; i++) + { + const uint32_t sel = pSelectors[i]; + z00 += pSelector_weights[sel].m_c[0]; + z10 += pSelector_weights[sel].m_c[1]; + z11 += pSelector_weights[sel].m_c[2]; + float w = pSelector_weights[sel].m_c[3]; + q00_r += w * pColors[i].m_c[0]; t_r += pColors[i].m_c[0]; + q00_g += w * pColors[i].m_c[1]; t_g += pColors[i].m_c[1]; + q00_b += w * pColors[i].m_c[2]; t_b += pColors[i].m_c[2]; + } + + q10_r = t_r - q00_r; + q10_g = t_g - q00_g; + q10_b = t_b - q00_b; + + z01 = z10; + + float det = z00 * z11 - z01 * z10; + if (det != 0.0f) + det = 1.0f / det; + + float iz00, iz01, iz10, iz11; + iz00 = z11 * det; + iz01 = -z01 * det; + iz10 = -z10 * det; + iz11 = z00 * det; + + pXl->m_c[0] = (float)(iz00 * q00_r + iz01 * q10_r); pXh->m_c[0] = (float)(iz10 * q00_r + iz11 * q10_r); + pXl->m_c[1] = (float)(iz00 * q00_g + iz01 * q10_g); pXh->m_c[1] = (float)(iz10 * q00_g + iz11 * q10_g); + pXl->m_c[2] = (float)(iz00 * q00_b + iz01 * q10_b); pXh->m_c[2] = (float)(iz10 * q00_b + iz11 * q10_b); + pXl->m_c[3] = 255.0f; pXh->m_c[3] = 255.0f; + + for (uint32_t c = 0; c < 3; c++) + { + if ((pXl->m_c[c] < 0.0f) || (pXh->m_c[c] > 255.0f)) + { + uint32_t lo_v = UINT32_MAX, hi_v = 0; + for (uint32_t i = 0; i < N; i++) + { + lo_v = minimumu(lo_v, pColors[i].m_c[c]); + hi_v = maximumu(hi_v, pColors[i].m_c[c]); + } + + if (lo_v == hi_v) + { + pXl->m_c[c] = (float)lo_v; + pXh->m_c[c] = (float)hi_v; + } + } + } +} + +static void compute_least_squares_endpoints_a(uint32_t N, const uint8_t* pSelectors, const vec4F* pSelector_weights, float* pXl, float* pXh, const color_rgba *pColors) +{ + // Least squares using normal equations: http://www.cs.cornell.edu/~bindel/class/cs3220-s12/notes/lec10.pdf + // I did this in matrix form first, expanded out all the ops, then optimized it a bit. + float z00 = 0.0f, z01 = 0.0f, z10 = 0.0f, z11 = 0.0f; + float q00_a = 0.0f, q10_a = 0.0f, t_a = 0.0f; + for (uint32_t i = 0; i < N; i++) + { + const uint32_t sel = pSelectors[i]; + + z00 += pSelector_weights[sel].m_c[0]; + z10 += pSelector_weights[sel].m_c[1]; + z11 += pSelector_weights[sel].m_c[2]; + + float w = pSelector_weights[sel].m_c[3]; + + q00_a += w * pColors[i].m_c[3]; t_a += pColors[i].m_c[3]; + } + + q10_a = t_a - q00_a; + + z01 = z10; + + float det = z00 * z11 - z01 * z10; + if (det != 0.0f) + det = 1.0f / det; + + float iz00, iz01, iz10, iz11; + iz00 = z11 * det; + iz01 = -z01 * det; + iz10 = -z10 * det; + iz11 = z00 * det; + + *pXl = (float)(iz00 * q00_a + iz01 * q10_a); *pXh = (float)(iz10 * q00_a + iz11 * q10_a); + + if ((*pXl < 0.0f) || (*pXh > 255.0f)) + { + uint32_t lo_v = UINT32_MAX, hi_v = 0; + for (uint32_t i = 0; i < N; i++) + { + lo_v = minimumu(lo_v, pColors[i].m_c[3]); + hi_v = maximumu(hi_v, pColors[i].m_c[3]); + } + + if (lo_v == hi_v) + { + *pXl = (float)lo_v; + *pXh = (float)hi_v; + } + } +} + +struct color_cell_compressor_params +{ + uint32_t m_num_pixels; + const color_rgba *m_pPixels; + uint32_t m_num_selector_weights; + const uint32_t *m_pSelector_weights; + const vec4F *m_pSelector_weightsx; + uint32_t m_comp_bits; + uint32_t m_weights[4]; + bool m_has_alpha; + bool m_has_pbits; + bool m_endpoints_share_pbit; + bool m_perceptual; +}; + +struct color_cell_compressor_results +{ + uint64_t m_best_overall_err; + color_rgba m_low_endpoint; + color_rgba m_high_endpoint; + uint32_t m_pbits[2]; + uint8_t *m_pSelectors; + uint8_t *m_pSelectors_temp; +}; + +static inline color_rgba scale_color(const color_rgba *pC, const color_cell_compressor_params *pParams) +{ + color_rgba results; + + const uint32_t n = pParams->m_comp_bits + (pParams->m_has_pbits ? 1 : 0); + assert((n >= 4) && (n <= 8)); + + for (uint32_t i = 0; i < 4; i++) + { + uint32_t v = pC->m_c[i] << (8 - n); + v |= (v >> n); + assert(v <= 255); + results.m_c[i] = (uint8_t)(v); + } + + return results; +} + +static inline uint64_t compute_color_distance_rgb(const color_rgba *pE1, const color_rgba *pE2, bool perceptual, const uint32_t weights[4]) +{ + int dr, dg, db; + + if (perceptual) + { + const int l1 = pE1->m_c[0] * 109 + pE1->m_c[1] * 366 + pE1->m_c[2] * 37; + const int cr1 = ((int)pE1->m_c[0] << 9) - l1; + const int cb1 = ((int)pE1->m_c[2] << 9) - l1; + const int l2 = pE2->m_c[0] * 109 + pE2->m_c[1] * 366 + pE2->m_c[2] * 37; + const int cr2 = ((int)pE2->m_c[0] << 9) - l2; + const int cb2 = ((int)pE2->m_c[2] << 9) - l2; + dr = (l1 - l2) >> 8; + dg = (cr1 - cr2) >> 8; + db = (cb1 - cb2) >> 8; + } + else + { + dr = (int)pE1->m_c[0] - (int)pE2->m_c[0]; + dg = (int)pE1->m_c[1] - (int)pE2->m_c[1]; + db = (int)pE1->m_c[2] - (int)pE2->m_c[2]; + } + + return weights[0] * (uint32_t)(dr * dr) + weights[1] * (uint32_t)(dg * dg) + weights[2] * (uint32_t)(db * db); +} + +static inline uint64_t compute_color_distance_rgba(const color_rgba *pE1, const color_rgba *pE2, bool perceptual, const uint32_t weights[4]) +{ + int da = (int)pE1->m_c[3] - (int)pE2->m_c[3]; + return compute_color_distance_rgb(pE1, pE2, perceptual, weights) + (weights[3] * (uint32_t)(da * da)); +} + +static uint64_t pack_mode1_to_one_color(const color_cell_compressor_params *pParams, color_cell_compressor_results *pResults, uint32_t r, uint32_t g, uint32_t b, uint8_t *pSelectors) +{ + uint32_t best_err = UINT_MAX; + uint32_t best_p = 0; + + for (uint32_t p = 0; p < 2; p++) + { + uint32_t err = g_bc7_mode_1_optimal_endpoints[r][p].m_error + g_bc7_mode_1_optimal_endpoints[g][p].m_error + g_bc7_mode_1_optimal_endpoints[b][p].m_error; + if (err < best_err) + { + best_err = err; + best_p = p; + if (!best_err) + break; + } + } + + const endpoint_err *pEr = &g_bc7_mode_1_optimal_endpoints[r][best_p]; + const endpoint_err *pEg = &g_bc7_mode_1_optimal_endpoints[g][best_p]; + const endpoint_err *pEb = &g_bc7_mode_1_optimal_endpoints[b][best_p]; + + color_quad_u8_set(&pResults->m_low_endpoint, pEr->m_lo, pEg->m_lo, pEb->m_lo, 0); + color_quad_u8_set(&pResults->m_high_endpoint, pEr->m_hi, pEg->m_hi, pEb->m_hi, 0); + pResults->m_pbits[0] = best_p; + pResults->m_pbits[1] = 0; + + memset(pSelectors, BC7ENC_MODE_1_OPTIMAL_INDEX, pParams->m_num_pixels); + + color_rgba p; + for (uint32_t i = 0; i < 3; i++) + { + uint32_t low = ((pResults->m_low_endpoint.m_c[i] << 1) | pResults->m_pbits[0]) << 1; + low |= (low >> 7); + + uint32_t high = ((pResults->m_high_endpoint.m_c[i] << 1) | pResults->m_pbits[0]) << 1; + high |= (high >> 7); + + p.m_c[i] = (uint8_t)((low * (64 - g_bc7_weights3[BC7ENC_MODE_1_OPTIMAL_INDEX]) + high * g_bc7_weights3[BC7ENC_MODE_1_OPTIMAL_INDEX] + 32) >> 6); + } + p.m_c[3] = 255; + + uint64_t total_err = 0; + for (uint32_t i = 0; i < pParams->m_num_pixels; i++) + total_err += compute_color_distance_rgb(&p, &pParams->m_pPixels[i], pParams->m_perceptual, pParams->m_weights); + + pResults->m_best_overall_err = total_err; + + return total_err; +} + +static uint64_t pack_mode7_to_one_color(const color_cell_compressor_params* pParams, color_cell_compressor_results* pResults, uint32_t r, uint32_t g, uint32_t b, uint32_t a, + uint8_t* pSelectors, uint32_t num_pixels, const color_rgba *pPixels) +{ + uint32_t best_err = UINT_MAX; + uint32_t best_p = 0; + + for (uint32_t p = 0; p < 4; p++) + { + uint32_t hi_p = p >> 1; + uint32_t lo_p = p & 1; + uint32_t err = g_bc7_mode_7_optimal_endpoints[r][hi_p][lo_p].m_error + g_bc7_mode_7_optimal_endpoints[g][hi_p][lo_p].m_error + g_bc7_mode_7_optimal_endpoints[b][hi_p][lo_p].m_error + g_bc7_mode_7_optimal_endpoints[a][hi_p][lo_p].m_error; + if (err < best_err) + { + best_err = err; + best_p = p; + if (!best_err) + break; + } + } + + uint32_t best_hi_p = best_p >> 1; + uint32_t best_lo_p = best_p & 1; + + const endpoint_err* pEr = &g_bc7_mode_7_optimal_endpoints[r][best_hi_p][best_lo_p]; + const endpoint_err* pEg = &g_bc7_mode_7_optimal_endpoints[g][best_hi_p][best_lo_p]; + const endpoint_err* pEb = &g_bc7_mode_7_optimal_endpoints[b][best_hi_p][best_lo_p]; + const endpoint_err* pEa = &g_bc7_mode_7_optimal_endpoints[a][best_hi_p][best_lo_p]; + + color_quad_u8_set(&pResults->m_low_endpoint, pEr->m_lo, pEg->m_lo, pEb->m_lo, pEa->m_lo); + color_quad_u8_set(&pResults->m_high_endpoint, pEr->m_hi, pEg->m_hi, pEb->m_hi, pEa->m_hi); + pResults->m_pbits[0] = best_lo_p; + pResults->m_pbits[1] = best_hi_p; + + for (uint32_t i = 0; i < num_pixels; i++) + pSelectors[i] = (uint8_t)BC7E_MODE_7_OPTIMAL_INDEX; + + color_rgba p; + + for (uint32_t i = 0; i < 4; i++) + { + uint32_t low = (pResults->m_low_endpoint.m_c[i] << 1) | pResults->m_pbits[0]; + uint32_t high = (pResults->m_high_endpoint.m_c[i] << 1) | pResults->m_pbits[1]; + + low = (low << 2) | (low >> 6); + high = (high << 2) | (high >> 6); + + p.m_c[i] = (uint8_t)((low * (64 - g_bc7_weights2[BC7E_MODE_7_OPTIMAL_INDEX]) + high * g_bc7_weights2[BC7E_MODE_7_OPTIMAL_INDEX] + 32) >> 6); + } + + uint64_t total_err = 0; + for (uint32_t i = 0; i < num_pixels; i++) + total_err += compute_color_distance_rgba(&p, &pPixels[i], pParams->m_perceptual, pParams->m_weights); + + pResults->m_best_overall_err = total_err; + + return total_err; +} + +static uint64_t evaluate_solution(const color_rgba *pLow, const color_rgba *pHigh, const uint32_t pbits[2], const color_cell_compressor_params *pParams, color_cell_compressor_results *pResults, + const bc7enc_compress_block_params* pComp_params) +{ + color_rgba quantMinColor = *pLow; + color_rgba quantMaxColor = *pHigh; + + if (pParams->m_has_pbits) + { + uint32_t minPBit, maxPBit; + + if (pParams->m_endpoints_share_pbit) + maxPBit = minPBit = pbits[0]; + else + { + minPBit = pbits[0]; + maxPBit = pbits[1]; + } + + quantMinColor.m_c[0] = (uint8_t)((pLow->m_c[0] << 1) | minPBit); + quantMinColor.m_c[1] = (uint8_t)((pLow->m_c[1] << 1) | minPBit); + quantMinColor.m_c[2] = (uint8_t)((pLow->m_c[2] << 1) | minPBit); + quantMinColor.m_c[3] = (uint8_t)((pLow->m_c[3] << 1) | minPBit); + + quantMaxColor.m_c[0] = (uint8_t)((pHigh->m_c[0] << 1) | maxPBit); + quantMaxColor.m_c[1] = (uint8_t)((pHigh->m_c[1] << 1) | maxPBit); + quantMaxColor.m_c[2] = (uint8_t)((pHigh->m_c[2] << 1) | maxPBit); + quantMaxColor.m_c[3] = (uint8_t)((pHigh->m_c[3] << 1) | maxPBit); + } + + color_rgba actualMinColor = scale_color(&quantMinColor, pParams); + color_rgba actualMaxColor = scale_color(&quantMaxColor, pParams); + + const uint32_t N = pParams->m_num_selector_weights; + + color_rgba weightedColors[16]; + weightedColors[0] = actualMinColor; + weightedColors[N - 1] = actualMaxColor; + + const uint32_t nc = pParams->m_has_alpha ? 4 : 3; + for (uint32_t i = 1; i < (N - 1); i++) + for (uint32_t j = 0; j < nc; j++) + weightedColors[i].m_c[j] = (uint8_t)((actualMinColor.m_c[j] * (64 - pParams->m_pSelector_weights[i]) + actualMaxColor.m_c[j] * pParams->m_pSelector_weights[i] + 32) >> 6); + + const int lr = actualMinColor.m_c[0]; + const int lg = actualMinColor.m_c[1]; + const int lb = actualMinColor.m_c[2]; + const int dr = actualMaxColor.m_c[0] - lr; + const int dg = actualMaxColor.m_c[1] - lg; + const int db = actualMaxColor.m_c[2] - lb; + + uint64_t total_err = 0; + + if (pComp_params->m_force_selectors) + { + for (uint32_t i = 0; i < pParams->m_num_pixels; i++) + { + const uint32_t best_sel = pComp_params->m_selectors[i]; + + uint64_t best_err; + if (pParams->m_has_alpha) + best_err = compute_color_distance_rgba(&weightedColors[best_sel], &pParams->m_pPixels[i], pParams->m_perceptual, pParams->m_weights); + else + best_err = compute_color_distance_rgb(&weightedColors[best_sel], &pParams->m_pPixels[i], pParams->m_perceptual, pParams->m_weights); + + total_err += best_err; + + pResults->m_pSelectors_temp[i] = (uint8_t)best_sel; + } + } + else if (!pParams->m_perceptual) + { + if (pParams->m_has_alpha) + { + const int la = actualMinColor.m_c[3]; + const int da = actualMaxColor.m_c[3] - la; + + const float f = N / (float)(squarei(dr) + squarei(dg) + squarei(db) + squarei(da) + .00000125f); + + for (uint32_t i = 0; i < pParams->m_num_pixels; i++) + { + const color_rgba *pC = &pParams->m_pPixels[i]; + int r = pC->m_c[0]; + int g = pC->m_c[1]; + int b = pC->m_c[2]; + int a = pC->m_c[3]; + + int best_sel = (int)((float)((r - lr) * dr + (g - lg) * dg + (b - lb) * db + (a - la) * da) * f + .5f); + best_sel = clampi(best_sel, 1, N - 1); + + uint64_t err0 = compute_color_distance_rgba(&weightedColors[best_sel - 1], pC, false, pParams->m_weights); + uint64_t err1 = compute_color_distance_rgba(&weightedColors[best_sel], pC, false, pParams->m_weights); + + if (err1 > err0) + { + err1 = err0; + --best_sel; + } + total_err += err1; + + pResults->m_pSelectors_temp[i] = (uint8_t)best_sel; + } + } + else + { + const float f = N / (float)(squarei(dr) + squarei(dg) + squarei(db) + .00000125f); + + for (uint32_t i = 0; i < pParams->m_num_pixels; i++) + { + const color_rgba *pC = &pParams->m_pPixels[i]; + int r = pC->m_c[0]; + int g = pC->m_c[1]; + int b = pC->m_c[2]; + + int sel = (int)((float)((r - lr) * dr + (g - lg) * dg + (b - lb) * db) * f + .5f); + sel = clampi(sel, 1, N - 1); + + uint64_t err0 = compute_color_distance_rgb(&weightedColors[sel - 1], pC, false, pParams->m_weights); + uint64_t err1 = compute_color_distance_rgb(&weightedColors[sel], pC, false, pParams->m_weights); + + int best_sel = sel; + uint64_t best_err = err1; + if (err0 < best_err) + { + best_err = err0; + best_sel = sel - 1; + } + + total_err += best_err; + + pResults->m_pSelectors_temp[i] = (uint8_t)best_sel; + } + } + } + else + { + // TODO: This could be improved. + for (uint32_t i = 0; i < pParams->m_num_pixels; i++) + { + uint64_t best_err = UINT64_MAX; + uint32_t best_sel = 0; + + if (pParams->m_has_alpha) + { + for (uint32_t j = 0; j < N; j++) + { + uint64_t err = compute_color_distance_rgba(&weightedColors[j], &pParams->m_pPixels[i], true, pParams->m_weights); + if (err < best_err) + { + best_err = err; + best_sel = j; + } + } + } + else + { + for (uint32_t j = 0; j < N; j++) + { + uint64_t err = compute_color_distance_rgb(&weightedColors[j], &pParams->m_pPixels[i], true, pParams->m_weights); + if (err < best_err) + { + best_err = err; + best_sel = j; + } + } + } + + total_err += best_err; + + pResults->m_pSelectors_temp[i] = (uint8_t)best_sel; + } + } + + if (total_err < pResults->m_best_overall_err) + { + pResults->m_best_overall_err = total_err; + + pResults->m_low_endpoint = *pLow; + pResults->m_high_endpoint = *pHigh; + + pResults->m_pbits[0] = pbits[0]; + pResults->m_pbits[1] = pbits[1]; + + memcpy(pResults->m_pSelectors, pResults->m_pSelectors_temp, sizeof(pResults->m_pSelectors[0]) * pParams->m_num_pixels); + } + + return total_err; +} + +static void fixDegenerateEndpoints(uint32_t mode, color_rgba *pTrialMinColor, color_rgba *pTrialMaxColor, const vec4F *pXl, const vec4F *pXh, uint32_t iscale, + const bc7enc_compress_block_params* pComp_params) +{ + //if ((mode == 1) || (mode == 7)) + //if (mode == 1) + if ( (mode == 1) || ((mode == 6) && (pComp_params->m_quant_mode6_endpoints)) ) + { + // fix degenerate case where the input collapses to a single colorspace voxel, and we loose all freedom (test with grayscale ramps) + for (uint32_t i = 0; i < 3; i++) + { + if (pTrialMinColor->m_c[i] == pTrialMaxColor->m_c[i]) + { + if (fabs(pXl->m_c[i] - pXh->m_c[i]) > 0.0f) + { + if (pTrialMinColor->m_c[i] > (iscale >> 1)) + { + if (pTrialMinColor->m_c[i] > 0) + pTrialMinColor->m_c[i]--; + else + if (pTrialMaxColor->m_c[i] < iscale) + pTrialMaxColor->m_c[i]++; + } + else + { + if (pTrialMaxColor->m_c[i] < iscale) + pTrialMaxColor->m_c[i]++; + else if (pTrialMinColor->m_c[i] > 0) + pTrialMinColor->m_c[i]--; + } + } + } + } + } +} + +static uint64_t find_optimal_solution(uint32_t mode, vec4F xl, vec4F xh, const color_cell_compressor_params *pParams, color_cell_compressor_results *pResults, + const bc7enc_compress_block_params* pComp_params) +{ + vec4F_saturate_in_place(&xl); vec4F_saturate_in_place(&xh); + + if (pParams->m_has_pbits) + { + const int iscalep = (1 << (pParams->m_comp_bits + 1)) - 1; + const float scalep = (float)iscalep; + + const int32_t totalComps = pParams->m_has_alpha ? 4 : 3; + + uint32_t best_pbits[2]; + color_rgba bestMinColor, bestMaxColor; + + if (!pParams->m_endpoints_share_pbit) + { + if ((pParams->m_comp_bits == 7) && (pComp_params->m_quant_mode6_endpoints)) + { + best_pbits[0] = 0; + bestMinColor.m_c[0] = g_mode6_reduced_quant[(int)((xl.m_c[0] * 2047.0f) + .5f)][0]; + bestMinColor.m_c[1] = g_mode6_reduced_quant[(int)((xl.m_c[1] * 2047.0f) + .5f)][0]; + bestMinColor.m_c[2] = g_mode6_reduced_quant[(int)((xl.m_c[2] * 2047.0f) + .5f)][0]; + bestMinColor.m_c[3] = g_mode6_reduced_quant[(int)((xl.m_c[3] * 2047.0f) + .5f)][0]; + + best_pbits[1] = 1; + bestMaxColor.m_c[0] = g_mode6_reduced_quant[(int)((xh.m_c[0] * 2047.0f) + .5f)][1]; + bestMaxColor.m_c[1] = g_mode6_reduced_quant[(int)((xh.m_c[1] * 2047.0f) + .5f)][1]; + bestMaxColor.m_c[2] = g_mode6_reduced_quant[(int)((xh.m_c[2] * 2047.0f) + .5f)][1]; + bestMaxColor.m_c[3] = g_mode6_reduced_quant[(int)((xh.m_c[3] * 2047.0f) + .5f)][1]; + } + else + { + float best_err0 = 1e+9; + float best_err1 = 1e+9; + + for (int p = 0; p < 2; p++) + { + color_rgba xMinColor, xMaxColor; + + // Notes: The pbit controls which quantization intervals are selected. + // total_levels=2^(comp_bits+1), where comp_bits=4 for mode 0, etc. + // pbit 0: v=(b*2)/(total_levels-1), pbit 1: v=(b*2+1)/(total_levels-1) where b is the component bin from [0,total_levels/2-1] and v is the [0,1] component value + // rearranging you get for pbit 0: b=floor(v*(total_levels-1)/2+.5) + // rearranging you get for pbit 1: b=floor((v*(total_levels-1)-1)/2+.5) + if (pParams->m_comp_bits == 5) + { + for (uint32_t c = 0; c < 4; c++) + { + int vl = (int)(xl.m_c[c] * 31.0f); + vl += (xl.m_c[c] > g_mode7_rgba_midpoints[vl][p]); + xMinColor.m_c[c] = (uint8_t)clampi(vl * 2 + p, p, 63 - 1 + p); + + int vh = (int)(xh.m_c[c] * 31.0f); + vh += (xh.m_c[c] > g_mode7_rgba_midpoints[vh][p]); + xMaxColor.m_c[c] = (uint8_t)clampi(vh * 2 + p, p, 63 - 1 + p); + } + } + else + { + for (uint32_t c = 0; c < 4; c++) + { + xMinColor.m_c[c] = (uint8_t)(clampi(((int)((xl.m_c[c] * scalep - p) / 2.0f + .5f)) * 2 + p, p, iscalep - 1 + p)); + xMaxColor.m_c[c] = (uint8_t)(clampi(((int)((xh.m_c[c] * scalep - p) / 2.0f + .5f)) * 2 + p, p, iscalep - 1 + p)); + } + } + + color_rgba scaledLow = scale_color(&xMinColor, pParams); + color_rgba scaledHigh = scale_color(&xMaxColor, pParams); + + float err0 = 0, err1 = 0; + for (int i = 0; i < totalComps; i++) + { + err0 += squaref(scaledLow.m_c[i] - xl.m_c[i] * 255.0f); + err1 += squaref(scaledHigh.m_c[i] - xh.m_c[i] * 255.0f); + } + + if (p == 1) + { + err0 *= pComp_params->m_pbit1_weight; + err1 *= pComp_params->m_pbit1_weight; + } + + if (err0 < best_err0) + { + best_err0 = err0; + best_pbits[0] = p; + + bestMinColor.m_c[0] = xMinColor.m_c[0] >> 1; + bestMinColor.m_c[1] = xMinColor.m_c[1] >> 1; + bestMinColor.m_c[2] = xMinColor.m_c[2] >> 1; + bestMinColor.m_c[3] = xMinColor.m_c[3] >> 1; + } + + if (err1 < best_err1) + { + best_err1 = err1; + best_pbits[1] = p; + + bestMaxColor.m_c[0] = xMaxColor.m_c[0] >> 1; + bestMaxColor.m_c[1] = xMaxColor.m_c[1] >> 1; + bestMaxColor.m_c[2] = xMaxColor.m_c[2] >> 1; + bestMaxColor.m_c[3] = xMaxColor.m_c[3] >> 1; + } + } + } + } + else + { + if ((mode == 1) && (pComp_params->m_bias_mode1_pbits)) + { + float x = 0.0f; + for (uint32_t c = 0; c < 3; c++) + x = std::max(std::max(x, xl.m_c[c]), xh.m_c[c]); + + int p = 0; + if (x > (253.0f / 255.0f)) + p = 1; + + color_rgba xMinColor, xMaxColor; + for (uint32_t c = 0; c < 4; c++) + { + int vl = (int)(xl.m_c[c] * 63.0f); + vl += (xl.m_c[c] > g_mode1_rgba_midpoints[vl][p]); + xMinColor.m_c[c] = (uint8_t)clampi(vl * 2 + p, p, 127 - 1 + p); + + int vh = (int)(xh.m_c[c] * 63.0f); + vh += (xh.m_c[c] > g_mode1_rgba_midpoints[vh][p]); + xMaxColor.m_c[c] = (uint8_t)clampi(vh * 2 + p, p, 127 - 1 + p); + } + + best_pbits[0] = p; + best_pbits[1] = p; + for (uint32_t j = 0; j < 4; j++) + { + bestMinColor.m_c[j] = xMinColor.m_c[j] >> 1; + bestMaxColor.m_c[j] = xMaxColor.m_c[j] >> 1; + } + } + else + { + // Endpoints share pbits + float best_err = 1e+9; + + for (int p = 0; p < 2; p++) + { + color_rgba xMinColor, xMaxColor; + if (pParams->m_comp_bits == 6) + { + for (uint32_t c = 0; c < 4; c++) + { + int vl = (int)(xl.m_c[c] * 63.0f); + vl += (xl.m_c[c] > g_mode1_rgba_midpoints[vl][p]); + xMinColor.m_c[c] = (uint8_t)clampi(vl * 2 + p, p, 127 - 1 + p); + + int vh = (int)(xh.m_c[c] * 63.0f); + vh += (xh.m_c[c] > g_mode1_rgba_midpoints[vh][p]); + xMaxColor.m_c[c] = (uint8_t)clampi(vh * 2 + p, p, 127 - 1 + p); + } + } + else + { + for (uint32_t c = 0; c < 4; c++) + { + xMinColor.m_c[c] = (uint8_t)(clampi(((int)((xl.m_c[c] * scalep - p) / 2.0f + .5f)) * 2 + p, p, iscalep - 1 + p)); + xMaxColor.m_c[c] = (uint8_t)(clampi(((int)((xh.m_c[c] * scalep - p) / 2.0f + .5f)) * 2 + p, p, iscalep - 1 + p)); + } + } + + color_rgba scaledLow = scale_color(&xMinColor, pParams); + color_rgba scaledHigh = scale_color(&xMaxColor, pParams); + + float err = 0; + for (int i = 0; i < totalComps; i++) + err += squaref((scaledLow.m_c[i] / 255.0f) - xl.m_c[i]) + squaref((scaledHigh.m_c[i] / 255.0f) - xh.m_c[i]); + + if (p == 1) + err *= pComp_params->m_pbit1_weight; + + if (err < best_err) + { + best_err = err; + best_pbits[0] = p; + best_pbits[1] = p; + for (uint32_t j = 0; j < 4; j++) + { + bestMinColor.m_c[j] = xMinColor.m_c[j] >> 1; + bestMaxColor.m_c[j] = xMaxColor.m_c[j] >> 1; + } + } + } + } + } + + fixDegenerateEndpoints(mode, &bestMinColor, &bestMaxColor, &xl, &xh, iscalep >> 1, pComp_params); + + if ((pResults->m_best_overall_err == UINT64_MAX) || color_quad_u8_notequals(&bestMinColor, &pResults->m_low_endpoint) || color_quad_u8_notequals(&bestMaxColor, &pResults->m_high_endpoint) || (best_pbits[0] != pResults->m_pbits[0]) || (best_pbits[1] != pResults->m_pbits[1])) + evaluate_solution(&bestMinColor, &bestMaxColor, best_pbits, pParams, pResults, pComp_params); + } + else + { + const int iscale = (1 << pParams->m_comp_bits) - 1; + const float scale = (float)iscale; + + color_rgba trialMinColor, trialMaxColor; + if (pParams->m_comp_bits == 7) + { + for (uint32_t c = 0; c < 4; c++) + { + int vl = (int)(xl.m_c[c] * 127.0f); + vl += (xl.m_c[c] > g_mode5_rgba_midpoints[vl]); + trialMinColor.m_c[c] = (uint8_t)clampi(vl, 0, 127); + + int vh = (int)(xh.m_c[c] * 127.0f); + vh += (xh.m_c[c] > g_mode5_rgba_midpoints[vh]); + trialMaxColor.m_c[c] = (uint8_t)clampi(vh, 0, 127); + } + } + else + { + color_quad_u8_set_clamped(&trialMinColor, (int)(xl.m_c[0] * scale + .5f), (int)(xl.m_c[1] * scale + .5f), (int)(xl.m_c[2] * scale + .5f), (int)(xl.m_c[3] * scale + .5f)); + color_quad_u8_set_clamped(&trialMaxColor, (int)(xh.m_c[0] * scale + .5f), (int)(xh.m_c[1] * scale + .5f), (int)(xh.m_c[2] * scale + .5f), (int)(xh.m_c[3] * scale + .5f)); + } + + fixDegenerateEndpoints(mode, &trialMinColor, &trialMaxColor, &xl, &xh, iscale, pComp_params); + + if ((pResults->m_best_overall_err == UINT64_MAX) || color_quad_u8_notequals(&trialMinColor, &pResults->m_low_endpoint) || color_quad_u8_notequals(&trialMaxColor, &pResults->m_high_endpoint)) + evaluate_solution(&trialMinColor, &trialMaxColor, pResults->m_pbits, pParams, pResults, pComp_params); + } + + return pResults->m_best_overall_err; +} + +static uint64_t color_cell_compression(uint32_t mode, const color_cell_compressor_params *pParams, color_cell_compressor_results *pResults, const bc7enc_compress_block_params *pComp_params) +{ + assert((mode == 6) || (mode == 7) || (!pParams->m_has_alpha)); + + pResults->m_best_overall_err = UINT64_MAX; + + // If the partition's colors are all the same in mode 1, then just pack them as a single color. + if (mode == 1) + { + const uint32_t cr = pParams->m_pPixels[0].m_c[0], cg = pParams->m_pPixels[0].m_c[1], cb = pParams->m_pPixels[0].m_c[2]; + + bool allSame = true; + for (uint32_t i = 1; i < pParams->m_num_pixels; i++) + { + if ((cr != pParams->m_pPixels[i].m_c[0]) || (cg != pParams->m_pPixels[i].m_c[1]) || (cb != pParams->m_pPixels[i].m_c[2])) + { + allSame = false; + break; + } + } + + if (allSame) + return pack_mode1_to_one_color(pParams, pResults, cr, cg, cb, pResults->m_pSelectors); + } + else if (mode == 7) + { + const uint32_t cr = pParams->m_pPixels[0].m_c[0], cg = pParams->m_pPixels[0].m_c[1], cb = pParams->m_pPixels[0].m_c[2], ca = pParams->m_pPixels[0].m_c[3]; + + bool allSame = true; + for (uint32_t i = 1; i < pParams->m_num_pixels; i++) + { + if ((cr != pParams->m_pPixels[i].m_c[0]) || (cg != pParams->m_pPixels[i].m_c[1]) || (cb != pParams->m_pPixels[i].m_c[2]) || (ca != pParams->m_pPixels[i].m_c[3])) + { + allSame = false; + break; + } + } + + if (allSame) + return pack_mode7_to_one_color(pParams, pResults, cr, cg, cb, ca, pResults->m_pSelectors, pParams->m_num_pixels, pParams->m_pPixels); + } + + // Compute partition's mean color and principle axis. + vec4F meanColor, axis; + vec4F_set_scalar(&meanColor, 0.0f); + + for (uint32_t i = 0; i < pParams->m_num_pixels; i++) + { + vec4F color = vec4F_from_color(&pParams->m_pPixels[i]); + meanColor = vec4F_add(&meanColor, &color); + } + + vec4F meanColorScaled = vec4F_mul(&meanColor, 1.0f / (float)(pParams->m_num_pixels)); + + meanColor = vec4F_mul(&meanColor, 1.0f / (float)(pParams->m_num_pixels * 255.0f)); + vec4F_saturate_in_place(&meanColor); + + if (pParams->m_has_alpha) + { + // Use incremental PCA for RGBA PCA, because it's simple. + vec4F_set_scalar(&axis, 0.0f); + for (uint32_t i = 0; i < pParams->m_num_pixels; i++) + { + vec4F color = vec4F_from_color(&pParams->m_pPixels[i]); + color = vec4F_sub(&color, &meanColorScaled); + vec4F a = vec4F_mul(&color, color.m_c[0]); + vec4F b = vec4F_mul(&color, color.m_c[1]); + vec4F c = vec4F_mul(&color, color.m_c[2]); + vec4F d = vec4F_mul(&color, color.m_c[3]); + vec4F n = i ? axis : color; + vec4F_normalize_in_place(&n); + axis.m_c[0] += vec4F_dot(&a, &n); + axis.m_c[1] += vec4F_dot(&b, &n); + axis.m_c[2] += vec4F_dot(&c, &n); + axis.m_c[3] += vec4F_dot(&d, &n); + } + vec4F_normalize_in_place(&axis); + } + else + { + // Use covar technique for RGB PCA, because it doesn't require per-pixel normalization. + float cov[6] = { 0, 0, 0, 0, 0, 0 }; + + for (uint32_t i = 0; i < pParams->m_num_pixels; i++) + { + const color_rgba *pV = &pParams->m_pPixels[i]; + float r = pV->m_c[0] - meanColorScaled.m_c[0]; + float g = pV->m_c[1] - meanColorScaled.m_c[1]; + float b = pV->m_c[2] - meanColorScaled.m_c[2]; + cov[0] += r*r; cov[1] += r*g; cov[2] += r*b; cov[3] += g*g; cov[4] += g*b; cov[5] += b*b; + } + + float vfr = .9f, vfg = 1.0f, vfb = .7f; + for (uint32_t iter = 0; iter < 3; iter++) + { + float r = vfr*cov[0] + vfg*cov[1] + vfb*cov[2]; + float g = vfr*cov[1] + vfg*cov[3] + vfb*cov[4]; + float b = vfr*cov[2] + vfg*cov[4] + vfb*cov[5]; + + float m = maximumf(maximumf(fabsf(r), fabsf(g)), fabsf(b)); + if (m > 1e-10f) + { + m = 1.0f / m; + r *= m; g *= m; b *= m; + } + + vfr = r; vfg = g; vfb = b; + } + + float len = vfr*vfr + vfg*vfg + vfb*vfb; + if (len < 1e-10f) + vec4F_set_scalar(&axis, 0.0f); + else + { + len = 1.0f / sqrtf(len); + vfr *= len; vfg *= len; vfb *= len; + vec4F_set(&axis, vfr, vfg, vfb, 0); + } + } + + // TODO: Try picking the 2 colors with the largest projection onto the axis, instead of computing new colors along the axis. + + if (vec4F_dot(&axis, &axis) < .5f) + { + if (pParams->m_perceptual) + vec4F_set(&axis, .213f, .715f, .072f, pParams->m_has_alpha ? .715f : 0); + else + vec4F_set(&axis, 1.0f, 1.0f, 1.0f, pParams->m_has_alpha ? 1.0f : 0); + vec4F_normalize_in_place(&axis); + } + + float l = 1e+9f, h = -1e+9f; + + for (uint32_t i = 0; i < pParams->m_num_pixels; i++) + { + vec4F color = vec4F_from_color(&pParams->m_pPixels[i]); + + vec4F q = vec4F_sub(&color, &meanColorScaled); + float d = vec4F_dot(&q, &axis); + + l = minimumf(l, d); + h = maximumf(h, d); + } + + l *= (1.0f / 255.0f); + h *= (1.0f / 255.0f); + + vec4F b0 = vec4F_mul(&axis, l); + vec4F b1 = vec4F_mul(&axis, h); + vec4F c0 = vec4F_add(&meanColor, &b0); + vec4F c1 = vec4F_add(&meanColor, &b1); + vec4F minColor = vec4F_saturate(&c0); + vec4F maxColor = vec4F_saturate(&c1); + + vec4F whiteVec; + vec4F_set_scalar(&whiteVec, 1.0f); + + if (vec4F_dot(&minColor, &whiteVec) > vec4F_dot(&maxColor, &whiteVec)) + { +#if 0 + // Don't compile correctly with VC 2019 in release. + vec4F temp = minColor; + minColor = maxColor; + maxColor = temp; +#else + float a = minColor.m_c[0], b = minColor.m_c[1], c = minColor.m_c[2], d = minColor.m_c[3]; + minColor.m_c[0] = maxColor.m_c[0]; + minColor.m_c[1] = maxColor.m_c[1]; + minColor.m_c[2] = maxColor.m_c[2]; + minColor.m_c[3] = maxColor.m_c[3]; + maxColor.m_c[0] = a; + maxColor.m_c[1] = b; + maxColor.m_c[2] = c; + maxColor.m_c[3] = d; +#endif + } + + // First find a solution using the block's PCA. + if (!find_optimal_solution(mode, minColor, maxColor, pParams, pResults, pComp_params)) + return 0; + + if (pComp_params->m_try_least_squares) + { + // Now try to refine the solution using least squares by computing the optimal endpoints from the current selectors. + vec4F xl, xh; + vec4F_set_scalar(&xl, 0.0f); + vec4F_set_scalar(&xh, 0.0f); + if (pParams->m_has_alpha) + compute_least_squares_endpoints_rgba(pParams->m_num_pixels, pResults->m_pSelectors, pParams->m_pSelector_weightsx, &xl, &xh, pParams->m_pPixels); + else + compute_least_squares_endpoints_rgb(pParams->m_num_pixels, pResults->m_pSelectors, pParams->m_pSelector_weightsx, &xl, &xh, pParams->m_pPixels); + + xl = vec4F_mul(&xl, (1.0f / 255.0f)); + xh = vec4F_mul(&xh, (1.0f / 255.0f)); + + if (!find_optimal_solution(mode, xl, xh, pParams, pResults, pComp_params)) + return 0; + } + + if (pComp_params->m_uber_level > 0) + { + // In uber level 1, try varying the selectors a little, somewhat like cluster fit would. First try incrementing the minimum selectors, + // then try decrementing the selectrors, then try both. + uint8_t selectors_temp[16], selectors_temp1[16]; + memcpy(selectors_temp, pResults->m_pSelectors, pParams->m_num_pixels); + + const int max_selector = pParams->m_num_selector_weights - 1; + + uint32_t min_sel = 16; + uint32_t max_sel = 0; + for (uint32_t i = 0; i < pParams->m_num_pixels; i++) + { + uint32_t sel = selectors_temp[i]; + min_sel = minimumu(min_sel, sel); + max_sel = maximumu(max_sel, sel); + } + + for (uint32_t i = 0; i < pParams->m_num_pixels; i++) + { + uint32_t sel = selectors_temp[i]; + if ((sel == min_sel) && (sel < (pParams->m_num_selector_weights - 1))) + sel++; + selectors_temp1[i] = (uint8_t)sel; + } + + vec4F xl, xh; + vec4F_set_scalar(&xl, 0.0f); + vec4F_set_scalar(&xh, 0.0f); + if (pParams->m_has_alpha) + compute_least_squares_endpoints_rgba(pParams->m_num_pixels, selectors_temp1, pParams->m_pSelector_weightsx, &xl, &xh, pParams->m_pPixels); + else + compute_least_squares_endpoints_rgb(pParams->m_num_pixels, selectors_temp1, pParams->m_pSelector_weightsx, &xl, &xh, pParams->m_pPixels); + + xl = vec4F_mul(&xl, (1.0f / 255.0f)); + xh = vec4F_mul(&xh, (1.0f / 255.0f)); + + if (!find_optimal_solution(mode, xl, xh, pParams, pResults, pComp_params)) + return 0; + + for (uint32_t i = 0; i < pParams->m_num_pixels; i++) + { + uint32_t sel = selectors_temp[i]; + if ((sel == max_sel) && (sel > 0)) + sel--; + selectors_temp1[i] = (uint8_t)sel; + } + + if (pParams->m_has_alpha) + compute_least_squares_endpoints_rgba(pParams->m_num_pixels, selectors_temp1, pParams->m_pSelector_weightsx, &xl, &xh, pParams->m_pPixels); + else + compute_least_squares_endpoints_rgb(pParams->m_num_pixels, selectors_temp1, pParams->m_pSelector_weightsx, &xl, &xh, pParams->m_pPixels); + + xl = vec4F_mul(&xl, (1.0f / 255.0f)); + xh = vec4F_mul(&xh, (1.0f / 255.0f)); + + if (!find_optimal_solution(mode, xl, xh, pParams, pResults, pComp_params)) + return 0; + + for (uint32_t i = 0; i < pParams->m_num_pixels; i++) + { + uint32_t sel = selectors_temp[i]; + if ((sel == min_sel) && (sel < (pParams->m_num_selector_weights - 1))) + sel++; + else if ((sel == max_sel) && (sel > 0)) + sel--; + selectors_temp1[i] = (uint8_t)sel; + } + + if (pParams->m_has_alpha) + compute_least_squares_endpoints_rgba(pParams->m_num_pixels, selectors_temp1, pParams->m_pSelector_weightsx, &xl, &xh, pParams->m_pPixels); + else + compute_least_squares_endpoints_rgb(pParams->m_num_pixels, selectors_temp1, pParams->m_pSelector_weightsx, &xl, &xh, pParams->m_pPixels); + + xl = vec4F_mul(&xl, (1.0f / 255.0f)); + xh = vec4F_mul(&xh, (1.0f / 255.0f)); + + if (!find_optimal_solution(mode, xl, xh, pParams, pResults, pComp_params)) + return 0; + + // In uber levels 2+, try taking more advantage of endpoint extrapolation by scaling the selectors in one direction or another. + const uint32_t uber_err_thresh = (pParams->m_num_pixels * 56) >> 4; + if ((pComp_params->m_uber_level >= 2) && (pResults->m_best_overall_err > uber_err_thresh)) + { + const int Q = (pComp_params->m_uber_level >= 4) ? (pComp_params->m_uber_level - 2) : 1; + for (int ly = -Q; ly <= 1; ly++) + { + for (int hy = max_selector - 1; hy <= (max_selector + Q); hy++) + { + if ((ly == 0) && (hy == max_selector)) + continue; + + for (uint32_t i = 0; i < pParams->m_num_pixels; i++) + selectors_temp1[i] = (uint8_t)clampf(floorf((float)max_selector * ((float)selectors_temp[i] - (float)ly) / ((float)hy - (float)ly) + .5f), 0, (float)max_selector); + + //vec4F xl, xh; + vec4F_set_scalar(&xl, 0.0f); + vec4F_set_scalar(&xh, 0.0f); + if (pParams->m_has_alpha) + compute_least_squares_endpoints_rgba(pParams->m_num_pixels, selectors_temp1, pParams->m_pSelector_weightsx, &xl, &xh, pParams->m_pPixels); + else + compute_least_squares_endpoints_rgb(pParams->m_num_pixels, selectors_temp1, pParams->m_pSelector_weightsx, &xl, &xh, pParams->m_pPixels); + + xl = vec4F_mul(&xl, (1.0f / 255.0f)); + xh = vec4F_mul(&xh, (1.0f / 255.0f)); + + if (!find_optimal_solution(mode, xl, xh, pParams, pResults, pComp_params)) + return 0; + } + } + } + } + + if (mode == 1) + { + // Try encoding the partition as a single color by using the optimal singe colors tables to encode the block to its mean. + color_cell_compressor_results avg_results = *pResults; + const uint32_t r = (int)(.5f + meanColor.m_c[0] * 255.0f), g = (int)(.5f + meanColor.m_c[1] * 255.0f), b = (int)(.5f + meanColor.m_c[2] * 255.0f); + uint64_t avg_err = pack_mode1_to_one_color(pParams, &avg_results, r, g, b, pResults->m_pSelectors_temp); + if (avg_err < pResults->m_best_overall_err) + { + *pResults = avg_results; + memcpy(pResults->m_pSelectors, pResults->m_pSelectors_temp, sizeof(pResults->m_pSelectors[0]) * pParams->m_num_pixels); + pResults->m_best_overall_err = avg_err; + } + } + else if (mode == 7) + { + // Try encoding the partition as a single color by using the optimal singe colors tables to encode the block to its mean. + color_cell_compressor_results avg_results = *pResults; + const uint32_t r = (int)(.5f + meanColor.m_c[0] * 255.0f), g = (int)(.5f + meanColor.m_c[1] * 255.0f), b = (int)(.5f + meanColor.m_c[2] * 255.0f), a = (int)(.5f + meanColor.m_c[3] * 255.0f); + uint64_t avg_err = pack_mode7_to_one_color(pParams, &avg_results, r, g, b, a, pResults->m_pSelectors_temp, pParams->m_num_pixels, pParams->m_pPixels); + if (avg_err < pResults->m_best_overall_err) + { + *pResults = avg_results; + memcpy(pResults->m_pSelectors, pResults->m_pSelectors_temp, sizeof(pResults->m_pSelectors[0]) * pParams->m_num_pixels); + pResults->m_best_overall_err = avg_err; + } + } + + return pResults->m_best_overall_err; +} + +static uint64_t color_cell_compression_est_mode1(uint32_t num_pixels, const color_rgba *pPixels, bool perceptual, uint32_t pweights[4], uint64_t best_err_so_far) +{ + // Find RGB bounds as an approximation of the block's principle axis + uint32_t lr = 255, lg = 255, lb = 255; + uint32_t hr = 0, hg = 0, hb = 0; + for (uint32_t i = 0; i < num_pixels; i++) + { + const color_rgba *pC = &pPixels[i]; + if (pC->m_c[0] < lr) lr = pC->m_c[0]; + if (pC->m_c[1] < lg) lg = pC->m_c[1]; + if (pC->m_c[2] < lb) lb = pC->m_c[2]; + if (pC->m_c[0] > hr) hr = pC->m_c[0]; + if (pC->m_c[1] > hg) hg = pC->m_c[1]; + if (pC->m_c[2] > hb) hb = pC->m_c[2]; + } + + color_rgba lowColor; color_quad_u8_set(&lowColor, lr, lg, lb, 0); + color_rgba highColor; color_quad_u8_set(&highColor, hr, hg, hb, 0); + + // Place endpoints at bbox diagonals and compute interpolated colors + const uint32_t N = 8; + color_rgba weightedColors[8]; + + weightedColors[0] = lowColor; + weightedColors[N - 1] = highColor; + for (uint32_t i = 1; i < (N - 1); i++) + { + weightedColors[i].m_c[0] = (uint8_t)((lowColor.m_c[0] * (64 - g_bc7_weights3[i]) + highColor.m_c[0] * g_bc7_weights3[i] + 32) >> 6); + weightedColors[i].m_c[1] = (uint8_t)((lowColor.m_c[1] * (64 - g_bc7_weights3[i]) + highColor.m_c[1] * g_bc7_weights3[i] + 32) >> 6); + weightedColors[i].m_c[2] = (uint8_t)((lowColor.m_c[2] * (64 - g_bc7_weights3[i]) + highColor.m_c[2] * g_bc7_weights3[i] + 32) >> 6); + } + + // Compute dots and thresholds + const int ar = highColor.m_c[0] - lowColor.m_c[0]; + const int ag = highColor.m_c[1] - lowColor.m_c[1]; + const int ab = highColor.m_c[2] - lowColor.m_c[2]; + + int dots[8]; + for (uint32_t i = 0; i < N; i++) + dots[i] = weightedColors[i].m_c[0] * ar + weightedColors[i].m_c[1] * ag + weightedColors[i].m_c[2] * ab; + + int thresh[8 - 1]; + for (uint32_t i = 0; i < (N - 1); i++) + thresh[i] = (dots[i] + dots[i + 1] + 1) >> 1; + + uint64_t total_err = 0; + if (perceptual) + { + // Transform block's interpolated colors to YCbCr + int l1[8], cr1[8], cb1[8]; + for (int j = 0; j < 8; j++) + { + const color_rgba *pE1 = &weightedColors[j]; + l1[j] = pE1->m_c[0] * 109 + pE1->m_c[1] * 366 + pE1->m_c[2] * 37; + cr1[j] = ((int)pE1->m_c[0] << 9) - l1[j]; + cb1[j] = ((int)pE1->m_c[2] << 9) - l1[j]; + } + + for (uint32_t i = 0; i < num_pixels; i++) + { + const color_rgba *pC = &pPixels[i]; + + int d = ar * pC->m_c[0] + ag * pC->m_c[1] + ab * pC->m_c[2]; + + // Find approximate selector + uint32_t s = 0; + if (d >= thresh[6]) + s = 7; + else if (d >= thresh[5]) + s = 6; + else if (d >= thresh[4]) + s = 5; + else if (d >= thresh[3]) + s = 4; + else if (d >= thresh[2]) + s = 3; + else if (d >= thresh[1]) + s = 2; + else if (d >= thresh[0]) + s = 1; + + // Compute error + const int l2 = pC->m_c[0] * 109 + pC->m_c[1] * 366 + pC->m_c[2] * 37; + const int cr2 = ((int)pC->m_c[0] << 9) - l2; + const int cb2 = ((int)pC->m_c[2] << 9) - l2; + + const int dl = (l1[s] - l2) >> 8; + const int dcr = (cr1[s] - cr2) >> 8; + const int dcb = (cb1[s] - cb2) >> 8; + + int ie = (pweights[0] * dl * dl) + (pweights[1] * dcr * dcr) + (pweights[2] * dcb * dcb); + + total_err += ie; + if (total_err > best_err_so_far) + break; + } + } + else + { + for (uint32_t i = 0; i < num_pixels; i++) + { + const color_rgba *pC = &pPixels[i]; + + int d = ar * pC->m_c[0] + ag * pC->m_c[1] + ab * pC->m_c[2]; + + // Find approximate selector + uint32_t s = 0; + if (d >= thresh[6]) + s = 7; + else if (d >= thresh[5]) + s = 6; + else if (d >= thresh[4]) + s = 5; + else if (d >= thresh[3]) + s = 4; + else if (d >= thresh[2]) + s = 3; + else if (d >= thresh[1]) + s = 2; + else if (d >= thresh[0]) + s = 1; + + // Compute error + const color_rgba *pE1 = &weightedColors[s]; + + int dr = (int)pE1->m_c[0] - (int)pC->m_c[0]; + int dg = (int)pE1->m_c[1] - (int)pC->m_c[1]; + int db = (int)pE1->m_c[2] - (int)pC->m_c[2]; + + total_err += pweights[0] * (dr * dr) + pweights[1] * (dg * dg) + pweights[2] * (db * db); + if (total_err > best_err_so_far) + break; + } + } + + return total_err; +} + +static uint64_t color_cell_compression_est_mode7(uint32_t num_pixels, const color_rgba * pPixels, bool perceptual, uint32_t pweights[4], uint64_t best_err_so_far) +{ + // Find RGB bounds as an approximation of the block's principle axis + uint32_t lr = 255, lg = 255, lb = 255, la = 255; + uint32_t hr = 0, hg = 0, hb = 0, ha = 0; + for (uint32_t i = 0; i < num_pixels; i++) + { + const color_rgba* pC = &pPixels[i]; + if (pC->m_c[0] < lr) lr = pC->m_c[0]; + if (pC->m_c[1] < lg) lg = pC->m_c[1]; + if (pC->m_c[2] < lb) lb = pC->m_c[2]; + if (pC->m_c[3] < la) la = pC->m_c[3]; + + if (pC->m_c[0] > hr) hr = pC->m_c[0]; + if (pC->m_c[1] > hg) hg = pC->m_c[1]; + if (pC->m_c[2] > hb) hb = pC->m_c[2]; + if (pC->m_c[3] > ha) ha = pC->m_c[3]; + } + + color_rgba lowColor; color_quad_u8_set(&lowColor, lr, lg, lb, la); + color_rgba highColor; color_quad_u8_set(&highColor, hr, hg, hb, ha); + + // Place endpoints at bbox diagonals and compute interpolated colors + const uint32_t N = 4; + color_rgba weightedColors[4]; + + weightedColors[0] = lowColor; + weightedColors[N - 1] = highColor; + for (uint32_t i = 1; i < (N - 1); i++) + { + weightedColors[i].m_c[0] = (uint8_t)((lowColor.m_c[0] * (64 - g_bc7_weights2[i]) + highColor.m_c[0] * g_bc7_weights2[i] + 32) >> 6); + weightedColors[i].m_c[1] = (uint8_t)((lowColor.m_c[1] * (64 - g_bc7_weights2[i]) + highColor.m_c[1] * g_bc7_weights2[i] + 32) >> 6); + weightedColors[i].m_c[2] = (uint8_t)((lowColor.m_c[2] * (64 - g_bc7_weights2[i]) + highColor.m_c[2] * g_bc7_weights2[i] + 32) >> 6); + weightedColors[i].m_c[3] = (uint8_t)((lowColor.m_c[3] * (64 - g_bc7_weights2[i]) + highColor.m_c[3] * g_bc7_weights2[i] + 32) >> 6); + } + + // Compute dots and thresholds + const int ar = highColor.m_c[0] - lowColor.m_c[0]; + const int ag = highColor.m_c[1] - lowColor.m_c[1]; + const int ab = highColor.m_c[2] - lowColor.m_c[2]; + const int aa = highColor.m_c[3] - lowColor.m_c[3]; + + int dots[4]; + for (uint32_t i = 0; i < N; i++) + dots[i] = weightedColors[i].m_c[0] * ar + weightedColors[i].m_c[1] * ag + weightedColors[i].m_c[2] * ab + weightedColors[i].m_c[3] * aa; + + int thresh[4 - 1]; + for (uint32_t i = 0; i < (N - 1); i++) + thresh[i] = (dots[i] + dots[i + 1] + 1) >> 1; + + uint64_t total_err = 0; + if (perceptual) + { + // Transform block's interpolated colors to YCbCr + int l1[4], cr1[4], cb1[4]; + for (int j = 0; j < 4; j++) + { + const color_rgba* pE1 = &weightedColors[j]; + l1[j] = pE1->m_c[0] * 109 + pE1->m_c[1] * 366 + pE1->m_c[2] * 37; + cr1[j] = ((int)pE1->m_c[0] << 9) - l1[j]; + cb1[j] = ((int)pE1->m_c[2] << 9) - l1[j]; + } + + for (uint32_t i = 0; i < num_pixels; i++) + { + const color_rgba* pC = &pPixels[i]; + + int d = ar * pC->m_c[0] + ag * pC->m_c[1] + ab * pC->m_c[2] + aa * pC->m_c[3]; + + // Find approximate selector + uint32_t s = 0; + if (d >= thresh[2]) + s = 3; + else if (d >= thresh[1]) + s = 2; + else if (d >= thresh[0]) + s = 1; + + // Compute error + const int l2 = pC->m_c[0] * 109 + pC->m_c[1] * 366 + pC->m_c[2] * 37; + const int cr2 = ((int)pC->m_c[0] << 9) - l2; + const int cb2 = ((int)pC->m_c[2] << 9) - l2; + + const int dl = (l1[s] - l2) >> 8; + const int dcr = (cr1[s] - cr2) >> 8; + const int dcb = (cb1[s] - cb2) >> 8; + + const int dca = (int)pC->m_c[3] - (int)weightedColors[s].m_c[3]; + + int ie = (pweights[0] * dl * dl) + (pweights[1] * dcr * dcr) + (pweights[2] * dcb * dcb) + (pweights[3] * dca * dca); + + total_err += ie; + if (total_err > best_err_so_far) + break; + } + } + else + { + for (uint32_t i = 0; i < num_pixels; i++) + { + const color_rgba* pC = &pPixels[i]; + + int d = ar * pC->m_c[0] + ag * pC->m_c[1] + ab * pC->m_c[2] + aa * pC->m_c[3]; + + // Find approximate selector + uint32_t s = 0; + if (d >= thresh[2]) + s = 3; + else if (d >= thresh[1]) + s = 2; + else if (d >= thresh[0]) + s = 1; + + // Compute error + const color_rgba* pE1 = &weightedColors[s]; + + int dr = (int)pE1->m_c[0] - (int)pC->m_c[0]; + int dg = (int)pE1->m_c[1] - (int)pC->m_c[1]; + int db = (int)pE1->m_c[2] - (int)pC->m_c[2]; + int da = (int)pE1->m_c[3] - (int)pC->m_c[3]; + + total_err += pweights[0] * (dr * dr) + pweights[1] * (dg * dg) + pweights[2] * (db * db) + pweights[3] * (da * da); + if (total_err > best_err_so_far) + break; + } + } + + return total_err; +} + +// This table contains bitmasks indicating which "key" partitions must be best ranked before this partition is worth evaluating. +// We first rank the best/most used 14 partitions (sorted by usefulness), record the best one found as the key partition, then use +// that to control the other partitions to evaluate. The quality loss is ~.08 dB RGB PSNR, the perf gain is up to ~11% (at uber level 0). +static const uint32_t g_partition_predictors[35] = +{ + UINT32_MAX, + UINT32_MAX, + UINT32_MAX, + UINT32_MAX, + UINT32_MAX, + (1 << 1) | (1 << 2) | (1 << 8), + (1 << 1) | (1 << 3) | (1 << 7), + UINT32_MAX, + UINT32_MAX, + (1 << 2) | (1 << 8) | (1 << 16), + (1 << 7) | (1 << 3) | (1 << 15), + UINT32_MAX, + (1 << 8) | (1 << 14) | (1 << 16), + (1 << 7) | (1 << 14) | (1 << 15), + UINT32_MAX, + UINT32_MAX, + UINT32_MAX, + UINT32_MAX, + (1 << 14) | (1 << 15), + (1 << 16) | (1 << 22) | (1 << 14), + (1 << 17) | (1 << 24) | (1 << 14), + (1 << 2) | (1 << 14) | (1 << 15) | (1 << 1), + UINT32_MAX, + (1 << 1) | (1 << 3) | (1 << 14) | (1 << 16) | (1 << 22), + UINT32_MAX, + (1 << 1) | (1 << 2) | (1 << 15) | (1 << 17) | (1 << 24), + (1 << 1) | (1 << 3) | (1 << 22), + UINT32_MAX, + UINT32_MAX, + UINT32_MAX, + (1 << 14) | (1 << 15) | (1 << 16) | (1 << 17), + UINT32_MAX, + UINT32_MAX, + (1 << 1) | (1 << 2) | (1 << 3) | (1 << 27) | (1 << 4) | (1 << 24), + (1 << 14) | (1 << 15) | (1 << 16) | (1 << 11) | (1 << 17) | (1 << 27) +}; + +// Estimate the partition used by modes 1/7. This scans through each partition and computes an approximate error for each. +static uint32_t estimate_partition(const color_rgba *pPixels, const bc7enc_compress_block_params *pComp_params, uint32_t pweights[4], uint32_t mode) +{ + const uint32_t total_partitions = minimumu(pComp_params->m_max_partitions, BC7ENC_MAX_PARTITIONS); + if (total_partitions <= 1) + return 0; + + uint64_t best_err = UINT64_MAX; + uint32_t best_partition = 0; + + // Partition order sorted by usage frequency across a large test corpus. Pattern 34 (checkerboard) must appear in slot 34. + // Using a sorted order allows the user to decrease the # of partitions to scan with minimal loss in quality. + static const uint8_t s_sorted_partition_order[64] = + { + 1 - 1, 14 - 1, 2 - 1, 3 - 1, 16 - 1, 15 - 1, 11 - 1, 17 - 1, + 4 - 1, 24 - 1, 27 - 1, 7 - 1, 8 - 1, 22 - 1, 20 - 1, 30 - 1, + 9 - 1, 5 - 1, 10 - 1, 21 - 1, 6 - 1, 32 - 1, 23 - 1, 18 - 1, + 19 - 1, 12 - 1, 13 - 1, 31 - 1, 25 - 1, 26 - 1, 29 - 1, 28 - 1, + 33 - 1, 34 - 1, 35 - 1, 46 - 1, 47 - 1, 52 - 1, 50 - 1, 51 - 1, + 49 - 1, 39 - 1, 40 - 1, 38 - 1, 54 - 1, 53 - 1, 55 - 1, 37 - 1, + 58 - 1, 59 - 1, 56 - 1, 42 - 1, 41 - 1, 43 - 1, 44 - 1, 60 - 1, + 45 - 1, 57 - 1, 48 - 1, 36 - 1, 61 - 1, 64 - 1, 63 - 1, 62 - 1 + }; + + assert(s_sorted_partition_order[34] == 34); + + int best_key_partition = 0; + + for (uint32_t partition_iter = 0; (partition_iter < total_partitions) && (best_err > 0); partition_iter++) + { + const uint32_t partition = s_sorted_partition_order[partition_iter]; + + // Check to see if we should bother evaluating this partition at all, depending on the best partition found from the first 14. + if (pComp_params->m_mode17_partition_estimation_filterbank) + { + if ((partition_iter >= 14) && (partition_iter <= 34)) + { + const uint32_t best_key_partition_bitmask = 1 << (best_key_partition + 1); + if ((g_partition_predictors[partition] & best_key_partition_bitmask) == 0) + { + if (partition_iter == 34) + break; + + continue; + } + } + } + + const uint8_t *pPartition = &g_bc7_partition2[partition * 16]; + + color_rgba subset_colors[2][16]; + uint32_t subset_total_colors[2] = { 0, 0 }; + for (uint32_t index = 0; index < 16; index++) + subset_colors[pPartition[index]][subset_total_colors[pPartition[index]]++] = pPixels[index]; + + uint64_t total_subset_err = 0; + for (uint32_t subset = 0; (subset < 2) && (total_subset_err < best_err); subset++) + { + if (mode == 7) + total_subset_err += color_cell_compression_est_mode7(subset_total_colors[subset], &subset_colors[subset][0], pComp_params->m_perceptual, pweights, best_err); + else + total_subset_err += color_cell_compression_est_mode1(subset_total_colors[subset], &subset_colors[subset][0], pComp_params->m_perceptual, pweights, best_err); + } + + if (partition < 16) + { + total_subset_err = (uint64_t)((double)total_subset_err * pComp_params->m_low_frequency_partition_weight + .5f); + } + + if (total_subset_err < best_err) + { + best_err = total_subset_err; + best_partition = partition; + } + + // If the checkerboard pattern doesn't get the highest ranking vs. the previous (lower frequency) patterns, then just stop now because statistically the subsequent patterns won't do well either. + if ((partition == 34) && (best_partition != 34)) + break; + + if (partition_iter == 13) + best_key_partition = best_partition; + + } // partition + + return best_partition; +} + +static void set_block_bits(uint8_t *pBytes, uint32_t val, uint32_t num_bits, uint32_t *pCur_ofs) +{ + assert((num_bits <= 32) && (val < (1ULL << num_bits))); + while (num_bits) + { + const uint32_t n = minimumu(8 - (*pCur_ofs & 7), num_bits); + pBytes[*pCur_ofs >> 3] |= (uint8_t)(val << (*pCur_ofs & 7)); + val >>= n; + num_bits -= n; + *pCur_ofs += n; + } + assert(*pCur_ofs <= 128); +} + +struct bc7_optimization_results +{ + uint32_t m_mode; + uint32_t m_partition; + uint8_t m_selectors[16]; + uint8_t m_alpha_selectors[16]; + color_rgba m_low[3]; + color_rgba m_high[3]; + uint32_t m_pbits[3][2]; + uint32_t m_rotation; + uint32_t m_index_selector; +}; + +void encode_bc7_block(void* pBlock, const bc7_optimization_results* pResults) +{ + assert(pResults->m_index_selector <= 1); + assert(pResults->m_rotation <= 3); + + const uint32_t best_mode = pResults->m_mode; + + const uint32_t total_subsets = g_bc7_num_subsets[best_mode]; + const uint32_t total_partitions = 1 << g_bc7_partition_bits[best_mode]; + //const uint32_t num_rotations = 1 << g_bc7_rotation_bits[best_mode]; + //const uint32_t num_index_selectors = (best_mode == 4) ? 2 : 1; + + const uint8_t* pPartition; + if (total_subsets == 1) + pPartition = &g_bc7_partition1[0]; + else if (total_subsets == 2) + pPartition = &g_bc7_partition2[pResults->m_partition * 16]; + else + pPartition = &g_bc7_partition3[pResults->m_partition * 16]; + + uint8_t color_selectors[16]; + memcpy(color_selectors, pResults->m_selectors, 16); + + uint8_t alpha_selectors[16]; + memcpy(alpha_selectors, pResults->m_alpha_selectors, 16); + + color_rgba low[3], high[3]; + memcpy(low, pResults->m_low, sizeof(low)); + memcpy(high, pResults->m_high, sizeof(high)); + + uint32_t pbits[3][2]; + memcpy(pbits, pResults->m_pbits, sizeof(pbits)); + + int anchor[3] = { -1, -1, -1 }; + + for (uint32_t k = 0; k < total_subsets; k++) + { + uint32_t anchor_index = 0; + if (k) + { + if ((total_subsets == 3) && (k == 1)) + anchor_index = g_bc7_table_anchor_index_third_subset_1[pResults->m_partition]; + else if ((total_subsets == 3) && (k == 2)) + anchor_index = g_bc7_table_anchor_index_third_subset_2[pResults->m_partition]; + else + anchor_index = g_bc7_table_anchor_index_second_subset[pResults->m_partition]; + } + + anchor[k] = anchor_index; + + const uint32_t color_index_bits = get_bc7_color_index_size(best_mode, pResults->m_index_selector); + const uint32_t num_color_indices = 1 << color_index_bits; + + if (color_selectors[anchor_index] & (num_color_indices >> 1)) + { + for (uint32_t i = 0; i < 16; i++) + if (pPartition[i] == k) + color_selectors[i] = (uint8_t)((num_color_indices - 1) - color_selectors[i]); + + if (get_bc7_mode_has_seperate_alpha_selectors(best_mode)) + { + for (uint32_t q = 0; q < 3; q++) + { + uint8_t t = low[k].m_c[q]; + low[k].m_c[q] = high[k].m_c[q]; + high[k].m_c[q] = t; + } + } + else + { + color_rgba tmp = low[k]; + low[k] = high[k]; + high[k] = tmp; + } + + if (!g_bc7_mode_has_shared_p_bits[best_mode]) + { + uint32_t t = pbits[k][0]; + pbits[k][0] = pbits[k][1]; + pbits[k][1] = t; + } + } + + if (get_bc7_mode_has_seperate_alpha_selectors(best_mode)) + { + const uint32_t alpha_index_bits = get_bc7_alpha_index_size(best_mode, pResults->m_index_selector); + const uint32_t num_alpha_indices = 1 << alpha_index_bits; + + if (alpha_selectors[anchor_index] & (num_alpha_indices >> 1)) + { + for (uint32_t i = 0; i < 16; i++) + if (pPartition[i] == k) + alpha_selectors[i] = (uint8_t)((num_alpha_indices - 1) - alpha_selectors[i]); + + uint8_t t = low[k].m_c[3]; + low[k].m_c[3] = high[k].m_c[3]; + high[k].m_c[3] = t; + } + } + } + + uint8_t* pBlock_bytes = (uint8_t*)(pBlock); + memset(pBlock_bytes, 0, BC7ENC_BLOCK_SIZE); + + uint32_t cur_bit_ofs = 0; + set_block_bits(pBlock_bytes, 1 << best_mode, best_mode + 1, &cur_bit_ofs); + + if ((best_mode == 4) || (best_mode == 5)) + set_block_bits(pBlock_bytes, pResults->m_rotation, 2, &cur_bit_ofs); + + if (best_mode == 4) + set_block_bits(pBlock_bytes, pResults->m_index_selector, 1, &cur_bit_ofs); + + if (total_partitions > 1) + set_block_bits(pBlock_bytes, pResults->m_partition, (total_partitions == 64) ? 6 : 4, &cur_bit_ofs); + + const uint32_t total_comps = (best_mode >= 4) ? 4 : 3; + for (uint32_t comp = 0; comp < total_comps; comp++) + { + for (uint32_t subset = 0; subset < total_subsets; subset++) + { + set_block_bits(pBlock_bytes, low[subset].m_c[comp], (comp == 3) ? g_bc7_alpha_precision_table[best_mode] : g_bc7_color_precision_table[best_mode], &cur_bit_ofs); + set_block_bits(pBlock_bytes, high[subset].m_c[comp], (comp == 3) ? g_bc7_alpha_precision_table[best_mode] : g_bc7_color_precision_table[best_mode], &cur_bit_ofs); + } + } + + if (g_bc7_mode_has_p_bits[best_mode]) + { + for (uint32_t subset = 0; subset < total_subsets; subset++) + { + set_block_bits(pBlock_bytes, pbits[subset][0], 1, &cur_bit_ofs); + if (!g_bc7_mode_has_shared_p_bits[best_mode]) + set_block_bits(pBlock_bytes, pbits[subset][1], 1, &cur_bit_ofs); + } + } + + for (uint32_t y = 0; y < 4; y++) + { + for (uint32_t x = 0; x < 4; x++) + { + int idx = x + y * 4; + + uint32_t n = pResults->m_index_selector ? get_bc7_alpha_index_size(best_mode, pResults->m_index_selector) : get_bc7_color_index_size(best_mode, pResults->m_index_selector); + + if ((idx == anchor[0]) || (idx == anchor[1]) || (idx == anchor[2])) + n--; + + set_block_bits(pBlock_bytes, pResults->m_index_selector ? alpha_selectors[idx] : color_selectors[idx], n, &cur_bit_ofs); + } + } + + if (get_bc7_mode_has_seperate_alpha_selectors(best_mode)) + { + for (uint32_t y = 0; y < 4; y++) + { + for (uint32_t x = 0; x < 4; x++) + { + int idx = x + y * 4; + + uint32_t n = pResults->m_index_selector ? get_bc7_color_index_size(best_mode, pResults->m_index_selector) : get_bc7_alpha_index_size(best_mode, pResults->m_index_selector); + + if ((idx == anchor[0]) || (idx == anchor[1]) || (idx == anchor[2])) + n--; + + set_block_bits(pBlock_bytes, pResults->m_index_selector ? color_selectors[idx] : alpha_selectors[idx], n, &cur_bit_ofs); + } + } + } + + assert(cur_bit_ofs == 128); +} + +static void handle_alpha_block_mode5(const color_rgba* pPixels, const bc7enc_compress_block_params* pComp_params, color_cell_compressor_params* pParams, uint32_t lo_a, uint32_t hi_a, bc7_optimization_results* pOpt_results5, uint64_t* pMode5_err, uint64_t* pMode5_alpha_err) +{ + pParams->m_pSelector_weights = g_bc7_weights2; + pParams->m_pSelector_weightsx = (const vec4F*)g_bc7_weights2x; + pParams->m_num_selector_weights = 4; + + pParams->m_comp_bits = 7; + pParams->m_has_pbits = false; + pParams->m_endpoints_share_pbit = false; + pParams->m_has_alpha = false; + + pParams->m_perceptual = pComp_params->m_perceptual; + + pParams->m_num_pixels = 16; + pParams->m_pPixels = pPixels; + + color_cell_compressor_results results5; + results5.m_pSelectors = pOpt_results5->m_selectors; + + uint8_t selectors_temp[16]; + results5.m_pSelectors_temp = selectors_temp; + + *pMode5_err = color_cell_compression(5, pParams, &results5, pComp_params); + assert(*pMode5_err == results5.m_best_overall_err); + + pOpt_results5->m_low[0] = results5.m_low_endpoint; + pOpt_results5->m_high[0] = results5.m_high_endpoint; + + if (lo_a == hi_a) + { + *pMode5_alpha_err = 0; + pOpt_results5->m_low[0].m_c[3] = (uint8_t)lo_a; + pOpt_results5->m_high[0].m_c[3] = (uint8_t)hi_a; + memset(pOpt_results5->m_alpha_selectors, 0, sizeof(pOpt_results5->m_alpha_selectors)); + } + else + { + *pMode5_alpha_err = UINT64_MAX; + + const uint32_t total_passes = (pComp_params->m_uber_level >= 1) ? 3 : 2; + for (uint32_t pass = 0; pass < total_passes; pass++) + { + int32_t vals[4]; + vals[0] = lo_a; + vals[3] = hi_a; + + const int32_t w_s1 = 21, w_s2 = 43; + vals[1] = (vals[0] * (64 - w_s1) + vals[3] * w_s1 + 32) >> 6; + vals[2] = (vals[0] * (64 - w_s2) + vals[3] * w_s2 + 32) >> 6; + + uint8_t trial_alpha_selectors[16]; + + uint64_t trial_alpha_err = 0; + for (uint32_t i = 0; i < 16; i++) + { + const int32_t a = pParams->m_pPixels[i].m_c[3]; + + int s = 0; + int32_t be = iabs32(a - vals[0]); + int e = iabs32(a - vals[1]); if (e < be) { be = e; s = 1; } + e = iabs32(a - vals[2]); if (e < be) { be = e; s = 2; } + e = iabs32(a - vals[3]); if (e < be) { be = e; s = 3; } + + trial_alpha_selectors[i] = (uint8_t)s; + + uint32_t a_err = (uint32_t)(be * be) * pParams->m_weights[3]; + + trial_alpha_err += a_err; + } + + if (trial_alpha_err < *pMode5_alpha_err) + { + *pMode5_alpha_err = trial_alpha_err; + pOpt_results5->m_low[0].m_c[3] = (uint8_t)lo_a; + pOpt_results5->m_high[0].m_c[3] = (uint8_t)hi_a; + memcpy(pOpt_results5->m_alpha_selectors, trial_alpha_selectors, sizeof(pOpt_results5->m_alpha_selectors)); + } + + if (pass != (total_passes - 1U)) + { + float xl, xh; + compute_least_squares_endpoints_a(16, trial_alpha_selectors, (const vec4F*)g_bc7_weights2x, &xl, &xh, pParams->m_pPixels); + + uint32_t new_lo_a = clampi((int)floor(xl + .5f), 0, 255); + uint32_t new_hi_a = clampi((int)floor(xh + .5f), 0, 255); + if (new_lo_a > new_hi_a) + swapu(&new_lo_a, &new_hi_a); + + if ((new_lo_a == lo_a) && (new_hi_a == hi_a)) + break; + + lo_a = new_lo_a; + hi_a = new_hi_a; + } + } + + *pMode5_err += *pMode5_alpha_err; + } +} + +static void handle_alpha_block(void *pBlock, const color_rgba *pPixels, const bc7enc_compress_block_params *pComp_params, color_cell_compressor_params *pParams) +{ + assert((pComp_params->m_mode_mask & (1 << 6)) || (pComp_params->m_mode_mask & (1 << 5)) || (pComp_params->m_mode_mask & (1 << 7))); + + pParams->m_pSelector_weights = g_bc7_weights4; + pParams->m_pSelector_weightsx = (const vec4F *)g_bc7_weights4x; + pParams->m_num_selector_weights = 16; + pParams->m_comp_bits = 7; + pParams->m_has_pbits = true; + pParams->m_endpoints_share_pbit = false; + pParams->m_has_alpha = true; + pParams->m_perceptual = pComp_params->m_perceptual; + pParams->m_num_pixels = 16; + pParams->m_pPixels = pPixels; + + bc7_optimization_results opt_results6, opt_results5, opt_results7; + color_cell_compressor_results results6; + memset(&results6, 0, sizeof(results6)); + + uint64_t best_err = UINT64_MAX; + uint32_t best_mode = 0; + uint8_t selectors_temp[16]; + + if (pComp_params->m_mode_mask & (1 << 6)) + { + results6.m_pSelectors = opt_results6.m_selectors; + results6.m_pSelectors_temp = selectors_temp; + + best_err = (uint64_t)(color_cell_compression(6, pParams, &results6, pComp_params) * pComp_params->m_mode6_error_weight + .5f); + best_mode = 6; + } + + if ((best_err > 0) && (pComp_params->m_mode_mask & (1 << 5))) + { + uint32_t lo_a = 255, hi_a = 0; + for (uint32_t i = 0; i < 16; i++) + { + uint32_t a = pPixels[i].m_c[3]; + lo_a = minimumu(lo_a, a); + hi_a = maximumu(hi_a, a); + } + + uint64_t mode5_err, mode5_alpha_err; + handle_alpha_block_mode5(pPixels, pComp_params, pParams, lo_a, hi_a, &opt_results5, &mode5_err, &mode5_alpha_err); + + mode5_err = (uint64_t)(mode5_err * pComp_params->m_mode5_error_weight + .5f); + + if (mode5_err < best_err) + { + best_err = mode5_err; + best_mode = 5; + } + } + + if ((best_err > 0) && (pComp_params->m_mode_mask & (1 << 7))) + { + const uint32_t trial_partition = estimate_partition(pPixels, pComp_params, pParams->m_weights, 7); + + pParams->m_pSelector_weights = g_bc7_weights2; + pParams->m_pSelector_weightsx = (const vec4F*)g_bc7_weights2x; + pParams->m_num_selector_weights = 4; + pParams->m_comp_bits = 5; + pParams->m_has_pbits = true; + pParams->m_endpoints_share_pbit = false; + pParams->m_has_alpha = true; + + const uint8_t* pPartition = &g_bc7_partition2[trial_partition * 16]; + + color_rgba subset_colors[2][16]; + + uint32_t subset_total_colors7[2] = { 0, 0 }; + + uint8_t subset_pixel_index7[2][16]; + uint8_t subset_selectors7[2][16]; + color_cell_compressor_results subset_results7[2]; + + for (uint32_t idx = 0; idx < 16; idx++) + { + const uint32_t p = pPartition[idx]; + subset_colors[p][subset_total_colors7[p]] = pPixels[idx]; + subset_pixel_index7[p][subset_total_colors7[p]] = (uint8_t)idx; + subset_total_colors7[p]++; + } + + uint64_t trial_err = 0; + for (uint32_t subset = 0; subset < 2; subset++) + { + pParams->m_num_pixels = subset_total_colors7[subset]; + pParams->m_pPixels = &subset_colors[subset][0]; + + color_cell_compressor_results* pResults = &subset_results7[subset]; + pResults->m_pSelectors = &subset_selectors7[subset][0]; + pResults->m_pSelectors_temp = selectors_temp; + uint64_t err = color_cell_compression(7, pParams, pResults, pComp_params); + trial_err += err; + if ((uint64_t)(trial_err * pComp_params->m_mode7_error_weight + .5f) > best_err) + break; + + } // subset + + const uint64_t mode7_trial_err = (uint64_t)(trial_err * pComp_params->m_mode7_error_weight + .5f); + + if (mode7_trial_err < best_err) + { + best_err = mode7_trial_err; + best_mode = 7; + opt_results7.m_mode = 7; + opt_results7.m_partition = trial_partition; + opt_results7.m_index_selector = 0; + opt_results7.m_rotation = 0; + for (uint32_t subset = 0; subset < 2; subset++) + { + for (uint32_t i = 0; i < subset_total_colors7[subset]; i++) + opt_results7.m_selectors[subset_pixel_index7[subset][i]] = subset_selectors7[subset][i]; + opt_results7.m_low[subset] = subset_results7[subset].m_low_endpoint; + opt_results7.m_high[subset] = subset_results7[subset].m_high_endpoint; + opt_results7.m_pbits[subset][0] = subset_results7[subset].m_pbits[0]; + opt_results7.m_pbits[subset][1] = subset_results7[subset].m_pbits[1]; + } + } + } + + if (best_mode == 7) + { + encode_bc7_block(pBlock, &opt_results7); + } + else if (best_mode == 5) + { + opt_results5.m_mode = 5; + opt_results5.m_partition = 0; + opt_results5.m_rotation = 0; + opt_results5.m_index_selector = 0; + + encode_bc7_block(pBlock, &opt_results5); + } + else if (best_mode == 6) + { + opt_results6.m_mode = 6; + opt_results6.m_partition = 0; + opt_results6.m_low[0] = results6.m_low_endpoint; + opt_results6.m_high[0] = results6.m_high_endpoint; + opt_results6.m_pbits[0][0] = results6.m_pbits[0]; + opt_results6.m_pbits[0][1] = results6.m_pbits[1]; + opt_results6.m_rotation = 0; + opt_results6.m_index_selector = 0; + + encode_bc7_block(pBlock, &opt_results6); + } + else + { + assert(0); + } +} + +static void handle_opaque_block(void *pBlock, const color_rgba *pPixels, const bc7enc_compress_block_params *pComp_params, color_cell_compressor_params *pParams) +{ + assert((pComp_params->m_mode_mask & (1 << 6)) || (pComp_params->m_mode_mask & (1 << 1))); + + uint8_t selectors_temp[16]; + + bc7_optimization_results opt_results; + + uint64_t best_err = UINT64_MAX; + + pParams->m_perceptual = pComp_params->m_perceptual; + pParams->m_num_pixels = 16; + pParams->m_pPixels = pPixels; + pParams->m_has_alpha = false; + + opt_results.m_partition = 0; + opt_results.m_index_selector = 0; + opt_results.m_rotation = 0; + + // Mode 6 + if (pComp_params->m_mode_mask & (1 << 6)) + { + pParams->m_pSelector_weights = g_bc7_weights4; + pParams->m_pSelector_weightsx = (const vec4F*)g_bc7_weights4x; + pParams->m_num_selector_weights = 16; + pParams->m_comp_bits = 7; + pParams->m_has_pbits = true; + pParams->m_endpoints_share_pbit = false; + + color_cell_compressor_results results6; + results6.m_pSelectors = opt_results.m_selectors; + results6.m_pSelectors_temp = selectors_temp; + + best_err = (uint64_t)(color_cell_compression(6, pParams, &results6, pComp_params) * pComp_params->m_mode6_error_weight + .5f); + + opt_results.m_mode = 6; + opt_results.m_low[0] = results6.m_low_endpoint; + opt_results.m_high[0] = results6.m_high_endpoint; + opt_results.m_pbits[0][0] = results6.m_pbits[0]; + opt_results.m_pbits[0][1] = results6.m_pbits[1]; + } + + // Mode 1 + if ((best_err > 0) && (pComp_params->m_max_partitions > 0) && (pComp_params->m_mode_mask & (1 << 1))) + { + const uint32_t trial_partition = estimate_partition(pPixels, pComp_params, pParams->m_weights, 1); + + pParams->m_pSelector_weights = g_bc7_weights3; + pParams->m_pSelector_weightsx = (const vec4F *)g_bc7_weights3x; + pParams->m_num_selector_weights = 8; + pParams->m_comp_bits = 6; + pParams->m_has_pbits = true; + pParams->m_endpoints_share_pbit = true; + + const uint8_t *pPartition = &g_bc7_partition2[trial_partition * 16]; + + color_rgba subset_colors[2][16]; + + uint32_t subset_total_colors1[2] = { 0, 0 }; + + uint8_t subset_pixel_index1[2][16]; + uint8_t subset_selectors1[2][16]; + color_cell_compressor_results subset_results1[2]; + + for (uint32_t idx = 0; idx < 16; idx++) + { + const uint32_t p = pPartition[idx]; + subset_colors[p][subset_total_colors1[p]] = pPixels[idx]; + subset_pixel_index1[p][subset_total_colors1[p]] = (uint8_t)idx; + subset_total_colors1[p]++; + } + + uint64_t trial_err = 0; + for (uint32_t subset = 0; subset < 2; subset++) + { + pParams->m_num_pixels = subset_total_colors1[subset]; + pParams->m_pPixels = &subset_colors[subset][0]; + + color_cell_compressor_results *pResults = &subset_results1[subset]; + pResults->m_pSelectors = &subset_selectors1[subset][0]; + pResults->m_pSelectors_temp = selectors_temp; + uint64_t err = color_cell_compression(1, pParams, pResults, pComp_params); + + trial_err += err; + if ((uint64_t)(trial_err * pComp_params->m_mode1_error_weight + .5f) > best_err) + break; + + } // subset + + const uint64_t mode1_trial_err = (uint64_t)(trial_err * pComp_params->m_mode1_error_weight + .5f); + if (mode1_trial_err < best_err) + { + best_err = mode1_trial_err; + opt_results.m_mode = 1; + opt_results.m_partition = trial_partition; + for (uint32_t subset = 0; subset < 2; subset++) + { + for (uint32_t i = 0; i < subset_total_colors1[subset]; i++) + opt_results.m_selectors[subset_pixel_index1[subset][i]] = subset_selectors1[subset][i]; + opt_results.m_low[subset] = subset_results1[subset].m_low_endpoint; + opt_results.m_high[subset] = subset_results1[subset].m_high_endpoint; + opt_results.m_pbits[subset][0] = subset_results1[subset].m_pbits[0]; + } + } + } + + encode_bc7_block(pBlock, &opt_results); +} + +bool bc7enc_compress_block(void *pBlock, const void *pPixelsRGBA, const bc7enc_compress_block_params *pComp_params) +{ + assert(g_bc7_mode_1_optimal_endpoints[255][0].m_hi != 0); + + const color_rgba *pPixels = (const color_rgba *)(pPixelsRGBA); + + color_cell_compressor_params params; + if (pComp_params->m_perceptual) + { + // https://en.wikipedia.org/wiki/YCbCr#ITU-R_BT.709_conversion + const float pr_weight = (.5f / (1.0f - .2126f)) * (.5f / (1.0f - .2126f)); + const float pb_weight = (.5f / (1.0f - .0722f)) * (.5f / (1.0f - .0722f)); + params.m_weights[0] = (int)(pComp_params->m_weights[0] * 4.0f); + params.m_weights[1] = (int)(pComp_params->m_weights[1] * 4.0f * pr_weight); + params.m_weights[2] = (int)(pComp_params->m_weights[2] * 4.0f * pb_weight); + params.m_weights[3] = pComp_params->m_weights[3] * 4; + } + else + memcpy(params.m_weights, pComp_params->m_weights, sizeof(params.m_weights)); + + if (pComp_params->m_force_alpha) + { + handle_alpha_block(pBlock, pPixels, pComp_params, ¶ms); + return true; + } + + for (uint32_t i = 0; i < 16; i++) + { + if (pPixels[i].m_c[3] < 255) + { + handle_alpha_block(pBlock, pPixels, pComp_params, ¶ms); + return true; + } + } + handle_opaque_block(pBlock, pPixels, pComp_params, ¶ms); + return false; +} + +static const uint8_t g_tdefl_small_dist_extra[512] = +{ + 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 5, + 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, + 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, + 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, + 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, + 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, + 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, + 7, 7, 7, 7, 7, 7, 7, 7 +}; + +static const uint8_t g_tdefl_large_dist_extra[128] = +{ + 0, 0, 8, 8, 9, 9, 9, 9, 10, 10, 10, 10, 10, 10, 10, 10, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, + 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, + 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13 +}; + +static inline uint32_t compute_match_cost_estimate(uint32_t dist, uint32_t match_len_in_bytes) +{ + assert(match_len_in_bytes <= 258); + + uint32_t len_cost = 6; + if (match_len_in_bytes >= 12) + len_cost = 9; + else if (match_len_in_bytes >= 8) + len_cost = 8; + else if (match_len_in_bytes >= 6) + len_cost = 7; + + uint32_t dist_cost = 5; + if (dist < 512) + dist_cost += g_tdefl_small_dist_extra[dist & 511]; + else + { + dist_cost += g_tdefl_large_dist_extra[std::min(dist, 32767) >> 8]; + while (dist >= 32768) + { + dist_cost++; + dist >>= 1; + } + } + return len_cost + dist_cost; +} + +class tracked_stat +{ +public: + tracked_stat() { clear(); } + + void clear() { m_num = 0; m_total = 0; m_total2 = 0; } + + void update(uint32_t val) { m_num++; m_total += val; m_total2 += val * val; } + + tracked_stat& operator += (uint32_t val) { update(val); return *this; } + + uint32_t get_number_of_values() { return m_num; } + uint64_t get_total() const { return m_total; } + uint64_t get_total2() const { return m_total2; } + + float get_average() const { return m_num ? (float)m_total / m_num : 0.0f; }; + float get_std_dev() const { return m_num ? sqrtf((float)(m_num * m_total2 - m_total * m_total)) / m_num : 0.0f; } + float get_variance() const { float s = get_std_dev(); return s * s; } + +private: + uint32_t m_num; + uint64_t m_total; + uint64_t m_total2; +}; + +static inline float compute_block_max_std_dev(const color_rgba* pPixels) +{ + tracked_stat r_stats, g_stats, b_stats, a_stats; + + for (uint32_t i = 0; i < 16; i++) + { + r_stats.update(pPixels[i].m_c[0]); + g_stats.update(pPixels[i].m_c[1]); + b_stats.update(pPixels[i].m_c[2]); + a_stats.update(pPixels[i].m_c[3]); + } + + return std::max(std::max(std::max(r_stats.get_std_dev(), g_stats.get_std_dev()), b_stats.get_std_dev()), a_stats.get_std_dev()); +} + +struct bc7_block +{ + uint8_t m_bytes[16]; + + uint32_t get_mode() const + { + uint32_t bc7_mode = 0; + while (((m_bytes[0] & (1 << bc7_mode)) == 0) && (bc7_mode < 8)) + bc7_mode++; + return bc7_mode; + } +}; + +/* +------------------------------------------------------------------------------ +This software is available under 2 licenses -- choose whichever you prefer. +If you use this software in a product, attribution / credits is requested but not required. +------------------------------------------------------------------------------ +ALTERNATIVE A - MIT License +Copyright(c) 2020-2021 Richard Geldreich, Jr. +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files(the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and / or sell copies +of the Software, and to permit persons to whom the Software is furnished to do +so, subject to the following conditions : +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. +------------------------------------------------------------------------------ +ALTERNATIVE B - Public Domain(www.unlicense.org) +This is free and unencumbered software released into the public domain. +Anyone is free to copy, modify, publish, use, compile, sell, or distribute this +software, either in source code form or as a compiled binary, for any purpose, +commercial or non - commercial, and by any means. +In jurisdictions that recognize copyright laws, the author or authors of this +software dedicate any and all copyright interest in the software to the public +domain.We make this dedication for the benefit of the public at large and to +the detriment of our heirs and successors.We intend this dedication to be an +overt act of relinquishment in perpetuity of all present and future rights to +this software under copyright law. +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE +AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN +ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +------------------------------------------------------------------------------ +*/ diff --git a/src/gfx/tex/bc7enc/bc7enc.h b/src/gfx/tex/bc7enc/bc7enc.h new file mode 100644 index 0000000..e5e31fa --- /dev/null +++ b/src/gfx/tex/bc7enc/bc7enc.h @@ -0,0 +1,123 @@ +// File: bc7enc.h - Richard Geldreich, Jr. - MIT license or public domain (see end of bc7enc.c) +// If you use this software in a product, attribution / credits is requested but not required. +#include +#include +#include +#include + +#define BC7ENC_BLOCK_SIZE (16) +#define BC7ENC_MAX_PARTITIONS (64) +#define BC7ENC_MAX_UBER_LEVEL (4) + +struct color_rgba { uint8_t m_c[4]; }; + +struct bc7enc_compress_block_params +{ + uint32_t m_mode_mask; + + // m_max_partitions may range from 0 (disables mode 1) to BC7ENC_MAX_PARTITIONS. The higher this value, the slower the compressor, but the higher the quality. + uint32_t m_max_partitions; + + // Relative RGBA or YCbCrA weights. + uint32_t m_weights[4]; + + // m_uber_level may range from 0 to BC7ENC_MAX_UBER_LEVEL. The higher this value, the slower the compressor, but the higher the quality. + uint32_t m_uber_level; + + // If m_perceptual is true, colorspace error is computed in YCbCr space, otherwise RGB. + bool m_perceptual; + + // Set m_try_least_squares to false for slightly faster/lower quality compression. + bool m_try_least_squares; + + // When m_mode17_partition_estimation_filterbank, the mode1 partition estimator skips lesser used partition patterns unless they are strongly predicted to be potentially useful. + // There's a slight loss in quality with this enabled (around .08 dB RGB PSNR or .05 dB Y PSNR), but up to a 11% gain in speed depending on the other settings. + bool m_mode17_partition_estimation_filterbank; + + bool m_force_alpha; + + bool m_force_selectors; + uint8_t m_selectors[16]; + + bool m_quant_mode6_endpoints; + bool m_bias_mode1_pbits; + + float m_pbit1_weight; + + float m_mode1_error_weight; + float m_mode5_error_weight; + float m_mode6_error_weight; + float m_mode7_error_weight; + + float m_low_frequency_partition_weight; + + void clear() + { + memset(this, 0, sizeof(*this)); + } + + void print() + { + printf("Mode mask: 0x%X\n", m_mode_mask); + printf("Max partitions: %u\n", m_max_partitions); + printf("Weights: %u %u %u %u\n", m_weights[0], m_weights[1], m_weights[2], m_weights[3]); + printf("Uber level: %u\n", m_uber_level); + printf("Perceptual: %u\n", m_perceptual); + printf("Try least squares: %u\n", m_try_least_squares); + printf("Mode 1/7 partition estimation filterbank: %u\n", m_mode17_partition_estimation_filterbank); + printf("Force alpha: %u\n", m_force_alpha); + printf("Quant mode 6 endpoints: %u\n", m_quant_mode6_endpoints); + printf("Bias mode 1 p-bits: %u\n", m_bias_mode1_pbits); + printf("p-bit 1 weight: %f\n", m_pbit1_weight); + printf("Mode error weights: %f %f %f %f\n", m_mode1_error_weight, m_mode5_error_weight, m_mode6_error_weight, m_mode7_error_weight); + printf("Low frequency partition weight: %f\n", m_low_frequency_partition_weight); + } +}; + +inline void bc7enc_compress_block_params_init_linear_weights(bc7enc_compress_block_params *p) +{ + p->m_perceptual = false; + p->m_weights[0] = 1; + p->m_weights[1] = 1; + p->m_weights[2] = 1; + p->m_weights[3] = 1; +} + +inline void bc7enc_compress_block_params_init_perceptual_weights(bc7enc_compress_block_params *p) +{ + p->m_perceptual = true; + p->m_weights[0] = 128; + p->m_weights[1] = 64; + p->m_weights[2] = 16; + p->m_weights[3] = 32; +} + +inline void bc7enc_compress_block_params_init(bc7enc_compress_block_params *p) +{ + p->m_mode_mask = UINT32_MAX; + p->m_max_partitions = BC7ENC_MAX_PARTITIONS; + p->m_try_least_squares = true; + p->m_mode17_partition_estimation_filterbank = true; + p->m_uber_level = 0; + p->m_force_selectors = false; + p->m_force_alpha = false; + p->m_quant_mode6_endpoints = false; + p->m_bias_mode1_pbits = false; + p->m_pbit1_weight = 1.0f; + p->m_mode1_error_weight = 1.0f; + p->m_mode5_error_weight = 1.0f; + p->m_mode6_error_weight = 1.0f; + p->m_mode7_error_weight = 1.0f; + p->m_low_frequency_partition_weight = 1.0f; + bc7enc_compress_block_params_init_perceptual_weights(p); +} + +// bc7enc_compress_block_init() MUST be called before calling bc7enc_compress_block() (or you'll get artifacts). +void bc7enc_compress_block_init(); + +// Packs a single block of 16x16 RGBA pixels (R first in memory) to 128-bit BC7 block pBlock, using either mode 1 and/or 6. +// Alpha blocks will always use mode 6, and by default opaque blocks will use either modes 1 or 6. +// Returns true if the block had any pixels with alpha < 255, otherwise it return false. (This is not an error code - a block is always encoded.) +bool bc7enc_compress_block(void *pBlock, const void *pPixelsRGBA, const bc7enc_compress_block_params *pComp_params); + + diff --git a/src/gfx/tex/bc7enc/rgbcx.cpp b/src/gfx/tex/bc7enc/rgbcx.cpp new file mode 100644 index 0000000..1d62109 --- /dev/null +++ b/src/gfx/tex/bc7enc/rgbcx.cpp @@ -0,0 +1,3083 @@ +// rgbcx.cpp - see license at end of rgbcx.h +#include "rgbcx.h" +#include +#include +#include + +namespace rgbcx +{ + const uint8_t g_bc1_to_linear[4] = { 0, 3, 1, 2 }; + + const uint32_t NUM_UNIQUE_TOTAL_ORDERINGS4 = 969; + +#ifdef _MSC_VER +#pragma region +#endif + // All total orderings for 16 pixels 2-bit selectors. + // BC1 selector order 0, 2, 3, 1 (i.e. the selectors are reordered into linear order). + static uint8_t g_unique_total_orders4[NUM_UNIQUE_TOTAL_ORDERINGS4][4] = + { + {0,8,2,6},{4,3,9,0},{4,8,1,3},{12,0,3,1},{11,3,2,0},{6,4,6,0},{7,5,0,4},{6,0,8,2},{1,0,0,15},{3,0,8,5},{1,1,13,1},{13,1,2,0},{0,14,1,1},{0,15,1,0},{0,13,0,3},{16,0,0,0},{4,3,4,5},{8,6,0,2},{0,10,0,6},{10,0,4,2},{7,2,1,6},{4,7,5,0},{1,4,7,4},{0,14,2,0},{2,7,2,5},{9,0,5,2},{9,2,2,3},{10,0,5,1},{2,3,7,4},{4,9,0,3},{1,5,0,10},{1,1,6,8}, + {6,6,4,0},{11,5,0,0},{11,2,0,3},{4,0,10,2},{2,3,10,1},{1,13,1,1},{0,14,0,2},{2,3,3,8},{12,3,1,0},{14,0,0,2},{9,1,3,3},{6,4,0,6},{1,1,5,9},{5,9,0,2},{2,10,1,3},{12,0,0,4},{4,6,6,0},{0,6,4,6},{3,7,4,2},{0,13,3,0},{3,10,0,3},{10,2,1,3},{1,12,1,2},{2,0,13,1},{11,0,5,0},{12,1,3,0},{6,4,5,1},{10,4,2,0},{3,6,1,6},{7,3,6,0},{10,4,0,2},{10,0,2,4}, + {0,5,9,2},{0,9,3,4},{6,4,2,4},{3,4,7,2},{3,3,5,5},{4,2,9,1},{6,2,8,0},{3,5,3,5},{4,10,1,1},{10,1,3,2},{5,7,0,4},{5,3,7,1},{6,8,1,1},{8,8,0,0},{11,1,0,4},{14,1,0,1},{9,3,2,2},{8,2,1,5},{0,0,2,14},{3,3,9,1},{10,1,5,0},{8,3,1,4},{1,5,8,2},{6,1,9,0},{3,2,1,10},{3,11,1,1},{7,6,3,0},{9,0,3,4},{5,2,5,4},{0,2,3,11},{15,0,0,1},{0,6,6,4}, + {3,4,9,0},{4,7,0,5},{0,4,4,8},{0,13,2,1},{2,4,1,9},{3,2,5,6},{10,6,0,0},{3,5,6,2},{8,0,4,4},{1,3,6,6},{7,7,0,2},{6,1,4,5},{0,11,1,4},{2,2,8,4},{0,1,2,13},{15,0,1,0},{7,2,6,1},{8,1,7,0},{1,8,4,3},{2,13,1,0},{1,0,7,8},{14,2,0,0},{1,8,1,6},{9,3,3,1},{0,0,7,9},{4,4,1,7},{9,0,6,1},{10,2,4,0},{1,7,3,5},{0,3,8,5},{5,2,4,5},{1,2,5,8}, + {0,8,7,1},{10,3,2,1},{12,0,4,0},{2,1,4,9},{5,2,2,7},{1,9,3,3},{15,1,0,0},{6,3,4,3},{9,5,0,2},{1,6,9,0},{6,6,0,4},{13,2,1,0},{5,1,8,2},{0,5,11,0},{7,1,0,8},{1,2,12,1},{0,3,3,10},{7,4,2,3},{5,1,4,6},{7,0,3,6},{3,12,0,1},{3,4,5,4},{1,10,0,5},{7,4,3,2},{10,5,0,1},{13,3,0,0},{2,5,4,5},{3,10,1,2},{5,1,2,8},{14,0,1,1},{1,5,4,6},{1,4,5,6}, + {2,3,11,0},{11,0,4,1},{11,2,2,1},{5,3,8,0},{1,3,10,2},{0,1,13,2},{3,1,4,8},{4,2,4,6},{1,5,6,4},{2,1,11,2},{1,2,9,4},{4,7,3,2},{6,2,5,3},{7,2,2,5},{8,1,4,3},{3,2,8,3},{12,1,0,3},{7,8,1,0},{7,0,2,7},{5,10,0,1},{0,2,14,0},{2,9,3,2},{7,0,0,9},{11,1,4,0},{10,4,1,1},{2,2,9,3},{5,7,2,2},{1,3,1,11},{13,2,0,1},{4,2,8,2},{2,3,1,10},{4,2,5,5}, + {7,0,7,2},{10,0,0,6},{0,8,5,3},{4,4,0,8},{12,4,0,0},{0,1,14,1},{8,0,1,7},{5,1,5,5},{11,0,3,2},{0,4,1,11},{0,8,8,0},{0,2,5,9},{7,3,2,4},{7,8,0,1},{1,0,3,12},{7,4,5,0},{1,6,7,2},{7,6,1,2},{9,6,1,0},{12,2,0,2},{4,1,6,5},{4,0,1,11},{8,4,4,0},{13,0,1,2},{8,6,2,0},{4,12,0,0},{2,7,5,2},{2,0,5,9},{5,4,5,2},{3,8,5,0},{7,3,3,3},{4,4,8,0}, + {2,1,3,10},{5,0,1,10},{6,4,3,3},{4,9,1,2},{1,4,0,11},{11,3,1,1},{4,0,12,0},{13,0,0,3},{6,1,6,3},{9,0,4,3},{8,0,0,8},{8,4,0,4},{0,12,1,3},{0,4,10,2},{3,4,8,1},{1,3,8,4},{9,2,5,0},{5,7,4,0},{1,0,11,4},{4,10,0,2},{1,3,12,0},{6,9,0,1},{5,0,9,2},{5,9,2,0},{13,1,0,2},{9,3,4,0},{9,4,0,3},{3,1,12,0},{2,4,3,7},{1,2,13,0},{2,2,4,8},{6,8,0,2}, + {9,2,1,4},{9,5,1,1},{2,0,4,10},{5,4,0,7},{0,0,6,10},{1,2,0,13},{4,7,2,3},{6,5,5,0},{3,3,1,9},{1,6,1,8},{12,2,1,1},{4,4,5,3},{1,0,6,9},{0,6,10,0},{4,8,3,1},{4,3,2,7},{2,1,7,6},{1,9,1,5},{3,1,3,9},{8,7,1,0},{1,2,3,10},{14,1,1,0},{5,4,4,3},{3,7,0,6},{7,4,1,4},{3,7,5,1},{1,1,0,14},{0,10,3,3},{0,4,3,9},{1,7,7,1},{2,0,10,4},{5,8,0,3}, + {6,7,3,0},{0,8,4,4},{5,7,3,1},{7,9,0,0},{7,6,2,1},{0,4,5,7},{6,3,5,2},{1,2,1,12},{5,2,0,9},{8,5,0,3},{4,6,1,5},{1,1,7,7},{10,5,1,0},{1,2,8,5},{1,8,2,5},{5,1,0,10},{6,9,1,0},{13,0,2,1},{8,3,5,0},{6,3,6,1},{2,11,3,0},{3,7,3,3},{1,5,2,8},{7,5,2,2},{0,6,7,3},{13,1,1,1},{5,3,4,4},{7,2,7,0},{5,8,3,0},{3,13,0,0},{0,7,9,0},{8,0,3,5}, + {1,3,7,5},{4,0,2,10},{12,0,1,3},{1,7,6,2},{3,9,0,4},{7,2,0,7},{0,1,7,8},{2,1,8,5},{0,13,1,2},{0,8,1,7},{5,0,11,0},{5,6,2,3},{0,3,0,13},{2,3,4,7},{5,6,3,2},{4,2,10,0},{3,3,7,3},{7,2,5,2},{1,1,11,3},{12,3,0,1},{5,1,1,9},{1,15,0,0},{9,7,0,0},{9,1,2,4},{0,7,3,6},{3,0,13,0},{3,0,11,2},{0,6,5,5},{8,2,2,4},{6,10,0,0},{4,8,4,0},{0,0,3,13}, + {0,4,12,0},{7,1,6,2},{3,5,0,8},{8,0,6,2},{6,2,3,5},{2,10,0,4},{4,11,0,1},{6,1,5,4},{5,1,3,7},{0,11,3,2},{4,6,0,6},{2,6,0,8},{3,1,7,5},{2,14,0,0},{2,9,2,3},{0,3,4,9},{11,0,1,4},{13,0,3,0},{8,3,0,5},{0,5,3,8},{5,11,0,0},{0,1,4,11},{2,1,9,4},{3,4,4,5},{7,1,2,6},{12,2,2,0},{9,4,1,2},{6,0,2,8},{4,6,2,4},{11,2,3,0},{3,2,2,9},{10,3,1,2}, + {1,1,2,12},{0,5,2,9},{0,1,11,4},{6,2,4,4},{2,8,2,4},{0,9,4,3},{11,0,2,3},{0,2,11,3},{6,0,7,3},{0,3,6,7},{4,5,5,2},{1,2,6,7},{7,5,1,3},{9,0,2,5},{2,6,4,4},{4,1,9,2},{4,8,2,2},{1,12,3,0},{0,9,6,1},{0,10,6,0},{3,1,5,7},{2,13,0,1},{2,2,1,11},{3,6,0,7},{5,6,5,0},{5,5,4,2},{4,0,3,9},{3,4,1,8},{0,11,2,3},{2,12,1,1},{7,1,3,5},{7,0,9,0}, + {8,0,8,0},{1,0,2,13},{3,3,10,0},{2,4,4,6},{2,3,8,3},{1,10,5,0},{7,3,0,6},{2,9,0,5},{1,4,6,5},{6,6,3,1},{5,6,0,5},{6,3,0,7},{3,10,2,1},{2,5,5,4},{3,8,4,1},{1,14,0,1},{10,3,3,0},{3,5,7,1},{1,1,3,11},{2,4,0,10},{9,3,1,3},{5,10,1,0},{3,0,6,7},{3,1,9,3},{11,2,1,2},{5,3,3,5},{0,5,1,10},{4,1,11,0},{10,2,0,4},{7,6,0,3},{2,7,0,7},{4,2,2,8}, + {6,1,7,2},{4,9,2,1},{0,0,8,8},{3,7,2,4},{9,6,0,1},{0,12,4,0},{6,7,1,2},{0,7,2,7},{1,0,10,5},{0,0,14,2},{2,7,3,4},{5,0,0,11},{7,7,1,1},{6,2,7,1},{4,5,3,4},{3,5,1,7},{5,9,1,1},{6,2,1,7},{3,2,0,11},{0,11,0,5},{3,11,2,0},{10,1,4,1},{7,0,4,5},{11,4,0,1},{10,3,0,3},{0,2,4,10},{0,15,0,1},{0,11,5,0},{6,7,2,1},{1,12,2,1},{4,1,3,8},{1,0,13,2}, + {1,8,5,2},{7,0,1,8},{3,12,1,0},{9,2,4,1},{1,7,4,4},{11,4,1,0},{4,3,8,1},{2,8,4,2},{1,11,3,1},{1,1,4,10},{4,10,2,0},{8,2,5,1},{1,0,9,6},{5,3,2,6},{0,9,7,0},{10,2,2,2},{5,8,1,2},{8,7,0,1},{0,3,12,1},{1,0,1,14},{4,8,0,4},{3,8,0,5},{4,6,5,1},{0,9,5,2},{10,2,3,1},{2,3,9,2},{1,0,12,3},{11,3,0,2},{4,5,2,5},{0,2,12,2},{9,1,0,6},{9,2,0,5}, + {1,2,7,6},{4,7,4,1},{0,12,2,2},{0,0,0,16},{2,8,3,3},{3,6,2,5},{0,6,3,7},{7,5,4,0},{3,3,3,7},{3,3,0,10},{5,0,6,5},{0,0,10,6},{8,5,3,0},{8,1,5,2},{6,0,9,1},{11,1,2,2},{2,11,2,1},{9,5,2,0},{3,0,4,9},{2,2,12,0},{2,6,6,2},{2,1,13,0},{6,0,5,5},{2,0,14,0},{2,11,1,2},{4,4,7,1},{2,0,11,3},{3,1,1,11},{2,9,4,1},{3,7,6,0},{14,0,2,0},{1,10,4,1}, + {8,0,7,1},{3,6,5,2},{0,3,11,2},{2,5,6,3},{11,1,3,1},{6,5,3,2},{3,8,1,4},{0,2,7,7},{2,10,2,2},{1,6,2,7},{11,0,0,5},{12,1,1,2},{12,1,2,1},{0,7,1,8},{0,3,9,4},{0,2,1,13},{7,1,4,4},{10,1,0,5},{4,0,8,4},{5,2,7,2},{0,2,0,14},{4,3,7,2},{2,7,1,6},{1,2,2,11},{6,3,3,4},{1,14,1,0},{2,4,6,4},{5,3,6,2},{5,3,5,3},{8,4,1,3},{1,3,0,12},{3,5,2,6}, + {1,8,7,0},{0,7,4,5},{2,1,6,7},{4,11,1,0},{7,2,4,3},{6,1,3,6},{4,5,4,3},{2,11,0,3},{1,5,7,3},{12,0,2,2},{5,0,4,7},{1,13,0,2},{7,7,2,0},{4,1,7,4},{4,5,0,7},{5,0,5,6},{6,5,4,1},{2,4,2,8},{1,10,1,4},{6,3,1,6},{3,3,8,2},{0,7,7,2},{4,4,2,6},{1,1,8,6},{1,12,0,3},{2,1,12,1},{1,9,2,4},{1,11,0,4},{2,5,2,7},{10,0,3,3},{4,6,3,3},{3,7,1,5}, + {1,9,0,6},{7,1,7,1},{1,6,5,4},{9,2,3,2},{6,2,2,6},{2,2,2,10},{8,3,3,2},{0,1,8,7},{2,0,8,6},{0,3,1,12},{9,4,2,1},{9,4,3,0},{6,2,6,2},{1,8,0,7},{5,1,10,0},{0,5,5,6},{8,2,4,2},{2,3,2,9},{6,0,3,7},{2,2,6,6},{2,6,2,6},{1,13,2,0},{9,3,0,4},{7,3,5,1},{6,5,2,3},{5,2,6,3},{2,0,12,2},{5,7,1,3},{8,1,3,4},{3,1,10,2},{1,0,15,0},{0,8,0,8}, + {5,0,7,4},{4,4,6,2},{0,1,0,15},{10,0,1,5},{7,3,4,2},{4,9,3,0},{2,5,7,2},{3,4,2,7},{8,3,2,3},{5,1,6,4},{0,10,2,4},{6,6,1,3},{6,0,0,10},{4,4,3,5},{1,3,9,3},{7,5,3,1},{3,0,7,6},{1,8,6,1},{4,3,0,9},{3,11,0,2},{6,0,6,4},{0,1,3,12},{0,4,2,10},{5,5,6,0},{4,1,4,7},{8,1,6,1},{5,6,4,1},{8,4,2,2},{4,3,1,8},{3,0,2,11},{1,11,4,0},{0,8,3,5}, + {5,1,7,3},{7,0,8,1},{4,3,5,4},{4,6,4,2},{3,2,4,7},{1,6,3,6},{0,7,8,1},{3,0,1,12},{9,1,4,2},{7,4,0,5},{1,7,0,8},{5,4,1,6},{9,1,5,1},{1,1,9,5},{4,1,1,10},{5,3,0,8},{2,2,5,7},{4,0,0,12},{9,0,7,0},{3,4,0,9},{0,2,6,8},{8,2,0,6},{3,2,6,5},{4,2,6,4},{3,6,4,3},{2,8,6,0},{5,0,3,8},{0,4,0,12},{0,16,0,0},{0,9,2,5},{4,0,11,1},{1,6,4,5}, + {0,1,6,9},{3,4,6,3},{3,0,10,3},{7,0,6,3},{1,4,9,2},{1,5,3,7},{8,5,2,1},{0,12,0,4},{7,2,3,4},{0,5,6,5},{11,1,1,3},{6,5,0,5},{2,1,5,8},{1,4,11,0},{9,1,1,5},{0,0,13,3},{5,8,2,1},{2,12,0,2},{3,3,6,4},{4,1,10,1},{4,0,5,7},{8,1,0,7},{5,1,9,1},{4,3,3,6},{0,2,2,12},{6,3,2,5},{0,0,12,4},{1,5,1,9},{2,6,5,3},{3,6,3,4},{2,12,2,0},{1,6,8,1}, + {10,1,1,4},{1,3,4,8},{7,4,4,1},{1,11,1,3},{1,2,10,3},{3,9,3,1},{8,5,1,2},{2,10,4,0},{4,2,0,10},{2,7,6,1},{8,2,3,3},{1,5,5,5},{3,1,0,12},{3,10,3,0},{8,0,5,3},{0,6,8,2},{0,3,13,0},{0,0,16,0},{1,9,4,2},{4,1,8,3},{1,6,6,3},{0,10,5,1},{0,1,12,3},{4,0,6,6},{3,8,3,2},{0,5,4,7},{1,0,14,1},{0,4,6,6},{3,9,1,3},{3,5,8,0},{3,6,6,1},{5,4,7,0}, + {3,0,12,1},{8,6,1,1},{2,9,5,0},{6,1,1,8},{4,1,2,9},{3,9,4,0},{5,2,9,0},{0,12,3,1},{1,4,10,1},{4,0,7,5},{3,1,2,10},{5,4,2,5},{5,5,5,1},{4,2,3,7},{1,7,5,3},{2,8,0,6},{8,1,2,5},{3,8,2,3},{6,1,2,7},{3,9,2,2},{9,0,0,7},{0,8,6,2},{8,4,3,1},{0,2,8,6},{6,5,1,4},{2,3,5,6},{2,10,3,1},{0,7,0,9},{4,2,7,3},{2,4,8,2},{7,1,1,7},{2,4,7,3}, + {2,4,10,0},{0,1,10,5},{4,7,1,4},{0,10,4,2},{9,0,1,6},{1,9,6,0},{3,3,4,6},{4,5,7,0},{5,5,2,4},{2,8,1,5},{2,3,6,5},{0,1,1,14},{3,2,3,8},{10,1,2,3},{9,1,6,0},{3,4,3,6},{2,2,0,12},{0,0,9,7},{4,0,9,3},{7,0,5,4},{4,5,6,1},{2,5,1,8},{2,5,9,0},{3,5,4,4},{1,3,11,1},{7,1,5,3},{3,2,7,4},{1,4,2,9},{1,11,2,2},{2,2,3,9},{5,0,10,1},{3,2,11,0}, + {1,10,3,2},{8,3,4,1},{3,6,7,0},{0,7,5,4},{1,3,3,9},{2,2,10,2},{1,9,5,1},{0,5,0,11},{3,0,3,10},{0,4,8,4},{2,7,7,0},{2,0,2,12},{1,2,11,2},{6,3,7,0},{0,6,2,8},{0,10,1,5},{0,9,0,7},{6,4,4,2},{6,0,1,9},{1,5,10,0},{5,4,6,1},{5,5,3,3},{0,0,4,12},{0,3,2,11},{1,4,1,10},{3,0,9,4},{5,5,0,6},{1,7,8,0},{2,0,3,11},{6,4,1,5},{10,0,6,0},{0,6,0,10}, + {0,4,11,1},{3,1,6,6},{2,5,8,1},{0,2,10,4},{3,1,11,1},{6,6,2,2},{1,1,10,4},{2,1,2,11},{6,1,8,1},{0,2,13,1},{0,7,6,3},{6,8,2,0},{3,0,0,13},{4,4,4,4},{6,2,0,8},{7,3,1,5},{0,11,4,1},{6,7,0,3},{2,6,3,5},{5,2,1,8},{7,1,8,0},{5,5,1,5},{1,8,3,4},{8,2,6,0},{6,0,10,0},{5,6,1,4},{1,4,4,7},{2,7,4,3},{1,4,8,3},{5,4,3,4},{1,10,2,3},{2,9,1,4}, + {2,2,11,1},{2,5,0,9},{0,0,1,15},{0,0,11,5},{0,4,7,5},{0,1,15,0},{2,1,0,13},{0,3,10,3},{8,0,2,6},{3,3,2,8},{3,5,5,3},{1,7,1,7},{1,3,2,10},{4,0,4,8},{2,0,9,5},{1,1,1,13},{2,2,7,5},{2,1,10,3},{4,2,1,9},{4,3,6,3},{1,3,5,7},{2,5,3,6},{1,0,8,7},{5,0,2,9},{2,8,5,1},{1,6,0,9},{0,0,5,11},{0,4,9,3},{2,0,7,7},{1,7,2,6},{2,1,1,12},{2,4,9,1}, + {0,5,7,4},{6,0,4,6},{3,2,10,1},{0,6,1,9},{2,6,1,7},{0,5,8,3},{4,1,0,11},{1,2,4,9},{4,1,5,6},{6,1,0,9},{1,4,3,8},{4,5,1,6},{1,0,5,10},{5,3,1,7},{0,9,1,6},{2,0,1,13},{2,0,6,8},{8,1,1,6},{1,5,9,1},{0,6,9,1},{0,3,5,8},{0,2,9,5},{5,2,8,1},{1,1,14,0},{3,2,9,2},{5,0,8,3},{0,5,10,1},{5,2,3,6},{2,6,7,1},{2,3,0,11},{0,1,9,6},{1,0,4,11}, + {3,0,5,8},{0,0,15,1},{2,4,5,5},{0,3,7,6},{2,0,0,14},{1,1,12,2},{2,6,8,0},{3,1,8,4},{0,1,5,10} + }; + + // All total orderings for 16 pixels [0,2] 2-bit selectors. + // BC1 selector order: 0, 1, 2 + // Note this is different from g_unique_total_orders4[], which reorders the selectors into linear order. + const uint32_t NUM_UNIQUE_TOTAL_ORDERINGS3 = 153; + static uint8_t g_unique_total_orders3[NUM_UNIQUE_TOTAL_ORDERINGS3][3] = + { + {6,0,10},{3,6,7},{3,0,13},{13,3,0},{12,4,0},{9,1,6},{2,13,1},{4,7,5},{7,5,4},{9,6,1},{7,4,5},{8,6,2},{16,0,0},{10,6,0},{2,7,7}, + {0,0,16},{0,3,13},{1,15,0},{0,2,14},{1,4,11},{15,1,0},{1,12,3},{9,2,5},{14,1,1},{8,2,6},{3,3,10},{4,2,10},{14,0,2},{0,14,2},{1,7,8},{6,6,4}, + {11,5,0},{6,4,6},{11,3,2},{4,3,9},{7,1,8},{10,4,2},{12,1,3},{11,0,5},{9,3,4},{1,0,15},{9,0,7},{2,6,8},{12,2,2},{6,2,8},{6,8,2},{15,0,1}, + {4,8,4},{0,4,12},{8,5,3},{5,9,2},{11,2,3},{12,3,1},{6,3,7},{1,1,14},{2,9,5},{1,8,7},{4,10,2},{7,7,2},{13,1,2},{0,15,1},{3,2,11},{7,0,9}, + {4,4,8},{3,8,5},{0,5,11},{13,2,1},{1,10,5},{4,11,1},{3,10,3},{5,10,1},{10,2,4},{0,6,10},{14,2,0},{11,4,1},{3,12,1},{1,13,2},{1,5,10},{5,11,0}, + {12,0,4},{8,1,7},{6,10,0},{3,13,0},{7,2,7},{0,7,9},{5,8,3},{0,12,4},{11,1,4},{13,0,3},{0,16,0},{5,7,4},{10,3,3},{10,0,6},{0,13,3},{4,6,6}, + {2,8,6},{2,5,9},{7,8,1},{2,1,13},{2,0,14},{7,3,6},{5,1,10},{3,11,2},{5,4,7},{8,3,5},{10,5,1},{6,9,1},{1,3,12},{4,5,7},{2,2,12},{4,1,11}, + {0,8,8},{4,12,0},{6,5,5},{8,7,1},{5,5,6},{3,7,6},{7,9,0},{4,9,3},{0,10,6},{8,0,8},{5,3,8},{10,1,5},{6,1,9},{7,6,3},{9,5,2},{0,1,15}, + {9,7,0},{2,14,0},{3,4,9},{8,4,4},{9,4,3},{0,9,7},{1,9,6},{3,9,4},{5,2,9},{2,3,11},{5,6,5},{1,14,1},{6,7,3},{2,4,10},{2,12,2},{8,8,0}, + {2,10,4},{4,0,12},{0,11,5},{2,11,3},{1,11,4},{3,5,8},{5,0,11},{3,1,12},{1,2,13},{1,6,9} + }; + + // For each total ordering, this table indicates which other total orderings are likely to improve quality using a least squares pass. Each array is sorted by usefulness. + static uint16_t g_best_total_orderings4[NUM_UNIQUE_TOTAL_ORDERINGS4][MAX_TOTAL_ORDERINGS4] = + { +#if RGBCX_USE_SMALLER_TABLES + #include "rgbcx_table4_small.h" +#else + #include "rgbcx_table4.h" +#endif + }; + + static uint8_t g_best_total_orderings3[NUM_UNIQUE_TOTAL_ORDERINGS3][32] = + { + { 12,1,3,5,27,2,4,38,8,7,16,18,6,10,41,79,40,23,46,9,20,88,22,37,14,19,24,126,99,119,35,11 }, + { 7,64,116,14,94,30,8,42,1,108,47,55,137,10,134,95,96,115,69,32,63,29,90,113,11,148,16,103,19,9,34,25 }, + { 12,1,0,5,3,7,4,27,8,6,38,40,41,16,18,46,9,10,20,23,79,62,14,22,88,99,37,126,92,19,120,11 }, + { 16,88,27,18,46,48,126,107,79,19,59,38,37,65,23,66,0,2,3,43,12,151,28,25,5,87,72,40,1,20,52,92 }, + { 79,48,88,16,27,65,18,38,46,19,37,4,72,33,126,41,52,0,12,92,5,1,2,107,3,77,23,91,43,51,22,74 }, + { 1,8,41,122,10,22,2,0,87,24,37,120,38,7,39,4,5,3,9,92,62,59,23,16,104,11,27,79,19,26,25,32 }, + { 2,76,99,28,40,86,93,21,138,60,6,0,17,128,145,119,98,144,141,82,147,54,67,75,5,12,27,132,146,1,38,14 }, + { 47,7,64,90,1,118,116,85,57,14,30,94,50,45,137,134,8,42,69,139,55,68,58,108,95,29,10,115,0,32,2,11 }, + { 49,8,10,30,124,11,32,113,130,58,125,9,100,53,104,115,131,103,24,7,1,39,45,36,139,0,137,22,90,44,114,105 }, + { 9,38,72,125,49,41,84,11,13,5,27,0,16,92,8,2,65,105,10,18,48,29,127,131,36,14,1,46,111,79,130,12 }, + { 130,8,10,100,104,131,49,32,53,39,30,36,113,24,11,22,124,44,83,58,7,103,1,4,9,125,5,0,91,33,115,74 }, + { 114,11,58,8,120,49,9,124,142,111,41,30,10,0,97,130,62,84,38,5,72,125,92,127,100,27,139,113,13,132,32,1 }, + { 60,46,28,27,40,20,0,17,18,2,126,16,6,38,86,23,79,54,1,93,5,88,41,14,21,111,7,48,3,84,72,62 }, + { 72,92,38,65,84,48,41,79,27,16,29,111,88,5,18,46,1,0,152,14,37,19,77,42,132,7,22,13,119,56,12,2 }, + { 7,55,1,95,29,56,64,116,143,8,14,30,47,94,152,90,65,67,10,133,42,72,146,84,16,48,6,0,25,108,77,21 }, + { 27,23,20,5,0,79,38,2,3,1,59,46,4,41,33,86,37,87,88,92,7,126,43,8,22,152,151,150,149,148,147,146 }, + { 12,0,1,2,7,6,3,5,28,4,8,14,60,40,17,19,21,86,126,93,10,18,9,29,48,99,65,25,84,119,72,41 }, + { 60,40,99,2,54,12,0,1,19,28,98,93,6,138,21,5,27,17,151,14,76,46,16,18,38,29,86,144,107,7,25,41 }, + { 12,0,1,2,3,5,6,7,4,28,8,60,14,40,16,17,21,10,19,9,86,38,126,41,93,27,29,48,62,84,79,99 }, + { 0,1,2,10,5,8,3,25,4,29,32,34,63,7,77,26,16,48,65,56,14,22,129,103,72,24,18,152,140,53,96,42 }, + { 46,126,18,54,12,16,1,0,5,2,27,98,20,23,6,3,88,48,28,7,19,8,4,60,151,38,37,21,79,14,65,40 }, + { 76,6,141,86,119,2,138,67,28,145,0,93,17,1,40,60,146,99,147,14,21,144,132,7,5,29,55,27,16,75,19,12 }, + { 71,5,51,39,22,80,0,43,10,122,8,62,41,24,104,87,35,37,2,91,33,120,36,38,1,131,9,100,130,66,3,4 }, + { 126,18,46,27,20,16,88,23,12,79,54,59,48,0,73,1,37,151,5,19,28,38,2,66,60,3,65,98,14,26,6,43 }, + { 22,10,8,5,0,71,35,80,104,39,24,51,100,1,62,32,2,130,11,41,7,9,53,43,49,83,122,120,30,44,37,38 }, + { 1,34,14,129,53,63,42,26,121,148,7,44,96,10,0,24,100,32,64,116,140,22,5,19,29,103,135,108,8,61,39,83 }, + { 1,7,34,63,44,25,135,14,24,108,22,0,83,94,5,129,35,101,47,121,2,19,42,53,6,110,103,8,148,10,16,123 }, + { 12,28,16,60,18,1,6,21,14,0,86,19,2,48,93,17,38,29,7,5,65,126,46,72,41,79,84,119,40,56,54,88 }, + { 0,2,12,27,5,46,38,40,41,79,88,99,3,23,1,62,20,4,22,37,92,35,18,8,16,24,10,60,7,120,98,54 }, + { 1,7,14,56,8,0,84,67,10,2,133,72,42,111,5,30,21,4,9,3,25,94,16,116,47,11,65,18,132,90,55,64 }, + { 30,8,124,139,45,11,58,90,113,137,7,115,10,32,1,49,94,85,9,47,108,103,0,97,63,14,50,114,53,106,100,25 }, + { 65,38,48,27,16,79,72,18,88,19,46,77,84,92,37,41,0,29,1,14,12,111,2,5,31,36,87,74,105,40,28,51 }, + { 10,8,30,113,130,100,53,32,115,103,104,7,1,121,39,49,131,44,24,36,63,137,34,45,22,90,108,83,26,11,94,139 }, + { 51,52,43,33,5,74,16,37,71,91,38,3,36,87,48,22,4,0,122,41,39,18,66,27,79,24,65,88,59,23,62,92 }, + { 1,7,63,53,108,121,94,44,103,100,14,10,129,47,32,26,24,25,148,42,135,22,0,61,83,8,39,104,5,64,115,34 }, + { 1,8,10,7,5,0,80,32,62,2,24,44,53,83,9,41,30,22,100,11,14,25,120,4,26,6,3,16,122,34,19,35 }, + { 74,4,36,48,33,91,39,79,22,16,65,5,131,38,24,71,27,52,0,105,51,18,88,104,3,31,10,37,72,19,41,130 }, + { 59,43,38,79,23,27,92,51,0,16,46,5,18,88,41,37,66,3,87,20,48,2,122,4,22,12,1,126,19,65,33,24 }, + { 12,28,1,27,0,16,2,46,65,60,21,3,5,18,6,19,48,14,4,7,79,88,86,29,22,72,93,40,23,8,17,41 }, + { 22,91,39,33,24,71,5,131,36,10,51,0,130,8,104,2,35,125,9,43,52,49,83,80,100,41,122,3,37,38,4,16 }, + { 12,0,1,2,5,3,4,8,7,27,18,38,10,6,16,46,9,20,41,23,126,79,22,14,19,99,88,54,37,48,62,35 }, + { 12,27,1,2,3,0,46,4,38,16,8,28,7,79,18,5,84,6,88,10,14,21,23,20,40,22,60,19,9,29,72,65 }, + { 1,14,7,55,95,29,8,94,30,56,10,108,77,116,152,64,32,48,63,42,143,148,16,25,137,65,11,0,115,9,19,72 }, + { 37,79,66,38,16,52,48,59,43,27,87,33,41,4,23,51,3,5,88,18,92,46,73,122,22,71,20,0,65,19,2,120 }, + { 24,32,83,22,53,1,8,10,7,30,35,5,103,0,100,101,121,113,34,123,63,2,44,25,71,115,80,14,26,108,51,39 }, + { 97,45,111,58,85,139,0,90,47,7,120,106,142,30,50,132,41,62,84,1,119,114,14,56,117,8,38,29,2,64,116,5 }, + { 12,28,16,18,1,60,6,14,2,21,0,86,126,19,48,93,7,27,17,29,5,65,54,38,72,79,84,88,119,145,8,111 }, + { 118,47,64,116,57,85,7,14,50,1,42,0,45,68,86,69,2,111,134,28,90,55,16,29,56,48,84,144,60,30,112,41 }, + { 12,1,2,0,7,6,28,5,3,4,8,14,60,21,18,40,17,86,10,9,16,29,19,93,126,79,38,84,72,27,111,119 }, + { 11,8,49,130,10,125,9,124,100,114,131,30,58,104,32,39,24,113,36,105,0,41,22,120,5,53,111,38,142,44,83,35 }, + { 50,70,47,118,85,57,106,0,45,7,64,90,81,14,2,134,28,62,86,55,69,1,78,119,68,56,18,67,16,60,29,21 }, + { 43,37,33,87,51,41,66,5,122,38,22,59,92,0,23,91,27,16,71,79,18,52,120,4,3,24,46,20,73,39,62,36 }, + { 79,48,4,16,27,88,43,33,18,38,65,37,46,3,19,51,52,22,66,87,74,5,41,91,23,59,0,71,122,72,20,92 }, + { 32,100,10,8,30,104,24,44,39,113,83,103,1,7,22,53,115,63,135,121,26,35,34,5,0,108,137,90,91,45,2,130 }, + { 0,1,2,5,16,12,6,7,14,3,19,18,29,20,4,21,40,8,17,35,23,48,126,22,25,56,26,10,98,27,38,65 }, + { 143,67,56,146,1,7,133,55,64,141,134,69,6,47,14,29,84,21,111,147,57,16,95,72,118,132,50,0,2,18,119,42 }, + { 1,7,67,14,133,111,8,84,0,21,2,47,64,132,55,10,95,147,119,42,16,5,72,56,4,3,6,29,9,25,18,30 }, + { 68,57,69,112,144,86,102,2,134,55,0,70,118,64,75,47,14,28,93,143,67,7,50,149,1,21,29,56,119,95,60,78 }, + { 58,97,114,30,124,45,11,139,8,90,0,142,7,10,41,113,84,62,49,111,85,1,9,5,137,120,32,14,2,117,47,38 }, + { 23,66,18,79,38,20,43,27,16,88,46,59,126,37,87,12,73,92,3,5,48,0,19,54,2,51,28,1,41,65,122,22 }, + { 0,12,2,27,5,40,46,38,1,41,3,79,88,23,99,4,20,62,22,54,92,18,8,37,16,35,10,7,19,120,144,24 }, + { 1,14,25,26,0,7,44,34,129,42,24,5,135,22,19,148,6,96,83,2,29,16,63,35,101,64,140,136,116,110,3,10 }, + { 12,1,2,27,3,4,38,5,7,8,18,16,46,6,0,40,41,10,79,23,88,9,20,22,14,19,37,92,48,126,28,21 }, + { 7,1,10,32,108,103,94,47,8,53,25,14,34,115,100,129,121,130,148,42,64,116,63,26,44,0,24,30,113,4,104,22 }, + { 47,134,7,14,55,69,64,95,1,29,85,118,56,116,45,57,102,143,50,90,42,30,16,94,0,8,67,75,133,2,18,48 }, + { 12,1,2,0,7,6,28,8,14,5,3,4,40,21,17,18,60,86,16,93,126,10,9,29,99,38,119,25,19,54,27,84 }, + { 59,16,27,18,23,88,79,37,46,66,38,20,73,126,3,43,48,87,92,51,41,12,19,5,52,107,65,0,151,122,54,2 }, + { 1,21,147,7,119,14,76,132,55,0,86,145,2,6,69,67,16,143,111,138,17,28,29,60,18,93,8,19,40,56,84,5 }, + { 144,86,112,2,68,102,69,0,149,93,75,28,57,55,145,60,21,67,99,134,143,40,146,119,82,110,62,6,29,26,78,14 }, + { 102,57,55,69,143,75,146,67,56,68,134,2,29,141,0,21,6,14,133,118,64,1,7,95,47,84,111,28,147,82,72,119 }, + { 0,70,57,119,50,145,2,86,28,118,69,78,149,47,60,68,67,55,93,81,134,21,14,62,64,7,5,1,132,85,41,16 }, + { 51,5,43,71,122,87,41,37,91,39,0,22,33,36,38,24,66,120,62,2,80,16,92,10,59,4,27,23,35,79,8,3 }, + { 12,1,2,0,7,6,28,5,8,14,3,21,40,4,60,17,86,18,16,93,10,9,126,119,99,29,19,41,38,27,25,92 }, + { 27,18,46,126,23,16,88,79,20,151,59,73,48,38,0,54,12,2,37,1,19,5,28,60,66,41,3,109,86,65,40,6 }, + { 48,79,4,33,16,74,65,38,88,27,91,52,18,36,22,19,46,0,37,3,51,5,71,39,72,43,24,41,92,87,2,10 }, + { 86,2,144,93,28,112,141,6,102,21,99,60,75,0,68,82,69,146,67,149,55,40,145,76,111,147,56,119,110,143,26,132 }, + { 6,138,2,99,86,17,40,93,28,21,145,141,0,60,119,147,128,76,67,54,1,12,5,27,144,14,38,98,146,41,29,19 }, + { 1,8,0,10,2,29,7,5,3,56,4,25,14,152,63,32,65,72,96,42,34,108,48,9,26,16,84,103,67,148,22,129 }, + { 149,145,0,86,2,28,93,144,62,60,119,101,21,41,5,35,78,99,26,40,12,68,57,67,110,120,69,18,55,76,132,70 }, + { 12,28,16,1,48,19,6,60,2,14,18,21,0,27,46,65,86,29,5,7,72,93,40,3,17,84,56,88,126,4,38,8 }, + { 1,8,5,10,7,24,2,62,0,41,22,122,120,9,4,3,32,87,11,37,38,83,100,44,25,104,16,26,39,80,14,6 }, + { 0,119,62,86,145,149,28,132,93,2,120,67,60,41,35,5,144,21,123,38,111,81,84,56,12,44,24,50,92,55,40,22 }, + { 2,93,99,28,40,144,60,0,86,150,76,21,149,98,6,25,1,61,82,26,12,5,54,141,7,18,145,16,27,138,110,38 }, + { 24,8,10,22,32,35,100,5,1,53,0,7,71,80,30,123,83,104,51,11,2,39,44,113,9,62,25,103,34,101,43,41 }, + { 12,1,2,0,7,6,28,5,40,60,8,16,3,18,14,4,86,21,17,93,41,10,9,99,27,119,38,19,126,22,48,145 }, + { 45,47,50,7,85,90,97,1,64,139,116,118,30,58,14,106,70,111,0,57,94,42,137,142,29,120,8,56,18,134,84,41 }, + { 12,0,2,5,27,38,1,46,41,40,79,144,3,22,88,23,28,60,99,62,6,24,26,7,4,16,10,35,37,18,14,20 }, + { 37,38,59,92,0,5,23,51,79,41,27,22,2,3,87,16,46,4,1,43,20,33,18,88,24,71,8,10,48,19,126,122 }, + { 12,28,16,60,1,18,6,21,19,14,48,0,2,86,93,5,46,29,17,27,65,7,3,72,38,126,119,40,84,37,56,4 }, + { 0,2,5,1,16,6,27,28,18,38,60,7,14,21,46,40,86,41,19,48,93,8,3,79,22,4,10,37,62,23,24,111 }, + { 85,7,90,30,47,139,45,50,94,58,137,1,8,64,14,116,118,115,113,11,124,108,0,10,97,57,32,70,42,106,29,114 }, + { 33,36,22,71,51,5,91,39,0,52,43,24,131,74,16,37,38,122,41,3,87,48,4,104,35,80,10,2,105,62,27,18 }, + { 12,1,27,2,0,16,3,28,46,18,4,6,5,72,21,79,38,7,14,60,88,8,65,19,48,29,23,40,22,20,86,126 }, + { 0,12,2,27,5,38,46,41,1,40,79,3,88,23,22,99,20,37,62,4,18,6,16,35,60,28,24,7,92,8,14,10 }, + { 7,47,1,30,137,8,116,94,90,64,14,115,108,118,57,10,148,113,42,85,32,11,63,50,103,45,124,134,55,9,69,34 }, + { 55,7,1,29,56,143,64,47,67,133,14,146,95,72,84,8,116,111,6,134,141,21,65,0,69,30,16,45,85,42,50,10 }, + { 14,1,42,8,10,29,108,63,55,148,95,32,7,19,25,115,103,34,56,129,77,0,16,152,94,30,113,26,2,5,48,4 }, + { 111,120,142,97,58,0,41,45,62,132,114,84,139,30,5,8,38,2,7,85,119,90,117,1,124,11,56,47,28,27,35,72 }, + { 1,0,14,2,6,5,16,19,7,29,42,18,3,25,12,35,21,8,26,17,40,4,20,48,109,99,22,96,55,101,10,61 }, + { 12,0,1,5,3,2,4,7,27,8,38,6,40,18,16,10,20,46,9,41,23,22,79,14,62,19,37,126,88,11,92,48 }, + { 10,8,104,39,24,32,22,83,44,100,30,130,53,91,113,5,11,1,35,33,7,49,0,2,103,71,36,124,9,80,131,34 }, + { 1,7,0,14,8,34,5,25,35,26,6,63,10,123,2,16,103,19,44,32,135,121,108,80,62,30,115,94,149,144,53,18 }, + { 75,68,146,141,102,67,2,21,6,57,69,143,0,55,82,86,28,144,147,29,93,112,56,119,133,14,76,60,84,134,111,145 }, + { 10,32,115,7,8,53,1,108,30,113,94,137,100,63,90,34,130,103,121,47,44,25,104,39,24,26,85,14,49,36,22,131 }, + { 39,24,10,22,8,130,91,104,83,49,5,33,100,11,0,35,32,131,71,36,9,44,53,2,80,51,30,1,41,7,43,62 }, + { 38,36,65,105,27,72,31,79,41,131,5,48,125,39,0,16,92,46,22,13,18,84,24,37,88,2,33,74,91,71,130,49 }, + { 0,106,62,50,45,119,85,81,132,28,2,86,41,47,38,60,35,117,5,29,7,30,145,90,55,70,14,111,18,67,93,56 }, + { 0,2,5,1,3,25,19,26,4,34,29,10,22,16,8,7,24,14,48,65,53,18,6,77,44,56,72,61,121,21,136,40 }, + { 7,1,94,8,47,115,10,32,113,103,30,108,137,63,14,64,116,148,129,42,90,25,34,118,53,57,11,49,85,9,96,50 }, + { 14,0,1,26,19,5,42,2,25,24,29,22,6,44,61,16,7,96,136,3,140,34,35,55,135,18,48,77,83,4,8,10 }, + { 1,7,14,0,25,6,34,5,26,16,63,2,19,8,35,101,108,29,94,10,18,42,123,144,129,47,61,21,3,62,149,4 }, + { 12,0,2,1,28,5,6,120,7,60,40,16,18,86,27,14,21,93,8,62,41,38,3,17,4,119,99,48,19,126,10,9 }, + { 86,144,93,2,28,149,0,60,99,112,110,145,40,21,102,26,75,62,69,1,12,101,119,25,76,67,7,68,55,5,6,14 }, + { 8,30,10,32,113,49,115,137,124,103,45,90,7,139,11,1,58,53,130,94,108,100,9,63,85,125,34,47,0,24,44,104 }, + { 120,142,111,41,58,114,97,0,11,62,84,124,5,30,8,38,132,127,27,139,92,10,72,45,49,9,28,2,29,56,16,1 }, + { 8,113,30,137,7,32,10,90,94,115,1,103,108,63,47,85,49,53,11,45,34,50,14,25,9,124,100,130,139,121,42,26 }, + { 64,7,14,47,134,55,1,42,95,69,116,90,94,30,8,29,56,137,45,108,85,10,57,16,102,143,118,19,63,32,11,50 }, + { 62,132,0,119,120,41,111,86,35,28,5,84,56,38,2,93,145,60,67,12,92,27,29,72,55,117,21,24,133,149,22,45 }, + { 57,68,69,118,134,64,50,47,55,14,7,2,102,144,0,112,70,86,85,1,95,29,116,143,42,75,16,56,28,45,21,48 }, + { 0,12,2,1,5,28,6,40,60,27,7,38,16,14,86,18,93,41,62,46,99,35,8,23,3,17,22,21,10,19,79,20 }, + { 12,1,2,27,16,3,38,111,4,0,18,5,7,46,40,8,79,6,14,28,88,10,48,41,19,84,21,9,22,23,20,72 }, + { 53,103,32,7,1,100,22,63,71,44,10,115,108,24,92,104,26,30,122,94,8,39,83,34,137,135,90,91,121,5,87,47 }, + { 87,37,41,0,22,38,2,92,1,24,4,8,3,59,10,5,39,23,71,79,122,27,16,46,33,7,91,20,18,51,9,120 }, + { 1,7,8,10,0,5,35,32,53,44,14,30,2,80,25,34,6,62,26,103,16,19,63,9,149,24,121,41,22,11,113,83 }, + { 11,58,8,30,124,49,10,113,9,114,139,45,97,32,7,137,90,1,0,130,115,125,100,24,5,94,53,41,14,13,35,38 }, + { 125,105,9,36,131,49,8,130,39,11,10,5,22,38,41,104,0,31,13,24,27,16,2,72,65,91,48,32,84,18,100,74 }, + { 12,1,0,2,6,3,7,5,4,8,14,28,16,60,18,10,21,17,19,9,40,27,86,93,29,38,54,11,25,48,46,41 }, + { 84,41,38,72,92,29,111,5,65,120,79,0,27,56,48,14,132,16,119,22,86,88,46,28,62,12,1,2,93,18,24,127 }, + { 99,28,40,60,2,93,138,0,98,17,86,54,76,12,27,1,21,144,128,38,5,14,46,18,25,16,109,6,41,145,7,29 }, + { 1,63,10,32,148,14,103,34,42,7,8,108,116,53,64,96,25,121,26,94,140,0,29,19,55,24,100,136,5,4,44,115 }, + { 131,100,130,49,10,8,36,104,39,0,48,41,11,38,4,24,27,22,16,44,79,5,33,2,53,9,125,74,91,120,32,83 }, + { 36,39,131,74,4,91,22,33,125,104,130,48,10,24,16,5,49,8,100,105,79,0,9,65,71,2,18,83,31,11,19,44 }, + { 0,12,2,1,6,5,7,28,40,60,16,14,18,62,86,27,93,8,17,38,21,41,35,99,3,19,10,23,22,4,9,48 }, + { 1,7,67,14,21,147,111,55,132,119,0,8,2,76,64,16,47,84,6,18,86,95,145,10,42,29,133,5,56,134,17,72 }, + { 69,55,47,134,102,143,7,57,118,95,14,64,29,56,1,50,75,67,146,2,0,133,68,16,21,6,141,85,116,18,72,65 }, + { 1,44,7,24,83,63,34,103,22,121,53,32,25,35,0,115,108,5,14,8,10,101,94,30,2,123,110,26,137,47,90,19 }, + { 14,1,25,42,34,0,26,96,19,29,140,5,53,10,2,121,3,24,44,22,55,77,129,7,63,16,8,4,6,61,100,48 }, + { 30,90,7,8,137,94,85,1,47,113,115,108,45,139,124,11,10,32,50,58,103,14,63,64,9,116,49,42,25,148,0,53 }, + { 40,99,2,60,28,17,0,54,93,98,86,138,6,12,21,76,1,5,27,144,128,38,19,46,14,41,145,7,16,67,3,109 }, + { 45,58,30,139,90,7,85,137,97,8,124,47,1,11,106,114,50,94,0,113,10,115,14,32,9,64,108,41,49,29,62,116 }, + { 14,42,10,1,63,96,32,25,34,8,129,29,0,103,55,19,26,53,77,5,95,2,4,7,3,16,148,56,18,24,121,108 }, + { 21,2,75,86,6,76,144,28,119,99,93,147,141,67,102,145,60,132,146,128,0,82,40,138,55,111,143,17,133,112,69,14 }, + { 111,120,41,62,84,132,0,5,38,119,56,92,72,142,27,28,29,35,58,80,2,86,65,79,12,14,1,24,145,16,21,48 }, + { 146,67,141,69,133,21,6,143,57,55,111,147,56,1,14,132,7,2,134,102,0,119,29,84,76,64,86,72,28,68,47,75 }, + { 12,1,0,5,27,3,7,4,38,8,6,41,16,40,46,10,18,79,2,9,23,86,20,22,62,14,37,88,92,19,24,11 }, + { 0,12,2,1,27,5,38,28,60,6,40,7,16,46,18,14,41,99,93,62,3,79,86,23,149,8,22,35,88,17,19,10 }, + { 141,6,21,67,147,102,146,2,76,119,132,69,55,111,86,75,28,133,143,0,1,145,14,128,56,99,17,60,29,93,84,68 }, + { 21,76,1,119,86,145,2,0,14,7,6,138,146,55,17,28,132,93,67,40,60,143,29,147,111,16,69,141,5,56,19,133 }, + { 1,8,108,14,7,116,64,42,10,63,94,32,115,103,113,96,30,34,55,47,95,148,29,140,129,25,134,53,69,26,19,11 }, + { 12,1,3,5,4,2,0,7,8,38,27,16,18,6,10,20,41,40,79,46,9,23,22,88,92,37,14,24,62,19,48,99 }, + { 1,14,7,0,6,25,5,16,19,2,42,26,29,35,61,8,18,129,101,21,3,110,34,148,96,10,17,4,22,40,12,20 }, + { 0,2,5,1,3,19,22,26,16,24,29,7,14,6,4,25,18,44,8,48,12,61,20,21,10,35,65,56,23,40,17,107 }, + { 1,7,8,29,56,0,10,14,2,42,72,5,4,65,3,30,84,94,67,9,25,133,111,11,32,108,16,63,21,96,26,48 } + }; +#ifdef _MSC_VER +#pragma endregion +#endif + + static inline uint32_t iabs(int32_t i) { return (i < 0) ? static_cast(-i) : static_cast(i); } + //static inline uint64_t iabs(int64_t i) { return (i < 0) ? static_cast(-i) : static_cast(i); } + + static inline uint8_t to_5(uint32_t v) { v = v * 31 + 128; return (uint8_t)((v + (v >> 8)) >> 8); } + static inline uint8_t to_6(uint32_t v) { v = v * 63 + 128; return (uint8_t)((v + (v >> 8)) >> 8); } + + template inline T square(T a) { return a * a; } + + static inline float clampf(float value, float low, float high) { if (value < low) value = low; else if (value > high) value = high; return value; } + + template inline S clamp(S value, S low, S high) { return (value < low) ? low : ((value > high) ? high : value); } + static inline int32_t clampi(int32_t value, int32_t low, int32_t high) { if (value < low) value = low; else if (value > high) value = high; return value; } + + static inline int squarei(int a) { return a * a; } + //static inline int absi(int a) { return (a < 0) ? -a : a; } + + template inline F lerp(F a, F b, F s) { return a + (b - a) * s; } + + static const uint32_t TOTAL_ORDER_4_0_16 = 15; + static const uint32_t TOTAL_ORDER_4_1_16 = 700; + static const uint32_t TOTAL_ORDER_4_2_16 = 753; + static const uint32_t TOTAL_ORDER_4_3_16 = 515; + static uint16_t g_total_ordering4_hash[4096]; + static float g_selector_factors4[NUM_UNIQUE_TOTAL_ORDERINGS4][3]; + + static const uint32_t TOTAL_ORDER_3_0_16 = 12; + static const uint32_t TOTAL_ORDER_3_1_16 = 15; + static const uint32_t TOTAL_ORDER_3_2_16 = 89; + static uint16_t g_total_ordering3_hash[256]; + static float g_selector_factors3[NUM_UNIQUE_TOTAL_ORDERINGS3][3]; + + struct hist4 + { + uint8_t m_hist[4]; + + hist4() + { + memset(m_hist, 0, sizeof(m_hist)); + } + + hist4(uint32_t i, uint32_t j, uint32_t k, uint32_t l) + { + m_hist[0] = (uint8_t)i; + m_hist[1] = (uint8_t)j; + m_hist[2] = (uint8_t)k; + m_hist[3] = (uint8_t)l; + } + + inline bool operator== (const hist4& h) const + { + if (m_hist[0] != h.m_hist[0]) return false; + if (m_hist[1] != h.m_hist[1]) return false; + if (m_hist[2] != h.m_hist[2]) return false; + if (m_hist[3] != h.m_hist[3]) return false; + return true; + } + + inline bool any_16() const + { + return (m_hist[0] == 16) || (m_hist[1] == 16) || (m_hist[2] == 16) || (m_hist[3] == 16); + } + + inline uint32_t lookup_total_ordering_index() const + { + if (m_hist[0] == 16) + return TOTAL_ORDER_4_0_16; + else if (m_hist[1] == 16) + return TOTAL_ORDER_4_1_16; + else if (m_hist[2] == 16) + return TOTAL_ORDER_4_2_16; + else if (m_hist[3] == 16) + return TOTAL_ORDER_4_3_16; + + // Must sum to 16, so m_hist[3] isn't needed. + return g_total_ordering4_hash[m_hist[0] | (m_hist[1] << 4) | (m_hist[2] << 8)]; + } + }; + + struct hist3 + { + uint8_t m_hist[3]; + + hist3() + { + memset(m_hist, 0, sizeof(m_hist)); + } + + hist3(uint32_t i, uint32_t j, uint32_t k) + { + m_hist[0] = (uint8_t)i; + m_hist[1] = (uint8_t)j; + m_hist[2] = (uint8_t)k; + } + + inline bool operator== (const hist3& h) const + { + if (m_hist[0] != h.m_hist[0]) return false; + if (m_hist[1] != h.m_hist[1]) return false; + if (m_hist[2] != h.m_hist[2]) return false; + return true; + } + + inline bool any_16() const + { + return (m_hist[0] == 16) || (m_hist[1] == 16) || (m_hist[2] == 16); + } + + inline uint32_t lookup_total_ordering_index() const + { + if (m_hist[0] == 16) + return TOTAL_ORDER_3_0_16; + else if (m_hist[1] == 16) + return TOTAL_ORDER_3_1_16; + else if (m_hist[2] == 16) + return TOTAL_ORDER_3_2_16; + + // Must sum to 16, so m_hist[2] isn't needed. + return g_total_ordering3_hash[m_hist[0] | (m_hist[1] << 4)]; + } + }; + + struct bc1_match_entry + { + uint8_t m_hi; + uint8_t m_lo; + uint8_t m_e; + }; + + static bc1_approx_mode g_bc1_approx_mode; + static bc1_match_entry g_bc1_match5_equals_1[256], g_bc1_match6_equals_1[256]; + static bc1_match_entry g_bc1_match5_half[256], g_bc1_match6_half[256]; + + static inline int scale_5_to_8(int v) { return (v << 3) | (v >> 2); } + static inline int scale_6_to_8(int v) { return (v << 2) | (v >> 4); } + + // v0, v1 = unexpanded DXT1 endpoint values (5/6-bits) + // c0, c1 = expanded DXT1 endpoint values (8-bits) + static inline int interp_5_6_ideal(int c0, int c1) { assert(c0 < 256 && c1 < 256); return (c0 * 2 + c1) / 3; } + static inline int interp_5_6_ideal_round(int c0, int c1) { assert(c0 < 256 && c1 < 256); return (c0 * 2 + c1 + 1) / 3; } + static inline int interp_half_5_6_ideal(int c0, int c1) { assert(c0 < 256 && c1 < 256); return (c0 + c1) / 2; } + + static inline int interp_5_nv(int v0, int v1) { assert(v0 < 32 && v1 < 32); return ((2 * v0 + v1) * 22) / 8; } + static inline int interp_6_nv(int c0, int c1) { assert(c0 < 256 && c1 < 256); const int gdiff = c1 - c0; return (256 * c0 + (gdiff / 4) + 128 + gdiff * 80) / 256; } + + static inline int interp_half_5_nv(int v0, int v1) { assert(v0 < 32 && v1 < 32); return ((v0 + v1) * 33) / 8; } + static inline int interp_half_6_nv(int c0, int c1) { assert(c0 < 256 && c1 < 256); const int gdiff = c1 - c0; return (256 * c0 + gdiff / 4 + 128 + gdiff * 128) / 256; } + + static inline int interp_5_6_amd(int c0, int c1) { assert(c0 < 256 && c1 < 256); return (c0 * 43 + c1 * 21 + 32) >> 6; } + static inline int interp_half_5_6_amd(int c0, int c1) { assert(c0 < 256 && c1 < 256); return (c0 + c1 + 1) >> 1; } + + static inline int interp_5(int v0, int v1, int c0, int c1, bc1_approx_mode mode) + { + assert(scale_5_to_8(v0) == c0 && scale_5_to_8(v1) == c1); + switch (mode) + { + case bc1_approx_mode::cBC1NVidia: return interp_5_nv(v0, v1); + case bc1_approx_mode::cBC1AMD: return interp_5_6_amd(c0, c1); + default: + case bc1_approx_mode::cBC1Ideal: return interp_5_6_ideal(c0, c1); + case bc1_approx_mode::cBC1IdealRound4: return interp_5_6_ideal_round(c0, c1); + } + } + + static inline int interp_6(int v0, int v1, int c0, int c1, bc1_approx_mode mode) + { + (void)v0; (void)v1; + assert(scale_6_to_8(v0) == c0 && scale_6_to_8(v1) == c1); + switch (mode) + { + case bc1_approx_mode::cBC1NVidia: return interp_6_nv(c0, c1); + case bc1_approx_mode::cBC1AMD: return interp_5_6_amd(c0, c1); + default: + case bc1_approx_mode::cBC1Ideal: return interp_5_6_ideal(c0, c1); + case bc1_approx_mode::cBC1IdealRound4: return interp_5_6_ideal_round(c0, c1); + } + } + + static inline int interp_half_5(int v0, int v1, int c0, int c1, bc1_approx_mode mode) + { + assert(scale_5_to_8(v0) == c0 && scale_5_to_8(v1) == c1); + switch (mode) + { + case bc1_approx_mode::cBC1NVidia: return interp_half_5_nv(v0, v1); + case bc1_approx_mode::cBC1AMD: return interp_half_5_6_amd(c0, c1); + case bc1_approx_mode::cBC1Ideal: + case bc1_approx_mode::cBC1IdealRound4: + default: + return interp_half_5_6_ideal(c0, c1); + } + } + + static inline int interp_half_6(int v0, int v1, int c0, int c1, bc1_approx_mode mode) + { + (void)v0; (void)v1; + assert(scale_6_to_8(v0) == c0 && scale_6_to_8(v1) == c1); + switch (mode) + { + case bc1_approx_mode::cBC1NVidia: return interp_half_6_nv(c0, c1); + case bc1_approx_mode::cBC1AMD: return interp_half_5_6_amd(c0, c1); + case bc1_approx_mode::cBC1Ideal: + case bc1_approx_mode::cBC1IdealRound4: + default: + return interp_half_5_6_ideal(c0, c1); + } + } + + static void prepare_bc1_single_color_table_half(bc1_match_entry* pTable, const uint8_t* pExpand, int size, bc1_approx_mode mode) + { + for (int i = 0; i < 256; i++) + { + int lowest_e = 256; + for (int lo = 0; lo < size; lo++) + { + const int lo_e = pExpand[lo]; + + for (int hi = 0; hi < size; hi++) + { + const int hi_e = pExpand[hi]; + + const int v = (size == 32) ? interp_half_5(hi, lo, hi_e, lo_e, mode) : interp_half_6(hi, lo, hi_e, lo_e, mode); + + int e = iabs(v - i); + + // We only need to factor in 3% error in BC1 ideal mode. + if ((mode == bc1_approx_mode::cBC1Ideal) || (mode == bc1_approx_mode::cBC1IdealRound4)) + e += (iabs(hi_e - lo_e) * 3) / 100; + + // Favor equal endpoints, for lower error on actual GPU's which approximate the interpolation. + if ((e < lowest_e) || ((e == lowest_e) && (lo == hi))) + { + pTable[i].m_hi = static_cast(hi); + pTable[i].m_lo = static_cast(lo); + + assert(e <= UINT8_MAX); + pTable[i].m_e = static_cast(e); + + lowest_e = e; + } + + } // hi + } // lo + } + } + + static void prepare_bc1_single_color_table(bc1_match_entry* pTable, const uint8_t* pExpand, int size, bc1_approx_mode mode) + { + for (int i = 0; i < 256; i++) + { + int lowest_e = 256; + for (int lo = 0; lo < size; lo++) + { + const int lo_e = pExpand[lo]; + + for (int hi = 0; hi < size; hi++) + { + const int hi_e = pExpand[hi]; + + const int v = (size == 32) ? interp_5(hi, lo, hi_e, lo_e, mode) : interp_6(hi, lo, hi_e, lo_e, mode); + + int e = iabs(v - i); + + if ((mode == bc1_approx_mode::cBC1Ideal) || (mode == bc1_approx_mode::cBC1IdealRound4)) + e += (iabs(hi_e - lo_e) * 3) / 100; + + // Favor equal endpoints, for lower error on actual GPU's which approximate the interpolation. + if ((e < lowest_e) || ((e == lowest_e) && (lo == hi))) + { + pTable[i].m_hi = static_cast(hi); + pTable[i].m_lo = static_cast(lo); + + assert(e <= UINT8_MAX); + pTable[i].m_e = static_cast(e); + + lowest_e = e; + } + + } // hi + } // lo + } + } + + // This table is: 9 * (w * w), 9 * ((1.0f - w) * w), 9 * ((1.0f - w) * (1.0f - w)) + // where w is [0,1/3,2/3,1]. 9 is the perfect multiplier. + static const uint32_t g_weight_vals4[4] = { 0x000009, 0x010204, 0x040201, 0x090000 }; + + // multiplier is 4 for 3-color + static const uint32_t g_weight_vals3[3] = { 0x000004, 0x040000, 0x010101 }; + + static inline void compute_selector_factors4(const hist4& h, float& iz00, float& iz10, float& iz11) + { + uint32_t weight_accum = 0; + for (uint32_t sel = 0; sel < 4; sel++) + weight_accum += g_weight_vals4[sel] * h.m_hist[sel]; + + float z00 = (float)((weight_accum >> 16) & 0xFF); + float z10 = (float)((weight_accum >> 8) & 0xFF); + float z11 = (float)(weight_accum & 0xFF); + float z01 = z10; + + float det = z00 * z11 - z01 * z10; + if (fabs(det) < 1e-8f) + det = 0.0f; + else + det = (3.0f / 255.0f) / det; + + iz00 = z11 * det; + iz10 = -z10 * det; + iz11 = z00 * det; + } + + static inline void compute_selector_factors3(const hist3& h, float& iz00, float& iz10, float& iz11) + { + uint32_t weight_accum = 0; + for (uint32_t sel = 0; sel < 3; sel++) + weight_accum += g_weight_vals3[sel] * h.m_hist[sel]; + + float z00 = (float)((weight_accum >> 16) & 0xFF); + float z10 = (float)((weight_accum >> 8) & 0xFF); + float z11 = (float)(weight_accum & 0xFF); + float z01 = z10; + + float det = z00 * z11 - z01 * z10; + if (fabs(det) < 1e-8f) + det = 0.0f; + else + det = (2.0f / 255.0f) / det; + + iz00 = z11 * det; + iz10 = -z10 * det; + iz11 = z00 * det; + } + + static bool g_initialized; + + void init(bc1_approx_mode mode) + { + g_bc1_approx_mode = mode; + + uint8_t bc1_expand5[32]; + for (int i = 0; i < 32; i++) + bc1_expand5[i] = static_cast((i << 3) | (i >> 2)); + prepare_bc1_single_color_table(g_bc1_match5_equals_1, bc1_expand5, 32, mode); + prepare_bc1_single_color_table_half(g_bc1_match5_half, bc1_expand5, 32, mode); + + uint8_t bc1_expand6[64]; + for (int i = 0; i < 64; i++) + bc1_expand6[i] = static_cast((i << 2) | (i >> 4)); + prepare_bc1_single_color_table(g_bc1_match6_equals_1, bc1_expand6, 64, mode); + prepare_bc1_single_color_table_half(g_bc1_match6_half, bc1_expand6, 64, mode); + + for (uint32_t i = 0; i < NUM_UNIQUE_TOTAL_ORDERINGS4; i++) + { + hist4 h; + h.m_hist[0] = (uint8_t)g_unique_total_orders4[i][0]; + h.m_hist[1] = (uint8_t)g_unique_total_orders4[i][1]; + h.m_hist[2] = (uint8_t)g_unique_total_orders4[i][2]; + h.m_hist[3] = (uint8_t)g_unique_total_orders4[i][3]; + + if (!h.any_16()) + { + const uint32_t index = h.m_hist[0] | (h.m_hist[1] << 4) | (h.m_hist[2] << 8); + assert(index < 4096); + g_total_ordering4_hash[index] = (uint16_t)i; + } + + compute_selector_factors4(h, g_selector_factors4[i][0], g_selector_factors4[i][1], g_selector_factors4[i][2]); + } + + for (uint32_t i = 0; i < NUM_UNIQUE_TOTAL_ORDERINGS3; i++) + { + hist3 h; + h.m_hist[0] = (uint8_t)g_unique_total_orders3[i][0]; + h.m_hist[1] = (uint8_t)g_unique_total_orders3[i][1]; + h.m_hist[2] = (uint8_t)g_unique_total_orders3[i][2]; + + if (!h.any_16()) + { + const uint32_t index = h.m_hist[0] | (h.m_hist[1] << 4); + assert(index < 256); + g_total_ordering3_hash[index] = (uint16_t)i; + } + + compute_selector_factors3(h, g_selector_factors3[i][0], g_selector_factors3[i][1], g_selector_factors3[i][2]); + } + + g_initialized = true; + } + + void encode_bc1_solid_block(void* pDst, uint32_t fr, uint32_t fg, uint32_t fb, bool allow_3color) + { + bc1_block* pDst_block = static_cast(pDst); + + uint32_t mask = 0xAA; + int max16 = -1, min16 = 0; + + if (allow_3color) + { + const uint32_t err4 = g_bc1_match5_equals_1[fr].m_e + g_bc1_match6_equals_1[fg].m_e + g_bc1_match5_equals_1[fb].m_e; + const uint32_t err3 = g_bc1_match5_half[fr].m_e + g_bc1_match6_half[fg].m_e + g_bc1_match5_half[fb].m_e; + + if (err3 < err4) + { + max16 = (g_bc1_match5_half[fr].m_hi << 11) | (g_bc1_match6_half[fg].m_hi << 5) | g_bc1_match5_half[fb].m_hi; + min16 = (g_bc1_match5_half[fr].m_lo << 11) | (g_bc1_match6_half[fg].m_lo << 5) | g_bc1_match5_half[fb].m_lo; + + if (max16 > min16) + std::swap(max16, min16); + } + } + + if (max16 == -1) + { + max16 = (g_bc1_match5_equals_1[fr].m_hi << 11) | (g_bc1_match6_equals_1[fg].m_hi << 5) | g_bc1_match5_equals_1[fb].m_hi; + min16 = (g_bc1_match5_equals_1[fr].m_lo << 11) | (g_bc1_match6_equals_1[fg].m_lo << 5) | g_bc1_match5_equals_1[fb].m_lo; + + if (min16 == max16) + { + // Always forbid 3 color blocks + // This is to guarantee that BC3 blocks never use punchthrough alpha (3 color) mode, which isn't supported on some (all?) GPU's. + mask = 0; + + // Make l > h + if (min16 > 0) + min16--; + else + { + // l = h = 0 + assert(min16 == max16 && max16 == 0); + + max16 = 1; + min16 = 0; + mask = 0x55; + } + + assert(max16 > min16); + } + + if (max16 < min16) + { + std::swap(max16, min16); + mask ^= 0x55; + } + } + + pDst_block->set_low_color(static_cast(max16)); + pDst_block->set_high_color(static_cast(min16)); + pDst_block->m_selectors[0] = static_cast(mask); + pDst_block->m_selectors[1] = static_cast(mask); + pDst_block->m_selectors[2] = static_cast(mask); + pDst_block->m_selectors[3] = static_cast(mask); + } + + static const float g_midpoint5[32] = { .015686f, .047059f, .078431f, .111765f, .145098f, .176471f, .207843f, .241176f, .274510f, .305882f, .337255f, .370588f, .403922f, .435294f, .466667f, .5f, .533333f, .564706f, .596078f, .629412f, .662745f, .694118f, .725490f, .758824f, .792157f, .823529f, .854902f, .888235f, .921569f, .952941f, .984314f, 1e+37f }; + static const float g_midpoint6[64] = { .007843f, .023529f, .039216f, .054902f, .070588f, .086275f, .101961f, .117647f, .133333f, .149020f, .164706f, .180392f, .196078f, .211765f, .227451f, .245098f, .262745f, .278431f, .294118f, .309804f, .325490f, .341176f, .356863f, .372549f, .388235f, .403922f, .419608f, .435294f, .450980f, .466667f, .482353f, .500000f, .517647f, .533333f, .549020f, .564706f, .580392f, .596078f, .611765f, .627451f, .643137f, .658824f, .674510f, .690196f, .705882f, .721569f, .737255f, .754902f, .772549f, .788235f, .803922f, .819608f, .835294f, .850980f, .866667f, .882353f, .898039f, .913725f, .929412f, .945098f, .960784f, .976471f, .992157f, 1e+37f }; + + struct vec3F { float c[3]; }; + + static inline void compute_least_squares_endpoints4_rgb( + vec3F* pXl, vec3F* pXh, + int total_r, int total_g, int total_b, + float iz00, float iz10, float iz11, + uint32_t s, const uint32_t r_sum[17], const uint32_t g_sum[17], const uint32_t b_sum[17]) + { + const float iz01 = iz10; + + const uint32_t f1 = g_unique_total_orders4[s][0]; + const uint32_t f2 = g_unique_total_orders4[s][0] + g_unique_total_orders4[s][1]; + const uint32_t f3 = g_unique_total_orders4[s][0] + g_unique_total_orders4[s][1] + g_unique_total_orders4[s][2]; + uint32_t uq00_r = (r_sum[f2] - r_sum[f1]) + (r_sum[f3] - r_sum[f2]) * 2 + (r_sum[16] - r_sum[f3]) * 3; + uint32_t uq00_g = (g_sum[f2] - g_sum[f1]) + (g_sum[f3] - g_sum[f2]) * 2 + (g_sum[16] - g_sum[f3]) * 3; + uint32_t uq00_b = (b_sum[f2] - b_sum[f1]) + (b_sum[f3] - b_sum[f2]) * 2 + (b_sum[16] - b_sum[f3]) * 3; + + float q10_r = (float)(total_r * 3 - uq00_r); + float q10_g = (float)(total_g * 3 - uq00_g); + float q10_b = (float)(total_b * 3 - uq00_b); + + pXl->c[0] = iz00 * (float)uq00_r + iz01 * q10_r; + pXh->c[0] = iz10 * (float)uq00_r + iz11 * q10_r; + + pXl->c[1] = iz00 * (float)uq00_g + iz01 * q10_g; + pXh->c[1] = iz10 * (float)uq00_g + iz11 * q10_g; + + pXl->c[2] = iz00 * (float)uq00_b + iz01 * q10_b; + pXh->c[2] = iz10 * (float)uq00_b + iz11 * q10_b; + } + + static inline bool compute_least_squares_endpoints4_rgb(const color32* pColors, const uint8_t* pSelectors, vec3F* pXl, vec3F* pXh, int total_r, int total_g, int total_b) + { + uint32_t uq00_r = 0, uq00_g = 0, uq00_b = 0; + uint32_t weight_accum = 0; + for (uint32_t i = 0; i < 16; i++) + { + const uint8_t r = pColors[i].c[0], g = pColors[i].c[1], b = pColors[i].c[2]; + const uint8_t sel = pSelectors[i]; + + weight_accum += g_weight_vals4[sel]; + uq00_r += sel * r; + uq00_g += sel * g; + uq00_b += sel * b; + } + + int q10_r = total_r * 3 - uq00_r; + int q10_g = total_g * 3 - uq00_g; + int q10_b = total_b * 3 - uq00_b; + + float z00 = (float)((weight_accum >> 16) & 0xFF); + float z10 = (float)((weight_accum >> 8) & 0xFF); + float z11 = (float)(weight_accum & 0xFF); + float z01 = z10; + + float det = z00 * z11 - z01 * z10; + if (fabs(det) < 1e-8f) + return false; + + det = (3.0f / 255.0f) / det; + + float iz00, iz01, iz10, iz11; + iz00 = z11 * det; + iz01 = -z01 * det; + iz10 = -z10 * det; + iz11 = z00 * det; + + pXl->c[0] = iz00 * (float)uq00_r + iz01 * q10_r; + pXh->c[0] = iz10 * (float)uq00_r + iz11 * q10_r; + + pXl->c[1] = iz00 * (float)uq00_g + iz01 * q10_g; + pXh->c[1] = iz10 * (float)uq00_g + iz11 * q10_g; + + pXl->c[2] = iz00 * (float)uq00_b + iz01 * q10_b; + pXh->c[2] = iz10 * (float)uq00_b + iz11 * q10_b; + + return true; + } + + static inline void compute_least_squares_endpoints3_rgb( + vec3F* pXl, vec3F* pXh, + int total_r, int total_g, int total_b, + float iz00, float iz10, float iz11, + uint32_t s, const uint32_t r_sum[17], const uint32_t g_sum[17], const uint32_t b_sum[17]) + { + const float iz01 = iz10; + + // Compensates for BC1 3-color ordering, which is selector 0, 2, 1 + const uint32_t f1 = g_unique_total_orders3[s][0]; + const uint32_t f2 = g_unique_total_orders3[s][0] + g_unique_total_orders3[s][2]; + uint32_t uq00_r = (r_sum[16] - r_sum[f2]) * 2 + (r_sum[f2] - r_sum[f1]); + uint32_t uq00_g = (g_sum[16] - g_sum[f2]) * 2 + (g_sum[f2] - g_sum[f1]); + uint32_t uq00_b = (b_sum[16] - b_sum[f2]) * 2 + (b_sum[f2] - b_sum[f1]); + + float q10_r = (float)(total_r * 2 - uq00_r); + float q10_g = (float)(total_g * 2 - uq00_g); + float q10_b = (float)(total_b * 2 - uq00_b); + + pXl->c[0] = iz00 * (float)uq00_r + iz01 * q10_r; + pXh->c[0] = iz10 * (float)uq00_r + iz11 * q10_r; + + pXl->c[1] = iz00 * (float)uq00_g + iz01 * q10_g; + pXh->c[1] = iz10 * (float)uq00_g + iz11 * q10_g; + + pXl->c[2] = iz00 * (float)uq00_b + iz01 * q10_b; + pXh->c[2] = iz10 * (float)uq00_b + iz11 * q10_b; + } + + static inline bool compute_least_squares_endpoints3_rgb(bool use_black, const color32* pColors, const uint8_t* pSelectors, vec3F* pXl, vec3F* pXh) + { + int uq00_r = 0, uq00_g = 0, uq00_b = 0; + uint32_t weight_accum = 0; + int total_r = 0, total_g = 0, total_b = 0; + for (uint32_t i = 0; i < 16; i++) + { + const uint8_t r = pColors[i].c[0], g = pColors[i].c[1], b = pColors[i].c[2]; + if (use_black) + { + if ((r | g | b) < 4) + continue; + } + + const uint8_t sel = pSelectors[i]; + assert(sel <= 3); + if (sel == 3) + continue; + + weight_accum += g_weight_vals3[sel]; + + static const uint8_t s_tran[3] = { 0, 2, 1 }; + const uint8_t tsel = s_tran[sel]; + uq00_r += tsel * r; + uq00_g += tsel * g; + uq00_b += tsel * b; + + total_r += r; + total_g += g; + total_b += b; + } + + int q10_r = total_r * 2 - uq00_r; + int q10_g = total_g * 2 - uq00_g; + int q10_b = total_b * 2 - uq00_b; + + float z00 = (float)((weight_accum >> 16) & 0xFF); + float z10 = (float)((weight_accum >> 8) & 0xFF); + float z11 = (float)(weight_accum & 0xFF); + float z01 = z10; + + float det = z00 * z11 - z01 * z10; + if (fabs(det) < 1e-8f) + return false; + + det = (2.0f / 255.0f) / det; + + float iz00, iz01, iz10, iz11; + iz00 = z11 * det; + iz01 = -z01 * det; + iz10 = -z10 * det; + iz11 = z00 * det; + + pXl->c[0] = iz00 * (float)uq00_r + iz01 * q10_r; + pXh->c[0] = iz10 * (float)uq00_r + iz11 * q10_r; + + pXl->c[1] = iz00 * (float)uq00_g + iz01 * q10_g; + pXh->c[1] = iz10 * (float)uq00_g + iz11 * q10_g; + + pXl->c[2] = iz00 * (float)uq00_b + iz01 * q10_b; + pXh->c[2] = iz10 * (float)uq00_b + iz11 * q10_b; + + return true; + } + + static inline void bc1_get_block_colors4(uint32_t block_r[4], uint32_t block_g[4], uint32_t block_b[4], uint32_t lr, uint32_t lg, uint32_t lb, uint32_t hr, uint32_t hg, uint32_t hb) + { + block_r[0] = (lr << 3) | (lr >> 2); block_g[0] = (lg << 2) | (lg >> 4); block_b[0] = (lb << 3) | (lb >> 2); + block_r[3] = (hr << 3) | (hr >> 2); block_g[3] = (hg << 2) | (hg >> 4); block_b[3] = (hb << 3) | (hb >> 2); + + if (g_bc1_approx_mode == bc1_approx_mode::cBC1Ideal) + { + block_r[1] = (block_r[0] * 2 + block_r[3]) / 3; block_g[1] = (block_g[0] * 2 + block_g[3]) / 3; block_b[1] = (block_b[0] * 2 + block_b[3]) / 3; + block_r[2] = (block_r[3] * 2 + block_r[0]) / 3; block_g[2] = (block_g[3] * 2 + block_g[0]) / 3; block_b[2] = (block_b[3] * 2 + block_b[0]) / 3; + } + else if (g_bc1_approx_mode == bc1_approx_mode::cBC1IdealRound4) + { + block_r[1] = (block_r[0] * 2 + block_r[3] + 1) / 3; block_g[1] = (block_g[0] * 2 + block_g[3] + 1) / 3; block_b[1] = (block_b[0] * 2 + block_b[3] + 1) / 3; + block_r[2] = (block_r[3] * 2 + block_r[0] + 1) / 3; block_g[2] = (block_g[3] * 2 + block_g[0] + 1) / 3; block_b[2] = (block_b[3] * 2 + block_b[0] + 1) / 3; + } + else if (g_bc1_approx_mode == bc1_approx_mode::cBC1AMD) + { + block_r[1] = interp_5_6_amd(block_r[0], block_r[3]); block_g[1] = interp_5_6_amd(block_g[0], block_g[3]); block_b[1] = interp_5_6_amd(block_b[0], block_b[3]); + block_r[2] = interp_5_6_amd(block_r[3], block_r[0]); block_g[2] = interp_5_6_amd(block_g[3], block_g[0]); block_b[2] = interp_5_6_amd(block_b[3], block_b[0]); + } + else + { + block_r[1] = interp_5_nv(lr, hr); block_g[1] = interp_6_nv(block_g[0], block_g[3]); block_b[1] = interp_5_nv(lb, hb); + block_r[2] = interp_5_nv(hr, lr); block_g[2] = interp_6_nv(block_g[3], block_g[0]); block_b[2] = interp_5_nv(hb, lb); + } + } + + static inline void bc1_get_block_colors3(uint32_t block_r[3], uint32_t block_g[3], uint32_t block_b[3], uint32_t lr, uint32_t lg, uint32_t lb, uint32_t hr, uint32_t hg, uint32_t hb) + { + block_r[0] = (lr << 3) | (lr >> 2); block_g[0] = (lg << 2) | (lg >> 4); block_b[0] = (lb << 3) | (lb >> 2); + block_r[1] = (hr << 3) | (hr >> 2); block_g[1] = (hg << 2) | (hg >> 4); block_b[1] = (hb << 3) | (hb >> 2); + + if ((g_bc1_approx_mode == bc1_approx_mode::cBC1Ideal) || (g_bc1_approx_mode == bc1_approx_mode::cBC1IdealRound4)) + { + block_r[2] = (block_r[0] + block_r[1]) / 2; block_g[2] = (block_g[0] + block_g[1]) / 2; block_b[2] = (block_b[0] + block_b[1]) / 2; + } + else if (g_bc1_approx_mode == bc1_approx_mode::cBC1AMD) + { + block_r[2] = interp_half_5_6_amd(block_r[0], block_r[1]); block_g[2] = interp_half_5_6_amd(block_g[0], block_g[1]); block_b[2] = interp_half_5_6_amd(block_b[0], block_b[1]); + } + else + { + block_r[2] = interp_half_5_nv(lr, hr); block_g[2] = interp_half_6_nv(block_g[0], block_g[1]); block_b[2] = interp_half_5_nv(lb, hb); + } + } + + static inline void bc1_find_sels4_noerr(const color32* pSrc_pixels, uint32_t lr, uint32_t lg, uint32_t lb, uint32_t hr, uint32_t hg, uint32_t hb, uint8_t sels[16], const uint8_t* pForce_selectors) + { + if (pForce_selectors) + { + memcpy(sels, pForce_selectors, 16); + return; + } + + uint32_t block_r[4], block_g[4], block_b[4]; + bc1_get_block_colors4(block_r, block_g, block_b, lr, lg, lb, hr, hg, hb); + + int ar = block_r[3] - block_r[0], ag = block_g[3] - block_g[0], ab = block_b[3] - block_b[0]; + + int dots[4]; + for (uint32_t i = 0; i < 4; i++) + dots[i] = (int)block_r[i] * ar + (int)block_g[i] * ag + (int)block_b[i] * ab; + + int t0 = dots[0] + dots[1], t1 = dots[1] + dots[2], t2 = dots[2] + dots[3]; + + ar *= 2; ag *= 2; ab *= 2; + + static const uint8_t s_sels[4] = { 3, 2, 1, 0 }; + + for (uint32_t i = 0; i < 16; i += 4) + { + const int d0 = pSrc_pixels[i + 0].r * ar + pSrc_pixels[i + 0].g * ag + pSrc_pixels[i + 0].b * ab; + const int d1 = pSrc_pixels[i + 1].r * ar + pSrc_pixels[i + 1].g * ag + pSrc_pixels[i + 1].b * ab; + const int d2 = pSrc_pixels[i + 2].r * ar + pSrc_pixels[i + 2].g * ag + pSrc_pixels[i + 2].b * ab; + const int d3 = pSrc_pixels[i + 3].r * ar + pSrc_pixels[i + 3].g * ag + pSrc_pixels[i + 3].b * ab; + + sels[i + 0] = s_sels[(d0 <= t0) + (d0 < t1) + (d0 < t2)]; + sels[i + 1] = s_sels[(d1 <= t0) + (d1 < t1) + (d1 < t2)]; + sels[i + 2] = s_sels[(d2 <= t0) + (d2 < t1) + (d2 < t2)]; + sels[i + 3] = s_sels[(d3 <= t0) + (d3 < t1) + (d3 < t2)]; + } + } + + static inline uint32_t bc1_find_sels4_fasterr(const color32* pSrc_pixels, uint32_t lr, uint32_t lg, uint32_t lb, uint32_t hr, uint32_t hg, uint32_t hb, uint8_t sels[16], uint32_t cur_err) + { + uint32_t block_r[4], block_g[4], block_b[4]; + bc1_get_block_colors4(block_r, block_g, block_b, lr, lg, lb, hr, hg, hb); + + int ar = block_r[3] - block_r[0], ag = block_g[3] - block_g[0], ab = block_b[3] - block_b[0]; + + int dots[4]; + for (uint32_t i = 0; i < 4; i++) + dots[i] = (int)block_r[i] * ar + (int)block_g[i] * ag + (int)block_b[i] * ab; + + int t0 = dots[0] + dots[1], t1 = dots[1] + dots[2], t2 = dots[2] + dots[3]; + + ar *= 2; ag *= 2; ab *= 2; + + static const uint8_t s_sels[4] = { 3, 2, 1, 0 }; + + uint32_t total_err = 0; + + for (uint32_t i = 0; i < 16; i += 4) + { + const int d0 = pSrc_pixels[i + 0].r * ar + pSrc_pixels[i + 0].g * ag + pSrc_pixels[i + 0].b * ab; + const int d1 = pSrc_pixels[i + 1].r * ar + pSrc_pixels[i + 1].g * ag + pSrc_pixels[i + 1].b * ab; + const int d2 = pSrc_pixels[i + 2].r * ar + pSrc_pixels[i + 2].g * ag + pSrc_pixels[i + 2].b * ab; + const int d3 = pSrc_pixels[i + 3].r * ar + pSrc_pixels[i + 3].g * ag + pSrc_pixels[i + 3].b * ab; + + uint8_t sel0 = s_sels[(d0 <= t0) + (d0 < t1) + (d0 < t2)]; + uint8_t sel1 = s_sels[(d1 <= t0) + (d1 < t1) + (d1 < t2)]; + uint8_t sel2 = s_sels[(d2 <= t0) + (d2 < t1) + (d2 < t2)]; + uint8_t sel3 = s_sels[(d3 <= t0) + (d3 < t1) + (d3 < t2)]; + + sels[i + 0] = sel0; + sels[i + 1] = sel1; + sels[i + 2] = sel2; + sels[i + 3] = sel3; + + total_err += squarei(pSrc_pixels[i + 0].r - block_r[sel0]) + squarei(pSrc_pixels[i + 0].g - block_g[sel0]) + squarei(pSrc_pixels[i + 0].b - block_b[sel0]); + total_err += squarei(pSrc_pixels[i + 1].r - block_r[sel1]) + squarei(pSrc_pixels[i + 1].g - block_g[sel1]) + squarei(pSrc_pixels[i + 1].b - block_b[sel1]); + total_err += squarei(pSrc_pixels[i + 2].r - block_r[sel2]) + squarei(pSrc_pixels[i + 2].g - block_g[sel2]) + squarei(pSrc_pixels[i + 2].b - block_b[sel2]); + total_err += squarei(pSrc_pixels[i + 3].r - block_r[sel3]) + squarei(pSrc_pixels[i + 3].g - block_g[sel3]) + squarei(pSrc_pixels[i + 3].b - block_b[sel3]); + + if (total_err >= cur_err) + break; + } + + return total_err; + } + + static inline uint32_t bc1_find_sels4_check2_err(const color32* pSrc_pixels, uint32_t lr, uint32_t lg, uint32_t lb, uint32_t hr, uint32_t hg, uint32_t hb, uint8_t sels[16], uint32_t cur_err) + { + uint32_t block_r[4], block_g[4], block_b[4]; + bc1_get_block_colors4(block_r, block_g, block_b, lr, lg, lb, hr, hg, hb); + + int dr = block_r[3] - block_r[0], dg = block_g[3] - block_g[0], db = block_b[3] - block_b[0]; + + const float f = 4.0f / (float)(squarei(dr) + squarei(dg) + squarei(db) + .00000125f); + + uint32_t total_err = 0; + + for (uint32_t i = 0; i < 16; i++) + { + const int r = pSrc_pixels[i].r; + const int g = pSrc_pixels[i].g; + const int b = pSrc_pixels[i].b; + + int sel = (int)((float)((r - (int)block_r[0]) * dr + (g - (int)block_g[0]) * dg + (b - (int)block_b[0]) * db) * f + .5f); + sel = clampi(sel, 1, 3); + + uint32_t err0 = squarei((int)block_r[sel - 1] - (int)r) + squarei((int)block_g[sel - 1] - (int)g) + squarei((int)block_b[sel - 1] - (int)b); + uint32_t err1 = squarei((int)block_r[sel] - (int)r) + squarei((int)block_g[sel] - (int)g) + squarei((int)block_b[sel] - (int)b); + + int best_sel = sel; + uint32_t best_err = err1; + if (err0 == err1) + { + // Prefer non-interpolation + if ((best_sel - 1) == 0) + best_sel = 0; + } + else if (err0 < best_err) + { + best_sel = sel - 1; + best_err = err0; + } + + total_err += best_err; + + if (total_err >= cur_err) + break; + + sels[i] = (uint8_t)best_sel; + } + return total_err; + } + + static inline uint32_t bc1_find_sels4_fullerr(const color32* pSrc_pixels, uint32_t lr, uint32_t lg, uint32_t lb, uint32_t hr, uint32_t hg, uint32_t hb, uint8_t sels[16], uint32_t cur_err) + { + uint32_t block_r[4], block_g[4], block_b[4]; + bc1_get_block_colors4(block_r, block_g, block_b, lr, lg, lb, hr, hg, hb); + + uint32_t total_err = 0; + + for (uint32_t i = 0; i < 16; i++) + { + const int r = pSrc_pixels[i].r; + const int g = pSrc_pixels[i].g; + const int b = pSrc_pixels[i].b; + + uint32_t best_err = squarei((int)block_r[0] - (int)r) + squarei((int)block_g[0] - (int)g) + squarei((int)block_b[0] - (int)b); + uint8_t best_sel = 0; + + for (uint32_t j = 1; (j < 4) && best_err; j++) + { + uint32_t err = squarei((int)block_r[j] - (int)r) + squarei((int)block_g[j] - (int)g) + squarei((int)block_b[j] - (int)b); + if ((err < best_err) || ((err == best_err) && (j == 3))) + { + best_err = err; + best_sel = (uint8_t)j; + } + } + + total_err += best_err; + + if (total_err >= cur_err) + break; + + sels[i] = (uint8_t)best_sel; + } + return total_err; + } + + static inline uint32_t bc1_find_sels4(uint32_t flags, const color32* pSrc_pixels, uint32_t lr, uint32_t lg, uint32_t lb, uint32_t hr, uint32_t hg, uint32_t hb, uint8_t sels[16], uint32_t cur_err, const uint8_t* pForce_selectors) + { + uint32_t err; + + if (pForce_selectors) + { + memcpy(sels, pForce_selectors, 16); + + uint32_t block_r[4], block_g[4], block_b[4]; + bc1_get_block_colors4(block_r, block_g, block_b, lr, lg, lb, hr, hg, hb); + + err = 0; + for (uint32_t i = 0; i < 16; i++) + { + const int r = pSrc_pixels[i].r; + const int g = pSrc_pixels[i].g; + const int b = pSrc_pixels[i].b; + + const uint32_t sel = pForce_selectors[i]; + assert(sel <= 3); + + err += squarei((int)block_r[sel] - (int)r) + squarei((int)block_g[sel] - (int)g) + squarei((int)block_b[sel] - (int)b); + } + } + else + { + if (flags & cEncodeBC1UseFasterMSEEval) + err = bc1_find_sels4_fasterr(pSrc_pixels, lr, lg, lb, hr, hg, hb, sels, cur_err); + else if (flags & cEncodeBC1UseFullMSEEval) + err = bc1_find_sels4_fullerr(pSrc_pixels, lr, lg, lb, hr, hg, hb, sels, cur_err); + else + err = bc1_find_sels4_check2_err(pSrc_pixels, lr, lg, lb, hr, hg, hb, sels, cur_err); + } + + return err; + } + + static inline uint32_t bc1_find_sels3_fullerr(bool use_black, const color32* pSrc_pixels, uint32_t lr, uint32_t lg, uint32_t lb, uint32_t hr, uint32_t hg, uint32_t hb, uint8_t sels[16], uint32_t cur_err, const uint8_t* pForce_selectors) + { + uint32_t block_r[4], block_g[4], block_b[4]; + bc1_get_block_colors3(block_r, block_g, block_b, lr, lg, lb, hr, hg, hb); + + uint32_t total_err = 0; + + if (pForce_selectors) + { + memcpy(sels, pForce_selectors, 16); + + //uint32_t block_r[4], block_g[4], block_b[4]; + //bc1_get_block_colors3(block_r, block_g, block_b, lr, lg, lb, hr, hg, hb); + + block_r[3] = 0; block_g[3] = 0; block_b[3] = 0; + + for (uint32_t i = 0; i < 16; i++) + { + const int r = pSrc_pixels[i].r; + const int g = pSrc_pixels[i].g; + const int b = pSrc_pixels[i].b; + + const uint32_t sel = pForce_selectors[i]; + assert(sel <= 3); + + total_err += squarei((int)block_r[sel] - (int)r) + squarei((int)block_g[sel] - (int)g) + squarei((int)block_b[sel] - (int)b); + } + } + else + { + for (uint32_t i = 0; i < 16; i++) + { + const int r = pSrc_pixels[i].r; + const int g = pSrc_pixels[i].g; + const int b = pSrc_pixels[i].b; + + uint32_t best_err = squarei((int)block_r[0] - (int)r) + squarei((int)block_g[0] - (int)g) + squarei((int)block_b[0] - (int)b); + uint32_t best_sel = 0; + + uint32_t err1 = squarei((int)block_r[1] - (int)r) + squarei((int)block_g[1] - (int)g) + squarei((int)block_b[1] - (int)b); + if (err1 < best_err) + { + best_err = err1; + best_sel = 1; + } + + uint32_t err2 = squarei((int)block_r[2] - (int)r) + squarei((int)block_g[2] - (int)g) + squarei((int)block_b[2] - (int)b); + if (err2 < best_err) + { + best_err = err2; + best_sel = 2; + } + + if (use_black) + { + uint32_t err3 = squarei(r) + squarei(g) + squarei(b); + if (err3 < best_err) + { + best_err = err3; + best_sel = 3; + } + } + + total_err += best_err; + if (total_err >= cur_err) + return total_err; + + sels[i] = (uint8_t)best_sel; + } + } + + return total_err; + } + + static inline void precise_round_565(const vec3F& xl, const vec3F& xh, + int& trial_lr, int& trial_lg, int& trial_lb, + int& trial_hr, int& trial_hg, int& trial_hb) + { + trial_lr = (int)(xl.c[0] * 31.0f); + trial_lg = (int)(xl.c[1] * 63.0f); + trial_lb = (int)(xl.c[2] * 31.0f); + + trial_hr = (int)(xh.c[0] * 31.0f); + trial_hg = (int)(xh.c[1] * 63.0f); + trial_hb = (int)(xh.c[2] * 31.0f); + + if ((uint32_t)(trial_lr | trial_lb | trial_hr | trial_hb) > 31U) + { + trial_lr = ((uint32_t)trial_lr > 31U) ? (~trial_lr >> 31) & 31 : trial_lr; + trial_hr = ((uint32_t)trial_hr > 31U) ? (~trial_hr >> 31) & 31 : trial_hr; + + trial_lb = ((uint32_t)trial_lb > 31U) ? (~trial_lb >> 31) & 31 : trial_lb; + trial_hb = ((uint32_t)trial_hb > 31U) ? (~trial_hb >> 31) & 31 : trial_hb; + } + + if ((uint32_t)(trial_lg | trial_hg) > 63U) + { + trial_lg = ((uint32_t)trial_lg > 63U) ? (~trial_lg >> 31) & 63 : trial_lg; + trial_hg = ((uint32_t)trial_hg > 63U) ? (~trial_hg >> 31) & 63 : trial_hg; + } + + trial_lr = (trial_lr + (xl.c[0] > g_midpoint5[trial_lr])) & 31; + trial_lg = (trial_lg + (xl.c[1] > g_midpoint6[trial_lg])) & 63; + trial_lb = (trial_lb + (xl.c[2] > g_midpoint5[trial_lb])) & 31; + + trial_hr = (trial_hr + (xh.c[0] > g_midpoint5[trial_hr])) & 31; + trial_hg = (trial_hg + (xh.c[1] > g_midpoint6[trial_hg])) & 63; + trial_hb = (trial_hb + (xh.c[2] > g_midpoint5[trial_hb])) & 31; + } + + static inline void precise_round_565_noscale(vec3F xl, vec3F xh, + int& trial_lr, int& trial_lg, int& trial_lb, + int& trial_hr, int& trial_hg, int& trial_hb) + { + xl.c[0] *= 1.0f / 255.0f; + xl.c[1] *= 1.0f / 255.0f; + xl.c[2] *= 1.0f / 255.0f; + + xh.c[0] *= 1.0f / 255.0f; + xh.c[1] *= 1.0f / 255.0f; + xh.c[2] *= 1.0f / 255.0f; + + precise_round_565(xl, xh, trial_lr, trial_lg, trial_lb, trial_hr, trial_hg, trial_hb); + } + + static inline void bc1_encode4(bc1_block* pDst_block, int lr, int lg, int lb, int hr, int hg, int hb, const uint8_t sels[16]) + { + uint32_t lc16 = bc1_block::pack_unscaled_color(lr, lg, lb); + uint32_t hc16 = bc1_block::pack_unscaled_color(hr, hg, hb); + + // Always forbid 3 color blocks + if (lc16 == hc16) + { + uint8_t mask = 0; + + // Make l > h + if (hc16 > 0) + hc16--; + else + { + // lc16 = hc16 = 0 + assert(lc16 == hc16 && hc16 == 0); + + hc16 = 0; + lc16 = 1; + mask = 0x55; // select hc16 + } + + assert(lc16 > hc16); + pDst_block->set_low_color(static_cast(lc16)); + pDst_block->set_high_color(static_cast(hc16)); + + pDst_block->m_selectors[0] = mask; + pDst_block->m_selectors[1] = mask; + pDst_block->m_selectors[2] = mask; + pDst_block->m_selectors[3] = mask; + } + else + { + uint8_t invert_mask = 0; + if (lc16 < hc16) + { + std::swap(lc16, hc16); + invert_mask = 0x55; + } + + assert(lc16 > hc16); + pDst_block->set_low_color((uint16_t)lc16); + pDst_block->set_high_color((uint16_t)hc16); + + uint32_t packed_sels = 0; + static const uint8_t s_sel_trans[4] = { 0, 2, 3, 1 }; + for (uint32_t i = 0; i < 16; i++) + packed_sels |= ((uint32_t)s_sel_trans[sels[i]] << (i * 2)); + + pDst_block->m_selectors[0] = (uint8_t)packed_sels ^ invert_mask; + pDst_block->m_selectors[1] = (uint8_t)(packed_sels >> 8) ^ invert_mask; + pDst_block->m_selectors[2] = (uint8_t)(packed_sels >> 16) ^ invert_mask; + pDst_block->m_selectors[3] = (uint8_t)(packed_sels >> 24) ^ invert_mask; + } + } + + static inline void bc1_encode3(bc1_block* pDst_block, int lr, int lg, int lb, int hr, int hg, int hb, const uint8_t sels[16]) + { + uint32_t lc16 = bc1_block::pack_unscaled_color(lr, lg, lb); + uint32_t hc16 = bc1_block::pack_unscaled_color(hr, hg, hb); + + bool invert_flag = false; + if (lc16 > hc16) + { + std::swap(lc16, hc16); + invert_flag = true; + } + + assert(lc16 <= hc16); + + pDst_block->set_low_color((uint16_t)lc16); + pDst_block->set_high_color((uint16_t)hc16); + + uint32_t packed_sels = 0; + + if (invert_flag) + { + static const uint8_t s_sel_trans_inv[4] = { 1, 0, 2, 3 }; + + for (uint32_t i = 0; i < 16; i++) + packed_sels |= ((uint32_t)s_sel_trans_inv[sels[i]] << (i * 2)); + } + else + { + for (uint32_t i = 0; i < 16; i++) + packed_sels |= ((uint32_t)sels[i] << (i * 2)); + } + + pDst_block->m_selectors[0] = (uint8_t)packed_sels; + pDst_block->m_selectors[1] = (uint8_t)(packed_sels >> 8); + pDst_block->m_selectors[2] = (uint8_t)(packed_sels >> 16); + pDst_block->m_selectors[3] = (uint8_t)(packed_sels >> 24); + } + + struct bc1_encode_results + { + int lr, lg, lb; + int hr, hg, hb; + uint8_t sels[16]; + bool m_3color; + }; + + static bool try_3color_block_useblack(const color32* pSrc_pixels, uint32_t flags, uint32_t& cur_err, bc1_encode_results& results, const uint8_t* pForce_selectors) + { + int total_r = 0, total_g = 0, total_b = 0; + int max_r = 0, max_g = 0, max_b = 0; + int min_r = 255, min_g = 255, min_b = 255; + int total_pixels = 0; + for (uint32_t i = 0; i < 16; i++) + { + const int r = pSrc_pixels[i].r, g = pSrc_pixels[i].g, b = pSrc_pixels[i].b; + if ((r | g | b) < 4) + continue; + + max_r = std::max(max_r, r); max_g = std::max(max_g, g); max_b = std::max(max_b, b); + min_r = std::min(min_r, r); min_g = std::min(min_g, g); min_b = std::min(min_b, b); + total_r += r; total_g += g; total_b += b; + + total_pixels++; + } + + if (!total_pixels) + return false; + + int half_total_pixels = total_pixels >> 1; + int avg_r = (total_r + half_total_pixels) / total_pixels; + int avg_g = (total_g + half_total_pixels) / total_pixels; + int avg_b = (total_b + half_total_pixels) / total_pixels; + + uint32_t low_c = 0, high_c = 0; + + int icov[6] = { 0, 0, 0, 0, 0, 0 }; + for (uint32_t i = 0; i < 16; i++) + { + int r = (int)pSrc_pixels[i].r; + int g = (int)pSrc_pixels[i].g; + int b = (int)pSrc_pixels[i].b; + + if ((r | g | b) < 4) + continue; + + r -= avg_r; + g -= avg_g; + b -= avg_b; + + icov[0] += r * r; + icov[1] += r * g; + icov[2] += r * b; + icov[3] += g * g; + icov[4] += g * b; + icov[5] += b * b; + } + + float cov[6]; + for (uint32_t i = 0; i < 6; i++) + cov[i] = (float)(icov[i]) * (1.0f / 255.0f); + + float xr = (float)(max_r - min_r); + float xg = (float)(max_g - min_g); + float xb = (float)(max_b - min_b); + + if (icov[2] < 0) + xr = -xr; + + if (icov[4] < 0) + xg = -xg; + + for (uint32_t power_iter = 0; power_iter < 4; power_iter++) + { + float r = xr * cov[0] + xg * cov[1] + xb * cov[2]; + float g = xr * cov[1] + xg * cov[3] + xb * cov[4]; + float b = xr * cov[2] + xg * cov[4] + xb * cov[5]; + xr = r; xg = g; xb = b; + } + + float k = maximum(fabsf(xr), fabsf(xg), fabsf(xb)); + int saxis_r = 306, saxis_g = 601, saxis_b = 117; + if (k >= 2) + { + float m = 1024.0f / k; + saxis_r = (int)(xr * m); + saxis_g = (int)(xg * m); + saxis_b = (int)(xb * m); + } + + int low_dot = INT_MAX, high_dot = INT_MIN; + for (uint32_t i = 0; i < 16; i++) + { + int r = (int)pSrc_pixels[i].r, g = (int)pSrc_pixels[i].g, b = (int)pSrc_pixels[i].b; + + if ((r | g | b) < 4) + continue; + + int dot = r * saxis_r + g * saxis_g + b * saxis_b; + if (dot < low_dot) + { + low_dot = dot; + low_c = i; + } + if (dot > high_dot) + { + high_dot = dot; + high_c = i; + } + } + + int lr = to_5(pSrc_pixels[low_c].r); + int lg = to_6(pSrc_pixels[low_c].g); + int lb = to_5(pSrc_pixels[low_c].b); + + int hr = to_5(pSrc_pixels[high_c].r); + int hg = to_6(pSrc_pixels[high_c].g); + int hb = to_5(pSrc_pixels[high_c].b); + + uint8_t trial_sels[16]; + uint32_t trial_err = bc1_find_sels3_fullerr(true, pSrc_pixels, lr, lg, lb, hr, hg, hb, trial_sels, UINT32_MAX, pForce_selectors); + + if (trial_err) + { + const uint32_t total_ls_passes = flags & cEncodeBC1TwoLeastSquaresPasses ? 2 : 1; + for (uint32_t trials = 0; trials < total_ls_passes; trials++) + { + vec3F xl, xh; + int lr2, lg2, lb2, hr2, hg2, hb2; + if (!compute_least_squares_endpoints3_rgb(true, pSrc_pixels, trial_sels, &xl, &xh)) + { + lr2 = g_bc1_match5_half[avg_r].m_hi; + lg2 = g_bc1_match6_half[avg_g].m_hi; + lb2 = g_bc1_match5_half[avg_b].m_hi; + + hr2 = g_bc1_match5_half[avg_r].m_lo; + hg2 = g_bc1_match6_half[avg_g].m_lo; + hb2 = g_bc1_match5_half[avg_b].m_lo; + } + else + { + precise_round_565(xl, xh, hr2, hg2, hb2, lr2, lg2, lb2); + } + + if ((lr == lr2) && (lg == lg2) && (lb == lb2) && (hr == hr2) && (hg == hg2) && (hb == hb2)) + break; + + uint8_t trial_sels2[16]; + uint32_t trial_err2 = bc1_find_sels3_fullerr(true, pSrc_pixels, lr2, lg2, lb2, hr2, hg2, hb2, trial_sels2, trial_err, pForce_selectors); + + if (trial_err2 < trial_err) + { + trial_err = trial_err2; + lr = lr2; lg = lg2; lb = lb2; + hr = hr2; hg = hg2; hb = hb2; + memcpy(trial_sels, trial_sels2, sizeof(trial_sels)); + } + else + break; + } + } + + if (trial_err < cur_err) + { + results.m_3color = true; + results.lr = lr; + results.lg = lg; + results.lb = lb; + results.hr = hr; + results.hg = hg; + results.hb = hb; + memcpy(results.sels, trial_sels, 16); + + cur_err = trial_err; + + return true; + } + + return false; + } + + static bool try_3color_block(const color32* pSrc_pixels, uint32_t flags, uint32_t& cur_err, + int avg_r, int avg_g, int avg_b, int lr, int lg, int lb, int hr, int hg, int hb, int total_r, int total_g, int total_b, uint32_t total_orderings_to_try, + bc1_encode_results& results, const uint8_t* pForce_selectors) + { + if (pForce_selectors) + { + for (uint32_t i = 0; i < 16; i++) + if (pForce_selectors[i] == 3) + return false; + } + + uint8_t trial_sels[16]; + uint32_t trial_err = bc1_find_sels3_fullerr(false, pSrc_pixels, lr, lg, lb, hr, hg, hb, trial_sels, UINT32_MAX, pForce_selectors); + + if (trial_err) + { + const uint32_t total_ls_passes = flags & cEncodeBC1TwoLeastSquaresPasses ? 2 : 1; + for (uint32_t trials = 0; trials < total_ls_passes; trials++) + { + vec3F xl, xh; + int lr2, lg2, lb2, hr2, hg2, hb2; + if (!compute_least_squares_endpoints3_rgb(false, pSrc_pixels, trial_sels, &xl, &xh)) + { + lr2 = g_bc1_match5_half[avg_r].m_hi; + lg2 = g_bc1_match6_half[avg_g].m_hi; + lb2 = g_bc1_match5_half[avg_b].m_hi; + + hr2 = g_bc1_match5_half[avg_r].m_lo; + hg2 = g_bc1_match6_half[avg_g].m_lo; + hb2 = g_bc1_match5_half[avg_b].m_lo; + } + else + { + precise_round_565(xl, xh, hr2, hg2, hb2, lr2, lg2, lb2); + } + + if ((lr == lr2) && (lg == lg2) && (lb == lb2) && (hr == hr2) && (hg == hg2) && (hb == hb2)) + break; + + uint8_t trial_sels2[16]; + uint32_t trial_err2 = bc1_find_sels3_fullerr(false, pSrc_pixels, lr2, lg2, lb2, hr2, hg2, hb2, trial_sels2, trial_err, pForce_selectors); + + if (trial_err2 < trial_err) + { + trial_err = trial_err2; + lr = lr2; lg = lg2; lb = lb2; + hr = hr2; hg = hg2; hb = hb2; + memcpy(trial_sels, trial_sels2, sizeof(trial_sels)); + } + else + break; + } + } + + if ((trial_err) && (flags & cEncodeBC1UseLikelyTotalOrderings) && (total_orderings_to_try)) + { + hist3 h; + for (uint32_t i = 0; i < 16; i++) + { + assert(trial_sels[i] < 3); + h.m_hist[trial_sels[i]]++; + } + + const uint32_t orig_total_order_index = h.lookup_total_ordering_index(); + + int r0, g0, b0, r3, g3, b3; + r0 = (lr << 3) | (lr >> 2); g0 = (lg << 2) | (lg >> 4); b0 = (lb << 3) | (lb >> 2); + r3 = (hr << 3) | (hr >> 2); g3 = (hg << 2) | (hg >> 4); b3 = (hb << 3) | (hb >> 2); + + int ar = r3 - r0, ag = g3 - g0, ab = b3 - b0; + + int dots[16]; + for (uint32_t i = 0; i < 16; i++) + { + int r = pSrc_pixels[i].r; + int g = pSrc_pixels[i].g; + int b = pSrc_pixels[i].b; + int d = 0x1000000 + (r * ar + g * ag + b * ab); + assert(d >= 0); + dots[i] = (d << 4) + i; + } + + std::sort(dots, dots + 16); + + uint32_t r_sum[17], g_sum[17], b_sum[17]; + uint32_t r = 0, g = 0, b = 0; + for (uint32_t i = 0; i < 16; i++) + { + const uint32_t p = dots[i] & 15; + + r_sum[i] = r; + g_sum[i] = g; + b_sum[i] = b; + + r += pSrc_pixels[p].r; + g += pSrc_pixels[p].g; + b += pSrc_pixels[p].b; + } + + r_sum[16] = total_r; + g_sum[16] = total_g; + b_sum[16] = total_b; + + const uint32_t q_total = (flags & cEncodeBC1Exhaustive) ? NUM_UNIQUE_TOTAL_ORDERINGS3 : std::min(total_orderings_to_try, MAX_TOTAL_ORDERINGS3); + for (uint32_t q = 0; q < q_total; q++) + { + const uint32_t s = (flags & cEncodeBC1Exhaustive) ? q : g_best_total_orderings3[orig_total_order_index][q]; + + int trial_lr, trial_lg, trial_lb, trial_hr, trial_hg, trial_hb; + + vec3F xl, xh; + + if ((s == TOTAL_ORDER_3_0_16) || (s == TOTAL_ORDER_3_1_16) || (s == TOTAL_ORDER_3_2_16)) + { + trial_lr = g_bc1_match5_half[avg_r].m_hi; + trial_lg = g_bc1_match6_half[avg_g].m_hi; + trial_lb = g_bc1_match5_half[avg_b].m_hi; + + trial_hr = g_bc1_match5_half[avg_r].m_lo; + trial_hg = g_bc1_match6_half[avg_g].m_lo; + trial_hb = g_bc1_match5_half[avg_b].m_lo; + } + else + { + compute_least_squares_endpoints3_rgb(&xl, &xh, total_r, total_g, total_b, + g_selector_factors3[s][0], g_selector_factors3[s][1], g_selector_factors3[s][2], s, r_sum, g_sum, b_sum); + + precise_round_565(xl, xh, trial_hr, trial_hg, trial_hb, trial_lr, trial_lg, trial_lb); + } + + uint8_t trial_sels2[16]; + uint32_t trial_err2 = bc1_find_sels3_fullerr(false, pSrc_pixels, trial_lr, trial_lg, trial_lb, trial_hr, trial_hg, trial_hb, trial_sels2, UINT32_MAX, pForce_selectors); + + if (trial_err2 < trial_err) + { + trial_err = trial_err2; + + lr = trial_lr; + lg = trial_lg; + lb = trial_lb; + + hr = trial_hr; + hg = trial_hg; + hb = trial_hb; + + memcpy(trial_sels, trial_sels2, sizeof(trial_sels)); + } + + } // s + } + + if (trial_err < cur_err) + { + results.m_3color = true; + results.lr = lr; + results.lg = lg; + results.lb = lb; + results.hr = hr; + results.hg = hg; + results.hb = hb; + memcpy(results.sels, trial_sels, 16); + + cur_err = trial_err; + + return true; + } + + return false; + } + + void encode_bc1(uint32_t level, void* pDst, const uint8_t* pPixels, bool allow_3color, bool allow_transparent_texels_for_black, const uint8_t* pForce_selectors) + { + uint32_t flags = 0, total_orderings4 = 1, total_orderings3 = 1; + + static_assert(MAX_TOTAL_ORDERINGS3 >= 32, "MAX_TOTAL_ORDERINGS3 >= 32"); + static_assert(MAX_TOTAL_ORDERINGS4 >= 32, "MAX_TOTAL_ORDERINGS4 >= 32"); + + switch (level) + { + case 0: + // Faster/higher quality than stb_dxt default. + flags = cEncodeBC1BoundingBoxInt; + break; + case 1: + // Faster/higher quality than stb_dxt default. A bit higher average quality vs. mode 0. + flags = cEncodeBC1Use2DLS; + break; + case 2: + // On average mode 2 is a little weaker than modes 0/1, but it's stronger on outliers (very tough textures). + // Slightly stronger than stb_dxt. + flags = 0; + break; + case 3: + // Slightly stronger than stb_dxt HIGHQUAL. + flags = cEncodeBC1TwoLeastSquaresPasses; + break; + case 4: + flags = cEncodeBC1TwoLeastSquaresPasses | cEncodeBC1UseFullMSEEval | cEncodeBC1Use6PowerIters; + break; + default: + case 5: + // stb_dxt HIGHQUAL + permit 3 color (if it's enabled). + flags = cEncodeBC1TwoLeastSquaresPasses | cEncodeBC1UseFasterMSEEval; + flags |= (allow_3color ? cEncodeBC1Use3ColorBlocks : 0) | (allow_transparent_texels_for_black ? cEncodeBC1Use3ColorBlocksForBlackPixels : 0); + break; + case 6: + flags = cEncodeBC1TwoLeastSquaresPasses | cEncodeBC1UseFasterMSEEval | cEncodeBC1UseLikelyTotalOrderings; + flags |= (allow_3color ? cEncodeBC1Use3ColorBlocks : 0) | (allow_transparent_texels_for_black ? cEncodeBC1Use3ColorBlocksForBlackPixels : 0); + break; + case 7: + flags = cEncodeBC1TwoLeastSquaresPasses | cEncodeBC1UseFasterMSEEval | cEncodeBC1UseLikelyTotalOrderings; + flags |= (allow_3color ? cEncodeBC1Use3ColorBlocks : 0) | (allow_transparent_texels_for_black ? cEncodeBC1Use3ColorBlocksForBlackPixels : 0); + total_orderings4 = 4; + break; + case 8: + flags = cEncodeBC1TwoLeastSquaresPasses | cEncodeBC1UseFasterMSEEval | cEncodeBC1UseLikelyTotalOrderings; + flags |= (allow_3color ? cEncodeBC1Use3ColorBlocks : 0) | (allow_transparent_texels_for_black ? cEncodeBC1Use3ColorBlocksForBlackPixels : 0); + total_orderings4 = 8; + break; + case 9: + flags = cEncodeBC1TwoLeastSquaresPasses | cEncodeBC1UseLikelyTotalOrderings; + flags |= (allow_3color ? cEncodeBC1Use3ColorBlocks : 0) | (allow_transparent_texels_for_black ? cEncodeBC1Use3ColorBlocksForBlackPixels : 0); + total_orderings4 = 11; + total_orderings3 = 3; + break; + case 10: + flags = cEncodeBC1TwoLeastSquaresPasses | cEncodeBC1UseLikelyTotalOrderings; + flags |= (allow_3color ? cEncodeBC1Use3ColorBlocks : 0) | (allow_transparent_texels_for_black ? cEncodeBC1Use3ColorBlocksForBlackPixels : 0); + total_orderings4 = 20; + total_orderings3 = 8; + break; + case 11: + flags = cEncodeBC1TwoLeastSquaresPasses | cEncodeBC1UseLikelyTotalOrderings; + flags |= (allow_3color ? cEncodeBC1Use3ColorBlocks : 0) | (allow_transparent_texels_for_black ? cEncodeBC1Use3ColorBlocksForBlackPixels : 0); + total_orderings4 = 28; + total_orderings3 = 16; + break; + case 12: + flags = cEncodeBC1TwoLeastSquaresPasses | cEncodeBC1UseLikelyTotalOrderings; + flags |= (allow_3color ? cEncodeBC1Use3ColorBlocks : 0) | (allow_transparent_texels_for_black ? cEncodeBC1Use3ColorBlocksForBlackPixels : 0); + total_orderings4 = 32; + total_orderings3 = 32; + break; + case 13: + flags = cEncodeBC1TwoLeastSquaresPasses | cEncodeBC1UseFullMSEEval | cEncodeBC1UseLikelyTotalOrderings | cEncodeBC1Use6PowerIters | (20 << cEncodeBC1EndpointSearchRoundsShift) | cEncodeBC1TryAllInitialEndponts; + flags |= (allow_3color ? cEncodeBC1Use3ColorBlocks : 0) | (allow_transparent_texels_for_black ? cEncodeBC1Use3ColorBlocksForBlackPixels : 0); + total_orderings4 = 32; + total_orderings3 = 32; + break; + case 14: + flags = cEncodeBC1TwoLeastSquaresPasses | cEncodeBC1UseFullMSEEval | cEncodeBC1UseLikelyTotalOrderings | cEncodeBC1Use6PowerIters | (32 << cEncodeBC1EndpointSearchRoundsShift) | cEncodeBC1TryAllInitialEndponts; + flags |= (allow_3color ? cEncodeBC1Use3ColorBlocks : 0) | (allow_transparent_texels_for_black ? cEncodeBC1Use3ColorBlocksForBlackPixels : 0); + total_orderings4 = 32; + total_orderings3 = 32; + break; + case 15: + flags = cEncodeBC1TwoLeastSquaresPasses | cEncodeBC1UseFullMSEEval | cEncodeBC1UseLikelyTotalOrderings | cEncodeBC1Use6PowerIters | (32 << cEncodeBC1EndpointSearchRoundsShift) | cEncodeBC1TryAllInitialEndponts; + flags |= (allow_3color ? cEncodeBC1Use3ColorBlocks : 0) | (allow_transparent_texels_for_black ? cEncodeBC1Use3ColorBlocksForBlackPixels : 0); + total_orderings4 = ((((32 + MAX_TOTAL_ORDERINGS4) / 2) + 32) / 2); + total_orderings3 = 32; + break; + case 16: + flags = cEncodeBC1TwoLeastSquaresPasses | cEncodeBC1UseFullMSEEval | cEncodeBC1UseLikelyTotalOrderings | cEncodeBC1Use6PowerIters | (256 << cEncodeBC1EndpointSearchRoundsShift) | cEncodeBC1TryAllInitialEndponts; + flags |= (allow_3color ? cEncodeBC1Use3ColorBlocks : 0) | (allow_transparent_texels_for_black ? cEncodeBC1Use3ColorBlocksForBlackPixels : 0); + total_orderings4 = (32 + MAX_TOTAL_ORDERINGS4) / 2; + total_orderings3 = 32; + break; + case 17: + flags = cEncodeBC1TwoLeastSquaresPasses | cEncodeBC1UseFullMSEEval | cEncodeBC1UseLikelyTotalOrderings | cEncodeBC1Use6PowerIters | (256 << cEncodeBC1EndpointSearchRoundsShift) | cEncodeBC1TryAllInitialEndponts; + flags |= (allow_3color ? cEncodeBC1Use3ColorBlocks : 0) | (allow_transparent_texels_for_black ? cEncodeBC1Use3ColorBlocksForBlackPixels : 0); + total_orderings4 = MAX_TOTAL_ORDERINGS4; + total_orderings3 = 32; + break; + case 18: + flags = cEncodeBC1TwoLeastSquaresPasses | cEncodeBC1UseFullMSEEval | cEncodeBC1UseLikelyTotalOrderings | cEncodeBC1Use6PowerIters | cEncodeBC1Iterative | (256 << cEncodeBC1EndpointSearchRoundsShift) | cEncodeBC1TryAllInitialEndponts; + flags |= (allow_3color ? cEncodeBC1Use3ColorBlocks : 0) | (allow_transparent_texels_for_black ? cEncodeBC1Use3ColorBlocksForBlackPixels : 0); + total_orderings4 = MAX_TOTAL_ORDERINGS4; + total_orderings3 = 32; + break; + case 19: + // This hidden mode is *extremely* slow and abuses the encoder. It's just for testing/training. + flags = cEncodeBC1TwoLeastSquaresPasses | cEncodeBC1UseFullMSEEval | cEncodeBC1UseLikelyTotalOrderings | cEncodeBC1Use6PowerIters | cEncodeBC1Exhaustive | cEncodeBC1Iterative | (256 << cEncodeBC1EndpointSearchRoundsShift) | cEncodeBC1TryAllInitialEndponts; + flags |= (allow_3color ? cEncodeBC1Use3ColorBlocks : 0) | (allow_transparent_texels_for_black ? cEncodeBC1Use3ColorBlocksForBlackPixels : 0); + total_orderings4 = 32; + total_orderings3 = 32; + break; + } + + encode_bc1(pDst, pPixels, flags, total_orderings4, total_orderings3, pForce_selectors); + } + + static inline void encode_bc1_pick_initial(const color32* pSrc_pixels, uint32_t flags, bool grayscale_flag, + int min_r, int min_g, int min_b, int max_r, int max_g, int max_b, + int avg_r, int avg_g, int avg_b, int total_r, int total_g, int total_b, + int& lr, int& lg, int& lb, int& hr, int& hg, int& hb) + { + if (grayscale_flag) + { + const int fr = pSrc_pixels[0].r; + + // Grayscale blocks are a common enough case to specialize. + if ((max_r - min_r) < 2) + { + lr = lb = hr = hb = to_5(fr); + lg = hg = to_6(fr); + } + else + { + lr = lb = to_5(min_r); + lg = to_6(min_r); + + hr = hb = to_5(max_r); + hg = to_6(max_r); + } + } + else if (flags & cEncodeBC1Use2DLS) + { + // 2D Least Squares approach from Humus's example, with added inset and optimal rounding. + int big_chan = 0, min_chan_val = min_r, max_chan_val = max_r; + if ((max_g - min_g) > (max_chan_val - min_chan_val)) + big_chan = 1, min_chan_val = min_g, max_chan_val = max_g; + + if ((max_b - min_b) > (max_chan_val - min_chan_val)) + big_chan = 2, min_chan_val = min_b, max_chan_val = max_b; + + int sum_xy_r = 0, sum_xy_g = 0, sum_xy_b = 0; + vec3F l, h; + if (big_chan == 0) + { + for (uint32_t i = 0; i < 16; i++) + { + const int r = pSrc_pixels[i].r, g = pSrc_pixels[i].g, b = pSrc_pixels[i].b; + sum_xy_r += r * r, sum_xy_g += r * g, sum_xy_b += r * b; + } + + int sum_x = total_r; + int sum_x2 = sum_xy_r; + + float div = (float)(16 * sum_x2 - sum_x * sum_x); + float b_y = 0.0f, b_z = 0.0f; + if (fabs(div) > 1e-8f) + { + div = 1.0f / div; + b_y = (16 * sum_xy_g - sum_x * total_g) * div; + b_z = (16 * sum_xy_b - sum_x * total_b) * div; + } + + float a_y = (total_g - b_y * sum_x) / 16.0f; + float a_z = (total_b - b_z * sum_x) / 16.0f; + + l.c[1] = a_y + b_y * min_chan_val; + l.c[2] = a_z + b_z * min_chan_val; + + h.c[1] = a_y + b_y * max_chan_val; + h.c[2] = a_z + b_z * max_chan_val; + + float dg = (h.c[1] - l.c[1]); + float db = (h.c[2] - l.c[2]); + + h.c[1] = l.c[1] + dg * (15.0f / 16.0f); + h.c[2] = l.c[2] + db * (15.0f / 16.0f); + + l.c[1] = l.c[1] + dg * (1.0f / 16.0f); + l.c[2] = l.c[2] + db * (1.0f / 16.0f); + + float d = (float)(max_chan_val - min_chan_val); + float fmin_chan_val = min_chan_val + d * (1.0f / 16.0f); + float fmax_chan_val = min_chan_val + d * (15.0f / 16.0f); + + l.c[0] = fmin_chan_val; + h.c[0] = fmax_chan_val; + } + else if (big_chan == 1) + { + for (uint32_t i = 0; i < 16; i++) + { + const int r = pSrc_pixels[i].r, g = pSrc_pixels[i].g, b = pSrc_pixels[i].b; + sum_xy_r += g * r, sum_xy_g += g * g, sum_xy_b += g * b; + } + + int sum_x = total_g; + int sum_x2 = sum_xy_g; + + float div = (float)(16 * sum_x2 - sum_x * sum_x); + float b_x = 0.0f, b_z = 0.0f; + if (fabs(div) > 1e-8f) + { + div = 1.0f / div; + b_x = (16 * sum_xy_r - sum_x * total_r) * div; + b_z = (16 * sum_xy_b - sum_x * total_b) * div; + } + + float a_x = (total_r - b_x * sum_x) / 16.0f; + float a_z = (total_b - b_z * sum_x) / 16.0f; + + l.c[0] = a_x + b_x * min_chan_val; + l.c[2] = a_z + b_z * min_chan_val; + + h.c[0] = a_x + b_x * max_chan_val; + h.c[2] = a_z + b_z * max_chan_val; + + float dr = (h.c[0] - l.c[0]); + float db = (h.c[2] - l.c[2]); + + h.c[0] = l.c[0] + dr * (15.0f / 16.0f); + h.c[2] = l.c[2] + db * (15.0f / 16.0f); + + l.c[0] = l.c[0] + dr * (1.0f / 16.0f); + l.c[2] = l.c[2] + db * (1.0f / 16.0f); + + float d = (float)(max_chan_val - min_chan_val); + float fmin_chan_val = min_chan_val + d * (1.0f / 16.0f); + float fmax_chan_val = min_chan_val + d * (15.0f / 16.0f); + + l.c[1] = fmin_chan_val; + h.c[1] = fmax_chan_val; + } + else + { + for (uint32_t i = 0; i < 16; i++) + { + const int r = pSrc_pixels[i].r, g = pSrc_pixels[i].g, b = pSrc_pixels[i].b; + sum_xy_r += b * r, sum_xy_g += b * g, sum_xy_b += b * b; + } + + int sum_x = total_b; + int sum_x2 = sum_xy_b; + + float div = (float)(16 * sum_x2 - sum_x * sum_x); + float b_x = 0.0f, b_y = 0.0f; + if (fabs(div) > 1e-8f) + { + div = 1.0f / div; + b_x = (16 * sum_xy_r - sum_x * total_r) * div; + b_y = (16 * sum_xy_g - sum_x * total_g) * div; + } + + float a_x = (total_r - b_x * sum_x) / 16.0f; + float a_y = (total_g - b_y * sum_x) / 16.0f; + + l.c[0] = a_x + b_x * min_chan_val; + l.c[1] = a_y + b_y * min_chan_val; + + h.c[0] = a_x + b_x * max_chan_val; + h.c[1] = a_y + b_y * max_chan_val; + + float dr = (h.c[0] - l.c[0]); + float dg = (h.c[1] - l.c[1]); + + h.c[0] = l.c[0] + dr * (15.0f / 16.0f); + h.c[1] = l.c[1] + dg * (15.0f / 16.0f); + + l.c[0] = l.c[0] + dr * (1.0f / 16.0f); + l.c[1] = l.c[1] + dg * (1.0f / 16.0f); + + float d = (float)(max_chan_val - min_chan_val); + float fmin_chan_val = min_chan_val + d * (1.0f / 16.0f); + float fmax_chan_val = min_chan_val + d * (15.0f / 16.0f); + + l.c[2] = fmin_chan_val; + h.c[2] = fmax_chan_val; + } + + precise_round_565_noscale(l, h, lr, lg, lb, hr, hg, hb); + } + else if (flags & cEncodeBC1BoundingBox) + { + // Algorithm from icbc.h compress_dxt1_fast() + vec3F l, h; + l.c[0] = min_r * (1.0f / 255.0f); + l.c[1] = min_g * (1.0f / 255.0f); + l.c[2] = min_b * (1.0f / 255.0f); + + h.c[0] = max_r * (1.0f / 255.0f); + h.c[1] = max_g * (1.0f / 255.0f); + h.c[2] = max_b * (1.0f / 255.0f); + + const float bias = 8.0f / 255.0f; + float inset_r = (h.c[0] - l.c[0] - bias) * (1.0f / 16.0f); + float inset_g = (h.c[1] - l.c[1] - bias) * (1.0f / 16.0f); + float inset_b = (h.c[2] - l.c[2] - bias) * (1.0f / 16.0f); + + l.c[0] = clampf(l.c[0] + inset_r, 0.0f, 1.0f); + l.c[1] = clampf(l.c[1] + inset_g, 0.0f, 1.0f); + l.c[2] = clampf(l.c[2] + inset_b, 0.0f, 1.0f); + + h.c[0] = clampf(h.c[0] - inset_r, 0.0f, 1.0f); + h.c[1] = clampf(h.c[1] - inset_g, 0.0f, 1.0f); + h.c[2] = clampf(h.c[2] - inset_b, 0.0f, 1.0f); + + int icov_xz = 0, icov_yz = 0; + for (uint32_t i = 0; i < 16; i++) + { + int r = (int)pSrc_pixels[i].r - avg_r; + int g = (int)pSrc_pixels[i].g - avg_g; + int b = (int)pSrc_pixels[i].b - avg_b; + icov_xz += r * b; + icov_yz += g * b; + } + + if (icov_xz < 0) + std::swap(l.c[0], h.c[0]); + + if (icov_yz < 0) + std::swap(l.c[1], h.c[1]); + + precise_round_565(l, h, lr, lg, lb, hr, hg, hb); + } + else if (flags & cEncodeBC1BoundingBoxInt) + { + // Algorithm from icbc.h compress_dxt1_fast(), but converted to integer. + int inset_r = (max_r - min_r - 8) >> 4; + int inset_g = (max_g - min_g - 8) >> 4; + int inset_b = (max_b - min_b - 8) >> 4; + + min_r += inset_r; + min_g += inset_g; + min_b += inset_b; + if ((uint32_t)(min_r | min_g | min_b) > 255U) + { + min_r = clampi(min_r, 0, 255); + min_g = clampi(min_g, 0, 255); + min_b = clampi(min_b, 0, 255); + } + + max_r -= inset_r; + max_g -= inset_g; + max_b -= inset_b; + if ((uint32_t)(max_r | max_g | max_b) > 255U) + { + max_r = clampi(max_r, 0, 255); + max_g = clampi(max_g, 0, 255); + max_b = clampi(max_b, 0, 255); + } + + int icov_xz = 0, icov_yz = 0; + for (uint32_t i = 0; i < 16; i++) + { + int r = (int)pSrc_pixels[i].r - avg_r; + int g = (int)pSrc_pixels[i].g - avg_g; + int b = (int)pSrc_pixels[i].b - avg_b; + icov_xz += r * b; + icov_yz += g * b; + } + + int x0 = min_r; + int y0 = min_g; + int x1 = max_r; + int y1 = max_g; + + if (icov_xz < 0) + std::swap(x0, x1); + + if (icov_yz < 0) + std::swap(y0, y1); + + lr = to_5(x0); + lg = to_6(y0); + lb = to_5(min_b); + + hr = to_5(x1); + hg = to_6(y1); + hb = to_5(max_b); + } + else + { + // Select 2 colors along the principle axis. (There must be a faster/simpler way.) + uint32_t low_c = 0, high_c = 0; + + int icov[6] = { 0, 0, 0, 0, 0, 0 }; + for (uint32_t i = 0; i < 16; i++) + { + int r = (int)pSrc_pixels[i].r - avg_r; + int g = (int)pSrc_pixels[i].g - avg_g; + int b = (int)pSrc_pixels[i].b - avg_b; + icov[0] += r * r; + icov[1] += r * g; + icov[2] += r * b; + icov[3] += g * g; + icov[4] += g * b; + icov[5] += b * b; + } + + int saxis_r = 306, saxis_g = 601, saxis_b = 117; + + float xr = (float)(max_r - min_r); + float xg = (float)(max_g - min_g); + float xb = (float)(max_b - min_b); + + if (icov[2] < 0) + xr = -xr; + + if (icov[4] < 0) + xg = -xg; + + float cov[6]; + for (uint32_t i = 0; i < 6; i++) + cov[i] = (float)(icov[i]) * (1.0f / 255.0f); + + const uint32_t total_power_iters = (flags & cEncodeBC1Use6PowerIters) ? 6 : 4; + for (uint32_t power_iter = 0; power_iter < total_power_iters; power_iter++) + { + float r = xr * cov[0] + xg * cov[1] + xb * cov[2]; + float g = xr * cov[1] + xg * cov[3] + xb * cov[4]; + float b = xr * cov[2] + xg * cov[4] + xb * cov[5]; + xr = r; xg = g; xb = b; + } + + float k = maximum(fabsf(xr), fabsf(xg), fabsf(xb)); + if (k >= 2) + { + float m = 2048.0f / k; + saxis_r = (int)(xr * m); + saxis_g = (int)(xg * m); + saxis_b = (int)(xb * m); + } + + int low_dot = INT_MAX, high_dot = INT_MIN; + + saxis_r = (int)((uint32_t)saxis_r << 4U); + saxis_g = (int)((uint32_t)saxis_g << 4U); + saxis_b = (int)((uint32_t)saxis_b << 4U); + + for (uint32_t i = 0; i < 16; i += 4) + { + int dot0 = ((pSrc_pixels[i].r * saxis_r + pSrc_pixels[i].g * saxis_g + pSrc_pixels[i].b * saxis_b) & ~0xF) + i; + int dot1 = ((pSrc_pixels[i + 1].r * saxis_r + pSrc_pixels[i + 1].g * saxis_g + pSrc_pixels[i + 1].b * saxis_b) & ~0xF) + i + 1; + int dot2 = ((pSrc_pixels[i + 2].r * saxis_r + pSrc_pixels[i + 2].g * saxis_g + pSrc_pixels[i + 2].b * saxis_b) & ~0xF) + i + 2; + int dot3 = ((pSrc_pixels[i + 3].r * saxis_r + pSrc_pixels[i + 3].g * saxis_g + pSrc_pixels[i + 3].b * saxis_b) & ~0xF) + i + 3; + + int min_d01 = std::min(dot0, dot1); + int max_d01 = std::max(dot0, dot1); + + int min_d23 = std::min(dot2, dot3); + int max_d23 = std::max(dot2, dot3); + + int min_d = std::min(min_d01, min_d23); + int max_d = std::max(max_d01, max_d23); + + low_dot = std::min(low_dot, min_d); + high_dot = std::max(high_dot, max_d); + } + low_c = low_dot & 15; + high_c = high_dot & 15; + + lr = to_5(pSrc_pixels[low_c].r); + lg = to_6(pSrc_pixels[low_c].g); + lb = to_5(pSrc_pixels[low_c].b); + + hr = to_5(pSrc_pixels[high_c].r); + hg = to_6(pSrc_pixels[high_c].g); + hb = to_5(pSrc_pixels[high_c].b); + } + } + + static const int8_t s_adjacent_voxels[16][4] = + { + { 1,0,0, 3 }, // 0 + { 0,1,0, 4 }, // 1 + { 0,0,1, 5 }, // 2 + { -1,0,0, 0 }, // 3 + { 0,-1,0, 1 }, // 4 + { 0,0,-1, 2 }, // 5 + { 1,1,0, 9 }, // 6 + { 1,0,1, 10 }, // 7 + { 0,1,1, 11 }, // 8 + { -1,-1,0, 6 }, // 9 + { -1,0,-1, 7 }, // 10 + { 0,-1,-1, 8 }, // 11 + { -1,1,0, 13 }, // 12 + { 1,-1,0, 12 }, // 13 + { 0,-1,1, 15 }, // 14 + { 0,1,-1, 14 }, // 15 + }; + + // From icbc's high quality mode. + static inline void encode_bc1_endpoint_search(const color32* pSrc_pixels, bool any_black_pixels, + uint32_t flags, bc1_encode_results& results, uint32_t cur_err, const uint8_t* pForce_selectors) + { + int& lr = results.lr, & lg = results.lg, & lb = results.lb, & hr = results.hr, & hg = results.hg, & hb = results.hb; + uint8_t* sels = results.sels; + + int prev_improvement_index = 0, forbidden_direction = -1; + + const int endpoint_search_rounds = (flags & cEncodeBC1EndpointSearchRoundsMask) >> cEncodeBC1EndpointSearchRoundsShift; + for (int i = 0; i < endpoint_search_rounds; i++) + { + assert(s_adjacent_voxels[s_adjacent_voxels[i & 15][3]][3] == (i & 15)); + + if (forbidden_direction == (i & 31)) + continue; + + const int8_t delta[3] = { s_adjacent_voxels[i & 15][0], s_adjacent_voxels[i & 15][1], s_adjacent_voxels[i & 15][2] }; + + int trial_lr = lr, trial_lg = lg, trial_lb = lb, trial_hr = hr, trial_hg = hg, trial_hb = hb; + + if ((i >> 4) & 1) + { + trial_lr = clampi(trial_lr + delta[0], 0, 31); + trial_lg = clampi(trial_lg + delta[1], 0, 63); + trial_lb = clampi(trial_lb + delta[2], 0, 31); + } + else + { + trial_hr = clampi(trial_hr + delta[0], 0, 31); + trial_hg = clampi(trial_hg + delta[1], 0, 63); + trial_hb = clampi(trial_hb + delta[2], 0, 31); + } + + uint8_t trial_sels[16]; + + uint32_t trial_err; + if (results.m_3color) + { + trial_err = bc1_find_sels3_fullerr( + ((any_black_pixels) && ((flags & cEncodeBC1Use3ColorBlocksForBlackPixels) != 0)), + pSrc_pixels, trial_lr, trial_lg, trial_lb, trial_hr, trial_hg, trial_hb, trial_sels, cur_err, pForce_selectors); + } + else + { + trial_err = bc1_find_sels4(flags, pSrc_pixels, trial_lr, trial_lg, trial_lb, trial_hr, trial_hg, trial_hb, trial_sels, cur_err, pForce_selectors); + } + + if (trial_err < cur_err) + { + cur_err = trial_err; + + forbidden_direction = s_adjacent_voxels[i & 15][3] | (i & 16); + + lr = trial_lr, lg = trial_lg, lb = trial_lb, hr = trial_hr, hg = trial_hg, hb = trial_hb; + + memcpy(sels, trial_sels, 16); + + prev_improvement_index = i; + } + + if (i - prev_improvement_index > 32) + break; + } + } + + void encode_bc1(void* pDst, const uint8_t* pPixels, uint32_t flags, uint32_t total_orderings_to_try, uint32_t total_orderings_to_try3, const uint8_t* pForce_selectors) + { + assert(g_initialized); + + const color32* pSrc_pixels = (const color32*)pPixels; + bc1_block* pDst_block = static_cast(pDst); + + int avg_r, avg_g, avg_b, min_r, min_g, min_b, max_r, max_g, max_b; + + const uint32_t fr = pSrc_pixels[0].r, fg = pSrc_pixels[0].g, fb = pSrc_pixels[0].b; + + uint32_t j; + for (j = 15; j >= 1; --j) + if ((pSrc_pixels[j].r != fr) || (pSrc_pixels[j].g != fg) || (pSrc_pixels[j].b != fb)) + break; + + if (j == 0) + { + encode_bc1_solid_block(pDst, fr, fg, fb, (flags & (cEncodeBC1Use3ColorBlocks | cEncodeBC1Use3ColorBlocksForBlackPixels)) != 0); + return; + } + + int total_r = fr, total_g = fg, total_b = fb; + + max_r = fr, max_g = fg, max_b = fb; + min_r = fr, min_g = fg, min_b = fb; + + uint32_t grayscale_flag = (fr == fg) && (fr == fb); + uint32_t any_black_pixels = (fr | fg | fb) < 4; + + for (uint32_t i = 1; i < 16; i++) + { + const int r = pSrc_pixels[i].r, g = pSrc_pixels[i].g, b = pSrc_pixels[i].b; + + grayscale_flag &= ((r == g) && (r == b)); + any_black_pixels |= ((r | g | b) < 4); + + max_r = std::max(max_r, r); max_g = std::max(max_g, g); max_b = std::max(max_b, b); + min_r = std::min(min_r, r); min_g = std::min(min_g, g); min_b = std::min(min_b, b); + total_r += r; total_g += g; total_b += b; + } + + avg_r = (total_r + 8) >> 4, avg_g = (total_g + 8) >> 4, avg_b = (total_b + 8) >> 4; + + bc1_encode_results results; + results.m_3color = false; + + uint8_t* sels = results.sels; + int& lr = results.lr, & lg = results.lg, & lb = results.lb, & hr = results.hr, & hg = results.hg, & hb = results.hb; + int orig_lr = 0, orig_lg = 0, orig_lb = 0, orig_hr = 0, orig_hg = 0, orig_hb = 0; + + lr = 0, lg = 0, lb = 0, hr = 0, hg = 0, hb = 0; + + const bool needs_block_error = ((flags & (cEncodeBC1UseLikelyTotalOrderings | cEncodeBC1Use3ColorBlocks | cEncodeBC1UseFullMSEEval | cEncodeBC1EndpointSearchRoundsMask)) != 0) || + (any_black_pixels && ((flags & cEncodeBC1Use3ColorBlocksForBlackPixels) != 0)); + + uint32_t cur_err = UINT32_MAX; + + if (!needs_block_error) + { + assert((flags & cEncodeBC1TryAllInitialEndponts) == 0); + + encode_bc1_pick_initial(pSrc_pixels, flags, grayscale_flag != 0, + min_r, min_g, min_b, max_r, max_g, max_b, + avg_r, avg_g, avg_b, total_r, total_g, total_b, + lr, lg, lb, hr, hg, hb); + + orig_lr = lr, orig_lg = lg, orig_lb = lb, orig_hr = hr, orig_hg = hg, orig_hb = hb; + + bc1_find_sels4_noerr(pSrc_pixels, lr, lg, lb, hr, hg, hb, sels, pForce_selectors); + + const uint32_t total_ls_passes = flags & cEncodeBC1TwoLeastSquaresPasses ? 2 : 1; + for (uint32_t ls_pass = 0; ls_pass < total_ls_passes; ls_pass++) + { + int trial_lr, trial_lg, trial_lb, trial_hr, trial_hg, trial_hb; + + vec3F xl, xh; + if (!compute_least_squares_endpoints4_rgb(pSrc_pixels, sels, &xl, &xh, total_r, total_g, total_b)) + { + // All selectors equal - treat it as a solid block which should always be equal or better. + trial_lr = g_bc1_match5_equals_1[avg_r].m_hi; + trial_lg = g_bc1_match6_equals_1[avg_g].m_hi; + trial_lb = g_bc1_match5_equals_1[avg_b].m_hi; + + trial_hr = g_bc1_match5_equals_1[avg_r].m_lo; + trial_hg = g_bc1_match6_equals_1[avg_g].m_lo; + trial_hb = g_bc1_match5_equals_1[avg_b].m_lo; + + // In high/higher quality mode, let it try again in case the optimal tables have caused the sels to diverge. + } + else + { + precise_round_565(xl, xh, trial_hr, trial_hg, trial_hb, trial_lr, trial_lg, trial_lb); + } + + if ((lr == trial_lr) && (lg == trial_lg) && (lb == trial_lb) && (hr == trial_hr) && (hg == trial_hg) && (hb == trial_hb)) + break; + + bc1_find_sels4_noerr(pSrc_pixels, trial_lr, trial_lg, trial_lb, trial_hr, trial_hg, trial_hb, sels, pForce_selectors); + + lr = trial_lr; + lg = trial_lg; + lb = trial_lb; + hr = trial_hr; + hg = trial_hg; + hb = trial_hb; + + } // ls_pass + } + else + { + const uint32_t total_rounds = (flags & cEncodeBC1TryAllInitialEndponts) ? 2 : 1; + for (uint32_t round = 0; round < total_rounds; round++) + { + uint32_t modified_flags = flags; + if (round == 1) + { + modified_flags &= ~(cEncodeBC1Use2DLS | cEncodeBC1BoundingBox); + modified_flags |= cEncodeBC1BoundingBox; + } + + int round_lr, round_lg, round_lb, round_hr, round_hg, round_hb; + uint8_t round_sels[16]; + + encode_bc1_pick_initial(pSrc_pixels, modified_flags, grayscale_flag != 0, + min_r, min_g, min_b, max_r, max_g, max_b, + avg_r, avg_g, avg_b, total_r, total_g, total_b, + round_lr, round_lg, round_lb, round_hr, round_hg, round_hb); + + int orig_round_lr = round_lr, orig_round_lg = round_lg, orig_round_lb = round_lb, orig_round_hr = round_hr, orig_round_hg = round_hg, orig_round_hb = round_hb; + + uint32_t round_err = bc1_find_sels4(flags, pSrc_pixels, round_lr, round_lg, round_lb, round_hr, round_hg, round_hb, round_sels, UINT32_MAX, pForce_selectors); + + const uint32_t total_ls_passes = flags & cEncodeBC1TwoLeastSquaresPasses ? 2 : 1; + for (uint32_t ls_pass = 0; ls_pass < total_ls_passes; ls_pass++) + { + int trial_lr, trial_lg, trial_lb, trial_hr, trial_hg, trial_hb; + + vec3F xl, xh; + if (!compute_least_squares_endpoints4_rgb(pSrc_pixels, round_sels, &xl, &xh, total_r, total_g, total_b)) + { + // All selectors equal - treat it as a solid block which should always be equal or better. + trial_lr = g_bc1_match5_equals_1[avg_r].m_hi; + trial_lg = g_bc1_match6_equals_1[avg_g].m_hi; + trial_lb = g_bc1_match5_equals_1[avg_b].m_hi; + + trial_hr = g_bc1_match5_equals_1[avg_r].m_lo; + trial_hg = g_bc1_match6_equals_1[avg_g].m_lo; + trial_hb = g_bc1_match5_equals_1[avg_b].m_lo; + + // In high/higher quality mode, let it try again in case the optimal tables have caused the sels to diverge. + } + else + { + precise_round_565(xl, xh, trial_hr, trial_hg, trial_hb, trial_lr, trial_lg, trial_lb); + } + + if ((round_lr == trial_lr) && (round_lg == trial_lg) && (round_lb == trial_lb) && (round_hr == trial_hr) && (round_hg == trial_hg) && (round_hb == trial_hb)) + break; + + uint8_t trial_sels[16]; + uint32_t trial_err = bc1_find_sels4(flags, pSrc_pixels, trial_lr, trial_lg, trial_lb, trial_hr, trial_hg, trial_hb, trial_sels, round_err, pForce_selectors); + + if (trial_err < round_err) + { + round_lr = trial_lr; + round_lg = trial_lg; + round_lb = trial_lb; + + round_hr = trial_hr; + round_hg = trial_hg; + round_hb = trial_hb; + + round_err = trial_err; + memcpy(round_sels, trial_sels, 16); + } + else + break; + + } // ls_pass + + if (round_err <= cur_err) + { + cur_err = round_err; + + lr = round_lr; + lg = round_lg; + lb = round_lb; + hr = round_hr; + hg = round_hg; + hb = round_hb; + + orig_lr = orig_round_lr; + orig_lg = orig_round_lg; + orig_lb = orig_round_lb; + orig_hr = orig_round_hr; + orig_hg = orig_round_hg; + orig_hb = orig_round_hb; + + memcpy(sels, round_sels, 16); + } + + } // round + } + + if ((cur_err) && (flags & cEncodeBC1UseLikelyTotalOrderings)) + { + assert(needs_block_error); + + const uint32_t total_iters = (flags & cEncodeBC1Iterative) ? 2 : 1; + for (uint32_t iter_index = 0; iter_index < total_iters; iter_index++) + { + const uint32_t orig_err = cur_err; + + hist4 h; + for (uint32_t i = 0; i < 16; i++) + { + assert(sels[i] < 4); + h.m_hist[sels[i]]++; + } + + const uint32_t orig_total_order_index = h.lookup_total_ordering_index(); + + int r0, g0, b0, r3, g3, b3; + r0 = (lr << 3) | (lr >> 2); g0 = (lg << 2) | (lg >> 4); b0 = (lb << 3) | (lb >> 2); + r3 = (hr << 3) | (hr >> 2); g3 = (hg << 2) | (hg >> 4); b3 = (hb << 3) | (hb >> 2); + + int ar = r3 - r0, ag = g3 - g0, ab = b3 - b0; + + int dots[16]; + for (uint32_t i = 0; i < 16; i++) + { + int r = pSrc_pixels[i].r; + int g = pSrc_pixels[i].g; + int b = pSrc_pixels[i].b; + int d = 0x1000000 + (r * ar + g * ag + b * ab); + assert(d >= 0); + dots[i] = (d << 4) + i; + } + + std::sort(dots, dots + 16); + + uint32_t r_sum[17], g_sum[17], b_sum[17]; + uint32_t r = 0, g = 0, b = 0; + for (uint32_t i = 0; i < 16; i++) + { + const uint32_t p = dots[i] & 15; + + r_sum[i] = r; + g_sum[i] = g; + b_sum[i] = b; + + r += pSrc_pixels[p].r; + g += pSrc_pixels[p].g; + b += pSrc_pixels[p].b; + } + + r_sum[16] = total_r; + g_sum[16] = total_g; + b_sum[16] = total_b; + + const uint32_t q_total = (flags & cEncodeBC1Exhaustive) ? NUM_UNIQUE_TOTAL_ORDERINGS4 : clampi(total_orderings_to_try, MIN_TOTAL_ORDERINGS, MAX_TOTAL_ORDERINGS4); + for (uint32_t q = 0; q < q_total; q++) + { + const uint32_t s = (flags & cEncodeBC1Exhaustive) ? q : g_best_total_orderings4[orig_total_order_index][q]; + + int trial_lr, trial_lg, trial_lb, trial_hr, trial_hg, trial_hb; + + vec3F xl, xh; + + if ((s == TOTAL_ORDER_4_0_16) || (s == TOTAL_ORDER_4_1_16) || (s == TOTAL_ORDER_4_2_16) || (s == TOTAL_ORDER_4_3_16)) + { + trial_lr = g_bc1_match5_equals_1[avg_r].m_hi; + trial_lg = g_bc1_match6_equals_1[avg_g].m_hi; + trial_lb = g_bc1_match5_equals_1[avg_b].m_hi; + + trial_hr = g_bc1_match5_equals_1[avg_r].m_lo; + trial_hg = g_bc1_match6_equals_1[avg_g].m_lo; + trial_hb = g_bc1_match5_equals_1[avg_b].m_lo; + } + else + { + compute_least_squares_endpoints4_rgb(&xl, &xh, total_r, total_g, total_b, + g_selector_factors4[s][0], g_selector_factors4[s][1], g_selector_factors4[s][2], s, r_sum, g_sum, b_sum); + + precise_round_565(xl, xh, trial_hr, trial_hg, trial_hb, trial_lr, trial_lg, trial_lb); + } + + uint8_t trial_sels[16]; + + uint32_t trial_err = bc1_find_sels4(flags, pSrc_pixels, trial_lr, trial_lg, trial_lb, trial_hr, trial_hg, trial_hb, trial_sels, cur_err, pForce_selectors); + + if (trial_err < cur_err) + { + cur_err = trial_err; + + lr = trial_lr; + lg = trial_lg; + lb = trial_lb; + + hr = trial_hr; + hg = trial_hg; + hb = trial_hb; + + memcpy(sels, trial_sels, 16); + } + + } // s + + if ((!cur_err) || (cur_err == orig_err)) + break; + + } // iter_index + } + + if (((flags & (cEncodeBC1Use3ColorBlocks | cEncodeBC1Use3ColorBlocksForBlackPixels)) != 0) && (cur_err)) + { + if (flags & cEncodeBC1Use3ColorBlocks) + { + assert(needs_block_error); + try_3color_block(pSrc_pixels, flags, cur_err, avg_r, avg_g, avg_b, orig_lr, orig_lg, orig_lb, orig_hr, orig_hg, orig_hb, total_r, total_g, total_b, total_orderings_to_try3, results, pForce_selectors); + } + + if ((any_black_pixels) && ((flags & cEncodeBC1Use3ColorBlocksForBlackPixels) != 0)) + { + assert(needs_block_error); + try_3color_block_useblack(pSrc_pixels, flags, cur_err, results, pForce_selectors); + } + } + + if ((flags & cEncodeBC1EndpointSearchRoundsMask) && (cur_err)) + { + assert(needs_block_error); + + encode_bc1_endpoint_search(pSrc_pixels, any_black_pixels != 0, flags, results, cur_err, pForce_selectors); + } + + if (results.m_3color) + bc1_encode3(pDst_block, results.lr, results.lg, results.lb, results.hr, results.hg, results.hb, results.sels); + else + bc1_encode4(pDst_block, results.lr, results.lg, results.lb, results.hr, results.hg, results.hb, results.sels); + } + + // BC3-5 + + void encode_bc4(void* pDst, const uint8_t* pPixels, uint32_t stride) + { + assert(g_initialized); + + uint32_t min0_v, max0_v, min1_v, max1_v, min2_v, max2_v, min3_v, max3_v; + + { + min0_v = max0_v = pPixels[0 * stride]; + min1_v = max1_v = pPixels[1 * stride]; + min2_v = max2_v = pPixels[2 * stride]; + min3_v = max3_v = pPixels[3 * stride]; + } + + { + uint32_t v0 = pPixels[4 * stride]; min0_v = std::min(min0_v, v0); max0_v = std::max(max0_v, v0); + uint32_t v1 = pPixels[5 * stride]; min1_v = std::min(min1_v, v1); max1_v = std::max(max1_v, v1); + uint32_t v2 = pPixels[6 * stride]; min2_v = std::min(min2_v, v2); max2_v = std::max(max2_v, v2); + uint32_t v3 = pPixels[7 * stride]; min3_v = std::min(min3_v, v3); max3_v = std::max(max3_v, v3); + } + + { + uint32_t v0 = pPixels[8 * stride]; min0_v = std::min(min0_v, v0); max0_v = std::max(max0_v, v0); + uint32_t v1 = pPixels[9 * stride]; min1_v = std::min(min1_v, v1); max1_v = std::max(max1_v, v1); + uint32_t v2 = pPixels[10 * stride]; min2_v = std::min(min2_v, v2); max2_v = std::max(max2_v, v2); + uint32_t v3 = pPixels[11 * stride]; min3_v = std::min(min3_v, v3); max3_v = std::max(max3_v, v3); + } + + { + uint32_t v0 = pPixels[12 * stride]; min0_v = std::min(min0_v, v0); max0_v = std::max(max0_v, v0); + uint32_t v1 = pPixels[13 * stride]; min1_v = std::min(min1_v, v1); max1_v = std::max(max1_v, v1); + uint32_t v2 = pPixels[14 * stride]; min2_v = std::min(min2_v, v2); max2_v = std::max(max2_v, v2); + uint32_t v3 = pPixels[15 * stride]; min3_v = std::min(min3_v, v3); max3_v = std::max(max3_v, v3); + } + + const uint32_t min_v = minimum(min0_v, min1_v, min2_v, min3_v); + const uint32_t max_v = maximum(max0_v, max1_v, max2_v, max3_v); + + uint8_t* pDst_bytes = static_cast(pDst); + pDst_bytes[0] = (uint8_t)max_v; + pDst_bytes[1] = (uint8_t)min_v; + + if (max_v == min_v) + { + memset(pDst_bytes + 2, 0, 6); + return; + } + + const uint32_t delta = max_v - min_v; + + // min_v is now 0. Compute thresholds between values by scaling max_v. It's x14 because we're adding two x7 scale factors. + const int t0 = delta * 13; + const int t1 = delta * 11; + const int t2 = delta * 9; + const int t3 = delta * 7; + const int t4 = delta * 5; + const int t5 = delta * 3; + const int t6 = delta * 1; + + // BC4 floors in its divisions, which we compensate for with the 4 bias. + // This function is optimal for all possible inputs (i.e. it outputs the same results as checking all 8 values and choosing the closest one). + const int bias = 4 - min_v * 14; + + static const uint32_t s_tran0[8] = { 1U , 7U , 6U , 5U , 4U , 3U , 2U , 0U }; + static const uint32_t s_tran1[8] = { 1U << 3U, 7U << 3U, 6U << 3U, 5U << 3U, 4U << 3U, 3U << 3U, 2U << 3U, 0U << 3U }; + static const uint32_t s_tran2[8] = { 1U << 6U, 7U << 6U, 6U << 6U, 5U << 6U, 4U << 6U, 3U << 6U, 2U << 6U, 0U << 6U }; + static const uint32_t s_tran3[8] = { 1U << 9U, 7U << 9U, 6U << 9U, 5U << 9U, 4U << 9U, 3U << 9U, 2U << 9U, 0U << 9U }; + + uint64_t a0, a1, a2, a3; + { + const int v0 = pPixels[0 * stride] * 14 + bias; + const int v1 = pPixels[1 * stride] * 14 + bias; + const int v2 = pPixels[2 * stride] * 14 + bias; + const int v3 = pPixels[3 * stride] * 14 + bias; + a0 = s_tran0[(v0 >= t0) + (v0 >= t1) + (v0 >= t2) + (v0 >= t3) + (v0 >= t4) + (v0 >= t5) + (v0 >= t6)]; + a1 = s_tran1[(v1 >= t0) + (v1 >= t1) + (v1 >= t2) + (v1 >= t3) + (v1 >= t4) + (v1 >= t5) + (v1 >= t6)]; + a2 = s_tran2[(v2 >= t0) + (v2 >= t1) + (v2 >= t2) + (v2 >= t3) + (v2 >= t4) + (v2 >= t5) + (v2 >= t6)]; + a3 = s_tran3[(v3 >= t0) + (v3 >= t1) + (v3 >= t2) + (v3 >= t3) + (v3 >= t4) + (v3 >= t5) + (v3 >= t6)]; + } + + { + const int v0 = pPixels[4 * stride] * 14 + bias; + const int v1 = pPixels[5 * stride] * 14 + bias; + const int v2 = pPixels[6 * stride] * 14 + bias; + const int v3 = pPixels[7 * stride] * 14 + bias; + a0 |= (uint64_t)(s_tran0[(v0 >= t0) + (v0 >= t1) + (v0 >= t2) + (v0 >= t3) + (v0 >= t4) + (v0 >= t5) + (v0 >= t6)] << 12U); + a1 |= (uint64_t)(s_tran1[(v1 >= t0) + (v1 >= t1) + (v1 >= t2) + (v1 >= t3) + (v1 >= t4) + (v1 >= t5) + (v1 >= t6)] << 12U); + a2 |= (uint64_t)(s_tran2[(v2 >= t0) + (v2 >= t1) + (v2 >= t2) + (v2 >= t3) + (v2 >= t4) + (v2 >= t5) + (v2 >= t6)] << 12U); + a3 |= (uint64_t)(s_tran3[(v3 >= t0) + (v3 >= t1) + (v3 >= t2) + (v3 >= t3) + (v3 >= t4) + (v3 >= t5) + (v3 >= t6)] << 12U); + } + + { + const int v0 = pPixels[8 * stride] * 14 + bias; + const int v1 = pPixels[9 * stride] * 14 + bias; + const int v2 = pPixels[10 * stride] * 14 + bias; + const int v3 = pPixels[11 * stride] * 14 + bias; + a0 |= (((uint64_t)s_tran0[(v0 >= t0) + (v0 >= t1) + (v0 >= t2) + (v0 >= t3) + (v0 >= t4) + (v0 >= t5) + (v0 >= t6)]) << 24U); + a1 |= (((uint64_t)s_tran1[(v1 >= t0) + (v1 >= t1) + (v1 >= t2) + (v1 >= t3) + (v1 >= t4) + (v1 >= t5) + (v1 >= t6)]) << 24U); + a2 |= (((uint64_t)s_tran2[(v2 >= t0) + (v2 >= t1) + (v2 >= t2) + (v2 >= t3) + (v2 >= t4) + (v2 >= t5) + (v2 >= t6)]) << 24U); + a3 |= (((uint64_t)s_tran3[(v3 >= t0) + (v3 >= t1) + (v3 >= t2) + (v3 >= t3) + (v3 >= t4) + (v3 >= t5) + (v3 >= t6)]) << 24U); + } + + { + const int v0 = pPixels[12 * stride] * 14 + bias; + const int v1 = pPixels[13 * stride] * 14 + bias; + const int v2 = pPixels[14 * stride] * 14 + bias; + const int v3 = pPixels[15 * stride] * 14 + bias; + a0 |= (((uint64_t)s_tran0[(v0 >= t0) + (v0 >= t1) + (v0 >= t2) + (v0 >= t3) + (v0 >= t4) + (v0 >= t5) + (v0 >= t6)]) << 36U); + a1 |= (((uint64_t)s_tran1[(v1 >= t0) + (v1 >= t1) + (v1 >= t2) + (v1 >= t3) + (v1 >= t4) + (v1 >= t5) + (v1 >= t6)]) << 36U); + a2 |= (((uint64_t)s_tran2[(v2 >= t0) + (v2 >= t1) + (v2 >= t2) + (v2 >= t3) + (v2 >= t4) + (v2 >= t5) + (v2 >= t6)]) << 36U); + a3 |= (((uint64_t)s_tran3[(v3 >= t0) + (v3 >= t1) + (v3 >= t2) + (v3 >= t3) + (v3 >= t4) + (v3 >= t5) + (v3 >= t6)]) << 36U); + } + + const uint64_t f = a0 | a1 | a2 | a3; + + pDst_bytes[2] = (uint8_t)f; + pDst_bytes[3] = (uint8_t)(f >> 8U); + pDst_bytes[4] = (uint8_t)(f >> 16U); + pDst_bytes[5] = (uint8_t)(f >> 24U); + pDst_bytes[6] = (uint8_t)(f >> 32U); + pDst_bytes[7] = (uint8_t)(f >> 40U); + } + + uint32_t encode_bc4_hq(void* pDst, const uint8_t* pPixels, uint32_t stride, uint32_t search_rad, uint32_t mode_flag, const uint8_t* pForce_selectors) + { + assert(mode_flag); + + uint8_t* pDst_bytes = static_cast(pDst); + + uint32_t min_val = 255, max_val = 0; + for (uint32_t i = 0; i < 16; i++) + { + uint32_t val = pPixels[i * stride]; + min_val = std::min(val, min_val); + max_val = std::max(val, max_val); + } + + if (min_val == max_val) + { + if (mode_flag & BC4_USE_MODE6_FLAG) + { + pDst_bytes[0] = (uint8_t)min_val; + pDst_bytes[1] = (uint8_t)min_val; + + memset(pDst_bytes + 2, 0, 6); + + assert(!(pDst_bytes[0] > pDst_bytes[1])); + } + else + { + // Use an 8 value encoding + if (min_val > 0) + { + pDst_bytes[0] = (uint8_t)min_val; + pDst_bytes[1] = (uint8_t)min_val - 1; + + memset(pDst_bytes + 2, 0, 6); + } + else + { + static const uint8_t s_const_1_vals[8] = { 1, 0, 0x49, 0x92, 0x24, 0x49, 0x92, 0x24 }; + memcpy(pDst_bytes, s_const_1_vals, 8); + } + + assert(pDst_bytes[0] > pDst_bytes[1]); + } + +#if defined(_DEBUG) || defined(DEBUG) + { + bc4_block* pBlock = (bc4_block*)pDst; + uint8_t pixels[16]; + unpack_bc4(pDst, pixels, 1); + for (uint32_t i = 0; i < 16; i++) + assert(pixels[i] == min_val); + if (mode_flag & BC4_USE_MODE6_FLAG) + { + assert(pBlock->is_alpha6_block()); + } + else + { + assert(!pBlock->is_alpha6_block()); + } + } +#endif + + return 0; + } + + uint32_t best_err = UINT32_MAX; + for (uint32_t mode = 0; mode < 2; mode++) + { + if ((mode_flag & (1 << mode)) == 0) + continue; + + for (int lo_delta = -(int)search_rad; lo_delta <= (int)search_rad; lo_delta++) + { + for (int hi_delta = -(int)search_rad; hi_delta <= (int)search_rad; hi_delta++) + { + bc4_block trial_block; + trial_block.m_endpoints[0] = (uint8_t)clamp(max_val + hi_delta, 0, 255); + trial_block.m_endpoints[1] = (uint8_t)clamp(min_val + lo_delta, 0, 255); + + if (trial_block.m_endpoints[0] == trial_block.m_endpoints[1]) + continue; + + if (mode == 0) + { + if (trial_block.is_alpha6_block()) + std::swap(trial_block.m_endpoints[0], trial_block.m_endpoints[1]); + } + else if (!trial_block.is_alpha6_block()) + std::swap(trial_block.m_endpoints[0], trial_block.m_endpoints[1]); + + uint8_t block_vals[8]; + trial_block.get_block_values(block_vals, trial_block.m_endpoints[0], trial_block.m_endpoints[1]); + + uint32_t trial_err = 0; + uint8_t trial_sels[16]; + + if (pForce_selectors) + { + memcpy(trial_sels, pForce_selectors, 16); + + for (uint32_t i = 0; i < 16; i++) + trial_err += squarei(block_vals[pForce_selectors[i]] - pPixels[i * stride]); + } + else + { + for (uint32_t i = 0; i < 16; i++) + { + uint32_t best_index_err = UINT32_MAX; + uint32_t best_index = 0; + for (uint32_t j = 0; j < 8; j++) + { + uint32_t err = squarei(block_vals[j] - pPixels[i * stride]); + if (err < best_index_err) + { + best_index_err = err; + best_index = j; + if (!err) + break; + } + } + + trial_err += best_index_err; + if (trial_err >= best_err) + break; + + trial_sels[i] = (uint8_t)best_index; + } // i + } + + if (trial_err < best_err) + { + best_err = trial_err; + + uint64_t sel_vals = 0; + for (uint32_t i = 0; i < 16; i++) + sel_vals |= ((uint64_t)trial_sels[i] << (i * 3)); + + trial_block.m_selectors[0] = (uint8_t)sel_vals; + trial_block.m_selectors[1] = (uint8_t)(sel_vals >> 8); + trial_block.m_selectors[2] = (uint8_t)(sel_vals >> 16); + trial_block.m_selectors[3] = (uint8_t)(sel_vals >> 24); + trial_block.m_selectors[4] = (uint8_t)(sel_vals >> 32); + trial_block.m_selectors[5] = (uint8_t)(sel_vals >> 40); + + memcpy(pDst_bytes, &trial_block, sizeof(bc4_block)); + } // if (trial_err < best_err) + + } // hi_delta + + } // lo_delta + + } // mode + + return best_err; + } + + void encode_bc3(void* pDst, const uint8_t* pPixels, uint32_t flags, uint32_t total_orderings_to_try) + { + assert(g_initialized); + + // 3-color blocks are not allowed with BC3 (on most GPU's). + flags &= ~(cEncodeBC1Use3ColorBlocksForBlackPixels | cEncodeBC1Use3ColorBlocks); + + encode_bc4(pDst, pPixels + 3, 4); + encode_bc1(static_cast(pDst) + 8, pPixels, flags, total_orderings_to_try); + } + + void encode_bc3(uint32_t level, void* pDst, const uint8_t* pPixels) + { + assert(g_initialized); + + encode_bc4(pDst, pPixels + 3, 4); + encode_bc1(level, static_cast(pDst) + 8, pPixels, false, false); + } + + void encode_bc3_hq(uint32_t level, void* pDst, const uint8_t* pPixels, uint32_t alpha_search_rad, uint32_t alpha_modes) + { + assert(g_initialized); + + encode_bc4_hq(pDst, pPixels + 3, 4, alpha_search_rad, alpha_modes); + encode_bc1(level, static_cast(pDst) + 8, pPixels, false, false); + } + + void encode_bc5(void* pDst, const uint8_t* pPixels, uint32_t chan0, uint32_t chan1, uint32_t stride) + { + assert(g_initialized); + + encode_bc4(pDst, pPixels + chan0, stride); + encode_bc4(static_cast(pDst) + 8, pPixels + chan1, stride); + } + + void encode_bc5_hq(void* pDst, const uint8_t* pPixels, uint32_t chan0, uint32_t chan1, uint32_t stride, uint32_t alpha_search_rad, uint32_t alpha_modes) + { + assert(g_initialized); + + encode_bc4_hq(pDst, pPixels + chan0, stride, alpha_search_rad, alpha_modes); + encode_bc4_hq(static_cast(pDst) + 8, pPixels + chan1, stride, alpha_search_rad, alpha_modes); + } + + bool unpack_bc1_block_colors(const void* pBlock_bits, color32* c, bc1_approx_mode mode) + { + const bc1_block* pBlock = static_cast(pBlock_bits); + + const uint32_t l = pBlock->get_low_color(); + const uint32_t h = pBlock->get_high_color(); + + const int cr0 = (l >> 11) & 31; + const int cg0 = (l >> 5) & 63; + const int cb0 = l & 31; + const int r0 = (cr0 << 3) | (cr0 >> 2); + const int g0 = (cg0 << 2) | (cg0 >> 4); + const int b0 = (cb0 << 3) | (cb0 >> 2); + + const int cr1 = (h >> 11) & 31; + const int cg1 = (h >> 5) & 63; + const int cb1 = h & 31; + const int r1 = (cr1 << 3) | (cr1 >> 2); + const int g1 = (cg1 << 2) | (cg1 >> 4); + const int b1 = (cb1 << 3) | (cb1 >> 2); + + bool used_punchthrough = false; + + if (l > h) + { + c[0].set_noclamp_rgba(r0, g0, b0, 255); + c[1].set_noclamp_rgba(r1, g1, b1, 255); + switch (mode) + { + case bc1_approx_mode::cBC1Ideal: + c[2].set_noclamp_rgba((r0 * 2 + r1) / 3, (g0 * 2 + g1) / 3, (b0 * 2 + b1) / 3, 255); + c[3].set_noclamp_rgba((r1 * 2 + r0) / 3, (g1 * 2 + g0) / 3, (b1 * 2 + b0) / 3, 255); + break; + case bc1_approx_mode::cBC1IdealRound4: + c[2].set_noclamp_rgba((r0 * 2 + r1 + 1) / 3, (g0 * 2 + g1 + 1) / 3, (b0 * 2 + b1 + 1) / 3, 255); + c[3].set_noclamp_rgba((r1 * 2 + r0 + 1) / 3, (g1 * 2 + g0 + 1) / 3, (b1 * 2 + b0 + 1) / 3, 255); + break; + case bc1_approx_mode::cBC1NVidia: + c[2].set_noclamp_rgba(interp_5_nv(cr0, cr1), interp_6_nv(g0, g1), interp_5_nv(cb0, cb1), 255); + c[3].set_noclamp_rgba(interp_5_nv(cr1, cr0), interp_6_nv(g1, g0), interp_5_nv(cb1, cb0), 255); + break; + case bc1_approx_mode::cBC1AMD: + c[2].set_noclamp_rgba(interp_5_6_amd(r0, r1), interp_5_6_amd(g0, g1), interp_5_6_amd(b0, b1), 255); + c[3].set_noclamp_rgba(interp_5_6_amd(r1, r0), interp_5_6_amd(g1, g0), interp_5_6_amd(b1, b0), 255); + break; + } + } + else + { + c[0].set_noclamp_rgba(r0, g0, b0, 255); + c[1].set_noclamp_rgba(r1, g1, b1, 255); + switch (mode) + { + case bc1_approx_mode::cBC1Ideal: + case bc1_approx_mode::cBC1IdealRound4: + c[2].set_noclamp_rgba((r0 + r1) / 2, (g0 + g1) / 2, (b0 + b1) / 2, 255); + break; + case bc1_approx_mode::cBC1NVidia: + c[2].set_noclamp_rgba(interp_half_5_nv(cr0, cr1), interp_half_6_nv(g0, g1), interp_half_5_nv(cb0, cb1), 255); + break; + case bc1_approx_mode::cBC1AMD: + c[2].set_noclamp_rgba(interp_half_5_6_amd(r0, r1), interp_half_5_6_amd(g0, g1), interp_half_5_6_amd(b0, b1), 255); + break; + } + + c[3].set_noclamp_rgba(0, 0, 0, 0); + used_punchthrough = true; + } + + return used_punchthrough; + } + + // Returns true if the block uses 3 color punchthrough alpha mode. + bool unpack_bc1(const void* pBlock_bits, void* pPixels, bool set_alpha, bc1_approx_mode mode) + { + color32* pDst_pixels = static_cast(pPixels); + + static_assert(sizeof(bc1_block) == 8, "sizeof(bc1_block) == 8"); + static_assert(sizeof(bc4_block) == 8, "sizeof(bc4_block) == 8"); + + const bc1_block* pBlock = static_cast(pBlock_bits); + + color32 c[4]; + const bool used_punchthrough = unpack_bc1_block_colors(pBlock_bits, c, mode); + + if (set_alpha) + { + for (uint32_t y = 0; y < 4; y++, pDst_pixels += 4) + { + pDst_pixels[0] = c[pBlock->get_selector(0, y)]; + pDst_pixels[1] = c[pBlock->get_selector(1, y)]; + pDst_pixels[2] = c[pBlock->get_selector(2, y)]; + pDst_pixels[3] = c[pBlock->get_selector(3, y)]; + } + } + else + { + for (uint32_t y = 0; y < 4; y++, pDst_pixels += 4) + { + pDst_pixels[0].set_rgb(c[pBlock->get_selector(0, y)]); + pDst_pixels[1].set_rgb(c[pBlock->get_selector(1, y)]); + pDst_pixels[2].set_rgb(c[pBlock->get_selector(2, y)]); + pDst_pixels[3].set_rgb(c[pBlock->get_selector(3, y)]); + } + } + + return used_punchthrough; + } + + void unpack_bc4(const void* pBlock_bits, uint8_t* pPixels, uint32_t stride) + { + static_assert(sizeof(bc4_block) == 8, "sizeof(bc4_block) == 8"); + + const bc4_block* pBlock = static_cast(pBlock_bits); + + uint8_t sel_values[8]; + bc4_block::get_block_values(sel_values, pBlock->get_low_alpha(), pBlock->get_high_alpha()); + + const uint64_t selector_bits = pBlock->get_selector_bits(); + + for (uint32_t y = 0; y < 4; y++, pPixels += (stride * 4U)) + { + pPixels[0] = sel_values[pBlock->get_selector(0, y, selector_bits)]; + pPixels[stride * 1] = sel_values[pBlock->get_selector(1, y, selector_bits)]; + pPixels[stride * 2] = sel_values[pBlock->get_selector(2, y, selector_bits)]; + pPixels[stride * 3] = sel_values[pBlock->get_selector(3, y, selector_bits)]; + } + } + + // Returns false if the block uses 3-color punchthrough alpha mode, which isn't supported on some GPU's for BC3. + bool unpack_bc3(const void* pBlock_bits, void* pPixels, bc1_approx_mode mode) + { + color32* pDst_pixels = static_cast(pPixels); + + bool success = true; + + if (unpack_bc1((const uint8_t*)pBlock_bits + sizeof(bc4_block), pDst_pixels, true, mode)) + success = false; + + unpack_bc4(pBlock_bits, &pDst_pixels[0].a, sizeof(color32)); + + return success; + } + + // writes RG + void unpack_bc5(const void* pBlock_bits, void* pPixels, uint32_t chan0, uint32_t chan1, uint32_t stride) + { + unpack_bc4(pBlock_bits, (uint8_t*)pPixels + chan0, stride); + unpack_bc4((const uint8_t*)pBlock_bits + sizeof(bc4_block), (uint8_t*)pPixels + chan1, stride); + } + +} // namespace rgbcx + + + diff --git a/src/gfx/tex/bc7enc/rgbcx.h b/src/gfx/tex/bc7enc/rgbcx.h new file mode 100644 index 0000000..2d96c69 --- /dev/null +++ b/src/gfx/tex/bc7enc/rgbcx.h @@ -0,0 +1,473 @@ +// rgbcx.h v1.13 +// High-performance scalar encoders and RDO (Rate Distortion Optimization) post processors for BC1-5. +// Public Domain or MIT license (you choose - see below), written by Richard Geldreich 2020 . +// +// Influential references: +// https://tinyurl.com/y3vxz457 (Ortego and Ramchandran, "Rate-distortion Methods for Image and Video Compression", 1998) +// http://sjbrown.co.uk/2006/01/19/dxt-compression-techniques/ +// https://github.com/nothings/stb/blob/master/stb_dxt.h +// https://gist.github.com/castano/c92c7626f288f9e99e158520b14a61cf +// https://github.com/castano/icbc/blob/master/icbc.h +// http://www.humus.name/index.php?page=3D&ID=79 +// +// This is a single header file library. Be sure to "#define RGBCX_IMPLEMENTATION" in one .cpp file somewhere. +// +// Instructions: +// +// The library MUST be initialized by calling this function at least once before using any encoder or decoder functions: +// +// void rgbcx::init(bc1_approx_mode mode = cBC1Ideal); +// +// This function manipulates global state, so it is not thread safe. +// You can call it multiple times to change the global BC1 approximation mode. +// Important: BC1/3 textures encoded using non-ideal BC1 approximation modes should only be sampled on parts from that vendor. +// If you encode for AMD, average error on AMD parts will go down, but average error on NVidia parts will go up and vice versa. +// If in doubt, encode in ideal BC1 mode. +// +// Call these functions to encode BC1-5: +// void rgbcx::encode_bc1(uint32_t level, void* pDst, const uint8_t* pPixels, bool allow_3color, bool use_transparent_texels_for_black); +// void rgbcx::encode_bc3(uint32_t level, void* pDst, const uint8_t* pPixels); +// void rgbcx::encode_bc4(void* pDst, const uint8_t* pPixels, uint32_t stride = 4); +// void rgbcx::encode_bc5(void* pDst, const uint8_t* pPixels, uint32_t chan0 = 0, uint32_t chan1 = 1, uint32_t stride = 4); +// +// - level ranges from MIN_LEVEL to MAX_LEVEL. The higher the level, the slower the encoder goes, but the higher the average quality. +// levels [0,4] are fast and compete against stb_dxt (default and HIGHQUAL). The remaining levels compete against squish/NVTT/icbc and icbc HQ. +// If in doubt just use level 10, set allow_3color to true and use_transparent_texels_for_black to false, and adjust as needed. +// +// - pDst is a pointer to the 8-byte (BC1/4) or 16-byte (BC3/5) destination block. +// +// - pPixels is a pointer to the 32-bpp pixels, in either RGBX or RGBA format (R is first in memory). +// Alpha is always ignored by encode_bc1(). +// +// - allow_3color: If true the encoder will use 3-color blocks. This flag is ignored unless level is >= 5 (because lower levels compete against stb_dxt and it doesn't support 3-color blocks). +// Do not enable on BC3-5 textures. 3-color block usage slows down encoding. +// +// - use_transparent_texels_for_black: If true the encoder will use 3-color block transparent black pixels to code very dark or black texels. Your engine/shader MUST ignore the sampled +// alpha value for textures encoded in this mode. This is how NVidia's classic "nvdxt" encoder (used by many original Xbox titles) used to work by default on DXT1C textures. It increases +// average quality substantially (because dark texels/black are very common) and is highly recommended. +// Do not enable on BC3-5 textures. +// +// - stride is the source pixel stride, in bytes. It's typically 4. +// +// - chan0 and chan1 are the source channels. Typically they will be 0 and 1. +// +// All encoding and decoding functions are threade-safe. +// +// To reduce the compiled size of the encoder, set #define RGBCX_USE_SMALLER_TABLES to 1 before including this header. +// +#ifndef RGBCX_INCLUDE_H +#define RGBCX_INCLUDE_H + +#ifdef _MSC_VER +#pragma warning (disable:4201) //nameless struct/union +#endif + +#include +#include +#include +#include +#include + +// By default, the table used to accelerate cluster fit on 4 color blocks uses a 969x128 entry table. +// To reduce the executable size, set RGBCX_USE_SMALLER_TABLES to 1, which selects the smaller 969x32 entry table. +#ifndef RGBCX_USE_SMALLER_TABLES +#define RGBCX_USE_SMALLER_TABLES 0 +#endif + +namespace rgbcx +{ + enum class bc1_approx_mode + { + // The default mode. No rounding for 4-color colors 2,3. My older tools/compressors use this mode. + // This matches the D3D10 docs on BC1. + cBC1Ideal = 0, + + // NVidia GPU mode. + cBC1NVidia = 1, + + // AMD GPU mode. + cBC1AMD = 2, + + // This mode matches AMD Compressonator's output. It rounds 4-color colors 2,3 (not 3-color color 2). + // This matches the D3D9 docs on DXT1. + cBC1IdealRound4 = 3 + }; + + enum class eNoClamp { cNoClamp }; + static inline uint8_t clamp255(int32_t i) { return (uint8_t)((i & 0xFFFFFF00U) ? (~(i >> 31)) : i); } + + template inline S maximum(S a, S b) { return (a > b) ? a : b; } + template inline S maximum(S a, S b, S c) { return maximum(maximum(a, b), c); } + template inline S maximum(S a, S b, S c, S d) { return maximum(maximum(maximum(a, b), c), d); } + + template inline S minimum(S a, S b) { return (a < b) ? a : b; } + template inline S minimum(S a, S b, S c) { return minimum(minimum(a, b), c); } + template inline S minimum(S a, S b, S c, S d) { return minimum(minimum(minimum(a, b), c), d); } + + struct color32 + { + union + { + struct + { + uint8_t r; + uint8_t g; + uint8_t b; + uint8_t a; + }; + + uint8_t c[4]; + + uint32_t m; + }; + + color32() { } + + color32(uint32_t vr, uint32_t vg, uint32_t vb, uint32_t va) { set(vr, vg, vb, va); } + color32(eNoClamp unused, uint32_t vr, uint32_t vg, uint32_t vb, uint32_t va) { (void)unused; set_noclamp_rgba(vr, vg, vb, va); } + + void set(uint32_t vr, uint32_t vg, uint32_t vb, uint32_t va) { c[0] = static_cast(vr); c[1] = static_cast(vg); c[2] = static_cast(vb); c[3] = static_cast(va); } + + void set_noclamp_rgb(uint32_t vr, uint32_t vg, uint32_t vb) { c[0] = static_cast(vr); c[1] = static_cast(vg); c[2] = static_cast(vb); } + void set_noclamp_rgba(uint32_t vr, uint32_t vg, uint32_t vb, uint32_t va) { set(vr, vg, vb, va); } + + void set_clamped(int vr, int vg, int vb, int va) { c[0] = clamp255(vr); c[1] = clamp255(vg); c[2] = clamp255(vb); c[3] = clamp255(va); } + + uint8_t operator[] (uint32_t idx) const { assert(idx < 4); return c[idx]; } + uint8_t& operator[] (uint32_t idx) { assert(idx < 4); return c[idx]; } + + bool operator== (const color32& rhs) const { return m == rhs.m; } + + void set_rgb(const color32& other) { c[0] = static_cast(other.c[0]); c[1] = static_cast(other.c[1]); c[2] = static_cast(other.c[2]); } + + static color32 comp_min(const color32& a, const color32& b) { return color32(eNoClamp::cNoClamp, std::min(a[0], b[0]), std::min(a[1], b[1]), std::min(a[2], b[2]), std::min(a[3], b[3])); } + static color32 comp_max(const color32& a, const color32& b) { return color32(eNoClamp::cNoClamp, std::max(a[0], b[0]), std::max(a[1], b[1]), std::max(a[2], b[2]), std::max(a[3], b[3])); } + }; + + // init() MUST be called once before using the BC1 encoder. + // This function may be called multiple times to change the BC1 approximation mode. + // This function initializes global state, so don't call it while other threads inside the encoder. + // Important: If you encode textures for a specific vendor's GPU's, beware that using that texture data on other GPU's may result in ugly artifacts. + // Encode to cBC1Ideal unless you know the texture data will only be deployed or used on a specific vendor's GPU. + void init(bc1_approx_mode mode = bc1_approx_mode::cBC1Ideal); + + // Optimally encodes a solid color block to BC1 format. + void encode_bc1_solid_block(void* pDst, uint32_t fr, uint32_t fg, uint32_t fb, bool allow_3color); + + // BC1 low-level API encoder flags. You can ignore this if you use the simple level API. + enum + { + // Try to improve quality using the most likely total orderings. + // The total_orderings_to_try parameter will then control the number of total orderings to try for 4 color blocks, and the + // total_orderings_to_try3 parameter will control the number of total orderings to try for 3 color blocks (if they are enabled). + cEncodeBC1UseLikelyTotalOrderings = 2, + + // Use 2 least squares pass, instead of one (same as stb_dxt's HIGHQUAL option). + // Recommended if you're enabling cEncodeBC1UseLikelyTotalOrderings. + cEncodeBC1TwoLeastSquaresPasses = 4, + + // cEncodeBC1Use3ColorBlocksForBlackPixels allows the BC1 encoder to use 3-color blocks for blocks containing black or very dark pixels. + // You shader/engine MUST ignore the alpha channel on textures encoded with this flag. + // Average quality goes up substantially for my 100 texture corpus (~.5 dB), so it's worth using if you can. + // Note the BC1 encoder does not actually support transparency in 3-color mode. + // Don't set when encoding to BC3. + cEncodeBC1Use3ColorBlocksForBlackPixels = 8, + + // If cEncodeBC1Use3ColorBlocks is set, the encoder can use 3-color mode for a small but noticeable gain in average quality, but lower perf. + // If you also specify the cEncodeBC1UseLikelyTotalOrderings flag, set the total_orderings_to_try3 paramter to the number of total orderings to try. + // Don't set when encoding to BC3. + cEncodeBC1Use3ColorBlocks = 16, + + // cEncodeBC1Iterative will greatly increase encode time, but is very slightly higher quality. + // Same as squish's iterative cluster fit option. Not really worth the tiny boost in quality, unless you just don't care about perf. at all. + cEncodeBC1Iterative = 32, + + // cEncodeBC1BoundingBox enables a fast all-integer PCA approximation on 4-color blocks. + // At level 0 options (no other flags), this is ~15% faster, and higher *average* quality. + cEncodeBC1BoundingBox = 64, + + // Use a slightly lower quality, but ~30% faster MSE evaluation function for 4-color blocks. + cEncodeBC1UseFasterMSEEval = 128, + + // Examine all colors to compute selectors/MSE (slower than default) + cEncodeBC1UseFullMSEEval = 256, + + // Use 2D least squares+inset+optimal rounding (the method used in Humus's GPU texture encoding demo), instead of PCA. + // Around 18% faster, very slightly lower average quality to better (depends on the content). + cEncodeBC1Use2DLS = 512, + + // Use 6 power iterations vs. 4 for PCA. + cEncodeBC1Use6PowerIters = 2048, + + // Check all total orderings - *very* slow. The encoder is not designed to be used in this way. + cEncodeBC1Exhaustive = 8192, + + // Try 2 different ways of choosing the initial endpoints. + cEncodeBC1TryAllInitialEndponts = 16384, + + // Same as cEncodeBC1BoundingBox, but implemented using integer math (faster, slightly less quality) + cEncodeBC1BoundingBoxInt = 32768, + + // Try refining the final endpoints by examining nearby colors. + cEncodeBC1EndpointSearchRoundsShift = 22, + cEncodeBC1EndpointSearchRoundsMask = 1023U << cEncodeBC1EndpointSearchRoundsShift, + }; + + const uint32_t MIN_TOTAL_ORDERINGS = 1; + const uint32_t MAX_TOTAL_ORDERINGS3 = 32; + +#if RGBCX_USE_SMALLER_TABLES + const uint32_t MAX_TOTAL_ORDERINGS4 = 32; +#else + const uint32_t MAX_TOTAL_ORDERINGS4 = 128; +#endif + + // DEFAULT_TOTAL_ORDERINGS_TO_TRY is around 3x faster than libsquish at slightly higher average quality. 10-16 is a good range to start to compete against libsquish. + const uint32_t DEFAULT_TOTAL_ORDERINGS_TO_TRY = 10; + + const uint32_t DEFAULT_TOTAL_ORDERINGS_TO_TRY3 = 1; + + // Encodes a 4x4 block of RGBX (X=ignored) pixels to BC1 format. + // This is the simplified interface for BC1 encoding, which accepts a level parameter and converts that to the best overall flags. + // The pixels are in RGBA format, where R is first in memory. The BC1 encoder completely ignores the alpha channel (i.e. there is no punchthrough alpha support). + // This is the recommended function to use for BC1 encoding, becuase it configures the encoder for you in the best possible way (on average). + // Note that the 3 color modes won't be used at all until level 5 or higher. + // No transparency supported, however if you set use_transparent_texels_for_black to true the encocer will use transparent selectors on very dark/black texels to reduce MSE. + const uint32_t MIN_LEVEL = 0, MAX_LEVEL = 18; + void encode_bc1(uint32_t level, void* pDst, const uint8_t* pPixels, bool allow_3color, bool use_transparent_texels_for_black, const uint8_t* pForce_selectors = nullptr); + + // Low-level interface for BC1 encoding. + // Always returns a 4 color block, unless cEncodeBC1Use3ColorBlocksForBlackPixels or cEncodeBC1Use3ColorBlock flags are specified. + // total_orderings_to_try controls the perf. vs. quality tradeoff on 4-color blocks when the cEncodeBC1UseLikelyTotalOrderings flag is used. It must range between [MIN_TOTAL_ORDERINGS, MAX_TOTAL_ORDERINGS4]. + // total_orderings_to_try3 controls the perf. vs. quality tradeoff on 3-color bocks when the cEncodeBC1UseLikelyTotalOrderings and the cEncodeBC1Use3ColorBlocks flags are used. Valid range is [0,MAX_TOTAL_ORDERINGS3] (0=disabled). + void encode_bc1(void* pDst, const uint8_t* pPixels, uint32_t flags = 0, uint32_t total_orderings_to_try = DEFAULT_TOTAL_ORDERINGS_TO_TRY, uint32_t total_orderings_to_try3 = DEFAULT_TOTAL_ORDERINGS_TO_TRY3, const uint8_t *pForce_selectors = nullptr); + + // Constants used for high quality BC4/BC5 encoding (and alpha of BC3) + const uint32_t BC4_DEFAULT_SEARCH_RAD = 3; + const uint32_t BC4_USE_MODE8_FLAG = 1; + const uint32_t BC4_USE_MODE6_FLAG = 2; + const uint32_t BC4_USE_ALL_MODES = 3; + + // Encodes a 4x4 block of RGBA pixels to BC3 format. + // There are two encode_bc3() functions. + // The first is the recommended function, which accepts a level parameter. + // The second is a low-level version that allows fine control over BC1 encoding. + void encode_bc3(uint32_t level, void* pDst, const uint8_t* pPixels); + void encode_bc3(void* pDst, const uint8_t* pPixels, uint32_t flags = 0, uint32_t total_orderings_to_try = DEFAULT_TOTAL_ORDERINGS_TO_TRY); + void encode_bc3_hq(uint32_t level, void* pDst, const uint8_t* pPixels, uint32_t alpha_search_rad = BC4_DEFAULT_SEARCH_RAD, uint32_t alpha_modes = BC4_USE_ALL_MODES); + + // Encodes a single channel to BC4. + // stride is the source pixel stride in bytes. + void encode_bc4(void* pDst, const uint8_t* pPixels, uint32_t stride = 4); + uint32_t encode_bc4_hq(void* pDst, const uint8_t* pPixels, uint32_t stride = 4, uint32_t search_rad = BC4_DEFAULT_SEARCH_RAD, uint32_t mode_flag = BC4_USE_ALL_MODES, const uint8_t* pForce_selectors = nullptr); + + // Encodes two channels to BC5. + // chan0/chan1 control which channels, stride is the source pixel stride in bytes. + void encode_bc5(void* pDst, const uint8_t* pPixels, uint32_t chan0 = 0, uint32_t chan1 = 1, uint32_t stride = 4); + void encode_bc5_hq(void* pDst, const uint8_t* pPixels, uint32_t chan0 = 0, uint32_t chan1 = 1, uint32_t stride = 4, uint32_t alpha_search_rad = BC4_DEFAULT_SEARCH_RAD, uint32_t alpha_modes = BC4_USE_ALL_MODES); + + // Decompression functions. + + bool unpack_bc1_block_colors(const void* pBlock_bits, color32* c, bc1_approx_mode mode = bc1_approx_mode::cBC1Ideal); + + // Returns true if the block uses 3 color punchthrough alpha mode. + bool unpack_bc1(const void* pBlock_bits, void* pPixels, bool set_alpha = true, bc1_approx_mode mode = bc1_approx_mode::cBC1Ideal); + + void unpack_bc4(const void* pBlock_bits, uint8_t* pPixels, uint32_t stride = 4); + + // Returns true if the block uses 3 color punchthrough alpha mode. + bool unpack_bc3(const void* pBlock_bits, void* pPixels, bc1_approx_mode mode = bc1_approx_mode::cBC1Ideal); + + void unpack_bc5(const void* pBlock_bits, void* pPixels, uint32_t chan0 = 0, uint32_t chan1 = 1, uint32_t stride = 4); + + // Rate Distortion Optimization (RDO) + enum dxt_constants + { + cDXT1SelectorBits = 2U, cDXT1SelectorValues = 1U << cDXT1SelectorBits, cDXT1SelectorMask = cDXT1SelectorValues - 1U, + cDXT5SelectorBits = 3U, cDXT5SelectorValues = 1U << cDXT5SelectorBits, cDXT5SelectorMask = cDXT5SelectorValues - 1U, + }; + + struct bc1_block + { + enum { cTotalEndpointBytes = 2, cTotalSelectorBytes = 4 }; + + uint8_t m_low_color[cTotalEndpointBytes]; + uint8_t m_high_color[cTotalEndpointBytes]; + uint8_t m_selectors[cTotalSelectorBytes]; + + inline uint32_t get_low_color() const { return m_low_color[0] | (m_low_color[1] << 8U); } + inline uint32_t get_high_color() const { return m_high_color[0] | (m_high_color[1] << 8U); } + inline bool is_3color() const { return get_low_color() <= get_high_color(); } + inline void set_low_color(uint16_t c) { m_low_color[0] = static_cast(c & 0xFF); m_low_color[1] = static_cast((c >> 8) & 0xFF); } + inline void set_high_color(uint16_t c) { m_high_color[0] = static_cast(c & 0xFF); m_high_color[1] = static_cast((c >> 8) & 0xFF); } + inline uint32_t get_selector(uint32_t x, uint32_t y) const { assert((x < 4U) && (y < 4U)); return (m_selectors[y] >> (x * cDXT1SelectorBits)) & cDXT1SelectorMask; } + inline void set_selector(uint32_t x, uint32_t y, uint32_t val) { assert((x < 4U) && (y < 4U) && (val < 4U)); m_selectors[y] &= (~(cDXT1SelectorMask << (x * cDXT1SelectorBits))); m_selectors[y] |= (val << (x * cDXT1SelectorBits)); } + + inline uint32_t get_endpoint_bits() const { return m_low_color[0] | (m_low_color[1] << 8) | (m_high_color[0] << 16) | (m_high_color[1] << 24); } + inline void set_endpoint_bits(uint32_t s) { m_low_color[0] = (uint8_t)s; m_low_color[1] = (uint8_t)(s >> 8); m_high_color[0] = (uint8_t)(s >> 16); m_high_color[1] = (uint8_t)(s >> 24); } + + inline uint32_t get_selector_bits() const { return m_selectors[0] | (m_selectors[1] << 8) | (m_selectors[2] << 16) | (m_selectors[3] << 24); } + inline void set_selector_bits(uint32_t s) { m_selectors[0] = (uint8_t)s; m_selectors[1] = (uint8_t)(s >> 8); m_selectors[2] = (uint8_t)(s >> 16); m_selectors[3] = (uint8_t)(s >> 24); } + + inline bool any_selectors_transparent() const + { + uint32_t sel_bits = get_selector_bits(); + for (uint32_t i = 0; i < 16; i++) + { + if ((sel_bits & 3) == 3) + return true; + + sel_bits >>= 2; + } + return false; + } + + static inline uint16_t pack_color(const color32& color, bool scaled, uint32_t bias = 127U) + { + uint32_t r = color.r, g = color.g, b = color.b; + if (scaled) + { + r = (r * 31U + bias) / 255U; + g = (g * 63U + bias) / 255U; + b = (b * 31U + bias) / 255U; + } + return static_cast(minimum(b, 31U) | (minimum(g, 63U) << 5U) | (minimum(r, 31U) << 11U)); + } + + static inline uint16_t pack_unscaled_color(uint32_t r, uint32_t g, uint32_t b) { return static_cast(b | (g << 5U) | (r << 11U)); } + + static inline void unpack_color(uint32_t c, uint32_t& r, uint32_t& g, uint32_t& b) + { + r = (c >> 11) & 31; + g = (c >> 5) & 63; + b = c & 31; + + r = (r << 3) | (r >> 2); + g = (g << 2) | (g >> 4); + b = (b << 3) | (b >> 2); + } + + static inline void unpack_color_unscaled(uint32_t c, uint32_t& r, uint32_t& g, uint32_t& b) + { + r = (c >> 11) & 31; + g = (c >> 5) & 63; + b = c & 31; + } + }; + + struct bc4_block + { + enum { cBC4SelectorBits = 3, cTotalSelectorBytes = 6, cMaxSelectorValues = 8 }; + uint8_t m_endpoints[2]; + + uint8_t m_selectors[cTotalSelectorBytes]; + + inline uint32_t get_low_alpha() const { return m_endpoints[0]; } + inline uint32_t get_high_alpha() const { return m_endpoints[1]; } + inline bool is_alpha6_block() const { return get_low_alpha() <= get_high_alpha(); } + + inline uint64_t get_selector_bits() const + { + return ((uint64_t)((uint32_t)m_selectors[0] | ((uint32_t)m_selectors[1] << 8U) | ((uint32_t)m_selectors[2] << 16U) | ((uint32_t)m_selectors[3] << 24U))) | + (((uint64_t)m_selectors[4]) << 32U) | + (((uint64_t)m_selectors[5]) << 40U); + } + + inline void set_selector_bits(uint64_t v) + { + for (uint32_t i = 0; i < 6; i++) + { + m_selectors[i] = (uint8_t)v; + v >>= 8; + } + } + + inline uint32_t get_selector(uint32_t x, uint32_t y, uint64_t selector_bits) const + { + assert((x < 4U) && (y < 4U)); + return (selector_bits >> (((y * 4) + x) * cBC4SelectorBits)) & (cMaxSelectorValues - 1); + } + + static inline uint32_t get_block_values6(uint8_t* pDst, uint32_t l, uint32_t h) + { + pDst[0] = static_cast(l); + pDst[1] = static_cast(h); + pDst[2] = static_cast((l * 4 + h) / 5); + pDst[3] = static_cast((l * 3 + h * 2) / 5); + pDst[4] = static_cast((l * 2 + h * 3) / 5); + pDst[5] = static_cast((l + h * 4) / 5); + pDst[6] = 0; + pDst[7] = 255; + return 6; + } + + static inline uint32_t get_block_values8(uint8_t* pDst, uint32_t l, uint32_t h) + { + pDst[0] = static_cast(l); + pDst[1] = static_cast(h); + pDst[2] = static_cast((l * 6 + h) / 7); + pDst[3] = static_cast((l * 5 + h * 2) / 7); + pDst[4] = static_cast((l * 4 + h * 3) / 7); + pDst[5] = static_cast((l * 3 + h * 4) / 7); + pDst[6] = static_cast((l * 2 + h * 5) / 7); + pDst[7] = static_cast((l + h * 6) / 7); + return 8; + } + + static inline uint32_t get_block_values(uint8_t* pDst, uint32_t l, uint32_t h) + { + if (l > h) + return get_block_values8(pDst, l, h); + else + return get_block_values6(pDst, l, h); + } + }; + +} +#endif // #ifndef RGBCX_INCLUDE_H + +#ifdef RGBCX_IMPLEMENTATION +#endif //#ifdef RGBCX_IMPLEMENTATION + +/* +------------------------------------------------------------------------------ +This software is available under 2 licenses -- choose whichever you prefer. +------------------------------------------------------------------------------ +ALTERNATIVE A - MIT License +Copyright(c) 2020 Richard Geldreich, Jr. +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files(the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and / or sell copies +of the Software, and to permit persons to whom the Software is furnished to do +so, subject to the following conditions : +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. +------------------------------------------------------------------------------ +ALTERNATIVE B - Public Domain(www.unlicense.org) +This is free and unencumbered software released into the public domain. +Anyone is free to copy, modify, publish, use, compile, sell, or distribute this +software, either in source code form or as a compiled binary, for any purpose, +commercial or non - commercial, and by any means. +In jurisdictions that recognize copyright laws, the author or authors of this +software dedicate any and all copyright interest in the software to the public +domain.We make this dedication for the benefit of the public at large and to +the detriment of our heirs and successors.We intend this dedication to be an +overt act of relinquishment in perpetuity of all present and future rights to +this software under copyright law. +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE +AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN +ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +------------------------------------------------------------------------------ +*/ + diff --git a/src/gfx/tex/bc7enc/rgbcx_table4.h b/src/gfx/tex/bc7enc/rgbcx_table4.h new file mode 100644 index 0000000..2342872 --- /dev/null +++ b/src/gfx/tex/bc7enc/rgbcx_table4.h @@ -0,0 +1,969 @@ +{ 202,403,51,450,318,23,120,13,494,5,523,260,77,21,15,128,48,141,33,817,137,269,12,102,1,475,4,326,7,32,453,64,196,115,730,49,165,2,82,342,6,401,180,31,899,59,402,351,197,17,125,457,14,386,40,3,61,58,37,221,95,16,99,719,98,20,10,854,117,153,27,576,134,341,308,216,11,194,30,24,922,285,9,50,86,65,19,615,391,44,75,237,177,90,352,93,56,22,18,116,70,96,247,291,45,110,105,60,57,54,43,36,0,217,421,203,46,629,569,365,210,92,69,66,38,28,144,551 }, +{ 13,115,51,23,141,217,269,137,318,202,77,21,317,5,64,4,291,2,352,7,32,33,6,372,0,48,365,3,102,349,12,341,177,180,10,237,9,221,403,165,752,22,11,244,28,308,15,128,247,197,49,16,494,125,98,40,120,498,31,90,304,143,95,93,437,196,144,86,44,864,153,18,678,342,901,39,24,30,203,160,117,105,475,14,482,253,457,873,59,35,579,36,569,450,206,72,99,17,717,401,241,453,110,546,201,83,421,61,37,326,260,509,284,275,96,8,109,57,391,386,468,134,82,961,822,576,89,68 }, +{ 29,58,262,74,287,52,151,331,171,500,396,334,75,27,635,220,53,1,26,409,6,324,5,70,145,223,163,112,73,50,45,549,445,113,297,19,675,651,46,61,502,307,14,17,34,573,186,80,394,436,583,315,138,38,21,767,250,491,820,350,64,2,32,357,42,852,25,764,309,67,802,96,631,48,606,780,243,881,103,241,115,807,13,4,155,181,11,106,611,614,278,153,23,108,69,845,841,308,400,72,853,200,66,196,226,237,587,632,24,281,785,133,128,125,117,380,95,474,294,83,197,90,161,134,366,236,760,311 }, +{ 40,51,62,33,453,14,23,56,730,196,475,99,12,34,139,775,153,403,585,440,102,38,130,47,162,17,108,880,117,11,202,57,361,84,123,27,184,236,4,128,19,2,670,136,54,26,111,134,13,161,277,719,152,6,377,183,5,59,211,53,412,485,46,328,77,711,78,21,556,452,65,283,421,129,37,514,507,402,176,29,25,229,401,488,757,45,76,205,899,322,32,177,494,527,383,497,457,249,300,114,215,555,386,119,133,0,349,115,3,862,1,165,73,41,523,72,200,64,291,217,80,305,22,318,194,308,18,714 }, +{ 33,51,102,202,23,128,196,13,64,153,40,12,141,99,403,15,10,269,77,139,475,453,5,318,117,2,14,134,1,86,165,120,401,3,0,22,137,6,197,277,17,402,180,21,421,494,37,115,31,82,38,9,95,342,93,7,110,351,125,457,65,36,361,49,54,19,260,111,44,291,237,28,8,11,194,961,719,352,217,18,485,523,308,730,386,326,210,450,328,300,98,899,27,32,341,48,25,670,34,50,629,26,24,503,498,285,317,67,817,143,365,177,509,41,164,116,757,55,46,203,569,898,373,114,144,16,349,775 }, +{ 13,23,51,115,77,177,202,4,141,32,40,33,217,64,128,90,269,102,59,318,11,291,0,137,403,22,21,7,165,197,15,210,247,180,16,117,153,352,14,2,125,494,10,49,237,18,95,453,6,9,1,24,28,349,475,216,12,342,317,304,3,196,86,98,275,134,31,17,35,25,365,308,19,341,44,36,372,30,93,20,401,76,241,402,139,752,57,244,105,386,37,26,260,110,288,120,116,48,326,421,221,285,901,38,99,82,498,39,468,55,523,143,111,482,65,300,214,450,203,457,351,43,29,27,8,576,678,569 }, +{ 235,26,19,648,624,630,78,47,122,444,145,27,665,112,85,42,390,416,813,113,130,65,680,64,711,25,453,108,644,6,53,73,138,256,95,412,22,730,29,289,90,194,34,525,74,690,32,297,128,45,196,250,287,133,445,86,283,214,236,237,137,222,884,200,746,59,52,312,77,280,632,0,17,114,835,51,123,14,288,881,28,307,180,57,381,141,183,70,4,472,10,33,76,337,3,389,247,315,2,18,368,849,874,573,23,58,544,153,151,436,102,322,99,352,36,1,177,714,719,125,597,63,165,519,285,49,13,396 }, +{ 64,352,751,141,217,247,237,304,269,954,437,372,177,947,312,86,318,349,180,875,95,864,291,597,498,128,678,77,232,92,115,317,752,32,202,588,59,197,579,90,638,392,652,76,210,546,260,1,448,126,340,164,18,303,13,165,275,14,582,120,633,205,179,69,156,296,35,230,391,888,267,572,28,16,379,851,660,96,717,314,0,23,863,49,195,39,223,5,386,137,75,509,107,571,563,46,194,526,309,21,450,342,288,140,83,24,330,87,70,50,12,9,514,365,246,199,100,65,36,385,55,125,658,244,326,264,696,453 }, +{ 642,898,638,180,901,341,82,197,951,10,762,253,165,15,752,811,479,244,143,365,569,873,242,110,506,351,31,8,116,55,137,115,317,499,141,352,44,338,206,373,386,854,164,22,758,93,37,217,36,1,417,77,629,661,21,717,33,9,564,160,401,456,30,4,203,559,5,509,86,28,326,102,282,268,535,0,13,308,170,269,11,498,391,51,120,2,457,64,704,965,40,32,23,299,421,482,148,615,39,318,533,3,169,961,801,202,260,851,576,117,247,221,349,372,670,968,403,468,291,285,128,824,134,719,237,734,177,342 }, +{ 51,221,23,254,308,438,21,125,166,39,530,656,159,401,49,13,158,115,386,421,688,613,629,228,743,48,518,190,627,453,0,829,333,30,105,912,341,9,670,424,827,165,406,217,404,840,669,391,137,274,197,301,286,272,170,468,457,95,89,22,770,242,33,310,109,569,938,393,960,1,101,36,88,773,538,364,890,131,616,100,28,219,456,344,64,479,679,734,327,846,365,261,55,317,32,5,4,492,180,68,67,482,237,475,793,385,2,367,16,7,347,293,371,224,86,35,719,762,578,332,202,102,442,865,12,949,351,98 }, +{ 1,143,31,197,44,116,242,2,180,125,23,55,206,13,93,498,5,8,115,341,36,752,638,338,141,268,64,901,165,456,0,203,951,546,479,365,873,253,417,4,569,137,15,762,506,237,12,352,965,120,3,299,244,170,164,7,22,282,51,144,6,86,864,21,269,384,558,128,37,9,457,326,844,492,160,961,391,499,202,509,599,870,260,317,685,434,127,217,824,393,373,489,629,77,98,903,308,33,559,332,811,473,301,318,401,102,963,28,386,564,918,367,110,105,911,285,959,523,134,482,740,678,169,617,14,96,758,450 }, +{ 12,23,51,37,13,99,14,38,15,196,328,54,21,134,153,64,453,401,117,128,2,33,139,120,308,5,82,141,361,0,277,29,403,351,719,165,10,115,629,457,46,475,89,450,22,32,110,260,1,31,269,93,9,137,8,386,202,125,86,18,40,111,52,180,17,49,48,4,102,72,477,523,474,326,318,45,898,3,44,402,197,77,730,203,28,24,6,19,421,854,961,352,237,194,114,65,155,373,116,50,36,498,242,236,221,98,55,16,25,7,217,391,569,880,615,41,546,144,509,59,316,79,57,514,494,317,899,488 }, +{ 13,23,15,961,457,115,569,51,165,38,197,2,4,1,474,5,719,99,629,77,0,14,37,401,3,33,7,94,6,128,11,40,10,453,59,64,153,79,509,482,54,17,21,328,111,89,139,277,12,102,498,188,9,152,141,734,76,157,873,386,196,402,32,475,177,117,72,391,36,8,730,758,237,45,304,57,403,134,579,148,202,775,22,523,341,29,431,318,317,308,120,546,62,313,468,437,86,405,217,180,361,50,269,46,342,90,31,143,16,48,44,339,184,19,49,27,18,749,41,450,880,542,205,901,300,216,164,52 }, +{ 15,10,12,0,1,23,961,197,5,82,2,4,3,120,134,457,7,165,260,33,51,9,6,11,351,31,22,21,14,64,8,180,40,450,110,141,386,898,719,523,37,17,153,102,77,117,44,341,217,32,326,93,36,854,269,317,115,18,730,352,49,922,196,365,202,901,28,38,99,59,291,817,125,318,16,569,453,403,55,498,475,111,48,30,94,372,349,342,752,41,391,139,72,474,128,19,899,373,210,203,57,873,661,177,494,98,79,86,143,95,509,27,76,401,89,864,615,304,116,431,144,482,758,137,35,45,29,277 }, +{ 23,51,13,961,719,457,730,401,453,386,758,165,99,899,328,0,10,9,2,1,308,15,134,33,7,6,5,117,54,12,153,11,523,4,391,403,197,629,40,139,14,22,361,28,202,509,128,402,236,231,888,488,369,111,87,57,196,514,79,115,880,317,65,35,817,377,3,291,46,176,77,49,44,38,31,8,450,64,585,477,244,180,141,498,120,507,412,283,102,59,775,475,114,108,62,52,546,237,188,160,157,152,137,94,452,383,260,95,56,440,215,830,774,757,622,569,494,485,472,342,330,300,275,210,203,201,187,144 }, +{ 134,898,82,117,13,23,33,102,153,77,351,341,196,197,110,260,961,180,901,10,120,165,115,111,457,8,450,854,177,40,569,51,94,365,37,11,922,523,277,4,326,291,0,79,55,342,1,9,93,642,12,217,59,352,817,202,5,730,317,21,873,719,141,31,661,811,629,349,16,98,509,373,386,453,372,44,64,18,54,32,3,139,203,116,615,275,22,125,269,210,899,38,318,128,2,144,752,57,479,49,704,17,28,90,30,6,564,638,95,7,403,475,137,304,494,758,474,300,48,36,391,237,498,482,105,499,14,39 }, +{ 60,126,167,191,18,71,35,121,199,16,271,24,212,92,146,150,118,575,107,68,727,566,446,132,21,273,463,108,246,7,279,467,48,653,695,608,517,589,1,101,9,598,5,354,14,649,55,562,345,522,97,98,441,375,140,0,195,22,407,640,149,28,32,223,10,156,49,298,125,13,423,755,508,953,363,221,23,411,314,46,50,781,346,447,154,37,12,818,356,95,647,302,31,230,43,69,674,2,388,20,783,809,232,621,359,189,267,362,124,175,67,293,105,51,109,65,360,493,520,236,163,607,535,75,165,103,39,33 }, +{ 136,62,129,123,128,41,162,789,211,618,249,214,183,678,248,507,38,216,710,119,152,177,269,130,64,17,57,141,245,597,318,205,184,4,179,90,503,77,59,402,305,106,3,313,229,157,266,54,210,139,102,257,275,556,209,45,117,40,13,11,292,99,328,76,457,33,339,23,452,378,217,86,291,80,194,471,369,947,349,383,188,524,497,304,542,381,255,153,519,751,494,240,361,84,803,588,202,32,440,372,115,403,742,164,56,529,12,137,37,790,21,5,504,432,207,196,10,14,285,215,134,342,36,15,587,0,358,165 }, +{ 523,494,475,403,51,899,817,453,202,23,450,730,421,13,805,120,128,757,102,5,4,719,33,21,342,318,958,326,196,153,260,402,7,1,386,12,48,133,65,20,32,457,141,9,8,615,16,114,108,14,10,77,19,0,361,137,17,300,84,82,59,351,123,3,194,576,11,700,221,401,269,95,64,49,177,541,27,40,57,26,115,125,117,22,474,308,99,37,862,701,304,193,56,503,91,697,551,134,105,283,50,129,86,24,249,240,2,942,814,673,671,579,569,519,490,468,437,397,389,365,352,349,306,291,281,247,218,208 }, +{ 59,4,84,62,77,40,3,102,76,119,27,33,56,177,32,12,161,73,21,196,136,123,17,210,11,193,216,80,90,18,64,57,128,51,6,25,13,611,300,63,202,65,48,415,42,605,106,23,152,0,99,275,22,217,803,26,49,205,122,503,291,53,5,2,37,29,349,87,667,453,402,318,209,361,19,690,283,403,718,445,342,129,14,61,194,79,436,245,370,397,297,141,86,862,485,269,240,52,91,85,494,304,510,153,95,573,115,1,45,416,249,184,139,757,475,255,133,81,529,292,41,469,111,680,250,813,113,31 }, +{ 43,414,422,319,20,819,945,7,0,376,61,904,97,470,325,173,707,5,804,693,22,104,14,49,560,426,330,95,595,32,392,17,107,147,207,887,577,246,46,21,324,356,77,74,344,725,58,52,1,198,113,784,347,28,27,18,66,38,76,427,135,105,329,31,858,25,12,158,526,291,263,112,9,523,357,239,232,120,115,362,81,230,359,118,19,10,715,450,119,241,64,44,259,424,315,24,260,140,13,942,292,350,165,102,56,45,37,36,861,125,879,817,719,600,86,33,30,2,814,204,500,476,205,98,26,16,180,179 }, +{ 13,23,51,217,115,77,141,2,33,10,4,0,64,7,180,6,102,40,11,22,317,9,14,269,352,3,1,5,202,197,15,177,341,165,59,237,291,125,365,16,12,128,28,18,247,318,308,31,153,196,349,95,372,117,36,17,24,137,49,498,98,752,32,403,304,134,90,475,342,457,453,210,143,401,93,391,19,86,44,120,30,901,275,386,8,57,105,216,244,164,569,39,65,494,509,20,29,110,25,421,864,961,139,116,873,38,55,27,719,37,35,82,50,221,482,46,260,56,26,546,450,316,468,351,326,45,899,523 }, +{ 28,49,24,105,35,65,159,0,55,109,1,16,239,320,14,7,347,95,424,158,168,170,124,9,2,185,67,68,301,18,169,654,286,149,12,289,175,36,5,127,154,21,10,71,69,114,98,64,740,83,512,39,108,13,272,48,164,327,31,23,623,420,374,395,46,50,713,125,371,518,51,194,338,577,312,6,37,143,916,22,346,344,429,570,32,86,835,103,336,761,101,928,44,638,242,890,737,60,133,202,237,293,115,538,30,762,128,141,137,38,302,398,747,484,479,165,439,189,96,844,610,92,221,299,892,308,54,131 }, +{ 15,10,0,12,1,457,13,165,5,2,82,9,4,117,120,7,961,3,134,6,64,22,197,14,11,260,351,37,33,32,51,21,31,719,77,141,180,38,341,17,8,365,450,317,102,217,153,28,523,269,110,115,44,93,898,372,854,202,386,18,196,40,291,326,99,901,49,730,349,453,352,16,128,125,922,36,237,72,111,318,509,177,76,59,342,403,48,475,817,401,98,391,29,30,89,752,90,474,373,569,27,137,45,704,24,275,19,95,328,55,494,54,498,86,899,143,437,139,210,304,629,758,57,25,402,116,546,873 }, +{ 7,97,71,14,92,150,140,60,149,189,18,16,517,69,336,398,298,154,124,66,302,35,43,68,175,467,273,38,230,118,251,508,314,429,24,2,28,399,172,232,20,0,882,36,279,55,423,173,267,6,46,1,107,126,67,375,83,566,108,22,74,65,547,29,607,104,192,5,426,246,135,847,52,81,531,420,109,596,501,10,809,9,13,952,387,533,505,608,484,169,21,458,330,674,628,49,823,86,25,345,158,51,191,705,48,101,637,12,386,723,439,570,324,653,462,185,168,697,779,535,783,796,346,114,722,861,799,95 }, +{ 102,4,77,59,33,210,11,342,40,32,12,64,128,139,624,5,503,196,202,90,494,21,300,91,51,233,318,403,23,17,291,177,416,269,275,678,141,817,450,13,529,0,25,10,3,485,277,153,151,49,37,42,789,216,496,457,115,57,197,680,214,22,18,788,523,249,205,27,389,194,86,72,453,122,145,480,81,26,961,280,148,138,134,111,104,74,65,6,899,730,204,165,120,89,209,804,475,348,123,95,19,636,396,174,693,651,287,125,869,750,629,306,257,76,48,386,349,221,61,45,803,784,544,402,397,240,58,31 }, +{ 25,42,256,235,19,122,85,436,813,6,297,680,53,26,573,416,233,445,390,63,605,73,250,343,80,27,65,630,389,283,106,255,194,444,45,17,138,544,396,690,525,503,648,269,141,64,119,33,145,123,611,113,128,102,174,112,84,91,29,789,70,624,4,412,77,0,736,483,209,358,205,597,179,56,117,249,243,280,287,108,153,650,196,619,114,32,136,78,678,18,746,348,5,129,2,495,200,57,14,378,76,240,40,511,86,137,161,90,216,10,684,881,23,665,222,22,432,12,289,51,177,1,21,214,370,245,134,11 }, +{ 102,300,402,128,202,403,33,23,342,77,40,99,21,757,84,12,115,196,483,318,19,119,117,210,494,139,90,32,25,4,569,153,469,277,26,475,91,134,63,59,80,523,73,14,275,51,485,453,44,36,31,10,6,48,576,37,49,22,352,193,56,165,18,790,152,415,53,42,880,125,370,291,177,74,13,1,197,180,81,64,62,3,814,611,343,161,136,106,391,386,138,129,66,54,45,38,34,27,17,2,678,11,378,770,697,543,511,504,120,65,82,226,961,899,862,730,718,690,643,317,256,235,89,87,72,61,47,605 }, +{ 158,109,9,39,49,0,35,65,168,68,24,124,159,344,185,154,22,55,333,577,272,175,254,289,810,169,16,327,101,346,95,105,570,149,747,627,286,374,71,310,703,67,439,793,30,28,252,429,131,419,1,455,912,114,610,671,347,18,424,364,7,36,835,100,194,14,21,420,239,336,86,688,10,634,69,108,83,538,706,709,578,484,23,518,2,320,64,51,13,166,48,12,60,118,962,170,128,5,190,133,913,398,50,46,125,677,535,137,127,96,98,345,302,189,826,584,103,251,31,224,638,37,265,92,160,722,219,32 }, +{ 237,498,546,180,115,197,873,287,5,34,923,61,297,509,165,45,400,80,445,2,311,341,3,58,52,468,176,231,186,881,250,64,13,56,151,48,17,217,62,490,14,153,141,23,513,901,51,391,21,667,334,247,352,903,171,84,482,161,549,10,72,401,243,53,738,592,196,46,316,73,719,308,869,666,396,365,645,106,130,585,125,887,635,569,215,86,749,1,207,864,472,177,211,317,22,659,573,37,95,40,227,136,33,372,527,29,507,483,128,651,90,469,229,629,436,833,721,787,758,515,437,54,36,27,117,32,430,304 }, +{ 120,968,704,260,373,450,202,137,661,326,717,269,349,110,217,77,318,473,1,652,95,851,291,93,177,244,33,21,372,141,719,102,351,82,518,156,51,48,848,340,210,928,713,160,922,98,296,128,817,146,86,827,304,264,32,197,225,115,623,317,212,96,23,751,386,854,35,13,7,863,698,180,919,494,385,64,800,253,730,678,523,140,15,70,65,303,453,437,275,247,49,642,341,352,127,37,888,737,5,942,898,167,776,87,658,744,692,615,389,203,101,36,478,196,190,393,59,55,363,132,118,850,163,131,916,836,811,599 }, +{ 44,116,144,268,489,434,367,384,127,918,98,855,940,948,617,206,203,959,93,137,105,31,332,393,492,22,131,9,911,282,276,141,219,202,0,49,284,28,299,120,326,36,39,417,269,1,51,523,30,473,935,899,293,318,352,125,23,395,253,64,13,244,55,160,10,551,128,95,403,373,201,143,450,115,260,8,692,386,21,86,109,110,187,730,165,456,272,737,159,791,261,499,170,7,16,341,728,295,77,951,5,48,96,944,559,242,258,365,371,494,24,254,2,317,421,578,752,18,224,716,164,197,391,285,301,498,924,12 }, +{ 77,13,23,51,102,33,141,64,115,0,177,291,217,202,269,6,2,7,4,318,40,14,180,128,59,1,10,196,117,153,22,11,16,15,352,95,134,9,137,3,165,197,210,18,86,403,17,237,28,49,304,342,125,275,12,349,24,341,247,317,453,5,90,19,475,216,57,25,36,494,65,365,27,20,457,31,120,143,93,498,372,308,401,402,391,29,26,752,260,98,110,719,312,82,961,164,45,56,386,35,44,139,21,99,509,30,730,194,38,244,144,351,8,37,111,203,421,546,50,300,96,288,285,208,46,54,39,901 }, +{ 15,4,11,40,141,165,23,32,202,180,21,197,64,269,125,13,59,386,120,93,51,110,77,5,128,57,318,457,82,3,12,523,260,0,17,49,10,450,730,134,117,31,403,1,719,352,8,102,137,22,115,90,153,899,351,98,44,326,237,2,494,475,139,203,391,144,9,111,961,56,37,402,116,177,210,48,801,453,65,196,6,95,217,341,509,14,421,99,76,291,7,86,216,373,758,401,498,18,28,752,27,36,275,365,300,72,342,615,901,854,898,247,317,45,922,105,127,30,308,152,304,123,817,551,19,791,393,55 }, +{ 176,231,585,62,56,161,412,34,236,527,14,440,202,3,57,17,283,361,84,402,136,23,369,51,128,4,803,13,130,472,383,548,322,507,46,719,457,555,53,73,38,514,503,250,381,119,59,200,162,64,77,452,401,495,153,469,11,229,318,29,205,211,196,497,453,152,2,730,556,240,33,390,27,372,210,184,106,961,215,183,123,65,757,629,99,494,108,129,52,37,18,0,21,40,529,515,475,54,377,828,275,209,197,217,117,1,899,543,300,269,245,862,483,305,45,504,378,342,297,111,102,76,50,24,10,95,12,611 }, +{ 101,9,237,180,0,352,217,230,498,141,100,88,702,115,68,317,83,175,213,39,365,1,252,30,345,752,64,546,482,208,321,864,265,341,490,562,13,372,818,336,952,399,128,69,36,244,197,212,96,410,21,154,689,269,158,28,251,289,166,194,65,382,419,48,443,759,596,901,873,23,468,67,51,16,547,831,797,165,570,55,285,264,930,329,49,756,646,346,124,694,144,391,579,799,854,247,35,190,189,164,699,755,24,364,203,777,323,238,429,569,768,435,37,724,466,201,349,12,509,723,840,50,789,826,160,676,291,137 }, +{ 28,22,49,109,9,39,1,95,0,105,158,168,83,185,159,55,86,30,96,208,141,272,128,12,420,5,505,13,31,327,169,160,131,69,67,137,23,202,21,64,115,251,286,254,51,35,98,127,2,101,244,44,238,194,48,197,50,531,10,7,217,927,180,533,175,352,584,103,285,269,165,37,65,218,100,535,317,323,68,374,946,239,597,24,837,735,289,797,896,418,341,347,333,601,170,116,346,365,237,143,318,345,678,32,344,125,16,164,498,627,253,752,402,577,424,310,4,219,99,403,252,6,312,144,578,166,789,242 }, +{ 72,76,38,89,51,477,4,413,308,179,474,12,152,401,57,148,59,11,205,99,54,421,670,23,245,17,629,431,79,488,405,90,94,184,13,569,21,453,32,229,128,3,358,45,197,14,475,188,77,528,157,464,316,165,257,543,115,41,288,202,457,428,961,216,15,341,403,40,734,266,5,339,749,10,64,33,2,452,328,217,645,460,141,775,111,313,468,1,247,361,102,277,432,743,365,381,587,482,62,509,37,269,0,769,497,471,285,137,402,177,794,449,155,6,318,542,377,86,805,181,129,9,369,7,153,524,349,243 }, +{ 23,13,457,961,15,51,165,629,12,719,11,3,197,2,1,37,0,117,5,569,7,509,33,153,40,196,6,134,328,99,9,542,391,237,17,474,139,111,277,64,453,54,730,546,386,401,57,4,38,14,514,45,775,59,734,361,115,21,102,41,498,477,77,873,431,475,880,369,339,19,587,62,89,377,29,27,523,479,248,128,758,308,482,352,291,275,210,32,8,468,413,164,94,155,659,428,864,752,751,579,494,405,403,397,342,247,240,216,184,180,171,148,122,106,91,84,72,56,52,34,721,188,157,79,654,597,312,95 }, +{ 254,310,100,166,625,9,265,272,30,731,190,39,404,557,688,1,578,131,98,593,101,616,438,264,0,31,829,252,795,716,127,21,382,51,812,49,88,93,613,48,105,827,478,274,18,639,329,36,44,656,23,203,363,68,411,96,682,159,16,116,327,160,242,128,333,13,5,938,910,224,35,447,286,676,144,165,921,530,67,228,137,520,125,219,223,22,724,455,86,33,846,110,284,118,627,905,120,24,77,115,65,83,737,435,32,158,856,664,95,326,924,821,102,518,167,960,202,385,395,931,371,406,10,69,403,55,709,40 }, +{ 51,23,202,99,128,12,15,13,120,403,10,453,153,196,141,64,165,5,318,117,475,82,197,269,401,33,260,2,37,3,1,110,0,14,180,93,386,22,21,450,421,31,402,115,125,137,17,351,134,326,102,494,6,38,308,457,44,36,4,77,11,95,523,28,49,7,9,8,719,237,352,730,32,203,116,391,373,57,18,98,59,361,86,54,19,89,285,775,328,277,757,498,341,854,111,16,27,139,72,45,48,41,143,79,509,961,898,758,342,291,65,474,488,30,569,217,144,56,901,242,899,670,34,24,678,817,922,194 }, +{ 38,99,542,453,139,196,117,457,23,328,13,111,37,134,961,11,12,775,401,51,587,40,474,153,277,477,54,629,475,33,719,79,165,41,14,569,157,4,413,188,89,115,64,339,734,308,15,361,369,197,313,102,94,247,155,45,248,236,57,514,72,758,402,128,181,485,377,509,86,237,488,0,403,266,141,46,10,164,211,471,184,588,65,180,29,2,875,528,490,95,76,22,5,50,17,7,721,670,391,49,18,503,817,789,431,21,257,216,32,208,305,177,136,1,880,803,352,140,342,260,229,202,67,867,597,59,757,899 }, +{ 85,6,233,343,91,25,138,26,63,174,256,235,122,29,19,81,283,65,605,648,42,718,445,370,4,74,59,573,389,416,194,396,45,177,544,297,355,64,141,348,145,77,114,216,123,280,436,881,133,680,432,87,70,511,137,108,27,128,57,630,0,11,415,18,90,503,619,86,412,40,287,17,397,789,222,14,73,269,32,66,52,291,204,243,56,22,209,275,597,104,76,72,529,148,678,84,80,119,53,643,304,210,208,285,33,46,2,523,5,311,813,636,510,381,255,10,524,112,102,403,289,214,152,483,62,736,525,349 }, +{ 49,5,197,408,193,18,20,61,713,350,173,21,64,97,509,95,729,841,263,0,241,32,165,104,43,422,65,125,35,510,60,474,7,120,102,523,397,14,48,232,260,128,347,68,546,387,38,58,693,403,16,105,81,71,115,784,10,560,91,72,6,862,27,92,51,707,344,22,9,237,319,29,414,107,55,66,643,147,663,450,391,498,240,357,70,468,90,2,751,370,98,701,84,819,903,330,207,623,141,63,52,718,569,308,864,475,25,159,74,283,356,326,158,86,12,900,494,137,171,42,221,202,85,734,386,847,873,650 }, +{ 434,144,268,489,116,384,367,855,940,206,617,332,959,911,127,93,44,98,203,137,276,282,219,202,131,22,141,31,417,284,918,9,36,105,28,201,269,258,244,120,473,23,318,523,51,326,253,1,450,39,0,30,352,160,948,143,10,13,8,110,49,64,125,55,551,935,951,692,295,187,128,21,86,261,567,260,499,559,373,96,224,170,899,386,242,115,299,817,95,662,728,293,165,197,811,395,159,48,492,365,109,317,498,403,37,752,341,393,836,638,717,5,180,254,77,15,272,851,164,24,944,494,237,385,16,7,301,391 }, +{ 106,17,255,32,64,62,136,292,162,129,476,352,237,214,123,519,241,720,141,752,498,247,211,41,119,209,864,507,249,180,179,290,710,954,425,148,317,618,197,90,186,21,5,115,248,77,76,269,217,183,45,48,14,227,130,3,263,12,304,38,165,128,243,546,23,2,358,207,117,164,57,579,51,245,205,54,454,313,152,350,177,13,155,873,339,270,50,4,428,378,157,655,449,496,305,400,365,171,86,1,437,742,556,372,369,408,471,328,266,216,33,80,153,139,460,181,597,542,316,40,102,29,751,59,361,257,84,341 }, +{ 2,29,52,58,5,50,27,6,53,26,171,45,366,151,14,287,61,19,357,74,17,73,764,1,113,34,297,309,218,262,181,38,133,48,583,70,487,600,739,334,25,396,250,760,281,96,186,445,516,75,502,635,315,114,308,108,3,223,80,549,42,545,103,331,324,106,200,163,112,21,785,67,436,145,603,238,401,243,51,37,651,433,155,834,491,675,236,99,745,209,161,13,845,32,227,115,409,23,894,388,765,12,56,36,311,255,220,920,766,138,323,453,573,24,594,62,59,65,832,754,160,86,54,307,283,83,285,629 }, +{ 130,711,412,381,59,196,210,730,236,77,402,401,803,202,453,361,99,108,283,291,153,57,183,128,503,421,4,51,14,719,432,11,318,275,117,23,529,757,3,475,13,339,457,133,488,123,670,899,603,369,134,33,65,46,266,386,2,47,754,12,32,389,114,38,838,56,300,40,102,177,739,111,961,523,629,328,758,64,54,90,216,514,269,21,594,0,743,37,775,657,576,894,403,152,650,194,471,34,200,5,832,162,50,494,805,472,18,165,22,6,29,181,95,285,10,129,188,480,15,1,322,24,229,342,139,714,79,86 }, +{ 13,23,51,217,115,77,141,33,4,64,10,202,102,180,0,40,317,352,177,11,291,269,197,165,9,32,22,59,7,2,128,318,341,18,15,21,237,137,117,6,16,5,95,3,153,403,125,349,365,14,247,752,49,12,196,28,31,453,90,1,24,98,304,17,143,494,372,308,244,475,498,134,210,221,120,901,401,36,342,421,86,93,44,30,216,260,35,164,457,19,110,275,569,391,57,386,203,105,326,55,864,82,402,523,241,139,8,450,39,873,730,719,144,65,20,253,160,27,351,116,72,29,437,509,37,629,285,546 }, +{ 23,13,202,51,21,269,141,120,5,48,128,137,318,523,15,102,125,98,1,450,93,403,165,260,64,197,77,33,180,12,457,32,221,82,817,115,44,110,31,7,352,494,2,386,475,308,117,4,153,326,14,401,719,116,196,421,37,237,16,453,9,144,730,351,17,99,6,30,291,402,203,18,11,899,19,24,217,96,498,40,391,20,10,39,95,854,3,341,50,134,349,293,373,177,615,35,365,65,139,0,45,285,372,61,8,317,90,922,805,901,342,59,43,898,76,29,509,393,127,86,752,170,208,105,89,72,67,38 }, +{ 2,1,46,14,52,29,171,67,38,103,58,75,70,69,163,24,83,223,5,96,309,181,54,186,112,334,108,45,394,675,155,366,220,35,238,151,16,236,409,18,281,760,307,133,537,400,36,71,189,328,227,433,28,218,766,61,545,65,17,487,149,7,48,552,21,502,0,12,262,19,87,6,765,114,25,388,267,150,113,820,418,50,549,27,26,37,251,22,99,335,430,486,32,60,536,287,68,587,9,828,55,64,74,243,10,516,137,160,86,66,128,311,600,696,739,852,583,451,336,315,514,72,135,80,175,145,89,294 }, +{ 15,2,1,13,0,10,153,5,9,7,12,23,3,14,117,6,165,134,4,719,457,196,82,341,180,197,33,22,120,115,77,8,961,21,64,217,102,317,569,365,351,141,260,32,31,18,28,291,11,16,110,111,37,125,93,386,17,202,40,450,269,372,730,901,99,898,44,36,177,854,38,342,349,352,48,523,758,318,453,57,128,401,391,304,72,76,403,19,922,326,752,29,629,49,509,475,237,137,116,98,89,55,373,95,24,27,26,30,494,247,59,482,86,25,210,96,54,46,437,899,139,65,253,817,661,498,308,873 }, +{ 509,197,546,391,165,180,308,13,23,498,569,29,873,115,901,5,401,61,457,903,80,734,629,468,3,287,34,2,45,758,311,445,250,579,297,341,961,237,17,186,171,749,386,51,14,38,217,231,48,58,62,400,46,752,247,64,211,176,490,106,719,56,764,738,667,227,215,349,151,11,558,316,502,513,833,32,161,243,54,317,153,196,50,21,53,40,365,583,155,923,10,6,587,334,305,483,33,4,369,372,181,136,1,125,119,12,482,130,552,263,57,52,95,123,814,266,84,832,495,645,739,864,96,15,787,472,129,36 }, +{ 73,19,250,27,714,200,444,472,26,34,813,53,390,322,17,283,78,297,106,630,436,113,122,690,123,432,136,128,176,862,161,389,503,684,611,495,80,378,62,42,108,529,14,56,469,57,25,412,236,177,59,47,65,102,6,216,275,3,381,680,235,179,527,605,84,129,383,130,619,183,368,46,255,814,33,23,318,2,789,77,141,13,453,194,245,51,11,445,45,4,152,63,650,269,22,90,64,0,133,440,29,349,40,585,86,678,736,291,95,196,153,184,361,18,210,38,58,803,401,5,117,10,114,358,61,730,249,119 }, +{ 17,21,3,45,62,32,155,119,106,38,5,328,14,828,227,76,2,401,209,123,129,243,270,115,540,29,56,536,46,12,221,488,99,13,629,57,72,719,181,51,552,290,361,136,41,37,514,165,543,152,421,408,84,48,77,308,207,255,90,306,23,263,569,11,454,52,587,162,758,378,4,89,80,754,130,402,236,197,186,440,457,27,383,157,50,1,183,311,832,128,10,794,425,59,576,961,211,741,453,805,790,86,742,496,386,34,313,54,700,600,266,148,659,202,19,94,341,22,880,509,229,240,697,770,670,548,73,177 }, +{ 115,341,873,365,197,13,901,180,569,752,317,1,509,498,10,143,15,4,634,165,268,943,261,5,258,23,237,116,141,0,127,601,244,843,282,253,164,44,12,33,64,860,638,533,219,77,144,9,567,31,134,7,2,352,864,276,160,535,51,206,217,824,125,391,8,28,951,295,11,93,332,201,40,964,844,308,137,6,3,120,531,260,22,36,482,98,177,111,318,474,417,37,170,82,961,291,196,102,169,924,434,203,153,94,954,395,734,898,384,269,242,479,965,735,405,326,30,728,662,617,386,159,24,18,944,401,301,367 }, +{ 453,51,23,33,403,475,421,196,153,102,670,13,117,753,134,40,494,523,4,730,342,0,21,65,59,22,12,880,77,11,86,801,401,719,202,48,28,899,18,805,757,64,14,10,24,128,1,629,457,99,108,57,308,5,2,217,317,139,16,485,569,165,141,36,361,341,365,177,143,125,283,386,120,82,44,9,115,221,7,817,576,300,152,137,124,123,98,61,50,46,32,6,450,111,775,543,488,417,269,197,114,110,95,93,54,35,31,402,34,17,3,911,642,564,561,554,511,431,332,277,261,253,242,239,194,170,116,96 }, +{ 23,13,51,33,12,453,196,117,99,153,134,40,102,403,14,128,202,77,15,401,475,37,139,38,54,11,64,165,421,21,2,0,141,115,308,318,719,328,65,108,670,197,10,46,457,361,269,730,79,1,22,402,28,111,120,95,82,18,3,386,9,389,6,17,283,50,36,24,86,236,114,125,775,5,494,180,4,260,31,94,41,32,899,523,19,210,110,137,59,391,7,326,488,450,351,291,217,177,412,341,93,300,485,477,509,880,377,44,39,30,277,52,221,16,8,961,629,365,203,29,758,133,26,817,350,342,237,194 }, +{ 2,7,14,5,52,29,70,61,112,46,87,592,135,1,66,45,38,16,32,6,186,21,315,290,48,227,24,72,307,113,75,172,425,20,243,58,74,97,666,128,845,631,76,226,155,294,18,738,23,270,25,35,64,141,334,115,0,89,22,50,43,163,151,37,51,171,77,207,69,17,220,36,177,202,19,262,10,13,137,9,90,232,4,54,86,92,263,269,255,278,59,496,655,60,28,95,140,26,106,27,180,12,454,192,148,67,40,209,102,318,33,292,400,849,49,65,324,331,409,405,217,491,311,461,549,476,431,210 }, +{ 33,102,128,202,51,23,141,196,64,153,13,269,318,403,77,117,10,137,134,40,1,5,453,2,6,165,3,0,14,475,86,99,139,15,180,120,12,22,494,402,17,197,352,260,19,7,95,342,21,110,237,65,82,115,93,125,31,36,37,277,18,217,210,9,49,44,25,177,291,326,523,38,450,401,194,32,457,421,285,386,26,28,351,16,111,719,27,4,11,114,48,341,203,300,8,752,98,373,143,730,361,498,57,54,365,317,24,164,961,116,569,108,275,899,50,34,391,45,41,901,96,244,29,629,133,72,509,144 }, +{ 92,16,43,20,126,107,7,35,71,97,14,60,140,246,279,68,18,149,230,121,575,356,375,191,362,24,783,150,118,55,695,566,345,48,0,359,175,173,319,598,298,232,939,22,653,441,167,463,21,414,125,819,467,407,33,69,346,38,422,146,77,446,154,608,28,649,336,172,10,199,102,101,104,168,189,637,251,212,49,314,376,2,124,51,9,595,399,221,23,66,5,392,541,755,196,470,386,13,6,158,117,768,393,147,628,551,493,683,165,67,46,36,259,108,81,1,727,535,364,342,325,672,267,281,674,120,203,153 }, +{ 13,23,77,115,51,33,59,4,177,40,90,64,102,32,141,202,10,128,22,210,291,0,11,21,117,95,137,217,16,318,196,9,180,153,7,197,165,49,28,269,6,342,14,18,2,216,1,237,134,86,98,352,349,203,326,24,403,36,494,31,125,450,120,475,275,15,93,351,341,260,194,386,105,3,247,48,57,365,373,20,372,402,317,457,44,5,139,110,17,35,30,25,12,498,76,144,116,72,39,453,304,19,300,244,208,82,60,401,143,285,65,214,55,38,26,817,569,50,678,551,391,308,123,111,46,854,509,523 }, +{ 507,162,129,211,41,183,38,123,4,248,59,62,57,556,130,99,361,402,305,128,202,11,266,136,17,3,77,803,152,497,313,452,184,210,369,33,40,471,339,56,117,757,157,54,328,229,205,494,381,23,139,457,153,12,403,84,503,257,19,34,65,13,76,514,119,275,37,188,542,14,80,106,440,475,775,318,618,102,21,134,453,300,27,432,64,880,95,89,49,141,86,196,32,529,377,249,73,79,460,177,50,5,10,90,53,619,548,342,94,401,269,179,288,587,961,383,148,114,579,115,477,216,72,45,805,285,291,217 }, +{ 643,123,193,718,650,18,802,65,389,84,415,91,177,25,90,619,256,216,87,289,511,183,736,370,217,324,380,235,108,63,808,500,432,102,249,57,275,194,152,6,673,510,304,862,240,524,355,588,561,397,233,372,13,523,130,81,96,4,0,343,605,141,1,22,103,77,59,283,214,64,789,437,306,630,23,85,153,80,33,475,597,529,210,544,349,196,129,56,291,32,27,847,300,114,889,550,138,86,886,444,269,124,28,26,416,421,280,208,174,70,10,2,814,401,115,19,851,117,42,730,469,352,204,17,202,150,73,51 }, +{ 23,13,51,15,5,1,2,7,115,0,12,6,165,141,21,3,4,457,9,197,180,32,77,14,37,202,217,10,33,120,341,352,11,269,22,8,16,93,569,961,31,401,318,102,247,719,453,308,629,17,365,18,260,403,177,317,901,386,40,48,509,137,752,44,730,28,391,498,72,117,523,128,110,153,59,450,98,494,99,196,36,372,237,30,90,49,475,134,291,326,125,24,210,95,349,89,899,482,38,116,421,468,351,304,19,76,873,82,20,139,57,734,203,86,342,373,546,54,39,437,45,65,817,854,758,922,144,96 }, +{ 51,23,13,453,21,403,12,475,202,165,141,64,128,115,32,5,95,523,125,730,494,180,197,401,899,308,48,120,37,719,386,269,318,817,457,4,99,421,3,326,1,6,237,17,450,2,260,10,19,22,14,0,217,194,18,402,77,221,90,247,15,177,27,25,26,961,59,9,7,758,137,16,670,391,352,117,11,801,312,33,110,82,50,28,241,57,40,351,123,153,49,615,317,291,72,341,210,24,86,54,61,29,196,102,216,93,56,569,31,304,170,98,275,289,46,42,20,629,361,108,58,89,389,283,144,45,84,203 }, +{ 204,74,135,174,192,66,138,85,6,280,353,348,580,172,7,673,97,889,426,500,792,568,95,87,355,32,620,64,712,29,65,91,233,81,232,222,52,448,43,872,25,825,42,104,173,888,22,289,226,63,526,86,387,5,312,636,861,49,90,237,278,2,343,415,194,560,21,0,715,92,38,214,145,140,20,256,835,324,235,330,18,10,750,61,28,75,808,1,4,26,681,288,77,36,24,70,69,58,107,45,874,347,125,14,112,314,33,72,337,102,298,241,294,707,59,729,108,180,37,115,370,40,287,76,48,356,164,117 }, +{ 35,68,149,24,69,0,65,133,103,16,189,9,28,18,50,114,175,336,124,101,1,71,154,46,83,251,14,7,2,55,283,118,230,108,218,238,38,398,345,140,705,75,54,302,443,39,596,22,532,309,335,181,96,361,418,21,399,12,52,394,565,433,412,267,346,109,60,5,29,36,486,236,48,160,375,732,831,545,23,480,487,516,51,220,92,366,86,429,10,13,64,323,328,67,49,252,30,37,330,514,364,212,100,439,547,158,389,95,87,458,194,115,32,646,891,930,246,208,70,641,171,163,6,191,223,722,167,99 }, +{ 101,118,16,71,24,9,18,154,191,212,0,35,60,520,124,68,694,806,167,364,375,252,604,346,55,302,562,628,28,39,149,917,439,126,419,92,65,826,329,100,706,589,166,108,175,265,264,21,1,273,199,109,22,701,48,925,676,49,158,7,150,363,404,10,146,942,5,88,121,815,105,411,14,967,682,67,23,535,69,13,818,566,882,50,777,907,647,429,95,159,51,96,30,12,333,621,31,2,857,848,32,107,140,722,98,575,821,246,37,847,46,382,905,221,677,169,36,336,125,553,695,83,467,593,185,759,114,286 }, +{ 35,16,9,0,68,24,65,1,18,71,149,103,67,189,101,28,21,48,212,114,23,335,486,124,13,51,39,118,167,191,55,154,7,133,238,433,50,30,443,141,96,230,22,100,140,218,60,5,137,64,537,352,115,12,88,330,217,223,75,10,128,399,37,269,49,83,411,202,32,283,175,36,31,221,246,264,98,163,398,2,281,723,375,302,69,166,478,447,125,108,487,180,54,372,317,589,44,99,520,336,181,361,345,418,318,363,177,86,695,341,664,565,77,160,95,545,562,252,70,237,116,752,532,190,72,105,92,14 }, +{ 13,23,202,77,51,128,33,32,115,102,21,269,177,141,318,64,5,217,137,494,291,11,4,10,0,48,22,153,95,210,117,86,90,196,3,15,6,317,2,49,125,349,40,18,12,342,352,9,31,17,247,304,403,194,134,36,16,7,14,450,19,275,98,180,120,1,61,44,372,216,28,341,65,165,221,326,116,76,20,260,59,523,237,365,285,203,24,35,105,453,288,143,386,678,30,351,101,68,60,55,25,817,421,475,96,901,401,367,308,293,99,93,82,39,37,27,752,312,144,615,323,730,58,72,437,859,899,316 }, +{ 16,92,60,35,7,18,68,150,24,126,149,97,191,246,375,118,14,230,107,140,71,273,575,175,517,653,154,566,55,43,298,345,167,399,0,121,28,467,20,108,279,212,101,441,346,302,336,66,423,232,508,48,22,674,124,38,356,608,314,9,21,2,189,199,359,446,49,10,695,46,5,69,1,463,65,815,589,12,172,439,607,755,251,146,37,362,67,392,23,173,6,109,13,722,125,809,702,796,628,36,32,51,95,637,74,135,354,398,52,132,83,29,429,221,407,768,882,105,672,50,64,104,86,271,501,364,535,96 }, +{ 76,90,464,21,179,316,205,288,460,245,32,257,184,148,45,358,152,57,270,48,241,38,449,89,14,2,1,350,769,645,229,155,5,497,221,54,17,524,37,243,12,59,227,405,266,428,4,452,413,308,749,432,50,619,99,720,51,29,292,476,217,11,290,46,157,115,79,188,339,129,313,216,401,381,431,471,171,341,247,77,10,94,106,513,400,3,23,64,317,141,52,7,0,365,773,474,477,72,430,528,36,659,162,369,453,181,62,214,128,33,237,741,186,9,6,281,328,13,202,102,502,349,536,498,177,291,180,372 }, +{ 19,297,26,27,813,80,436,495,17,73,53,250,122,106,200,113,378,390,690,42,4,45,469,361,59,25,742,128,136,161,77,255,202,736,445,605,402,123,862,630,40,63,318,78,383,803,56,11,235,102,283,444,119,527,6,62,210,503,680,14,57,33,148,472,209,129,684,256,84,412,667,611,18,573,85,365,483,0,177,12,3,300,814,153,504,65,22,76,139,61,34,205,133,341,117,64,51,23,13,58,269,152,90,257,10,236,2,358,497,38,432,342,95,47,368,243,183,494,757,453,619,196,184,548,171,714,475,322 }, +{ 6,26,235,138,145,112,19,42,624,122,70,648,27,331,262,85,78,47,25,500,52,632,630,226,113,444,307,644,409,453,29,222,665,163,75,50,849,324,2,65,108,390,294,64,675,53,416,711,130,220,22,34,196,17,73,813,87,32,95,680,14,280,223,852,412,133,730,151,74,237,315,33,86,45,115,66,197,287,241,58,236,153,583,51,256,28,5,114,1,297,283,792,180,4,165,38,90,63,135,102,337,690,250,21,3,171,820,396,546,36,853,889,357,10,183,137,46,200,445,56,475,125,99,461,603,309,117,0 }, +{ 16,7,14,92,24,35,60,18,46,140,126,71,87,50,171,54,9,2,38,5,0,220,108,69,23,230,21,13,246,128,163,70,12,107,394,58,20,68,64,121,155,150,330,309,28,48,22,67,32,10,51,141,267,137,55,314,236,202,146,1,37,281,115,181,502,461,269,189,307,852,232,774,132,97,571,328,75,43,223,65,278,199,52,217,86,820,77,366,335,172,72,112,186,6,409,622,318,99,61,95,103,149,262,49,780,552,177,133,36,227,29,294,462,430,17,334,135,352,572,45,675,563,270,66,33,451,39,180 }, +{ 257,90,205,184,460,769,245,32,179,524,5,45,229,497,288,152,619,21,57,432,452,2,17,292,59,381,266,14,476,1,4,48,106,241,316,358,216,255,29,77,38,217,11,54,155,58,313,64,263,157,425,76,471,12,129,6,227,72,243,148,186,141,3,79,247,37,128,7,339,464,214,188,290,209,207,99,720,519,369,317,177,40,50,115,365,33,710,51,89,270,269,102,221,588,86,46,162,61,202,529,237,291,111,454,94,10,62,372,408,875,23,277,350,618,210,645,349,123,308,19,52,25,180,36,449,352,119,181 }, +{ 15,4,5,11,13,23,141,21,32,40,202,180,1,0,33,269,3,128,17,318,352,134,51,120,2,64,153,117,450,31,326,12,137,196,22,197,102,82,260,44,6,752,165,59,10,49,7,48,177,217,494,57,341,115,403,523,95,901,237,386,457,37,210,14,317,98,351,365,90,56,9,961,203,817,498,719,125,93,28,730,18,110,19,372,116,291,244,678,86,475,615,8,194,76,342,27,45,453,349,36,61,139,96,873,864,216,402,391,111,72,373,16,275,143,285,30,25,247,39,144,300,253,24,922,214,38,70,164 }, +{ 130,711,47,412,108,453,730,196,283,390,78,381,27,183,236,503,472,51,14,200,803,128,153,719,59,250,133,432,34,318,57,73,123,4,114,117,65,17,53,177,33,576,714,216,657,56,378,322,475,46,23,3,529,754,134,369,275,339,19,11,269,12,389,899,291,813,402,650,368,40,603,401,21,444,361,436,266,210,99,457,297,202,480,77,102,54,666,592,513,349,403,129,194,106,894,38,471,0,523,32,739,137,18,22,2,690,136,386,594,152,161,684,678,181,162,548,26,670,141,37,10,184,814,743,421,789,45,629 }, +{ 12,277,51,474,153,111,99,23,37,13,629,961,431,94,569,38,453,542,139,115,79,134,117,401,196,33,40,188,157,89,11,775,457,72,485,15,405,54,4,339,413,300,377,197,477,148,313,328,165,59,734,308,102,719,76,14,45,210,179,21,128,730,152,3,64,17,509,498,386,10,266,880,2,77,32,391,57,482,41,475,0,471,403,304,141,177,898,523,82,18,365,490,205,758,275,5,217,29,402,546,372,488,260,351,341,120,24,202,452,358,237,22,352,873,468,361,6,269,317,216,155,437,162,48,1,229,31,811 }, +{ 495,469,17,378,106,26,27,483,161,862,383,742,19,73,80,527,53,436,814,136,297,84,62,119,611,56,250,25,440,209,113,42,6,122,34,4,255,128,240,14,3,684,200,59,585,129,65,45,29,123,504,57,573,2,5,11,445,361,141,630,61,64,205,179,86,833,76,32,667,690,33,235,202,245,162,618,40,283,680,548,318,472,402,269,503,77,555,102,194,184,108,183,207,95,151,21,152,112,177,514,46,153,887,491,311,803,54,58,257,12,38,10,507,90,648,196,51,176,22,614,306,48,0,229,130,145,1,114 }, +{ 681,636,81,750,91,370,138,718,0,104,348,784,174,397,673,693,18,29,355,192,173,280,74,426,306,65,207,6,701,715,86,319,64,124,671,85,52,32,289,95,510,22,90,511,887,204,343,712,560,879,526,422,240,233,214,7,49,312,40,120,72,77,519,43,24,707,484,4,577,353,42,523,84,63,11,177,27,145,835,730,57,474,643,151,448,756,28,25,10,61,19,21,20,260,415,102,194,888,580,87,861,610,825,33,288,232,193,115,66,59,817,76,45,172,847,17,2,719,457,273,114,26,23,450,168,13,414,302 }, +{ 15,33,77,102,115,117,4,134,153,40,196,11,13,217,1,5,23,317,21,3,365,2,291,341,32,12,59,51,177,139,342,6,0,349,210,7,372,37,180,48,351,111,17,90,569,141,275,36,57,197,457,202,137,72,89,304,14,308,9,120,854,901,898,300,277,10,55,96,64,148,247,961,160,27,165,450,128,437,269,719,352,260,752,38,401,318,523,216,730,734,93,629,76,221,94,482,56,579,18,16,143,253,873,922,453,61,638,179,49,19,22,31,244,498,125,8,697,45,24,817,44,245,403,485,79,326,84,899 }, +{ 9,0,28,68,35,65,101,39,67,175,1,124,154,69,24,22,55,18,16,238,103,30,336,114,12,49,251,109,149,96,50,13,21,51,7,23,252,5,345,100,596,218,158,141,831,418,189,480,532,37,10,166,115,346,2,433,64,323,217,128,137,48,443,88,86,160,36,439,75,237,547,269,202,71,289,866,646,429,118,335,389,364,285,230,317,930,197,352,180,159,143,181,486,503,402,283,99,133,281,95,223,194,498,208,165,168,318,32,60,14,83,4,657,105,212,131,265,694,745,956,54,185,163,484,6,403,77,722 }, +{ 23,33,102,40,51,77,13,403,117,196,153,64,11,128,453,134,137,300,59,291,141,4,10,202,22,475,180,95,57,719,139,99,0,342,86,65,494,90,12,28,9,49,485,421,21,115,114,1,194,901,752,210,177,165,120,108,15,37,18,349,328,2,323,269,899,450,391,351,260,197,125,36,961,523,457,361,48,14,341,958,954,947,922,854,817,758,730,546,509,498,412,365,289,275,236,160,93,54,32,16,7,847,805,576,401,217,962,570,467,285,218,124,105,874,835,775,701,670,629,577,569,541,386,373,372,347,277,247 }, +{ 214,90,289,874,6,65,256,235,64,751,25,42,312,249,194,746,288,875,525,519,247,835,174,85,233,348,544,217,437,32,524,352,343,95,123,138,26,63,355,954,86,122,91,280,630,416,0,177,605,22,204,317,752,648,222,736,141,77,45,579,864,180,74,216,415,304,389,59,597,57,66,306,81,237,287,56,881,588,243,115,84,208,19,445,370,108,498,114,183,87,29,671,4,316,49,283,76,145,275,18,444,205,573,396,240,52,28,112,337,104,10,636,202,624,70,368,11,511,353,33,21,135,102,650,680,137,291,619 }, +{ 1,22,2,28,67,36,50,5,12,49,0,95,14,168,55,7,83,46,103,105,114,69,194,9,38,21,159,584,35,65,239,133,169,109,24,218,23,16,158,208,10,323,48,170,54,51,96,37,160,597,164,13,29,6,18,320,244,708,32,115,892,202,128,503,505,424,185,874,776,253,312,64,347,143,68,75,238,654,137,141,756,31,124,480,301,531,39,181,269,289,824,3,251,835,646,389,361,318,789,52,98,189,797,532,30,638,285,180,165,308,420,99,286,751,127,237,17,283,45,4,175,44,577,125,533,154,72,197 }, +{ 269,318,352,202,141,13,77,51,180,102,349,23,372,291,137,403,33,128,125,197,21,44,31,494,342,317,217,32,402,117,1,365,16,901,475,177,49,22,11,5,116,98,93,386,165,160,153,115,59,40,18,752,253,244,15,14,12,10,7,2,0,4,947,704,692,678,661,615,597,576,473,373,326,208,203,48,39,30,6,90,28,24,498,453,421,127,96,17,961,899,817,474,408,401,350,308,304,285,263,241,196,194,185,169,168,148,134,113,111,101,95,94,89,86,71,64,55,36,35,769,638,569,275,257,152,9,8,968 }, +{ 435,144,274,88,203,418,410,778,190,30,530,960,521,795,686,466,1,321,326,96,367,100,166,822,382,264,692,855,539,131,438,576,335,213,9,265,201,98,613,921,93,615,859,44,31,840,765,403,800,83,772,662,225,744,523,494,284,473,116,957,924,0,909,36,478,217,817,141,28,966,69,934,831,551,137,21,120,731,269,829,219,128,948,669,406,51,639,625,23,5,230,206,39,13,268,77,944,372,110,617,238,260,404,125,716,557,67,856,691,224,32,7,442,349,332,16,805,492,254,242,228,10,704,291,223,35,842,64 }, +{ 76,72,90,179,205,21,245,316,428,288,32,464,184,37,38,257,460,413,148,54,645,405,57,749,45,308,12,358,152,221,431,14,449,4,155,350,229,48,401,50,270,2,59,1,99,5,497,769,17,11,528,524,452,474,266,241,115,51,243,247,10,128,773,477,79,741,432,281,430,217,129,188,77,181,619,292,64,659,157,339,46,313,227,106,23,453,476,720,36,202,468,7,328,743,381,488,216,94,341,141,3,0,498,794,471,197,137,421,180,536,317,403,237,629,490,29,62,482,86,13,6,165,540,369,33,402,352,162 }, +{ 77,102,33,141,64,115,23,40,177,13,51,291,2,269,202,217,6,128,10,210,180,318,1,196,7,14,352,137,197,0,165,3,22,4,153,237,95,117,19,17,59,5,275,15,134,139,86,11,342,125,9,16,317,18,25,349,341,12,36,26,494,216,49,403,31,498,752,27,93,143,111,65,300,24,110,312,28,120,457,391,372,386,44,98,21,247,20,901,164,304,29,82,203,453,402,401,365,99,569,57,719,56,509,475,961,194,260,34,730,678,546,45,116,32,50,48,244,35,38,864,37,308,629,46,450,326,8,54 }, +{ 397,32,788,81,65,693,249,681,90,422,4,804,194,123,21,59,56,348,11,64,23,61,95,91,690,3,725,718,84,319,0,280,152,237,138,43,355,817,174,6,510,57,494,17,523,899,141,114,49,214,52,165,414,370,784,77,526,60,86,51,386,217,74,40,12,904,85,365,102,450,326,208,18,120,945,750,636,343,242,233,115,650,615,5,20,511,342,29,33,887,457,888,524,880,350,263,180,137,133,108,36,24,15,13,222,204,183,177,160,582,532,391,278,135,730,719,289,715,671,597,10,566,421,403,221,218,207,202 }, +{ 126,60,16,314,150,121,199,146,107,92,246,7,132,35,71,359,232,18,24,14,298,392,649,672,423,517,140,441,97,279,508,388,95,653,32,462,877,195,108,21,230,598,607,407,167,46,172,451,5,118,68,22,387,357,38,267,640,55,522,191,28,156,467,360,271,380,594,0,10,953,733,49,48,2,356,64,66,955,633,809,603,273,278,75,149,65,50,1,52,660,783,135,43,12,362,354,674,236,711,212,702,86,309,13,9,115,493,37,330,20,125,36,220,312,241,54,501,324,101,87,29,345,23,90,550,928,446,77 }, +{ 217,15,141,115,317,365,341,13,23,260,51,137,120,202,77,372,180,33,349,4,269,352,64,102,40,1,110,373,450,291,197,21,32,36,160,117,244,752,318,128,5,854,351,0,11,922,59,247,134,661,9,304,10,901,165,12,153,22,457,177,143,96,253,28,86,82,48,523,2,196,730,494,57,90,811,49,403,7,326,237,16,83,3,8,18,437,125,39,817,30,72,111,139,55,37,17,961,898,498,899,308,719,24,475,386,76,164,35,717,288,873,704,6,89,221,569,14,642,391,241,203,67,56,509,210,285,95,342 }, +{ 111,134,117,474,23,13,961,569,12,431,51,37,115,99,277,15,405,38,197,89,72,629,148,457,76,54,4,45,165,11,139,94,719,498,106,59,873,14,79,128,40,10,196,33,482,3,17,179,188,2,64,413,730,775,245,177,153,205,471,57,509,477,401,358,453,0,403,6,485,22,5,152,304,1,180,402,275,475,300,341,901,36,880,339,237,210,82,141,386,32,110,7,758,120,452,77,365,317,260,129,391,31,95,202,308,46,864,542,523,8,757,86,18,44,752,217,162,93,65,734,437,497,421,678,269,229,546,285 }, +{ 23,13,51,5,202,21,32,1,12,2,141,15,165,115,269,7,4,453,197,403,318,6,48,120,9,0,3,386,128,64,180,14,125,37,475,308,401,137,457,16,10,99,18,450,260,196,523,33,352,237,494,730,719,17,77,11,421,341,326,31,22,40,221,102,8,391,90,24,82,217,30,93,59,117,177,153,351,365,28,19,44,110,899,98,317,569,39,72,61,629,961,817,49,134,509,57,498,247,76,20,29,402,45,27,203,38,36,96,752,89,241,50,56,373,144,25,304,758,86,291,46,54,65,35,349,116,901,576 }, +{ 13,141,23,5,115,202,4,51,269,352,217,11,2,318,180,137,317,21,7,341,77,3,6,32,197,752,33,1,291,12,64,372,10,365,128,9,102,22,0,165,403,40,15,349,494,498,482,48,308,177,901,31,28,16,37,873,44,237,14,49,125,153,247,18,244,98,30,59,453,120,143,55,401,475,304,36,39,457,342,93,17,72,95,221,24,196,35,117,134,253,86,90,509,579,76,116,203,864,326,961,576,569,210,717,523,859,144,421,678,160,8,719,110,127,216,89,391,260,468,730,65,386,629,450,437,805,164,105 }, +{ 165,125,391,197,21,13,558,380,23,48,386,237,900,221,66,509,180,14,326,120,457,33,32,357,115,961,97,426,401,298,135,278,308,719,51,98,758,841,172,75,38,56,12,6,571,204,770,74,19,72,554,867,629,408,393,77,670,64,63,25,10,1,42,498,462,134,40,7,563,247,102,715,773,90,31,763,324,808,494,451,523,450,317,293,244,210,177,43,29,20,171,22,572,501,681,582,350,743,615,365,335,262,217,203,105,87,69,490,482,641,370,26,11,81,73,65,53,4,579,192,963,367,342,267,260,241,239,216 }, +{ 120,82,260,450,351,15,141,21,854,77,217,13,23,51,1,202,817,180,33,898,110,48,326,93,102,32,128,137,5,165,523,269,352,318,125,291,922,2,177,203,275,403,317,4,210,12,373,494,115,64,197,96,386,7,144,0,342,196,36,457,752,37,349,453,9,6,16,221,50,237,615,59,10,475,341,14,372,730,244,18,22,223,285,160,67,365,86,498,661,421,402,401,31,901,308,719,391,70,3,99,35,44,24,95,90,28,473,899,69,30,65,864,704,116,11,29,8,83,367,17,163,49,551,304,961,216,61,692 }, +{ 23,51,13,15,961,0,2,10,115,1,457,5,569,453,4,7,9,719,14,33,629,11,165,40,197,6,77,21,64,3,474,139,38,401,59,22,72,57,102,12,111,475,32,79,341,730,386,128,54,37,509,177,17,8,196,391,36,237,141,202,277,18,89,148,76,99,734,28,758,308,41,117,498,180,90,403,873,365,482,31,45,328,94,546,48,153,86,210,93,317,24,468,494,247,304,300,269,82,16,95,402,120,670,134,46,488,757,194,110,55,62,125,155,29,188,52,405,49,899,775,50,275,221,880,477,44,152,523 }, +{ 264,166,435,9,39,254,30,382,921,404,438,363,656,190,100,897,411,478,93,203,812,274,625,1,795,691,88,156,101,473,326,110,265,639,340,51,447,272,141,18,35,165,131,33,329,67,0,354,77,910,616,291,914,530,105,68,118,197,731,137,120,23,21,578,128,36,333,49,593,494,271,98,676,310,102,44,664,613,48,349,28,228,195,5,960,658,800,180,24,724,96,682,678,403,160,688,523,127,829,242,521,83,31,716,13,342,159,69,16,144,260,352,32,269,55,167,158,219,576,364,704,4,615,202,65,60,318,456 }, +{ 252,9,0,166,265,100,364,329,404,39,101,520,676,419,68,118,553,604,382,88,167,274,18,212,593,333,158,264,16,49,806,865,438,48,812,21,154,30,677,310,1,35,656,24,96,124,51,857,22,13,23,105,28,60,31,71,286,562,65,777,435,455,190,98,639,159,759,109,917,254,344,191,647,125,10,55,907,694,363,221,44,127,5,557,131,327,682,411,67,628,95,83,538,936,149,925,272,36,664,175,375,478,121,64,793,146,821,50,671,69,302,271,921,116,625,541,942,446,530,724,709,706,33,634,463,170,128,223 }, +{ 15,4,11,141,5,13,21,23,202,40,269,32,12,165,523,3,180,64,318,17,352,457,197,1,93,0,110,2,125,120,128,386,51,59,77,31,117,326,49,10,719,403,961,82,153,48,137,134,899,450,115,260,494,196,44,33,37,730,6,57,203,817,177,752,22,98,9,90,18,373,237,8,7,217,958,351,291,14,139,95,341,144,27,210,475,615,402,391,56,453,901,116,19,143,86,349,244,317,72,105,498,123,485,365,111,152,342,401,76,28,30,247,678,801,65,45,216,36,569,300,308,509,221,99,127,84,164,873 }, +{ 24,35,69,18,71,68,46,67,16,1,2,50,14,149,108,133,189,28,38,75,83,0,7,9,218,65,238,114,52,335,29,54,251,60,55,309,236,175,118,336,486,267,96,394,181,220,366,87,418,124,154,5,545,412,516,140,70,22,487,12,21,36,101,171,433,641,230,328,58,48,458,345,361,398,283,92,6,37,103,514,302,906,10,388,596,565,273,443,64,160,532,163,399,705,223,891,86,150,262,537,32,51,39,831,23,109,95,375,155,13,99,480,571,45,49,128,281,696,112,194,754,25,137,294,602,208,323,191 }, +{ 90,214,289,874,64,712,751,77,51,13,177,32,312,23,192,66,194,202,216,59,288,4,835,875,65,0,249,247,95,519,102,57,5,123,33,22,524,234,104,450,173,49,437,500,889,210,217,636,319,597,21,153,416,196,38,20,117,954,304,115,134,120,18,715,86,583,113,494,61,204,10,784,291,28,275,342,147,318,879,43,74,142,11,588,888,403,864,237,2,198,352,426,752,707,48,1,639,17,81,180,750,861,52,315,207,560,24,128,402,40,141,16,671,31,372,36,130,455,260,12,9,317,37,325,349,389,904,306 }, +{ 9,0,49,127,28,301,98,371,395,159,512,158,916,737,761,31,101,24,623,16,272,68,18,39,35,293,242,1,286,109,44,385,327,65,170,299,55,125,51,23,22,518,105,836,21,116,616,137,685,284,48,599,344,252,7,124,13,276,64,187,95,506,5,131,890,202,456,627,36,2,219,30,254,333,419,763,14,347,12,154,578,320,96,713,141,935,201,144,338,442,870,118,71,128,10,364,168,374,67,60,37,310,221,100,32,318,424,93,269,393,910,165,403,86,688,160,793,908,50,83,69,203,421,492,401,6,244,166 }, +{ 17,136,62,214,123,129,249,710,119,162,618,292,209,141,32,183,77,128,211,106,64,45,4,11,179,269,507,519,207,59,177,130,3,40,90,86,255,378,33,202,76,57,205,655,790,152,678,38,21,245,37,742,248,5,102,597,476,306,556,41,291,229,117,349,80,339,153,313,266,318,84,54,947,184,56,12,155,134,240,197,305,157,304,217,148,186,165,275,257,789,99,216,13,497,317,72,137,115,10,196,460,383,619,61,51,188,361,457,369,425,50,290,440,14,342,263,180,503,452,241,208,237,194,432,381,65,111,22 }, +{ 5,107,581,441,356,493,362,660,279,534,32,683,298,423,259,786,22,612,590,359,21,13,49,147,61,508,241,263,929,48,97,246,14,126,0,132,7,465,939,232,64,230,607,95,121,941,23,92,771,115,470,314,51,387,626,172,1,58,12,347,779,392,407,17,150,687,16,357,28,146,43,77,87,60,199,197,37,40,711,178,501,65,10,9,180,140,35,33,24,595,102,120,56,30,70,207,344,324,125,46,31,165,18,20,577,414,376,2,350,4,260,427,173,408,195,141,247,76,603,217,86,11,379,55,729,90,75,50 }, +{ 51,475,453,13,23,15,730,403,457,421,386,523,899,719,670,401,12,361,6,99,33,1,128,269,961,141,27,19,5,494,165,21,64,2,450,153,202,117,65,115,236,7,196,77,95,32,328,37,14,10,0,318,134,17,391,197,102,308,40,26,120,9,22,4,125,412,389,16,34,194,57,569,402,54,53,8,352,25,758,291,59,3,880,757,503,488,217,90,38,72,48,45,801,177,11,304,509,326,283,260,180,139,137,76,47,41,867,350,629,558,949,838,832,803,775,773,770,754,749,657,597,576,543,514,468,297,255,250 }, +{ 49,159,272,28,9,327,254,158,22,347,95,627,374,286,131,424,105,168,35,39,912,65,68,578,688,0,538,55,24,518,175,154,124,239,634,346,289,224,169,913,577,16,320,364,747,219,185,616,101,716,67,344,1,890,570,170,30,810,109,333,228,420,21,36,439,64,149,23,127,371,194,13,706,827,125,51,114,312,83,187,98,429,613,10,137,829,610,86,310,166,910,31,835,190,406,71,18,242,202,846,141,938,160,44,14,2,385,69,7,48,308,301,258,128,638,237,115,12,165,96,100,103,562,703,221,116,164,50 }, +{ 15,33,77,4,102,317,341,141,117,365,134,13,180,1,40,153,5,115,23,11,197,10,217,196,21,901,22,64,752,352,143,253,244,351,137,51,32,2,269,177,82,59,165,12,3,291,569,202,120,0,854,139,342,86,55,36,372,31,9,898,48,498,349,37,260,7,28,111,125,17,482,49,318,160,457,6,450,44,128,90,275,237,95,210,57,386,873,638,98,523,105,164,89,93,30,719,922,961,509,308,8,14,72,951,403,401,730,391,304,116,326,39,468,56,38,717,94,24,579,817,76,61,96,16,18,546,629,494 }, +{ 134,13,15,23,12,474,569,51,37,10,197,457,961,99,115,4,38,2,165,0,153,5,3,11,719,64,139,431,1,277,120,82,33,6,94,873,14,401,110,7,89,117,351,40,498,128,260,148,54,93,72,629,453,180,21,31,22,9,405,143,141,8,17,77,36,102,44,386,41,509,196,59,450,125,730,55,342,523,86,403,326,45,95,898,98,32,854,237,365,217,391,775,116,349,291,210,179,203,202,317,752,901,372,341,352,177,402,49,57,76,922,475,28,30,144,373,421,318,308,269,90,494,413,46,18,678,899,242 }, +{ 7,2,66,20,58,29,52,74,128,14,135,5,32,97,192,43,38,61,6,631,16,151,87,21,25,315,172,177,46,48,173,186,45,884,70,90,113,77,227,72,226,23,102,51,137,35,210,337,202,59,40,104,402,155,115,269,89,592,81,33,122,13,60,174,112,107,24,19,243,0,145,222,580,76,37,49,64,10,416,318,738,18,353,204,26,491,678,95,4,290,141,54,232,69,92,22,415,140,712,270,217,12,171,496,86,220,27,17,549,294,209,1,355,609,494,425,285,50,665,148,403,560,36,216,400,75,275,307 }, +{ 77,202,13,23,128,318,33,269,141,102,403,117,4,342,153,134,51,40,64,196,32,0,49,177,22,494,291,137,11,21,59,65,95,475,98,165,210,90,18,115,48,217,197,15,275,105,10,120,127,86,28,180,9,453,352,36,39,317,31,125,5,194,402,57,450,349,372,14,116,44,16,300,6,2,284,96,24,20,30,457,12,859,678,299,289,242,114,110,109,61,7,1,216,214,100,326,304,260,237,3,82,789,752,579,576,490,347,344,293,207,205,144,139,93,37,615,373,351,285,421,401,961,391,203,817,923,922,901 }, +{ 2,14,1,46,29,38,6,52,103,50,65,36,67,5,108,75,83,7,22,24,25,69,28,86,208,133,0,323,19,26,484,18,309,782,96,283,236,238,124,160,135,21,10,12,74,87,66,168,17,262,70,412,23,389,16,51,650,278,289,532,45,194,3,756,218,32,220,58,646,244,42,115,95,37,27,13,48,886,49,267,366,226,302,128,54,610,64,516,547,584,137,253,9,308,143,388,141,717,335,480,401,164,185,357,505,109,150,732,602,420,99,285,172,202,891,112,189,324,181,35,418,696,331,90,114,55,294,458 }, +{ 15,5,1,2,3,4,12,11,13,10,0,341,21,180,7,33,197,6,134,17,77,32,365,23,901,37,117,9,8,40,165,102,153,961,51,498,196,569,14,873,752,217,22,94,64,317,59,36,352,474,31,509,99,141,110,55,291,111,457,120,391,57,19,16,82,48,177,401,38,386,349,202,308,139,93,342,453,72,275,18,44,143,351,28,27,90,128,403,269,372,45,137,76,260,318,89,326,210,86,277,898,413,237,719,148,864,730,96,95,79,373,475,61,24,46,30,523,253,98,482,83,546,54,49,50,41,899,758 }, +{ 203,268,206,940,93,417,959,499,473,137,31,44,728,692,8,120,202,559,564,450,127,141,260,326,817,244,22,10,0,23,551,318,144,125,13,523,269,128,299,98,1,253,373,51,494,403,9,116,341,143,64,110,638,285,5,615,642,859,367,55,918,576,21,421,180,386,922,165,105,36,401,2,365,115,899,242,352,28,951,393,704,86,30,901,475,7,762,276,77,384,948,237,32,197,49,492,434,160,661,489,170,15,39,752,317,498,854,453,282,717,6,18,16,569,395,479,351,805,12,958,898,456,48,293,851,308,301,509 }, +{ 15,120,82,10,165,31,0,197,64,260,33,8,93,110,22,351,40,44,77,23,153,13,9,102,4,51,196,125,326,11,450,49,139,28,237,203,386,98,457,18,32,373,95,523,1,21,277,59,854,180,111,202,12,141,898,17,36,116,144,14,115,922,86,57,128,5,45,30,2,391,217,7,817,37,177,730,90,509,719,72,99,16,76,62,453,291,352,475,318,269,403,341,48,134,961,55,24,3,143,899,308,401,137,65,38,615,210,629,317,901,758,342,569,494,661,365,39,247,164,421,704,546,61,6,811,41,642,752 }, +{ 60,7,16,71,43,0,14,20,28,97,68,22,92,24,149,2,35,10,154,13,126,615,608,51,124,362,140,55,805,18,958,576,566,175,23,359,6,1,48,453,158,150,703,21,421,260,173,475,298,153,107,74,64,36,29,925,46,12,882,899,441,801,81,357,246,196,77,69,38,25,9,117,65,49,314,37,484,386,730,336,115,570,403,356,324,221,135,121,19,401,99,939,420,399,859,942,628,610,289,285,259,125,101,95,33,26,702,677,165,4,653,375,185,192,91,515,302,197,104,326,595,318,230,202,191,143,102,90 }, +{ 26,19,6,27,80,25,84,45,17,5,504,42,129,2,240,123,618,29,56,106,113,710,4,76,119,128,14,136,65,64,73,59,1,209,483,53,162,51,90,548,122,205,249,202,40,99,3,460,32,257,62,292,61,214,77,245,57,12,378,141,183,453,34,297,11,184,108,23,255,469,0,403,22,33,58,95,10,86,814,318,769,114,179,402,611,667,130,475,21,151,210,137,742,161,38,177,494,7,436,269,102,194,115,445,18,36,421,13,47,153,880,775,54,133,196,37,49,495,306,311,112,46,52,117,250,28,491,48 }, +{ 15,77,13,33,4,23,102,11,51,5,40,177,0,117,10,202,64,8,93,22,291,32,21,44,31,153,110,1,141,49,115,59,82,180,9,17,318,196,98,269,351,403,137,453,36,95,457,217,3,90,139,128,48,28,898,12,165,450,475,349,341,197,523,18,30,210,317,86,16,260,401,2,961,901,719,57,116,494,854,326,373,125,365,817,352,730,275,134,39,386,105,76,24,7,308,372,37,342,6,421,120,704,144,14,899,922,203,111,304,569,638,72,661,61,629,96,127,99,811,535,498,65,615,752,247,83,402,758 }, +{ 195,271,98,223,167,132,407,146,360,591,1,212,834,121,522,393,293,199,259,354,446,668,590,562,807,191,156,48,765,125,107,147,941,163,727,541,535,465,126,284,21,346,31,279,23,55,18,5,687,16,60,640,13,697,586,178,534,35,20,608,581,150,955,427,68,70,589,43,142,791,463,198,116,51,883,362,9,963,781,202,49,0,818,96,493,7,948,281,120,165,367,798,450,10,44,318,77,105,14,325,221,92,28,22,61,770,101,777,58,93,32,97,467,141,386,33,144,371,75,127,37,108,929,269,411,598,551,128 }, +{ 210,128,202,77,402,318,102,403,33,342,275,40,196,475,6,300,139,29,177,99,494,269,42,503,111,757,277,141,117,52,2,4,678,13,153,12,0,23,59,285,37,137,85,81,7,291,859,657,351,122,82,576,44,31,880,260,216,120,91,78,63,14,11,1,745,65,64,93,854,27,684,22,485,180,76,74,19,3,21,805,49,38,281,48,966,838,631,452,450,421,218,134,26,25,569,341,115,898,738,592,110,98,10,315,788,707,491,483,432,396,319,268,242,125,116,95,62,57,56,50,17,16,9,655,526,453,426,422 }, +{ 40,33,102,23,51,13,117,196,153,77,134,64,128,202,4,59,141,453,65,475,11,403,22,95,0,137,14,139,15,342,210,12,86,318,269,165,115,1,49,10,494,177,28,180,36,291,421,21,57,5,6,217,18,194,99,401,2,457,108,719,125,19,111,260,82,37,27,48,17,3,523,197,133,26,402,120,114,670,352,7,24,730,93,216,90,46,237,50,32,386,361,9,341,351,25,349,300,283,277,38,34,901,365,326,275,98,110,285,45,41,391,44,317,576,961,242,143,16,509,170,31,854,308,54,116,817,757,629 }, +{ 7,14,16,24,18,28,71,2,92,60,35,22,20,0,154,118,168,6,1,97,69,703,302,149,140,150,185,38,109,43,66,68,108,420,29,55,10,158,610,12,65,67,5,701,189,273,570,95,230,429,46,124,246,289,298,51,21,169,143,49,175,23,107,747,126,671,13,64,48,159,25,925,677,137,36,52,398,32,9,251,584,467,164,128,942,19,86,577,847,83,173,336,403,74,50,424,202,232,882,172,17,344,279,638,375,105,314,104,37,125,115,441,356,505,99,253,239,26,221,359,329,194,533,837,517,165,494,601 }, +{ 23,1,15,51,2,13,5,0,7,180,21,9,6,115,197,120,12,165,4,33,457,386,523,202,141,3,14,260,450,48,8,16,77,93,31,341,37,269,40,403,44,730,110,326,32,102,317,137,318,453,352,719,365,18,128,217,153,82,475,36,98,64,391,752,401,901,899,421,308,196,11,498,22,28,351,99,116,10,117,24,817,96,221,30,39,494,509,569,177,203,49,961,144,291,373,615,59,17,801,372,482,72,758,90,89,854,50,629,134,127,35,86,349,922,160,55,244,105,237,143,468,898,210,54,46,670,342,45 }, +{ 132,146,199,107,60,121,126,279,150,441,92,649,16,955,493,195,598,423,653,298,35,407,356,640,362,508,32,21,953,7,360,359,522,18,271,783,5,246,167,0,660,156,534,939,71,191,22,517,590,24,49,259,314,10,607,28,392,13,232,147,68,97,95,14,43,883,118,1,37,581,48,501,354,809,55,108,4,591,12,20,23,347,345,212,467,230,446,241,125,50,941,75,9,31,98,683,929,77,2,672,33,713,463,702,273,61,40,59,70,46,101,58,172,586,357,296,115,779,36,350,109,727,388,236,207,87,105,408 }, +{ 31,44,276,284,98,299,935,116,201,662,9,39,492,144,489,131,567,371,268,0,49,918,127,137,367,254,28,434,293,737,30,203,1,295,22,456,93,125,219,385,23,202,187,393,21,716,105,599,578,51,36,128,141,836,272,269,244,318,384,13,159,395,48,261,96,110,224,616,326,242,35,16,253,5,95,55,64,160,258,158,143,18,100,699,165,24,115,948,120,37,403,86,166,310,12,924,457,855,940,77,10,442,494,101,109,7,197,317,206,68,2,373,617,761,352,285,908,217,332,341,916,228,32,33,518,688,730,473 }, +{ 13,15,23,51,1,0,10,2,4,115,77,5,9,202,3,197,457,7,22,11,165,6,961,12,33,14,59,177,403,21,102,40,453,180,719,217,18,31,120,141,32,475,8,318,569,37,16,341,210,17,386,450,401,28,269,64,99,326,494,317,730,308,44,523,93,352,291,90,125,365,260,36,275,24,134,49,139,30,110,498,421,98,57,196,391,19,117,72,342,153,48,817,873,349,482,901,221,629,76,237,111,899,86,95,216,509,752,304,137,247,82,372,116,758,89,27,38,39,65,203,55,402,45,29,373,300,143,615 }, +{ 162,123,257,184,183,17,229,130,84,136,497,556,152,383,129,452,99,80,3,205,62,57,56,266,106,119,12,128,40,4,161,378,27,14,495,504,381,202,769,469,249,59,77,432,11,34,33,305,64,90,403,240,38,54,5,102,45,32,23,73,618,475,51,361,440,139,775,37,402,471,318,53,21,494,65,141,19,372,453,313,421,177,880,472,757,217,245,548,250,527,2,76,524,349,61,86,216,153,210,196,115,179,10,619,328,269,291,214,72,483,117,209,6,26,22,477,111,95,13,401,114,157,369,211,36,134,48,306 }, +{ 196,153,117,453,134,33,40,23,102,51,13,475,0,14,22,15,12,2,64,670,21,401,730,4,18,99,65,10,9,11,1,28,95,57,128,77,5,38,46,37,403,386,202,719,24,6,108,457,899,139,31,59,16,82,36,165,32,120,961,86,236,48,8,342,141,775,125,308,137,7,54,523,50,269,197,494,318,115,300,260,39,758,450,180,421,277,90,217,341,629,361,49,3,326,114,93,391,291,210,194,328,317,111,17,352,558,110,377,29,275,283,133,485,817,898,615,801,351,160,19,488,569,350,373,109,84,56,183 }, +{ 224,219,187,258,442,871,908,385,836,131,944,574,839,127,98,613,31,116,926,44,137,829,860,160,827,276,39,254,284,36,144,228,201,96,406,1,371,530,202,856,716,93,30,51,9,203,268,737,902,492,23,49,21,190,13,318,843,531,326,269,924,125,367,141,64,77,120,166,816,935,800,242,299,115,456,260,293,274,931,494,395,567,110,217,662,5,317,22,717,100,143,434,128,28,48,86,533,10,295,105,332,55,489,180,365,918,418,102,578,197,244,83,341,576,403,384,625,165,261,177,159,352,291,822,863,473,251,206 }, +{ 195,121,360,590,146,407,156,786,522,132,883,591,626,929,941,259,687,279,379,586,296,465,467,150,178,354,5,21,55,534,70,60,783,771,13,346,340,48,271,581,49,493,163,1,199,147,608,668,125,955,167,562,535,98,518,108,362,18,371,212,446,107,761,878,31,126,50,463,35,818,105,75,120,598,87,0,767,223,165,640,22,447,293,777,159,16,23,9,142,115,658,92,481,44,95,37,807,93,61,58,24,191,33,7,846,395,12,197,4,385,127,423,71,140,82,427,144,116,17,117,2,363,698,246,39,64,32,781 }, +{ 2,1,29,14,6,46,52,26,75,38,5,25,19,50,96,70,108,114,309,36,103,74,17,65,27,112,45,262,236,220,145,218,58,42,86,516,24,366,22,7,67,21,12,66,532,87,412,18,480,283,602,323,137,3,163,226,238,0,194,51,32,28,10,69,171,37,23,115,208,128,331,394,135,48,99,64,13,433,285,160,61,782,113,357,308,222,95,484,401,16,141,294,151,287,223,83,646,124,389,361,894,866,545,421,54,324,4,278,388,396,186,650,594,53,168,453,289,202,59,253,181,90,273,138,150,403,756,606 }, +{ 15,82,120,10,197,260,0,165,351,8,64,31,110,93,9,22,117,13,457,450,1,44,33,23,326,102,49,125,898,153,719,77,854,523,18,14,98,12,40,7,961,2,115,373,180,5,51,386,36,111,141,95,730,28,196,922,139,203,55,116,4,817,498,899,128,30,177,86,569,144,661,99,37,202,16,24,65,318,21,873,11,59,6,277,217,391,269,341,38,32,291,143,237,352,210,317,3,365,509,474,349,758,704,642,216,39,615,137,453,275,403,401,94,300,304,629,247,372,242,342,90,48,752,901,801,308,221,811 }, +{ 7,66,97,172,232,192,226,2,74,324,52,29,135,92,568,222,43,107,353,87,387,580,500,38,620,448,331,526,140,330,5,138,294,174,356,20,14,262,298,32,357,145,173,853,6,359,21,126,58,16,204,808,64,888,337,104,314,673,35,22,729,95,872,25,278,380,70,802,37,583,426,1,49,12,112,69,75,65,86,61,45,60,48,10,246,90,230,0,42,355,24,220,415,85,81,889,46,885,36,825,287,28,194,72,171,792,392,632,560,280,312,50,71,115,76,54,26,18,151,155,707,587,89,17,4,764,147,59 }, +{ 62,129,123,162,183,507,618,136,249,152,211,57,77,184,205,59,130,4,128,41,11,40,117,17,229,497,361,257,3,33,38,313,452,266,378,248,153,64,106,90,210,383,119,12,328,214,99,432,216,157,196,86,202,556,179,54,102,80,37,381,402,471,134,245,177,5,514,369,188,339,440,305,275,524,141,45,269,84,710,529,291,403,139,342,115,56,318,209,76,504,421,372,742,619,292,240,237,79,27,14,775,22,19,10,23,457,95,21,207,488,13,494,180,542,65,300,401,306,32,34,114,26,215,25,217,769,151,110 }, +{ 13,1,4,5,2,23,115,15,3,51,6,0,7,77,341,217,11,141,177,33,317,12,180,21,365,197,202,32,318,352,269,102,901,9,291,10,165,40,403,752,59,22,8,31,401,569,349,494,120,308,304,44,110,90,17,386,64,93,37,247,153,210,326,14,450,28,373,117,16,372,48,196,457,260,98,275,475,498,237,453,125,30,18,203,523,342,49,509,391,36,961,128,873,134,719,116,629,24,482,421,39,221,216,139,244,72,730,253,95,817,55,86,143,615,57,661,351,704,144,127,899,105,76,864,576,82,922,758 }, +{ 6,26,235,145,19,47,112,78,648,624,630,27,85,64,42,95,122,444,25,644,113,70,108,453,29,632,130,711,53,222,74,390,32,65,237,412,416,280,665,287,813,138,730,307,73,680,283,14,196,52,34,4,312,51,236,226,90,262,17,337,22,180,59,331,500,128,86,884,315,194,690,114,133,498,849,137,50,200,3,256,21,2,49,45,33,23,13,66,141,409,445,297,288,57,381,719,11,396,289,247,87,163,165,1,250,5,46,115,153,525,99,183,76,746,324,63,792,675,294,316,56,214,28,583,751,125,285,503 }, +{ 51,23,99,12,15,13,153,10,117,403,120,37,82,165,128,453,401,64,202,197,2,115,110,14,260,457,21,475,5,308,0,38,93,719,31,3,22,134,141,1,33,196,386,180,351,17,421,569,44,326,125,523,961,11,730,269,137,402,318,450,4,494,8,49,36,9,98,18,7,95,391,6,54,629,328,32,237,28,40,19,86,775,373,57,116,41,854,77,79,59,899,352,94,474,102,203,509,477,65,758,89,341,285,221,901,217,111,177,16,361,144,30,880,757,498,46,29,27,670,105,898,143,24,62,72,277,317,55 }, +{ 16,24,71,18,92,108,35,60,146,64,7,118,246,199,126,121,32,50,9,267,649,156,167,830,271,133,273,140,36,68,5,230,132,247,21,14,1,640,12,563,48,0,87,388,75,2,237,220,330,86,46,354,115,65,672,304,128,55,751,522,180,360,95,28,236,191,516,114,163,696,149,953,189,22,54,565,141,51,137,217,674,412,352,49,340,23,633,37,309,458,202,39,10,394,58,103,70,597,125,262,72,38,947,726,591,502,437,195,164,363,67,641,514,602,954,124,864,448,107,150,143,101,366,283,13,678,317,177 }, +{ 15,2,0,13,1,4,3,5,23,7,217,6,9,115,51,8,341,10,11,317,372,180,365,77,33,12,22,247,901,18,120,197,457,14,16,165,291,28,21,202,31,349,961,719,102,260,32,453,30,352,523,40,752,569,730,24,82,450,304,308,110,437,93,117,44,177,36,351,196,403,64,17,318,475,153,899,134,269,494,922,401,386,128,48,482,316,326,37,137,49,57,873,125,39,854,59,55,509,90,579,342,35,817,629,758,373,210,98,498,237,898,116,65,111,144,139,615,19,670,241,20,56,638,468,89,95,275,801 }, +{ 234,639,178,416,77,455,5,49,142,202,450,427,0,147,329,21,198,315,61,557,344,13,113,32,318,120,22,23,325,48,259,534,494,105,95,347,518,798,39,884,159,58,481,99,817,623,878,291,112,711,70,64,17,1,31,320,51,265,158,65,465,128,362,192,137,236,845,260,403,44,310,141,196,342,177,30,7,271,241,195,135,269,28,290,210,929,15,4,307,170,125,663,102,33,275,263,132,859,408,294,194,786,581,402,301,153,86,523,98,150,423,346,167,25,237,415,36,858,682,592,352,117,100,12,37,10,709,146 }, +{ 31,1,44,36,116,64,55,5,180,2,22,352,98,86,127,141,12,164,13,498,125,115,144,253,170,4,242,137,752,197,128,105,7,244,0,160,237,23,341,21,165,9,365,299,901,301,3,951,317,10,6,28,864,51,338,269,15,268,844,83,169,217,202,873,638,93,569,509,96,824,546,8,37,456,740,293,512,203,77,395,479,318,206,14,776,599,95,506,33,367,276,11,48,685,954,285,239,457,535,201,492,762,717,401,965,533,102,393,482,208,49,109,391,601,558,489,67,870,284,947,961,403,708,946,35,434,923,40 }, +{ 120,217,260,141,23,77,13,51,317,1,180,269,15,115,202,450,137,110,373,291,21,352,64,349,5,177,82,326,351,48,318,854,341,36,365,304,165,197,0,32,922,372,128,96,9,93,752,12,22,523,160,10,2,244,7,457,4,28,898,386,86,901,661,498,719,33,16,275,90,247,203,143,11,253,482,40,37,83,403,730,494,102,50,18,961,615,473,221,237,8,817,30,968,125,704,216,39,59,14,67,6,117,210,35,49,95,72,342,308,24,89,453,55,134,475,76,69,116,17,61,31,3,437,864,70,899,576,98 }, +{ 26,396,648,6,112,122,624,145,287,19,222,416,42,25,573,138,74,651,644,151,66,113,445,45,85,665,280,29,525,204,64,174,337,680,87,884,135,53,65,70,436,27,95,58,297,881,194,73,80,746,544,235,17,86,2,90,243,32,22,5,128,1,609,353,311,141,61,315,307,491,226,106,0,52,81,813,614,209,255,7,632,75,114,10,36,186,580,415,289,63,611,14,34,33,49,214,28,792,117,250,312,283,77,125,172,108,389,76,192,208,355,237,667,56,4,91,165,294,40,348,148,635,102,123,256,153,46,549 }, +{ 156,360,522,271,146,354,21,446,121,132,5,463,591,340,195,727,586,18,48,407,13,49,167,32,668,60,199,566,23,16,447,698,955,575,9,658,296,493,608,150,126,0,534,35,24,581,92,71,590,363,107,279,141,125,640,31,212,4,929,411,467,562,10,1,51,115,783,165,191,22,623,68,598,379,61,12,293,273,202,95,197,303,163,663,269,108,37,101,221,118,621,391,76,28,17,40,109,7,767,260,55,518,120,33,11,77,259,39,347,941,44,252,713,478,98,140,408,350,93,626,346,264,900,509,818,30,170,763 }, +{ 21,5,61,49,98,23,95,32,612,178,259,51,182,48,13,481,120,77,683,660,427,291,147,64,70,4,465,33,344,269,241,0,17,767,318,263,202,141,107,415,817,450,180,22,534,392,362,125,116,347,11,929,350,121,595,408,423,352,293,260,196,165,117,105,102,87,86,59,37,18,7,3,557,376,845,663,579,541,490,356,316,115,90,163,928,711,28,132,590,407,941,922,704,645,523,360,346,284,268,232,210,203,195,177,144,134,118,97,65,60,40,16,1,298,153,146,58,937,798,414,113,31,771,246,207,142,281,647 }, +{ 468,89,358,79,365,94,179,205,317,405,482,498,245,72,115,341,217,180,413,490,111,734,247,431,188,216,37,474,152,452,11,629,569,873,752,749,497,197,4,339,308,13,12,59,901,428,864,579,40,76,23,372,291,875,10,352,134,961,117,77,542,528,401,51,588,349,471,529,33,464,546,477,645,139,221,1,377,38,21,99,36,177,509,64,449,316,14,2,304,143,102,437,9,32,196,741,3,165,5,275,202,128,54,277,237,153,210,743,457,430,300,773,485,460,141,17,0,350,22,288,421,257,270,48,8,184,137,15 }, +{ 24,35,71,68,16,18,92,118,375,108,7,101,302,175,336,154,60,273,14,124,9,398,230,140,0,150,55,149,28,345,189,399,246,191,65,346,69,722,212,46,1,458,167,22,429,314,2,267,126,674,21,388,439,97,67,38,114,10,48,252,628,95,364,251,882,50,133,637,36,49,39,109,12,823,172,768,602,103,236,650,5,83,520,66,232,64,517,32,562,847,694,815,589,702,107,96,51,594,23,695,363,298,37,535,462,484,29,13,796,86,467,886,604,806,868,121,419,125,54,755,451,168,100,135,570,158,6,169 }, +{ 421,453,51,801,730,475,23,523,386,670,401,719,13,629,899,365,403,758,457,115,494,569,805,958,576,543,697,165,615,221,423,817,450,33,488,308,197,102,66,21,514,388,56,14,38,734,341,117,7,326,69,467,961,880,323,196,153,19,202,886,770,128,273,97,120,811,642,532,458,207,110,24,10,6,1,540,67,0,753,554,390,135,134,92,73,52,43,894,37,32,743,214,114,103,90,65,29,891,847,782,125,451,269,208,75,754,602,480,244,229,200,133,130,119,16,9,426,162,83,40,22,2,42,25,203,123,4,901 }, +{ 113,311,614,29,491,145,45,112,80,6,151,631,209,315,74,61,27,2,73,5,287,66,26,243,25,70,396,611,667,52,138,19,58,454,869,122,222,17,186,106,87,255,445,32,64,496,42,297,1,135,684,227,4,128,644,7,86,624,76,337,95,416,65,22,72,85,59,14,204,148,90,21,738,174,250,53,119,141,280,10,483,0,833,137,651,690,549,46,75,226,11,378,37,665,155,742,200,56,194,207,57,177,179,525,12,114,84,77,48,495,40,469,172,292,307,161,49,38,884,887,655,573,881,89,425,18,36,33 }, +{ 471,339,188,4,11,59,377,79,12,99,94,102,77,33,51,111,542,37,342,152,961,13,474,403,40,202,196,153,128,3,452,453,177,229,431,475,401,23,139,117,184,266,157,57,134,115,494,210,300,17,291,402,129,90,32,529,629,5,141,76,775,72,64,275,38,21,318,757,569,277,216,162,14,421,313,123,269,880,89,1,2,497,257,62,197,405,27,45,285,54,304,148,485,217,15,457,41,10,6,179,165,205,180,137,477,349,130,488,245,19,523,432,437,260,22,120,95,769,372,0,352,381,249,618,482,308,237,7 }, +{ 15,40,4,11,33,120,10,82,165,77,110,93,31,8,260,0,23,22,102,51,13,59,44,351,125,64,197,139,32,9,21,326,28,450,196,180,98,203,57,141,386,49,373,12,111,95,1,391,5,457,99,403,115,18,453,116,202,134,475,523,719,854,37,117,128,401,237,48,144,177,14,898,36,217,2,90,308,269,342,758,30,210,318,277,730,494,17,509,922,352,291,817,86,3,65,16,901,24,194,402,105,38,137,7,341,317,29,961,39,629,55,72,76,300,661,752,421,899,242,54,775,247,349,365,569,143,615,485 }, +{ 16,71,24,35,68,18,118,60,0,28,124,92,7,55,9,419,375,14,109,175,149,101,126,252,302,346,150,439,108,154,158,570,169,185,364,22,273,49,65,917,701,329,168,962,420,191,925,39,628,107,2,1,604,46,140,67,97,21,535,265,38,69,159,942,677,10,722,336,105,345,694,467,703,48,5,95,246,553,810,882,20,83,333,43,12,706,13,637,826,100,23,847,671,36,167,279,121,793,189,199,374,50,125,51,212,230,286,66,272,566,6,484,634,344,32,429,289,327,86,103,96,601,114,441,517,37,64,298 }, +{ 17,45,227,106,209,243,21,290,255,400,207,48,263,270,62,186,119,2,496,454,50,32,3,5,52,221,29,425,136,408,430,46,513,171,14,659,54,84,787,721,741,56,281,38,536,80,181,378,240,742,587,552,1,306,76,123,241,541,655,328,428,61,129,41,666,72,502,57,89,12,760,540,115,155,148,383,449,794,37,96,361,152,11,34,214,4,311,51,440,474,592,431,64,790,202,179,405,350,292,36,90,67,663,6,141,59,13,498,103,583,130,697,514,40,211,99,58,10,128,237,217,770,27,869,157,828,834,391 }, +{ 340,354,586,658,668,156,195,698,363,447,883,296,1,411,303,18,379,9,919,360,51,621,98,31,48,591,371,13,21,446,163,5,121,70,132,125,23,759,846,127,761,478,60,293,223,271,146,385,120,664,691,108,763,82,634,638,75,4,260,24,807,774,144,44,522,407,30,457,32,16,914,463,126,777,335,93,771,367,71,49,39,12,0,687,351,118,299,201,40,7,50,848,518,284,922,854,758,538,364,279,11,10,724,310,217,326,64,393,77,541,767,167,150,137,105,103,101,87,58,17,115,286,225,33,401,867,850,663 }, +{ 277,153,111,12,23,51,474,99,38,139,37,457,453,117,542,41,79,629,196,401,13,11,14,569,961,102,33,134,188,89,40,157,54,328,72,775,57,115,17,94,413,339,431,15,377,369,3,477,308,719,248,165,313,4,468,45,485,148,32,197,76,734,216,62,2,21,10,587,405,64,5,59,29,34,6,210,82,509,471,128,155,152,730,177,490,7,266,361,52,120,475,1,0,873,19,305,758,9,899,125,22,110,386,300,211,93,529,229,48,260,237,180,65,50,528,880,77,391,86,36,27,588,326,546,482,402,514,56 }, +{ 18,16,24,0,159,35,68,49,105,9,60,71,101,28,344,55,7,118,320,109,424,14,327,455,239,22,286,170,272,310,1,374,252,154,419,329,167,158,124,518,23,95,709,39,2,21,51,65,13,301,5,890,48,92,265,98,191,212,12,10,108,137,347,128,30,538,31,364,125,202,37,553,20,149,121,557,185,36,677,67,333,169,338,146,127,346,46,175,64,6,126,100,577,50,254,32,512,916,33,403,86,395,69,242,806,371,115,96,44,77,318,221,479,694,616,102,165,623,83,910,627,737,917,38,793,578,846,375 }, +{ 0,105,9,16,49,18,24,68,101,518,158,320,28,35,301,170,272,890,344,286,109,327,39,65,55,512,7,1,127,916,98,118,252,60,737,159,239,371,21,71,124,395,154,14,48,455,623,242,31,23,51,419,374,364,761,5,2,12,137,310,95,125,293,333,846,202,13,616,329,347,338,385,30,37,131,44,938,479,128,836,709,22,538,265,506,64,793,627,10,67,167,299,403,827,212,713,578,149,116,32,36,219,100,910,254,553,175,318,677,108,185,50,6,96,46,685,221,421,424,191,577,92,93,269,599,187,165,284 }, +{ 141,180,1,5,352,365,13,2,4,217,341,115,269,317,15,23,752,3,7,901,6,0,197,21,51,31,77,202,44,318,32,64,372,873,498,10,93,8,11,244,33,9,165,12,102,137,22,253,49,48,237,116,30,98,125,39,177,569,401,28,143,291,308,864,509,40,120,349,403,37,105,391,144,153,482,203,450,110,421,127,72,576,494,95,453,260,247,96,475,373,326,117,36,717,304,131,128,134,90,89,14,457,854,221,206,196,55,811,82,947,629,351,922,386,59,961,734,670,17,437,661,224,201,16,88,76,164,187 }, +{ 51,4,23,102,33,300,40,485,453,880,153,19,17,196,27,342,78,26,53,403,47,77,210,0,475,117,73,2,57,34,11,5,200,13,99,494,14,1,22,12,176,421,64,111,41,21,368,361,59,757,231,130,123,6,139,128,134,401,106,18,56,49,322,317,177,45,775,48,115,165,108,10,125,65,504,495,197,32,605,229,80,341,221,183,390,813,46,37,618,306,95,719,554,152,753,736,543,444,161,402,277,119,630,236,216,122,113,29,730,237,25,899,217,714,383,297,472,445,378,250,38,678,670,527,503,469,412,318 }, +{ 62,440,56,184,229,183,556,130,3,136,548,152,266,99,162,161,17,57,84,452,305,12,383,257,123,403,34,40,527,14,205,349,504,555,497,328,4,361,202,139,775,33,38,37,128,475,129,880,313,381,757,102,11,196,59,119,401,432,402,54,471,453,215,77,23,514,507,585,51,211,153,80,494,495,469,236,477,117,111,64,421,240,369,106,21,214,277,157,308,5,291,141,378,249,13,2,803,619,318,45,76,177,27,485,472,134,41,32,412,339,216,65,524,197,10,72,114,176,165,53,90,372,22,188,245,86,217,543 }, +{ 13,23,115,51,141,202,77,64,269,217,33,318,102,32,21,128,137,0,177,180,4,5,291,9,48,352,10,197,90,196,317,40,153,7,372,18,15,349,22,2,247,165,275,6,403,11,117,125,237,59,3,16,752,494,28,342,304,31,341,134,14,98,86,210,36,241,12,244,49,95,24,498,30,116,39,1,143,901,453,20,450,144,365,402,221,17,475,93,203,127,65,253,216,72,326,120,164,576,35,19,96,44,437,160,421,308,83,67,864,954,569,817,719,25,101,100,61,55,57,284,805,523,401,60,293,285,105,88 }, +{ 1,22,36,105,170,28,55,239,86,31,2,95,5,9,127,0,98,301,12,64,67,7,49,320,109,654,44,83,10,169,208,143,21,312,35,13,23,128,141,740,125,242,14,512,293,116,160,237,51,395,50,338,37,159,96,6,933,114,115,708,299,751,185,137,844,505,269,168,69,48,638,103,597,928,202,180,244,323,16,352,253,165,3,713,197,194,892,39,24,599,65,54,341,285,318,286,46,685,456,951,4,623,824,32,133,158,365,584,601,535,38,498,144,218,251,841,533,506,30,371,217,479,8,503,284,870,752,276 }, +{ 15,0,1,2,5,13,23,7,901,9,6,8,12,51,4,3,457,180,341,898,82,22,115,10,141,120,117,31,351,14,202,260,197,28,269,33,153,961,64,30,44,37,93,18,569,854,11,21,16,77,217,137,365,386,719,237,134,391,110,128,523,49,450,36,629,498,352,102,125,326,317,40,401,509,730,318,196,24,203,546,308,39,116,922,55,752,32,86,817,17,403,90,98,35,453,95,373,59,99,873,177,758,111,143,38,291,242,139,65,48,244,899,372,285,642,349,247,144,734,638,615,482,277,72,253,421,89,475 }, +{ 274,98,100,39,1,265,438,190,310,625,382,30,530,223,909,96,264,593,166,778,88,812,410,960,731,557,252,31,125,21,539,23,254,9,48,51,116,639,44,478,393,131,127,13,101,0,293,284,329,795,404,840,144,656,435,333,724,616,5,219,228,321,272,421,834,165,921,957,224,105,32,36,93,128,697,367,541,669,765,242,221,49,676,137,418,455,160,748,83,897,163,202,682,578,475,453,553,386,807,217,213,551,613,406,335,403,141,466,197,827,180,948,37,158,187,115,905,821,791,203,352,286,258,371,120,863,730,2 }, +{ 212,18,118,447,167,781,363,411,271,647,146,121,60,562,664,621,463,815,478,1,16,446,9,777,354,628,21,936,48,101,932,132,68,191,98,520,522,223,71,264,5,24,759,55,199,575,727,566,126,13,23,724,156,360,691,125,0,407,591,608,49,293,604,221,51,35,92,195,346,31,107,252,108,375,586,96,154,914,682,541,807,150,39,589,124,848,265,69,22,105,50,834,32,364,302,393,329,12,4,10,382,535,404,28,166,765,100,668,273,33,149,88,706,806,163,279,95,410,598,44,335,955,7,30,67,284,159,467 }, +{ 24,28,14,7,35,16,22,65,0,1,2,158,49,109,55,159,124,95,18,67,149,105,71,239,169,68,108,185,424,289,9,12,69,46,577,10,175,154,320,5,164,347,6,747,344,21,835,36,286,50,38,83,170,654,892,23,420,92,114,51,64,143,60,13,374,189,302,429,140,48,103,570,118,32,37,29,336,610,133,327,312,398,39,484,128,638,168,301,272,125,202,137,86,150,30,194,230,671,345,584,703,96,54,115,273,251,310,283,141,518,98,308,20,538,401,101,127,52,31,221,346,455,535,236,403,389,458,267 }, +{ 105,272,131,22,286,239,320,327,98,127,219,9,185,224,374,187,55,28,578,86,31,258,36,913,616,535,1,95,538,64,164,837,634,109,116,601,44,654,312,170,910,49,708,623,115,23,5,0,713,371,201,159,13,505,2,197,83,892,7,35,67,180,533,928,51,933,317,737,301,751,10,716,352,284,6,160,776,237,509,916,165,242,498,251,531,12,395,39,924,137,638,385,293,269,30,276,168,141,217,254,752,208,144,569,896,836,143,346,24,824,21,4,546,468,169,308,14,761,128,345,244,318,482,365,3,597,391,384 }, +{ 28,301,22,31,127,512,105,98,395,109,44,299,293,95,9,320,1,599,685,900,55,0,239,36,763,338,242,49,116,125,841,168,870,456,286,23,276,506,86,35,284,51,185,65,492,159,2,24,374,479,13,393,928,5,144,7,64,169,713,141,12,137,327,21,202,14,268,237,16,935,10,201,558,272,538,67,401,367,37,143,83,308,424,164,312,128,158,115,48,165,918,654,194,269,923,740,39,347,244,623,18,6,489,253,318,114,963,160,352,289,124,180,68,96,420,170,203,175,131,421,219,434,197,391,221,762,453,93 }, +{ 2,58,29,52,1,186,334,46,5,14,151,75,45,400,155,50,61,227,163,262,502,69,549,70,220,223,48,38,311,112,67,409,113,780,7,243,675,767,27,17,209,24,103,80,394,96,21,315,54,307,513,536,83,287,787,309,37,281,6,16,106,807,171,587,496,267,19,32,631,474,270,491,12,35,181,26,583,454,74,869,820,852,36,324,721,255,845,278,64,73,335,357,89,87,140,72,537,25,760,290,18,659,614,430,189,66,97,133,663,114,108,592,552,128,331,666,764,141,606,3,86,380,486,366,76,65,148,10 }, +{ 7,107,232,135,97,387,92,356,298,66,14,2,192,359,278,172,52,357,126,324,448,35,60,526,314,16,38,29,46,330,380,43,362,259,392,230,20,226,74,140,262,246,279,147,24,825,64,58,5,121,568,87,294,32,220,501,132,21,612,465,95,150,86,104,620,18,12,71,173,581,54,534,0,893,37,22,808,583,178,353,90,10,580,550,146,707,802,600,606,331,194,112,222,28,69,660,853,65,70,199,171,68,560,1,174,36,50,77,49,48,779,500,204,683,493,414,6,61,25,764,55,138,33,102,750,888,307,45 }, +{ 20,43,104,414,560,426,707,784,173,319,861,7,819,81,715,422,879,0,376,330,74,95,97,52,32,192,64,38,22,49,526,356,66,204,681,397,207,140,232,5,470,65,693,61,359,636,387,500,107,324,28,91,712,18,10,29,147,77,21,595,392,113,577,172,729,2,325,24,138,92,33,347,125,86,945,102,90,306,40,115,4,198,370,583,230,174,12,904,180,11,858,312,153,124,17,587,750,76,59,69,344,298,3,135,13,37,46,825,718,357,165,348,315,214,72,887,117,289,353,246,196,141,84,36,427,51,237,48 }, +{ 104,74,636,204,355,222,66,81,0,145,25,319,784,746,192,750,29,20,173,174,792,64,90,138,280,715,65,194,91,879,525,32,22,4,348,287,43,95,18,422,7,214,415,86,77,42,40,6,681,289,59,524,151,33,10,52,712,2,28,544,945,13,597,102,45,177,97,21,57,11,243,49,85,36,312,500,426,693,141,5,673,624,396,16,1,122,87,115,648,24,26,186,632,233,67,17,172,414,304,416,123,12,353,389,117,651,580,69,588,751,153,113,874,112,14,23,227,216,128,58,217,197,27,875,114,635,137,707 }, +{ 101,0,9,35,68,154,252,39,65,28,364,124,166,100,149,336,158,289,346,439,429,419,67,55,114,570,694,30,16,265,24,109,484,88,1,329,69,194,404,637,49,189,389,547,133,398,703,826,671,283,22,168,610,175,212,190,7,83,345,103,64,14,768,399,71,274,333,344,230,95,799,50,21,706,36,722,677,420,10,185,962,51,23,18,553,140,254,159,125,967,535,676,46,48,310,2,86,96,952,264,13,12,169,382,5,596,108,323,705,782,251,118,208,857,702,756,562,865,438,218,60,267,131,237,37,755,105,272 }, +{ 57,14,4,236,231,59,369,361,300,585,342,51,23,719,401,13,38,176,46,202,62,402,3,56,210,54,12,2,412,803,381,457,215,11,507,128,40,757,555,77,514,961,32,730,629,322,161,328,37,162,33,133,283,108,129,21,211,34,102,494,17,152,130,29,99,139,153,52,527,472,181,79,123,475,229,6,497,899,453,240,758,194,205,184,188,183,177,0,136,377,305,134,155,248,165,157,425,290,440,714,495,209,739,196,277,269,80,50,556,10,805,64,65,523,503,386,257,141,16,7,5,179,403,84,600,469,318,180 }, +{ 2,1,3,5,141,6,7,15,0,269,180,64,33,77,10,352,102,13,202,128,318,11,197,17,14,4,291,12,134,117,9,120,752,165,153,196,137,217,498,341,349,260,110,21,342,93,40,31,237,23,317,44,82,115,22,16,59,19,86,51,351,678,32,57,36,961,37,450,111,125,365,457,27,143,901,569,95,79,203,49,403,873,326,8,139,494,864,372,28,98,402,116,18,244,56,854,509,48,719,386,947,29,922,954,76,94,373,38,45,523,25,391,285,99,730,546,144,475,275,72,90,164,210,26,24,453,817,253 }, +{ 5,21,61,48,32,202,1,70,259,13,58,115,22,23,178,90,0,494,51,12,475,453,182,18,465,403,49,17,76,450,120,142,318,141,137,147,534,455,402,146,132,128,64,31,11,522,260,771,953,922,859,523,420,269,241,221,207,175,121,72,37,6,4,3,2,768,205,179,415,376,362,28,10,237,30,937,702,113,68,16,9,798,247,590,326,197,165,626,344,279,347,878,125,195,955,933,928,868,854,851,821,815,795,780,765,727,678,668,657,576,562,540,503,502,493,488,482,430,423,411,365,288,285,281,273,118,87,65 }, +{ 205,141,59,152,245,4,352,72,497,76,216,247,79,452,11,588,94,21,188,217,89,875,77,339,864,954,464,269,115,148,64,12,37,529,38,111,32,358,180,5,316,405,471,490,45,474,365,317,349,752,177,288,437,498,51,99,90,468,102,372,128,318,48,1,237,17,542,23,947,241,431,221,54,3,33,291,2,734,197,413,377,270,460,13,401,961,873,341,14,196,579,482,244,569,629,40,308,645,184,257,304,678,720,449,36,10,202,749,275,134,117,428,15,290,153,143,453,57,139,7,430,6,350,769,281,210,476,403 }, +{ 15,1,0,2,5,13,4,3,7,6,23,365,341,115,8,51,9,77,31,10,457,93,44,569,120,197,202,141,12,961,165,33,317,450,901,11,260,55,110,352,22,719,730,453,217,98,82,64,401,18,102,40,21,269,523,403,70,386,49,351,752,105,498,28,16,629,308,318,111,61,898,30,117,475,509,391,817,95,922,342,36,59,494,210,134,899,137,421,873,116,854,14,32,177,196,139,153,734,128,143,291,24,20,349,37,758,372,247,90,326,690,801,113,216,48,275,670,27,125,25,39,57,638,17,304,35,237,244 }, +{ 5,2,50,58,171,186,46,29,14,52,45,38,61,155,17,218,1,281,334,151,227,400,48,487,502,36,54,760,67,133,103,114,21,96,328,236,366,243,80,552,209,83,311,828,27,745,160,920,19,24,766,32,6,108,832,65,12,223,106,514,3,433,70,787,842,99,28,549,113,112,834,540,361,163,26,37,536,516,496,137,765,75,480,454,754,532,780,0,430,966,739,22,238,86,474,51,767,283,18,25,583,115,285,587,309,7,73,287,72,255,23,600,412,128,822,807,13,323,315,208,76,119,545,10,956,270,64,16 }, +{ 457,795,70,120,846,48,23,761,125,318,64,751,415,291,758,221,763,202,391,269,87,401,77,352,237,21,165,128,96,163,558,13,1,260,137,494,5,51,851,767,141,197,546,557,867,597,49,518,102,349,450,623,509,33,98,177,678,922,900,743,498,293,31,223,342,403,371,32,180,873,854,0,217,884,196,153,95,4,44,308,7,2,541,17,30,127,765,58,351,372,386,117,807,6,903,704,182,845,22,134,663,210,105,39,27,402,326,859,307,159,285,281,82,817,719,455,353,312,192,61,931,523,203,947,247,207,385,45 }, +{ 23,51,13,40,453,102,33,4,196,77,117,475,64,153,0,22,10,11,65,59,15,12,134,21,1,403,139,202,95,14,210,28,177,108,18,115,141,165,57,137,128,457,86,2,31,99,6,37,36,401,269,49,9,318,82,7,275,180,5,24,217,342,485,16,111,277,125,386,291,44,32,300,197,19,421,352,46,48,194,730,341,237,17,120,93,494,283,133,90,25,523,260,3,285,114,89,216,8,317,899,38,961,719,498,312,50,26,20,365,351,67,391,54,110,373,326,289,208,168,546,752,569,308,304,98,377,402,361 }, +{ 229,152,266,57,381,452,432,313,471,12,99,17,184,339,369,62,4,129,59,157,128,77,497,3,11,38,123,202,162,37,291,188,257,40,403,136,5,216,21,205,102,111,775,79,51,475,14,33,32,45,402,453,54,619,90,119,115,64,76,177,2,106,361,377,72,880,139,23,494,130,277,757,183,349,84,401,27,56,308,89,80,19,318,141,117,1,94,10,328,529,474,210,245,249,41,36,217,269,421,86,618,6,148,556,342,305,588,179,383,0,134,34,514,477,304,769,542,22,214,378,275,137,13,65,300,95,285,82 }, +{ 49,109,159,28,272,22,327,9,105,95,254,168,131,374,286,347,39,35,424,158,55,175,169,420,627,578,312,67,346,194,0,912,1,224,219,65,86,239,892,36,83,518,114,688,320,538,127,634,584,913,439,371,64,616,30,69,24,10,535,187,336,2,228,7,716,289,68,98,601,23,16,31,164,654,208,505,51,258,14,13,170,874,128,44,12,837,103,890,160,115,385,137,345,50,185,143,116,756,251,141,101,96,190,747,333,125,202,5,166,708,133,21,637,570,242,237,46,835,189,165,638,149,610,706,100,751,48,577 }, +{ 5,61,29,58,311,2,45,209,80,52,454,227,243,106,17,496,592,869,1,151,255,334,75,667,287,70,738,549,171,48,46,483,663,74,155,14,50,163,207,666,290,6,186,119,32,27,378,21,220,659,263,112,136,84,742,307,833,7,400,780,469,767,62,64,72,513,113,148,262,56,409,270,161,331,315,635,425,89,19,38,495,631,3,76,73,383,887,502,69,66,34,223,128,26,240,54,86,306,87,491,96,67,845,408,135,814,37,4,25,445,181,95,294,24,90,226,103,10,405,237,12,297,141,22,655,129,11,16 }, +{ 260,244,373,276,44,269,141,228,110,131,574,219,137,30,839,717,224,217,661,36,352,567,406,120,1,116,253,93,318,160,202,341,31,21,88,201,39,190,187,77,372,752,96,180,64,851,268,55,98,317,82,22,258,613,326,86,351,48,51,23,115,295,213,100,539,291,144,860,35,922,864,9,450,261,489,28,365,105,13,177,442,840,515,321,83,166,856,854,49,935,203,530,284,0,274,349,816,69,197,954,251,165,67,10,811,434,247,778,143,385,531,254,968,957,237,498,843,127,384,669,466,824,102,223,32,5,871,800 }, +{ 99,12,139,79,453,11,277,196,474,51,37,542,111,401,23,94,188,775,33,13,40,102,313,339,961,629,153,431,477,4,38,485,475,300,115,569,134,117,377,157,210,59,14,471,266,15,89,328,880,148,369,57,457,54,413,405,308,403,197,152,128,72,64,165,734,10,3,2,21,17,670,488,719,202,229,82,757,0,5,402,32,41,77,275,177,45,76,29,305,22,342,36,498,1,730,46,494,184,28,318,93,31,529,162,514,6,269,27,482,62,110,873,528,452,179,120,141,137,86,421,317,7,237,129,8,18,361,428 }, +{ 68,35,16,149,24,71,9,69,0,18,67,118,101,212,65,191,114,230,167,133,124,103,399,140,154,1,50,283,246,398,60,7,64,565,55,695,39,218,336,723,375,302,796,28,108,238,267,705,83,189,330,100,23,486,589,702,562,458,21,443,175,13,361,48,75,345,92,96,22,51,181,237,532,755,46,30,2,115,88,14,10,86,363,650,732,335,12,36,49,54,891,520,217,411,516,141,166,95,412,394,641,180,37,5,906,478,447,223,251,247,264,433,252,389,146,480,98,596,602,545,830,346,664,309,271,121,364,628 }, +{ 88,274,435,30,613,100,131,829,166,530,382,93,960,795,190,127,856,466,957,187,521,264,473,778,31,254,228,716,406,1,686,39,863,203,410,224,800,457,137,9,160,704,219,438,28,924,944,120,450,144,836,492,299,326,260,44,116,840,98,202,83,96,321,36,0,21,921,55,141,367,341,827,639,385,265,158,128,772,201,49,539,64,258,317,180,110,86,269,851,442,295,125,908,77,662,276,23,669,744,272,373,242,105,82,931,48,35,968,351,168,318,859,115,578,822,33,557,418,371,616,32,625,67,213,217,109,523,365 }, +{ 167,118,212,18,16,71,60,24,191,375,411,68,363,146,447,101,199,589,271,562,126,302,647,121,815,9,628,566,273,35,575,664,21,92,55,781,621,1,48,777,478,0,149,936,818,108,520,150,354,154,446,847,463,124,346,132,264,98,107,7,65,522,156,28,759,13,22,23,39,49,932,727,882,5,14,223,32,640,608,467,100,51,604,125,695,10,69,724,806,246,221,50,140,195,517,67,252,96,768,279,293,31,360,848,88,166,105,691,95,682,674,46,535,103,2,175,649,109,159,598,458,265,329,364,914,12,653,33 }, +{ 141,64,86,177,128,597,77,269,95,180,237,352,304,291,275,147,588,372,102,194,125,947,13,678,49,875,202,349,120,437,115,4,59,137,153,751,197,40,208,342,465,954,67,23,5,318,135,32,216,660,470,534,196,182,98,31,22,612,90,38,312,448,107,7,10,581,165,353,192,172,707,51,262,498,494,117,33,21,17,1,389,76,203,14,46,565,563,65,569,546,391,330,285,284,277,210,205,139,134,26,25,20,19,12,9,6,3,620,217,11,326,97,853,629,259,278,232,789,380,365,294,289,244,55,43,0,247,111 }, +{ 18,289,701,65,240,194,90,403,523,214,421,283,475,389,217,102,862,202,847,437,494,453,650,64,697,874,51,352,751,401,817,33,84,0,899,13,597,5,196,805,32,23,11,954,95,3,27,22,24,757,875,141,671,312,108,21,153,19,306,114,118,247,450,288,77,17,730,56,4,12,177,942,273,958,814,117,519,86,61,128,859,207,835,193,880,123,249,838,28,120,342,49,134,300,2,6,801,216,719,57,457,576,864,304,125,302,124,14,615,372,16,210,40,237,60,577,504,119,770,608,752,62,10,9,643,48,1,15 }, +{ 51,23,13,202,115,21,12,5,32,128,403,453,141,401,165,308,120,318,457,15,475,4,269,64,1,0,197,10,719,494,99,2,450,37,48,730,3,180,523,77,7,18,386,14,9,6,16,59,22,217,260,125,11,17,90,177,341,237,117,421,153,24,221,899,33,352,317,102,28,326,137,247,82,817,961,19,40,365,402,391,569,27,629,57,95,49,25,31,8,216,210,110,758,304,72,29,93,54,351,26,38,291,98,20,134,61,196,76,56,45,44,241,275,139,372,349,509,36,65,342,350,615,678,285,89,214,50,670 }, +{ 98,223,393,125,31,791,834,116,271,1,93,202,697,44,64,167,127,293,5,541,51,446,801,77,450,212,48,284,807,234,120,23,765,21,318,13,421,18,195,948,639,551,132,360,730,770,141,142,562,146,163,96,386,49,963,144,668,354,591,197,346,128,260,221,0,198,82,165,416,269,494,670,367,949,817,147,33,178,522,70,32,535,455,719,137,453,326,203,842,22,102,687,61,156,55,121,407,4,315,2,325,105,403,883,110,329,217,342,95,692,10,781,615,463,798,12,958,291,351,7,17,39,878,586,113,196,28,68 }, +{ 15,4,40,11,23,165,13,51,110,93,120,32,82,77,125,33,197,141,260,202,59,10,0,21,102,64,180,5,57,31,453,22,386,12,269,8,318,128,403,134,3,139,95,475,457,730,44,9,49,719,326,117,203,1,115,351,98,450,17,352,28,99,116,217,494,523,210,373,391,961,37,177,90,144,137,2,342,18,153,317,401,56,854,14,48,65,237,341,36,758,421,291,111,277,402,365,509,30,7,922,242,105,275,372,349,24,86,901,801,899,898,300,39,308,817,72,6,670,16,45,752,498,27,38,76,304,170,615 }, +{ 15,1,0,13,2,5,901,7,23,6,9,341,8,3,4,51,961,12,115,10,898,180,82,569,134,31,22,141,351,202,457,120,44,269,165,77,260,365,403,401,318,317,14,33,217,11,854,93,28,30,453,18,137,37,386,21,110,64,719,36,498,352,128,450,16,117,922,475,509,49,177,55,111,391,40,730,523,99,326,98,102,629,758,153,32,39,642,372,24,873,752,116,291,308,421,125,373,95,143,482,203,817,494,35,899,196,86,304,349,17,244,38,237,65,275,247,242,59,105,402,210,638,48,615,144,342,148,127 }, +{ 234,416,77,315,5,639,202,325,147,113,198,49,450,455,61,142,329,178,21,0,427,470,494,342,58,32,557,22,318,230,725,43,858,112,120,70,102,13,48,263,20,344,260,884,425,817,128,798,115,290,105,518,291,137,95,904,23,403,64,141,491,159,241,845,523,31,121,33,319,97,320,392,246,196,153,17,173,104,10,39,663,269,117,592,501,125,51,347,72,30,163,28,253,170,408,194,376,107,18,265,544,326,352,270,341,4,738,615,534,475,595,98,259,66,631,449,301,922,158,76,134,720,512,312,9,180,859,690 }, +{ 146,60,271,126,16,156,360,199,18,121,167,640,522,132,24,446,354,591,71,463,118,340,150,92,598,273,441,195,107,212,22,0,407,10,5,21,108,467,28,279,586,575,423,566,727,191,101,493,649,534,68,95,608,517,55,35,32,783,653,9,408,48,49,508,363,296,607,698,246,13,668,33,64,117,109,356,98,314,39,658,140,105,7,362,660,447,809,411,955,93,31,621,50,303,298,153,347,12,158,125,65,953,713,264,163,350,359,141,124,14,143,51,939,478,149,115,36,75,4,781,581,225,207,102,77,30,562,44 }, +{ 4,11,361,300,77,56,377,368,14,59,40,161,51,128,33,200,210,23,378,27,54,495,102,554,38,494,453,17,21,12,202,80,444,108,53,757,37,46,78,13,245,457,342,390,139,133,123,386,84,523,32,605,196,862,527,736,803,99,73,57,19,630,561,475,3,542,509,421,317,250,181,136,111,277,269,450,197,472,183,48,26,485,730,297,62,714,402,381,403,391,880,122,719,341,318,306,145,25,5,514,256,61,899,546,106,90,50,34,383,833,236,775,497,165,129,85,401,366,184,469,436,838,358,328,316,291,275,241 }, +{ 141,351,217,82,352,120,854,260,64,15,180,372,1,922,269,317,36,137,752,373,13,21,349,51,898,450,77,202,110,23,318,160,93,115,291,128,365,48,197,177,901,96,5,165,244,864,10,86,457,4,40,237,523,247,33,579,341,32,304,11,661,125,83,143,498,326,817,2,203,275,437,69,102,954,717,16,59,961,35,386,851,730,22,475,28,253,31,7,57,98,95,134,144,391,12,6,719,117,210,116,17,196,139,0,3,37,494,473,403,114,49,30,29,164,899,704,559,153,90,72,55,9,45,509,482,453,393,230 }, +{ 15,77,13,1,0,102,23,33,2,4,51,5,10,291,7,9,3,450,6,217,117,120,177,317,341,260,11,12,817,342,365,115,8,134,21,153,180,16,18,22,349,40,165,82,14,64,141,453,372,31,457,523,901,196,197,326,275,352,210,961,719,730,28,304,475,403,569,44,128,32,899,93,59,269,401,24,17,49,752,98,318,110,36,629,216,30,386,125,95,494,90,137,48,351,308,116,111,57,139,37,86,99,498,65,615,247,482,144,39,237,221,20,96,19,244,421,300,758,203,670,194,509,38,854,402,873,437,391 }, +{ 15,23,13,77,141,260,137,120,51,202,180,115,1,365,217,0,341,450,269,291,317,318,5,197,128,21,9,177,373,351,110,352,165,2,12,523,7,64,349,326,10,48,36,901,32,33,22,403,854,82,90,96,4,922,16,37,28,18,752,457,40,386,244,372,102,494,6,308,8,160,86,24,30,14,401,49,661,817,453,719,421,402,11,704,961,475,237,873,342,55,3,39,730,391,59,196,93,221,117,143,216,615,898,44,498,275,253,210,304,35,285,95,125,31,98,17,139,509,72,153,811,482,89,83,144,899,99,576 }, +{ 348,174,85,138,6,280,74,355,233,192,204,636,580,353,343,289,673,66,65,91,81,104,712,750,25,42,214,792,95,64,681,173,194,32,715,135,90,312,29,879,172,825,63,235,835,86,22,87,145,370,5,256,7,222,415,52,0,874,426,97,500,519,17,26,237,43,4,288,36,18,10,389,20,33,560,76,21,61,40,544,45,59,249,888,77,577,746,232,28,102,2,72,49,568,123,287,56,525,448,751,180,11,337,526,707,671,226,27,19,247,24,319,117,48,422,114,12,69,108,84,208,597,37,1,861,67,889,58 }, +{ 152,452,497,59,216,4,269,188,77,11,79,339,128,529,94,471,318,141,202,588,32,76,377,177,678,33,111,291,349,90,5,102,21,137,51,72,12,179,542,38,99,64,257,184,37,1,245,217,460,196,48,372,115,947,148,45,402,474,40,89,342,210,17,352,205,288,3,153,769,13,117,10,2,134,292,54,275,304,139,875,498,431,23,405,241,403,247,317,569,229,401,453,494,285,358,180,197,86,300,57,0,476,961,6,954,308,36,143,524,165,277,475,316,597,14,95,237,15,7,221,341,58,157,735,129,266,214,421 }, +{ 417,253,244,499,141,10,559,564,110,8,260,352,728,143,180,951,120,642,638,661,752,922,341,373,811,137,269,901,717,197,93,206,498,44,165,31,116,203,160,854,384,351,36,873,762,326,704,268,217,1,64,115,282,317,959,55,144,15,365,82,898,968,509,164,86,479,318,13,851,22,21,332,473,434,51,202,77,940,911,569,23,125,450,864,824,237,261,9,242,615,372,617,127,48,201,102,128,4,33,5,96,506,0,391,117,98,28,401,386,40,295,105,32,758,531,276,338,37,83,35,349,59,533,153,177,629,11,12 }, +{ 77,13,23,51,177,102,33,115,59,4,40,0,141,128,90,210,202,137,64,22,269,291,217,10,196,28,153,117,318,7,49,2,14,216,134,18,180,16,6,11,9,24,403,1,275,197,3,95,36,86,125,165,247,342,31,352,494,19,32,402,12,139,453,317,20,237,98,65,15,349,44,475,26,285,17,30,120,111,57,93,21,372,105,27,25,719,194,457,341,203,304,386,260,391,730,116,421,365,35,29,5,99,326,123,498,208,39,569,509,244,288,752,450,143,678,144,110,523,401,308,82,214,300,351,43,37,48,164 }, +{ 1,2,22,50,67,14,36,5,46,28,103,12,0,38,83,114,65,168,194,7,69,133,86,95,24,49,218,29,54,21,55,96,109,160,51,10,584,16,128,6,9,23,48,158,18,75,756,37,35,105,169,32,389,289,159,323,238,181,52,244,202,13,503,646,164,137,532,141,239,115,361,480,185,64,17,143,789,283,124,708,253,505,946,597,3,717,45,776,58,170,108,269,70,735,312,874,285,610,68,782,189,318,424,71,531,31,308,237,320,171,4,892,72,402,99,328,220,418,251,420,394,19,180,577,487,401,347,344 }, +{ 17,106,119,378,742,255,306,136,207,240,84,383,62,790,80,655,61,45,5,56,311,3,209,504,440,667,76,151,32,58,263,483,179,29,2,833,292,548,214,123,129,425,186,454,128,243,205,738,445,64,21,245,148,52,38,14,26,249,57,27,328,19,90,6,4,555,11,34,59,54,162,202,184,507,469,141,592,229,48,215,887,710,152,73,86,40,161,72,618,460,266,257,491,77,358,211,227,287,25,869,183,318,297,50,290,495,10,216,130,614,155,1,476,177,137,361,12,315,37,112,33,22,46,95,113,269,194,171 }, +{ 141,128,202,3,5,2,1,269,64,318,6,33,17,102,15,180,0,196,137,10,153,7,120,352,197,13,77,117,51,134,403,14,165,12,23,402,11,93,9,342,260,82,110,21,494,19,22,237,31,351,4,125,86,475,457,450,326,27,44,32,203,40,36,95,291,961,453,115,25,285,143,111,678,98,341,37,139,99,523,817,49,57,752,498,45,386,18,38,217,8,116,65,16,244,28,144,317,365,56,719,373,901,194,854,79,898,277,59,29,576,26,421,30,569,401,730,947,177,164,864,48,72,391,922,24,873,94,349 }, +{ 38,514,328,377,11,57,266,248,41,880,313,471,556,152,361,4,485,99,305,757,403,157,12,102,339,54,369,40,211,130,542,457,3,37,188,14,139,23,13,153,494,117,33,475,775,229,507,381,300,196,77,453,452,59,555,162,488,184,277,585,236,477,440,629,62,128,51,111,421,134,402,183,215,961,670,79,548,123,342,210,202,89,401,527,474,94,65,587,719,543,432,181,45,17,257,291,80,64,72,21,619,108,32,413,828,46,5,29,0,95,2,275,129,308,10,56,504,205,86,618,155,76,177,49,431,569,90,497 }, +{ 68,520,167,101,264,118,478,562,0,9,664,777,936,604,212,100,191,252,154,18,124,806,265,16,759,21,48,1,71,375,88,724,628,24,329,35,682,411,410,363,166,302,815,447,65,419,589,55,23,39,647,149,917,909,821,382,125,942,98,346,925,772,13,701,146,96,223,51,108,31,10,221,271,22,60,28,321,44,92,905,69,30,676,639,49,593,364,882,274,694,521,907,64,105,5,158,67,621,781,812,126,133,553,189,848,897,213,686,95,32,121,404,522,199,103,116,156,50,140,83,429,706,763,273,115,165,127,77 }, +{ 230,689,699,213,466,831,352,217,30,443,418,854,144,201,840,855,251,203,957,822,96,530,1,539,93,36,137,800,317,83,190,128,326,669,752,351,748,494,88,206,160,31,473,859,335,202,141,44,321,253,735,498,269,435,116,367,187,86,35,927,332,180,237,69,98,778,244,285,219,82,131,576,228,617,208,922,866,930,372,100,120,64,2,934,403,349,531,702,48,21,731,817,274,23,5,10,13,406,365,224,947,824,795,345,898,291,382,295,223,692,318,286,238,39,341,276,863,77,615,166,260,625,678,51,28,67,410,125 }, +{ 77,33,102,117,134,153,196,291,13,51,15,23,82,64,40,202,128,120,0,141,260,351,450,22,4,95,59,137,269,342,177,386,318,326,11,115,10,210,65,373,86,180,217,49,403,1,165,275,28,18,453,57,36,125,98,352,494,523,139,197,14,31,854,475,2,194,216,9,7,817,21,719,93,6,730,24,457,237,90,12,116,16,44,349,110,3,5,402,99,961,317,114,372,19,48,32,285,105,96,17,365,341,300,143,8,203,401,247,164,108,421,25,752,144,26,30,20,391,39,111,304,109,758,50,221,208,37,289 }, +{ 11,40,33,51,542,328,13,117,38,14,23,139,153,134,54,102,111,12,485,377,775,37,99,403,231,277,401,719,369,3,475,453,514,4,629,308,211,880,457,291,305,89,266,152,130,2,57,196,961,6,488,215,5,176,59,555,471,313,72,300,477,183,62,184,79,0,507,248,115,21,670,339,249,548,361,1,165,292,29,197,229,77,556,474,236,386,421,157,413,188,452,17,758,41,162,391,449,587,32,46,123,428,509,52,247,569,136,899,757,523,210,753,155,129,76,50,237,64,730,720,476,383,312,440,275,90,45,342 }, +{ 77,64,141,202,33,102,128,153,2,117,269,6,23,318,1,13,0,134,137,51,5,180,3,196,10,7,40,165,15,14,352,197,86,291,237,115,403,95,17,177,22,217,36,139,210,9,120,19,125,342,494,275,82,475,457,49,453,16,12,31,4,65,21,402,11,143,260,194,110,93,450,25,28,351,317,111,498,326,27,18,349,752,44,341,59,164,386,26,285,116,365,98,719,208,32,391,300,509,401,523,99,20,24,38,114,244,373,730,55,48,901,372,203,312,961,8,277,160,678,304,34,37,57,854,56,546,29,144 }, +{ 15,4,93,5,13,21,11,1,23,32,110,141,64,82,180,2,33,40,3,98,352,120,6,10,0,12,77,7,341,197,37,51,102,351,269,854,95,17,202,31,9,752,901,44,165,117,48,318,153,291,134,137,115,365,237,196,372,59,498,260,349,450,14,128,90,36,317,244,899,22,342,922,437,125,143,873,453,16,160,72,105,482,179,817,19,89,401,457,57,661,403,523,177,164,45,308,864,373,253,245,8,898,49,421,569,223,76,719,28,61,18,247,139,56,96,27,490,210,468,111,494,475,144,38,961,391,304,25 }, +{ 1,2,50,46,14,67,103,38,24,83,133,29,52,96,75,28,114,5,36,238,65,54,108,171,70,0,58,18,69,181,160,532,236,7,22,412,433,754,283,55,163,220,480,361,487,124,12,223,545,45,328,394,21,398,16,155,9,366,48,35,956,6,154,87,323,646,309,514,112,186,822,137,17,109,149,418,745,175,547,86,37,285,68,800,866,10,429,732,516,25,208,302,51,32,118,334,281,99,531,71,61,194,537,458,23,891,389,19,151,675,409,336,128,168,13,400,307,95,262,832,227,49,273,420,3,115,484,64 }, +{ 258,276,860,201,843,295,137,160,567,116,261,144,268,943,284,202,187,935,662,203,699,131,219,224,44,141,127,36,326,96,98,31,318,269,244,964,23,93,384,489,55,21,434,332,13,403,51,120,1,617,918,352,908,367,110,206,180,365,254,197,492,473,871,442,48,82,10,143,800,260,39,574,28,944,299,940,282,253,9,115,494,418,341,125,855,77,8,49,128,836,535,165,100,576,30,523,385,105,822,217,395,959,564,22,531,911,37,83,393,851,717,64,817,752,421,716,5,317,901,551,373,859,0,170,291,599,533,251 }, +{ 7,2,14,87,46,52,278,92,16,140,75,29,135,294,262,70,38,35,172,171,409,24,60,58,69,5,97,112,66,314,1,163,571,71,50,461,155,267,232,330,18,220,54,334,357,126,380,307,21,74,12,230,48,309,186,675,32,37,298,331,226,246,151,45,774,107,64,337,25,324,6,67,606,189,622,20,448,150,43,61,0,400,22,849,287,10,68,9,86,563,28,95,227,583,192,394,620,103,36,764,572,108,128,335,83,55,587,550,13,145,600,549,536,65,141,852,366,49,415,90,26,236,526,950,17,4,181,72 }, +{ 13,23,51,115,2,33,77,0,1,10,4,7,217,5,9,141,102,6,3,180,11,40,317,22,15,64,14,197,352,202,16,12,341,269,365,165,18,318,177,17,59,153,453,196,752,291,32,117,308,237,125,403,31,128,8,36,401,24,28,349,372,49,498,137,475,247,95,304,143,569,901,21,98,468,457,134,44,139,19,90,421,86,210,120,386,629,93,37,116,482,494,57,110,873,342,55,961,275,111,391,546,35,509,244,203,30,27,99,29,437,48,20,260,65,719,216,734,25,38,54,450,326,50,351,46,864,105,253 }, +{ 74,145,204,66,138,42,25,6,337,174,525,222,192,29,26,544,746,287,415,632,85,609,135,7,353,280,112,580,87,396,712,355,64,122,81,104,2,45,173,673,416,648,172,43,792,226,52,22,97,651,151,194,70,65,0,19,32,20,715,95,825,348,5,624,233,113,86,635,91,879,294,10,644,235,1,90,500,36,573,560,49,636,243,58,63,21,331,426,61,75,389,27,445,14,69,750,28,289,46,38,17,888,311,312,18,24,140,186,76,283,141,12,874,16,232,278,72,461,707,53,4,526,597,255,665,180,114,881 }, +{ 77,217,33,102,317,291,202,23,13,51,141,15,275,269,318,115,342,180,21,4,196,352,64,128,11,494,32,5,137,403,40,2,0,125,3,177,10,98,365,349,12,7,372,6,341,453,165,197,153,475,48,752,22,237,31,117,9,95,59,93,16,90,120,247,308,450,49,221,244,36,28,18,144,86,14,134,326,421,203,44,17,304,24,116,523,260,110,864,210,498,401,901,1,35,241,386,82,127,817,39,72,437,160,105,65,143,730,719,351,216,457,402,30,350,293,164,37,859,96,367,139,284,19,579,576,391,285,373 }, +{ 187,258,871,219,442,574,908,839,926,160,116,44,137,224,276,201,36,860,131,202,613,228,144,531,406,93,31,902,567,385,843,318,203,268,800,98,662,856,494,23,190,120,127,418,30,295,533,836,1,284,96,822,244,64,944,717,86,851,55,13,21,51,22,840,434,269,251,367,489,326,48,110,141,699,816,260,39,384,261,371,473,83,206,254,859,10,669,492,28,317,373,128,49,365,450,77,180,863,559,530,82,829,341,918,253,32,9,197,125,351,332,165,143,943,728,535,67,831,576,855,940,403,217,716,105,274,69,959 }, +{ 1,372,23,5,141,21,744,13,291,77,225,349,144,48,459,96,839,260,304,269,442,335,691,352,32,30,622,203,303,284,330,64,45,521,180,317,217,160,88,752,418,120,93,190,948,166,954,340,919,686,342,244,228,202,201,189,137,110,321,914,411,447,781,772,254,51,765,31,656,264,237,167,724,478,842,437,274,127,125,69,44,662,466,435,351,326,82,9,897,410,253,716,286,678,479,299,238,197,68,7,2,934,223,100,70,58,154,98,682,393,967,960,922,856,805,762,475,402,365,341,336,285,247,221,177,169,163,131 }, +{ 66,222,74,135,87,29,145,2,337,294,52,172,138,331,6,7,25,226,42,97,632,70,192,112,353,609,525,415,461,204,287,500,26,174,75,43,324,1,58,32,220,5,232,544,580,85,620,635,278,45,746,64,426,568,448,396,262,122,173,151,69,20,95,19,61,526,140,65,889,86,307,14,104,280,21,46,81,355,92,38,872,22,387,651,186,648,113,194,416,72,163,673,90,24,35,155,49,16,330,37,792,0,888,243,17,36,76,861,10,48,849,67,644,227,50,4,89,853,334,63,27,314,312,12,825,712,91,298 }, +{ 45,17,106,209,255,119,186,5,263,207,454,29,2,48,425,243,21,496,62,290,3,270,400,408,52,155,136,659,171,14,56,513,50,80,84,32,378,46,61,655,666,742,221,58,383,38,1,54,311,306,240,721,181,440,502,430,334,129,72,536,227,123,76,790,592,214,89,328,281,555,34,787,241,148,64,215,663,287,587,37,11,405,162,27,237,504,57,90,130,152,541,179,12,211,548,41,36,552,184,4,741,115,431,292,449,869,229,96,77,507,223,128,183,40,7,86,51,249,205,245,161,6,738,236,103,151,59,10 }, +{ 213,88,260,120,93,373,141,30,100,450,166,36,372,321,459,473,264,180,110,326,1,225,144,704,717,351,82,855,530,269,466,352,217,265,190,22,274,131,96,539,165,86,457,291,28,244,64,187,318,752,39,349,661,330,9,160,202,219,77,410,197,44,908,254,116,49,137,968,276,317,125,406,31,521,523,435,15,98,228,128,32,299,35,662,293,143,115,21,934,778,224,33,856,923,876,854,689,403,371,341,10,935,851,498,437,8,922,48,642,776,903,839,836,730,613,365,268,242,55,0,944,899,492,898,615,385,304,285 }, +{ 266,57,152,381,471,313,369,99,229,12,339,62,157,77,4,37,188,40,475,162,38,3,452,775,129,184,403,17,59,128,11,432,202,111,453,33,56,183,291,401,216,361,328,102,136,402,79,54,440,14,139,880,123,494,51,23,514,277,507,130,757,474,377,21,94,529,177,548,421,349,64,76,211,90,117,89,72,308,13,318,383,5,141,210,134,556,342,153,305,497,257,555,119,485,84,269,275,106,670,2,115,477,488,45,196,41,300,205,32,542,10,543,304,431,86,180,504,19,27,82,179,36,629,249,46,50,237,245 }, +{ 352,217,317,141,752,372,365,180,341,15,349,77,864,21,291,244,115,13,120,269,1,498,5,901,579,98,144,64,23,48,482,351,51,237,102,201,82,437,125,137,318,4,2,247,202,0,31,854,203,143,253,260,32,177,160,490,10,96,110,44,22,494,251,33,128,403,9,3,898,717,164,28,197,72,93,11,873,450,36,165,386,95,373,6,223,661,30,12,49,922,342,206,523,16,457,127,109,468,308,391,116,531,332,473,187,39,88,35,730,453,326,221,7,719,69,86,105,100,83,851,475,90,350,230,678,421,101,304 }, +{ 14,514,369,880,102,403,377,51,719,485,153,23,453,401,457,11,40,13,328,12,57,4,33,38,629,236,670,54,730,117,111,361,961,59,37,134,99,758,757,165,2,46,386,412,421,391,277,29,308,543,477,494,188,600,6,196,21,542,197,139,300,739,475,52,899,79,509,523,488,775,583,283,231,7,803,552,32,5,402,89,828,3,507,176,77,50,15,155,211,64,17,569,215,115,18,381,237,305,266,202,90,34,342,194,128,62,555,389,242,125,108,65,0,471,338,136,56,548,506,221,181,133,10,440,114,248,95,1 }, +{ 107,172,7,359,126,92,14,135,314,278,60,150,298,46,392,279,232,192,2,441,356,330,357,16,32,380,97,534,423,581,35,526,38,448,493,132,52,288,246,660,5,121,66,24,230,324,572,21,387,226,462,259,247,95,220,146,140,77,18,508,147,262,90,48,312,707,20,71,603,54,33,195,550,362,86,236,108,739,199,830,23,64,43,29,895,802,451,22,36,28,929,294,606,50,87,70,501,13,808,177,55,1,194,178,711,316,4,0,102,590,366,49,893,600,37,582,309,872,267,117,388,241,620,115,786,939,571,360 }, +{ 59,4,90,45,216,13,25,0,177,23,77,51,27,95,72,26,115,65,12,91,210,123,275,37,389,19,102,33,11,49,22,18,194,5,214,21,85,53,48,202,343,291,31,57,6,128,42,746,93,73,58,44,32,196,283,81,319,61,141,249,153,64,138,117,422,355,40,348,139,403,789,145,788,300,247,240,134,74,34,804,619,370,104,86,38,1,304,180,524,494,217,20,597,544,318,352,323,233,174,114,76,17,475,280,204,588,342,285,14,3,43,725,287,7,888,636,235,52,29,63,337,232,122,784,125,397,817,678 }, +{ 5,315,416,49,455,639,202,350,77,408,21,197,137,450,402,13,61,509,329,113,713,344,569,347,401,48,0,841,494,318,734,749,64,70,308,95,903,120,342,884,237,557,32,194,546,115,845,1,23,165,663,128,873,102,291,518,22,58,817,391,498,263,17,33,221,269,403,923,260,105,241,125,51,196,312,39,10,31,153,234,112,523,158,18,4,265,12,141,9,159,928,623,326,207,25,177,310,3,2,859,98,28,65,30,163,11,19,275,491,290,6,117,541,16,15,124,76,72,27,922,210,44,341,349,134,773,147,24 }, +{ 130,47,381,390,59,90,200,472,214,714,65,289,6,122,874,85,64,29,751,648,52,50,74,875,247,194,624,249,288,26,312,437,32,217,630,881,4,123,95,78,835,352,196,665,519,33,57,711,524,954,108,42,19,77,22,153,597,416,412,0,145,444,27,243,25,14,183,304,117,453,51,256,141,730,102,177,134,236,315,680,86,112,34,11,115,235,287,237,113,45,283,151,317,588,579,99,3,752,216,342,813,128,445,864,180,56,46,28,18,73,12,23,114,13,389,53,644,719,21,396,58,17,372,947,137,316,10,250 }, +{ 51,453,23,719,13,730,165,457,37,197,99,12,386,401,475,961,4,17,11,64,59,308,54,115,57,38,328,758,391,6,5,2,899,3,753,403,32,523,670,10,509,40,33,421,629,120,21,65,19,22,0,202,125,95,569,117,90,558,29,26,514,494,1,775,128,134,14,450,361,34,9,86,194,177,350,269,72,27,25,7,880,757,111,62,153,316,236,141,50,56,260,217,196,867,546,161,130,106,77,48,41,31,16,8,317,152,468,242,221,136,734,80,89,968,949,862,814,801,717,704,495,469,440,428,412,383,378,326 }, +{ 15,13,1,5,23,2,7,0,6,3,12,51,4,180,115,21,9,165,457,197,141,352,11,14,202,217,10,37,8,32,341,120,22,117,752,269,16,77,134,961,901,719,318,48,317,365,33,110,93,31,569,18,102,28,40,260,17,509,128,44,153,137,82,498,59,629,351,72,386,403,36,49,401,453,730,308,196,450,89,873,30,523,125,99,391,326,177,98,64,247,494,854,24,291,95,475,38,86,482,39,203,922,372,349,116,90,144,57,35,758,55,864,221,468,244,210,421,139,817,76,373,899,54,105,19,304,342,96 }, +{ 0,9,35,28,68,24,67,65,1,16,18,114,69,103,50,101,149,22,175,124,55,39,12,218,133,96,154,189,5,83,21,71,23,7,336,433,251,13,49,75,118,335,2,51,48,486,30,14,10,283,181,745,115,223,137,54,230,109,418,37,345,64,128,141,108,281,99,36,60,100,202,956,537,163,532,480,221,766,302,596,487,95,252,346,58,158,46,180,217,361,237,70,398,86,269,160,31,32,105,657,443,140,285,125,87,212,166,167,502,6,88,375,165,323,831,317,754,29,318,439,77,98,364,38,197,429,44,765 }, +{ 28,105,22,320,109,49,301,0,9,170,95,1,35,55,127,65,24,168,98,159,713,16,512,31,395,36,623,2,7,293,740,14,338,347,424,185,286,12,44,23,21,169,124,5,158,68,67,928,13,371,18,299,242,125,51,289,175,916,64,761,479,374,39,737,685,154,48,272,6,114,599,116,10,86,141,37,194,83,149,506,327,312,841,202,763,237,50,900,870,538,284,115,131,420,346,137,654,385,456,165,187,762,30,276,239,32,308,69,143,269,429,96,570,71,128,180,933,160,164,46,197,221,401,634,965,219,101,318 }, +{ 77,13,51,102,4,59,23,33,202,210,115,40,403,177,128,196,141,153,117,64,494,269,318,134,475,217,342,0,291,11,90,453,22,402,137,15,216,139,10,180,275,86,95,65,57,6,36,18,523,1,125,21,317,365,197,49,28,421,32,14,352,25,19,7,2,899,730,719,391,341,152,24,401,48,27,17,12,901,247,123,114,16,9,111,99,26,349,285,576,50,20,3,386,372,326,260,194,120,110,351,283,482,293,242,165,96,29,37,30,817,801,498,457,450,393,373,299,237,214,203,615,569,116,93,34,82,958,805 }, +{ 13,23,77,51,33,141,115,102,0,217,10,2,6,4,7,64,22,1,11,177,180,40,3,14,202,269,9,291,352,15,165,197,17,128,317,153,196,5,318,18,372,117,16,28,59,36,19,137,95,210,237,247,125,49,86,349,341,304,134,365,24,275,12,31,27,453,403,752,25,143,308,26,57,20,901,120,475,44,401,457,29,498,90,139,65,8,391,93,342,494,216,509,164,98,99,35,55,386,244,111,32,45,37,21,260,38,110,56,450,105,39,730,402,523,46,437,82,30,719,569,203,961,873,326,285,194,116,421 }, +{ 456,116,492,268,949,867,203,51,719,8,391,918,791,457,730,499,13,386,23,206,685,125,31,551,170,870,93,417,401,10,22,479,506,338,940,44,509,143,64,692,670,558,453,0,473,197,523,758,629,341,165,1,564,55,899,762,308,961,421,959,36,638,115,569,801,180,642,86,299,164,28,37,141,965,498,559,9,5,120,237,95,546,260,873,599,137,253,12,244,903,901,2,817,752,728,128,352,450,105,202,7,615,127,365,958,21,301,49,951,403,740,326,239,898,373,922,475,6,854,242,15,3,318,4,269,98,65,99 }, +{ 17,237,106,62,180,45,136,32,498,64,115,41,255,129,546,752,864,197,21,227,241,490,352,482,57,123,292,350,247,509,449,162,365,468,214,38,23,13,4,476,148,76,437,155,507,141,165,77,3,720,317,290,270,873,211,164,72,153,209,117,5,659,54,48,249,217,341,186,710,33,40,14,12,496,119,11,519,90,183,618,139,179,37,328,579,221,431,248,372,405,130,59,99,400,263,205,425,2,358,349,901,51,177,171,89,428,50,454,304,207,542,1,457,308,734,721,391,316,29,569,94,243,143,339,196,313,128,111 }, +{ 1,5,180,15,2,4,0,352,3,13,901,341,7,752,6,141,217,23,365,317,115,197,21,51,165,269,12,11,9,202,31,33,77,569,44,93,318,137,110,120,36,10,22,372,260,498,116,64,32,873,125,48,102,308,8,349,98,482,291,386,401,14,373,326,28,237,37,55,86,134,203,457,391,16,117,177,253,144,509,421,127,153,40,206,403,961,864,468,221,494,437,453,18,17,196,304,143,128,96,854,450,72,247,35,579,89,642,758,82,342,160,629,719,282,811,59,351,922,730,95,638,523,210,546,332,661,49,734 }, +{ 141,205,497,216,269,152,588,59,452,128,4,76,352,875,188,77,79,11,94,678,947,339,954,217,471,372,90,529,72,318,247,64,32,89,177,12,288,37,1,304,349,202,5,51,291,33,474,102,111,23,115,21,99,179,864,38,148,377,13,460,431,498,437,316,180,2,961,40,358,542,137,490,405,241,464,317,3,45,54,7,184,196,17,10,402,752,245,342,210,14,873,48,143,285,153,257,197,308,292,0,36,569,769,237,134,482,139,275,165,468,341,629,413,734,86,476,365,6,300,851,117,751,494,401,270,229,453,29 }, +{ 247,64,317,217,237,437,180,498,349,115,752,579,304,372,864,482,291,141,352,468,197,177,13,391,164,490,23,546,33,365,77,153,72,216,954,597,277,10,708,269,134,569,275,51,695,509,318,179,0,678,244,140,312,640,117,76,629,196,751,128,36,953,674,260,102,901,32,21,22,225,401,86,717,143,92,873,351,16,133,114,191,386,253,165,69,59,9,458,341,202,149,82,649,35,453,398,166,100,95,93,90,125,15,373,330,923,316,48,854,586,213,486,230,922,5,411,823,734,189,139,103,65,50,24,18,12,830,167 }, +{ 2,1,5,7,6,3,77,0,141,13,180,15,217,64,33,23,102,10,352,14,117,9,11,197,165,40,269,17,291,4,115,134,12,153,341,317,752,372,237,51,349,22,498,202,318,16,457,128,196,36,177,365,32,21,59,19,125,901,342,8,86,120,139,93,45,961,57,95,38,143,509,28,29,391,111,210,27,260,873,244,82,137,18,386,164,37,719,25,48,31,304,49,864,98,402,275,569,110,351,629,253,546,26,41,90,494,50,730,56,523,453,44,54,144,24,99,30,403,678,76,898,450,437,55,954,482,308,277 }, +{ 11,40,38,328,542,33,339,313,12,377,157,775,485,99,51,471,41,369,23,37,475,57,54,457,14,117,13,188,305,629,102,403,79,4,139,514,880,961,111,266,719,152,248,474,134,453,401,153,211,342,477,21,2,59,361,89,587,277,94,29,308,196,64,32,413,77,229,210,184,215,402,128,431,381,46,670,5,247,197,165,556,757,386,115,300,50,17,391,803,494,875,490,730,177,162,450,421,275,202,0,6,1,257,76,72,65,48,19,10,129,569,236,130,15,7,555,901,898,588,86,9,543,488,90,452,749,619,507 }, +{ 77,102,33,196,40,51,202,23,153,128,117,13,342,64,134,141,4,59,0,269,210,177,22,318,137,403,65,86,10,475,139,453,95,115,28,99,11,165,217,36,180,197,49,402,291,125,1,352,18,15,494,421,194,275,31,300,110,120,57,90,93,12,237,401,317,6,457,14,2,82,24,114,260,9,386,44,285,108,361,98,16,7,21,349,326,216,5,116,730,341,372,19,133,719,203,48,32,391,39,25,143,111,105,67,576,752,670,961,899,308,498,37,351,304,20,30,283,901,523,365,27,244,208,55,3,164,485,8 }, +{ 472,34,80,495,250,469,176,389,527,17,14,161,53,231,4,128,56,283,216,106,84,57,27,483,177,59,119,503,436,45,585,297,194,29,378,3,42,62,269,136,26,40,11,2,862,678,619,33,73,236,255,514,719,200,25,6,64,240,179,141,52,102,789,361,618,129,597,5,153,611,349,205,65,245,803,402,210,457,123,383,90,304,667,318,814,504,328,133,130,51,32,48,684,108,714,412,215,13,139,19,95,77,47,275,99,12,311,211,165,1,432,207,322,196,117,61,23,524,152,115,369,183,134,114,573,54,162,46 }, +{ 752,352,317,180,141,217,365,269,244,341,498,115,253,901,1,372,202,21,318,4,5,23,700,717,13,137,2,349,197,864,10,873,143,7,482,0,51,3,64,11,67,308,531,12,8,6,28,237,22,31,468,579,533,291,160,221,35,30,96,48,55,33,9,961,120,49,44,39,32,247,83,69,450,326,260,77,36,98,851,509,165,37,16,457,437,110,93,88,82,14,345,822,765,638,421,219,190,131,128,114,203,144,99,800,401,844,453,125,494,475,373,351,241,206,170,164,148,134,127,116,105,100,86,40,38,18,811,19 }, +{ 9,39,101,265,18,333,520,329,100,593,553,364,310,252,167,16,68,604,0,363,404,118,166,121,254,158,272,24,60,271,212,286,639,327,159,35,146,1,109,30,455,411,793,105,557,49,132,647,31,71,621,21,656,354,438,627,676,688,98,48,682,28,156,538,190,22,5,55,23,195,65,44,848,264,51,446,857,709,127,374,781,447,191,344,865,625,821,13,124,96,93,7,578,125,419,907,360,463,32,116,126,37,274,777,88,731,108,67,154,92,95,36,199,12,137,131,10,562,128,326,664,203,382,50,806,242,921,435 }, +{ 15,1,901,180,2,4,5,3,0,752,7,6,341,13,115,317,365,23,217,141,9,197,352,165,51,21,137,11,36,202,12,33,22,873,308,77,269,10,64,86,498,569,31,372,55,93,40,120,160,44,509,318,349,482,638,8,117,37,116,164,102,32,28,401,110,128,196,144,206,244,153,457,468,951,17,291,16,203,96,260,811,143,373,642,125,134,437,386,546,268,247,177,14,48,758,717,719,391,579,72,421,89,661,629,111,730,559,961,221,59,403,523,304,851,35,30,169,854,494,127,82,326,450,251,898,417,475,734 }, +{ 131,39,829,166,613,827,716,578,9,100,224,310,406,30,98,616,228,1,254,856,219,438,938,190,31,846,265,931,0,127,96,625,274,924,44,371,101,385,116,252,272,863,51,137,88,21,333,731,557,36,23,404,48,223,125,49,264,160,187,737,242,105,28,258,144,93,13,530,158,944,382,284,395,203,812,593,518,442,5,456,836,128,326,293,68,329,403,35,202,816,77,115,83,909,688,86,890,478,120,64,22,639,16,165,276,393,455,871,410,201,286,299,33,576,960,656,24,217,95,159,110,32,367,421,492,221,55,102 }, +{ 17,64,62,141,106,136,751,292,129,352,41,38,954,86,476,214,217,128,177,32,237,947,180,162,123,209,710,269,519,247,90,864,157,77,318,99,57,76,59,4,437,40,179,358,11,45,618,119,313,5,304,211,597,148,875,23,249,197,130,95,507,37,54,3,216,457,248,58,210,115,288,12,425,678,186,164,328,72,454,339,245,102,80,498,312,165,372,139,155,2,152,56,33,21,655,243,720,183,207,14,29,241,117,266,290,471,137,84,51,188,305,275,205,449,7,263,48,542,13,270,742,171,752,509,61,317,349,369 }, +{ 42,25,235,650,736,65,605,630,233,343,123,256,122,85,26,416,6,389,63,141,249,368,444,194,511,174,544,19,108,597,348,524,138,90,27,183,32,370,304,525,84,56,53,283,789,445,18,78,280,91,130,947,588,619,45,355,17,680,73,4,0,222,112,3,813,119,145,64,678,297,250,80,21,573,523,86,214,216,102,76,11,746,40,33,269,240,29,289,624,177,323,450,57,24,561,77,152,114,184,74,287,28,396,665,648,378,372,137,37,5,390,106,70,22,847,51,49,349,874,47,881,23,817,432,337,243,153,124 }, +{ 184,229,152,432,266,452,57,497,381,619,17,313,471,257,59,129,369,339,4,12,205,99,157,77,3,5,45,128,123,106,62,202,162,32,21,216,40,136,11,188,119,249,402,349,2,403,14,37,38,54,64,291,618,524,102,475,76,177,80,90,214,33,79,141,453,23,210,111,245,372,137,27,318,269,217,494,775,86,94,139,10,1,19,115,51,401,56,769,183,361,84,529,474,277,757,72,48,880,6,247,29,209,710,179,377,89,41,292,285,328,13,588,180,342,130,556,36,477,308,275,460,378,61,197,196,0,165,421 }, +{ 137,202,160,131,860,567,318,36,295,141,843,39,800,93,30,105,187,224,276,201,261,943,964,98,49,269,219,851,116,9,22,473,1,352,203,127,822,100,206,699,144,268,662,244,253,10,717,535,332,120,44,258,143,31,64,450,704,96,859,190,180,115,55,365,531,326,95,217,28,110,564,752,86,373,284,228,317,21,77,341,13,48,434,221,197,51,911,940,159,23,384,533,260,494,559,935,291,901,88,109,574,282,128,442,367,308,959,617,5,83,489,299,251,728,170,164,165,177,492,406,926,417,293,401,349,871,125,968 }, +{ 125,64,493,0,49,165,35,293,55,93,18,98,9,325,68,10,279,137,31,48,453,389,194,197,147,13,788,14,132,95,107,77,391,371,342,174,61,43,581,105,28,22,5,259,319,104,758,671,590,128,44,32,20,767,761,457,146,121,725,465,198,126,757,289,116,100,23,21,110,763,612,12,663,199,142,120,102,101,7,427,4,356,37,941,963,867,848,498,156,88,86,58,51,33,690,558,376,180,2,864,163,955,70,623,804,422,657,202,1,40,534,90,649,470,798,522,392,352,945,502,386,359,237,221,203,177,159,144 }, +{ 15,33,4,77,13,102,23,40,32,5,11,349,51,177,21,141,117,93,202,59,110,153,64,291,318,196,31,217,269,115,10,180,82,137,1,8,44,3,98,125,134,0,165,22,12,128,351,17,95,116,2,304,120,139,403,457,210,197,49,341,453,144,372,450,494,86,48,352,386,523,36,90,719,898,6,275,30,365,57,730,28,9,317,247,326,854,76,373,475,7,72,37,421,817,638,901,401,111,961,342,899,216,402,277,260,18,237,758,569,203,99,629,437,535,39,61,16,96,752,14,300,208,922,704,105,83,308,143 }, +{ 110,854,253,811,352,141,951,661,244,642,180,752,143,498,384,269,559,317,922,10,911,160,351,533,365,373,206,902,638,873,332,261,345,260,137,717,116,843,31,417,120,44,959,531,217,341,535,201,125,36,699,860,82,55,564,434,926,318,268,940,197,901,164,574,864,276,251,824,295,876,127,450,64,617,943,93,258,776,15,282,326,165,546,817,855,851,203,96,187,144,728,83,1,372,601,219,224,762,237,284,509,21,170,169,499,98,391,479,115,768,482,473,406,86,898,662,349,48,202,131,51,5,0,896,871,401,238,228 }, +{ 2,29,70,75,220,52,112,331,145,1,163,26,74,324,6,69,58,307,19,409,278,287,38,226,46,223,394,87,25,294,606,50,66,14,632,135,583,151,42,67,45,27,396,357,171,380,675,138,5,21,113,267,262,97,24,7,309,764,635,53,537,222,103,48,335,500,83,32,122,16,550,186,96,17,73,334,849,155,189,644,852,802,37,61,54,64,571,140,35,12,582,651,461,486,853,108,36,80,18,133,451,315,624,65,337,785,573,366,445,72,86,774,92,89,297,400,10,114,95,416,587,648,462,172,71,600,820,238 }, +{ 23,13,77,51,115,177,64,141,33,102,4,217,202,40,128,269,291,59,0,180,10,2,6,7,318,11,196,14,137,90,32,22,95,153,352,210,9,1,117,15,165,16,197,49,3,125,86,237,349,18,403,216,19,17,247,21,317,304,342,24,341,134,475,453,12,275,25,36,28,20,98,494,372,57,31,93,65,44,110,27,26,120,139,365,752,111,37,35,326,164,116,402,901,260,457,30,288,421,105,56,498,29,144,391,82,203,5,401,308,285,719,386,99,143,48,39,351,55,194,437,312,450,96,523,76,509,244,316 }, +{ 340,897,691,478,914,658,724,382,363,812,698,682,100,156,166,521,264,447,909,411,1,404,88,354,303,438,96,296,274,772,39,656,418,921,9,30,410,144,23,744,848,639,664,435,21,586,919,13,271,31,335,254,51,5,850,98,367,225,960,77,203,759,621,190,48,127,676,120,32,668,33,83,4,67,44,36,272,217,69,291,131,284,286,360,821,101,285,116,242,18,223,686,102,265,905,842,317,329,93,68,82,822,195,128,125,576,692,351,103,64,238,115,652,475,349,765,326,141,966,578,110,118,0,260,403,201,163,105 }, +{ 310,18,9,326,265,252,121,51,159,77,195,105,101,120,363,33,24,354,271,102,272,709,167,98,16,13,132,639,23,158,260,55,333,327,704,450,35,170,60,202,49,557,125,93,411,254,1,364,95,455,146,817,28,520,338,494,7,286,137,156,109,39,21,14,48,128,116,64,40,31,859,320,118,212,141,360,890,318,293,4,523,615,269,576,169,20,165,153,75,196,329,373,210,177,92,69,0,301,291,781,479,371,217,627,407,403,342,127,71,44,36,910,374,5,50,668,647,351,936,682,578,538,593,553,203,185,180,110 }, +{ 57,313,471,99,369,12,339,157,266,152,38,475,328,453,775,37,403,401,514,40,188,342,139,77,11,59,377,361,229,474,33,4,51,880,277,54,3,211,14,102,62,381,23,111,162,196,130,529,556,305,757,275,128,79,115,629,13,402,421,507,308,485,210,129,123,184,177,494,41,477,153,488,117,542,452,248,300,216,202,236,134,94,670,431,89,64,17,183,440,569,136,555,72,76,734,432,457,543,961,257,215,291,304,10,413,15,205,548,90,56,2,21,46,165,318,247,349,197,86,32,0,283,412,141,36,828,386,82 }, +{ 16,24,35,60,140,71,92,7,14,230,189,108,18,69,46,150,388,246,278,2,172,68,1,451,38,330,135,126,236,309,149,443,50,267,121,565,67,582,21,75,48,462,52,563,594,314,29,399,66,191,877,103,572,55,232,366,97,696,357,380,733,167,83,220,517,5,28,674,74,12,65,0,54,9,118,262,335,146,516,32,37,653,915,22,36,133,251,96,324,87,273,394,64,199,723,223,107,10,486,458,711,13,6,95,163,571,86,726,448,212,641,192,345,70,181,132,23,49,114,550,98,20,271,606,298,633,99,603 }, +{ 206,417,940,959,93,473,728,559,499,203,137,202,8,450,564,31,44,120,269,260,318,141,116,326,244,253,10,127,268,143,22,125,98,817,434,1,373,144,128,615,51,638,23,384,692,523,341,64,922,0,851,13,489,110,352,105,9,951,717,86,28,386,367,403,299,704,642,242,55,551,901,36,958,282,859,160,77,49,291,494,801,968,30,180,492,365,911,661,498,276,854,165,164,873,37,762,456,197,21,317,12,15,5,7,569,285,918,421,752,629,576,479,115,95,2,349,96,401,758,899,393,170,898,217,457,391,48,39 }, +{ 15,1,2,13,0,5,23,7,4,3,6,51,9,217,77,341,115,10,8,11,33,349,12,317,291,180,177,14,102,16,365,120,22,901,304,165,457,21,372,260,197,32,18,40,403,453,31,202,82,961,719,475,28,247,523,117,450,64,93,141,730,110,30,352,351,752,569,44,401,153,17,386,196,59,308,24,134,36,854,494,318,899,48,137,342,326,125,629,90,49,128,437,37,817,95,275,922,421,498,758,98,210,111,203,99,237,391,116,216,57,482,144,373,39,35,20,19,468,139,615,55,221,288,898,873,509,244,86 }, +{ 2,1,14,5,50,48,46,290,21,45,29,32,263,155,207,52,38,425,171,76,408,72,449,17,292,227,89,58,513,400,428,119,476,7,61,54,67,720,75,241,243,186,502,655,69,221,36,148,24,96,6,306,3,214,106,37,0,103,223,10,179,281,16,83,350,64,90,129,430,464,181,663,666,22,413,405,18,240,710,84,262,790,123,431,28,12,741,163,115,460,255,536,56,128,35,334,141,9,220,65,205,51,86,659,99,249,23,541,95,245,618,209,519,202,70,592,270,237,528,62,27,217,133,4,394,137,114,675 }, +{ 132,121,354,167,146,463,271,195,668,446,98,223,407,360,18,212,522,447,1,781,60,48,591,411,156,562,586,363,664,21,777,393,199,293,125,807,936,9,23,590,163,16,13,658,5,126,541,55,101,941,107,259,621,191,51,834,786,68,647,765,31,150,929,759,284,35,71,49,24,465,221,118,520,7,92,883,116,608,279,727,535,70,96,44,687,478,105,50,346,697,467,963,281,33,10,566,22,581,626,93,77,296,493,4,102,108,115,640,147,75,39,770,87,32,914,127,144,598,178,0,117,273,12,2,37,783,691,932 }, +{ 131,578,371,219,224,716,105,616,737,385,187,761,258,916,254,159,49,910,9,623,98,127,836,924,272,286,442,39,28,688,890,320,627,518,293,0,944,871,574,242,31,395,22,201,327,44,116,158,276,109,908,301,860,284,36,51,23,137,95,926,713,512,30,160,125,333,170,239,96,21,456,347,228,13,86,64,128,1,68,101,839,202,506,424,295,662,938,144,55,168,24,567,48,843,406,83,65,185,613,438,203,268,35,165,492,244,299,902,221,384,16,141,115,166,827,93,10,829,37,364,763,261,169,269,5,656,434,685 }, +{ 302,467,484,97,453,6,398,124,69,36,51,14,421,298,1,66,273,87,74,523,81,426,2,403,67,168,18,386,886,756,701,801,158,13,65,0,26,23,475,24,758,86,22,108,719,25,547,615,401,159,117,423,150,730,847,563,154,75,19,646,29,7,505,817,189,278,457,165,42,458,717,572,267,138,782,531,429,153,135,450,134,49,700,208,958,671,715,622,336,244,172,143,114,95,85,63,60,52,650,197,46,899,140,92,160,28,10,791,885,718,327,314,12,5,610,109,43,703,673,670,370,341,326,289,38,35,805,879 }, +{ 190,30,778,530,539,88,100,625,382,1,960,410,731,274,96,840,957,435,321,795,748,264,669,438,466,39,166,131,36,418,144,265,9,44,228,213,23,219,224,921,223,160,406,251,31,863,83,98,310,116,21,613,13,557,203,51,128,0,252,367,335,137,254,5,924,521,284,352,829,48,765,856,258,475,827,69,201,93,49,909,404,716,238,101,616,822,33,421,32,456,593,217,125,127,944,752,67,286,276,679,931,831,329,403,816,158,77,812,221,772,317,639,37,35,28,242,120,268,578,50,230,966,686,494,187,551,478,285 }, +{ 141,202,128,2,269,1,64,5,180,3,318,33,197,6,352,15,7,102,77,0,134,13,137,10,196,153,165,120,117,14,17,11,12,82,110,4,9,237,23,291,260,494,93,51,403,217,21,498,752,450,342,341,31,32,351,22,326,125,402,457,317,143,86,36,115,19,203,40,111,961,285,16,48,475,98,116,44,678,365,95,8,37,57,28,244,27,18,901,49,45,864,569,349,373,386,391,453,59,523,144,719,139,509,25,854,372,38,24,56,898,194,164,947,79,29,817,94,253,99,873,954,730,55,65,597,61,26,546 }, +{ 44,201,567,489,116,662,224,268,131,219,434,295,187,144,31,276,699,384,137,110,442,258,36,160,228,261,367,613,93,30,64,120,908,244,284,203,260,141,202,385,1,9,269,940,39,855,253,918,98,96,373,22,406,856,326,105,206,143,318,10,217,127,55,523,115,77,125,717,617,86,49,13,836,51,28,23,574,332,128,83,190,180,871,839,371,393,944,352,317,21,164,473,935,197,450,341,959,0,177,911,840,254,531,165,291,716,418,899,800,88,829,282,827,494,48,492,661,854,5,351,100,82,299,851,822,15,860,365 }, +{ 51,12,23,15,99,37,13,10,453,165,117,38,0,197,629,569,82,115,401,120,64,134,22,474,457,11,110,719,31,93,260,9,14,475,111,32,153,79,1,94,21,40,308,128,125,8,28,5,18,54,403,351,3,44,180,202,89,450,328,33,961,2,139,36,4,49,77,45,141,523,326,386,730,48,17,217,30,431,148,405,317,237,29,98,16,341,775,413,196,365,352,55,318,72,758,116,6,59,24,7,477,898,734,509,58,39,203,95,372,482,86,854,19,391,421,402,269,144,494,90,102,291,922,817,57,373,899,27 }, +{ 135,7,172,2,92,140,66,52,97,74,324,14,226,29,232,46,38,380,87,262,314,357,278,330,16,298,571,448,35,126,60,220,620,606,75,6,24,192,107,69,246,230,331,1,5,267,294,70,622,353,222,853,550,563,71,572,583,138,150,600,189,500,25,43,50,309,21,526,32,872,12,174,54,58,37,20,893,802,22,48,145,86,64,580,462,95,67,18,888,42,950,65,387,36,171,108,451,112,10,501,26,726,335,441,103,785,83,359,204,582,279,337,461,49,0,72,423,45,443,409,61,68,356,28,392,163,673,114 }, +{ 125,386,391,23,963,60,51,949,165,221,21,13,541,393,118,719,150,197,326,758,421,791,517,801,566,558,308,48,362,7,834,670,453,551,173,401,203,102,43,20,5,71,36,457,251,77,697,279,140,867,33,126,115,32,92,149,422,743,367,232,193,0,770,4,98,509,40,674,81,18,341,11,783,192,577,546,397,16,494,510,508,217,189,104,705,610,330,2,596,105,823,293,202,117,10,314,796,64,862,629,475,468,281,246,242,38,403,29,474,298,31,27,14,858,414,95,942,779,773,569,523,430,317,301,172,134,116,91 }, +{ 85,6,42,138,235,25,222,174,525,256,746,26,289,64,280,214,544,65,32,337,90,204,355,348,74,312,95,415,145,122,233,609,416,66,194,19,29,630,835,63,22,87,45,237,874,5,605,353,112,86,519,343,135,17,648,76,736,70,287,249,180,681,0,288,21,580,27,123,444,4,226,192,52,247,81,1,56,751,216,78,58,75,49,91,792,23,48,53,183,396,108,59,61,28,3,636,624,2,208,84,370,368,12,207,37,114,10,151,153,389,197,172,51,673,72,33,671,597,18,115,125,31,730,573,179,141,73,250 }, +{ 1,14,50,24,5,67,46,2,69,103,18,54,58,48,16,38,83,96,0,35,28,502,21,36,45,181,155,12,61,7,223,513,430,22,270,65,108,9,75,10,37,133,540,29,32,290,71,52,17,171,238,114,218,72,236,55,99,309,807,70,149,328,89,433,163,227,834,68,64,23,189,51,666,6,766,765,221,115,474,86,60,366,552,541,488,13,487,243,425,95,592,536,76,780,141,477,767,186,128,663,267,251,3,794,49,741,137,418,263,202,160,207,545,27,543,920,335,428,388,408,400,150,760,148,124,106,19,269 }, +{ 811,351,642,951,180,752,110,638,253,10,352,854,197,82,365,564,873,341,499,55,143,244,535,36,498,559,901,417,22,661,141,9,28,282,165,30,206,898,373,35,509,260,115,922,728,8,39,332,49,317,31,44,762,964,93,15,911,533,203,160,1,217,120,391,137,24,116,33,105,102,479,943,77,569,717,338,134,117,968,125,965,164,64,473,127,196,153,531,384,269,601,308,98,37,959,23,177,13,867,109,617,291,864,482,169,21,372,242,546,843,434,824,692,40,32,4,551,299,758,5,468,11,629,903,12,89,949,791 }, +{ 51,23,13,453,403,12,475,115,4,15,165,202,523,719,21,457,386,99,308,37,401,2,730,5,197,32,494,0,33,1,120,450,10,59,102,128,17,14,117,3,961,77,153,9,217,569,11,196,134,899,421,40,318,391,247,90,6,629,64,7,16,57,210,48,341,177,141,269,38,260,402,180,22,468,29,27,758,317,291,817,25,19,72,18,56,361,326,133,304,54,221,757,236,34,26,288,300,275,670,365,514,509,125,237,36,65,139,137,108,45,76,52,28,498,351,801,82,805,576,114,89,61,352,41,615,24,8,854 }, +{ 120,13,23,141,77,51,260,217,202,326,1,137,291,269,93,102,82,21,203,450,351,15,180,110,165,318,352,5,128,177,373,48,854,197,64,922,523,317,96,36,349,32,0,386,372,144,33,2,9,125,752,115,59,12,210,7,457,16,817,90,403,244,367,14,898,237,160,28,35,342,719,6,275,10,18,494,304,37,30,341,50,39,24,498,730,365,391,22,473,901,4,86,223,285,247,116,31,704,453,678,83,221,661,216,44,69,61,615,201,421,65,402,8,475,268,401,11,468,49,29,98,88,899,300,242,68,3,418 }, +{ 1,5,22,12,0,2,36,21,28,48,10,14,86,32,54,50,37,7,49,9,23,65,51,55,202,96,95,13,24,208,128,3,137,323,67,17,6,99,4,16,168,18,38,194,83,115,114,281,58,45,64,318,141,745,402,46,269,31,160,480,77,103,503,105,11,403,164,69,35,218,181,72,133,29,822,433,735,430,75,180,223,89,109,584,244,341,197,361,289,165,657,169,401,239,757,177,389,98,124,217,851,238,159,33,158,956,30,59,143,70,102,39,308,44,494,754,40,253,90,155,61,766,163,540,221,76,270,756 }, +{ 219,127,98,258,23,308,301,51,910,401,276,616,284,170,395,293,105,165,22,13,327,242,512,125,201,944,115,900,374,320,109,924,964,763,272,239,567,28,860,629,391,131,116,386,55,197,935,453,843,341,9,943,743,365,506,206,457,261,758,295,36,180,670,31,95,237,538,535,662,44,224,456,185,558,332,64,911,479,0,143,169,569,492,634,10,841,317,421,867,338,282,253,49,144,685,740,384,110,901,468,187,734,719,86,417,844,509,268,12,564,141,286,713,762,918,475,299,120,1,578,164,137,159,403,870,168,551,498 }, +{ 64,180,237,5,80,165,61,58,498,250,297,197,186,2,752,864,34,445,95,869,881,17,311,391,231,115,29,151,546,549,3,312,247,483,153,469,45,27,22,365,334,73,26,136,933,62,56,495,32,52,243,215,21,90,472,592,396,509,288,651,48,738,106,53,490,482,161,211,11,316,287,177,13,369,635,378,141,86,262,667,119,663,216,128,179,40,19,209,50,164,719,457,145,23,10,352,176,241,196,527,666,171,833,269,383,125,758,14,46,70,341,507,317,873,579,464,255,400,37,875,405,210,117,33,143,331,84,6 }, +{ 77,13,102,23,64,33,141,2,115,51,1,0,10,7,177,217,6,269,180,40,3,14,202,128,22,15,318,291,196,5,197,165,153,9,117,352,210,4,237,95,17,134,16,11,125,137,86,18,317,139,12,31,93,59,28,36,49,341,24,275,304,19,349,365,110,44,342,143,403,27,120,29,719,752,457,498,164,372,111,45,203,453,82,25,569,391,8,494,216,144,98,21,247,386,475,26,401,402,30,312,37,99,57,65,20,901,38,678,244,116,326,260,56,308,450,32,35,509,300,34,961,597,194,730,48,629,55,351 }, +{ 15,120,450,51,23,13,202,523,153,196,260,457,719,386,326,4,730,128,318,141,21,11,64,403,453,33,77,32,475,269,95,5,494,125,102,165,48,12,98,1,115,2,401,117,116,10,0,180,40,197,817,7,9,308,99,899,3,352,137,90,16,37,6,18,14,237,22,17,59,217,421,402,134,8,291,49,31,961,82,28,177,670,341,351,342,247,61,758,221,194,19,391,30,317,44,93,214,365,24,57,27,20,275,25,241,216,312,285,29,288,65,576,349,629,70,72,26,39,304,498,36,96,569,123,210,46,289,854 }, +{ 2,5,1,29,45,14,58,207,263,425,52,32,46,76,186,72,292,50,7,61,476,655,408,227,48,38,155,21,720,89,17,75,243,119,6,449,171,148,306,70,241,163,106,334,214,592,54,400,36,179,90,64,790,663,24,666,84,3,16,67,69,240,10,464,56,405,496,128,0,710,74,86,270,129,22,290,460,519,255,209,220,18,141,428,223,123,659,205,96,65,19,513,83,37,4,12,27,25,112,103,115,358,66,23,9,202,95,262,249,221,51,151,28,26,767,59,35,77,350,87,137,780,245,40,409,99,237,618 }, +{ 15,4,13,11,5,23,21,32,1,40,141,3,33,2,180,17,197,64,12,269,31,352,44,117,165,102,202,120,37,318,153,0,6,137,51,10,196,341,134,22,7,49,77,752,59,217,128,48,260,203,125,237,901,82,93,317,115,365,498,90,14,110,342,111,450,95,177,873,72,326,61,8,494,244,457,57,45,19,36,210,139,386,96,144,56,89,509,143,98,76,372,275,719,27,9,160,403,94,351,349,546,854,704,678,817,391,947,253,453,304,99,569,864,308,401,247,29,70,216,758,717,670,475,474,961,116,523,221 }, +{ 214,76,5,129,710,249,618,123,45,460,2,257,17,179,769,205,245,519,207,61,184,106,162,29,209,90,119,32,1,58,6,14,255,292,183,128,77,229,64,21,141,3,655,48,27,38,52,19,263,57,59,4,40,497,151,25,202,152,464,72,7,26,80,99,102,306,311,269,33,115,288,130,318,524,86,54,177,790,62,358,50,243,452,137,46,10,84,186,454,0,425,51,12,148,619,11,210,56,23,556,22,240,112,266,402,36,217,216,37,432,476,136,227,65,70,403,738,155,237,95,111,194,381,139,241,317,180,13 }, +{ 51,23,141,1,202,13,120,21,326,15,269,137,180,165,318,197,352,5,128,48,260,2,77,0,217,12,33,102,7,403,450,203,93,9,32,110,386,115,16,14,64,6,36,4,82,523,373,457,125,18,494,237,291,351,24,342,196,96,37,453,475,817,3,752,391,177,244,615,50,144,28,854,730,210,35,719,99,30,8,317,341,401,160,31,10,44,901,86,221,98,421,22,349,402,576,365,116,922,153,308,39,704,40,72,20,285,17,59,498,372,367,393,661,49,899,67,95,551,70,55,90,275,29,247,89,83,65,69 }, +{ 7,135,2,172,97,87,66,29,52,14,112,70,58,232,337,92,46,226,74,43,262,192,331,461,20,409,16,307,5,294,107,155,334,186,171,278,140,38,151,32,849,324,415,35,75,21,25,45,609,24,227,6,145,1,298,222,60,448,61,64,620,353,387,126,37,86,69,287,549,356,173,48,400,330,644,10,314,583,18,12,22,113,220,526,54,128,359,49,853,36,95,0,76,525,163,67,568,50,65,872,28,104,72,243,138,90,137,587,71,357,4,141,17,246,544,26,230,888,177,194,380,122,474,311,315,635,83,631 }, +{ 574,187,926,860,384,258,531,434,269,110,160,244,141,253,116,871,533,940,261,717,699,959,902,219,661,224,201,373,143,36,911,352,137,260,268,31,851,617,817,728,10,318,523,180,206,295,332,144,811,489,44,843,922,64,251,662,816,120,450,55,127,93,276,855,535,839,98,131,752,951,284,559,824,164,406,642,854,203,1,864,202,317,237,228,365,345,776,498,326,567,442,197,125,282,341,367,96,21,217,417,13,943,86,351,83,679,115,908,564,954,473,82,800,896,165,968,23,128,35,669,48,5,22,170,901,873,169,385 }, +{ 354,156,296,586,1,98,850,163,223,698,379,291,64,44,31,937,48,340,96,349,182,125,237,120,658,77,284,39,30,367,23,385,221,195,293,18,202,144,105,774,128,771,687,541,342,197,116,49,371,121,118,93,5,626,360,16,692,591,453,421,386,352,318,242,153,117,36,10,795,759,260,141,463,372,326,102,87,33,697,11,4,931,446,13,35,948,103,958,899,827,781,752,664,638,558,551,497,475,457,415,304,269,217,205,201,179,166,140,131,127,126,113,107,100,92,89,88,72,55,37,24,883,203,70,21,7,914,846 }, +{ 250,34,472,80,495,483,194,469,141,176,17,64,53,161,33,814,4,667,527,73,27,231,84,56,833,269,14,102,297,585,283,503,59,597,119,106,77,862,255,128,789,11,90,153,209,389,95,887,57,76,3,240,58,23,678,61,236,40,202,134,113,117,151,51,123,304,342,136,369,137,22,237,86,65,45,32,62,445,412,311,200,573,129,436,803,328,19,10,611,196,177,133,491,214,514,72,36,947,26,5,52,139,402,378,318,504,114,0,216,180,287,322,13,381,396,358,629,475,285,46,28,2,383,179,130,112,738,247 }, +{ 97,140,66,172,7,192,314,81,92,426,74,232,298,462,501,43,508,330,69,135,279,6,230,189,24,48,21,885,104,14,448,861,38,712,126,173,715,572,60,563,246,267,16,91,633,423,71,526,52,36,357,29,362,387,18,204,63,443,32,568,324,681,879,441,138,107,67,22,20,0,150,517,85,636,607,12,825,5,893,2,174,278,65,37,149,28,726,86,87,35,779,380,560,42,125,565,95,750,83,124,46,1,49,370,251,653,77,64,877,25,4,10,221,54,33,61,672,40,23,451,98,102,31,9,51,120,348,915 }, +{ 44,31,116,144,268,434,489,367,918,492,384,948,393,127,98,9,940,22,206,105,203,0,855,332,49,93,395,959,28,456,617,911,293,137,1,299,39,125,282,202,51,141,473,55,963,120,417,326,23,95,159,21,599,109,276,730,242,13,36,10,131,8,272,284,253,551,219,30,318,523,2,352,165,512,403,269,18,64,5,7,899,244,16,301,616,450,158,115,12,170,559,935,160,924,143,260,791,386,86,692,201,421,128,457,24,719,728,499,373,37,261,916,110,197,578,48,951,239,286,576,811,494,308,35,96,564,14,498 }, +{ 202,128,141,3,51,64,15,269,5,120,33,2,102,13,403,318,1,23,196,197,153,10,165,17,0,137,82,6,110,117,180,260,453,93,352,12,475,14,125,7,351,450,494,22,11,523,457,402,31,326,77,342,86,99,386,237,9,32,36,21,19,4,134,44,27,203,18,730,37,115,8,373,95,817,421,719,40,291,139,401,49,143,341,98,285,57,111,116,25,961,498,28,391,38,854,317,144,26,752,56,16,217,308,65,277,899,365,898,509,59,45,79,922,48,72,30,41,678,757,569,24,177,615,34,29,864,873,244 }, +{ 127,98,31,105,395,293,28,9,512,299,22,44,0,49,763,599,242,685,109,284,320,159,276,95,1,116,456,870,55,286,900,125,492,36,713,506,158,51,23,935,393,24,327,16,35,239,623,144,201,13,2,65,272,21,39,7,338,12,168,141,18,202,424,5,268,14,170,347,137,86,489,841,68,367,918,301,269,64,374,185,124,165,48,567,479,221,403,662,37,963,308,131,30,401,6,318,67,616,948,115,83,197,237,434,160,295,96,169,128,421,203,154,10,143,538,194,916,352,101,371,253,928,910,261,180,164,244,737 }, +{ 7,24,97,124,2,69,66,1,14,92,18,6,140,67,20,71,43,16,21,118,29,23,149,52,150,51,32,81,5,429,189,48,36,484,172,273,38,60,65,28,22,25,74,83,46,91,13,0,298,314,12,420,610,154,42,108,398,701,467,135,35,505,173,221,576,104,421,168,336,423,37,232,403,302,143,185,847,267,137,115,10,703,158,251,308,508,330,75,68,49,86,61,26,253,202,230,107,401,64,799,388,128,99,443,102,72,607,165,426,453,570,387,547,326,105,125,85,109,517,386,95,138,615,87,126,192,671,63 }, +{ 1,349,64,303,284,442,622,202,691,141,652,447,682,67,717,917,340,237,154,352,269,919,752,547,429,411,197,293,260,44,335,180,160,863,238,201,191,167,318,9,751,730,98,453,33,418,956,475,360,69,77,765,217,140,96,31,153,120,87,724,664,389,323,268,206,116,922,888,102,948,854,842,805,196,127,399,586,658,873,851,850,796,781,695,657,385,330,244,149,137,0,662,367,351,82,947,774,551,498,363,225,146,125,35,212,909,899,848,807,776,772,770,759,726,604,576,541,523,480,450,403,393,372,332,329,327,301,296 }, +{ 1,2,141,15,7,10,180,3,5,6,64,0,13,197,77,269,33,11,352,165,9,102,14,4,349,12,291,341,23,237,134,752,22,120,117,202,498,128,36,217,318,115,51,17,317,40,153,93,365,110,342,961,82,31,864,901,125,137,59,143,196,372,16,86,21,260,111,44,873,32,49,947,457,95,57,244,326,8,351,37,28,569,954,203,19,373,678,18,139,450,29,30,48,386,72,27,79,45,854,391,144,509,55,546,94,99,164,98,116,20,76,83,253,56,898,922,38,25,24,39,61,177,210,719,58,523,494,402 }, +{ 139,99,12,453,196,277,775,475,40,401,215,33,51,111,130,23,11,77,313,211,14,38,266,129,474,477,37,328,339,369,153,719,162,152,54,231,3,308,57,700,117,403,305,188,629,102,880,457,569,183,62,514,229,961,157,556,555,115,176,41,177,471,248,13,507,670,342,4,542,257,17,123,291,236,72,79,304,134,452,89,34,184,90,2,59,76,383,488,94,421,80,32,361,128,440,377,202,119,56,316,21,494,46,165,757,288,45,543,197,528,730,585,249,431,136,5,0,432,52,210,402,120,485,29,1,529,548,536 }, +{ 77,102,33,141,128,202,23,64,51,59,269,153,4,40,115,177,13,196,318,117,291,210,0,137,134,403,22,90,217,342,10,95,180,494,18,49,11,1,475,65,32,6,16,352,216,36,15,139,86,7,194,24,402,275,14,28,9,2,21,19,125,453,165,20,25,120,57,237,3,260,26,341,317,114,349,197,326,214,48,31,12,5,82,17,300,111,421,308,365,523,450,98,110,93,247,304,351,457,285,498,39,27,312,123,44,152,99,401,37,615,373,108,372,170,143,116,68,67,56,96,719,678,361,29,961,205,35,133 }, +{ 7,14,2,58,46,16,70,112,87,38,52,135,5,155,29,66,97,294,172,32,20,334,24,92,186,35,128,227,18,171,6,21,107,37,64,75,780,666,400,61,140,43,60,48,220,232,0,23,141,177,1,54,202,307,13,592,461,115,22,77,12,45,50,51,126,163,137,409,10,315,192,270,25,845,69,90,269,76,72,513,28,113,74,210,49,262,243,9,95,89,71,318,402,631,86,884,298,40,431,102,4,275,290,278,17,324,217,19,180,151,36,237,230,67,356,291,352,314,65,403,59,33,26,609,387,226,285,330 }, +{ 15,13,1,5,2,23,3,0,365,10,7,4,341,51,11,6,9,115,180,317,14,33,77,12,217,102,40,197,21,153,165,17,64,8,117,141,569,22,901,134,32,16,202,352,37,36,457,291,128,31,269,125,237,18,139,196,44,453,391,177,372,28,349,59,719,93,482,629,86,509,961,318,873,498,99,55,752,386,19,401,210,143,48,137,120,342,50,38,49,29,27,403,24,30,758,111,110,57,475,95,203,730,46,98,260,116,90,45,523,494,326,54,105,72,96,65,546,402,351,242,277,164,67,450,734,253,83,26 }, +{ 2,1,75,29,52,14,69,220,67,262,46,38,70,87,103,6,74,24,83,267,25,226,357,366,394,96,26,189,451,112,606,486,135,145,388,324,294,108,335,278,50,66,42,238,163,58,16,331,150,71,19,149,545,222,35,236,18,418,696,133,114,7,218,582,5,36,45,433,516,537,60,380,171,138,641,600,65,550,251,151,21,54,309,603,853,223,181,12,27,287,28,0,739,565,22,17,785,48,86,68,140,461,594,37,97,10,32,423,458,95,160,733,337,583,61,122,307,281,336,64,571,186,273,85,55,774,632,172 }, +{ 9,105,18,101,39,557,16,890,1,252,272,100,265,938,21,326,23,51,301,49,0,68,24,60,264,13,35,310,254,159,737,363,167,593,137,127,121,166,98,846,118,827,156,48,202,30,333,31,55,5,411,395,128,329,132,146,578,44,520,512,242,77,731,682,170,320,371,120,71,916,518,450,373,28,821,286,33,447,327,212,354,271,32,195,639,125,7,364,95,704,102,65,616,385,615,36,688,203,64,12,110,523,455,96,141,197,75,910,4,116,836,506,88,165,93,604,404,22,318,931,2,10,260,86,338,817,40,905 }, +{ 80,209,667,61,45,17,106,483,255,6,833,151,454,29,5,738,378,2,887,742,112,311,27,119,26,469,495,113,491,161,58,145,396,19,25,84,73,70,315,207,814,76,56,136,42,64,32,1,243,62,52,306,614,287,128,445,14,74,383,53,631,3,90,240,34,141,179,86,297,4,72,148,496,65,250,611,214,138,527,46,194,123,137,186,21,655,95,66,549,50,881,292,592,790,48,87,7,11,129,263,205,122,22,440,361,59,222,573,307,38,202,227,75,269,57,85,10,358,245,644,684,54,200,862,651,425,36,171 }, +{ 23,13,51,5,21,115,12,202,15,1,141,32,2,165,0,269,180,197,401,7,4,3,9,457,308,403,453,6,341,37,318,10,719,14,48,120,128,352,33,77,22,137,11,730,217,523,40,386,99,16,260,31,475,569,18,64,450,317,28,237,391,90,17,494,421,365,177,59,125,8,102,72,93,30,629,247,961,24,221,117,44,153,498,758,351,291,196,509,899,402,817,49,304,134,76,89,210,98,326,82,482,45,57,110,38,36,752,901,39,139,468,20,96,372,116,216,19,349,203,288,29,61,95,275,546,670,27,576 }, +{ 99,12,453,277,139,474,369,339,51,38,157,196,37,775,401,188,23,475,111,328,11,313,471,153,40,629,266,79,54,4,477,33,102,59,14,57,248,880,152,342,13,457,305,542,117,377,403,514,94,308,115,229,569,488,961,210,300,211,431,41,89,734,361,485,275,670,134,128,184,381,3,72,529,2,556,719,402,64,413,21,15,215,46,177,757,165,77,197,236,32,62,421,155,50,130,17,162,202,5,45,587,52,507,10,183,555,82,247,490,129,588,123,405,148,216,76,141,7,0,1,181,29,432,6,730,90,468,452 }, +{ 7,92,232,126,97,298,387,140,107,16,60,246,14,66,35,462,150,441,104,357,314,359,279,330,633,423,380,172,230,199,24,121,38,392,356,71,132,192,451,324,22,146,893,508,493,568,289,65,560,95,0,5,32,74,189,830,2,135,21,173,18,603,64,46,43,707,20,501,52,362,90,214,712,550,877,86,607,28,426,312,48,36,29,10,69,278,12,194,49,37,1,6,672,874,204,81,267,68,526,108,835,259,733,895,388,500,779,649,448,915,61,125,149,443,50,55,54,399,590,70,72,939,572,191,124,711,517,167 }, +{ 13,23,318,202,77,177,51,269,33,102,141,115,494,4,90,403,210,153,40,196,117,32,59,128,11,137,342,134,217,291,64,21,12,475,352,31,341,14,2,98,93,10,7,95,700,49,317,275,216,20,19,16,44,22,365,116,6,453,110,247,144,180,96,9,24,17,5,82,0,498,402,349,300,127,48,3,125,57,111,87,28,86,1,203,197,35,25,401,18,165,351,326,139,60,43,26,36,450,873,864,551,473,393,373,372,288,678,576,285,37,523,120,70,65,817,67,752,697,529,421,415,386,304,214,164,152,123,61 }, +{ 13,77,2,23,141,6,0,1,217,115,7,3,180,5,10,51,33,14,197,269,352,64,102,177,9,291,15,4,317,40,17,165,22,11,341,202,318,237,196,117,153,349,16,365,28,210,372,752,134,19,18,128,304,498,36,247,12,125,59,139,95,8,275,457,86,31,27,49,21,24,29,391,342,569,509,25,93,45,143,901,137,32,244,403,26,57,38,719,111,873,386,120,44,864,20,37,546,56,494,39,961,164,758,401,30,453,110,475,35,730,98,65,216,437,34,203,41,326,99,402,300,253,82,90,55,46,194,351 }, +{ 15,4,110,13,120,93,11,23,5,21,32,1,40,180,10,82,165,2,3,33,197,141,51,37,0,48,901,102,12,31,6,7,125,117,352,77,752,341,144,153,351,196,9,237,64,134,498,143,17,269,457,342,59,202,115,98,365,450,318,22,44,401,291,873,8,244,217,475,57,391,253,14,72,349,854,372,55,36,509,89,898,403,90,719,19,386,697,139,111,546,661,27,260,61,56,177,96,569,482,961,16,638,332,758,494,28,864,148,76,45,241,79,350,308,247,437,421,70,474,559,221,210,41,164,160,137,453,275 }, +{ 15,1,13,2,0,23,4,5,3,7,6,51,77,9,341,10,115,291,11,33,217,8,12,102,177,365,317,14,165,16,22,120,349,40,180,197,31,457,901,453,18,21,450,153,961,141,719,260,523,32,44,82,64,117,304,28,93,372,110,17,210,475,59,30,134,352,386,342,569,401,752,275,196,730,90,403,326,202,36,49,629,125,817,216,98,308,247,269,351,24,203,494,899,37,95,111,139,482,758,373,391,19,48,116,237,922,509,57,144,86,137,498,854,421,128,670,873,898,253,437,65,300,55,99,20,704,244,615 }, +{ 20,43,198,325,904,104,173,147,234,422,66,97,426,319,861,77,376,712,416,725,427,945,0,7,5,470,450,192,142,32,707,178,330,819,639,38,202,22,500,21,784,583,414,2,324,230,858,61,889,49,315,113,671,81,455,18,392,74,207,232,52,24,329,107,342,33,560,13,140,595,10,715,291,120,102,95,48,1,64,359,693,600,879,16,798,58,312,494,4,14,344,835,764,306,105,526,290,40,23,289,259,90,159,158,884,577,246,37,9,170,204,681,138,29,239,194,112,12,17,31,318,72,28,226,70,397,153,65 }, +{ 105,28,0,9,49,301,159,127,512,16,170,623,24,109,35,395,158,98,68,1,65,22,55,31,761,371,713,916,18,293,286,7,242,124,347,95,338,737,327,44,39,21,2,14,272,424,154,36,506,685,23,168,101,125,299,51,12,479,48,13,870,763,374,284,64,385,320,599,5,67,202,71,276,116,131,141,185,137,836,456,175,344,239,37,900,187,30,10,577,538,6,616,289,740,60,518,118,86,169,83,364,912,219,96,50,115,928,128,935,165,318,149,747,221,346,32,114,69,810,254,237,627,269,308,144,393,201,492 }, +{ 213,466,689,88,230,435,321,30,699,795,201,831,352,144,217,443,854,530,840,190,539,957,418,1,96,203,382,855,251,100,669,410,335,93,83,748,367,44,662,960,31,934,36,206,141,752,351,317,822,128,731,116,778,120,284,274,131,98,228,927,332,187,69,438,521,800,21,242,160,224,48,372,219,310,137,735,686,557,326,285,237,576,127,922,5,859,269,260,13,494,264,82,523,403,35,244,202,615,166,39,616,551,345,930,180,67,253,921,299,966,2,10,9,258,23,473,365,692,265,197,866,32,276,51,816,617,856,223 }, +{ 130,453,196,47,57,4,236,14,59,412,51,711,381,730,153,77,202,108,128,361,23,46,283,402,210,11,300,719,13,117,38,803,200,401,183,78,34,475,54,342,318,27,369,432,275,2,757,133,503,472,12,3,585,134,62,862,161,529,37,328,457,56,390,494,102,73,176,50,440,322,111,670,65,52,377,899,181,629,17,114,33,136,99,266,123,40,514,339,291,53,250,84,177,21,188,216,389,217,386,139,32,961,155,480,162,754,832,714,894,739,383,657,269,152,106,603,543,527,523,497,277,485,838,137,86,64,48,758 }, +{ 1,2,14,50,5,38,46,36,22,67,103,0,114,65,29,12,28,218,54,7,133,83,86,96,69,21,75,208,24,6,52,168,194,389,10,128,51,160,55,17,23,48,18,181,238,45,37,32,95,16,283,532,58,137,503,202,49,124,171,361,480,13,735,9,289,109,782,285,3,115,70,756,141,169,484,584,35,244,717,64,402,4,328,220,163,646,158,803,505,19,253,789,164,185,108,25,155,487,236,269,143,946,394,318,610,401,335,105,433,403,851,87,800,223,866,309,420,61,27,180,597,99,366,308,59,754,514,418 }, +{ 6,74,26,19,165,453,391,197,14,138,730,29,357,70,51,457,1,50,108,125,500,509,27,145,386,112,366,52,46,235,133,262,13,308,47,331,2,719,78,42,309,75,603,25,223,394,785,23,600,218,38,163,115,114,17,558,34,53,220,128,487,58,236,122,65,130,766,401,498,96,409,307,45,85,66,444,899,73,516,632,64,711,54,630,412,196,5,103,21,180,894,113,221,137,624,241,237,594,773,758,867,629,433,390,475,820,546,12,648,67,32,153,285,606,99,545,745,569,644,324,583,3,181,675,903,283,802,670 }, +{ 10,9,30,376,0,68,55,39,18,43,20,49,174,104,88,501,120,342,450,858,28,2,137,523,470,453,197,757,603,37,414,165,31,325,730,729,671,595,455,222,204,77,64,58,22,21,17,13,5,194,110,95,389,318,260,177,128,102,97,33,1,904,725,7,788,568,387,90,54,46,32,14,11,259,98,147,61,817,392,518,961,958,922,899,873,854,819,805,785,739,719,713,661,615,550,534,528,494,477,475,474,465,457,449,386,372,355,351,349,337,298,294,291,289,269,262,247,196,173,171,155,135,122,105,94,56,48,36 }, +{ 15,23,13,1,202,0,51,77,5,9,341,141,137,120,4,260,115,2,351,21,128,7,12,450,82,901,197,180,269,318,10,8,11,22,403,32,854,33,165,40,6,18,3,16,37,494,523,457,102,177,28,64,352,453,14,317,922,217,36,365,48,421,401,719,475,291,30,110,59,24,569,817,93,961,308,730,17,752,125,117,898,31,342,153,196,285,44,99,576,86,275,642,49,90,402,39,143,139,98,221,386,244,116,373,55,57,629,144,210,661,349,134,203,859,899,95,498,35,482,372,72,237,253,105,216,96,111,758 }, +{ 272,131,105,578,219,159,371,49,616,716,9,224,320,286,737,623,187,910,385,28,258,98,761,127,254,347,916,424,22,109,627,158,713,95,924,239,0,242,836,39,201,31,301,912,944,442,125,170,116,327,293,44,538,688,374,23,55,168,276,36,64,284,51,871,185,395,35,506,221,512,518,202,890,137,101,13,860,169,68,574,479,86,908,10,24,333,16,1,65,144,269,364,308,228,344,141,21,338,567,30,577,165,128,160,747,18,763,124,839,926,299,261,456,268,295,164,662,406,197,67,312,175,7,928,421,685,154,810 }, +{ 457,51,13,23,961,719,12,453,629,99,401,57,515,3,4,165,11,45,62,41,14,17,2,569,37,77,33,32,5,0,40,59,1,308,76,115,197,38,72,79,730,21,474,102,7,475,6,54,9,64,403,10,56,775,237,477,202,29,152,188,341,328,90,509,111,89,215,128,880,734,339,247,757,134,386,84,421,317,313,217,129,94,22,318,157,34,130,117,361,210,670,555,369,365,285,269,266,186,181,153,123,24,899,485,120,542,25,391,758,514,468,277,270,243,227,196,180,179,155,148,137,52,50,49,46,18,471,236 }, +{ 202,318,5,120,450,494,77,33,102,260,21,1,128,13,165,403,523,23,269,342,12,326,817,32,475,141,402,64,2,17,82,457,180,137,48,730,125,291,197,453,37,93,51,615,922,719,386,210,15,177,854,351,98,7,899,576,275,6,44,25,961,4,196,117,115,285,95,14,217,31,391,352,237,65,38,29,27,20,758,40,10,9,153,194,367,116,110,105,50,49,35,16,139,498,341,61,163,39,30,11,349,293,216,144,108,99,76,58,54,43,0,203,3,134,113,70,959,958,940,933,923,900,855,841,750,725,704,639 }, +{ 141,269,352,180,217,202,349,64,160,137,317,318,372,494,752,418,244,954,864,437,753,237,13,341,291,165,717,22,197,115,304,260,365,93,36,21,351,247,120,922,77,82,48,498,32,5,579,144,10,23,342,373,33,854,95,128,0,391,164,143,125,98,482,51,403,28,326,898,386,76,72,1,951,678,638,453,201,86,110,30,457,312,253,221,163,96,44,475,102,901,719,421,196,134,190,166,131,100,88,474,401,203,859,597,521,509,466,459,435,393,321,284,228,108,65,59,40,4,851,576,350,308,288,241,216,214,179,99 }, +{ 70,145,29,112,74,26,2,75,287,52,220,226,632,87,294,6,135,307,163,66,19,396,278,138,635,409,25,151,58,113,1,122,331,222,5,97,42,644,324,461,27,849,500,69,651,45,7,172,262,853,337,53,648,573,624,64,334,445,171,21,67,186,32,73,297,46,38,416,14,17,140,80,61,50,436,155,884,223,24,22,85,48,380,852,92,65,86,889,12,315,36,774,571,103,549,37,10,95,243,491,83,16,72,665,227,89,357,394,76,309,35,106,54,675,808,267,474,250,609,71,881,209,133,606,96,232,583,400 }, +{ 351,82,317,141,752,898,901,180,352,341,15,365,1,498,217,115,33,253,32,143,10,160,55,0,854,5,197,269,21,77,4,36,137,23,64,2,110,482,169,873,120,244,96,7,89,13,349,86,164,102,72,93,372,148,922,40,16,345,90,531,9,83,437,22,251,35,260,457,546,165,202,6,864,48,291,51,28,719,59,238,18,308,247,128,11,304,117,12,569,237,65,961,817,509,450,535,196,95,125,730,523,468,144,98,49,3,899,661,342,318,373,153,579,734,14,490,116,203,134,559,391,312,130,123,109,84,76,326 }, +{ 39,9,310,254,101,252,265,272,100,0,557,30,49,688,455,105,190,166,329,627,159,286,625,709,327,333,18,68,158,404,520,578,131,438,731,16,553,24,98,364,31,676,35,118,593,1,518,48,344,44,167,23,65,21,93,604,829,795,127,60,890,96,88,616,22,363,656,51,242,28,419,716,125,124,846,71,274,36,264,613,13,137,95,116,212,938,109,224,827,411,154,910,160,639,371,664,128,5,374,144,530,395,228,170,219,865,67,10,203,759,121,83,301,382,320,737,55,682,37,647,86,271,793,7,821,115,916,285 }, +{ 2,58,29,151,52,186,70,5,549,75,1,112,45,400,61,163,113,227,307,315,311,287,220,46,7,14,155,74,243,87,409,66,27,80,209,631,845,780,69,262,135,48,50,171,767,496,869,454,331,6,491,17,334,278,106,32,19,64,614,21,73,145,26,294,24,592,67,25,38,659,223,16,97,255,172,72,226,502,852,128,37,140,721,666,89,667,290,36,103,324,86,849,267,76,141,54,35,83,65,207,22,148,96,270,12,635,10,4,95,92,738,474,18,445,119,611,425,137,431,263,114,396,663,0,122,133,90,237 }, +{ 141,217,269,352,317,180,23,13,21,115,77,349,202,137,51,64,372,165,197,32,365,752,318,291,341,5,237,102,15,177,498,33,48,244,304,128,125,4,864,253,39,30,44,11,247,22,9,0,31,901,308,403,1,717,201,342,49,10,98,16,7,221,95,144,40,88,494,90,12,28,117,93,678,120,68,326,100,203,86,96,468,2,475,206,116,546,457,160,437,8,873,210,127,110,76,65,55,37,418,386,350,72,18,453,223,143,24,153,851,351,260,284,579,576,401,268,482,196,35,421,82,3,402,166,101,954,391,83 }, +{ 68,35,101,149,124,154,0,65,9,175,24,398,28,16,189,71,118,429,55,346,67,114,7,345,18,230,399,364,133,14,252,722,39,283,1,547,140,69,108,484,302,694,100,439,103,46,705,83,166,267,389,2,458,92,336,50,212,289,21,246,375,251,419,109,36,520,329,158,22,637,12,60,952,48,265,236,702,596,96,806,30,732,5,826,604,64,194,49,412,701,388,88,420,191,535,264,562,167,886,23,796,168,51,650,238,570,782,868,86,13,218,95,917,768,38,10,32,309,404,706,37,54,221,602,125,159,105,676 }, +{ 66,7,29,112,52,97,2,192,20,151,74,43,145,173,222,135,226,525,415,172,644,104,337,544,353,580,32,45,25,287,186,609,58,70,204,155,14,5,21,712,122,87,227,174,331,61,849,549,426,90,355,81,334,560,6,232,416,243,632,46,0,92,324,177,64,113,673,624,107,59,37,635,216,140,396,138,22,500,16,38,4,707,128,65,48,307,17,77,86,137,91,35,12,721,665,789,141,24,294,884,85,42,414,194,115,526,72,311,825,11,126,262,597,76,89,10,28,75,1,387,209,60,304,171,51,147,651,40 }, +{ 31,125,44,299,116,22,170,55,242,456,599,685,36,870,64,393,492,268,1,479,164,86,558,28,506,144,10,740,23,206,903,9,2,0,963,237,127,546,301,13,434,143,367,5,169,93,51,239,12,654,965,384,137,498,7,923,203,918,489,98,180,95,391,141,762,115,352,417,638,841,253,6,105,14,951,8,940,165,197,844,128,948,457,401,959,244,160,308,282,185,21,312,109,286,202,332,341,49,168,386,752,269,538,365,4,37,24,512,911,473,873,509,551,634,499,318,719,3,208,395,708,217,450,421,96,901,35,16 }, +{ 188,11,79,12,99,377,542,94,339,33,474,37,40,139,111,102,453,4,51,775,475,277,13,471,23,196,629,153,431,401,117,210,134,266,152,403,115,38,961,128,59,300,57,77,880,342,157,485,313,569,3,229,202,14,177,275,477,54,308,184,488,328,21,89,72,670,32,64,2,76,10,402,15,90,165,369,148,141,5,162,318,405,17,269,257,205,216,413,757,129,62,361,0,494,734,41,421,179,45,452,197,217,291,120,82,22,130,386,123,1,50,237,358,125,381,497,498,6,137,285,36,180,48,482,372,29,46,341 }, +{ 658,698,340,303,31,363,919,98,44,156,296,850,586,668,354,87,914,120,163,30,127,70,23,385,51,48,1,478,125,49,77,774,223,260,291,82,691,349,13,854,225,217,116,39,10,724,21,922,898,447,351,293,201,75,9,269,284,759,541,480,281,268,221,165,621,141,442,622,411,371,318,187,153,115,105,102,65,777,93,64,33,652,352,202,744,453,433,398,330,191,180,177,137,128,117,95,69,35,931,638,40,5,888,372,146,140,100,88,565,457,450,335,326,317,219,190,159,158,134,36,32,22,518,482,401,394,276,947 }, +{ 15,82,4,197,13,33,1,23,901,21,5,11,10,165,32,0,120,180,2,40,3,51,110,77,642,7,8,351,326,115,6,386,365,217,457,9,117,401,12,37,102,260,523,134,719,758,569,22,36,59,317,730,31,57,342,898,961,450,453,153,17,141,291,137,475,55,177,48,116,90,196,93,44,403,14,64,352,629,54,372,817,28,143,18,269,139,16,958,96,202,494,349,421,275,752,210,128,49,94,86,244,29,498,899,801,318,851,38,670,72,615,474,304,148,41,89,25,19,308,482,160,373,253,70,61,30,277,125 }, +{ 15,4,11,13,23,5,21,180,141,32,202,269,3,40,1,352,318,17,12,64,33,165,197,31,2,0,51,120,128,117,326,59,457,10,153,196,134,817,6,48,125,93,237,82,44,137,450,77,494,403,49,203,386,752,37,523,7,22,961,217,260,8,115,110,341,57,177,498,719,90,615,9,317,102,365,139,14,730,901,210,98,402,72,453,19,116,65,18,111,391,45,56,95,351,27,899,99,349,293,291,300,475,86,28,36,401,25,55,144,244,275,509,247,216,38,678,76,61,393,29,372,576,485,373,30,253,551,468 }, +{ 91,233,370,718,85,6,81,25,65,256,63,235,42,138,511,397,123,26,343,74,650,87,249,643,355,544,18,64,194,510,636,45,174,280,90,22,108,348,681,104,1,84,86,114,389,4,415,66,19,0,673,283,214,141,122,524,289,56,53,193,208,95,240,750,297,29,119,605,27,573,445,436,736,10,14,11,287,597,32,630,49,24,192,57,5,17,204,475,103,61,102,70,947,152,51,588,23,16,183,77,619,730,523,453,444,59,561,2,813,701,874,323,75,33,28,804,129,96,403,368,21,9,145,177,396,255,693,611 }, +{ 23,51,202,13,21,128,318,141,403,120,450,137,48,494,269,5,1,165,125,64,32,260,15,523,102,475,115,12,14,457,31,33,197,77,817,44,98,453,2,308,95,49,99,16,7,180,18,93,326,37,221,6,4,352,421,401,116,291,61,386,203,82,59,36,9,40,30,24,719,117,110,20,217,153,730,17,615,351,402,237,50,854,10,196,39,19,342,3,922,25,96,341,35,28,22,11,105,177,90,72,58,38,317,247,54,961,805,576,207,68,57,46,349,210,134,86,704,75,70,899,373,65,901,670,0,391,89,45 }, +{ 365,180,317,752,341,217,115,352,901,482,372,498,291,15,253,244,141,13,1,579,269,349,7,5,21,4,318,23,351,82,0,2,51,77,93,125,10,898,89,160,202,44,6,854,165,3,96,308,221,148,137,332,247,197,33,11,304,851,531,144,120,494,40,717,468,251,403,72,177,391,32,12,546,453,421,260,203,437,961,574,258,533,48,509,55,922,880,661,615,373,261,134,241,873,475,345,949,867,734,728,699,569,490,473,434,295,288,237,219,143,127,110,27,17,16,187,153,117,876,350,99,36,9,947,128,86,64,679 }, +{ 115,498,901,197,341,468,13,352,482,546,165,509,237,569,9,873,64,23,365,22,180,95,391,317,758,558,49,752,734,109,28,105,217,51,844,86,837,143,629,131,254,876,12,159,158,719,39,406,308,148,141,679,117,490,83,637,457,10,708,166,36,706,102,89,35,505,168,442,372,125,55,530,221,37,208,669,535,332,272,153,103,98,903,401,261,219,768,114,33,864,364,187,169,345,4,574,434,258,242,224,67,30,3,253,961,868,164,228,816,77,843,120,438,384,351,230,134,128,100,99,2,753,50,44,31,584,347,170 }, +{ 23,13,51,202,141,21,180,165,5,197,125,1,115,269,32,120,12,128,318,64,15,137,523,2,33,386,7,260,14,403,48,457,102,4,450,6,93,99,82,401,719,9,730,110,16,31,237,77,326,308,421,98,453,352,37,221,475,90,44,117,30,0,18,144,899,153,10,817,3,203,391,196,11,59,351,40,217,494,177,17,341,134,116,39,8,24,95,752,498,402,20,509,615,61,28,96,19,247,961,291,36,72,365,86,317,285,241,76,49,373,38,29,25,342,70,304,58,57,242,546,576,22,350,349,901,629,801,551 }, +{ 214,289,90,174,874,280,355,138,65,85,64,6,751,81,312,194,249,835,233,348,636,875,91,343,288,519,32,0,524,247,415,104,29,370,681,750,74,204,22,95,217,437,235,86,123,25,352,192,42,597,954,256,77,18,673,63,864,752,671,306,87,353,746,287,712,237,397,389,115,588,222,177,180,145,580,715,21,141,59,474,317,207,66,56,49,52,33,28,579,45,544,10,84,61,784,102,4,511,202,693,57,40,24,117,173,76,319,240,208,5,216,26,304,134,291,205,114,525,72,17,153,577,168,11,108,422,718,498 }, +{ 15,4,11,5,13,23,21,32,33,1,37,31,2,141,3,180,64,102,44,12,49,196,120,77,40,6,17,7,153,117,352,134,0,82,269,197,51,22,93,10,752,260,165,217,98,317,137,365,342,202,901,318,95,341,450,115,48,30,9,128,177,351,125,523,110,498,854,291,237,89,90,8,817,45,59,14,57,86,457,873,326,373,372,72,61,922,139,56,275,482,36,864,391,96,898,386,111,403,144,27,38,509,899,453,70,116,29,58,253,244,76,160,475,437,304,316,210,16,41,569,308,300,203,719,288,221,546,148 }, +{ 217,33,15,13,117,196,153,134,23,51,77,141,0,2,64,115,102,10,4,5,7,6,180,3,1,11,165,352,202,22,9,40,317,269,197,14,128,318,341,237,17,177,16,291,12,95,365,120,125,18,752,28,453,372,59,137,498,349,49,36,31,24,403,32,110,98,210,386,19,457,401,8,391,901,475,44,86,260,342,304,143,247,308,730,93,275,244,494,164,719,21,864,65,509,57,961,523,25,90,105,546,139,216,55,38,482,569,402,35,29,450,99,873,421,48,670,45,27,26,326,20,37,437,39,629,579,56,203 }, +{ 15,33,77,4,102,117,40,115,134,11,153,13,217,21,1,5,317,196,32,23,3,2,59,12,291,341,365,51,139,120,177,180,372,349,48,141,260,17,342,110,6,197,37,7,72,482,90,165,111,57,352,93,898,752,0,326,457,210,82,854,901,450,269,202,14,36,64,125,523,373,10,137,89,922,203,719,498,730,961,569,244,817,143,44,318,8,76,304,277,403,253,275,31,494,128,386,308,160,22,247,56,899,148,144,453,468,61,638,579,300,96,437,221,116,873,38,485,94,99,237,216,9,27,49,45,19,86,401 }, +{ 15,13,217,2,1,23,0,4,5,3,317,51,341,7,6,8,115,9,365,202,10,11,291,12,22,33,269,31,82,180,308,457,120,28,901,30,18,77,197,450,351,21,961,40,221,372,523,165,730,318,719,854,141,102,64,453,110,260,14,569,93,117,44,16,817,36,32,349,386,498,482,98,403,898,196,134,35,153,24,49,125,752,579,326,899,237,128,137,247,401,342,48,475,177,39,304,661,20,922,509,494,19,116,59,37,203,468,95,329,144,758,55,17,65,244,86,111,391,90,118,437,113,91,122,143,629,864,99 }, +{ 135,66,2,7,192,337,97,52,14,20,673,355,29,43,104,107,331,226,415,376,353,222,294,147,74,112,46,172,560,155,32,5,324,470,609,173,174,21,825,16,64,58,298,145,92,707,38,232,77,35,356,4,10,414,22,525,287,262,171,888,86,632,544,595,635,568,178,90,48,580,87,334,202,597,72,37,60,587,474,6,151,95,307,49,177,59,25,0,142,17,858,61,409,128,1,12,721,583,318,259,362,500,141,126,269,198,102,319,70,494,13,230,853,45,227,54,849,612,400,65,204,186,33,819,246,76,644,279 }, +{ 195,132,146,142,271,167,121,363,878,178,687,647,786,212,354,77,522,929,60,202,411,120,35,234,147,259,101,360,520,626,446,781,98,0,450,668,5,199,55,771,562,23,16,706,777,49,936,18,621,604,48,158,318,639,13,31,407,22,125,21,591,590,465,481,159,534,51,109,198,419,24,126,364,191,141,7,682,551,252,28,20,806,634,156,105,14,447,948,494,107,10,116,39,43,1,269,260,33,265,128,640,291,93,403,137,326,95,203,421,379,102,765,9,82,692,110,416,32,883,664,553,455,694,475,842,523,293,239 }, +{ 202,77,20,318,104,66,269,291,494,102,177,43,128,0,342,216,173,5,2,97,52,450,234,22,7,33,750,29,112,319,74,275,90,337,32,49,198,210,678,544,147,204,58,24,141,789,21,349,64,18,304,16,10,14,580,120,192,194,325,13,416,142,403,500,196,61,59,137,115,86,712,153,113,12,226,174,415,40,23,81,4,28,70,525,402,1,9,230,72,35,178,214,636,46,11,6,904,37,95,87,48,217,355,597,334,31,798,39,17,38,427,186,222,65,376,117,426,285,36,135,51,208,353,673,155,792,172,260 }, +{ 107,362,612,359,414,356,97,392,683,259,819,43,298,660,465,147,581,729,173,779,7,595,501,20,319,470,534,858,246,376,279,126,230,14,5,178,427,232,493,35,92,49,132,426,885,60,32,104,387,0,786,423,66,38,21,121,878,22,16,146,861,441,95,357,560,830,142,2,199,61,172,10,81,707,150,70,590,422,207,48,140,46,58,929,52,324,603,192,18,508,202,314,77,37,347,33,263,13,135,28,241,12,23,29,125,1,345,124,102,98,74,64,568,24,198,380,40,4,687,626,105,607,117,72,177,577,65,318 }, +{ 6,19,13,26,453,165,457,386,51,23,14,74,401,391,308,758,719,603,47,867,29,138,2,70,730,27,366,309,108,324,197,1,25,196,743,711,78,235,262,130,42,475,133,50,115,145,670,487,112,58,53,52,85,17,421,894,96,34,412,5,331,516,509,163,33,500,569,629,482,388,45,468,153,218,128,64,341,283,594,545,444,65,125,600,217,114,236,754,75,899,73,785,630,137,95,38,801,328,895,103,961,550,901,200,11,350,558,117,4,220,180,54,46,745,3,134,433,766,287,739,122,223,317,226,390,241,21,357 }, +{ 179,205,72,245,317,247,152,180,216,490,76,752,352,79,217,59,864,94,498,4,497,452,89,148,11,875,111,141,405,115,188,21,468,588,873,197,579,471,365,464,37,645,372,482,5,77,64,431,341,237,10,529,349,1,12,23,2,474,128,134,316,339,291,36,32,51,38,413,437,629,954,734,308,99,13,143,48,102,165,14,288,90,377,749,184,401,460,542,33,428,3,7,269,449,961,569,901,350,86,40,257,244,202,0,270,117,6,304,54,241,402,45,137,221,457,403,177,9,164,139,318,17,528,15,453,421,358,769 }, +{ 107,359,7,314,392,298,387,279,581,32,232,60,14,126,97,493,534,150,356,121,423,362,172,92,259,147,441,95,35,501,22,357,21,16,246,590,0,380,786,49,18,465,13,64,132,5,28,146,178,10,508,779,23,140,48,24,71,939,929,660,51,135,607,324,603,115,230,43,46,241,38,192,683,462,247,66,612,568,941,835,819,77,12,550,90,36,560,199,4,312,33,9,893,347,330,65,59,52,20,519,2,278,50,626,58,202,37,345,771,294,117,40,102,70,470,451,156,448,350,316,707,236,173,288,31,75,54,672 }, +{ 156,354,586,271,195,591,668,132,48,360,340,121,23,21,463,446,1,5,13,18,658,296,407,51,146,70,698,883,626,55,60,221,777,163,32,363,379,621,590,562,58,49,167,467,647,522,150,61,118,24,411,781,115,447,223,767,786,7,16,31,87,308,687,22,9,259,279,212,44,17,518,346,71,10,371,45,98,2,919,241,761,33,12,93,64,39,541,37,608,35,774,341,759,75,664,941,663,125,846,581,401,120,638,566,303,68,936,260,0,932,199,105,634,270,706,30,453,141,102,95,86,110,107,771,273,535,82,727 }, +{ 23,51,13,453,4,475,0,12,457,10,2,115,401,15,11,403,5,1,40,99,14,59,961,165,569,629,9,32,77,37,308,3,719,7,57,421,21,64,17,494,6,22,33,102,18,202,177,128,730,197,523,670,139,141,899,16,90,341,8,56,217,210,402,48,24,386,125,27,318,95,45,86,72,775,269,758,50,65,19,76,111,38,317,180,89,62,342,54,46,734,28,247,41,29,36,509,237,26,880,196,352,365,275,468,120,216,153,152,108,25,137,757,349,221,181,117,277,49,477,123,34,291,304,801,82,31,678,597 }, +{ 48,125,221,21,165,23,197,13,763,423,508,193,98,293,457,391,5,867,693,386,441,81,18,16,51,92,60,314,64,683,401,49,237,96,104,43,430,35,397,558,241,180,126,523,105,32,758,140,38,72,510,308,474,356,230,95,279,102,501,784,517,663,858,541,31,633,498,718,149,127,91,422,97,418,118,900,159,144,115,511,44,203,246,366,24,223,480,453,361,350,218,160,71,22,701,719,393,40,173,124,963,421,370,148,69,643,93,743,598,493,298,253,191,170,164,116,7,1,939,730,120,74,33,20,326,63,192,269 }, +{ 202,5,450,77,120,132,494,195,523,318,326,1,260,146,121,271,167,576,203,0,817,615,18,21,403,692,234,604,354,142,49,13,35,23,7,212,20,14,2,682,158,252,147,137,101,102,128,32,33,265,48,16,291,551,342,419,364,60,22,966,475,43,107,9,178,363,51,639,17,10,687,12,821,6,859,159,520,416,97,848,805,31,360,269,109,93,198,329,259,39,55,64,562,786,402,92,82,553,878,70,61,647,455,180,411,95,315,191,4,842,105,239,165,822,638,115,98,634,24,453,346,126,99,156,899,199,522,28 }, +{ 39,166,9,101,404,158,333,30,0,190,310,252,274,68,438,419,100,344,88,49,254,124,694,865,154,530,656,364,577,28,810,265,65,35,131,455,159,857,16,21,24,793,22,539,212,777,105,625,747,677,709,562,95,48,286,1,329,703,18,228,71,125,272,36,109,13,10,289,538,219,676,706,778,51,31,98,557,86,23,175,60,553,406,55,962,671,840,118,224,168,221,83,669,67,213,382,96,570,64,960,346,327,170,613,731,191,242,44,115,759,439,957,627,748,114,634,264,688,593,37,251,137,320,127,7,520,167,518 }, +{ 15,110,4,120,11,5,1,23,13,165,93,21,10,2,32,40,180,82,197,3,51,6,33,0,7,125,351,901,391,352,77,9,115,260,102,341,117,217,317,134,31,141,752,453,12,48,8,153,386,509,873,203,457,401,196,37,898,372,59,326,475,55,14,202,569,57,498,403,719,17,64,308,237,494,318,269,94,90,253,961,22,44,730,36,697,342,177,482,56,474,137,89,72,421,349,38,670,576,139,111,30,28,210,148,96,758,143,734,629,16,244,160,128,18,291,41,805,247,25,19,84,373,245,61,488,579,205,523 }, +{ 2,29,52,1,26,58,6,262,5,50,14,70,103,46,19,38,309,27,96,112,67,516,75,163,133,331,218,145,108,223,17,236,45,83,287,25,487,74,114,171,785,409,357,113,238,24,433,151,53,36,388,220,739,34,545,396,307,532,583,675,61,600,181,42,65,73,764,160,635,21,603,394,12,80,594,186,150,324,894,632,3,7,334,99,54,137,315,891,69,445,37,412,323,48,745,537,18,283,87,458,138,822,308,243,155,765,86,711,895,398,285,28,651,366,226,115,766,849,294,23,480,820,51,221,297,22,732,209 }, +{ 120,77,260,141,51,13,217,23,317,1,450,291,15,202,137,326,373,110,177,180,21,269,165,318,128,197,352,115,349,64,32,203,5,342,351,0,102,48,341,96,854,93,82,523,2,244,457,9,160,752,386,36,922,12,365,372,37,898,7,304,275,403,86,10,210,253,817,16,704,28,83,90,901,498,143,6,615,4,719,22,494,237,18,661,730,473,30,24,308,50,221,40,76,72,55,8,59,35,11,33,391,285,67,89,144,125,61,961,99,39,401,14,3,475,247,402,44,31,70,453,49,509,642,153,208,482,17,216 }, +{ 444,183,291,297,358,196,554,255,56,59,123,269,33,605,108,122,153,177,275,210,77,57,341,117,130,128,475,467,730,152,134,42,403,4,50,880,19,381,315,401,51,65,421,216,630,349,283,21,503,235,133,85,23,11,719,368,46,202,99,494,432,229,40,14,29,561,665,339,670,480,736,588,523,453,102,15,899,650,326,256,249,236,197,125,318,2,775,757,711,624,396,365,328,524,78,64,3,529,194,18,13,0,250,12,6,690,25,445,266,48,511,402,285,26,789,678,260,242,165,120,343,769,543,488,409,390,369,366 }, +{ 23,13,51,12,165,14,120,153,38,197,117,37,99,401,134,386,453,54,128,202,141,196,64,5,328,40,237,21,137,139,15,260,33,2,719,475,125,133,180,457,0,46,403,242,77,236,6,208,1,36,96,32,4,391,361,115,308,52,194,629,17,569,730,477,86,102,546,817,450,19,10,758,961,933,923,903,558,160,11,29,22,18,3,188,111,79,114,108,65,48,95,291,56,349,157,94,41,338,221,89,82,34,59,277,177,49,317,304,506,372,351,50,26,421,217,44,31,7,72,341,352,350,316,312,207,183,143,130 }, +{ 95,64,348,33,467,56,343,174,554,511,77,312,233,120,108,180,0,561,6,355,543,13,306,370,197,165,859,25,18,29,297,256,91,63,899,237,49,770,650,412,235,85,138,19,42,523,65,671,90,102,22,9,32,368,194,40,23,10,4,576,426,416,204,161,153,117,116,86,26,27,104,448,775,721,719,457,450,344,326,275,217,72,24,605,730,283,245,196,98,76,17,5,577,519,123,835,475,192,81,78,353,588,386,365,269,260,222,220,214,179,170,141,134,125,119,105,97,66,51,28,11,881,273,242,172,746,707,643 }, +{ 120,51,260,326,450,202,23,77,141,203,93,13,137,318,21,1,82,165,110,15,523,5,351,269,197,64,102,373,128,33,217,48,180,386,32,291,457,125,817,352,922,854,403,115,144,116,177,349,210,12,96,196,615,475,7,719,372,367,237,40,730,2,342,704,4,98,391,10,275,59,494,453,401,0,37,6,421,317,50,16,90,36,44,752,308,9,498,160,31,393,18,899,221,341,14,3,30,45,402,8,898,304,83,692,99,24,551,72,947,247,216,28,961,153,864,678,661,473,88,509,70,22,285,17,576,300,76,35 }, +{ 15,4,11,13,5,23,21,32,1,64,93,33,2,3,180,40,22,37,141,10,82,0,6,77,102,44,31,352,7,120,117,165,341,12,48,49,110,197,98,134,9,752,17,901,153,498,342,269,196,115,291,365,128,260,36,317,137,51,95,59,217,318,237,202,349,72,8,509,391,14,873,177,16,523,86,922,386,247,143,304,210,125,29,45,57,457,453,90,854,482,403,111,579,160,244,18,569,437,89,61,253,76,27,899,351,864,275,450,719,56,758,116,875,961,277,468,164,148,25,817,401,316,84,139,546,490,216,46 }, +{ 15,341,13,141,23,33,217,4,77,180,1,102,40,317,5,137,0,365,202,51,11,21,351,854,9,269,82,10,115,372,32,64,922,901,22,117,260,352,197,153,128,36,28,196,752,134,165,59,318,349,3,120,2,12,30,177,244,37,450,143,291,31,86,457,48,139,160,16,8,7,898,247,253,125,523,18,49,44,57,403,304,55,730,569,961,6,111,110,14,96,498,494,17,308,719,72,39,237,642,386,475,90,98,24,342,45,401,817,326,873,661,76,277,95,509,105,421,210,93,811,453,717,144,391,99,89,164,221 }, +{ 272,105,131,327,286,320,578,109,22,28,219,239,224,159,9,616,371,49,95,623,538,98,716,910,713,127,258,737,187,185,385,761,424,168,170,64,55,254,347,169,928,916,0,158,164,1,924,31,23,301,51,293,39,627,374,125,913,201,44,836,242,912,36,395,13,116,65,479,634,24,442,654,86,346,284,276,35,312,137,308,512,160,115,194,420,10,175,2,154,7,338,21,128,16,944,83,143,456,12,67,601,202,762,141,908,124,535,289,574,871,688,165,401,900,5,763,96,892,933,384,317,506,237,860,14,37,68,30 }, +{ 16,68,35,71,18,60,118,24,191,126,92,101,150,212,167,154,302,175,9,346,55,0,273,7,124,108,107,375,628,28,566,14,722,149,815,199,589,439,882,575,520,363,121,562,246,467,21,364,1,806,279,140,22,49,48,252,517,695,65,230,345,847,647,97,621,411,69,146,336,46,653,39,10,109,694,604,535,5,189,768,399,446,637,50,441,2,755,419,125,298,608,43,314,423,13,649,158,706,463,398,100,67,818,508,38,12,37,20,826,95,354,264,23,32,169,166,132,271,105,96,221,674,83,51,932,429,356,54 }, +{ 20,470,147,43,904,376,798,427,325,595,142,178,858,198,319,0,97,173,414,22,259,61,422,819,77,945,202,14,32,5,362,49,426,392,107,207,887,725,660,481,263,465,234,104,450,534,494,861,95,28,18,33,120,10,612,64,344,105,13,30,324,39,21,455,817,115,46,146,102,31,132,4,246,356,318,65,158,66,260,40,7,48,523,581,9,35,159,315,113,729,23,230,125,347,112,38,128,878,359,59,36,60,12,239,600,182,237,192,153,342,196,683,98,16,170,2,306,298,357,54,279,101,70,560,501,345,199,845 }, +{ 51,23,13,12,99,453,33,202,128,64,153,102,15,196,37,403,141,40,120,117,475,38,134,115,14,137,77,401,139,10,0,165,318,269,82,308,21,2,5,197,54,3,180,260,22,31,494,421,17,86,93,1,95,44,111,457,277,237,402,450,9,110,18,6,98,719,125,65,351,32,28,217,49,342,326,328,730,629,352,386,285,361,221,11,50,36,19,341,48,569,177,41,59,8,194,27,24,89,79,523,29,775,4,961,365,7,670,291,144,373,210,114,46,317,391,34,116,509,757,203,236,108,30,26,817,55,498,57 }, +{ 383,136,17,62,84,440,119,504,56,240,548,3,80,129,378,123,106,128,555,205,4,184,507,59,162,11,14,183,57,361,249,38,202,469,152,64,179,495,483,90,32,215,257,214,245,130,209,54,161,27,250,503,328,229,618,211,5,73,402,45,12,306,318,40,34,514,790,76,452,77,65,26,86,297,141,19,53,497,742,269,814,283,37,33,255,311,194,436,667,114,177,61,217,803,95,207,266,2,50,432,769,757,236,102,21,527,36,556,46,22,10,72,115,41,494,611,381,29,196,99,216,210,151,248,445,137,0,576 }, +{ 70,652,48,5,638,846,1,21,163,340,888,767,221,125,761,349,260,562,269,303,518,658,591,586,13,698,623,23,883,774,98,87,49,31,9,371,296,223,51,763,850,863,120,541,360,346,141,634,64,535,39,30,2,0,845,7,622,156,318,195,44,95,202,900,32,12,931,379,6,558,768,293,271,115,777,401,58,354,818,61,55,17,132,668,450,308,385,914,351,919,93,922,391,457,128,22,590,212,291,77,807,762,758,252,159,101,663,352,538,372,100,446,75,520,18,937,284,96,146,626,326,127,118,116,771,522,463,69 }, +{ 66,135,74,97,278,6,172,69,324,140,267,808,462,14,550,204,7,357,138,87,189,85,298,75,29,2,500,52,335,38,25,92,582,802,174,571,572,841,563,42,380,1,24,314,21,350,16,451,67,46,280,220,501,443,150,620,230,103,486,423,35,353,622,60,83,226,71,192,32,889,26,22,65,903,95,5,48,232,330,64,108,86,565,792,641,580,36,448,70,509,197,222,43,348,558,426,241,726,125,262,0,877,893,50,872,81,309,63,233,950,37,749,18,388,149,145,12,91,237,10,355,606,294,246,114,508,673,28 }, +{ 51,23,33,102,13,128,40,196,202,77,64,65,139,141,99,12,14,453,117,153,475,403,134,10,86,137,4,11,22,0,318,269,115,401,36,114,108,21,38,5,120,180,421,165,402,59,28,15,95,82,210,37,2,177,494,31,342,9,260,217,277,523,386,3,308,450,6,457,730,17,133,18,719,24,46,361,1,352,49,194,143,111,54,50,351,197,300,94,44,237,341,164,365,221,389,67,32,48,283,19,485,291,29,236,27,854,93,961,758,110,326,817,317,285,125,275,90,52,45,34,26,7,391,181,57,8,752,670 }, +{ 88,264,435,100,166,274,772,921,478,382,909,639,438,960,521,96,897,410,30,1,466,190,530,778,265,321,682,691,418,144,724,329,686,39,9,447,93,31,914,131,404,203,44,411,765,335,593,475,795,367,98,625,254,664,120,213,613,403,0,744,539,116,83,656,363,238,5,48,13,576,77,812,805,110,23,21,141,966,69,125,616,829,326,286,201,82,35,520,225,51,821,127,604,285,284,165,260,217,33,36,310,731,340,842,223,102,137,101,615,64,831,252,228,49,10,67,224,557,219,443,692,406,272,268,910,506,197,351 }, +{ 440,62,136,548,504,56,84,3,555,383,507,129,361,128,123,162,4,17,59,119,184,14,202,183,152,161,402,495,328,57,205,514,527,229,215,80,27,257,38,11,378,73,757,497,472,240,211,130,503,452,803,54,469,77,53,34,494,381,318,283,266,250,106,12,102,33,245,210,432,64,403,19,40,880,236,90,5,556,179,21,412,377,2,32,200,300,99,196,269,585,153,488,141,51,275,216,45,37,117,249,529,194,214,372,76,862,177,134,133,313,26,421,0,139,471,41,305,65,248,483,13,114,678,23,217,50,72,775 }, +{ 911,617,959,332,206,253,244,141,282,120,110,384,951,143,352,260,269,10,160,373,717,531,811,417,940,55,728,661,533,36,434,450,93,137,559,203,473,498,642,116,180,326,855,261,295,44,268,638,922,854,752,31,499,164,318,351,201,535,144,851,564,341,365,64,8,115,165,197,843,1,217,317,704,864,187,699,567,127,202,873,237,251,22,258,574,824,98,86,968,860,489,23,817,82,125,21,169,509,926,219,13,276,15,601,28,345,9,372,51,901,349,943,871,5,367,291,776,308,131,391,96,523,105,83,386,902,338,35 }, +{ 120,260,450,141,77,15,1,23,351,326,137,64,373,82,203,33,13,291,51,21,217,523,269,177,102,128,922,165,110,197,854,5,318,457,180,352,48,32,125,144,93,817,160,202,386,96,50,7,372,36,349,304,210,2,342,730,115,898,719,16,0,196,247,494,59,10,9,473,22,4,403,116,18,86,67,341,752,615,95,75,901,317,275,704,24,475,453,437,367,208,164,163,143,37,90,899,244,98,12,3,365,968,551,300,237,206,55,35,29,6,801,805,678,576,401,393,391,288,268,216,194,99,72,61,58,49,39,242 }, +{ 15,0,1,2,5,51,9,23,7,13,6,901,12,3,8,180,4,141,82,115,14,10,341,120,197,33,93,730,21,40,125,401,260,202,102,22,110,64,569,719,457,165,37,961,898,453,18,351,31,28,203,16,137,352,217,450,77,326,11,403,629,308,128,196,32,30,36,269,44,98,116,318,365,99,475,373,670,237,24,899,854,523,391,95,421,49,48,317,177,221,817,153,55,117,752,39,86,144,38,642,105,922,242,291,54,143,17,134,494,498,244,59,661,96,473,65,160,509,139,170,247,89,114,90,638,20,704,482 }, +{ 104,7,97,107,356,560,66,232,707,359,298,568,289,324,712,392,192,38,14,426,173,65,95,64,172,387,22,43,92,32,357,362,214,20,259,526,90,0,330,380,126,246,501,49,5,86,893,74,500,194,874,147,135,612,230,10,204,779,52,2,729,808,550,140,35,835,21,28,314,77,312,60,16,465,132,715,279,423,414,600,885,802,683,18,81,24,36,13,636,861,150,33,178,29,819,519,288,4,31,58,660,448,121,534,12,168,61,48,581,278,70,37,146,102,441,23,46,199,142,6,751,249,76,319,202,189,124,237 }, +{ 6,66,1,2,74,67,29,14,24,69,83,135,25,458,52,150,108,451,75,38,103,380,267,278,7,87,26,42,324,36,357,138,602,114,97,19,85,238,594,273,65,149,398,335,172,46,160,204,70,18,711,550,50,16,309,220,71,732,96,696,5,133,603,12,891,86,174,28,280,500,418,467,226,32,22,21,222,189,27,60,124,423,486,547,895,302,733,236,532,388,208,64,63,0,35,17,13,298,95,262,45,906,336,646,582,37,145,194,571,20,531,112,23,58,140,516,484,48,886,218,118,115,99,81,462,429,54,55 }, +{ 51,23,13,453,12,115,202,21,403,401,5,4,457,719,475,37,99,165,32,308,64,3,1,2,523,10,730,0,128,17,197,15,141,494,11,22,14,6,269,961,7,318,120,386,77,899,18,247,421,450,817,48,16,117,59,90,33,125,217,9,19,153,180,629,26,758,237,24,177,28,221,95,57,341,40,25,38,54,569,391,27,260,402,29,216,317,36,72,291,102,56,134,365,45,76,49,468,304,352,137,210,203,34,89,196,342,82,275,194,349,437,46,123,326,50,20,8,670,482,86,509,139,498,316,53,61,576,288 }, +{ 57,59,77,4,381,11,412,369,210,291,421,283,803,53,250,542,377,14,339,471,275,108,99,670,19,300,453,403,23,27,401,368,40,51,202,361,775,21,12,554,236,183,402,813,33,342,38,378,475,503,46,37,444,10,736,390,152,90,56,561,285,153,134,123,117,650,73,899,523,281,0,529,494,63,386,880,432,162,514,128,58,605,102,817,743,690,495,349,277,256,196,137,111,79,18,9,801,235,139,78,867,391,200,119,758,130,42,26,480,389,719,714,467,341,80,269,122,730,221,216,188,177,161,133,114,45,29,22 }, +{ 15,1,5,2,0,7,23,6,13,51,9,3,180,115,12,457,4,8,14,141,901,719,197,33,21,82,961,365,22,120,165,341,31,351,64,202,308,77,37,401,260,117,16,153,28,11,110,93,730,10,32,44,269,137,217,569,18,98,102,49,128,317,453,125,30,36,99,523,48,352,134,752,450,196,386,318,326,40,95,24,758,898,17,116,629,39,373,203,86,72,899,237,403,59,177,35,873,90,89,55,291,105,817,144,734,475,244,139,421,65,498,76,247,494,20,38,854,349,160,253,509,372,96,482,221,216,615,143 }, +{ 64,237,180,95,312,197,146,32,90,247,316,929,352,126,165,288,317,132,304,13,60,490,77,217,955,522,92,314,86,22,4,23,7,5,21,626,40,14,11,12,652,498,291,177,232,164,199,33,10,482,51,141,115,150,633,462,579,340,216,16,529,128,37,360,638,388,121,359,143,468,275,59,107,49,534,125,202,658,509,269,888,267,1,598,581,588,76,17,467,451,296,28,751,660,441,349,194,29,31,71,698,279,253,45,640,57,48,546,134,117,18,2,830,647,372,102,65,437,493,351,98,733,622,423,273,241,309,137 }, +{ 1,23,13,51,120,15,202,180,5,21,260,141,0,326,2,137,7,77,197,450,269,33,165,318,9,128,352,110,48,6,102,523,12,115,342,317,4,32,16,386,14,403,373,203,817,36,18,217,64,96,37,457,93,752,82,28,341,494,24,615,3,22,291,351,401,8,475,10,308,453,421,391,901,221,196,59,719,730,854,30,177,90,160,365,402,125,35,498,31,285,99,39,11,244,237,72,20,899,44,40,704,86,17,922,83,50,210,349,569,576,98,95,67,76,372,509,49,143,116,253,961,65,144,89,55,551,661,29 }, +{ 14,46,2,16,24,75,69,35,220,52,7,38,1,140,267,29,71,50,309,60,87,92,70,171,189,67,18,54,163,262,335,108,230,58,366,103,150,5,675,236,181,112,223,278,409,537,21,83,294,48,155,68,12,486,149,37,571,582,96,0,246,334,135,186,443,45,328,28,32,133,641,97,451,151,388,64,307,126,25,9,502,760,394,281,357,22,65,66,774,36,10,55,172,114,61,6,565,462,461,251,552,563,400,74,820,330,238,218,600,72,99,86,606,314,17,380,430,227,587,433,20,13,23,583,516,545,121,115 }, +{ 127,98,31,44,299,276,284,9,935,293,116,599,49,0,456,201,492,28,39,105,144,371,662,22,159,125,272,137,242,131,737,489,393,567,268,395,51,1,761,578,367,202,158,30,109,918,916,685,219,254,203,616,23,434,36,21,55,141,295,95,13,24,16,187,93,385,128,18,96,948,48,269,763,160,318,64,5,165,86,101,244,326,115,518,35,286,836,261,37,512,7,10,68,327,623,716,384,120,403,253,688,110,333,224,2,890,457,143,100,197,12,386,252,301,320,258,963,310,65,14,237,924,221,170,494,421,83,699 }, +{ 53,573,445,297,436,73,6,27,151,113,26,250,491,19,614,145,611,200,315,112,45,396,881,138,80,42,58,70,34,684,25,631,17,222,287,106,74,472,122,61,624,714,690,235,5,90,32,65,255,64,648,29,14,66,95,85,322,2,209,243,549,311,644,22,1,52,665,280,47,161,86,76,87,194,312,651,3,667,133,416,75,307,114,128,78,62,137,51,57,4,226,141,50,136,46,862,483,49,289,884,119,440,632,814,12,21,176,237,148,56,236,283,174,36,135,63,123,337,10,453,33,412,84,469,214,59,108,23 }, +{ 90,788,804,32,115,693,494,81,84,56,725,403,119,21,18,681,99,77,202,526,65,240,214,576,64,817,91,0,784,350,241,102,397,342,690,848,104,61,24,945,318,69,825,453,421,365,210,134,123,117,33,20,707,402,60,293,263,207,194,114,49,888,370,97,43,673,671,523,726,326,125,510,879,773,718,715,701,462,283,249,180,153,59,57,51,11,887,835,814,111,23,4,889,468,289,247,221,95,89,86,66,120,719,325,192,899,730,683,595,422,757,486,484,426,351,312,306,302,285,275,260,238,216,203,196,177,174,165 }, +{ 24,7,14,149,71,16,18,108,65,69,28,154,189,35,2,140,124,92,68,150,118,175,429,302,0,66,273,67,336,38,55,97,251,46,60,36,547,1,267,484,22,458,230,29,83,6,9,388,86,650,172,314,95,109,114,52,135,21,64,886,602,74,505,10,232,5,20,12,596,823,705,398,32,103,49,101,420,570,236,531,289,533,50,298,168,169,48,133,462,451,380,23,882,246,283,51,799,13,43,25,185,797,194,847,75,357,278,158,37,439,346,594,330,733,399,423,610,164,192,535,701,104,345,467,143,443,125,137 }, +{ 68,0,101,9,212,65,124,35,154,189,69,149,114,230,67,39,252,100,1,336,702,329,16,265,520,83,103,191,264,755,64,589,88,562,28,705,30,24,283,167,118,18,777,732,478,12,21,723,71,166,23,49,175,133,398,335,596,55,664,218,237,429,389,532,818,604,364,50,158,115,484,289,246,352,238,410,141,22,759,419,323,375,345,36,13,10,806,128,51,547,796,269,213,5,48,722,140,247,321,96,565,480,164,180,251,330,803,553,37,676,95,143,202,86,75,490,443,7,936,724,99,2,190,694,346,197,782,917 }, +{ 5,2,186,61,29,52,45,58,311,17,171,80,227,209,48,155,1,106,243,496,50,46,334,454,592,21,513,14,502,255,869,666,151,663,262,223,75,207,38,32,270,667,62,6,27,738,659,119,56,3,807,54,780,287,378,84,721,74,220,290,96,163,483,72,263,89,67,549,103,281,136,37,583,76,430,181,536,7,70,408,12,394,64,787,73,675,148,495,469,69,161,400,409,36,833,86,65,34,19,474,587,742,767,221,26,324,128,4,309,115,24,83,306,425,25,22,431,10,241,760,90,141,764,541,133,240,95,383 }, +{ 15,1,13,2,5,3,4,0,341,7,6,23,10,11,14,12,9,33,51,17,165,153,197,365,569,180,77,40,102,8,115,117,21,901,64,217,317,22,141,16,128,32,457,196,31,134,28,36,291,18,177,139,137,237,719,961,37,57,120,111,352,349,386,59,202,453,629,86,752,19,758,38,93,873,269,44,391,372,49,143,95,342,730,30,475,498,509,260,210,318,125,110,304,55,27,29,403,82,48,24,277,72,482,275,116,494,25,450,98,54,351,76,90,326,41,79,373,99,899,546,523,194,144,474,94,56,203,39 }, +{ 13,23,15,51,0,1,10,4,2,5,77,9,115,3,12,14,22,102,11,197,33,457,7,961,165,453,403,40,59,6,21,475,494,18,719,64,180,28,202,569,32,401,31,37,17,16,8,210,177,341,523,120,141,386,730,49,217,99,44,318,57,450,308,36,629,421,125,93,153,342,90,117,24,111,498,139,269,237,30,196,817,291,134,317,326,98,95,365,899,260,352,19,48,82,275,391,72,38,89,110,509,482,86,873,216,76,349,148,247,79,27,54,116,474,65,203,901,615,55,45,351,128,758,137,304,752,94,50 }, +{ 15,13,0,1,23,2,10,102,5,9,33,7,4,3,6,77,51,115,12,11,341,217,196,40,14,117,153,18,317,569,22,8,457,365,523,16,342,180,197,165,120,719,453,961,59,134,730,21,64,141,260,31,901,401,291,899,386,28,139,82,210,177,349,269,49,24,482,202,17,372,817,44,450,93,629,111,32,125,351,326,300,352,37,318,110,128,30,36,475,95,579,468,57,308,19,137,38,98,247,437,304,275,20,65,752,237,498,391,494,758,615,90,86,373,854,116,277,99,72,96,48,421,203,45,216,105,898,89 }, +{ 166,39,274,190,438,310,30,333,530,100,252,656,625,539,265,404,778,88,101,455,9,0,557,960,593,329,254,131,1,676,382,709,724,158,553,840,731,228,98,31,669,23,68,219,264,48,21,748,957,812,406,49,224,127,44,13,344,36,272,286,51,159,520,419,105,96,116,125,613,83,5,93,795,124,213,28,863,22,223,35,478,10,816,352,293,69,67,829,16,827,258,616,364,160,64,242,639,578,120,167,128,65,518,217,321,154,32,137,187,604,180,4,679,435,118,18,251,212,144,86,95,759,141,77,410,793,371,37 }, +{ 734,94,148,308,79,111,413,468,89,37,431,115,179,401,569,197,629,341,188,13,873,474,490,365,72,12,11,134,205,245,317,117,4,247,358,59,40,152,509,51,482,901,961,76,99,528,23,180,10,498,217,216,477,339,77,139,102,38,165,55,749,421,196,743,579,143,33,542,428,391,875,752,471,497,54,1,452,153,177,21,488,588,3,64,141,349,453,485,670,277,291,457,2,128,5,57,403,36,475,529,184,540,202,48,377,90,7,372,14,50,405,210,437,15,758,402,464,386,352,300,157,6,17,0,45,794,9,342 }, +{ 539,228,224,219,190,816,840,871,30,669,530,957,258,748,863,406,93,187,778,531,160,213,88,466,839,36,131,137,116,202,728,679,926,120,44,352,39,321,373,318,274,276,851,31,574,442,860,144,730,217,201,203,613,251,856,55,559,533,944,141,435,1,450,242,230,22,127,86,110,64,800,28,49,326,473,9,260,752,268,254,166,418,717,567,456,98,96,776,902,385,968,100,719,180,269,616,317,918,351,83,827,492,10,625,931,494,125,831,908,197,21,365,372,82,371,35,77,716,854,795,689,295,704,5,332,345,244,291 }, +{ 801,393,125,791,386,730,23,551,51,21,144,116,48,203,98,221,175,165,949,93,575,31,126,13,958,326,697,601,421,345,788,293,367,22,963,541,110,455,60,608,670,273,197,44,463,4,32,677,623,127,33,259,834,35,948,453,375,133,107,120,77,40,223,0,407,342,68,719,121,43,14,446,621,590,523,373,308,102,49,16,899,1,815,475,199,770,725,191,493,804,96,36,319,141,101,92,39,146,690,71,55,7,118,38,805,352,649,735,615,602,533,531,480,323,279,246,208,201,196,180,160,105,64,743,589,132,117,598 }, +{ 13,23,77,51,115,141,33,217,4,2,177,6,0,102,40,7,64,11,10,14,3,1,180,269,22,202,9,59,16,15,17,352,318,128,18,197,291,247,32,165,196,153,117,134,317,19,5,90,341,125,237,24,137,210,36,28,304,349,403,49,12,95,86,25,275,372,453,365,31,308,752,216,342,20,143,139,475,93,27,98,498,57,26,401,21,120,386,391,457,326,8,569,316,494,421,111,65,901,203,110,105,30,260,468,288,164,437,241,45,48,29,35,99,44,482,719,82,402,509,144,37,244,55,546,56,285,961,293 }, +{ 2,52,29,70,112,7,58,14,75,87,46,220,307,186,171,163,151,1,334,262,135,278,66,5,69,155,294,97,38,140,74,16,45,172,287,549,92,227,400,24,849,226,331,113,61,324,25,48,852,32,35,21,37,6,461,67,409,50,19,12,644,315,267,145,26,243,223,311,27,64,60,54,232,122,71,18,83,17,571,583,189,86,36,853,103,72,65,394,380,587,780,774,309,20,43,22,357,42,96,314,298,128,209,335,536,95,474,767,675,270,337,76,80,89,330,222,0,90,721,290,230,10,137,108,246,126,28,448 }, +{ 321,88,213,435,466,100,689,382,30,230,1,795,352,410,96,217,699,921,190,831,521,530,443,36,83,686,144,418,840,264,438,854,128,251,539,957,44,141,180,557,752,335,69,223,317,960,31,265,48,137,855,116,934,201,98,748,160,669,909,93,166,203,812,731,403,21,274,351,593,39,778,206,367,772,219,64,228,131,455,202,9,284,5,372,35,326,238,127,23,735,82,51,656,285,67,158,310,125,639,800,221,187,224,332,269,120,365,576,494,13,897,252,345,822,478,765,349,702,77,268,421,242,208,110,318,859,101,237 }, +{ 363,520,411,354,101,195,668,132,447,9,156,905,364,664,121,146,360,23,18,759,781,1,98,765,271,51,96,676,252,264,13,777,5,31,586,621,591,478,223,167,848,21,100,691,936,551,33,393,857,70,48,883,125,682,77,16,39,687,948,821,44,367,116,93,24,407,265,127,593,834,110,49,842,522,296,4,293,166,626,812,865,64,404,203,333,102,538,212,36,638,40,55,446,50,647,118,291,656,897,144,639,60,771,697,128,403,379,67,35,692,541,165,463,163,604,453,658,199,590,69,791,286,7,32,12,317,217,724 }, +{ 51,23,13,453,401,475,12,403,457,730,719,21,5,308,99,115,2,32,11,4,3,421,961,1,165,15,629,899,0,670,37,10,6,17,197,9,33,18,22,523,569,64,40,202,28,19,56,26,7,57,27,16,72,48,196,194,24,59,494,177,153,45,36,139,77,386,210,128,117,65,758,34,8,391,341,817,468,217,54,29,450,275,221,76,757,509,350,141,134,90,89,46,38,291,180,880,237,102,304,216,775,734,546,402,361,277,137,130,129,67,41,25,14,247,148,62,901,498,477,283,262,260,133,123,74,42,801,365 }, +{ 72,76,89,205,179,90,245,528,148,54,184,37,12,431,38,308,152,4,257,59,474,316,428,401,464,57,460,749,32,288,358,405,229,23,477,645,743,99,11,21,51,115,497,77,217,1,794,79,157,14,5,629,540,488,453,216,45,449,17,155,2,221,770,165,48,670,421,569,328,266,50,452,197,341,181,3,128,769,10,365,137,64,13,317,587,241,36,457,734,188,247,0,339,313,7,536,350,471,9,202,94,468,524,509,721,482,141,46,129,102,773,697,33,432,40,180,270,237,281,619,741,381,86,361,961,543,476,720 }, +{ 376,43,147,470,20,595,173,107,858,97,319,142,427,819,414,178,729,362,426,325,5,259,7,32,861,392,61,798,246,14,104,198,230,560,885,324,77,21,422,904,356,0,465,126,357,48,707,234,38,330,22,49,207,35,501,612,66,13,202,95,534,359,298,2,10,577,76,28,16,70,113,64,132,4,17,481,318,715,102,60,58,18,784,59,120,40,72,11,90,29,500,23,74,279,879,146,33,1,112,660,387,56,52,45,315,81,232,9,494,177,140,36,115,37,153,600,57,199,217,31,12,121,347,51,403,878,671,164 }, +{ 202,269,141,494,318,51,137,403,128,77,453,4,475,217,678,349,421,5,342,352,96,102,291,177,372,317,64,341,180,21,2,40,10,59,32,752,33,11,61,3,515,120,304,36,0,365,6,31,13,22,7,39,30,95,125,260,237,12,717,275,115,28,23,98,244,482,268,105,947,851,402,805,203,194,144,954,576,450,351,48,859,289,283,86,65,55,49,44,795,197,82,90,498,312,285,241,201,182,165,160,57,56,14,411,293,284,9,822,326,899,523,931,921,897,863,854,839,821,735,691,668,579,569,442,360,251,214,207 }, +{ 77,202,318,102,494,33,128,269,23,13,51,196,153,177,117,342,134,291,403,0,141,137,64,15,349,82,22,351,203,49,275,615,373,21,551,393,473,217,95,859,31,210,4,475,402,1,352,65,11,10,450,120,32,180,86,285,36,18,2,7,3,453,12,260,6,28,98,14,125,39,421,48,90,5,372,40,194,59,44,165,817,16,9,115,326,341,523,17,678,216,30,197,317,116,576,96,127,237,61,93,300,24,365,124,70,401,105,730,214,160,114,304,289,110,329,20,704,163,35,386,208,899,144,692,244,67,50,719 }, +{ 253,951,352,110,811,499,244,854,10,638,642,752,559,180,143,141,564,8,260,341,498,417,922,661,901,197,206,165,911,373,137,93,282,873,351,384,15,959,55,332,120,762,717,728,269,160,898,82,473,64,326,164,36,365,203,22,704,533,317,509,31,479,86,968,116,318,115,202,531,1,268,217,33,535,28,44,9,617,237,13,434,4,261,23,77,940,125,5,372,11,851,864,40,32,338,21,102,0,569,386,30,96,482,349,169,824,421,51,105,546,12,401,242,758,576,391,35,127,98,170,2,117,144,844,295,457,24,291 }, +{ 141,23,13,5,4,269,352,51,180,21,317,115,202,217,1,197,2,11,365,318,32,341,3,6,372,137,7,498,77,64,349,291,48,752,33,0,165,237,9,15,22,12,901,221,253,16,144,244,143,403,93,28,401,30,453,110,475,24,304,40,39,120,125,494,421,203,308,72,160,569,18,386,177,35,102,482,31,14,260,44,10,36,391,95,473,373,153,49,468,864,247,117,873,128,100,88,59,134,69,546,241,76,717,89,86,196,17,37,437,166,509,351,164,98,854,67,8,90,704,629,82,851,450,96,811,367,326,284 }, +{ 252,101,18,9,0,68,16,118,329,35,364,24,39,167,60,333,265,109,158,553,71,212,677,28,419,604,100,793,49,286,159,154,191,105,55,310,272,694,327,455,124,810,806,48,22,166,254,593,21,7,374,344,1,520,146,121,538,5,917,65,346,706,857,634,92,562,23,627,404,126,925,13,98,185,51,108,30,14,907,271,169,709,31,942,95,10,446,96,149,628,557,125,375,170,175,67,688,199,363,132,639,239,190,656,107,621,320,12,912,638,137,36,150,44,815,302,50,463,865,221,438,32,37,578,671,439,848,131 }, +{ 28,109,158,9,168,49,39,175,35,159,22,0,55,185,65,289,272,67,95,194,114,346,333,577,169,584,254,1,344,327,105,570,286,336,24,83,86,30,747,68,627,374,124,505,69,101,810,610,837,439,36,154,535,133,103,7,310,16,14,835,131,756,601,345,2,429,874,149,389,50,799,793,252,347,283,208,547,51,251,46,23,100,128,10,189,312,164,424,912,21,484,5,64,688,637,12,190,578,455,96,239,420,13,518,143,634,137,419,533,364,18,398,166,48,709,320,218,538,127,160,170,230,323,125,71,37,98,202 }, +{ 15,13,1,5,2,3,4,341,11,23,0,33,6,51,7,12,40,115,10,9,77,102,14,197,365,180,17,901,21,217,317,196,165,32,569,22,141,8,64,961,453,457,16,153,59,117,18,352,31,202,125,37,139,291,401,629,28,386,269,134,719,177,475,237,93,318,120,730,44,48,372,110,24,111,899,36,210,403,19,57,523,391,349,752,326,49,38,128,308,342,260,494,498,116,95,509,98,137,30,873,304,758,90,801,27,56,86,203,615,144,275,45,482,450,65,670,373,54,41,50,734,94,143,25,82,244,105,79 }, +{ 173,693,18,104,422,20,102,0,77,49,61,5,784,560,526,707,226,32,13,33,397,64,193,344,134,117,91,66,21,153,715,263,81,95,172,232,207,196,31,11,6,65,510,42,25,414,197,180,135,48,29,22,10,43,72,819,4,470,324,40,85,546,86,887,347,51,319,204,57,28,27,426,359,283,498,376,138,128,90,194,392,643,879,701,474,87,74,59,52,38,9,387,681,7,862,597,192,814,847,718,713,334,291,270,217,165,125,83,37,56,23,202,177,141,122,825,241,509,330,942,817,523,370,269,115,98,84,44 }, +{ 23,51,386,308,13,401,221,421,801,949,719,165,468,730,21,770,341,125,697,453,867,14,115,670,457,6,391,403,1,475,958,523,450,117,102,197,629,365,33,569,607,7,2,758,120,66,899,515,743,134,217,791,342,97,16,326,67,733,74,350,150,773,749,260,202,52,29,18,9,501,456,805,809,734,482,56,0,317,135,4,906,458,153,3,380,103,645,579,558,490,24,93,83,388,196,108,817,11,99,903,685,177,69,32,561,267,236,335,5,180,594,696,554,451,642,576,564,559,546,349,291,238,122,89,87,76,75,73 }, +{ 49,0,24,16,105,159,28,9,35,158,68,320,18,109,239,55,65,170,124,7,1,154,301,347,71,95,14,286,518,101,2,344,327,272,127,118,39,21,60,374,890,98,577,623,12,149,22,5,916,338,512,168,36,713,395,23,31,67,51,371,13,48,185,175,125,137,10,424,169,64,202,737,479,30,747,538,6,761,37,108,128,346,242,32,740,293,131,46,50,302,654,69,44,252,419,289,810,86,114,92,83,616,141,385,846,299,164,254,364,506,403,677,310,671,375,191,221,116,938,318,634,100,165,439,115,143,333,96 }, +{ 2,5,29,45,207,58,61,1,76,292,32,655,263,14,476,52,72,6,119,17,306,70,186,7,106,46,179,214,790,90,148,243,720,38,227,50,240,112,151,710,255,592,84,89,64,48,25,464,75,241,209,21,129,74,155,460,358,3,128,454,496,425,36,77,19,22,141,738,10,115,56,26,86,16,123,66,408,311,27,245,0,249,95,65,205,287,519,290,618,40,24,177,549,171,202,87,4,59,163,307,54,334,237,51,23,62,269,12,102,37,18,449,869,137,80,69,113,28,99,33,67,13,135,180,318,315,769,288 }, +{ 125,21,115,95,715,197,49,165,64,56,66,32,6,278,13,33,713,681,808,350,204,237,120,408,263,241,172,135,81,22,324,87,509,74,0,85,97,4,90,86,468,65,888,353,174,558,391,841,192,180,23,950,872,645,571,554,650,580,141,5,2,412,283,467,341,673,59,138,38,108,903,177,91,51,7,511,370,164,75,31,620,475,203,893,170,98,712,104,44,19,885,10,1,561,426,415,308,63,403,196,153,117,102,84,389,128,27,825,569,210,77,347,53,123,386,316,159,143,835,194,900,448,93,11,546,232,18,572 }, +{ 612,325,427,202,107,494,5,403,470,147,518,376,64,362,49,21,392,788,465,98,14,342,125,623,318,178,48,31,845,804,534,858,423,70,690,595,414,278,93,422,269,453,7,660,102,77,33,693,61,953,767,798,819,581,307,298,234,198,116,110,97,39,32,30,10,237,4,671,663,389,289,194,128,43,0,20,904,725,761,711,678,509,501,493,366,319,236,232,220,174,104,99,95,88,87,68,55,44,38,37,18,9,900,293,13,715,441,315,105,291,163,141,391,112,848,397,22,763,713,558,846,835,828,822,757,657,557,546 }, +{ 32,76,21,72,14,241,449,50,292,2,350,5,1,155,48,221,46,38,89,179,270,45,476,90,54,720,214,430,148,464,17,281,129,460,37,205,413,741,181,36,245,519,29,227,52,290,257,184,405,12,123,710,3,528,162,7,773,243,541,10,358,0,288,431,67,57,249,536,316,152,99,96,23,106,171,51,513,4,743,308,618,128,137,6,115,64,474,477,103,408,86,207,769,229,263,9,328,83,133,180,59,183,119,552,794,787,540,114,217,488,425,218,62,11,421,659,77,645,760,721,22,524,202,58,400,401,141,770 }, +{ 24,154,149,28,124,16,35,0,71,7,14,68,175,108,65,18,55,92,109,302,118,9,336,570,484,398,69,140,60,2,420,67,189,22,101,46,150,185,158,346,230,1,439,169,252,273,168,419,289,39,49,38,505,251,83,345,701,535,799,610,637,267,21,36,97,10,650,50,95,12,429,246,103,533,114,388,458,133,5,48,399,314,86,6,671,375,159,66,364,29,236,329,962,703,105,547,23,51,172,601,333,283,64,265,886,126,96,194,52,13,232,722,32,20,389,54,37,677,164,100,143,125,327,602,75,917,344,286 }, +{ 5,14,1,50,48,2,46,38,67,54,45,103,24,270,21,96,513,155,290,83,58,61,281,69,32,36,181,72,502,0,29,171,17,12,16,7,76,263,37,18,28,52,227,35,425,207,89,666,22,221,243,133,541,663,65,10,741,218,408,223,114,536,108,474,75,9,186,6,592,552,51,23,99,238,328,115,309,794,540,3,433,834,86,292,760,477,148,449,428,64,476,128,400,528,119,655,55,236,241,106,202,141,765,766,487,70,13,163,137,366,95,71,587,697,488,720,179,160,19,418,807,308,659,149,27,90,431,4 }, +{ 961,13,569,37,197,23,474,94,148,79,12,629,401,111,901,341,15,51,165,405,734,873,99,4,413,188,134,89,10,11,358,152,179,117,308,115,40,59,72,468,245,139,457,33,153,247,365,477,205,277,339,317,77,758,5,38,498,76,579,453,216,180,0,3,542,762,36,490,2,177,102,143,482,217,196,471,14,452,128,437,55,1,54,304,22,210,9,377,64,638,202,17,21,488,349,431,300,775,528,485,32,7,141,318,372,86,8,509,157,45,291,497,137,184,6,864,402,670,275,880,403,110,543,328,529,875,90,657 }, +{ 102,33,51,77,23,40,128,202,64,13,141,139,196,318,403,153,269,117,99,137,22,4,453,134,59,115,12,10,11,475,0,14,86,65,95,180,15,2,120,1,210,494,342,28,18,165,352,6,36,49,177,217,300,260,82,719,401,421,24,402,386,21,5,457,7,197,730,3,125,37,450,19,291,48,523,17,351,237,27,114,326,93,50,111,38,341,317,25,285,98,26,277,110,54,194,46,9,391,361,275,16,44,31,961,32,308,96,349,899,34,133,854,569,29,485,757,108,105,283,67,144,373,57,509,678,498,221,116 }, +{ 24,35,1,67,0,28,18,16,103,9,68,69,50,65,83,149,12,5,71,133,14,7,22,114,75,96,2,55,189,218,108,48,21,54,223,175,281,251,181,23,46,58,37,124,101,51,336,238,36,70,13,154,118,163,418,10,99,60,745,766,137,537,39,128,38,335,202,141,487,49,361,64,32,45,115,486,155,765,283,109,502,345,230,140,87,430,171,29,269,398,30,267,95,221,543,92,86,820,236,217,6,412,285,17,160,302,77,177,596,61,318,180,458,488,31,52,540,165,352,4,237,105,920,545,125,657,705,273 }, +{ 384,617,940,911,959,855,332,206,282,434,244,253,141,268,352,93,120,144,10,269,137,110,160,201,417,36,143,450,116,203,260,44,811,31,717,951,55,295,318,64,187,261,373,567,202,276,489,258,326,498,98,642,559,180,473,127,22,728,860,217,1,661,219,499,86,23,851,638,8,197,692,341,13,531,165,224,351,523,817,115,9,854,164,125,752,365,28,533,367,51,922,15,564,131,21,96,317,128,82,5,662,699,843,105,871,372,615,237,0,83,386,30,864,509,242,800,901,284,291,551,77,2,776,349,7,704,37,574 }, +{ 93,30,203,473,326,704,137,652,450,31,800,269,848,160,127,190,274,144,254,219,187,625,349,303,166,352,120,372,116,521,224,110,131,752,435,88,217,202,141,260,291,77,367,156,340,228,822,744,5,382,197,296,658,438,180,863,410,44,730,698,578,317,418,1,859,165,850,829,735,494,102,795,616,159,264,921,98,960,373,36,100,23,13,341,128,21,64,258,10,0,523,330,318,39,717,321,778,709,7,968,125,919,942,177,49,86,158,9,365,225,901,478,105,686,531,33,924,888,851,96,2,363,371,442,385,747,691,615 }, +{ 53,73,27,250,26,200,19,297,630,714,472,611,42,122,235,25,17,34,6,436,65,684,106,680,690,80,14,256,113,322,57,813,85,108,64,45,59,648,283,4,161,194,573,445,236,389,128,862,444,114,624,390,469,78,56,51,141,133,2,90,22,84,11,176,491,0,18,255,495,453,416,32,3,412,23,47,145,137,63,29,665,123,95,269,46,378,62,61,614,483,86,361,814,10,5,605,136,396,216,177,112,50,249,619,730,77,129,119,503,13,432,202,179,12,719,285,318,24,597,102,151,183,76,21,130,742,37,209 }, +{ 15,2,1,5,7,3,0,6,141,180,10,4,13,33,11,352,77,197,9,217,14,134,64,12,341,317,365,165,752,40,498,372,269,102,23,17,115,117,291,8,349,22,51,153,196,59,32,237,16,901,36,120,21,260,202,342,961,128,569,143,82,57,864,37,45,318,873,110,93,111,457,18,72,391,386,76,28,86,351,125,137,509,95,31,56,954,326,450,79,48,27,854,139,29,482,19,38,44,546,89,373,99,49,244,90,55,84,719,523,94,403,35,177,54,579,453,277,24,164,922,41,253,30,203,730,61,20,58 }, +{ 254,530,960,613,39,829,688,221,406,190,166,475,827,438,30,9,669,269,180,23,98,51,31,228,21,541,202,274,141,840,318,131,938,105,48,158,127,125,96,44,846,137,453,49,393,219,352,116,13,1,679,421,856,625,286,317,36,308,100,333,310,403,115,890,144,217,791,101,931,616,569,10,386,551,401,0,876,272,910,55,258,88,102,77,327,656,224,629,252,64,468,731,160,385,860,518,926,28,697,344,326,159,33,5,197,93,944,627,128,165,816,293,574,455,187,765,371,364,843,404,242,120,22,284,95,32,345,506 }, +{ 39,9,166,28,158,35,101,30,175,364,68,168,190,0,67,346,194,100,49,88,131,36,333,420,274,310,570,344,404,252,83,289,706,109,55,69,208,1,86,160,137,610,96,65,967,251,438,352,637,336,114,16,577,159,128,254,756,154,345,219,22,64,230,799,228,810,95,24,51,747,224,185,141,818,149,584,768,202,103,419,429,21,826,752,124,530,857,31,656,10,13,169,323,505,237,143,865,562,189,165,793,238,694,535,23,217,874,455,164,116,105,7,197,127,44,547,125,286,180,213,868,539,71,2,703,272,498,115 }, +{ 62,555,548,56,3,507,161,440,34,215,527,136,514,162,361,4,211,17,130,328,383,757,14,84,183,123,205,152,494,11,184,57,585,38,236,202,377,59,54,504,452,129,402,128,176,556,229,880,80,372,300,381,495,40,33,378,37,469,106,266,472,119,12,214,497,248,102,803,403,342,249,73,53,77,245,412,313,51,196,27,153,543,257,485,210,471,76,283,488,99,305,200,421,13,139,157,23,349,432,79,45,90,339,32,117,240,46,179,714,457,292,108,134,250,719,318,41,217,64,21,22,369,710,188,72,133,277,453 }, +{ 126,107,92,356,493,279,359,7,246,97,298,362,392,441,508,60,199,35,423,16,146,132,121,14,43,230,501,232,939,20,649,150,173,5,779,660,414,607,672,314,590,259,534,21,66,140,729,683,387,191,18,71,941,955,953,32,104,49,48,22,38,0,46,319,640,95,819,172,147,167,24,357,28,10,2,13,858,603,324,195,68,598,12,1,345,595,426,61,64,653,522,407,612,470,4,885,77,23,52,236,125,37,33,702,31,687,581,6,118,74,98,711,399,212,40,75,70,376,465,517,360,135,271,105,102,50,192,156 }, +{ 291,141,349,373,372,82,15,260,351,120,922,854,326,217,318,77,450,1,352,898,93,269,523,102,202,21,203,144,13,64,23,304,5,752,137,437,32,51,473,98,704,247,180,177,317,237,197,342,59,457,947,817,494,316,76,33,719,615,300,961,12,36,968,954,475,386,367,268,165,115,86,83,69,67,48,39,22,2,851,341,275,210,201,110,662,576,418,403,391,365,284,241,194,116,105,96,88,54,44,31,7,730,485,205,125,40,117,875,873,864,661,579,498,490,482,468,393,332,308,244,221,190,185,169,160,128,109,100 }, +{ 141,217,352,115,269,180,317,752,341,244,13,291,21,77,23,197,365,5,372,165,51,64,237,349,48,9,4,304,482,102,32,864,28,318,247,137,11,202,308,33,39,98,7,3,1,498,351,125,2,468,437,110,10,82,490,253,120,100,88,954,515,177,93,901,31,12,401,201,0,49,922,72,203,569,55,30,6,342,260,143,22,117,898,717,44,89,35,153,144,854,128,36,386,350,101,68,196,40,373,221,134,96,206,873,391,316,288,241,164,160,105,16,90,579,403,326,645,629,523,421,284,116,65,61,961,59,24,18 }, +{ 78,47,390,108,130,711,813,19,453,412,27,283,730,444,690,196,250,123,236,128,389,503,200,650,183,14,59,4,26,65,472,133,73,34,381,297,153,51,177,57,269,11,53,318,33,216,719,17,194,630,680,322,619,114,113,368,714,46,684,803,432,56,77,29,657,6,480,141,117,3,445,22,603,122,42,235,84,361,90,899,13,64,23,102,249,2,754,524,605,457,99,63,0,475,349,789,40,38,588,210,18,152,106,134,678,436,24,10,25,597,50,529,291,594,401,52,245,95,402,894,129,266,86,471,339,119,523,137 }, +{ 250,297,34,80,472,495,311,64,53,17,667,61,148,27,56,483,469,136,90,73,141,161,45,243,194,3,106,597,527,176,128,751,352,151,436,62,86,874,738,4,209,153,833,491,881,614,396,32,383,84,26,76,445,231,95,5,659,14,887,237,29,378,573,431,947,196,405,58,23,255,119,117,65,11,113,59,33,719,134,814,292,287,57,177,721,240,12,585,611,247,269,51,22,283,200,498,2,583,304,180,19,145,312,165,678,205,202,10,835,349,72,217,114,514,40,555,315,588,137,130,37,875,389,197,207,179,0,13 }, +{ 51,23,551,421,453,13,33,102,77,21,326,153,401,958,308,120,615,196,5,48,202,221,115,692,523,791,403,899,342,4,801,475,210,805,260,107,450,291,966,93,730,576,32,117,842,541,468,18,1,16,203,116,110,697,60,96,126,121,386,68,31,40,2,494,259,125,463,407,137,154,24,704,427,373,218,180,92,10,575,244,35,104,64,175,302,628,351,318,197,160,98,71,817,770,743,670,612,12,859,158,17,590,199,650,7,945,815,752,346,949,851,717,642,597,473,439,402,367,253,252,217,146,56,44,6,3,36,847 }, +{ 98,223,393,363,411,834,478,664,1,691,791,284,156,447,724,914,759,697,293,354,541,807,51,586,421,48,948,668,765,848,264,195,9,163,658,23,801,18,125,551,21,386,96,340,772,909,682,963,730,698,31,453,360,13,371,949,905,770,520,5,621,781,221,744,271,296,116,202,647,857,132,70,118,335,821,44,656,842,367,897,719,121,167,446,32,69,457,450,318,101,100,670,379,638,127,634,144,137,410,761,662,217,846,758,146,401,932,615,75,777,883,538,87,165,33,225,936,591,286,93,166,269,197,812,326,141,105,919 }, +{ 232,7,107,92,14,46,135,172,359,220,314,278,60,5,171,192,2,526,16,620,21,262,52,75,35,32,126,294,29,872,24,132,18,77,246,87,448,571,230,48,1,54,12,58,38,236,50,606,247,86,23,70,177,112,461,298,121,380,97,71,37,888,217,392,66,279,150,356,155,64,13,20,441,108,678,304,751,572,147,226,115,334,140,61,291,785,153,739,579,43,451,178,146,186,947,437,387,394,90,51,330,195,194,875,307,216,352,597,587,357,117,0,269,181,45,603,349,210,199,137,415,128,267,95,366,315,156,31 }, +{ 32,76,2,241,21,14,292,1,476,72,48,5,45,720,50,179,270,214,46,89,90,29,148,129,464,155,17,290,519,38,227,460,205,123,245,350,243,358,52,184,54,257,171,710,249,408,7,400,186,221,405,513,3,36,162,618,12,106,37,288,181,57,502,413,0,263,10,428,316,99,281,207,6,229,4,58,152,64,115,425,430,769,67,741,119,59,22,128,659,308,773,183,23,202,96,86,431,83,75,62,141,9,524,51,16,103,328,645,69,237,497,255,209,217,130,11,24,496,666,28,137,403,528,18,247,536,452,77 }, +{ 15,77,33,102,13,23,51,117,4,32,11,202,40,90,0,1,5,141,153,247,177,128,134,214,288,21,137,269,17,453,3,318,291,403,8,110,82,180,10,59,165,494,22,6,342,197,457,351,64,475,260,9,93,2,120,523,326,48,31,217,115,57,421,386,12,373,44,196,16,19,719,730,25,139,96,352,341,36,961,49,210,317,349,203,125,194,401,18,275,7,24,28,98,817,285,898,365,95,308,901,450,30,99,661,372,86,223,65,854,615,922,402,56,899,697,758,208,103,14,237,116,105,569,37,304,221,704,638 }, +{ 66,74,135,69,6,380,278,267,97,14,2,29,1,75,24,87,324,52,150,38,335,138,357,189,204,172,67,103,25,388,85,7,462,220,83,423,108,550,149,71,603,42,733,571,696,174,26,582,140,36,458,565,16,808,895,802,486,46,238,35,19,70,298,21,18,114,65,226,22,86,60,418,32,251,192,92,64,451,280,273,500,309,95,594,222,501,12,50,5,145,563,398,353,0,889,28,443,48,607,711,641,517,112,262,336,348,37,125,580,596,906,705,622,572,10,160,81,43,63,54,58,133,792,68,620,124,165,294 }, +{ 152,471,4,339,59,188,79,77,11,377,128,94,529,102,33,202,402,111,12,99,51,216,542,452,291,40,474,342,210,196,13,141,184,64,403,37,453,153,134,318,17,177,117,494,23,961,139,5,32,76,431,3,300,257,205,757,497,115,401,275,269,10,229,179,72,475,21,2,86,90,14,57,89,349,1,0,180,157,285,277,148,629,137,266,197,38,36,304,45,217,485,6,498,678,129,405,700,95,569,245,437,734,247,460,22,413,313,352,19,372,775,308,421,292,880,48,237,143,769,503,358,873,488,54,7,618,710,386 }, +{ 15,1,196,0,33,13,2,77,5,10,9,102,217,23,7,6,3,12,117,14,4,134,177,153,40,11,457,291,210,275,22,730,16,115,18,165,21,386,341,719,120,82,180,64,197,51,141,523,317,8,569,365,450,202,32,961,372,260,28,349,342,125,352,95,37,31,899,351,17,128,318,269,48,110,49,403,19,93,901,59,475,401,72,326,24,139,36,65,111,61,237,758,494,90,99,30,854,304,817,44,482,137,391,56,509,898,437,25,308,752,98,38,629,247,76,922,421,86,57,373,670,26,253,70,203,221,801,579 }, +{ 209,106,17,207,45,255,119,655,5,742,306,378,61,62,136,425,263,80,311,84,240,29,790,2,58,56,186,32,3,383,738,243,454,592,869,76,151,14,179,129,290,123,48,72,440,148,21,52,292,64,90,504,287,667,214,46,1,128,54,408,507,50,476,34,27,833,549,57,38,548,155,227,162,86,241,19,70,152,89,205,6,334,171,26,496,483,112,183,181,141,130,400,249,59,245,266,184,555,663,75,270,358,77,229,361,37,215,95,445,4,297,10,211,25,33,666,519,881,11,257,202,73,710,269,237,40,221,180 }, +{ 120,450,260,817,523,15,23,1,51,326,98,13,202,82,125,64,5,351,318,21,31,269,457,32,116,93,115,165,102,77,33,12,95,922,719,30,494,110,44,615,128,48,14,180,141,36,2,730,61,9,137,342,403,899,197,86,49,10,704,551,386,237,8,203,96,83,67,37,7,177,291,352,90,22,18,16,402,196,153,373,179,144,76,39,17,6,961,854,475,898,692,401,241,207,148,105,100,65,50,20,453,349,344,3,217,901,678,341,293,263,247,210,99,88,69,45,40,11,958,801,317,132,126,107,35,24,752,661 }, +{ 116,268,492,918,93,206,203,940,23,51,551,417,523,499,473,959,22,421,8,125,791,692,730,120,31,401,949,10,0,13,899,165,44,386,450,260,867,453,64,9,629,115,36,719,559,341,728,170,55,137,599,299,197,615,569,180,457,28,391,141,1,326,685,301,564,479,475,338,506,86,509,105,638,202,95,817,308,242,253,762,244,2,373,801,670,127,901,21,49,870,951,5,558,237,143,269,37,752,239,12,642,922,403,873,7,352,4,758,365,318,128,18,965,456,32,320,221,14,805,512,317,498,961,16,704,98,33,898 }, +{ 15,13,165,23,197,4,0,1,180,12,11,117,5,8,82,901,719,341,2,120,51,9,523,260,509,7,3,730,141,386,10,450,351,6,33,752,758,134,115,498,352,961,317,365,153,898,14,899,854,64,217,922,873,22,569,326,93,128,817,31,269,237,291,202,21,110,391,37,801,615,28,318,17,457,44,16,77,18,32,98,99,36,49,30,349,401,453,403,102,38,373,958,35,629,116,40,137,125,55,19,482,203,304,242,86,546,474,96,24,144,65,308,475,139,143,247,210,402,89,54,69,177,196,48,485,661,83,734 }, +{ 14,24,69,66,7,108,2,398,267,150,67,97,189,29,149,74,38,71,36,6,83,52,273,388,140,1,135,18,46,172,602,65,28,124,429,302,154,380,336,451,547,16,278,114,732,68,484,324,75,733,92,175,103,118,22,462,0,25,251,357,86,596,55,160,87,705,594,35,60,531,133,467,423,711,236,95,650,109,458,443,298,21,886,5,50,138,26,12,505,238,168,309,314,42,32,10,646,220,49,70,335,192,232,906,420,204,48,20,64,37,43,603,13,19,533,96,85,23,565,51,330,226,550,221,570,208,283,54 }, +{ 330,335,96,839,523,662,367,443,615,418,141,403,717,244,692,576,922,372,1,352,269,217,752,349,64,498,36,854,966,180,822,831,954,873,864,317,284,291,776,201,318,144,69,98,351,678,7,21,5,160,77,203,83,82,442,137,551,2,260,223,622,187,494,44,13,385,128,202,23,475,75,341,901,48,31,437,238,304,247,230,765,93,373,120,393,22,67,251,371,293,125,661,342,859,285,898,163,365,30,32,450,29,110,459,326,934,51,10,197,842,851,399,246,189,800,421,237,0,37,116,221,165,16,579,127,9,321,88 }, +{ 184,257,205,90,245,229,769,524,152,432,497,619,17,57,5,452,32,45,381,266,106,2,59,21,179,4,292,217,313,460,288,1,214,129,14,76,77,3,128,216,48,207,471,339,157,141,710,11,202,38,64,519,369,6,61,209,12,37,119,79,99,123,29,317,72,188,7,62,255,33,58,162,269,247,40,54,102,115,177,51,618,372,476,318,10,89,137,86,249,148,241,263,180,316,352,402,50,403,655,155,19,210,23,94,136,291,308,349,358,237,453,377,111,56,139,425,365,27,341,46,143,277,401,153,588,221,0,186 }, +{ 7,14,16,46,107,32,35,24,70,60,23,307,172,5,18,2,38,87,92,58,13,21,20,10,177,51,202,77,135,48,37,128,0,97,232,115,171,112,126,43,9,220,140,318,12,141,102,71,852,155,4,54,291,22,90,1,269,52,163,33,137,298,192,28,66,64,217,356,61,132,294,278,359,147,349,461,95,29,45,609,448,121,259,186,314,76,474,304,362,17,108,6,150,415,59,400,780,146,36,342,31,275,67,86,246,402,334,279,678,75,247,50,620,30,409,502,39,372,72,49,230,40,494,227,69,465,872,68 }, +{ 140,66,97,172,92,7,267,189,14,314,16,69,298,380,60,135,451,230,150,35,24,71,74,357,423,232,330,38,324,246,192,443,278,563,2,572,877,550,462,126,448,52,893,508,67,501,6,568,29,21,603,46,565,607,83,388,204,633,22,279,18,149,723,733,399,65,48,1,95,64,526,915,5,32,726,28,75,36,841,25,350,87,138,108,68,500,895,10,43,0,251,808,125,104,12,426,49,81,50,20,37,309,517,674,86,802,174,107,103,653,173,582,220,226,85,387,26,55,712,903,241,197,121,237,262,54,711,191 }, +{ 121,167,354,212,132,407,191,107,147,146,271,199,101,446,126,647,195,35,259,18,16,60,55,20,590,465,43,411,68,363,178,198,142,562,9,427,463,92,640,14,346,7,727,798,941,520,364,325,815,5,581,13,1,621,21,23,522,534,0,589,279,158,28,806,706,125,439,109,98,97,118,48,22,694,955,393,604,362,150,932,608,51,105,33,252,165,234,265,108,49,598,156,96,419,818,10,467,197,154,493,12,765,376,120,70,786,169,777,967,102,687,38,202,32,682,246,64,39,31,447,781,612,58,159,61,260,40,450 }, +{ 76,90,460,205,179,257,245,32,288,184,21,45,769,358,316,524,229,241,152,5,57,48,497,2,476,148,720,619,452,266,72,155,1,17,38,432,14,54,243,290,292,227,449,381,12,4,89,247,50,29,313,645,270,59,350,11,129,216,51,217,106,157,221,128,99,58,37,471,64,188,77,214,408,339,141,115,186,3,369,23,52,202,79,46,428,425,308,6,317,464,7,255,405,162,413,33,171,519,496,372,365,341,269,401,10,0,94,36,137,875,318,263,237,62,588,352,123,474,181,431,40,177,291,180,102,61,207,618 }, +{ 147,259,427,178,798,581,878,786,198,534,465,325,929,142,362,20,376,43,107,771,35,481,132,470,234,5,626,146,590,279,97,522,49,70,77,345,21,199,640,450,612,202,13,48,14,102,150,0,33,725,595,846,941,23,22,7,58,195,423,660,501,121,687,126,518,260,117,904,120,858,953,39,342,60,455,1,196,165,125,158,105,159,246,92,28,153,10,31,360,315,346,271,761,683,603,212,18,298,128,711,167,379,318,113,623,817,95,32,291,61,16,134,72,359,108,221,197,115,98,163,767,523,493,191,329,30,87,50 }, +{ 473,778,450,203,855,144,93,617,30,704,141,466,530,330,372,459,201,326,934,321,88,839,254,36,505,403,128,559,83,120,523,217,86,213,131,613,717,110,332,728,344,251,264,352,686,434,82,39,578,317,206,35,625,435,44,795,840,961,539,274,244,116,137,55,662,265,228,863,577,406,96,924,601,564,417,384,351,341,225,95,49,28,800,418,21,831,719,531,367,219,202,31,64,822,224,335,960,968,922,854,829,688,365,197,22,160,410,165,873,752,716,518,456,347,242,125,731,269,180,948,498,478,349,237,98,77,33,9 }, +{ 523,475,899,730,453,51,23,403,719,13,421,457,33,386,196,817,801,4,120,117,153,11,450,260,102,202,515,361,670,165,758,494,125,108,128,961,283,236,99,21,3,880,5,401,32,59,77,543,615,10,6,318,402,221,183,137,650,57,40,7,2,1,90,847,412,247,241,210,207,115,12,208,181,150,133,114,63,42,467,775,389,342,111,56,39,31,30,28,22,9,0,576,558,304,229,216,177,152,123,113,47,29,25,958,554,391,629,968,967,966,965,964,963,962,960,959,957,956,955,954,953,952,951,950,949,948,947,946 }, +{ 1,5,15,2,13,0,7,23,10,3,317,341,6,9,4,365,11,115,33,180,217,77,17,64,197,165,102,117,141,40,14,21,51,32,134,12,8,291,901,752,16,569,22,509,352,237,482,372,143,18,498,153,457,59,128,349,269,125,202,177,961,546,31,391,342,196,45,36,873,57,38,28,37,86,48,386,24,19,139,318,111,95,29,44,403,27,719,304,120,41,110,629,93,72,164,275,61,55,253,99,56,90,49,137,730,210,864,758,35,260,76,50,579,116,94,300,221,26,25,277,326,98,54,475,453,30,79,244 }, +{ 19,119,40,4,27,84,77,56,202,73,102,59,757,33,485,26,300,210,504,494,128,80,862,23,402,63,42,139,880,342,129,32,5,361,25,13,196,51,12,11,2,99,21,6,153,469,123,10,710,53,65,618,64,161,117,736,90,38,22,106,17,803,95,76,0,45,37,14,548,813,250,245,108,297,453,378,162,57,113,460,403,611,255,205,256,383,217,1,235,114,86,742,209,152,292,630,576,240,343,790,667,475,16,48,133,122,18,136,134,85,115,15,283,34,179,436,180,78,29,9,200,605,561,318,49,269,684,495 }, +{ 32,20,13,21,2,38,5,43,29,23,61,12,52,115,90,6,202,37,64,48,95,207,51,81,66,7,25,49,17,74,26,76,178,147,98,46,362,54,22,63,19,14,0,259,128,577,125,97,58,91,77,56,182,65,197,465,376,27,141,427,316,1,241,291,113,878,217,247,312,137,120,4,858,481,142,595,165,31,237,93,230,379,534,482,180,116,86,72,42,11,44,203,263,470,356,352,279,102,84,40,33,687,626,835,786,107,683,260,18,318,24,414,298,468,367,326,304,216,117,105,99,3,933,612,415,347,501,344 }, +{ 339,188,11,377,79,4,99,94,12,542,102,111,474,37,51,33,471,453,152,40,475,342,77,403,59,196,277,139,153,57,134,117,13,775,229,128,401,38,880,23,210,64,184,202,431,115,961,157,300,3,266,72,485,402,89,629,21,54,15,129,177,76,14,162,32,17,216,10,148,275,269,5,757,452,361,477,141,90,205,313,569,494,2,65,179,120,257,381,421,237,197,405,86,488,328,36,318,48,22,137,165,413,50,308,0,1,341,130,82,95,49,457,161,543,245,18,304,260,62,45,285,180,291,41,529,498,125,56 }, +{ 128,283,503,34,318,529,432,176,495,275,231,412,527,291,402,161,381,202,59,3,389,469,177,803,53,585,216,14,269,57,200,17,73,472,27,56,250,136,678,4,152,51,862,361,80,245,11,789,619,194,123,719,401,23,714,84,507,210,77,12,65,322,106,99,436,130,129,46,62,2,13,38,162,369,236,108,33,183,133,457,153,64,483,211,297,741,514,403,114,18,629,961,378,555,440,119,383,597,475,29,453,229,40,165,240,215,196,655,611,556,21,249,117,54,667,134,102,308,814,19,50,86,391,22,0,197,95,26 }, +{ 15,217,13,341,317,141,365,115,23,51,180,349,291,1,120,77,260,269,202,137,21,5,450,352,373,110,0,372,64,33,36,128,177,4,318,351,102,9,197,2,304,854,10,901,7,12,32,165,326,752,40,244,922,160,96,22,523,117,11,48,16,86,661,90,457,82,386,494,403,308,59,28,55,253,247,18,498,134,8,143,153,221,14,6,83,30,196,93,3,37,615,453,401,898,421,817,24,95,961,285,475,719,873,17,35,39,391,730,125,67,237,49,98,811,704,642,717,216,203,69,482,99,342,402,851,50,57,569 }, +{ 13,961,15,4,11,12,457,51,3,37,2,197,5,10,569,99,115,23,6,72,89,14,0,1,134,7,59,33,40,38,165,111,196,76,405,79,139,117,148,498,873,339,719,300,277,485,17,471,157,152,128,21,94,431,32,177,188,509,474,64,45,54,482,730,153,36,629,386,41,22,28,141,901,864,18,180,77,523,9,401,678,437,102,304,210,308,16,775,579,48,31,317,179,413,365,758,352,57,24,341,86,752,546,137,58,318,245,44,27,402,95,899,19,108,269,216,202,49,184,110,93,954,880,56,55,452,358,8 }, +{ 15,13,1,2,0,5,10,23,33,9,7,3,4,6,12,102,115,14,11,217,51,117,77,40,569,153,341,196,317,18,457,22,8,16,961,197,719,180,64,365,165,730,120,134,21,141,899,202,82,31,523,291,125,372,386,28,17,139,93,349,128,304,269,177,110,482,32,342,111,37,318,95,453,579,36,901,59,24,352,450,44,437,137,403,49,260,401,210,237,247,326,817,494,98,99,57,752,19,48,203,30,86,351,485,758,116,402,391,26,898,308,38,27,25,275,801,498,854,65,56,629,76,90,300,922,468,144,35 }, +{ 76,292,5,214,45,32,2,519,710,460,129,179,29,1,90,58,123,249,464,207,205,618,17,14,245,257,72,61,48,263,106,21,358,425,769,241,119,52,184,6,162,186,50,46,38,7,655,148,209,243,128,255,3,155,288,183,290,64,141,454,227,89,77,476,54,115,4,25,59,19,57,229,36,202,269,408,152,99,70,151,130,86,177,75,592,216,720,62,318,12,217,40,26,10,37,27,496,306,33,247,524,311,316,56,497,350,84,51,137,171,790,102,270,23,0,112,11,22,16,452,334,237,432,402,96,180,95,449 }, +{ 76,32,72,21,38,4,54,89,57,17,179,12,14,99,148,45,23,59,51,90,488,3,11,241,401,2,428,37,543,152,205,528,453,128,5,421,13,115,129,77,10,474,245,184,350,670,449,413,1,569,137,79,403,629,475,328,308,221,155,464,0,361,123,743,202,460,214,257,33,40,64,794,292,165,405,770,805,41,476,62,162,270,229,358,86,285,36,402,775,316,341,94,197,50,576,106,431,720,46,29,540,217,139,773,119,838,48,15,216,181,657,177,290,514,288,7,457,227,961,9,22,102,247,828,6,141,365,180 }, +{ 98,223,393,1,834,284,791,264,697,724,293,909,772,905,541,478,51,821,807,363,897,48,682,421,765,447,411,730,801,410,163,948,386,96,9,31,963,23,21,691,664,453,125,5,382,100,13,521,949,551,656,44,116,770,686,670,639,221,39,367,914,265,166,144,719,335,812,848,371,0,101,842,318,321,127,88,252,520,662,329,354,32,759,438,33,698,93,202,593,201,286,156,450,658,281,115,165,49,401,120,604,586,82,2,22,36,225,217,77,744,10,498,141,102,615,180,118,435,69,30,921,404,326,197,433,475,83,70 }, +{ 341,13,23,509,762,638,569,242,901,629,165,8,197,873,10,961,506,456,391,499,116,338,180,457,492,206,12,51,115,268,143,365,951,1,93,965,642,867,417,37,2,253,546,64,5,203,3,498,758,752,719,0,401,15,352,44,31,7,36,244,86,4,141,386,734,237,6,22,308,730,55,844,164,918,473,326,125,82,564,949,898,551,940,791,120,38,9,260,523,137,661,373,559,110,14,854,959,615,317,11,54,33,102,692,269,99,351,903,89,870,740,217,708,170,824,40,801,728,864,817,717,160,128,28,117,776,479,77 }, +{ 818,230,755,443,352,420,1,2,244,50,14,5,38,46,29,36,114,218,65,96,22,133,52,181,58,0,17,28,21,103,75,54,45,86,67,175,48,171,7,12,323,137,24,285,208,51,487,6,283,18,10,70,23,155,32,83,194,361,3,503,163,389,328,128,223,37,803,532,16,19,168,735,69,13,124,61,202,160,186,433,9,115,484,27,236,514,238,782,95,25,754,108,64,657,112,141,281,220,412,289,55,756,401,394,584,151,308,49,545,851,822,4,227,717,866,334,269,646,403,402,26,400,87,35,745,366,99,143 }, +{ 481,878,32,202,5,21,23,269,13,318,494,142,77,51,70,1,182,557,141,137,33,291,58,178,415,465,61,48,590,259,922,115,2,929,640,403,207,884,4,260,120,87,817,626,534,349,342,102,195,0,132,351,59,427,252,109,35,7,961,678,345,247,153,72,845,158,169,49,12,10,6,196,112,798,450,687,279,841,713,475,457,128,40,146,28,3,416,290,265,75,767,854,771,947,795,753,704,637,631,597,592,581,546,509,493,354,352,341,326,316,288,285,241,237,212,197,159,90,54,52,46,38,37,36,17,14,39,9 }, +{ 15,1,5,4,2,13,11,341,0,3,7,12,10,6,21,365,9,180,32,134,317,197,33,23,37,117,40,165,17,77,217,498,8,752,153,901,64,102,569,141,873,51,14,961,196,115,59,110,352,509,22,128,57,111,120,143,125,72,38,48,372,36,291,308,349,99,139,31,90,202,260,269,457,44,16,89,76,49,55,401,342,237,27,93,177,94,82,18,45,29,326,28,19,386,221,351,137,86,277,546,391,41,50,35,403,453,54,164,318,719,474,629,373,203,98,95,475,854,253,304,148,24,450,244,56,758,542,300 }, +{ 25,26,119,42,19,6,27,45,122,84,469,4,17,790,814,113,80,76,59,73,483,2,29,611,128,32,202,40,1,77,90,862,106,56,209,33,240,0,684,136,102,14,292,416,5,311,11,504,196,318,123,680,378,255,12,129,161,22,690,151,53,117,63,179,667,10,85,72,61,115,65,210,402,396,57,141,64,614,134,757,153,137,62,18,99,445,51,269,66,177,7,3,24,710,256,648,36,655,235,38,23,306,58,618,70,491,48,454,21,403,742,361,49,34,86,494,275,383,342,28,300,145,13,74,112,52,297,245 }, +{ 14,7,24,2,1,16,69,6,38,67,108,22,65,28,124,0,46,18,29,86,66,35,83,5,71,398,36,12,168,289,140,95,610,189,150,149,10,21,92,756,52,302,23,109,97,20,584,25,50,267,194,64,49,51,32,60,103,55,505,13,429,172,128,273,420,158,37,458,19,547,118,48,75,115,137,74,135,9,185,164,143,388,154,202,26,701,650,251,133,336,17,114,169,68,230,236,43,54,141,244,671,253,703,105,208,90,3,160,159,278,99,462,314,403,380,87,389,175,357,324,484,318,232,708,125,799,451,886 }, +{ 128,51,202,23,141,120,403,15,64,13,318,82,10,269,99,33,12,197,475,453,260,153,137,165,196,0,3,402,351,5,494,1,22,14,110,117,180,102,93,17,2,31,37,38,125,9,326,450,49,21,86,44,401,457,7,6,36,352,115,18,77,523,237,28,134,95,285,386,421,719,32,48,342,98,11,54,203,730,57,341,961,373,139,8,854,361,757,111,898,45,4,40,308,194,65,16,922,277,41,880,678,116,291,24,498,19,27,391,143,56,217,899,59,569,30,79,365,775,50,503,208,55,144,817,61,105,901,94 }, +{ 16,24,35,18,71,68,9,0,65,189,1,7,133,50,69,149,60,28,103,101,108,537,67,118,230,75,114,23,335,55,167,21,124,92,12,191,181,433,13,39,140,5,14,48,223,64,212,96,51,218,443,281,487,154,141,163,22,217,99,37,70,565,2,115,49,267,58,54,394,766,46,246,545,83,30,128,10,330,820,202,32,361,137,622,125,238,237,100,121,77,165,412,175,87,143,352,399,171,317,86,126,745,146,486,418,177,36,291,269,318,283,336,105,302,398,695,4,98,197,180,31,345,251,271,109,309,430,38 }, +{ 50,2,1,46,14,29,38,52,171,67,5,58,103,96,36,24,218,133,223,83,114,236,45,70,54,181,163,186,108,155,75,28,65,238,334,151,18,433,160,745,328,0,366,112,532,48,516,17,514,61,832,400,281,21,227,22,760,19,6,69,754,766,412,12,7,309,545,409,27,361,283,137,16,220,675,480,55,86,765,822,502,552,32,10,307,99,956,285,25,113,9,398,323,37,51,800,26,388,394,64,891,128,124,243,549,828,3,458,194,208,35,646,95,262,23,311,150,966,820,866,894,149,13,115,87,430,80,287 }, +{ 21,32,5,76,72,14,17,2,3,38,54,89,45,51,1,12,23,4,10,48,90,137,36,148,221,401,22,0,421,179,115,86,99,543,11,453,13,37,128,270,46,241,285,29,290,64,428,57,6,361,202,59,540,50,403,328,114,475,528,65,155,449,77,308,794,569,181,7,805,27,40,141,292,133,33,576,263,129,514,205,341,9,413,350,208,19,838,629,323,28,430,480,119,657,474,214,741,123,245,402,102,720,15,770,476,408,494,56,425,106,269,62,207,775,243,152,477,670,180,318,197,405,165,95,58,281,18,697 }, +{ 434,384,268,144,940,855,617,206,959,911,332,116,93,203,137,489,282,244,141,120,36,44,160,269,253,367,110,450,10,127,202,276,417,219,98,258,326,131,260,31,64,373,201,9,187,318,692,473,22,224,523,817,143,86,352,1,55,30,23,51,28,295,567,105,559,499,261,717,811,0,125,951,180,8,115,13,217,128,551,39,165,49,96,341,284,77,21,851,728,564,662,498,317,386,752,48,351,531,5,642,365,197,83,661,237,615,800,291,242,15,170,164,494,82,95,638,704,349,37,854,228,7,32,24,935,968,35,16 }, +{ 2,5,1,15,7,13,0,10,6,3,180,217,23,9,141,11,317,4,341,17,352,117,197,14,77,33,64,165,134,365,115,12,102,752,237,40,498,153,291,32,372,51,22,21,8,349,901,269,16,196,457,45,569,128,36,143,59,57,509,125,27,38,19,342,86,202,864,873,120,31,546,318,110,37,28,961,29,164,177,95,48,244,111,137,93,391,18,719,139,44,49,82,386,629,54,24,482,56,99,26,453,260,50,579,76,25,304,35,98,253,326,41,55,277,210,72,275,30,730,437,90,523,403,203,351,61,79,450 }, +{ 113,25,2,29,6,833,45,61,145,7,32,74,151,0,42,1,210,128,177,5,19,4,72,14,64,106,655,59,22,122,26,112,174,66,40,77,738,209,525,402,138,90,148,20,483,104,684,141,51,76,17,27,425,18,21,416,23,10,202,95,16,255,52,13,115,624,318,137,102,311,33,269,85,81,49,887,87,746,544,491,275,86,119,790,454,690,65,24,9,222,216,665,291,117,80,204,58,243,134,48,476,814,217,46,349,287,70,28,285,43,348,36,84,342,11,757,194,91,207,315,31,38,73,153,192,355,135,300 }, +{ 116,268,918,551,203,692,31,940,403,791,421,206,417,93,499,523,299,44,8,473,23,51,805,22,13,450,120,475,202,0,959,730,386,125,599,326,137,9,10,401,899,105,1,576,817,559,453,564,165,393,494,21,28,36,456,5,127,949,64,670,728,638,391,242,685,2,260,762,457,197,180,615,569,141,170,253,308,963,49,115,341,492,55,143,719,479,873,37,18,4,86,7,77,506,48,244,801,867,95,318,269,901,512,98,951,33,6,301,958,642,338,30,352,870,39,221,365,12,16,395,373,24,237,629,164,102,159,239 }, +{ 107,612,126,132,362,146,259,199,279,660,493,590,121,20,939,683,534,43,147,941,376,35,595,953,649,581,60,470,640,955,356,522,5,97,195,150,407,49,465,16,92,14,7,687,359,298,22,858,246,0,13,360,21,48,18,427,271,167,423,191,10,441,325,392,32,598,929,28,33,31,178,414,1,77,232,173,55,230,156,125,24,68,23,95,345,4,904,626,105,212,319,98,786,783,70,61,202,142,44,702,58,71,40,314,37,198,51,64,2,623,163,159,819,93,446,158,59,346,357,12,501,118,260,46,663,116,120,102 }, +{ 15,13,1,2,0,23,5,4,3,77,102,33,10,51,6,7,9,115,341,12,11,217,342,317,14,117,40,22,365,180,120,8,569,18,196,165,457,141,21,210,197,64,177,31,291,153,16,961,59,523,817,453,719,450,134,730,93,260,352,44,269,275,49,386,401,32,901,110,82,28,17,629,125,349,139,475,899,24,128,37,326,111,752,95,202,36,318,403,372,498,216,90,308,247,30,98,482,137,86,237,116,615,300,48,65,203,19,57,351,55,391,509,99,38,734,922,468,244,144,221,854,304,579,373,39,143,758,148 }, +{ 53,27,161,17,469,73,378,527,136,250,383,19,862,495,62,26,56,200,297,106,84,585,440,80,34,14,436,128,119,45,4,3,472,123,129,813,504,240,483,361,59,444,205,65,611,5,202,548,108,255,742,514,283,78,814,113,179,162,690,57,402,183,38,555,714,6,245,390,122,54,42,803,184,25,61,33,630,318,209,12,194,64,77,684,11,141,328,757,503,269,99,236,2,102,389,51,22,618,176,152,257,453,47,21,48,210,86,680,40,381,148,133,217,880,650,215,46,10,130,377,18,0,412,497,23,114,13,229 }, +{ 209,45,17,106,207,119,5,255,454,243,263,425,290,655,306,186,84,29,62,61,2,408,136,58,742,378,21,56,311,240,3,592,48,32,383,50,52,80,227,790,270,46,171,14,72,440,148,287,334,76,659,400,666,179,504,496,869,64,155,241,129,1,123,54,38,214,738,75,663,162,90,141,89,221,151,292,57,358,237,507,513,130,34,548,152,70,205,37,128,183,211,184,667,41,229,328,476,266,115,555,181,86,549,405,6,281,163,269,215,12,96,245,11,220,502,720,257,36,313,430,288,180,833,40,202,165,449,27 }, +{ 152,4,452,59,128,471,529,339,79,216,11,188,377,77,291,202,318,94,102,402,32,33,99,5,90,51,269,76,349,12,184,257,542,474,17,177,111,141,37,21,342,210,23,205,275,678,13,72,403,38,285,3,2,40,64,229,588,57,10,769,494,48,134,45,372,196,117,217,129,6,497,1,148,460,757,245,453,475,0,137,153,157,89,139,7,115,961,292,179,498,304,524,266,197,14,629,431,477,405,775,288,437,180,15,165,214,237,86,401,308,119,123,54,22,221,120,9,162,27,619,300,62,317,316,31,488,277,880 }, +{ 15,1,2,5,0,7,3,4,6,13,23,115,9,12,51,180,8,317,197,10,961,901,11,22,77,21,752,33,202,141,110,341,14,16,217,134,120,102,31,165,365,40,93,269,457,352,64,44,28,18,37,351,260,117,569,82,308,125,32,196,49,36,153,719,128,482,318,99,17,401,326,137,450,403,203,111,177,730,386,291,55,873,523,373,98,30,237,24,453,89,48,95,221,475,39,59,116,372,349,105,144,139,509,143,72,96,758,391,898,494,854,86,342,899,210,629,57,468,247,90,304,65,148,83,817,421,160,20 }, +{ 951,752,638,811,642,253,351,180,10,197,341,901,873,352,110,244,143,762,165,8,898,55,564,141,15,365,854,499,82,137,417,498,509,36,535,922,479,559,206,661,373,317,282,115,31,22,338,160,93,569,164,260,717,217,391,86,533,33,44,1,28,9,242,965,728,120,116,77,64,968,21,102,203,125,37,117,506,30,13,386,5,4,531,269,40,911,153,23,0,169,384,2,326,372,105,196,851,170,134,109,704,332,268,349,96,127,48,291,32,758,237,318,24,177,202,35,11,961,401,39,308,83,3,629,51,210,12,434 }, +{ 6,197,350,74,262,841,509,26,47,115,13,196,583,903,165,4,235,138,50,308,21,130,749,324,675,19,38,331,391,145,711,569,33,75,241,453,734,64,153,223,52,558,236,468,99,220,34,70,112,11,117,546,237,125,3,27,32,1,42,134,409,51,78,25,163,29,14,648,23,645,180,820,46,2,85,108,65,773,802,624,122,394,341,785,22,730,95,386,606,57,412,247,923,316,66,498,357,221,630,133,457,59,853,90,53,226,873,632,48,113,719,629,45,889,17,307,141,766,69,12,37,444,83,171,54,600,5,765 }, +{ 125,391,165,867,558,386,221,758,23,457,21,51,13,197,401,308,963,115,961,97,719,77,791,743,451,603,949,48,180,670,770,341,89,120,730,352,291,148,450,426,607,304,7,403,393,66,64,350,316,217,134,135,14,421,74,247,177,50,380,24,696,236,172,87,629,10,141,475,144,35,901,801,523,453,138,509,467,456,241,210,150,98,18,565,357,267,103,582,545,342,301,278,237,216,20,16,99,69,903,650,551,412,389,283,81,32,298,739,543,498,370,365,317,309,300,260,170,140,116,85,52,37,33,30,25,5,273,203 }, +{ 14,1,46,2,24,16,38,50,5,69,58,35,7,75,18,67,54,48,0,155,513,171,163,61,103,45,29,52,430,270,37,223,21,70,12,28,9,780,32,267,60,290,83,227,71,36,666,767,22,189,140,807,10,72,96,65,108,181,64,186,220,309,17,335,92,394,23,592,243,89,536,663,115,281,51,6,474,55,68,112,141,400,128,133,87,13,425,99,741,236,95,675,278,230,76,221,114,86,263,113,820,552,502,27,269,202,49,760,537,149,262,328,408,90,366,137,150,207,77,357,315,307,218,241,19,148,25,66 }, +{ 23,51,13,115,453,12,0,2,4,10,403,1,202,5,401,475,15,21,457,59,77,165,9,3,37,99,32,11,197,719,308,6,14,64,7,569,17,629,961,318,180,22,33,494,120,18,730,141,40,102,269,217,177,57,16,128,523,450,341,8,90,421,386,237,28,48,125,19,117,31,247,402,72,210,899,221,260,291,134,89,342,317,365,509,38,24,391,216,352,468,817,153,54,36,45,49,27,196,56,326,82,65,76,304,349,139,50,25,86,275,44,137,482,372,670,498,26,734,546,62,95,758,29,41,46,93,351,79 }, +{ 129,84,17,27,56,495,80,378,19,548,123,162,161,53,469,618,73,26,504,106,40,257,205,59,3,183,184,862,4,136,62,6,128,202,14,77,11,65,769,5,527,108,45,57,383,361,2,497,250,249,200,130,23,34,22,90,402,483,245,240,12,51,214,38,0,139,25,99,64,318,102,152,32,210,300,10,297,757,21,494,880,775,114,710,111,33,54,29,460,403,452,36,42,133,18,61,86,141,436,177,814,269,13,194,119,179,49,28,342,453,217,95,115,514,229,556,76,137,37,1,742,283,485,401,503,488,475,24 }, +{ 28,109,49,9,22,67,168,158,39,185,159,420,1,55,0,83,35,95,194,272,86,105,69,103,36,114,169,327,50,254,208,175,65,584,286,323,535,346,251,345,797,30,131,756,374,12,333,533,128,2,218,68,96,133,344,160,101,23,64,289,7,597,189,24,646,577,874,896,13,5,601,51,46,503,531,837,16,238,747,627,115,14,347,10,98,310,789,137,124,21,202,127,164,578,54,799,141,154,230,532,547,31,312,708,285,480,37,810,637,596,48,143,336,100,389,835,252,439,237,38,518,912,219,424,44,239,253,868 }, +{ 242,391,456,116,23,13,492,8,51,341,401,165,93,206,499,457,867,338,268,569,417,509,386,22,1,762,479,638,10,629,918,758,64,551,203,730,901,719,12,31,197,791,115,5,86,14,125,949,0,559,37,2,642,951,873,959,237,961,692,55,170,752,870,120,44,180,940,9,141,564,523,3,308,36,7,4,15,899,450,365,728,137,21,473,253,898,558,143,128,453,740,164,801,260,903,6,685,202,403,670,965,54,114,28,854,326,817,844,352,95,82,498,11,923,244,98,299,922,239,110,38,317,105,599,373,661,160,99 }, +{ 211,162,248,130,556,57,507,4,305,266,183,41,152,361,11,229,129,471,62,38,377,313,300,514,77,339,184,157,381,59,128,452,99,3,328,40,202,123,54,210,440,494,369,555,342,136,117,542,402,757,64,548,139,23,196,188,880,457,775,33,153,56,14,102,432,13,177,205,257,497,17,12,37,291,51,89,403,215,961,111,141,90,349,216,488,269,485,803,477,180,165,115,10,475,437,383,79,197,95,21,84,134,503,421,86,453,318,308,1,670,76,194,719,618,106,32,629,15,730,65,22,5,245,94,214,45,482,288 }, +{ 92,7,97,298,140,16,126,232,314,60,246,279,35,173,230,71,107,14,43,508,423,150,779,653,441,607,66,189,359,501,24,362,20,172,104,387,356,21,48,38,18,399,199,330,5,674,192,877,81,191,729,517,74,672,426,939,560,633,0,267,2,649,69,121,149,392,324,135,462,22,95,29,46,52,125,598,37,723,707,49,10,357,68,345,6,32,830,98,64,1,702,380,118,28,900,12,695,251,861,885,493,783,603,146,65,715,61,55,278,36,236,273,138,33,75,25,31,124,175,221,293,83,165,763,712,451,414,319 }, +{ 15,1,2,5,0,7,13,6,23,9,3,12,180,4,51,115,8,901,457,117,14,165,22,341,82,120,141,10,37,197,961,31,21,11,202,351,64,260,77,365,110,16,134,28,217,93,269,33,44,36,569,18,719,308,450,317,128,102,125,137,32,629,153,98,401,55,352,30,318,373,49,116,898,730,48,17,752,523,196,237,386,24,95,86,326,403,291,99,453,40,39,72,38,177,89,203,873,391,144,247,349,498,899,59,734,90,475,372,54,494,421,143,482,304,854,105,758,35,210,65,76,244,160,216,221,83,45,546 }, +{ 403,18,0,102,25,523,64,6,91,85,42,74,342,32,24,13,51,306,95,49,22,33,475,151,681,880,421,128,899,750,397,214,122,757,730,77,28,519,288,247,23,5,817,805,317,879,861,673,273,177,145,21,120,52,283,81,275,701,577,453,104,59,202,193,697,636,401,287,249,216,207,196,180,165,138,125,123,118,90,84,60,48,43,587,260,389,16,10,4,1,351,326,318,933,804,635,498,280,237,197,186,172,153,148,137,134,117,99,61,29,9,671,289,958,838,814,579,312,238,124,65,56,847,719,654,650,457,424 }, +{ 56,65,554,33,561,18,84,123,63,64,343,194,0,90,49,22,6,27,19,289,240,129,73,659,431,214,475,523,370,368,233,899,701,287,257,95,85,28,102,183,736,650,467,256,153,25,207,124,108,80,32,24,120,26,389,874,718,635,51,525,891,752,751,730,648,614,498,496,450,249,237,209,184,177,162,114,59,45,29,235,77,42,817,497,444,137,112,74,53,352,511,681,630,643,196,947,942,847,764,719,696,651,615,610,603,602,543,457,412,386,337,326,312,288,262,260,247,227,226,179,152,139,134,119,117,105,86,76 }, +{ 127,98,31,293,299,395,44,9,105,0,49,599,284,28,276,116,242,685,159,456,22,763,125,935,272,492,158,1,623,393,109,144,201,39,36,95,18,51,24,320,137,23,286,55,16,512,916,489,202,21,662,301,268,367,327,371,35,761,567,13,434,65,7,101,918,616,2,5,141,737,506,870,12,64,131,68,30,48,900,948,86,128,295,10,219,239,269,910,403,627,338,578,143,14,37,124,160,203,318,384,170,244,165,261,96,347,254,713,963,168,457,237,187,93,67,154,421,221,253,401,344,424,164,479,6,386,83,855 }, +{ 1,14,5,2,46,50,38,58,171,61,45,155,7,48,69,270,290,24,21,29,75,52,16,67,54,666,32,663,227,72,502,0,425,223,18,592,89,17,35,243,37,207,76,186,103,263,36,163,96,83,741,6,408,10,400,22,181,28,9,428,12,536,64,221,220,70,262,281,23,267,115,106,65,807,292,309,119,148,474,449,394,128,141,86,241,3,655,430,51,189,476,140,431,541,587,95,334,738,675,90,114,13,780,108,71,720,19,133,27,202,335,413,4,767,112,77,25,405,513,179,659,99,218,787,55,180,528,151 }, +{ 23,51,13,453,457,12,719,401,961,4,629,730,99,475,115,2,11,14,403,17,40,5,15,165,1,3,64,0,10,57,33,21,77,569,32,50,37,308,7,386,9,16,38,670,197,59,41,217,128,494,46,6,758,62,22,775,65,45,18,86,48,29,72,899,514,361,328,139,102,247,153,54,56,421,468,79,523,95,108,757,196,52,19,317,117,482,342,341,24,734,509,210,202,180,177,237,488,402,114,36,134,880,76,498,90,275,111,34,28,291,67,20,474,120,743,125,82,391,130,752,490,437,405,318,277,27,141,89 }, +{ 968,967,966,965,964,963,962,961,960,959,958,957,956,955,954,953,952,951,950,949,948,947,946,945,944,943,942,941,940,939,938,937,936,935,934,933,932,931,930,929,928,927,926,925,924,923,922,921,920,919,918,917,916,915,914,913,912,911,910,909,908,907,906,905,904,903,902,901,900,899,898,897,896,895,894,893,892,891,890,889,888,887,886,885,884,883,882,881,880,879,878,877,876,875,874,873,872,871,870,869,868,867,866,865,864,863,862,861,860,859,858,857,856,855,854,853,852,851,850,849,848,847,846,845,844,843,842,841 }, +{ 2,1,29,103,75,52,14,67,46,309,6,83,70,238,38,24,108,220,218,133,74,96,262,36,114,87,388,26,50,25,366,236,69,160,112,5,65,58,66,394,19,145,226,7,532,135,163,42,18,45,150,545,458,433,891,331,357,594,17,28,412,27,294,418,86,278,21,124,398,149,732,894,324,537,283,223,273,222,71,12,606,739,16,646,547,138,267,48,696,171,137,151,0,175,602,800,22,37,487,181,99,208,32,194,287,785,336,64,23,13,113,55,95,335,54,97,323,154,61,711,128,68,302,118,10,308,600,484 }, +{ 7,92,71,97,140,60,150,16,298,24,189,230,14,149,423,35,508,314,18,43,279,232,66,399,273,246,251,38,607,69,126,172,517,674,2,173,28,375,74,467,104,68,0,809,20,823,336,118,267,175,135,108,191,154,21,345,501,796,36,52,125,653,29,81,330,107,124,362,356,6,387,55,359,46,779,302,192,22,5,575,48,67,877,462,10,388,441,9,566,723,783,221,1,65,49,398,101,37,278,702,83,91,357,705,392,12,109,121,236,426,380,32,212,695,443,86,596,103,868,608,560,165,830,167,13,558,25,541 }, +{ 51,23,202,120,141,13,1,260,128,21,269,450,77,137,318,48,115,165,15,32,102,5,197,64,180,494,82,351,326,125,291,817,12,349,342,98,523,403,203,93,7,33,854,352,2,31,14,719,457,6,110,144,44,4,386,177,730,96,116,308,99,221,9,217,10,153,17,210,922,453,18,16,402,237,90,704,37,36,117,25,421,11,615,95,20,898,373,30,475,40,401,391,19,317,8,3,304,498,576,69,285,901,275,223,293,247,196,86,899,134,83,341,127,59,45,39,365,216,692,372,50,0,801,752,367,163,139,100 }, +{ 77,102,33,13,177,23,51,141,202,64,40,128,115,4,269,59,318,6,217,137,210,291,0,7,2,1,180,196,14,22,10,95,153,11,117,125,165,15,86,352,197,134,90,403,216,275,3,16,9,18,237,453,494,342,49,31,19,139,25,65,28,5,17,317,349,93,36,304,475,26,82,12,98,44,341,27,120,365,57,24,20,110,372,386,21,247,457,719,194,402,498,116,29,730,99,260,8,143,144,523,164,114,569,48,32,111,326,285,208,391,401,244,30,509,312,961,300,752,56,160,105,203,34,450,546,67,482,170 }, +{ 101,9,363,676,264,18,411,682,604,905,520,821,271,167,621,647,364,121,252,404,118,39,166,848,781,265,100,16,382,664,447,212,777,478,354,146,907,812,132,446,360,724,0,60,463,759,24,329,23,1,88,71,35,936,68,857,195,21,48,593,865,639,932,156,897,31,98,96,51,191,13,286,333,125,656,49,694,30,254,105,591,55,806,562,109,158,44,438,419,272,521,124,154,407,274,5,65,310,553,28,144,691,33,116,159,127,93,32,77,22,67,50,410,37,538,108,706,393,115,83,36,92,12,199,327,223,10,418 }, +{ 144,203,326,382,166,96,822,418,438,93,859,88,921,1,367,141,744,269,349,110,576,473,83,77,410,23,966,21,403,120,692,39,494,274,521,317,291,217,615,523,855,303,817,13,639,264,800,32,100,909,691,225,36,478,914,765,506,5,498,652,44,435,265,321,260,160,910,551,404,137,48,772,202,897,453,450,208,101,960,254,31,863,342,286,190,51,30,593,831,616,419,730,682,98,340,372,341,197,69,33,9,625,724,848,686,284,466,704,656,223,180,102,335,125,617,352,285,363,0,662,244,530,919,812,393,374,351,318 }, +{ 21,13,180,23,32,115,48,125,197,191,165,51,5,199,237,146,12,225,18,16,93,901,350,271,22,241,375,0,71,118,408,49,447,264,126,1,167,31,24,522,302,98,65,64,575,864,569,92,566,247,744,86,58,391,352,713,212,141,60,44,9,4,88,562,459,308,303,203,202,100,45,37,17,2,818,463,360,316,919,897,698,586,478,363,293,221,213,103,72,40,691,909,608,558,509,166,68,35,727,340,517,217,724,629,546,467,446,341,304,156,120,108,11,640,288,110,7,734,296,150,591,914,346,354,321,317,269,114 }, +{ 15,13,23,51,102,0,8,77,1,5,12,33,9,4,2,10,180,82,165,197,115,110,18,403,202,22,120,326,457,7,11,351,260,898,141,342,14,373,401,901,961,64,36,386,3,291,453,31,16,40,28,93,21,6,352,59,44,99,27,341,450,217,719,49,117,48,125,475,128,37,391,308,153,95,269,24,318,177,854,730,203,615,421,98,30,32,38,137,922,317,498,704,523,57,817,509,56,569,629,116,899,133,83,365,758,105,752,144,661,196,304,638,494,134,300,670,206,402,221,210,208,86,535,143,65,532,109,35 }, +{ 77,33,102,64,202,141,128,51,40,196,23,13,117,318,269,153,134,1,177,115,10,342,137,180,210,2,0,95,6,139,403,86,14,22,15,7,217,291,4,352,165,197,36,59,237,275,3,11,5,494,65,125,49,9,453,475,16,19,28,18,120,82,216,17,402,31,93,110,260,326,12,341,194,349,401,203,98,25,99,27,24,457,44,317,365,498,372,26,351,285,719,143,386,450,300,21,752,391,116,29,312,38,114,304,111,244,164,961,509,57,247,569,523,144,308,373,901,34,421,96,56,160,437,35,20,678,32,8 }, +{ 174,104,544,151,624,74,525,25,644,66,29,177,636,416,0,204,750,122,216,6,81,45,186,222,319,138,2,145,287,112,665,192,64,59,243,7,77,32,355,22,715,128,784,17,173,20,311,135,789,280,52,33,549,113,879,4,227,21,194,90,5,141,491,102,86,42,95,43,137,58,12,27,496,275,65,529,614,11,348,678,304,651,402,580,712,503,269,422,49,37,14,318,210,673,396,202,792,10,18,500,869,36,148,85,426,247,405,681,19,226,97,825,72,16,196,40,61,609,597,115,24,217,233,51,28,881,91,153 }, +{ 372,141,304,352,947,954,291,177,217,269,349,128,318,437,498,77,678,202,864,64,102,115,86,210,873,137,751,597,247,13,180,5,275,32,16,35,21,950,752,92,197,83,87,7,457,160,776,70,33,24,179,36,317,245,165,126,107,569,288,717,95,402,237,23,263,108,774,156,546,509,340,37,18,12,946,342,872,75,800,244,114,103,851,735,365,341,146,121,99,60,579,312,120,1,195,354,296,140,888,71,51,260,223,418,758,622,537,473,363,196,98,69,10,526,48,820,817,765,704,586,450,316,246,208,148,96,90,50 }, +{ 200,161,714,53,73,27,472,383,17,440,34,585,62,56,136,250,106,78,527,322,19,862,361,4,3,514,80,84,390,14,176,179,236,378,162,377,129,548,202,47,469,495,217,26,11,494,128,757,555,453,402,412,130,880,328,507,257,183,51,45,5,38,300,184,123,497,54,12,57,33,444,59,556,283,205,245,108,485,297,209,803,152,342,40,119,99,318,452,2,46,102,211,23,196,64,229,65,77,215,684,21,814,6,240,0,153,368,403,90,730,475,13,117,133,61,37,317,305,32,255,813,22,504,181,503,29,436,95 }, +{ 32,76,72,21,38,14,241,89,54,428,449,5,45,148,37,181,90,179,2,17,221,350,12,292,46,129,48,476,155,1,794,99,477,205,720,3,770,50,464,773,4,540,308,36,57,460,270,245,413,743,184,214,741,328,115,23,123,474,137,257,10,358,11,0,281,431,29,488,421,405,162,51,288,59,64,401,710,536,430,152,290,128,218,519,22,316,697,552,7,587,86,670,361,487,6,227,13,52,618,133,249,543,77,106,62,165,769,141,243,202,403,229,114,453,263,67,475,745,341,119,197,207,497,285,9,40,180,920 }, +{ 128,141,202,64,102,33,23,77,51,13,196,318,269,153,2,3,180,165,40,137,5,15,1,6,403,10,117,139,197,86,352,134,14,17,0,19,494,210,120,342,22,7,402,237,453,291,177,475,125,115,93,217,95,21,12,82,260,110,36,457,11,65,341,450,31,25,27,194,99,9,4,326,498,275,961,349,26,49,111,401,317,44,203,391,351,277,38,28,386,18,421,37,752,164,16,143,29,285,98,24,48,678,114,116,32,373,509,300,244,208,54,719,56,59,365,57,8,50,546,144,20,523,30,569,372,898,90,308 }, +{ 39,137,96,840,30,202,9,0,613,88,406,669,100,141,180,576,160,28,49,530,403,1,269,22,228,822,829,679,856,190,244,166,219,352,816,966,765,827,36,55,68,83,128,274,418,23,318,98,44,442,35,10,253,258,51,863,77,31,317,251,21,876,158,494,475,213,95,65,116,254,859,268,125,48,13,16,931,187,367,131,805,574,105,5,365,800,102,177,93,115,33,224,127,242,120,110,291,924,217,960,144,64,842,616,165,531,385,351,101,159,276,839,778,203,201,492,109,67,24,284,748,860,345,692,295,421,272,456 }, +{ 180,352,141,752,1,901,269,365,115,5,217,2,15,4,317,23,197,7,13,341,0,244,3,253,6,51,498,202,21,318,31,372,873,165,44,77,98,9,33,10,116,137,237,93,482,125,64,864,105,12,32,48,349,509,128,131,22,120,579,11,143,8,102,569,717,37,391,177,260,127,308,110,28,30,291,450,40,559,49,39,401,160,203,144,14,437,373,564,72,24,453,403,386,546,421,96,326,16,95,36,148,661,951,247,533,494,219,109,201,159,55,961,251,475,728,468,642,117,254,221,134,18,811,153,210,82,345,224 }, +{ 1,67,28,50,46,2,103,83,14,65,114,24,0,38,133,69,36,55,18,75,9,54,124,29,22,283,7,218,5,238,16,154,35,52,96,160,480,109,108,547,12,68,398,429,646,323,181,732,175,220,21,361,171,71,335,236,389,418,70,394,48,49,149,86,58,866,531,87,309,366,37,484,208,328,251,118,194,6,10,302,412,487,420,189,336,23,168,545,800,51,803,433,185,137,128,163,754,39,45,505,169,285,596,486,13,503,533,60,101,155,95,64,32,289,570,223,158,267,927,140,745,516,17,115,514,346,956,99 }, +{ 180,352,365,317,115,217,752,901,341,1,253,141,873,2,15,4,0,5,197,3,244,308,951,31,7,498,318,44,6,372,291,165,269,202,98,137,120,127,160,36,116,373,105,260,482,13,131,33,509,110,51,125,77,39,569,349,304,437,9,11,8,450,96,64,22,21,391,221,23,93,638,717,28,219,12,421,10,403,187,83,17,102,49,661,564,851,401,128,247,811,351,523,117,535,546,468,224,201,86,35,559,48,16,148,494,134,143,579,968,922,854,817,800,551,475,55,801,758,728,615,196,177,153,144,14,32,30,533 }, +{ 32,5,21,347,13,95,49,713,51,48,197,23,107,60,126,64,237,180,153,77,165,241,33,379,850,1,263,296,196,195,4,121,134,132,102,28,115,22,117,217,0,291,16,841,929,928,652,534,660,39,493,247,31,279,18,50,36,626,150,90,76,40,407,883,509,362,349,260,9,356,125,10,177,92,441,340,259,35,24,752,351,199,65,11,71,809,546,953,933,854,312,75,61,56,37,30,7,598,207,44,423,372,298,522,783,590,354,156,645,608,591,939,937,638,498,450,437,391,359,303,273,86,57,342,316,59,640,446 }, +{ 341,365,180,901,317,115,752,15,873,217,564,82,951,197,110,165,0,1,141,351,4,3,482,352,6,372,2,12,269,13,10,559,253,854,5,468,7,244,282,23,535,18,898,14,661,417,349,308,318,102,33,509,125,811,704,36,143,77,964,28,498,326,237,137,51,533,692,638,38,475,31,22,9,642,867,391,30,8,202,120,203,966,758,373,338,206,24,569,291,922,494,403,332,144,116,49,44,697,55,450,386,128,943,842,768,728,706,405,346,299,295,261,260,258,242,160,148,39,27,26,19,17,11,50,48,46,843,634 }, +{ 45,17,106,21,227,209,62,243,5,155,32,3,290,38,255,119,54,270,14,2,207,181,587,48,263,12,794,29,46,552,540,186,659,37,430,408,770,496,721,50,454,136,84,52,221,425,56,76,308,828,99,697,400,328,115,123,741,80,488,1,528,474,401,57,218,281,41,129,541,513,421,36,487,128,10,787,61,241,378,477,543,64,51,742,306,133,240,413,11,72,428,4,165,734,23,152,655,453,211,34,920,311,59,90,236,137,214,58,162,629,431,179,832,341,6,366,583,89,77,202,313,361,383,317,514,86,745,365 }, +{ 16,24,35,18,50,7,71,75,60,14,69,0,9,1,189,108,68,67,21,181,28,12,48,223,92,5,23,140,46,103,51,133,13,2,267,230,54,163,149,65,22,335,70,281,10,32,55,96,87,37,64,171,128,394,114,141,502,58,115,486,236,99,820,98,766,202,38,137,83,309,118,246,269,49,36,95,125,191,155,167,121,430,39,218,101,77,6,150,237,126,641,774,31,318,221,487,180,443,177,433,251,545,220,565,388,29,86,72,352,399,165,217,552,112,61,105,537,17,852,45,44,30,212,175,124,89,307,102 }, +{ 127,308,293,98,165,395,258,219,13,286,23,629,401,51,115,301,569,197,457,276,512,391,509,105,616,239,924,338,22,734,170,55,961,295,31,758,169,910,272,567,116,763,201,320,944,341,143,261,237,506,180,479,873,365,141,185,206,36,242,86,374,762,498,546,913,131,44,64,923,327,843,601,386,332,492,160,125,860,317,654,964,900,284,95,28,634,943,164,137,10,901,282,224,740,468,456,144,109,841,558,719,268,120,417,535,384,299,844,482,903,110,93,9,1,935,662,578,217,253,928,730,187,393,2,752,244,531,434 }, +{ 748,957,840,213,669,539,466,88,30,251,863,321,217,679,190,137,93,144,317,530,352,831,228,230,160,617,800,689,219,203,816,96,418,202,36,473,258,406,141,871,326,83,855,752,120,854,778,82,44,100,187,201,1,435,224,351,345,719,335,116,110,443,274,28,310,55,21,450,332,382,822,574,86,931,494,531,365,31,839,128,39,180,341,367,898,23,613,498,372,731,860,9,410,926,197,704,5,817,131,69,35,125,434,166,22,944,318,924,856,268,64,48,567,244,206,291,922,523,269,32,876,717,98,49,13,349,169,859 }, +{ 5,48,21,14,1,2,54,50,36,430,38,32,181,270,45,281,12,72,17,67,46,0,290,218,99,22,28,83,76,155,24,541,37,10,513,89,114,65,58,133,61,103,96,488,51,221,425,263,3,9,328,29,23,487,697,207,741,115,18,86,745,160,428,474,148,408,6,16,502,7,64,171,137,794,108,52,920,477,842,227,55,236,552,543,13,119,514,361,449,202,238,828,536,128,308,292,476,243,592,90,241,186,179,666,285,4,27,655,663,480,421,401,106,208,141,165,433,528,532,19,834,95,283,197,11,180,720,403 }, +{ 13,23,51,115,217,33,77,0,4,2,9,317,102,141,7,11,1,10,180,5,40,202,22,197,3,6,352,365,12,269,341,165,14,18,16,28,64,15,59,318,177,403,752,31,24,196,308,32,372,125,128,291,498,137,17,247,153,21,49,401,453,349,98,117,44,8,95,237,901,36,86,93,494,342,475,30,482,210,468,457,304,873,569,90,105,143,421,391,386,37,509,39,134,139,110,116,35,275,244,144,120,326,55,961,19,546,203,46,864,27,719,221,111,65,164,253,285,216,127,402,730,50,437,38,67,20,288,54 }, +{ 13,23,165,15,37,54,134,12,117,38,457,51,64,197,99,115,153,89,961,719,33,10,82,509,14,128,120,237,3,196,308,86,141,328,0,1,629,72,450,277,139,102,523,386,351,77,8,401,65,4,546,95,155,202,180,21,2,498,391,40,352,7,5,32,11,6,730,474,31,475,110,137,899,854,453,922,569,291,114,108,22,898,44,9,177,403,148,49,361,342,285,260,514,269,93,17,758,901,210,864,479,59,34,29,28,67,217,494,318,236,179,90,76,56,50,18,242,417,528,488,206,98,413,94,817,789,468,402 }, +{ 5,21,32,14,2,48,17,3,1,12,72,38,45,54,51,10,36,22,76,23,0,89,540,13,137,86,37,4,430,270,50,114,11,221,99,285,115,64,6,61,46,361,29,181,290,65,58,403,208,155,28,133,480,281,148,128,323,453,657,401,514,541,754,421,697,202,90,9,7,475,241,263,40,56,328,308,95,59,207,576,805,165,838,513,57,27,402,19,425,18,775,488,119,283,218,741,428,341,24,141,77,33,794,408,227,197,102,194,52,179,832,55,180,494,269,365,449,164,236,350,745,318,171,859,244,474,108,880 }, +{ 202,318,77,291,128,275,102,33,269,342,177,210,402,678,141,403,494,40,20,196,0,137,7,23,49,32,64,216,4,416,104,139,285,59,352,43,61,66,13,304,234,90,22,475,319,450,11,349,117,300,113,5,153,58,947,17,112,21,6,115,120,70,544,38,16,142,2,315,194,134,95,18,498,9,74,597,237,51,329,884,25,12,453,864,24,789,173,415,639,859,155,277,52,81,86,222,178,10,48,560,14,230,665,198,97,31,355,557,65,1,186,105,817,111,39,226,57,546,121,29,147,28,15,87,60,96,265,523 }, +{ 1,24,46,2,14,67,69,75,103,38,35,50,52,83,29,108,18,16,71,220,149,394,309,236,189,70,96,133,171,54,68,366,28,58,5,238,181,163,87,7,433,0,335,60,218,114,251,487,65,267,537,36,328,418,223,9,486,55,516,262,112,150,155,12,388,48,21,45,675,175,336,22,409,118,37,186,281,140,412,696,6,124,25,514,151,294,61,99,154,10,334,32,17,273,92,135,160,283,760,278,458,227,766,64,307,19,86,361,398,230,552,641,532,400,137,565,23,502,345,51,13,95,765,26,66,72,128,399 }, +{ 15,1,5,2,13,0,7,23,6,3,180,12,51,4,9,115,457,165,197,21,8,14,141,901,37,752,117,16,77,10,32,120,202,11,317,217,110,961,22,33,365,341,82,269,719,93,352,31,102,18,128,153,134,28,260,48,569,64,137,40,318,196,386,351,509,17,44,30,326,125,730,308,450,49,177,36,95,453,89,72,98,99,203,873,523,401,498,403,59,291,116,38,24,629,372,90,349,76,55,391,373,144,758,494,237,244,96,482,475,54,854,39,468,143,45,139,105,221,247,86,899,50,148,253,35,210,304,421 }, +{ 28,65,124,14,0,154,24,67,69,55,175,7,9,429,46,2,109,35,149,336,68,1,189,83,114,251,108,16,133,398,484,103,36,18,38,22,50,267,420,140,283,289,71,570,158,236,389,101,505,230,86,346,185,39,12,169,168,799,194,118,49,596,29,732,782,302,309,5,533,52,252,95,388,419,535,21,439,458,705,75,797,160,412,6,10,54,531,218,92,345,208,96,159,610,48,60,64,323,756,105,37,23,646,238,51,128,329,886,547,278,13,650,637,32,172,135,364,164,143,584,602,115,701,601,137,333,399,443 }, +{ 162,403,56,775,129,880,161,527,99,3,17,40,33,475,23,4,361,128,196,14,200,183,34,202,102,184,548,402,383,53,130,153,757,494,27,257,108,453,38,12,136,117,51,485,123,64,57,65,19,585,141,73,13,139,421,134,229,106,11,80,556,469,670,84,59,152,62,78,714,412,378,77,5,495,21,54,236,488,497,2,6,90,514,86,111,730,22,180,26,46,507,440,95,269,37,472,115,32,401,45,283,452,618,0,328,47,18,29,266,390,10,211,165,899,250,24,300,862,719,504,133,543,197,318,277,50,176,457 }, +{ 151,29,112,2,58,311,186,52,70,113,315,45,66,74,61,631,5,87,287,243,491,209,1,869,6,227,614,135,7,307,75,80,27,145,496,226,25,454,17,14,549,26,845,19,32,46,73,106,294,163,155,64,255,659,849,667,632,644,592,72,220,331,21,334,222,172,42,69,400,48,76,138,396,337,148,635,50,86,611,445,128,171,97,122,141,95,738,119,16,37,38,4,207,67,425,290,409,53,24,65,262,89,90,12,297,278,192,461,137,22,624,36,20,483,881,84,292,684,43,10,833,56,651,655,114,263,204,115 }, +{ 6,74,75,1,70,66,138,29,25,85,26,42,87,220,52,2,135,226,222,802,324,112,145,357,103,19,278,108,380,67,114,309,235,14,500,69,451,133,606,280,38,24,582,83,65,262,163,86,97,394,294,267,486,96,50,36,889,388,603,641,58,256,32,895,238,223,853,45,122,594,46,537,64,21,204,150,27,335,792,102,7,115,337,218,13,418,71,33,160,632,5,189,696,571,308,461,95,565,16,545,331,125,4,137,134,153,113,117,17,35,558,54,287,149,22,171,18,711,174,630,247,63,12,23,0,194,462,90 }, +{ 15,13,23,51,1,120,0,180,202,5,260,9,2,102,450,115,77,7,141,12,21,269,318,197,137,523,403,6,165,326,33,351,341,817,110,901,342,128,365,36,14,4,16,457,308,401,18,494,8,48,32,59,64,82,352,22,475,10,217,386,730,317,28,93,37,3,373,854,453,719,752,31,615,24,44,30,421,291,11,210,221,39,899,99,96,961,55,86,98,40,116,177,402,922,160,898,244,203,35,576,17,569,117,509,144,391,90,49,372,704,349,153,873,285,498,253,275,125,89,143,300,196,72,247,67,95,629,127 }, +{ 5,45,17,14,46,2,50,48,186,181,227,61,29,38,155,281,54,52,80,106,400,209,21,58,3,171,243,487,540,32,328,311,103,218,133,430,255,62,67,12,828,454,36,56,536,236,27,1,207,496,787,119,697,366,114,37,19,108,6,760,513,541,600,587,592,84,745,842,99,920,96,502,151,76,13,26,270,583,263,86,221,115,83,832,666,51,23,136,34,334,65,10,290,72,308,137,663,869,739,361,514,25,721,738,794,4,659,543,89,401,667,11,64,576,378,148,22,24,834,0,474,357,73,483,754,516,41,53 }, +{ 18,16,35,60,68,121,265,158,159,24,329,105,9,132,167,252,327,709,101,55,286,39,21,71,146,49,7,109,374,23,22,126,604,107,419,333,271,14,28,1,48,0,191,120,92,212,20,310,272,890,553,118,5,326,137,199,260,455,51,95,446,320,13,32,450,259,170,627,621,639,2,10,195,647,128,98,364,301,557,463,43,37,239,77,576,848,64,100,351,82,102,125,65,285,202,12,354,6,363,346,178,344,197,165,493,407,147,29,523,242,907,96,538,932,403,33,279,916,97,704,36,31,402,520,318,156,912,846 }, +{ 467,650,389,108,283,56,412,123,177,216,670,619,349,269,33,475,899,775,453,503,789,401,678,65,543,51,196,403,730,128,84,23,194,523,13,421,719,576,457,11,597,153,880,141,249,847,3,801,117,817,57,133,18,0,202,114,59,24,150,32,386,64,40,95,291,4,183,450,130,524,162,5,554,152,90,22,9,361,134,236,102,17,27,120,304,19,961,629,210,432,21,770,86,16,14,10,77,165,46,28,594,381,377,240,137,26,229,494,947,372,602,342,318,275,125,98,630,743,701,605,184,103,97,63,50,49,958,444 }, +{ 514,11,377,328,361,3,4,880,507,403,485,57,130,14,757,152,38,102,300,211,556,215,54,176,236,62,229,56,494,40,585,555,266,34,183,59,342,12,162,440,453,548,51,37,77,402,488,305,184,196,542,421,775,471,123,527,210,369,313,33,457,136,719,23,202,231,13,128,153,381,629,452,803,543,129,412,157,17,248,339,111,84,139,497,475,205,99,119,32,828,188,177,961,349,383,117,46,89,79,730,216,504,277,401,161,134,308,291,322,76,249,283,72,65,64,21,2,22,670,503,241,41,257,90,108,618,472,94 }, +{ 3,555,62,266,507,215,130,440,99,514,229,152,471,40,57,139,403,12,56,548,305,33,475,38,313,775,361,14,123,211,880,328,196,102,23,556,381,184,136,757,377,77,277,54,183,494,453,421,4,11,205,349,291,129,162,452,34,236,84,17,369,37,59,485,497,111,401,402,51,153,157,477,585,13,670,128,117,339,176,161,41,488,248,202,383,300,457,249,119,76,432,46,342,134,79,188,89,214,2,474,72,527,961,21,32,308,719,543,165,177,257,730,412,133,413,64,283,216,210,245,106,618,803,90,15,629,115,460 }, +{ 120,318,450,260,77,15,202,13,817,51,269,5,33,494,922,128,1,326,102,23,403,523,137,141,21,177,351,40,342,48,291,82,854,153,125,64,7,615,32,196,22,704,210,402,98,4,475,12,373,115,93,2,96,453,285,44,275,349,36,61,31,116,216,859,197,117,49,28,16,899,678,203,165,59,90,180,139,50,24,6,144,217,95,898,692,170,110,45,18,17,10,0,730,352,72,37,207,457,576,341,237,35,20,9,661,372,194,961,223,214,958,949,811,805,791,551,424,391,365,244,242,221,208,160,127,99,89,76 }, +{ 23,1,15,51,13,2,5,0,7,6,180,12,9,115,21,3,197,165,4,457,386,120,141,33,202,453,14,730,523,719,401,37,40,32,341,8,31,403,260,82,450,352,16,48,110,269,317,77,217,308,44,137,22,102,18,365,318,901,93,351,421,117,475,752,11,326,28,391,49,98,153,498,128,99,64,30,569,10,899,509,36,196,961,24,221,59,39,89,177,758,854,134,494,116,17,629,90,817,670,468,86,247,72,95,35,105,96,203,482,55,373,125,372,873,144,210,922,291,38,244,237,898,54,50,20,349,615,65 }, +{ 15,180,351,317,82,115,365,217,141,898,10,854,901,197,143,77,23,752,341,352,33,110,13,36,64,102,51,4,1,260,253,244,120,9,21,5,22,40,498,55,165,137,117,28,11,873,372,922,482,134,59,0,72,153,196,160,32,48,202,3,2,961,90,12,35,811,164,291,177,349,31,86,30,308,7,125,509,642,450,128,6,139,17,96,210,237,89,951,523,342,457,93,730,57,37,39,661,16,83,437,373,546,269,49,18,14,531,8,717,318,111,45,24,569,247,76,533,719,221,67,326,282,468,386,203,52,44,38 }, +{ 104,289,707,66,712,173,97,214,90,64,426,20,874,414,568,43,65,32,194,715,7,204,312,560,74,500,192,835,324,0,38,376,819,861,147,319,95,636,22,86,729,21,470,879,288,5,107,526,52,81,178,2,751,49,10,519,142,595,427,247,77,356,232,249,4,750,237,597,28,885,784,36,889,858,51,202,802,14,135,298,115,13,59,524,808,1,330,325,23,92,180,359,392,31,24,583,681,61,18,259,172,48,177,123,825,798,37,357,12,208,33,174,387,362,140,102,35,76,40,17,137,198,11,216,600,16,671,91 }, +{ 84,65,56,18,194,249,0,108,123,643,847,554,63,524,416,141,453,24,26,403,22,511,467,630,475,128,73,137,33,9,80,899,95,64,370,76,45,16,133,42,289,19,650,214,27,240,730,719,210,196,119,114,49,494,402,304,102,90,32,11,10,588,561,283,235,202,523,183,25,736,609,580,576,500,457,433,357,349,343,209,124,60,57,39,28,3,1,757,150,23,368,670,51,597,947,813,667,659,529,431,291,287,243,236,233,216,179,177,153,134,122,117,112,89,85,72,31,30,6,421,389,165,125,895,874,619,519,361 }, +{ 21,350,308,13,346,197,826,352,101,841,570,68,165,509,23,115,100,694,749,569,903,51,212,401,9,0,221,734,391,154,777,252,88,124,610,265,237,39,558,125,48,30,217,230,421,35,289,468,64,55,535,475,213,546,175,909,1,403,382,166,589,28,645,812,419,180,32,706,98,593,410,703,759,141,317,629,65,241,83,194,818,773,22,321,752,689,923,341,962,96,671,699,158,95,16,329,429,854,264,724,190,439,31,722,316,72,900,144,10,93,247,49,831,676,372,24,223,36,553,269,293,69,365,490,191,5,639,251 }, +{ 92,16,60,24,7,71,126,18,35,140,330,14,246,46,50,267,121,146,236,108,75,220,199,571,394,68,309,150,132,107,314,448,230,54,232,633,5,366,118,2,21,9,278,38,388,37,87,64,1,181,328,12,641,55,70,672,582,726,48,526,0,622,172,516,563,86,69,32,103,10,133,36,156,167,28,65,101,135,195,52,171,950,149,22,830,189,99,649,191,451,67,13,29,262,294,606,83,545,58,517,462,271,23,572,95,163,696,128,675,594,620,392,514,77,785,45,155,192,217,653,335,279,522,212,237,51,177,565 }, +{ 15,82,898,365,33,77,341,180,115,901,21,10,102,197,55,5,4,351,13,165,117,36,11,1,217,23,317,141,40,752,120,153,450,134,253,12,2,260,143,196,873,51,89,457,32,3,177,352,349,854,372,244,160,59,7,719,961,64,202,96,6,110,291,137,523,498,111,0,210,482,28,31,30,17,951,125,318,22,128,509,139,811,730,386,169,304,9,391,37,164,48,326,269,203,93,373,237,148,535,342,72,569,704,308,247,38,533,49,98,922,86,734,109,14,45,275,216,95,817,403,44,437,57,899,661,41,39,968 }, +{ 16,24,35,71,18,68,108,140,189,69,103,60,267,50,149,118,230,7,1,133,191,92,46,167,9,75,246,0,2,443,309,236,486,335,14,54,212,330,218,516,65,67,38,394,641,21,181,55,28,101,48,83,399,366,563,412,545,114,220,696,388,51,328,13,23,5,64,458,273,124,126,12,238,487,52,345,37,537,87,121,29,915,723,150,175,32,302,154,163,361,146,375,674,22,10,171,271,96,514,251,86,433,70,336,36,571,622,223,906,796,237,726,398,602,283,39,262,115,702,565,891,58,695,562,199,128,894,49 }, +{ 60,20,16,43,7,71,18,92,118,14,35,628,608,68,154,28,375,126,175,107,0,279,150,173,22,356,97,149,566,55,153,169,49,140,69,33,13,230,117,38,23,10,362,51,2,77,783,414,421,109,298,246,359,196,125,124,446,102,104,386,722,66,21,1,601,575,621,326,439,429,12,392,134,32,882,192,74,345,551,533,419,185,336,191,4,232,37,273,595,81,189,791,314,202,158,917,576,467,570,6,5,302,48,422,115,801,319,197,165,677,535,925,420,387,9,423,399,251,517,515,403,291,199,260,694,553,64,40 }, +{ 187,258,871,295,434,489,219,201,224,268,839,384,699,940,110,261,116,160,131,44,36,574,228,144,141,93,137,567,406,244,926,120,98,253,260,373,64,442,206,269,127,908,318,55,31,911,959,855,617,10,1,531,332,523,202,613,203,661,326,352,860,217,717,367,276,21,51,143,180,96,23,105,13,450,22,385,533,284,662,197,851,86,77,251,30,115,83,9,48,816,164,282,811,317,365,39,856,165,351,843,669,473,899,341,125,922,902,752,291,817,372,918,345,417,177,728,82,293,242,951,498,968,854,386,559,15,349,49 }, +{ 66,97,7,172,192,712,232,204,324,387,74,426,448,526,104,173,107,568,560,43,356,825,808,500,729,135,359,52,707,92,32,95,330,20,636,64,65,22,140,38,138,81,298,2,289,715,885,5,802,392,29,49,14,174,0,889,91,86,226,348,90,750,861,6,357,280,314,312,126,214,246,792,353,10,36,24,380,580,21,835,37,28,414,58,230,69,222,595,362,194,673,16,278,12,77,85,33,46,874,18,819,61,60,681,147,858,888,612,600,13,241,25,501,102,48,35,67,189,87,71,40,4,583,519,779,1,237,150 }, +{ 15,1,5,341,4,13,0,3,2,134,33,12,11,7,10,77,6,9,21,23,40,197,8,32,365,37,17,901,111,102,51,117,180,317,961,153,217,165,59,14,22,16,82,141,31,457,94,202,115,291,453,403,196,57,36,372,475,318,719,28,349,18,401,72,269,99,48,386,110,498,64,494,120,352,93,351,54,139,27,509,474,177,45,44,30,670,90,421,326,76,56,89,128,898,642,629,304,260,450,19,482,137,29,55,391,873,342,24,49,730,203,62,373,143,210,79,752,41,39,758,661,817,157,152,98,523,579,402 }, +{ 0,24,35,28,124,68,9,154,65,16,109,149,55,175,158,71,39,252,7,419,14,108,101,346,168,22,185,289,429,49,18,420,169,67,439,671,703,336,302,1,159,92,610,962,484,118,333,95,69,2,677,398,344,100,364,810,272,345,105,46,329,114,60,535,265,327,577,30,10,189,21,12,254,286,133,194,83,23,140,793,553,86,36,50,570,637,13,103,5,722,310,51,374,230,48,38,64,32,747,706,505,701,273,251,150,164,601,455,166,627,143,634,125,283,424,239,458,799,128,246,267,388,547,6,137,96,37,538 }, +{ 7,92,14,16,46,140,220,60,35,24,2,126,38,87,75,278,71,172,262,135,52,314,107,54,294,246,394,150,18,50,330,29,309,267,230,171,1,70,232,108,21,5,121,620,97,236,69,622,32,572,563,950,366,409,58,199,12,48,64,328,37,571,189,146,448,132,66,68,155,872,112,192,461,67,441,181,675,279,13,298,28,226,0,451,23,163,423,606,9,582,128,20,86,526,36,334,307,55,380,462,888,357,335,61,43,10,22,45,83,337,331,186,353,359,103,726,133,633,95,51,65,202,72,74,785,114,141,149 }, +{ 92,7,126,16,314,172,60,140,35,135,278,14,232,2,107,246,46,150,330,24,66,38,71,230,52,380,97,298,279,448,563,192,357,262,199,121,87,18,582,309,462,441,29,75,74,132,220,5,146,451,526,324,423,50,21,550,69,32,68,226,508,1,70,37,633,64,12,108,606,267,48,572,493,54,189,726,366,0,392,359,22,95,43,58,571,603,28,10,20,86,236,55,171,600,36,9,388,163,65,607,387,6,67,49,294,649,830,583,356,362,90,622,394,620,167,259,939,149,443,83,580,114,25,103,118,672,61,45 }, +{ 235,26,53,297,6,436,73,113,27,445,42,25,250,611,90,19,65,214,648,614,85,624,289,64,122,200,491,684,416,32,17,194,45,665,145,34,80,95,573,151,112,322,86,680,714,315,630,106,690,14,29,472,255,141,396,56,312,57,22,247,76,209,4,128,61,288,123,84,352,249,11,874,108,3,138,5,217,256,70,119,59,74,0,835,283,161,114,751,269,597,525,389,136,77,51,10,40,115,287,47,2,52,78,176,881,62,13,524,519,23,137,33,133,28,46,177,746,311,18,236,12,63,180,453,237,631,179,243 }, +{ 352,141,752,854,36,1,351,217,230,244,661,110,816,258,160,224,180,219,82,201,269,567,144,295,260,373,922,93,120,203,489,64,35,202,406,317,206,699,137,116,268,871,717,10,839,251,434,228,864,253,326,96,21,131,9,30,83,384,44,442,39,450,574,69,332,48,318,418,728,898,55,127,187,642,365,86,276,88,28,341,831,498,22,901,443,100,367,523,128,531,940,213,5,689,748,190,860,679,0,31,811,843,559,164,7,669,143,662,851,77,564,372,165,335,98,800,261,2,959,349,284,170,115,291,824,345,50,855 }, +{ 126,60,16,35,107,18,191,121,68,92,279,446,598,463,375,43,167,493,7,20,118,199,212,566,14,150,97,28,575,608,589,55,407,101,246,154,146,24,346,21,0,175,48,71,362,815,354,49,628,132,939,5,467,22,649,10,9,1,356,259,147,271,149,783,439,140,23,108,32,109,230,376,683,125,13,621,345,441,363,31,51,647,612,273,124,562,173,695,768,105,38,252,2,727,169,818,640,195,755,590,637,158,722,364,104,37,953,535,12,44,64,66,359,33,202,806,298,46,419,98,221,95,77,185,660,399,653,520 }, +{ 1,15,2,5,4,3,13,0,23,11,7,6,341,115,77,51,10,9,12,33,180,197,102,40,217,165,141,365,901,21,317,32,202,8,14,64,177,22,269,352,59,17,291,31,569,318,125,210,16,93,386,453,196,457,401,18,494,403,153,752,37,28,475,961,117,110,498,120,342,523,139,116,629,44,326,134,349,203,98,372,49,128,275,57,308,48,237,391,719,30,509,24,95,36,86,19,421,260,758,247,90,730,615,216,137,373,450,873,111,482,144,29,899,304,817,670,25,300,143,801,55,20,244,82,45,27,39,242 }, +{ 51,23,13,202,21,5,403,137,32,141,318,128,1,120,165,125,12,48,15,475,269,180,453,197,115,64,401,33,2,7,450,6,523,308,260,421,221,4,494,37,386,99,16,14,196,457,730,237,93,817,102,77,61,82,352,9,719,98,110,10,20,899,326,19,18,3,117,90,17,391,402,95,0,341,31,11,217,25,961,629,351,96,134,72,49,153,30,26,569,801,203,372,40,509,24,50,317,285,27,758,116,349,342,70,59,58,76,39,44,291,89,57,45,29,854,28,22,144,177,35,194,576,86,670,54,365,678,247 }, +{ 131,716,371,224,737,385,219,254,616,924,836,187,258,98,761,916,944,127,39,105,442,578,9,49,272,31,688,44,116,242,137,871,908,159,395,518,890,28,293,284,574,202,623,51,160,23,0,201,276,36,827,938,839,22,333,125,456,299,286,144,109,438,158,13,95,910,64,30,829,228,327,166,101,1,860,926,613,96,86,512,846,310,21,295,10,406,55,318,252,320,93,843,128,301,627,567,203,165,48,662,492,100,268,37,531,457,221,190,434,403,68,384,421,239,393,261,902,35,533,169,170,120,110,260,244,308,5,506 }, +{ 15,5,1,2,3,4,7,0,6,11,13,10,180,197,12,217,341,141,33,134,77,17,165,9,352,40,365,32,317,64,498,752,102,21,117,901,59,14,237,8,23,291,349,36,864,269,372,153,57,37,82,961,22,115,569,196,120,509,873,342,202,48,51,351,110,16,93,45,31,457,125,19,76,72,111,244,27,260,128,143,318,79,546,56,139,28,18,86,44,98,90,391,94,61,55,854,95,38,386,29,482,898,49,99,89,253,210,177,137,730,450,35,221,84,25,304,954,758,326,922,474,403,203,20,373,50,247,523 }, +{ 66,74,192,173,222,226,174,7,29,712,52,204,353,43,20,500,2,138,145,97,104,673,426,355,6,64,560,87,0,580,331,172,414,90,32,25,337,65,22,49,135,825,861,112,280,5,287,81,95,525,86,21,707,42,38,792,415,85,750,889,356,36,324,819,10,632,232,45,526,330,1,107,544,91,319,140,746,151,294,194,58,92,28,14,67,289,4,636,115,37,17,12,262,715,470,16,155,59,27,61,233,387,298,18,376,568,216,26,122,69,77,113,24,23,888,70,13,114,51,75,448,729,11,609,102,348,312,33 }, +{ 786,259,5,534,590,493,465,929,581,21,279,941,49,13,612,132,32,362,23,147,61,48,35,121,939,626,683,146,178,107,771,687,408,347,713,345,58,427,126,70,18,60,4,197,878,150,344,640,379,1,195,841,33,120,263,356,51,509,522,260,28,22,0,115,55,10,376,663,207,350,199,16,423,142,407,221,156,246,623,595,241,105,31,903,298,11,71,68,95,92,64,17,40,450,441,90,167,308,165,518,44,783,316,247,953,360,351,98,30,102,82,65,159,955,290,87,817,134,20,182,39,24,230,117,59,501,315,569 }, +{ 278,14,135,2,140,16,267,69,92,24,7,46,35,189,38,52,71,29,60,75,357,172,87,309,66,380,262,74,324,230,462,220,1,451,150,335,226,571,108,18,97,50,443,606,563,103,67,246,550,6,641,126,330,70,388,366,5,83,149,68,21,486,48,314,25,12,54,565,298,394,600,622,32,423,294,232,37,582,236,222,572,28,65,251,603,36,192,64,58,145,0,171,331,55,950,853,95,133,86,22,10,114,112,399,620,138,448,26,163,353,508,500,583,96,45,877,238,42,726,9,785,121,279,181,696,72,19,223 }, +{ 165,13,308,391,197,23,401,509,758,457,569,51,629,734,180,115,873,317,901,961,3,453,719,29,46,546,80,34,2,482,217,468,181,867,48,106,45,341,52,17,14,287,297,5,730,445,386,552,428,62,221,186,15,11,171,56,50,12,38,365,250,328,133,61,211,540,153,215,119,53,41,903,670,58,743,311,99,305,231,176,27,236,4,54,130,881,403,151,6,600,57,136,84,40,764,787,114,475,862,472,469,73,32,26,33,400,227,137,21,1,255,196,270,281,155,65,752,635,440,405,326,161,110,549,477,430,396,592 }, +{ 1,14,22,2,28,7,67,168,65,24,49,0,35,46,55,50,95,69,12,159,36,105,16,114,109,158,5,83,194,103,38,133,239,169,9,289,164,185,18,424,6,21,10,708,124,170,320,283,312,389,29,68,51,23,420,143,189,347,505,336,37,175,835,892,86,149,108,64,874,54,13,577,251,48,32,654,756,547,96,71,115,301,286,128,154,202,236,344,160,253,484,638,52,75,208,230,747,610,308,39,218,533,137,140,267,141,374,3,345,31,127,237,799,98,601,597,244,272,323,125,398,401,118,17,327,99,412,101 }, +{ 34,453,130,196,322,3,14,236,361,47,51,377,11,730,514,711,4,412,153,57,161,440,485,102,176,59,162,108,17,62,283,757,202,56,136,38,328,880,77,494,184,12,719,128,402,403,200,23,78,117,27,475,21,300,40,381,54,472,33,53,183,152,129,99,2,714,452,13,585,803,80,210,862,555,383,73,46,123,37,139,84,670,556,503,548,6,507,421,5,111,527,29,19,342,229,457,26,775,134,318,469,133,115,90,106,257,277,401,250,497,205,1,543,266,211,542,629,899,45,488,215,217,65,72,275,339,432,32 }, +{ 48,18,23,21,35,60,16,24,156,51,13,71,121,221,5,132,46,141,163,7,1,43,271,202,115,92,411,178,340,137,447,308,32,363,150,698,125,223,195,781,541,318,147,142,28,22,0,108,360,621,401,241,2,463,354,49,658,586,67,303,269,97,20,761,50,296,146,70,65,763,867,691,638,182,647,93,82,10,767,167,31,914,670,290,128,107,75,61,225,98,64,4,120,14,12,758,110,29,949,457,316,165,105,95,293,30,668,932,180,39,446,724,743,666,421,391,372,351,114,96,86,66,591,352,33,942,922,459 }, +{ 13,165,629,457,961,569,23,197,341,401,41,758,901,719,17,509,51,115,38,421,62,54,157,12,32,45,117,57,37,339,328,106,188,14,365,734,99,546,488,477,76,873,2,453,308,542,248,72,10,184,40,721,214,211,4,386,305,479,474,152,3,243,90,139,11,5,1,153,313,21,468,134,471,217,148,129,123,179,77,89,670,255,162,130,94,7,391,136,59,102,111,536,196,828,247,237,743,428,6,514,277,556,128,33,475,155,266,229,29,528,209,720,413,241,64,361,476,183,9,0,519,405,317,202,181,805,775,431 }, +{ 77,141,2,102,33,1,64,6,5,3,7,269,202,128,180,23,318,14,0,10,13,291,352,153,40,197,165,137,237,51,17,177,15,217,117,115,196,9,86,4,139,134,11,19,22,12,210,403,16,342,95,125,36,457,317,21,498,752,349,275,143,59,341,120,27,93,82,18,28,111,494,110,164,38,25,365,901,475,391,57,8,203,509,260,31,961,49,453,386,48,678,351,326,56,37,719,402,99,401,546,372,569,44,65,304,864,32,730,300,194,26,45,244,873,277,29,285,41,208,597,20,253,160,523,24,98,54,312 }, +{ 212,68,154,124,562,101,191,0,9,375,118,520,167,149,818,65,16,722,302,21,777,35,24,71,604,350,100,806,252,478,346,264,664,841,1,759,265,48,13,695,429,55,701,329,51,23,724,398,589,18,936,115,882,484,755,88,749,69,903,125,189,410,22,221,535,382,108,308,166,917,411,39,28,223,67,92,509,32,10,197,64,5,694,650,114,558,95,246,60,909,7,83,241,96,165,363,391,593,98,230,321,103,364,826,553,925,639,49,458,734,628,419,336,447,768,399,140,105,175,37,886,682,905,674,821,404,109,812 }, +{ 48,23,51,98,125,541,598,259,293,13,807,126,21,115,221,493,963,223,35,31,670,386,121,0,107,770,165,144,90,203,116,719,949,783,727,279,590,197,32,326,10,284,146,49,401,407,132,64,199,16,955,342,97,39,834,1,194,453,191,522,427,114,105,92,30,14,7,465,393,763,271,50,725,615,195,108,22,77,44,9,640,291,20,202,269,941,767,671,591,498,467,360,147,128,28,704,551,389,289,33,150,141,60,43,697,534,362,589,58,325,621,523,475,403,260,237,218,210,140,137,127,109,102,87,83,46,36,867 }, +{ 21,23,13,5,1,586,32,48,167,51,781,31,647,33,271,165,49,141,591,44,115,522,77,197,221,22,360,18,93,146,391,558,105,932,217,447,118,153,961,352,60,900,317,761,196,459,0,363,260,241,82,4,12,698,463,7,291,98,27,247,815,156,120,36,623,35,10,191,628,95,65,46,17,50,922,341,202,45,411,64,521,372,102,16,934,308,269,663,410,351,326,14,116,686,349,225,110,457,29,2,509,71,621,854,701,475,354,342,159,158,132,127,24,478,446,321,246,199,640,848,777,759,371,318,316,273,223,126 }, +{ 2,29,61,52,7,1,5,14,58,70,45,425,6,290,112,738,32,207,46,72,66,186,87,76,655,38,243,263,74,151,21,75,16,227,25,50,135,17,148,64,89,255,292,128,24,106,155,141,315,476,209,226,90,26,833,19,113,27,306,115,48,454,549,592,18,4,163,307,0,20,119,77,10,86,36,270,54,177,35,171,720,22,65,137,496,202,40,887,59,179,95,294,790,23,69,269,220,172,631,334,9,67,449,28,408,318,483,311,145,13,97,845,84,287,240,214,51,37,43,241,405,33,49,102,663,80,12,180 }, +{ 39,265,9,333,100,310,411,254,363,101,447,404,18,272,98,639,264,166,1,286,203,327,682,676,31,905,156,520,821,16,93,656,593,30,625,354,105,24,438,44,664,688,23,116,51,557,0,326,329,21,812,195,49,190,159,35,759,60,478,125,118,121,48,364,132,627,13,167,33,110,921,242,68,274,158,96,127,252,137,668,77,5,731,578,692,102,795,865,131,435,165,22,604,586,271,473,455,382,146,120,10,88,144,65,64,966,212,71,67,128,83,691,897,160,576,551,36,724,615,223,82,50,647,7,95,197,621,115 }, +{ 6,26,2,25,74,1,29,19,42,138,51,108,114,70,324,14,23,386,133,103,5,96,85,13,719,75,453,52,27,66,730,602,220,38,309,65,388,457,58,17,112,122,50,128,323,357,64,235,280,238,670,421,135,165,150,475,7,523,197,32,145,278,87,500,899,401,550,3,218,273,758,208,113,86,83,603,532,222,115,711,12,137,21,895,394,46,24,163,53,45,97,67,125,283,28,458,236,117,34,11,262,294,90,36,801,792,141,516,484,341,302,33,802,297,204,57,95,4,756,594,226,256,134,480,891,545,99,450 }, +{ 20,43,356,414,49,595,858,359,819,107,392,319,97,7,422,470,376,612,173,5,246,0,427,14,683,147,197,22,660,95,230,347,945,713,32,344,92,61,830,362,325,259,33,31,21,12,4,904,707,623,126,509,18,2,77,357,232,165,140,105,102,729,196,153,426,117,408,841,518,98,64,10,17,16,501,663,560,76,44,104,38,558,125,391,23,324,178,134,46,54,29,13,40,51,330,120,493,279,168,581,263,180,158,116,241,455,159,81,28,903,298,90,900,66,465,441,423,202,198,30,534,237,58,24,3,52,887,845 }, +{ 68,35,0,101,65,9,124,154,175,114,336,67,28,69,149,189,103,133,1,24,39,83,50,230,55,345,16,429,251,7,283,218,252,346,18,389,547,71,118,364,100,2,238,140,646,109,46,22,12,439,166,289,36,23,49,484,30,399,14,51,732,13,86,930,532,398,64,96,54,323,694,88,21,722,158,265,5,75,38,48,329,181,705,212,194,952,480,361,419,443,208,302,868,782,520,10,95,108,503,128,191,115,246,237,335,267,637,37,167,160,137,723,702,141,806,565,309,105,264,831,826,701,604,535,60,330,562,375 }, +{ 23,13,51,115,15,5,202,12,1,165,2,0,21,197,453,457,4,318,269,10,32,7,141,9,6,401,3,180,14,403,37,11,341,77,22,719,120,569,217,475,308,177,33,18,260,365,17,961,64,59,99,16,317,237,48,28,629,128,450,730,90,117,137,386,352,523,31,40,494,291,102,247,8,421,509,72,24,498,125,391,196,76,49,44,57,221,36,93,468,203,326,30,45,153,210,899,89,304,216,482,372,38,110,95,82,19,275,134,402,758,342,139,349,98,901,678,752,670,873,546,734,27,56,351,65,817,61,86 }, +{ 121,195,407,126,60,271,146,199,167,16,107,493,98,132,191,223,35,279,1,212,20,362,727,92,48,43,18,21,150,522,446,259,393,467,5,783,590,598,360,640,612,939,649,354,71,246,7,156,834,68,49,125,293,24,0,147,55,608,591,163,14,376,534,31,660,541,463,683,807,97,13,345,589,23,953,668,562,22,32,818,221,108,955,118,284,96,581,595,10,50,28,470,765,695,575,346,116,2,37,61,356,535,566,95,12,51,105,101,70,281,64,273,423,230,202,441,44,9,755,75,697,653,465,38,768,173,175,77 }, +{ 31,44,299,116,393,492,144,268,918,367,456,434,105,489,22,948,127,384,963,9,98,28,0,49,125,242,293,206,395,940,36,512,55,51,203,1,855,93,911,617,159,95,272,23,959,332,763,301,137,21,13,141,64,109,86,282,10,170,120,286,320,551,202,417,599,239,165,685,7,2,616,5,128,730,143,457,352,719,48,39,8,253,160,37,692,326,327,386,450,421,158,269,12,276,18,506,791,473,115,24,16,244,391,164,558,318,403,338,623,131,219,14,523,401,284,30,197,499,900,951,308,479,221,260,728,169,910,935 }, +{ 23,457,51,401,13,758,719,165,629,453,961,730,197,901,386,569,308,670,391,3,29,14,12,739,231,34,52,153,832,828,115,341,2,40,46,514,176,365,5,21,33,0,838,181,4,328,45,54,555,130,509,494,540,734,552,583,215,196,507,227,11,211,99,117,361,180,38,10,1,351,894,134,27,26,186,50,48,543,152,749,369,202,62,236,421,102,600,468,403,171,161,155,57,9,528,17,880,753,36,477,19,440,903,488,377,22,585,108,37,7,6,697,899,757,523,475,576,326,67,479,469,281,867,805,801,659,657,587 }, +{ 105,131,22,36,127,115,180,341,201,219,272,31,365,169,98,1,44,317,9,64,239,187,141,197,498,143,873,320,752,286,901,352,86,164,160,258,327,244,224,185,109,159,217,13,535,253,538,144,116,0,374,276,261,776,170,28,83,634,531,2,569,533,269,616,95,896,251,824,482,165,37,55,128,282,10,5,578,49,546,654,947,913,371,384,295,208,8,860,21,137,202,954,312,284,30,23,567,509,125,574,39,285,318,268,206,203,843,623,33,708,12,77,716,385,837,7,638,120,237,301,254,308,293,16,951,3,395,15 }, +{ 6,1,2,29,14,25,52,38,74,66,26,65,67,114,42,19,108,5,69,83,46,388,75,86,324,135,103,7,17,27,85,380,36,24,70,150,278,87,138,97,594,886,357,267,50,51,18,484,650,23,458,45,646,21,732,398,95,208,64,96,273,13,238,0,124,22,711,550,12,532,323,302,28,220,160,37,335,63,32,194,421,133,262,451,602,283,386,309,730,10,172,467,453,235,168,3,401,782,115,610,670,48,719,81,145,16,20,500,137,226,58,280,165,603,244,866,141,308,222,197,189,112,547,53,99,73,475,180 }, +{ 51,453,386,23,719,730,670,13,758,6,457,401,899,475,629,19,2,29,569,47,26,165,14,894,421,130,801,236,197,108,412,27,74,1,308,754,5,50,17,961,133,45,52,285,523,711,657,78,324,196,128,137,65,34,115,25,99,138,357,59,480,42,114,46,361,867,53,739,58,4,743,403,603,366,11,859,38,262,12,594,202,22,235,153,64,391,218,0,516,487,10,145,200,3,33,444,70,117,901,57,223,134,54,745,390,112,283,95,123,309,96,21,66,803,220,86,341,18,365,402,176,37,328,77,488,73,322,250 }, +{ 252,121,18,101,16,9,167,132,271,60,146,419,364,35,265,363,212,0,158,604,329,682,68,647,109,848,907,191,520,463,39,24,333,28,446,159,821,55,621,553,118,126,593,411,49,71,407,23,286,634,105,195,354,107,404,857,13,676,374,310,259,22,538,562,92,100,31,10,21,793,199,1,327,639,108,48,98,806,51,7,709,346,522,932,166,147,5,590,96,137,254,465,272,20,150,694,64,706,12,14,203,169,95,36,124,77,2,116,360,44,865,402,125,154,128,33,627,576,185,810,455,156,822,777,37,326,127,842 }, +{ 283,503,128,432,269,529,63,26,275,789,193,678,318,389,102,122,291,77,59,177,256,6,862,342,25,18,235,57,216,445,27,4,573,605,349,65,11,85,403,523,643,84,813,630,494,397,233,40,56,73,475,202,138,416,210,718,42,32,133,415,381,736,453,412,511,19,619,91,240,255,80,22,64,114,23,289,194,0,17,123,561,87,174,152,343,81,436,66,300,297,148,95,78,3,217,51,803,510,370,341,139,137,130,49,45,119,396,372,90,237,544,141,74,476,129,86,21,650,421,378,352,209,196,33,288,247,402,701 }, +{ 2,75,29,52,87,74,226,69,220,1,278,135,46,262,70,267,14,380,6,66,324,38,357,294,7,394,222,140,145,97,331,25,335,189,24,67,92,16,112,853,571,26,42,309,163,50,138,35,550,582,103,172,83,96,337,451,71,108,150,5,58,486,785,537,21,60,298,19,462,583,600,461,171,32,230,48,65,366,45,500,606,54,774,12,18,36,149,223,114,37,443,641,133,632,287,563,64,409,85,22,764,388,314,238,151,950,95,61,27,307,155,86,28,415,565,17,246,423,43,572,113,0,186,609,251,396,10,802 }, +{ 97,298,172,140,69,66,7,189,267,92,501,314,60,423,135,24,232,81,71,16,43,150,462,74,14,330,192,517,173,426,733,149,251,35,885,273,6,279,38,18,221,67,278,508,83,607,2,91,380,52,125,448,124,29,387,458,87,20,36,443,230,861,108,915,711,302,104,21,568,526,723,324,12,68,48,5,362,1,336,335,0,28,22,46,541,126,357,103,25,451,877,726,246,398,565,50,138,879,175,963,65,86,165,809,391,715,197,705,191,118,117,42,712,386,204,160,823,674,54,107,467,399,441,633,831,572,563,236 }, +{ 475,421,805,899,403,453,958,615,817,51,801,523,120,401,670,730,450,326,494,23,576,202,386,629,569,260,115,365,697,77,165,110,153,351,318,24,117,13,317,0,341,221,16,704,137,5,196,342,308,757,543,22,719,197,11,4,457,20,9,302,484,758,300,10,7,217,6,193,901,391,91,361,1,791,515,862,291,273,118,102,66,38,36,2,402,125,114,661,643,352,474,134,82,33,133,756,642,511,48,373,240,847,735,608,480,172,135,52,43,21,14,859,244,116,29,370,124,754,180,35,42,898,886,851,842,811,770,558 }, +{ 7,128,20,14,177,97,16,112,43,202,269,77,107,66,415,318,2,135,141,275,147,38,18,58,35,52,307,173,10,356,298,29,210,414,172,32,6,0,137,192,24,291,70,22,102,494,9,87,334,13,104,60,115,64,40,678,342,46,49,186,33,217,5,121,48,90,23,21,4,155,86,232,403,59,28,74,132,37,227,324,319,61,362,402,36,126,95,92,431,337,178,400,609,1,376,71,51,294,665,631,392,151,549,105,81,359,353,304,230,196,67,349,15,134,544,311,285,278,884,352,259,76,427,262,194,50,496,171 }, +{ 24,14,7,28,2,16,0,22,124,1,65,35,108,168,149,67,109,69,71,18,46,55,158,49,289,12,92,95,185,38,6,169,5,420,83,10,189,154,159,584,140,150,36,68,429,164,21,671,484,50,398,60,105,175,29,302,143,703,239,103,86,23,64,344,424,32,51,9,336,251,230,273,577,114,747,133,37,118,194,66,13,48,267,97,128,835,505,52,170,137,320,892,253,115,236,570,388,347,374,125,286,54,20,345,202,458,314,638,246,654,610,25,75,708,17,172,312,96,135,677,141,19,221,3,650,799,547,601 }, +{ 26,80,122,73,27,25,42,684,19,209,445,573,667,17,680,690,106,6,255,297,611,483,4,495,59,119,250,45,84,378,113,469,65,436,128,311,85,64,29,53,200,56,648,396,235,61,742,11,136,76,383,77,0,86,32,161,62,22,814,361,256,14,12,5,137,123,40,389,194,491,503,813,283,269,862,21,3,95,112,454,179,63,141,129,33,177,233,57,202,803,2,102,70,10,34,240,402,18,416,624,275,145,605,614,52,358,504,665,527,343,133,114,108,139,50,881,151,58,205,51,790,24,210,38,174,49,216,285 }, +{ 107,259,465,362,376,595,534,660,683,470,581,147,612,20,858,43,427,49,5,178,246,786,0,35,97,356,279,878,21,423,929,61,142,14,230,126,70,132,146,904,590,325,319,493,22,199,48,32,501,392,298,953,95,77,414,640,347,345,359,798,31,13,198,7,173,58,64,121,105,711,28,125,150,344,12,518,10,941,713,603,4,60,33,450,1,481,92,202,98,120,687,55,207,23,16,18,626,197,102,522,819,163,232,771,113,577,17,318,37,65,357,263,40,324,422,260,623,59,44,663,408,702,494,441,68,455,845,237 }, +{ 131,30,190,856,228,406,224,530,88,863,219,778,274,944,613,539,816,258,187,100,435,871,39,160,321,840,44,116,31,166,93,625,137,36,957,531,127,442,213,669,466,716,385,1,144,839,492,202,827,829,96,382,908,242,662,731,9,254,120,367,276,456,371,960,203,795,201,98,730,574,83,13,924,410,800,926,326,21,217,851,318,523,836,717,268,473,931,352,533,418,23,49,141,28,260,110,77,22,86,310,567,494,457,64,10,55,244,737,251,373,728,128,299,125,105,0,51,48,679,822,489,5,616,265,295,284,615,102 }, +{ 113,311,611,6,491,80,667,25,209,45,26,27,684,42,122,73,85,396,255,145,106,17,614,29,648,297,690,19,454,61,64,624,4,5,416,445,112,32,2,138,76,65,74,151,483,90,174,66,665,119,22,287,1,59,86,378,128,833,194,573,881,141,70,52,0,525,742,214,315,243,544,680,14,87,216,469,11,746,179,250,495,887,95,12,10,222,21,348,814,289,280,204,84,72,436,77,123,56,33,114,177,235,53,137,102,81,161,631,136,46,790,503,58,240,292,208,200,153,36,202,63,7,186,50,135,57,49,37 }, +{ 15,0,1,5,23,9,51,13,2,7,12,120,115,77,260,82,180,341,202,4,450,141,10,8,351,197,21,901,6,817,403,137,102,14,64,22,165,342,457,269,93,16,3,32,11,110,37,128,318,177,401,40,494,18,365,523,854,453,33,36,421,352,475,217,28,210,922,31,569,898,317,125,44,719,30,730,59,86,144,24,48,203,961,17,49,373,116,98,99,308,326,117,752,386,576,153,55,291,39,629,899,95,221,196,139,642,661,402,237,143,244,134,160,90,509,670,391,65,498,349,89,54,127,57,805,275,253,758 }, +{ 219,127,258,98,395,924,421,276,293,697,944,567,201,284,453,401,242,512,308,51,860,475,23,105,295,131,137,272,770,301,843,670,116,299,578,386,935,125,261,13,763,31,576,456,791,115,44,492,403,221,662,268,9,730,801,269,202,49,365,964,55,805,616,159,699,943,22,141,506,224,327,206,541,21,551,144,39,393,165,326,0,842,535,203,36,743,910,28,170,332,318,282,494,949,716,253,569,197,1,143,10,341,533,5,160,615,457,391,916,317,911,187,737,286,384,93,24,599,109,371,180,900,783,244,352,12,623,48 }, +{ 15,352,752,317,180,217,854,141,351,82,1,898,36,341,260,115,23,13,365,21,120,110,10,33,291,922,64,901,372,77,349,160,143,244,51,137,32,498,661,269,5,4,202,304,197,48,253,40,237,102,450,457,373,318,11,22,35,482,251,67,55,9,719,165,59,117,72,153,177,196,12,83,28,961,579,247,326,45,96,17,864,69,523,2,468,125,0,86,730,811,128,817,134,93,90,148,144,437,37,111,403,342,76,57,546,230,704,345,308,139,89,58,873,29,203,164,531,50,30,3,275,559,127,855,629,473,8,56 }, +{ 129,123,257,184,162,17,205,249,183,769,80,119,497,229,106,5,45,130,84,90,59,152,3,99,128,4,618,27,57,77,245,40,62,452,556,378,19,214,460,136,14,12,240,56,26,2,202,432,6,33,61,266,11,209,402,403,64,21,269,102,381,25,141,29,504,76,139,318,54,73,305,34,217,619,179,1,10,32,177,38,361,207,475,383,37,483,22,210,255,115,494,58,196,524,65,23,710,775,53,306,153,742,86,180,421,36,469,137,117,15,51,216,48,134,313,0,111,72,161,285,288,7,757,250,880,503,151,372 }, +{ 102,33,128,64,77,23,51,202,141,269,13,137,40,10,196,22,0,86,117,153,403,318,342,210,65,1,115,139,15,95,134,475,453,180,165,2,177,217,14,6,494,291,197,99,36,4,59,49,28,5,7,11,402,120,12,114,194,3,18,9,19,31,125,237,82,352,17,16,21,110,275,401,44,260,24,285,719,349,25,93,326,457,164,351,111,300,216,27,317,421,133,26,37,203,143,312,116,108,8,277,498,730,365,678,341,98,244,20,386,523,308,105,29,752,372,32,485,450,901,391,361,57,283,48,30,208,96,38 }, +{ 7,14,172,107,92,2,16,135,46,35,232,294,60,97,87,5,278,24,126,29,220,52,38,21,262,66,192,202,58,20,70,48,43,171,359,54,32,448,356,337,37,353,226,112,75,298,1,526,18,12,872,132,64,409,140,22,155,387,334,128,121,392,357,318,307,461,331,0,150,314,49,146,61,415,86,69,71,620,673,230,494,380,324,6,362,355,330,45,259,77,888,137,10,13,246,95,186,141,74,23,50,279,147,194,28,25,51,9,4,609,151,177,55,849,145,199,31,115,102,550,597,59,36,216,90,17,441,222 }, +{ 576,403,615,805,523,475,326,817,494,421,120,51,450,859,137,202,453,260,670,402,77,23,551,401,386,33,958,318,704,342,269,197,196,0,165,480,373,323,65,153,899,753,629,10,803,697,102,7,569,59,4,2,351,90,12,13,962,677,110,9,801,118,40,6,95,5,128,28,143,730,446,124,122,114,14,917,306,207,82,621,407,86,16,164,115,154,64,842,886,848,735,661,484,397,302,247,174,149,141,71,60,1,719,216,104,81,925,628,457,208,168,49,25,22,844,824,685,338,175,109,259,810,236,933,932,854,804,788 }, +{ 141,352,202,269,137,217,180,372,349,9,18,0,51,115,23,13,16,365,221,318,165,317,2,494,752,7,125,120,77,98,20,10,64,49,341,291,22,901,95,260,854,351,308,55,30,28,403,144,93,44,922,873,326,223,82,48,39,35,21,14,197,203,24,153,33,102,753,498,473,156,88,65,386,32,717,275,268,31,196,177,817,730,704,678,661,482,450,437,393,373,288,241,195,132,121,118,110,105,101,36,1,947,475,237,5,253,244,206,201,168,164,158,143,128,961,954,864,751,615,576,551,546,523,509,489,453,442,421 }, +{ 23,202,51,141,13,165,1,21,5,180,197,269,318,137,48,128,15,7,403,120,12,102,125,64,2,33,14,326,523,450,260,93,37,16,386,32,9,4,237,6,115,352,77,203,457,153,196,82,18,117,0,110,421,494,453,401,40,24,817,98,351,475,99,221,730,144,391,31,3,44,96,291,30,719,308,36,39,341,342,11,217,35,8,899,50,20,402,854,615,922,116,160,72,28,17,22,45,10,373,317,89,88,61,59,285,509,752,365,372,58,19,901,46,139,70,349,25,177,65,86,293,90,247,498,54,576,758,898 }, +{ 25,151,145,6,122,174,29,138,74,665,42,416,113,614,45,746,525,287,348,311,396,544,85,66,26,81,280,243,624,355,52,222,32,64,65,680,4,59,19,0,491,2,22,90,7,204,194,104,77,216,112,445,1,61,5,128,86,177,651,91,214,17,95,209,611,141,648,297,80,415,21,102,87,233,454,33,27,43,881,503,20,137,636,573,10,18,14,117,289,76,58,869,202,153,337,496,186,196,789,106,315,11,40,28,135,49,667,269,192,36,13,57,12,115,134,750,70,208,63,72,73,46,597,529,51,48,684,173 }, +{ 166,190,382,254,30,100,731,829,438,264,274,88,795,131,613,578,716,625,9,93,921,616,435,404,265,1,530,203,31,44,960,36,39,827,160,127,272,326,98,924,96,473,224,23,410,116,252,77,144,310,137,856,105,219,101,33,83,0,863,110,800,13,812,51,128,406,478,228,125,521,120,639,21,49,836,102,593,217,242,28,897,772,931,5,688,957,286,115,64,141,691,269,656,317,321,10,494,159,187,197,291,86,363,576,944,67,22,450,201,523,55,411,373,82,822,557,260,456,704,329,367,225,276,48,202,268,213,32 }, +{ 132,21,379,5,32,13,23,195,626,51,48,121,70,1,850,296,687,883,156,771,61,49,407,95,146,125,105,55,652,4,98,761,77,58,33,518,260,18,623,586,349,340,120,82,523,522,354,163,24,16,0,293,45,87,786,165,929,75,64,888,763,535,308,102,31,698,40,269,202,159,37,35,9,845,622,207,663,153,117,900,608,386,817,591,351,347,310,241,221,199,170,158,115,76,65,59,28,22,706,223,774,640,629,461,415,116,101,71,303,60,941,638,427,421,401,391,372,197,196,127,299,937,854,768,475,453,450,403 }, +{ 254,9,272,0,49,688,39,101,578,131,159,518,158,327,333,105,286,68,716,224,252,28,890,344,616,310,846,829,219,827,228,613,24,35,364,16,30,18,938,371,406,166,419,65,455,22,95,627,793,709,98,910,124,154,51,100,31,127,48,1,21,385,190,23,320,242,137,187,374,347,258,36,44,13,301,404,109,856,116,170,125,118,96,160,442,656,424,865,438,931,67,168,64,55,239,623,128,737,265,924,553,221,202,10,71,677,86,83,88,916,60,329,293,538,7,577,512,5,761,395,912,421,212,144,816,944,506,346 }, +{ 16,35,60,20,7,18,126,68,14,43,92,107,118,101,55,71,28,121,9,154,0,97,109,24,621,150,2,191,419,252,5,202,279,363,917,22,21,124,364,169,439,604,199,49,346,446,175,158,12,167,575,48,38,13,493,246,185,10,362,32,95,51,259,375,1,39,598,628,932,566,421,6,23,64,29,108,146,925,61,46,356,494,318,132,140,345,694,149,359,147,65,407,329,376,125,403,697,273,302,212,96,69,137,50,848,25,467,298,77,453,173,230,570,826,221,66,70,143,677,535,265,608,551,58,815,128,907,100 }, +{ 15,1,13,2,341,5,0,11,4,3,10,7,33,23,12,6,117,21,9,40,17,901,197,8,134,77,22,37,365,14,32,165,317,153,102,180,217,457,202,51,961,569,141,128,115,318,38,31,64,19,16,291,137,269,196,372,111,59,44,36,25,352,349,57,99,18,509,72,30,27,752,120,93,54,45,41,342,48,110,49,453,498,79,177,28,719,94,403,402,386,89,82,474,29,285,139,86,482,56,237,361,95,55,391,26,277,116,873,34,210,76,758,326,50,39,143,90,328,260,308,247,125,898,164,24,494,401,421 }, +{ 78,444,47,19,26,390,813,130,108,453,711,27,412,730,65,122,680,283,690,113,51,235,236,73,128,630,196,297,14,250,123,53,194,183,200,133,64,445,368,90,719,3,381,42,23,389,269,6,472,650,249,56,475,32,59,4,57,45,34,648,605,624,503,114,17,25,736,256,22,803,141,318,153,315,95,214,0,684,119,99,18,432,216,899,85,289,46,457,243,33,11,436,480,50,322,177,29,152,129,54,13,657,304,285,84,52,401,117,881,255,714,202,665,597,2,961,12,339,38,361,77,524,115,58,24,145,670,402 }, +{ 2,29,7,52,66,61,45,112,738,113,74,14,5,151,6,315,20,592,70,128,32,425,16,25,1,222,87,135,177,43,64,491,186,59,38,655,21,72,76,46,290,90,243,141,115,77,631,58,106,19,337,207,145,137,4,24,454,833,97,0,26,40,227,311,402,192,36,27,89,210,35,292,269,48,86,10,202,209,496,18,217,22,102,226,138,17,887,204,122,65,318,33,28,104,172,51,255,13,23,42,148,644,216,790,307,287,174,180,92,69,291,95,49,415,75,416,483,549,179,275,263,270,107,9,334,869,214,155 }, +{ 145,112,74,66,26,6,29,70,396,287,138,87,135,222,226,25,644,151,42,19,122,307,651,624,635,337,648,45,2,113,294,58,85,632,849,5,416,75,52,172,61,573,445,27,1,315,884,665,64,7,53,204,32,461,95,17,97,297,174,163,436,220,280,525,192,65,792,73,14,186,86,609,881,353,36,311,549,72,243,22,194,331,148,500,235,80,69,46,680,278,38,312,48,90,21,106,37,409,491,128,10,114,614,631,209,67,415,227,50,108,237,250,580,155,76,141,171,544,262,24,620,12,43,133,611,4,334,389 }, +{ 92,126,60,16,7,24,35,232,71,14,150,140,121,246,314,146,172,132,199,107,18,267,46,278,572,230,108,451,330,236,380,388,462,135,68,38,830,633,392,279,97,2,448,309,649,517,672,32,357,21,5,75,9,118,22,55,167,87,582,192,526,594,366,156,359,298,191,195,64,653,28,50,189,324,563,13,220,0,1,66,467,12,10,423,441,149,733,52,262,877,48,29,70,95,49,516,133,36,54,508,407,360,354,598,550,101,69,37,58,65,271,640,711,212,571,86,345,895,696,273,125,522,23,90,603,72,739,163 }, +{ 13,115,569,734,219,341,127,365,197,509,629,23,98,538,873,901,286,762,55,206,258,165,253,332,170,169,638,535,180,391,116,276,844,31,44,12,546,0,395,164,1,293,10,752,911,36,964,143,282,37,384,512,237,498,308,616,301,740,5,141,352,8,261,9,2,943,654,317,120,284,38,338,913,601,295,244,105,7,3,201,468,924,125,923,910,144,239,21,4,479,482,242,6,86,131,417,961,185,22,64,758,374,110,841,533,401,93,160,15,944,434,82,951,567,843,268,14,564,837,299,272,719,28,860,109,11,351,935 }, +{ 151,58,396,53,287,74,27,73,297,549,29,651,145,70,445,112,113,315,436,26,331,6,573,250,75,45,334,52,307,17,5,34,80,163,2,138,19,491,61,881,220,409,631,262,1,614,632,226,171,106,611,186,32,200,66,50,62,311,25,46,845,767,500,21,135,644,243,223,87,14,48,42,64,3,472,108,324,76,122,161,635,684,65,294,56,38,136,95,72,255,114,849,780,278,209,37,90,852,84,714,7,51,89,148,624,237,383,36,207,236,502,22,12,133,86,322,659,227,119,69,222,11,4,96,309,263,179,33 }, +{ 214,289,90,874,104,751,64,835,312,249,204,65,750,288,875,348,519,174,247,194,74,715,81,138,32,280,524,784,192,636,319,437,879,0,95,85,6,681,217,173,954,91,29,177,86,597,792,77,317,671,422,22,115,864,888,52,202,752,216,673,588,712,18,123,180,233,306,43,355,304,352,20,76,693,579,66,4,5,25,51,21,343,59,237,2,28,61,426,235,577,33,10,397,316,707,287,145,370,24,102,49,291,17,108,84,945,222,97,57,208,56,401,42,526,560,13,730,40,402,23,117,207,474,7,484,72,494,36 }, +{ 9,39,166,28,35,158,49,30,101,175,439,0,346,1,364,64,88,274,100,168,36,67,404,310,333,952,159,55,345,505,190,706,254,420,86,237,23,336,68,22,251,344,51,69,194,114,164,131,570,208,180,756,10,799,143,399,109,83,50,115,141,967,438,768,584,868,128,752,105,95,103,189,286,252,352,5,323,2,230,65,596,289,137,498,7,702,165,13,12,577,503,535,160,125,202,48,547,455,244,610,96,21,16,253,169,857,217,154,797,646,656,228,197,530,747,149,810,124,930,272,865,597,98,185,31,24,389,625 }, +{ 341,901,15,1,365,10,569,0,2,5,180,4,3,115,6,31,7,197,9,13,22,317,951,165,44,629,143,23,253,401,116,244,762,55,93,242,479,752,30,217,51,37,141,28,873,137,638,36,202,203,12,21,86,386,421,77,24,758,11,268,453,318,269,110,33,576,64,811,352,326,164,16,494,506,498,308,670,82,717,642,351,206,120,338,170,261,32,125,40,851,805,260,102,859,8,403,54,661,961,291,144,105,965,98,35,898,482,391,373,153,509,492,958,117,17,499,615,38,824,559,456,384,854,457,282,89,564,237 }, +{ 202,450,120,77,260,318,494,128,102,817,15,269,51,13,342,403,23,291,5,523,137,326,21,141,33,1,922,32,82,351,48,854,177,859,402,64,196,70,615,31,475,40,4,153,285,59,0,12,22,10,2,9,95,98,61,90,28,210,30,17,453,194,117,139,8,352,96,49,275,35,678,899,113,349,165,65,180,6,730,93,692,115,58,704,217,37,203,719,116,576,125,50,457,75,44,421,163,216,36,134,7,39,3,416,14,67,197,25,373,110,341,207,112,86,386,87,56,127,99,11,223,551,365,300,105,16,19,237 }, +{ 241,32,13,115,197,180,165,316,64,288,125,21,23,237,247,141,95,16,22,92,167,522,191,546,498,391,341,28,752,177,586,4,48,509,447,118,459,90,5,352,59,901,264,225,304,350,35,199,126,363,18,217,24,410,468,691,246,841,744,10,478,102,77,649,386,117,86,49,317,308,302,212,146,33,71,954,156,72,46,674,93,31,0,640,411,923,903,645,686,271,698,2,521,140,602,591,120,36,375,68,12,864,321,695,873,758,340,336,318,269,179,143,65,897,772,446,880,839,682,366,190,57,30,818,781,303,953,388 }, +{ 16,24,35,71,60,14,267,18,46,140,69,1,7,108,189,2,92,50,75,230,309,68,236,38,335,220,394,54,366,52,486,21,150,149,246,48,29,87,67,5,571,388,9,181,133,0,103,443,28,163,537,171,55,223,126,545,121,70,12,96,262,328,565,83,32,118,65,516,58,218,37,191,330,563,64,167,22,114,13,51,23,238,251,622,281,278,487,294,399,99,696,115,36,10,582,135,514,86,6,128,517,950,675,674,273,98,101,594,172,49,72,146,95,314,451,155,45,409,212,433,221,345,141,66,247,97,733,125 }, +{ 15,898,180,197,901,341,638,165,33,10,115,77,365,1,4,217,317,5,102,117,253,13,951,752,3,134,2,762,153,569,37,21,0,196,40,11,244,23,6,82,7,120,143,291,32,141,137,51,31,873,55,12,351,44,260,349,352,401,372,479,177,93,22,342,116,36,9,629,308,59,758,450,386,210,854,139,110,111,482,457,164,17,922,421,961,202,719,269,203,373,247,523,28,125,160,48,437,64,326,242,57,704,509,304,14,318,730,498,403,506,148,817,670,8,90,453,494,282,144,72,127,275,579,221,206,899,54,89 }, +{ 523,193,18,84,730,817,90,56,65,240,214,475,453,196,33,719,4,233,202,32,51,153,289,152,123,421,11,194,403,129,99,96,386,249,135,102,40,27,23,361,847,352,59,511,91,457,177,370,134,128,117,510,13,561,75,753,643,388,278,247,204,115,494,450,415,304,77,524,217,216,64,6,5,3,174,701,880,659,609,437,433,432,381,287,260,257,165,139,120,111,87,74,66,1,402,372,318,269,21,899,183,63,141,650,605,770,743,735,718,667,629,611,569,544,543,348,306,288,209,207,162,95,93,86,61,57,49,48 }, +{ 151,29,112,74,222,66,145,2,287,624,52,6,416,45,7,113,186,25,122,665,549,337,87,204,243,5,135,644,311,58,884,226,138,396,61,32,192,70,42,525,415,614,315,19,64,491,651,26,4,227,174,609,544,20,21,43,14,27,128,22,172,632,0,59,209,65,353,97,631,72,1,85,580,86,17,445,80,454,307,104,90,334,81,496,177,46,635,141,255,48,37,77,75,10,155,869,659,76,294,106,49,73,11,95,38,36,115,673,69,18,216,712,148,173,792,33,331,102,280,12,16,24,137,881,89,297,91,28 }, +{ 13,2,23,0,77,5,217,7,1,6,33,3,15,51,10,115,141,102,180,64,4,9,40,317,11,197,14,117,165,153,341,22,352,237,17,365,134,269,291,125,16,202,12,196,372,59,349,177,752,498,128,28,18,32,318,8,36,95,457,19,86,49,210,143,391,901,24,342,21,304,569,31,509,137,139,120,27,386,57,25,164,98,29,111,401,403,45,26,244,864,482,546,110,44,961,308,38,275,453,37,35,93,48,402,247,629,20,50,46,719,873,55,475,65,56,30,72,579,203,523,99,253,494,468,170,54,116,76 }, +{ 28,67,0,65,1,114,103,83,69,50,9,55,2,109,124,46,133,24,14,35,22,154,36,68,38,547,389,218,7,283,323,16,54,18,12,160,194,420,49,251,175,5,75,185,238,39,289,429,158,96,208,168,532,29,86,480,505,531,484,189,503,169,149,797,782,361,21,101,570,52,346,71,181,336,927,533,48,95,37,108,51,596,10,252,23,159,128,799,105,285,418,64,118,535,335,398,732,6,70,803,419,137,220,13,866,171,756,394,30,32,309,328,140,115,87,60,230,433,345,141,58,439,831,402,610,366,99,754 }, +{ 121,271,132,167,146,18,363,101,411,621,354,212,520,60,9,647,932,446,806,16,195,407,463,777,364,562,35,360,522,68,55,1,98,118,31,252,706,694,191,0,24,107,936,664,676,604,199,48,126,781,23,590,156,28,591,150,21,447,39,125,5,109,264,49,759,265,96,22,404,33,13,10,100,44,51,905,259,71,346,857,682,116,223,848,941,7,92,154,93,105,108,478,77,467,286,821,166,64,638,907,765,691,628,279,178,14,203,102,12,30,120,393,158,159,593,834,724,419,147,50,293,163,815,786,367,70,634,2 }, +{ 26,6,85,122,396,624,42,25,573,445,174,416,665,19,680,648,138,145,348,881,64,436,214,113,235,65,297,544,194,90,45,746,525,280,112,289,74,233,32,53,27,355,29,222,287,389,17,151,86,80,95,813,81,73,256,5,249,141,283,91,597,0,343,76,22,66,87,204,415,874,209,312,690,524,611,255,288,63,644,123,243,70,614,216,4,128,2,77,119,61,370,106,52,491,315,58,10,56,651,21,12,519,18,14,835,59,250,352,247,114,49,179,636,205,684,619,311,57,84,28,337,667,177,108,11,208,503,240 }, +{ 146,16,199,24,71,18,126,60,92,121,271,156,360,953,640,108,132,674,35,246,118,273,32,354,195,649,7,314,150,458,522,95,517,167,446,191,21,140,340,463,22,388,68,107,566,575,241,727,10,345,591,359,296,48,586,14,407,28,5,608,149,672,594,602,660,1,467,363,212,13,9,823,49,267,101,225,124,55,232,37,653,0,392,877,50,447,330,702,12,809,462,441,230,508,46,668,189,77,303,236,733,23,698,621,115,423,411,399,298,97,350,247,72,796,451,64,658,217,69,589,309,51,928,264,168,102,163,133 }, +{ 51,23,13,453,475,730,719,899,403,202,401,21,15,64,523,115,95,457,33,12,141,308,32,102,4,128,40,19,11,421,5,117,386,3,99,22,153,125,26,817,120,25,6,961,165,1,180,196,17,197,494,57,134,221,0,194,2,237,260,7,670,48,27,365,37,629,450,49,170,65,14,569,137,342,236,38,352,10,56,29,20,16,468,316,241,402,391,77,177,350,84,34,283,269,217,82,9,247,546,351,318,42,59,372,903,841,558,317,129,86,47,46,41,28,18,326,615,341,90,291,164,801,509,119,76,63,54,24 }, +{ 151,396,53,113,73,445,145,6,27,297,573,26,287,315,112,436,74,58,138,491,45,70,881,19,614,34,250,61,29,549,611,80,17,631,632,5,307,2,200,644,32,222,64,25,52,106,42,1,243,87,66,75,845,226,311,186,334,849,95,122,472,322,294,21,651,714,255,684,48,171,331,194,65,161,209,14,128,86,72,141,46,624,50,135,22,635,262,227,85,108,136,7,220,90,76,163,235,176,114,312,280,62,409,10,337,4,148,56,454,283,237,12,469,37,51,780,214,667,236,3,767,133,36,89,495,155,0,23 }, +{ 223,774,1,888,385,260,98,863,202,269,421,730,96,349,697,453,141,393,622,834,318,751,87,418,791,765,801,922,163,21,335,352,5,7,120,13,291,854,64,23,717,851,293,30,284,372,31,678,371,217,947,351,70,51,32,137,48,77,128,82,442,450,44,49,36,752,125,386,541,931,304,873,0,498,22,2,201,276,807,317,10,670,180,102,6,75,237,39,373,33,69,770,144,177,95,17,16,160,116,100,842,203,197,968,949,221,115,275,165,494,475,127,898,58,963,523,403,12,954,537,45,50,433,872,652,526,268,241 }, +{ 60,16,126,92,121,35,150,246,7,598,279,199,191,167,441,107,71,97,146,195,271,18,298,24,407,356,517,230,1,14,98,508,423,140,674,132,359,223,653,212,48,68,399,783,345,21,314,108,607,43,467,232,362,649,118,392,189,20,5,0,149,702,727,55,22,695,28,446,672,69,66,49,493,10,50,388,809,273,2,522,31,46,640,293,125,9,13,95,101,267,173,501,393,834,877,38,575,67,32,75,37,796,346,779,608,12,156,589,236,566,163,96,221,104,360,375,535,64,711,463,23,175,755,172,414,562,953,354 }, +{ 22,105,28,1,239,95,55,170,320,301,127,36,98,31,49,109,9,2,0,168,35,67,185,169,12,14,86,512,7,44,395,159,5,194,293,312,114,64,83,928,713,740,65,24,23,623,21,299,10,338,16,51,242,50,125,141,933,420,13,237,37,164,844,286,143,69,175,584,892,202,48,6,46,371,103,133,269,116,685,39,916,137,115,160,599,208,272,128,96,284,158,737,762,638,180,244,276,187,763,308,165,841,761,479,505,401,347,424,318,389,30,352,870,18,131,32,197,219,965,327,538,54,456,900,506,253,385,874 }, +{ 2,292,6,119,29,207,76,61,306,45,5,790,17,240,32,58,106,1,151,476,710,14,214,209,25,179,84,52,70,64,90,7,112,26,311,19,618,72,128,129,27,255,123,655,460,59,249,141,86,4,80,40,113,22,115,46,10,245,464,205,145,243,263,358,77,56,36,74,65,315,504,454,148,3,186,95,0,519,50,177,75,202,21,269,38,257,66,42,137,23,16,769,51,738,48,136,33,162,18,184,62,241,99,227,24,318,194,87,491,37,11,12,57,425,833,180,402,49,237,89,287,216,13,720,102,54,631,138 }, +{ 51,221,23,115,254,438,48,421,530,541,13,21,401,613,308,39,688,386,125,840,228,551,453,960,827,1,166,96,406,326,393,627,670,158,49,697,656,829,846,770,333,856,404,5,180,30,863,190,202,669,518,890,219,83,159,98,50,131,33,102,310,217,165,137,743,77,32,101,31,752,679,15,105,615,468,391,341,317,0,224,9,498,352,258,286,455,364,187,778,197,344,44,865,539,457,482,272,128,116,37,28,949,403,876,385,963,373,709,931,719,64,574,68,35,160,274,268,616,345,442,95,492,450,293,843,839,22,261 }, +{ 5,1,22,21,2,12,14,36,48,0,54,86,32,10,28,50,65,37,38,23,51,17,7,208,3,9,114,45,281,13,4,137,323,46,24,6,99,95,18,49,58,55,115,480,72,202,96,745,285,194,181,67,64,16,270,89,133,218,83,168,403,11,430,29,61,155,128,160,503,361,540,223,77,244,221,389,318,757,164,103,141,283,76,402,502,401,59,859,920,40,341,543,290,31,803,269,513,735,124,165,308,33,75,253,421,143,838,754,69,90,180,197,125,102,19,494,453,475,966,105,27,289,109,584,576,569,765,148 }, +{ 182,23,363,284,296,447,340,98,1,291,698,367,93,668,658,156,144,21,269,70,433,379,31,318,478,354,203,44,691,335,180,13,223,125,116,937,586,411,326,110,107,48,18,421,664,724,273,197,141,51,692,195,87,5,67,914,687,486,759,349,551,386,375,268,232,149,102,60,35,33,115,795,251,69,65,49,415,364,77,771,626,163,453,281,165,127,121,24,10,360,806,791,393,336,39,475,372,217,850,662,942,682,518,484,361,359,221,196,146,133,128,117,75,64,22,12,7,883,591,457,132,16,647,153,120,68,801,670 }, +{ 45,498,17,106,546,115,32,62,873,41,197,180,509,54,129,468,243,255,123,752,165,155,38,341,13,241,292,217,720,449,57,136,536,734,569,148,51,476,248,428,214,139,76,369,221,482,328,237,542,365,72,89,339,181,23,227,350,64,90,211,117,11,519,270,290,405,721,587,457,21,901,50,128,741,710,141,552,313,430,162,2,540,130,202,15,102,528,247,488,317,209,157,77,48,14,3,391,152,431,308,629,4,618,743,281,179,5,183,33,40,12,37,358,361,266,46,86,961,828,471,773,372,421,543,249,507,119,10 }, +{ 32,95,64,237,92,180,246,22,241,197,125,115,312,652,126,638,49,316,13,21,5,165,288,534,7,4,48,0,28,23,498,392,296,107,888,340,90,303,658,626,247,70,441,10,16,365,953,468,260,230,546,18,24,774,12,379,883,98,490,482,863,31,877,508,391,141,51,830,933,354,9,517,347,150,314,385,177,132,65,146,645,40,653,76,33,2,341,217,14,269,44,698,263,156,195,850,649,633,199,120,360,359,563,923,509,216,128,86,60,733,873,267,35,1,442,168,330,293,929,545,11,622,591,87,77,641,437,660 }, +{ 15,33,341,217,13,77,141,4,317,102,23,40,115,180,365,351,117,21,5,352,854,1,137,197,11,10,153,51,82,901,32,134,59,64,202,196,36,244,269,22,922,165,2,291,260,177,752,143,0,120,898,48,3,318,12,9,372,253,128,457,31,569,28,37,160,30,247,86,139,55,44,72,7,57,349,110,90,237,498,342,304,111,719,579,450,96,8,730,403,308,811,49,523,164,386,35,125,482,437,17,6,14,961,39,98,494,116,93,275,642,401,210,95,221,76,277,105,717,373,203,216,38,873,899,18,817,326,509 }, +{ 120,260,1,15,141,217,352,13,77,82,93,351,110,23,180,317,349,51,202,21,854,326,165,64,269,128,137,5,291,304,752,36,373,372,922,450,318,177,203,197,115,48,457,10,342,2,0,898,102,160,12,16,901,22,125,386,86,9,144,32,96,35,719,28,7,365,341,244,275,494,210,730,18,95,83,661,4,403,237,523,498,247,143,864,473,90,14,961,437,33,817,40,37,8,678,482,253,579,569,98,72,24,50,475,391,67,69,954,717,288,164,65,49,6,206,216,55,17,134,947,899,576,308,76,316,196,490,402 }, +{ 23,13,51,115,77,217,141,177,33,202,4,64,269,102,0,128,10,2,7,318,40,59,32,14,6,11,352,22,9,291,90,247,180,237,137,1,165,16,317,21,210,197,117,153,196,18,125,49,403,3,95,15,17,349,341,304,12,372,216,365,342,19,36,134,453,86,98,28,308,24,31,244,494,93,275,143,475,57,401,44,48,5,498,457,241,120,203,20,8,105,144,221,25,326,110,752,29,65,523,164,37,55,30,719,421,402,139,39,160,27,116,260,26,96,35,300,437,961,82,569,38,45,901,99,72,253,678,316 }, +{ 478,264,724,520,682,98,1,772,905,664,759,821,9,223,21,48,411,604,909,447,363,23,265,100,13,410,897,166,5,907,293,31,88,329,252,39,101,0,51,125,639,848,32,167,96,647,691,807,781,541,686,382,284,521,44,321,777,393,621,274,118,212,18,676,116,163,69,271,936,834,221,593,812,16,24,914,30,10,127,765,115,37,49,354,335,217,144,68,93,386,105,404,586,22,33,4,83,70,806,165,553,697,932,60,77,50,744,463,71,64,286,421,254,281,435,102,35,12,656,371,191,225,963,446,121,213,201,763 }, +{ 29,20,128,7,43,77,174,74,2,23,32,416,52,102,66,491,104,113,81,33,13,6,51,342,222,202,173,14,0,21,525,59,137,45,192,141,319,5,318,64,10,210,22,291,631,18,72,122,90,145,269,196,151,86,61,58,16,665,4,177,40,355,204,153,134,402,28,285,42,738,544,135,48,216,25,275,115,85,24,65,217,9,12,87,76,833,415,226,179,198,38,138,114,49,97,84,243,337,315,311,287,36,426,869,422,172,89,255,17,209,784,112,454,317,425,35,349,117,95,27,494,208,91,194,503,232,30,11 }, +{ 2,1,5,14,58,7,52,29,46,61,290,45,75,38,592,425,50,270,70,16,32,163,24,155,21,186,171,72,6,227,69,334,263,207,220,17,243,663,48,18,89,76,0,67,400,655,738,35,408,54,112,87,148,36,64,449,66,23,106,22,307,10,262,128,292,74,409,115,119,135,9,476,90,65,141,278,37,767,103,83,86,28,720,140,95,780,255,96,25,405,27,77,3,241,51,12,13,306,20,97,666,179,223,40,19,177,4,428,92,151,496,209,172,26,502,113,84,202,790,137,60,431,833,240,659,331,217,324 }, +{ 80,483,209,106,6,26,17,495,255,833,378,469,113,742,27,25,19,814,119,445,161,887,45,42,76,667,61,64,491,151,90,573,73,235,84,29,65,32,136,614,179,383,128,112,145,297,122,2,527,396,214,62,56,240,85,5,53,862,194,648,123,311,611,624,207,141,58,86,315,95,14,22,306,292,34,4,454,436,59,684,790,0,1,289,440,250,129,738,205,70,416,361,3,249,245,57,269,287,665,52,108,402,202,46,12,177,11,114,216,21,137,36,10,200,33,115,72,180,74,138,50,28,318,247,312,51,585,148 }, +{ 98,223,393,834,1,354,195,791,697,447,293,284,360,541,781,807,51,664,591,156,421,411,48,801,163,883,948,668,765,386,18,586,132,121,23,963,363,125,21,446,551,759,31,621,949,770,407,777,271,658,146,730,167,670,13,116,936,453,5,367,296,478,144,371,221,96,70,9,44,202,105,719,127,687,379,49,118,55,60,698,101,842,626,318,771,217,647,212,691,137,281,638,340,165,75,522,463,706,724,32,93,401,39,364,269,16,520,562,35,141,77,50,265,914,450,33,203,197,268,498,264,69,857,335,22,317,103,905 }, +{ 679,876,36,816,141,144,93,137,752,406,669,228,180,269,96,251,332,21,1,48,352,203,351,202,855,160,831,318,345,498,83,5,116,187,120,617,201,64,317,902,258,261,840,23,839,613,473,110,864,466,190,574,219,32,860,13,873,418,450,326,539,926,748,856,260,44,51,384,704,143,50,295,531,197,489,843,661,206,10,944,778,530,217,567,871,88,82,268,901,800,128,822,125,372,365,86,957,31,30,165,37,442,244,728,224,768,851,373,854,505,625,863,386,367,276,55,827,237,115,922,349,420,274,523,223,169,72,61 }, +{ 15,13,2,1,341,5,23,3,0,4,7,11,10,6,51,12,14,9,33,165,115,569,197,180,17,365,217,153,77,64,196,102,22,21,40,453,317,8,32,901,141,457,117,961,134,31,18,37,16,386,629,291,719,110,28,44,475,177,120,93,269,202,86,237,372,24,95,19,125,59,49,403,57,401,352,318,48,349,730,36,111,326,137,128,98,38,82,391,27,139,65,30,873,275,242,203,54,260,29,45,494,55,25,342,210,450,99,56,46,116,509,498,482,752,373,50,351,899,523,308,247,90,143,35,437,76,72,26 }, +{ 202,13,23,51,403,21,494,5,141,128,48,269,450,125,120,523,64,12,318,180,165,453,95,475,15,33,817,32,197,1,137,115,102,7,260,899,4,37,10,99,2,98,326,93,49,196,730,719,221,457,77,16,237,9,31,22,401,6,421,14,386,308,0,117,134,18,352,40,82,20,19,177,615,59,44,61,402,17,391,203,116,110,27,25,90,341,351,153,30,28,285,105,217,3,291,8,958,854,96,24,50,805,922,54,961,758,576,39,26,801,247,347,509,46,757,342,317,365,498,11,194,72,670,210,38,139,275,170 }, +{ 16,24,71,146,60,18,92,199,126,118,246,35,121,108,140,167,132,230,649,271,360,267,7,9,640,388,150,156,232,191,133,522,273,68,236,101,107,212,0,672,55,446,953,14,149,458,674,330,46,67,314,566,95,189,602,633,392,562,195,64,591,103,21,467,39,354,69,463,407,114,247,98,32,5,50,172,10,1,733,399,278,366,49,36,653,309,412,65,823,22,28,915,288,877,316,124,75,345,340,264,37,12,594,516,135,563,598,451,225,86,38,572,830,125,31,30,359,517,115,571,363,164,726,398,48,237,116,251 }, +{ 269,141,177,678,202,77,318,947,128,349,33,291,40,102,352,210,498,137,954,120,205,864,216,64,196,597,304,275,217,494,461,23,402,115,342,90,13,403,450,300,95,49,197,111,14,326,22,139,237,10,0,125,873,372,182,751,86,117,615,7,44,195,36,31,83,67,752,356,153,98,46,96,59,252,109,97,800,357,285,774,523,550,208,609,569,546,391,247,116,93,66,24,19,18,17,15,6,5,3,2,30,922,414,373,180,159,147,105,20,485,75,859,854,392,376,317,298,288,277,260,143,134,107,39,28,735,415,409 }, +{ 16,71,24,92,35,60,18,191,167,108,246,126,150,118,140,68,7,273,121,149,212,21,388,14,399,314,517,267,796,230,48,146,823,199,271,458,1,467,189,695,877,674,345,9,55,97,46,702,101,575,602,594,733,841,0,350,22,375,107,95,232,28,236,462,32,566,69,298,50,672,13,2,5,727,133,451,154,302,125,653,463,23,446,124,10,589,223,132,64,67,175,38,37,49,98,65,354,423,398,103,649,172,562,755,75,66,336,221,197,903,508,915,809,363,359,51,380,411,309,608,237,330,412,241,12,640,509,36 }, +{ 2,14,75,46,29,52,1,70,171,69,163,220,58,38,112,24,16,7,409,67,262,223,35,334,155,186,5,151,267,394,54,307,87,140,103,309,50,45,83,189,400,18,71,60,92,335,760,227,502,48,21,181,108,61,96,537,12,820,25,278,6,852,113,549,66,357,366,32,287,587,17,37,19,536,27,606,36,150,135,97,74,64,26,380,486,65,0,552,583,28,230,281,600,294,787,582,849,68,149,133,675,22,315,311,236,72,474,243,114,10,324,571,238,86,226,764,9,55,780,218,145,209,80,462,328,95,430,767 }, +{ 265,166,252,100,404,9,329,382,520,593,812,39,88,274,604,101,264,0,553,1,639,676,438,865,664,333,310,68,30,656,167,821,118,363,411,96,23,682,18,158,48,21,478,364,190,105,286,51,98,49,254,435,759,724,647,13,212,16,907,272,124,921,159,31,223,419,125,905,557,5,447,93,154,321,44,393,83,410,327,455,848,35,67,621,293,271,127,191,71,530,354,344,32,24,60,625,213,418,242,109,131,36,69,33,221,144,765,28,121,806,65,50,777,116,22,238,960,146,834,781,897,697,778,165,163,466,115,948 }, +{ 16,35,60,18,24,68,158,71,0,7,55,118,109,14,49,22,159,105,9,28,329,126,344,92,20,107,10,154,553,265,2,191,101,95,424,23,21,5,327,121,13,1,286,239,43,747,51,419,48,455,108,374,168,39,272,677,137,169,124,149,12,199,310,150,46,128,38,279,320,37,709,170,346,6,64,98,167,202,31,185,252,67,810,65,36,125,917,175,577,97,212,32,942,44,518,29,925,628,86,301,69,146,375,132,538,575,913,890,566,403,634,439,333,61,33,50,326,147,165,120,66,446,102,273,421,96,493,116 }, +{ 13,15,23,51,0,1,2,5,77,4,10,115,3,6,9,197,7,177,202,165,961,22,11,12,457,318,217,180,31,14,21,141,33,403,40,102,32,120,453,269,8,341,59,18,16,28,44,17,64,291,93,210,719,37,352,401,110,569,49,317,134,117,90,203,475,260,365,30,450,386,494,36,730,523,125,326,196,153,24,498,98,275,901,111,99,421,308,137,349,817,48,139,752,82,72,95,116,237,86,304,391,247,873,19,509,76,372,216,482,629,57,342,899,128,373,39,758,89,65,864,351,437,105,55,221,38,144,96 }, +{ 160,251,137,93,317,36,180,1,345,120,352,752,617,318,217,332,202,96,365,531,269,498,10,901,141,83,851,341,55,143,64,816,35,968,679,800,260,473,351,230,144,21,373,244,717,82,831,201,876,533,110,372,295,559,258,418,206,203,50,28,704,855,237,349,22,669,86,261,48,898,115,860,253,926,482,9,406,748,197,67,854,873,574,7,434,564,839,69,661,450,15,902,164,228,2,219,843,268,125,187,98,5,728,16,922,165,221,613,335,109,224,442,367,384,868,77,539,840,871,291,822,417,579,116,31,918,44,326 }, +{ 42,25,6,59,85,611,233,128,122,26,4,27,19,343,91,269,614,318,45,177,283,503,138,680,77,80,73,11,216,573,145,102,445,297,0,402,235,210,275,209,174,63,202,389,84,14,803,311,18,33,529,396,255,648,416,280,40,65,256,483,10,64,2,57,678,1,667,624,250,81,24,348,304,17,196,76,194,5,544,454,114,789,151,285,133,90,74,106,51,29,690,12,349,141,61,123,119,152,137,32,355,9,52,56,7,153,525,21,237,28,13,881,36,22,87,436,111,576,300,117,605,495,370,361,291,53,23,494 }, +{ 95,64,81,91,32,237,51,65,180,7,347,74,343,207,204,511,498,577,38,197,348,56,312,128,23,20,370,49,104,43,18,138,22,233,124,86,0,90,554,28,10,681,453,21,14,280,87,750,202,141,25,546,561,29,306,115,63,16,636,241,172,165,147,85,52,24,11,467,650,580,353,192,84,6,13,713,194,673,500,283,9,368,718,151,145,72,57,33,355,263,19,544,475,412,324,318,269,235,173,140,125,108,102,4,933,174,899,519,42,643,879,751,525,403,278,247,221,217,148,123,120,112,111,97,75,73,61,59 }, +{ 326,202,120,817,260,450,494,318,403,137,704,859,128,523,77,576,5,615,553,51,342,291,23,13,1,421,234,475,142,15,48,269,178,968,147,16,21,33,639,922,285,20,49,351,7,854,682,70,58,363,265,373,198,252,402,401,158,82,416,60,132,55,455,354,0,68,102,24,2,329,141,35,101,64,18,6,121,43,95,325,165,31,9,848,798,341,36,12,805,195,32,14,411,10,146,107,75,912,569,116,96,793,424,374,167,156,98,125,17,670,692,163,604,465,25,965,629,349,115,907,427,61,457,65,621,86,71,520 }, +{ 43,20,107,356,362,595,126,92,7,359,319,858,392,493,422,246,441,259,173,683,414,49,97,5,819,279,21,104,60,16,230,35,945,232,0,612,376,48,660,121,32,199,147,132,649,22,18,71,14,663,61,423,470,13,146,672,81,623,298,1,105,693,125,159,408,58,427,98,51,165,501,263,729,95,68,4,24,12,325,779,140,55,23,953,426,28,560,72,804,715,241,581,767,314,939,885,202,118,40,204,102,33,37,17,955,192,713,70,221,598,397,77,2,387,251,344,64,44,31,10,941,75,6,465,603,534,46,761 }, +{ 42,25,6,77,210,483,690,29,102,33,122,342,0,27,4,59,177,40,19,91,300,680,18,21,648,153,196,26,475,202,117,494,128,119,790,48,22,2,343,85,145,56,36,24,32,757,81,1,684,106,292,269,28,833,9,814,151,402,137,84,45,134,70,64,23,139,112,464,396,73,655,115,52,20,614,14,10,403,50,216,63,13,358,233,370,86,7,5,76,37,17,12,485,72,65,397,476,291,57,51,80,611,209,881,665,491,179,99,74,49,416,11,667,624,453,174,148,113,95,235,138,275,61,742,544,341,510,746 }, +{ 31,44,299,116,599,393,456,144,492,125,22,268,242,367,28,963,9,0,105,489,948,434,301,918,36,127,384,1,95,55,49,170,98,512,109,506,206,51,320,23,940,93,558,13,900,86,203,2,10,239,64,5,959,457,141,395,143,7,293,282,855,911,137,202,12,253,165,8,417,159,338,21,24,332,115,401,498,327,551,286,719,197,352,16,791,617,269,391,128,120,870,37,763,244,180,479,14,168,386,308,65,164,39,421,473,160,6,18,692,841,403,276,499,326,730,48,951,272,685,523,713,365,169,903,450,424,559,35 }, +{ 410,521,686,367,321,662,201,772,144,335,88,934,1,98,96,921,264,284,897,274,435,443,909,744,692,203,225,551,120,393,33,960,639,44,418,382,190,217,260,93,64,523,442,197,77,9,100,30,237,166,102,948,244,31,326,765,475,438,165,615,82,10,791,473,717,351,238,206,125,12,805,498,466,213,69,842,617,268,778,899,576,127,116,110,37,5,318,187,0,699,530,303,230,855,854,822,817,613,403,269,39,35,32,21,2,489,141,23,291,704,160,478,131,966,539,905,851,768,730,352,272,247,180,175,137,36,22,922 }, +{ 878,142,178,234,786,132,195,929,416,202,146,147,77,639,522,259,687,167,271,450,481,771,626,212,590,465,49,534,455,427,198,5,315,407,120,0,360,318,48,640,325,23,125,329,113,35,883,22,562,163,346,70,581,13,21,31,98,591,1,61,121,199,107,798,279,150,818,159,20,260,379,668,354,446,44,557,494,269,43,32,10,371,293,95,55,105,141,386,58,33,344,884,64,158,467,116,768,102,93,18,16,362,342,817,60,30,518,191,296,128,711,99,223,39,535,87,170,291,777,97,51,75,402,783,7,156,363,347 }, +{ 254,39,131,578,272,9,716,310,827,829,0,224,518,166,616,333,613,846,228,938,101,30,49,252,105,890,100,219,404,406,371,159,190,265,98,286,158,688,127,68,438,385,455,931,327,910,31,242,856,1,187,709,28,924,737,258,23,329,16,21,557,51,137,96,656,625,125,44,442,116,24,18,48,35,36,13,160,88,65,731,124,863,274,22,95,293,344,395,5,93,109,553,816,128,299,627,530,916,364,761,836,301,154,284,871,118,202,676,201,144,83,276,165,419,944,456,203,10,55,512,32,7,320,724,520,60,115,593 }, +{ 699,854,230,855,473,450,831,704,351,137,217,498,352,800,251,202,160,144,253,203,201,418,206,317,332,752,443,141,1,96,36,180,326,82,822,372,93,244,83,494,335,35,365,817,69,859,237,689,898,21,617,864,318,901,44,702,384,922,954,349,345,341,98,10,48,64,824,367,187,128,295,873,284,120,67,531,579,269,7,86,5,22,678,268,2,261,840,927,957,930,28,735,13,291,31,559,947,116,143,30,851,669,373,723,23,16,260,959,661,125,868,717,403,9,748,437,110,50,213,115,223,539,55,615,466,946,285,533 }, +{ 403,475,202,453,77,318,51,402,494,23,13,33,342,102,128,576,269,141,137,450,120,899,421,523,196,4,117,40,90,275,153,365,349,59,177,291,65,210,64,217,84,0,134,670,15,372,95,197,401,817,1,180,805,341,237,96,260,194,3,285,139,114,99,18,873,326,615,503,393,386,317,293,125,24,11,389,165,70,216,800,752,745,730,719,697,678,657,277,115,108,87,83,69,36,32,21,859,111,457,901,788,323,300,283,266,130,118,86,76,57,12,352,351,9,7,6,2,842,725,541,361,329,312,221,168,124,49,945 }, +{ 144,23,291,268,197,182,202,116,98,44,264,284,415,96,70,100,125,367,141,203,39,110,905,411,1,107,93,363,120,180,296,658,82,310,447,692,897,238,361,190,77,664,260,105,88,520,48,478,166,682,118,576,340,30,18,966,51,865,13,349,379,403,795,551,921,326,687,301,254,581,223,101,676,457,404,156,812,195,242,213,208,9,165,364,323,269,65,60,615,498,21,765,621,271,137,128,917,914,890,855,481,217,154,102,68,232,883,593,523,354,131,948,318,219,691,842,846,771,272,252,0,626,382,149,87,824,822,791 }, +{ 15,23,13,77,1,51,202,120,180,115,141,260,5,0,21,9,137,450,269,128,12,2,318,365,7,197,326,351,48,102,341,177,342,352,523,217,291,403,165,6,64,110,32,373,4,317,18,16,37,36,901,22,14,10,494,90,457,28,854,453,82,24,386,817,40,33,308,8,11,475,401,752,96,922,93,3,244,30,275,719,730,349,210,421,86,160,99,17,898,285,31,961,50,39,569,44,221,59,704,49,117,35,372,237,153,55,143,899,391,253,203,116,134,615,576,144,196,402,65,125,629,98,247,482,83,20,873,509 }, +{ 0,95,18,577,32,207,64,193,29,784,65,887,715,102,52,61,671,138,49,91,74,180,104,701,237,693,474,280,403,241,804,847,22,263,6,174,344,583,453,862,475,77,85,643,500,397,38,37,306,498,494,312,289,2,81,33,5,681,942,510,370,27,283,422,316,192,25,155,347,84,901,319,113,576,24,835,355,348,43,933,814,353,72,194,240,204,42,19,298,650,523,125,86,718,173,247,51,721,389,124,879,490,707,291,153,141,58,54,17,889,202,97,402,1,546,342,134,482,36,511,421,805,750,673,426,196,120,117 }, +{ 101,9,0,252,68,364,166,100,265,124,154,329,39,16,419,212,24,404,71,118,191,676,158,35,88,18,274,553,520,382,65,562,694,28,962,677,671,60,55,30,333,302,706,149,167,604,375,49,109,857,21,703,967,917,865,1,264,593,806,344,310,438,346,190,48,254,96,105,439,23,22,701,812,13,125,175,777,67,925,51,159,656,83,589,570,7,455,95,628,289,108,98,31,286,10,5,221,429,69,810,826,336,36,435,92,131,557,639,722,32,168,793,638,272,273,14,44,128,126,114,50,170,146,37,238,127,484,634 }, +{ 16,24,191,71,167,118,212,18,35,68,149,375,101,108,302,589,60,92,755,9,21,562,674,124,154,246,0,273,48,1,126,363,398,411,55,7,447,146,458,350,841,695,140,65,818,150,388,199,628,345,478,264,22,13,189,647,95,271,602,32,69,823,267,121,28,50,67,796,14,346,23,650,49,175,5,621,520,664,51,882,399,10,903,133,815,777,252,98,100,197,649,64,115,781,46,847,314,517,566,221,509,223,103,575,12,953,125,114,236,230,354,37,336,2,39,96,107,308,640,749,463,936,165,724,462,886,604,237 }, +{ 69,14,2,24,46,267,71,189,38,29,67,278,52,35,135,1,103,108,149,75,309,16,388,83,150,7,60,87,18,236,74,66,220,335,68,251,262,6,50,140,582,357,451,324,133,28,226,366,238,516,172,565,65,36,25,486,114,70,336,0,92,55,594,5,443,458,273,175,54,418,96,571,97,394,906,733,21,26,124,380,12,118,9,218,58,22,154,641,606,398,86,230,48,222,160,37,19,705,42,596,331,550,112,10,412,138,95,45,399,181,294,171,163,145,64,462,32,831,600,603,345,696,517,711,602,545,894,302 }, +{ 13,23,1,4,115,51,2,0,5,3,11,180,33,7,77,10,15,9,341,141,6,365,12,102,217,202,197,317,40,22,165,269,352,318,59,403,64,901,17,21,14,32,752,28,31,49,177,308,457,401,498,453,8,18,873,128,291,494,16,125,117,475,569,196,95,509,36,93,237,153,24,98,372,137,961,386,349,37,139,44,30,326,210,134,342,48,719,482,391,90,120,55,111,86,110,247,203,39,421,116,523,304,260,450,19,546,629,65,105,734,730,27,275,57,468,402,758,143,35,864,351,38,54,144,244,216,300,373 }, +{ 447,1,698,411,125,98,31,21,363,5,919,848,48,44,691,144,759,293,225,914,459,120,13,478,354,156,4,116,922,32,23,127,64,668,163,857,371,202,49,82,30,839,284,586,70,17,223,93,865,664,621,340,165,87,51,39,421,321,260,897,110,551,88,724,330,264,335,318,221,77,3,2,821,772,744,442,190,242,195,634,520,360,201,682,963,916,898,854,685,372,351,349,291,269,217,180,141,105,102,96,95,75,59,40,686,622,949,938,687,658,228,909,100,931,867,638,446,407,761,558,463,456,342,310,237,197,166,90 }, +{ 854,351,180,82,217,352,141,752,898,317,15,244,1,922,21,36,10,64,77,115,260,372,365,35,110,22,291,28,13,341,40,48,120,901,349,4,864,5,661,32,23,160,304,498,237,137,33,11,269,51,202,143,373,93,144,59,164,96,326,98,83,450,318,165,203,102,253,230,55,45,197,2,247,201,67,128,579,17,125,86,76,177,69,37,57,0,16,31,44,728,117,72,27,9,52,116,251,482,342,817,457,7,523,811,367,559,50,719,12,134,961,717,494,187,127,105,49,3,90,873,569,899,418,196,18,316,6,61 }, +{ 13,1,5,0,4,10,33,23,9,2,82,11,3,77,120,6,21,7,40,12,102,8,51,31,117,260,351,134,22,16,153,110,64,450,93,32,196,457,523,115,28,165,18,719,98,59,44,730,30,17,326,386,116,197,898,854,217,961,180,177,291,36,49,817,125,48,342,95,141,90,317,341,922,57,758,202,373,899,139,14,365,352,349,24,203,37,372,901,801,615,128,275,318,269,210,453,39,144,111,237,401,391,72,55,137,105,752,403,86,38,569,494,642,704,99,475,509,498,661,304,811,277,61,247,65,873,308,551 }, +{ 51,403,475,23,13,21,12,494,453,115,99,1,5,523,817,202,7,421,49,22,64,450,899,95,221,32,0,120,308,730,6,180,37,141,48,2,20,10,401,128,44,31,14,753,125,402,3,719,260,11,9,318,801,40,33,19,17,4,386,289,237,105,90,86,16,457,326,36,361,269,170,102,76,39,38,30,165,26,25,961,341,958,576,365,247,240,197,117,98,89,77,61,35,34,350,137,93,82,805,782,756,701,671,670,569,503,484,468,413,405,352,317,241,217,216,214,208,179,134,8,281,316,270,144,83,68,945,901 }, +{ 752,180,352,864,498,317,141,217,237,39,365,115,901,30,88,930,579,230,0,83,101,9,197,372,482,873,128,546,64,100,341,36,1,166,437,927,702,35,213,96,23,13,175,285,68,252,269,349,251,28,689,190,69,868,443,735,304,21,562,723,457,137,399,759,509,244,831,419,10,274,77,321,539,165,818,67,530,201,219,194,228,840,418,345,410,93,131,144,86,570,143,48,154,854,247,646,224,866,49,12,699,669,402,596,954,212,203,208,160,158,800,31,177,694,358,24,117,51,957,406,382,202,22,346,134,7,438,16 }, +{ 16,24,35,71,7,0,18,28,60,14,22,158,109,68,154,118,124,92,168,2,55,149,159,49,747,185,169,424,1,108,105,95,10,577,150,344,175,126,20,239,65,9,677,69,5,67,302,6,12,21,671,46,329,38,107,97,164,23,51,43,286,375,170,48,50,125,140,327,64,320,39,420,29,36,701,374,289,83,101,273,570,37,143,32,13,347,137,191,345,189,230,86,66,553,265,638,128,202,429,455,252,54,272,703,403,246,535,103,52,221,421,654,398,279,98,194,121,114,310,336,467,584,298,165,25,453,346,96 }, +{ 15,341,23,13,0,51,77,1,141,202,33,4,137,9,260,115,120,180,102,269,351,40,11,291,5,318,82,7,2,21,10,901,197,12,22,128,854,922,165,450,16,117,352,59,8,177,217,28,3,64,457,349,90,32,36,403,37,110,18,30,14,401,6,17,523,134,898,365,317,453,48,752,57,31,719,386,817,308,196,421,153,93,24,244,139,86,143,475,730,569,39,99,494,961,373,55,642,44,49,498,285,96,210,253,275,661,125,342,35,237,203,372,95,899,326,144,402,247,629,160,98,391,61,304,576,221,216,208 }, +{ 68,35,24,149,18,71,16,118,189,108,14,7,101,154,0,9,230,175,140,336,69,124,65,267,28,114,67,133,302,46,55,398,251,375,345,2,1,236,458,103,388,83,399,596,60,50,283,38,92,346,732,412,602,36,22,212,309,21,273,39,650,722,246,12,109,48,172,429,29,218,191,906,75,52,364,443,5,86,167,547,54,10,135,252,49,64,238,95,330,37,278,723,96,23,705,516,520,439,628,891,13,51,366,565,158,915,930,232,562,30,868,6,335,637,696,150,952,194,451,484,32,220,100,694,702,886,389,168 }, +{ 308,13,197,115,391,165,546,509,125,629,569,237,749,468,23,9,558,28,903,338,217,801,734,317,352,401,740,254,341,49,36,180,22,386,51,39,159,901,873,95,365,490,506,530,168,961,498,131,166,438,64,482,247,219,579,83,185,109,93,752,656,187,345,439,613,857,457,688,316,101,86,21,158,627,258,843,89,829,637,312,10,55,679,105,967,224,170,251,574,332,406,221,175,228,48,758,902,261,960,505,44,31,876,116,37,12,868,840,72,424,347,1,371,98,15,518,274,141,32,827,333,77,190,892,816,272,253,143 }, +{ 90,64,217,32,115,128,141,216,77,312,86,597,95,59,4,33,247,875,13,172,352,197,147,237,177,288,102,437,210,22,97,5,194,137,202,37,57,11,182,208,153,76,61,14,304,134,117,65,40,751,23,291,947,10,864,342,196,21,362,49,356,178,60,45,874,317,0,54,48,612,165,107,51,259,588,752,324,192,819,298,275,180,20,678,372,307,289,173,43,120,112,415,285,207,164,72,36,35,707,44,7,359,89,46,802,414,156,2,789,729,579,168,123,104,98,92,55,24,712,524,125,87,70,66,470,387,465,148 }, +{ 1,22,28,105,36,95,67,49,2,55,239,0,5,12,83,170,50,9,159,312,86,109,7,114,320,21,208,14,103,69,35,168,301,31,164,64,194,23,892,127,169,323,160,654,98,51,141,65,48,158,10,96,143,133,37,505,13,218,584,38,6,46,751,54,115,24,185,244,347,128,424,44,824,202,253,503,597,16,269,32,237,238,39,137,286,180,512,420,3,352,740,533,29,646,165,638,251,242,318,338,623,175,125,4,18,776,395,797,480,285,713,197,531,116,532,30,928,68,341,401,293,371,99,933,124,789,844,272 }, +{ 18,9,16,101,159,68,252,35,24,105,0,60,310,265,49,39,890,329,320,71,455,118,272,557,7,55,167,28,327,121,286,1,137,170,98,364,271,100,518,333,301,65,14,520,51,23,254,13,212,553,21,604,5,363,109,202,938,128,239,709,30,31,127,95,419,64,22,125,578,846,132,191,616,2,154,158,146,124,371,48,647,395,92,67,621,12,242,627,10,916,411,344,424,116,44,403,32,108,512,338,446,737,374,326,166,86,149,793,36,120,688,421,69,827,318,93,293,126,346,731,20,463,37,260,639,450,848,264 }, +{ 129,123,249,618,214,257,460,205,769,184,17,5,245,162,106,45,90,119,76,183,128,2,209,229,61,80,179,497,6,59,130,1,64,202,99,3,152,27,77,19,710,14,207,84,4,306,40,12,29,141,255,57,240,62,26,452,10,790,58,25,269,177,619,556,21,524,38,266,32,742,217,56,136,86,102,519,7,137,115,432,318,33,292,311,11,151,22,402,48,0,216,51,37,378,288,23,210,54,655,667,381,95,36,34,113,65,72,194,403,139,50,305,111,361,504,483,46,833,285,15,494,180,475,317,112,16,263,275 }, +{ 51,719,23,453,899,457,730,13,475,386,670,4,523,15,401,961,11,12,196,99,153,403,32,3,57,117,21,33,450,59,775,165,40,115,130,421,236,77,308,202,128,120,134,758,108,6,65,391,217,90,64,125,177,197,0,361,5,34,26,260,2,19,1,629,300,183,47,216,54,141,328,817,78,123,22,17,180,342,801,7,95,27,37,241,210,214,269,221,14,10,56,237,275,242,194,494,317,558,412,25,743,73,38,247,48,102,28,488,381,318,9,82,29,326,291,711,288,137,53,304,129,46,277,402,339,283,266,24 }, +{ 173,192,66,204,104,74,636,560,426,825,20,7,43,289,750,707,712,97,861,414,214,348,65,715,81,500,90,138,95,64,835,32,312,174,319,22,526,194,280,874,792,52,0,580,91,681,232,29,6,819,330,889,673,879,5,888,86,77,568,519,422,85,376,288,135,729,387,172,370,2,222,33,38,61,10,21,784,355,49,353,107,249,324,25,69,24,18,42,36,577,226,28,356,102,140,76,448,59,92,751,207,115,671,40,325,4,237,31,359,247,524,343,180,470,145,147,397,16,392,153,117,13,120,26,177,597,125,45 }, +{ 23,51,13,202,21,141,5,1,165,120,269,180,48,12,197,32,15,2,318,115,403,125,128,260,450,137,7,9,523,64,82,33,352,93,386,326,0,6,308,4,453,401,421,102,237,37,457,475,31,14,110,351,77,16,3,221,341,391,99,494,30,44,18,117,8,11,817,203,98,730,17,217,39,719,153,899,373,40,196,24,10,61,498,402,72,365,509,50,90,854,569,317,116,28,144,247,134,19,291,59,96,752,45,285,961,89,615,922,342,58,49,25,35,36,22,349,177,20,546,241,57,670,88,54,29,105,576,65 }, +{ 200,78,34,322,472,27,390,714,161,136,19,59,283,444,250,128,236,176,862,108,47,62,527,14,57,412,53,813,73,3,381,17,453,711,56,585,210,4,46,361,77,196,440,26,65,129,51,153,130,432,383,730,216,183,297,803,690,402,401,45,469,202,133,389,123,529,5,495,162,33,90,275,318,754,194,12,378,177,291,32,257,102,38,719,117,184,114,789,480,13,23,11,106,48,503,814,2,61,629,84,684,54,475,245,21,657,555,231,40,678,152,99,300,507,137,119,80,0,436,113,457,369,650,6,403,86,22,523 }, +{ 347,33,204,498,66,546,681,56,174,5,426,95,135,348,97,172,353,864,49,22,928,923,873,808,73,263,241,10,568,370,27,197,180,77,554,207,81,580,102,21,42,237,368,32,751,526,192,134,117,19,0,350,86,63,715,448,468,115,380,123,76,25,4,29,64,44,31,36,6,91,7,511,40,645,387,85,74,861,408,298,80,736,69,173,872,13,501,355,108,712,138,153,87,61,28,17,901,18,467,72,673,59,620,343,232,161,233,312,800,509,324,278,125,45,30,14,650,636,90,3,451,415,577,500,65,561,165,70 }, +{ 131,224,219,187,385,442,258,836,371,944,871,908,737,924,254,127,98,31,827,716,574,44,39,829,116,201,613,276,839,616,242,860,926,284,137,228,395,144,293,36,96,9,1,578,406,30,456,0,299,567,93,21,13,125,202,23,843,160,166,49,51,64,856,105,846,902,203,761,935,938,662,492,128,295,159,83,48,916,438,268,190,100,115,269,5,403,310,272,318,489,367,110,77,141,931,28,10,531,55,326,165,918,688,32,86,576,22,699,88,158,120,717,530,261,244,143,421,341,434,217,2,512,197,960,625,164,37,333 }, +{ 1,4,5,2,13,180,3,15,7,141,6,115,23,352,901,341,51,0,217,317,197,77,752,365,33,21,11,165,12,269,32,9,202,102,291,177,318,93,40,372,247,10,137,31,349,498,22,120,8,308,403,44,17,873,569,110,401,203,116,59,125,36,457,28,326,196,48,304,14,37,494,386,373,98,144,134,153,128,90,64,117,475,758,260,509,72,18,16,206,55,49,523,86,237,210,95,961,629,421,482,453,30,391,253,450,951,719,342,139,638,730,576,39,864,275,268,579,241,19,82,221,24,546,437,89,57,127,811 }, +{ 370,91,681,636,397,718,510,74,81,240,355,66,84,582,278,289,214,511,701,138,90,18,782,6,104,288,5,65,64,32,693,117,247,77,63,192,56,0,814,306,750,673,312,233,85,40,453,847,671,415,11,643,317,207,134,72,61,42,33,102,4,942,519,343,123,115,27,1,86,25,348,249,22,580,537,461,396,352,153,75,484,784,475,403,304,194,21,10,235,29,3,193,119,583,287,196,114,76,59,52,319,256,124,26,23,13,874,579,389,610,462,302,238,17,710,817,597,353,129,96,51,48,217,164,89,835,804,751 }, +{ 15,13,23,4,197,51,165,1,2,120,180,341,0,260,386,5,8,82,12,7,9,3,141,450,317,153,11,10,391,217,6,351,326,115,64,77,14,457,117,901,498,523,854,509,291,110,134,730,401,758,33,961,237,352,22,202,898,125,922,93,21,365,308,196,31,44,615,269,128,817,99,453,801,28,203,899,116,373,873,752,629,18,40,55,349,137,98,37,318,372,403,36,49,475,569,32,38,86,546,95,16,661,143,704,102,719,30,342,59,39,35,17,642,304,494,482,48,24,638,144,177,300,194,139,958,29,54,492 }, +{ 32,76,292,2,5,476,45,214,72,29,1,241,14,519,464,460,21,179,48,129,290,90,17,710,148,205,123,263,249,50,58,358,52,425,186,243,408,46,257,618,207,38,245,184,89,227,155,769,162,270,171,350,3,7,288,119,61,6,106,57,316,64,54,400,449,128,183,229,255,75,4,141,36,496,115,152,524,655,99,209,720,334,497,62,12,202,59,77,37,86,10,405,0,56,51,221,130,666,23,25,40,592,22,181,237,269,11,247,432,137,16,217,454,84,27,452,428,24,663,619,19,513,318,177,659,26,67,341 }, +{ 13,569,115,165,509,197,629,734,17,873,23,659,106,457,180,45,41,32,961,901,38,536,341,428,62,155,587,474,391,54,21,431,99,546,255,51,243,528,308,365,136,241,37,468,405,89,123,129,14,482,181,2,211,401,139,72,12,148,542,719,221,117,794,214,76,749,498,449,350,317,292,157,743,540,29,758,328,152,552,188,162,11,3,119,720,488,339,770,413,237,227,179,50,248,48,209,697,313,90,741,476,130,4,477,128,1,773,519,277,40,421,507,46,266,57,249,33,245,361,137,153,202,453,369,471,64,133,358 }, +{ 68,101,24,35,71,16,154,124,9,0,149,118,175,65,302,18,108,375,346,28,55,92,252,364,212,429,191,7,439,60,694,39,520,14,398,806,100,22,109,329,166,167,484,273,336,604,67,69,419,189,562,21,826,589,1,701,695,345,95,140,230,265,246,49,10,264,48,158,114,637,755,168,103,917,133,46,628,2,64,722,126,650,399,88,404,535,125,768,570,289,150,458,13,706,83,30,50,36,388,32,420,23,267,12,185,5,882,674,818,96,169,51,676,925,37,363,283,602,86,105,38,221,333,967,159,286,942,553 }, +{ 0,9,68,35,1,65,101,230,69,28,124,23,83,39,141,154,67,13,21,352,212,217,51,30,100,252,88,16,336,115,48,175,864,103,24,317,189,5,180,114,166,128,96,443,36,399,10,12,238,64,329,562,264,478,177,32,18,265,752,77,55,22,321,372,291,218,160,702,149,520,167,50,437,304,202,498,247,137,118,269,99,49,596,365,589,221,759,486,349,197,165,382,346,323,410,2,664,480,223,419,213,777,604,433,418,75,7,4,335,482,15,547,532,37,251,363,237,125,936,72,143,755,341,553,308,190,109,201 }, +{ 909,265,100,410,382,812,88,593,321,213,1,30,252,166,676,639,264,435,466,230,689,39,9,48,724,352,478,21,98,553,96,329,190,223,101,23,274,51,217,13,83,5,921,310,699,438,795,125,69,821,905,530,293,752,759,35,36,656,557,128,144,393,443,520,0,541,284,521,604,137,419,32,682,831,141,455,317,418,664,855,854,251,31,165,772,221,404,386,93,201,180,203,33,686,421,335,539,840,115,55,127,44,957,160,285,158,28,344,68,67,367,77,202,345,254,697,102,669,562,897,807,702,765,286,10,351,64,269 }, +{ 113,61,904,43,0,204,848,453,389,315,788,198,174,5,22,899,222,59,37,102,33,207,422,290,288,128,97,329,306,46,31,30,77,470,817,523,159,58,40,942,122,65,725,690,324,291,104,64,263,450,426,20,54,337,304,247,205,196,153,134,117,105,95,49,408,757,615,351,326,194,115,82,28,18,9,177,137,32,350,887,372,347,331,325,217,90,48,2,701,344,342,320,316,158,317,804,376,341,819,319,849,685,666,600,580,579,553,501,498,403,392,365,357,349,330,301,270,262,242,237,180,112,76,44,39,36,7,147 }, +{ 24,16,141,71,18,0,352,269,75,108,140,7,60,35,330,12,199,217,65,9,486,28,498,864,87,246,64,92,68,50,318,267,126,954,118,115,55,230,70,5,146,177,950,545,13,372,197,752,271,101,98,14,133,77,180,641,349,317,99,36,726,458,23,537,487,156,102,437,22,10,622,165,21,563,516,832,202,2,341,149,48,565,86,457,304,273,220,237,116,44,674,526,117,100,39,33,120,49,37,365,360,830,58,137,103,291,135,522,181,166,121,88,54,51,412,167,284,223,46,15,394,1,696,852,463,361,264,212 }, +{ 271,167,121,60,199,146,191,463,18,446,212,132,575,126,598,354,407,16,955,727,640,566,1,608,195,35,562,279,522,21,48,107,98,71,118,783,223,493,150,467,68,589,24,5,360,781,92,156,363,591,411,668,447,55,13,534,818,101,23,9,108,346,581,647,777,590,49,621,0,259,125,362,936,664,939,7,273,28,31,22,51,163,32,807,10,535,834,221,147,520,246,33,293,541,50,37,586,14,815,375,393,4,116,12,105,96,70,2,44,77,39,93,252,149,281,441,46,158,75,61,759,154,97,649,95,423,364,140 }, +{ 15,115,217,317,365,341,180,13,141,33,23,260,120,4,82,110,102,351,51,40,854,77,372,137,352,21,64,1,5,32,36,10,197,117,898,752,11,901,269,202,922,165,59,9,160,244,0,661,373,134,2,143,22,48,457,349,153,253,450,12,386,196,318,482,247,326,128,308,177,55,72,90,498,523,7,125,961,719,93,28,37,86,3,291,811,509,237,96,57,873,16,437,89,579,391,730,111,203,31,864,18,6,615,35,468,210,216,14,569,275,139,164,304,17,342,76,899,403,221,494,642,316,817,251,30,531,717,95 }, +{ 173,97,356,7,43,387,107,426,104,729,414,359,232,20,560,392,595,885,858,861,707,819,66,81,612,5,49,22,362,319,126,192,230,32,660,330,0,74,357,92,246,147,784,715,14,324,298,172,470,879,95,263,2,422,259,16,427,38,314,207,683,61,376,140,21,681,35,6,29,397,91,60,18,241,10,279,28,64,52,77,24,830,500,493,31,636,33,37,501,12,4,86,779,204,135,58,577,370,65,568,441,40,199,48,526,13,98,581,712,105,534,125,325,17,102,72,44,347,750,138,120,70,59,46,465,25,45,30 }, +{ 15,23,13,82,120,51,110,93,31,33,116,98,95,196,453,4,165,40,197,260,117,0,386,2,180,217,12,1,351,11,77,5,8,134,102,9,141,326,457,115,341,291,3,153,10,401,21,450,317,523,854,202,352,14,7,391,898,6,64,961,373,719,22,125,475,901,28,922,615,403,18,177,269,498,203,237,44,349,99,758,509,32,30,318,128,37,817,16,899,629,308,57,365,670,36,421,49,139,275,661,730,111,569,137,24,39,48,86,704,144,801,752,35,17,65,109,494,342,558,38,372,551,221,216,170,46,55,638 }, +{ 100,9,120,260,30,827,795,373,264,77,137,166,202,326,127,578,1,39,254,82,704,318,217,450,18,101,395,737,351,0,731,291,265,317,156,349,341,31,716,105,102,93,141,197,55,33,968,98,269,5,296,128,478,131,24,851,252,371,342,32,859,938,921,44,616,411,403,404,385,924,144,109,165,40,456,274,180,10,494,836,16,512,854,410,299,195,110,68,28,7,518,442,36,438,286,242,13,244,167,96,363,850,379,656,521,284,95,523,125,64,658,615,340,116,67,922,212,115,447,187,937,935,916,724,936,856,822,372 }, +{ 14,24,2,67,69,46,28,108,1,65,38,83,7,398,18,0,124,154,114,103,36,50,267,133,29,55,236,189,149,35,16,547,52,68,429,175,283,71,336,9,160,118,309,22,302,251,109,458,646,75,6,140,388,86,135,484,278,412,12,596,5,389,650,172,531,54,238,60,218,420,602,96,21,66,208,87,886,220,194,505,782,150,185,169,92,64,10,891,168,95,25,101,797,335,289,533,570,532,74,906,37,230,48,357,49,273,262,323,366,696,346,158,23,516,443,39,32,799,70,565,58,705,137,51,13,439,418,128 }, +{ 66,69,14,267,189,24,71,7,150,172,140,97,149,16,60,380,135,462,2,35,67,74,388,92,38,18,451,278,108,29,83,68,251,6,357,52,46,1,230,443,36,324,423,298,458,65,273,28,336,517,314,823,232,192,398,330,711,118,594,103,87,0,75,565,175,55,335,399,705,124,22,86,25,21,603,448,906,568,204,95,48,12,50,236,5,563,309,43,550,10,154,138,64,32,345,302,37,596,893,20,133,572,54,467,114,126,607,42,26,160,246,220,733,375,809,226,429,526,501,723,9,895,49,808,70,500,602,915 }, +{ 15,1,13,5,2,341,4,365,3,0,23,10,11,7,12,180,6,117,9,17,33,21,134,115,77,14,32,217,197,102,317,165,8,141,457,153,64,40,901,37,51,128,22,961,569,291,202,196,16,31,269,177,352,36,137,38,873,372,28,93,19,351,59,55,349,111,44,509,342,498,752,18,48,125,318,99,304,139,120,386,143,237,116,308,72,401,86,391,110,30,95,54,629,27,482,210,76,45,164,24,89,79,29,90,453,403,25,719,57,275,203,216,82,98,41,758,579,277,247,46,260,49,523,244,26,285,326,144 }, +{ 1,22,12,5,28,36,0,21,2,86,49,48,95,55,37,23,10,50,9,7,96,13,202,32,51,105,14,54,208,168,65,115,128,24,83,16,160,67,4,141,137,31,64,99,318,3,269,194,109,6,323,18,164,159,17,114,35,170,239,281,402,285,180,11,238,745,69,584,103,197,165,480,158,45,341,181,169,403,77,244,44,46,38,218,143,98,433,289,401,30,223,58,717,125,657,503,352,217,851,33,177,185,29,946,597,757,708,39,365,312,317,72,800,253,75,789,102,678,124,418,133,237,59,956,90,127,308,430 }, +{ 123,605,630,63,108,736,444,65,650,368,19,561,619,389,122,216,25,114,183,177,26,53,27,813,133,42,102,194,84,445,249,899,297,56,421,349,90,401,33,597,588,250,137,78,73,130,11,730,51,283,45,524,629,235,59,57,14,141,196,153,32,40,343,255,719,134,117,467,304,119,76,17,13,0,152,6,436,77,390,4,323,511,678,475,457,136,64,594,342,23,403,256,12,129,503,218,208,86,62,61,5,416,233,372,52,3,469,184,275,381,106,947,789,554,214,112,680,450,95,309,217,18,9,529,432,291,269,80 }, +{ 127,39,9,254,98,395,371,31,0,385,836,44,1,30,299,284,100,166,116,578,131,187,272,28,935,49,293,276,21,125,23,16,219,310,599,51,737,442,827,48,242,616,137,18,101,908,5,144,13,846,24,265,456,716,35,512,96,252,93,88,202,22,201,68,159,36,158,7,77,55,224,924,938,141,492,203,128,190,688,269,518,258,32,333,64,37,318,228,2,65,829,326,274,244,761,165,12,115,67,286,120,931,217,110,367,404,105,839,33,102,567,83,268,916,662,95,438,14,86,50,944,109,574,557,69,327,625,10 }, +{ 2,29,61,45,151,6,655,112,207,52,25,425,32,833,76,7,70,106,1,113,14,119,292,790,58,26,72,5,887,64,209,128,17,90,255,315,66,19,145,4,186,306,243,476,115,59,483,27,77,74,21,177,311,38,87,86,40,263,141,738,46,290,36,23,148,454,287,10,240,95,42,22,84,16,137,0,179,202,135,33,631,491,814,75,227,51,269,89,50,210,65,214,20,12,80,222,48,318,24,28,49,13,11,216,18,155,358,102,99,56,464,122,37,644,138,549,710,54,180,337,205,117,496,67,172,869,123,720 }, +{ 2,17,5,29,27,3,6,401,45,58,51,26,23,19,53,52,14,34,46,453,151,133,366,13,61,236,50,218,114,1,73,99,308,487,603,108,21,25,181,670,287,421,297,80,115,396,48,74,56,137,171,412,764,445,202,38,281,62,457,42,65,629,516,480,475,341,113,549,64,573,600,386,57,403,894,250,106,730,54,436,651,491,86,145,12,334,208,155,832,128,328,614,361,262,165,283,84,32,719,323,745,200,90,141,37,36,711,657,631,67,760,754,583,739,635,569,7,594,488,315,161,10,285,59,402,758,243,430 }, +{ 22,170,31,301,127,1,98,44,28,36,239,105,55,299,512,293,86,395,116,685,242,841,9,95,125,338,599,900,2,0,5,654,23,320,169,109,13,456,870,49,64,185,763,10,51,7,237,12,141,144,164,479,143,965,312,35,137,923,21,268,115,14,933,492,128,168,393,284,506,67,286,762,276,83,37,244,6,114,202,160,180,928,24,352,634,194,367,201,197,159,308,165,269,253,558,844,401,538,48,65,208,272,935,391,584,50,420,341,365,131,203,913,318,16,498,3,489,327,96,4,457,374,317,892,386,32,434,638 }, +{ 14,21,50,48,5,2,1,32,270,46,38,290,76,72,45,155,54,17,263,181,408,207,425,281,29,430,221,67,36,96,292,89,449,52,428,37,476,513,241,83,103,119,0,171,7,541,720,528,3,12,10,24,227,148,22,58,179,61,536,502,214,243,655,6,413,90,9,350,23,218,106,51,16,86,69,129,65,186,28,133,115,114,99,431,552,400,760,794,464,64,474,128,433,123,328,460,18,405,137,487,306,84,108,477,245,710,35,236,205,4,13,787,27,238,308,587,309,770,141,540,19,56,790,240,95,223,488,202 }, +{ 17,119,106,378,240,383,136,84,62,80,306,790,56,504,742,3,440,207,209,128,255,483,123,45,5,667,61,32,57,64,179,27,129,14,205,548,297,4,245,2,214,311,141,90,11,76,833,655,184,151,29,361,59,469,152,249,26,38,162,19,58,215,328,257,73,495,161,34,54,229,269,202,250,266,292,21,12,402,514,555,216,113,194,887,130,1,445,503,507,183,6,25,318,237,186,72,52,814,710,618,99,10,86,211,148,452,263,243,65,137,37,40,41,23,36,33,155,497,13,403,53,95,313,227,494,180,77,738 }, +{ 13,1,2,15,23,5,115,0,51,10,4,3,6,341,7,11,365,217,180,317,9,33,197,77,165,40,22,12,14,102,64,141,117,153,196,453,457,8,202,901,17,21,269,18,128,28,125,569,31,386,59,318,32,16,134,349,352,291,372,237,36,719,403,752,475,873,961,730,509,391,177,137,24,401,95,57,44,37,93,498,120,308,139,494,110,98,326,111,247,86,304,30,482,49,143,48,55,629,210,260,758,523,19,351,65,450,72,116,46,76,342,29,275,203,50,170,105,164,89,82,734,242,27,38,54,144,899,90 }, +{ 772,335,662,96,367,744,897,1,303,686,264,521,652,340,116,520,141,31,919,201,966,260,888,203,44,64,905,321,144,225,269,98,691,658,349,284,410,698,692,363,120,88,848,717,863,187,551,914,909,418,576,202,646,443,850,639,523,69,404,336,958,293,128,730,131,615,274,382,296,166,960,948,258,228,224,219,751,486,160,125,82,411,191,765,244,223,475,453,352,326,100,318,676,596,67,351,899,703,403,389,435,947,589,498,102,33,954,291,265,156,922,854,402,93,759,664,447,372,212,137,478,329,724,562,530,286,217,942 }, +{ 1,28,5,0,36,24,65,83,50,14,67,22,12,96,2,48,218,114,281,54,18,21,9,133,55,103,46,181,37,58,160,38,16,7,99,10,223,69,238,32,430,433,502,45,35,51,766,49,108,86,23,124,361,155,487,765,532,283,75,137,270,115,17,838,163,109,480,95,13,61,6,543,154,754,513,70,29,72,202,540,822,657,323,920,398,89,68,64,141,966,3,221,208,285,128,290,956,547,488,269,236,175,168,429,194,474,800,646,171,39,412,775,52,308,531,866,149,880,389,4,180,403,318,842,834,402,217,185 }, +{ 138,174,348,42,25,6,145,81,544,280,525,74,85,26,648,204,355,287,396,91,233,66,636,104,122,746,416,29,64,65,90,45,0,235,194,681,87,22,222,651,95,624,573,32,214,289,750,415,343,445,192,52,112,86,17,370,19,5,135,881,7,151,2,337,113,605,243,63,389,256,49,59,10,4,673,680,77,18,297,61,312,76,614,97,72,249,283,11,524,70,353,33,27,28,216,114,874,173,311,57,43,21,14,597,36,1,141,102,58,24,217,12,436,665,80,172,715,792,148,123,619,56,133,20,491,75,784,117 }, +{ 24,16,35,68,0,18,158,28,71,159,49,7,55,154,109,14,22,60,105,124,344,65,118,424,9,239,95,1,320,2,149,577,286,170,92,169,108,347,12,21,327,374,175,185,67,10,272,5,101,51,48,23,302,455,39,6,671,301,46,13,69,310,20,518,64,150,36,346,50,32,137,202,709,677,191,126,98,125,375,168,538,289,164,38,37,128,638,30,703,747,890,273,127,634,31,419,140,143,439,86,403,114,252,83,962,654,221,762,623,338,115,29,212,103,701,570,308,107,512,189,54,97,167,230,479,345,312,913 }, +{ 854,699,230,352,93,689,144,251,160,137,855,202,617,203,217,36,669,332,450,120,473,351,831,201,213,752,326,800,206,141,258,748,82,418,345,30,180,816,957,318,840,190,443,228,44,35,863,968,10,1,219,871,704,317,64,22,679,83,466,539,498,224,55,96,898,31,116,406,851,384,86,244,295,28,873,98,817,531,187,822,864,373,494,131,559,372,530,901,574,365,69,127,839,735,128,268,67,533,442,868,860,922,88,269,927,859,9,367,613,253,564,930,567,261,260,728,39,237,109,284,276,164,730,489,166,926,125,776 }, +{ 15,13,5,1,2,23,3,0,317,51,115,4,10,11,7,6,217,33,9,77,17,14,341,165,102,117,180,365,134,40,197,153,64,22,12,32,196,21,8,141,391,110,752,372,291,498,352,125,569,16,36,901,202,59,260,28,237,482,349,18,269,457,961,120,318,93,57,509,37,29,128,177,49,31,19,401,48,453,342,38,719,873,45,55,629,111,24,44,143,468,386,50,54,403,86,139,27,308,304,95,210,546,730,475,46,137,98,247,35,72,90,99,82,76,437,494,864,30,326,758,96,52,275,116,402,277,164,56 }, +{ 104,173,20,319,43,66,426,355,712,77,97,0,202,7,198,74,580,204,174,500,32,5,59,2,102,337,90,325,226,234,4,52,22,450,904,155,222,376,861,21,750,33,192,414,194,147,324,342,889,58,544,636,18,291,416,112,784,81,334,64,707,95,331,422,639,715,36,400,14,141,560,792,65,29,177,49,37,315,427,28,57,10,120,24,1,40,48,107,673,11,494,318,230,61,524,128,196,16,12,470,632,232,138,115,329,455,113,142,186,153,819,294,17,525,526,70,117,87,25,247,587,105,269,583,588,158,171,348 }, +{ 23,13,51,5,15,1,2,115,0,141,12,7,4,21,6,165,32,3,269,9,10,457,202,180,197,14,77,11,37,453,22,318,352,120,33,719,217,341,401,403,16,961,17,40,308,18,31,28,8,48,64,475,569,730,260,365,59,128,247,317,137,177,90,93,237,102,99,629,523,44,386,117,36,450,110,153,30,752,899,38,349,72,134,98,509,498,49,326,901,125,24,391,196,421,57,82,291,76,95,19,203,494,89,45,817,304,468,221,139,210,758,351,546,65,373,372,96,54,482,734,144,116,86,39,29,216,402,288 }, +{ 15,1,13,0,2,5,4,23,3,317,115,51,7,6,341,9,8,365,217,22,10,33,77,31,457,197,44,901,18,12,28,180,11,120,165,202,961,102,719,569,40,308,21,730,20,260,351,523,141,36,401,64,482,82,55,49,450,453,403,269,93,318,16,30,498,468,342,14,372,352,110,61,817,125,137,386,117,70,899,25,898,291,509,196,494,475,24,629,854,39,153,349,98,32,134,221,391,416,234,758,59,37,237,95,86,19,91,48,27,177,247,326,903,17,734,873,89,128,421,105,922,57,111,143,638,300,304,90 }, +{ 15,1,3,2,0,4,10,5,9,13,82,12,8,11,16,7,31,23,351,898,365,93,44,854,22,64,21,457,217,180,6,141,317,730,110,318,18,752,719,352,341,51,961,28,269,33,260,77,922,120,202,291,98,55,372,197,349,901,105,165,30,450,14,817,494,523,403,899,128,125,453,95,36,115,116,475,386,326,49,401,111,94,811,642,758,479,474,391,373,203,201,187,153,144,137,38,37,24,102,59,421,282,206,48,569,342,275,216,210,864,846,801,795,692,661,597,559,442,253,244,230,196,177,170,143,127,123,89 }, +{ 2,5,14,1,29,50,38,46,45,17,52,58,114,133,171,218,21,181,48,186,36,6,155,3,19,65,487,12,32,54,27,61,480,323,70,22,51,86,361,137,96,0,208,115,285,227,283,151,23,412,10,281,28,108,37,236,112,328,7,334,163,400,25,4,223,26,67,75,657,194,18,103,128,24,13,514,745,99,401,202,308,532,59,803,389,16,760,11,243,72,64,80,106,9,502,83,832,113,453,56,552,421,403,503,402,95,735,341,956,69,124,782,366,90,141,409,77,57,160,822,433,89,209,244,40,76,177,484 }, +{ 68,101,0,9,212,124,154,65,149,35,16,100,118,24,252,71,562,589,189,336,114,167,265,191,39,329,398,88,67,429,1,264,478,69,166,484,55,30,777,18,399,520,759,13,818,722,175,419,230,48,283,664,796,115,23,28,375,724,346,140,133,103,410,21,705,806,64,51,701,604,83,246,7,345,158,389,60,36,382,302,14,2,108,50,755,5,695,694,237,702,547,917,92,639,411,936,732,213,267,238,32,404,190,10,364,321,289,49,180,553,447,247,650,72,218,46,593,925,468,458,96,905,596,217,95,546,223,439 }, +{ 14,2,1,22,46,67,7,65,28,24,0,69,38,168,50,36,83,289,5,12,16,29,35,55,86,114,103,133,109,49,18,158,95,108,6,194,584,124,185,610,52,169,10,159,189,105,51,23,21,484,283,9,505,389,236,128,239,267,420,164,13,547,143,54,32,64,71,115,149,75,37,140,251,68,48,208,160,253,708,309,137,96,154,244,312,170,782,202,320,412,577,175,308,424,25,141,336,429,218,344,60,892,835,874,347,398,401,531,230,17,278,388,92,135,797,638,732,237,3,323,39,19,286,99,747,4,66,403 }, +{ 23,13,51,15,0,10,115,1,4,2,9,961,457,77,11,5,453,12,33,3,403,14,40,102,197,7,22,165,59,475,64,6,37,401,21,569,719,18,308,99,32,17,177,730,16,141,523,8,341,31,629,217,180,117,386,202,153,196,120,57,28,49,111,125,95,139,899,494,128,421,72,65,210,317,36,342,391,90,19,93,89,44,86,269,365,24,291,237,110,76,318,873,509,482,54,48,300,498,247,134,450,82,275,260,352,98,27,137,326,38,304,758,148,50,79,817,45,670,372,30,179,29,203,734,152,116,349,216 }, +{ 15,1,0,2,5,9,7,23,13,51,901,8,6,12,3,4,180,10,719,341,120,14,31,115,22,141,898,197,82,260,93,21,28,44,457,37,217,125,202,16,153,351,64,110,401,18,11,33,36,961,569,30,77,49,352,450,98,269,139,317,117,116,32,40,453,326,308,128,523,318,365,203,137,730,24,629,854,403,102,39,237,165,55,291,99,475,95,196,48,86,498,134,752,899,373,391,817,244,38,105,242,421,922,59,144,386,57,143,35,349,54,509,482,372,65,253,221,177,304,494,247,638,17,170,72,127,164,114 }, +{ 100,252,265,88,724,101,382,410,48,676,213,909,321,9,593,0,812,329,553,21,352,1,30,230,166,39,419,520,23,51,264,217,221,478,689,435,190,664,13,604,466,677,694,125,752,777,36,98,699,639,83,759,96,35,158,68,128,562,141,223,344,274,310,69,455,115,5,180,404,317,656,364,203,293,831,438,175,308,443,401,333,31,144,570,154,854,105,137,212,346,49,857,530,418,840,855,201,55,921,28,539,421,557,806,821,127,124,67,251,10,634,165,936,22,610,795,16,286,957,167,541,32,962,905,682,686,241,44 }, +{ 58,171,2,46,5,50,14,29,186,1,52,45,334,151,400,38,155,61,103,96,67,227,502,281,223,54,48,17,83,181,552,69,70,309,75,163,24,549,21,311,27,366,80,536,243,113,218,787,106,36,112,433,765,430,209,133,12,6,236,487,19,780,675,394,766,238,587,545,32,37,767,807,834,7,108,114,328,287,315,65,262,820,16,496,583,26,18,454,513,0,3,600,220,25,474,86,307,73,28,35,255,869,537,631,22,72,89,491,409,357,64,270,828,267,189,845,160,221,10,592,137,119,99,659,128,516,541,785 }, +{ 23,202,120,141,51,13,318,269,21,128,260,165,137,197,180,326,1,450,5,77,33,82,15,48,93,64,110,351,32,125,403,203,102,523,854,386,352,457,922,115,7,2,12,494,373,817,615,196,342,153,221,217,16,6,210,144,402,453,36,4,391,14,37,730,9,96,475,291,98,0,99,901,3,421,551,177,18,40,401,237,752,349,31,898,86,719,341,24,704,285,117,11,308,393,116,35,61,160,8,30,65,90,59,44,275,10,241,50,39,216,509,372,67,72,961,247,22,899,28,569,293,83,139,801,473,76,95,46 }, +{ 341,197,901,10,13,569,23,638,15,479,8,873,961,180,165,143,12,115,0,1,951,509,116,37,642,365,499,498,242,253,5,36,898,4,752,2,352,141,268,206,51,55,401,391,7,22,3,417,506,492,82,9,93,11,137,456,338,244,457,64,317,120,110,629,351,546,431,33,21,386,31,86,260,217,164,6,758,965,28,762,474,918,661,326,551,153,125,14,719,94,421,854,203,801,373,269,44,16,730,202,615,237,844,791,811,523,450,40,18,128,864,282,117,111,148,824,940,102,959,17,139,559,170,32,958,867,968,38 }, +{ 23,1,21,51,202,13,15,141,120,5,180,2,197,165,7,269,352,48,12,318,260,326,137,33,386,403,9,450,0,32,6,4,115,128,14,16,523,457,102,37,110,82,93,18,217,3,64,221,752,421,203,453,237,125,77,153,351,494,308,475,196,401,719,31,44,391,373,817,24,730,8,342,30,36,11,40,341,901,96,50,99,615,98,116,117,39,10,22,498,317,17,59,35,160,144,86,28,20,854,961,291,244,177,899,72,402,509,90,922,350,349,95,29,247,285,19,127,661,365,372,49,569,758,65,576,241,898,61 }, +{ 2,29,58,50,262,52,53,27,6,151,14,1,171,5,287,34,324,17,74,46,26,113,309,396,45,223,61,334,96,73,635,445,583,38,19,75,67,103,80,357,70,331,181,573,112,785,236,675,163,281,549,145,366,54,108,315,760,218,220,491,453,297,12,138,62,250,133,436,51,409,200,487,600,128,502,37,64,106,114,651,48,83,394,24,25,820,433,59,766,307,23,155,36,614,311,65,66,137,57,4,765,186,69,21,99,202,56,13,3,587,165,767,7,631,238,86,780,115,283,161,322,545,221,42,32,739,400,47 }, +{ 13,51,23,141,202,115,217,77,317,4,269,33,5,11,318,352,0,102,180,2,64,10,341,7,21,9,137,3,40,128,32,6,22,291,752,15,197,12,365,1,165,31,125,28,498,177,16,403,494,237,18,308,44,59,372,98,349,24,93,48,244,386,49,90,30,342,95,901,475,39,120,196,14,117,457,116,453,17,247,391,153,8,143,482,36,304,203,110,509,326,450,260,37,134,144,210,468,523,86,221,401,569,873,546,719,139,253,35,160,127,285,105,373,421,961,864,576,82,367,437,351,55,579,19,216,275,402,164 }, +{ 1,24,67,50,69,103,35,18,16,14,28,0,5,83,9,54,223,2,96,58,181,12,46,68,48,65,108,133,71,281,149,22,7,163,21,114,36,75,38,55,155,502,238,37,189,99,70,218,45,10,430,433,236,60,765,251,171,61,175,540,23,820,32,51,418,270,29,487,513,52,124,118,336,17,64,49,920,154,13,141,115,545,72,267,221,537,745,290,86,488,309,128,335,39,227,89,328,160,95,834,361,6,101,486,514,543,112,137,474,394,186,202,388,109,150,140,552,807,412,283,307,345,87,780,366,398,77,516 }, +{ 13,23,51,141,202,115,77,269,64,217,102,128,318,33,32,291,177,180,137,10,352,4,21,16,197,0,11,22,9,7,165,90,40,494,125,2,210,247,14,18,153,49,59,31,6,95,317,28,196,349,24,304,5,341,403,342,98,237,117,120,372,44,3,275,35,86,48,1,93,752,36,15,241,12,475,144,30,19,17,216,110,365,260,116,326,134,82,105,143,308,203,20,39,402,386,221,901,127,72,498,437,244,453,76,401,65,38,96,285,316,160,55,351,25,288,482,164,57,139,457,678,490,201,523,509,391,83,817 }, +{ 115,365,180,317,873,498,752,482,341,569,217,901,13,197,352,546,468,1,864,457,509,579,165,23,143,21,15,2,185,36,7,148,12,134,332,22,351,9,261,51,16,5,10,0,346,93,30,141,868,734,372,253,345,116,474,94,77,876,76,28,896,679,384,203,160,39,37,6,98,14,391,109,530,187,843,177,86,629,33,637,11,437,18,4,902,574,434,144,120,44,31,837,72,613,295,169,349,505,228,190,102,95,49,48,706,406,405,96,83,961,617,597,254,237,926,903,717,531,420,258,164,110,860,268,661,477,464,449 }, +{ 184,257,229,205,432,497,152,619,452,17,381,57,266,524,5,45,90,106,313,471,245,32,128,129,59,4,214,2,339,119,769,77,369,12,123,21,14,99,157,3,202,38,710,1,618,216,64,217,292,188,162,76,11,40,249,141,62,51,33,19,372,102,179,37,209,23,27,6,111,318,29,61,403,86,460,349,207,54,269,402,10,48,139,115,36,136,79,137,7,255,72,80,177,519,291,453,529,401,288,58,0,153,475,377,247,306,25,56,26,94,210,22,84,494,50,790,89,775,474,361,328,148,241,588,285,317,678,277 }, +{ 13,23,1,2,0,51,10,5,115,4,15,3,365,7,9,33,11,180,77,6,217,341,22,102,317,40,197,14,12,64,141,165,18,8,17,31,202,28,125,16,269,21,117,901,453,196,177,59,153,36,128,475,318,569,32,24,352,291,403,134,93,401,98,44,752,237,391,372,49,509,457,961,873,308,498,386,95,342,349,116,137,37,139,494,19,30,210,247,143,304,86,629,48,203,111,55,57,719,523,110,482,546,120,38,65,54,326,421,216,730,50,99,899,758,402,260,105,275,39,72,450,96,144,35,25,242,29,437 }, +{ 626,70,687,771,379,767,846,518,878,49,937,591,82,761,31,786,481,407,845,195,590,351,98,44,163,557,13,120,125,48,87,883,371,623,360,817,450,260,178,522,21,5,854,523,116,326,105,291,142,4,296,929,884,259,127,0,132,922,615,77,713,293,217,197,165,22,146,64,931,608,465,457,349,196,32,941,455,279,621,315,202,201,126,110,40,16,11,10,391,666,467,446,373,354,346,144,58,33,581,763,512,916,535,795,768,658,558,447,344,150,92,51,28,24,9,265,212,93,586,562,180,156,147,134,117,1,783,663 }, +{ 264,909,410,897,772,1,686,521,478,914,98,335,96,284,691,639,382,44,744,321,724,100,765,438,223,662,88,144,475,367,31,265,125,116,921,9,682,656,166,363,127,225,21,201,842,0,447,905,393,48,443,421,5,82,329,252,120,805,411,51,821,13,919,69,834,948,934,664,33,23,435,551,274,77,217,131,268,7,93,854,418,520,340,110,30,213,141,453,165,101,39,791,260,922,403,238,698,898,351,102,593,433,64,203,83,303,286,293,197,32,960,812,697,523,466,285,180,692,404,658,35,873,254,163,352,553,190,615 }, +{ 13,23,2,51,1,5,115,0,10,77,15,217,7,6,180,33,3,317,40,4,141,102,9,197,11,341,14,64,165,22,365,17,117,196,352,202,12,269,153,16,291,237,453,128,457,498,752,32,18,134,177,59,21,125,386,318,901,28,8,372,31,349,391,403,95,475,36,137,569,49,308,961,401,509,482,143,98,19,24,44,730,86,120,719,139,546,402,342,93,304,110,29,210,421,873,45,38,629,57,260,326,247,111,275,30,523,55,90,450,46,116,253,50,37,244,65,494,105,99,468,758,25,170,48,96,216,35,27 }, +{ 141,269,77,115,13,318,64,202,177,23,137,217,102,291,5,51,180,48,21,352,0,32,7,16,33,18,11,2,197,128,4,15,10,752,24,244,14,494,317,165,96,9,3,349,247,237,93,35,28,12,6,49,44,1,421,60,20,403,36,341,95,22,475,372,326,308,275,196,144,90,40,39,117,31,678,954,901,304,284,268,221,203,110,98,89,68,46,365,153,134,76,67,576,453,432,402,381,342,179,125,118,86,72,71,69,59,50,17,498,160,164,143,120,111,82,961,873,851,401,377,335,288,285,241,223,210,121,116 }, +{ 23,13,51,15,0,10,453,2,115,1,40,961,33,11,14,4,9,457,22,401,77,102,7,5,569,475,197,12,64,3,21,165,59,719,629,6,196,18,177,386,37,32,17,111,153,57,117,139,31,16,28,730,308,99,38,141,8,403,36,128,670,391,217,210,72,90,523,79,482,341,202,237,45,474,180,125,41,509,46,494,304,275,48,54,95,94,899,29,277,134,120,76,758,65,44,247,498,24,291,148,89,468,86,49,437,82,56,30,402,143,365,349,317,93,50,757,269,734,342,98,352,450,678,62,188,19,152,328 }, +{ 1,22,36,5,12,0,105,28,2,49,31,21,95,9,86,55,13,23,128,7,141,98,170,44,115,239,37,48,127,83,160,4,180,164,51,202,10,269,3,96,6,208,159,137,67,244,320,197,312,50,318,352,64,285,301,125,341,32,165,109,116,217,14,402,708,39,365,16,654,35,30,242,317,169,168,99,253,15,678,11,69,751,569,54,531,752,8,901,947,338,735,77,24,238,143,323,498,158,65,401,17,237,293,946,18,717,740,597,93,512,638,286,403,395,892,144,347,308,299,424,457,185,418,713,194,221,757,824 }, +{ 100,252,101,88,265,0,382,759,329,410,724,9,213,321,553,676,419,593,166,39,21,30,909,694,562,520,68,13,812,264,23,435,677,230,35,352,478,48,190,154,689,212,1,217,570,51,466,639,124,221,83,806,604,175,699,826,703,125,98,308,55,671,664,346,32,158,610,706,96,962,274,31,317,289,344,69,438,49,28,831,917,115,656,455,936,404,165,116,208,777,310,5,854,857,16,201,128,36,333,752,194,67,24,71,144,795,391,197,589,141,818,364,921,131,44,86,203,251,748,105,65,22,180,557,149,702,350,855 }, +{ 539,88,840,190,30,321,530,957,435,748,669,778,144,96,100,466,418,863,410,203,228,213,382,274,795,406,613,131,36,822,1,83,326,856,219,352,335,494,251,230,679,31,224,960,44,831,855,141,859,944,217,9,367,160,201,137,625,128,166,93,258,662,120,800,521,403,116,98,187,28,576,752,816,269,202,557,39,931,21,265,686,616,443,921,10,180,731,438,689,523,473,125,871,317,23,77,197,457,827,55,264,35,617,692,254,33,719,260,86,966,817,32,829,206,276,127,242,475,456,284,51,615,252,0,854,238,492,551 }, +{ 7,97,173,92,232,298,314,107,387,729,356,43,359,126,246,140,362,441,501,426,392,230,560,20,14,707,885,104,16,66,330,779,172,683,60,35,493,861,22,357,5,0,95,49,81,279,38,568,18,24,21,74,192,71,414,830,423,259,324,2,121,526,125,633,508,32,672,52,319,10,150,48,28,147,29,46,715,347,6,64,199,380,189,61,858,69,660,33,204,135,649,713,36,595,1,98,37,879,31,146,132,75,819,263,784,443,408,607,448,54,12,663,138,68,58,117,102,70,13,267,65,55,712,91,153,77,399,165 }, +{ 14,2,7,58,46,1,38,16,75,5,70,52,24,155,29,45,220,35,163,69,50,171,61,186,87,227,18,270,92,48,666,54,32,290,140,21,278,592,513,307,67,112,262,6,66,60,37,0,135,243,97,767,64,425,23,89,71,72,409,400,128,845,113,17,334,36,22,172,20,10,315,115,76,141,9,28,51,12,663,103,86,74,77,207,13,65,177,90,263,83,267,202,95,137,43,269,151,294,189,27,25,431,19,49,428,430,68,40,223,536,852,106,738,324,226,655,474,780,96,502,496,217,114,108,318,230,102,148 }, +{ 447,411,664,363,647,478,936,621,354,271,98,781,759,223,777,1,360,18,691,668,48,932,167,156,132,121,212,195,591,446,724,520,118,586,146,21,23,463,31,293,9,264,393,16,51,914,13,125,807,101,49,834,284,44,60,163,682,24,848,221,658,541,5,105,35,71,68,55,96,39,116,562,407,32,905,604,22,252,772,0,33,765,522,69,10,963,67,191,93,100,638,676,166,340,364,120,281,698,608,286,821,404,50,115,382,144,77,265,697,64,28,102,909,126,410,30,628,82,662,95,367,7,815,127,744,335,40,296 }, +{ 14,2,1,46,67,65,38,7,69,36,29,24,83,28,50,0,22,103,5,6,52,124,108,484,168,12,114,86,75,133,208,18,16,194,236,283,289,389,21,610,10,109,756,160,25,267,35,23,51,95,32,309,55,505,135,584,71,13,185,66,547,128,189,37,732,54,96,48,169,646,17,49,412,9,420,158,87,19,323,115,278,244,26,64,218,164,70,143,253,74,238,140,220,398,172,45,137,58,149,262,797,141,92,308,4,3,503,302,202,60,68,388,154,159,118,650,531,90,516,366,150,285,105,532,335,251,671,429 }, +{ 386,421,51,791,801,697,958,730,453,23,165,551,221,670,615,326,401,391,576,403,13,102,457,899,805,523,842,770,48,24,450,475,0,203,125,949,541,719,566,77,193,115,308,197,21,18,398,153,33,393,317,117,43,202,137,743,120,300,7,124,467,5,91,108,9,71,342,291,118,42,32,16,97,134,85,279,61,260,173,289,281,517,494,14,859,817,196,60,809,758,692,36,882,25,194,388,20,373,302,140,65,547,429,273,851,717,458,412,122,104,95,509,233,81,492,318,6,362,832,110,69,92,49,149,35,508,298,608 }, +{ 319,104,945,422,81,20,715,43,0,879,693,681,306,32,784,74,173,207,61,7,22,52,397,671,414,904,887,636,240,95,6,115,835,192,5,707,91,426,18,2,76,29,819,330,316,263,56,85,138,84,28,232,577,365,64,97,113,519,312,725,526,17,344,370,204,49,202,214,329,24,38,560,403,750,66,494,10,804,500,241,25,105,289,718,159,587,523,77,124,102,86,861,348,498,450,292,51,288,90,117,140,65,899,510,168,347,158,16,448,45,817,237,107,324,33,9,194,180,120,318,92,790,40,280,21,356,320,874 }, +{ 2,29,70,1,52,262,75,112,26,145,46,163,309,58,103,67,14,50,6,220,96,74,38,394,223,331,357,19,83,238,45,69,5,324,171,287,27,409,151,25,366,307,24,583,218,537,133,108,632,42,267,396,451,53,36,433,87,114,65,17,113,635,186,764,550,545,226,600,138,675,388,66,37,21,61,54,48,181,122,516,12,606,294,236,418,73,35,189,18,80,86,486,487,651,135,155,150,7,32,16,137,644,243,739,281,222,445,334,278,380,573,149,849,106,227,582,603,160,335,34,64,283,696,648,765,624,297,72 }, +{ 771,929,786,941,590,626,687,259,465,178,70,878,132,5,13,21,534,146,49,518,279,48,107,35,379,493,31,147,125,767,105,108,87,195,221,481,126,121,44,163,22,623,23,127,61,581,407,362,0,150,165,937,427,58,846,345,371,142,883,467,167,761,120,33,10,640,98,75,591,360,391,1,663,271,77,30,199,4,845,763,522,60,185,102,291,7,349,159,156,137,128,246,450,51,308,296,293,239,32,11,158,423,558,557,342,269,116,39,260,112,17,386,232,502,441,281,196,153,95,37,12,783,953,612,535,401,197,15 }, +{ 5,186,45,2,61,29,50,227,52,400,58,17,46,48,171,209,14,155,106,243,80,311,38,1,430,281,496,54,502,536,255,181,454,513,103,96,592,760,3,334,207,21,666,67,119,552,587,869,663,62,151,541,27,32,583,309,721,84,56,37,659,218,133,366,221,136,83,540,6,834,72,36,223,270,667,290,738,12,328,34,807,89,287,263,474,19,487,262,236,238,148,76,75,115,69,425,240,378,99,26,394,24,766,433,483,128,114,306,70,549,65,324,600,675,828,357,764,53,780,108,545,73,137,161,7,833,90,765 }, +{ 0,95,453,64,32,389,494,817,342,207,312,365,577,690,194,403,848,237,202,316,180,671,5,482,757,61,490,805,102,289,498,450,65,835,475,523,113,49,401,241,887,115,120,263,329,21,17,18,576,22,33,137,344,13,318,402,77,3,1,899,11,48,347,4,210,141,118,125,90,51,933,942,260,23,645,874,306,308,117,134,597,615,27,247,12,196,24,122,19,6,2,751,697,743,317,468,291,25,16,288,177,730,629,28,86,84,349,770,421,214,611,592,491,483,476,425,56,36,670,300,275,153,217,621,350,39,71,60 }, +{ 23,13,51,202,115,15,5,1,12,4,2,77,403,165,0,21,10,318,453,32,457,33,128,197,9,7,6,141,3,401,475,59,177,269,14,11,308,17,37,494,22,719,217,99,64,341,180,117,18,90,40,102,120,16,317,730,961,569,48,365,28,421,386,352,629,8,38,523,450,237,31,291,402,125,210,137,247,216,19,391,196,57,221,36,509,326,260,72,93,24,349,275,153,203,49,95,498,342,304,899,45,134,82,44,76,30,468,372,98,89,27,817,25,139,86,110,758,65,54,29,111,546,351,116,241,46,20,482 }, +{ 119,292,6,45,306,76,2,26,240,209,17,61,710,106,5,80,29,19,27,25,214,84,113,32,64,618,129,255,128,1,179,90,123,4,59,65,151,14,58,667,40,460,311,56,504,22,245,86,249,207,0,790,141,205,42,95,77,483,3,833,10,112,115,202,7,23,18,887,137,194,11,52,57,36,62,136,269,476,177,162,99,33,491,21,257,73,769,454,12,70,38,28,402,13,24,145,16,48,72,318,184,196,614,445,122,102,53,114,49,117,153,210,548,285,46,54,464,134,180,51,396,50,297,183,315,243,237,814 }, +{ 15,1,23,13,51,180,0,2,120,5,7,9,450,523,33,12,115,260,197,102,165,6,202,77,4,21,141,14,457,351,403,341,16,901,8,326,82,308,365,3,269,18,36,137,93,318,48,128,386,59,31,37,719,217,453,401,730,817,22,110,44,475,64,317,352,10,32,28,421,494,40,30,116,98,11,899,342,373,752,196,24,615,221,177,86,291,49,854,203,39,99,509,961,96,144,569,898,55,922,35,391,210,117,244,629,125,349,498,873,17,50,247,372,89,160,758,237,670,300,95,127,143,153,65,285,402,90,801 }, +{ 6,138,222,280,74,85,66,226,337,87,42,204,235,25,145,294,64,135,95,174,609,29,112,415,256,26,65,525,792,461,353,32,580,22,52,122,746,172,630,544,289,70,348,416,237,287,19,278,49,312,86,605,75,10,108,2,0,194,192,5,620,1,7,21,90,97,736,220,63,45,632,500,114,214,61,355,36,151,233,396,180,58,69,24,125,141,72,28,14,648,27,519,673,889,872,624,241,444,288,18,343,197,48,426,33,207,133,665,263,232,113,73,347,91,17,568,448,316,103,208,165,128,104,53,380,12,81,498 }, +{ 39,9,333,101,0,158,252,310,254,272,455,68,49,627,327,344,159,286,419,364,18,16,709,329,24,688,105,265,109,30,100,553,118,374,35,557,166,124,190,154,677,131,65,578,912,71,793,22,60,518,1,538,51,167,21,23,98,28,212,48,95,438,604,625,13,404,7,31,96,55,127,520,36,731,137,890,857,125,320,616,694,656,806,716,44,577,5,10,170,224,67,83,829,168,242,219,228,128,191,14,239,827,88,86,301,347,12,424,160,747,810,37,115,64,846,346,371,613,69,910,221,33,149,2,562,93,865,676 }, +{ 21,32,14,5,38,76,72,2,48,54,270,45,1,17,50,46,221,290,181,89,36,430,3,741,428,218,408,155,37,540,281,449,0,10,12,148,179,90,241,263,207,29,292,133,425,22,477,328,99,51,697,421,114,67,770,487,115,476,23,4,528,227,65,28,6,119,243,488,214,720,552,541,86,474,24,52,350,64,61,7,9,137,11,186,103,536,401,129,745,413,106,58,83,96,920,308,236,13,57,513,123,18,842,464,543,361,245,160,56,59,205,128,655,431,514,358,828,285,171,108,460,55,62,400,27,16,453,40 }, +{ 137,269,141,202,120,450,260,318,494,922,351,291,82,77,5,326,678,352,704,854,128,523,1,817,21,15,64,349,180,342,32,13,475,90,177,947,7,372,61,51,49,23,165,95,403,11,207,115,312,102,4,48,197,237,402,719,203,50,96,308,30,6,2,373,205,86,70,69,67,36,210,241,105,898,859,851,615,208,194,217,901,285,25,453,216,182,163,87,65,46,45,8,0,457,31,307,139,112,58,37,12,3,117,968,158,127,20,730,386,273,199,178,168,147,126,108,60,28,18,16,10,9,954,795,597,275,247,134 }, +{ 16,68,71,24,149,35,191,118,167,212,18,124,101,246,108,399,189,154,140,92,267,9,60,375,65,458,695,302,133,67,398,589,7,0,69,230,114,562,702,1,705,103,55,915,755,283,21,388,602,336,412,48,14,175,273,650,50,345,823,674,411,46,121,146,28,363,236,447,39,10,64,672,22,664,478,796,264,309,330,271,100,520,647,37,95,218,199,12,906,23,565,36,83,32,252,777,88,166,628,126,5,13,125,49,75,2,366,51,96,86,732,486,346,54,815,237,733,429,98,891,759,516,633,818,115,443,723,238 }, +{ 28,109,0,9,67,22,1,55,65,114,103,158,39,49,83,194,168,69,35,420,185,289,36,50,208,24,68,133,2,124,505,251,218,175,86,46,7,389,323,154,95,169,159,346,12,547,160,14,756,101,646,16,570,105,30,5,799,189,533,38,531,96,503,238,596,252,128,64,535,23,54,18,21,344,51,782,272,336,610,419,283,532,345,333,480,13,100,10,37,137,327,115,789,584,149,286,230,361,48,75,285,181,202,29,254,374,164,429,927,141,71,896,143,671,930,577,32,237,140,810,244,484,439,31,131,601,402,310 }, +{ 325,147,427,198,178,142,798,202,259,5,376,20,534,43,465,450,904,318,234,470,858,77,725,455,120,581,329,315,70,97,61,929,105,878,786,113,362,260,58,557,481,403,660,14,639,21,173,0,170,49,159,494,319,269,48,23,344,523,128,107,301,595,1,845,518,416,33,884,817,324,320,13,326,199,22,475,402,158,37,270,132,51,40,38,35,32,7,95,590,121,4,422,239,141,104,66,146,347,112,615,263,230,126,75,28,307,76,207,704,414,360,265,195,9,922,453,356,298,19,17,137,72,342,290,102,87,50,395 }, +{ 28,0,65,109,9,35,55,175,124,67,289,24,39,158,68,114,14,336,570,154,420,1,22,7,185,168,69,133,194,49,16,429,149,101,46,505,83,189,2,103,169,346,50,252,484,36,547,389,610,283,18,756,95,419,159,535,345,251,86,71,105,12,108,398,100,230,30,140,439,333,533,344,584,703,21,208,272,286,671,38,5,327,267,10,64,601,23,254,51,96,118,810,13,48,164,782,265,128,364,236,29,143,160,837,329,374,677,54,577,218,310,637,399,60,37,962,797,92,166,323,793,705,32,238,732,75,6,596 }, +{ 141,180,115,269,13,23,4,1,352,5,202,318,51,317,2,217,21,752,365,77,3,7,165,341,197,32,0,137,11,308,10,6,901,33,15,9,31,22,177,102,403,64,401,12,30,44,498,237,291,48,143,244,93,349,569,40,494,28,8,39,253,125,372,90,221,98,453,304,386,482,873,509,203,120,110,116,18,59,49,457,144,128,95,247,391,864,326,117,16,421,37,153,14,196,475,36,629,134,105,210,260,961,373,160,164,55,719,546,437,216,734,35,468,350,201,17,139,96,89,579,473,72,758,717,111,24,743,275 }, +{ 15,0,1,2,9,23,13,5,7,51,12,6,4,8,901,33,3,180,14,197,10,341,115,77,102,93,120,141,260,22,21,82,457,899,31,64,453,28,165,16,450,18,523,11,351,475,202,36,110,37,203,401,125,30,352,898,44,403,961,730,719,116,128,569,421,365,196,49,40,98,32,217,137,373,269,39,308,237,144,55,854,153,177,318,317,86,326,24,17,117,95,629,48,817,752,59,300,670,498,99,391,494,386,291,134,38,105,90,642,342,922,873,242,143,244,253,393,473,221,50,247,372,57,349,139,65,111,661 }, +{ 6,74,138,66,75,85,25,324,42,380,278,135,26,500,87,226,1,220,97,29,70,222,889,841,52,350,19,606,280,235,294,394,112,571,69,853,21,38,145,335,903,267,582,65,550,309,14,172,792,509,32,2,197,64,749,262,237,174,241,357,122,95,461,486,808,67,337,256,537,96,103,115,774,546,204,63,125,27,391,22,558,462,83,451,7,238,630,5,308,802,24,165,108,734,46,163,140,950,648,641,92,45,13,298,189,416,583,525,150,72,923,86,600,48,114,53,331,221,388,12,620,180,468,426,90,645,746,348 }, +{ 51,23,13,453,12,115,403,4,457,401,165,475,15,0,1,10,11,37,59,5,99,40,3,308,64,197,32,2,33,202,17,9,719,77,629,730,7,14,22,6,961,21,569,523,494,421,386,18,120,141,57,125,177,318,269,102,237,128,90,38,899,341,45,450,111,76,16,180,217,19,36,31,134,117,247,49,670,139,72,28,210,95,291,196,402,153,41,25,24,89,509,216,260,8,93,86,54,221,82,137,65,468,758,26,56,44,27,317,391,352,817,365,304,62,349,123,498,48,34,326,275,152,775,342,114,372,300,757 }, +{ 788,180,319,21,5,83,32,498,314,547,526,17,693,61,817,48,11,4,27,510,33,12,848,77,288,104,24,160,0,814,120,64,3,804,153,134,117,56,519,342,196,102,207,725,326,690,523,450,69,40,403,95,899,306,60,958,90,961,942,801,500,232,174,155,84,43,38,115,945,457,386,595,475,615,247,887,715,671,494,482,365,192,13,312,202,165,10,9,81,51,566,65,71,119,113,861,835,802,770,757,756,746,743,729,692,683,629,587,583,551,550,525,449,426,422,418,414,324,291,260,221,217,214,186,170,125,123,105 }, +{ 15,1,2,13,23,4,5,3,0,11,7,6,341,51,10,9,12,33,77,115,102,40,217,14,197,180,165,141,365,901,569,317,202,64,22,21,8,16,59,32,403,31,17,269,177,318,475,18,196,386,453,352,153,93,401,120,125,117,28,291,457,494,210,110,37,961,629,326,128,391,237,260,139,49,44,349,98,372,116,719,342,36,523,95,57,86,111,137,19,24,203,450,308,752,30,275,38,730,134,90,509,304,373,247,300,144,48,498,27,45,20,758,482,65,143,72,105,55,99,35,670,76,873,402,615,82,29,817 }, +{ 101,520,18,167,9,118,212,604,364,16,60,694,252,0,264,917,24,68,146,55,363,562,35,191,121,166,265,806,329,71,676,100,706,647,411,271,39,907,628,621,925,21,419,848,404,28,682,942,48,1,815,154,777,446,109,132,124,5,821,463,49,857,158,13,199,905,967,23,932,126,88,108,22,346,96,382,354,759,51,575,125,105,98,92,32,522,65,286,333,31,447,812,639,7,553,664,566,727,159,593,30,10,149,150,865,156,375,14,897,936,407,95,638,274,478,175,439,272,67,221,634,302,50,2,12,107,144,44 }, +{ 13,23,51,115,33,217,77,141,64,0,4,102,202,9,2,317,7,269,180,21,291,352,32,16,10,128,5,11,318,6,177,40,341,3,18,22,165,197,137,15,14,153,247,237,125,28,24,12,59,196,365,31,349,117,90,752,49,403,372,98,221,93,308,17,95,44,86,342,48,494,1,304,210,498,30,36,120,475,453,134,401,241,244,386,143,275,19,391,901,72,110,35,39,203,144,326,116,260,164,20,437,457,37,402,509,421,8,482,55,523,105,82,96,316,65,350,450,864,139,216,25,76,285,29,27,579,546,127 }, +{ 66,135,172,74,97,204,278,6,138,324,7,174,87,192,85,448,140,353,29,232,280,620,500,52,802,92,25,226,808,38,568,380,889,42,298,69,426,580,792,314,571,330,2,872,348,95,26,572,622,43,563,32,526,222,893,22,14,65,5,21,81,64,673,24,75,267,950,1,355,885,220,70,357,726,145,104,86,63,888,91,387,189,16,46,289,294,126,71,501,20,107,233,60,19,48,462,335,67,173,28,45,194,49,241,550,235,230,37,90,0,312,83,18,415,61,58,36,262,12,35,112,633,10,72,150,712,246,343 }, +{ 97,24,2,69,66,423,298,6,29,7,267,172,74,52,14,189,67,38,25,607,150,149,83,71,1,36,18,135,733,467,81,0,140,501,426,517,21,33,398,278,160,118,124,809,42,154,26,19,462,85,273,92,20,46,429,302,117,60,336,568,508,204,28,192,715,232,770,705,165,65,16,153,596,173,102,279,50,91,87,175,43,23,448,45,12,861,386,37,13,63,421,104,48,51,324,388,314,197,134,711,138,32,531,526,330,681,712,443,22,451,75,608,185,4,949,697,221,882,636,236,370,196,54,420,5,309,885,458 }, +{ 158,9,0,68,39,333,109,344,49,101,24,16,35,159,124,154,254,272,310,793,252,327,419,65,627,286,71,22,364,455,18,374,105,577,747,28,30,168,912,118,677,55,95,185,709,100,688,149,60,175,671,131,190,346,166,265,518,578,538,329,169,703,424,1,212,21,7,694,962,289,706,439,347,191,108,51,810,48,10,634,23,67,14,616,239,553,170,302,557,98,320,125,31,2,570,857,625,13,5,36,12,128,375,86,137,127,114,37,716,88,44,913,420,92,83,69,96,50,64,224,638,562,242,301,167,890,731,160 }, +{ 15,10,33,77,197,4,180,341,365,115,13,102,23,317,141,117,217,5,1,901,11,40,253,165,134,153,21,752,244,143,32,36,352,51,638,137,2,196,12,55,3,873,569,951,372,82,37,120,291,177,7,59,898,349,6,64,0,509,31,342,269,22,351,498,260,308,160,93,14,139,854,44,202,28,86,48,457,17,482,90,401,450,111,719,72,203,116,125,110,523,128,961,164,57,304,922,9,386,318,38,730,326,49,762,535,275,391,277,629,210,242,373,99,98,148,96,734,817,95,89,237,717,247,30,479,45,758,54 }, +{ 100,382,166,478,265,264,404,274,593,438,88,39,921,724,98,639,682,1,363,411,447,435,905,812,821,9,96,30,897,31,656,23,223,252,254,310,664,410,116,190,51,144,329,691,21,520,393,44,127,101,0,909,48,125,13,625,333,284,557,521,418,286,367,960,772,5,93,77,321,118,105,83,948,616,272,530,33,293,203,795,156,765,676,68,466,102,67,551,131,604,120,242,49,35,16,354,848,421,18,32,553,730,69,158,335,165,759,791,110,731,914,163,613,115,10,386,217,24,36,778,834,65,395,4,578,238,842,364 }, +{ 19,283,436,297,53,26,813,122,42,432,503,27,605,25,736,73,390,235,389,128,630,123,250,529,63,256,80,65,108,445,690,573,269,789,106,444,216,113,381,412,6,14,133,680,200,78,318,650,177,678,17,194,57,114,619,368,4,349,84,472,59,112,85,275,291,56,18,378,714,40,32,396,183,102,11,152,33,416,136,495,341,236,46,403,130,161,90,243,561,255,184,24,129,597,0,742,86,45,469,76,145,862,401,285,249,141,947,665,13,179,51,524,287,50,527,803,233,117,684,12,304,148,205,29,719,23,139,61 }, +{ 202,128,494,141,33,269,137,36,51,0,13,102,180,6,2,1,196,65,4,86,402,95,29,475,403,117,59,40,457,77,450,15,210,177,134,49,22,21,18,291,153,12,3,260,114,90,817,386,318,120,25,16,32,23,11,961,922,576,237,217,194,165,115,14,7,453,342,216,873,373,197,164,64,719,874,854,789,757,751,704,597,558,391,372,326,304,301,300,285,275,208,203,159,133,110,105,99,93,81,74,70,66,52,48,47,42,41,34,28,26,24,10,5,139,76,958,899,801,730,615,551,160,963,901,752,657,584,421 }, +{ 18,60,16,68,167,191,101,118,212,121,35,126,199,146,647,520,55,9,628,806,621,71,24,107,575,346,363,132,446,463,364,0,566,92,562,28,271,411,252,1,727,21,815,48,407,150,604,598,694,279,932,154,608,49,589,493,439,22,5,7,124,109,706,419,10,108,13,354,14,23,640,467,375,96,273,158,848,302,955,535,175,195,39,818,534,105,125,100,221,522,676,264,777,12,51,50,159,329,32,265,65,259,362,917,98,20,826,783,149,37,31,581,95,169,43,67,97,46,223,147,907,166,590,64,246,36,2,967 }, +{ 352,230,531,160,36,748,854,217,93,669,728,137,251,689,871,699,373,228,574,258,202,752,351,717,559,120,144,268,318,260,219,203,269,201,831,533,839,567,539,187,840,213,816,489,679,82,1,317,851,332,44,141,55,295,855,206,30,617,863,418,83,345,926,96,180,443,800,473,116,10,143,326,110,35,406,564,131,224,22,860,276,64,450,28,98,9,21,190,957,466,822,968,244,164,31,434,856,169,384,661,69,898,0,365,730,824,284,125,442,253,128,372,873,498,88,530,922,776,261,165,494,48,237,86,68,940,39,927 }, +{ 15,13,51,23,0,77,8,4,5,1,202,33,10,11,165,180,82,22,12,110,115,9,102,197,40,59,342,403,141,120,351,326,7,453,373,31,386,18,93,457,260,49,2,401,318,475,128,28,901,64,269,291,44,494,36,854,719,21,177,14,30,523,961,16,898,57,3,349,421,125,922,99,341,32,730,98,137,203,56,217,317,450,117,615,76,391,498,275,24,352,569,365,95,6,105,308,144,37,196,899,86,65,153,39,535,90,210,629,116,704,247,83,208,237,48,661,752,638,509,17,758,402,133,551,143,139,817,244 }, +{ 101,9,100,0,88,217,252,68,39,35,352,562,175,30,115,230,752,213,265,419,13,317,197,166,180,154,237,83,212,158,124,64,321,777,382,208,1,336,410,65,194,759,329,694,36,141,826,289,23,365,724,341,429,128,509,69,165,570,345,676,909,96,346,756,553,21,593,372,854,702,498,55,799,264,399,67,86,190,28,468,589,768,51,364,308,144,677,755,689,323,125,722,482,610,547,391,333,137,789,812,189,546,49,32,952,310,247,274,131,72,16,344,98,10,671,351,435,386,251,478,285,244,901,167,160,484,706,164 }, +{ 32,64,95,147,288,21,90,5,519,470,49,61,0,356,22,48,115,362,97,247,312,141,835,387,560,57,56,4,182,128,20,237,178,59,18,11,31,13,107,214,202,28,298,217,44,76,316,7,65,43,259,180,819,874,51,40,23,216,173,330,207,197,150,25,2,86,729,324,120,376,279,140,612,392,595,104,72,66,29,6,352,37,12,210,678,414,172,683,779,490,482,317,291,165,134,102,81,77,58,45,33,26,195,864,795,304,858,87,54,660,427,707,878,465,357,501,232,753,205,177,170,126,117,91,19,597,230,212 }, +{ 24,16,14,35,7,18,69,1,50,46,75,5,2,58,163,67,54,70,0,71,28,502,60,48,38,12,9,21,189,103,155,140,267,223,281,108,37,171,430,92,394,65,68,22,230,513,83,10,23,181,32,55,309,87,45,36,236,51,335,29,220,149,307,99,61,133,112,141,64,13,780,52,6,115,128,227,270,72,89,852,537,328,17,150,474,49,807,114,767,221,552,186,96,290,95,20,246,366,77,443,400,137,202,86,269,486,666,388,237,540,25,278,477,217,4,536,218,765,177,19,125,180,165,126,113,318,251,39 }, +{ 195,18,354,265,252,411,39,9,363,167,101,682,271,132,16,121,60,1,333,203,21,593,35,146,520,156,364,447,639,360,212,5,24,327,23,627,668,310,404,13,48,664,286,100,538,0,777,676,77,98,821,7,68,33,692,446,865,93,158,576,159,116,115,647,966,883,551,407,272,107,51,687,120,591,463,326,137,71,102,857,31,781,421,37,586,30,44,329,379,50,125,374,166,105,92,96,621,709,142,65,36,127,562,475,109,403,14,126,118,70,346,223,479,199,805,450,190,43,20,191,936,905,656,128,254,12,2,202 }, +{ 4,115,13,5,23,141,51,180,2,202,1,217,11,269,318,3,77,341,7,6,317,33,365,0,21,12,352,197,165,9,15,32,102,10,137,40,177,308,752,64,291,31,22,901,349,372,28,8,59,403,90,494,30,237,498,44,120,37,401,125,48,39,14,93,17,244,49,457,569,509,128,153,16,386,98,143,304,36,873,475,196,391,117,482,342,253,210,203,110,453,116,247,373,221,326,24,216,18,144,72,421,260,95,55,134,450,961,576,86,139,719,105,468,35,164,82,57,127,160,89,864,678,523,351,88,275,67,546 }, +{ 7,71,24,14,16,35,140,92,189,149,150,108,60,68,18,230,97,69,66,314,273,267,458,388,336,38,2,65,232,251,175,399,118,46,398,172,67,28,462,246,36,55,375,0,705,1,21,124,154,22,298,302,135,95,451,733,877,126,594,423,345,64,467,101,9,74,517,29,48,86,602,236,6,20,191,83,596,52,330,380,32,357,43,5,10,674,50,12,133,49,429,723,103,711,212,346,104,192,114,107,915,278,167,387,37,443,324,309,121,823,54,125,96,633,566,221,796,484,75,109,547,603,13,23,289,809,575,607 }, +{ 1,22,31,5,36,12,105,44,2,86,21,170,98,180,127,0,55,141,64,23,7,352,28,13,9,128,116,125,164,239,4,301,49,115,6,95,197,37,3,160,242,51,341,752,202,83,498,269,10,96,165,144,48,253,244,285,901,217,320,365,208,569,137,8,338,299,318,395,293,109,654,93,512,143,32,50,237,317,159,402,864,11,15,708,776,873,169,201,14,35,99,67,638,39,954,30,844,531,479,678,77,69,456,740,533,16,403,308,268,120,751,951,33,961,947,457,286,238,401,276,482,203,601,17,506,312,509,18 }, +{ 97,66,192,7,560,52,20,2,135,74,43,173,324,104,580,414,226,107,147,750,204,29,174,500,376,568,825,707,222,0,356,470,77,32,331,194,90,38,729,16,819,298,92,172,355,22,60,64,888,595,357,232,24,18,259,712,202,353,426,81,5,362,262,874,427,526,49,330,14,178,95,885,387,21,10,138,37,861,142,6,33,102,86,28,889,4,359,65,312,36,858,126,59,1,112,792,673,337,58,151,177,91,35,113,319,291,9,115,145,13,636,501,342,448,278,140,12,87,61,71,583,294,751,465,237,246,230,325 }, +{ 101,0,68,9,252,154,124,364,166,35,39,100,24,65,419,16,149,694,71,265,329,28,439,158,346,88,30,289,212,404,55,671,703,562,570,49,274,175,429,21,706,589,18,553,118,375,1,722,67,302,22,109,962,190,677,382,333,108,60,23,967,191,168,95,69,48,344,14,51,336,114,254,676,610,484,13,398,10,83,92,7,520,310,857,818,125,264,593,189,64,133,185,826,438,12,32,159,865,221,103,535,50,194,695,286,105,36,604,96,169,420,230,283,701,167,241,131,140,2,350,656,115,272,5,239,637,812,98 }, +{ 39,100,127,30,737,836,9,254,856,31,166,190,98,395,0,385,88,931,625,131,373,264,863,93,827,28,228,120,44,125,1,219,116,137,110,382,371,55,13,101,731,299,260,144,908,442,21,77,265,23,456,64,203,51,599,242,33,187,49,326,310,22,224,578,202,5,450,36,272,18,276,16,141,795,82,105,916,284,165,274,318,48,95,512,944,35,716,478,24,935,258,201,616,851,32,557,102,96,252,109,68,269,128,351,435,492,83,724,86,67,7,10,704,293,772,59,921,329,846,156,924,286,197,438,717,4,291,244 }, +{ 17,5,45,21,14,2,48,3,32,155,106,227,186,181,38,29,61,209,62,46,328,133,400,80,243,54,56,52,1,51,832,23,19,50,99,255,114,430,6,84,543,536,12,514,401,236,27,37,137,361,552,453,119,136,34,311,58,218,171,221,76,72,4,754,115,281,108,13,285,496,65,454,629,513,41,207,308,540,421,26,290,805,480,541,488,920,657,202,475,11,487,787,128,86,89,569,57,739,151,838,502,36,270,59,403,208,90,10,7,721,474,22,263,745,576,0,457,659,592,663,587,25,161,129,165,148,323,894 }, +{ 131,613,224,856,228,406,716,274,827,30,100,219,190,39,166,863,88,931,924,829,944,31,382,44,625,9,127,1,442,385,187,98,116,731,93,578,258,254,530,816,137,871,160,616,960,96,36,310,265,0,371,203,144,836,410,438,264,435,795,125,242,21,202,120,201,531,276,908,51,921,128,23,101,49,839,846,28,737,48,326,5,83,778,13,110,64,284,105,35,492,272,10,557,33,938,115,299,77,252,158,456,367,268,669,217,260,165,213,32,177,22,800,352,473,86,102,69,197,574,321,318,567,141,935,593,403,55,180 }, +{ 352,217,752,141,954,864,64,372,437,269,180,115,498,177,349,776,579,291,317,247,77,947,873,717,304,318,678,237,179,23,197,128,33,0,35,165,13,275,68,210,457,341,459,1,67,410,32,143,365,65,244,202,102,9,160,140,137,225,569,12,36,18,125,75,51,134,93,751,15,546,509,86,708,482,928,713,330,312,201,164,144,100,10,7,521,268,246,44,24,21,16,558,55,851,682,342,260,245,120,117,101,98,39,28,726,363,116,110,30,189,72,478,83,744,953,841,824,672,563,358,166,114,92,31,5,386,196,69 }, +{ 141,217,317,180,352,365,115,341,5,372,23,13,752,498,253,244,15,1,237,2,51,197,269,0,7,165,21,202,64,349,77,6,137,143,4,318,3,9,901,546,33,177,308,11,32,10,160,12,48,49,482,30,22,125,468,98,373,247,120,102,93,31,864,110,509,260,531,391,28,44,494,437,36,450,569,90,40,490,453,326,221,219,203,16,128,96,82,386,164,72,873,291,251,224,187,117,403,401,210,39,131,116,24,18,473,144,86,37,629,148,105,8,533,951,758,475,421,228,139,170,134,95,65,35,734,704,617,275 }, +{ 5,2,45,17,14,29,48,1,21,58,181,38,50,61,3,133,46,155,171,186,114,52,32,27,281,19,6,218,36,236,227,54,361,151,51,12,754,209,26,400,80,137,514,243,65,106,552,487,13,480,334,23,285,86,412,328,108,4,283,502,311,37,56,99,323,22,745,657,208,202,25,62,72,128,115,838,760,59,430,7,34,536,64,70,454,11,76,10,28,255,119,803,194,787,67,421,0,576,57,859,496,103,84,96,18,532,75,165,89,453,389,223,24,920,401,163,73,113,308,828,402,77,739,112,287,148,457,513 }, +{ 6,25,2,119,790,29,113,76,26,45,483,655,887,42,292,32,19,4,59,128,151,207,1,306,84,17,814,106,240,90,112,209,61,145,14,177,77,64,65,52,833,27,122,179,491,210,70,0,72,311,115,40,255,58,614,74,85,7,214,202,710,22,5,57,469,21,141,11,123,80,476,95,36,396,33,10,24,269,38,137,102,66,425,23,416,49,194,86,275,504,13,216,56,402,318,46,358,18,138,28,460,624,9,148,50,174,129,618,51,205,87,75,16,291,186,12,217,249,222,287,454,20,54,99,243,315,117,114 }, +{ 23,13,51,115,2,4,5,0,141,180,7,11,217,3,33,77,6,1,202,9,317,10,102,197,21,269,40,352,341,32,12,165,16,22,64,318,365,28,128,15,14,18,752,137,59,24,291,308,177,125,31,17,403,498,237,494,372,98,49,117,901,93,196,30,44,153,457,349,8,37,120,134,90,116,247,342,36,482,401,95,35,210,143,221,86,453,260,203,144,475,326,48,244,110,391,386,55,304,873,39,19,569,961,421,105,509,719,546,450,253,864,216,20,402,29,523,72,57,50,139,25,730,373,468,275,65,82,629 }, +{ 23,13,51,202,21,141,115,5,165,318,32,12,403,197,453,1,269,120,15,128,2,180,475,125,308,457,401,7,48,6,386,64,0,730,4,9,719,137,3,10,523,18,33,14,37,421,16,99,494,11,260,450,341,221,17,196,93,77,40,217,22,117,90,95,237,899,31,391,247,352,326,28,24,102,19,153,30,317,98,961,817,134,82,44,25,59,8,241,110,49,365,20,26,291,27,61,177,36,351,498,509,569,402,39,203,758,29,72,57,45,629,350,54,116,316,96,752,50,38,349,67,901,65,46,342,304,285,58 }, +{ 39,9,100,30,31,127,44,88,0,276,98,131,1,385,254,442,166,49,908,116,187,190,935,201,219,284,827,299,28,224,144,829,96,228,36,274,137,141,101,21,265,716,382,48,51,258,202,23,352,22,125,836,93,856,613,128,839,158,371,264,578,395,13,55,16,180,662,244,269,252,217,567,492,110,310,35,64,77,272,83,268,410,931,203,120,326,752,260,68,318,18,489,293,574,625,160,24,33,115,86,159,924,242,373,165,438,5,450,69,102,616,213,197,918,863,321,295,717,291,82,731,177,456,317,406,7,599,737 }, +{ 22,109,49,28,159,185,95,9,105,131,272,327,254,1,67,286,36,39,35,168,86,83,55,169,505,158,224,374,420,127,584,208,98,219,312,578,31,347,535,601,0,69,708,424,44,114,103,30,160,627,346,64,141,194,187,518,175,96,239,345,616,892,10,128,164,143,896,116,137,50,12,5,2,237,228,913,533,13,23,7,21,371,320,597,688,538,751,912,352,323,251,716,165,115,634,202,258,197,498,637,37,531,180,797,874,51,253,768,48,101,890,16,189,201,613,24,752,385,14,170,285,365,406,789,244,218,46,133 }, +{ 5,1,21,2,48,14,12,32,22,36,0,17,54,10,38,3,37,86,50,45,65,28,51,72,281,23,4,6,58,114,89,46,137,13,270,99,430,11,7,208,76,202,181,61,115,403,155,9,133,18,540,64,361,221,29,323,480,285,49,128,24,95,218,401,657,421,290,745,19,576,194,55,308,754,59,453,513,920,148,880,96,341,502,52,16,475,77,67,842,40,283,859,318,197,402,494,171,141,474,244,543,757,165,775,90,541,27,33,180,57,83,227,164,217,428,328,269,108,425,569,514,207,31,143,223,263,102,503 }, +{ 141,352,269,372,351,217,854,82,752,922,318,180,954,260,317,15,349,1,202,64,120,898,717,77,67,35,864,291,69,365,137,851,437,36,22,5,247,86,373,4,457,450,244,165,189,163,140,68,13,237,901,115,128,304,23,21,341,98,125,177,51,93,32,28,961,160,96,494,110,95,10,0,719,253,661,523,326,767,83,730,275,44,386,418,40,12,817,210,19,17,102,59,45,18,7,579,498,482,615,403,144,11,6,3,203,76,822,475,49,48,31,33,293,284,268,228,187,947,899,587,583,485,474,367,342,300,245,100 }, +{ 144,613,203,406,669,418,326,137,228,96,679,494,160,816,1,817,822,36,317,859,840,83,856,876,855,217,251,219,863,831,944,202,576,365,539,93,800,5,345,530,77,128,44,957,748,141,13,839,30,23,21,778,143,341,115,473,617,332,201,64,120,190,492,456,829,531,349,367,716,258,98,48,578,116,180,924,403,187,31,515,966,692,574,442,131,86,931,719,523,450,9,224,351,291,860,752,82,28,10,244,51,371,22,276,39,102,125,730,261,50,851,269,827,335,318,871,704,916,268,110,898,69,67,498,206,177,615,805 }, +{ 15,13,23,51,202,1,21,33,5,269,180,217,2,141,308,12,120,0,352,165,7,221,115,117,48,32,6,318,197,4,153,9,260,82,196,453,134,457,3,40,386,523,730,110,14,719,77,401,31,403,450,137,326,37,102,341,16,351,8,128,752,11,475,93,18,498,317,44,365,391,64,30,98,237,72,10,49,421,22,99,901,494,509,28,24,899,17,125,39,817,569,854,116,203,59,291,177,89,961,95,36,244,629,758,105,247,90,372,241,864,349,546,670,898,96,35,144,86,65,350,373,50,139,304,61,160,922,615 }, +{ 23,115,13,51,77,0,1,4,365,2,141,341,5,33,202,217,11,10,317,9,180,102,3,7,6,15,269,22,318,197,165,64,352,12,40,494,403,21,28,31,128,308,125,32,453,59,177,18,291,16,401,475,901,14,24,752,237,93,342,137,349,457,569,44,8,17,372,30,153,98,120,95,49,210,36,498,196,117,110,509,482,421,116,247,386,873,304,37,391,39,629,244,48,139,90,203,143,134,55,86,523,450,326,468,260,275,57,961,144,719,65,35,402,216,730,221,19,253,373,46,50,734,437,678,576,96,758,551 }, +{ 531,373,728,160,943,559,717,260,93,269,332,206,533,244,352,384,261,535,964,564,120,110,10,253,295,137,365,318,141,55,143,959,180,617,36,851,911,843,197,661,203,873,282,1,752,940,341,699,968,901,82,417,326,21,144,450,22,351,317,434,64,642,498,855,824,574,202,926,473,860,800,896,164,23,28,898,8,51,13,258,15,116,35,816,567,127,187,237,776,9,951,219,601,268,115,811,165,217,201,864,131,98,406,854,77,523,251,902,922,125,48,345,44,31,16,704,96,499,86,83,489,871,367,299,177,72,817,24 }, +{ 1,31,36,170,55,86,22,44,127,5,301,64,98,164,2,116,740,105,299,12,10,242,237,125,169,338,9,0,21,143,7,512,23,28,841,128,141,13,239,51,395,180,654,208,293,685,37,144,870,923,95,6,599,83,197,115,160,456,900,352,67,498,506,253,165,638,49,3,268,492,137,965,4,479,14,393,763,269,244,185,708,202,109,341,96,365,50,762,509,933,276,48,284,951,367,505,864,751,752,546,320,601,597,35,8,569,634,312,323,217,558,201,391,824,317,401,308,901,457,32,837,159,535,93,873,54,114,38 }, +{ 13,23,51,115,32,141,77,177,202,33,64,269,90,4,102,318,128,40,59,21,217,291,137,180,352,210,95,10,48,0,22,5,196,494,15,7,237,14,117,49,18,216,197,125,342,98,31,304,11,9,165,153,403,317,247,86,752,134,28,275,349,16,24,116,365,341,2,6,76,44,453,17,12,20,30,300,221,93,39,143,139,36,1,19,3,402,326,386,312,105,457,65,244,194,8,576,498,901,308,110,99,72,372,120,401,203,50,25,241,144,61,26,208,96,678,450,160,615,421,391,127,114,57,46,37,35,27,947 }, +{ 202,260,120,450,318,51,269,77,23,351,1,326,13,128,82,137,141,21,33,15,494,102,523,817,5,403,110,165,922,64,203,197,854,125,32,180,177,291,48,153,196,93,116,12,342,98,352,373,40,4,115,7,475,144,349,36,217,90,2,453,386,16,0,285,95,44,615,237,59,49,31,457,704,6,898,210,24,58,96,86,30,719,117,70,401,216,8,275,28,22,14,402,17,247,50,39,29,18,421,99,37,20,10,391,304,61,25,752,341,730,678,9,365,859,576,473,139,45,961,692,293,661,498,317,214,132,35,76 }, +{ 899,523,102,730,33,153,196,117,719,23,475,51,817,13,15,403,453,5,4,342,77,494,40,120,21,11,450,202,421,59,457,12,805,165,139,134,26,19,17,2,1,300,386,801,961,57,32,27,7,6,3,880,197,99,361,757,670,128,125,72,58,56,48,45,37,14,318,64,49,770,247,82,576,260,884,775,635,629,509,485,456,402,391,180,177,141,95,89,84,76,70,62,61,54,53,38,29,25,758,401,269,262,583,409,365,341,331,217,52,968,967,966,965,964,963,962,960,959,958,957,956,955,954,953,952,951,950,949 }, +{ 202,494,403,475,523,817,453,450,318,402,120,576,51,13,77,102,342,33,260,128,23,326,5,899,15,4,82,615,210,269,137,351,291,59,196,141,90,730,40,386,285,214,177,21,719,389,49,11,65,61,30,22,48,153,117,457,114,12,3,98,64,44,39,31,1,922,194,216,84,349,165,95,32,28,10,0,757,197,421,474,123,38,961,887,805,690,629,551,401,283,263,207,139,125,118,113,58,848,503,115,41,880,275,24,17,422,319,104,942,854,803,801,704,671,509,416,400,393,372,334,237,227,217,186,180,151,134,96 }, +{ 2,29,52,112,66,87,70,151,135,7,226,58,74,307,186,294,172,287,549,644,334,5,14,97,46,75,45,227,331,113,145,6,25,315,155,337,222,1,61,192,32,243,163,311,64,220,38,400,43,262,171,69,21,37,624,461,20,884,122,849,631,416,26,16,48,86,27,24,42,22,19,278,138,92,353,635,36,609,232,0,128,140,10,500,324,50,17,72,665,59,12,95,76,90,409,209,35,4,396,491,614,65,18,496,415,54,721,568,592,67,632,49,141,137,80,28,89,204,659,869,60,106,426,845,448,177,207,115 }, +{ 1,349,652,5,260,291,77,39,342,102,120,23,100,888,13,269,64,340,264,51,303,698,32,2,202,21,9,296,33,98,70,385,163,638,318,137,327,87,293,58,50,128,48,195,69,774,141,75,121,7,658,850,326,96,371,284,223,931,863,759,132,126,92,71,221,678,203,82,551,961,937,615,575,563,494,105,83,67,55,36,30,27,17,771,418,115,90,31,156,914,687,393,372,12,6,3,622,144,922,898,854,731,691,478,403,367,363,352,351,333,310,286,170,146,0,950,230,93,95,49,919,954,947,865,851,765,751,719 }, +{ 1,5,2,13,4,23,7,3,15,0,6,180,51,115,141,341,11,77,269,352,21,901,217,197,365,33,12,32,177,752,317,202,318,165,40,9,102,10,291,93,8,569,31,22,349,498,120,90,44,110,401,59,37,304,326,48,17,64,372,403,196,117,16,98,14,457,210,453,125,494,28,36,386,629,247,153,475,30,308,260,134,873,116,253,373,450,18,237,244,482,203,49,128,961,275,137,144,216,509,523,421,391,143,342,24,719,105,95,55,72,76,89,39,206,437,139,468,57,579,758,730,864,546,127,221,96,734,19 }, +{ 14,7,16,46,24,2,35,70,92,58,5,38,75,18,140,60,87,171,220,54,1,155,69,307,50,21,163,48,71,32,112,0,37,12,23,278,780,502,126,9,61,52,186,172,135,45,28,64,29,13,67,227,20,107,22,115,97,513,51,10,267,72,68,270,128,230,89,314,666,330,246,141,6,232,189,36,400,177,66,108,290,150,17,309,55,430,767,334,77,474,121,262,103,243,409,223,137,83,43,113,202,592,217,461,49,95,315,571,65,181,394,357,86,281,269,774,294,236,380,852,536,845,76,90,25,102,298,4 }, +{ 226,135,87,2,29,66,74,52,294,278,222,220,75,7,172,145,331,70,324,262,97,6,25,112,138,337,140,42,1,92,461,26,69,38,500,58,357,14,380,5,287,16,46,571,232,620,808,632,267,163,32,24,35,583,853,774,43,314,609,45,171,21,298,353,19,802,95,60,151,415,448,409,64,67,192,22,65,396,85,525,37,174,307,189,872,204,20,71,50,12,330,86,622,61,48,27,36,186,394,764,113,54,335,563,114,108,309,150,950,72,155,17,10,572,122,635,544,103,96,606,28,893,246,18,49,230,550,334 }, +{ 15,33,77,117,134,153,102,4,40,196,1,217,21,11,291,13,32,5,177,202,51,115,3,23,141,2,48,317,180,349,59,137,372,128,17,269,342,210,341,96,57,64,365,120,247,12,216,82,221,139,318,110,6,260,93,90,352,898,450,165,10,36,31,197,275,523,7,37,72,494,203,244,457,56,351,579,125,111,8,237,817,326,403,0,752,98,922,961,901,160,277,22,719,730,144,44,206,14,386,899,86,45,482,437,304,373,143,89,569,9,116,453,475,308,253,300,61,105,76,27,638,164,95,49,490,29,18,94 }, +{ 217,15,317,82,351,141,260,854,13,77,23,120,51,922,115,1,21,180,110,137,365,341,352,372,269,349,898,291,36,202,64,5,32,450,318,373,177,128,752,96,304,523,93,33,48,165,326,197,102,0,10,40,901,9,4,160,498,247,90,244,661,203,22,59,482,2,12,143,86,437,386,83,457,28,473,30,67,253,961,11,35,16,221,24,125,117,391,39,7,811,494,88,37,8,69,719,559,3,817,704,237,216,76,72,61,14,873,403,18,642,31,899,210,164,45,730,717,49,179,134,139,851,475,421,251,223,100,56 }, +{ 190,373,31,30,863,908,120,88,110,127,260,442,856,318,254,326,717,935,131,836,228,93,187,44,39,385,299,100,968,274,219,203,82,166,224,197,116,661,276,77,1,217,98,269,86,450,51,371,36,128,180,829,137,372,341,160,704,141,64,435,202,49,395,144,9,23,918,410,28,22,730,778,244,242,351,55,737,523,827,177,851,625,944,382,272,530,317,115,349,352,96,613,492,291,954,32,931,744,716,125,268,258,284,253,686,48,615,515,367,247,21,33,456,165,321,795,10,567,466,225,83,839,102,578,521,919,817,752 }, +{ 23,13,308,115,401,51,165,9,391,21,629,254,125,159,773,197,743,569,49,468,217,438,656,627,341,89,39,558,221,32,317,386,509,350,28,95,530,228,457,867,109,166,406,876,679,48,669,364,219,158,365,346,168,912,810,101,170,105,72,64,143,30,347,258,967,237,148,90,688,274,131,670,272,185,749,843,33,12,10,187,55,175,141,4,734,758,242,196,190,453,261,518,224,902,747,286,482,164,37,333,241,2,439,102,0,424,420,393,706,613,338,153,67,372,351,68,40,310,120,816,634,332,840,35,320,88,86,289 }, +{ 49,5,165,125,21,173,43,623,104,193,422,102,13,7,91,18,95,22,0,192,141,64,48,197,900,693,558,105,51,23,32,232,391,387,314,98,31,116,20,779,202,81,92,319,408,241,16,126,38,77,494,33,784,643,474,11,221,28,712,138,862,663,595,403,350,879,29,510,27,633,180,140,729,397,172,44,4,858,237,61,718,501,359,71,60,58,511,448,370,356,750,490,482,347,344,316,308,177,93,37,24,498,293,115,158,145,72,65,40,830,246,207,63,85,107,841,569,468,414,352,330,326,269,203,69,14,12,2 }, +{ 4,1,2,5,13,23,77,115,15,3,0,341,7,51,6,11,33,202,102,217,12,180,10,141,9,40,21,317,318,177,365,901,197,269,32,165,352,22,59,8,64,31,291,14,494,403,93,17,401,569,752,16,453,37,342,210,110,153,275,18,475,44,125,196,90,28,308,326,386,98,120,117,116,372,457,128,349,36,95,629,49,421,237,48,139,203,260,244,450,498,137,134,304,30,24,523,144,247,373,719,216,57,391,961,86,482,143,253,39,730,509,111,55,221,817,96,615,19,65,873,351,758,99,670,437,72,50,734 }, +{ 843,295,93,120,160,206,36,943,567,373,384,332,268,137,261,141,201,260,110,10,203,531,144,55,64,617,434,197,258,699,116,244,282,318,559,564,964,269,44,352,911,717,253,860,202,180,31,326,533,1,940,143,473,417,728,661,276,351,959,187,22,341,851,535,219,871,855,28,367,127,82,131,115,450,365,86,901,752,926,317,21,51,98,48,77,165,23,105,224,96,968,662,217,13,8,902,574,489,125,817,349,800,523,515,83,704,49,811,498,164,494,39,5,393,228,32,291,251,128,9,12,898,642,576,69,67,873,839 }, +{ 192,81,43,173,426,879,7,715,172,104,712,91,636,526,232,825,568,97,560,5,95,681,448,66,49,861,32,387,370,74,20,750,33,204,22,707,330,729,718,140,61,0,6,356,397,69,21,117,422,347,135,359,38,92,102,263,348,64,174,4,414,153,888,237,138,319,76,298,40,13,885,52,2,107,86,500,25,196,784,595,77,819,241,180,134,58,207,85,29,42,31,355,65,577,72,18,28,1,343,48,392,23,10,126,89,324,37,189,344,233,314,11,933,71,14,511,16,408,115,673,46,498,45,671,125,60,858,246 }, +{ 475,453,403,51,33,102,494,23,196,13,202,153,117,77,0,318,128,757,421,22,342,730,899,817,18,805,65,15,958,450,134,4,402,24,36,59,269,40,49,210,401,1,28,697,670,64,11,114,291,48,16,576,108,386,523,141,57,177,21,20,86,165,133,95,90,120,365,317,326,260,221,180,137,31,10,115,752,99,498,457,391,361,352,70,50,32,349,113,9,6,3,864,838,770,743,373,351,338,268,242,206,125,116,109,5,283,253,84,17,961,801,791,546,372,341,237,217,214,205,194,163,143,96,72,69,68,30,14 }, +{ 141,269,260,318,352,202,120,351,349,137,854,82,1,851,922,5,180,372,291,64,32,947,21,15,77,752,494,13,48,304,33,7,51,717,23,217,373,177,128,678,450,115,237,326,317,70,898,36,403,704,312,4,160,102,86,90,968,247,95,87,196,49,275,244,197,22,165,67,96,35,288,203,817,735,125,342,144,93,16,216,163,50,44,31,83,954,2,69,597,457,241,59,153,961,221,110,98,39,30,9,8,901,661,551,523,473,442,386,367,284,210,864,117,61,12,0,800,29,24,863,859,730,719,475,453,385,316,293 }, +{ 15,1,2,13,5,4,3,0,23,7,6,115,217,12,180,197,9,51,21,11,352,961,8,10,165,77,457,341,141,317,22,14,901,33,31,117,93,120,102,40,134,752,202,365,260,16,37,269,569,110,32,44,28,318,64,18,17,48,351,719,153,386,177,82,291,36,196,98,873,450,49,125,59,128,498,403,72,326,482,30,523,401,99,854,308,203,24,453,89,137,95,391,509,111,349,86,139,730,372,247,144,55,210,116,342,494,304,475,373,148,922,90,105,237,661,35,579,899,143,221,39,19,629,275,421,758,244,468 }, +{ 166,39,274,404,101,252,190,30,333,9,310,0,100,438,656,158,265,88,68,625,329,530,553,419,344,539,254,455,793,778,21,48,759,228,49,676,1,131,960,709,557,731,520,154,23,35,382,13,124,364,604,677,105,28,272,16,286,406,212,31,51,18,613,219,224,593,840,159,167,125,36,98,65,518,669,865,118,24,44,96,213,748,264,777,115,83,175,22,86,538,957,221,127,55,327,5,10,857,694,688,137,258,795,410,806,627,95,71,577,289,128,116,578,32,50,242,69,724,187,352,562,346,616,67,33,827,846,191 }, +{ 28,0,1,9,22,65,12,83,67,55,109,50,24,5,36,49,39,96,35,238,16,18,2,21,114,124,69,37,7,218,48,23,323,103,158,160,168,54,51,128,285,154,13,68,185,137,14,194,99,141,95,169,181,418,289,420,202,433,86,30,75,480,175,10,505,105,115,32,402,281,223,389,269,208,646,503,745,64,133,46,180,657,159,800,531,547,251,956,101,429,4,318,31,149,163,6,70,346,58,532,822,533,484,757,797,165,335,252,197,735,570,352,283,361,71,98,29,118,3,535,927,766,221,217,38,799,237,308 }, +{ 15,1,2,0,23,5,51,13,7,6,9,180,12,115,3,457,197,8,719,21,730,4,120,901,141,202,14,33,165,341,453,386,82,31,260,351,365,523,308,64,37,22,44,401,450,32,40,16,93,77,110,10,98,961,403,36,102,899,28,217,11,125,352,196,48,318,137,317,475,153,18,269,49,569,116,752,99,326,421,128,24,30,221,95,39,203,117,873,144,817,629,758,177,55,86,35,494,509,898,373,59,237,105,17,65,291,96,89,498,670,482,247,253,134,468,38,143,54,244,72,90,372,160,854,127,210,391,801 }, +{ 9,39,166,49,30,28,0,158,22,131,180,100,219,35,352,190,345,228,1,752,88,498,115,141,274,317,482,109,83,128,930,251,36,217,95,365,230,55,101,160,254,165,224,159,168,23,341,13,105,31,208,187,258,927,310,197,69,137,333,901,864,364,64,185,269,86,344,505,252,202,404,68,286,873,539,372,96,669,438,51,10,164,67,5,868,346,244,637,439,2,7,21,748,323,406,418,318,16,237,44,272,50,840,706,420,265,127,98,946,735,793,768,579,169,125,530,175,148,327,857,702,678,238,116,12,48,533,335 }, +{ 61,311,80,209,45,29,5,58,2,151,454,106,17,6,667,243,255,738,52,592,70,496,287,74,483,1,833,112,549,27,207,119,26,315,32,227,378,19,64,14,138,25,84,263,75,742,186,631,113,21,136,56,48,887,62,46,7,307,469,72,76,145,50,73,66,161,128,495,148,425,814,651,491,4,34,95,869,86,38,87,3,334,240,290,383,396,845,42,53,306,12,171,65,141,163,445,135,155,11,22,54,297,90,331,179,226,655,632,37,614,361,292,663,89,400,13,36,23,10,137,294,666,99,659,250,220,59,51 }, +{ 31,44,299,22,125,599,456,685,116,28,492,393,170,242,144,268,963,301,506,1,95,558,367,36,239,55,489,0,918,109,206,23,105,9,10,434,127,51,384,13,86,286,948,479,98,64,841,5,338,2,93,12,203,498,164,7,512,168,21,143,165,49,903,940,320,401,391,137,8,37,900,417,352,141,115,128,546,253,457,959,237,14,185,197,308,169,202,395,282,421,374,911,180,740,332,293,386,6,24,312,855,538,509,752,244,473,65,928,762,160,638,617,341,269,961,867,949,730,654,403,551,499,965,523,48,719,16,453 }, +{ 187,258,926,839,574,860,93,219,406,228,871,160,531,224,137,902,201,120,116,36,203,144,669,268,318,131,442,567,816,295,434,908,851,533,843,44,384,717,202,800,559,373,489,728,141,332,855,206,261,276,1,260,269,251,617,326,180,317,385,940,110,613,217,959,55,31,450,352,968,365,115,197,64,699,679,244,98,96,83,23,10,840,341,77,492,473,899,856,831,661,143,127,190,918,752,539,21,284,418,13,943,748,662,911,564,530,48,165,291,523,704,128,345,125,822,351,169,954,86,349,367,922,776,535,177,51,824,22 }, +{ 7,107,24,60,349,232,16,51,18,75,9,291,87,23,71,269,92,126,77,141,64,177,307,150,461,35,220,14,330,37,121,0,46,279,115,947,267,132,32,12,172,55,112,309,13,10,448,278,202,21,70,2,372,210,852,678,441,304,28,5,328,217,147,54,48,786,660,259,359,318,155,90,334,181,180,140,872,108,22,163,102,31,409,314,394,352,341,281,171,68,30,622,294,236,20,423,49,626,33,760,186,58,687,415,50,39,407,128,43,771,365,197,237,1,400,929,498,888,590,366,342,270,192,135,100,76,61,675 }, +{ 15,1,2,0,5,7,13,6,3,4,115,23,180,9,12,901,51,8,341,961,365,197,10,22,141,82,33,351,14,11,77,120,31,202,457,165,217,308,134,352,28,64,102,21,269,569,110,18,260,317,40,44,16,37,36,719,93,318,237,137,99,177,450,30,125,128,386,49,752,153,55,498,17,391,98,24,111,403,32,39,196,117,401,116,373,898,730,523,326,291,453,95,854,509,304,247,86,59,372,546,482,349,143,475,899,203,244,494,139,35,48,642,253,811,144,105,210,160,758,421,148,629,221,90,164,65,127,638 }, +{ 23,51,13,202,5,12,115,21,141,165,1,32,2,318,197,15,403,180,453,7,6,3,401,269,4,0,128,308,37,10,99,457,14,11,9,120,64,22,48,341,137,386,494,475,17,77,16,719,18,217,125,31,352,523,421,59,730,90,153,365,391,237,177,317,28,40,450,260,72,221,33,8,569,93,509,961,44,19,89,102,291,98,247,326,498,76,24,49,817,899,758,30,57,629,117,61,36,110,29,139,20,210,402,82,304,196,95,45,351,752,38,285,116,203,134,349,241,96,468,54,216,901,482,576,546,25,275,342 }, +{ 2,1,77,64,141,6,3,7,0,33,102,5,10,13,23,14,269,180,15,217,165,291,352,40,117,197,9,17,115,202,237,4,128,11,22,341,12,153,349,134,196,16,51,86,318,317,177,36,19,125,137,95,342,18,752,8,28,365,31,210,457,143,139,498,59,372,21,164,57,304,82,120,45,27,38,275,24,49,110,391,386,111,93,44,244,901,260,509,56,32,41,450,98,65,99,351,30,961,55,37,403,864,569,494,29,678,48,312,453,326,546,203,20,25,54,401,79,373,402,61,253,26,46,62,719,76,475,523 }, +{ 345,531,332,269,717,260,373,317,728,752,351,180,352,533,10,365,498,82,160,251,244,36,901,143,217,341,93,55,1,141,873,864,898,35,9,564,64,851,951,22,417,137,253,28,617,169,67,69,120,559,854,197,601,482,318,546,115,768,21,96,83,661,261,876,110,372,237,164,144,535,574,203,800,206,16,86,349,31,824,699,50,48,831,902,843,860,434,968,125,221,579,959,700,679,384,926,922,295,7,51,23,291,148,418,230,202,24,943,468,896,201,5,776,77,839,44,258,954,816,13,811,940,844,637,473,868,822,308 }, +{ 92,16,189,7,77,60,388,140,33,153,196,102,35,117,230,134,150,733,314,24,14,71,21,915,246,1,108,594,97,18,32,48,66,126,69,517,121,462,64,22,98,13,399,723,65,5,291,95,67,451,674,267,232,217,653,298,2,23,38,223,104,711,83,0,197,90,36,423,149,46,125,49,167,28,68,195,20,172,120,251,86,12,289,214,508,115,386,633,236,133,10,50,350,458,191,165,330,51,107,31,260,6,43,55,271,841,450,457,273,326,796,146,509,221,603,380,237,702,9,194,4,202,199,103,52,357,29,74 }, +{ 234,142,178,202,639,5,49,450,77,70,786,416,929,48,259,455,21,0,878,147,315,481,198,329,427,318,1,125,105,120,61,113,534,846,58,344,12,132,95,518,465,325,22,31,163,557,494,13,159,32,761,767,128,590,146,37,798,195,269,711,371,291,402,265,50,342,158,362,260,39,98,221,884,687,30,64,279,137,2,167,10,771,165,35,640,626,522,403,293,310,102,23,817,347,170,28,581,99,141,236,127,150,107,51,271,87,763,623,845,212,7,18,423,194,17,523,112,345,663,33,44,376,239,199,121,391,116,75 }, +{ 81,715,192,712,681,636,104,173,750,91,370,0,74,95,718,22,7,825,879,20,49,560,43,64,422,172,397,66,138,207,784,86,500,65,28,18,707,204,414,10,577,174,319,426,32,38,312,888,348,330,4,526,289,52,355,61,511,861,347,33,237,29,102,232,448,31,21,353,835,124,5,6,2,643,36,85,97,117,153,77,306,59,134,819,140,673,671,24,40,120,165,729,568,240,135,280,324,105,125,98,344,519,233,387,194,288,196,693,474,241,214,701,44,107,583,314,260,450,847,287,69,25,872,587,343,84,9,817 }, +{ 23,13,51,15,0,10,9,115,1,453,2,961,4,77,5,457,11,40,33,401,3,7,475,197,102,14,569,165,22,59,12,6,719,32,64,21,99,730,57,17,177,18,37,8,217,386,403,196,28,308,16,31,141,139,111,341,523,120,125,180,153,117,629,391,210,36,45,237,72,899,49,95,202,128,485,247,38,482,24,48,44,509,90,27,260,450,30,19,137,317,365,41,65,291,269,89,134,342,86,304,498,421,474,82,79,148,54,277,758,318,352,46,76,56,326,468,29,670,349,494,275,351,437,300,93,50,546,873 }, +{ 64,297,5,80,445,61,95,311,250,312,472,34,237,86,45,180,738,17,151,58,22,667,869,247,2,14,73,288,194,106,53,21,128,197,352,177,32,29,186,469,833,217,52,165,287,255,3,1,495,391,396,243,176,137,90,546,125,56,161,141,136,592,48,10,209,27,0,231,76,751,62,11,437,315,119,887,115,334,179,23,84,498,651,207,77,4,864,145,13,881,389,304,6,59,49,631,372,65,16,240,164,153,112,378,549,483,584,527,316,26,862,509,409,752,51,7,875,18,933,171,269,196,457,283,241,148,114,19 }, +{ 71,16,24,18,60,35,7,92,118,154,68,149,150,14,175,28,124,55,302,97,0,273,126,108,429,140,230,43,375,2,246,109,20,628,69,336,189,9,101,66,38,298,107,65,439,22,517,346,279,345,1,467,419,21,67,169,399,49,48,46,158,847,420,10,570,185,232,251,6,653,29,252,441,314,12,566,637,191,5,535,51,52,508,423,364,533,722,36,32,398,172,74,882,674,917,168,121,95,39,23,621,83,221,159,105,50,505,356,13,701,96,37,484,694,962,173,608,604,135,267,86,104,64,826,329,103,363,236 }, +{ 195,360,156,591,687,771,163,626,70,883,146,132,586,371,407,379,354,48,846,121,293,13,296,31,761,98,18,767,49,105,385,44,279,522,668,21,1,55,23,125,5,518,271,127,590,878,340,10,608,223,33,807,167,259,77,447,116,75,60,658,102,446,265,929,115,128,35,457,221,87,786,301,284,4,95,51,165,395,39,481,698,541,781,291,178,664,372,150,252,777,557,64,30,24,16,153,120,40,558,535,281,107,465,763,638,783,663,393,101,937,9,467,58,196,217,144,61,401,212,199,117,299,941,32,22,817,260,118 }, +{ 13,77,202,318,23,51,33,269,141,64,137,32,291,0,128,102,21,494,4,10,403,217,177,342,5,196,9,349,117,22,153,134,98,95,115,59,31,120,753,18,16,40,48,28,90,372,317,237,49,450,1,86,65,244,352,475,194,11,180,165,142,100,44,125,76,12,116,93,523,275,127,121,24,20,453,365,341,61,859,401,144,82,402,210,198,178,147,88,37,17,2,351,234,105,7,6,326,197,110,30,203,143,922,854,851,427,325,259,118,107,101,72,55,704,421,260,114,36,208,329,265,38,164,864,752,717,558,498 }, +{ 192,81,172,173,7,426,97,232,861,715,43,879,66,712,104,387,330,568,825,74,526,893,91,298,885,448,636,681,140,92,49,560,6,204,135,779,22,5,95,572,707,64,125,726,750,14,370,563,314,20,324,10,362,29,174,246,713,0,718,356,107,138,25,397,85,63,18,38,71,21,441,98,58,500,230,31,501,32,319,408,69,729,189,13,663,422,24,784,359,241,858,165,16,347,263,77,357,623,673,26,888,2,12,353,900,141,23,52,126,37,33,622,278,830,197,48,233,40,443,355,44,42,45,267,87,19,61,237 }, +{ 14,2,1,67,6,24,69,66,7,29,38,108,36,65,83,124,46,52,150,86,25,484,28,302,74,22,388,18,267,398,458,103,50,273,168,97,12,5,16,0,650,135,26,19,160,75,172,189,21,10,208,32,289,278,380,547,782,95,610,140,51,602,118,37,114,451,429,357,42,71,23,324,505,194,149,420,48,143,92,87,17,13,421,115,60,54,70,35,584,96,756,462,20,238,109,185,27,164,64,45,220,244,253,594,3,154,99,138,43,283,335,701,133,309,128,403,797,49,58,732,550,202,169,646,401,423,218,847 }, +{ 202,77,33,13,102,128,51,318,23,64,494,4,141,196,59,0,115,403,40,22,269,210,177,153,137,117,32,90,10,475,291,21,342,15,11,134,352,95,18,65,49,217,86,16,180,36,197,453,165,275,57,1,402,9,194,48,24,341,719,28,105,6,123,349,317,285,523,125,120,372,216,14,450,365,20,457,2,237,7,82,576,98,3,421,12,5,260,247,110,730,116,44,31,901,752,300,8,208,30,498,351,96,93,139,678,164,46,19,401,386,308,817,615,214,50,35,961,758,859,326,160,17,503,391,293,242,221,203 }, +{ 269,318,141,291,77,202,349,217,947,304,33,177,102,851,372,352,753,260,351,120,5,137,854,752,494,275,403,32,922,22,180,475,21,317,30,10,954,31,64,44,418,365,49,28,48,450,39,23,95,341,661,373,82,11,4,36,93,13,61,128,7,6,3,2,859,51,0,197,800,576,96,59,901,69,165,40,203,144,247,1,86,898,12,76,70,160,498,437,342,237,125,9,433,330,238,103,98,67,453,312,864,367,326,253,87,58,361,221,99,90,888,717,678,579,65,774,241,115,50,45,968,951,704,615,473,277,244,134 }, +{ 6,74,138,85,66,222,226,87,42,280,25,135,26,792,500,294,75,29,220,204,70,174,337,145,235,112,278,853,461,19,52,97,1,32,122,38,808,2,256,172,64,889,69,324,380,774,620,802,609,525,7,95,353,21,571,5,65,580,746,415,22,108,140,96,14,241,24,331,63,45,544,287,67,237,267,125,262,92,394,348,103,86,163,632,426,90,194,606,165,630,550,58,53,36,233,76,91,192,357,28,27,17,141,189,416,72,582,486,81,49,114,16,622,113,78,115,10,928,444,537,872,462,46,605,83,350,335,150 }, +{ 9,0,105,39,101,18,16,68,310,24,333,272,252,127,98,518,846,938,28,371,1,916,35,158,286,737,455,31,301,265,327,100,395,616,49,30,254,7,578,55,23,827,118,512,109,44,329,21,65,890,60,320,242,116,137,709,170,51,385,125,48,761,166,14,5,2,13,71,293,131,520,627,154,159,299,557,344,836,12,553,364,93,202,96,167,88,623,128,36,124,506,64,688,212,604,924,910,931,120,284,50,144,10,419,22,165,219,32,37,95,77,110,121,326,67,403,404,203,442,197,187,6,141,676,276,260,239,33 }, +{ 2,14,1,67,46,24,108,29,83,69,38,103,52,75,133,236,309,36,114,28,50,18,398,135,6,65,458,278,7,238,74,87,160,388,124,220,732,0,267,262,66,149,532,335,283,25,412,71,16,5,96,55,547,154,70,273,602,22,302,418,429,218,516,366,646,150,336,357,86,696,175,26,189,54,12,68,118,226,35,172,324,58,582,19,486,208,9,21,594,109,531,112,45,484,140,596,251,60,95,323,17,10,32,37,194,394,64,650,48,168,163,145,42,571,97,222,23,420,505,27,181,92,389,533,606,171,361,99 }, +{ 22,1,28,105,95,49,55,67,168,159,239,65,14,320,35,0,2,36,114,7,170,109,83,12,169,347,194,50,301,69,9,24,5,424,164,185,103,133,654,46,312,420,16,127,10,175,158,64,584,21,23,51,98,143,31,86,286,13,37,289,6,336,389,48,283,505,96,124,68,38,638,512,160,208,39,141,202,18,115,189,128,395,54,623,874,338,237,713,547,44,272,308,740,137,149,371,125,251,32,30,518,345,844,535,180,3,293,928,756,916,242,29,835,165,538,708,327,323,99,737,484,221,401,218,533,374,253,154 }, +{ 66,7,172,97,232,92,314,568,298,140,192,380,135,330,74,324,387,448,357,14,526,126,38,246,204,16,426,107,572,52,2,808,712,501,359,60,69,278,29,230,32,779,441,500,35,95,550,5,21,22,893,6,43,563,392,104,267,46,24,150,362,64,726,138,189,423,173,65,825,71,356,20,462,279,0,12,802,87,86,174,560,10,81,18,49,37,226,830,633,25,48,28,90,289,58,36,67,75,707,91,508,241,312,1,85,451,220,70,443,199,214,636,50,108,83,61,54,222,121,353,262,125,280,45,888,750,237,42 }, +{ 2,29,52,50,46,14,5,1,6,26,133,114,58,218,366,70,19,108,45,27,112,17,236,137,75,38,262,25,74,13,283,516,53,487,223,151,145,208,96,48,51,412,287,163,23,401,323,285,86,65,171,61,36,220,21,480,594,42,128,80,331,73,103,754,181,115,12,739,113,3,532,4,194,54,602,635,186,99,202,409,675,745,34,308,583,307,357,309,782,396,150,57,361,7,764,165,66,735,324,64,966,457,453,90,24,386,445,11,484,32,389,141,629,59,87,244,155,956,711,670,421,22,545,832,758,334,328,573 }, +{ 6,74,1,25,29,42,75,26,66,2,70,138,52,19,85,220,103,388,38,14,226,87,135,324,67,357,108,112,235,69,238,603,222,65,550,500,96,114,145,594,83,309,133,516,27,36,150,451,218,294,262,532,711,51,58,602,280,13,163,50,380,63,137,97,802,23,122,278,458,64,5,394,86,53,606,696,792,46,891,160,461,267,732,12,7,204,582,113,308,453,24,545,889,386,331,45,21,165,398,486,391,115,32,423,401,95,337,758,632,17,208,3,853,457,256,537,335,78,630,141,307,189,641,37,646,444,670,10 }, +{ 22,28,109,49,9,105,131,95,1,36,159,39,86,185,31,98,272,169,127,55,44,219,83,254,286,208,141,128,35,327,352,160,30,180,116,0,224,202,115,239,13,535,187,137,5,96,12,374,317,269,2,244,837,312,752,67,531,21,533,253,237,197,64,217,51,143,168,158,578,251,318,505,37,258,165,285,365,10,7,23,498,201,538,371,901,320,50,424,347,678,48,69,164,776,708,601,341,597,616,345,228,170,873,238,634,6,947,3,482,93,125,144,518,242,946,913,864,716,4,735,8,627,221,420,100,688,910,323 }, +{ 203,822,77,859,326,576,23,403,39,817,137,100,102,473,494,691,411,291,33,177,704,70,523,217,447,96,367,478,18,373,160,13,363,265,1,658,615,93,966,520,310,144,30,692,800,578,342,182,180,120,90,51,254,296,40,158,682,333,9,354,921,349,273,127,121,101,83,60,59,24,731,264,115,821,364,44,268,260,752,857,810,272,196,166,153,131,118,48,16,457,156,67,506,481,450,415,230,15,0,593,968,914,212,937,829,395,365,351,242,225,224,219,194,187,146,141,132,117,110,87,82,49,36,35,22,905,901,656 }, +{ 15,33,117,77,102,134,153,4,115,196,13,40,217,11,341,1,23,5,317,3,365,2,291,21,32,12,349,569,177,59,51,342,0,6,372,139,7,37,901,210,10,180,17,90,111,629,48,275,82,120,197,244,304,141,14,253,137,57,202,247,401,638,308,457,300,9,719,961,482,437,260,277,351,579,31,44,93,216,165,269,36,125,450,72,523,453,89,38,352,94,730,854,116,752,18,817,143,98,386,64,56,96,318,16,468,55,160,148,61,54,76,22,485,110,403,99,221,922,19,474,79,127,65,105,588,27,490,128 }, +{ 15,13,23,51,0,8,197,180,1,165,9,33,82,5,4,10,12,120,2,102,386,115,260,475,7,351,141,110,77,901,326,14,21,18,11,217,22,457,40,93,898,453,373,36,450,341,16,6,125,31,64,854,352,719,401,117,32,391,28,730,3,291,177,203,24,961,523,202,59,317,403,99,49,44,210,922,615,37,237,111,801,153,128,498,342,629,196,509,30,899,569,421,304,752,48,758,817,276,258,137,269,130,393,98,365,57,116,65,338,144,55,704,134,349,105,206,661,551,308,242,38,95,494,275,86,546,318,56 }, +{ 23,51,1,13,21,141,5,180,15,2,202,120,7,0,197,165,12,9,6,4,115,48,352,269,403,386,33,260,318,93,3,32,457,110,37,137,14,450,475,16,82,128,523,453,102,31,18,326,125,421,8,77,44,221,391,64,308,730,401,341,217,153,11,117,98,719,498,196,30,351,99,40,752,494,24,237,317,39,72,28,36,10,59,203,49,373,365,17,116,817,90,899,96,509,22,569,35,901,144,961,615,342,45,50,291,177,86,89,244,661,55,134,20,54,854,65,873,801,402,160,95,247,38,350,629,127,105,29 }, +{ 15,1,0,2,5,3,7,4,6,8,13,10,341,23,9,180,51,82,120,77,31,110,197,961,115,12,22,93,116,165,569,730,202,217,365,752,457,453,719,102,125,44,351,30,403,141,352,33,11,260,523,317,291,475,28,21,450,318,64,898,401,98,899,386,203,36,210,40,177,269,16,242,638,421,326,153,349,372,18,817,494,854,117,275,342,59,111,196,14,137,629,49,134,55,39,143,70,32,37,139,95,758,244,958,498,801,87,24,922,86,670,253,144,170,208,90,105,615,373,873,391,308,247,482,216,61,393,304 }, +{ 559,661,922,564,373,141,260,533,244,332,269,317,642,143,10,93,110,295,728,752,261,876,531,351,206,843,120,345,203,352,180,811,253,384,36,197,144,434,717,901,959,365,498,55,567,1,943,9,28,535,817,22,160,873,902,341,326,911,35,82,601,860,39,30,679,489,824,704,417,926,217,251,940,699,574,137,617,854,201,816,116,282,898,115,44,372,855,258,237,473,662,509,187,169,864,499,349,284,125,31,468,224,21,669,64,318,164,546,951,395,964,839,450,8,219,170,776,638,268,851,23,131,98,482,367,299,48,730 }, +{ 15,5,1,2,23,0,13,7,6,51,3,180,12,9,115,4,165,719,197,457,21,8,14,33,901,22,37,141,82,341,120,10,730,202,16,110,32,31,386,11,317,153,117,752,260,351,453,217,401,18,352,77,308,93,961,48,28,44,134,569,365,391,49,269,102,523,128,98,196,758,137,99,498,36,40,30,450,64,403,17,318,509,24,89,95,475,629,854,72,125,221,177,86,873,326,247,96,899,817,898,39,38,421,139,116,291,482,55,494,59,105,203,210,144,65,468,29,20,237,54,349,90,148,372,244,734,35,45 }, +{ 198,234,5,0,325,77,416,202,61,32,147,49,639,20,43,21,315,22,450,113,95,455,142,427,48,318,230,207,342,18,494,13,329,904,577,725,64,178,28,10,70,58,344,470,102,403,128,819,269,263,104,137,1,319,112,246,17,23,347,291,7,65,887,798,120,12,241,4,36,858,107,237,66,76,376,817,39,422,316,72,97,33,945,105,2,51,90,141,290,884,557,253,9,518,251,671,11,173,86,31,115,27,159,158,194,3,24,312,289,362,260,259,192,859,414,426,16,14,306,523,37,126,475,712,576,310,453,98 }, +{ 447,411,264,363,676,9,664,682,821,812,759,478,897,166,100,905,354,156,1,382,98,265,404,724,39,18,691,656,668,101,520,195,23,144,223,21,865,271,438,593,639,848,31,96,13,5,51,521,125,586,48,127,604,88,909,30,252,765,393,647,360,77,33,121,167,781,367,44,621,132,93,0,410,274,286,118,203,254,120,272,105,102,293,446,116,83,32,658,68,310,371,284,777,115,364,4,834,16,948,69,333,163,24,212,190,40,340,329,551,914,117,49,110,153,70,165,146,35,296,418,10,67,842,242,883,326,697,238 }, +{ 24,71,35,14,69,149,189,108,267,18,16,68,46,2,7,67,140,388,60,38,236,83,251,1,133,118,309,230,336,52,103,29,458,278,28,65,135,175,150,50,172,92,273,55,0,154,75,443,114,9,705,124,399,66,565,398,412,602,335,366,36,101,823,596,451,74,220,345,375,302,594,330,696,516,87,238,6,891,21,218,5,12,54,262,48,96,86,22,582,357,324,97,191,64,70,232,283,723,95,10,37,160,831,732,346,246,563,380,32,25,109,486,733,167,58,314,212,226,192,13,429,99,547,23,181,126,572,394 }, +{ 35,16,7,20,14,60,18,121,24,43,107,68,109,126,199,326,92,120,77,907,2,147,23,259,265,475,403,260,553,1,13,523,132,252,12,146,71,55,33,0,5,859,191,318,817,38,6,450,82,202,848,402,9,286,51,21,329,159,494,419,351,28,465,327,102,10,158,48,682,374,576,198,178,97,32,421,272,621,269,615,590,279,128,941,22,446,167,142,70,493,4,310,604,581,19,105,640,463,453,407,153,694,64,810,150,108,90,29,534,427,212,346,285,709,638,557,793,137,101,39,37,25,291,165,118,95,58,50 }, +{ 30,131,276,187,613,442,44,228,39,31,190,839,935,224,116,856,219,88,9,406,201,908,574,100,244,127,567,137,36,98,110,144,1,93,202,863,258,141,829,260,28,284,49,120,269,295,318,716,21,827,254,661,166,160,203,373,299,55,385,0,22,662,352,944,13,82,530,77,253,86,268,23,96,717,489,64,180,51,326,274,778,918,48,351,492,5,752,197,83,206,125,128,450,115,217,860,32,242,931,578,293,213,102,924,494,851,95,10,101,836,105,272,143,291,699,371,35,434,165,843,864,367,403,159,177,418,261,68 }, +{ 98,223,1,393,812,834,421,697,48,265,593,791,51,410,100,382,293,284,21,23,125,96,541,213,656,88,801,453,31,321,230,765,730,386,221,13,689,9,264,670,639,163,217,5,909,807,116,144,44,352,137,286,551,30,770,948,39,252,699,466,438,435,201,401,127,93,842,475,203,165,921,478,180,166,949,724,367,963,831,36,128,404,455,805,141,676,854,202,2,317,318,772,840,190,32,418,251,855,105,35,333,77,83,197,102,335,49,443,576,957,719,351,539,308,69,55,82,310,115,0,120,557,752,329,403,269,686,669 }, +{ 98,616,219,127,258,51,293,512,924,395,105,421,272,276,308,201,401,284,763,116,23,301,295,453,327,567,944,31,44,843,261,13,268,860,170,770,115,125,569,900,131,386,144,935,456,55,578,365,320,143,341,662,137,109,28,403,434,242,221,36,203,670,286,141,10,326,22,629,21,943,95,9,743,685,160,384,623,506,180,49,206,949,533,719,734,558,282,299,185,159,244,64,8,0,110,964,911,169,32,4,224,551,202,475,11,187,910,617,492,901,253,841,716,1,489,269,959,940,165,699,615,374,855,332,237,713,86,371 }, +{ 253,951,352,110,811,244,206,141,180,332,854,642,282,10,143,638,417,752,559,533,498,260,55,661,911,499,717,120,351,959,535,93,564,160,873,531,137,341,269,197,922,384,373,36,365,728,261,617,901,8,44,82,116,165,31,317,164,64,203,217,509,824,115,473,968,268,144,326,704,434,295,601,86,15,345,851,864,237,943,251,450,169,1,843,201,762,13,372,22,258,308,125,898,96,98,202,127,855,940,4,33,489,21,28,187,9,40,546,117,83,569,338,964,51,5,32,479,391,276,23,102,318,482,349,219,153,128,817 }, +{ 49,254,272,159,9,158,627,131,327,39,518,688,0,578,28,286,68,105,101,347,424,374,364,344,224,22,333,616,65,219,35,24,124,154,716,890,95,577,16,371,320,109,810,538,228,168,310,793,747,127,30,419,613,829,827,187,385,239,846,170,21,1,98,18,36,71,938,406,910,252,31,55,346,175,23,289,51,190,48,258,86,185,83,912,100,623,301,166,128,149,125,67,118,96,44,137,64,242,13,737,857,160,703,671,694,455,202,442,169,60,856,10,761,924,221,709,116,706,634,114,439,677,194,479,7,570,212,395 }, +{ 105,272,22,131,327,374,286,320,109,224,219,28,239,95,98,578,9,169,127,185,538,159,187,616,55,258,371,36,634,623,64,164,49,713,86,910,716,385,168,654,170,420,424,1,10,312,737,201,35,254,293,284,51,23,143,13,44,761,141,601,160,395,933,31,928,916,276,301,924,208,116,67,346,237,137,347,535,39,194,242,0,442,836,83,762,165,584,338,638,505,892,244,251,175,345,114,740,12,391,944,837,128,158,202,197,479,261,253,908,352,912,125,531,96,308,295,144,24,21,30,7,269,268,2,751,323,228,5 }, +{ 691,478,340,724,658,914,698,1,363,744,772,156,411,447,682,909,303,284,296,264,354,919,335,98,9,21,686,664,225,223,586,410,96,367,382,897,51,13,521,33,848,765,125,144,23,32,120,759,48,69,662,260,44,31,166,40,93,77,5,4,551,668,35,269,30,82,191,167,163,318,351,87,404,371,102,88,812,438,165,70,39,385,293,116,65,64,656,364,101,622,286,948,254,271,393,321,141,75,11,18,854,842,443,433,118,100,418,274,202,421,16,774,67,326,329,905,149,922,110,37,386,201,12,197,127,49,203,907 }, +{ 16,24,71,35,18,108,140,60,92,267,68,189,7,230,191,246,149,14,118,167,388,46,69,236,458,399,674,212,273,9,133,1,126,150,21,309,330,50,101,55,2,48,103,121,67,906,602,5,0,345,723,412,232,172,271,146,565,28,516,696,702,75,38,65,451,32,796,12,37,594,633,154,278,733,375,366,302,199,124,314,36,336,398,443,823,64,695,54,135,114,29,22,95,97,462,915,52,563,582,251,705,575,891,589,175,10,83,566,572,13,220,223,23,467,218,830,163,363,394,66,562,132,107,446,641,517,49,86 }, +{ 9,127,0,395,371,98,39,31,737,49,512,385,272,44,28,293,299,101,836,761,158,252,254,16,242,1,284,159,116,18,518,35,276,310,333,24,187,51,21,599,137,30,301,22,131,68,286,23,219,327,202,125,616,48,442,935,456,578,916,5,109,166,685,55,265,7,65,100,688,128,105,201,36,890,623,96,144,13,908,329,12,2,846,64,37,95,141,344,455,318,403,124,93,492,154,32,404,14,203,269,938,627,67,258,118,224,170,244,326,110,268,165,367,763,86,60,827,221,716,160,924,553,120,419,489,662,10,83 }, +{ 16,18,60,68,35,101,252,0,118,71,9,419,28,24,109,364,604,191,126,167,55,121,329,92,7,158,265,39,212,22,346,14,925,49,107,694,185,169,621,146,439,333,628,124,199,150,942,21,159,48,154,553,706,175,108,575,279,677,10,962,20,105,100,848,917,815,446,363,907,132,5,793,271,375,43,149,23,65,13,566,520,806,286,810,95,166,593,374,302,51,1,97,634,598,404,407,12,31,2,272,493,46,467,727,463,562,589,125,326,327,310,50,967,96,682,168,37,344,140,608,203,98,38,647,821,246,362,67 }, +{ 116,268,203,692,93,206,940,31,551,417,473,959,499,44,202,523,559,8,137,120,450,728,299,494,403,22,51,576,0,326,564,817,10,260,13,23,127,1,899,9,105,421,125,615,318,492,638,98,386,918,141,393,341,269,253,21,791,5,28,2,144,64,37,244,805,762,373,180,128,401,599,55,901,242,352,30,165,197,453,642,730,475,170,873,115,49,569,36,4,859,7,143,391,498,86,48,308,922,752,39,395,719,365,951,285,6,457,456,958,12,110,402,704,384,367,95,512,949,18,24,16,515,479,221,32,948,801,282 }, +{ 225,1,744,919,21,459,914,48,691,284,330,385,141,96,98,335,622,166,5,293,51,32,120,223,948,340,260,443,203,88,286,30,831,372,50,363,351,269,82,64,934,70,658,9,190,127,403,352,13,115,418,682,411,371,87,144,393,909,521,922,776,272,102,90,69,61,839,23,15,4,774,349,327,321,268,244,202,197,100,93,551,75,848,662,523,442,478,373,206,201,180,125,76,59,45,36,33,963,538,410,274,237,163,156,821,303,264,772,724,450,382,367,958,954,899,898,865,864,859,854,765,752,730,668,664,506,494,475 }, +{ 5,1,14,2,36,50,28,0,48,281,21,67,24,54,22,83,218,12,38,18,65,181,58,45,114,133,430,32,17,46,96,10,9,103,37,160,502,61,155,99,270,55,72,7,29,513,16,51,108,487,238,23,543,745,86,3,290,6,540,89,361,115,834,13,52,202,221,433,842,488,480,137,236,76,69,328,223,64,838,514,49,532,35,666,474,128,171,283,766,536,227,124,966,308,75,19,552,95,70,208,775,141,323,27,541,822,697,425,754,765,4,285,207,148,657,109,318,412,243,263,77,592,832,663,741,186,807,576 }, +{ 450,137,704,202,326,120,260,851,318,968,494,269,457,182,403,291,351,349,77,678,141,23,310,373,817,859,576,661,523,890,60,128,82,64,795,110,854,70,515,96,5,159,285,1,961,341,13,717,9,421,922,203,180,95,244,55,379,658,577,347,105,75,49,822,687,115,363,800,557,301,160,36,937,475,411,67,59,237,18,16,354,195,864,771,615,520,424,415,344,83,68,51,28,0,805,447,386,364,268,232,107,101,90,86,7,668,481,342,271,786,735,901,897,846,751,731,719,638,621,586,518,402,296,253,239,208,197,165 }, +{ 15,4,33,77,40,13,102,11,23,5,32,21,110,117,217,51,93,153,59,1,247,134,115,141,64,202,120,2,372,180,3,10,12,31,82,291,177,116,139,125,269,196,8,317,17,22,165,137,98,351,318,36,44,128,203,197,28,0,352,48,260,349,898,144,57,403,450,437,6,719,341,72,304,111,457,523,90,365,494,9,326,49,37,730,7,210,39,86,30,14,475,95,386,453,638,961,854,373,76,401,901,237,277,752,56,342,160,817,244,758,402,308,899,143,89,206,216,569,421,615,61,535,45,18,316,127,164,579 }, +{ 23,13,51,1,15,5,2,7,0,21,180,6,115,12,141,9,165,3,197,457,202,4,14,352,719,48,33,32,16,269,37,217,341,8,11,401,120,318,308,22,386,77,31,18,128,752,40,110,260,10,28,453,730,93,137,498,901,403,391,153,82,317,365,117,44,102,17,523,961,569,64,450,509,72,99,49,134,36,758,59,98,221,24,629,89,30,351,475,196,326,125,177,494,291,90,421,237,29,899,203,35,39,247,864,817,95,372,96,349,20,116,854,65,38,54,160,19,144,57,86,244,304,873,55,342,373,350,316 }, +{ 219,258,127,276,98,944,395,201,567,284,131,293,935,860,116,662,295,224,737,105,578,576,242,403,843,371,137,551,272,456,512,421,51,31,475,716,202,697,261,44,492,187,924,616,268,9,49,326,206,254,144,299,39,385,22,115,916,805,93,699,308,203,943,159,160,761,836,763,125,23,13,0,332,453,401,386,36,120,28,770,141,21,327,615,384,393,282,958,221,318,494,367,365,143,217,128,434,64,692,940,10,244,110,617,301,911,55,964,730,320,5,966,523,86,719,623,457,96,95,1,269,317,24,670,197,859,842,286 }, +{ 16,35,60,18,7,92,20,14,118,0,68,126,71,28,154,24,107,43,158,22,2,109,49,55,677,168,185,1,97,6,121,12,21,175,10,5,159,703,150,202,169,105,191,942,95,38,494,279,37,137,23,13,9,329,230,140,628,48,69,420,32,553,124,326,167,128,403,120,747,101,74,51,108,925,31,65,66,621,246,475,29,46,36,576,344,52,419,25,575,81,104,98,375,173,570,421,346,39,265,199,86,50,671,269,424,362,58,318,54,450,44,149,125,143,817,566,615,302,61,439,260,453,170,165,493,402,67,286 }, +{ 728,531,160,93,373,574,559,717,120,860,533,295,269,260,318,926,851,434,258,36,902,137,251,332,617,141,201,261,699,206,384,940,968,843,816,345,352,244,187,839,564,55,776,661,144,959,871,116,64,219,44,473,489,268,535,203,10,567,31,911,498,143,896,110,679,450,523,224,824,800,197,164,276,601,351,406,662,202,873,752,943,317,365,169,855,228,704,282,131,180,253,326,82,954,1,669,125,854,367,901,98,217,165,417,817,341,86,442,831,8,898,642,284,13,127,876,237,21,947,864,22,83,23,748,418,115,946,868 }, +{ 28,9,0,1,22,109,49,39,83,55,12,158,67,96,69,5,35,30,65,95,168,323,159,128,185,238,21,50,36,105,2,24,160,23,194,208,103,7,68,169,137,218,141,48,37,505,16,420,51,13,202,86,251,285,18,114,269,31,115,531,124,10,289,418,101,402,797,154,433,32,99,175,64,866,735,252,180,646,272,100,286,480,503,54,333,533,244,14,344,131,352,44,318,127,98,327,6,346,217,570,254,197,237,181,657,800,535,745,374,4,335,75,317,165,281,756,143,584,757,419,532,345,956,365,223,831,403,341 }, +{ 23,13,51,5,141,21,1,165,202,2,15,32,12,197,115,180,7,0,48,269,120,9,4,6,453,386,403,318,308,401,37,457,14,33,137,352,64,3,125,128,523,16,31,93,391,260,730,719,475,341,450,237,99,18,82,110,11,10,44,77,326,421,8,217,494,40,28,90,102,59,49,17,117,36,98,30,221,899,247,317,22,365,72,153,509,24,196,351,39,498,569,629,752,89,116,134,177,203,20,373,96,961,901,372,817,241,291,45,86,57,19,61,76,304,38,758,468,65,50,160,105,144,55,402,29,244,216,25 }, +{ 5,21,13,49,23,43,32,20,48,14,7,51,97,107,61,115,38,713,22,24,1,12,70,46,652,955,142,58,16,590,0,626,197,2,640,147,64,279,581,132,350,150,121,303,81,493,340,888,841,104,36,60,87,18,95,345,92,241,296,221,509,98,349,308,622,91,202,195,638,141,125,35,408,407,156,379,269,146,66,28,31,9,786,534,17,128,903,929,518,178,29,354,4,108,105,263,522,687,247,65,259,850,126,44,237,199,54,774,535,271,260,50,347,6,446,318,159,90,10,288,137,783,360,76,223,598,55,586 }, +{ 9,28,0,39,101,30,49,35,68,1,67,166,175,83,22,158,100,252,141,55,88,69,65,109,36,23,180,217,323,115,64,269,345,103,21,124,10,317,13,251,289,238,364,498,154,352,230,5,419,131,190,31,7,51,194,48,50,346,96,128,439,159,160,105,202,24,274,143,952,95,16,168,12,646,341,254,752,86,868,365,637,596,244,127,137,570,433,318,418,114,864,208,44,37,503,404,372,2,32,344,197,333,116,532,218,77,98,187,237,164,125,723,535,443,480,169,265,901,189,310,221,420,336,873,797,403,402,285 }, +{ 260,120,450,351,854,82,269,922,5,32,13,125,33,23,202,291,165,21,197,77,137,1,141,203,457,349,318,51,4,326,177,144,180,753,730,275,386,523,817,237,116,217,373,719,403,98,93,31,372,128,241,115,12,899,704,341,304,196,102,44,40,391,48,2,342,898,954,864,801,494,284,208,110,76,59,365,352,285,207,947,758,663,615,599,558,492,482,456,408,350,263,210,201,160,64,57,49,19,18,16,14,11,10,7,6,961,317,361,37,968,752,661,498,475,453,402,323,300,293,268,244,206,154,127,105,96,95,86 }, +{ 121,16,60,363,35,18,101,20,167,68,621,259,126,191,604,407,107,132,55,364,43,146,199,252,7,14,92,419,590,147,212,71,9,326,5,463,520,21,694,917,465,24,354,446,647,109,13,271,682,0,28,22,120,203,10,97,806,821,48,169,77,279,727,362,2,346,195,581,640,932,178,589,118,115,102,125,23,202,51,439,941,907,848,411,450,33,857,427,634,165,553,154,108,180,39,494,325,185,373,329,142,475,265,818,49,342,158,967,403,260,1,523,374,612,110,962,12,534,955,37,615,376,150,793,692,562,137,638 }, +{ 23,13,51,5,1,12,2,141,15,21,115,0,202,180,7,32,197,3,165,4,6,9,401,37,269,14,308,457,352,11,318,453,341,386,120,403,10,16,33,22,8,719,18,391,40,31,77,48,217,28,137,17,99,365,523,475,569,128,730,44,93,59,260,102,498,450,90,49,509,24,30,317,110,752,421,64,247,125,72,82,237,221,177,629,98,351,961,494,153,36,89,19,117,758,901,373,196,326,57,38,39,29,20,65,546,304,134,116,95,899,817,139,372,76,35,55,316,241,291,468,482,203,144,210,45,402,873,54 }, +{ 201,144,206,443,418,332,203,435,330,96,494,326,473,372,335,269,434,268,187,403,822,817,459,1,88,498,564,384,839,382,617,367,83,264,934,559,295,276,30,717,344,521,141,831,321,274,44,202,576,31,9,686,709,190,100,921,699,812,252,55,230,137,208,450,616,457,284,180,128,93,28,859,410,966,228,265,456,352,69,523,318,244,158,795,961,747,735,662,584,531,421,855,213,168,960,225,478,959,899,716,578,577,489,466,417,291,223,116,36,35,23,13,954,952,237,800,349,285,77,744,866,772,752,346,197,166,98,32 }, +{ 44,276,31,116,201,284,662,567,131,489,144,268,295,434,918,30,9,39,98,137,187,219,299,224,935,367,49,384,385,36,93,22,28,0,244,261,141,127,492,699,203,110,254,1,202,836,160,442,253,258,105,96,64,269,393,51,128,371,125,23,716,55,86,318,326,10,908,120,206,940,855,77,143,190,827,13,21,228,352,217,100,35,159,48,165,115,272,944,373,948,260,617,829,102,197,166,578,737,83,16,332,293,95,717,450,24,523,88,180,403,109,18,177,37,164,5,899,242,959,613,730,285,291,15,7,616,911,473 }, +{ 520,664,478,604,759,264,167,1,724,777,9,647,411,682,48,806,936,363,101,223,265,410,21,68,118,252,100,907,848,447,772,98,96,781,0,212,88,905,16,821,39,925,917,23,329,909,382,18,166,942,125,191,639,221,13,24,51,676,124,932,321,897,621,628,521,5,71,154,69,31,30,815,686,293,541,49,284,271,60,593,765,55,32,812,404,435,335,807,163,562,44,834,553,35,691,65,364,763,105,83,146,28,127,274,10,158,7,190,393,64,238,213,419,67,701,421,354,33,22,254,697,149,116,272,201,115,77,144 }, +{ 70,23,87,21,60,75,163,182,267,92,291,120,115,107,146,379,195,132,446,795,937,246,126,121,98,71,48,18,7,415,931,566,232,133,51,24,125,371,821,668,535,411,401,340,318,308,221,116,77,44,40,32,11,4,281,850,293,563,537,407,273,217,202,199,140,13,5,807,197,64,591,385,33,883,386,626,687,891,823,820,770,766,754,743,670,658,633,627,615,598,551,545,541,458,394,392,372,364,361,349,342,338,333,330,301,284,283,268,264,216,210,105,102,100,95,49,35,31,866,774,597,498,335,244,238,237,160,2 }, +{ 39,9,737,127,0,310,98,827,846,395,100,31,371,916,254,265,385,105,272,101,18,30,836,1,242,166,44,512,16,557,116,252,131,578,938,931,137,93,301,28,125,68,333,24,51,404,23,120,286,616,856,35,299,21,55,13,64,202,593,716,128,96,48,326,329,219,260,158,7,110,264,5,688,456,77,203,165,36,49,373,924,761,442,118,863,639,144,518,318,284,228,88,438,60,327,363,12,450,33,190,82,10,411,731,102,65,293,109,455,95,170,908,599,623,494,115,478,187,141,2,14,22,812,685,276,403,935,197 }, +{ 107,126,279,20,43,356,362,92,7,359,16,598,683,595,246,441,60,392,939,653,399,35,199,319,125,649,97,259,414,71,0,493,121,858,230,150,140,232,191,49,48,14,702,173,575,68,95,298,422,674,18,64,98,590,147,22,612,118,28,24,672,104,819,407,345,146,77,660,695,10,953,375,423,31,945,314,293,33,640,23,115,93,158,44,167,149,132,154,21,783,5,175,81,807,508,376,105,109,59,13,40,391,779,387,237,124,589,470,55,426,4,330,727,729,51,427,169,120,796,223,91,189,446,117,153,541,463,755 }, +{ 206,417,93,959,728,559,499,473,137,120,8,564,141,244,260,253,450,31,44,10,269,352,203,318,373,143,341,202,638,116,752,180,144,951,940,326,268,110,160,851,115,922,127,98,717,642,384,125,434,64,13,36,51,282,615,704,365,661,197,498,23,901,817,30,762,489,276,692,15,873,165,22,968,86,1,854,128,386,299,105,217,811,55,0,242,421,317,351,523,367,237,911,21,170,49,77,9,349,391,28,509,479,187,177,82,201,576,285,164,898,492,401,96,456,291,569,494,800,338,33,32,5,39,372,12,261,295,293 }, +{ 259,465,147,581,534,132,590,687,146,427,199,279,941,195,955,640,786,107,178,522,798,121,929,362,150,5,198,325,626,142,35,21,878,376,407,360,167,48,271,1,423,10,70,20,13,126,77,22,125,98,493,31,163,49,212,202,0,105,43,783,33,14,156,345,883,102,612,767,28,97,660,346,470,221,23,450,60,12,771,92,191,379,120,591,2,467,318,9,32,61,4,535,128,153,117,116,562,95,953,18,115,595,159,494,17,371,44,293,818,58,165,51,7,68,354,246,234,269,87,342,845,196,260,72,6,134,481,93 }, +{ 523,403,817,202,15,899,450,120,51,730,494,421,719,128,457,453,13,23,342,102,401,318,64,196,33,475,141,180,49,5,115,269,308,197,237,221,165,99,4,77,137,95,260,0,757,576,365,316,125,65,48,615,402,326,159,11,671,389,312,289,283,247,210,203,194,148,105,22,21,386,835,12,958,749,743,697,485,351,350,341,300,177,170,158,155,151,136,133,129,127,123,122,119,114,113,112,90,89,86,84,80,72,62,61,59,58,56,45,40,34,27,26,25,19,17,3,901,752,153,134,117,961,922,968,967,966,965,964 }, +{ 141,559,244,10,661,253,564,143,365,533,752,180,55,110,901,160,922,373,341,317,93,535,352,206,260,531,36,197,332,269,82,642,282,964,498,911,717,261,728,873,943,351,137,64,417,854,120,1,811,9,115,217,898,169,28,165,164,86,384,203,22,15,83,21,959,96,617,499,318,35,31,8,482,295,951,251,345,372,902,5,473,601,144,32,546,843,800,237,13,864,44,509,109,824,67,268,450,391,308,349,23,258,77,567,851,291,202,116,949,615,125,926,954,638,551,569,457,51,326,102,59,434,817,418,958,719,629,523 }, +{ 219,258,276,127,284,98,935,860,943,843,567,201,137,295,964,131,261,116,535,662,492,44,160,253,31,224,206,203,615,268,144,332,293,692,924,918,384,244,944,326,36,128,434,395,716,737,367,456,187,55,96,403,551,318,911,617,616,385,699,533,115,940,202,365,10,836,576,966,523,959,77,28,21,93,9,120,489,141,1,260,48,371,531,217,143,282,564,125,110,180,51,704,421,299,39,23,64,363,254,393,442,242,494,417,105,822,473,317,578,269,272,697,22,855,102,805,475,13,948,341,0,373,33,285,82,49,932,574 }, +{ 422,319,0,945,207,887,32,577,693,43,95,20,904,22,344,61,104,804,707,306,426,124,5,725,7,241,153,325,316,49,715,414,113,671,115,239,263,329,173,198,180,102,18,365,117,83,24,33,376,510,942,355,324,140,134,69,21,9,232,77,76,240,237,196,189,177,160,158,36,10,835,292,341,526,334,331,168,92,48,16,330,37,25,204,64,901,595,416,392,347,858,362,244,174,164,149,97,86,60,58,39,31,30,27,790,770,222,65,2,350,40,11,4,814,13,933,424,159,819,397,697,623,610,491,403,387,315,288 }, +{ 1,22,12,5,2,0,36,21,28,49,105,86,95,37,55,9,7,48,23,32,141,83,10,51,31,96,269,202,13,128,50,4,160,239,208,318,14,168,159,6,3,170,44,67,164,54,115,285,99,35,109,98,244,180,64,137,402,65,24,127,16,352,320,18,197,69,323,30,158,11,169,301,708,678,238,312,77,717,597,39,143,165,584,194,125,17,253,217,341,735,281,114,116,218,654,181,403,237,424,103,347,317,185,177,45,401,15,776,242,29,46,72,68,418,498,433,223,33,569,901,480,531,40,657,365,800,851,8 }, +{ 15,13,2,1,5,23,3,0,4,7,6,51,115,9,10,12,197,11,14,77,22,217,269,8,165,31,961,33,457,202,32,180,21,341,40,141,120,102,177,16,569,453,28,318,317,17,260,291,44,365,93,37,117,134,110,30,352,18,403,719,153,196,49,401,59,475,901,36,326,98,64,304,386,125,498,730,349,494,450,137,48,247,95,90,523,308,752,24,128,210,111,99,82,139,372,203,873,57,76,116,237,342,72,629,351,391,482,509,19,421,39,144,373,35,55,216,89,275,437,817,20,86,105,56,899,96,38,65 }, +{ 23,202,77,51,13,120,1,141,15,326,180,269,260,137,318,102,450,5,128,21,110,217,291,165,197,352,115,317,373,2,48,12,7,0,523,33,203,386,403,9,96,64,93,342,6,351,82,32,14,457,177,16,36,817,349,752,494,37,4,475,18,210,901,854,341,285,615,59,275,90,498,50,244,160,391,453,99,308,86,10,365,401,22,719,922,8,221,24,730,28,704,402,421,11,125,83,67,237,661,3,40,898,30,153,35,961,31,196,65,253,39,576,17,44,247,20,569,300,95,49,899,116,509,473,117,143,70,372 }, +{ 15,1,0,2,51,13,23,5,7,180,9,6,12,4,115,33,197,523,21,120,3,8,14,457,901,165,351,141,450,260,202,365,341,82,16,40,730,31,37,308,719,401,77,93,44,18,10,899,64,453,403,32,137,317,386,217,36,48,110,98,475,102,326,28,22,128,269,30,11,116,569,421,352,318,817,752,24,196,961,221,99,39,49,144,153,629,55,89,17,203,373,35,117,59,801,898,291,873,177,509,86,125,494,244,95,160,482,134,498,854,72,372,96,615,349,468,38,143,758,237,90,127,391,54,342,670,734,247 }, +{ 16,24,18,92,35,71,60,7,75,126,140,108,50,132,9,21,68,121,14,246,13,0,146,23,267,1,48,128,220,107,70,230,150,32,12,199,118,5,394,181,87,137,537,46,51,133,269,189,141,54,64,22,167,10,202,156,2,330,281,820,171,77,99,291,774,217,177,349,58,65,39,37,103,69,163,28,95,67,622,318,309,633,86,236,101,360,372,6,563,36,195,304,115,38,641,314,30,149,522,232,20,591,352,354,49,545,55,98,571,271,317,526,649,407,672,278,100,640,586,45,223,114,31,678,430,237,388,328 }, +{ 180,341,901,15,1,365,5,2,0,4,197,7,6,115,3,752,10,13,569,317,873,23,165,8,55,141,9,143,44,93,31,498,217,51,638,253,509,308,352,12,22,401,244,36,116,33,21,11,206,64,144,120,951,391,629,137,16,37,86,372,269,125,734,386,134,110,351,40,18,421,417,260,77,282,117,326,268,499,102,457,164,318,482,373,237,203,170,127,95,35,661,642,82,202,177,559,468,261,201,196,160,221,105,961,32,28,854,551,169,24,349,811,762,728,564,338,14,242,717,291,109,153,139,128,403,312,437,851 }, +{ 9,0,101,175,35,28,39,68,364,67,252,65,124,289,158,336,154,166,83,55,100,114,439,103,419,69,49,194,346,1,30,189,345,570,109,88,22,168,64,251,50,16,547,36,230,96,596,133,389,868,265,238,149,86,429,24,694,21,128,218,637,399,141,703,23,646,51,826,13,671,208,677,159,706,95,137,7,797,505,610,237,12,115,756,10,404,484,131,768,190,2,333,71,164,18,799,329,323,344,967,283,202,274,420,782,930,48,105,286,140,217,702,480,143,212,54,37,962,269,14,857,185,310,532,254,535,197,118 }, +{ 64,115,180,197,237,165,391,247,217,352,386,316,304,32,90,13,21,468,95,72,317,76,498,89,365,291,141,125,22,509,349,288,179,86,546,98,372,23,341,225,437,49,116,569,645,796,923,459,18,411,308,177,48,640,4,586,490,16,5,12,591,312,241,933,117,35,65,134,482,752,672,269,164,77,156,302,71,1,264,873,410,321,118,446,360,271,102,59,24,682,674,246,953,88,149,751,51,521,92,823,447,144,864,37,7,11,695,702,166,781,698,662,647,588,458,403,401,363,283,239,216,194,170,153,57,33,28,17 }, +{ 15,13,1,2,5,23,0,4,7,3,115,6,51,217,9,12,197,10,141,21,11,961,457,180,8,341,77,33,165,22,14,32,31,317,202,120,40,37,177,260,16,102,93,269,901,365,352,28,44,247,318,18,117,110,134,17,386,569,372,719,752,59,48,30,36,498,153,403,196,49,450,291,82,72,125,137,304,128,453,391,730,401,98,873,349,373,523,139,509,24,64,326,39,90,475,111,99,308,899,116,95,89,482,203,351,210,86,922,144,437,629,55,57,421,148,494,342,854,35,76,20,179,817,758,275,105,237,216 }, +{ 132,199,146,121,407,60,493,279,640,598,534,522,195,581,590,955,360,126,259,107,150,783,362,35,5,156,271,49,167,939,18,16,147,612,941,591,467,21,883,465,423,929,0,10,191,92,22,298,441,356,446,13,70,48,7,31,626,28,77,24,32,296,178,97,212,33,354,108,608,786,623,68,345,586,727,660,71,649,687,102,163,125,9,105,115,683,55,4,463,379,75,23,87,98,14,953,293,159,50,44,818,761,376,61,246,40,64,763,713,109,1,120,117,767,359,427,95,101,20,344,58,43,607,221,575,663,142,153 }, +{ 28,1,0,22,9,24,65,55,12,5,67,36,96,83,114,50,18,16,2,14,35,103,21,109,49,48,7,218,160,54,124,69,433,37,281,238,23,10,181,51,223,133,137,39,154,745,68,58,99,480,202,13,95,86,75,128,657,115,283,532,285,46,32,163,323,822,38,141,175,547,30,389,766,108,45,168,646,64,194,429,158,169,185,155,361,505,418,70,420,800,531,105,208,502,269,430,6,101,71,866,318,765,402,289,838,398,149,31,251,221,29,484,757,543,4,3,533,308,180,503,17,403,72,754,171,118,217,89 }, +{ 93,120,957,968,459,77,30,160,352,217,473,110,102,372,202,373,613,291,137,260,190,752,704,141,317,31,318,961,269,177,44,326,203,851,330,863,450,264,321,265,228,224,728,457,304,247,88,924,116,840,800,219,144,365,197,115,28,187,717,49,39,944,559,908,829,406,934,349,213,916,244,131,64,225,36,98,82,9,730,403,342,275,210,180,109,100,86,33,938,855,55,831,521,367,719,10,661,165,922,935,258,839,918,864,669,539,351,274,196,166,83,341,15,959,822,772,617,285,897,531,410,716,686,478,871,251,901,836 }, +{ 15,0,1,13,9,23,2,5,51,7,12,6,120,4,8,10,102,197,523,901,341,180,14,260,115,82,77,141,33,165,3,16,21,18,202,59,450,22,403,457,110,817,93,475,64,351,11,453,137,37,401,898,421,128,28,40,719,31,961,217,36,48,569,365,30,125,269,308,318,373,203,317,210,32,494,98,44,116,730,352,24,86,49,326,386,117,342,854,95,629,17,153,144,55,391,177,922,899,99,291,237,143,196,39,805,134,221,96,576,670,139,752,35,253,244,661,300,50,57,127,402,65,638,349,19,498,206,111 }, +{ 206,417,244,499,141,253,44,8,137,559,269,144,93,10,31,728,116,564,203,143,268,352,717,638,120,260,160,752,951,318,384,341,180,282,373,959,434,127,110,197,202,125,851,64,36,901,642,473,498,326,332,115,165,450,201,98,873,242,922,23,22,1,217,261,276,762,55,13,489,86,661,51,317,365,15,940,898,811,82,855,854,704,9,295,0,21,372,164,351,37,968,187,569,367,911,5,576,28,617,401,237,170,615,128,776,349,7,48,105,421,479,386,16,96,95,299,291,12,258,824,864,308,817,2,30,403,49,692 }, +{ 530,840,254,228,406,829,669,613,827,96,21,48,51,31,23,960,137,224,219,1,166,131,39,98,679,438,310,190,856,116,93,403,274,221,144,127,258,846,36,202,180,938,688,120,318,765,421,385,931,317,475,13,77,816,876,105,44,187,30,115,5,625,203,125,752,442,141,32,367,308,365,284,924,778,341,842,50,28,268,333,242,217,860,160,100,22,616,539,110,88,871,159,37,345,9,165,33,839,332,299,83,574,351,576,68,404,371,10,518,326,82,748,716,863,492,578,401,265,456,102,269,656,128,926,557,531,498,494 }, +{ 15,13,197,457,0,719,23,8,730,1,82,165,120,134,9,10,5,180,2,141,7,12,115,51,260,901,4,899,351,6,341,352,33,3,22,386,450,31,202,11,117,569,523,269,128,93,18,898,44,110,758,28,752,373,365,64,30,961,77,318,498,873,21,854,111,49,102,37,16,40,99,137,125,317,14,391,55,217,24,116,326,98,403,474,36,494,482,817,509,372,35,661,277,139,922,615,38,801,203,143,291,153,475,86,453,253,629,304,148,437,402,237,48,642,954,564,864,431,244,95,242,144,39,401,72,17,196,958 }, +{ 68,0,16,24,35,9,252,71,419,18,28,101,154,158,109,124,118,60,39,55,329,364,333,49,65,677,346,22,7,185,265,149,159,175,169,191,810,302,553,100,14,92,344,108,439,671,703,793,105,168,375,286,272,327,374,21,694,706,1,10,67,126,254,634,310,95,212,289,701,2,48,167,150,455,166,627,577,30,747,420,273,538,46,69,962,36,51,23,424,13,12,64,857,429,593,96,5,570,925,86,917,199,37,125,31,50,83,826,628,912,107,239,604,98,638,114,688,221,967,131,709,137,170,336,140,20,589,128 }, +{ 51,1,23,13,15,180,120,2,5,197,202,141,0,7,21,165,260,326,33,9,4,450,12,102,6,523,48,386,115,403,269,137,457,352,14,318,16,128,110,77,3,40,196,32,18,93,217,82,317,37,341,8,817,96,31,401,453,36,752,342,351,44,719,203,901,28,475,221,421,365,64,730,24,391,153,30,308,373,10,494,22,615,59,11,39,99,98,50,116,899,498,961,177,372,291,35,854,144,509,244,17,49,922,569,86,72,247,65,349,125,160,210,482,127,117,402,20,134,55,873,38,83,898,67,275,88,285,237 }, +{ 535,253,352,564,365,110,82,180,854,341,10,533,873,55,244,898,752,901,143,642,141,559,498,36,317,951,638,160,661,115,197,282,964,137,531,165,634,351,64,308,164,217,169,206,417,811,86,499,482,28,332,125,373,509,15,569,93,546,83,728,762,965,96,22,468,601,260,911,269,717,9,349,922,109,345,800,291,734,31,23,338,170,1,372,538,629,221,943,8,120,261,44,21,318,743,494,401,128,202,13,822,579,403,251,617,959,719,770,67,864,219,185,127,117,48,286,384,299,730,475,453,402,350,69,421,391,98,39 }, +{ 31,125,44,116,338,64,242,55,164,36,299,1,237,86,456,10,22,558,506,144,498,492,13,23,762,170,268,143,206,685,93,870,180,844,599,5,903,546,197,115,2,203,509,169,479,165,141,127,0,352,253,393,752,740,8,137,12,51,28,341,391,638,7,244,9,457,160,367,873,98,128,384,951,963,282,202,918,21,269,37,6,105,569,629,434,365,120,301,4,708,318,3,417,489,923,901,719,317,308,401,386,535,217,14,15,734,95,551,634,499,208,533,332,49,758,83,824,961,730,239,776,601,523,751,948,717,864,654 }, +{ 23,13,51,4,115,202,5,217,77,141,1,11,180,2,317,0,3,7,33,269,318,15,365,341,21,9,102,10,32,6,40,12,352,177,22,197,403,64,165,128,901,59,752,31,494,137,308,291,28,8,18,349,401,16,125,90,30,14,237,498,372,93,37,98,153,49,453,17,36,873,44,475,457,39,342,196,210,24,48,326,569,386,421,247,95,203,120,509,275,116,391,961,117,244,110,143,57,304,144,216,482,719,134,139,468,86,523,55,221,450,67,76,50,253,35,105,260,65,437,373,730,72,859,629,46,692,576,111 }, +{ 39,9,166,364,68,101,158,333,0,154,562,28,404,30,65,124,252,35,175,274,344,310,190,706,49,289,570,100,168,438,694,419,346,703,826,55,16,24,439,21,88,149,254,577,429,810,109,159,962,857,671,212,455,656,22,610,95,194,1,336,747,13,125,67,36,131,530,865,48,71,625,793,10,589,23,64,841,265,484,539,86,224,558,350,69,237,420,51,835,96,272,105,406,677,165,228,83,18,31,141,114,286,557,221,347,115,137,169,722,778,7,391,98,818,816,308,840,634,709,44,185,128,960,777,143,258,669,60 }, +{ 15,341,13,33,23,77,141,4,0,351,1,260,102,51,82,9,40,349,854,11,115,217,269,137,180,202,922,5,901,22,10,117,21,365,318,197,120,352,64,12,7,153,177,59,291,32,128,2,165,196,372,36,403,317,457,28,18,8,16,304,30,14,450,31,898,37,3,752,48,134,139,494,421,6,453,401,719,90,86,569,523,110,24,55,475,210,49,44,386,17,730,95,247,244,961,143,125,308,342,817,629,98,498,93,96,76,39,275,509,326,99,285,373,57,237,35,402,160,111,253,105,391,221,116,899,72,127,661 } diff --git a/src/gfx/tex/bc7enc/rgbcx_table4_small.h b/src/gfx/tex/bc7enc/rgbcx_table4_small.h new file mode 100644 index 0000000..176af11 --- /dev/null +++ b/src/gfx/tex/bc7enc/rgbcx_table4_small.h @@ -0,0 +1,969 @@ +{ 202, 120, 13, 318, 15, 23, 403, 450, 5, 51, 260, 128, 77, 21, 33, 494, 515, 523, 4, 141, 269, 1, 2, 700, 137, 49, 48, 102, 7, 64, 753, 82 }, +{ 13,141,23,217,115,51,77,2,64,21,0,4,5,317,137,269,202,33,318,7,291,352,9,10,3,180,32,6,365,102,341,349 }, +{ 29,58,262,1,52,74,6,171,5,287,151,334,27,500,75,26,331,223,53,635,220,19,50,45,46,17,14,396,163,409,324,70 }, +{ 40,51,33,453,14,23,62,56,12,196,730,475,153,99,403,775,117,130,585,34,4,17,162,11,139,57,102,38,108,47,123,440 }, +{ 33,23,51,13,102,64,202,128,12,40,15,196,153,10,1,2,77,99,141,0,515,5,117,3,120,403,700,165,22,14,269,453 }, +{ 13,23,51,4,77,141,202,33,115,64,32,128,0,11,177,40,15,102,2,217,7,137,269,21,90,59,515,1,180,403,22,6 }, +{ 26,235,19,47,648,624,78,145,27,112,122,64,444,6,630,453,25,42,65,130,711,85,390,113,416,108,665,29,730,138,644,95 }, +{ 64,141,352,751,217,247,237,437,177,269,86,954,947,875,32,318,95,77,304,92,597,180,232,291,128,864,349,588,372,202,312,1 }, +{ 642,898,180,638,901,341,82,197,10,951,15,515,165,762,700,253,811,753,752,365,143,479,244,569,8,110,351,873,55,31,499,116 }, +{ 221,23,51,125,438,254,13,21,39,49,308,656,0,115,530,159,158,401,30,166,912,386,165,688,518,9,105,627,424,22,421,33 }, +{ 143,31,1,44,197,8,180,125,116,55,13,498,23,341,638,242,93,15,2,141,0,901,752,115,36,206,165,479,338,365,515,762 }, +{ 12,23,51,13,14,15,37,99,515,38,700,117,2,196,134,153,753,64,54,33,128,120,21,0,328,5,139,82,453,719,457,1 }, +{ 13,15,23,515,961,700,457,753,51,115,4,165,197,2,38,569,1,474,0,37,99,719,5,12,629,14,11,3,33,77,64,10 }, +{ 15,515,700,753,1,0,2,4,3,23,134,12,961,5,10,197,11,33,82,120,457,51,165,7,6,341,217,21,77,9,40,180 }, +{ 13,51,23,457,719,961,730,401,165,453,0,117,386,15,134,1,758,153,12,54,515,99,11,2,700,5,753,4,308,33,6,899 }, +{ 134,898,82,117,13,33,77,102,23,260,341,351,120,901,197,153,961,111,196,110,180,457,854,10,450,8,165,40,4,115,0,365 }, +{ 60,18,126,167,35,16,191,71,24,92,121,271,68,107,212,146,118,150,199,7,21,1,9,575,727,5,566,48,0,132,108,273 }, +{ 62,136,129,123,128,41,162,17,249,211,214,789,618,710,38,678,248,507,57,64,152,269,119,3,177,183,597,106,4,179,216,90 }, +{ 403,523,51,475,494,453,817,899,202,23,450,13,421,120,102,730,33,128,4,1,805,5,7,153,757,260,318,196,77,457,326,65 }, +{ 4,59,3,62,12,33,56,193,27,21,102,17,40,77,76,84,32,0,6,123,119,177,128,11,18,611,605,25,13,51,73,210 }, +{ 43,20,319,422,414,945,0,7,819,61,5,376,325,173,804,904,470,693,97,707,14,49,22,104,147,107,95,32,426,1,330,577 }, +{ 13,23,51,2,0,115,4,141,217,33,10,77,1,15,64,180,3,515,7,6,22,102,11,5,40,9,165,700,202,197,317,341 }, +{ 28,49,0,105,1,24,65,159,35,55,95,239,16,2,109,7,9,14,170,320,347,168,424,158,10,301,124,5,67,21,64,36 }, +{ 15,515,700,753,0,1,13,2,117,4,12,10,5,165,457,3,9,134,11,7,6,51,77,64,961,82,33,197,14,341,120,141 }, +{ 7,71,14,149,97,18,60,16,150,92,398,189,140,124,24,273,35,2,69,302,154,68,0,336,517,43,66,28,118,251,230,1 }, +{ 4,102,33,77,40,59,11,624,210,12,128,342,5,503,91,139,64,32,25,494,202,678,416,0,403,275,21,450,196,318,523,177 }, +{ 25,19,42,6,122,813,256,235,85,26,436,53,297,573,680,390,445,63,27,416,80,233,65,73,389,283,45,605,194,17,250,343 }, +{ 402,102,202,128,33,300,403,23,12,77,40,21,342,117,483,99,25,494,6,4,63,32,84,569,139,757,475,318,19,26,196,134 }, +{ 158,9,0,109,39,49,65,22,35,168,55,24,68,124,159,16,185,344,333,154,254,272,175,289,1,577,95,28,105,810,30,169 }, +{ 197,180,115,237,498,165,2,5,287,546,400,3,61,34,509,13,297,80,341,52,45,186,58,881,23,873,468,176,64,17,311,250 }, +{ 120,968,373,260,704,110,450,202,137,318,77,95,269,326,217,717,661,652,851,349,93,1,518,98,827,291,21,177,82,33,848,719 }, +{ 44,116,144,268,434,489,367,384,98,127,918,93,948,31,206,940,855,0,203,137,9,22,617,141,332,105,393,492,959,282,299,131 }, +{ 13,77,23,33,51,0,64,141,102,4,2,115,1,6,202,15,10,128,269,7,177,180,3,40,22,11,515,217,117,318,700,137 }, +{ 15,515,700,753,4,11,141,40,165,23,64,180,13,202,32,3,51,125,5,197,21,128,0,93,77,1,120,82,269,117,110,59 }, +{ 176,231,585,62,34,14,412,161,56,236,527,57,17,3,51,202,4,23,369,283,128,13,472,440,84,361,136,457,381,130,719,53 }, +{ 9,0,180,217,237,101,141,352,88,100,230,64,175,317,115,498,68,39,30,1,702,83,213,36,365,208,752,13,252,321,952,546 }, +{ 28,9,22,1,49,0,109,39,83,95,86,30,13,105,128,55,141,168,158,67,31,159,208,12,96,5,185,2,160,64,137,23 }, +{ 72,4,38,12,51,89,477,11,57,76,401,308,23,474,99,148,413,179,59,13,431,152,54,569,17,3,205,629,197,421,405,15 }, +{ 457,13,23,961,15,51,515,700,165,12,753,629,11,1,719,117,0,3,2,37,569,197,40,328,33,5,153,134,99,64,38,196 }, +{ 254,100,310,9,30,1,39,625,166,265,190,0,272,557,131,731,31,98,578,688,404,93,101,88,49,21,127,264,44,36,252,478 }, +{ 51,23,12,13,15,128,99,120,10,202,515,153,64,82,700,33,165,2,5,117,403,1,141,0,3,196,37,453,753,197,260,93 }, +{ 38,99,542,139,453,117,196,23,457,13,328,111,37,134,961,11,12,51,40,775,587,401,474,54,153,477,41,629,33,475,14,277 }, +{ 6,85,25,233,343,91,26,63,138,29,19,65,283,4,81,235,42,122,605,64,648,256,174,370,74,389,718,59,45,194,445,416 }, +{ 49,5,97,20,197,21,18,193,0,64,408,729,173,350,43,422,165,7,14,104,61,32,509,713,523,102,120,95,125,397,35,232 }, +{ 144,116,268,434,384,489,367,206,93,855,940,44,98,332,617,127,959,911,137,282,203,31,22,219,141,9,131,276,417,0,1,120 }, +{ 17,106,64,62,32,255,136,292,476,162,129,241,123,141,41,237,720,214,209,352,519,211,186,148,752,247,507,90,21,77,197,119 }, +{ 2,29,52,50,5,58,14,6,27,1,366,357,45,53,17,19,171,151,26,181,133,38,218,764,287,583,61,113,3,487,600,281 }, +{ 130,59,196,412,381,730,711,236,77,210,202,402,453,99,401,108,361,803,291,283,153,4,57,51,128,183,14,719,503,117,23,11 }, +{ 13,23,51,141,77,4,33,64,115,0,217,10,180,202,2,102,11,9,15,165,40,21,128,352,22,7,197,3,317,515,269,1 }, +{ 23,13,202,51,120,15,21,5,141,1,128,269,137,515,64,102,125,48,98,33,260,523,318,93,700,165,450,77,2,12,403,82 }, +{ 1,2,14,46,29,67,38,52,5,171,58,24,103,69,96,70,83,181,54,75,163,223,16,45,112,309,155,0,186,35,18,108 }, +{ 15,515,700,753,13,0,1,2,153,5,23,10,117,3,9,7,134,165,12,6,341,33,4,14,77,457,115,21,719,180,217,82 }, +{ 197,165,509,13,391,180,308,115,23,546,5,498,2,29,3,401,901,61,34,80,14,457,250,569,237,873,38,297,45,15,468,386 }, +{ 19,73,27,250,200,714,444,472,26,53,34,17,813,322,283,390,128,297,78,123,432,14,436,136,106,690,57,122,389,80,503,3 }, +{ 3,17,21,45,62,32,38,12,155,14,2,328,5,99,401,536,828,13,227,488,106,51,719,119,540,76,165,221,115,629,209,41 }, +{ 115,341,873,197,365,13,901,180,569,752,317,1,10,498,143,634,261,0,509,15,943,237,44,31,116,601,165,127,282,23,141,64 }, +{ 453,51,23,403,33,421,475,102,15,153,196,515,13,700,117,523,12,40,753,21,4,134,0,494,670,899,22,801,730,10,11,401 }, +{ 23,13,51,33,12,117,153,134,453,196,15,99,515,40,14,700,128,102,11,753,77,64,403,202,0,401,475,37,65,2,3,38 }, +{ 2,7,5,14,70,1,29,61,52,45,6,112,66,16,21,32,592,46,38,135,87,58,186,315,290,128,113,0,64,48,227,23 }, +{ 33,23,102,51,128,13,64,202,141,1,77,10,153,40,196,117,2,3,0,5,15,269,403,12,137,134,318,165,120,6,453,99 }, +{ 16,92,7,20,43,35,126,71,60,14,107,18,68,97,0,121,279,149,24,246,191,48,118,575,55,140,362,783,230,150,375,566 }, +{ 13,23,4,33,77,64,51,102,141,128,32,10,0,202,40,115,59,22,90,11,177,21,291,6,7,318,180,117,137,2,95,165 }, +{ 507,162,129,41,4,211,62,38,123,59,57,248,183,130,99,11,3,361,202,17,402,556,266,305,803,210,128,184,152,136,313,117 }, +{ 643,123,193,650,802,18,25,389,718,256,65,289,84,91,619,511,415,90,235,63,57,510,324,216,862,102,6,183,108,397,217,736 }, +{ 13,23,15,1,515,51,0,2,700,5,753,165,141,115,12,3,4,180,21,197,457,7,6,10,120,9,33,202,77,32,8,11 }, +{ 23,51,13,453,64,403,12,21,5,202,128,475,165,141,523,95,125,115,3,1,4,730,120,32,2,494,180,719,457,197,450,401 }, +{ 204,74,135,66,6,174,192,7,138,172,85,353,348,580,280,97,95,500,29,64,426,32,87,889,65,81,25,2,52,43,568,673 }, +{ 35,0,68,69,24,9,1,16,65,103,149,133,18,114,28,50,83,2,189,7,46,14,101,336,175,124,251,55,71,218,38,238 }, +{ 16,101,0,118,9,18,24,68,35,154,71,124,60,212,191,520,55,806,694,167,28,39,364,375,1,346,252,65,604,302,22,21 }, +{ 0,9,16,35,1,24,68,18,65,21,103,67,13,149,28,189,71,23,101,238,114,7,335,133,486,141,22,212,48,50,30,118 }, +{ 13,202,23,77,33,51,128,5,21,141,115,32,102,64,4,0,318,269,10,15,291,2,494,177,11,217,3,515,22,137,6,700 }, +{ 16,92,60,35,7,18,24,68,150,149,14,71,0,375,97,126,118,107,230,191,246,273,140,55,175,653,9,575,2,28,566,517 }, +{ 76,90,21,179,316,148,205,32,464,288,184,257,245,1,89,2,460,57,152,45,38,358,645,5,12,449,350,48,37,17,4,14 }, +{ 19,27,26,813,80,297,17,495,436,53,73,200,4,378,250,59,106,25,45,128,361,42,113,469,122,390,77,40,736,6,11,136 }, +{ 6,26,235,138,19,145,112,70,331,262,25,42,52,624,27,453,122,47,500,78,648,85,29,2,630,632,409,113,50,226,108,75 }, +{ 7,16,14,24,92,35,18,2,46,9,60,140,0,87,50,5,54,13,12,38,171,23,126,21,58,64,1,70,128,71,220,163 }, +{ 90,205,257,184,32,179,460,5,245,45,2,288,769,524,57,21,152,229,17,1,497,4,292,59,619,452,432,76,476,11,266,14 }, +{ 15,515,700,753,4,5,11,141,13,1,33,3,0,128,202,23,180,21,2,64,269,32,117,134,120,40,102,318,153,17,137,352 }, +{ 47,130,711,108,453,412,730,196,390,283,78,27,51,183,381,236,128,200,719,14,153,472,503,34,59,250,3,4,57,803,123,432 }, +{ 12,277,51,474,111,153,23,99,13,37,961,94,629,542,569,431,79,139,38,134,117,453,33,188,196,40,115,15,11,157,401,515 }, +{ 17,495,469,106,26,378,80,27,161,483,19,742,527,436,383,862,73,136,53,814,297,6,119,84,62,56,25,3,209,611,4,128 }, +{ 81,681,636,91,0,750,370,104,718,138,18,693,173,784,29,397,348,74,192,673,174,65,6,207,64,280,306,52,671,32,355,319 }, +{ 15,515,700,753,33,77,4,102,115,117,40,13,1,153,134,11,5,217,23,196,2,21,3,317,32,365,0,341,291,59,12,51 }, +{ 0,9,28,35,68,1,65,67,101,39,69,175,16,238,13,22,96,124,18,24,251,30,55,12,23,2,50,141,114,5,154,103 }, +{ 23,33,77,13,117,40,11,102,64,4,51,403,153,453,10,0,196,134,128,65,12,291,86,99,95,59,15,141,202,180,137,719 }, +{ 214,90,289,6,874,64,25,65,235,42,751,249,256,312,194,85,746,875,174,32,525,288,519,835,247,348,233,544,217,524,437,352 }, +{ 1,22,2,0,36,67,28,5,49,95,12,50,168,83,105,55,7,9,14,194,103,23,114,21,584,46,10,13,38,69,208,159 }, +{ 269,141,13,202,33,180,318,77,291,137,102,352,128,23,349,51,31,217,372,317,125,197,44,21,11,5,901,1,18,0,4,494 }, +{ 435,144,274,88,203,418,30,1,190,410,96,778,100,530,521,326,466,795,686,166,960,321,382,264,367,822,131,31,692,9,213,93 }, +{ 76,72,90,21,37,179,12,205,32,428,148,38,308,405,4,413,57,184,749,245,316,221,54,645,288,1,152,155,464,257,2,14 }, +{ 77,33,64,102,13,141,23,2,40,1,51,10,0,115,6,180,202,128,4,3,177,269,15,7,22,165,291,14,217,318,137,11 }, +{ 397,81,4,32,65,788,693,804,681,11,249,21,91,64,690,494,3,0,422,56,348,725,194,123,23,59,523,319,61,510,95,90 }, +{ 60,126,16,7,92,121,314,246,35,107,150,132,14,146,24,18,199,298,232,71,359,140,672,97,392,649,5,423,95,21,22,388 }, +{ 15,515,141,217,115,700,13,23,120,317,753,180,33,260,110,137,341,51,1,365,4,77,64,202,0,40,36,352,197,269,10,21 }, +{ 111,134,117,474,23,13,961,12,569,431,37,15,51,115,515,700,277,99,753,38,197,405,457,4,72,94,629,45,11,89,54,148 }, +{ 23,13,51,5,1,15,2,21,12,202,141,0,515,165,120,32,4,64,700,3,115,197,269,125,753,7,9,128,6,180,453,403 }, +{ 13,141,4,23,5,2,115,217,202,51,180,137,269,352,77,1,317,3,21,318,0,15,9,64,10,197,11,341,33,515,752,7 }, +{ 165,125,197,13,391,21,23,558,48,380,97,120,298,33,14,426,66,115,32,386,900,180,6,98,357,237,326,509,51,278,221,457 }, +{ 120,82,15,260,515,1,351,77,450,700,13,21,141,23,753,202,217,93,110,33,51,854,5,128,326,102,137,180,817,48,269,352 }, +{ 23,13,15,51,515,700,961,753,0,457,1,2,4,115,10,453,569,5,33,165,11,719,14,40,64,197,3,21,474,629,38,401 }, +{ 264,166,39,30,9,100,435,254,93,921,190,363,1,625,411,382,897,656,203,478,404,812,438,110,473,88,18,691,156,141,274,272 }, +{ 9,0,252,100,166,39,101,265,364,68,88,329,520,18,419,676,118,167,404,604,16,1,21,30,212,158,553,49,382,274,48,13 }, +{ 15,515,700,753,4,11,141,5,3,13,202,1,180,21,2,165,269,23,40,64,0,318,12,32,128,51,77,117,523,197,120,457 }, +{ 24,1,2,69,35,16,67,18,14,50,0,46,68,9,38,7,133,71,83,149,28,108,189,218,65,114,238,29,75,54,5,96 }, +{ 90,289,214,64,874,13,77,712,66,751,4,23,51,192,32,0,202,194,312,177,33,65,234,104,875,288,59,5,835,416,102,95 }, +{ 0,9,49,127,98,31,301,28,371,159,1,395,512,737,158,761,916,623,16,44,242,39,170,18,293,105,24,272,101,22,23,385 }, +{ 17,62,136,214,123,129,32,292,119,209,710,106,141,162,128,64,45,4,77,249,11,618,211,3,207,130,519,183,38,177,21,269 }, +{ 5,107,581,356,279,32,441,362,493,660,13,298,0,534,49,147,21,22,132,121,97,423,7,590,259,683,14,786,126,508,60,246 }, +{ 51,13,15,730,453,23,515,719,386,457,12,700,403,475,899,1,6,523,753,421,99,401,165,33,2,19,361,5,0,670,120,27 }, +{ 49,28,9,159,272,22,254,131,158,327,95,105,0,39,35,168,347,286,374,55,65,627,424,912,68,578,1,24,239,175,688,169 }, +{ 15,515,700,33,753,4,77,141,341,317,1,10,13,180,102,22,40,117,115,365,5,901,23,197,134,11,217,351,64,82,21,137 }, +{ 134,15,13,515,23,700,12,753,51,474,37,961,197,10,457,569,4,0,99,2,115,38,165,153,94,3,139,11,1,82,33,5 }, +{ 7,2,20,58,5,14,128,66,6,29,32,43,21,52,16,38,631,61,74,97,46,135,113,25,202,192,13,0,884,45,112,87 }, +{ 77,13,33,202,23,128,102,4,141,342,117,0,269,318,134,22,11,21,32,153,403,291,49,64,137,51,40,15,494,5,196,98 }, +{ 2,1,14,6,46,38,29,65,5,36,67,0,103,7,22,86,133,50,108,208,52,83,24,323,283,69,28,18,10,25,23,75 }, +{ 15,515,700,753,1,5,4,2,3,13,0,11,180,341,12,33,10,197,134,365,77,23,21,901,6,117,165,7,37,32,17,102 }, +{ 203,268,206,93,417,940,31,8,120,137,44,499,959,473,202,692,728,559,0,260,10,326,141,564,817,127,341,1,450,22,110,23 }, +{ 15,82,515,120,700,0,10,753,33,8,64,165,110,31,260,93,13,197,23,22,40,4,351,44,77,9,11,153,102,51,1,196 }, +{ 60,0,16,7,14,43,20,71,28,10,2,22,154,18,13,24,92,1,51,576,35,615,805,925,68,126,124,149,97,64,23,55 }, +{ 19,6,26,80,5,84,27,17,25,2,504,129,45,240,56,123,4,119,618,1,76,106,64,51,14,3,128,65,32,710,0,42 }, +{ 15,515,700,753,13,4,77,23,33,51,0,5,8,10,11,31,44,1,82,22,202,64,110,102,93,21,291,40,141,180,9,49 }, +{ 195,98,271,223,132,167,146,407,1,360,121,834,393,591,212,199,293,259,522,107,354,147,156,191,807,590,48,18,125,16,765,541 }, +{ 128,202,77,210,402,318,33,102,6,40,403,29,342,269,196,757,99,139,2,111,42,4,494,117,275,300,13,12,678,0,177,122 }, +{ 13,33,23,40,51,102,4,117,77,64,134,0,128,153,202,196,453,11,15,12,1,22,403,141,59,14,10,475,515,65,700,95 }, +{ 7,16,14,24,18,2,28,0,92,71,1,22,6,35,60,20,168,10,154,118,5,302,124,69,97,109,703,158,420,12,149,66 }, +{ 15,1,515,23,0,13,700,2,51,753,180,5,120,165,197,21,115,4,33,9,141,7,12,6,3,457,386,202,260,523,8,31 }, +{ 60,107,121,132,146,126,199,279,150,92,16,649,441,35,955,7,21,0,423,5,18,195,598,298,493,356,32,653,22,362,953,10 }, +{ 31,44,98,276,284,299,116,935,9,201,0,131,39,127,144,662,1,137,371,492,567,489,93,254,49,268,22,28,30,293,434,737 }, +{ 13,15,23,515,700,0,1,51,753,4,2,10,77,202,5,115,3,165,197,457,9,12,11,961,33,120,22,141,180,7,6,40 }, +{ 123,162,184,257,17,183,229,130,129,3,84,136,99,152,556,383,57,497,12,205,4,62,56,452,80,266,128,14,40,119,27,106 }, +{ 196,33,117,40,153,23,134,13,51,102,453,0,15,475,12,14,515,2,22,700,4,21,753,64,401,670,730,1,9,11,10,99 }, +{ 224,219,187,131,258,385,442,871,836,31,98,908,44,574,127,944,137,839,116,36,613,1,254,39,926,160,829,96,93,371,860,827 }, +{ 121,195,156,132,146,360,590,407,786,522,883,591,259,929,626,941,150,687,5,55,296,379,467,178,586,465,279,21,1,13,60,354 }, +{ 2,1,14,29,6,5,46,52,38,19,114,75,26,65,108,96,25,50,36,70,103,309,17,236,218,74,12,86,0,3,10,112 }, +{ 15,515,82,700,120,753,10,0,8,197,260,165,351,64,13,110,117,93,31,1,9,33,22,23,457,44,450,77,102,898,40,49 }, +{ 7,66,97,2,172,74,226,52,29,135,192,232,43,324,92,5,38,20,222,14,6,568,87,107,353,620,580,16,138,174,448,32 }, +{ 62,129,123,162,136,249,618,183,507,57,4,152,17,59,11,184,117,77,3,128,211,41,130,205,12,40,33,106,64,229,38,313 }, +{ 1,13,15,2,4,515,23,0,3,115,700,5,51,77,341,141,753,180,33,217,197,202,901,6,21,165,11,365,318,317,10,102 }, +{ 6,26,235,19,145,47,112,78,64,27,453,95,29,444,25,624,85,108,648,70,32,130,74,42,711,630,632,138,65,122,113,730 }, +{ 23,51,12,15,13,99,515,153,117,10,700,37,120,82,165,2,753,64,128,0,403,3,5,1,134,197,453,31,202,457,110,21 }, +{ 16,24,18,71,64,35,92,7,246,146,9,108,60,118,199,5,140,2,267,0,230,830,32,133,1,68,50,330,247,563,36,12 }, +{ 15,515,700,753,0,1,13,2,23,3,4,217,51,5,115,8,9,180,341,10,7,6,317,77,33,372,901,197,365,11,120,165 }, +{ 234,639,178,202,77,142,5,455,450,49,416,0,147,427,198,21,315,329,13,318,325,557,120,344,113,259,22,128,61,105,23,494 }, +{ 1,31,36,44,141,180,55,2,64,22,98,116,13,352,0,115,10,127,5,164,253,498,237,165,341,197,4,86,15,170,125,23 }, +{ 15,120,13,141,23,260,217,515,1,77,51,110,180,700,317,82,269,137,115,202,21,753,64,5,351,291,0,450,352,93,36,326 }, +{ 26,6,112,396,19,145,25,122,648,287,42,74,624,222,416,45,138,66,644,151,113,651,29,573,64,280,445,27,525,85,70,58 }, +{ 156,360,5,146,121,21,271,522,354,132,49,13,18,195,16,340,60,591,446,586,727,0,107,407,167,48,1,463,199,566,32,23 }, +{ 5,61,49,147,178,612,660,120,21,182,23,427,259,683,33,4,77,70,13,3,376,98,64,0,481,344,48,595,291,263,141,51 }, +{ 89,79,468,179,358,205,94,405,115,498,72,180,365,431,37,111,341,734,188,317,482,217,11,4,245,152,413,216,12,474,490,752 }, +{ 24,16,35,68,18,71,7,92,0,108,9,14,118,101,336,175,375,302,28,124,154,55,149,60,398,1,65,2,140,273,345,230 }, +{ 51,730,421,801,453,386,23,523,13,475,719,401,670,365,899,403,115,457,758,165,33,494,450,6,423,805,629,56,569,514,958,388 }, +{ 113,45,6,311,29,2,151,614,145,491,112,80,5,27,61,74,315,66,209,631,19,25,58,17,73,26,1,243,70,64,611,287 }, +{ 4,339,188,471,11,59,79,12,377,94,99,33,77,102,51,111,37,152,13,961,474,542,40,342,3,23,128,403,202,177,184,57 }, +{ 15,4,515,11,700,33,82,40,0,120,753,10,8,110,13,93,23,165,77,260,64,31,22,51,44,102,351,1,125,9,197,21 }, +{ 16,24,18,0,35,68,28,71,124,118,60,7,9,55,14,92,109,101,419,175,22,252,154,375,149,302,158,346,2,49,1,126 }, +{ 17,45,227,21,106,3,2,243,209,5,48,32,221,62,207,50,29,186,290,270,263,52,14,496,400,119,46,255,54,430,38,721 }, +{ 340,354,586,658,156,195,698,668,1,296,9,18,883,363,447,379,303,98,411,13,31,163,51,5,371,48,919,846,121,21,360,70 }, +{ 277,153,111,12,23,51,474,99,38,37,139,117,41,457,79,453,542,13,11,33,134,157,629,188,961,14,196,401,102,569,15,94 }, +{ 0,18,16,159,49,24,9,105,35,68,7,28,22,1,60,344,55,101,109,2,14,158,13,23,71,118,455,286,272,424,5,327 }, +{ 0,105,9,49,16,18,158,28,518,24,101,320,1,68,170,301,272,127,7,286,35,890,109,39,159,98,21,344,31,55,371,23 }, +{ 141,1,180,15,13,2,365,217,515,352,317,115,341,0,4,5,269,700,23,21,3,752,197,77,753,51,31,901,10,202,8,64 }, +{ 4,23,51,33,19,17,102,153,485,880,40,403,196,26,300,453,27,117,78,0,12,200,47,5,11,14,342,99,53,77,475,2 }, +{ 62,184,56,440,130,229,183,3,556,152,99,162,12,266,17,548,136,57,305,161,123,14,452,4,383,403,257,34,40,84,33,139 }, +{ 13,23,77,141,64,202,33,51,269,115,0,102,21,4,217,128,5,32,318,137,291,9,15,2,180,10,3,317,177,515,7,6 }, +{ 1,22,36,105,170,0,86,2,31,28,239,64,55,5,10,98,9,44,127,95,654,67,301,143,13,12,49,23,320,141,83,21 }, +{ 15,515,700,753,0,1,13,2,23,901,5,8,51,82,9,180,457,4,7,12,3,6,10,120,341,141,22,898,197,351,115,260 }, +{ 1,39,274,98,100,265,190,30,438,310,166,223,88,96,909,31,264,625,530,9,382,812,21,252,593,0,254,539,44,131,23,778 }, +{ 18,212,167,118,363,1,447,411,146,60,271,16,781,121,647,9,621,562,21,478,664,68,815,5,354,98,48,101,24,446,777,463 }, +{ 24,28,22,0,7,1,2,16,14,65,35,49,158,95,109,159,55,105,10,18,124,9,67,5,239,149,12,289,108,68,21,424 }, +{ 105,22,131,272,286,98,55,239,1,31,320,9,127,327,36,185,28,374,86,219,0,64,187,44,578,164,224,913,535,115,601,13 }, +{ 22,31,28,301,127,98,44,0,105,1,512,395,9,293,109,299,95,338,239,125,242,116,36,320,55,841,900,685,599,23,13,763 }, +{ 2,1,58,29,5,14,52,46,186,334,45,155,151,50,400,75,38,69,502,61,48,227,223,7,163,17,262,67,549,21,70,113 }, +{ 7,107,135,232,97,14,2,92,66,16,172,192,278,387,298,356,38,35,448,52,46,43,60,29,20,126,324,526,357,359,64,5 }, +{ 20,43,104,426,173,7,560,414,707,784,319,81,0,861,422,819,38,74,715,52,376,97,879,32,330,22,49,64,66,95,192,526 }, +{ 104,74,636,66,204,0,355,81,222,25,29,319,145,784,20,65,90,4,174,194,7,64,6,746,138,173,750,715,91,43,192,32 }, +{ 0,9,101,35,68,39,65,28,252,124,67,154,364,336,100,166,30,1,289,55,149,346,16,114,158,88,439,24,429,22,570,194 }, +{ 57,14,4,231,236,585,176,59,369,23,361,13,719,51,300,342,12,457,56,3,62,38,202,401,34,46,2,322,11,215,210,507 }, +{ 1,2,15,3,141,0,515,5,33,700,13,64,77,180,6,128,753,10,4,269,102,202,11,7,134,197,352,120,117,318,12,291 }, +{ 5,1,21,202,13,32,48,23,0,61,259,22,494,120,70,49,51,18,137,128,465,12,178,115,2,453,403,141,58,3,90,450 }, +{ 141,205,4,72,59,79,245,11,352,94,152,76,247,216,21,188,452,217,497,12,89,37,111,339,588,77,64,875,864,115,358,464 }, +{ 15,515,700,753,0,1,2,13,5,4,23,3,8,341,365,51,115,10,120,457,6,141,77,197,31,7,165,9,202,450,961,260 }, +{ 5,2,50,14,58,38,171,46,29,1,45,186,17,52,155,218,48,281,61,487,54,36,67,21,328,334,151,227,760,114,400,133 }, +{ 457,120,70,125,318,64,23,48,795,291,202,761,751,415,77,846,269,758,21,237,96,260,391,165,87,1,128,5,221,13,137,763 }, +{ 13,23,51,33,4,40,117,102,453,64,153,196,0,77,15,11,12,475,1,65,134,10,515,22,21,14,700,59,403,141,2,753 }, +{ 229,152,57,266,452,381,432,12,313,184,99,471,17,4,62,339,157,3,129,59,128,11,369,37,77,38,40,123,5,497,188,257 }, +{ 49,28,109,22,159,9,272,95,105,131,55,35,254,168,39,327,169,0,1,286,175,374,347,158,420,67,36,194,312,424,627,346 }, +{ 5,2,61,29,45,58,80,311,1,17,209,227,52,243,106,869,454,151,592,496,48,334,14,155,6,186,46,171,75,21,255,667 }, +{ 244,44,110,141,260,30,269,352,839,131,574,228,373,276,1,406,219,717,217,137,253,224,120,93,36,31,567,116,661,187,341,88 }, +{ 12,99,79,139,11,453,196,51,277,474,111,23,542,37,94,188,33,13,401,775,40,961,313,102,4,339,153,485,629,134,300,431 }, +{ 16,35,9,0,68,24,149,69,67,18,1,114,65,230,71,7,103,133,50,167,212,118,101,191,140,64,399,28,124,283,55,565 }, +{ 88,30,274,435,131,613,190,100,93,829,166,1,187,795,530,127,382,957,960,160,31,137,466,264,39,800,406,254,28,473,521,219 }, +{ 167,16,18,118,212,24,60,71,101,68,191,9,375,411,363,35,0,1,589,199,302,21,447,55,146,126,92,271,647,121,562,48 }, +{ 64,141,86,177,77,128,147,597,304,95,269,102,275,4,352,49,120,5,372,194,465,13,588,237,947,216,202,180,612,751,107,534 }, +{ 18,65,90,403,523,289,240,214,194,102,701,475,202,217,283,862,389,51,33,0,494,421,453,817,84,64,847,899,352,13,23,437 }, +{ 13,51,23,202,5,12,21,128,15,115,0,1,141,120,64,32,4,2,515,403,165,457,3,10,700,99,453,318,719,450,308,401 }, +{ 98,223,393,31,1,271,834,791,167,44,202,64,93,697,5,116,77,125,450,446,212,18,541,293,51,120,195,132,284,13,807,765 }, +{ 15,515,700,753,4,11,23,13,40,51,82,165,0,110,93,33,141,64,120,5,10,77,3,102,180,32,202,125,8,197,31,21 }, +{ 15,515,700,753,0,1,13,2,901,23,5,341,3,51,82,8,4,180,961,9,115,10,12,6,898,7,351,141,134,22,31,120 }, +{ 234,416,77,5,315,639,325,202,147,198,113,49,450,61,455,142,0,21,22,342,329,494,178,58,102,427,318,230,13,120,43,470 }, +{ 60,146,16,18,156,126,121,271,199,360,132,24,167,0,640,10,71,522,21,92,5,340,107,354,118,150,22,195,446,35,28,212 }, +{ 4,361,11,14,56,368,377,161,27,12,300,77,59,200,17,554,202,33,40,494,495,21,210,80,757,25,128,23,19,38,444,53 }, +{ 141,82,217,351,15,352,120,1,180,260,515,64,854,36,700,317,752,372,13,269,77,753,922,21,349,23,202,110,93,137,51,373 }, +{ 15,515,700,753,77,13,0,1,23,33,102,2,51,4,3,5,291,217,10,9,450,120,341,7,317,6,11,117,115,8,260,180 }, +{ 15,515,120,13,700,23,77,141,1,260,0,753,180,51,137,202,115,365,110,291,217,5,128,9,21,341,197,269,2,450,317,165 }, +{ 174,6,348,85,138,74,280,204,66,233,192,355,289,65,81,580,636,353,25,91,104,343,673,214,64,95,42,712,792,32,194,90 }, +{ 152,497,452,59,4,216,11,79,94,77,128,188,269,339,588,33,76,529,318,32,141,471,12,202,111,21,5,51,37,90,72,177 }, +{ 417,499,10,141,253,244,110,559,8,564,180,260,728,120,352,638,642,341,951,206,143,752,901,93,137,661,922,373,44,31,811,197 }, +{ 13,77,23,33,4,51,0,102,128,59,141,40,64,115,177,10,137,22,202,2,7,11,90,1,117,180,269,14,49,6,134,3 }, +{ 1,2,22,0,36,5,67,50,14,28,12,86,38,46,83,168,194,65,103,114,49,7,10,95,21,69,23,24,128,51,55,13 }, +{ 17,106,119,207,255,306,742,378,84,62,136,45,3,5,240,80,61,56,209,383,311,790,655,32,2,440,76,151,58,29,179,263 }, +{ 3,128,1,141,2,202,33,5,64,15,0,515,102,13,269,10,700,180,134,51,120,6,77,318,23,137,17,117,753,197,82,153 }, +{ 514,38,377,328,11,57,41,248,880,266,556,4,152,361,471,757,485,403,305,102,3,211,313,99,457,130,12,14,157,40,23,54 }, +{ 68,0,167,101,9,118,264,520,16,18,21,478,562,1,124,212,100,936,664,777,191,88,806,154,48,24,759,604,35,252,265,65 }, +{ 230,689,699,213,466,352,217,831,30,443,418,144,854,201,840,855,1,251,203,317,530,957,96,93,822,539,36,752,351,137,83,800 }, +{ 33,77,102,117,15,82,13,134,23,64,0,515,120,153,51,4,40,128,700,260,202,141,196,22,753,11,351,10,1,326,95,269 }, +{ 11,40,33,51,117,13,542,328,14,134,38,153,23,12,485,231,102,54,775,37,3,377,111,139,211,4,457,403,369,475,99,719 }, +{ 33,64,77,128,141,2,1,202,102,13,23,117,0,15,3,153,51,134,10,40,6,5,515,269,137,180,318,165,700,7,196,753 }, +{ 15,515,700,753,4,1,5,11,13,21,33,180,93,141,64,2,23,77,82,3,0,102,32,40,352,341,10,197,98,110,117,901 }, +{ 1,2,14,67,50,46,38,24,103,83,0,5,36,28,29,133,114,96,65,52,18,75,54,108,22,7,238,58,160,9,361,69 }, +{ 258,201,276,137,160,860,116,261,295,843,567,144,131,44,187,268,943,219,284,31,202,935,141,98,662,203,127,96,36,93,224,1 }, +{ 7,2,14,16,46,87,75,52,92,278,29,38,140,70,1,5,35,294,24,262,135,69,171,172,58,409,112,60,50,66,97,12 }, +{ 13,23,0,2,51,1,33,4,115,10,15,141,77,3,5,180,217,515,9,7,64,11,700,6,102,40,197,22,317,753,165,202 }, +{ 74,145,6,66,25,204,42,29,222,337,138,26,7,525,192,174,746,287,544,135,415,2,609,632,112,64,87,0,85,45,712,396 }, +{ 77,33,102,15,217,13,23,141,202,515,51,700,291,4,269,753,317,180,21,64,318,115,128,0,275,2,352,196,3,5,137,11 }, +{ 187,219,258,871,44,442,160,574,137,224,908,116,839,131,36,926,276,201,93,228,202,860,31,613,144,531,406,1,902,30,190,318 }, +{ 1,372,141,5,21,77,225,744,96,30,23,349,13,291,269,284,69,442,459,144,303,839,217,622,160,330,260,48,120,410,189,352 }, +{ 66,222,2,74,29,87,135,6,7,145,52,25,294,337,226,172,138,331,42,70,97,112,26,1,632,192,43,5,415,609,461,353 }, +{ 45,17,106,209,5,2,21,29,48,207,3,186,243,155,255,263,454,119,400,496,270,14,290,62,425,1,171,32,659,52,38,56 }, +{ 93,88,141,120,30,213,260,373,100,717,459,82,110,1,166,450,180,321,217,372,36,269,131,225,22,352,326,466,473,187,244,410 }, +{ 266,57,152,381,313,471,12,229,99,369,339,62,157,3,4,37,77,38,188,17,11,162,40,184,129,59,475,775,128,452,403,453 }, +{ 217,352,317,141,752,15,180,515,372,365,700,341,753,349,77,21,291,1,115,244,64,120,13,98,269,82,5,498,864,351,23,144 }, +{ 14,514,369,102,403,377,51,719,880,153,23,13,457,11,485,4,401,12,328,453,33,40,117,57,629,38,730,236,134,670,361,961 }, +{ 107,7,172,14,92,135,2,359,60,314,46,16,126,278,232,150,279,32,38,392,298,5,35,97,24,192,259,288,330,52,356,312 }, +{ 0,4,25,13,59,90,65,23,26,19,18,12,5,216,91,51,389,33,77,11,22,85,27,81,21,177,746,45,42,194,37,123 }, +{ 5,49,315,202,416,77,455,639,450,21,197,137,350,13,408,0,329,318,494,344,61,402,64,509,347,120,113,48,95,713,308,401 }, +{ 130,47,381,390,59,90,200,214,289,6,65,472,29,64,874,648,50,751,624,26,52,32,4,194,875,714,85,249,247,33,881,19 }, +{ 51,23,453,13,719,12,457,165,37,730,99,4,386,197,401,17,11,2,3,15,5,961,475,6,515,64,54,700,32,115,0,403 }, +{ 15,515,1,13,700,2,23,0,753,5,3,180,51,4,165,12,141,21,197,457,7,115,6,9,352,10,120,202,8,341,11,77 }, +{ 0,9,1,67,35,28,68,16,24,65,18,69,50,114,103,12,22,13,5,101,2,96,23,83,149,21,39,55,7,175,433,124 }, +{ 28,105,22,0,1,320,170,9,49,301,109,95,127,31,98,55,65,35,2,24,168,159,36,713,16,740,13,338,21,44,512,23 }, +{ 13,77,4,51,23,33,102,202,128,59,40,0,64,141,117,403,115,11,15,318,153,269,22,515,475,134,10,494,177,1,90,210 }, +{ 13,23,0,51,77,33,2,141,4,10,1,64,115,102,3,6,22,15,217,11,180,7,40,515,165,202,177,9,269,128,700,5 }, +{ 456,116,492,8,949,268,867,391,203,51,499,13,719,386,31,791,457,918,125,10,23,93,479,685,417,0,22,338,506,551,870,730 }, +{ 17,237,45,180,106,62,32,64,115,41,136,498,255,21,197,129,241,13,3,227,23,352,165,752,350,365,449,155,4,546,476,38 }, +{ 1,15,180,515,0,2,341,700,901,352,4,141,13,3,752,5,753,217,317,115,365,23,197,21,51,165,31,6,269,202,77,7 }, +{ 205,141,216,269,497,4,588,76,59,152,128,452,79,77,875,11,72,94,188,217,352,12,247,37,90,64,32,1,474,23,947,372 }, +{ 64,247,217,237,317,180,752,115,349,141,498,13,437,304,23,372,352,164,579,291,33,864,177,197,0,490,72,10,482,77,269,51 }, +{ 2,1,0,13,15,141,3,77,5,515,64,33,23,180,6,700,4,117,217,7,10,11,102,165,753,197,115,134,40,352,12,269 }, +{ 11,40,38,328,33,542,12,313,41,339,23,157,377,117,369,51,471,99,775,485,13,305,457,57,14,475,37,248,4,54,188,719 }, +{ 33,77,102,40,13,23,0,51,4,128,64,202,117,141,22,196,153,10,134,15,59,269,1,137,65,11,403,318,453,86,515,177 }, +{ 472,80,34,250,495,161,17,14,469,176,128,4,389,106,283,436,216,527,3,297,483,177,53,56,231,194,119,84,719,57,255,59 }, +{ 317,352,180,141,217,752,115,341,365,244,1,269,202,901,253,15,21,498,372,4,137,515,13,2,700,318,5,197,23,143,753,349 }, +{ 9,39,101,18,265,100,333,520,252,16,0,329,593,1,553,364,68,167,310,30,121,254,118,158,363,166,60,604,272,24,286,404 }, +{ 15,515,1,180,700,901,0,2,753,341,752,4,3,13,115,365,317,5,23,197,141,217,165,352,6,22,36,9,137,51,7,10 }, +{ 131,39,9,829,166,613,578,827,1,30,716,254,100,98,31,224,0,406,228,310,616,219,44,846,127,190,938,96,265,371,856,438 }, +{ 17,64,62,106,141,751,136,292,32,129,352,41,38,476,86,128,214,237,5,177,123,209,217,45,269,954,162,710,180,3,90,4 }, +{ 25,42,235,65,650,736,605,6,630,85,123,343,233,256,26,122,63,389,141,249,416,444,368,194,19,108,138,174,90,0,544,511 }, +{ 184,229,152,57,266,432,497,452,17,381,619,257,313,12,4,205,59,3,99,471,157,128,5,129,339,369,77,11,32,45,202,2 }, +{ 137,202,160,860,141,30,93,567,36,276,295,261,131,39,9,964,201,843,1,98,800,318,116,22,943,187,10,219,206,44,269,535 }, +{ 0,493,125,64,49,9,279,10,35,18,93,55,293,31,14,13,194,165,325,48,22,132,21,107,98,389,44,581,342,259,174,137 }, +{ 15,515,700,753,4,33,13,77,23,5,51,32,102,40,93,11,349,141,21,8,82,202,64,31,110,10,117,0,1,44,3,318 }, +{ 110,253,854,811,352,141,244,951,180,642,661,384,498,143,752,317,911,10,269,206,559,351,261,120,902,533,922,959,365,160,332,217 }, +{ 2,29,70,1,75,52,6,220,26,112,145,331,74,163,19,69,38,324,46,58,14,5,25,21,278,223,50,307,66,7,67,409 }, +{ 13,23,77,33,51,4,64,141,115,102,0,2,128,177,40,11,202,10,6,180,7,15,269,1,32,217,59,22,291,3,137,515 }, +{ 340,897,691,478,658,264,914,382,100,812,363,1,724,156,166,698,88,521,39,404,682,447,296,96,303,411,30,909,9,274,656,772 }, +{ 9,18,310,101,265,159,326,120,105,158,33,363,77,195,51,55,13,39,354,132,23,7,28,639,16,137,98,1,252,272,709,49 }, +{ 57,313,471,12,99,369,157,339,266,152,38,37,475,453,328,775,11,40,59,188,77,514,401,403,342,4,139,33,377,51,229,14 }, +{ 16,7,24,14,35,140,60,92,18,69,71,2,189,1,46,230,108,388,150,38,21,172,278,67,246,267,50,309,236,135,451,0 }, +{ 206,417,93,940,959,473,499,203,8,137,559,728,31,202,44,120,450,141,10,260,116,564,22,326,269,318,268,244,0,1,253,638 }, +{ 15,515,700,753,1,0,13,2,23,4,3,51,5,217,7,77,341,115,8,9,10,33,6,180,317,349,291,120,11,165,457,901 }, +{ 1,2,5,14,48,21,290,32,50,45,38,46,263,207,155,72,76,29,17,408,425,171,89,52,7,0,292,449,3,227,513,428 }, +{ 121,132,354,167,271,223,146,98,18,463,1,668,446,195,407,60,212,447,781,48,360,363,411,522,156,393,807,9,21,16,293,13 }, +{ 131,578,105,371,219,224,716,616,187,49,9,254,737,159,385,98,258,127,272,761,0,916,623,910,28,286,39,31,22,518,924,242 }, +{ 302,467,97,6,273,1,24,484,124,51,36,18,2,398,453,421,523,69,7,23,13,403,386,150,66,0,298,65,426,165,22,158 }, +{ 30,190,530,88,1,100,778,539,625,274,382,410,96,731,960,39,795,321,9,131,264,144,840,748,44,166,669,957,36,31,435,228 }, +{ 141,1,2,128,64,33,15,202,3,0,180,5,13,77,515,134,269,102,197,700,10,137,318,6,120,165,753,352,4,82,23,117 }, +{ 44,201,567,116,131,224,295,662,489,268,219,31,434,144,187,276,110,384,93,261,699,137,36,442,120,1,613,30,228,64,141,244 }, +{ 12,15,51,23,515,37,99,13,700,0,10,117,753,38,165,82,134,120,11,453,197,64,115,569,1,629,401,22,457,474,110,153 }, +{ 7,135,2,92,172,14,66,140,38,52,97,46,29,74,16,324,278,226,6,87,1,571,262,5,357,232,35,380,69,314,24,330 }, +{ 125,386,23,963,949,60,51,391,165,221,13,197,118,21,719,193,541,421,517,150,393,7,401,453,308,5,791,551,326,558,48,173 }, +{ 6,85,42,25,138,222,174,235,280,256,525,289,26,214,64,746,90,32,544,65,204,19,66,337,355,95,348,415,74,29,5,312 }, +{ 1,14,5,50,2,67,24,0,46,69,48,21,58,103,16,12,18,38,54,96,83,7,502,45,36,181,35,9,430,28,10,155 }, +{ 811,351,642,180,951,752,110,638,253,10,82,352,197,341,365,564,499,854,873,55,9,417,282,901,244,22,559,143,206,141,28,898 }, +{ 23,13,51,15,12,453,403,165,4,515,115,719,475,457,700,523,2,21,0,99,202,197,14,5,386,753,128,401,37,308,33,117 }, +{ 120,13,23,77,141,1,15,93,217,82,260,51,137,202,110,515,21,180,165,5,128,102,64,351,291,700,269,352,326,203,177,0 }, +{ 1,5,0,22,12,2,36,21,10,23,86,13,28,51,9,128,48,14,32,50,7,3,96,137,54,4,202,49,37,65,208,323 }, +{ 219,98,23,127,301,51,258,308,170,910,13,165,22,105,293,616,125,242,276,401,201,395,964,115,55,284,31,374,327,206,512,900 }, +{ 64,180,80,165,5,237,2,250,34,58,297,61,197,17,22,29,186,498,231,445,247,3,752,311,95,32,483,153,27,45,115,469 }, +{ 13,77,23,33,0,2,1,64,141,51,102,10,15,3,115,40,180,6,515,128,7,22,269,202,4,217,700,5,177,117,14,165 }, +{ 15,120,51,515,13,450,23,700,202,153,196,753,260,64,128,141,730,4,326,386,21,523,33,318,5,457,95,32,403,1,77,269 }, +{ 2,1,5,29,32,45,207,263,14,425,58,72,76,21,7,408,48,46,52,186,17,292,38,6,61,89,476,50,155,720,119,3 }, +{ 15,515,700,753,4,13,11,5,1,23,33,21,3,141,32,2,40,180,117,64,269,202,102,197,0,165,120,51,341,352,153,12 }, +{ 76,5,214,129,2,123,45,710,17,249,618,460,179,32,1,257,205,519,90,207,245,184,162,61,769,209,292,106,6,29,14,128 }, +{ 1,15,23,13,120,141,51,515,202,21,700,165,0,180,137,2,5,77,128,93,753,260,269,197,326,33,110,352,82,102,318,48 }, +{ 7,2,135,14,29,87,66,52,97,172,70,112,5,58,46,337,92,16,20,43,1,38,232,155,74,294,6,461,409,151,262,32 }, +{ 574,187,384,926,860,110,258,434,269,531,141,244,160,261,253,116,699,959,940,717,533,36,219,31,902,661,871,295,201,352,10,260 }, +{ 156,354,296,1,182,586,64,379,340,937,850,698,31,48,98,44,120,18,163,23,30,658,195,125,77,284,223,291,774,481,96,39 }, +{ 250,80,34,472,17,495,176,469,33,194,64,483,4,297,141,14,161,27,53,667,56,833,73,527,585,231,106,51,84,814,2,59 }, +{ 97,7,81,140,66,92,172,192,24,298,43,6,74,69,314,426,462,14,501,16,21,508,60,189,267,232,230,104,48,20,135,330 }, +{ 31,44,116,144,268,393,492,434,367,489,127,98,918,0,384,9,22,206,948,105,93,203,1,456,332,940,299,28,137,49,293,125 }, +{ 15,128,33,3,13,51,141,1,202,64,23,2,515,120,102,0,5,82,10,700,165,197,269,153,403,110,753,137,196,318,117,12 }, +{ 31,98,127,9,0,105,22,28,44,512,293,395,299,1,242,49,685,763,320,599,125,116,109,276,284,95,870,159,23,456,36,900 }, +{ 7,24,124,1,6,97,2,69,14,18,23,92,21,67,66,16,5,484,43,20,118,65,36,22,28,0,51,140,13,71,29,150 }, +{ 1,64,442,303,284,349,202,141,622,67,154,447,260,44,652,429,9,335,237,919,197,98,167,33,682,269,547,77,863,411,340,201 }, +{ 1,15,2,141,515,0,700,13,3,180,10,753,5,64,77,33,4,6,7,197,102,269,165,23,134,11,352,341,291,349,22,120 }, +{ 99,139,12,453,196,277,775,40,475,33,23,401,215,51,11,14,77,111,313,130,38,211,37,266,129,15,339,153,719,3,369,515 }, +{ 33,77,102,4,23,128,13,141,202,64,51,0,40,59,269,115,117,137,153,1,318,11,10,177,15,134,22,90,196,2,403,32 }, +{ 7,2,14,58,70,112,16,5,87,38,46,52,6,128,135,1,32,21,155,29,66,64,0,97,92,186,172,294,13,23,20,37 }, +{ 15,13,515,1,700,2,23,0,753,5,3,4,51,10,341,115,365,180,11,33,317,77,6,7,217,12,197,165,117,9,64,102 }, +{ 2,1,14,29,75,69,67,6,52,46,38,24,103,220,83,25,70,87,262,74,96,267,50,366,26,16,226,394,357,66,108,19 }, +{ 9,105,18,39,1,0,16,557,101,272,252,890,326,49,265,21,137,100,23,938,13,310,159,5,31,24,254,51,30,128,202,132 }, +{ 80,209,45,61,667,17,6,106,5,2,151,29,483,255,454,833,27,311,112,19,738,378,1,58,113,26,25,469,119,887,32,64 }, +{ 13,23,51,15,5,1,515,0,21,2,12,141,700,165,202,115,753,32,180,4,3,197,10,120,457,9,269,128,64,341,7,33 }, +{ 99,12,453,277,139,157,369,474,339,51,38,23,37,196,188,401,775,111,11,313,328,475,153,266,4,471,79,40,33,629,102,14 }, +{ 7,92,16,232,97,140,126,14,60,107,66,35,298,387,314,104,246,462,441,150,0,38,24,2,172,357,230,330,5,633,22,289 }, +{ 13,77,23,202,318,141,33,4,51,269,102,177,115,403,137,2,40,494,90,11,342,128,31,117,21,32,7,12,64,134,14,10 }, +{ 13,2,0,23,141,1,77,3,180,33,6,64,15,10,115,51,4,5,217,197,7,165,515,102,22,11,700,269,40,352,177,14 }, +{ 15,515,700,753,4,11,1,93,13,5,180,110,82,21,120,23,2,33,10,141,3,165,197,102,901,0,32,341,117,40,153,12 }, +{ 15,515,700,753,1,13,0,2,23,4,77,51,3,5,341,291,7,33,6,115,10,9,8,217,11,177,120,180,102,165,197,365 }, +{ 20,43,198,325,173,904,104,234,66,147,77,319,416,422,97,426,5,0,7,450,861,202,712,725,2,32,639,376,38,324,945,315 }, +{ 105,0,9,28,49,301,170,1,127,159,22,16,31,98,512,623,24,109,158,395,35,68,371,65,713,55,2,242,293,21,44,18 }, +{ 213,88,689,466,230,30,321,435,699,352,217,201,795,831,144,854,1,443,96,539,530,840,418,251,855,190,93,100,669,31,957,662 }, +{ 130,453,47,196,4,57,14,59,236,711,51,153,730,77,412,381,23,202,108,128,361,13,283,117,11,719,200,46,34,78,210,2 }, +{ 1,2,5,14,0,50,36,22,38,46,65,67,12,86,114,28,103,29,208,7,10,128,21,83,218,23,96,54,194,6,133,51 }, +{ 6,26,74,19,165,453,14,730,1,125,197,50,29,51,138,357,13,2,108,391,70,719,46,457,47,500,386,262,112,23,235,52 }, +{ 9,10,376,20,43,0,49,18,30,120,2,33,325,104,501,470,77,788,725,102,523,39,858,5,904,414,174,55,137,37,342,13 }, +{ 15,515,700,753,0,1,13,23,51,77,120,202,341,82,5,4,9,260,2,137,141,128,115,351,901,8,180,10,197,21,450,33 }, +{ 105,131,272,578,9,49,371,219,159,616,286,320,224,187,716,98,28,22,0,623,127,258,910,737,385,31,239,347,254,109,424,95 }, +{ 457,51,13,23,961,12,719,99,453,15,4,515,165,401,629,3,700,11,17,14,2,37,753,41,57,569,38,45,0,33,5,32 }, +{ 202,120,5,33,318,77,450,102,1,260,403,128,494,21,165,13,269,12,326,23,342,523,402,2,817,64,15,141,125,82,457,475 }, +{ 141,269,352,217,180,64,349,137,202,160,317,15,372,515,700,752,318,753,244,13,437,291,165,864,22,237,5,82,954,21,77,418 }, +{ 70,29,2,145,74,112,26,6,75,52,19,66,632,1,87,220,5,135,163,287,307,25,226,7,58,396,294,278,113,409,69,151 }, +{ 82,351,317,15,752,180,898,352,141,901,515,341,10,700,365,1,753,498,0,217,253,115,55,854,33,5,143,32,21,160,36,197 }, +{ 39,9,310,254,0,30,101,49,252,272,100,265,105,455,159,557,190,333,286,688,18,166,1,158,709,16,625,627,31,131,327,329 }, +{ 2,58,29,5,1,151,186,52,70,45,7,549,14,75,112,400,113,155,61,46,227,163,311,315,66,6,307,27,17,220,287,74 }, +{ 141,217,13,21,352,23,269,77,180,115,317,64,202,15,349,137,5,51,165,291,318,752,372,4,0,102,33,365,197,32,341,125 }, +{ 68,35,0,9,65,101,149,124,24,154,175,16,28,7,67,1,18,189,114,398,55,14,345,39,118,133,69,2,230,429,71,283 }, +{ 66,7,29,2,112,52,20,43,97,151,74,192,135,5,173,525,337,45,145,58,415,25,14,32,644,70,544,226,222,21,6,580 }, +{ 31,125,44,22,116,299,242,55,1,170,64,36,479,870,456,685,10,599,558,0,268,506,28,740,23,903,492,164,393,206,2,86 }, +{ 188,11,79,12,99,377,94,33,542,339,40,474,111,37,4,51,102,453,139,775,13,475,23,961,277,471,134,57,431,266,115,117 }, +{ 658,698,340,98,296,303,1,31,850,363,156,919,44,774,586,385,120,77,82,10,223,30,354,291,23,914,478,87,260,163,48,13 }, +{ 15,515,700,753,82,4,1,13,901,33,197,11,5,10,23,165,2,0,180,3,21,77,51,120,365,115,217,40,117,102,32,401 }, +{ 15,515,700,753,4,11,5,13,1,141,3,180,23,202,21,2,269,64,165,33,40,32,0,318,120,128,12,197,117,352,51,17 }, +{ 91,6,233,85,370,718,81,65,25,256,63,343,42,74,235,123,138,511,397,249,26,194,650,355,64,87,544,18,90,643,66,214 }, +{ 23,13,202,51,21,120,1,5,141,128,450,64,318,403,15,137,260,33,12,48,32,31,125,494,269,102,165,515,77,2,197,14 }, +{ 180,317,365,341,752,217,115,352,901,482,372,498,1,141,15,253,515,244,2,700,0,21,13,82,23,4,579,351,753,291,269,77 }, +{ 13,115,197,341,9,352,468,237,64,498,23,165,22,509,901,546,482,180,28,569,317,51,365,873,391,95,86,217,49,837,752,706 }, +{ 13,23,51,1,141,5,165,202,21,120,64,125,180,15,2,33,197,115,128,32,260,269,12,82,4,515,137,7,318,93,0,700 }, +{ 214,289,90,174,874,6,138,280,65,81,64,85,355,751,194,233,312,348,835,91,0,32,343,636,249,29,875,288,519,104,247,74 }, +{ 15,515,700,753,4,5,11,13,1,33,23,21,2,3,102,32,141,77,180,117,31,64,0,40,134,196,120,352,12,44,197,6 }, +{ 33,15,13,515,117,23,700,217,134,753,0,51,153,77,141,2,4,64,196,1,3,180,10,115,5,102,6,11,22,202,165,7 }, +{ 15,515,700,753,33,4,77,102,1,40,13,117,11,115,134,5,21,153,23,217,3,32,2,317,120,196,180,141,51,12,59,260 }, +{ 15,515,700,753,13,0,1,23,2,217,51,3,4,5,8,317,115,9,341,10,202,180,6,365,7,82,457,22,120,901,33,291 }, +{ 7,2,135,20,97,14,66,52,337,673,192,29,43,355,353,5,16,294,107,376,147,226,331,560,64,470,222,104,415,32,4,324 }, +{ 195,132,142,167,146,77,363,271,121,354,202,120,647,178,786,212,687,0,101,878,16,522,60,5,450,411,35,55,98,639,259,318 }, +{ 202,77,20,0,318,66,104,128,102,269,177,43,33,7,216,291,494,5,2,342,74,173,97,112,450,22,337,10,234,52,64,678 }, +{ 107,362,612,356,359,97,414,43,259,20,392,7,298,147,819,683,465,173,729,660,319,14,5,779,581,595,246,35,501,92,0,230 }, +{ 6,165,14,453,13,51,19,23,386,457,74,391,308,2,26,401,47,758,603,108,719,366,1,29,309,730,324,197,133,70,115,867 }, +{ 179,72,205,180,247,245,4,490,352,59,317,152,79,498,94,217,148,76,752,864,11,216,141,405,89,452,197,111,497,188,37,21 }, +{ 107,7,298,314,14,359,32,392,232,279,172,97,60,581,387,126,121,0,534,493,356,92,441,95,13,21,35,147,22,5,16,362 }, +{ 156,271,354,586,360,132,591,195,121,18,340,1,5,13,21,48,668,446,23,463,296,658,60,55,407,698,146,70,626,51,163,24 }, +{ 13,23,51,4,0,12,457,15,11,453,2,515,5,1,99,10,115,165,700,475,401,403,3,961,40,14,37,753,719,32,64,569 }, +{ 48,125,21,165,13,221,23,763,423,508,197,5,98,92,193,16,441,386,64,314,293,457,391,140,49,60,102,693,683,51,35,867 }, +{ 202,77,120,450,5,318,1,494,0,195,18,132,523,403,326,604,354,260,121,576,203,167,234,817,682,49,35,615,21,20,13,102 }, +{ 39,9,166,30,0,101,158,68,404,190,333,274,252,310,88,100,49,28,344,35,21,22,419,131,438,1,16,65,530,694,124,10 }, +{ 15,515,700,753,110,4,1,11,165,180,93,13,82,5,2,197,33,120,0,3,10,23,21,115,901,217,341,77,317,51,32,117 }, +{ 2,29,1,14,6,52,5,46,50,26,70,19,103,58,38,67,96,262,516,309,218,133,108,27,75,17,112,114,24,487,331,83 }, +{ 120,77,15,13,1,141,260,23,515,217,110,51,137,700,317,202,165,291,180,21,753,128,0,177,326,93,450,82,64,269,197,5 }, +{ 255,59,554,297,183,56,33,444,108,358,123,196,269,122,77,153,57,177,117,730,19,467,605,130,128,50,275,4,291,475,134,133 }, +{ 13,23,51,12,153,14,117,120,165,134,99,401,38,453,15,128,197,719,64,515,475,403,37,33,196,700,40,125,5,0,54,2 }, +{ 64,33,174,348,95,108,467,554,56,0,25,306,233,6,63,511,343,120,13,85,29,561,543,707,319,180,899,355,77,49,256,18 }, +{ 120,260,51,23,77,15,202,1,93,82,141,450,13,326,515,137,21,5,64,33,110,700,128,165,318,203,269,102,351,753,197,125 }, +{ 15,515,700,753,4,13,11,1,5,21,23,2,33,64,3,180,32,141,22,102,77,0,10,93,82,352,117,40,341,31,165,6 }, +{ 15,515,700,753,341,13,23,141,33,1,0,217,4,77,180,10,82,351,51,137,5,64,9,317,21,11,102,40,260,202,854,115 }, +{ 105,272,131,22,327,286,28,239,320,9,109,578,219,49,98,224,95,159,538,371,616,127,187,64,713,55,0,170,168,258,716,623 }, +{ 16,18,68,35,24,60,71,118,92,126,0,9,101,191,7,55,154,175,212,14,167,150,302,28,375,1,107,124,346,273,21,108 }, +{ 20,147,43,470,376,142,904,178,427,798,0,595,198,325,858,319,61,202,173,97,5,422,14,22,107,259,32,49,887,77,414,392 }, +{ 13,23,51,12,33,15,99,64,128,515,453,202,117,153,37,102,700,40,134,196,120,0,2,753,141,14,38,3,82,403,77,21 }, +{ 383,17,62,136,84,119,56,440,3,504,240,80,378,129,123,548,106,128,4,11,14,555,162,32,184,361,59,64,205,5,469,57 }, +{ 70,1,48,652,5,638,846,888,21,349,269,260,340,562,767,761,163,883,774,141,125,518,591,0,23,9,87,13,371,303,622,31 }, +{ 66,135,6,97,74,278,69,7,14,324,267,172,2,140,462,1,357,38,808,550,92,841,189,29,16,25,298,87,75,204,24,335 }, +{ 51,23,33,13,102,40,12,128,64,77,10,202,0,196,117,4,14,99,134,453,65,153,11,475,139,403,22,141,86,2,21,15 }, +{ 88,100,264,166,274,435,772,1,382,921,96,478,30,438,639,909,897,521,190,466,960,410,9,144,530,418,31,329,265,691,778,93 }, +{ 62,440,136,56,84,3,504,548,555,383,4,17,129,128,507,361,123,59,119,162,14,57,152,328,161,11,202,495,184,27,80,215 }, +{ 911,617,332,959,206,141,253,244,282,384,110,120,10,260,352,143,951,811,269,373,160,417,93,531,728,203,434,940,137,55,36,717 }, +{ 120,15,260,141,77,1,515,82,700,351,33,23,450,13,110,326,64,217,269,753,203,137,102,5,165,21,51,291,93,177,373,128 }, +{ 15,515,700,753,0,1,2,23,13,51,5,9,82,901,180,8,3,4,120,6,7,141,93,12,197,341,10,33,115,730,64,125 }, +{ 7,104,97,107,356,232,66,560,298,289,14,707,38,568,359,64,20,0,65,324,22,214,92,32,192,5,387,43,712,90,172,95 }, +{ 6,1,2,66,67,14,74,24,108,29,69,83,458,7,25,38,135,103,36,150,451,114,52,594,75,65,380,18,267,602,19,278 }, +{ 13,23,51,12,115,21,202,5,457,15,4,1,64,719,0,403,2,3,453,165,99,141,401,128,32,515,10,37,523,197,120,700 }, +{ 57,59,4,11,412,381,77,53,421,291,250,368,99,14,27,369,803,283,23,108,403,19,339,210,0,401,12,444,236,40,361,736 }, +{ 15,515,700,1,0,753,2,13,23,5,51,180,3,115,6,7,457,4,9,8,12,82,197,165,141,901,120,719,33,64,21,22 }, +{ 64,95,180,247,929,146,90,126,197,32,237,60,288,165,316,92,5,13,77,7,217,955,522,22,16,314,132,4,317,10,312,86 }, +{ 15,1,120,13,23,515,0,51,700,180,141,2,5,202,21,260,753,165,137,33,77,110,197,128,326,7,450,4,102,9,269,12 }, +{ 14,2,16,46,1,7,24,69,75,35,38,50,29,220,52,140,267,67,18,54,70,309,5,60,92,189,171,87,71,163,58,0 }, +{ 31,98,127,44,9,299,0,276,293,284,116,49,935,599,105,22,456,201,28,1,39,125,242,137,371,144,131,492,159,272,51,395 }, +{ 6,27,151,53,573,445,297,113,26,73,436,19,491,250,396,315,45,112,145,58,614,881,25,34,611,200,17,80,70,5,138,631 }, +{ 32,693,81,788,90,804,403,56,494,21,84,397,202,65,18,77,64,681,214,725,523,784,526,33,102,825,240,0,115,241,817,91 }, +{ 24,7,14,2,18,16,65,0,108,149,28,69,1,71,154,36,124,35,67,140,189,429,92,68,66,22,55,118,302,150,9,6 }, +{ 0,68,9,35,65,101,189,212,114,67,124,69,1,154,149,39,230,64,252,16,88,702,103,100,18,336,28,329,520,83,30,755 }, +{ 5,2,186,29,61,45,17,1,52,48,58,171,155,227,80,209,311,21,14,46,50,106,243,513,334,502,496,38,3,6,32,592 }, +{ 15,515,700,753,13,1,2,0,3,4,5,23,341,11,10,33,6,51,165,117,153,7,180,12,365,901,77,569,197,115,64,9 }, +{ 13,15,23,515,0,51,1,700,4,2,753,10,3,5,12,77,33,961,165,457,197,11,115,9,22,102,40,403,202,21,14,59 }, +{ 15,515,700,753,13,0,1,23,2,33,102,5,4,10,9,3,51,115,77,7,6,341,12,11,217,40,457,196,180,165,8,523 }, +{ 166,39,30,274,190,100,333,438,530,310,88,252,0,9,539,265,1,656,404,101,625,131,778,254,31,455,676,329,724,158,21,23 }, +{ 734,148,94,308,431,115,37,89,111,413,79,468,197,629,341,474,569,12,13,873,179,401,11,4,180,23,205,72,59,365,134,51 }, +{ 539,228,224,219,816,190,30,258,871,840,669,93,406,530,957,187,160,531,748,137,131,88,863,36,728,839,44,213,352,116,202,466 }, +{ 393,791,125,801,730,551,386,23,31,175,93,98,51,13,144,788,126,203,21,345,116,22,949,110,575,165,326,44,0,4,60,221 }, +{ 13,23,77,141,0,4,51,2,33,115,64,1,10,3,6,15,11,102,7,217,180,40,515,22,128,177,202,9,700,269,165,5 }, +{ 2,29,7,70,52,14,1,58,112,46,75,5,171,163,87,220,307,151,186,334,38,66,155,16,69,135,278,45,262,97,6,21 }, +{ 88,321,213,100,230,435,689,466,1,382,30,352,217,699,410,96,795,36,921,752,190,141,144,180,44,831,317,83,443,31,840,251 }, +{ 363,411,101,520,354,9,195,668,132,156,447,1,905,364,18,23,765,664,146,5,360,13,121,96,98,31,252,39,100,759,264,551 }, +{ 13,23,51,730,12,719,453,457,401,475,5,21,403,2,0,1,15,4,3,899,99,32,165,11,515,308,197,115,6,961,700,523 }, +{ 72,76,89,12,37,4,308,179,38,528,90,431,54,205,148,184,401,57,152,474,23,59,51,245,428,11,32,99,405,316,257,21 }, +{ 376,20,43,147,470,173,97,595,107,319,414,142,819,5,729,178,858,7,427,32,426,104,14,0,392,362,259,61,230,77,560,246 }, +{ 202,141,269,494,318,137,51,128,403,4,217,96,77,5,64,177,291,180,15,352,102,10,33,349,2,317,0,341,120,515,21,453 }, +{ 77,202,33,128,102,318,494,269,13,0,117,23,342,291,403,15,134,51,153,141,177,515,82,137,196,700,203,64,22,351,753,4 }, +{ 253,110,951,352,499,811,10,854,180,638,244,559,642,752,564,8,141,143,417,341,901,260,206,197,922,661,93,15,498,373,165,911 }, +{ 141,13,23,180,4,217,5,1,269,317,21,0,2,202,115,51,352,77,3,197,64,341,318,15,291,9,137,93,32,165,515,33 }, +{ 9,0,18,252,16,101,68,39,24,118,35,109,158,329,28,167,60,364,333,265,49,100,22,419,553,55,1,677,71,7,212,159 }, +{ 28,109,9,39,0,158,49,22,168,35,55,175,1,65,67,185,194,159,289,95,272,114,30,105,86,584,36,169,254,2,83,24 }, +{ 15,515,13,700,1,753,2,23,0,3,4,5,33,341,11,51,6,10,197,115,901,180,77,40,102,12,365,165,141,217,7,317 }, +{ 173,693,104,422,5,18,61,32,102,0,20,13,784,560,33,66,397,526,49,207,29,25,510,707,65,6,11,344,21,263,81,77 }, +{ 23,13,386,51,308,801,719,221,401,949,21,730,165,421,102,115,125,33,341,670,468,117,770,1,120,6,197,14,403,97,67,958 }, +{ 0,49,105,16,28,24,159,9,158,320,1,68,35,239,170,18,109,7,55,65,2,95,301,124,347,14,21,154,22,127,286,31 }, +{ 2,5,1,207,45,29,32,58,76,61,6,263,292,655,72,14,17,476,7,119,52,306,70,64,21,90,186,214,106,38,3,790 }, +{ 21,6,125,49,13,64,715,66,115,95,197,33,22,32,204,165,56,278,0,408,241,120,4,808,681,350,263,85,81,571,135,509 }, +{ 612,427,325,107,202,5,376,49,64,392,403,470,21,147,31,788,494,14,362,465,858,98,20,804,518,43,845,318,125,97,725,534 }, +{ 32,21,76,72,2,1,14,5,241,449,89,38,350,221,155,48,50,292,37,46,45,90,270,54,17,179,214,12,148,430,476,413 }, +{ 24,0,28,16,7,124,35,154,14,149,65,18,9,68,55,108,175,71,2,1,22,109,92,67,484,336,118,69,302,398,570,420 }, +{ 1,5,14,2,48,50,38,67,46,21,0,54,45,270,281,12,24,32,155,96,513,103,290,83,61,58,36,17,37,72,69,181 }, +{ 13,961,569,197,37,15,23,474,515,94,148,111,12,165,629,341,700,79,901,401,51,405,753,10,134,4,115,734,873,11,89,117 }, +{ 33,23,102,51,13,40,77,128,64,202,141,15,4,12,0,1,2,117,22,11,10,403,153,515,99,318,137,269,139,196,700,134 }, +{ 0,1,24,67,9,16,18,35,28,69,103,50,5,2,65,12,83,68,7,96,14,22,21,149,75,114,13,133,23,71,218,54 }, +{ 384,617,940,332,855,911,206,959,434,282,141,10,93,253,244,110,144,268,120,36,352,137,417,203,116,31,44,269,160,201,143,951 }, +{ 30,93,473,137,31,704,450,652,190,203,800,254,166,274,326,144,269,160,127,303,120,625,88,848,110,435,77,521,349,131,340,744 }, +{ 53,27,73,26,19,250,297,200,25,630,17,6,611,122,34,42,714,235,472,65,436,14,80,684,690,106,45,113,680,108,64,4 }, +{ 15,515,1,2,700,0,753,3,5,141,180,4,13,77,33,10,217,6,7,134,11,352,197,64,165,341,317,23,12,115,102,40 }, +{ 254,530,39,613,688,221,30,31,438,190,228,960,1,44,141,21,180,406,23,166,9,202,13,96,137,48,131,829,317,269,393,51 }, +{ 9,39,28,35,30,166,158,36,0,175,101,346,364,67,49,68,168,420,88,1,194,131,100,352,55,83,190,64,137,570,86,65 }, +{ 62,56,3,548,555,507,440,161,34,4,215,136,162,514,361,527,17,14,211,130,328,11,383,123,84,183,38,57,184,152,205,494 }, +{ 92,126,107,7,356,493,97,279,359,298,16,246,35,60,14,441,362,121,43,423,5,132,392,20,508,230,199,146,232,173,150,414 }, +{ 15,82,141,515,291,922,349,700,217,260,372,120,351,93,77,753,318,352,373,854,1,326,269,21,13,102,144,202,64,23,203,137 }, +{ 141,217,352,115,180,13,269,317,752,77,23,21,341,197,5,372,244,291,9,64,51,102,4,1,365,2,165,33,3,48,237,351 }, +{ 78,47,390,19,130,453,108,27,711,813,730,444,412,283,196,690,123,14,128,26,250,389,650,236,200,65,51,4,34,183,297,73 }, +{ 34,250,297,80,472,64,495,17,311,3,148,45,667,61,176,53,243,27,90,161,469,141,483,151,62,128,29,4,58,56,5,231 }, +{ 51,23,33,13,551,77,102,326,421,21,523,120,5,899,453,692,202,153,308,615,115,958,450,401,791,68,221,93,475,18,403,4 }, +{ 98,223,393,363,411,1,478,834,664,156,284,691,447,791,914,293,354,724,697,9,807,541,759,51,18,421,48,264,948,586,195,848 }, +{ 7,14,107,232,16,92,2,60,46,5,359,121,24,526,220,620,135,1,172,21,126,314,132,77,18,75,32,278,12,23,52,38 }, +{ 32,76,2,1,21,72,241,14,5,48,292,89,476,45,720,270,179,90,17,214,148,38,50,29,129,155,350,46,290,227,123,464 }, +{ 15,515,700,753,13,23,33,77,51,4,102,0,32,202,1,11,128,82,117,141,40,5,110,8,3,90,137,21,10,318,403,165 }, +{ 66,6,69,2,1,74,14,135,278,267,380,24,29,97,67,38,103,75,7,388,324,25,52,150,87,83,189,357,335,108,204,172 }, +{ 152,4,339,59,79,471,188,11,77,94,128,33,529,377,12,111,102,202,452,402,216,99,13,542,51,40,474,37,64,291,23,961 }, +{ 15,515,700,753,1,0,196,13,33,2,77,5,23,102,3,10,9,7,217,4,6,153,117,177,14,457,115,12,40,730,11,134 }, +{ 17,209,45,106,207,5,255,119,62,2,61,3,263,742,306,655,425,378,32,56,29,136,84,80,311,58,186,240,243,383,14,21 }, +{ 120,260,450,15,1,23,817,13,515,523,326,5,700,51,82,31,202,64,21,753,318,93,32,269,98,33,351,77,102,125,457,165 }, +{ 116,492,268,93,23,206,203,0,551,918,13,51,8,22,417,940,120,10,499,31,949,791,125,523,165,473,341,730,421,959,401,391 }, +{ 15,515,700,753,165,13,0,1,197,23,4,82,120,2,180,12,260,719,8,3,386,117,5,523,901,11,341,51,10,9,141,351 }, +{ 14,24,69,7,2,66,108,1,67,6,36,398,18,267,150,97,29,38,83,149,65,74,28,0,189,71,388,16,273,124,46,22 }, +{ 330,96,523,335,367,662,141,839,1,922,372,615,244,717,269,443,418,352,403,692,217,854,752,180,36,64,498,576,349,201,98,284 }, +{ 184,90,257,205,245,229,57,152,769,17,524,5,32,497,45,432,619,2,452,266,4,106,1,21,179,59,76,3,460,292,381,128 }, +{ 7,14,16,2,46,5,70,107,87,13,58,307,92,32,38,23,202,0,172,24,18,21,60,128,77,35,20,10,9,4,171,112 }, +{ 7,66,140,16,14,92,97,69,267,172,189,24,380,2,35,60,298,451,230,135,314,74,150,71,38,357,6,330,67,423,21,443 }, +{ 121,167,354,132,18,446,147,101,212,146,407,16,55,35,647,191,20,271,199,68,60,259,463,107,9,126,363,7,195,43,14,411 }, +{ 76,90,179,32,205,21,184,460,257,288,45,245,316,5,57,152,241,2,358,1,229,72,524,148,48,769,17,4,12,38,14,720 }, +{ 147,259,178,878,427,465,581,198,786,798,142,534,325,929,20,362,35,132,107,376,43,5,279,77,49,146,70,202,590,771,33,14 }, +{ 473,93,450,778,141,30,855,466,144,203,330,530,88,523,459,372,201,617,839,704,254,321,934,326,39,36,82,717,332,213,559,403 }, +{ 523,475,51,899,730,453,23,719,403,33,457,13,421,386,4,120,117,196,102,153,15,801,450,817,515,260,202,11,700,99,165,125 }, +{ 15,1,13,515,0,2,700,5,23,753,4,3,341,317,10,115,180,11,33,64,217,77,117,165,197,7,6,365,9,141,102,134 }, +{ 19,4,119,40,33,202,27,84,102,56,77,73,504,485,26,494,757,63,862,59,23,300,25,12,128,11,5,13,342,880,469,6 }, +{ 32,20,2,13,5,21,23,6,12,38,43,29,64,7,95,51,61,207,48,147,90,178,17,182,49,0,115,202,52,362,37,22 }, +{ 339,188,11,79,4,94,377,12,99,111,542,102,37,33,474,51,471,40,453,152,77,13,59,403,342,23,117,57,475,134,128,38 }, +{ 34,128,283,176,495,231,318,432,503,275,529,527,161,53,3,202,56,291,585,469,73,17,14,412,57,27,80,245,250,381,402,51 }, +{ 15,515,13,700,1,217,141,120,23,180,753,115,365,51,317,341,77,260,0,291,110,137,202,5,21,269,64,36,349,2,4,10 }, +{ 13,15,961,515,700,753,4,12,2,457,3,11,197,51,37,569,115,23,5,0,99,10,1,134,6,111,165,33,72,40,38,79 }, +{ 15,515,700,753,13,1,0,2,23,33,5,3,10,4,9,115,7,102,6,51,12,217,77,11,40,457,569,341,117,317,14,719 }, +{ 5,76,2,32,292,214,45,1,129,519,123,179,90,710,17,29,460,72,14,207,21,249,58,205,464,263,618,48,6,245,3,257 }, +{ 72,76,32,4,21,12,38,23,99,54,89,3,14,17,51,57,11,90,13,488,179,2,59,148,45,37,5,115,401,1,10,421 }, +{ 98,223,393,1,834,264,284,791,724,293,478,772,697,909,363,682,905,447,541,821,411,51,421,9,807,48,765,31,730,96,386,410 }, +{ 341,13,509,8,23,638,165,901,762,10,569,242,391,197,873,642,506,499,629,961,15,180,116,456,206,546,417,1,338,457,515,867 }, +{ 1,2,5,50,14,38,46,114,0,36,29,22,218,65,86,96,137,21,133,285,12,10,323,181,17,58,51,23,67,7,28,6 }, +{ 481,878,202,13,5,23,182,32,269,21,1,318,77,142,557,494,141,33,640,137,70,291,2,51,260,415,929,403,120,58,4,259 }, +{ 15,515,700,753,1,4,13,0,2,5,341,3,11,180,134,12,10,317,197,365,33,21,23,165,117,6,77,7,217,37,32,498 }, +{ 25,119,19,6,26,42,27,17,4,790,45,814,2,469,483,84,122,1,0,33,32,128,76,80,611,113,73,56,5,240,202,77 }, +{ 14,2,7,1,24,0,65,6,16,69,67,22,124,28,108,5,18,36,86,10,38,46,66,398,289,168,12,83,21,23,610,13 }, +{ 51,23,128,13,15,202,12,120,33,64,141,82,10,515,0,403,700,3,1,99,117,269,153,165,753,5,318,197,102,260,2,137 }, +{ 16,35,24,0,9,18,7,1,68,69,50,71,103,65,67,189,133,23,28,13,60,537,149,335,75,21,64,5,114,2,12,14 }, +{ 754,803,133,576,880,543,2,1,657,50,14,38,46,5,29,67,218,36,58,171,52,96,24,103,775,0,114,83,181,54,65,45 }, +{ 21,32,5,3,2,17,14,72,76,1,12,23,38,51,4,54,10,0,89,13,99,137,45,36,421,115,543,11,22,128,221,48 }, +{ 434,384,268,144,855,940,617,206,332,116,93,911,959,282,203,137,141,489,44,120,10,110,244,36,98,31,269,253,367,417,160,9 }, +{ 15,2,1,0,13,515,5,700,3,23,180,217,141,10,753,4,117,6,77,33,64,7,11,197,352,317,341,134,165,115,12,9 }, +{ 2,113,6,25,1,0,29,4,7,833,5,45,32,61,128,19,77,151,74,145,64,42,14,210,655,106,59,177,27,17,21,738 }, +{ 116,268,918,203,551,31,8,692,206,791,403,499,417,93,940,421,0,23,22,120,13,523,44,51,299,473,959,1,10,475,202,125 }, +{ 107,126,132,612,362,279,20,146,259,493,199,121,590,43,660,147,35,376,939,60,941,534,683,5,0,953,16,7,49,649,595,470 }, +{ 15,515,700,753,13,1,0,23,2,33,77,4,3,51,5,102,115,10,9,341,6,7,11,342,217,12,120,180,40,317,141,8 }, +{ 53,27,17,161,469,378,73,527,19,136,383,250,495,56,862,26,62,84,80,106,200,4,34,14,440,297,3,128,585,5,129,123 }, +{ 17,45,209,106,5,207,243,454,119,255,2,263,186,290,29,3,21,62,425,61,84,32,58,56,48,408,655,136,306,14,742,227 }, +{ 4,152,59,452,128,79,216,11,339,471,529,188,94,77,202,12,291,33,318,377,99,51,23,5,402,349,32,474,102,13,205,111 }, +{ 15,515,700,753,1,0,2,13,3,5,23,4,180,51,115,9,6,12,7,8,197,33,10,961,901,77,141,752,110,22,120,341 }, +{ 951,752,638,811,351,642,180,253,10,341,197,901,110,873,8,244,15,352,165,898,143,515,564,762,499,55,365,700,82,753,141,854 }, +{ 6,262,197,350,74,26,115,509,841,583,165,38,21,13,47,50,235,19,33,324,453,4,308,196,138,99,64,903,675,1,223,130 }, +{ 125,165,391,23,386,221,21,13,558,457,51,867,197,115,401,758,77,97,308,791,7,180,48,120,963,451,743,89,603,134,403,450 }, +{ 1,14,2,5,16,46,7,38,58,24,50,0,69,48,35,67,54,18,12,75,21,45,513,155,430,37,270,9,61,163,223,32 }, +{ 23,13,51,0,12,15,4,1,115,2,515,453,10,457,5,3,202,21,165,700,403,11,37,64,77,401,9,197,753,59,475,99 }, +{ 129,84,17,56,27,495,19,548,80,123,162,378,3,504,161,469,618,73,40,53,4,26,205,184,106,183,62,6,257,128,862,12 }, +{ 28,9,22,49,109,1,67,0,39,55,168,158,83,36,35,86,420,194,185,159,95,105,69,208,272,103,50,114,2,254,169,30 }, +{ 242,391,8,456,116,13,23,492,341,165,867,51,499,457,479,638,338,509,719,10,1,642,417,762,401,93,206,268,901,569,22,197 }, +{ 211,162,248,130,57,4,41,556,507,266,183,152,305,361,11,129,62,229,38,471,514,313,157,300,377,3,440,128,123,328,339,59 }, +{ 7,92,97,16,298,140,60,126,14,35,279,314,232,246,43,230,508,173,71,107,423,24,150,779,20,189,66,18,607,21,0,653 }, +{ 15,515,700,753,1,0,2,13,23,5,3,180,51,901,6,4,7,12,9,115,8,457,165,82,120,197,10,64,141,341,22,117 }, +{ 0,18,403,25,523,74,6,24,42,91,22,102,13,51,49,193,475,681,95,85,730,64,899,397,273,750,247,673,32,805,757,288 }, +{ 56,0,18,65,33,554,84,343,64,6,90,561,22,19,899,108,27,63,289,475,240,467,370,32,233,214,24,123,95,287,28,194 }, +{ 31,98,127,9,0,44,293,105,395,299,49,242,28,22,599,116,1,284,276,125,456,685,763,159,272,623,23,935,393,144,201,137 }, +{ 1,5,2,14,38,46,50,48,21,7,58,45,270,61,155,171,0,290,69,32,29,54,67,16,24,666,663,17,37,75,502,52 }, +{ 23,51,13,453,457,12,719,4,15,99,401,2,961,3,11,730,475,515,0,1,165,115,629,700,14,17,403,40,5,33,37,64 }, +{ 968,967,966,965,964,963,962,961,960,959,958,957,956,955,954,953,952,951,950,949,948,947,946,945,944,943,942,941,940,939,938,937 }, +{ 2,1,14,29,67,103,6,46,52,75,24,133,38,218,83,309,36,108,70,114,96,5,238,74,25,26,220,236,65,50,69,87 }, +{ 7,71,16,92,24,60,14,97,150,140,35,189,149,298,18,230,43,508,2,423,69,0,38,314,66,279,399,517,251,20,232,273 }, +{ 23,1,120,51,13,202,77,141,260,21,15,5,128,82,2,450,269,165,102,318,48,32,137,515,125,64,12,115,351,180,33,7 }, +{ 77,13,33,23,64,51,4,102,141,128,40,1,2,202,0,6,177,115,137,15,59,10,11,7,269,22,515,180,318,3,700,95 }, +{ 101,9,18,363,264,520,411,604,676,682,905,271,16,821,167,0,621,364,39,100,121,118,166,781,647,252,1,848,447,265,404,60 }, +{ 144,203,326,382,166,418,93,88,96,822,1,141,859,77,744,438,110,269,921,367,521,274,100,39,494,120,403,473,217,576,13,291 }, +{ 13,21,180,125,5,23,191,32,18,16,146,199,115,24,165,118,0,225,22,1,60,197,64,901,375,241,48,12,408,71,522,818 }, +{ 15,515,700,753,13,0,23,8,1,51,82,102,2,33,4,9,180,165,5,77,10,110,12,197,120,260,18,326,351,403,22,457 }, +{ 33,77,102,64,13,23,128,51,141,202,1,40,0,2,117,10,15,4,6,318,269,134,22,515,180,115,177,153,137,196,3,700 }, +{ 174,544,104,525,74,0,151,25,6,624,29,66,2,636,81,45,204,177,64,416,7,644,5,138,222,319,355,77,22,122,789,216 }, +{ 141,304,372,352,291,947,177,269,128,954,77,349,217,202,64,318,498,437,102,864,86,13,115,180,137,5,210,197,32,950,678,7 }, +{ 161,200,53,17,714,27,34,73,472,62,585,56,440,383,136,78,527,19,4,3,106,361,14,250,80,514,377,84,322,390,862,548 }, +{ 32,76,72,21,38,14,89,54,12,37,2,241,5,428,17,1,181,221,350,45,3,4,449,90,148,179,99,292,794,770,477,46 }, +{ 33,23,128,64,141,13,77,51,102,202,2,15,1,3,40,10,5,153,269,515,165,0,117,196,180,318,6,700,137,134,120,22 }, +{ 96,137,30,0,9,39,840,202,669,406,141,530,613,1,180,88,22,160,679,576,28,403,31,219,49,228,829,100,36,15,10,856 }, +{ 180,141,352,1,15,752,115,0,217,365,2,515,13,901,341,317,23,4,197,700,269,5,3,31,753,244,21,165,253,202,51,44 }, +{ 1,2,67,0,28,50,83,65,14,46,103,114,24,38,36,9,69,5,18,7,22,133,55,218,16,124,29,54,96,160,12,480 }, +{ 180,115,352,317,365,217,752,901,141,15,341,1,515,253,700,0,753,873,2,197,31,137,165,244,4,120,160,44,98,5,202,3 }, +{ 5,32,347,49,13,21,95,713,23,1,77,33,60,64,107,4,126,928,296,850,0,241,197,102,652,195,180,534,165,153,379,10 }, +{ 341,180,365,901,317,115,15,752,515,700,217,873,753,82,0,110,197,141,951,165,1,564,13,351,253,12,10,3,2,4,308,244 }, +{ 17,45,21,3,106,5,155,38,227,32,2,209,62,54,12,243,14,181,552,587,46,540,207,794,37,48,430,119,255,221,770,29 }, +{ 16,24,35,18,7,0,50,1,9,14,75,69,2,5,12,21,60,13,67,71,23,48,10,108,223,181,189,103,46,64,92,51 }, +{ 127,13,98,165,308,23,286,293,258,51,219,395,197,115,301,401,31,391,22,105,457,170,239,276,55,338,629,116,180,479,509,569 }, +{ 539,213,748,840,957,669,30,466,88,217,144,251,863,190,137,93,230,228,679,352,317,203,617,321,258,530,160,219,96,831,816,689 }, +{ 5,48,1,21,2,14,0,36,12,38,32,54,430,181,50,270,72,99,281,45,17,10,46,22,37,218,67,3,290,76,23,51 }, +{ 13,23,0,4,33,51,2,115,141,1,77,217,180,10,9,317,3,102,11,5,15,197,7,202,22,165,40,64,515,6,341,31 }, +{ 13,15,117,515,23,12,37,134,165,700,38,54,457,753,51,64,153,197,14,10,33,82,961,0,99,89,115,719,141,3,4,1 }, +{ 5,21,2,3,1,32,14,12,48,17,0,10,51,23,38,22,4,72,13,54,36,45,137,76,99,114,86,37,11,64,540,430 }, +{ 202,128,77,318,291,33,269,102,275,141,494,342,40,678,0,177,20,210,402,7,4,5,137,6,13,450,403,32,49,120,23,22 }, +{ 1,2,24,14,67,46,69,50,38,103,16,18,75,35,83,29,52,96,5,108,0,7,54,71,149,394,236,309,70,133,220,58 }, +{ 15,515,1,700,0,2,753,13,23,5,180,3,51,4,165,457,12,197,115,6,7,21,9,141,8,901,33,82,120,77,10,110 }, +{ 0,28,65,14,67,2,124,24,1,9,7,69,55,154,36,16,46,114,175,35,83,22,429,18,109,149,68,189,108,336,251,133 }, +{ 56,162,403,3,129,775,99,161,17,40,527,33,880,4,14,128,475,12,548,23,102,202,361,117,34,184,383,200,183,196,64,53 }, +{ 151,2,29,58,112,45,186,113,5,70,52,1,311,6,315,66,61,7,74,27,631,17,80,87,287,243,209,227,14,491,19,869 }, +{ 6,1,74,2,75,29,25,66,26,70,52,138,67,324,357,42,19,220,14,85,87,108,38,451,309,103,24,69,380,135,114,65 }, +{ 15,515,700,13,23,0,1,120,753,51,180,2,260,202,5,141,77,102,9,450,115,21,197,165,7,137,110,33,12,269,901,4 }, +{ 5,45,17,2,14,46,48,38,181,50,155,3,186,54,61,29,21,227,281,80,540,106,12,400,52,1,58,32,328,171,209,487 }, +{ 16,18,265,121,158,35,60,9,39,7,329,105,252,68,24,1,132,167,159,22,0,49,286,101,21,146,23,327,120,709,5,14 }, +{ 108,467,283,56,389,650,123,412,33,177,899,475,216,453,269,349,619,65,51,730,403,670,23,196,523,128,84,13,401,789,503,543 }, +{ 514,3,11,377,328,4,361,507,57,403,14,880,130,485,176,215,236,38,152,102,211,56,62,757,54,585,300,556,34,555,40,229 }, +{ 3,555,62,266,130,99,507,139,514,12,152,229,215,305,57,40,440,33,403,471,38,56,475,14,361,313,775,328,196,548,123,23 }, +{ 120,202,318,15,77,13,1,450,33,269,515,260,5,128,494,51,23,700,102,141,40,753,326,403,817,137,523,21,177,922,342,7 }, +{ 15,1,515,23,0,13,700,2,51,753,180,5,165,21,197,12,3,120,115,4,141,6,9,7,457,33,386,202,82,8,31,341 }, +{ 15,180,515,82,351,700,10,317,753,115,217,365,141,898,33,901,13,23,110,854,752,77,1,197,4,341,143,36,64,352,102,9 }, +{ 104,289,66,707,214,90,712,64,97,173,20,0,414,194,874,43,32,7,568,560,65,38,426,312,715,192,376,74,835,5,324,147 }, +{ 84,56,0,554,63,65,453,249,123,643,18,26,847,475,511,403,416,561,524,289,370,73,9,19,45,42,719,194,27,467,33,730 }, +{ 21,346,13,350,308,826,197,101,352,68,570,0,165,23,9,841,115,100,509,694,221,230,35,217,569,88,124,749,1,777,212,154 }, +{ 16,92,7,24,60,18,35,140,126,14,50,71,46,330,2,75,246,5,121,267,571,1,230,309,220,0,9,64,146,236,54,108 }, +{ 82,15,515,898,365,700,180,33,341,753,77,901,10,115,55,351,21,5,1,4,13,102,36,217,2,165,752,120,197,117,11,317 }, +{ 16,24,35,18,69,71,140,1,103,7,189,68,0,50,9,108,2,133,60,267,230,46,149,67,167,118,92,14,75,21,191,38 }, +{ 60,71,16,18,7,20,43,118,35,68,375,28,608,0,175,566,154,92,14,149,628,33,22,13,2,10,279,23,107,356,55,117 }, +{ 187,258,871,295,201,434,219,224,489,384,268,110,261,839,44,699,93,116,36,131,141,228,144,160,940,567,244,406,137,574,98,253 }, +{ 66,7,97,172,192,712,232,324,204,74,43,448,387,426,568,20,526,107,104,135,356,729,173,0,22,5,32,95,2,64,500,560 }, +{ 15,515,700,753,1,4,0,341,13,3,134,2,5,33,11,77,12,10,23,197,365,901,7,40,217,32,21,6,51,180,961,37 }, +{ 0,28,24,9,35,65,16,124,68,55,109,154,7,39,22,149,158,14,175,1,49,252,18,71,2,168,289,419,108,420,67,101 }, +{ 7,16,14,92,2,46,140,24,220,35,38,60,75,1,50,18,87,54,5,126,29,52,278,262,314,107,71,21,172,135,330,394 }, +{ 7,92,16,14,172,126,2,60,140,35,135,314,278,46,24,38,232,107,330,66,5,18,150,246,230,97,52,1,121,563,279,21 }, +{ 6,26,235,53,297,436,27,19,25,73,113,445,90,214,65,42,64,289,250,611,624,32,45,648,614,17,85,491,34,122,200,416 }, +{ 352,141,1,217,854,752,351,180,244,36,110,661,82,258,816,160,295,219,567,224,230,269,922,144,260,268,93,201,137,116,489,202 }, +{ 16,60,35,18,126,107,68,191,92,121,7,14,598,20,493,279,167,446,118,0,28,43,463,55,24,212,375,566,9,150,575,21 }, +{ 15,1,515,2,4,13,0,700,3,5,23,753,341,77,51,115,33,11,180,10,197,141,6,165,7,901,102,40,9,202,217,12 }, +{ 23,51,13,202,21,5,1,120,15,137,128,125,32,2,12,141,33,165,64,515,403,318,700,48,180,7,6,450,115,523,475,260 }, +{ 131,716,224,371,219,187,737,616,385,254,9,98,105,924,31,258,836,39,127,578,49,916,44,761,272,137,944,159,0,242,442,22 }, +{ 15,515,700,1,753,2,5,0,4,13,3,180,11,141,197,10,341,217,33,134,165,6,77,7,317,12,352,64,365,32,102,40 }, +{ 66,74,7,173,174,29,192,2,222,20,226,43,353,52,712,6,0,138,500,204,97,145,64,104,426,673,355,90,25,5,65,87 }, +{ 5,259,786,534,590,493,279,49,13,581,465,21,929,35,941,132,147,32,23,612,362,626,107,121,178,0,146,61,48,939,10,18 }, +{ 2,14,16,7,278,69,135,140,46,24,267,35,92,38,1,189,29,52,309,60,66,75,71,172,74,357,18,87,67,6,230,5 }, +{ 165,13,308,197,391,23,401,15,51,457,180,509,115,569,3,629,961,719,34,758,317,734,14,29,46,2,17,901,38,453,5,217 }, +{ 1,22,2,14,0,28,7,168,67,49,65,24,36,95,5,105,55,35,12,46,69,16,114,159,194,50,10,9,158,83,164,109 }, +{ 34,453,3,196,130,14,322,11,47,51,377,236,361,4,730,153,514,711,57,440,62,17,161,108,176,59,485,56,162,412,202,117 }, +{ 18,16,21,23,48,13,24,35,121,5,156,60,51,1,7,132,141,221,163,115,0,271,447,340,363,202,125,71,2,781,22,698 }, +{ 165,13,457,23,197,961,629,569,341,41,12,38,401,901,54,51,115,17,15,509,421,37,62,45,719,57,32,328,117,758,157,99 }, +{ 2,1,77,141,33,64,3,102,0,23,13,5,128,10,6,15,180,202,269,40,51,515,7,165,137,117,318,4,700,153,197,352 }, +{ 68,212,0,124,101,9,154,16,562,191,21,149,65,24,35,1,118,167,818,350,520,100,722,841,264,71,13,302,478,23,375,346 }, +{ 98,23,48,598,13,293,541,21,125,121,51,807,0,31,35,259,126,7,386,1,223,783,10,107,199,20,221,144,342,963,49,64 }, +{ 21,13,5,586,1,23,167,48,33,781,647,49,165,18,51,271,77,32,761,118,0,82,391,22,146,141,459,31,197,156,115,4 }, +{ 2,1,5,61,29,7,58,45,14,6,425,32,70,52,290,738,207,21,72,112,66,76,655,17,186,46,64,263,38,0,128,87 }, +{ 39,265,9,100,1,333,363,101,18,411,447,254,166,310,31,98,264,30,639,404,156,286,16,93,593,203,272,682,0,905,44,821 }, +{ 6,2,1,19,29,51,26,108,25,74,5,23,14,114,13,386,133,103,42,66,453,70,309,138,719,324,65,38,64,96,52,75 }, +{ 20,43,356,107,49,858,595,7,414,359,0,5,392,319,97,612,422,819,14,376,173,246,22,470,147,427,230,92,197,33,683,95 }, +{ 0,9,68,35,65,67,114,101,28,1,124,175,336,69,154,103,83,24,189,133,39,16,50,7,2,149,55,251,18,345,230,36 }, +{ 23,13,51,15,0,1,515,115,165,2,5,12,700,202,4,21,141,457,753,197,10,3,180,120,32,9,318,11,453,64,6,269 }, +{ 121,195,60,16,126,107,98,271,146,407,132,35,1,167,199,223,493,191,279,20,18,5,43,7,21,92,48,393,0,362,212,467 }, +{ 31,44,299,116,393,144,492,456,268,22,105,0,367,918,384,434,127,489,98,9,963,125,242,948,1,28,206,49,36,51,93,293 }, +{ 23,13,457,51,165,401,719,758,197,453,961,629,308,14,15,12,730,3,386,569,391,29,739,515,34,828,832,901,115,514,670,341 }, +{ 105,36,131,22,180,115,341,127,169,1,9,31,64,98,44,365,317,141,272,143,160,55,219,86,197,776,239,187,0,535,13,752 }, +{ 1,6,2,14,66,25,29,5,108,67,65,114,19,38,26,52,74,7,24,18,69,86,36,388,64,51,17,83,23,46,42,75 }, +{ 51,386,23,453,719,13,730,6,457,670,758,19,401,165,2,475,47,26,899,14,108,17,1,5,197,29,894,754,236,74,27,285 }, +{ 252,18,9,101,121,16,132,0,419,167,364,60,604,35,265,363,146,271,39,158,68,109,28,329,848,24,647,907,682,159,212,55 }, +{ 283,503,128,432,26,193,63,269,789,529,102,122,389,275,678,6,25,318,445,4,342,27,573,605,177,862,643,291,216,57,235,59 }, +{ 2,1,29,75,69,52,14,6,46,74,87,7,220,226,278,38,135,66,267,70,16,262,25,24,380,324,357,140,67,394,97,222 }, +{ 97,298,69,7,66,140,189,24,16,267,172,423,60,150,14,314,92,71,81,501,43,35,74,6,517,232,149,607,83,330,18,2 }, +{ 475,421,403,899,51,805,523,958,453,817,23,615,401,801,120,326,202,670,494,730,450,386,115,629,260,576,77,365,569,0,165,13 }, +{ 7,20,14,128,77,97,112,202,2,177,16,415,269,318,275,66,107,43,141,414,135,38,307,10,58,0,6,291,32,5,4,40 }, +{ 24,14,7,0,2,1,22,28,16,65,168,124,35,67,108,109,18,49,10,149,69,158,5,95,289,12,55,6,36,71,46,21 }, +{ 26,80,27,73,122,25,19,17,6,42,684,209,445,573,667,106,45,690,4,611,255,680,297,495,65,59,128,119,483,113,64,53 }, +{ 107,259,362,376,465,20,470,147,595,534,612,683,660,43,5,49,581,0,858,35,427,246,97,786,178,356,14,21,142,878,7,279 }, +{ 131,30,228,190,856,406,224,88,219,530,863,613,778,274,944,816,187,39,100,160,258,31,44,93,1,321,539,36,871,137,435,531 }, +{ 113,6,311,25,45,491,80,611,27,26,209,667,17,73,122,42,684,396,19,85,106,5,614,4,2,255,151,29,1,64,648,61 }, +{ 15,515,700,753,0,1,23,51,120,2,13,82,5,260,9,4,341,77,180,115,141,10,7,12,450,8,202,901,197,351,165,93 }, +{ 219,127,98,258,395,421,924,293,242,201,697,105,276,51,308,23,453,272,401,944,512,137,13,31,284,567,386,365,116,131,964,125 }, +{ 15,180,352,141,515,752,217,82,1,317,854,700,351,753,115,341,110,13,260,120,21,36,33,898,23,10,5,365,4,160,901,137 }, +{ 129,123,17,257,162,184,205,249,183,769,5,80,3,4,229,130,119,45,90,99,618,106,57,497,12,128,2,84,59,152,27,40 }, +{ 33,102,23,77,64,128,51,13,0,202,10,141,40,15,1,22,117,137,2,86,4,403,269,153,515,196,65,11,700,115,99,5 }, +{ 7,14,2,16,172,107,46,92,5,135,35,202,294,87,38,232,29,97,20,21,24,1,60,220,66,43,12,0,126,52,54,70 }, +{ 403,576,615,523,475,326,805,817,494,421,51,202,120,450,137,453,23,859,260,401,402,77,33,670,0,958,15,197,386,515,165,480 }, +{ 141,352,217,137,0,180,202,349,9,269,23,51,115,291,77,372,13,317,120,752,365,351,93,22,2,341,64,10,82,854,28,18 }, +{ 1,23,13,51,202,141,5,165,21,15,120,180,64,2,197,125,33,102,12,7,137,515,48,128,269,318,93,700,0,403,9,4 }, +{ 25,151,6,145,122,29,174,45,113,74,4,665,42,138,2,614,416,287,19,348,746,0,66,26,1,7,64,243,311,396,81,624 }, +{ 30,190,254,166,100,382,731,829,88,131,264,795,9,93,625,274,438,1,578,613,716,31,44,39,530,36,616,921,265,203,160,77 }, +{ 132,5,21,13,1,23,32,195,379,687,156,121,626,296,48,70,850,146,51,82,883,771,35,49,652,407,60,4,260,0,845,33 }, +{ 9,254,0,49,272,131,39,159,688,101,105,578,518,158,286,28,327,333,68,224,252,219,344,16,22,1,716,31,30,228,24,890 }, +{ 16,7,35,60,18,20,14,68,9,0,28,118,43,92,126,55,107,2,101,154,24,71,5,202,121,109,22,252,21,97,1,621 }, +{ 15,515,700,753,13,1,341,2,0,4,3,5,11,23,10,33,117,12,901,197,6,134,77,8,165,317,21,365,217,7,17,40 }, +{ 78,19,444,47,26,390,27,453,130,813,108,730,711,65,412,122,51,680,113,235,690,196,630,283,128,236,14,64,73,53,200,445 }, +{ 2,7,29,5,61,6,45,1,66,113,112,14,52,315,738,128,32,151,74,16,20,64,70,21,592,0,25,4,425,43,491,222 }, +{ 145,112,74,66,6,29,26,70,19,396,25,87,2,287,135,151,138,222,5,226,42,122,7,307,1,644,45,58,113,651,635,632 }, +{ 92,16,7,60,126,24,140,35,14,232,18,121,246,71,46,267,172,150,107,314,132,146,230,2,278,108,330,199,236,5,38,572 }, +{ 13,115,197,538,569,341,98,55,165,127,365,762,219,286,844,23,170,206,734,638,535,901,169,253,629,0,873,509,180,10,332,258 }, +{ 58,151,74,53,287,27,29,396,6,70,2,73,5,52,112,26,651,1,297,113,17,75,19,45,334,445,145,34,315,549,436,331 }, +{ 214,289,90,874,104,751,64,65,312,835,204,249,750,194,74,81,875,32,519,288,348,0,174,247,636,715,138,192,784,6,524,280 }, +{ 9,39,28,35,30,0,166,49,1,175,439,158,64,346,36,101,67,364,86,88,274,100,168,55,23,10,420,22,190,141,505,180 }, +{ 341,901,15,515,700,753,1,365,10,0,569,180,2,197,115,31,165,3,5,4,44,22,317,13,9,951,23,253,116,143,762,93 }, +{ 120,202,77,450,260,15,128,318,102,515,494,13,817,700,269,5,403,51,1,33,23,753,82,326,141,342,291,137,21,523,351,32 }, +{ 13,115,241,64,180,32,125,197,165,4,118,22,21,23,16,247,237,28,225,191,95,141,167,5,0,341,288,35,459,18,177,24 }, +{ 16,24,35,14,1,2,7,69,18,46,60,50,267,140,71,189,108,38,75,92,0,5,9,230,67,21,309,335,54,236,394,220 }, +{ 15,515,700,753,898,180,901,341,197,638,10,165,33,1,115,4,77,365,317,13,102,217,117,0,5,2,253,3,82,569,21,752 }, +{ 193,523,18,84,56,730,233,65,4,817,90,33,643,403,91,511,453,240,59,11,214,51,719,196,153,475,32,123,64,847,102,561 }, +{ 112,29,151,2,74,6,66,7,222,145,287,45,5,624,52,25,113,416,58,122,19,70,186,204,4,87,644,549,337,884,32,0 }, +{ 13,0,23,2,1,15,33,3,77,515,141,5,4,217,10,51,64,180,700,115,6,117,11,7,753,40,102,165,197,22,317,153 }, +{ 28,0,1,67,65,9,2,114,83,69,103,50,36,22,55,24,46,14,124,109,35,7,16,38,133,160,389,323,18,12,154,5 }, +{ 121,132,18,167,271,146,101,363,621,9,411,647,16,354,520,60,212,932,1,806,55,0,195,446,68,35,31,364,777,252,407,118 }, +{ 26,6,85,396,122,624,25,19,42,445,64,648,573,416,174,680,665,214,45,348,90,65,194,145,113,881,138,289,112,436,297,544 }, +{ 16,146,18,92,24,199,60,71,121,126,35,108,156,953,271,674,132,7,32,640,360,246,649,118,21,95,5,517,14,9,1,314 }, +{ 51,13,23,453,475,730,719,15,457,403,64,115,33,95,4,523,3,12,21,6,899,102,5,128,401,202,11,141,308,515,22,125 }, +{ 151,396,6,53,27,113,58,26,73,112,74,287,45,29,297,19,145,70,138,445,315,436,34,2,17,573,5,61,549,491,1,80 }, +{ 223,1,888,774,260,98,269,385,349,202,96,141,421,622,730,863,318,697,87,453,393,418,922,834,751,5,163,335,120,291,352,30 }, +{ 16,60,92,35,126,121,7,150,246,18,107,1,598,24,167,195,14,97,71,279,98,441,191,199,517,146,356,223,298,271,230,0 }, +{ 22,1,105,28,239,170,0,55,95,31,36,301,2,320,98,127,9,49,44,64,35,67,10,86,5,12,109,23,168,13,21,312 }, +{ 2,6,5,207,292,76,1,119,45,32,17,29,61,306,790,58,240,106,14,64,214,151,476,710,7,72,84,128,4,179,70,25 }, +{ 51,23,221,254,115,13,438,530,125,48,21,39,541,960,386,49,1,613,15,840,228,308,627,131,688,401,5,326,421,158,165,83 }, +{ 1,5,2,0,12,22,21,36,10,14,48,86,23,13,32,54,3,4,28,65,51,50,137,37,208,114,9,38,17,7,281,202 }, +{ 363,23,447,182,296,340,1,93,698,478,379,156,284,144,18,269,21,98,141,70,668,411,664,658,110,914,67,937,180,691,335,291 }, +{ 17,32,45,498,41,115,180,197,106,62,54,38,546,165,13,155,468,509,341,243,241,217,542,15,57,536,428,51,117,721,292,129 }, +{ 32,95,64,246,22,92,180,13,5,652,125,241,638,237,7,49,4,126,21,115,197,296,888,316,0,165,774,23,16,392,1,534 }, +{ 15,515,700,753,33,341,13,217,4,141,77,23,180,317,1,10,102,351,82,115,40,5,854,21,137,11,352,901,365,117,197,0 }, +{ 15,120,1,82,93,217,515,260,77,141,13,110,700,351,352,23,180,753,21,854,202,317,64,349,269,51,165,137,5,128,291,36 }, +{ 13,23,51,141,77,0,33,4,115,64,2,10,102,202,217,128,1,177,269,11,7,22,6,21,32,9,180,40,15,3,165,318 }, +{ 478,264,1,520,98,724,9,682,223,664,21,759,13,772,604,100,23,363,411,48,821,5,0,905,909,447,31,265,88,101,166,39 }, +{ 20,29,7,2,77,416,6,128,33,5,0,113,104,32,43,13,491,66,23,21,102,51,74,210,202,525,64,318,10,81,174,14 }, +{ 2,1,5,14,7,58,61,29,45,290,46,38,52,21,32,270,6,592,425,0,75,155,16,48,17,50,72,70,207,24,263,663 }, +{ 80,6,17,209,106,26,483,113,19,469,255,25,378,27,495,833,45,64,161,2,61,667,76,742,32,90,445,5,814,65,887,119 }, +{ 98,223,393,1,354,834,195,791,447,697,284,293,360,541,781,156,51,807,18,664,421,411,163,668,48,31,591,765,883,386,948,23 }, +{ 679,141,816,36,93,406,876,144,228,137,1,180,669,21,332,251,5,269,116,187,96,351,202,752,317,64,203,831,574,466,855,345 }, +{ 15,515,700,13,1,753,2,0,23,341,3,5,4,10,51,11,33,165,6,7,115,197,12,64,180,153,217,77,9,569,901,317 }, +{ 13,23,202,51,5,21,403,15,120,64,1,450,128,141,12,523,33,165,494,125,2,515,269,7,48,102,318,95,260,180,453,197 }, +{ 16,18,24,60,71,92,146,246,199,35,140,7,9,118,121,108,167,230,126,132,0,640,156,14,68,133,267,360,649,271,64,55 }, +{ 269,141,678,177,202,77,128,318,33,947,40,120,291,349,102,137,64,352,210,864,461,498,13,342,196,23,275,450,954,0,205,111 }, +{ 16,24,92,18,71,60,35,7,108,191,167,246,140,14,126,21,1,68,150,118,149,388,399,9,273,0,121,796,230,48,212,517 }, +{ 2,14,1,29,46,75,52,70,69,171,38,7,58,163,16,5,24,220,67,112,223,54,50,409,155,35,267,186,151,334,394,140 }, +{ 9,252,100,265,166,39,88,404,329,0,1,520,382,812,101,593,264,274,604,676,30,118,68,553,18,664,363,23,639,865,21,411 }, +{ 16,18,35,24,0,60,158,7,22,68,14,49,109,159,55,9,28,71,2,10,5,105,1,118,329,13,344,23,92,20,21,126 }, +{ 15,13,515,700,23,0,753,1,51,2,4,10,77,5,3,197,115,165,961,202,9,457,180,12,141,22,33,120,6,11,318,31 }, +{ 160,93,251,137,317,1,180,36,120,217,345,752,617,352,332,10,96,531,498,318,365,202,141,269,816,341,901,679,143,35,83,968 }, +{ 6,25,42,128,19,59,122,4,85,26,611,27,269,233,45,0,343,91,318,80,11,177,283,73,33,614,2,77,64,138,445,216 }, +{ 95,64,74,7,32,81,51,204,0,20,237,65,56,38,91,23,207,180,347,343,29,6,511,52,49,10,25,18,554,370,14,312 }, +{ 202,120,326,260,450,817,494,318,137,403,128,77,523,553,859,5,704,1,15,23,13,576,7,16,615,51,682,291,515,0,21,234 }, +{ 20,43,107,356,362,126,595,92,359,7,422,319,493,16,858,5,392,246,414,683,60,0,35,945,441,21,259,819,49,97,279,173 }, +{ 25,42,6,77,33,102,0,122,4,690,29,483,210,27,21,19,2,300,18,648,680,119,117,59,1,10,342,12,26,153,91,684 }, +{ 31,44,299,116,125,242,456,599,22,393,0,144,492,28,268,1,9,963,301,105,367,36,127,170,384,434,206,98,918,10,13,93 }, +{ 410,521,686,367,662,88,335,321,201,96,98,772,144,1,934,921,443,435,284,274,264,551,120,897,44,100,33,225,744,418,909,960 }, +{ 142,178,878,234,132,786,195,202,77,416,147,929,146,522,167,259,687,639,450,271,626,481,590,5,198,212,771,49,0,465,315,427 }, +{ 254,39,131,9,272,0,578,716,310,224,30,49,105,827,518,829,166,333,616,228,613,846,101,219,1,31,890,98,159,938,252,100 }, +{ 230,699,854,473,450,351,831,137,855,217,352,704,800,202,251,498,160,144,206,203,317,201,253,752,418,141,1,332,82,180,443,36 }, +{ 403,202,475,453,494,23,51,77,318,402,13,33,128,102,137,141,120,342,269,0,450,4,899,576,40,421,275,117,217,177,196,64 }, +{ 23,44,98,182,291,144,116,39,110,141,96,82,905,70,367,264,125,93,77,411,120,1,658,202,100,415,107,363,197,30,447,105 }, +{ 15,515,1,13,700,23,77,120,0,753,51,180,202,141,260,5,21,115,2,137,128,9,450,197,365,269,12,326,110,102,318,7 }, +{ 0,32,18,95,207,577,193,29,61,104,64,784,715,102,693,887,81,91,583,671,403,5,52,474,397,180,138,49,37,344,38,263 }, +{ 0,101,9,68,252,16,100,39,166,364,124,24,154,265,212,88,18,35,329,419,28,118,71,30,65,158,191,55,1,694,21,676 }, +{ 16,24,191,18,35,71,167,118,149,68,212,9,0,1,21,108,101,92,60,375,302,7,589,755,124,674,350,48,562,246,13,363 }, +{ 2,14,69,24,1,67,46,16,38,103,29,267,7,35,189,135,278,71,108,18,83,309,52,6,149,388,75,236,60,0,150,66 }, +{ 13,1,23,0,4,2,51,15,180,33,3,115,5,515,141,10,77,700,11,9,197,341,202,165,217,102,22,7,753,317,365,6 }, +{ 447,1,698,411,31,363,98,5,919,4,156,125,759,691,13,64,459,354,44,21,48,293,30,914,478,225,82,120,2,922,848,839 }, +{ 854,82,351,217,141,180,352,15,515,752,1,700,317,898,753,244,10,21,922,115,77,36,4,260,64,110,372,13,5,365,120,11 }, +{ 0,1,4,13,5,2,82,33,3,120,10,23,9,11,77,260,21,102,8,31,40,6,351,51,64,450,22,117,93,110,7,457 }, +{ 51,13,403,23,12,475,1,2,21,5,453,523,115,202,817,7,0,99,3,6,450,120,494,64,22,95,49,899,10,37,32,141 }, +{ 180,752,352,141,498,864,317,217,9,0,115,237,230,39,30,197,83,1,930,64,35,365,372,13,579,88,702,36,101,901,482,21 }, +{ 16,24,0,7,22,18,28,35,14,158,71,2,109,60,1,168,49,154,124,68,10,55,92,118,159,9,5,747,95,105,65,6 }, +{ 15,515,700,753,0,23,1,13,341,51,120,77,141,4,137,33,260,82,202,9,180,5,351,2,291,10,11,901,21,115,40,128 }, +{ 24,68,35,149,18,16,0,7,9,14,189,108,69,65,67,1,71,2,118,28,140,101,114,336,230,124,175,133,46,55,251,154 }, +{ 13,308,197,115,125,9,165,237,391,23,509,569,546,28,49,629,22,338,317,254,749,180,468,159,903,386,217,352,558,39,36,734 }, +{ 64,90,32,217,77,4,141,216,172,312,128,13,86,33,597,147,352,95,5,115,875,22,59,11,102,14,182,437,97,177,874,707 }, +{ 1,22,36,0,105,28,2,67,95,49,55,5,239,12,86,9,83,170,312,64,31,21,23,10,164,50,114,159,208,13,7,320 }, +{ 9,18,16,0,159,105,101,252,49,310,24,68,35,39,265,272,7,1,60,28,455,890,329,557,118,286,55,137,327,167,5,13 }, +{ 129,123,214,249,618,17,5,257,205,184,460,76,2,162,769,245,90,106,128,45,119,1,183,4,3,12,179,64,6,229,99,209 }, +{ 51,453,719,457,23,13,730,899,475,386,4,15,11,12,670,196,515,523,961,401,153,3,700,99,753,117,403,32,120,165,57,0 }, +{ 173,66,192,204,20,74,104,636,7,43,289,426,825,712,560,214,81,750,65,97,707,0,90,414,64,348,32,500,22,861,95,6 }, +{ 13,23,1,51,5,21,141,120,202,15,165,2,515,180,12,125,0,64,82,700,197,269,32,48,260,128,115,93,9,137,33,753 }, +{ 200,34,322,78,472,390,27,714,19,14,136,161,453,176,236,444,59,3,62,128,108,57,283,862,73,53,47,17,412,813,4,56 }, +{ 33,347,66,204,426,498,56,172,97,95,5,681,546,22,10,0,135,180,4,241,19,174,6,353,263,21,7,370,42,197,27,808 }, +{ 131,224,219,187,385,371,258,442,254,737,31,98,836,127,924,944,44,871,908,716,39,827,201,574,116,137,36,1,276,242,578,616 }, +{ 1,15,2,180,4,141,13,515,0,5,3,115,700,901,341,23,217,352,753,51,197,77,317,33,365,752,165,21,6,7,269,93 }, +{ 370,91,718,74,81,510,397,66,636,240,355,84,138,511,18,278,6,681,701,289,90,5,214,582,64,104,0,643,192,65,750,32 }, +{ 15,515,700,753,13,120,1,0,165,2,197,23,260,180,4,82,51,386,8,5,12,10,3,141,351,341,326,9,450,7,64,6 }, +{ 32,2,76,5,1,292,72,45,476,214,21,241,29,14,17,48,129,90,179,460,464,123,290,148,519,205,3,263,249,38,710,89 }, +{ 13,165,115,17,197,569,23,509,457,45,32,41,106,180,62,38,659,734,155,536,341,629,961,873,587,54,431,37,391,99,405,428 }, +{ 68,24,35,16,0,101,9,124,154,71,149,65,18,175,28,118,7,55,302,108,92,14,22,346,1,39,429,252,375,364,10,67 }, +{ 0,9,1,68,230,65,35,69,83,23,101,13,141,67,217,352,21,39,16,88,28,124,212,100,115,154,51,64,30,36,10,317 }, +{ 100,265,88,909,410,382,812,593,1,213,321,30,252,230,352,264,9,166,689,39,676,98,21,466,724,639,478,217,13,48,553,101 }, +{ 113,61,198,904,43,0,5,37,899,325,20,59,33,523,204,725,817,389,470,329,222,40,174,58,22,453,690,848,122,104,788,105 }, +{ 16,24,141,18,7,0,71,140,35,269,75,352,12,9,108,217,5,330,60,64,199,70,22,13,486,246,318,133,65,50,23,498 }, +{ 271,167,121,60,18,191,146,199,16,1,446,132,575,212,463,354,126,35,598,566,727,98,107,21,608,955,640,407,5,24,223,68 }, +{ 15,515,115,217,700,13,317,753,141,180,33,23,110,120,4,341,82,10,1,260,365,36,64,854,351,21,51,352,137,77,40,0 }, +{ 173,7,97,356,43,107,20,387,729,104,426,232,560,595,359,392,414,707,885,81,5,0,66,858,612,49,861,14,22,32,819,230 }, +{ 15,515,700,753,13,23,51,82,0,33,165,120,196,4,1,2,197,453,260,351,180,12,40,8,386,110,5,326,9,141,217,457 }, +{ 9,100,120,30,77,795,137,82,202,39,264,827,578,127,0,166,373,318,18,326,141,260,1,450,731,31,33,395,217,291,341,254 }, +{ 14,2,67,1,24,69,0,28,65,7,46,18,114,108,36,83,38,398,9,16,124,133,103,154,50,55,22,267,29,160,35,547 }, +{ 14,7,69,24,66,16,2,267,189,67,71,150,140,97,18,60,172,35,6,1,38,149,388,92,83,135,108,74,462,380,29,36 }, +{ 15,515,700,753,1,13,2,0,4,341,5,3,23,365,11,117,180,10,12,33,134,115,77,197,217,165,6,7,317,102,21,9 }, +{ 1,22,0,12,5,2,36,28,21,10,86,13,23,49,128,9,95,51,55,96,208,141,48,202,4,137,37,64,105,3,50,7 }, +{ 605,630,63,123,736,650,65,108,444,368,561,389,19,25,42,619,122,194,183,27,53,33,84,26,297,813,114,73,256,235,249,216 }, +{ 127,39,9,0,31,371,98,254,1,385,395,44,30,836,187,131,100,116,284,578,299,166,28,21,737,16,276,272,23,49,137,935 }, +{ 2,61,6,29,45,151,1,655,7,207,32,5,112,425,17,76,833,4,14,64,58,106,119,25,113,128,72,52,70,21,292,790 }, +{ 2,5,17,14,3,29,23,27,13,401,46,6,51,58,1,453,45,53,34,52,133,19,236,26,181,114,99,366,151,108,218,38 }, +{ 31,22,170,1,301,44,127,98,36,28,55,105,239,0,338,116,512,299,293,125,86,10,242,395,2,13,9,64,841,23,95,685 }, +{ 1,21,2,14,5,32,48,50,38,270,46,76,290,72,45,54,17,0,155,221,263,207,37,281,430,3,89,12,181,408,36,67 }, +{ 17,106,119,378,84,240,62,80,383,136,306,3,56,790,742,5,207,504,64,440,32,128,45,2,123,209,14,4,61,57,297,667 }, +{ 13,15,1,515,23,2,0,700,4,115,3,51,5,10,753,180,33,341,217,11,165,317,365,197,6,77,40,64,22,9,7,117 }, +{ 772,335,96,744,1,367,662,686,652,897,303,264,521,31,225,410,141,520,260,116,64,44,321,98,144,88,919,966,340,269,349,284 }, +{ 1,0,5,28,36,2,12,22,83,67,65,50,24,14,9,96,21,218,18,114,48,281,54,10,7,160,181,103,37,23,133,99 }, +{ 25,6,145,42,138,81,174,348,525,544,26,74,85,280,287,648,746,91,66,0,29,396,204,64,636,90,122,194,355,104,65,233 }, +{ 16,24,0,18,28,158,7,35,49,22,68,159,55,1,14,109,105,2,9,71,65,154,124,95,424,344,60,239,118,577,21,10 }, +{ 352,854,699,230,93,689,137,144,217,160,251,36,669,202,351,120,617,855,752,203,332,82,450,180,141,748,831,30,258,201,1,816 }, +{ 15,13,515,700,1,2,0,753,23,5,4,3,51,33,10,115,11,317,217,77,180,341,117,165,6,134,197,153,64,9,102,7 }, +{ 104,20,43,173,66,319,0,77,202,7,198,5,97,580,355,74,2,204,174,52,712,234,426,155,102,192,32,4,500,337,226,904 }, +{ 13,23,15,51,1,515,0,2,5,700,141,4,753,165,115,12,3,21,457,10,180,269,32,6,197,202,9,7,120,11,77,33 }, +{ 15,515,700,753,0,1,13,2,23,115,4,317,8,3,5,51,9,341,10,217,22,365,33,457,6,180,77,901,197,120,18,7 }, +{ 15,515,700,753,1,0,2,3,4,13,5,141,23,16,82,217,457,10,365,180,9,317,51,21,269,898,64,202,11,12,318,341 }, +{ 2,5,1,14,50,38,29,17,114,46,133,3,45,21,58,171,181,36,218,12,6,52,0,48,137,65,361,23,155,4,285,51 }, +{ 0,68,9,65,101,124,35,212,16,149,154,100,24,1,114,336,67,589,252,39,71,189,69,562,18,13,30,398,118,88,265,264 }, +{ 1,2,14,22,0,7,67,65,28,36,24,46,168,5,86,69,38,16,49,12,289,10,194,50,83,114,95,6,18,23,55,158 }, +{ 13,23,15,51,515,0,700,4,1,753,2,10,115,961,457,12,33,11,3,5,197,9,165,77,102,403,453,40,64,22,37,59 }, +{ 15,515,700,753,0,1,2,13,23,5,51,901,8,9,180,3,7,82,4,120,12,10,719,341,6,31,141,457,197,22,115,93 }, +{ 100,252,88,101,0,265,9,724,48,1,21,352,213,676,410,382,321,230,30,329,593,909,39,812,553,217,23,689,520,264,166,419 }, +{ 2,5,1,58,171,14,46,50,29,52,45,38,186,155,67,54,151,281,334,61,48,96,17,181,103,400,502,227,21,223,12,69 }, +{ 23,120,13,1,202,141,51,21,165,128,260,15,5,269,137,64,33,180,82,318,93,197,77,326,515,125,110,700,450,2,32,48 }, +{ 341,197,10,901,13,15,8,638,569,515,479,23,180,873,700,165,143,642,0,961,753,951,1,115,509,499,116,12,498,242,82,206 }, +{ 15,1,23,13,515,21,120,51,2,141,202,700,5,180,165,0,753,197,12,7,33,260,352,137,269,4,82,128,48,9,110,6 }, +{ 2,29,50,58,1,6,5,52,14,262,17,46,27,53,151,34,171,74,324,26,38,309,45,113,19,96,287,396,223,67,73,583 }, +{ 13,23,141,51,4,202,0,115,77,2,33,217,5,317,180,64,10,269,3,9,15,21,1,128,102,137,318,11,352,515,22,31 }, +{ 1,67,0,24,50,5,14,18,16,69,2,9,103,35,83,12,96,28,54,7,58,223,21,46,281,48,65,181,22,38,36,108 }, +{ 13,23,141,51,77,64,202,115,33,102,128,4,0,269,10,21,217,32,180,318,9,137,2,11,22,291,7,177,16,31,165,197 }, +{ 317,115,180,365,873,498,217,341,13,752,482,197,569,352,1,901,36,23,457,468,165,346,546,143,509,134,579,876,868,2,332,21 }, +{ 184,257,205,229,152,17,57,497,266,432,452,524,5,619,381,32,4,90,2,12,313,128,45,59,245,106,3,471,129,769,339,214 }, +{ 13,23,1,0,15,2,4,515,51,3,10,33,5,700,115,180,753,77,11,365,341,217,9,6,197,7,102,165,317,40,22,64 }, +{ 626,70,771,687,379,846,767,761,518,878,82,481,31,786,49,591,178,163,407,44,87,13,845,125,590,371,195,120,98,557,937,351 }, +{ 264,1,410,909,772,897,686,521,335,478,98,96,691,639,100,44,284,382,31,321,744,88,914,724,662,765,223,9,682,363,0,367 }, +{ 13,23,1,2,0,15,51,515,5,10,4,33,115,77,180,700,3,141,217,40,6,753,317,197,64,165,7,11,102,9,341,22 }, +{ 141,77,13,64,269,23,115,21,318,217,5,202,102,33,137,2,15,291,177,51,48,180,32,4,515,352,128,7,0,10,96,11 }, +{ 13,15,23,515,51,0,700,753,1,2,4,10,33,11,961,453,115,40,457,14,12,3,9,5,165,401,197,77,22,21,64,102 }, +{ 1,22,0,36,2,31,5,12,13,105,28,9,49,86,141,21,23,95,128,55,44,115,170,10,164,98,180,4,137,239,83,51 }, +{ 100,101,88,0,252,9,265,30,21,39,759,724,213,329,321,13,419,68,562,382,676,352,694,35,553,410,1,166,909,593,230,23 }, +{ 539,88,30,190,321,530,840,144,669,435,957,748,778,100,96,418,203,213,1,131,410,228,466,274,36,382,219,863,613,83,822,352 }, +{ 7,97,92,173,298,107,43,314,232,140,16,356,20,387,729,362,126,359,246,14,230,501,426,441,0,5,560,66,104,779,35,60 }, +{ 2,14,7,1,58,5,46,16,38,70,75,45,24,155,29,0,21,52,61,163,220,50,69,270,35,48,32,171,18,6,64,54 }, +{ 447,411,363,664,647,98,621,1,354,271,223,478,18,777,781,936,360,759,167,132,121,48,21,156,9,195,118,293,23,691,13,264 }, +{ 2,1,14,65,36,67,0,7,46,22,69,5,38,24,28,6,83,29,86,114,168,50,124,208,12,18,108,10,194,484,103,16 }, +{ 421,386,51,791,730,958,165,801,23,453,697,403,615,13,221,523,24,899,401,326,551,670,576,102,18,33,125,77,566,115,203,197 }, +{ 104,319,422,945,0,81,20,43,715,32,784,693,879,7,397,74,306,207,52,681,671,2,61,173,6,636,904,95,887,5,18,192 }, +{ 2,29,1,46,14,52,70,262,6,26,50,67,75,96,309,38,103,112,58,19,5,163,145,83,74,220,223,357,24,69,331,25 }, +{ 786,929,590,771,687,626,941,178,465,259,70,5,13,21,35,534,107,518,132,49,878,48,146,121,379,279,31,767,147,195,108,125 }, +{ 5,2,186,45,17,29,48,50,14,61,46,155,400,1,227,171,52,58,38,54,430,209,80,281,3,106,536,311,181,243,21,502 }, +{ 0,32,64,95,817,494,342,403,207,202,194,389,453,365,312,180,316,5,690,237,848,577,450,61,102,523,475,289,49,241,65,482 }, +{ 23,13,51,15,202,515,1,0,2,12,115,4,700,5,165,753,77,457,21,141,128,10,3,64,403,32,197,318,9,11,33,117 }, +{ 119,2,6,76,5,17,45,292,306,240,32,1,19,84,64,61,4,209,710,80,26,0,106,27,214,25,128,129,29,179,3,113 }, +{ 15,515,700,1,753,0,13,23,180,120,51,2,5,33,165,197,9,450,7,260,115,523,4,12,202,141,82,77,21,102,8,6 }, +{ 6,138,74,280,222,85,66,226,25,42,87,204,64,337,29,135,95,174,235,26,145,65,19,32,792,294,112,52,256,2,5,22 }, +{ 39,9,0,101,333,158,49,252,310,254,272,68,16,18,159,286,344,455,30,109,627,327,24,105,419,100,364,22,35,1,329,709 }, +{ 21,5,32,14,2,1,38,72,76,54,17,3,48,221,270,0,45,46,12,181,37,89,36,50,540,290,430,10,4,741,99,23 }, +{ 120,137,202,269,141,260,318,450,922,494,77,291,82,15,5,351,128,1,515,326,64,854,700,352,342,21,753,678,349,32,523,90 }, +{ 16,24,68,35,71,18,149,118,191,167,9,0,212,124,65,246,7,67,140,189,399,101,133,60,1,108,267,114,69,92,695,154 }, +{ 28,0,9,67,1,22,109,36,55,65,194,114,39,83,49,69,2,35,103,50,158,208,86,420,168,289,505,24,7,185,5,323 }, +{ 147,325,198,427,142,178,202,798,5,376,20,318,259,43,120,450,77,234,534,904,470,465,878,725,329,14,315,0,260,858,70,61 }, +{ 28,0,65,9,109,1,55,67,35,22,24,39,289,7,175,14,114,2,158,124,420,194,68,16,336,36,49,69,168,570,154,505 }, +{ 141,180,13,115,1,23,4,269,2,202,0,317,217,51,15,5,21,352,77,318,3,752,197,10,165,365,137,341,9,515,33,64 }, +{ 15,515,700,0,753,1,2,13,23,9,51,5,4,901,33,7,8,3,12,93,180,120,197,6,82,341,10,141,22,260,457,115 }, +{ 6,74,66,1,25,75,324,380,278,26,138,85,135,500,87,42,220,841,97,350,29,19,70,226,38,21,52,606,235,889,2,14 }, +{ 13,23,51,12,4,15,453,0,457,1,403,165,115,3,11,2,64,5,401,10,515,37,202,33,40,32,99,475,197,700,308,17 }, +{ 788,180,5,83,693,319,314,4,32,21,17,11,817,3,510,498,33,12,24,104,814,120,64,117,306,804,523,450,288,160,102,43 }, +{ 15,1,515,13,2,700,23,0,4,753,3,341,5,51,33,11,10,6,77,7,115,102,180,165,141,9,197,217,901,40,12,64 }, +{ 101,18,9,167,520,16,0,118,60,212,604,364,694,24,55,252,68,917,264,35,1,121,146,363,39,100,806,5,21,166,191,28 }, +{ 13,23,0,33,51,141,77,4,64,2,115,217,9,102,7,202,21,10,180,3,15,128,5,269,6,32,11,16,165,352,22,317 }, +{ 66,135,97,74,172,6,278,7,204,324,138,174,29,85,2,87,25,140,92,192,52,38,802,69,448,500,808,620,22,1,280,232 }, +{ 2,24,69,6,97,7,1,0,14,298,423,66,67,29,150,25,189,267,124,74,607,18,36,81,172,33,83,38,52,273,71,809 }, +{ 0,9,158,39,68,49,109,16,24,333,35,344,101,22,159,254,272,30,124,65,28,18,793,154,310,252,327,105,627,419,286,55 }, +{ 15,515,700,10,753,33,77,180,4,341,1,197,13,115,365,23,901,317,5,102,11,217,165,117,141,40,2,3,253,21,134,55 }, +{ 100,166,382,478,265,264,88,39,98,1,404,274,9,593,724,921,639,438,363,682,411,31,30,812,96,447,821,905,252,0,223,435 }, +{ 19,283,436,53,297,26,813,432,27,128,42,25,390,503,122,736,73,123,605,63,389,529,630,250,690,65,381,444,6,269,108,216 }, +{ 33,202,13,128,494,0,51,141,269,1,4,2,102,180,15,137,65,95,6,450,77,40,117,59,457,36,196,817,134,86,49,515 }, +{ 18,16,60,68,101,167,191,118,35,121,9,212,55,0,126,1,24,647,199,146,520,107,628,621,363,71,21,28,346,92,806,727 }, +{ 352,230,217,531,160,93,36,669,748,854,689,258,137,871,728,699,752,251,574,202,373,351,228,120,717,260,144,219,268,82,816,1 }, +{ 15,515,700,753,13,23,0,51,8,1,4,82,165,77,110,33,10,180,5,202,11,22,120,12,9,197,115,93,403,141,40,351 }, +{ 0,9,101,217,35,88,352,100,39,175,30,68,562,752,13,317,252,115,180,197,64,1,83,141,65,213,165,230,194,36,28,265 }, +{ 32,64,5,470,288,90,21,147,0,95,356,22,20,519,835,312,819,18,247,182,11,97,13,4,387,49,43,298,316,48,107,7 }, +{ 16,14,24,1,7,2,35,0,5,50,18,69,46,12,58,75,9,67,70,163,21,54,38,48,223,502,281,37,140,60,28,10 }, +{ 18,265,9,252,39,195,354,411,1,16,132,101,121,682,167,203,5,363,146,593,35,333,21,271,60,13,100,0,156,327,7,520 }, +{ 4,13,1,115,141,23,2,180,5,0,51,3,217,202,77,15,33,269,341,318,317,21,165,515,11,10,197,365,9,137,64,352 }, +{ 7,24,16,14,71,35,18,92,140,189,108,149,68,60,69,150,2,230,97,66,0,458,67,1,65,251,38,314,388,267,36,46 }, +{ 1,22,31,36,0,2,44,5,141,105,180,170,12,64,13,98,86,55,23,21,28,164,115,127,10,125,128,4,9,239,352,197 }, +{ 66,7,97,2,192,20,52,43,135,74,560,107,104,0,750,147,414,29,580,173,324,376,226,194,77,174,204,38,356,64,16,470 }, +{ 0,101,9,68,35,124,24,65,39,16,252,100,154,166,28,364,149,694,30,88,55,346,1,419,71,439,265,289,22,21,175,158 }, +{ 39,9,100,30,127,0,737,856,31,836,827,254,98,931,166,88,93,1,44,190,131,228,120,395,625,385,863,264,219,373,110,28 }, +{ 5,17,2,3,21,45,14,155,48,32,38,1,328,181,186,46,23,51,12,61,227,29,106,54,99,133,62,832,13,37,514,543 }, +{ 131,613,30,224,228,716,274,100,827,406,219,856,39,190,31,88,1,166,9,44,829,863,931,93,0,187,625,924,127,98,137,254 }, +{ 352,217,64,141,752,269,180,864,437,372,954,115,498,177,77,349,317,318,579,291,947,197,247,0,23,717,237,304,128,457,776,678 }, +{ 141,217,180,317,352,115,15,341,1,23,13,365,515,752,2,64,0,5,498,700,372,165,51,237,753,77,244,197,137,4,21,253 }, +{ 5,2,17,14,1,45,3,38,21,29,181,58,46,48,50,133,114,171,61,155,32,6,186,281,361,12,36,54,4,13,52,514 }, +{ 6,2,119,25,790,4,45,483,655,113,1,29,76,26,32,19,887,17,128,0,292,833,59,61,106,64,77,814,14,151,84,42 }, +{ 13,23,2,0,51,4,1,115,141,3,5,180,33,217,77,9,202,11,7,15,10,6,317,64,21,197,515,165,102,128,22,269 }, +{ 13,23,51,202,21,5,1,15,141,165,120,2,115,12,32,0,515,128,318,64,125,700,4,403,197,453,180,457,3,7,10,6 }, +{ 9,39,31,30,0,127,1,44,100,131,98,187,385,276,88,442,219,908,254,116,49,166,935,28,201,36,141,827,137,299,284,21 }, +{ 22,49,28,109,9,185,105,95,1,131,159,272,36,67,86,254,39,55,35,0,505,31,83,169,208,327,286,98,168,535,312,708 }, +{ 5,1,2,21,0,12,48,22,10,14,36,3,32,17,23,54,86,38,4,51,13,37,137,50,65,281,114,45,28,99,58,202 }, +{ 141,269,352,82,217,351,180,854,372,922,752,1,15,260,317,318,515,202,64,700,120,349,954,753,77,35,67,717,898,137,365,115 }, +{ 144,203,613,418,326,406,96,669,137,679,1,228,494,822,840,317,36,83,855,160,817,859,856,816,217,831,345,93,876,77,44,251 }, +{ 15,13,515,23,700,753,51,1,33,0,202,21,2,5,180,141,120,165,217,82,12,117,4,352,269,197,115,32,3,9,134,260 }, +{ 13,23,115,1,0,51,4,77,2,33,15,141,10,5,341,180,515,3,217,202,9,365,317,64,700,102,11,165,197,22,753,7 }, +{ 531,943,373,160,728,93,206,260,261,559,964,269,717,535,332,384,365,295,110,533,141,10,180,352,244,137,120,55,959,564,36,253 }, +{ 1,31,36,170,22,55,44,10,86,64,127,0,2,98,301,164,740,338,237,143,5,125,116,13,242,141,299,180,23,169,105,12 }, +{ 13,23,141,77,51,4,64,32,33,202,115,269,102,128,21,0,177,180,318,90,40,10,7,5,137,15,217,352,9,291,59,22 }, +{ 202,120,260,318,77,15,450,269,1,82,33,23,141,13,51,515,351,128,700,5,64,326,137,21,102,110,753,494,93,523,817,165 }, +{ 523,899,102,33,730,15,23,403,719,117,153,13,515,51,475,4,700,5,453,817,196,753,494,40,202,120,1,2,450,457,17,421 }, +{ 202,403,494,450,120,817,523,475,318,453,33,402,128,77,13,51,260,576,342,102,15,23,515,4,700,5,82,753,326,210,137,615 }, +{ 2,29,112,66,7,52,70,151,58,87,135,5,74,226,307,6,14,186,1,45,549,172,644,25,113,287,46,155,334,64,294,97 }, +{ 1,77,349,291,260,120,652,102,5,39,64,269,9,33,340,342,13,98,888,698,23,296,100,318,51,202,87,137,638,128,50,850 }, +{ 1,13,15,2,0,4,23,515,5,141,180,3,700,341,115,51,753,269,77,901,197,352,217,33,21,11,365,6,165,202,7,317 }, +{ 7,14,16,2,46,5,70,58,1,38,24,35,92,163,0,75,21,18,50,54,140,12,87,220,155,69,171,23,60,9,13,307 }, +{ 2,29,66,226,135,7,87,74,52,278,6,75,222,220,294,70,97,1,145,25,172,262,324,38,69,112,331,92,5,14,140,26 }, +{ 15,515,700,753,33,77,117,4,1,102,134,40,153,11,13,196,217,21,5,51,23,115,32,3,2,202,141,137,128,291,48,177 }, +{ 15,217,82,515,351,141,317,1,13,700,260,77,110,120,115,854,23,753,180,51,21,36,137,922,5,64,365,352,291,202,93,341 }, +{ 31,190,30,373,120,110,863,88,44,127,908,856,260,318,82,98,93,187,836,717,935,39,442,131,141,254,228,219,1,968,77,116 }, +{ 23,13,308,9,165,115,51,21,401,125,49,39,197,391,159,254,217,743,28,438,773,629,558,386,341,95,32,317,876,679,109,166 }, +{ 49,5,43,165,7,0,21,104,125,22,173,422,64,13,623,102,20,18,314,95,91,141,23,31,193,51,391,900,779,558,92,232 }, +{ 1,4,13,2,15,0,23,515,77,3,341,33,5,700,115,51,202,753,141,180,11,10,102,217,6,901,40,7,197,318,317,365 }, +{ 93,843,295,120,36,160,206,261,10,137,567,110,384,141,943,268,201,332,258,55,1,180,64,116,44,144,699,203,282,31,260,373 }, +{ 81,7,192,426,43,173,172,104,879,91,5,712,715,526,6,97,568,95,448,66,33,861,560,32,49,20,0,636,232,825,2,22 }, +{ 475,403,51,453,33,102,13,23,494,202,0,196,15,77,153,18,4,117,515,450,318,22,730,128,700,421,65,753,269,402,134,817 }, +{ 141,269,260,318,202,120,352,349,82,351,1,5,854,137,64,291,15,922,180,851,32,77,515,372,21,700,7,217,13,947,33,753 }, +{ 15,515,700,753,1,13,0,2,4,23,3,5,180,115,197,12,51,165,217,10,961,9,6,141,352,21,8,7,33,77,457,120 }, +{ 39,166,9,30,0,101,274,404,252,333,190,100,158,438,310,88,68,265,656,21,1,530,329,344,49,539,625,254,13,131,48,419 }, +{ 0,1,28,9,22,12,65,83,67,36,5,2,50,55,96,109,16,13,24,23,21,238,49,18,285,160,128,39,69,114,7,323 }, +{ 15,515,700,1,0,753,23,2,13,51,5,180,115,6,3,9,197,12,457,120,7,165,901,82,4,21,8,141,31,33,719,341 }, +{ 9,39,30,0,28,166,22,49,180,1,352,35,317,158,88,141,498,131,115,345,752,128,228,217,100,83,219,930,13,251,365,36 }, +{ 5,61,45,2,80,29,311,209,6,17,58,1,151,106,454,667,243,70,52,496,287,592,255,738,64,74,483,14,27,32,112,19 }, +{ 31,125,22,44,299,456,685,242,599,116,170,28,0,1,492,393,506,144,558,10,268,301,239,23,13,36,963,367,55,206,105,95 }, +{ 187,258,926,574,839,93,228,860,406,219,871,160,137,531,224,116,120,902,669,201,36,131,44,144,843,533,318,384,442,1,434,268 }, +{ 7,107,75,16,87,9,64,177,24,18,291,77,349,141,60,232,23,0,51,269,132,14,5,21,70,32,678,112,126,121,71,947 }, +{ 15,515,700,753,1,0,2,13,5,3,23,180,4,115,901,51,6,8,961,9,7,10,12,82,197,22,141,341,33,120,365,457 }, +{ 13,23,51,1,5,202,2,12,15,21,165,141,0,115,3,4,32,515,197,10,180,318,128,120,64,700,6,7,403,269,457,137 }, +{ 1,2,0,77,64,3,141,13,33,15,23,10,6,102,5,515,180,4,117,7,700,165,11,217,269,40,753,115,128,17,197,134 }, +{ 345,531,332,269,260,317,717,752,373,351,180,352,728,82,10,365,160,533,217,143,498,251,244,93,341,901,36,1,141,898,55,864 }, +{ 16,7,33,189,92,77,388,60,140,35,102,24,14,1,230,21,150,117,733,314,18,915,71,13,108,134,5,64,69,2,98,22 }, +{ 142,202,234,178,5,786,77,49,70,0,416,450,639,878,1,48,21,929,147,259,315,455,198,120,12,481,163,113,846,329,318,22 }, +{ 81,715,192,0,173,712,681,104,636,91,74,20,750,370,7,718,95,879,22,43,825,560,422,64,207,49,172,18,397,10,426,319 }, +{ 13,23,15,51,515,0,700,753,4,1,961,2,10,115,457,11,33,453,3,5,9,40,12,197,165,77,401,475,64,102,22,569 }, +{ 64,297,5,445,95,61,250,311,80,34,17,312,45,2,86,472,58,14,180,53,22,151,869,738,247,237,29,1,128,165,21,288 }, +{ 16,24,18,71,7,35,118,92,14,154,60,68,0,149,28,302,124,150,55,175,2,9,97,1,429,20,108,273,22,65,43,126 }, +{ 195,360,156,771,132,163,626,687,591,371,883,146,121,846,70,586,379,13,293,98,407,48,761,296,354,18,31,1,55,49,21,105 }, +{ 202,13,77,23,318,33,51,0,4,141,5,21,217,32,291,102,64,128,15,10,9,494,269,137,515,403,1,31,117,700,120,317 }, +{ 7,192,97,81,172,66,426,173,43,715,712,232,861,879,104,330,568,298,74,893,885,526,387,825,92,140,91,14,636,6,5,448 }, +{ 2,1,14,6,67,7,65,69,24,36,66,124,108,83,38,29,22,86,0,18,484,5,28,46,12,10,25,302,150,16,650,74 }, +{ 33,77,13,202,102,4,0,23,128,51,141,64,318,22,403,269,137,10,15,40,494,117,32,59,11,153,1,21,177,196,515,115 }, +{ 269,141,318,77,349,291,217,202,33,15,372,304,515,22,102,177,351,700,352,120,5,137,10,317,260,753,64,851,854,403,49,21 }, +{ 6,74,66,85,138,25,87,42,135,26,226,222,280,29,75,500,220,278,792,70,19,2,1,294,204,64,32,145,853,112,52,174 }, +{ 9,0,105,39,16,18,1,101,272,31,127,98,24,518,333,252,310,28,68,737,846,371,158,916,938,49,30,7,286,35,301,455 }, +{ 2,1,14,67,24,46,83,108,69,29,38,103,114,36,6,133,18,0,28,7,65,52,236,75,50,398,5,309,135,16,278,160 }, +{ 22,1,28,105,49,95,0,2,67,55,36,239,168,159,65,35,14,170,320,164,9,7,10,5,114,12,83,64,194,109,24,301 }, +{ 7,66,172,97,92,140,232,568,298,14,192,314,16,380,135,324,2,330,74,38,357,448,126,69,35,5,107,6,387,60,204,572 }, +{ 2,29,1,14,5,6,46,133,114,50,52,26,218,108,19,13,366,236,27,45,70,17,58,23,86,51,137,65,112,38,25,12 }, +{ 6,1,74,25,2,26,29,66,42,19,75,14,388,67,108,70,52,85,103,65,38,138,357,133,114,594,324,516,603,96,309,69 }, +{ 22,9,28,1,36,49,109,105,86,95,131,31,169,39,0,141,272,159,44,55,98,180,13,30,185,115,83,128,352,137,64,208 }, +{ 203,822,326,23,77,859,403,494,576,39,473,182,33,1,691,100,18,217,13,817,411,447,363,102,93,966,96,478,291,704,310,120 }, +{ 15,515,700,753,33,77,117,4,102,134,115,153,13,1,40,217,11,196,341,2,5,3,23,317,365,0,21,291,32,51,12,569 }, +{ 15,515,700,753,13,0,23,1,8,82,51,165,197,120,180,2,9,33,4,110,5,12,10,260,351,386,141,7,457,475,93,901 }, +{ 1,23,13,15,51,0,21,2,515,5,141,180,120,165,700,202,197,4,753,12,33,9,7,82,115,93,3,352,260,6,110,48 }, +{ 15,515,700,753,0,1,2,5,3,4,8,13,180,341,10,23,7,6,9,51,77,197,961,115,165,82,120,31,22,202,457,217 }, +{ 559,661,922,564,141,533,10,317,373,110,143,269,244,260,332,261,93,642,752,295,351,876,531,843,180,206,728,384,352,1,434,120 }, +{ 15,515,1,0,700,2,13,23,753,5,51,180,3,165,12,6,197,115,4,9,7,21,719,8,457,82,141,120,33,22,901,10 }, +{ 198,234,0,325,5,77,202,416,20,147,32,43,639,315,49,61,450,455,142,21,113,230,22,318,725,342,207,13,95,904,494,10 }, +{ 447,264,363,9,411,676,682,1,156,664,821,478,166,354,812,39,100,905,382,897,98,18,759,404,31,101,724,5,265,223,88,13 }, +{ 24,14,69,16,35,18,2,7,108,189,71,67,267,149,1,46,68,83,38,140,0,236,251,9,388,60,133,103,65,28,29,50 }, +{ 16,7,35,20,14,18,109,2,43,120,107,60,1,121,326,907,553,77,13,147,23,82,68,260,0,403,5,24,202,126,265,199 }, +{ 30,131,187,276,31,44,613,442,39,9,190,228,1,839,116,935,908,219,127,88,244,224,110,137,93,201,98,141,36,567,0,856 }, +{ 98,223,1,393,812,265,100,421,593,834,697,48,51,410,791,382,21,88,31,284,9,125,96,293,230,23,213,217,656,689,541,5 }, +{ 98,51,127,219,616,258,105,293,395,421,924,512,31,308,23,201,116,44,301,272,763,276,125,13,453,170,401,295,261,944,115,567 }, +{ 253,110,951,352,811,206,332,180,141,244,282,10,854,417,642,638,559,752,143,911,260,55,93,533,499,498,661,120,351,959,564,341 }, +{ 49,9,159,254,272,158,0,131,28,39,627,105,327,286,22,518,688,578,68,347,374,101,224,424,95,35,219,24,16,364,65,344 }, +{ 105,22,131,272,98,286,327,109,374,239,28,95,320,219,9,224,55,127,187,36,578,169,64,185,538,1,159,10,371,634,49,616 }, +{ 691,478,340,1,658,914,724,363,744,698,156,772,411,296,682,447,9,284,335,98,264,303,909,21,354,410,225,13,664,686,88,919 }, +{ 16,24,35,18,71,7,140,108,189,267,92,60,14,230,68,69,9,1,149,46,246,191,388,167,2,0,118,236,133,21,674,5 }, +{ 9,0,127,31,98,371,395,39,737,49,1,44,385,272,512,28,293,242,836,761,254,299,101,16,187,22,116,158,159,131,18,21 }, +{ 16,18,68,0,60,35,9,101,252,28,118,24,419,55,7,109,604,71,39,121,22,364,14,158,191,167,925,126,329,21,92,49 }, +{ 116,268,203,93,206,692,551,31,417,940,499,8,473,44,202,523,959,0,120,137,559,22,450,403,576,10,728,299,13,326,51,1 }, +{ 225,459,744,1,919,914,691,330,622,21,141,223,5,284,934,335,88,538,340,82,385,839,363,120,478,98,48,30,64,32,686,166 }, +{ 5,1,2,0,14,36,21,281,12,48,50,67,22,28,54,83,24,218,38,10,181,9,32,18,65,58,45,114,430,17,99,37 }, +{ 137,450,202,704,120,260,326,318,968,269,851,403,291,77,23,141,182,310,494,373,351,457,82,890,349,110,60,128,817,678,105,96 }, +{ 15,515,700,753,4,33,13,23,77,5,40,11,102,93,1,21,110,51,82,117,141,2,10,8,32,64,120,31,202,3,217,115 }, +{ 15,13,1,23,515,0,51,2,700,5,753,21,180,141,165,3,12,115,197,4,7,6,457,9,352,202,33,8,719,120,77,341 }, +{ 219,127,258,98,276,201,131,395,944,293,116,284,567,31,242,105,137,935,295,44,403,860,51,224,576,456,9,371,578,475,202,512 }, +{ 16,7,18,35,60,0,14,20,118,28,68,22,2,24,1,92,158,107,5,49,154,126,109,12,43,10,55,6,677,71,21,168 }, +{ 93,728,531,160,559,373,574,120,295,860,533,269,717,260,926,902,258,318,36,201,261,434,851,137,617,141,187,352,843,384,332,251 }, +{ 28,0,1,9,22,109,83,39,49,12,36,67,55,5,96,2,128,30,158,69,21,23,160,208,35,13,65,323,50,141,194,238 }, +{ 23,13,51,1,5,15,141,21,0,2,165,515,202,700,12,197,180,120,32,115,4,753,64,9,7,269,6,3,125,386,48,453 }, +{ 5,21,13,49,14,20,7,23,43,32,1,0,652,48,713,22,38,2,16,132,955,107,12,279,24,888,197,640,70,303,18,638 }, +{ 9,0,28,39,1,30,35,101,22,67,83,141,49,175,36,68,55,88,13,251,10,69,23,158,180,115,64,100,217,65,345,166 }, +{ 260,120,82,269,5,450,351,1,202,141,854,13,77,922,32,33,137,4,23,125,291,21,15,515,165,349,177,700,318,326,180,753 }, +{ 121,16,18,35,363,101,60,20,107,14,68,259,621,55,604,43,7,252,9,364,126,0,167,191,5,407,132,28,199,419,146,10 }, +{ 13,23,51,1,15,0,2,141,5,515,12,21,700,115,165,180,4,753,3,197,202,32,9,120,7,8,6,11,37,10,457,269 }, +{ 201,144,206,443,418,203,435,96,335,459,187,1,88,332,330,321,269,934,30,372,822,521,268,326,44,523,382,141,410,264,494,473 }, +{ 31,44,276,201,116,131,284,662,567,144,9,489,98,295,268,434,0,30,137,39,93,1,187,22,219,918,110,299,141,36,224,384 }, +{ 520,478,664,1,264,604,9,167,777,759,411,0,806,724,48,21,101,68,647,936,363,223,118,682,410,18,100,16,252,98,265,13 }, +{ 23,70,21,87,60,75,120,182,163,379,92,18,7,937,71,121,446,132,24,98,931,126,107,77,795,195,115,44,411,146,51,850 }, +{ 39,9,0,737,127,31,846,98,1,827,105,310,371,30,254,100,44,18,395,242,272,101,385,916,836,16,265,131,938,93,166,557 }, +{ 279,20,43,126,107,7,92,16,356,362,60,595,246,359,598,35,0,683,939,653,121,97,125,441,399,392,150,199,48,230,14,649 }, +{ 206,417,93,959,499,728,8,559,120,473,137,141,10,564,31,260,44,450,203,341,253,244,373,116,143,638,268,180,352,110,318,940 }, +{ 259,465,147,132,590,687,534,199,581,146,941,427,107,640,279,178,121,5,195,150,522,955,198,35,786,929,798,142,1,21,325,626 }, +{ 523,15,120,450,202,515,403,51,817,700,13,753,23,457,33,899,128,64,730,102,494,342,115,719,453,196,49,99,318,421,308,5 }, +{ 141,559,10,244,365,564,661,180,253,143,752,110,55,317,533,341,901,93,373,206,535,160,82,922,260,36,531,964,352,332,261,197 }, +{ 219,258,98,127,276,964,943,137,843,535,201,935,131,860,261,295,284,567,206,44,116,31,253,492,203,332,160,615,36,93,55,692 }, +{ 0,319,422,207,945,693,577,887,32,804,95,344,104,904,61,20,5,43,7,725,113,510,306,102,49,263,153,426,33,83,22,9 }, +{ 1,22,0,5,12,2,36,21,28,86,49,105,9,10,23,13,141,95,31,55,128,37,51,4,83,202,3,64,96,7,32,44 }, +{ 15,515,13,700,1,0,753,2,23,3,4,5,51,10,115,197,6,33,12,9,165,7,8,77,11,961,180,269,141,22,120,457 }, +{ 15,1,23,120,77,13,515,51,141,202,700,180,110,137,260,753,326,5,128,102,0,21,2,165,269,33,197,450,318,217,93,115 }, +{ 15,515,1,700,0,753,13,2,23,180,51,5,120,4,9,115,197,12,7,165,21,33,6,82,3,8,523,901,31,141,457,260 }, +{ 16,18,24,7,92,35,60,75,9,13,71,14,0,108,50,21,126,121,1,140,23,5,132,146,2,12,128,10,64,141,70,87 }, +{ 180,341,901,15,515,1,365,700,0,2,197,753,115,4,10,13,752,5,3,8,165,317,141,23,143,873,44,31,569,55,93,6 }, +{ 9,0,175,35,101,28,39,67,68,1,65,83,30,69,364,336,22,114,55,124,194,158,100,289,252,166,64,345,103,36,50,88 }, +{ 64,165,180,197,115,247,217,237,21,13,32,316,22,141,352,72,288,304,95,225,76,391,386,16,468,90,49,35,365,640,372,23 }, +{ 15,515,700,13,753,1,0,2,23,4,5,3,115,51,141,197,12,10,180,961,7,9,21,33,217,6,8,165,457,11,77,341 }, +{ 132,121,199,146,60,279,493,640,407,598,126,195,534,581,955,590,107,5,150,35,522,49,259,16,18,360,156,0,147,362,21,167 }, +{ 0,1,28,9,22,5,36,12,65,24,67,96,2,83,18,50,114,55,21,16,7,10,23,14,13,160,137,51,48,218,103,69 }, +{ 93,120,957,77,30,968,459,110,137,160,613,102,202,352,373,141,31,372,217,330,190,318,269,260,203,44,28,473,228,177,863,704 }, +{ 15,515,700,0,753,1,13,23,2,51,5,9,120,82,4,7,901,197,10,8,260,180,341,12,33,6,3,523,165,102,115,141 }, +{ 206,417,8,141,499,44,244,93,31,10,137,253,559,116,728,144,120,564,269,638,203,352,143,260,341,752,268,717,951,180,160,110 }, +{ 530,254,228,1,96,21,406,39,827,31,669,840,613,829,137,679,166,98,23,51,960,438,131,93,48,224,219,317,310,36,876,190 }, +{ 15,515,700,753,13,457,0,197,719,1,165,82,23,8,120,730,2,10,12,180,134,5,9,141,260,4,351,51,115,3,341,899 }, +{ 0,16,68,9,24,28,18,35,252,109,39,419,124,158,154,55,101,71,22,118,60,7,49,65,333,14,1,10,329,364,677,346 }, +{ 1,15,13,23,515,51,120,0,700,180,2,165,5,753,141,197,21,33,202,102,260,4,9,12,7,326,137,450,115,6,82,110 }, +{ 535,253,352,564,110,365,82,180,341,10,854,533,55,898,244,901,873,141,752,143,642,559,498,317,36,951,115,964,638,282,661,197 }, +{ 31,44,125,338,116,64,242,36,1,10,55,22,456,237,180,13,299,164,506,86,23,165,558,143,0,762,492,479,844,546,93,8 }, +{ 13,23,4,1,202,2,0,51,115,77,141,180,5,15,217,3,33,11,515,317,9,10,102,21,700,341,365,318,269,64,32,128 }, +{ 9,39,0,166,68,101,28,364,30,158,562,35,175,65,333,154,49,404,706,124,21,252,274,168,190,289,100,570,16,1,310,346 }, +{ 15,515,700,753,341,13,0,23,1,33,141,4,260,82,77,51,351,180,9,5,115,137,10,217,11,120,102,40,349,269,202,854 } diff --git a/src/gfx/tex/bc7key.c b/src/gfx/tex/bc7key.c new file mode 100644 index 0000000..ef48ed5 --- /dev/null +++ b/src/gfx/tex/bc7key.c @@ -0,0 +1,23 @@ +// The key of a surface in the BC7 cache: two 64-bit lanes over the bytes, the size and block size mixed in first. +#include + +#include "bc7key.h" + +static uint64_t mix(uint64_t h, uint64_t v) { h ^= v * 0x9E3779B97F4A7C15ull; h = (h << 27 | h >> 37) * 0xC2B2AE3D27D4EB4Full; return h; } +static uint64_t fin(uint64_t h) { h ^= h >> 33; h *= 0xFF51AFD7ED558CCDull; h ^= h >> 33; h *= 0xC4CEB9FE1A85EC53ull; h ^= h >> 33; return h; } + +bc7_key bc7_key_of(const uint8_t *blocks, size_t n, uint32_t w, uint32_t h, uint32_t depth, uint32_t bw, uint32_t bh) { + uint64_t a = 0x5432102030405060ull, b = 0x0123456789ABCDEFull; + uint64_t p0 = (uint64_t)w << 32 | h, p1 = (uint64_t)depth << 32 | bw << 16 | bh << 8; + a = mix(a, p0); a = mix(a, p1); b = mix(b, p1); b = mix(b, p0); + size_t i = 0; + for (; i + 16 <= n; i += 16) { + uint64_t x, y; + memcpy(&x, blocks + i, 8); memcpy(&y, blocks + i + 8, 8); + a = mix(a, x); b = mix(b, y); + } + for (; i < n; i++) { a = mix(a, blocks[i]); b = mix(b, (uint64_t)blocks[i] << 8); } + bc7_key k = {fin(a ^ n), fin(b ^ (n << 1))}; + if (!k.a && !k.b) k.a = 1; // 0,0 marks an empty slot in the cache's index + return k; +} diff --git a/src/gfx/tex/bc7key.h b/src/gfx/tex/bc7key.h new file mode 100644 index 0000000..50f9c57 --- /dev/null +++ b/src/gfx/tex/bc7key.h @@ -0,0 +1,10 @@ +// The key of a surface in the BC7 cache (src/gfx/vkb_bc7.c, tools/bc7build): shared by the game and the build, which +// must compute the same key for the same texture whichever way it was laid out. +#pragma once +#include +#include + +typedef struct { uint64_t a, b; } bc7_key; +// blocks: the surface's ASTC blocks in rows (n bytes; a 3D level: its slices one after another); w x h texels, depth +// slices, block size bw x bh. The colour space is not part of it: the BC7 bytes serve an sRGB and a UNORM view alike. +bc7_key bc7_key_of(const uint8_t *blocks, size_t n, uint32_t w, uint32_t h, uint32_t depth, uint32_t bw, uint32_t bh); diff --git a/src/gfx/tex/nvn_deswizzle.c b/src/gfx/tex/nvn_deswizzle.c new file mode 100644 index 0000000..a62fa98 --- /dev/null +++ b/src/gfx/tex/nvn_deswizzle.c @@ -0,0 +1,103 @@ +// Tegra X1 块线性排布（公开描述见 deko3d / Ryujinx BlockLinearLayout）： +// GOB = 64 字节 × 8 行 = 512 字节；GOB 内 (x 字节, y 行) 的偏移 = +// (x%64/32)·256 + (y%8/2)·64 + (x%32/16)·32 + (y%2)·16 + x%16 +// 块 = 1 个 GOB 宽 × 2^tileH 个 GOB 高 × 2^tileD 个 GOB 深，块内先排 y 再排 z；块按 x → y → z 行优先铺满本级。 +// 级 / 层起点与本级实际块高块深取自 nvn_tex.c，这里只做级内寻址与搬运。 +// Tegra X1 block-linear layout (public descriptions: deko3d / Ryujinx BlockLinearLayout): +// GOB = 64 bytes × 8 rows = 512 bytes; offset of (x byte, y row) within a GOB = +// (x%64/32)·256 + (y%8/2)·64 + (x%32/16)·32 + (y%2)·16 + x%16 +// block = 1 GOB wide × 2^tileH GOBs high × 2^tileD GOBs deep, ordered y first then z within a block; blocks fill the level in x → y → z row-major order. +// Level / layer start offsets and the actual block height and block depth of this level come from nvn_tex.c; here we only do intra-level addressing and copying. +#include "nvn_deswizzle.h" +#include + +static uint32_t gob_off(uint32_t x, uint32_t y) { + return (x % 64 / 32) * 256 + (y % 8 / 2) * 64 + (x % 32 / 16) * 32 + (y % 2) * 16 + x % 16; +} + +uint64_t nvn_bl_offset(uint32_t x, uint32_t y, uint32_t z, uint32_t row_bytes, uint32_t rows, unsigned th, unsigned td) { + uint32_t bh = 1u << th, bd = 1u << td; + uint64_t gobs_x = (row_bytes + 63) / 64, blocks_y = (rows + 8 * bh - 1) / (8 * bh); + uint64_t block = 512ull * bh * bd; // 一个块的字节 | bytes in one block + return (z / bd) * gobs_x * blocks_y * block // 第几层块（深度方向） | which block layer (along depth) + + (y / (8 * bh)) * gobs_x * block // 第几行块 | which block row + + (x / 64) * block // 第几列块 | which block column + + ((z % bd) * bh + (y % (8 * bh)) / 8) * 512ull // 块内第几个 GOB | which GOB within the block + + gob_off(x, y); +} + +static uint32_t mip(uint32_t s, unsigned l) { s >>= l; return s ? s : 1; } + +// 本级几何：块宽字节数、块行数、切片数，以及本级本层在存储里的起点；不支持的贴图返回 -1 +// Geometry of this level: width-in-blocks byte count, block row count, slice count, and where this level and layer starts in storage; returns -1 for unsupported textures +int nvn_deswizzle_geom(const nvn_texdesc *d, unsigned level, unsigned layer, + uint32_t *row_bytes, uint32_t *rows, uint32_t *slices, uint64_t *base, unsigned *th, unsigned *td, uint32_t *bpb) { + if (d->target == 5 || d->target == 6 || d->target == 10 || nvn_tex_is_linear(d) || d->samples > 1) return -1; // MS / BUFFER / LINEAR + int bw, bh; *bpb = (uint32_t)nvn_fmt_bpb(d->format, &bw, &bh); + if (!*bpb || level >= (d->levels ? d->levels : 1)) return -1; + nvn_texlayout L; nvn_tex_layout(d, &L); + if (layer >= L.layers) return -1; + int one_d = d->target == 0 || d->target == 3; + uint32_t w = mip(d->width, level), h = one_d ? 1 : mip(d->height, level); + *slices = d->target == 2 ? mip(d->depth ? d->depth : 1, level) : 1; + *row_bytes = (w + bw - 1) / bw * *bpb; *rows = (h + bh - 1) / bh; + *base = (uint64_t)layer * L.layer_size + nvn_tex_level_offset(d, level); + nvn_tex_level_tiles(d, level, th, td); + return 0; +} + +static int move(const nvn_texdesc *d, unsigned level, unsigned layer, uint8_t *bl, uint8_t *lin, size_t pitch, int to_linear) { + if (nvn_tex_is_linear(d)) { + if (level || layer || d->samples > 1 || (d->target != 0 && d->target != 1 && d->target != 7)) return -1; + int bw, bh, bpb = nvn_fmt_bpb(d->format, &bw, &bh); + if (!bpb) return -1; + size_t row = (size_t)((d->width + bw - 1) / bw) * bpb; + uint32_t rows = d->target == 0 ? 1 : (d->height + bh - 1) / bh, stride = nvn_tex_linear_stride(d); + if (pitch < row || stride < row) return -1; + for (uint32_t y = 0; y < rows; y++) { + if (to_linear) memcpy(lin + y * pitch, bl + (size_t)y * stride, row); + else memcpy(bl + (size_t)y * stride, lin + y * pitch, row); + } + return 0; + } + uint32_t row_bytes, rows, slices, bpb; uint64_t base; unsigned th, td; + if (nvn_deswizzle_geom(d, level, layer, &row_bytes, &rows, &slices, &base, &th, &td, &bpb)) return -1; + if (pitch < row_bytes) return -1; + bl += base; + // [port] GOB by GOB: the address of each GOB is computed once, and its 8 rows x 4 16-byte sectors are copied at fixed + // offsets (a 16-byte memcpy is inlined). Per 16 bytes, the full address with its divisions and a library memcpy call + // took ~12% of the recording thread on the Great Sky Island (texture uploads, 2026-10-24). + uint32_t bh = 1u << th, bd = 1u << td, gobs_x = (row_bytes + 63) / 64, blocks_y = (rows + 8 * bh - 1) / (8 * bh); + uint64_t block = 512ull * bh * bd; + for (uint32_t z = 0; z < slices; z++) + for (uint32_t y0 = 0; y0 < rows; y0 += 8) { // a row of GOBs + uint64_t gy = (uint64_t)(z / bd) * gobs_x * blocks_y * block + (uint64_t)(y0 / (8 * bh)) * gobs_x * block + + ((uint64_t)(z % bd) * bh + (y0 % (8 * bh)) / 8) * 512ull; + uint32_t ny = rows - y0 < 8 ? rows - y0 : 8; + for (uint32_t gx = 0; gx < gobs_x; gx++) { + uint8_t *gob = bl + gy + (uint64_t)gx * block; + uint32_t x0 = gx * 64, nx = row_bytes - x0 < 64 ? row_bytes - x0 : 64; + for (uint32_t r = 0; r < ny; r++) { + uint8_t *row = lin + ((size_t)z * rows + y0 + r) * pitch + x0; + uint8_t *g = gob + (r / 2) * 64 + (r % 2) * 16; + if (nx == 64) { // the 4 sectors of the row: +0, +32, +256, +288 + if (to_linear) { memcpy(row, g, 16); memcpy(row + 16, g + 32, 16); memcpy(row + 32, g + 256, 16); memcpy(row + 48, g + 288, 16); } + else { memcpy(g, row, 16); memcpy(g + 32, row + 16, 16); memcpy(g + 256, row + 32, 16); memcpy(g + 288, row + 48, 16); } + } else + for (uint32_t x = 0; x < nx; x += 16) { // the last, partial GOB column (16 divides every block width) + uint32_t n = nx - x < 16 ? nx - x : 16; + uint8_t *p = g + (x / 32) * 256 + (x % 32 / 16) * 32; + if (to_linear) memcpy(row + x, p, n); else memcpy(p, row + x, n); + } + } + } + } + return 0; +} + +int nvn_deswizzle(const nvn_texdesc *d, unsigned level, unsigned layer, const void *src, void *dst, size_t dst_pitch) { + return move(d, level, layer, (uint8_t *)src, (uint8_t *)dst, dst_pitch, 1); +} +int nvn_swizzle(const nvn_texdesc *d, unsigned level, unsigned layer, const void *src, size_t src_pitch, void *dst) { + return move(d, level, layer, (uint8_t *)dst, (uint8_t *)src, src_pitch, 0); +} diff --git a/src/gfx/tex/nvn_deswizzle.h b/src/gfx/tex/nvn_deswizzle.h new file mode 100644 index 0000000..1d1d8a7 --- /dev/null +++ b/src/gfx/tex/nvn_deswizzle.h @@ -0,0 +1,26 @@ +// Tegra 块线性 ↔ 线性。级 / 层偏移与逐级块高块深一律取 nvn/nvn_tex.c（nvn_tex_level_offset / nvn_tex_layout / nvn_tex_level_tiles）。 +// Tegra block-linear ↔ linear. Level / layer offsets and per-level block height and block depth always come from nvn/nvn_tex.c (nvn_tex_level_offset / nvn_tex_layout / nvn_tex_level_tiles). +#pragma once +#include +#include +#include "../nvn/nvn_tex.h" + +// 把第 level 级、第 layer 层（数组 / 立方体的层；非分层贴图取 0）从整张贴图存储 src_storage 解到线性 dst_linear。 +// dst 按块行排：每行 dst_pitch 字节（≥ 本级块宽 × 每块字节），压缩格式一行 = 一行块。 +// 3D 贴图：一次解出本级全部深度切片，切片 k 位于 dst_linear + k × dst_pitch × 本级块行数。 +// 线性 / 线性渲染目标按行距复制；不支持多重采样、BUFFER，返回 -1。成功返回 0。 +// Decode level number level, layer number layer (layer of an array / cube; 0 for non-layered textures) from the whole-texture storage src_storage into linear dst_linear. +// dst is laid out by block rows: dst_pitch bytes per row (≥ width in blocks of this level × bytes per block); for compressed formats one row = one row of blocks. +// 3D textures: all depth slices of this level are decoded at once; slice k is at dst_linear + k × dst_pitch × block row count of this level. +// Linear / linear-render-target storage is copied by pitch. Multisampling and BUFFER return -1; success returns 0. +int nvn_deswizzle(const nvn_texdesc *d, unsigned level, unsigned layer, const void *src_storage, void *dst_linear, size_t dst_pitch); +// 反向（单测往返用）：线性 src_linear → 写进整张贴图存储 dst_storage 的对应级 / 层 +// Reverse (for unit test round trips): linear src_linear → written into the corresponding level / layer of the whole-texture storage dst_storage +int nvn_swizzle(const nvn_texdesc *d, unsigned level, unsigned layer, const void *src_linear, size_t src_pitch, void *dst_storage); +// 单点地址：本级本层内，线性 (x 字节, y 块行, z 切片) → 相对本级本层起点的块线性字节偏移（单测按手算核） +// Single-point address: within this level and layer, linear (x byte, y block row, z slice) → block-linear byte offset relative to the start of this level and layer (unit tests verify against hand calculation) +uint64_t nvn_bl_offset(uint32_t x, uint32_t y, uint32_t z, uint32_t row_bytes, uint32_t rows, unsigned tileH, unsigned tileD); + +int nvn_deswizzle_geom(const nvn_texdesc *d, unsigned level, unsigned layer, + uint32_t *row_bytes, uint32_t *rows, uint32_t *slices, + uint64_t *base, unsigned *tileH, unsigned *tileD, uint32_t *bpb); diff --git a/src/gfx/vkb.h b/src/gfx/vkb.h new file mode 100644 index 0000000..34693e3 --- /dev/null +++ b/src/gfx/vkb.h @@ -0,0 +1,184 @@ +// Internal interface of the Vulkan NVN replayer (backend_vk.c, vkb_tex.c). +#pragma once +#include + +#include "nvn/nvn_mtl.h" +#include "nvn/nvn_obj.h" +#include "../vk/vk.h" + +#define VKB_SLOTS 8 // command buffers in flight (backend_vk.c) + +// ---- backend_vk.c +const p_set_rt *vkb_rt(void); // current render targets +uint64_t vkb_frame(void); // presents so far +int vkb_rt_count(void); // colour targets of the current pass +VkExtent2D vkb_rt_extent(void); // the current pass's render area +int vkb_slot(void); // the command slot of the submission being replayed (per-slot rings / pools) +void vkb_gprof_report(void); // TOTK_GPU_PROF: GPU time per shader of one frame (development) +// Begins the render pass on the current targets if needed (0: not drawable); with no targets at all: a pass without +// attachments over `bare`. fb: the targets the draws also sample (bit i: colour i, VKB_FB_DEPTH: depth), which get the +// attachment feedback loop layout. +enum { VKB_FB_DEPTH = 1u << 8 }; +int vkb_pass_begin(VkExtent2D bare, uint32_t fb); +uint32_t vkb_pass_fb(void); // the current pass's feedback targets +uint64_t vkb_pass_serial(void); // changes with every pass begun +void vkb_pass_end(void); +enum { PE_OTHER, PE_SET_RT, PE_BARRIER, PE_COPY, PE_UPDATE, PE_CLEAR_TEX, PE_SUBMIT, PE_STORES, PE_SNAP, PE_FEEDBACK, PE_IMAGES, + PE_MIRROR, PE_DISPATCH, PE_TEXCOPY, PE_SHADOW, PE_COUNT }; // why a pass ended (counted, the per-frame log line) +void vkb_pass_end_r(int why); +int vkb_stores_take(void); +int vkb_scissor(VkRect2D *r); +int vkb_viewport0(float v[4]); // the scissor set by the game (0: none) +uint64_t vkb_draw_count(void); // draws and dispatches recorded so far // vkb_draw.c: storage writes since the last pass began (and forgets them) +// Destroys a view / an image and its memory once the GPU has finished every submission that may use them: the one being +// recorded now included (its earlier commands may already refer to them). +void vkb_destroy_later(VkImageView view, VkImage img, VkDeviceMemory mem); +void vkb_passprof_note(const char *key); // TOTK_GPU_PASSPROF: names the open pass by its first shader +void vkb_checkpoint(const char *what, const char *k1, const char *k2); // TOTK_VK_CHECKPOINTS: names the next GPU command +int vkb_in_pass(void); +// Host-visible upload memory of the current submission (reused after its fence): n bytes aligned to `align`. +void *vkb_ring_alloc(VkDeviceSize n, VkDeviceSize align, VkBuffer *buf, VkDeviceSize *off); +VkCommandBuffer vkb_cmd(void); // the command buffer of the submission being replayed +int vkb_render_suppressed(void); // rendering disabled or conditional (recorded state; audit) +int vkb_ready(void); // 1 when replaying on a Vulkan device + +// ---- vkb_tex.c: NVN textures as Vulkan images (one per texture serial: the game copies texture objects by value) +typedef struct vkb_image { + uint64_t serial; + nvn_texdesc d; + VkImage img; + VkDeviceMemory mem; + VkFormat format; + VkImageAspectFlags aspect; + VkImageLayout layout; // the whole image's current layout (replay thread) + uint32_t layers, levels, width, height, depth; + int astc; // ASTC in the game: decoded to RGBA8 on upload, or BC7 from the cache (bc7) + int bc7; // an ASTC texture held as BC7 (1) or BC1 (2) from the cache (vkb_bc7.c); 0: RGBA8 + int uploading; // an upload is being recorded (several threads: the others wait for it, vkb_tex_upload) + VkImageUsageFlags usage; + int compressed; // block-compressed format: no attachment / storage use + int gpu_written; // rendered / cleared / copied to by the GPU: never overwritten from game memory ([temporary]) + uint64_t upload_stamp; // nvn_dirty generation of the last upload from game memory (0 = never) + uint64_t dirty_checked; // the submission in which the game memory was last checked for CPU writes + int stale; // that check found CPU writes: upload before the next use + int cpu_written; // WriteTexels changed its game memory: upload before the next use (any pool) + int not_uploadable; // contents in game memory that the upload cannot take (depth / 3-component formats) + uint64_t vpool_gen; // in a virtual pool: the mapping generation of the last upload (0: not in one) + const uint8_t *storage; // the texture's storage in game memory (block-linear / linear) + uint64_t write_seq; // vkb_image_written: when the GPU last wrote it (0 = never) + struct vkb_image *refresh_from; // newer contents at the same memory in another size: scaled in before use (vkb_tex.c) +} vkb_image; +void vkb_image_written(vkb_image *im); // the GPU writes it (render target, clear, copy, storage) +int vkb_tex_refresh(VkCommandBuffer cb, vkb_image *im); // outside a pass: the pending refresh_from copy; 1 if recorded + +vkb_image *vkb_image_of(const tex_t *t); // creates it on first use; NULL for textures that cannot be images yet +// Moves the image to `layout` before work of `stage`/`access` (records a barrier on the current command buffer). +void vkb_image_use(VkCommandBuffer cb, vkb_image *im, VkImageLayout layout, VkPipelineStageFlags2 stage, VkAccessFlags2 access); +// A view of the image as NVN's view (NULL = the whole texture); cached per image. +// attachment: 0 sampled, 1 render target, 2 storage image. view_type: a VkImageViewType the shader samples it as, or -1 +// (from the texture / view target). VK_NULL_HANDLE when the image cannot be used so. +VkImageView vkb_view(vkb_image *im, const view_t *v, int attachment, int view_type); +// Brings the image's contents up to date from game memory when the CPU wrote them (records copies on cb; outside a pass). +// 1 when it recorded an upload. +int vkb_tex_upload(VkCommandBuffer cb, vkb_image *im); +void vkb_tex_image_stats(uint64_t *made, uint64_t *shared, uint64_t *taken_back); // images by memory and shape + +// ---- vkb_draw.c: device-local mirrors of written game memory for the current submission +int vkb_gpu_addr(uint64_t addr, uint64_t len, VkBuffer *buf, VkDeviceSize *off); // the mirror, else the imported pool +void vkb_mirrors_evict(uint64_t addr, uint64_t len); // mirrors straddling the range go back (outside a pass) +void vkb_mirrors_submit(VkCommandBuffer cb); // end of a submission: all back to game memory + +// ---- backend_vk.c / vkb_shadow.c: copies of the memory pools in device memory (discrete GPUs) +void vkb_pool_use(uint64_t addr, uint64_t len, int write); // before a GPU access is recorded (write: may end the pass) +int vkb_pool_addr(uint64_t addr, uint64_t len, VkBuffer *buf, VkDeviceSize *off); // the copy when current, else game memory +int vkb_shadow_on(void); +uint32_t vkb_shadow_alloc(VkDeviceSize size, VkBuffer *buf, VkDeviceSize *off); // 0: none +void vkb_shadow_free(uint32_t h); // reused after the submission being recorded +uint64_t vkb_shadow_bytes(void); + +// ---- vkb_ucap.c: uniform contents as of the submit call +int vkb_ucap_on(void); +uint64_t mtl_done(void); // backend_vk.c: the last submission the GPU completed +int vkb_ucap_range(uint64_t addr, uint64_t size, VkBuffer *buf, VkDeviceSize *off); // the copy of a bound uniform range +void vkb_ucap_read(uint64_t addr, void *out, size_t n); // CPU read of uniform memory, from the copy where there is one +void vkb_ucap_drop(uint64_t addr, uint64_t len); +int vkb_ucap_why(uint64_t addr, uint64_t size); // a GPU write: copies it overlaps are no longer used +void vkb_ucap_stats(uint64_t *caps, uint64_t *bytes, uint64_t *ns, uint64_t *hits, uint64_t *drops); + +// ---- vkb_bc7.c: ASTC textures as BC7, from a cache by the contents of the ASTC blocks +typedef struct { uint64_t a, b; } vkb_bc7_key; +int vkb_bc7_on(void); // TOTK_BC7=0: ASTC decoded to RGBA8 (A/B) +// The key of one surface: its ASTC blocks in rows (n bytes), its size in texels (depth: slices of a 3D level), block size, sRGB. +vkb_bc7_key vkb_bc7_key_of(const uint8_t *blocks, size_t n, uint32_t w, uint32_t h, uint32_t depth, uint32_t bw, uint32_t bh, int srgb); +enum { VKB_BC_7 = 0, VKB_BC_1 = 1 }; // the format of a cache entry +int vkb_bc7_lookup(vkb_bc7_key k, int *fmt, size_t *len); // in the cache: its format and length +int vkb_bc7_has(vkb_bc7_key k, size_t n); // n bytes of BC7 blocks for it in the cache +int vkb_bc7_read(vkb_bc7_key k, uint8_t *out, size_t n); +void vkb_bc7_store(vkb_bc7_key k, const uint8_t *data, size_t n); +void vkb_bc_store(vkb_bc7_key k, const uint8_t *data, size_t n, int fmt); +void vkb_bc7_queue(vkb_bc7_key k, const uint8_t *rgba, uint32_t w, uint32_t h, uint32_t d, int srgb); // encoded in the background +void vkb_bc7_encode_now(vkb_bc7_key k, const uint8_t *rgba, uint32_t w, uint32_t h, uint32_t d, int srgb, int fmt, uint8_t *out); +void vkb_bc7_stats(uint64_t *hits, uint64_t *misses, uint64_t *encoded, uint64_t *dropped, uint64_t *encode_ns, size_t *queued); + +// ---- vkb_desc.c: descriptors in buffers (VK_EXT_descriptor_buffer) +int vkb_db_on(void); +void vkb_db_layouts(const VkDescriptorSetLayout layout[2]); +void vkb_db_slot_reset(int slot); +uint64_t vkb_db_gen(void); // changes when offsets made before are void (descriptor memory replaced) +VkDeviceSize vkb_db_set(VkCommandBuffer cb, int slot, uint64_t rec, const VkWriteDescriptorSet *w, int n, int compute); +void vkb_db_bind(VkCommandBuffer cb, int slot, uint64_t rec); // the memory offsets refer to, bound to this command buffer +void vkb_db_stats(uint64_t *get_calls, uint64_t *cache_hits); + +// ---- vkb_copy.c: NVN copies between game memory and textures (outside a pass; they end one) +void vkb_copy_stride(int64_t stride); // SetCopyRowStride +int64_t vkb_copy_stride_get(void); +int vkb_copy_t2b_range(const p_copy_b2t *c, uint64_t *addr, uint64_t *len); // the buffer range CopyTextureToBuffer writes +void vkb_copy_buf_tex(const p_copy_b2t *c, int to_tex); // CopyBufferToTexture (1) / CopyTextureToBuffer (0) +void vkb_copy_tex_tex(const p_copy_t2t *c); // CopyTextureToTexture +int vkb_tex_stale(vkb_image *im); // the CPU changed its game memory since the upload (checked once per submission) +VkFormat vkb_format(int nvn_format, VkImageAspectFlags *aspect, int *astc); +void vkb_tex_release(const tex_t *t); +int vkb_tex_snap_wanted(const char *vkey, const char *fkey, uint64_t frame); // TOTK_SNAP (development) +void vkb_tex_snap(VkCommandBuffer cb, vkb_image *im, const char *what); +void vkb_tex_snap_flush(void); +void vkb_tex_lumlog(const tex_t *t, uint64_t present); // TOTK_LUMLOG (development) +void vkb_tex_dump(const char *dir); // TOTK_DUMP: the GPU-written images as BMP (development) // the game finalized the texture: image freed after the GPU is done + +// ---- vkb_draw.c: render state, texture / sampler pools, draws +void vkb_slot_reset(int slot); // the slot's previous submission has completed: reuse its memory +void vkb_state_reset(void); +void vkb_state(uint32_t op, const uint8_t *payload); // state / binding commands +void vkb_draw(const uint8_t *payload, int indexed); // OP_DRAW / OP_DRAW_IDX +void vkb_dispatch(const uint8_t *payload); // OP_DISPATCH +void vkb_draw_stats(uint64_t *draws, uint64_t *skipped); +size_t vkb_state_size(void); // the render state as bytes (a chunk's starting state, vkb_mt.c) +void vkb_state_save(void *dst); +void vkb_state_load(const void *src); +void vkb_rec_begin(void); // this thread starts a command buffer: its recording caches start empty +void vkb_draw_mt_init(void); // objects made on first use, made before threads record (prologue) +void vkb_draw_prescan(const uint8_t *payload, int indexed, int dispatch); // the GPU writes a draw / dispatch makes (ucap) + +// ---- vkb_mt.c: submissions recorded by several threads (2026-10-24) +// The translation thread walks a submission once in order (vkb_mt_submit): render state, events and counters, +// conditional rendering, the GPU writes that void captured uniforms, and cuts it into chunks at render target changes, +// each with its starting state. Recording threads (the translation thread too, once the walk is done) record the chunks +// into command buffers of their own. Work that must come before every chunk (texture uploads, pool copies) goes into the +// submission's prologue (vkb_pro_begin / vkb_pro_end, under a lock). Each chunk tracks the layouts its images need first +// and leave last (vkb_trk); between chunks a small command buffer moves them (and waits for all earlier work). +int vkb_mt_on(void); // TOTK_VK_RECORD_THREADS (default: by the number of processors; 1: single thread) +int vkb_mt_recording(void); // this thread is recording a chunk or the prologue +uint64_t vkb_seq(void); // the position of the command being replayed within its submission +void vkb_seq_set(uint64_t seq); +void vkb_pro_begin(void); // records into the prologue from here (a no-op on one thread) +void vkb_pro_end(void); +void vkb_cmd_set(VkCommandBuffer cb); // this thread's command buffer (VK_NULL_HANDLE: the submission's own) +void vkb_mt_slot_reset(int slot); // the slot's previous submission has completed (backend_vk.c mtl_sub_begin) +void vkb_mt_submit(const uint64_t *h, int n); // records the submission's command handles (after mtl_sub_begin) +// Image layouts per chunk (vkb_tex.c): the first layout each image needs in it and the one it leaves it in. +typedef struct { vkb_image *im; VkImageLayout first, cur; } vkb_trk_e; +typedef struct { vkb_trk_e *e; int n, cap; int *tab; int tcap; } vkb_trk; +void vkb_trk_set(vkb_trk *t); // this thread records with this tracking (NULL: images' own layouts, one thread) +vkb_trk *vkb_trk_get(void); +void vkb_trk_reset(vkb_trk *t); +int vkb_layout_is(vkb_image *im, VkImageLayout l); // the image is in l as recorded so far (a first use: it will be) +VkImageLayout vkb_layout(vkb_image *im); // its layout as recorded so far diff --git a/src/gfx/vkb_audit.c b/src/gfx/vkb_audit.c new file mode 100644 index 0000000..06248b1 --- /dev/null +++ b/src/gfx/vkb_audit.c @@ -0,0 +1,131 @@ +// Fidelity audit of the Vulkan NVN replayer: counting of substitutions and skips (vkb_audit.h). +// Counting is always on (array increments on the replay thread); the per-shader tables and the report only within +// a TOTK_AUDIT window. +#include +#include +#include +#include + +#include "compat/gfx_compat.h" +#include "host/hle.h" +#include "vkb_audit.h" + +#define VLOG(...) LOG("[vk] " __VA_ARGS__) + +static const char *const g_name[A_COUNT] = { + "draw skipped: no program / translation", "draw skipped: primitive type", "draw skipped: targets not attachable", + "draw skipped: index buffer not imported", "draw skipped: resource outside layout", "render target left out", + "texture slot unbound", "separate texture handle", "texture: no image", "texture not in sampled layout", + "texture contents not uploadable", "texture view adjusted to the shader", "storage image missing", + "uniform buffer not imported (zeros)", "uniform buffer copied (misaligned)", "storage buffer missing", + "storage buffer source unknown", "vertex buffer not imported (zeros)", "integer attribute read as float", + "fragment input not written by vertex shader", "multisampling replayed single-sampled", + "draw while rendering disabled / conditional", "masked clear not replayed", "buffer copy skipped", + "uniform update skipped", "border colour approximated", "command not replayed", + "texture copy not replayed (form)", "written buffer left in host memory (slow)", + "mirror sent back early (straddling access)"}; + +typedef struct { char key[13]; uint64_t n; } keyc; +typedef struct { uint64_t events, draws, frames; keyc keys[16]; int nkeys, frame_hit; } stat_t; +static stat_t g_stat[A_COUNT]; +static uint32_t g_draw_mask[(A_COUNT + 31) / 32]; +static uint64_t g_draws, g_presents; +static int g_on; // inside a window +static uint64_t g_from, g_to; // the window [from, to) +static const char *g_spec; // the rest of TOTK_AUDIT (next windows) +static int g_init; + +static void next_window(void) { + g_from = g_to = 0; + if (!g_spec || !*g_spec) return; + char *end; + g_from = strtoull(g_spec, &end, 10); + uint64_t n = *end == ':' ? strtoull(end + 1, &end, 10) : 300; + g_to = g_from + n; + g_spec = strchr(end, ','); + if (g_spec) g_spec++; +} + +static void flush_draw(void) { + for (int r = 0; r < A_COUNT; r++) + if (g_draw_mask[r / 32] >> (r % 32) & 1) { + g_stat[r].draws++; + if (!g_stat[r].frame_hit) { g_stat[r].frame_hit = 1; g_stat[r].frames++; } + } + memset(g_draw_mask, 0, sizeof g_draw_mask); +} + +void vkb_audit_draw_begin(void) { + if (!g_on) { // off (and nothing left of the last window): every recording thread passes here, so no writes + int any = 0; + for (unsigned i = 0; i < sizeof g_draw_mask / sizeof *g_draw_mask; i++) any |= g_draw_mask[i] != 0; + if (!any) return; + } + flush_draw(); + if (g_on) g_draws++; +} + +void vkb_audit(int reason, const char *key) { + if (!g_on || reason < 0 || reason >= A_COUNT) return; + stat_t *s = &g_stat[reason]; + s->events++; + g_draw_mask[reason / 32] |= 1u << (reason % 32); + if (!key) key = "-"; + for (int i = 0; i < s->nkeys; i++) + if (!strncmp(s->keys[i].key, key, 12)) { s->keys[i].n++; return; } + if (s->nkeys < 16) { snprintf(s->keys[s->nkeys].key, 13, "%.12s", key); s->keys[s->nkeys++].n = 1; } +} + +static void report(void) { + char path[256]; + _mkdir("diag"); // development output in the game folder (it describes the game's frames) + snprintf(path, sizeof path, "diag/audit-%llu.md", (unsigned long long)g_from); + FILE *f = fopen(path, "w"); + uint64_t frames = g_presents ? g_presents : 1; + char line[512]; + snprintf(line, sizeof line, "Audit of presents %llu..%llu: %llu draws/dispatches (%.0f per frame)", + (unsigned long long)g_from, (unsigned long long)g_to - 1, (unsigned long long)g_draws, (double)g_draws / frames); + VLOG("%s", line); + if (f) fprintf(f, "# %s\n\n| reason | draws per frame | events per frame | frames | top shaders |\n|---|---|---|---|---|\n", line); + int order[A_COUNT]; + for (int i = 0; i < A_COUNT; i++) order[i] = i; + for (int i = 0; i < A_COUNT; i++) // by affected draws + for (int j = i + 1; j < A_COUNT; j++) + if (g_stat[order[j]].draws > g_stat[order[i]].draws) { int t = order[i]; order[i] = order[j]; order[j] = t; } + for (int k = 0; k < A_COUNT; k++) { + stat_t *s = &g_stat[order[k]]; + if (!s->events) continue; + for (int i = 0; i < s->nkeys; i++) // shaders by count + for (int j = i + 1; j < s->nkeys; j++) + if (s->keys[j].n > s->keys[i].n) { keyc t = s->keys[i]; s->keys[i] = s->keys[j]; s->keys[j] = t; } + char keys[256] = ""; + int n = 0; + for (int i = 0; i < s->nkeys && i < 6 && n < 200; i++) n += snprintf(keys + n, sizeof keys - n, "%s%s %llu", i ? ", " : "", s->keys[i].key, (unsigned long long)s->keys[i].n); + VLOG(" %-48s %8.1f draws/frame %8.1f events/frame %5llu frames %s", g_name[order[k]], (double)s->draws / frames, + (double)s->events / frames, (unsigned long long)s->frames, keys); + if (f) fprintf(f, "| %s | %.1f | %.1f | %llu | %s |\n", g_name[order[k]], (double)s->draws / frames, (double)s->events / frames, + (unsigned long long)s->frames, keys); + } + if (f) fclose(f); +} + +void vkb_audit_present(uint64_t present) { + if (!g_init) { + g_init = 1; + g_spec = getenv("TOTK_AUDIT"); + next_window(); + } + flush_draw(); + for (int r = 0; r < A_COUNT; r++) g_stat[r].frame_hit = 0; + if (g_on) g_presents++; + if (g_on && present + 1 >= g_to) { // the window ends with this present + report(); + g_on = 0; + next_window(); + } + if (!g_on && g_to && present + 1 >= g_from && present + 1 < g_to) { + g_on = 1; + memset(g_stat, 0, sizeof g_stat); + g_draws = g_presents = 0; + } +} diff --git a/src/gfx/vkb_audit.h b/src/gfx/vkb_audit.h new file mode 100644 index 0000000..681e57b --- /dev/null +++ b/src/gfx/vkb_audit.h @@ -0,0 +1,44 @@ +// Fidelity audit of the Vulkan NVN replayer (vkb_audit.c): every place where the replay substitutes something +// (zeros, an empty texture, a default shader variant) or skips work is counted with its reason, per frame and per +// shader. TOTK_AUDIT=:[,...]: a report per window (markdown table) in the log and in +// diag/audit-.md. The table decides which replay mechanisms to complete first. +#pragma once +#include + +enum { + A_DRAW_NO_PROGRAM, // draw / dispatch skipped: no program or no translation of a shader + A_DRAW_TOPOLOGY, // draw skipped: primitive type without a Vulkan topology (line loop, quads) + A_DRAW_NO_TARGETS, // draw skipped: its render targets cannot be attached + A_DRAW_NO_INDEX, // draw skipped: index buffer outside the imported pools + A_SHADER_UNSUPPORTED, // draw skipped: a shader resource outside the universal layout + A_RT_NOT_ATTACHABLE, // a render target left out of the pass (compressed / unsupported format) + A_TEX_UNBOUND, // a sampled texture slot with no texture handle: empty texture + A_TEX_SEPARATE, // separate texture / sampler handles (not resolved yet): empty texture + A_TEX_NO_IMAGE, // the handle names no image (format or target not created): empty texture + A_TEX_NOT_READY, // the image is not in a sampled layout (e.g. bound as a target of this pass): empty texture + A_TEX_NOT_UPLOADED, // the contents come from game memory but cannot be uploaded (depth / 3-component formats) + A_TEX_VIEW_ADJUSTED, // the shader's texture type does not fit the image (cube of < 6 layers, 3D of a 2D image) + A_IMG_MISSING, // a storage image slot with no usable image: empty image + A_UBO_MISSING, // uniform buffer outside the imported pools: zeros + A_UBO_COPIED, // uniform buffer misaligned for this GPU: a copy of the current contents + A_SSBO_MISSING, // storage buffer outside the imported pools / misaligned / size 0: empty buffer + A_SSBO_UNKNOWN_SOURCE, // storage buffer whose address source the translation could not tell + A_VBUF_MISSING, // vertex buffer outside the imported pools (virtual pools): zeros + A_ATTR_INT_AS_FLOAT, // integer vertex attribute given as float (no variant could be made) + A_FS_INPUT_UNWRITTEN, // the fragment shader reads an input the vertex shader does not write (undefined) + A_MULTISAMPLE, // multisampled state or target replayed single-sampled + A_RENDER_DISABLED, // draw recorded while rendering was disabled / conditional (replayed anyway) + A_CLEAR_MASKED, // clear with a channel mask (not replayed) + A_COPY_SKIPPED, // buffer copy outside the imported pools (skipped) + A_UPDATE_SKIPPED, // inline uniform update not 4-byte aligned (skipped) + A_BORDER_APPROX, // sampler border colour approximated by a standard one + A_OP_NOT_REPLAYED, // a recorded command the replay does not handle + A_TEX_COPY_UNSUPPORTED, + A_SSBO_HOST_WRITE, + A_MIRROR_EVICTED, // a mirror sent back early: an access straddled its bounds // a written storage buffer left in host memory (no device-local mirror): slow over PCIe // a texture copy in a form not replayed (ASTC / depth / 3-component formats, scaled depth) + A_COUNT +}; + +void vkb_audit(int reason, const char *key); // key: the shader key (or NULL) +void vkb_audit_draw_begin(void); // a draw / dispatch starts: reasons from here count it once each +void vkb_audit_present(uint64_t present); diff --git a/src/gfx/vkb_bc7.c b/src/gfx/vkb_bc7.c new file mode 100644 index 0000000..339596e --- /dev/null +++ b/src/gfx/vkb_bc7.c @@ -0,0 +1,218 @@ +// ASTC textures as BC7 (decided 2026-10-24): desktop GPUs and the Steam Deck have no ASTC, so +// ASTC was decoded on the CPU to RGBA8 at every upload (the hitches when new parts of the world stream in, 4x the video +// memory). BC7 is what these GPUs sample natively, at 8 bits per texel - not fewer bits than TOTK's ASTC blocks. +// A cache of BC7 data by the contents of the ASTC blocks: the key of a surface (one level of one layer; every slice of a +// 3D level) is a 128-bit hash of its ASTC blocks in rows (as deswizzled for an upload) with its size and block size +// (tex/bc7key.c) - the same bytes whichever way the game lays the texture out in memory, so the build can fill the cache +// from the RomFS (next step) and the game finds the entries. Until then the cache fills as textures are met: a miss is +// decoded as before and encoded to BC7 in the background for the next time. +// Files (TOTK_BC7_DIR, default data/textures): bc7.pack (the blocks, appended) and bc7.idx (32-byte records: key, +// offset, length, format - 0 BC7, 1 BC1; appended after the blocks are written, so an entry always has its data). +// BC1: the interface's 12x12 ASTC maps when opaque (tools/bc7build: a texture is BC1 only if none of its texels is +// transparent; half the bytes of BC7, and 12x12 ASTC holds less detail than BC1 keeps). Nothing from the game is +// in the repository or a build: the cache is made on the player's machine from the player's copy (README.md, Legal). +#include +#include +#include +#include +#include +#include + +#include "compat/gfx_compat.h" +#include "host/hle.h" +#include "vkb.h" +#include "tex/bc7.h" +#include "tex/bc7key.h" + +#define VLOG(...) LOG("[vk] " __VA_ARGS__) + +int vkb_bc7_on(void) { // TOTK_BC7=0: decode ASTC to RGBA8 as before (A/B) + static int on = -1; + if (on < 0) { const char *e = getenv("TOTK_BC7"); on = !(e && *e == '0'); } + return on; +} + +// ---- the key (tex/bc7key.c: the build computes the same) +vkb_bc7_key vkb_bc7_key_of(const uint8_t *blocks, size_t n, uint32_t w, uint32_t h, uint32_t depth, uint32_t bw, uint32_t bh, int srgb) { + (void)srgb; // not part of the key: the BC7 bytes serve both views + bc7_key k = bc7_key_of(blocks, n, w, h, depth, bw, bh); + return (vkb_bc7_key){k.a, k.b}; +} + +// ---- the index in memory (open addressing) and the files +typedef struct { uint64_t a, b, off; uint32_t len, fmt; } rec_t; // also the record in bc7.idx (fmt: VKB_BC_*) +static rec_t *g_tab; +static size_t g_cap, g_n; +static FILE *g_pack, *g_idx; +static pthread_mutex_t g_mu = PTHREAD_MUTEX_INITIALIZER; +static int g_open; // 1 open, -1 cannot be used +static uint64_t g_hits, g_misses, g_encoded, g_dropped, g_encode_ns; + +static rec_t *slot(uint64_t a, uint64_t b) { // g_mu held, g_cap a power of two + size_t h = (size_t)(a ^ (b * 0x9E3779B97F4A7C15ull)) & (g_cap - 1); + while ((g_tab[h].a || g_tab[h].b) && !(g_tab[h].a == a && g_tab[h].b == b)) h = (h + 1) & (g_cap - 1); + return &g_tab[h]; +} +static void put(const rec_t *r) { // g_mu held + if ((g_n + 1) * 2 > g_cap) { + rec_t *old = g_tab; + size_t n = g_cap; + g_cap = g_cap ? g_cap * 2 : 1 << 16; + if (!(g_tab = calloc(g_cap, sizeof *g_tab))) die("BC7 index"); + for (size_t i = 0; i < n; i++) if (old[i].a || old[i].b) *slot(old[i].a, old[i].b) = old[i]; + free(old); + } + rec_t *s = slot(r->a, r->b); + if (!s->a && !s->b) g_n++; + *s = *r; +} +static const char *dir(void) { + const char *d = getenv("TOTK_BC7_DIR"); + return d && *d ? d : "data/textures"; +} +static int open_files(void) { // g_mu held + if (g_open) return g_open > 0; + g_open = -1; + char path[512]; + const char *d = dir(); + for (char *p = strcpy(path, d) + 1; *p; p++) if (*p == '/' || *p == '\\') { char c = *p; *p = 0; _mkdir(path); *p = c; } + _mkdir(path); + snprintf(path, sizeof path, "%s/bc7.pack", d); + if (!(g_pack = fopen(path, "r+b")) && !(g_pack = fopen(path, "w+b"))) { VLOG("BC7 cache: cannot open %s", path); return 0; } + _fseeki64(g_pack, 0, SEEK_END); + int64_t pack_size = _ftelli64(g_pack); + snprintf(path, sizeof path, "%s/bc7.idx", d); + if (!(g_idx = fopen(path, "r+b")) && !(g_idx = fopen(path, "w+b"))) { VLOG("BC7 cache: cannot open %s", path); return 0; } + rec_t r; + int64_t good = 0; + while (fread(&r, sizeof r, 1, g_idx) == 1) { + if ((int64_t)(r.off + r.len) > pack_size) break; // the blocks of an interrupted write: dropped + put(&r); + good += sizeof r; + } + _fseeki64(g_idx, good, SEEK_SET); + VLOG("BC7 cache %s: %zu surfaces, %.1f MiB", d, g_n, pack_size / 1048576.0); + g_open = 1; + return 1; +} + +int vkb_bc7_lookup(vkb_bc7_key k, int *fmt, size_t *len) { + pthread_mutex_lock(&g_mu); + int ok = open_files() && g_cap; + if (ok) { rec_t *s = slot(k.a, k.b); ok = (s->a || s->b) && s->len; if (ok) { *fmt = (int)s->fmt; *len = s->len; } } + pthread_mutex_unlock(&g_mu); + return ok; +} +int vkb_bc7_has(vkb_bc7_key k, size_t n) { + int fmt; size_t len; + return vkb_bc7_lookup(k, &fmt, &len) && fmt == VKB_BC_7 && len == n; +} +int vkb_bc7_read(vkb_bc7_key k, uint8_t *out, size_t n) { + pthread_mutex_lock(&g_mu); + int ok = open_files() && g_cap; + rec_t r = {0}; + if (ok) { r = *slot(k.a, k.b); ok = (r.a || r.b) && r.len == n; } + if (ok) ok = !_fseeki64(g_pack, (int64_t)r.off, SEEK_SET) && fread(out, 1, n, g_pack) == n; + if (ok) g_hits++; else g_misses++; + pthread_mutex_unlock(&g_mu); + return ok; +} +void vkb_bc7_store(vkb_bc7_key k, const uint8_t *data, size_t n) { vkb_bc_store(k, data, n, VKB_BC_7); } +void vkb_bc_store(vkb_bc7_key k, const uint8_t *data, size_t n, int fmt) { + pthread_mutex_lock(&g_mu); + if (open_files() && !(g_cap && (slot(k.a, k.b)->a || slot(k.a, k.b)->b))) { + _fseeki64(g_pack, 0, SEEK_END); + rec_t r = {k.a, k.b, (uint64_t)_ftelli64(g_pack), (uint32_t)n, (uint32_t)fmt}; + if (fwrite(data, 1, n, g_pack) == n && !fflush(g_pack)) { + _fseeki64(g_idx, 0, SEEK_END); + if (fwrite(&r, sizeof r, 1, g_idx) == 1) { fflush(g_idx); put(&r); } + } + } + pthread_mutex_unlock(&g_mu); +} + +// ---- encoding in the background (a miss: next time it is there) +typedef struct job { vkb_bc7_key k; uint8_t *rgba; uint32_t w, h, d; int srgb; struct job *next; } job; +static job *g_jobs, **g_tail = &g_jobs; +static size_t g_queued_bytes; +static pthread_mutex_t g_jmu = PTHREAD_MUTEX_INITIALIZER; +static pthread_cond_t g_jcv = PTHREAD_COND_INITIALIZER; +enum { QUEUE_MAX = 512u << 20, ENCODERS = 2 }; +static int quality(void) { // TOTK_BC7_QUALITY=0..4 (bc7enc's uber level; background encoding) + static int q = -1; + // 0 by default: met in play this is a stopgap (one large surface took ~0.75 s at level 1); the build fills the cache + if (q < 0) { const char *e = getenv("TOTK_BC7_QUALITY"); q = e && *e ? atoi(e) : 0; } + return q; +} +static void *encoder(void *unused) { + (void)unused; + pthread_setname_np("totk-bc7"); + SetThreadPriority(GetCurrentThread(), THREAD_PRIORITY_LOWEST); // the game's and the recording threads come first + for (;;) { + pthread_mutex_lock(&g_jmu); + while (!g_jobs) pthread_cond_wait(&g_jcv, &g_jmu); + job *j = g_jobs; + g_jobs = j->next; + if (!g_jobs) g_tail = &g_jobs; + size_t in = (size_t)j->w * j->h * j->d * 4; + pthread_mutex_unlock(&g_jmu); + size_t n = bc7_size(j->w, j->h, j->d); + if (!vkb_bc7_has(j->k, n)) { + uint8_t *out = malloc(n); + if (out) { + uint64_t t0 = runtime_now_ns(); + bc7_encode(j->rgba, j->w, j->h, j->d, j->srgb, quality(), out); + vkb_bc7_store(j->k, out, n); + __atomic_fetch_add(&g_encode_ns, runtime_now_ns() - t0, __ATOMIC_RELAXED); + __atomic_fetch_add(&g_encoded, 1, __ATOMIC_RELAXED); + free(out); + } + } + free(j->rgba); + free(j); + pthread_mutex_lock(&g_jmu); + g_queued_bytes -= in; + pthread_mutex_unlock(&g_jmu); + } + return NULL; +} +void vkb_bc7_queue(vkb_bc7_key k, const uint8_t *rgba, uint32_t w, uint32_t h, uint32_t d, int srgb) { + size_t in = (size_t)w * h * (d ? d : 1) * 4; + pthread_mutex_lock(&g_jmu); + if (g_queued_bytes + in > QUEUE_MAX) { g_dropped++; pthread_mutex_unlock(&g_jmu); return; } // met again later + g_queued_bytes += in; + pthread_mutex_unlock(&g_jmu); + job *j = malloc(sizeof *j); + uint8_t *copy = malloc(in); + if (!j || !copy) { free(j); free(copy); pthread_mutex_lock(&g_jmu); g_queued_bytes -= in; pthread_mutex_unlock(&g_jmu); return; } + memcpy(copy, rgba, in); + *j = (job){k, copy, w, h, d ? d : 1, srgb, NULL}; + static int started; + pthread_mutex_lock(&g_jmu); + if (!started) { + started = 1; + for (int i = 0; i < ENCODERS; i++) { + pthread_t t; + if (pthread_create(&t, NULL, encoder, NULL) == 0) pthread_detach(t); + } + } + *g_tail = j; + g_tail = &j->next; + pthread_cond_signal(&g_jcv); + pthread_mutex_unlock(&g_jmu); +} +// A miss on an image that is BC7 / BC1 already (its contents changed after it became one): encoded now, quickly. +void vkb_bc7_encode_now(vkb_bc7_key k, const uint8_t *rgba, uint32_t w, uint32_t h, uint32_t d, int srgb, int fmt, uint8_t *out) { + if (fmt == VKB_BC_1) { bc1_encode(rgba, w, h, d ? d : 1, 4, out); return; } // (not kept: the cache keys BC1 by the build's choice) + bc7_encode(rgba, w, h, d ? d : 1, srgb, 0, out); + vkb_bc7_store(k, out, bc7_size(w, h, d ? d : 1)); +} +void vkb_bc7_stats(uint64_t *hits, uint64_t *misses, uint64_t *encoded, uint64_t *dropped, uint64_t *encode_ns, size_t *queued) { + pthread_mutex_lock(&g_mu); + *hits = g_hits; *misses = g_misses; + pthread_mutex_unlock(&g_mu); + *encoded = __atomic_load_n(&g_encoded, __ATOMIC_RELAXED); *encode_ns = __atomic_load_n(&g_encode_ns, __ATOMIC_RELAXED); + pthread_mutex_lock(&g_jmu); + *dropped = g_dropped; *queued = g_queued_bytes; + pthread_mutex_unlock(&g_jmu); +} diff --git a/src/gfx/vkb_copy.c b/src/gfx/vkb_copy.c new file mode 100644 index 0000000..cda7de2 --- /dev/null +++ b/src/gfx/vkb_copy.c @@ -0,0 +1,170 @@ +// GPU copies between game memory and textures: NVN CopyBufferToTexture, CopyTextureToBuffer, CopyTextureToTexture +// (with SetCopyRowStride), after MacNativeGames' nvn_mtl.m copy_b2t / copy_t2b / copy_t2t (MIT). These were not replayed +// at all: TOTK uploads every frame of a pre-rendered cutscene from its NV12 buffer with CopyBufferToTexture, and the +// movie texture stayed empty (a green picture: zero luma and chroma). +// A texture a copy writes holds GPU contents from then on (gpu_written): it is not uploaded from game memory again, +// whose bytes a GPU copy does not change here (as MacNativeGames' GPU-owned textures). +#include + +#include "compat/gfx_compat.h" +#include "host/hle.h" +#include "nvn/nvn_mtl.h" +#include "nvn/nvn_obj.h" +#include "nvn/nvn_tex.h" +#include "vkb.h" +#include "vkb_audit.h" + +#define VLOG(...) LOG("[vk] " __VA_ARGS__) + +static __thread int64_t g_stride; // SetCopyRowStride: bytes between rows in the buffer, 0 = tightly packed (per recording thread) +void vkb_copy_stride(int64_t stride) { g_stride = stride; } +int64_t vkb_copy_stride_get(void) { return g_stride; } + +static void barrier(VkCommandBuffer cb, VkPipelineStageFlags2 src, VkAccessFlags2 sa, VkPipelineStageFlags2 dst, VkAccessFlags2 da) { + VkMemoryBarrier2 mb = {VK_STRUCTURE_TYPE_MEMORY_BARRIER_2, .srcStageMask = src, .srcAccessMask = sa, .dstStageMask = dst, .dstAccessMask = da}; + VkDependencyInfo dep = {VK_STRUCTURE_TYPE_DEPENDENCY_INFO, .memoryBarrierCount = 1, .pMemoryBarriers = &mb}; + vk.vkCmdPipelineBarrier2(cb, &dep); +} +static void unsupported(const char *what, const vkb_image *im) { + vkb_audit(A_TEX_COPY_UNSUPPORTED, NULL); + static int said; + if (said++ < 8) VLOG("[temporary] %s: texture %llu format %#x not copied", what, im ? (unsigned long long)im->serial : 0ull, im ? im->d.format : 0); +} +// Colour formats stored here as NVN lays them out (what the upload copies byte for byte): not ASTC (decoded on the +// CPU), not depth (D16 / D24 are kept as D32F), not the 3-component formats (expanded to 4). +static int raw_layout(const vkb_image *im, int *bw, int *bh, int *bpb) { + *bpb = nvn_fmt_bpb(im->d.format, bw, bh); + return *bpb && !im->astc && im->aspect == VK_IMAGE_ASPECT_COLOR_BIT && !(im->d.format >= 0x19 && im->d.format <= 0x24); +} +static uint32_t level_extent(uint32_t v, unsigned level) { return v >> level ? v >> level : 1; } + +// The buffer range a copy between a buffer and a texture covers, with its texture and region checked (0: not copied). +static vkb_image *copy_range(const p_copy_b2t *c, const char *what, uint64_t *len_out, int *bw, int *bh, int *bpb, int quiet) { + vkb_image *im = c->tex ? vkb_image_of(c->tex) : NULL; + if (!im || !raw_layout(im, bw, bh, bpb)) { if (!quiet) unsupported(what, im); return NULL; } + unsigned level = c->has_view && c->view.set_levels ? c->view.base_level : 0, layer0 = c->has_view && c->view.set_layers ? c->view.min_layer : 0; + if (level >= im->levels) { if (!quiet) unsupported(what, im); return NULL; } + int is3d = im->depth > 1, nz = c->r.depth > 0 ? c->r.depth : 1; + uint32_t lw = level_extent(im->width, level), lh = level_extent(im->height, level), ld = is3d ? level_extent(im->depth, level) : 1; + if (c->r.width <= 0 || c->r.height <= 0 || c->r.xoffset < 0 || c->r.yoffset < 0 || c->r.zoffset < 0) return NULL; + if ((uint32_t)(c->r.xoffset + c->r.width) > lw || (uint32_t)(c->r.yoffset + c->r.height) > lh || + (is3d ? (uint32_t)(c->r.zoffset + nz) > ld : layer0 + c->r.zoffset + nz > im->layers)) { if (!quiet) unsupported(what, im); return NULL; } + uint64_t wb = (uint64_t)((c->r.width + *bw - 1) / *bw) * *bpb, row = g_stride > 0 ? (uint64_t)g_stride : wb; + uint64_t rows = (uint64_t)(c->r.height + *bh - 1) / *bh, img = row * rows; + *len_out = img * (nz - 1) + row * (rows - 1) + wb; + if (row < wb || row % *bpb) { if (!quiet) vkb_audit(A_COPY_SKIPPED, NULL); return NULL; } + return im; +} +int vkb_copy_t2b_range(const p_copy_b2t *c, uint64_t *addr, uint64_t *len) { // vkb_mt.c: the walk's GPU writes + int bw, bh, bpb; + if (!copy_range(c, "", len, &bw, &bh, &bpb, 1)) return 0; + *addr = c->src; + return 1; +} + +// to_tex: CopyBufferToTexture (c->src is the buffer); else CopyTextureToBuffer (c->src is the destination buffer). +void vkb_copy_buf_tex(const p_copy_b2t *c, int to_tex) { + const char *what = to_tex ? "buffer -> texture copy" : "texture -> buffer copy"; + int bw, bh, bpb; + uint64_t len; + vkb_image *im = copy_range(c, what, &len, &bw, &bh, &bpb, 0); + if (!im) return; + unsigned level = c->has_view && c->view.set_levels ? c->view.base_level : 0, layer0 = c->has_view && c->view.set_layers ? c->view.min_layer : 0; + int is3d = im->depth > 1, nz = c->r.depth > 0 ? c->r.depth : 1; + uint64_t wb = (uint64_t)((c->r.width + bw - 1) / bw) * bpb, row = g_stride > 0 ? (uint64_t)g_stride : wb; + uint64_t rows = (uint64_t)(c->r.height + bh - 1) / bh; + vkb_pass_end_r(PE_TEXCOPY); + vkb_mirrors_evict(c->src, len); // a range that straddles a mirror: game memory first + vkb_pool_use(c->src, len, !to_tex); + if (!to_tex) vkb_ucap_drop(c->src, len); + VkBuffer buf; + VkDeviceSize off; + if (!vkb_gpu_addr(c->src, len, &buf, &off)) { vkb_audit(A_COPY_SKIPPED, NULL); return; } + if (off % (bpb < 4 ? 4 : bpb)) { unsupported(what, im); return; } // [temporary] Vulkan's buffer offset alignment + VkCommandBuffer cb = vkb_cmd(); + vkb_tex_upload(cb, im); // the texture's current contents (the source, or the rest of the destination) + VkBufferImageCopy r = {.bufferOffset = off, .bufferRowLength = (uint32_t)(row / bpb * bw), .bufferImageHeight = (uint32_t)(rows * bh), + .imageSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, level, is3d ? 0 : layer0 + (uint32_t)c->r.zoffset, is3d ? 1 : (uint32_t)nz}, + .imageOffset = {c->r.xoffset, c->r.yoffset, is3d ? c->r.zoffset : 0}, + .imageExtent = {(uint32_t)c->r.width, (uint32_t)c->r.height, is3d ? (uint32_t)nz : 1}}; + if (to_tex) { // the buffer may have been written by earlier GPU work + barrier(cb, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_WRITE_BIT, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_READ_BIT); + vkb_image_use(cb, im, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT); + vkb_checkpoint("copy buffer -> texture", NULL, NULL); + vk.vkCmdCopyBufferToImage(cb, buf, im->img, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, 1, &r); + vkb_image_written(im); + } else { + barrier(cb, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT, + VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT); + vkb_image_use(cb, im, VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_READ_BIT); + vkb_checkpoint("copy texture -> buffer", NULL, NULL); + vk.vkCmdCopyImageToBuffer(cb, im->img, VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, buf, 1, &r); + // later commands (and the CPU after the game's wait) read the buffer + barrier(cb, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT | VK_PIPELINE_STAGE_2_HOST_BIT, + VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT | VK_ACCESS_2_HOST_READ_BIT); + } +} + +static VkFormatFeatureFlags features(VkFormat f) { + VkFormatProperties p; + vk.vkGetPhysicalDeviceFormatProperties(vkc.gpu, f, &p); + return p.optimalTilingFeatures; +} + +// CopyTextureToTexture: the same size as a plain copy (formats of one texel size: a reinterpretation, as on the +// Switch); scaling, mirroring or a different texel size as a blit (filtered with LINEAR_FILTER). +void vkb_copy_tex_tex(const p_copy_t2t *c) { + vkb_image *s = c->src ? vkb_image_of(c->src) : NULL, *d = c->dst ? vkb_image_of(c->dst) : NULL; + if (!s || !d) { unsupported("texture -> texture copy", s ? s : d); return; } + unsigned sl = c->has_sv && c->sv.set_levels ? c->sv.base_level : 0, dl = c->has_dv && c->dv.set_levels ? c->dv.base_level : 0; + unsigned sz0 = c->has_sv && c->sv.set_layers ? c->sv.min_layer : 0, dz0 = c->has_dv && c->dv.set_layers ? c->dv.min_layer : 0; + int s3 = s->depth > 1, d3 = d->depth > 1; + int snz = c->sr.depth > 0 ? c->sr.depth : 1, dnz = c->dr.depth > 0 ? c->dr.depth : 1; + if (sl >= s->levels || dl >= d->levels || s->aspect != d->aspect || s3 != d3 || s->astc != d->astc || + c->sr.width <= 0 || c->sr.height <= 0 || c->dr.width <= 0 || c->dr.height <= 0 || + (!s3 && (sz0 + c->sr.zoffset + snz > s->layers || dz0 + c->dr.zoffset + dnz > d->layers))) { + unsupported("texture -> texture copy", s); return; + } + int sbw, sbh, dbw, dbh, sbpb = nvn_fmt_bpb(s->d.format, &sbw, &sbh), dbpb = nvn_fmt_bpb(d->d.format, &dbw, &dbh); + int mirror = c->flags & (NVN_COPY_FLAGS_MIRROR_X_BIT | NVN_COPY_FLAGS_MIRROR_Y_BIT | NVN_COPY_FLAGS_MIRROR_Z_BIT); + int same = c->sr.width == c->dr.width && c->sr.height == c->dr.height && snz == dnz && !mirror; + int plain = same && (s->format == d->format || (sbpb && sbpb == dbpb && sbw == dbw && sbh == dbh && !s->astc)); + VkFilter filter = c->flags & NVN_COPY_FLAGS_LINEAR_FILTER_BIT ? VK_FILTER_LINEAR : VK_FILTER_NEAREST; + if (!plain) { // a blit: both formats must allow it (and filtering, when asked for) + VkFormatFeatureFlags sf = features(s->format), df = features(d->format); + if (!(sf & VK_FORMAT_FEATURE_BLIT_SRC_BIT) || !(df & VK_FORMAT_FEATURE_BLIT_DST_BIT) || (mirror & NVN_COPY_FLAGS_MIRROR_Z_BIT) || + (s->aspect != VK_IMAGE_ASPECT_COLOR_BIT && (s->format != d->format || !same))) { + unsupported("texture -> texture copy (scaled / converted)", s); return; + } + if (!(sf & VK_FORMAT_FEATURE_SAMPLED_IMAGE_FILTER_LINEAR_BIT)) filter = VK_FILTER_NEAREST; + } + vkb_pass_end_r(PE_TEXCOPY); + VkCommandBuffer cb = vkb_cmd(); + vkb_tex_upload(cb, s); + vkb_tex_upload(cb, d); // the rest of the destination + VkImageLayout sly = VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, dly = VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL; + if (s == d) { // within one image (another level or layer): one layout for both ends + sly = dly = VK_IMAGE_LAYOUT_GENERAL; + vkb_image_use(cb, s, sly, VK_PIPELINE_STAGE_2_ALL_TRANSFER_BIT, VK_ACCESS_2_TRANSFER_READ_BIT | VK_ACCESS_2_TRANSFER_WRITE_BIT); + } else { + vkb_image_use(cb, s, sly, VK_PIPELINE_STAGE_2_ALL_TRANSFER_BIT, VK_ACCESS_2_TRANSFER_READ_BIT); + vkb_image_use(cb, d, dly, VK_PIPELINE_STAGE_2_ALL_TRANSFER_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT); + } + VkImageSubresourceLayers ss = {s->aspect, sl, s3 ? 0 : sz0 + (uint32_t)c->sr.zoffset, s3 ? 1 : (uint32_t)snz}; + VkImageSubresourceLayers ds = {d->aspect, dl, d3 ? 0 : dz0 + (uint32_t)c->dr.zoffset, d3 ? 1 : (uint32_t)dnz}; + vkb_checkpoint(plain ? "copy texture -> texture" : "blit texture -> texture", NULL, NULL); + if (plain) { + VkImageCopy r = {ss, {c->sr.xoffset, c->sr.yoffset, s3 ? c->sr.zoffset : 0}, ds, {c->dr.xoffset, c->dr.yoffset, d3 ? c->dr.zoffset : 0}, + {(uint32_t)c->sr.width, (uint32_t)c->sr.height, s3 ? (uint32_t)snz : 1}}; + vk.vkCmdCopyImage(cb, s->img, sly, d->img, dly, 1, &r); + } else { + VkImageBlit r = {ss, {{c->sr.xoffset, c->sr.yoffset, s3 ? c->sr.zoffset : 0}, + {c->sr.xoffset + c->sr.width, c->sr.yoffset + c->sr.height, s3 ? c->sr.zoffset + snz : 1}}, + ds, {{c->dr.xoffset, c->dr.yoffset, d3 ? c->dr.zoffset : 0}, + {c->dr.xoffset + c->dr.width, c->dr.yoffset + c->dr.height, d3 ? c->dr.zoffset + dnz : 1}}}; + if (c->flags & NVN_COPY_FLAGS_MIRROR_X_BIT) { int32_t t = r.dstOffsets[0].x; r.dstOffsets[0].x = r.dstOffsets[1].x; r.dstOffsets[1].x = t; } + if (c->flags & NVN_COPY_FLAGS_MIRROR_Y_BIT) { int32_t t = r.dstOffsets[0].y; r.dstOffsets[0].y = r.dstOffsets[1].y; r.dstOffsets[1].y = t; } + vk.vkCmdBlitImage(cb, s->img, sly, d->img, dly, 1, &r, filter); + } + vkb_image_written(d); +} diff --git a/src/gfx/vkb_desc.c b/src/gfx/vkb_desc.c new file mode 100644 index 0000000..435a865 --- /dev/null +++ b/src/gfx/vkb_desc.c @@ -0,0 +1,201 @@ +// Descriptors in buffers (VK_EXT_descriptor_buffer): the target design (bindless, no descriptor set updates +// per draw). A draw's descriptors are bytes written into the submission's descriptor memory at the layout's binding +// offsets; the shaders find them through one offset set per draw (vkCmdSetDescriptorBufferOffsetsEXT). The bytes of each +// resource (a buffer range, a view with its sampler) come from the driver once per submission (vkGetDescriptorEXT) and +// are copied from then on. Before, each draw allocated a descriptor set and wrote it through vkUpdateDescriptorSets: +// with ~6000 draws a frame on the Great Sky Island, ~45 % of the replay thread was in the driver (2026-10-24). +// Within a submission no object a descriptor names is destroyed or recreated (destruction waits for the GPU, pools are +// deleted between submissions), so the per-submission caches cannot see a reused handle. +// TOTK_VK_DESCRIPTOR_SETS=1 keeps descriptor sets (A/B; also where the extension is missing). +#include +#include + +#include "compat/gfx_compat.h" +#include "host/hle.h" +#include "vkb.h" +#include "vkb_shader.h" +#include "vkb_lock.h" + +#define VLOG(...) LOG("[vk] " __VA_ARGS__) + +int vkb_db_on(void) { + static int v = -1; + if (v < 0) { + const char *e = getenv("TOTK_VK_DESCRIPTOR_SETS"); + v = vkc.has_descriptor_buffer && !(e && *e && *e != '0'); + VLOG("descriptors: %s", v ? "descriptor buffers" : "descriptor sets"); + } + return v; +} + +// ---- the layouts: size and the offset of every binding +static VkDeviceSize g_size[2], g_off[2][128]; +void vkb_db_layouts(const VkDescriptorSetLayout layout[2]) { + for (int c = 0; c < 2; c++) { + vk.vkGetDescriptorSetLayoutSizeEXT(vkc.device, layout[c], &g_size[c]); + for (uint32_t b = 0; b < 128; b++) { + int used = b % VKB_STAGE_BINDINGS < VKB_B_IMG + VKB_NIMG && (!c || b < VKB_STAGE_BINDINGS); // compute: one stage + if (used) vk.vkGetDescriptorSetLayoutBindingOffsetEXT(vkc.device, layout[c], b, &g_off[c][b]); + } + } + VLOG("descriptor buffer: set sizes %llu (graphics) / %llu (compute) bytes", (unsigned long long)g_size[0], (unsigned long long)g_size[1]); +} + +// ---- descriptor memory: per command slot and recording thread (vkb_mt.c), host-visible (device-local with resizable +// BAR), grown by replacing it +typedef struct { VkBuffer buf; VkDeviceMemory mem; uint8_t *map; VkDeviceSize size, used; VkDeviceAddress addr; } dmem_t; +static dmem_t g_mem[VKB_SLOTS][VKB_MAXREC]; +static dmem_t g_old[VKB_SLOTS][VKB_MAXREC][8]; // replaced during a submission: freed when the slot comes round again +static int g_nold[VKB_SLOTS][VKB_MAXREC]; +static __thread uint64_t g_gen = 1; // changes whenever this thread's memory changes (offsets made before are void) +static __thread uint64_t g_bound_rec, g_bound_gen; +static uint32_t g_type = UINT32_MAX; // the same answer whichever thread finds it + +static int dmem_new(dmem_t *m, VkDeviceSize size) { + VkBufferCreateInfo bi = {VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .size = size, + .usage = VK_BUFFER_USAGE_RESOURCE_DESCRIPTOR_BUFFER_BIT_EXT | VK_BUFFER_USAGE_SAMPLER_DESCRIPTOR_BUFFER_BIT_EXT | + VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT}; + if (!vk_check(vk.vkCreateBuffer(vkc.device, &bi, NULL, &m->buf), "descriptor buffer")) return 0; + VkMemoryRequirements req; + vk.vkGetBufferMemoryRequirements(vkc.device, m->buf, &req); + if (g_type == UINT32_MAX) { + VkMemoryPropertyFlags want = VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT | VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT; + for (uint32_t i = 0; i < vkc.memory.memoryTypeCount && g_type == UINT32_MAX; i++) + if ((req.memoryTypeBits & (1u << i)) && (vkc.memory.memoryTypes[i].propertyFlags & want) == want && + vkc.memory.memoryHeaps[vkc.memory.memoryTypes[i].heapIndex].size >= (1ull << 30)) g_type = i; // not the 256 MiB BAR window + want = VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT; + for (uint32_t i = 0; i < vkc.memory.memoryTypeCount && g_type == UINT32_MAX; i++) + if ((req.memoryTypeBits & (1u << i)) && (vkc.memory.memoryTypes[i].propertyFlags & want) == want) g_type = i; + } + VkMemoryAllocateFlagsInfo fl = {VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_FLAGS_INFO, .flags = VK_MEMORY_ALLOCATE_DEVICE_ADDRESS_BIT}; + VkMemoryAllocateInfo ai = {VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO, &fl, .allocationSize = req.size, .memoryTypeIndex = g_type}; + if (g_type == UINT32_MAX || !vk_check(vk.vkAllocateMemory(vkc.device, &ai, NULL, &m->mem), "descriptor memory")) { + vk.vkDestroyBuffer(vkc.device, m->buf, NULL); + return 0; + } + vk.vkBindBufferMemory(vkc.device, m->buf, m->mem, 0); + void *p; + if (!vk_check(vk.vkMapMemory(vkc.device, m->mem, 0, VK_WHOLE_SIZE, 0, &p), "map descriptor memory")) return 0; + VkBufferDeviceAddressInfo ad = {VK_STRUCTURE_TYPE_BUFFER_DEVICE_ADDRESS_INFO, .buffer = m->buf}; + m->map = p; m->size = size; m->used = 0; m->addr = vk.vkGetBufferDeviceAddress(vkc.device, &ad); + return 1; +} +static void dmem_free(dmem_t *m) { + if (!m->buf) return; + vk.vkDestroyBuffer(vkc.device, m->buf, NULL); + vk.vkFreeMemory(vkc.device, m->mem, NULL); + memset(m, 0, sizeof *m); +} +void vkb_db_slot_reset(int slot) { // the slot's previous submission is complete + for (int w = 0; w < VKB_MAXREC; w++) { + for (int i = 0; i < g_nold[slot][w]; i++) dmem_free(&g_old[slot][w][i]); + g_nold[slot][w] = 0; + g_mem[slot][w].used = 0; + } +} +uint64_t vkb_db_gen(void) { return g_gen; } + +// ---- the bytes of one descriptor, per submission (tables per recording thread) +enum { CACHE = 1 << 15 }; +typedef struct { uint64_t rec, k0, k1, k2, k3; uint8_t len, bytes[63]; } dcache_t; +static __thread dcache_t *g_cache; +static vkb_cnt g_get_calls, g_cache_hits; +typedef struct { uint64_t rec; VkBuffer buf; VkDeviceAddress addr; } addr_t; +static __thread addr_t *g_addr; // 4096 + +static VkDeviceAddress buffer_address(VkBuffer b, uint64_t rec) { + if (!g_addr && !(g_addr = calloc(4096, sizeof *g_addr))) die("buffer addresses"); + uint32_t h = (uint32_t)(((uint64_t)(uintptr_t)b * 0x9e3779b97f4a7c15ull) >> 52); + for (uint32_t k = h;; k = (k + 1) & 4095) { + if (g_addr[k].rec == rec && g_addr[k].buf == b) return g_addr[k].addr; + if (g_addr[k].rec != rec) { + VkBufferDeviceAddressInfo ad = {VK_STRUCTURE_TYPE_BUFFER_DEVICE_ADDRESS_INFO, .buffer = b}; + g_addr[k].rec = rec; g_addr[k].buf = b; g_addr[k].addr = vk.vkGetBufferDeviceAddress(vkc.device, &ad); + return g_addr[k].addr; + } + } +} +static size_t type_size(VkDescriptorType t) { + switch (t) { + case VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER: return vkc.db.uniformBufferDescriptorSize; + case VK_DESCRIPTOR_TYPE_STORAGE_BUFFER: return vkc.db.storageBufferDescriptorSize; + case VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER: return vkc.db.combinedImageSamplerDescriptorSize; + case VK_DESCRIPTOR_TYPE_STORAGE_IMAGE: return vkc.db.storageImageDescriptorSize; + default: return 0; + } +} +static const uint8_t *descriptor(const VkWriteDescriptorSet *w, uint64_t rec, size_t *len) { + uint64_t k0 = (uint64_t)w->descriptorType, k1, k2, k3; + if (w->pBufferInfo) { k1 = (uint64_t)(uintptr_t)w->pBufferInfo->buffer; k2 = w->pBufferInfo->offset; k3 = w->pBufferInfo->range; } + else { k1 = (uint64_t)(uintptr_t)w->pImageInfo->sampler; k2 = (uint64_t)(uintptr_t)w->pImageInfo->imageView; k3 = (uint64_t)w->pImageInfo->imageLayout; } + if (!g_cache && !(g_cache = calloc(CACHE, sizeof *g_cache))) die("descriptor cache"); + uint64_t h = (k0 * 0x9e3779b97f4a7c15ull) ^ (k1 * 0xc2b2ae3d27d4eb4full) ^ (k2 * 0x165667b19e3779f9ull) ^ (k3 * 0x27d4eb2f165667c5ull); + uint32_t at = (uint32_t)(h >> 49) & (CACHE - 1); + dcache_t *e = NULL; + for (int probe = 0; probe < 16; probe++) { + dcache_t *c = &g_cache[(at + (uint32_t)probe) & (CACHE - 1)]; + if (c->rec == rec && c->k0 == k0 && c->k1 == k1 && c->k2 == k2 && c->k3 == k3) { vkb_cnt_add(g_cache_hits, 1); *len = c->len; return c->bytes; } + if (c->rec != rec) { e = c; break; } + } + static __thread uint8_t spill[64]; // the table region is full: no caching for this one + uint8_t *out = e ? e->bytes : spill; + size_t n = type_size(w->descriptorType); + if (n > 63) die("descriptor of %zu bytes", n); + VkDescriptorGetInfoEXT gi = {VK_STRUCTURE_TYPE_DESCRIPTOR_GET_INFO_EXT, .type = w->descriptorType}; + VkDescriptorAddressInfoEXT ai = {VK_STRUCTURE_TYPE_DESCRIPTOR_ADDRESS_INFO_EXT}; + if (w->pBufferInfo) { + ai.address = buffer_address(w->pBufferInfo->buffer, rec) + w->pBufferInfo->offset; + ai.range = w->pBufferInfo->range; + if (w->descriptorType == VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER) gi.data.pUniformBuffer = &ai; else gi.data.pStorageBuffer = &ai; + } else if (w->descriptorType == VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER) gi.data.pCombinedImageSampler = w->pImageInfo; + else gi.data.pStorageImage = w->pImageInfo; + vk.vkGetDescriptorEXT(vkc.device, &gi, n, out); + vkb_cnt_add(g_get_calls, 1); + if (e) { e->rec = rec; e->k0 = k0; e->k1 = k1; e->k2 = k2; e->k3 = k3; e->len = (uint8_t)n; } + *len = n; + return out; +} + +// This thread's descriptor memory of the slot bound to the command buffer (rec): once per command buffer, and again +// whenever the memory was replaced. +void vkb_db_bind(VkCommandBuffer cb, int slot, uint64_t rec) { + dmem_t *m = &g_mem[slot][vkb_worker()]; + if (!m->buf || (g_bound_rec == rec && g_bound_gen == g_gen)) return; + VkDescriptorBufferBindingInfoEXT bb = {VK_STRUCTURE_TYPE_DESCRIPTOR_BUFFER_BINDING_INFO_EXT, .address = m->addr, + .usage = VK_BUFFER_USAGE_RESOURCE_DESCRIPTOR_BUFFER_BIT_EXT | VK_BUFFER_USAGE_SAMPLER_DESCRIPTOR_BUFFER_BIT_EXT}; + vk.vkCmdBindDescriptorBuffersEXT(cb, 1, &bb); + g_bound_rec = rec; g_bound_gen = g_gen; +} +// A draw's descriptors (the writes' dstSet is ignored) in the slot's descriptor memory: their offset there. +VkDeviceSize vkb_db_set(VkCommandBuffer cb, int slot, uint64_t rec, const VkWriteDescriptorSet *w, int n, int compute) { + int wk = vkb_worker(); + dmem_t *m = &g_mem[slot][wk]; + VkDeviceSize a = vkc.db.descriptorBufferOffsetAlignment ? vkc.db.descriptorBufferOffsetAlignment : 64; + VkDeviceSize at = (m->used + a - 1) & ~(a - 1); + if (!m->buf || at + g_size[compute] > m->size) { // more memory: the old one stays until the slot's submission is done + VkDeviceSize size = m->size ? m->size * 2 : 16u << 20; + VkDeviceSize cap = vkc.db.maxResourceDescriptorBufferRange < vkc.db.maxSamplerDescriptorBufferRange + ? vkc.db.maxResourceDescriptorBufferRange : vkc.db.maxSamplerDescriptorBufferRange; + if (cap && size > cap) size = cap; + if (m->buf) { + if (g_nold[slot][wk] == 8) die("descriptor memory: replaced 8 times in one submission"); + g_old[slot][wk][g_nold[slot][wk]++] = *m; + memset(m, 0, sizeof *m); + } + if (!dmem_new(m, size)) die("descriptor memory"); + VLOG("descriptor memory of slot %d, recording thread %d: %llu MiB", slot, wk, (unsigned long long)(size >> 20)); + g_gen++; + at = 0; + } + vkb_db_bind(cb, slot, rec); + uint64_t sub = mtl_last_fence() + 1; // the bytes stay valid for the submission (rec: this command buffer) + uint8_t *base = m->map + at; + for (int i = 0; i < n; i++) { + size_t len; + const uint8_t *d = descriptor(&w[i], sub, &len); + memcpy(base + g_off[compute][w[i].dstBinding], d, len); + } + m->used = at + g_size[compute]; + return at; +} +void vkb_db_stats(uint64_t *get_calls, uint64_t *cache_hits) { *get_calls = vkb_cnt_sum(g_get_calls); *cache_hits = vkb_cnt_sum(g_cache_hits); } diff --git a/src/gfx/vkb_draw.c b/src/gfx/vkb_draw.c new file mode 100644 index 0000000..6b50c11 --- /dev/null +++ b/src/gfx/vkb_draw.c @@ -0,0 +1,1779 @@ +// Drawing for the Vulkan NVN replayer (milestone 3.3 step 5): the render state the command stream sets, the texture and +// sampler pools, descriptors, vertex input, draws and dispatches. +// - State: what the frontend records (nvn_obj.h payloads), kept as NVN values and turned into Vulkan dynamic state at +// each draw (shader objects: every state is dynamic). [temporary] All of it is set before every draw; skipping unchanged +// state is a later optimisation measured against the profile. +// - Descriptors: one set per draw from the submission's pool in the universal layout (vkb_shader.h). Constant buffers +// c3..c16 are the game's uniform buffers (slot - 3), c1 the shader's own constants, c0 / c2 the NVN driver banks, +// built here from the bound storage buffers and handles (the offsets of MacNativeGames' totk_xlate Service.cs). +// - Textures and samplers: the game's texture / sampler pool registrations (mtl_reg_tex / mtl_reg_samp); a combined +// handle is texture id | sampler id << 20. +#include +#include +#include + +#include "compat/gfx_compat.h" +#include "host/hle.h" +#include "vkb_shader.h" +#include "vkb_audit.h" +#include "vkb_lock.h" + +#define VLOG(...) LOG("[vk] " __VA_ARGS__) + +// ---------------- upload memory and descriptor pools, per command slot ---------------- +typedef struct { VkBuffer buf; VkDeviceMemory mem; uint8_t *map; VkDeviceSize size, used; } chunk_t; +typedef struct { chunk_t c[32]; int n, cur; VkDescriptorPool pool[64]; int npool, curpool; } slot_res; +static slot_res g_res[VKB_SLOTS][VKB_MAXREC]; // per slot and recording thread (vkb_mt.c) +static struct { VkBuffer buf; VkDeviceMemory mem; VkDeviceSize used; } g_mpool[VKB_SLOTS]; // device memory for mirrors +static uint32_t g_ring_type = UINT32_MAX; + +static uint32_t memory_type(uint32_t bits, VkMemoryPropertyFlags want) { + for (uint32_t i = 0; i < vkc.memory.memoryTypeCount; i++) + if ((bits & (1u << i)) && (vkc.memory.memoryTypes[i].propertyFlags & want) == want) return i; + return UINT32_MAX; +} + +static int chunk_new(chunk_t *c, VkDeviceSize size) { + VkBufferCreateInfo bi = {VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .size = size, + .usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT | VK_BUFFER_USAGE_UNIFORM_BUFFER_BIT | VK_BUFFER_USAGE_STORAGE_BUFFER_BIT | + VK_BUFFER_USAGE_VERTEX_BUFFER_BIT | VK_BUFFER_USAGE_INDEX_BUFFER_BIT | VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT}; + if (!vk_check(vk.vkCreateBuffer(vkc.device, &bi, NULL, &c->buf), "upload buffer")) return 0; + VkMemoryRequirements req; + vk.vkGetBufferMemoryRequirements(vkc.device, c->buf, &req); + if (g_ring_type == UINT32_MAX) { // device-local and mapped where the GPU offers it (resizable BAR), else host memory + g_ring_type = memory_type(req.memoryTypeBits, VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT | VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT); + if (g_ring_type != UINT32_MAX && vkc.memory.memoryHeaps[vkc.memory.memoryTypes[g_ring_type].heapIndex].size < (1ull << 30)) + g_ring_type = UINT32_MAX; // the 256 MiB window without resizable BAR: keep it for the driver + if (g_ring_type == UINT32_MAX) g_ring_type = memory_type(req.memoryTypeBits, VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT); + } + VkMemoryAllocateFlagsInfo fl = {VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_FLAGS_INFO, .flags = VK_MEMORY_ALLOCATE_DEVICE_ADDRESS_BIT}; // descriptor buffers + VkMemoryAllocateInfo ai = {VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO, &fl, .allocationSize = req.size, .memoryTypeIndex = g_ring_type}; + if (!vk_check(vk.vkAllocateMemory(vkc.device, &ai, NULL, &c->mem), "upload memory")) return 0; + vk.vkBindBufferMemory(vkc.device, c->buf, c->mem, 0); + void *m; + if (!vk_check(vk.vkMapMemory(vkc.device, c->mem, 0, VK_WHOLE_SIZE, 0, &m), "map upload memory")) return 0; + c->map = m; c->size = size; c->used = 0; + return 1; +} + +static __thread uint64_t g_rec; // the command buffer this thread records (its caches below compare against it) +static _Atomic uint64_t g_rec_next; // unique across threads +// Upload memory, descriptor memory and descriptor sets are this thread's for the whole submission: what refers to +// them is cached per submission (g_sub); what a command buffer has bound or set, per command buffer (g_rec). +static __thread uint64_t g_sub; +void vkb_rec_begin(void) { g_rec = atomic_fetch_add(&g_rec_next, 1) + 1; g_sub = mtl_last_fence() + 1; } +static int no_cache(void) { // TOTK_VK_NO_RECORD_CACHE=1 (development): every draw records all its state again (A/B) + static int v = -1; + if (v < 0) { const char *e = getenv("TOTK_VK_NO_RECORD_CACHE"); v = e && *e && *e != '0'; } + return v; +} +void vkb_slot_reset(int slot) { // the slot's previous submission has completed; a new one starts recording + vkb_rec_begin(); + if (vkb_db_on()) vkb_db_slot_reset(slot); + for (int w = 0; w < VKB_MAXREC; w++) { + slot_res *r = &g_res[slot][w]; + for (int i = 0; i < r->n; i++) r->c[i].used = 0; + r->cur = 0; + for (int i = 0; i < r->npool; i++) vk.vkResetDescriptorPool(vkc.device, r->pool[i], 0); + r->curpool = 0; + } + g_mpool[slot].used = 0; +} + +void *vkb_ring_alloc(VkDeviceSize n, VkDeviceSize align, VkBuffer *buf, VkDeviceSize *off) { + slot_res *r = &g_res[vkb_slot()][vkb_worker()]; + for (;;) { + if (r->cur < r->n) { + chunk_t *c = &r->c[r->cur]; + VkDeviceSize at = (c->used + align - 1) & ~(align - 1); + if (at + n <= c->size) { c->used = at + n; *buf = c->buf; *off = at; return c->map + at; } + r->cur++; + continue; + } + if (r->n == 32) die("upload memory: 32 chunks (2 GiB) in one submission"); + if (!chunk_new(&r->c[r->n], n > (64u << 20) ? n : (64u << 20))) die("upload memory"); + r->n++; + } +} + +static VkDescriptorSet alloc_set(int compute) { + slot_res *r = &g_res[vkb_slot()][vkb_worker()]; + VkDescriptorSetLayout layout = vkb_set_layout(compute); + for (;;) { + if (r->curpool < r->npool) { + VkDescriptorSetAllocateInfo ai = {VK_STRUCTURE_TYPE_DESCRIPTOR_SET_ALLOCATE_INFO, .descriptorPool = r->pool[r->curpool], + .descriptorSetCount = 1, .pSetLayouts = &layout}; + VkDescriptorSet set; + VkResult res = vk.vkAllocateDescriptorSets(vkc.device, &ai, &set); + if (res == VK_SUCCESS) return set; + r->curpool++; + continue; + } + if (r->npool == 64) die("descriptor pools: 64 in one submission"); + VkDescriptorPoolSize sizes[] = {{VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER, 2048 * 2 * 18}, {VK_DESCRIPTOR_TYPE_STORAGE_BUFFER, 2048 * 2 * 16}, + {VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER, 2048 * 2 * 24}, {VK_DESCRIPTOR_TYPE_STORAGE_IMAGE, 2048 * 2 * 4}}; + VkDescriptorPoolCreateInfo pi = {VK_STRUCTURE_TYPE_DESCRIPTOR_POOL_CREATE_INFO, .maxSets = 2048, .poolSizeCount = 4, .pPoolSizes = sizes}; + if (!vk_check(vk.vkCreateDescriptorPool(vkc.device, &pi, NULL, &r->pool[r->npool]), "descriptor pool")) die("descriptor pool"); + r->npool++; + } +} + +// ---------------- the render state the command stream sets ---------------- +static __thread struct { + const prog_t *prog; uint64_t prog_serial; const vkb_shader *sh[6]; int prog_ok; + blend_st blend[8]; color_st color; chmask_st chmask; ms_st ms; poly_st poly; ds_st ds; + int has_blend[8], has_color, has_chmask, has_ms, has_poly, has_ds; + float vp[16][6]; int nvp; float dr[2]; int has_dr; + p_scissor sc; int has_sc; + struct { uint32_t value_mask, write_mask, ref; } stencil[2]; // front, back + p_poly_offset po; float blend_color[4]; + uint32_t vattr[16]; int nvattr; vs_st vstream[16]; int nvstream; NVNbufferRange vbuf[16]; + p_bind_buf ubo[6][18], ssbo[6][16]; + uint64_t tex[6][32], img[6][8]; + int depth_clamp, raster_discard; + const char *bad_key; // the unsupported shader of a program that cannot be drawn (audit) + int64_t copy_stride; +} S; // this recording thread's (vkb_mt.c gives each chunk the state at its start) + +void vkb_state_reset(void) { + memset(&S, 0, sizeof S); + for (int f = 0; f < 2; f++) { S.stencil[f].value_mask = 0xFF; S.stencil[f].write_mask = 0xFF; } + S.chmask.mask = 0xFFFFFFFF; +} +size_t vkb_state_size(void) { return sizeof S; } +void vkb_state_save(void *dst) { memcpy(dst, &S, sizeof S); } +void vkb_state_load(const void *src) { memcpy(&S, src, sizeof S); } + +void vkb_state(uint32_t op, const uint8_t *a) { + switch (op) { + case OP_BIND_PROGRAM: { + const p_bind_program *b = (const p_bind_program *)a; + S.prog = b->prog; + S.prog_serial = 0; // resolved at the next draw + break; + } + case OP_BIND_STATE: { + const p_bind_state *b = (const p_bind_state *)a; + switch (b->kind) { + case 0: { blend_st s; memcpy(&s, b->bytes, sizeof s); S.blend[s.target & 7] = s; S.has_blend[s.target & 7] = 1; break; } + case 1: memcpy(&S.color, b->bytes, sizeof S.color); S.has_color = 1; break; + case 2: memcpy(&S.chmask, b->bytes, sizeof S.chmask); S.has_chmask = 1; break; + case 3: memcpy(&S.ms, b->bytes, sizeof S.ms); S.has_ms = 1; break; + case 4: memcpy(&S.poly, b->bytes, sizeof S.poly); S.has_poly = 1; break; + case 5: memcpy(&S.ds, b->bytes, sizeof S.ds); S.has_ds = 1; break; + } + break; + } + case OP_VIEWPORTS: { + const p_viewports *v = (const p_viewports *)a; + for (int i = 0; i < v->n && v->first + i < 16; i++) memcpy(S.vp[v->first + i], v->v[i], sizeof S.vp[0]); + if (v->first + v->n > S.nvp) S.nvp = v->first + v->n > 16 ? 16 : v->first + v->n; + break; + } + case OP_SCISSOR: memcpy(&S.sc, a, sizeof S.sc); S.has_sc = 1; break; + case OP_DEPTH_RANGE: { const p_depth_range *d = (const p_depth_range *)a; S.dr[0] = d->n; S.dr[1] = d->f; S.has_dr = 1; break; } + case OP_STENCIL: { + const p_stencil *s = (const p_stencil *)a; + for (int f = 0; f < 2; f++) { + if (!(s->face & (1 << f))) continue; + if (s->which == 0) S.stencil[f].value_mask = (uint32_t)s->value_mask; + else if (s->which == 1) S.stencil[f].write_mask = (uint32_t)s->mask; + else S.stencil[f].ref = (uint32_t)s->ref; + } + break; + } + case OP_POLY_OFFSET: memcpy(&S.po, a, sizeof S.po); break; + case OP_BLEND_COLOR: memcpy(S.blend_color, a, sizeof S.blend_color); break; + case OP_BIND_VATTR: { + const p_bind_vattr *v = (const p_bind_vattr *)a; + for (int i = 0; i < v->n && v->first + i < 16; i++) S.vattr[v->first + i] = v->st[i]; + S.nvattr = v->first + v->n > 16 ? 16 : v->first + v->n; + break; + } + case OP_BIND_VSTREAM: { + const p_bind_vstream *v = (const p_bind_vstream *)a; + for (int i = 0; i < v->n && v->first + i < 16; i++) S.vstream[v->first + i] = v->st[i]; + S.nvstream = v->first + v->n > 16 ? 16 : v->first + v->n; + break; + } + case OP_BIND_VBUF: { + const p_bind_vbuf *v = (const p_bind_vbuf *)a; + for (int i = 0; i < v->n && v->first + i < 16; i++) S.vbuf[v->first + i] = v->r[i]; + break; + } + case OP_BIND_UBO: { const p_bind_buf *b = (const p_bind_buf *)a; if (b->stage < 6 && b->index >= 0 && b->index < 18) S.ubo[b->stage][b->index] = *b; break; } + case OP_BIND_SSBO: { const p_bind_buf *b = (const p_bind_buf *)a; if (b->stage < 6 && b->index >= 0 && b->index < 16) S.ssbo[b->stage][b->index] = *b; break; } + case OP_BIND_TEX: case OP_BIND_IMG: { + const p_bind_handles *b = (const p_bind_handles *)a; + if (b->stage < 0 || b->stage >= 6) break; + uint64_t *dst = op == OP_BIND_TEX ? S.tex[b->stage] : S.img[b->stage]; + int max = op == OP_BIND_TEX ? 32 : 8; + for (int i = 0; i < b->n && b->first + i < max; i++) dst[b->first + i] = b->h[i]; + break; + } + case OP_DEPTH_CLAMP: S.depth_clamp = ((const p_flag *)a)->value; break; + case OP_RASTER_DISCARD: S.raster_discard = ((const p_flag *)a)->value; break; + case OP_COPY_STRIDE: S.copy_stride = ((const p_copy_stride *)a)->stride; break; + default: break; + } +} + +// ---------------- texture and sampler pools ---------------- +typedef struct { tex_t t; view_t v; uint8_t has_view, valid; } treg_t; +typedef struct { samp_par p; VkSampler s; uint8_t valid; } sreg_t; +static treg_t *g_treg; +static sreg_t *g_sreg; +static int g_ntreg, g_nsreg; +static vkb_rw g_reg_mu; // game threads register while the recording threads read + +static void *grow(void *a, int *n, int need, size_t elem) { + if (need < *n) return a; + int m = *n ? *n : 1024; + while (m <= need) m *= 2; + a = realloc(a, (size_t)m * elem); + if (!a) die("pool registry"); + memset((uint8_t *)a + (size_t)*n * elem, 0, (size_t)(m - *n) * elem); + *n = m; + return a; +} +void mtl_reg_tex(int id, const tex_t *t, const view_t *v) { + if (id < 0 || id >= (1 << 20)) return; + vkb_rw_write(&g_reg_mu, LK_TEXREG); + g_treg = grow(g_treg, &g_ntreg, id, sizeof *g_treg); + treg_t *r = &g_treg[id]; + r->valid = t != NULL; + if (t) r->t = *t; + r->has_view = v != NULL; + if (v) r->v = *v; + vkb_rw_write_end(&g_reg_mu); +} +void mtl_reg_samp(int id, const samp_par *p) { + if (id < 0 || id >= (1 << 12)) return; + vkb_rw_write(&g_reg_mu, LK_TEXREG); + g_sreg = grow(g_sreg, &g_nsreg, id, sizeof *g_sreg); + sreg_t *r = &g_sreg[id]; + VkSampler old = VK_NULL_HANDLE; + if (r->s && (!p || memcmp(&r->p, p, sizeof *p))) { old = r->s; r->s = VK_NULL_HANDLE; } + r->valid = p != NULL; + if (p) r->p = *p; + vkb_rw_write_end(&g_reg_mu); + if (old) { mtl_finish(); vk.vkDestroySampler(vkc.device, old, NULL); } // [temporary] waits for the GPU +} +void mtl_set_tex_pool_n(int n) { (void)n; } + +static VkSamplerAddressMode wrap(uint8_t w) { + switch (w) { + case 1: return VK_SAMPLER_ADDRESS_MODE_REPEAT; + case 2: case 3: case 4: return VK_SAMPLER_ADDRESS_MODE_MIRROR_CLAMP_TO_EDGE; + case 5: return VK_SAMPLER_ADDRESS_MODE_CLAMP_TO_BORDER; + case 6: return VK_SAMPLER_ADDRESS_MODE_MIRRORED_REPEAT; + default: return VK_SAMPLER_ADDRESS_MODE_CLAMP_TO_EDGE; // CLAMP (GL legacy), CLAMP_TO_EDGE + } +} +static VkSampler sampler(int id) { + static VkSampler fallback; // made before threads record (vkb_draw_mt_init) + vkb_rw_read(&g_reg_mu, LK_TEXREG); + int valid = id >= 0 && id < g_nsreg && g_sreg[id].valid; + VkSampler have = valid ? g_sreg[id].s : VK_NULL_HANDLE; + samp_par par; + if (valid) par = g_sreg[id].p; + vkb_rw_read_end(&g_reg_mu); + if (have) return have; + if (!valid) { + if (!fallback) { + VkSamplerCreateInfo ci = {VK_STRUCTURE_TYPE_SAMPLER_CREATE_INFO, .magFilter = VK_FILTER_LINEAR, .minFilter = VK_FILTER_LINEAR, + .maxLod = VK_LOD_CLAMP_NONE}; + vk.vkCreateSampler(vkc.device, &ci, NULL, &fallback); + } + return fallback; + } + const samp_par *p = ∥ + int mip = p->minf >= 2; + VkSamplerReductionModeCreateInfo red = {VK_STRUCTURE_TYPE_SAMPLER_REDUCTION_MODE_CREATE_INFO, + .reductionMode = p->reduction == 1 ? VK_SAMPLER_REDUCTION_MODE_MIN : p->reduction == 2 ? VK_SAMPLER_REDUCTION_MODE_MAX + : VK_SAMPLER_REDUCTION_MODE_WEIGHTED_AVERAGE}; + // [temporary] border colour: the nearest standard one (custom border colours: VK_EXT_custom_border_color later) + { + const float *b = p->border; + int standard = (b[0] == 0 && b[1] == 0 && b[2] == 0 && (b[3] == 0 || b[3] == 1)) || (b[0] == 1 && b[1] == 1 && b[2] == 1 && b[3] == 1); + if (!standard && (p->ws == 5 || p->wt == 5 || p->wr == 5 || p->ws == 4 || p->wt == 4 || p->wr == 4)) vkb_audit(A_BORDER_APPROX, NULL); + } + VkBorderColor border = p->border[3] < 0.5f ? VK_BORDER_COLOR_FLOAT_TRANSPARENT_BLACK + : p->border[0] > 0.5f ? VK_BORDER_COLOR_FLOAT_OPAQUE_WHITE : VK_BORDER_COLOR_FLOAT_OPAQUE_BLACK; + VkSamplerCreateInfo ci = {VK_STRUCTURE_TYPE_SAMPLER_CREATE_INFO, &red, + .magFilter = p->magf ? VK_FILTER_LINEAR : VK_FILTER_NEAREST, + .minFilter = p->minf == 1 || p->minf == 3 || p->minf == 5 ? VK_FILTER_LINEAR : VK_FILTER_NEAREST, + .mipmapMode = p->minf == 4 || p->minf == 5 ? VK_SAMPLER_MIPMAP_MODE_LINEAR : VK_SAMPLER_MIPMAP_MODE_NEAREST, + .addressModeU = wrap(p->ws), .addressModeV = wrap(p->wt), .addressModeW = wrap(p->wr), + .mipLodBias = p->lod_bias, .anisotropyEnable = p->aniso > 1.0f, .maxAnisotropy = p->aniso > 16.0f ? 16.0f : p->aniso, + .compareEnable = p->cmp_mode == 1, .compareOp = (VkCompareOp)(p->cmp_func ? p->cmp_func - 1 : 7), + .minLod = p->lod_min, .maxLod = mip ? p->lod_max : 0.25f, // no mipmap filter: the base level only (GL) + .borderColor = border}; + if (ci.maxLod < ci.minLod) ci.maxLod = ci.minLod; + VkSampler s; + if (!vk_check(vk.vkCreateSampler(vkc.device, &ci, NULL, &s), "sampler")) return sampler(-1); + vkb_rw_write(&g_reg_mu, LK_TEXREG); + int kept = id < g_nsreg && g_sreg[id].valid && !memcmp(&g_sreg[id].p, &par, sizeof par) && !g_sreg[id].s; + if (kept) g_sreg[id].s = s; + else if (id < g_nsreg && g_sreg[id].valid && !memcmp(&g_sreg[id].p, &par, sizeof par) && g_sreg[id].s) { // another thread made one meanwhile + VkSampler other = g_sreg[id].s; + vkb_rw_write_end(&g_reg_mu); + vk.vkDestroySampler(vkc.device, s, NULL); + return other; + } + vkb_rw_write_end(&g_reg_mu); + return s; // [temporary] a sampler re-registered meanwhile leaks this one +} + +// A 1x1 black image for unbound texture slots, one per view type. Created (cleared) outside a pass: dummy_init. +static VkImage g_dummy; +static void dummy_init(void) { + VkImage *imgp = &g_dummy; + if (*imgp) return; + { + VkImage img; + VkImageCreateInfo ci = {VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO, .flags = VK_IMAGE_CREATE_CUBE_COMPATIBLE_BIT, + .imageType = VK_IMAGE_TYPE_2D, .format = VK_FORMAT_R8G8B8A8_UNORM, .extent = {1, 1, 1}, .mipLevels = 1, .arrayLayers = 6, + .samples = VK_SAMPLE_COUNT_1_BIT, .tiling = VK_IMAGE_TILING_OPTIMAL, + .usage = VK_IMAGE_USAGE_SAMPLED_BIT | VK_IMAGE_USAGE_STORAGE_BIT | VK_IMAGE_USAGE_TRANSFER_DST_BIT}; + vk.vkCreateImage(vkc.device, &ci, NULL, &img); + VkMemoryRequirements req; + vk.vkGetImageMemoryRequirements(vkc.device, img, &req); + VkMemoryAllocateInfo ai = {VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO, .allocationSize = req.size, + .memoryTypeIndex = memory_type(req.memoryTypeBits, VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT)}; + VkDeviceMemory m; + vk.vkAllocateMemory(vkc.device, &ai, NULL, &m); + vk.vkBindImageMemory(vkc.device, img, m, 0); + VkCommandBuffer cb = vkb_cmd(); // cleared to 0 and left in GENERAL (sampled and storage) + VkImageMemoryBarrier2 b = {VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2, .dstStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + .dstAccessMask = VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT, .oldLayout = VK_IMAGE_LAYOUT_UNDEFINED, + .newLayout = VK_IMAGE_LAYOUT_GENERAL, .srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, + .image = img, .subresourceRange = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 1, 0, 6}}; + VkDependencyInfo di = {VK_STRUCTURE_TYPE_DEPENDENCY_INFO, .imageMemoryBarrierCount = 1, .pImageMemoryBarriers = &b}; + vk.vkCmdPipelineBarrier2(cb, &di); + VkClearColorValue z = {{0, 0, 0, 0}}; + VkImageSubresourceRange rg = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 1, 0, 6}; + vk.vkCmdClearColorImage(cb, img, VK_IMAGE_LAYOUT_GENERAL, &z, 1, &rg); + b.srcStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT; b.srcAccessMask = VK_ACCESS_2_MEMORY_WRITE_BIT; + b.oldLayout = VK_IMAGE_LAYOUT_GENERAL; + vk.vkCmdPipelineBarrier2(cb, &di); + *imgp = img; + } +} +static VkImageView g_dummy_views[8]; // made before threads record (vkb_draw_mt_init) +static VkImageView dummy_view(VkImageViewType type) { + VkImageView *views = g_dummy_views; + VkImage img = g_dummy; + if (type > VK_IMAGE_VIEW_TYPE_CUBE_ARRAY) type = VK_IMAGE_VIEW_TYPE_2D; + if (type == VK_IMAGE_VIEW_TYPE_3D) type = VK_IMAGE_VIEW_TYPE_2D_ARRAY; // [temporary] a 2D image cannot have a 3D view + if (!views[type]) { + uint32_t layers = type == VK_IMAGE_VIEW_TYPE_CUBE || type == VK_IMAGE_VIEW_TYPE_CUBE_ARRAY ? 6 : 1; + VkImageViewCreateInfo ci = {VK_STRUCTURE_TYPE_IMAGE_VIEW_CREATE_INFO, .image = img, .viewType = type, .format = VK_FORMAT_R8G8B8A8_UNORM, + .subresourceRange = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 1, 0, layers}}; + vk.vkCreateImageView(vkc.device, &ci, NULL, &views[type]); + } + return views[type]; +} + +static VkImageViewType view_type_of(uint16_t t) { // Ryujinx SamplerType + int array = (t & 0x100) != 0; + switch (t & 0xff) { + case 1: return array ? VK_IMAGE_VIEW_TYPE_1D_ARRAY : VK_IMAGE_VIEW_TYPE_1D; + case 4: return VK_IMAGE_VIEW_TYPE_3D; + case 5: return array ? VK_IMAGE_VIEW_TYPE_CUBE_ARRAY : VK_IMAGE_VIEW_TYPE_CUBE; + default: return array ? VK_IMAGE_VIEW_TYPE_2D_ARRAY : VK_IMAGE_VIEW_TYPE_2D; + } +} + +// The image a texture handle names (NULL: none / not an image yet); *view points to a copy of the pool entry's view. +static vkb_image *handle_image(uint64_t h, const view_t **view) { + static __thread view_t views[64]; + static __thread unsigned next; + int id = (int)(h & 0xFFFFF); + vkb_rw_read(&g_reg_mu, LK_TEXREG); + if (id <= 0 || id >= g_ntreg || !g_treg[id].valid) { vkb_rw_read_end(&g_reg_mu); *view = NULL; return NULL; } + tex_t t = g_treg[id].t; + view_t *v = NULL; + if (g_treg[id].has_view) { v = &views[next++ & 63]; *v = g_treg[id].v; } + vkb_rw_read_end(&g_reg_mu); + *view = v; + return vkb_image_of(&t); +} + +// ---------------- sampled render targets ---------------- +// TOTK samples textures bound as targets of the same draw (decals reading the G-buffer, effects reading depth). With +// VK_EXT_attachment_feedback_loop_layout such targets stay attached in the feedback loop layout and are sampled there; +// without it they would leave the pass, and the draw saw an empty texture instead (the audit's top substitution). +static __thread vkb_image *g_fb_rt[9]; // this draw's targets: colour 0-7, depth 8 +static __thread uint32_t g_fb; // the targets it samples (vkb.h VKB_FB_DEPTH) + +static int fb_index(const vkb_image *im) { // the target bit of a sampled image, -1: not a target of this draw + for (int i = 0; im && i < 9; i++) if (g_fb_rt[i] == im) return i; + return -1; +} +static void feedback_scan(const vkb_shader *vs, const vkb_shader *fs) { + g_fb = 0; + memset(g_fb_rt, 0, sizeof g_fb_rt); + if (!vkc.has_feedback_loop) return; + const p_set_rt *rt = vkb_rt(); + int n = rt->n < 0 ? 0 : rt->n > 8 ? 8 : rt->n, any = 0; + for (int i = 0; i < n; i++) if (rt->color[i]) any |= (g_fb_rt[i] = vkb_image_of(rt->color[i])) != NULL; + if (rt->depth) any |= (g_fb_rt[8] = vkb_image_of(rt->depth)) != NULL; + if (!any) return; + for (int st = 0; st < 2; st++) { + const vkb_shader *sh = st ? fs : vs; + for (int i = 0; sh && i < sh->ntex; i++) { + const view_t *v = NULL; + int b = fb_index(sh->tex[i].separate ? NULL : handle_image(S.tex[st][sh->tex[i].slot], &v)); + if (b >= 0) g_fb |= 1u << b; + } + } +} +static int fb_sampled(const vkb_image *im) { int b = fb_index(im); return b >= 0 && ((g_fb >> b) & 1); } + +// ---------------- buffers ---------------- +static VkBuffer g_zero; // 64 KiB of zeros for unbound buffers +static __thread const char *g_akey; // the shader whose bindings are written (audit) +static VkBuffer g_support; +static void fixed_buffers(void) { + if (g_zero) return; + // Ryujinx SupportBuffer: alpha test off, no BGRA swaps, render scale 1 (73 entries from byte 192), 1392 bytes + for (int k = 0; k < 2; k++) { + VkBuffer *b = k ? &g_support : &g_zero; + VkBufferCreateInfo bi = {VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .size = 65536, + .usage = VK_BUFFER_USAGE_UNIFORM_BUFFER_BIT | VK_BUFFER_USAGE_STORAGE_BUFFER_BIT | VK_BUFFER_USAGE_VERTEX_BUFFER_BIT | + VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT}; + vk.vkCreateBuffer(vkc.device, &bi, NULL, b); + VkMemoryRequirements req; + vk.vkGetBufferMemoryRequirements(vkc.device, *b, &req); + VkMemoryAllocateFlagsInfo fl = {VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_FLAGS_INFO, .flags = VK_MEMORY_ALLOCATE_DEVICE_ADDRESS_BIT}; + VkMemoryAllocateInfo ai = {VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO, &fl, .allocationSize = req.size, + .memoryTypeIndex = memory_type(req.memoryTypeBits, VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT)}; + VkDeviceMemory m; + vk.vkAllocateMemory(vkc.device, &ai, NULL, &m); + vk.vkBindBufferMemory(vkc.device, *b, m, 0); + uint8_t *p; + vk.vkMapMemory(vkc.device, m, 0, VK_WHOLE_SIZE, 0, (void **)&p); + memset(p, 0, 65536); + if (k) for (int i = 0; i < 73; i++) { float one = 1.0f; memcpy(p + 192 + 16 * i, &one, 4); } + } +} + +// ---------------- device-local mirrors of written storage buffers ---------------- +// The game's memory pools are host memory imported into Vulkan (zero copy), so a shader that writes a storage buffer +// writes across PCIe; atomics to one address are serialised there (TOTK's sun-shadow statistics: 114 ms of GPU time for +// one 400x224 fullscreen draw, 2026-10-23). A written range gets a copy in device memory for the rest of the submission: +// copied in when a shader first writes it, used by every later GPU access to the range in that submission (storage and +// uniform buffers, vertex and index buffers, copies, updates), copied back once at its end. Before, each draw had a copy +// of its own, in and out: on the Great Sky Island ~360 draws a frame write one 128 KiB tile buffer, ~90 MB of copies a +// frame; the mirror memory ran out mid-frame and the rest wrote host memory (a 400x224 atomic pass took 59 ms: 6 FPS, +// 2026-10-24). An access that straddles a mirror's bounds sends the mirror back first (vkb_mirrors_evict). +// [temporary] The copy back writes the whole range: CPU writes to it while the submission is in flight are overwritten. +// Device-local pools with explicit CPU sync (NVN's Flush / InvalidateMappedRange) replace this (planned). +// The table is sorted by address; mirrors never overlap (a range that straddles one sends it back before another is made). +enum { MIRROR_POOL = 32u << 20, MIRROR_MAX = 4u << 20, MIRRORS = 1024 }; +typedef struct { uint64_t addr, size; VkBuffer host, dev; VkDeviceSize hoff, doff; } mirror_t; +static mirror_t g_mir[MIRRORS]; +static int g_nmir; +static uint64_t g_mirrors_made, g_mirrors_evicted; + +static void barrier(VkCommandBuffer cb, VkPipelineStageFlags2 src, VkAccessFlags2 sa, VkPipelineStageFlags2 dst, VkAccessFlags2 da) { + VkMemoryBarrier2 mb = {VK_STRUCTURE_TYPE_MEMORY_BARRIER_2, .srcStageMask = src, .srcAccessMask = sa, .dstStageMask = dst, .dstAccessMask = da}; + VkDependencyInfo dep = {VK_STRUCTURE_TYPE_DEPENDENCY_INFO, .memoryBarrierCount = 1, .pMemoryBarriers = &mb}; + vk.vkCmdPipelineBarrier2(cb, &dep); +} +// Mirror memory of the current submission's slot. The offset keeps the address's position within 256 bytes, so a +// range in the mirror meets the same offset alignment rules as in the imported pool (pools start on page boundaries). +static int mirror_alloc(uint64_t addr, VkDeviceSize n, VkBuffer *buf, VkDeviceSize *off) { + int sl = vkb_slot(); + if (sl < 0) return 0; + if (!g_mpool[sl].buf) { + VkBufferCreateInfo bi = {VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .size = MIRROR_POOL, + .usage = VK_BUFFER_USAGE_STORAGE_BUFFER_BIT | VK_BUFFER_USAGE_UNIFORM_BUFFER_BIT | VK_BUFFER_USAGE_VERTEX_BUFFER_BIT | + VK_BUFFER_USAGE_INDEX_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT | + VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT}; + if (!vk_check(vk.vkCreateBuffer(vkc.device, &bi, NULL, &g_mpool[sl].buf), "mirror buffer")) return 0; + VkMemoryRequirements req; + vk.vkGetBufferMemoryRequirements(vkc.device, g_mpool[sl].buf, &req); + VkMemoryAllocateFlagsInfo fl = {VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_FLAGS_INFO, .flags = VK_MEMORY_ALLOCATE_DEVICE_ADDRESS_BIT}; + VkMemoryAllocateInfo ai = {VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO, &fl, .allocationSize = req.size, + .memoryTypeIndex = memory_type(req.memoryTypeBits, VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT)}; + if (ai.memoryTypeIndex == UINT32_MAX || !vk_check(vk.vkAllocateMemory(vkc.device, &ai, NULL, &g_mpool[sl].mem), "mirror memory")) die("mirror memory"); + vk.vkBindBufferMemory(vkc.device, g_mpool[sl].buf, g_mpool[sl].mem, 0); + } + VkDeviceSize at = ((g_mpool[sl].used + 255) & ~(VkDeviceSize)255) + (addr & 255); + if (at + n > MIRROR_POOL) return 0; + g_mpool[sl].used = at + n; + *buf = g_mpool[sl].buf; + *off = at; + return 1; +} +static int mirror_first(uint64_t addr) { // index of the first mirror that ends after addr + int lo = 0, hi = g_nmir; + while (lo < hi) { int m = (lo + hi) / 2; if (g_mir[m].addr + g_mir[m].size <= addr) lo = m + 1; else hi = m; } + return lo; +} +static const mirror_t *mirror_find(uint64_t addr, uint64_t len) { // the mirror that holds [addr, addr+len) entirely + int k = mirror_first(addr); + return k < g_nmir && addr >= g_mir[k].addr && addr + len <= g_mir[k].addr + g_mir[k].size ? &g_mir[k] : NULL; +} +static int mirrors_straddle(uint64_t addr, uint64_t len) { // a mirror overlaps [addr, addr+len) without holding it + int k = mirror_first(addr); + return k < g_nmir && g_mir[k].addr < addr + len && !(addr >= g_mir[k].addr && addr + len <= g_mir[k].addr + g_mir[k].size); +} +static void mirror_add(const mirror_t *m) { + int k = mirror_first(m->addr); + memmove(&g_mir[k + 1], &g_mir[k], (size_t)(g_nmir - k) * sizeof g_mir[0]); + g_mir[k] = *m; + g_nmir++; +} +int vkb_gpu_addr(uint64_t addr, uint64_t len, VkBuffer *buf, VkDeviceSize *off) { + const mirror_t *m = mirror_find(addr, len); + if (m) { *buf = m->dev; *off = m->doff + (addr - m->addr); return 1; } + return vkb_pool_addr(addr, len, buf, off); // the pool's copy in device memory when current, else game memory +} +static void mirror_back(VkCommandBuffer cb, const mirror_t *m) { + VkBufferCopy r = {m->doff, m->hoff, m->size}; + char at[32]; snprintf(at, sizeof at, "%#llx+%llu", (unsigned long long)m->addr, (unsigned long long)m->size); + vkb_checkpoint("mirror out", at, NULL); + vk.vkCmdCopyBuffer(cb, m->dev, m->host, 1, &r); +} +void vkb_mirrors_evict(uint64_t addr, uint64_t len) { + VkCommandBuffer cb = vkb_cmd(); + int n = 0; + for (int k = mirror_first(addr); k < g_nmir && g_mir[k].addr < addr + len;) { + const mirror_t *m = &g_mir[k]; + if (addr >= m->addr && addr + len <= m->addr + m->size) { k++; continue; } + if (!n++) barrier(cb, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_WRITE_BIT, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_READ_BIT); + mirror_back(cb, m); + memmove(&g_mir[k], &g_mir[k + 1], (size_t)(--g_nmir - k) * sizeof g_mir[0]); + g_mirrors_evicted++; + vkb_audit(A_MIRROR_EVICTED, NULL); + } + if (n) barrier(cb, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT); +} +void vkb_mirrors_submit(VkCommandBuffer cb) { + if (!g_nmir) return; + barrier(cb, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_WRITE_BIT, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_READ_BIT); + for (int k = 0; k < g_nmir; k++) mirror_back(cb, &g_mir[k]); + barrier(cb, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT | VK_PIPELINE_STAGE_2_HOST_BIT, + VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT | VK_ACCESS_2_HOST_READ_BIT); + g_nmir = 0; +} + +// A buffer range for a guest address: the imported pool (offset aligned as the descriptor type needs) or a copy now. +static VkDescriptorBufferInfo guest_range(uint64_t addr, uint64_t size, VkDeviceSize align, int copy_ok) { + if (size > 65536 && align == vkc.props.limits.minUniformBufferOffsetAlignment) size = 65536; + VkBuffer buf; + VkDeviceSize off; + if (addr && size && vkb_ucap_range(addr, size, &buf, &off) && !(off % align)) // contents as of the submit call (stores dropped it) + return (VkDescriptorBufferInfo){buf, off, size}; + if (addr && size && vkb_gpu_addr(addr, size, &buf, &off)) { + if (!(off % align)) return (VkDescriptorBufferInfo){buf, off, size}; + if (copy_ok) { // [temporary] misaligned for this GPU: a copy of the current contents (later GPU-ordered updates missed) + vkb_audit(A_UBO_COPIED, g_akey); + VkBuffer rb; + VkDeviceSize ro; + void *p = vkb_ring_alloc(size, align, &rb, &ro); + memcpy(p, (const void *)(uintptr_t)addr, size); + return (VkDescriptorBufferInfo){rb, ro, size}; + } + } + return (VkDescriptorBufferInfo){g_zero, 0, 65536}; +} + +// The NVN driver bank c0 or c2 of a stage, as the shader reads it (bank 0: storage buffer table, compute uniform buffer +// table; bank 2: combined texture handles at 0x20, images at 0x120). +// Rebuilt only when what it holds changed since the last one of that bank and stage in this submission (four 4 KiB +// banks a draw were built and cleared before: ~96 MB a frame on the Great Sky Island). +typedef struct { uint64_t rec; VkBuffer b; VkDeviceSize off; uint8_t src[1024]; size_t n; } bank_t; +static __thread bank_t (*t_bank)[6]; +static VkDescriptorBufferInfo driver_bank(int bank, int stage) { + if (!t_bank && !(t_bank = calloc(2, sizeof *t_bank))) die("driver banks"); + bank_t (*last)[6] = t_bank; + uint8_t src[1024]; + size_t n = 0; + if (bank == 0) { + memcpy(src, S.ssbo[stage], sizeof S.ssbo[stage]); n = sizeof S.ssbo[stage]; + if (stage == 5) { memcpy(src + n, S.ubo[5], sizeof S.ubo[5]); n += sizeof S.ubo[5]; } + } else { + memcpy(src, S.tex[stage], sizeof S.tex[stage]); n = sizeof S.tex[stage]; + memcpy(src + n, S.img[stage], sizeof S.img[stage]); n += sizeof S.img[stage]; + } + int bi = bank ? 1 : 0; + if (last[bi][stage].rec == g_sub && last[bi][stage].n == n && !memcmp(last[bi][stage].src, src, n) && !no_cache()) + return (VkDescriptorBufferInfo){last[bi][stage].b, last[bi][stage].off, 0x1000}; + VkBuffer b; + VkDeviceSize off; + uint8_t *p = vkb_ring_alloc(0x1000, vkc.props.limits.minUniformBufferOffsetAlignment, &b, &off); + last[bi][stage].rec = g_sub; last[bi][stage].b = b; last[bi][stage].off = off; last[bi][stage].n = n; + memcpy(last[bi][stage].src, src, n); + memset(p, 0, 0x1000); + if (bank == 0) { + int base = stage == 0 ? 0x110 : stage == 1 ? 0x510 : 0x310; + for (int r = 0; r < 16; r++) { + uint32_t n = (uint32_t)S.ssbo[stage][r].size; + memcpy(p + base + 16 * r, &S.ssbo[stage][r].addr, 8); + memcpy(p + base + 16 * r + 8, &n, 4); + } + if (stage == 5) + for (int r = 0; r < 16; r++) { + uint32_t n = (uint32_t)S.ubo[5][r].size; + memcpy(p + 0x210 + 16 * r, &S.ubo[5][r].addr, 8); + memcpy(p + 0x210 + 16 * r + 8, &n, 4); + } + } else { + for (int r = 0; r < 32; r++) memcpy(p + 0x20 + 8 * r, &S.tex[stage][r], 8); + for (int r = 0; r < 8; r++) memcpy(p + 0x120 + 8 * r, &S.img[stage][r], 8); + } + return (VkDescriptorBufferInfo){b, off, 0x1000}; +} + +// ---------------- descriptors ---------------- +typedef struct { + VkWriteDescriptorSet w[128]; + VkDescriptorBufferInfo bi[128]; + VkDescriptorImageInfo ii[128]; + int n; +} writes_t; + +// ---- the recording thread's own tables (heap: only recording threads have them) +#define SETS 8192 +enum { PROGS = 8192 }; +enum { D_VP, D_SC, D_RDISC, D_TOPO, D_RESTART, D_CULL, D_FRONT, D_POLY, D_BIAS_EN, D_BIAS, D_LINEW, D_CLAMP, D_CLIP, D_CLIPNEG, + D_PROVOKE, D_SAMPLES, D_SMASK, D_A2C, D_A2ONE, D_DT, D_DW, D_DCMP, D_DBT, D_ST, D_SOP0, D_SOP1, D_SCM0, D_SCM1, D_SWM0, + D_SWM1, D_SREF0, D_SREF1, D_BLEN, D_BEQ, D_WMASK, D_LOPEN, D_LOP, D_BCONST, D_FBLOOP, D_SHADERS, D_VINPUT, D_VB0, + D_COUNT = D_VB0 + 16 }; +typedef struct { const vkb_shader *sh; uint64_t rec; VkBuffer b; VkDeviceSize off; } c1_t; +typedef struct { + struct { uint64_t rec, h1, h2; VkDescriptorSet set; } sets[2][SETS]; + // descriptor buffers (vkb_desc.c): the same reuse by contents, of offsets in the thread's descriptor memory + struct { uint64_t rec, gen, h1, h2; VkDeviceSize off; } dsets[2][SETS]; + struct { uint64_t serial; const vkb_shader *sh[6]; const char *bad_key; int ok; } progs[PROGS]; + struct { uint64_t rec; uint8_t valid[D_COUNT]; uint16_t len[D_COUNT]; uint8_t v[D_COUNT][2048]; } dc; + c1_t c1[1024]; // the shaders' own constants (c1) in the command buffer's upload memory + writes_t writes; // one draw at a time +} big_t; +static __thread big_t *B; +static void big_init(void) { if (!B && !(B = calloc(1, sizeof *B))) die("recording tables"); } + +static void put_buf(writes_t *W, VkDescriptorSet set, uint32_t binding, VkDescriptorType type, VkDescriptorBufferInfo info) { + W->bi[W->n] = info; + W->w[W->n] = (VkWriteDescriptorSet){VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET, .dstSet = set, .dstBinding = binding, + .descriptorCount = 1, .descriptorType = type, .pBufferInfo = &W->bi[W->n]}; + W->n++; +} +static void put_img(writes_t *W, VkDescriptorSet set, uint32_t binding, VkDescriptorType type, VkDescriptorImageInfo info) { + W->ii[W->n] = info; + W->w[W->n] = (VkWriteDescriptorSet){VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET, .dstSet = set, .dstBinding = binding, + .descriptorCount = 1, .descriptorType = type, .pImageInfo = &W->ii[W->n]}; + W->n++; +} + +// Textures and images a shader samples: uploads and layout changes (outside a pass). 1 = all set. +static void prepare_images(const vkb_shader *sh, int stage) { + VkCommandBuffer cb = vkb_cmd(); + for (int i = 0; i < sh->ntex; i++) { + const view_t *v = NULL; + vkb_image *im = sh->tex[i].separate ? NULL : handle_image(S.tex[stage][sh->tex[i].slot], &v); + if (im && im->refresh_from) vkb_tex_refresh(cb, im); + if (!im || fb_sampled(im)) continue; // a sampled target: the pass sets its layout + vkb_tex_upload(cb, im); + if (!vkb_layout_is(im, VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL)) + vkb_image_use(cb, im, VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_SHADER_READ_BIT); + } + for (int i = 0; i < sh->nimg; i++) { + const view_t *v = NULL; + vkb_image *im = handle_image(S.img[stage][sh->img[i].slot], &v); + if (im && (im->usage & VK_IMAGE_USAGE_STORAGE_BIT)) { + if (!vkb_layout_is(im, VK_IMAGE_LAYOUT_GENERAL)) + vkb_image_use(cb, im, VK_IMAGE_LAYOUT_GENERAL, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_SHADER_READ_BIT | VK_ACCESS_2_SHADER_WRITE_BIT); + vkb_image_written(im); + } + } +} +// Whether prepare_images would record anything (then a pass must end first). Threads recording: uploads go into the +// submission's prologue right here (vkb_tex_upload), so they do not end the pass. +static int images_need_work(const vkb_shader *sh, int stage) { + int mt = vkb_mt_recording(); + for (int i = 0; i < sh->ntex; i++) { + const view_t *v = NULL; + vkb_image *im = sh->tex[i].separate ? NULL : handle_image(S.tex[stage][sh->tex[i].slot], &v); + // render targets have no upload stamp: only their layout matters (this ended the pass at every draw before) + if (!im || fb_sampled(im)) continue; + if (mt) vkb_tex_upload(VK_NULL_HANDLE, im); + if (im->refresh_from || !vkb_layout_is(im, VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL)) return 1; + if (!mt && ((!im->upload_stamp && !im->gpu_written) || vkb_tex_stale(im))) return 1; + } + for (int i = 0; i < sh->nimg; i++) { + const view_t *v = NULL; + vkb_image *im = handle_image(S.img[stage][sh->img[i].slot], &v); + if (im && (im->usage & VK_IMAGE_USAGE_STORAGE_BIT) && !vkb_layout_is(im, VK_IMAGE_LAYOUT_GENERAL)) return 1; + } + return 0; +} + +// The guest range of a shader's storage buffer i (0 size: unknown / unbound). +static void sbuf_range(const vkb_shader *sh, int stage, int i, uint64_t *addr, uint64_t *size) { + *addr = 0; *size = 0; + uint8_t slot = sh->sbuf[i].slot; + if (slot == 0xFF) return; + if (sh->sbuf[i].kind == VKB_SB_TABLE) { *addr = S.ssbo[stage][slot].addr; *size = S.ssbo[stage][slot].size; } + else if (sh->sbuf[i].kind == VKB_SB_UBO) { *addr = S.ubo[stage][slot].addr; *size = S.ubo[stage][slot].size; } + else { // a pointer inside the game's uniform buffer: {address, size} + const p_bind_buf *u = &S.ubo[stage][slot]; + if (u->addr && sh->sbuf[i].off + 12 <= u->size) { + uint32_t n; + vkb_ucap_read(u->addr + sh->sbuf[i].off, addr, 8); + vkb_ucap_read(u->addr + sh->sbuf[i].off + 8, &n, 4); + *size = n; + } + } +} + +// Before a draw / dispatch: accesses that straddle a mirror get game memory back, and the storage buffers the shader +// writes get a mirror for the rest of the submission (the pass ends only when commands are recorded here). +static void mirrors_prepare(const vkb_shader *sh, int stage) { + static int off_env = -1; // TOTK_VK_NO_MIRRORS=1 (development): written buffers stay in host memory + if (off_env < 0) off_env = getenv("TOTK_VK_NO_MIRRORS") != NULL || vkb_shadow_on(); // pool copies take their place + // [temporary] several threads recording: no mirrors (their order across chunks is not tracked); measure on the Deck + int off = off_env || vkb_mt_recording(); + for (int w = 1; w >= 0; w--) // pools in device memory: written ones first (they may end the pass) + for (int i = 0; i < sh->nsbuf; i++) { + uint64_t addr, size; + if (sh->sbuf[i].write != w) continue; + sbuf_range(sh, stage, i, &addr, &size); + vkb_pool_use(addr, size, w); + if (w && addr && size) vkb_ucap_drop(addr, size); // a store: copied uniforms there are stale from now on + } + for (int slot = 3; slot < VKB_NCBUF; slot++) + if (sh->cbuf_mask & (1u << slot)) vkb_pool_use(S.ubo[stage][slot - 3].addr, S.ubo[stage][slot - 3].size > 65536 ? 65536 : S.ubo[stage][slot - 3].size, 0); + for (int i = 0; i < sh->nsbuf; i++) { + uint64_t addr, size; + sbuf_range(sh, stage, i, &addr, &size); + if (addr && size && mirrors_straddle(addr, size)) { vkb_pass_end_r(PE_MIRROR); vkb_mirrors_evict(addr, size); } + } + for (int slot = 3; slot < VKB_NCBUF; slot++) { + const p_bind_buf *u = &S.ubo[stage][slot - 3]; + uint64_t size = u->size > 65536 ? 65536 : u->size; + if ((sh->cbuf_mask & (1u << slot)) && u->addr && size && mirrors_straddle(u->addr, size)) { vkb_pass_end_r(PE_MIRROR); vkb_mirrors_evict(u->addr, size); } + } + int copied = 0; + for (int i = 0; i < sh->nsbuf && !off; i++) { + if (!sh->sbuf[i].write) continue; + uint64_t addr, size; + sbuf_range(sh, stage, i, &addr, &size); + if (!addr || !size || mirror_find(addr, size)) continue; + int overlap = 0; // two copies of one range would diverge: a range another of the shader's buffers reaches stays put + for (int j = 0; j < sh->nsbuf && !overlap; j++) { + uint64_t a2, s2; + if (j == i) continue; + sbuf_range(sh, stage, j, &a2, &s2); + overlap = s2 && a2 < addr + size && addr < a2 + s2; + } + void *hb; + uint64_t hoff; + mirror_t m = {addr, size}; + const char *why = size > MIRROR_MAX ? "larger than a mirror" : overlap ? "overlaps another buffer of the shader" + : g_nmir >= MIRRORS ? "too many mirrors" : !mtl_addr(addr, size, &hb, &hoff) ? "not in an imported pool" + : !mirror_alloc(addr, size, &m.dev, &m.doff) ? "mirror memory full" : NULL; + if (why) { // the shader writes host memory across PCIe (atomics there are serialised: slow) + vkb_audit(A_SSBO_HOST_WRITE, sh->key); + static int said; + if (said++ < 16) VLOG("[temporary] shader %.12s: written storage buffer %#llx+%llu stays in host memory: %s", sh->key ? sh->key : "?", + (unsigned long long)addr, (unsigned long long)size, why); + continue; + } + m.host = (VkBuffer)hb; m.hoff = hoff; + VkCommandBuffer cb = vkb_cmd(); + if (!copied++) { + vkb_pass_end_r(PE_MIRROR); + barrier(cb, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_WRITE_BIT, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_READ_BIT); + } + VkBufferCopy r = {m.hoff, m.doff, m.size}; + char at[32]; snprintf(at, sizeof at, "%#llx+%llu", (unsigned long long)addr, (unsigned long long)size); + vkb_checkpoint("mirror in", at, sh->key); + vk.vkCmdCopyBuffer(cb, m.host, m.dev, 1, &r); + mirror_add(&m); + g_mirrors_made++; + } + if (copied) barrier(vkb_cmd(), VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT); +} +// Vertex and index buffers of a draw that straddle a mirror (outside a pass when it records). +static void mirrors_prepare_fixed(int indexed, const p_draw_idx *di) { + for (uint32_t s = 0; s < 16; s++) vkb_pool_use(S.vbuf[s].address, S.vbuf[s].size, 0); + if (indexed && di->ibuf) vkb_pool_use(di->ibuf, (uint64_t)di->count * (di->itype == 0 ? 1 : di->itype == 1 ? 2 : 4), 0); + for (uint32_t s = 0; s < 16; s++) + if (S.vbuf[s].address && S.vbuf[s].size && mirrors_straddle(S.vbuf[s].address, S.vbuf[s].size)) { + vkb_pass_end_r(PE_MIRROR); vkb_mirrors_evict(S.vbuf[s].address, S.vbuf[s].size); + } + if (indexed && di->ibuf) { + uint64_t n = (uint64_t)di->count * (di->itype == 0 ? 1 : di->itype == 1 ? 2 : 4); + if (mirrors_straddle(di->ibuf, n)) { vkb_pass_end_r(PE_MIRROR); vkb_mirrors_evict(di->ibuf, n); } + } +} + +static void write_stage(writes_t *W, VkDescriptorSet set, const vkb_shader *sh, int stage) { + uint32_t base = stage == 1 ? VKB_STAGE_BINDINGS : 0; + VkDeviceSize ua = vkc.props.limits.minUniformBufferOffsetAlignment, sa = vkc.props.limits.minStorageBufferOffsetAlignment; + g_akey = sh->key; + put_buf(W, set, base + VKB_B_SUPPORT, VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER, (VkDescriptorBufferInfo){g_support, 0, 1392}); + for (int slot = 0; slot < VKB_NCBUF; slot++) { + if (!(sh->cbuf_mask & (1u << slot))) continue; + VkDescriptorBufferInfo info; + if (slot == 1) { // the shader's own constants: one copy per command buffer (they never change) + uint32_t n = sh->c1_size ? (sh->c1_size + 15) & ~15u : 16; + c1_t *c = &B->c1[((uintptr_t)sh * 0x9E3779B97F4A7C15ull) >> 54]; + if (c->sh != sh || c->rec != g_sub || no_cache()) { + uint8_t *p = vkb_ring_alloc(n, ua, &c->b, &c->off); + memset(p, 0, n); + if (sh->c1_size) memcpy(p, sh->c1, sh->c1_size); + c->sh = sh; c->rec = g_sub; + } + info = (VkDescriptorBufferInfo){c->b, c->off, n}; + } else if (slot == 0 || slot == 2) info = driver_bank(slot, stage); + else { + const p_bind_buf *u = &S.ubo[stage][slot - 3]; + info = guest_range(u->addr, u->size, ua, 1); + if (info.buffer == g_zero && u->addr) { + vkb_audit(A_UBO_MISSING, sh->key); + static int said; + if (said++ < 16) VLOG("[temporary] shader %.12s: uniform buffer c%d %#llx+%llu not in an imported pool: zeros are bound", + sh->key ? sh->key : "?", slot, (unsigned long long)u->addr, (unsigned long long)u->size); + } + } + put_buf(W, set, base + VKB_B_CBUF + (uint32_t)slot, VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER, info); + } + for (int i = 0; i < sh->nsbuf; i++) { + uint64_t addr, size; + sbuf_range(sh, stage, i, &addr, &size); + VkDescriptorBufferInfo bi = guest_range(addr, size, sa, 0); + if (sh->sbuf[i].slot == 0xFF) vkb_audit(A_SSBO_UNKNOWN_SOURCE, sh->key); + else if (bi.buffer == g_zero) vkb_audit(A_SSBO_MISSING, sh->key); + if (bi.buffer == g_zero && addr) { + static int said; + if (said++ < 16) VLOG("[temporary] shader %.12s: storage buffer %#llx+%llu not in an imported pool (or misaligned): an empty one is bound", + sh->key ? sh->key : "?", (unsigned long long)addr, (unsigned long long)size); + } + put_buf(W, set, base + VKB_B_SBUF + (uint32_t)i, VK_DESCRIPTOR_TYPE_STORAGE_BUFFER, bi); + } + for (int i = 0; i < sh->ntex; i++) { + uint64_t h = sh->tex[i].separate ? 0 : S.tex[stage][sh->tex[i].slot]; + const view_t *v = NULL; + vkb_image *im = h ? handle_image(h, &v) : NULL; + VkImageViewType vt = view_type_of(sh->tex[i].type); + int readable = im && (fb_sampled(im) ? vkb_layout_is(im, VK_IMAGE_LAYOUT_ATTACHMENT_FEEDBACK_LOOP_OPTIMAL_EXT) + : vkb_layout_is(im, VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL)); + VkImageView view = readable ? vkb_view(im, v, 0, (int)vt) : VK_NULL_HANDLE; + if (sh->tex[i].separate) vkb_audit(A_TEX_SEPARATE, sh->key); + else if (!(h & 0xFFFFF)) vkb_audit(A_TEX_UNBOUND, sh->key); + else if (!im) vkb_audit(A_TEX_NO_IMAGE, sh->key); + else if (!view) vkb_audit(A_TEX_NOT_READY, sh->key); + else if (im->not_uploadable) vkb_audit(A_TEX_NOT_UPLOADED, sh->key); + VkDescriptorImageInfo info = {sampler((int)(h >> 20) & 0xFFF), view ? view : dummy_view(vt), + view ? vkb_layout(im) : VK_IMAGE_LAYOUT_GENERAL}; + put_img(W, set, base + VKB_B_TEX + (uint32_t)i, VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER, info); + } + for (int i = 0; i < sh->nimg; i++) { + const view_t *v = NULL; + vkb_image *im = handle_image(S.img[stage][sh->img[i].slot], &v); + VkImageView view = im && vkb_layout_is(im, VK_IMAGE_LAYOUT_GENERAL) ? vkb_view(im, v, 2, -1) : VK_NULL_HANDLE; + if (!view) vkb_audit(A_IMG_MISSING, sh->key); + VkDescriptorImageInfo info = {VK_NULL_HANDLE, view ? view : dummy_view(VK_IMAGE_VIEW_TYPE_2D), VK_IMAGE_LAYOUT_GENERAL}; + put_img(W, set, base + VKB_B_IMG + (uint32_t)i, VK_DESCRIPTOR_TYPE_STORAGE_IMAGE, info); + } +} + +// A descriptor set with these writes: one made earlier in this submission with the same contents (two 64-bit hashes +// of every binding, buffer range, view, sampler and layout), else a new one. Each draw allocated and wrote a set of its +// own before (SavageWind measured ~45 % of them identical in BotW). +static vkb_cnt g_sets_new, g_sets_reused; +static __thread VkDescriptorSet g_bound[2]; +static __thread uint64_t g_bound_rec[2]; +static __thread struct { uint64_t rec, gen; VkDeviceSize off; } g_dbound[2]; +static void hmix(uint64_t *h1, uint64_t *h2, uint64_t v) { + *h1 = (*h1 ^ v) * 0x100000001b3ull; + *h2 = (*h2 + v) * 0x9e3779b97f4a7c15ull; + *h2 ^= *h2 >> 29; +} +static VkDescriptorSet resolve_set(writes_t *W, int compute) { + uint64_t h1 = 0xcbf29ce484222325ull, h2 = 0x84222325cbf29ce4ull; + for (int i = 0; i < W->n; i++) { + const VkWriteDescriptorSet *w = &W->w[i]; + hmix(&h1, &h2, (uint64_t)w->dstBinding << 32 | (uint32_t)w->descriptorType); + if (w->pBufferInfo) { + hmix(&h1, &h2, (uint64_t)(uintptr_t)w->pBufferInfo->buffer); + hmix(&h1, &h2, w->pBufferInfo->offset); + hmix(&h1, &h2, w->pBufferInfo->range); + } else { + hmix(&h1, &h2, (uint64_t)(uintptr_t)w->pImageInfo->sampler); + hmix(&h1, &h2, (uint64_t)(uintptr_t)w->pImageInfo->imageView); + hmix(&h1, &h2, (uint64_t)w->pImageInfo->imageLayout); + } + } + uint32_t at = (uint32_t)(h1 ^ h2 >> 32) & (SETS - 1); + if (vkb_db_on()) { // an offset in descriptor memory, returned in place of a set + uint64_t gen = vkb_db_gen(); + for (int probe = 0; probe < 8 && !no_cache(); probe++) { + uint32_t k = (at + (uint32_t)probe) & (SETS - 1); + if (B->dsets[compute][k].rec == g_sub && B->dsets[compute][k].gen == gen && B->dsets[compute][k].h1 == h1 && B->dsets[compute][k].h2 == h2) { + vkb_cnt_add(g_sets_reused, 1); + return (VkDescriptorSet)(uintptr_t)(B->dsets[compute][k].off + 1); + } + if (B->dsets[compute][k].rec != g_sub || B->dsets[compute][k].gen != gen) { at = k; break; } + } + VkDeviceSize off = vkb_db_set(vkb_cmd(), vkb_slot(), g_rec, W->w, W->n, compute); + B->dsets[compute][at].rec = g_sub; B->dsets[compute][at].gen = vkb_db_gen(); B->dsets[compute][at].h1 = h1; B->dsets[compute][at].h2 = h2; + B->dsets[compute][at].off = off; + vkb_cnt_add(g_sets_new, 1); + return (VkDescriptorSet)(uintptr_t)(off + 1); + } + for (int probe = 0; probe < 8 && !no_cache(); probe++) { + uint32_t k = (at + (uint32_t)probe) & (SETS - 1); + if (B->sets[compute][k].rec == g_sub && B->sets[compute][k].h1 == h1 && B->sets[compute][k].h2 == h2) { vkb_cnt_add(g_sets_reused, 1); return B->sets[compute][k].set; } + if (B->sets[compute][k].rec != g_sub) { at = k; break; } + } + VkDescriptorSet set = alloc_set(compute); + for (int i = 0; i < W->n; i++) W->w[i].dstSet = set; + vk.vkUpdateDescriptorSets(vkc.device, (uint32_t)W->n, W->w, 0, NULL); + B->sets[compute][at].rec = g_sub; B->sets[compute][at].h1 = h1; B->sets[compute][at].h2 = h2; B->sets[compute][at].set = set; + vkb_cnt_add(g_sets_new, 1); + return set; +} +static void bind_set(VkCommandBuffer cb, VkDescriptorSet set, int compute) { + if (vkb_db_on()) { // `set` is offset + 1 in descriptor memory + VkDeviceSize off = (VkDeviceSize)(uintptr_t)set - 1; + uint64_t gen = vkb_db_gen(); + if (g_dbound[compute].rec == g_rec && g_dbound[compute].gen == gen && g_dbound[compute].off == off && !no_cache()) return; + g_dbound[compute].rec = g_rec; g_dbound[compute].gen = gen; g_dbound[compute].off = off; + vkb_db_bind(cb, vkb_slot(), g_rec); // an offset made by an earlier command buffer of this thread: its memory bound here too + uint32_t idx = 0; + vk.vkCmdSetDescriptorBufferOffsetsEXT(cb, compute ? VK_PIPELINE_BIND_POINT_COMPUTE : VK_PIPELINE_BIND_POINT_GRAPHICS, + vkb_pipeline_layout(compute), 0, 1, &idx, &off); + return; + } + if (g_bound_rec[compute] == g_rec && g_bound[compute] == set && !no_cache()) return; + g_bound_rec[compute] = g_rec; g_bound[compute] = set; + vk.vkCmdBindDescriptorSets(cb, compute ? VK_PIPELINE_BIND_POINT_COMPUTE : VK_PIPELINE_BIND_POINT_GRAPHICS, vkb_pipeline_layout(compute), 0, 1, &set, 0, NULL); +} + +// The program's shaders, resolved once per program serial. +// Programs resolved before, by serial (a new serial with every SetShaders): resolving hashes the shaders' code +// (vkb_shader_get, SHA-256 of tens of KiB), and TOTK switches programs hundreds of times a frame - with only the +// last one remembered, hashing took ~20% of the recording thread (Great Sky Island, 2026-10-24). +static int resolve_program(void) { + if (!S.prog) return 0; + if (S.prog_serial == S.prog->serial && S.prog_serial) return S.prog_ok; + S.prog_serial = S.prog->serial; + uint32_t at = (uint32_t)(S.prog_serial * 0x9E3779B97F4A7C15ull >> 51) & (PROGS - 1); + if (S.prog_serial && B->progs[at].serial == S.prog_serial) { + memcpy(S.sh, B->progs[at].sh, sizeof S.sh); + S.bad_key = B->progs[at].bad_key; + return S.prog_ok = B->progs[at].ok; + } + S.bad_key = NULL; + memset(S.sh, 0, sizeof S.sh); + S.prog_ok = 1; + for (unsigned i = 0; i < S.prog->nstage && i < 6; i++) { + const vkb_shader *sh = vkb_shader_get(&S.prog->sd[i]); + if (!sh || sh->unsupported) { S.prog_ok = 0; S.bad_key = sh ? sh->key : NULL; continue; } + S.sh[sh->stage] = sh; + } + B->progs[at].serial = S.prog_serial; + memcpy(B->progs[at].sh, S.sh, sizeof S.sh); + B->progs[at].bad_key = S.bad_key; + B->progs[at].ok = S.prog_ok; + return S.prog_ok; +} + +// ---------------- dynamic state ---------------- +static VkBlendFactor blend_factor(uint32_t f) { + switch (f) { + case 1: return VK_BLEND_FACTOR_ZERO; case 2: return VK_BLEND_FACTOR_ONE; + case 3: return VK_BLEND_FACTOR_SRC_COLOR; case 4: return VK_BLEND_FACTOR_ONE_MINUS_SRC_COLOR; + case 5: return VK_BLEND_FACTOR_SRC_ALPHA; case 6: return VK_BLEND_FACTOR_ONE_MINUS_SRC_ALPHA; + case 7: return VK_BLEND_FACTOR_DST_ALPHA; case 8: return VK_BLEND_FACTOR_ONE_MINUS_DST_ALPHA; + case 9: return VK_BLEND_FACTOR_DST_COLOR; case 0xA: return VK_BLEND_FACTOR_ONE_MINUS_DST_COLOR; + case 0xB: return VK_BLEND_FACTOR_SRC_ALPHA_SATURATE; + case 0x10: return VK_BLEND_FACTOR_SRC1_COLOR; case 0x11: return VK_BLEND_FACTOR_ONE_MINUS_SRC1_COLOR; + case 0x12: return VK_BLEND_FACTOR_SRC1_ALPHA; case 0x13: return VK_BLEND_FACTOR_ONE_MINUS_SRC1_ALPHA; + case 0x61: return VK_BLEND_FACTOR_CONSTANT_COLOR; case 0x62: return VK_BLEND_FACTOR_ONE_MINUS_CONSTANT_COLOR; + case 0x63: return VK_BLEND_FACTOR_CONSTANT_ALPHA; case 0x64: return VK_BLEND_FACTOR_ONE_MINUS_CONSTANT_ALPHA; + default: return VK_BLEND_FACTOR_ONE; + } +} +static VkBlendOp blend_op(uint32_t e) { + switch (e) { + case 2: return VK_BLEND_OP_SUBTRACT; case 3: return VK_BLEND_OP_REVERSE_SUBTRACT; + case 4: return VK_BLEND_OP_MIN; case 5: return VK_BLEND_OP_MAX; + default: return VK_BLEND_OP_ADD; + } +} +static VkCompareOp compare(uint32_t f, VkCompareOp dflt) { return f >= 1 && f <= 8 ? (VkCompareOp)(f - 1) : dflt; } +static VkStencilOp stencil_op(uint32_t o) { return o >= 1 && o <= 8 ? (VkStencilOp)(o - 1) : VK_STENCIL_OP_KEEP; } + +// ---------------- per-submission caches of recorded state ---------------- +// Dynamic state, bound shaders, vertex input and vertex buffers stay set in a command buffer until set again (render +// passes do not reset them): a value equal to the one recorded last in this submission is not recorded again. Every draw +// used to record ~45 state commands, the shaders and the vertex input; on the Great Sky Island (~6000 draws a frame) the +// driver's share of recording was ~60 % of the replay thread (2026-10-24). +static vkb_cnt g_dc_hits, g_dc_calls, g_local_barriers; +static int dyn(int k, const void *v, size_t n) { // 1: record it (changed since the last time in this command buffer) + if (B->dc.rec != g_rec) { memset(B->dc.valid, 0, sizeof B->dc.valid); B->dc.rec = g_rec; } + vkb_cnt_add(g_dc_calls, 1); + if (n > sizeof B->dc.v[0] || no_cache()) return 1; + if (B->dc.valid[k] && B->dc.len[k] == n && !memcmp(B->dc.v[k], v, n)) { vkb_cnt_add(g_dc_hits, 1); return 0; } + memcpy(B->dc.v[k], v, n); + B->dc.len[k] = (uint16_t)n; + B->dc.valid[k] = 1; + return 1; +} +#define DYN(k, T, val, call) do { T v_ = (val); if (dyn(k, &v_, sizeof v_)) call; } while (0) + +static void set_dynamic_state(VkCommandBuffer cb, int ncolor, VkExtent2D ext, VkPrimitiveTopology topo) { + // viewports (TOTK uses the upper-left window origin, as Vulkan) and the depth range + VkViewport vps[16]; + int nvp = S.nvp ? S.nvp : 1; + for (int i = 0; i < nvp; i++) { + const float *v = S.vp[i]; + vps[i] = S.nvp ? (VkViewport){v[0], v[1], v[2], v[3], v[4], v[5]} : (VkViewport){0, 0, (float)ext.width, (float)ext.height, 0, 1}; + if (S.has_dr) { vps[i].minDepth = S.dr[0]; vps[i].maxDepth = S.dr[1]; } + if (vps[i].width <= 0) vps[i].width = 1; + if (vps[i].height == 0) vps[i].height = 1; + // The translator writes clip space with y up (GL / Metal, MacNativeGames' target); Vulkan's points down. + // A negative-height viewport flips it back (core since Vulkan 1.1), so the shaders stay as translated. + vps[i].y += vps[i].height; + vps[i].height = -vps[i].height; + } + if (dyn(D_VP, vps, sizeof vps[0] * (size_t)nvp)) vk.vkCmdSetViewportWithCount(cb, (uint32_t)nvp, vps); + VkRect2D sc[16]; + memset(sc, 0, sizeof sc); + for (int i = 0; i < nvp; i++) { + sc[i] = S.has_sc ? (VkRect2D){{S.sc.x < 0 ? 0 : S.sc.x, S.sc.y < 0 ? 0 : S.sc.y}, {(uint32_t)(S.sc.w < 0 ? 0 : S.sc.w), (uint32_t)(S.sc.h < 0 ? 0 : S.sc.h)}} + : (VkRect2D){{0, 0}, ext}; + } + if (dyn(D_SC, sc, sizeof sc[0] * (size_t)nvp)) vk.vkCmdSetScissorWithCount(cb, (uint32_t)nvp, sc); + DYN(D_RDISC, VkBool32, S.raster_discard != 0, vk.vkCmdSetRasterizerDiscardEnable(cb, v_)); + DYN(D_TOPO, VkPrimitiveTopology, topo, vk.vkCmdSetPrimitiveTopology(cb, v_)); + DYN(D_RESTART, VkBool32, VK_FALSE, vk.vkCmdSetPrimitiveRestartEnable(cb, v_)); // [temporary] NVN primitive restart state is not recorded yet + // polygon state + uint32_t cull = S.has_poly ? S.poly.cull : 0, front = S.has_poly ? S.poly.front : 1, mode = S.has_poly ? S.poly.mode : 2; + DYN(D_CULL, VkCullModeFlags, (VkCullModeFlags)cull, vk.vkCmdSetCullMode(cb, v_)); + DYN(D_FRONT, VkFrontFace, front ? VK_FRONT_FACE_COUNTER_CLOCKWISE : VK_FRONT_FACE_CLOCKWISE, vk.vkCmdSetFrontFace(cb, v_)); + DYN(D_POLY, VkPolygonMode, mode == 0 ? VK_POLYGON_MODE_POINT : mode == 1 ? VK_POLYGON_MODE_LINE : VK_POLYGON_MODE_FILL, vk.vkCmdSetPolygonModeEXT(cb, v_)); + int bias = S.has_poly && (S.poly.offset_en & 4); + DYN(D_BIAS_EN, VkBool32, (VkBool32)bias, vk.vkCmdSetDepthBiasEnable(cb, v_)); + { float b[3] = {S.po.units, S.po.clamp, S.po.factor}; if (dyn(D_BIAS, b, sizeof b)) vk.vkCmdSetDepthBias(cb, b[0], b[1], b[2]); } + DYN(D_LINEW, float, 1.0f, vk.vkCmdSetLineWidth(cb, v_)); + DYN(D_CLAMP, VkBool32, S.depth_clamp != 0, vk.vkCmdSetDepthClampEnableEXT(cb, v_)); + DYN(D_CLIP, VkBool32, !S.depth_clamp, vk.vkCmdSetDepthClipEnableEXT(cb, v_)); + DYN(D_CLIPNEG, VkBool32, VK_FALSE, vk.vkCmdSetDepthClipNegativeOneToOneEXT(cb, v_)); // the translator converts z = (z + w) / 2 + DYN(D_PROVOKE, VkProvokingVertexModeEXT, VK_PROVOKING_VERTEX_MODE_FIRST_VERTEX_EXT, vk.vkCmdSetProvokingVertexModeEXT(cb, v_)); + // multisample + DYN(D_SAMPLES, VkSampleCountFlagBits, VK_SAMPLE_COUNT_1_BIT, vk.vkCmdSetRasterizationSamplesEXT(cb, v_)); // [temporary] single-sampled targets + DYN(D_SMASK, VkSampleMask, 0xFFFFFFFF, vk.vkCmdSetSampleMaskEXT(cb, VK_SAMPLE_COUNT_1_BIT, &v_)); + DYN(D_A2C, VkBool32, S.has_ms && S.ms.a2c, vk.vkCmdSetAlphaToCoverageEnableEXT(cb, v_)); + DYN(D_A2ONE, VkBool32, VK_FALSE, vk.vkCmdSetAlphaToOneEnableEXT(cb, v_)); + // depth / stencil + const ds_st *d = &S.ds; + DYN(D_DT, VkBool32, S.has_ds && d->dt, vk.vkCmdSetDepthTestEnable(cb, v_)); + DYN(D_DW, VkBool32, S.has_ds && d->dw, vk.vkCmdSetDepthWriteEnable(cb, v_)); + VkCompareOp dcmp = compare(d->df, VK_COMPARE_OP_LESS); + { // TOTK_VK_DEPTH_ALWAYS=,...: those draws pass the depth test (development experiment) + static const char *list; + static int init; + if (!init) { init = 1; list = getenv("TOTK_VK_DEPTH_ALWAYS"); } + for (const char *p = list; p && *p && S.sh[1] && S.sh[1]->key;) { + size_t n = strcspn(p, ","); + if (n && !strncmp(S.sh[1]->key, p, n)) dcmp = VK_COMPARE_OP_ALWAYS; + p += n + (p[n] == ','); + } + } + DYN(D_DCMP, VkCompareOp, dcmp, vk.vkCmdSetDepthCompareOp(cb, v_)); + DYN(D_DBT, VkBool32, VK_FALSE, vk.vkCmdSetDepthBoundsTestEnable(cb, v_)); + DYN(D_ST, VkBool32, S.has_ds && d->st, vk.vkCmdSetStencilTestEnable(cb, v_)); + for (int f = 0; f < 2; f++) { + VkStencilFaceFlags face = f ? VK_STENCIL_FACE_BACK_BIT : VK_STENCIL_FACE_FRONT_BIT; + int op[4] = {f ? d->fail_b : d->fail_f, f ? d->zpass_b : d->zpass_f, f ? d->zfail_b : d->zfail_f, f ? d->sf_b : d->sf_f}; + if (dyn(D_SOP0 + f, op, sizeof op)) + vk.vkCmdSetStencilOp(cb, face, stencil_op(op[0]), stencil_op(op[1]), stencil_op(op[2]), compare(op[3], VK_COMPARE_OP_ALWAYS)); + DYN(D_SCM0 + f, uint32_t, S.stencil[f].value_mask, vk.vkCmdSetStencilCompareMask(cb, face, v_)); + DYN(D_SWM0 + f, uint32_t, S.stencil[f].write_mask, vk.vkCmdSetStencilWriteMask(cb, face, v_)); + DYN(D_SREF0 + f, uint32_t, S.stencil[f].ref, vk.vkCmdSetStencilReference(cb, face, v_)); + } + // colour: blend per target, write masks, logic op + if (ncolor > 0) { + VkBool32 en[8]; + VkColorBlendEquationEXT eq[8]; + VkColorComponentFlags wm[8]; + memset(eq, 0, sizeof eq); + for (int i = 0; i < ncolor; i++) { + en[i] = S.has_color && ((S.color.blend_en >> i) & 1); + const blend_st *b = &S.blend[i]; + eq[i] = S.has_blend[i] + ? (VkColorBlendEquationEXT){blend_factor(b->src_rgb), blend_factor(b->dst_rgb), blend_op(b->eq_rgb), + blend_factor(b->src_a), blend_factor(b->dst_a), blend_op(b->eq_a)} + : (VkColorBlendEquationEXT){VK_BLEND_FACTOR_ONE, VK_BLEND_FACTOR_ZERO, VK_BLEND_OP_ADD, VK_BLEND_FACTOR_ONE, VK_BLEND_FACTOR_ZERO, VK_BLEND_OP_ADD}; + wm[i] = (S.chmask.mask >> (4 * i)) & 0xF; + } + if (dyn(D_BLEN, en, sizeof en[0] * (size_t)ncolor)) vk.vkCmdSetColorBlendEnableEXT(cb, 0, (uint32_t)ncolor, en); + if (dyn(D_BEQ, eq, sizeof eq[0] * (size_t)ncolor)) vk.vkCmdSetColorBlendEquationEXT(cb, 0, (uint32_t)ncolor, eq); + if (dyn(D_WMASK, wm, sizeof wm[0] * (size_t)ncolor)) vk.vkCmdSetColorWriteMaskEXT(cb, 0, (uint32_t)ncolor, wm); + } + uint32_t lop = S.has_color ? S.color.logic_op : 3; + DYN(D_LOPEN, VkBool32, lop != 3, vk.vkCmdSetLogicOpEnableEXT(cb, v_)); + DYN(D_LOP, VkLogicOp, (VkLogicOp)lop, vk.vkCmdSetLogicOpEXT(cb, v_)); + if (dyn(D_BCONST, S.blend_color, sizeof S.blend_color)) vk.vkCmdSetBlendConstants(cb, S.blend_color); +} + +// NVN vertex attribute formats (NVNformat values) as vertex input formats. +static VkFormat vertex_format(int f) { + switch (f) { + case 0x19: return VK_FORMAT_R8G8B8_UNORM; case 0x1A: return VK_FORMAT_R8G8B8_SNORM; + case 0x1B: return VK_FORMAT_R8G8B8_UINT; case 0x1C: return VK_FORMAT_R8G8B8_SINT; + case 0x1D: return VK_FORMAT_R16G16B16_SFLOAT; case 0x1E: return VK_FORMAT_R16G16B16_UNORM; + case 0x1F: return VK_FORMAT_R16G16B16_SNORM; case 0x20: return VK_FORMAT_R16G16B16_UINT; + case 0x21: return VK_FORMAT_R16G16B16_SINT; case 0x22: return VK_FORMAT_R32G32B32_SFLOAT; + case 0x23: return VK_FORMAT_R32G32B32_UINT; case 0x24: return VK_FORMAT_R32G32B32_SINT; + case 0x51: return VK_FORMAT_R8_USCALED; case 0x52: return VK_FORMAT_R8_SSCALED; + case 0x53: return VK_FORMAT_R16_USCALED; case 0x54: return VK_FORMAT_R16_SSCALED; + case 0x57: return VK_FORMAT_R8G8_USCALED; case 0x58: return VK_FORMAT_R8G8_SSCALED; + case 0x59: return VK_FORMAT_R16G16_USCALED; case 0x5A: return VK_FORMAT_R16G16_SSCALED; + case 0x5D: return VK_FORMAT_R8G8B8_USCALED; case 0x5E: return VK_FORMAT_R8G8B8_SSCALED; + case 0x5F: return VK_FORMAT_R16G16B16_USCALED; case 0x60: return VK_FORMAT_R16G16B16_SSCALED; + case 0x63: return VK_FORMAT_R8G8B8A8_USCALED; case 0x64: return VK_FORMAT_R8G8B8A8_SSCALED; + case 0x65: return VK_FORMAT_R16G16B16A16_USCALED; case 0x66: return VK_FORMAT_R16G16B16A16_SSCALED; + case 0x69: return VK_FORMAT_A2B10G10R10_SNORM_PACK32; case 0x6A: return VK_FORMAT_A2B10G10R10_SINT_PACK32; + case 0x6B: return VK_FORMAT_A2B10G10R10_USCALED_PACK32; case 0x6C: return VK_FORMAT_A2B10G10R10_SSCALED_PACK32; + default: { + VkFormat v = vkb_format(f, NULL, NULL); + return f >= 0x79 ? VK_FORMAT_UNDEFINED : v; // no compressed / ASTC vertex formats + } + } +} + +// Integer vertex formats: 1 unsigned, 2 signed, 0 otherwise. +static int int_kind(VkFormat f) { + switch (f) { + case VK_FORMAT_R8_UINT: case VK_FORMAT_R8G8_UINT: case VK_FORMAT_R8G8B8_UINT: case VK_FORMAT_R8G8B8A8_UINT: + case VK_FORMAT_R16_UINT: case VK_FORMAT_R16G16_UINT: case VK_FORMAT_R16G16B16_UINT: case VK_FORMAT_R16G16B16A16_UINT: + case VK_FORMAT_R32_UINT: case VK_FORMAT_R32G32_UINT: case VK_FORMAT_R32G32B32_UINT: case VK_FORMAT_R32G32B32A32_UINT: + case VK_FORMAT_A2B10G10R10_UINT_PACK32: return 1; + case VK_FORMAT_R8_SINT: case VK_FORMAT_R8G8_SINT: case VK_FORMAT_R8G8B8_SINT: case VK_FORMAT_R8G8B8A8_SINT: + case VK_FORMAT_R16_SINT: case VK_FORMAT_R16G16_SINT: case VK_FORMAT_R16G16B16_SINT: case VK_FORMAT_R16G16B16A16_SINT: + case VK_FORMAT_R32_SINT: case VK_FORMAT_R32G32_SINT: case VK_FORMAT_R32G32B32_SINT: case VK_FORMAT_R32G32B32A32_SINT: + case VK_FORMAT_A2B10G10R10_SINT_PACK32: return 2; + default: return 0; + } +} + +// Locations of the bound attributes in integer formats (signed ones also in *sint). +static uint32_t int_attributes(uint32_t *sint) { + uint32_t m = 0; + *sint = 0; + for (int i = 0; i < S.nvattr; i++) { + va_st a; + memcpy(&a, &S.vattr[i], sizeof a); + int k = a.fmt ? int_kind(vertex_format((int)a.fmt)) : 0; + if (k) m |= 1u << i; + if (k == 2) *sint |= 1u << i; + } + return m; +} + +// Integer attributes the shader variant reads as integers (int_inputs) are given in their own format. Any other +// integer attribute still goes to a float input: as a scaled format ([temporary]: only if no variant could be made), +// 32-bit ones as float bits. +static VkFormat float_read(VkFormat f) { + switch (f) { + case VK_FORMAT_R8_UINT: return VK_FORMAT_R8_USCALED; case VK_FORMAT_R8_SINT: return VK_FORMAT_R8_SSCALED; + case VK_FORMAT_R8G8_UINT: return VK_FORMAT_R8G8_USCALED; case VK_FORMAT_R8G8_SINT: return VK_FORMAT_R8G8_SSCALED; + case VK_FORMAT_R8G8B8_UINT: return VK_FORMAT_R8G8B8_USCALED; case VK_FORMAT_R8G8B8_SINT: return VK_FORMAT_R8G8B8_SSCALED; + case VK_FORMAT_R8G8B8A8_UINT: return VK_FORMAT_R8G8B8A8_USCALED; case VK_FORMAT_R8G8B8A8_SINT: return VK_FORMAT_R8G8B8A8_SSCALED; + case VK_FORMAT_R16_UINT: return VK_FORMAT_R16_USCALED; case VK_FORMAT_R16_SINT: return VK_FORMAT_R16_SSCALED; + case VK_FORMAT_R16G16_UINT: return VK_FORMAT_R16G16_USCALED; case VK_FORMAT_R16G16_SINT: return VK_FORMAT_R16G16_SSCALED; + case VK_FORMAT_R16G16B16_UINT: return VK_FORMAT_R16G16B16_USCALED; case VK_FORMAT_R16G16B16_SINT: return VK_FORMAT_R16G16B16_SSCALED; + case VK_FORMAT_R16G16B16A16_UINT: return VK_FORMAT_R16G16B16A16_USCALED; case VK_FORMAT_R16G16B16A16_SINT: return VK_FORMAT_R16G16B16A16_SSCALED; + case VK_FORMAT_A2B10G10R10_UINT_PACK32: return VK_FORMAT_A2B10G10R10_USCALED_PACK32; + case VK_FORMAT_A2B10G10R10_SINT_PACK32: return VK_FORMAT_A2B10G10R10_SSCALED_PACK32; + case VK_FORMAT_R32_UINT: case VK_FORMAT_R32_SINT: return VK_FORMAT_R32_SFLOAT; // bits reinterpreted (no 32-bit scaled) + case VK_FORMAT_R32G32_UINT: case VK_FORMAT_R32G32_SINT: return VK_FORMAT_R32G32_SFLOAT; + case VK_FORMAT_R32G32B32_UINT: case VK_FORMAT_R32G32B32_SINT: return VK_FORMAT_R32G32B32_SFLOAT; + case VK_FORMAT_R32G32B32A32_UINT: case VK_FORMAT_R32G32B32A32_SINT: return VK_FORMAT_R32G32B32A32_SFLOAT; + default: return f; + } +} + +static void set_vertex_input(VkCommandBuffer cb, uint32_t int_inputs) { + VkVertexInputBindingDescription2EXT bind[16]; + VkVertexInputAttributeDescription2EXT attr[16]; + memset(bind, 0, sizeof bind); + memset(attr, 0, sizeof attr); + uint32_t nb = 0, na = 0, used = 0; + for (int i = 0; i < S.nvattr; i++) { + va_st a; + memcpy(&a, &S.vattr[i], sizeof a); + VkFormat raw = a.fmt ? vertex_format((int)a.fmt) : VK_FORMAT_UNDEFINED; + VkFormat f = int_inputs >> i & 1 ? raw : float_read(raw); + if (f == VK_FORMAT_UNDEFINED) continue; + { // a vertex format the GPU cannot fetch reads garbage: say so once per format + static uint8_t checked[256]; + if ((unsigned)f < 256 && !checked[f]) { + checked[f] = 1; + VkFormatProperties fp; + vk.vkGetPhysicalDeviceFormatProperties(vkc.gpu, f, &fp); + if (!(fp.bufferFeatures & VK_FORMAT_FEATURE_VERTEX_BUFFER_BIT)) + VLOG("[temporary] vertex format %d (NVN %#x) is not supported for vertex fetch on this GPU", (int)f, (unsigned)a.fmt); + } + } + attr[na++] = (VkVertexInputAttributeDescription2EXT){VK_STRUCTURE_TYPE_VERTEX_INPUT_ATTRIBUTE_DESCRIPTION_2_EXT, .location = (uint32_t)i, + .binding = a.stream, .format = f, .offset = a.offset}; + used |= 1u << a.stream; + } + for (uint32_t s = 0; s < 16; s++) { + if (!(used & (1u << s))) continue; + bind[nb++] = (VkVertexInputBindingDescription2EXT){VK_STRUCTURE_TYPE_VERTEX_INPUT_BINDING_DESCRIPTION_2_EXT, .binding = s, + .stride = S.vstream[s].stride, .inputRate = S.vstream[s].divisor ? VK_VERTEX_INPUT_RATE_INSTANCE : VK_VERTEX_INPUT_RATE_VERTEX, + .divisor = 1}; + VkBuffer buf; + VkDeviceSize off; + VkDeviceSize size = S.vbuf[s].size; + if (S.vbuf[s].address && size && vkb_gpu_addr(S.vbuf[s].address, size, &buf, &off)) { + VkDeviceSize key[3] = {(VkDeviceSize)(uintptr_t)buf, off, size}; + if (dyn(D_VB0 + (int)s, key, sizeof key)) vk.vkCmdBindVertexBuffers2(cb, s, 1, &buf, &off, &size, NULL); + } + else { + if (S.vbuf[s].address) { // [temporary] not in an imported pool (a virtual pool: sparse binding later): zeros + vkb_audit(A_VBUF_MISSING, S.sh[0] ? S.sh[0]->key : NULL); + static int said; + if (said++ < 16) VLOG("[temporary] vertex buffer %#llx+%llu (stream %u) not in an imported pool: zeros are bound", + (unsigned long long)S.vbuf[s].address, (unsigned long long)S.vbuf[s].size, s); + } + VkDeviceSize z = 0, zs = 65536, key[3] = {(VkDeviceSize)(uintptr_t)g_zero, 0, 65536}; + if (dyn(D_VB0 + (int)s, key, sizeof key)) vk.vkCmdBindVertexBuffers2(cb, s, 1, &g_zero, &z, &zs, NULL); + } + } + struct { uint32_t nb, na; VkVertexInputBindingDescription2EXT b[16]; VkVertexInputAttributeDescription2EXT a[16]; } vi; + memset(&vi, 0, sizeof vi); + vi.nb = nb; vi.na = na; + memcpy(vi.b, bind, sizeof bind[0] * nb); + memcpy(vi.a, attr, sizeof attr[0] * na); + if (dyn(D_VINPUT, &vi, sizeof vi)) vk.vkCmdSetVertexInputEXT(cb, nb, bind, na, attr); +} + +static VkPrimitiveTopology topology(int prim, int *ok) { + *ok = 1; + switch (prim) { + case 0: return VK_PRIMITIVE_TOPOLOGY_POINT_LIST; + case 1: return VK_PRIMITIVE_TOPOLOGY_LINE_LIST; + case 3: return VK_PRIMITIVE_TOPOLOGY_LINE_STRIP; + case 4: return VK_PRIMITIVE_TOPOLOGY_TRIANGLE_LIST; + case 5: return VK_PRIMITIVE_TOPOLOGY_TRIANGLE_STRIP; + case 6: case 9: return VK_PRIMITIVE_TOPOLOGY_TRIANGLE_FAN; // POLYGON: a convex fan + case 0xA: return VK_PRIMITIVE_TOPOLOGY_LINE_LIST_WITH_ADJACENCY; + case 0xB: return VK_PRIMITIVE_TOPOLOGY_LINE_STRIP_WITH_ADJACENCY; + case 0xC: return VK_PRIMITIVE_TOPOLOGY_TRIANGLE_LIST_WITH_ADJACENCY; + case 0xD: return VK_PRIMITIVE_TOPOLOGY_TRIANGLE_STRIP_WITH_ADJACENCY; + case 0xE: return VK_PRIMITIVE_TOPOLOGY_PATCH_LIST; + default: *ok = 0; return VK_PRIMITIVE_TOPOLOGY_TRIANGLE_LIST; // LINE_LOOP, QUADS, QUAD_STRIP: [temporary] not drawn + } +} + +// ---------------- draws ---------------- +static vkb_cnt g_draws, g_skipped[4]; // skipped: no program / no shader translation, topology, targets, index buffer +static __thread int g_stores_pending; // a recorded draw wrote storage buffers / images: barrier before the next pass +// Earlier draws wrote storage buffers / images: the next pass sees their writes (backend_vk.c pass_begin asks before any +// pass, a clear's too). TOTK's tiled lighting marks tiles from a fragment shader and the next pass's vertex shader reads +// the marks to drop tiles; without the barrier it read the cleared marks and lit every tile with every variant (the +// overexposed title screen, 2026-10-23). Draws within one pass do not wait for each other's stores (on the console that +// takes the game's own barrier, replayed as a full one): ending the pass after every storing draw ended ~590 passes a +// frame on the Great Sky Island (2026-10-24). +// Within a pass, a draw waits for earlier draws' stores only when it reads what they wrote: TOTK's tile marking draws +// (~360 a frame on the Great Sky Island) write one 128 KiB buffer and do not read it, the lighting draws read it. +enum { WRITTEN = 256 }; +static __thread struct { uint64_t addr, size; } g_wr[WRITTEN]; +static __thread int g_nwr, g_wr_all, g_wr_img; // all: too many ranges (every read waits); img: storage images were written +int vkb_stores_take(void) { int p = g_stores_pending; g_stores_pending = 0; g_nwr = 0; g_wr_all = 0; g_wr_img = 0; return p; } +static void written_add(const vkb_shader *sh, int stage) { + for (int i = 0; i < sh->nsbuf; i++) { + if (!sh->sbuf[i].write) continue; + uint64_t addr, size; + sbuf_range(sh, stage, i, &addr, &size); + if (!addr || !size) { g_wr_all = 1; continue; } + int k = 0; + while (k < g_nwr && !(g_wr[k].addr == addr && g_wr[k].size == size)) k++; + if (k < g_nwr) continue; + if (g_nwr == WRITTEN) g_wr_all = 1; else g_wr[g_nwr].addr = addr, g_wr[g_nwr++].size = size; + } + if (sh->nimg) g_wr_img = 1; +} +static int written_overlaps(uint64_t addr, uint64_t size) { + if (!addr || !size) return 0; + if (g_wr_all) return 1; + for (int k = 0; k < g_nwr; k++) if (addr < g_wr[k].addr + g_wr[k].size && g_wr[k].addr < addr + size) return 1; + return 0; +} +static int reads_written(const vkb_shader *sh, int stage) { // the shader reads what earlier stores wrote + if (g_wr_img && (sh->ntex || sh->nimg)) return 1; + for (int i = 0; i < sh->nsbuf; i++) { + uint64_t addr, size; + sbuf_range(sh, stage, i, &addr, &size); + if (sh->sbuf[i].read && written_overlaps(addr, size)) return 1; + } + for (int slot = 3; slot < VKB_NCBUF; slot++) + if ((sh->cbuf_mask & (1u << slot)) && written_overlaps(S.ubo[stage][slot - 3].addr, S.ubo[stage][slot - 3].size)) return 1; + return 0; +} +uint64_t vkb_draw_count(void) { return vkb_cnt_sum(g_draws); } +int vkb_viewport0(float v[4]) { if (!S.nvp) return 0; memcpy(v, S.vp[0], 4 * sizeof *v); return 1; } // x y w h (trace) +int vkb_scissor(VkRect2D *r) { // the game's scissor (clears are limited to it, as on NVN); 0: none set + if (!S.has_sc) return 0; + int32_t x = S.sc.x < 0 ? 0 : S.sc.x, y = S.sc.y < 0 ? 0 : S.sc.y; + int32_t w = S.sc.w - (x - S.sc.x), h = S.sc.h - (y - S.sc.y); + *r = (VkRect2D){{x, y}, {(uint32_t)(w < 0 ? 0 : w), (uint32_t)(h < 0 ? 0 : h)}}; + return 1; +} +void vkb_draw_stats(uint64_t *draws, uint64_t *skipped) { + *draws = vkb_cnt_sum(g_draws); + uint64_t sk[4] = {vkb_cnt_sum(g_skipped[0]), vkb_cnt_sum(g_skipped[1]), vkb_cnt_sum(g_skipped[2]), vkb_cnt_sum(g_skipped[3])}; + uint64_t dc_calls = vkb_cnt_sum(g_dc_calls), dc_hits = vkb_cnt_sum(g_dc_hits); + uint64_t gets = 0, hits = 0; + if (vkb_db_on()) vkb_db_stats(&gets, &hits); + if (dc_calls) VLOG("recording caches: state %.0f %% not recorded again, descriptor sets %llu made / %llu reused (descriptor bytes: %llu from the driver, %llu cached), mirrors %llu made / %llu sent back early, barriers inside passes %llu", + 100.0 * (double)dc_hits / (double)dc_calls, (unsigned long long)vkb_cnt_sum(g_sets_new), (unsigned long long)vkb_cnt_sum(g_sets_reused), + (unsigned long long)gets, (unsigned long long)hits, (unsigned long long)g_mirrors_made, (unsigned long long)g_mirrors_evicted, + (unsigned long long)vkb_cnt_sum(g_local_barriers)); + *skipped = sk[0] + sk[1] + sk[2] + sk[3]; + static uint64_t said; + if (*skipped && *draws + *skipped >= said * 2) { + said = *draws + *skipped; + VLOG("skipped draws: no program / translation %llu, topology %llu, no targets %llu, index buffer %llu", + (unsigned long long)sk[0], (unsigned long long)sk[1], (unsigned long long)sk[2], (unsigned long long)sk[3]); + } +} + +// TOTK_VK_DRAWTRACE=:: n draws from that present on (development): targets, shaders, counts, outcome. +static int draw_trace(void) { + static long long at = -1, left; + if (at == -1) { + const char *e = getenv("TOTK_VK_DRAWTRACE"); + at = e ? atoll(e) : 0; + left = e && strchr(e, ':') ? atoll(strchr(e, ':') + 1) : 200; + } + return at > 0 && vkb_frame() >= (uint64_t)at && left-- > 0; +} +// ---------------- GPU profile (development) ---------------- +// TOTK_GPU_PROF=: a timestamp before and after every draw and dispatch of that frame; the next present prints +// the GPU time per shader (sorted) and the frame's span. Timestamps inside a pass overlap with neighbouring draws, so +// the per-draw times are approximate; the span is exact. +enum { GPROF_MAX = 8192 }; +typedef struct { char key[13]; uint8_t compute; uint32_t count, inst, w, h; } gprof_info; +static VkQueryPool g_qp; +static int g_qn, g_qdone; +static gprof_info g_qinfo[GPROF_MAX]; +static long long gprof_frame(void) { + static long long at = -1; + if (at == -1) { const char *e = getenv("TOTK_GPU_PROF"); at = e ? atoll(e) : 0; } + return at; +} +static int gprof_begin(void) { + long long at = gprof_frame(); + if (at <= 0 || g_qdone || vkb_frame() != (uint64_t)at || g_qn >= GPROF_MAX) return -1; + if (!g_qp) { + VkQueryPoolCreateInfo ci = {VK_STRUCTURE_TYPE_QUERY_POOL_CREATE_INFO, .queryType = VK_QUERY_TYPE_TIMESTAMP, .queryCount = 2 * GPROF_MAX}; + if (!vk_check(vk.vkCreateQueryPool(vkc.device, &ci, NULL, &g_qp), "query pool")) { g_qdone = 1; return -1; } + vk.vkResetQueryPool(vkc.device, g_qp, 0, 2 * GPROF_MAX); + } + vk.vkCmdWriteTimestamp2(vkb_cmd(), VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, g_qp, (uint32_t)(2 * g_qn)); + return g_qn++; +} +static void gprof_end(int q, const vkb_shader *sh, int compute, uint32_t count, uint32_t inst) { + if (q < 0) return; + vk.vkCmdWriteTimestamp2(vkb_cmd(), VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, g_qp, (uint32_t)(2 * q + 1)); + gprof_info *i = &g_qinfo[q]; + snprintf(i->key, sizeof i->key, "%.12s", sh && sh->key ? sh->key : "?"); + VkExtent2D e = vkb_rt_extent(); + i->compute = (uint8_t)compute; i->count = count; i->inst = inst; i->w = e.width; i->h = e.height; +} +void vkb_gprof_report(void) { + long long at = gprof_frame(); + if (at <= 0 || g_qdone || !g_qn || vkb_frame() <= (uint64_t)at) return; + g_qdone = 1; + mtl_finish(); + uint64_t *ts = malloc(sizeof *ts * 2 * (size_t)g_qn); + if (!ts) return; + vk.vkGetQueryPoolResults(vkc.device, g_qp, 0, (uint32_t)(2 * g_qn), sizeof *ts * 2 * (size_t)g_qn, ts, sizeof *ts, + VK_QUERY_RESULT_64_BIT | VK_QUERY_RESULT_WAIT_BIT); + double ns = vkc.props.limits.timestampPeriod, sum = 0; + typedef struct { char key[13]; double ms; int n; uint8_t compute; uint32_t w, h; uint64_t prims; } agg; + agg *a = calloc((size_t)g_qn, sizeof *a); + int na = 0; + uint64_t lo = UINT64_MAX, hi = 0; + for (int q = 0; q < g_qn; q++) { + uint64_t t0 = ts[2 * q], t1 = ts[2 * q + 1]; + if (t0 < lo) lo = t0; + if (t1 > hi) hi = t1; + double ms = t1 > t0 ? (double)(t1 - t0) * ns / 1e6 : 0; + sum += ms; + int k = 0; + while (k < na && strcmp(a[k].key, g_qinfo[q].key)) k++; + if (k == na) { memcpy(a[na].key, g_qinfo[q].key, 13); a[na].compute = g_qinfo[q].compute; a[na].w = g_qinfo[q].w; a[na].h = g_qinfo[q].h; na++; } + a[k].ms += ms; a[k].n++; a[k].prims += (uint64_t)g_qinfo[q].count * (g_qinfo[q].inst ? g_qinfo[q].inst : 1); + } + for (int i = 0; i < na; i++) // by time, descending + for (int j = i + 1; j < na; j++) if (a[j].ms > a[i].ms) { agg t = a[i]; a[i] = a[j]; a[j] = t; } + VLOG("GPU profile of present %lld: %d draws/dispatches, %.2f ms summed, %.2f ms from first to last", at, g_qn, sum, + hi > lo ? (double)(hi - lo) * ns / 1e6 : 0.0); + for (int i = 0; i < na && i < 30; i++) + VLOG(" %8.3f ms %4d x %s %s target %ux%u vertices*instances %llu", a[i].ms, a[i].n, a[i].compute ? "cs" : "fs", a[i].key, + a[i].w, a[i].h, (unsigned long long)a[i].prims); + free(a); + free(ts); +} + +// What a traced shader reads and writes: textures, images (serial, format, size) and storage buffers (address, size). +static void trace_resources(const vkb_shader *sh, int stage) { + char line[1024]; + int n = snprintf(line, sizeof line, "[draw] %s %.12s:", stage == 0 ? "vs" : stage == 1 ? "fs" : "cs", sh->key ? sh->key : "?"); + for (int i = 0; i < sh->ntex && n < 900; i++) { + const view_t *v = NULL; + vkb_image *im = sh->tex[i].separate ? NULL : handle_image(S.tex[stage][sh->tex[i].slot], &v); + n += im ? snprintf(line + n, sizeof line - n, " t%d=%llu/%#x/%ux%u%s@%p#%04llx", sh->tex[i].slot, (unsigned long long)im->serial, im->d.format, + im->width, im->height, im->gpu_written ? "/gpu" : "", (const void *)im->storage, + (unsigned long long)((uintptr_t)im->img & 0xffff)) + : snprintf(line + n, sizeof line - n, " t%d=-", sh->tex[i].slot); + int sid = (int)(S.tex[stage][sh->tex[i].slot] >> 20) & 0xFFF; + vkb_rw_read(&g_reg_mu, LK_TEXREG); + if (sid < g_nsreg && g_sreg[sid].valid) { + const samp_par *sp = &g_sreg[sid].p; + if (sp->cmp_mode) n += snprintf(line + n, sizeof line - n, "(cmp %u func %u)", sp->cmp_mode, sp->cmp_func); + if (getenv("TOTK_VK_TRACE_SAMPLERS")) // NVN values: wrap 0 clamp .. 7 clamp to edge, filters 0 nearest 1 linear + n += snprintf(line + n, sizeof line - n, "(s%d wrap %u %u %u min %u mag %u border %g %g %g %g)", sid, sp->ws, sp->wt, + sp->wr, sp->minf, sp->magf, sp->border[0], sp->border[1], sp->border[2], sp->border[3]); + } + vkb_rw_read_end(&g_reg_mu); + if (sh->tex[i].type & 0x400) n += snprintf(line + n, sizeof line - n, "(shadow)"); // Ryujinx SamplerType.Shadow + } + for (int i = 0; i < sh->nimg && n < 900; i++) { + const view_t *v = NULL; + vkb_image *im = handle_image(S.img[stage][sh->img[i].slot], &v); + n += im ? snprintf(line + n, sizeof line - n, " i%d=%llu/%#x/%ux%u", sh->img[i].slot, (unsigned long long)im->serial, im->d.format, im->width, im->height) + : snprintf(line + n, sizeof line - n, " i%d=-", sh->img[i].slot); + } + for (int slot = 3; slot < VKB_NCBUF && n < 900; slot++) + if (sh->cbuf_mask & (1u << slot)) + n += snprintf(line + n, sizeof line - n, " c%d=%#llx+%llu", slot, (unsigned long long)S.ubo[stage][slot - 3].addr, + (unsigned long long)S.ubo[stage][slot - 3].size); + for (int i = 0; i < sh->nsbuf && n < 900; i++) { + uint8_t slot = sh->sbuf[i].slot; + const p_bind_buf *b = slot == 0xFF ? NULL : sh->sbuf[i].kind == VKB_SB_TABLE ? &S.ssbo[stage][slot] : &S.ubo[stage][slot]; + n += snprintf(line + n, sizeof line - n, " s%d(%c%d)=%#llx+%llu", i, "tup"[sh->sbuf[i].kind], slot, + b ? (unsigned long long)b->addr : 0ull, b ? (unsigned long long)b->size : 0ull); + } + VLOG("%s", line); + // TOTK_VK_PEEK=::-: those constants of traced draws with that shader + const char *pk = getenv("TOTK_VK_PEEK"); + size_t kl = pk && strchr(pk, ':') ? (size_t)(strchr(pk, ':') - pk) : 0; + if (kl && sh->key && !strncmp(sh->key, pk, kl)) { + int slot = 0, a = 0, b = 0; + if (sscanf(pk + kl, ":%d:%d-%d", &slot, &a, &b) == 3 && slot >= 3 && slot < VKB_NCBUF) { + const p_bind_buf *u = &S.ubo[stage][slot - 3]; + for (int v = a; v <= b && u->addr && (uint64_t)(v + 1) * 16 <= u->size; v++) { + float f[4]; + memcpy(f, (const void *)(uintptr_t)(u->addr + (uint64_t)v * 16), 16); + VLOG("[draw] c%d[%d] = %g %g %g %g", slot, v, f[0], f[1], f[2], f[3]); + } + } + } +} + +// TOTK_VK_SKIP=,...: draws with these vertex / fragment shaders are not drawn (development: bisecting) +static int skip_listed(void) { + static const char *skip; + static int init; + if (!init) { skip = getenv("TOTK_VK_SKIP"); init = 1; } + for (const char *p = skip; p && *p;) { + size_t n = strcspn(p, ","); + for (int s = 0; s < 2; s++) + if (n && S.sh[s] && S.sh[s]->key && !strncmp(S.sh[s]->key, p, n)) return 1; + p += n + (p[n] == ','); + } + return 0; +} + +// The walk of a submission (vkb_mt.c): the storage buffers a draw or dispatch writes void the uniforms captured there +// from its position on (vkb_ucap_drop), as mirrors_prepare does when one thread records. +static void prescan_stores(const vkb_shader *sh, int stage) { + for (int i = 0; i < sh->nsbuf; i++) { + if (!sh->sbuf[i].write) continue; + uint64_t addr, size; + sbuf_range(sh, stage, i, &addr, &size); + if (addr && size) vkb_ucap_drop(addr, size); + } +} +void vkb_draw_prescan(const uint8_t *a, int indexed, int dispatch) { + big_init(); + if (!resolve_program()) return; + if (dispatch) { if (S.sh[5]) prescan_stores(S.sh[5], 5); return; } + if (!S.sh[0] || skip_listed()) return; + int ok; + topology(indexed ? ((const p_draw_idx *)a)->prim : ((const p_draw *)a)->prim, &ok); + if (!ok) return; + prescan_stores(S.sh[0], 0); + if (S.sh[1]) prescan_stores(S.sh[1], 1); +} + +// Objects made on first use: made before threads record (the prologue is this thread's command buffer then). +void vkb_draw_mt_init(void) { + big_init(); + fixed_buffers(); + if (!g_dummy) dummy_init(); + static const VkImageViewType types[] = {VK_IMAGE_VIEW_TYPE_2D, VK_IMAGE_VIEW_TYPE_2D_ARRAY, VK_IMAGE_VIEW_TYPE_CUBE, VK_IMAGE_VIEW_TYPE_CUBE_ARRAY}; + for (unsigned t = 0; t < sizeof types / sizeof *types; t++) dummy_view(types[t]); // (1D ones: as before, on first use) + sampler(-1); +} + +void vkb_draw(const uint8_t *a, int indexed) { + big_init(); + fixed_buffers(); + vkb_audit_draw_begin(); + const p_draw *dr = (const p_draw *)a; + const p_draw_idx *di = (const p_draw_idx *)a; + int trace = draw_trace(); + if (trace) { + const p_set_rt *rt = vkb_rt(); + const tex_t *c0 = rt->n > 0 ? rt->color[0] : NULL; + resolve_program(); + VLOG("[draw] frame %llu %s prim %d count %d inst %d | rt %d: tex %llu fmt %#x %ux%u%s | prog %llu vs %s fs %s | %d attrs", + (unsigned long long)vkb_frame(), indexed ? "indexed" : "arrays", indexed ? di->prim : dr->prim, indexed ? di->count : dr->count, + indexed ? di->instances : dr->instances, rt->n, c0 ? (unsigned long long)c0->serial : 0ull, c0 ? c0->d.format : 0, + c0 ? c0->d.width : 0, c0 ? c0->d.height : 0, rt->depth ? " +depth" : "", + S.prog ? (unsigned long long)S.prog->serial : 0ull, S.sh[0] ? "ok" : "-", S.sh[1] ? "ok" : "-", S.nvattr); + char tl[512]; // every target with its memory (two texture objects over the same memory: aliasing) + int tn = 0; + for (int i = 0; i < rt->n && i < 8; i++) + if (rt->color[i]) { + vkb_image *ti = vkb_image_of(rt->color[i]); + tn += snprintf(tl + tn, sizeof tl - (size_t)tn, " c%d=%llu@%p#%04llx", i, (unsigned long long)rt->color[i]->serial, rt->color[i]->mem, + ti ? (unsigned long long)((uintptr_t)ti->img & 0xffff) : 0ull); + } + if (rt->depth) { + vkb_image *ti = vkb_image_of(rt->depth); + tn += snprintf(tl + tn, sizeof tl - (size_t)tn, " d=%llu@%p#%04llx", (unsigned long long)rt->depth->serial, rt->depth->mem, + ti ? (unsigned long long)((uintptr_t)ti->img & 0xffff) : 0ull); + } + VLOG("[draw] targets:%s", tn ? tl : " -"); + const blend_st *b0 = &S.blend[0]; + VLOG("[draw] state: blend en %#x t0 %s %u/%u op %u, a %u/%u op %u | chmask %#x | depth %s test %u write %u func %u stencil %u", + S.has_color ? (unsigned)S.color.blend_en : 0u, S.has_blend[0] ? "set" : "-", (unsigned)b0->src_rgb, (unsigned)b0->dst_rgb, (unsigned)b0->eq_rgb, + (unsigned)b0->src_a, (unsigned)b0->dst_a, (unsigned)b0->eq_a, S.chmask.mask, S.has_ds ? "set" : "-", (unsigned)S.ds.dt, (unsigned)S.ds.dw, (unsigned)S.ds.df, (unsigned)S.ds.st); + { + const p_set_rt *r = vkb_rt(); + VLOG("[draw] depth: range %s %g..%g, viewport0 z %g..%g, offset %s (units %g factor %g clamp %g), clamp %d, target %s format %#x", + S.has_dr ? "set" : "-", S.dr[0], S.dr[1], S.nvp ? S.vp[0][4] : 0.0f, S.nvp ? S.vp[0][5] : 1.0f, + S.has_poly && (S.poly.offset_en & 4) ? "on" : "off", S.po.units, S.po.factor, S.po.clamp, S.depth_clamp, + r->depth ? "yes" : "no", r->depth ? r->depth->d.format : 0); + } + if (S.sh[0]) trace_resources(S.sh[0], 0); + if (S.sh[1]) trace_resources(S.sh[1], 1); + } + if (!resolve_program() || !S.sh[0]) { + vkb_cnt_add(g_skipped[0], 1); + vkb_audit(S.bad_key ? A_SHADER_UNSUPPORTED : A_DRAW_NO_PROGRAM, S.bad_key); + if (trace) VLOG("[draw] skipped: no program / translation"); + return; + } + if (skip_listed()) return; + int ok; + VkPrimitiveTopology topo = topology(indexed ? di->prim : dr->prim, &ok); + if (!ok) { + vkb_cnt_add(g_skipped[1], 1); + vkb_audit(A_DRAW_TOPOLOGY, S.sh[1] ? S.sh[1]->key : S.sh[0]->key); + static int said; if (!said++) VLOG("[temporary] primitive %d not drawn (no Vulkan topology)", indexed ? di->prim : dr->prim); + return; + } + const vkb_shader *vs = S.sh[0], *fs = S.sh[1]; + static int snap_after; // TOTK_SNAP_AFTER=1: the draw's render targets after it as well (a_rt*) + if (vkb_tex_snap_wanted(vs->key, fs ? fs->key : NULL, vkb_frame())) { // TOTK_SNAP: what this draw sees + snap_after = getenv("TOTK_SNAP_AFTER") != NULL; + vkb_pass_end_r(PE_SNAP); + char what[16]; + for (int st = 0; st < 2; st++) { + const vkb_shader *sh = st ? fs : vs; + for (int i = 0; sh && i < sh->ntex; i++) { + const view_t *v = NULL; + vkb_image *im = sh->tex[i].separate ? NULL : handle_image(S.tex[st][sh->tex[i].slot], &v); + snprintf(what, sizeof what, "%ct%d", st ? 'f' : 'v', sh->tex[i].slot); + if (im) vkb_tex_snap(vkb_cmd(), im, what); + } + } + const p_set_rt *rt = vkb_rt(); + for (int i = 0; i < rt->n && i < 8; i++) { + snprintf(what, sizeof what, "rt%d", i); + if (rt->color[i]) vkb_tex_snap(vkb_cmd(), vkb_image_of(rt->color[i]), what); + } + if (rt->depth) vkb_tex_snap(vkb_cmd(), vkb_image_of(rt->depth), "rtdepth"); + } + if (!g_dummy) { vkb_pass_end(); dummy_init(); } + if (g_stores_pending) { // the draw reads what earlier storing draws wrote: their writes first (outside the pass) + static int every = -1; // TOTK_VK_STORES_EVERY=1 (development A/B): every draw waits, as before 2026-10-24 + if (every < 0) { const char *e = getenv("TOTK_VK_STORES_EVERY"); every = e && *e && *e != '0'; } + int reads = every || reads_written(vs, 0) || (fs && reads_written(fs, 1)); + for (uint32_t vb = 0; vb < 16 && !reads; vb++) reads = written_overlaps(S.vbuf[vb].address, S.vbuf[vb].size); + if (!reads && indexed && di->ibuf) reads = written_overlaps(di->ibuf, (uint64_t)di->count * (di->itype == 0 ? 1 : di->itype == 1 ? 2 : 4)); + if (reads) vkb_pass_end_r(PE_STORES); // pass_begin records the barrier + static long long log_at = -1, left = 40; // TOTK_VK_STORES_LOG=: these decisions (development) + if (log_at < 0) { const char *e = getenv("TOTK_VK_STORES_LOG"); log_at = e ? atoll(e) : 0; } + if (log_at && vkb_frame() >= (uint64_t)log_at && left > 0) { + left--; + VLOG("[stores] vs %.12s fs %.12s: reads written %d (ranges %d, all %d, images %d; first %#llx+%llu)", vs->key, fs ? fs->key : "-", reads, + g_nwr, g_wr_all, g_wr_img, g_nwr ? (unsigned long long)g_wr[0].addr : 0ull, g_nwr ? (unsigned long long)g_wr[0].size : 0ull); + for (int st = 0; st < 2; st++) { + const vkb_shader *sh = st ? fs : vs; + for (int i = 0; sh && i < sh->nsbuf; i++) { + uint64_t addr, size; + sbuf_range(sh, st, i, &addr, &size); + VLOG("[stores] %s sbuf %d %#llx+%llu read %d write %d", st ? "fs" : "vs", i, (unsigned long long)addr, (unsigned long long)size, sh->sbuf[i].read, sh->sbuf[i].write); + } + for (int slot = 3; sh && slot < VKB_NCBUF; slot++) + if (sh->cbuf_mask & (1u << slot)) + VLOG("[stores] %s c%d %#llx+%llu", st ? "fs" : "vs", slot, (unsigned long long)S.ubo[st][slot - 3].addr, (unsigned long long)S.ubo[st][slot - 3].size); + } + } + } + feedback_scan(vs, fs); + if (vkb_in_pass() && vkb_pass_fb() != g_fb) vkb_pass_end_r(PE_FEEDBACK); // [temporary] a pass per change of sampled targets + if ((vkb_in_pass() && (images_need_work(vs, 0) || (fs && images_need_work(fs, 1))))) vkb_pass_end_r(PE_IMAGES); + if (!vkb_in_pass()) { + prepare_images(vs, 0); + if (fs) prepare_images(fs, 1); + } + mirrors_prepare(vs, 0); // written storage buffers: device-local copies for the submission + if (fs) mirrors_prepare(fs, 1); + mirrors_prepare_fixed(indexed, di); + VkExtent2D bare = {0, 0}; // a draw without targets covers its viewport + if (S.nvp && S.vp[0][2] > 0 && S.vp[0][3] > 0) + bare = (VkExtent2D){(uint32_t)ceilf(S.vp[0][0] + S.vp[0][2]), (uint32_t)ceilf(S.vp[0][1] + S.vp[0][3])}; + if (!vkb_pass_begin(bare, g_fb)) { vkb_cnt_add(g_skipped[2], 1); vkb_audit(A_DRAW_NO_TARGETS, fs ? fs->key : vs->key); return; } + VkCommandBuffer cb = vkb_cmd(); + writes_t *W = &B->writes; + W->n = 0; + write_stage(W, VK_NULL_HANDLE, vs, 0); + if (fs) write_stage(W, VK_NULL_HANDLE, fs, 1); + bind_set(cb, resolve_set(W, 0), 0); + VkShaderStageFlagBits stages[5] = {VK_SHADER_STAGE_VERTEX_BIT, VK_SHADER_STAGE_FRAGMENT_BIT, VK_SHADER_STAGE_TESSELLATION_CONTROL_BIT, + VK_SHADER_STAGE_TESSELLATION_EVALUATION_BIT, VK_SHADER_STAGE_GEOMETRY_BIT}; + uint32_t sint, ints = int_attributes(&sint) & vs->in_mask; // integer attributes: the vertex shader's variant for them + VkShaderEXT vobj = ints ? vkb_shader_variant(vs, ints, sint) : vs->obj; + uint32_t int_inputs = vobj != vs->obj ? ints : 0; + if (ints && !int_inputs) vkb_audit(A_ATTR_INT_AS_FLOAT, vs->key); + if (fs && (fs->in_mask & ~vs->out_mask)) vkb_audit(A_FS_INPUT_UNWRITTEN, fs->key); + if (S.has_ms && S.ms.enable && S.ms.samples > 1) vkb_audit(A_MULTISAMPLE, fs ? fs->key : vs->key); + if (vkb_render_suppressed()) vkb_audit(A_RENDER_DISABLED, fs ? fs->key : vs->key); + VkShaderEXT objs[5] = {vobj, fs ? fs->obj : VK_NULL_HANDLE, VK_NULL_HANDLE, VK_NULL_HANDLE, VK_NULL_HANDLE}; + if (dyn(D_SHADERS, objs, sizeof objs)) vk.vkCmdBindShadersEXT(cb, 5, stages, objs); + set_dynamic_state(cb, vkb_rt_count(), vkb_rt_extent(), topo); + if (vkc.has_feedback_loop) { // required state with shader objects once the feature is enabled + VkImageAspectFlags loop = (g_fb & 0xFF ? VK_IMAGE_ASPECT_COLOR_BIT : 0) | (g_fb & VKB_FB_DEPTH ? g_fb_rt[8]->aspect : 0); + DYN(D_FBLOOP, VkImageAspectFlags, loop, vk.vkCmdSetAttachmentFeedbackLoopEnableEXT(cb, v_)); + } + // A draw that samples a target of its pass sees what earlier draws of the pass wrote there only after a barrier + // inside the pass (VK_KHR_dynamic_rendering_local_read, by region: each pixel waits for its own writes). Without it + // the read may see old contents (as SavageWind found with BotW's terrain, which ORs bits into its G-buffer pixel). + static __thread uint64_t wrote_pass; // the pass in which a draw was recorded since the last such barrier + static int no_local = -1; // TOTK_VK_NO_LOCAL_BARRIER=1 (development A/B): no barrier inside the pass + if (no_local < 0) { const char *e = getenv("TOTK_VK_NO_LOCAL_BARRIER"); no_local = e && *e == '1'; } + if (g_fb && vkc.has_local_read && !no_local && wrote_pass == vkb_pass_serial()) { + VkMemoryBarrier2 mb = {VK_STRUCTURE_TYPE_MEMORY_BARRIER_2, + .srcStageMask = VK_PIPELINE_STAGE_2_COLOR_ATTACHMENT_OUTPUT_BIT | VK_PIPELINE_STAGE_2_EARLY_FRAGMENT_TESTS_BIT | VK_PIPELINE_STAGE_2_LATE_FRAGMENT_TESTS_BIT, + .srcAccessMask = VK_ACCESS_2_COLOR_ATTACHMENT_WRITE_BIT | VK_ACCESS_2_DEPTH_STENCIL_ATTACHMENT_WRITE_BIT, + .dstStageMask = VK_PIPELINE_STAGE_2_FRAGMENT_SHADER_BIT, + .dstAccessMask = VK_ACCESS_2_INPUT_ATTACHMENT_READ_BIT}; + VkDependencyInfo dep = {VK_STRUCTURE_TYPE_DEPENDENCY_INFO, .dependencyFlags = VK_DEPENDENCY_BY_REGION_BIT, + .memoryBarrierCount = 1, .pMemoryBarriers = &mb}; + vk.vkCmdPipelineBarrier2(cb, &dep); + vkb_cnt_add(g_local_barriers, 1); + } + wrote_pass = vkb_pass_serial(); + set_vertex_input(cb, int_inputs); + uint32_t inst = indexed ? (uint32_t)(di->instances ? di->instances : 1) : (uint32_t)(dr->instances ? dr->instances : 1); + vkb_checkpoint(indexed ? "draw indexed" : "draw", vs->key, fs ? fs->key : NULL); + vkb_passprof_note(fs ? fs->key : vs->key); + if (indexed) { + VkBuffer buf; + VkDeviceSize off; + uint32_t isz = di->itype == 0 ? 1 : di->itype == 1 ? 2 : 4; + if (!di->ibuf || !vkb_gpu_addr(di->ibuf, (uint64_t)di->count * isz, &buf, &off)) { + vkb_cnt_add(g_skipped[3], 1); + vkb_audit(A_DRAW_NO_INDEX, vs->key); + return; + } + VkIndexType it = di->itype == 0 ? VK_INDEX_TYPE_UINT8_EXT : di->itype == 1 ? VK_INDEX_TYPE_UINT16 : VK_INDEX_TYPE_UINT32; + vk.vkCmdBindIndexBuffer(cb, buf, off, it); + int gq = gprof_begin(); + vk.vkCmdDrawIndexed(cb, (uint32_t)di->count, inst, 0, di->base_vertex, (uint32_t)di->base_instance); + gprof_end(gq, fs ? fs : vs, 0, (uint32_t)di->count, inst); + } else { + int gq = gprof_begin(); + vk.vkCmdDraw(cb, (uint32_t)dr->count, inst, (uint32_t)dr->first, (uint32_t)dr->base_instance); + gprof_end(gq, fs ? fs : vs, 0, (uint32_t)dr->count, inst); + } + if (vs->stores || (fs && fs->stores)) { + g_stores_pending = 1; + if (vs->stores) written_add(vs, 0); + if (fs && fs->stores) written_add(fs, 1); + } + vkb_cnt_add(g_draws, 1); + if (snap_after) { + snap_after = 0; + vkb_pass_end_r(PE_SNAP); + const p_set_rt *rt = vkb_rt(); + char what[16]; + for (int i = 0; i < rt->n && i < 8; i++) { + snprintf(what, sizeof what, "a_rt%d", i); + if (rt->color[i]) vkb_tex_snap(vkb_cmd(), vkb_image_of(rt->color[i]), what); + } + } +} + +void vkb_dispatch(const uint8_t *a) { + big_init(); + fixed_buffers(); + const p_dispatch *d = (const p_dispatch *)a; + if (draw_trace()) { + VLOG("[draw] frame %llu dispatch %dx%dx%d | prog %llu cs %s", (unsigned long long)vkb_frame(), (int)d->x, (int)d->y, (int)d->z, + S.prog ? (unsigned long long)S.prog->serial : 0ull, resolve_program() && S.sh[5] ? "ok" : "-"); + if (resolve_program() && S.sh[5]) trace_resources(S.sh[5], 5); + } + vkb_audit_draw_begin(); + if (!resolve_program() || !S.sh[5]) { vkb_cnt_add(g_skipped[0], 1); vkb_audit(S.bad_key ? A_SHADER_UNSUPPORTED : A_DRAW_NO_PROGRAM, S.bad_key); return; } + vkb_pass_end_r(PE_DISPATCH); + dummy_init(); + const vkb_shader *cs = S.sh[5]; + g_fb = 0; // no targets: every sampled image leaves any feedback layout + prepare_images(cs, 5); + VkCommandBuffer cb = vkb_cmd(); + mirrors_prepare(cs, 5); + writes_t *W = &B->writes; + W->n = 0; + write_stage(W, VK_NULL_HANDLE, cs, 5); + VkDescriptorSet set = resolve_set(W, 1); + // [temporary] compute sees all earlier work and later work sees it: full barriers + VkMemoryBarrier2 mb = {VK_STRUCTURE_TYPE_MEMORY_BARRIER_2, .srcStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + .srcAccessMask = VK_ACCESS_2_MEMORY_WRITE_BIT, .dstStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + .dstAccessMask = VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT}; + VkDependencyInfo dep = {VK_STRUCTURE_TYPE_DEPENDENCY_INFO, .memoryBarrierCount = 1, .pMemoryBarriers = &mb}; + vk.vkCmdPipelineBarrier2(cb, &dep); + g_stores_pending = 0; // the full barrier covers the draws' stores too + bind_set(cb, set, 1); + VkShaderStageFlagBits st = VK_SHADER_STAGE_COMPUTE_BIT; + vk.vkCmdBindShadersEXT(cb, 1, &st, &cs->obj); + int gq = gprof_begin(); + vkb_checkpoint("dispatch", cs->key, NULL); + vk.vkCmdDispatch(cb, (uint32_t)d->x, (uint32_t)d->y, (uint32_t)d->z); + gprof_end(gq, cs, 1, (uint32_t)(d->x * d->y * d->z), 1); + vk.vkCmdPipelineBarrier2(cb, &dep); + vkb_cnt_add(g_draws, 1); +} diff --git a/src/gfx/vkb_lock.h b/src/gfx/vkb_lock.h new file mode 100644 index 0000000..a5a72ab --- /dev/null +++ b/src/gfx/vkb_lock.h @@ -0,0 +1,40 @@ +// Contention of the replayer's shared tables (development measurement, the per-frame log line): a lock that is taken +// at once costs nothing extra; a contended one adds its wait to its kind's total. +#pragma once +#include +#include +#include + +enum { LK_TEXREG, LK_IMAGES, LK_POOLS, LK_COUNT }; // texture registry (vkb_draw.c), images (vkb_tex.c), memory pools (backend_vk.c) +extern _Atomic uint64_t vkb_lock_wait_ns[LK_COUNT], vkb_lock_waits[LK_COUNT]; +uint64_t runtime_now_ns(void); +// Time spent in some replay steps (development measurement, the per-frame log line) +enum { PK_CREATE, PK_UPLOAD, PK_SUBMIT, PK_COUNT }; // new images (vkCreateImage + memory), uploads from game memory, queue submits +extern _Atomic uint64_t vkb_step_ns[PK_COUNT], vkb_step_n[PK_COUNT]; +static inline void vkb_step_add(int kind, uint64_t t0) { + atomic_fetch_add_explicit(&vkb_step_ns[kind], runtime_now_ns() - t0, memory_order_relaxed); + atomic_fetch_add_explicit(&vkb_step_n[kind], 1, memory_order_relaxed); +} +static inline void vkb_lock(pthread_mutex_t *m, int kind) { + if (!pthread_mutex_trylock(m)) return; + uint64_t t0 = runtime_now_ns(); + pthread_mutex_lock(m); + atomic_fetch_add_explicit(&vkb_lock_wait_ns[kind], runtime_now_ns() - t0, memory_order_relaxed); + atomic_fetch_add_explicit(&vkb_lock_waits[kind], 1, memory_order_relaxed); +} + +// A reader-writer lock for the tables the recording threads read for every draw (pools, the texture registry): readers +// do not exclude each other. Windows SRWLOCK (pointer-sized, zero = unlocked); backend_vk.c implements it. +typedef struct { void *opaque; } vkb_rw; +void vkb_rw_read(vkb_rw *l, int kind); +void vkb_rw_read_end(vkb_rw *l); +void vkb_rw_write(vkb_rw *l, int kind); +void vkb_rw_write_end(vkb_rw *l); + +// Counters the recording threads bump per draw (the per-frame log lines): one cache line per recording thread, summed +// when read, so that threads do not contend for one counter. +#define VKB_MAXREC 8 // recording threads at most (vkb_mt.c), the translation thread included +typedef struct { uint64_t v; char pad[56]; } vkb_cnt[VKB_MAXREC]; +int vkb_worker(void); // vkb_mt.c: this thread's recording index (0: the translation thread, or any other thread) +static inline void vkb_cnt_add(vkb_cnt c, uint64_t n) { c[vkb_worker()].v += n; } +static inline uint64_t vkb_cnt_sum(const vkb_cnt c) { uint64_t s = 0; for (int i = 0; i < VKB_MAXREC; i++) s += c[i].v; return s; } diff --git a/src/gfx/vkb_shader.c b/src/gfx/vkb_shader.c new file mode 100644 index 0000000..2500525 --- /dev/null +++ b/src/gfx/vkb_shader.c @@ -0,0 +1,440 @@ +// Shaders of the NVN replayer (milestone 3.3 step 4): the game's Maxwell shaders, translated to SPIR-V by Ryujinx's +// translator (MacNativeGames' totk_xlate), become Vulkan shader objects (VK_EXT_shader_object): no pipelines to build +// at draw time, every other state is dynamic. +// - A shader is identified as MacNativeGames does: SHA-256 of its control section (0x880 bytes) and its code (length +// at control+0x704); the stage is at control+0x714 (0 vertex, 1 fragment, 5 compute). +// - [temporary] The translation is the default variant from the translator comparison (work/shaders/v143/ryujinx: every +// texture 2D-typed by its own reflection, attributes and outputs float). The runtime translation service with +// variants per bound state (MacNativeGames' totk_shader_rt) replaces it before the open world. +// - Descriptors: one universal layout. Each stage owns 64 bindings of set 0 (vertex 0-63, fragment 64-127; compute its +// own layout): +0 Ryujinx support buffer, +1..17 constant buffers c0-c16, +18..33 storage buffers, +34..57 combined +// textures, +58..61 images. The translator numbers bindings densely per shader; they are rewritten in the SPIR-V +// when it is loaded, so any vertex shader combines with any fragment shader under one pipeline layout. +#include +#include // after windows.h (its types) +#include +#include +#include +#include + +#include "compat/gfx_compat.h" +#include "host/hle.h" +#include "vkb.h" +#include "vkb_shader.h" +#include "vkb_spv.h" + +#define VLOG(...) LOG("[vk] " __VA_ARGS__) + +static VkDescriptorSetLayout g_layout[2]; // 0 graphics, 1 compute +static VkPipelineLayout g_playout[2]; + +static void make_layouts(void) { + for (int compute = 0; compute < 2; compute++) { + VkDescriptorSetLayoutBinding b[128]; + VkDescriptorBindingFlags fl[128]; + uint32_t n = 0; + for (int st = 0; st < (compute ? 1 : 2); st++) { + VkShaderStageFlags sf = compute ? VK_SHADER_STAGE_COMPUTE_BIT : st ? VK_SHADER_STAGE_FRAGMENT_BIT : VK_SHADER_STAGE_VERTEX_BIT; + uint32_t base = (uint32_t)st * VKB_STAGE_BINDINGS; + struct { uint32_t off, count; VkDescriptorType type; } r[] = { + {VKB_B_SUPPORT, 1, VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER}, {VKB_B_CBUF, VKB_NCBUF, VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER}, + {VKB_B_SBUF, VKB_NSBUF, VK_DESCRIPTOR_TYPE_STORAGE_BUFFER}, {VKB_B_TEX, VKB_NTEX, VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER}, + {VKB_B_IMG, VKB_NIMG, VK_DESCRIPTOR_TYPE_STORAGE_IMAGE}}; + for (unsigned k = 0; k < sizeof r / sizeof *r; k++) + for (uint32_t i = 0; i < r[k].count; i++) { + b[n] = (VkDescriptorSetLayoutBinding){base + r[k].off + i, r[k].type, 1, sf, NULL}; + fl[n++] = VK_DESCRIPTOR_BINDING_PARTIALLY_BOUND_BIT; // only what the shaders use is written + } + } + VkDescriptorSetLayoutBindingFlagsCreateInfo fi = {VK_STRUCTURE_TYPE_DESCRIPTOR_SET_LAYOUT_BINDING_FLAGS_CREATE_INFO, + .bindingCount = n, .pBindingFlags = fl}; + VkDescriptorSetLayoutCreateInfo ci = {VK_STRUCTURE_TYPE_DESCRIPTOR_SET_LAYOUT_CREATE_INFO, &fi, .bindingCount = n, .pBindings = b}; + if (vkb_db_on()) { ci.pNext = NULL; ci.flags = VK_DESCRIPTOR_SET_LAYOUT_CREATE_DESCRIPTOR_BUFFER_BIT_EXT; } // vkb_desc.c + if (!vk_check(vk.vkCreateDescriptorSetLayout(vkc.device, &ci, NULL, &g_layout[compute]), "descriptor set layout")) die("layout"); + VkPipelineLayoutCreateInfo pi = {VK_STRUCTURE_TYPE_PIPELINE_LAYOUT_CREATE_INFO, .setLayoutCount = 1, .pSetLayouts = &g_layout[compute]}; + if (!vk_check(vk.vkCreatePipelineLayout(vkc.device, &pi, NULL, &g_playout[compute]), "pipeline layout")) die("layout"); + } + if (vkb_db_on()) vkb_db_layouts(g_layout); +} +VkDescriptorSetLayout vkb_set_layout(int compute) { if (!g_layout[0]) make_layouts(); return g_layout[compute]; } +VkPipelineLayout vkb_pipeline_layout(int compute) { if (!g_layout[0]) make_layouts(); return g_playout[compute]; } + +// ---------------- the reflection (.res.json): a small reader for its fixed shape ---------------- +static const char *json_array(const char *j, const char *name) { // just after the '[' of "name": [ + char pat[64]; + snprintf(pat, sizeof pat, "\"%s\"", name); + const char *p = strstr(j, pat); + if (!p) return NULL; + p = strchr(p + strlen(pat), '['); + return p ? p + 1 : NULL; +} +// next {...} object of an array: its bounds; NULL at the closing ']' +static const char *json_next(const char *p, const char **end) { + while (*p && *p != '{' && *p != ']') p++; + if (*p != '{') return NULL; + int depth = 0; + for (const char *q = p; *q; q++) { + if (*q == '{') depth++; + else if (*q == '}' && !--depth) { *end = q + 1; return p; } + } + return NULL; +} +static long json_int(const char *obj, const char *end, const char *key, long dflt) { + char pat[64]; + snprintf(pat, sizeof pat, "\"%s\":", key); + const char *p = strstr(obj, pat); + if (!p || p >= end) return dflt; + p += strlen(pat); + while (*p == ' ') p++; + if (!strncmp(p, "true", 4)) return 1; + if (!strncmp(p, "false", 5)) return 0; + return strtol(p, NULL, 10); +} +static long json_top_int(const char *j, const char *key, long dflt) { return json_int(j, j + strlen(j), key, dflt); } + +static char *read_file(const char *path, size_t *n) { + FILE *f = fopen(path, "rb"); + if (!f) return NULL; + fseek(f, 0, SEEK_END); + long len = ftell(f); + fseek(f, 0, SEEK_SET); + char *b = malloc((size_t)len + 1); + if (b && fread(b, 1, (size_t)len, f) != (size_t)len) { free(b); b = NULL; } + if (b) b[len] = 0; + fclose(f); + if (n) *n = (size_t)len; + return b; +} + +// ---------------- shaders by key ---------------- +typedef struct shentry { char key[65]; int c1_set, prewarmed; vkb_shader sh; struct shentry *next; } shentry; // prewarmed: made ahead (prewarm_thread) +static shentry *g_tab[4096]; +static pthread_mutex_t g_mu = PTHREAD_MUTEX_INITIALIZER; +static uint64_t g_loaded, g_missing; + +static BCRYPT_ALG_HANDLE g_sha; // opened once (opening it per hash cost more than the hash) +static void sha_open(void) { if (BCryptOpenAlgorithmProvider(&g_sha, BCRYPT_SHA256_ALGORITHM, NULL, 0)) die("SHA-256 provider"); } +static void sha256_hex(const void *a, size_t na, const void *b, size_t nb, char out[65]) { + static pthread_once_t once = PTHREAD_ONCE_INIT; + pthread_once(&once, sha_open); + BCRYPT_ALG_HANDLE alg = g_sha; + BCRYPT_HASH_HANDLE h; + uint8_t md[32]; + if (BCryptCreateHash(alg, &h, NULL, 0, NULL, 0, 0)) die("SHA-256 provider"); + BCryptHashData(h, (PUCHAR)a, (ULONG)na, 0); + BCryptHashData(h, (PUCHAR)b, (ULONG)nb, 0); + BCryptFinishHash(h, md, sizeof md, 0); + BCryptDestroyHash(h); + static const char hex[] = "0123456789abcdef"; + for (int i = 0; i < 32; i++) { out[2 * i] = hex[md[i] >> 4]; out[2 * i + 1] = hex[md[i] & 15]; } + out[64] = 0; +} + +static const char *cache_dir(void) { + static const char *d; + if (!d) { d = getenv("TOTK_SHADER_DIR"); if (!d) d = "data/shaders/ryujinx"; } + return d; +} + +// The translator declares gl_PerVertex with ClipDistance[8] and CullDistance[1]: 9 > maxCombinedClipAndCullDistances (8) +// on NVIDIA, though no shader writes the cull distance. The CullDistance member (the block's last) is dropped: its +// OpTypeStruct operand, OpMemberName and OpMemberDecorate. Returns the new word count. +static size_t drop_cull_distance(uint32_t *w, size_t n) { + uint32_t st = 0, mem = 0; + for (size_t i = 5; i < n;) { + uint32_t op = w[i] & 0xffff, wc = w[i] >> 16; + if (!wc || i + wc > n) return n; + if (op == 72 && wc == 5 && w[i + 3] == 11 && w[i + 4] == 4) { st = w[i + 1]; mem = w[i + 2]; } // BuiltIn CullDistance + i += wc; + } + if (!st) return n; + size_t o = 5; + for (size_t i = 5; i < n;) { + uint32_t op = w[i] & 0xffff, wc = w[i] >> 16; + int drop = (op == 72 || op == 6) && wc >= 3 && w[i + 1] == st && w[i + 2] == mem; // OpMemberDecorate / OpMemberName + if (op == 30 && w[i + 1] == st && wc - 2 == mem + 1) { // OpTypeStruct: the member is the last operand + memmove(&w[o], &w[i], (wc - 1) * 4); + w[o] = ((wc - 1) << 16) | 30; + o += wc - 1; + } else if (!drop) { + memmove(&w[o], &w[i], wc * 4); + o += wc; + } + i += wc; + } + return o; +} + +static VkResult create_object(VkShaderStageFlagBits stage, const uint32_t *spv, size_t bytes, VkShaderEXT *obj) { + VkDescriptorSetLayout layout = vkb_set_layout(stage == VK_SHADER_STAGE_COMPUTE_BIT); + VkShaderCreateInfoEXT ci = {VK_STRUCTURE_TYPE_SHADER_CREATE_INFO_EXT, .stage = stage, + .nextStage = stage == VK_SHADER_STAGE_VERTEX_BIT ? VK_SHADER_STAGE_FRAGMENT_BIT : 0, + .codeType = VK_SHADER_CODE_TYPE_SPIRV_EXT, .codeSize = bytes, .pCode = spv, .pName = "main", + .setLayoutCount = 1, .pSetLayouts = &layout}; + return vk.vkCreateShadersEXT(vkc.device, 1, &ci, NULL, obj); +} + +// A vertex shader for integer attributes at the locations in intmask (signed where sintmask has the bit): those +// inputs read the raw bits (vkb_spv.c). Made on first use and kept; 0 masks give the shader itself. +// [temporary] made during play (a compile on first use); the variant list and building ahead of time - phase 5. +VkShaderEXT vkb_shader_variant(const vkb_shader *cs, uint32_t intmask, uint32_t sintmask) { + vkb_shader *s = (vkb_shader *)cs; + intmask &= s->in_mask; + sintmask &= intmask; + if (!intmask || !s->spv) return s->obj; + pthread_mutex_lock(&g_mu); + for (int i = 0; i < s->nvariants; i++) + if (s->variants[i].intmask == intmask && s->variants[i].sintmask == sintmask) { pthread_mutex_unlock(&g_mu); return s->variants[i].obj; } + pthread_mutex_unlock(&g_mu); + uint32_t *out = NULL; + size_t nw = 0; + VkShaderEXT obj = VK_NULL_HANDLE; + if (vkb_spv_int_inputs(s->spv, s->nspv_words, intmask, sintmask, &out, &nw)) { + if (getenv("TOTK_SPV_DUMP")) { // development: the variant's SPIR-V for spirv-val / spirv-dis + char path[256]; + snprintf(path, sizeof path, "%s/%.16s-i%x-s%x.spv", getenv("TOTK_SPV_DUMP"), s->key ? s->key : "x", intmask, sintmask); + FILE *f = fopen(path, "wb"); + if (f) { fwrite(out, 4, nw, f); fclose(f); } + } + if (create_object(VK_SHADER_STAGE_VERTEX_BIT, out, nw * 4, &obj) != VK_SUCCESS) obj = VK_NULL_HANDLE; + free(out); + } + if (!obj) { + static int said; + if (said++ < 8) VLOG("[temporary] shader %.12s: no variant for integer inputs %#x: the float reading stays", s->key ? s->key : "?", intmask); + obj = s->obj; + } else { + static int made; + if (made++ < 16) VLOG("shader %.12s: variant for integer inputs %#x (signed %#x)", s->key ? s->key : "?", intmask, sintmask); + } + pthread_mutex_lock(&g_mu); + if (s->nvariants < VKB_VARIANTS) s->variants[s->nvariants++] = (vkb_variant){intmask, sintmask, obj}; + pthread_mutex_unlock(&g_mu); + return obj; +} + +// Reads the reflection, rewrites the SPIR-V bindings into the universal layout, creates the shader object. +static int load(vkb_shader *s, const char *key, int nvn_stage) { + char path[1024]; + snprintf(path, sizeof path, "%s/%s.res.json", cache_dir(), key); + char *j = read_file(path, NULL); + snprintf(path, sizeof path, "%s/%s.spv", cache_dir(), key); + size_t nspv = 0; + uint32_t *spv = (uint32_t *)read_file(path, &nspv); + if (!j || !spv || nspv < 20 || (nspv & 3) || spv[0] != 0x07230203) { free(j); free(spv); return 0; } + s->stage = nvn_stage; + uint32_t base = nvn_stage == 1 ? VKB_STAGE_BINDINGS : 0; + uint32_t remap[256]; // translator binding -> universal binding (UINT32_MAX: unknown) + for (int i = 0; i < 256; i++) remap[i] = UINT32_MAX; + remap[0] = base + VKB_B_SUPPORT; + const char *p, *e; + // constant buffers: slot = NVN constant buffer index + if ((p = json_array(j, "cbufs"))) + for (; (p = json_next(p, &e)); p = e) { + long b = json_int(p, e, "binding", -1), slot = json_int(p, e, "slot", -1); + if (b < 0 || b > 255 || slot < 0 || slot >= VKB_NCBUF) continue; + remap[b] = base + VKB_B_CBUF + (uint32_t)slot; + s->cbuf_mask |= 1u << slot; + } + // storage buffers: the address comes from the NVN driver bank c0, 16 bytes per slot: the stage's storage buffer + // table (vertex 0x110, fragment 0x510, compute 0x310) or, for compute, its uniform buffer table (0x210): TOTK's + // compute shaders also use uniform buffers as storage (MacNativeGames' "ubo_ptr") + int sb_base = nvn_stage == 0 ? 0x110 : nvn_stage == 1 ? 0x510 : 0x310; + if ((p = json_array(j, "sbufs"))) + for (; (p = json_next(p, &e)); p = e) { + long b = json_int(p, e, "binding", -1), off = json_int(p, e, "cb_offset", -1) * 4; + if (b < 0 || b > 255 || s->nsbuf >= VKB_NSBUF) { s->unsupported = 1; continue; } + long cb = json_int(p, e, "cb_slot", 0); + int write = (json_int(p, e, "flags", 0) & 1) != 0; // Ryujinx BufferUsageFlags.Write + if (write) s->stores = 1; + int slot = -1, kind = VKB_SB_TABLE; + if (cb == 0 && off >= sb_base && off < sb_base + 0x100 && !(off & 15)) slot = (int)(off - sb_base) / 16; + else if (cb == 0 && nvn_stage == 5 && off >= 0x210 && off < 0x310 && !(off & 15)) { slot = (int)(off - 0x210) / 16; kind = VKB_SB_UBO; } + else if (cb >= 3 && cb <= 16 && off >= 0 && off < 0x10000) { slot = (int)cb - 3; kind = VKB_SB_UBO_PTR; } + if (slot < 0 || slot >= VKB_NSBUF) { + static int said; if (said++ < 8) VLOG("shader %s: storage buffer address from c%ld+%#lx: unknown source", key, cb, off); + slot = 0xFF; + } + remap[b] = base + VKB_B_SBUF + (uint32_t)s->nsbuf; + s->sbuf[s->nsbuf].slot = (uint8_t)slot; + s->sbuf[s->nsbuf].kind = (uint8_t)kind; + s->sbuf[s->nsbuf].write = (uint8_t)write; + s->sbuf[s->nsbuf++].off = (uint16_t)(kind == VKB_SB_UBO_PTR ? off : 0); + } + // textures: the handle is a word offset into the NVN driver bank c2 - combined textures at 0x20 (8 bytes per slot), + // separate textures at 0x168 (MacNativeGames' totk_xlate Service.cs) + if ((p = json_array(j, "textures"))) + for (; (p = json_next(p, &e)); p = e) { + long b = json_int(p, e, "binding", -1), h = json_int(p, e, "handle", -1) * 4, type = json_int(p, e, "type", 3); + if (b < 0 || b > 255 || s->ntex >= VKB_NTEX || h < 0x20) { s->unsupported = 1; continue; } + vkb_tex_ref *t = &s->tex[s->ntex]; + t->separate = h >= 0x168; + t->slot = (uint8_t)((h - (t->separate ? 0x168 : 0x20)) / 8); + t->type = (uint16_t)type; + remap[b] = base + VKB_B_TEX + (uint32_t)s->ntex++; + } + if ((p = json_array(j, "images"))) + for (; (p = json_next(p, &e)); p = e) { + long b = json_int(p, e, "binding", -1), h = json_int(p, e, "handle", -1) * 4, type = json_int(p, e, "type", 3); + if (b < 0 || b > 255 || s->nimg >= VKB_NIMG || h < 0x120) { s->unsupported = 1; continue; } + s->img[s->nimg] = (vkb_tex_ref){(uint8_t)((h - 0x120) / 8), 0, (uint16_t)type}; + s->stores = 1; // [temporary] images are taken as written (the reflection has no read-only mark) + remap[b] = base + VKB_B_IMG + (uint32_t)s->nimg++; + } + s->frag_outputs = (uint32_t)json_top_int(j, "frag_out_map", 0); + free(j); + // rewrite Binding decorations (OpDecorate %id Binding n: opcode 71, decoration 33) + size_t nw = drop_cull_distance(spv, nspv / 4); + nspv = nw * 4; + for (size_t i = 5; i < nw;) { + uint32_t op = spv[i] & 0xffff, wc = spv[i] >> 16; + if (!wc || i + wc > nw) break; + if (op == 71 && wc == 4 && spv[i + 2] == 33) { + uint32_t old = spv[i + 3]; + if (old > 255 || remap[old] == UINT32_MAX) { + VLOG("shader %s: binding %u not in its reflection", key, old); + s->unsupported = 1; + } else spv[i + 3] = remap[old]; + } + i += wc; + } + VkShaderStageFlagBits stage = nvn_stage == 0 ? VK_SHADER_STAGE_VERTEX_BIT : nvn_stage == 1 ? VK_SHADER_STAGE_FRAGMENT_BIT : VK_SHADER_STAGE_COMPUTE_BIT; + VkResult r = create_object(stage, spv, nspv, &s->obj); + if (r == VK_SUCCESS) { // input / output locations: vertex variants; fragment inputs no vertex shader writes (audit) + s->in_mask = vkb_spv_input_mask(spv, nspv / 4); + s->out_mask = vkb_spv_output_mask(spv, nspv / 4); + uint32_t reads = vkb_spv_sbuf_reads(spv, nspv / 4, base + VKB_B_SBUF, s->nsbuf); + for (int i = 0; i < s->nsbuf; i++) s->sbuf[i].read = (uint8_t)(reads >> i & 1); + } + if (stage == VK_SHADER_STAGE_VERTEX_BIT && r == VK_SUCCESS) { // kept for attribute variants (vkb_shader_variant) + s->spv = spv; + s->nspv_words = nspv / 4; + } else free(spv); + if (r != VK_SUCCESS) { VLOG("shader %s: vkCreateShadersEXT %d", key, (int)r); return 0; } + if (getenv("TOTK_VK_VALIDATION")) VLOG("shader %.12s = VkShaderEXT %#llx", key, (unsigned long long)(uintptr_t)s->obj); // names validation messages + s->vk_stage = stage; + return 1; +} + +// ---- shaders made ahead: creating a shader object compiles it in the driver, ~17% of the recording thread on the way +// into the world, where new shaders arrive in bursts (submissions held up to 2.3 s, Great Sky Island, 2026-10-24). +// Every shader loaded is listed (/used.txt: key and NVN stage, appended); at the first shader lookup of a run, +// PREWARM_THREADS low-priority threads create the listed ones that are not loaded yet. The first run fills the list; +// later runs find those shaders made (as SavageWind's pipeline list). The constants c1 come from the game's program +// at the first real lookup (c1_set). +enum { PREWARM_THREADS = 2 }; +static uint64_t g_prewarm_made, g_prewarm_hits; +uint64_t vkb_shader_load_ns; // shaders made when first drawn with (long submission log, backend_vk.c) +static char *g_list; // used.txt, read once +static size_t g_list_n; +static _Atomic size_t g_list_at; +static void list_add(const char *key, int stage) { // g_mu held + char path[1024]; + snprintf(path, sizeof path, "%s/used.txt", cache_dir()); + FILE *f = fopen(path, "ab"); + if (!f) return; + fprintf(f, "%s %d\n", key, stage); + fclose(f); +} +static void *prewarm_thread(void *unused) { + (void)unused; + pthread_setname_np("totk-shader-prewarm"); + SetThreadPriority(GetCurrentThread(), THREAD_PRIORITY_BELOW_NORMAL); + for (;;) { + size_t at = atomic_fetch_add(&g_list_at, 67); // one line: 64 hex digits, space, stage, newline (67 bytes) + if (at + 67 > g_list_n) break; + const char *l = g_list + at; + if (l[64] != ' ' || l[66] != '\n') continue; + char key[65]; + memcpy(key, l, 64); key[64] = 0; + int stage = l[65] - '0'; + uint32_t h = (uint32_t)strtoul(key + 56, NULL, 16) & 4095; + pthread_mutex_lock(&g_mu); + shentry *e = g_tab[h]; + while (e && strcmp(e->key, key)) e = e->next; + pthread_mutex_unlock(&g_mu); + if (e) continue; + shentry *n = calloc(1, sizeof *n); + if (!n) die("shader entry"); + memcpy(n->key, key, sizeof key); + n->sh.key = n->key; + if (!load(&n->sh, key, stage)) { free(n); continue; } + n->prewarmed = 1; + pthread_mutex_lock(&g_mu); + e = g_tab[h]; + while (e && strcmp(e->key, key)) e = e->next; + if (!e) { n->next = g_tab[h]; g_tab[h] = n; g_prewarm_made++; g_loaded++; } + pthread_mutex_unlock(&g_mu); + if (e) { vk.vkDestroyShaderEXT(vkc.device, n->sh.obj, NULL); free(n->sh.spv); free(n); } // loaded meanwhile + } + static _Atomic int left = PREWARM_THREADS; + if (atomic_fetch_sub(&left, 1) == 1) VLOG("shaders made ahead from the list: %llu", (unsigned long long)g_prewarm_made); + return NULL; +} +static void prewarm_start(void) { + const char *e = getenv("TOTK_SHADER_PREWARM"); // =0: shaders are made only when first drawn with (A/B) + if (e && *e == '0') return; + char path[1024]; + snprintf(path, sizeof path, "%s/used.txt", cache_dir()); + size_t n = 0; + if (!(g_list = read_file(path, &n))) return; + g_list_n = n; + VLOG("shader list: %zu shaders to make ahead", n / 67); + for (int i = 0; i < PREWARM_THREADS; i++) { + pthread_t t; + if (pthread_create(&t, NULL, prewarm_thread, NULL) == 0) pthread_detach(t); + } +} + +const vkb_shader *vkb_shader_get(const NVNshaderData *sd) { + const uint8_t *control = sd->control; + if (!control || !sd->data) return NULL; + uint32_t length, stage; + memcpy(&length, control + 0x704, 4); + memcpy(&stage, control + 0x714, 4); + if (!length || length > (16u << 20) || (stage != 0 && stage != 1 && stage != 5)) return NULL; + char key[65]; + sha256_hex(control, 0x880, (const void *)(uintptr_t)sd->data, length, key); + static pthread_once_t once = PTHREAD_ONCE_INIT; + pthread_once(&once, prewarm_start); + uint32_t h = (uint32_t)strtoul(key + 56, NULL, 16) & 4095; + pthread_mutex_lock(&g_mu); + shentry *e = g_tab[h]; + while (e && strcmp(e->key, key)) e = e->next; + if (!e) { + e = calloc(1, sizeof *e); + if (!e) die("shader entry"); + memcpy(e->key, key, sizeof key); + e->sh.key = e->key; + uint64_t t0 = runtime_now_ns(); + int ok = load(&e->sh, key, (int)stage); + vkb_shader_load_ns += runtime_now_ns() - t0; + if (ok) { + g_loaded++; + list_add(key, (int)stage); + } + else { + e->sh.obj = VK_NULL_HANDLE; + if (g_missing++ < 16) VLOG("shader %s (stage %u): no translation in %s", key, stage, cache_dir()); + } + e->next = g_tab[h]; + g_tab[h] = e; + } + if (e->sh.obj && !e->c1_set) { // loaded now or ahead (prewarm): the constants come with the game's program + e->c1_set = 1; + if (e->prewarmed) g_prewarm_hits++; + uint32_t c1off; + memcpy(&c1off, control + 0x700, 4); // the shader's constants: the code's tail (c1) + if (c1off < length) { // copied: the program's memory may be freed and reused while the key stays cached + uint8_t *c = malloc(length - c1off); + if (!c) die("shader constants"); + memcpy(c, (const uint8_t *)(uintptr_t)sd->data + c1off, length - c1off); + e->sh.c1 = c; + e->sh.c1_size = length - c1off; + } + } + pthread_mutex_unlock(&g_mu); + return e->sh.obj ? &e->sh : NULL; +} + +void vkb_shader_stats(uint64_t *loaded, uint64_t *missing) { *loaded = g_loaded; *missing = g_missing; } diff --git a/src/gfx/vkb_shader.h b/src/gfx/vkb_shader.h new file mode 100644 index 0000000..ccc4ae9 --- /dev/null +++ b/src/gfx/vkb_shader.h @@ -0,0 +1,52 @@ +// Shaders of the Vulkan NVN replayer (vkb_shader.c): translated SPIR-V as shader objects, universal descriptor layout. +#pragma once +#include + +#include "vkb.h" + +// Universal layout: per stage VKB_STAGE_BINDINGS bindings of set 0 (vertex from 0, fragment from 64; compute: own layout). +enum { + VKB_STAGE_BINDINGS = 64, + VKB_B_SUPPORT = 0, // Ryujinx support buffer + VKB_B_CBUF = 1, VKB_NCBUF = 17, // c0..c16 at +1..17 + VKB_B_SBUF = 18, VKB_NSBUF = 16, // storage buffers in the shader's order + VKB_B_TEX = 34, VKB_NTEX = 24, // combined textures in the shader's order + VKB_B_IMG = 58, VKB_NIMG = 4, // images in the shader's order +}; + +enum { VKB_SB_TABLE, VKB_SB_UBO, VKB_SB_UBO_PTR }; +typedef struct { uint8_t slot, separate; uint16_t type; } vkb_tex_ref; // NVN binding slot; Ryujinx SamplerType + +enum { VKB_VARIANTS = 8 }; +typedef struct { uint32_t intmask, sintmask; VkShaderEXT obj; } vkb_variant; + +typedef struct { + VkShaderEXT obj; + VkShaderStageFlagBits vk_stage; + int stage; // NVN: 0 vertex, 1 fragment, 5 compute + uint32_t cbuf_mask; // hardware constant buffer slots read (binding +1+slot): c3..c16 = game uniform + // buffer slot-3, c1 = the shader's own constants (c1 below), c0 = NVN driver bank + const uint8_t *c1; uint32_t c1_size; // code[control+0x700 .. length): the shader's constants (a copy) + uint64_t c1_rec; VkBuffer c1_buf; VkDeviceSize c1_off; // its copy in the submission being recorded (vkb_draw.c) + // storage buffers, binding +18+i. Where the address comes from (MacNativeGames' totk_xlate Service.cs): + // VKB_SB_TABLE: the NVN storage buffer slot (driver table in c0); VKB_SB_UBO: (compute) the uniform buffer slot, its + // range used as storage; VKB_SB_UBO_PTR: a 64-bit address stored in the game's uniform buffer `slot` at byte `off`; + // slot 0xFF: unknown (an empty buffer is bound) + struct { uint8_t slot, kind, write, read; uint16_t off; } sbuf[VKB_NSBUF]; int nsbuf; // write / read: stores to / loads from it + vkb_tex_ref tex[VKB_NTEX]; int ntex; // binding +34+i + vkb_tex_ref img[VKB_NIMG]; int nimg; // binding +58+i + uint32_t frag_outputs; // colour outputs written (bit per target) + int stores; // writes storage buffers or images (later work must wait for it) + int unsupported; // a resource outside the layout: draws with it are skipped + const char *key; // SHA-256 key of the translation (traces) + uint32_t *spv; size_t nspv_words; // vertex shaders: the module as loaded (bindings rewritten), for variants + uint32_t in_mask, out_mask; // input / output locations + vkb_variant variants[VKB_VARIANTS]; int nvariants; // integer-attribute variants made so far +} vkb_shader; + +const vkb_shader *vkb_shader_get(const NVNshaderData *sd); // NULL: no translation (logged) +// The vertex shader object for integer attributes at intmask (signed: sintmask): see vkb_spv.c. +VkShaderEXT vkb_shader_variant(const vkb_shader *vs, uint32_t intmask, uint32_t sintmask); +VkDescriptorSetLayout vkb_set_layout(int compute); +VkPipelineLayout vkb_pipeline_layout(int compute); +void vkb_shader_stats(uint64_t *loaded, uint64_t *missing); diff --git a/src/gfx/vkb_shadow.c b/src/gfx/vkb_shadow.c new file mode 100644 index 0000000..eb4a7a6 --- /dev/null +++ b/src/gfx/vkb_shadow.c @@ -0,0 +1,152 @@ +// Device memory for the copies of game memory pools (backend_vk.c, "pool shadows"): blocks of BLOCK bytes, cut into +// pieces first-fit (a pool larger than a block gets a block of its own). One allocation per pool would exhaust the +// driver's allocation count (~4096 on Windows) with TOTK's ~2900 pools. A freed piece is reused once the GPU has finished +// every submission recorded up to then. +#include +#include + +#include "compat/gfx_compat.h" +#include "host/hle.h" +#include "vkb.h" + +#define VLOG(...) LOG("[vk] " __VA_ARGS__) +enum { BLOCK = 64u << 20, ALIGN = 256 }; + +typedef struct { VkDeviceSize off, size; } span; +typedef struct { VkBuffer buf; VkDeviceMemory mem; VkDeviceSize size; span *free; int nfree, capfree; } block_t; +typedef struct { int block; VkDeviceSize off, size; uint64_t fence; } piece; // fence: freed, reusable when done +static block_t *g_blocks; +static int g_nblocks; +static piece *g_pieces; // handle - 1 -> piece +static int g_npieces, g_cappieces; +static int *g_pending; // freed handles waiting for their fence +static int g_npending, g_cappending; +static uint64_t g_bytes; // in use +static pthread_mutex_t g_mu = PTHREAD_MUTEX_INITIALIZER; + +int vkb_shadow_on(void) { // discrete GPUs: game memory is across PCIe from the GPU + static int v = -1; + if (v < 0) { + const char *e = getenv("TOTK_VK_POOL_SHADOW"); + v = e && *e ? *e != '0' : vkc.props.deviceType == VK_PHYSICAL_DEVICE_TYPE_DISCRETE_GPU; + VLOG("copies of game memory in device memory: %s", v ? "on (discrete GPU)" : "off (memory shared with the GPU)"); + } + return v; +} + +static int block_new(VkDeviceSize size) { + block_t b = {0}; + b.size = size; + VkBufferCreateInfo bi = {VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .size = size, + .usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT | VK_BUFFER_USAGE_UNIFORM_BUFFER_BIT | + VK_BUFFER_USAGE_STORAGE_BUFFER_BIT | VK_BUFFER_USAGE_INDEX_BUFFER_BIT | VK_BUFFER_USAGE_VERTEX_BUFFER_BIT | + VK_BUFFER_USAGE_INDIRECT_BUFFER_BIT | VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT}; + if (!vk_check(vk.vkCreateBuffer(vkc.device, &bi, NULL, &b.buf), "shadow buffer")) return -1; + VkMemoryRequirements req; + vk.vkGetBufferMemoryRequirements(vkc.device, b.buf, &req); + uint32_t type = UINT32_MAX; + for (uint32_t i = 0; i < vkc.memory.memoryTypeCount && type == UINT32_MAX; i++) + if ((req.memoryTypeBits & (1u << i)) && (vkc.memory.memoryTypes[i].propertyFlags & VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT) && + !(vkc.memory.memoryTypes[i].propertyFlags & VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT)) type = i; + VkMemoryAllocateFlagsInfo fl = {VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_FLAGS_INFO, .flags = VK_MEMORY_ALLOCATE_DEVICE_ADDRESS_BIT}; + VkMemoryAllocateInfo ai = {VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO, &fl, .allocationSize = req.size, .memoryTypeIndex = type}; + if (type == UINT32_MAX || vk.vkAllocateMemory(vkc.device, &ai, NULL, &b.mem) != VK_SUCCESS) { + vk.vkDestroyBuffer(vkc.device, b.buf, NULL); + VLOG("shadow memory: no %llu MiB of device memory", (unsigned long long)(size >> 20)); + return -1; + } + vk.vkBindBufferMemory(vkc.device, b.buf, b.mem, 0); + b.capfree = 16; + b.free = malloc(sizeof *b.free * (size_t)b.capfree); + if (!b.free) die("shadow free list"); + b.free[0] = (span){0, size}; + b.nfree = 1; + g_blocks = realloc(g_blocks, sizeof *g_blocks * (size_t)(g_nblocks + 1)); + if (!g_blocks) die("shadow blocks"); + g_blocks[g_nblocks] = b; + return g_nblocks++; +} +static void span_free(block_t *b, VkDeviceSize off, VkDeviceSize size) { // back into the sorted list, merged + int i = 0; + while (i < b->nfree && b->free[i].off < off) i++; + if (b->nfree == b->capfree) { + b->capfree *= 2; + b->free = realloc(b->free, sizeof *b->free * (size_t)b->capfree); + if (!b->free) die("shadow free list"); + } + memmove(&b->free[i + 1], &b->free[i], sizeof *b->free * (size_t)(b->nfree - i)); + b->free[i] = (span){off, size}; + b->nfree++; + if (i + 1 < b->nfree && b->free[i].off + b->free[i].size == b->free[i + 1].off) { + b->free[i].size += b->free[i + 1].size; + memmove(&b->free[i + 1], &b->free[i + 2], sizeof *b->free * (size_t)(b->nfree - i - 2)); + b->nfree--; + } + if (i > 0 && b->free[i - 1].off + b->free[i - 1].size == b->free[i].off) { + b->free[i - 1].size += b->free[i].size; + memmove(&b->free[i], &b->free[i + 1], sizeof *b->free * (size_t)(b->nfree - i - 1)); + b->nfree--; + } +} +static void reclaim(void) { // freed pieces whose GPU work is done + for (int k = 0; k < g_npending;) { + piece *p = &g_pieces[g_pending[k] - 1]; + if (!mtl_wait(p->fence, 0)) { k++; continue; } + span_free(&g_blocks[p->block], p->off, p->size); + p->size = 0; + g_pending[k] = g_pending[--g_npending]; + } +} + +uint32_t vkb_shadow_alloc(VkDeviceSize size, VkBuffer *buf, VkDeviceSize *off) { + size = (size + ALIGN - 1) & ~(VkDeviceSize)(ALIGN - 1); + pthread_mutex_lock(&g_mu); + reclaim(); + int bi = -1; + VkDeviceSize at = 0; + for (int i = 0; i < g_nblocks && bi < 0; i++) + for (int k = 0; k < g_blocks[i].nfree; k++) + if (g_blocks[i].free[k].size >= size) { + bi = i; at = g_blocks[i].free[k].off; + g_blocks[i].free[k].off += size; g_blocks[i].free[k].size -= size; + if (!g_blocks[i].free[k].size) { + memmove(&g_blocks[i].free[k], &g_blocks[i].free[k + 1], sizeof g_blocks[i].free[0] * (size_t)(g_blocks[i].nfree - k - 1)); + g_blocks[i].nfree--; + } + break; + } + if (bi < 0) { + bi = block_new(size > BLOCK ? size : BLOCK); + if (bi < 0) { pthread_mutex_unlock(&g_mu); return 0; } + at = 0; + g_blocks[bi].free[0].off = size; g_blocks[bi].free[0].size -= size; + if (!g_blocks[bi].free[0].size) g_blocks[bi].nfree = 0; + } + if (g_npieces == g_cappieces) { + g_cappieces = g_cappieces ? g_cappieces * 2 : 1024; + g_pieces = realloc(g_pieces, sizeof *g_pieces * (size_t)g_cappieces); + if (!g_pieces) die("shadow pieces"); + } + g_pieces[g_npieces] = (piece){bi, at, size, 0}; + uint32_t h = (uint32_t)++g_npieces; + g_bytes += size; + *buf = g_blocks[bi].buf; + *off = at; + pthread_mutex_unlock(&g_mu); + return h; +} +void vkb_shadow_free(uint32_t h) { + if (!h) return; + pthread_mutex_lock(&g_mu); + piece *p = &g_pieces[h - 1]; + p->fence = mtl_last_fence() + 1; // the submission being recorded may still use it + g_bytes -= p->size; + if (g_npending == g_cappending) { + g_cappending = g_cappending ? g_cappending * 2 : 256; + g_pending = realloc(g_pending, sizeof *g_pending * (size_t)g_cappending); + if (!g_pending) die("shadow pending"); + } + g_pending[g_npending++] = (int)h; + pthread_mutex_unlock(&g_mu); +} +uint64_t vkb_shadow_bytes(void) { return g_bytes; } diff --git a/src/gfx/vkb_spv.c b/src/gfx/vkb_spv.c new file mode 100644 index 0000000..691c2d9 --- /dev/null +++ b/src/gfx/vkb_spv.c @@ -0,0 +1,166 @@ +// SPIR-V rewrites of translated shaders for the bind state (milestone 3.3 leftovers, the first part of the variant +// service). Vertex attributes of integer formats: Maxwell hands a shader the attribute's raw bits, and TOTK's skinned +// vertex shaders read bone indices that way (floatBitsToInt(attr) * 48). The default translation declares every input +// vec4 float, so the attribute had to be given as a scaled format (5 -> 5.0f) and the bone index was garbage. Here the +// input becomes uvec4 / ivec4 (the attribute is then given in its own integer format) and every load from it is +// bitcast back to float: the rest of the shader is unchanged and sees the raw bits, as on the hardware. +#include +#include + +#include "vkb_spv.h" + +enum { OP_TYPE_INT = 21, OP_TYPE_FLOAT = 22, OP_TYPE_VECTOR = 23, OP_TYPE_POINTER = 32, OP_VARIABLE = 59, OP_LOAD = 61, + OP_ACCESS_CHAIN = 65, OP_IN_BOUNDS_ACCESS_CHAIN = 66, OP_DECORATE = 71, OP_BITCAST = 124 }; +enum { DEC_LOCATION = 30, SC_INPUT = 1 }; + +typedef struct { uint32_t *w; size_t n, cap; } words; +static void put(words *o, uint32_t v) { + if (o->n == o->cap) { + o->cap = o->cap ? o->cap * 2 : 1024; + o->w = realloc(o->w, o->cap * sizeof *o->w); + if (!o->w) abort(); + } + o->w[o->n++] = v; +} +static void put_ins(words *o, uint32_t op, int n, const uint32_t *ops) { + put(o, (uint32_t)(n + 1) << 16 | op); + for (int i = 0; i < n; i++) put(o, ops[i]); +} + +int vkb_spv_int_inputs(const uint32_t *in, size_t nw, uint32_t locmask, uint32_t signedmask, uint32_t **out, size_t *outw) { + if (nw < 5 || !locmask) return 0; + uint32_t bound = in[3]; + // ids by kind (0: absent) + uint32_t f32 = 0, vec4f = 0, i32[2] = {0, 0}, ivec4[2] = {0, 0}; // [0] unsigned, [1] signed + uint32_t *loc_of = calloc(bound, sizeof *loc_of); // id -> location + 1 (decorated) + uint32_t *ptr_pointee = calloc(bound, sizeof *ptr_pointee); // Input pointer type id -> pointee + uint8_t *var = calloc(bound, 1); // id -> 1 + signedness of a patched input + uint8_t *chain = calloc(bound, 1); // access chain result -> 1 + signedness + if (!loc_of || !ptr_pointee || !var || !chain) abort(); + int ok = 1, patched = 0; + for (size_t i = 5; i < nw;) { + uint32_t op = in[i] & 0xffff, wc = in[i] >> 16; + if (!wc || i + wc > nw) { ok = 0; break; } + const uint32_t *a = in + i + 1; + if (op == OP_DECORATE && wc >= 4 && a[1] == DEC_LOCATION && a[0] < bound) loc_of[a[0]] = a[2] + 1; + else if (op == OP_TYPE_FLOAT && a[1] == 32) f32 = a[0]; + else if (op == OP_TYPE_INT && a[1] == 32) i32[a[2] ? 1 : 0] = a[0]; + else if (op == OP_TYPE_VECTOR && a[2] == 4) { + if (a[1] == f32) vec4f = a[0]; + else if (a[1] == i32[0] && i32[0]) ivec4[0] = a[0]; + else if (a[1] == i32[1] && i32[1]) ivec4[1] = a[0]; + } else if (op == OP_TYPE_POINTER && a[1] == SC_INPUT && a[0] < bound) ptr_pointee[a[0]] = a[2]; + else if (op == OP_VARIABLE && a[2] == SC_INPUT && a[1] < bound && loc_of[a[1]]) { + uint32_t loc = loc_of[a[1]] - 1; + if (loc < 32 && (locmask >> loc & 1)) { + if (a[0] >= bound || ptr_pointee[a[0]] != vec4f || !vec4f) ok = 0; // only vec4 float inputs + else { var[a[1]] = (uint8_t)(1 + (signedmask >> loc & 1)); patched++; } + } + } + i += wc; + } + if (!ok || !patched) { free(loc_of); free(ptr_pointee); free(var); free(chain); return 0; } + // new ids: missing integer scalar / vector types, Input pointers to them + uint32_t next = bound, new_i32[2] = {0, 0}, new_ivec4[2] = {0, 0}, p_vec[2], p_scalar[2]; + for (int s = 0; s < 2; s++) { + if (!i32[s]) i32[s] = new_i32[s] = next++; + if (!ivec4[s]) ivec4[s] = new_ivec4[s] = next++; + p_vec[s] = next++; + p_scalar[s] = next++; + } + words o = {0}; + for (int k = 0; k < 5; k++) put(&o, in[k]); + int types_done = 0; + for (size_t i = 5; i < nw;) { + uint32_t op = in[i] & 0xffff, wc = in[i] >> 16; + const uint32_t *a = in + i + 1; + if (op == OP_VARIABLE && a[1] < bound && var[a[1]]) { + if (!types_done) { // the new types, before the first patched input (all they refer to is declared above) + for (int s = 0; s < 2; s++) { + if (new_i32[s]) put_ins(&o, OP_TYPE_INT, 3, (uint32_t[]){i32[s], 32, (uint32_t)s}); + if (new_ivec4[s]) put_ins(&o, OP_TYPE_VECTOR, 3, (uint32_t[]){ivec4[s], i32[s], 4}); + put_ins(&o, OP_TYPE_POINTER, 3, (uint32_t[]){p_vec[s], SC_INPUT, ivec4[s]}); + put_ins(&o, OP_TYPE_POINTER, 3, (uint32_t[]){p_scalar[s], SC_INPUT, i32[s]}); + } + types_done = 1; + } + int s = var[a[1]] - 1; + put(&o, in[i]); + put(&o, p_vec[s]); + for (uint32_t k = 2; k < wc; k++) put(&o, in[i + k]); + } else if ((op == OP_ACCESS_CHAIN || op == OP_IN_BOUNDS_ACCESS_CHAIN) && a[2] < bound && var[a[2]]) { + int s = var[a[2]] - 1; + if (a[1] < bound) chain[a[1]] = (uint8_t)(1 + s); + if (wc != 5) ok = 0; // a component of the vector: one index + put(&o, in[i]); + put(&o, p_scalar[s]); + for (uint32_t k = 2; k < wc; k++) put(&o, in[i + k]); + } else if (op == OP_LOAD && a[2] < bound && (var[a[2]] || chain[a[2]])) { + int whole = var[a[2]] != 0, s = (whole ? var[a[2]] : chain[a[2]]) - 1; + uint32_t t = whole ? ivec4[s] : i32[s], raw = next++; + put(&o, in[i]); + put(&o, t); + put(&o, raw); + for (uint32_t k = 3; k < wc; k++) put(&o, in[i + k]); + put_ins(&o, OP_BITCAST, 3, (uint32_t[]){a[0], a[1], raw}); // the original result id, as float again + } else { + for (uint32_t k = 0; k < wc; k++) put(&o, in[i + k]); + } + i += wc; + } + free(loc_of); free(ptr_pointee); free(var); free(chain); + if (!ok) { free(o.w); return 0; } + o.w[3] = next; + *out = o.w; + *outw = o.n; + return patched; +} + +// Storage buffers the shader reads: bindings first..first+n-1, bit i when a load or an atomic (they read too) goes through a +// pointer into binding first+i. The translator's reflection only tells writes; a draw that only writes a buffer need not +// wait for earlier draws' stores to it (vkb_draw.c). Pointers are followed through access chains and copies, which in +// the translator's SPIR-V come before their uses. +uint32_t vkb_spv_sbuf_reads(const uint32_t *in, size_t nw, uint32_t first, int n) { + if (nw < 5 || n <= 0) return 0; + uint32_t bound = in[3], mask = 0; + int8_t *of = malloc(bound); // id -> storage buffer index, -1: none + uint32_t *bind = calloc(bound, sizeof *bind); + if (!of || !bind) abort(); + memset(of, -1, bound); + for (size_t i = 5; i < nw;) { + uint32_t op = in[i] & 0xffff, wc = in[i] >> 16; + if (!wc || i + wc > nw) break; + const uint32_t *a = in + i + 1; + if (op == OP_DECORATE && wc >= 4 && a[1] == 33 && a[0] < bound) bind[a[0]] = a[2] + 1; // Binding + else if (op == OP_VARIABLE && wc >= 4 && a[1] < bound && bind[a[1]] > first && bind[a[1]] - 1 - first < (uint32_t)n) + of[a[1]] = (int8_t)(bind[a[1]] - 1 - first); + else if ((op == OP_ACCESS_CHAIN || op == OP_IN_BOUNDS_ACCESS_CHAIN || op == 67 || op == 83) && wc >= 4 && a[1] < bound && a[2] < bound) + of[a[1]] = of[a[2]]; // OpPtrAccessChain 67, OpCopyObject 83 + else if ((op == OP_LOAD || (op >= 227 && op <= 242 && op != 228)) && wc >= 4 && a[2] < bound && of[a[2]] >= 0) + mask |= 1u << of[a[2]]; // OpLoad, OpAtomic* except OpAtomicStore + i += wc; + } + free(of); + free(bind); + return mask; +} +static uint32_t location_mask(const uint32_t *in, size_t nw, uint32_t storage); +uint32_t vkb_spv_input_mask(const uint32_t *in, size_t nw) { return location_mask(in, nw, SC_INPUT); } +uint32_t vkb_spv_output_mask(const uint32_t *in, size_t nw) { return location_mask(in, nw, 3); } // StorageClass Output +static uint32_t location_mask(const uint32_t *in, size_t nw, uint32_t storage) { + if (nw < 5) return 0; + uint32_t bound = in[3], mask = 0; + uint32_t *loc_of = calloc(bound, sizeof *loc_of); + if (!loc_of) abort(); + for (size_t i = 5; i < nw;) { + uint32_t op = in[i] & 0xffff, wc = in[i] >> 16; + if (!wc || i + wc > nw) break; + const uint32_t *a = in + i + 1; + if (op == OP_DECORATE && wc >= 4 && a[1] == DEC_LOCATION && a[0] < bound) loc_of[a[0]] = a[2] + 1; + else if (op == OP_VARIABLE && a[2] == storage && a[1] < bound && loc_of[a[1]] && loc_of[a[1]] <= 32) + mask |= 1u << (loc_of[a[1]] - 1); + i += wc; + } + free(loc_of); + return mask; +} diff --git a/src/gfx/vkb_spv.h b/src/gfx/vkb_spv.h new file mode 100644 index 0000000..ef82537 --- /dev/null +++ b/src/gfx/vkb_spv.h @@ -0,0 +1,14 @@ +// SPIR-V rewrites of translated shaders for the bind state (vkb_spv.c). +#pragma once +#include +#include + +// Vertex inputs at the locations in locmask (vec4 float in the translation) become uvec4, or ivec4 where signedmask +// has the bit; loads from them are bitcast back to float. *out: a new module (malloc) of *outw words. Returns the +// number of inputs changed (0: nothing to do or a shape not handled - the caller keeps the original). +int vkb_spv_int_inputs(const uint32_t *in, size_t nwords, uint32_t locmask, uint32_t signedmask, uint32_t **out, size_t *outw); +// Locations of the module's Input variables (bit per location). +uint32_t vkb_spv_input_mask(const uint32_t *in, size_t nwords); +// Locations of the module's Output variables (bit per location). +uint32_t vkb_spv_sbuf_reads(const uint32_t *in, size_t nw, uint32_t first, int n); // storage buffers read (bit per index) +uint32_t vkb_spv_output_mask(const uint32_t *in, size_t nwords); diff --git a/src/gfx/vkb_tex.c b/src/gfx/vkb_tex.c new file mode 100644 index 0000000..4bc4518 --- /dev/null +++ b/src/gfx/vkb_tex.c @@ -0,0 +1,1509 @@ +// NVN textures as Vulkan images (see vkb.h). Contents uploaded from game memory come with drawing (milestone 3.3 +// step 4/5); here images exist for render targets, clears, copies and presentation. +// The NVN format table follows LASRecomp runtime/nvn/vulkan/textures.c (MIT, refs/LASRecomp): verified on a Switch +// game on Vulkan. Changes: ASTC, which desktop GPUs lack, becomes BC7 / BC1 from the cache made at build time +// (vkb_bc7.c), else RGBA8 decoded on upload; DEPTH24_STENCIL8 is D24S8 where the GPU can attach it, D32S8 otherwise +// (AMD, Steam Deck). +#include +#include +#include +#include + +#include "compat/gfx_compat.h" +#include "host/hle.h" +#include "vkb.h" +#include "nvn/nvn_dirty.h" +#include "tex/astc.h" +#include "tex/bc7.h" +#include "tex/nvn_deswizzle.h" +#include "nvn/totk_vpool.h" +#include "vkb_audit.h" +#include "vkb_lock.h" + +#define VLOG(...) LOG("[vk] " __VA_ARGS__) + +static int supports(VkFormat f, VkFormatFeatureFlags want) { + VkFormatProperties p; + vk.vkGetPhysicalDeviceFormatProperties(vkc.gpu, f, &p); + return (p.optimalTilingFeatures & want) == want; +} + +// Depth buffers are kept as 32-bit float (D16 and D24 too). TOTK classifies lighting tiles with an EQUAL depth test: +// one pass writes gl_FragDepth = material id, the lighting passes draw quads at the same z. In a UNORM depth buffer +// the two values reach the buffer through different conversions (fragment depth export and rasterised z), whose +// rounding may differ by one step on PC GPUs - the tiles failed the test and the world was unlit (2026-10-23, D16). +// Maxwell rounds both the same way; a float buffer stores both exactly. The precision only grows. +static VkFormat depth24_stencil8(void) { + static VkFormat f; + if (!f) { + VkFormatFeatureFlags need = VK_FORMAT_FEATURE_DEPTH_STENCIL_ATTACHMENT_BIT | VK_FORMAT_FEATURE_SAMPLED_IMAGE_BIT | + VK_FORMAT_FEATURE_TRANSFER_SRC_BIT | VK_FORMAT_FEATURE_TRANSFER_DST_BIT; + f = supports(VK_FORMAT_D32_SFLOAT_S8_UINT, need) ? VK_FORMAT_D32_SFLOAT_S8_UINT : VK_FORMAT_D24_UNORM_S8_UINT; + VLOG("DEPTH24_STENCIL8 -> %s", f == VK_FORMAT_D24_UNORM_S8_UINT ? "D24_UNORM_S8_UINT" : "D32_SFLOAT_S8_UINT"); + } + return f; +} + +VkFormat vkb_format(int f, VkImageAspectFlags *aspect, int *astc) { + if (aspect) *aspect = f == 0x31 ? VK_IMAGE_ASPECT_STENCIL_BIT + : f == 0x35 || f == 0x36 ? VK_IMAGE_ASPECT_DEPTH_BIT | VK_IMAGE_ASPECT_STENCIL_BIT + : f >= 0x32 && f <= 0x34 ? VK_IMAGE_ASPECT_DEPTH_BIT : VK_IMAGE_ASPECT_COLOR_BIT; + if (astc) *astc = f >= 0x79 && f <= 0x94; + if (f >= 0x79 && f <= 0x86) return VK_FORMAT_R8G8B8A8_UNORM; // ASTC LDR + if (f >= 0x87 && f <= 0x94) return VK_FORMAT_R8G8B8A8_SRGB; // ASTC sRGB + switch (f) { + case 0x01: return VK_FORMAT_R8_UNORM; case 0x02: return VK_FORMAT_R8_SNORM; + case 0x03: return VK_FORMAT_R8_UINT; case 0x04: return VK_FORMAT_R8_SINT; + case 0x05: return VK_FORMAT_R16_SFLOAT; case 0x06: return VK_FORMAT_R16_UNORM; + case 0x07: return VK_FORMAT_R16_SNORM; case 0x08: return VK_FORMAT_R16_UINT; + case 0x09: return VK_FORMAT_R16_SINT; case 0x0A: return VK_FORMAT_R32_SFLOAT; + case 0x0B: return VK_FORMAT_R32_UINT; case 0x0C: return VK_FORMAT_R32_SINT; + case 0x0D: return VK_FORMAT_R8G8_UNORM; case 0x0E: return VK_FORMAT_R8G8_SNORM; + case 0x0F: return VK_FORMAT_R8G8_UINT; case 0x10: return VK_FORMAT_R8G8_SINT; + case 0x11: return VK_FORMAT_R16G16_SFLOAT; case 0x12: return VK_FORMAT_R16G16_UNORM; + case 0x13: return VK_FORMAT_R16G16_SNORM; case 0x14: return VK_FORMAT_R16G16_UINT; + case 0x15: return VK_FORMAT_R16G16_SINT; case 0x16: return VK_FORMAT_R32G32_SFLOAT; + case 0x17: return VK_FORMAT_R32G32_UINT; case 0x18: return VK_FORMAT_R32G32_SINT; + case 0x19: return VK_FORMAT_R8G8B8A8_UNORM; // RGB8: expanded on upload + case 0x25: return VK_FORMAT_R8G8B8A8_UNORM; case 0x26: return VK_FORMAT_R8G8B8A8_SNORM; + case 0x27: return VK_FORMAT_R8G8B8A8_UINT; case 0x28: return VK_FORMAT_R8G8B8A8_SINT; + case 0x29: return VK_FORMAT_R16G16B16A16_SFLOAT; case 0x2A: return VK_FORMAT_R16G16B16A16_UNORM; + case 0x2B: return VK_FORMAT_R16G16B16A16_SNORM; case 0x2C: return VK_FORMAT_R16G16B16A16_UINT; + case 0x2D: return VK_FORMAT_R16G16B16A16_SINT; case 0x2E: return VK_FORMAT_R32G32B32A32_SFLOAT; + case 0x2F: return VK_FORMAT_R32G32B32A32_UINT; case 0x30: return VK_FORMAT_R32G32B32A32_SINT; + case 0x31: return VK_FORMAT_S8_UINT; case 0x32: return VK_FORMAT_D32_SFLOAT; // D16: see depth24_stencil8 + case 0x33: return VK_FORMAT_D32_SFLOAT; case 0x34: return VK_FORMAT_D32_SFLOAT; + case 0x35: return depth24_stencil8(); case 0x36: return VK_FORMAT_D32_SFLOAT_S8_UINT; + case 0x37: case 0x38: return VK_FORMAT_R8G8B8A8_SRGB; + case 0x39: return VK_FORMAT_R4G4B4A4_UNORM_PACK16; + case 0x3A: return VK_FORMAT_R5G6B5_UNORM_PACK16; + case 0x3B: return VK_FORMAT_R5G5B5A1_UNORM_PACK16; + case 0x3C: return VK_FORMAT_R5G6B5_UNORM_PACK16; + case 0x3D: return VK_FORMAT_A2B10G10R10_UNORM_PACK32; + case 0x3E: return VK_FORMAT_A2B10G10R10_UINT_PACK32; + case 0x3F: return VK_FORMAT_B10G11R11_UFLOAT_PACK32; + case 0x40: return VK_FORMAT_E5B9G9R9_UFLOAT_PACK32; + case 0x41: case 0x42: return VK_FORMAT_BC1_RGBA_UNORM_BLOCK; + case 0x43: return VK_FORMAT_BC2_UNORM_BLOCK; case 0x44: return VK_FORMAT_BC3_UNORM_BLOCK; + case 0x45: case 0x46: return VK_FORMAT_BC1_RGBA_SRGB_BLOCK; + case 0x47: return VK_FORMAT_BC2_SRGB_BLOCK; case 0x48: return VK_FORMAT_BC3_SRGB_BLOCK; + case 0x49: return VK_FORMAT_BC4_UNORM_BLOCK; case 0x4A: return VK_FORMAT_BC4_SNORM_BLOCK; + case 0x4B: return VK_FORMAT_BC5_UNORM_BLOCK; case 0x4C: return VK_FORMAT_BC5_SNORM_BLOCK; + case 0x4D: return VK_FORMAT_BC7_UNORM_BLOCK; case 0x4E: return VK_FORMAT_BC7_SRGB_BLOCK; + case 0x4F: return VK_FORMAT_BC6H_SFLOAT_BLOCK; case 0x50: return VK_FORMAT_BC6H_UFLOAT_BLOCK; + case 0x6D: return VK_FORMAT_R8G8B8A8_UNORM; case 0x71: return VK_FORMAT_R16G16B16A16_SFLOAT; + case 0x76: return VK_FORMAT_R32G32B32A32_SFLOAT; + case 0x95: case 0x96: case 0x97: return VK_FORMAT_R5G5B5A1_UNORM_PACK16; + case 0x98: case 0x99: return VK_FORMAT_B8G8R8A8_UNORM; + case 0x9A: case 0x9B: return VK_FORMAT_B8G8R8A8_SRGB; + default: return VK_FORMAT_UNDEFINED; + } +} + +// ---------------- images: one per memory and shape ---------------- +// The game re-initialises texture objects over the same memory with the same shape (its render targets every frame, +// copies of objects, pool entries of finalised ones): on the console that is one texture that keeps its contents. Each +// serial used to get an image of its own: ~30 new images a frame on the Great Sky Island (~8 ms of the replay thread in +// vkCreateImage / vkAllocateMemory), each starting empty where the console kept the previous frame's contents +// (2026-10-24; SavageWind found the same in BotW). Serials now map to entries shared by memory and shape. An entry no +// serial holds stays aside for RECYCLE_FRAMES presents (the next frame's object takes it back, contents included) and is +// then destroyed once the GPU is done with it. Taken back after the CPU wrote its memory meanwhile, it uploads again. +#define VIEWS 16 // per texture: levels of a mip chain drawn one by one, layers, formats, shader types +typedef struct entry { + vkb_image im; VkImageView view[VIEWS]; uint8_t view_key[VIEWS][17]; int nviews; + int refs, shaped; // serials holding it; in the shape index (it has memory) + uint64_t released, released_stamp; // aside since this present / dirty-tracking generation + uint8_t has_layout, layout[8]; + struct entry *shape_next, *all_next, **all_pprev; +} entry; +typedef struct { uint64_t serial; entry *e; } tslot; // serial 0: free; e NULL: removed +static tslot *g_tab; +static uint32_t g_cap, g_n, g_tomb; // live serials, removed slots until the next grow +static entry *g_all; // every entry (dumps, alias logs) +enum { SHAPES = 4096, RECYCLE_FRAMES = 120, RECYCLE_MAX = 512 }; +static entry *g_shape[SHAPES]; +static entry *g_aside[RECYCLE_MAX]; +static int g_naside; +static uint64_t g_images_made, g_images_shared, g_images_taken_back; +static pthread_mutex_t g_mu = PTHREAD_MUTEX_INITIALIZER; + +static tslot *slot(uint64_t serial) { // g_mu held; g_cap a power of two + uint32_t h = (uint32_t)(serial * 0x9E3779B97F4A7C15ull >> 40) & (g_cap - 1); + while (g_tab[h].serial && g_tab[h].serial != serial) h = (h + 1) & (g_cap - 1); + return &g_tab[h]; +} +static void grow(void) { + tslot *old = g_tab; + uint32_t n = g_cap; + g_cap = g_cap ? g_cap * 2 : 1024; + g_tab = calloc(g_cap, sizeof *g_tab); + if (!g_tab) die("image table"); + for (uint32_t i = 0; i < n; i++) if (old[i].serial && old[i].e) *slot(old[i].serial) = old[i]; + g_tomb = 0; + free(old); +} +static uint32_t shape_of(const void *mem, const nvn_texdesc *d) { + uint64_t h = (uint64_t)(uintptr_t)mem * 0x9E3779B97F4A7C15ull; + h ^= ((uint64_t)(uint32_t)d->format << 32 | d->width) * 0xC2B2AE3D27D4EB4Full; + h ^= ((uint64_t)d->height << 32 | d->levels) * 0x165667B19E3779F9ull; + return (uint32_t)(h >> 52) & (SHAPES - 1); +} +static int same_shape(const entry *e, const tex_t *t) { + const nvn_texdesc *a = &e->im.d, *b = &t->d; + return e->im.storage == t->mem && a->target == b->target && a->format == b->format && a->flags == b->flags && + a->samples == b->samples && a->width == b->width && a->height == b->height && a->depth == b->depth && + a->levels == b->levels && a->stride == b->stride && a->pkg_block_h == b->pkg_block_h && a->swizzle_r == b->swizzle_r && + a->swizzle_g == b->swizzle_g && a->swizzle_b == b->swizzle_b && a->swizzle_a == b->swizzle_a && + e->has_layout == t->has_layout && (!t->has_layout || !memcmp(e->layout, t->layout, sizeof e->layout)); +} +static void entry_drop(entry *e) { // g_mu held: out of the indexes, destroyed when the GPU is done with it + if (e->shaped) { + entry **p = &g_shape[shape_of(e->im.storage, &e->im.d)]; + while (*p && *p != e) p = &(*p)->shape_next; + if (*p) *p = e->shape_next; + } + *e->all_pprev = e->all_next; + if (e->all_next) e->all_next->all_pprev = e->all_pprev; + for (int i = 0; i < e->nviews; i++) vkb_destroy_later(e->view[i], VK_NULL_HANDLE, VK_NULL_HANDLE); + vkb_destroy_later(VK_NULL_HANDLE, e->im.img, e->im.mem); + free(e); +} +static void aside_purge(void) { // g_mu held: entries aside too long (or too many) go + uint64_t now = vkb_frame(); + while (g_naside && (g_naside >= RECYCLE_MAX || g_aside[0]->released + RECYCLE_FRAMES < now)) { + entry *e = g_aside[0]; + memmove(g_aside, g_aside + 1, (size_t)--g_naside * sizeof g_aside[0]); + entry_drop(e); + } +} +void vkb_tex_image_stats(uint64_t *made, uint64_t *shared, uint64_t *taken_back) { + *made = g_images_made; *shared = g_images_shared; *taken_back = g_images_taken_back; +} + +static uint32_t memory_type(uint32_t bits, VkMemoryPropertyFlags want) { + for (uint32_t i = 0; i < vkc.memory.memoryTypeCount; i++) + if ((bits & (1u << i)) && (vkc.memory.memoryTypes[i].propertyFlags & want) == want) return i; + return UINT32_MAX; +} + +static int create(vkb_image *im, const tex_t *t) { + const nvn_texdesc *d = &t->d; + VkImageAspectFlags aspect; + int astc; + VkFormat f = vkb_format(d->format, &aspect, &astc); + if (f == VK_FORMAT_UNDEFINED || d->target == NVN_TEXTURE_TARGET_BUFFER) return 0; + nvn_texlayout L; + nvn_tex_layout(d, &L); + VkImageType type = d->target == NVN_TEXTURE_TARGET_3D ? VK_IMAGE_TYPE_3D + : d->target == NVN_TEXTURE_TARGET_1D || d->target == NVN_TEXTURE_TARGET_1D_ARRAY ? VK_IMAGE_TYPE_1D : VK_IMAGE_TYPE_2D; + uint32_t w = d->width ? d->width : 1, h = type == VK_IMAGE_TYPE_1D ? 1 : (d->height ? d->height : 1); + uint32_t depth = type == VK_IMAGE_TYPE_3D ? (d->depth ? d->depth : 1) : 1, layers = type == VK_IMAGE_TYPE_3D ? 1 : (L.layers ? L.layers : 1); + uint32_t levels = d->levels ? d->levels : 1; + int ms = d->target == NVN_TEXTURE_TARGET_2D_MULTISAMPLE || d->target == NVN_TEXTURE_TARGET_2D_MULTISAMPLE_ARRAY; + VkImageUsageFlags usage = VK_IMAGE_USAGE_SAMPLED_BIT | VK_IMAGE_USAGE_TRANSFER_SRC_BIT | VK_IMAGE_USAGE_TRANSFER_DST_BIT; + VkFormatFeatureFlags feat = 0; + { VkFormatProperties p; vk.vkGetPhysicalDeviceFormatProperties(vkc.gpu, f, &p); feat = p.optimalTilingFeatures; } + if (aspect & (VK_IMAGE_ASPECT_DEPTH_BIT | VK_IMAGE_ASPECT_STENCIL_BIT)) { + if (feat & VK_FORMAT_FEATURE_DEPTH_STENCIL_ATTACHMENT_BIT) usage |= VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT; + } else if (!astc) { + if (feat & VK_FORMAT_FEATURE_COLOR_ATTACHMENT_BIT) usage |= VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT; + if (feat & VK_FORMAT_FEATURE_STORAGE_IMAGE_BIT) usage |= VK_IMAGE_USAGE_STORAGE_BIT; + } + // a target may be sampled while bound (backend_vk.c pass_begin) + if (vkc.has_feedback_loop && (usage & (VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT | VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT))) + usage |= VK_IMAGE_USAGE_ATTACHMENT_FEEDBACK_LOOP_BIT_EXT; + VkImageCreateFlags flags = VK_IMAGE_CREATE_MUTABLE_FORMAT_BIT | VK_IMAGE_CREATE_EXTENDED_USAGE_BIT; // NVN views may reinterpret the format + int compressed = !astc && f >= VK_FORMAT_BC1_RGB_UNORM_BLOCK && f <= VK_FORMAT_BC7_SRGB_BLOCK; + if (compressed) flags |= VK_IMAGE_CREATE_BLOCK_TEXEL_VIEW_COMPATIBLE_BIT; // read a block as RGBA16UI / RG32UI (TOTK's GPU BC encoder) + if (type == VK_IMAGE_TYPE_2D && !ms && layers >= 6 && w == h) flags |= VK_IMAGE_CREATE_CUBE_COMPATIBLE_BIT; + if (type == VK_IMAGE_TYPE_3D) flags |= VK_IMAGE_CREATE_2D_ARRAY_COMPATIBLE_BIT; + VkImageCreateInfo ci = {VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO, .flags = flags, .imageType = type, .format = f, + .extent = {w, h, depth}, .mipLevels = levels, .arrayLayers = layers, + .samples = ms && d->samples > 1 ? (VkSampleCountFlagBits)d->samples : VK_SAMPLE_COUNT_1_BIT, + .tiling = VK_IMAGE_TILING_OPTIMAL, .usage = usage, .sharingMode = VK_SHARING_MODE_EXCLUSIVE, + .initialLayout = VK_IMAGE_LAYOUT_UNDEFINED}; + VkResult r = vk.vkCreateImage(vkc.device, &ci, NULL, &im->img); + if (r != VK_SUCCESS) { + static int said; + if (said++ < 16) VLOG("texture %llu (format %#x target %d %ux%ux%u, %u levels, %u layers): vkCreateImage %d", + (unsigned long long)t->serial, d->format, d->target, w, h, depth, levels, layers, (int)r); + return 0; + } + VkMemoryRequirements req; + vk.vkGetImageMemoryRequirements(vkc.device, im->img, &req); + // [temporary] one allocation per texture; a suballocator before the open world (thousands of textures) + VkMemoryAllocateInfo ai = {VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO, .allocationSize = req.size, + .memoryTypeIndex = memory_type(req.memoryTypeBits, VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT)}; + if (ai.memoryTypeIndex == UINT32_MAX || !vk_check(vk.vkAllocateMemory(vkc.device, &ai, NULL, &im->mem), "image memory")) { + vk.vkDestroyImage(vkc.device, im->img, NULL); + return 0; + } + vk.vkBindImageMemory(vkc.device, im->img, im->mem, 0); + im->serial = t->serial; im->d = *d; im->format = f; im->aspect = aspect; im->layout = VK_IMAGE_LAYOUT_UNDEFINED; + im->width = w; im->height = h; im->depth = depth; im->layers = layers; im->levels = levels; im->astc = astc; + im->storage = t->mem; + im->usage = usage; + im->compressed = compressed; + return 1; +} + +// ---- one memory, several sizes: TOTK's dynamic resolution re-creates its render targets at another size over the same +// memory (1600x896 <-> 1432x808 on the Great Sky Island when the camera turns to the sky and back). On the console the +// new objects see what was rendered last through the old ones; here each size is an image of its own, and coming back +// to a size took back its image as it was frames earlier: depth from when the camera faced the sky, so the sea of +// clouds was drawn over the whole scene, and the lighting of the title menu read another shot's targets - the scene went +// dark for 4-6 frames (2026-10-24). As GPU emulators' texture caches do, an image that a texture object starts using +// takes the contents of the image last written at the same memory in the same format but another size, scaled +// (vkb_tex_refresh, before its next use outside a pass). Off (TOTK_VK_SIZE_REFRESH=1 to try): TOTK also reuses that memory +// for small transient targets of the same format (its environment probe renders 64x64 into the main depth's memory), so +// "the last written" was often such a transient and the full-size target got its stretched copy; the drop-outs stayed. +// The fix is to keep the size fixed (no dynamic resolution). +static _Atomic uint64_t g_write_seq; +static uint64_t g_refreshes; +void vkb_image_written(vkb_image *im) { im->gpu_written = 1; im->write_seq = atomic_fetch_add(&g_write_seq, 1) + 1; } +static void refresh_mark(entry *e) { // g_mu held: e is starting to be used by a new texture object + entry *best = NULL; + for (entry *o = g_all; o; o = o->all_next) + if (o != e && o->im.storage == e->im.storage && o->im.format == e->im.format && o->im.gpu_written && o->im.aspect == e->im.aspect && + (o->im.width != e->im.width || o->im.height != e->im.height) && o->im.write_seq > e->im.write_seq && + (!best || o->im.write_seq > best->im.write_seq)) best = o; + e->im.refresh_from = best ? &best->im : NULL; +} +int vkb_tex_refresh(VkCommandBuffer cb, vkb_image *im) { + vkb_image *src = im->refresh_from; + im->refresh_from = NULL; + if (!src || vkb_in_pass()) return 0; + vkb_lock(&g_mu, LK_IMAGES); + int alive = 0; // the source may have been destroyed since it was marked + for (entry *o = g_all; o && !alive; o = o->all_next) alive = &o->im == src; + pthread_mutex_unlock(&g_mu); + if (!alive || src->write_seq <= im->write_seq) return 0; + VkFormatFeatureFlags need = VK_FORMAT_FEATURE_BLIT_SRC_BIT | VK_FORMAT_FEATURE_BLIT_DST_BIT; + VkFormatProperties fp; + vk.vkGetPhysicalDeviceFormatProperties(vkc.gpu, im->format, &fp); + if ((fp.optimalTilingFeatures & need) != need) return 0; + int linear = im->aspect == VK_IMAGE_ASPECT_COLOR_BIT && (fp.optimalTilingFeatures & VK_FORMAT_FEATURE_SAMPLED_IMAGE_FILTER_LINEAR_BIT); + vkb_image_use(cb, src, VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, VK_PIPELINE_STAGE_2_BLIT_BIT, VK_ACCESS_2_TRANSFER_READ_BIT); + vkb_image_use(cb, im, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, VK_PIPELINE_STAGE_2_BLIT_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT); + uint32_t layers = src->layers < im->layers ? src->layers : im->layers; + VkImageBlit r = {.srcSubresource = {im->aspect, 0, 0, layers}, .srcOffsets = {{0, 0, 0}, {(int32_t)src->width, (int32_t)src->height, 1}}, + .dstSubresource = {im->aspect, 0, 0, layers}, .dstOffsets = {{0, 0, 0}, {(int32_t)im->width, (int32_t)im->height, 1}}}; + vkb_checkpoint("refresh from another size", NULL, NULL); + vk.vkCmdBlitImage(cb, src->img, VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, im->img, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, 1, &r, + linear ? VK_FILTER_LINEAR : VK_FILTER_NEAREST); + im->gpu_written = 1; + im->write_seq = src->write_seq; + g_refreshes++; + static int said; + if (said++ < 8) VLOG("image %llu (%ux%u, format %#x) takes the contents of image %llu (%ux%u) at the same memory", (unsigned long long)im->serial, + im->width, im->height, im->d.format, (unsigned long long)src->serial, src->width, src->height); + return 1; +} + +// Recording threads look up the same serials draw after draw: a small cache per thread, valid for one submission (no +// image is freed while a submission is recorded: releases run on the translation thread between submissions). +typedef struct { uint64_t serial, sub; vkb_image *im; } icache_t; +static __thread icache_t *t_icache; +vkb_image *vkb_image_of(const tex_t *t) { + if (!t || !t->serial || !vkb_ready()) return NULL; + icache_t *c = NULL; + uint64_t sub = mtl_last_fence() + 1; + if (vkb_mt_recording()) { + if (!t_icache && !(t_icache = calloc(1024, sizeof *t_icache))) die("image cache"); + c = &t_icache[(t->serial * 0x9E3779B97F4A7C15ull) >> 54]; + if (c->serial == t->serial && c->sub == sub) return c->im; + } + vkb_lock(&g_mu, LK_IMAGES); + if ((g_n + g_tomb + 1) * 2 >= g_cap) grow(); + tslot *s = slot(t->serial); + if (s->serial == t->serial && s->e) { + vkb_image *found = &s->e->im; + pthread_mutex_unlock(&g_mu); + if (c) *c = (icache_t){t->serial, sub, found}; + return found; + } + entry *e = NULL; + static int no_share = -1; // TOTK_VK_NO_IMAGE_SHARE=1 (development A/B): an image per texture object, as before + if (no_share < 0) { const char *ns = getenv("TOTK_VK_NO_IMAGE_SHARE"); no_share = ns && *ns == '1'; } + if (t->mem && !no_share) for (e = g_shape[shape_of(t->mem, &t->d)]; e && !same_shape(e, t); e = e->shape_next) {} + if ((!e || !e->refs) && t->mem) { // a new or taken-back image over memory another live image uses: what differs (the game sees one texture there) + static int said; + static long long from = -1; // TOTK_SHAPE_LOG=: from that present on (development) + if (from < 0) { const char *f = getenv("TOTK_SHAPE_LOG"); from = f ? atoll(f) : 0; } + for (entry *o = g_all; o && from && vkb_frame() >= (uint64_t)from && said < 48; o = o->all_next) { + if (o == e || o->im.storage != t->mem || !o->refs) continue; + const nvn_texdesc *a = &o->im.d, *b = &t->d; + said++; + VLOG("[shape] texture %llu (%s) over the memory of image %llu (%p): target %u/%u format %#x/%#x flags %#x/%#x %ux%ux%u/%ux%ux%u " + "levels %u/%u samples %u/%u stride %u/%u block_h %u/%u swizzle %u%u%u%u/%u%u%u%u layout %d/%d%s", + (unsigned long long)t->serial, e ? "taken back" : "new", (unsigned long long)o->im.serial, t->mem, a->target, b->target, a->format, b->format, + a->flags, b->flags, a->width, a->height, a->depth, b->width, b->height, b->depth, a->levels, b->levels, a->samples, b->samples, + a->stride, b->stride, a->pkg_block_h, b->pkg_block_h, a->swizzle_r, a->swizzle_g, a->swizzle_b, a->swizzle_a, + b->swizzle_r, b->swizzle_g, b->swizzle_b, b->swizzle_a, o->has_layout, t->has_layout, + o->has_layout && t->has_layout && memcmp(o->layout, t->layout, sizeof o->layout) ? " (layouts differ)" : ""); + } + } + if (e && !e->refs) { // taken back from the side: written by the CPU meanwhile -> its new contents upload + int k = 0; + while (k < g_naside && g_aside[k] != e) k++; + if (k < g_naside) memmove(&g_aside[k], &g_aside[k + 1], (size_t)(--g_naside - k) * sizeof g_aside[0]); + nvn_texlayout L; + nvn_tex_layout(&e->im.d, &L); + if (nvn_dirty_tracked(e->im.storage, L.size) && nvn_dirty_since(e->im.storage, L.size, e->released_stamp)) { + e->im.gpu_written = 0; e->im.upload_stamp = 0; + } + g_images_taken_back++; + } else if (e) g_images_shared++; + if (!e) { + e = calloc(1, sizeof *e); + if (!e) die("image entry"); + uint64_t t0 = runtime_now_ns(); + int ok = create(&e->im, t); + vkb_step_add(PK_CREATE, t0); + if (!ok) { free(e); pthread_mutex_unlock(&g_mu); return NULL; } + e->has_layout = t->has_layout; + memcpy(e->layout, t->layout, sizeof e->layout); + if (t->mem) { uint32_t h = shape_of(t->mem, &t->d); e->shape_next = g_shape[h]; g_shape[h] = e; e->shaped = 1; } + e->all_next = g_all; e->all_pprev = &g_all; + if (g_all) g_all->all_pprev = &e->all_next; + g_all = e; + g_images_made++; + } + if (t->mem && getenv("TOTK_VK_SIZE_REFRESH")) refresh_mark(e); // [experiment, off] picks transient aliases (see above) + e->refs++; + if (s->serial == t->serial) g_tomb--; else s->serial = t->serial; + s->e = e; + g_n++; + pthread_mutex_unlock(&g_mu); + if (c) *c = (icache_t){t->serial, sub, &e->im}; + return &e->im; +} + +// ---- layouts as a chunk records them (vkb_mt.c): an image the chunk has not used yet is taken to be in the layout it +// is first needed in (no barrier recorded); the command buffer between chunks moves it there from where the previous +// chunk left it. One thread recording (t_trk NULL): the image's own layout, a barrier at every change. +static __thread vkb_trk *t_trk; +void vkb_trk_set(vkb_trk *t) { t_trk = t; } +vkb_trk *vkb_trk_get(void) { return t_trk; } +void vkb_trk_reset(vkb_trk *t) { + t->n = 0; + if (t->tab) memset(t->tab, 0xff, sizeof *t->tab * (size_t)t->tcap); +} +static uint32_t trk_hash(const vkb_image *im, int cap) { return (uint32_t)(((uint64_t)(uintptr_t)im * 0x9E3779B97F4A7C15ull) >> 40) & (uint32_t)(cap - 1); } +static vkb_trk_e *trk_find(vkb_trk *t, const vkb_image *im) { + if (!t->tcap) return NULL; + for (uint32_t h = trk_hash(im, t->tcap);; h = (h + 1) & (uint32_t)(t->tcap - 1)) { + int k = t->tab[h]; + if (k < 0) return NULL; + if (t->e[k].im == im) return &t->e[k]; + } +} +static vkb_trk_e *trk_add(vkb_trk *t, vkb_image *im, VkImageLayout l) { + if ((t->n + 1) * 2 > t->tcap) { // the index stays at most half full + int cap = t->tcap ? t->tcap * 2 : 256; + int *tab = malloc(sizeof *tab * (size_t)cap); + if (!tab) die("layout tracking"); + memset(tab, 0xff, sizeof *tab * (size_t)cap); + for (int k = 0; k < t->n; k++) { + uint32_t h = trk_hash(t->e[k].im, cap); + while (tab[h] >= 0) h = (h + 1) & (uint32_t)(cap - 1); + tab[h] = k; + } + free(t->tab); t->tab = tab; t->tcap = cap; + } + if (t->n == t->cap) { + t->cap = t->cap ? t->cap * 2 : 128; + if (!(t->e = realloc(t->e, sizeof *t->e * (size_t)t->cap))) die("layout tracking"); + } + vkb_trk_e *e = &t->e[t->n]; + *e = (vkb_trk_e){im, l, l}; + uint32_t h = trk_hash(im, t->tcap); + while (t->tab[h] >= 0) h = (h + 1) & (uint32_t)(t->tcap - 1); + t->tab[h] = t->n++; + return e; +} +int vkb_layout_is(vkb_image *im, VkImageLayout l) { + if (!t_trk) return im->layout == l; + vkb_trk_e *e = trk_find(t_trk, im); + if (!e) { trk_add(t_trk, im, l); return 1; } + return e->cur == l; +} +VkImageLayout vkb_layout(vkb_image *im) { + if (!t_trk) return im->layout; + vkb_trk_e *e = trk_find(t_trk, im); + return e ? e->cur : im->layout; +} + +void vkb_image_use(VkCommandBuffer cb, vkb_image *im, VkImageLayout layout, VkPipelineStageFlags2 stage, VkAccessFlags2 access) { + VkImageLayout old = im->layout; + if (t_trk) { + vkb_trk_e *e = trk_find(t_trk, im); + if (!e) { trk_add(t_trk, im, layout); return; } // its first use in the chunk: the barrier between chunks + old = e->cur; + e->cur = layout; + } else im->layout = layout; + // [temporary] every use waits for all earlier work on the image: precise per-pass barriers come with the pass tracker + VkImageMemoryBarrier2 b = {VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2, .srcStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + .srcAccessMask = VK_ACCESS_2_MEMORY_WRITE_BIT, .dstStageMask = stage, .dstAccessMask = access, .oldLayout = old, + .newLayout = layout, .srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, + .image = im->img, .subresourceRange = {im->aspect, 0, VK_REMAINING_MIP_LEVELS, 0, VK_REMAINING_ARRAY_LAYERS}}; + VkDependencyInfo di = {VK_STRUCTURE_TYPE_DEPENDENCY_INFO, .imageMemoryBarrierCount = 1, .pImageMemoryBarriers = &b}; + vk.vkCmdPipelineBarrier2(cb, &di); +} + +static VkComponentSwizzle swz(uint8_t s, VkComponentSwizzle identity) { + switch (s) { + case NVN_TEXTURE_SWIZZLE_ZERO: return VK_COMPONENT_SWIZZLE_ZERO; + case NVN_TEXTURE_SWIZZLE_ONE: return VK_COMPONENT_SWIZZLE_ONE; + case NVN_TEXTURE_SWIZZLE_R: return VK_COMPONENT_SWIZZLE_R; + case NVN_TEXTURE_SWIZZLE_G: return VK_COMPONENT_SWIZZLE_G; + case NVN_TEXTURE_SWIZZLE_B: return VK_COMPONENT_SWIZZLE_B; + case NVN_TEXTURE_SWIZZLE_A: return VK_COMPONENT_SWIZZLE_A; + default: return identity; + } +} + +// Formats without alpha read a = 1 (RGBX, RGB DXT1): NVN_FORMAT values, as MacNativeGames' format table. +static int alpha_one(int f) { + return f == 0x37 || f == 0x41 || f == 0x45 || (f >= 0x6D && f <= 0x78) || f == 0x96 || f == 0x98 || f == 0x9A; +} + +VkImageView vkb_view(vkb_image *im, const view_t *v, int attachment, int view_type) { + entry *e = (entry *)im; // im is the first member + uint8_t key[17] = {0}; + if (v) memcpy(key, v, 16); + key[16] = (uint8_t)(attachment ? 0xFF : view_type + 1); + // Recording threads look views up without a lock: entries are only appended (a full table takes no more), each + // published by the count after it is written. + int nv = __atomic_load_n(&e->nviews, __ATOMIC_ACQUIRE); + for (int i = 0; i < nv; i++) if (!memcmp(e->view_key[i], key, sizeof key)) return e->view[i]; + uint32_t base = v && v->set_levels ? v->base_level : 0, nlev = v && v->set_levels && v->nlevels ? v->nlevels : im->levels - base; + uint32_t layer = v && v->set_layers ? v->min_layer : 0, nlay = v && v->set_layers && v->nlayers ? v->nlayers : im->layers - layer; + if (attachment) nlev = 1; + if (attachment == 1 && !(im->usage & (VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT | VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT))) return VK_NULL_HANDLE; + if (attachment == 2 && !(im->usage & VK_IMAGE_USAGE_STORAGE_BIT)) return VK_NULL_HANDLE; + VkFormat f = im->format; + if (v && v->set_format && !im->astc) { + VkFormat vf = vkb_format(v->format, NULL, NULL); + if (vf != VK_FORMAT_UNDEFINED) f = vf; + } + int reinterpret_block = im->compressed && !(f >= VK_FORMAT_BC1_RGB_UNORM_BLOCK && f <= VK_FORMAT_BC7_SRGB_BLOCK); + if (reinterpret_block) { // [temporary] an uncompressed view of a compressed image: one level, one layer (and never a target) + if (attachment) return VK_NULL_HANDLE; + nlev = 1; + nlay = 1; + } + if (attachment == 2) { // storage views: the UNORM form of sRGB formats + if (f == VK_FORMAT_R8G8B8A8_SRGB) f = VK_FORMAT_R8G8B8A8_UNORM; + else if (f == VK_FORMAT_B8G8R8A8_SRGB) f = VK_FORMAT_B8G8R8A8_UNORM; + } + int target = v && v->set_target ? v->target : im->d.target; + VkImageViewType type = im->depth > 1 ? VK_IMAGE_VIEW_TYPE_3D + : target == NVN_TEXTURE_TARGET_1D ? VK_IMAGE_VIEW_TYPE_1D : target == NVN_TEXTURE_TARGET_1D_ARRAY ? VK_IMAGE_VIEW_TYPE_1D_ARRAY + : target == NVN_TEXTURE_TARGET_CUBEMAP && !attachment ? VK_IMAGE_VIEW_TYPE_CUBE + : target == NVN_TEXTURE_TARGET_CUBEMAP_ARRAY && !attachment ? VK_IMAGE_VIEW_TYPE_CUBE_ARRAY + : nlay > 1 || target == NVN_TEXTURE_TARGET_2D_ARRAY || target == NVN_TEXTURE_TARGET_2D_MULTISAMPLE_ARRAY ? VK_IMAGE_VIEW_TYPE_2D_ARRAY + : VK_IMAGE_VIEW_TYPE_2D; + if (type == VK_IMAGE_VIEW_TYPE_3D && attachment) { type = VK_IMAGE_VIEW_TYPE_2D_ARRAY; nlay = im->depth; } + if (!attachment && view_type >= 0 && im->depth <= 1) { // as the shader declares it (layer counts must fit the type) + type = (VkImageViewType)view_type; + if (((type == VK_IMAGE_VIEW_TYPE_CUBE || type == VK_IMAGE_VIEW_TYPE_CUBE_ARRAY) && nlay < 6) || type == VK_IMAGE_VIEW_TYPE_3D) + vkb_audit(A_TEX_VIEW_ADJUSTED, NULL); + if ((type == VK_IMAGE_VIEW_TYPE_CUBE && nlay < 6) || (type == VK_IMAGE_VIEW_TYPE_CUBE_ARRAY && nlay < 6)) + type = nlay > 1 ? VK_IMAGE_VIEW_TYPE_2D_ARRAY : VK_IMAGE_VIEW_TYPE_2D; + if (type == VK_IMAGE_VIEW_TYPE_CUBE) nlay = 6; + if (type == VK_IMAGE_VIEW_TYPE_CUBE_ARRAY) nlay -= nlay % 6; + if (type == VK_IMAGE_VIEW_TYPE_2D || type == VK_IMAGE_VIEW_TYPE_1D) nlay = 1; + if (type == VK_IMAGE_VIEW_TYPE_3D) type = VK_IMAGE_VIEW_TYPE_2D_ARRAY; // a 2D image declared 3D: closest valid view + } else if (!attachment && view_type == VK_IMAGE_VIEW_TYPE_3D && im->depth > 1) type = VK_IMAGE_VIEW_TYPE_3D; + if (type == VK_IMAGE_VIEW_TYPE_3D) { layer = 0; nlay = 1; } + if (type == VK_IMAGE_VIEW_TYPE_2D_ARRAY && im->depth > 1) { // slices of a 3D image: the depth of the view's level + uint32_t dl = im->depth >> base ? im->depth >> base : 1; + nlev = 1; + if (layer + nlay > dl) { if (layer >= dl) layer = 0; nlay = dl - layer; } + } + if (im->depth <= 1 && layer + nlay > im->layers) { if (layer >= im->layers) layer = 0; nlay = im->layers - layer; } + if (base + nlev > im->levels) { if (base >= im->levels) base = 0; nlev = im->levels - base; } + // swizzle: the view's, else the texture's own (TextureBuilder SetSwizzle); formats without alpha read a = 1 + VkComponentMapping cm = {VK_COMPONENT_SWIZZLE_IDENTITY, VK_COMPONENT_SWIZZLE_IDENTITY, VK_COMPONENT_SWIZZLE_IDENTITY, VK_COMPONENT_SWIZZLE_IDENTITY}; + if (!attachment) { + if (v && v->set_swizzle) + cm = (VkComponentMapping){swz(v->swz[0], VK_COMPONENT_SWIZZLE_R), swz(v->swz[1], VK_COMPONENT_SWIZZLE_G), + swz(v->swz[2], VK_COMPONENT_SWIZZLE_B), swz(v->swz[3], VK_COMPONENT_SWIZZLE_A)}; + else if (im->d.swizzle_r | im->d.swizzle_g | im->d.swizzle_b | im->d.swizzle_a) + cm = (VkComponentMapping){swz((uint8_t)im->d.swizzle_r, VK_COMPONENT_SWIZZLE_R), swz((uint8_t)im->d.swizzle_g, VK_COMPONENT_SWIZZLE_G), + swz((uint8_t)im->d.swizzle_b, VK_COMPONENT_SWIZZLE_B), swz((uint8_t)im->d.swizzle_a, VK_COMPONENT_SWIZZLE_A)}; + if (alpha_one(v && v->set_format ? v->format : im->d.format)) { + if (cm.a == VK_COMPONENT_SWIZZLE_IDENTITY || cm.a == VK_COMPONENT_SWIZZLE_A) cm.a = VK_COMPONENT_SWIZZLE_ONE; + } + } + VkImageAspectFlags aspect = im->aspect; + if (!attachment && aspect == (VK_IMAGE_ASPECT_DEPTH_BIT | VK_IMAGE_ASPECT_STENCIL_BIT)) aspect = VK_IMAGE_ASPECT_DEPTH_BIT; // sampled: depth + // the view's own usage: an image made for storage too must not make every view storage (sRGB views cannot be) + VkImageViewUsageCreateInfo vu = {VK_STRUCTURE_TYPE_IMAGE_VIEW_USAGE_CREATE_INFO, + .usage = attachment == 2 ? VK_IMAGE_USAGE_STORAGE_BIT + : attachment == 1 ? im->usage & (VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT | VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT) + : VK_IMAGE_USAGE_SAMPLED_BIT}; + // views used in the feedback loop layout: as attachment and sampled at once (a sampled view in another format + // keeps plain sampling: the format may not be renderable) + if ((im->usage & VK_IMAGE_USAGE_ATTACHMENT_FEEDBACK_LOOP_BIT_EXT) && (attachment == 1 || (!attachment && f == im->format))) + vu.usage = im->usage & (VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT | VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT | + VK_IMAGE_USAGE_SAMPLED_BIT | VK_IMAGE_USAGE_ATTACHMENT_FEEDBACK_LOOP_BIT_EXT); + VkImageViewCreateInfo ci = {VK_STRUCTURE_TYPE_IMAGE_VIEW_CREATE_INFO, &vu, .image = im->img, .viewType = type, .format = f, + .components = cm, .subresourceRange = {aspect, base, nlev, layer, nlay}}; + vkb_lock(&g_mu, LK_IMAGES); + nv = e->nviews; + for (int i = 0; i < nv; i++) + if (!memcmp(e->view_key[i], key, sizeof key)) { pthread_mutex_unlock(&g_mu); return e->view[i]; } // made meanwhile + VkImageView view = VK_NULL_HANDLE; + if (!vk_check(vk.vkCreateImageView(vkc.device, &ci, NULL, &view), "image view")) { pthread_mutex_unlock(&g_mu); return VK_NULL_HANDLE; } + if (nv < VIEWS) { + memcpy(e->view_key[nv], key, sizeof key); + e->view[nv] = view; + __atomic_store_n(&e->nviews, nv + 1, __ATOMIC_RELEASE); + } else { // [temporary] many distinct views of one texture: this one is not kept (destroyed when the GPU is done with it) + static int said; if (said++ < 4) VLOG("texture %llu: more than %d views", (unsigned long long)im->serial, VIEWS); + vkb_destroy_later(view, VK_NULL_HANDLE, VK_NULL_HANDLE); + } + pthread_mutex_unlock(&g_mu); + return view; +} + +void vkb_tex_release(const tex_t *t) { + if (!t || !t->serial || !vkb_ready()) return; + vkb_lock(&g_mu, LK_IMAGES); + tslot *s = g_cap ? slot(t->serial) : NULL; + if (s && s->serial == t->serial && s->e) { + entry *e = s->e; + s->e = NULL; g_n--; g_tomb++; + if (--e->refs == 0) { + if (!e->shaped) entry_drop(e); // no memory: nothing can take it back + else { + if (g_naside == RECYCLE_MAX) { entry *o = g_aside[0]; memmove(g_aside, g_aside + 1, (size_t)--g_naside * sizeof g_aside[0]); entry_drop(o); } + e->released = vkb_frame(); + e->released_stamp = nvn_dirty_now(); + g_aside[g_naside++] = e; + } + } + } + aside_purge(); + pthread_mutex_unlock(&g_mu); +} + +// ---------------- contents from game memory ---------------- +// Every level and layer: block-linear (or linear) storage -> rows of blocks in upload memory (nvn_deswizzle) -> image. +// ASTC: the blocks are decoded to RGBA8 on the CPU first ([temporary], phase 5: BC7 at build time). Depth / stencil and +// 3-component formats are not uploaded yet (render targets; logged once). +// The CPU cannot write game memory while a submission is replayed (the replay runs on the submitting thread), so one +// check per texture per submission is enough; a texture found changed is uploaded before its next use (the pass ends +// for it). Without this, textures the CPU updates while drawn every frame (the font atlas: new glyphs, the save slot +// thumbnails) kept their first contents. +int vkb_tex_stale(vkb_image *im) { + uint64_t sub = mtl_last_fence() + 1; + if (!im->upload_stamp || im->gpu_written || !im->storage) return 0; + if (im->cpu_written) return 1; + if (im->vpool_gen && im->vpool_gen != totk_vpool_generation()) return 1; // a virtual pool's mapping changed + if (__atomic_load_n(&im->dirty_checked, __ATOMIC_ACQUIRE) != sub) { + int mt = vkb_mt_recording(); + if (mt) { // one thread checks; the others wait for its answer + vkb_lock(&g_mu, LK_IMAGES); + if (im->dirty_checked == sub) { pthread_mutex_unlock(&g_mu); return im->stale; } + } + nvn_texlayout L; + nvn_tex_layout(&im->d, &L); + // a virtual pool: the physical pages behind it; [temporary] untracked memory (CPU-no-access pools hold GPU data) + // counts as unchanged: uploaded once + int v = im->vpool_gen ? totk_vpool_dirty_since((uintptr_t)im->storage, L.size, im->upload_stamp) : -1; + im->stale = v >= 0 ? v : nvn_dirty_tracked(im->storage, L.size) && nvn_dirty_since(im->storage, L.size, im->upload_stamp); + static long long log_serial = -1; // TOTK_VK_STALE_LOG=: the check's answers (development) + if (log_serial == -1) { const char *e = getenv("TOTK_VK_STALE_LOG"); log_serial = e ? atoll(e) : 0; } + if (log_serial && (long long)im->serial == log_serial) + VLOG("[stale] texture %llu submission %llu: tracked %d stale %d (stamp %llu now %llu)", (unsigned long long)im->serial, + (unsigned long long)sub, nvn_dirty_tracked(im->storage, L.size), im->stale, (unsigned long long)im->upload_stamp, + (unsigned long long)nvn_dirty_now()); + __atomic_store_n(&im->dirty_checked, sub, __ATOMIC_RELEASE); + if (mt) pthread_mutex_unlock(&g_mu); + } + return im->stale; +} + +// ASTC is decoded on the CPU (desktop GPUs have no ASTC); entering the world, decoding new textures on the recording +// thread held single submissions for 0.3-2.3 s (28% of that thread there, Great Sky Island, 2026-10-24). A texture's +// levels, layers and slices are gathered (astc_add) and decoded together (astc_run) by ASTC_WORKERS threads and the +// recording thread, in pieces of ASTC_CHUNK block rows: most of TOTK's textures and mip levels are small, so splitting +// texture by texture left the workers idle. +// ASTC -> BC7 when the port is built (vkb_bc7.c, tools/bc7_build.py) takes most decoding out of play. +enum { ASTC_WORKERS = 4, ASTC_CHUNK = 4 }; // block rows per piece taken +typedef struct { const uint8_t *src; uint8_t *dst; uint32_t w, h, bw, bh, first; int srgb; } astc_item; // first: its first piece +static astc_item *g_ai; +static int g_nai, g_capai; +static uint32_t g_apieces; // pieces of all items +static _Atomic uint32_t g_anext, g_adone; // the next piece to take / pieces decoded +static uint64_t g_aj_serial; +static int g_aj_open, g_aj_active; // a batch is being decoded / workers in it +static pthread_mutex_t g_aj_mu = PTHREAD_MUTEX_INITIALIZER; +static pthread_cond_t g_aj_cv = PTHREAD_COND_INITIALIZER, g_aj_done_cv = PTHREAD_COND_INITIALIZER; +uint64_t vkb_astc_ns; // decoding time on the recording thread (long submission log) +static void astc_add(const uint8_t *src, uint32_t w, uint32_t h, uint32_t bw, uint32_t bh, int srgb, uint8_t *dst) { + if (g_nai == g_capai) { + g_capai = g_capai ? g_capai * 2 : 64; + if (!(g_ai = realloc(g_ai, sizeof *g_ai * (size_t)g_capai))) die("ASTC items"); + } + g_ai[g_nai++] = (astc_item){src, dst, w, h, bw, bh, g_apieces, srgb}; + g_apieces += ((h + bh - 1) / bh + ASTC_CHUNK - 1) / ASTC_CHUNK; +} +static void astc_work(void) { // pieces until none is left + int k = 0; + for (;;) { + uint32_t p = atomic_fetch_add(&g_anext, 1); + if (p >= g_apieces) return; + while (k + 1 < g_nai && g_ai[k + 1].first <= p) k++; // pieces are taken in order: the item only moves forward + while (k > 0 && g_ai[k].first > p) k--; + const astc_item *it = &g_ai[k]; + uint32_t r = (p - it->first) * ASTC_CHUNK; + astc_decode_ldr_rows(it->src, it->w, it->h, it->bw, it->bh, it->srgb, it->dst, r, r + ASTC_CHUNK); + if (atomic_fetch_add(&g_adone, 1) + 1 == g_apieces) { + pthread_mutex_lock(&g_aj_mu); + pthread_cond_broadcast(&g_aj_done_cv); + pthread_mutex_unlock(&g_aj_mu); + } + } +} +static void *astc_worker(void *unused) { + (void)unused; + pthread_setname_np("totk-astc"); + uint64_t seen = 0; + for (;;) { + pthread_mutex_lock(&g_aj_mu); + while (g_aj_serial == seen) pthread_cond_wait(&g_aj_cv, &g_aj_mu); + seen = g_aj_serial; + int open = g_aj_open; + if (open) g_aj_active++; + pthread_mutex_unlock(&g_aj_mu); + if (!open) continue; + astc_work(); + pthread_mutex_lock(&g_aj_mu); + g_aj_active--; + pthread_cond_broadcast(&g_aj_done_cv); + pthread_mutex_unlock(&g_aj_mu); + } + return NULL; +} +static void astc_start(void) { + for (int i = 0; i < ASTC_WORKERS; i++) { + pthread_t t; + if (pthread_create(&t, NULL, astc_worker, NULL) == 0) pthread_detach(t); + } +} +static void astc_run(void) { // the gathered items, decoded; then the batch is empty + if (!g_nai) return; + uint64_t t0 = runtime_now_ns(); + atomic_store(&g_anext, 0); atomic_store(&g_adone, 0); + if (g_apieces > 2) { + static pthread_once_t once = PTHREAD_ONCE_INIT; + pthread_once(&once, astc_start); + pthread_mutex_lock(&g_aj_mu); + g_aj_open = 1; + g_aj_serial++; + pthread_cond_broadcast(&g_aj_cv); + pthread_mutex_unlock(&g_aj_mu); + } + astc_work(); + if (g_apieces > 2) { + pthread_mutex_lock(&g_aj_mu); + while (atomic_load(&g_adone) < g_apieces) pthread_cond_wait(&g_aj_done_cv, &g_aj_mu); + g_aj_open = 0; // a worker that wakes late takes nothing + while (g_aj_active) pthread_cond_wait(&g_aj_done_cv, &g_aj_mu); + pthread_mutex_unlock(&g_aj_mu); + } + g_nai = 0; g_apieces = 0; + vkb_astc_ns += runtime_now_ns() - t0; +} + +static int tex_upload(VkCommandBuffer cb, vkb_image *im); +// Several threads recording: into the submission's prologue, which runs before every chunk (the CPU wrote the bytes +// before the submission, so earlier is as good; a GPU write to the image in a chunk still comes after it). +static pthread_mutex_t g_up_mu = PTHREAD_MUTEX_INITIALIZER; // one upload at a time: the ASTC batch is shared +int vkb_tex_upload(VkCommandBuffer cb, vkb_image *im) { + uint64_t stamp = __atomic_load_n(&im->upload_stamp, __ATOMIC_ACQUIRE); + if (vkb_mt_recording() && __atomic_load_n(&im->uploading, __ATOMIC_ACQUIRE)) { // another thread is uploading it: its + pthread_mutex_lock(&g_up_mu); // image may change (BC7), wait + pthread_mutex_unlock(&g_up_mu); + stamp = __atomic_load_n(&im->upload_stamp, __ATOMIC_ACQUIRE); + } + if (!im->storage || im->gpu_written || (stamp && !vkb_tex_stale(im))) return 0; // the common case: nothing to do + uint64_t t0 = runtime_now_ns(); + int r; + if (vkb_mt_recording()) { + pthread_mutex_lock(&g_up_mu); + vkb_pro_begin(); + r = tex_upload(vkb_cmd(), im); // checks again: another thread may have uploaded it meanwhile + vkb_pro_end(); + pthread_mutex_unlock(&g_up_mu); + } else r = tex_upload(cb, im); + vkb_step_add(PK_UPLOAD, t0); + return r; +} +// ---- ASTC as BC7 (vkb_bc7.c). At its first upload an ASTC texture becomes a BC7 image when the cache holds every one of +// its surfaces (no view of it exists yet, nothing has drawn with it); its contents then come from the cache. Otherwise +// it stays RGBA8, decoded as before, and each surface is queued for encoding: the next time it is in the cache. +// [temporary] 1D and 3D ASTC textures stay RGBA8. +static int to_bc7(vkb_image *im, int srgb, int fmt) { + entry *e = (entry *)im; + if (__atomic_load_n(&e->nviews, __ATOMIC_ACQUIRE) || im->gpu_written || im->depth > 1 || im->d.target == NVN_TEXTURE_TARGET_1D || + im->d.target == NVN_TEXTURE_TARGET_1D_ARRAY) return 0; + VkFormat f = fmt == VKB_BC_1 ? (srgb ? VK_FORMAT_BC1_RGB_SRGB_BLOCK : VK_FORMAT_BC1_RGB_UNORM_BLOCK) + : (srgb ? VK_FORMAT_BC7_SRGB_BLOCK : VK_FORMAT_BC7_UNORM_BLOCK); + VkFormatProperties fp; + vk.vkGetPhysicalDeviceFormatProperties(vkc.gpu, f, &fp); + if (!(fp.optimalTilingFeatures & VK_FORMAT_FEATURE_SAMPLED_IMAGE_BIT)) return 0; + VkImageUsageFlags usage = VK_IMAGE_USAGE_SAMPLED_BIT | VK_IMAGE_USAGE_TRANSFER_SRC_BIT | VK_IMAGE_USAGE_TRANSFER_DST_BIT; + VkImageCreateInfo ci = {VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO, + .flags = im->layers >= 6 && im->width == im->height ? VK_IMAGE_CREATE_CUBE_COMPATIBLE_BIT : 0, + .imageType = VK_IMAGE_TYPE_2D, .format = f, .extent = {im->width, im->height, 1}, .mipLevels = im->levels, + .arrayLayers = im->layers, .samples = VK_SAMPLE_COUNT_1_BIT, .tiling = VK_IMAGE_TILING_OPTIMAL, .usage = usage, + .sharingMode = VK_SHARING_MODE_EXCLUSIVE, .initialLayout = VK_IMAGE_LAYOUT_UNDEFINED}; + VkImage img; + if (vk.vkCreateImage(vkc.device, &ci, NULL, &img) != VK_SUCCESS) return 0; + VkMemoryRequirements req; + vk.vkGetImageMemoryRequirements(vkc.device, img, &req); + VkMemoryAllocateInfo ai = {VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO, .allocationSize = req.size, + .memoryTypeIndex = memory_type(req.memoryTypeBits, VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT)}; + VkDeviceMemory mem; + if (ai.memoryTypeIndex == UINT32_MAX || vk.vkAllocateMemory(vkc.device, &ai, NULL, &mem) != VK_SUCCESS) { vk.vkDestroyImage(vkc.device, img, NULL); return 0; } + vk.vkBindImageMemory(vkc.device, img, mem, 0); + vkb_destroy_later(VK_NULL_HANDLE, im->img, im->mem); // the RGBA8 image no command has used + im->img = img; im->mem = mem; im->format = f; im->usage = usage; im->compressed = 1; im->bc7 = fmt == VKB_BC_1 ? 2 : 1; + im->layout = VK_IMAGE_LAYOUT_UNDEFINED; + return 1; +} +typedef struct { const uint8_t *rows; uint8_t *rgba; vkb_bc7_key k; uint32_t level, layer, lw, lh, ld; int have, fmt; VkDeviceSize slice; } surf_t; +static uint64_t g_bc7_images; // images made BC7 (the log line) +static void astc_upload(VkCommandBuffer cb, vkb_image *im, const uint8_t *src, int bw, int bh, int srgb) { + const nvn_texdesc *d = &im->d; + int ns = (int)(im->levels * im->layers), k = 0, all = 1, fmt = -1; + surf_t *sf = calloc((size_t)(ns ? ns : 1), sizeof *sf); + if (!sf) die("ASTC surfaces"); + for (uint32_t level = 0; level < im->levels; level++) { + uint32_t lw = im->width >> level ? im->width >> level : 1, lh = im->height >> level ? im->height >> level : 1; + uint32_t ld = im->depth >> level ? im->depth >> level : 1; + uint32_t bx = (lw + bw - 1) / bw, by = (lh + bh - 1) / bh, pitch = bx * 16; + VkDeviceSize slice = (VkDeviceSize)pitch * by, bytes = slice * ld; + for (uint32_t layer = 0; layer < im->layers; layer++, k++) { + VkBuffer buf; + VkDeviceSize off; + uint8_t *rows = vkb_ring_alloc(bytes, 16, &buf, &off); + surf_t *s = &sf[k]; + *s = (surf_t){NULL, NULL, {0, 0}, level, layer, lw, lh, ld, 0, 0, slice}; + if (nvn_deswizzle(d, level, layer, src, rows, pitch)) { all = 0; continue; } + s->rows = rows; + s->k = vkb_bc7_key_of(rows, (size_t)bytes, lw, lh, ld, (uint32_t)bw, (uint32_t)bh, srgb); + size_t len; + s->have = vkb_bc7_lookup(s->k, &s->fmt, &len) && + len == (s->fmt == VKB_BC_1 ? bc1_size(lw, lh, ld) : bc7_size(lw, lh, ld)); + all &= s->have && (fmt < 0 || s->fmt == fmt); // one format for the whole image + if (s->have && fmt < 0) fmt = s->fmt; + } + } + if (!im->bc7 && all && ns && to_bc7(im, srgb, fmt)) { + g_bc7_images++; + static int said; // TOTK_BC7_LOG=1: every image made BC7 (development) + if (getenv("TOTK_BC7_LOG") && said++ < 400) + VLOG("[bc7] texture %llu format %#x %ux%u target %d levels %u layers %u%s", (unsigned long long)im->serial, d->format, im->width, + im->height, d->target, im->levels, im->layers, srgb ? " sRGB" : ""); + } + if (!im->bc7 && getenv("TOTK_BC7_LOG")) { // not in the cache (development: where the rest of the ASTC comes from) + int have = 0; + for (int i = 0; i < ns; i++) have += sf[i].have; + static int said; + if (said++ < 400) + VLOG("[bc7] texture %llu format %#x %ux%u target %d levels %u layers %u: %d of %d surfaces in the cache, storage %p", + (unsigned long long)im->serial, d->format, im->width, im->height, d->target, im->levels, im->layers, have, ns, (const void *)im->storage); + } + vkb_image_use(cb, im, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT); + if (im->bc7) { // blocks from the cache (a surface missing from it: encoded now) + int bcf = im->bc7 == 2 ? VKB_BC_1 : VKB_BC_7; + for (int i = 0; i < ns; i++) { + surf_t *s = &sf[i]; + if (!s->rows) continue; + size_t n = bcf == VKB_BC_1 ? bc1_size(s->lw, s->lh, s->ld) : bc7_size(s->lw, s->lh, s->ld); + VkBuffer buf; + VkDeviceSize off; + uint8_t *blocks = vkb_ring_alloc(n, 16, &buf, &off); + if (!vkb_bc7_read(s->k, blocks, n)) { + VkBuffer rb; + VkDeviceSize ro; + uint8_t *rgba = vkb_ring_alloc((VkDeviceSize)s->lw * s->lh * 4 * s->ld, 16, &rb, &ro); + for (uint32_t z = 0; z < s->ld; z++) + astc_add(s->rows + s->slice * z, s->lw, s->lh, (uint32_t)bw, (uint32_t)bh, srgb, rgba + (VkDeviceSize)s->lw * s->lh * 4 * z); + astc_run(); + vkb_bc7_encode_now(s->k, rgba, s->lw, s->lh, s->ld, srgb, bcf, blocks); + } + VkBufferImageCopy r = {.bufferOffset = off, .bufferRowLength = (s->lw + 3) / 4 * 4, .bufferImageHeight = (s->lh + 3) / 4 * 4, + .imageSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, s->level, s->layer, 1}, .imageExtent = {s->lw, s->lh, s->ld}}; + vkb_checkpoint("upload texture (BC7)", NULL, NULL); + vk.vkCmdCopyBufferToImage(cb, buf, im->img, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, 1, &r); + } + } else { // decoded to RGBA8 (all surfaces together), then queued for encoding + for (int i = 0; i < ns; i++) { + surf_t *s = &sf[i]; + if (!s->rows) continue; + VkBuffer buf; + VkDeviceSize off; + s->rgba = vkb_ring_alloc((VkDeviceSize)s->lw * s->lh * 4 * s->ld, 16, &buf, &off); + for (uint32_t z = 0; z < s->ld; z++) + astc_add(s->rows + s->slice * z, s->lw, s->lh, (uint32_t)bw, (uint32_t)bh, srgb, s->rgba + (VkDeviceSize)s->lw * s->lh * 4 * z); + VkBufferImageCopy r = {.bufferOffset = off, .bufferRowLength = s->lw, .bufferImageHeight = s->lh, + .imageSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, s->level, s->layer, 1}, .imageExtent = {s->lw, s->lh, s->ld}}; + vkb_checkpoint("upload texture", NULL, NULL); + vk.vkCmdCopyBufferToImage(cb, buf, im->img, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, 1, &r); + } + astc_run(); + for (int i = 0; i < ns; i++) + if (sf[i].rgba && !sf[i].have) vkb_bc7_queue(sf[i].k, sf[i].rgba, sf[i].lw, sf[i].lh, sf[i].ld, srgb); + } + free(sf); +} +uint64_t vkb_bc7_images(void) { return g_bc7_images; } + +static int tex_upload(VkCommandBuffer cb, vkb_image *im) { + if (!im->storage || im->gpu_written) return 0; + const nvn_texdesc *d = &im->d; + nvn_texlayout L; + nvn_tex_layout(d, &L); + if (im->upload_stamp && !vkb_tex_stale(im)) return 0; // unchanged since the upload (checked once per submission) + im->stale = 0; + im->cpu_written = 0; + int bw = 1, bh = 1, bpb = nvn_fmt_bpb(d->format, &bw, &bh); + // Depth textures the game fills from the CPU (D16 / D32F, stored as D32_SFLOAT here): converted to floats on the way. + // TOTK sets one up at every camera cut of the title menu (192x192 D16, read by the chain that lights its environment + // probes); left empty, the probe and so the scene went dark for 4-6 frames (2026-10-24). [temporary] D24 (0x33) and + // the depth-stencil formats: the packing of their words is not settled yet. + int depth_up = im->aspect == VK_IMAGE_ASPECT_DEPTH_BIT && (d->format == 0x32 || d->format == 0x34) && bw == 1 && bh == 1; + if ((im->aspect != VK_IMAGE_ASPECT_COLOR_BIT && !depth_up) || !bpb || (d->format >= 0x19 && d->format <= 0x24)) { + im->not_uploadable = 1; + static int said; if (said++ < 4) VLOG("[temporary] texture %llu format %#x: contents not uploaded", (unsigned long long)im->serial, d->format); + im->upload_stamp = nvn_dirty_now(); + return 0; + } + if (!im->upload_stamp) { // first upload: does a GPU-written texture use the same memory? (aliasing, development log) + static int said; + vkb_lock(&g_mu, LK_IMAGES); + for (entry *e = g_all; e && said < 32; e = e->all_next) { + if (&e->im == im || !e->im.gpu_written || !e->im.storage) continue; + nvn_texlayout L2; + nvn_tex_layout(&e->im.d, &L2); + if (e->im.storage < im->storage + L.size && im->storage < e->im.storage + L2.size) { + said++; + VLOG("[alias] texture %llu (format %#x %ux%u) is read from memory that GPU-written texture %llu (format %#x %ux%u) uses", + (unsigned long long)im->serial, d->format, im->width, im->height, (unsigned long long)e->im.serial, + e->im.d.format, e->im.width, e->im.height); + } + } + pthread_mutex_unlock(&g_mu); + } + __atomic_store_n(&im->uploading, 1, __ATOMIC_SEQ_CST); // before the stamp: a thread that sees the stamp waits (vkb_tex_upload) + __atomic_store_n(&im->upload_stamp, nvn_dirty_now(), __ATOMIC_RELEASE); // before reading: a write during the upload shows as dirty next time + // A texture in a virtual pool (TOTK streams world textures this way): its addresses are only reserved, the bytes + // are in physical pools' pages mapped through the page table — gathered into a copy first. It uploads again when + // the pages behind it are written or the mapping changes ([temporary]: any MapVirtual; Vulkan sparse binding later). + const uint8_t *src = im->storage; + uint8_t *gathered = NULL; + uint64_t gen = totk_vpool_generation(); + if (totk_vpool_gather((uintptr_t)im->storage, 0, NULL)) { // (length 0: only whether it is in a virtual pool) + gathered = malloc(L.size ? L.size : 1); + if (!gathered) die("texture gather"); + totk_vpool_gather((uintptr_t)im->storage, L.size, gathered); + src = gathered; + im->vpool_gen = gen; + } + int srgb = d->format >= 0x87 && d->format <= 0x94; + if (im->astc && vkb_bc7_on() && bpb == 16) { + astc_upload(cb, im, src, bw, bh, srgb); + free(gathered); + __atomic_store_n(&im->uploading, 0, __ATOMIC_RELEASE); + return 1; + } + vkb_image_use(cb, im, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT); + for (uint32_t level = 0; level < im->levels; level++) { + uint32_t lw = im->width >> level ? im->width >> level : 1, lh = im->height >> level ? im->height >> level : 1; + uint32_t ld = im->depth >> level ? im->depth >> level : 1; + uint32_t bx = (lw + bw - 1) / bw, by = (lh + bh - 1) / bh, pitch = bx * (uint32_t)bpb; + VkDeviceSize slice = (VkDeviceSize)pitch * by, bytes = slice * ld; + for (uint32_t layer = 0; layer < im->layers; layer++) { + VkBuffer buf; + VkDeviceSize off; + uint8_t *rows = vkb_ring_alloc(bytes, 16, &buf, &off); + if (nvn_deswizzle(d, level, layer, src, rows, pitch)) continue; + VkBufferImageCopy r = {.bufferOffset = off, .bufferRowLength = bx * (uint32_t)bw, .bufferImageHeight = by * (uint32_t)bh, + .imageSubresource = {depth_up ? VK_IMAGE_ASPECT_DEPTH_BIT : VK_IMAGE_ASPECT_COLOR_BIT, level, layer, 1}, + .imageExtent = {lw, lh, ld}}; + if (depth_up && bpb == 2) { // D16 -> 32-bit float depth + float *f = vkb_ring_alloc((VkDeviceSize)lw * lh * ld * 4, 16, &buf, &off); + for (uint64_t k = 0; k < (uint64_t)lw * lh * ld; k++) { uint16_t v; memcpy(&v, rows + 2 * k, 2); f[k] = v / 65535.0f; } + r.bufferOffset = off; + } + if (im->astc) { // decoded texels, slice by slice + uint8_t *rgba = vkb_ring_alloc((VkDeviceSize)lw * lh * 4 * ld, 16, &buf, &off); + for (uint32_t z = 0; z < ld; z++) + astc_add(rows + slice * z, lw, lh, (uint32_t)bw, (uint32_t)bh, srgb, rgba + (VkDeviceSize)lw * lh * 4 * z); + r.bufferOffset = off; r.bufferRowLength = lw; r.bufferImageHeight = lh; + } + vkb_checkpoint("upload texture", NULL, NULL); + vk.vkCmdCopyBufferToImage(cb, buf, im->img, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, 1, &r); + } + } + astc_run(); // the copies recorded above read these texels when the submission runs + free(gathered); + __atomic_store_n(&im->uploading, 0, __ATOMIC_RELEASE); + return 1; +} + +// ---------------- CPU reads of texels (NVN ReadTexels) ---------------- +// TOTK reads GPU results on the CPU this way; the null backend's zeros made the title screen several times too bright. +// GPU-written images: the region is copied to upload memory and read after the GPU is done. Others: the game memory is +// the truth, de-laid-out as for an upload. Tightly packed rows unless the caller gives strides (block rows for BC). +void mtl_read_texels(const tex_t *t, unsigned level, unsigned layer0, const NVNcopyRegion *r, uint8_t *dst, size_t row, size_t img) { + if (!t || !r || !dst || r->width <= 0 || r->height <= 0) return; + int bw = 1, bh = 1, bpb = nvn_fmt_bpb(t->d.format, &bw, &bh); + uint32_t bx = (uint32_t)(r->width + bw - 1) / bw, by = (uint32_t)(r->height + bh - 1) / bh, nz = r->depth > 0 ? (uint32_t)r->depth : 1; + size_t wb = (size_t)bx * (size_t)bpb; + if (!row) row = wb; + if (!img) img = row * by; + vkb_image *im = vkb_image_of(t); + int is3d = t->d.target == NVN_TEXTURE_TARGET_3D; + static int calls; + if (calls++ < 8) VLOG("ReadTexels: texture %llu format %#x level %u region %dx%dx%d, %s", (unsigned long long)t->serial, t->d.format, + level, r->width, r->height, r->depth, im && im->gpu_written ? "GPU-written" : "game memory"); + if (im && im->gpu_written && !im->astc && bpb && im->aspect == VK_IMAGE_ASPECT_COLOR_BIT && !(t->d.format >= 0x19 && t->d.format <= 0x24)) { + mtl_sub_begin(); + VkCommandBuffer cb = vkb_cmd(); + VkBuffer b; + VkDeviceSize off; + uint8_t *px = vkb_ring_alloc((VkDeviceSize)wb * by * nz, 16, &b, &off); + vkb_image_use(cb, im, VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_READ_BIT); + VkBufferImageCopy c = {.bufferOffset = off, .bufferRowLength = bx * (uint32_t)bw, .bufferImageHeight = by * (uint32_t)bh, + .imageSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, level, is3d ? 0 : layer0 + (uint32_t)r->zoffset, is3d ? 1 : nz}, + .imageOffset = {r->xoffset, r->yoffset, is3d ? r->zoffset : 0}, .imageExtent = {(uint32_t)r->width, (uint32_t)r->height, is3d ? nz : 1}}; + vk.vkCmdCopyImageToBuffer(cb, im->img, VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, b, 1, &c); + mtl_wait(mtl_sub_commit(), UINT64_MAX); + for (uint32_t z = 0; z < nz; z++) + for (uint32_t y = 0; y < by; y++) memcpy(dst + z * img + y * row, px + ((size_t)z * by + y) * wb, wb); + return; + } + // from game memory: the whole level / layer, then the region + uint32_t lw = t->d.width >> level ? t->d.width >> level : 1, lh = t->d.height >> level ? t->d.height >> level : 1; + uint32_t ld = is3d ? (t->d.depth >> level ? t->d.depth >> level : 1) : 1; + uint32_t lbx = (lw + bw - 1) / bw, lby = (lh + bh - 1) / bh, pitch = lbx * (uint32_t)bpb; + uint8_t *lin = bpb && t->mem ? malloc((size_t)pitch * lby * ld) : NULL; + for (uint32_t z = 0; z < nz; z++) { + uint32_t layer = is3d ? 0 : layer0 + (uint32_t)r->zoffset + z, slice = is3d ? (uint32_t)r->zoffset + z : 0; + int ok = lin && (is3d ? z == 0 : 1) ? !nvn_deswizzle(&t->d, level, layer, t->mem, lin, pitch) : lin != NULL; + for (uint32_t y = 0; y < by; y++) { + uint8_t *o = dst + z * img + y * row; + uint32_t sy = (uint32_t)r->yoffset / bh + y, sx = (uint32_t)r->xoffset / bw; + if (ok && sy < lby && slice < ld && sx + bx <= lbx) memcpy(o, lin + ((size_t)slice * lby + sy) * pitch + (size_t)sx * bpb, wb); + else memset(o, 0, wb); + } + } + free(lin); + static int said; + if (!bpb && said++ < 4) VLOG("[temporary] ReadTexels: texture %llu format %#x not read", (unsigned long long)t->serial, t->d.format); +} + +// ---------------- CPU writes of texels (NVN WriteTexels) ---------------- +// The region goes into the game memory in the texture's own layout (the truth for textures the CPU fills: the next +// upload sees the pages dirty); an image the GPU has written gets the region copied in directly as well. TOTK fills +// its light falloff table this way: with the null backend's no-op the lights read garbage and the title screen was +// several times too bright (2026-10-23). +void mtl_write_texels(const tex_t *t, unsigned level, unsigned layer0, const NVNcopyRegion *r, const uint8_t *src, size_t row, size_t img) { + if (!t || !r || !src || r->width <= 0 || r->height <= 0 || !t->mem) return; + int bw = 1, bh = 1, bpb = nvn_fmt_bpb(t->d.format, &bw, &bh); + if (!bpb) { static int said; if (said++ < 4) VLOG("[temporary] WriteTexels: texture %llu format %#x not written", (unsigned long long)t->serial, t->d.format); return; } + uint32_t bx = (uint32_t)(r->width + bw - 1) / bw, by = (uint32_t)(r->height + bh - 1) / bh, nz = r->depth > 0 ? (uint32_t)r->depth : 1; + size_t wb = (size_t)bx * (size_t)bpb; + if (!row) row = wb; + if (!img) img = row * by; + int is3d = t->d.target == NVN_TEXTURE_TARGET_3D; + uint32_t lw = t->d.width >> level ? t->d.width >> level : 1, lh = t->d.height >> level ? t->d.height >> level : 1; + uint32_t ld = is3d ? (t->d.depth >> level ? t->d.depth >> level : 1) : 1; + uint32_t lbx = (lw + bw - 1) / bw, lby = (lh + bh - 1) / bh, pitch = lbx * (uint32_t)bpb; + uint32_t sx = (uint32_t)r->xoffset / bw, sy = (uint32_t)r->yoffset / bh; + if (sx + bx > lbx || sy + by > lby) { VLOG("WriteTexels: region outside texture %llu", (unsigned long long)t->serial); return; } + uint8_t *lin = malloc((size_t)pitch * lby * ld); + if (!lin) die("WriteTexels"); + uint8_t *storage = (uint8_t *)t->mem; + for (uint32_t z = 0; z < (is3d ? 1 : nz); z++) { // 3D: one pass over the level's slices + uint32_t layer = is3d ? 0 : layer0 + (uint32_t)r->zoffset + z; + if (nvn_deswizzle(&t->d, level, layer, storage, lin, pitch)) { VLOG("WriteTexels: texture %llu layout not supported", (unsigned long long)t->serial); break; } + for (uint32_t s = 0; s < (is3d ? nz : 1); s++) { + uint32_t slice = is3d ? (uint32_t)r->zoffset + s : 0; + if (slice >= ld) break; + for (uint32_t y = 0; y < by; y++) + memcpy(lin + ((size_t)slice * lby + sy + y) * pitch + (size_t)sx * bpb, src + (is3d ? s : z) * img + y * row, wb); + } + nvn_swizzle(&t->d, level, layer, lin, pitch, storage); + } + free(lin); + vkb_image *im = vkb_image_of(t); + if (im && !im->gpu_written) im->cpu_written = 1; // the pool may not be write-watched: say so directly + if (im && im->gpu_written && !im->astc && im->aspect == VK_IMAGE_ASPECT_COLOR_BIT && !(t->d.format >= 0x19 && t->d.format <= 0x24)) { + mtl_sub_begin(); + VkCommandBuffer cb = vkb_cmd(); + VkBuffer b; + VkDeviceSize off; + uint8_t *px = vkb_ring_alloc((VkDeviceSize)wb * by * nz, 16, &b, &off); + for (uint32_t z = 0; z < nz; z++) + for (uint32_t y = 0; y < by; y++) memcpy(px + ((size_t)z * by + y) * wb, src + z * img + y * row, wb); + vkb_image_use(cb, im, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT); + VkBufferImageCopy c = {.bufferOffset = off, .bufferRowLength = bx * (uint32_t)bw, .bufferImageHeight = by * (uint32_t)bh, + .imageSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, level, is3d ? 0 : layer0 + (uint32_t)r->zoffset, is3d ? 1 : nz}, + .imageOffset = {r->xoffset, r->yoffset, is3d ? r->zoffset : 0}, .imageExtent = {(uint32_t)r->width, (uint32_t)r->height, is3d ? nz : 1}}; + vkb_checkpoint("write texels", NULL, NULL); + vk.vkCmdCopyBufferToImage(cb, b, im->img, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, 1, &c); + mtl_wait(mtl_sub_commit(), UINT64_MAX); + } + static int calls; + if (calls++ < 8) VLOG("WriteTexels: texture %llu format %#x level %u region %dx%dx%d at %d,%d,%d%s", (unsigned long long)t->serial, + t->d.format, level, r->width, r->height, r->depth, r->xoffset, r->yoffset, r->zoffset, + im && im->gpu_written ? " (GPU-written image updated too)" : ""); +} + +// ---------------- development: dump the frame's render targets ---------------- +// TOTK_DUMP=:: every image the GPU has written (level 0, layer 0) in a format read back here +// (RGBA8 / BGRA8 / R11G11B10F / RGBA16F) is saved as /__x.bmp after that present. +static float half_to_float(uint16_t h) { + uint32_t s = (h >> 15) & 1, e = (h >> 10) & 31, m = h & 1023; + float v = e == 0 ? m / 1024.0f / 16384.0f : e == 31 ? 65504.0f : (1.0f + m / 1024.0f) * (float)(1 << e) / 32768.0f; + return s ? -v : v; +} +static float uf(uint32_t bits, int mbits) { // unsigned small float: 5-bit exponent, mbits mantissa (R11G11B10F) + uint32_t e = bits >> mbits, m = bits & ((1u << mbits) - 1); + if (e == 0) return m / (float)(1 << mbits) / 16384.0f; + if (e == 31) return 65504.0f; + return (1.0f + m / (float)(1 << mbits)) * (float)(1 << e) / 32768.0f; +} +static uint8_t tone(float v) { v = v / (1.0f + v); return v <= 0 ? 0 : v >= 1 ? 255 : (uint8_t)(v * 255.0f + 0.5f); } // HDR: Reinhard + +// Bytes per texel of the formats the dump reads back (0: not dumped); depth: the float depth aspect. +static uint32_t dump_bpp(const vkb_image *im, int *depth) { + VkFormat f = im->format; + *depth = f == VK_FORMAT_D32_SFLOAT || f == VK_FORMAT_D32_SFLOAT_S8_UINT; // depth: the float values, as R32F + if (im->astc || im->depth > 1 || (im->aspect != VK_IMAGE_ASPECT_COLOR_BIT && !*depth)) return 0; + return f == VK_FORMAT_R16G16B16A16_SFLOAT ? 8 : (f == VK_FORMAT_R8G8B8A8_UNORM || f == VK_FORMAT_R8G8B8A8_SRGB || + f == VK_FORMAT_B8G8R8A8_UNORM || f == VK_FORMAT_B8G8R8A8_SRGB || f == VK_FORMAT_B10G11R11_UFLOAT_PACK32 || + f == VK_FORMAT_R32_SFLOAT || *depth) ? 4 : f == VK_FORMAT_R16_SFLOAT ? 2 : 0; +} +// Records the copy of level 0, layer `layer` into upload memory (valid until the submission's slot is reused). +static uint8_t *dump_copy(VkCommandBuffer cb, vkb_image *im, uint32_t layer, uint32_t *bpp) { + int depth; + *bpp = dump_bpp(im, &depth); + if (!*bpp) return NULL; + VkBuffer b; + VkDeviceSize off; + uint8_t *px = vkb_ring_alloc((VkDeviceSize)im->width * im->height * *bpp, 16, &b, &off); + vkb_image_use(cb, im, VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_READ_BIT); + VkBufferImageCopy r = {.bufferOffset = off, .imageSubresource = {depth ? VK_IMAGE_ASPECT_DEPTH_BIT : VK_IMAGE_ASPECT_COLOR_BIT, 0, layer, 1}, + .imageExtent = {im->width, im->height, 1}}; + vk.vkCmdCopyImageToBuffer(cb, im->img, VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, b, 1, &r); + return px; +} +// /__x.bmp (float formats Reinhard-mapped) and value statistics in the log. +static void dump_save(const char *dir, const char *prefix, const vkb_image *im, const uint8_t *px, uint32_t bpp) { + uint32_t w = im->width, h = im->height, size = w * h * 4; + uint8_t *out = malloc(size); + if (!out) return; + for (uint32_t p = 0; p < w * h; p++) { + const uint8_t *s = px + (size_t)p * bpp; + uint8_t *o = out + (size_t)p * 4; // BMP: B G R A + switch (im->format) { + case VK_FORMAT_R16G16B16A16_SFLOAT: { + uint16_t c[4]; memcpy(c, s, 8); + o[2] = tone(half_to_float(c[0])); o[1] = tone(half_to_float(c[1])); o[0] = tone(half_to_float(c[2])); o[3] = 255; + break; + } + case VK_FORMAT_B10G11R11_UFLOAT_PACK32: { + uint32_t v; memcpy(&v, s, 4); + o[2] = tone(uf(v & 0x7FF, 6)); o[1] = tone(uf((v >> 11) & 0x7FF, 6)); o[0] = tone(uf(v >> 22, 5)); o[3] = 255; + break; + } + case VK_FORMAT_B8G8R8A8_UNORM: case VK_FORMAT_B8G8R8A8_SRGB: o[0] = s[0]; o[1] = s[1]; o[2] = s[2]; o[3] = 255; break; + case VK_FORMAT_R32_SFLOAT: case VK_FORMAT_D32_SFLOAT: case VK_FORMAT_D32_SFLOAT_S8_UINT: + { float v; memcpy(&v, s, 4); o[0] = o[1] = o[2] = tone(v < 0 ? -v : v); o[3] = 255; break; } + case VK_FORMAT_R16_SFLOAT: { uint16_t v; memcpy(&v, s, 2); float fv = half_to_float(v); o[0] = o[1] = o[2] = tone(fv < 0 ? -fv : fv); o[3] = 255; break; } + default: o[0] = s[2]; o[1] = s[1]; o[2] = s[0]; o[3] = 255; break; + } + } + if (im->format == VK_FORMAT_R16_SFLOAT) { // a table (light falloff): its corners and middle row + const uint16_t *h16 = (const uint16_t *)px; + char row[512]; + int rn = 0; + for (uint32_t x = 0; x < w && rn < 480; x += w / 16 ? w / 16 : 1) rn += snprintf(row + rn, sizeof row - rn, " %.4g", half_to_float(h16[(size_t)(h / 2) * w + x])); + VLOG("[dump] %stexture %llu R16F %ux%u: row %u:%s; corners %g %g %g %g", prefix, (unsigned long long)im->serial, w, h, h / 2, row, + half_to_float(h16[0]), half_to_float(h16[w - 1]), half_to_float(h16[(size_t)(h - 1) * w]), half_to_float(h16[(size_t)h * w - 1])); + } + if (im->format == VK_FORMAT_R32_SFLOAT || im->format == VK_FORMAT_D32_SFLOAT || im->format == VK_FORMAT_D32_SFLOAT_S8_UINT) { // single channel: the values + double sum = 0; float lo = 3e38f, hi = -3e38f; + for (uint32_t p = 0; p < w * h; p++) { float v; memcpy(&v, px + (size_t)p * 4, 4); sum += v; if (v < lo) lo = v; if (v > hi) hi = v; } + VLOG("[dump] %stexture %llu %s %ux%u: min %g mean %g max %g, centre %g", prefix, (unsigned long long)im->serial, + im->format == VK_FORMAT_R32_SFLOAT ? "R32F" : "depth", w, h, lo, sum / (w * h), hi, ((const float *)px)[(size_t)(h / 2) * w + w / 2]); + } + char path[1024]; + snprintf(path, sizeof path, "%s/%s%llu_%02x_%ux%u.bmp", dir, prefix, (unsigned long long)im->serial, im->d.format, w, h); + FILE *f = fopen(path, "wb"); + if (f) { + uint8_t hdr[54] = {'B', 'M'}; + uint32_t v = 54 + size; memcpy(hdr + 2, &v, 4); + v = 54; memcpy(hdr + 10, &v, 4); + v = 40; memcpy(hdr + 14, &v, 4); + memcpy(hdr + 18, &w, 4); + int32_t nh = -(int32_t)h; memcpy(hdr + 22, &nh, 4); + hdr[26] = 1; hdr[28] = 32; + memcpy(hdr + 34, &size, 4); + fwrite(hdr, 1, sizeof hdr, f); + fwrite(out, 1, size, f); + fclose(f); + } + free(out); +} + +// ---------------- development: snapshots in the middle of a frame ---------------- +// TOTK_SNAP=::: before the first draw of that present whose vertex or fragment shader +// key starts so, every texture the draw samples (and its render targets) is copied; saved as /snap_... at the +// next present. Answers "what did this pass see", which the end-of-frame dump cannot. +typedef struct { vkb_image *im; uint8_t *px; uint32_t bpp; char prefix[48]; } snap_t; +static snap_t g_snap[64]; +static int g_nsnap; +static const char *g_snap_dir; +static int g_snap_key; // the index of the key of the draw being snapped (file names) +static long long g_snap_at_jump; // TOTK_SNAP_ON_JUMP=1: the TOTK_SNAP frame is replaced by the present after the first upward jump (lumlog) +int vkb_tex_snap_wanted(const char *vkey, const char *fkey, uint64_t frame) { // keys: k1,k2,...: the first draw of each + static long long at = -1; + static char keys[8][65]; + static int nkeys, done[8]; + if (at == -1) { + const char *e = getenv("TOTK_SNAP"); + at = e ? atoll(e) : 0; + const char *k = e ? strchr(e, ':') : NULL, *d = k ? strchr(k + 1, ':') : NULL; + if (!k || !d) at = 0; + for (const char *p = k ? k + 1 : NULL; p && p < d && nkeys < 8;) { + size_t n = strcspn(p, ",:"); + if (n && n < 65) { memcpy(keys[nkeys], p, n); keys[nkeys++][n] = 0; } + p += n + (p[n] == ','); + if (*p == ':' || p >= d) break; + } + if (d) g_snap_dir = d + 1; + } + if (g_snap_at_jump) at = g_snap_at_jump; + if (at <= 0 || frame != (uint64_t)at) return 0; + for (int i = 0; i < nkeys; i++) { + size_t n = strlen(keys[i]); + if (done[i] || !((vkey && !strncmp(vkey, keys[i], n)) || (fkey && !strncmp(fkey, keys[i], n)))) continue; + done[i] = 1; // one draw per key + g_snap_key = i; + return 1; + } + return 0; +} +void vkb_tex_snap(VkCommandBuffer cb, vkb_image *im, const char *what) { + if (!im) return; + for (uint32_t layer = 0; layer < (im->layers > 1 ? 2u : 1u) && g_nsnap < 64; layer++) { + snap_t *s = &g_snap[g_nsnap]; + s->px = dump_copy(cb, im, layer, &s->bpp); + if (!s->px) return; + s->im = im; + snprintf(s->prefix, sizeof s->prefix, "snap%d_%s%s_%04llx_", g_snap_key, what, layer ? "_l1" : "", + (unsigned long long)((uintptr_t)im->img & 0xffff)); // the key's index; the VkImage's low bits: same image or not + g_nsnap++; + } +} +void vkb_tex_snap_flush(void) { + if (!g_nsnap) return; + mtl_finish(); + _mkdir(g_snap_dir); + for (int k = 0; k < g_nsnap; k++) { + dump_save(g_snap_dir, g_snap[k].prefix, g_snap[k].im, g_snap[k].px, g_snap[k].bpp); + const vkb_image *im = g_snap[k].im; + if (g_snap[k].bpp == 8) { // RGBA16F: per-channel means and pixels below 1 (flags and masks) + uint32_t np = im->width * im->height, below[4] = {0}; + double ch[4] = {0}; + for (uint32_t p = 0; p < np; p++) { + uint16_t h[4]; memcpy(h, g_snap[k].px + (size_t)p * 8, 8); + for (int q = 0; q < 4; q++) { float v = half_to_float(h[q]); ch[q] += v; if (v < 1.0f) below[q]++; } + } + VLOG("snapshot %s%ux%u: rgba %.4g %.4g %.4g %.4g, below 1: %u %u %u %u of %u", g_snap[k].prefix, im->width, im->height, + ch[0] / np, ch[1] / np, ch[2] / np, ch[3] / np, below[0], below[1], below[2], below[3], np); + } + } + VLOG("snapshot: %d images to %s", g_nsnap, g_snap_dir); + g_nsnap = 0; +} + +// TOTK_SAMEMEM=: at that present, every group of live images at exactly the same memory, with what differs +// (development: the game sees one texture where the backend has several images). +static int by_storage(const void *a, const void *b) { + const entry *x = *(entry *const *)a, *y = *(entry *const *)b; + return (uintptr_t)x->im.storage < (uintptr_t)y->im.storage ? -1 : (uintptr_t)x->im.storage > (uintptr_t)y->im.storage; +} +static void samemem_report(uint64_t present) { + static long long at = -1; + if (at == -1) { const char *e = getenv("TOTK_SAMEMEM"); at = e ? atoll(e) : 0; } + if (at <= 0 || present != (uint64_t)at) return; + vkb_lock(&g_mu, LK_IMAGES); + int n = 0; + for (entry *e = g_all; e; e = e->all_next) if (e->refs && e->im.storage) n++; + entry **v = malloc(sizeof *v * (size_t)(n ? n : 1)); + n = 0; + for (entry *e = g_all; v && e; e = e->all_next) if (e->refs && e->im.storage) v[n++] = e; + if (v) qsort(v, (size_t)n, sizeof *v, by_storage); + int groups = 0; + for (int i = 0; v && i < n;) { + int j = i + 1; + while (j < n && v[j]->im.storage == v[i]->im.storage) j++; + if (j - i > 1) { + groups++; + for (int k = i; k < j; k++) { + const nvn_texdesc *d = &v[k]->im.d; + VLOG("[samemem] %p image %llu refs %d: target %u format %#x flags %#x %ux%ux%u levels %u samples %u stride %u block_h %u " + "swizzle %u%u%u%u layout %d %02x%02x%02x%02x%02x%02x%02x%02x gpu_written %d", v[k]->im.storage, (unsigned long long)v[k]->im.serial, + v[k]->refs, d->target, d->format, d->flags, d->width, d->height, d->depth, d->levels, d->samples, d->stride, d->pkg_block_h, + d->swizzle_r, d->swizzle_g, d->swizzle_b, d->swizzle_a, v[k]->has_layout, v[k]->layout[0], v[k]->layout[1], v[k]->layout[2], + v[k]->layout[3], v[k]->layout[4], v[k]->layout[5], v[k]->layout[6], v[k]->layout[7], v[k]->im.gpu_written); + } + } + i = j; + } + VLOG("[samemem] present %llu: %d live images, %d memory addresses with more than one", (unsigned long long)present, n, groups); + free(v); + pthread_mutex_unlock(&g_mu); +} + +// TOTK_LUMLOG=:: the mean of the presented image for count presents from that one (development: +// frame-to-frame flicker). Waits for the GPU at each of those presents. +// TOTK_TEXLOG=
,...: within the TOTK_LUMLOG presents, the mean of every layer (level 0) of the images at those +// memory addresses (development: how a GPU-built texture, such as an environment probe, evolves frame by frame). +static void texlog(uint64_t present) { + static uint64_t addr[8], serial[8]; + static int n = -1; + if (n < 0) { + n = 0; + const char *e = getenv("TOTK_TEXLOG"); + for (const char *p = e; p && *p && n < 8;) { // s: the image with that serial (the draw trace's numbers) + serial[n] = *p == 's' ? strtoull(p + 1, NULL, 10) : 0; + addr[n++] = *p == 's' ? 0 : strtoull(p, NULL, 16); + p = strchr(p, ','); if (p) p++; + } + } + for (int k = 0; k < n; k++) { + vkb_image *im = NULL; + vkb_lock(&g_mu, LK_IMAGES); + for (entry *e = g_all; e; e = e->all_next) + if (e->refs && (serial[k] ? e->im.serial == serial[k] : (uint64_t)(uintptr_t)e->im.storage == addr[k])) { im = &e->im; break; } + pthread_mutex_unlock(&g_mu); + if (!im) continue; + char line[512]; + int ln = snprintf(line, sizeof line, "[texlog] present %llu %#llx image %llu (game memory %p, %ux%u):", (unsigned long long)present, + (unsigned long long)addr[k], (unsigned long long)im->serial, (const void *)im->storage, im->width, im->height); + for (uint32_t layer = 0; layer < im->layers && layer < 6; layer++) { + mtl_sub_begin(); + uint32_t bpp; + uint8_t *px = dump_copy(vkb_cmd(), im, layer, &bpp); + mtl_wait(mtl_sub_commit(), UINT64_MAX); + if (!px) continue; + double s = 0, ch[4] = {0}; + uint32_t np = im->width * im->height, below1[4] = {0}; + for (uint32_t p = 0; p < np; p++) { + if (bpp == 4 && im->format == VK_FORMAT_B10G11R11_UFLOAT_PACK32) { // 11/11/10-bit unsigned floats + uint32_t v; memcpy(&v, px + (size_t)p * 4, 4); + uint32_t c[3] = {v & 0x7ff, (v >> 11) & 0x7ff, v >> 22}, mb[3] = {6, 6, 5}; + for (int q = 0; q < 3; q++) { + uint32_t e = c[q] >> mb[q], m = c[q] & ((1u << mb[q]) - 1); + s += (e ? ldexp(1.0 + m / (double)(1u << mb[q]), (int)e - 15) : ldexp(m / (double)(1u << mb[q]), -14)) / 3.0; + } + } + else if (bpp == 4) s += (px[(size_t)p * 4] + px[(size_t)p * 4 + 1] + px[(size_t)p * 4 + 2]) / 3.0; + else if (bpp == 8) { // RGBA16F: per channel below + uint16_t h[4]; memcpy(h, px + (size_t)p * 8, 8); + for (int q = 0; q < 4; q++) { float v = half_to_float(h[q]); ch[q] += v; if (v < 1.0f) below1[q]++; } + s += (half_to_float(h[0]) + half_to_float(h[1]) + half_to_float(h[2])) / 3.0; + } + } + ln += snprintf(line + ln, sizeof line - (size_t)ln, " %.4g", np ? s / np : 0); + if (bpp == 8 && np) + ln += snprintf(line + ln, sizeof line - (size_t)ln, " [rgba %.3g %.3g %.3g %.3g, below 1: %u %u %u %u]", ch[0] / np, ch[1] / np, ch[2] / np, + ch[3] / np, below1[0], below1[1], below1[2], below1[3]); + } + VLOG("%s", line); + } +} + +// TOTK_MEMLOG=
:[,...]: within the TOTK_LUMLOG presents (the GPU done with the frame), a summary of game +// memory there - nonzero 32-bit words, the sum of their absolute float values, an FNV hash - and with TOTK_LUMLOG_DIR the +// bytes as /mem_
_.bin (development: GPU-built buffers such as a light list, frame by frame). +static void memlog(uint64_t present) { + static uint64_t addr[4], len[4]; + static int n = -1; + if (n < 0) { + n = 0; + for (const char *p = getenv("TOTK_MEMLOG"); p && *p && n < 4;) { + char *e; + addr[n] = strtoull(p, &e, 16); + len[n] = *e == ':' ? strtoull(e + 1, &e, 0) : 0; + if (len[n]) n++; + p = strchr(p, ','); + if (p) p++; + } + } + for (int k = 0; k < n; k++) { + const uint8_t *m = (const uint8_t *)(uintptr_t)addr[k]; + uint64_t nz = 0, h = 1469598103934665603ull; + double s = 0; + for (uint64_t i = 0; i + 4 <= len[k]; i += 4) { + uint32_t w; float f; + memcpy(&w, m + i, 4); memcpy(&f, m + i, 4); + if (w) { nz++; if (f == f && f < 1e30f && f > -1e30f) s += f < 0 ? -f : f; } + h = (h ^ w) * 1099511628211ull; + } + VLOG("[memlog] present %llu %#llx+%llu: nonzero words %llu, sum |f| %.6g, hash %016llx", (unsigned long long)present, + (unsigned long long)addr[k], (unsigned long long)len[k], (unsigned long long)nz, s, (unsigned long long)h); + const char *dir = getenv("TOTK_LUMLOG_DIR"); + if (dir && *dir) { + _mkdir(dir); + char path[512]; + snprintf(path, sizeof path, "%s/mem_%llx_%llu.bin", dir, (unsigned long long)addr[k], (unsigned long long)present); + FILE *f = fopen(path, "wb"); + if (f) { fwrite(m, 1, (size_t)len[k], f); fclose(f); } + } + } +} + +void vkb_tex_dump(const char *dir); +void vkb_tex_lumlog(const tex_t *t, uint64_t present) { + samemem_report(present); + { + static long long at = -1, count; + if (at == -1) { const char *e = getenv("TOTK_LUMLOG"); at = e ? atoll(e) : 0; count = e && strchr(e, ':') ? atoll(strchr(e, ':') + 1) : 0; } + if (at > 0 && present >= (uint64_t)at && present < (uint64_t)(at + count)) { texlog(present); } + } + static long long at = -1, count; + if (at == -1) { const char *e = getenv("TOTK_LUMLOG"); at = e ? atoll(e) : 0; count = e && strchr(e, ':') ? atoll(strchr(e, ':') + 1) : 0; } + if (at <= 0 || present < (uint64_t)at || present >= (uint64_t)(at + count)) return; + vkb_image *im = vkb_image_of(t); + if (!im) return; + mtl_sub_begin(); + uint32_t bpp; + uint8_t *px = dump_copy(vkb_cmd(), im, 0, &bpp); + mtl_wait(mtl_sub_commit(), UINT64_MAX); + if (!px || bpp != 4) return; + double sum[3] = {0}, low = 0; // low: the lower half (the ground on the island; clouds there when it is missing) + for (uint32_t p = 0; p < im->width * im->height; p += 7) + for (int c = 0; c < 3; c++) { sum[c] += px[(size_t)p * 4 + c]; if (p / im->width >= im->height / 2) low += px[(size_t)p * 4 + c]; } + double n = (double)(im->width * im->height / 7) * 255.0; + static uint64_t last_draws; + uint64_t draws = vkb_draw_count(); + VLOG("[lum] present %llu: mean %.3f %.3f %.3f, lower half %.3f, draws %llu", (unsigned long long)present, sum[0] / n, sum[1] / n, sum[2] / n, + low / (n * 1.5), (unsigned long long)(draws - last_draws)); + last_draws = draws; + memlog(present); // the GPU has finished this frame (waited for above) + // and each of those frames as /f_...bmp; with TOTK_LUMLOG_JUMP= only frames where the lower + // half's mean jumps by more than delta, with the frame before and the two after (catching a short drop-out) + const char *dir = getenv("TOTK_LUMLOG_DIR"), *jump = getenv("TOTK_LUMLOG_JUMP"); + static double prev_low = -1; + static uint8_t *prev_px; + static uint64_t prev_present; + static int save_left; + double lowm = low / (n * 1.5); + if (dir && *dir) { + _mkdir(dir); + char prefix[32]; + size_t bytes = (size_t)im->width * im->height * bpp; + int save = !jump; + if (jump && prev_low >= 0 && (lowm - prev_low > atof(jump) || prev_low - lowm > atof(jump))) { + save = 1; save_left = 2; + static int jumps; + if (prev_px && jumps++ < 8) { snprintf(prefix, sizeof prefix, "f%llu_before_", (unsigned long long)prev_present); dump_save(dir, prefix, im, prev_px, bpp); } + VLOG("[lum] jump at present %llu: lower half %.3f -> %.3f", (unsigned long long)present, prev_low, lowm); + // 1: after the first jump up (the drop-out ends); down[]: the n-th drop (default the first) + const char *soj = getenv("TOTK_SNAP_ON_JUMP"); + int down = soj && !strncmp(soj, "down", 4); + static int drops; + if (down && lowm < prev_low) drops++; + if (soj && !g_snap_at_jump && (down ? lowm < prev_low && drops >= (atoi(soj + 4) > 0 ? atoi(soj + 4) : 1) : lowm > prev_low)) { + g_snap_at_jump = (long long)vkb_frame() + 1; // the next frame recorded (the drop-out lasts a few) + VLOG("[lum] snapshot set for present %lld", g_snap_at_jump); + } + static int dumps; // TOTK_LUMLOG_JUMP_DUMP=1: the first jumps also dump every render target of the frame + if (getenv("TOTK_LUMLOG_JUMP_DUMP") && dumps++ < 6) { + char d2[512]; + snprintf(d2, sizeof d2, "%s/jump%llu", dir, (unsigned long long)present); + _mkdir(d2); + vkb_tex_dump(d2); + } + } else if (save_left > 0) { save = 1; save_left--; } + static long every = -1; // TOTK_LUMLOG_EVERY=: also every n-th frame (where the scripted run went) + if (every < 0) every = getenv("TOTK_LUMLOG_EVERY") ? atol(getenv("TOTK_LUMLOG_EVERY")) : 0; + if (every > 0 && present % (uint64_t)every == 0) save = 1; + if (save) { snprintf(prefix, sizeof prefix, "f%llu_", (unsigned long long)present); dump_save(dir, prefix, im, px, bpp); } + if (jump) { // the frame, kept for "before" (the ring memory is reused) + if (!prev_px) prev_px = malloc(bytes); + if (prev_px) memcpy(prev_px, px, bytes); + prev_present = present; + } + } + prev_low = lowm; +} + +void vkb_tex_dump(const char *dir) { + typedef struct { vkb_image *im; uint8_t *px; uint32_t bpp; } job; + job jobs[512]; + int n = 0; + mtl_sub_begin(); + VkCommandBuffer cb = vkb_cmd(); + vkb_lock(&g_mu, LK_IMAGES); + for (entry *e = g_all; e && n < 512; e = e->all_next) { + // TOTK_DUMP_ALL=1: textures uploaded from game memory too (tables, LUTs) + if (!e->refs || (!e->im.gpu_written && !(e->im.upload_stamp && getenv("TOTK_DUMP_ALL")))) continue; + uint32_t bpp; + uint8_t *px = dump_copy(cb, &e->im, 0, &bpp); + if (px) jobs[n++] = (job){&e->im, px, bpp}; + } + pthread_mutex_unlock(&g_mu); + mtl_wait(mtl_sub_commit(), UINT64_MAX); + _mkdir(dir); // one level: the parent must exist + for (int k = 0; k < n; k++) dump_save(dir, "", jobs[k].im, jobs[k].px, jobs[k].bpp); + VLOG("dumped %d render targets to %s", n, dir); + { // GPU-written textures that share memory (aliasing): each is a separate image here + int said = 0; + vkb_lock(&g_mu, LK_IMAGES); + for (entry *a = g_all; a && said < 40; a = a->all_next) { + if (!a->refs || !a->im.gpu_written || !a->im.storage) continue; + nvn_texlayout La; + nvn_tex_layout(&a->im.d, &La); + for (entry *b = a->all_next; b && said < 40; b = b->all_next) { + if (!b->refs || !b->im.gpu_written || !b->im.storage) continue; + nvn_texlayout Lb; + nvn_tex_layout(&b->im.d, &Lb); + if (a->im.storage < b->im.storage + Lb.size && b->im.storage < a->im.storage + La.size) { + said++; + VLOG("[alias] GPU-written textures %llu (format %#x %ux%u at %p) and %llu (format %#x %ux%u at %p) share memory", + (unsigned long long)a->im.serial, a->im.d.format, a->im.width, a->im.height, (const void *)a->im.storage, + (unsigned long long)b->im.serial, b->im.d.format, b->im.width, b->im.height, (const void *)b->im.storage); + } + } + } + pthread_mutex_unlock(&g_mu); + } + // TOTK_DUMP_MEM=:[,...]: raw guest memory (GPU-written buffers) as /mem_
.bin + for (const char *p = getenv("TOTK_DUMP_MEM"); p && *p;) { + char *end; + unsigned long long addr = strtoull(p, &end, 0), size = *end == ':' ? strtoull(end + 1, &end, 0) : 0; + if (addr && size && size < (64u << 20)) { + char path[1024]; + snprintf(path, sizeof path, "%s/mem_%llx.bin", dir, addr); + FILE *f = fopen(path, "wb"); + if (f) { fwrite((const void *)(uintptr_t)addr, 1, size, f); fclose(f); } + } + p = strchr(end, ','); + if (p) p++; + } +} diff --git a/src/gfx/vkb_ucap.c b/src/gfx/vkb_ucap.c new file mode 100644 index 0000000..9fd5753 --- /dev/null +++ b/src/gfx/vkb_ucap.c @@ -0,0 +1,227 @@ +// Uniform contents as of the submit call (after MacNativeGames' nvn_mtl.m usnap / mtl_sub_capture_ubos, MIT). +// On the console nvnQueueSubmitCommands returns at once and the GPU reads the constants soon after; the game, having +// handed the frame over, rebuilds its per-object and per-frame constant pools on other threads right away. Our replay +// runs later (the translation thread) and the GPU later still, so draws read constants of the next frame or bytes the game +// cleared for refilling: TOTK's light parameters (c13 of its lighting shaders) read as zeros every other frame, and on a +// camera cut of the title menu the scene went black for 4-6 frames (2026-10-24; MacNativeGames saw distant meshes +// drawn in front of the camera from the same race). +// So the uniform ranges a submission binds are copied in the submit call (game thread, mtl_sub_capture_ubos), uploaded +// at the start of its replay into device memory of the submission's slot (mtl_sub_snapshot_ubos_cap), and bindings of +// those ranges read the copy (vkb_ucap_range). A GPU write to a copied range during the submission (inline uniform update, +// buffer copy, texture-to-buffer copy, storage buffer store) drops that copy from then on: later readers see the GPU's +// result in memory, as MacNativeGames does. TOTK_UBO_SNAPSHOT=0 reads live memory as before (A/B). +// "From then on" is the position of the command in the submission (vkb_seq): with several recording threads the walk of +// the submission (vkb_mt.c) drops in order and the threads only look, each at its own position. +#include +#include +#include + +#include "compat/gfx_compat.h" +#include "host/hle.h" +#include "nvn/nvn_mtl.h" +#include "nvn/nvn_obj.h" +#include "vkb.h" +#include "vkb_lock.h" + +#define VLOG(...) LOG("[vk] " __VA_ARGS__) +enum { UBO_MAX = 65536, OTHER_MAX = 1 << 20, MERGE_MAX = 4 << 20 }; + +typedef struct { uint64_t lo, hi; } rng; +typedef struct { int n; rng *r; size_t *off; uint8_t *bytes; uint64_t done; } ucap; // ranges sorted, disjoint; off: into bytes; done: the GPU had completed this submission at the capture + +int vkb_ucap_on(void) { + static int on = -1; + if (on < 0) { const char *e = getenv("TOTK_UBO_SNAPSHOT"); on = !(e && *e == '0'); VLOG("uniform contents as of the submit call: %s", on ? "on" : "off (live memory, A/B)"); } + return on; +} + +static void collect(rng **v, int *n, int *cap, const uint8_t *p, int depth) { + for (;;) { + op_hdr h; + memcpy(&h, p, sizeof h); + if (h.size < 8 || (h.size & 7) || h.op >= OP_COUNT_) die("corrupt command record at %p while copying uniforms (op %u size %u)", (const void *)p, h.op, h.size); + if (h.op == OP_END) return; + if (h.op == OP_JUMP) { memcpy(&p, p + 8, 8); continue; } + if (h.op == OP_CALL) { uint64_t t; memcpy(&t, p + 8, 8); if (depth < 16) collect(v, n, cap, (const uint8_t *)(uintptr_t)t, depth + 1); } + else if (h.op == OP_BIND_UBO || h.op == OP_BIND_SSBO || h.op == OP_COPY_B2B) { + // uniform buffers; storage buffers (TOTK's light list, read by its light shaders) and the sources of buffer + // copies (its light parameters reach their uniform buffer by a GPU copy every frame): all bytes the game + // rewrites for the next frame as soon as it has submitted this one + uint64_t addr, sz; + if (h.op == OP_COPY_B2B) { memcpy(&addr, p + 8, 8); memcpy(&sz, p + 24, 8); if (sz > OTHER_MAX) sz = 0; } + else { + const p_bind_buf *b = (const p_bind_buf *)(p + 8); + addr = b->addr; sz = b->size; + if (h.op == OP_BIND_UBO && sz > UBO_MAX) sz = UBO_MAX; + if (h.op == OP_BIND_SSBO && sz > OTHER_MAX) sz = 0; + } + if (addr && sz) { + if (*n == *cap) { *cap = *cap ? *cap * 2 : 4096; if (!(*v = realloc(*v, sizeof **v * (size_t)*cap))) die("uniform capture ranges"); } + (*v)[(*n)++] = (rng){addr, addr + sz}; + } + } + p += h.size; + } +} +static int rng_cmp(const void *a, const void *b) { const rng *x = a, *y = b; return x->lo < y->lo ? -1 : x->lo > y->lo; } + +static _Atomic uint64_t g_cap_bytes, g_cap_ns, g_caps; +void *mtl_sub_capture_ubos(const uint64_t *h, int n) { // game thread, in the submit call + if (!vkb_ucap_on()) return NULL; + uint64_t t0 = runtime_now_ns(); + rng *v = NULL; + int nv = 0, cv = 0; + for (int i = 0; i < n; i++) collect(&v, &nv, &cv, (const uint8_t *)(uintptr_t)h[i], 0); + ucap *c = calloc(1, sizeof *c); + if (!c) die("uniform capture"); + c->done = mtl_done(); // before the bytes: a submission completing meanwhile only makes this conservative + if (nv) { + qsort(v, (size_t)nv, sizeof *v, rng_cmp); + int k = -1; // overlapping / touching bindings merged (up to MERGE_MAX); ranges outside every imported pool dropped + for (int i = 0; i < nv; i++) { + void *mb; uint64_t mo; + if (k >= 0 && v[i].lo <= v[k].hi) { + if (v[i].hi <= v[k].hi) continue; + if (v[i].hi - v[k].lo <= MERGE_MAX && mtl_addr(v[k].lo, v[i].hi - v[k].lo, &mb, &mo)) { v[k].hi = v[i].hi; continue; } + } + if (!mtl_addr(v[i].lo, v[i].hi - v[i].lo, &mb, &mo)) continue; + v[++k] = v[i]; + } + c->n = k + 1; + c->off = malloc(sizeof *c->off * (size_t)(c->n ? c->n : 1)); + if (!c->off) die("uniform capture"); + size_t total = 0; + for (int i = 0; i < c->n; i++) { c->off[i] = total; total += (size_t)(v[i].hi - v[i].lo); } + c->bytes = malloc(total ? total : 1); + if (!c->bytes) die("uniform capture bytes"); + for (int i = 0; i < c->n; i++) memcpy(c->bytes + c->off[i], (const void *)(uintptr_t)v[i].lo, (size_t)(v[i].hi - v[i].lo)); + c->r = v; v = NULL; + atomic_fetch_add(&g_cap_bytes, total); + } + free(v); + atomic_fetch_add(&g_cap_ns, runtime_now_ns() - t0); + atomic_fetch_add(&g_caps, 1); + return c; +} + +// The replay's table: the captured ranges in device memory of the slot (per-slot buffer, grown as needed). +typedef struct { uint64_t lo, hi; VkDeviceSize dev; _Atomic uint64_t drop_at; } urng; // drop_at: the first position not served (UINT64_MAX: none) +static urng *g_u; +static int g_nu, g_capu, g_uslot = -1; +static const uint8_t *g_ubytes; // the captured bytes (CPU reads of pointers kept in uniform buffers) +static size_t *g_uoff; +static ucap *g_cur; +typedef struct { VkBuffer buf; VkDeviceMemory mem; VkDeviceSize cap; } arena_t; +static arena_t g_arena[VKB_SLOTS]; +static vkb_cnt g_hits; +static uint64_t g_drops; +// GPU writes of submissions recorded so far (inline updates, copies, stores), kept until the GPU has completed them: a +// capture taken before that holds the bytes from before the write, so ranges such a write overlaps read memory (the GPU +// timeline) instead of the copy. +typedef struct { uint64_t lo, hi, fence; } pend_t; +static pend_t *g_pend; +static int g_npend, g_cappend; + +static int arena_get(int slot, VkDeviceSize need) { + arena_t *a = &g_arena[slot]; + if (a->cap >= need) return 1; + if (a->buf) { vk.vkDestroyBuffer(vkc.device, a->buf, NULL); vk.vkFreeMemory(vkc.device, a->mem, NULL); } // the slot's last submission is done + *a = (arena_t){0}; + VkDeviceSize cap = 1u << 20; + while (cap < need) cap *= 2; + VkBufferCreateInfo bi = {VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .size = cap, + .usage = VK_BUFFER_USAGE_UNIFORM_BUFFER_BIT | VK_BUFFER_USAGE_STORAGE_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT | + VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT}; + if (!vk_check(vk.vkCreateBuffer(vkc.device, &bi, NULL, &a->buf), "uniform copy buffer")) return 0; + VkMemoryRequirements req; + vk.vkGetBufferMemoryRequirements(vkc.device, a->buf, &req); + uint32_t type = UINT32_MAX; + for (uint32_t i = 0; i < vkc.memory.memoryTypeCount && type == UINT32_MAX; i++) + if ((req.memoryTypeBits & (1u << i)) && (vkc.memory.memoryTypes[i].propertyFlags & VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT)) type = i; + VkMemoryAllocateFlagsInfo fl = {VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_FLAGS_INFO, .flags = VK_MEMORY_ALLOCATE_DEVICE_ADDRESS_BIT}; + VkMemoryAllocateInfo ai = {VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO, &fl, .allocationSize = req.size, .memoryTypeIndex = type}; + if (type == UINT32_MAX || !vk_check(vk.vkAllocateMemory(vkc.device, &ai, NULL, &a->mem), "uniform copy memory")) { + vk.vkDestroyBuffer(vkc.device, a->buf, NULL); *a = (arena_t){0}; return 0; + } + vk.vkBindBufferMemory(vkc.device, a->buf, a->mem, 0); + a->cap = cap; + return 1; +} + +void mtl_sub_snapshot_ubos_cap(const uint64_t *h, int n, void *capv) { // replay thread, the submission just begun + ucap *c = capv ? capv : mtl_sub_capture_ubos(h, n); // without the translation thread: the replay is in the submit call + if (g_cur) { free(g_cur->r); free(g_cur->off); free(g_cur->bytes); free(g_cur); g_cur = NULL; } + g_nu = 0; g_uslot = -1; g_ubytes = NULL; g_uoff = NULL; + if (!c) return; + g_cur = c; + if (!c->n || vkb_slot() < 0) return; + if (g_capu < c->n) { g_capu = c->n; if (!(g_u = realloc(g_u, sizeof *g_u * (size_t)g_capu))) die("uniform copy table"); } + uint64_t done = mtl_done(); + int k = 0; + for (int i = 0; i < g_npend; i++) if (g_pend[i].fence > done) g_pend[k++] = g_pend[i]; // completed writes are in memory + g_npend = k; + VkDeviceSize at = 0; // each range keeps its address's position within 256 bytes: bindings stay aligned as in the pool + for (int i = 0; i < c->n; i++) { + at = ((at + 255) & ~(VkDeviceSize)255) + (c->r[i].lo & 255); + g_u[i] = (urng){c->r[i].lo, c->r[i].hi, at, UINT64_MAX}; + for (int j = 0; j < g_npend && g_u[i].drop_at; j++) // a GPU write the capture predates: read memory + if (g_pend[j].fence > c->done && g_pend[j].lo < c->r[i].hi && c->r[i].lo < g_pend[j].hi) { g_u[i].drop_at = 0; g_drops++; } + at += c->r[i].hi - c->r[i].lo; + } + int slot = vkb_slot(); + if (!arena_get(slot, at)) return; + VkBuffer rb; + VkDeviceSize ro; + uint8_t *p = vkb_ring_alloc(at, 256, &rb, &ro); + for (int i = 0; i < c->n; i++) memcpy(p + g_u[i].dev, c->bytes + c->off[i], (size_t)(c->r[i].hi - c->r[i].lo)); + VkCommandBuffer cb = vkb_cmd(); + VkBufferCopy r = {ro, 0, at}; + vk.vkCmdCopyBuffer(cb, rb, g_arena[slot].buf, 1, &r); + VkMemoryBarrier2 mb = {VK_STRUCTURE_TYPE_MEMORY_BARRIER_2, .srcStageMask = VK_PIPELINE_STAGE_2_COPY_BIT, .srcAccessMask = VK_ACCESS_2_TRANSFER_WRITE_BIT, + .dstStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, .dstAccessMask = VK_ACCESS_2_MEMORY_READ_BIT}; + VkDependencyInfo dep = {VK_STRUCTURE_TYPE_DEPENDENCY_INFO, .memoryBarrierCount = 1, .pMemoryBarriers = &mb}; + vk.vkCmdPipelineBarrier2(cb, &dep); + g_nu = c->n; g_uslot = slot; g_ubytes = c->bytes; g_uoff = c->off; +} + +static int dropped(const urng *u) { return atomic_load_explicit(&u->drop_at, memory_order_acquire) <= vkb_seq(); } +static int find(uint64_t a) { // the first range ending above a + int lo = 0, hi = g_nu; + while (lo < hi) { int m = (lo + hi) / 2; if (g_u[m].hi <= a) lo = m + 1; else hi = m; } + return lo; +} +int vkb_ucap_range(uint64_t addr, uint64_t size, VkBuffer *buf, VkDeviceSize *off) { + if (!g_nu || g_uslot != vkb_slot()) return 0; // another slot (a split submission): its copy is not this one's + int i = find(addr); + if (i >= g_nu || addr < g_u[i].lo || addr + size > g_u[i].hi || dropped(&g_u[i])) return 0; + *buf = g_arena[g_uslot].buf; + *off = g_u[i].dev + (addr - g_u[i].lo); + vkb_cnt_add(g_hits, 1); + return 1; +} +int vkb_ucap_why(uint64_t addr, uint64_t size) { // development: 0 in the copy, 1 not captured, 2 dropped by a GPU write, 3 other slot + if (!g_nu) return 1; + if (g_uslot != vkb_slot()) return 3; + int i = find(addr); + if (i >= g_nu || addr < g_u[i].lo || addr + size > g_u[i].hi) return 1; + return dropped(&g_u[i]) ? 2 : 0; +} +void vkb_ucap_read(uint64_t addr, void *out, size_t n) { // CPU reads of uniform memory (pointers in uniform buffers) + if (g_nu) { + int i = find(addr); + if (i < g_nu && addr >= g_u[i].lo && addr + n <= g_u[i].hi && !dropped(&g_u[i])) { memcpy(out, g_ubytes + g_uoff[i] + (addr - g_u[i].lo), n); return; } + } + memcpy(out, (const void *)(uintptr_t)addr, n); +} +void vkb_ucap_drop(uint64_t addr, uint64_t len) { // a GPU write to [addr, addr+len): copies it overlaps are dropped + if (vkb_mt_recording()) return; // the walk of the submission dropped it already (vkb_mt.c) + uint64_t at = vkb_seq(); + for (int i = find(addr); i < g_nu && g_u[i].lo < addr + len; i++) + if (atomic_load(&g_u[i].drop_at) > at) { atomic_store_explicit(&g_u[i].drop_at, at, memory_order_release); g_drops++; } + if (!vkb_ucap_on()) return; + if (g_npend == g_cappend) { g_cappend = g_cappend ? g_cappend * 2 : 1024; if (!(g_pend = realloc(g_pend, sizeof *g_pend * (size_t)g_cappend))) die("pending GPU writes"); } + g_pend[g_npend++] = (pend_t){addr, addr + len, mtl_last_fence() + 1}; +} +void vkb_ucap_stats(uint64_t *caps, uint64_t *bytes, uint64_t *ns, uint64_t *hits, uint64_t *drops) { + *caps = g_caps; *bytes = g_cap_bytes; *ns = g_cap_ns; *hits = vkb_cnt_sum(g_hits); *drops = g_drops; +} diff --git a/src/host/account.c b/src/host/account.c new file mode 100644 index 0000000..9a795b6 --- /dev/null +++ b/src/host/account.c @@ -0,0 +1,97 @@ +/* Offline local user identity: one persistent user, validated handles. + * Ported from MacNativeGames games/totk/runtime/account.c @b75be8c (MIT). Changes: guest-register binding through + * HLE_NATIVE; Windows file locking is not needed (the game slot guarantees one game process); the lock-guard + * diagnostics are dropped. */ +#include +#include +#include +#include +#include +#include +#include +#include + +#include "host.h" +#include "../runtime/hle.h" + +typedef struct { uint64_t lo, hi; } Uid; +typedef struct { Uid uid; uint64_t token; } UserHandle; +static Uid local; +static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER; +static uint64_t tokens[64], next_token = 1; + +const char *runtime_profile_dir(void) { + const char *dir = getenv("TOTK_PROFILE_DIR"); + return dir ? dir : "userdata"; +} + +static void initialize(void) { + pthread_mutex_lock(&lock); + if (local.lo || local.hi) { pthread_mutex_unlock(&lock); return; } + const char *dir = runtime_profile_dir(); + if (mkdir(dir, 0700) && errno != EEXIST) runtime_fail("create local account directory"); + char path[4096]; + if (snprintf(path, sizeof path, "%s/account.uid", dir) >= (int)sizeof path) runtime_fail("local account path length"); + int fd = open(path, O_RDWR | O_CREAT | O_BINARY, 0600); + if (fd < 0) runtime_fail("open local account identity"); + struct stat st; + if (fstat(fd, &st) || !S_ISREG(st.st_mode)) runtime_fail("invalid local account identity file"); + if (st.st_size == 0) { + do { local.lo = ((uint64_t)arc4random() << 32) | arc4random(); local.hi = ((uint64_t)arc4random() << 32) | arc4random(); } while (!local.lo && !local.hi); + if (write(fd, &local, sizeof local) != sizeof local || fsync(fd)) runtime_fail("persist local account identity"); + } else if (st.st_size != sizeof local || pread(fd, &local, sizeof local, 0) != sizeof local || (!local.lo && !local.hi)) { + runtime_fail("corrupt local account identity"); + } + close(fd); + pthread_mutex_unlock(&lock); + fprintf(stderr, "[totk] persistent local account initialized\n"); +} +static bool same(const Uid *uid) { return uid && uid->lo == local.lo && uid->hi == local.hi; } +static bool valid(const UserHandle *h) { + if (!h || !same(&h->uid)) return false; + for (unsigned i = 0; i < 64; i++) if (tokens[i] && tokens[i] == h->token) return true; + return false; +} +static unsigned preselected(UserHandle *out) { + if (!out) runtime_fail("account null user handle"); + initialize(); + pthread_mutex_lock(&lock); + for (unsigned i = 0; i < 64; i++) + if (!tokens[i]) { tokens[i] = next_token++; *out = (UserHandle){local, tokens[i]}; pthread_mutex_unlock(&lock); return 1; } + pthread_mutex_unlock(&lock); + return 0; +} +static uint32_t user_id(Uid *out, const UserHandle *h) { + pthread_mutex_lock(&lock); + bool ok = out && valid(h); + if (ok) *out = local; + pthread_mutex_unlock(&lock); + return ok ? 0 : 0x7c67; +} +static void close_user(const UserHandle *h) { + pthread_mutex_lock(&lock); + if (!valid(h)) runtime_fail("invalid local account handle close"); + for (unsigned i = 0; i < 64; i++) if (tokens[i] == h->token) { tokens[i] = 0; break; } + pthread_mutex_unlock(&lock); +} +static uint32_t existence(unsigned char *out, const Uid *uid) { + if (!out || !uid) return 0x7c67; + initialize(); + *out = same(uid); + return 0; +} +static uint32_t nickname(char *out, const Uid *uid) { + initialize(); + if (!out || !same(uid)) return 0x7c67; + memset(out, 0, 33); + memcpy(out, "Local", 5); + return 0; +} +void runtime_account_uid(uint64_t out[2]) { initialize(); out[0] = local.lo; out[1] = local.hi; } + +HLE_NATIVE("_ZN2nn7account10InitializeEv", initialize, 'v') +HLE_NATIVE("_ZN2nn7account22TryOpenPreselectedUserEPNS0_10UserHandleE", preselected, 'b') +HLE_NATIVE("_ZN2nn7account9CloseUserERKNS0_10UserHandleE", close_user, 'v') +HLE_NATIVE("_ZN2nn7account9GetUserIdEPNS0_3UidERKNS0_10UserHandleE", user_id, 'w') +HLE_NATIVE("_ZN2nn7account16GetUserExistenceEPbRKNS0_3UidE", existence, 'w') +HLE_NATIVE("_ZN2nn7account11GetNicknameEPNS0_8NicknameERKNS0_3UidE", nickname, 'w') diff --git a/src/host/audio_effects.c b/src/host/audio_effects.c new file mode 100644 index 0000000..86f2cc9 --- /dev/null +++ b/src/host/audio_effects.c @@ -0,0 +1,184 @@ +/* From MacNativeGames games/totk/runtime/audio_effects.c @b75be8c (MIT). [port] marks the Windows changes. */ +/* Delay, standard reverb and compressor DSP, ported from Ryujinx.Audio. + Copyright (c) Ryujinx Team and Contributors. MIT license: + third_party/licenses/Ryujinx-Audio.txt. Runtime samples are normalized floats. */ +#include "audio_effects.h" +#include "host.h" // [port] runtime.h +#include +#include +#include + +typedef struct { float *data; unsigned max, count, at; } DelayLine; +struct AudioEffectDsp { + unsigned type, revision, channels; + union { AudrenDelayParam delay; AudrenReverbParam reverb; AudrenCompressorParam compressor; } p; + DelayLine line[10]; + float history[6], direct[4], previous[4], decay_rate; + unsigned early_time[10], pre_time; + float early_gain[10]; + float input_mean, gain_mean, last_target, last_alpha, output_gain, makeup; + float stats_min, stats_mean, stats_last[6]; +}; +static int channel_ok(unsigned n) { return n==1 || n==2 || n==4 || n==6; } +static unsigned sample_count(unsigned rate, float ms) { return (unsigned)nearbyintf(rate*ms); } +/* The reference DelayLine wraps at CurrentSampleCount-1, including when its + backing allocation is larger. Preserve this timing for sample comparisons. */ +static void line_set(DelayLine *d, unsigned n) { d->count=nmax?n:d->max; d->at=0; } +static void line_init(DelayLine *d, unsigned rate, float ms) { + d->max=sample_count(rate,ms); d->data=calloc((size_t)d->max+1,sizeof(float)); + if(!d->data)runtime_fail("audio effect delay allocation"); line_set(d,d->max); +} +static float line_read(const DelayLine *d) { return d->data[d->at]; } +static float line_update(DelayLine *d,float v) { + float old=line_read(d);d->data[d->at++]=v; + if(d->at >= (d->count?d->count-1:0))d->at=0; + return old; +} +static float line_tap(const DelayLine *d,unsigned sample,int offset) { + int at=(int)d->at-(int)sample-offset; + if(at<0)at+=(int)d->count; + if(at<0 || (unsigned)at>d->max)runtime_fail("audio effect delay tap bounds"); + return d->data[at]; +} +static float gain10(float x) { return x>=0?1:x<=-5.3f?0:powf(10,x); } +static const float fdn_time[4][4]={{53.953247f,79.192566f,116.238770f,130.615295f},{53.953247f,79.192566f,116.238770f,170.615295f},{5,10,5,10},{47.03f,71,103,170}}; +static const float decay_time[4][4]={{7,9,13,17},{7,9,13,17},{1,1,1,1},{7,7,13,9}}; +/* Early reflections are indexed by EarlyMode. These are Ryujinx.Audio's five + tables; its own version reads the first row for every mode. */ +static const float early_time[5][10]={ + {0,3.5f,2.8f,3.9f,2.7f,13.4f,7.9f,8.4f,9.9f,12}, + {0,11.8f,5.5f,11.2f,10.4f,38.1f,22.2f,29.6f,21.2f,24.8f}, + {0,41.5f,20.5f,41.3f,0,29.5f,33.8f,45.2f,46.8f,0}, + {33.1f,43.3f,22.8f,37.9f,14.9f,35.3f,17.9f,34.2f,0,43.3f}, + {0} +}; +static const float early_gain[5][10]={ + {.70f,.68f,.70f,.68f,.70f,.68f,.70f,.68f,.68f,.68f}, + {.70f,.68f,.70f,.68f,.70f,.68f,.68f,.68f,.68f,.68f}, + {.50f,.70f,.70f,.68f,.50f,.68f,.68f,.70f,.68f,0}, + {.93f,.92f,.87f,.86f,.94f,.81f,.80f,.77f,.76f,.65f}, + {0} +}; +static const float pre_time[5]={12.5f,40,50,50,0}; +static void validate_delay(const AudrenDelayParam *p) { + unsigned rate=(p->sample_rate>>14)/1000; + if(!channel_ok(p->channels) || p->channel_maxchannels || rate<1 || rate>192 || p->delay_max>20000 || p->delay>p->delay_max) + runtime_fail("audio delay parameters"); +} +static void validate_reverb(const AudrenReverbParam *p) { + if(!channel_ok(p->channels) || p->channel_maxchannels || p->sample_rate<16384 || p->sample_rate>192*16384 || + p->early_mode<0 || p->early_mode>4 || p->late_mode<0 || p->late_mode>3 || p->decay<=0 || p->hf_decay<=0 || p->pre_delay<0) + runtime_fail("audio standard reverb parameters"); +} +static void validate_compressor(const AudrenCompressorParam *p) { + if(!channel_ok(p->channels) || p->channel_maxchannels || !isfinite(p->threshold) || !(p->ratio>0) || !isfinite(p->ratio) || + !(p->input_gain>=0 && p->input_gain<=1) || !(p->attack>=0 && p->attack<=1) || !(p->release>=0 && p->release<=1) || !isfinite(p->output_gain)) + runtime_fail("audio compressor parameters"); +} +AudioEffectDsp *audio_effect_create(unsigned type,const void *param,unsigned revision) { + AudioEffectDsp *s=calloc(1,sizeof *s);if(!s)runtime_fail("audio effect allocation"); + s->type=type;s->revision=revision;s->gain_mean=s->last_target=s->stats_min=1; + if(type==3) { + AudrenDelayParam p;memcpy(&p,param,sizeof p);validate_delay(&p);s->channels=p.channels; + for(unsigned c=0;cchannels;c++){line_init(&s->line[c],(p.sample_rate>>14)/1000,p.delay_max);line_set(&s->line[c],sample_count((p.sample_rate>>14)/1000,p.delay));} + p.status=0;audio_effect_update(s,&p); + } else if(type==4) { + AudrenReverbParam p;memcpy(&p,param,sizeof p);validate_reverb(&p);s->channels=p.channels;unsigned khz=(unsigned)p.sample_rate>>14; + for(int i=0;i<4;i++){line_init(&s->line[i],khz,fdn_time[1][i]);line_init(&s->line[i+4],khz,decay_time[1][i]);} + line_init(&s->line[8],khz,revision>=3?350:150);line_init(&s->line[9],khz,5); + p.status=0;audio_effect_update(s,&p); + } else if(type==9) { + AudrenCompressorParam p;memcpy(&p,param,sizeof p);validate_compressor(&p);s->channels=p.channels; + p.status=0;audio_effect_update(s,&p); + } else runtime_fail("audio effect DSP type"); + return s; +} +void audio_effect_free(AudioEffectDsp *s) { if(s){for(int i=0;i<10;i++)free(s->line[i].data);free(s);} } +void audio_effect_update(AudioEffectDsp *s,const void *param) { + if(s->type==3){ + AudrenDelayParam p;memcpy(&p,param,sizeof p);validate_delay(&p); + if(p.channels!=s->channels || (s->p.delay.sample_rate && (p.sample_rate!=s->p.delay.sample_rate || p.delay_max!=s->p.delay.delay_max)))runtime_fail("audio delay requires reset"); + s->p.delay=p; + } else if(s->type==4){ + AudrenReverbParam p;memcpy(&p,param,sizeof p);validate_reverb(&p);if(p.channels!=s->channels)runtime_fail("audio reverb requires reset");s->p.reverb=p; + if(p.status>=2)return; + unsigned rate=(unsigned)p.sample_rate>>14; + for(int i=0;i<10;i++){ + unsigned n=sample_count(rate,early_time[p.early_mode][i]+p.pre_delay/16384.f);s->early_time[i]=(nline[8].max?n:s->line[8].max)+1; + s->early_gain[i]=early_gain[p.early_mode][i]*(p.early_gain/16384.f); + } + if(p.channels==2){s->early_gain[4]*=.5f;s->early_gain[5]*=.5f;} + unsigned n=sample_count(rate,pre_time[p.early_mode]+p.pre_delay/16384.f);s->pre_time=nline[8].max?n:s->line[8].max; + float decay=p.decay/16384.f,hf=p.hf_decay/16384.f,co=cosf((1280.f/rate)*(3.14159265358979323846f/180)); + s->decay_rate=.6f*(1-p.coloration/16384.f); + for(int i=0;i<4;i++){ + line_set(&s->line[i],sample_count(rate,fdn_time[p.late_mode][i]));line_set(&s->line[4+i],sample_count(rate,decay_time[p.late_mode][i])); + float a=-3.f*(s->line[i].count+s->line[i+4].count),b=a/(decay*rate),c=0; + if(hf<.995f){float e=gain10(((1/hf-1)*2)/100*(b/10)),f=1-e,g=2-e*2*co;c=(g-sqrtf(g*g-f*f*4))/(f*2);} + s->direct[i]=gain10(b/1000)*(1-c)*.7071f;s->previous[i]=c;s->history[i]=0; + } + } else { + AudrenCompressorParam p;memcpy(&p,param,sizeof p);validate_compressor(&p);if(p.channels!=s->channels)runtime_fail("audio compressor requires reset");s->p.compressor=p; + if(p.status<2){s->last_alpha=p.attack;s->makeup=p.makeup?p.threshold*.5f*(1/p.ratio-1)-3:0;s->output_gain=powf(10,(p.output_gain+s->makeup)/20);} + } +} +static void delay_process(AudioEffectDsp *s,float *const in[6],float *const out[6],int frames) { + const AudrenDelayParam *p=&s->p.delay;unsigned n=s->channels; + float fb=p->feedback/16384.f,spread=p->spread/16384.f,base=(1-spread)*fb*.98f,cross=spread*fb*.98f/(n>=4?2:1),lp=.95f*(p->lowpass/16384.f); + static const int quad[4][2]={{1,2},{0,3},{0,3},{1,2}}; + static const int surround[6][2]={{2,4},{2,5},{0,1},{3,3},{0,5},{1,4}}; + for(int k=0;kline[c]);} + for(unsigned c=0;c=4 && !(n==6 && c==3)){const int *q=n==4?quad[c]:surround[c];feedback+=(d[q[0]]+d[q[1]])*cross;} + v[c]=x[c]*(p->in_gain/16384.f)+feedback; + float z=lp*s->history[c]+v[c]*(1-lp);s->history[c]=z;line_update(&s->line[c],z); + out[c][k]=(x[c]*(p->dry_gain/16384.f)+d[c]*(p->out_gain/16384.f))/64; + } + } +} +static void reverb_process(AudioEffectDsp *s,float *const in[6],float *const out[6],int frames) { + const AudrenReverbParam *p=&s->p.reverb;unsigned n=s->channels; + static const int mono[10]={0,0,0,0,0,0,0,0,0,0},stereo[10]={0,0,1,1,0,1,0,0,1,1},quad[10]={0,0,1,1,0,1,2,2,3,3},surround[10]={0,0,1,1,4,4,2,2,3,3}; + static const int feedback_stereo[4]={2,0,3,1},output_stereo[4]={0,0,1,1}; + const int *map=n==1?mono:n==2?stereo:n==4?quad:surround; + for(int k=0;kline[8],s->early_time[i],0)*s->early_gain[i];y[map[i]]+=tap;if(n==6)y[5]+=tap;} + if(n==6)y[5]*=.2f; + for(unsigned c=0;creverb_gain/16384.f);}line_update(&s->line[8],sum); + unsigned at=s->pre_time;if(at>=s->line[8].count)at=s->line[8].count-1; + float late=line_tap(&s->line[8],at,-1)*(p->late_gain/16384.f); + for(int i=0;i<4;i++){v[i]=line_read(&s->line[i])*s->direct[i]+s->history[i]*s->previous[i];s->history[i]=v[i];} + fb[0]=v[2]+v[1];fb[1]=-v[0]-v[3];fb[2]=v[0]-v[3];fb[3]=v[1]-v[2]; + for(int i=0;i<4;i++){float d=line_read(&s->line[4+i]),z=fb[i]+late-s->decay_rate*d;v[i]=line_update(&s->line[4+i],z)+z*s->decay_rate;line_update(&s->line[i],v[i]);} + if(n==1)for(int i=0;i<4;i++)y[0]+=v[i]; + else if(n==2)for(int i=0;i<4;i++)y[output_stereo[i]]+=v[feedback_stereo[i]]; + else {for(int i=0;i<4;i++)y[i]+=v[i];if(n==6){y[5]+=v[3];y[4]+=line_update(&s->line[9],(v[2]-v[3])*.5f);}} + for(unsigned c=0;cout_gain/16384.f)+x[c]*(p->dry_gain/16384.f))/64; + } +} +static void compressor_process(AudioEffectDsp *s,float *const in[6],float *const out[6],int frames) { + const AudrenCompressorParam *p=&s->p.compressor; + if(p->reset_statistics){s->stats_min=1;s->stats_mean=0;memset(s->stats_last,0,sizeof s->stats_last);} + for(int k=0;kchannels;c++){x[c]=in[c][k];mean+=x[c]*x[c];}mean/=s->channels; + s->input_mean+=p->input_gain*(mean-s->input_mean);float db=10*log10f(fmaxf(s->input_mean,1e-10f)),z=1; + if(db>=p->threshold-1.5f){float g=db>=p->threshold+1.5f?(1/p->ratio-1)*(db-p->threshold):(db-p->threshold+1.5f)*(db-p->threshold+1.5f)*(-(1-1/p->ratio)/6);z=powf(10,g/20);} + float alpha,target=z,diff=s->last_target-z; + if(diff>.08f)alpha=p->attack; + else if(diff<-.08f)alpha=p->release; + else {if(fabsf(s->gain_mean-z)>=.001f)target=s->last_target;alpha=s->last_alpha;} + s->gain_mean+=alpha*(z-s->gain_mean);s->last_target=target;s->last_alpha=alpha; + for(unsigned c=0;cchannels;c++){out[c][k]=x[c]*s->gain_mean*s->output_gain;s->stats_last[c]=fabsf(x[c]);} + s->stats_min=fminf(s->stats_min,s->gain_mean*s->output_gain);s->stats_mean=fmaxf(s->stats_mean,mean); + } +} +void audio_effect_process(AudioEffectDsp *s,float *const input[6],float *const output[6],int frames) { + if(s->type==3)delay_process(s,input,output,frames);else if(s->type==4)reverb_process(s,input,output,frames);else compressor_process(s,input,output,frames); +} +void audio_effect_statistics(AudioEffectDsp *s,void *output,unsigned bytes) { + if(s && s->type==9 && bytes>=32){float stats[8]={s->stats_mean,s->stats_min};memcpy(stats+2,s->stats_last,24);memcpy(output,stats,32);} +} diff --git a/src/host/audio_effects.h b/src/host/audio_effects.h new file mode 100644 index 0000000..b37b23e --- /dev/null +++ b/src/host/audio_effects.h @@ -0,0 +1,37 @@ +/* From MacNativeGames games/totk/runtime/audio_effects.h @b75be8c (MIT). [port] marks the Windows changes. */ +/* Additional DSP effects. Wire layouts and algorithms follow Ryujinx.Audio + (MIT, third_party/licenses/Ryujinx-Audio.txt). No device or guest-memory access. */ +#pragma once +#include +#include +typedef struct { + uint8_t input[6], output[6]; uint16_t channel_max, channels; + uint32_t delay_max, delay, sample_rate, in_gain, feedback, out_gain, dry_gain, spread, lowpass; + uint8_t status; +} AudrenDelayParam; +typedef struct { + uint8_t input[6], output[6]; uint16_t channel_max, channels; + int32_t sample_rate, early_mode, early_gain, pre_delay, late_mode, late_gain, + decay, hf_decay, coloration, reverb_gain, out_gain, dry_gain; + uint8_t status; +} AudrenReverbParam; +typedef struct { + uint8_t input[6], output[6]; uint16_t channel_max, channels; int32_t sample_rate; + float threshold, ratio; int32_t attack_time, release_time; + float input_gain, attack, release, output_gain; + uint8_t status, makeup, statistics, reset_statistics; +} AudrenCompressorParam; +typedef struct { uint8_t input[24], output[24]; float volume[24]; uint32_t count; } AudrenBufferMixParam; +typedef struct { uint8_t input[6], output[6]; int16_t numerator[3], denominator[2]; uint8_t channels, status; } AudrenBiquadEffectV1; +typedef struct { uint8_t input[6], output[6]; uint32_t padding; float numerator[3], denominator[2]; uint8_t channels, status; uint16_t reserved; } AudrenBiquadEffectV2; +_Static_assert(offsetof(AudrenDelayParam,status)==52,"delay wire layout"); +_Static_assert(offsetof(AudrenReverbParam,status)==64,"reverb wire layout"); +_Static_assert(offsetof(AudrenCompressorParam,status)==52,"compressor wire layout"); +_Static_assert(sizeof(AudrenBufferMixParam)==148,"buffer-mix wire layout"); +_Static_assert(sizeof(AudrenBiquadEffectV1)==24 && sizeof(AudrenBiquadEffectV2)==40,"biquad wire layouts"); +typedef struct AudioEffectDsp AudioEffectDsp; +AudioEffectDsp *audio_effect_create(unsigned type, const void *param, unsigned revision); +void audio_effect_free(AudioEffectDsp *s); +void audio_effect_update(AudioEffectDsp *s, const void *param); +void audio_effect_process(AudioEffectDsp *s, float *const input[6], float *const output[6], int frames); +void audio_effect_statistics(AudioEffectDsp *s, void *output, unsigned bytes); diff --git a/src/host/audio_mixer.c b/src/host/audio_mixer.c new file mode 100644 index 0000000..cb9360c --- /dev/null +++ b/src/host/audio_mixer.c @@ -0,0 +1,740 @@ +/* From MacNativeGames games/totk/runtime/audio_mixer.c @b75be8c (MIT). [port] marks the Windows changes. */ +/* Audio renderer mixing stage and the IAudioDevice service. + Per 5 ms frame (called by audio_renderer.c with the renderer lock held, before voices advance): + voices are decoded (PCM8/16/32, float, Nintendo DSP-ADPCM) and resampled to the output rate with linear + interpolation along exactly the sample path the voice clock consumes, scaled by voice volume and by the + channel-resource or splitter-destination volumes into the buffers of their destination mix; submixes are + folded into their destination in dependency order; the device sink's inputs (5.1 downmixed to stereo) go to + CoreAudio through a ring buffer. Splitter wire formats follow the public Ryujinx description (REV11: SNDH + header, SNDI infos with 12 reserved bytes after the ids, SNDD destinations, 16-byte padded section). + Two per-voice Q14 biquads run after resampling, before volume/routing. Delay, reverb, biquad, compressor, auxiliary/capture and limiter effects run on each completed mix in effect order. + Effect algorithms/layouts follow Ryujinx.Audio (MIT; third_party/licenses/Ryujinx-Audio.txt). */ +#include "host.h" // [port] runtime.h +#include "audio_mixer.h" +#include "audio_reverb.h" +#include "audio_effects.h" +#include // [port] output device (CoreAudio before) +#include +#include +#include +#include +#include + +#define MAX_FRAME 1024 +#define RING_FRAMES 16384 /* stereo frames, ~340 ms at 48 kHz */ +#define SPLITTER_UNUSED 0xffffffffu + +typedef struct { float vol[24]; int32_t dest; uint8_t used; AudrenBiquadFloat biquads[2]; float state[2][2]; } SplitDest; +typedef struct { int32_t ndest; int32_t *ids; } SplitInfo; +typedef struct { + Reverb3d *reverb; Reverb3dParam param; int pending; + float *delay, detector[6], compression[6], peak[6], minimum[6]; int position[6], delay_max, channels; + AudioEffectDsp *dsp; unsigned type; uint8_t wire[160]; + float biquad_z[6][2], numerator[3], denominator[2]; uint8_t biquad_channels; +} EffectState; +typedef struct Mixer { + float *buf; int nbuf, frame; + int *offset; /* first mix buffer of each mix id */ + SplitInfo *split; int nsplit; SplitDest *dest; int ndest; + int unit_ok; SDL_AudioStream *stream; /* [port] SDL3 output instead of a CoreAudio unit */ + float ring[RING_FRAMES * 2]; _Atomic uint64_t wr, rd, underruns, overruns; + _Atomic int playing; int ready; _Atomic uint64_t missing_frames; + _Atomic uint64_t flush_until; _Atomic unsigned stop_epoch; unsigned callback_epoch; + _Atomic uint64_t callback_frames,callback_nonzero; + EffectState *effect; int *effect_order; unsigned effect_reported; + uint64_t *sink_address; uint32_t *sink_size, *sink_offset, *sink_old, *sink_written; + unsigned reported; /* one-time reports */ + float voice_scratch[6][MAX_FRAME],filter_scratch[MAX_FRAME],output_scratch[MAX_FRAME*2]; + double level_sum,level_peak;uint64_t level_count,level_frames; + FILE *capture;float *capture_buf;uint64_t capture_frames,capture_limit,capture_skip; + pthread_t capture_thread;pthread_mutex_t capture_lock;pthread_cond_t capture_cond; + int capture_started,capture_done,capture_closing,capture_worker; +} Mixer; + +static _Atomic float device_volume = 1.0f; +static unsigned audio_revision(const AudioRenderer *r) { unsigned byte=r->revision>>24;return byte>='0'?byte-'0':0; } +static void report_once(Mixer *m, unsigned bit, const char *msg, int value) { + if (m->reported & (1u << bit)) return; + m->reported |= 1u << bit; fprintf(stderr, "[totk-audio] 【红】%s (%d)\n", msg, value); +} + +/* Diagnostic recording is optional. The render thread only copies to pre-touched + memory: filesystem writes and close/flush belong to this worker. */ +static void *capture_write(void *ctx) { + Mixer *m=ctx; + pthread_mutex_lock(&m->capture_lock); + while(!m->capture_done && !m->capture_closing)pthread_cond_wait(&m->capture_cond,&m->capture_lock); + uint64_t n=m->capture_frames; + pthread_mutex_unlock(&m->capture_lock); + int ok=fwrite(m->capture_buf,2*sizeof(float),(size_t)n,m->capture)==n; + if(fclose(m->capture))ok=0; + fprintf(stderr,"[totk-audio] diagnostic capture %s: %llu stereo frames\n",ok?"complete":"write failed",(unsigned long long)n); + return NULL; +} +static void capture_copy(Mixer *m,const float *lr,int frames) { + if(!m->capture_worker || m->capture_done)return; + if(!m->capture_started)for(int k=0;k<2*frames;k++)if(lr[k]!=0){m->capture_started=1;break;} + if(!m->capture_started)return; + uint64_t skip=m->capture_skip<(uint64_t)frames?m->capture_skip:(uint64_t)frames;m->capture_skip-=skip; + uint64_t n=m->capture_limit-m->capture_frames;if(n>(uint64_t)frames-skip)n=(uint64_t)frames-skip; + memcpy(m->capture_buf+2*m->capture_frames,lr+2*skip,(size_t)n*2*sizeof(float));m->capture_frames+=n; + if(m->capture_frames==m->capture_limit){ + pthread_mutex_lock(&m->capture_lock);m->capture_done=1;pthread_cond_signal(&m->capture_cond);pthread_mutex_unlock(&m->capture_lock); + } +} + +/* ---------- CoreAudio output ---------- */ +/* How much audio the output waits for before it starts, and again after an underrun: TOTK_AUDIO_PREFILL_MS, rounded up + to whole 5-ms render blocks (the renderer's period), default 20 ms = four blocks (the user, 2026-10-18 evening; it was + 40 ms that morning, and ten sessions at 0 / 20 / 40 ms had no underrun). The sound runs that much behind the picture. + With the 512-frame (10.7 ms) device buffer, 20 ms leaves about 10 ms for a stall of the mixing thread (40 ms left about + 30 ms); 0 starts as soon as one device buffer is there. + Returns -1 for a value that is not a number of milliseconds from 0 to 500. */ +static int prefill_blocks_for(const char *e) { + if (!e || !*e) return 4; + char *end; double ms = strtod(e, &end); + if (end == e || *end || !(ms >= 0) || ms > 500) return -1; + return (int)ceil(ms / 5.0); +} +static int prefill_blocks(void) { // read once before the device starts (audio_mixer_open); the callback only reads it + static int blocks = -2; + if (blocks == -2) { + const char *e = getenv("TOTK_AUDIO_PREFILL_MS"); int b = prefill_blocks_for(e); + if (b < 0) { fprintf(stderr, "[totk-audio] 【红】TOTK_AUDIO_PREFILL_MS=%s is not 0..500 ms: using 20\n", e); b = 4; } + fprintf(stderr, "[totk-audio] output starts after %d ms of queued audio (TOTK_AUDIO_PREFILL_MS%s)\n", b * 5, e && *e ? "" : ", default"); + blocks = b; + } + return blocks; +} +/* [port] The device callback: fills `frames` stereo float frames at `out` (CoreAudio render callback before). */ +static int pull(void *ctx, float *out, uint32_t frames) { + Mixer *m = ctx; + atomic_fetch_add(&m->callback_frames,frames); + unsigned epoch=atomic_load_explicit(&m->stop_epoch,memory_order_acquire); + uint64_t rd = atomic_load(&m->rd), wr = atomic_load_explicit(&m->wr, memory_order_acquire), have = wr - rd; + if(epoch!=m->callback_epoch){uint64_t until=atomic_load_explicit(&m->flush_until,memory_order_acquire);if(until>rd){rd=untilrd,rd,memory_order_release);}have=wr-rd;m->callback_epoch=epoch;m->ready=0;} + if(!atomic_load_explicit(&m->playing,memory_order_acquire)){ + memset(out,0,(size_t)frames*2*sizeof(float));m->ready=0;atomic_store_explicit(&m->rd,wr,memory_order_release);return 0; + } + if(!m->ready){ + /* prefill_blocks() 5-ms blocks (default four, 20 ms) of scheduling headroom for the producer. + A real underrun still counts and requires a fresh prefill. */ + uint64_t prefill=(uint64_t)m->frame*(uint64_t)prefill_blocks();if(prefillRING_FRAMES)prefill=RING_FRAMES; + if(haveready=1; + } + uint32_t n = frames < have ? frames : (uint32_t)have; + uint64_t nonzero=0; + for (uint32_t i = 0; i < n; i++) { uint64_t k = (rd + i) % RING_FRAMES; out[2 * i] = m->ring[2 * k]; out[2 * i + 1] = m->ring[2 * k + 1];if(out[2*i]!=0 || out[2*i+1]!=0)nonzero++; } + atomic_fetch_add(&m->callback_nonzero,nonzero); + if (n < frames) { memset(out + 2 * n, 0, (frames - n) * 2 * sizeof(float));atomic_fetch_add(&m->underruns,1);atomic_fetch_add(&m->missing_frames,frames-n);m->ready=0; } + atomic_store_explicit(&m->rd, rd + n, memory_order_release); + return 0; +} +/* [port] SDL3 asks for `additional` bytes of float stereo; the ring gives them (pull, with its prefill and underrun + accounting). SDL's own device buffer is 480 frames (window_sdl.c). */ +static void SDLCALL device_pull(void *ctx, SDL_AudioStream *stream, int additional, int total) { + (void)total; + Mixer *m = ctx; float lr[2 * 512]; + for (int frames = additional / 8; frames > 0;) { + int n = frames < 512 ? frames : 512; + pull(m, lr, (uint32_t)n); + SDL_PutAudioStreamData(stream, lr, n * 8); + frames -= n; + } +} +static void output_open(Mixer *m, int rate) { + if (getenv("TOTK_AUDIO_OFF")) { fprintf(stderr, "[totk-audio] TOTK_AUDIO_OFF: mixing without an output device\n"); return; } + if (!SDL_WasInit(SDL_INIT_AUDIO)) { fprintf(stderr, "[totk-audio] no window (headless): mixing without an output device\n"); return; } + SDL_AudioSpec spec = {SDL_AUDIO_F32, 2, rate}; + m->stream = SDL_OpenAudioDeviceStream(SDL_AUDIO_DEVICE_DEFAULT_PLAYBACK, &spec, device_pull, m); + if (!m->stream) { fprintf(stderr, "[totk-audio] 【红】no output device (%s): mixing without sound\n", SDL_GetError()); return; } + m->unit_ok = 1; + SDL_ResumeAudioStreamDevice(m->stream); + fprintf(stderr, "[totk-audio] SDL3 output %d Hz stereo on \"%s\"\n", rate, SDL_GetAudioDeviceName(SDL_GetAudioStreamDevice(m->stream))); +} +static void output_push(Mixer *m, const float *lr, int frames) { + uint64_t wr = atomic_load(&m->wr), rd = atomic_load_explicit(&m->rd, memory_order_acquire); + if (m->unit_ok && wr + (uint64_t)frames - rd > RING_FRAMES) { atomic_fetch_add(&m->overruns, 1); return; } /* device stalled: drop this frame */ + for (int i = 0; i < frames; i++) { uint64_t k = (wr + (uint64_t)i) % RING_FRAMES; m->ring[2 * k] = lr[2 * i]; m->ring[2 * k + 1] = lr[2 * i + 1]; } + atomic_store_explicit(&m->wr, wr + (uint64_t)frames, memory_order_release); + if (!m->unit_ok) atomic_store(&m->rd, wr + (uint64_t)frames); /* no device: the ring only keeps the latest frames */ +} + +void audio_mixer_open(AudioRenderer *r) { + Mixer *m = calloc(1, sizeof *m); + if (!m) runtime_fail("audio mixer allocation"); + m->frame = r->param.sample_count; m->nbuf = r->param.mix_buffer_count; + if (m->frame <= 0 || m->frame > MAX_FRAME || m->nbuf <= 0 || m->nbuf > 512) runtime_fail("audio mixer frame / buffer count"); + m->buf = calloc((size_t)m->nbuf * (size_t)m->frame, sizeof(float)); + m->offset = calloc((size_t)r->param.submix_count + 1, sizeof(int)); + m->nsplit = r->param.splitter_count; m->ndest = r->param.splitter_destination_count; + m->split = calloc((size_t)(m->nsplit ? m->nsplit : 1), sizeof *m->split); m->dest = calloc((size_t)(m->ndest ? m->ndest : 1), sizeof *m->dest); + if (!m->buf || !m->offset || !m->split || !m->dest) runtime_fail("audio mixer tables"); + m->effect = calloc((size_t)(r->param.effect_count ? r->param.effect_count : 1), sizeof *m->effect); + m->effect_order = calloc((size_t)(r->param.effect_count ? r->param.effect_count : 1), sizeof *m->effect_order); + if (!m->effect || !m->effect_order) runtime_fail("audio effect tables"); + size_t sinks=(size_t)r->param.sink_count; + m->sink_address=calloc(sinks,sizeof(uint64_t));m->sink_size=calloc(sinks,sizeof(uint32_t)); + m->sink_offset=calloc(sinks,sizeof(uint32_t));m->sink_old=calloc(sinks,sizeof(uint32_t));m->sink_written=calloc(sinks,sizeof(uint32_t)); + if(!m->sink_address || !m->sink_size || !m->sink_offset || !m->sink_old || !m->sink_written)runtime_fail("audio sink state allocation"); + r->mixer = m; + atomic_store(&m->playing,r->state==0); + const char *capture=getenv("TOTK_AUDIO_CAPTURE"); + if(capture && *capture){m->capture=fopen(capture,"wbx");if(m->capture){ + const char *skip=getenv("TOTK_AUDIO_CAPTURE_DELAY_S");char *end=NULL;unsigned long seconds=skip?strtoul(skip,&end,10):0; + if(skip && (!*skip || *end || seconds>600))runtime_fail("audio capture delay must be 0..600 seconds"); + m->capture_limit=(uint64_t)r->param.sample_rate*30;m->capture_skip=(uint64_t)r->param.sample_rate*seconds; + size_t bytes=(size_t)m->capture_limit*2*sizeof(float);m->capture_buf=calloc(1,bytes); + if(!m->capture_buf)runtime_fail("audio capture memory"); + for(size_t i=0;icapture_buf)[i]=0; + if(pthread_mutex_init(&m->capture_lock,NULL) || pthread_cond_init(&m->capture_cond,NULL) || pthread_create(&m->capture_thread,NULL,capture_write,m))runtime_fail("audio capture worker"); + m->capture_worker=1; + fprintf(stderr,"[totk-audio] diagnostic capture armed: %d Hz stereo float32, 30 seconds after %lu-second delay from first nonzero frame\n",r->param.sample_rate,seconds); + }else fprintf(stderr,"[totk-audio] diagnostic capture unavailable (file exists or cannot be opened)\n");} + prefill_blocks(); // decided before the device's callback can run + output_open(m, r->param.sample_rate); +} +void audio_mixer_set_running(AudioRenderer *r,int running){ + Mixer *m=r->mixer;if(!m)return; + if(!running){atomic_store_explicit(&m->flush_until,atomic_load_explicit(&m->wr,memory_order_acquire),memory_order_release);atomic_fetch_add_explicit(&m->stop_epoch,1,memory_order_release);} + atomic_store_explicit(&m->playing,running,memory_order_release); +} +void audio_mixer_close(AudioRenderer *r) { + Mixer *m = r->mixer; if (!m) return; + if (m->stream) SDL_DestroyAudioStream(m->stream); /* [port] */ + if(m->capture_worker){ + pthread_mutex_lock(&m->capture_lock);m->capture_closing=1;pthread_cond_signal(&m->capture_cond);pthread_mutex_unlock(&m->capture_lock); + pthread_join(m->capture_thread,NULL);pthread_cond_destroy(&m->capture_cond);pthread_mutex_destroy(&m->capture_lock);free(m->capture_buf); + } + for (int i = 0; i < m->nsplit; i++) free(m->split[i].ids); + for (int i = 0; i < r->param.effect_count; i++) { audio_reverb_free(m->effect[i].reverb); audio_effect_free(m->effect[i].dsp); free(m->effect[i].delay); } + free(m->sink_address);free(m->sink_size);free(m->sink_offset);free(m->sink_old);free(m->sink_written); + free(m->effect); free(m->effect_order); + free(m->buf); free(m->offset); free(m->split); free(m->dest); free(m); r->mixer = NULL; +} + +/* ---------- routing input: splitters and mixes ---------- */ +static void splitters_update(AudioRenderer *r, const uint8_t *p, uint32_t n) { + Mixer *m = r->mixer; + if (!n) return; + uint32_t hdr[3]; + if (n < 0x20) runtime_fail("audio splitter section header"); + memcpy(hdr, p, 12); + if (hdr[0] != 0x48444e53u) runtime_fail("audio splitter header magic (SNDH)"); + uint32_t ni = hdr[1], nd = hdr[2], at = 0x20; + { static int dumped; if (!dumped++ && getenv("TOTK_AUDIO_DUMP")) { FILE *f = fopen("analysis/audio-splitter-section.bin", "wb"); if (f) { fwrite(p, 1, n, f); fclose(f); } + fprintf(stderr, "[totk-audio] splitter section %u bytes, header infos %u destinations %u\n", n, ni, nd); } } + for (uint32_t i = 0; i < ni; i++) { + uint32_t h[4]; if (at + 16 > n) runtime_fail("audio splitter info overruns section"); + memcpy(h, p + at, 16); at += 16; + if (h[0] != 0x49444e53u || h[1] >= (uint32_t)m->nsplit || h[3] > (uint32_t)m->ndest || at + 4u * h[3] > n) { + FILE *f = fopen("analysis/audio-splitter-section-bad.bin", "wb"); if (f) { fwrite(p, 1, n, f); fclose(f); } + fprintf(stderr, "[totk-audio] splitter info %u at %#x: %08x %08x %08x %08x (section %u bytes, infos %u, destinations %u)\n", i, at - 16, h[0], h[1], h[2], h[3], n, ni, nd); + runtime_fail("audio splitter info (SNDI)"); + } + SplitInfo *s = &m->split[h[1]]; free(s->ids); + s->ndest = (int32_t)h[3]; s->ids = calloc(h[3] ? h[3] : 1, sizeof(int32_t)); + if (!s->ids) runtime_fail("audio splitter destinations"); + memcpy(s->ids, p + at, 4u * h[3]); at += 4u * h[3] + 12; /* 12 reserved bytes follow the ids (measured, TOTK REV11) */ + if (at > n) runtime_fail("audio splitter info padding overruns section"); + for (uint32_t k = 0; k < h[3]; k++) if (s->ids[k] < 0 || s->ids[k] >= m->ndest) runtime_fail("audio splitter destination id"); + } + for (uint32_t i = 0; i < nd; i++) { + unsigned rev=(r->revision>>24)-'0',stride=rev>=15?0xa8:rev>=12?0x90:0x70; + uint8_t e[0xa8]={0}; if (at>n||stride>n-at) runtime_fail("audio splitter destination overruns section"); + memcpy(e, p + at, stride); at += stride; + uint32_t magic, id; int32_t dst; memcpy(&magic, e, 4); memcpy(&id, e + 4, 4); memcpy(&dst, e + 0x68, 4); + if (magic != 0x44444e53u || id >= (uint32_t)m->ndest) runtime_fail("audio splitter destination (SNDD)"); + SplitDest *d = &m->dest[id]; memcpy(d->vol, e + 8, sizeof d->vol); d->dest = dst; + d->used=e[rev>=15?0x9c:rev>=12?0x84:0x6c]; + for(unsigned f=0;f<2;f++){ + AudrenBiquadFloat b={0}; + if(rev>=15)memcpy(&b,e+0x6c+f*sizeof b,sizeof b); + else if(rev>=12){AudrenBiquad q;memcpy(&q,e+0x6c+f*sizeof q,sizeof q);b.enable=q.enable;for(unsigned c=0;c<3;c++)b.numerator[c]=q.numerator[c]/16384.0f;for(unsigned c=0;c<2;c++)b.denominator[c]=q.denominator[c]/16384.0f;} + if(b.enable){for(unsigned c=0;c<3;c++)if(!isfinite(b.numerator[c]))runtime_fail("audio splitter nonfinite numerator");for(unsigned c=0;c<2;c++)if(!isfinite(b.denominator[c]))runtime_fail("audio splitter nonfinite denominator");} + if(!b.enable||!d->biquads[f].enable||!d->used)memset(d->state[f],0,sizeof d->state[f]); + d->biquads[f]=b; + } + } + if (at <= n && n - at < 16) at = n; /* the section is padded to 16 bytes; the padding is not cleared by the game (measured) */ + if (at != n) { fprintf(stderr, "[totk-audio] splitter section %u bytes, parsed %u (infos %u, destinations %u)\n", n, at, ni, nd); runtime_fail("audio splitter section size"); } +} +void audio_mixer_update_routing(AudioRenderer *r, const uint8_t *splitters, uint32_t splitter_bytes, const uint8_t *mixes, uint32_t mix_bytes, int mix_count) { + splitters_update(r, splitters, splitter_bytes); + if (mix_bytes == (uint32_t)mix_count * sizeof(AudrenMixIn)) { memcpy(r->mixes, mixes, mix_bytes); return; } + /* MixInParameterDirtyOnly: 0x20-byte header {u32 magic 0xCAFEBABE, u32 count, pad} followed by the dirty mixes */ + if (mix_bytes >= 0x20 && (mix_bytes - 0x20) % sizeof(AudrenMixIn) == 0) { + uint32_t magic, n; memcpy(&magic, mixes, 4); memcpy(&n, mixes + 4, 4); + if (magic != 0xcafebabeu || n * sizeof(AudrenMixIn) != mix_bytes - 0x20) runtime_fail("audio dirty mix header"); + for (uint32_t i = 0; i < n; i++) { + AudrenMixIn m; memcpy(&m, mixes + 0x20 + i * sizeof m, sizeof m); + if (m.mix_id < 0 || m.mix_id >= mix_count) runtime_fail("audio mix id"); + r->mixes[m.mix_id] = m; + } + return; + } + fprintf(stderr, "[totk-audio] mix section %u bytes for %d mixes\n", mix_bytes, mix_count); + runtime_fail("audio mix section size"); +} + +/* Effect parameters arrive once per IPC update, while the DSP may process multiple frames. */ +void audio_mixer_update_effects(AudioRenderer *r) { + Mixer *m = r->mixer; + for (int i=0;iparam.effect_count;i++) { + const AudrenEffectIn *e=&r->effects[i]; EffectState *s=&m->effect[i]; + unsigned revision=audio_revision(r); + int reset=e->is_new || s->type!=e->type; + if(reset){audio_effect_free(s->dsp);s->dsp=NULL;memset(s->biquad_z,0,sizeof s->biquad_z);} + if(e->type==3 || e->type==4 || e->type==9) { + unsigned status,channels;int dimensions_changed=0; + if(e->type==3){AudrenDelayParam p,old;memcpy(&p,e->specific,sizeof p);memcpy(&old,s->wire,sizeof old);status=p.status;channels=p.channels;dimensions_changed=p.channels!=old.channels || p.sample_rate!=old.sample_rate || p.delay_max!=old.delay_max;} + else if(e->type==4){AudrenReverbParam p,old;memcpy(&p,e->specific,sizeof p);memcpy(&old,s->wire,sizeof old);status=p.status;channels=p.channels;dimensions_changed=p.channels!=old.channels || p.sample_rate!=old.sample_rate;} + else {AudrenCompressorParam p,old;memcpy(&p,e->specific,sizeof p);memcpy(&old,s->wire,sizeof old);status=p.status;channels=p.channels;dimensions_changed=p.channels!=old.channels;} + if(status==0 || dimensions_changed){audio_effect_free(s->dsp);s->dsp=NULL;} + if(e->enabled && channels){if(!s->dsp)s->dsp=audio_effect_create(e->type,e->specific,revision);else audio_effect_update(s->dsp,e->specific);} + } else if(s->dsp){audio_effect_free(s->dsp);s->dsp=NULL;} + if(e->type==6){ + unsigned status; + if(revision>=15){AudrenBiquadEffectV2 p;memcpy(&p,e->specific,sizeof p);s->biquad_channels=p.channels;status=p.status;memcpy(s->numerator,p.numerator,sizeof p.numerator);memcpy(s->denominator,p.denominator,sizeof p.denominator);} + else {AudrenBiquadEffectV1 p;memcpy(&p,e->specific,sizeof p);s->biquad_channels=p.channels;status=p.status;for(int k=0;k<3;k++)s->numerator[k]=p.numerator[k]/16384.f;for(int k=0;k<2;k++)s->denominator[k]=p.denominator[k]/16384.f;} + if(status==0 || (status==1 && revision<6))memset(s->biquad_z,0,sizeof s->biquad_z); + if(e->enabled && (s->biquad_channels<1 || s->biquad_channels>6))runtime_fail("biquad effect channel count"); + } + s->type=e->type;memcpy(s->wire,e->specific,sizeof s->wire); + if(e->type!=5 || e->is_new) { audio_reverb_free(s->reverb); s->reverb=NULL; s->pending=1; } + if(e->type==7) { + AudrenLimiterParam p; memcpy(&p,e->specific,sizeof p); + if(e->is_new || !s->delay || p.status==0 || p.channels!=s->channels || p.delay_max!=s->delay_max) { + free(s->delay);s->delay=NULL;s->delay_max=p.delay_max;s->channels=p.channels; + memset(s->detector,0,sizeof s->detector);memset(s->position,0,sizeof s->position); + for(int c=0;c<6;c++)s->compression[c]=s->minimum[c]=1; + memset(s->peak,0,sizeof s->peak); + if(e->enabled) { + if(p.channels<1 || p.channels>6 || p.delay_max<1 || p.delay_max>192000)runtime_fail("limiter delay dimensions"); + s->delay=calloc((size_t)p.channels*p.delay_max,sizeof(float));if(!s->delay)runtime_fail("limiter delay allocation"); + } + } + if(p.statistics_reset) { memset(s->peak,0,sizeof s->peak); for(int c=0;c<6;c++)s->minimum[c]=1; } + } else if(s->delay) { free(s->delay);s->delay=NULL; } + if(e->type!=5) continue; + Reverb3dParam p; memcpy(&p,e->specific,sizeof p); + if(s->reverb && p.sample_rate!=s->param.sample_rate) { audio_reverb_free(s->reverb); s->reverb=NULL; } + if(memcmp(&p,&s->param,72) || p.status<2) s->pending=1; + s->param=p; + } +} + +/* ---------- voice decoding along the voice clock's sample path ---------- */ +typedef struct { ServerVoice *v; unsigned index; uint64_t offset; int32_t loops; uint8_t valid[4]; int ended; int16_t h1, h2; int32_t abuf; uint64_t aat; } Walk; +static int fmt_bytes(int f) { return f == FMT_PCM8 ? 1 : f == FMT_PCM16 ? 2 : f==FMT_PCM24?3:f == FMT_PCM32 || f == FMT_PCMF32 ? 4 : 0; } +/* DSP-ADPCM: 8-byte frames of a predictor/scale byte and 14 4-bit samples; the history runs across samples. */ +static int16_t adpcm_next(Walk *w, const AudrenWaveBuffer *b, const int16_t *coef, uint64_t i) { + const uint8_t *d = (const uint8_t *)(uintptr_t)b->address; + uint64_t frame = i / 14, nib = i % 14; + if ((frame + 1) * 8 > b->size) return 0; + uint8_t ps = d[frame * 8]; int scale = 1 << (ps & 15), pred = (ps >> 4) & 7; + int n = (d[frame * 8 + 1 + nib / 2] >> ((nib & 1) ? 0 : 4)) & 15; if (n >= 8) n -= 16; + int64_t s = (int64_t)n * scale * 2048 + 1024 + (int64_t)coef[pred * 2] * w->h1 + (int64_t)coef[pred * 2 + 1] * w->h2; + s >>= 11; if (s > 32767) s = 32767; if (s < -32768) s = -32768; + w->h2 = w->h1; w->h1 = (int16_t)s; return (int16_t)s; +} +static float sample_at(Walk *w, int ch) { + if (w->ended || !w->valid[w->index]) return 0; + ServerVoice *v = w->v; const AudrenWaveBuffer *b = &v->buffers[w->index]; + uint64_t i = (uint64_t)b->start + w->offset; int f = v->in.sample_format, cc = (int)v->in.channel_count; + if (f == FMT_ADPCM) { + if (cc != 1 || !v->in.extra_address || v->in.extra_size < 32) return 0; + const int16_t *coef = (const int16_t *)(uintptr_t)v->in.extra_address; + if (w->abuf != (int32_t)w->index || w->aat > i + 1) { /* (re)start this buffer: from its context, else from the frame boundary with carried history */ + w->abuf = (int32_t)w->index; + if (b->context_address && b->context_size >= 6) { const int16_t *c = (const int16_t *)(uintptr_t)b->context_address; w->h1 = c[1]; w->h2 = c[2]; w->aat = (uint64_t)b->start; } + else w->aat = (uint64_t)b->start - (uint64_t)b->start % 14; + if (i < w->aat) w->aat = i; + } + while (w->aat <= i) adpcm_next(w, b, coef, w->aat++); + return w->h1 / 32768.0f; + } + int bytes = fmt_bytes(f); if (!bytes || ch >= cc) return 0; + uint64_t at = (i * (uint64_t)cc + (uint64_t)ch) * (uint64_t)bytes; + if (at + (uint64_t)bytes > b->size) return 0; + const uint8_t *p = (const uint8_t *)(uintptr_t)b->address + at; + switch (f) { + case FMT_PCM8: return (int8_t)p[0] / 128.0f; + case FMT_PCM16: { int16_t s; memcpy(&s, p, 2); return s / 32768.0f; } + case FMT_PCM24: {uint32_t word=(uint32_t)p[0]|(uint32_t)p[1]<<8|(uint32_t)p[2]<<16;return ((int32_t)(word<<8)>>8)/8388608.f;} + case FMT_PCM32: { int32_t s; memcpy(&s, p, 4); return (float)(s / 2147483648.0); } + default: { float s; memcpy(&s, p, 4); return s; } + } +} +static void walk_skip_empty(Walk *w) { /* the same as skip_empty in audio_renderer.c: no samples, played on arrival */ + while (!w->ended && w->valid[w->index] && audio_buffer_empty(&w->v->buffers[w->index])) { + const AudrenWaveBuffer *b = &w->v->buffers[w->index]; + if (b->loop && b->loop_count < 0) break; + w->valid[w->index] = 0; w->offset = 0; w->loops = 0; w->index = (w->index + 1) & 3; + if (b->end_of_stream) w->ended = 1; + } +} +static void walk_step(Walk *w) { /* the same transitions as advance_voice in audio_renderer.c */ + if (w->ended || !w->valid[w->index]) { w->ended = 1; return; } + const AudrenWaveBuffer *b = &w->v->buffers[w->index]; + uint64_t length = (uint64_t)(b->end - b->start); + if (++w->offset < length) return; + if (b->loop && (b->loop_count < 0 || w->loops < b->loop_count)) { w->loops++; w->offset = b->loop_start > b->start && b->loop_start < b->end ? (uint64_t)(b->loop_start - b->start) : 0; return; } + w->valid[w->index] = 0; w->offset = 0; w->loops = 0; w->index = (w->index + 1) & 3; + if (b->end_of_stream) w->ended = 1; + walk_skip_empty(w); +} +/* Decodes `frames` output samples per channel into out[c][k]; the renderer advances the voice afterwards by the same amount. */ +static void voice_render(AudioRenderer *r, ServerVoice *v, float out[6][MAX_FRAME], int frames) { + const double step = v->in.sample_rate * (v->in.pitch > 0 ? v->in.pitch : 1.0) / r->param.sample_rate; + Walk w = {v, v->index, v->offset, v->loops_done, {0}, 0, v->adpcm.history[0], v->adpcm.history[1], v->adpcm.buffer, v->adpcm.at}; + memcpy(w.valid, v->valid, sizeof w.valid); + walk_skip_empty(&w); + int cc = (int)v->in.channel_count; double frac = v->fraction; + for (int k = 0; k < frames; k++) { + Walk next = w; walk_step(&next); + for (int c = 0; c < cc; c++) { float s0 = sample_at(&w, c), s1 = sample_at(&next, c); out[c][k] = (float)(s0 + (s1 - s0) * frac); } + frac += step; + while (frac >= 1.0) { frac -= 1.0; walk_step(&w); } + } + v->adpcm.history[0] = w.h1; v->adpcm.history[1] = w.h2; v->adpcm.buffer = w.abuf; v->adpcm.at = w.aat; +} + +/* ---------- mixing ---------- */ +static float *mixbuf(Mixer *m, int i) { return m->buf + (size_t)i * (size_t)m->frame; } +static void accumulate(Mixer *m, int buffer, const float *src, float vol, int frames) { + if (vol == 0 || buffer < 0 || buffer >= m->nbuf) return; + float *d = mixbuf(m, buffer); for (int k = 0; k < frames; k++) d[k] += src[k] * vol; +} +static int mix_valid(AudioRenderer *r, int32_t id) { return id >= 0 && id <= r->param.submix_count && r->mixes[id].in_use; } +static int mix_depth(AudioRenderer *r, int id) { + int d = 0; + for (int guard = 0; id > 0 && guard < 64; guard++, d++) { + if (!mix_valid(r, r->mixes[id].dest_mix_id)) return d + 1; + id = r->mixes[id].dest_mix_id; + } + return d; +} + +/* Guest custom effects exchange int32 samples in two rings. Each ring has CPU and DSP + 0x40-byte headers followed by samples; only the DSP header's owned cursor is advanced. */ +static void mix_aux(AudioRenderer *r, int mix, const AudrenEffectIn *e) { + Mixer *m=r->mixer; AudrenAuxParam p; memcpy(&p,e->specific,sizeof p); + if(!p.send_info || !p.return_info || !p.capacity || !p.channels) return; + if(p.channels>24 || p.capacity>16*1024*1024)runtime_fail("auxiliary effect dimensions"); + uint8_t *send=ipc_guest(p.send_info,128+(uint64_t)p.capacity*4), *ret=ipc_guest(p.return_info,128+(uint64_t)p.capacity*4); + uint32_t *sh=(uint32_t *)(send+64),*rh=(uint32_t *)(ret+64); int32_t *sd=(int32_t *)(send+128),*rd=(int32_t *)(ret+128); + uint32_t wr=sh[1],read=rh[0],total=p.channels*(uint32_t)m->frame; + for(unsigned c=0;c=r->mixes[mix].buffer_count || p.output[c]>=r->mixes[mix].buffer_count)runtime_fail("auxiliary effect mix buffer"); + float *input=mixbuf(m,m->offset[mix]+p.input[c]),*output=mixbuf(m,m->offset[mix]+p.output[c]); + if(!e->enabled) { if(input!=output)memmove(output,input,m->frame*sizeof(float)); continue; } + uint64_t w=(uint64_t)wr+c*m->frame,at=(uint64_t)read+c*m->frame; + for(int k=0;kframe;k++) { + double scaled=trunc((double)input[k]*32768); int32_t value=scaled>INT32_MAX?INT32_MAX:scaledframe;k++)output[k]=at<=p.capacity?rd[(at+(unsigned)k)%p.capacity]/32768.f:0; + if(c==p.channels-1) { if(w<=p.capacity)sh[1]=(wr+total)%p.capacity; if(at<=p.capacity)rh[0]=(read+total)%p.capacity; } + } + if(!e->enabled) { sh[0]=sh[1]=sh[3]=0; rh[0]=rh[1]=rh[3]=0; } +} + +void audio_mixer_effect_result(AudioRenderer *r, int id, void *output, unsigned size) { + Mixer *m=r->mixer; + if(size>=48 && r->effects[id].type==7) { memcpy(output,m->effect[id].peak,24);memcpy((uint8_t *)output+24,m->effect[id].minimum,24); } + if(r->effects[id].type==9)audio_effect_statistics(m->effect[id].dsp,output,size); +} +static void mix_limiter(AudioRenderer *r, int mix, int id) { + Mixer *m=r->mixer; EffectState *s=&m->effect[id]; const AudrenEffectIn *e=&r->effects[id]; + AudrenLimiterParam p;memcpy(&p,e->specific,sizeof p); + if(p.channels<1 || p.channels>6) { if(e->enabled)runtime_fail("limiter channel count");return; } + if(e->enabled && (!s->delay || p.delay_min<1 || p.delay_min>p.delay_max || p.attack<0 || p.attack>1 || p.release<0 || p.release>1))runtime_fail("limiter parameters"); + for(int c=0;c=r->mixes[mix].buffer_count || p.output[c]>=r->mixes[mix].buffer_count)runtime_fail("limiter mix buffer"); + float *input=mixbuf(m,m->offset[mix]+p.input[c]),*output=mixbuf(m,m->offset[mix]+p.output[c]); + if(!e->enabled) { if(input!=output)memmove(output,input,m->frame*sizeof(float));continue; } + for(int k=0;kframe;k++) { + float x=(input[k]*(32768.f/32767.f))*p.input_gain,level=fabsf(x); + float coeff=level>s->detector[c]?p.attack:p.release;s->detector[c]+=coeff*(level-s->detector[c]); + float attenuation=s->detector[c]>p.threshold?p.threshold/s->detector[c]:1; + coeff=s->compression[c]>attenuation?p.attack:p.release;s->compression[c]+=coeff*(attenuation-s->compression[c]); + int slot=c*p.delay_max+s->position[c];output[k]=s->delay[slot]*s->compression[c]*p.output_gain*(32767.f/32768.f); + s->delay[slot]=x;s->position[c]=(s->position[c]+1)%p.delay_min; + if(p.statistics_enabled) { s->peak[c]=fmaxf(s->peak[c],level);s->minimum[c]=fminf(s->minimum[c],s->compression[c]); } + } + } +} + +static int32_t effect_integer(float value) { + double x=trunc((double)value*32768);return !isfinite(x)?0:x>INT32_MAX?INT32_MAX:x=size){guest_prepare_write(base,size);return;} + uint32_t first=(uint32_t)bytes;if(first>size-at)first=size-at; + guest_prepare_write(base+at,first);if(bytes>first)guest_prepare_write(base,(size_t)bytes-first); +} +/* SDK SampleCounter::Add advances the producer counter itself, not the + producer-consumer difference or the ring cursor. Saturate on diff overflow. */ +static void effect_counter_add(uint32_t cpu,uint32_t *dsp,uint32_t count) { + uint32_t previous=*dsp-cpu,next=*dsp+count; + *dsp=next-cpumixer;AudrenAuxParam p;memcpy(&p,e->specific,sizeof p); + if(!p.send_info || !p.capacity || !p.channels)return; + if(p.channels>24 || p.capacity>16*1024*1024)runtime_fail("capture effect dimensions"); + uint8_t *base=ipc_guest(p.send_info,128+(uint64_t)p.capacity*4);uint32_t *cpu=(uint32_t *)base,*dsp=(uint32_t *)(base+64);int32_t *data=(int32_t *)(base+128); + guest_prepare_write(base+64,64); + if(!e->enabled){dsp[0]=dsp[1]=dsp[3]=0;return;} + uint32_t wr=dsp[1],total=p.channels*(uint32_t)m->frame; + for(unsigned c=0;c=r->mixes[mix].buffer_count)runtime_fail("capture effect input buffer"); + float *in=mixbuf(m,m->offset[mix]+p.input[c]);uint64_t at=(uint64_t)wr+c*m->frame; + if(at<=p.capacity)prepare_ring_write((uint8_t *)data,p.capacity*4,(uint32_t)(at%p.capacity)*4,(uint64_t)m->frame*4); + for(int k=0;kframe;k++){int32_t value=effect_integer(in[k]);in[k]=value/32768.f;if(at<=p.capacity)data[(at+(unsigned)k)%p.capacity]=value;} + } + if((uint64_t)wr+(p.channels-1)*(uint32_t)m->frame<=p.capacity){ + uint32_t diff=dsp[3]-cpu[3]; + if(diff>=p.capacity)effect_counter_add(cpu[2],&dsp[2],total); + dsp[1]=(wr+total)%p.capacity;effect_counter_add(cpu[3],&dsp[3],total); + } +} +static void mix_simple_effect(AudioRenderer *r,int mix,int id) { + Mixer *m=r->mixer;const AudrenEffectIn *e=&r->effects[id];EffectState *s=&m->effect[id]; + if(e->type==1){ + AudrenBufferMixParam p;memcpy(&p,e->specific,sizeof p);if(!e->enabled)return; + if(p.count>24)runtime_fail("buffer-mix effect count"); + for(unsigned c=0;c=r->mixes[mix].buffer_count || p.output[c]>=r->mixes[mix].buffer_count)runtime_fail("buffer-mix effect index"); + float *in=mixbuf(m,m->offset[mix]+p.input[c]),*out=mixbuf(m,m->offset[mix]+p.output[c]); + for(int k=0;kframe;k++)out[k]+=ceilf(in[k]*32768*p.volume[c])/32768; + }return; + } + unsigned channels;if(e->type==6)channels=s->biquad_channels;else {uint16_t n;memcpy(&n,e->specific+14,2);channels=n;} + if(channels<1 || channels>6){if(e->enabled)runtime_fail("audio DSP effect channels");return;} + float *in[6]={0},*out[6]={0};unsigned revision=audio_revision(r); + for(unsigned c=0;ctype==3 && revision<11)||(e->type==4 && revision>=11))){static const unsigned map[6]={0,1,4,5,2,3};idx=map[c];} + if(e->specific[idx]>=r->mixes[mix].buffer_count || e->specific[6+idx]>=r->mixes[mix].buffer_count)runtime_fail("audio DSP effect buffer index"); + in[c]=mixbuf(m,m->offset[mix]+e->specific[idx]);out[c]=mixbuf(m,m->offset[mix]+e->specific[6+idx]); + } + if(!e->enabled){for(unsigned c=0;cframe*sizeof(float));return;} + if(e->type==6){ + for(unsigned c=0;cframe;k++){ + float x=in[c][k],y=s->numerator[0]*x+s->biquad_z[c][0]; + s->biquad_z[c][0]=s->numerator[1]*x+s->denominator[0]*y+s->biquad_z[c][1];s->biquad_z[c][1]=s->numerator[2]*x+s->denominator[1]*y;out[c][k]=y; + } + } else {if(!s->dsp)runtime_fail("audio DSP effect state missing");audio_effect_process(s->dsp,in,out,m->frame);} +} +static void mix_effects(AudioRenderer *r, int mix) { + Mixer *m=r->mixer; int count=0; + for(int i=0;iparam.effect_count;i++) if(r->effects[i].type && r->effects[i].mix_id==mix) { + int j=count++; while(j>0 && r->effects[m->effect_order[j-1]].order>r->effects[i].order) { m->effect_order[j]=m->effect_order[j-1]; j--; } m->effect_order[j]=i; + } + for(int n=0;neffect_order[n]; const AudrenEffectIn *e=&r->effects[id]; EffectState *s=&m->effect[id]; + if(e->type==1 || e->type==3 || e->type==4 || e->type==6 || e->type==9){mix_simple_effect(r,mix,id);continue;} + if(e->type==2) { mix_aux(r,mix,e); continue; } + if(e->type==8) { mix_capture(r,mix,e);continue; } + if(e->type==7) { mix_limiter(r,mix,id); continue; } + if(e->type!=5) { + if(e->enabled && e->type<32 && !(m->effect_reported&(1u<type))) { m->effect_reported|=1u<type; fprintf(stderr,"[totk-audio] 【红】effect type %u not implemented\n",e->type); } + continue; + } + Reverb3dParam *p=&s->param; float *input[6]={0},*output[6]={0}; + if(p->channels!=1 && p->channels!=2 && p->channels!=4 && p->channels!=6) { if(e->enabled)runtime_fail("audio reverb channel count"); continue; } + for(int c=0;cchannels;c++) { + int idx=c; /* REV11+ channel resources use L,R,C,LFE,BL,BR; the reference DSP uses legacy order. */ + if(p->channels==6 && (r->revision>>24)>='0'+11) { static const int map[6]={0,1,4,5,2,3}; idx=map[c]; } + if(p->input[idx]>=r->mixes[mix].buffer_count || p->output[idx]>=r->mixes[mix].buffer_count)runtime_fail("audio reverb buffer index"); + input[c]=mixbuf(m,m->offset[mix]+p->input[idx]); output[c]=mixbuf(m,m->offset[mix]+p->output[idx]); + } + if(!e->enabled) { for(int c=0;cchannels;c++)if(input[c]!=output[c])memmove(output[c],input[c],m->frame*sizeof(float)); continue; } + if(!s->reverb)s->reverb=audio_reverb_create(p); else if(s->pending)audio_reverb_update(s->reverb,p); + s->pending=0; audio_reverb_process(s->reverb,input,output,m->frame); + } +} + +static const float *splitter_filter(SplitDest *d,const float *input,float *output,int frames){ + if(!d->biquads[0].enable&&!d->biquads[1].enable)return input; + memcpy(output,input,(size_t)frames*sizeof *output); + for(unsigned f=0;f<2;f++)if(d->biquads[f].enable){ + const AudrenBiquadFloat *b=&d->biquads[f];float *z=d->state[f]; + for(int k=0;knumerator[0]*x+z[0]; + z[0]=b->numerator[1]*x+b->denominator[0]*y+z[1]; + z[1]=b->numerator[2]*x+b->denominator[1]*y;output[k]=y; + } + } + return output; +} +typedef struct { + uint64_t address;uint32_t size,channels,samples,last_read; + uint8_t format,reserved[3],input[6],padding[2]; +} CircularSinkParam; +_Static_assert(__builtin_offsetof(CircularSinkParam,input)==28,"circular sink wire layout"); +static void mix_circular_sink(AudioRenderer *r,int id) { + Mixer *m=r->mixer;CircularSinkParam p;memcpy(&p,r->sinks[id].name,sizeof p); + if(!p.address || !p.size)return; + if(p.channels<1 || p.channels>6 || p.format!=FMT_PCM16 || p.size>256*1024*1024 || p.size%2)runtime_fail("circular audio sink parameters"); + if(m->sink_address[id]!=p.address || m->sink_size[id]!=p.size){m->sink_address[id]=p.address;m->sink_size[id]=p.size;m->sink_offset[id]=m->sink_old[id]=m->sink_written[id]=0;} + uint8_t *data=ipc_guest(p.address,p.size);uint32_t at=m->sink_offset[id]; + prepare_ring_write(data,p.size,at,(uint64_t)p.channels*m->frame*2); + float gain=r->mixes[0].in_use?r->mixes[0].volume:1; + for(unsigned c=0;c=m->nbuf)runtime_fail("circular audio sink input index"); + float *in=mixbuf(m,p.input[c]); + for(int k=0;kframe;k++){ + float scaled=in[k]*gain*32768;int16_t value=!isfinite(scaled)?0:scaled>=32767?32767:scaled<=-32768?-32768:(int16_t)scaled; + memcpy(data+at,&value,2);at=(at+2)%p.size; + } + } + m->sink_written[id]=m->sink_old[id];m->sink_old[id]=m->sink_offset[id];m->sink_offset[id]=at; +} +uint32_t audio_mixer_sink_written(AudioRenderer *r,int id) {Mixer *m=r->mixer;return m?m->sink_written[id]:0;} + +void audio_mixer_render(AudioRenderer *r, double seconds) { + (void)seconds; + Mixer *m = r->mixer; if (!m) return; + const int frames = m->frame, mc = r->param.submix_count + 1; + memset(m->buf, 0, (size_t)m->nbuf * (size_t)frames * sizeof(float)); + int at = 0; + for (int i = 0; i < mc; i++) { m->offset[i] = at; if (r->mixes[i].in_use) at += r->mixes[i].buffer_count; } + if (at > m->nbuf) { fprintf(stderr, "[totk-audio] mixes use %d buffers, renderer has %d\n", at, m->nbuf); runtime_fail("audio mix buffer layout"); } + + float (*vs)[MAX_FRAME]=m->voice_scratch,*filtered=m->filter_scratch; + for (int i = 0; i < r->param.voice_count; i++) { + ServerVoice *v = &r->voices[i]; + if (!v->in.in_use || v->state != SV_STARTED || !v->in.sample_rate || !v->valid[v->index]) continue; + int cc = (int)v->in.channel_count; if (cc < 1 || cc > 6) { report_once(m, 2, "voice channel count unsupported", cc); continue; } + voice_render(r, v, vs, frames); + for (int f = 0; f < 2; f++) if (v->in.biquads[f].enable) { + const AudrenBiquad *b = &v->in.biquads[f]; + const AudrenBiquadFloat *bf=&v->float_biquads[f]; + const float a0 = v->use_float_biquads?bf->numerator[0]:b->numerator[0]/16384.0f; + const float a1 = v->use_float_biquads?bf->numerator[1]:b->numerator[1]/16384.0f; + const float a2 = v->use_float_biquads?bf->numerator[2]:b->numerator[2]/16384.0f; + const float b1 = v->use_float_biquads?bf->denominator[0]:b->denominator[0]/16384.0f; + const float b2 = v->use_float_biquads?bf->denominator[1]:b->denominator[1]/16384.0f; + for (int c = 0; c < cc; c++) { + float *z = v->biquad_state[f][c]; + for (int k = 0; k < frames; k++) { + float x = vs[c][k], y = a0 * x + z[0]; + z[0] = a1 * x + b1 * y + z[1]; + z[1] = a2 * x + b2 * y; + vs[c][k] = y; + } + } + } + if (v->in.splitter_id != SPLITTER_UNUSED && v->in.splitter_id < (uint32_t)m->nsplit) { + const SplitInfo *s = &m->split[v->in.splitter_id]; + for (int c = 0; c < cc; c++) for (int k = c; k < s->ndest; k += cc) { + SplitDest *d = &m->dest[s->ids[k]]; + if (!d->used || !mix_valid(r, d->dest)) continue; + const float *source=splitter_filter(d,vs[c],filtered,frames); + for (int b = 0; b < r->mixes[d->dest].buffer_count && b < 24; b++) accumulate(m, m->offset[d->dest] + b, source, d->vol[b] * v->in.volume, frames); + } + } else if (mix_valid(r, v->in.mix_id)) { + for (int c = 0; c < cc; c++) { + int32_t id = v->in.channel_ids[c]; if (id < 0 || id >= r->param.voice_count * 6) continue; + const AudrenChannelIn *ch = &r->channels[id]; + for (int b = 0; b < r->mixes[v->in.mix_id].buffer_count && b < 24; b++) accumulate(m, m->offset[v->in.mix_id] + b, vs[c], ch->mix[b] * v->in.volume, frames); + } + } + } + /* submixes: deepest first, so every mix is complete before it is folded into its destination */ + int order[256], n = 0, depth[256]; + for (int i = 1; i < mc && n < 256; i++) if (r->mixes[i].in_use) { depth[n] = mix_depth(r, i); order[n++] = i; } + for (int a = 0; a < n; a++) for (int b = a + 1; b < n; b++) if (depth[b] > depth[a]) { int t = order[a]; order[a] = order[b]; order[b] = t; t = depth[a]; depth[a] = depth[b]; depth[b] = t; } + for (int a = 0; a < n; a++) { + mix_effects(r, order[a]); + const AudrenMixIn *x = &r->mixes[order[a]]; int src = m->offset[order[a]]; + if (x->dest_splitter_id != SPLITTER_UNUSED && x->dest_splitter_id < (uint32_t)m->nsplit) { + const SplitInfo *s = &m->split[x->dest_splitter_id]; + for (int c = 0; c < x->buffer_count && c < 24; c++) for (int k = c; k < s->ndest; k += x->buffer_count) { + SplitDest *d = &m->dest[s->ids[k]]; + if (!d->used || !mix_valid(r, d->dest)) continue; + const float *source=splitter_filter(d,mixbuf(m,src+c),filtered,frames); + for (int b = 0; b < r->mixes[d->dest].buffer_count && b < 24; b++) accumulate(m, m->offset[d->dest] + b, source, d->vol[b] * x->volume, frames); + } + } else if (mix_valid(r, x->dest_mix_id)) { + for (int c = 0; c < x->buffer_count && c < 24; c++) + for (int b = 0; b < r->mixes[x->dest_mix_id].buffer_count && b < 24; b++) accumulate(m, m->offset[x->dest_mix_id] + b, mixbuf(m, src + c), x->mix[c][b] * x->volume, frames); + } + } + if (r->mixes[0].in_use) mix_effects(r, 0); + /* device sink */ + const AudrenSinkIn *sink = NULL;int circular=0; + for (int i = 0; i < r->param.sink_count; i++) if (r->sinks[i].in_use) { + if(r->sinks[i].type!=SINK_CIRCULAR){m->sink_address[i]=0;m->sink_size[i]=m->sink_offset[i]=m->sink_old[i]=m->sink_written[i]=0;} + if (r->sinks[i].type == SINK_DEVICE && !sink) sink = &r->sinks[i]; + else if (r->sinks[i].type == SINK_CIRCULAR){mix_circular_sink(r,i);circular=1;} + } else {m->sink_address[i]=0;m->sink_size[i]=m->sink_offset[i]=m->sink_old[i]=m->sink_written[i]=0;} + float *lr=m->output_scratch; + memset(lr, 0, sizeof(float) * 2 * (size_t)frames); + if (sink && sink->input_count >= 1 && sink->input_count <= 6) { + const float gain = atomic_load(&device_volume) * (r->mixes[0].in_use ? r->mixes[0].volume : 1.0f); + const float *ch[6] = {0}; + for (uint32_t c = 0; c < sink->input_count; c++) if (sink->inputs[c] >= 0 && sink->inputs[c] < m->nbuf) ch[c] = mixbuf(m, sink->inputs[c]); + for (int k = 0; k < frames; k++) { + float l, rr; + if (sink->input_count >= 6) { /* FL FR C LFE RL RR -> stereo */ + float ff = sink->downmix_enabled ? sink->downmix[0] : 1.0f, cf = sink->downmix_enabled ? sink->downmix[1] : 0.7071f, sf = sink->downmix_enabled ? sink->downmix[2] : 0.7071f; + float C = ch[2] ? ch[2][k] : 0; + l = ff * (ch[0] ? ch[0][k] : 0) + cf * C + sf * (ch[4] ? ch[4][k] : 0); + rr = ff * (ch[1] ? ch[1][k] : 0) + cf * C + sf * (ch[5] ? ch[5][k] : 0); + } else if (sink->input_count >= 2) { l = ch[0] ? ch[0][k] : 0; rr = ch[1] ? ch[1][k] : 0; } + else l = rr = ch[0] ? ch[0][k] : 0; + l *= gain; rr *= gain; + lr[2 * k] = l > 1 ? 1 : l < -1 ? -1 : l; lr[2 * k + 1] = rr > 1 ? 1 : rr < -1 ? -1 : rr; + } + } else if (!sink && !circular) report_once(m, 5, "no device sink in use", r->param.sink_count); + { /* Per-renderer level report; independent renderers never share scratch or meter state. */ + for (int k = 0; k < 2 * frames; k++) { double x = lr[k];m->level_sum+=x*x;if(fabs(x)>m->level_peak)m->level_peak=fabs(x); } + m->level_count+=2*(uint64_t)frames;m->level_frames+=(uint64_t)frames; + if (m->level_frames >= (uint64_t)r->param.sample_rate * 5) { + fprintf(stderr, "[totk-audio] output level: rms %.4f peak %.4f over %.1f s, underruns %llu, overruns %llu, missing frames %llu, callback frames %llu / nonzero %llu\n", sqrt(m->level_sum/(double)m->level_count),m->level_peak, + m->level_frames/(double)r->param.sample_rate,(unsigned long long)atomic_load(&m->underruns),(unsigned long long)atomic_load(&m->overruns),(unsigned long long)atomic_load(&m->missing_frames),(unsigned long long)atomic_load(&m->callback_frames),(unsigned long long)atomic_load(&m->callback_nonzero)); + m->level_sum=m->level_peak=0;m->level_count=m->level_frames=0; + } } + capture_copy(m,lr,frames); + output_push(m, lr, frames); +} + +/* Probe hook: the latest mixed stereo frames. */ +int audio_mixer_read(AudioRenderer *r, float *lr, int frames) { + Mixer *m = r->mixer; uint64_t wr = atomic_load(&m->wr); if ((uint64_t)frames > wr || frames > RING_FRAMES) return 0; + for (int i = 0; i < frames; i++) { uint64_t k = (wr - (uint64_t)frames + (uint64_t)i) % RING_FRAMES; lr[2 * i] = m->ring[2 * k]; lr[2 * i + 1] = m->ring[2 * k + 1]; } + return frames; +} + +/* ---------- IAudioDevice: one stereo output device ---------- */ +static const char device_name[] = "AudioTvOutput"; +static void put_name(const IpcBuffer *b, IpcResponse *rs, int pointer) { + if (!b->size) return; + char *dst = ipc_guest(b->addr, b->size); size_t n = b->size < sizeof device_name ? b->size : sizeof device_name; + memset(dst, 0, b->size); memcpy(dst, device_name, n); + if (pointer) rs->xout[rs->nxout++] = (IpcBuffer){b->addr, b->size, b->index}; +} +static IpcBuffer out_buffer(const IpcRequest *rq, int *pointer) { + *pointer = !(rq->nb && rq->b[0].size) && rq->nc && rq->c[0].size; + return *pointer ? rq->c[0] : rq->nb ? rq->b[0] : (IpcBuffer){0}; +} +void audio_device_handler(IpcSession *s, const IpcRequest *rq, IpcResponse *rs) { + int pointer; + switch (rq->cmd) { + case 0: case 6: { IpcBuffer b = out_buffer(rq, &pointer); int32_t n = b.size >= 0x100 ? 1 : 0; if (n) put_name(&b, rs, pointer); IPC_OUT(rs, n); return; } /* ListAudioDeviceName(Auto) */ + case 1: case 7: device_volume = IPC_ARG(rq, float, 0); return; /* SetAudioDeviceOutputVolume(Auto) */ + case 2: case 8: {float volume=atomic_load(&device_volume);IPC_OUT(rs,volume);return;} /* GetAudioDeviceOutputVolume(Auto) */ + case 3: case 10: { IpcBuffer b = out_buffer(rq, &pointer); put_name(&b, rs, pointer); return; } /* GetActiveAudioDeviceName(Auto) */ + case 5: { int32_t n = 2; IPC_OUT(rs, n); return; } /* GetActiveChannelCount */ + case 13: { IpcBuffer b = out_buffer(rq, &pointer); put_name(&b, rs, pointer); return; } /* GetActiveAudioOutputDeviceName [13.0.0+] */ + case 14: { IpcBuffer b = out_buffer(rq, &pointer); int32_t n = b.size >= 0x100 ? 1 : 0; if (n) put_name(&b, rs, pointer); IPC_OUT(rs, n); return; } /* ListAudioOutputDeviceName [13.0.0+] */ + case 4: case 11: case 12: { static uint32_t ev; if (!ev) ev = runtime_event_new(); rs->copy[rs->ncopy++] = runtime_event_readable(ev); return; } /* device change events: never signalled, the device does not change */ + default: ipc_unimplemented(s, rq); + } +} diff --git a/src/host/audio_mixer.h b/src/host/audio_mixer.h new file mode 100644 index 0000000..8441ddf --- /dev/null +++ b/src/host/audio_mixer.h @@ -0,0 +1,42 @@ +/* From MacNativeGames games/totk/runtime/audio_mixer.h @b75be8c (MIT). [port] marks the Windows changes. */ +/* Shared server-side audio renderer state (audio_renderer.c) and the mixer interface (audio_mixer.c). */ +#pragma once +#include "audio_renderer.h" +#include "ipc.h" +#include + +typedef struct { + AudrenVoiceIn in; + AudrenBiquadFloat float_biquads[2]; uint8_t use_float_biquads; + AudrenWaveBuffer buffers[4]; uint8_t valid[4]; + uint8_t pending[4]; /* sent while the voice was not playing: valid once it plays (the DSP's IsSendToAudioProcessor) */ + uint8_t state; /* server play state (SV_*): the client's Start / Stop / Pause, plus Stopping until the next render frame */ + uint32_t flush, flush_from; /* FlushWaveBuffers requested since the last render frame, from the client's current buffer */ + unsigned index; uint64_t offset; double fraction; int32_t loops_done; + uint64_t played; uint32_t consumed, drops; + struct { int16_t history[2]; int32_t buffer; uint64_t at; } adpcm; /* DSP-ADPCM decode cursor (buffer slot, next sample) and history */ + float biquad_state[2][6][2]; /* filter, channel, transposed direct-form II delays */ +} ServerVoice; + +enum { SV_STOPPED, SV_STARTED, SV_PAUSED, SV_STOPPING }; +typedef struct AudioRenderer { + AudrenParameter param; + pthread_mutex_t lock; pthread_t thread; int running; + uint32_t state, revision, time_limit, event; int logged; + uint64_t behavior_flags, elapsed_frames; + uint64_t stop_released, empty_played, flushed, logged_released; /* wave buffers given back by Stop / without samples / by FlushWaveBuffers */ + ServerVoice *voices; AudrenPoolIn *pools; AudrenMixIn *mixes; AudrenSinkIn *sinks; AudrenEffectIn *effects; AudrenChannelIn *channels; + void *mixer; +} AudioRenderer; + +void audio_mixer_open(AudioRenderer *r); +void audio_mixer_set_running(AudioRenderer *r, int running); +void audio_mixer_close(AudioRenderer *r); +void audio_mixer_update_routing(AudioRenderer *r, const uint8_t *splitters, uint32_t splitter_bytes, const uint8_t *mixes, uint32_t mix_bytes, int mix_count); +void audio_mixer_update_effects(AudioRenderer *r); +void audio_mixer_effect_result(AudioRenderer *r, int id, void *output, unsigned size); +uint32_t audio_mixer_sink_written(AudioRenderer *r, int id); +void audio_mixer_render(AudioRenderer *r, double seconds); /* called with r->lock held, after audio_voices_prepare, before voices advance */ +void audio_voices_prepare(AudioRenderer *r); /* audio_renderer.c: each render frame's voice state changes (flush, stop, start) */ +int audio_buffer_empty(const AudrenWaveBuffer *b); /* no samples to decode: played as soon as it is reached */ +void audio_device_handler(IpcSession *s, const IpcRequest *rq, IpcResponse *rs); diff --git a/src/host/audio_renderer.c b/src/host/audio_renderer.c new file mode 100644 index 0000000..849a156 --- /dev/null +++ b/src/host/audio_renderer.c @@ -0,0 +1,600 @@ +/* From MacNativeGames games/totk/runtime/audio_renderer.c @b75be8c (MIT). [port] marks the Windows changes. */ +/* audren:u — host audio renderer service. The SDK's nn::audio client builds RequestUpdate input + unchanged; this service validates every section against the header, keeps server-side voice state, + advances it in real time (one 5 ms frame per render tick, signalling the frame event) and writes the + output status the client expects. Sample mixing to CoreAudio is done by audio_mixer.c. */ +#include "host.h" // [port] runtime.h +#include "ipc.h" +#include "audio_renderer.h" +#include "audio_mixer.h" +#include "audio_reverb.h" +#include +#include +#include +#include +#include +#include +#include // [port] mach / QoS +#undef near // [port] windef.h macro + +#define REV(n) (0x30564552u + ((uint32_t)(n) << 24)) /* "REVn" */ +#define WORK_BUFFER_BYTES 0x10000u /* host keeps its state in host memory */ +#define BEHAVIOR_OUT 0xb0u +#define RENDER_INFO_OUT 0x10u + +static int revision_number(uint32_t rev) { return (int)(rev >> 24) - '0'; } +static void need(int c, const char *m) { if (!c) { fprintf(stderr, "[totk-audio] %s\n", m); runtime_fail("audio renderer protocol"); } } + +static AudioRenderer *renderer_new(const AudrenParameter *p) { + AudioRenderer *r = calloc(1, sizeof *r); + need(r != NULL, "renderer allocation"); + r->param = *p; + r->revision=p->revision; + need(p->sample_rate == 48000 || p->sample_rate == 32000, "unsupported output rate"); + need(p->voice_count > 0 && p->voice_count <= 1024 && p->sink_count > 0 && p->sink_count <= 256 && p->effect_count >= 0 && p->effect_count <= 4096 && p->submix_count >= 0 && p->submix_count < 256, "renderer counts"); + r->voices = calloc((size_t)p->voice_count, sizeof *r->voices); + r->pools = calloc((size_t)p->effect_count + 4u * (size_t)p->voice_count, sizeof *r->pools); + r->mixes = calloc((size_t)p->submix_count + 1, sizeof *r->mixes); + r->sinks = calloc((size_t)p->sink_count, sizeof *r->sinks); + r->effects = calloc((size_t)(p->effect_count ? p->effect_count : 1), sizeof *r->effects); + r->channels = calloc((size_t)p->voice_count * 6, sizeof *r->channels); + need(r->voices && r->pools && r->mixes && r->sinks && r->effects && r->channels, "renderer tables"); + pthread_mutex_init(&r->lock, NULL); + r->event = runtime_event_new(); + r->state = 1; /* stopped */ + return r; +} + +/* ---------- RequestUpdate ---------- */ +typedef struct { const uint8_t *p, *end; } Reader; +static const uint8_t *take(Reader *in, uint32_t n, const char *what) { + if ((size_t)(in->end - in->p) < n) { fprintf(stderr, "[totk-audio] input section %s (%u bytes) overruns request\n", what, n); runtime_fail("audio renderer protocol"); } + const uint8_t *p = in->p; in->p += n; return p; +} +static void voice_reset(ServerVoice *v) { + memset(v->valid, 0, sizeof v->valid); memset(v->pending, 0, sizeof v->pending); + v->state = SV_STOPPED; v->flush = 0; v->flush_from = 0; + v->index = 0; v->offset = 0; v->fraction = 0; v->played = 0; v->consumed = 0; v->loops_done = 0; + memset(&v->adpcm, 0, sizeof v->adpcm); v->adpcm.buffer = -1; + memset(v->biquad_state, 0, sizeof v->biquad_state); +} +static void update_voice(AudioRenderer *r, const AudrenVoiceIn *in) { + need(in->id < (uint32_t)r->param.voice_count, "voice id out of range"); + ServerVoice *v = &r->voices[in->id]; + if (in->is_new || !in->in_use) voice_reset(v); + for (int i = 0; i < 2; i++) if (!in->biquads[i].enable) + memset(v->biquad_state[i], 0, sizeof v->biquad_state[i]); + v->in = *in; + v->use_float_biquads=0; + if (!in->in_use) return; + need(in->wave_buffer_index < 4, "wave buffer index"); + /* The server's play state the way the DSP keeps it (Ryujinx VoiceInfo.UpdatePlayState): Stop on a voice that is not + stopped becomes Stopping, which the next render frame turns into Stopped after giving back every queued buffer. */ + if (in->play_state == PLAY_STARTED) v->state = SV_STARTED; + else if (in->play_state == PLAY_PAUSED) v->state = SV_PAUSED; + else if (v->state != SV_STOPPED) v->state = SV_STOPPING; + for (unsigned i = 0; i < 4; i++) { /* a new buffer waits until a render frame finds the voice playing */ + const AudrenWaveBuffer *w = &in->wave_buffers[i]; + if (!w->sent) { v->buffers[i] = *w; v->pending[i] = 1; } + } + v->flush += in->flush_wave_buffer_count; v->flush_from = in->wave_buffer_index; + if (in->clear_voice_drop) v->drops = 0; +} + +/* No samples (no memory, no size or an empty range): the DSP decodes nothing and counts the buffer as played when it + reaches it, so a stream's empty last buffer does not hold up the buffers queued behind it. */ +int audio_buffer_empty(const AudrenWaveBuffer *b) { return !b->address || !b->size || b->end <= b->start; } +static void skip_empty(AudioRenderer *r, ServerVoice *v) { + while (v->valid[v->index] && audio_buffer_empty(&v->buffers[v->index])) { + const AudrenWaveBuffer *b = &v->buffers[v->index]; + if (b->loop && b->loop_count < 0) break; /* an endless empty loop starves the voice, as on the DSP */ + v->valid[v->index] = 0; v->consumed++; v->offset = 0; v->loops_done = 0; v->index = (v->index + 1) & 3; + if (r) r->empty_played++; + if (b->end_of_stream) { v->played = 0; break; } + } +} +static int stop_release(void) { + static int on = -1; + if (on < 0) { + const char *e = getenv("TOTK_AUDIO_STOP_RELEASE"); on = !(e && !strcmp(e, "0")); + if (!on) fprintf(stderr, "[totk-audio] TOTK_AUDIO_STOP_RELEASE=0: a stopped voice keeps its queued wave buffers (the old behaviour, for comparison)\n"); + } + return on; +} +/* Each render frame, before mixing (Ryujinx VoiceInfo.UpdateParametersForCommandGeneration): flush, stop, start. */ +void audio_voices_prepare(AudioRenderer *r) { + for (int k = 0; k < r->param.voice_count; k++) { + ServerVoice *v = &r->voices[k]; + if (!v->in.in_use) continue; + if (v->flush) { /* FlushWaveBuffers: from the client's current buffer, the queued ones count as played */ + unsigned i = v->flush_from & 3; + for (uint32_t n = v->flush; n; n--, i = (i + 1) & 3) { + v->pending[i] = 0; + if (v->valid[i]) { v->valid[i] = 0; v->consumed++; v->index = (v->index + 1) & 3; v->offset = 0; v->loops_done = 0; r->flushed++; } + } + v->flush = 0; + } + if (v->state == SV_STOPPING) { + if (!stop_release()) { v->state = SV_STOPPED; continue; } + for (int i = 0; i < 4; i++) { /* every queued buffer counts as played; ones never sent to the DSP are dropped */ + if (v->valid[i]) { v->consumed++; v->index = (v->index + 1) & 3; r->stop_released++; } + v->valid[i] = 0; v->pending[i] = 0; + } + v->offset = 0; v->played = 0; v->fraction = 0; v->loops_done = 0; + memset(&v->adpcm, 0, sizeof v->adpcm); v->adpcm.buffer = -1; + v->state = SV_STOPPED; + continue; + } + if (v->state == SV_STARTED) { + for (int i = 0; i < 4; i++) if (v->pending[i]) { + v->pending[i] = 0; v->valid[i] = 1; + if ((unsigned)i == v->index) { v->offset = 0; v->loops_done = 0; } + } + skip_empty(r, v); + } + } +} + +static uint32_t effect_out_bytes(const AudioRenderer *r) { return revision_number(r->revision) >= 9 ? 0x90u : 0x10u; } + +static void request_update(AudioRenderer *r, const uint8_t *input, uint64_t in_size, uint8_t *output, uint64_t out_size, uint64_t perf_size) { + need(in_size >= sizeof(AudrenHeader), "update input too small"); + AudrenHeader h; memcpy(&h, input, sizeof h); + need(h.total <= in_size, "update total size"); + Reader in = {input + sizeof h, input + h.total}; + pthread_mutex_lock(&r->lock); + r->revision = h.revision; + const int vc = r->param.voice_count, ec = r->param.effect_count, sc = r->param.sink_count, mc = r->param.submix_count + 1; + const uint32_t npools = (uint32_t)ec + 4u * (uint32_t)vc; + if (!r->logged) { + r->logged = 1; + fprintf(stderr, "[totk-audio] update REV%d sizes behavior=%u pools=%u voices=%u channels=%u effects=%u mixes=%u sinks=%u perf=%u splitters=%u render=%u total=%u out=%llu perf_out=%llu\n", + revision_number(h.revision), h.behavior, h.memory_pools, h.voices, h.voice_resources, h.effects, h.mixes, h.sinks, h.performance, h.splitters, h.render_info, h.total, + (unsigned long long)out_size, (unsigned long long)perf_size); + } + need(h.behavior == 0x10, "behavior section size"); + const uint8_t *behavior = take(&in, h.behavior, "behavior"); memcpy(&r->behavior_flags, behavior + 8, 8); + need(h.memory_pools == npools * sizeof(AudrenPoolIn), "memory pool section size"); + const AudrenPoolIn *pools = (const AudrenPoolIn *)take(&in, h.memory_pools, "memory pools"); + need(h.voice_resources % sizeof(AudrenChannelIn) == 0 && h.voice_resources / sizeof(AudrenChannelIn) <= (uint32_t)vc * 6, "voice channel section size"); + const AudrenChannelIn *chans = (const AudrenChannelIn *)take(&in, h.voice_resources, "voice channels"); + int float_voices=revision_number(h.revision)>=15; + size_t voice_stride=float_voices?sizeof(AudrenVoiceInV2):sizeof(AudrenVoiceIn); + need(h.voices == (uint32_t)vc * voice_stride, "voice section size"); + const uint8_t *voices = take(&in, h.voices, "voices"); + need(h.effects == (uint32_t)ec * sizeof(AudrenEffectIn), "effect section size"); + const AudrenEffectIn *effects = (const AudrenEffectIn *)take(&in, h.effects, "effects"); + const uint8_t *splitters = take(&in, h.splitters, "splitters"); + const uint8_t *mixes = take(&in, h.mixes, "mixes"); + need(h.sinks == (uint32_t)sc * sizeof(AudrenSinkIn), "sink section size"); + const AudrenSinkIn *sinks = (const AudrenSinkIn *)take(&in, h.sinks, "sinks"); + take(&in, h.performance, "performance"); + take(&in, h.render_info, "render info"); + need(in.p == in.end, "unconsumed update input"); + + for (uint32_t i = 0; i < h.voice_resources / sizeof(AudrenChannelIn); i++) { + need(chans[i].id < (uint32_t)vc * 6, "channel resource id"); + r->channels[chans[i].id] = chans[i]; + } + for (int i = 0; i < vc; i++) { + AudrenVoiceIn v; + AudrenVoiceInV2 v2; + if(float_voices){ + memcpy(&v2,voices+i*voice_stride,sizeof v2);memset(&v,0,sizeof v); + memcpy(&v,v2.prefix,sizeof v2.prefix); + memcpy((uint8_t *)&v+0x3c,v2.tail,sizeof v2.tail); + for(unsigned f=0;f<2;f++){ + v.biquads[f].enable=v2.biquads[f].enable; + if(v.biquads[f].enable){ + for(unsigned c=0;c<3;c++)need(isfinite(v2.biquads[f].numerator[c]),"nonfinite voice biquad numerator"); + for(unsigned c=0;c<2;c++)need(isfinite(v2.biquads[f].denominator[c]),"nonfinite voice biquad denominator"); + } + } + }else memcpy(&v,voices+i*voice_stride,sizeof v); + update_voice(r,&v); + ServerVoice *server=&r->voices[v.id];server->use_float_biquads=float_voices; + if(float_voices)memcpy(server->float_biquads,v2.biquads,sizeof v2.biquads); + } + for (int i = 0; i < ec; i++) r->effects[i] = effects[i]; + if (getenv("TOTK_AUDIO_DUMP")) { static unsigned dumped; unsigned active = 0; for (int i=0;i dumped) { dumped = active; FILE *f = fopen("analysis/audio-effects.bin", "wb"); if (f) { fwrite(effects, sizeof *effects, ec, f); fclose(f); } } + } + audio_mixer_update_effects(r); + audio_mixer_update_routing(r, splitters, h.splitters, mixes, h.mixes, mc); + for (int i = 0; i < sc; i++) r->sinks[i] = sinks[i]; + + /* ---- output ---- */ + const uint32_t eo = effect_out_bytes(r); + AudrenHeader o = {0}; + o.revision = REV(revision_number(h.revision) >= 1 ? revision_number(h.revision) : 1); + o.memory_pools = npools * sizeof(AudrenPoolOut); o.voices = (uint32_t)vc * sizeof(AudrenVoiceOut); + o.effects = (uint32_t)ec * eo; o.sinks = (uint32_t)sc * sizeof(AudrenSinkOut); + o.performance = 0x10; o.behavior = BEHAVIOR_OUT; o.render_info = RENDER_INFO_OUT; + o.total = (uint32_t)sizeof o + o.memory_pools + o.voices + o.effects + o.sinks + o.performance + o.behavior + o.render_info; + if (o.total > out_size) { fprintf(stderr, "[totk-audio] output needs %u bytes, client gave %llu (effect out %u)\n", o.total, (unsigned long long)out_size, eo); runtime_fail("audio renderer output size"); } + memset(output, 0, out_size); + uint8_t *w = output; memcpy(w, &o, sizeof o); w += sizeof o; + for (uint32_t i = 0; i < npools; i++) { + uint32_t s = pools[i].state; + if (s == POOL_REQUEST_ATTACH) s = POOL_ATTACHED; else if (s == POOL_REQUEST_DETACH) s = POOL_DETACHED; + r->pools[i] = pools[i]; r->pools[i].state = s; + AudrenPoolOut po = {s, {0}}; memcpy(w, &po, sizeof po); w += sizeof po; + } + for (int i = 0; i < vc; i++) { + AudrenVoiceOut vo = {r->voices[i].played, r->voices[i].consumed, r->voices[i].drops}; + memcpy(w, &vo, sizeof vo); w += sizeof vo; + } + for (int i = 0; i < ec; i++) { w[0] = effects[i].enabled ? 3 : 4; if(eo>16)audio_mixer_effect_result(r,i,w+16,eo-16); w += eo; } /* EffectStatus Enabled=3 / Disabled=4 */ + for (int i = 0; i < sc; i++) { AudrenSinkOut so={0};if(r->sinks[i].type==SINK_CIRCULAR)so.last_written_offset=audio_mixer_sink_written(r,i);memcpy(w,&so,sizeof so);w += sizeof so; } + w += 0x10; /* performance: nothing written */ + w += BEHAVIOR_OUT; /* behavior: no errors */ + memcpy(w, &r->elapsed_frames, 8); + pthread_mutex_unlock(&r->lock); +} + +/* ---------- Render clock: one frame every sample_count / sample_rate seconds ---------- */ +static void advance_voice(AudioRenderer *r, ServerVoice *v, double out_seconds) { + if (!v->in.in_use || v->state != SV_STARTED || !v->in.sample_rate) return; + double want = v->fraction + out_seconds * v->in.sample_rate * (v->in.pitch > 0 ? v->in.pitch : 1.0); + uint64_t samples = (uint64_t)want; v->fraction = want - (double)samples; + while (samples && v->valid[v->index]) { + AudrenWaveBuffer *b = &v->buffers[v->index]; + uint64_t length = (uint64_t)(b->end - b->start), left = length - v->offset; + uint64_t step = samples < left ? samples : left; + v->offset += step; v->played += step; samples -= step; + if (v->offset < length) break; + if (b->loop && (b->loop_count < 0 || v->loops_done < b->loop_count)) { v->loops_done++; v->offset = b->loop_start > b->start && b->loop_start < b->end ? (uint64_t)(b->loop_start - b->start) : 0; continue; } + v->valid[v->index] = 0; v->consumed++; v->offset = 0; v->loops_done = 0; v->index = (v->index + 1) & 3; + if (b->end_of_stream) { v->played = 0; break; } + skip_empty(r, v); + } +} +static void *render_thread(void *p) { + AudioRenderer *r = p; + SetThreadPriority(GetCurrentThread(), THREAD_PRIORITY_TIME_CRITICAL); // [port] QoS user-interactive + // [port] nanosleep sleeps in whole scheduler ticks on Windows; a high-resolution waitable timer keeps the 5 ms period. + HANDLE timer = CreateWaitableTimerExW(NULL, NULL, CREATE_WAITABLE_TIMER_HIGH_RESOLUTION, TIMER_ALL_ACCESS); + const double frame = (double)r->param.sample_count / r->param.sample_rate; + struct timespec next; clock_gettime(CLOCK_MONOTONIC, &next); + while (__atomic_load_n(&r->running, __ATOMIC_ACQUIRE)) { + pthread_mutex_lock(&r->lock); + if (r->state == 0) { + audio_voices_prepare(r); + audio_mixer_render(r, frame); + for (int i = 0; i < r->param.voice_count; i++) advance_voice(r, &r->voices[i], frame); + r->elapsed_frames++; + if (r->stop_released + r->empty_played + r->flushed != r->logged_released && r->elapsed_frames % 1000 == 0) { + r->logged_released = r->stop_released + r->empty_played + r->flushed; + fprintf(stderr, "[totk-audio] wave buffers given back: %llu by stopped voices, %llu without samples, %llu flushed (since start)\n", + (unsigned long long)r->stop_released, (unsigned long long)r->empty_played, (unsigned long long)r->flushed); + } + } + pthread_mutex_unlock(&r->lock); + runtime_event_signal(r->event); + uint64_t ns = (uint64_t)(frame * 1e9); + next.tv_nsec += (long)ns; while (next.tv_nsec >= 1000000000L) { next.tv_nsec -= 1000000000L; next.tv_sec++; } + struct timespec now; clock_gettime(CLOCK_MONOTONIC, &now); + int64_t wait = (int64_t)(next.tv_sec - now.tv_sec) * 1000000000LL + (next.tv_nsec - now.tv_nsec); + if (wait > 0) { // [port] + if (timer) { LARGE_INTEGER due = {.QuadPart = -(wait / 100)}; SetWaitableTimer(timer, &due, 0, NULL, NULL, FALSE); WaitForSingleObject(timer, INFINITE); } + else { struct timespec d = {wait / 1000000000LL, wait % 1000000000LL}; nanosleep(&d, NULL); } + } + else if (wait < -100000000LL) next = now; /* fell far behind (debugger, sleep): resynchronise */ + } + if (timer) CloseHandle(timer); // [port] + return NULL; +} + +/* ---------- IAudioRenderer ---------- */ +static IpcBuffer pick(const IpcBuffer *a, unsigned na, const IpcBuffer *b, unsigned nb, unsigned i) { + IpcBuffer x = i < na ? a[i] : (IpcBuffer){0}, y = i < nb ? b[i] : (IpcBuffer){0}; + return x.size ? x : y; +} +static void renderer_handler(IpcSession *s, const IpcRequest *rq, IpcResponse *rs) { + AudioRenderer *r = s->state; + switch (rq->cmd) { + case 0: { int32_t v = r->param.sample_rate; IPC_OUT(rs, v); return; } + case 1: { int32_t v = r->param.sample_count; IPC_OUT(rs, v); return; } + case 2: { int32_t v = r->param.mix_buffer_count; IPC_OUT(rs, v); return; } + case 3: { uint32_t v = r->state; IPC_OUT(rs, v); return; } + case 4: case 10: { /* RequestUpdate / RequestUpdateAuto: out(output), out(performance), in(input) */ + IpcBuffer in = pick(rq->a, rq->na, rq->x, rq->nx, 0); + IpcBuffer out = pick(rq->b, rq->nb, rq->c, rq->nc, 0), perf = pick(rq->b, rq->nb, rq->c, rq->nc, 1); + if (!r->logged) fprintf(stderr, "[totk-audio] RequestUpdate cmd %u in=%#llx+%#llx out=%#llx+%#llx perf=%#llx+%#llx (a%u b%u x%u c%u)\n", rq->cmd, + (unsigned long long)in.addr, (unsigned long long)in.size, (unsigned long long)out.addr, (unsigned long long)out.size, (unsigned long long)perf.addr, (unsigned long long)perf.size, rq->na, rq->nb, rq->nx, rq->nc); + need(in.size && out.size, "RequestUpdate buffers"); + request_update(r, ipc_guest(in.addr, in.size), in.size, ipc_guest(out.addr, out.size), out.size, perf.size); + for (unsigned i = 0; i < rq->nc && rq->cmd == 10; i++) if (rq->c[i].size && (i >= rq->nb || !rq->b[i].size)) rs->xout[rs->nxout++] = (IpcBuffer){rq->c[i].addr, rq->c[i].size, i}; + return; + } + case 5: pthread_mutex_lock(&r->lock); r->state = 0;audio_mixer_set_running(r,1); pthread_mutex_unlock(&r->lock); return; + case 6: pthread_mutex_lock(&r->lock); r->state = 1;audio_mixer_set_running(r,0); pthread_mutex_unlock(&r->lock); return; + case 7: rs->copy[rs->ncopy++] = runtime_event_readable(r->event); return; + case 8: r->time_limit = IPC_ARG(rq, uint32_t, 0); return; + case 9: { uint32_t v = r->time_limit ? r->time_limit : 100; IPC_OUT(rs, v); return; } + default: ipc_unimplemented(s, rq); + } +} +static void renderer_close(void *p) { + AudioRenderer *r = p; + __atomic_store_n(&r->running, 0, __ATOMIC_RELEASE); pthread_join(r->thread, NULL); + audio_mixer_close(r); + free(r->voices); free(r->pools); free(r->mixes); free(r->sinks); free(r->effects); free(r->channels); free(r); +} + +/* ---------- IAudioRendererManager ---------- */ +static void manager_handler(IpcSession *s, const IpcRequest *rq, IpcResponse *rs) { + switch (rq->cmd) { + case 0: { /* OpenAudioRenderer(param, pad, work size, aruid; copy tmem, process) */ + AudrenParameter p = IPC_ARG(rq, AudrenParameter, 0); + need(rq->ncopy >= 1, "OpenAudioRenderer handles"); + AudioRenderer *r = renderer_new(&p); + r->running = 1; audio_mixer_open(r); + need(pthread_create(&r->thread, NULL, render_thread, r) == 0, "render thread"); + fprintf(stderr, "[totk-audio] renderer open %d Hz x%d, voices %d effects %d submixes %d sinks %d splitters %d/%d REV%d\n", p.sample_rate, p.sample_count, + p.voice_count, p.effect_count, p.submix_count, p.sink_count, p.splitter_count, p.splitter_destination_count, revision_number(p.revision)); + rs->move[rs->nmove++] = runtime_ipc_new_session("IAudioRenderer", renderer_handler, r, renderer_close); + return; + } + case 1: { uint64_t v = WORK_BUFFER_BYTES; IPC_OUT(rs, v); return; } + case 2: case 4: rs->move[rs->nmove++] = runtime_ipc_new_session("IAudioDevice", audio_device_handler, NULL, NULL); return; + default: ipc_unimplemented(s, rq); + } +} + +void runtime_audio_register(void) { runtime_ipc_register("audren:u", manager_handler, NULL, NULL); } +// [port] registered from host_services_register (services_lm.c) + +/* ---------- make probe-audio: mixer checks on constructed renderer state (no game, no audio device) ---------- */ +int audio_mixer_read(AudioRenderer *r, float *lr, int frames); +static int near(float a, float b) { return fabsf(a - b) <= 1e-4f; } +/* Complete REV15 wire request -> parser -> float voice/splitter filters -> sink. + * Expected samples use separate double-precision recurrences, including state + * across two requests. Coefficients deliberately cannot be represented in Q14. */ +static int probe_rev15(void){ + AudrenParameter p={.sample_rate=48000,.sample_count=240,.mix_buffer_count=2,.voice_count=1,.sink_count=1,.splitter_count=1,.splitter_destination_count=1,.revision=REV(15)}; + AudioRenderer *r=renderer_new(&p);audio_mixer_open(r); + static float pcm[480];for(unsigned k=0;k<480;k++)pcm[k]=(float)(0.3*sin(k*0.071)+(k%37==0?0.1:0)); + AudrenHeader h={.revision=REV(15),.behavior=16,.memory_pools=4*sizeof(AudrenPoolIn),.voice_resources=sizeof(AudrenChannelIn),.voices=sizeof(AudrenVoiceInV2),.splitters=0x40+0xa8,.mixes=sizeof(AudrenMixIn),.sinks=sizeof(AudrenSinkIn),.performance=16}; + h.total=sizeof h+h.behavior+h.memory_pools+h.voice_resources+h.voices+h.splitters+h.mixes+h.sinks+h.performance; + uint8_t *input=calloc(1,h.total),output[4096];need(input!=NULL,"REV15 probe allocation"); + AudrenVoiceIn v={.id=0,.is_new=1,.in_use=1,.play_state=PLAY_STARTED,.sample_format=FMT_PCMF32,.sample_rate=48000,.channel_count=1,.pitch=1,.volume=1,.mix_id=0x7fffffff,.splitter_id=0}; + v.wave_buffers[0]=(AudrenWaveBuffer){.address=(uintptr_t)pcm,.size=sizeof pcm,.end=480}; + for(unsigned k=1;k<4;k++)v.wave_buffers[k].sent=1; + AudrenBiquadFloat voice={.enable=1,.numerator={0.1234567f,0.0712345f,0},.denominator={0.25f,0}}; + AudrenBiquadFloat splitter={.enable=1,.numerator={0.7654321f,0,0},.denominator={0.125f,0}}; + double vx=0,vy=0,sy=0;int checked=0; + for(unsigned block=0;block<2;block++){ + uint8_t *at=input;memcpy(at,&h,sizeof h);at+=sizeof h+h.behavior+h.memory_pools+h.voice_resources; + AudrenVoiceInV2 wire={0};memcpy(wire.prefix,&v,sizeof wire.prefix);memcpy(wire.tail,(uint8_t *)&v+0x3c,sizeof wire.tail);wire.biquads[0]=voice; + memcpy(at,&wire,sizeof wire);at+=sizeof wire; + uint32_t sh[3]={0x48444e53,1,1},si[5]={0x49444e53,0,48000,1,0},magic=0x44444e53; + memcpy(at,sh,12);memcpy(at+0x20,si,sizeof si); + uint8_t *dest=at+0x40;memcpy(dest,&magic,4);float volumes[24]={1,0.5f};memcpy(dest+8,volumes,sizeof volumes);memcpy(dest+0x6c,&splitter,sizeof splitter);dest[0x9c]=1; + at+=h.splitters; + AudrenMixIn mix={.volume=1,.sample_rate=48000,.buffer_count=2,.in_use=1,.mix_id=0,.dest_mix_id=0x7fffffff,.dest_splitter_id=0xffffffff};memcpy(at,&mix,sizeof mix);at+=sizeof mix; + AudrenSinkIn sink={.type=SINK_DEVICE,.in_use=1,.input_count=2,.inputs={0,1}};memcpy(at,&sink,sizeof sink); + request_update(r,input,h.total,output,sizeof output,0); + AudrenHeader reply;memcpy(&reply,output,sizeof reply);need(reply.revision==REV(15)&&reply.voices==sizeof(AudrenVoiceOut)&&reply.total<=sizeof output,"REV15 probe response"); + if(getenv("TOTK_AUDIO_REV15_NEGATIVE"))r->voices[0].use_float_biquads=0; + audio_voices_prepare(r);audio_mixer_render(r,240.0/48000);advance_voice(r,&r->voices[0],240.0/48000); + float samples[480];need(audio_mixer_read(r,samples,240)==240,"REV15 probe ring"); + for(unsigned k=0;k<240;k++){ + double x=pcm[block*240+k],y=voice.numerator[0]*x+voice.numerator[1]*vx+voice.denominator[0]*vy;vx=x;vy=y; + double z=splitter.numerator[0]*y+splitter.denominator[0]*sy;sy=z; + need(fabs(samples[2*k]-z)<1e-6&&fabs(samples[2*k+1]-z*0.5)<1e-6,"audio probe: REV15 float voice and splitter samples");checked+=2; + } + v.is_new=0;v.wave_buffers[0].sent=1; + } + free(input);audio_mixer_close(r);free(r->voices);free(r->pools);free(r->mixes);free(r->sinks);free(r->effects);free(r->channels);free(r); + return checked; +} +void runtime_audio_probe(void) { + setenv("TOTK_AUDIO_OFF", "1", 1); + const int neg = getenv("TOTK_PROBE_NEGATIVE") != NULL; + AudrenParameter p = {48000, 240, 8, 1, 4, 1, 2, 0, 0, 0, 0, 0, 1, 2, 0, REV(11)}; + AudioRenderer *r = renderer_new(&p); audio_mixer_open(r); + int checked = probe_rev15(); + /* final mix 0: buffers 0-1; submix 1: buffers 2-3 -> mix 0 with a swap matrix at volume 0.5 */ + r->mixes[0] = (AudrenMixIn){.volume = 1, .sample_rate = 48000, .buffer_count = 2, .in_use = 1, .mix_id = 0, .dest_mix_id = 0x7fffffff, .dest_splitter_id = 0xffffffffu}; + r->mixes[1] = (AudrenMixIn){.volume = 0.5f, .sample_rate = 48000, .buffer_count = 2, .in_use = 1, .mix_id = 1, .dest_mix_id = 0, .dest_splitter_id = 0xffffffffu}; + r->mixes[1].mix[0][1] = 1; r->mixes[1].mix[1][0] = 1; + r->sinks[0] = (AudrenSinkIn){.type = SINK_DEVICE, .in_use = 1, .input_count = 2, .inputs = {0, 1}}; + /* 1. PCM16 sine through a channel resource: L 0.5, R 0.125 */ + static int16_t pcm[4800]; for (int i = 0; i < 4800; i++) pcm[i] = (int16_t)(16000 * sin(2 * M_PI * 1000 * i / 48000.0)); + AudrenVoiceIn v = {.id = 0, .node_id = 0, .is_new = 1, .in_use = 1, .play_state = PLAY_STARTED, .sample_format = FMT_PCM16, .sample_rate = 48000, .channel_count = 1, + .pitch = 1, .volume = 0.5f, .mix_id = 0, .splitter_id = 0xffffffffu, .channel_ids = {0}}; + v.wave_buffers[0] = (AudrenWaveBuffer){.address = (uint64_t)(uintptr_t)pcm, .size = sizeof pcm, .start = 0, .end = 4800, .loop = 1, .loop_count = -1}; + r->channels[0].mix[0] = 1; r->channels[0].mix[1] = neg ? 0.3f : 0.25f; + update_voice(r, &v); + const double frame = 240.0 / 48000; + float lr[480]; + for (int f = 0; f < 3; f++) { audio_voices_prepare(r); audio_mixer_render(r, frame); advance_voice(r, &r->voices[0], frame); } + if (audio_mixer_read(r, lr, 240) != 240) runtime_fail("audio probe ring"); + for (int k = 0; k < 240; k++) { float s = pcm[480 + k] / 32768.0f; + if (!near(lr[2 * k], 0.5f * s) || !near(lr[2 * k + 1], 0.125f * s)) { fprintf(stderr, "pcm frame sample %d: %f %f want %f %f\n", k, lr[2 * k], lr[2 * k + 1], 0.5f * s, 0.125f * s); runtime_fail("audio probe: PCM16 mix"); } checked += 2; } + /* 2. pitch 0.5: linear interpolation between neighbouring samples */ + v.is_new = 1; v.pitch = 0.5f; update_voice(r, &v); + audio_voices_prepare(r); audio_mixer_render(r, frame); advance_voice(r, &r->voices[0], frame); audio_mixer_read(r, lr, 240); + for (int k = 0; k < 240; k++) { int i = k / 2; float s = (k & 1) ? (pcm[i] + pcm[i + 1]) / 65536.0f : pcm[i] / 32768.0f; + if (!near(lr[2 * k], 0.5f * s)) { fprintf(stderr, "pitch sample %d: %f want %f\n", k, lr[2 * k], 0.5f * s); runtime_fail("audio probe: resampling"); } checked++; } + /* 3. DSP-ADPCM, coefficients {1.0, 0} for predictor 0 (an integrator), scale 2^2 */ + static uint8_t adpcm[8 * 20]; static int16_t coef[16]; coef[0] = 2048; + int16_t want[280]; int32_t h = 0; + for (int fr = 0; fr < 20; fr++) { adpcm[fr * 8] = 0x02; for (int j = 0; j < 7; j++) adpcm[fr * 8 + 1 + j] = (uint8_t)(((j + fr) & 7) << 4 | ((15 - j) & 15)); + for (int j = 0; j < 14; j++) { int n = (adpcm[fr * 8 + 1 + j / 2] >> ((j & 1) ? 0 : 4)) & 15; if (n >= 8) n -= 16; h += n * 4; if (h > 32767) h = 32767; if (h < -32768) h = -32768; want[fr * 14 + j] = (int16_t)h; } } + AudrenVoiceIn a = v; a.is_new = 1; a.pitch = 1; a.sample_format = FMT_ADPCM; a.extra_address = (uint64_t)(uintptr_t)coef; a.extra_size = sizeof coef; + a.wave_buffers[0] = (AudrenWaveBuffer){.address = (uint64_t)(uintptr_t)adpcm, .size = sizeof adpcm, .start = 0, .end = 280}; + update_voice(r, &a); + audio_voices_prepare(r); audio_mixer_render(r, frame); advance_voice(r, &r->voices[0], frame); audio_mixer_read(r, lr, 240); + for (int k = 0; k < 240; k++) { float s = want[k] / 32768.0f; + if (!near(lr[2 * k], 0.5f * s)) { fprintf(stderr, "adpcm sample %d: %f want %f\n", k, lr[2 * k], 0.5f * s); runtime_fail("audio probe: ADPCM"); } checked++; } + /* 4. splitter: voice -> destination 0 (submix 1, volumes 1 / 0.5) -> mix 1 swapped into mix 0 at 0.5 */ + a.in_use = 0; update_voice(r, &a); + uint8_t split[0x20 + 0x20 + 0x70] = {0}; uint32_t h0[3] = {0x48444e53u, 1, 1}, h1[5] = {0x49444e53u, 0, 48000, 1, 0}; memcpy(split, h0, 12); memcpy(split + 0x20, h1, 20); /* info: 16 + 4 ids + 12 reserved */ + uint8_t *d = split + 0x40; uint32_t dm = 0x44444e53u, did = 0; int32_t dst = 1; float vol[24] = {1, 0.5f}; memcpy(d, &dm, 4); memcpy(d + 4, &did, 4); memcpy(d + 8, vol, sizeof vol); memcpy(d + 0x68, &dst, 4); d[0x6c] = 1; + uint8_t mixes[2 * sizeof(AudrenMixIn)]; memcpy(mixes, r->mixes, sizeof mixes); + audio_mixer_update_routing(r, split, sizeof split, mixes, sizeof mixes, 2); + AudrenVoiceIn sv = v; sv.id = 1; sv.is_new = 1; sv.pitch = 1; sv.mix_id = 0x7fffffff; sv.splitter_id = 0; update_voice(r, &sv); + audio_voices_prepare(r); audio_mixer_render(r, frame); advance_voice(r, &r->voices[1], frame); audio_mixer_read(r, lr, 240); + for (int k = 0; k < 240; k++) { float s = pcm[k] / 32768.0f * 0.5f * 0.5f; /* voice 0.5, mix 1 volume 0.5; buffer 2 (x1) -> R, buffer 3 (x0.5) -> L */ + if (!near(lr[2 * k], 0.5f * s) || !near(lr[2 * k + 1], s)) { fprintf(stderr, "splitter sample %d: %f %f want %f %f\n", k, lr[2 * k], lr[2 * k + 1], 0.5f * s, s); runtime_fail("audio probe: splitter routing"); } checked += 2; } + /* 5. Two cascaded Q14 filters, independent stereo channels and state across frames. + The oracle uses double-precision direct form I; production uses float transposed form II. */ + sv.in_use = 0; update_voice(r, &sv); + static int16_t stereo[240 * 6 * 2]; + for (int k = 0; k < 240 * 6; k++) { + stereo[2 * k] = (int16_t)(7000 * sin(k * 0.043) + (k % 71 == 0 ? 4000 : 0)); + stereo[2 * k + 1] = (int16_t)(5000 * cos(k * 0.071) - (k % 53 == 0 ? 3000 : 0)); + } + AudrenBiquad filters[2] = {{.enable = 1, .numerator = {8192, 4096, 2048}, .denominator = {6144, -2048}}, + {.enable = 1, .numerator = {12288, -4096, 1024}, .denominator = {4096, -1024}}}; + v.is_new = 1; v.pitch = 1; v.volume = 1; v.channel_count = 2; v.channel_ids[1] = 1; + v.wave_buffers[0] = (AudrenWaveBuffer){.address = (uint64_t)(uintptr_t)stereo, .size = sizeof stereo, .end = 240 * 6}; + memset(r->channels, 0, 2 * sizeof *r->channels); r->channels[0].mix[0] = 1; r->channels[1].mix[1] = 1; + double history[2][2][4] = {{{0}}}; int cursor = 0; + for (int block = 0; block < 6; block++) { + filters[0].enable = block != 2; + if (block == 2) memset(history[0], 0, sizeof history[0]); + if (block == 4) { v.is_new = 1; v.wave_buffers[0].sent = 0; cursor = 0; memset(history, 0, sizeof history); } + memcpy(v.biquads, filters, sizeof filters); + if (getenv("TOTK_AUDIO_BIQUAD_NEGATIVE")) v.biquads[0].enable = v.biquads[1].enable = 0; + update_voice(r, &v); + audio_voices_prepare(r); audio_mixer_render(r, frame); advance_voice(r, &r->voices[0], frame); audio_mixer_read(r, lr, 240); + for (int k = 0; k < 240; k++) for (int c = 0; c < 2; c++) { + double x = stereo[2 * (cursor + k) + c] / 32768.0; + for (int f = 0; f < 2; f++) if (filters[f].enable) { + double *h = history[f][c]; const AudrenBiquad *b = &filters[f]; + double y = (b->numerator[0] * x + b->numerator[1] * h[0] + b->numerator[2] * h[1] + + b->denominator[0] * h[2] + b->denominator[1] * h[3]) / 16384.0; + h[1] = h[0]; h[0] = x; h[3] = h[2]; h[2] = y; x = y; + } + if (!near(lr[2 * k + c], (float)x)) { + fprintf(stderr, "biquad block %d sample %d channel %d: %.8f want %.8f\n", block, k, c, lr[2 * k + c], x); + runtime_fail("audio probe: biquad state/routing"); + } + checked++; + } + cursor += 240; v.is_new = 0; v.wave_buffers[0].sent = 1; + } + /* 6. Ordered submix effects, distinct output buffers, disabled bypass and downstream routing. */ + memset(v.biquads, 0, sizeof v.biquads); v.mix_id=1; v.is_new=1; v.wave_buffers[0].sent=0; + r->mixes[1].buffer_count=4; memset(r->mixes[1].mix,0,sizeof r->mixes[1].mix); + r->mixes[1].mix[2][1]=1; r->mixes[1].mix[3][0]=1; + Reverb3dParam rp={.input={0,1},.output={2,3},.channel_max=6,.channels=2,.sample_rate=48000, + .room_hf=-2000,.hf_reference=4000,.decay_time=.8f,.hf_decay_ratio=.7f,.room_gain=-20000, + .diffusion=50,.density=70,.dry_gain=.25f,.status=2}; + r->effects[0]=(AudrenEffectIn){.type=5,.is_new=1,.enabled=1,.mix_id=1,.order=1}; + r->effects[1]=(AudrenEffectIn){.type=5,.is_new=1,.enabled=1,.mix_id=1,.order=0}; + memcpy(r->effects[0].specific,&rp,sizeof rp); rp.output[0]=0;rp.output[1]=1;rp.dry_gain=.5f;memcpy(r->effects[1].specific,&rp,sizeof rp); + for(int block=0;block<2;block++) { + update_voice(r,&v); r->effects[0].enabled=block==0; + if(getenv("TOTK_AUDIO_REVERB_NEGATIVE"))r->effects[0].order=-1; + audio_mixer_update_effects(r);audio_voices_prepare(r);audio_mixer_render(r,frame);audio_mixer_read(r,lr,240); + float gain=block==0?.0625f:.25f; + for(int k=0;k<240;k++)for(int c=0;c<2;c++){float want=stereo[2*k+1-c]/32768.f*gain; + if(!near(lr[2*k+c],want))runtime_fail("audio probe: reverb effect order/bypass/routing");checked++;} + } + /* Real wet response across IPC updates: the DSP kernel is independently compared to Ryujinx by probe-reverb. */ + r->effects[1].type=0; r->effects[0].enabled=1;r->effects[0].is_new=1;r->effects[0].order=0; + rp.room_gain=-800;rp.reverb_gain=500;rp.dry_gain=.1f;rp.output[0]=2;rp.output[1]=3; + memcpy(r->effects[0].specific,&rp,sizeof rp); Reverb3d *ref=audio_reverb_create(&rp); + float input[2][240],output[2][240],*ip[6]={input[0],input[1]},*op[6]={output[0],output[1]}; + for(int block=0;block<80;block++){ + v.is_new=1;v.wave_buffers[0].sent=0;update_voice(r,&v); + for(int c=0;c<2;c++)for(int k=0;k<240;k++)input[c][k]=stereo[2*k+c]/32768.f; + audio_reverb_process(ref,ip,op,240);audio_mixer_update_effects(r);audio_voices_prepare(r);audio_mixer_render(r,frame);audio_mixer_read(r,lr,240); + for(int c=0;c<2;c++)for(int k=0;k<240;k++){if(!near(lr[2*k+c],output[1-c][k]*.5f))runtime_fail("audio probe: reverb continuity in mix graph");checked++;} + r->effects[0].is_new=0; + } + audio_reverb_free(ref); + /* 7. Auxiliary int32 exchange: stereo wrap, DSP-owned cursors, CPU metadata and disabled passthrough. */ + uint32_t send_ring[32+960],return_ring[32+960];for(int i=0;i<32+960;i++){send_ring[i]=0x12345678;return_ring[i]=(uint32_t)(int32_t)(i-500);} + send_ring[17]=return_ring[16]=720;send_ring[18]=return_ring[18]=7; + AudrenAuxParam ap={.input={0,1},.output={2,3},.channels=2,.sample_rate=48000,.capacity=960,.channel_max=2, + .send_info=(uint64_t)(uintptr_t)send_ring,.return_info=(uint64_t)(uintptr_t)return_ring}; + r->effects[0]=(AudrenEffectIn){.type=2,.enabled=1,.mix_id=1};memcpy(r->effects[0].specific,&ap,sizeof ap); + audio_mixer_update_effects(r);update_voice(r,&v);audio_voices_prepare(r);audio_mixer_render(r,frame);audio_mixer_read(r,lr,240); + for(int c=0;c<2;c++)for(int k=0;k<240;k++){ + int idx=(720+c*240+k)%960,want_sample=idx+32-500; + if(!near(lr[2*k+1-c],want_sample/65536.f) || (int32_t)send_ring[32+idx]!=stereo[2*k+c])runtime_fail("audio probe: auxiliary sample/wrap/routing");checked+=2; + } + if(send_ring[17]!=240 || return_ring[16]!=240 || send_ring[0]!=0x12345678 || send_ring[18]!=7)runtime_fail("audio probe: auxiliary cursors");checked+=4; + r->effects[0].enabled=0;update_voice(r,&v);audio_voices_prepare(r);audio_mixer_render(r,frame);audio_mixer_read(r,lr,240); + for(int c=0;c<2;c++)for(int k=0;k<240;k++){if(!near(lr[2*k+1-c],stereo[2*k+c]/65536.f))runtime_fail("audio probe: auxiliary bypass");checked++;} + if(send_ring[16]||send_ring[17]||send_ring[19]||return_ring[16]||return_ring[17]||return_ring[19]||send_ring[18]!=7)runtime_fail("audio probe: auxiliary reset");checked+=7; + /* 8. Limiter zero-state delay and hard threshold. Independent expected rectangular response. */ + r->mixes[1].in_use=0;v.mix_id=0;v.sample_format=FMT_PCMF32;static float limiter_input[480]; + for(int k=0;k<240;k++){limiter_input[2*k]=.75f;limiter_input[2*k+1]=-.125f;} + v.wave_buffers[0]=(AudrenWaveBuffer){.address=(uint64_t)(uintptr_t)limiter_input,.size=sizeof limiter_input,.end=240}; + AudrenLimiterParam lp={.input={0,1},.output={0,1},.channel_max=6,.channels=2,.sample_rate=48000, + .attack=1,.release=1,.threshold=.5f,.input_gain=1,.output_gain=1,.delay_min=9,.delay_max=13,.status=2,.statistics_enabled=1}; + r->effects[0]=(AudrenEffectIn){.type=7,.enabled=1,.is_new=1,.mix_id=0};memcpy(r->effects[0].specific,&lp,sizeof lp); + update_voice(r,&v);audio_mixer_update_effects(r);audio_voices_prepare(r);audio_mixer_render(r,frame);audio_mixer_read(r,lr,240); + for(int c=0;c<2;c++)for(int k=0;k<240;k++){float expected=k<9?0:c==0?.5f*(32767.f/32768.f):-.125f; + if(!near(lr[2*k+c],expected))runtime_fail("audio probe: limiter delay/threshold");checked++;} + float statistics[12];audio_mixer_effect_result(r,0,statistics,sizeof statistics); + if(!near(statistics[0],.75f*32768.f/32767.f)||!near(statistics[6],.5f/(.75f*32768.f/32767.f)))runtime_fail("audio probe: limiter statistics");checked+=2; + const char *fixture=getenv("TOTK_LIMITER_REFERENCE"); + if(fixture){FILE *f=fopen(fixture,"rb");if(!f)runtime_fail("limiter reference fixture");lp.attack=.2f;lp.release=.02f;lp.threshold=.5f;lp.input_gain=1.3f;lp.output_gain=.8f; + for(int block=0;block<150;block++){ + if(block==70){lp.threshold=.3f;lp.attack=.4f;}lp.statistics_reset=block==50; + r->effects[0].is_new=block==0||block==120;r->effects[0].enabled=block<100||block>=110; + memcpy(r->effects[0].specific,&lp,sizeof lp); + for(int c=0;c<2;c++)for(int k=0;k<240;k++){int t=block*240+k;limiter_input[2*k+c]=((t*17+c*31)%257-128)/128.f*(block%7==0?1.8f:.3f);} + v.is_new=1;v.wave_buffers[0].sent=0;update_voice(r,&v);audio_mixer_update_effects(r);audio_voices_prepare(r);audio_mixer_render(r,frame);audio_mixer_read(r,lr,240); + audio_mixer_effect_result(r,0,statistics,sizeof statistics); + for(int k=0;k<492;k++){float want;need(fread(&want,4,1,f)==1,"limiter reference length");float got=k<480?lr[k]:statistics[k-480]; + if(getenv("TOTK_LIMITER_NEGATIVE") && k<480)got=limiter_input[k]; + if(!isfinite(got)||fabsf(got-want)>2e-5f){fprintf(stderr,"limiter block %d index %d got %.9g want %.9g\n",block,k,got,want);runtime_fail("limiter independent reference");}checked++; + } + }need(fgetc(f)==EOF,"limiter reference trailing data");fclose(f); + } + audio_mixer_close(r); + /* 7. Voice lifetime as the DSP keeps it (Ryujinx VoiceInfo / DataSourceHelper): a buffer without samples counts as + played when it is reached, so the one behind it still plays; Stop gives back every queued buffer on the next render + frame and starts the voice over; buffers sent while stopped wait for Start; FlushWaveBuffers counts the queued + ones from the client's current buffer as played. 2026-10-17 the game over "retry" waited on stopped voices. */ + { + static int16_t a7[480], b7[480]; for (int k = 0; k < 480; k++) { a7[k] = (int16_t)(k * 50); b7[k] = (int16_t)(-k * 50); } + AudrenVoiceIn x = {.id = 2, .is_new = 1, .in_use = 1, .play_state = PLAY_STARTED, .sample_format = FMT_PCM16, .sample_rate = 48000, + .channel_count = 1, .pitch = 1, .volume = 1, .mix_id = 0x7fffffff, .splitter_id = 0xffffffffu, .channel_ids = {2}}; + x.wave_buffers[0] = (AudrenWaveBuffer){.address = (uint64_t)(uintptr_t)a7, .size = sizeof a7, .end = 480}; + x.wave_buffers[1] = (AudrenWaveBuffer){.address = 0, .size = 0, .end = 0}; /* nothing to decode */ + x.wave_buffers[2] = (AudrenWaveBuffer){.address = (uint64_t)(uintptr_t)b7, .size = sizeof b7, .end = 480}; + x.wave_buffers[3].sent = 1; + ServerVoice *sv7 = &r->voices[2]; + update_voice(r, &x); + for (int f = 0; f < 3; f++) { audio_voices_prepare(r); audio_mixer_render(r, frame); advance_voice(r, sv7, frame); } + if (sv7->consumed != 2 || sv7->index != 2 || sv7->played != 720 || sv7->offset != 240) { + fprintf(stderr, "empty buffer: consumed %u index %u played %llu offset %llu, want 2 2 720 240\n", sv7->consumed, sv7->index, (unsigned long long)sv7->played, (unsigned long long)sv7->offset); + runtime_fail("audio probe: a buffer without samples holds up the voice"); + } + checked += 4; + x.is_new = 0; for (int i = 0; i < 4; i++) x.wave_buffers[i].sent = 1; + x.play_state = PLAY_STOPPED; update_voice(r, &x); + if (sv7->consumed != 2) runtime_fail("audio probe: Stop gave buffers back before a render frame"); + audio_voices_prepare(r); audio_mixer_render(r, frame); advance_voice(r, sv7, frame); + if (sv7->consumed != 3 || sv7->played != 0 || sv7->offset != 0 || sv7->valid[0] | sv7->valid[1] | sv7->valid[2] | sv7->valid[3] || sv7->state != SV_STOPPED) { + fprintf(stderr, "stop: consumed %u played %llu offset %llu state %u, want 3 0 0 stopped\n", sv7->consumed, (unsigned long long)sv7->played, (unsigned long long)sv7->offset, sv7->state); + runtime_fail("audio probe: a stopped voice keeps its queued wave buffers"); + } + checked += 5; + x.wave_buffers[3] = (AudrenWaveBuffer){.address = (uint64_t)(uintptr_t)a7, .size = sizeof a7, .end = 480}; /* sent while stopped */ + update_voice(r, &x); audio_voices_prepare(r); audio_mixer_render(r, frame); advance_voice(r, sv7, frame); + if (sv7->valid[3] || !sv7->pending[3] || sv7->consumed != 3 || sv7->played != 0) runtime_fail("audio probe: a buffer sent while stopped played"); + x.wave_buffers[3].sent = 1; x.wave_buffers[0] = (AudrenWaveBuffer){.address = (uint64_t)(uintptr_t)b7, .size = sizeof b7, .end = 480}; + x.play_state = PLAY_STARTED; update_voice(r, &x); audio_voices_prepare(r); + if (!sv7->valid[3] || !sv7->valid[0] || sv7->index != 3) runtime_fail("audio probe: Start did not queue the waiting buffers"); + checked += 2; + x.wave_buffers[0].sent = 1; x.flush_wave_buffer_count = 2; x.wave_buffer_index = 3; update_voice(r, &x); + audio_voices_prepare(r); audio_mixer_render(r, frame); advance_voice(r, sv7, frame); + if (sv7->consumed != 5 || sv7->valid[3] || sv7->valid[0] || sv7->played != 0) { + fprintf(stderr, "flush: consumed %u valid %u%u played %llu, want 5 00 0\n", sv7->consumed, sv7->valid[3], sv7->valid[0], (unsigned long long)sv7->played); + runtime_fail("audio probe: FlushWaveBuffers"); + } + checked += 3; + x.flush_wave_buffer_count = 0; x.in_use = 0; update_voice(r, &x); + } + printf("{\"status\":\"pass\",\"probe\":\"probe-audio\",\"checked\":%d,\"scope\":\"REV15 wire parsing, native float voice/splitter biquads and consecutive request state; PCM16 channel mix, pitch interpolation, DSP-ADPCM decode, splitter to submix to final mix, device sink, stereo cascaded biquads, ordered submix reverb, auxiliary ring exchange, output limiter, bypass, frame continuity and reset\"}\n", checked); +} diff --git a/src/host/audio_renderer.h b/src/host/audio_renderer.h new file mode 100644 index 0000000..c8a22e7 --- /dev/null +++ b/src/host/audio_renderer.h @@ -0,0 +1,87 @@ +/* From MacNativeGames games/totk/runtime/audio_renderer.h @b75be8c (MIT). [port] marks the Windows changes. */ +/* nn::audio renderer service (audren:u) wire formats. Layouts follow the public libnx audren.h + description; per-revision section sizes are cross-checked against the request header at runtime. */ +#pragma once +#include + +typedef struct { + int32_t sample_rate, sample_count, mix_buffer_count, submix_count, voice_count, sink_count, effect_count, + performance_frame_count; + uint8_t voice_drop_enabled, unknown, rendering_device, execution_mode; + int32_t splitter_count, splitter_destination_count; + uint32_t external_context_size, revision; +} AudrenParameter; +_Static_assert(sizeof(AudrenParameter) == 0x34, "AudioRendererParameter"); + +typedef struct { + uint32_t revision, behavior, memory_pools, voices, voice_resources, effects, mixes, sinks, performance, + splitters, render_info, reserved[4], total; +} AudrenHeader; +_Static_assert(sizeof(AudrenHeader) == 0x40, "UpdateDataHeader"); + +typedef struct { uint64_t address, size; uint32_t state, pad[3]; } AudrenPoolIn; +typedef struct { uint32_t state, pad[3]; } AudrenPoolOut; +enum { POOL_INVALID, POOL_NEW, POOL_REQUEST_DETACH, POOL_DETACHED, POOL_REQUEST_ATTACH, POOL_ATTACHED, POOL_RELEASED }; + +typedef struct { uint32_t id; float mix[24]; uint8_t in_use, pad[11]; } AudrenChannelIn; +_Static_assert(sizeof(AudrenChannelIn) == 0x70, "voice channel resource"); + +typedef struct { uint8_t enable, pad; int16_t numerator[3], denominator[2]; } AudrenBiquad; +typedef struct { uint8_t enable,pad[3]; float numerator[3],denominator[2]; } AudrenBiquadFloat; +_Static_assert(sizeof(AudrenBiquadFloat)==0x18,"REV15 float biquad"); +typedef struct { + uint64_t address, size; int32_t start, end; uint8_t loop, end_of_stream, sent, pad; + int32_t loop_count; uint64_t context_address, context_size; int32_t loop_start, loop_end; +} AudrenWaveBuffer; +_Static_assert(sizeof(AudrenWaveBuffer) == 0x38, "wave buffer"); + +typedef struct { + uint32_t id, node_id; uint8_t is_new, in_use, play_state, sample_format; + uint32_t sample_rate; int32_t priority, sort_order; uint32_t channel_count; float pitch, volume; + AudrenBiquad biquads[2]; uint32_t wave_buffer_count; uint16_t wave_buffer_index, pad0; uint32_t pad1; + uint64_t extra_address, extra_size; int32_t mix_id; uint32_t splitter_id; + AudrenWaveBuffer wave_buffers[4]; int32_t channel_ids[6]; + uint8_t clear_voice_drop, flush_wave_buffer_count, pad2[2], flags, pad3, src_quality, pad4[17]; +} AudrenVoiceIn; +_Static_assert(sizeof(AudrenVoiceIn) == 0x170, "voice in parameter"); +/* REV15 enlarges only the two filter records (Q14 -> native float). The + * surrounding fields retain their order; the tail moves by 24 bytes. */ +typedef struct { uint8_t prefix[0x24]; AudrenBiquadFloat biquads[2]; uint8_t tail[0x134]; } AudrenVoiceInV2; +_Static_assert(sizeof(AudrenVoiceInV2)==0x188,"REV15 voice in parameter"); +typedef struct { uint64_t played_samples; uint32_t wave_buffers_consumed, voice_drops; } AudrenVoiceOut; +enum { PLAY_STARTED, PLAY_STOPPED, PLAY_PAUSED }; +enum { FMT_INVALID, FMT_PCM8, FMT_PCM16, FMT_PCM24, FMT_PCM32, FMT_PCMF32, FMT_ADPCM }; + +typedef struct { + float volume; uint32_t sample_rate; int32_t buffer_count; uint8_t in_use, dirty, pad0[2]; + int32_t mix_id; uint32_t effect_count, node_id, pad1[2]; float mix[24][24]; int32_t dest_mix_id; uint32_t dest_splitter_id, pad2; +} AudrenMixIn; +_Static_assert(sizeof(AudrenMixIn) == 0x930, "mix in parameter"); + +typedef struct { + uint8_t type, in_use, pad0[2]; uint32_t node_id; uint64_t pad1[3]; + char name[0x100]; uint32_t input_count; int8_t inputs[6]; uint8_t pad2, downmix_enabled; float downmix[4]; +} AudrenSinkIn; +_Static_assert(sizeof(AudrenSinkIn) == 0x140, "sink in parameter"); +typedef struct { uint32_t last_written_offset, pad0; uint64_t pad1[3]; } AudrenSinkOut; +enum { SINK_INVALID, SINK_DEVICE, SINK_CIRCULAR }; + +typedef struct { uint8_t type, is_new, enabled, pad0; int32_t mix_id; uint64_t buffer_address, buffer_size; int32_t order; uint32_t pad1; uint8_t specific[0xa0]; } AudrenEffectIn; +_Static_assert(sizeof(AudrenEffectIn) == 0xc0, "effect in parameter"); + +typedef struct { + uint8_t input[6], output[6]; uint16_t channel_max, channels; + int32_t sample_rate, lookahead_max, attack_time, release_time, lookahead; + float attack, release, threshold, input_gain, output_gain; + int32_t delay_min, delay_max; uint8_t status, statistics_enabled, statistics_reset, reserved; +} AudrenLimiterParam; +_Static_assert(sizeof(AudrenLimiterParam) == 68, "limiter effect wire layout"); + +typedef struct { + uint8_t input[24], output[24]; uint32_t channels, sample_rate, capacity, channel_max; + uint64_t send_info, send_storage, return_info, return_storage; + uint32_t sample_size, total_samples, frame_samples; +} AudrenAuxParam; +_Static_assert(__builtin_offsetof(AudrenAuxParam, send_info) == 64, "auxiliary effect wire layout"); + +void runtime_audio_register(void); diff --git a/src/host/audio_reverb.c b/src/host/audio_reverb.c new file mode 100644 index 0000000..e95e3a7 --- /dev/null +++ b/src/host/audio_reverb.c @@ -0,0 +1,83 @@ +/* From MacNativeGames games/totk/runtime/audio_reverb.c @b75be8c (MIT). [port] marks the Windows changes. */ +/* Compatible 3D reverb network, based on Ryujinx.Audio's Reverb3dState/Command, + DelayLine3d and DecayDelay. Copyright (c) Ryujinx Team and Contributors. + MIT license: third_party/licenses/Ryujinx-Audio.txt. Samples are normalized floats. */ +#include "audio_reverb.h" +#include "host.h" // [port] runtime.h +#include +#include +#include + +typedef struct { float *data; unsigned cap, delay, read, write; } Delay; +struct Reverb3d { + Reverb3dParam p; Delay fdn[4], allpass1[4], allpass2[4], pre, center; + unsigned early_time[20], reflection_time; + float pre_previous, pre_gain, pre_target, early_gain, late_gain, diffusion; + float direct[4], current[4], output[4], history[4]; +}; +static const float fdn_min[4] = {5,6,13,14}, fdn_max[4] = {45.704f,82.782f,149.94f,271.58f}; +static const float ap1_max[4] = {17,13,9,7}, ap2_max[4] = {19,11,10,6}; +static const float early_times[20] = {.017136f,.059154f,.16173f,.39019f,.42526f,.45541f,.68974f,.74591f,.83384f,.8595f,0,.075024f,.16879f,.2999f,.33744f,.3719f,.59901f,.71674f,.81786f,.85166f}; +static const float early_gains[20] = {.67096f,.61027f,1,.35680f,.68361f,.65978f,.51939f,.24712f,.45945f,.45021f,.64196f,.54879f,.92925f,.38270f,.72867f,.69794f,.5464f,.24563f,.45214f,.44042f}; +static unsigned samples(unsigned khz, float ms) { return (unsigned)nearbyintf(khz * ms); } +static void delay_set(Delay *d, unsigned n) { if (n > d->cap) runtime_fail("reverb delay exceeds capacity"); d->delay = n; d->write = (d->read + n) % (d->cap + 1); } +static void delay_init(Delay *d, unsigned khz, float ms) { d->cap = samples(khz, ms); d->data = calloc(d->cap + 1, sizeof(float)); if (!d->data) runtime_fail("reverb delay allocation"); delay_set(d, d->cap); } +static float delay_update(Delay *d, float v) { + d->data[d->write++] = v; float out = d->data[d->read++]; + if (d->read >= d->cap) d->read = 0; + if (d->write >= d->cap) d->write = 0; + return out; +} +static float delay_tap(const Delay *d, unsigned at) { int i = (int)d->write - (int)at - 1; if (i < 0) i += (int)d->cap + 1; return d->data[i]; } +static float allpass(Delay *d, float x, float gain) { float v = x - gain * d->data[d->read]; return delay_update(d, v) + gain * v; } +static float gain10(float x) { return x >= 0 ? 1 : x <= -5.3f ? 0 : powf(10, x); } +static float radians(float x) { return x * 3.14159265358979323846f / 180; } +static void validate(const Reverb3dParam *p) { + if ((p->channels != 1 && p->channels != 2 && p->channels != 4 && p->channels != 6) || p->sample_rate < 1000 || p->sample_rate > 192000 || + !(p->decay_time > 0) || !(p->hf_decay_ratio > 0) || !(p->density >= 0 && p->density <= 100) || + !(p->reflection_delay >= 0 && p->reverb_delay >= 0 && p->reflection_delay + p->reverb_delay <= .4f)) runtime_fail("invalid reverb3d parameters"); +} +Reverb3d *audio_reverb_create(const Reverb3dParam *p) { + validate(p); Reverb3d *s = calloc(1, sizeof *s); if (!s) runtime_fail("reverb allocation"); unsigned khz = p->sample_rate / 1000; + for (int i=0;i<4;i++) { delay_init(&s->fdn[i],khz,fdn_max[i]); delay_init(&s->allpass1[i],khz,ap1_max[i]); delay_init(&s->allpass2[i],khz,ap2_max[i]); } + delay_init(&s->pre,khz,400); delay_init(&s->center,khz,5); audio_reverb_update(s,p); return s; +} +void audio_reverb_free(Reverb3d *s) { if (!s) return; for (int i=0;i<4;i++) { free(s->fdn[i].data); free(s->allpass1[i].data); free(s->allpass2[i].data); } free(s->pre.data); free(s->center.data); free(s); } +void audio_reverb_update(Reverb3d *s, const Reverb3dParam *p) { + validate(p); s->p = *p; unsigned khz = p->sample_rate / 1000; + memset(s->history,0,sizeof s->history); s->pre_previous=0; + s->early_gain=gain10(fminf(p->room_gain+p->reflections_gain,5000)/2000); s->late_gain=gain10(fminf(p->room_gain+p->reverb_gain,5000)/2000); + float hf=gain10(p->room_hf/2000); + if (hf<1) { float a=1-hf,b=2-2*hf*cosf(radians(256*p->hf_reference/p->sample_rate)); s->pre_gain=(b-sqrtf(b*b-4*a*a))/(2*a); s->pre_target=1-s->pre_gain; } + else { s->pre_gain=0; s->pre_target=1; } + s->reflection_time=samples(khz,1000*(p->reflection_delay+p->reverb_delay)); s->diffusion=.6f*p->diffusion*.01f; + for (int i=0;i<4;i++) { + delay_set(&s->fdn[i],samples(khz,fdn_min[i]+p->density/100*(fdn_max[i]-fdn_min[i]))); + float n=(float)(s->fdn[i].delay+s->allpass1[i].delay+s->allpass2[i].delay); + float a=-60*n/(p->decay_time*p->sample_rate),b=a/p->hf_decay_ratio; + float angle=radians(128*.5f*p->hf_reference/p->sample_rate),c=cosf(angle)/sinf(angle),d=gain10((b-a)/40),e=gain10((b+a)/40)*.7071f; + s->direct[i]=e*(d*c+1)/(c+d); s->current[i]=e*(1-d*c)/(c+d); s->output[i]=(c-d)/(c+d); + } + for (int i=0;i<20;i++) { unsigned n=samples(khz,p->reflection_delay*1000+early_times[i]*1000*(p->reverb_delay*.9998f+.02f)); s->early_time[i]=npre.cap?n:s->pre.cap; } +} +void audio_reverb_process(Reverb3d *s, float *const input[6], float *const output[6], int frames) { + static const unsigned stereo[20]={0,0,0,1,1,1,1,0,0,0,1,1,1,0,0,0,0,1,1,1}; + static const unsigned quad[20]={0,0,0,1,1,1,1,2,2,2,1,1,1,0,0,0,0,3,3,3}; + /* Preserve the reference's legacy surround channel ordering (L,R,BL,BR,C,LFE). */ + static const unsigned surround[20]={4,5,0,5,0,5,1,5,1,5,1,5,1,5,2,5,2,5,2,5}; + const int channels=s->p.channels; + for (int k=0;kpre,s->reflection_time); + for (int i=0;i<20;i++) { unsigned out=channels==1?0:channels==2?stereo[i]:channels==4?quad[i]:surround[i]; float x=delay_tap(&s->pre,s->early_time[i])*early_gains[i]; early[out]+=x; if(channels==6)early[out]+=x; } + for (int c=0;cpre_previous=sum*s->pre_target+s->pre_previous*s->pre_gain; delay_update(&s->pre,s->pre_previous); + for (int i=0;i<4;i++) { float x=s->fdn[i].data[s->fdn[i].read]; fb[i]=x*s->direct[i]+s->history[i]; s->history[i]=x*s->current[i]+fb[i]*s->output[i]; } + const float feedback[4]={fb[2]+fb[1],-fb[0]-fb[3],fb[0]-fb[3],fb[1]-fb[2]}; + for (int i=0;i<4;i++) { v[i]=allpass(&s->allpass2[i],allpass(&s->allpass1[i],tap*s->late_gain+feedback[i],s->diffusion),-.9f*s->diffusion); delay_update(&s->fdn[i],v[i]); } + for (int c=0;cearly_gain+delay_update(&s->center,(v[2]-v[3])*.5f)+dry[c]*s->p.dry_gain; + else output[c][k]=early[c]*s->early_gain+v[c==5?3:c]+dry[c]*s->p.dry_gain; + } + if(channels==1)output[0][k]+=v[1]; + } +} diff --git a/src/host/audio_reverb.h b/src/host/audio_reverb.h new file mode 100644 index 0000000..8a750c2 --- /dev/null +++ b/src/host/audio_reverb.h @@ -0,0 +1,18 @@ +/* From MacNativeGames games/totk/runtime/audio_reverb.h @b75be8c (MIT). [port] marks the Windows changes. */ +#ifndef TOTK_AUDIO_REVERB_H +#define TOTK_AUDIO_REVERB_H +#include +typedef struct { + uint8_t input[6], output[6]; uint16_t channel_max, channels; + uint32_t reserved, sample_rate; + float room_hf, hf_reference, decay_time, hf_decay_ratio, room_gain, reflections_gain, + reverb_gain, diffusion, reflection_delay, reverb_delay, density, dry_gain; + uint8_t status; +} Reverb3dParam; +_Static_assert(__builtin_offsetof(Reverb3dParam, status) == 72, "reverb3d wire layout"); +typedef struct Reverb3d Reverb3d; +Reverb3d *audio_reverb_create(const Reverb3dParam *p); +void audio_reverb_free(Reverb3d *s); +void audio_reverb_update(Reverb3d *s, const Reverb3dParam *p); +void audio_reverb_process(Reverb3d *s, float *const input[6], float *const output[6], int frames); +#endif diff --git a/src/host/boot.c b/src/host/boot.c new file mode 100644 index 0000000..5ad94df --- /dev/null +++ b/src/host/boot.c @@ -0,0 +1,142 @@ +/* Starts the recompiled game the way MacNativeGames' loader.c boot() does: the main guest thread enters the SDK's own + * nn::init::Start(main thread handle, 0, loader callback, module initializers, module finalizers). + * totk + */ +#include +#include + +#include "host.h" +#include "version.h" // build/version.h (tools/build_game.py, tools/version.py) +#ifdef _WIN32 +#include +#endif + +static void loader_ready(Cpu *c) { (void)c; fprintf(stderr, "[totk] rtld callback: modules linked\n"); } + +static void module_inits(Cpu *c) { + (void)c; + fprintf(stderr, "[totk] SDK startup returned; running module constructors\n"); + for (int i = 0; i < g_nmodules; i++) { + Module *m = &g_modules[i]; + if (!m->init) continue; + fprintf(stderr, "[totk] ENTER %s DT_INIT\n", m->name); + GCALL0(m->init); + fprintf(stderr, "[totk] RETURN %s DT_INIT\n", m->name); + } +} + +static void module_finis(Cpu *c) { (void)c; runtime_fail("guest exited: module finalizers not yet implemented"); } + +// The guest window is claimed before anything else in the process can allocate there (the C runtime, thread stacks). +// The executable is also linked without high-entropy ASLR, which otherwise scatters the process's own allocations +// over the whole 47-bit space, the window included. +static bool g_window; +__attribute__((constructor(101))) static void claim_guest_window(void) { g_window = plat_reserve_guest(GUEST_LO, GUEST_HI); } + +static void *game_main(void *unused); + +// The game's folder (tools/setup.py makes it: data\\, userdata\\, ultratears.cfg, UltraTears.exe) becomes the working +// directory: the exe's folder, or its parent for build\\UltraTears.exe. Every path below is relative to it; the defaults +// are setup.py's layout (data\\exefs, data\\romfs, data\\shaders\\ryujinx, data\\textures, data\\fonts, userdata). +// ultratears.cfg there may name other places: lines "name=value" give the ExeFS (exefs=), the RomFS image (romfs=), +// shaders, texture cache, saves and fonts, each as the TOTK_* variable it stands for unless that is already set (a +// TOTK_* line sets one directly). +static char g_exefs[4096]; +static const char *default_exefs(void) { + const char *exefs = "data/exefs"; +#ifdef _WIN32 + char path[MAX_PATH]; + DWORD n = GetModuleFileNameA(NULL, path, sizeof path); + char *slash = n && n < sizeof path ? strrchr(path, '\\') : NULL; + if (!slash) return exefs; + *slash = 0; + char cfg[MAX_PATH + 32]; + snprintf(cfg, sizeof cfg, "%s\\ultratears.cfg", path); + FILE *f = fopen(cfg, "r"); + if (!f && (slash = strrchr(path, '\\')) && !_stricmp(slash, "\\build")) { // build\\UltraTears.exe: the folder above + *slash = 0; + snprintf(cfg, sizeof cfg, "%s\\ultratears.cfg", path); + f = fopen(cfg, "r"); + } + if (!SetCurrentDirectoryA(path)) fprintf(stderr, "[totk] cannot enter %s\n", path); + if (f) { + char line[4600]; + while (fgets(line, sizeof line, f)) { + line[strcspn(line, "\r\n")] = 0; + char *eq = strchr(line, '='); + if (!eq || line[0] == '#') continue; + *eq = 0; + static const char *const keys[][2] = { // the file's names -> the runtime's variables + {"romfs", "TOTK_ROMFS"}, {"shaders", "TOTK_SHADER_DIR"}, {"textures", "TOTK_BC7_DIR"}, + {"saves", "TOTK_PROFILE_DIR"}, {"fonts", "TOTK_SHARED_FONTS"}}; + if (!strcmp(line, "exefs")) snprintf(g_exefs, sizeof g_exefs, "%s", eq + 1), exefs = g_exefs; + else if (!strncmp(line, "TOTK_", 5)) setenv(line, eq + 1, 0); + else + for (unsigned i = 0; i < sizeof keys / sizeof *keys; i++) + if (!strcmp(line, keys[i][0])) setenv(keys[i][1], eq + 1, 0); + } + fclose(f); + fprintf(stderr, "[totk] game folder %s (%s)\n", path, cfg); + } else + fprintf(stderr, "[totk] game folder %s (no ultratears.cfg: setup.py's layout)\n", path); +#endif + return exefs; +} + +int main(int argc, char **argv) { + setvbuf(stderr, NULL, _IONBF, 0); + fprintf(stderr, "UltraTears %s (commit %s)\n", ULTRS_VERSION, ULTRS_COMMIT); // first line of every log + if (argc > 2) { fprintf(stderr, "usage: UltraTears [exefs dir]\n"); return 2; } + const char *exefs = argc == 2 ? argv[1] : default_exefs(); + plat_init(); + plat_crash_handler_install(); + if (!g_window) { +#ifdef _WIN32 + for (uint64_t a = GUEST_LO; a < GUEST_HI;) { // what already occupies the window + MEMORY_BASIC_INFORMATION mi; + if (!VirtualQuery((void *)(uintptr_t)a, &mi, sizeof mi)) break; + if (mi.State != MEM_FREE) + fprintf(stderr, " occupied %llx +%llx state %lx type %lx\n", (unsigned long long)(uintptr_t)mi.BaseAddress, + (unsigned long long)mi.RegionSize, mi.State, mi.Type); + a = (uint64_t)(uintptr_t)mi.BaseAddress + mi.RegionSize; + } +#endif + runtime_fail("cannot reserve the guest address window"); + } + uint64_t t0 = runtime_now_ns(); + loader_load(exefs); + dispatch_init(); + fprintf(stderr, "[totk] modules loaded and linked in %.0f ms\n", (runtime_now_ns() - t0) / 1e6); + + runtime_profiler_start(); + + // The window takes the main thread (SDL wants its events there); the game gets its own. TOTK_HEADLESS=1: no + // window, no sound device, the game on the main thread (automated runs). + const char *headless = getenv("TOTK_HEADLESS"); + if (headless && strcmp(headless, "0")) game_main(NULL); + if (!window_init()) runtime_fail("window (TOTK_HEADLESS=1 runs without one)"); + pthread_attr_t a; + pthread_attr_init(&a); + pthread_attr_setstacksize(&a, 16 << 20); // the host stack under the main guest thread's calls + pthread_t t; + if (pthread_create(&t, &a, game_main, NULL)) runtime_fail("game thread"); + window_loop(); +} + +static void *game_main(void *unused) { + (void)unused; + runtime_memory_init(); + host_services_register(); + uint32_t main_handle = svc_thread_main_init(); + Cpu *c = guest_cpu_new(runtime_main_stack(0x100000), guest_tls_alloc()); // main.npdm: 1 MiB main stack + guest_cpu_set(c); + + uint64_t start = (uint64_t)(uintptr_t)runtime_symbol("_ZN2nn4init5StartEmmPFvvES2_S2_"); + if (!start) runtime_fail("nn::init::Start not found"); + uint64_t ready = hle_register("host_loader_ready", loader_ready); + uint64_t inits = hle_register("host_module_inits", module_inits); + uint64_t finis = hle_register("host_module_finis", module_finis); + fprintf(stderr, "[totk] ENTER nn::init::Start\n"); + GCALL(start, main_handle, 0, ready, inits, finis); + runtime_fail("SDK Start unexpectedly returned"); +} diff --git a/src/host/fs.c b/src/host/fs.c new file mode 100644 index 0000000..4f4b1b2 --- /dev/null +++ b/src/host/fs.c @@ -0,0 +1,730 @@ +/* nn::fs for the recompiled game: the read-only RomFS and writable save, temporary and cache storage. + * Ported from MacNativeGames games/totk/runtime/fs_host.c and romfs.c @b75be8c (MIT). Changes: + * - guest-register binding through HLE_NATIVE (every nn::fs entry point takes integers and pointers only); + * - Windows file calls (mkdir, directory types via stat, rename that refuses to replace, _commit); + * - RomFS: the runtime index (tools/make_romfs_index.py, MacNativeGames' TOTKRFS1 layout) and the extracted + * romfs.bin, both already verified at extraction, or (the player's dump) a folder of plain files, indexed by + * walking it at start (tools/setup.py checks it against the expected version); overlays (mods, data patches), split images and the startup + * re-hash are not ported yet. RomFS directory listing stays unsupported, as in MacNativeGames. + * Save, temporary and cache storage are host directories under the profile directory (TOTK_PROFILE_DIR, default + * userdata); application save data is created on first start, as the system launcher does. */ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#ifdef _WIN32 +#include +#endif + +#include "host.h" +#include "../runtime/hle.h" + +#define R_PATH_NOT_FOUND 0x202u +#define R_ALREADY_EXISTS 0x402u +#define R_NOT_EMPTY 0x1002u +#define R_TARGET_NOT_FOUND 0x7d402u +#define R_INVALID_ARGUMENT 0x2ee202u +#define R_INVALID_PATH 0x2ee602u +#define R_OPEN_MODE 0x307402u +#define R_APPEND_REQUIRED 0x2f5e02u +#define R_ROM_UNSUPPORTED 0x6402u +#define R_ROM_INVALID 0xe402u +#define TOTK_APPLICATION_ID 0x0100f2c0115b6000ULL +enum { M_FREE, M_SAVE, M_TEMP, M_CACHE }; +enum { OPEN_READ = 1, OPEN_WRITE = 2, OPEN_APPEND = 4 }; +#define ROM_BASE 0x100000000ULL +#define NROM 65536 +#define FILE_BASE 0x200000000ULL +#define DIR_BASE 0x300000000ULL +#define NFILES 1024 +#define NDIRS 256 + +void runtime_account_uid(uint64_t out[2]); +const char *runtime_profile_dir(void); + +static void need(int c, const char *m) { if (!c) runtime_fail(m); } +static int make_dir_p(const char *p) { return mkdir(p, 0700); } + +/* ======================================== RomFS ======================================== */ +static uint8_t *index_bytes; +static uint64_t file_count, strings_offset; +static int image_fd = -1; +static char mount_name[32]; +static uint64_t rom_handles[NROM]; /* file index + 1; 0 is free */ +static pthread_mutex_t rom_lock = PTHREAD_MUTEX_INITIALIZER; +typedef struct { uint64_t offset, size; uint32_t name, len; } Entry; +_Static_assert(sizeof(Entry) == 24, "index entry layout"); +static Entry entry(uint64_t i) { Entry e; memcpy(&e, index_bytes + 24 + i * 24, 24); return e; } +static const char *entry_name(Entry e) { return (const char *)index_bytes + strings_offset + e.name; } + +/* A RomFS folder (the player's extraction: plain files): the index is built by walking it, in the same layout as the + runtime index, with each entry's offset unused; a file is opened by its path when the game opens it. */ +static char rom_dir[4096]; +static int rom_folder; +static int rom_fds[NROM]; +typedef struct { char *path; uint64_t size; } Walked; +static Walked *walked; +static size_t nwalked, walked_cap; +static void walk_add(const char *path, uint64_t size) { + if (nwalked == walked_cap) { + walked_cap = walked_cap ? walked_cap * 2 : 65536; + walked = realloc(walked, walked_cap * sizeof *walked); + need(walked != NULL, "RomFS folder index memory"); + } + walked[nwalked].path = strdup(path); + walked[nwalked++].size = size; +} +static void walk(const char *root, const char *rel) { + char pattern[4096]; + snprintf(pattern, sizeof pattern, "%s/%s%s*", root, rel, *rel ? "/" : ""); +#ifdef _WIN32 + WIN32_FIND_DATAA d; + HANDLE h = FindFirstFileExA(pattern, FindExInfoBasic, &d, FindExSearchNameMatch, NULL, FIND_FIRST_EX_LARGE_FETCH); + if (h == INVALID_HANDLE_VALUE) return; + do { + if (!strcmp(d.cFileName, ".") || !strcmp(d.cFileName, "..")) continue; + char sub[4096]; + snprintf(sub, sizeof sub, "%s%s%s", rel, *rel ? "/" : "", d.cFileName); + if (d.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) walk(root, sub); + else walk_add(sub, (uint64_t)d.nFileSizeHigh << 32 | d.nFileSizeLow); + } while (FindNextFileA(h, &d)); + FindClose(h); +#else + pattern[strlen(pattern) - 1] = 0; + DIR *dir = opendir(pattern); + if (!dir) return; + for (struct dirent *e; (e = readdir(dir));) { + if (!strcmp(e->d_name, ".") || !strcmp(e->d_name, "..")) continue; + char sub[4096], full[8192]; + snprintf(sub, sizeof sub, "%s%s%s", rel, *rel ? "/" : "", e->d_name); + snprintf(full, sizeof full, "%s/%s", root, sub); + struct stat st; + if (stat(full, &st)) continue; + if (S_ISDIR(st.st_mode)) walk(root, sub); + else walk_add(sub, (uint64_t)st.st_size); + } + closedir(dir); +#endif +} +static int walked_cmp(const void *a, const void *b) { return strcmp(((const Walked *)a)->path, ((const Walked *)b)->path); } +static void index_folder(const char *dir) { + uint64_t t0 = runtime_now_ns(); + walk(dir, ""); + need(nwalked > 0, "RomFS folder is empty"); + qsort(walked, nwalked, sizeof *walked, walked_cmp); + size_t names = 0; + for (size_t i = 0; i < nwalked; i++) names += strlen(walked[i].path) + 1; + file_count = nwalked; + strings_offset = 24 + file_count * 24; + index_bytes = malloc(strings_offset + names); + need(index_bytes != NULL, "RomFS folder index memory"); + memcpy(index_bytes, "TOTKRFS1", 8); + memcpy(index_bytes + 8, &file_count, 8); + memcpy(index_bytes + 16, &strings_offset, 8); + uint32_t at = 0; + for (size_t i = 0; i < nwalked; i++) { + uint32_t len = (uint32_t)strlen(walked[i].path) + 1; + Entry e = {0, walked[i].size, at, len}; + memcpy(index_bytes + 24 + i * 24, &e, 24); + memcpy(index_bytes + strings_offset + at, walked[i].path, len); + at += len; + free(walked[i].path); + } + free(walked); + walked = NULL; + fprintf(stderr, "[totk] RomFS: %llu files in the folder %s (indexed in %.0f ms)\n", (unsigned long long)file_count, dir, + (runtime_now_ns() - t0) / 1e6); +} + +void runtime_fs_init(void) { + if (image_fd >= 0 || rom_folder) return; + const char *dir = getenv("TOTK_ROMFS"); + if (!dir) dir = "data/romfs"; + char path[4096]; + snprintf(path, sizeof path, "%s/romfs.bin", dir); + struct stat st; + if (stat(path, &st)) { // no image: a folder of plain files + need(snprintf(rom_dir, sizeof rom_dir, "%s", dir) < (int)sizeof rom_dir, "RomFS folder path"); + rom_folder = 1; + index_folder(rom_dir); + return; + } + snprintf(path, sizeof path, "%s/runtime-index.bin", dir); + FILE *f = fopen(path, "rb"); + if (!f) { fprintf(stderr, "[totk] %s missing: run tools/make_romfs_index.py\n", path); runtime_fail("RomFS index missing"); } + fseek(f, 0, SEEK_END); + long size = ftell(f); + fseek(f, 0, SEEK_SET); + need(size >= 24 && size < 128L * 1024 * 1024, "RomFS index size"); + index_bytes = malloc((size_t)size); + need(index_bytes && fread(index_bytes, 1, (size_t)size, f) == (size_t)size, "read RomFS index"); + fclose(f); + need(!memcmp(index_bytes, "TOTKRFS1", 8), "RomFS index format"); + memcpy(&file_count, index_bytes + 8, 8); + memcpy(&strings_offset, index_bytes + 16, 8); + need(file_count < 1000000 && strings_offset == 24 + file_count * 24 && strings_offset < (uint64_t)size, "RomFS index layout"); + snprintf(path, sizeof path, "%s/romfs.bin", dir); + image_fd = open(path, O_RDONLY | O_BINARY); + need(image_fd >= 0, "RomFS image missing"); + fprintf(stderr, "[totk] RomFS: %llu files from %s\n", (unsigned long long)file_count, dir); +} +/* First entry whose path is >= p */ +static uint64_t lower(const char *p) { + uint64_t lo = 0, hi = file_count; + while (lo < hi) { uint64_t m = lo + (hi - lo) / 2; if (strcmp(entry_name(entry(m)), p) < 0) lo = m + 1; else hi = m; } + return lo; +} +static uint64_t file_index(const char *p) { + uint64_t i = lower(p); + return i < file_count && !strcmp(p, entry_name(entry(i))) ? i : UINT64_MAX; +} +static int folder_exists(const char *folder) { + size_t n = strlen(folder); + uint64_t i = lower(folder); + return i < file_count && !strncmp(entry_name(entry(i)), folder, n); +} +/* "mount:/a/b" (or "mount://a" from audio stream paths) -> "a/b"; NULL on traversal or a foreign mount */ +static const char *rom_path(const char *path) { + size_t n = strlen(mount_name); + if (!n || !path || strncmp(path, mount_name, n) || path[n] != ':' || path[n + 1] != '/') return NULL; + path += n + 2; + while (*path == '/') ++path; + if (!*path || strchr(path, '\\')) return NULL; + for (const char *part = path; *part;) { + const char *end = strchr(part, '/'); + size_t size = end ? (size_t)(end - part) : strlen(part); + if (!size || (size == 1 && part[0] == '.') || (size == 2 && part[0] == '.' && part[1] == '.')) return NULL; + if (!end) break; + part = end + 1; + } + return path; +} +static int fs_log_on = -1; +static uint64_t fs_log_t0; +void runtime_fs_log(const char *what, const char *path) { + if (fs_log_on < 0) { fs_log_on = getenv("TOTK_FS_LOG") != NULL; fs_log_t0 = runtime_now_ns(); } + if (fs_log_on) fprintf(stderr, "[totk-fs] %8.3f %s %s\n", (runtime_now_ns() - fs_log_t0) / 1e9, what, path); +} +static int romfs_owns(const char *path) { + size_t n = strlen(mount_name); + return n && path && !strncmp(path, mount_name, n) && path[n] == ':'; +} +static int romfs_handle(uint64_t h) { return h > ROM_BASE && h < ROM_BASE + NROM; } +static unsigned romfs_open(uint64_t *out, const char *path, int mode) { + runtime_fs_log("rom", path ? path : "(null)"); + if (!out || mode != OPEN_READ) return R_ROM_UNSUPPORTED; + const char *p = rom_path(path); + if (!p) return R_PATH_NOT_FOUND; + uint64_t i = file_index(p); + if (i == UINT64_MAX) return R_PATH_NOT_FOUND; + int fd = -1; + if (rom_folder) { + char full[8192]; + snprintf(full, sizeof full, "%s/%s", rom_dir, p); + if ((fd = open(full, O_RDONLY | O_BINARY)) < 0) return R_PATH_NOT_FOUND; + } + pthread_mutex_lock(&rom_lock); + for (uint64_t h = 1; h < NROM; h++) + if (!rom_handles[h]) { rom_handles[h] = i + 1; rom_fds[h] = fd; *out = ROM_BASE + h; pthread_mutex_unlock(&rom_lock); return 0; } + pthread_mutex_unlock(&rom_lock); + runtime_fail("RomFS handle table full"); +} +static unsigned romfs_read(uint64_t *out, uint64_t h, int64_t offset, void *buffer, uint64_t size) { + if (!out || offset < 0 || (!buffer && size)) return R_ROM_INVALID; + uint64_t i = rom_handles[h - ROM_BASE]; + if (!i) return R_ROM_INVALID; + Entry e = entry(i - 1); + if ((uint64_t)offset > e.size) return R_ROM_INVALID; + uint64_t n = size < e.size - (uint64_t)offset ? size : e.size - (uint64_t)offset, done = 0; + guest_prepare_write(buffer, n); + while (done < n) { + ssize_t r = pread(rom_folder ? rom_fds[h - ROM_BASE] : image_fd, (uint8_t *)buffer + done, (size_t)(n - done), (off_t)(e.offset + (uint64_t)offset + done)); + need(r > 0, "RomFS image read"); + done += (uint64_t)r; + } + *out = n; + return 0; +} +static unsigned romfs_size(int64_t *out, uint64_t h) { + uint64_t i = rom_handles[h - ROM_BASE]; + if (!out || !i) return R_ROM_INVALID; + *out = (int64_t)entry(i - 1).size; + return 0; +} +static void romfs_close(uint64_t h) { + pthread_mutex_lock(&rom_lock); + need(rom_handles[h - ROM_BASE] != 0, "close of a closed RomFS handle"); + rom_handles[h - ROM_BASE] = 0; + if (rom_folder) close(rom_fds[h - ROM_BASE]); + pthread_mutex_unlock(&rom_lock); +} +static unsigned romfs_entry_type(int *out, const char *path) { + if (!out) return R_ROM_INVALID; + const char *p = rom_path(path); + if (!p) return R_PATH_NOT_FOUND; + if (file_index(p) != UINT64_MAX) { *out = 1; return 0; } + char dir[4096]; + if (snprintf(dir, sizeof dir, "%s/", p) >= (int)sizeof dir) return R_ROM_INVALID; + if (folder_exists(dir)) { *out = 0; return 0; } + return R_PATH_NOT_FOUND; +} +static unsigned mount_rom(const char *name, void *cache, uint64_t size) { + (void)cache; (void)size; + runtime_fs_init(); + if (!name || !*name || strlen(name) >= sizeof mount_name || strchr(name, '/') || strchr(name, ':')) return R_ROM_INVALID; + if (*mount_name && strcmp(name, mount_name)) runtime_fail("second RomFS mount not implemented"); + strcpy(mount_name, name); + fprintf(stderr, "[totk] fs mount %s: -> RomFS\n", name); + return 0; +} +static unsigned rom_cache_size(uint64_t *out) { if (!out) return R_ROM_INVALID; *out = 0; return 0; } + +/* =============================== save, temporary and cache storage =============================== */ +typedef struct { int kind; char name[32]; char root[1024]; } Mount; +typedef struct { int fd, mode, save; } File; /* save: on a save-data mount (counts for runtime_fs_save_busy) */ +typedef struct { DIR *dir; char path[1024]; int mode; } Dir; +static Mount mounts[16]; +static File files[NFILES]; +static Dir dirs[NDIRS]; +static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER; +static char profile[1024], save_root[1024]; +static pthread_once_t once = PTHREAD_ONCE_INIT; + +/* Save in progress: a save-data file open for writing, or save data changed in the last 3 s and not committed yet. */ +static _Atomic int save_writers, save_dirty; +static _Atomic uint64_t save_changed_ns; +static void save_touched(void) { atomic_store(&save_changed_ns, runtime_now_ns()); atomic_store(&save_dirty, 1); } +int runtime_fs_save_busy(void) { + if (atomic_load(&save_writers) > 0) return 1; + return atomic_load(&save_dirty) && runtime_now_ns() - atomic_load(&save_changed_ns) < 3000000000ull; +} +static void make_dir(const char *p) { if (make_dir_p(p) && errno != EEXIST) runtime_fail("create profile storage directory"); } +static int is_dir(const char *p) { struct stat st; return !stat(p, &st) && S_ISDIR(st.st_mode); } +static int remove_tree(const char *path) { + DIR *d = opendir(path); + if (!d) return errno == ENOENT ? 0 : -1; + struct dirent *e; + int fail = 0; + char child[2048]; + while ((e = readdir(d))) { + if (!strcmp(e->d_name, ".") || !strcmp(e->d_name, "..")) continue; + snprintf(child, sizeof child, "%s/%s", path, e->d_name); + fail |= is_dir(child) ? remove_tree(child) : unlink(child); + } + closedir(d); + return fail | rmdir(path); +} +static void init(void) { + need(snprintf(profile, sizeof profile, "%s", runtime_profile_dir()) < (int)sizeof profile, "profile path"); + uint64_t uid[2]; + runtime_account_uid(uid); + char p[1100]; + make_dir(profile); + snprintf(p, sizeof p, "%s/save", profile); make_dir(p); + need(snprintf(save_root, sizeof save_root, "%s/save/%016llx%016llx", profile, (unsigned long long)uid[1], (unsigned long long)uid[0]) < (int)sizeof save_root, "save path"); + make_dir(save_root); /* launcher-style EnsureSaveData */ + snprintf(p, sizeof p, "%s/temp", profile); remove_tree(p); make_dir(p); + snprintf(p, sizeof p, "%s/cache", profile); make_dir(p); + fprintf(stderr, "[totk] host save storage: %s\n", save_root); +} +static void ready(void) { pthread_once(&once, init); } + +static unsigned mount_add(const char *name, int kind, const char *root) { + if (!name || !*name || strlen(name) >= 32 || strchr(name, ':') || strchr(name, '/')) return R_INVALID_ARGUMENT; + pthread_mutex_lock(&lock); + int slot = -1; + for (int i = 0; i < 16; i++) { + if (mounts[i].kind && !strcmp(mounts[i].name, name)) { pthread_mutex_unlock(&lock); return R_ALREADY_EXISTS; } + if (!mounts[i].kind && slot < 0) slot = i; + } + if (slot < 0) runtime_fail("fs mount table full"); + mounts[slot].kind = kind; + strcpy(mounts[slot].name, name); + snprintf(mounts[slot].root, sizeof mounts[slot].root, "%s", root); + pthread_mutex_unlock(&lock); + fprintf(stderr, "[totk] fs mount %s: -> %s\n", name, root); + return 0; +} +static int component_ok(const char *c, size_t n) { return n && !(n == 1 && c[0] == '.') && !(n == 2 && c[0] == '.' && c[1] == '.'); } +/* "name:/a/b" -> host path; 0 on success; *kind = the mount's kind (M_SAVE ...) when kind is given */ +static unsigned host_path_kind(const char *path, char *out, size_t cap, int *kind) { + if (!path) return R_INVALID_PATH; + const char *colon = strchr(path, ':'); + if (!colon || colon[1] != '/') return R_INVALID_PATH; + size_t n = (size_t)(colon - path); + pthread_mutex_lock(&lock); + const Mount *m = NULL; + for (int i = 0; i < 16; i++) if (mounts[i].kind && strlen(mounts[i].name) == n && !strncmp(mounts[i].name, path, n)) m = &mounts[i]; + char root[1024]; + if (m) strcpy(root, m->root); + int k = m ? m->kind : M_FREE; + pthread_mutex_unlock(&lock); + if (!m) return R_PATH_NOT_FOUND; + if (kind) *kind = k; + const char *rest = colon + 1; + for (const char *q = rest + 1; *q;) { + const char *e = strchr(q, '/'); + size_t len = e ? (size_t)(e - q) : strlen(q); + if (!e && !len) break; + if (!component_ok(q, len) || memchr(q, '\\', len)) return R_INVALID_PATH; + q = e ? e + 1 : q + len; + } + if (snprintf(out, cap, "%s%s", root, rest) >= (int)cap) return R_INVALID_PATH; + size_t L = strlen(out); + while (L > strlen(root) && out[L - 1] == '/') out[--L] = 0; + return 0; +} +static unsigned host_path(const char *path, char *out, size_t cap) { return host_path_kind(path, out, cap, NULL); } +static unsigned save_change_path(const char *path, char *out, size_t cap) { + int kind = M_FREE; + unsigned r = host_path_kind(path, out, cap, &kind); + if (!r && kind == M_SAVE) save_touched(); + return r; +} +static unsigned errno_result(void) { + return errno == ENOENT || errno == ENOTDIR ? R_PATH_NOT_FOUND : errno == EEXIST ? R_ALREADY_EXISTS + : errno == ENOTEMPTY ? R_NOT_EMPTY : R_INVALID_ARGUMENT; +} + +static unsigned mount_save(const char *name, const uint64_t *uid) { + ready(); + uint64_t me[2]; + runtime_account_uid(me); + if (!uid || uid[0] != me[0] || uid[1] != me[1]) return R_TARGET_NOT_FOUND; + return mount_add(name, M_SAVE, save_root); +} +static unsigned char save_exists(uint64_t application, const uint64_t *uid) { + ready(); + uint64_t me[2]; + runtime_account_uid(me); + return application == TOTK_APPLICATION_ID && uid && uid[0] == me[0] && uid[1] == me[1]; +} +/* Other titles' save data (e.g. data-link bonuses) is not present on this host. */ +static unsigned mount_save_ro(const char *name, uint64_t application, const uint64_t *uid) { + if (!save_exists(application, uid)) return R_TARGET_NOT_FOUND; + return mount_add(name, M_SAVE, save_root); +} +static unsigned mount_temp(const char *name) { ready(); char p[1100]; snprintf(p, sizeof p, "%s/temp", profile); return mount_add(name, M_TEMP, p); } +static void cache_dir(int index, char *out, size_t cap) { snprintf(out, cap, "%s/cache/%d", profile, index); } +static unsigned create_cache(int index, int64_t size, int64_t journal) { + ready(); + if (index < 0 || size < 0 || journal < 0) return R_INVALID_ARGUMENT; + char p[1100]; + cache_dir(index, p, sizeof p); + if (make_dir_p(p)) return errno == EEXIST ? R_ALREADY_EXISTS : errno_result(); + return 0; +} +static unsigned delete_cache(int index) { ready(); char p[1100]; cache_dir(index, p, sizeof p); return remove_tree(p) ? R_TARGET_NOT_FOUND : 0; } +static unsigned mount_cache(const char *name, int index) { + ready(); + char p[1100]; + cache_dir(index, p, sizeof p); + if (!is_dir(p)) return R_TARGET_NOT_FOUND; + return mount_add(name, M_CACHE, p); +} +typedef struct { int32_t index; uint8_t reserved[0x1c]; } CacheStorageInfo; +typedef struct { int list[64]; int count, next; } CacheList; +static unsigned open_cache_list(uint64_t *out) { + ready(); + if (!out) return R_INVALID_ARGUMENT; + CacheList *l = calloc(1, sizeof *l); + need(l != NULL, "cache list allocation"); + char p[1100]; + snprintf(p, sizeof p, "%s/cache", profile); + DIR *d = opendir(p); + struct dirent *e; + while (d && (e = readdir(d)) && l->count < 64) { + char *end; + long v = strtol(e->d_name, &end, 10); + if (*e->d_name && !*end && v >= 0) l->list[l->count++] = (int)v; + } + if (d) closedir(d); + *out = (uint64_t)(uintptr_t)l; + return 0; +} +static unsigned read_cache_list(int *count, CacheStorageInfo *out, uint64_t h, int max) { + CacheList *l = (CacheList *)(uintptr_t)h; + if (!l || !count || max < 0 || (!out && max)) return R_INVALID_ARGUMENT; + int n = 0; + while (n < max && l->next < l->count) { memset(&out[n], 0, sizeof *out); out[n++].index = l->list[l->next++]; } + *count = n; + return 0; +} +static void close_cache_list(uint64_t h) { free((void *)(uintptr_t)h); } +static void unmount(const char *name) { + pthread_mutex_lock(&lock); + for (int i = 0; i < 16; i++) + if (mounts[i].kind && !strcmp(mounts[i].name, name)) { mounts[i].kind = M_FREE; pthread_mutex_unlock(&lock); return; } + pthread_mutex_unlock(&lock); + if (!romfs_owns(name) && strcmp(name, mount_name)) runtime_fail("unmount of unknown fs mount"); +} +static unsigned commit(const char *name) { + pthread_mutex_lock(&lock); + int found = 0, save = 0; + for (int i = 0; i < 16; i++) if (mounts[i].kind && !strcmp(mounts[i].name, name)) { found = 1; save = mounts[i].kind == M_SAVE; } + pthread_mutex_unlock(&lock); + if (!found) return R_PATH_NOT_FOUND; + for (int i = 0; i < NFILES; i++) { + pthread_mutex_lock(&lock); + int fd = files[i].fd; + pthread_mutex_unlock(&lock); + if (fd > 0) fsync(fd); + } + if (save) atomic_store(&save_dirty, 0); + return 0; +} + +static File *file_of(uint64_t h, int *fd, int *mode) { + if (h < FILE_BASE || h >= FILE_BASE + NFILES) return NULL; + pthread_mutex_lock(&lock); + File *f = &files[h - FILE_BASE]; + int v = f->fd; + if (fd) *fd = f->fd; + if (mode) *mode = f->mode; + pthread_mutex_unlock(&lock); + return v > 0 ? f : NULL; +} +static unsigned open_file(uint64_t *out, const char *path, int mode) { + if (romfs_owns(path)) return romfs_open(out, path, mode); + if (path) runtime_fs_log(mode & 2 ? "write" : "read", path); + if (!out || !(mode & (OPEN_READ | OPEN_WRITE)) || (mode & ~7)) return R_INVALID_ARGUMENT; + char p[1100]; + int kind = M_FREE; + unsigned r = host_path_kind(path, p, sizeof p, &kind); + if (r) return r; + struct stat st; + if (stat(p, &st) || !S_ISREG(st.st_mode)) return R_PATH_NOT_FOUND; + int save_writer = kind == M_SAVE && (mode & OPEN_WRITE); + int fd = open(p, (mode & OPEN_WRITE ? (mode & OPEN_READ ? O_RDWR : O_WRONLY) : O_RDONLY) | O_BINARY); + if (fd < 0) return errno_result(); + pthread_mutex_lock(&lock); + for (int i = 0; i < NFILES; i++) + if (files[i].fd <= 0) { + files[i] = (File){fd, mode, save_writer}; + *out = FILE_BASE + (uint64_t)i; + pthread_mutex_unlock(&lock); + if (save_writer) { atomic_fetch_add(&save_writers, 1); save_touched(); } + return 0; + } + pthread_mutex_unlock(&lock); + runtime_fail("host file handle table full"); +} +static unsigned read_file(uint64_t *out, uint64_t h, int64_t offset, void *buffer, uint64_t size) { + if (romfs_handle(h)) return romfs_read(out, h, offset, buffer, size); + int fd, mode; + if (!out || offset < 0 || (!buffer && size) || !file_of(h, &fd, &mode)) return R_INVALID_ARGUMENT; + if (!(mode & OPEN_READ)) return R_OPEN_MODE; + guest_prepare_write(buffer, size); + uint64_t done = 0; + while (done < size) { + ssize_t n = pread(fd, (uint8_t *)buffer + done, (size_t)(size - done), offset + (off_t)done); + if (n < 0) runtime_fail("host save read failure"); + if (!n) break; + done += (uint64_t)n; + } + *out = done; + return 0; +} +static unsigned read_options(uint64_t *out, uint64_t h, int64_t off, void *buf, uint64_t size, const void *options) { + (void)options; + return read_file(out, h, off, buf, size); +} +static unsigned read_exact(uint64_t h, int64_t off, void *buf, uint64_t size) { + uint64_t n = 0; + unsigned r = read_file(&n, h, off, buf, size); + return r ? r : n == size ? 0 : R_INVALID_ARGUMENT; +} +static unsigned write_file(uint64_t h, int64_t offset, const void *buffer, uint64_t size, const int32_t *option) { + int fd, mode; + File *f = offset < 0 || (!buffer && size) ? NULL : file_of(h, &fd, &mode); + if (!f) return R_INVALID_ARGUMENT; + if (!(mode & OPEN_WRITE)) return R_OPEN_MODE; + struct stat st; + if (fstat(fd, &st)) runtime_fail("host save stat"); + if ((uint64_t)offset + size > (uint64_t)st.st_size && !(mode & OPEN_APPEND)) return R_APPEND_REQUIRED; + if (f->save) save_touched(); + uint64_t done = 0; + while (done < size) { + ssize_t n = pwrite(fd, (const uint8_t *)buffer + done, (size_t)(size - done), offset + (off_t)done); + if (n <= 0) { + fprintf(stderr, "[totk] save write at %#llx (%#llx bytes from %p) failed: errno %d\n", + (unsigned long long)(offset + (off_t)done), (unsigned long long)(size - done), + (const void *)((const uint8_t *)buffer + done), errno); + runtime_fail("host save write failure"); + } + done += (uint64_t)n; + } + if (option && (*option & 1)) fsync(fd); + return 0; +} +static unsigned file_size(int64_t *out, uint64_t h) { + if (romfs_handle(h)) return romfs_size(out, h); + int fd; + struct stat st; + if (!out || !file_of(h, &fd, NULL) || fstat(fd, &st)) return R_INVALID_ARGUMENT; + *out = st.st_size; + return 0; +} +static unsigned set_file_size(uint64_t h, int64_t size) { + int fd, mode; + if (size < 0 || !file_of(h, &fd, &mode)) return R_INVALID_ARGUMENT; + if (!(mode & OPEN_WRITE)) return R_OPEN_MODE; + if (files[h - FILE_BASE].save) save_touched(); + return _chsize_s(fd, size) ? errno_result() : 0; +} +static unsigned flush_file(uint64_t h) { int fd; if (!file_of(h, &fd, NULL)) return R_INVALID_ARGUMENT; fsync(fd); return 0; } +static void close_file(uint64_t h) { + if (romfs_handle(h)) { romfs_close(h); return; } + if (!file_of(h, NULL, NULL)) runtime_fail("close invalid file handle"); + pthread_mutex_lock(&lock); + int fd = files[h - FILE_BASE].fd, save = files[h - FILE_BASE].save; + files[h - FILE_BASE].fd = 0; + pthread_mutex_unlock(&lock); + close(fd); + if (save) { save_touched(); atomic_fetch_sub(&save_writers, 1); } +} +static unsigned create_file(const char *path, int64_t size) { + char p[1100]; + unsigned r = save_change_path(path, p, sizeof p); + if (r) return r; + if (size < 0) return R_INVALID_ARGUMENT; + int fd = open(p, O_WRONLY | O_CREAT | O_EXCL | O_BINARY, 0600); + if (fd < 0) return errno_result(); + int bad = _chsize_s(fd, size); + close(fd); + return bad ? errno_result() : 0; +} +static unsigned delete_file(const char *path) { + char p[1100]; + unsigned r = save_change_path(path, p, sizeof p); + if (r) return r; + return unlink(p) ? errno_result() : 0; +} +static unsigned rename_file(const char *from, const char *to) { + char a[1100], b[1100]; + unsigned r = save_change_path(from, a, sizeof a); + if (!r) r = host_path(to, b, sizeof b); + if (r) return r; + struct stat st; + if (!stat(b, &st)) return R_ALREADY_EXISTS; + return rename(a, b) ? errno_result() : 0; +} +static unsigned entry_type(int *out, const char *path) { + if (romfs_owns(path)) return romfs_entry_type(out, path); + char p[1100]; + unsigned r = host_path(path, p, sizeof p); + if (r) return r; + struct stat st; + if (!out) return R_INVALID_ARGUMENT; + if (stat(p, &st)) return R_PATH_NOT_FOUND; + *out = S_ISDIR(st.st_mode) ? 0 : 1; + return 0; +} + +typedef struct { char name[769]; uint8_t pad0[3]; int8_t type; uint8_t pad1[3]; int64_t size; } DirectoryEntry; +_Static_assert(sizeof(DirectoryEntry) == 0x310, "nn::fs::DirectoryEntry"); +static unsigned create_directory(const char *path) { + char p[1100]; + unsigned r = save_change_path(path, p, sizeof p); + if (r) return r; + return make_dir_p(p) ? errno_result() : 0; +} +static unsigned delete_recursive(const char *path) { + char p[1100]; + unsigned r = save_change_path(path, p, sizeof p); + if (r) return r; + struct stat st; + if (stat(p, &st)) return R_PATH_NOT_FOUND; + return remove_tree(p) ? R_INVALID_ARGUMENT : 0; +} +static unsigned open_directory(uint64_t *out, const char *path, int mode) { + if (romfs_owns(path)) { fprintf(stderr, "[totk] OpenDirectory on RomFS %s\n", path); runtime_fail("RomFS directory listing not implemented"); } + char p[1100]; + unsigned r = host_path(path, p, sizeof p); + if (r) return r; + if (!out || mode < 1 || mode > 3) return R_INVALID_ARGUMENT; + DIR *d = opendir(p); + if (!d) return errno_result(); + pthread_mutex_lock(&lock); + for (int i = 0; i < NDIRS; i++) + if (!dirs[i].dir) { + dirs[i].dir = d; dirs[i].mode = mode; strcpy(dirs[i].path, p); + *out = DIR_BASE + (uint64_t)i; + pthread_mutex_unlock(&lock); + return 0; + } + pthread_mutex_unlock(&lock); + runtime_fail("host directory handle table full"); +} +static unsigned read_directory(int64_t *count, DirectoryEntry *out, uint64_t h, int64_t max) { + if (h < DIR_BASE || h >= DIR_BASE + NDIRS || !count || max < 0 || (!out && max)) return R_INVALID_ARGUMENT; + Dir *d = &dirs[h - DIR_BASE]; + if (!d->dir) return R_INVALID_ARGUMENT; + int64_t n = 0; + struct dirent *e; + while (n < max && (e = readdir(d->dir))) { + if (!strcmp(e->d_name, ".") || !strcmp(e->d_name, "..") || strlen(e->d_name) > 768) continue; + char child[2048]; + snprintf(child, sizeof child, "%s/%s", d->path, e->d_name); + struct stat st; + if (stat(child, &st)) continue; + int dir = S_ISDIR(st.st_mode); + if (!(d->mode & (dir ? 1 : 2))) continue; + memset(&out[n], 0, sizeof *out); + strcpy(out[n].name, e->d_name); + out[n].type = dir ? 0 : 1; + out[n].size = dir ? 0 : st.st_size; + n++; + } + *count = n; + return 0; +} +static void close_directory(uint64_t h) { + if (h < DIR_BASE || h >= DIR_BASE + NDIRS || !dirs[h - DIR_BASE].dir) runtime_fail("close invalid directory handle"); + pthread_mutex_lock(&lock); + closedir(dirs[h - DIR_BASE].dir); + dirs[h - DIR_BASE].dir = NULL; + pthread_mutex_unlock(&lock); +} +static void set_priority(int priority) { (void)priority; } /* fs priority orders I/O inside the system fs process */ +static void fs_initialize(void) { runtime_fs_init(); } + +HLE_NATIVE("_ZN2nn2fs24InitializeForApplicationEv", fs_initialize, 'v') +HLE_NATIVE("_ZN2nn2fs8MountRomEPKcPvm", mount_rom, 'w') +HLE_NATIVE("_ZN2nn2fs22QueryMountRomCacheSizeEPm", rom_cache_size, 'w') +HLE_NATIVE("_ZN2nn2fs8OpenFileEPNS0_10FileHandleEPKci", open_file, 'w') +HLE_NATIVE("_ZN2nn2fs11GetFileSizeEPlNS0_10FileHandleE", file_size, 'w') +HLE_NATIVE("_ZN2nn2fs9CloseFileENS0_10FileHandleE", close_file, 'v') +HLE_NATIVE("_ZN2nn2fs12GetEntryTypeEPNS0_18DirectoryEntryTypeEPKc", entry_type, 'w') +HLE_NATIVE("_ZN2nn2fs8ReadFileEPmNS0_10FileHandleElPvm", read_file, 'w') +HLE_NATIVE("_ZN2nn2fs8ReadFileEPmNS0_10FileHandleElPvmRKNS0_10ReadOptionE", read_options, 'w') +HLE_NATIVE("_ZN2nn2fs8ReadFileENS0_10FileHandleElPvm", read_exact, 'w') +HLE_NATIVE("_ZN2nn2fs9WriteFileENS0_10FileHandleElPKvmRKNS0_11WriteOptionE", write_file, 'w') +HLE_NATIVE("_ZN2nn2fs11SetFileSizeENS0_10FileHandleEl", set_file_size, 'w') +HLE_NATIVE("_ZN2nn2fs9FlushFileENS0_10FileHandleE", flush_file, 'w') +HLE_NATIVE("_ZN2nn2fs10CreateFileEPKcl", create_file, 'w') +HLE_NATIVE("_ZN2nn2fs10DeleteFileEPKc", delete_file, 'w') +HLE_NATIVE("_ZN2nn2fs10RenameFileEPKcS2_", rename_file, 'w') +HLE_NATIVE("_ZN2nn2fs15CreateDirectoryEPKc", create_directory, 'w') +HLE_NATIVE("_ZN2nn2fs26DeleteDirectoryRecursivelyEPKc", delete_recursive, 'w') +HLE_NATIVE("_ZN2nn2fs13OpenDirectoryEPNS0_15DirectoryHandleEPKci", open_directory, 'w') +HLE_NATIVE("_ZN2nn2fs13ReadDirectoryEPlPNS0_14DirectoryEntryENS0_15DirectoryHandleEl", read_directory, 'w') +HLE_NATIVE("_ZN2nn2fs14CloseDirectoryENS0_15DirectoryHandleE", close_directory, 'v') +HLE_NATIVE("_ZN2nn2fs13MountSaveDataEPKcRKNS_7account3UidE", mount_save, 'w') +HLE_NATIVE("_ZN2nn2fs21MountSaveDataReadOnlyEPKcNS_13ApplicationIdERKNS_7account3UidE", mount_save_ro, 'w') +HLE_NATIVE("_ZN2nn2fs18IsSaveDataExistingENS_13ApplicationIdERKNS_7account3UidE", save_exists, 'b') +HLE_NATIVE("_ZN2nn2fs21MountTemporaryStorageEPKc", mount_temp, 'w') +HLE_NATIVE("_ZN2nn2fs7UnmountEPKc", unmount, 'v') +HLE_NATIVE("_ZN2nn2fs6CommitEPKc", commit, 'w') +HLE_NATIVE("_ZN2nn2fs18CreateCacheStorageEill", create_cache, 'w') +HLE_NATIVE("_ZN2nn2fs18DeleteCacheStorageEi", delete_cache, 'w') +HLE_NATIVE("_ZN2nn2fs17MountCacheStorageEPKci", mount_cache, 'w') +HLE_NATIVE("_ZN2nn2fs20OpenCacheStorageListEPNS0_22CacheStorageListHandleE", open_cache_list, 'w') +HLE_NATIVE("_ZN2nn2fs20ReadCacheStorageListEPiPNS0_16CacheStorageInfoENS0_22CacheStorageListHandleEi", read_cache_list, 'w') +HLE_NATIVE("_ZN2nn2fs21CloseCacheStorageListENS0_22CacheStorageListHandleE", close_cache_list, 'v') +HLE_NATIVE("_ZN2nn2fs26SetPriorityOnCurrentThreadENS0_8PriorityE", set_priority, 'v') diff --git a/src/host/guest.c b/src/host/guest.c new file mode 100644 index 0000000..b327ef1 --- /dev/null +++ b/src/host/guest.c @@ -0,0 +1,97 @@ +// Glue between host threads and the recompiled game: per-thread Cpu, Horizon TLS blocks, host -> guest calls, SVCs. +#include +#include +#include +#include + +#include "host.h" + +static __thread Cpu *t_cpu; + +Cpu *guest_cpu(void) { return t_cpu; } +void guest_cpu_set(Cpu *c) { t_cpu = c; } + +// Horizon thread-local region (tpidrro_el0): 0x200 bytes; +0x000 IPC message buffer, +0x1f8 the SDK's thread pointer. +// MacNativeGames (loader.c runtime_thread_init) points +0x1f8 at a zeroed 0x400-byte placeholder until the SDK sets it. +uint64_t guest_tls_alloc(void) { + static _Atomic uint64_t next = TLS_BASE; + uint64_t a = atomic_fetch_add(&next, 0x200); + if (a + 0x200 > TLS_BASE + TLS_SIZE) runtime_fail("out of guest TLS blocks"); + plat_commit(a, 0x200); + memset((void *)(uintptr_t)a, 0, 0x200); + uint64_t placeholder = (uint64_t)(uintptr_t)hle_alloc(0x400); + memcpy((void *)(uintptr_t)(a + 0x1f8), &placeholder, 8); + return a; +} + +Cpu *guest_cpu_new(uint64_t stack_top, uint64_t tls) { + Cpu *c = _aligned_malloc(sizeof(Cpu), 64); + if (!c) runtime_fail("guest CPU allocation"); + memset(c, 0, sizeof *c); + c->sp = stack_top & ~0xFull; + c->tpidrro = tls; + return c; +} + +uint64_t guest_call_args(uint64_t fn, int n, const uint64_t *args) { + Cpu *c = t_cpu; + if (!c) runtime_fail("guest call from a host thread"); + if (n > 8) runtime_fail("guest call with more than 8 arguments"); + Cpu *saved = _aligned_malloc(sizeof(Cpu), 64); + if (!saved) runtime_fail("guest call state allocation"); + memcpy(saved, c, sizeof *c); + for (int i = 0; i < n; i++) c->x[i] = args[i]; + guest_call(c, fn); + uint64_t r = c->x[0]; + // Only the caller's registers come back (x0-x30, sp, pc, flags, vector registers). Thread state the guest changed + // during the call stays: TPIDR_EL0 (the SDK sets it with msr while initializing threads), FPCR/FPSR, the monitor. + memcpy(c->x, saved->x, sizeof c->x); + c->sp = saved->sp; + c->pc = saved->pc; + c->nzcv = saved->nzcv; + memcpy(c->v, saved->v, sizeof c->v); + _aligned_free(saved); + return r; +} + +// Every `svc #imm` of the recompiled code lands here (cpu.h). +void rt_svc(Cpu *c, uint32_t imm, uint64_t pc) { runtime_svc(imm, c->x, pc); } + +_Noreturn void runtime_fail(const char *message) { + fprintf(stderr, "[totk] FATAL: %s\n", message); + rt_fatal_trace(); +} + +uint64_t runtime_now_ns(void) { return plat_now_ns(); } + +void *runtime_symbol(const char *name) { return (void *)(uintptr_t)loader_find_symbol(name); } + +// Guest call chain from a frame pointer (x29) and link register (x30): AArch64 frame records are {fp, lr} pairs. +void runtime_trace_guest(uint64_t fp, uint64_t lr) { + for (int depth = 0; depth < 32; depth++) { + uint64_t off = 0; + const char *s = lr ? loader_symbol_at(lr - 4, &off) : NULL; + fprintf(stderr, " #%-2d %#llx %s+%#llx\n", depth, (unsigned long long)lr, s ? s : "?", (unsigned long long)(off + 4)); + if (!fp || fp < GUEST_LO || fp >= GUEST_HI || (fp & 7)) break; + uint64_t next = ((uint64_t *)(uintptr_t)fp)[0]; + lr = ((uint64_t *)(uintptr_t)fp)[1]; + if (next <= fp) break; + fp = next; + } +} + +// Crash report: the faulting thread's guest registers and guest call chain (src/runtime/dispatch.c rt_fault_report). +static void guest_fault_report(void) { + Cpu *c = t_cpu; + if (!c) { fprintf(stderr, " (no guest thread)\n"); return; } + fprintf(stderr, " guest sp=%#llx tpidrro=%#llx thread handle %#x\n", (unsigned long long)c->sp, + (unsigned long long)c->tpidrro, runtime_thread_current_handle()); + for (int i = 0; i < 31; i += 4) + fprintf(stderr, " x%-2d %016llx x%-2d %016llx x%-2d %016llx x%-2d %016llx\n", i, (unsigned long long)c->x[i], + i + 1, (unsigned long long)c->x[i + 1], i + 2, (unsigned long long)c->x[i + 2], i + 3, + (unsigned long long)c->x[i + 3]); + fprintf(stderr, " guest call chain (x30 / frame records):\n"); + runtime_trace_guest(c->x[29], c->x[30]); +} +extern void (*rt_fault_extra)(void); +__attribute__((constructor)) static void install_fault_report(void) { rt_fault_extra = guest_fault_report; } diff --git a/src/host/hid_feedback.c b/src/host/hid_feedback.c new file mode 100644 index 0000000..2b7f421 --- /dev/null +++ b/src/host/hid_feedback.c @@ -0,0 +1,153 @@ +/* From MacNativeGames games/totk/runtime/hid_feedback.c @b75be8c (MIT). [port] marks the Windows changes: the + * functions are bound with HLE_NATIVE (always: the TOTK_RUMBLE=0 / TOTK_GYRO=0 switches that hand the handle queries + * back to the SDK are not ported), and the controller side (rumble motors, motion sensors) is empty until the SDL3 + * game controller layer of milestone 3.2. */ +/* Controller rumble and motion on the game's side (2026-10-17): the hid functions TOTK uses for HD rumble and for the + six-axis sensor (motion aiming). + Opt-in at start, because the game asks for its vibration and six-axis handles only when a controller's style is set + up (main+0x29b45cc, from the per-frame controller update main+0x29b442c), so they cannot appear later: + TOTK_RUMBLE=1 the controller has its two vibration devices (left, right); what the game plays goes to the Mac + controller's haptics (runtime/gamepad_feedback.m); + TOTK_GYRO=1 the controller has its six-axis sensor; the game gets the Mac controller's motion. + Off (the default) the game sees neither, exactly as before (no handles, nothing below is called). + While the game runs, "flag rumble 0|1" / "flag gyro 0|1" mute and unmute them (muted motion = a controller lying still). + The handles are the SDK's own: nn::hid::utility computes them without the hid service (byte 0 = style index, 3 full-key, + 4 handheld; byte 1 = controller id; byte 2 = device index: vibration 0 left / 1 right, six-axis 2). Everything under + them (nn::hid::detail, which the SDK calls through its PLT, including the batches VibrationTarget sends through + detail::EnqueueVibrationValue -> detail::SendVibrationValues) is answered here instead of by the hid service. + Layouts: VibrationValue {amplitude low, frequency low, amplitude high, frequency high} (0..1 and Hz; neutral 0/160/0/320, + sdk+0x4a2df8); VibrationDeviceInfo {type, position} (8 bytes, sdk+0x49da78); SixAxisSensorState 0x60 bytes. */ +#include "host.h" // [port] runtime.h +#include "../runtime/hle.h" +#include +#include +#include +#include +#include +#include +#include + +/* [port] The controller side (MacNativeGames gamepad_feedback.m): SDL3 in window_sdl.c, motion_fusion.c; host.h. */ + +typedef struct { float amp_low, freq_low, amp_high, freq_high; } VibrationValue; +typedef struct { uint32_t type, position; } VibrationDeviceInfo; /* type 1 = linear resonant actuator; position 1 left, 2 right */ +typedef struct { int64_t delta_ns, sampling; float acc[3], gyro[3], angle[3], dir[9]; uint32_t attributes, reserved; } SixAxisState; +_Static_assert(sizeof(VibrationValue) == 16 && sizeof(VibrationDeviceInfo) == 8 && sizeof(SixAxisState) == 0x60, "hid ABI"); +enum { STYLE_FULL_KEY = 3, STYLE_HANDHELD = 4, SIX_AXIS_CONNECTED = 1 }; + +/* On unless =0 (2026-10-17: default on, the user's request). An agent's test session (TOTK_SLOT_OWNER claude / codex) leaves + them off unless set, so a test does not rumble the user's controller and tests stay as measured before. */ +static bool env_on(const char *name) { + const char *e = getenv(name); if (e) return strcmp(e, "0") != 0; + const char *o = getenv("TOTK_SLOT_OWNER"); + return !(o && (!strcmp(o, "claude") || !strcmp(o, "codex"))); +} +int runtime_rumble_enabled(void) { return env_on("TOTK_RUMBLE"); } +int runtime_gyro_enabled(void) { return env_on("TOTK_GYRO"); } +static _Atomic int rumble_muted, gyro_muted; +static pthread_mutex_t lastLock = PTHREAD_MUTEX_INITIALIZER; +static VibrationValue last[2] = {{0, 160, 0, 320}, {0, 160, 0, 320}}; /* lastLock */ +static _Atomic uint64_t values_in, reads; + +static void note_once(_Atomic int *flag, const char *text, uint32_t handle) { + if (!atomic_exchange(flag, 1)) fprintf(stderr, "[hid] %s (handle %#x)\n", text, handle); +} +/* The device a handle names, or -1 (a style this runtime never reports, logged once). */ +static int vibration_side(const uint32_t *h) { + if (!h) runtime_fail("vibration device handle pointer"); + unsigned style = *h & 0xff, index = (*h >> 16) & 0xff; + if ((style == STYLE_FULL_KEY || style == STYLE_HANDHELD) && index < 2) return (int)index; + static _Atomic int warned; note_once(&warned, "vibration call with a handle this controller does not have; ignored", *h); + return -1; +} +static void play(int side, const VibrationValue *v) { + if (!v) runtime_fail("vibration value pointer"); + if (side < 0) return; + VibrationValue c = *v; + if (!isfinite(c.amp_low) || !isfinite(c.amp_high) || !isfinite(c.freq_low) || !isfinite(c.freq_high)) c = (VibrationValue){0, 160, 0, 320}; + pthread_mutex_lock(&lastLock); last[side] = c; pthread_mutex_unlock(&lastLock); atomic_fetch_add(&values_in, 1); + static _Atomic int first; if (!atomic_exchange(&first, 1)) fprintf(stderr, "[hid] first vibration value from the game: %s %.2f@%.0f Hz %.2f@%.0f Hz\n", + side ? "right" : "left", c.amp_low, c.freq_low, c.amp_high, c.freq_high); + if (atomic_load(&rumble_muted)) c.amp_low = c.amp_high = 0; + runtime_rumble_set((unsigned)side, c.amp_low, c.freq_low, c.amp_high, c.freq_high); +} +static uint32_t vib_init(const uint32_t *h) { + int side = vibration_side(h); static _Atomic int once; + if (side >= 0) note_once(&once, "the game initialized its vibration devices", *h); + return 0; +} +static uint32_t vib_send(const uint32_t *h, const VibrationValue *v) { play(vibration_side(h), v); return 0; } +static uint32_t vib_send_many(const uint32_t *h, const VibrationValue *v, int n) { + if (n < 0 || (n && (!h || !v))) runtime_fail("vibration batch"); + for (int i = 0; i < n; i++) play(vibration_side(&h[i]), &v[i]); + return 0; +} +static uint32_t vib_actual(VibrationValue *out, const uint32_t *h) { + if (!out) runtime_fail("vibration value output pointer"); + int side = vibration_side(h); *out = (VibrationValue){0, 160, 0, 320}; + if (side >= 0) { pthread_mutex_lock(&lastLock); *out = last[side]; pthread_mutex_unlock(&lastLock); } + return 0; +} +static uint32_t vib_info(VibrationDeviceInfo *out, const uint32_t *h) { + if (!out) runtime_fail("vibration device info pointer"); + int side = vibration_side(h); *out = (VibrationDeviceInfo){side >= 0 ? 1u : 0u, side >= 0 ? (uint32_t)side + 1 : 0}; return 0; +} +static uint32_t vib_mounted(bool *out, const uint32_t *h) { if (!out) runtime_fail("vibration mounted pointer"); *out = vibration_side(h) >= 0; return 0; } +static uint32_t vib_permitted(bool *out) { if (!out) runtime_fail("vibration permitted pointer"); *out = true; return 0; } + +static void six_axis_check(const uint32_t *h) { + if (!h) runtime_fail("six-axis handle pointer"); + unsigned style = *h & 0xff; + if (style != STYLE_FULL_KEY && style != STYLE_HANDHELD) { static _Atomic int warned; note_once(&warned, "six-axis call with a handle this controller does not have", *h); } +} +static uint32_t six_start(const uint32_t *h) { + six_axis_check(h); static _Atomic int once; note_once(&once, "the game started the six-axis sensor", *h); + runtime_motion_start(); return 0; +} +static uint32_t six_stop(const uint32_t *h) { six_axis_check(h); return 0; } +/* Newest first, like the hid service's ring. No motion (no controller with sensors, or muted): one state of a controller + lying still, not marked connected. */ +static uint32_t six_states(int *out, SixAxisState *s, int max, const uint32_t *h) { + if (!out || (max > 0 && !s)) runtime_fail("six-axis state output"); + six_axis_check(h); atomic_fetch_add(&reads, 1); + if (max <= 0) { *out = 0; return 0; } + RuntimeMotionState m[RUNTIME_MOTION_MAX]; + int n = atomic_load(&gyro_muted) ? 0 : runtime_motion_read(m, max < RUNTIME_MOTION_MAX ? max : RUNTIME_MOTION_MAX); + if (n <= 0) { + s[0] = (SixAxisState){.acc = {0, 0, -1}, .dir = {1, 0, 0, 0, 1, 0, 0, 0, 1}}; + *out = 1; return 0; + } + for (int i = 0; i < n; i++) { + SixAxisState *d = &s[i]; *d = (SixAxisState){.delta_ns = (int64_t)m[i].delta_ns, .sampling = (int64_t)m[i].sampling, .attributes = SIX_AXIS_CONNECTED}; + memcpy(d->acc, m[i].acc, sizeof d->acc); memcpy(d->gyro, m[i].gyro, sizeof d->gyro); + memcpy(d->angle, m[i].angle, sizeof d->angle); memcpy(d->dir, m[i].dir, sizeof d->dir); + } + *out = n; return 0; +} +static uint32_t six_state(SixAxisState *s, const uint32_t *h) { int n; return six_states(&n, s, 1, h); } + +/* "flag rumble 0|1", "flag gyro 0|1": 1 if the name was one of these. */ +int runtime_hid_feedback_flag(const char *name, int v) { + _Atomic int *muted = !strcmp(name, "rumble") ? &rumble_muted : !strcmp(name, "gyro") ? &gyro_muted : NULL; + if (!muted) return 0; + if (v != 0 && v != 1) runtime_fail("flag rumble / gyro takes 0 or 1"); + int on = muted == &rumble_muted ? runtime_rumble_enabled() : runtime_gyro_enabled(); + atomic_store(muted, !v); + if (!v && muted == &rumble_muted) { runtime_rumble_set(0, 0, 160, 0, 320); runtime_rumble_set(1, 0, 160, 0, 320); } + fprintf(stderr, "[hid] %s %s%s\n", name, v ? "on" : "muted", on ? "" : " (no effect: the game has no such device in this run; start it with TOTK_RUMBLE=1 / TOTK_GYRO=1)"); + return 1; +} +void runtime_hid_feedback_stats(uint64_t *values, uint64_t *motion_reads) { *values = atomic_load(&values_in); *motion_reads = atomic_load(&reads); } + +// [port] MacNativeGames' runtime_hid_feedback_override table as HLE bindings (always on) +HLE_NATIVE("_ZN2nn3hid6detail25InitializeVibrationDeviceERKNS0_21VibrationDeviceHandleE", vib_init, 'w') +HLE_NATIVE("_ZN2nn3hid6detail18SendVibrationValueERKNS0_21VibrationDeviceHandleERKNS0_14VibrationValueE", vib_send, 'w') +HLE_NATIVE("_ZN2nn3hid6detail19SendVibrationValuesEPKNS0_21VibrationDeviceHandleEPKNS0_14VibrationValueEi", vib_send_many, 'w') +HLE_NATIVE("_ZN2nn3hid6detail23GetActualVibrationValueEPNS0_14VibrationValueERKNS0_21VibrationDeviceHandleE", vib_actual, 'w') +HLE_NATIVE("_ZN2nn3hid6detail22GetVibrationDeviceInfoEPNS0_19VibrationDeviceInfoERKNS0_21VibrationDeviceHandleE", vib_info, 'w') +HLE_NATIVE("_ZN2nn3hid6detail24IsVibrationDeviceMountedEPbRKNS0_21VibrationDeviceHandleE", vib_mounted, 'w') +HLE_NATIVE("_ZN2nn3hid6detail20IsVibrationPermittedEPb", vib_permitted, 'w') +HLE_NATIVE("_ZN2nn3hid6detail18StartSixAxisSensorERKNS0_19SixAxisSensorHandleE", six_start, 'w') +HLE_NATIVE("_ZN2nn3hid6detail17StopSixAxisSensorERKNS0_19SixAxisSensorHandleE", six_stop, 'w') +HLE_NATIVE("_ZN2nn3hid6detail22GetSixAxisSensorStatesEPiPNS0_18SixAxisSensorStateEiRKNS0_19SixAxisSensorHandleE", six_states, 'w') +HLE_NATIVE("_ZN2nn3hid6detail21GetSixAxisSensorStateEPNS0_18SixAxisSensorStateERKNS0_19SixAxisSensorHandleE", six_state, 'w') diff --git a/src/host/host.h b/src/host/host.h new file mode 100644 index 0000000..d76fa82 --- /dev/null +++ b/src/host/host.h @@ -0,0 +1,79 @@ +// Horizon host for the recompiled game: what MacNativeGames' runtime/runtime.h declares, for the parts ported here. +// The game's code is recompiled C working on a Cpu (src/runtime/cpu.h), so the host reaches guest registers through +// Cpu and calls guest functions through guest_call (gcall below), never as native function pointers. +#pragma once +#include +#include +#include + +#include "../runtime/rt.h" + +_Noreturn void runtime_fail(const char *message); +uint64_t runtime_now_ns(void); +void *runtime_symbol(const char *name); // guest address of an exported symbol (as a pointer), NULL if none + +// ---- the calling guest thread (guest.c) +Cpu *guest_cpu(void); // this thread's guest CPU state, NULL on a pure host thread +void guest_cpu_set(Cpu *c); +Cpu *guest_cpu_new(uint64_t stack_top, uint64_t tls); +uint64_t guest_tls_alloc(void); // a thread's TLS block (tpidrro_el0): IPC message buffer at +0, 0x200 bytes +// Call a guest function with integer/pointer arguments from host code running on a guest thread. The caller's guest +// registers are saved and restored around the call (host code may run inside an SVC or a host replacement). +uint64_t guest_call_args(uint64_t fn, int n, const uint64_t *args); +// Before the OS itself writes guest memory (file reads into a game buffer): guest pages are committed on the first +// CPU access through a fault, which the kernel's own writes do not take, so the range is committed first. +static inline void guest_prepare_write(void *addr, uint64_t size) { + uint64_t a = (uint64_t)(uintptr_t)addr; + if (size && a >= GUEST_LO && a < GUEST_HI) plat_commit(a, size); +} +#define GCALL(fn, ...) guest_call_args((uint64_t)(uintptr_t)(fn), \ + (int)(sizeof((uint64_t[]){0, __VA_ARGS__}) / sizeof(uint64_t)) - 1, (uint64_t[]){0, __VA_ARGS__} + 1) +#define GCALL0(fn) guest_call_args((uint64_t)(uintptr_t)(fn), 0, NULL) + +// ---- svc.c +uint64_t runtime_svc(unsigned num, uint64_t *regs, uint64_t pc); +void runtime_memory_init(void); +uint64_t runtime_image_bytes(void); +uint64_t runtime_main_stack(uint64_t size); // the main thread's guest stack (top), from the stack region + +// ---- threads.c +uint32_t svc_thread_main_init(void); +int svc_thread(int num, uint64_t *regs); +void runtime_trace_guest(uint64_t fp, uint64_t lr); +void host_services_register(void); // services_*.c: IPC services the SDK opens +void hle_bind_system_event(void *event, int autoclear); +void hle_signal_system_event(void *event); +uint32_t runtime_thread_current_handle(void); + +// ---- loader (src/runtime/loader.c): module memory for QueryMemory / SetMemoryPermission +int runtime_module_query(uint64_t address, uint64_t *base, uint64_t *size, unsigned *state, unsigned *permission, + unsigned *attribute); + +// ---- input (overrides_input.c), fed by the window (window_sdl.c) +typedef struct { uint64_t buttons; int32_t lx, ly, rx, ry; int connected; } RuntimeGamepadState; +void runtime_gamepad_set(const RuntimeGamepadState *s); +void runtime_input_key(unsigned code, int down); // Mac virtual key codes (MacNativeGames' key layout) +void runtime_input_clear(void); + +// ---- window (window_sdl.c): SDL3 on the main thread +int window_init(void); // 1 = window up (presentation may still be missing: logged) +_Noreturn void window_loop(void); + +// ---- rumble and motion (hid_feedback.c <- window_sdl.c, motion_fusion.c; MacNativeGames runtime.h) +#define RUNTIME_MOTION_MAX 32 +typedef struct { uint64_t sampling, delta_ns; float acc[3], gyro[3], angle[3], dir[9]; } RuntimeMotionState; // G, rev/s, rev, 3 axes +void runtime_rumble_set(unsigned side, float amp_low, float freq_low, float amp_high, float freq_high); // side 0 left, 1 right +int runtime_motion_read(RuntimeMotionState *out, int max); // newest first; 0 = no motion now +void runtime_motion_start(void); // the game (re)started its sensor: angle and direction start over +// One sample in the Switch six-axis frame (x right, y toward the top edge, z out of the face): acceleration in G with +// gravity (face up reads z = -1), rotation in rad/s. +void runtime_motion_feed(const double acc_g[3], const double gyro_rad_s[3], uint64_t now_ns); +int runtime_motion_axes(const char *spec); // TOTK_GYRO_AXES ("-x,y,-z"): 1 set, 0 unset, -1 malformed +void runtime_motion_deadzone_default(double deg_s); +void runtime_motion_live(int on); // a controller with motion attached (1) / detached (0) +void runtime_hid_feedback_stats(uint64_t *values, uint64_t *motion_reads); + +// ---- profiler.c: sampling CPU profiler for development (TOTK_PROF=:[:]) +void runtime_profiler_start(void); +uint64_t runtime_symbolize(uint64_t rip, char *out, size_t n); // host address -> function (f_ + guest symbol) +void runtime_watch_tick(uint64_t present); // TOTK_WATCH: arms / reports the memory write watch (watch.c) diff --git a/src/host/ipc.c b/src/host/ipc.c new file mode 100644 index 0000000..eba58bb --- /dev/null +++ b/src/host/ipc.c @@ -0,0 +1,266 @@ +/* Ported from MacNativeGames games/totk/runtime/ipc.c @b75be8c (MIT). Change: the message buffer is the calling + guest thread's TLS block (Cpu tpidrro_el0) instead of a Darwin TSD slot; CMIF domains (TOTK 1.4.3's SDK makes its lm session a + domain), framed as the public libnx cmif.h describes. */ +/* In-process Horizon IPC server. Message framing follows the public HIPC/CMIF/TIPC description + (libnx sf/hipc.h, cmif.h, tipc.h; ISC licence). The SDK client code runs unchanged: requests are + read from the calling thread's TLS message buffer and answered in place. Unknown services and + commands stop the runtime with their name and id instead of returning a fake success. */ +#include "host.h" +#include "ipc.h" +#include +#include +#include +#include + +#define CMIF_IN_MAGIC 0x49434653u /* "SFCI" */ +#define CMIF_OUT_MAGIC 0x4f434653u /* "SFCO" */ +#define MSG_BYTES 0x100 +#define POINTER_BUFFER_SIZE 0x8000 + +enum { T_CLOSE = 2, T_REQUEST = 4, T_CONTROL = 5, T_REQUEST_CTX = 6, T_CONTROL_CTX = 7, T_TIPC_BASE = 16, T_TIPC_CLOSE = 15 }; + +typedef struct { const char *name; IpcHandler handler; IpcOpen open; void (*close)(void *); } Service; +static Service services[32]; +static unsigned nservices; +static __thread uint64_t trace_fp, trace_lr; /* guest frame of the current request, for failure reports */ +static pthread_mutex_t registry = PTHREAD_MUTEX_INITIALIZER; + +void runtime_ipc_register(const char *name, IpcHandler handler, IpcOpen open, void (*close)(void *)) { + pthread_mutex_lock(®istry); + if (nservices == sizeof services / sizeof *services) runtime_fail("IPC service registry full"); + services[nservices++] = (Service){name, handler, open, close}; + pthread_mutex_unlock(®istry); +} + +static IpcSession *session_new(const char *name, IpcHandler handler, void *state, void (*close)(void *)) { + IpcSession *s = calloc(1, sizeof *s); + if (!s) runtime_fail("IPC session allocation"); + *s = (IpcSession){name, handler, state, close}; + return s; +} +uint32_t runtime_ipc_new_session(const char *name, IpcHandler handler, void *state, void (*close)(void *)) { + return runtime_handle_new(1, session_new(name, handler, state, close)); +} +static void object_free(IpcSession *s) { + if (s->close) s->close(s->state); + free(s); +} +void runtime_ipc_session_closed(void *p) { + IpcSession *s = p; + if (s->domain == s) { /* a domain: its objects go with it (object 1 shares the session's state) */ + for (unsigned i = 2; i < IPC_MAX_OBJECTS; i++) if (s->objects[i]) object_free(s->objects[i]); + free(s->objects[1]); + } + object_free(s); +} +void ipc_out_object(IpcSession *s, IpcResponse *rs, const char *name, IpcHandler handler, void *state, void (*close)(void *)) { + if (!s->domain) { rs->move[rs->nmove++] = runtime_ipc_new_session(name, handler, state, close); return; } + IpcSession *d = s->domain; + unsigned id = 2; + while (id < IPC_MAX_OBJECTS && d->objects[id]) id++; + if (id == IPC_MAX_OBJECTS) runtime_fail("IPC domain full"); + IpcSession *o = session_new(name, handler, state, close); + o->domain = d; + d->objects[id] = o; + if (rs->nout_objects == 8) runtime_fail("IPC too many out objects"); + rs->out_objects[rs->nout_objects++] = id; +} +void runtime_ipc_tmem_closed(void *p) { free(p); } + +void *ipc_guest(uint64_t addr, uint64_t size) { + if (!addr && size) runtime_fail("IPC null buffer"); + if (addr + size < addr) runtime_fail("IPC buffer overflow"); + return (void *)(uintptr_t)addr; +} +const void *ipc_arg(const IpcRequest *rq, uint32_t off, uint32_t n) { + if (off + n > rq->data_size) { fprintf(stderr, "[totk-ipc] cmd %u argument %u+%u beyond %u bytes\n", rq->cmd, off, n, rq->data_size); runtime_fail("IPC argument out of range"); } + return rq->data + off; +} +void ipc_out(IpcResponse *rs, const void *p, uint32_t n) { + if (rs->data_size + n > sizeof rs->data) runtime_fail("IPC response too large"); + memcpy(rs->data + rs->data_size, p, n); rs->data_size += n; +} +_Noreturn void ipc_unimplemented(const IpcSession *s, const IpcRequest *rq) { + fprintf(stderr, "[totk-ipc] %s: command %u not implemented (type %u, data %u bytes, x%u a%u b%u w%u c%u)\n", + s->name, rq->cmd, rq->type, rq->data_size, rq->nx, rq->na, rq->nb, rq->nw, rq->nc); + runtime_trace_guest(trace_fp, trace_lr); + runtime_fail("IPC command not implemented"); +} + +/* ---------- HIPC framing ---------- */ +static uint64_t static_addr(const uint32_t *d) { return d[1] | ((uint64_t)((d[0] >> 12) & 0xf) << 32) | ((uint64_t)((d[0] >> 6) & 0x3f) << 36); } +static uint64_t buffer_addr(const uint32_t *d) { return d[1] | ((uint64_t)((d[2] >> 28) & 0xf) << 32) | ((uint64_t)((d[2] >> 2) & 0x3fffff) << 36); } +static uint64_t buffer_size(const uint32_t *d) { return d[0] | ((uint64_t)((d[2] >> 24) & 0xf) << 32); } + +static void parse(const uint32_t *m, IpcRequest *rq, const uint32_t **data_words, unsigned *ndata) { + memset(rq, 0, sizeof *rq); + rq->type = m[0] & 0xffff; + rq->nx = (m[0] >> 16) & 0xf; rq->na = (m[0] >> 20) & 0xf; rq->nb = (m[0] >> 24) & 0xf; rq->nw = (m[0] >> 28) & 0xf; + *ndata = m[1] & 0x3ff; + unsigned recv_mode = (m[1] >> 10) & 0xf, special = m[1] >> 31; + if (rq->nx > IPC_MAX_BUF || rq->na > IPC_MAX_BUF || rq->nb > IPC_MAX_BUF || rq->nw > IPC_MAX_BUF) runtime_fail("IPC too many buffers"); + const uint32_t *p = m + 2; + if (special) { + uint32_t h = *p++; + rq->has_pid = h & 1; rq->ncopy = (h >> 1) & 0xf; rq->nmove = (h >> 5) & 0xf; + if (rq->ncopy > 8 || rq->nmove > 8) runtime_fail("IPC too many handles"); + if (rq->has_pid) { memcpy(&rq->pid, p, 8); p += 2; } + for (unsigned i = 0; i < rq->ncopy; i++) rq->copy[i] = *p++; + for (unsigned i = 0; i < rq->nmove; i++) rq->move[i] = *p++; + } + for (unsigned i = 0; i < rq->nx; i++, p += 2) rq->x[i] = (IpcBuffer){static_addr(p), p[0] >> 16, p[0] & 0x3f}; + IpcBuffer *lists[3] = {rq->a, rq->b, rq->w}; unsigned counts[3] = {rq->na, rq->nb, rq->nw}; + for (unsigned k = 0; k < 3; k++) for (unsigned i = 0; i < counts[k]; i++, p += 3) lists[k][i] = (IpcBuffer){buffer_addr(p), buffer_size(p), i}; + *data_words = p; + const uint32_t *recv = p + *ndata; + rq->nc = recv_mode == 2 ? 1 : recv_mode > 2 ? recv_mode - 2 : 0; + if (rq->nc > IPC_MAX_BUF) runtime_fail("IPC too many receive buffers"); + for (unsigned i = 0; i < rq->nc; i++, recv += 2) rq->c[i] = (IpcBuffer){recv[0] | ((uint64_t)(recv[1] & 0xffff) << 32), recv[1] >> 16, i}; + if ((const uint8_t *)recv - (const uint8_t *)m > MSG_BYTES) runtime_fail("IPC request exceeds message buffer"); +} + +static void respond(uint32_t *m, int tipc, int domain, const IpcResponse *rs) { + uint32_t out[MSG_BYTES / 4]; memset(out, 0, sizeof out); + unsigned special = rs->ncopy || rs->nmove; + uint32_t *p = out + 2; + if (special) { + *p++ = (rs->ncopy << 1) | (rs->nmove << 5); + for (unsigned i = 0; i < rs->ncopy; i++) *p++ = rs->copy[i]; + for (unsigned i = 0; i < rs->nmove; i++) *p++ = rs->move[i]; + } + for (unsigned i = 0; i < rs->nxout; i++, p += 2) { + uint64_t a = rs->xout[i].addr; + p[0] = (rs->xout[i].index & 0x3f) | (uint32_t)(((a >> 36) & 0x3f) << 6) | (uint32_t)(((a >> 32) & 0xf) << 12) | (uint32_t)(rs->xout[i].size << 16); + p[1] = (uint32_t)a; + } + uint8_t *words = (uint8_t *)p, *payload; + unsigned header = tipc ? 4 : domain ? 32 : 16; + if (tipc) payload = words; + else payload = (uint8_t *)out + (((words - (uint8_t *)out) + 15) & ~15); + unsigned used = (unsigned)(payload - words) + header + rs->data_size + (domain ? rs->nout_objects * 4 : 0); + unsigned nwords = tipc ? (used + 3) / 4 : (used + 16 + 3) / 4; /* CMIF keeps alignment slack like libnx */ + if ((size_t)(words - (uint8_t *)out) + nwords * 4 > MSG_BYTES) runtime_fail("IPC response exceeds message buffer"); + if (tipc) memcpy(payload, &rs->result, 4); + else { /* a domain response: a 16-byte domain header (out object count), then the CMIF header */ + uint32_t h[4] = {CMIF_OUT_MAGIC, 0, rs->result, 0}; + if (domain) { uint32_t d[4] = {rs->nout_objects, 0, 0, 0}; memcpy(payload, d, 16); } + memcpy(payload + (domain ? 16 : 0), h, 16); + } + memcpy(payload + header, rs->data, rs->data_size); + if (domain) memcpy(payload + header + rs->data_size, rs->out_objects, rs->nout_objects * 4); + out[0] = (rs->nxout & 0xf) << 16; + out[1] = (nwords & 0x3ff) | (special ? 1u << 31 : 0); + memcpy(m, out, MSG_BYTES); +} + +/* ---------- Control requests (CMIF) ---------- */ +static void control(IpcSession *s, const IpcRequest *rq, IpcResponse *rs) { + switch (rq->cmd) { + case 3: { uint16_t n = POINTER_BUFFER_SIZE; IPC_OUT(rs, n); return; } /* QueryPointerBufferSize */ + case 2: case 4: /* CloneCurrentObject(Ex) */ + if (s->domain) runtime_fail("IPC: cloning a domain session is not implemented"); + rs->move[rs->nmove++] = runtime_ipc_new_session(s->name, s->handler, s->state, NULL); + return; + case 0: { /* ConvertCurrentObjectToDomain -> object id */ + if (s->domain) runtime_fail("IPC: session is already a domain"); + IpcSession *own = session_new(s->name, s->handler, s->state, NULL); /* object 1: the session's own interface */ + own->domain = s; + s->domain = s; + s->objects[1] = own; + uint32_t id = 1; IPC_OUT(rs, id); + return; + } + default: + fprintf(stderr, "[totk-ipc] %s: control command %u not implemented\n", s->name, rq->cmd); + runtime_fail("IPC control command not implemented"); + } +} + +static void dispatch(IpcSession *s, uint32_t *m) { + IpcRequest rq; const uint32_t *words; unsigned ndata; + parse(m, &rq, &words, &ndata); + if (getenv("TOTK_IPC_DUMP") && rq.na + rq.nb) { fprintf(stderr, "[totk-ipc] raw %s:", s->name); for (unsigned i = 0; i < 20; i++) fprintf(stderr, " %08x", m[i]); fputc('\n', stderr); } + IpcResponse rs; memset(&rs, 0, sizeof rs); + if (rq.type == T_CLOSE || rq.type == T_TIPC_CLOSE) return; + int tipc = rq.type >= T_TIPC_BASE; + const uint8_t *bytes = (const uint8_t *)words, *end = bytes + ndata * 4; + if (tipc) { rq.cmd = rq.type - T_TIPC_BASE; rq.data = bytes; rq.data_size = (uint32_t)(end - bytes); } + else if (rq.type == T_REQUEST || rq.type == T_REQUEST_CTX || rq.type == T_CONTROL || rq.type == T_CONTROL_CTX) { + const uint8_t *payload = (const uint8_t *)m + (((bytes - (const uint8_t *)m) + 15) & ~15); + if (s->domain == s && (rq.type == T_REQUEST || rq.type == T_REQUEST_CTX)) { + /* domain header: u8 type (1 send, 2 close), u8 in objects, u16 data size, u32 object id, u32 pad, u32 token */ + uint8_t dtype = payload[0], nin = payload[1]; uint16_t dsize; uint32_t object; + memcpy(&dsize, payload + 2, 2); memcpy(&object, payload + 4, 4); + IpcSession *o = object < IPC_MAX_OBJECTS ? s->objects[object] : NULL; + if (!o) { fprintf(stderr, "[totk-ipc] %s: domain object %u does not exist\n", s->name, object); runtime_fail("IPC bad domain object"); } + if (dtype == 2) { /* close the object; the client reads no response */ + if (object != 1) { object_free(o); s->objects[object] = NULL; } + respond(m, 0, 1, &rs); + return; + } + if (dtype != 1) runtime_fail("IPC unknown domain message type"); + if (nin > 8) runtime_fail("IPC too many domain in objects"); + rq.nin_objects = nin; + memcpy(rq.in_objects, payload + 16 + dsize, nin * 4u); + payload += 16; + end = payload + dsize; + s = o; + } + uint32_t h[4]; memcpy(h, payload, 16); + if (h[0] != CMIF_IN_MAGIC) { fprintf(stderr, "[totk-ipc] %s: bad CMIF magic %#x\n", s->name, h[0]); runtime_fail("IPC bad CMIF header"); } + rq.cmd = h[2]; rq.data = payload + 16; rq.data_size = payload + 16 <= end ? (uint32_t)(end - payload - 16) : 0; + } else { + fprintf(stderr, "[totk-ipc] %s: message type %u at %p:", s->name, rq.type, (void *)m); + for (unsigned i = 0; i < 16; i++) fprintf(stderr, " %08x", m[i]); + fputc('\n', stderr); runtime_fail("IPC message type not implemented"); + } + if (rq.type == T_CONTROL || rq.type == T_CONTROL_CTX) control(s, &rq, &rs); + else s->handler(s, &rq, &rs); + respond(m, tipc, s->domain && (rq.type == T_REQUEST || rq.type == T_REQUEST_CTX), &rs); +} + +/* ---------- sm: service manager ---------- */ +static void sm(IpcSession *s, const IpcRequest *rq, IpcResponse *rs) { + switch (rq->cmd) { + case 0: return; /* RegisterClient / Initialize */ + case 1: { /* GetServiceHandle(name) */ + char name[9] = {0}; memcpy(name, ipc_arg(rq, 0, 8), 8); + pthread_mutex_lock(®istry); + const Service *found = NULL; + for (unsigned i = 0; i < nservices; i++) if (!strcmp(services[i].name, name)) found = &services[i]; + pthread_mutex_unlock(®istry); + if (!found) { fprintf(stderr, "[totk-ipc] sm: service \"%s\" not implemented\n", name); runtime_trace_guest(trace_fp, trace_lr); runtime_fail("IPC service not implemented"); } + rs->move[rs->nmove++] = runtime_ipc_new_session(found->name, found->handler, found->open ? found->open() : NULL, found->close); + fprintf(stderr, "[totk-ipc] opened service %s\n", found->name); + return; + } + default: ipc_unimplemented(s, rq); + } +} + +/* ---------- SVCs ---------- */ +int runtime_ipc_svc(unsigned num, uint64_t *r) { + switch (num) { + case 0x1f: { /* ConnectToNamedPort(x1 name) -> w1 */ + char name[13] = {0}; memcpy(name, (const void *)(uintptr_t)r[1], 12); + if (strcmp(name, "sm:")) { fprintf(stderr, "[totk-ipc] named port \"%s\"\n", name); runtime_fail("IPC named port not implemented"); } + r[0] = 0; r[1] = runtime_ipc_new_session("sm:", sm, NULL, NULL); return 1; + } + case 0x21: case 0x22: { /* SendSyncRequest(w0) / WithUserBuffer(x0, x1, w2) */ + uint32_t h = (uint32_t)(num == 0x21 ? r[0] : r[2]); + IpcSession *s = runtime_handle_get(h, 1); + if (!s) { r[0] = 0xe401; return 1; } + uint32_t *m = num == 0x21 ? (uint32_t *)(uintptr_t)guest_cpu()->tpidrro : (uint32_t *)(uintptr_t)r[0]; + if (!m || (num == 0x22 && r[1] < MSG_BYTES)) runtime_fail("IPC message buffer"); + trace_fp = r[29]; trace_lr = r[30]; + dispatch(s, m); r[0] = 0; return 1; + } + case 0x15: { /* CreateTransferMemory(x1 addr, x2 size, w3 perm) -> w1 */ + IpcTransferMemory *t = malloc(sizeof *t); + if (!t) runtime_fail("transfer memory allocation"); + *t = (IpcTransferMemory){r[1], r[2], (uint32_t)r[3]}; + r[0] = 0; r[1] = runtime_handle_new(0, t); return 1; + } + default: return 0; + } +} diff --git a/src/host/ipc.h b/src/host/ipc.h new file mode 100644 index 0000000..0126748 --- /dev/null +++ b/src/host/ipc.h @@ -0,0 +1,66 @@ +/* From MacNativeGames games/totk/runtime/ipc.h @b75be8c (MIT). Added: CMIF domains (objects in one session). */ +/* Host-side Horizon IPC: HIPC message framing with CMIF/TIPC payloads, served in-process. + The SDK client libraries run unchanged; services are host implementations registered by name. */ +#pragma once +#include +#include + +#define IPC_MAX_BUF 8 +typedef struct { uint64_t addr, size; unsigned index; } IpcBuffer; + +typedef struct { + unsigned type, cmd; // HIPC type (4/5/6/7 CMIF, >=16 TIPC) and command id + int has_pid; uint64_t pid; + uint32_t copy[8], move[8]; unsigned ncopy, nmove; + IpcBuffer x[IPC_MAX_BUF], a[IPC_MAX_BUF], b[IPC_MAX_BUF], w[IPC_MAX_BUF], c[IPC_MAX_BUF]; + unsigned nx, na, nb, nw, nc; + const uint8_t *data; uint32_t data_size; // command arguments after the CMIF header + uint32_t in_objects[8]; unsigned nin_objects; // domain requests: object ids passed in +} IpcRequest; + +typedef struct { + uint32_t result; + uint8_t data[0x100]; uint32_t data_size; + uint32_t copy[8], move[8]; unsigned ncopy, nmove; + IpcBuffer xout[IPC_MAX_BUF]; unsigned nxout; // out-pointer data placed by the service into client C buffers + uint32_t out_objects[8]; unsigned nout_objects; // domain responses: new object ids +} IpcResponse; + +struct IpcSession; +typedef void (*IpcHandler)(struct IpcSession *s, const IpcRequest *rq, IpcResponse *rs); +#define IPC_MAX_OBJECTS 64 +typedef struct IpcSession { + const char *name; IpcHandler handler; void *state; void (*close)(void *state); + struct IpcSession *domain; // the domain this object lives in (NULL: a plain session) + struct IpcSession *objects[IPC_MAX_OBJECTS]; // a domain session: its objects by id (1 = its own interface) +} IpcSession; + +/* Service registry: open(state out) creates per-session state for a named service. */ +typedef void *(*IpcOpen)(void); +void runtime_ipc_register(const char *name, IpcHandler handler, IpcOpen open, void (*close)(void *)); +uint32_t runtime_ipc_new_session(const char *name, IpcHandler handler, void *state, void (*close)(void *)); +int runtime_ipc_svc(unsigned num, uint64_t *regs); /* returns 1 when handled */ + +/* Response helpers */ +void ipc_out(IpcResponse *rs, const void *p, uint32_t n); +#define IPC_OUT(rs, v) ipc_out((rs), &(v), sizeof(v)) +#define IPC_ARG(rq, T, off) (*(const T *)ipc_arg((rq), (off), sizeof(T))) +const void *ipc_arg(const IpcRequest *rq, uint32_t off, uint32_t n); +void *ipc_guest(uint64_t addr, uint64_t size); /* validated guest pointer */ +_Noreturn void ipc_unimplemented(const IpcSession *s, const IpcRequest *rq); +/* A command that returns an interface: a new object id in a domain, a new session handle otherwise. */ +void ipc_out_object(IpcSession *s, IpcResponse *rs, const char *name, IpcHandler handler, void *state, void (*close)(void *)); + +/* threads.c */ +uint32_t runtime_handle_new(int session, void *obj); +void *runtime_handle_get(uint32_t h, int session); +uint32_t runtime_event_new(void); +uint32_t runtime_event_readable(uint32_t writable); +void runtime_event_signal(uint32_t writable); +void runtime_ipc_session_closed(void *session); +void runtime_ipc_tmem_closed(void *tmem); + +/* Transfer memory created by svcCreateTransferMemory */ +typedef struct { uint64_t addr, size; uint32_t perm; } IpcTransferMemory; + +#define IPC_RESULT_UNKNOWN_COMMAND 0xf601u /* sf: unknown command (module 10, description 123) */ diff --git a/src/host/motion_fusion.c b/src/host/motion_fusion.c new file mode 100644 index 0000000..97302cd --- /dev/null +++ b/src/host/motion_fusion.c @@ -0,0 +1,147 @@ +/* From MacNativeGames games/totk/runtime/motion_fusion.c @b75be8c (MIT). [port]: fed by the SDL3 controller + * sensors (window_sdl.c), which are converted to the frame below there; the code itself is unchanged. */ +/* Six-axis states from the Mac controller's motion samples (2026-10-17; fed by runtime/gamepad_feedback.m, read by the + game through runtime/hid_feedback.c, only with TOTK_GYRO=1). + Input per sample: acceleration in G including gravity and rotation rate in rad/s, in the controller's frame (GCMotion: + x to the right, y toward the top edge, z out of the face; lying face up it reads z = -1 G). That is the Switch's + six-axis frame, so by default nothing is remapped; TOTK_GYRO_AXES="-x,y,-z" style remaps (a reflection also flips the + rotation's sense). Output per sample, like the Switch: acceleration in G, angular velocity in revolutions per second, + angle = the integrated revolutions per axis, direction = the controller's three axes in a reference frame whose z is + up. The orientation is integrated from the gyro and, while the controller is not being accelerated (|a| within 15 % + of 1 G), pulled toward the measured gravity (Mahony, gain 0.5/s: about 2 s), as the Switch's sensor fusion does; yaw + has no reference and drifts with the gyro's bias (well under a degree a minute on a DualSense at rest). The newest 64 + samples are kept; the game reads up to 16, newest first, and goes by their sampling numbers. + Dead zone (2026-10-19): rotation slower than TOTK_GYRO_DEADZONE deg/s counts as none, and the dead zone is taken off + faster rotation, so the rate never jumps at its edge. Default 0, except 1.2 while a controller's motion comes from its + HID reports (runtime_motion_deadzone_default, gamepad_feedback.m): an 8BitDo Pro Controller lying on the desk read + 0.83 deg/s on average (bias 0.05, 0.73, 0.38 deg/s after its factory calibration), at most 1.07 deg/s, in 616 samples, + enough for the aim to creep; a DualSense through GameController needs none. */ +#include "host.h" // [port] runtime.h +#include +#include +#include +#include +#include + +enum { MOTION_RING = 64 }; +static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER; +static RuntimeMotionState ring[MOTION_RING]; +static uint64_t produced, last_ns, logged, log_at, reads_logged; +static double R[3][3]; /* controller -> reference; row i = reference axis i in controller coordinates */ +static double angle[3]; +static int restart = 1, live; +static int from[3] = {0, 1, 2}; +static double sign[3] = {1, 1, 1}, det = 1; +static double deadzone = -1; /* rad/s; -1 until read from TOTK_GYRO_DEADZONE */ +static double deadzone_default; /* deg/s, when TOTK_GYRO_DEADZONE is unset */ + +/* TOTK_GYRO_AXES: reference axis i = sign * controller axis from[i]. 0 if unset, -1 if malformed. */ +int runtime_motion_axes(const char *spec) { + int f[3], used[3] = {0}; double s[3], parity = 1; const char *p = spec; + if (!spec) return 0; + for (int i = 0; i < 3; i++) { + s[i] = 1; if (*p == '-') { s[i] = -1; p++; } else if (*p == '+') p++; + if (*p < 'x' || *p > 'z' || used[*p - 'x']) return -1; + f[i] = *p - 'x'; used[f[i]] = 1; p++; + if (i < 2 && *p++ != ',') return -1; + } + if (*p) return -1; + for (int i = 0; i < 3; i++) for (int j = i + 1; j < 3; j++) if (f[i] > f[j]) parity = -parity; + pthread_mutex_lock(&lock); + for (int i = 0; i < 3; i++) { from[i] = f[i]; sign[i] = s[i]; } + det = parity * s[0] * s[1] * s[2]; + pthread_mutex_unlock(&lock); + return 1; +} +static double norm3(const double v[3]) { return sqrt(v[0] * v[0] + v[1] * v[1] + v[2] * v[2]); } +static void cross3(const double a[3], const double b[3], double out[3]) { + out[0] = a[1] * b[2] - a[2] * b[1]; out[1] = a[2] * b[0] - a[0] * b[2]; out[2] = a[0] * b[1] - a[1] * b[0]; +} +static void orthonormalize(void) { /* rows, from the up row (the one the gravity correction keeps right) */ + double n = norm3(R[2]); for (int j = 0; j < 3; j++) R[2][j] /= n; + double d = R[0][0] * R[2][0] + R[0][1] * R[2][1] + R[0][2] * R[2][2]; + for (int j = 0; j < 3; j++) R[0][j] -= d * R[2][j]; + n = norm3(R[0]); for (int j = 0; j < 3; j++) R[0][j] /= n; + cross3(R[2], R[0], R[1]); +} +/* A fresh orientation: z up against the measured gravity, x as close to the controller's x as that allows. */ +static void align_to(const double acc[3]) { + double n = norm3(acc), up[3] = {0, 0, 1}, x[3] = {1, 0, 0}; + if (n > 0.5) for (int j = 0; j < 3; j++) up[j] = -acc[j] / n; + if (fabs(up[0]) > 0.9) { x[0] = 0; x[1] = 1; } + memcpy(R[2], up, sizeof up); memcpy(R[0], x, sizeof x); orthonormalize(); +} +static void integrate(const double gyro[3], const double acc[3], double dt) { + double w[3] = {gyro[0], gyro[1], gyro[2]}, n = norm3(acc); + if (fabs(n - 1) < 0.15) { /* not being accelerated: turn the estimated down toward the measured one */ + const double kp = 0.5; + double measured[3] = {acc[0] / n, acc[1] / n, acc[2] / n}, estimated[3] = {-R[2][0], -R[2][1], -R[2][2]}, e[3]; + cross3(measured, estimated, e); + for (int j = 0; j < 3; j++) w[j] += kp * e[j]; + } + double rate = norm3(w), theta = rate * dt; + if (theta < 1e-12) return; + double k[3] = {w[0] / rate, w[1] / rate, w[2] / rate}, s = sin(theta), c = 1 - cos(theta); + double K[3][3] = {{0, -k[2], k[1]}, {k[2], 0, -k[0]}, {-k[1], k[0], 0}}, E[3][3], out[3][3]; + for (int i = 0; i < 3; i++) for (int j = 0; j < 3; j++) { + double kk = 0; for (int m = 0; m < 3; m++) kk += K[i][m] * K[m][j]; + E[i][j] = (i == j) + s * K[i][j] + c * kk; + } + static int negative = -1; if (negative < 0) { const char *e = getenv("TOTK_PROBE_NEGATIVE"); negative = e && atoi(e) == 20; } + for (int i = 0; i < 3; i++) for (int j = 0; j < 3; j++) { /* body-frame rate: R * E (probe negative 20: E * R, a reference-frame rate) */ + double v = 0; for (int m = 0; m < 3; m++) v += negative ? E[i][m] * R[m][j] : R[i][m] * E[m][j]; + out[i][j] = v; + } + memcpy(R, out, sizeof R); orthonormalize(); +} +void runtime_motion_feed(const double raw_acc[3], const double raw_gyro[3], uint64_t now) { + double acc[3], gyro[3]; + pthread_mutex_lock(&lock); + for (int i = 0; i < 3; i++) { acc[i] = sign[i] * raw_acc[from[i]]; gyro[i] = det * sign[i] * raw_gyro[from[i]]; } + if (deadzone < 0) { + const char *e = getenv("TOTK_GYRO_DEADZONE"); double v = deadzone_default; + if (e) { char *end; v = strtod(e, &end); if (end == e || *end || !(v >= 0) || v > 20) runtime_fail("TOTK_GYRO_DEADZONE must be a number of degrees a second in [0, 20]"); } + deadzone = v * M_PI / 180; + if (e) fprintf(stderr, "[gyro] dead zone %.2f deg/s (TOTK_GYRO_DEADZONE)\n", v); + } + double rate = norm3(gyro), keep = rate > deadzone ? (rate - deadzone) / rate : 0; + for (int i = 0; i < 3; i++) gyro[i] *= keep; + double dt = last_ns && now > last_ns ? (now - last_ns) / 1e9 : 0.004; + if (dt > 0.05) dt = 0.004; /* a gap (sensors were off, the app was in the background): no integration across it */ + last_ns = now; live = 1; + if (restart) { align_to(acc); memset(angle, 0, sizeof angle); restart = 0; } + else integrate(gyro, acc, dt); + RuntimeMotionState *s = &ring[produced % MOTION_RING]; + *s = (RuntimeMotionState){.sampling = produced + 1, .delta_ns = (uint64_t)(dt * 1e9)}; + for (int i = 0; i < 3; i++) { + double rps = gyro[i] / (2 * M_PI); angle[i] += rps * dt; + s->acc[i] = (float)acc[i]; s->gyro[i] = (float)rps; s->angle[i] = (float)angle[i]; + for (int j = 0; j < 3; j++) s->dir[i * 3 + j] = (float)R[j][i]; /* direction i = controller axis i in the reference frame */ + } + produced++; + if (!log_at) log_at = now; + if (now - log_at >= 30000000000ull) { + uint64_t values, reads; runtime_hid_feedback_stats(&values, &reads); + fprintf(stderr, "[gyro] last 30 s: %llu samples (%.0f a second), the game read the states %llu times; now acc (%+.2f %+.2f %+.2f) G, gyro (%+.3f %+.3f %+.3f) rev/s\n", + (unsigned long long)(produced - logged), (produced - logged) / ((now - log_at) / 1e9), (unsigned long long)(reads - reads_logged), + s->acc[0], s->acc[1], s->acc[2], s->gyro[0], s->gyro[1], s->gyro[2]); + logged = produced; reads_logged = reads; log_at = now; + } + pthread_mutex_unlock(&lock); +} +int runtime_motion_read(RuntimeMotionState *out, int max) { + if (max <= 0) return 0; + int n = 0; + pthread_mutex_lock(&lock); + if (live && runtime_now_ns() - last_ns < 200000000ull) { /* no sample for 200 ms: no motion */ + uint64_t have = produced < MOTION_RING ? produced : MOTION_RING; + n = (int)(have < (uint64_t)max ? have : (uint64_t)max); + for (int i = 0; i < n; i++) out[i] = ring[(produced - 1 - (uint64_t)i) % MOTION_RING]; + } + pthread_mutex_unlock(&lock); + return n; +} +void runtime_motion_start(void) { pthread_mutex_lock(&lock); restart = 1; pthread_mutex_unlock(&lock); } +/* The dead zone in deg/s when TOTK_GYRO_DEADZONE is unset: the HID reader sets 1.2, a GameController controller 0. */ +void runtime_motion_deadzone_default(double deg) { pthread_mutex_lock(&lock); deadzone_default = deg; deadzone = -1; pthread_mutex_unlock(&lock); } +void runtime_motion_live(int on) { pthread_mutex_lock(&lock); live = on; if (on) restart = 1; pthread_mutex_unlock(&lock); } diff --git a/src/host/movie_host.c b/src/host/movie_host.c new file mode 100644 index 0000000..9b6ea14 --- /dev/null +++ b/src/host/movie_host.c @@ -0,0 +1,498 @@ +/* From MacNativeGames games/totk/runtime/movie_host.c @b75be8c (MIT). [port] marks the Windows changes: + guest functions are called through GCALL (the game is recompiled C, not native ARM code), the entries are bound by + symbol name with HLE_NATIVE, input buffers the game writes live in guest-visible host memory, NV12 interleave in + SSE2 instead of NEON, and the probe hooks (movie_probe.c) are left out. */ +/* movie::Decoder for VP9 video on the host with libvpx. + TOTK's pre-rendered cutscenes (the .webm files in Event/Movie, 28 of them: VP9 profile 0, 1920x1080, 30 fps, video only) + play through the SDK movie library in subsdk0, which is Android's MediaCodec over NVIDIA's NvMM. Its video decoder opens + Tegra's NVDEC through nvdrv (NvMMLiteOpenVideo -> NvRmChannelOpen -> NvOsDrvOpen), which this port does not have, so the + SDK aborted at the first cutscene (here: Ganondorf at the end of the prologue cave, 2026-10-23). + main imports the constructor and calls the 17 virtual methods through the vtable for movie::Decoder, whose slots are + relocated by symbol name: overriding the names replaces the slots (src/runtime/loader.c resolve). A VP9 buffer-mode + decoder (type 3, mode 1) built by the constructor is a host object, +0 the library's vtable and +8 a host Movie where + the library keeps its DecoderImpl; every other decoder goes to the original code. Behaviour follows DecoderImpl: + - output format 4 is packed NV12 (Y plane, then interleaved UV; stride = width), written into the client's Buffer; + - input and output buffers are indices handed out FIFO, as MediaCodec does; input flag 4 / output flag 4 = end of stream; + - client events {input available, output available, format changed, error} (DecoderEvents, 4 EventType pointers) are + signalled with nn::os::SignalEvent. + Decoding is synchronous in SendInputBufferForDecode. While the game holds every output slot, inputs wait and are decoded + when it releases one (ReleaseOutputBufferIndex). + Pacing: the game shows one decoded frame per game frame, which plays a 30 fps movie at 30 fps only at the original 30 + fps. So decoded frames are handed to the game when their presentation time is due, counted from the first frame + (movie_host_tick, called at every present on the game's thread); in between the movie texture keeps the last frame. + At most one frame is handed over per game frame, and when the game falls behind a frame more than 1.5 frame periods + late is dropped if the next one is due too: the picture stays in time with the cutscene's sound. + TOTK_MOVIE_PACE=0 hands every frame over as soon as it is decoded; TOTK_MOVIE_HOST=0 leaves every decoder to the + library (which aborts). */ +#include "host.h" // [port] runtime.h +#include "../runtime/hle.h" +#include // [port] arm_neon.h +#include +#include +#include +#include +#include +#include +#include + +enum { IN_BUFS = 4, IN_CAP = 2 << 20, OUT_BUFS = 8, MAX_MOVIES = 16 }; +enum { TYPE_VP9 = 3, MODE_BUFFER = 1, FORMAT_NV12 = 4, FLAG_EOS = 4 }; +enum { ST_OK = 0, ST_ERROR = 0x1000, ST_NULL = 0x100b, ST_NOT_READY = 0x100c, ST_NO_OUTPUT = 0x100e, ST_NO_INPUT = 0x1011, + ST_UNSUPPORTED = 0x2001, ST_NOT_CONFIGURED = 0x2002 }; +enum { EV_INPUT, EV_OUTPUT, EV_FORMAT, EV_ERROR }; +enum { MOVIE_CTOR, MOVIE_D2, MOVIE_D0, MOVIE_CONFIGURE, MOVIE_REGISTER_EVENTS, MOVIE_GET_OUTPUT_FORMAT, MOVIE_START, + MOVIE_STOP, MOVIE_RESET, MOVIE_FLUSH, MOVIE_ACQUIRE_INPUT, MOVIE_GET_INPUT, MOVIE_SEND_INPUT, MOVIE_ACQUIRE_OUTPUT, + MOVIE_GET_OUTPUT, MOVIE_GET_OUTPUT_SIZE, MOVIE_RELEASE_OUTPUT, MOVIE_GET_LAST_ERROR, MOVIE_GET_OUTPUT_MD, + MOVIE_GET_OUTPUT_BUFFER_FORMAT, MOVIE_ENTRIES }; + +/* [port] guest addresses of the functions the decoder calls (GCALL), and the original movie::Decoder code */ +typedef struct { + void *signal_event, *buf_set_range, *buf_set_data, *buf_base, *buf_capacity; + void *md_find_int32, *md_find_string, *md_set_int32, *md_set_rect, *md_set_string; + void *vptr; /* vtable for movie::Decoder + 0x10: what the constructor stores at +0 */ + void *orig[MOVIE_ENTRIES]; +} MovieGuest; + +typedef struct { int index; unsigned flags; int64_t time; } Ready; +typedef struct { int index, offset, size; int64_t time; unsigned flags; } Pending; +typedef struct { + void *guest; /* the movie::Decoder object */ + int format, configured, started, w, h, eos_in, format_sent, last_error; + void **events; /* DecoderEvents: EventType* per EV_* */ + vpx_codec_ctx_t vpx; int vpx_live; + uint8_t *in[IN_BUFS]; int in_free[IN_BUFS], nin_free; + Pending pending[IN_BUFS]; int npending; + uint8_t *out[OUT_BUFS]; int out_bytes[OUT_BUFS], out_held[OUT_BUFS]; + Ready ready[OUT_BUFS]; int nready; + Ready decoded[OUT_BUFS]; int ndecoded; /* paced: decoded, not due yet */ + uint64_t t0; int64_t pts0; /* paced: host time and presentation time of the first frame handed over */ + uint64_t frames, decode_ns, errors, dropped; + pthread_mutex_t mu; +} Movie; + +static MovieGuest G; static int g_ready; +static int pace_on(void) { static int v = -1; if (v < 0) { const char *e = getenv("TOTK_MOVIE_PACE"); v = !(e && !strcmp(e, "0")); } return v; } +static Movie *g_movies[MAX_MOVIES]; +static pthread_mutex_t g_lock = PTHREAD_MUTEX_INITIALIZER; + +static const char *const NAMES[MOVIE_ENTRIES] = { + "_ZN5movie7DecoderC1ENS_11DecoderTypeENS_19DecoderOutputFormatENS_11DecoderModeEi", "_ZN5movie7DecoderD2Ev", + "_ZN5movie7DecoderD0Ev", "_ZN5movie7Decoder9ConfigureEPNS_9MediaDataE", + "_ZN5movie7Decoder20RegisterClientEventsEPNS_13DecoderEventsE", "_ZN5movie7Decoder15GetOutputFormatEPNS_9MediaDataE", + "_ZN5movie7Decoder5StartEv", "_ZN5movie7Decoder4StopEv", "_ZN5movie7Decoder5ResetEv", "_ZN5movie7Decoder5FlushEv", + "_ZN5movie7Decoder23AcquireInputBufferIndexEPiS1_", "_ZN5movie7Decoder14GetInputBufferEiPNS_6BufferE", + "_ZN5movie7Decoder24SendInputBufferForDecodeEiiilj", "_ZN5movie7Decoder24AcquireOutputBufferIndexEPiPlPjS1_", + "_ZN5movie7Decoder15GetOutputBufferEiPNS_6BufferE", "_ZN5movie7Decoder19GetOutputBufferSizeEPiS1_", + "_ZN5movie7Decoder24ReleaseOutputBufferIndexEi", "_ZN5movie7Decoder12GetLastErrorEv", + "_ZN5movie7Decoder15GetOutputBufferEiPNS_6BufferEPNS_9MediaDataE", "_ZN5movie7Decoder21GetOutputBufferFormatEiPNS_9MediaDataE"}; + +#define MLOG(...) do { fprintf(stderr, "[movie] " __VA_ARGS__); fputc('\n', stderr); } while (0) +#define U(x) ((uint64_t)(uintptr_t)(x)) +static uint64_t now_ns(void) { return runtime_now_ns(); } + +/* [port] the guest side: SDK and movie library functions, called on the calling guest thread */ +static void g_signal(void *ev) { GCALL(G.signal_event, U(ev)); } +static void buf_set_range(void *buf, int offset, int size) { GCALL(G.buf_set_range, U(buf), (uint64_t)offset, (uint64_t)size); } +static void buf_set_data(void *buf, void *data, size_t cap) { GCALL(G.buf_set_data, U(buf), U(data), (uint64_t)cap); } +static uint8_t *buf_base(const void *buf) { return (uint8_t *)(uintptr_t)GCALL(G.buf_base, U(buf)); } +static size_t buf_capacity(const void *buf) { return (size_t)GCALL(G.buf_capacity, U(buf)); } +static bool md_find_int32(const void *md, const char *key, int *v) { return (GCALL(G.md_find_int32, U(md), U(key), U(v)) & 0xFF) != 0; } +static bool md_find_string(const void *md, const char *key, const char **v) { return (GCALL(G.md_find_string, U(md), U(key), U(v)) & 0xFF) != 0; } +static void md_set_int32(void *md, const char *key, int v) { GCALL(G.md_set_int32, U(md), U(key), (uint64_t)(uint32_t)v); } +static void md_set_rect(void *md, const char *key, int l, int t, int r, int b) { + GCALL(G.md_set_rect, U(md), U(key), (uint64_t)(uint32_t)l, (uint64_t)(uint32_t)t, (uint64_t)(uint32_t)r, (uint64_t)(uint32_t)b); +} +static void md_set_string(void *md, const char *key, const char *v) { GCALL(G.md_set_string, U(md), U(key), U(v)); } +static uint64_t orig(int which, int n, const uint64_t *args) { return guest_call_args(U(G.orig[which]), n, args); } + +/* [port] input buffers are written by the game's code: guest-visible host memory, which has no free, so they are kept + for reuse (one cutscene at a time in practice; 28 cutscenes would otherwise use up the heap) */ +static uint8_t *g_in_pool[MAX_MOVIES * IN_BUFS]; static int g_in_pooled; +static uint8_t *in_get(void) { + pthread_mutex_lock(&g_lock); + uint8_t *p = g_in_pooled ? g_in_pool[--g_in_pooled] : NULL; + pthread_mutex_unlock(&g_lock); + return p ? p : hle_alloc(IN_CAP); +} +static void in_put(uint8_t *p) { + if (!p) return; + pthread_mutex_lock(&g_lock); + if (g_in_pooled < (int)(sizeof g_in_pool / sizeof g_in_pool[0])) g_in_pool[g_in_pooled++] = p; + pthread_mutex_unlock(&g_lock); +} + +static void need_symbol(void **slot, const char *name) { + *slot = runtime_symbol(name); /* the module's own definition, not the host entry bound in its place */ + if (!*slot) { fprintf(stderr, "[movie] missing guest symbol %s\n", name); runtime_fail("movie decoder: guest symbol missing"); } +} +/* The guest functions are resolved the first time any entry runs (the modules are loaded and relocated by then). */ +static void guest_init(void) { + pthread_mutex_lock(&g_lock); + if (!g_ready) { + need_symbol(&G.signal_event, "_ZN2nn2os11SignalEventEPNS0_9EventTypeE"); + need_symbol(&G.buf_set_range, "_ZN5movie6Buffer8SetRangeEii"); + need_symbol(&G.buf_set_data, "_ZN5movie6Buffer18SetDataAndCapacityEPvm"); + need_symbol(&G.buf_base, "_ZNK5movie6Buffer4BaseEv"); + need_symbol(&G.buf_capacity, "_ZNK5movie6Buffer8CapacityEv"); + need_symbol(&G.md_find_int32, "_ZNK5movie9MediaData9FindInt32EPKcPi"); + need_symbol(&G.md_find_string, "_ZNK5movie9MediaData10FindStringEPKcPS2_"); + need_symbol(&G.md_set_int32, "_ZN5movie9MediaData8SetInt32EPKci"); + need_symbol(&G.md_set_rect, "_ZN5movie9MediaData7SetRectEPKciiii"); + need_symbol(&G.md_set_string, "_ZN5movie9MediaData9SetStringEPKcS2_"); + void *vt; need_symbol(&vt, "_ZTVN5movie7DecoderE"); G.vptr = (uint8_t *)vt + 0x10; + for (int i = 0; i < MOVIE_ENTRIES; i++) need_symbol(&G.orig[i], NAMES[i]); + g_ready = 1; + } + pthread_mutex_unlock(&g_lock); +} + +static Movie *find(const void *guest) { + if (!g_ready) guest_init(); + pthread_mutex_lock(&g_lock); Movie *m = NULL; + for (int i = 0; i < MAX_MOVIES; i++) if (g_movies[i] && g_movies[i]->guest == guest) m = g_movies[i]; + pthread_mutex_unlock(&g_lock); + return m; +} + +/* Events are signalled after the decoder's lock is released: the woken game thread calls straight back in. */ +typedef struct { void *ev[8]; int n; } Signals; +static void note(Movie *m, Signals *s, int which) { if (m->events && m->events[which] && s->n < 8) s->ev[s->n++] = m->events[which]; } +static void fire(const Signals *s) { for (int i = 0; i < s->n; i++) g_signal(s->ev[i]); } + +static int frame_bytes(const Movie *m) { return m->w * m->h + (m->w / 2) * (m->h / 2) * 2; } + +/* I420 (libvpx) -> packed NV12: Y rows, then rows of interleaved U, V. */ +static void to_nv12(const vpx_image_t *img, uint8_t *dst, int w, int h) { + for (int y = 0; y < h; y++) memcpy(dst + (size_t)y * w, img->planes[VPX_PLANE_Y] + (size_t)y * img->stride[VPX_PLANE_Y], (size_t)w); + uint8_t *uv = dst + (size_t)w * h; + for (int y = 0; y < h / 2; y++) { + const uint8_t *u = img->planes[VPX_PLANE_U] + (size_t)y * img->stride[VPX_PLANE_U]; + const uint8_t *v = img->planes[VPX_PLANE_V] + (size_t)y * img->stride[VPX_PLANE_V]; + uint8_t *o = uv + (size_t)y * w; int x = 0; + for (; x + 16 <= w / 2; x += 16) { // [port] SSE2 + __m128i a = _mm_loadu_si128((const __m128i *)(u + x)), b = _mm_loadu_si128((const __m128i *)(v + x)); + _mm_storeu_si128((__m128i *)(o + 2 * x), _mm_unpacklo_epi8(a, b)); + _mm_storeu_si128((__m128i *)(o + 2 * x + 16), _mm_unpackhi_epi8(a, b)); + } + for (; x < w / 2; x++) { o[2 * x] = u[x]; o[2 * x + 1] = v[x]; } + } +} +static int free_out(const Movie *m) { for (int i = 0; i < OUT_BUFS; i++) if (!m->out_held[i]) return i; return -1; } +static void push_ready(Movie *m, Signals *s, int slot, unsigned flags, int64_t time, int bytes) { + m->out_held[slot] = 1; m->out_bytes[slot] = bytes; + if (pace_on()) { m->decoded[m->ndecoded++] = (Ready){slot, flags, time}; return; } + m->ready[m->nready++] = (Ready){slot, flags, time}; + note(m, s, EV_OUTPUT); +} +/* Host time a decoded frame is due (ns); end of stream is due as soon as it is first in line. */ +static uint64_t due_at(Movie *m, const Ready *d) { + int64_t ahead = d->time - m->pts0; if (ahead < 0) ahead = 0; + return m->t0 + (uint64_t)ahead * 1000; +} +/* One game frame (a present): drop leading frames that are more than 1.5 frame periods late while the next one is due + too, then hand over at most one due frame (end of stream right after the last frame). */ +static void release_due(Movie *m, Signals *s, uint64_t now) { + while (m->ndecoded > 1 && !(m->decoded[0].flags & FLAG_EOS) && !(m->decoded[1].flags & FLAG_EOS) && m->t0) { + uint64_t due = due_at(m, &m->decoded[0]), next = due_at(m, &m->decoded[1]); + if (now < next || now - due <= (next - due) * 3 / 2) break; + m->out_held[m->decoded[0].index] = 0; m->dropped++; + memmove(m->decoded, m->decoded + 1, (size_t)--m->ndecoded * sizeof m->decoded[0]); + } + if (!m->ndecoded) return; + Ready d = m->decoded[0]; + if (!(d.flags & FLAG_EOS)) { + if (!m->t0) { m->t0 = now; m->pts0 = d.time; } + if (now < due_at(m, &d)) return; + } + memmove(m->decoded, m->decoded + 1, (size_t)--m->ndecoded * sizeof m->decoded[0]); + m->ready[m->nready++] = d; note(m, s, EV_OUTPUT); +} +/* Called by nvnQueuePresentTexture (src/gfx/nvn/hle_nvn.c) on the game's thread. */ +void movie_host_tick(void) { /* lock order g_lock -> Movie.mu, as in destroy; events fire after both are released */ + Signals s = {0}; + pthread_mutex_lock(&g_lock); + for (int i = 0; i < MAX_MOVIES; i++) { + Movie *m = g_movies[i]; if (!m) continue; + pthread_mutex_lock(&m->mu); if (m->started) release_due(m, &s, now_ns()); pthread_mutex_unlock(&m->mu); + } + pthread_mutex_unlock(&g_lock); + fire(&s); +} +/* Decode one input into output slots; returns 0 when it had to wait for a free output slot. */ +static int decode_one(Movie *m, Signals *s, const Pending *p) { + int need = p->size > 0 ? 1 : 0; + if (p->flags & FLAG_EOS && !m->eos_in) need++; + int freeslots = 0; for (int i = 0; i < OUT_BUFS; i++) freeslots += !m->out_held[i]; + if (freeslots < need) return 0; + if (p->size > 0) { + uint64_t t0 = now_ns(); + vpx_codec_err_t e = vpx_codec_decode(&m->vpx, m->in[p->index] + p->offset, (unsigned)p->size, NULL, 0); + if (e != VPX_CODEC_OK) { + m->errors++; m->last_error = ST_ERROR; note(m, s, EV_ERROR); + if (m->errors <= 4) MLOG("VP9 decode error on a %d-byte frame at %lld us: %s", p->size, (long long)p->time, vpx_codec_error_detail(&m->vpx) ? vpx_codec_error_detail(&m->vpx) : vpx_codec_err_to_string(e)); + } else { + vpx_codec_iter_t it = NULL; const vpx_image_t *img; + while ((img = vpx_codec_get_frame(&m->vpx, &it)) != NULL) { + if (img->fmt != VPX_IMG_FMT_I420 || (int)img->d_w != m->w || (int)img->d_h != m->h) { + MLOG("VP9 frame %ux%u format %d differs from the configured %dx%d I420", img->d_w, img->d_h, img->fmt, m->w, m->h); + runtime_fail("movie decoder: unexpected VP9 frame layout"); + } + int slot = free_out(m); if (slot < 0) runtime_fail("movie decoder: output slots counted wrong"); + if (!m->format_sent) { m->format_sent = 1; note(m, s, EV_FORMAT); } + to_nv12(img, m->out[slot], m->w, m->h); + push_ready(m, s, slot, 0, p->time, frame_bytes(m)); + if (!m->frames++) MLOG("first frame decoded (%lld us, Y at the centre %d)", (long long)p->time, m->out[slot][(size_t)m->w * (m->h / 2) + m->w / 2]); + } + } + m->decode_ns += now_ns() - t0; + if (m->frames && m->frames % 300 == 0) MLOG("%llu frames, decode %.2f ms per frame", (unsigned long long)m->frames, m->decode_ns / 1e6 / (double)m->frames); + } + if (p->flags & FLAG_EOS && !m->eos_in) { + m->eos_in = 1; int slot = free_out(m); if (slot < 0) runtime_fail("movie decoder: no slot for end of stream"); + push_ready(m, s, slot, FLAG_EOS, p->time, 0); + MLOG("end of stream after %llu frames (%.2f ms per frame, %llu errors)", (unsigned long long)m->frames, m->frames ? m->decode_ns / 1e6 / (double)m->frames : 0.0, (unsigned long long)m->errors); + } + m->in_free[m->nin_free++] = p->index; note(m, s, EV_INPUT); + return 1; +} +static void drain_pending(Movie *m, Signals *s) { + while (m->npending && decode_one(m, s, &m->pending[0])) { memmove(m->pending, m->pending + 1, (size_t)--m->npending * sizeof m->pending[0]); } +} +static void release_codec(Movie *m) { + if (m->vpx_live) { vpx_codec_destroy(&m->vpx); m->vpx_live = 0; } + for (int i = 0; i < IN_BUFS; i++) { in_put(m->in[i]); m->in[i] = NULL; } + for (int i = 0; i < OUT_BUFS; i++) { free(m->out[i]); m->out[i] = NULL; } + m->configured = m->started = 0; m->nin_free = m->npending = m->nready = m->ndecoded = 0; m->t0 = 0; +} +static void reset_queues(Movie *m) { + m->nin_free = 0; for (int i = 0; i < IN_BUFS; i++) m->in_free[m->nin_free++] = i; + memset(m->out_held, 0, sizeof m->out_held); m->npending = m->nready = m->ndecoded = 0; m->eos_in = 0; m->t0 = 0; +} +static int host_off(void) { static int off = -1; if (off < 0) { const char *e = getenv("TOTK_MOVIE_HOST"); off = e && !strcmp(e, "0"); } return off; } + +/* ---- entries ---- */ +static void h_ctor(void *obj, int type, int format, int mode, int priority) { + if (!g_ready) guest_init(); + if (host_off() || type != TYPE_VP9 || mode != MODE_BUFFER) { + MLOG("decoder type %d mode %d format %d: left to the library (only VP9 in buffer mode runs on the host)", type, mode, format); + orig(MOVIE_CTOR, 5, (uint64_t[]){U(obj), (uint64_t)type, (uint64_t)format, (uint64_t)mode, (uint64_t)priority}); return; + } + Movie *m = calloc(1, sizeof *m); if (!m) runtime_fail("movie decoder allocation"); + m->guest = obj; m->format = format; pthread_mutex_init(&m->mu, NULL); + pthread_mutex_lock(&g_lock); int i = 0; + while (i < MAX_MOVIES && g_movies[i] && g_movies[i]->guest != obj) i++; + if (i == MAX_MOVIES) runtime_fail("movie decoder table full"); + Movie *old = g_movies[i]; + g_movies[i] = m; pthread_mutex_unlock(&g_lock); + if (old) { MLOG("decoder object %p reused without destruction", obj); release_codec(old); free(old); } + ((void **)obj)[0] = G.vptr; ((void **)obj)[1] = m; + MLOG("VP9 decoder %p on the host (libvpx %s), output format %d", obj, vpx_codec_version_str(), format); +} +static void destroy(void *obj) { + pthread_mutex_lock(&g_lock); Movie *m = NULL; + for (int i = 0; i < MAX_MOVIES; i++) if (g_movies[i] && g_movies[i]->guest == obj) { m = g_movies[i]; g_movies[i] = NULL; } + if (m) { pthread_mutex_lock(&m->mu); pthread_mutex_unlock(&m->mu); } /* wait for an operation in progress (movie_host_tick) */ + pthread_mutex_unlock(&g_lock); + if (!m) return; + MLOG("decoder %p destroyed after %llu frames", obj, (unsigned long long)m->frames); + release_codec(m); pthread_mutex_destroy(&m->mu); free(m); + ((void **)obj)[0] = G.vptr; ((void **)obj)[1] = NULL; /* what the library's destructor leaves (no DecoderImpl) */ +} +static void h_d2(void *obj) { if (!find(obj)) { orig(MOVIE_D2, 1, (uint64_t[]){U(obj)}); return; } destroy(obj); } +/* Deleting destructor: free the host state, then the library's own deleting destructor frees the object (impl now 0). */ +static void h_d0(void *obj) { if (find(obj)) destroy(obj); orig(MOVIE_D0, 1, (uint64_t[]){U(obj)}); } + +static int h_configure(void *obj, void *format) { + Movie *m = find(obj); if (!m) return (int)orig(MOVIE_CONFIGURE, 2, (uint64_t[]){U(obj), U(format)}); + if (!format) return ST_NULL; + if (m->format != FORMAT_NV12) { MLOG("output format %d not implemented on the host (NV12 = 4 only)", m->format); return ST_UNSUPPORTED; } + const char *mime = NULL; int w = 0, h = 0; + md_find_string(format, "mime", &mime); md_find_int32(format, "width", &w); md_find_int32(format, "height", &h); + if (!mime || strcmp(mime, "video/x-vnd.on2.vp9") || w <= 0 || h <= 0 || w > 8192 || h > 8192 || (w | h) & 1) { + MLOG("configure: mime %s %dx%d not supported", mime ? mime : "(none)", w, h); return ST_UNSUPPORTED; + } + pthread_mutex_lock(&m->mu); + release_codec(m); m->w = w; m->h = h; + vpx_codec_dec_cfg_t cfg = {.threads = 4, .w = (unsigned)w, .h = (unsigned)h}; + if (vpx_codec_dec_init(&m->vpx, vpx_codec_vp9_dx(), &cfg, 0) != VPX_CODEC_OK) runtime_fail("libvpx VP9 decoder initialisation"); + m->vpx_live = 1; + for (int i = 0; i < IN_BUFS; i++) m->in[i] = in_get(); + for (int i = 0; i < OUT_BUFS; i++) if (!(m->out[i] = malloc((size_t)frame_bytes(m)))) runtime_fail("movie output buffers"); + reset_queues(m); m->format_sent = 0; m->configured = 1; m->last_error = 0; + pthread_mutex_unlock(&m->mu); + MLOG("configured %dx%d VP9, %d input buffers of %d KiB, %d output frames of %d bytes", w, h, IN_BUFS, IN_CAP >> 10, OUT_BUFS, frame_bytes(m)); + return ST_OK; +} +static int h_register_events(void *obj, void *events) { + Movie *m = find(obj); if (!m) return (int)orig(MOVIE_REGISTER_EVENTS, 2, (uint64_t[]){U(obj), U(events)}); + m->events = events; return ST_OK; +} +static void set_format(Movie *m, void *md) { + md_set_string(md, "mime", "video/raw"); + md_set_int32(md, "width", m->w); md_set_int32(md, "height", m->h); + md_set_int32(md, "stride", m->w); md_set_int32(md, "slice-height", m->h); + md_set_int32(md, "color-format", 21); /* COLOR_FormatYUV420SemiPlanar */ + md_set_rect(md, "crop", 0, 0, m->w - 1, m->h - 1); + md_set_int32(md, "crop-width", m->w); md_set_int32(md, "crop-height", m->h); + md_set_int32(md, "nv12-y-stride", m->w); md_set_int32(md, "nv12-uv-stride", m->w); + md_set_int32(md, "nv12-y-offset", 0); md_set_int32(md, "nv12-uv-offset", m->w * m->h); +} +static int h_get_output_format(void *obj, void *md) { + Movie *m = find(obj); if (!m) return (int)orig(MOVIE_GET_OUTPUT_FORMAT, 2, (uint64_t[]){U(obj), U(md)}); + if (!md) return ST_NULL; + if (!m->configured) return ST_ERROR; + set_format(m, md); return ST_OK; +} +static int h_start(void *obj) { + Movie *m = find(obj); if (!m) return (int)orig(MOVIE_START, 1, (uint64_t[]){U(obj)}); + Signals s = {0}; + pthread_mutex_lock(&m->mu); + if (!m->configured) { pthread_mutex_unlock(&m->mu); return ST_NOT_CONFIGURED; } + m->started = 1; note(m, &s, EV_INPUT); + pthread_mutex_unlock(&m->mu); fire(&s); + return ST_OK; +} +static int h_stop(void *obj) { + Movie *m = find(obj); if (!m) return (int)orig(MOVIE_STOP, 1, (uint64_t[]){U(obj)}); + pthread_mutex_lock(&m->mu); MLOG("stop after %llu frames (%llu dropped to keep time)", (unsigned long long)m->frames, (unsigned long long)m->dropped); release_codec(m); pthread_mutex_unlock(&m->mu); + return ST_OK; +} +static int h_reset(void *obj) { + Movie *m = find(obj); if (!m) return (int)orig(MOVIE_RESET, 1, (uint64_t[]){U(obj)}); + pthread_mutex_lock(&m->mu); release_codec(m); m->last_error = 0; pthread_mutex_unlock(&m->mu); + return ST_OK; +} +static int h_flush(void *obj) { + Movie *m = find(obj); if (!m) return (int)orig(MOVIE_FLUSH, 1, (uint64_t[]){U(obj)}); + pthread_mutex_lock(&m->mu); + if (m->configured) { /* drop queued work; the next input starts a new decode (after a seek: a key frame) */ + vpx_codec_destroy(&m->vpx); m->vpx_live = 0; + vpx_codec_dec_cfg_t cfg = {.threads = 4, .w = (unsigned)m->w, .h = (unsigned)m->h}; + if (vpx_codec_dec_init(&m->vpx, vpx_codec_vp9_dx(), &cfg, 0) != VPX_CODEC_OK) runtime_fail("libvpx VP9 decoder re-initialisation"); + m->vpx_live = 1; reset_queues(m); + } + pthread_mutex_unlock(&m->mu); + return ST_OK; +} +static int h_acquire_input(void *obj, int *index, int *remaining) { + Movie *m = find(obj); if (!m) return (int)orig(MOVIE_ACQUIRE_INPUT, 3, (uint64_t[]){U(obj), U(index), U(remaining)}); + if (!index || !remaining) return ST_NULL; + pthread_mutex_lock(&m->mu); + *index = -1; *remaining = 0; int st = ST_NO_INPUT; + if (m->started && m->nin_free > 0) { + *index = m->in_free[0]; memmove(m->in_free, m->in_free + 1, (size_t)--m->nin_free * sizeof m->in_free[0]); + *remaining = m->nin_free; st = ST_OK; + } + pthread_mutex_unlock(&m->mu); return st; +} +static int h_get_input(void *obj, int index, void *buf) { + Movie *m = find(obj); if (!m) return (int)orig(MOVIE_GET_INPUT, 3, (uint64_t[]){U(obj), (uint64_t)index, U(buf)}); + if (!buf) return ST_NULL; + if (index < 0 || index >= IN_BUFS || !m->in[index]) { buf_set_range(buf, 0, 0); buf_set_data(buf, NULL, 0); return ST_ERROR; } + buf_set_range(buf, 0, IN_CAP); buf_set_data(buf, m->in[index], IN_CAP); + return ST_OK; +} +static int h_send_input(void *obj, int index, int offset, int size, int64_t time, unsigned flags) { + Movie *m = find(obj); + if (!m) return (int)orig(MOVIE_SEND_INPUT, 6, (uint64_t[]){U(obj), (uint64_t)index, (uint64_t)offset, (uint64_t)size, (uint64_t)time, (uint64_t)flags}); + if (index < 0 || index >= IN_BUFS || offset < 0 || size < 0 || (int64_t)offset + size > IN_CAP) return ST_ERROR; + Signals s = {0}; + pthread_mutex_lock(&m->mu); + if (!m->started) { pthread_mutex_unlock(&m->mu); return ST_ERROR; } + Pending p = {index, offset, size, time, flags}; + if (m->eos_in) { m->in_free[m->nin_free++] = index; note(m, &s, EV_INPUT); } /* after end of stream: hand the buffer back */ + else if (m->npending || !decode_one(m, &s, &p)) m->pending[m->npending++] = p; + pthread_mutex_unlock(&m->mu); fire(&s); + return ST_OK; +} +static int h_acquire_output(void *obj, int *index, int64_t *time, unsigned *flags, int *remaining) { + Movie *m = find(obj); if (!m) return (int)orig(MOVIE_ACQUIRE_OUTPUT, 5, (uint64_t[]){U(obj), U(index), U(time), U(flags), U(remaining)}); + if (!index || !time || !flags || !remaining) return ST_NULL; + pthread_mutex_lock(&m->mu); /* paced frames become ready only at presents (movie_host_tick): one per game frame */ + *index = -1; *time = 0; *flags = 0; *remaining = 0; int st = ST_NO_OUTPUT; + if (m->nready > 0) { + Ready r = m->ready[0]; memmove(m->ready, m->ready + 1, (size_t)--m->nready * sizeof m->ready[0]); + *index = r.index; *time = r.time; *flags = r.flags; *remaining = m->nready; st = ST_OK; + static int said; if (said++ < 2) MLOG("output %d handed to the game (%lld us, flags %u)", r.index, (long long)r.time, r.flags); + } + pthread_mutex_unlock(&m->mu); + return st; +} +static int h_get_output(void *obj, int index, void *buf) { + Movie *m = find(obj); if (!m) return (int)orig(MOVIE_GET_OUTPUT, 3, (uint64_t[]){U(obj), (uint64_t)index, U(buf)}); + if (!buf) return ST_NULL; + buf_set_range(buf, 0, 0); + /* [port] the guest calls run before the lock: the decoder's lock is never held across guest code */ + uint8_t *dst = buf_base(buf); size_t cap = buf_capacity(buf); + pthread_mutex_lock(&m->mu); + int st = ST_ERROR, bytes = 0; + if (index >= 0 && index < OUT_BUFS && m->out_held[index]) { + bytes = m->out_bytes[index]; + if (!bytes) st = ST_OK; + else if (dst && cap >= (size_t)bytes) { memcpy(dst, m->out[index], (size_t)bytes); st = ST_OK; } + else MLOG("output buffer of %zu bytes for a %d-byte frame", cap, bytes); + } + pthread_mutex_unlock(&m->mu); + if (st == ST_OK && bytes) buf_set_range(buf, 0, bytes); + static int said; if (said++ < 2) MLOG("output %d: %d bytes copied to %p (capacity %zu), status %#x", index, bytes, (void *)dst, cap, st); + return st; +} +static int h_get_output_size(void *obj, int *size, int *count) { + Movie *m = find(obj); if (!m) return (int)orig(MOVIE_GET_OUTPUT_SIZE, 3, (uint64_t[]){U(obj), U(size), U(count)}); + if (!size || !count) return ST_NULL; + *count = 1; *size = m->configured ? frame_bytes(m) : 0; + return ST_OK; +} +static int h_release_output(void *obj, int index) { + Movie *m = find(obj); if (!m) return (int)orig(MOVIE_RELEASE_OUTPUT, 2, (uint64_t[]){U(obj), (uint64_t)index}); + Signals s = {0}; + pthread_mutex_lock(&m->mu); + int st = ST_ERROR; + if (index >= 0 && index < OUT_BUFS && m->out_held[index]) { + int queued = 0; for (int i = 0; i < m->nready; i++) queued |= m->ready[i].index == index; + for (int i = 0; i < m->ndecoded; i++) queued |= m->decoded[i].index == index; + if (!queued) { m->out_held[index] = 0; drain_pending(m, &s); st = ST_OK; } + } + pthread_mutex_unlock(&m->mu); fire(&s); + return st; +} +static int h_get_last_error(void *obj) { + Movie *m = find(obj); if (!m) return (int)orig(MOVIE_GET_LAST_ERROR, 1, (uint64_t[]){U(obj)}); + return m->last_error; +} +static int h_get_output_buffer_format(void *obj, int index, void *md) { + Movie *m = find(obj); if (!m) return (int)orig(MOVIE_GET_OUTPUT_BUFFER_FORMAT, 3, (uint64_t[]){U(obj), (uint64_t)index, U(md)}); + if (!md) return ST_NULL; + if (index < 0 || index >= OUT_BUFS || !m->out_held[index]) return ST_ERROR; + set_format(m, md); return ST_OK; +} +static int h_get_output_md(void *obj, int index, void *buf, void *md) { + Movie *m = find(obj); if (!m) return (int)orig(MOVIE_GET_OUTPUT_MD, 4, (uint64_t[]){U(obj), (uint64_t)index, U(buf), U(md)}); + if (!buf && !md) return ST_NULL; + int st = md ? h_get_output_buffer_format(obj, index, md) : ST_OK; + return st == ST_OK && buf ? h_get_output(obj, index, buf) : st; +} + +/* [port] bound by name: the vtable slots, main's imports and the library's own calls (loader.c resolve) */ +HLE_NATIVE("_ZN5movie7DecoderC1ENS_11DecoderTypeENS_19DecoderOutputFormatENS_11DecoderModeEi", h_ctor, 'v') +HLE_NATIVE("_ZN5movie7DecoderC2ENS_11DecoderTypeENS_19DecoderOutputFormatENS_11DecoderModeEi", h_ctor, 'v') +HLE_NATIVE("_ZN5movie7DecoderD2Ev", h_d2, 'v') +HLE_NATIVE("_ZN5movie7DecoderD1Ev", h_d2, 'v') +HLE_NATIVE("_ZN5movie7DecoderD0Ev", h_d0, 'v') +HLE_NATIVE("_ZN5movie7Decoder9ConfigureEPNS_9MediaDataE", h_configure, 'w') +HLE_NATIVE("_ZN5movie7Decoder20RegisterClientEventsEPNS_13DecoderEventsE", h_register_events, 'w') +HLE_NATIVE("_ZN5movie7Decoder15GetOutputFormatEPNS_9MediaDataE", h_get_output_format, 'w') +HLE_NATIVE("_ZN5movie7Decoder5StartEv", h_start, 'w') +HLE_NATIVE("_ZN5movie7Decoder4StopEv", h_stop, 'w') +HLE_NATIVE("_ZN5movie7Decoder5ResetEv", h_reset, 'w') +HLE_NATIVE("_ZN5movie7Decoder5FlushEv", h_flush, 'w') +HLE_NATIVE("_ZN5movie7Decoder23AcquireInputBufferIndexEPiS1_", h_acquire_input, 'w') +HLE_NATIVE("_ZN5movie7Decoder14GetInputBufferEiPNS_6BufferE", h_get_input, 'w') +HLE_NATIVE("_ZN5movie7Decoder24SendInputBufferForDecodeEiiilj", h_send_input, 'w') +HLE_NATIVE("_ZN5movie7Decoder24AcquireOutputBufferIndexEPiPlPjS1_", h_acquire_output, 'w') +HLE_NATIVE("_ZN5movie7Decoder15GetOutputBufferEiPNS_6BufferE", h_get_output, 'w') +HLE_NATIVE("_ZN5movie7Decoder19GetOutputBufferSizeEPiS1_", h_get_output_size, 'w') +HLE_NATIVE("_ZN5movie7Decoder24ReleaseOutputBufferIndexEi", h_release_output, 'w') +HLE_NATIVE("_ZN5movie7Decoder12GetLastErrorEv", h_get_last_error, 'w') +HLE_NATIVE("_ZN5movie7Decoder15GetOutputBufferEiPNS_6BufferEPNS_9MediaDataE", h_get_output_md, 'w') +HLE_NATIVE("_ZN5movie7Decoder21GetOutputBufferFormatEiPNS_9MediaDataE", h_get_output_buffer_format, 'w') diff --git a/src/host/opus_codec.c b/src/host/opus_codec.c new file mode 100644 index 0000000..df13c99 --- /dev/null +++ b/src/host/opus_codec.c @@ -0,0 +1,143 @@ +/* From MacNativeGames games/totk/runtime/opus_codec.c @b75be8c (MIT). [port] marks the Windows changes. */ +/* nn::codec hardware Opus decoder, implemented with libopus on the host (the hwopus service is a + hardware-accelerated libopus). Input frames carry Nintendo's 8-byte big-endian header + {payload size, final range}; the decoder's final range is checked against it. A header range of 0 is not a + recorded value: none of the 280,723 packets in 1.4.3's English voice streams has it, while a fan dub's encoder + writes 0 into every packet (all 186,758 of the PKG PT-BR dub, which decode without error). Those are not checked. */ +#include "host.h" // [port] runtime.h +#include "../runtime/hle.h" +#include // [port] MSYS2 layout +#include +#include +#include +#include +#include + +#define R_INVALID 0x1fcd1u /* codec: invalid argument (module 111) */ +/* A fixed-capacity host table could terminate a running game when another + * stream opened. Allocate one stable node per live decoder and free it at + * Finalize; growing the registry must not move a decoder another thread uses. + * Registry -> decoder is the lock order. A caller acquires the decoder lock + * before dropping the registry lock, so Finalize cannot free a borrowed node. + * Different streams can still decode in parallel. */ +typedef struct Decoder { + const void *guest; + OpusDecoder *dec; + int rate, channels; + struct Decoder *next; + pthread_mutex_t use_lock; +} Decoder; +static Decoder *decoders; +static size_t decoder_count, decoder_high_water, decoder_created, decoder_finalized, decoder_reinitialized; +static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER; + +static void decoder_stats(const char *event) { /* registry lock held */ + fprintf(stderr, "[totk-audio] Opus %s: active %zu, peak %zu, created %zu, finalized %zu, reinitialized %zu (dynamic registry)\n", + event, decoder_count, decoder_high_water, decoder_created, decoder_finalized, decoder_reinitialized); +} +static int codec_trace(void) { + const char *v = getenv("TOTK_CODEC_TRACE"); + return v && *v && strcmp(v, "0"); +} +static Decoder *acquire(const void *guest) { + pthread_mutex_lock(&lock); + Decoder *d = decoders; + while (d && d->guest != guest) d = d->next; + if (d) pthread_mutex_lock(&d->use_lock); + pthread_mutex_unlock(&lock); + if (!d) runtime_fail("unknown hardware Opus decoder"); + return d; +} +static void release(Decoder *d) { pthread_mutex_unlock(&d->use_lock); } +static int valid_format(int rate, int channels) { + return (rate == 48000 || rate == 24000 || rate == 16000 || rate == 12000 || rate == 8000) && (channels == 1 || channels == 2); +} +static size_t work_size(int rate, int channels, int option) { + (void)option; if (!valid_format(rate, channels)) return 0; + return (size_t)opus_decoder_get_size(channels) + 0x1000; /* host keeps the state; the game's buffer is reserved but unused */ +} +static unsigned initialize(void *guest, int rate, int channels, void *work, size_t size, int option) { + (void)work; (void)option; + if (!guest || !valid_format(rate, channels) || size < work_size(rate, channels, 0)) return R_INVALID; + Decoder *d = calloc(1, sizeof *d); + if (!d) runtime_fail("hardware Opus decoder allocation"); + int err = 0; + d->dec = opus_decoder_create(rate, channels, &err); + if (!d->dec || err != OPUS_OK) { + if (d->dec) opus_decoder_destroy(d->dec); + free(d); runtime_fail("libopus decoder creation"); + } + if (pthread_mutex_init(&d->use_lock, NULL)) { + opus_decoder_destroy(d->dec); free(d); runtime_fail("hardware Opus decoder mutex"); + } + d->guest = guest; d->rate = rate; d->channels = channels; + pthread_mutex_lock(&lock); + for (Decoder *old = decoders; old; old = old->next) if (old->guest == guest) { + /* Preserve #18: the game may initialize the same object without + * Finalize. Replace its state, waiting for any in-flight operation, + * rather than adding another entry for the same guest object. */ + pthread_mutex_lock(&old->use_lock); + OpusDecoder *previous = old->dec; + old->dec = d->dec; old->rate = rate; old->channels = channels; + decoder_reinitialized++; + if (decoder_reinitialized == 1) + fprintf(stderr, "[totk] hardware Opus decoder %p initialized again without Finalize: its slot is reused (logged once)\n", guest); + if (codec_trace()) decoder_stats("reinitialize"); + pthread_mutex_unlock(&lock); + opus_decoder_destroy(previous); release(old); + pthread_mutex_destroy(&d->use_lock); free(d); + return 0; + } + d->next = decoders; decoders = d; decoder_count++; decoder_created++; + int peak = decoder_count > decoder_high_water; + if (peak) decoder_high_water = decoder_count; + /* past the old fixed 64: at 65 and every power of two, so a leak shows in a normal log */ + if (codec_trace() || (peak && decoder_count > 64 && (decoder_count == 65 || !(decoder_count & (decoder_count - 1))))) + decoder_stats("initialize"); + pthread_mutex_unlock(&lock); + return 0; +} +static void finalize(void *guest) { + pthread_mutex_lock(&lock); + Decoder **slot = &decoders; + while (*slot && (*slot)->guest != guest) slot = &(*slot)->next; + Decoder *d = *slot; + if (!d) { pthread_mutex_unlock(&lock); runtime_fail("unknown hardware Opus decoder"); } + pthread_mutex_lock(&d->use_lock); + *slot = d->next; decoder_count--; decoder_finalized++; + if (codec_trace()) decoder_stats("finalize"); + pthread_mutex_unlock(&lock); + opus_decoder_destroy(d->dec); + release(d); pthread_mutex_destroy(&d->use_lock); free(d); +} +static int sample_rate(const void *guest) { Decoder *d = acquire(guest); int v = d->rate; release(d); return v; } +static int channel_count(const void *guest) { Decoder *d = acquire(guest); int v = d->channels; release(d); return v; } +static unsigned reset(void *guest) { + Decoder *d = acquire(guest); int ok = opus_decoder_ctl(d->dec, OPUS_RESET_STATE); release(d); + return ok == OPUS_OK ? 0 : R_INVALID; +} +static uint32_t be32(const uint8_t *p) { return (uint32_t)p[0] << 24 | (uint32_t)p[1] << 16 | (uint32_t)p[2] << 8 | p[3]; } +static unsigned decode(void *guest, size_t *consumed, int *samples, int16_t *out, size_t out_size, const uint8_t *in, size_t in_size) { + if (!consumed || !samples || !out || !in || in_size < 8) return R_INVALID; + uint32_t n = be32(in), range = be32(in + 4); + if (n > in_size - 8 || n > INT_MAX) return R_INVALID; + Decoder *d = acquire(guest); + size_t capacity = out_size / (sizeof(int16_t) * (size_t)d->channels); + if (capacity > INT_MAX) { release(d); return R_INVALID; } + int max = (int)capacity; + guest_prepare_write(out, out_size); + int got = opus_decode(d->dec, in + 8, (opus_int32)n, out, max, 0); + if (got < 0) { fprintf(stderr, "[totk] opus_decode: %s\n", opus_strerror(got)); release(d); return R_INVALID; } + opus_uint32 final = 0; opus_decoder_ctl(d->dec, OPUS_GET_FINAL_RANGE(&final)); + if (n && range && final != range) { fprintf(stderr, "[totk] Opus final range %#x != stream %#x\n", final, range); runtime_fail("Opus stream integrity"); } + *consumed = 8 + n; *samples = got; release(d); return 0; +} + +// [port] MacNativeGames' runtime_codec_override table as HLE bindings +HLE_NATIVE("_ZN2nn5codec36GetHardwareOpusDecoderWorkBufferSizeEiii", work_size, 'x') +HLE_NATIVE("_ZN2nn5codec29InitializeHardwareOpusDecoderEPNS0_23HardwareOpusDecoderTypeEiiPvmi", initialize, 'w') +HLE_NATIVE("_ZN2nn5codec27FinalizeHardwareOpusDecoderEPNS0_23HardwareOpusDecoderTypeE", finalize, 'v') +HLE_NATIVE("_ZN2nn5codec32GetHardwareOpusDecoderSampleRateEPKNS0_23HardwareOpusDecoderTypeE", sample_rate, 'w') +HLE_NATIVE("_ZN2nn5codec34GetHardwareOpusDecoderChannelCountEPKNS0_23HardwareOpusDecoderTypeE", channel_count, 'w') +HLE_NATIVE("_ZN2nn5codec24ResetHardwareOpusDecoderEPNS0_23HardwareOpusDecoderTypeE", reset, 'w') +HLE_NATIVE("_ZN2nn5codec33DecodeOpusInterleavedWithHardwareEPNS0_23HardwareOpusDecoderTypeEPmPiPsmPKvm", decode, 'w') diff --git a/src/host/overrides_applet.c b/src/host/overrides_applet.c new file mode 100644 index 0000000..79ed853 --- /dev/null +++ b/src/host/overrides_applet.c @@ -0,0 +1,127 @@ +/* Host side of applet, telemetry, NFC and error-display functions the game reaches. + * Ported from MacNativeGames games/totk/runtime/applet_host.c @b75be8c (MIT): client-side SDK code (PlayReport + * encoding, keyboard presets, diag logging) runs unchanged; only the calls that would talk to system services are + * taken here. */ +#include +#include +#include + +#include "host.h" +#include "../runtime/hle.h" + +/* ---------- nn::oe / nn::applet ---------- */ +static float balance[2] = {1.0f, 1.0f}; +static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER; +static pthread_cond_t message_cv = PTHREAD_COND_INITIALIZER; +static uint32_t messages[32]; +static unsigned head, tail; +static uint64_t start_ns; +__attribute__((constructor)) static void started(void) { start_ns = runtime_now_ns(); } + +void runtime_oe_push_message(uint32_t message) { + pthread_mutex_lock(&lock); + if (tail - head < 32) { messages[tail++ % 32] = message; pthread_cond_broadcast(&message_cv); } + pthread_mutex_unlock(&lock); +} +static void get_balance(float *l, float *r) { + pthread_mutex_lock(&lock); + if (l) *l = balance[0]; + if (r) *r = balance[1]; + pthread_mutex_unlock(&lock); +} +static int64_t total_active_time(void) { return (int64_t)(runtime_now_ns() - start_ns); } /* TimeSpan in ns */ +static void inactivity_extended(unsigned char on) { (void)on; } /* the host never sleeps the console */ +static void finish_startup_logo(void) { fprintf(stderr, "[totk] game finished startup logo\n"); } +/* The executable was launched directly: no predecessor in an ExecuteProgram chain (-1 keeps the normal path). */ +static int previous_program_index(void) { return -1; } +/* A stable, nonzero resource id for this process's one application. */ +static uint64_t applet_resource_user_id(void) { return 1; } +static unsigned char try_pop_launch_parameter(uint64_t *size, void *buffer, uint64_t capacity) { + (void)buffer; (void)capacity; + if (size) *size = 0; + return 0; +} +static unsigned char try_pop(uint32_t *out) { + pthread_mutex_lock(&lock); + int ok = head != tail; + if (ok && out) *out = messages[head++ % 32]; + pthread_mutex_unlock(&lock); + return (unsigned char)ok; +} +int runtime_core_away(void); +void runtime_core_back(int *away); +static uint32_t pop(void) { + __attribute__((cleanup(runtime_core_back))) int away = runtime_core_away(); /* blocks like a Horizon wait */ + pthread_mutex_lock(&lock); + while (head == tail) pthread_cond_wait(&message_cv, &lock); + uint32_t m = messages[head++ % 32]; + pthread_mutex_unlock(&lock); + return m; +} +HLE_FUNC("_ZN2nn2oe24SetExpectedVolumeBalanceEff", set_balance) { /* float arguments: v0, v1 */ + float l, r; + memcpy(&l, &c->v[0], 4); + memcpy(&r, &c->v[1], 4); + if (l < 0 || l > 1 || r < 0 || r > 1) runtime_fail("volume balance out of range"); + pthread_mutex_lock(&lock); + balance[0] = l; balance[1] = r; + pthread_mutex_unlock(&lock); +} + +/* ---------- nn::prepo: play reports are not transmitted (no Nintendo servers) ---------- */ +static uint64_t reports; +static unsigned report_save(const void *report) { + (void)report; + if (!(reports++ % 100)) fprintf(stderr, "[totk] play report kept local (not transmitted), count %llu\n", (unsigned long long)reports); + return 0; +} +static unsigned report_save_uid(const void *report, const void *uid) { (void)uid; return report_save(report); } +static unsigned immediate_transmission(void) { return 0; } +static unsigned connection_status(void *report, const char *a, const char *b, const char *c) { (void)report; (void)a; (void)b; (void)c; return 0; } + +/* ---------- nn::nfp: the host has no NFC reader ---------- */ +#define NFP_DEVICE_NOT_FOUND 0x8073u /* nfp module 115, description 64 */ +static int nfp_state; +static unsigned nfp_initialize(void) { nfp_state = 1; return 0; } +static unsigned nfp_finalize(void) { nfp_state = 0; return 0; } +static int nfp_get_state(void) { return nfp_state; } +static unsigned nfp_list(void *handles, int *count, int max) { (void)handles; (void)max; if (count) *count = 0; return NFP_DEVICE_NOT_FOUND; } +static unsigned nfp_attach_availability(void *event) { if (!event) runtime_fail("nfp availability event"); hle_bind_system_event(event, 0); return 0; } +static unsigned nfp_no_device(void) { return NFP_DEVICE_NOT_FOUND; } +static int nfp_device_state(const void *h) { (void)h; return 0; } + +/* ---------- nn::err ---------- */ +static void show_error(uint32_t result) { + fprintf(stderr, "[totk] game error dialog: %04u-%04u (result %#x)\n", 2000 + (result & 0x1ff), (result >> 9) & 0x1fff, result); +} + +HLE_NATIVE("_ZN2nn6applet23GetPreviousProgramIndexEv", previous_program_index, 'w') +HLE_NATIVE("_ZN2nn6applet23GetAppletResourceUserIdEv", applet_resource_user_id, 'x') +HLE_NATIVE("_ZN2nn2am23GetAppletResourceUserIdEv", applet_resource_user_id, 'x') +HLE_NATIVE("_ZN2nn2oe21TryPopLaunchParameterEPmPvm", try_pop_launch_parameter, 'b') +HLE_NATIVE("_ZN2nn2oe24GetExpectedVolumeBalanceEPfS1_", get_balance, 'v') +HLE_NATIVE("_ZN2nn2oe25GetProgramTotalActiveTimeEv", total_active_time, 'x') +HLE_NATIVE("_ZN2nn2oe38SetUserInactivityDetectionTimeExtendedEb", inactivity_extended, 'v') +HLE_NATIVE("_ZN2nn2oe44SetUserInactivityDetectionTimeExtendedUnsafeEb", inactivity_extended, 'v') +HLE_NATIVE("_ZN2nn2oe17FinishStartupLogoEv", finish_startup_logo, 'v') +HLE_NATIVE("_ZN2nn2oe25TryPopNotificationMessageEPj", try_pop, 'b') +HLE_NATIVE("_ZN2nn2oe22PopNotificationMessageEv", pop, 'w') +HLE_NATIVE("_ZN2nn5prepo10PlayReport4SaveEv", report_save, 'w') +HLE_NATIVE("_ZN2nn5prepo10PlayReport4SaveERKNS_7account3UidE", report_save_uid, 'w') +HLE_NATIVE("_ZN2nn5prepo28RequestImmediateTransmissionEv", immediate_transmission, 'w') +HLE_NATIVE("_ZN2nn5prepo27AddInternetConnectionStatusEPNS0_10PlayReportEPKcS4_S4_", connection_status, 'w') +HLE_NATIVE("_ZN2nn3nfp10InitializeEv", nfp_initialize, 'w') +HLE_NATIVE("_ZN2nn3nfp8FinalizeEv", nfp_finalize, 'w') +HLE_NATIVE("_ZN2nn3nfp8GetStateEv", nfp_get_state, 'w') +HLE_NATIVE("_ZN2nn3nfp11ListDevicesEPNS0_12DeviceHandleEPii", nfp_list, 'w') +HLE_NATIVE("_ZN2nn3nfp29AttachAvailabilityChangeEventEPNS_2os15SystemEventTypeE", nfp_attach_availability, 'w') +HLE_NATIVE("_ZN2nn3nfp14GetDeviceStateERKNS0_12DeviceHandleE", nfp_device_state, 'w') +HLE_NATIVE("_ZN2nn3nfp5MountERKNS0_12DeviceHandleENS0_9ModelTypeENS0_11MountTargetE", nfp_no_device, 'w') +HLE_NATIVE("_ZN2nn3nfp9GetNpadIdEPjRKNS0_12DeviceHandleE", nfp_no_device, 'w') +HLE_NATIVE("_ZN2nn3nfp19AttachActivateEventEPNS_2os15SystemEventTypeERKNS0_12DeviceHandleE", nfp_no_device, 'w') +HLE_NATIVE("_ZN2nn3nfp21AttachDeactivateEventEPNS_2os15SystemEventTypeERKNS0_12DeviceHandleE", nfp_no_device, 'w') +HLE_NATIVE("_ZN2nn3nfp14StartDetectionERKNS0_12DeviceHandleE", nfp_no_device, 'w') +HLE_NATIVE("_ZN2nn3nfp13StopDetectionERKNS0_12DeviceHandleE", nfp_no_device, 'w') +HLE_NATIVE("_ZN2nn3nfp10GetTagInfoEPNS0_7TagInfoERKNS0_12DeviceHandleE", nfp_no_device, 'w') +HLE_NATIVE("_ZN2nn3nfp12GetModelInfoEPNS0_9ModelInfoERKNS0_12DeviceHandleE", nfp_no_device, 'w') +HLE_NATIVE("_ZN2nn3err9ShowErrorENS_6ResultE", show_error, 'v') diff --git a/src/host/overrides_input.c b/src/host/overrides_input.c new file mode 100644 index 0000000..cacac88 --- /dev/null +++ b/src/host/overrides_input.c @@ -0,0 +1,204 @@ +/* nn::hid: one full-key controller (docked) or the handheld controller, fed by the keyboard and game controllers. + * Ported from MacNativeGames games/totk/runtime/input.c @b75be8c (MIT); state ABI follows libnx HID. Changes: key codes + * stay MacNativeGames' (Mac virtual key codes) and the window layer translates its own keys into them; the game + * controller source is runtime_gamepad_read (none until the window / SDL3 layer of milestone 3.2); the diagnostic input + * script / file commands are not ported. */ +#include +#include +#include +#include +#include + +#include "host.h" +#include "../runtime/hle.h" + +typedef struct { int64_t sampling; uint64_t buttons; int32_t lx, ly, rx, ry; uint32_t attributes, pad; } NpadState; +_Static_assert(sizeof(NpadState) == 40, "Npad full-key ABI"); + +int runtime_handheld_mode(void); + +static pthread_mutex_t input_lock = PTHREAD_MUTEX_INITIALIZER; +static unsigned char keys[128]; +static uint64_t pending; +static uint32_t styles = 1; +static bool enabled, allowed = true; +static int64_t sample; + +/* Mac virtual key codes -> Switch buttons (MacNativeGames' layout: J=A, K/Space=B, I=X, U=Y, Z/C=L/R ...). */ +static uint64_t button(unsigned code) { + switch (code) { + case 38: return 1; case 40: case 49: return 2; case 34: return 4; case 32: return 8; case 6: return 16; case 8: return 32; + case 12: return 64; case 14: return 128; case 58: return 256; case 56: return 512; case 36: return 1024; + case 48: case 53: return 2048; case 3: return 4096; case 17: return 8192; case 4: return 16384; case 5: return 32768; + default: return 0; + } +} + +/* Game controllers: filled by the window layer (SDL3, milestone 3.2). */ +static RuntimeGamepadState g_pad; +void runtime_gamepad_set(const RuntimeGamepadState *s) { pthread_mutex_lock(&input_lock); g_pad = *s; pthread_mutex_unlock(&input_lock); } +static void gamepad_read(RuntimeGamepadState *out) { *out = g_pad; } + +void runtime_input_key(unsigned code, int down) { + if (code >= 128) return; + pthread_mutex_lock(&input_lock); + if (down && !keys[code]) pending |= button(code); + keys[code] = (unsigned char)!!down; + pthread_mutex_unlock(&input_lock); +} +void runtime_input_clear(void) { + pthread_mutex_lock(&input_lock); + memset(keys, 0, sizeof keys); + pending = 0; + memset(&g_pad, 0, sizeof g_pad); + pthread_mutex_unlock(&input_lock); +} + +/* TOTK_INPUT_SCRIPT=:[*],... (development: drive the game without a person, e.g. + "520:J,700:J*30,900:W*120"). Frames are the graphics frontend's frames; keys are named as on the keyboard + (A-Z, RETURN, SPACE, TAB, ESCAPE, UP, DOWN, LEFT, RIGHT, LSHIFT, LALT) and go through the same layout as typing. */ +static int script_code(const char *k, size_t n) { + static const struct { const char *name; int code; } named[] = { + {"RETURN", 36}, {"SPACE", 49}, {"TAB", 48}, {"ESCAPE", 53}, {"UP", 126}, {"DOWN", 125}, {"LEFT", 123}, + {"RIGHT", 124}, {"LSHIFT", 56}, {"LALT", 58}}; + static const int letters[26] = {0, 11, 8, 2, 14, 3, 5, 4, 34, 38, 40, 37, 46, 45, 31, 35, 12, 15, 1, 17, 32, 9, 13, 7, 16, 6}; + if (n == 1 && k[0] >= 'A' && k[0] <= 'Z') return letters[k[0] - 'A']; + for (unsigned i = 0; i < sizeof named / sizeof *named; i++) + if (strlen(named[i].name) == n && !strncmp(named[i].name, k, n)) return named[i].code; + return -1; +} +void runtime_input_script_tick(uint64_t frame) { + static const char *script; + static int init; + if (!init) { init = 1; script = getenv("TOTK_INPUT_SCRIPT"); } + for (const char *p = script; p && *p;) { + char *end; + uint64_t at = strtoull(p, &end, 10); + const char *k = *end == ':' ? end + 1 : end; + size_t n = strcspn(k, "*,"); + uint64_t hold = k[n] == '*' ? strtoull(k + n + 1, NULL, 10) : 6; + int code = script_code(k, n); + if (code >= 0 && frame == at) { fprintf(stderr, "[totk] input script: frame %llu press %.*s\n", (unsigned long long)frame, (int)n, k); runtime_input_key((unsigned)code, 1); } + if (code >= 0 && frame == at + hold) runtime_input_key((unsigned)code, 0); + p = strchr(k, ','); + if (p) p++; + } +} + +static void initialize(void) { + pthread_mutex_lock(&input_lock); enabled = true; pthread_mutex_unlock(&input_lock); + fprintf(stderr, "[totk] keyboard / game controller input initialized\n"); +} +/* Docked: one full-key controller as player 1 (id 0, style bit 0). Handheld (TOTK_OPERATION_MODE=handheld): the same + input is the handheld controller (id 0x20, style bit 1) and player 1 is not connected. */ +static uint32_t pad_id(void) { return runtime_handheld_mode() ? 0x20u : 0u; } +static uint32_t pad_style(void) { return runtime_handheld_mode() ? 2u : 1u; } +static void supported_ids(const uint32_t *ids, uint64_t n) { + if (n > 10 || (!ids && n)) runtime_fail("invalid controller ids"); + pthread_mutex_lock(&input_lock); + allowed = false; + for (uint64_t i = 0; i < n; i++) if (ids[i] == pad_id()) allowed = true; + pthread_mutex_unlock(&input_lock); +} +static void supported_styles(uint32_t value) { pthread_mutex_lock(&input_lock); styles = value; pthread_mutex_unlock(&input_lock); } +static uint32_t style(const uint32_t *id) { + if (!id) runtime_fail("controller id pointer"); + pthread_mutex_lock(&input_lock); + uint32_t v = enabled && allowed && *id == pad_id() ? (styles & pad_style()) : 0; + pthread_mutex_unlock(&input_lock); + return v; +} +static int pad_states(NpadState *out, int max, const uint32_t *id, uint32_t want_style) { + if (max <= 0) return 0; + if (!out || !id) runtime_fail("controller output pointer"); + pthread_mutex_lock(&input_lock); + NpadState state = {.sampling = ++sample}; + if (enabled && allowed && *id == pad_id() && want_style == pad_style() && (styles & want_style)) { + state.attributes = 1; + for (unsigned i = 0; i < 128; i++) if (keys[i]) state.buttons |= button(i); + state.buttons |= pending; + pending = 0; + state.lx = 32767 * ((int)keys[2] - (int)keys[0]); /* D / A */ + state.ly = 32767 * ((int)keys[13] - (int)keys[1]); /* W / S */ + state.rx = 32767 * ((int)keys[124] - (int)keys[123]); /* arrow right / left */ + state.ry = 32767 * ((int)keys[126] - (int)keys[125]); /* arrow up / down */ + RuntimeGamepadState pad; + gamepad_read(&pad); + state.buttons |= pad.buttons; + if (abs(pad.lx) > abs(state.lx)) state.lx = pad.lx; + if (abs(pad.ly) > abs(state.ly)) state.ly = pad.ly; + if (abs(pad.rx) > abs(state.rx)) state.rx = pad.rx; + if (abs(pad.ry) > abs(state.ry)) state.ry = pad.ry; + } + *out = state; + pthread_mutex_unlock(&input_lock); + return 1; +} +static int full_states(NpadState *out, int max, const uint32_t *id) { return pad_states(out, max, id, 1); } +static int handheld_states(NpadState *out, int max, const uint32_t *id) { return pad_states(out, max, id, 2); } +static int disconnected_states(NpadState *out, int max, const uint32_t *id) { + (void)id; + if (max <= 0) return 0; + if (!out) runtime_fail("controller output pointer"); + pthread_mutex_lock(&input_lock); + *out = (NpadState){.sampling = ++sample}; + pthread_mutex_unlock(&input_lock); + return 1; +} +static void hold_type(int type) { if (type < 0 || type > 1) runtime_fail("invalid joy hold type"); } +static void disconnect(const uint32_t *id) { + if (!id) runtime_fail("disconnect id"); + if (*id == pad_id()) { /* only the pad in use: in handheld mode TOTK disconnects player 1 */ + pthread_mutex_lock(&input_lock); allowed = false; pthread_mutex_unlock(&input_lock); + runtime_input_clear(); + } +} + +/* Mouse and keyboard devices: the keyboard drives the controller, so the game sees no mouse and a keyboard with no keys. */ +typedef struct { int64_t sampling; int32_t x, y, dx, dy, wx, wy; uint32_t buttons, attributes; } MouseState; +typedef struct { int64_t sampling; uint32_t modifiers, attributes; uint64_t keys[4]; } KeyboardState; +_Static_assert(sizeof(MouseState) == 40 && sizeof(KeyboardState) == 48, "mouse/keyboard ABI"); +static bool mouse_ready, keyboard_ready; +static void initialize_mouse(void) { pthread_mutex_lock(&input_lock); mouse_ready = true; pthread_mutex_unlock(&input_lock); } +static void initialize_keyboard(void) { pthread_mutex_lock(&input_lock); keyboard_ready = true; pthread_mutex_unlock(&input_lock); } +static int mouse_states(MouseState *out, int max) { + if (max <= 0) return 0; + if (!out) runtime_fail("mouse output pointer"); + pthread_mutex_lock(&input_lock); + if (!mouse_ready) runtime_fail("mouse used before InitializeMouse"); + *out = (MouseState){.sampling = ++sample}; + pthread_mutex_unlock(&input_lock); + return 1; +} +static void keyboard_state(KeyboardState *out) { + if (!out) runtime_fail("keyboard output pointer"); + pthread_mutex_lock(&input_lock); + if (!keyboard_ready) runtime_fail("keyboard used before InitializeKeyboard"); + *out = (KeyboardState){.sampling = ++sample}; + pthread_mutex_unlock(&input_lock); +} +/* Controller support applet: exactly one controller here, so it returns at once (one player, on the controller in use). */ +typedef struct { int8_t players; uint8_t pad[3]; uint32_t selected; } ControllerSupportResult; +_Static_assert(sizeof(ControllerSupportResult) == 8, "controller support result ABI"); +static uint32_t show_controller_support(ControllerSupportResult *out, const void *arg) { + (void)arg; + if (out) *out = (ControllerSupportResult){.players = 1, .selected = pad_id()}; + return 0; +} + +HLE_NATIVE("_ZN2nn3hid14InitializeNpadEv", initialize, 'v') +HLE_NATIVE("_ZN2nn3hid22SetSupportedNpadIdTypeEPKjm", supported_ids, 'v') +HLE_NATIVE("_ZN2nn3hid24SetSupportedNpadStyleSetENS_4util10BitFlagSetILi32ENS0_12NpadStyleTagEEE", supported_styles, 'v') +HLE_NATIVE("_ZN2nn3hid15GetNpadStyleSetERKj", style, 'w') +HLE_NATIVE("_ZN2nn3hid13GetNpadStatesEPNS0_16NpadFullKeyStateEiRKj", full_states, 'w') +HLE_NATIVE("_ZN2nn3hid13GetNpadStatesEPNS0_16NpadJoyDualStateEiRKj", disconnected_states, 'w') +HLE_NATIVE("_ZN2nn3hid13GetNpadStatesEPNS0_16NpadJoyLeftStateEiRKj", disconnected_states, 'w') +HLE_NATIVE("_ZN2nn3hid13GetNpadStatesEPNS0_17NpadJoyRightStateEiRKj", disconnected_states, 'w') +HLE_NATIVE("_ZN2nn3hid13GetNpadStatesEPNS0_17NpadHandheldStateEiRKj", handheld_states, 'w') +HLE_NATIVE("_ZN2nn3hid18SetNpadJoyHoldTypeENS0_15NpadJoyHoldTypeE", hold_type, 'v') +HLE_NATIVE("_ZN2nn3hid14DisconnectNpadERKj", disconnect, 'v') +HLE_NATIVE("_ZN2nn3hid21ShowControllerSupportEPNS0_27ControllerSupportResultInfoERKNS0_20ControllerSupportArgE", show_controller_support, 'w') +HLE_NATIVE("_ZN2nn3hid15InitializeMouseEv", initialize_mouse, 'v') +HLE_NATIVE("_ZN2nn3hid18InitializeKeyboardEv", initialize_keyboard, 'v') +HLE_NATIVE("_ZN2nn3hid14GetMouseStatesEPNS0_10MouseStateEi", mouse_states, 'w') +HLE_NATIVE("_ZN2nn3hid16GetKeyboardStateEPNS0_13KeyboardStateE", keyboard_state, 'v') diff --git a/src/host/overrides_oe.c b/src/host/overrides_oe.c new file mode 100644 index 0000000..b3355b5 --- /dev/null +++ b/src/host/overrides_oe.c @@ -0,0 +1,107 @@ +/* Host replacements of SDK functions (application environment, add-on content, error/parental-control/diagnostics + * setup), bound by symbol name. Ported from MacNativeGames games/totk/runtime/services.c runtime_override() @b75be8c + * (MIT): the same behavior, written against the guest registers (AAPCS64: x0.. integer arguments and results). */ +#include +#include +#include +#ifdef _WIN32 +#include +#endif + +#include "host.h" +#include "../runtime/hle.h" + +/* TOTK_OPERATION_MODE=handheld reports the Switch handheld mode (oe OperationMode 0, PerformanceMode Normal 0) instead + of docked (1, Boost 1): the game then uses its own handheld settings (1280x720 output, lower internal resolution). */ +static int handheld_mode(void) { + static int v = -1; + if (v < 0) { + const char *e = getenv("TOTK_OPERATION_MODE"); + if (!e || !strcmp(e, "docked")) v = 0; + else if (!strcmp(e, "handheld")) v = 1; + else runtime_fail("TOTK_OPERATION_MODE must be docked or handheld"); + fprintf(stderr, "[totk] operation mode reported to the game: %s\n", v ? "handheld (performance normal)" : "docked (performance boost)"); + } + return v; +} +int runtime_handheld_mode(void) { return handheld_mode(); } + +/* The game's language: TOTK_LANG as given (a Switch language code), otherwise from TOTK_LOCALE or the user's + Windows UI language. Brazil maps to "pt-BR" in 1.4.3, whose RomFS has the official Brazilian text. */ +static uint64_t desired_language(void) { + const char *explicit_code = getenv("TOTK_LANG"); + char code[8] = {0}; + if (explicit_code) { + if (strlen(explicit_code) > 7) runtime_fail("language code too long"); + memcpy(code, explicit_code, strlen(explicit_code)); + } else { + char tag[LOCALE_NAME_MAX_LENGTH] = {0}; + const char *env = getenv("TOTK_LOCALE"); + if (env && *env) snprintf(tag, sizeof tag, "%s", env); +#ifdef _WIN32 + else { + wchar_t w[LOCALE_NAME_MAX_LENGTH]; + if (GetUserDefaultLocaleName(w, LOCALE_NAME_MAX_LENGTH)) WideCharToMultiByte(CP_UTF8, 0, w, -1, tag, sizeof tag, NULL, NULL); + } +#endif + char language[16] = {0}, region[4] = {0}; + size_t n = strcspn(tag, "-_"); + if (n < sizeof language) memcpy(language, tag, n); + for (const char *p = tag + n; *p;) { + p += strspn(p, "-_"); + size_t k = strcspn(p, "-_"); + if (k == 2 && isalpha((unsigned char)p[0]) && isalpha((unsigned char)p[1])) { + region[0] = (char)toupper((unsigned char)p[0]); + region[1] = (char)toupper((unsigned char)p[1]); + } + p += k; + } + const char *portuguese = !strcmp(region, "BR") ? "pt-BR" : "pt"; + const char *selected = !strcmp(language, "ja") ? "ja" : !strcmp(language, "de") ? "de" : !strcmp(language, "fr") ? "fr" + : !strcmp(language, "es") ? "es" : !strcmp(language, "it") ? "it" : !strcmp(language, "nl") ? "nl" + : !strcmp(language, "ru") ? "ru" : !strcmp(language, "ko") ? "ko" : !strcmp(language, "zh") ? "zh-Hans" + : !strcmp(language, "pt") ? portuguese : "en-US"; + memcpy(code, selected, strlen(selected)); + } + uint64_t result; + memcpy(&result, code, 8); + return result; +} + +static int focus_policy; +static unsigned resume_notification, operation_notification, performance_notification; +static uint32_t performance_configuration[2]; + +HLE_FUNC("_ZN2nn2oe18GetDesiredLanguageEv", oe_desired_language) { c->x[0] = desired_language(); } +HLE_FUNC("_ZN2nn2oe27SetPerformanceConfigurationENS0_15PerformanceModeEi", oe_performance_config) { + int mode = (int)c->x[0]; + if (mode < 0 || mode > 1) runtime_fail("invalid performance mode"); + performance_configuration[mode] = (uint32_t)c->x[1]; + fprintf(stderr, "[totk] performance request mode=%d cfg=%x: host scheduling\n", mode, (uint32_t)c->x[1]); +} +HLE_FUNC("_ZN2nn2oe15SetCpuBoostModeENS0_12CpuBoostModeE", oe_cpu_boost) { + if ((int)c->x[0] < 0 || (int)c->x[0] > 2) runtime_fail("unknown CPU boost mode"); +} +HLE_FUNC("_ZN2nn2oe16GetOperationModeEv", oe_operation_mode) { c->x[0] = handheld_mode() ? 0 : 1; } +HLE_FUNC("_ZN2nn2oe18GetPerformanceModeEv", oe_performance_mode) { c->x[0] = handheld_mode() ? 0 : 1; } +HLE_FUNC("_ZN2nn2oe20SetFocusHandlingModeENS0_17FocusHandlingModeE", oe_focus_mode) { + int mode = (int)c->x[0]; + if (mode < 0 || mode > 3) runtime_fail("unknown focus handling mode"); + focus_policy = mode; + fprintf(stderr, "[totk] focus policy stored: %d (window lifecycle pending)\n", focus_policy); +} +HLE_FUNC("_ZN2nn2oe20GetCurrentFocusStateEv", oe_focus_state) { c->x[0] = 1; } /* InFocus (no window yet) */ +HLE_FUNC("_ZN2nn2oe28SetResumeNotificationEnabledEb", oe_resume_enabled) { resume_notification = c->x[0] & 255; } +HLE_FUNC("_ZN2nn2oe42SetOperationModeChangedNotificationEnabledEb", oe_operation_enabled) { operation_notification = c->x[0] & 255; } +HLE_FUNC("_ZN2nn2oe44SetPerformanceModeChangedNotificationEnabledEb", oe_performance_enabled) { performance_notification = c->x[0] & 255; } +HLE_FUNC("nndiagStartup", host_diag_init) { (void)c; fprintf(stderr, "[totk] native diagnostics initialized\n"); } +HLE_FUNC("_ZN2nn2oe10InitializeEv", host_oe_init) { (void)c; fprintf(stderr, "[totk] application environment: local Windows process\n"); } +HLE_FUNC("_ZN2nn3aoc10InitializeEv", host_aoc_init) { (void)c; fprintf(stderr, "[totk] add-on content: no DLC\n"); } +HLE_FUNC("_ZN2nn3aoc17CountAddOnContentEv", host_aoc_count) { c->x[0] = 0; } +HLE_FUNC("_ZN2nn3aoc16ListAddOnContentEPiii", host_aoc_list) { + if ((int)c->x[1] < 0 || (int)c->x[2] < 0) runtime_fail("invalid add-on content list range"); + c->x[0] = 0; +} +HLE_FUNC("_ZN2nn3err26InitializeContextWriterApiEv", host_error_init) { (void)c; fprintf(stderr, "[totk] error contexts: local diagnostics\n"); } +HLE_FUNC("_ZN2nn4pctl10InitializeEv", host_pctl_init) { (void)c; fprintf(stderr, "[totk] local application parental controls initialized\n"); } +HLE_FUNC("_ZN2nn4diag34InitializeApplicationAbortObserverEv", host_abort_init) { (void)c; fprintf(stderr, "[totk] native crash observer initialized\n"); } diff --git a/src/host/overrides_time.c b/src/host/overrides_time.c new file mode 100644 index 0000000..5722e98 --- /dev/null +++ b/src/host/overrides_time.c @@ -0,0 +1,150 @@ +/* nn::time for the functions the game imports, backed by the host clock and time zone. + * Ported from MacNativeGames games/totk/runtime/time_host.c @b75be8c (MIT). Changes: guest-register wrappers; + * Windows clock and time-zone calls. User and network system clocks both follow the host wall clock; the steady clock + * counts seconds of host monotonic time with one source id per process. ClockSnapshot uses the public 0xD0-byte + * layout because game code may read its fields through inline accessors. + * TODO(port): the snapshot's location name is "UTC"; map the Windows time zone to its IANA name. */ +#include +#include +#include +#include +#ifdef _WIN32 +#include +#endif + +#include "host.h" +#include "../runtime/hle.h" + +typedef struct { int64_t value; uint8_t source[16]; } SteadyClockTimePoint; +typedef struct { int64_t offset; SteadyClockTimePoint point; } SystemClockContext; +typedef struct { int16_t year; int8_t month, day, hour, minute, second, pad; } CalendarTime; +typedef struct { int32_t day_of_week, day_of_year; char name[8]; uint8_t dst, pad[3]; int32_t utc_offset; } CalendarAdditionalInfo; +typedef struct { + SystemClockContext user, network; int64_t user_time, network_time; + CalendarTime user_calendar, network_calendar; CalendarAdditionalInfo user_info, network_info; + SteadyClockTimePoint steady; char location[0x24]; uint8_t automatic_correction, type, pad[2]; +} ClockSnapshot; +_Static_assert(sizeof(SteadyClockTimePoint) == 0x18 && sizeof(CalendarTime) == 8 && sizeof(CalendarAdditionalInfo) == 0x18, "nn::time types"); +_Static_assert(sizeof(ClockSnapshot) == 0xd0, "nn::time::ClockSnapshot"); + +#define R_INVALID 0x74u /* time module 116: invalid argument family */ +#define R_SOURCE_MISMATCH 0x18074u +static uint8_t source_id[16]; +static int initialized; +static pthread_once_t once = PTHREAD_ONCE_INIT; +static void make_source(void) { + uint64_t a = runtime_now_ns() ^ ((uint64_t)rand() << 32), b = (uint64_t)time(NULL) * 0x9e3779b97f4a7c15ULL ^ (uint64_t)rand(); + memcpy(source_id, &a, 8); memcpy(source_id + 8, &b, 8); +} + +static int64_t steady_seconds(void) { return (int64_t)(runtime_now_ns() / 1000000000u); } +static SteadyClockTimePoint steady_now(void) { + pthread_once(&once, make_source); + SteadyClockTimePoint p = {steady_seconds(), {0}}; + memcpy(p.source, source_id, 16); + return p; +} +static int64_t posix_now(void) { + static long off = -1; + if (off < 0) { const char *e = getenv("TOTK_TIME_OFFSET"); off = e ? atol(e) : 0; if (off < 0) off = 0; } /* diagnostic */ + return (int64_t)time(NULL) + off; +} + +static int32_t local_utc_offset(int *dst) { +#ifdef _WIN32 + TIME_ZONE_INFORMATION tz; + DWORD r = GetTimeZoneInformation(&tz); + *dst = r == TIME_ZONE_ID_DAYLIGHT; + return -(int32_t)(tz.Bias + (r == TIME_ZONE_ID_DAYLIGHT ? tz.DaylightBias : r == TIME_ZONE_ID_STANDARD ? tz.StandardBias : 0)) * 60; +#else + *dst = 0; + return 0; +#endif +} + +static void fill_calendar(CalendarTime *c, CalendarAdditionalInfo *info, int64_t posix, int local) { + time_t t = (time_t)posix; + struct tm tm; + if ((local ? localtime_s(&tm, &t) : gmtime_s(&tm, &t)) != 0) runtime_fail("calendar conversion"); + *c = (CalendarTime){(int16_t)(tm.tm_year + 1900), (int8_t)(tm.tm_mon + 1), (int8_t)tm.tm_mday, (int8_t)tm.tm_hour, + (int8_t)tm.tm_min, (int8_t)tm.tm_sec, 0}; + if (info) { + memset(info, 0, sizeof *info); + info->day_of_week = tm.tm_wday; + info->day_of_year = tm.tm_yday; + int dst = 0; + int32_t offset = local ? local_utc_offset(&dst) : 0; + snprintf(info->name, sizeof info->name, "%s", local ? "LOCAL" : "UTC"); + info->dst = (uint8_t)(local && dst); + info->utc_offset = offset; + } +} + +static ClockSnapshot *snap(uint64_t a) { return (ClockSnapshot *)(uintptr_t)a; } + +HLE_FUNC("_ZN2nn4time10InitializeEv", time_initialize) { initialized = 1; pthread_once(&once, make_source); c->x[0] = 0; } +HLE_FUNC("_ZN2nn4time8FinalizeEv", time_finalize) { initialized = 0; c->x[0] = 0; } +HLE_FUNC("_ZN2nn4time13IsInitializedEv", time_is_initialized) { c->x[0] = (unsigned char)initialized; } +HLE_FUNC("_ZN2nn4time23StandardUserSystemClock14GetCurrentTimeEPNS0_9PosixTimeE", time_user_now) { + int64_t *out = (int64_t *)(uintptr_t)c->x[0]; + if (!out) { c->x[0] = R_INVALID; return; } + *out = posix_now(); c->x[0] = 0; +} +HLE_FUNC("_ZN2nn4time26StandardNetworkSystemClock14GetCurrentTimeEPNS0_9PosixTimeE", time_network_now) { + int64_t *out = (int64_t *)(uintptr_t)c->x[0]; + if (!out) { c->x[0] = R_INVALID; return; } + *out = posix_now(); c->x[0] = 0; +} +/* StandardSteadyClock::now() returns the chrono-style time_point: the int64 seconds value only (in x0). */ +HLE_FUNC("_ZN2nn4time19StandardSteadyClock3nowEv", time_steady_now) { c->x[0] = (uint64_t)steady_now().value; } +HLE_FUNC("_ZN2nn4time14ToCalendarTimeEPNS0_12CalendarTimeEPNS0_22CalendarAdditionalInfoERKNS0_9PosixTimeE", time_to_calendar) { + CalendarTime *cal = (CalendarTime *)(uintptr_t)c->x[0]; + CalendarAdditionalInfo *info = (CalendarAdditionalInfo *)(uintptr_t)c->x[1]; + const int64_t *posix = (const int64_t *)(uintptr_t)c->x[2]; + if (!cal || !posix) { c->x[0] = R_INVALID; return; } + fill_calendar(cal, info, *posix, 1); + c->x[0] = 0; +} +HLE_FUNC("_ZN2nn4time19ToCalendarTimeInUtcERKNS0_9PosixTimeE", time_to_calendar_utc) { /* returns the 8-byte struct in x0 */ + const int64_t *posix = (const int64_t *)(uintptr_t)c->x[0]; + CalendarTime cal; + fill_calendar(&cal, NULL, posix ? *posix : 0, 0); + uint64_t v; memcpy(&v, &cal, 8); c->x[0] = v; +} +HLE_FUNC("_ZN2nn4time18ToPosixTimeFromUtcERKNS0_12CalendarTimeE", time_to_posix_utc) { + const CalendarTime *cal = (const CalendarTime *)(uintptr_t)c->x[0]; + struct tm tm = {0}; + tm.tm_year = cal->year - 1900; tm.tm_mon = cal->month - 1; tm.tm_mday = cal->day; + tm.tm_hour = cal->hour; tm.tm_min = cal->minute; tm.tm_sec = cal->second; + c->x[0] = (uint64_t)(int64_t)_mkgmtime(&tm); +} +HLE_FUNC("_ZN2nn4time13ClockSnapshotC1Ev", time_snapshot_init) { memset(snap(c->x[0]), 0, sizeof(ClockSnapshot)); } +HLE_FUNC("_ZN2nn4time13ClockSnapshot29CreateWithStandardSystemClockEPS1_", time_snapshot_create) { + ClockSnapshot *s = snap(c->x[0]); + if (!s) { c->x[0] = R_INVALID; return; } + SteadyClockTimePoint steady = steady_now(); + int64_t now = posix_now(); + memset(s, 0, sizeof *s); + s->user = (SystemClockContext){now - steady.value, steady}; + s->network = s->user; + s->user_time = now; s->network_time = now; + fill_calendar(&s->user_calendar, &s->user_info, now, 1); + fill_calendar(&s->network_calendar, &s->network_info, now, 1); + s->steady = steady; + snprintf(s->location, sizeof s->location, "UTC"); + s->automatic_correction = 1; s->type = 0; + c->x[0] = 0; +} +HLE_FUNC("_ZN2nn4time20CalculateSpanBetweenEPNS_8TimeSpanERKNS0_13ClockSnapshotES5_", time_span_between) { + int64_t *out = (int64_t *)(uintptr_t)c->x[0]; + const ClockSnapshot *a = snap(c->x[1]), *b = snap(c->x[2]); + if (!out || !a || !b) { c->x[0] = R_INVALID; return; } + if (memcmp(a->steady.source, b->steady.source, 16)) { c->x[0] = R_SOURCE_MISMATCH; return; } + *out = (b->steady.value - a->steady.value) * 1000000000LL; + c->x[0] = 0; +} +/* Change of the user clock offset between snapshots (manual adjustments), as a TimeSpan in x0. */ +HLE_FUNC("_ZN2nn4time48CalculateStandardUserSystemClockDifferenceByUserERKNS0_13ClockSnapshotES3_", time_user_difference) { + const ClockSnapshot *a = snap(c->x[0]), *b = snap(c->x[1]); + c->x[0] = memcmp(a->user.point.source, b->user.point.source, 16) ? 0 : (uint64_t)((b->user.offset - a->user.offset) * 1000000000LL); +} diff --git a/src/host/profiler.c b/src/host/profiler.c new file mode 100644 index 0000000..e7ac048 --- /dev/null +++ b/src/host/profiler.c @@ -0,0 +1,314 @@ +// Sampling CPU profiler for development (TOTK_PROF=:[:]). A host thread wakes about +// 1000 times a second, suspends every other thread of the process in turn, reads its instruction pointer and resumes +// it. At the end the samples are mapped to functions through the PDB (dbghelp) and a report is written: +// - self samples per function (recompiled game functions f_ also get the guest symbol, if any); +// - per thread: busy share and its top functions. +// A thread that is blocked in the kernel (waits, sleeps) reports an ntdll address: those count as waiting. +// Suspending threads costs a few microseconds each; at 1 kHz it slows the game by about a percent. +#include +#include +#include +#include +#include +#include + +#include "host.h" + +#define MAX_THREADS 128 +#define TABLE (1u << 20) // open addressing: rip -> count (per thread) + +typedef struct { uint64_t rip; uint32_t count; uint16_t thread; } Slot; +static Slot *table; +static uint64_t samples_total, samples_lost; +static DWORD tids[MAX_THREADS]; +static uint64_t thread_samples[MAX_THREADS], thread_wait[MAX_THREADS]; +static unsigned nthreads; +static HMODULE ntdll, win32u; +static uint64_t ntdll_lo, ntdll_hi, win32u_lo, win32u_hi, exe_lo, exe_hi; +#define VIA (1ull << 63) // a table key with this bit: the game executable's function that called into a DLL +#define WAIT (1ull << 62) // ... or into a kernel wait +#define WAIT4 (1ull << 61) // ... the fourth such function up the stack of a wait +#define FLAGS (VIA | WAIT | WAIT4) + +static void module_range(HMODULE m, uint64_t *lo, uint64_t *hi) { + if (!m) return; + IMAGE_NT_HEADERS *nt = (IMAGE_NT_HEADERS *)((char *)m + ((IMAGE_DOS_HEADER *)m)->e_lfanew); + *lo = (uint64_t)(uintptr_t)m; + *hi = *lo + nt->OptionalHeader.SizeOfImage; +} + +static unsigned thread_index(DWORD tid) { + for (unsigned i = 0; i < nthreads; i++) if (tids[i] == tid) return i; + if (nthreads == MAX_THREADS) return MAX_THREADS; + tids[nthreads] = tid; + return nthreads++; +} + +static void add(uint64_t rip, unsigned t) { + uint64_t h = (rip * 0x9E3779B97F4A7C15ull ^ t) >> 44; + for (unsigned k = 0; k < TABLE; k++) { + Slot *s = &table[(h + k) & (TABLE - 1)]; + if (s->count && s->rip == rip && s->thread == t) { s->count++; return; } + if (!s->count) { *s = (Slot){rip, 1, (uint16_t)t}; return; } + } + samples_lost++; +} + +// The process's threads, re-listed every 500 ms (a thread snapshot costs milliseconds: listing them on every pass held +// the sampler to ~20 passes a second). Handles stay open between passes. +static HANDLE handles[MAX_THREADS]; +static unsigned slot_of[MAX_THREADS], nhandles; +static void list_threads(DWORD self) { + for (unsigned i = 0; i < nhandles; i++) CloseHandle(handles[i]); + nhandles = 0; + HANDLE snap = CreateToolhelp32Snapshot(TH32CS_SNAPTHREAD, 0); + if (snap == INVALID_HANDLE_VALUE) return; + THREADENTRY32 te = {sizeof te}; + DWORD pid = GetCurrentProcessId(); + for (BOOL ok = Thread32First(snap, &te); ok && nhandles < MAX_THREADS; ok = Thread32Next(snap, &te)) { + if (te.th32OwnerProcessID != pid || te.th32ThreadID == self) continue; + unsigned t = thread_index(te.th32ThreadID); + if (t >= MAX_THREADS) continue; + HANDLE h = OpenThread(THREAD_SUSPEND_RESUME | THREAD_GET_CONTEXT, FALSE, te.th32ThreadID); + if (!h) continue; + handles[nhandles] = h; + slot_of[nhandles++] = t; + } + CloseHandle(snap); +} + +static void sample_all(void) { + for (unsigned i = 0; i < nhandles; i++) { + HANDLE h = handles[i]; + if (SuspendThread(h) == (DWORD)-1) continue; // exited + CONTEXT ctx = {.ContextFlags = CONTEXT_FULL}; + if (GetThreadContext(h, &ctx)) { + unsigned t = slot_of[i]; + uint64_t rip = ctx.Rip; + thread_samples[t]++; + int waiting = (rip >= ntdll_lo && rip < ntdll_hi) || (rip >= win32u_lo && rip < win32u_hi); + if (waiting) thread_wait[t]++; + else add(rip, t); + // outside the game executable (a DLL: the GPU driver, the C runtime; or a wait in the kernel): the game + // executable's functions on the stack, by unwinding - the first (VIA) and, for waits, the fourth (WAIT4: + // past the pthread wrappers, the code that waits) + int outside = !(rip >= exe_lo && rip < exe_hi), seen = 0; + for (int d = 0; outside && d < 48 && rip; d++) { + DWORD64 base; + PRUNTIME_FUNCTION fe = RtlLookupFunctionEntry(rip, &base, NULL); + if (fe) { + void *hd; DWORD64 fr; + RtlVirtualUnwind(UNW_FLAG_NHANDLER, base, rip, fe, &ctx, &hd, &fr, NULL); + } else { // a leaf function: the return address is on top of the stack + ctx.Rip = *(DWORD64 *)(uintptr_t)ctx.Rsp; + ctx.Rsp += 8; + } + rip = ctx.Rip; + if (!(rip >= exe_lo && rip < exe_hi)) continue; + seen++; + if (seen == 1) add(rip | VIA | (waiting ? WAIT : 0), t); + if (!waiting || seen == 4) { if (waiting) add(rip | VIA | WAIT | WAIT4, t); break; } + } + samples_total++; + } + ResumeThread(h); + } +} + +typedef struct { uint64_t fn; uint32_t count; uint16_t thread; char *name; } Fn; +static int by_fn(const void *a, const void *b) { + const Fn *x = a, *y = b; + return x->fn < y->fn ? -1 : x->fn > y->fn ? 1 : 0; +} +static int by_thread_fn(const void *a, const void *b) { + const Fn *x = a, *y = b; + if (x->thread != y->thread) return x->thread < y->thread ? -1 : 1; + return by_fn(a, b); +} +static int by_count(const void *a, const void *b) { + uint64_t x = ((const Fn *)a)->count, y = ((const Fn *)b)->count; + return x < y ? 1 : x > y ? -1 : 0; +} +// Adds up runs of equal function (and thread, when per_thread) in a sorted array, in place; returns the new length. +static unsigned fold(Fn *e, unsigned n, int per_thread) { + unsigned out = 0; + for (unsigned i = 0; i < n; i++) { + if (out && e[out - 1].fn == e[i].fn && (!per_thread || e[out - 1].thread == e[i].thread)) e[out - 1].count += e[i].count; + else e[out++] = e[i]; + } + return out; +} + + +// Function start and name of a host address; recompiled functions f_ get the guest symbol as well. +static uint64_t describe(HANDLE proc, uint64_t rip, char *out, size_t n) { + char buf[sizeof(SYMBOL_INFO) + 256]; + SYMBOL_INFO *si = (SYMBOL_INFO *)buf; + si->SizeOfStruct = sizeof(SYMBOL_INFO); + si->MaxNameLen = 255; + DWORD64 off = 0; + if (!SymFromAddr(proc, rip, &off, si)) { + HMODULE m = NULL; + char mod[MAX_PATH] = "?"; + if (GetModuleHandleExA(GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT, + (LPCSTR)(uintptr_t)rip, &m)) + GetModuleFileNameA(m, mod, sizeof mod); + const char *base = strrchr(mod, '\\'); + snprintf(out, n, "[%s]", base ? base + 1 : mod); + return (uint64_t)(uintptr_t)m; + } + uint64_t guest = 0; + const char *g = NULL; + uint64_t goff = 0; + if (sscanf(si->Name, "f_%llx", (unsigned long long *)&guest) == 1) g = loader_symbol_at(guest, &goff); + if (g && !goff) snprintf(out, n, "%s (%s)", si->Name, g); + else if (g) snprintf(out, n, "%s (%s+%#llx)", si->Name, g, (unsigned long long)goff); + else snprintf(out, n, "%s", si->Name); + HMODULE m = NULL; // outside the game's executable: the module too (a DLL's nearest export may not be the function) + if (GetModuleHandleExA(GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT, (LPCSTR)(uintptr_t)rip, &m) && + m != GetModuleHandleA(NULL)) { + char mod[MAX_PATH] = "?"; + GetModuleFileNameA(m, mod, sizeof mod); + const char *base = strrchr(mod, '\\'); + size_t l = strlen(out); + snprintf(out + l, n - l, " [%s]", base ? base + 1 : mod); + } + return si->Address; +} + + +static void sym_init(HANDLE proc) { + static int done; + if (done++) return; + SymSetOptions(SYMOPT_UNDNAME | SYMOPT_DEFERRED_LOADS); + SymInitialize(proc, NULL, TRUE); +} +// For other development tools (watch.c): the function a host address belongs to, as in the profile report. +uint64_t runtime_symbolize(uint64_t rip, char *out, size_t n) { + HANDLE proc = GetCurrentProcess(); + sym_init(proc); + return describe(proc, rip, out, n); +} + +static void report(const char *path, double seconds) { + HANDLE proc = GetCurrentProcess(); + sym_init(proc); + CreateDirectoryA("diag", NULL); // the default's folder (development output in the game folder) + FILE *f = fopen(path, "w"); + if (!f) { fprintf(stderr, "[prof] cannot write %s\n", path); return; } + unsigned n = 0; + for (unsigned i = 0; i < TABLE; i++) n += table[i].count != 0; + Fn *e = calloc(n ? n : 1, sizeof *e), *t = calloc(n ? n : 1, sizeof *t); + if (!e || !t) runtime_fail("profiler report memory"); + // one entry per (rip, thread): its function start; names only for the functions that make the report + unsigned k = 0; + for (unsigned i = 0; i < TABLE; i++) { + Slot *s = &table[i]; + if (!s->count) continue; + char name[160]; + e[k++] = (Fn){describe(proc, s->rip & ~FLAGS, name, sizeof name) | (s->rip & FLAGS), s->count, s->thread, NULL}; + } + memcpy(t, e, n * sizeof *e); + qsort(e, n, sizeof *e, by_fn); + unsigned nall = fold(e, n, 0); + qsort(e, nall, sizeof *e, by_count); + qsort(t, n, sizeof *t, by_thread_fn); + unsigned nthr = fold(t, n, 1); + uint64_t busy = 0; + for (unsigned i = 0; i < nthreads; i++) busy += thread_samples[i] - thread_wait[i]; + fprintf(f, "TOTK CPU profile: %.1f s, %llu samples (%llu running, %llu waiting in the kernel), %u threads, %llu lost\n", + seconds, (unsigned long long)samples_total, (unsigned long long)busy, + (unsigned long long)(samples_total - busy), nthreads, (unsigned long long)samples_lost); + fprintf(f, "One sample per thread per ~1 ms: a thread busy all the time collects ~1000 samples a second.\n\n"); + fprintf(f, "== Running samples by function (all threads) ==\n"); + for (unsigned i = 0, shown = 0; i < nall && shown < 100; i++) { + if (e[i].fn & FLAGS) continue; + shown++; + char name[160]; + describe(proc, e[i].fn, name, sizeof name); + fprintf(f, "%6.2f%% %8u %s\n", 100.0 * e[i].count / (busy ? busy : 1), e[i].count, name); + } + fprintf(f, "\n== Threads (busy = share of the thread's samples outside kernel waits) ==\n"); + for (unsigned th = 0; th < nthreads; th++) { + uint64_t run = thread_samples[th] - thread_wait[th]; + if (run < 50) continue; + fprintf(f, "thread %5lu: busy %5.1f%% (%llu running samples)\n", tids[th], 100.0 * run / thread_samples[th], + (unsigned long long)run); + unsigned a = 0; + while (a < nthr && t[a].thread != th) a++; + unsigned b = a; + while (b < nthr && t[b].thread == th) b++; + qsort(t + a, b - a, sizeof *t, by_count); + for (unsigned i = a, shown = 0; i < b && shown < 24; i++) { + if (t[i].fn & FLAGS) continue; + shown++; + char name[160]; + describe(proc, t[i].fn, name, sizeof name); + fprintf(f, " %5.1f%% %s\n", 100.0 * t[i].count / run, name); + } + fprintf(f, " time in DLLs, by the game executable's function that called them:\n"); + for (unsigned i = a, shown = 0; i < b && shown < 16; i++) { + if ((t[i].fn & FLAGS) != VIA) continue; + shown++; + char name[160]; + describe(proc, t[i].fn & ~FLAGS, name, sizeof name); + fprintf(f, " %5.1f%% %s\n", 100.0 * t[i].count / run, name); + } + for (int w = 0; w < 2; w++) { + fprintf(f, " waits (share of all its samples), by the %s game executable's function on the stack:\n", w ? "fourth" : "first"); + for (unsigned i = a, shown = 0; i < b && shown < 12; i++) { + if ((t[i].fn & FLAGS) != (w ? FLAGS : (VIA | WAIT))) continue; + shown++; + char name[160]; + describe(proc, t[i].fn & ~FLAGS, name, sizeof name); + fprintf(f, " %5.1f%% %s\n", 100.0 * t[i].count / thread_samples[th], name); + } + } + } + fclose(f); + free(e); + free(t); + fprintf(stderr, "[prof] report written to %s\n", path); +} + +static double start_s, duration_s; +static char out_path[512] = "diag/profile.txt"; + +static DWORD WINAPI profiler(void *unused) { + (void)unused; + timeBeginPeriod(1); + Sleep((DWORD)(start_s * 1000)); + fprintf(stderr, "[prof] sampling for %.0f s\n", duration_s); + DWORD self = GetCurrentThreadId(); + uint64_t t0 = runtime_now_ns(), end = t0 + (uint64_t)(duration_s * 1e9); + uint64_t listed = 0, passes = 0; + for (uint64_t now; (now = runtime_now_ns()) < end; passes++) { + if (now - listed >= 500000000ull) { list_threads(self); listed = now; } + sample_all(); + Sleep(1); + } + fprintf(stderr, "[prof] %.0f passes a second\n", passes / duration_s); + report(out_path, (runtime_now_ns() - t0) / 1e9); + return 0; +} + +void runtime_profiler_start(void) { + const char *e = getenv("TOTK_PROF"); + if (!e || !*e) return; + char path[512] = ""; + if (sscanf(e, "%lf:%lf:%511s", &start_s, &duration_s, path) < 2 || start_s < 0 || duration_s <= 0) { + fprintf(stderr, "[prof] TOTK_PROF=%s: want :[:]\n", e); + return; + } + if (*path) snprintf(out_path, sizeof out_path, "%s", path); + table = calloc(TABLE, sizeof *table); + if (!table) runtime_fail("profiler table"); + ntdll = GetModuleHandleA("ntdll.dll"); + win32u = GetModuleHandleA("win32u.dll"); + module_range(ntdll, &ntdll_lo, &ntdll_hi); + module_range(win32u, &win32u_lo, &win32u_hi); + module_range(GetModuleHandleA(NULL), &exe_lo, &exe_hi); + HANDLE h = CreateThread(NULL, 0, profiler, NULL, 0, NULL); + if (h) SetThreadPriority(h, THREAD_PRIORITY_HIGHEST); + fprintf(stderr, "[prof] armed: %.0f s from now, for %.0f s, report %s\n", start_s, duration_s, out_path); +} diff --git a/src/host/services_lm.c b/src/host/services_lm.c new file mode 100644 index 0000000..966e1e5 --- /dev/null +++ b/src/host/services_lm.c @@ -0,0 +1,39 @@ +/* lm: the system log manager. TOTK 1.4.3's SDK opens it unconditionally in nninitInitializeSdkModule + * (nn::lm::Initialize -> CreateLogServiceByHipc). Command ids follow the public service description (libnx / Switchbrew: + * lm OpenLogger = 0; ILogger Log = 0, SetDestination = 1). Log packets are accepted and dropped unless + * TOTK_LM_LOG is set, which counts them. */ +#include +#include + +#include "host.h" +#include "ipc.h" + +static unsigned long g_packets; + +static void logger(IpcSession *s, const IpcRequest *rq, IpcResponse *rs) { + switch (rq->cmd) { + case 0: /* Log(buffer) */ + if (getenv("TOTK_LM_LOG") && (++g_packets % 100) == 1) + fprintf(stderr, "[totk-lm] %lu log packets (%llu bytes last)\n", g_packets, + (unsigned long long)(rq->nx ? rq->x[0].size : rq->na ? rq->a[0].size : 0)); + return; + case 1: /* SetDestination(u32) */ + return; + default: ipc_unimplemented(s, rq); + } +} + +static void lm(IpcSession *s, const IpcRequest *rq, IpcResponse *rs) { + switch (rq->cmd) { + case 0: /* OpenLogger(pid) -> ILogger */ + ipc_out_object(s, rs, "lm:ILogger", logger, NULL, NULL); + return; + default: ipc_unimplemented(s, rq); + } +} + +void runtime_audio_register(void); // audio_renderer.c +void host_services_register(void) { + runtime_ipc_register("lm", lm, NULL, NULL); + runtime_audio_register(); +} diff --git a/src/host/shared_font.c b/src/host/shared_font.c new file mode 100644 index 0000000..20e8cc1 --- /dev/null +++ b/src/host/shared_font.c @@ -0,0 +1,69 @@ +/* nn::pl shared fonts: the player's system BFTTF files (extracted from their firmware by + * MacNativeGames tools/extract_shared_fonts.py as -.bfttf) laid out back to back as in the pl shared memory. + * After MacNativeGames games/totk/runtime/shared_font.c @b75be8c (MIT): GetSharedFontAddress points 8 bytes past each + * file start (after the BFTTF header), as the SDK adds the service-provided offset to the shared memory base; the size + * is the decoded font size. Changes: the directory comes from TOTK_SHARED_FONTS (default data/fonts), + * and files are checked by their BFTTF header instead of SHA-256 rows of one firmware version. */ +#include +#include +#include +#include + +#include "host.h" +#include "../runtime/hle.h" + +#define SHARED_FONT_COUNT 6 +static const char *const names[SHARED_FONT_COUNT] = { + "0-standard", "1-chinese-simplified", "2-chinese-simplified-extension", "3-chinese-traditional", "4-korean", + "5-nintendo-extension", +}; +static uint8_t *region; +static uint64_t offsets[SHARED_FONT_COUNT], font_bytes[SHARED_FONT_COUNT]; +static pthread_once_t once = PTHREAD_ONCE_INIT; + +static void need(int c, const char *m) { if (!c) runtime_fail(m); } +static uint32_t be32(const uint8_t *p) { return (uint32_t)p[0] << 24 | (uint32_t)p[1] << 16 | (uint32_t)p[2] << 8 | p[3]; } + +static void load(void) { + const char *dir = getenv("TOTK_SHARED_FONTS"); + if (!dir) dir = "data/fonts"; + FILE *f[SHARED_FONT_COUNT]; + uint64_t sizes[SHARED_FONT_COUNT], total = 0; + for (int i = 0; i < SHARED_FONT_COUNT; i++) { + char path[4096]; + need(snprintf(path, sizeof path, "%s/%s.bfttf", dir, names[i]) < (int)sizeof path, "shared font path"); + f[i] = fopen(path, "rb"); + if (!f[i]) { fprintf(stderr, "[totk] %s missing (TOTK_SHARED_FONTS)\n", path); runtime_fail("open shared font"); } + fseek(f[i], 0, SEEK_END); + sizes[i] = (uint64_t)ftell(f[i]); + fseek(f[i], 0, SEEK_SET); + need(sizes[i] > 8 && sizes[i] < (64u << 20), "shared font size"); + total += sizes[i]; + } + region = malloc(total); + need(region != NULL, "shared font memory"); + uint64_t at = 0; + for (int i = 0; i < SHARED_FONT_COUNT; i++) { + need(fread(region + at, 1, sizes[i], f[i]) == sizes[i], "shared font read"); + fclose(f[i]); + // BFTTF header: magic and size, each XORed with the same key; the magic decodes to 0x18029a7f. + uint32_t key = be32(region + at) ^ 0x18029a7fu, n = be32(region + at + 4) ^ key; + need(n == sizes[i] - 8, "shared font BFTTF header"); + offsets[i] = at + 8; + font_bytes[i] = n; + at += sizes[i]; + } + fprintf(stderr, "[totk] shared fonts loaded: %d fonts, %llu bytes\n", SHARED_FONT_COUNT, (unsigned long long)total); +} +static int valid(int type) { + if (type < 0 || type >= SHARED_FONT_COUNT) runtime_fail("invalid shared font type"); + pthread_once(&once, load); + return type; +} +static int load_state(int type) { valid(type); return 1; } /* SharedFontLoadState_Loaded */ +static const void *address(int type) { return region + offsets[valid(type)]; } +static size_t size(int type) { return (size_t)font_bytes[valid(type)]; } + +HLE_NATIVE("_ZN2nn2pl22GetSharedFontLoadStateENS0_14SharedFontTypeE", load_state, 'w') +HLE_NATIVE("_ZN2nn2pl20GetSharedFontAddressENS0_14SharedFontTypeE", address, 'x') +HLE_NATIVE("_ZN2nn2pl17GetSharedFontSizeENS0_14SharedFontTypeE", size, 'x') diff --git a/src/host/svc.c b/src/host/svc.c new file mode 100644 index 0000000..b51ef3b --- /dev/null +++ b/src/host/svc.c @@ -0,0 +1,349 @@ +/* Horizon memory and system SVCs for the recompiled game. + * Ported from MacNativeGames games/totk/runtime/services.c (runtime_svc, memory_init, alias_map, stack_move) and + * loader.c (module permissions / query) @b75be8c (MIT). Changes: guest regions live inside the window that + * src/runtime reserves at startup (pages are committed when first touched); unmapping decommits pages, so a page + * mapped again reads as zeros. Darwin's 16 KiB page handling is gone (Windows pages are 4 KiB, as on the Switch). + * Unmapped guest pages are not made inaccessible yet: a stray access reads zeros instead of faulting. */ +#include +#include +#include +#include +#include +#ifdef _WIN32 +#include +#endif + +#include "host.h" +#include "ipc.h" + +/* Guest address layout. The SDK encodes IPC buffer addresses with 39 bits, so all guest memory lies below 2^39. + MacNativeGames' layout: a 64 GiB alias region (the real 39-bit Horizon size, which the game's allocators depend + on) and 6 GiB for stacks and the heap below it. */ +#define ALIAS_START 0x7000000000ULL +#define ALIAS_SIZE (64ULL << 30) +#define LOW_START 0x1000000000ULL +#define LOW_SIZE (6ULL << 30) +#define STACK_OFF 0x10000000ULL +#define STACK_SIZE 0x10000000ULL +#define STACK_START (LOW_START + STACK_OFF) +#define HEAP_START (LOW_START + STACK_OFF + STACK_SIZE + 0x10000000ULL) +#define HEAP_CAP (4ULL << 30) /* 1.4.3 sets a 2.2 GiB heap during nninitStartup */ +#define ASLR_START LOW_START +#define ASLR_SIZE ((1ULL << 39) - LOW_START) +#define SYSTEM_RESOURCE_BUDGET (32ULL << 20) +#define APP_POOL 0xcd500000ULL /* retail application pool, 3.2 GiB (MacNativeGames' default without 2K heaps) */ + +static uint64_t heap_size; +static unsigned char alias_pages[ALIAS_SIZE / 4096]; +static uint64_t alias_bytes; +static uint64_t main_stack_lo, main_stack_hi; +static uint8_t *user_exception_context; +static pthread_mutex_t memory_lock = PTHREAD_MUTEX_INITIALIZER; + +static void decommit(uint64_t addr, uint64_t size) { +#ifdef _WIN32 + if (size && !VirtualFree((void *)(uintptr_t)addr, size, MEM_DECOMMIT)) runtime_fail("decommit guest pages"); +#else + (void)addr; (void)size; +#endif +} + +void runtime_memory_init(void) { + static int done; + if (done) return; + done = 1; + user_exception_context = hle_alloc(0x1000); + fprintf(stderr, "[totk] guest layout: alias %#llx+%#llx stack %#llx+%#llx heap %#llx (cap %#llx)\n", + ALIAS_START, ALIAS_SIZE, STACK_START, STACK_SIZE, HEAP_START, HEAP_CAP); +} + +uint64_t runtime_main_stack(uint64_t size) { + /* The main thread's stack (main.npdm: 1 MiB) sits at the top of the stack region; the SDK maps further thread + stacks below it with MapMemory. */ + main_stack_hi = STACK_START + STACK_SIZE; + main_stack_lo = main_stack_hi - size; + plat_commit(main_stack_lo, size); + return main_stack_hi; +} + +uint64_t runtime_image_bytes(void) { + uint64_t total = 0; + for (int i = 0; i < g_nmodules; i++) total += g_modules[i].size; + return (total + 0xfff) & ~0xfffULL; +} + +/* ---------- MapMemory / UnmapMemory into the stack region (MacNativeGames stack_move) ---------- */ +typedef struct { uint64_t dst, src, size; } StackMap; +static StackMap stack_maps[4096]; +static unsigned nstack_maps; +static int overlaps(uint64_t a, uint64_t n, uint64_t b, uint64_t m) { return a < b + m && b < a + n; } +static uint64_t stack_move(int unmap, uint64_t dst, uint64_t src, uint64_t size) { + if (!size || ((dst | src | size) & 4095)) return 0xcc01; + if (dst < STACK_START || size > STACK_SIZE || dst - STACK_START > STACK_SIZE - size) return 0xdc01; + int inHeap = src >= HEAP_START && size <= heap_size && src - HEAP_START <= heap_size - size; + int inAlias = src >= ALIAS_START && size <= ALIAS_SIZE && src - ALIAS_START <= ALIAS_SIZE - size; + if (!inHeap && !inAlias) return 0xdc01; + pthread_mutex_lock(&memory_lock); + if (inAlias) { + unsigned first = (unsigned)((src - ALIAS_START) / 4096); + for (uint64_t i = 0; i < size / 4096; i++) + if (!alias_pages[first + i]) { pthread_mutex_unlock(&memory_lock); return 0xdc01; } + } + unsigned found = nstack_maps; + for (unsigned i = 0; i < nstack_maps; i++) + if (stack_maps[i].dst == dst && stack_maps[i].src == src && stack_maps[i].size == size) found = i; + if (unmap) { + if (found == nstack_maps) { pthread_mutex_unlock(&memory_lock); return 0xd401; } + memcpy((void *)(uintptr_t)src, (const void *)(uintptr_t)dst, size); + stack_maps[found] = stack_maps[--nstack_maps]; + decommit(dst, size); + } else { + if (nstack_maps == 4096) { pthread_mutex_unlock(&memory_lock); return 0xd401; } + // The main thread's stack is mapped too (QueryMemory reports it): the SDK places thread stacks at random + // addresses and retries when the kernel refuses one, as Horizon does for memory that is in use. + if (overlaps(dst, size, main_stack_lo, main_stack_hi - main_stack_lo)) { pthread_mutex_unlock(&memory_lock); return 0xdc01; } + for (unsigned i = 0; i < nstack_maps; i++) + if (overlaps(dst, size, stack_maps[i].dst, stack_maps[i].size) || + overlaps(src, size, stack_maps[i].src, stack_maps[i].size)) { + pthread_mutex_unlock(&memory_lock); + return 0xdc01; + } + plat_commit(dst, size); + memcpy((void *)(uintptr_t)dst, (const void *)(uintptr_t)src, size); + stack_maps[nstack_maps++] = (StackMap){dst, src, size}; + } + pthread_mutex_unlock(&memory_lock); + return 0; +} + +/* ---------- MapPhysicalMemory / UnmapPhysicalMemory in the alias region (MacNativeGames alias_map) ---------- */ +static uint64_t alias_map(int unmap, uint64_t addr, uint64_t size) { + if (!size || ((addr | size) & 4095)) return 0xcc01; + if (addr < ALIAS_START || size > ALIAS_SIZE || addr - ALIAS_START > ALIAS_SIZE - size) return 0xdc01; + pthread_mutex_lock(&memory_lock); + unsigned first = (unsigned)((addr - ALIAS_START) / 4096), count = (unsigned)(size / 4096); + /* Horizon MapPhysicalMemory maps only the still-unmapped pages of the range and leaves mapped pages (and their + contents) untouched; TOTK's physical-memory heaps grow by re-mapping ranges that overlap what they own. */ + uint64_t fresh = 0; + if (!unmap) + for (unsigned i = 0; i < count; i++) if (!alias_pages[first + i]) fresh += 4096; + /* Out of the application pool is LimitReached (0x10801), which the SDK treats as recoverable out-of-memory. */ + if (!unmap && fresh + alias_bytes + heap_size + runtime_image_bytes() > APP_POOL) { + pthread_mutex_unlock(&memory_lock); + return 0x10801; + } + if (unmap) { + for (unsigned i = 0; i < nstack_maps; i++) + if (overlaps(addr, size, stack_maps[i].src, stack_maps[i].size)) { pthread_mutex_unlock(&memory_lock); return 0xdc01; } + uint64_t released = 0; + for (unsigned i = 0; i < count; i++) + if (alias_pages[first + i]) { + alias_pages[first + i] = 0; + released += 4096; + decommit(addr + (uint64_t)i * 4096, 4096); // mapped again later: zero-filled like a fresh page + } + alias_bytes -= released; + } else { + for (unsigned i = 0; i < count; i++) alias_pages[first + i] = 1; + alias_bytes += fresh; + } + pthread_mutex_unlock(&memory_lock); + return 0; +} + +/* ---------- module memory: permissions (RELRO) and QueryMemory (MacNativeGames loader.c) ---------- */ +static pthread_mutex_t module_permission_lock = PTHREAD_MUTEX_INITIALIZER; +static unsigned char *module_permissions[MAX_MODULES]; +static unsigned module_page_permission(int i, uint64_t off) { + Module *m = &g_modules[i]; + if (off < m->data_off) return off < ((m->text_off + m->text_size + 0xfff) & ~0xfffULL) ? 5 : 1; + return module_permissions[i] ? module_permissions[i][(off - m->data_off) / 4096] : 3; +} +static unsigned char *permission_table(int i) { + Module *m = &g_modules[i]; + if (!module_permissions[i]) { + size_t count = (m->size - m->data_off + 4095) / 4096; + module_permissions[i] = malloc(count); + if (!module_permissions[i]) runtime_fail("module permission table"); + memset(module_permissions[i], 3, count); + } + return module_permissions[i]; +} +static uint64_t module_permission(uint64_t address, uint64_t size, unsigned permission) { + if (address & 4095) return 0xcc01; + if (!size || (size & 4095)) return 0xca01; + if (permission != 0 && permission != 1 && permission != 3) return 0xd801; + for (int i = 0; i < g_nmodules; i++) { + Module *m = &g_modules[i]; + if (address < m->base || address - m->base < m->data_off || address - m->base >= m->size || + size > m->size - (address - m->base)) continue; + uint64_t off = address - m->base; + pthread_mutex_lock(&module_permission_lock); + unsigned char *t = permission_table(i); + for (uint64_t p = 0; p < size / 4096; p++) + if (t[(off - m->data_off) / 4096 + p] & 16) { pthread_mutex_unlock(&module_permission_lock); return 0xd401; } + memset(t + (off - m->data_off) / 4096, permission, size / 4096); + plat_protect(address, size, permission == 0 ? PLAT_PROT_NONE : permission == 1 ? PLAT_PROT_READ : PLAT_PROT_RW); + pthread_mutex_unlock(&module_permission_lock); + fprintf(stderr, "[totk] %s data permission +%#llx size=%#llx perm=%u\n", m->name, (unsigned long long)off, + (unsigned long long)size, permission); + return 0; + } + return 0xd401; +} +static uint64_t module_lock_permission(uint64_t address, uint64_t size, unsigned mask, unsigned attribute) { + if (address & 4095) return 0xcc01; + if (!size || (size & 4095)) return 0xca01; + if (mask != 16 || attribute != 16) return 0xd801; /* the SDK only uses the irreversible RELRO lock */ + for (int i = 0; i < g_nmodules; i++) { + Module *m = &g_modules[i]; + if (address < m->base || address - m->base < m->data_off || address - m->base >= m->size || + size > m->size - (address - m->base)) continue; + pthread_mutex_lock(&module_permission_lock); + unsigned char *t = permission_table(i); + size_t first = (address - m->base - m->data_off) / 4096; + for (uint64_t p = 0; p < size / 4096; p++) t[first + p] |= 16; + pthread_mutex_unlock(&module_permission_lock); + return 0; + } + return 0xd401; +} +int runtime_module_query(uint64_t address, uint64_t *base, uint64_t *size, unsigned *state, unsigned *permission, + unsigned *attribute) { + for (int i = 0; i < g_nmodules; i++) { + Module *m = &g_modules[i]; + if (address < m->base || address - m->base >= m->size) continue; + uint64_t off = (address - m->base) & ~4095ULL, lo = off, hi = off + 4096; + pthread_mutex_lock(&module_permission_lock); + unsigned perm = module_page_permission(i, off); + int data = off >= m->data_off; + uint64_t limit = data ? m->data_off : 0, end = data ? m->size : m->data_off; + while (lo > limit && module_page_permission(i, lo - 4096) == perm) lo -= 4096; + while (hi < end && module_page_permission(i, hi) == perm) hi += 4096; + pthread_mutex_unlock(&module_permission_lock); + *base = m->base + lo; *size = hi - lo; *state = data ? 4 : 3; *permission = perm & 7; *attribute = perm & 16; + return 1; + } + return 0; +} + +/* ---------- the SVC dispatcher (MacNativeGames runtime_svc) ---------- */ +uint64_t runtime_svc(unsigned num, uint64_t *r, uint64_t pc) { + static _Atomic int trace = -1; + int enabled = atomic_load_explicit(&trace, memory_order_relaxed); + if (enabled < 0) { enabled = getenv("TOTK_TRACE_SVC") != NULL; atomic_store_explicit(&trace, enabled, memory_order_relaxed); } + if (enabled) + fprintf(stderr, "[totk] SVC 0x%x at %#llx x0=0x%llx x1=0x%llx x2=0x%llx x3=0x%llx\n", num, (unsigned long long)pc, + (unsigned long long)r[0], (unsigned long long)r[1], (unsigned long long)r[2], (unsigned long long)r[3]); + if (svc_thread((int)num, r)) return r[0]; + if (runtime_ipc_svc(num, r)) return r[0]; + runtime_memory_init(); + switch (num) { + case 0x02: r[0] = module_permission(r[0], r[1], (unsigned)r[2]); return r[0]; + case 0x03: r[0] = module_lock_permission(r[0], r[1], (unsigned)r[2], (unsigned)r[3]); return r[0]; + case 0x29: { /* GetInfo */ + uint64_t value = 0; + unsigned id = (unsigned)r[1]; + switch (id) { + case 0: value = 7; break; + case 1: value = 0x0ffffffff0000000ULL; break; + case 2: value = ALIAS_START; break; + case 3: value = ALIAS_SIZE; break; + case 4: value = HEAP_START; break; + case 5: value = HEAP_CAP; break; + case 6: case 21: value = APP_POOL; break; + case 7: case 22: value = runtime_image_bytes() + heap_size + alias_bytes; break; + case 8: value = 0; break; + case 10: value = 0; break; + case 11: value = ((uint64_t)rand() << 32) ^ (uint64_t)rand() ^ runtime_now_ns(); break; + case 12: value = ASLR_START; break; + case 13: value = ASLR_SIZE; break; + case 14: value = STACK_START; break; + case 15: value = STACK_SIZE; break; + case 16: value = SYSTEM_RESOURCE_BUDGET; break; + case 17: value = sizeof(stack_maps) + sizeof(alias_pages); break; + case 18: value = 0x0100f2c0115b6000ULL; break; + case 20: value = (uintptr_t)user_exception_context; break; + case 23: value = 1; break; + case 28: value = 0; break; /* AliasRegionExtraSize: no kernel-owned extension */ + default: fprintf(stderr, "unknown GetInfo id %u\n", id); runtime_fail("GetInfo id not implemented"); + } + r[0] = 0; r[1] = value; + return 0; + } + case 0x2c: case 0x2d: { + uint64_t a = r[0], n = r[1]; + r[0] = alias_map(num == 0x2d, a, n); + if (r[0]) + fprintf(stderr, "[totk] %sMapPhysicalMemory %#llx+%#llx failed %#llx (alias %#llx used, heap %#llx)\n", + num == 0x2d ? "Un" : "", (unsigned long long)a, (unsigned long long)n, (unsigned long long)r[0], + (unsigned long long)alias_bytes, (unsigned long long)heap_size); + return r[0]; + } + case 0x04: case 0x05: r[0] = stack_move(num == 5, r[0], r[1], r[2]); return r[0]; + case 0x06: { /* QueryMemory(x0 info out, x2 address) */ + uint64_t addr = r[2], base = addr & ~0xfffULL, size = 0x1000; + unsigned state = 0, perm = 0, attr = 0; + uint64_t heap_end = HEAP_START + heap_size; + if (addr >= main_stack_lo && addr < main_stack_hi) { base = main_stack_lo; size = main_stack_hi - main_stack_lo; state = 0xb; perm = 3; } + else if (addr >= HEAP_START && addr < heap_end) { base = HEAP_START; size = heap_size; state = 5; perm = 3; } + else if (addr >= LOW_START && addr < LOW_START + LOW_SIZE) { + if (addr < HEAP_START) { base = LOW_START; size = HEAP_START - base; } + else { base = heap_end; size = LOW_START + LOW_SIZE - base; } + } else if (addr >= LOW_START + LOW_SIZE && addr < ALIAS_START) { + base = LOW_START + LOW_SIZE; size = ALIAS_START - base; state = 0x10; perm = 0; /* host-owned: Inaccessible */ + } else if (!addr) { base = 0; size = ASLR_START; state = 0; perm = 0; } + else if (runtime_module_query(addr, &base, &size, &state, &perm, &attr)) {} + else { state = 3; perm = 5; } + pthread_mutex_lock(&memory_lock); + if (addr >= ALIAS_START && addr < ALIAS_START + ALIAS_SIZE) { + unsigned p = (unsigned)((addr - ALIAS_START) / 4096), low = p, high = p + 1; + unsigned allocated = alias_pages[p]; + while (low > 0 && alias_pages[low - 1] == allocated) low--; + while (high < sizeof(alias_pages) && alias_pages[high] == allocated) high++; + base = ALIAS_START + (uint64_t)low * 4096; size = (uint64_t)(high - low) * 4096; + state = allocated ? 5 : 0; perm = allocated ? 3 : 0; + } + for (unsigned i = 0; i < nstack_maps; i++) { + StackMap *m = &stack_maps[i]; + uint64_t end = m->dst + m->size, send = m->src + m->size; + if (addr >= m->dst && addr < end) { base = m->dst; size = m->size; state = 0xb; perm = 3; break; } + if (addr >= m->src && addr < send) { base = m->src; size = m->size; state = 5; perm = 0; attr = 4; break; } + if (state == 0) { uint64_t oldEnd = base + size; if (end <= addr && end > base) base = end; if (m->dst > addr && m->dst < oldEnd) oldEnd = m->dst; size = oldEnd - base; } + else if (state == 5) { uint64_t oldEnd = base + size; if (send <= addr && send > base) base = send; if (m->src > addr && m->src < oldEnd) oldEnd = m->src; size = oldEnd - base; } + } + pthread_mutex_unlock(&memory_lock); + struct { uint64_t base, size; uint32_t state, attr, perm, ipc, device, pad; } info = {base, size, state, attr, perm, 0, 0, 0}; + if (r[0]) memcpy((void *)(uintptr_t)r[0], &info, sizeof(info)); + r[0] = 0; r[1] = 0; + return 0; + } + case 0x01: { /* SetHeapSize */ + uint64_t size = r[1]; + if (size > HEAP_CAP || size % 0x200000 || size + runtime_image_bytes() > APP_POOL) { r[0] = 0xd401; return r[0]; } + if (size < heap_size) decommit(HEAP_START + size, heap_size - size); + heap_size = size; + r[0] = 0; r[1] = HEAP_START; + return 0; + } + case 0x26: { /* Break */ + uint32_t result = 0; + if (r[1] && r[2] >= 4) memcpy(&result, (void *)(uintptr_t)r[1], 4); + fprintf(stderr, "[totk] guest Break reason=%llx result=%x lr=%#llx\n", (unsigned long long)r[0], result, + (unsigned long long)r[30]); + if (r[0] & 0x80000000ULL) return 0; + runtime_fail("original SDK Break"); + } + case 0x27: { /* OutputDebugString */ + size_t n = r[1]; + if (n > 65536) runtime_fail("debug string too long"); + fwrite((void *)(uintptr_t)r[0], 1, n, stderr); + r[0] = 0; + return 0; + } + default: + fprintf(stderr, "[totk] unimplemented SVC 0x%x at %#llx lr=%#llx\n", num, (unsigned long long)pc, (unsigned long long)r[30]); + if (num == 0x1f) fprintf(stderr, "[totk] named port %.12s\n", (const char *)(uintptr_t)r[1]); + runtime_fail("system service not implemented"); + } +} diff --git a/src/host/threads.c b/src/host/threads.c new file mode 100644 index 0000000..075b290 --- /dev/null +++ b/src/host/threads.c @@ -0,0 +1,619 @@ +/* Horizon kernel threads and synchronization for the recompiled game. + * Ported from MacNativeGames games/totk/runtime/threads.c @b75be8c (MIT, "Adapted from pinned D2R Mobile + * host/hle_thread.c, MIT"). Changes: each guest thread runs the recompiled code on its own host thread with a Cpu + * (guest_call) instead of entering ARM code on the guest stack; Darwin QoS / Mach diagnostics removed; host -> guest + * SDK calls go through GCALL. */ +/* Original header: Adapted from pinned D2R Mobile host/hle_thread.c, MIT. + * Game-specific diagnostics and optional timing accounting removed. + * Core affinity is virtual bookkeeping; Darwin schedules pthreads. */ +// Horizon 内核的线程与同步 SVC，原生实现：每个客体线程 = 一个宿主 pthread，跑在 SDK 给的客体栈上。 +// SDK 的 nn::os 原样运行，只把它下面的「内核」换成宿主： +// 线程 0x08 CreateThread 0x09 StartThread 0x0A ExitThread 0x0B SleepThread 0x0C/0x0D 优先级 0x0E/0x0F 核掩码 +// 0x10 GetCurrentProcessorNumber 0x25 GetThreadId 0x19 CancelSynchronization +// 锁 0x1A ArbitrateLock 0x1B ArbitrateUnlock（标签字 = 持有者句柄 | 0x40000000 有等待者） +// 条件 0x1C WaitProcessWideKeyAtomic 0x1D SignalProcessWideKey +// 地址 0x34 WaitForAddress 0x35 SignalToAddress +// 事件 0x45 CreateEvent 0x11 SignalEvent 0x12 ClearEvent 0x17 ResetSignal 0x18 WaitSynchronization 0x16 CloseHandle +// 语义按公开 Horizon 内核行为（Atmosphère mesosphère 的 KConditionVariable / KAddressArbiter / KSynchronization）。 +// 所有等待挂在一把全局锁 G 上，每线程一个条件变量；先求正确，性能后量。 +// Thread and synchronization SVCs of the Horizon kernel, implemented natively: each guest thread = one host pthread, running on the guest stack given by the SDK. +// The SDK's nn::os runs as is; only the "kernel" underneath it is replaced by the host: +// Threads 0x08 CreateThread 0x09 StartThread 0x0A ExitThread 0x0B SleepThread 0x0C/0x0D priority 0x0E/0x0F core mask +// 0x10 GetCurrentProcessorNumber 0x25 GetThreadId 0x19 CancelSynchronization +// Locks 0x1A ArbitrateLock 0x1B ArbitrateUnlock (tag word = owner handle | 0x40000000 has waiters) +// Condition 0x1C WaitProcessWideKeyAtomic 0x1D SignalProcessWideKey +// Address 0x34 WaitForAddress 0x35 SignalToAddress +// Events 0x45 CreateEvent 0x11 SignalEvent 0x12 ClearEvent 0x17 ResetSignal 0x18 WaitSynchronization 0x16 CloseHandle +// Semantics follow the public Horizon kernel behavior (KConditionVariable / KAddressArbiter / KSynchronization of Atmosphère mesosphère). +// All waits hang on one global lock G, with one condition variable per thread; correctness first, performance to be measured later. +#include "host.h" +#include "ipc.h" +#include +#include +#include +#include +#include +#include +#undef LOG +#define LOG(...) do {fprintf(stderr,"[totk-thread] " __VA_ARGS__);fputc('\n',stderr);}while(0) +#define die(...) do {LOG(__VA_ARGS__);runtime_fail("thread runtime failure");}while(0) +#define now_ns runtime_now_ns +#define guest_ns runtime_now_ns +static inline int dm_detclock_enabled(void) {return 0;} + +#define R_TIMEOUT 0xEA01u // 内核模块 1，描述 117 | Kernel module 1, description 117 +#define R_CANCELLED 0xEC01u // 118 +#define R_BAD_HANDLE 0xE401u // 114 +#define R_BAD_STATE 0xFA01u // 125 +#define R_OUT_OF_RES 0xCE01u // 103 +#define WAITERS_BIT 0x40000000u +#define H_CUR_THREAD 0xFFFF8000u + +enum { O_FREE, O_THREAD, O_EVENT_R, O_EVENT_W, O_SESSION, O_TMEM }; // O_SESSION / O_TMEM: IPC objects owned by ipc.c +typedef struct event { int signaled; int refs; } event_t; +typedef struct thr { + uint32_t handle; uint64_t id; + uint64_t entry, arg, stack_top; uint64_t host_context[2]; int prio, core; uint64_t mask; + uint64_t born_ns, exit_ns; uint32_t born_tick, exit_tick; // 建立 / 退出的时刻与当时的游戏拍（现场导出用：作业工人是不是在卡住前退的） | Time of creation / exit and the game tick at that time (for the state dump: did the job worker exit before the stall) + int started, exited, cancel; + pthread_t pt; pthread_cond_t cv; + // 当前等待：kind 0=无 1=锁 2=条件 3=地址 4=同步对象 + // Current wait: kind 0=none 1=lock 2=condition 3=address 4=synchronization object + int wkind, woke; uint64_t waddr, wkey; uint32_t wtag; int64_t wres; + uint64_t wseq; // 登记等待的先后序号：同优先级先到先醒（与 Horizon 内核 KThread 等待树同序） | Sequence number of wait registration: at equal priority first come, first woken (same order as the Horizon kernel's KThread wait tree) + struct thr *next; // 等待链 | Wait chain + struct thr *all; // 全部线程链（现场导出用） | All-threads chain (for the state dump) + int last_svc; uint64_t last_lr, last_fp; + int closed; // 句柄已关：与 exited 同时成立才释放对象（Horizon：线程对象随最后一个引用消失） | Handle closed: the object is freed only when this holds together with exited (Horizon: the thread object goes away with the last reference) + uint64_t nsvc; // SVC 次数（现场导出用） | SVC count (for the state dump) + uint32_t qos; + int cur_core; // virtual core held while running guest code, -1 while blocked / not started + int preempt; // a higher-priority thread waits for this thread's core + uint64_t cseq, core_since; struct thr *cnext; + uint32_t wh[4]; int nwh; // handles of the current WaitSynchronization (state dump) + int last_hle; // 1 + wrapper index of the last host call made while "hletrace" was on (state dump) + uint64_t yrun_ns, yrun_svc; // the current run of yields: when it started, and nsvc at its last yield (yield_backoff_ns) +} thr_t; +static thr_t *g_all; + +static pthread_mutex_t G = PTHREAD_MUTEX_INITIALIZER; +#define MAX_H 4096 +static struct { int type; void *obj; } g_h[MAX_H]; +/* [port] the calling guest thread: a thread-local instead of a pthread key (winpthreads hands out key 0, which + MacNativeGames' "g_keythr ? ... : NULL" checks took for "no key") */ +static __thread thr_t *t_self; +static uint64_t g_next_id = 0x51; +static thr_t *g_wait; // 全部在等的线程（锁 / 条件 / 地址 / 同步） | All waiting threads (lock / condition / address / synchronization) + +static uint32_t h_new(int type, void *obj) { // 调用方持 G | Caller holds G + for (int i = 1; i < MAX_H; i++) if (!g_h[i].type) { g_h[i].type = type; g_h[i].obj = obj; return 0x1000 + i; } + die("句柄表满"); +} +static void *h_peek(uint32_t h, int type) { // 只查不报 | Lookup only, no report + if (h == H_CUR_THREAD && type == O_THREAD) return t_self; + unsigned i = h - 0x1000; + if (i == 0 || i >= MAX_H || g_h[i].type != type) return NULL; + return g_h[i].obj; +} +static void *h_get(uint32_t h, int type) { // 查不到即报（句柄错是真错误） | Reports if not found (a wrong handle is a real error) + void *o = h_peek(h, type); + if (!o) LOG("句柄无效 %#x（要类型 %d）", h, type); + return o; +} +static thr_t *cur(void) { + thr_t *t = t_self; + if (!t) die("非客体线程调用了线程 SVC"); + return t; +} + +// ---------- 等待原语（持 G 调用）---------- +// ---------- Wait primitives (called holding G) ---------- +static void wait_unlink(thr_t *t) { + for (thr_t **p = &g_wait; *p; p = &(*p)->next) if (*p == t) { *p = t->next; break; } + t->next = NULL; +} +// 按内核顺序挑等待者：优先级数值小者优先，同优先级先登记者优先（持 G）。kind 1 按锁地址、2 按条件键、3 按地址匹配 +// Pick a waiter in kernel order: numerically lower priority first, at equal priority the earlier registered first (holding G). kind 1 matches by lock address, 2 by condition key, 3 by address +static uint64_t g_wseq; +static thr_t *pick(int kind, uint64_t a) { + thr_t *b = NULL; + for (thr_t *t = g_wait; t; t = t->next) { + if (t->wkind != kind || (kind == 2 ? t->wkey : t->waddr) != a) continue; + if (!b || t->prio < b->prio || (t->prio == b->prio && t->wseq < b->wseq)) b = t; + } + return b; +} +static void wake(thr_t *t, int64_t res) { wait_unlink(t); t->wkind = 0; t->woke = 1; t->wres = res; pthread_cond_signal(&t->cv); } +// 挂起直到被唤醒或超时（ns<0 无限、0 立即超时）；返回 wres 或 R_TIMEOUT +// Suspend until woken or timed out (ns<0 infinite, 0 immediate timeout); returns wres or R_TIMEOUT +static int64_t block_inner(thr_t *t, int kind, int64_t ns); +// ---------- Virtual cores ---------- +// Horizon runs at most one thread per core, and TOTK keys per-core data (allocators, job queues) by +// GetCurrentCoreNumber without locks. Guest threads therefore hold one of the three application cores while they +// run guest code; a core is given up only where Horizon would switch threads (waits, sleep, yield, exit, blocking +// host services) and handed to the best waiting thread (lowest priority value, then first come). A thread that a +// higher-priority waiter wants to preempt gives its core up at its next SVC. TOTK_FREE_CORES=1 disables this. +#define NCORE 3 +static thr_t *g_core_owner[NCORE]; +static thr_t *g_core_wait; +static int g_cores_on = -1; +static int cores_enabled(void) { if (g_cores_on < 0) g_cores_on = getenv("TOTK_FREE_CORES") ? 0 : 1; return g_cores_on; } +static int core_allowed(const thr_t *t, int c) { return (int)((t->mask >> c) & 1); } +static void core_grant(int c) { // holding G: give free core c to the best waiter + thr_t *b = NULL, **bp = NULL; + for (thr_t **p = &g_core_wait; *p; p = &(*p)->cnext) + if (core_allowed(*p, c) && (!b || (*p)->prio < b->prio || ((*p)->prio == b->prio && (*p)->cseq < b->cseq))) { b = *p; bp = p; } + if (!b) return; + *bp = b->cnext; b->cnext = NULL; b->cur_core = c; b->core_since = now_ns(); g_core_owner[c] = b; pthread_cond_signal(&b->cv); +} +static void core_put(thr_t *t) { // holding G + if (t->cur_core < 0) return; + int c = t->cur_core; t->cur_core = -1; t->preempt = 0; g_core_owner[c] = NULL; core_grant(c); +} +static void core_take(thr_t *t) { // holding G; waits until a core in t's mask is handed over + if (!cores_enabled() || t->cur_core >= 0) return; + if (!(t->mask & ((1u << NCORE) - 1))) die("thread %#x core mask %#llx has no application core", t->handle, (unsigned long long)t->mask); + int c = t->core >= 0 && t->core < NCORE && core_allowed(t, t->core) && !g_core_owner[t->core] ? t->core : -1; + for (int i = 0; c < 0 && i < NCORE; i++) if (core_allowed(t, i) && !g_core_owner[i]) c = i; + if (c >= 0) { t->cur_core = c; t->core_since = now_ns(); g_core_owner[c] = t; return; } + t->cseq = ++g_wseq; t->cnext = g_core_wait; g_core_wait = t; + for (int i = 0; i < NCORE; i++) if (core_allowed(t, i) && g_core_owner[i] && g_core_owner[i]->prio > t->prio) g_core_owner[i]->preempt = 1; + while (t->cur_core < 0) pthread_cond_wait(&t->cv, &G); +} +static thr_t *me_or_null(void) { return t_self; } +void runtime_thread_note_hle(unsigned i) { thr_t *t = me_or_null(); if (t) t->last_hle = (int)i + 1; } // hle_stack.c, while tracing +uint32_t runtime_thread_current_handle(void) { thr_t *t = me_or_null(); return t ? t->handle : 0; } +// Host services that wait for other threads or the GPU give the core up around the wait (no host locks held). +int runtime_core_away(void) { + thr_t *t = me_or_null(); if (!t || t->cur_core < 0) return 0; + pthread_mutex_lock(&G); core_put(t); pthread_mutex_unlock(&G); return 1; +} +void runtime_core_back(int *away) { + thr_t *t = me_or_null(); if (!*away || !t) return; + pthread_mutex_lock(&G); core_take(t); pthread_mutex_unlock(&G); +} +// Preemption point at SVC entry: a higher-priority thread waits for this core. +void runtime_core_checkpoint(void) { + thr_t *t = me_or_null(); if (!t || !t->preempt) return; + pthread_mutex_lock(&G); t->preempt = 0; core_put(t); core_take(t); pthread_mutex_unlock(&G); +} +int runtime_current_core(void) { thr_t *t = me_or_null(); return t ? (t->cur_core >= 0 ? t->cur_core : t->core) : -1; } + +static int64_t block(thr_t *t,int kind,int64_t ns) { + if (ns == 0) return block_inner(t, kind, ns); + core_put(t); int64_t r = block_inner(t, kind, ns); core_take(t); return r; +} +static int64_t block_inner(thr_t *t, int kind, int64_t ns) { + t->wkind = kind; t->woke = 0; t->next = g_wait; g_wait = t; t->wseq = ++g_wseq; + if (ns == 0) { wait_unlink(t); t->wkind = 0; return R_TIMEOUT; } + struct timespec dl; + int det_wait = ns > 0 && dm_detclock_enabled(); + uint64_t guest_deadline = det_wait ? guest_ns() + (uint64_t)ns : 0; + if (ns > 0 && !det_wait) { clock_gettime(CLOCK_REALTIME, &dl); dl.tv_sec += ns / 1000000000; dl.tv_nsec += ns % 1000000000; + if (dl.tv_nsec >= 1000000000) { dl.tv_sec++; dl.tv_nsec -= 1000000000; } } + while (!t->woke) { + int rc; + if (det_wait) { + uint64_t now = guest_ns(); + if (now >= guest_deadline) { wait_unlink(t); t->wkind = 0; return R_TIMEOUT; } + (void)(guest_deadline - now); + rc = pthread_cond_wait(&t->cv, &G); + } else rc = ns > 0 ? pthread_cond_timedwait(&t->cv, &G, &dl) : pthread_cond_wait(&t->cv, &G); + if (rc == ETIMEDOUT && !t->woke && (!det_wait || guest_ns() >= guest_deadline)) { wait_unlink(t); t->wkind = 0; return R_TIMEOUT; } + } + return t->wres; +} + +// ---------- 锁 ---------- +// ---------- Locks ---------- +static inline uint32_t ld32(uint64_t a) { return __atomic_load_n((uint32_t *)a, __ATOMIC_SEQ_CST); } +static inline void st32(uint64_t a, uint32_t v) { __atomic_store_n((uint32_t *)a, v, __ATOMIC_SEQ_CST); } +static inline int cas32(uint64_t a, uint32_t *exp, uint32_t v) { return __atomic_compare_exchange_n((uint32_t *)a, exp, v, 0, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST); } + +// 释放 addr 处的锁：交给等在它上面的第一个线程，或清零（持 G） +// Release the lock at addr: hand it to the first thread waiting on it, or clear it to zero (holding G) +static void lock_handoff(uint64_t addr) { + thr_t *w = pick(1, addr); int more = 0; + if (!w) { st32(addr, 0); return; } + for (thr_t *t = g_wait; t; t = t->next) if (t != w && t->wkind == 1 && t->waddr == addr) { more = 1; break; } + st32(addr, w->wtag | (more ? WAITERS_BIT : 0)); + wake(w, 0); +} +// 让 t 去拿 addr 处的锁：空闲就直接拿到并唤醒，否则置等待者位并转为锁等待（持 G） +// Make t take the lock at addr: if free, it gets it directly and is woken, otherwise set the waiters bit and turn into a lock wait (holding G) +static void lock_acquire_for(thr_t *t, uint64_t addr, uint32_t tag) { + uint32_t v = ld32(addr); + for (;;) { + if (v == 0) { if (cas32(addr, &v, tag)) { wake(t, 0); return; } continue; } + if (cas32(addr, &v, v | WAITERS_BIT)) break; + } + t->wkind = 1; t->waddr = addr; t->wtag = tag; // 仍在 g_wait 里，改为锁等待 | Still in g_wait, changed to a lock wait + t->wseq = ++g_wseq; // 内核把它新插进锁持有者的等待队列：同优先级排在已有锁等待者之后 | The kernel inserts it anew into the lock owner's wait queue: at equal priority it goes after the existing lock waiters +} + +// ---------- 线程 ---------- +// ---------- Threads ---------- +uint32_t (*dm_thread_tick_fn)(void); // the current game tick, when diagnostics register it +static void *thread_main(void *p) { + thr_t *t = p; + t_self = t; + Cpu *c = guest_cpu_new(t->stack_top, guest_tls_alloc()); + guest_cpu_set(c); + pthread_mutex_lock(&G); while (!t->started) pthread_cond_wait(&t->cv, &G); core_take(t); pthread_mutex_unlock(&G); + c->x[0] = t->arg; + guest_call(c, t->entry); // does not return: the guest leaves through ExitThread (svc 0x0A) + die("guest thread entry returned (Horizon requires ExitThread)"); +} +static void thr_free(thr_t *t) { // 持 G；从全部线程链摘下并释放 | Holding G; unlink from the all-threads chain and free + for (thr_t **p = &g_all; *p; p = &(*p)->all) if (*p == t) { *p = t->all; break; } + pthread_cond_destroy(&t->cv); free(t); +} + +static _Noreturn void thread_exit_on_host(void *argument) { + thr_t *t=argument; + t->exit_ns = now_ns(); t->exit_tick = dm_thread_tick_fn ? dm_thread_tick_fn() : 0; + // 这里不打日志：退出发生在客体线程自己的栈上（有的只有十几 KiB），带浮点的 printf 会把栈撑爆（真机冒烟：LibcurlResolver 退出时 SIGBUS，回溯 fprintf → __dtoa）。时刻只记下来，现场导出时再打 + // No logging here: the exit happens on the guest thread's own stack (some are only a dozen or so KiB), and a printf with floating point would blow the stack (real-device smoke test: SIGBUS when LibcurlResolver exits, backtrace fprintf → __dtoa). Only the time is recorded, and it is printed later in the state dump + pthread_mutex_lock(&G); + core_put(t); + t->exited = 1; + for (thr_t *w = g_wait; w; ) { thr_t *n = w->next; if (w->wkind == 4) wake(w, -1); w = n; } // 同步等待者重查 | Synchronization waiters re-check + t_self = NULL; + if (t->closed) thr_free(t); + pthread_mutex_unlock(&G); + pthread_exit(NULL); + __builtin_unreachable(); // winpthreads does not declare pthread_exit noreturn +} +static _Noreturn void thread_exit(thr_t *t) { thread_exit_on_host(t); } // already on the thread's host stack + +// 主线程登记，返回其句柄（交给 nnosInitialize 作主线程句柄） +// Register the main thread and return its handle (passed to nnosInitialize as the main thread handle) +uint32_t svc_thread_main_init(void) { + thr_t *t = calloc(1, sizeof *t); + pthread_cond_init(&t->cv, NULL); + t->started = 1; t->prio = 44; t->core = 0; t->mask = 7; t->pt = pthread_self(); + t->cur_core = -1; + pthread_mutex_lock(&G); t->handle = h_new(O_THREAD, t); t->id = g_next_id++; t->all = g_all; g_all = t; core_take(t); pthread_mutex_unlock(&G); + t_self = t; + return t->handle; +} + +static _Atomic uint64_t g_main_yield_n, g_main_yield_ns; // 只有客体主线程写；诊断行在同一线程读 | Written only by the guest main thread; the diagnostic line reads it on the same thread +void dm_main_yield_stats(uint64_t *n, uint64_t *ns) { *n = g_main_yield_n; *ns = g_main_yield_ns; } + +// ---------- 同步对象 ---------- +// ---------- Synchronization objects ---------- +static int sig_obj(uint32_t h, int *bad) { // 1=已触发（持 G） | 1 = signaled (holding G) + unsigned i = h - 0x1000; + if (h == H_CUR_THREAD) return 0; + if (i == 0 || i >= MAX_H) { *bad = 1; return 0; } + switch (g_h[i].type) { + case O_THREAD: return ((thr_t *)g_h[i].obj)->exited; + case O_EVENT_R: return ((event_t *)g_h[i].obj)->signaled; + default: *bad = 1; return 0; + } +} +static void wake_sync_waiters(void) { + for (thr_t *w = g_wait; w; ) { thr_t *n = w->next; if (w->wkind == 4) wake(w, -1); w = n; } +} + +static int64_t now_mono_ns(void) { struct timespec ts; clock_gettime(CLOCK_MONOTONIC, &ts); return ts.tv_sec * 1000000000ll + ts.tv_nsec; } +void dm_guest_clock_advanced(void) { + if (!dm_detclock_enabled()) return; + pthread_mutex_lock(&G); + for (thr_t *t = g_wait; t; t = t->next) if (t->wkind) pthread_cond_signal(&t->cv); + pthread_mutex_unlock(&G); +} + +// 处理线程 / 同步类 SVC；返回 1 = 已处理 +// Handle thread / synchronization SVCs; returns 1 = handled +/* Yield back-off (after nxnative e06a8ec / b703be2). On the Switch a yield with nothing else runnable costs nothing; here + a guest loop that polls and yields becomes a host busy spin (on the iPhone the system logged about 1,300 wake-ups a + second for the game). A run of yields with no other SVC between them stays plain sched_yield for its first 2 ms: + shorter waits are handoffs between working threads, such as the main thread waiting for its ModuleSystem workers on + the frame's critical path (nxnative started at 200 µs and 1.4.3 then dropped itself to 20 fps). After that each yield + sleeps an eighth of the run so far, 10 to 250 µs, with the virtual core given up, so a wait ends at most an eighth late. + Off by default: on the Mac at about 100 fps (1.2.1 field save, 300 s each) it saved 3% CPU (708 -> 685 s) but frames + dropped steadily, 10-15 per 300 instead of 0-3. TOTK_YIELD_BACKOFF=on turns it on (250:2000); =[:] + sets the cap and the start. Returns the sleep for a yield whose run started run_ns ago, 0 for a plain yield. */ +static uint64_t yield_backoff_ns(uint64_t run_ns) { + static int64_t cap = -1, start = 2000000; + if (cap < 0) { + const char *e = getenv("TOTK_YIELD_BACKOFF"); int64_t c = 0; + if (e && !strcmp(e, "on")) c = 250000; + else if (e && *e) { c = atoll(e) * 1000; const char *colon = strchr(e, ':'); if (colon) start = atoll(colon + 1) * 1000; } + if (c) LOG("TOTK_YIELD_BACKOFF: a run of yields backs off after %lld us, sleeping at most %lld us", (long long)(start / 1000), (long long)(c / 1000)); + cap = c; + } + if (!cap || run_ns < (uint64_t)start) return 0; + uint64_t s = run_ns / 8; if (s < 10000) s = 10000; + return s < (uint64_t)cap ? s : (uint64_t)cap; +} + +int svc_thread(int num, uint64_t *regs) { + { thr_t *me = t_self; if (me) { me->last_svc = num; me->last_lr = regs[30]; me->last_fp = regs[29]; me->nsvc++; } } + + switch (num) { + case 0x08: { // CreateThread(→w1 句柄; x1 入口, x2 参数, x3 栈顶, w4 优先级, w5 核) | CreateThread(→w1 handle; x1 entry, x2 argument, x3 stack top, w4 priority, w5 core) + if(!regs[1]||!regs[3]||(regs[3]&15)||(int)regs[4]<28||(int)regs[4]>59||(int)regs[5]<-3||(int)regs[5]>2){regs[0]=R_BAD_STATE;return 1;} + thr_t *t = calloc(1, sizeof *t); + if(!t)die("thread allocation"); + t->cur_core = -1; + pthread_cond_init(&t->cv, NULL); + t->entry = regs[1]; t->arg = regs[2]; t->stack_top = regs[3] & ~0xFull; + t->prio = (int)regs[4]; t->core = (int)regs[5]; t->mask = t->core >= 0 ? 1ull << t->core : 7; + t->born_ns = now_ns(); t->born_tick = dm_thread_tick_fn ? dm_thread_tick_fn() : 0; + pthread_mutex_lock(&G); t->handle = h_new(O_THREAD, t); t->id = g_next_id++; t->all = g_all; g_all = t; pthread_mutex_unlock(&G); + pthread_attr_t a; pthread_attr_init(&a); pthread_attr_setstacksize(&a, 8 << 20); // also the host-service stack of this guest thread + pthread_attr_setdetachstate(&a, PTHREAD_CREATE_DETACHED); // 退出即回收宿主资源（客体经句柄等待，不 join） | Host resources are reclaimed on exit (the guest waits through the handle, no join) + if (pthread_create(&t->pt, &a, thread_main, t)) die("pthread_create"); + pthread_attr_destroy(&a); + LOG("CreateThread h=%#x entry=%#llx sp=%#llx", t->handle,t->entry,t->stack_top); + regs[0] = 0; regs[1] = t->handle; return 1; + } + case 0x09: { // StartThread(w0) + pthread_mutex_lock(&G); + thr_t *t = h_get((uint32_t)regs[0], O_THREAD); + if (t) { t->started = 1; pthread_cond_signal(&t->cv); } + pthread_mutex_unlock(&G); + regs[0] = t ? 0 : R_BAD_HANDLE; return 1; + } + case 0x0A: thread_exit(cur()); return 1; + case 0x0B: { // SleepThread(x0 ns)；0/-1/-2 = 各类让出 | SleepThread(x0 ns); 0/-1/-2 = the various kinds of yield + int64_t ns = (int64_t)regs[0]; + + thr_t *self = cur(); + if (ns > 0) { pthread_mutex_lock(&G); core_put(self); pthread_mutex_unlock(&G); } + if (ns > 0 && dm_detclock_enabled()) { + uint64_t deadline = guest_ns() + (uint64_t)ns; + while (guest_ns() < deadline) { struct timespec ts = {0, 1000000}; nanosleep(&ts, 0); } + } else if (ns > 0) { struct timespec ts = {ns / 1000000000, ns % 1000000000}; nanosleep(&ts, 0); } + else { // 让出（0 / -1 / -2）。主线程的让出次数与耗时记账：游戏渲染任务图「等全部做完」是纯轮询 + 让出（main+0x15ee898），量它占一帧多少 | Yield (0 / -1 / -2). Count and time of the main thread's yields are recorded: the game's render task graph "wait for everything to finish" is pure polling + yield (main+0x15ee898); measure how much of a frame it takes + thr_t *me = t_self; + uint64_t y0 = now_ns(); + if (me && me->nsvc != me->yrun_svc + 1) me->yrun_ns = y0; // another SVC since the last yield: a new run + if (me) me->yrun_svc = me->nsvc; + pthread_mutex_lock(&G); // Horizon yield: hand the core to a waiting thread of equal or higher priority + int give = 0; for (thr_t *w = g_core_wait; w && !give; w = w->cnext) give = me->cur_core >= 0 && core_allowed(w, me->cur_core) && w->prio <= me->prio; + if (give) { core_put(me); core_take(me); } + pthread_mutex_unlock(&G); + uint64_t backoff = !give && me ? yield_backoff_ns(y0 - me->yrun_ns) : 0; + if (backoff) { // a long poll: sleep with the core given up, as SleepThread(ns) does + pthread_mutex_lock(&G); core_put(me); pthread_mutex_unlock(&G); + struct timespec ts = {0, (long)backoff}; nanosleep(&ts, 0); + pthread_mutex_lock(&G); core_take(me); pthread_mutex_unlock(&G); + } else if (!give) sched_yield(); + if (me && me->handle == 0x1001) { g_main_yield_ns += now_ns() - y0; g_main_yield_n++; } + } + if (ns > 0) { pthread_mutex_lock(&G); core_take(self); pthread_mutex_unlock(&G); } + + return 1; + } + case 0x0C: { pthread_mutex_lock(&G);thr_t *t = h_get((uint32_t)regs[1], O_THREAD); regs[0] = t ? 0 : R_BAD_HANDLE; regs[1] = t ? (uint64_t)t->prio : 0; pthread_mutex_unlock(&G);return 1; } + case 0x0D: { if(regs[1]<28||regs[1]>59){regs[0]=R_BAD_STATE;return 1;}pthread_mutex_lock(&G);thr_t *t = h_get((uint32_t)regs[0], O_THREAD); if (t) t->prio = (int)regs[1]; regs[0] = t ? 0 : R_BAD_HANDLE;pthread_mutex_unlock(&G); return 1; } + case 0x0E: { pthread_mutex_lock(&G);thr_t *t = h_get((uint32_t)regs[2], O_THREAD); + regs[0] = t ? 0 : R_BAD_HANDLE; regs[1] = t ? (uint64_t)(uint32_t)t->core : 0; regs[2] = t ? t->mask : 0;pthread_mutex_unlock(&G); return 1; } + case 0x0F: { // SetThreadCoreMask(w0, w1 理想核, x2 掩码)：-1 = 用进程缺省、-2/-3 = 不改理想核；宿主只记账 | SetThreadCoreMask(w0, w1 ideal core, x2 mask): -1 = use the process default, -2/-3 = leave the ideal core unchanged; the host only keeps the bookkeeping + pthread_mutex_lock(&G);thr_t *t = h_get((uint32_t)regs[0], O_THREAD); + if (t) { int c = (int)regs[1]; if (c >= 0) t->core = c; else if (c == -1) t->core = 0; if (regs[2]) t->mask = regs[2]; + if (t == cur() && t->cur_core >= 0 && !core_allowed(t, t->cur_core)) { core_put(t); core_take(t); } } // migrate off a core no longer allowed + regs[0] = t ? 0 : R_BAD_HANDLE;pthread_mutex_unlock(&G); return 1; + } + case 0x10: { thr_t *t = cur(); regs[0] = (uint64_t)(uint32_t)(t->cur_core >= 0 ? t->cur_core : t->core); return 1; } // GetCurrentProcessorNumber + case 0x25: { pthread_mutex_lock(&G);thr_t *t = h_get((uint32_t)regs[1], O_THREAD); regs[0] = t ? 0 : R_BAD_HANDLE; regs[1] = t ? t->id : 0;pthread_mutex_unlock(&G); return 1; } + + case 0x1A: { // ArbitrateLock(w0 持有者句柄, x1 地址, w2 本线程标签) | ArbitrateLock(w0 owner handle, x1 address, w2 this thread's tag) + thr_t *t = cur(); uint64_t addr = regs[1]; + pthread_mutex_lock(&G); + int64_t r = 0; + if (ld32(addr) == ((uint32_t)regs[0] | WAITERS_BIT)) { t->waddr = addr; t->wtag = (uint32_t)regs[2]; r = block(t, 1, -1); } + pthread_mutex_unlock(&G); + regs[0] = (uint64_t)r; return 1; + } + case 0x1B: pthread_mutex_lock(&G); lock_handoff(regs[0]); pthread_mutex_unlock(&G); regs[0] = 0; return 1; + case 0x1C: { // WaitProcessWideKeyAtomic(x0 锁地址, x1 条件键, w2 标签, x3 超时) | WaitProcessWideKeyAtomic(x0 lock address, x1 condition key, w2 tag, x3 timeout) + thr_t *t = cur(); uint64_t addr = regs[0], key = regs[1]; + pthread_mutex_lock(&G); + // 顺序要紧：先置条件键 = 1，再放互斥锁。SDK 的 InternalConditionVariable::Signal 在用户态先读条件键、为 0 就不发 svc 0x1D； + // 用户态抢互斥锁也不经过 G。原先先 lock_handoff 后置键：放锁后、置键前，别的线程可在用户态抢到锁、改完条件、读到键 0 就不唤醒， + // 本线程随后挂上去永远等不到（并行负载下资源表读写锁 main+0x320e5e0 卡死：读者计数 0、锁空闲、写者条件键 1、写者仍在等）。 + // 挂进等待链在 block 里、仍持 G：svc 0x1D 要取 G，不会落在这之前 + // The order matters: first set the condition key = 1, then release the mutex. The SDK's InternalConditionVariable::Signal first reads the condition key in user mode, and does not issue svc 0x1D if it is 0; + // grabbing the mutex in user mode does not go through G either. It used to do lock_handoff first and set the key after: after the release and before the key was set, another thread could grab the lock in user mode, change the condition, read key 0 and not wake anyone, + // and this thread would then link itself in and wait forever (under parallel load the resource table read-write lock main+0x320e5e0 deadlocked: reader count 0, lock free, writer condition key 1, writer still waiting). + // Linking into the wait chain happens inside block, still holding G: svc 0x1D has to take G, so it cannot land before this +#ifdef DM_TEST_HOOKS +#define CV_TEST_HOOK() { extern void (*dm_test_cv_after_release)(void); if (dm_test_cv_after_release) dm_test_cv_after_release(); } // 单测：紧跟放锁之后、挂链之前 | Unit test: right after releasing the lock, before linking into the wait chain +#else +#define CV_TEST_HOOK() +#endif +#ifndef DM_TEST_CV_OLD_ORDER + st32(key, 1); + t->wkey = key; t->waddr = addr; t->wtag = (uint32_t)regs[2]; + lock_handoff(addr); CV_TEST_HOOK(); +#else // 仅单测 t_condvar 的负对照：还原修复前「先放锁、后置键」的顺序 | Negative control for the unit test t_condvar only: restores the pre-fix order "release the lock first, set the key after" + lock_handoff(addr); CV_TEST_HOOK(); + st32(key, 1); + t->wkey = key; t->waddr = addr; t->wtag = (uint32_t)regs[2]; +#endif + int64_t r = block(t, 2, (int64_t)regs[3]); // 超时：不重新拿锁（用户态自己拿） | Timeout: the lock is not re-acquired (user mode takes it itself) + pthread_mutex_unlock(&G); + regs[0] = (uint64_t)r; return 1; + } + case 0x1D: { // SignalProcessWideKey(x0 条件键, w1 个数，<=0 全部) | SignalProcessWideKey(x0 condition key, w1 count, <=0 all) + uint64_t key = regs[0]; int cnt = (int)regs[1], n = 0; + pthread_mutex_lock(&G); + for (thr_t *w; (cnt <= 0 || n < cnt) && (w = pick(2, key)); n++) lock_acquire_for(w, w->waddr, w->wtag); // 按内核顺序；转为锁等待或被唤醒后不再是 kind 2 | In kernel order; once turned into a lock wait or woken it is no longer kind 2 + int left = 0; for (thr_t *w = g_wait; w; w = w->next) if (w->wkind == 2 && w->wkey == key) { left = 1; break; } + if (!left) st32(key, 0); + pthread_mutex_unlock(&G); + regs[0] = 0; return 1; + } + + case 0x34: { // WaitForAddress(x0 地址, w1 类型, w2 值, x3 超时)：0 小于则等、1 小于则减一再等、2 相等则等 | WaitForAddress(x0 address, w1 type, w2 value, x3 timeout): 0 wait if less than, 1 decrement if less than then wait, 2 wait if equal + thr_t *t = cur(); uint64_t addr = regs[0]; int type = (int)regs[1]; int32_t val = (int32_t)regs[2]; + pthread_mutex_lock(&G); + int64_t r; uint32_t v = ld32(addr); + int ok = type == 2 ? (int32_t)v == val : (int32_t)v < val; + if (ok && type == 1) { while (!cas32(addr, &v, v - 1)) if (!((int32_t)v < val)) { ok = 0; break; } } + if (!ok) r = R_BAD_STATE; + else { t->waddr = addr; r = block(t, 3, (int64_t)regs[3]); } + pthread_mutex_unlock(&G); + regs[0] = (uint64_t)r; return 1; + } + case 0x35: { // SignalToAddress(x0 地址, w1 类型, w2 值, w3 个数)：0 唤醒、1 相等则加一并唤醒、2 相等则按等待数改值并唤醒 | SignalToAddress(x0 address, w1 type, w2 value, w3 count): 0 wake, 1 increment if equal and wake, 2 if equal modify the value by waiter count and wake + uint64_t addr = regs[0]; int type = (int)regs[1]; int32_t val = (int32_t)regs[2]; int cnt = (int)regs[3]; + pthread_mutex_lock(&G); + int nw = 0; for (thr_t *w = g_wait; w; w = w->next) if (w->wkind == 3 && w->waddr == addr) nw++; + int64_t r = 0; + if (type != 0) { + int32_t nv = val + 1; + if (type == 2) { + if (cnt <= 0) nv = nw > 0 ? val - 1 : val + 1; + else nv = nw == 0 ? val + 1 : nw <= cnt ? val - 1 : val; + } + uint32_t e = (uint32_t)val; + if (!cas32(addr, &e, (uint32_t)nv)) r = R_BAD_STATE; + } + if (!r) { int n = 0; + for (thr_t *w; (cnt <= 0 || n < cnt) && (w = pick(3, addr)); n++) wake(w, 0); } + pthread_mutex_unlock(&G); + regs[0] = (uint64_t)r; return 1; + } + + case 0x45: { // CreateEvent(→w1 写端, w2 读端) | CreateEvent(→w1 write end, w2 read end) + event_t *e = calloc(1, sizeof *e); e->refs = 2; + pthread_mutex_lock(&G); uint32_t hw = h_new(O_EVENT_W, e), hr = h_new(O_EVENT_R, e); pthread_mutex_unlock(&G); + regs[0] = 0; regs[1] = hw; regs[2] = hr; return 1; + } + case 0x11: case 0x12: case 0x17: { // SignalEvent(写端) / ClearEvent(任一端) / ResetSignal(读端，未触发报错) | SignalEvent(write end) / ClearEvent(either end) / ResetSignal(read end, error if not signaled) + pthread_mutex_lock(&G); + // SignalEvent 只认写端；ClearEvent 两端都认；ResetSignal 只认读端 + // SignalEvent accepts only the write end; ClearEvent accepts both ends; ResetSignal accepts only the read end + uint32_t h = (uint32_t)regs[0]; + event_t *e = num == 0x11 ? h_peek(h, O_EVENT_W) : num == 0x17 ? h_peek(h, O_EVENT_R) + : (h_peek(h, O_EVENT_W) ? h_peek(h, O_EVENT_W) : h_peek(h, O_EVENT_R)); + int64_t r = 0; + if (!e) { r = (num == 0x17 && h_peek(h, O_THREAD)) ? R_BAD_STATE : R_BAD_HANDLE; + if (r == R_BAD_HANDLE) LOG("事件 SVC %#x 句柄无效 %#x", num, h); } + else if (num == 0x11) { e->signaled = 1; wake_sync_waiters(); } + else if (num == 0x17 && !e->signaled) r = R_BAD_STATE; + else e->signaled = 0; + pthread_mutex_unlock(&G); + regs[0] = (uint64_t)r; return 1; + } + case 0x18: { // WaitSynchronization(→w1 下标; x1 句柄数组, w2 个数, x3 超时) | WaitSynchronization(→w1 index; x1 handle array, w2 count, x3 timeout) + thr_t *t = cur(); const uint32_t *hs = (const uint32_t *)regs[1]; int n = (int)regs[2]; int64_t ns = (int64_t)regs[3]; + if(n<0||n>64||(!hs&&n)){regs[0]=R_BAD_STATE;return 1;} + int64_t deadline = ns > 0 ? (dm_detclock_enabled() ? (int64_t)guest_ns() : now_mono_ns()) + ns : 0; + pthread_mutex_lock(&G); + t->nwh = n; for (int i = 0; i < n && i < 4; i++) t->wh[i] = hs[i]; + int64_t r; int idx = -1; + for (;;) { + if (t->cancel) { t->cancel = 0; r = R_CANCELLED; break; } + int bad = 0; + for (int i = 0; i < n && idx < 0; i++) if (sig_obj(hs[i], &bad)) idx = i; + if (bad) { r = R_BAD_HANDLE; break; } + if (idx >= 0) { r = 0; break; } + int64_t left = ns < 0 ? -1 : ns == 0 ? 0 : deadline - (dm_detclock_enabled() ? (int64_t)guest_ns() : now_mono_ns()); + if (ns > 0 && left <= 0) { r = R_TIMEOUT; break; } + if (block(t, 4, left) == R_TIMEOUT) { r = R_TIMEOUT; break; } + } + pthread_mutex_unlock(&G); + regs[0] = (uint64_t)r; regs[1] = idx >= 0 ? (uint64_t)idx : 0; return 1; + } + case 0x19: { // CancelSynchronization(w0 线程) | CancelSynchronization(w0 thread) + pthread_mutex_lock(&G); + thr_t *t = h_get((uint32_t)regs[0], O_THREAD); + if (t) { t->cancel = 1; if (t->wkind == 4) wake(t, -1); } + pthread_mutex_unlock(&G); + regs[0] = t ? 0 : R_BAD_HANDLE; return 1; + } + case 0x16: { // CloseHandle(w0)：事件两端都关才释放；线程句柄关闭后槽位回收，对象在线程也退出后释放 | CloseHandle(w0): an event is freed only when both ends are closed; after a thread handle is closed the slot is recycled, and the object is freed once the thread has also exited + pthread_mutex_lock(&G); + unsigned i = (uint32_t)regs[0] - 0x1000;int valid=0; + if (i > 0 && i < MAX_H && (g_h[i].type == O_EVENT_R || g_h[i].type == O_EVENT_W)) { + valid=1;event_t *e = g_h[i].obj; if (--e->refs == 0) free(e); + g_h[i].type = O_FREE; g_h[i].obj = NULL; + } else if (i > 0 && i < MAX_H && (g_h[i].type == O_SESSION || g_h[i].type == O_TMEM)) { + valid=1;void *o = g_h[i].obj; int session = g_h[i].type == O_SESSION; + g_h[i].type = O_FREE; g_h[i].obj = NULL; + pthread_mutex_unlock(&G); + if (session) runtime_ipc_session_closed(o); else runtime_ipc_tmem_closed(o); + pthread_mutex_lock(&G); + } else if (i > 0 && i < MAX_H && g_h[i].type == O_THREAD) { + valid=1;thr_t *t = g_h[i].obj; + g_h[i].type = O_FREE; g_h[i].obj = NULL; + t->closed = 1; + if (t->exited) thr_free(t); + } + pthread_mutex_unlock(&G); + regs[0] = valid?0:R_BAD_HANDLE; return 1; + } + } + return 0; +} + +// ---------- 服务层的 SystemEvent ---------- +// 真机：服务把内核事件的读端交给客户端，客体经 SDK nn::os 等待。宿主：在客体线程上调 SDK 自己的 +// CreateSystemEvent(跨进程=true)，SDK 经 SVC 0x45 建出内核事件；记下写端句柄，之后任何线程（含宿主音频线程）都可触发。 +// ---------- SystemEvent of the service layer ---------- +// Real device: the service hands the read end of a kernel event to the client, and the guest waits on it through SDK nn::os. Host: on a guest thread, call the SDK's own +// CreateSystemEvent(inter-process=true), and the SDK creates the kernel event through SVC 0x45; the write-end handle is recorded, after which any thread (including the host audio thread) can signal it. +static struct { void *ev; uint32_t wh; } g_se[512]; +static int g_nse; +static pthread_mutex_t g_se_lock = PTHREAD_MUTEX_INITIALIZER; +static uint32_t se_find(void *ev) { + for (int i = 0; i < g_nse; i++) if (g_se[i].ev == ev) return g_se[i].wh; + return 0; +} +void hle_bind_system_event(void *ev, int autoclear) { // 必须在客体线程上调（要执行 SDK 代码） | Must be called on a guest thread (it has to execute SDK code) + pthread_mutex_lock(&g_se_lock); uint32_t wh = se_find(ev); pthread_mutex_unlock(&g_se_lock); + if (wh) return; + if (!t_self) die("hle_bind_system_event 须在客体线程上调用"); + static void *create, *getw; + if (!create) { create = runtime_symbol("_ZN2nn2os17CreateSystemEventEPNS0_15SystemEventTypeENS0_14EventClearModeEb"); + getw = runtime_symbol("_ZN2nn2os30GetWritableHandleOfSystemEventEPKNS0_15SystemEventTypeE"); } + uint32_t r = (uint32_t)GCALL(create, (uint64_t)(uintptr_t)ev, (uint64_t)autoclear, 1); + if (r) die("CreateSystemEvent 失败 %#x", r); + wh = (uint32_t)GCALL(getw, (uint64_t)(uintptr_t)ev); + pthread_mutex_lock(&g_se_lock); + if (g_nse >= (int)(sizeof g_se / sizeof g_se[0])) die("SystemEvent 绑定表满"); + g_se[g_nse].ev = ev; g_se[g_nse++].wh = wh; + pthread_mutex_unlock(&g_se_lock); +} +void hle_signal_system_event(void *ev) { // 未绑定的按手动清除即时绑定（只能在客体线程上发生） | An unbound one is bound on the spot as manual-clear (can only happen on a guest thread) + pthread_mutex_lock(&g_se_lock); uint32_t wh = se_find(ev); pthread_mutex_unlock(&g_se_lock); + if (!wh) { hle_bind_system_event(ev, 0); pthread_mutex_lock(&g_se_lock); wh = se_find(ev); pthread_mutex_unlock(&g_se_lock); } + pthread_mutex_lock(&G); + event_t *e = h_get(wh, O_EVENT_W); + if (!e) die("SystemEvent 写端句柄失效 %#x", wh); + e->signaled = 1; wake_sync_waiters(); + pthread_mutex_unlock(&G); +} + +// ---------- IPC support (ipc.c): sessions, transfer memory and service-owned events ---------- +// Weak fallbacks keep standalone thread tests linkable; the runtime links the real ipc.c versions. +__attribute__((weak)) void runtime_ipc_session_closed(void *s) { (void)s; die("IPC session closed without ipc.c"); } +__attribute__((weak)) void runtime_ipc_tmem_closed(void *t) { (void)t; die("transfer memory closed without ipc.c"); } +uint32_t runtime_handle_new(int session, void *obj) { pthread_mutex_lock(&G); uint32_t h = h_new(session ? O_SESSION : O_TMEM, obj); pthread_mutex_unlock(&G); return h; } +void *runtime_handle_get(uint32_t h, int session) { pthread_mutex_lock(&G); void *o = h_peek(h, session ? O_SESSION : O_TMEM); pthread_mutex_unlock(&G); return o; } +// Service-side event: returns the writable handle kept by the service. +uint32_t runtime_event_new(void) { + event_t *e = calloc(1, sizeof *e); if (!e) die("event allocation"); e->refs = 1; + pthread_mutex_lock(&G); uint32_t hw = h_new(O_EVENT_W, e); pthread_mutex_unlock(&G); return hw; +} +// Copy-handle transfer of the readable end: each call gives the client its own handle to the same event. +uint32_t runtime_event_readable(uint32_t hw) { + pthread_mutex_lock(&G); event_t *e = h_get(hw, O_EVENT_W); if (!e) die("service event handle %#x", hw); + e->refs++; uint32_t hr = h_new(O_EVENT_R, e); pthread_mutex_unlock(&G); return hr; +} +void runtime_event_signal(uint32_t hw) { + pthread_mutex_lock(&G); event_t *e = h_get(hw, O_EVENT_W); if (!e) die("service event handle %#x", hw); + e->signaled = 1; wake_sync_waiters(); pthread_mutex_unlock(&G); +} + +// ---------- Diagnostics ---------- +// MacNativeGames dumps every guest thread's wait state and guest return chain through Mach APIs (TOTK_DUMP_AFTER). +// TODO(port): a Windows version for hang diagnosis. +void runtime_dump_threads(void) { LOG("thread dump not ported yet"); } diff --git a/src/host/watch.c b/src/host/watch.c new file mode 100644 index 0000000..4cfa5e1 --- /dev/null +++ b/src/host/watch.c @@ -0,0 +1,205 @@ +// Memory write watch for development: which code writes a range of game memory. +// TOTK_WATCH=
:[@] +// From that present on (default 1) the range's pages are guard pages. Every write into the range is recorded with the +// writing instruction, the value written (as float and hex) and a short call chain; the access then completes by +// single-stepping it, and the guard is set again. TOTK_WATCH_FRAMES (default 20) presents later, or after 256 writes, +// the report goes to stderr: one line per writing instruction (count, function f_ with the guest +// symbol, last value) and its call chain once. Reads of the pages are passed through the same way without being recorded. +#include +#include +#include +#include + +#include "host.h" + +#define MAX_HITS 256 +#define CHAIN 8 + +typedef struct { uint64_t rip, addr; uint32_t count, last; uint64_t chain[CHAIN]; int nchain; uint64_t x[6], lr; } hit_t; // x, lr: guest registers at the last write +static hit_t hits[MAX_HITS]; +static volatile LONG nhits, nwrites, nfaults; +static uint64_t lo, hi, page_lo, page_hi; +static long long arm_at = -1, report_at; +static volatile LONG armed, done; +static __thread uint64_t rearm_addr; // a guarded access is being single-stepped on this thread +static SRWLOCK lock = SRWLOCK_INIT; +static uint64_t find_at; // TOTK_WATCH_FIND: the 16 bytes there, captured when their last word is written +static uint8_t find_pat[16], zero16[16]; +static volatile int find_have; + +static volatile LONG guarded; // the pages existed and got the guard (they may be committed later than the arming) +static void guard(void) { + DWORD old; + if (VirtualProtect((void *)(uintptr_t)page_lo, (SIZE_T)(page_hi - page_lo), PAGE_READWRITE | PAGE_GUARD, &old)) guarded = 1; +} + +static void chain_of(const CONTEXT *ctx, uint64_t *out, int *n) { + CONTEXT c = *ctx; + *n = 0; + for (int i = 0; i < CHAIN; i++) { + DWORD64 base = 0; + RUNTIME_FUNCTION *f = RtlLookupFunctionEntry(c.Rip, &base, NULL); + if (!f) break; + void *data; + DWORD64 frame; + RtlVirtualUnwind(UNW_FLAG_NHANDLER, base, c.Rip, f, &c, &data, &frame, NULL); + if (!c.Rip) break; + out[(*n)++] = c.Rip; + } +} + +// TOTK_WATCH_FIND=: the 16 bytes now at that offset of the range, searched for in the rest of game memory +// (where a copied value comes from). +static void find_copies(void) { + if (!find_at) return; + uint64_t at = find_at; + if (!find_have) { fprintf(stderr, "[watch] +%#llx: no non-zero write captured\n", (unsigned long long)(at - lo)); return; } + uint8_t pat[16]; + memcpy(pat, find_pat, 16); + float f[4]; + memcpy(f, pat, 16); + fprintf(stderr, "[watch] searching for %g %g %g %g (written at +%#llx)\n", f[0], f[1], f[2], f[3], (unsigned long long)(at - lo)); + int found = 0; + for (uint64_t a = GUEST_LO; a < GUEST_HI && found < 32;) { + MEMORY_BASIC_INFORMATION mi; + if (!VirtualQuery((void *)(uintptr_t)a, &mi, sizeof mi)) break; + uint64_t end = (uint64_t)(uintptr_t)mi.BaseAddress + mi.RegionSize; + if (mi.State == MEM_COMMIT && !(mi.Protect & (PAGE_NOACCESS | PAGE_GUARD))) + for (uint64_t p = (uint64_t)(uintptr_t)mi.BaseAddress; p + 16 <= end && found < 32; p += 4) + if (!memcmp((const void *)(uintptr_t)p, pat, 16) && (p < lo || p >= hi)) { + fprintf(stderr, "[watch] the 16 bytes at +%#llx also at %#llx\n", (unsigned long long)(at - lo), (unsigned long long)p); + found++; + } + a = end; + } + if (!found) fprintf(stderr, "[watch] the 16 bytes at +%#llx are nowhere else in game memory\n", (unsigned long long)(at - lo)); + // TOTK_WATCH_RATIO=1: also three floats in the same proportion as the first three (a colour scaled by an intensity) + if (!getenv("TOTK_WATCH_RATIO") || !(f[0] > 0)) return; + float g = f[1] / f[0], b = f[2] / f[0]; + found = 0; + for (uint64_t a = GUEST_LO; a < GUEST_HI && found < 16;) { + MEMORY_BASIC_INFORMATION mi; + if (!VirtualQuery((void *)(uintptr_t)a, &mi, sizeof mi)) break; + uint64_t end = (uint64_t)(uintptr_t)mi.BaseAddress + mi.RegionSize; + if (mi.State == MEM_COMMIT && !(mi.Protect & (PAGE_NOACCESS | PAGE_GUARD))) + for (uint64_t p = (uint64_t)(uintptr_t)mi.BaseAddress; p + 12 <= end && found < 16; p += 4) { + float v[3]; + memcpy(v, (const void *)(uintptr_t)p, 12); + if (!(v[0] > 1e-6f) || v[0] == f[0]) continue; + float dg = v[1] / v[0] - g, db = v[2] / v[0] - b; + if (dg > -1e-4f && dg < 1e-4f && db > -1e-4f && db < 1e-4f) { + fprintf(stderr, "[watch] proportional at %#llx: %g %g %g (x %g)\n", (unsigned long long)p, v[0], v[1], v[2], f[0] / v[0]); + found++; + } + } + a = end; + } +} + +static LONG CALLBACK handler(EXCEPTION_POINTERS *e) { + DWORD code = e->ExceptionRecord->ExceptionCode; + if (code == STATUS_GUARD_PAGE_VIOLATION) { + uint64_t a = e->ExceptionRecord->ExceptionInformation[1]; + if (a < page_lo || a >= page_hi) return EXCEPTION_CONTINUE_SEARCH; + InterlockedIncrement(&nfaults); + int write = e->ExceptionRecord->ExceptionInformation[0] == 1; + if (write && a >= lo && a < hi) { + InterlockedIncrement(&nwrites); + uint64_t rip = e->ContextRecord->Rip; + AcquireSRWLockExclusive(&lock); + int k = 0; + while (k < nhits && hits[k].rip != rip) k++; + if (k == nhits && nhits < MAX_HITS) { + hits[k].rip = rip; + chain_of(e->ContextRecord, hits[k].chain, &hits[k].nchain); + nhits++; + } + if (k < nhits) { + hits[k].count++; + hits[k].addr = a; + Cpu *c = guest_cpu(); + if (c) { memcpy(hits[k].x, c->x, sizeof hits[k].x); hits[k].lr = c->x[30]; } + } + ReleaseSRWLockExclusive(&lock); + } + if (done) return EXCEPTION_CONTINUE_EXECUTION; // the guard is gone (one-shot): stop watching + rearm_addr = a; + e->ContextRecord->EFlags |= 0x100; // trap after the access, then guard again + return EXCEPTION_CONTINUE_EXECUTION; + } + if (code == STATUS_SINGLE_STEP && rearm_addr) { + uint64_t a = rearm_addr; + rearm_addr = 0; + if (find_at && a == find_at + 12 && memcmp((const void *)(uintptr_t)find_at, zero16, 16)) { // TOTK_WATCH_FIND: as written + memcpy(find_pat, (const void *)(uintptr_t)find_at, 16); + if (!find_have++) find_copies(); // at once: the source may change by the next frame + } + if (a >= lo && a < hi) { // the value now in memory at the last recorded write + AcquireSRWLockExclusive(&lock); + for (int k = 0; k < nhits; k++) + if (hits[k].addr == a) memcpy(&hits[k].last, (const void *)(uintptr_t)(a & ~3ull), 4); + ReleaseSRWLockExclusive(&lock); + } + if (!done) guard(); + e->ContextRecord->EFlags &= ~0x100u; + return EXCEPTION_CONTINUE_EXECUTION; + } + return EXCEPTION_CONTINUE_SEARCH; +} + +static void report(void) { + fprintf(stderr, "[watch] %#llx+%llu: %ld writes, %ld guard faults, %ld writing instructions\n", (unsigned long long)lo, + (unsigned long long)(hi - lo), nwrites, nfaults, nhits); + for (int k = 0; k < nhits; k++) { + char name[200]; + float f; + memcpy(&f, &hits[k].last, 4); + runtime_symbolize(hits[k].rip, name, sizeof name); + fprintf(stderr, "[watch] %6u x +%#llx last %g (%#x) %s\n", hits[k].count, (unsigned long long)(hits[k].addr - lo), f, + hits[k].last, name); + fprintf(stderr, "[watch] guest x0..x5 %#llx %#llx %#llx %#llx %#llx %#llx lr %#llx\n", + (unsigned long long)hits[k].x[0], (unsigned long long)hits[k].x[1], (unsigned long long)hits[k].x[2], + (unsigned long long)hits[k].x[3], (unsigned long long)hits[k].x[4], (unsigned long long)hits[k].x[5], + (unsigned long long)hits[k].lr); + for (int i = 0; i < hits[k].nchain; i++) { + runtime_symbolize(hits[k].chain[i], name, sizeof name); + fprintf(stderr, "[watch] <- %s\n", name); + } + } +} + +void runtime_watch_tick(uint64_t present) { + if (arm_at == -1) { + const char *e = getenv("TOTK_WATCH"); + arm_at = 0; + if (e) { + char *end; + lo = strtoull(e, &end, 0); + uint64_t n = *end == ':' ? strtoull(end + 1, &end, 0) : 0; + arm_at = *end == '@' ? strtoll(end + 1, NULL, 10) : 1; + hi = lo + n; + page_lo = lo & ~0xFFFull; + page_hi = (hi + 0xFFF) & ~0xFFFull; + if (!n) arm_at = 0; + const char *f = getenv("TOTK_WATCH_FIND"); + if (f) find_at = lo + strtoull(f, NULL, 0); + } + } + if (arm_at <= 0 || done) return; + if (!armed && present >= (uint64_t)arm_at) { + armed = 1; + const char *r = getenv("TOTK_WATCH_FRAMES"); + report_at = (long long)present + (r ? atoll(r) : 20); + AddVectoredExceptionHandler(1, handler); + guard(); + fprintf(stderr, "[watch] armed on %#llx+%llu at present %llu\n", (unsigned long long)lo, (unsigned long long)(hi - lo), + (unsigned long long)present); + } else if (armed && !guarded) { + guard(); + } else if (armed && ((long long)present >= report_at || nhits >= MAX_HITS)) { + done = 1; + DWORD old; + VirtualProtect((void *)(uintptr_t)page_lo, (SIZE_T)(page_hi - page_lo), PAGE_READWRITE, &old); + report(); + } +} diff --git a/src/host/window_sdl.c b/src/host/window_sdl.c new file mode 100644 index 0000000..b258d6e --- /dev/null +++ b/src/host/window_sdl.c @@ -0,0 +1,209 @@ +// The window (milestone 3.2): SDL3 on the process's main thread — window, keyboard, game controllers with rumble — +// while the game runs on its own thread (boot.c). The picture goes through src/vk/present.c. +// - keyboard: SDL scancodes become the Mac virtual key codes the controller emulation of overrides_input.c (from +// MacNativeGames) expects, so the key layout is theirs (J/K/I/U = A/B/X/Y, WASD + arrows = sticks, ...); +// - game controllers: buttons by position as on a Switch controller (the right face button is A), sticks with Y up; +// - rumble: the game's vibration values (hid_feedback.c) drive the controller's low / high frequency motors. +// - motion: the controller's gyro and accelerometer feed motion_fusion.c (gyro aiming); F11 / Alt+Enter: fullscreen. +#include +#define SDL_MAIN_HANDLED // main() is boot.c's; SDL only gets SDL_SetMainReady +#include +#include +#include +#include + +#include "host.h" +#include "../vk/present.h" + +static SDL_Window *window; +static SDL_Gamepad *pad; +static SDL_Mutex *pad_lock; + +static int mac_key(SDL_Scancode s) { + switch (s) { + case SDL_SCANCODE_A: return 0; case SDL_SCANCODE_S: return 1; case SDL_SCANCODE_D: return 2; + case SDL_SCANCODE_F: return 3; case SDL_SCANCODE_H: return 4; case SDL_SCANCODE_G: return 5; + case SDL_SCANCODE_Z: return 6; case SDL_SCANCODE_X: return 7; case SDL_SCANCODE_C: return 8; + case SDL_SCANCODE_V: return 9; case SDL_SCANCODE_B: return 11; case SDL_SCANCODE_Q: return 12; + case SDL_SCANCODE_W: return 13; case SDL_SCANCODE_E: return 14; case SDL_SCANCODE_R: return 15; + case SDL_SCANCODE_Y: return 16; case SDL_SCANCODE_T: return 17; case SDL_SCANCODE_O: return 31; + case SDL_SCANCODE_U: return 32; case SDL_SCANCODE_I: return 34; case SDL_SCANCODE_P: return 35; + case SDL_SCANCODE_L: return 37; case SDL_SCANCODE_J: return 38; case SDL_SCANCODE_K: return 40; + case SDL_SCANCODE_N: return 45; case SDL_SCANCODE_M: return 46; case SDL_SCANCODE_RETURN: return 36; + case SDL_SCANCODE_TAB: return 48; case SDL_SCANCODE_SPACE: return 49; case SDL_SCANCODE_ESCAPE: return 53; + case SDL_SCANCODE_LSHIFT: return 56; case SDL_SCANCODE_RSHIFT: return 60; case SDL_SCANCODE_LALT: return 58; + case SDL_SCANCODE_RALT: return 61; case SDL_SCANCODE_LCTRL: return 59; case SDL_SCANCODE_RCTRL: return 62; + case SDL_SCANCODE_LEFT: return 123; case SDL_SCANCODE_RIGHT: return 124; case SDL_SCANCODE_DOWN: return 125; + case SDL_SCANCODE_UP: return 126; + default: return -1; + } +} + +// Switch npad button bits (libnx HidNpadButton): A B X Y StickL StickR L R ZL ZR Plus Minus Left Up Right Down +static int32_t stick(Sint16 v, int invert) { + int32_t x = invert ? -(int32_t)v : v; + return x > 32767 ? 32767 : x < -32767 ? -32767 : x; +} +static void pad_update(void) { + RuntimeGamepadState s = {0}; + if (pad) { + static const struct { SDL_GamepadButton b; uint64_t bit; } map[] = { + {SDL_GAMEPAD_BUTTON_EAST, 1}, {SDL_GAMEPAD_BUTTON_SOUTH, 2}, {SDL_GAMEPAD_BUTTON_NORTH, 4}, + {SDL_GAMEPAD_BUTTON_WEST, 8}, {SDL_GAMEPAD_BUTTON_LEFT_STICK, 16}, {SDL_GAMEPAD_BUTTON_RIGHT_STICK, 32}, + {SDL_GAMEPAD_BUTTON_LEFT_SHOULDER, 64}, {SDL_GAMEPAD_BUTTON_RIGHT_SHOULDER, 128}, + {SDL_GAMEPAD_BUTTON_START, 1024}, {SDL_GAMEPAD_BUTTON_BACK, 2048}, {SDL_GAMEPAD_BUTTON_DPAD_LEFT, 4096}, + {SDL_GAMEPAD_BUTTON_DPAD_UP, 8192}, {SDL_GAMEPAD_BUTTON_DPAD_RIGHT, 16384}, + {SDL_GAMEPAD_BUTTON_DPAD_DOWN, 32768}}; + for (unsigned i = 0; i < sizeof map / sizeof *map; i++) + if (SDL_GetGamepadButton(pad, map[i].b)) s.buttons |= map[i].bit; + if (SDL_GetGamepadAxis(pad, SDL_GAMEPAD_AXIS_LEFT_TRIGGER) > 16384) s.buttons |= 256; + if (SDL_GetGamepadAxis(pad, SDL_GAMEPAD_AXIS_RIGHT_TRIGGER) > 16384) s.buttons |= 512; + s.lx = stick(SDL_GetGamepadAxis(pad, SDL_GAMEPAD_AXIS_LEFTX), 0); + s.ly = stick(SDL_GetGamepadAxis(pad, SDL_GAMEPAD_AXIS_LEFTY), 1); + s.rx = stick(SDL_GetGamepadAxis(pad, SDL_GAMEPAD_AXIS_RIGHTX), 0); + s.ry = stick(SDL_GetGamepadAxis(pad, SDL_GAMEPAD_AXIS_RIGHTY), 1); + s.connected = 1; + } + runtime_gamepad_set(&s); +} + +// hid_feedback.c: the game's vibration value for one side (0 left, 1 right). A Switch controller has a low and a high +// frequency actuator per side; an SDL controller has a low and a high frequency motor, so each takes the stronger side. +// The game sends values every frame; the 250 ms duration stops the motors if it stops sending. +void runtime_rumble_set(unsigned side, float amp_low, float freq_low, float amp_high, float freq_high) { + (void)freq_low; (void)freq_high; + static float low[2], high[2]; + if (side > 1 || !pad_lock) return; + SDL_LockMutex(pad_lock); + low[side] = amp_low < 0 ? 0 : amp_low > 1 ? 1 : amp_low; + high[side] = amp_high < 0 ? 0 : amp_high > 1 ? 1 : amp_high; + float l = low[0] > low[1] ? low[0] : low[1], h = high[0] > high[1] ? high[0] : high[1]; + if (pad) SDL_RumbleGamepad(pad, (Uint16)(l * 65535), (Uint16)(h * 65535), 250); + SDL_UnlockMutex(pad_lock); +} + +int window_init(void) { + SDL_SetMainReady(); + SDL_SetHint(SDL_HINT_AUDIO_DEVICE_SAMPLE_FRAMES, "480"); // 10 ms device buffer at 48 kHz + SDL_SetHint(SDL_HINT_WINDOWS_INTRESOURCE_ICON, "1"); // the exe's icon resource (assets/icon.ico via build/ultratears.rc) for the window + SDL_SetHint(SDL_HINT_WINDOWS_INTRESOURCE_ICON_SMALL, "1"); + if (!SDL_Init(SDL_INIT_VIDEO | SDL_INIT_GAMEPAD | SDL_INIT_AUDIO)) { + fprintf(stderr, "[totk] SDL: %s\n", SDL_GetError()); + return 0; + } + pad_lock = SDL_CreateMutex(); + window = SDL_CreateWindow("UltraTears", 1280, 720, SDL_WINDOW_VULKAN | SDL_WINDOW_RESIZABLE | SDL_WINDOW_HIGH_PIXEL_DENSITY); + if (!window) { fprintf(stderr, "[totk] window: %s\n", SDL_GetError()); return 0; } + if (!vkp_init(window)) fprintf(stderr, "[totk] no Vulkan presentation: the game runs without a picture\n"); + return 1; +} + +// Motion: the controller's accelerometer and gyro (SDL_EVENT_GAMEPAD_SENSOR_UPDATE) go to motion_fusion.c. +static void pad_attached(void) { // pad_lock held; pad may be NULL + if (!pad) { runtime_motion_live(0); return; } + fprintf(stderr, "[totk] game controller: %s\n", SDL_GetGamepadName(pad)); + if (SDL_GamepadHasSensor(pad, SDL_SENSOR_GYRO) && SDL_GamepadHasSensor(pad, SDL_SENSOR_ACCEL) && + SDL_SetGamepadSensorEnabled(pad, SDL_SENSOR_GYRO, true) && SDL_SetGamepadSensorEnabled(pad, SDL_SENSOR_ACCEL, true)) { + const char *axes = getenv("TOTK_GYRO_AXES"); + if (runtime_motion_axes(axes) < 0) fprintf(stderr, "[totk] TOTK_GYRO_AXES=%s is not like \"-x,y,-z\": ignored\n", axes); + runtime_motion_live(1); + fprintf(stderr, "[totk] controller motion sensors on (%.0f Hz gyro)\n", SDL_GetGamepadSensorDataRate(pad, SDL_SENSOR_GYRO)); + } else { + runtime_motion_live(0); + fprintf(stderr, "[totk] controller has no motion sensors: no gyro aiming\n"); + } +} +// SDL's controller frame: x right, y out of the face, z toward the player (the bottom edge); acceleration in m/s^2 as +// the force holding it up (face up reads y = +9.8). The Switch frame (runtime_motion_feed): x right, y toward the top +// edge, z out of the face, acceleration as gravity (face up reads z = -1 G). (x, -z, y) is a rotation, so the gyro +// keeps its sense; the acceleration also changes sign. +static void pad_sensor(const SDL_GamepadSensorEvent *e) { + static double acc[3] = {0, 0, -1}; + const float *d = e->data; + if (e->sensor == SDL_SENSOR_ACCEL) { + acc[0] = -d[0] / SDL_STANDARD_GRAVITY; + acc[1] = d[2] / SDL_STANDARD_GRAVITY; + acc[2] = -d[1] / SDL_STANDARD_GRAVITY; + } else if (e->sensor == SDL_SENSOR_GYRO) { + double gyro[3] = {d[0], -d[2], d[1]}; + runtime_motion_feed(acc, gyro, runtime_now_ns()); + } +} + +static void open_pad(SDL_JoystickID id) { + SDL_LockMutex(pad_lock); + if (!pad) { + pad = SDL_OpenGamepad(id); + pad_attached(); + } + SDL_UnlockMutex(pad_lock); + pad_update(); +} +static void close_pad(SDL_JoystickID id) { + SDL_LockMutex(pad_lock); + if (pad && SDL_GetGamepadID(pad) == id) { + SDL_CloseGamepad(pad); + pad = NULL; + int n = 0; // another one still connected takes over + SDL_JoystickID *ids = SDL_GetGamepads(&n); + if (ids && n) pad = SDL_OpenGamepad(ids[0]); + SDL_free(ids); + pad_attached(); + } + SDL_UnlockMutex(pad_lock); + pad_update(); +} + +static void toggle_fullscreen(void) { + bool on = (SDL_GetWindowFlags(window) & SDL_WINDOW_FULLSCREEN) != 0; + SDL_SetWindowFullscreen(window, !on); // borderless desktop fullscreen (no display mode change) +} + +// Game frame rate and shown frame rate in the title, twice a second. +static void title_update(void) { + static uint64_t t0, game0, shown0; + uint64_t now = runtime_now_ns(), game = vkp_frames(), shown = vkp_presented(); + if (!t0) { t0 = now; game0 = game; shown0 = shown; return; } + if (now - t0 < 500000000) return; + double s = (now - t0) / 1e9; + char title[128]; + snprintf(title, sizeof title, "totk-pc-port | game %.1f fps | shown %.1f fps", (game - game0) / s, (shown - shown0) / s); + SDL_SetWindowTitle(window, title); + t0 = now; game0 = game; shown0 = shown; +} + +_Noreturn void window_loop(void) { + for (;;) { + SDL_Event e; + if (SDL_WaitEventTimeout(&e, 100)) { + do { + switch (e.type) { + case SDL_EVENT_QUIT: + fprintf(stderr, "[totk] window closed\n"); + _Exit(0); + case SDL_EVENT_KEY_DOWN: case SDL_EVENT_KEY_UP: { + if (e.type == SDL_EVENT_KEY_DOWN && !e.key.repeat && + (e.key.scancode == SDL_SCANCODE_F11 || + (e.key.scancode == SDL_SCANCODE_RETURN && (e.key.mod & SDL_KMOD_ALT)))) { + toggle_fullscreen(); + break; + } + int k = mac_key(e.key.scancode); + if (k >= 0 && !e.key.repeat) runtime_input_key((unsigned)k, e.type == SDL_EVENT_KEY_DOWN); + break; + } + case SDL_EVENT_WINDOW_FOCUS_LOST: runtime_input_clear(); break; + case SDL_EVENT_WINDOW_PIXEL_SIZE_CHANGED: vkp_resized(); break; + case SDL_EVENT_GAMEPAD_ADDED: open_pad(e.gdevice.which); break; + case SDL_EVENT_GAMEPAD_REMOVED: close_pad(e.gdevice.which); break; + case SDL_EVENT_GAMEPAD_AXIS_MOTION: case SDL_EVENT_GAMEPAD_BUTTON_DOWN: case SDL_EVENT_GAMEPAD_BUTTON_UP: + pad_update(); + break; + case SDL_EVENT_GAMEPAD_SENSOR_UPDATE: pad_sensor(&e.gsensor); break; + default: break; + } + } while (SDL_PollEvent(&e)); + } + title_update(); + } +} diff --git a/src/probe/probe_cpu.c b/src/probe/probe_cpu.c new file mode 100644 index 0000000..eb32ddf --- /dev/null +++ b/src/probe/probe_cpu.c @@ -0,0 +1,176 @@ +// Phase 2 probe: the recompiled TOTK modules load, relocate and run. Calls exported SDK functions (libc, math) through +// the recompiled code and compares the results with the host's libc; times a few of them. +// probe_cpu +#include +#include +#include +#include + +#include "../runtime/hle.h" +#include "../runtime/rt.h" + +static Cpu *g_cpu; +static uint64_t g_stack_top; +static int g_fail, g_pass; + +static void reset_cpu(void) { + memset(g_cpu, 0, sizeof *g_cpu); + g_cpu->sp = g_stack_top; + g_cpu->tpidrro = (uint64_t)(uintptr_t)hle_alloc(0x200); +} + +static uint64_t call(const char *name, uint64_t a0, uint64_t a1, uint64_t a2, uint64_t a3) { + uint64_t f = loader_find_symbol(name); + if (!f) { fprintf(stderr, "missing %s\n", name); g_fail++; return 0; } + g_cpu->x[0] = a0; g_cpu->x[1] = a1; g_cpu->x[2] = a2; g_cpu->x[3] = a3; + g_cpu->sp = g_stack_top; + guest_call(g_cpu, f); + return g_cpu->x[0]; +} + +static float callf(const char *name, float a, float b) { + uint64_t f = loader_find_symbol(name); + if (!f) { fprintf(stderr, "missing %s\n", name); g_fail++; return 0; } + memset(g_cpu->v, 0, 2 * sizeof g_cpu->v[0]); + memcpy(&g_cpu->v[0], &a, 4); memcpy(&g_cpu->v[1], &b, 4); + g_cpu->sp = g_stack_top; + guest_call(g_cpu, f); + float r; memcpy(&r, &g_cpu->v[0], 4); + return r; +} + +static double calld(const char *name, double a, double b) { + uint64_t f = loader_find_symbol(name); + if (!f) { fprintf(stderr, "missing %s\n", name); g_fail++; return 0; } + memset(g_cpu->v, 0, 2 * sizeof g_cpu->v[0]); + memcpy(&g_cpu->v[0], &a, 8); memcpy(&g_cpu->v[1], &b, 8); + g_cpu->sp = g_stack_top; + guest_call(g_cpu, f); + double r; memcpy(&r, &g_cpu->v[0], 8); + return r; +} + +static void check(const char *what, int ok) { + if (ok) g_pass++; else g_fail++; + printf(" %-44s %s\n", what, ok ? "ok" : "FAIL"); +} + +// A host function the guest calls back (qsort comparator): int cmp(const int *a, const int *b). +static void host_cmp(Cpu *c) { + int a = *(int *)(uintptr_t)c->x[0], b = *(int *)(uintptr_t)c->x[1]; + c->x[0] = (uint64_t)(int64_t)(a < b ? -1 : a > b); +} + +static int near_f(float a, float b) { return fabsf(a - b) <= 1e-6f * fmaxf(1.0f, fabsf(b)); } +static int near_d(double a, double b) { return fabs(a - b) <= 1e-14 * fmax(1.0, fabs(b)); } + +int main(int argc, char **argv) { + if (argc != 2) { fprintf(stderr, "usage: probe_cpu \n"); return 2; } + plat_init(); + plat_crash_handler_install(); + if (!plat_reserve_guest(GUEST_LO, GUEST_HI)) FATAL("cannot reserve the guest window %llx-%llx\n", GUEST_LO, GUEST_HI); + uint64_t t0 = plat_now_ns(); + loader_load(argv[1]); + dispatch_init(); + printf("load+relocate: %.1f ms\n", (plat_now_ns() - t0) / 1e6); + + g_cpu = _aligned_malloc(sizeof(Cpu), 64); + g_stack_top = (uint64_t)(uintptr_t)hle_alloc(1 << 20) + (1 << 20); + reset_cpu(); + + printf("libc through recompiled sdk:\n"); + char *s = hle_alloc(64); + strcpy(s, "Tears of the Kingdom"); + check("strlen", call("strlen", (uintptr_t)s, 0, 0, 0) == strlen(s)); + char *t = hle_alloc(64); + strcpy(t, "Tears of the Kingdon"); + check("memcmp <", (int)call("memcmp", (uintptr_t)s, (uintptr_t)t, 20, 0) < 0); + check("strcmp ==", (int)call("strcmp", (uintptr_t)s, (uintptr_t)s, 0, 0) == 0); + char *num = hle_alloc(32); + strcpy(num, "-123456789"); + check("strtol", (long)call("strtol", (uintptr_t)num, 0, 10, 0) == -123456789L); + uint8_t *src = hle_alloc(4096), *dst = hle_alloc(4096); + for (int i = 0; i < 4096; i++) src[i] = (uint8_t)(i * 7 + 3); + call("memcpy", (uintptr_t)dst + 3, (uintptr_t)src + 1, 3001, 0); + check("memcpy (unaligned 3001 bytes)", !memcmp(dst + 3, src + 1, 3001)); + call("memset", (uintptr_t)dst, 0xab, 4096, 0); + int all = 1; + for (int i = 0; i < 4096; i++) all &= dst[i] == 0xab; + check("memset", all); + + printf("math (float/double, NEON/FP registers):\n"); + float fx[] = {0.5f, 1.25f, -2.0f, 3.14159f, 10.0f}; + int fok = 1; + for (int i = 0; i < 5; i++) { + fok &= near_f(callf("sinf", fx[i], 0), sinf(fx[i])); + fok &= near_f(callf("cosf", fx[i], 0), cosf(fx[i])); + fok &= near_f(callf("expf", fx[i], 0), expf(fx[i])); + fok &= near_f(callf("atan2f", fx[i], 0.75f), atan2f(fx[i], 0.75f)); + fok &= near_f(callf("powf", fabsf(fx[i]), 1.7f), powf(fabsf(fx[i]), 1.7f)); + fok &= callf("sqrtf", fabsf(fx[i]), 0) == sqrtf(fabsf(fx[i])); + } + check("sinf cosf expf atan2f powf sqrtf (5 inputs)", fok); + int dok = 1; + for (int i = 0; i < 5; i++) { + dok &= near_d(calld("sin", fx[i], 0), sin(fx[i])); + dok &= near_d(calld("exp", fx[i], 0), exp(fx[i])); + dok &= near_d(calld("pow", fabs(fx[i]), 2.5), pow(fabs(fx[i]), 2.5)); + } + check("sin exp pow (double, 5 inputs)", dok); + + printf("formatting and callbacks:\n"); + char *out = hle_alloc(128), *fmt = hle_alloc(64); + strcpy(fmt, "%d|%x|%s"); + { // varargs (AAPCS64: in registers x3..x5 here) + uint64_t f = loader_find_symbol("snprintf"); + g_cpu->x[4] = 0xbeef; g_cpu->x[5] = (uintptr_t)s; + g_cpu->x[0] = (uintptr_t)out; g_cpu->x[1] = 128; g_cpu->x[2] = (uintptr_t)fmt; g_cpu->x[3] = 42; + g_cpu->sp = g_stack_top; + guest_call(g_cpu, f); + } + printf(" snprintf -> \"%s\"\n", out); + check("snprintf %d|%x|%s", !strcmp(out, "42|beef|Tears of the Kingdom")); + int *arr = hle_alloc(1000 * sizeof(int)); + for (int i = 0; i < 1000; i++) arr[i] = (i * 7919) % 1000 - 500; + uint64_t cmp = hle_register("probe_host_cmp", host_cmp); + call("qsort", (uintptr_t)arr, 1000, sizeof(int), cmp); + int sorted = 1; + for (int i = 1; i < 1000; i++) sorted &= arr[i - 1] <= arr[i]; + check("qsort with a host comparator (guest -> host)", sorted); + + printf("timing (recompiled vs host):\n"); + uint8_t *big_src = hle_alloc(32 << 20), *big_dst = hle_alloc(32 << 20); + memset(big_src, 1, 32 << 20); + t0 = plat_now_ns(); + for (int i = 0; i < 20; i++) call("memcpy", (uintptr_t)big_dst, (uintptr_t)big_src, 32 << 20, 0); + double g_ms = (plat_now_ns() - t0) / 1e6; + t0 = plat_now_ns(); + for (int i = 0; i < 20; i++) memcpy(big_dst, big_src, 32 << 20); + double h_ms = (plat_now_ns() - t0) / 1e6; + printf(" memcpy 20 x 32 MiB: recompiled %.1f ms (%.1f GB/s), host %.1f ms\n", g_ms, 20 * 32 / 1024.0 / (g_ms / 1e3), h_ms); + uint64_t fsin = loader_find_symbol("sinf"); + volatile float acc = 0; + t0 = plat_now_ns(); + for (int i = 0; i < 1000000; i++) { + float a = i * 1e-5f; + memcpy(&g_cpu->v[0], &a, 4); + g_cpu->sp = g_stack_top; + guest_call(g_cpu, fsin); + float r; memcpy(&r, &g_cpu->v[0], 4); + acc += r; + } + g_ms = (plat_now_ns() - t0) / 1e6; + t0 = plat_now_ns(); + for (int i = 0; i < 1000000; i++) acc += sinf(i * 1e-5f); + h_ms = (plat_now_ns() - t0) / 1e6; + printf(" sinf x 1e6: recompiled %.1f ms (%.1f ns/call), host %.1f ms\n", g_ms, g_ms, h_ms); + + printf("%d passed, %d failed\n", g_pass, g_fail); + return g_fail != 0; +} + +// No Horizon host in this probe: a system call means the tested code reached the OS. +void rt_svc(Cpu *c, uint32_t imm, uint64_t pc) { + (void)c; + FATAL("svc 0x%x at %llx (no Horizon host in probe_cpu)\n", imm, (unsigned long long)pc); +} diff --git a/src/runtime/cpu.h b/src/runtime/cpu.h new file mode 100644 index 0000000..0526e6e --- /dev/null +++ b/src/runtime/cpu.h @@ -0,0 +1,197 @@ +// Guest CPU state and helpers used by recompiled code. +#pragma once +#include +#include +#include + +typedef struct __attribute__((aligned(16))) Cpu { + uint64_t x[32]; // x0-x30 (x[31] unused; sp is separate) + uint64_t sp; // 256 + uint64_t pc; // 264: target for indirect dispatch + uint64_t nzcv; // 272: flags in bits 31..28 + uint64_t tpidr; // 280 + uint64_t tpidrro; // 288 + uint64_t excl_addr; // 296 + uint64_t excl_val; // 304 + void *thread; // 312: runtime thread object + uint64_t fpcr; // 320: guest FPCR/FPSR (portable FP/SIMD lowering only; the asm path uses the host's) + uint64_t fpsr; // 328 + uint8_t _pad[512 - 336]; + __uint128_t v[32]; // 512 +} Cpu; + +_Static_assert(offsetof(Cpu, sp) == 256, "sp"); +_Static_assert(offsetof(Cpu, pc) == 264, "pc"); +_Static_assert(offsetof(Cpu, nzcv) == 272, "nzcv"); +_Static_assert(offsetof(Cpu, fpcr) == 320, "fpcr"); +_Static_assert(offsetof(Cpu, fpsr) == 328, "fpsr"); +_Static_assert(offsetof(Cpu, v) == 512, "v"); + +typedef void (*GuestFn)(Cpu *); + +// ---- runtime services (runtime/*.c) +void rt_dispatch(Cpu *c); // call function at c->pc +void rt_svc(Cpu *c, uint32_t imm, uint64_t pc); +void rt_brk(Cpu *c, uint32_t imm, uint64_t pc) __attribute__((noreturn)); +void rt_udf(Cpu *c, uint64_t pc) __attribute__((noreturn)); +uint64_t rt_ticks(void); + +#if defined(__aarch64__) +static inline uint64_t rt_get_fpcr(void) { uint64_t v; __asm__ volatile("mrs %0, fpcr" : "=r"(v)); return v; } +static inline uint64_t rt_get_fpsr(void) { uint64_t v; __asm__ volatile("mrs %0, fpsr" : "=r"(v)); return v; } +static inline void rt_set_fpcr(uint64_t v) { __asm__ volatile("msr fpcr, %0" :: "r"(v)); } +static inline void rt_set_fpsr(uint64_t v) { __asm__ volatile("msr fpsr, %0" :: "r"(v)); } +#else +// Host FP state is not the guest's here: code lifted with --simd=c keeps FPCR/FPSR in Cpu (c->fpcr, zeroed +// with the Cpu), so the host-side reset in thread start-up has nothing to do. +static inline uint64_t rt_get_fpcr(void) { return 0; } +static inline uint64_t rt_get_fpsr(void) { return 0; } +static inline void rt_set_fpcr(uint64_t v) { (void)v; } +static inline void rt_set_fpsr(uint64_t v) { (void)v; } +#endif + +// ---- memory: guest addresses are host addresses +typedef uint16_t __attribute__((aligned(1), may_alias)) u16u; +typedef uint32_t __attribute__((aligned(1), may_alias)) u32u; +typedef uint64_t __attribute__((aligned(1), may_alias)) u64u; +typedef uint8_t __attribute__((may_alias)) u8a; + +// Function-entry profiler (runtime/fnprof.c, LARECOMP_FNPROF): one predictable branch per call when off. +extern volatile int g_fnprof; +void fnprof_hit(uint64_t addr, uint64_t lr, uint64_t x0); +#define FN_ENTER(a) do { if (__builtin_expect(g_fnprof, 0)) fnprof_hit(a, c->x[30], c->x[0]); } while (0) + +#define LD8(a) (*(const u8a *)(uintptr_t)(a)) +#define LD16(a) (*(const u16u *)(uintptr_t)(a)) +#define LD32(a) (*(const u32u *)(uintptr_t)(a)) +#define LD64(a) (*(const u64u *)(uintptr_t)(a)) +#define ST8(a, v) (*(u8a *)(uintptr_t)(a) = (uint8_t)(v)) +#define ST16(a, v) (*(u16u *)(uintptr_t)(a) = (uint16_t)(v)) +#define ST32(a, v) (*(u32u *)(uintptr_t)(a) = (uint32_t)(v)) +#define ST64(a, v) (*(u64u *)(uintptr_t)(a) = (uint64_t)(v)) + +// ---- exclusive monitor (emulated with compare-and-swap) +#define EXCL(N, T) \ + static inline uint64_t ldx##N(Cpu *c, uint64_t a) { \ + T v = __atomic_load_n((T *)(uintptr_t)a, __ATOMIC_ACQUIRE); \ + c->excl_addr = a; \ + c->excl_val = v; \ + return v; \ + } \ + static inline uint32_t stx##N(Cpu *c, uint64_t a, uint64_t v) { \ + if (c->excl_addr != a) return 1; \ + c->excl_addr = ~0ull; \ + T expected = (T)c->excl_val; \ + return !__atomic_compare_exchange_n((T *)(uintptr_t)a, &expected, (T)v, 0, \ + __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST); \ + } \ + static inline uint64_t lda##N(uint64_t a) { \ + return __atomic_load_n((T *)(uintptr_t)a, __ATOMIC_ACQUIRE); \ + } \ + static inline void stl##N(uint64_t a, uint64_t v) { \ + __atomic_store_n((T *)(uintptr_t)a, (T)v, __ATOMIC_RELEASE); \ + } +EXCL(8, uint8_t) +EXCL(16, uint16_t) +EXCL(32, uint32_t) +EXCL(64, uint64_t) + +// ---- flags (packed NZCV in bits 31..28) +#define FLAG_N(c) (((c)->nzcv >> 31) & 1) +#define FLAG_Z(c) (((c)->nzcv >> 30) & 1) +#define FLAG_C(c) (((c)->nzcv >> 29) & 1) +#define FLAG_V(c) (((c)->nzcv >> 28) & 1) +#define COND_EQ(c) (FLAG_Z(c)) +#define COND_NE(c) (!FLAG_Z(c)) +#define COND_CS(c) (FLAG_C(c)) +#define COND_CC(c) (!FLAG_C(c)) +#define COND_MI(c) (FLAG_N(c)) +#define COND_PL(c) (!FLAG_N(c)) +#define COND_VS(c) (FLAG_V(c)) +#define COND_VC(c) (!FLAG_V(c)) +#define COND_HI(c) (FLAG_C(c) && !FLAG_Z(c)) +#define COND_LS(c) (!(FLAG_C(c) && !FLAG_Z(c))) +#define COND_GE(c) (FLAG_N(c) == FLAG_V(c)) +#define COND_LT(c) (FLAG_N(c) != FLAG_V(c)) +#define COND_GT(c) (!FLAG_Z(c) && FLAG_N(c) == FLAG_V(c)) +#define COND_LE(c) (!(!FLAG_Z(c) && FLAG_N(c) == FLAG_V(c))) + +#define MKFLAGS(n, z, cy, v) \ + (((uint64_t)(n) << 31) | ((uint64_t)(z) << 30) | ((uint64_t)(cy) << 29) | ((uint64_t)(v) << 28)) + +static inline uint64_t adds64(Cpu *c, uint64_t a, uint64_t b) { + uint64_t r = a + b; + c->nzcv = MKFLAGS(r >> 63, r == 0, r < a, ((a ^ r) & (b ^ r)) >> 63); + return r; +} +static inline uint64_t subs64(Cpu *c, uint64_t a, uint64_t b) { + uint64_t r = a - b; + c->nzcv = MKFLAGS(r >> 63, r == 0, a >= b, ((a ^ b) & (a ^ r)) >> 63); + return r; +} +static inline uint32_t adds32(Cpu *c, uint32_t a, uint32_t b) { + uint32_t r = a + b; + c->nzcv = MKFLAGS(r >> 31, r == 0, r < a, ((a ^ r) & (b ^ r)) >> 31); + return r; +} +static inline uint32_t subs32(Cpu *c, uint32_t a, uint32_t b) { + uint32_t r = a - b; + c->nzcv = MKFLAGS(r >> 31, r == 0, a >= b, ((a ^ b) & (a ^ r)) >> 31); + return r; +} +static inline uint64_t logic64(Cpu *c, uint64_t r) { c->nzcv = MKFLAGS(r >> 63, r == 0, 0, 0); return r; } +static inline uint32_t logic32(Cpu *c, uint32_t r) { c->nzcv = MKFLAGS(r >> 31, r == 0, 0, 0); return r; } + +#define ADC_IMPL(N, S) \ + static inline uint##N##_t adc##N(Cpu *c, uint##N##_t a, uint##N##_t b) { \ + return a + b + (uint##N##_t)FLAG_C(c); \ + } \ + static inline uint##N##_t sbc##N(Cpu *c, uint##N##_t a, uint##N##_t b) { \ + return a - b - 1 + (uint##N##_t)FLAG_C(c); \ + } \ + static inline uint##N##_t adcs##N(Cpu *c, uint##N##_t a, uint##N##_t b) { \ + uint##N##_t cin = FLAG_C(c), r = a + b + cin; \ + int carry = cin ? r <= a : r < a; \ + c->nzcv = MKFLAGS(r >> (N - 1), r == 0, carry, ((a ^ r) & (b ^ r)) >> (N - 1)); \ + return r; \ + } \ + static inline uint##N##_t sbcs##N(Cpu *c, uint##N##_t a, uint##N##_t b) { \ + return adcs##N(c, a, (uint##N##_t)~b); \ + } +ADC_IMPL(64, ) +ADC_IMPL(32, ) + +// ---- bit ops +#define ROR64(v, r) ((uint64_t)(v) >> ((r) & 63) | (uint64_t)(v) << ((64 - ((r) & 63)) & 63)) +#define ROR32(v, r) ((uint32_t)(v) >> ((r) & 31) | (uint32_t)(v) << ((32 - ((r) & 31)) & 31)) +static inline uint64_t clz64(uint64_t v) { return v ? __builtin_clzll(v) : 64; } +static inline uint32_t clz32(uint32_t v) { return v ? __builtin_clz(v) : 32; } +static inline uint64_t cls64(uint64_t v) { return clz64(v ^ (uint64_t)((int64_t)v >> 1)) - 1; } +static inline uint32_t cls32(uint32_t v) { return clz32(v ^ (uint32_t)((int32_t)v >> 1)) - 1; } +static inline uint64_t rbit64(uint64_t v) { return __builtin_bitreverse64(v); } +static inline uint32_t rbit32(uint32_t v) { return __builtin_bitreverse32(v); } +static inline uint32_t rev16_32(uint32_t v) { return ((v & 0x00ff00ffu) << 8) | ((v >> 8) & 0x00ff00ffu); } +static inline uint64_t rev16_64(uint64_t v) { + return ((v & 0x00ff00ff00ff00ffull) << 8) | ((v >> 8) & 0x00ff00ff00ff00ffull); +} +static inline uint64_t rev32_64(uint64_t v) { + return ((uint64_t)__builtin_bswap32((uint32_t)v)) | ((uint64_t)__builtin_bswap32((uint32_t)(v >> 32)) << 32); +} +static inline uint64_t udiv64(uint64_t a, uint64_t b) { return b ? a / b : 0; } +static inline uint32_t udiv32(uint32_t a, uint32_t b) { return b ? a / b : 0; } +static inline uint64_t sdiv64(uint64_t a, uint64_t b) { + if (!b) return 0; + if ((int64_t)a == INT64_MIN && (int64_t)b == -1) return a; + return (uint64_t)((int64_t)a / (int64_t)b); +} +static inline uint32_t sdiv32(uint32_t a, uint32_t b) { + if (!b) return 0; + if ((int32_t)a == INT32_MIN && (int32_t)b == -1) return a; + return (uint32_t)((int32_t)a / (int32_t)b); +} + +// Portable FP/SIMD helpers for code lifted with --simd=c (default off AArch64; gen.py includes simd.h itself +// in that mode, so AArch64 hosts can build it too). +#if !defined(__cplusplus) && !defined(__aarch64__) +#include "simd.h" +#endif diff --git a/src/runtime/dispatch.c b/src/runtime/dispatch.c new file mode 100644 index 0000000..fc222df --- /dev/null +++ b/src/runtime/dispatch.c @@ -0,0 +1,254 @@ +// Guest address -> recompiled function lookup, and the host (HLE) function registry. +// Based on LASRecomp runtime/dispatch.c (MIT); TOTK: three modules linked statically into the executable. +#include +#include +#include + +#include "hle.h" +#include "rt.h" + +// Tables emitted by tools/lifter/gen.py for each module (_table.c). +#define GAME_TABLE(n) extern const uint64_t n##_fn_addrs[]; extern const GuestFn n##_fn_ptrs[]; extern const uint64_t n##_fn_count; +GAME_TABLE(main) +GAME_TABLE(subsdk0) +GAME_TABLE(sdk) + +typedef struct { const uint64_t *addrs; const GuestFn *fns; uint64_t count, lo, hi; } FnTable; +static FnTable tables[MAX_MODULES]; + +// HLE stubs live at fake addresses HLE_BASE + 16*i +#define MAX_HLE 4096 +typedef struct { const char *name; HostFn fn; void *native; char ret; } HleSlot; +static HleSlot hle_slots[MAX_HLE]; +static int hle_count; + +// Overrides of recompiled guest functions (by address) +#define MAX_OVERRIDE 1024 +static struct { uint64_t addr; HostFn fn; const char *name; } overrides[MAX_OVERRIDE]; +static int override_count; + +// Host implementations registered by name (HLE_FUNC, hle.h) +#if defined(__linux__) +extern const HleImplEntry __start_hle_impls[], __stop_hle_impls[]; +#else +// PE/COFF: the linker sorts .hle$a < .hle$m (entries) < .hle$z by the text after '$'. +__attribute__((section(".hle$a"), used)) static const HleImplEntry __start_hle_impls[1]; +__attribute__((section(".hle$z"), used)) static const HleImplEntry __stop_hle_impls[1]; +#endif + +static const HleImplEntry *hle_lookup_entry(const char *name) { + const HleImplEntry *start = __start_hle_impls, *stop = __stop_hle_impls; + __asm__("" : "+r"(start), "+r"(stop)); // separate objects: keep the compiler from reasoning about them + for (const HleImplEntry *p = start; p < stop; p++) + if (p->name && !strcmp(p->name, name)) return p; + return NULL; +} +const void *hle_lookup_entry_public(const char *name) { return hle_lookup_entry(name); } +HostFn hle_lookup_impl(const char *name) { + const HleImplEntry *e = hle_lookup_entry(name); + return e ? e->fn : NULL; +} + +// HLE_NATIVE: the slot being called (set by dispatch_slow, which every host call passes through: host stubs are not +// cached in the dispatch cache). +static __thread const HleSlot *t_native; +void hle_native_thunk(Cpu *c) { + const HleSlot *s = t_native; + uint64_t (*f)(uint64_t, uint64_t, uint64_t, uint64_t, uint64_t, uint64_t, uint64_t, uint64_t) = s->native; + uint64_t r = f(c->x[0], c->x[1], c->x[2], c->x[3], c->x[4], c->x[5], c->x[6], c->x[7]); + switch (s->ret) { + case 'v': break; + case 'b': c->x[0] = (uint8_t)r; break; + case 'w': c->x[0] = (uint32_t)r; break; + default: c->x[0] = r; break; + } +} + +static __thread uint64_t unimpl_pc; +static void hle_unimplemented(Cpu *c) { + int i = (int)((unimpl_pc - HLE_BASE) / 16); + FATAL("unimplemented host function %s (x0=%llx x1=%llx lr=%llx)\n", hle_slots[i].name, + (unsigned long long)c->x[0], (unsigned long long)c->x[1], (unsigned long long)c->x[30]); +} + +uint64_t hle_register(const char *name, HostFn fn) { + for (int i = 0; i < hle_count; i++) + if (!strcmp(hle_slots[i].name, name)) return HLE_BASE + 16ull * i; + if (hle_count == MAX_HLE) FATAL("too many HLE stubs\n"); + hle_slots[hle_count].name = strdup(name); + hle_slots[hle_count].fn = fn; + return HLE_BASE + 16ull * hle_count++; +} + +uint64_t hle_register_entry(const char *name, const void *entry) { + const HleImplEntry *e = entry; + uint64_t a = hle_register(name, e->fn); + HleSlot *s = &hle_slots[(a - HLE_BASE) / 16]; + s->native = e->native; + s->ret = e->ret; + return a; +} + +void hle_override(uint64_t addr, HostFn fn, const char *name) { + if (override_count == MAX_OVERRIDE) FATAL("too many overrides\n"); + overrides[override_count].addr = addr; + overrides[override_count].fn = fn; + overrides[override_count].name = name; + override_count++; +} + +static uint64_t hle_next = HLE_HEAP; +void *hle_alloc(uint64_t size) { + static pthread_mutex_t m = PTHREAD_MUTEX_INITIALIZER; + pthread_mutex_lock(&m); + uint64_t a = hle_next; + hle_next += (size + 63) & ~63ull; + if (hle_next > HLE_HEAP + HLE_HEAP_SIZE) FATAL("HLE heap exhausted\n"); + pthread_mutex_unlock(&m); + plat_commit(a, size); + memset((void *)a, 0, size); + return (void *)a; +} + +void dispatch_init(void) { + const uint64_t *addrs[] = {main_fn_addrs, subsdk0_fn_addrs, sdk_fn_addrs}; + const GuestFn *fns[] = {main_fn_ptrs, subsdk0_fn_ptrs, sdk_fn_ptrs}; + uint64_t counts[] = {main_fn_count, subsdk0_fn_count, sdk_fn_count}; + for (int t = 0; t < MAX_MODULES; t++) + tables[t] = (FnTable){addrs[t], fns[t], counts[t], addrs[t][0], addrs[t][counts[t] - 1]}; +} + +GuestFn dispatch_lookup(uint64_t addr) { + for (int t = 0; t < MAX_MODULES; t++) { + FnTable *ft = &tables[t]; + if (addr < ft->lo || addr > ft->hi) continue; + uint64_t lo = 0, hi = ft->count; + while (lo < hi) { + uint64_t mid = (lo + hi) / 2; + if (ft->addrs[mid] < addr) lo = mid + 1; + else hi = mid; + } + if (lo < ft->count && ft->addrs[lo] == addr) return ft->fns[lo]; + } + return NULL; +} + +// Small direct-mapped cache in front of the binary search +#define DCACHE 4096 +static __thread struct { uint64_t addr; GuestFn fn; } dcache[DCACHE]; + +static GuestFn dispatch_slow(Cpu *c) { + uint64_t pc = c->pc; + if (pc >= HLE_BASE && pc < HLE_BASE + HLE_SIZE) { + int i = (int)((pc - HLE_BASE) / 16); + if (i < hle_count) { + if (hle_slots[i].native) { t_native = &hle_slots[i]; return hle_native_thunk; } + if (hle_slots[i].fn) return hle_slots[i].fn; + unimpl_pc = pc; + return hle_unimplemented; + } + } + for (int i = 0; i < override_count; i++) + if (overrides[i].addr == pc) return overrides[i].fn; + GuestFn f = dispatch_lookup(pc); + if (!f) { + uint64_t off = 0; + const char *s = loader_symbol_at(c->x[30], &off); + FATAL("no function at %llx (lr=%llx %s+%llx)\n", (unsigned long long)pc, (unsigned long long)c->x[30], + s ? s : "?", (unsigned long long)off); + } + unsigned h = (unsigned)(pc >> 2) & (DCACHE - 1); + dcache[h].addr = pc; + dcache[h].fn = f; + return f; +} + +// Always tail-calls, so chains of indirect jumps don't grow the host stack. +void rt_dispatch(Cpu *c) { + unsigned h = (unsigned)(c->pc >> 2) & (DCACHE - 1); + GuestFn f = dcache[h].addr == c->pc ? dcache[h].fn : dispatch_slow(c); + __attribute__((musttail)) return f(c); +} + +void guest_call(Cpu *c, uint64_t addr) { + uint64_t lr = c->x[30]; + c->x[30] = 0; + c->pc = addr; + rt_dispatch(c); + c->x[30] = lr; +} + +void rt_udf(Cpu *c, uint64_t pc) { + (void)c; + FATAL("undefined instruction / trap at %llx\n", (unsigned long long)pc); +} + +void rt_brk(Cpu *c, uint32_t imm, uint64_t pc) { + (void)c; + FATAL("brk #%u at %llx\n", imm, (unsigned long long)pc); +} + +void rt_fatal_trace(void) { + plat_print_backtrace(); + fflush(stderr); + abort(); +} + +// ---- crash reporting (platform/*.c calls these) + +// Host pc -> recompiled guest function containing it ("f_"). +static int cmp_host(const void *a, const void *b) { + uint64_t x = ((const uint64_t *)a)[0], y = ((const uint64_t *)b)[0]; + return x < y ? -1 : x > y; +} +const char *rt_host_fn_name(uint64_t pc, uint64_t *offset) { + static uint64_t (*byhost)[2]; // {host address, guest address}, sorted by host address + static uint64_t n; + static __thread char name[32]; + if (!byhost) { + uint64_t total = 0, k = 0; + for (int t = 0; t < MAX_MODULES; t++) total += tables[t].count; + uint64_t (*v)[2] = malloc(total * sizeof *v); + if (!v) return NULL; + for (int t = 0; t < MAX_MODULES; t++) + for (uint64_t i = 0; i < tables[t].count; i++) { + v[k][0] = (uint64_t)(uintptr_t)tables[t].fns[i]; + v[k++][1] = tables[t].addrs[i]; + } + qsort(v, k, sizeof *v, cmp_host); + n = k; + byhost = v; + } + uint64_t lo = 0, hi = n; // last entry with host address <= pc + while (lo < hi) { + uint64_t mid = (lo + hi) / 2; + if (byhost[mid][0] <= pc) lo = mid + 1; + else hi = mid; + } + if (!lo || pc - byhost[lo - 1][0] > 0x100000) return NULL; + snprintf(name, sizeof name, "f_%llx", (unsigned long long)byhost[lo - 1][1]); + if (offset) *offset = pc - byhost[lo - 1][0]; + return name; +} + +bool rt_fault_watch(uint64_t fa, uint64_t pc) { (void)fa; (void)pc; return false; } + +void (*rt_fault_extra)(void); // the host's guest-state report (registers, guest call chain), set at startup + +void rt_fault_report(const char *what, uint64_t fa, uint64_t pc, uint64_t lr) { + uint64_t po = 0, lo = 0; + const char *ps = rt_host_fn_name(pc, &po), *ls = rt_host_fn_name(lr, &lo); + if (!ps) ps = plat_symbol(pc, &po); + fprintf(stderr, "\n*** %s accessing %#llx\n host pc %llx (%s+%llx)\n host lr %llx (%s)\n", what, + (unsigned long long)fa, (unsigned long long)pc, ps ? ps : "?", (unsigned long long)po, + (unsigned long long)lr, ls ? ls : "?"); + if (rt_fault_extra) rt_fault_extra(); + rt_fatal_trace(); +} + +// ---- function-entry profiler hook (cpu.h FN_ENTER): off unless a profiler sets g_fnprof +volatile int g_fnprof; +void fnprof_hit(uint64_t addr, uint64_t lr, uint64_t x0) { (void)addr; (void)lr; (void)x0; } + +// ---- CNTPCT_EL0: the Switch's 19.2 MHz system counter +uint64_t rt_ticks(void) { return (uint64_t)((unsigned __int128)plat_now_ns() * 12 / 625); } diff --git a/src/runtime/hle.h b/src/runtime/hle.h new file mode 100644 index 0000000..7d8625d --- /dev/null +++ b/src/runtime/hle.h @@ -0,0 +1,29 @@ +// Registration of host implementations for guest symbols (from LASRecomp runtime/hle.h, MIT). +#pragma once +#include "rt.h" + +// A host implementation of a guest symbol: either a function on the guest registers (HLE_FUNC), or an ordinary C +// function with integer/pointer arguments (HLE_NATIVE), called with x0..x7 and its result put in x0. +typedef struct { const char *name; HostFn fn; void *native; char ret; } HleImplEntry; +// HLE_NATIVE result kinds: 'v' void, 'b' bool/u8, 'w' 32-bit (int, unsigned, Result), 'x' 64-bit or pointer. +// Win64 passes the first four integer arguments in registers and the rest on the stack, so calling any +// integer/pointer-only function as fn(u64 x 8) is correct; float/double arguments or results need HLE_FUNC. +void hle_native_thunk(Cpu *c); + +// HLE_FUNC("mangled_symbol", c_function) binds a guest import to a host function. +#define HLE_CAT2(a, b) a##b +#define HLE_CAT(a, b) HLE_CAT2(a, b) +#if defined(__linux__) +#define HLE_SECTION "hle_impls" // ELF: the linker defines __start_ / __stop_ for C-identifier section names +#else +#define HLE_SECTION ".hle$m" // PE/COFF: between the .hle$a / .hle$z markers in dispatch.c +#endif +#define HLE_FUNC(sym, fn) \ + static void fn(Cpu *c); \ + __attribute__((used, section(HLE_SECTION))) static const HleImplEntry \ + HLE_CAT(hle_entry_, __LINE__) = {sym, fn, 0, 0}; \ + static void fn(Cpu *c) + +#define HLE_NATIVE(sym, cfn, kind) \ + __attribute__((used, section(HLE_SECTION))) static const HleImplEntry \ + HLE_CAT(hle_entry_, __LINE__) = {sym, hle_native_thunk, (void *)(cfn), kind}; diff --git a/src/runtime/loader.c b/src/runtime/loader.c new file mode 100644 index 0000000..06d30ea --- /dev/null +++ b/src/runtime/loader.c @@ -0,0 +1,227 @@ +// NSO loading, relocation and symbol resolution. +// Based on LASRecomp runtime/loader.c (MIT); TOTK: three modules, DT_RELR, DT_GNU_HASH. +#include +#include + +#include "hle.h" +#include "rt.h" + +Module g_modules[MAX_MODULES]; +int g_nmodules; + +enum { DT_NULL = 0, DT_PLTRELSZ = 2, DT_HASH = 4, DT_STRTAB = 5, DT_SYMTAB = 6, DT_RELA = 7, DT_RELASZ = 8, + DT_INIT = 12, DT_FINI = 13, DT_JMPREL = 23, DT_INIT_ARRAY = 25, DT_FINI_ARRAY = 26, DT_INIT_ARRAYSZ = 27, + DT_FINI_ARRAYSZ = 28, DT_RELRSZ = 35, DT_RELR = 36, DT_GNU_HASH = 0x6FFFFEF5 }; +enum { R_ABS64 = 257, R_GLOB_DAT = 1025, R_JUMP_SLOT = 1026, R_RELATIVE = 1027 }; + +typedef struct { uint32_t name; uint8_t info, other; uint16_t shndx; uint64_t value, size; } Elf64Sym; +typedef struct { uint64_t off, info; int64_t addend; } Elf64Rela; + +static size_t lz4_decompress(const uint8_t *src, size_t srclen, uint8_t *dst, size_t dstlen) { + const uint8_t *s = src, *se = src + srclen; + uint8_t *d = dst, *de = dst + dstlen; + while (s < se) { + uint8_t tok = *s++; + size_t lit = tok >> 4; + if (lit == 15) { uint8_t b; do { b = *s++; lit += b; } while (b == 255); } + if (d + lit > de || s + lit > se) break; + memcpy(d, s, lit); + d += lit; s += lit; + if (s >= se) break; + size_t off = s[0] | (s[1] << 8); + s += 2; + size_t ml = tok & 15; + if (ml == 15) { uint8_t b; do { b = *s++; ml += b; } while (b == 255); } + ml += 4; + uint8_t *m = d - off; + if (m < dst || d + ml > de) break; + while (ml--) *d++ = *m++; + } + return d - dst; +} + +static uint8_t *read_file(const char *path, size_t *len) { + FILE *f = fopen(path, "rb"); + if (!f) return NULL; + fseek(f, 0, SEEK_END); + *len = ftell(f); + fseek(f, 0, SEEK_SET); + uint8_t *p = malloc(*len); + if (p && fread(p, 1, *len, f) != *len) { free(p); p = NULL; } + fclose(f); + return p; +} + +// Symbol count from DT_GNU_HASH: one past the highest symbol any bucket chain reaches. +static uint64_t gnu_hash_count(uint64_t table) { + const uint32_t *h = (const uint32_t *)table; + uint32_t nbuckets = h[0], symoffset = h[1], bloom = h[2]; + const uint32_t *buckets = (const uint32_t *)(table + 16 + bloom * 8ull), *chains = buckets + nbuckets; + uint32_t last = 0; + for (uint32_t i = 0; i < nbuckets; i++) if (buckets[i] > last) last = buckets[i]; + if (last < symoffset) return symoffset; + while (!(chains[last - symoffset] & 1)) last++; + return last + 1; +} + +static void load_nso(Module *m, const char *path, uint64_t base) { + size_t len; + uint8_t *raw = read_file(path, &len); + if (!raw || len < 0x100 || memcmp(raw, "NSO0", 4)) FATAL("cannot load %s\n", path); + uint32_t flags = *(uint32_t *)(raw + 0xC); + uint32_t seg[3][3]; // file_off, mem_off, size + for (int i = 0; i < 3; i++) memcpy(seg[i], raw + 0x10 + i * 0x10, 12); + uint32_t bss = *(uint32_t *)(raw + 0x3C); + uint64_t size = (seg[2][1] + seg[2][2] + bss + 0xFFF) & ~0xFFFull; + plat_commit(base, size); // inside the guest window: zero-filled, read/write + for (int i = 0; i < 3; i++) { + uint32_t csize = *(uint32_t *)(raw + 0x60 + i * 4); + uint8_t *dst = (uint8_t *)base + seg[i][1]; + if (flags & (1 << i)) { + size_t got = lz4_decompress(raw + seg[i][0], csize, dst, seg[i][2]); + if (got != seg[i][2]) FATAL("lz4 %s seg %d: %zu != %u\n", path, i, got, seg[i][2]); + } else { + memcpy(dst, raw + seg[i][0], seg[i][2]); + } + } + free(raw); + m->base = base; + m->size = size; + m->text_off = seg[0][1]; m->text_size = seg[0][2]; + m->ro_off = seg[1][1]; m->ro_size = seg[1][2]; + m->data_off = seg[2][1]; m->data_size = seg[2][2] + bss; + + uint32_t mod0 = *(uint32_t *)(base + 4); + uint64_t mod = base + mod0; + if (memcmp((void *)mod, "MOD0", 4)) FATAL("%s: no MOD0\n", path); + m->dynamic = mod + *(int32_t *)(mod + 4); + uint64_t gnu_hash = 0; + for (uint64_t *d = (uint64_t *)m->dynamic; d[0] != DT_NULL; d += 2) { + switch (d[0]) { + case DT_SYMTAB: m->symtab = base + d[1]; break; + case DT_STRTAB: m->strtab = base + d[1]; break; + case DT_HASH: m->nsyms = *(uint32_t *)(base + d[1] + 4); break; + case DT_GNU_HASH: gnu_hash = base + d[1]; break; + case DT_INIT: m->init = base + d[1]; break; + case DT_FINI: m->fini = base + d[1]; break; + case DT_INIT_ARRAY: m->init_array = base + d[1]; break; + case DT_INIT_ARRAYSZ: m->init_array_sz = d[1]; break; + case DT_FINI_ARRAY: m->fini_array = base + d[1]; break; + case DT_FINI_ARRAYSZ: m->fini_array_sz = d[1]; break; + } + } + if (!m->nsyms && gnu_hash) m->nsyms = gnu_hash_count(gnu_hash); + LOG("loaded %-7s at %llx size %llx syms %llu\n", m->name, (unsigned long long)base, (unsigned long long)size, + (unsigned long long)m->nsyms); +} + +static Elf64Sym *sym(Module *m, uint64_t i) { return (Elf64Sym *)m->symtab + i; } +static const char *symname(Module *m, Elf64Sym *s) { return (const char *)(m->strtab + s->name); } + +uint64_t loader_find_symbol(const char *name) { + for (int mi = 0; mi < g_nmodules; mi++) { + Module *m = &g_modules[mi]; + for (uint64_t i = 1; i < m->nsyms; i++) { + Elf64Sym *s = sym(m, i); + if (s->shndx && !strcmp(symname(m, s), name)) return m->base + s->value; + } + } + return 0; +} + +const char *loader_symbol_at(uint64_t addr, uint64_t *offset) { + const char *best = NULL; + uint64_t best_addr = 0; + for (int mi = 0; mi < g_nmodules; mi++) { + Module *m = &g_modules[mi]; + if (addr < m->base || addr >= m->base + m->size) continue; + for (uint64_t i = 1; i < m->nsyms; i++) { + Elf64Sym *s = sym(m, i); + uint64_t a = m->base + s->value; + if (s->shndx && (s->info & 0xF) == 2 && a <= addr && a > best_addr) { + best = symname(m, s); + best_addr = a; + } + } + } + if (offset) *offset = addr - best_addr; + return best; +} + +static uint64_t resolve(Module *m, uint32_t symidx) { + Elf64Sym *s = sym(m, symidx); + const char *name = symname(m, s); + // A host implementation wins even over the module's own definition: an SDK module calls its own exported + // functions through its PLT, so this is where its internal calls are replaced too (as in MacNativeGames). + const void *impl = name[0] ? hle_lookup_entry_public(name) : NULL; + if (impl) return hle_register_entry(name, impl); + if (s->shndx) return m->base + s->value; + uint64_t a = loader_find_symbol(name); + if (a) return a; + if ((s->info >> 4) == 2) return 0; // weak undefined + return hle_register(name, NULL); +} + +static void relocate(Module *m) { + uint64_t rela = 0, relasz = 0, jmprel = 0, pltsz = 0, relr = 0, relrsz = 0; + for (uint64_t *d = (uint64_t *)m->dynamic; d[0] != DT_NULL; d += 2) { + if (d[0] == DT_RELA) rela = m->base + d[1]; + if (d[0] == DT_RELASZ) relasz = d[1]; + if (d[0] == DT_JMPREL) jmprel = m->base + d[1]; + if (d[0] == DT_PLTRELSZ) pltsz = d[1]; + if (d[0] == DT_RELR) relr = m->base + d[1]; + if (d[0] == DT_RELRSZ) relrsz = d[1]; + } + int unresolved = 0; + uint64_t count = 0; + for (int pass = 0; pass < 2; pass++) { + Elf64Rela *r = (Elf64Rela *)(pass ? jmprel : rela); + uint64_t n = (pass ? pltsz : relasz) / sizeof(Elf64Rela); + for (uint64_t i = 0; i < n; i++, count++) { + uint64_t *where = (uint64_t *)(m->base + r[i].off); + uint32_t type = (uint32_t)r[i].info, si = r[i].info >> 32; + switch (type) { + case R_RELATIVE: *where = m->base + r[i].addend; break; + case R_ABS64: + case R_GLOB_DAT: + case R_JUMP_SLOT: { + uint64_t s = resolve(m, si); + *where = s ? s + r[i].addend : 0; + if (s >= HLE_BASE && s < HLE_BASE + HLE_SIZE) unresolved++; + break; + } + default: FATAL("%s: reloc type %u\n", m->name, type); + } + } + } + // DT_RELR: an even entry is an address (relocated, next word follows); an odd entry is a bitmap of the + // following 63 words. Each relocated word already holds its addend. + uint64_t where = 0; + for (uint64_t i = 0; i < relrsz / 8; i++) { + uint64_t e = ((uint64_t *)relr)[i]; + if (!(e & 1)) { + *(uint64_t *)(m->base + e) += m->base; + where = e + 8; + count++; + } else { + for (int bit = 0; bit < 63; bit++) + if (e >> (bit + 1) & 1) { *(uint64_t *)(m->base + where + bit * 8ull) += m->base; count++; } + where += 63 * 8; + } + } + LOG("relocated %-7s %llu relocations (%d imports bound to host stubs)\n", m->name, (unsigned long long)count, + unresolved); +} + +void loader_load(const char *exefs) { + char path[1024]; + static const struct { const char *name; uint64_t base; } mods[] = { + {"main", MAIN_BASE}, {"subsdk0", SUBSDK0_BASE}, {"sdk", SDK_BASE}}; + for (unsigned i = 0; i < MAX_MODULES; i++) { + Module *m = &g_modules[g_nmodules++]; + m->name = mods[i].name; + snprintf(path, sizeof path, "%s/%s", exefs, m->name); + load_nso(m, path, mods[i].base); + } + for (int i = 0; i < g_nmodules; i++) relocate(&g_modules[i]); +} diff --git a/src/runtime/platform.h b/src/runtime/platform.h new file mode 100644 index 0000000..cdc28df --- /dev/null +++ b/src/runtime/platform.h @@ -0,0 +1,74 @@ +// Host OS layer: everything the runtime needs that differs between macOS and Windows. +// Implementations: runtime/platform/macos.c, runtime/platform/windows.c (the Makefile picks one). +#pragma once +#include +#include +#include + +#ifdef _WIN32 +#include "platform/win_compat.h" +#elif defined(__linux__) +#include "platform/linux_compat.h" +#endif + +#ifdef __cplusplus +extern "C" { +#endif + +// ---- guest address space +// Claim [lo, hi) for the guest at exactly that address, readable/writable (pages may be backed lazily). +bool plat_reserve_guest(uint64_t lo, uint64_t hi); +// Make sure [addr, addr+size) inside the guest window is backed (before the OS itself writes there, e.g. +// file reads); a no-op where the whole window is backed from the start. +void plat_commit(uint64_t addr, uint64_t size); +// In-place reboot: every page of [lo, hi) back to zero (and unbacked), the window still claimed. +bool plat_reset_guest(uint64_t lo, uint64_t hi); +// Pages of [addr, addr+size) in the guest window the CPU wrote since the last call over them (write tracking: Windows +// MEM_WRITE_WATCH), their addresses into pages[] (at most max), and the tracking reset for the range. Returns the +// count (max: maybe more), or -1 where there is no tracking (macOS, Linux). Page size: *page. +int plat_written_pages(uint64_t addr, uint64_t size, uint64_t *pages, int max, uint64_t *page); +enum { PLAT_PROT_NONE = 0, PLAT_PROT_READ = 1, PLAT_PROT_RW = 3 }; +int plat_protect(uint64_t addr, uint64_t size, int prot); // 0 on success +#define PLAT_PAGE 0x4000ull // largest page size we deal with (macOS 16K; multiple of Windows' 4K) + +// ---- time +uint64_t plat_now_ns(void); // monotonic +void plat_sleep_until_ns(uint64_t t); // precise wait for plat_now_ns() >= t +void plat_thread_realtime(uint64_t period_ns); +void plat_thread_boost(void); // a thread the frame waits for (the replay thread): above the game's workers // periodic timing thread (vsync): don't coalesce its timers + +// ---- process +void plat_init(void); // first thing in main() +bool plat_exe_path(char *buf, size_t size); +void plat_exec(char *const argv[]); // replace the process (NULL = same command line); returns only on failure +#if defined(_WIN32) || defined(__linux__) +void plat_terminate(void); // end the process now, skipping DLL teardown (exit status 0) +#endif + +// Child processes (seed generator, game-file extraction). stdout + stderr go to log_path (NULL = /dev/null). +typedef intptr_t PlatProc; // 0 = failed to start +PlatProc plat_spawn(const char *const argv[], const char *log_path, bool append_log); +int plat_proc_wait(PlatProc p, bool block); // exit code, or -1 while still running (block = false) +void plat_proc_kill(PlatProc p); // end it and what it started (then it's gone: no wait) +bool plat_remove_tree(const char *dir); // rm -rf +#ifdef _WIN32 +#define PLAT_EXE_SUFFIX ".exe" +#define PLAT_VENV_PYTHON "venv/Scripts/python.exe" // relative to the venv's parent +#else +#define PLAT_EXE_SUFFIX "" +#define PLAT_VENV_PYTHON "venv/bin/python" +#endif + +// ---- diagnostics +// Name of the host function containing pc (guest functions are f_), NULL if unknown. +const char *plat_symbol(uint64_t pc, uint64_t *offset); +void plat_print_backtrace(void); +void plat_fatal_notice(void); // a fatal error is about to end the process: tell the user (Windows release: message box) +void plat_crash_handler_install(void); // per thread where the OS needs it (alternate signal stack) +// Called by the crash handler (runtime/dispatch.c): true = fault handled (watchpoint), resume. +bool rt_fault_watch(uint64_t fault_addr, uint64_t host_pc); +void rt_fault_report(const char *what, uint64_t fault_addr, uint64_t host_pc, uint64_t host_lr) __attribute__((noreturn)); + +#ifdef __cplusplus +} +#endif diff --git a/src/runtime/platform/linux.c b/src/runtime/platform/linux.c new file mode 100644 index 0000000..d263687 --- /dev/null +++ b/src/runtime/platform/linux.c @@ -0,0 +1,272 @@ +// Linux host layer (runtime/platform.h). x86-64 and aarch64. +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../platform.h" + +#ifndef MAP_FIXED_NOREPLACE +#define MAP_FIXED_NOREPLACE 0x100000 +#endif + +// ---- guest address space +// Everything readable/writable from the start, like macOS: the guest maps at 4K granularity, and untouched pages +// cost nothing (MAP_NORESERVE: the 12 GB window doesn't count against overcommit). +bool plat_reserve_guest(uint64_t lo, uint64_t hi) { + void *p = mmap((void *)(uintptr_t)lo, hi - lo, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE | MAP_FIXED_NOREPLACE, -1, 0); + if (p == MAP_FAILED) return false; + if (p != (void *)(uintptr_t)lo) { munmap(p, hi - lo); return false; } // (kernels before 4.17: just a hint) + return true; +} + +// Fresh zero pages over the whole window (the old ones are released at once). +bool plat_reset_guest(uint64_t lo, uint64_t hi) { + return mmap((void *)(uintptr_t)lo, hi - lo, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE | MAP_FIXED, + -1, 0) == (void *)(uintptr_t)lo; +} + +void plat_commit(uint64_t addr, uint64_t size) { (void)addr; (void)size; } + +int plat_written_pages(uint64_t addr, uint64_t size, uint64_t *pages, int max, uint64_t *page) { // (no write tracking) + (void)addr; (void)size; (void)pages; (void)max; (void)page; + return -1; +} + +int plat_protect(uint64_t addr, uint64_t size, int prot) { + int p = prot == PLAT_PROT_NONE ? PROT_NONE : prot == PLAT_PROT_READ ? PROT_READ : PROT_READ | PROT_WRITE; + return mprotect((void *)(uintptr_t)addr, size, p); +} + +// ---- time +uint64_t plat_now_ns(void) { + struct timespec ts; + clock_gettime(CLOCK_MONOTONIC, &ts); + return (uint64_t)ts.tv_sec * 1000000000u + (uint64_t)ts.tv_nsec; +} + +void plat_sleep_until_ns(uint64_t t) { + struct timespec ts = {.tv_sec = (time_t)(t / 1000000000u), .tv_nsec = (long)(t % 1000000000u)}; + while (clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &ts, NULL) == EINTR) {} +} + +// Real-time scheduling where the user may have it (rtkit / limits.conf), else just a higher priority: keeps the +// vsync thread on time while the guest threads load every core. +void plat_thread_boost(void) {} // [temporary] Linux / Steam Deck: nice / SCHED_* for the replay thread - with the Deck port + +void plat_thread_realtime(uint64_t period_ns) { + (void)period_ns; + struct sched_param sp = {.sched_priority = 1}; + if (pthread_setschedparam(pthread_self(), SCHED_FIFO, &sp) == 0) return; + setpriority(PRIO_PROCESS, (id_t)gettid(), -10); // (per thread on Linux; fails quietly without CAP_SYS_NICE) +} + +// ---- process +// Started from a desktop (the AppImage) there is no terminal: stdout / stderr go to lasrecomp.log in the data folder +// (runtime/paths.h; rt_paths_init runs later, so the same rule is applied here). Processes started by this one (the +// start screen re-executing itself for the game) append to the same log. +static char g_log_path[1024]; + +static void log_to_file_without_terminal(void) { + if (isatty(STDERR_FILENO)) return; + char dir[900]; + const char *env = getenv("LARECOMP_DATA_ROOT"), *xdg = getenv("XDG_DATA_HOME"), *home = getenv("HOME"); + struct stat st; + if (env && *env) snprintf(dir, sizeof dir, "%s", env); + else if (stat("extracted", &st) == 0 && S_ISDIR(st.st_mode)) return; // dev mode: wherever stderr goes already + else if (xdg && *xdg == '/') snprintf(dir, sizeof dir, "%s/lasrecomp", xdg); + else snprintf(dir, sizeof dir, "%s/.local/share/lasrecomp", home && *home ? home : "/tmp"); + for (char *c = dir + 1; *c; c++) + if (*c == '/') { *c = 0; mkdir(dir, 0755); *c = '/'; } + mkdir(dir, 0755); + snprintf(g_log_path, sizeof g_log_path, "%s/lasrecomp.log", dir); + bool append = getenv("LASRECOMP_LOG_APPEND") != NULL; + if (!freopen(g_log_path, append ? "a" : "w", stderr)) { g_log_path[0] = 0; return; } + dup2(STDERR_FILENO, STDOUT_FILENO); + setenv("LASRECOMP_LOG_APPEND", "1", 1); +} + +#if defined(__x86_64__) +// The recompiled game code is built for x86-64-v3 (Makefile GENARCH: AVX2, FMA, BMI2): say so instead of crashing +// with an illegal instruction on an older CPU. +static void require_cpu(void) { + __builtin_cpu_init(); + if (__builtin_cpu_supports("avx2") && __builtin_cpu_supports("fma") && __builtin_cpu_supports("bmi2")) return; + fprintf(stderr, "FATAL: LAS Recomp needs a CPU with AVX2 (Intel Haswell / 2013 or newer, AMD Excavator / Zen or " + "newer).\n"); + exit(1); +} +#else +static void require_cpu(void) {} +#endif + +void plat_init(void) { + log_to_file_without_terminal(); + require_cpu(); +} + +// (inside an AppImage: the binary in its mount, which stays mounted while this process lives - re-exec included) +bool plat_exe_path(char *buf, size_t size) { + ssize_t n = readlink("/proc/self/exe", buf, size - 1); + if (n <= 0) return false; + buf[n] = 0; + return true; +} + +// Our own command line, from /proc/self/cmdline (NUL-separated). +static char **self_argv(void) { + static char buf[32768]; + static char *argv[256]; + int fd = open("/proc/self/cmdline", O_RDONLY); + if (fd < 0) return NULL; + ssize_t n = read(fd, buf, sizeof buf - 1); + close(fd); + if (n <= 0) return NULL; + buf[n] = 0; + int k = 0; + for (char *p = buf; p < buf + n && k < 255; p += strlen(p) + 1) argv[k++] = p; + argv[k] = NULL; + return argv; +} + +void plat_exec(char *const argv[]) { + if (argv) { execv(argv[0], argv); return; } + char path[4096]; + char **av = self_argv(); + if (av && plat_exe_path(path, sizeof path)) execv(path, av); +} + +// ---- diagnostics +// Exported symbols only (the Makefile links with -rdynamic); frames in the generated code are left to the caller, +// which maps them to guest functions (runtime/dispatch.c). Static runtime functions come out as the nearest guest +// function, so backtrace lines also give the offset in the executable (addr2line -fe lasrecomp ). +const char *plat_symbol(uint64_t pc, uint64_t *offset) { + Dl_info di = {0}; + if (!dladdr((void *)(uintptr_t)pc, &di) || !di.dli_sname) return NULL; + if (offset) *offset = pc - (uint64_t)(uintptr_t)di.dli_saddr; + return di.dli_sname; +} + +// Fatal error without a terminal: say so in a message box rather than vanishing (the details are in the log). +void plat_fatal_notice(void) { + if (!g_log_path[0]) return; + fflush(stderr); + char text[1200]; + snprintf(text, sizeof text, "LAS Recomp stopped because of an error.\n\nDetails: %s", g_log_path); + SDL_ShowSimpleMessageBox(SDL_MESSAGEBOX_ERROR, "LAS Recomp", text, NULL); +} + +const char *rt_host_fn_name(uint64_t pc, uint64_t *offset); // runtime/dispatch.c + +void plat_print_backtrace(void) { + void *bt[64]; + int n = backtrace(bt, 64); + Dl_info self = {0}; + dladdr((void *)plat_print_backtrace, &self); + for (int i = 0; i < n; i++) { + uint64_t pc = (uint64_t)(uintptr_t)bt[i], off = 0; + Dl_info di = {0}; + if (!dladdr(bt[i], &di)) di.dli_fname = NULL; + bool ours = di.dli_fbase && di.dli_fbase == self.dli_fbase; + const char *s = plat_symbol(pc, &off); + if (!s && ours) s = rt_host_fn_name(pc, &off); + const char *mod = di.dli_fname ? (strrchr(di.dli_fname, '/') ? strrchr(di.dli_fname, '/') + 1 : di.dli_fname) : "?"; + uint64_t moff = pc - (uint64_t)(uintptr_t)di.dli_fbase; + if (s && ours) fprintf(stderr, " #%-2d %016llx %s+%llx (lasrecomp+%llx)\n", i, (unsigned long long)pc, s, + (unsigned long long)off, (unsigned long long)moff); + else if (s) fprintf(stderr, " #%-2d %016llx %s+%llx (%s)\n", i, (unsigned long long)pc, s, (unsigned long long)off, mod); + else fprintf(stderr, " #%-2d %016llx %s+%llx\n", i, (unsigned long long)pc, mod, (unsigned long long)moff); + } +} + +static void crash_handler(int sig, siginfo_t *si, void *ctx) { + ucontext_t *uc = ctx; + static volatile int depth; + if (sig == SIGABRT && depth++ > 0) { signal(SIGABRT, SIG_DFL); abort(); } // (nested: just die) + uint64_t fa = (uint64_t)(uintptr_t)si->si_addr; +#if defined(__aarch64__) + uint64_t pc = uc->uc_mcontext.pc, lr = uc->uc_mcontext.regs[30]; +#else + // x86-64 has no link register: report the return address on top of the stack (valid at function entry) + uint64_t pc = (uint64_t)uc->uc_mcontext.gregs[REG_RIP], sp = (uint64_t)uc->uc_mcontext.gregs[REG_RSP], lr = 0; + if (sig != SIGSEGV || (fa & ~0xFFFull) != (sp & ~0xFFFull)) lr = *(uint64_t *)(uintptr_t)sp; +#endif + if (rt_fault_watch(fa, pc)) return; + char what[32]; + snprintf(what, sizeof what, "signal %d", sig); + rt_fault_report(what, fa, pc, lr); +} + +void plat_crash_handler_install(void) { + stack_t ss = {.ss_sp = malloc(1 << 20), .ss_size = 1 << 20}; + sigaltstack(&ss, NULL); + struct sigaction sa = {0}; + sa.sa_sigaction = crash_handler; + sa.sa_flags = SA_SIGINFO | SA_ONSTACK; + sigaction(SIGSEGV, &sa, NULL); + sigaction(SIGBUS, &sa, NULL); + sigaction(SIGILL, &sa, NULL); + sigaction(SIGFPE, &sa, NULL); + sigaction(SIGABRT, &sa, NULL); +} + +// ---- child processes +#include +#include +#include +extern char **environ; + +PlatProc plat_spawn(const char *const argv[], const char *log_path, bool append_log) { + posix_spawn_file_actions_t fa; + posix_spawn_file_actions_init(&fa); + posix_spawn_file_actions_addopen(&fa, 1, log_path ? log_path : "/dev/null", + O_WRONLY | O_CREAT | (append_log ? O_APPEND : O_TRUNC), 0644); + posix_spawn_file_actions_adddup2(&fa, 1, 2); + posix_spawnattr_t at; // (its own process group: plat_proc_kill ends what it starts, e.g. a compiler's own child) + posix_spawnattr_init(&at); + posix_spawnattr_setflags(&at, POSIX_SPAWN_SETPGROUP); + posix_spawnattr_setpgroup(&at, 0); + pid_t pid; + int rc = posix_spawn(&pid, argv[0], &fa, &at, (char *const *)argv, environ); + posix_spawnattr_destroy(&at); + posix_spawn_file_actions_destroy(&fa); + return rc ? 0 : (PlatProc)pid; +} + +void plat_proc_kill(PlatProc p) { + kill(-(pid_t)p, SIGKILL); + waitpid((pid_t)p, NULL, 0); +} + +int plat_proc_wait(PlatProc p, bool block) { + int st = 0; + pid_t r = waitpid((pid_t)p, &st, block ? 0 : WNOHANG); + if (r == 0) return -1; + if (r < 0) return 255; + return WIFEXITED(st) ? WEXITSTATUS(st) : 255; +} + +static int rm_entry(const char *path, const struct stat *sb, int flag, struct FTW *ftw) { + (void)sb, (void)flag, (void)ftw; + return remove(path); +} +bool plat_remove_tree(const char *dir) { return nftw(dir, rm_entry, 16, FTW_DEPTH | FTW_PHYS) == 0; } + +// End the process at once, skipping the library teardown: as on Windows, the GPU driver's threads crashed in exit() +// while the game's threads still submitted work. +void plat_terminate(void) { _exit(0); } diff --git a/src/runtime/platform/linux_compat.h b/src/runtime/platform/linux_compat.h new file mode 100644 index 0000000..806f1bf --- /dev/null +++ b/src/runtime/platform/linux_compat.h @@ -0,0 +1,37 @@ +// What glibc before 2.38 lacks (included via runtime/platform.h on Linux only): the AppImage is built against +// Ubuntu 22.04's glibc 2.35 (no strlcpy / strlcat until 2.38, no arc4random until 2.36). +#pragma once +#include +#include +#include + +#if !__GLIBC_PREREQ(2, 38) +static inline size_t strlcpy(char *d, const char *s, size_t n) { + size_t len = strlen(s); + if (n) { + size_t k = len < n - 1 ? len : n - 1; + memcpy(d, s, k); + d[k] = 0; + } + return len; +} +static inline size_t strlcat(char *d, const char *s, size_t n) { + size_t dl = strnlen(d, n); + if (dl == n) return n + strlen(s); + return dl + strlcpy(d + dl, s, n - dl); +} +#endif + +#if !__GLIBC_PREREQ(2, 36) +static inline uint32_t arc4random(void) { + uint32_t v = 0; + while (getrandom(&v, sizeof v, 0) != sizeof v) {} + return v; +} +static inline uint32_t arc4random_uniform(uint32_t n) { + if (n < 2) return 0; + uint32_t min = -n % n, r; // reject the low values that would bias the modulo + do r = arc4random(); while (r < min); + return r % n; +} +#endif diff --git a/src/runtime/platform/win_compat.h b/src/runtime/platform/win_compat.h new file mode 100644 index 0000000..913a5c9 --- /dev/null +++ b/src/runtime/platform/win_compat.h @@ -0,0 +1,75 @@ +// POSIX bits mingw-w64 lacks (included via runtime/platform.h on Windows only). +// Build with -D_FILE_OFFSET_BITS=64 so off_t / stat sizes are 64-bit (romfs.bin is > 4 GB). +#pragma once +#include +#include +#include +#include +#include +#include +#include +#include + +// Positional read/write (thread-safe: does not move the descriptor's file pointer). +ssize_t pread(int fd, void *buf, size_t n, off_t off); +ssize_t pwrite(int fd, const void *buf, size_t n, off_t off); + +static inline int setenv(const char *k, const char *v, int overwrite) { + if (!overwrite && getenv(k)) return 0; + return _putenv_s(k, v); +} +static inline int unsetenv(const char *k) { return _putenv_s(k, ""); } + +static inline int fsync(int fd) { return _commit(fd); } + +#ifndef _CRT_RAND_S +int __cdecl rand_s(unsigned int *v); // (declared by stdlib.h only with _CRT_RAND_S) +#endif +static inline uint32_t arc4random(void) { + unsigned v = 0; + rand_s(&v); + return v; +} + +static inline uint32_t arc4random_uniform(uint32_t n) { + if (n < 2) return 0; + uint32_t min = -n % n, r; // reject the low values that would bias the modulo + do r = arc4random(); while (r < min); + return r % n; +} + +static inline size_t strlcpy(char *d, const char *s, size_t n) { + size_t len = strlen(s); + if (n) { + size_t k = len < n - 1 ? len : n - 1; + memcpy(d, s, k); + d[k] = 0; + } + return len; +} +static inline size_t strlcat(char *d, const char *s, size_t n) { + size_t dl = strnlen(d, n); + if (dl == n) return n + strlen(s); + return dl + strlcpy(d + dl, s, n - dl); +} +static inline char *strsep(char **sp, const char *delim) { + char *s = *sp; + if (!s) return NULL; + char *e = s + strcspn(s, delim); + *sp = *e ? e + 1 : NULL; + *e = 0; + return s; +} + +// POSIX mkdir(path, mode); mingw's takes only the path. +static inline int plat_mkdir(const char *p, int mode) { (void)mode; return _mkdir(p); } +#define mkdir(p, m) plat_mkdir(p, m) + +// realpath(path, out[PATH_MAX]): absolute form of an existing path (links not resolved; callers check them). +#ifndef PATH_MAX +#define PATH_MAX 260 +#endif +static inline char *realpath(const char *p, char *out) { + struct stat st; + return stat(p, &st) == 0 ? _fullpath(out, p, PATH_MAX) : NULL; +} diff --git a/src/runtime/platform/windows.c b/src/runtime/platform/windows.c new file mode 100644 index 0000000..d2c18b5 --- /dev/null +++ b/src/runtime/platform/windows.c @@ -0,0 +1,424 @@ +// Windows host layer (runtime/platform.h). +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../platform.h" + +// ---- guest address space +// The window is reserved up front and committed per region (plat_commit from the loader / kernel). Anything +// the guest touches outside a committed region is committed on its first access by the exception handler +// below, so the window behaves like macOS's all-RW reservation. The OS itself (ReadFile into a guest buffer) +// does not fault: such buffers must be inside committed regions. +static uint64_t g_lo, g_hi; +#define COMMIT_CHUNK 0x10000ull + +static bool lazy_commit(uint64_t a) { + if (a < g_lo || a >= g_hi) return false; + MEMORY_BASIC_INFORMATION mi; + if (!VirtualQuery((void *)(uintptr_t)a, &mi, sizeof mi)) return false; + // Another thread faulted on the same chunk and committed it first: retry the access. + if (mi.State == MEM_COMMIT) return mi.Protect == PAGE_READWRITE; + if (mi.State != MEM_RESERVE) return false; + uint64_t base = a & ~(COMMIT_CHUNK - 1); + return VirtualAlloc((void *)(uintptr_t)base, COMMIT_CHUNK, MEM_COMMIT, PAGE_READWRITE) != NULL; +} + +static bool g_report_crashes; + +static LONG CALLBACK vectored_handler(EXCEPTION_POINTERS *ep) { + EXCEPTION_RECORD *er = ep->ExceptionRecord; + if (er->ExceptionCode != EXCEPTION_ACCESS_VIOLATION || er->NumberParameters < 2) return EXCEPTION_CONTINUE_SEARCH; + uint64_t fa = er->ExceptionInformation[1]; + if (lazy_commit(fa)) return EXCEPTION_CONTINUE_EXECUTION; + if (g_report_crashes && rt_fault_watch(fa, ep->ContextRecord->Rip)) return EXCEPTION_CONTINUE_EXECUTION; + return EXCEPTION_CONTINUE_SEARCH; +} + +// The window keeps write tracking (MEM_WRITE_WATCH: the Vulkan renderer's vertex / index cache re-copies only pages the +// CPU wrote, runtime/nvn/vulkan/exec.c). Without it (refused): no tracking, the renderer reads guest memory in place. +static bool g_write_watch; + +bool plat_reserve_guest(uint64_t lo, uint64_t hi) { + void *p = VirtualAlloc((void *)(uintptr_t)lo, hi - lo, MEM_RESERVE | MEM_WRITE_WATCH, PAGE_READWRITE); + if (p == (void *)(uintptr_t)lo) g_write_watch = true; + else p = VirtualAlloc((void *)(uintptr_t)lo, hi - lo, MEM_RESERVE, PAGE_READWRITE); + if (p != (void *)(uintptr_t)lo) return false; + g_lo = lo; + g_hi = hi; + AddVectoredExceptionHandler(1, vectored_handler); + return true; +} + +int plat_written_pages(uint64_t addr, uint64_t size, uint64_t *pages, int max, uint64_t *page) { + if (!g_write_watch || addr < g_lo || addr + size > g_hi || max <= 0) return -1; + ULONG_PTR n = (ULONG_PTR)max; + DWORD gran = 4096; + if (GetWriteWatch(WRITE_WATCH_FLAG_RESET, (void *)(uintptr_t)addr, (SIZE_T)size, (PVOID *)pages, &n, &gran) != 0) return -1; + *page = gran; + if (n >= (ULONG_PTR)max) ResetWriteWatch((void *)(uintptr_t)addr, (SIZE_T)size); // (more than fit: caller takes all) + return (int)n; +} + +// Decommit everything: the range stays reserved, and pages come back zeroed when committed again (lazily). +bool plat_reset_guest(uint64_t lo, uint64_t hi) { + return VirtualFree((void *)(uintptr_t)lo, hi - lo, MEM_DECOMMIT) != 0; +} + +void plat_commit(uint64_t addr, uint64_t size) { + if (!size) return; + uint64_t a = addr & ~0xFFFull, e = (addr + size + 0xFFF) & ~0xFFFull; + if (!VirtualAlloc((void *)(uintptr_t)a, e - a, MEM_COMMIT, PAGE_READWRITE)) + fprintf(stderr, "[plat] commit %llx+%llx failed (%lu)\n", (unsigned long long)addr, + (unsigned long long)size, GetLastError()); +} + +int plat_protect(uint64_t addr, uint64_t size, int prot) { + DWORD p = prot == PLAT_PROT_NONE ? PAGE_NOACCESS : prot == PLAT_PROT_READ ? PAGE_READONLY : PAGE_READWRITE, old; + if (addr >= g_lo && addr < g_hi) plat_commit(addr, size); + return VirtualProtect((void *)(uintptr_t)addr, size, p, &old) ? 0 : -1; +} + +// ---- time +static LARGE_INTEGER qpf; + +uint64_t plat_now_ns(void) { + if (!qpf.QuadPart) QueryPerformanceFrequency(&qpf); + LARGE_INTEGER t; + QueryPerformanceCounter(&t); + return (uint64_t)((unsigned __int128)t.QuadPart * 1000000000u / (uint64_t)qpf.QuadPart); +} + +#ifndef CREATE_WAITABLE_TIMER_HIGH_RESOLUTION +#define CREATE_WAITABLE_TIMER_HIGH_RESOLUTION 0x2 +#endif + +void plat_sleep_until_ns(uint64_t t) { + static __thread HANDLE timer; + if (!timer) { + timer = CreateWaitableTimerExW(NULL, NULL, CREATE_WAITABLE_TIMER_HIGH_RESOLUTION, TIMER_ALL_ACCESS); + if (!timer) timer = CreateWaitableTimerExW(NULL, NULL, 0, TIMER_ALL_ACCESS); + } + for (;;) { + uint64_t now = plat_now_ns(); + if (now >= t) return; + uint64_t left = t - now; + if (left < 200000) { YieldProcessor(); continue; } // last 0.2 ms: spin (timer slack) + LARGE_INTEGER due = {.QuadPart = -(LONGLONG)((left - 100000) / 100)}; // relative, 100 ns units + if (!timer || !SetWaitableTimer(timer, &due, 0, NULL, NULL, FALSE)) { Sleep(1); continue; } + WaitForSingleObject(timer, INFINITE); + } +} + +// A thread every frame waits for (the replay of NVN command lists into Vulkan): above the game's ~50 worker threads, +// which otherwise preempt it on a busy CPU (MacNativeGames measured 6.6 -> 13 ms per frame on macOS; on the Great Sky +// Island the replay ran ~40 ms of a 42 ms frame with ~29 ms of CPU time, 2026-10-24). Not time-critical: Windows and the +// game's audio threads stay above it. +void plat_thread_boost(void) { SetThreadPriority(GetCurrentThread(), THREAD_PRIORITY_HIGHEST); } + +void plat_thread_realtime(uint64_t period_ns) { + (void)period_ns; + SetThreadPriority(GetCurrentThread(), THREAD_PRIORITY_TIME_CRITICAL); +} + +// ---- process +// Release builds are GUI-subsystem executables (no console window): stdout / stderr go to lasrecomp.log next to the +// exe, and a fatal error shows a message box (plat_fatal_notice) instead of vanishing. Processes started by this one +// (the start screen re-executing itself for the game) append to the same log. +static char g_log_path[MAX_PATH]; + +static void log_to_file_without_console(void) { + if (GetStdHandle(STD_ERROR_HANDLE) && GetStdHandle(STD_ERROR_HANDLE) != INVALID_HANDLE_VALUE) return; // console / pipe + char exe[MAX_PATH]; + DWORD n = GetModuleFileNameA(NULL, exe, sizeof exe); + if (!n || n >= sizeof exe) return; + char *slash = strrchr(exe, '\\'); + if (slash) slash[1] = 0; + snprintf(g_log_path, sizeof g_log_path, "%slasrecomp.log", exe); + bool append = getenv("LASRECOMP_LOG_APPEND") != NULL; + if (!freopen(g_log_path, append ? "ab" : "wb", stderr)) { g_log_path[0] = 0; return; } + setvbuf(stderr, NULL, _IONBF, 0); + _dup2(_fileno(stderr), _fileno(stdout)); + _putenv("LASRECOMP_LOG_APPEND=1"); +} + +// The recompiled game code is built for x86-64-v3 (Makefile GENARCH: AVX2, FMA, BMI2): say so instead of crashing +// with an illegal instruction on an older CPU. +static void require_cpu(void) { + __builtin_cpu_init(); + if (__builtin_cpu_supports("avx2") && __builtin_cpu_supports("fma") && __builtin_cpu_supports("bmi2")) return; + MessageBoxA(NULL, "LAS Recomp needs a CPU with AVX2 (Intel Haswell / 2013 or newer, AMD Excavator / Zen or newer).", + "UltraTears", MB_ICONERROR | MB_OK); + ExitProcess(1); +} + +// Fatal error (rt_fatal_trace): the last FATAL / exception line of the log in a message box, when there is no console. +void plat_fatal_notice(void) { + if (!g_log_path[0]) return; + fflush(stderr); + char text[1024] = "LAS Recomp stopped because of an error."; + FILE *f = fopen(g_log_path, "rb"); + if (f) { + static char tail[16384]; + fseek(f, 0, SEEK_END); + long sz = ftell(f), from = sz > (long)sizeof tail - 1 ? sz - (long)sizeof tail + 1 : 0; + fseek(f, from, SEEK_SET); + size_t got = fread(tail, 1, sizeof tail - 1, f); + tail[got] = 0; + fclose(f); + char *line = NULL; + for (char *p = tail; (p = strstr(p, "FATAL: ")) != NULL; p++) line = p; + if (!line) for (char *p = tail; (p = strstr(p, "*** ")) != NULL; p++) line = p; + if (line) { + char *end = strchr(line, '\n'); + if (end) *end = 0; + snprintf(text, sizeof text, "LAS Recomp stopped because of an error:\n\n%.600s", line); + if (strstr(line, "Vulkan") || strstr(line, "VK_") || strstr(line, "vk")) { + size_t l = strlen(text); + snprintf(text + l, sizeof text - l, "\n\nLAS Recomp needs a graphics driver with Vulkan 1.3: install the " + "latest driver for your GPU (NVIDIA, AMD or Intel)."); + } + } + } + size_t l = strlen(text); + snprintf(text + l, sizeof text - l, "\n\nDetails: %s", g_log_path); + MessageBoxA(NULL, text, "UltraTears", MB_ICONERROR | MB_OK); +} + +void plat_init(void) { + log_to_file_without_console(); + require_cpu(); + _fmode = _O_BINARY; // fopen/open default to binary (the runtime's "r"/"w" files are binary or LF text) + _setmode(_fileno(stdout), _O_BINARY); + _setmode(_fileno(stderr), _O_BINARY); + SetConsoleOutputCP(CP_UTF8); + timeBeginPeriod(1); // 1 ms Sleep / condition-variable timeouts (default 15.6 ms) + // Opt out of EcoQoS and of Windows 11 ignoring the timer resolution while the window is occluded. + PROCESS_POWER_THROTTLING_STATE pt = { + .Version = PROCESS_POWER_THROTTLING_CURRENT_VERSION, + .ControlMask = PROCESS_POWER_THROTTLING_EXECUTION_SPEED | PROCESS_POWER_THROTTLING_IGNORE_TIMER_RESOLUTION, + .StateMask = 0, + }; + SetProcessInformation(GetCurrentProcess(), ProcessPowerThrottling, &pt, sizeof pt); +} + +bool plat_exe_path(char *buf, size_t size) { + DWORD n = GetModuleFileNameA(NULL, buf, (DWORD)size); + return n > 0 && n < size; +} + +// Quote one argument the way CommandLineToArgvW / the CRT parse it back. +static void append_arg(char *out, size_t cap, const char *a) { + size_t n = strlen(out); + bool q = !*a || strpbrk(a, " \t\"") != NULL; + if (n && n + 1 < cap) out[n++] = ' '; + if (q && n + 1 < cap) out[n++] = '"'; + for (const char *p = a; *p; p++) { + int bs = 0; + while (*p == '\\') { bs++; p++; } + int reps = !*p ? (q ? 2 * bs : bs) : *p == '"' ? 2 * bs + 1 : bs; + for (int i = 0; i < reps && n + 1 < cap; i++) out[n++] = '\\'; + if (!*p) break; + if (n + 1 < cap) out[n++] = *p; + } + if (q && n + 1 < cap) out[n++] = '"'; + out[n] = 0; +} + +// No exec() on Windows: start the new process with the same console / standard handles, then exit. +void plat_exec(char *const argv[]) { + STARTUPINFOW si = {.cb = sizeof si}; + PROCESS_INFORMATION pi; + BOOL ok; + fflush(stdout); + fflush(stderr); + if (!argv) { + ok = CreateProcessW(NULL, GetCommandLineW(), NULL, NULL, TRUE, 0, NULL, NULL, &si, &pi); + } else { + static char cmd[32768]; + cmd[0] = 0; + for (int i = 0; argv[i]; i++) append_arg(cmd, sizeof cmd, argv[i]); + static wchar_t wcmd[32768]; + MultiByteToWideChar(CP_UTF8, 0, cmd, -1, wcmd, 32768); + ok = CreateProcessW(NULL, wcmd, NULL, NULL, TRUE, 0, NULL, NULL, &si, &pi); + } + if (!ok) return; + CloseHandle(pi.hThread); + CloseHandle(pi.hProcess); + ExitProcess(0); +} + +// ---- diagnostics +// DLL frames are named module+offset; frames in our own executable are left to the caller, which maps +// them to guest functions (runtime/dispatch.c) - mingw builds carry DWARF, not PDB symbols. +const char *plat_symbol(uint64_t pc, uint64_t *offset) { + static __thread char name[MAX_PATH]; + HMODULE m; + if (!GetModuleHandleExA(GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT, + (LPCSTR)(uintptr_t)pc, &m)) + return NULL; + if (m == GetModuleHandleA(NULL)) return NULL; + char path[MAX_PATH]; + if (!GetModuleFileNameA(m, path, sizeof path)) return NULL; + const char *b = strrchr(path, '\\'); + snprintf(name, sizeof name, "%s", b ? b + 1 : path); + if (offset) *offset = pc - (uint64_t)(uintptr_t)m; + return name; +} + +const char *rt_host_fn_name(uint64_t pc, uint64_t *offset); // runtime/dispatch.c + +void plat_print_backtrace(void) { + void *bt[64]; + USHORT n = RtlCaptureStackBackTrace(0, 64, bt, NULL); + uint64_t exe = (uint64_t)(uintptr_t)GetModuleHandleA(NULL); + for (int i = 0; i < n; i++) { + uint64_t pc = (uint64_t)(uintptr_t)bt[i], off = 0; + const char *s = plat_symbol(pc, &off); + if (!s) s = rt_host_fn_name(pc, &off); + if (s) fprintf(stderr, " #%-2d %016llx %s+%llx\n", i, (unsigned long long)pc, s, (unsigned long long)off); + else fprintf(stderr, " #%-2d %016llx lasrecomp+%llx\n", i, (unsigned long long)pc, + (unsigned long long)(pc - exe)); + } +} + +static LONG WINAPI unhandled_filter(EXCEPTION_POINTERS *ep) { + EXCEPTION_RECORD *er = ep->ExceptionRecord; + uint64_t fa = er->NumberParameters >= 2 ? er->ExceptionInformation[1] : 0; + char what[48]; + snprintf(what, sizeof what, "exception %08lx", er->ExceptionCode); + MEMORY_BASIC_INFORMATION fm; + if (er->NumberParameters >= 2 && VirtualQuery((void *)(uintptr_t)fa, &fm, sizeof fm)) + fprintf(stderr, "[plat] fault %s %#llx: region %p+%#llx state %#lx protect %#lx (allocation %p)\n", + er->ExceptionInformation[0] == 1 ? "write" : er->ExceptionInformation[0] == 8 ? "execute" : "read", + (unsigned long long)fa, fm.BaseAddress, (unsigned long long)fm.RegionSize, fm.State, fm.Protect, + fm.AllocationBase); + // x64 has no link register: report the return address on top of the stack (valid at function entry). + uint64_t lr = 0; + MEMORY_BASIC_INFORMATION mi; + if (VirtualQuery((void *)(uintptr_t)ep->ContextRecord->Rsp, &mi, sizeof mi) && mi.State == MEM_COMMIT) + lr = *(uint64_t *)(uintptr_t)ep->ContextRecord->Rsp; + rt_fault_report(what, fa, ep->ContextRecord->Rip, lr); +} + +void plat_crash_handler_install(void) { + ULONG guarantee = 64 << 10; // room for the report after a stack overflow + SetThreadStackGuarantee(&guarantee); + if (g_report_crashes) return; + g_report_crashes = true; + SetUnhandledExceptionFilter(unhandled_filter); +} + +// ---- POSIX shims (win_compat.h) +// The kernel does not take the lazy-commit fault: a guest buffer handed to ReadFile / WriteFile fails with +// ERROR_NOACCESS on reserved pages. Commit those first (only reserved ones: committed pages keep their protection). +static void commit_reserved(uint64_t a, uint64_t n) { + if (!n || a >= g_hi || a + n <= g_lo) return; + uint64_t end = a + n; + while (a < end) { + MEMORY_BASIC_INFORMATION mi; + if (!VirtualQuery((void *)(uintptr_t)a, &mi, sizeof mi)) return; + uint64_t next = (uint64_t)(uintptr_t)mi.BaseAddress + mi.RegionSize; + if (mi.State == MEM_RESERVE) plat_commit(a, (next < end ? next : end) - a); + a = next; + } +} + +static ssize_t prw(int fd, void *buf, size_t n, off_t off, bool write) { + HANDLE h = (HANDLE)_get_osfhandle(fd); + if (h == INVALID_HANDLE_VALUE) return -1; + commit_reserved((uint64_t)(uintptr_t)buf, n); + size_t done = 0; + while (done < n) { + OVERLAPPED ov = {0}; + uint64_t o = (uint64_t)off + done; + ov.Offset = (DWORD)o; + ov.OffsetHigh = (DWORD)(o >> 32); + DWORD chunk = n - done > 0x40000000 ? 0x40000000 : (DWORD)(n - done), got = 0; + BOOL ok = write ? WriteFile(h, (const char *)buf + done, chunk, &got, &ov) + : ReadFile(h, (char *)buf + done, chunk, &got, &ov); + if (!ok) { + if (GetLastError() == ERROR_HANDLE_EOF) break; + return done ? (ssize_t)done : -1; + } + if (!got) break; + done += got; + } + return (ssize_t)done; +} +ssize_t pread(int fd, void *buf, size_t n, off_t off) { return prw(fd, buf, n, off, false); } +ssize_t pwrite(int fd, const void *buf, size_t n, off_t off) { return prw(fd, (void *)buf, n, off, true); } + +// ---- child processes +PlatProc plat_spawn(const char *const argv[], const char *log_path, bool append_log) { + static __thread char cmd[32768]; + cmd[0] = 0; + for (int i = 0; argv[i]; i++) append_arg(cmd, sizeof cmd, argv[i]); + static __thread wchar_t wcmd[32768], wlog[MAX_PATH]; + MultiByteToWideChar(CP_UTF8, 0, cmd, -1, wcmd, 32768); + MultiByteToWideChar(CP_UTF8, 0, log_path ? log_path : "NUL", -1, wlog, MAX_PATH); + SECURITY_ATTRIBUTES sa = {.nLength = sizeof sa, .bInheritHandle = TRUE}; + HANDLE log = CreateFileW(wlog, append_log ? FILE_APPEND_DATA : GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE, + &sa, append_log ? OPEN_ALWAYS : CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL); + if (log == INVALID_HANDLE_VALUE) return 0; + STARTUPINFOW si = {.cb = sizeof si, .dwFlags = STARTF_USESTDHANDLES, .hStdInput = NULL, .hStdOutput = log, + .hStdError = log}; + PROCESS_INFORMATION pi; + // CreateProcess doesn't resolve relative paths written with '/' (e.g. a helper under tools/): pass the program + // as a full path, like posix_spawn takes it as given + static __thread wchar_t wprog[MAX_PATH], wfull[MAX_PATH]; + MultiByteToWideChar(CP_UTF8, 0, argv[0], -1, wprog, MAX_PATH); + for (wchar_t *p = wprog; *p; p++) if (*p == L'/') *p = L'\\'; + const wchar_t *app = GetFullPathNameW(wprog, MAX_PATH, wfull, NULL) ? wfull : NULL; + BOOL ok = CreateProcessW(app, wcmd, NULL, NULL, TRUE, CREATE_NO_WINDOW | CREATE_SUSPENDED, NULL, NULL, &si, &pi); + if (!ok) fprintf(stderr, "[plat] could not start %s (error %lu)\n", argv[0], GetLastError()); + CloseHandle(log); + if (!ok) return 0; + // every child in one job that ends with this process (a compiler's own children too: no leftovers on quit) + static HANDLE job; + if (!job && (job = CreateJobObjectW(NULL, NULL))) { + JOBOBJECT_EXTENDED_LIMIT_INFORMATION li = {0}; + li.BasicLimitInformation.LimitFlags = JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE; + SetInformationJobObject(job, JobObjectExtendedLimitInformation, &li, sizeof li); + } + if (job) AssignProcessToJobObject(job, pi.hProcess); + ResumeThread(pi.hThread); + CloseHandle(pi.hThread); + return (PlatProc)pi.hProcess; +} + +void plat_proc_kill(PlatProc p) { // (what it started ends with this process: the job above) + TerminateProcess((HANDLE)p, 255); + WaitForSingleObject((HANDLE)p, INFINITE); + CloseHandle((HANDLE)p); +} + +int plat_proc_wait(PlatProc p, bool block) { + HANDLE h = (HANDLE)p; + DWORD w = WaitForSingleObject(h, block ? INFINITE : 0); + if (w == WAIT_TIMEOUT) return -1; + if (w != WAIT_OBJECT_0) return 255; // (not a process we can wait for: ended as far as the caller is concerned) + DWORD code = 255; + GetExitCodeProcess(h, &code); + CloseHandle(h); + return code > 255 ? 255 : (int)code; // (a crash's NTSTATUS, e.g. 0xC0000409, is not "still running" (-1)) +} + +bool plat_remove_tree(const char *dir) { + // SHFileOperation wants a double-NUL-terminated path with backslashes + static __thread wchar_t w[MAX_PATH + 2]; + int n = MultiByteToWideChar(CP_UTF8, 0, dir, -1, w, MAX_PATH); + if (n <= 1) return false; + w[n] = 0; + for (wchar_t *p = w; *p; p++) if (*p == L'/') *p = L'\\'; + SHFILEOPSTRUCTW op = {.wFunc = FO_DELETE, .pFrom = w, .fFlags = FOF_NO_UI}; + return SHFileOperationW(&op) == 0 && !op.fAnyOperationsAborted; +} + +// End the process at once, skipping DLL teardown (from an atexit handler, after flushing). Exit status 0. +void plat_terminate(void) { TerminateProcess(GetCurrentProcess(), 0); } diff --git a/src/runtime/rt.h b/src/runtime/rt.h new file mode 100644 index 0000000..c27aaa6 --- /dev/null +++ b/src/runtime/rt.h @@ -0,0 +1,55 @@ +// Runtime-internal declarations: guest address layout, loader, dispatch. +// Structure follows LASRecomp runtime/rt.h (MIT); the layout and module set are TOTK's. +#pragma once +#include +#include + +#include "cpu.h" +#include "platform.h" + +#define LOG(...) fprintf(stderr, __VA_ARGS__) +#define FATAL(...) (fprintf(stderr, "FATAL: " __VA_ARGS__), rt_fatal_trace()) + +// ---- guest address layout (guest addresses == host addresses) +// The SDK encodes IPC buffer addresses in 39 bits, so all guest memory lies below 2^39. The window is reserved at +// startup; pages are committed when first used. The code bases are baked into the generated C (tools/gen_game.py). +#define GUEST_LO 0x0800000000ull +#define GUEST_HI 0x8000000000ull // 2^39 +#define MAIN_BASE 0x0800000000ull +#define SUBSDK0_BASE 0x0804000000ull +#define SDK_BASE 0x0805000000ull +#define HLE_BASE 0x0808000000ull // fake guest addresses of host-implemented functions (16 bytes each) +#define HLE_SIZE 0x0000100000ull +#define TLS_BASE 0x0809000000ull // per-thread TLS blocks (tpidrro_el0) +#define TLS_SIZE 0x0000100000ull +#define HLE_HEAP 0x080A000000ull // guest-visible memory owned by host code +#define HLE_HEAP_SIZE 0x000E000000ull + +// ---- loader.c +typedef struct Module { + const char *name; + uint64_t base, size; + uint64_t text_off, text_size, ro_off, ro_size, data_off, data_size; + uint64_t dynamic, init, fini, init_array, init_array_sz, fini_array, fini_array_sz; + uint64_t symtab, strtab, nsyms; +} Module; + +#define MAX_MODULES 3 +extern Module g_modules[MAX_MODULES]; +extern int g_nmodules; +void loader_load(const char *exefs_dir); +uint64_t loader_find_symbol(const char *name); +const char *loader_symbol_at(uint64_t addr, uint64_t *offset); + +// ---- dispatch.c +typedef void (*HostFn)(Cpu *c); +void dispatch_init(void); +GuestFn dispatch_lookup(uint64_t addr); +uint64_t hle_register(const char *name, HostFn fn); // returns the fake guest address +HostFn hle_lookup_impl(const char *name); +uint64_t hle_register_entry(const char *name, const void *entry); // a registry entry (hle.h HleImplEntry) +const void *hle_lookup_entry_public(const char *name); // the registry entry for a symbol, or NULL +void hle_override(uint64_t guest_addr, HostFn fn, const char *name); +void *hle_alloc(uint64_t size); +void guest_call(Cpu *c, uint64_t addr); +void rt_fatal_trace(void) __attribute__((noreturn)); diff --git a/src/runtime/simd.h b/src/runtime/simd.h new file mode 100644 index 0000000..ed2a7bc --- /dev/null +++ b/src/runtime/simd.h @@ -0,0 +1,833 @@ +// Portable C lowering of AArch64 FP / AdvSIMD / crypto / CRC32 (tools/simd_c.py emits calls with constant +// fields; everything is always_inline so the size/lane/op switches fold at -O1). +// Semantics follow the ARM pseudocode, not the host: NaN propagation (first signalling NaN quietened, then +// first quiet NaN), FPCR.DN default NaN, FPCR.FZ flushing (inputs and pre-rounding-tiny results), FPCR.RMode +// for arithmetic / conversions / FRINTX/I, saturating FP->int conversions, fused multiply-add, ARM estimate +// tables. Not modelled: FPSR cumulative exception bits and QC, FPCR trap enables, FP16 arithmetic (ARMv8.2). +// FPCR.AHP is honoured by the half-precision conversions. Host assumptions: IEEE binary32/64, MXCSR (or the +// host FPU) in round-to-nearest with denormals enabled; fma must be correctly rounded (build x86 with -mfma). +#pragma once +#include +#include +#include + +#include "cpu.h" + +typedef __uint128_t u128; +#define SI static inline __attribute__((always_inline)) +#define SN static __attribute__((noinline)) +#define SIMD_CAT2(a, b) a##_##b +#define SIMD_CAT(a, b) SIMD_CAT2(a, b) + +#define SIMD_VT(name, T) typedef T name __attribute__((vector_size(16), may_alias)); +SIMD_VT(vu8, uint8_t) SIMD_VT(vs8, int8_t) SIMD_VT(vu16, uint16_t) SIMD_VT(vs16, int16_t) +SIMD_VT(vu32, uint32_t) SIMD_VT(vs32, int32_t) SIMD_VT(vu64, uint64_t) SIMD_VT(vs64, int64_t) +SIMD_VT(vf32, float) SIMD_VT(vf64, double) +typedef uint16_t __attribute__((may_alias)) sa16; +typedef uint32_t __attribute__((may_alias)) sa32; +typedef uint64_t __attribute__((may_alias)) sa64; +typedef float __attribute__((may_alias)) saf32; +typedef double __attribute__((may_alias)) saf64; +typedef u128 __attribute__((aligned(1), may_alias)) u128u; + +// ---- registers: VREG = whole V register as a vector type; VQ = source with the upper half cleared when +// Q=0; VW = write, zeroing the upper half when Q=0. Scalar views of lane 0: VS8..VS64, VSF32/VSF64. +#define VREG(T, n) (*(T *)&c->v[n]) +#define VQ(T, Q, n) ((T)((Q) ? c->v[n] : (u128)(uint64_t)c->v[n])) +#define VW(T, Q, d, e) do { T _w = (e); c->v[d] = (Q) ? (u128)_w : (u128)(uint64_t)(u128)_w; } while (0) +#define VS8(n) (*(uint8_t *)&c->v[n]) +#define VS16(n) (*(sa16 *)&c->v[n]) +#define VS32(n) (*(sa32 *)&c->v[n]) +#define VS64(n) (*(sa64 *)&c->v[n]) +#define VSF32(n) (*(saf32 *)&c->v[n]) +#define VSF64(n) (*(saf64 *)&c->v[n]) +#define VHI64(n) (((sa64 *)&c->v[n])[1]) +#define LD128(a) (*(const u128u *)(uintptr_t)(a)) +#define ST128(a, v) (*(u128u *)(uintptr_t)(a) = (u128)(v)) +#define SHUF __builtin_shufflevector +#define CVT __builtin_convertvector + +#define FPCR_AHP (1u << 26) +#define FPCR_DN (1u << 25) +#define FPCR_FZ (1u << 24) +#define FPCR_RM (3u << 22) +#define FPCR_SLOW (FPCR_FZ | FPCR_RM) + +// MSR FPCR (the lifter emits this): FPCR.FZ also sets the host thread's MXCSR flush-to-zero and denormals-are-zero +// (FTZ bit 15, DAZ bit 6), so FP arithmetic under FZ runs natively instead of through the flushing slow path. TOTK sets +// FZ on its threads; the slow path took ~10-15 % of the game's CPU time (2026-10-21 profile), and SavageWind measured +// ~40 % of BotW's main thread for the same software flushing (2026-10-20). The one difference from ARM: a result that +// rounds up to the smallest normal from below stays (x86 detects tininess after rounding, ARM before). Each guest +// thread runs on its own host thread, so MXCSR follows that thread's FPCR. FMIN/FMAX and NaN fixes, which return +// operand bits, flush their operands themselves. +SI void rt_fpcr_write(Cpu *c, uint64_t v) { + c->fpcr = v & 0x7c00000ull; + unsigned m = _mm_getcsr() & ~0x8040u; + if (v & FPCR_FZ) m |= 0x8040u; + _mm_setcsr(m); +} + +enum { FOP_ADD, FOP_SUB, FOP_MUL, FOP_MULX, FOP_NMUL, FOP_DIV, FOP_ABD, FOP_SQRT, FOP_MADD, FOP_RECPS, FOP_RSQRTS, + FOP_MAX, FOP_MIN, FOP_MAXNM, FOP_MINNM }; + +static const uint8_t simd_recip_tab[256] = { + 255, 253, 251, 249, 247, 245, 243, 241, 240, 238, 236, 234, 232, 230, 229, 227, + 225, 223, 221, 220, 218, 216, 215, 213, 211, 210, 208, 206, 205, 203, 201, 200, + 198, 197, 195, 194, 192, 191, 189, 188, 186, 185, 183, 182, 180, 179, 177, 176, + 174, 173, 172, 170, 169, 167, 166, 165, 163, 162, 161, 159, 158, 157, 156, 154, + 153, 152, 150, 149, 148, 147, 145, 144, 143, 142, 141, 139, 138, 137, 136, 135, + 134, 132, 131, 130, 129, 128, 127, 126, 124, 123, 122, 121, 120, 119, 118, 117, + 116, 115, 114, 113, 112, 111, 110, 109, 108, 107, 106, 105, 104, 103, 102, 101, + 100, 99, 98, 97, 96, 95, 94, 93, 92, 91, 90, 89, 88, 88, 87, 86, + 85, 84, 83, 82, 81, 81, 80, 79, 78, 77, 76, 75, 75, 74, 73, 72, + 71, 70, 70, 69, 68, 67, 66, 66, 65, 64, 63, 63, 62, 61, 60, 59, + 59, 58, 57, 56, 56, 55, 54, 53, 53, 52, 51, 51, 50, 49, 48, 48, + 47, 46, 46, 45, 44, 44, 43, 42, 42, 41, 40, 40, 39, 38, 38, 37, + 36, 36, 35, 34, 34, 33, 32, 32, 31, 30, 30, 29, 29, 28, 27, 27, + 26, 26, 25, 24, 24, 23, 23, 22, 21, 21, 20, 20, 19, 18, 18, 17, + 17, 16, 16, 15, 15, 14, 13, 13, 12, 12, 11, 11, 10, 10, 9, 9, + 8, 7, 7, 6, 6, 5, 5, 4, 4, 3, 3, 2, 2, 1, 1, 0, +}; +static const uint8_t simd_rsqrt_tab[384] = { + 255, 253, 251, 249, 247, 245, 243, 242, 240, 238, 236, 234, 233, 231, 229, 228, + 226, 224, 223, 221, 219, 218, 216, 215, 213, 212, 210, 209, 207, 206, 204, 203, + 201, 200, 198, 197, 196, 194, 193, 192, 190, 189, 188, 186, 185, 184, 183, 181, + 180, 179, 178, 176, 175, 174, 173, 172, 170, 169, 168, 167, 166, 165, 164, 163, + 162, 160, 159, 158, 157, 156, 155, 154, 153, 152, 151, 150, 149, 148, 147, 146, + 145, 144, 143, 142, 141, 140, 140, 139, 138, 137, 136, 135, 134, 133, 132, 131, + 131, 130, 129, 128, 127, 126, 126, 125, 124, 123, 122, 121, 121, 120, 119, 118, + 118, 117, 116, 115, 114, 114, 113, 112, 111, 111, 110, 109, 109, 108, 107, 106, + 105, 105, 104, 104, 103, 103, 101, 101, 100, 100, 99, 99, 97, 97, 96, 96, + 95, 95, 93, 93, 92, 92, 91, 91, 90, 90, 88, 88, 87, 87, 86, 86, + 85, 85, 84, 84, 82, 82, 81, 81, 80, 80, 79, 79, 78, 78, 77, 77, + 76, 76, 75, 75, 74, 74, 72, 72, 71, 71, 70, 70, 69, 69, 68, 68, + 67, 67, 66, 66, 65, 65, 64, 64, 63, 63, 62, 62, 61, 61, 60, 60, + 60, 60, 59, 59, 58, 58, 57, 57, 56, 56, 55, 55, 54, 54, 53, 53, + 52, 52, 51, 51, 51, 51, 50, 50, 49, 49, 48, 48, 47, 47, 46, 46, + 46, 46, 45, 45, 44, 44, 43, 43, 42, 42, 42, 42, 41, 41, 40, 40, + 39, 39, 38, 38, 38, 38, 37, 37, 36, 36, 35, 35, 35, 35, 34, 34, + 33, 33, 33, 33, 32, 32, 31, 31, 30, 30, 30, 30, 29, 29, 28, 28, + 28, 28, 27, 27, 26, 26, 26, 26, 25, 25, 24, 24, 24, 24, 23, 23, + 22, 22, 22, 22, 21, 21, 20, 20, 20, 20, 19, 19, 19, 19, 18, 18, + 17, 17, 17, 17, 16, 16, 16, 16, 15, 15, 14, 14, 14, 14, 13, 13, + 13, 13, 12, 12, 11, 11, 11, 11, 10, 10, 10, 10, 9, 9, 9, 9, + 8, 8, 8, 8, 7, 7, 6, 6, 6, 6, 5, 5, 5, 5, 4, 4, + 4, 4, 3, 3, 3, 3, 2, 2, 2, 2, 1, 1, 1, 1, 0, 0, +}; +static const uint32_t simd_crc32_tab[256] = { + 0x00000000, 0x77073096, 0xee0e612c, 0x990951ba, 0x076dc419, 0x706af48f, 0xe963a535, 0x9e6495a3, + 0x0edb8832, 0x79dcb8a4, 0xe0d5e91e, 0x97d2d988, 0x09b64c2b, 0x7eb17cbd, 0xe7b82d07, 0x90bf1d91, + 0x1db71064, 0x6ab020f2, 0xf3b97148, 0x84be41de, 0x1adad47d, 0x6ddde4eb, 0xf4d4b551, 0x83d385c7, + 0x136c9856, 0x646ba8c0, 0xfd62f97a, 0x8a65c9ec, 0x14015c4f, 0x63066cd9, 0xfa0f3d63, 0x8d080df5, + 0x3b6e20c8, 0x4c69105e, 0xd56041e4, 0xa2677172, 0x3c03e4d1, 0x4b04d447, 0xd20d85fd, 0xa50ab56b, + 0x35b5a8fa, 0x42b2986c, 0xdbbbc9d6, 0xacbcf940, 0x32d86ce3, 0x45df5c75, 0xdcd60dcf, 0xabd13d59, + 0x26d930ac, 0x51de003a, 0xc8d75180, 0xbfd06116, 0x21b4f4b5, 0x56b3c423, 0xcfba9599, 0xb8bda50f, + 0x2802b89e, 0x5f058808, 0xc60cd9b2, 0xb10be924, 0x2f6f7c87, 0x58684c11, 0xc1611dab, 0xb6662d3d, + 0x76dc4190, 0x01db7106, 0x98d220bc, 0xefd5102a, 0x71b18589, 0x06b6b51f, 0x9fbfe4a5, 0xe8b8d433, + 0x7807c9a2, 0x0f00f934, 0x9609a88e, 0xe10e9818, 0x7f6a0dbb, 0x086d3d2d, 0x91646c97, 0xe6635c01, + 0x6b6b51f4, 0x1c6c6162, 0x856530d8, 0xf262004e, 0x6c0695ed, 0x1b01a57b, 0x8208f4c1, 0xf50fc457, + 0x65b0d9c6, 0x12b7e950, 0x8bbeb8ea, 0xfcb9887c, 0x62dd1ddf, 0x15da2d49, 0x8cd37cf3, 0xfbd44c65, + 0x4db26158, 0x3ab551ce, 0xa3bc0074, 0xd4bb30e2, 0x4adfa541, 0x3dd895d7, 0xa4d1c46d, 0xd3d6f4fb, + 0x4369e96a, 0x346ed9fc, 0xad678846, 0xda60b8d0, 0x44042d73, 0x33031de5, 0xaa0a4c5f, 0xdd0d7cc9, + 0x5005713c, 0x270241aa, 0xbe0b1010, 0xc90c2086, 0x5768b525, 0x206f85b3, 0xb966d409, 0xce61e49f, + 0x5edef90e, 0x29d9c998, 0xb0d09822, 0xc7d7a8b4, 0x59b33d17, 0x2eb40d81, 0xb7bd5c3b, 0xc0ba6cad, + 0xedb88320, 0x9abfb3b6, 0x03b6e20c, 0x74b1d29a, 0xead54739, 0x9dd277af, 0x04db2615, 0x73dc1683, + 0xe3630b12, 0x94643b84, 0x0d6d6a3e, 0x7a6a5aa8, 0xe40ecf0b, 0x9309ff9d, 0x0a00ae27, 0x7d079eb1, + 0xf00f9344, 0x8708a3d2, 0x1e01f268, 0x6906c2fe, 0xf762575d, 0x806567cb, 0x196c3671, 0x6e6b06e7, + 0xfed41b76, 0x89d32be0, 0x10da7a5a, 0x67dd4acc, 0xf9b9df6f, 0x8ebeeff9, 0x17b7be43, 0x60b08ed5, + 0xd6d6a3e8, 0xa1d1937e, 0x38d8c2c4, 0x4fdff252, 0xd1bb67f1, 0xa6bc5767, 0x3fb506dd, 0x48b2364b, + 0xd80d2bda, 0xaf0a1b4c, 0x36034af6, 0x41047a60, 0xdf60efc3, 0xa867df55, 0x316e8eef, 0x4669be79, + 0xcb61b38c, 0xbc66831a, 0x256fd2a0, 0x5268e236, 0xcc0c7795, 0xbb0b4703, 0x220216b9, 0x5505262f, + 0xc5ba3bbe, 0xb2bd0b28, 0x2bb45a92, 0x5cb36a04, 0xc2d7ffa7, 0xb5d0cf31, 0x2cd99e8b, 0x5bdeae1d, + 0x9b64c2b0, 0xec63f226, 0x756aa39c, 0x026d930a, 0x9c0906a9, 0xeb0e363f, 0x72076785, 0x05005713, + 0x95bf4a82, 0xe2b87a14, 0x7bb12bae, 0x0cb61b38, 0x92d28e9b, 0xe5d5be0d, 0x7cdcefb7, 0x0bdbdf21, + 0x86d3d2d4, 0xf1d4e242, 0x68ddb3f8, 0x1fda836e, 0x81be16cd, 0xf6b9265b, 0x6fb077e1, 0x18b74777, + 0x88085ae6, 0xff0f6a70, 0x66063bca, 0x11010b5c, 0x8f659eff, 0xf862ae69, 0x616bffd3, 0x166ccf45, + 0xa00ae278, 0xd70dd2ee, 0x4e048354, 0x3903b3c2, 0xa7672661, 0xd06016f7, 0x4969474d, 0x3e6e77db, + 0xaed16a4a, 0xd9d65adc, 0x40df0b66, 0x37d83bf0, 0xa9bcae53, 0xdebb9ec5, 0x47b2cf7f, 0x30b5ffe9, + 0xbdbdf21c, 0xcabac28a, 0x53b39330, 0x24b4a3a6, 0xbad03605, 0xcdd70693, 0x54de5729, 0x23d967bf, + 0xb3667a2e, 0xc4614ab8, 0x5d681b02, 0x2a6f2b94, 0xb40bbe37, 0xc30c8ea1, 0x5a05df1b, 0x2d02ef8d, +}; +static const uint32_t simd_crc32c_tab[256] = { + 0x00000000, 0xf26b8303, 0xe13b70f7, 0x1350f3f4, 0xc79a971f, 0x35f1141c, 0x26a1e7e8, 0xd4ca64eb, + 0x8ad958cf, 0x78b2dbcc, 0x6be22838, 0x9989ab3b, 0x4d43cfd0, 0xbf284cd3, 0xac78bf27, 0x5e133c24, + 0x105ec76f, 0xe235446c, 0xf165b798, 0x030e349b, 0xd7c45070, 0x25afd373, 0x36ff2087, 0xc494a384, + 0x9a879fa0, 0x68ec1ca3, 0x7bbcef57, 0x89d76c54, 0x5d1d08bf, 0xaf768bbc, 0xbc267848, 0x4e4dfb4b, + 0x20bd8ede, 0xd2d60ddd, 0xc186fe29, 0x33ed7d2a, 0xe72719c1, 0x154c9ac2, 0x061c6936, 0xf477ea35, + 0xaa64d611, 0x580f5512, 0x4b5fa6e6, 0xb93425e5, 0x6dfe410e, 0x9f95c20d, 0x8cc531f9, 0x7eaeb2fa, + 0x30e349b1, 0xc288cab2, 0xd1d83946, 0x23b3ba45, 0xf779deae, 0x05125dad, 0x1642ae59, 0xe4292d5a, + 0xba3a117e, 0x4851927d, 0x5b016189, 0xa96ae28a, 0x7da08661, 0x8fcb0562, 0x9c9bf696, 0x6ef07595, + 0x417b1dbc, 0xb3109ebf, 0xa0406d4b, 0x522bee48, 0x86e18aa3, 0x748a09a0, 0x67dafa54, 0x95b17957, + 0xcba24573, 0x39c9c670, 0x2a993584, 0xd8f2b687, 0x0c38d26c, 0xfe53516f, 0xed03a29b, 0x1f682198, + 0x5125dad3, 0xa34e59d0, 0xb01eaa24, 0x42752927, 0x96bf4dcc, 0x64d4cecf, 0x77843d3b, 0x85efbe38, + 0xdbfc821c, 0x2997011f, 0x3ac7f2eb, 0xc8ac71e8, 0x1c661503, 0xee0d9600, 0xfd5d65f4, 0x0f36e6f7, + 0x61c69362, 0x93ad1061, 0x80fde395, 0x72966096, 0xa65c047d, 0x5437877e, 0x4767748a, 0xb50cf789, + 0xeb1fcbad, 0x197448ae, 0x0a24bb5a, 0xf84f3859, 0x2c855cb2, 0xdeeedfb1, 0xcdbe2c45, 0x3fd5af46, + 0x7198540d, 0x83f3d70e, 0x90a324fa, 0x62c8a7f9, 0xb602c312, 0x44694011, 0x5739b3e5, 0xa55230e6, + 0xfb410cc2, 0x092a8fc1, 0x1a7a7c35, 0xe811ff36, 0x3cdb9bdd, 0xceb018de, 0xdde0eb2a, 0x2f8b6829, + 0x82f63b78, 0x709db87b, 0x63cd4b8f, 0x91a6c88c, 0x456cac67, 0xb7072f64, 0xa457dc90, 0x563c5f93, + 0x082f63b7, 0xfa44e0b4, 0xe9141340, 0x1b7f9043, 0xcfb5f4a8, 0x3dde77ab, 0x2e8e845f, 0xdce5075c, + 0x92a8fc17, 0x60c37f14, 0x73938ce0, 0x81f80fe3, 0x55326b08, 0xa759e80b, 0xb4091bff, 0x466298fc, + 0x1871a4d8, 0xea1a27db, 0xf94ad42f, 0x0b21572c, 0xdfeb33c7, 0x2d80b0c4, 0x3ed04330, 0xccbbc033, + 0xa24bb5a6, 0x502036a5, 0x4370c551, 0xb11b4652, 0x65d122b9, 0x97baa1ba, 0x84ea524e, 0x7681d14d, + 0x2892ed69, 0xdaf96e6a, 0xc9a99d9e, 0x3bc21e9d, 0xef087a76, 0x1d63f975, 0x0e330a81, 0xfc588982, + 0xb21572c9, 0x407ef1ca, 0x532e023e, 0xa145813d, 0x758fe5d6, 0x87e466d5, 0x94b49521, 0x66df1622, + 0x38cc2a06, 0xcaa7a905, 0xd9f75af1, 0x2b9cd9f2, 0xff56bd19, 0x0d3d3e1a, 0x1e6dcdee, 0xec064eed, + 0xc38d26c4, 0x31e6a5c7, 0x22b65633, 0xd0ddd530, 0x0417b1db, 0xf67c32d8, 0xe52cc12c, 0x1747422f, + 0x49547e0b, 0xbb3ffd08, 0xa86f0efc, 0x5a048dff, 0x8ecee914, 0x7ca56a17, 0x6ff599e3, 0x9d9e1ae0, + 0xd3d3e1ab, 0x21b862a8, 0x32e8915c, 0xc083125f, 0x144976b4, 0xe622f5b7, 0xf5720643, 0x07198540, + 0x590ab964, 0xab613a67, 0xb831c993, 0x4a5a4a90, 0x9e902e7b, 0x6cfbad78, 0x7fab5e8c, 0x8dc0dd8f, + 0xe330a81a, 0x115b2b19, 0x020bd8ed, 0xf0605bee, 0x24aa3f05, 0xd6c1bc06, 0xc5914ff2, 0x37faccf1, + 0x69e9f0d5, 0x9b8273d6, 0x88d28022, 0x7ab90321, 0xae7367ca, 0x5c18e4c9, 0x4f48173d, 0xbd23943e, + 0xf36e6f75, 0x0105ec76, 0x12551f82, 0xe03e9c81, 0x34f4f86a, 0xc69f7b69, 0xd5cf889d, 0x27a40b9e, + 0x79b737ba, 0x8bdcb4b9, 0x988c474d, 0x6ae7c44e, 0xbe2da0a5, 0x4c4623a6, 0x5f16d052, 0xad7d5351, +}; +static const uint8_t simd_aes_sbox[256] = { + 0x63, 0x7c, 0x77, 0x7b, 0xf2, 0x6b, 0x6f, 0xc5, 0x30, 0x01, 0x67, 0x2b, 0xfe, 0xd7, 0xab, 0x76, + 0xca, 0x82, 0xc9, 0x7d, 0xfa, 0x59, 0x47, 0xf0, 0xad, 0xd4, 0xa2, 0xaf, 0x9c, 0xa4, 0x72, 0xc0, + 0xb7, 0xfd, 0x93, 0x26, 0x36, 0x3f, 0xf7, 0xcc, 0x34, 0xa5, 0xe5, 0xf1, 0x71, 0xd8, 0x31, 0x15, + 0x04, 0xc7, 0x23, 0xc3, 0x18, 0x96, 0x05, 0x9a, 0x07, 0x12, 0x80, 0xe2, 0xeb, 0x27, 0xb2, 0x75, + 0x09, 0x83, 0x2c, 0x1a, 0x1b, 0x6e, 0x5a, 0xa0, 0x52, 0x3b, 0xd6, 0xb3, 0x29, 0xe3, 0x2f, 0x84, + 0x53, 0xd1, 0x00, 0xed, 0x20, 0xfc, 0xb1, 0x5b, 0x6a, 0xcb, 0xbe, 0x39, 0x4a, 0x4c, 0x58, 0xcf, + 0xd0, 0xef, 0xaa, 0xfb, 0x43, 0x4d, 0x33, 0x85, 0x45, 0xf9, 0x02, 0x7f, 0x50, 0x3c, 0x9f, 0xa8, + 0x51, 0xa3, 0x40, 0x8f, 0x92, 0x9d, 0x38, 0xf5, 0xbc, 0xb6, 0xda, 0x21, 0x10, 0xff, 0xf3, 0xd2, + 0xcd, 0x0c, 0x13, 0xec, 0x5f, 0x97, 0x44, 0x17, 0xc4, 0xa7, 0x7e, 0x3d, 0x64, 0x5d, 0x19, 0x73, + 0x60, 0x81, 0x4f, 0xdc, 0x22, 0x2a, 0x90, 0x88, 0x46, 0xee, 0xb8, 0x14, 0xde, 0x5e, 0x0b, 0xdb, + 0xe0, 0x32, 0x3a, 0x0a, 0x49, 0x06, 0x24, 0x5c, 0xc2, 0xd3, 0xac, 0x62, 0x91, 0x95, 0xe4, 0x79, + 0xe7, 0xc8, 0x37, 0x6d, 0x8d, 0xd5, 0x4e, 0xa9, 0x6c, 0x56, 0xf4, 0xea, 0x65, 0x7a, 0xae, 0x08, + 0xba, 0x78, 0x25, 0x2e, 0x1c, 0xa6, 0xb4, 0xc6, 0xe8, 0xdd, 0x74, 0x1f, 0x4b, 0xbd, 0x8b, 0x8a, + 0x70, 0x3e, 0xb5, 0x66, 0x48, 0x03, 0xf6, 0x0e, 0x61, 0x35, 0x57, 0xb9, 0x86, 0xc1, 0x1d, 0x9e, + 0xe1, 0xf8, 0x98, 0x11, 0x69, 0xd9, 0x8e, 0x94, 0x9b, 0x1e, 0x87, 0xe9, 0xce, 0x55, 0x28, 0xdf, + 0x8c, 0xa1, 0x89, 0x0d, 0xbf, 0xe6, 0x42, 0x68, 0x41, 0x99, 0x2d, 0x0f, 0xb0, 0x54, 0xbb, 0x16, +}; +static const uint8_t simd_aes_isbox[256] = { + 0x52, 0x09, 0x6a, 0xd5, 0x30, 0x36, 0xa5, 0x38, 0xbf, 0x40, 0xa3, 0x9e, 0x81, 0xf3, 0xd7, 0xfb, + 0x7c, 0xe3, 0x39, 0x82, 0x9b, 0x2f, 0xff, 0x87, 0x34, 0x8e, 0x43, 0x44, 0xc4, 0xde, 0xe9, 0xcb, + 0x54, 0x7b, 0x94, 0x32, 0xa6, 0xc2, 0x23, 0x3d, 0xee, 0x4c, 0x95, 0x0b, 0x42, 0xfa, 0xc3, 0x4e, + 0x08, 0x2e, 0xa1, 0x66, 0x28, 0xd9, 0x24, 0xb2, 0x76, 0x5b, 0xa2, 0x49, 0x6d, 0x8b, 0xd1, 0x25, + 0x72, 0xf8, 0xf6, 0x64, 0x86, 0x68, 0x98, 0x16, 0xd4, 0xa4, 0x5c, 0xcc, 0x5d, 0x65, 0xb6, 0x92, + 0x6c, 0x70, 0x48, 0x50, 0xfd, 0xed, 0xb9, 0xda, 0x5e, 0x15, 0x46, 0x57, 0xa7, 0x8d, 0x9d, 0x84, + 0x90, 0xd8, 0xab, 0x00, 0x8c, 0xbc, 0xd3, 0x0a, 0xf7, 0xe4, 0x58, 0x05, 0xb8, 0xb3, 0x45, 0x06, + 0xd0, 0x2c, 0x1e, 0x8f, 0xca, 0x3f, 0x0f, 0x02, 0xc1, 0xaf, 0xbd, 0x03, 0x01, 0x13, 0x8a, 0x6b, + 0x3a, 0x91, 0x11, 0x41, 0x4f, 0x67, 0xdc, 0xea, 0x97, 0xf2, 0xcf, 0xce, 0xf0, 0xb4, 0xe6, 0x73, + 0x96, 0xac, 0x74, 0x22, 0xe7, 0xad, 0x35, 0x85, 0xe2, 0xf9, 0x37, 0xe8, 0x1c, 0x75, 0xdf, 0x6e, + 0x47, 0xf1, 0x1a, 0x71, 0x1d, 0x29, 0xc5, 0x89, 0x6f, 0xb7, 0x62, 0x0e, 0xaa, 0x18, 0xbe, 0x1b, + 0xfc, 0x56, 0x3e, 0x4b, 0xc6, 0xd2, 0x79, 0x20, 0x9a, 0xdb, 0xc0, 0xfe, 0x78, 0xcd, 0x5a, 0xf4, + 0x1f, 0xdd, 0xa8, 0x33, 0x88, 0x07, 0xc7, 0x31, 0xb1, 0x12, 0x10, 0x59, 0x27, 0x80, 0xec, 0x5f, + 0x60, 0x51, 0x7f, 0xa9, 0x19, 0xb5, 0x4a, 0x0d, 0x2d, 0xe5, 0x7a, 0x9f, 0x93, 0xc9, 0x9c, 0xef, + 0xa0, 0xe0, 0x3b, 0x4d, 0xae, 0x2a, 0xf5, 0xb0, 0xc8, 0xeb, 0xbb, 0x3c, 0x83, 0x53, 0x99, 0x61, + 0x17, 0x2b, 0x04, 0x7e, 0xba, 0x77, 0xd6, 0x26, 0xe1, 0x69, 0x14, 0x63, 0x55, 0x21, 0x0c, 0x7d, +}; + +SI int simd_fe_mode(uint64_t fpcr) { + static const int m[4] = {FE_TONEAREST, FE_UPWARD, FE_DOWNWARD, FE_TOWARDZERO}; + return m[(fpcr >> 22) & 3]; +} + +#define F s +#define T float +#define U uint32_t +#define NB 32 +#define EB 8 +#define MB 23 +#define VT vf32 +#define VTU vu32 +#define VTS vs32 +#define NL 4 +#include "simd_fp.h" +#undef F +#undef T +#undef U +#undef NB +#undef EB +#undef MB +#undef VT +#undef VTU +#undef VTS +#undef NL +#define F d +#define T double +#define U uint64_t +#define NB 64 +#define EB 11 +#define MB 52 +#define VT vf64 +#define VTU vu64 +#define VTS vs64 +#define NL 2 +#include "simd_fp.h" +#undef F +#undef T +#undef U +#undef NB +#undef EB +#undef MB +#undef VT +#undef VTU +#undef VTS +#undef NL + +// ---- precision conversions (FCVT, FCVTL/N, FCVTXN); FZ flushes single/double inputs, never halves +SI double simd_s2d(Cpu *c, float x) { + uint32_t u = fb_s(fz_s(c, x)); + if (isnan_s(u)) { + if (c->fpcr & FPCR_DN) return fv_d(0x7ff8000000000000ull); + return fv_d(((uint64_t)(u >> 31) << 63) | 0x7ff8000000000000ull | ((uint64_t)(u & 0x3fffff) << 29)); + } + return (double)fv_s(u); +} +SN float simd_d2s_slow(Cpu *c, double x) { +#pragma STDC FENV_ACCESS ON + int old = fegetround(); + fesetround(simd_fe_mode(c->fpcr)); + volatile double vx = x; + volatile float r = (float)vx; + fesetround(old); + return r; +} +SI float simd_d2s(Cpu *c, double x) { + uint64_t u = fb_d(fz_d(c, x)); + if (isnan_d(u)) { + if (c->fpcr & FPCR_DN) return fv_s(0x7fc00000u); + return fv_s((uint32_t)(u >> 63) << 31 | 0x7fc00000u | (uint32_t)((u >> 29) & 0x3fffff)); + } + x = fv_d(u); + if (__builtin_expect(c->fpcr & FPCR_SLOW, 0)) { + if ((c->fpcr & FPCR_FZ) && __builtin_fabs(x) < 0x1p-126 && x != 0) return fv_s((uint32_t)(u >> 63) << 31); + if (c->fpcr & FPCR_RM) return simd_d2s_slow(c, x); // [TOTK] FZ alone: the host's nearest mode is right + } + return (float)x; +} +// round to odd (FCVTXN) +SI float simd_d2s_odd(Cpu *c, double x) { + uint64_t u = fb_d(fz_d(c, x)); + if (isnan_d(u)) return simd_d2s(c, x); + x = fv_d(u); + if ((c->fpcr & FPCR_FZ) && __builtin_fabs(x) < 0x1p-126 && x != 0) return fv_s((uint32_t)(u >> 63) << 31); + float f = (float)x; + if ((double)f != x && x == x) { + uint32_t b = fb_s(f); + if (__builtin_fabs((double)f) > __builtin_fabs(x)) b--; + return fv_s(b | 1); + } + return f; +} +SI uint64_t simd_rnd_shift(uint64_t m, int sh, int rm, int neg) { + if (sh >= 64) return (rm == 1 && !neg && m) || (rm == 2 && neg && m); + uint64_t q = m >> sh, rem = m & ((1ull << sh) - 1), half = 1ull << (sh - 1); + if (rm == 0) q += rem > half || (rem == half && (q & 1)); + else if (rm == 1) q += rem && !neg; + else if (rm == 2) q += rem && neg; + return q; +} +// double (exact image of the single/double source, NaN payload left-aligned) -> half, FPCR rounding +SI uint16_t simd_d2h(Cpu *c, uint64_t u) { + uint64_t fpcr = c->fpcr; + int ahp = !!(fpcr & FPCR_AHP), rm = (fpcr >> 22) & 3, neg = (int)(u >> 63); + uint16_t s = (uint16_t)(neg << 15); + int e = (int)((u >> 52) & 0x7ff); + uint64_t f = u & ((1ull << 52) - 1); + if (e == 0x7ff) { + if (f) return ahp ? s : (fpcr & FPCR_DN) ? 0x7e00 : (uint16_t)(s | 0x7e00 | (f >> 42)); + return ahp ? (uint16_t)(s | 0x7fff) : (uint16_t)(s | 0x7c00); + } + if (!e && !f) return s; + uint64_t m = e ? f | (1ull << 52) : f; + int E = e ? e - 1075 : -1074, ue = e ? e - 1023 : -1023; + uint64_t q; + int bits; + if (ue < -14) { + q = simd_rnd_shift(m, -24 - E, rm, neg); + bits = (int)q; + } else { + q = simd_rnd_shift(m, 42, rm, neg); + bits = ((ue + 14) << 10) + (int)q; + } + if (ahp) return (uint16_t)(s | (bits > 0x7fff ? 0x7fff : bits)); + if (bits >= 0x7c00) { + int inf = rm == 0 || (rm == 1 && !neg) || (rm == 2 && neg); + return (uint16_t)(s | (inf ? 0x7c00 : 0x7bff)); + } + return (uint16_t)(s | bits); +} +SI uint16_t simd_s2h(Cpu *c, float x) { + uint32_t u = fb_s(fz_s(c, x)); + uint64_t d = isnan_s(u) ? ((uint64_t)(u >> 31) << 63) | 0x7ff0000000000000ull | ((uint64_t)(u & 0x7fffff) << 29) + : fb_d((double)fv_s(u)); + return simd_d2h(c, d); +} +SI uint16_t simd_dd2h(Cpu *c, double x) { return simd_d2h(c, fb_d(fz_d(c, x))); } +SI uint64_t simd_h2dbits(Cpu *c, uint16_t h) { + uint64_t s = (uint64_t)(h >> 15) << 63; + int e = (h >> 10) & 31, f = h & 0x3ff; + if (e == 31 && !(c->fpcr & FPCR_AHP)) { + if (!f) return s | 0x7ff0000000000000ull; + if (c->fpcr & FPCR_DN) return 0x7ff8000000000000ull; + return s | 0x7ff8000000000000ull | ((uint64_t)f << 42); + } + if (!e) return fb_d((s ? -1.0 : 1.0) * f * 0x1p-24); + return s | ((uint64_t)(e - 15 + 1023) << 52) | ((uint64_t)f << 42); +} +SI double simd_h2d(Cpu *c, uint16_t h) { return fv_d(simd_h2dbits(c, h)); } +SI float simd_h2s(Cpu *c, uint16_t h) { + uint64_t d = simd_h2dbits(c, h); + if (isnan_d(d)) return fv_s((uint32_t)(d >> 63) << 31 | 0x7fc00000u | (uint32_t)((d >> 29) & 0x3fffff)); + return (float)fv_d(d); +} + +// ---- scalar FP instructions (FP data processing) +#define FP_SETS(d, x) (c->v[d] = fb_s(x)) +#define FP_SETD(d, x) (c->v[d] = fb_d(x)) +SI void fp_op_s(Cpu *c, int op, int d, int n, int m) { FP_SETS(d, op_s(c, op, VSF32(n), VSF32(m), 0)); } +SI void fp_op_d(Cpu *c, int op, int d, int n, int m) { FP_SETD(d, op_d(c, op, VSF64(n), VSF64(m), 0)); } +// FMADD family: neg_a / neg_n negate the addend / first factor before the fused op +SI void fp_fma_s(Cpu *c, int na, int nn, int d, int n, int m, int a) { + float x = VSF32(a), y = VSF32(n); + if (na) x = neg_s(x); + if (nn) y = neg_s(y); + FP_SETS(d, op_s(c, FOP_MADD, x, y, VSF32(m))); +} +SI void fp_fma_d(Cpu *c, int na, int nn, int d, int n, int m, int a) { + double x = VSF64(a), y = VSF64(n); + if (na) x = neg_d(x); + if (nn) y = neg_d(y); + FP_SETD(d, op_d(c, FOP_MADD, x, y, VSF64(m))); +} + +// ---- integer lanes (complex ops run lane by lane; constant sizes fold to plain loads/stores) +SI uint64_t lmask(int sz) { return sz == 3 ? ~0ull : (1ull << (8 << sz)) - 1; } +SI int64_t lsx(uint64_t v, int sz) { int s = 64 - (8 << sz); return (int64_t)(v << s) >> s; } +SI uint64_t lget(const u128 *v, int sz, int i) { + const uint8_t *p = (const uint8_t *)v + (i << sz); + switch (sz) { + case 0: return *p; + case 1: return *(const sa16 *)p; + case 2: return *(const sa32 *)p; + default: return *(const sa64 *)p; + } +} +SI void lset(u128 *v, int sz, int i, uint64_t x) { + uint8_t *p = (uint8_t *)v + (i << sz); + switch (sz) { + case 0: *p = (uint8_t)x; break; + case 1: *(sa16 *)p = (uint16_t)x; break; + case 2: *(sa32 *)p = (uint32_t)x; break; + default: *(sa64 *)p = x; break; + } +} +SI uint64_t sat_s(__int128 v, int sz) { + __int128 hi = ((__int128)1 << ((8 << sz) - 1)) - 1, lo = -hi - 1; + return (uint64_t)(v > hi ? hi : v < lo ? lo : v) & lmask(sz); +} +SI uint64_t sat_u(__int128 v, int sz) { + __int128 hi = ((__int128)1 << (8 << sz)) - 1; + return (uint64_t)(v > hi ? hi : v < 0 ? 0 : v); +} +SI uint64_t sat_uu(unsigned __int128 v, int sz) { return v > lmask(sz) ? lmask(sz) : (uint64_t)v; } +SI __int128 shl128(__int128 v, int s) { return (__int128)((unsigned __int128)v << s); } +// right shift by s >= 1 with optional rounding, as an infinite-precision value +SI __int128 lshr_s(int64_t v, int s, int rnd) { + if (s > 100) s = 100; + return rnd ? ((__int128)v + ((__int128)1 << (s - 1))) >> s : (__int128)v >> s; +} +SI unsigned __int128 lshr_u(uint64_t v, int s, int rnd) { + if (s > 100) s = 100; + return rnd ? ((unsigned __int128)v + ((unsigned __int128)1 << (s - 1))) >> s : (unsigned __int128)v >> s; +} +SI uint64_t pmul64(uint64_t a, uint64_t b, uint64_t *hi) { // carry-less + uint64_t lo = 0, h = 0; + for (int i = 0; i < 64; i++) + if ((b >> i) & 1) lo ^= a << i, h ^= i ? a >> (64 - i) : 0; + if (hi) *hi = h; + return lo; +} + +enum { + LOP_SQADD, LOP_UQADD, LOP_SQSUB, LOP_UQSUB, LOP_SUQADD, LOP_USQADD, + LOP_SHL, LOP_RSHL, LOP_QSHL, LOP_QRSHL, // register shifts (b = signed byte), U selects unsigned + LOP_SHLI, LOP_QSHLI, LOP_QSHLUI, LOP_SHRI, LOP_RSHRI, LOP_SRAI, LOP_RSRAI, LOP_SRI, LOP_SLI, + LOP_SQDMULH, LOP_SQRDMULH, LOP_PMUL, + LOP_SQABS, LOP_SQNEG, LOP_ABS, LOP_NEG, LOP_CLS, LOP_CLZ, LOP_CNT, LOP_RBIT, LOP_URECPE, LOP_URSQRTE, + LOP_HADD, LOP_RHADD, LOP_HSUB, LOP_ABD, LOP_ABA, + // long: result lanes sz+1 + LOP_ADDL, LOP_SUBL, LOP_ABDL, LOP_ABAL, LOP_MLAL, LOP_MLSL, LOP_MULL, LOP_ADDW, LOP_SUBW, + LOP_SQDMULL, LOP_SQDMLAL, LOP_SQDMLSL, LOP_PMULL, LOP_SHLL, + // narrow: sz = destination lane size, a/b are sz+1 + LOP_XTN, LOP_SQXTN, LOP_UQXTN, LOP_SQXTUN, LOP_SHRN, LOP_RSHRN, LOP_QSHRN, LOP_QRSHRN, LOP_QSHRUN, + LOP_QRSHRUN, LOP_ADDHN, LOP_RADDHN, LOP_SUBHN, LOP_RSUBHN, + LOP_SHLLI, // SSHLL/USHLL: extend, shift (long) +}; +#define LOP_U 256 // unsigned variant flag + +SI uint64_t lop(int op, int sz, uint64_t a, uint64_t b, uint64_t d) { + int u = op >> 8, es = 8 << sz; + op &= 255; + int64_t sa = lsx(a, sz), sb = lsx(b, sz); + __int128 xa = u ? (__int128)a : (__int128)sa, xb = u ? (__int128)b : (__int128)sb; // extended operands + int sh = (int8_t)b; + switch (op) { + case LOP_SQADD: return sat_s((__int128)sa + sb, sz); + case LOP_UQADD: return sat_u((__int128)a + b, sz); + case LOP_SQSUB: return sat_s((__int128)sa - sb, sz); + case LOP_UQSUB: return sat_u((__int128)a - b, sz); + case LOP_SUQADD: return sat_s((__int128)lsx(d, sz) + a, sz); + case LOP_USQADD: return sat_u((__int128)d + sa, sz); + case LOP_SHL: case LOP_RSHL: case LOP_QSHL: case LOP_QRSHL: { + int sat = op == LOP_QSHL || op == LOP_QRSHL, rnd = op == LOP_RSHL || op == LOP_QRSHL; + if (sh >= 0) { + if (sat) return u ? sat_uu((unsigned __int128)a << (sh > 64 ? 64 : sh), sz) + : sat_s(shl128(sa, sh > 64 ? 64 : sh), sz); + return sh >= es ? 0 : a << sh; + } + return u ? (uint64_t)lshr_u(a, -sh, rnd) : (uint64_t)lshr_s(sa, -sh, rnd); + } + case LOP_SHLI: return a << b; + case LOP_QSHLI: return u ? sat_uu((unsigned __int128)a << b, sz) : sat_s(shl128(sa, (int)b), sz); + case LOP_QSHLUI: return sat_u(shl128(sa, (int)b), sz); + case LOP_SHRI: return u ? (uint64_t)lshr_u(a, (int)b, 0) : (uint64_t)lshr_s(sa, (int)b, 0); + case LOP_RSHRI: return u ? (uint64_t)lshr_u(a, (int)b, 1) : (uint64_t)lshr_s(sa, (int)b, 1); + case LOP_SRAI: return d + (u ? (uint64_t)lshr_u(a, (int)b, 0) : (uint64_t)lshr_s(sa, (int)b, 0)); + case LOP_RSRAI: return d + (u ? (uint64_t)lshr_u(a, (int)b, 1) : (uint64_t)lshr_s(sa, (int)b, 1)); + case LOP_SRI: { + uint64_t m = (int)b >= es ? 0 : lmask(sz) >> b; + return (d & ~m) | ((int)b >= es ? 0 : a >> b); + } + case LOP_SLI: { uint64_t m = (lmask(sz) << b) & lmask(sz); return (d & ~m) | ((a << b) & m); } + case LOP_SQDMULH: return sat_s(((__int128)sa * sb * 2) >> es, sz); + case LOP_SQRDMULH: return sat_s(((__int128)sa * sb * 2 + ((__int128)1 << (es - 1))) >> es, sz); + case LOP_PMUL: return pmul64(a, b, 0); + case LOP_SQABS: return sat_s(sa < 0 ? -(__int128)sa : sa, sz); + case LOP_SQNEG: return sat_s(-(__int128)sa, sz); + case LOP_ABS: return sa < 0 ? (uint64_t)-sa : (uint64_t)sa; + case LOP_NEG: return -a; + case LOP_CLS: { + uint64_t x = (uint64_t)(sa ^ (sa >> 1)) & lmask(sz); + return (x ? (uint64_t)__builtin_clzll(x << (64 - es)) : (uint64_t)es) - 1; + } + case LOP_CLZ: return a ? (uint64_t)__builtin_clzll(a << (64 - es)) : (uint64_t)es; + case LOP_CNT: return (uint64_t)__builtin_popcountll(a); + case LOP_RBIT: return __builtin_bitreverse8((uint8_t)a); + case LOP_URECPE: return (a >> 31) ? (uint64_t)(256 | simd_recip_tab[(a >> 23) & 255]) << 23 : 0xffffffffull; + case LOP_URSQRTE: return (a >> 30) ? (uint64_t)(256 | simd_rsqrt_tab[((a >> 23) & 511) - 128]) << 23 : 0xffffffffull; + case LOP_HADD: return (uint64_t)((xa + xb) >> 1); + case LOP_RHADD: return (uint64_t)((xa + xb + 1) >> 1); + case LOP_HSUB: return (uint64_t)((xa - xb) >> 1); + case LOP_ABD: return (uint64_t)(xa > xb ? xa - xb : xb - xa); + case LOP_ABA: return d + (uint64_t)(xa > xb ? xa - xb : xb - xa); + case LOP_ADDL: return (uint64_t)(xa + xb); + case LOP_SUBL: return (uint64_t)(xa - xb); + case LOP_ABDL: return (uint64_t)(xa > xb ? xa - xb : xb - xa); + case LOP_ABAL: return d + (uint64_t)(xa > xb ? xa - xb : xb - xa); + case LOP_MLAL: return d + (uint64_t)(xa * xb); + case LOP_MLSL: return d - (uint64_t)(xa * xb); + case LOP_MULL: return (uint64_t)(xa * xb); + case LOP_ADDW: return a + (uint64_t)xb; + case LOP_SUBW: return a - (uint64_t)xb; + case LOP_SQDMULL: return sat_s((__int128)sa * sb * 2, sz + 1); + case LOP_SQDMLAL: return sat_s((__int128)lsx(d, sz + 1) + (__int128)lsx(sat_s((__int128)sa * sb * 2, sz + 1), sz + 1), sz + 1); + case LOP_SQDMLSL: return sat_s((__int128)lsx(d, sz + 1) - (__int128)lsx(sat_s((__int128)sa * sb * 2, sz + 1), sz + 1), sz + 1); + case LOP_PMULL: return pmul64(a, b, 0); + case LOP_SHLL: return a << es; + // narrowing: a (and b for xHN) are 2*es wide + case LOP_XTN: return a; + case LOP_SQXTN: return sat_s(lsx(a, sz + 1), sz); + case LOP_UQXTN: return sat_u(a, sz); + case LOP_SQXTUN: return sat_u(lsx(a, sz + 1), sz); + case LOP_SHRN: return a >> b; + case LOP_RSHRN: return (uint64_t)lshr_u(a, (int)b, 1); + case LOP_QSHRN: case LOP_QRSHRN: { + int rnd = op == LOP_QRSHRN; + return u ? sat_u((__int128)lshr_u(a, (int)b, rnd), sz) : sat_s(lshr_s(lsx(a, sz + 1), (int)b, rnd), sz); + } + case LOP_QSHRUN: return sat_u(lshr_s(lsx(a, sz + 1), (int)b, 0), sz); + case LOP_QRSHRUN: return sat_u(lshr_s(lsx(a, sz + 1), (int)b, 1), sz); + case LOP_ADDHN: return (a + b) >> es; + case LOP_RADDHN: return (a + b + (1ull << (es - 1))) >> es; + case LOP_SUBHN: return (a - b) >> es; + case LOP_RSUBHN: return (a - b + (1ull << (es - 1))) >> es; + case LOP_SHLLI: return (uint64_t)xa << b; + } + return 0; +} + +// Generic lane loop: ne lanes; result lane i (size szd) goes to lane offd+i of d (other lanes kept when keep, +// else zero); a = n lane offn+i (size szn), b = m lane offm+i (size szm), or lane bidx of m when bidx >= 0, +// or the constant bimm when m < 0; the accumulator is d's old lane offd+i. +SI void sl_gen(Cpu *c, int op, int sz, int ne, int d, int szd, int offd, int keep, int n, int szn, int offn, int m, + int szm, int offm, int bidx, uint64_t bimm) { + u128 a = c->v[n], b = m >= 0 ? c->v[m] : 0, o = c->v[d], r = keep ? o : 0; + for (int i = 0; i < ne; i++) { + uint64_t y = m < 0 ? bimm : lget(&b, szm, bidx >= 0 ? bidx : offm + i); + lset(&r, szd, offd + i, lop(op, sz, lget(&a, szn, offn + i), y, lget(&o, szd, offd + i))); + } + c->v[d] = r; +} +// SADDLP/UADDLP/SADALP/UADALP +SI void sl_addlp(Cpu *c, int u, int acc, int Q, int sz, int d, int n) { + u128 a = c->v[n], o = c->v[d], r = 0; + for (int i = 0; i < (4 << Q) >> sz; i++) { + uint64_t x = lget(&a, sz, 2 * i), y = lget(&a, sz, 2 * i + 1); + uint64_t s = u ? x + y : (uint64_t)(lsx(x, sz) + lsx(y, sz)); + lset(&r, sz + 1, i, s + (acc ? lget(&o, sz + 1, i) : 0)); + } + c->v[d] = r; +} +// ADDV, SADDLV/UADDLV, SMAXV/UMAXV, SMINV/UMINV (op 0..3; u = unsigned) +SI void sl_across(Cpu *c, int op, int u, int Q, int sz, int d, int n) { + u128 a = c->v[n]; + int ne = (8 << Q) >> sz; + int64_t acc = u ? (int64_t)lget(&a, sz, 0) : lsx(lget(&a, sz, 0), sz); + for (int i = 1; i < ne; i++) { + int64_t x = u ? (int64_t)lget(&a, sz, i) : lsx(lget(&a, sz, i), sz); + if (op <= 1) acc += x; + else if (op == 2) acc = u ? ((uint64_t)x > (uint64_t)acc ? x : acc) : (x > acc ? x : acc); + else acc = u ? ((uint64_t)x < (uint64_t)acc ? x : acc) : (x < acc ? x : acc); + } + c->v[d] = (uint64_t)acc & lmask(op == 1 ? sz + 1 : sz); +} +// PMULL2/PMULL 1Q (64x64 -> 128 carry-less) +SI void sl_pmull64(Cpu *c, int part, int d, int n, int m) { + uint64_t hi, lo = pmul64(((sa64 *)&c->v[n])[part], ((sa64 *)&c->v[m])[part], &hi); + c->v[d] = ((u128)hi << 64) | lo; +} + +// ---- FP vector helpers: lanes of the 64-bit form are computed on a zero upper half +SI vf32 vsplat_s(vf32 v, int i, int Q) { float x = v[i]; return Q ? (vf32){x, x, x, x} : (vf32){x, x, 0, 0}; } +SI vf64 vsplat_d(vf64 v, int i, int Q) { double x = v[i]; (void)Q; return (vf64){x, x}; } +#define SV_FOP_S(op, Q, d, n, m, a) VW(vf32, Q, d, vop_s(c, op, n, m, a)) +#define SV_FOP_D(op, Q, d, n, m, a) VW(vf64, Q, d, vop_d(c, op, n, m, a)) +SI vf32 vneg_s(vf32 v) { return (vf32)((vu32)v ^ 0x80000000u); } +SI vf64 vneg_d(vf64 v) { return (vf64)((vu64)v ^ 0x8000000000000000ull); } +SI float pair_s(Cpu *c, int op, float a, float b) { return op_s(c, op, a, b, 0); } +SI double pair_d(Cpu *c, int op, double a, double b) { return op_d(c, op, a, b, 0); } +// FMAXV/FMINV/FMAXNMV/FMINNMV (4S): pairwise tree +SI void sv_fred_s(Cpu *c, int op, int d, int n) { + vf32 v = VREG(vf32, n); + c->v[d] = fb_s(pair_s(c, op, pair_s(c, op, v[0], v[1]), pair_s(c, op, v[2], v[3]))); +} +// lane-wise FP conversions +SI vu32 vtoint_s(Cpu *c, vf32 v, int rm, int fbits, int uns) { + vu32 r; + for (int i = 0; i < 4; i++) r[i] = (uint32_t)toint_s(c, v[i], rm, fbits, 32, uns); + return r; +} +SI vu64 vtoint_d(Cpu *c, vf64 v, int rm, int fbits, int uns) { + vu64 r; + for (int i = 0; i < 2; i++) r[i] = toint_d(c, v[i], rm, fbits, 64, uns); + return r; +} +SI vf32 vfromint_s(Cpu *c, vu32 v, int fbits, int uns) { + vf32 r; + if (!(c->fpcr & FPCR_RM)) { + r = uns ? CVT(v, vf32) : CVT((vs32)v, vf32); + if (fbits) r = r / (float)(1ull << (fbits - 1)) / 2.0f; + return r; + } + for (int i = 0; i < 4; i++) r[i] = fromint_s(c, v[i], 32, uns, fbits); + return r; +} +SI vf64 vfromint_d(Cpu *c, vu64 v, int fbits, int uns) { + vf64 r; + for (int i = 0; i < 2; i++) r[i] = fromint_d(c, v[i], 64, uns, fbits); + return r; +} +SI vf32 vrecpe_s(Cpu *c, vf32 v) { for (int i = 0; i < 4; i++) v[i] = recpe_s(c, v[i]); return v; } +SI vf64 vrecpe_d(Cpu *c, vf64 v) { for (int i = 0; i < 2; i++) v[i] = recpe_d(c, v[i]); return v; } +SI vf32 vrsqrte_s(Cpu *c, vf32 v) { for (int i = 0; i < 4; i++) v[i] = rsqrte_s(c, v[i]); return v; } +SI vf64 vrsqrte_d(Cpu *c, vf64 v) { for (int i = 0; i < 2; i++) v[i] = rsqrte_d(c, v[i]); return v; } + +// FCVTL/FCVTL2 (h->s, s->d), FCVTN/FCVTN2 (s->h, d->s), FCVTXN/FCVTXN2 +SI void sv_fcvtl(Cpu *c, int sz, int part, int d, int n) { + u128 a = c->v[n], r = 0; + if (sz == 0) { + vf32 o; + for (int i = 0; i < 4; i++) o[i] = simd_h2s(c, (uint16_t)lget(&a, 1, 4 * part + i)); + r = (u128)o; + } else { + vf64 o; + for (int i = 0; i < 2; i++) o[i] = simd_s2d(c, fv_s((uint32_t)lget(&a, 2, 2 * part + i))); + r = (u128)o; + } + c->v[d] = r; +} +SI void sv_fcvtn(Cpu *c, int sz, int part, int odd, int d, int n) { + u128 a = c->v[n], r = part ? (uint64_t)c->v[d] : 0; + if (sz == 0) { + for (int i = 0; i < 4; i++) lset(&r, 1, 4 * part + i, simd_s2h(c, fv_s((uint32_t)lget(&a, 2, i)))); + } else { + for (int i = 0; i < 2; i++) { + double x = fv_d(lget(&a, 3, i)); + lset(&r, 2, 2 * part + i, fb_s(odd ? simd_d2s_odd(c, x) : simd_d2s(c, x))); + } + } + c->v[d] = r; +} + +// TBL / TBX with len+1 table registers +SI void sv_tbl(Cpu *c, int Q, int len, int tbx, int d, int n, int m) { + uint8_t tab[64]; + for (int i = 0; i <= len; i++) memcpy(tab + 16 * i, &c->v[(n + i) & 31], 16); + u128 idx = c->v[m], o = c->v[d], r = 0; + for (int i = 0; i < 8 << Q; i++) { + unsigned k = (unsigned)lget(&idx, 0, i); + lset(&r, 0, i, k < 16u * (len + 1) ? tab[k] : tbx ? lget(&o, 0, i) : 0); + } + c->v[d] = r; +} + +// ---- CRC32 / CRC32C (bytes = 1, 2, 4, 8) +SI uint32_t simd_crc(const uint32_t *t, uint32_t acc, uint64_t v, int bytes) { + for (int i = 0; i < bytes; i++) acc = t[(acc ^ (uint8_t)(v >> (8 * i))) & 0xff] ^ (acc >> 8); + return acc; +} +#define CRC32(acc, v, n) simd_crc(simd_crc32_tab, (uint32_t)(acc), (v), n) +#define CRC32C(acc, v, n) simd_crc(simd_crc32c_tab, (uint32_t)(acc), (v), n) + +// ---- crypto (AES, SHA1, SHA256) +SI uint8_t aes_xt(uint8_t x) { return (uint8_t)((x << 1) ^ ((x >> 7) * 0x1b)); } +SI uint8_t aes_mul(uint8_t x, int k) { // k in {9, 11, 13, 14} + uint8_t x2 = aes_xt(x), x4 = aes_xt(x2), x8 = aes_xt(x4); + return (uint8_t)(x8 ^ (k & 4 ? x4 : 0) ^ (k & 2 ? x2 : 0) ^ (k & 1 ? x : 0)); +} +SI void sv_aes(Cpu *c, int op, int d, int n) { // 0 AESE, 1 AESD, 2 AESMC, 3 AESIMC + uint8_t s[16], r[16]; + u128 v = op < 2 ? c->v[d] ^ c->v[n] : c->v[n]; + memcpy(s, &v, 16); + if (op == 0) { + for (int i = 0; i < 16; i++) r[i] = simd_aes_sbox[s[(i + 4 * (i & 3)) & 15]]; + } else if (op == 1) { + for (int i = 0; i < 16; i++) r[i] = simd_aes_isbox[s[(i - 4 * (i & 3)) & 15]]; + } else { + for (int col = 0; col < 4; col++) { + uint8_t *a = s + 4 * col, *o = r + 4 * col; + for (int i = 0; i < 4; i++) { + uint8_t a0 = a[i], a1 = a[(i + 1) & 3], a2 = a[(i + 2) & 3], a3 = a[(i + 3) & 3]; + o[i] = op == 2 ? (uint8_t)(aes_xt(a0) ^ aes_xt(a1) ^ a1 ^ a2 ^ a3) + : (uint8_t)(aes_mul(a0, 14) ^ aes_mul(a1, 11) ^ aes_mul(a2, 13) ^ aes_mul(a3, 9)); + } + } + } + memcpy(&v, r, 16); + c->v[d] = v; +} +SI uint32_t rol32_(uint32_t x, int s) { return (x << s) | (x >> (32 - s)); } +SI uint32_t ror32_(uint32_t x, int s) { return (x >> s) | (x << (32 - s)); } +SI void sv_sha1hash(Cpu *c, int op, int d, int n, int m) { // 0 SHA1C, 1 SHA1P, 2 SHA1M + uint32_t x[4], w[4], y = (uint32_t)c->v[n]; + memcpy(x, &c->v[d], 16); + memcpy(w, &c->v[m], 16); + for (int e = 0; e < 4; e++) { + uint32_t t = op == 0 ? (((x[2] ^ x[3]) & x[1]) ^ x[3]) : op == 1 ? (x[1] ^ x[2] ^ x[3]) + : ((x[1] & x[2]) | ((x[1] | x[2]) & x[3])); + y = y + rol32_(x[0], 5) + t + w[e]; + x[1] = rol32_(x[1], 30); + uint32_t top = x[3]; + x[3] = x[2], x[2] = x[1], x[1] = x[0], x[0] = y, y = top; + } + memcpy(&c->v[d], x, 16); +} +SI void sv_sha256hash(Cpu *c, int part1, int d, int n, int m) { // SHA256H (part1), SHA256H2 + uint32_t x[4], y[4], w[4]; + memcpy(part1 ? x : y, &c->v[d], 16); + memcpy(part1 ? y : x, &c->v[n], 16); + memcpy(w, &c->v[m], 16); + for (int e = 0; e < 4; e++) { + uint32_t chs = ((y[1] ^ y[2]) & y[0]) ^ y[2]; + uint32_t maj = (x[0] & x[1]) | ((x[0] | x[1]) & x[2]); + uint32_t t1 = y[3] + (ror32_(y[0], 6) ^ ror32_(y[0], 11) ^ ror32_(y[0], 25)) + chs + w[e]; + x[3] = t1 + x[3]; + y[3] = t1 + (ror32_(x[0], 2) ^ ror32_(x[0], 13) ^ ror32_(x[0], 22)) + maj; + uint32_t t = y[3]; + y[3] = y[2], y[2] = y[1], y[1] = y[0], y[0] = x[3]; + x[3] = x[2], x[2] = x[1], x[1] = x[0], x[0] = t; + } + memcpy(&c->v[d], part1 ? x : y, 16); +} +SI void sv_sha_misc(Cpu *c, int op, int d, int n, int m) { // 0 SHA1SU0, 1 SHA1SU1, 2 SHA256SU0, 3 SHA256SU1, 4 SHA1H + uint32_t a[4], b[4], k[4], r[4], t[4]; + memcpy(a, &c->v[d], 16); + memcpy(b, &c->v[n], 16); + memcpy(k, &c->v[m], 16); + switch (op) { + case 0: + t[0] = a[2], t[1] = a[3], t[2] = b[0], t[3] = b[1]; + for (int i = 0; i < 4; i++) r[i] = t[i] ^ a[i] ^ k[i]; + break; + case 1: + for (int i = 0; i < 4; i++) t[i] = a[i] ^ (i < 3 ? b[i + 1] : 0); + for (int i = 0; i < 4; i++) r[i] = rol32_(t[i], 1); + r[3] ^= rol32_(t[0], 2); + break; + case 2: + t[0] = a[1], t[1] = a[2], t[2] = a[3], t[3] = b[0]; + for (int i = 0; i < 4; i++) r[i] = (ror32_(t[i], 7) ^ ror32_(t[i], 18) ^ (t[i] >> 3)) + a[i]; + break; + case 3: { + uint32_t t0[4] = {b[1], b[2], b[3], k[0]}; + for (int e = 0; e < 4; e++) { + uint32_t x = e < 2 ? k[e + 2] : r[e - 2]; + r[e] = (ror32_(x, 17) ^ ror32_(x, 19) ^ (x >> 10)) + a[e] + t0[e]; + } + break; + } + default: + c->v[d] = rol32_(b[0], 30); + return; + } + memcpy(&c->v[d], r, 16); +} + +// ---- loads / stores of lanes +SI uint64_t simd_ld(uint64_t a, int sz) { + switch (sz) { + case 0: return LD8(a); + case 1: return LD16(a); + case 2: return LD32(a); + default: return LD64(a); + } +} +SI void simd_st(uint64_t a, int sz, uint64_t v) { + switch (sz) { + case 0: ST8(a, v); break; + case 1: ST16(a, v); break; + case 2: ST32(a, v); break; + default: ST64(a, v); break; + } +} +// LDn/STn (multiple structures): selem interleaved registers, rpt repeats (LD1 with 2-4 registers) +SI void sl_ldm(Cpu *c, int Q, int sz, int selem, int rpt, int t, uint64_t ea) { + u128 r[4] = {0, 0, 0, 0}; + int ne = (8 << Q) >> sz; + for (int k = 0; k < rpt; k++) + for (int e = 0; e < ne; e++) + for (int s = 0; s < selem; s++, ea += 1 << sz) lset(&r[k + s], sz, e, simd_ld(ea, sz)); + for (int i = 0; i < rpt + selem - 1; i++) c->v[(t + i) & 31] = r[i]; +} +SI void sl_stm(Cpu *c, int Q, int sz, int selem, int rpt, int t, uint64_t ea) { + u128 r[4]; + int ne = (8 << Q) >> sz; + for (int i = 0; i < rpt + selem - 1; i++) r[i] = c->v[(t + i) & 31]; + for (int k = 0; k < rpt; k++) + for (int e = 0; e < ne; e++) + for (int s = 0; s < selem; s++, ea += 1 << sz) simd_st(ea, sz, lget(&r[k + s], sz, e)); +} +// single structure: lane idx of selem registers; LDnR replicates into every lane +SI void sl_lds(Cpu *c, int sz, int selem, int idx, int t, uint64_t ea) { + for (int s = 0; s < selem; s++) { + u128 v = c->v[(t + s) & 31]; + lset(&v, sz, idx, simd_ld(ea + (s << sz), sz)); + c->v[(t + s) & 31] = v; + } +} +SI void sl_sts(Cpu *c, int sz, int selem, int idx, int t, uint64_t ea) { + for (int s = 0; s < selem; s++) simd_st(ea + (s << sz), sz, lget(&c->v[(t + s) & 31], sz, idx)); +} +SI void sl_ldr(Cpu *c, int Q, int sz, int selem, int t, uint64_t ea) { + uint64_t x[4]; + for (int s = 0; s < selem; s++) x[s] = simd_ld(ea + (s << sz), sz); + for (int s = 0; s < selem; s++) { + u128 v = 0; + for (int e = 0; e < (8 << Q) >> sz; e++) lset(&v, sz, e, x[s]); + c->v[(t + s) & 31] = v; + } +} diff --git a/src/runtime/simd_fp.h b/src/runtime/simd_fp.h new file mode 100644 index 0000000..c63da68 --- /dev/null +++ b/src/runtime/simd_fp.h @@ -0,0 +1,347 @@ +// FP helpers for one precision; included twice by simd.h (F = s/float, d/double). No include guard. +// Scalar entry points take/return T, vector ones VT (all lanes; callers zero the unused half for Q=0). + +#define FN(x) SIMD_CAT(x, F) +#define SGN ((U)1 << (NB - 1)) +#define EXPM ((((U)1 << EB) - 1) << MB) +#define QB ((U)1 << (MB - 1)) +#define DNAN (EXPM | QB) +#define MINN ((U)1 << MB) // smallest normal +#define VK(v) ((VT){} + (T)(v)) +#define K(v) ((T)(v)) + +SI U FN(fb)(T x) { U u; memcpy(&u, &x, sizeof u); return u; } +SI T FN(fv)(U u) { T x; memcpy(&x, &u, sizeof x); return x; } +SI int FN(isnan)(U u) { return (u & ~SGN) > EXPM; } +SI int FN(issnan)(U u) { return FN(isnan)(u) && !(u & QB); } +SI int FN(isqnan)(U u) { return FN(isnan)(u) && (u & QB); } +SI int FN(isinf)(U u) { return (u & ~SGN) == EXPM; } +SI int FN(iszero)(U u) { return !(u & ~SGN); } +SI int FN(isden)(U u) { return !(u & EXPM) && (u & ~SGN); } +SI T FN(flush)(T x) { U u = FN(fb)(x); return FN(isden)(u) ? FN(fv)(u & SGN) : x; } +SI T FN(fz)(Cpu *c, T x) { return __builtin_expect(c->fpcr & FPCR_FZ, 0) ? FN(flush)(x) : x; } +SI T FN(neg)(T x) { return FN(fv)(FN(fb)(x) ^ SGN); } + +// FPProcessNaN(s): first signalling NaN (quietened), else first quiet NaN, else (invalid op) default NaN +SN T FN(nan1)(Cpu *c, T a) { + U x = FN(fb)(a); + if (!(c->fpcr & FPCR_DN) && FN(isnan)(x)) return FN(fv)(x | QB); + return FN(fv)(DNAN); +} +SN T FN(nan2)(Cpu *c, T a, T b) { + U x = FN(fb)(a), y = FN(fb)(b); + if (!(c->fpcr & FPCR_DN)) { + if (FN(issnan)(x)) return FN(fv)(x | QB); + if (FN(issnan)(y)) return FN(fv)(y | QB); + if (FN(isnan)(x)) return a; + if (FN(isnan)(y)) return b; + } + return FN(fv)(DNAN); +} +SN T FN(nan3)(Cpu *c, T a, T b, T d) { + U x = FN(fb)(a), y = FN(fb)(b), z = FN(fb)(d); + if (!(c->fpcr & FPCR_DN)) { + if (FN(issnan)(x)) return FN(fv)(x | QB); + if (FN(issnan)(y)) return FN(fv)(y | QB); + if (FN(issnan)(z)) return FN(fv)(z | QB); + if (FN(isnan)(x)) return a; + if (FN(isnan)(y)) return b; + if (FN(isnan)(z)) return d; + } + return FN(fv)(DNAN); +} + +// The arithmetic itself, for scalars (KK = K) and vectors (KK = VK). MADD: a + b*d (a = addend). +// RECPS/RSQRTS get their first operand already negated (FPNeg happens before NaN processing). +#define FCALC(op, a, b, d, KK) \ + ((op) == FOP_ADD ? (a) + (b) : (op) == FOP_SUB ? (a) - (b) \ + : (op) == FOP_MUL || (op) == FOP_MULX ? (a) * (b) : (op) == FOP_NMUL ? -((a) * (b)) \ + : (op) == FOP_DIV ? (a) / (b) : (op) == FOP_ABD ? __builtin_elementwise_abs((a) - (b)) \ + : (op) == FOP_SQRT ? __builtin_elementwise_sqrt(a) \ + : (op) == FOP_MADD ? __builtin_elementwise_fma(b, d, a) \ + : (op) == FOP_RECPS ? __builtin_elementwise_fma(a, b, KK(2)) \ + : (op) == FOP_RSQRTS ? __builtin_elementwise_fma(a, b, KK(3)) * KK(0.5) : (a)) + +// Lanes whose fast result needs fixing: NaN results (NaN operand or invalid op), RSQRTS overflow. +#define FBAD(op, r) ((op) == FOP_RSQRTS ? ((r) != (r)) | (__builtin_elementwise_abs(r) == K(INFINITY)) : (r) != (r)) + +// (3 + a*b) / 2 with one rounding (a already negated): halve a normal operand first, so a product beyond +// the format's range (which the fast path rounds to inf, or to the largest normal when rounding down) is exact. +SN T FN(rsqrts)(T a, T b) { + if (__builtin_elementwise_abs(a) >= FN(fv)(2 * MINN)) return __builtin_elementwise_fma(a * K(0.5), b, K(1.5)); + if (__builtin_elementwise_abs(b) >= FN(fv)(2 * MINN)) return __builtin_elementwise_fma(a, b * K(0.5), K(1.5)); + return __builtin_elementwise_fma(a, b, K(3)) * K(0.5); +} + +// ARM result for a lane flagged by FBAD (operands already flushed). +SN T FN(fix)(Cpu *c, int op, T a, T b, T d) { + U x = FN(fb)(a), y = FN(fb)(b), z = FN(fb)(d); + int nan = FN(isnan)(x) || FN(isnan)(y); + switch (op) { + case FOP_SQRT: return FN(nan1)(c, a); // NaN in, or negative -> default NaN + case FOP_MADD: + if (FN(isnan)(x) || FN(isnan)(y) || FN(isnan)(z)) { + if (FN(isqnan)(x) && ((FN(isinf)(y) && FN(iszero)(z)) || (FN(iszero)(y) && FN(isinf)(z)))) + return FN(fv)(DNAN); + return FN(nan3)(c, a, b, d); + } + return FN(fv)(DNAN); + case FOP_MULX: + if (!nan) return FN(fv)(((x ^ y) & SGN) | FN(fb)(K(2))); + break; + case FOP_RECPS: + if (!nan) return K(2); + break; + case FOP_RSQRTS: + if (!nan) { + if ((FN(isinf)(x) && FN(iszero)(y)) || (FN(iszero)(x) && FN(isinf)(y))) return K(1.5); + return FN(rsqrts)(a, b); + } + break; + case FOP_NMUL: return FN(neg)(FN(nan2)(c, a, b)); + case FOP_ABD: return FN(fv)(FN(fb)(FN(nan2)(c, a, b)) & ~SGN); + } + return FN(nan2)(c, a, b); +} + +// FMAX/FMIN/FMAXNM/FMINNM (operands already flushed) +SI T FN(minmax)(Cpu *c, int op, T a, T b) { + U x = FN(fb)(a), y = FN(fb)(b); + if (__builtin_expect(FN(isnan)(x) || FN(isnan)(y), 0)) { + if (op == FOP_MAXNM || op == FOP_MINNM) { + if (FN(isqnan)(x) && !FN(isnan)(y)) return b; + if (FN(isqnan)(y) && !FN(isnan)(x)) return a; + } + return FN(nan2)(c, a, b); + } + if (op == FOP_MAX || op == FOP_MAXNM) return a == b ? FN(fv)(x & y) : a > b ? a : b; + return a == b ? FN(fv)(x | y) : a < b ? a : b; +} +#define FMINMAX(op) ((op) >= FOP_MAX && (op) <= FOP_MINNM) + +// FPCR.FZ / RMode path: flush inputs, round in the FPCR mode, flush tiny results (tininess is detected +// before rounding on ARM: a result rounded up to the smallest normal is recomputed toward zero). +SN T FN(slow)(Cpu *c, int op, T a, T b, T d) { +#pragma STDC FENV_ACCESS ON + uint64_t fpcr = c->fpcr; + if (fpcr & FPCR_FZ) a = FN(flush)(a), b = FN(flush)(b), d = FN(flush)(d); + if (FMINMAX(op)) return FN(minmax)(c, op, a, b); + // [TOTK] The game runs with FPCR.FZ set, so every FP op takes this path. The host thread is always in + // round-to-nearest (only this code changes the mode, and it puts nearest back): with RMode nearest no switch is + // needed. fegetround / fesetround cost ~70% of the game's CPU time otherwise (mingw sets both x87 and SSE state). + int mode = simd_fe_mode(fpcr), changed = mode != FE_TONEAREST; + if (changed) fesetround(mode); + volatile T va = a, vb = b, vd = d; + volatile T vr = op == FOP_RSQRTS ? FN(rsqrts)(va, vb) : FCALC(op, va, vb, vd, K); + T r = vr; + if (FBAD(op, r)) vr = FN(fix)(c, op, a, b, d), r = vr; + if ((fpcr & FPCR_FZ) && (FN(fb)(r) & ~SGN) <= MINN && !FN(iszero)(FN(fb)(r))) { + if ((FN(fb)(r) & ~SGN) == MINN) { + fesetround(FE_TOWARDZERO); + changed = 1; + volatile T vz = FCALC(op, va, vb, vd, K); + if ((FN(fb)(vz) & ~SGN) < MINN) r = FN(fv)(FN(fb)(r) & SGN); + } else { + r = FN(fv)(FN(fb)(r) & SGN); + } + } + if (changed) fesetround(FE_TONEAREST); + return r; +} + +// Scalar FP op (constant op folds the selection). FZ alone is the host's own mode (rt_fpcr_write: MXCSR DAZ / FTZ): +// only another rounding mode takes the slow path. +SI T FN(op)(Cpu *c, int op, T a, T b, T d) { + if (__builtin_expect(c->fpcr & FPCR_RM, 0)) return FN(slow)(c, op, a, b, d); + if (FMINMAX(op)) return FN(minmax)(c, op, FN(fz)(c, a), FN(fz)(c, b)); + T r = FCALC(op, a, b, d, K); + if (__builtin_expect(FBAD(op, r), 0)) r = FN(fix)(c, op, FN(fz)(c, a), FN(fz)(c, b), FN(fz)(c, d)); + return r; +} + +SN VT FN(vslow)(Cpu *c, int op, VT a, VT b, VT d) { + VT r; + for (int i = 0; i < NL; i++) r[i] = FN(op)(c, op, a[i], b[i], d[i]); + return r; +} + +SI VTU FN(vsel)(VTS m, VTU x, VTU y) { return (x & (VTU)m) | (y & ~(VTU)m); } + +// Vector FP op: native lanes, any NaN (or FPCR.FZ/RMode) reruns all lanes through the scalar path. +SI VT FN(vfz)(Cpu *c, VT a); +SI VT FN(vop)(Cpu *c, int op, VT a, VT b, VT d) { + if (__builtin_expect(c->fpcr & FPCR_RM, 0)) return FN(vslow)(c, op, a, b, d); + VT r; + if (FMINMAX(op)) { + a = FN(vfz)(c, a), b = FN(vfz)(c, b); // the result is an operand's bits: flushed as ARM's FZ would + if (__builtin_expect(__builtin_reduce_or((a != a) | (b != b)) != 0, 0)) return FN(vslow)(c, op, a, b, d); + VTU x = (VTU)a, y = (VTU)b; + if (op == FOP_MAX || op == FOP_MAXNM) + r = (VT)FN(vsel)(a > b, x, FN(vsel)(a == b, x & y, y)); + else + r = (VT)FN(vsel)(a < b, x, FN(vsel)(a == b, x | y, y)); + return r; + } + r = FCALC(op, a, b, d, VK); + if (__builtin_expect(__builtin_reduce_or(FBAD(op, r)) != 0, 0)) return FN(vslow)(c, op, a, b, d); + return r; +} + +SI VT FN(vfz)(Cpu *c, VT a) { + if (__builtin_expect(c->fpcr & FPCR_FZ, 0)) { + VTU u = (VTU)a, mag = u & ~((VTU){} + SGN); + VTS den = (mag != 0) & ((u & EXPM) == 0); + a = (VT)FN(vsel)(den, u & SGN, u); + } + return a; +} + +// ---- compares +SI uint64_t FN(cmp)(Cpu *c, T a, T b) { // FCMP NZCV + a = FN(fz)(c, a), b = FN(fz)(c, b); + return a == b ? 0x60000000ull : a < b ? 0x80000000ull : a > b ? 0x20000000ull : 0x30000000ull; +} +// FCMEQ/FCMGE/FCMGT/FACGE/FACGT (op 0..4) +SI VTU FN(vcmp)(Cpu *c, int op, VT a, VT b) { + a = FN(vfz)(c, a), b = FN(vfz)(c, b); + if (op >= 3) a = __builtin_elementwise_abs(a), b = __builtin_elementwise_abs(b); + switch (op) { + case 0: return (VTU)(a == b); + case 1: case 3: return (VTU)(a >= b); + default: return (VTU)(a > b); + } +} + +// ---- rounding to integral (FRINT*): rm 0 N, 1 P, 2 M, 3 Z, 4 A; 5 X / 6 I = FPCR mode +// (rint = ties-to-even: the host stays in round-to-nearest outside the slow paths; no roundeven in MinGW's CRT) +SI T FN(rnd)(T x, int rm) { + switch (rm) { + case 0: return __builtin_elementwise_rint(x); + case 1: return __builtin_elementwise_ceil(x); + case 2: return __builtin_elementwise_floor(x); + case 3: return __builtin_elementwise_trunc(x); + default: return __builtin_elementwise_round(x); + } +} +SI T FN(frint)(Cpu *c, T x, int rm) { + x = FN(fz)(c, x); + if (rm >= 5) rm = (int)((c->fpcr >> 22) & 3); + T r = FN(rnd)(x, rm); + if (__builtin_expect(r != r, 0)) r = FN(nan1)(c, x); + return r; +} +SI VT FN(vfrint)(Cpu *c, VT x, int rm) { + x = FN(vfz)(c, x); + if (rm >= 5) rm = (int)((c->fpcr >> 22) & 3); + VT r; + switch (rm) { + case 0: r = __builtin_elementwise_rint(x); break; + case 1: r = __builtin_elementwise_ceil(x); break; + case 2: r = __builtin_elementwise_floor(x); break; + case 3: r = __builtin_elementwise_trunc(x); break; + default: r = __builtin_elementwise_round(x); break; + } + if (__builtin_expect(__builtin_reduce_or(r != r) != 0, 0)) + for (int i = 0; i < NL; i++) + if (r[i] != r[i]) r[i] = FN(nan1)(c, x[i]); + return r; +} + +// ---- FP -> integer (FCVT[NPMZA][SU], fixed point): saturating, NaN -> 0. ib = 32/64. +SI uint64_t FN(toint)(Cpu *c, T x, int rm, int fbits, int ib, int uns) { + x = FN(fz)(c, x); + if (fbits) x = x * (T)(1ull << (fbits - 1)) * K(2); + T r = FN(rnd)(x, rm); + if (r != r) return 0; + if (uns) { + if (r <= K(0)) return 0; + if (ib == 32) return r >= K(4294967296.0) ? 0xffffffffull : (uint32_t)r; + return r >= K(18446744073709551616.0) ? ~0ull : (uint64_t)r; + } + if (ib == 32) return r >= K(2147483648.0) ? 0x7fffffffull : r < K(-2147483648.0) ? 0x80000000ull : (uint32_t)(int32_t)r; + return r >= K(9223372036854775808.0) ? 0x7fffffffffffffffull : r < K(-9223372036854775808.0) ? 0x8000000000000000ull + : (uint64_t)(int64_t)r; +} + +// ---- integer -> FP (SCVTF/UCVTF, fixed point), rounded in the FPCR mode +SN T FN(fromint_slow)(Cpu *c, uint64_t v, int ib, int uns, int fbits) { +#pragma STDC FENV_ACCESS ON + int old = fegetround(); + fesetround(simd_fe_mode(c->fpcr)); + volatile uint64_t vv = v; + volatile T r; + if (uns) r = ib == 32 ? (T)(uint32_t)vv : (T)vv; + else r = ib == 32 ? (T)(int32_t)vv : (T)(int64_t)vv; + fesetround(old); + T x = r; + if (fbits) x = x / (T)(1ull << (fbits - 1)) / K(2); + return x; +} +SI T FN(fromint)(Cpu *c, uint64_t v, int ib, int uns, int fbits) { + if (__builtin_expect(c->fpcr & FPCR_RM, 0)) return FN(fromint_slow)(c, v, ib, uns, fbits); + T r; + if (uns) r = ib == 32 ? (T)(uint32_t)v : (T)v; + else r = ib == 32 ? (T)(int32_t)v : (T)(int64_t)v; + if (fbits) r = r / (T)(1ull << (fbits - 1)) / K(2); + return r; +} + +// ---- reciprocal estimates (bit-exact ARMv8.0 pseudocode) +SI T FN(recpe)(Cpu *c, T a) { + U x = FN(fb)(FN(fz)(c, a)), s = x & SGN; + if (FN(isnan)(x)) return FN(nan1)(c, a); + if (FN(isinf)(x)) return FN(fv)(s); + if (FN(iszero)(x)) return FN(fv)(s | EXPM); + U mag = x & ~SGN; + if (mag < ((U)1 << (MB - 2))) { // |x| < 2^(emin - 2): result overflows + int rm = (c->fpcr >> 22) & 3; + int inf = rm == 0 || (rm == 1 && !s) || (rm == 2 && s); + return FN(fv)(s | (inf ? EXPM : EXPM - 1)); + } + int64_t e = (int64_t)(mag >> MB); + U frac = mag & (((U)1 << MB) - 1); + if ((c->fpcr & FPCR_FZ) && e >= (int64_t)((1 << EB) - 3)) return FN(fv)(s); // result would be tiny + if (e == 0) { + if (!(frac >> (MB - 1))) e = -1, frac = (frac << 2) & (((U)1 << MB) - 1); + else frac = (frac << 1) & (((U)1 << MB) - 1); + } + int scaled = 256 | (int)(frac >> (MB - 8)); + int64_t re = (int64_t)((1 << EB) - 3) - e; + U est = (U)simd_recip_tab[scaled - 256]; // 8 fraction bits + U f = est << (MB - 8); + if (re == 0) f = (f >> 1) | ((U)1 << (MB - 1)); + else if (re == -1) f = (f >> 2) | ((U)1 << (MB - 2)), re = 0; + return FN(fv)(s | ((U)re << MB) | f); +} +SI T FN(rsqrte)(Cpu *c, T a) { + U x = FN(fb)(FN(fz)(c, a)), s = x & SGN; + if (FN(isnan)(x)) return FN(nan1)(c, a); + if (FN(iszero)(x)) return FN(fv)(s | EXPM); + if (s) return FN(fv)(DNAN); + if (FN(isinf)(x)) return K(0); + int64_t e = (int64_t)(x >> MB); + U frac = x & (((U)1 << MB) - 1); + if (e == 0) { + while (!(frac >> (MB - 1))) frac <<= 1, e--; + frac = (frac << 1) & (((U)1 << MB) - 1); + } + int scaled = (e & 1) ? 128 | (int)(frac >> (MB - 7)) : 256 | (int)(frac >> (MB - 8)); + int64_t re = ((int64_t)(3 * ((1 << (EB - 1)) - 1) - 1) - e) / 2; // (380 - e) / 2, (3068 - e) / 2 + U est = (U)simd_rsqrt_tab[scaled - 128]; + return FN(fv)(((U)re << MB) | (est << (MB - 8))); +} +SI T FN(recpx)(Cpu *c, T a) { + U x = FN(fb)(FN(fz)(c, a)), s = x & SGN; + if (FN(isnan)(x)) return FN(nan1)(c, a); + if (!(x & EXPM)) return FN(fv)(s | (EXPM - MINN)); + return FN(fv)(s | (~x & EXPM)); +} + +#undef FN +#undef SGN +#undef EXPM +#undef QB +#undef DNAN +#undef MINN +#undef VK +#undef K diff --git a/src/vk/present.c b/src/vk/present.c new file mode 100644 index 0000000..6ec5e97 --- /dev/null +++ b/src/vk/present.c @@ -0,0 +1,304 @@ +// Vulkan presentation (see present.h). The functions are loaded through SDL's vkGetInstanceProcAddr, so the exe does +// not link vulkan-1 and starts (without a picture) on a machine with no Vulkan driver. +// The present thread waits for a frame from the game, acquires a swapchain image, fits the newest game frame into it +// (the replayer's output image) and presents it. The game never waits on the display here; its pacing comes from the vsync events (front_win.c). +// Mailbox when the surface offers it (the newest frame wins, no tearing), FIFO otherwise. +#include "present.h" +#include "vk.h" + +#include +#include +#include +#include +#include +#include +#include + +#define LOG(...) (fprintf(stderr, "[totk-vk] " __VA_ARGS__), fputc('\n', stderr)) +#define FRAMES 2 // frames in flight +#define MAX_IMAGES 8 + +static SDL_Window *g_window; +static VkSwapchainKHR swapchain; +static VkFormat format; +static VkExtent2D extent; +static VkImage images[MAX_IMAGES]; +static uint32_t nimages; +static VkSemaphore image_ready[FRAMES], render_done[MAX_IMAGES]; +static VkFence in_flight[FRAMES]; +static VkCommandPool pool; +static VkCommandBuffer cmd[FRAMES]; +static VkPresentModeKHR mode; + +static pthread_mutex_t mu = PTHREAD_MUTEX_INITIALIZER; +static pthread_cond_t cv = PTHREAD_COND_INITIALIZER; +static uint64_t frames_in; // under mu +static _Atomic uint64_t presented; +static _Atomic int resized; + +// Command buffers and sync objects of the present thread (the device is vk.c's). +static int create_present_objects(void) { + VkCommandPoolCreateInfo pi = {VK_STRUCTURE_TYPE_COMMAND_POOL_CREATE_INFO, + .flags = VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT, .queueFamilyIndex = vkc.family}; + if (!vk_check(vk.vkCreateCommandPool(vkc.device, &pi, NULL, &pool), "vkCreateCommandPool")) return 0; + VkCommandBufferAllocateInfo ai = {VK_STRUCTURE_TYPE_COMMAND_BUFFER_ALLOCATE_INFO, .commandPool = pool, + .level = VK_COMMAND_BUFFER_LEVEL_PRIMARY, .commandBufferCount = FRAMES}; + if (!vk_check(vk.vkAllocateCommandBuffers(vkc.device, &ai, cmd), "vkAllocateCommandBuffers")) return 0; + VkSemaphoreCreateInfo si = {VK_STRUCTURE_TYPE_SEMAPHORE_CREATE_INFO}; + VkFenceCreateInfo fi = {VK_STRUCTURE_TYPE_FENCE_CREATE_INFO, .flags = VK_FENCE_CREATE_SIGNALED_BIT}; + for (int i = 0; i < FRAMES; i++) + if (!vk_check(vk.vkCreateSemaphore(vkc.device, &si, NULL, &image_ready[i]), "vkCreateSemaphore") || + !vk_check(vk.vkCreateFence(vkc.device, &fi, NULL, &in_flight[i]), "vkCreateFence")) return 0; + return 1; +} + +static void destroy_swapchain(void) { + for (uint32_t i = 0; i < nimages; i++) vk.vkDestroySemaphore(vkc.device, render_done[i], NULL); + if (swapchain) vk.vkDestroySwapchainKHR(vkc.device, swapchain, NULL); + swapchain = VK_NULL_HANDLE; + nimages = 0; +} + +// 0: the window has no area now (minimised) — try again later +static int create_swapchain(void) { + VkSurfaceCapabilitiesKHR caps; + if (!vk_check(vk.vkGetPhysicalDeviceSurfaceCapabilitiesKHR(vkc.gpu, vkc.surface, &caps), "surface capabilities")) return 0; + extent = caps.currentExtent; + if (extent.width == UINT32_MAX) { + int w, h; + SDL_GetWindowSizeInPixels(g_window, &w, &h); + extent = (VkExtent2D){(uint32_t)w, (uint32_t)h}; + } + if (!extent.width || !extent.height) return 0; + uint32_t nf = 0; + vk.vkGetPhysicalDeviceSurfaceFormatsKHR(vkc.gpu, vkc.surface, &nf, NULL); + VkSurfaceFormatKHR *fm = calloc(nf ? nf : 1, sizeof *fm); + vk.vkGetPhysicalDeviceSurfaceFormatsKHR(vkc.gpu, vkc.surface, &nf, fm); + VkSurfaceFormatKHR pick = fm[0]; + for (uint32_t i = 0; i < nf; i++) + if ((fm[i].format == VK_FORMAT_B8G8R8A8_UNORM || fm[i].format == VK_FORMAT_R8G8B8A8_UNORM) && + fm[i].colorSpace == VK_COLOR_SPACE_SRGB_NONLINEAR_KHR) { pick = fm[i]; break; } + free(fm); + format = pick.format; + uint32_t nm = 0; + vk.vkGetPhysicalDeviceSurfacePresentModesKHR(vkc.gpu, vkc.surface, &nm, NULL); + VkPresentModeKHR modes[16]; + if (nm > 16) nm = 16; + vk.vkGetPhysicalDeviceSurfacePresentModesKHR(vkc.gpu, vkc.surface, &nm, modes); + mode = VK_PRESENT_MODE_FIFO_KHR; + for (uint32_t i = 0; i < nm; i++) if (modes[i] == VK_PRESENT_MODE_MAILBOX_KHR) mode = VK_PRESENT_MODE_MAILBOX_KHR; + uint32_t count = caps.minImageCount + 1; + if (caps.maxImageCount && count > caps.maxImageCount) count = caps.maxImageCount; + VkSwapchainCreateInfoKHR ci = {VK_STRUCTURE_TYPE_SWAPCHAIN_CREATE_INFO_KHR, .surface = vkc.surface, + .minImageCount = count, .imageFormat = format, .imageColorSpace = pick.colorSpace, .imageExtent = extent, + .imageArrayLayers = 1, .imageUsage = VK_IMAGE_USAGE_TRANSFER_DST_BIT | VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT, + .imageSharingMode = VK_SHARING_MODE_EXCLUSIVE, .preTransform = caps.currentTransform, + .compositeAlpha = VK_COMPOSITE_ALPHA_OPAQUE_BIT_KHR, .presentMode = mode, .clipped = VK_TRUE}; + if (!vk_check(vk.vkCreateSwapchainKHR(vkc.device, &ci, NULL, &swapchain), "vkCreateSwapchainKHR")) return 0; + nimages = MAX_IMAGES; + vk.vkGetSwapchainImagesKHR(vkc.device, swapchain, &nimages, images); + VkSemaphoreCreateInfo si = {VK_STRUCTURE_TYPE_SEMAPHORE_CREATE_INFO}; + for (uint32_t i = 0; i < nimages; i++) vk.vkCreateSemaphore(vkc.device, &si, NULL, &render_done[i]); + LOG("swapchain %ux%u, %u images, %s", extent.width, extent.height, nimages, + mode == VK_PRESENT_MODE_MAILBOX_KHR ? "mailbox" : "fifo"); + return 1; +} + +static void barrier(VkCommandBuffer cb, VkImage img, VkImageLayout from, VkImageLayout to, VkAccessFlags src, + VkAccessFlags dst, VkPipelineStageFlags s0, VkPipelineStageFlags s1) { + VkImageMemoryBarrier b = {VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER, .srcAccessMask = src, .dstAccessMask = dst, + .oldLayout = from, .newLayout = to, .srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, + .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, .image = img, + .subresourceRange = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 1, 0, 1}}; + vk.vkCmdPipelineBarrier(cb, s0, s1, 0, 0, NULL, 0, NULL, 1, &b); +} + +// Until the first game frame (and for the null device): a colour that cycles with the frame number. +static VkClearColorValue frame_colour(uint64_t n) { + float t = (float)(n % 240) / 240.0f * 6.2831853f; + return (VkClearColorValue){{0.15f + 0.1f * sinf(t), 0.15f + 0.1f * sinf(t + 2.094f), 0.2f + 0.1f * sinf(t + 4.188f), 1}}; +} + +// ---- output images: written by the replay thread, shown by the present thread +#define OUTS 3 +typedef struct { + VkImage img; VkDeviceMemory mem; uint32_t w, h; + int state; // 0 free, 1 written (pending), 2 being shown + uint64_t seq; // when it was written (newest pending wins) + VkSemaphore timeline; uint64_t value; int crop[4]; +} out_t; +static out_t g_out[OUTS]; +static uint64_t g_out_seq; // under mu + +static uint32_t memory_type(uint32_t bits, VkMemoryPropertyFlags want) { + for (uint32_t i = 0; i < vkc.memory.memoryTypeCount; i++) + if ((bits & (1u << i)) && (vkc.memory.memoryTypes[i].propertyFlags & want) == want) return i; + return UINT32_MAX; +} + +int vkp_active(void) { return swapchain != VK_NULL_HANDLE || g_window != NULL; } + +int vkp_out_acquire(uint32_t w, uint32_t h, VkImage *image) { + pthread_mutex_lock(&mu); + int pick = -1; + for (int i = 0; i < OUTS && pick < 0; i++) if (g_out[i].state == 0) pick = i; + if (pick < 0) // none free: overwrite the oldest pending one (it was never shown; newer frames win) + for (int i = 0; i < OUTS; i++) if (g_out[i].state == 1 && (pick < 0 || g_out[i].seq < g_out[pick].seq)) pick = i; + if (pick >= 0) g_out[pick].state = 3; // being written + pthread_mutex_unlock(&mu); + if (pick < 0) return -1; + out_t *o = &g_out[pick]; + if (o->img && (o->w != w || o->h != h)) { // free: no present work uses it + vk.vkDestroyImage(vkc.device, o->img, NULL); + vk.vkFreeMemory(vkc.device, o->mem, NULL); + o->img = VK_NULL_HANDLE; + } + if (!o->img) { + VkImageCreateInfo ci = {VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO, .imageType = VK_IMAGE_TYPE_2D, .format = VK_FORMAT_B8G8R8A8_UNORM, + .extent = {w, h, 1}, .mipLevels = 1, .arrayLayers = 1, .samples = VK_SAMPLE_COUNT_1_BIT, .tiling = VK_IMAGE_TILING_OPTIMAL, + .usage = VK_IMAGE_USAGE_TRANSFER_SRC_BIT | VK_IMAGE_USAGE_TRANSFER_DST_BIT, .initialLayout = VK_IMAGE_LAYOUT_UNDEFINED}; + VkMemoryRequirements req; + if (!vk_check(vk.vkCreateImage(vkc.device, &ci, NULL, &o->img), "output image")) goto fail; + vk.vkGetImageMemoryRequirements(vkc.device, o->img, &req); + VkMemoryAllocateInfo ai = {VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO, .allocationSize = req.size, + .memoryTypeIndex = memory_type(req.memoryTypeBits, VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT)}; + if (!vk_check(vk.vkAllocateMemory(vkc.device, &ai, NULL, &o->mem), "output memory")) { + vk.vkDestroyImage(vkc.device, o->img, NULL); + o->img = VK_NULL_HANDLE; + goto fail; + } + vk.vkBindImageMemory(vkc.device, o->img, o->mem, 0); + o->w = w; o->h = h; + } + *image = o->img; + return pick; +fail: + pthread_mutex_lock(&mu); + o->state = 0; + pthread_mutex_unlock(&mu); + return -1; +} + +void vkp_out_ready(int slot, VkSemaphore timeline, uint64_t value, const int crop[4]) { + pthread_mutex_lock(&mu); + out_t *o = &g_out[slot]; + o->timeline = timeline; o->value = value; o->seq = ++g_out_seq; o->state = 1; + if (crop) memcpy(o->crop, crop, sizeof o->crop); else memset(o->crop, 0, sizeof o->crop); + for (int i = 0; i < OUTS; i++) // older pending frames are never shown + if (i != slot && g_out[i].state == 1 && g_out[i].seq < o->seq) g_out[i].state = 0; + frames_in++; + pthread_cond_signal(&cv); + pthread_mutex_unlock(&mu); +} + +// One window frame: the newest game frame (out, may be NULL: the cycling colour) fitted with its aspect ratio. +static int present_one(uint64_t n, unsigned slot, out_t *out) { + if (atomic_exchange(&resized, 0) || !swapchain) { + vk.vkWaitForFences(vkc.device, FRAMES, in_flight, VK_TRUE, UINT64_MAX); // the present thread's own work only + destroy_swapchain(); + if (!create_swapchain()) return 0; + } + vk.vkWaitForFences(vkc.device, 1, &in_flight[slot], VK_TRUE, UINT64_MAX); + uint32_t idx; + VkResult r = vk.vkAcquireNextImageKHR(vkc.device, swapchain, UINT64_MAX, image_ready[slot], VK_NULL_HANDLE, &idx); + if (r == VK_ERROR_OUT_OF_DATE_KHR) { atomic_store(&resized, 1); return 0; } + if (r != VK_SUCCESS && r != VK_SUBOPTIMAL_KHR) { vk_check(r, "vkAcquireNextImageKHR"); return 0; } + vk.vkResetFences(vkc.device, 1, &in_flight[slot]); + VkCommandBuffer cb = cmd[slot]; + vk.vkResetCommandBuffer(cb, 0); + VkCommandBufferBeginInfo bi = {VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO, + .flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT}; + vk.vkBeginCommandBuffer(cb, &bi); + barrier(cb, images[idx], VK_IMAGE_LAYOUT_UNDEFINED, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, 0, + VK_ACCESS_TRANSFER_WRITE_BIT, VK_PIPELINE_STAGE_TRANSFER_BIT, VK_PIPELINE_STAGE_TRANSFER_BIT); + VkClearColorValue c = out ? (VkClearColorValue){{0, 0, 0, 1}} : frame_colour(n); // bars around the game picture + VkImageSubresourceRange range = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 1, 0, 1}; + vk.vkCmdClearColorImage(cb, images[idx], VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, &c, 1, &range); + if (out) { + // source rectangle: the game's crop (NVN WindowCrop; 0 size = the whole texture) + int sx = out->crop[0], sy = out->crop[1], sw = out->crop[2], sh = out->crop[3]; + if (sw <= 0 || sh <= 0 || sx + sw > (int)out->w || sy + sh > (int)out->h) { sx = sy = 0; sw = (int)out->w; sh = (int)out->h; } + double scale = (double)extent.width / sw < (double)extent.height / sh ? (double)extent.width / sw : (double)extent.height / sh; + int dw = (int)(sw * scale + 0.5), dh = (int)(sh * scale + 0.5), dx = ((int)extent.width - dw) / 2, dy = ((int)extent.height - dh) / 2; + barrier(cb, images[idx], VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, + VK_ACCESS_TRANSFER_WRITE_BIT, VK_ACCESS_TRANSFER_WRITE_BIT, VK_PIPELINE_STAGE_TRANSFER_BIT, VK_PIPELINE_STAGE_TRANSFER_BIT); + VkImageBlit b = {{VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1}, {{sx, sy, 0}, {sx + sw, sy + sh, 1}}, + {VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1}, {{dx, dy, 0}, {dx + dw, dy + dh, 1}}}; + vk.vkCmdBlitImage(cb, out->img, VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, images[idx], VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, + 1, &b, VK_FILTER_LINEAR); + } + barrier(cb, images[idx], VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, VK_IMAGE_LAYOUT_PRESENT_SRC_KHR, + VK_ACCESS_TRANSFER_WRITE_BIT, 0, VK_PIPELINE_STAGE_TRANSFER_BIT, VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT); + vk.vkEndCommandBuffer(cb); + // wait for the swapchain image and, with a game frame, for the replayer's copy into the output image + VkSemaphoreSubmitInfo waits[2] = { + {VK_STRUCTURE_TYPE_SEMAPHORE_SUBMIT_INFO, .semaphore = image_ready[slot], .stageMask = VK_PIPELINE_STAGE_2_TRANSFER_BIT}, + {VK_STRUCTURE_TYPE_SEMAPHORE_SUBMIT_INFO, .semaphore = out ? out->timeline : VK_NULL_HANDLE, .value = out ? out->value : 0, + .stageMask = VK_PIPELINE_STAGE_2_TRANSFER_BIT}}; + VkSemaphoreSubmitInfo sig = {VK_STRUCTURE_TYPE_SEMAPHORE_SUBMIT_INFO, .semaphore = render_done[idx], + .stageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT}; + VkCommandBufferSubmitInfo cbi = {VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = cb}; + VkSubmitInfo2 si = {VK_STRUCTURE_TYPE_SUBMIT_INFO_2, .waitSemaphoreInfoCount = out ? 2u : 1u, .pWaitSemaphoreInfos = waits, + .commandBufferInfoCount = 1, .pCommandBufferInfos = &cbi, .signalSemaphoreInfoCount = 1, + .pSignalSemaphoreInfos = &sig}; + VkPresentInfoKHR pi = {VK_STRUCTURE_TYPE_PRESENT_INFO_KHR, .waitSemaphoreCount = 1, + .pWaitSemaphores = &render_done[idx], .swapchainCount = 1, .pSwapchains = &swapchain, + .pImageIndices = &idx}; + vk_queue_lock(); // the replayer submits on the same queue + VkResult s = vk.vkQueueSubmit2(vkc.queue, 1, &si, in_flight[slot]); + r = s == VK_SUCCESS ? vk.vkQueuePresentKHR(vkc.queue, &pi) : s; + vk_queue_unlock(); + if (!vk_check(s, "vkQueueSubmit2")) return 0; + if (r == VK_ERROR_OUT_OF_DATE_KHR || r == VK_SUBOPTIMAL_KHR) atomic_store(&resized, 1); + else if (r != VK_SUCCESS) { vk_check(r, "vkQueuePresentKHR"); return 0; } + if (out) vk.vkWaitForFences(vkc.device, 1, &in_flight[slot], VK_TRUE, UINT64_MAX); // the output image is free again after this + atomic_fetch_add(&presented, 1); + return 1; +} + +static void *present_thread(void *unused) { + (void)unused; + uint64_t shown = 0; + unsigned slot = 0; + for (;;) { + pthread_mutex_lock(&mu); + while (frames_in == shown) pthread_cond_wait(&cv, &mu); + shown = frames_in; // frames that came while the last one was shown are skipped: only the newest counts + int pick = -1; + for (int i = 0; i < OUTS; i++) if (g_out[i].state == 1 && (pick < 0 || g_out[i].seq > g_out[pick].seq)) pick = i; + if (pick >= 0) g_out[pick].state = 2; + pthread_mutex_unlock(&mu); + if (present_one(shown, slot, pick >= 0 ? &g_out[pick] : NULL)) slot = (slot + 1) % FRAMES; + if (pick >= 0) { + pthread_mutex_lock(&mu); + g_out[pick].state = 0; + pthread_mutex_unlock(&mu); + } + } + return NULL; +} + +int vkp_init(SDL_Window *window) { + g_window = window; + if (!vk_init(window) || !vkc.surface || !create_present_objects()) return 0; + if (!create_swapchain()) LOG("window has no area yet; swapchain deferred"); + pthread_t t; + if (pthread_create(&t, NULL, present_thread, NULL)) { LOG("present thread"); return 0; } + return 1; +} + +void vkp_frame(void) { + pthread_mutex_lock(&mu); + frames_in++; + pthread_cond_signal(&cv); + pthread_mutex_unlock(&mu); +} + +void vkp_resized(void) { atomic_store(&resized, 1); } +uint64_t vkp_presented(void) { return atomic_load(&presented); } +uint64_t vkp_frames(void) { + pthread_mutex_lock(&mu); + uint64_t n = frames_in; + pthread_mutex_unlock(&mu); + return n; +} diff --git a/src/vk/present.h b/src/vk/present.h new file mode 100644 index 0000000..c3413ce --- /dev/null +++ b/src/vk/present.h @@ -0,0 +1,23 @@ +// Window presentation through Vulkan: the swapchain and a present thread. The replayer (src/gfx/backend_vk.c) copies +// each frame the game presents into one of three output images; the present thread shows the newest one, scaled to +// the window with the game's aspect ratio. Until the first game frame, a colour cycling per frame shows that frames +// arrive. +#pragma once +#include + +#include "vk.h" + +typedef struct SDL_Window SDL_Window; + +int vkp_init(SDL_Window *window); // 1 = ready; on failure logs why and returns 0 (the game runs without a picture) +void vkp_frame(void); // the game presented a frame without a picture (null device): count it, cycle colour +void vkp_resized(void); // the window changed size: the swapchain is rebuilt before the next present +uint64_t vkp_presented(void); // frames shown so far +uint64_t vkp_frames(void); // frames the game presented so far +int vkp_active(void); // a window with a swapchain is there (else the replayer skips the copy) + +// Output images (replay thread): acquire one of width x height (B8G8R8A8_UNORM, created / resized here), record the +// copy of the game's frame into it (layout UNDEFINED -> TRANSFER_DST -> TRANSFER_SRC), then hand it over with the +// timeline value after which the copy is done. -1 = none free (the frame is dropped). +int vkp_out_acquire(uint32_t width, uint32_t height, VkImage *image); +void vkp_out_ready(int slot, VkSemaphore timeline, uint64_t value, const int crop[4]); diff --git a/src/vk/vk.c b/src/vk/vk.c new file mode 100644 index 0000000..7aff4dd --- /dev/null +++ b/src/vk/vk.c @@ -0,0 +1,238 @@ +// The shared Vulkan instance and device (see vk.h). +#include "vk.h" + +#include +#include +#include +#include +#include +#include +#include + +#define LOG(...) (fprintf(stderr, "[totk-vk] " __VA_ARGS__), fputc('\n', stderr)) + +VkFunctions vk; +VkContext vkc; +static pthread_mutex_t queue_mu = PTHREAD_MUTEX_INITIALIZER; +void vk_queue_lock(void) { pthread_mutex_lock(&queue_mu); } +void vk_queue_unlock(void) { pthread_mutex_unlock(&queue_mu); } + +int vk_check(VkResult r, const char *what) { + if (r == VK_SUCCESS) return 1; + LOG("%s failed (%d)", what, (int)r); + return 0; +} + +static int has_layer(const char *name) { + uint32_t n = 0; + vk.vkEnumerateInstanceLayerProperties(&n, NULL); + VkLayerProperties *p = calloc(n ? n : 1, sizeof *p); + vk.vkEnumerateInstanceLayerProperties(&n, p); + int found = 0; + for (uint32_t i = 0; i < n; i++) if (!strcmp(p[i].layerName, name)) found = 1; + free(p); + return found; +} + +static int create_instance(SDL_Window *window) { + PFN_vkGetInstanceProcAddr gipa = NULL; + if (window) gipa = (PFN_vkGetInstanceProcAddr)SDL_Vulkan_GetVkGetInstanceProcAddr(); + else { // no window (TOTK_HEADLESS): SDL's video subsystem is not up, so the loader is loaded here + HMODULE m = LoadLibraryA("vulkan-1.dll"); + if (m) gipa = (PFN_vkGetInstanceProcAddr)(void *)GetProcAddress(m, "vkGetInstanceProcAddr"); + } + if (!gipa) { LOG("no Vulkan loader (vulkan-1.dll)"); return 0; } +#define LOAD(f) vk.f = (PFN_##f)gipa(NULL, #f); + VK_FN_GLOBAL(LOAD) +#undef LOAD + Uint32 next = 0; + const char *const *exts = window ? SDL_Vulkan_GetInstanceExtensions(&next) : NULL; + // TOTK_VK_VALIDATION=1: the Khronos validation layer (Vulkan SDK), for development + const char *layer = "VK_LAYER_KHRONOS_validation"; + const char *v = getenv("TOTK_VK_VALIDATION"); + int validate = v && strcmp(v, "0") && has_layer(layer); + if (v && strcmp(v, "0") && !validate) LOG("TOTK_VK_VALIDATION: %s not installed", layer); + VkApplicationInfo app = {VK_STRUCTURE_TYPE_APPLICATION_INFO, .pApplicationName = "totk-pc-port", + .apiVersion = VK_API_VERSION_1_3}; + VkInstanceCreateInfo ci = {VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO, .pApplicationInfo = &app, + .enabledExtensionCount = next, .ppEnabledExtensionNames = exts, + .enabledLayerCount = validate ? 1u : 0u, .ppEnabledLayerNames = &layer}; + if (!vk_check(vk.vkCreateInstance(&ci, NULL, &vkc.instance), "vkCreateInstance")) return 0; +#define LOAD(f) vk.f = (PFN_##f)gipa(vkc.instance, #f); + VK_FN_INSTANCE(LOAD) +#undef LOAD + if (validate) LOG("validation layer on"); + if (window && !SDL_Vulkan_CreateSurface(window, vkc.instance, NULL, &vkc.surface)) { + LOG("window surface: %s", SDL_GetError()); + vkc.surface = VK_NULL_HANDLE; + } + return 1; +} + +typedef struct { VkExtensionProperties *e; uint32_t n; } ExtList; +static ExtList device_extensions(VkPhysicalDevice p) { + ExtList l = {0}; + vk.vkEnumerateDeviceExtensionProperties(p, NULL, &l.n, NULL); + l.e = calloc(l.n ? l.n : 1, sizeof *l.e); + vk.vkEnumerateDeviceExtensionProperties(p, NULL, &l.n, l.e); + return l; +} +static int ext_in(const ExtList *l, const char *name) { + for (uint32_t i = 0; i < l->n; i++) if (!strcmp(l->e[i].extensionName, name)) return 1; + return 0; +} + +// A Vulkan 1.3 GPU with a queue family for graphics and compute (and presentation, with a window); discrete first. +static int pick_device(void) { + uint32_t n = 0; + vk.vkEnumeratePhysicalDevices(vkc.instance, &n, NULL); + if (!n) { LOG("no Vulkan GPU"); return 0; } + VkPhysicalDevice *all = calloc(n, sizeof *all); + vk.vkEnumeratePhysicalDevices(vkc.instance, &n, all); + int best = -1; + for (uint32_t i = 0; i < n; i++) { + VkPhysicalDeviceProperties pr; + vk.vkGetPhysicalDeviceProperties(all[i], &pr); + ExtList ex = device_extensions(all[i]); + int ok = pr.apiVersion >= VK_API_VERSION_1_3 && ext_in(&ex, VK_EXT_EXTERNAL_MEMORY_HOST_EXTENSION_NAME) && + (!vkc.surface || ext_in(&ex, VK_KHR_SWAPCHAIN_EXTENSION_NAME)); + free(ex.e); + if (!ok) continue; + uint32_t nq = 0; + vk.vkGetPhysicalDeviceQueueFamilyProperties(all[i], &nq, NULL); + VkQueueFamilyProperties *q = calloc(nq, sizeof *q); + vk.vkGetPhysicalDeviceQueueFamilyProperties(all[i], &nq, q); + for (uint32_t k = 0; k < nq; k++) { + VkBool32 present = VK_TRUE; + if (vkc.surface) vk.vkGetPhysicalDeviceSurfaceSupportKHR(all[i], k, vkc.surface, &present); + if ((q[k].queueFlags & (VK_QUEUE_GRAPHICS_BIT | VK_QUEUE_COMPUTE_BIT)) != (VK_QUEUE_GRAPHICS_BIT | VK_QUEUE_COMPUTE_BIT) || !present) + continue; + int score = pr.deviceType == VK_PHYSICAL_DEVICE_TYPE_DISCRETE_GPU ? 2 : 1; + if (best < score) { best = score; vkc.gpu = all[i]; vkc.family = k; } + break; + } + free(q); + } + free(all); + if (best < 0) { LOG("no Vulkan 1.3 GPU with VK_EXT_external_memory_host%s", vkc.surface ? " that presents to the window" : ""); return 0; } + vk.vkGetPhysicalDeviceProperties(vkc.gpu, &vkc.props); + vk.vkGetPhysicalDeviceMemoryProperties(vkc.gpu, &vkc.memory); + vkc.db = (VkPhysicalDeviceDescriptorBufferPropertiesEXT){VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_DESCRIPTOR_BUFFER_PROPERTIES_EXT}; + VkPhysicalDeviceExternalMemoryHostPropertiesEXT host = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_EXTERNAL_MEMORY_HOST_PROPERTIES_EXT, &vkc.db}; + VkPhysicalDeviceProperties2 p2 = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_PROPERTIES_2, &host}; + vk.vkGetPhysicalDeviceProperties2(vkc.gpu, &p2); + vkc.host_pointer_alignment = host.minImportedHostPointerAlignment; + LOG("GPU: %s (Vulkan %u.%u.%u, driver %#x)", vkc.props.deviceName, VK_API_VERSION_MAJOR(vkc.props.apiVersion), + VK_API_VERSION_MINOR(vkc.props.apiVersion), VK_API_VERSION_PATCH(vkc.props.apiVersion), vkc.props.driverVersion); + return 1; +} + +// Feature structs: queried with everything the GPU supports and passed back unchanged to vkCreateDevice. +static VkPhysicalDeviceVulkan11Features f11 = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_VULKAN_1_1_FEATURES}; +static VkPhysicalDeviceVulkan12Features f12 = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_VULKAN_1_2_FEATURES}; +static VkPhysicalDeviceVulkan13Features f13 = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_VULKAN_1_3_FEATURES}; +static VkPhysicalDeviceExtendedDynamicState3FeaturesEXT feds3 = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_EXTENDED_DYNAMIC_STATE_3_FEATURES_EXT}; +static VkPhysicalDeviceVertexInputDynamicStateFeaturesEXT fvids = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_VERTEX_INPUT_DYNAMIC_STATE_FEATURES_EXT}; +static VkPhysicalDeviceDescriptorBufferFeaturesEXT fdb = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_DESCRIPTOR_BUFFER_FEATURES_EXT}; +static VkPhysicalDeviceGraphicsPipelineLibraryFeaturesEXT fgpl = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_GRAPHICS_PIPELINE_LIBRARY_FEATURES_EXT}; +static VkPhysicalDeviceDepthClipControlFeaturesEXT fdcc = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_DEPTH_CLIP_CONTROL_FEATURES_EXT}; +static VkPhysicalDeviceDepthClipEnableFeaturesEXT fdce = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_DEPTH_CLIP_ENABLE_FEATURES_EXT}; +static VkPhysicalDeviceCustomBorderColorFeaturesEXT fcbc = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_CUSTOM_BORDER_COLOR_FEATURES_EXT}; +static VkPhysicalDeviceProvokingVertexFeaturesEXT fpv = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_PROVOKING_VERTEX_FEATURES_EXT}; +static VkPhysicalDeviceConditionalRenderingFeaturesEXT fcr = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_CONDITIONAL_RENDERING_FEATURES_EXT}; +static VkPhysicalDeviceIndexTypeUint8FeaturesEXT fu8 = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_INDEX_TYPE_UINT8_FEATURES_EXT}; +static VkPhysicalDevicePrimitiveTopologyListRestartFeaturesEXT fplr = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_PRIMITIVE_TOPOLOGY_LIST_RESTART_FEATURES_EXT}; +static VkPhysicalDeviceShaderObjectFeaturesEXT fso = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SHADER_OBJECT_FEATURES_EXT}; +static VkPhysicalDeviceFragmentShaderInterlockFeaturesEXT ffsi = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_FRAGMENT_SHADER_INTERLOCK_FEATURES_EXT}; +static VkPhysicalDeviceAttachmentFeedbackLoopLayoutFeaturesEXT ffl = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_ATTACHMENT_FEEDBACK_LOOP_LAYOUT_FEATURES_EXT}; +static VkPhysicalDeviceAttachmentFeedbackLoopDynamicStateFeaturesEXT ffld = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_ATTACHMENT_FEEDBACK_LOOP_DYNAMIC_STATE_FEATURES_EXT}; +static int has_fl, has_fld; +static VkPhysicalDeviceDynamicRenderingLocalReadFeaturesKHR flr = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_DYNAMIC_RENDERING_LOCAL_READ_FEATURES_KHR}; +static int has_lr; + +static int create_device(void) { + ExtList ex = device_extensions(vkc.gpu); + const char *names[32]; + uint32_t nnames = 0; + VkPhysicalDeviceFeatures2 f2 = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_FEATURES_2, &f11}; + f11.pNext = &f12; + f12.pNext = &f13; + void **tail = &f13.pNext; + // extension name, its feature struct (NULL: none), the flag in vkc + struct { const char *name; void *features; int *flag; } want[] = { + {VK_KHR_SWAPCHAIN_EXTENSION_NAME, NULL, NULL}, + {VK_EXT_EXTERNAL_MEMORY_HOST_EXTENSION_NAME, NULL, NULL}, + {VK_KHR_PUSH_DESCRIPTOR_EXTENSION_NAME, NULL, NULL}, + {VK_EXT_DESCRIPTOR_BUFFER_EXTENSION_NAME, &fdb, &vkc.has_descriptor_buffer}, + {VK_KHR_PIPELINE_LIBRARY_EXTENSION_NAME, NULL, NULL}, + {VK_EXT_GRAPHICS_PIPELINE_LIBRARY_EXTENSION_NAME, &fgpl, &vkc.has_gpl}, + {VK_EXT_EXTENDED_DYNAMIC_STATE_3_EXTENSION_NAME, &feds3, &vkc.has_eds3}, + {VK_EXT_VERTEX_INPUT_DYNAMIC_STATE_EXTENSION_NAME, &fvids, &vkc.has_vertex_input_ds}, + {VK_EXT_DEPTH_CLIP_CONTROL_EXTENSION_NAME, &fdcc, &vkc.has_depth_clip_control}, + {VK_EXT_DEPTH_CLIP_ENABLE_EXTENSION_NAME, &fdce, NULL}, + {VK_EXT_CUSTOM_BORDER_COLOR_EXTENSION_NAME, &fcbc, NULL}, + {VK_EXT_PROVOKING_VERTEX_EXTENSION_NAME, &fpv, NULL}, + {VK_EXT_CONDITIONAL_RENDERING_EXTENSION_NAME, &fcr, NULL}, + {VK_EXT_INDEX_TYPE_UINT8_EXTENSION_NAME, &fu8, NULL}, + {VK_EXT_PRIMITIVE_TOPOLOGY_LIST_RESTART_EXTENSION_NAME, &fplr, NULL}, + {VK_EXT_SHADER_OBJECT_EXTENSION_NAME, &fso, &vkc.has_shader_object}, + {VK_EXT_FRAGMENT_SHADER_INTERLOCK_EXTENSION_NAME, &ffsi, &vkc.has_interlock}, + // TOTK samples textures that are attachments of the same pass (G-buffer decals, depth-reading effects) + {VK_EXT_ATTACHMENT_FEEDBACK_LOOP_LAYOUT_EXTENSION_NAME, &ffl, &has_fl}, + {VK_EXT_ATTACHMENT_FEEDBACK_LOOP_DYNAMIC_STATE_EXTENSION_NAME, &ffld, &has_fld}, + // ...and a draw that samples a target earlier draws of the pass wrote waits for them inside the pass + {VK_KHR_DYNAMIC_RENDERING_LOCAL_READ_EXTENSION_NAME, &flr, &has_lr}, + // development (TOTK_VK_CHECKPOINTS=1): the markers a lost device reached + {VK_NV_DEVICE_DIAGNOSTIC_CHECKPOINTS_EXTENSION_NAME, NULL, &vkc.has_checkpoints}, + }; + for (unsigned i = 0; i < sizeof want / sizeof *want; i++) { + if (!strcmp(want[i].name, VK_KHR_SWAPCHAIN_EXTENSION_NAME) && !vkc.surface) continue; + if (want[i].flag == &vkc.has_checkpoints && !getenv("TOTK_VK_CHECKPOINTS")) continue; + if (!ext_in(&ex, want[i].name)) { LOG("extension %s not available", want[i].name); continue; } + names[nnames++] = want[i].name; + if (want[i].flag) *want[i].flag = 1; + if (want[i].features) { *tail = want[i].features; tail = (void **)&((VkBaseOutStructure *)want[i].features)->pNext; } + } + free(ex.e); + vk.vkGetPhysicalDeviceFeatures2(vkc.gpu, &f2); + vkc.has_feedback_loop = has_fl && has_fld && ffl.attachmentFeedbackLoopLayout && ffld.attachmentFeedbackLoopDynamicState; + if (!vkc.has_feedback_loop) LOG("no attachment feedback loops: textures that are attachments of their pass read empty"); + vkc.has_local_read = has_lr && flr.dynamicRenderingLocalRead; + if (vkc.has_feedback_loop && !vkc.has_local_read) LOG("no barriers inside a pass: a draw sampling its own target may miss earlier draws' writes"); + // TOTK_VK_ROBUST=1 (development): bounds-checked buffer access, to tell a shader's out-of-range access from other + // causes of a lost device + if (!getenv("TOTK_VK_ROBUST")) f2.features.robustBufferAccess = VK_FALSE; + else LOG("robust buffer access %s (TOTK_VK_ROBUST)", f2.features.robustBufferAccess ? "on" : "not supported"); + vkc.has_descriptor_buffer = vkc.has_descriptor_buffer && fdb.descriptorBuffer; + if (vkc.has_descriptor_buffer) fdb.descriptorBufferCaptureReplay = VK_FALSE; // tools only, costs driver work + if (vkc.has_descriptor_buffer) + LOG("descriptor buffer: uniform %zu, storage %zu, combined image sampler %zu, storage image %zu bytes; offset alignment %llu; " + "ranges resource %llu / sampler %llu", vkc.db.uniformBufferDescriptorSize, vkc.db.storageBufferDescriptorSize, + vkc.db.combinedImageSamplerDescriptorSize, vkc.db.storageImageDescriptorSize, (unsigned long long)vkc.db.descriptorBufferOffsetAlignment, + (unsigned long long)vkc.db.maxResourceDescriptorBufferRange, (unsigned long long)vkc.db.maxSamplerDescriptorBufferRange); + fdb.descriptorBufferImageLayoutIgnored = fdb.descriptorBufferImageLayoutIgnored && vkc.has_descriptor_buffer; + float prio = 1.0f; + VkDeviceQueueCreateInfo qi = {VK_STRUCTURE_TYPE_DEVICE_QUEUE_CREATE_INFO, .queueFamilyIndex = vkc.family, + .queueCount = 1, .pQueuePriorities = &prio}; + VkDeviceCreateInfo ci = {VK_STRUCTURE_TYPE_DEVICE_CREATE_INFO, .pNext = &f2, .queueCreateInfoCount = 1, + .pQueueCreateInfos = &qi, .enabledExtensionCount = nnames, .ppEnabledExtensionNames = names}; + if (!vk_check(vk.vkCreateDevice(vkc.gpu, &ci, NULL, &vkc.device), "vkCreateDevice")) return 0; +#define LOAD(f) vk.f = (PFN_##f)vk.vkGetDeviceProcAddr(vkc.device, #f); + VK_FN_DEVICE(LOAD) +#undef LOAD + vk.vkGetDeviceQueue(vkc.device, vkc.family, 0, &vkc.queue); + LOG("device: %u extensions; descriptor buffer %d, pipeline library %d, dynamic state 3 %d, vertex input %d, " + "timeline semaphores %d, host pointer alignment %#llx", nnames, vkc.has_descriptor_buffer, vkc.has_gpl, + vkc.has_eds3, vkc.has_vertex_input_ds, f12.timelineSemaphore, (unsigned long long)vkc.host_pointer_alignment); + if (!f12.timelineSemaphore || !f13.dynamicRendering || !f13.synchronization2) { LOG("GPU lacks Vulkan 1.3 core features"); return 0; } + return 1; +} + +int vk_init(SDL_Window *window) { + static int state; // 0 not tried, 1 ready, -1 failed + static pthread_mutex_t mu = PTHREAD_MUTEX_INITIALIZER; + pthread_mutex_lock(&mu); + if (!state) state = create_instance(window) && pick_device() && create_device() ? 1 : -1; + pthread_mutex_unlock(&mu); + return state == 1; +} diff --git a/src/vk/vk.h b/src/vk/vk.h new file mode 100644 index 0000000..3b18829 --- /dev/null +++ b/src/vk/vk.h @@ -0,0 +1,76 @@ +// The process's one Vulkan device, shared by the NVN replayer (src/gfx/backend_vk.c) and the window presentation +// (present.c). Functions are loaded at run time through SDL's vkGetInstanceProcAddr (no vulkan-1 link), into the table +// below; a function is called as vk.Name(...). +// The device enables every feature the GPU supports except robustBufferAccess (bounds checks the game does not need, +// at a cost on every buffer access), and the extensions of the replayer's design (2026-10-21). +#pragma once +#define VK_NO_PROTOTYPES +#include +#include + +#define VK_FN_GLOBAL(X) X(vkCreateInstance) X(vkEnumerateInstanceLayerProperties) X(vkEnumerateInstanceExtensionProperties) +#define VK_FN_INSTANCE(X) X(vkDestroyInstance) X(vkEnumeratePhysicalDevices) X(vkGetPhysicalDeviceProperties) \ + X(vkGetPhysicalDeviceProperties2) X(vkGetPhysicalDeviceFeatures2) X(vkGetPhysicalDeviceMemoryProperties) \ + X(vkGetPhysicalDeviceQueueFamilyProperties) X(vkGetPhysicalDeviceFormatProperties) \ + X(vkGetPhysicalDeviceSurfaceSupportKHR) X(vkGetPhysicalDeviceSurfaceCapabilitiesKHR) \ + X(vkGetPhysicalDeviceSurfaceFormatsKHR) X(vkGetPhysicalDeviceSurfacePresentModesKHR) \ + X(vkEnumerateDeviceExtensionProperties) X(vkCreateDevice) X(vkGetDeviceProcAddr) X(vkDestroySurfaceKHR) +#define VK_FN_DEVICE(X) X(vkGetDeviceQueue) X(vkDeviceWaitIdle) X(vkQueueSubmit) X(vkQueueSubmit2) X(vkQueueWaitIdle) \ + X(vkCreateSwapchainKHR) X(vkDestroySwapchainKHR) X(vkGetSwapchainImagesKHR) X(vkAcquireNextImageKHR) X(vkQueuePresentKHR) \ + X(vkCreateCommandPool) X(vkResetCommandPool) X(vkAllocateCommandBuffers) X(vkBeginCommandBuffer) \ + X(vkEndCommandBuffer) X(vkResetCommandBuffer) X(vkCmdPipelineBarrier) X(vkCmdPipelineBarrier2) \ + X(vkCmdClearColorImage) X(vkCmdCopyBuffer) X(vkCmdUpdateBuffer) X(vkCmdBlitImage) X(vkCmdFillBuffer) \ + X(vkCreateSemaphore) X(vkDestroySemaphore) X(vkCreateFence) X(vkWaitForFences) X(vkResetFences) X(vkDestroyFence) \ + X(vkWaitSemaphores) X(vkGetSemaphoreCounterValue) X(vkMapMemory) X(vkUnmapMemory) \ + X(vkAllocateMemory) X(vkFreeMemory) X(vkCreateBuffer) X(vkDestroyBuffer) X(vkBindBufferMemory) \ + X(vkGetBufferMemoryRequirements) X(vkGetMemoryHostPointerPropertiesEXT) X(vkGetBufferDeviceAddress) \ + X(vkCreateImage) X(vkDestroyImage) X(vkGetImageMemoryRequirements) X(vkBindImageMemory) X(vkCreateImageView) \ + X(vkDestroyImageView) X(vkCmdBeginRendering) X(vkCmdEndRendering) X(vkCmdClearAttachments) X(vkCmdCopyImage) \ + X(vkCmdClearDepthStencilImage) X(vkCmdCopyBufferToImage) X(vkCmdCopyImageToBuffer) \ + X(vkCreateDescriptorSetLayout) X(vkCreatePipelineLayout) X(vkCreateDescriptorPool) X(vkResetDescriptorPool) \ + X(vkAllocateDescriptorSets) X(vkUpdateDescriptorSets) X(vkCmdBindDescriptorSets) X(vkCreateSampler) X(vkDestroySampler) \ + X(vkCreateShadersEXT) X(vkDestroyShaderEXT) X(vkCmdBindShadersEXT) X(vkCmdDraw) X(vkCmdDrawIndexed) X(vkCmdDispatch) \ + X(vkCmdBindVertexBuffers2) X(vkCmdBindIndexBuffer) X(vkCmdSetVertexInputEXT) X(vkCmdSetViewportWithCount) \ + X(vkCmdSetScissorWithCount) X(vkCmdSetPrimitiveTopology) X(vkCmdSetPrimitiveRestartEnable) X(vkCmdSetRasterizerDiscardEnable) \ + X(vkCmdSetCullMode) X(vkCmdSetFrontFace) X(vkCmdSetPolygonModeEXT) X(vkCmdSetDepthBiasEnable) X(vkCmdSetDepthBias) \ + X(vkCmdSetDepthTestEnable) X(vkCmdSetDepthWriteEnable) X(vkCmdSetDepthCompareOp) X(vkCmdSetDepthBoundsTestEnable) \ + X(vkCmdSetStencilTestEnable) X(vkCmdSetStencilOp) X(vkCmdSetStencilCompareMask) X(vkCmdSetStencilWriteMask) \ + X(vkCmdSetStencilReference) X(vkCmdSetRasterizationSamplesEXT) X(vkCmdSetSampleMaskEXT) X(vkCmdSetAlphaToCoverageEnableEXT) \ + X(vkCmdSetAlphaToOneEnableEXT) X(vkCmdSetColorBlendEnableEXT) X(vkCmdSetColorBlendEquationEXT) X(vkCmdSetColorWriteMaskEXT) \ + X(vkCmdSetLogicOpEnableEXT) X(vkCmdSetLogicOpEXT) X(vkCmdSetBlendConstants) X(vkCmdSetDepthClampEnableEXT) \ + X(vkCmdSetLineWidth) X(vkCmdSetDepthClipNegativeOneToOneEXT) X(vkCmdSetViewport) X(vkCmdSetScissor) \ + X(vkCmdSetProvokingVertexModeEXT) X(vkCmdSetDepthClipEnableEXT) X(vkCmdSetDepthBounds) \ + X(vkCreateQueryPool) X(vkResetQueryPool) X(vkCmdResetQueryPool) X(vkCmdWriteTimestamp2) X(vkGetQueryPoolResults) \ + X(vkCmdSetAttachmentFeedbackLoopEnableEXT) X(vkCmdSetCheckpointNV) X(vkGetQueueCheckpointDataNV) \ + X(vkGetDescriptorSetLayoutSizeEXT) X(vkGetDescriptorSetLayoutBindingOffsetEXT) X(vkGetDescriptorEXT) \ + X(vkCmdBindDescriptorBuffersEXT) X(vkCmdSetDescriptorBufferOffsetsEXT) +#define VK_FN_DECLARE(f) PFN_##f f; +typedef struct { VK_FN_GLOBAL(VK_FN_DECLARE) VK_FN_INSTANCE(VK_FN_DECLARE) VK_FN_DEVICE(VK_FN_DECLARE) } VkFunctions; +#undef VK_FN_DECLARE +extern VkFunctions vk; + +typedef struct { + VkInstance instance; + VkSurfaceKHR surface; // the window's, VK_NULL_HANDLE without one + VkPhysicalDevice gpu; + VkDevice device; + VkQueue queue; // graphics + compute + transfer (+ present when there is a window); vk_queue_lock() + uint32_t family; + VkPhysicalDeviceProperties props; + VkPhysicalDeviceMemoryProperties memory; + uint64_t host_pointer_alignment; // VK_EXT_external_memory_host + int has_descriptor_buffer, has_gpl, has_eds3, has_vertex_input_ds, has_depth_clip_control, has_shader_object, has_interlock; + int has_feedback_loop; // VK_EXT_attachment_feedback_loop_layout + _dynamic_state: sampling a bound attachment + int has_local_read; // VK_KHR_dynamic_rendering_local_read: barriers inside a pass (draws reading what earlier draws wrote) + VkPhysicalDeviceDescriptorBufferPropertiesEXT db; // VK_EXT_descriptor_buffer: descriptor sizes, alignment, ranges + int has_checkpoints; // TOTK_VK_CHECKPOINTS=1: VK_NV_device_diagnostic_checkpoints (where a lost device stopped) +} VkContext; +extern VkContext vkc; + +typedef struct SDL_Window SDL_Window; +// Creates the instance and device once (the first caller's window decides the instance extensions for presentation; +// NULL = no window, e.g. TOTK_HEADLESS). 1 = ready; 0 = no usable Vulkan 1.3 GPU (logged). +int vk_init(SDL_Window *window); +void vk_queue_lock(void); // the queue is used by the replayer's submit thread and the present thread +void vk_queue_unlock(void); +int vk_check(VkResult r, const char *what); // 1 on success, logs the failure otherwise diff --git a/third_party/README.md b/third_party/README.md new file mode 100644 index 0000000..f62d381 --- /dev/null +++ b/third_party/README.md @@ -0,0 +1,13 @@ +# Third-party code in this repository + +Every piece keeps its license; the license file sits next to the code. All are compatible with the port's +GPL-3.0-or-later. + +| Folder | What | From | License | +|---|---|---|---| +| `maxwell_shader/` | The Ryujinx 1.3.2 shader translator, unchanged apart from the patches below (Ryujinx Team and contributors) (`Ryujinx.Graphics.Shader`, `Ryujinx.Common`, `Spv.Generator`) | MacNativeGames `vendor/ryujinx` @b75be8c (Ryujinx 1.3.2, unmodified there), with the four patches of `shader_translator/patches/` applied here | MIT (`maxwell_shader/LICENSE.txt`) | +| `shader_translator/` | MacNativeGames' `totk_xlate`: the service around that translator that turns the game's NVN shaders into SPIR-V; built as `ultrs-shaders.exe` by `tools/build_xlate.py` | MacNativeGames `games/totk/tools/totk_xlate` @b75be8c, without its macOS build script | MIT (`shader_translator/LICENSE`) | + +Elsewhere in the tree: `src/gfx/tex/bc7enc/` (bc7enc and rgbcx, Rich Geldreich, MIT) and the files ported from +MacNativeGames (MIT), LASRecomp (MIT) and B2recomp (GPL-3.0), each named in its header comment; +`tools/shaders/inventory.py` (MacNativeGames, MIT). diff --git a/third_party/licenses/MacNativeGames.txt b/third_party/licenses/MacNativeGames.txt new file mode 100644 index 0000000..6102617 --- /dev/null +++ b/third_party/licenses/MacNativeGames.txt @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 MacNativeGames contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/third_party/maxwell_shader/.gitignore b/third_party/maxwell_shader/.gitignore new file mode 100644 index 0000000..59fc676 --- /dev/null +++ b/third_party/maxwell_shader/.gitignore @@ -0,0 +1,2 @@ +bin/ +obj/ diff --git a/third_party/maxwell_shader/Directory.Build.props b/third_party/maxwell_shader/Directory.Build.props new file mode 100644 index 0000000..de0e8e1 --- /dev/null +++ b/third_party/maxwell_shader/Directory.Build.props @@ -0,0 +1,7 @@ + + + net10.0 + preview + none + + diff --git a/third_party/maxwell_shader/Directory.Packages.props b/third_party/maxwell_shader/Directory.Packages.props new file mode 100644 index 0000000..34248e2 --- /dev/null +++ b/third_party/maxwell_shader/Directory.Packages.props @@ -0,0 +1,104 @@ + + + true + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/third_party/maxwell_shader/LICENSE.txt b/third_party/maxwell_shader/LICENSE.txt new file mode 100644 index 0000000..2c5385a --- /dev/null +++ b/third_party/maxwell_shader/LICENSE.txt @@ -0,0 +1,9 @@ +MIT License + +Copyright (c) Ryujinx Team and Contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/third_party/maxwell_shader/README.md b/third_party/maxwell_shader/README.md new file mode 100644 index 0000000..4e2d0d4 --- /dev/null +++ b/third_party/maxwell_shader/README.md @@ -0,0 +1,8 @@ +# Ryujinx shader translator (vendored subset) + +Three projects from Ryujinx 1.3.2, unmodified: `Ryujinx.Graphics.Shader`, `Ryujinx.Common` and `Spv.Generator`. +The shader translation tool in `dm/gfx/tools/dmsh_xlate` references them to translate the original Switch shaders to SPIR-V. +License: MIT, see `LICENSE.txt`. Upstream: . + +Ryujinx 1.3.2 的三个项目，未做修改：`Ryujinx.Graphics.Shader`、`Ryujinx.Common`、`Spv.Generator`。 +`dm/gfx/tools/dmsh_xlate` 的着色器翻译工具引用它们，把 Switch 原版着色器翻译成 SPIR-V。许可证 MIT，见 `LICENSE.txt`。 diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/AsyncWorkQueue.cs b/third_party/maxwell_shader/src/Ryujinx.Common/AsyncWorkQueue.cs new file mode 100644 index 0000000..a719764 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/AsyncWorkQueue.cs @@ -0,0 +1,102 @@ +using System; +using System.Collections.Concurrent; +using System.Threading; + +namespace Ryujinx.Common +{ + public sealed class AsyncWorkQueue : IDisposable + { + private readonly Thread _workerThread; + private readonly CancellationTokenSource _cts; + private readonly Action _workerAction; + private readonly BlockingCollection _queue; + + public bool IsCancellationRequested => _cts.IsCancellationRequested; + + public AsyncWorkQueue(Action callback, string name = null) : this(callback, name, new BlockingCollection()) + { + } + + public AsyncWorkQueue(Action callback, string name, BlockingCollection collection) + { + _cts = new CancellationTokenSource(); + _queue = collection; + _workerAction = callback; + _workerThread = new Thread(DoWork) + { + Name = name, + IsBackground = true, + }; + _workerThread.Start(); + } + + private void DoWork() + { + try + { + foreach (T item in _queue.GetConsumingEnumerable(_cts.Token)) + { + _workerAction(item); + } + } + catch (OperationCanceledException) + { + } + } + + public void Cancel() + { + _cts.Cancel(); + } + + public void CancelAfter(int millisecondsDelay) + { + _cts.CancelAfter(millisecondsDelay); + } + + public void CancelAfter(TimeSpan delay) + { + _cts.CancelAfter(delay); + } + + public void Add(T workItem) + { + _queue.Add(workItem); + } + + public void Add(T workItem, CancellationToken cancellationToken) + { + _queue.Add(workItem, cancellationToken); + } + + public bool TryAdd(T workItem) + { + return _queue.TryAdd(workItem); + } + + public bool TryAdd(T workItem, int millisecondsDelay) + { + return _queue.TryAdd(workItem, millisecondsDelay); + } + + public bool TryAdd(T workItem, int millisecondsDelay, CancellationToken cancellationToken) + { + return _queue.TryAdd(workItem, millisecondsDelay, cancellationToken); + } + + public bool TryAdd(T workItem, TimeSpan timeout) + { + return _queue.TryAdd(workItem, timeout); + } + + public void Dispose() + { + _queue.CompleteAdding(); + _cts.Cancel(); + _workerThread.Join(); + + _queue.Dispose(); + _cts.Dispose(); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Collections/BitMap.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Collections/BitMap.cs new file mode 100644 index 0000000..d64cddd --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Collections/BitMap.cs @@ -0,0 +1,226 @@ +namespace Ryujinx.Common.Collections +{ + /// + /// Represents a collection that can store 1 bit values. + /// + public struct BitMap + { + /// + /// Size in bits of the integer used internally for the groups of bits. + /// + public const int IntSize = 64; + + private const int IntShift = 6; + private const int IntMask = IntSize - 1; + + private readonly long[] _masks; + + /// + /// Gets or sets the value of a bit. + /// + /// Bit to access + /// Bit value + public bool this[int bit] + { + get => IsSet(bit); + set + { + if (value) + { + Set(bit); + } + else + { + Clear(bit); + } + } + } + + /// + /// Creates a new bitmap. + /// + /// Total number of bits + public BitMap(int count) + { + _masks = new long[(count + IntMask) / IntSize]; + } + + /// + /// Checks if any bit is set. + /// + /// True if any bit is set, false otherwise + public bool AnySet() + { + for (int i = 0; i < _masks.Length; i++) + { + if (_masks[i] != 0) + { + return true; + } + } + + return false; + } + + /// + /// Checks if a specific bit is set. + /// + /// Bit to be checked + /// True if set, false otherwise + public bool IsSet(int bit) + { + int wordIndex = bit >> IntShift; + int wordBit = bit & IntMask; + + long wordMask = 1L << wordBit; + + return (_masks[wordIndex] & wordMask) != 0; + } + + /// + /// Checks if any bit inside a given range of bits is set. + /// + /// Start bit of the range + /// End bit of the range (inclusive) + /// True if any bit is set, false otherwise + public bool IsSet(int start, int end) + { + if (start == end) + { + return IsSet(start); + } + + int startIndex = start >> IntShift; + int startBit = start & IntMask; + long startMask = -1L << startBit; + + int endIndex = end >> IntShift; + int endBit = end & IntMask; + long endMask = (long)(ulong.MaxValue >> (IntMask - endBit)); + + if (startIndex == endIndex) + { + return (_masks[startIndex] & startMask & endMask) != 0; + } + + if ((_masks[startIndex] & startMask) != 0) + { + return true; + } + + for (int i = startIndex + 1; i < endIndex; i++) + { + if (_masks[i] != 0) + { + return true; + } + } + + if ((_masks[endIndex] & endMask) != 0) + { + return true; + } + + return false; + } + + /// + /// Sets the value of a bit to 1. + /// + /// Bit to be set + /// True if the bit was 0 and then changed to 1, false if it was already 1 + public bool Set(int bit) + { + int wordIndex = bit >> IntShift; + int wordBit = bit & IntMask; + + long wordMask = 1L << wordBit; + + if ((_masks[wordIndex] & wordMask) != 0) + { + return false; + } + + _masks[wordIndex] |= wordMask; + + return true; + } + + /// + /// Sets a given range of bits to 1. + /// + /// Start bit of the range + /// End bit of the range (inclusive) + public void SetRange(int start, int end) + { + if (start == end) + { + Set(start); + return; + } + + int startIndex = start >> IntShift; + int startBit = start & IntMask; + long startMask = -1L << startBit; + + int endIndex = end >> IntShift; + int endBit = end & IntMask; + long endMask = (long)(ulong.MaxValue >> (IntMask - endBit)); + + if (startIndex == endIndex) + { + _masks[startIndex] |= startMask & endMask; + } + else + { + _masks[startIndex] |= startMask; + + for (int i = startIndex + 1; i < endIndex; i++) + { + _masks[i] |= -1; + } + + _masks[endIndex] |= endMask; + } + } + + /// + /// Sets a given bit to 0. + /// + /// Bit to be cleared + public void Clear(int bit) + { + int wordIndex = bit >> IntShift; + int wordBit = bit & IntMask; + + long wordMask = 1L << wordBit; + + _masks[wordIndex] &= ~wordMask; + } + + /// + /// Sets all bits to 0. + /// + public void Clear() + { + for (int i = 0; i < _masks.Length; i++) + { + _masks[i] = 0; + } + } + + /// + /// Sets one or more groups of bits to 0. + /// See for how many bits are inside each group. + /// + /// Start index of the group + /// End index of the group (inclusive) + public void ClearInt(int start, int end) + { + for (int i = start; i <= end; i++) + { + _masks[i] = 0; + } + } + } +} \ No newline at end of file diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Collections/IntervalTree.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Collections/IntervalTree.cs new file mode 100644 index 0000000..d966e23 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Collections/IntervalTree.cs @@ -0,0 +1,507 @@ +using System; +using System.Collections.Generic; +using System.Linq; + +namespace Ryujinx.Common.Collections +{ + /// + /// An Augmented Interval Tree based off of the "TreeDictionary"'s Red-Black Tree. Allows fast overlap checking of ranges. + /// + /// Key + /// Value + public class IntervalTree : IntrusiveRedBlackTreeImpl> where TKey : IComparable + { + private const int ArrayGrowthSize = 32; + + #region Public Methods + + /// + /// Gets the values of the interval whose key is . + /// + /// Key of the node value to get + /// Overlaps array to place results in + /// Number of values found + /// is null + public int Get(TKey key, ref TValue[] overlaps) + { + if (!typeof(TKey).IsValueType) + { + ArgumentNullException.ThrowIfNull(key); + } + + IntervalTreeNode node = GetNode(key); + + if (node == null) + { + return 0; + } + + if (node.Values.Count > overlaps.Length) + { + Array.Resize(ref overlaps, node.Values.Count); + } + + int overlapsCount = 0; + foreach (RangeNode value in node.Values) + { + overlaps[overlapsCount++] = value.Value; + } + + return overlapsCount; + } + + /// + /// Returns the values of the intervals whose start and end keys overlap the given range. + /// + /// Start of the range + /// End of the range + /// Overlaps array to place results in + /// Index to start writing results into the array. Defaults to 0 + /// Number of values found + /// or is null + public int Get(TKey start, TKey end, ref TValue[] overlaps, int overlapCount = 0) + { + ArgumentNullException.ThrowIfNull(start); + ArgumentNullException.ThrowIfNull(end); + + GetValues(Root, start, end, ref overlaps, ref overlapCount); + + return overlapCount; + } + + /// + /// Adds a new interval into the tree whose start is , end is and value is . + /// + /// Start of the range to add + /// End of the range to insert + /// Value to add + /// , or are null + public void Add(TKey start, TKey end, TValue value) + { + ArgumentNullException.ThrowIfNull(start); + ArgumentNullException.ThrowIfNull(end); + ArgumentNullException.ThrowIfNull(value); + + Insert(start, end, value); + } + + /// + /// Removes the given from the tree, searching for it with . + /// + /// Key of the node to remove + /// Value to remove + /// is null + /// Number of deleted values + public int Remove(TKey key, TValue value) + { + if (!typeof(TKey).IsValueType) + { + ArgumentNullException.ThrowIfNull(key); + } + + int removed = Delete(key, value); + + Count -= removed; + + return removed; + } + + /// + /// Adds all the nodes in the dictionary into . + /// + /// A list of all RangeNodes sorted by Key Order + public List> AsList() + { + List> list = []; + + AddToList(Root, list); + + return list; + } + + #endregion + + #region Private Methods (BST) + + /// + /// Adds all RangeNodes that are children of or contained within into , in Key Order. + /// + /// The node to search for RangeNodes within + /// The list to add RangeNodes to + private static void AddToList(IntervalTreeNode node, List> list) + { + if (node == null) + { + return; + } + + AddToList(node.Left, list); + + list.AddRange(node.Values); + + AddToList(node.Right, list); + } + + /// + /// Retrieve the node reference whose key is , or null if no such node exists. + /// + /// Key of the node to get + /// Node reference in the tree + /// is null + private IntervalTreeNode GetNode(TKey key) + { + if (!typeof(TKey).IsValueType) + { + ArgumentNullException.ThrowIfNull(key); + } + + IntervalTreeNode node = Root; + while (node != null) + { + int cmp = key.CompareTo(node.Start); + if (cmp < 0) + { + node = node.Left; + } + else if (cmp > 0) + { + node = node.Right; + } + else + { + return node; + } + } + + return null; + } + + /// + /// Retrieve all values that overlap the given start and end keys. + /// + /// Start of the range + /// End of the range + /// Overlaps array to place results in + /// Overlaps count to update + private static void GetValues(IntervalTreeNode node, TKey start, TKey end, ref TValue[] overlaps, ref int overlapCount) + { + if (node == null || start.CompareTo(node.Max) >= 0) + { + return; + } + + GetValues(node.Left, start, end, ref overlaps, ref overlapCount); + + bool endsOnRight = end.CompareTo(node.Start) > 0; + if (endsOnRight) + { + if (start.CompareTo(node.End) < 0) + { + // Contains this node. Add overlaps to list. + foreach (RangeNode overlap in node.Values) + { + if (start.CompareTo(overlap.End) < 0) + { + if (overlaps.Length <= overlapCount) + { + Array.Resize(ref overlaps, overlapCount + ArrayGrowthSize); + } + + overlaps[overlapCount++] = overlap.Value; + } + } + } + + GetValues(node.Right, start, end, ref overlaps, ref overlapCount); + } + } + + /// + /// Inserts a new node into the tree with a given , and . + /// + /// Start of the range to insert + /// End of the range to insert + /// Value to insert + private void Insert(TKey start, TKey end, TValue value) + { + IntervalTreeNode newNode = BSTInsert(start, end, value); + RestoreBalanceAfterInsertion(newNode); + } + + /// + /// Propagate an increase in max value starting at the given node, heading up the tree. + /// This should only be called if the max increases - not for rebalancing or removals. + /// + /// The node to start propagating from + private static void PropagateIncrease(IntervalTreeNode node) + { + TKey max = node.Max; + IntervalTreeNode ptr = node; + + while ((ptr = ptr.Parent) != null) + { + if (max.CompareTo(ptr.Max) > 0) + { + ptr.Max = max; + } + else + { + break; + } + } + } + + /// + /// Propagate recalculating max value starting at the given node, heading up the tree. + /// This fully recalculates the max value from all children when there is potential for it to decrease. + /// + /// The node to start propagating from + private static void PropagateFull(IntervalTreeNode node) + { + IntervalTreeNode ptr = node; + + do + { + TKey max = ptr.End; + + if (ptr.Left != null && ptr.Left.Max.CompareTo(max) > 0) + { + max = ptr.Left.Max; + } + + if (ptr.Right != null && ptr.Right.Max.CompareTo(max) > 0) + { + max = ptr.Right.Max; + } + + ptr.Max = max; + } while ((ptr = ptr.Parent) != null); + } + + /// + /// Insertion Mechanism for the interval tree. Similar to a BST insert, with the start of the range as the key. + /// Iterates the tree starting from the root and inserts a new node where all children in the left subtree are less than , and all children in the right subtree are greater than . + /// Each node can contain multiple values, and has an end address which is the maximum of all those values. + /// Post insertion, the "max" value of the node and all parents are updated. + /// + /// Start of the range to insert + /// End of the range to insert + /// Value to insert + /// The inserted Node + private IntervalTreeNode BSTInsert(TKey start, TKey end, TValue value) + { + IntervalTreeNode parent = null; + IntervalTreeNode node = Root; + + while (node != null) + { + parent = node; + int cmp = start.CompareTo(node.Start); + if (cmp < 0) + { + node = node.Left; + } + else if (cmp > 0) + { + node = node.Right; + } + else + { + node.Values.Add(new RangeNode(start, end, value)); + + if (end.CompareTo(node.End) > 0) + { + node.End = end; + if (end.CompareTo(node.Max) > 0) + { + node.Max = end; + PropagateIncrease(node); + } + } + + Count++; + return node; + } + } + + IntervalTreeNode newNode = new(start, end, value, parent); + if (newNode.Parent == null) + { + Root = newNode; + } + else if (start.CompareTo(parent.Start) < 0) + { + parent.Left = newNode; + } + else + { + parent.Right = newNode; + } + + PropagateIncrease(newNode); + Count++; + return newNode; + } + + /// + /// Removes instances of from the dictionary after searching for it with . + /// + /// Key to search for + /// Value to delete + /// Number of deleted values + private int Delete(TKey key, TValue value) + { + IntervalTreeNode nodeToDelete = GetNode(key); + + if (nodeToDelete == null) + { + return 0; + } + + int removed = nodeToDelete.Values.RemoveAll(node => node.Value.Equals(value)); + + if (nodeToDelete.Values.Count > 0) + { + if (removed > 0) + { + nodeToDelete.End = nodeToDelete.Values.Max(node => node.End); + + // Recalculate max from children and new end. + PropagateFull(nodeToDelete); + } + + return removed; + } + + IntervalTreeNode replacementNode; + + if (LeftOf(nodeToDelete) == null || RightOf(nodeToDelete) == null) + { + replacementNode = nodeToDelete; + } + else + { + replacementNode = PredecessorOf(nodeToDelete); + } + + IntervalTreeNode tmp = LeftOf(replacementNode) ?? RightOf(replacementNode); + + tmp?.Parent = ParentOf(replacementNode); + + if (ParentOf(replacementNode) == null) + { + Root = tmp; + } + else if (replacementNode == LeftOf(ParentOf(replacementNode))) + { + ParentOf(replacementNode).Left = tmp; + } + else + { + ParentOf(replacementNode).Right = tmp; + } + + if (replacementNode != nodeToDelete) + { + nodeToDelete.Start = replacementNode.Start; + nodeToDelete.Values = replacementNode.Values; + nodeToDelete.End = replacementNode.End; + nodeToDelete.Max = replacementNode.Max; + } + + PropagateFull(replacementNode); + + if (tmp != null && ColorOf(replacementNode) == Black) + { + RestoreBalanceAfterRemoval(tmp); + } + + return removed; + } + + #endregion + + protected override void RotateLeft(IntervalTreeNode node) + { + if (node != null) + { + base.RotateLeft(node); + + PropagateFull(node); + } + } + + protected override void RotateRight(IntervalTreeNode node) + { + if (node != null) + { + base.RotateRight(node); + + PropagateFull(node); + } + } + + public bool ContainsKey(TKey key) + { + ArgumentNullException.ThrowIfNull(key); + + return GetNode(key) != null; + } + } + + /// + /// Represents a value and its start and end keys. + /// + /// + /// + public readonly struct RangeNode + { + public readonly TKey Start; + public readonly TKey End; + public readonly TValue Value; + + public RangeNode(TKey start, TKey end, TValue value) + { + Start = start; + End = end; + Value = value; + } + } + + /// + /// Represents a node in the IntervalTree which contains start and end keys of type K, and a value of generic type V. + /// + /// Key type of the node + /// Value type of the node + public class IntervalTreeNode : IntrusiveRedBlackTreeNode> + { + /// + /// The start of the range. + /// + internal TKey Start; + + /// + /// The end of the range - maximum of all in the Values list. + /// + internal TKey End; + + /// + /// The maximum end value of this node and all its children. + /// + internal TKey Max; + + /// + /// Values contained on the node that shares a common Start value. + /// + internal List> Values; + + internal IntervalTreeNode(TKey start, TKey end, TValue value, IntervalTreeNode parent) + { + Start = start; + End = end; + Max = end; + Values = [new RangeNode(start, end, value)]; + Parent = parent; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Collections/IntrusiveRedBlackTree.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Collections/IntrusiveRedBlackTree.cs new file mode 100644 index 0000000..af6ddc7 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Collections/IntrusiveRedBlackTree.cs @@ -0,0 +1,353 @@ +using System; + +namespace Ryujinx.Common.Collections +{ + /// + /// Tree that provides the ability for O(logN) lookups for keys that exist in the tree, and O(logN) lookups for keys immediately greater than or less than a specified key. + /// + /// Derived node type + public class IntrusiveRedBlackTree : IntrusiveRedBlackTreeImpl where T : IntrusiveRedBlackTreeNode, IComparable + { + #region Public Methods + + /// + /// Adds a new node into the tree. + /// + /// Node to be added + /// Node to be added under + /// is null + public void Add(T node, T parent = null) + { + ArgumentNullException.ThrowIfNull(node); + + Insert(node, parent); + } + + /// + /// Removes a node from the tree. + /// + /// Note to be removed + /// is null + public void Remove(T node) + { + ArgumentNullException.ThrowIfNull(node); + + if (Delete(node) != null) + { + Count--; + } + } + + /// + /// Retrieve the node that is considered equal to the specified node by the comparator. + /// + /// Node to compare with + /// Node that is equal to + /// is null + public T GetNode(T searchNode) + { + ArgumentNullException.ThrowIfNull(searchNode); + + T node = Root; + while (node != null) + { + int cmp = searchNode.CompareTo(node); + if (cmp < 0) + { + node = node.Left; + } + else if (cmp > 0) + { + node = node.Right; + } + else + { + return node; + } + } + + return null; + } + + #endregion + + #region Private Methods (BST) + + /// + /// Inserts a new node into the tree. + /// + /// Node to be inserted + /// Node to be inserted under + private void Insert(T node, T parent = null) + { + T newNode = parent != null ? InsertWithParent(node, parent) : BSTInsert(node); + + RestoreBalanceAfterInsertion(newNode); + } + + /// + /// Insertion Mechanism for a Binary Search Tree (BST). + ///

+ /// Iterates the tree starting from the root and inserts a new node + /// where all children in the left subtree are less than , + /// and all children in the right subtree are greater than . + /// + /// Node to be inserted + /// The inserted Node + private T BSTInsert(T newNode) + { + T parent = null; + T node = Root; + + while (node != null) + { + parent = node; + int cmp = newNode.CompareTo(node); + if (cmp < 0) + { + node = node.Left; + } + else if (cmp > 0) + { + node = node.Right; + } + else + { + return node; + } + } + + newNode.Parent = parent; + if (parent == null) + { + Root = newNode; + } + else if (newNode.CompareTo(parent) < 0) + { + parent.Left = newNode; + + newNode.Successor = parent; + + if (parent.Predecessor != null) + { + newNode.Predecessor = parent.Predecessor; + newNode.Predecessor.Successor = newNode; + } + + parent.Predecessor = newNode; + } + else + { + parent.Right = newNode; + + newNode.Predecessor = parent; + + if (parent.Successor != null) + { + newNode.Successor = parent.Successor; + newNode.Successor.Predecessor = newNode; + } + + parent.Successor = newNode; + } + Count++; + return newNode; + } + + /// + /// Insertion Mechanism for a Binary Search Tree (BST). + ///

+ /// Inserts a new node directly under a parent node + /// where all children in the left subtree are less than , + /// and all children in the right subtree are greater than . + /// + /// Node to be inserted + /// Node to be inserted under + /// The inserted Node + private T InsertWithParent(T newNode, T parent) + { + newNode.Parent = parent; + + if (newNode.CompareTo(parent) < 0) + { + parent.Left = newNode; + + newNode.Successor = parent; + + if (parent.Predecessor != null) + { + newNode.Predecessor = parent.Predecessor; + parent.Predecessor = newNode; + newNode.Predecessor.Successor = newNode; + } + + parent.Predecessor = newNode; + } + else + { + parent.Right = newNode; + + newNode.Predecessor = parent; + + if (parent.Successor != null) + { + newNode.Successor = parent.Successor; + newNode.Successor.Predecessor = newNode; + } + + parent.Successor = newNode; + } + + Count++; + return newNode; + } + + /// + /// Removes from the tree, if it exists. + /// + /// Node to be removed + /// The deleted Node + private T Delete(T nodeToDelete) + { + if (nodeToDelete == null) + { + return null; + } + + T old = nodeToDelete; + T child; + T parent; + bool color; + + if (LeftOf(nodeToDelete) == null) + { + child = RightOf(nodeToDelete); + } + else if (RightOf(nodeToDelete) == null) + { + child = LeftOf(nodeToDelete); + } + else + { + T element = nodeToDelete.Successor; + + child = RightOf(element); + parent = ParentOf(element); + color = ColorOf(element); + + child?.Parent = parent; + + if (parent == null) + { + Root = child; + } + else if (element == LeftOf(parent)) + { + parent.Left = child; + } + else + { + parent.Right = child; + } + + element.Color = old.Color; + element.Left = old.Left; + element.Right = old.Right; + element.Parent = old.Parent; + element.Predecessor = old.Predecessor; + element.Predecessor?.Successor = element; + + if (ParentOf(old) == null) + { + Root = element; + } + else if (old == LeftOf(ParentOf(old))) + { + ParentOf(old).Left = element; + } + else + { + ParentOf(old).Right = element; + } + + LeftOf(old).Parent = element; + + if (RightOf(old) != null) + { + RightOf(old).Parent = element; + } + + if (child != null && color == Black) + { + RestoreBalanceAfterRemoval(child); + } + + return old; + } + + parent = ParentOf(nodeToDelete); + color = ColorOf(nodeToDelete); + + child?.Parent = parent; + + if (parent == null) + { + Root = child; + } + else if (nodeToDelete == LeftOf(parent)) + { + parent.Left = child; + } + else + { + parent.Right = child; + } + + if (child != null && color == Black) + { + RestoreBalanceAfterRemoval(child); + } + + old.Successor?.Predecessor = old.Predecessor; + old.Predecessor?.Successor = old.Successor; + + return old; + } + + #endregion + } + + public static class IntrusiveRedBlackTreeExtensions + { + /// + /// Retrieve the node that is considered equal to the key by the comparator. + /// + /// Tree to search at + /// Key of the node to be found + /// Node that is equal to + public static TNode GetNodeByKey(this IntrusiveRedBlackTree tree, TKey key) + where TNode : IntrusiveRedBlackTreeNode, IComparable, IComparable + where TKey : struct + { + TNode node = tree.RootNode; + while (node != null) + { + int cmp = node.CompareTo(key); + if (cmp < 0) + { + node = node.Right; + } + else if (cmp > 0) + { + node = node.Left; + } + else + { + return node; + } + } + + return null; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Collections/IntrusiveRedBlackTreeImpl.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Collections/IntrusiveRedBlackTreeImpl.cs new file mode 100644 index 0000000..5b47413 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Collections/IntrusiveRedBlackTreeImpl.cs @@ -0,0 +1,361 @@ +using System; + +namespace Ryujinx.Common.Collections +{ + /// + /// Tree that provides the ability for O(logN) lookups for keys that exist in the tree, and O(logN) lookups for keys immediately greater than or less than a specified key. + /// + /// Derived node type + public class IntrusiveRedBlackTreeImpl where T : IntrusiveRedBlackTreeNode + { + protected const bool Black = true; + protected const bool Red = false; + protected T Root; + + internal T RootNode => Root; + + /// + /// Number of nodes on the tree. + /// + public int Count { get; protected set; } + + /// + /// Removes all nodes on the tree. + /// + public void Clear() + { + Root = null; + Count = 0; + } + + /// + /// Finds the node whose key is immediately greater than . + /// + /// Node to find the successor of + /// Successor of + internal static T SuccessorOf(T node) + { + if (node.Right != null) + { + return Minimum(node.Right); + } + + T parent = node.Parent; + while (parent != null && node == parent.Right) + { + node = parent; + parent = parent.Parent; + } + + return parent; + } + + /// + /// Finds the node whose key is immediately less than . + /// + /// Node to find the predecessor of + /// Predecessor of + internal static T PredecessorOf(T node) + { + if (node.Left != null) + { + return Maximum(node.Left); + } + + T parent = node.Parent; + while (parent != null && node == parent.Left) + { + node = parent; + parent = parent.Parent; + } + + return parent; + } + + /// + /// Returns the node with the largest key where is considered the root node. + /// + /// Root node + /// Node with the maximum key in the tree of + protected static T Maximum(T node) + { + T tmp = node; + while (tmp.Right != null) + { + tmp = tmp.Right; + } + + return tmp; + } + + /// + /// Returns the node with the smallest key where is considered the root node. + /// + /// Root node + /// Node with the minimum key in the tree of + /// is null + protected static T Minimum(T node) + { + ArgumentNullException.ThrowIfNull(node); + + T tmp = node; + while (tmp.Left != null) + { + tmp = tmp.Left; + } + + return tmp; + } + + protected void RestoreBalanceAfterRemoval(T balanceNode) + { + T ptr = balanceNode; + + while (ptr != Root && ColorOf(ptr) == Black) + { + if (ptr == LeftOf(ParentOf(ptr))) + { + T sibling = RightOf(ParentOf(ptr)); + + if (ColorOf(sibling) == Red) + { + SetColor(sibling, Black); + SetColor(ParentOf(ptr), Red); + RotateLeft(ParentOf(ptr)); + sibling = RightOf(ParentOf(ptr)); + } + + if (ColorOf(LeftOf(sibling)) == Black && ColorOf(RightOf(sibling)) == Black) + { + SetColor(sibling, Red); + ptr = ParentOf(ptr); + } + else + { + if (ColorOf(RightOf(sibling)) == Black) + { + SetColor(LeftOf(sibling), Black); + SetColor(sibling, Red); + RotateRight(sibling); + sibling = RightOf(ParentOf(ptr)); + } + + SetColor(sibling, ColorOf(ParentOf(ptr))); + SetColor(ParentOf(ptr), Black); + SetColor(RightOf(sibling), Black); + RotateLeft(ParentOf(ptr)); + ptr = Root; + } + } + else + { + T sibling = LeftOf(ParentOf(ptr)); + + if (ColorOf(sibling) == Red) + { + SetColor(sibling, Black); + SetColor(ParentOf(ptr), Red); + RotateRight(ParentOf(ptr)); + sibling = LeftOf(ParentOf(ptr)); + } + + if (ColorOf(RightOf(sibling)) == Black && ColorOf(LeftOf(sibling)) == Black) + { + SetColor(sibling, Red); + ptr = ParentOf(ptr); + } + else + { + if (ColorOf(LeftOf(sibling)) == Black) + { + SetColor(RightOf(sibling), Black); + SetColor(sibling, Red); + RotateLeft(sibling); + sibling = LeftOf(ParentOf(ptr)); + } + + SetColor(sibling, ColorOf(ParentOf(ptr))); + SetColor(ParentOf(ptr), Black); + SetColor(LeftOf(sibling), Black); + RotateRight(ParentOf(ptr)); + ptr = Root; + } + } + } + + SetColor(ptr, Black); + } + + protected void RestoreBalanceAfterInsertion(T balanceNode) + { + SetColor(balanceNode, Red); + while (balanceNode != null && balanceNode != Root && ColorOf(ParentOf(balanceNode)) == Red) + { + if (ParentOf(balanceNode) == LeftOf(ParentOf(ParentOf(balanceNode)))) + { + T sibling = RightOf(ParentOf(ParentOf(balanceNode))); + + if (ColorOf(sibling) == Red) + { + SetColor(ParentOf(balanceNode), Black); + SetColor(sibling, Black); + SetColor(ParentOf(ParentOf(balanceNode)), Red); + balanceNode = ParentOf(ParentOf(balanceNode)); + } + else + { + if (balanceNode == RightOf(ParentOf(balanceNode))) + { + balanceNode = ParentOf(balanceNode); + RotateLeft(balanceNode); + } + + SetColor(ParentOf(balanceNode), Black); + SetColor(ParentOf(ParentOf(balanceNode)), Red); + RotateRight(ParentOf(ParentOf(balanceNode))); + } + } + else + { + T sibling = LeftOf(ParentOf(ParentOf(balanceNode))); + + if (ColorOf(sibling) == Red) + { + SetColor(ParentOf(balanceNode), Black); + SetColor(sibling, Black); + SetColor(ParentOf(ParentOf(balanceNode)), Red); + balanceNode = ParentOf(ParentOf(balanceNode)); + } + else + { + if (balanceNode == LeftOf(ParentOf(balanceNode))) + { + balanceNode = ParentOf(balanceNode); + RotateRight(balanceNode); + } + + SetColor(ParentOf(balanceNode), Black); + SetColor(ParentOf(ParentOf(balanceNode)), Red); + RotateLeft(ParentOf(ParentOf(balanceNode))); + } + } + } + + SetColor(Root, Black); + } + + protected virtual void RotateLeft(T node) + { + if (node != null) + { + T right = RightOf(node); + node.Right = LeftOf(right); + node.Right?.Parent = node; + + T nodeParent = ParentOf(node); + right.Parent = nodeParent; + if (nodeParent == null) + { + Root = right; + } + else if (node == LeftOf(nodeParent)) + { + nodeParent.Left = right; + } + else + { + nodeParent.Right = right; + } + + right.Left = node; + node.Parent = right; + } + } + + protected virtual void RotateRight(T node) + { + if (node != null) + { + T left = LeftOf(node); + node.Left = RightOf(left); + node.Left?.Parent = node; + + T nodeParent = ParentOf(node); + left.Parent = nodeParent; + if (nodeParent == null) + { + Root = left; + } + else if (node == RightOf(nodeParent)) + { + nodeParent.Right = left; + } + else + { + nodeParent.Left = left; + } + + left.Right = node; + node.Parent = left; + } + } + + #region Safety-Methods + + // These methods save memory by allowing us to forego sentinel nil nodes, as well as serve as protection against NullReferenceExceptions. + + /// + /// Returns the color of , or Black if it is null. + /// + /// Node + /// The boolean color of , or black if null + protected static bool ColorOf(T node) + { + return node == null || node.Color; + } + + /// + /// Sets the color of node to . + ///

+ /// This method does nothing if is null. + /// + /// Node to set the color of + /// Color (Boolean) + protected static void SetColor(T node, bool color) + { + node?.Color = color; + } + + /// + /// This method returns the left node of , or null if is null. + /// + /// Node to retrieve the left child from + /// Left child of + protected static T LeftOf(T node) + { + return node?.Left; + } + + /// + /// This method returns the right node of , or null if is null. + /// + /// Node to retrieve the right child from + /// Right child of + protected static T RightOf(T node) + { + return node?.Right; + } + + /// + /// Returns the parent node of , or null if is null. + /// + /// Node to retrieve the parent from + /// Parent of + protected static T ParentOf(T node) + { + return node?.Parent; + } + + #endregion + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Collections/IntrusiveRedBlackTreeNode.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Collections/IntrusiveRedBlackTreeNode.cs new file mode 100644 index 0000000..a2911dc --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Collections/IntrusiveRedBlackTreeNode.cs @@ -0,0 +1,15 @@ +namespace Ryujinx.Common.Collections +{ + /// + /// Represents a node in the Red-Black Tree. + /// + public class IntrusiveRedBlackTreeNode where T : IntrusiveRedBlackTreeNode + { + public bool Color = true; + public T Left; + public T Right; + public T Parent; + public T Predecessor; + public T Successor; + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Collections/TreeDictionary.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Collections/TreeDictionary.cs new file mode 100644 index 0000000..2c1666c --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Collections/TreeDictionary.cs @@ -0,0 +1,632 @@ +using System; +using System.Collections; +using System.Collections.Generic; +using System.Diagnostics.CodeAnalysis; + +namespace Ryujinx.Common.Collections +{ + /// + /// Dictionary that provides the ability for O(logN) Lookups for keys that exist in the Dictionary, and O(logN) lookups for keys immediately greater than or less than a specified key. + /// + /// Key + /// Value + public class TreeDictionary : IntrusiveRedBlackTreeImpl>, IDictionary where TKey : IComparable + { + #region Public Methods + + /// + /// Returns the value of the node whose key is , or the default value if no such node exists. + /// + /// Key of the node value to get + /// Value associated w/ + /// is null + public TValue Get(TKey key) + { + ArgumentNullException.ThrowIfNull(key); + + Node node = GetNode(key); + + if (node == null) + { + return default; + } + + return node.Value; + } + + /// + /// Adds a new node into the tree whose key is key and value is . + ///

+ /// Note: Adding the same key multiple times will cause the value for that key to be overwritten. + /// + /// Key of the node to add + /// Value of the node to add + /// or are null + public void Add(TKey key, TValue value) + { + ArgumentNullException.ThrowIfNull(key); + ArgumentNullException.ThrowIfNull(value); + + Insert(key, value); + } + + /// + /// Removes the node whose key is from the tree. + /// + /// Key of the node to remove + /// is null + public void Remove(TKey key) + { + ArgumentNullException.ThrowIfNull(key); + + if (Delete(key) != null) + { + Count--; + } + } + + /// + /// Returns the value whose key is equal to or immediately less than . + /// + /// Key for which to find the floor value of + /// Key of node immediately less than + /// is null + public TKey Floor(TKey key) + { + Node node = FloorNode(key); + if (node != null) + { + return node.Key; + } + + return default; + } + + /// + /// Returns the node whose key is equal to or immediately greater than . + /// + /// Key for which to find the ceiling node of + /// Key of node immediately greater than + /// is null + public TKey Ceiling(TKey key) + { + Node node = CeilingNode(key); + if (node != null) + { + return node.Key; + } + + return default; + } + + /// + /// Finds the value whose key is immediately greater than . + /// + /// Key to find the successor of + /// Value + public TKey SuccessorOf(TKey key) + { + Node node = GetNode(key); + if (node != null) + { + Node successor = node.Successor; + + return successor != null ? successor.Key : default; + } + + return default; + } + + /// + /// Finds the value whose key is immediately less than . + /// + /// Key to find the predecessor of + /// Value + public TKey PredecessorOf(TKey key) + { + Node node = GetNode(key); + if (node != null) + { + Node predecessor = node.Predecessor; + + return predecessor != null ? predecessor.Key : default; + } + + return default; + } + + /// + /// Adds all the nodes in the dictionary as key/value pairs into a list. + ///

+ /// The key/value pairs will be added in Level Order. + /// + public List> AsLevelOrderList() + { + List> list = []; + + Queue> nodes = new(); + + if (this.Root != null) + { + nodes.Enqueue(this.Root); + } + + while (nodes.TryDequeue(out Node node)) + { + list.Add(new KeyValuePair(node.Key, node.Value)); + if (node.Left != null) + { + nodes.Enqueue(node.Left); + } + + if (node.Right != null) + { + nodes.Enqueue(node.Right); + } + } + + return list; + } + + /// + /// Adds all the nodes in the dictionary into a list. + /// + /// A list of all KeyValuePairs sorted by Key Order + public List> AsList() + { + List> list = []; + + AddToList(Root, list); + + return list; + } + + #endregion + + #region Private Methods (BST) + + /// + /// Adds all nodes that are children of or contained within into , in Key Order. + /// + /// The node to search for nodes within + /// The list to add node to + private static void AddToList(Node node, List> list) + { + if (node == null) + { + return; + } + + AddToList(node.Left, list); + + list.Add(new KeyValuePair(node.Key, node.Value)); + + AddToList(node.Right, list); + } + + /// + /// Retrieve the node reference whose key is , or null if no such node exists. + /// + /// Key of the node to get + /// Node reference in the tree + /// is null + private Node GetNode(TKey key) + { + ArgumentNullException.ThrowIfNull(key); + + Node node = Root; + while (node != null) + { + int cmp = key.CompareTo(node.Key); + if (cmp < 0) + { + node = node.Left; + } + else if (cmp > 0) + { + node = node.Right; + } + else + { + return node; + } + } + + return null; + } + + /// + /// Inserts a new node into the tree whose key is and value is . + ///

+ /// Adding the same key multiple times will overwrite the previous value. + /// + /// Key of the node to insert + /// Value of the node to insert + private void Insert(TKey key, TValue value) + { + Node newNode = BSTInsert(key, value); + RestoreBalanceAfterInsertion(newNode); + } + + /// + /// Insertion Mechanism for a Binary Search Tree (BST). + ///

+ /// Iterates the tree starting from the root and inserts a new node where all children in the left subtree are less than , and all children in the right subtree are greater than . + ///

+ /// Note: If a node whose key is already exists, it's value will be overwritten. + /// + /// Key of the node to insert + /// Value of the node to insert + /// The inserted Node + private Node BSTInsert(TKey key, TValue value) + { + Node parent = null; + Node node = Root; + + while (node != null) + { + parent = node; + int cmp = key.CompareTo(node.Key); + if (cmp < 0) + { + node = node.Left; + } + else if (cmp > 0) + { + node = node.Right; + } + else + { + node.Value = value; + return node; + } + } + + Node newNode = new(key, value, parent); + if (parent == null) + { + Root = newNode; + } + else if (key.CompareTo(parent.Key) < 0) + { + parent.Left = newNode; + } + else + { + parent.Right = newNode; + } + + Count++; + return newNode; + } + + /// + /// Removes from the dictionary, if it exists. + /// + /// Key of the node to delete + /// The deleted Node + private Node Delete(TKey key) + { + // O(1) Retrieval + Node nodeToDelete = GetNode(key); + + if (nodeToDelete == null) + { + return null; + } + + Node replacementNode; + + if (LeftOf(nodeToDelete) == null || RightOf(nodeToDelete) == null) + { + replacementNode = nodeToDelete; + } + else + { + replacementNode = PredecessorOf(nodeToDelete); + } + + Node tmp = LeftOf(replacementNode) ?? RightOf(replacementNode); + + tmp?.Parent = ParentOf(replacementNode); + + if (ParentOf(replacementNode) == null) + { + Root = tmp; + } + else if (replacementNode == LeftOf(ParentOf(replacementNode))) + { + ParentOf(replacementNode).Left = tmp; + } + else + { + ParentOf(replacementNode).Right = tmp; + } + + if (replacementNode != nodeToDelete) + { + nodeToDelete.Key = replacementNode.Key; + nodeToDelete.Value = replacementNode.Value; + } + + if (tmp != null && ColorOf(replacementNode) == Black) + { + RestoreBalanceAfterRemoval(tmp); + } + + return replacementNode; + } + + /// + /// Returns the node whose key immediately less than or equal to . + /// + /// Key for which to find the floor node of + /// Node whose key is immediately less than or equal to , or null if no such node is found. + /// is null + private Node FloorNode(TKey key) + { + ArgumentNullException.ThrowIfNull(key); + + Node tmp = Root; + + while (tmp != null) + { + int cmp = key.CompareTo(tmp.Key); + if (cmp > 0) + { + if (tmp.Right != null) + { + tmp = tmp.Right; + } + else + { + return tmp; + } + } + else if (cmp < 0) + { + if (tmp.Left != null) + { + tmp = tmp.Left; + } + else + { + Node parent = tmp.Parent; + Node ptr = tmp; + while (parent != null && ptr == parent.Left) + { + ptr = parent; + parent = parent.Parent; + } + + return parent; + } + } + else + { + return tmp; + } + } + + return null; + } + + /// + /// Returns the node whose key is immediately greater than or equal to than . + /// + /// Key for which to find the ceiling node of + /// Node whose key is immediately greater than or equal to , or null if no such node is found. + /// is null + private Node CeilingNode(TKey key) + { + ArgumentNullException.ThrowIfNull(key); + + Node tmp = Root; + + while (tmp != null) + { + int cmp = key.CompareTo(tmp.Key); + if (cmp < 0) + { + if (tmp.Left != null) + { + tmp = tmp.Left; + } + else + { + return tmp; + } + } + else if (cmp > 0) + { + if (tmp.Right != null) + { + tmp = tmp.Right; + } + else + { + Node parent = tmp.Parent; + Node ptr = tmp; + while (parent != null && ptr == parent.Right) + { + ptr = parent; + parent = parent.Parent; + } + + return parent; + } + } + else + { + return tmp; + } + } + + return null; + } + + #endregion + + #region Interface Implementations + + // Method descriptions are not provided as they are already included as part of the interface. + public bool ContainsKey(TKey key) + { + ArgumentNullException.ThrowIfNull(key); + + return GetNode(key) != null; + } + + bool IDictionary.Remove(TKey key) + { + int count = Count; + Remove(key); + return count > Count; + } + + public bool TryGetValue(TKey key, [MaybeNullWhen(false)] out TValue value) + { + ArgumentNullException.ThrowIfNull(key); + + Node node = GetNode(key); + value = node != null ? node.Value : default; + return node != null; + } + + public void Add(KeyValuePair item) + { + ArgumentNullException.ThrowIfNull(item.Key); + + Add(item.Key, item.Value); + } + + public bool Contains(KeyValuePair item) + { + if (item.Key == null) + { + return false; + } + + Node node = GetNode(item.Key); + if (node != null) + { + return node.Key.Equals(item.Key) && node.Value.Equals(item.Value); + } + + return false; + } + + public void CopyTo(KeyValuePair[] array, int arrayIndex) + { + if (arrayIndex < 0 || array.Length - arrayIndex < this.Count) + { + throw new ArgumentOutOfRangeException(nameof(arrayIndex)); + } + + SortedList list = GetKeyValues(); + + int offset = 0; + + for (int i = arrayIndex; i < array.Length && offset < list.Count; i++) + { + array[i] = new KeyValuePair(list.Keys[i], list.Values[i]); + offset++; + } + } + + public bool Remove(KeyValuePair item) + { + Node node = GetNode(item.Key); + + if (node == null) + { + return false; + } + + if (node.Value.Equals(item.Value)) + { + int count = Count; + Remove(item.Key); + return count > Count; + } + + return false; + } + + public IEnumerator> GetEnumerator() + { + return GetKeyValues().GetEnumerator(); + } + + IEnumerator IEnumerable.GetEnumerator() + { + return GetKeyValues().GetEnumerator(); + } + + public ICollection Keys => GetKeyValues().Keys; + + public ICollection Values => GetKeyValues().Values; + + public bool IsReadOnly => false; + + public TValue this[TKey key] + { + get => Get(key); + set => Add(key, value); + } + + #endregion + + #region Private Interface Helper Methods + + /// + /// Returns a sorted list of all the node keys / values in the tree. + /// + /// List of node keys + private SortedList GetKeyValues() + { + SortedList set = new(); + Queue> queue = new(); + if (Root != null) + { + queue.Enqueue(Root); + } + + while (queue.TryDequeue(out Node node)) + { + set.Add(node.Key, node.Value); + if (null != node.Left) + { + queue.Enqueue(node.Left); + } + + if (null != node.Right) + { + queue.Enqueue(node.Right); + } + } + + return set; + } + + #endregion + } + + /// + /// Represents a node in the TreeDictionary which contains a key and value of generic type K and V, respectively. + /// + /// Key of the node + /// Value of the node + public class Node : IntrusiveRedBlackTreeNode> where TKey : IComparable + { + internal TKey Key; + internal TValue Value; + + internal Node(TKey key, TValue value, Node parent) + { + Key = key; + Value = value; + Parent = parent; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/AntiAliasing.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/AntiAliasing.cs new file mode 100644 index 0000000..8f4b272 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/AntiAliasing.cs @@ -0,0 +1,15 @@ +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration +{ + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum AntiAliasing + { + None, + Fxaa, + SmaaLow, + SmaaMedium, + SmaaHigh, + SmaaUltra, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/AppDataManager.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/AppDataManager.cs new file mode 100644 index 0000000..9f3dba3 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/AppDataManager.cs @@ -0,0 +1,328 @@ +using Ryujinx.Common.Logging; +using Ryujinx.Common.Utilities; +using System; +using System.IO; +using System.Runtime.Versioning; + +namespace Ryujinx.Common.Configuration +{ + public static class AppDataManager + { + private const string DefaultBaseDir = "Ryujinx"; + private const string DefaultPortableDir = "portable"; + + // The following 3 are always part of Base Directory + private const string GamesDir = "games"; + private const string ProfilesDir = "profiles"; + private const string KeysDir = "system"; + + public enum LaunchMode + { + UserProfile, + Portable, + Custom, + } + + public static LaunchMode Mode { get; private set; } + + public static string BaseDirPath { get; private set; } + public static string GamesDirPath { get; private set; } + public static string ProfilesDirPath { get; private set; } + public static string KeysDirPath { get; private set; } + public static string KeysDirPathUser { get; } + + public static string LogsDirPath { get; private set; } + + public const string DefaultNandDir = "bis"; + public const string DefaultSdcardDir = "sdcard"; + private const string DefaultModsDir = "mods"; + + public static string CustomModsPath { get; set; } + public static string CustomSdModsPath { get; set; } + public static string CustomNandPath { get; set; } // TODO: Actually implement this into VFS + public static string CustomSdCardPath { get; set; } // TODO: Actually implement this into VFS + + static AppDataManager() + { + KeysDirPathUser = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), ".switch"); + } + + public static void Initialize(string baseDirPath) + { + string appDataPath = Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData); + + if (appDataPath.Length == 0) + { + appDataPath = Environment.GetFolderPath(Environment.SpecialFolder.UserProfile); + } + + string userProfilePath = Path.Combine(appDataPath, DefaultBaseDir); + string portablePath = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, DefaultPortableDir); + + // On macOS, check for a portable directory next to the app bundle as well. + if (OperatingSystem.IsMacOS() && !Directory.Exists(portablePath)) + { + string bundlePath = Path.GetFullPath(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "..", "..")); + // Make sure we're actually running within an app bundle. + if (bundlePath.EndsWith(".app")) + { + portablePath = Path.GetFullPath(Path.Combine(bundlePath, "..", DefaultPortableDir)); + } + } + + if (Directory.Exists(portablePath)) + { + BaseDirPath = portablePath; + Mode = LaunchMode.Portable; + } + else + { + BaseDirPath = userProfilePath; + Mode = LaunchMode.UserProfile; + } + + if (baseDirPath != null && baseDirPath != userProfilePath) + { + if (!Directory.Exists(baseDirPath)) + { + Logger.Error?.Print(LogClass.Application, $"Custom Data Directory '{baseDirPath}' does not exist. Falling back to {Mode}..."); + } + else + { + BaseDirPath = baseDirPath; + Mode = LaunchMode.Custom; + } + } + + BaseDirPath = Path.GetFullPath(BaseDirPath); // convert relative paths + + if (IsPathSymlink(BaseDirPath)) + { + Logger.Warning?.Print(LogClass.Application, "Application data directory is a symlink. This may be unintended."); + } + + SetupBasePaths(); + } + + public static string GetOrCreateLogsDir() + { + if (Directory.Exists(LogsDirPath)) + { + return LogsDirPath; + } + + Logger.Notice.Print(LogClass.Application, "Logging directory not found; attempting to create new logging directory."); + LogsDirPath = SetUpLogsDir(); + + return LogsDirPath; + } + + private static string SetUpLogsDir() + { + string logDir = string.Empty; + + if (Mode == LaunchMode.Portable) + { + logDir = Path.Combine(BaseDirPath, "Logs"); + try + { + Directory.CreateDirectory(logDir); + } + catch + { + Logger.Warning?.Print(LogClass.Application, $"Logging directory could not be created '{logDir}'"); + + return null; + } + } + else + { + if (OperatingSystem.IsMacOS()) + { + // NOTE: Should evaluate to "~/Library/Logs/Ryujinx/". + logDir = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), "Library", "Logs", DefaultBaseDir); + try + { + Directory.CreateDirectory(logDir); + } + catch + { + Logger.Warning?.Print(LogClass.Application, $"Logging directory could not be created '{logDir}'"); + logDir = string.Empty; + } + + if (string.IsNullOrEmpty(logDir)) + { + // NOTE: Should evaluate to "~/Library/Application Support/Ryujinx/Logs". + logDir = Path.Combine(BaseDirPath, "Logs"); + + try + { + Directory.CreateDirectory(logDir); + } + catch + { + Logger.Warning?.Print(LogClass.Application, $"Logging directory could not be created '{logDir}'"); + + return null; + } + } + } + else if (OperatingSystem.IsWindows()) + { + // NOTE: Should evaluate to a "Logs" directory in whatever directory Ryujinx was launched from. + logDir = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "Logs"); + try + { + Directory.CreateDirectory(logDir); + } + catch + { + Logger.Warning?.Print(LogClass.Application, $"Logging directory could not be created '{logDir}'"); + logDir = string.Empty; + } + + if (string.IsNullOrEmpty(logDir)) + { + // NOTE: Should evaluate to "C:\Users\user\AppData\Roaming\Ryujinx\Logs". + logDir = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData), DefaultBaseDir, "Logs"); + + try + { + Directory.CreateDirectory(logDir); + } + catch + { + Logger.Warning?.Print(LogClass.Application, $"Logging directory could not be created '{logDir}'"); + + return null; + } + } + } + else if (OperatingSystem.IsLinux()) + { + // NOTE: Should evaluate to "~/.config/Ryujinx/Logs". + logDir = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData), DefaultBaseDir, "Logs"); + + try + { + Directory.CreateDirectory(logDir); + } + catch + { + Logger.Warning?.Print(LogClass.Application, $"Logging directory could not be created '{logDir}'"); + + return null; + } + } + } + + return logDir; + } + + private static void SetupBasePaths() + { + Directory.CreateDirectory(BaseDirPath); + LogsDirPath = SetUpLogsDir(); + Directory.CreateDirectory(GamesDirPath = Path.Combine(BaseDirPath, GamesDir)); + Directory.CreateDirectory(ProfilesDirPath = Path.Combine(BaseDirPath, ProfilesDir)); + Directory.CreateDirectory(KeysDirPath = Path.Combine(BaseDirPath, KeysDir)); + } + + // Check if existing old baseDirPath is a symlink, to prevent possible errors. + // Should be removed, when the existence of the old directory isn't checked anymore. + private static bool IsPathSymlink(string path) + { + try + { + FileAttributes attributes = File.GetAttributes(path); + return (attributes & FileAttributes.ReparsePoint) == FileAttributes.ReparsePoint; + } + catch + { + return false; + } + } + + [SupportedOSPlatform("macos")] + public static void FixMacOSConfigurationFolders() + { + string oldConfigPath = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), + ".config", DefaultBaseDir); + if (Path.Exists(oldConfigPath) && !IsPathSymlink(oldConfigPath) && !Path.Exists(BaseDirPath)) + { + FileSystemUtils.MoveDirectory(oldConfigPath, BaseDirPath); + Directory.CreateSymbolicLink(oldConfigPath, BaseDirPath); + } + + string correctApplicationDataDirectoryPath = + Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData), DefaultBaseDir); + if (IsPathSymlink(correctApplicationDataDirectoryPath)) + { + //copy the files somewhere temporarily + string tempPath = Path.Combine(Path.GetTempPath(), DefaultBaseDir); + try + { + FileSystemUtils.CopyDirectory(correctApplicationDataDirectoryPath, tempPath, true); + } + catch (Exception exception) + { + Logger.Error?.Print(LogClass.Application, + $"Critical error copying Ryujinx application data into the temp folder. {exception}"); + try + { + FileSystemInfo resolvedDirectoryInfo = + Directory.ResolveLinkTarget(correctApplicationDataDirectoryPath, true); + string resolvedPath = resolvedDirectoryInfo.FullName; + Logger.Error?.Print(LogClass.Application, $"Please manually move your Ryujinx data from {resolvedPath} to {correctApplicationDataDirectoryPath}, and remove the symlink."); + } + catch (Exception symlinkException) + { + Logger.Error?.Print(LogClass.Application, $"Unable to resolve the symlink for Ryujinx application data: {symlinkException}. Follow the symlink at {correctApplicationDataDirectoryPath} and move your data back to the Application Support folder."); + } + + return; + } + + //delete the symlink + try + { + //This will fail if this is an actual directory, so there is no way we can actually delete user data here. + File.Delete(correctApplicationDataDirectoryPath); + } + catch (Exception exception) + { + Logger.Error?.Print(LogClass.Application, + $"Critical error deleting the Ryujinx application data folder symlink at {correctApplicationDataDirectoryPath}. {exception}"); + try + { + FileSystemInfo resolvedDirectoryInfo = + Directory.ResolveLinkTarget(correctApplicationDataDirectoryPath, true); + string resolvedPath = resolvedDirectoryInfo.FullName; + Logger.Error?.Print(LogClass.Application, $"Please manually move your Ryujinx data from {resolvedPath} to {correctApplicationDataDirectoryPath}, and remove the symlink."); + } + catch (Exception symlinkException) + { + Logger.Error?.Print(LogClass.Application, $"Unable to resolve the symlink for Ryujinx application data: {symlinkException}. Follow the symlink at {correctApplicationDataDirectoryPath} and move your data back to the Application Support folder."); + } + + return; + } + + //put the files back + try + { + FileSystemUtils.CopyDirectory(tempPath, correctApplicationDataDirectoryPath, true); + } + catch (Exception exception) + { + Logger.Error?.Print(LogClass.Application, + $"Critical error copying Ryujinx application data into the correct location. {exception}. Please manually move your application data from {tempPath} to {correctApplicationDataDirectoryPath}."); + } + } + } + + public static string GetModsPath() => CustomModsPath ?? Directory.CreateDirectory(Path.Combine(BaseDirPath, DefaultModsDir)).FullName; + public static string GetSdModsPath() => CustomSdModsPath ?? Directory.CreateDirectory(Path.Combine(BaseDirPath, DefaultSdcardDir, "atmosphere")).FullName; + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/AspectRatioExtensions.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/AspectRatioExtensions.cs new file mode 100644 index 0000000..35e3f5c --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/AspectRatioExtensions.cs @@ -0,0 +1,70 @@ +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration +{ + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum AspectRatio + { + Fixed4x3, + Fixed16x9, + Fixed16x10, + Fixed21x9, + Fixed32x9, + Stretched, + } + + public static class AspectRatioExtensions + { + public static float ToFloat(this AspectRatio aspectRatio) + { + return aspectRatio.ToFloatX() / aspectRatio.ToFloatY(); + } + + public static float ToFloatX(this AspectRatio aspectRatio) + { + return aspectRatio switch + { +#pragma warning disable IDE0055 // Disable formatting + AspectRatio.Fixed4x3 => 4.0f, + AspectRatio.Fixed16x9 => 16.0f, + AspectRatio.Fixed16x10 => 16.0f, + AspectRatio.Fixed21x9 => 21.0f, + AspectRatio.Fixed32x9 => 32.0f, + _ => 16.0f, +#pragma warning restore IDE0055 + }; + } + + + + public static float ToFloatY(this AspectRatio aspectRatio) + { + return aspectRatio switch + { +#pragma warning disable IDE0055 // Disable formatting + AspectRatio.Fixed4x3 => 3.0f, + AspectRatio.Fixed16x9 => 9.0f, + AspectRatio.Fixed16x10 => 10.0f, + AspectRatio.Fixed21x9 => 9.0f, + AspectRatio.Fixed32x9 => 9.0f, + _ => 9.0f, +#pragma warning restore IDE0055 + }; + } + + public static string ToText(this AspectRatio aspectRatio) + { + return aspectRatio switch + { +#pragma warning disable IDE0055 // Disable formatting + AspectRatio.Fixed4x3 => "4:3", + AspectRatio.Fixed16x9 => "16:9", + AspectRatio.Fixed16x10 => "16:10", + AspectRatio.Fixed21x9 => "21:9", + AspectRatio.Fixed32x9 => "32:9", + _ => "Stretched", +#pragma warning restore IDE0055 + }; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/BackendThreading.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/BackendThreading.cs new file mode 100644 index 0000000..6135c83 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/BackendThreading.cs @@ -0,0 +1,12 @@ +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration +{ + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum BackendThreading + { + Auto, + Off, + On, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/DirtyHack.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/DirtyHack.cs new file mode 100644 index 0000000..2e8ff50 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/DirtyHack.cs @@ -0,0 +1,62 @@ +using Gommon; +using System; +using System.Collections.Generic; +using System.Linq; + +namespace Ryujinx.Common.Configuration +{ + [Flags] + public enum DirtyHack : byte + { + Xc2MenuSoftlockFix = 1, + // ShaderTranslationDelay = 2 + NifmServiceDisableIsAnyInternetRequestAccepted = 3 + } + + public readonly struct EnabledDirtyHack(DirtyHack hack, int value) + { + public DirtyHack Hack => hack; + public int Value => value; + + + + public ulong Pack() => Raw.PackBitFields(PackedFormat); + + public static EnabledDirtyHack Unpack(ulong packedHack) + { + uint[] unpackedFields = packedHack.UnpackBitFields(PackedFormat); + // ReSharper disable once PatternAlwaysMatches + if (unpackedFields is not [uint hack, uint value]) + throw new Exception("The unpack operation on the integer resulted in an invalid unpacked result."); + + return new EnabledDirtyHack((DirtyHack)hack, (int)value); + } + + private uint[] Raw => [(uint)Hack, (uint)Value.CoerceAtLeast(0)]; + + public static readonly byte[] PackedFormat = [8, 32]; + } + + public class DirtyHacks : Dictionary + { + public DirtyHacks(IEnumerable hacks) + => hacks.ForEach(edh => Add(edh.Hack, edh.Value)); + + public DirtyHacks(ulong[] packedHacks) : this(packedHacks.Select(EnabledDirtyHack.Unpack)) { } + + public ulong[] PackEntries() + => Entries.Select(it => it.Pack()).ToArray(); + + public EnabledDirtyHack[] Entries + => this + .Select(it => new EnabledDirtyHack(it.Key, it.Value)) + .ToArray(); + + public static implicit operator DirtyHacks(EnabledDirtyHack[] hacks) => new(hacks); + public static implicit operator DirtyHacks(ulong[] packedHacks) => new(packedHacks); + + public new int this[DirtyHack hack] => TryGetValue(hack, out int value) ? value : -1; + + public bool IsEnabled(DirtyHack hack) => ContainsKey(hack); + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/DownloadableContentContainer.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/DownloadableContentContainer.cs new file mode 100644 index 0000000..e5c2d07 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/DownloadableContentContainer.cs @@ -0,0 +1,13 @@ +using System.Collections.Generic; +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration +{ + public struct DownloadableContentContainer + { + [JsonPropertyName("path")] + public string ContainerPath { get; set; } + [JsonPropertyName("dlc_nca_list")] + public List DownloadableContentNcaList { get; set; } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/DownloadableContentJsonSerializerContext.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/DownloadableContentJsonSerializerContext.cs new file mode 100644 index 0000000..5ac017a --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/DownloadableContentJsonSerializerContext.cs @@ -0,0 +1,11 @@ +using System.Collections.Generic; +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration +{ + [JsonSourceGenerationOptions(WriteIndented = true)] + [JsonSerializable(typeof(List))] + public partial class DownloadableContentJsonSerializerContext : JsonSerializerContext + { + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/DownloadableContentNca.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/DownloadableContentNca.cs new file mode 100644 index 0000000..3584824 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/DownloadableContentNca.cs @@ -0,0 +1,14 @@ +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration +{ + public struct DownloadableContentNca + { + [JsonPropertyName("path")] + public string FullPath { get; set; } + [JsonPropertyName("title_id")] + public ulong TitleId { get; set; } + [JsonPropertyName("is_enabled")] + public bool Enabled { get; set; } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/GraphicsBackend.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/GraphicsBackend.cs new file mode 100644 index 0000000..b423611 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/GraphicsBackend.cs @@ -0,0 +1,11 @@ +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration +{ + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum GraphicsBackend + { + Vulkan, + OpenGl, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/GraphicsDebugLevel.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/GraphicsDebugLevel.cs new file mode 100644 index 0000000..0ecf717 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/GraphicsDebugLevel.cs @@ -0,0 +1,13 @@ +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration +{ + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum GraphicsDebugLevel + { + None, + Error, + Slowdowns, + All, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/AssignedInputDevice.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/AssignedInputDevice.cs new file mode 100644 index 0000000..dba3c2d --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/AssignedInputDevice.cs @@ -0,0 +1,11 @@ +namespace Ryujinx.Common.Configuration.Hid +{ + public class AssignedInputDevice + { + public AssignedInputDeviceType Type { get; set; } + + public string Id { get; set; } + + public string ProfileName { get; set; } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/AssignedInputDeviceType.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/AssignedInputDeviceType.cs new file mode 100644 index 0000000..aa6306e --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/AssignedInputDeviceType.cs @@ -0,0 +1,11 @@ +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration.Hid +{ + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum AssignedInputDeviceType + { + Keyboard, + Controller, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/GamepadInputId.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/GamepadInputId.cs new file mode 100644 index 0000000..9067dac --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/GamepadInputId.cs @@ -0,0 +1,58 @@ +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration.Hid.Controller +{ + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum GamepadInputId : byte + { + Unbound, + A, + B, + X, + Y, + LeftStick, + RightStick, + LeftShoulder, + RightShoulder, + + // Likely axis + LeftTrigger, + // Likely axis + RightTrigger, + + DpadUp, + DpadDown, + DpadLeft, + DpadRight, + + // Special buttons + + Minus, + Plus, + + Back = Minus, + Start = Plus, + + Guide, + Misc1, + + // Xbox Elite paddle + Paddle1, + Paddle2, + Paddle3, + Paddle4, + + // PS5 touchpad button + Touchpad, + + // Virtual buttons for single joycon (left) + SingleLeftTrigger0, + SingleRightTrigger0, + + // Virtual buttons for single joycon (right) + SingleLeftTrigger1, + SingleRightTrigger1, + + Count, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/GenericControllerInputConfig.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/GenericControllerInputConfig.cs new file mode 100644 index 0000000..475a9f1 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/GenericControllerInputConfig.cs @@ -0,0 +1,87 @@ +using Ryujinx.Common.Configuration.Hid.Controller.Motion; +using System; +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration.Hid.Controller +{ + public class GenericControllerInputConfig : GenericInputConfigurationCommon where TButton : unmanaged where TStick : unmanaged + { + [JsonIgnore] + private float _deadzoneLeft; + [JsonIgnore] + private float _deadzoneRight; + [JsonIgnore] + private float _triggerThreshold; + + /// + /// Left JoyCon Controller Stick Bindings + /// + public JoyconConfigControllerStick LeftJoyconStick { get; set; } + + /// + /// Right JoyCon Controller Stick Bindings + /// + public JoyconConfigControllerStick RightJoyconStick { get; set; } + + /// + /// Controller Left Analog Stick Deadzone + /// + public float DeadzoneLeft + { + get => _deadzoneLeft; set + { + _deadzoneLeft = MathF.Round(value, 3); + OnPropertyChanged(); + } + } + + /// + /// Controller Right Analog Stick Deadzone + /// + public float DeadzoneRight + { + get => _deadzoneRight; set + { + _deadzoneRight = MathF.Round(value, 3); + OnPropertyChanged(); + } + } + + /// + /// Controller Left Analog Stick Range + /// + public float RangeLeft { get; set; } + + /// + /// Controller Right Analog Stick Range + /// + public float RangeRight { get; set; } + + /// + /// Controller Trigger Threshold + /// + public float TriggerThreshold + { + get => _triggerThreshold; set + { + _triggerThreshold = MathF.Round(value, 3); + OnPropertyChanged(); + } + } + + /// + /// Controller Motion Settings + /// + public MotionConfigController Motion { get; set; } + + /// + /// Controller Rumble Settings + /// + public RumbleConfigController Rumble { get; set; } + + /// + /// Controller LED Settings + /// + public LedConfigController Led { get; set; } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/JoyconConfigControllerStick.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/JoyconConfigControllerStick.cs new file mode 100644 index 0000000..6e9133d --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/JoyconConfigControllerStick.cs @@ -0,0 +1,13 @@ +namespace Ryujinx.Common.Configuration.Hid.Controller +{ + public class JoyconConfigControllerStick + where TButton : unmanaged + where TStick : unmanaged + { + public TStick Joystick { get; set; } + public bool InvertStickX { get; set; } + public bool InvertStickY { get; set; } + public bool Rotate90CW { get; set; } + public TButton StickButton { get; set; } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/LedConfigController.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/LedConfigController.cs new file mode 100644 index 0000000..33b8b8c --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/LedConfigController.cs @@ -0,0 +1,25 @@ +namespace Ryujinx.Common.Configuration.Hid.Controller +{ + public class LedConfigController + { + /// + /// Enable LED color changing by the emulator + /// + public bool EnableLed { get; set; } + + /// + /// Ignores the color and disables the LED entirely. + /// + public bool TurnOffLed { get; set; } + + /// + /// Ignores the color and uses the rainbow color functionality for the LED. + /// + public bool UseRainbow { get; set; } + + /// + /// Packed RGB int of the color + /// + public uint LedColor { get; set; } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/Motion/CemuHookMotionConfigController.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/Motion/CemuHookMotionConfigController.cs new file mode 100644 index 0000000..5c005f4 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/Motion/CemuHookMotionConfigController.cs @@ -0,0 +1,30 @@ +namespace Ryujinx.Common.Configuration.Hid.Controller.Motion +{ + public class CemuHookMotionConfigController : MotionConfigController + { + /// + /// Motion Controller Slot + /// + public int Slot { get; set; } + + /// + /// Motion Controller Alternative Slot, for RightJoyCon in Pair mode + /// + public int AltSlot { get; set; } + + /// + /// Mirror motion input in Pair mode + /// + public bool MirrorInput { get; set; } + + /// + /// Host address of the DSU Server + /// + public string DsuServerHost { get; set; } + + /// + /// Port of the DSU Server + /// + public int DsuServerPort { get; set; } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/Motion/JsonMotionConfigControllerConverter.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/Motion/JsonMotionConfigControllerConverter.cs new file mode 100644 index 0000000..743dc73 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/Motion/JsonMotionConfigControllerConverter.cs @@ -0,0 +1,79 @@ +using Ryujinx.Common.Utilities; +using System; +using System.Text.Json; +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration.Hid.Controller.Motion +{ + class JsonMotionConfigControllerConverter : JsonConverter + { + private static readonly MotionConfigJsonSerializerContext _serializerContext = new(JsonHelper.GetDefaultSerializerOptions()); + + private static MotionInputBackendType GetMotionInputBackendType(ref Utf8JsonReader reader) + { + // Temporary reader to get the backend type + Utf8JsonReader tempReader = reader; + + MotionInputBackendType result = MotionInputBackendType.Invalid; + + while (tempReader.Read()) + { + // NOTE: We scan all properties ignoring the depth entirely on purpose. + // The reason behind this is that we cannot track in a reliable way the depth of the object because Utf8JsonReader never emit the first TokenType == StartObject if the json start with an object. + // As such, this code will try to parse very field named "motion_backend" to the correct enum. + if (tempReader.TokenType == JsonTokenType.PropertyName) + { + string propertyName = tempReader.GetString(); + + if (propertyName.Equals("motion_backend")) + { + tempReader.Read(); + + if (tempReader.TokenType == JsonTokenType.String) + { + string backendTypeRaw = tempReader.GetString(); + + if (!Enum.TryParse(backendTypeRaw, out result)) + { + result = MotionInputBackendType.Invalid; + } + else + { + break; + } + } + } + } + } + + return result; + } + + public override MotionConfigController Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) + { + MotionInputBackendType motionBackendType = GetMotionInputBackendType(ref reader); + + return motionBackendType switch + { + MotionInputBackendType.GamepadDriver => JsonSerializer.Deserialize(ref reader, _serializerContext.StandardMotionConfigController), + MotionInputBackendType.CemuHook => JsonSerializer.Deserialize(ref reader, _serializerContext.CemuHookMotionConfigController), + _ => throw new InvalidOperationException($"Unknown backend type {motionBackendType}"), + }; + } + + public override void Write(Utf8JsonWriter writer, MotionConfigController value, JsonSerializerOptions options) + { + switch (value.MotionBackend) + { + case MotionInputBackendType.GamepadDriver: + JsonSerializer.Serialize(writer, value as StandardMotionConfigController, _serializerContext.StandardMotionConfigController); + break; + case MotionInputBackendType.CemuHook: + JsonSerializer.Serialize(writer, value as CemuHookMotionConfigController, _serializerContext.CemuHookMotionConfigController); + break; + default: + throw new ArgumentException($"Unknown motion backend type {value.MotionBackend}"); + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/Motion/MotionConfigController.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/Motion/MotionConfigController.cs new file mode 100644 index 0000000..10e26b5 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/Motion/MotionConfigController.cs @@ -0,0 +1,25 @@ +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration.Hid.Controller.Motion +{ + [JsonConverter(typeof(JsonMotionConfigControllerConverter))] + public class MotionConfigController + { + public MotionInputBackendType MotionBackend { get; set; } + + /// + /// Gyro Sensitivity + /// + public int Sensitivity { get; set; } + + /// + /// Gyro Deadzone + /// + public double GyroDeadzone { get; set; } + + /// + /// Enable Motion Controls + /// + public bool EnableMotion { get; set; } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/Motion/MotionConfigJsonSerializerContext.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/Motion/MotionConfigJsonSerializerContext.cs new file mode 100644 index 0000000..5138b5f --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/Motion/MotionConfigJsonSerializerContext.cs @@ -0,0 +1,12 @@ +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration.Hid.Controller.Motion +{ + [JsonSourceGenerationOptions(WriteIndented = true)] + [JsonSerializable(typeof(MotionConfigController))] + [JsonSerializable(typeof(CemuHookMotionConfigController))] + [JsonSerializable(typeof(StandardMotionConfigController))] + public partial class MotionConfigJsonSerializerContext : JsonSerializerContext + { + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/Motion/MotionInputBackendType.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/Motion/MotionInputBackendType.cs new file mode 100644 index 0000000..f1b92c3 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/Motion/MotionInputBackendType.cs @@ -0,0 +1,12 @@ +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration.Hid.Controller.Motion +{ + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum MotionInputBackendType : byte + { + Invalid, + GamepadDriver, + CemuHook, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/Motion/StandardMotionConfigController.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/Motion/StandardMotionConfigController.cs new file mode 100644 index 0000000..febbe15 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/Motion/StandardMotionConfigController.cs @@ -0,0 +1,4 @@ +namespace Ryujinx.Common.Configuration.Hid.Controller.Motion +{ + public class StandardMotionConfigController : MotionConfigController { } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/RumbleConfigController.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/RumbleConfigController.cs new file mode 100644 index 0000000..9766bd4 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/RumbleConfigController.cs @@ -0,0 +1,25 @@ +namespace Ryujinx.Common.Configuration.Hid.Controller +{ + public class RumbleConfigController + { + /// + /// Controller Strong Rumble Multiplier + /// + public float StrongRumble { get; set; } + + /// + /// Controller Weak Rumble Multiplier + /// + public float WeakRumble { get; set; } + + /// + /// Enable Rumble + /// + public bool EnableRumble { get; set; } + + /// + /// Enable HD Rumble support + /// { } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/StickInputId.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/StickInputId.cs new file mode 100644 index 0000000..5014db4 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Controller/StickInputId.cs @@ -0,0 +1,14 @@ +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration.Hid.Controller +{ + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum StickInputId : byte + { + Unbound, + Left, + Right, + + Count, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/ControllerType.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/ControllerType.cs new file mode 100644 index 0000000..7afcc6d --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/ControllerType.cs @@ -0,0 +1,22 @@ +using System; +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration.Hid +{ + // This enum was duplicated from Ryujinx.HLE.HOS.Services.Hid.PlayerIndex and should be kept identical + [Flags] + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum ControllerType + { + None, + ProController = 1 << 0, + Handheld = 1 << 1, + JoyconPair = 1 << 2, + JoyconLeft = 1 << 3, + JoyconRight = 1 << 4, + Invalid = 1 << 5, + Pokeball = 1 << 6, + SystemExternal = 1 << 29, + System = 1 << 30, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/GenericInputConfigurationCommon.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/GenericInputConfigurationCommon.cs new file mode 100644 index 0000000..daac2ae --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/GenericInputConfigurationCommon.cs @@ -0,0 +1,15 @@ +namespace Ryujinx.Common.Configuration.Hid +{ + public class GenericInputConfigurationCommon : InputConfig where TButton : unmanaged + { + /// + /// Left JoyCon Controller Bindings + /// + public LeftJoyconCommonConfig LeftJoycon { get; set; } + + /// + /// Right JoyCon Controller Bindings + /// + public RightJoyconCommonConfig RightJoycon { get; set; } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/InputBackendType.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/InputBackendType.cs new file mode 100644 index 0000000..b9d45a5 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/InputBackendType.cs @@ -0,0 +1,13 @@ +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration.Hid +{ + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum InputBackendType + { + Invalid, + WindowKeyboard, + GamepadSDL2, //backcompat + GamepadSDL3, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/InputConfig.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/InputConfig.cs new file mode 100644 index 0000000..b2e403f --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/InputConfig.cs @@ -0,0 +1,52 @@ +using System.ComponentModel; +using System.Runtime.CompilerServices; +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration.Hid +{ + [JsonConverter(typeof(JsonInputConfigConverter))] + public class InputConfig : INotifyPropertyChanged + { + /// + /// The current version of the input file format + /// + public const int CurrentVersion = 1; + + public int Version { get; set; } + + public InputBackendType Backend { get; set; } + + /// + /// Controller id + /// + public string Id { get; set; } + + /// + /// Controller name + /// + public string Name { get; set; } + + /// + /// Controller's Type + /// + public ControllerType ControllerType { get; set; } + + /// + /// Player's Index for the controller + /// + public PlayerIndex PlayerIndex { get; set; } + + /// + /// Allow a keyboard configuration to temporarily promote to a connected gamepad, + /// while preserving the existing keyboard fallback path when that gamepad disappears. + /// + public bool EnableDynamicGamepadSwap { get; set; } + + public event PropertyChangedEventHandler PropertyChanged; + + protected virtual void OnPropertyChanged([CallerMemberName] string propertyName = null) + { + PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(propertyName)); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/InputConfigJsonSerializerContext.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/InputConfigJsonSerializerContext.cs new file mode 100644 index 0000000..c840171 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/InputConfigJsonSerializerContext.cs @@ -0,0 +1,14 @@ +using Ryujinx.Common.Configuration.Hid.Controller; +using Ryujinx.Common.Configuration.Hid.Keyboard; +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration.Hid +{ + [JsonSourceGenerationOptions(WriteIndented = true)] + [JsonSerializable(typeof(InputConfig))] + [JsonSerializable(typeof(StandardKeyboardInputConfig))] + [JsonSerializable(typeof(StandardControllerInputConfig))] + public partial class InputConfigJsonSerializerContext : JsonSerializerContext + { + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/JsonInputConfigConverter.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/JsonInputConfigConverter.cs new file mode 100644 index 0000000..8a5ecea --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/JsonInputConfigConverter.cs @@ -0,0 +1,81 @@ +using Ryujinx.Common.Configuration.Hid.Controller; +using Ryujinx.Common.Configuration.Hid.Keyboard; +using Ryujinx.Common.Utilities; +using System; +using System.Text.Json; +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration.Hid +{ + public class JsonInputConfigConverter : JsonConverter + { + private static readonly InputConfigJsonSerializerContext _serializerContext = new(JsonHelper.GetDefaultSerializerOptions()); + + private static InputBackendType GetInputBackendType(ref Utf8JsonReader reader) + { + // Temporary reader to get the backend type + Utf8JsonReader tempReader = reader; + + InputBackendType result = InputBackendType.Invalid; + + while (tempReader.Read()) + { + // NOTE: We scan all properties ignoring the depth entirely on purpose. + // The reason behind this is that we cannot track in a reliable way the depth of the object because Utf8JsonReader never emit the first TokenType == StartObject if the json start with an object. + // As such, this code will try to parse very field named "backend" to the correct enum. + if (tempReader.TokenType == JsonTokenType.PropertyName) + { + string propertyName = tempReader.GetString(); + + if (propertyName.Equals("backend")) + { + tempReader.Read(); + + if (tempReader.TokenType == JsonTokenType.String) + { + string backendTypeRaw = tempReader.GetString(); + + if (!Enum.TryParse(backendTypeRaw, out result)) + { + result = InputBackendType.Invalid; + } + else + { + break; + } + } + } + } + } + + return result; + } + + public override InputConfig Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) + { + InputBackendType backendType = GetInputBackendType(ref reader); + + return backendType switch + { + InputBackendType.WindowKeyboard => JsonSerializer.Deserialize(ref reader, _serializerContext.StandardKeyboardInputConfig), + InputBackendType.GamepadSDL2 or InputBackendType.GamepadSDL3 => JsonSerializer.Deserialize(ref reader, _serializerContext.StandardControllerInputConfig), + _ => throw new InvalidOperationException($"Unknown backend type {backendType}"), + }; + } + + public override void Write(Utf8JsonWriter writer, InputConfig value, JsonSerializerOptions options) + { + switch (value.Backend) + { + case InputBackendType.WindowKeyboard: + JsonSerializer.Serialize(writer, value as StandardKeyboardInputConfig, _serializerContext.StandardKeyboardInputConfig); + break; + case InputBackendType.GamepadSDL2 or InputBackendType.GamepadSDL3: + JsonSerializer.Serialize(writer, value as StandardControllerInputConfig, _serializerContext.StandardControllerInputConfig); + break; + default: + throw new ArgumentException($"Unknown backend type {value.Backend}"); + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Key.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Key.cs new file mode 100644 index 0000000..6f73ce1 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Key.cs @@ -0,0 +1,142 @@ +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration.Hid +{ + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum Key + { + Unknown, + ShiftLeft, + ShiftRight, + ControlLeft, + ControlRight, + AltLeft, + AltRight, + WinLeft, + WinRight, + Menu, + F1, + F2, + F3, + F4, + F5, + F6, + F7, + F8, + F9, + F10, + F11, + F12, + F13, + F14, + F15, + F16, + F17, + F18, + F19, + F20, + F21, + F22, + F23, + F24, + F25, + F26, + F27, + F28, + F29, + F30, + F31, + F32, + F33, + F34, + F35, + Up, + Down, + Left, + Right, + Enter, + Escape, + Space, + Tab, + BackSpace, + Insert, + Delete, + PageUp, + PageDown, + Home, + End, + CapsLock, + ScrollLock, + PrintScreen, + Pause, + NumLock, + Clear, + Keypad0, + Keypad1, + Keypad2, + Keypad3, + Keypad4, + Keypad5, + Keypad6, + Keypad7, + Keypad8, + Keypad9, + KeypadDivide, + KeypadMultiply, + KeypadSubtract, + KeypadAdd, + KeypadDecimal, + KeypadEnter, + A, + B, + C, + D, + E, + F, + G, + H, + I, + J, + K, + L, + M, + N, + O, + P, + Q, + R, + S, + T, + U, + V, + W, + X, + Y, + Z, + Number0, + Number1, + Number2, + Number3, + Number4, + Number5, + Number6, + Number7, + Number8, + Number9, + Tilde, + Grave, + Minus, + Plus, + BracketLeft, + BracketRight, + Semicolon, + Quote, + Comma, + Period, + Slash, + BackSlash, + Unbound, + + Count, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Keyboard/GenericKeyboardInputConfig.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Keyboard/GenericKeyboardInputConfig.cs new file mode 100644 index 0000000..7faa930 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Keyboard/GenericKeyboardInputConfig.cs @@ -0,0 +1,15 @@ +namespace Ryujinx.Common.Configuration.Hid.Keyboard +{ + public class GenericKeyboardInputConfig : GenericInputConfigurationCommon where TKey : unmanaged + { + /// + /// Left JoyCon Controller Stick Bindings + /// + public JoyconConfigKeyboardStick LeftJoyconStick { get; set; } + + /// + /// Right JoyCon Controller Stick Bindings + /// + public JoyconConfigKeyboardStick RightJoyconStick { get; set; } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Keyboard/JoyconConfigKeyboardStick.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Keyboard/JoyconConfigKeyboardStick.cs new file mode 100644 index 0000000..0e8a586 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Keyboard/JoyconConfigKeyboardStick.cs @@ -0,0 +1,11 @@ +namespace Ryujinx.Common.Configuration.Hid.Keyboard +{ + public class JoyconConfigKeyboardStick where TKey : unmanaged + { + public TKey StickUp { get; set; } + public TKey StickDown { get; set; } + public TKey StickLeft { get; set; } + public TKey StickRight { get; set; } + public TKey StickButton { get; set; } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Keyboard/StandardKeyboardInputConfig.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Keyboard/StandardKeyboardInputConfig.cs new file mode 100644 index 0000000..c695c68 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/Keyboard/StandardKeyboardInputConfig.cs @@ -0,0 +1,4 @@ +namespace Ryujinx.Common.Configuration.Hid.Keyboard +{ + public class StandardKeyboardInputConfig : GenericKeyboardInputConfig { } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/KeyboardHotkeys.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/KeyboardHotkeys.cs new file mode 100644 index 0000000..8e65e60 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/KeyboardHotkeys.cs @@ -0,0 +1,19 @@ +namespace Ryujinx.Common.Configuration.Hid +{ + public class KeyboardHotkeys + { + public Key ToggleVSyncMode { get; set; } + public Key Screenshot { get; set; } + public Key ShowUI { get; set; } + public Key Pause { get; set; } + public Key ToggleMute { get; set; } + public Key ResScaleUp { get; set; } + public Key ResScaleDown { get; set; } + public Key VolumeUp { get; set; } + public Key VolumeDown { get; set; } + public Key CustomVSyncIntervalIncrement { get; set; } + public Key CustomVSyncIntervalDecrement { get; set; } + public Key TurboMode { get; set; } + public bool TurboModeWhileHeld { get; set; } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/LeftJoyconCommonConfig.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/LeftJoyconCommonConfig.cs new file mode 100644 index 0000000..737bdf3 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/LeftJoyconCommonConfig.cs @@ -0,0 +1,15 @@ +namespace Ryujinx.Common.Configuration.Hid +{ + public class LeftJoyconCommonConfig + { + public TButton ButtonMinus { get; set; } + public TButton ButtonL { get; set; } + public TButton ButtonZl { get; set; } + public TButton ButtonSl { get; set; } + public TButton ButtonSr { get; set; } + public TButton DpadUp { get; set; } + public TButton DpadDown { get; set; } + public TButton DpadLeft { get; set; } + public TButton DpadRight { get; set; } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/PhysicalKey.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/PhysicalKey.cs new file mode 100644 index 0000000..29fcc72 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/PhysicalKey.cs @@ -0,0 +1,142 @@ +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration.Hid +{ + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum PhysicalKey + { + Unknown, + ShiftLeft, + ShiftRight, + ControlLeft, + ControlRight, + AltLeft, + AltRight, + WinLeft, + WinRight, + Menu, + F1, + F2, + F3, + F4, + F5, + F6, + F7, + F8, + F9, + F10, + F11, + F12, + F13, + F14, + F15, + F16, + F17, + F18, + F19, + F20, + F21, + F22, + F23, + F24, + F25, + F26, + F27, + F28, + F29, + F30, + F31, + F32, + F33, + F34, + F35, + Up, + Down, + Left, + Right, + Enter, + Escape, + Space, + Tab, + BackSpace, + Insert, + Delete, + PageUp, + PageDown, + Home, + End, + CapsLock, + ScrollLock, + PrintScreen, + Pause, + NumLock, + Clear, + Keypad0, + Keypad1, + Keypad2, + Keypad3, + Keypad4, + Keypad5, + Keypad6, + Keypad7, + Keypad8, + Keypad9, + KeypadDivide, + KeypadMultiply, + KeypadSubtract, + KeypadAdd, + KeypadDecimal, + KeypadEnter, + A, + B, + C, + D, + E, + F, + G, + H, + I, + J, + K, + L, + M, + N, + O, + P, + Q, + R, + S, + T, + U, + V, + W, + X, + Y, + Z, + Number0, + Number1, + Number2, + Number3, + Number4, + Number5, + Number6, + Number7, + Number8, + Number9, + Tilde, + Grave, + Minus, + Plus, + BracketLeft, + BracketRight, + Semicolon, + Quote, + Comma, + Period, + Slash, + BackSlash, + Unbound, + + Count, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/PlayerIndex.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/PlayerIndex.cs new file mode 100644 index 0000000..f2e5b99 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/PlayerIndex.cs @@ -0,0 +1,21 @@ +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration.Hid +{ + // This enum was duplicated from Ryujinx.HLE.HOS.Services.Hid.PlayerIndex and should be kept identical + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum PlayerIndex + { + Player1 = 0, + Player2 = 1, + Player3 = 2, + Player4 = 3, + Player5 = 4, + Player6 = 5, + Player7 = 6, + Player8 = 7, + Handheld = 8, + Unknown = 9, + Auto = 10, // Shouldn't be used directly + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/PlayerInputAssignment.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/PlayerInputAssignment.cs new file mode 100644 index 0000000..e7d6bdd --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/PlayerInputAssignment.cs @@ -0,0 +1,13 @@ +using System.Collections.Generic; + +namespace Ryujinx.Common.Configuration.Hid +{ + public class PlayerInputAssignment + { + public PlayerIndex PlayerIndex { get; set; } + + public bool EnableDynamicInputSwap { get; set; } + + public List Devices { get; set; } = []; + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/PlayerInputAssignmentHelper.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/PlayerInputAssignmentHelper.cs new file mode 100644 index 0000000..fa279b6 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/PlayerInputAssignmentHelper.cs @@ -0,0 +1,166 @@ +using Ryujinx.Common.Configuration.Hid.Keyboard; +using System; +using System.Collections.Generic; +using System.Linq; + +namespace Ryujinx.Common.Configuration.Hid +{ + public static class PlayerInputAssignmentHelper + { + public static AssignedInputDevice CreatePrimaryDevice(InputConfig inputConfig) + { + if (inputConfig == null || string.IsNullOrWhiteSpace(inputConfig.Id)) + { + return null; + } + + return new AssignedInputDevice + { + Type = inputConfig is StandardKeyboardInputConfig + ? AssignedInputDeviceType.Keyboard + : AssignedInputDeviceType.Controller, + Id = inputConfig.Id, + }; + } + + public static PlayerInputAssignment Normalize(PlayerInputAssignment assignment, AssignedInputDevice preferredDevice = null) + { + if (assignment == null) + { + return null; + } + + PlayerInputAssignment normalized = new() + { + PlayerIndex = assignment.PlayerIndex, + EnableDynamicInputSwap = assignment.EnableDynamicInputSwap, + }; + + List distinctDevices = Deduplicate(assignment.Devices); + + if (assignment.EnableDynamicInputSwap) + { + normalized.Devices.AddRange(distinctDevices.Select(Clone)); + return normalized; + } + + AssignedInputDevice primaryDevice = SelectPrimaryDevice(distinctDevices, preferredDevice) ?? Clone(preferredDevice); + + if (primaryDevice != null) + { + normalized.Devices.Add(Clone(primaryDevice)); + } + + return normalized; + } + + public static bool AreEquivalent( + PlayerInputAssignment left, + PlayerInputAssignment right, + AssignedInputDevice leftPreferredDevice = null, + AssignedInputDevice rightPreferredDevice = null) + { + if (left == null || right == null) + { + return left == right; + } + + PlayerInputAssignment normalizedLeft = Normalize(left, leftPreferredDevice); + PlayerInputAssignment normalizedRight = Normalize(right, rightPreferredDevice); + + if (normalizedLeft.EnableDynamicInputSwap != normalizedRight.EnableDynamicInputSwap) + { + return false; + } + + List<(AssignedInputDeviceType Type, string Id, string ProfileName)> leftDevices = normalizedLeft.Devices + .Select(device => (Type: device.Type, Id: device.Id, ProfileName: device.ProfileName ?? string.Empty)) + .OrderBy(device => device.Type) + .ThenBy(device => device.Id, StringComparer.Ordinal) + .ThenBy(device => device.ProfileName, StringComparer.Ordinal) + .ToList(); + + List<(AssignedInputDeviceType Type, string Id, string ProfileName)> rightDevices = normalizedRight.Devices + .Select(device => (Type: device.Type, Id: device.Id, ProfileName: device.ProfileName ?? string.Empty)) + .OrderBy(device => device.Type) + .ThenBy(device => device.Id, StringComparer.Ordinal) + .ThenBy(device => device.ProfileName, StringComparer.Ordinal) + .ToList(); + + return leftDevices.SequenceEqual(rightDevices); + } + + private static List Deduplicate(IEnumerable devices) + { + List result = []; + + if (devices == null) + { + return result; + } + + foreach (AssignedInputDevice device in devices) + { + if (device == null || string.IsNullOrWhiteSpace(device.Id)) + { + continue; + } + + int existingIndex = result.FindIndex(existing => + existing.Type == device.Type && + string.Equals(existing.Id, device.Id, StringComparison.Ordinal)); + + if (existingIndex == -1) + { + result.Add(Clone(device)); + continue; + } + + if (!string.IsNullOrWhiteSpace(device.ProfileName) || + string.IsNullOrWhiteSpace(result[existingIndex].ProfileName)) + { + result[existingIndex].ProfileName = device.ProfileName; + } + } + + return result; + } + + private static AssignedInputDevice SelectPrimaryDevice(List devices, AssignedInputDevice preferredDevice) + { + if (devices == null || devices.Count == 0) + { + return null; + } + + if (preferredDevice != null) + { + AssignedInputDevice matchedDevice = devices.FirstOrDefault(device => + device.Type == preferredDevice.Type && + string.Equals(device.Id, preferredDevice.Id, StringComparison.Ordinal)); + + if (matchedDevice != null) + { + return matchedDevice; + } + } + + return devices[0]; + } + + private static AssignedInputDevice Clone(AssignedInputDevice device) + { + if (device == null) + { + return null; + } + + return new AssignedInputDevice + { + Type = device.Type, + Id = device.Id, + ProfileName = device.ProfileName, + }; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/RightJoyconCommonConfig.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/RightJoyconCommonConfig.cs new file mode 100644 index 0000000..a15eff4 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Hid/RightJoyconCommonConfig.cs @@ -0,0 +1,15 @@ +namespace Ryujinx.Common.Configuration.Hid +{ + public class RightJoyconCommonConfig + { + public TButton ButtonPlus { get; set; } + public TButton ButtonR { get; set; } + public TButton ButtonZr { get; set; } + public TButton ButtonSl { get; set; } + public TButton ButtonSr { get; set; } + public TButton ButtonX { get; set; } + public TButton ButtonB { get; set; } + public TButton ButtonY { get; set; } + public TButton ButtonA { get; set; } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/HideCursorMode.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/HideCursorMode.cs new file mode 100644 index 0000000..1a66a33 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/HideCursorMode.cs @@ -0,0 +1,9 @@ +namespace Ryujinx.Common.Configuration +{ + public enum HideCursorMode + { + Never, + OnIdle, + Always, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/MemoryManagerMode.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/MemoryManagerMode.cs new file mode 100644 index 0000000..6fc69a8 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/MemoryManagerMode.cs @@ -0,0 +1,12 @@ +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration +{ + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum MemoryManagerMode : byte + { + SoftwarePageTable, + HostMapped, + HostMappedUnsafe, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Mod.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Mod.cs new file mode 100644 index 0000000..7b80642 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Mod.cs @@ -0,0 +1,9 @@ +namespace Ryujinx.Common.Configuration +{ + public class Mod + { + public string Name { get; set; } + public string Path { get; set; } + public bool Enabled { get; set; } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/ModMetadata.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/ModMetadata.cs new file mode 100644 index 0000000..527abb3 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/ModMetadata.cs @@ -0,0 +1,14 @@ +using System.Collections.Generic; + +namespace Ryujinx.Common.Configuration +{ + public struct ModMetadata + { + public List Mods { get; set; } + + public ModMetadata() + { + Mods = []; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/ModMetadataJsonSerializerContext.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/ModMetadataJsonSerializerContext.cs new file mode 100644 index 0000000..5037481 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/ModMetadataJsonSerializerContext.cs @@ -0,0 +1,10 @@ +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration +{ + [JsonSourceGenerationOptions(WriteIndented = true)] + [JsonSerializable(typeof(ModMetadata))] + public partial class ModMetadataJsonSerializerContext : JsonSerializerContext + { + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Multiplayer/MultiplayerMode.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Multiplayer/MultiplayerMode.cs new file mode 100644 index 0000000..a25377a --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/Multiplayer/MultiplayerMode.cs @@ -0,0 +1,9 @@ +namespace Ryujinx.Common.Configuration.Multiplayer +{ + public enum MultiplayerMode + { + Disabled, + LdnRyu, + LdnMitm, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/ScalingFilter.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/ScalingFilter.cs new file mode 100644 index 0000000..d53fb4d --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/ScalingFilter.cs @@ -0,0 +1,13 @@ +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration +{ + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum ScalingFilter + { + Bilinear, + Nearest, + Fsr, + Area, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/TitleUpdateMetadata.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/TitleUpdateMetadata.cs new file mode 100644 index 0000000..b3994b1 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/TitleUpdateMetadata.cs @@ -0,0 +1,10 @@ +using System.Collections.Generic; + +namespace Ryujinx.Common.Configuration +{ + public struct TitleUpdateMetadata + { + public string Selected { get; set; } + public List Paths { get; set; } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/TitleUpdateMetadataJsonSerializerContext.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/TitleUpdateMetadataJsonSerializerContext.cs new file mode 100644 index 0000000..2c12460 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/TitleUpdateMetadataJsonSerializerContext.cs @@ -0,0 +1,10 @@ +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Configuration +{ + [JsonSourceGenerationOptions(WriteIndented = true)] + [JsonSerializable(typeof(TitleUpdateMetadata))] + public partial class TitleUpdateMetadataJsonSerializerContext : JsonSerializerContext + { + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/VSyncMode.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/VSyncMode.cs new file mode 100644 index 0000000..3f2d697 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Configuration/VSyncMode.cs @@ -0,0 +1,21 @@ +namespace Ryujinx.Common.Configuration +{ + public enum VSyncMode + { + Switch, + Unbounded, + Custom + } + + public static class VSyncModeExtensions + { + public static VSyncMode Next(this VSyncMode vsync, bool customEnabled = false) => + vsync switch + { + VSyncMode.Switch => customEnabled ? VSyncMode.Custom : VSyncMode.Unbounded, + VSyncMode.Unbounded => VSyncMode.Switch, + VSyncMode.Custom => VSyncMode.Unbounded, + _ => VSyncMode.Switch + }; + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Extensions/BinaryReaderExtensions.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Extensions/BinaryReaderExtensions.cs new file mode 100644 index 0000000..587090d --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Extensions/BinaryReaderExtensions.cs @@ -0,0 +1,14 @@ +using System.IO; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; + +namespace Ryujinx.Common +{ + public static class BinaryReaderExtensions + { + public static T ReadStruct(this BinaryReader reader) where T : unmanaged + { + return MemoryMarshal.Cast(reader.ReadBytes(Unsafe.SizeOf()))[0]; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Extensions/BinaryWriterExtensions.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Extensions/BinaryWriterExtensions.cs new file mode 100644 index 0000000..97aabaa --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Extensions/BinaryWriterExtensions.cs @@ -0,0 +1,27 @@ +using System; +using System.IO; +using System.Runtime.InteropServices; + +namespace Ryujinx.Common +{ + public static class BinaryWriterExtensions + { + public static void WriteStruct(this BinaryWriter writer, T value) where T : unmanaged + { + ReadOnlySpan data = MemoryMarshal.Cast(MemoryMarshal.CreateReadOnlySpan(ref value, 1)); + + writer.Write(data); + } + + public static void Write(this BinaryWriter writer, UInt128 value) + { + writer.Write((ulong)value); + writer.Write((ulong)(value >> 64)); + } + + public static void Write(this BinaryWriter writer, MemoryStream stream) + { + stream.CopyTo(writer.BaseStream); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Extensions/SequenceReaderExtensions.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Extensions/SequenceReaderExtensions.cs new file mode 100644 index 0000000..0e4f001 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Extensions/SequenceReaderExtensions.cs @@ -0,0 +1,182 @@ +using System; +using System.Buffers; +using System.Diagnostics; +using System.IO; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; + +namespace Ryujinx.Common.Extensions +{ + public static class SequenceReaderExtensions + { + /// + /// Dumps the entire to a file, restoring its previous location afterward. + /// Useful for debugging purposes. + /// + /// The to write to a file + /// The path and name of the file to create and dump to + public static void DumpToFile(this ref SequenceReader reader, string fileFullName) + { + long initialConsumed = reader.Consumed; + + reader.Rewind(initialConsumed); + + using (FileStream fileStream = System.IO.File.Create(fileFullName, 4096, System.IO.FileOptions.None)) + { + while (reader.End == false) + { + ReadOnlySpan span = reader.CurrentSpan; + fileStream.Write(span); + reader.Advance(span.Length); + } + } + + reader.SetConsumed(initialConsumed); + } + + /// + /// Returns a reference to the desired value. This ref should always be used. The argument passed in should never be used, as this is only used for storage if the value + /// must be copied from multiple segments held by the . + /// + /// Type to get + /// The to read from + /// A location used as storage if (and only if) the value to be read spans multiple segments + /// A reference to the desired value, either directly to memory in the , or to if it has been used for copying the value in to + /// + /// DO NOT use after calling this method, as it will only + /// contain a value if the value couldn't be referenced directly because it spans multiple segments. + /// To discourage use, it is recommended to call this method like the following: + /// + /// ref readonly MyStruct value = ref sequenceReader.GetRefOrRefToCopy{MyStruct}(out _); + /// + /// + /// The does not contain enough data to read a value of type + public static ref readonly T GetRefOrRefToCopy(this scoped ref SequenceReader reader, out T copyDestinationIfRequiredDoNotUse) where T : unmanaged + { + int lengthRequired = Unsafe.SizeOf(); + + ReadOnlySpan span = reader.UnreadSpan; + if (lengthRequired <= span.Length) + { + reader.Advance(lengthRequired); + + copyDestinationIfRequiredDoNotUse = default; + + ReadOnlySpan spanOfT = MemoryMarshal.Cast(span); + + return ref spanOfT[0]; + } + else + { + copyDestinationIfRequiredDoNotUse = default; + + Span valueSpan = MemoryMarshal.CreateSpan(ref copyDestinationIfRequiredDoNotUse, 1); + + Span valueBytesSpan = MemoryMarshal.AsBytes(valueSpan); + + if (!reader.TryCopyTo(valueBytesSpan)) + { + throw new ArgumentOutOfRangeException(nameof(reader), "The sequence is not long enough to read the desired value."); + } + + reader.Advance(lengthRequired); + + return ref valueSpan[0]; + } + } + + /// + /// Reads an as little endian. + /// + /// The to read from + /// A location to receive the read value + /// Thrown if there wasn't enough data for an + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void ReadLittleEndian(this ref SequenceReader reader, out int value) + { + if (!reader.TryReadLittleEndian(out value)) + { + throw new ArgumentOutOfRangeException(nameof(value), "The sequence is not long enough to read the desired value."); + } + } + + /// + /// Reads the desired unmanaged value by copying it to the specified . + /// + /// Type to read + /// The to read from + /// The target that will receive the read value + /// The does not contain enough data to read a value of type + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void ReadUnmanaged(this ref SequenceReader reader, out T value) where T : unmanaged + { + if (!reader.TryReadUnmanaged(out value)) + { + throw new ArgumentOutOfRangeException(nameof(value), "The sequence is not long enough to read the desired value."); + } + } + + /// + /// Sets the reader's position as bytes consumed. + /// + /// The to set the position + /// The number of bytes consumed + public static void SetConsumed(ref this SequenceReader reader, long consumed) + { + reader.Rewind(reader.Consumed); + reader.Advance(consumed); + } + + /// + /// Try to read the given type out of the buffer if possible. Warning: this is dangerous to use with arbitrary + /// structs - see remarks for full details. + /// + /// Type to read + /// + /// IMPORTANT: The read is a straight copy of bits. If a struct depends on specific state of it's members to + /// behave correctly this can lead to exceptions, etc. If reading endian specific integers, use the explicit + /// overloads such as + /// + /// + /// True if successful. will be default if failed (due to lack of space). + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static unsafe bool TryReadUnmanaged(ref this SequenceReader reader, out T value) where T : unmanaged + { + ReadOnlySpan span = reader.UnreadSpan; + + if (span.Length < sizeof(T)) + { + return TryReadUnmanagedMultiSegment(ref reader, out value); + } + + value = Unsafe.ReadUnaligned(ref MemoryMarshal.GetReference(span)); + + reader.Advance(sizeof(T)); + + return true; + } + + private static unsafe bool TryReadUnmanagedMultiSegment(ref SequenceReader reader, out T value) where T : unmanaged + { + Debug.Assert(reader.UnreadSpan.Length < sizeof(T)); + + // Not enough data in the current segment, try to peek for the data we need. + T buffer = default; + + Span tempSpan = new(&buffer, sizeof(T)); + + if (!reader.TryCopyTo(tempSpan)) + { + value = default; + return false; + } + + value = Unsafe.ReadUnaligned(ref MemoryMarshal.GetReference(tempSpan)); + + reader.Advance(sizeof(T)); + + return true; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Extensions/StreamExtensions.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Extensions/StreamExtensions.cs new file mode 100644 index 0000000..f21360b --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Extensions/StreamExtensions.cs @@ -0,0 +1,138 @@ +using System; +using System.Buffers.Binary; +using System.IO; +using System.Runtime.InteropServices; + +namespace Ryujinx.Common +{ + public static class StreamExtensions + { + /// + /// Writes an int span to this stream. + /// + /// This default implementation converts each buffer value to a stack-allocated + /// byte array, then writes it to the Stream using . + /// + /// The stream to be written to + /// The buffer of values to be written + public static void Write(this Stream stream, ReadOnlySpan buffer) + { + if (buffer.Length == 0) + { + return; + } + + if (BitConverter.IsLittleEndian) + { + ReadOnlySpan byteBuffer = MemoryMarshal.Cast(buffer); + stream.Write(byteBuffer); + } + else + { + Span byteBuffer = stackalloc byte[sizeof(int)]; + + foreach (int value in buffer) + { + BinaryPrimitives.WriteInt32LittleEndian(byteBuffer, value); + stream.Write(byteBuffer); + } + } + } + + /// + /// Writes a four-byte signed integer to this stream. The current position + /// of the stream is advanced by four. + /// + /// The stream to be written to + /// The value to be written + public static void Write(this Stream stream, int value) + { + Span buffer = stackalloc byte[sizeof(int)]; + BinaryPrimitives.WriteInt32LittleEndian(buffer, value); + stream.Write(buffer); + } + + /// + /// Writes an eight-byte signed integer to this stream. The current position + /// of the stream is advanced by eight. + /// + /// The stream to be written to + /// The value to be written + public static void Write(this Stream stream, long value) + { + Span buffer = stackalloc byte[sizeof(long)]; + BinaryPrimitives.WriteInt64LittleEndian(buffer, value); + stream.Write(buffer); + } + + /// + /// Writes a four-byte unsigned integer to this stream. The current position + /// of the stream is advanced by four. + /// + /// The stream to be written to + /// The value to be written + public static void Write(this Stream stream, uint value) + { + Span buffer = stackalloc byte[sizeof(uint)]; + BinaryPrimitives.WriteUInt32LittleEndian(buffer, value); + stream.Write(buffer); + } + + /// + /// Writes an eight-byte unsigned integer to this stream. The current + /// position of the stream is advanced by eight. + /// + /// The stream to be written to + /// The value to be written + public static void Write(this Stream stream, ulong value) + { + Span buffer = stackalloc byte[sizeof(ulong)]; + BinaryPrimitives.WriteUInt64LittleEndian(buffer, value); + stream.Write(buffer); + } + + /// + /// Writes the contents of source to stream by calling source.CopyTo(stream). + /// Provides consistency with other Stream.Write methods. + /// + /// The stream to be written to + /// The stream to be read from + public static void Write(this Stream stream, Stream source) + { + source.CopyTo(stream); + } + + /// + /// Writes a sequence of bytes to the Stream. + /// + /// The stream to be written to. + /// The byte to be written + /// The number of times the value should be written + public static void WriteByte(this Stream stream, byte value, int count) + { + if (count <= 0) + { + return; + } + + const int BlockSize = 16; + + int blockCount = count / BlockSize; + if (blockCount > 0) + { + Span span = stackalloc byte[BlockSize]; + span.Fill(value); + for (int x = 0; x < blockCount; x++) + { + stream.Write(span); + } + } + + int nonBlockBytes = count % BlockSize; + for (int x = 0; x < nonBlockBytes; x++) + { + stream.WriteByte(value); + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/GraphicsDriver/DriverUtilities.cs b/third_party/maxwell_shader/src/Ryujinx.Common/GraphicsDriver/DriverUtilities.cs new file mode 100644 index 0000000..6f74de4 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/GraphicsDriver/DriverUtilities.cs @@ -0,0 +1,42 @@ +using Ryujinx.Common.Utilities; +using System; + +namespace Ryujinx.Common.GraphicsDriver +{ + public static class DriverUtilities + { + private static void AddMesaFlags(string envVar, string newFlags) + { + string existingFlags = Environment.GetEnvironmentVariable(envVar); + + string flags = existingFlags == null ? newFlags : $"{existingFlags},{newFlags}"; + + OsUtils.SetEnvironmentVariableNoCaching(envVar, flags); + } + + public static void InitDriverConfig(bool oglThreading) + { + if (OperatingSystem.IsLinux()) + { + AddMesaFlags("RADV_DEBUG", "nodcc"); + } + + ToggleOGLThreading(oglThreading); + } + + public static void ToggleOGLThreading(bool enabled) + { + OsUtils.SetEnvironmentVariableNoCaching("mesa_glthread", enabled.ToString().ToLower()); + OsUtils.SetEnvironmentVariableNoCaching("__GL_THREADED_OPTIMIZATIONS", enabled ? "1" : "0"); + + try + { + NVThreadedOptimization.SetThreadedOptimization(enabled); + } + catch + { + // NVAPI is not available, or couldn't change the application profile. + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/GraphicsDriver/NVAPI/Nvapi.cs b/third_party/maxwell_shader/src/Ryujinx.Common/GraphicsDriver/NVAPI/Nvapi.cs new file mode 100644 index 0000000..9dd82b2 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/GraphicsDriver/NVAPI/Nvapi.cs @@ -0,0 +1,11 @@ +namespace Ryujinx.Common.GraphicsDriver.NVAPI +{ + enum Nvapi : uint + { + OglThreadControlId = 0x20C1221E, + + OglThreadControlDefault = 0, + OglThreadControlEnable = 1, + OglThreadControlDisable = 2, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/GraphicsDriver/NVAPI/NvapiUnicodeString.cs b/third_party/maxwell_shader/src/Ryujinx.Common/GraphicsDriver/NVAPI/NvapiUnicodeString.cs new file mode 100644 index 0000000..6faf557 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/GraphicsDriver/NVAPI/NvapiUnicodeString.cs @@ -0,0 +1,42 @@ +using System.Runtime.InteropServices; +using System.Text; + +namespace Ryujinx.Common.GraphicsDriver.NVAPI +{ + [StructLayout(LayoutKind.Sequential, Pack = 4)] + public unsafe struct NvapiUnicodeString + { + private fixed byte _data[4096]; + + public NvapiUnicodeString(string text) + { + Set(text); + } + + public readonly string Get() + { + fixed (byte* data = _data) + { + string text = Encoding.Unicode.GetString(data, 4096); + + int index = text.IndexOf('\0'); + if (index > -1) + { + text = text[..index]; + } + + return text; + } + } + + public readonly void Set(string text) + { + text += '\0'; + fixed (char* textPtr = text) + fixed (byte* data = _data) + { + int written = Encoding.Unicode.GetBytes(textPtr, text.Length, data, 4096); + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/GraphicsDriver/NVAPI/NvdrsApplicationV4.cs b/third_party/maxwell_shader/src/Ryujinx.Common/GraphicsDriver/NVAPI/NvdrsApplicationV4.cs new file mode 100644 index 0000000..fcaf5b6 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/GraphicsDriver/NVAPI/NvdrsApplicationV4.cs @@ -0,0 +1,17 @@ +using System.Runtime.InteropServices; + +namespace Ryujinx.Common.GraphicsDriver.NVAPI +{ + [StructLayout(LayoutKind.Sequential, Pack = 4)] + struct NvdrsApplicationV4 + { + public uint Version; + public uint IsPredefined; + public NvapiUnicodeString AppName; + public NvapiUnicodeString UserFriendlyName; + public NvapiUnicodeString Launcher; + public NvapiUnicodeString FileInFolder; + public uint Flags; + public NvapiUnicodeString CommandLine; + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/GraphicsDriver/NVAPI/NvdrsProfile.cs b/third_party/maxwell_shader/src/Ryujinx.Common/GraphicsDriver/NVAPI/NvdrsProfile.cs new file mode 100644 index 0000000..3849273 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/GraphicsDriver/NVAPI/NvdrsProfile.cs @@ -0,0 +1,15 @@ +using System.Runtime.InteropServices; + +namespace Ryujinx.Common.GraphicsDriver.NVAPI +{ + [StructLayout(LayoutKind.Sequential, Pack = 1)] + struct NvdrsProfile + { + public uint Version; + public NvapiUnicodeString ProfileName; + public uint GpuSupport; + public uint IsPredefined; + public uint NumOfApps; + public uint NumOfSettings; + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/GraphicsDriver/NVAPI/NvdrsSetting.cs b/third_party/maxwell_shader/src/Ryujinx.Common/GraphicsDriver/NVAPI/NvdrsSetting.cs new file mode 100644 index 0000000..5b1bd30 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/GraphicsDriver/NVAPI/NvdrsSetting.cs @@ -0,0 +1,49 @@ +using System.Runtime.InteropServices; + +namespace Ryujinx.Common.GraphicsDriver.NVAPI +{ + enum NvdrsSettingType : uint + { + NvdrsDwordType, + NvdrsBinaryType, + NvdrsStringType, + NvdrsWstringType, + } + + enum NvdrsSettingLocation : uint + { + NvdrsCurrentProfileLocation, + NvdrsGlobalProfileLocation, + NvdrsBaseProfileLocation, + NvdrsDefaultProfileLocation, + } + + [StructLayout(LayoutKind.Explicit, Size = 0x3020)] + struct NvdrsSetting + { + [FieldOffset(0x0)] + public uint Version; + [FieldOffset(0x4)] + public NvapiUnicodeString SettingName; + [FieldOffset(0x1004)] + public Nvapi SettingId; + [FieldOffset(0x1008)] + public NvdrsSettingType SettingType; + [FieldOffset(0x100C)] + public NvdrsSettingLocation SettingLocation; + [FieldOffset(0x1010)] + public uint IsCurrentPredefined; + [FieldOffset(0x1014)] + public uint IsPredefinedValid; + + [FieldOffset(0x1018)] + public uint PredefinedValue; + [FieldOffset(0x1018)] + public NvapiUnicodeString PredefinedString; + + [FieldOffset(0x201C)] + public uint CurrentValue; + [FieldOffset(0x201C)] + public NvapiUnicodeString CurrentString; + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/GraphicsDriver/NVThreadedOptimization.cs b/third_party/maxwell_shader/src/Ryujinx.Common/GraphicsDriver/NVThreadedOptimization.cs new file mode 100644 index 0000000..05d2a7a --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/GraphicsDriver/NVThreadedOptimization.cs @@ -0,0 +1,161 @@ +using Ryujinx.Common.GraphicsDriver.NVAPI; +using System; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +// ReSharper disable InconsistentNaming + +namespace Ryujinx.Common.GraphicsDriver +{ + static partial class NVThreadedOptimization + { + private const string ProfileName = "Ryujinx Nvidia Profile"; + + private const uint NvAPI_Initialize_ID = 0x0150E828; + private const uint NvAPI_DRS_CreateSession_ID = 0x0694D52E; + private const uint NvAPI_DRS_LoadSettings_ID = 0x375DBD6B; + private const uint NvAPI_DRS_FindProfileByName_ID = 0x7E4A9A0B; + private const uint NvAPI_DRS_CreateProfile_ID = 0x0CC176068; + private const uint NvAPI_DRS_CreateApplication_ID = 0x4347A9DE; + private const uint NvAPI_DRS_SetSetting_ID = 0x577DD202; + private const uint NvAPI_DRS_SaveSettings_ID = 0xFCBC7E14; + private const uint NvAPI_DRS_DestroySession_ID = 0x0DAD9CFF8; + + [LibraryImport("nvapi64")] + private static partial nint nvapi_QueryInterface(uint id); + + private delegate int NvAPI_InitializeDelegate(); + private static NvAPI_InitializeDelegate NvAPI_Initialize; + + private delegate int NvAPI_DRS_CreateSessionDelegate(out nint handle); + private static NvAPI_DRS_CreateSessionDelegate NvAPI_DRS_CreateSession; + + private delegate int NvAPI_DRS_LoadSettingsDelegate(nint handle); + private static NvAPI_DRS_LoadSettingsDelegate NvAPI_DRS_LoadSettings; + + private delegate int NvAPI_DRS_FindProfileByNameDelegate(nint handle, NvapiUnicodeString profileName, out nint profileHandle); + private static NvAPI_DRS_FindProfileByNameDelegate NvAPI_DRS_FindProfileByName; + + private delegate int NvAPI_DRS_CreateProfileDelegate(nint handle, ref NvdrsProfile profileInfo, out nint profileHandle); + private static NvAPI_DRS_CreateProfileDelegate NvAPI_DRS_CreateProfile; + + private delegate int NvAPI_DRS_CreateApplicationDelegate(nint handle, nint profileHandle, ref NvdrsApplicationV4 app); + private static NvAPI_DRS_CreateApplicationDelegate NvAPI_DRS_CreateApplication; + + private delegate int NvAPI_DRS_SetSettingDelegate(nint handle, nint profileHandle, ref NvdrsSetting setting); + private static NvAPI_DRS_SetSettingDelegate NvAPI_DRS_SetSetting; + + private delegate int NvAPI_DRS_SaveSettingsDelegate(nint handle); + private static NvAPI_DRS_SaveSettingsDelegate NvAPI_DRS_SaveSettings; + + private delegate int NvAPI_DRS_DestroySessionDelegate(nint handle); + private static NvAPI_DRS_DestroySessionDelegate NvAPI_DRS_DestroySession; + + private static bool _initialized; + + private static void Check(int status) + { + if (status != 0) + { + throw new Exception($"NVAPI Error: {status}"); + } + } + + private static void Initialize() + { + if (!_initialized) + { + NvAPI_Initialize = NvAPI_Delegate(NvAPI_Initialize_ID); + + Check(NvAPI_Initialize()); + + NvAPI_DRS_CreateSession = NvAPI_Delegate(NvAPI_DRS_CreateSession_ID); + NvAPI_DRS_LoadSettings = NvAPI_Delegate(NvAPI_DRS_LoadSettings_ID); + NvAPI_DRS_FindProfileByName = NvAPI_Delegate(NvAPI_DRS_FindProfileByName_ID); + NvAPI_DRS_CreateProfile = NvAPI_Delegate(NvAPI_DRS_CreateProfile_ID); + NvAPI_DRS_CreateApplication = NvAPI_Delegate(NvAPI_DRS_CreateApplication_ID); + NvAPI_DRS_SetSetting = NvAPI_Delegate(NvAPI_DRS_SetSetting_ID); + NvAPI_DRS_SaveSettings = NvAPI_Delegate(NvAPI_DRS_SaveSettings_ID); + NvAPI_DRS_DestroySession = NvAPI_Delegate(NvAPI_DRS_DestroySession_ID); + + _initialized = true; + } + } + + private static uint MakeVersion(uint version) where T : unmanaged + { + return (uint)Unsafe.SizeOf() | version << 16; + } + + public static void SetThreadedOptimization(bool enabled) + { + Initialize(); + + uint targetValue = (uint)(enabled ? Nvapi.OglThreadControlEnable : Nvapi.OglThreadControlDisable); + + Check(NvAPI_Initialize()); + + Check(NvAPI_DRS_CreateSession(out nint handle)); + + Check(NvAPI_DRS_LoadSettings(handle)); + + // Check if the profile already exists. + + int status = NvAPI_DRS_FindProfileByName(handle, new NvapiUnicodeString(ProfileName), out nint profileHandle); + + if (status != 0) + { + NvdrsProfile profile = new() + { + Version = MakeVersion(1), + IsPredefined = 0, + GpuSupport = uint.MaxValue, + }; + profile.ProfileName.Set(ProfileName); + Check(NvAPI_DRS_CreateProfile(handle, ref profile, out profileHandle)); + + NvdrsApplicationV4 application = new() + { + Version = MakeVersion(4), + IsPredefined = 0, + Flags = 3, // IsMetro, IsCommandLine + }; + application.AppName.Set("Ryujinx.exe"); + application.UserFriendlyName.Set("Ryujinx"); + application.Launcher.Set(string.Empty); + application.FileInFolder.Set(string.Empty); + + Check(NvAPI_DRS_CreateApplication(handle, profileHandle, ref application)); + } + + NvdrsSetting setting = new() + { + Version = MakeVersion(1), + SettingId = Nvapi.OglThreadControlId, + SettingType = NvdrsSettingType.NvdrsDwordType, + SettingLocation = NvdrsSettingLocation.NvdrsCurrentProfileLocation, + IsCurrentPredefined = 0, + IsPredefinedValid = 0, + CurrentValue = targetValue, + PredefinedValue = targetValue, + }; + + Check(NvAPI_DRS_SetSetting(handle, profileHandle, ref setting)); + + Check(NvAPI_DRS_SaveSettings(handle)); + + NvAPI_DRS_DestroySession(handle); + } + + private static T NvAPI_Delegate(uint id) where T : class + { + nint ptr = nvapi_QueryInterface(id); + + if (ptr != nint.Zero) + { + return Marshal.GetDelegateForFunctionPointer(ptr); + } + + return null; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Hash128.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Hash128.cs new file mode 100644 index 0000000..55cb9ae --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Hash128.cs @@ -0,0 +1,739 @@ +using System; +using System.Buffers.Binary; +using System.Diagnostics; +using System.Numerics; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using System.Runtime.Intrinsics; +using System.Runtime.Intrinsics.X86; +// ReSharper disable InconsistentNaming + +namespace Ryujinx.Common +{ + [StructLayout(LayoutKind.Sequential)] + public struct Hash128(ulong low, ulong high) : IEquatable + { + public ulong Low = low; + public ulong High = high; + + public readonly override string ToString() => $"{High:x16}{Low:x16}"; + + public static bool operator ==(Hash128 x, Hash128 y) => x.Equals(y); + + public static bool operator !=(Hash128 x, Hash128 y) => !x.Equals(y); + + public readonly override bool Equals(object obj) => obj is Hash128 hash128 && Equals(hash128); + + public readonly bool Equals(Hash128 cmpObj) => Low == cmpObj.Low && High == cmpObj.High; + + public readonly override int GetHashCode() => HashCode.Combine(Low, High); + + public static Hash128 ComputeHash(ReadOnlySpan input) => Xxh3128bitsInternal(input, Xxh3KSecret, 0UL); + + #region Hash computation + + private const int StripeLen = 64; + private const int AccNb = StripeLen / sizeof(ulong); + private const int SecretConsumeRate = 8; + private const int SecretLastAccStart = 7; + private const int SecretMergeAccsStart = 11; + private const int SecretSizeMin = 136; + private const int MidSizeStartOffset = 3; + private const int MidSizeLastOffset = 17; + + private const uint Prime32_1 = 0x9E3779B1U; + private const uint Prime32_2 = 0x85EBCA77U; + private const uint Prime32_3 = 0xC2B2AE3DU; + private const uint Prime32_4 = 0x27D4EB2FU; + private const uint Prime32_5 = 0x165667B1U; + + private const ulong Prime64_1 = 0x9E3779B185EBCA87UL; + private const ulong Prime64_2 = 0xC2B2AE3D27D4EB4FUL; + private const ulong Prime64_3 = 0x165667B19E3779F9UL; + private const ulong Prime64_4 = 0x85EBCA77C2B2AE63UL; + private const ulong Prime64_5 = 0x27D4EB2F165667C5UL; + + private static readonly ulong[] _xxh3InitAcc = + [ + Prime32_3, + Prime64_1, + Prime64_2, + Prime64_3, + Prime64_4, + Prime32_2, + Prime64_5, + Prime32_1 + ]; + + private static ReadOnlySpan Xxh3KSecret => + [ + 0xb8, + 0xfe, + 0x6c, + 0x39, + 0x23, + 0xa4, + 0x4b, + 0xbe, + 0x7c, + 0x01, + 0x81, + 0x2c, + 0xf7, + 0x21, + 0xad, + 0x1c, + 0xde, + 0xd4, + 0x6d, + 0xe9, + 0x83, + 0x90, + 0x97, + 0xdb, + 0x72, + 0x40, + 0xa4, + 0xa4, + 0xb7, + 0xb3, + 0x67, + 0x1f, + 0xcb, + 0x79, + 0xe6, + 0x4e, + 0xcc, + 0xc0, + 0xe5, + 0x78, + 0x82, + 0x5a, + 0xd0, + 0x7d, + 0xcc, + 0xff, + 0x72, + 0x21, + 0xb8, + 0x08, + 0x46, + 0x74, + 0xf7, + 0x43, + 0x24, + 0x8e, + 0xe0, + 0x35, + 0x90, + 0xe6, + 0x81, + 0x3a, + 0x26, + 0x4c, + 0x3c, + 0x28, + 0x52, + 0xbb, + 0x91, + 0xc3, + 0x00, + 0xcb, + 0x88, + 0xd0, + 0x65, + 0x8b, + 0x1b, + 0x53, + 0x2e, + 0xa3, + 0x71, + 0x64, + 0x48, + 0x97, + 0xa2, + 0x0d, + 0xf9, + 0x4e, + 0x38, + 0x19, + 0xef, + 0x46, + 0xa9, + 0xde, + 0xac, + 0xd8, + 0xa8, + 0xfa, + 0x76, + 0x3f, + 0xe3, + 0x9c, + 0x34, + 0x3f, + 0xf9, + 0xdc, + 0xbb, + 0xc7, + 0xc7, + 0x0b, + 0x4f, + 0x1d, + 0x8a, + 0x51, + 0xe0, + 0x4b, + 0xcd, + 0xb4, + 0x59, + 0x31, + 0xc8, + 0x9f, + 0x7e, + 0xc9, + 0xd9, + 0x78, + 0x73, + 0x64, + 0xea, + 0xc5, + 0xac, + 0x83, + 0x34, + 0xd3, + 0xeb, + 0xc3, + 0xc5, + 0x81, + 0xa0, + 0xff, + 0xfa, + 0x13, + 0x63, + 0xeb, + 0x17, + 0x0d, + 0xdd, + 0x51, + 0xb7, + 0xf0, + 0xda, + 0x49, + 0xd3, + 0x16, + 0x55, + 0x26, + 0x29, + 0xd4, + 0x68, + 0x9e, + 0x2b, + 0x16, + 0xbe, + 0x58, + 0x7d, + 0x47, + 0xa1, + 0xfc, + 0x8f, + 0xf8, + 0xb8, + 0xd1, + 0x7a, + 0xd0, + 0x31, + 0xce, + 0x45, + 0xcb, + 0x3a, + 0x8f, + 0x95, + 0x16, + 0x04, + 0x28, + 0xaf, + 0xd7, + 0xfb, + 0xca, + 0xbb, + 0x4b, + 0x40, + 0x7e + ]; + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong Mult32To64(ulong x, ulong y) => (uint)x * (ulong)(uint)y; + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Hash128 Mult64To128(ulong lhs, ulong rhs) + { + ulong high = Math.BigMul(lhs, rhs, out ulong low); + + return new Hash128 + { + Low = low, + High = high, + }; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong Mul128Fold64(ulong lhs, ulong rhs) + { + Hash128 product = Mult64To128(lhs, rhs); + + return product.Low ^ product.High; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong XorShift64(ulong v64, int shift) + { + Debug.Assert(shift is >= 0 and < 64); + + return v64 ^ (v64 >> shift); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong Xxh3Avalanche(ulong h64) + { + h64 = XorShift64(h64, 37); + h64 *= 0x165667919E3779F9UL; + h64 = XorShift64(h64, 32); + + return h64; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong Xxh64Avalanche(ulong h64) + { + h64 ^= h64 >> 33; + h64 *= Prime64_2; + h64 ^= h64 >> 29; + h64 *= Prime64_3; + h64 ^= h64 >> 32; + + return h64; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private unsafe static void Xxh3Accumulate512(Span acc, ReadOnlySpan input, ReadOnlySpan secret) + { + if (Avx2.IsSupported) + { + fixed (ulong* pAcc = acc) + { + fixed (byte* pInput = input, pSecret = secret) + { + Vector256* xAcc = (Vector256*)pAcc; + Vector256* xInput = (Vector256*)pInput; + Vector256* xSecret = (Vector256*)pSecret; + + for (ulong i = 0; i < StripeLen / 32; i++) + { + Vector256 dataVec = xInput[i]; + Vector256 keyVec = xSecret[i]; + Vector256 dataKey = Avx2.Xor(dataVec, keyVec); + Vector256 dataKeyLo = Avx2.Shuffle(dataKey.AsUInt32(), 0b00110001); + Vector256 product = Avx2.Multiply(dataKey.AsUInt32(), dataKeyLo); + Vector256 dataSwap = Avx2.Shuffle(dataVec.AsUInt32(), 0b01001110); + Vector256 sum = Avx2.Add(xAcc[i], dataSwap.AsUInt64()); + xAcc[i] = Avx2.Add(product, sum); + } + } + } + } + else if (Sse2.IsSupported) + { + fixed (ulong* pAcc = acc) + { + fixed (byte* pInput = input, pSecret = secret) + { + Vector128* xAcc = (Vector128*)pAcc; + Vector128* xInput = (Vector128*)pInput; + Vector128* xSecret = (Vector128*)pSecret; + + for (ulong i = 0; i < StripeLen / 16; i++) + { + Vector128 dataVec = xInput[i]; + Vector128 keyVec = xSecret[i]; + Vector128 dataKey = Sse2.Xor(dataVec, keyVec); + Vector128 dataKeyLo = Sse2.Shuffle(dataKey.AsUInt32(), 0b00110001); + Vector128 product = Sse2.Multiply(dataKey.AsUInt32(), dataKeyLo); + Vector128 dataSwap = Sse2.Shuffle(dataVec.AsUInt32(), 0b01001110); + Vector128 sum = Sse2.Add(xAcc[i], dataSwap.AsUInt64()); + xAcc[i] = Sse2.Add(product, sum); + } + } + } + } + else + { + for (int i = 0; i < AccNb; i++) + { + ulong dataVal = BinaryPrimitives.ReadUInt64LittleEndian(input[(i * sizeof(ulong))..]); + ulong dataKey = dataVal ^ BinaryPrimitives.ReadUInt64LittleEndian(secret[(i * sizeof(ulong))..]); + acc[i ^ 1] += dataVal; + acc[i] += Mult32To64((uint)dataKey, dataKey >> 32); + } + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private unsafe static void Xxh3ScrambleAcc(Span acc, ReadOnlySpan secret) + { + if (Avx2.IsSupported) + { + fixed (ulong* pAcc = acc) + { + fixed (byte* pSecret = secret) + { + Vector256 prime32 = Vector256.Create(Prime32_1); + Vector256* xAcc = (Vector256*)pAcc; + Vector256* xSecret = (Vector256*)pSecret; + + for (ulong i = 0; i < StripeLen / 32; i++) + { + Vector256 accVec = xAcc[i]; + Vector256 shifted = Avx2.ShiftRightLogical(accVec, 47); + Vector256 dataVec = Avx2.Xor(accVec, shifted); + + Vector256 keyVec = xSecret[i]; + Vector256 dataKey = Avx2.Xor(dataVec.AsUInt32(), keyVec.AsUInt32()); + + Vector256 dataKeyHi = Avx2.Shuffle(dataKey.AsUInt32(), 0b00110001); + Vector256 prodLo = Avx2.Multiply(dataKey, prime32); + Vector256 prodHi = Avx2.Multiply(dataKeyHi, prime32); + + xAcc[i] = Avx2.Add(prodLo, Avx2.ShiftLeftLogical(prodHi, 32)); + } + } + } + } + else if (Sse2.IsSupported) + { + fixed (ulong* pAcc = acc) + { + fixed (byte* pSecret = secret) + { + Vector128 prime32 = Vector128.Create(Prime32_1); + Vector128* xAcc = (Vector128*)pAcc; + Vector128* xSecret = (Vector128*)pSecret; + + for (ulong i = 0; i < StripeLen / 16; i++) + { + Vector128 accVec = xAcc[i]; + Vector128 shifted = Sse2.ShiftRightLogical(accVec, 47); + Vector128 dataVec = Sse2.Xor(accVec, shifted); + + Vector128 keyVec = xSecret[i]; + Vector128 dataKey = Sse2.Xor(dataVec.AsUInt32(), keyVec.AsUInt32()); + + Vector128 dataKeyHi = Sse2.Shuffle(dataKey.AsUInt32(), 0b00110001); + Vector128 prodLo = Sse2.Multiply(dataKey, prime32); + Vector128 prodHi = Sse2.Multiply(dataKeyHi, prime32); + + xAcc[i] = Sse2.Add(prodLo, Sse2.ShiftLeftLogical(prodHi, 32)); + } + } + } + } + else + { + for (int i = 0; i < AccNb; i++) + { + ulong key64 = BinaryPrimitives.ReadUInt64LittleEndian(secret[(i * sizeof(ulong))..]); + ulong acc64 = acc[i]; + acc64 = XorShift64(acc64, 47); + acc64 ^= key64; + acc64 *= Prime32_1; + acc[i] = acc64; + } + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void Xxh3Accumulate(Span acc, ReadOnlySpan input, ReadOnlySpan secret, int nbStripes) + { + for (int n = 0; n < nbStripes; n++) + { + ReadOnlySpan inData = input[(n * StripeLen)..]; + Xxh3Accumulate512(acc, inData, secret[(n * SecretConsumeRate)..]); + } + } + + private static void Xxh3HashLongInternalLoop(Span acc, ReadOnlySpan input, ReadOnlySpan secret) + { + int nbStripesPerBlock = (secret.Length - StripeLen) / SecretConsumeRate; + int blockLen = StripeLen * nbStripesPerBlock; + int nbBlocks = (input.Length - 1) / blockLen; + + Debug.Assert(secret.Length >= SecretSizeMin); + + for (int n = 0; n < nbBlocks; n++) + { + Xxh3Accumulate(acc, input[(n * blockLen)..], secret, nbStripesPerBlock); + Xxh3ScrambleAcc(acc, secret[^StripeLen..]); + } + + Debug.Assert(input.Length > StripeLen); + + int nbStripes = (input.Length - 1 - (blockLen * nbBlocks)) / StripeLen; + Debug.Assert(nbStripes <= (secret.Length / SecretConsumeRate)); + Xxh3Accumulate(acc, input[(nbBlocks * blockLen)..], secret, nbStripes); + + ReadOnlySpan p = input[^StripeLen..]; + Xxh3Accumulate512(acc, p, secret[(secret.Length - StripeLen - SecretLastAccStart)..]); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong Xxh3Mix2Accs(Span acc, ReadOnlySpan secret) + { + return Mul128Fold64( + acc[0] ^ BinaryPrimitives.ReadUInt64LittleEndian(secret), + acc[1] ^ BinaryPrimitives.ReadUInt64LittleEndian(secret[8..])); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong Xxh3MergeAccs(Span acc, ReadOnlySpan secret, ulong start) + { + ulong result64 = start; + + for (int i = 0; i < 4; i++) + { + result64 += Xxh3Mix2Accs(acc[(2 * i)..], secret[(16 * i)..]); + } + + return Xxh3Avalanche(result64); + } + + [SkipLocalsInit] + private static Hash128 Xxh3HashLong128bInternal(ReadOnlySpan input, ReadOnlySpan secret) + { + Span acc = stackalloc ulong[AccNb]; + _xxh3InitAcc.CopyTo(acc); + + Xxh3HashLongInternalLoop(acc, input, secret); + + Debug.Assert(acc.Length == 8); + Debug.Assert(secret.Length >= acc.Length * sizeof(ulong) + SecretMergeAccsStart); + + return new Hash128 + { + Low = Xxh3MergeAccs(acc, secret[SecretMergeAccsStart..], (ulong)input.Length * Prime64_1), + High = Xxh3MergeAccs( + acc, + secret[(secret.Length - acc.Length * sizeof(ulong) - SecretMergeAccsStart)..], + ~((ulong)input.Length * Prime64_2)), + }; + } + + private static Hash128 Xxh3Len1To3128b(ReadOnlySpan input, ReadOnlySpan secret, ulong seed) + { + Debug.Assert(input.Length is >= 1 and <= 3); + + byte c1 = input[0]; + byte c2 = input[input.Length >> 1]; + byte c3 = input[^1]; + + uint combinedL = ((uint)c1 << 16) | ((uint)c2 << 24) | c3 | ((uint)input.Length << 8); + uint combinedH = BitOperations.RotateLeft(BinaryPrimitives.ReverseEndianness(combinedL), 13); + ulong bitFlipL = (BinaryPrimitives.ReadUInt32LittleEndian(secret) ^ BinaryPrimitives.ReadUInt32LittleEndian(secret[4..])) + seed; + ulong bitFlipH = (BinaryPrimitives.ReadUInt32LittleEndian(secret[8..]) ^ BinaryPrimitives.ReadUInt32LittleEndian(secret[12..])) - seed; + ulong keyedLo = combinedL ^ bitFlipL; + ulong keyedHi = combinedH ^ bitFlipH; + + return new Hash128 + { + Low = Xxh64Avalanche(keyedLo), + High = Xxh64Avalanche(keyedHi), + }; + } + + private static Hash128 Xxh3Len4To8128b(ReadOnlySpan input, ReadOnlySpan secret, ulong seed) + { + Debug.Assert(input.Length is >= 4 and <= 8); + + seed ^= BinaryPrimitives.ReverseEndianness((uint)seed) << 32; + + uint inputLo = BinaryPrimitives.ReadUInt32LittleEndian(input); + uint inputHi = BinaryPrimitives.ReadUInt32LittleEndian(input[^4..]); + ulong input64 = inputLo + ((ulong)inputHi << 32); + ulong bitFlip = (BinaryPrimitives.ReadUInt64LittleEndian(secret[16..]) ^ BinaryPrimitives.ReadUInt64LittleEndian(secret[24..])) + seed; + ulong keyed = input64 ^ bitFlip; + + Hash128 m128 = Mult64To128(keyed, Prime64_1 + ((ulong)input.Length << 2)); + + m128.High += m128.Low << 1; + m128.Low ^= m128.High >> 3; + + m128.Low = XorShift64(m128.Low, 35); + m128.Low *= 0x9FB21C651E98DF25UL; + m128.Low = XorShift64(m128.Low, 28); + m128.High = Xxh3Avalanche(m128.High); + + return m128; + } + + private static Hash128 Xxh3Len9To16128b(ReadOnlySpan input, ReadOnlySpan secret, ulong seed) + { + Debug.Assert(input.Length is >= 9 and <= 16); + + ulong bitFlipL = (BinaryPrimitives.ReadUInt64LittleEndian(secret[32..]) ^ BinaryPrimitives.ReadUInt64LittleEndian(secret[40..])) - seed; + ulong bitFlipH = (BinaryPrimitives.ReadUInt64LittleEndian(secret[48..]) ^ BinaryPrimitives.ReadUInt64LittleEndian(secret[56..])) + seed; + ulong inputLo = BinaryPrimitives.ReadUInt64LittleEndian(input); + ulong inputHi = BinaryPrimitives.ReadUInt64LittleEndian(input[^8..]); + + Hash128 m128 = Mult64To128(inputLo ^ inputHi ^ bitFlipL, Prime64_1); + m128.Low += ((ulong)input.Length - 1) << 54; + inputHi ^= bitFlipH; + m128.High += inputHi + Mult32To64((uint)inputHi, Prime32_2 - 1); + m128.Low ^= BinaryPrimitives.ReverseEndianness(m128.High); + + Hash128 h128 = Mult64To128(m128.Low, Prime64_2); + h128.High += m128.High * Prime64_2; + h128.Low = Xxh3Avalanche(h128.Low); + h128.High = Xxh3Avalanche(h128.High); + + return h128; + } + + private static Hash128 Xxh3Len0To16128b(ReadOnlySpan input, ReadOnlySpan secret, ulong seed) + { + Debug.Assert(input.Length <= 16); + + if (input.Length > 8) + { + return Xxh3Len9To16128b(input, secret, seed); + } + + if (input.Length >= 4) + { + return Xxh3Len4To8128b(input, secret, seed); + } + + if (input.Length != 0) + { + return Xxh3Len1To3128b(input, secret, seed); + } + + Hash128 h128 = new(); + ulong bitFlipL = BinaryPrimitives.ReadUInt64LittleEndian(secret[64..]) ^ BinaryPrimitives.ReadUInt64LittleEndian(secret[72..]); + ulong bitFlipH = BinaryPrimitives.ReadUInt64LittleEndian(secret[80..]) ^ BinaryPrimitives.ReadUInt64LittleEndian(secret[88..]); + h128.Low = Xxh64Avalanche(seed ^ bitFlipL); + h128.High = Xxh64Avalanche(seed ^ bitFlipH); + + return h128; + } + + private static ulong Xxh3Mix16b(ReadOnlySpan input, ReadOnlySpan secret, ulong seed) + { + ulong inputLo = BinaryPrimitives.ReadUInt64LittleEndian(input); + ulong inputHi = BinaryPrimitives.ReadUInt64LittleEndian(input[8..]); + + return Mul128Fold64( + inputLo ^ (BinaryPrimitives.ReadUInt64LittleEndian(secret) + seed), + inputHi ^ (BinaryPrimitives.ReadUInt64LittleEndian(secret[8..]) - seed)); + } + + private static Hash128 Xxh128Mix32b(Hash128 acc, ReadOnlySpan input, ReadOnlySpan input2, ReadOnlySpan secret, ulong seed) + { + acc.Low += Xxh3Mix16b(input, secret, seed); + acc.Low ^= BinaryPrimitives.ReadUInt64LittleEndian(input2) + BinaryPrimitives.ReadUInt64LittleEndian(input2[8..]); + acc.High += Xxh3Mix16b(input2, secret[16..], seed); + acc.High ^= BinaryPrimitives.ReadUInt64LittleEndian(input) + BinaryPrimitives.ReadUInt64LittleEndian(input[8..]); + + return acc; + } + + private static Hash128 Xxh3Len17To128128b(ReadOnlySpan input, ReadOnlySpan secret, ulong seed) + { + Debug.Assert(secret.Length >= SecretSizeMin); + Debug.Assert(input.Length is > 16 and <= 128); + + Hash128 acc = new() + { + Low = (ulong)input.Length * Prime64_1, + High = 0, + }; + + if (input.Length > 32) + { + if (input.Length > 64) + { + if (input.Length > 96) + { + acc = Xxh128Mix32b(acc, input[48..], input[^64..], secret[96..], seed); + } + + acc = Xxh128Mix32b(acc, input[32..], input[^48..], secret[64..], seed); + } + + acc = Xxh128Mix32b(acc, input[16..], input[^32..], secret[32..], seed); + } + + acc = Xxh128Mix32b(acc, input, input[^16..], secret, seed); + + Hash128 h128 = new() + { + Low = acc.Low + acc.High, + High = acc.Low * Prime64_1 + acc.High * Prime64_4 + ((ulong)input.Length - seed) * Prime64_2, + }; + h128.Low = Xxh3Avalanche(h128.Low); + h128.High = 0UL - Xxh3Avalanche(h128.High); + + return h128; + } + + private static Hash128 Xxh3Len129To240128b(ReadOnlySpan input, ReadOnlySpan secret, ulong seed) + { + Debug.Assert(secret.Length >= SecretSizeMin); + Debug.Assert(input.Length is > 128 and <= 240); + + Hash128 acc = new(); + + int nbRounds = input.Length / 32; + acc.Low = (ulong)input.Length * Prime64_1; + acc.High = 0; + + for (int i = 0; i < 4; i++) + { + acc = Xxh128Mix32b(acc, input[(32 * i)..], input[(32 * i + 16)..], secret[(32 * i)..], seed); + } + + acc.Low = Xxh3Avalanche(acc.Low); + acc.High = Xxh3Avalanche(acc.High); + Debug.Assert(nbRounds >= 4); + + for (int i = 4; i < nbRounds; i++) + { + acc = Xxh128Mix32b(acc, input[(32 * i)..], input[(32 * i + 16)..], secret[(MidSizeStartOffset + 32 * (i - 4))..], seed); + } + + acc = Xxh128Mix32b(acc, input[^16..], input[^32..], secret[(SecretSizeMin - MidSizeLastOffset - 16)..], 0UL - seed); + + Hash128 h128 = new() + { + Low = acc.Low + acc.High, + High = acc.Low * Prime64_1 + acc.High * Prime64_4 + ((ulong)input.Length - seed) * Prime64_2, + }; + h128.Low = Xxh3Avalanche(h128.Low); + h128.High = 0UL - Xxh3Avalanche(h128.High); + + return h128; + } + + private static Hash128 Xxh3128bitsInternal(ReadOnlySpan input, ReadOnlySpan secret, ulong seed) + { + Debug.Assert(secret.Length >= SecretSizeMin); + + return input.Length switch + { + <= 16 => Xxh3Len0To16128b(input, secret, seed), + <= 128 => Xxh3Len17To128128b(input, secret, seed), + <= 240 => Xxh3Len129To240128b(input, secret, seed), + _ => Xxh3HashLong128bInternal(input, secret) + }; + } + + #endregion + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/ConsoleHelper.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/ConsoleHelper.cs new file mode 100644 index 0000000..2c81115 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/ConsoleHelper.cs @@ -0,0 +1,64 @@ +using Ryujinx.Common.Logging; +using System; +using System.Runtime.InteropServices; +using System.Runtime.Versioning; + +namespace Ryujinx.Common.Helper +{ + public static partial class ConsoleHelper + { + [SupportedOSPlatform("windows")] + [LibraryImport("kernel32")] + private static partial nint GetConsoleWindow(); + + [SupportedOSPlatform("windows")] + [LibraryImport("kernel32", SetLastError = true)] + [return: MarshalAs(UnmanagedType.Bool)] + private static partial bool FreeConsole(); + + public static bool SetConsoleWindowStateSupported => OperatingSystem.IsWindows(); + public static bool HasConsoleWindow => OperatingSystem.IsWindows() && GetConsoleWindow() != nint.Zero; + + public static void SetConsoleWindowState(bool show) + { + if (OperatingSystem.IsWindows()) + { + SetConsoleWindowStateWindows(show); + } + else if (show == false) + { + Logger.Warning?.Print(LogClass.Application, "OS doesn't support hiding console window"); + } + } + + [SupportedOSPlatform("windows")] + private static void SetConsoleWindowStateWindows(bool show) + { + if (show) + { + if (GetConsoleWindow() != nint.Zero) + { + Logger.SetConsoleTargetEnabled(true); + } + return; + } + + Logger.SetConsoleTargetEnabled(false); + DetachConsole(); + } + + [SupportedOSPlatform("windows")] + private static void DetachConsole() + { + if (GetConsoleWindow() == nint.Zero) + { + return; + } + + if (!FreeConsole()) + { + Logger.Warning?.Print(LogClass.Application, "Attempted to detach console window but the operation failed"); + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/FileAssociationHelper.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/FileAssociationHelper.cs new file mode 100644 index 0000000..65aa715 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/FileAssociationHelper.cs @@ -0,0 +1,195 @@ +using Microsoft.Win32; +using Ryujinx.Common.Logging; +using System; +using System.Diagnostics; +using System.IO; +using System.Linq; +using System.Runtime.InteropServices; +using System.Runtime.Versioning; + +namespace Ryujinx.Common.Helper +{ + public static partial class FileAssociationHelper + { + private static readonly string[] _fileExtensions = [".nca", ".nro", ".nso", ".nsp", ".xci"]; + + [SupportedOSPlatform("linux")] + private static readonly string _mimeDbPath = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), ".local", "share", "mime"); + + private const int SHCNE_ASSOCCHANGED = 0x8000000; + private const int SHCNF_FLUSH = 0x1000; + + [LibraryImport("shell32.dll", SetLastError = true)] + public static partial void SHChangeNotify(uint wEventId, uint uFlags, nint dwItem1, nint dwItem2); + + public static bool IsTypeAssociationSupported => (OperatingSystem.IsLinux() || OperatingSystem.IsWindows()); + + // NOTE: On macOS, users have a more robust file association system (via Right-Click > Get Info > "Open with:" > Ryujinx > "Change All...) + // Custom file association isn't strictly necessary and will not provide any additional benefit to macOS users. + + public static bool AreMimeTypesRegistered + { + get + { + if (OperatingSystem.IsLinux()) + { + return AreMimeTypesRegisteredLinux(); + } + + if (OperatingSystem.IsWindows()) + { + return AreMimeTypesRegisteredWindows(); + } + + return false; + } + } + + [SupportedOSPlatform("linux")] + private static bool AreMimeTypesRegisteredLinux() => File.Exists(Path.Combine(_mimeDbPath, "packages", "Ryujinx.xml")); + + [SupportedOSPlatform("linux")] + private static bool InstallLinuxMimeTypes(bool uninstall = false) + { + string installKeyword = uninstall ? "uninstall" : "install"; + + if ((uninstall && AreMimeTypesRegisteredLinux()) || (!uninstall && !AreMimeTypesRegisteredLinux())) + { + string mimeTypesFile = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "mime", "Ryujinx.xml"); + string additionalArgs = !uninstall ? "--novendor" : string.Empty; + + using Process mimeProcess = new(); + + mimeProcess.StartInfo.FileName = "xdg-mime"; + mimeProcess.StartInfo.Arguments = $"{installKeyword} {additionalArgs} --mode user {mimeTypesFile}"; + + mimeProcess.Start(); + mimeProcess.WaitForExit(); + + if (mimeProcess.ExitCode != 0) + { + Logger.Error?.PrintMsg(LogClass.Application, $"Unable to {installKeyword} mime types. Make sure xdg-utils is installed. Process exited with code: {mimeProcess.ExitCode}"); + + return false; + } + + using Process updateMimeProcess = new(); + + updateMimeProcess.StartInfo.FileName = "update-mime-database"; + updateMimeProcess.StartInfo.Arguments = _mimeDbPath; + + updateMimeProcess.Start(); + updateMimeProcess.WaitForExit(); + + if (updateMimeProcess.ExitCode != 0) + { + Logger.Error?.PrintMsg(LogClass.Application, $"Could not update local mime database. Process exited with code: {updateMimeProcess.ExitCode}"); + } + } + + return true; + } + + [SupportedOSPlatform("windows")] + private static bool AreMimeTypesRegisteredWindows() + { + return _fileExtensions.Aggregate(false, + (current, ext) => current | CheckRegistering(ext) + ); + + static bool CheckRegistering(string ext) + { + RegistryKey key = Registry.CurrentUser.OpenSubKey(@$"Software\Classes\{ext}"); + + RegistryKey openCmd = key?.OpenSubKey(@"shell\open\command"); + + if (openCmd is null) + { + return false; + } + + string keyValue = (string)openCmd.GetValue(string.Empty); + + return keyValue is not null && (keyValue.Contains("Ryujinx") || keyValue.Contains(AppDomain.CurrentDomain.FriendlyName)); + } + } + + [SupportedOSPlatform("windows")] + private static bool InstallWindowsMimeTypes(bool uninstall = false) + { + bool registered = _fileExtensions.Aggregate(false, + (current, ext) => current | RegisterExtension(ext, uninstall) + ); + + // Notify Explorer the file association has been changed. + SHChangeNotify(SHCNE_ASSOCCHANGED, SHCNF_FLUSH, nint.Zero, nint.Zero); + + return registered; + + static bool RegisterExtension(string ext, bool uninstall = false) + { + string keyString = @$"Software\Classes\{ext}"; + + if (uninstall) + { + // If the types don't already exist, there's nothing to do, and we can call this operation successful. + if (!AreMimeTypesRegisteredWindows()) + { + return true; + } + + Logger.Debug?.Print(LogClass.Application, $"Removing type association {ext}"); + Registry.CurrentUser.DeleteSubKeyTree(keyString); + Logger.Debug?.Print(LogClass.Application, $"Removed type association {ext}"); + } + else + { + using RegistryKey key = Registry.CurrentUser.CreateSubKey(keyString); + + if (key is null) + { + return false; + } + + Logger.Debug?.Print(LogClass.Application, $"Adding type association {ext}"); + using RegistryKey openCmd = key.CreateSubKey(@"shell\open\command"); + openCmd.SetValue(string.Empty, $"\"{Environment.ProcessPath}\" \"%1\""); + Logger.Debug?.Print(LogClass.Application, $"Added type association {ext}"); + + } + + return true; + } + } + + public static bool Install() + { + if (OperatingSystem.IsLinux()) + { + return InstallLinuxMimeTypes(); + } + + if (OperatingSystem.IsWindows()) + { + return InstallWindowsMimeTypes(); + } + + return false; + } + + public static bool Uninstall() + { + if (OperatingSystem.IsLinux()) + { + return InstallLinuxMimeTypes(true); + } + + if (OperatingSystem.IsWindows()) + { + return InstallWindowsMimeTypes(true); + } + + return false; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/LinuxHelper.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/LinuxHelper.cs new file mode 100644 index 0000000..611677a --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/LinuxHelper.cs @@ -0,0 +1,62 @@ +using System; +using System.Diagnostics; +using System.IO; +using System.Runtime.Versioning; + +namespace Ryujinx.Common.Helper +{ + [SupportedOSPlatform("linux")] + public static class LinuxHelper + { + // NOTE: This value was determined by manual tests and might need to be increased again. + public const int RecommendedVmMaxMapCount = 524288; + public const string VmMaxMapCountPath = "/proc/sys/vm/max_map_count"; + public const string SysCtlConfigPath = "/etc/sysctl.d/99-Ryujinx.conf"; + public static int VmMaxMapCount => int.Parse(File.ReadAllText(VmMaxMapCountPath)); + public static string PkExecPath { get; } = GetBinaryPath("pkexec"); + + private static string GetBinaryPath(string binary) + { + string pathVar = Environment.GetEnvironmentVariable("PATH"); + + if (pathVar is null || string.IsNullOrEmpty(binary)) + { + return null; + } + + foreach (string searchPath in pathVar.Split(":", StringSplitOptions.TrimEntries | StringSplitOptions.RemoveEmptyEntries)) + { + string binaryPath = Path.Combine(searchPath, binary); + + if (File.Exists(binaryPath)) + { + return binaryPath; + } + } + + return null; + } + + public static int RunPkExec(string command) + { + if (PkExecPath == null) + { + return 1; + } + + using Process process = new() + { + StartInfo = + { + FileName = PkExecPath, + ArgumentList = { "sh", "-c", command }, + }, + }; + + process.Start(); + process.WaitForExit(); + + return process.ExitCode; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/ObjectiveC.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/ObjectiveC.cs new file mode 100644 index 0000000..6abf9e3 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/ObjectiveC.cs @@ -0,0 +1,159 @@ +using System.Runtime.InteropServices; +using System.Runtime.Versioning; + +namespace Ryujinx.Common.Helper +{ + [SupportedOSPlatform("macos")] + public static partial class ObjectiveC + { + private const string ObjCRuntime = "/usr/lib/libobjc.A.dylib"; + + [LibraryImport(ObjCRuntime, StringMarshalling = StringMarshalling.Utf8)] + private static partial nint sel_getUid(string name); + + [LibraryImport(ObjCRuntime, StringMarshalling = StringMarshalling.Utf8)] + private static partial nint objc_getClass(string name); + + [LibraryImport(ObjCRuntime)] + private static partial void objc_msgSend(nint receiver, Selector selector); + + [LibraryImport(ObjCRuntime)] + private static partial void objc_msgSend(nint receiver, Selector selector, byte value); + + [LibraryImport(ObjCRuntime)] + private static partial void objc_msgSend(nint receiver, Selector selector, nint value); + + [LibraryImport(ObjCRuntime)] + private static partial void objc_msgSend(nint receiver, Selector selector, NSRect point); + + [LibraryImport(ObjCRuntime)] + private static partial void objc_msgSend(nint receiver, Selector selector, double value); + + [LibraryImport(ObjCRuntime, EntryPoint = "objc_msgSend")] + private static partial nint nint_objc_msgSend(nint receiver, Selector selector); + + [LibraryImport(ObjCRuntime, EntryPoint = "objc_msgSend")] + private static partial nint nint_objc_msgSend(nint receiver, Selector selector, nint param); + + [LibraryImport(ObjCRuntime, EntryPoint = "objc_msgSend", StringMarshalling = StringMarshalling.Utf8)] + private static partial nint nint_objc_msgSend(nint receiver, Selector selector, string param); + + [LibraryImport(ObjCRuntime, EntryPoint = "objc_msgSend")] + [return: MarshalAs(UnmanagedType.Bool)] + private static partial bool bool_objc_msgSend(nint receiver, Selector selector, nint param); + + public readonly struct Object + { + public readonly nint ObjPtr; + + private Object(nint pointer) + { + ObjPtr = pointer; + } + + public Object(string name) + { + ObjPtr = objc_getClass(name); + } + + public void SendMessage(Selector selector) + { + objc_msgSend(ObjPtr, selector); + } + + public void SendMessage(Selector selector, byte value) + { + objc_msgSend(ObjPtr, selector, value); + } + + public void SendMessage(Selector selector, Object obj) + { + objc_msgSend(ObjPtr, selector, obj.ObjPtr); + } + + public void SendMessage(Selector selector, NSRect point) + { + objc_msgSend(ObjPtr, selector, point); + } + + public void SendMessage(Selector selector, double value) + { + objc_msgSend(ObjPtr, selector, value); + } + + public Object GetFromMessage(Selector selector) + { + return new Object(nint_objc_msgSend(ObjPtr, selector)); + } + + public Object GetFromMessage(Selector selector, Object obj) + { + return new Object(nint_objc_msgSend(ObjPtr, selector, obj.ObjPtr)); + } + + public Object GetFromMessage(Selector selector, NSString nsString) + { + return new Object(nint_objc_msgSend(ObjPtr, selector, nsString.StrPtr)); + } + + public Object GetFromMessage(Selector selector, string param) + { + return new Object(nint_objc_msgSend(ObjPtr, selector, param)); + } + + public bool GetBoolFromMessage(Selector selector, Object obj) + { + return bool_objc_msgSend(ObjPtr, selector, obj.ObjPtr); + } + } + + public readonly struct Selector + { + public readonly nint SelPtr; + + private Selector(string name) + { + SelPtr = sel_getUid(name); + } + + public static implicit operator Selector(string value) => new(value); + } + + public readonly struct NSString + { + public readonly nint StrPtr; + + public NSString(string aString) + { + nint nsString = objc_getClass("NSString"); + StrPtr = nint_objc_msgSend(nsString, "stringWithUTF8String:", aString); + } + + public static implicit operator nint(NSString nsString) => nsString.StrPtr; + } + + public readonly struct NSPoint + { + public readonly double X; + public readonly double Y; + + public NSPoint(double x, double y) + { + X = x; + Y = y; + } + } + + public readonly struct NSRect + { + public readonly NSPoint Pos; + public readonly NSPoint Size; + + public NSRect(double x, double y, double width, double height) + { + Pos = new NSPoint(x, y); + Size = new NSPoint(width, height); + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/OpenHelper.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/OpenHelper.cs new file mode 100644 index 0000000..2c37186 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/OpenHelper.cs @@ -0,0 +1,115 @@ +using Gommon; +using Ryujinx.Common.Logging; +using System; +using System.Diagnostics; +using System.IO; +using System.Runtime.InteropServices; + +namespace Ryujinx.Common.Helper +{ + public static partial class OpenHelper + { + [LibraryImport("shell32.dll", SetLastError = true)] + private static partial int SHOpenFolderAndSelectItems(nint pidlFolder, uint cidl, nint apidl, uint dwFlags); + + [LibraryImport("shell32.dll", SetLastError = true)] + private static partial void ILFree(nint pidlList); + + [LibraryImport("shell32.dll", SetLastError = true)] + private static partial nint ILCreateFromPathW([MarshalAs(UnmanagedType.LPWStr)] string pszPath); + + public static void OpenFolder(string path) + { + if (Directory.Exists(path)) + { + Process.Start(new ProcessStartInfo + { + FileName = path, + UseShellExecute = true, + Verb = "open", + }); + } + else + { + Logger.Notice.Print(LogClass.Application, $"Directory \"{path}\" doesn't exist!"); + } + } + + public static void OpenFolder(FilePath path) => OpenFolder(path.Path); + + public static void LocateFile(string path) + { + if (File.Exists(path)) + { + if (OperatingSystem.IsWindows()) + { + nint pidlList = ILCreateFromPathW(path); + if (pidlList != nint.Zero) + { + try + { + Marshal.ThrowExceptionForHR(SHOpenFolderAndSelectItems(pidlList, 0, nint.Zero, 0)); + } + finally + { + ILFree(pidlList); + } + } + } + else if (OperatingSystem.IsMacOS()) + { + ObjectiveC.NSString nsStringPath = new(path); + ObjectiveC.Object nsUrl = new("NSURL"); + ObjectiveC.Object urlPtr = nsUrl.GetFromMessage("fileURLWithPath:", nsStringPath); + + ObjectiveC.Object nsArray = new("NSArray"); + ObjectiveC.Object urlArray = nsArray.GetFromMessage("arrayWithObject:", urlPtr); + + ObjectiveC.Object nsWorkspace = new("NSWorkspace"); + ObjectiveC.Object sharedWorkspace = nsWorkspace.GetFromMessage("sharedWorkspace"); + + sharedWorkspace.SendMessage("activateFileViewerSelectingURLs:", urlArray); + } + else if (OperatingSystem.IsLinux()) + { + Process.Start("dbus-send", $"--session --print-reply --dest=org.freedesktop.FileManager1 --type=method_call /org/freedesktop/FileManager1 org.freedesktop.FileManager1.ShowItems array:string:\"file://{path}\" string:\"\""); + } + else + { + OpenFolder(Path.GetDirectoryName(path)); + } + } + else + { + Logger.Notice.Print(LogClass.Application, $"File \"{path}\" doesn't exist!"); + } + } + + public static void OpenUrl(string url) + { + if (OperatingSystem.IsWindows()) + { + Process.Start(new ProcessStartInfo("cmd", $"/c start {url.Replace("&", "^&")}")); + } + else if (OperatingSystem.IsLinux()) + { + Process.Start("xdg-open", url); + } + else if (OperatingSystem.IsMacOS()) + { + ObjectiveC.NSString nsStringPath = new(url); + ObjectiveC.Object nsUrl = new("NSURL"); + ObjectiveC.Object urlPtr = nsUrl.GetFromMessage("URLWithString:", nsStringPath); + + ObjectiveC.Object nsWorkspace = new("NSWorkspace"); + ObjectiveC.Object sharedWorkspace = nsWorkspace.GetFromMessage("sharedWorkspace"); + + sharedWorkspace.GetBoolFromMessage("openURL:", urlPtr); + } + else + { + Logger.Notice.Print(LogClass.Application, $"Cannot open url \"{url}\" on this platform!"); + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/Patterns.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/Patterns.cs new file mode 100644 index 0000000..3d31acf --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/Patterns.cs @@ -0,0 +1,126 @@ +using System.Text.RegularExpressions; + +namespace Ryujinx.Common.Helper +{ + public static partial class Patterns + { + #region Accessors + + public static readonly Regex Numeric = NumericRegex(); + + public static readonly Regex AmdGcn = AmdGcnRegex(); + public static readonly Regex AmdRdna3 = AmdRdna3Regex(); + public static readonly Regex NvidiaConsumerClass = NvidiaConsumerClassRegex(); + public static readonly Regex IntelWindowsDriverVersion = IntelWindowsDriverVersionRegex(); + + public static readonly Regex DomainLp1Ns = DomainLp1NsRegex(); + public static readonly Regex DomainLp1Lp1Npln = DomainLp1Lp1NplnRegex(); + public static readonly Regex DomainLp1Znc = DomainLp1ZncRegex(); + public static readonly Regex DomainSbApi = DomainSbApiRegex(); + public static readonly Regex DomainSbAccounts = DomainSbAccountsRegex(); + public static readonly Regex DomainAccounts = DomainAccountsRegex(); + + public static readonly Regex Module = ModuleRegex(); + public static readonly Regex FsSdk = FsSdkRegex(); + public static readonly Regex SdkMw = SdkMwRegex(); + + // ReSharper disable once InconsistentNaming + public static readonly Regex CJK = CJKRegex(); + + public static readonly Regex LdnPassphrase = LdnPassphraseRegex(); + + public static readonly Regex CleanText = CleanTextRegex(); + + #endregion + + #region Generated pattern stubs + + #region Numeric validation + + [GeneratedRegex("[0-9]|.")] + internal static partial Regex NumericRegex(); + + #endregion + + #region GPU names + + [GeneratedRegex( + "Radeon (((HD|R(5|7|9|X)) )?((M?[2-6]\\d{2}(\\D|$))|([7-8]\\d{3}(\\D|$))|Fury|Nano))|(Pro Duo)")] + internal static partial Regex AmdGcnRegex(); + + [GeneratedRegex("Radeon ([7-8](\\d{2}\\d?)[MS]|PRO [VW]7(\\d{2}\\d?)|RX 7\\d{3}([MS]?| XTX?| GRE)?)")] + public static partial Regex AmdRdna3Regex(); + + [GeneratedRegex("NVIDIA GeForce (R|G)?TX? (\\d{3}\\d?)M?")] + internal static partial Regex NvidiaConsumerClassRegex(); + + [GeneratedRegex(@"^\d+\.\d+\.\d+\.\d+$")] + internal static partial Regex IntelWindowsDriverVersionRegex(); + + #endregion + + #region DNS blocking + + public static readonly Regex[] BlockedHosts = + [ + DomainLp1Ns, + DomainLp1Lp1Npln, + DomainLp1Znc, + DomainSbApi, + DomainSbAccounts, + DomainAccounts + ]; + + const RegexOptions DnsRegexOpts = + RegexOptions.CultureInvariant | RegexOptions.IgnoreCase | RegexOptions.ExplicitCapture; + + [GeneratedRegex(@"^(.*)\-lp1\.(n|s)\.n\.srv\.nintendo\.net$", DnsRegexOpts)] + internal static partial Regex DomainLp1NsRegex(); + + [GeneratedRegex(@"^(.*)\-lp1\.lp1\.t\.npln\.srv\.nintendo\.net$", DnsRegexOpts)] + internal static partial Regex DomainLp1Lp1NplnRegex(); + + [GeneratedRegex(@"^(.*)\-lp1\.(znc|p)\.srv\.nintendo\.net$", DnsRegexOpts)] + internal static partial Regex DomainLp1ZncRegex(); + + [GeneratedRegex(@"^(.*)\-sb\-api\.accounts\.nintendo\.com$", DnsRegexOpts)] + internal static partial Regex DomainSbApiRegex(); + + [GeneratedRegex(@"^(.*)\-sb\.accounts\.nintendo\.com$", DnsRegexOpts)] + internal static partial Regex DomainSbAccountsRegex(); + + [GeneratedRegex(@"^accounts\.nintendo\.com$", DnsRegexOpts)] + internal static partial Regex DomainAccountsRegex(); + + #endregion + + #region Executable information + + [GeneratedRegex(@"[a-z]:[\\/][ -~]{5,}\.nss", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant)] + internal static partial Regex ModuleRegex(); + + [GeneratedRegex(@"sdk_version: ([0-9.]*)")] + internal static partial Regex FsSdkRegex(); + + [GeneratedRegex(@"SDK MW[ -~]*")] + internal static partial Regex SdkMwRegex(); + + #endregion + + #region CJK + + [GeneratedRegex( + "\\p{IsHangulJamo}|\\p{IsCJKRadicalsSupplement}|\\p{IsCJKSymbolsandPunctuation}|\\p{IsEnclosedCJKLettersandMonths}|\\p{IsCJKCompatibility}|\\p{IsCJKUnifiedIdeographsExtensionA}|\\p{IsCJKUnifiedIdeographs}|\\p{IsHangulSyllables}|\\p{IsCJKCompatibilityForms}")] + private static partial Regex CJKRegex(); + + #endregion + + [GeneratedRegex("Ryujinx-[0-9a-f]{8}")] + private static partial Regex LdnPassphraseRegex(); + + [GeneratedRegex(@"[^\u0000\u0009\u000A\u000D\u0020-\uFFFF]..")] + private static partial Regex CleanTextRegex(); + + #endregion + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/RefEvent.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/RefEvent.cs new file mode 100644 index 0000000..d3f92a5 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/RefEvent.cs @@ -0,0 +1,58 @@ +using Gommon; +using System.Collections.Generic; +using System.Threading; + +namespace Ryujinx.Common.Helper +{ + public class RefEvent + { + public delegate void Handler(ref T arg); + + private readonly Lock _subLock = new(); + private readonly List _subscriptions = []; + + public bool HasSubscribers + { + get + { + lock (_subLock) + return _subscriptions.Count != 0; + } + } + + public IReadOnlyList Subscriptions + { + get + { + lock (_subLock) + return _subscriptions; + } + } + + public void Add(Handler subscriber) + { + Guard.Require(subscriber, nameof(subscriber)); + lock (_subLock) + _subscriptions.Add(subscriber); + } + + public void Remove(Handler subscriber) + { + Guard.Require(subscriber, nameof(subscriber)); + lock (_subLock) + _subscriptions.Remove(subscriber); + } + + public void Clear() + { + lock (_subLock) + _subscriptions.Clear(); + } + + public void Call(ref T arg) + { + foreach (Handler subscription in Subscriptions) + subscription(ref arg); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/RunningPlatform.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/RunningPlatform.cs new file mode 100644 index 0000000..d300cd5 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Helpers/RunningPlatform.cs @@ -0,0 +1,46 @@ +using System; +using System.Runtime.InteropServices; +// ReSharper disable MemberCanBePrivate.Global +// ReSharper disable InconsistentNaming + +namespace Ryujinx.Common.Helper +{ + public enum OperatingSystemType + { + MacOS, + Linux, + Windows + } + + public static class RunningPlatform + { + public static readonly OperatingSystemType CurrentOS + = IsMacOS + ? OperatingSystemType.MacOS + : IsWindows + ? OperatingSystemType.Windows + : IsLinux + ? OperatingSystemType.Linux + : throw new PlatformNotSupportedException(); + + public static Architecture Architecture => RuntimeInformation.OSArchitecture; + public static Architecture CurrentProcessArchitecture => RuntimeInformation.ProcessArchitecture; + + public static bool IsMacOS => OperatingSystem.IsMacOS(); + public static bool IsWindows => OperatingSystem.IsWindows(); + public static bool IsLinux => OperatingSystem.IsLinux(); + + public static bool IsArm => Architecture is Architecture.Arm64; + + public static bool IsX64 => Architecture is Architecture.X64; + + public static bool IsIntelMac => IsMacOS && IsX64; + public static bool IsArmMac => IsMacOS && IsArm; + + public static bool IsX64Windows => IsWindows && IsX64; + public static bool IsArmWindows => IsWindows && IsArm; + + public static bool IsX64Linux => IsLinux && IsX64; + public static bool IsArmLinux => IsLinux && IsArmMac; + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Formatters/DefaultLogFormatter.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Formatters/DefaultLogFormatter.cs new file mode 100644 index 0000000..b04cf34 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Formatters/DefaultLogFormatter.cs @@ -0,0 +1,51 @@ +using System.Diagnostics; +using System.Text; + +namespace Ryujinx.Common.Logging.Formatters +{ + internal class DefaultLogFormatter : ILogFormatter + { + private static readonly ObjectPool _stringBuilderPool = SharedPools.Default(); + + public string Format(LogEventArgs args) + { + StringBuilder sb = _stringBuilderPool.Allocate(); + + try + { + sb.Clear(); + + sb.Append($@"{args.Time:hh\:mm\:ss\.fff}"); + sb.Append($" |{args.Level.ToString()[0]}| "); + + if (args.ThreadName != null) + { + sb.Append(args.ThreadName); + sb.Append(' '); + } + + sb.Append(args.Message); + + if (args.Data is not null) + { + if (args.Data is StackTrace trace) + { + sb.Append('\n'); + sb.Append(trace); + + return sb.ToString(); + } + + sb.Append(' '); + DynamicObjectFormatter.Format(sb, args.Data); + } + + return sb.ToString(); + } + finally + { + _stringBuilderPool.Release(sb); + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Formatters/DynamicObjectFormatter.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Formatters/DynamicObjectFormatter.cs new file mode 100644 index 0000000..7da0d48 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Formatters/DynamicObjectFormatter.cs @@ -0,0 +1,84 @@ +#nullable enable +using System; +using System.Reflection; +using System.Text; + +namespace Ryujinx.Common.Logging.Formatters +{ + internal static class DynamicObjectFormatter + { + private static readonly ObjectPool _stringBuilderPool = SharedPools.Default(); + + public static string? Format(object? dynamicObject) + { + if (dynamicObject is null) + { + return null; + } + + StringBuilder sb = _stringBuilderPool.Allocate(); + + try + { + Format(sb, dynamicObject); + + return sb.ToString(); + } + finally + { + _stringBuilderPool.Release(sb); + } + } + + public static void Format(StringBuilder sb, object? dynamicObject) + { + if (dynamicObject is null) + { + return; + } + + PropertyInfo[] props = dynamicObject.GetType().GetProperties(); + + sb.Append('{'); + + foreach (PropertyInfo prop in props) + { + sb.Append(prop.Name); + sb.Append(": "); + + if (typeof(Array).IsAssignableFrom(prop.PropertyType)) + { + Array? array = (Array?)prop.GetValue(dynamicObject); + + if (array is not null) + { + foreach (object? item in array) + { + sb.Append(item); + sb.Append(", "); + } + + if (array.Length > 0) + { + sb.Remove(sb.Length - 2, 2); + } + } + } + else + { + sb.Append(prop.GetValue(dynamicObject)); + } + + sb.Append(" ; "); + } + + // We remove the final ';' from the string + if (props.Length > 0) + { + sb.Remove(sb.Length - 3, 3); + } + + sb.Append('}'); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Formatters/ILogFormatter.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Formatters/ILogFormatter.cs new file mode 100644 index 0000000..56d5795 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Formatters/ILogFormatter.cs @@ -0,0 +1,7 @@ +namespace Ryujinx.Common.Logging.Formatters +{ + interface ILogFormatter + { + string Format(LogEventArgs args); + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Logging/LogClass.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/LogClass.cs new file mode 100644 index 0000000..7d98eb1 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/LogClass.cs @@ -0,0 +1,79 @@ +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Logging +{ + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum LogClass + { + Application, + Audio, + AudioRenderer, + Configuration, + Cpu, + Emulation, + FFmpeg, + GdbStub, + Font, + Gpu, + Hid, + Host1x, + Kernel, + KernelIpc, + KernelScheduler, + KernelSvc, + Loader, + ModLoader, + Nvdec, + Ptc, + Service, + ServiceAcc, + ServiceAm, + ServiceApm, + ServiceAudio, + ServiceBcat, + ServiceBsd, + ServiceBtm, + ServiceCaps, + ServiceEctx, + ServiceFatal, + ServiceFriend, + ServiceFs, + ServiceHid, + ServiceIrs, + ServiceLdn, + ServiceLdr, + ServiceLm, + ServiceMii, + ServiceMm, + ServiceMnpp, + ServiceNfc, + ServiceNfp, + ServiceNgct, + ServiceNifm, + ServiceNim, + ServiceNotification, + ServiceNs, + ServiceNsd, + ServiceNtc, + ServiceNv, + ServiceOlsc, + ServicePctl, + ServicePcv, + ServicePl, + ServicePrepo, + ServicePsm, + ServicePtm, + ServiceSet, + ServiceSfdnsres, + ServiceSm, + ServiceSsl, + ServiceSss, + ServiceTime, + ServiceVi, + SurfaceFlinger, + TamperMachine, + UI, + Vic, + XCIFileTrimmer + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Logging/LogEventArgs.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/LogEventArgs.cs new file mode 100644 index 0000000..0180a89 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/LogEventArgs.cs @@ -0,0 +1,23 @@ +using System; + +namespace Ryujinx.Common.Logging +{ + public class LogEventArgs : EventArgs + { + public readonly LogLevel Level; + public readonly TimeSpan Time; + public readonly string ThreadName; + + public readonly string Message; + public readonly object Data; + + public LogEventArgs(LogLevel level, TimeSpan time, string threadName, string message, object data = null) + { + Level = level; + Time = time; + ThreadName = threadName; + Message = message; + Data = data; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Logging/LogLevel.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/LogLevel.cs new file mode 100644 index 0000000..869ceab --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/LogLevel.cs @@ -0,0 +1,19 @@ +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Logging +{ + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum LogLevel + { + Debug, + Stub, + Info, + Warning, + Error, + Guest, + AccessLog, + NetLog, + Notice, + Trace, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Logger.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Logger.cs new file mode 100644 index 0000000..8568a16 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Logger.cs @@ -0,0 +1,266 @@ +using Ryujinx.Common.Logging.Targets; +using Ryujinx.Common.SystemInterop; +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.IO; +using System.Linq; +using System.Runtime.CompilerServices; +using System.Threading; + +namespace Ryujinx.Common.Logging +{ + public static class Logger + { + private static readonly Stopwatch _time; + + private static readonly bool[] _enabledClasses; + + private static readonly List _logTargets; + + private static readonly StdErrAdapter _stdErrAdapter; + + public static event EventHandler Updated; + + public readonly struct Log + { + private static readonly string _homeDir = Environment.GetFolderPath(Environment.SpecialFolder.UserProfile); + private static readonly string _homeDirRedacted = Path.Combine(Directory.GetParent(_homeDir)!.FullName, "[redacted]"); + + internal readonly LogLevel Level; + + internal Log(LogLevel level) + { + Level = level; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void PrintMsg(LogClass logClass, string message) + { + if (_enabledClasses[(int)logClass]) + { + Updated?.Invoke(null, new LogEventArgs(Level, _time.Elapsed, Thread.CurrentThread.Name, FormatMessage(logClass, string.Empty, message))); + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void Print(LogClass logClass, string message, [CallerMemberName] string caller = "") + { + if (_enabledClasses[(int)logClass]) + { + Updated?.Invoke(null, new LogEventArgs(Level, _time.Elapsed, Thread.CurrentThread.Name, FormatMessage(logClass, caller, message))); + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void Print(LogClass logClass, string message, object data, [CallerMemberName] string caller = "") + { + if (_enabledClasses[(int)logClass]) + { + Updated?.Invoke(null, new LogEventArgs(Level, _time.Elapsed, Thread.CurrentThread.Name, FormatMessage(logClass, caller, message), data)); + } + } + + [StackTraceHidden] + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void PrintStack(LogClass logClass, string message, [CallerMemberName] string caller = "") + { + if (_enabledClasses[(int)logClass]) + { + Updated?.Invoke(null, new LogEventArgs(Level, _time.Elapsed, Thread.CurrentThread.Name, FormatMessage(logClass, caller, message), new StackTrace(true))); + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void PrintStub(LogClass logClass, string message = "", [CallerMemberName] string caller = "") + { + if (_enabledClasses[(int)logClass]) + { + Updated?.Invoke(null, new LogEventArgs(Level, _time.Elapsed, Thread.CurrentThread.Name, FormatMessage(logClass, caller, "Stubbed. " + message))); + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void PrintStub(LogClass logClass, object data, [CallerMemberName] string caller = "") + { + if (_enabledClasses[(int)logClass]) + { + Updated?.Invoke(null, new LogEventArgs(Level, _time.Elapsed, Thread.CurrentThread.Name, FormatMessage(logClass, caller, "Stubbed."), data)); + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void PrintStub(LogClass logClass, string message, object data, [CallerMemberName] string caller = "") + { + if (_enabledClasses[(int)logClass]) + { + Updated?.Invoke(null, new LogEventArgs(Level, _time.Elapsed, Thread.CurrentThread.Name, FormatMessage(logClass, caller, "Stubbed. " + message), data)); + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void PrintRawMsg(string message) + { + Updated?.Invoke(null, new LogEventArgs(Level, _time.Elapsed, Thread.CurrentThread.Name, message)); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static string FormatMessage(LogClass logClass, string caller, string message) + { + message = message.Replace(_homeDir, _homeDirRedacted); + + return $"{logClass} {caller}: {message}"; + } + } + + public static Log? Debug { get; private set; } + public static Log? Info { get; private set; } + public static Log? Warning { get; private set; } + public static Log? Error { get; private set; } + public static Log? Guest { get; private set; } + public static Log? AccessLog { get; private set; } + public static Log? NetLog { get; private set; } + public static Log? Stub { get; private set; } + public static Log? Trace { get; private set; } + public static Log Notice { get; } // Always enabled + + static Logger() + { + _enabledClasses = new bool[Enum.GetNames().Length]; + + for (int index = 0; index < _enabledClasses.Length; index++) + { + _enabledClasses[index] = true; + } + + _logTargets = []; + + _time = Stopwatch.StartNew(); + + SetConsoleTargetEnabled(true); + + Notice = new Log(LogLevel.Notice); + + // Enable important log levels before configuration is loaded + Error = new Log(LogLevel.Error); + Warning = new Log(LogLevel.Warning); + Info = new Log(LogLevel.Info); + Trace = new Log(LogLevel.Trace); + + _stdErrAdapter = new StdErrAdapter(); + } + + public static void RestartTime() + { + _time.Restart(); + } + + private static ILogTarget GetTarget(string targetName) + => _logTargets.FirstOrDefault(target => target.Name.Equals(targetName)); + + public static void AddTarget(ILogTarget target) + { + if (_logTargets.Any(t => t.Name == target.Name)) + { + target.Dispose(); + return; + } + + _logTargets.Add(target); + + Updated += target.Log; + } + + public static void SetConsoleTargetEnabled(bool enabled) + { + if (enabled) + { + AddTarget(new AsyncLogTargetWrapper( + new ConsoleLogTarget("console"), + 1000, + AsyncLogTargetOverflowAction.Discard)); + } + else + { + RemoveTarget("console"); + } + } + + public static void RemoveTarget(string target) + { + ILogTarget logTarget = GetTarget(target); + + if (logTarget != null) + { + Updated -= logTarget.Log; + + _logTargets.Remove(logTarget); + + logTarget.Dispose(); + } + } + + public static void Flush() + { + foreach (ILogTarget target in _logTargets) + { + if (target is AsyncLogTargetWrapper asyncTarget) + { + asyncTarget.Flush(); + } + } + } + + public static void Shutdown() + { + Updated = null; + + _stdErrAdapter.Dispose(); + + foreach (ILogTarget target in _logTargets) + { + target.Dispose(); + } + + _logTargets.Clear(); + } + + public static IReadOnlyCollection GetEnabledLevels() + { + Log?[] logs = [Debug, Info, Warning, Error, Guest, AccessLog, Stub, Trace]; + List levels = new(logs.Length); + foreach (Log? log in logs) + { + if (log.HasValue) + levels.Add(log.Value.Level); + } + + return levels; + } + + public static void SetEnable(LogLevel logLevel, bool enabled) + { + switch (logLevel) + { +#pragma warning disable IDE0055 // Disable formatting + case LogLevel.Debug : Debug = enabled ? new Log(LogLevel.Debug) : null; break; + case LogLevel.Info : Info = enabled ? new Log(LogLevel.Info) : null; break; + case LogLevel.Warning : Warning = enabled ? new Log(LogLevel.Warning) : null; break; + case LogLevel.Error : Error = enabled ? new Log(LogLevel.Error) : null; break; + case LogLevel.Guest : Guest = enabled ? new Log(LogLevel.Guest) : null; break; + case LogLevel.AccessLog : AccessLog = enabled ? new Log(LogLevel.AccessLog) : null; break; + case LogLevel.NetLog : NetLog = enabled ? new Log(LogLevel.NetLog) : null; break; + case LogLevel.Stub : Stub = enabled ? new Log(LogLevel.Stub) : null; break; + case LogLevel.Trace : Trace = enabled ? new Log(LogLevel.Trace) : null; break; + case LogLevel.Notice : break; + default: throw new ArgumentException("Unknown Log Level", nameof(logLevel)); +#pragma warning restore IDE0055 + } + } + + public static void SetEnable(LogClass logClass, bool enabled) + { + _enabledClasses[(int)logClass] = enabled; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Targets/AsyncLogTargetWrapper.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Targets/AsyncLogTargetWrapper.cs new file mode 100644 index 0000000..d358a5a --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Targets/AsyncLogTargetWrapper.cs @@ -0,0 +1,117 @@ +using System; +using System.Collections.Concurrent; +using System.Threading; + +namespace Ryujinx.Common.Logging.Targets +{ + public enum AsyncLogTargetOverflowAction + { + /// + /// Block until there's more room in the queue + /// + Block = 0, + + /// + /// Discard the overflowing item + /// + Discard = 1, + } + + public class AsyncLogTargetWrapper : ILogTarget + { + private readonly ILogTarget _target; + + private readonly Thread _messageThread; + + private readonly BlockingCollection _messageQueue; + + private readonly int _overflowTimeout; + + private sealed class FlushEventArgs : LogEventArgs + { + public readonly ManualResetEventSlim SignalEvent; + + public FlushEventArgs(ManualResetEventSlim signalEvent) + : base(LogLevel.Notice, TimeSpan.Zero, string.Empty, string.Empty) + { + SignalEvent = signalEvent; + } + } + + string ILogTarget.Name => _target.Name; + + public AsyncLogTargetWrapper(ILogTarget target, int queueLimit = -1, AsyncLogTargetOverflowAction overflowAction = AsyncLogTargetOverflowAction.Block) + { + _target = target; + _messageQueue = new BlockingCollection(queueLimit); + _overflowTimeout = overflowAction == AsyncLogTargetOverflowAction.Block ? -1 : 0; + + _messageThread = new Thread(() => + { + while (!_messageQueue.IsCompleted) + { + try + { + LogEventArgs item = _messageQueue.Take(); + + if (item is FlushEventArgs flush) + { + flush.SignalEvent.Set(); + continue; + } + + _target.Log(this, item); + } + catch (InvalidOperationException) + { + // IOE means that Take() was called on a completed collection. + // Some other thread can call CompleteAdding after we pass the + // IsCompleted check but before we call Take. + // We can simply catch the exception since the loop will break + // on the next iteration. + } + } + }) + { + Name = "Logger.MessageThread", + IsBackground = true, + }; + _messageThread.Start(); + } + + public void Log(object sender, LogEventArgs e) + { + if (!_messageQueue.IsAddingCompleted) + { + _messageQueue.TryAdd(e, _overflowTimeout); + } + } + + public void Flush() + { + if (_messageQueue.Count == 0 || _messageQueue.IsAddingCompleted) + { + return; + } + + using ManualResetEventSlim signal = new ManualResetEventSlim(false); + try + { + _messageQueue.Add(new FlushEventArgs(signal)); + } + catch (InvalidOperationException) + { + return; + } + + signal.Wait(); + } + + public void Dispose() + { + GC.SuppressFinalize(this); + _messageQueue.CompleteAdding(); + _messageThread.Join(); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Targets/ConsoleLogTarget.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Targets/ConsoleLogTarget.cs new file mode 100644 index 0000000..f930725 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Targets/ConsoleLogTarget.cs @@ -0,0 +1,44 @@ +using Ryujinx.Common.Logging.Formatters; +using System; + +namespace Ryujinx.Common.Logging.Targets +{ + public class ConsoleLogTarget : ILogTarget + { + private readonly DefaultLogFormatter _formatter; + + private readonly string _name; + + string ILogTarget.Name { get => _name; } + + private static ConsoleColor GetLogColor(LogLevel level) => level switch + { + LogLevel.Info => ConsoleColor.White, + LogLevel.Warning => ConsoleColor.Yellow, + LogLevel.Error => ConsoleColor.Red, + LogLevel.Stub => ConsoleColor.DarkGray, + LogLevel.Notice => ConsoleColor.Cyan, + LogLevel.Trace => ConsoleColor.DarkCyan, + _ => ConsoleColor.Gray, + }; + + public ConsoleLogTarget(string name) + { + _formatter = new DefaultLogFormatter(); + _name = name; + } + + public void Log(object sender, LogEventArgs args) + { + Console.ForegroundColor = GetLogColor(args.Level); + Console.WriteLine(_formatter.Format(args)); + Console.ResetColor(); + } + + public void Dispose() + { + GC.SuppressFinalize(this); + Console.ResetColor(); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Targets/FileLogTarget.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Targets/FileLogTarget.cs new file mode 100644 index 0000000..43489cb --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Targets/FileLogTarget.cs @@ -0,0 +1,109 @@ +using Ryujinx.Common.Logging.Formatters; +using System; +using System.IO; +using System.Linq; + +namespace Ryujinx.Common.Logging.Targets +{ + public class FileLogTarget : ILogTarget + { + private readonly StreamWriter _logWriter; + private readonly DefaultLogFormatter _formatter; + private readonly string _name; + + string ILogTarget.Name { get => _name; } + + public FileLogTarget(string name, FileStream fileStream) + { + _name = name; + _logWriter = new StreamWriter(fileStream); + _formatter = new DefaultLogFormatter(); + } + + public static FileStream PrepareLogFile(string path) + { + // Ensure directory is present + DirectoryInfo logDir; + try + { + logDir = new DirectoryInfo(path); + } + catch (ArgumentException exception) + { + Logger.Warning?.Print(LogClass.Application, $"Logging directory path ('{path}') was invalid: {exception}"); + + return null; + } + + try + { + logDir.Create(); + } + catch (IOException exception) + { + Logger.Warning?.Print(LogClass.Application, $"Logging directory could not be created '{logDir}': {exception}"); + + return null; + } + + // Clean up old logs, should only keep 3 + FileInfo[] files = logDir.GetFiles("*.log").OrderBy(info => info.CreationTime).ToArray(); + for (int i = 0; i < files.Length - 2; i++) + { + try + { + files[i].Delete(); + } + catch (UnauthorizedAccessException exception) + { + Logger.Warning?.Print(LogClass.Application, $"Old log file could not be deleted '{files[i].FullName}': {exception}"); + + return null; + } + catch (IOException exception) + { + Logger.Warning?.Print(LogClass.Application, $"Old log file could not be deleted '{files[i].FullName}': {exception}"); + + return null; + } + } + + string version = ReleaseInformation.Version; + string appName = ReleaseInformation.IsCanaryBuild ? "Ryujinx_Canary" : "Ryujinx"; + + // Get path for the current time + path = Path.Combine(logDir.FullName, $"{appName}_{version}_{DateTime.Now:yyyy-MM-dd_HH-mm-ss}.log"); + + try + { + return File.Open(path, FileMode.Append, FileAccess.Write, FileShare.Read); + } + catch (UnauthorizedAccessException exception) + { + Logger.Warning?.Print(LogClass.Application, $"Log file could not be created '{path}': {exception}"); + + return null; + } + catch (IOException exception) + { + Logger.Warning?.Print(LogClass.Application, $"Log file could not be created '{path}': {exception}"); + + return null; + } + } + + public void Log(object sender, LogEventArgs args) + { + _logWriter.WriteLine(_formatter.Format(args)); + _logWriter.Flush(); + } + + public void Dispose() + { + GC.SuppressFinalize(this); + _logWriter.WriteLine("---- End of Log ----"); + _logWriter.Flush(); + _logWriter.Dispose(); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Targets/ILogTarget.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Targets/ILogTarget.cs new file mode 100644 index 0000000..c657c55 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/Targets/ILogTarget.cs @@ -0,0 +1,11 @@ +using System; + +namespace Ryujinx.Common.Logging.Targets +{ + public interface ILogTarget : IDisposable + { + void Log(object sender, LogEventArgs args); + + string Name { get; } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Logging/XCIFileTrimmerLog.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/XCIFileTrimmerLog.cs new file mode 100644 index 0000000..31de666 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Logging/XCIFileTrimmerLog.cs @@ -0,0 +1,30 @@ +using Ryujinx.Common.Utilities; + +namespace Ryujinx.Common.Logging +{ + public class XCIFileTrimmerLog : XCIFileTrimmer.ILog + { + public virtual void Progress(long current, long total, string text, bool complete) + { + } + + public void Write(XCIFileTrimmer.LogType logType, string text) + { + switch (logType) + { + case XCIFileTrimmer.LogType.Info: + Logger.Notice.Print(LogClass.XCIFileTrimmer, text); + break; + case XCIFileTrimmer.LogType.Warn: + Logger.Warning?.Print(LogClass.XCIFileTrimmer, text); + break; + case XCIFileTrimmer.LogType.Error: + Logger.Error?.Print(LogClass.XCIFileTrimmer, text); + break; + case XCIFileTrimmer.LogType.Progress: + Logger.Info?.Print(LogClass.XCIFileTrimmer, text); + break; + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Memory/ArrayPtr.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/ArrayPtr.cs new file mode 100644 index 0000000..38aaffb --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/ArrayPtr.cs @@ -0,0 +1,123 @@ +using System; +using System.Diagnostics.CodeAnalysis; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; + +namespace Ryujinx.Common.Memory +{ + /// + /// Represents an array of unmanaged resources. + /// + /// Array element type + public unsafe struct ArrayPtr : IEquatable>, IArray where T : unmanaged + { + private nint _ptr; + + /// + /// Null pointer. + /// + public static ArrayPtr Null => new() { _ptr = nint.Zero }; + + /// + /// True if the pointer is null, false otherwise. + /// + public readonly bool IsNull => _ptr == nint.Zero; + + /// + /// Number of elements on the array. + /// + public int Length { get; } + + /// + /// Gets a reference to the item at the given index. + /// + /// + /// No bounds checks are performed, this allows negative indexing, + /// but care must be taken if the index may be out of bounds. + /// + /// Index of the element + /// Reference to the element at the given index + public readonly ref T this[int index] => ref Unsafe.AsRef((T*)_ptr + index); + + /// + /// Creates a new array from a given reference. + /// + /// + /// For data on the heap, proper pinning is necessary during + /// use. Failure to do so will result in memory corruption and crashes. + /// + /// Reference of the first array element + /// Number of elements on the array + public ArrayPtr(ref T value, int length) + { + _ptr = (nint)Unsafe.AsPointer(ref value); + Length = length; + } + + /// + /// Creates a new array from a given pointer. + /// + /// Array base pointer + /// Number of elements on the array + public ArrayPtr(T* ptr, int length) + { + _ptr = (nint)ptr; + Length = length; + } + + /// + /// Creates a new array from a given pointer. + /// + /// Array base pointer + /// Number of elements on the array + public ArrayPtr(nint ptr, int length) + { + _ptr = ptr; + Length = length; + } + + /// + /// Splits the array starting at the specified position. + /// + /// Index where the new array should start + /// New array starting at the specified position + public ArrayPtr Slice(int start) => new(ref this[start], Length - start); + + /// + /// Gets a span from the array. + /// + /// Span of the array + public Span AsSpan() => Length == 0 ? Span.Empty : MemoryMarshal.CreateSpan(ref this[0], Length); + + /// + /// Gets the array base pointer. + /// + /// Base pointer + public readonly T* ToPointer() => (T*)_ptr; + + public readonly override bool Equals(object obj) + { + return obj is ArrayPtr other && Equals(other); + } + + public readonly bool Equals([AllowNull] ArrayPtr other) + { + return _ptr == other._ptr && Length == other.Length; + } + + public readonly override int GetHashCode() + { + return HashCode.Combine(_ptr, Length); + } + + public static bool operator ==(ArrayPtr left, ArrayPtr right) + { + return left.Equals(right); + } + + public static bool operator !=(ArrayPtr left, ArrayPtr right) + { + return !(left == right); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Memory/Box.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/Box.cs new file mode 100644 index 0000000..430f3c0 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/Box.cs @@ -0,0 +1,12 @@ +namespace Ryujinx.Common.Memory +{ + public class Box where T : unmanaged + { + public T Data; + + public Box() + { + Data = new T(); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Memory/IArray.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/IArray.cs new file mode 100644 index 0000000..e962c58 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/IArray.cs @@ -0,0 +1,28 @@ +using System; + +namespace Ryujinx.Common.Memory +{ + /// + /// Array interface. + /// + /// Element type + public interface IArray where T : unmanaged + { + /// + /// Used to index the array. + /// + /// Element index + /// Element at the specified index + ref T this[int index] { get; } + + /// + /// Number of elements on the array. + /// + int Length { get; } + + /// + /// Number of elements on the array. + /// + public Span AsSpan(); + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Memory/MemoryOwner.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/MemoryOwner.cs new file mode 100644 index 0000000..a91da88 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/MemoryOwner.cs @@ -0,0 +1,256 @@ +#nullable enable +using System; +using System.Buffers; +using System.Collections.Generic; +using System.Diagnostics.CodeAnalysis; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using System.Threading; + +namespace Ryujinx.Common.Memory +{ + /// + /// An implementation with an embedded length and fast + /// accessor, with memory allocated from . + /// + /// The type of item to store. + public sealed class MemoryOwner : IMemoryOwner + { + private static class ArrayPooling + { + public class Holder(T[]? array = null) : IComparable, IComparable + { + public int SkipCount; + public readonly T[]? Array = array; + + public int CompareTo(Holder? other) + { + return Array!.Length.CompareTo(other!.Array!.Length); + } + + public int CompareTo(int other) + { + int self = Array!.Length; + + if (self < other) + { + SkipCount++; + return -1; + } + + if (self > other * 4) + { + return 1; + } + + return 0; + } + } + + // ReSharper disable once StaticMemberInGenericType + private static int _maxCacheCount = 50; + + private const int MaxSkipCount = 50; + + static readonly List _pool = new(); + + // ReSharper disable once StaticMemberInGenericType + static readonly Lock _lock = new(); + + private static int BinarySearch(List list, int size) + { + int min = 0; + int max = list.Count-1; + + while (min <= max) + { + int mid = (min + max) / 2; + int comparison = list[mid].CompareTo(size); + if (comparison == 0) + { + return mid; + } + if (comparison < 0) + { + min = mid+1; + } + else + { + max = mid-1; + } + } + return ~min; + } + + public static T[] Get(int minimumSize) + { + lock (_lock) + { + int index = BinarySearch(_pool, minimumSize); + + if (index >= 0) + { + Holder holder = _pool[index]; + + _pool.Remove(holder); + return holder.Array!; + } + + return new T[minimumSize]; + } + } + + public static void Return(T[] array) + { + + lock (_lock) + { + Holder holder = new(array); + int i = _pool.BinarySearch(holder); + if (i < 0) + { + _pool.Insert(~i, holder); + } + + if (_pool.Count >= _maxCacheCount) + { + for (int index = 0; index < _pool.Count; index++) + { + Holder h = _pool[index]; + + if (h.SkipCount >= MaxSkipCount) + { + _pool.Remove(h); + index--; + } + } + + _maxCacheCount = _pool.Count * 2; + } + } + } + } + + private readonly int _length; + private T[]? _array; + + /// + /// Initializes a new instance of the class with the specified parameters. + /// + /// The length of the new memory buffer to use + private MemoryOwner(int length) + { + _length = length; + _array = ArrayPooling.Get(length); + } + + /// + /// Creates a new instance with the specified length. + /// + /// The length of the new memory buffer to use + /// A instance of the requested length + /// Thrown when is not valid + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static MemoryOwner Rent(int length) => new(length); + + /// + /// Creates a new instance with the specified length and the content cleared. + /// + /// The length of the new memory buffer to use + /// A instance of the requested length and the content cleared + /// Thrown when is not valid + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static MemoryOwner RentCleared(int length) + { + MemoryOwner result = new(length); + + result._array.AsSpan(0, length).Clear(); + + return result; + } + + /// + /// Creates a new instance with the content copied from the specified buffer. + /// + /// The buffer to copy + /// A instance with the same length and content as + public static MemoryOwner RentCopy(ReadOnlySpan buffer) + { + MemoryOwner result = new(buffer.Length); + + buffer.CopyTo(result._array); + + return result; + } + + /// + /// Gets the number of items in the current instance. + /// + public int Length + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => _length; + } + + /// + public Memory Memory + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get + { + T[]? array = _array; + + if (array is null) + { + ThrowObjectDisposedException(); + } + + return new(array, 0, _length); + } + } + + /// + /// Gets a wrapping the memory belonging to the current instance. + /// + /// + /// Uses a trick made possible by the .NET 6+ runtime array layout. + /// + public Span Span + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get + { + T[]? array = _array; + + if (array is null) + { + ThrowObjectDisposedException(); + } + + ref T firstElementRef = ref MemoryMarshal.GetArrayDataReference(array); + + return MemoryMarshal.CreateSpan(ref firstElementRef, _length); + } + } + + /// + public void Dispose() + { + T[]? array = Interlocked.Exchange(ref _array, null); + + if (array is not null) + { + ArrayPooling.Return(array); + } + } + + /// + /// Throws an when is . + /// + [DoesNotReturn] + private static void ThrowObjectDisposedException() + { + throw new ObjectDisposedException(nameof(MemoryOwner), "The buffer has already been disposed."); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Memory/MemoryStreamManager.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/MemoryStreamManager.cs new file mode 100644 index 0000000..14cfb78 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/MemoryStreamManager.cs @@ -0,0 +1,114 @@ +using Microsoft.IO; +using System; + +namespace Ryujinx.Common.Memory +{ + public static class MemoryStreamManager + { + private static readonly RecyclableMemoryStreamManager _shared = new(); + + private static readonly ObjectPool _streamPool = + new(() => new RecyclableMemoryStream(_shared, Guid.NewGuid(), null, 0)); + + /// + /// We don't expose the RecyclableMemoryStreamManager directly because version 2.x + /// returns them as MemoryStream. This Shared class is here to a) offer only the GetStream() versions we use + /// and b) return them as RecyclableMemoryStream so we don't have to cast. + /// + public static class Shared + { + /// + /// Retrieve a new MemoryStream object with no tag and a default initial capacity. + /// + /// A RecyclableMemoryStream + public static RecyclableMemoryStream GetStream() + { + RecyclableMemoryStream stream = _streamPool.Allocate(); + stream.SetLength(0); + + return stream; + } + + /// + /// Retrieve a new MemoryStream object with the contents copied from the provided + /// buffer. The provided buffer is not wrapped or used after construction. + /// + /// The new stream's position is set to the beginning of the stream when returned. + /// The byte buffer to copy data from + /// A RecyclableMemoryStream + public static RecyclableMemoryStream GetStream(byte[] buffer) + => GetStream(Guid.NewGuid(), null, buffer, 0, buffer.Length); + + /// + /// Retrieve a new MemoryStream object with the given tag and with contents copied from the provided + /// buffer. The provided buffer is not wrapped or used after construction. + /// + /// The new stream's position is set to the beginning of the stream when returned. + /// The byte buffer to copy data from + /// A RecyclableMemoryStream + public static RecyclableMemoryStream GetStream(ReadOnlySpan buffer) + => GetStream(Guid.NewGuid(), null, buffer); + + /// + /// Retrieve a new RecyclableMemoryStream object with the given tag and with contents copied from the provided + /// buffer. The provided buffer is not wrapped or used after construction. + /// + /// The new stream's position is set to the beginning of the stream when returned. + /// A unique identifier which can be used to trace usages of the stream + /// A tag which can be used to track the source of the stream + /// The byte buffer to copy data from + /// A RecyclableMemoryStream + public static RecyclableMemoryStream GetStream(Guid id, string tag, ReadOnlySpan buffer) + { + RecyclableMemoryStream stream = null; + try + { + stream = _streamPool.Allocate(); + stream.SetLength(0); + stream.Write(buffer); + stream.Position = 0; + return stream; + } + catch + { + stream?.Dispose(); + throw; + } + } + + /// + /// Retrieve a new RecyclableMemoryStream object with the given tag and with contents copied from the provided + /// buffer. The provided buffer is not wrapped or used after construction. + /// + /// The new stream's position is set to the beginning of the stream when returned + /// A unique identifier which can be used to trace usages of the stream + /// A tag which can be used to track the source of the stream + /// The byte buffer to copy data from + /// The offset from the start of the buffer to copy from + /// The number of bytes to copy from the buffer + /// A RecyclableMemoryStream + public static RecyclableMemoryStream GetStream(Guid id, string tag, byte[] buffer, int offset, int count) + { + RecyclableMemoryStream stream = null; + try + { + stream = _streamPool.Allocate(); + stream.SetLength(0); + stream.Write(buffer, offset, count); + stream.Position = 0; + return stream; + } + catch + { + stream?.Dispose(); + throw; + } + } + + public static void ReleaseStream(RecyclableMemoryStream stream) + { + _streamPool.Release(stream); + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Memory/PartialUnmaps/NativeReaderWriterLock.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/PartialUnmaps/NativeReaderWriterLock.cs new file mode 100644 index 0000000..1b54412 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/PartialUnmaps/NativeReaderWriterLock.cs @@ -0,0 +1,85 @@ +using System.Runtime.InteropServices; +using System.Threading; +using static Ryujinx.Common.Memory.PartialUnmaps.PartialUnmapHelpers; + +namespace Ryujinx.Common.Memory.PartialUnmaps +{ + /// + /// A simple implementation of a ReaderWriterLock which can be used from native code. + /// + [StructLayout(LayoutKind.Sequential, Pack = 4)] + public struct NativeReaderWriterLock + { + public int WriteLock; + public int ReaderCount; + + public static readonly int WriteLockOffset; + public static readonly int ReaderCountOffset; + + /// + /// Populates the field offsets for use when emitting native code. + /// + static NativeReaderWriterLock() + { + NativeReaderWriterLock instance = new(); + + WriteLockOffset = OffsetOf(ref instance, ref instance.WriteLock); + ReaderCountOffset = OffsetOf(ref instance, ref instance.ReaderCount); + } + + /// + /// Acquires the reader lock. + /// + public void AcquireReaderLock() + { + // Must take write lock for a very short time to become a reader. + + while (Interlocked.CompareExchange(ref WriteLock, 1, 0) != 0) + { + } + + Interlocked.Increment(ref ReaderCount); + + Interlocked.Exchange(ref WriteLock, 0); + } + + /// + /// Releases the reader lock. + /// + public void ReleaseReaderLock() + { + Interlocked.Decrement(ref ReaderCount); + } + + /// + /// Upgrades to a writer lock. The reader lock is temporarily released while obtaining the writer lock. + /// + public void UpgradeToWriterLock() + { + // Prevent any more threads from entering reader. + // If the write lock is already taken, wait for it to not be taken. + + Interlocked.Decrement(ref ReaderCount); + + while (Interlocked.CompareExchange(ref WriteLock, 1, 0) != 0) + { + } + + // Wait for reader count to drop to 0, then take the lock again as the only reader. + + while (Interlocked.CompareExchange(ref ReaderCount, 1, 0) != 0) + { + } + } + + /// + /// Downgrades from a writer lock, back to a reader one. + /// + public void DowngradeFromWriterLock() + { + // Release the WriteLock. + + Interlocked.Exchange(ref WriteLock, 0); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Memory/PartialUnmaps/PartialUnmapHelpers.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/PartialUnmaps/PartialUnmapHelpers.cs new file mode 100644 index 0000000..a5bab83 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/PartialUnmaps/PartialUnmapHelpers.cs @@ -0,0 +1,20 @@ +using System.Runtime.CompilerServices; + +namespace Ryujinx.Common.Memory.PartialUnmaps +{ + static class PartialUnmapHelpers + { + /// + /// Calculates a byte offset of a given field within a struct. + /// + /// Struct type + /// Field type + /// Parent struct + /// Field + /// The byte offset of the given field in the given struct + public static int OffsetOf(ref T2 storage, ref T target) + { + return (int)Unsafe.ByteOffset(ref Unsafe.As(ref storage), ref target); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Memory/PartialUnmaps/PartialUnmapState.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/PartialUnmaps/PartialUnmapState.cs new file mode 100644 index 0000000..29b2a89 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/PartialUnmaps/PartialUnmapState.cs @@ -0,0 +1,161 @@ +using System; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using System.Runtime.Versioning; +using System.Threading; +using static Ryujinx.Common.Memory.PartialUnmaps.PartialUnmapHelpers; + +namespace Ryujinx.Common.Memory.PartialUnmaps +{ + /// + /// State for partial unmaps. Intended to be used on Windows. + /// + [StructLayout(LayoutKind.Sequential, Pack = 1)] + public partial struct PartialUnmapState + { + public NativeReaderWriterLock PartialUnmapLock; + public int PartialUnmapsCount; + public ThreadLocalMap LocalCounts; + + public readonly static int PartialUnmapLockOffset; + public readonly static int PartialUnmapsCountOffset; + public readonly static int LocalCountsOffset; + + public readonly static nint GlobalState; + + [SupportedOSPlatform("windows")] + [LibraryImport("kernel32.dll")] + private static partial int GetCurrentThreadId(); + + [SupportedOSPlatform("windows")] + [LibraryImport("kernel32.dll", SetLastError = true)] + private static partial nint OpenThread(int dwDesiredAccess, [MarshalAs(UnmanagedType.Bool)] bool bInheritHandle, uint dwThreadId); + + [SupportedOSPlatform("windows")] + [LibraryImport("kernel32.dll", SetLastError = true)] + [return: MarshalAs(UnmanagedType.Bool)] + private static partial bool CloseHandle(nint hObject); + + [SupportedOSPlatform("windows")] + [LibraryImport("kernel32.dll", SetLastError = true)] + [return: MarshalAs(UnmanagedType.Bool)] + private static partial bool GetExitCodeThread(nint hThread, out uint lpExitCode); + + /// + /// Creates a global static PartialUnmapState and populates the field offsets. + /// + static unsafe PartialUnmapState() + { + PartialUnmapState instance = new(); + + PartialUnmapLockOffset = OffsetOf(ref instance, ref instance.PartialUnmapLock); + PartialUnmapsCountOffset = OffsetOf(ref instance, ref instance.PartialUnmapsCount); + LocalCountsOffset = OffsetOf(ref instance, ref instance.LocalCounts); + + int size = Unsafe.SizeOf(); + GlobalState = Marshal.AllocHGlobal(size); + Unsafe.InitBlockUnaligned((void*)GlobalState, 0, (uint)size); + } + + /// + /// Resets the global state. + /// + public static unsafe void Reset() + { + int size = Unsafe.SizeOf(); + Unsafe.InitBlockUnaligned((void*)GlobalState, 0, (uint)size); + } + + /// + /// Gets a reference to the global state. + /// + /// A reference to the global state + public static unsafe ref PartialUnmapState GetRef() + { + return ref Unsafe.AsRef((void*)GlobalState); + } + + /// + /// Checks if an access violation handler should retry execution due to a fault caused by partial unmap. + /// + /// + /// Due to Windows limitations, might need to unmap more memory than requested. + /// The additional memory that was unmapped is later remapped, however this leaves a time gap where the + /// memory might be accessed but is unmapped. Users of the API must compensate for that by catching the + /// access violation and retrying if it happened between the unmap and remap operation. + /// This method can be used to decide if retrying in such cases is necessary or not. + /// + /// This version of the function is not used, but serves as a reference for the native + /// implementation in ARMeilleure. + /// + /// True if execution should be retried, false otherwise + [SupportedOSPlatform("windows")] + public bool RetryFromAccessViolation() + { + PartialUnmapLock.AcquireReaderLock(); + + int threadID = GetCurrentThreadId(); + int threadIndex = LocalCounts.GetOrReserve(threadID, 0); + + if (threadIndex == -1) + { + // Out of thread local space... try again later. + + PartialUnmapLock.ReleaseReaderLock(); + + return true; + } + + ref int threadLocalPartialUnmapsCount = ref LocalCounts.GetValue(threadIndex); + + bool retry = threadLocalPartialUnmapsCount != PartialUnmapsCount; + if (retry) + { + threadLocalPartialUnmapsCount = PartialUnmapsCount; + } + + PartialUnmapLock.ReleaseReaderLock(); + + return retry; + } + + /// + /// Iterates and trims threads in the thread -> count map that + /// are no longer active. + /// + [SupportedOSPlatform("windows")] + public void TrimThreads() + { + const uint ExitCodeStillActive = 259; + const int ThreadQueryInformation = 0x40; + + Span ids = LocalCounts.ThreadIds.AsSpan(); + + for (int i = 0; i < ids.Length; i++) + { + int id = ids[i]; + + if (id != 0) + { + nint handle = OpenThread(ThreadQueryInformation, false, (uint)id); + + if (handle == nint.Zero) + { + Interlocked.CompareExchange(ref ids[i], 0, id); + } + else + { + GetExitCodeThread(handle, out uint exitCode); + + if (exitCode != ExitCodeStillActive) + { + Interlocked.CompareExchange(ref ids[i], 0, id); + } + + CloseHandle(handle); + } + } + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Memory/PartialUnmaps/ThreadLocalMap.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/PartialUnmaps/ThreadLocalMap.cs new file mode 100644 index 0000000..9364393 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/PartialUnmaps/ThreadLocalMap.cs @@ -0,0 +1,95 @@ +using System; +using System.Runtime.InteropServices; +using System.Threading; +using static Ryujinx.Common.Memory.PartialUnmaps.PartialUnmapHelpers; + +namespace Ryujinx.Common.Memory.PartialUnmaps +{ + /// + /// A simple fixed size thread safe map that can be used from native code. + /// Integer thread IDs map to corresponding structs. + /// + /// The value type for the map + [StructLayout(LayoutKind.Sequential, Pack = 1)] + public struct ThreadLocalMap where T : unmanaged + { + public const int MapSize = 20; + + public Array20 ThreadIds; + public Array20 Structs; + + public static readonly int ThreadIdsOffset; + public static readonly int StructsOffset; + + /// + /// Populates the field offsets for use when emitting native code. + /// + static ThreadLocalMap() + { + ThreadLocalMap instance = new(); + + ThreadIdsOffset = OffsetOf(ref instance, ref instance.ThreadIds); + StructsOffset = OffsetOf(ref instance, ref instance.Structs); + } + + /// + /// Gets the index of a given thread ID in the map, or reserves one. + /// When reserving a struct, its value is set to the given initial value. + /// Returns -1 when there is no space to reserve a new entry. + /// + /// Thread ID to use as a key + /// Initial value of the associated struct. + /// The index of the entry, or -1 if none + public int GetOrReserve(int threadId, T initial) + { + // Try get a match first. + + Span threadIdsSpan = ThreadIds.AsSpan(); + Span structsSpan = Structs.AsSpan(); + + for (int i = 0; i < MapSize; i++) + { + int compare = Interlocked.CompareExchange(ref threadIdsSpan[i], threadId, threadId); + + if (compare == threadId) + { + return i; + } + } + + // Try get a free entry. Since the id is assumed to be unique to this thread, we know it doesn't exist yet. + + for (int i = 0; i < MapSize; i++) + { + int compare = Interlocked.CompareExchange(ref threadIdsSpan[i], threadId, 0); + + if (compare == 0) + { + structsSpan[i] = initial; + return i; + } + } + + return -1; + } + + /// + /// Gets the struct value for a given map entry. + /// + /// Index of the entry + /// A reference to the struct value + public ref T GetValue(int index) + { + return ref Structs[index]; + } + + /// + /// Releases an entry from the map. + /// + /// Index of the entry to release + public void Release(int index) + { + Interlocked.Exchange(ref ThreadIds[index], 0); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Memory/Ptr.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/Ptr.cs new file mode 100644 index 0000000..82cd549 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/Ptr.cs @@ -0,0 +1,68 @@ +using System; +using System.Diagnostics.CodeAnalysis; +using System.Runtime.CompilerServices; + +namespace Ryujinx.Common.Memory +{ + /// + /// Represents a pointer to an unmanaged resource. + /// + /// Type of the unmanaged resource + public unsafe struct Ptr : IEquatable> where T : unmanaged + { + private nint _ptr; + + /// + /// Null pointer. + /// + public static Ptr Null => new() { _ptr = nint.Zero }; + + /// + /// True if the pointer is null, false otherwise. + /// + public readonly bool IsNull => _ptr == nint.Zero; + + /// + /// Gets a reference to the value. + /// + public readonly ref T Value => ref Unsafe.AsRef((void*)_ptr); + + /// + /// Creates a new pointer to an unmanaged resource. + /// + /// + /// For data on the heap, proper pinning is necessary during + /// use. Failure to do so will result in memory corruption and crashes. + /// + /// Reference to the unmanaged resource + public Ptr(ref T value) + { + _ptr = (nint)Unsafe.AsPointer(ref value); + } + + public readonly override bool Equals(object obj) + { + return obj is Ptr other && Equals(other); + } + + public readonly bool Equals([AllowNull] Ptr other) + { + return _ptr == other._ptr; + } + + public readonly override int GetHashCode() + { + return _ptr.GetHashCode(); + } + + public static bool operator ==(Ptr left, Ptr right) + { + return left.Equals(right); + } + + public static bool operator !=(Ptr left, Ptr right) + { + return !(left == right); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Memory/SpanOwner.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/SpanOwner.cs new file mode 100644 index 0000000..6c6a5dd --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/SpanOwner.cs @@ -0,0 +1,114 @@ +using System; +using System.Buffers; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; + +namespace Ryujinx.Common.Memory +{ + /// + /// A stack-only type that rents a buffer of a specified length from . + /// It does not implement to avoid being boxed, but should still be disposed. This + /// is easy since C# 8, which allows use of C# `using` constructs on any type that has a public Dispose() method. + /// To keep this type simple, fast, and read-only, it does not check or guard against multiple disposals. + /// For all these reasons, all usage should be with a `using` block or statement. + /// + /// The type of item to store. + public readonly ref struct SpanOwner + { + private readonly int _length; + private readonly T[] _array; + + /// + /// Initializes a new instance of the struct with the specified parameters. + /// + /// The length of the new memory buffer to use + private SpanOwner(int length) + { + _length = length; + _array = ArrayPool.Shared.Rent(length); + } + + /// + /// Gets an empty instance. + /// + public static SpanOwner Empty + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => new(0); + } + + /// + /// Creates a new instance with the specified length. + /// + /// The length of the new memory buffer to use + /// A instance of the requested length + /// Thrown when is not valid + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static SpanOwner Rent(int length) => new(length); + + /// + /// Creates a new instance with the length and the content cleared. + /// + /// The length of the new memory buffer to use + /// A instance of the requested length and the content cleared + /// Thrown when is not valid + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static SpanOwner RentCleared(int length) + { + SpanOwner result = new(length); + + result._array.AsSpan(0, length).Clear(); + + return result; + } + + /// + /// Creates a new instance with the content copied from the specified buffer. + /// + /// The buffer to copy + /// A instance with the same length and content as + public static SpanOwner RentCopy(ReadOnlySpan buffer) + { + SpanOwner result = new(buffer.Length); + + buffer.CopyTo(result._array); + + return result; + } + + /// + /// Gets the number of items in the current instance + /// + public int Length + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => _length; + } + + /// + /// Gets a wrapping the memory belonging to the current instance. + /// + /// + /// Uses a trick made possible by the .NET 6+ runtime array layout. + /// + public Span Span + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get + { + ref T firstElementRef = ref MemoryMarshal.GetArrayDataReference(_array); + + return MemoryMarshal.CreateSpan(ref firstElementRef, _length); + } + } + + /// + /// Implements the duck-typed method. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void Dispose() + { + ArrayPool.Shared.Return(_array, RuntimeHelpers.IsReferenceOrContainsReferences()); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Memory/SpanReader.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/SpanReader.cs new file mode 100644 index 0000000..eb7c680 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/SpanReader.cs @@ -0,0 +1,74 @@ +using System; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; + +namespace Ryujinx.Common.Memory +{ + public ref struct SpanReader + { + private ReadOnlySpan _input; + + public readonly int Length => _input.Length; + + public SpanReader(ReadOnlySpan input) + { + _input = input; + } + + public T Read() where T : unmanaged + { + T value = MemoryMarshal.Cast(_input)[0]; + + _input = _input[Unsafe.SizeOf()..]; + + return value; + } + + public bool TryRead(out T value) where T : unmanaged + { + int valueSize = Unsafe.SizeOf(); + + if (valueSize > _input.Length) + { + value = default; + + return false; + } + + value = MemoryMarshal.Cast(_input)[0]; + + _input = _input[valueSize..]; + + return true; + } + + public ReadOnlySpan GetSpan(int size) + { + ReadOnlySpan data = _input[..size]; + + _input = _input[size..]; + + return data; + } + + public ReadOnlySpan GetSpanSafe(int size) + { + return GetSpan((int)Math.Min((uint)_input.Length, (uint)size)); + } + + public readonly T ReadAt(int offset) where T : unmanaged + { + return MemoryMarshal.Cast(_input[offset..])[0]; + } + + public readonly ReadOnlySpan GetSpanAt(int offset, int size) + { + return _input.Slice(offset, size); + } + + public void Skip(int size) + { + _input = _input[size..]; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Memory/SpanWriter.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/SpanWriter.cs new file mode 100644 index 0000000..b61d5d0 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/SpanWriter.cs @@ -0,0 +1,45 @@ +using System; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; + +namespace Ryujinx.Common.Memory +{ + public ref struct SpanWriter + { + private Span _output; + + public readonly int Length => _output.Length; + + public SpanWriter(Span output) + { + _output = output; + } + + public void Write(T value) where T : unmanaged + { + MemoryMarshal.Cast(_output)[0] = value; + _output = _output[Unsafe.SizeOf()..]; + } + + public void Write(ReadOnlySpan data) + { + data.CopyTo(_output[..data.Length]); + _output = _output[data.Length..]; + } + + public readonly void WriteAt(int offset, T value) where T : unmanaged + { + MemoryMarshal.Cast(_output[offset..])[0] = value; + } + + public readonly void WriteAt(int offset, ReadOnlySpan data) + { + data.CopyTo(_output.Slice(offset, data.Length)); + } + + public void Skip(int size) + { + _output = _output[size..]; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Memory/StructArrayHelpers.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/StructArrayHelpers.cs new file mode 100644 index 0000000..dbc5003 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/StructArrayHelpers.cs @@ -0,0 +1,847 @@ +using System; +using System.Diagnostics.Contracts; +using System.Runtime.InteropServices; + +#pragma warning disable CS0169, IDE0051 // Remove unused private member +namespace Ryujinx.Common.Memory +{ + public struct Array1 : IArray where T : unmanaged + { + T _e0; + public readonly int Length => 1; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array2 : IArray where T : unmanaged + { + T _e0; + Array1 _other; + public readonly int Length => 2; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array3 : IArray where T : unmanaged + { + T _e0; + Array2 _other; + public readonly int Length => 3; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array4 : IArray where T : unmanaged + { + T _e0; + Array3 _other; + public readonly int Length => 4; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array5 : IArray where T : unmanaged + { + T _e0; + Array4 _other; + public readonly int Length => 5; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array6 : IArray where T : unmanaged + { + T _e0; + Array5 _other; + public readonly int Length => 6; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array7 : IArray where T : unmanaged + { + T _e0; + Array6 _other; + public readonly int Length => 7; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array8 : IArray where T : unmanaged + { + T _e0; + Array7 _other; + public readonly int Length => 8; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array9 : IArray where T : unmanaged + { + T _e0; + Array8 _other; + public readonly int Length => 9; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array10 : IArray where T : unmanaged + { + T _e0; + Array9 _other; + public readonly int Length => 10; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array11 : IArray where T : unmanaged + { + T _e0; + Array10 _other; + public readonly int Length => 11; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array12 : IArray where T : unmanaged + { + T _e0; + Array11 _other; + public readonly int Length => 12; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array13 : IArray where T : unmanaged + { + T _e0; + Array12 _other; + public readonly int Length => 13; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array14 : IArray where T : unmanaged + { + T _e0; + Array13 _other; + public readonly int Length => 14; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array15 : IArray where T : unmanaged + { + T _e0; + Array14 _other; + public readonly int Length => 15; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array16 : IArray where T : unmanaged + { + T _e0; + Array15 _other; + public readonly int Length => 16; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array17 : IArray where T : unmanaged + { + T _e0; + Array16 _other; + public readonly int Length => 17; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array18 : IArray where T : unmanaged + { + T _e0; + Array17 _other; + public readonly int Length => 18; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array19 : IArray where T : unmanaged + { + T _e0; + Array18 _other; + public readonly int Length => 19; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array20 : IArray where T : unmanaged + { + T _e0; + Array19 _other; + public readonly int Length => 20; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array21 : IArray where T : unmanaged + { + T _e0; + Array20 _other; + public readonly int Length => 21; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array22 : IArray where T : unmanaged + { + T _e0; + Array21 _other; + public readonly int Length => 22; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array23 : IArray where T : unmanaged + { + T _e0; + Array22 _other; + public readonly int Length => 23; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array24 : IArray where T : unmanaged + { + T _e0; + Array23 _other; + + public readonly int Length => 24; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array25 : IArray where T : unmanaged + { + T _e0; + Array24 _other; + + public readonly int Length => 25; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array26 : IArray where T : unmanaged + { + T _e0; + Array25 _other; + + public readonly int Length => 26; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array27 : IArray where T : unmanaged + { + T _e0; + Array26 _other; + + public readonly int Length => 27; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array28 : IArray where T : unmanaged + { + T _e0; + Array27 _other; + + public readonly int Length => 28; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array29 : IArray where T : unmanaged + { + T _e0; + Array28 _other; + + public readonly int Length => 29; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array30 : IArray where T : unmanaged + { + T _e0; + Array29 _other; + + public readonly int Length => 30; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array31 : IArray where T : unmanaged + { + T _e0; + Array30 _other; + + public readonly int Length => 31; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array32 : IArray where T : unmanaged + { + T _e0; + Array31 _other; + + public readonly int Length => 32; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array33 : IArray where T : unmanaged + { + T _e0; + Array32 _other; + + public readonly int Length => 33; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array34 : IArray where T : unmanaged + { + T _e0; + Array33 _other; + + public readonly int Length => 34; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array35 : IArray where T : unmanaged + { + T _e0; + Array34 _other; + + public readonly int Length => 35; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array36 : IArray where T : unmanaged + { + T _e0; + Array35 _other; + + public readonly int Length => 36; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array37 : IArray where T : unmanaged + { + T _e0; + Array36 _other; + + public readonly int Length => 37; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array38 : IArray where T : unmanaged + { + T _e0; + Array37 _other; + + public readonly int Length => 38; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array39 : IArray where T : unmanaged + { + T _e0; + Array38 _other; + + public readonly int Length => 39; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array40 : IArray where T : unmanaged + { + T _e0; + Array39 _other; + + public readonly int Length => 40; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array41 : IArray where T : unmanaged + { + T _e0; + Array40 _other; + + public readonly int Length => 41; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array42 : IArray where T : unmanaged + { + T _e0; + Array41 _other; + + public readonly int Length => 42; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array43 : IArray where T : unmanaged + { + T _e0; + Array42 _other; + + public readonly int Length => 43; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array44 : IArray where T : unmanaged + { + T _e0; + Array43 _other; + + public readonly int Length => 44; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array45 : IArray where T : unmanaged + { + T _e0; + Array44 _other; + + public readonly int Length => 45; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array46 : IArray where T : unmanaged + { + T _e0; + Array45 _other; + + public readonly int Length => 46; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array47 : IArray where T : unmanaged + { + T _e0; + Array46 _other; + + public readonly int Length => 47; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array48 : IArray where T : unmanaged + { + T _e0; + Array47 _other; + + public readonly int Length => 48; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array49 : IArray where T : unmanaged + { + T _e0; + Array48 _other; + + public readonly int Length => 49; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array50 : IArray where T : unmanaged + { + T _e0; + Array49 _other; + + public readonly int Length => 50; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array51 : IArray where T : unmanaged + { + T _e0; + Array50 _other; + + public readonly int Length => 51; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array52 : IArray where T : unmanaged + { + T _e0; + Array51 _other; + + public readonly int Length => 52; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array53 : IArray where T : unmanaged + { + T _e0; + Array52 _other; + + public readonly int Length => 53; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array54 : IArray where T : unmanaged + { + T _e0; + Array53 _other; + + public readonly int Length => 54; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array55 : IArray where T : unmanaged + { + T _e0; + Array54 _other; + + public readonly int Length => 55; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array56 : IArray where T : unmanaged + { + T _e0; + Array55 _other; + + public readonly int Length => 56; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array57 : IArray where T : unmanaged + { + T _e0; + Array56 _other; + + public readonly int Length => 57; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array58 : IArray where T : unmanaged + { + T _e0; + Array57 _other; + + public readonly int Length => 58; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array59 : IArray where T : unmanaged + { + T _e0; + Array58 _other; + + public readonly int Length => 59; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array60 : IArray where T : unmanaged + { + T _e0; + Array59 _other; + public readonly int Length => 60; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array61 : IArray where T : unmanaged + { + T _e0; + Array60 _other; + public readonly int Length => 61; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array62 : IArray where T : unmanaged + { + T _e0; + Array61 _other; + public readonly int Length => 62; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array63 : IArray where T : unmanaged + { + T _e0; + Array62 _other; + public readonly int Length => 63; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array64 : IArray where T : unmanaged + { + T _e0; + Array63 _other; + public readonly int Length => 64; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array65 : IArray where T : unmanaged + { + T _e0; + Array64 _other; + public readonly int Length => 65; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array73 : IArray where T : unmanaged + { + T _e0; + Array64 _other; + Array8 _other2; + public readonly int Length => 73; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array96 : IArray where T : unmanaged + { + T _e0; + Array64 _other; + Array31 _other2; + public readonly int Length => 96; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array127 : IArray where T : unmanaged + { + T _e0; + Array64 _other; + Array62 _other2; + public readonly int Length => 127; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array128 : IArray where T : unmanaged + { + T _e0; + Array64 _other; + Array63 _other2; + public readonly int Length => 128; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array140 : IArray where T : unmanaged + { + T _e0; + Array64 _other; + Array64 _other2; + Array11 _other3; + public readonly int Length => 140; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array256 : IArray where T : unmanaged + { + T _e0; + Array128 _other; + Array127 _other2; + public readonly int Length => 256; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } + + public struct Array384 : IArray where T : unmanaged + { + T _e0; + Array64 _other; + Array64 _other2; + Array64 _other3; + Array64 _other4; + Array64 _other5; + Array63 _other6; + public readonly int Length => 384; + public ref T this[int index] => ref AsSpan()[index]; + + [Pure] + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _e0, Length); + } +} +#pragma warning restore CS0169, IDE0051 diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Memory/StructByteArrayHelpers.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/StructByteArrayHelpers.cs new file mode 100644 index 0000000..ff92936 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Memory/StructByteArrayHelpers.cs @@ -0,0 +1,89 @@ +using System; +using System.Runtime.InteropServices; + +namespace Ryujinx.Common.Memory +{ + [StructLayout(LayoutKind.Sequential, Size = Size, Pack = 1)] + public struct ByteArray128 : IArray + { + private const int Size = 128; + + byte _element; + + public readonly int Length => Size; + public ref byte this[int index] => ref AsSpan()[index]; + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _element, Size); + } + + [StructLayout(LayoutKind.Sequential, Size = Size, Pack = 1)] + public struct ByteArray256 : IArray + { + private const int Size = 256; + + byte _element; + + public readonly int Length => Size; + public ref byte this[int index] => ref AsSpan()[index]; + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _element, Size); + } + + [StructLayout(LayoutKind.Sequential, Size = Size, Pack = 1)] + public struct ByteArray512 : IArray + { + private const int Size = 512; + + byte _element; + + public readonly int Length => Size; + public ref byte this[int index] => ref AsSpan()[index]; + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _element, Size); + } + + [StructLayout(LayoutKind.Sequential, Size = Size, Pack = 1)] + public struct ByteArray1024 : IArray + { + private const int Size = 1024; + + byte _element; + + public readonly int Length => Size; + public ref byte this[int index] => ref AsSpan()[index]; + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _element, Size); + } + + [StructLayout(LayoutKind.Sequential, Size = Size, Pack = 1)] + public struct ByteArray2048 : IArray + { + private const int Size = 2048; + + byte _element; + + public readonly int Length => Size; + public ref byte this[int index] => ref AsSpan()[index]; + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _element, Size); + } + + [StructLayout(LayoutKind.Sequential, Size = Size, Pack = 1)] + public struct ByteArray3000 : IArray + { + private const int Size = 3000; + + byte _element; + + public readonly int Length => Size; + public ref byte this[int index] => ref AsSpan()[index]; + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _element, Size); + } + + [StructLayout(LayoutKind.Sequential, Size = Size, Pack = 1)] + public struct ByteArray4096 : IArray + { + private const int Size = 4096; + + byte _element; + + public readonly int Length => Size; + public ref byte this[int index] => ref AsSpan()[index]; + public Span AsSpan() => MemoryMarshal.CreateSpan(ref _element, Size); + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/PerformanceCounter.cs b/third_party/maxwell_shader/src/Ryujinx.Common/PerformanceCounter.cs new file mode 100644 index 0000000..f6542d1 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/PerformanceCounter.cs @@ -0,0 +1,82 @@ +using System.Diagnostics; + +namespace Ryujinx.Common +{ + public static class PerformanceCounter + { + private static readonly double _ticksToNs; + + /// + /// Represents the number of ticks in 1 day. + /// + public static long TicksPerDay { get; } + + /// + /// Represents the number of ticks in 1 hour. + /// + public static long TicksPerHour { get; } + + /// + /// Represents the number of ticks in 1 minute. + /// + public static long TicksPerMinute { get; } + + /// + /// Represents the number of ticks in 1 second. + /// + public static long TicksPerSecond { get; } + + /// + /// Represents the number of ticks in 1 millisecond. + /// + public static long TicksPerMillisecond { get; } + + /// + /// Gets the number of ticks elapsed since the system started. + /// + public static long ElapsedTicks + { + get + { + return Stopwatch.GetTimestamp(); + } + } + + /// + /// Gets the number of milliseconds elapsed since the system started. + /// + public static long ElapsedMilliseconds + { + get + { + long timestamp = Stopwatch.GetTimestamp(); + + return timestamp / TicksPerMillisecond; + } + } + + /// + /// Gets the number of nanoseconds elapsed since the system started. + /// + public static long ElapsedNanoseconds + { + get + { + long timestamp = Stopwatch.GetTimestamp(); + + return (long)(timestamp * _ticksToNs); + } + } + + static PerformanceCounter() + { + TicksPerMillisecond = Stopwatch.Frequency / 1000; + TicksPerSecond = Stopwatch.Frequency; + TicksPerMinute = TicksPerSecond * 60; + TicksPerHour = TicksPerMinute * 60; + TicksPerDay = TicksPerHour * 24; + + _ticksToNs = 1000000000.0 / Stopwatch.Frequency; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Pools/ObjectPool.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Pools/ObjectPool.cs new file mode 100644 index 0000000..062a808 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Pools/ObjectPool.cs @@ -0,0 +1,34 @@ +using System; +using System.Collections.Concurrent; + +namespace Ryujinx.Common +{ + public class ObjectPool(Func factory, int size = -1) + where T : class + { + private int _size = size; + private readonly ConcurrentBag _items = new(); + + public T Allocate() + { + bool success = _items.TryTake(out T instance); + + if (!success) + { + instance = factory(); + } + + return instance; + } + + public void Release(T obj) + { + if (_size < 0 || _items.Count < _size) + { + _items.Add(obj); + } + } + + public void Clear() => _items.Clear(); + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Pools/SharedPools.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Pools/SharedPools.cs new file mode 100644 index 0000000..915ddd3 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Pools/SharedPools.cs @@ -0,0 +1,17 @@ +namespace Ryujinx.Common +{ + public static class SharedPools + { + private static class DefaultPool + where T : class, new() + { + public static readonly ObjectPool Instance = new(() => new T(), 20); + } + + public static ObjectPool Default() + where T : class, new() + { + return DefaultPool.Instance; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Pools/ThreadStaticArray.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Pools/ThreadStaticArray.cs new file mode 100644 index 0000000..755836e --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Pools/ThreadStaticArray.cs @@ -0,0 +1,17 @@ +using System; + +namespace Ryujinx.Common.Pools +{ + public static class ThreadStaticArray + { + [ThreadStatic] + private static T[] _array; + + public static ref T[] Get() + { + _array ??= new T[1]; + + return ref _array; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/IPreciseSleepEvent.cs b/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/IPreciseSleepEvent.cs new file mode 100644 index 0000000..1bb097b --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/IPreciseSleepEvent.cs @@ -0,0 +1,38 @@ +using System; + +namespace Ryujinx.Common.PreciseSleep +{ + /// + /// An event which works similarly to an AutoResetEvent, but is backed by a + /// more precise timer that allows waits of less than a millisecond. + /// + public interface IPreciseSleepEvent : IDisposable + { + /// + /// Adjust a timepoint to better fit the host clock. + /// When no adjustment is made, the input timepoint will be returned. + /// + /// Timepoint to adjust + /// Requested timeout in nanoseconds + /// Adjusted timepoint + long AdjustTimePoint(long timePoint, long timeoutNs); + + /// + /// Sleep until a timepoint, or a signal is received. + /// Given no signal, may wake considerably before, or slightly after the timeout. + /// + /// Timepoint to sleep until + /// True if signalled or waited, false if a wait could not be performed + bool SleepUntil(long timePoint); + + /// + /// Sleep until a signal is received. + /// + void Sleep(); + + /// + /// Signal the event, waking any sleeping thread or the next attempted sleep. + /// + void Signal(); + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/Nanosleep.cs b/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/Nanosleep.cs new file mode 100644 index 0000000..8b1cde9 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/Nanosleep.cs @@ -0,0 +1,160 @@ +using System; +using System.Runtime.InteropServices; +using System.Runtime.Versioning; + +namespace Ryujinx.Common.PreciseSleep +{ + /// + /// Access to Linux/MacOS nanosleep, with platform specific bias to improve precision. + /// + [SupportedOSPlatform("macos")] + [SupportedOSPlatform("linux")] + [SupportedOSPlatform("android")] + [SupportedOSPlatform("ios")] + internal static partial class Nanosleep + { + private const long LinuxBaseNanosleepBias = 50000; // 0.05ms + + // Penalty for max allowed sleep duration + private const long LinuxNanosleepAccuracyPenaltyThreshold = 200000; // 0.2ms + private const long LinuxNanosleepAccuracyPenalty = 30000; // 0.03ms + + // Penalty for base sleep duration + private const long LinuxNanosleepBasePenaltyThreshold = 500000; // 0.5ms + private const long LinuxNanosleepBasePenalty = 30000; // 0.03ms + private const long LinuxNanosleepPenaltyPerMillisecond = 18000; // 0.018ms + private const long LinuxNanosleepPenaltyCap = 18000; // 0.018ms + + private const long LinuxStrictBiasOffset = 150_000; // 0.15ms + + // Nanosleep duration is biased depending on the requested timeout on MacOS. + // These match the results when measuring on an M1 processor at AboveNormal priority. + private const long MacosBaseNanosleepBias = 5000; // 0.005ms + private const long MacosBiasPerMillisecond = 140000; // 0.14ms + private const long MacosBiasMaxNanoseconds = 20_000_000; // 20ms + private const long MacosStrictBiasOffset = 150_000; // 0.15ms + + public static long Bias { get; } + + /// + /// Get bias for a given nanosecond timeout. + /// Some platforms calculate their bias differently, this method can be used to counteract it. + /// + /// Nanosecond timeout + /// Bias in nanoseconds + public static long GetBias(long timeoutNs) + { + if (OperatingSystem.IsMacOS() || OperatingSystem.IsIOS()) + { + long biasNs = Math.Min(timeoutNs, MacosBiasMaxNanoseconds); + return MacosBaseNanosleepBias + biasNs * MacosBiasPerMillisecond / 1_000_000; + } + else + { + long bias = LinuxBaseNanosleepBias; + + if (timeoutNs > LinuxNanosleepBasePenaltyThreshold) + { + long penalty = (timeoutNs - LinuxNanosleepBasePenaltyThreshold) * LinuxNanosleepPenaltyPerMillisecond / 1_000_000; + bias += LinuxNanosleepBasePenalty + Math.Min(LinuxNanosleepPenaltyCap, penalty); + } + + return bias; + } + } + + /// + /// Get a stricter bias for a given nanosecond timeout, + /// which can improve the chances the sleep completes before the timeout. + /// Some platforms calculate their bias differently, this method can be used to counteract it. + /// + /// Nanosecond timeout + /// Strict bias in nanoseconds + public static long GetStrictBias(long timeoutNs) + { + if (OperatingSystem.IsMacOS() || OperatingSystem.IsIOS()) + { + return GetBias(timeoutNs) + MacosStrictBiasOffset; + } + else + { + long bias = GetBias(timeoutNs) + LinuxStrictBiasOffset; + + if (timeoutNs > LinuxNanosleepAccuracyPenaltyThreshold) + { + bias += LinuxNanosleepAccuracyPenalty; + } + + return bias; + } + } + + static Nanosleep() + { + Bias = GetBias(0); + } + + [StructLayout(LayoutKind.Sequential)] + private struct Timespec + { + public long tv_sec; // Seconds + public long tv_nsec; // Nanoseconds + } + + [LibraryImport("libc", SetLastError = true)] + private static partial int nanosleep(ref Timespec req, ref Timespec rem); + + /// + /// Convert a timeout in nanoseconds to a timespec for nanosleep. + /// + /// Timeout in nanoseconds + /// Timespec for nanosleep + private static Timespec GetTimespecFromNanoseconds(ulong nanoseconds) + { + return new Timespec + { + tv_sec = (long)(nanoseconds / 1_000_000_000), + tv_nsec = (long)(nanoseconds % 1_000_000_000) + }; + } + + /// + /// Sleep for approximately a given time period in nanoseconds. + /// + /// Time to sleep for in nanoseconds + public static void Sleep(long nanoseconds) + { + nanoseconds -= GetBias(nanoseconds); + + if (nanoseconds >= 0) + { + Timespec req = GetTimespecFromNanoseconds((ulong)nanoseconds); + Timespec rem = new(); + + nanosleep(ref req, ref rem); + } + } + + /// + /// Sleep for at most a given time period in nanoseconds. + /// Uses a stricter bias to wake before the requested duration. + /// + /// + /// Due to OS scheduling behaviour, this timeframe may still be missed. + /// + /// Maximum allowed time for sleep + public static void SleepAtMost(long nanoseconds) + { + // Stricter bias to ensure we wake before the timepoint. + nanoseconds -= GetStrictBias(nanoseconds); + + if (nanoseconds >= 0) + { + Timespec req = GetTimespecFromNanoseconds((ulong)nanoseconds); + Timespec rem = new(); + + nanosleep(ref req, ref rem); + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/NanosleepEvent.cs b/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/NanosleepEvent.cs new file mode 100644 index 0000000..267ead4 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/NanosleepEvent.cs @@ -0,0 +1,85 @@ +using System; +using System.Runtime.Versioning; +using System.Threading; + +namespace Ryujinx.Common.PreciseSleep +{ + /// + /// A precise sleep event for linux and macos that uses nanosleep for more precise timeouts. + /// + [SupportedOSPlatform("macos")] + [SupportedOSPlatform("linux")] + [SupportedOSPlatform("android")] + [SupportedOSPlatform("ios")] + internal class NanosleepEvent : IPreciseSleepEvent + { + private readonly AutoResetEvent _waitEvent = new(false); + private readonly NanosleepPool _pool; + + public NanosleepEvent() + { + _pool = new NanosleepPool(_waitEvent); + } + + public long AdjustTimePoint(long timePoint, long timeoutNs) + { + // No adjustment + return timePoint; + } + + public bool SleepUntil(long timePoint) + { + long now = PerformanceCounter.ElapsedTicks; + long delta = (timePoint - now); + long ms = Math.Min(delta / PerformanceCounter.TicksPerMillisecond, int.MaxValue); + long ns = (delta * 1_000_000) / PerformanceCounter.TicksPerMillisecond; + + if (ms > 0) + { + _waitEvent.WaitOne((int)ms); + + return true; + } + else if (ns - Nanosleep.Bias > 0) + { + // Don't bother starting a sleep if there's already a signal active. + if (_waitEvent.WaitOne(0)) + { + return true; + } + + // The 1ms wait will be interrupted by the nanosleep timeout if it completes. + if (!_pool.SleepAndSignal(ns, timePoint)) + { + // Too many threads on the pool. + return false; + } + + _waitEvent.WaitOne(1); + _pool.IgnoreSignal(); + + return true; + } + + return false; + } + + public void Sleep() + { + _waitEvent.WaitOne(); + } + + public void Signal() + { + _waitEvent.Set(); + } + + public void Dispose() + { + GC.SuppressFinalize(this); + + _pool.Dispose(); + _waitEvent.Dispose(); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/NanosleepPool.cs b/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/NanosleepPool.cs new file mode 100644 index 0000000..90252eb --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/NanosleepPool.cs @@ -0,0 +1,228 @@ +using System; +using System.Collections.Generic; +using System.Runtime.Versioning; +using System.Threading; + +namespace Ryujinx.Common.PreciseSleep +{ + /// + /// A pool of threads used to allow "interruptable" nanosleep for a single target event. + /// + [SupportedOSPlatform("macos")] + [SupportedOSPlatform("linux")] + [SupportedOSPlatform("android")] + [SupportedOSPlatform("ios")] + internal class NanosleepPool : IDisposable + { + public const int MaxThreads = 8; + + /// + /// A thread that nanosleeps and may signal an event on wake. + /// When a thread is assigned a nanosleep to perform, it also gets a signal ID. + /// The pool's target event is only signalled if this ID matches the latest dispatched one. + /// + private class NanosleepThread : IDisposable + { + private static readonly long _timePointEpsilon; + + static NanosleepThread() + { + _timePointEpsilon = PerformanceCounter.TicksPerMillisecond / 100; // 0.01ms + } + + private readonly Thread _thread; + private readonly NanosleepPool _parent; + private readonly AutoResetEvent _newWaitEvent; + private bool _running = true; + + private long _signalId; + private long _nanoseconds; + private long _timePoint; + + public long SignalId => _signalId; + + /// + /// Creates a new NanosleepThread for a parent pool, with a specified thread ID. + /// + /// Parent NanosleepPool + /// Thread ID + public NanosleepThread(NanosleepPool parent, int id) + { + _parent = parent; + _newWaitEvent = new(false); + + _thread = new Thread(Loop) + { + Name = $"Common.Nanosleep.{id}", + Priority = ThreadPriority.AboveNormal, + IsBackground = true + }; + + _thread.Start(); + } + + /// + /// Service requests to perform a nanosleep, signal parent pool when complete. + /// + private void Loop() + { + _newWaitEvent.WaitOne(); + + while (_running) + { + Nanosleep.Sleep(_nanoseconds); + + _parent.Signal(this); + _newWaitEvent.WaitOne(); + } + + _newWaitEvent.Dispose(); + } + + /// + /// Assign a nanosleep for this thread to perform, then signal at the end. + /// + /// Nanoseconds to sleep + /// Signal ID + /// Target timepoint + public void SleepAndSignal(long nanoseconds, long signalId, long timePoint) + { + _signalId = signalId; + _nanoseconds = nanoseconds; + _timePoint = timePoint; + _newWaitEvent.Set(); + } + + /// + /// Resurrect an active nanosleep's signal if its target timepoint is a close enough match. + /// + /// New signal id to assign the nanosleep + /// Target timepoint + /// True if resurrected, false otherwise + public bool Resurrect(long signalId, long timePoint) + { + if (Math.Abs(timePoint - _timePoint) < _timePointEpsilon) + { + _signalId = signalId; + + return true; + } + + return false; + } + + /// + /// Dispose the NanosleepThread, interrupting its worker loop. + /// + public void Dispose() + { + if (_running) + { + _running = false; + _newWaitEvent.Set(); + } + } + } + + private readonly Lock _lock = new(); + private readonly List _threads = []; + private readonly List _active = []; + private readonly Stack _free = new(); + private readonly AutoResetEvent _signalTarget; + + private long _signalId; + + /// + /// Creates a new NanosleepPool with a target event to signal when a nanosleep completes. + /// + /// Event to signal when nanosleeps complete + public NanosleepPool(AutoResetEvent signalTarget) + { + _signalTarget = signalTarget; + } + + /// + /// Signal the target event (if the source sleep has not been superseded) + /// and free the nanosleep thread. + /// + /// Nanosleep thread that completed + private void Signal(NanosleepThread thread) + { + lock (_lock) + { + _active.Remove(thread); + _free.Push(thread); + + if (thread.SignalId == _signalId) + { + _signalTarget.Set(); + } + } + } + + /// + /// Sleep for the given number of nanoseconds and signal the target event. + /// This does not block the caller thread. + /// + /// Nanoseconds to sleep + /// Target timepoint + /// True if the signal will be set, false otherwise + public bool SleepAndSignal(long nanoseconds, long timePoint) + { + lock (_lock) + { + _signalId++; + + // Check active sleeps, if any line up with the requested timepoint then resurrect that nanosleep. + foreach (NanosleepThread existing in _active) + { + if (existing.Resurrect(_signalId, timePoint)) + { + return true; + } + } + + if (!_free.TryPop(out NanosleepThread thread)) + { + if (_threads.Count >= MaxThreads) + { + return false; + } + + thread = new NanosleepThread(this, _threads.Count); + + _threads.Add(thread); + } + + _active.Add(thread); + + thread.SleepAndSignal(nanoseconds, _signalId, timePoint); + + return true; + } + } + + /// + /// Ignore the latest nanosleep. + /// + public void IgnoreSignal() + { + _signalId++; + } + + /// + /// Dispose the NanosleepPool, disposing all of its active threads. + /// + public void Dispose() + { + GC.SuppressFinalize(this); + + foreach (NanosleepThread thread in _threads) + { + thread.Dispose(); + } + + _threads.Clear(); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/PreciseSleepHelper.cs b/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/PreciseSleepHelper.cs new file mode 100644 index 0000000..f590bb3 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/PreciseSleepHelper.cs @@ -0,0 +1,104 @@ +using Ryujinx.Common.SystemInterop; +using System; +using System.Threading; + +namespace Ryujinx.Common.PreciseSleep +{ + public static class PreciseSleepHelper + { + /// + /// Create a precise sleep event for the current platform. + /// + /// A precise sleep event + public static IPreciseSleepEvent CreateEvent() + { + if (OperatingSystem.IsLinux() || OperatingSystem.IsMacOS() || OperatingSystem.IsIOS() || OperatingSystem.IsAndroid()) + { + return new NanosleepEvent(); + } + else if (OperatingSystem.IsWindows()) + { + return new WindowsSleepEvent(); + } + else + { + return new SleepEvent(); + } + } + + /// + /// Sleeps up to the closest point to the timepoint that the OS reasonably allows. + /// The provided event is used by the timer to wake the current thread, and should not be signalled from any other source. + /// + /// Event used to wake this thread + /// Target timepoint in host ticks + public static void SleepUntilTimePoint(EventWaitHandle evt, long timePoint) + { + if (OperatingSystem.IsWindows()) + { + WindowsGranularTimer.Instance.SleepUntilTimePointWithoutExternalSignal(evt, timePoint); + } + else + { + // Events might oversleep by a little, depending on OS. + // We don't want to miss the timepoint, so bias the wait to be lower. + // Nanosleep can possibly handle it better, too. + long accuracyBias = PerformanceCounter.TicksPerMillisecond / 2; + long now = PerformanceCounter.ElapsedTicks + accuracyBias; + long ms = Math.Min((timePoint - now) / PerformanceCounter.TicksPerMillisecond, int.MaxValue); + + if (ms > 0) + { + evt.WaitOne((int)ms); + } + + if (OperatingSystem.IsLinux() || OperatingSystem.IsMacOS() || OperatingSystem.IsIOS() || OperatingSystem.IsAndroid()) + { + // Do a nanosleep. + now = PerformanceCounter.ElapsedTicks; + long ns = ((timePoint - now) * 1_000_000) / PerformanceCounter.TicksPerMillisecond; + + Nanosleep.SleepAtMost(ns); + } + } + } + + /// + /// Spinwait until the given timepoint. If wakeSignal is or becomes 1, return early. + /// Thread is allowed to yield. + /// + /// Target timepoint in host ticks + /// Returns early if this is set to 1 + public static void SpinWaitUntilTimePoint(long timePoint, ref long wakeSignal) + { + SpinWait spinWait = new(); + + while (Interlocked.Read(ref wakeSignal) != 1 && PerformanceCounter.ElapsedTicks < timePoint) + { + // Our time is close - don't let SpinWait go off and potentially Thread.Sleep(). + if (spinWait.NextSpinWillYield) + { + Thread.Yield(); + + spinWait.Reset(); + } + else + { + spinWait.SpinOnce(); + } + } + } + + /// + /// Spinwait until the given timepoint, with no opportunity to wake early. + /// + /// Target timepoint in host ticks + public static void SpinWaitUntilTimePoint(long timePoint) + { + while (PerformanceCounter.ElapsedTicks < timePoint) + { + Thread.SpinWait(5); + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/SleepEvent.cs b/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/SleepEvent.cs new file mode 100644 index 0000000..2dd5ce8 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/SleepEvent.cs @@ -0,0 +1,51 @@ +using System; +using System.Threading; + +namespace Ryujinx.Common.PreciseSleep +{ + /// + /// A cross-platform precise sleep event that has millisecond granularity. + /// + internal class SleepEvent : IPreciseSleepEvent + { + private readonly AutoResetEvent _waitEvent = new(false); + + public long AdjustTimePoint(long timePoint, long timeoutNs) + { + // No adjustment + return timePoint; + } + + public bool SleepUntil(long timePoint) + { + long now = PerformanceCounter.ElapsedTicks; + long ms = Math.Min((timePoint - now) / PerformanceCounter.TicksPerMillisecond, int.MaxValue); + + if (ms > 0) + { + _waitEvent.WaitOne((int)ms); + + return true; + } + + return false; + } + + public void Sleep() + { + _waitEvent.WaitOne(); + } + + public void Signal() + { + _waitEvent.Set(); + } + + public void Dispose() + { + GC.SuppressFinalize(this); + + _waitEvent.Dispose(); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/WindowsGranularTimer.cs b/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/WindowsGranularTimer.cs new file mode 100644 index 0000000..6146e25 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/WindowsGranularTimer.cs @@ -0,0 +1,220 @@ +using System; +using System.Collections.Generic; +using System.Runtime.InteropServices; +using System.Runtime.Versioning; +using System.Threading; + +namespace Ryujinx.Common.SystemInterop +{ + /// + /// Timer that attempts to align with the hardware timer interrupt, + /// and can alert listeners on ticks. + /// + [SupportedOSPlatform("windows")] + internal partial class WindowsGranularTimer + { + private const int MinimumGranularity = 5000; + + private static readonly WindowsGranularTimer _instance = new(); + public static WindowsGranularTimer Instance => _instance; + + private readonly struct WaitingObject + { + public readonly long Id; + public readonly EventWaitHandle Signal; + public readonly long TimePoint; + + public WaitingObject(long id, EventWaitHandle signal, long timePoint) + { + Id = id; + Signal = signal; + TimePoint = timePoint; + } + } + + [LibraryImport("ntdll.dll", SetLastError = true)] + private static partial int NtSetTimerResolution(int DesiredResolution, [MarshalAs(UnmanagedType.Bool)] bool SetResolution, out int CurrentResolution); + + [LibraryImport("ntdll.dll", SetLastError = true)] + private static partial int NtQueryTimerResolution(out int MaximumResolution, out int MinimumResolution, out int CurrentResolution); + + [LibraryImport("ntdll.dll", SetLastError = true)] + private static partial uint NtDelayExecution([MarshalAs(UnmanagedType.Bool)] bool Alertable, ref long DelayInterval); + + public long GranularityNs => _granularityNs; + public long GranularityTicks => _granularityTicks; + + private readonly Thread _timerThread; + private long _granularityNs = MinimumGranularity * 100L; + private long _granularityTicks; + private long _lastTicks = PerformanceCounter.ElapsedTicks; + private long _lastId; + + private readonly Lock _lock = new(); + private readonly List _waitingObjects = []; + + private WindowsGranularTimer() + { + _timerThread = new Thread(Loop) + { + IsBackground = true, + Name = "Common.WindowsTimer", + Priority = ThreadPriority.Highest + }; + + _timerThread.Start(); + } + + /// + /// Measure and initialize the timer's target granularity. + /// + private void Initialize() + { + NtQueryTimerResolution(out _, out int min, out int curr); + + if (min > 0) + { + min = Math.Max(min, MinimumGranularity); + + _granularityNs = min * 100L; + NtSetTimerResolution(min, true, out _); + } + else + { + _granularityNs = curr * 100L; + } + + _granularityTicks = (_granularityNs * PerformanceCounter.TicksPerMillisecond) / 1_000_000; + } + + /// + /// Main loop for the timer thread. Wakes every clock tick and signals any listeners, + /// as well as keeping track of clock alignment. + /// + private void Loop() + { + Initialize(); + while (true) + { + long delayInterval = -1; // Next tick + NtSetTimerResolution((int)(_granularityNs / 100), true, out _); + NtDelayExecution(false, ref delayInterval); + + long newTicks = PerformanceCounter.ElapsedTicks; + long nextTicks = newTicks + _granularityTicks; + + lock (_lock) + { + for (int i = 0; i < _waitingObjects.Count; i++) + { + if (nextTicks > _waitingObjects[i].TimePoint) + { + // The next clock tick will be after the timepoint, we need to signal now. + _waitingObjects[i].Signal.Set(); + + _waitingObjects.RemoveAt(i--); + } + } + + _lastTicks = newTicks; + } + } + } + + /// + /// Sleep until a timepoint. + /// + /// Reset event to use to be awoken by the clock tick, or an external signal + /// Target timepoint + /// True if waited or signalled, false otherwise + public bool SleepUntilTimePoint(AutoResetEvent evt, long timePoint) + { + if (evt.WaitOne(0)) + { + return true; + } + + long id; + + lock (_lock) + { + // Return immediately if the next tick is after the requested timepoint. + long nextTicks = _lastTicks + _granularityTicks; + + if (nextTicks > timePoint) + { + return false; + } + + id = ++_lastId; + + _waitingObjects.Add(new WaitingObject(id, evt, timePoint)); + } + + evt.WaitOne(); + + lock (_lock) + { + for (int i = 0; i < _waitingObjects.Count; i++) + { + if (id == _waitingObjects[i].Id) + { + _waitingObjects.RemoveAt(i--); + break; + } + } + } + + return true; + } + + /// + /// Sleep until a timepoint, but don't expect any external signals. + /// + /// + /// Saves some effort compared to the sleep that expects to be signalled. + /// + /// Reset event to use to be awoken by the clock tick + /// Target timepoint + /// True if waited, false otherwise + public bool SleepUntilTimePointWithoutExternalSignal(EventWaitHandle evt, long timePoint) + { + long id; + + lock (_lock) + { + // Return immediately if the next tick is after the requested timepoint. + long nextTicks = _lastTicks + _granularityTicks; + + if (nextTicks > timePoint) + { + return false; + } + + id = ++_lastId; + + _waitingObjects.Add(new WaitingObject(id, evt, timePoint)); + } + + evt.WaitOne(); + + return true; + } + + /// + /// Returns the two nearest clock ticks for a given timepoint. + /// + /// Target timepoint + /// The nearest clock ticks before and after the given timepoint + public (long, long) ReturnNearestTicks(long timePoint) + { + long last = _lastTicks; + long delta = timePoint - last; + + long lowTicks = delta / _granularityTicks; + long highTicks = (delta + _granularityTicks - 1) / _granularityTicks; + + return (last + lowTicks * _granularityTicks, last + highTicks * _granularityTicks); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/WindowsSleepEvent.cs b/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/WindowsSleepEvent.cs new file mode 100644 index 0000000..bbd52cb --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/PreciseSleep/WindowsSleepEvent.cs @@ -0,0 +1,92 @@ +using Ryujinx.Common.SystemInterop; +using System; +using System.Runtime.Versioning; +using System.Threading; + +namespace Ryujinx.Common.PreciseSleep +{ + /// + /// A precise sleep event that uses Windows specific methods to increase clock resolution beyond 1ms, + /// use the clock's phase for more precise waits, and potentially align timepoints with it. + /// + [SupportedOSPlatform("windows")] + internal class WindowsSleepEvent : IPreciseSleepEvent + { + /// + /// The clock can drift a bit, so add this to encourage the clock to still wait if the next tick is forecasted slightly before it. + /// + private const long ErrorBias = 50000; + + /// + /// Allowed to be 0.05ms away from the clock granularity to reduce precision. + /// + private const long ClockAlignedBias = 50000; + + /// + /// The fraction of clock granularity above the timepoint that will align it down to the lower timepoint. + /// Currently set to the lower 1/4, so for 0.5ms granularity: 0.1ms would be rounded down, 0.2 ms would be rounded up. + /// + private const long ReverseTimePointFraction = 4; + + private readonly AutoResetEvent _waitEvent = new(false); + private readonly WindowsGranularTimer _timer = WindowsGranularTimer.Instance; + + /// + /// Set to true to disable timepoint realignment. + /// + public bool Precise { get; set; } = false; + + public long AdjustTimePoint(long timePoint, long timeoutNs) + { + if (Precise || timePoint == long.MaxValue) + { + return timePoint; + } + + // Does the timeout align with the host clock? + + long granularity = _timer.GranularityNs; + long misalignment = timeoutNs % granularity; + + if ((misalignment < ClockAlignedBias || misalignment > granularity - ClockAlignedBias) && timeoutNs > ClockAlignedBias) + { + // Inaccurate sleep for 0.5ms increments, typically. + + (long low, long high) = _timer.ReturnNearestTicks(timePoint); + + if (timePoint - low < _timer.GranularityTicks / ReverseTimePointFraction) + { + timePoint = low; + } + else + { + timePoint = high; + } + } + + return timePoint; + } + + public bool SleepUntil(long timePoint) + { + return _timer.SleepUntilTimePoint(_waitEvent, timePoint + (ErrorBias * PerformanceCounter.TicksPerMillisecond) / 1_000_000); + } + + public void Sleep() + { + _waitEvent.WaitOne(); + } + + public void Signal() + { + _waitEvent.Set(); + } + + public void Dispose() + { + GC.SuppressFinalize(this); + + _waitEvent.Dispose(); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/ReactiveObject.cs b/third_party/maxwell_shader/src/Ryujinx.Common/ReactiveObject.cs new file mode 100644 index 0000000..c4d28bc --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/ReactiveObject.cs @@ -0,0 +1,89 @@ +using Ryujinx.Common.Logging; +using System; +using System.Globalization; +using System.Threading; + +namespace Ryujinx.Common +{ + public class ReactiveObject + { + private readonly ReaderWriterLockSlim _rwLock = new(); + private bool _isInitialized; + private T _value; + + public event EventHandler> Event; + + public T Value + { + get + { + _rwLock.EnterReadLock(); + T value = _value; + _rwLock.ExitReadLock(); + + return value; + } + set + { + _rwLock.EnterWriteLock(); + + T oldValue = _value; + + bool oldIsInitialized = _isInitialized; + + _isInitialized = true; + _value = value; + + _rwLock.ExitWriteLock(); + + if (!oldIsInitialized || oldValue == null || !oldValue.Equals(_value)) + { + Event?.Invoke(this, new ReactiveEventArgs(oldValue, value)); + } + } + } + + public void LogChangesToValue(string valueName, LogClass logClass = LogClass.Configuration) + => Event += (_, e) => ReactiveObjectHelper.LogValueChange(logClass, e, valueName); + + public static implicit operator T(ReactiveObject obj) => obj.Value; + } + + public static class ReactiveObjectHelper + { + public static void LogValueChange(LogClass logClass, ReactiveEventArgs eventArgs, string valueName) + { + if (eventArgs.AreValuesEqual) + return; + + string message = string.Create(CultureInfo.InvariantCulture, $"{valueName} set to: {eventArgs.NewValue}"); + + Logger.Info?.Print(logClass, message); + } + + public static void Toggle(this ReactiveObject rBoolean) => rBoolean.Value = !rBoolean.Value; + } + + public class ReactiveEventArgs(T oldValue, T newValue) + { + public T OldValue { get; } = oldValue; + public T NewValue { get; } = newValue; + + public bool AreValuesEqual + { + get + { + if (OldValue == null && NewValue == null) + return true; + + if (OldValue == null && NewValue != null) + return false; + + if (OldValue != null && NewValue == null) + return false; + + return OldValue!.Equals(NewValue); + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/ReleaseInformation.cs b/third_party/maxwell_shader/src/Ryujinx.Common/ReleaseInformation.cs new file mode 100644 index 0000000..9b8b97f --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/ReleaseInformation.cs @@ -0,0 +1,37 @@ +using System; +using System.Reflection; + +namespace Ryujinx.Common +{ + // DO NOT EDIT, filled by CI + public static class ReleaseInformation + { + private const string CanaryChannel = "canary"; + private const string ReleaseChannel = "release"; + + private const string BuildVersion = "%%RYUJINX_BUILD_VERSION%%"; + private const string BuildGitHash = "%%RYUJINX_BUILD_GIT_HASH%%"; + private const string ReleaseChannelName = "%%RYUJINX_TARGET_RELEASE_CHANNEL_NAME%%"; + private const string ConfigFileName = "%%RYUJINX_CONFIG_FILE_NAME%%"; + + public static string ConfigName => !ConfigFileName.StartsWith("%%") ? ConfigFileName : "Config.json"; + + public static bool IsValid => + !BuildGitHash.StartsWith("%%") && + !ReleaseChannelName.StartsWith("%%") && + !ConfigFileName.StartsWith("%%"); + + public static bool IsCanaryBuild => IsValid && ReleaseChannelName.Equals(CanaryChannel); + + public static bool IsReleaseBuild => IsValid && ReleaseChannelName.Equals(ReleaseChannel); + + public static string Version => IsValid ? BuildVersion : Assembly.GetEntryAssembly()!.GetCustomAttribute()?.InformationalVersion; + + public static string GetChangelogUrl(Version currentVersion, Version newVersion) => + IsCanaryBuild + ? $"https://git.ryujinx.app/projects/Ryubing/compare/Canary-{currentVersion}...Canary-{newVersion}" + : $"https://git.ryujinx.app/projects/Ryubing/releases/tag/{newVersion}"; + } + + +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Ryujinx.Common.csproj b/third_party/maxwell_shader/src/Ryujinx.Common/Ryujinx.Common.csproj new file mode 100644 index 0000000..26afa15 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Ryujinx.Common.csproj @@ -0,0 +1,17 @@ + + + + true + $(DefineConstants);$(ExtraDefineConstants) + $(DefaultItemExcludes);._* + true + + + + + + + + + + diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/RyujinxException.cs b/third_party/maxwell_shader/src/Ryujinx.Common/RyujinxException.cs new file mode 100644 index 0000000..37d0938 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/RyujinxException.cs @@ -0,0 +1,10 @@ +using System; + +namespace Ryujinx.Common +{ + public class RyujinxException : Exception + { + public RyujinxException(string message) : base(message) + { } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/SharedConstants.cs b/third_party/maxwell_shader/src/Ryujinx.Common/SharedConstants.cs new file mode 100644 index 0000000..e462729 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/SharedConstants.cs @@ -0,0 +1,19 @@ +namespace Ryujinx.Common +{ + public static class SharedConstants + { + public const string DefaultLanPlayHost = "ldn.ryujinx.app"; + public const short LanPlayPort = 30456; + public const string DefaultLanPlayWebHost = DefaultLanPlayHost; + + public const string AmiiboTagsUrl = "https://raw.githubusercontent.com/Ryubing/Nfc/refs/heads/main/tags.json"; + + public const string FaqWikiUrl = "https://git.ryujinx.app/projects/Ryubing/wiki/FAQ-%26-Troubleshooting"; + + public const string SetupGuideWikiUrl = + "https://git.ryujinx.app/projects/Ryubing/wiki/Setup-%26-Configuration-Guide"; + + public const string MultiplayerWikiUrl = + "https://git.ryujinx.app/projects/Ryubing/wiki/Multiplayer-(LDN-Local-Wireless)-Guide"; + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/SystemInterop/DisplaySleep.cs b/third_party/maxwell_shader/src/Ryujinx.Common/SystemInterop/DisplaySleep.cs new file mode 100644 index 0000000..8e8f2df --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/SystemInterop/DisplaySleep.cs @@ -0,0 +1,35 @@ +using System; +using System.Runtime.InteropServices; + +namespace Ryujinx.Common.SystemInterop +{ + public partial class DisplaySleep + { + [Flags] + enum EXECUTION_STATE : uint + { + ES_CONTINUOUS = 0x80000000, + ES_DISPLAY_REQUIRED = 0x00000002, + ES_SYSTEM_REQUIRED = 0x00000001, + } + + [LibraryImport("kernel32.dll", SetLastError = true)] + private static partial EXECUTION_STATE SetThreadExecutionState(EXECUTION_STATE esFlags); + + static public void Prevent() + { + if (OperatingSystem.IsWindows()) + { + SetThreadExecutionState(EXECUTION_STATE.ES_CONTINUOUS | EXECUTION_STATE.ES_SYSTEM_REQUIRED | EXECUTION_STATE.ES_DISPLAY_REQUIRED); + } + } + + static public void Restore() + { + if (OperatingSystem.IsWindows()) + { + SetThreadExecutionState(EXECUTION_STATE.ES_CONTINUOUS); + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/SystemInterop/ForceDpiAware.cs b/third_party/maxwell_shader/src/Ryujinx.Common/SystemInterop/ForceDpiAware.cs new file mode 100644 index 0000000..6e8b52f --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/SystemInterop/ForceDpiAware.cs @@ -0,0 +1,97 @@ +using Ryujinx.Common.Logging; +using System; +using System.Globalization; +using System.Runtime.InteropServices; + +namespace Ryujinx.Common.SystemInterop +{ + public static partial class ForceDpiAware + { + [LibraryImport("user32.dll")] + [return: MarshalAs(UnmanagedType.Bool)] + private static partial bool SetProcessDPIAware(); + + private const string X11LibraryName = "libX11.so.6"; + + [LibraryImport(X11LibraryName)] + private static partial nint XOpenDisplay([MarshalAs(UnmanagedType.LPStr)] string display); + + [LibraryImport(X11LibraryName)] + private static partial nint XGetDefault(nint display, [MarshalAs(UnmanagedType.LPStr)] string program, [MarshalAs(UnmanagedType.LPStr)] string option); + + [LibraryImport(X11LibraryName)] + private static partial int XDisplayWidth(nint display, int screenNumber); + + [LibraryImport(X11LibraryName)] + private static partial int XDisplayWidthMM(nint display, int screenNumber); + + [LibraryImport(X11LibraryName)] + private static partial int XCloseDisplay(nint display); + + private const double StandardDpiScale = 96.0; + private const double MaxScaleFactor = 1.25; + + /// + /// Marks the application as DPI-Aware when running on the Windows operating system. + /// + public static void Windows() + { + // Make process DPI aware for proper window sizing on high-res screens. + if (OperatingSystem.IsWindowsVersionAtLeast(6)) + { + SetProcessDPIAware(); + } + } + + public static double GetActualScaleFactor() + { + double userDpiScale = 96.0; + + try + { + if (OperatingSystem.IsWindows()) + { + userDpiScale = GdiPlusHelper.GetDpiX(nint.Zero); + } + else if (OperatingSystem.IsLinux()) + { + string xdgSessionType = Environment.GetEnvironmentVariable("XDG_SESSION_TYPE")?.ToLower(); + + if (xdgSessionType is null or "x11") + { + nint display = XOpenDisplay(null); + string dpiString = Marshal.PtrToStringAnsi(XGetDefault(display, "Xft", "dpi")); + if (dpiString == null || !double.TryParse(dpiString, NumberStyles.Any, CultureInfo.InvariantCulture, out userDpiScale)) + { + userDpiScale = XDisplayWidth(display, 0) * 25.4 / XDisplayWidthMM(display, 0); + } + + _ = XCloseDisplay(display); + } + else if (xdgSessionType == "wayland") + { + // TODO + Logger.Warning?.Print(LogClass.Application, "Couldn't determine monitor DPI: Wayland not yet supported"); + } + else + { + Logger.Warning?.Print(LogClass.Application, $"Couldn't determine monitor DPI: Unrecognised XDG_SESSION_TYPE: {xdgSessionType}"); + } + } + } + catch (Exception e) + { + Logger.Warning?.Print(LogClass.Application, $"Couldn't determine monitor DPI: {e.Message}"); + } + + return userDpiScale; + } + + public static double GetWindowScaleFactor() + { + double userDpiScale = GetActualScaleFactor(); + + return Math.Min(userDpiScale / StandardDpiScale, MaxScaleFactor); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/SystemInterop/GdiPlusHelper.cs b/third_party/maxwell_shader/src/Ryujinx.Common/SystemInterop/GdiPlusHelper.cs new file mode 100644 index 0000000..024c6aa --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/SystemInterop/GdiPlusHelper.cs @@ -0,0 +1,76 @@ +using System; +using System.Runtime.InteropServices; +using System.Runtime.Versioning; + +namespace Ryujinx.Common.SystemInterop +{ + [SupportedOSPlatform("windows")] + public static partial class GdiPlusHelper + { + private const string LibraryName = "gdiplus.dll"; + + private static readonly nint _initToken; + + static GdiPlusHelper() + { + CheckStatus(GdiplusStartup(out _initToken, StartupInputEx.Default, out _)); + } + + private static void CheckStatus(int gdiStatus) + { + if (gdiStatus != 0) + { + throw new Exception($"GDI Status Error: {gdiStatus}"); + } + } + + private struct StartupInputEx + { + public int GdiplusVersion; + +#pragma warning disable CS0649 // Field is never assigned to + public nint DebugEventCallback; + public int SuppressBackgroundThread; + public int SuppressExternalCodecs; + public int StartupParameters; +#pragma warning restore CS0649 + + public static StartupInputEx Default => new() + { + // We assume Windows 8 and upper + GdiplusVersion = 2, + DebugEventCallback = nint.Zero, + SuppressBackgroundThread = 0, + SuppressExternalCodecs = 0, + StartupParameters = 0, + }; + } + + private struct StartupOutput + { + public nint NotificationHook; + public nint NotificationUnhook; + } + + [LibraryImport(LibraryName)] + private static partial int GdiplusStartup(out nint token, in StartupInputEx input, out StartupOutput output); + + [LibraryImport(LibraryName)] + private static partial int GdipCreateFromHWND(nint hwnd, out nint graphics); + + [LibraryImport(LibraryName)] + private static partial int GdipDeleteGraphics(nint graphics); + + [LibraryImport(LibraryName)] + private static partial int GdipGetDpiX(nint graphics, out float dpi); + + public static float GetDpiX(nint hwnd) + { + CheckStatus(GdipCreateFromHWND(hwnd, out nint graphicsHandle)); + CheckStatus(GdipGetDpiX(graphicsHandle, out float result)); + CheckStatus(GdipDeleteGraphics(graphicsHandle)); + + return result; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/SystemInterop/StdErrAdapter.cs b/third_party/maxwell_shader/src/Ryujinx.Common/SystemInterop/StdErrAdapter.cs new file mode 100644 index 0000000..291016a --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/SystemInterop/StdErrAdapter.cs @@ -0,0 +1,103 @@ +using Microsoft.Win32.SafeHandles; +using Ryujinx.Common.Logging; +using System; +using System.IO; +using System.Runtime.InteropServices; +using System.Runtime.Versioning; +using System.Threading; +using System.Threading.Tasks; + +namespace Ryujinx.Common.SystemInterop +{ + public partial class StdErrAdapter : IDisposable + { + private bool _disposable; + private FileStream _pipeReader; + private FileStream _pipeWriter; + private CancellationTokenSource _cancellationTokenSource; + private Task _worker; + + public StdErrAdapter() + { + if (OperatingSystem.IsLinux() || OperatingSystem.IsMacOS()) + { + RegisterPosix(); + } + } + + [SupportedOSPlatform("linux")] + [SupportedOSPlatform("macos")] + private void RegisterPosix() + { + const int StdErrFileno = 2; + + (int readFd, int writeFd) = MakePipe(); + dup2(writeFd, StdErrFileno); + + _pipeReader = CreateFileDescriptorStream(readFd); + _pipeWriter = CreateFileDescriptorStream(writeFd); + + _cancellationTokenSource = new CancellationTokenSource(); + _worker = Task.Run(async () => await EventWorkerAsync(_cancellationTokenSource.Token), _cancellationTokenSource.Token); + _disposable = true; + } + + [SupportedOSPlatform("linux")] + [SupportedOSPlatform("macos")] + private async Task EventWorkerAsync(CancellationToken cancellationToken) + { + using StreamReader reader = new(_pipeReader, leaveOpen: true); + string line; + while (cancellationToken.IsCancellationRequested == false && (line = await reader.ReadLineAsync(cancellationToken)) != null) + { + Logger.Error?.PrintRawMsg(line); + } + } + + public void Dispose() + { + GC.SuppressFinalize(this); + + if (_disposable) + { + _disposable = false; + + if (OperatingSystem.IsLinux() || OperatingSystem.IsMacOS()) + { + _cancellationTokenSource.Cancel(); + _worker.Wait(0); + _pipeReader?.Close(); + _pipeWriter?.Close(); + } + } + } + + [LibraryImport("libc", SetLastError = true)] + private static partial int dup2(int fd, int fd2); + + [LibraryImport("libc", SetLastError = true)] + private static partial int pipe(Span pipefd); + + private static (int, int) MakePipe() + { + Span pipefd = stackalloc int[2]; + + if (pipe(pipefd) == 0) + { + return (pipefd[0], pipefd[1]); + } + + throw new(); + } + + [SupportedOSPlatform("linux")] + [SupportedOSPlatform("macos")] + private static FileStream CreateFileDescriptorStream(int fd) + { + return new FileStream( + new SafeFileHandle(fd, ownsHandle: true), + FileAccess.ReadWrite + ); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/SystemInterop/WindowsMultimediaTimerResolution.cs b/third_party/maxwell_shader/src/Ryujinx.Common/SystemInterop/WindowsMultimediaTimerResolution.cs new file mode 100644 index 0000000..d3d20cb --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/SystemInterop/WindowsMultimediaTimerResolution.cs @@ -0,0 +1,114 @@ +using Ryujinx.Common.Logging; +using System; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using System.Runtime.Versioning; + +namespace Ryujinx.Common.SystemInterop +{ + /// + /// Handle Windows Multimedia timer resolution. + /// + [SupportedOSPlatform("windows")] + public partial class WindowsMultimediaTimerResolution : IDisposable + { + [StructLayout(LayoutKind.Sequential)] + public struct TimeCaps + { + public uint wPeriodMin; + public uint wPeriodMax; + } + + [LibraryImport("winmm.dll", EntryPoint = "timeGetDevCaps", SetLastError = true)] + private static partial uint TimeGetDevCaps(ref TimeCaps timeCaps, uint sizeTimeCaps); + + [LibraryImport("winmm.dll", EntryPoint = "timeBeginPeriod")] + private static partial uint TimeBeginPeriod(uint uMilliseconds); + + [LibraryImport("winmm.dll", EntryPoint = "timeEndPeriod")] + private static partial uint TimeEndPeriod(uint uMilliseconds); + + private uint _targetResolutionInMilliseconds; + private bool _isActive; + + /// + /// Create a new and activate the given resolution. + /// + /// + public WindowsMultimediaTimerResolution(uint targetResolutionInMilliseconds) + { + _targetResolutionInMilliseconds = targetResolutionInMilliseconds; + + EnsureResolutionSupport(); + Activate(); + } + + private void EnsureResolutionSupport() + { + TimeCaps timeCaps = default; + + uint result = TimeGetDevCaps(ref timeCaps, (uint)Unsafe.SizeOf()); + + if (result != 0) + { + Logger.Notice.Print(LogClass.Application, $"timeGetDevCaps failed with result: {result}"); + } + else + { + uint supportedTargetResolutionInMilliseconds = Math.Min(Math.Max(timeCaps.wPeriodMin, _targetResolutionInMilliseconds), timeCaps.wPeriodMax); + + if (supportedTargetResolutionInMilliseconds != _targetResolutionInMilliseconds) + { + Logger.Notice.Print(LogClass.Application, $"Target resolution isn't supported by OS, using closest resolution: {supportedTargetResolutionInMilliseconds}ms"); + + _targetResolutionInMilliseconds = supportedTargetResolutionInMilliseconds; + } + } + } + + private void Activate() + { + uint result = TimeBeginPeriod(_targetResolutionInMilliseconds); + + if (result != 0) + { + Logger.Notice.Print(LogClass.Application, $"timeBeginPeriod failed with result: {result}"); + } + else + { + _isActive = true; + } + } + + private void Disable() + { + if (_isActive) + { + uint result = TimeEndPeriod(_targetResolutionInMilliseconds); + + if (result != 0) + { + Logger.Notice.Print(LogClass.Application, $"timeEndPeriod failed with result: {result}"); + } + else + { + _isActive = false; + } + } + } + + public void Dispose() + { + Dispose(true); + GC.SuppressFinalize(this); + } + + protected virtual void Dispose(bool disposing) + { + if (disposing) + { + Disable(); + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/TitleIDs.cs b/third_party/maxwell_shader/src/Ryujinx.Common/TitleIDs.cs new file mode 100644 index 0000000..5e48a27 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/TitleIDs.cs @@ -0,0 +1,253 @@ +using Gommon; +using System.Linq; + +namespace Ryujinx.Common +{ + public static class TitleIDs + { + public static ReactiveObject> CurrentApplication { get; } = new(); + + public static string GetDiscordGameAsset(string titleId) + => DiscordGameAssetKeys.Contains(titleId) ? titleId : "game"; + + public static readonly string[] DiscordGameAssetKeys = + [ + //All games are in Alphabetical order by Game name. + + //Dragon Quest Franchise + "010008900705c000", // Dragon Quest Builders + "010042000a986000", // Dragon Quest Builders 2 + "010021701c45e000", // Dragon Quest VII Reimagined Demo + + //Fire Emblem Franchise + "0100a6301214e000", // Fire Emblem Engage + "0100a12011cc8000", // Fire Emblem: Shadow Dragon + "010055d009f78000", // Fire Emblem: Three Houses + "0100f15003e64000", // Fire Emblem Warriors + "010071f0143ea000", // Fire Emblem Warriors: Three Hopes + + //Kirby Franchise + "01004d300c5ae000", // Kirby and the Forgotten Land + "0100a8e016236000", // Kirby's Dream Buffet + "0100227010460000", // Kirby Fighters 2 + "01006b601380e000", // Kirby's Return to Dream Land Deluxe + "01007e3006dda000", // Kirby Star Allies + "01003fb00c5a8000", // Super Kirby Clash + + //The Zelda Franchise + "01000b900d8b0000", // Cadence of Hyrule + "0100ae00096ea000", // Hyrule Warriors: Definitive Edition + "01002b00111a2000", // Hyrule Warriors: Age of Calamity + "01007ef00011e000", // The Legend of Zelda: Breath of the Wild + "01006bb00c6f0000", // The Legend of Zelda: Link's Awakening + "01002da013484000", // The Legend of Zelda: Skyward Sword HD + "0100f2c0115b6000", // The Legend of Zelda: Tears of the Kingdom + "01008cf01baac000", // The Legend of Zelda: Echoes of Wisdom + + //Luigi Franchise + "010048701995e000", // Luigi's Mansion 2 HD + "0100dca0064a6000", // Luigi's Mansion 3 + + //Metroid Franchise + "010093801237c000", // Metroid Dread + "010012101468c000", // Metroid Prime Remastered + + //Monster Hunter Franchise + "0100770008dd8000", // Monster Hunter Generations Ultimate + "0100b04011742000", // Monster Hunter Rise + + //Mario Franchise + "010021d00812a000", // Arcade Archives VS. SUPER MARIO BROS. + "01007fe0221d8000", // Hello, Mario! + "01006d0017f7a000", // Mario & Luigi: Brothership + "010003000e146000", // Mario & Sonic at the Olympic Games Tokyo 2020 + "010067300059a000", // Mario + Rabbids: Kingdom Battle + "0100317013770000", // Mario + Rabbids: Sparks of Hope + "0100c9c00e25c000", // Mario Golf: Super Rush + "0100152000022000", // Mario Kart 8 Deluxe + "01006fe013472000", // Mario Party Superstars + "010019401051c000", // Mario Strikers: Battle League + "0100bde00862a000", // Mario Tennis Aces + "0100b99019412000", // Mario vs. Donkey Kong + "010049900f546000", // Super Mario 3D All-Stars + "010049900f546001", // Super Mario 3D All-Stars | Super Mario 64 + "010049900f546002", // Super Mario 3D All-Stars | Super Mario Sunshine + "010049900f546003", // Super Mario 3D All-Stars | Super Mario Galaxy + "010028600ebda000", // Super Mario 3D World + Bowser's Fury + "0100ea80032ea000", // Super Mario Bros. U Deluxe + "010015100b514000", // Super Mario Bros. Wonder + "01009b90006dc000", // Super Mario Maker 2 + "0100000000010000", // SUPER MARIO ODYSSEY + "010036b0034e4000", // Super Mario Party + "0100965017338000", // Super Mario Party Jamboree + "0100bc0018138000", // Super Mario RPG + "0100a3900c3e2000", // Paper Mario: The Origami King + "0100ecd018ebe000", // Paper Mario: The Thousand-Year Door + "01009cC022b96000", // Super Mario Galaxy + "0100fd8022daa000", // Super Mario Galaxy 2 + + //Pikmin Franchise + "0100aa80194b0000", // Pikmin 1 + "0100d680194b2000", // Pikmin 2 + "0100f4c009322000", // Pikmin 3 Deluxe + "0100b7c00933a000", // Pikmin 4 + + //The Pokémon Franchise + "0100f4300bf2c000", // New Pokémon Snap + "0100000011d90000", // Pokémon Brilliant Diamond + "010008c01e742000", // Pokémon Friends + "01001f5010dfa000", // Pokémon Legends: Arceus + "01003d200baa2000", // Pokémon Mystery Dungeon - Rescue Team DX + "0100a3d008c5c000", // Pokémon Scarlet + "01008db008c2c000", // Pokémon Shield + "010018e011d92000", // Pokémon Shining Pearl + "0100abf008968000", // Pokémon Sword + "01008f6008c5e000", // Pokémon Violet + "0100b3f000be2000", // Pokkén Tournament DX + "0100187003a36000", // Pokémon: Let's Go Eevee! + "010003f003a34000", // Pokémon: Let's Go Pikachu! + "0100f43008c44000", // Pokémon Legends: Z-A + "0100554023408000", // Pokémon FireRed Version (EN) + "01006fa0233f8000", // Pokémon FireRed Version (JP) + "0100fd6023430000", // Pokémon LeafGreen Version (DE) + "0100f1e0233fa000", // Pokémon LeafGreen Version (JP) + "01005b7008c52000", // Pokémon Champions + + //Splatoon Franchise + "0100f8f0000a2000", // Splatoon 2 (EU) + "01003c700009c000", // Splatoon 2 (JP) + "01003bc0000a0000", // Splatoon 2 (US) + "0100c2500fc20000", // Splatoon 3 + "0100ba0018500000", // Splatoon 3: Splatfest World Premiere + + //NSO Membership games + "0100d870045b6000", // NES - Nintendo Switch Online + "01008d300c50c000", // SNES - Nintendo Switch Online + "0100c62011050000", // GB - Nintendo Switch Online + "010012f017576000", // GBA - Nintendo Switch Online + "0100c9a00ece6000", // N64 - Nintendo Switch Online + "0100e0601c632000", // N64 - Nintendo Switch Online 18+ + "0100b3c014bda000", // SEGA Genesis - Nintendo Switch Online + "0100bfc01d976000", // Virtual Boy - Nintendo Switch Online + "0100ccf019c8c000", // F-ZERO 99 + "0100ad9012510000", // PAC-MAN 99 + "010040600c5ce000", // Tetris 99 + "0100277011f1a000", // Super Mario Bros. 35 + + //Misc Nintendo 1st party games + "01000320000cc000", // 1-2 Switch + "0100300012f2a000", // Advance Wars 1+2: Re-Boot Camp + "01006f8002326000", // Animal Crossing: New Horizons + "0100620012d6e000", // Big Brain Academy: Brain vs. Brain + "010018300d006000", // BOXBOY! + BOXGIRL! + "0100c1f0051b6000", // Donkey Kong Country: Tropical Freeze + "0100ed000d390000", // Dr. Kawashima's Brain Training + "010067b017588000", // Endless Ocean Luminous + "01003da010e8a000", // Miitopia + "01006b5012b32000", // Part Time UFO + "0100107027310000", // Rhythm Heave Groove: Starter Demo + "0100704000B3A000", // Snipperclips + "01006a800016e000", // Super Smash Bros. Ultimate + "0100a9400c9c2000", // Tokyo Mirage Sessions #FE Encore + "0100ca502552a000", // Tomodachi Life: Living the Dream - Welcome Edition + "010051f0207b2000", // Tomodachi Life: Living the Dream + + //Bayonetta Franchise + "010076f0049a2000", // Bayonetta + "01007960049a0000", // Bayonetta 2 + "01004a4010fea000", // Bayonetta 3 + "0100cf5010fec000", // Bayonetta Origins: Cereza and the Lost Demon + + // Famicom Detective Club Franchise + "010054601d54c000", // Emio - The Smiling Man: Famicom Detective Series (DEMO) + + //Persona Franchise + "0100dcd01525a000", // Persona 3 Portable + "010075a016a3a000", // Persona 4 Arena Ultimax + "010062b01525c000", // Persona 4 Golden + "01005ca01580e000", // Persona 5 Royal + "0100801011c3e000", // Persona 5 Strikers + "010087701b092000", // Persona 5 Tactica + + //Sonic Franchise + "01004ad014bf0000", // Sonic Frontiers + "01009aa000faa000", // Sonic Mania + "01005ea01c0fc000", // SONIC X SHADOW GENERATIONS + "01005ea01c0fc001", // ^ TODO: (Needs to be differentiated like 3D All Stars) + "01006e001823c000", // Sonic Racing: CrossWorlds + "010064b0242be000", // Sonic Racing: CrossWorlds - Demo + + //Xenoblade Franchise + "0100ff500e34a000", // Xenoblade Chronicles - Definitive Edition + "0100e95004038000", // Xenoblade Chronicles 2 + "010074f013262000", // Xenoblade Chronicles 3 + "0100453019aa8000", // Xenoblade Chronicles: X Definitive Edition + + //Misc Games + "01003670066de000", // 36 Fragments of Midnight + "010056e00853a000", // A Hat in Time + "0100c9f00aaee000", // Ascendence + "0100fd1014726000", // Baldurs Gate: Dark Alliance + "01008c2019598000", // Bluey: The Video Game + "010096f00ff22000", // Borderlands 2: Game of the Year Edition + "010007400ff24000", // Borderlands: The Pre-Sequel Ultimate Edition + "0100c6800b934000", // Brawlhalla + "0100dbf01000a000", // Burnout Paradise Remastered + "0100744001588000", // Cars 3: Driven to Win + "0100b41013c82000", // Cruis'n Blast + "010085900337e000", // Death Squared + "0100a0d022a68000", // DELTARUNE + "01001b300b9be000", // Diablo III: Eternal Collection + "010027400cdc6000", // Divinity Original 2 - Definitive Edition + "01008c8012920000", // Dying Light Platinum Edition + "0100d11013e6a000", // Eschatos + "01000490067ae000", // Frederic 2: Evil Strikes Back + "01001cc01b2d4000", // Goat Simulator 3 + "01003620068ea000", // Hand of Fate 2 + "01007ac00e012000", // HEXAGRAVITY + "0100f7e00c70e000", // Hogwarts Legacy + "010013c00e930000", // Hollow Knight: Silksong + "010085500130a000", // Lego City: Undercover + "010073c01af34000", // LEGO Horizon Adventures + "0100653026dde000", // MARVEL Cosmic Invasion Demo + "0100419014456000", // Mary Skelter Finale + "0100d71004694000", // Minecraft + "01007430037f6000", // Monopoly + "0100853015e86000", // No Man's Sky + "0100f85014ed0000", // No More Heroes + "0100463014ed4000", // No More Heroes 2 + "0100f7d00a1bc000", // NO THING + "0100e570094e8000", // Owlboy + "01007bb017812000", // Portal + "0100abd01785c000", // Portal 2 + "01009f100bc52000", // Psikyo Collection 1 + "01009d400c4a8000", // Psikyo Collection 2 + "01008e200c5c2000", // Muse Dash + "01005ff002e2a000", // Rayman Legends + "01007820196a6000", // Red Dead Redemption + "01007a800d520000", // REFUNCT + "0100e8300a67a000", // Risk + "01002f7013224000", // Rune Factory 5 + "01008d100d43e000", // Saints Row IV + "0100de600beee000", // Saints Row: The Third - The Full Package + "01001180021fa000", // Shovel Knight: Specter of Torment + "0100ccc0002e6000", // Skylanders Imaginators (US) + "010048d000bfc000", // Skylanders Imaginators (EU) + "010079f00671c000", // Sparkle 2: Evo + "010077b00e046000", // Spyro: Reignited Trilogy + "0100e1D01eb2e000", // Squeakross: Home Squeak Home + "0100e65002bb8000", // Stardew Valley + "0100d7a01b7a2000", // Star Wars: Bounty Hunter + "0100800015926000", // Suika Game + "01007ad00013e000", // Super Bomberman R + "0100e46006708000", // Terraria + "0100605008268000", // Titan Quest + "01000a10041ea000", // The Elder Scrolls V: Skyrim + "010057a01e4d4000", // TSUKIHIME -A piece of blue glass moon- + "010080b00ad66000", // Undertale + "010069401adb8000", // Unicorn Overlord + "01005c600ac68000", // Valkyria Chronicles 4 + "0100534009ff2000", // Yonder - The cloud catcher chronicles + ]; + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/UI/UserError.cs b/third_party/maxwell_shader/src/Ryujinx.Common/UI/UserError.cs new file mode 100644 index 0000000..3128c19 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/UI/UserError.cs @@ -0,0 +1,39 @@ +namespace Ryujinx.Common.UI +{ + /// + /// Represent a common error that could be reported to the user by the emulator. + /// + public enum UserError + { + /// + /// No error to report. + /// + Success = 0x0, + + /// + /// No keys are present. + /// + NoKeys = 0x1, + + /// + /// No firmware is installed. + /// + NoFirmware = 0x2, + + /// + /// Firmware parsing failed. + /// + /// Most likely related to keys. + FirmwareParsingFailed = 0x3, + + /// + /// No application was found at the given path. + /// + ApplicationNotFound = 0x4, + + /// + /// An unknown error. + /// + Unknown = 0xDEAD, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/BitUtils.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/BitUtils.cs new file mode 100644 index 0000000..68a63f3 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/BitUtils.cs @@ -0,0 +1,76 @@ +using System.Numerics; + +namespace Ryujinx.Common +{ + public static class BitUtils + { + public static T AlignUp(T value, T size) where T : IBinaryInteger + => (value + (size - T.One)) & -size; + + public static T AlignDown(T value, T size) where T : IBinaryInteger + => value & -size; + + public static T DivRoundUp(T value, T dividend) where T : IBinaryInteger + => (value + (dividend - T.One)) / dividend; + + public static int Pow2RoundDown(int value) => BitOperations.IsPow2(value) ? value : Pow2RoundUp(value) >> 1; + + public static long ReverseBits64(long value) => (long)ReverseBits64((ulong)value); + + public static int Pow2RoundUp(int value) + { + value--; + + value |= (value >> 1); + value |= (value >> 2); + value |= (value >> 4); + value |= (value >> 8); + value |= (value >> 16); + + return ++value; + } + + private static ulong ReverseBits64(ulong value) + { + value = ((value & 0xaaaaaaaaaaaaaaaa) >> 1) | ((value & 0x5555555555555555) << 1); + value = ((value & 0xcccccccccccccccc) >> 2) | ((value & 0x3333333333333333) << 2); + value = ((value & 0xf0f0f0f0f0f0f0f0) >> 4) | ((value & 0x0f0f0f0f0f0f0f0f) << 4); + value = ((value & 0xff00ff00ff00ff00) >> 8) | ((value & 0x00ff00ff00ff00ff) << 8); + value = ((value & 0xffff0000ffff0000) >> 16) | ((value & 0x0000ffff0000ffff) << 16); + + return (value >> 32) | (value << 32); + } + + // Never actually written bit packing logic before, so I looked it up. + // This code is from https://gist.github.com/Alan-FGR/04938e93e2bffdf5802ceb218a37c195 + + public static ulong PackBitFields(this uint[] values, byte[] bitFields) + { + ulong retVal = values[0]; //we set the first value right away + for (int f = 1; f < values.Length; f++) + { + retVal <<= bitFields[f]; // we shift the previous value + retVal += values[f];// and add our current value + } + + return retVal; + } + + public static uint[] UnpackBitFields(this ulong packed, byte[] bitFields) + { + int fields = bitFields.Length - 1; // number of fields to unpack + uint[] retArr = new uint[fields + 1]; // init return array + int curPos = 0; // current field bit position (start) + int lastEnd; // position where last field ended + for (int f = fields; f >= 0; f--) // loop from last + { + lastEnd = curPos; // we store where the last value ended + curPos += bitFields[f]; // we get where the current value starts + int leftShift = 64 - curPos; // we figure how much left shift we gotta apply for the other numbers to overflow into oblivion + retArr[f] = (uint)((packed << leftShift) >> leftShift + lastEnd); // we do magic + } + + return retArr; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/BitfieldExtensions.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/BitfieldExtensions.cs new file mode 100644 index 0000000..95bf1e6 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/BitfieldExtensions.cs @@ -0,0 +1,57 @@ +using System.Numerics; +using System.Runtime.CompilerServices; + +namespace Ryujinx.Common.Utilities +{ + public static class BitfieldExtensions + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool Extract(this T value, int lsb) where T : IBinaryInteger + { + int bitSize = Unsafe.SizeOf() * 8; + lsb &= bitSize - 1; + + return !T.IsZero((value >>> lsb) & T.One); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static T Extract(this T value, int lsb, int length) where T : IBinaryInteger + { + int bitSize = Unsafe.SizeOf() * 8; + lsb &= bitSize - 1; + + return (value >>> lsb) & (~T.Zero >>> (bitSize - length)); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static T ExtractSx(this T value, int lsb, int length) where T : IBinaryInteger + { + int bitSize = Unsafe.SizeOf() * 8; + int shift = lsb & (bitSize - 1); + + return (value << (bitSize - (shift + length))) >> (bitSize - length); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static T Insert(this T value, int lsb, bool toInsert) where T : IBinaryInteger + { + int bitSize = Unsafe.SizeOf() * 8; + lsb &= bitSize - 1; + + T mask = T.One << lsb; + + return (value & ~mask) | (toInsert ? mask : T.Zero); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static T Insert(this T value, int lsb, int length, T toInsert) where T : IBinaryInteger + { + int bitSize = Unsafe.SizeOf() * 8; + lsb &= bitSize - 1; + + T mask = (~T.Zero >>> (bitSize - length)) << lsb; + + return (value & ~mask) | ((toInsert << lsb) & mask); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/Buffers.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/Buffers.cs new file mode 100644 index 0000000..94f1734 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/Buffers.cs @@ -0,0 +1,59 @@ +using System; +using System.Diagnostics; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; + +namespace Ryujinx.Common.Utilities +{ + [DebuggerDisplay("{ToString()}")] + [StructLayout(LayoutKind.Sequential, Size = 16)] + public struct Buffer16 + { + [DebuggerBrowsable(DebuggerBrowsableState.Never)] private readonly ulong _dummy0; + [DebuggerBrowsable(DebuggerBrowsableState.Never)] private readonly ulong _dummy1; + + public byte this[int i] + { + get => Bytes[i]; + set => Bytes[i] = value; + } + + public Span Bytes => SpanHelpers.AsByteSpan(ref this); + + // Prevent a defensive copy by changing the read-only in reference to a reference with Unsafe.AsRef() + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static implicit operator Span(in Buffer16 value) + { + return SpanHelpers.AsByteSpan(ref Unsafe.AsRef(in value)); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static implicit operator ReadOnlySpan(in Buffer16 value) + { + return SpanHelpers.AsReadOnlyByteSpan(ref Unsafe.AsRef(in value)); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public ref T As() where T : unmanaged + { + if (Unsafe.SizeOf() > (uint)Unsafe.SizeOf()) + { + throw new ArgumentException(); + } + + return ref MemoryMarshal.GetReference(AsSpan()); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public Span AsSpan() where T : unmanaged + { + return SpanHelpers.AsSpan(ref this); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public readonly ReadOnlySpan AsReadOnlySpan() where T : unmanaged + { + return SpanHelpers.AsReadOnlySpan(ref Unsafe.AsRef(in this)); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/CommonJsonContext.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/CommonJsonContext.cs new file mode 100644 index 0000000..fb305d0 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/CommonJsonContext.cs @@ -0,0 +1,11 @@ +using System.Collections.Generic; +using System.Text.Json.Serialization; + +namespace Ryujinx.Common.Utilities +{ + [JsonSerializable(typeof(string[]), TypeInfoPropertyName = "StringArray")] + [JsonSerializable(typeof(Dictionary), TypeInfoPropertyName = "StringDictionary")] + public partial class CommonJsonContext : JsonSerializerContext + { + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/EmbeddedResources.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/EmbeddedResources.cs new file mode 100644 index 0000000..088efdf --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/EmbeddedResources.cs @@ -0,0 +1,152 @@ +using Ryujinx.Common.Memory; +using Ryujinx.Common.Utilities; +using System; +using System.IO; +using System.Linq; +using System.Reflection; +using System.Threading.Tasks; + +namespace Ryujinx.Common +{ + public static class EmbeddedResources + { + private readonly static Assembly _resourceAssembly; + + static EmbeddedResources() + { + _resourceAssembly = Assembly.GetAssembly(typeof(EmbeddedResources)); + } + + public static byte[] Read(string filename) + { + (Assembly assembly, string path) = ResolveManifestPath(filename); + + return Read(assembly, path); + } + + public static Task ReadAsync(string filename) + { + (Assembly assembly, string path) = ResolveManifestPath(filename); + + return ReadAsync(assembly, path); + } + + public static byte[] Read(Assembly assembly, string filename) + { + using Stream stream = GetStream(assembly, filename); + if (stream == null) + { + return null; + } + + return StreamUtils.StreamToBytes(stream); + } + + public static MemoryOwner ReadFileToRentedMemory(string filename) + { + (Assembly assembly, string path) = ResolveManifestPath(filename); + + return ReadFileToRentedMemory(assembly, path); + } + + public static MemoryOwner ReadFileToRentedMemory(Assembly assembly, string filename) + { + using Stream stream = GetStream(assembly, filename); + + return stream is null + ? null + : StreamUtils.StreamToRentedMemory(stream); + } + + public async static Task ReadAsync(Assembly assembly, string filename) + { + using Stream stream = GetStream(assembly, filename); + if (stream == null) + { + return null; + } + + return await StreamUtils.StreamToBytesAsync(stream); + } + + public static string ReadAllText(string filename) + { + (Assembly assembly, string path) = ResolveManifestPath(filename); + + return ReadAllText(assembly, path); + } + + public static Task ReadAllTextAsync(string filename) + { + (Assembly assembly, string path) = ResolveManifestPath(filename); + + return ReadAllTextAsync(assembly, path); + } + + public static string ReadAllText(Assembly assembly, string filename) + { + using Stream stream = GetStream(assembly, filename); + if (stream == null) + { + return null; + } + + using StreamReader reader = new(stream); + return reader.ReadToEnd(); + } + + public async static Task ReadAllTextAsync(Assembly assembly, string filename) + { + using Stream stream = GetStream(assembly, filename); + if (stream == null) + { + return null; + } + + using StreamReader reader = new(stream); + return await reader.ReadToEndAsync(); + } + + public static Stream GetStream(string filename) + { + (Assembly assembly, string path) = ResolveManifestPath(filename); + + return GetStream(assembly, path); + } + + public static Stream GetStream(Assembly assembly, string filename) + { + string @namespace = assembly.GetName().Name; + string manifestUri = @namespace + "." + filename.Replace('/', '.'); + + Stream stream = assembly.GetManifestResourceStream(manifestUri); + + return stream; + } + + public static string[] GetAllAvailableResources(string path, string ext = "") + { + return ResolveManifestPath(path).Item1.GetManifestResourceNames() + .Where(r => r.StartsWith(path.Replace('/', '.')) && r.EndsWith(ext)) + .ToArray(); + } + + private static (Assembly, string) ResolveManifestPath(string filename) + { + string[] segments = filename.Split('/', 2, StringSplitOptions.RemoveEmptyEntries); + + if (segments.Length >= 2) + { + foreach (Assembly assembly in AppDomain.CurrentDomain.GetAssemblies()) + { + if (assembly.GetName().Name == segments[0]) + { + return (assembly, segments[1]); + } + } + } + + return (_resourceAssembly, filename); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/FileSystemUtils.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/FileSystemUtils.cs new file mode 100644 index 0000000..a3e0fcd --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/FileSystemUtils.cs @@ -0,0 +1,56 @@ +using System.Collections.Generic; +using System.IO; +using System.Linq; + +namespace Ryujinx.Common.Utilities +{ + public static class FileSystemUtils + { + public static void CopyDirectory(string sourceDir, string destinationDir, bool recursive) + { + // Get information about the source directory + DirectoryInfo dir = new(sourceDir); + + // Check if the source directory exists + if (!dir.Exists) + { + throw new DirectoryNotFoundException($"Source directory not found: {dir.FullName}"); + } + + // Cache directories before we start copying + DirectoryInfo[] dirs = dir.GetDirectories(); + + // Create the destination directory + Directory.CreateDirectory(destinationDir); + + // Get the files in the source directory and copy to the destination directory + foreach (FileInfo file in dir.GetFiles()) + { + string targetFilePath = Path.Combine(destinationDir, file.Name); + file.CopyTo(targetFilePath); + } + + // If recursive and copying subdirectories, recursively call this method + if (recursive) + { + foreach (DirectoryInfo subDir in dirs) + { + string newDestinationDir = Path.Combine(destinationDir, subDir.Name); + CopyDirectory(subDir.FullName, newDestinationDir, true); + } + } + } + + public static void MoveDirectory(string sourceDir, string destinationDir) + { + CopyDirectory(sourceDir, destinationDir, true); + Directory.Delete(sourceDir, true); + } + + public static string SanitizeFileName(string fileName) + { + HashSet reservedChars = [.. Path.GetInvalidFileNameChars()]; + return string.Concat(fileName.Select(c => reservedChars.Contains(c) ? '_' : c)); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/HexUtils.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/HexUtils.cs new file mode 100644 index 0000000..820ab97 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/HexUtils.cs @@ -0,0 +1,90 @@ +using System; +using System.Text; + +namespace Ryujinx.Common +{ + public static class HexUtils + { + private static readonly char[] _hexChars = "0123456789ABCDEF".ToCharArray(); + + private const int HexTableColumnWidth = 8; + private const int HexTableColumnSpace = 3; + + // Modified for Ryujinx + // Original by Pascal Ganaye - CPOL License + // https://www.codeproject.com/Articles/36747/Quick-and-Dirty-HexDump-of-a-Byte-Array + public static string HexTable(byte[] bytes, int bytesPerLine = 16) + { + if (bytes == null) + { + return ""; + } + + int bytesLength = bytes.Length; + + int firstHexColumn = + HexTableColumnWidth + + HexTableColumnSpace; + + int firstCharColumn = firstHexColumn + + bytesPerLine * HexTableColumnSpace + + (bytesPerLine - 1) / HexTableColumnWidth + + 2; + + int lineLength = firstCharColumn + + bytesPerLine + + Environment.NewLine.Length; + + char[] line = (new String(' ', lineLength - Environment.NewLine.Length) + Environment.NewLine).ToCharArray(); + + int expectedLines = (bytesLength + bytesPerLine - 1) / bytesPerLine; + + StringBuilder result = new(expectedLines * lineLength); + + for (int i = 0; i < bytesLength; i += bytesPerLine) + { + line[0] = _hexChars[(i >> 28) & 0xF]; + line[1] = _hexChars[(i >> 24) & 0xF]; + line[2] = _hexChars[(i >> 20) & 0xF]; + line[3] = _hexChars[(i >> 16) & 0xF]; + line[4] = _hexChars[(i >> 12) & 0xF]; + line[5] = _hexChars[(i >> 8) & 0xF]; + line[6] = _hexChars[(i >> 4) & 0xF]; + line[7] = _hexChars[(i >> 0) & 0xF]; + + int hexColumn = firstHexColumn; + int charColumn = firstCharColumn; + + for (int j = 0; j < bytesPerLine; j++) + { + if (j > 0 && (j & 7) == 0) + { + hexColumn++; + } + + if (i + j >= bytesLength) + { + line[hexColumn] = ' '; + line[hexColumn + 1] = ' '; + line[charColumn] = ' '; + } + else + { + byte b = bytes[i + j]; + + line[hexColumn] = _hexChars[(b >> 4) & 0xF]; + line[hexColumn + 1] = _hexChars[b & 0xF]; + line[charColumn] = (b < 32 ? '·' : (char)b); + } + + hexColumn += 3; + charColumn++; + } + + result.Append(line); + } + + return result.ToString(); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/JsonHelper.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/JsonHelper.cs new file mode 100644 index 0000000..09fc7c0 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/JsonHelper.cs @@ -0,0 +1,86 @@ +using System; +using System.IO; +using System.Text; +using System.Text.Json; +using System.Text.Json.Serialization.Metadata; + +namespace Ryujinx.Common.Utilities +{ + public class JsonHelper + { + private static readonly JsonNamingPolicy _snakeCasePolicy = new SnakeCaseNamingPolicy(); + private const int DefaultFileWriteBufferSize = 4096; + + /// + /// Creates new serializer options with default settings. + /// + /// + /// It is REQUIRED for you to save returned options statically or as a part of static serializer context + /// in order to avoid performance issues. You can safely modify returned options for your case before storing. + /// + public static JsonSerializerOptions GetDefaultSerializerOptions(bool indented = true) => + new() + { + DictionaryKeyPolicy = _snakeCasePolicy, + PropertyNamingPolicy = _snakeCasePolicy, + WriteIndented = indented, + AllowTrailingCommas = true, + ReadCommentHandling = JsonCommentHandling.Skip + }; + + public static string Serialize(T value, JsonTypeInfo typeInfo) + => JsonSerializer.Serialize(value, typeInfo); + + public static T Deserialize(string value, JsonTypeInfo typeInfo) + => JsonSerializer.Deserialize(value, typeInfo); + + public static T Deserialize(ReadOnlySpan utf8Value, JsonTypeInfo typeInfo) + => JsonSerializer.Deserialize(utf8Value, typeInfo); + + public static void SerializeToFile(string filePath, T value, JsonTypeInfo typeInfo) + { + using FileStream file = File.Create(filePath, DefaultFileWriteBufferSize, FileOptions.WriteThrough); + JsonSerializer.Serialize(file, value, typeInfo); + } + + public static T DeserializeFromFile(string filePath, JsonTypeInfo typeInfo) + { + using FileStream file = File.OpenRead(filePath); + return JsonSerializer.Deserialize(file, typeInfo); + } + + public static void SerializeToStream(Stream stream, T value, JsonTypeInfo typeInfo) => JsonSerializer.Serialize(stream, value, typeInfo); + + private class SnakeCaseNamingPolicy : JsonNamingPolicy + { + public override string ConvertName(string name) + { + if (string.IsNullOrEmpty(name)) + { + return name; + } + + StringBuilder builder = new(); + + for (int i = 0; i < name.Length; i++) + { + char c = name[i]; + + if (char.IsUpper(c)) + { + if (!(i == 0 || char.IsUpper(name[i - 1]))) + builder.Append('_'); + + builder.Append(char.ToLowerInvariant(c)); + } + else + { + builder.Append(c); + } + } + + return builder.ToString(); + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/MessagePackObjectFormatter.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/MessagePackObjectFormatter.cs new file mode 100644 index 0000000..1881455 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/MessagePackObjectFormatter.cs @@ -0,0 +1,299 @@ +using MsgPack; +using System; +using System.Collections.Generic; +using System.Text; + +namespace Ryujinx.Common.Utilities +{ + public static class MessagePackObjectFormatter + { + public static string ToString(this MessagePackObject obj, bool pretty) + { + if (pretty) + { + return Format(obj); + } + + return obj.ToString(); + } + + public static string Format(MessagePackObject obj) + { + IndentedStringBuilder builder = new(); + + FormatMsgPackObj(obj, builder); + + return builder.ToString(); + } + + private static void FormatMsgPackObj(MessagePackObject obj, IndentedStringBuilder builder) + { + if (obj.IsMap || obj.IsDictionary) + { + FormatMsgPackMap(obj, builder); + } + else if (obj.IsArray || obj.IsList) + { + FormatMsgPackArray(obj, builder); + } + else if (obj.IsNil) + { + builder.Append("null"); + } + else + { + object literal = obj.ToObject(); + + if (literal is String) + { + builder.AppendQuotedString(obj.AsStringUtf16()); + } + else if (literal is byte[] byteArray) + { + FormatByteArray(byteArray, builder); + } + else if (literal is MessagePackExtendedTypeObject extObject) + { + builder.Append('{'); + + // Indent + builder.IncreaseIndent() + .AppendLine(); + + // Print TypeCode field + builder.AppendQuotedString("TypeCode") + .Append(": ") + .Append(extObject.TypeCode) + .AppendLine(","); + + // Print Value field + builder.AppendQuotedString("Value") + .Append(": "); + + FormatByteArrayAsString(extObject.GetBody(), builder, true); + + // Unindent + builder.DecreaseIndent() + .AppendLine(); + + builder.Append('}'); + } + else + { + builder.Append(literal); + } + } + } + + private static void FormatByteArray(byte[] arr, IndentedStringBuilder builder) + { + builder.Append("[ "); + + foreach (byte b in arr) + { + builder.Append("0x"); + builder.Append(ToHexChar(b >> 4)); + builder.Append(ToHexChar(b & 0xF)); + builder.Append(", "); + } + + // Remove trailing comma + builder.Remove(builder.Length - 2, 2); + + builder.Append(" ]"); + } + + private static void FormatByteArrayAsString(byte[] arr, IndentedStringBuilder builder, bool withPrefix) + { + builder.Append('"'); + + if (withPrefix) + { + builder.Append("0x"); + } + + foreach (byte b in arr) + { + builder.Append(ToHexChar(b >> 4)); + builder.Append(ToHexChar(b & 0xF)); + } + + builder.Append('"'); + } + + private static void FormatMsgPackMap(MessagePackObject obj, IndentedStringBuilder builder) + { + MessagePackObjectDictionary map = obj.AsDictionary(); + + builder.Append('{'); + + // Indent + builder.IncreaseIndent() + .AppendLine(); + + foreach (KeyValuePair item in map) + { + FormatMsgPackObj(item.Key, builder); + + builder.Append(": "); + + FormatMsgPackObj(item.Value, builder); + + builder.AppendLine(","); + } + + // Remove the trailing new line and comma + builder.TrimLastLine() + .Remove(builder.Length - 1, 1); + + // Unindent + builder.DecreaseIndent() + .AppendLine(); + + builder.Append('}'); + } + + private static void FormatMsgPackArray(MessagePackObject obj, IndentedStringBuilder builder) + { + IList arr = obj.AsList(); + + builder.Append("[ "); + + foreach (MessagePackObject item in arr) + { + FormatMsgPackObj(item, builder); + + builder.Append(", "); + } + + // Remove trailing comma + builder.Remove(builder.Length - 2, 2); + + builder.Append(" ]"); + } + + private static char ToHexChar(int b) + { + if (b < 10) + { + return unchecked((char)('0' + b)); + } + + return unchecked((char)('A' + (b - 10))); + } + + internal class IndentedStringBuilder + { + const string DefaultIndent = " "; + + private int _indentCount; + private int _newLineIndex; + private readonly StringBuilder _builder; + + public string IndentString { get; set; } = DefaultIndent; + + public IndentedStringBuilder(StringBuilder builder) + { + _builder = builder; + } + + public IndentedStringBuilder() + : this(new StringBuilder()) + { } + + public IndentedStringBuilder(string str) + : this(new StringBuilder(str)) + { } + + public IndentedStringBuilder(int length) + : this(new StringBuilder(length)) + { } + + public int Length { get => _builder.Length; } + + public IndentedStringBuilder IncreaseIndent() + { + _indentCount++; + + return this; + } + + public IndentedStringBuilder DecreaseIndent() + { + _indentCount--; + + return this; + } + + public IndentedStringBuilder Append(char value) + { + _builder.Append(value); + + return this; + } + + public IndentedStringBuilder Append(string value) + { + _builder.Append(value); + + return this; + } + + public IndentedStringBuilder Append(object value) + { + this.Append(value.ToString()); + + return this; + } + + public IndentedStringBuilder AppendQuotedString(string value) + { + _builder.Append('"'); + _builder.Append(value); + _builder.Append('"'); + + return this; + } + + public IndentedStringBuilder AppendLine() + { + _newLineIndex = _builder.Length; + + _builder.AppendLine(); + + for (int i = 0; i < _indentCount; i++) + _builder.Append(IndentString); + + return this; + } + + public IndentedStringBuilder AppendLine(string value) + { + _builder.Append(value); + + this.AppendLine(); + + return this; + } + + public IndentedStringBuilder TrimLastLine() + { + _builder.Remove(_newLineIndex, _builder.Length - _newLineIndex); + + return this; + } + + public IndentedStringBuilder Remove(int startIndex, int length) + { + _builder.Remove(startIndex, length); + + return this; + } + + public override string ToString() + { + return _builder.ToString(); + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/NetworkHelpers.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/NetworkHelpers.cs new file mode 100644 index 0000000..d770a30 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/NetworkHelpers.cs @@ -0,0 +1,89 @@ +using System.Buffers.Binary; +using System.Net; +using System.Net.NetworkInformation; +using System.Runtime.InteropServices; + +namespace Ryujinx.Common.Utilities +{ + public static class NetworkHelpers + { + private static (IPInterfaceProperties, UnicastIPAddressInformation) GetLocalInterface(NetworkInterface adapter, bool isPreferred) + { + IPInterfaceProperties properties = adapter.GetIPProperties(); + + if (isPreferred || (properties.GatewayAddresses.Count > 0 && properties.DnsAddresses.Count > 0)) + { + foreach (UnicastIPAddressInformation info in properties.UnicastAddresses) + { + // Only accept an IPv4 address + if (info.Address.GetAddressBytes().Length == 4) + { + return (properties, info); + } + } + } + + return (null, null); + } + + public static (IPInterfaceProperties, UnicastIPAddressInformation) GetLocalInterface(string lanInterfaceId = "0") + { + if (!NetworkInterface.GetIsNetworkAvailable()) + { + return (null, null); + } + + IPInterfaceProperties targetProperties = null; + UnicastIPAddressInformation targetAddressInfo = null; + + NetworkInterface[] interfaces = NetworkInterface.GetAllNetworkInterfaces(); + + string guid = lanInterfaceId; + bool hasPreference = guid != "0"; + + foreach (NetworkInterface adapter in interfaces) + { + bool isPreferred = adapter.Id == guid; + + // Ignore loopback and non IPv4 capable interface. + if (isPreferred || (targetProperties == null && adapter.NetworkInterfaceType != NetworkInterfaceType.Loopback && adapter.Supports(NetworkInterfaceComponent.IPv4))) + { + (IPInterfaceProperties properties, UnicastIPAddressInformation info) = GetLocalInterface(adapter, isPreferred); + + if (properties != null) + { + targetProperties = properties; + targetAddressInfo = info; + + if (isPreferred || !hasPreference) + { + break; + } + } + } + } + + return (targetProperties, targetAddressInfo); + } + + public static bool SupportsDynamicDns() + { + return RuntimeInformation.IsOSPlatform(OSPlatform.Windows); + } + + public static uint ConvertIpv4Address(IPAddress ipAddress) + { + return BinaryPrimitives.ReadUInt32BigEndian(ipAddress.GetAddressBytes()); + } + + public static uint ConvertIpv4Address(string ipAddress) + { + return ConvertIpv4Address(IPAddress.Parse(ipAddress)); + } + + public static IPAddress ConvertUint(uint ipAddress) + { + return new IPAddress(new byte[] { (byte)((ipAddress >> 24) & 0xFF), (byte)((ipAddress >> 16) & 0xFF), (byte)((ipAddress >> 8) & 0xFF), (byte)(ipAddress & 0xFF) }); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/OsUtils.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/OsUtils.cs new file mode 100644 index 0000000..519c91b --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/OsUtils.cs @@ -0,0 +1,49 @@ +using System; +using System.Diagnostics; +using System.Runtime.InteropServices; + +namespace Ryujinx.Common.Utilities +{ + public partial class OsUtils + { + [LibraryImport("libc", SetLastError = true)] + private static partial int setenv([MarshalAs(UnmanagedType.LPStr)] string name, [MarshalAs(UnmanagedType.LPStr)] string value, int overwrite); + + public static void SetEnvironmentVariableNoCaching(string key, string value) + { + // Set the value in the cached environment variables, too. + Environment.SetEnvironmentVariable(key, value); + + if (!OperatingSystem.IsWindows()) + { + int res = setenv(key, value, 1); + Debug.Assert(res != -1); + } + } + + // "dumpable" attribute of the calling process + private const int PR_GET_DUMPABLE = 3; + private const int PR_SET_DUMPABLE = 4; + + [LibraryImport("libc", SetLastError = true)] + private static partial int prctl(int option, int arg2); + + public static void SetCoreDumpable(bool dumpable) + { + if (OperatingSystem.IsLinux()) + { + int dumpableInt = dumpable ? 1 : 0; + int result = prctl(PR_SET_DUMPABLE, dumpableInt); + Debug.Assert(result == 0); + } + } + + // Use the below line to display dumpable status in the console: + // Console.WriteLine($"{OsUtils.IsCoreDumpable()}"); + public static bool IsCoreDumpable() + { + int result = prctl(PR_GET_DUMPABLE, 0); + return result == 1; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/Rainbow.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/Rainbow.cs new file mode 100644 index 0000000..09b2674 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/Rainbow.cs @@ -0,0 +1,131 @@ +using Gommon; +using Ryujinx.Common.Helper; +using System; +using System.Drawing; +using System.Threading; +using System.Threading.Tasks; + +namespace Ryujinx.Common.Utilities +{ + public static class Rainbow + { + public static bool CyclingEnabled { get; set; } + + public static void Enable() + { + if (!CyclingEnabled) + { + CyclingEnabled = true; + Executor.ExecuteBackgroundAsync(async () => + { + while (CyclingEnabled) + { + await Task.Delay(20); + Tick(); + } + }); + } + } + + public static void Disable() + { + CyclingEnabled = false; + } + + + public static float Speed { get; set; } = 1; + + private static readonly Lock _lock = new(); + + private static Color _color = Color.Blue; + + public static ref Color Color + { + get + { + lock (_lock) + { + return ref _color; + } + } + } + + public static void Tick() + { + lock (_lock) + { + _color = HsbToRgb((_color.GetHue() + Speed) / 360); + + _updatedHandler.Call(ref _color); + } + } + + public static void Reset() + { + _updatedHandler.Clear(); + + lock (_lock) + _color = Color.Blue; + } + + public static event RefEvent.Handler Updated + { + add => _updatedHandler.Add(value); + remove => _updatedHandler.Remove(value); + } + + private static readonly RefEvent _updatedHandler = new(); + + private static Color HsbToRgb(float hue, float saturation = 1, float brightness = 1) + { + int r = 0, g = 0, b = 0; + if (saturation == 0) + { + r = g = b = (int)(brightness * 255.0f + 0.5f); + } + else + { + float h = (hue - (float)Math.Floor(hue)) * 6.0f; + float f = h - (float)Math.Floor(h); + float p = brightness * (1.0f - saturation); + float q = brightness * (1.0f - saturation * f); + float t = brightness * (1.0f - (saturation * (1.0f - f))); + switch ((int)h) + { + case 0: + r = (int)(brightness * 255.0f + 0.5f); + g = (int)(t * 255.0f + 0.5f); + b = (int)(p * 255.0f + 0.5f); + break; + case 1: + r = (int)(q * 255.0f + 0.5f); + g = (int)(brightness * 255.0f + 0.5f); + b = (int)(p * 255.0f + 0.5f); + break; + case 2: + r = (int)(p * 255.0f + 0.5f); + g = (int)(brightness * 255.0f + 0.5f); + b = (int)(t * 255.0f + 0.5f); + break; + case 3: + r = (int)(p * 255.0f + 0.5f); + g = (int)(q * 255.0f + 0.5f); + b = (int)(brightness * 255.0f + 0.5f); + break; + case 4: + r = (int)(t * 255.0f + 0.5f); + g = (int)(p * 255.0f + 0.5f); + b = (int)(brightness * 255.0f + 0.5f); + break; + case 5: + r = (int)(brightness * 255.0f + 0.5f); + g = (int)(p * 255.0f + 0.5f); + b = (int)(q * 255.0f + 0.5f); + break; + } + } + + return Color.FromArgb(Convert.ToByte(255), Convert.ToByte(r), Convert.ToByte(g), Convert.ToByte(b)); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/SpanHelpers.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/SpanHelpers.cs new file mode 100644 index 0000000..88370a9 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/SpanHelpers.cs @@ -0,0 +1,61 @@ +using System; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; + +namespace Ryujinx.Common.Utilities +{ + public static class SpanHelpers + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Span CreateSpan(scoped ref T reference, int length) + { + return MemoryMarshal.CreateSpan(ref reference, length); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Span AsSpan(scoped ref T reference) where T : unmanaged + { + return CreateSpan(ref reference, 1); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Span AsSpan(scoped ref TStruct reference) + where TStruct : unmanaged where TSpan : unmanaged + { + return CreateSpan(ref Unsafe.As(ref reference), + Unsafe.SizeOf() / Unsafe.SizeOf()); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Span AsByteSpan(scoped ref T reference) where T : unmanaged + { + return CreateSpan(ref Unsafe.As(ref reference), Unsafe.SizeOf()); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static ReadOnlySpan CreateReadOnlySpan(scoped ref T reference, int length) + { + return MemoryMarshal.CreateReadOnlySpan(ref reference, length); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static ReadOnlySpan AsReadOnlySpan(scoped ref T reference) where T : unmanaged + { + return CreateReadOnlySpan(ref reference, 1); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static ReadOnlySpan AsReadOnlySpan(scoped ref TStruct reference) + where TStruct : unmanaged where TSpan : unmanaged + { + return CreateReadOnlySpan(ref Unsafe.As(ref reference), + Unsafe.SizeOf() / Unsafe.SizeOf()); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static ReadOnlySpan AsReadOnlyByteSpan(scoped ref T reference) where T : unmanaged + { + return CreateReadOnlySpan(ref Unsafe.As(ref reference), Unsafe.SizeOf()); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/StreamUtils.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/StreamUtils.cs new file mode 100644 index 0000000..35e6dbf --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/StreamUtils.cs @@ -0,0 +1,90 @@ +using Microsoft.IO; +using Ryujinx.Common.Memory; +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace Ryujinx.Common.Utilities +{ + public static class StreamUtils + { + public static byte[] StreamToBytes(Stream input) + { + using RecyclableMemoryStream output = StreamToRecyclableMemoryStream(input); + + return output.ToArray(); + } + + public static MemoryOwner StreamToRentedMemory(Stream input) + { + if (input is MemoryStream inputMemoryStream) + { + return MemoryStreamToRentedMemory(inputMemoryStream); + } + else if (input.CanSeek) + { + long bytesExpected = input.Length; + + MemoryOwner ownedMemory = MemoryOwner.Rent(checked((int)bytesExpected)); + + Span destSpan = ownedMemory.Span; + + int totalBytesRead = 0; + + while (totalBytesRead < bytesExpected) + { + int bytesRead = input.Read(destSpan[totalBytesRead..]); + + if (bytesRead == 0) + { + ownedMemory.Dispose(); + + throw new IOException($"Tried reading {bytesExpected} but the stream closed after reading {totalBytesRead}."); + } + + totalBytesRead += bytesRead; + } + + return ownedMemory; + } + else + { + // If input is (non-seekable) then copy twice: first into a RecyclableMemoryStream, then to a rented IMemoryOwner. + using RecyclableMemoryStream output = StreamToRecyclableMemoryStream(input); + + return MemoryStreamToRentedMemory(output); + } + } + + public static async Task StreamToBytesAsync(Stream input, CancellationToken cancellationToken = default) + { + using RecyclableMemoryStream stream = MemoryStreamManager.Shared.GetStream(); + + await input.CopyToAsync(stream, cancellationToken); + + return stream.ToArray(); + } + + private static MemoryOwner MemoryStreamToRentedMemory(MemoryStream input) + { + input.Position = 0; + + MemoryOwner ownedMemory = MemoryOwner.Rent(checked((int)input.Length)); + + // Discard the return value because we assume reading a MemoryStream always succeeds completely. + _ = input.Read(ownedMemory.Span); + + return ownedMemory; + } + + private static RecyclableMemoryStream StreamToRecyclableMemoryStream(Stream input) + { + RecyclableMemoryStream stream = MemoryStreamManager.Shared.GetStream(); + + input.CopyTo(stream); + + return stream; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/UInt128Utils.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/UInt128Utils.cs new file mode 100644 index 0000000..7686ced --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/UInt128Utils.cs @@ -0,0 +1,20 @@ +using System; +using System.Globalization; + +namespace Ryujinx.Common.Utilities +{ + public static class UInt128Utils + { + public static UInt128 FromHex(string hex) => + new( + ulong.Parse(hex.AsSpan(0, 16), NumberStyles.HexNumber), + ulong.Parse(hex.AsSpan(16), NumberStyles.HexNumber) + ); + + public static Int128 NextInt128(this Random rand) => + new((ulong)rand.NextInt64(), (ulong)rand.NextInt64()); + + public static UInt128 NextUInt128(this Random rand) => + new((ulong)rand.NextInt64(), (ulong)rand.NextInt64()); + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/XCIFileTrimmer.cs b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/XCIFileTrimmer.cs new file mode 100644 index 0000000..fb0544c --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Common/Utilities/XCIFileTrimmer.cs @@ -0,0 +1,524 @@ +// Uncomment the line below to ensure XCIFileTrimmer does not modify files +//#define XCI_TRIMMER_READ_ONLY_MODE + +using Gommon; +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.IO; +using System.Linq; +using System.Threading; + +namespace Ryujinx.Common.Utilities +{ + public sealed class XCIFileTrimmer + { + private const long BytesInAMegabyte = 1024 * 1024; + private const int BufferSize = 8 * (int)BytesInAMegabyte; + + private const long CartSizeMBinFormattedGB = 952; + private const int CartKeyAreaSize = 0x1000; + private const byte PaddingByte = 0xFF; + private const int HeaderFilePos = 0x100; + private const int CartSizeFilePos = 0x10D; + private const int DataSizeFilePos = 0x118; + private const string HeaderMagicValue = "HEAD"; + + /// + /// Cartridge Sizes (ByteIdentifier, SizeInGB) + /// + private static readonly Dictionary _cartSizesGB = new() + { + { 0xFA, 1 }, + { 0xF8, 2 }, + { 0xF0, 4 }, + { 0xE0, 8 }, + { 0xE1, 16 }, + { 0xE2, 32 } + }; + + private static long RecordsToByte(long records) + { + return 512 + (records * 512); + } + + public static bool CanTrim(string filename, ILog log = null) + { + if (Path.GetExtension(filename).Equals(".XCI", StringComparison.InvariantCultureIgnoreCase)) + { + XCIFileTrimmer trimmer = new(filename, log); + return trimmer.CanBeTrimmed; + } + + return false; + } + + public static bool CanUntrim(string filename, ILog log = null) + { + if (Path.GetExtension(filename).Equals(".XCI", StringComparison.InvariantCultureIgnoreCase)) + { + XCIFileTrimmer trimmer = new(filename, log); + return trimmer.CanBeUntrimmed; + } + + return false; + } + + private ILog _log; + private string _filename; + private FileStream _fileStream; + private BinaryReader _binaryReader; + private long _offsetB, _dataSizeB, _cartSizeB, _fileSizeB; + private bool _fileOK = true; + private bool _freeSpaceChecked = false; + private bool _freeSpaceValid = false; + + public enum OperationOutcome + { + Undetermined, + InvalidXCIFile, + NoTrimNecessary, + NoUntrimPossible, + FreeSpaceCheckFailed, + FileIOWriteError, + ReadOnlyFileCannotFix, + FileSizeChanged, + Successful, + Cancelled + } + + public enum LogType + { + Info, + Warn, + Error, + Progress + } + + public interface ILog + { + public void Write(LogType logType, string text); + public void Progress(long current, long total, string text, bool complete); + } + + public bool FileOK => _fileOK; + public bool Trimmed => _fileOK && FileSizeB < UntrimmedFileSizeB; + public bool ContainsKeyArea => _offsetB != 0; + public bool CanBeTrimmed => _fileOK && FileSizeB > TrimmedFileSizeB; + public bool CanBeUntrimmed => _fileOK && FileSizeB < UntrimmedFileSizeB; + public bool FreeSpaceChecked => _fileOK && _freeSpaceChecked; + public bool FreeSpaceValid => _fileOK && _freeSpaceValid; + public long DataSizeB => _dataSizeB; + public long CartSizeB => _cartSizeB; + public long FileSizeB => _fileSizeB; + public long DiskSpaceSavedB => CartSizeB - FileSizeB; + public long DiskSpaceSavingsB => CartSizeB - DataSizeB; + public long TrimmedFileSizeB => _offsetB + _dataSizeB; + public long UntrimmedFileSizeB => _offsetB + _cartSizeB; + + public ILog Log + { + get => _log; + set => _log = value; + } + + public String Filename + { + get => _filename; + set + { + _filename = value; + Reset(); + } + } + + public long Pos + { + get => _fileStream.Position; + set => _fileStream.Position = value; + } + + public XCIFileTrimmer(string path, ILog log = null) + { + Log = log; + Filename = path; + ReadHeader(); + } + + public void CheckFreeSpace(CancellationToken? cancelToken = null) + { + if (FreeSpaceChecked) + return; + + try + { + if (CanBeTrimmed) + { + _freeSpaceValid = false; + + OpenReaders(); + + try + { + Pos = TrimmedFileSizeB; + bool freeSpaceValid = true; + long readSizeB = FileSizeB - TrimmedFileSizeB; + + Stopwatch timedSw = Lambda.Timed(() => + { + freeSpaceValid = CheckPadding(readSizeB, cancelToken); + }); + + if (timedSw.Elapsed.TotalSeconds > 0) + { + Log?.Write(LogType.Info, $"Checked at {readSizeB / (double)XCIFileTrimmer.BytesInAMegabyte / timedSw.Elapsed.TotalSeconds:N} Mb/sec"); + } + + if (freeSpaceValid) + Log?.Write(LogType.Info, "Free space is valid"); + + _freeSpaceValid = freeSpaceValid; + } + finally + { + CloseReaders(); + } + } + else + { + Log?.Write(LogType.Warn, "There is no free space to check."); + _freeSpaceValid = false; + } + } + finally + { + _freeSpaceChecked = true; + } + } + + private bool CheckPadding(long readSizeB, CancellationToken? cancelToken = null) + { + long maxReads = readSizeB / XCIFileTrimmer.BufferSize; + long read = 0; + byte[] buffer = new byte[BufferSize]; + + while (true) + { + if (cancelToken.HasValue && cancelToken.Value.IsCancellationRequested) + { + return false; + } + + int bytes = _fileStream.Read(buffer, 0, XCIFileTrimmer.BufferSize); + if (bytes == 0) + break; + + Log?.Progress(read, maxReads, "Verifying file can be trimmed", false); + if (buffer.Take(bytes).AsParallel().Any(b => b != XCIFileTrimmer.PaddingByte)) + { + Log?.Write(LogType.Warn, "Free space is NOT valid"); + return false; + } + + read++; + } + + return true; + } + + private void Reset() + { + _freeSpaceChecked = false; + _freeSpaceValid = false; + ReadHeader(); + } + + public OperationOutcome Trim(CancellationToken? cancelToken = null) + { + if (!FileOK) + { + return OperationOutcome.InvalidXCIFile; + } + + if (!CanBeTrimmed) + { + return OperationOutcome.NoTrimNecessary; + } + + if (!FreeSpaceChecked) + { + CheckFreeSpace(cancelToken); + } + + if (!FreeSpaceValid) + { + if (cancelToken.HasValue && cancelToken.Value.IsCancellationRequested) + { + return OperationOutcome.Cancelled; + } + else + { + return OperationOutcome.FreeSpaceCheckFailed; + } + } + + Log?.Write(LogType.Info, "Trimming..."); + + try + { + FileInfo info = new(Filename); + if ((info.Attributes & FileAttributes.ReadOnly) == FileAttributes.ReadOnly) + { + try + { + Log?.Write(LogType.Info, "Attempting to remove ReadOnly attribute"); + File.SetAttributes(Filename, info.Attributes & ~FileAttributes.ReadOnly); + } + catch (Exception e) + { + Log?.Write(LogType.Error, e.ToString()); + return OperationOutcome.ReadOnlyFileCannotFix; + } + } + + if (info.Length != FileSizeB) + { + Log?.Write(LogType.Error, "File size has changed, cannot safely trim."); + return OperationOutcome.FileSizeChanged; + } + + FileStream outfileStream = new(_filename, FileMode.Open, FileAccess.Write, FileShare.Write); + + try + { + +#if !XCI_TRIMMER_READ_ONLY_MODE + outfileStream.SetLength(TrimmedFileSizeB); +#endif + return OperationOutcome.Successful; + } + finally + { + outfileStream.Close(); + Reset(); + } + } + catch (Exception e) + { + Log?.Write(LogType.Error, e.ToString()); + return OperationOutcome.FileIOWriteError; + } + } + + public OperationOutcome Untrim(CancellationToken? cancelToken = null) + { + if (!FileOK) + { + return OperationOutcome.InvalidXCIFile; + } + + if (!CanBeUntrimmed) + { + return OperationOutcome.NoUntrimPossible; + } + + try + { + Log?.Write(LogType.Info, "Untrimming..."); + + FileInfo info = new(Filename); + if ((info.Attributes & FileAttributes.ReadOnly) == FileAttributes.ReadOnly) + { + try + { + Log?.Write(LogType.Info, "Attempting to remove ReadOnly attribute"); + File.SetAttributes(Filename, info.Attributes & ~FileAttributes.ReadOnly); + } + catch (Exception e) + { + Log?.Write(LogType.Error, e.ToString()); + return OperationOutcome.ReadOnlyFileCannotFix; + } + } + + if (info.Length != FileSizeB) + { + Log?.Write(LogType.Error, "File size has changed, cannot safely untrim."); + return OperationOutcome.FileSizeChanged; + } + + FileStream outfileStream = new(_filename, FileMode.Append, FileAccess.Write, FileShare.Write); + long bytesToWriteB = UntrimmedFileSizeB - FileSizeB; + + try + { + Stopwatch timedSw = Lambda.Timed(() => + { + WritePadding(outfileStream, bytesToWriteB, cancelToken); + }); + + if (timedSw.Elapsed.TotalSeconds > 0) + { + Log?.Write(LogType.Info, $"Wrote at {bytesToWriteB / (double)XCIFileTrimmer.BytesInAMegabyte / timedSw.Elapsed.TotalSeconds:N} Mb/sec"); + } + + if (cancelToken.HasValue && cancelToken.Value.IsCancellationRequested) + { + return OperationOutcome.Cancelled; + } + else + { + return OperationOutcome.Successful; + } + } + finally + { + outfileStream.Close(); + Reset(); + } + } + catch (Exception e) + { + Log?.Write(LogType.Error, e.ToString()); + return OperationOutcome.FileIOWriteError; + } + } + + private void WritePadding(FileStream outfileStream, long bytesToWriteB, CancellationToken? cancelToken = null) + { + long bytesLeftToWriteB = bytesToWriteB; + long writes = bytesLeftToWriteB / XCIFileTrimmer.BufferSize; + int write = 0; + + try + { + byte[] buffer = new byte[BufferSize]; + Array.Fill(buffer, XCIFileTrimmer.PaddingByte); + + while (bytesLeftToWriteB > 0) + { + if (cancelToken.HasValue && cancelToken.Value.IsCancellationRequested) + { + return; + } + + long bytesToWrite = Math.Min(XCIFileTrimmer.BufferSize, bytesLeftToWriteB); + +#if !XCI_TRIMMER_READ_ONLY_MODE + outfileStream.Write(buffer, 0, (int)bytesToWrite); +#endif + + bytesLeftToWriteB -= bytesToWrite; + Log?.Progress(write, writes, "Writing padding data...", false); + write++; + } + } + finally + { + Log?.Progress(write, writes, "Writing padding data...", true); + } + } + + private void OpenReaders() + { + if (_binaryReader == null) + { + _fileStream = new FileStream(_filename, FileMode.Open, FileAccess.Read, FileShare.Read); + _binaryReader = new BinaryReader(_fileStream); + } + } + + private void CloseReaders() + { + if (_binaryReader != null && _binaryReader.BaseStream != null) + _binaryReader.Close(); + _binaryReader = null; + _fileStream = null; + GC.Collect(); + } + + private void ReadHeader() + { + try + { + OpenReaders(); + + try + { + // Attempt without key area + bool success = CheckAndReadHeader(false); + + if (!success) + { + // Attempt with key area + success = CheckAndReadHeader(true); + } + + _fileOK = success; + } + finally + { + CloseReaders(); + } + } + catch (Exception ex) + { + Log?.Write(LogType.Error, ex.Message); + _fileOK = false; + _dataSizeB = 0; + _cartSizeB = 0; + _fileSizeB = 0; + _offsetB = 0; + } + } + + private bool CheckAndReadHeader(bool assumeKeyArea) + { + // Read file size + _fileSizeB = _fileStream.Length; + if (_fileSizeB < 32 * 1024) + { + Log?.Write(LogType.Error, "The source file doesn't look like an XCI file as the data size is too small"); + return false; + } + + // Setup offset + _offsetB = (long)(assumeKeyArea ? XCIFileTrimmer.CartKeyAreaSize : 0); + + // Check header + Pos = _offsetB + XCIFileTrimmer.HeaderFilePos; + string head = System.Text.Encoding.ASCII.GetString(_binaryReader.ReadBytes(4)); + if (head != XCIFileTrimmer.HeaderMagicValue) + { + if (!assumeKeyArea) + { + Log?.Write(LogType.Warn, $"Incorrect header found, file mat contain a key area..."); + } + else + { + Log?.Write(LogType.Error, "The source file doesn't look like an XCI file as the header is corrupted"); + } + + return false; + } + + // Read Cart Size + Pos = _offsetB + XCIFileTrimmer.CartSizeFilePos; + byte cartSizeId = _binaryReader.ReadByte(); + if (!_cartSizesGB.TryGetValue(cartSizeId, out long cartSizeNGB)) + { + Log?.Write(LogType.Error, $"The source file doesn't look like an XCI file as the Cartridge Size is incorrect (0x{cartSizeId:X2})"); + return false; + } + + _cartSizeB = cartSizeNGB * XCIFileTrimmer.CartSizeMBinFormattedGB * XCIFileTrimmer.BytesInAMegabyte; + + // Read data size + Pos = _offsetB + XCIFileTrimmer.DataSizeFilePos; + long records = (long)BitConverter.ToUInt32(_binaryReader.ReadBytes(4), 0); + _dataSizeB = RecordsToByte(records); + + return true; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/AlphaTestOp.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/AlphaTestOp.cs new file mode 100644 index 0000000..078e050 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/AlphaTestOp.cs @@ -0,0 +1,14 @@ +namespace Ryujinx.Graphics.Shader +{ + public enum AlphaTestOp + { + Never = 1, + Less, + Equal, + LessOrEqual, + Greater, + NotEqual, + GreaterOrEqual, + Always, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/AttributeType.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/AttributeType.cs new file mode 100644 index 0000000..8f1abe5 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/AttributeType.cs @@ -0,0 +1,45 @@ +using Ryujinx.Graphics.Shader.Translation; +using System; + +namespace Ryujinx.Graphics.Shader +{ + [Flags] + public enum AttributeType : byte + { + // Generic types. + Float, + Sint, + Uint, + Sscaled, + Uscaled, + + Packed = 1 << 6, + PackedRgb10A2Signed = 1 << 7, + AnyPacked = Packed | PackedRgb10A2Signed, + } + + static class AttributeTypeExtensions + { + extension(AttributeType type) + { + public AggregateType Aggregate => + (type & ~AttributeType.AnyPacked) switch + { + AttributeType.Float => AggregateType.FP32, + AttributeType.Sint => AggregateType.S32, + AttributeType.Uint => AggregateType.U32, + _ => throw new ArgumentException($"Invalid attribute type \"{type}\"."), + }; + + public AggregateType AsAggregate(bool supportsScaledFormats) => (type & ~AttributeType.AnyPacked) switch + { + AttributeType.Float => AggregateType.FP32, + AttributeType.Sint => AggregateType.S32, + AttributeType.Uint => AggregateType.U32, + AttributeType.Sscaled => supportsScaledFormats ? AggregateType.FP32 : AggregateType.S32, + AttributeType.Uscaled => supportsScaledFormats ? AggregateType.FP32 : AggregateType.U32, + _ => throw new ArgumentException($"Invalid attribute type \"{type}\"."), + }; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/BufferDescriptor.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/BufferDescriptor.cs new file mode 100644 index 0000000..cbabbc1 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/BufferDescriptor.cs @@ -0,0 +1,34 @@ +namespace Ryujinx.Graphics.Shader +{ + public readonly struct BufferDescriptor + { + // New fields should be added to the end of the struct to keep disk shader cache compatibility. + + public readonly int Set; + public readonly int Binding; + public readonly byte Slot; + public readonly byte SbCbSlot; + public readonly ushort SbCbOffset; + public readonly BufferUsageFlags Flags; + + public BufferDescriptor(int set, int binding, int slot) + { + Set = set; + Binding = binding; + Slot = (byte)slot; + SbCbSlot = 0; + SbCbOffset = 0; + Flags = BufferUsageFlags.None; + } + + public BufferDescriptor(int set, int binding, int slot, int sbCbSlot, int sbCbOffset, BufferUsageFlags flags) + { + Set = set; + Binding = binding; + Slot = (byte)slot; + SbCbSlot = (byte)sbCbSlot; + SbCbOffset = (ushort)sbCbOffset; + Flags = flags; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/BufferUsageFlags.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/BufferUsageFlags.cs new file mode 100644 index 0000000..901b10a --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/BufferUsageFlags.cs @@ -0,0 +1,18 @@ +using System; + +namespace Ryujinx.Graphics.Shader +{ + /// + /// Flags that indicate how a buffer will be used in a shader. + /// + [Flags] + public enum BufferUsageFlags : byte + { + None = 0, + + /// + /// Buffer is written to. + /// + Write = 1 << 0, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/CodeGenParameters.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/CodeGenParameters.cs new file mode 100644 index 0000000..873ebfd --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/CodeGenParameters.cs @@ -0,0 +1,31 @@ +using Ryujinx.Graphics.Shader.StructuredIr; +using Ryujinx.Graphics.Shader.Translation; + +namespace Ryujinx.Graphics.Shader.CodeGen +{ + readonly struct CodeGenParameters + { + public readonly AttributeUsage AttributeUsage; + public readonly ShaderDefinitions Definitions; + public readonly ShaderProperties Properties; + public readonly HostCapabilities HostCapabilities; + public readonly ILogger Logger; + public readonly TargetApi TargetApi; + + public CodeGenParameters( + AttributeUsage attributeUsage, + ShaderDefinitions definitions, + ShaderProperties properties, + HostCapabilities hostCapabilities, + ILogger logger, + TargetApi targetApi) + { + AttributeUsage = attributeUsage; + Definitions = definitions; + Properties = properties; + HostCapabilities = hostCapabilities; + Logger = logger; + TargetApi = targetApi; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/CodeGenContext.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/CodeGenContext.cs new file mode 100644 index 0000000..1c7c15c --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/CodeGenContext.cs @@ -0,0 +1,105 @@ +using Ryujinx.Graphics.Shader.StructuredIr; +using Ryujinx.Graphics.Shader.Translation; +using System.Text; + +namespace Ryujinx.Graphics.Shader.CodeGen.Glsl +{ + class CodeGenContext + { + public const string Tab = " "; + + public StructuredFunction CurrentFunction { get; set; } + + public StructuredProgramInfo Info { get; } + + public AttributeUsage AttributeUsage { get; } + public ShaderDefinitions Definitions { get; } + public ShaderProperties Properties { get; } + public HostCapabilities HostCapabilities { get; } + public ILogger Logger { get; } + public TargetApi TargetApi { get; } + + public OperandManager OperandManager { get; } + + private readonly StringBuilder _sb; + + private int _level; + + private string _indentation; + + public CodeGenContext(StructuredProgramInfo info, CodeGenParameters parameters) + { + Info = info; + AttributeUsage = parameters.AttributeUsage; + Definitions = parameters.Definitions; + Properties = parameters.Properties; + HostCapabilities = parameters.HostCapabilities; + Logger = parameters.Logger; + TargetApi = parameters.TargetApi; + + OperandManager = new OperandManager(); + + _sb = new StringBuilder(); + } + + public void AppendLine() + { + _sb.AppendLine(); + } + + public void AppendLine(string str) + { + _sb.AppendLine(_indentation + str); + } + + public string GetCode() + { + return _sb.ToString(); + } + + public void EnterScope() + { + AppendLine("{"); + + _level++; + + UpdateIndentation(); + } + + public void LeaveScope(string suffix = "") + { + if (_level == 0) + { + return; + } + + _level--; + + UpdateIndentation(); + + AppendLine("}" + suffix); + } + + public StructuredFunction GetFunction(int id) + { + return Info.Functions[id]; + } + + private void UpdateIndentation() + { + _indentation = GetIndentation(_level); + } + + private static string GetIndentation(int level) + { + StringBuilder indentationBuilder = new(); + + for (int index = 0; index < level; index++) + { + indentationBuilder.Append(Tab); + } + + return indentationBuilder.ToString(); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Declarations.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Declarations.cs new file mode 100644 index 0000000..d027396 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Declarations.cs @@ -0,0 +1,659 @@ +using Ryujinx.Common; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.StructuredIr; +using Ryujinx.Graphics.Shader.Translation; +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; + +namespace Ryujinx.Graphics.Shader.CodeGen.Glsl +{ + static class Declarations + { + public static void Declare(CodeGenContext context, StructuredProgramInfo info) + { + context.AppendLine(context.TargetApi == TargetApi.Vulkan ? "#version 460 core" : "#version 450 core"); + context.AppendLine("#extension GL_ARB_gpu_shader_int64 : enable"); + + if (context.HostCapabilities.SupportsShaderBallot) + { + context.AppendLine("#extension GL_ARB_shader_ballot : enable"); + } + else + { + context.AppendLine("#extension GL_KHR_shader_subgroup_basic : enable"); + context.AppendLine("#extension GL_KHR_shader_subgroup_ballot : enable"); + context.AppendLine("#extension GL_KHR_shader_subgroup_shuffle : enable"); + } + + context.AppendLine("#extension GL_ARB_shader_group_vote : enable"); + context.AppendLine("#extension GL_EXT_shader_image_load_formatted : enable"); + context.AppendLine("#extension GL_EXT_texture_shadow_lod : enable"); + + if (context.Definitions.Stage == ShaderStage.Compute) + { + context.AppendLine("#extension GL_ARB_compute_shader : enable"); + } + else if (context.Definitions.Stage == ShaderStage.Fragment) + { + if (context.HostCapabilities.SupportsFragmentShaderInterlock) + { + context.AppendLine("#extension GL_ARB_fragment_shader_interlock : enable"); + } + else if (context.HostCapabilities.SupportsFragmentShaderOrderingIntel) + { + context.AppendLine("#extension GL_INTEL_fragment_shader_ordering : enable"); + } + } + else + { + if (context.Definitions.Stage == ShaderStage.Vertex) + { + context.AppendLine("#extension GL_ARB_shader_draw_parameters : enable"); + } + + context.AppendLine("#extension GL_ARB_shader_viewport_layer_array : enable"); + } + + if (context.Definitions.GpPassthrough && context.HostCapabilities.SupportsGeometryShaderPassthrough) + { + context.AppendLine("#extension GL_NV_geometry_shader_passthrough : enable"); + } + + if (context.HostCapabilities.SupportsViewportMask) + { + context.AppendLine("#extension GL_NV_viewport_array2 : enable"); + } + + context.AppendLine("#pragma optionNV(fastmath off)"); + context.AppendLine(); + + context.AppendLine($"const int {DefaultNames.UndefinedName} = 0;"); + context.AppendLine(); + + DeclareConstantBuffers(context, context.Properties.ConstantBuffers.Values); + DeclareStorageBuffers(context, context.Properties.StorageBuffers.Values); + DeclareMemories(context, context.Properties.LocalMemories.Values, isShared: false); + DeclareMemories(context, context.Properties.SharedMemories.Values, isShared: true); + DeclareSamplers(context, context.Properties.Textures.Values); + DeclareImages(context, context.Properties.Images.Values); + + if (context.Definitions.Stage != ShaderStage.Compute) + { + if (context.Definitions.Stage == ShaderStage.Geometry) + { + string inPrimitive = context.Definitions.InputTopology.GlslString; + + context.AppendLine($"layout (invocations = {context.Definitions.ThreadsPerInputPrimitive}, {inPrimitive}) in;"); + + if (context.Definitions.GpPassthrough && context.HostCapabilities.SupportsGeometryShaderPassthrough) + { + context.AppendLine($"layout (passthrough) in gl_PerVertex"); + context.EnterScope(); + context.AppendLine("vec4 gl_Position;"); + context.AppendLine("float gl_PointSize;"); + context.AppendLine("float gl_ClipDistance[];"); + context.LeaveScope(";"); + } + else + { + string outPrimitive = context.Definitions.OutputTopology.GlslString; + int maxOutputVertices = context.Definitions.MaxOutputVertices; + + context.AppendLine($"layout ({outPrimitive}, max_vertices = {maxOutputVertices}) out;"); + } + + context.AppendLine(); + } + else if (context.Definitions.Stage == ShaderStage.TessellationControl) + { + int threadsPerInputPrimitive = context.Definitions.ThreadsPerInputPrimitive; + + context.AppendLine($"layout (vertices = {threadsPerInputPrimitive}) out;"); + context.AppendLine(); + } + else if (context.Definitions.Stage == ShaderStage.TessellationEvaluation) + { + bool tessCw = context.Definitions.TessCw; + + if (context.TargetApi == TargetApi.Vulkan) + { + // We invert the front face on Vulkan backend, so we need to do that here aswell. + tessCw = !tessCw; + } + + string patchType = context.Definitions.TessPatchType.Glsl; + string spacing = context.Definitions.TessSpacing.Glsl; + string windingOrder = tessCw ? "cw" : "ccw"; + + context.AppendLine($"layout ({patchType}, {spacing}, {windingOrder}) in;"); + context.AppendLine(); + } + + static bool IsUserDefined(IoDefinition ioDefinition, StorageKind storageKind) + { + return ioDefinition.StorageKind == storageKind && ioDefinition.IoVariable == IoVariable.UserDefined; + } + + static bool IsUserDefinedOutput(ShaderStage stage, IoDefinition ioDefinition) + { + IoVariable ioVariable = stage == ShaderStage.Fragment ? IoVariable.FragmentOutputColor : IoVariable.UserDefined; + return ioDefinition.StorageKind == StorageKind.Output && ioDefinition.IoVariable == ioVariable; + } + + DeclareInputAttributes(context, info.IoDefinitions.Where(x => IsUserDefined(x, StorageKind.Input))); + DeclareOutputAttributes(context, info.IoDefinitions.Where(x => IsUserDefinedOutput(context.Definitions.Stage, x))); + DeclareInputAttributesPerPatch(context, info.IoDefinitions.Where(x => IsUserDefined(x, StorageKind.InputPerPatch))); + DeclareOutputAttributesPerPatch(context, info.IoDefinitions.Where(x => IsUserDefined(x, StorageKind.OutputPerPatch))); + + if (context.Definitions.TransformFeedbackEnabled && context.Definitions.LastInVertexPipeline) + { + TransformFeedbackOutput tfOutput = context.Definitions.GetTransformFeedbackOutput(AttributeConsts.PositionX); + if (tfOutput.Valid) + { + context.AppendLine($"layout (xfb_buffer = {tfOutput.Buffer}, xfb_offset = {tfOutput.Offset}, xfb_stride = {tfOutput.Stride}) out gl_PerVertex"); + context.EnterScope(); + context.AppendLine("vec4 gl_Position;"); + context.LeaveScope(context.Definitions.Stage == ShaderStage.TessellationControl ? " gl_out[];" : ";"); + } + } + } + else + { + string localSizeX = NumberFormatter.FormatInt(context.Definitions.ComputeLocalSizeX); + string localSizeY = NumberFormatter.FormatInt(context.Definitions.ComputeLocalSizeY); + string localSizeZ = NumberFormatter.FormatInt(context.Definitions.ComputeLocalSizeZ); + + context.AppendLine( + "layout (" + + $"local_size_x = {localSizeX}, " + + $"local_size_y = {localSizeY}, " + + $"local_size_z = {localSizeZ}) in;"); + context.AppendLine(); + } + + if (context.Definitions.Stage == ShaderStage.Fragment) + { + if (context.Definitions.EarlyZForce) + { + context.AppendLine("layout (early_fragment_tests) in;"); + context.AppendLine(); + } + + if (context.Definitions.OriginUpperLeft) + { + context.AppendLine("layout (origin_upper_left) in vec4 gl_FragCoord;"); + context.AppendLine(); + } + } + + if ((info.HelperFunctionsMask & HelperFunctionsMask.MultiplyHighS32) != 0) + { + AppendHelperFunction(context, "Ryujinx.Graphics.Shader/CodeGen/Glsl/HelperFunctions/MultiplyHighS32.glsl"); + } + + if ((info.HelperFunctionsMask & HelperFunctionsMask.MultiplyHighU32) != 0) + { + AppendHelperFunction(context, "Ryujinx.Graphics.Shader/CodeGen/Glsl/HelperFunctions/MultiplyHighU32.glsl"); + } + + if ((info.HelperFunctionsMask & HelperFunctionsMask.SwizzleAdd) != 0) + { + AppendHelperFunction(context, "Ryujinx.Graphics.Shader/CodeGen/Glsl/HelperFunctions/SwizzleAdd.glsl"); + } + } + + public static void DeclareLocals(CodeGenContext context, StructuredFunction function) + { + foreach (AstOperand decl in function.Locals) + { + string name = context.OperandManager.DeclareLocal(decl); + + context.AppendLine(GetVarTypeName(context, decl.VarType) + " " + name + ";"); + } + } + + public static string GetVarTypeName(CodeGenContext context, AggregateType type, bool precise = true) + { + if (context.HostCapabilities.ReducedPrecision) + { + precise = false; + } + + return type switch + { + AggregateType.Void => "void", + AggregateType.Bool => "bool", + AggregateType.FP32 => precise ? "precise float" : "float", + AggregateType.FP64 => "double", + AggregateType.S32 => "int", + AggregateType.U32 => "uint", + AggregateType.Vector2 | AggregateType.Bool => "bvec2", + AggregateType.Vector2 | AggregateType.FP32 => precise ? "precise vec2" : "vec2", + AggregateType.Vector2 | AggregateType.FP64 => "dvec2", + AggregateType.Vector2 | AggregateType.S32 => "ivec2", + AggregateType.Vector2 | AggregateType.U32 => "uvec2", + AggregateType.Vector3 | AggregateType.Bool => "bvec3", + AggregateType.Vector3 | AggregateType.FP32 => precise ? "precise vec3" : "vec3", + AggregateType.Vector3 | AggregateType.FP64 => "dvec3", + AggregateType.Vector3 | AggregateType.S32 => "ivec3", + AggregateType.Vector3 | AggregateType.U32 => "uvec3", + AggregateType.Vector4 | AggregateType.Bool => "bvec4", + AggregateType.Vector4 | AggregateType.FP32 => precise ? "precise vec4" : "vec4", + AggregateType.Vector4 | AggregateType.FP64 => "dvec4", + AggregateType.Vector4 | AggregateType.S32 => "ivec4", + AggregateType.Vector4 | AggregateType.U32 => "uvec4", + _ => throw new ArgumentException($"Invalid variable type \"{type}\"."), + }; + } + + private static void DeclareConstantBuffers(CodeGenContext context, IEnumerable buffers) + { + DeclareBuffers(context, buffers, "uniform"); + } + + private static void DeclareStorageBuffers(CodeGenContext context, IEnumerable buffers) + { + DeclareBuffers(context, buffers, "buffer"); + } + + private static void DeclareBuffers(CodeGenContext context, IEnumerable buffers, string declType) + { + foreach (BufferDefinition buffer in buffers) + { + string layout = buffer.Layout switch + { + BufferLayout.Std140 => "std140", + _ => "std430", + }; + + string set = string.Empty; + + if (context.TargetApi == TargetApi.Vulkan) + { + set = $"set = {buffer.Set}, "; + } + + context.AppendLine($"layout ({set}binding = {buffer.Binding}, {layout}) {declType} _{buffer.Name}"); + context.EnterScope(); + + foreach (StructureField field in buffer.Type.Fields) + { + if (field.Type.HasFlag(AggregateType.Array)) + { + string typeName = GetVarTypeName(context, field.Type & ~AggregateType.Array); + + if (field.ArrayLength > 0) + { + string arraySize = field.ArrayLength.ToString(CultureInfo.InvariantCulture); + + context.AppendLine($"{typeName} {field.Name}[{arraySize}];"); + } + else + { + context.AppendLine($"{typeName} {field.Name}[];"); + } + } + else + { + string typeName = GetVarTypeName(context, field.Type); + + context.AppendLine($"{typeName} {field.Name};"); + } + } + + context.LeaveScope($" {buffer.Name};"); + context.AppendLine(); + } + } + + private static void DeclareMemories(CodeGenContext context, IEnumerable memories, bool isShared) + { + string prefix = isShared ? "shared " : string.Empty; + + foreach (MemoryDefinition memory in memories) + { + string typeName = GetVarTypeName(context, memory.Type & ~AggregateType.Array); + + if (memory.Type.HasFlag(AggregateType.Array)) + { + if (memory.ArrayLength > 0) + { + string arraySize = memory.ArrayLength.ToString(CultureInfo.InvariantCulture); + + context.AppendLine($"{prefix}{typeName} {memory.Name}[{arraySize}];"); + } + else + { + context.AppendLine($"{prefix}{typeName} {memory.Name}[];"); + } + } + else + { + context.AppendLine($"{prefix}{typeName} {memory.Name};"); + } + } + } + + private static void DeclareSamplers(CodeGenContext context, IEnumerable definitions) + { + foreach (TextureDefinition definition in definitions) + { + string arrayDecl = string.Empty; + + if (definition.ArrayLength > 1) + { + arrayDecl = $"[{NumberFormatter.FormatInt(definition.ArrayLength)}]"; + } + else if (definition.ArrayLength == 0) + { + arrayDecl = "[]"; + } + + string samplerTypeName = definition.Separate ? definition.Type.GlslTextureTypeName : definition.Type.GlslSamplerTypeName; + + string layout = string.Empty; + + if (context.TargetApi == TargetApi.Vulkan) + { + layout = $", set = {definition.Set}"; + } + + context.AppendLine($"layout (binding = {definition.Binding}{layout}) uniform {samplerTypeName} {definition.Name}{arrayDecl};"); + } + } + + private static void DeclareImages(CodeGenContext context, IEnumerable definitions) + { + foreach (TextureDefinition definition in definitions) + { + string arrayDecl = string.Empty; + + if (definition.ArrayLength > 1) + { + arrayDecl = $"[{NumberFormatter.FormatInt(definition.ArrayLength)}]"; + } + else if (definition.ArrayLength == 0) + { + arrayDecl = "[]"; + } + + string imageTypeName = definition.Type.GetGlslImageTypeName(definition.Format.GetComponentType()); + + if (definition.Flags.HasFlag(TextureUsageFlags.ImageCoherent)) + { + imageTypeName = "coherent " + imageTypeName; + } + + string layout = definition.Format.ToGlslFormat(); + + if (!string.IsNullOrEmpty(layout)) + { + layout = ", " + layout; + } + + if (context.TargetApi == TargetApi.Vulkan) + { + layout = $", set = {definition.Set}{layout}"; + } + + context.AppendLine($"layout (binding = {definition.Binding}{layout}) uniform {imageTypeName} {definition.Name}{arrayDecl};"); + } + } + + private static void DeclareInputAttributes(CodeGenContext context, IEnumerable inputs) + { + if (context.Definitions.IaIndexing) + { + string suffix = context.Definitions.Stage == ShaderStage.Geometry ? "[]" : string.Empty; + + context.AppendLine($"layout (location = 0) in vec4 {DefaultNames.IAttributePrefix}{suffix}[{Constants.MaxAttributes}];"); + context.AppendLine(); + } + else + { + foreach (IoDefinition ioDefinition in inputs.OrderBy(x => x.Location)) + { + DeclareInputAttribute(context, ioDefinition.Location, ioDefinition.Component); + } + + if (inputs.Any()) + { + context.AppendLine(); + } + } + } + + private static void DeclareInputAttributesPerPatch(CodeGenContext context, IEnumerable inputs) + { + foreach (IoDefinition ioDefinition in inputs.OrderBy(x => x.Location)) + { + DeclareInputAttributePerPatch(context, ioDefinition.Location); + } + + if (inputs.Any()) + { + context.AppendLine(); + } + } + + private static void DeclareInputAttribute(CodeGenContext context, int location, int component) + { + string suffix = IsArrayAttributeGlsl(context.Definitions.Stage, isOutAttr: false) ? "[]" : string.Empty; + string iq = string.Empty; + + if (context.Definitions.Stage == ShaderStage.Fragment) + { + iq = context.Definitions.ImapTypes[location].GetFirstUsedType() switch + { + PixelImap.Constant => "flat ", + PixelImap.ScreenLinear => "noperspective ", + _ => string.Empty, + }; + } + + string name = $"{DefaultNames.IAttributePrefix}{location}"; + + if (context.Definitions.TransformFeedbackEnabled && context.Definitions.Stage == ShaderStage.Fragment) + { + bool hasComponent = context.Definitions.HasPerLocationInputOrOutputComponent(IoVariable.UserDefined, location, component, isOutput: false); + + if (hasComponent) + { + char swzMask = "xyzw"[component]; + + context.AppendLine($"layout (location = {location}, component = {component}) {iq}in float {name}_{swzMask}{suffix};"); + } + else + { + int components = context.Definitions.GetTransformFeedbackOutputComponents(location, 0); + + string type = components switch + { + 2 => "vec2", + 3 => "vec3", + 4 => "vec4", + _ => "float", + }; + + context.AppendLine($"layout (location = {location}) in {type} {name};"); + } + } + else + { + bool passthrough = (context.AttributeUsage.PassthroughAttributes & (1 << location)) != 0; + string pass = passthrough && context.HostCapabilities.SupportsGeometryShaderPassthrough ? "passthrough, " : string.Empty; + string type = GetVarTypeName(context, context.Definitions.GetUserDefinedType(location, isOutput: false), false); + + context.AppendLine($"layout ({pass}location = {location}) {iq}in {type} {name}{suffix};"); + } + } + + private static void DeclareInputAttributePerPatch(CodeGenContext context, int patchLocation) + { + int location = context.AttributeUsage.GetPerPatchAttributeLocation(patchLocation); + string name = $"{DefaultNames.PerPatchAttributePrefix}{patchLocation}"; + + context.AppendLine($"layout (location = {location}) patch in vec4 {name};"); + } + + private static void DeclareOutputAttributes(CodeGenContext context, IEnumerable outputs) + { + if (context.Definitions.OaIndexing) + { + context.AppendLine($"layout (location = 0) out vec4 {DefaultNames.OAttributePrefix}[{Constants.MaxAttributes}];"); + context.AppendLine(); + } + else + { + outputs = outputs.OrderBy(x => x.Location); + + if (context.Definitions.Stage == ShaderStage.Fragment && context.Definitions.DualSourceBlend) + { + IoDefinition firstOutput = outputs.ElementAtOrDefault(0); + + DeclareOutputDualSourceBlendAttribute(context, firstOutput.Location); + outputs = outputs.Skip(2); + } + + foreach (IoDefinition ioDefinition in outputs) + { + DeclareOutputAttribute(context, ioDefinition.Location, ioDefinition.Component); + } + + if (outputs.Any()) + { + context.AppendLine(); + } + } + } + + private static void DeclareOutputAttribute(CodeGenContext context, int location, int component) + { + string suffix = IsArrayAttributeGlsl(context.Definitions.Stage, isOutAttr: true) ? "[]" : string.Empty; + string name = $"{DefaultNames.OAttributePrefix}{location}{suffix}"; + + if (context.Definitions.TransformFeedbackEnabled && context.Definitions.LastInVertexPipeline) + { + bool hasComponent = context.Definitions.HasPerLocationInputOrOutputComponent(IoVariable.UserDefined, location, component, isOutput: true); + + if (hasComponent) + { + char swzMask = "xyzw"[component]; + + string xfb = string.Empty; + + TransformFeedbackOutput tfOutput = context.Definitions.GetTransformFeedbackOutput(location, component); + if (tfOutput.Valid) + { + xfb = $", xfb_buffer = {tfOutput.Buffer}, xfb_offset = {tfOutput.Offset}, xfb_stride = {tfOutput.Stride}"; + } + + context.AppendLine($"layout (location = {location}, component = {component}{xfb}) out float {name}_{swzMask};"); + } + else + { + int components = context.Definitions.GetTransformFeedbackOutputComponents(location, 0); + + string type = components switch + { + 2 => "vec2", + 3 => "vec3", + 4 => "vec4", + _ => "float", + }; + + string xfb = string.Empty; + + TransformFeedbackOutput tfOutput = context.Definitions.GetTransformFeedbackOutput(location, 0); + if (tfOutput.Valid) + { + xfb = $", xfb_buffer = {tfOutput.Buffer}, xfb_offset = {tfOutput.Offset}, xfb_stride = {tfOutput.Stride}"; + } + + context.AppendLine($"layout (location = {location}{xfb}) out {type} {name};"); + } + } + else + { + string type = context.Definitions.Stage != ShaderStage.Fragment ? "vec4" : + GetVarTypeName(context, context.Definitions.GetFragmentOutputColorType(location), false); + + if (context.HostCapabilities.ReducedPrecision && context.Definitions.Stage == ShaderStage.Vertex && location == 0) + { + context.AppendLine($"layout (location = {location}) invariant out {type} {name};"); + } + else + { + context.AppendLine($"layout (location = {location}) out {type} {name};"); + } + } + } + + private static void DeclareOutputDualSourceBlendAttribute(CodeGenContext context, int location) + { + string name = $"{DefaultNames.OAttributePrefix}{location}"; + string name2 = $"{DefaultNames.OAttributePrefix}{(location + 1)}"; + + context.AppendLine($"layout (location = {location}, index = 0) out vec4 {name};"); + context.AppendLine($"layout (location = {location}, index = 1) out vec4 {name2};"); + } + + private static void DeclareOutputAttributesPerPatch(CodeGenContext context, IEnumerable outputs) + { + foreach (IoDefinition ioDefinition in outputs) + { + DeclareOutputAttributePerPatch(context, ioDefinition.Location); + } + + if (outputs.Any()) + { + context.AppendLine(); + } + } + + private static void DeclareOutputAttributePerPatch(CodeGenContext context, int patchLocation) + { + int location = context.AttributeUsage.GetPerPatchAttributeLocation(patchLocation); + string name = $"{DefaultNames.PerPatchAttributePrefix}{patchLocation}"; + + context.AppendLine($"layout (location = {location}) patch out vec4 {name};"); + } + + private static bool IsArrayAttributeGlsl(ShaderStage stage, bool isOutAttr) + { + if (isOutAttr) + { + return stage == ShaderStage.TessellationControl; + } + else + { + return stage is ShaderStage.TessellationControl or + ShaderStage.TessellationEvaluation or + ShaderStage.Geometry; + } + } + + private static void AppendHelperFunction(CodeGenContext context, string filename) + { + string code = EmbeddedResources.ReadAllText(filename); + + code = code.Replace("\t", CodeGenContext.Tab); + + if (context.HostCapabilities.SupportsShaderBallot) + { + code = code.Replace("$SUBGROUP_INVOCATION$", "gl_SubGroupInvocationARB"); + code = code.Replace("$SUBGROUP_BROADCAST$", "readInvocationARB"); + } + else + { + code = code.Replace("$SUBGROUP_INVOCATION$", "gl_SubgroupInvocationID"); + code = code.Replace("$SUBGROUP_BROADCAST$", "subgroupBroadcast"); + } + + context.AppendLine(code); + context.AppendLine(); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/DefaultNames.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/DefaultNames.cs new file mode 100644 index 0000000..56c2c91 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/DefaultNames.cs @@ -0,0 +1,15 @@ +namespace Ryujinx.Graphics.Shader.CodeGen.Glsl +{ + static class DefaultNames + { + public const string LocalNamePrefix = "temp"; + + public const string PerPatchAttributePrefix = "patch_attr_"; + public const string IAttributePrefix = "in_attr"; + public const string OAttributePrefix = "out_attr"; + + public const string ArgumentNamePrefix = "a"; + + public const string UndefinedName = "undef"; + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/GlslGenerator.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/GlslGenerator.cs new file mode 100644 index 0000000..7fb1cbb --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/GlslGenerator.cs @@ -0,0 +1,177 @@ +using Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions; +using Ryujinx.Graphics.Shader.StructuredIr; +using Ryujinx.Graphics.Shader.Translation; +using System; + +using static Ryujinx.Graphics.Shader.CodeGen.Glsl.TypeConversion; + +namespace Ryujinx.Graphics.Shader.CodeGen.Glsl +{ + static class GlslGenerator + { + private const string MainFunctionName = "main"; + + public static string Generate(StructuredProgramInfo info, CodeGenParameters parameters) + { + CodeGenContext context = new(info, parameters); + + Declarations.Declare(context, info); + + if (info.Functions.Count != 0) + { + for (int i = 1; i < info.Functions.Count; i++) + { + context.AppendLine($"{GetFunctionSignature(context, info.Functions[i])};"); + } + + context.AppendLine(); + + for (int i = 1; i < info.Functions.Count; i++) + { + PrintFunction(context, info.Functions[i]); + + context.AppendLine(); + } + } + + PrintFunction(context, info.Functions[0], MainFunctionName); + + return context.GetCode(); + } + + private static void PrintFunction(CodeGenContext context, StructuredFunction function, string funcName = null) + { + context.CurrentFunction = function; + + context.AppendLine(GetFunctionSignature(context, function, funcName)); + context.EnterScope(); + + Declarations.DeclareLocals(context, function); + + PrintBlock(context, function.MainBlock, funcName == MainFunctionName); + + context.LeaveScope(); + } + + private static string GetFunctionSignature(CodeGenContext context, StructuredFunction function, string funcName = null) + { + string[] args = new string[function.InArguments.Length + function.OutArguments.Length]; + + for (int i = 0; i < function.InArguments.Length; i++) + { + args[i] = $"{Declarations.GetVarTypeName(context, function.InArguments[i])} {OperandManager.GetArgumentName(i)}"; + } + + for (int i = 0; i < function.OutArguments.Length; i++) + { + int j = i + function.InArguments.Length; + + args[j] = $"out {Declarations.GetVarTypeName(context, function.OutArguments[i])} {OperandManager.GetArgumentName(j)}"; + } + + return $"{Declarations.GetVarTypeName(context, function.ReturnType)} {funcName ?? function.Name}({string.Join(", ", args)})"; + } + + private static void PrintBlock(CodeGenContext context, AstBlock block, bool isMainFunction) + { + AstBlockVisitor visitor = new(block); + + visitor.BlockEntered += (sender, e) => + { + switch (e.Block.Type) + { + case AstBlockType.DoWhile: + context.AppendLine("do"); + break; + + case AstBlockType.Else: + context.AppendLine("else"); + break; + + case AstBlockType.ElseIf: + context.AppendLine($"else if ({GetCondExpr(context, e.Block.Condition)})"); + break; + + case AstBlockType.If: + context.AppendLine($"if ({GetCondExpr(context, e.Block.Condition)})"); + break; + + default: + throw new InvalidOperationException($"Found unexpected block type \"{e.Block.Type}\"."); + } + + context.EnterScope(); + }; + + visitor.BlockLeft += (sender, e) => + { + context.LeaveScope(); + + if (e.Block.Type == AstBlockType.DoWhile) + { + context.AppendLine($"while ({GetCondExpr(context, e.Block.Condition)});"); + } + }; + + bool supportsBarrierDivergence = context.HostCapabilities.SupportsShaderBarrierDivergence; + bool mayHaveReturned = false; + + foreach (IAstNode node in visitor.Visit()) + { + if (node is AstOperation operation) + { + if (!supportsBarrierDivergence) + { + if (operation.Inst == IntermediateRepresentation.Instruction.Barrier) + { + // Barrier on divergent control flow paths may cause the GPU to hang, + // so skip emitting the barrier for those cases. + if (visitor.Block.Type != AstBlockType.Main || mayHaveReturned || !isMainFunction) + { + context.Logger.Log("Shader has barrier on potentially divergent block, the barrier will be removed."); + + continue; + } + } + else if (operation.Inst == IntermediateRepresentation.Instruction.Return) + { + mayHaveReturned = true; + } + } + + string expr = InstGen.GetExpression(context, operation); + + if (expr != null) + { + context.AppendLine(expr + ";"); + } + } + else if (node is AstAssignment assignment) + { + AggregateType dstType = OperandManager.GetNodeDestType(context, assignment.Destination); + AggregateType srcType = OperandManager.GetNodeDestType(context, assignment.Source); + + string dest = InstGen.GetExpression(context, assignment.Destination); + string src = ReinterpretCast(context, assignment.Source, srcType, dstType); + + context.AppendLine(dest + " = " + src + ";"); + } + else if (node is AstComment comment) + { + context.AppendLine("// " + comment.Comment); + } + else + { + throw new InvalidOperationException($"Found unexpected node type \"{node?.GetType().Name ?? "null"}\"."); + } + } + } + + private static string GetCondExpr(CodeGenContext context, IAstNode cond) + { + AggregateType srcType = OperandManager.GetNodeDestType(context, cond); + + return ReinterpretCast(context, cond, srcType, AggregateType.Bool); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/HelperFunctions/HelperFunctionNames.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/HelperFunctions/HelperFunctionNames.cs new file mode 100644 index 0000000..d96b243 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/HelperFunctions/HelperFunctionNames.cs @@ -0,0 +1,10 @@ +namespace Ryujinx.Graphics.Shader.CodeGen.Glsl +{ + static class HelperFunctionNames + { + public static string MultiplyHighS32 = "Helper_MultiplyHighS32"; + public static string MultiplyHighU32 = "Helper_MultiplyHighU32"; + + public static string SwizzleAdd = "Helper_SwizzleAdd"; + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/HelperFunctions/MultiplyHighS32.glsl b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/HelperFunctions/MultiplyHighS32.glsl new file mode 100644 index 0000000..5e55aab --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/HelperFunctions/MultiplyHighS32.glsl @@ -0,0 +1,7 @@ +int Helper_MultiplyHighS32(int x, int y) +{ + int msb; + int lsb; + imulExtended(x, y, msb, lsb); + return msb; +} \ No newline at end of file diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/HelperFunctions/MultiplyHighU32.glsl b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/HelperFunctions/MultiplyHighU32.glsl new file mode 100644 index 0000000..f91a55e --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/HelperFunctions/MultiplyHighU32.glsl @@ -0,0 +1,7 @@ +uint Helper_MultiplyHighU32(uint x, uint y) +{ + uint msb; + uint lsb; + umulExtended(x, y, msb, lsb); + return msb; +} \ No newline at end of file diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/HelperFunctions/SwizzleAdd.glsl b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/HelperFunctions/SwizzleAdd.glsl new file mode 100644 index 0000000..d8306c0 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/HelperFunctions/SwizzleAdd.glsl @@ -0,0 +1,7 @@ +float Helper_SwizzleAdd(float x, float y, int mask) +{ + vec4 xLut = vec4(1.0, -1.0, 1.0, 0.0); + vec4 yLut = vec4(1.0, 1.0, -1.0, 1.0); + int lutIdx = (mask >> (int($SUBGROUP_INVOCATION$ & 3u) * 2)) & 3; + return x * xLut[lutIdx] + y * yLut[lutIdx]; +} \ No newline at end of file diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGen.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGen.cs new file mode 100644 index 0000000..bc5684c --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGen.cs @@ -0,0 +1,208 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.StructuredIr; +using Ryujinx.Graphics.Shader.Translation; +using System; +using System.Text; + +using static Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions.InstGenBallot; +using static Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions.InstGenCall; +using static Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions.InstGenFSI; +using static Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions.InstGenHelper; +using static Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions.InstGenMemory; +using static Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions.InstGenPacking; +using static Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions.InstGenShuffle; +using static Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions.InstGenVector; +using static Ryujinx.Graphics.Shader.StructuredIr.InstructionInfo; + +namespace Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions +{ + static class InstGen + { + public static string GetExpression(CodeGenContext context, IAstNode node) + { + if (node is AstOperation operation) + { + return GetExpression(context, operation); + } + else if (node is AstOperand operand) + { + return context.OperandManager.GetExpression(context, operand); + } + + throw new ArgumentException($"Invalid node type \"{node?.GetType().Name ?? "null"}\"."); + } + + public static string Negate(CodeGenContext context, AstOperation operation, InstInfo info) + { + IAstNode src = operation.GetSource(0); + + AggregateType type = GetSrcVarType(operation.Inst, 0); + + string srcExpr = GetSourceExpr(context, src, type); + string zero; + + if (type == AggregateType.FP64) + { + zero = "0.0"; + } + else + { + NumberFormatter.TryFormat(0, type, out zero); + } + + // Starting in the 496.13 NVIDIA driver, there's an issue with assigning variables to negated expressions. + // (-expr) does not work, but (0.0 - expr) does. This should be removed once the issue is resolved. + + return $"{zero} - {Enclose(srcExpr, src, operation.Inst, info, false)}"; + } + + private static string GetExpression(CodeGenContext context, AstOperation operation) + { + Instruction inst = operation.Inst; + + InstInfo info = GetInstructionInfo(inst); + + if ((info.Type & InstType.Call) != 0) + { + bool atomic = (info.Type & InstType.Atomic) != 0; + + int arity = (int)(info.Type & InstType.ArityMask); + + StringBuilder builder = new(); + + if (atomic && (operation.StorageKind == StorageKind.StorageBuffer || operation.StorageKind == StorageKind.SharedMemory)) + { + builder.Append(GenerateLoadOrStore(context, operation, isStore: false)); + + AggregateType dstType = operation.Inst is Instruction.AtomicMaxS32 or Instruction.AtomicMinS32 + ? AggregateType.S32 + : AggregateType.U32; + + for (int argIndex = operation.SourcesCount - arity + 2; argIndex < operation.SourcesCount; argIndex++) + { + builder.Append($", {GetSourceExpr(context, operation.GetSource(argIndex), dstType)}"); + } + } + else + { + for (int argIndex = 0; argIndex < arity; argIndex++) + { + if (argIndex != 0) + { + builder.Append(", "); + } + + AggregateType dstType = GetSrcVarType(inst, argIndex); + + builder.Append(GetSourceExpr(context, operation.GetSource(argIndex), dstType)); + } + } + + return $"{info.OpName}({builder})"; + } + else if ((info.Type & InstType.Op) != 0) + { + string op = info.OpName; + + // Return may optionally have a return value (and in this case it is unary). + if (inst == Instruction.Return && operation.SourcesCount != 0) + { + return $"{op} {GetSourceExpr(context, operation.GetSource(0), context.CurrentFunction.ReturnType)}"; + } + + int arity = (int)(info.Type & InstType.ArityMask); + + string[] expr = new string[arity]; + + for (int index = 0; index < arity; index++) + { + IAstNode src = operation.GetSource(index); + + string srcExpr = GetSourceExpr(context, src, GetSrcVarType(inst, index)); + + bool isLhs = arity == 2 && index == 0; + + expr[index] = Enclose(srcExpr, src, inst, info, isLhs); + } + + switch (arity) + { + case 0: + return op; + + case 1: + return op + expr[0]; + + case 2: + return $"{expr[0]} {op} {expr[1]}"; + + case 3: + return $"{expr[0]} {op[0]} {expr[1]} {op[1]} {expr[2]}"; + } + } + else if ((info.Type & InstType.Special) != 0) + { + switch (inst & Instruction.Mask) + { + case Instruction.Ballot: + return Ballot(context, operation); + + case Instruction.Call: + return Call(context, operation); + + case Instruction.FSIBegin: + return FSIBegin(context); + + case Instruction.FSIEnd: + return FSIEnd(context); + + case Instruction.ImageLoad: + case Instruction.ImageStore: + case Instruction.ImageAtomic: + return ImageLoadOrStore(context, operation); + + case Instruction.Load: + return Load(context, operation); + + case Instruction.Lod: + return Lod(context, operation); + + case Instruction.Negate: + return Negate(context, operation, info); + + case Instruction.PackDouble2x32: + return PackDouble2x32(context, operation); + + case Instruction.PackHalf2x16: + return PackHalf2x16(context, operation); + + case Instruction.Shuffle: + return Shuffle(context, operation); + + case Instruction.Store: + return Store(context, operation); + + case Instruction.TextureSample: + return TextureSample(context, operation); + + case Instruction.TextureQuerySamples: + return TextureQuerySamples(context, operation); + + case Instruction.TextureQuerySize: + return TextureQuerySize(context, operation); + + case Instruction.UnpackDouble2x32: + return UnpackDouble2x32(context, operation); + + case Instruction.UnpackHalf2x16: + return UnpackHalf2x16(context, operation); + + case Instruction.VectorExtract: + return VectorExtract(context, operation); + } + } + + throw new InvalidOperationException($"Unexpected instruction type \"{info.Type}\"."); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenBallot.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenBallot.cs new file mode 100644 index 0000000..007551b --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenBallot.cs @@ -0,0 +1,28 @@ +using Ryujinx.Graphics.Shader.StructuredIr; +using Ryujinx.Graphics.Shader.Translation; + +using static Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions.InstGenHelper; +using static Ryujinx.Graphics.Shader.StructuredIr.InstructionInfo; + +namespace Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions +{ + static class InstGenBallot + { + public static string Ballot(CodeGenContext context, AstOperation operation) + { + AggregateType dstType = GetSrcVarType(operation.Inst, 0); + + string arg = GetSourceExpr(context, operation.GetSource(0), dstType); + char component = "xyzw"[operation.Index]; + + if (context.HostCapabilities.SupportsShaderBallot) + { + return $"unpackUint2x32(ballotARB({arg})).{component}"; + } + else + { + return $"subgroupBallot({arg}).{component}"; + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenCall.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenCall.cs new file mode 100644 index 0000000..30d72d3 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenCall.cs @@ -0,0 +1,29 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.StructuredIr; +using System.Diagnostics; + +using static Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions.InstGenHelper; + +namespace Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions +{ + static class InstGenCall + { + public static string Call(CodeGenContext context, AstOperation operation) + { + AstOperand funcId = (AstOperand)operation.GetSource(0); + + Debug.Assert(funcId.Type == OperandType.Constant); + + StructuredFunction function = context.GetFunction(funcId.Value); + + string[] args = new string[operation.SourcesCount - 1]; + + for (int i = 0; i < args.Length; i++) + { + args[i] = GetSourceExpr(context, operation.GetSource(i + 1), function.GetArgumentType(i)); + } + + return $"{function.Name}({string.Join(", ", args)})"; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenFSI.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenFSI.cs new file mode 100644 index 0000000..5cf5284 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenFSI.cs @@ -0,0 +1,29 @@ +namespace Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions +{ + static class InstGenFSI + { + public static string FSIBegin(CodeGenContext context) + { + if (context.HostCapabilities.SupportsFragmentShaderInterlock) + { + return "beginInvocationInterlockARB()"; + } + else if (context.HostCapabilities.SupportsFragmentShaderOrderingIntel) + { + return "beginFragmentShaderOrderingINTEL()"; + } + + return null; + } + + public static string FSIEnd(CodeGenContext context) + { + if (context.HostCapabilities.SupportsFragmentShaderInterlock) + { + return "endInvocationInterlockARB()"; + } + + return null; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenHelper.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenHelper.cs new file mode 100644 index 0000000..2d18e9c --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenHelper.cs @@ -0,0 +1,222 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.StructuredIr; +using Ryujinx.Graphics.Shader.Translation; + +using static Ryujinx.Graphics.Shader.CodeGen.Glsl.TypeConversion; + +namespace Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions +{ + static class InstGenHelper + { + private static readonly InstInfo[] _infoTable; + + static InstGenHelper() + { + _infoTable = new InstInfo[(int)Instruction.Count]; + +#pragma warning disable IDE0055 // Disable formatting + Add(Instruction.AtomicAdd, InstType.AtomicBinary, "atomicAdd"); + Add(Instruction.AtomicAnd, InstType.AtomicBinary, "atomicAnd"); + Add(Instruction.AtomicCompareAndSwap, InstType.AtomicTernary, "atomicCompSwap"); + Add(Instruction.AtomicMaxU32, InstType.AtomicBinary, "atomicMax"); + Add(Instruction.AtomicMinU32, InstType.AtomicBinary, "atomicMin"); + Add(Instruction.AtomicOr, InstType.AtomicBinary, "atomicOr"); + Add(Instruction.AtomicSwap, InstType.AtomicBinary, "atomicExchange"); + Add(Instruction.AtomicXor, InstType.AtomicBinary, "atomicXor"); + Add(Instruction.Absolute, InstType.CallUnary, "abs"); + Add(Instruction.Add, InstType.OpBinaryCom, "+", 2); + Add(Instruction.Ballot, InstType.Special); + Add(Instruction.Barrier, InstType.CallNullary, "barrier"); + Add(Instruction.BitCount, InstType.CallUnary, "bitCount"); + Add(Instruction.BitfieldExtractS32, InstType.CallTernary, "bitfieldExtract"); + Add(Instruction.BitfieldExtractU32, InstType.CallTernary, "bitfieldExtract"); + Add(Instruction.BitfieldInsert, InstType.CallQuaternary, "bitfieldInsert"); + Add(Instruction.BitfieldReverse, InstType.CallUnary, "bitfieldReverse"); + Add(Instruction.BitwiseAnd, InstType.OpBinaryCom, "&", 6); + Add(Instruction.BitwiseExclusiveOr, InstType.OpBinaryCom, "^", 7); + Add(Instruction.BitwiseNot, InstType.OpUnary, "~", 0); + Add(Instruction.BitwiseOr, InstType.OpBinaryCom, "|", 8); + Add(Instruction.Call, InstType.Special); + Add(Instruction.Ceiling, InstType.CallUnary, "ceil"); + Add(Instruction.Clamp, InstType.CallTernary, "clamp"); + Add(Instruction.ClampU32, InstType.CallTernary, "clamp"); + Add(Instruction.CompareEqual, InstType.OpBinaryCom, "==", 5); + Add(Instruction.CompareGreater, InstType.OpBinary, ">", 4); + Add(Instruction.CompareGreaterOrEqual, InstType.OpBinary, ">=", 4); + Add(Instruction.CompareGreaterOrEqualU32, InstType.OpBinary, ">=", 4); + Add(Instruction.CompareGreaterU32, InstType.OpBinary, ">", 4); + Add(Instruction.CompareLess, InstType.OpBinary, "<", 4); + Add(Instruction.CompareLessOrEqual, InstType.OpBinary, "<=", 4); + Add(Instruction.CompareLessOrEqualU32, InstType.OpBinary, "<=", 4); + Add(Instruction.CompareLessU32, InstType.OpBinary, "<", 4); + Add(Instruction.CompareNotEqual, InstType.OpBinaryCom, "!=", 5); + Add(Instruction.ConditionalSelect, InstType.OpTernary, "?:", 12); + Add(Instruction.ConvertFP32ToFP64, InstType.CallUnary, "double"); + Add(Instruction.ConvertFP64ToFP32, InstType.CallUnary, "float"); + Add(Instruction.ConvertFP32ToS32, InstType.CallUnary, "int"); + Add(Instruction.ConvertFP32ToU32, InstType.CallUnary, "uint"); + Add(Instruction.ConvertFP64ToS32, InstType.CallUnary, "int"); + Add(Instruction.ConvertFP64ToU32, InstType.CallUnary, "uint"); + Add(Instruction.ConvertS32ToFP32, InstType.CallUnary, "float"); + Add(Instruction.ConvertS32ToFP64, InstType.CallUnary, "double"); + Add(Instruction.ConvertU32ToFP32, InstType.CallUnary, "float"); + Add(Instruction.ConvertU32ToFP64, InstType.CallUnary, "double"); + Add(Instruction.Cosine, InstType.CallUnary, "cos"); + Add(Instruction.Ddx, InstType.CallUnary, "dFdx"); + Add(Instruction.Ddy, InstType.CallUnary, "dFdy"); + Add(Instruction.Discard, InstType.OpNullary, "discard"); + Add(Instruction.Divide, InstType.OpBinary, "/", 1); + Add(Instruction.EmitVertex, InstType.CallNullary, "EmitVertex"); + Add(Instruction.EndPrimitive, InstType.CallNullary, "EndPrimitive"); + Add(Instruction.ExponentB2, InstType.CallUnary, "exp2"); + Add(Instruction.FSIBegin, InstType.Special); + Add(Instruction.FSIEnd, InstType.Special); + Add(Instruction.FindLSB, InstType.CallUnary, "findLSB"); + Add(Instruction.FindMSBS32, InstType.CallUnary, "findMSB"); + Add(Instruction.FindMSBU32, InstType.CallUnary, "findMSB"); + Add(Instruction.Floor, InstType.CallUnary, "floor"); + Add(Instruction.FusedMultiplyAdd, InstType.CallTernary, "fma"); + Add(Instruction.GroupMemoryBarrier, InstType.CallNullary, "groupMemoryBarrier"); + Add(Instruction.ImageLoad, InstType.Special); + Add(Instruction.ImageStore, InstType.Special); + Add(Instruction.ImageAtomic, InstType.Special); + Add(Instruction.IsNan, InstType.CallUnary, "isnan"); + Add(Instruction.Load, InstType.Special); + Add(Instruction.Lod, InstType.Special); + Add(Instruction.LogarithmB2, InstType.CallUnary, "log2"); + Add(Instruction.LogicalAnd, InstType.OpBinaryCom, "&&", 9); + Add(Instruction.LogicalExclusiveOr, InstType.OpBinaryCom, "^^", 10); + Add(Instruction.LogicalNot, InstType.OpUnary, "!", 0); + Add(Instruction.LogicalOr, InstType.OpBinaryCom, "||", 11); + Add(Instruction.LoopBreak, InstType.OpNullary, "break"); + Add(Instruction.LoopContinue, InstType.OpNullary, "continue"); + Add(Instruction.PackDouble2x32, InstType.Special); + Add(Instruction.PackHalf2x16, InstType.Special); + Add(Instruction.Maximum, InstType.CallBinary, "max"); + Add(Instruction.MaximumU32, InstType.CallBinary, "max"); + Add(Instruction.MemoryBarrier, InstType.CallNullary, "memoryBarrier"); + Add(Instruction.Minimum, InstType.CallBinary, "min"); + Add(Instruction.MinimumU32, InstType.CallBinary, "min"); + Add(Instruction.Modulo, InstType.CallBinary, "mod"); + Add(Instruction.Multiply, InstType.OpBinaryCom, "*", 1); + Add(Instruction.MultiplyHighS32, InstType.CallBinary, HelperFunctionNames.MultiplyHighS32); + Add(Instruction.MultiplyHighU32, InstType.CallBinary, HelperFunctionNames.MultiplyHighU32); + Add(Instruction.Negate, InstType.Special); + Add(Instruction.ReciprocalSquareRoot, InstType.CallUnary, "inversesqrt"); + Add(Instruction.Return, InstType.OpNullary, "return"); + Add(Instruction.Round, InstType.CallUnary, "roundEven"); + Add(Instruction.ShiftLeft, InstType.OpBinary, "<<", 3); + Add(Instruction.ShiftRightS32, InstType.OpBinary, ">>", 3); + Add(Instruction.ShiftRightU32, InstType.OpBinary, ">>", 3); + Add(Instruction.Shuffle, InstType.Special); + Add(Instruction.ShuffleDown, InstType.CallBinary, "subgroupShuffleDown"); + Add(Instruction.ShuffleUp, InstType.CallBinary, "subgroupShuffleUp"); + Add(Instruction.ShuffleXor, InstType.CallBinary, "subgroupShuffleXor"); + Add(Instruction.Sine, InstType.CallUnary, "sin"); + Add(Instruction.SquareRoot, InstType.CallUnary, "sqrt"); + Add(Instruction.Store, InstType.Special); + Add(Instruction.Subtract, InstType.OpBinary, "-", 2); + Add(Instruction.SwizzleAdd, InstType.CallTernary, HelperFunctionNames.SwizzleAdd); + Add(Instruction.TextureSample, InstType.Special); + Add(Instruction.TextureQuerySamples, InstType.Special); + Add(Instruction.TextureQuerySize, InstType.Special); + Add(Instruction.Truncate, InstType.CallUnary, "trunc"); + Add(Instruction.UnpackDouble2x32, InstType.Special); + Add(Instruction.UnpackHalf2x16, InstType.Special); + Add(Instruction.VectorExtract, InstType.Special); + Add(Instruction.VoteAll, InstType.CallUnary, "allInvocationsARB"); + Add(Instruction.VoteAllEqual, InstType.CallUnary, "allInvocationsEqualARB"); + Add(Instruction.VoteAny, InstType.CallUnary, "anyInvocationARB"); +#pragma warning restore IDE0055 + } + + private static void Add(Instruction inst, InstType flags, string opName = null, int precedence = 0) + { + _infoTable[(int)inst] = new InstInfo(flags, opName, precedence); + } + + public static InstInfo GetInstructionInfo(Instruction inst) + { + return _infoTable[(int)(inst & Instruction.Mask)]; + } + + public static string GetSourceExpr(CodeGenContext context, IAstNode node, AggregateType dstType) + { + return ReinterpretCast(context, node, OperandManager.GetNodeDestType(context, node), dstType); + } + + public static string Enclose(string expr, IAstNode node, Instruction pInst, bool isLhs) + { + InstInfo pInfo = GetInstructionInfo(pInst); + + return Enclose(expr, node, pInst, pInfo, isLhs); + } + + public static string Enclose(string expr, IAstNode node, Instruction pInst, InstInfo pInfo, bool isLhs = false) + { + if (NeedsParenthesis(node, pInst, pInfo, isLhs)) + { + expr = "(" + expr + ")"; + } + + return expr; + } + + public static bool NeedsParenthesis(IAstNode node, Instruction pInst, InstInfo pInfo, bool isLhs) + { + // If the node isn't a operation, then it can only be a operand, + // and those never needs to be surrounded in parenthesis. + if (node is not AstOperation operation) + { + // This is sort of a special case, if this is a negative constant, + // and it is consumed by a unary operation, we need to put on the parenthesis, + // as in GLSL a sequence like --2 or ~-1 is not valid. + if (IsNegativeConst(node) && pInfo.Type == InstType.OpUnary) + { + return true; + } + + return false; + } + + if ((pInfo.Type & (InstType.Call | InstType.Special)) != 0) + { + return false; + } + + InstInfo info = _infoTable[(int)(operation.Inst & Instruction.Mask)]; + + if ((info.Type & (InstType.Call | InstType.Special)) != 0) + { + return false; + } + + if (info.Precedence < pInfo.Precedence) + { + return false; + } + + if (info.Precedence == pInfo.Precedence && isLhs) + { + return false; + } + + if (pInst == operation.Inst && info.Type == InstType.OpBinaryCom) + { + return false; + } + + return true; + } + + private static bool IsNegativeConst(IAstNode node) + { + if (node is not AstOperand operand) + { + return false; + } + + return operand.Type == OperandType.Constant && operand.Value < 0; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenMemory.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenMemory.cs new file mode 100644 index 0000000..9b7d081 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenMemory.cs @@ -0,0 +1,701 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.StructuredIr; +using Ryujinx.Graphics.Shader.Translation; +using System; +using System.Text; +using static Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions.InstGenHelper; +using static Ryujinx.Graphics.Shader.StructuredIr.InstructionInfo; + +namespace Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions +{ + static class InstGenMemory + { + public static string ImageLoadOrStore(CodeGenContext context, AstOperation operation) + { + AstTextureOperation texOp = (AstTextureOperation)operation; + + bool isArray = (texOp.Type & SamplerType.Array) != 0; + + StringBuilder texCallBuilder = new(); + + if (texOp.Inst == Instruction.ImageAtomic) + { + texCallBuilder.Append((texOp.Flags & TextureFlags.AtomicMask) switch + { +#pragma warning disable IDE0055 // Disable formatting + TextureFlags.Add => "imageAtomicAdd", + TextureFlags.Minimum => "imageAtomicMin", + TextureFlags.Maximum => "imageAtomicMax", + TextureFlags.Increment => "imageAtomicAdd", // TODO: Clamp value. + TextureFlags.Decrement => "imageAtomicAdd", // TODO: Clamp value. + TextureFlags.BitwiseAnd => "imageAtomicAnd", + TextureFlags.BitwiseOr => "imageAtomicOr", + TextureFlags.BitwiseXor => "imageAtomicXor", + TextureFlags.Swap => "imageAtomicExchange", + TextureFlags.CAS => "imageAtomicCompSwap", + _ => "imageAtomicAdd", +#pragma warning restore IDE0055 + }); + } + else + { + texCallBuilder.Append(texOp.Inst == Instruction.ImageLoad ? "imageLoad" : "imageStore"); + } + + int srcIndex = 0; + + string Src(AggregateType type) + { + return GetSourceExpr(context, texOp.GetSource(srcIndex++), type); + } + + string imageName = GetImageName(context, texOp, ref srcIndex); + + texCallBuilder.Append('('); + texCallBuilder.Append(imageName); + + int coordsCount = texOp.Type.Dimensions; + + int pCount = coordsCount + (isArray ? 1 : 0); + + void Append(string str) + { + texCallBuilder.Append(", "); + texCallBuilder.Append(str); + } + + if (pCount > 1) + { + string[] elems = new string[pCount]; + + for (int index = 0; index < pCount; index++) + { + elems[index] = Src(AggregateType.S32); + } + + Append($"ivec{pCount}({string.Join(", ", elems)})"); + } + else + { + Append(Src(AggregateType.S32)); + } + + if (texOp.Inst == Instruction.ImageStore) + { + AggregateType type = texOp.Format.GetComponentType(); + + string[] cElems = new string[4]; + + for (int index = 0; index < 4; index++) + { + if (srcIndex < texOp.SourcesCount) + { + cElems[index] = Src(type); + } + else + { + cElems[index] = type switch + { + AggregateType.S32 => NumberFormatter.FormatInt(0), + AggregateType.U32 => NumberFormatter.FormatUint(0), + _ => NumberFormatter.FormatFloat(0), + }; + } + } + + string prefix = type switch + { + AggregateType.S32 => "i", + AggregateType.U32 => "u", + _ => string.Empty, + }; + + Append($"{prefix}vec4({string.Join(", ", cElems)})"); + } + + if (texOp.Inst == Instruction.ImageAtomic) + { + AggregateType type = texOp.Format.GetComponentType(); + + if ((texOp.Flags & TextureFlags.AtomicMask) == TextureFlags.CAS) + { + Append(Src(type)); // Compare value. + } + + string value = (texOp.Flags & TextureFlags.AtomicMask) switch + { + TextureFlags.Increment => NumberFormatter.FormatInt(1, type), // TODO: Clamp value + TextureFlags.Decrement => NumberFormatter.FormatInt(-1, type), // TODO: Clamp value + _ => Src(type), + }; + + Append(value); + + texCallBuilder.Append(')'); + + if (type != AggregateType.S32) + { + texCallBuilder + .Insert(0, "int(") + .Append(')'); + } + } + else + { + texCallBuilder.Append(')'); + + if (texOp.Inst == Instruction.ImageLoad) + { + texCallBuilder.Append(GetMaskMultiDest(texOp.Index)); + } + } + + return texCallBuilder.ToString(); + } + + public static string Load(CodeGenContext context, AstOperation operation) + { + return GenerateLoadOrStore(context, operation, isStore: false); + } + + public static string Lod(CodeGenContext context, AstOperation operation) + { + AstTextureOperation texOp = (AstTextureOperation)operation; + + int coordsCount = texOp.Type.Dimensions; + int coordsIndex = 0; + + string samplerName = GetSamplerName(context, texOp, ref coordsIndex); + + string coordsExpr; + + if (coordsCount > 1) + { + string[] elems = new string[coordsCount]; + + for (int index = 0; index < coordsCount; index++) + { + elems[index] = GetSourceExpr(context, texOp.GetSource(coordsIndex + index), AggregateType.FP32); + } + + coordsExpr = "vec" + coordsCount + "(" + string.Join(", ", elems) + ")"; + } + else + { + coordsExpr = GetSourceExpr(context, texOp.GetSource(coordsIndex), AggregateType.FP32); + } + + return $"textureQueryLod({samplerName}, {coordsExpr}){GetMask(texOp.Index)}"; + } + + public static string Store(CodeGenContext context, AstOperation operation) + { + return GenerateLoadOrStore(context, operation, isStore: true); + } + + public static string TextureSample(CodeGenContext context, AstOperation operation) + { + AstTextureOperation texOp = (AstTextureOperation)operation; + + bool isGather = (texOp.Flags & TextureFlags.Gather) != 0; + bool hasDerivatives = (texOp.Flags & TextureFlags.Derivatives) != 0; + bool intCoords = (texOp.Flags & TextureFlags.IntCoords) != 0; + bool hasLodBias = (texOp.Flags & TextureFlags.LodBias) != 0; + bool hasLodLevel = (texOp.Flags & TextureFlags.LodLevel) != 0; + bool hasOffset = (texOp.Flags & TextureFlags.Offset) != 0; + bool hasOffsets = (texOp.Flags & TextureFlags.Offsets) != 0; + + bool isArray = (texOp.Type & SamplerType.Array) != 0; + bool isMultisample = (texOp.Type & SamplerType.Multisample) != 0; + bool isShadow = (texOp.Type & SamplerType.Shadow) != 0; + + SamplerType type = texOp.Type & SamplerType.Mask; + + bool is2D = type == SamplerType.Texture2D; + bool isCube = type == SamplerType.TextureCube; + + // 2D Array and Cube shadow samplers with LOD level or bias requires an extension. + // If the extension is not supported, just remove the LOD parameter. + if (isArray && isShadow && (is2D || isCube) && !context.HostCapabilities.SupportsTextureShadowLod) + { + hasLodBias = false; + hasLodLevel = false; + } + + // Cube shadow samplers with LOD level requires an extension. + // If the extension is not supported, just remove the LOD level parameter. + if (isShadow && isCube && !context.HostCapabilities.SupportsTextureShadowLod) + { + hasLodLevel = false; + } + + string texCall = intCoords ? "texelFetch" : "texture"; + + if (isGather) + { + texCall += "Gather"; + } + else if (hasDerivatives) + { + texCall += "Grad"; + } + else if (hasLodLevel && !intCoords) + { + texCall += "Lod"; + } + + if (hasOffset) + { + texCall += "Offset"; + } + else if (hasOffsets) + { + texCall += "Offsets"; + } + + int srcIndex = 0; + + string Src(AggregateType type) + { + return GetSourceExpr(context, texOp.GetSource(srcIndex++), type); + } + + string samplerName = GetSamplerName(context, texOp, ref srcIndex); + + texCall += "(" + samplerName; + + int coordsCount = texOp.Type.Dimensions; + + int pCount = coordsCount; + + int arrayIndexElem = -1; + + if (isArray) + { + arrayIndexElem = pCount++; + } + + // The sampler 1D shadow overload expects a + // dummy value on the middle of the vector, who knows why... + bool hasDummy1DShadowElem = texOp.Type == (SamplerType.Texture1D | SamplerType.Shadow); + + if (hasDummy1DShadowElem) + { + pCount++; + } + + if (isShadow && !isGather) + { + pCount++; + } + + // On textureGather*, the comparison value is + // always specified as an extra argument. + bool hasExtraCompareArg = isShadow && isGather; + + if (pCount == 5) + { + pCount = 4; + + hasExtraCompareArg = true; + } + + void Append(string str) + { + texCall += ", " + str; + } + + AggregateType coordType = intCoords ? AggregateType.S32 : AggregateType.FP32; + + string AssemblePVector(int count) + { + if (count > 1) + { + string[] elems = new string[count]; + + for (int index = 0; index < count; index++) + { + if (arrayIndexElem == index) + { + elems[index] = Src(AggregateType.S32); + + if (!intCoords) + { + elems[index] = "float(" + elems[index] + ")"; + } + } + else if (index == 1 && hasDummy1DShadowElem) + { + elems[index] = NumberFormatter.FormatFloat(0); + } + else + { + elems[index] = Src(coordType); + } + } + + string prefix = intCoords ? "i" : string.Empty; + + return prefix + "vec" + count + "(" + string.Join(", ", elems) + ")"; + } + else + { + return Src(coordType); + } + } + + Append(AssemblePVector(pCount)); + + string AssembleDerivativesVector(int count) + { + if (count > 1) + { + string[] elems = new string[count]; + + for (int index = 0; index < count; index++) + { + elems[index] = Src(AggregateType.FP32); + } + + return "vec" + count + "(" + string.Join(", ", elems) + ")"; + } + else + { + return Src(AggregateType.FP32); + } + } + + if (hasExtraCompareArg) + { + Append(Src(AggregateType.FP32)); + } + + if (hasDerivatives) + { + Append(AssembleDerivativesVector(coordsCount)); // dPdx + Append(AssembleDerivativesVector(coordsCount)); // dPdy + } + + if (isMultisample) + { + Append(Src(AggregateType.S32)); + } + else if (hasLodLevel) + { + Append(Src(coordType)); + } + + string AssembleOffsetVector(int count) + { + if (count > 1) + { + string[] elems = new string[count]; + + for (int index = 0; index < count; index++) + { + elems[index] = Src(AggregateType.S32); + } + + return "ivec" + count + "(" + string.Join(", ", elems) + ")"; + } + else + { + return Src(AggregateType.S32); + } + } + + if (hasOffset) + { + Append(AssembleOffsetVector(coordsCount)); + } + else if (hasOffsets) + { + texCall += $", ivec{coordsCount}[4]("; + + texCall += AssembleOffsetVector(coordsCount) + ", "; + texCall += AssembleOffsetVector(coordsCount) + ", "; + texCall += AssembleOffsetVector(coordsCount) + ", "; + texCall += AssembleOffsetVector(coordsCount) + ")"; + } + + if (hasLodBias) + { + Append(Src(AggregateType.FP32)); + } + + // textureGather* optional extra component index, + // not needed for shadow samplers. + if (isGather && !isShadow) + { + Append(Src(AggregateType.S32)); + } + + bool colorIsVector = isGather || !isShadow; + + texCall += ")" + (colorIsVector ? GetMaskMultiDest(texOp.Index) : string.Empty); + + return texCall; + } + + public static string TextureQuerySamples(CodeGenContext context, AstOperation operation) + { + AstTextureOperation texOp = (AstTextureOperation)operation; + + int srcIndex = 0; + + string samplerName = GetSamplerName(context, texOp, ref srcIndex); + + return $"textureSamples({samplerName})"; + } + + public static string TextureQuerySize(CodeGenContext context, AstOperation operation) + { + AstTextureOperation texOp = (AstTextureOperation)operation; + + int srcIndex = 0; + + string samplerName = GetSamplerName(context, texOp, ref srcIndex); + + if (texOp.Index == 3) + { + return $"textureQueryLevels({samplerName})"; + } + else + { + context.Properties.Textures.TryGetValue(texOp.GetTextureSetAndBinding(), out TextureDefinition definition); + bool hasLod = !definition.Type.HasFlag(SamplerType.Multisample) && (definition.Type & SamplerType.Mask) != SamplerType.TextureBuffer; + string texCall; + + if (hasLod) + { + IAstNode lod = operation.GetSource(srcIndex); + string lodExpr = GetSourceExpr(context, lod, GetSrcVarType(operation.Inst, srcIndex)); + + texCall = $"textureSize({samplerName}, {lodExpr}){GetMask(texOp.Index)}"; + } + else + { + texCall = $"textureSize({samplerName}){GetMask(texOp.Index)}"; + } + + return texCall; + } + } + + public static string GenerateLoadOrStore(CodeGenContext context, AstOperation operation, bool isStore) + { + StorageKind storageKind = operation.StorageKind; + + string varName; + AggregateType varType; + int srcIndex = 0; + bool isStoreOrAtomic = operation.Inst == Instruction.Store || operation.Inst.IsAtomic(); + int inputsCount = isStoreOrAtomic ? operation.SourcesCount - 1 : operation.SourcesCount; + + if (operation.Inst == Instruction.AtomicCompareAndSwap) + { + inputsCount--; + } + + switch (storageKind) + { + case StorageKind.ConstantBuffer: + case StorageKind.StorageBuffer: + if (operation.GetSource(srcIndex++) is not AstOperand bindingIndex || bindingIndex.Type != OperandType.Constant) + { + throw new InvalidOperationException($"First input of {operation.Inst} with {storageKind} storage must be a constant operand."); + } + + int binding = bindingIndex.Value; + BufferDefinition buffer = storageKind == StorageKind.ConstantBuffer + ? context.Properties.ConstantBuffers[binding] + : context.Properties.StorageBuffers[binding]; + + if (operation.GetSource(srcIndex++) is not AstOperand fieldIndex || fieldIndex.Type != OperandType.Constant) + { + throw new InvalidOperationException($"Second input of {operation.Inst} with {storageKind} storage must be a constant operand."); + } + + StructureField field = buffer.Type.Fields[fieldIndex.Value]; + varName = $"{buffer.Name}.{field.Name}"; + varType = field.Type; + break; + + case StorageKind.LocalMemory: + case StorageKind.SharedMemory: + if (operation.GetSource(srcIndex++) is not AstOperand { Type: OperandType.Constant } bindingId) + { + throw new InvalidOperationException($"First input of {operation.Inst} with {storageKind} storage must be a constant operand."); + } + + MemoryDefinition memory = storageKind == StorageKind.LocalMemory + ? context.Properties.LocalMemories[bindingId.Value] + : context.Properties.SharedMemories[bindingId.Value]; + + varName = memory.Name; + varType = memory.Type; + break; + + case StorageKind.Input: + case StorageKind.InputPerPatch: + case StorageKind.Output: + case StorageKind.OutputPerPatch: + if (operation.GetSource(srcIndex++) is not AstOperand varId || varId.Type != OperandType.Constant) + { + throw new InvalidOperationException($"First input of {operation.Inst} with {storageKind} storage must be a constant operand."); + } + + IoVariable ioVariable = (IoVariable)varId.Value; + bool isOutput = storageKind.IsOutput(); + bool isPerPatch = storageKind.IsPerPatch(); + int location = -1; + int component = 0; + + if (context.Definitions.HasPerLocationInputOrOutput(ioVariable, isOutput)) + { + if (operation.GetSource(srcIndex++) is not AstOperand vecIndex || vecIndex.Type != OperandType.Constant) + { + throw new InvalidOperationException($"Second input of {operation.Inst} with {storageKind} storage must be a constant operand."); + } + + location = vecIndex.Value; + + if (operation.SourcesCount > srcIndex && + operation.GetSource(srcIndex) is AstOperand elemIndex && + elemIndex.Type == OperandType.Constant && + context.Definitions.HasPerLocationInputOrOutputComponent(ioVariable, location, elemIndex.Value, isOutput)) + { + component = elemIndex.Value; + srcIndex++; + } + } + + (varName, varType) = IoMap.GetGlslVariable( + context.Definitions, + context.HostCapabilities, + ioVariable, + location, + component, + isOutput, + isPerPatch); + + if (IoMap.IsPerVertexBuiltIn(context.Definitions.Stage, ioVariable, isOutput)) + { + // Since those exist both as input and output on geometry and tessellation shaders, + // we need the gl_in and gl_out prefixes to disambiguate. + + if (storageKind == StorageKind.Input) + { + string expr = GetSourceExpr(context, operation.GetSource(srcIndex++), AggregateType.S32); + varName = $"gl_in[{expr}].{varName}"; + } + else if (storageKind == StorageKind.Output) + { + string expr = GetSourceExpr(context, operation.GetSource(srcIndex++), AggregateType.S32); + varName = $"gl_out[{expr}].{varName}"; + } + } + + break; + + default: + throw new InvalidOperationException($"Invalid storage kind {storageKind}."); + } + + int firstSrcIndex = srcIndex; + + for (; srcIndex < inputsCount; srcIndex++) + { + IAstNode src = operation.GetSource(srcIndex); + + if ((varType & AggregateType.ElementCountMask) != 0 && + srcIndex == inputsCount - 1 && + src is AstOperand elementIndex && + elementIndex.Type == OperandType.Constant) + { + varName += "." + "xyzw"[elementIndex.Value & 3]; + } + else if (srcIndex == firstSrcIndex && context.Definitions.Stage == ShaderStage.TessellationControl && storageKind == StorageKind.Output) + { + // GLSL requires that for tessellation control shader outputs, + // that the index expression must be *exactly* "gl_InvocationID", + // otherwise the compilation fails. + // TODO: Get rid of this and use expression propagation to make sure we generate the correct code from IR. + varName += "[gl_InvocationID]"; + } + else + { + varName += $"[{GetSourceExpr(context, src, AggregateType.S32)}]"; + } + } + + if (isStore) + { + varType &= AggregateType.ElementTypeMask; + varName = $"{varName} = {GetSourceExpr(context, operation.GetSource(srcIndex), varType)}"; + } + + return varName; + } + + private static string GetSamplerName(CodeGenContext context, AstTextureOperation texOp, ref int srcIndex) + { + TextureDefinition textureDefinition = context.Properties.Textures[texOp.GetTextureSetAndBinding()]; + string name = textureDefinition.Name; + + if (textureDefinition.ArrayLength != 1) + { + name = $"{name}[{GetSourceExpr(context, texOp.GetSource(srcIndex++), AggregateType.S32)}]"; + } + + if (texOp.IsSeparate) + { + TextureDefinition samplerDefinition = context.Properties.Textures[texOp.GetSamplerSetAndBinding()]; + string samplerName = samplerDefinition.Name; + + if (samplerDefinition.ArrayLength != 1) + { + samplerName = $"{samplerName}[{GetSourceExpr(context, texOp.GetSource(srcIndex++), AggregateType.S32)}]"; + } + + name = $"{texOp.Type.GlslSamplerTypeName}({name}, {samplerName})"; + } + + return name; + } + + private static string GetImageName(CodeGenContext context, AstTextureOperation texOp, ref int srcIndex) + { + TextureDefinition definition = context.Properties.Images[texOp.GetTextureSetAndBinding()]; + string name = definition.Name; + + if (definition.ArrayLength != 1) + { + name = $"{name}[{GetSourceExpr(context, texOp.GetSource(srcIndex++), AggregateType.S32)}]"; + } + + return name; + } + + private static string GetMask(int index) + { + return $".{"rgba".AsSpan(index, 1)}"; + } + + private static string GetMaskMultiDest(int mask) + { + StringBuilder swizzleBuilder = new(); + swizzleBuilder.Append('.'); + + for (int i = 0; i < 4; i++) + { + if ((mask & (1 << i)) != 0) + { + swizzleBuilder.Append("xyzw"[i]); + } + } + + return swizzleBuilder.ToString(); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenPacking.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenPacking.cs new file mode 100644 index 0000000..d9c1c84 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenPacking.cs @@ -0,0 +1,56 @@ +using Ryujinx.Graphics.Shader.StructuredIr; +using System; + +using static Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions.InstGenHelper; +using static Ryujinx.Graphics.Shader.StructuredIr.InstructionInfo; + +namespace Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions +{ + static class InstGenPacking + { + public static string PackDouble2x32(CodeGenContext context, AstOperation operation) + { + IAstNode src0 = operation.GetSource(0); + IAstNode src1 = operation.GetSource(1); + + string src0Expr = GetSourceExpr(context, src0, GetSrcVarType(operation.Inst, 0)); + string src1Expr = GetSourceExpr(context, src1, GetSrcVarType(operation.Inst, 1)); + + return $"packDouble2x32(uvec2({src0Expr}, {src1Expr}))"; + } + + public static string PackHalf2x16(CodeGenContext context, AstOperation operation) + { + IAstNode src0 = operation.GetSource(0); + IAstNode src1 = operation.GetSource(1); + + string src0Expr = GetSourceExpr(context, src0, GetSrcVarType(operation.Inst, 0)); + string src1Expr = GetSourceExpr(context, src1, GetSrcVarType(operation.Inst, 1)); + + return $"packHalf2x16(vec2({src0Expr}, {src1Expr}))"; + } + + public static string UnpackDouble2x32(CodeGenContext context, AstOperation operation) + { + IAstNode src = operation.GetSource(0); + + string srcExpr = GetSourceExpr(context, src, GetSrcVarType(operation.Inst, 0)); + + return $"unpackDouble2x32({srcExpr}){GetMask(operation.Index)}"; + } + + public static string UnpackHalf2x16(CodeGenContext context, AstOperation operation) + { + IAstNode src = operation.GetSource(0); + + string srcExpr = GetSourceExpr(context, src, GetSrcVarType(operation.Inst, 0)); + + return $"unpackHalf2x16({srcExpr}){GetMask(operation.Index)}"; + } + + private static string GetMask(int index) + { + return $".{"xy".AsSpan(index, 1)}"; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenShuffle.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenShuffle.cs new file mode 100644 index 0000000..8646fd4 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenShuffle.cs @@ -0,0 +1,25 @@ +using Ryujinx.Graphics.Shader.StructuredIr; +using Ryujinx.Graphics.Shader.Translation; + +using static Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions.InstGenHelper; + +namespace Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions +{ + static class InstGenShuffle + { + public static string Shuffle(CodeGenContext context, AstOperation operation) + { + string value = GetSourceExpr(context, operation.GetSource(0), AggregateType.FP32); + string index = GetSourceExpr(context, operation.GetSource(1), AggregateType.U32); + + if (context.HostCapabilities.SupportsShaderBallot) + { + return $"readInvocationARB({value}, {index})"; + } + else + { + return $"subgroupShuffle({value}, {index})"; + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenVector.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenVector.cs new file mode 100644 index 0000000..9419a16 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstGenVector.cs @@ -0,0 +1,32 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.StructuredIr; + +using static Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions.InstGenHelper; +using static Ryujinx.Graphics.Shader.StructuredIr.InstructionInfo; + +namespace Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions +{ + static class InstGenVector + { + public static string VectorExtract(CodeGenContext context, AstOperation operation) + { + IAstNode vector = operation.GetSource(0); + IAstNode index = operation.GetSource(1); + + string vectorExpr = GetSourceExpr(context, vector, OperandManager.GetNodeDestType(context, vector)); + + if (index is AstOperand indexOperand && indexOperand.Type == OperandType.Constant) + { + char elem = "xyzw"[indexOperand.Value]; + + return $"{vectorExpr}.{elem}"; + } + else + { + string indexExpr = GetSourceExpr(context, index, GetSrcVarType(operation.Inst, 1)); + + return $"{vectorExpr}[{indexExpr}]"; + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstInfo.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstInfo.cs new file mode 100644 index 0000000..1d869b4 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstInfo.cs @@ -0,0 +1,18 @@ +namespace Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions +{ + readonly struct InstInfo + { + public InstType Type { get; } + + public string OpName { get; } + + public int Precedence { get; } + + public InstInfo(InstType type, string opName, int precedence) + { + Type = type; + OpName = opName; + Precedence = precedence; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstType.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstType.cs new file mode 100644 index 0000000..a7407cb --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/InstType.cs @@ -0,0 +1,33 @@ +using System; + +namespace Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions +{ + [Flags] + enum InstType + { + OpNullary = Op | 0, + OpUnary = Op | 1, + OpBinary = Op | 2, + OpBinaryCom = Op | 2 | Commutative, + OpTernary = Op | 3, + + CallNullary = Call | 0, + CallUnary = Call | 1, + CallBinary = Call | 2, + CallTernary = Call | 3, + CallQuaternary = Call | 4, + + // The atomic instructions have one extra operand, + // for the storage slot and offset pair. + AtomicBinary = Call | Atomic | 3, + AtomicTernary = Call | Atomic | 4, + + Commutative = 1 << 8, + Op = 1 << 9, + Call = 1 << 10, + Atomic = 1 << 11, + Special = 1 << 12, + + ArityMask = 0xff, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/IoMap.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/IoMap.cs new file mode 100644 index 0000000..ab8b5cb --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/Instructions/IoMap.cs @@ -0,0 +1,144 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using System.Globalization; + +namespace Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions +{ + static class IoMap + { + public static (string, AggregateType) GetGlslVariable( + ShaderDefinitions definitions, + HostCapabilities hostCapabilities, + IoVariable ioVariable, + int location, + int component, + bool isOutput, + bool isPerPatch) + { + return ioVariable switch + { + IoVariable.BackColorDiffuse => ("gl_BackColor", AggregateType.Vector4 | AggregateType.FP32), // Deprecated. + IoVariable.BackColorSpecular => ("gl_BackSecondaryColor", AggregateType.Vector4 | AggregateType.FP32), // Deprecated. + IoVariable.BaseInstance => ("gl_BaseInstanceARB", AggregateType.S32), + IoVariable.BaseVertex => ("gl_BaseVertexARB", AggregateType.S32), + IoVariable.ClipDistance => ("gl_ClipDistance", AggregateType.Array | AggregateType.FP32), + IoVariable.CtaId => ("gl_WorkGroupID", AggregateType.Vector3 | AggregateType.U32), + IoVariable.DrawIndex => ("gl_DrawIDARB", AggregateType.S32), + IoVariable.FogCoord => ("gl_FogFragCoord", AggregateType.FP32), // Deprecated. + IoVariable.FragmentCoord => ("gl_FragCoord", AggregateType.Vector4 | AggregateType.FP32), + IoVariable.FragmentOutputColor => GetFragmentOutputColorVariableName(definitions, location), + IoVariable.FragmentOutputDepth => ("gl_FragDepth", AggregateType.FP32), + IoVariable.FrontColorDiffuse => ("gl_FrontColor", AggregateType.Vector4 | AggregateType.FP32), // Deprecated. + IoVariable.FrontColorSpecular => ("gl_FrontSecondaryColor", AggregateType.Vector4 | AggregateType.FP32), // Deprecated. + IoVariable.FrontFacing => ("gl_FrontFacing", AggregateType.Bool), + IoVariable.GlobalId => ("gl_GlobalInvocationID", AggregateType.Vector3 | AggregateType.U32), + IoVariable.InstanceId => ("gl_InstanceID", AggregateType.S32), + IoVariable.InstanceIndex => ("gl_InstanceIndex", AggregateType.S32), + IoVariable.InvocationId => ("gl_InvocationID", AggregateType.S32), + IoVariable.Layer => ("gl_Layer", AggregateType.S32), + IoVariable.PatchVertices => ("gl_PatchVerticesIn", AggregateType.S32), + IoVariable.PointCoord => ("gl_PointCoord", AggregateType.Vector2 | AggregateType.FP32), + IoVariable.PointSize => ("gl_PointSize", AggregateType.FP32), + IoVariable.Position => ("gl_Position", AggregateType.Vector4 | AggregateType.FP32), + IoVariable.PrimitiveId => GetPrimitiveIdVariableName(definitions.Stage, isOutput), + IoVariable.SubgroupEqMask => GetSubgroupMaskVariableName(hostCapabilities.SupportsShaderBallot, "Eq"), + IoVariable.SubgroupGeMask => GetSubgroupMaskVariableName(hostCapabilities.SupportsShaderBallot, "Ge"), + IoVariable.SubgroupGtMask => GetSubgroupMaskVariableName(hostCapabilities.SupportsShaderBallot, "Gt"), + IoVariable.SubgroupLaneId => GetSubgroupInvocationIdVariableName(hostCapabilities.SupportsShaderBallot), + IoVariable.SubgroupLeMask => GetSubgroupMaskVariableName(hostCapabilities.SupportsShaderBallot, "Le"), + IoVariable.SubgroupLtMask => GetSubgroupMaskVariableName(hostCapabilities.SupportsShaderBallot, "Lt"), + IoVariable.TessellationCoord => ("gl_TessCoord", AggregateType.Vector3 | AggregateType.FP32), + IoVariable.TessellationLevelInner => ("gl_TessLevelInner", AggregateType.Array | AggregateType.FP32), + IoVariable.TessellationLevelOuter => ("gl_TessLevelOuter", AggregateType.Array | AggregateType.FP32), + IoVariable.TextureCoord => ("gl_TexCoord", AggregateType.Array | AggregateType.Vector4 | AggregateType.FP32), // Deprecated. + IoVariable.ThreadId => ("gl_LocalInvocationID", AggregateType.Vector3 | AggregateType.U32), + IoVariable.ThreadKill => ("gl_HelperInvocation", AggregateType.Bool), + IoVariable.UserDefined => GetUserDefinedVariableName(definitions, location, component, isOutput, isPerPatch), + IoVariable.VertexId => ("gl_VertexID", AggregateType.S32), + IoVariable.VertexIndex => ("gl_VertexIndex", AggregateType.S32), + IoVariable.ViewportIndex => ("gl_ViewportIndex", AggregateType.S32), + IoVariable.ViewportMask => ("gl_ViewportMask", AggregateType.Array | AggregateType.S32), + _ => (null, AggregateType.Invalid), + }; + } + + public static bool IsPerVertexBuiltIn(ShaderStage stage, IoVariable ioVariable, bool isOutput) + { + switch (ioVariable) + { + case IoVariable.Layer: + case IoVariable.ViewportIndex: + case IoVariable.PointSize: + case IoVariable.Position: + case IoVariable.ClipDistance: + case IoVariable.PointCoord: + case IoVariable.ViewportMask: + if (isOutput) + { + return stage == ShaderStage.TessellationControl; + } + else + { + return stage is ShaderStage.TessellationControl or + ShaderStage.TessellationEvaluation or + ShaderStage.Geometry; + } + } + + return false; + } + + private static (string, AggregateType) GetFragmentOutputColorVariableName(ShaderDefinitions definitions, int location) + { + if (location < 0) + { + return (DefaultNames.OAttributePrefix, definitions.GetFragmentOutputColorType(0)); + } + + string name = DefaultNames.OAttributePrefix + location.ToString(CultureInfo.InvariantCulture); + + return (name, definitions.GetFragmentOutputColorType(location)); + } + + private static (string, AggregateType) GetPrimitiveIdVariableName(ShaderStage stage, bool isOutput) + { + // The geometry stage has an additional gl_PrimitiveIDIn variable. + return (isOutput || stage != ShaderStage.Geometry ? "gl_PrimitiveID" : "gl_PrimitiveIDIn", AggregateType.S32); + } + + private static (string, AggregateType) GetSubgroupMaskVariableName(bool supportsShaderBallot, string cc) + { + return supportsShaderBallot + ? ($"unpackUint2x32(gl_SubGroup{cc}MaskARB)", AggregateType.Vector2 | AggregateType.U32) + : ($"gl_Subgroup{cc}Mask", AggregateType.Vector4 | AggregateType.U32); + } + + private static (string, AggregateType) GetSubgroupInvocationIdVariableName(bool supportsShaderBallot) + { + return supportsShaderBallot + ? ("gl_SubGroupInvocationARB", AggregateType.U32) + : ("gl_SubgroupInvocationID", AggregateType.U32); + } + + private static (string, AggregateType) GetUserDefinedVariableName(ShaderDefinitions definitions, int location, int component, bool isOutput, bool isPerPatch) + { + string name = isPerPatch + ? DefaultNames.PerPatchAttributePrefix + : (isOutput ? DefaultNames.OAttributePrefix : DefaultNames.IAttributePrefix); + + if (location < 0) + { + return (name, definitions.GetUserDefinedType(0, isOutput)); + } + + name += location.ToString(CultureInfo.InvariantCulture); + + if (definitions.HasPerLocationInputOrOutputComponent(IoVariable.UserDefined, location, component, isOutput)) + { + name += "_" + "xyzw"[component & 3]; + } + + return (name, definitions.GetUserDefinedType(location, isOutput)); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/NumberFormatter.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/NumberFormatter.cs new file mode 100644 index 0000000..afdde6f --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/NumberFormatter.cs @@ -0,0 +1,104 @@ +using Ryujinx.Graphics.Shader.Translation; +using System; +using System.Globalization; + +namespace Ryujinx.Graphics.Shader.CodeGen.Glsl +{ + static class NumberFormatter + { + private const int MaxDecimal = 256; + + public static bool TryFormat(int value, AggregateType dstType, out string formatted) + { + if (dstType == AggregateType.FP32) + { + return TryFormatFloat(BitConverter.Int32BitsToSingle(value), out formatted); + } + else if (dstType == AggregateType.S32) + { + formatted = FormatInt(value); + } + else if (dstType == AggregateType.U32) + { + formatted = FormatUint((uint)value); + } + else if (dstType == AggregateType.Bool) + { + formatted = value != 0 ? "true" : "false"; + } + else + { + throw new ArgumentException($"Invalid variable type \"{dstType}\"."); + } + + return true; + } + + public static string FormatFloat(float value) + { + if (!TryFormatFloat(value, out string formatted)) + { + throw new ArgumentException("Failed to convert float value to string."); + } + + return formatted; + } + + public static bool TryFormatFloat(float value, out string formatted) + { + if (float.IsNaN(value) || float.IsInfinity(value)) + { + formatted = null; + + return false; + } + + formatted = value.ToString("G9", CultureInfo.InvariantCulture); + + if (!(formatted.Contains('.') || + formatted.Contains('e') || + formatted.Contains('E'))) + { + formatted += ".0"; + } + + return true; + } + + public static string FormatInt(int value, AggregateType dstType) + { + if (dstType == AggregateType.S32) + { + return FormatInt(value); + } + else if (dstType == AggregateType.U32) + { + return FormatUint((uint)value); + } + else + { + throw new ArgumentException($"Invalid variable type \"{dstType}\"."); + } + } + + public static string FormatInt(int value) + { + if (value is <= MaxDecimal and >= (-MaxDecimal)) + { + return value.ToString(CultureInfo.InvariantCulture); + } + + return "0x" + value.ToString("X", CultureInfo.InvariantCulture); + } + + public static string FormatUint(uint value) + { + if (value is <= MaxDecimal and >= 0) + { + return value.ToString(CultureInfo.InvariantCulture) + "u"; + } + + return "0x" + value.ToString("X", CultureInfo.InvariantCulture) + "u"; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/OperandManager.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/OperandManager.cs new file mode 100644 index 0000000..d2c3ba1 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/OperandManager.cs @@ -0,0 +1,178 @@ +using Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.StructuredIr; +using Ryujinx.Graphics.Shader.Translation; +using System; +using System.Collections.Generic; +using System.Diagnostics; +using static Ryujinx.Graphics.Shader.StructuredIr.InstructionInfo; + +namespace Ryujinx.Graphics.Shader.CodeGen.Glsl +{ + class OperandManager + { + private readonly Dictionary _locals; + + public OperandManager() + { + _locals = new Dictionary(); + } + + public string DeclareLocal(AstOperand operand) + { + string name = $"{DefaultNames.LocalNamePrefix}_{_locals.Count}"; + + _locals.Add(operand, name); + + return name; + } + + public string GetExpression(CodeGenContext context, AstOperand operand) + { + return operand.Type switch + { + OperandType.Argument => GetArgumentName(operand.Value), + OperandType.Constant => NumberFormatter.FormatInt(operand.Value), + OperandType.LocalVariable => _locals[operand], + OperandType.Undefined => DefaultNames.UndefinedName, + _ => throw new ArgumentException($"Invalid operand type \"{operand.Type}\"."), + }; + } + + public static string GetArgumentName(int argIndex) + { + return $"{DefaultNames.ArgumentNamePrefix}{argIndex}"; + } + + public static AggregateType GetNodeDestType(CodeGenContext context, IAstNode node) + { + // TODO: Get rid of that function entirely and return the type from the operation generation + // functions directly, like SPIR-V does. + + if (node is AstOperation operation) + { + if (operation.Inst == Instruction.Load || operation.Inst.IsAtomic()) + { + switch (operation.StorageKind) + { + case StorageKind.ConstantBuffer: + case StorageKind.StorageBuffer: + if (operation.GetSource(0) is not AstOperand bindingIndex || bindingIndex.Type != OperandType.Constant) + { + throw new InvalidOperationException($"First input of {operation.Inst} with {operation.StorageKind} storage must be a constant operand."); + } + + if (operation.GetSource(1) is not AstOperand fieldIndex || fieldIndex.Type != OperandType.Constant) + { + throw new InvalidOperationException($"Second input of {operation.Inst} with {operation.StorageKind} storage must be a constant operand."); + } + + BufferDefinition buffer = operation.StorageKind == StorageKind.ConstantBuffer + ? context.Properties.ConstantBuffers[bindingIndex.Value] + : context.Properties.StorageBuffers[bindingIndex.Value]; + StructureField field = buffer.Type.Fields[fieldIndex.Value]; + + return field.Type & AggregateType.ElementTypeMask; + + case StorageKind.LocalMemory: + case StorageKind.SharedMemory: + if (operation.GetSource(0) is not AstOperand { Type: OperandType.Constant } bindingId) + { + throw new InvalidOperationException($"First input of {operation.Inst} with {operation.StorageKind} storage must be a constant operand."); + } + + MemoryDefinition memory = operation.StorageKind == StorageKind.LocalMemory + ? context.Properties.LocalMemories[bindingId.Value] + : context.Properties.SharedMemories[bindingId.Value]; + + return memory.Type & AggregateType.ElementTypeMask; + + case StorageKind.Input: + case StorageKind.InputPerPatch: + case StorageKind.Output: + case StorageKind.OutputPerPatch: + if (operation.GetSource(0) is not AstOperand varId || varId.Type != OperandType.Constant) + { + throw new InvalidOperationException($"First input of {operation.Inst} with {operation.StorageKind} storage must be a constant operand."); + } + + IoVariable ioVariable = (IoVariable)varId.Value; + bool isOutput = operation.StorageKind is StorageKind.Output or StorageKind.OutputPerPatch; + bool isPerPatch = operation.StorageKind is StorageKind.InputPerPatch or StorageKind.OutputPerPatch; + int location = 0; + int component = 0; + + if (context.Definitions.HasPerLocationInputOrOutput(ioVariable, isOutput)) + { + if (operation.GetSource(1) is not AstOperand vecIndex || vecIndex.Type != OperandType.Constant) + { + throw new InvalidOperationException($"Second input of {operation.Inst} with {operation.StorageKind} storage must be a constant operand."); + } + + location = vecIndex.Value; + + if (operation.SourcesCount > 2 && + operation.GetSource(2) is AstOperand elemIndex && + elemIndex.Type == OperandType.Constant && + context.Definitions.HasPerLocationInputOrOutputComponent(ioVariable, location, elemIndex.Value, isOutput)) + { + component = elemIndex.Value; + } + } + + (_, AggregateType varType) = IoMap.GetGlslVariable( + context.Definitions, + context.HostCapabilities, + ioVariable, + location, + component, + isOutput, + isPerPatch); + + return varType & AggregateType.ElementTypeMask; + } + } + else if (operation.Inst == Instruction.Call) + { + AstOperand funcId = (AstOperand)operation.GetSource(0); + + Debug.Assert(funcId.Type == OperandType.Constant); + + return context.GetFunction(funcId.Value).ReturnType; + } + else if (operation.Inst == Instruction.VectorExtract) + { + return GetNodeDestType(context, operation.GetSource(0)) & ~AggregateType.ElementCountMask; + } + else if (operation is AstTextureOperation texOp) + { + if (texOp.Inst.IsImage()) + { + return texOp.GetVectorType(texOp.Format.GetComponentType()); + } + else if (texOp.Inst == Instruction.TextureSample) + { + return texOp.GetVectorType(GetDestVarType(operation.Inst)); + } + } + + return GetDestVarType(operation.Inst); + } + else if (node is AstOperand operand) + { + if (operand.Type == OperandType.Argument) + { + int argIndex = operand.Value; + + return context.CurrentFunction.GetArgumentType(argIndex); + } + + return OperandInfo.GetVarType(operand); + } + else + { + throw new ArgumentException($"Invalid node type \"{node?.GetType().Name ?? "null"}\"."); + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/TypeConversion.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/TypeConversion.cs new file mode 100644 index 0000000..3708705 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Glsl/TypeConversion.cs @@ -0,0 +1,93 @@ +using Ryujinx.Graphics.Shader.CodeGen.Glsl.Instructions; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.StructuredIr; +using Ryujinx.Graphics.Shader.Translation; +using System; + +namespace Ryujinx.Graphics.Shader.CodeGen.Glsl +{ + static class TypeConversion + { + public static string ReinterpretCast( + CodeGenContext context, + IAstNode node, + AggregateType srcType, + AggregateType dstType) + { + if (node is AstOperand operand && operand.Type == OperandType.Constant) + { + if (NumberFormatter.TryFormat(operand.Value, dstType, out string formatted)) + { + return formatted; + } + } + + string expr = InstGen.GetExpression(context, node); + + return ReinterpretCast(expr, node, srcType, dstType); + } + + private static string ReinterpretCast(string expr, IAstNode node, AggregateType srcType, AggregateType dstType) + { + if (srcType == dstType) + { + return expr; + } + + if (srcType == AggregateType.FP32) + { + switch (dstType) + { + case AggregateType.Bool: + return $"(floatBitsToInt({expr}) != 0)"; + case AggregateType.S32: + return $"floatBitsToInt({expr})"; + case AggregateType.U32: + return $"floatBitsToUint({expr})"; + } + } + else if (dstType == AggregateType.FP32) + { + switch (srcType) + { + case AggregateType.Bool: + return $"intBitsToFloat({ReinterpretBoolToInt(expr, node, AggregateType.S32)})"; + case AggregateType.S32: + return $"intBitsToFloat({expr})"; + case AggregateType.U32: + return $"uintBitsToFloat({expr})"; + } + } + else if (srcType == AggregateType.Bool) + { + return ReinterpretBoolToInt(expr, node, dstType); + } + else if (dstType == AggregateType.Bool) + { + expr = InstGenHelper.Enclose(expr, node, Instruction.CompareNotEqual, isLhs: true); + + return $"({expr} != 0)"; + } + else if (dstType == AggregateType.S32) + { + return $"int({expr})"; + } + else if (dstType == AggregateType.U32) + { + return $"uint({expr})"; + } + + throw new ArgumentException($"Invalid reinterpret cast from \"{srcType}\" to \"{dstType}\"."); + } + + private static string ReinterpretBoolToInt(string expr, IAstNode node, AggregateType dstType) + { + string trueExpr = NumberFormatter.FormatInt(IrConsts.True, dstType); + string falseExpr = NumberFormatter.FormatInt(IrConsts.False, dstType); + + expr = InstGenHelper.Enclose(expr, node, Instruction.ConditionalSelect, isLhs: false); + + return $"({expr} ? {trueExpr} : {falseExpr})"; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/CodeGenContext.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/CodeGenContext.cs new file mode 100644 index 0000000..e911ba8 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/CodeGenContext.cs @@ -0,0 +1,375 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.StructuredIr; +using Ryujinx.Graphics.Shader.Translation; +using Spv.Generator; +using System; +using System.Collections.Generic; +using Instruction = Spv.Generator.Instruction; + +namespace Ryujinx.Graphics.Shader.CodeGen.Spirv +{ + using IrOperandType = IntermediateRepresentation.OperandType; + + partial class CodeGenContext : Module + { + private const uint SpirvVersionMajor = 1; + private const uint SpirvVersionMinor = 3; + private const uint SpirvVersionRevision = 0; + private const uint SpirvVersionPacked = (SpirvVersionMajor << 16) | (SpirvVersionMinor << 8) | SpirvVersionRevision; + + public StructuredProgramInfo Info { get; } + + public AttributeUsage AttributeUsage { get; } + public ShaderDefinitions Definitions { get; } + public ShaderProperties Properties { get; } + public HostCapabilities HostCapabilities { get; } + public ILogger Logger { get; } + public TargetApi TargetApi { get; } + + public Dictionary ConstantBuffers { get; } = new(); + public Dictionary StorageBuffers { get; } = new(); + + public Dictionary LocalMemories { get; } = new(); + public Dictionary SharedMemories { get; } = new(); + + public Dictionary SamplersTypes { get; } = new(); + public Dictionary Samplers { get; } = new(); + public Dictionary Images { get; } = new(); + + public Dictionary Inputs { get; } = new(); + public Dictionary Outputs { get; } = new(); + public Dictionary InputsPerPatch { get; } = new(); + public Dictionary OutputsPerPatch { get; } = new(); + + public StructuredFunction CurrentFunction { get; set; } + private readonly Dictionary _locals = new(); + private readonly Dictionary _funcArgs = new(); + private readonly Dictionary _functions = new(); + + private class BlockState + { + private int _entryCount; + private readonly List _labels = []; + + public Instruction GetNextLabel(CodeGenContext context) + { + return GetLabel(context, _entryCount); + } + + public Instruction GetNextLabelAutoIncrement(CodeGenContext context) + { + return GetLabel(context, _entryCount++); + } + + public Instruction GetLabel(CodeGenContext context, int index) + { + while (index >= _labels.Count) + { + _labels.Add(context.Label()); + } + + return _labels[index]; + } + } + + private readonly Dictionary _labels = new(); + + public Dictionary LoopTargets { get; set; } + + public AstBlock CurrentBlock { get; private set; } + + public SpirvDelegates Delegates { get; } + + public bool IsMainFunction { get; private set; } + public bool MayHaveReturned { get; set; } + public bool WasNonUniformAccessDeclared { get; set; } + + public CodeGenContext( + StructuredProgramInfo info, + CodeGenParameters parameters, + GeneratorPool instPool, + GeneratorPool integerPool) : base(SpirvVersionPacked, instPool, integerPool) + { + WasNonUniformAccessDeclared = false; + + Info = info; + AttributeUsage = parameters.AttributeUsage; + Definitions = parameters.Definitions; + Properties = parameters.Properties; + HostCapabilities = parameters.HostCapabilities; + Logger = parameters.Logger; + TargetApi = parameters.TargetApi; + + Delegates = new SpirvDelegates(this); + } + + public void StartFunction(bool isMainFunction) + { + IsMainFunction = isMainFunction; + MayHaveReturned = false; + _locals.Clear(); + _funcArgs.Clear(); + } + + public void EnterBlock(AstBlock block) + { + CurrentBlock = block; + AddLabel(GetBlockStateLazy(block).GetNextLabelAutoIncrement(this)); + } + + public Instruction GetFirstLabel(AstBlock block) + { + return GetBlockStateLazy(block).GetLabel(this, 0); + } + + public Instruction GetNextLabel(AstBlock block) + { + return GetBlockStateLazy(block).GetNextLabel(this); + } + + private BlockState GetBlockStateLazy(AstBlock block) + { + if (!_labels.TryGetValue(block, out BlockState blockState)) + { + blockState = new BlockState(); + + _labels.Add(block, blockState); + } + + return blockState; + } + + public Instruction NewBlock() + { + Instruction label = Label(); + Branch(label); + AddLabel(label); + return label; + } + + public Instruction[] GetMainInterface() + { + List mainInterface = []; + + mainInterface.AddRange(Inputs.Values); + mainInterface.AddRange(Outputs.Values); + mainInterface.AddRange(InputsPerPatch.Values); + mainInterface.AddRange(OutputsPerPatch.Values); + + return mainInterface.ToArray(); + } + + public void DeclareLocal(AstOperand local, Instruction spvLocal) + { + _locals.Add(local, spvLocal); + } + + public void DeclareArgument(int argIndex, Instruction spvLocal) + { + _funcArgs.Add(argIndex, spvLocal); + } + + public void DeclareFunction(int funcIndex, StructuredFunction function, Instruction spvFunc) + { + _functions.Add(funcIndex, (function, spvFunc)); + } + + public Instruction GetFP32(IAstNode node) + { + return Get(AggregateType.FP32, node); + } + + public Instruction GetFP64(IAstNode node) + { + return Get(AggregateType.FP64, node); + } + + public Instruction GetS32(IAstNode node) + { + return Get(AggregateType.S32, node); + } + + public Instruction GetU32(IAstNode node) + { + return Get(AggregateType.U32, node); + } + + public Instruction Get(AggregateType type, IAstNode node) + { + if (node is AstOperation operation) + { + OperationResult opResult = Instructions.Generate(this, operation); + return BitcastIfNeeded(type, opResult.Type, opResult.Value); + } + else if (node is AstOperand operand) + { + return operand.Type switch + { + IrOperandType.Argument => GetArgument(type, operand), + IrOperandType.Constant => GetConstant(type, operand), + IrOperandType.LocalVariable => GetLocal(type, operand), + IrOperandType.Undefined => GetUndefined(type), + _ => throw new ArgumentException($"Invalid operand type \"{operand.Type}\"."), + }; + } + + throw new NotImplementedException(node.GetType().Name); + } + + public Instruction GetWithType(IAstNode node, out AggregateType type) + { + if (node is AstOperation operation) + { + OperationResult opResult = Instructions.Generate(this, operation); + type = opResult.Type; + return opResult.Value; + } + else if (node is AstOperand operand) + { + switch (operand.Type) + { + case IrOperandType.LocalVariable: + type = operand.VarType; + return GetLocal(type, operand); + default: + throw new ArgumentException($"Invalid operand type \"{operand.Type}\"."); + } + } + + throw new NotImplementedException(node.GetType().Name); + } + + private Instruction GetUndefined(AggregateType type) + { + return type switch + { + AggregateType.Bool => ConstantFalse(TypeBool()), + AggregateType.FP32 => Constant(TypeFP32(), 0f), + AggregateType.FP64 => Constant(TypeFP64(), 0d), + _ => Constant(GetType(type), 0), + }; + } + + public Instruction GetConstant(AggregateType type, AstOperand operand) + { + return type switch + { + AggregateType.Bool => operand.Value != 0 ? ConstantTrue(TypeBool()) : ConstantFalse(TypeBool()), + AggregateType.FP32 => Constant(TypeFP32(), BitConverter.Int32BitsToSingle(operand.Value)), + AggregateType.FP64 => Constant(TypeFP64(), (double)BitConverter.Int32BitsToSingle(operand.Value)), + AggregateType.S32 => Constant(TypeS32(), operand.Value), + AggregateType.U32 => Constant(TypeU32(), (uint)operand.Value), + _ => throw new ArgumentException($"Invalid type \"{type}\"."), + }; + } + + public Instruction GetLocalPointer(AstOperand local) + { + return _locals[local]; + } + + public Instruction GetArgumentPointer(AstOperand funcArg) + { + return _funcArgs[funcArg.Value]; + } + + public Instruction GetLocal(AggregateType dstType, AstOperand local) + { + AggregateType srcType = local.VarType; + return BitcastIfNeeded(dstType, srcType, Load(GetType(srcType), GetLocalPointer(local))); + } + + public Instruction GetArgument(AggregateType dstType, AstOperand funcArg) + { + AggregateType srcType = funcArg.VarType; + return BitcastIfNeeded(dstType, srcType, Load(GetType(srcType), GetArgumentPointer(funcArg))); + } + + public (StructuredFunction, Instruction) GetFunction(int funcIndex) + { + return _functions[funcIndex]; + } + + public Instruction GetType(AggregateType type, int length = 1) + { + if ((type & AggregateType.Array) != 0) + { + if (length > 0) + { + return TypeArray(GetType(type & ~AggregateType.Array), Constant(TypeU32(), length)); + } + else + { + return TypeRuntimeArray(GetType(type & ~AggregateType.Array)); + } + } + else if ((type & AggregateType.ElementCountMask) != 0) + { + int vectorLength = (type & AggregateType.ElementCountMask) switch + { + AggregateType.Vector2 => 2, + AggregateType.Vector3 => 3, + AggregateType.Vector4 => 4, + _ => 1, + }; + + return TypeVector(GetType(type & ~AggregateType.ElementCountMask), vectorLength); + } + + return type switch + { + AggregateType.Void => TypeVoid(), + AggregateType.Bool => TypeBool(), + AggregateType.FP32 => TypeFP32(), + AggregateType.FP64 => TypeFP64(), + AggregateType.S32 => TypeS32(), + AggregateType.U32 => TypeU32(), + _ => throw new ArgumentException($"Invalid attribute type \"{type}\"."), + }; + } + + public Instruction BitcastIfNeeded(AggregateType dstType, AggregateType srcType, Instruction value) + { + if (dstType == srcType) + { + return value; + } + + if (dstType == AggregateType.Bool) + { + return INotEqual(TypeBool(), BitcastIfNeeded(AggregateType.S32, srcType, value), Constant(TypeS32(), 0)); + } + else if (srcType == AggregateType.Bool) + { + Instruction intTrue = Constant(TypeS32(), IrConsts.True); + Instruction intFalse = Constant(TypeS32(), IrConsts.False); + + return BitcastIfNeeded(dstType, AggregateType.S32, Select(TypeS32(), value, intTrue, intFalse)); + } + else + { + return Bitcast(GetType(dstType, 1), value); + } + } + + public Instruction TypeS32() + { + return TypeInt(32, true); + } + + public Instruction TypeU32() + { + return TypeInt(32, false); + } + + public Instruction TypeFP32() + { + return TypeFloat(32); + } + + public Instruction TypeFP64() + { + return TypeFloat(64); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/Declarations.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/Declarations.cs new file mode 100644 index 0000000..0bc0fe4 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/Declarations.cs @@ -0,0 +1,659 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.StructuredIr; +using Ryujinx.Graphics.Shader.Translation; +using Spv.Generator; +using System; +using System.Collections.Generic; +using System.Diagnostics; +using static Spv.Specification; +using SpvInstruction = Spv.Generator.Instruction; + +namespace Ryujinx.Graphics.Shader.CodeGen.Spirv +{ + static class Declarations + { + public static void DeclareParameters(CodeGenContext context, StructuredFunction function) + { + DeclareParameters(context, function.InArguments, 0); + DeclareParameters(context, function.OutArguments, function.InArguments.Length); + } + + private static void DeclareParameters(CodeGenContext context, IEnumerable argTypes, int argIndex) + { + foreach (AggregateType argType in argTypes) + { + SpvInstruction argPointerType = context.TypePointer(StorageClass.Function, context.GetType(argType)); + SpvInstruction spvArg = context.FunctionParameter(argPointerType); + + context.DeclareArgument(argIndex++, spvArg); + } + } + + public static void DeclareLocals(CodeGenContext context, StructuredFunction function) + { + foreach (AstOperand local in function.Locals) + { + SpvInstruction localPointerType = context.TypePointer(StorageClass.Function, context.GetType(local.VarType)); + SpvInstruction spvLocal = context.Variable(localPointerType, StorageClass.Function); + + context.AddLocalVariable(spvLocal); + context.DeclareLocal(local, spvLocal); + } + } + + public static void DeclareAll(CodeGenContext context, StructuredProgramInfo info) + { + DeclareConstantBuffers(context, context.Properties.ConstantBuffers.Values); + DeclareStorageBuffers(context, context.Properties.StorageBuffers.Values); + DeclareMemories(context, context.Properties.LocalMemories, context.LocalMemories, StorageClass.Private); + DeclareMemories(context, context.Properties.SharedMemories, context.SharedMemories, StorageClass.Workgroup); + DeclareSamplers(context, context.Properties.Textures.Values); + DeclareImages(context, context.Properties.Images.Values); + DeclareInputsAndOutputs(context, info); + } + + private static void DeclareMemories( + CodeGenContext context, + IReadOnlyDictionary memories, + Dictionary dict, + StorageClass storage) + { + foreach ((int id, MemoryDefinition memory) in memories) + { + SpvInstruction pointerType = context.TypePointer(storage, context.GetType(memory.Type, memory.ArrayLength)); + SpvInstruction variable = context.Variable(pointerType, storage); + + context.AddGlobalVariable(variable); + + dict.Add(id, variable); + } + } + + private static void DeclareConstantBuffers(CodeGenContext context, IEnumerable buffers) + { + DeclareBuffers(context, buffers, isBuffer: false); + } + + private static void DeclareStorageBuffers(CodeGenContext context, IEnumerable buffers) + { + DeclareBuffers(context, buffers, isBuffer: true); + } + + private static void DeclareBuffers(CodeGenContext context, IEnumerable buffers, bool isBuffer) + { + HashSet decoratedTypes = []; + + foreach (BufferDefinition buffer in buffers) + { + int setIndex = context.TargetApi == TargetApi.Vulkan ? buffer.Set : 0; + int alignment = buffer.Layout == BufferLayout.Std140 ? 16 : 4; + int alignmentMask = alignment - 1; + int offset = 0; + + SpvInstruction[] structFieldTypes = new SpvInstruction[buffer.Type.Fields.Length]; + int[] structFieldOffsets = new int[buffer.Type.Fields.Length]; + + for (int fieldIndex = 0; fieldIndex < buffer.Type.Fields.Length; fieldIndex++) + { + StructureField field = buffer.Type.Fields[fieldIndex]; + int fieldSize = (field.Type.GetSizeInBytes() + alignmentMask) & ~alignmentMask; + + structFieldTypes[fieldIndex] = context.GetType(field.Type, field.ArrayLength); + structFieldOffsets[fieldIndex] = offset; + + if (field.Type.HasFlag(AggregateType.Array)) + { + // We can't decorate the type more than once. + if (decoratedTypes.Add(structFieldTypes[fieldIndex])) + { + context.Decorate(structFieldTypes[fieldIndex], Decoration.ArrayStride, (LiteralInteger)fieldSize); + } + + // Zero lengths are assumed to be a "runtime array" (which does not have a explicit length + // specified on the shader, and instead assumes the bound buffer length). + // It is only valid as the last struct element. + + Debug.Assert(field.ArrayLength > 0 || fieldIndex == buffer.Type.Fields.Length - 1); + + offset += fieldSize * field.ArrayLength; + } + else + { + offset += fieldSize; + } + } + + SpvInstruction structType = context.TypeStruct(false, structFieldTypes); + + if (decoratedTypes.Add(structType)) + { + context.Decorate(structType, isBuffer ? Decoration.BufferBlock : Decoration.Block); + + for (int fieldIndex = 0; fieldIndex < structFieldOffsets.Length; fieldIndex++) + { + context.MemberDecorate(structType, fieldIndex, Decoration.Offset, (LiteralInteger)structFieldOffsets[fieldIndex]); + } + } + + SpvInstruction pointerType = context.TypePointer(StorageClass.Uniform, structType); + SpvInstruction variable = context.Variable(pointerType, StorageClass.Uniform); + + context.Name(variable, buffer.Name); + context.Decorate(variable, Decoration.DescriptorSet, (LiteralInteger)setIndex); + context.Decorate(variable, Decoration.Binding, (LiteralInteger)buffer.Binding); + context.AddGlobalVariable(variable); + + if (isBuffer) + { + context.StorageBuffers.Add(buffer.Binding, variable); + } + else + { + context.ConstantBuffers.Add(buffer.Binding, variable); + } + } + } + + private static void DeclareSamplers(CodeGenContext context, IEnumerable samplers) + { + foreach (TextureDefinition sampler in samplers) + { + int setIndex = context.TargetApi == TargetApi.Vulkan ? sampler.Set : 0; + + SpvInstruction imageType; + SpvInstruction sampledImageType; + + if (sampler.Type != SamplerType.None) + { + Dim dim = (sampler.Type & SamplerType.Mask) switch + { + SamplerType.Texture1D => Dim.Dim1D, + SamplerType.Texture2D => Dim.Dim2D, + SamplerType.Texture3D => Dim.Dim3D, + SamplerType.TextureCube => Dim.Cube, + SamplerType.TextureBuffer => Dim.Buffer, + _ => throw new InvalidOperationException($"Invalid sampler type \"{sampler.Type & SamplerType.Mask}\"."), + }; + + imageType = context.TypeImage( + context.TypeFP32(), + dim, + sampler.Type.HasFlag(SamplerType.Shadow), + sampler.Type.HasFlag(SamplerType.Array), + sampler.Type.HasFlag(SamplerType.Multisample), + 1, + ImageFormat.Unknown); + + sampledImageType = context.TypeSampledImage(imageType); + } + else + { + imageType = sampledImageType = context.TypeSampler(); + } + + SpvInstruction sampledOrSeparateImageType = sampler.Separate ? imageType : sampledImageType; + SpvInstruction sampledImagePointerType = context.TypePointer(StorageClass.UniformConstant, sampledOrSeparateImageType); + SpvInstruction sampledImageArrayPointerType = sampledImagePointerType; + + if (sampler.ArrayLength == 0) + { + SpvInstruction sampledImageArrayType = context.TypeRuntimeArray(sampledOrSeparateImageType); + sampledImageArrayPointerType = context.TypePointer(StorageClass.UniformConstant, sampledImageArrayType); + } + else if (sampler.ArrayLength != 1) + { + SpvInstruction sampledImageArrayType = context.TypeArray(sampledOrSeparateImageType, context.Constant(context.TypeU32(), sampler.ArrayLength)); + sampledImageArrayPointerType = context.TypePointer(StorageClass.UniformConstant, sampledImageArrayType); + } + + SpvInstruction sampledImageVariable = context.Variable(sampledImageArrayPointerType, StorageClass.UniformConstant); + + context.Samplers.Add(new(sampler.Set, sampler.Binding), new SamplerDeclaration( + imageType, + sampledImageType, + sampledImagePointerType, + sampledImageVariable, + sampler.ArrayLength != 1)); + context.SamplersTypes.Add(new(sampler.Set, sampler.Binding), sampler.Type); + + context.Name(sampledImageVariable, sampler.Name); + context.Decorate(sampledImageVariable, Decoration.DescriptorSet, (LiteralInteger)setIndex); + context.Decorate(sampledImageVariable, Decoration.Binding, (LiteralInteger)sampler.Binding); + context.AddGlobalVariable(sampledImageVariable); + } + } + + private static void DeclareImages(CodeGenContext context, IEnumerable images) + { + foreach (TextureDefinition image in images) + { + int setIndex = context.TargetApi == TargetApi.Vulkan ? image.Set : 0; + + Dim dim = GetDim(image.Type); + + SpvInstruction imageType = context.TypeImage( + context.GetType(image.Format.GetComponentType()), + dim, + image.Type.HasFlag(SamplerType.Shadow), + image.Type.HasFlag(SamplerType.Array), + image.Type.HasFlag(SamplerType.Multisample), + AccessQualifier.ReadWrite, + GetImageFormat(image.Format)); + + SpvInstruction imagePointerType = context.TypePointer(StorageClass.UniformConstant, imageType); + SpvInstruction imageArrayPointerType = imagePointerType; + + if (image.ArrayLength == 0) + { + SpvInstruction imageArrayType = context.TypeRuntimeArray(imageType); + imageArrayPointerType = context.TypePointer(StorageClass.UniformConstant, imageArrayType); + } + else if (image.ArrayLength != 1) + { + SpvInstruction imageArrayType = context.TypeArray(imageType, context.Constant(context.TypeU32(), image.ArrayLength)); + imageArrayPointerType = context.TypePointer(StorageClass.UniformConstant, imageArrayType); + } + + SpvInstruction imageVariable = context.Variable(imageArrayPointerType, StorageClass.UniformConstant); + + context.Images.Add(new(image.Set, image.Binding), new ImageDeclaration(imageType, imagePointerType, imageVariable, image.ArrayLength != 1)); + + context.Name(imageVariable, image.Name); + context.Decorate(imageVariable, Decoration.DescriptorSet, (LiteralInteger)setIndex); + context.Decorate(imageVariable, Decoration.Binding, (LiteralInteger)image.Binding); + + if (image.Flags.HasFlag(TextureUsageFlags.ImageCoherent)) + { + context.Decorate(imageVariable, Decoration.Coherent); + } + + context.AddGlobalVariable(imageVariable); + } + } + + private static Dim GetDim(SamplerType type) + { + return (type & SamplerType.Mask) switch + { + SamplerType.Texture1D => Dim.Dim1D, + SamplerType.Texture2D => Dim.Dim2D, + SamplerType.Texture3D => Dim.Dim3D, + SamplerType.TextureCube => Dim.Cube, + SamplerType.TextureBuffer => Dim.Buffer, + _ => throw new ArgumentException($"Invalid sampler type \"{type & SamplerType.Mask}\"."), + }; + } + + private static ImageFormat GetImageFormat(TextureFormat format) + { + return format switch + { + TextureFormat.Unknown => ImageFormat.Unknown, + TextureFormat.R8Unorm => ImageFormat.R8, + TextureFormat.R8Snorm => ImageFormat.R8Snorm, + TextureFormat.R8Uint => ImageFormat.R8ui, + TextureFormat.R8Sint => ImageFormat.R8i, + TextureFormat.R16Float => ImageFormat.R16f, + TextureFormat.R16Unorm => ImageFormat.R16, + TextureFormat.R16Snorm => ImageFormat.R16Snorm, + TextureFormat.R16Uint => ImageFormat.R16ui, + TextureFormat.R16Sint => ImageFormat.R16i, + TextureFormat.R32Float => ImageFormat.R32f, + TextureFormat.R32Uint => ImageFormat.R32ui, + TextureFormat.R32Sint => ImageFormat.R32i, + TextureFormat.R8G8Unorm => ImageFormat.Rg8, + TextureFormat.R8G8Snorm => ImageFormat.Rg8Snorm, + TextureFormat.R8G8Uint => ImageFormat.Rg8ui, + TextureFormat.R8G8Sint => ImageFormat.Rg8i, + TextureFormat.R16G16Float => ImageFormat.Rg16f, + TextureFormat.R16G16Unorm => ImageFormat.Rg16, + TextureFormat.R16G16Snorm => ImageFormat.Rg16Snorm, + TextureFormat.R16G16Uint => ImageFormat.Rg16ui, + TextureFormat.R16G16Sint => ImageFormat.Rg16i, + TextureFormat.R32G32Float => ImageFormat.Rg32f, + TextureFormat.R32G32Uint => ImageFormat.Rg32ui, + TextureFormat.R32G32Sint => ImageFormat.Rg32i, + TextureFormat.R8G8B8A8Unorm => ImageFormat.Rgba8, + TextureFormat.R8G8B8A8Snorm => ImageFormat.Rgba8Snorm, + TextureFormat.R8G8B8A8Uint => ImageFormat.Rgba8ui, + TextureFormat.R8G8B8A8Sint => ImageFormat.Rgba8i, + TextureFormat.R16G16B16A16Float => ImageFormat.Rgba16f, + TextureFormat.R16G16B16A16Unorm => ImageFormat.Rgba16, + TextureFormat.R16G16B16A16Snorm => ImageFormat.Rgba16Snorm, + TextureFormat.R16G16B16A16Uint => ImageFormat.Rgba16ui, + TextureFormat.R16G16B16A16Sint => ImageFormat.Rgba16i, + TextureFormat.R32G32B32A32Float => ImageFormat.Rgba32f, + TextureFormat.R32G32B32A32Uint => ImageFormat.Rgba32ui, + TextureFormat.R32G32B32A32Sint => ImageFormat.Rgba32i, + TextureFormat.R10G10B10A2Unorm => ImageFormat.Rgb10A2, + TextureFormat.R10G10B10A2Uint => ImageFormat.Rgb10a2ui, + TextureFormat.R11G11B10Float => ImageFormat.R11fG11fB10f, + _ => throw new ArgumentException($"Invalid texture format \"{format}\"."), + }; + } + + private static void DeclareInputsAndOutputs(CodeGenContext context, StructuredProgramInfo info) + { + int firstLocation = int.MaxValue; + + if (context.Definitions.Stage == ShaderStage.Fragment && context.Definitions.DualSourceBlend) + { + foreach (IoDefinition ioDefinition in info.IoDefinitions) + { + if (ioDefinition.IoVariable == IoVariable.FragmentOutputColor && ioDefinition.Location < firstLocation) + { + firstLocation = ioDefinition.Location; + } + } + } + + foreach (IoDefinition ioDefinition in info.IoDefinitions) + { + PixelImap iq = PixelImap.Unused; + + if (context.Definitions.Stage == ShaderStage.Fragment) + { + IoVariable ioVariable = ioDefinition.IoVariable; + if (ioVariable == IoVariable.UserDefined) + { + iq = context.Definitions.ImapTypes[ioDefinition.Location].GetFirstUsedType(); + } + else + { + (_, AggregateType varType) = IoMap.GetSpirvBuiltIn(ioVariable); + AggregateType elemType = varType & AggregateType.ElementTypeMask; + + if (elemType is AggregateType.S32 or AggregateType.U32) + { + iq = PixelImap.Constant; + } + } + } + else if (IoMap.IsPerVertexBuiltIn(ioDefinition.IoVariable)) + { + continue; + } + + bool isOutput = ioDefinition.StorageKind.IsOutput(); + bool isPerPatch = ioDefinition.StorageKind.IsPerPatch(); + + DeclareInputOrOutput(context, ioDefinition, isOutput, isPerPatch, iq, firstLocation); + } + + DeclarePerVertexBlock(context); + } + + private static void DeclarePerVertexBlock(CodeGenContext context) + { + if (context.Definitions.Stage.IsVtg) + { + if (context.Definitions.Stage != ShaderStage.Vertex) + { + SpvInstruction perVertexInputStructType = CreatePerVertexStructType(context); + int arraySize = context.Definitions.Stage == ShaderStage.Geometry ? context.Definitions.InputTopology.InputVertexCount : 32; + SpvInstruction perVertexInputArrayType = context.TypeArray(perVertexInputStructType, context.Constant(context.TypeU32(), arraySize)); + SpvInstruction perVertexInputPointerType = context.TypePointer(StorageClass.Input, perVertexInputArrayType); + SpvInstruction perVertexInputVariable = context.Variable(perVertexInputPointerType, StorageClass.Input); + + context.Name(perVertexInputVariable, "gl_in"); + + context.AddGlobalVariable(perVertexInputVariable); + context.Inputs.Add(new IoDefinition(StorageKind.Input, IoVariable.Position), perVertexInputVariable); + + if (context.Definitions.Stage == ShaderStage.Geometry && + context.Definitions.GpPassthrough && + context.HostCapabilities.SupportsGeometryShaderPassthrough) + { + context.MemberDecorate(perVertexInputStructType, 0, Decoration.PassthroughNV); + context.MemberDecorate(perVertexInputStructType, 1, Decoration.PassthroughNV); + context.MemberDecorate(perVertexInputStructType, 2, Decoration.PassthroughNV); + context.MemberDecorate(perVertexInputStructType, 3, Decoration.PassthroughNV); + } + } + + SpvInstruction perVertexOutputStructType = CreatePerVertexStructType(context); + + void DecorateTfo(IoVariable ioVariable, int fieldIndex) + { + if (context.Definitions.TryGetTransformFeedbackOutput(ioVariable, 0, 0, out TransformFeedbackOutput transformFeedbackOutput)) + { + context.MemberDecorate(perVertexOutputStructType, fieldIndex, Decoration.XfbBuffer, (LiteralInteger)transformFeedbackOutput.Buffer); + context.MemberDecorate(perVertexOutputStructType, fieldIndex, Decoration.XfbStride, (LiteralInteger)transformFeedbackOutput.Stride); + context.MemberDecorate(perVertexOutputStructType, fieldIndex, Decoration.Offset, (LiteralInteger)transformFeedbackOutput.Offset); + } + } + + DecorateTfo(IoVariable.Position, 0); + DecorateTfo(IoVariable.PointSize, 1); + DecorateTfo(IoVariable.ClipDistance, 2); + + SpvInstruction perVertexOutputArrayType; + + if (context.Definitions.Stage == ShaderStage.TessellationControl) + { + int arraySize = context.Definitions.ThreadsPerInputPrimitive; + perVertexOutputArrayType = context.TypeArray(perVertexOutputStructType, context.Constant(context.TypeU32(), arraySize)); + } + else + { + perVertexOutputArrayType = perVertexOutputStructType; + } + + SpvInstruction perVertexOutputPointerType = context.TypePointer(StorageClass.Output, perVertexOutputArrayType); + SpvInstruction perVertexOutputVariable = context.Variable(perVertexOutputPointerType, StorageClass.Output); + + context.AddGlobalVariable(perVertexOutputVariable); + context.Outputs.Add(new IoDefinition(StorageKind.Output, IoVariable.Position), perVertexOutputVariable); + } + } + + private static SpvInstruction CreatePerVertexStructType(CodeGenContext context) + { + SpvInstruction vec4FloatType = context.TypeVector(context.TypeFP32(), 4); + SpvInstruction floatType = context.TypeFP32(); + SpvInstruction array8FloatType = context.TypeArray(context.TypeFP32(), context.Constant(context.TypeU32(), 8)); + SpvInstruction array1FloatType = context.TypeArray(context.TypeFP32(), context.Constant(context.TypeU32(), 1)); + + SpvInstruction perVertexStructType = context.TypeStruct(true, vec4FloatType, floatType, array8FloatType, array1FloatType); + + context.Name(perVertexStructType, "gl_PerVertex"); + + context.MemberName(perVertexStructType, 0, "gl_Position"); + context.MemberName(perVertexStructType, 1, "gl_PointSize"); + context.MemberName(perVertexStructType, 2, "gl_ClipDistance"); + context.MemberName(perVertexStructType, 3, "gl_CullDistance"); + + context.Decorate(perVertexStructType, Decoration.Block); + + if (context.HostCapabilities.ReducedPrecision) + { + context.MemberDecorate(perVertexStructType, 0, Decoration.Invariant); + } + + context.MemberDecorate(perVertexStructType, 0, Decoration.BuiltIn, (LiteralInteger)BuiltIn.Position); + context.MemberDecorate(perVertexStructType, 1, Decoration.BuiltIn, (LiteralInteger)BuiltIn.PointSize); + context.MemberDecorate(perVertexStructType, 2, Decoration.BuiltIn, (LiteralInteger)BuiltIn.ClipDistance); + context.MemberDecorate(perVertexStructType, 3, Decoration.BuiltIn, (LiteralInteger)BuiltIn.CullDistance); + + return perVertexStructType; + } + + private static void DeclareInputOrOutput( + CodeGenContext context, + IoDefinition ioDefinition, + bool isOutput, + bool isPerPatch, + PixelImap iq = PixelImap.Unused, + int firstLocation = 0) + { + IoVariable ioVariable = ioDefinition.IoVariable; + StorageClass storageClass = isOutput ? StorageClass.Output : StorageClass.Input; + + bool isBuiltIn; + BuiltIn builtIn = default; + AggregateType varType; + + if (ioVariable == IoVariable.UserDefined) + { + varType = context.Definitions.GetUserDefinedType(ioDefinition.Location, isOutput); + isBuiltIn = false; + } + else if (ioVariable == IoVariable.FragmentOutputColor) + { + varType = context.Definitions.GetFragmentOutputColorType(ioDefinition.Location); + isBuiltIn = false; + } + else + { + (builtIn, varType) = IoMap.GetSpirvBuiltIn(ioVariable); + isBuiltIn = true; + + if (varType == AggregateType.Invalid) + { + throw new InvalidOperationException($"Unknown variable {ioVariable}."); + } + } + + bool hasComponent = context.Definitions.HasPerLocationInputOrOutputComponent(ioVariable, ioDefinition.Location, ioDefinition.Component, isOutput); + + if (hasComponent) + { + varType &= AggregateType.ElementTypeMask; + } + else if (ioVariable == IoVariable.UserDefined && context.Definitions.HasTransformFeedbackOutputs(isOutput)) + { + varType &= AggregateType.ElementTypeMask; + varType |= context.Definitions.GetTransformFeedbackOutputComponents(ioDefinition.Location, ioDefinition.Component) switch + { + 2 => AggregateType.Vector2, + 3 => AggregateType.Vector3, + 4 => AggregateType.Vector4, + _ => AggregateType.Invalid, + }; + } + + SpvInstruction spvType = context.GetType(varType, IoMap.GetSpirvBuiltInArrayLength(ioVariable)); + bool builtInPassthrough = false; + + if (!isPerPatch && IoMap.IsPerVertex(ioVariable, context.Definitions.Stage, isOutput)) + { + int arraySize = context.Definitions.Stage == ShaderStage.Geometry ? context.Definitions.InputTopology.InputVertexCount : 32; + spvType = context.TypeArray(spvType, context.Constant(context.TypeU32(), arraySize)); + + if (context.Definitions.GpPassthrough && context.HostCapabilities.SupportsGeometryShaderPassthrough) + { + builtInPassthrough = true; + } + } + + if (context.Definitions.Stage == ShaderStage.TessellationControl && isOutput && !isPerPatch) + { + spvType = context.TypeArray(spvType, context.Constant(context.TypeU32(), context.Definitions.ThreadsPerInputPrimitive)); + } + + SpvInstruction spvPointerType = context.TypePointer(storageClass, spvType); + SpvInstruction spvVar = context.Variable(spvPointerType, storageClass); + + if (builtInPassthrough) + { + context.Decorate(spvVar, Decoration.PassthroughNV); + } + + if (isBuiltIn) + { + if (isPerPatch) + { + context.Decorate(spvVar, Decoration.Patch); + } + + if (context.HostCapabilities.ReducedPrecision && ioVariable == IoVariable.Position) + { + context.Decorate(spvVar, Decoration.Invariant); + } + + context.Decorate(spvVar, Decoration.BuiltIn, (LiteralInteger)builtIn); + } + else if (isPerPatch) + { + context.Decorate(spvVar, Decoration.Patch); + + if (ioVariable == IoVariable.UserDefined) + { + int location = context.AttributeUsage.GetPerPatchAttributeLocation(ioDefinition.Location); + + context.Decorate(spvVar, Decoration.Location, (LiteralInteger)location); + } + } + else if (ioVariable == IoVariable.UserDefined) + { + context.Decorate(spvVar, Decoration.Location, (LiteralInteger)ioDefinition.Location); + + if (hasComponent) + { + context.Decorate(spvVar, Decoration.Component, (LiteralInteger)ioDefinition.Component); + } + + if (!isOutput && + !isPerPatch && + (context.AttributeUsage.PassthroughAttributes & (1 << ioDefinition.Location)) != 0 && + context.HostCapabilities.SupportsGeometryShaderPassthrough) + { + context.Decorate(spvVar, Decoration.PassthroughNV); + } + } + else if (ioVariable == IoVariable.FragmentOutputColor) + { + int location = ioDefinition.Location; + + if (context.Definitions.Stage == ShaderStage.Fragment && context.Definitions.DualSourceBlend) + { + int index = location - firstLocation; + + if ((uint)index < 2) + { + context.Decorate(spvVar, Decoration.Location, (LiteralInteger)firstLocation); + context.Decorate(spvVar, Decoration.Index, (LiteralInteger)index); + } + else + { + context.Decorate(spvVar, Decoration.Location, (LiteralInteger)location); + } + } + else + { + context.Decorate(spvVar, Decoration.Location, (LiteralInteger)location); + } + } + + if (!isOutput) + { + switch (iq) + { + case PixelImap.Constant: + context.Decorate(spvVar, Decoration.Flat); + break; + case PixelImap.ScreenLinear: + context.Decorate(spvVar, Decoration.NoPerspective); + break; + } + } + else if (context.Definitions.TryGetTransformFeedbackOutput( + ioVariable, + ioDefinition.Location, + ioDefinition.Component, + out TransformFeedbackOutput transformFeedbackOutput)) + { + context.Decorate(spvVar, Decoration.XfbBuffer, (LiteralInteger)transformFeedbackOutput.Buffer); + context.Decorate(spvVar, Decoration.XfbStride, (LiteralInteger)transformFeedbackOutput.Stride); + context.Decorate(spvVar, Decoration.Offset, (LiteralInteger)transformFeedbackOutput.Offset); + } + + context.AddGlobalVariable(spvVar); + + Dictionary dict = isPerPatch + ? (isOutput ? context.OutputsPerPatch : context.InputsPerPatch) + : (isOutput ? context.Outputs : context.Inputs); + dict.Add(ioDefinition, spvVar); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/EnumConversion.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/EnumConversion.cs new file mode 100644 index 0000000..2417b04 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/EnumConversion.cs @@ -0,0 +1,22 @@ +using System; +using static Spv.Specification; + +namespace Ryujinx.Graphics.Shader.CodeGen.Spirv +{ + static class EnumConversion + { + public static ExecutionModel Convert(this ShaderStage stage) + { + return stage switch + { + ShaderStage.Compute => ExecutionModel.GLCompute, + ShaderStage.Vertex => ExecutionModel.Vertex, + ShaderStage.TessellationControl => ExecutionModel.TessellationControl, + ShaderStage.TessellationEvaluation => ExecutionModel.TessellationEvaluation, + ShaderStage.Geometry => ExecutionModel.Geometry, + ShaderStage.Fragment => ExecutionModel.Fragment, + _ => throw new ArgumentException($"Invalid shader stage \"{stage}\"."), + }; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/ImageDeclaration.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/ImageDeclaration.cs new file mode 100644 index 0000000..1fbd31f --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/ImageDeclaration.cs @@ -0,0 +1,20 @@ +using Spv.Generator; + +namespace Ryujinx.Graphics.Shader.CodeGen.Spirv +{ + readonly struct ImageDeclaration + { + public readonly Instruction ImageType; + public readonly Instruction ImagePointerType; + public readonly Instruction Image; + public readonly bool IsIndexed; + + public ImageDeclaration(Instruction imageType, Instruction imagePointerType, Instruction image, bool isIndexed) + { + ImageType = imageType; + ImagePointerType = imagePointerType; + Image = image; + IsIndexed = isIndexed; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/Instructions.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/Instructions.cs new file mode 100644 index 0000000..bb2dd67 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/Instructions.cs @@ -0,0 +1,2190 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.StructuredIr; +using Ryujinx.Graphics.Shader.Translation; +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Numerics; +using static Spv.Specification; + +namespace Ryujinx.Graphics.Shader.CodeGen.Spirv +{ + using SpvInstruction = Spv.Generator.Instruction; + using SpvLiteralInteger = Spv.Generator.LiteralInteger; + + static class Instructions + { + private const MemorySemanticsMask DefaultMemorySemantics = + MemorySemanticsMask.ImageMemory | + MemorySemanticsMask.AtomicCounterMemory | + MemorySemanticsMask.WorkgroupMemory | + MemorySemanticsMask.UniformMemory | + MemorySemanticsMask.AcquireRelease; + + private static readonly Func[] _instTable; + + static Instructions() + { + _instTable = new Func[(int)Instruction.Count]; + +#pragma warning disable IDE0055 // Disable formatting + Add(Instruction.Absolute, GenerateAbsolute); + Add(Instruction.Add, GenerateAdd); + Add(Instruction.AtomicAdd, GenerateAtomicAdd); + Add(Instruction.AtomicAnd, GenerateAtomicAnd); + Add(Instruction.AtomicCompareAndSwap, GenerateAtomicCompareAndSwap); + Add(Instruction.AtomicMinS32, GenerateAtomicMinS32); + Add(Instruction.AtomicMinU32, GenerateAtomicMinU32); + Add(Instruction.AtomicMaxS32, GenerateAtomicMaxS32); + Add(Instruction.AtomicMaxU32, GenerateAtomicMaxU32); + Add(Instruction.AtomicOr, GenerateAtomicOr); + Add(Instruction.AtomicSwap, GenerateAtomicSwap); + Add(Instruction.AtomicXor, GenerateAtomicXor); + Add(Instruction.Ballot, GenerateBallot); + Add(Instruction.Barrier, GenerateBarrier); + Add(Instruction.BitCount, GenerateBitCount); + Add(Instruction.BitfieldExtractS32, GenerateBitfieldExtractS32); + Add(Instruction.BitfieldExtractU32, GenerateBitfieldExtractU32); + Add(Instruction.BitfieldInsert, GenerateBitfieldInsert); + Add(Instruction.BitfieldReverse, GenerateBitfieldReverse); + Add(Instruction.BitwiseAnd, GenerateBitwiseAnd); + Add(Instruction.BitwiseExclusiveOr, GenerateBitwiseExclusiveOr); + Add(Instruction.BitwiseNot, GenerateBitwiseNot); + Add(Instruction.BitwiseOr, GenerateBitwiseOr); + Add(Instruction.Call, GenerateCall); + Add(Instruction.Ceiling, GenerateCeiling); + Add(Instruction.Clamp, GenerateClamp); + Add(Instruction.ClampU32, GenerateClampU32); + Add(Instruction.Comment, GenerateComment); + Add(Instruction.CompareEqual, GenerateCompareEqual); + Add(Instruction.CompareGreater, GenerateCompareGreater); + Add(Instruction.CompareGreaterOrEqual, GenerateCompareGreaterOrEqual); + Add(Instruction.CompareGreaterOrEqualU32, GenerateCompareGreaterOrEqualU32); + Add(Instruction.CompareGreaterU32, GenerateCompareGreaterU32); + Add(Instruction.CompareLess, GenerateCompareLess); + Add(Instruction.CompareLessOrEqual, GenerateCompareLessOrEqual); + Add(Instruction.CompareLessOrEqualU32, GenerateCompareLessOrEqualU32); + Add(Instruction.CompareLessU32, GenerateCompareLessU32); + Add(Instruction.CompareNotEqual, GenerateCompareNotEqual); + Add(Instruction.ConditionalSelect, GenerateConditionalSelect); + Add(Instruction.ConvertFP32ToFP64, GenerateConvertFP32ToFP64); + Add(Instruction.ConvertFP32ToS32, GenerateConvertFP32ToS32); + Add(Instruction.ConvertFP32ToU32, GenerateConvertFP32ToU32); + Add(Instruction.ConvertFP64ToFP32, GenerateConvertFP64ToFP32); + Add(Instruction.ConvertFP64ToS32, GenerateConvertFP64ToS32); + Add(Instruction.ConvertFP64ToU32, GenerateConvertFP64ToU32); + Add(Instruction.ConvertS32ToFP32, GenerateConvertS32ToFP32); + Add(Instruction.ConvertS32ToFP64, GenerateConvertS32ToFP64); + Add(Instruction.ConvertU32ToFP32, GenerateConvertU32ToFP32); + Add(Instruction.ConvertU32ToFP64, GenerateConvertU32ToFP64); + Add(Instruction.Cosine, GenerateCosine); + Add(Instruction.Ddx, GenerateDdx); + Add(Instruction.Ddy, GenerateDdy); + Add(Instruction.Discard, GenerateDiscard); + Add(Instruction.Divide, GenerateDivide); + Add(Instruction.EmitVertex, GenerateEmitVertex); + Add(Instruction.EndPrimitive, GenerateEndPrimitive); + Add(Instruction.ExponentB2, GenerateExponentB2); + Add(Instruction.FSIBegin, GenerateFSIBegin); + Add(Instruction.FSIEnd, GenerateFSIEnd); + Add(Instruction.FindLSB, GenerateFindLSB); + Add(Instruction.FindMSBS32, GenerateFindMSBS32); + Add(Instruction.FindMSBU32, GenerateFindMSBU32); + Add(Instruction.Floor, GenerateFloor); + Add(Instruction.FusedMultiplyAdd, GenerateFusedMultiplyAdd); + Add(Instruction.GroupMemoryBarrier, GenerateGroupMemoryBarrier); + Add(Instruction.ImageAtomic, GenerateImageAtomic); + Add(Instruction.ImageLoad, GenerateImageLoad); + Add(Instruction.ImageStore, GenerateImageStore); + Add(Instruction.IsNan, GenerateIsNan); + Add(Instruction.Load, GenerateLoad); + Add(Instruction.Lod, GenerateLod); + Add(Instruction.LogarithmB2, GenerateLogarithmB2); + Add(Instruction.LogicalAnd, GenerateLogicalAnd); + Add(Instruction.LogicalExclusiveOr, GenerateLogicalExclusiveOr); + Add(Instruction.LogicalNot, GenerateLogicalNot); + Add(Instruction.LogicalOr, GenerateLogicalOr); + Add(Instruction.LoopBreak, GenerateLoopBreak); + Add(Instruction.LoopContinue, GenerateLoopContinue); + Add(Instruction.Maximum, GenerateMaximum); + Add(Instruction.MaximumU32, GenerateMaximumU32); + Add(Instruction.MemoryBarrier, GenerateMemoryBarrier); + Add(Instruction.Minimum, GenerateMinimum); + Add(Instruction.MinimumU32, GenerateMinimumU32); + Add(Instruction.Modulo, GenerateModulo); + Add(Instruction.Multiply, GenerateMultiply); + Add(Instruction.MultiplyHighS32, GenerateMultiplyHighS32); + Add(Instruction.MultiplyHighU32, GenerateMultiplyHighU32); + Add(Instruction.Negate, GenerateNegate); + Add(Instruction.PackDouble2x32, GeneratePackDouble2x32); + Add(Instruction.PackHalf2x16, GeneratePackHalf2x16); + Add(Instruction.ReciprocalSquareRoot, GenerateReciprocalSquareRoot); + Add(Instruction.Return, GenerateReturn); + Add(Instruction.Round, GenerateRound); + Add(Instruction.ShiftLeft, GenerateShiftLeft); + Add(Instruction.ShiftRightS32, GenerateShiftRightS32); + Add(Instruction.ShiftRightU32, GenerateShiftRightU32); + Add(Instruction.Shuffle, GenerateShuffle); + Add(Instruction.ShuffleDown, GenerateShuffleDown); + Add(Instruction.ShuffleUp, GenerateShuffleUp); + Add(Instruction.ShuffleXor, GenerateShuffleXor); + Add(Instruction.Sine, GenerateSine); + Add(Instruction.SquareRoot, GenerateSquareRoot); + Add(Instruction.Store, GenerateStore); + Add(Instruction.Subtract, GenerateSubtract); + Add(Instruction.SwizzleAdd, GenerateSwizzleAdd); + Add(Instruction.TextureSample, GenerateTextureSample); + Add(Instruction.TextureQuerySamples, GenerateTextureQuerySamples); + Add(Instruction.TextureQuerySize, GenerateTextureQuerySize); + Add(Instruction.Truncate, GenerateTruncate); + Add(Instruction.UnpackDouble2x32, GenerateUnpackDouble2x32); + Add(Instruction.UnpackHalf2x16, GenerateUnpackHalf2x16); + Add(Instruction.VectorExtract, GenerateVectorExtract); + Add(Instruction.VoteAll, GenerateVoteAll); + Add(Instruction.VoteAllEqual, GenerateVoteAllEqual); + Add(Instruction.VoteAny, GenerateVoteAny); +#pragma warning restore IDE0055 + } + + private static void Add(Instruction inst, Func handler) + { + _instTable[(int)(inst & Instruction.Mask)] = handler; + } + + public static OperationResult Generate(CodeGenContext context, AstOperation operation) + { + Func handler = _instTable[(int)(operation.Inst & Instruction.Mask)]; + if (handler != null) + { + return handler(context, operation); + } + else + { + throw new NotImplementedException(operation.Inst.ToString()); + } + } + + private static OperationResult GenerateAbsolute(CodeGenContext context, AstOperation operation) + { + return GenerateUnary(context, operation, context.Delegates.GlslFAbs, context.Delegates.GlslSAbs); + } + + private static OperationResult GenerateAdd(CodeGenContext context, AstOperation operation) + { + return GenerateBinary(context, operation, context.Delegates.FAdd, context.Delegates.IAdd); + } + + private static OperationResult GenerateAtomicAdd(CodeGenContext context, AstOperation operation) + { + return GenerateAtomicMemoryBinary(context, operation, context.Delegates.AtomicIAdd); + } + + private static OperationResult GenerateAtomicAnd(CodeGenContext context, AstOperation operation) + { + return GenerateAtomicMemoryBinary(context, operation, context.Delegates.AtomicAnd); + } + + private static OperationResult GenerateAtomicCompareAndSwap(CodeGenContext context, AstOperation operation) + { + return GenerateAtomicMemoryCas(context, operation); + } + + private static OperationResult GenerateAtomicMinS32(CodeGenContext context, AstOperation operation) + { + return GenerateAtomicMemoryBinary(context, operation, context.Delegates.AtomicSMin); + } + + private static OperationResult GenerateAtomicMinU32(CodeGenContext context, AstOperation operation) + { + return GenerateAtomicMemoryBinary(context, operation, context.Delegates.AtomicUMin); + } + + private static OperationResult GenerateAtomicMaxS32(CodeGenContext context, AstOperation operation) + { + return GenerateAtomicMemoryBinary(context, operation, context.Delegates.AtomicSMax); + } + + private static OperationResult GenerateAtomicMaxU32(CodeGenContext context, AstOperation operation) + { + return GenerateAtomicMemoryBinary(context, operation, context.Delegates.AtomicUMax); + } + + private static OperationResult GenerateAtomicOr(CodeGenContext context, AstOperation operation) + { + return GenerateAtomicMemoryBinary(context, operation, context.Delegates.AtomicOr); + } + + private static OperationResult GenerateAtomicSwap(CodeGenContext context, AstOperation operation) + { + return GenerateAtomicMemoryBinary(context, operation, context.Delegates.AtomicExchange); + } + + private static OperationResult GenerateAtomicXor(CodeGenContext context, AstOperation operation) + { + return GenerateAtomicMemoryBinary(context, operation, context.Delegates.AtomicXor); + } + + private static OperationResult GenerateBallot(CodeGenContext context, AstOperation operation) + { + IAstNode source = operation.GetSource(0); + + SpvInstruction uvec4Type = context.TypeVector(context.TypeU32(), 4); + SpvInstruction execution = context.Constant(context.TypeU32(), Scope.Subgroup); + + SpvInstruction maskVector = context.GroupNonUniformBallot(uvec4Type, execution, context.Get(AggregateType.Bool, source)); + SpvInstruction mask = context.CompositeExtract(context.TypeU32(), maskVector, (SpvLiteralInteger)operation.Index); + + return new OperationResult(AggregateType.U32, mask); + } + + private static OperationResult GenerateBarrier(CodeGenContext context, AstOperation operation) + { + // Barrier on divergent control flow paths may cause the GPU to hang, + // so skip emitting the barrier for those cases. + if (!context.HostCapabilities.SupportsShaderBarrierDivergence && + (context.CurrentBlock.Type != AstBlockType.Main || context.MayHaveReturned || !context.IsMainFunction)) + { + context.Logger.Log("Shader has barrier on potentially divergent block, the barrier will be removed."); + + return OperationResult.Invalid; + } + + context.ControlBarrier( + context.Constant(context.TypeU32(), Scope.Workgroup), + context.Constant(context.TypeU32(), Scope.Workgroup), + context.Constant(context.TypeU32(), MemorySemanticsMask.WorkgroupMemory | MemorySemanticsMask.AcquireRelease)); + + return OperationResult.Invalid; + } + + private static OperationResult GenerateBitCount(CodeGenContext context, AstOperation operation) + { + return GenerateUnaryS32(context, operation, context.Delegates.BitCount); + } + + private static OperationResult GenerateBitfieldExtractS32(CodeGenContext context, AstOperation operation) + { + return GenerateBitfieldExtractS32(context, operation, context.Delegates.BitFieldSExtract); + } + + private static OperationResult GenerateBitfieldExtractU32(CodeGenContext context, AstOperation operation) + { + return GenerateTernaryU32(context, operation, context.Delegates.BitFieldUExtract); + } + + private static OperationResult GenerateBitfieldInsert(CodeGenContext context, AstOperation operation) + { + return GenerateBitfieldInsert(context, operation, context.Delegates.BitFieldInsert); + } + + private static OperationResult GenerateBitfieldReverse(CodeGenContext context, AstOperation operation) + { + return GenerateUnaryS32(context, operation, context.Delegates.BitReverse); + } + + private static OperationResult GenerateBitwiseAnd(CodeGenContext context, AstOperation operation) + { + return GenerateBinaryS32(context, operation, context.Delegates.BitwiseAnd); + } + + private static OperationResult GenerateBitwiseExclusiveOr(CodeGenContext context, AstOperation operation) + { + return GenerateBinaryS32(context, operation, context.Delegates.BitwiseXor); + } + + private static OperationResult GenerateBitwiseNot(CodeGenContext context, AstOperation operation) + { + return GenerateUnaryS32(context, operation, context.Delegates.Not); + } + + private static OperationResult GenerateBitwiseOr(CodeGenContext context, AstOperation operation) + { + return GenerateBinaryS32(context, operation, context.Delegates.BitwiseOr); + } + + private static OperationResult GenerateCall(CodeGenContext context, AstOperation operation) + { + AstOperand funcId = (AstOperand)operation.GetSource(0); + + Debug.Assert(funcId.Type == OperandType.Constant); + + (StructuredFunction function, SpvInstruction spvFunc) = context.GetFunction(funcId.Value); + + SpvInstruction[] args = new SpvInstruction[operation.SourcesCount - 1]; + + for (int i = 0; i < args.Length; i++) + { + IAstNode operand = operation.GetSource(i + 1); + + AstOperand local = (AstOperand)operand; + Debug.Assert(local.Type == OperandType.LocalVariable); + args[i] = context.GetLocalPointer(local); + } + + AggregateType retType = function.ReturnType; + SpvInstruction result = context.FunctionCall(context.GetType(retType), spvFunc, args); + return new OperationResult(retType, result); + } + + private static OperationResult GenerateCeiling(CodeGenContext context, AstOperation operation) + { + return GenerateUnary(context, operation, context.Delegates.GlslCeil, null); + } + + private static OperationResult GenerateClamp(CodeGenContext context, AstOperation operation) + { + return GenerateTernary(context, operation, context.Delegates.GlslFClamp, context.Delegates.GlslSClamp); + } + + private static OperationResult GenerateClampU32(CodeGenContext context, AstOperation operation) + { + return GenerateTernaryU32(context, operation, context.Delegates.GlslUClamp); + } + + private static OperationResult GenerateComment(CodeGenContext context, AstOperation operation) + { + return OperationResult.Invalid; + } + + private static OperationResult GenerateCompareEqual(CodeGenContext context, AstOperation operation) + { + return GenerateCompare(context, operation, context.Delegates.FOrdEqual, context.Delegates.IEqual); + } + + private static OperationResult GenerateCompareGreater(CodeGenContext context, AstOperation operation) + { + return GenerateCompare(context, operation, context.Delegates.FOrdGreaterThan, context.Delegates.SGreaterThan); + } + + private static OperationResult GenerateCompareGreaterOrEqual(CodeGenContext context, AstOperation operation) + { + return GenerateCompare(context, operation, context.Delegates.FOrdGreaterThanEqual, context.Delegates.SGreaterThanEqual); + } + + private static OperationResult GenerateCompareGreaterOrEqualU32(CodeGenContext context, AstOperation operation) + { + return GenerateCompareU32(context, operation, context.Delegates.UGreaterThanEqual); + } + + private static OperationResult GenerateCompareGreaterU32(CodeGenContext context, AstOperation operation) + { + return GenerateCompareU32(context, operation, context.Delegates.UGreaterThan); + } + + private static OperationResult GenerateCompareLess(CodeGenContext context, AstOperation operation) + { + return GenerateCompare(context, operation, context.Delegates.FOrdLessThan, context.Delegates.SLessThan); + } + + private static OperationResult GenerateCompareLessOrEqual(CodeGenContext context, AstOperation operation) + { + return GenerateCompare(context, operation, context.Delegates.FOrdLessThanEqual, context.Delegates.SLessThanEqual); + } + + private static OperationResult GenerateCompareLessOrEqualU32(CodeGenContext context, AstOperation operation) + { + return GenerateCompareU32(context, operation, context.Delegates.ULessThanEqual); + } + + private static OperationResult GenerateCompareLessU32(CodeGenContext context, AstOperation operation) + { + return GenerateCompareU32(context, operation, context.Delegates.ULessThan); + } + + private static OperationResult GenerateCompareNotEqual(CodeGenContext context, AstOperation operation) + { + return GenerateCompare(context, operation, context.Delegates.FOrdNotEqual, context.Delegates.INotEqual); + } + + private static OperationResult GenerateConditionalSelect(CodeGenContext context, AstOperation operation) + { + IAstNode src1 = operation.GetSource(0); + IAstNode src2 = operation.GetSource(1); + IAstNode src3 = operation.GetSource(2); + + SpvInstruction cond = context.Get(AggregateType.Bool, src1); + + if (operation.Inst.HasFlag(Instruction.FP64)) + { + return new OperationResult(AggregateType.FP64, context.Select(context.TypeFP64(), cond, context.GetFP64(src2), context.GetFP64(src3))); + } + else if (operation.Inst.HasFlag(Instruction.FP32)) + { + return new OperationResult(AggregateType.FP32, context.Select(context.TypeFP32(), cond, context.GetFP32(src2), context.GetFP32(src3))); + } + else + { + return new OperationResult(AggregateType.S32, context.Select(context.TypeS32(), cond, context.GetS32(src2), context.GetS32(src3))); + } + } + + private static OperationResult GenerateConvertFP32ToFP64(CodeGenContext context, AstOperation operation) + { + IAstNode source = operation.GetSource(0); + + return new OperationResult(AggregateType.FP64, context.FConvert(context.TypeFP64(), context.GetFP32(source))); + } + + private static OperationResult GenerateConvertFP32ToS32(CodeGenContext context, AstOperation operation) + { + IAstNode source = operation.GetSource(0); + + return new OperationResult(AggregateType.S32, context.ConvertFToS(context.TypeS32(), context.GetFP32(source))); + } + + private static OperationResult GenerateConvertFP32ToU32(CodeGenContext context, AstOperation operation) + { + IAstNode source = operation.GetSource(0); + + return new OperationResult(AggregateType.U32, context.ConvertFToU(context.TypeU32(), context.GetFP32(source))); + } + + private static OperationResult GenerateConvertFP64ToFP32(CodeGenContext context, AstOperation operation) + { + IAstNode source = operation.GetSource(0); + + return new OperationResult(AggregateType.FP32, context.FConvert(context.TypeFP32(), context.GetFP64(source))); + } + + private static OperationResult GenerateConvertFP64ToS32(CodeGenContext context, AstOperation operation) + { + IAstNode source = operation.GetSource(0); + + return new OperationResult(AggregateType.S32, context.ConvertFToS(context.TypeS32(), context.GetFP64(source))); + } + + private static OperationResult GenerateConvertFP64ToU32(CodeGenContext context, AstOperation operation) + { + IAstNode source = operation.GetSource(0); + + return new OperationResult(AggregateType.U32, context.ConvertFToU(context.TypeU32(), context.GetFP64(source))); + } + + private static OperationResult GenerateConvertS32ToFP32(CodeGenContext context, AstOperation operation) + { + IAstNode source = operation.GetSource(0); + + return new OperationResult(AggregateType.FP32, context.ConvertSToF(context.TypeFP32(), context.GetS32(source))); + } + + private static OperationResult GenerateConvertS32ToFP64(CodeGenContext context, AstOperation operation) + { + IAstNode source = operation.GetSource(0); + + return new OperationResult(AggregateType.FP64, context.ConvertSToF(context.TypeFP64(), context.GetS32(source))); + } + + private static OperationResult GenerateConvertU32ToFP32(CodeGenContext context, AstOperation operation) + { + IAstNode source = operation.GetSource(0); + + return new OperationResult(AggregateType.FP32, context.ConvertUToF(context.TypeFP32(), context.GetU32(source))); + } + + private static OperationResult GenerateConvertU32ToFP64(CodeGenContext context, AstOperation operation) + { + IAstNode source = operation.GetSource(0); + + return new OperationResult(AggregateType.FP64, context.ConvertUToF(context.TypeFP64(), context.GetU32(source))); + } + + private static OperationResult GenerateCosine(CodeGenContext context, AstOperation operation) + { + return GenerateUnary(context, operation, context.Delegates.GlslCos, null); + } + + private static OperationResult GenerateDdx(CodeGenContext context, AstOperation operation) + { + return GenerateUnaryFP32(context, operation, context.Delegates.DPdx); + } + + private static OperationResult GenerateDdy(CodeGenContext context, AstOperation operation) + { + return GenerateUnaryFP32(context, operation, context.Delegates.DPdy); + } + + private static OperationResult GenerateDiscard(CodeGenContext context, AstOperation operation) + { + context.Kill(); + return OperationResult.Invalid; + } + + private static OperationResult GenerateDivide(CodeGenContext context, AstOperation operation) + { + return GenerateBinary(context, operation, context.Delegates.FDiv, context.Delegates.SDiv); + } + + private static OperationResult GenerateEmitVertex(CodeGenContext context, AstOperation operation) + { + context.EmitVertex(); + + return OperationResult.Invalid; + } + + private static OperationResult GenerateEndPrimitive(CodeGenContext context, AstOperation operation) + { + context.EndPrimitive(); + + return OperationResult.Invalid; + } + + private static OperationResult GenerateExponentB2(CodeGenContext context, AstOperation operation) + { + return GenerateUnary(context, operation, context.Delegates.GlslExp2, null); + } + + private static OperationResult GenerateFSIBegin(CodeGenContext context, AstOperation operation) + { + if (context.HostCapabilities.SupportsFragmentShaderInterlock) + { + context.BeginInvocationInterlockEXT(); + } + + return OperationResult.Invalid; + } + + private static OperationResult GenerateFSIEnd(CodeGenContext context, AstOperation operation) + { + if (context.HostCapabilities.SupportsFragmentShaderInterlock) + { + context.EndInvocationInterlockEXT(); + } + + return OperationResult.Invalid; + } + + private static OperationResult GenerateFindLSB(CodeGenContext context, AstOperation operation) + { + SpvInstruction source = context.GetU32(operation.GetSource(0)); + return new OperationResult(AggregateType.U32, context.GlslFindILsb(context.TypeU32(), source)); + } + + private static OperationResult GenerateFindMSBS32(CodeGenContext context, AstOperation operation) + { + SpvInstruction source = context.GetS32(operation.GetSource(0)); + return new OperationResult(AggregateType.U32, context.GlslFindSMsb(context.TypeU32(), source)); + } + + private static OperationResult GenerateFindMSBU32(CodeGenContext context, AstOperation operation) + { + SpvInstruction source = context.GetU32(operation.GetSource(0)); + return new OperationResult(AggregateType.U32, context.GlslFindUMsb(context.TypeU32(), source)); + } + + private static OperationResult GenerateFloor(CodeGenContext context, AstOperation operation) + { + return GenerateUnary(context, operation, context.Delegates.GlslFloor, null); + } + + private static OperationResult GenerateFusedMultiplyAdd(CodeGenContext context, AstOperation operation) + { + return GenerateTernary(context, operation, context.Delegates.GlslFma, null); + } + + private static OperationResult GenerateGroupMemoryBarrier(CodeGenContext context, AstOperation operation) + { + context.MemoryBarrier(context.Constant(context.TypeU32(), Scope.Workgroup), context.Constant(context.TypeU32(), DefaultMemorySemantics)); + return OperationResult.Invalid; + } + + private static void MarkNonUniform(CodeGenContext context, SpvInstruction inst) + { + if (context.HostCapabilities.SupportsShaderNonUniformIndexing) + { + if (!context.WasNonUniformAccessDeclared) + { + context.AddExtension("SPV_EXT_descriptor_indexing"); + context.AddCapability(Capability.ShaderNonUniform); + context.AddCapability(Capability.SampledImageArrayNonUniformIndexing); + context.AddCapability(Capability.StorageImageArrayNonUniformIndexing); + } + + context.Decorate(inst, Decoration.NonUniform); + context.WasNonUniformAccessDeclared = true; + } + } + + private static OperationResult GenerateImageAtomic(CodeGenContext context, AstOperation operation) + { + AstTextureOperation texOp = (AstTextureOperation)operation; + + AggregateType componentType = texOp.Format.GetComponentType(); + + bool isArray = (texOp.Type & SamplerType.Array) != 0; + + int srcIndex = 0; + + SpvInstruction Src(AggregateType type) + { + return context.Get(type, texOp.GetSource(srcIndex++)); + } + + ImageDeclaration declaration = context.Images[texOp.GetTextureSetAndBinding()]; + SpvInstruction image = declaration.Image; + + SpvInstruction resultType = context.GetType(componentType); + SpvInstruction imagePointerType = context.TypePointer(StorageClass.Image, resultType); + + if (declaration.IsIndexed) + { + SpvInstruction textureIndex = Src(AggregateType.S32); + + image = context.AccessChain(imagePointerType, image, textureIndex); + MarkNonUniform(context, image); + } + + int coordsCount = texOp.Type.Dimensions; + + int pCount = coordsCount + (isArray ? 1 : 0); + + SpvInstruction pCoords; + + if (pCount > 1) + { + SpvInstruction[] elems = new SpvInstruction[pCount]; + + for (int i = 0; i < pCount; i++) + { + elems[i] = Src(AggregateType.S32); + } + + SpvInstruction vectorType = context.TypeVector(context.TypeS32(), pCount); + pCoords = context.CompositeConstruct(vectorType, elems); + } + else + { + pCoords = Src(AggregateType.S32); + } + + SpvInstruction value = Src(componentType); + + SpvInstruction pointer = context.ImageTexelPointer(imagePointerType, image, pCoords, context.Constant(context.TypeU32(), 0)); + SpvInstruction one = context.Constant(context.TypeU32(), 1); + SpvInstruction zero = context.Constant(context.TypeU32(), 0); + + SpvInstruction result = (texOp.Flags & TextureFlags.AtomicMask) switch + { + TextureFlags.Add => context.AtomicIAdd(resultType, pointer, one, zero, value), + TextureFlags.Minimum => componentType == AggregateType.S32 + ? context.AtomicSMin(resultType, pointer, one, zero, value) + : context.AtomicUMin(resultType, pointer, one, zero, value), + TextureFlags.Maximum => componentType == AggregateType.S32 + ? context.AtomicSMax(resultType, pointer, one, zero, value) + : context.AtomicUMax(resultType, pointer, one, zero, value), + TextureFlags.Increment => context.AtomicIIncrement(resultType, pointer, one, zero), + TextureFlags.Decrement => context.AtomicIDecrement(resultType, pointer, one, zero), + TextureFlags.BitwiseAnd => context.AtomicAnd(resultType, pointer, one, zero, value), + TextureFlags.BitwiseOr => context.AtomicOr(resultType, pointer, one, zero, value), + TextureFlags.BitwiseXor => context.AtomicXor(resultType, pointer, one, zero, value), + TextureFlags.Swap => context.AtomicExchange(resultType, pointer, one, zero, value), + TextureFlags.CAS => context.AtomicCompareExchange(resultType, pointer, one, zero, zero, Src(componentType), value), + _ => context.AtomicIAdd(resultType, pointer, one, zero, value), + }; + + return new OperationResult(componentType, result); + } + + private static OperationResult GenerateImageLoad(CodeGenContext context, AstOperation operation) + { + AstTextureOperation texOp = (AstTextureOperation)operation; + + AggregateType componentType = texOp.Format.GetComponentType(); + + bool isArray = (texOp.Type & SamplerType.Array) != 0; + + int srcIndex = 0; + + SpvInstruction Src(AggregateType type) + { + return context.Get(type, texOp.GetSource(srcIndex++)); + } + + ImageDeclaration declaration = context.Images[texOp.GetTextureSetAndBinding()]; + SpvInstruction image = declaration.Image; + bool isIndexed = declaration.IsIndexed; + + if (isIndexed) + { + SpvInstruction textureIndex = Src(AggregateType.S32); + + image = context.AccessChain(declaration.ImagePointerType, image, textureIndex); + MarkNonUniform(context, image); + } + + image = context.Load(declaration.ImageType, image); + if (isIndexed) + { + MarkNonUniform(context, image); + } + + int coordsCount = texOp.Type.Dimensions; + + int pCount = coordsCount + (isArray ? 1 : 0); + + SpvInstruction pCoords; + + if (pCount > 1) + { + SpvInstruction[] elems = new SpvInstruction[pCount]; + + for (int i = 0; i < pCount; i++) + { + elems[i] = Src(AggregateType.S32); + } + + SpvInstruction vectorType = context.TypeVector(context.TypeS32(), pCount); + pCoords = context.CompositeConstruct(vectorType, elems); + } + else + { + pCoords = Src(AggregateType.S32); + } + + SpvInstruction imageComponentType = context.GetType(componentType); + AggregateType swizzledResultType = texOp.GetVectorType(componentType); + + SpvInstruction texel = context.ImageRead(context.TypeVector(imageComponentType, 4), image, pCoords, ImageOperandsMask.MaskNone); + SpvInstruction result = GetSwizzledResult(context, texel, swizzledResultType, texOp.Index); + + return new OperationResult(componentType, result); + } + + private static OperationResult GenerateImageStore(CodeGenContext context, AstOperation operation) + { + AstTextureOperation texOp = (AstTextureOperation)operation; + + bool isArray = (texOp.Type & SamplerType.Array) != 0; + + int srcIndex = 0; + + SpvInstruction Src(AggregateType type) + { + return context.Get(type, texOp.GetSource(srcIndex++)); + } + + ImageDeclaration declaration = context.Images[texOp.GetTextureSetAndBinding()]; + SpvInstruction image = declaration.Image; + bool isIndexed = declaration.IsIndexed; + + if (isIndexed) + { + SpvInstruction textureIndex = Src(AggregateType.S32); + + image = context.AccessChain(declaration.ImagePointerType, image, textureIndex); + MarkNonUniform(context, image); + } + + image = context.Load(declaration.ImageType, image); + if (isIndexed) + { + MarkNonUniform(context, image); + } + + int coordsCount = texOp.Type.Dimensions; + + int pCount = coordsCount + (isArray ? 1 : 0); + + SpvInstruction pCoords; + + if (pCount > 1) + { + SpvInstruction[] elems = new SpvInstruction[pCount]; + + for (int i = 0; i < pCount; i++) + { + elems[i] = Src(AggregateType.S32); + } + + SpvInstruction vectorType = context.TypeVector(context.TypeS32(), pCount); + pCoords = context.CompositeConstruct(vectorType, elems); + } + else + { + pCoords = Src(AggregateType.S32); + } + + AggregateType componentType = texOp.Format.GetComponentType(); + + const int ComponentsCount = 4; + + SpvInstruction[] cElems = new SpvInstruction[ComponentsCount]; + + for (int i = 0; i < ComponentsCount; i++) + { + if (srcIndex < texOp.SourcesCount) + { + cElems[i] = Src(componentType); + } + else + { + cElems[i] = componentType switch + { + AggregateType.S32 => context.Constant(context.TypeS32(), 0), + AggregateType.U32 => context.Constant(context.TypeU32(), 0u), + _ => context.Constant(context.TypeFP32(), 0f), + }; + } + } + + SpvInstruction texel = context.CompositeConstruct(context.TypeVector(context.GetType(componentType), ComponentsCount), cElems); + + context.ImageWrite(image, pCoords, texel, ImageOperandsMask.MaskNone); + + return OperationResult.Invalid; + } + + private static OperationResult GenerateIsNan(CodeGenContext context, AstOperation operation) + { + IAstNode source = operation.GetSource(0); + + SpvInstruction result; + + if (operation.Inst.HasFlag(Instruction.FP64)) + { + result = context.IsNan(context.TypeBool(), context.GetFP64(source)); + } + else + { + result = context.IsNan(context.TypeBool(), context.GetFP32(source)); + } + + return new OperationResult(AggregateType.Bool, result); + } + + private static OperationResult GenerateLoad(CodeGenContext context, AstOperation operation) + { + return GenerateLoadOrStore(context, operation, isStore: false); + } + + private static OperationResult GenerateLod(CodeGenContext context, AstOperation operation) + { + AstTextureOperation texOp = (AstTextureOperation)operation; + + int srcIndex = 0; + + SpvInstruction Src(AggregateType type) + { + return context.Get(type, texOp.GetSource(srcIndex++)); + } + + SamplerDeclaration declaration = context.Samplers[texOp.GetTextureSetAndBinding()]; + SpvInstruction image = GenerateSampledImageLoad(context, texOp, declaration, ref srcIndex); + + int pCount = texOp.Type.Dimensions; + + SpvInstruction pCoords; + + if (pCount > 1) + { + SpvInstruction[] elems = new SpvInstruction[pCount]; + + for (int i = 0; i < pCount; i++) + { + elems[i] = Src(AggregateType.FP32); + } + + SpvInstruction vectorType = context.TypeVector(context.TypeFP32(), pCount); + pCoords = context.CompositeConstruct(vectorType, elems); + } + else + { + pCoords = Src(AggregateType.FP32); + } + + SpvInstruction resultType = context.TypeVector(context.TypeFP32(), 2); + SpvInstruction packed = context.ImageQueryLod(resultType, image, pCoords); + SpvInstruction result = context.CompositeExtract(context.TypeFP32(), packed, (SpvLiteralInteger)texOp.Index); + + return new OperationResult(AggregateType.FP32, result); + } + + private static OperationResult GenerateLogarithmB2(CodeGenContext context, AstOperation operation) + { + return GenerateUnary(context, operation, context.Delegates.GlslLog2, null); + } + + private static OperationResult GenerateLogicalAnd(CodeGenContext context, AstOperation operation) + { + return GenerateBinaryBool(context, operation, context.Delegates.LogicalAnd); + } + + private static OperationResult GenerateLogicalExclusiveOr(CodeGenContext context, AstOperation operation) + { + return GenerateBinaryBool(context, operation, context.Delegates.LogicalNotEqual); + } + + private static OperationResult GenerateLogicalNot(CodeGenContext context, AstOperation operation) + { + return GenerateUnaryBool(context, operation, context.Delegates.LogicalNot); + } + + private static OperationResult GenerateLogicalOr(CodeGenContext context, AstOperation operation) + { + return GenerateBinaryBool(context, operation, context.Delegates.LogicalOr); + } + + private static OperationResult GenerateLoopBreak(CodeGenContext context, AstOperation operation) + { + AstBlock loopBlock = context.CurrentBlock; + while (loopBlock.Type != AstBlockType.DoWhile) + { + loopBlock = loopBlock.Parent; + } + + context.Branch(context.GetNextLabel(loopBlock.Parent)); + + return OperationResult.Invalid; + } + + private static OperationResult GenerateLoopContinue(CodeGenContext context, AstOperation operation) + { + AstBlock loopBlock = context.CurrentBlock; + while (loopBlock.Type != AstBlockType.DoWhile) + { + loopBlock = loopBlock.Parent; + } + + (_, SpvInstruction continueTarget) = context.LoopTargets[loopBlock]; + + context.Branch(continueTarget); + + return OperationResult.Invalid; + } + + private static OperationResult GenerateMaximum(CodeGenContext context, AstOperation operation) + { + return GenerateBinary(context, operation, context.Delegates.GlslFMax, context.Delegates.GlslSMax); + } + + private static OperationResult GenerateMaximumU32(CodeGenContext context, AstOperation operation) + { + return GenerateBinaryU32(context, operation, context.Delegates.GlslUMax); + } + + private static OperationResult GenerateMemoryBarrier(CodeGenContext context, AstOperation operation) + { + context.MemoryBarrier(context.Constant(context.TypeU32(), Scope.Device), context.Constant(context.TypeU32(), DefaultMemorySemantics)); + return OperationResult.Invalid; + } + + private static OperationResult GenerateMinimum(CodeGenContext context, AstOperation operation) + { + return GenerateBinary(context, operation, context.Delegates.GlslFMin, context.Delegates.GlslSMin); + } + + private static OperationResult GenerateMinimumU32(CodeGenContext context, AstOperation operation) + { + return GenerateBinaryU32(context, operation, context.Delegates.GlslUMin); + } + + private static OperationResult GenerateModulo(CodeGenContext context, AstOperation operation) + { + return GenerateBinary(context, operation, context.Delegates.FMod, null); + } + + private static OperationResult GenerateMultiply(CodeGenContext context, AstOperation operation) + { + return GenerateBinary(context, operation, context.Delegates.FMul, context.Delegates.IMul); + } + + private static OperationResult GenerateMultiplyHighS32(CodeGenContext context, AstOperation operation) + { + IAstNode src1 = operation.GetSource(0); + IAstNode src2 = operation.GetSource(1); + + SpvInstruction resultType = context.TypeStruct(false, context.TypeS32(), context.TypeS32()); + SpvInstruction result = context.SMulExtended(resultType, context.GetS32(src1), context.GetS32(src2)); + result = context.CompositeExtract(context.TypeS32(), result, 1); + + return new OperationResult(AggregateType.S32, result); + } + + private static OperationResult GenerateMultiplyHighU32(CodeGenContext context, AstOperation operation) + { + IAstNode src1 = operation.GetSource(0); + IAstNode src2 = operation.GetSource(1); + + SpvInstruction resultType = context.TypeStruct(false, context.TypeU32(), context.TypeU32()); + SpvInstruction result = context.UMulExtended(resultType, context.GetU32(src1), context.GetU32(src2)); + result = context.CompositeExtract(context.TypeU32(), result, 1); + + return new OperationResult(AggregateType.U32, result); + } + + private static OperationResult GenerateNegate(CodeGenContext context, AstOperation operation) + { + return GenerateUnary(context, operation, context.Delegates.FNegate, context.Delegates.SNegate); + } + + private static OperationResult GeneratePackDouble2x32(CodeGenContext context, AstOperation operation) + { + SpvInstruction value0 = context.GetU32(operation.GetSource(0)); + SpvInstruction value1 = context.GetU32(operation.GetSource(1)); + SpvInstruction vector = context.CompositeConstruct(context.TypeVector(context.TypeU32(), 2), value0, value1); + SpvInstruction result = context.GlslPackDouble2x32(context.TypeFP64(), vector); + + return new OperationResult(AggregateType.FP64, result); + } + + private static OperationResult GeneratePackHalf2x16(CodeGenContext context, AstOperation operation) + { + SpvInstruction value0 = context.GetFP32(operation.GetSource(0)); + SpvInstruction value1 = context.GetFP32(operation.GetSource(1)); + SpvInstruction vector = context.CompositeConstruct(context.TypeVector(context.TypeFP32(), 2), value0, value1); + SpvInstruction result = context.GlslPackHalf2x16(context.TypeU32(), vector); + + return new OperationResult(AggregateType.U32, result); + } + + private static OperationResult GenerateReciprocalSquareRoot(CodeGenContext context, AstOperation operation) + { + return GenerateUnary(context, operation, context.Delegates.GlslInverseSqrt, null); + } + + private static OperationResult GenerateReturn(CodeGenContext context, AstOperation operation) + { + context.MayHaveReturned = true; + + if (operation.SourcesCount != 0) + { + context.ReturnValue(context.Get(context.CurrentFunction.ReturnType, operation.GetSource(0))); + } + else + { + context.Return(); + } + + return OperationResult.Invalid; + } + + private static OperationResult GenerateRound(CodeGenContext context, AstOperation operation) + { + return GenerateUnary(context, operation, context.Delegates.GlslRoundEven, null); + } + + private static OperationResult GenerateShiftLeft(CodeGenContext context, AstOperation operation) + { + return GenerateBinaryS32(context, operation, context.Delegates.ShiftLeftLogical); + } + + private static OperationResult GenerateShiftRightS32(CodeGenContext context, AstOperation operation) + { + return GenerateBinaryS32(context, operation, context.Delegates.ShiftRightArithmetic); + } + + private static OperationResult GenerateShiftRightU32(CodeGenContext context, AstOperation operation) + { + return GenerateBinaryS32(context, operation, context.Delegates.ShiftRightLogical); + } + + private static OperationResult GenerateShuffle(CodeGenContext context, AstOperation operation) + { + SpvInstruction value = context.GetFP32(operation.GetSource(0)); + SpvInstruction index = context.GetU32(operation.GetSource(1)); + + SpvInstruction result = context.GroupNonUniformShuffle(context.TypeFP32(), context.Constant(context.TypeU32(), (int)Scope.Subgroup), value, index); + + return new OperationResult(AggregateType.FP32, result); + } + + private static OperationResult GenerateShuffleDown(CodeGenContext context, AstOperation operation) + { + SpvInstruction value = context.GetFP32(operation.GetSource(0)); + SpvInstruction index = context.GetU32(operation.GetSource(1)); + + SpvInstruction result = context.GroupNonUniformShuffleDown(context.TypeFP32(), context.Constant(context.TypeU32(), (int)Scope.Subgroup), value, index); + + return new OperationResult(AggregateType.FP32, result); + } + + private static OperationResult GenerateShuffleUp(CodeGenContext context, AstOperation operation) + { + SpvInstruction value = context.GetFP32(operation.GetSource(0)); + SpvInstruction index = context.GetU32(operation.GetSource(1)); + + SpvInstruction result = context.GroupNonUniformShuffleUp(context.TypeFP32(), context.Constant(context.TypeU32(), (int)Scope.Subgroup), value, index); + + return new OperationResult(AggregateType.FP32, result); + } + + private static OperationResult GenerateShuffleXor(CodeGenContext context, AstOperation operation) + { + SpvInstruction value = context.GetFP32(operation.GetSource(0)); + SpvInstruction index = context.GetU32(operation.GetSource(1)); + + SpvInstruction result = context.GroupNonUniformShuffleXor(context.TypeFP32(), context.Constant(context.TypeU32(), (int)Scope.Subgroup), value, index); + + return new OperationResult(AggregateType.FP32, result); + } + + private static OperationResult GenerateSine(CodeGenContext context, AstOperation operation) + { + return GenerateUnary(context, operation, context.Delegates.GlslSin, null); + } + + private static OperationResult GenerateSquareRoot(CodeGenContext context, AstOperation operation) + { + return GenerateUnary(context, operation, context.Delegates.GlslSqrt, null); + } + + private static OperationResult GenerateStore(CodeGenContext context, AstOperation operation) + { + return GenerateLoadOrStore(context, operation, isStore: true); + } + + private static OperationResult GenerateSubtract(CodeGenContext context, AstOperation operation) + { + return GenerateBinary(context, operation, context.Delegates.FSub, context.Delegates.ISub); + } + + private static OperationResult GenerateSwizzleAdd(CodeGenContext context, AstOperation operation) + { + SpvInstruction x = context.Get(AggregateType.FP32, operation.GetSource(0)); + SpvInstruction y = context.Get(AggregateType.FP32, operation.GetSource(1)); + SpvInstruction mask = context.Get(AggregateType.U32, operation.GetSource(2)); + + SpvInstruction v4float = context.TypeVector(context.TypeFP32(), 4); + SpvInstruction one = context.Constant(context.TypeFP32(), 1.0f); + SpvInstruction minusOne = context.Constant(context.TypeFP32(), -1.0f); + SpvInstruction zero = context.Constant(context.TypeFP32(), 0.0f); + SpvInstruction xLut = context.ConstantComposite(v4float, one, minusOne, one, zero); + SpvInstruction yLut = context.ConstantComposite(v4float, one, one, minusOne, one); + + SpvInstruction three = context.Constant(context.TypeU32(), 3); + + SpvInstruction threadId = GetScalarInput(context, IoVariable.SubgroupLaneId); + SpvInstruction shift = context.BitwiseAnd(context.TypeU32(), threadId, three); + shift = context.ShiftLeftLogical(context.TypeU32(), shift, context.Constant(context.TypeU32(), 1)); + SpvInstruction lutIdx = context.ShiftRightLogical(context.TypeU32(), mask, shift); + lutIdx = context.BitwiseAnd(context.TypeU32(), lutIdx, three); + + SpvInstruction xLutValue = context.VectorExtractDynamic(context.TypeFP32(), xLut, lutIdx); + SpvInstruction yLutValue = context.VectorExtractDynamic(context.TypeFP32(), yLut, lutIdx); + + SpvInstruction xResult = context.FMul(context.TypeFP32(), x, xLutValue); + SpvInstruction yResult = context.FMul(context.TypeFP32(), y, yLutValue); + SpvInstruction result = context.FAdd(context.TypeFP32(), xResult, yResult); + + return new OperationResult(AggregateType.FP32, result); + } + + private static OperationResult GenerateTextureSample(CodeGenContext context, AstOperation operation) + { + AstTextureOperation texOp = (AstTextureOperation)operation; + + bool isGather = (texOp.Flags & TextureFlags.Gather) != 0; + bool hasDerivatives = (texOp.Flags & TextureFlags.Derivatives) != 0; + bool intCoords = (texOp.Flags & TextureFlags.IntCoords) != 0; + bool hasLodBias = (texOp.Flags & TextureFlags.LodBias) != 0; + bool hasLodLevel = (texOp.Flags & TextureFlags.LodLevel) != 0; + bool hasOffset = (texOp.Flags & TextureFlags.Offset) != 0; + bool hasOffsets = (texOp.Flags & TextureFlags.Offsets) != 0; + + bool isArray = (texOp.Type & SamplerType.Array) != 0; + bool isMultisample = (texOp.Type & SamplerType.Multisample) != 0; + bool isShadow = (texOp.Type & SamplerType.Shadow) != 0; + + int srcIndex = 0; + + SpvInstruction Src(AggregateType type) + { + return context.Get(type, texOp.GetSource(srcIndex++)); + } + + SamplerDeclaration declaration = context.Samplers[texOp.GetTextureSetAndBinding()]; + SpvInstruction image = GenerateSampledImageLoad(context, texOp, declaration, ref srcIndex); + + int coordsCount = texOp.Type.Dimensions; + + int pCount = coordsCount; + + int arrayIndexElem = -1; + + if (isArray) + { + arrayIndexElem = pCount++; + } + + AggregateType coordType = intCoords ? AggregateType.S32 : AggregateType.FP32; + + SpvInstruction AssemblePVector(int count) + { + if (count > 1) + { + SpvInstruction[] elems = new SpvInstruction[count]; + + for (int index = 0; index < count; index++) + { + if (arrayIndexElem == index) + { + elems[index] = Src(AggregateType.S32); + + if (!intCoords) + { + elems[index] = context.ConvertSToF(context.TypeFP32(), elems[index]); + } + } + else + { + elems[index] = Src(coordType); + } + } + + SpvInstruction vectorType = context.TypeVector(intCoords ? context.TypeS32() : context.TypeFP32(), count); + return context.CompositeConstruct(vectorType, elems); + } + else + { + return Src(coordType); + } + } + + SpvInstruction pCoords = AssemblePVector(pCount); + + if (!intCoords) + { + SpvInstruction nfFp32 = context.TypeFP32(); + SpvInstruction nfType = pCount > 1 ? context.TypeVector(nfFp32, pCount) : nfFp32; + SpvInstruction nfZero = context.Constant(nfFp32, 0f); + + if (pCount > 1) + { + SpvInstruction[] nfElems = new SpvInstruction[pCount]; + for (int nfI = 0; nfI < pCount; nfI++) { nfElems[nfI] = nfZero; } + nfZero = context.ConstantComposite(nfType, nfElems); + } + + SpvInstruction nfBool = pCount > 1 ? context.TypeVector(context.TypeBool(), pCount) : context.TypeBool(); + pCoords = context.Select(nfType, context.IsNan(nfBool, pCoords), nfZero, pCoords); + } + + SpvInstruction AssembleDerivativesVector(int count) + { + if (count > 1) + { + SpvInstruction[] elems = new SpvInstruction[count]; + + for (int index = 0; index < count; index++) + { + elems[index] = Src(AggregateType.FP32); + } + + SpvInstruction vectorType = context.TypeVector(context.TypeFP32(), count); + return context.CompositeConstruct(vectorType, elems); + } + else + { + return Src(AggregateType.FP32); + } + } + + SpvInstruction dRef = null; + + if (isShadow) + { + dRef = Src(AggregateType.FP32); + } + + SpvInstruction[] derivatives = null; + + if (hasDerivatives) + { + derivatives = + [ + AssembleDerivativesVector(coordsCount), // dPdx + AssembleDerivativesVector(coordsCount) // dPdy + ]; + } + + SpvInstruction sample = null; + SpvInstruction lod = null; + + if (isMultisample) + { + sample = Src(AggregateType.S32); + } + else if (hasLodLevel) + { + lod = Src(coordType); + } + + SpvInstruction AssembleOffsetVector(int count) + { + if (count > 1) + { + SpvInstruction[] elems = new SpvInstruction[count]; + + for (int index = 0; index < count; index++) + { + elems[index] = Src(AggregateType.S32); + } + + SpvInstruction vectorType = context.TypeVector(context.TypeS32(), count); + + return context.ConstantComposite(vectorType, elems); + } + else + { + return Src(AggregateType.S32); + } + } + + SpvInstruction[] offsets = null; + + if (hasOffset) + { + offsets = [AssembleOffsetVector(coordsCount)]; + } + else if (hasOffsets) + { + offsets = + [ + AssembleOffsetVector(coordsCount), + AssembleOffsetVector(coordsCount), + AssembleOffsetVector(coordsCount), + AssembleOffsetVector(coordsCount) + ]; + } + + SpvInstruction lodBias = null; + + if (hasLodBias) + { + lodBias = Src(AggregateType.FP32); + } + + if (!isGather && !intCoords && !isMultisample && !hasLodLevel && !hasDerivatives && context.Definitions.Stage != ShaderStage.Fragment) + { + // Implicit LOD is only valid on fragment. + // Use the LOD bias as explicit LOD if available. + + lod = lodBias ?? context.Constant(context.TypeFP32(), 0f); + + lodBias = null; + hasLodBias = false; + hasLodLevel = true; + } + + SpvInstruction compIdx = null; + + // textureGather* optional extra component index, + // not needed for shadow samplers. + if (isGather && !isShadow) + { + compIdx = Src(AggregateType.S32); + } + + List operandsList = []; + ImageOperandsMask operandsMask = ImageOperandsMask.MaskNone; + + if (hasLodBias) + { + operandsMask |= ImageOperandsMask.Bias; + operandsList.Add(lodBias); + } + + if (!isMultisample && hasLodLevel) + { + operandsMask |= ImageOperandsMask.Lod; + operandsList.Add(lod); + } + + if (hasDerivatives) + { + operandsMask |= ImageOperandsMask.Grad; + operandsList.Add(derivatives[0]); + operandsList.Add(derivatives[1]); + } + + if (hasOffset) + { + operandsMask |= ImageOperandsMask.ConstOffset; + operandsList.Add(offsets[0]); + } + else if (hasOffsets) + { + operandsMask |= ImageOperandsMask.ConstOffsets; + SpvInstruction arrayv2 = context.TypeArray(context.TypeVector(context.TypeS32(), 2), context.Constant(context.TypeU32(), 4)); + operandsList.Add(context.ConstantComposite(arrayv2, offsets[0], offsets[1], offsets[2], offsets[3])); + } + + if (isMultisample) + { + operandsMask |= ImageOperandsMask.Sample; + operandsList.Add(sample); + } + + bool colorIsVector = isGather || !isShadow; + + SpvInstruction resultType = colorIsVector ? context.TypeVector(context.TypeFP32(), 4) : context.TypeFP32(); + + if (intCoords) + { + image = context.Image(declaration.ImageType, image); + } + + SpvInstruction[] operands = operandsList.ToArray(); + + SpvInstruction result; + + if (intCoords) + { + result = context.ImageFetch(resultType, image, pCoords, operandsMask, operands); + } + else if (isGather) + { + if (isShadow) + { + result = context.ImageDrefGather(resultType, image, pCoords, dRef, operandsMask, operands); + } + else + { + result = context.ImageGather(resultType, image, pCoords, compIdx, operandsMask, operands); + } + } + else if (isShadow) + { + if (hasLodLevel) + { + result = context.ImageSampleDrefExplicitLod(resultType, image, pCoords, dRef, operandsMask, operands); + } + else + { + result = context.ImageSampleDrefImplicitLod(resultType, image, pCoords, dRef, operandsMask, operands); + } + } + else if (hasDerivatives || hasLodLevel) + { + result = context.ImageSampleExplicitLod(resultType, image, pCoords, operandsMask, operands); + } + else + { + result = context.ImageSampleImplicitLod(resultType, image, pCoords, operandsMask, operands); + } + + AggregateType swizzledResultType = AggregateType.FP32; + + if (colorIsVector) + { + swizzledResultType = texOp.GetVectorType(swizzledResultType); + + result = GetSwizzledResult(context, result, swizzledResultType, texOp.Index); + } + + return new OperationResult(swizzledResultType, result); + } + + private static OperationResult GenerateTextureQuerySamples(CodeGenContext context, AstOperation operation) + { + AstTextureOperation texOp = (AstTextureOperation)operation; + + int srcIndex = 0; + + SamplerDeclaration declaration = context.Samplers[texOp.GetTextureSetAndBinding()]; + SpvInstruction image = GenerateSampledImageLoad(context, texOp, declaration, ref srcIndex); + + image = context.Image(declaration.ImageType, image); + + SpvInstruction result = context.ImageQuerySamples(context.TypeS32(), image); + + return new OperationResult(AggregateType.S32, result); + } + + private static OperationResult GenerateTextureQuerySize(CodeGenContext context, AstOperation operation) + { + AstTextureOperation texOp = (AstTextureOperation)operation; + + int srcIndex = 0; + + SamplerDeclaration declaration = context.Samplers[texOp.GetTextureSetAndBinding()]; + SpvInstruction image = GenerateSampledImageLoad(context, texOp, declaration, ref srcIndex); + + image = context.Image(declaration.ImageType, image); + + if (texOp.Index == 3) + { + return new OperationResult(AggregateType.S32, context.ImageQueryLevels(context.TypeS32(), image)); + } + else + { + SamplerType type = context.SamplersTypes[texOp.GetTextureSetAndBinding()]; + bool hasLod = !type.HasFlag(SamplerType.Multisample) && type != SamplerType.TextureBuffer; + + int dimensions = (type & SamplerType.Mask) == SamplerType.TextureCube ? 2 : type.Dimensions; + + if (type.HasFlag(SamplerType.Array)) + { + dimensions++; + } + + SpvInstruction resultType = dimensions == 1 ? context.TypeS32() : context.TypeVector(context.TypeS32(), dimensions); + + SpvInstruction result; + + if (hasLod) + { + SpvInstruction lod = context.GetS32(operation.GetSource(srcIndex)); + result = context.ImageQuerySizeLod(resultType, image, lod); + } + else + { + result = context.ImageQuerySize(resultType, image); + } + + if (dimensions != 1) + { + result = context.CompositeExtract(context.TypeS32(), result, texOp.Index); + } + + return new OperationResult(AggregateType.S32, result); + } + } + + private static OperationResult GenerateTruncate(CodeGenContext context, AstOperation operation) + { + return GenerateUnary(context, operation, context.Delegates.GlslTrunc, null); + } + + private static OperationResult GenerateUnpackDouble2x32(CodeGenContext context, AstOperation operation) + { + SpvInstruction value = context.GetFP64(operation.GetSource(0)); + SpvInstruction vector = context.GlslUnpackDouble2x32(context.TypeVector(context.TypeU32(), 2), value); + SpvInstruction result = context.CompositeExtract(context.TypeU32(), vector, operation.Index); + + return new OperationResult(AggregateType.U32, result); + } + + private static OperationResult GenerateUnpackHalf2x16(CodeGenContext context, AstOperation operation) + { + SpvInstruction value = context.GetU32(operation.GetSource(0)); + SpvInstruction vector = context.GlslUnpackHalf2x16(context.TypeVector(context.TypeFP32(), 2), value); + SpvInstruction result = context.CompositeExtract(context.TypeFP32(), vector, operation.Index); + + return new OperationResult(AggregateType.FP32, result); + } + + private static OperationResult GenerateVectorExtract(CodeGenContext context, AstOperation operation) + { + SpvInstruction vector = context.GetWithType(operation.GetSource(0), out AggregateType vectorType); + AggregateType scalarType = vectorType & ~AggregateType.ElementCountMask; + SpvInstruction resultType = context.GetType(scalarType); + SpvInstruction result; + + if (operation.GetSource(1) is AstOperand indexOperand && indexOperand.Type == OperandType.Constant) + { + result = context.CompositeExtract(resultType, vector, (SpvLiteralInteger)indexOperand.Value); + } + else + { + SpvInstruction index = context.Get(AggregateType.S32, operation.GetSource(1)); + result = context.VectorExtractDynamic(resultType, vector, index); + } + + return new OperationResult(scalarType, result); + } + + private static OperationResult GenerateVoteAll(CodeGenContext context, AstOperation operation) + { + SpvInstruction execution = context.Constant(context.TypeU32(), Scope.Subgroup); + SpvInstruction result = context.GroupNonUniformAll(context.TypeBool(), execution, context.Get(AggregateType.Bool, operation.GetSource(0))); + return new OperationResult(AggregateType.Bool, result); + } + + private static OperationResult GenerateVoteAllEqual(CodeGenContext context, AstOperation operation) + { + SpvInstruction execution = context.Constant(context.TypeU32(), Scope.Subgroup); + SpvInstruction result = context.GroupNonUniformAllEqual(context.TypeBool(), execution, context.Get(AggregateType.Bool, operation.GetSource(0))); + return new OperationResult(AggregateType.Bool, result); + } + + private static OperationResult GenerateVoteAny(CodeGenContext context, AstOperation operation) + { + SpvInstruction execution = context.Constant(context.TypeU32(), Scope.Subgroup); + SpvInstruction result = context.GroupNonUniformAny(context.TypeBool(), execution, context.Get(AggregateType.Bool, operation.GetSource(0))); + return new OperationResult(AggregateType.Bool, result); + } + + private static OperationResult GenerateCompare( + CodeGenContext context, + AstOperation operation, + Func emitF, + Func emitI) + { + IAstNode src1 = operation.GetSource(0); + IAstNode src2 = operation.GetSource(1); + + SpvInstruction result; + + if (operation.Inst.HasFlag(Instruction.FP64)) + { + result = emitF(context.TypeBool(), context.GetFP64(src1), context.GetFP64(src2)); + } + else if (operation.Inst.HasFlag(Instruction.FP32)) + { + result = emitF(context.TypeBool(), context.GetFP32(src1), context.GetFP32(src2)); + } + else + { + result = emitI(context.TypeBool(), context.GetS32(src1), context.GetS32(src2)); + } + + return new OperationResult(AggregateType.Bool, result); + } + + private static OperationResult GenerateCompareU32( + CodeGenContext context, + AstOperation operation, + Func emitU) + { + IAstNode src1 = operation.GetSource(0); + IAstNode src2 = operation.GetSource(1); + + SpvInstruction result = emitU(context.TypeBool(), context.GetU32(src1), context.GetU32(src2)); + + return new OperationResult(AggregateType.Bool, result); + } + + private static OperationResult GenerateAtomicMemoryBinary( + CodeGenContext context, + AstOperation operation, + Func emitU) + { + SpvInstruction elemPointer = GetStoragePointer(context, operation, out AggregateType varType); + + SpvInstruction value = context.Get(varType, operation.GetSource(operation.SourcesCount - 1)); + + SpvInstruction one = context.Constant(context.TypeU32(), 1); + SpvInstruction zero = context.Constant(context.TypeU32(), 0); + + return new OperationResult(varType, emitU(context.GetType(varType), elemPointer, one, zero, value)); + } + + private static OperationResult GenerateAtomicMemoryCas(CodeGenContext context, AstOperation operation) + { + SpvInstruction elemPointer = GetStoragePointer(context, operation, out AggregateType varType); + + SpvInstruction value0 = context.Get(varType, operation.GetSource(operation.SourcesCount - 2)); + SpvInstruction value1 = context.Get(varType, operation.GetSource(operation.SourcesCount - 1)); + + SpvInstruction one = context.Constant(context.TypeU32(), 1); + SpvInstruction zero = context.Constant(context.TypeU32(), 0); + + return new OperationResult(varType, context.AtomicCompareExchange(context.GetType(varType), elemPointer, one, zero, zero, value1, value0)); + } + + private static OperationResult GenerateLoadOrStore(CodeGenContext context, AstOperation operation, bool isStore) + { + SpvInstruction pointer = GetStoragePointer(context, operation, out AggregateType varType); + + if (isStore) + { + context.Store(pointer, context.Get(varType, operation.GetSource(operation.SourcesCount - 1))); + return OperationResult.Invalid; + } + else + { + SpvInstruction result = context.Load(context.GetType(varType), pointer); + return new OperationResult(varType, result); + } + } + + private static SpvInstruction GetStoragePointer(CodeGenContext context, AstOperation operation, out AggregateType varType) + { + StorageKind storageKind = operation.StorageKind; + + StorageClass storageClass; + SpvInstruction baseObj; + int srcIndex = 0; + IoVariable? perVertexBuiltIn = null; + + switch (storageKind) + { + case StorageKind.ConstantBuffer: + case StorageKind.StorageBuffer: + if (operation.GetSource(srcIndex++) is not AstOperand bindingIndex || bindingIndex.Type != OperandType.Constant) + { + throw new InvalidOperationException($"First input of {operation.Inst} with {storageKind} storage must be a constant operand."); + } + + if (operation.GetSource(srcIndex) is not AstOperand fieldIndex || fieldIndex.Type != OperandType.Constant) + { + throw new InvalidOperationException($"Second input of {operation.Inst} with {storageKind} storage must be a constant operand."); + } + + BufferDefinition buffer = storageKind == StorageKind.ConstantBuffer + ? context.Properties.ConstantBuffers[bindingIndex.Value] + : context.Properties.StorageBuffers[bindingIndex.Value]; + StructureField field = buffer.Type.Fields[fieldIndex.Value]; + + storageClass = StorageClass.Uniform; + varType = field.Type & AggregateType.ElementTypeMask; + baseObj = storageKind == StorageKind.ConstantBuffer + ? context.ConstantBuffers[bindingIndex.Value] + : context.StorageBuffers[bindingIndex.Value]; + break; + + case StorageKind.LocalMemory: + case StorageKind.SharedMemory: + if (operation.GetSource(srcIndex++) is not AstOperand { Type: OperandType.Constant } bindingId) + { + throw new InvalidOperationException($"First input of {operation.Inst} with {storageKind} storage must be a constant operand."); + } + + if (storageKind == StorageKind.LocalMemory) + { + storageClass = StorageClass.Private; + varType = context.Properties.LocalMemories[bindingId.Value].Type & AggregateType.ElementTypeMask; + baseObj = context.LocalMemories[bindingId.Value]; + } + else + { + storageClass = StorageClass.Workgroup; + varType = context.Properties.SharedMemories[bindingId.Value].Type & AggregateType.ElementTypeMask; + baseObj = context.SharedMemories[bindingId.Value]; + } + + break; + + case StorageKind.Input: + case StorageKind.InputPerPatch: + case StorageKind.Output: + case StorageKind.OutputPerPatch: + if (operation.GetSource(srcIndex++) is not AstOperand varId || varId.Type != OperandType.Constant) + { + throw new InvalidOperationException($"First input of {operation.Inst} with {storageKind} storage must be a constant operand."); + } + + IoVariable ioVariable = (IoVariable)varId.Value; + bool isOutput = storageKind.IsOutput(); + bool isPerPatch = storageKind.IsPerPatch(); + int location = 0; + int component = 0; + + if (context.Definitions.HasPerLocationInputOrOutput(ioVariable, isOutput)) + { + if (operation.GetSource(srcIndex++) is not AstOperand vecIndex || vecIndex.Type != OperandType.Constant) + { + throw new InvalidOperationException($"Second input of {operation.Inst} with {storageKind} storage must be a constant operand."); + } + + location = vecIndex.Value; + + if (operation.SourcesCount > srcIndex && + operation.GetSource(srcIndex) is AstOperand elemIndex && + elemIndex.Type == OperandType.Constant && + context.Definitions.HasPerLocationInputOrOutputComponent(ioVariable, location, elemIndex.Value, isOutput)) + { + component = elemIndex.Value; + srcIndex++; + } + } + + if (ioVariable == IoVariable.UserDefined) + { + varType = context.Definitions.GetUserDefinedType(location, isOutput); + } + else if (ioVariable == IoVariable.FragmentOutputColor) + { + varType = context.Definitions.GetFragmentOutputColorType(location); + } + else + { + (_, varType) = IoMap.GetSpirvBuiltIn(ioVariable); + + if (IoMap.IsPerVertexBuiltIn(ioVariable)) + { + perVertexBuiltIn = ioVariable; + ioVariable = IoVariable.Position; + } + } + + varType &= AggregateType.ElementTypeMask; + + storageClass = isOutput ? StorageClass.Output : StorageClass.Input; + + IoDefinition ioDefinition = new(storageKind, ioVariable, location, component); + Dictionary dict = isPerPatch + ? (isOutput ? context.OutputsPerPatch : context.InputsPerPatch) + : (isOutput ? context.Outputs : context.Inputs); + + baseObj = dict[ioDefinition]; + break; + + default: + throw new InvalidOperationException($"Invalid storage kind {storageKind}."); + } + + bool isStoreOrAtomic = operation.Inst == Instruction.Store || operation.Inst.IsAtomic(); + int inputsCount = (isStoreOrAtomic ? operation.SourcesCount - 1 : operation.SourcesCount) - srcIndex; + + if (perVertexBuiltIn.HasValue) + { + int fieldIndex = IoMap.GetPerVertexStructFieldIndex(perVertexBuiltIn.Value); + + SpvInstruction[] indexes = new SpvInstruction[inputsCount + 1]; + int index = 0; + + if (IoMap.IsPerVertexArrayBuiltIn(storageKind, context.Definitions.Stage)) + { + indexes[index++] = context.Get(AggregateType.S32, operation.GetSource(srcIndex++)); + indexes[index++] = context.Constant(context.TypeS32(), fieldIndex); + } + else + { + indexes[index++] = context.Constant(context.TypeS32(), fieldIndex); + } + + for (; index < inputsCount + 1; srcIndex++, index++) + { + indexes[index] = context.Get(AggregateType.S32, operation.GetSource(srcIndex)); + } + + return context.AccessChain(context.TypePointer(storageClass, context.GetType(varType)), baseObj, indexes); + } + + if (operation.Inst == Instruction.AtomicCompareAndSwap) + { + inputsCount--; + } + + SpvInstruction e0, e1, e2; + SpvInstruction pointer; + + switch (inputsCount) + { + case 0: + pointer = baseObj; + break; + case 1: + e0 = context.Get(AggregateType.S32, operation.GetSource(srcIndex++)); + pointer = context.AccessChain(context.TypePointer(storageClass, context.GetType(varType)), baseObj, e0); + break; + case 2: + e0 = context.Get(AggregateType.S32, operation.GetSource(srcIndex++)); + e1 = context.Get(AggregateType.S32, operation.GetSource(srcIndex++)); + pointer = context.AccessChain(context.TypePointer(storageClass, context.GetType(varType)), baseObj, e0, e1); + break; + case 3: + e0 = context.Get(AggregateType.S32, operation.GetSource(srcIndex++)); + e1 = context.Get(AggregateType.S32, operation.GetSource(srcIndex++)); + e2 = context.Get(AggregateType.S32, operation.GetSource(srcIndex++)); + pointer = context.AccessChain(context.TypePointer(storageClass, context.GetType(varType)), baseObj, e0, e1, e2); + break; + default: + SpvInstruction[] indexes = new SpvInstruction[inputsCount]; + int index = 0; + + for (; index < inputsCount; srcIndex++, index++) + { + indexes[index] = context.Get(AggregateType.S32, operation.GetSource(srcIndex)); + } + + pointer = context.AccessChain(context.TypePointer(storageClass, context.GetType(varType)), baseObj, indexes); + break; + } + + return pointer; + } + + private static SpvInstruction GetScalarInput(CodeGenContext context, IoVariable ioVariable) + { + (_, AggregateType varType) = IoMap.GetSpirvBuiltIn(ioVariable); + varType &= AggregateType.ElementTypeMask; + + IoDefinition ioDefinition = new(StorageKind.Input, ioVariable); + + return context.Load(context.GetType(varType), context.Inputs[ioDefinition]); + } + + private static SpvInstruction GetSwizzledResult(CodeGenContext context, SpvInstruction vector, AggregateType swizzledResultType, int mask) + { + if ((swizzledResultType & AggregateType.ElementCountMask) != 0) + { + SpvLiteralInteger[] components = new SpvLiteralInteger[BitOperations.PopCount((uint)mask)]; + + int componentIndex = 0; + + for (int i = 0; i < 4; i++) + { + if ((mask & (1 << i)) != 0) + { + components[componentIndex++] = i; + } + } + + return context.VectorShuffle(context.GetType(swizzledResultType), vector, vector, components); + } + else + { + int componentIndex = (int)BitOperations.TrailingZeroCount(mask); + + return context.CompositeExtract(context.GetType(swizzledResultType), vector, (SpvLiteralInteger)componentIndex); + } + } + + private static SpvInstruction GenerateSampledImageLoad(CodeGenContext context, AstTextureOperation texOp, SamplerDeclaration declaration, ref int srcIndex) + { + SpvInstruction image = declaration.Image; + bool imageIndexed = declaration.IsIndexed; + + if (imageIndexed) + { + SpvInstruction textureIndex = context.Get(AggregateType.S32, texOp.GetSource(srcIndex++)); + + image = context.AccessChain(declaration.SampledImagePointerType, image, textureIndex); + MarkNonUniform(context, image); + } + + if (texOp.IsSeparate) + { + image = context.Load(declaration.ImageType, image); + if (imageIndexed) + { + MarkNonUniform(context, image); + } + + SamplerDeclaration samplerDeclaration = context.Samplers[texOp.GetSamplerSetAndBinding()]; + + SpvInstruction sampler = samplerDeclaration.Image; + bool samplerIndexed = samplerDeclaration.IsIndexed; + + if (samplerIndexed) + { + SpvInstruction samplerIndex = context.Get(AggregateType.S32, texOp.GetSource(srcIndex++)); + + sampler = context.AccessChain(samplerDeclaration.SampledImagePointerType, sampler, samplerIndex); + MarkNonUniform(context, sampler); + } + + sampler = context.Load(samplerDeclaration.ImageType, sampler); + if (samplerIndexed) + { + MarkNonUniform(context, sampler); + } + + image = context.SampledImage(declaration.SampledImageType, image, sampler); + if (imageIndexed || samplerIndexed) + { + MarkNonUniform(context, image); + } + } + else + { + image = context.Load(declaration.SampledImageType, image); + if (imageIndexed) + { + MarkNonUniform(context, image); + } + } + + return image; + } + + private static OperationResult GenerateUnary( + CodeGenContext context, + AstOperation operation, + Func emitF, + Func emitI) + { + IAstNode source = operation.GetSource(0); + + if (operation.Inst.HasFlag(Instruction.FP64)) + { + return new OperationResult(AggregateType.FP64, emitF(context.TypeFP64(), context.GetFP64(source))); + } + else if (operation.Inst.HasFlag(Instruction.FP32)) + { + return new OperationResult(AggregateType.FP32, emitF(context.TypeFP32(), context.GetFP32(source))); + } + else + { + return new OperationResult(AggregateType.S32, emitI(context.TypeS32(), context.GetS32(source))); + } + } + + private static OperationResult GenerateUnaryBool( + CodeGenContext context, + AstOperation operation, + Func emitB) + { + IAstNode source = operation.GetSource(0); + return new OperationResult(AggregateType.Bool, emitB(context.TypeBool(), context.Get(AggregateType.Bool, source))); + } + + private static OperationResult GenerateUnaryFP32( + CodeGenContext context, + AstOperation operation, + Func emit) + { + IAstNode source = operation.GetSource(0); + return new OperationResult(AggregateType.FP32, emit(context.TypeFP32(), context.GetFP32(source))); + } + + private static OperationResult GenerateUnaryS32( + CodeGenContext context, + AstOperation operation, + Func emitS) + { + IAstNode source = operation.GetSource(0); + return new OperationResult(AggregateType.S32, emitS(context.TypeS32(), context.GetS32(source))); + } + + private static OperationResult GenerateBinary( + CodeGenContext context, + AstOperation operation, + Func emitF, + Func emitI) + { + IAstNode src1 = operation.GetSource(0); + IAstNode src2 = operation.GetSource(1); + + if (operation.Inst.HasFlag(Instruction.FP64)) + { + SpvInstruction result = emitF(context.TypeFP64(), context.GetFP64(src1), context.GetFP64(src2)); + + if (!context.HostCapabilities.ReducedPrecision || operation.ForcePrecise) + { + context.Decorate(result, Decoration.NoContraction); + } + + return new OperationResult(AggregateType.FP64, result); + } + else if (operation.Inst.HasFlag(Instruction.FP32)) + { + SpvInstruction result = emitF(context.TypeFP32(), context.GetFP32(src1), context.GetFP32(src2)); + + if (!context.HostCapabilities.ReducedPrecision || operation.ForcePrecise) + { + context.Decorate(result, Decoration.NoContraction); + } + + return new OperationResult(AggregateType.FP32, result); + } + else + { + return new OperationResult(AggregateType.S32, emitI(context.TypeS32(), context.GetS32(src1), context.GetS32(src2))); + } + } + + private static OperationResult GenerateBinaryBool( + CodeGenContext context, + AstOperation operation, + Func emitB) + { + IAstNode src1 = operation.GetSource(0); + IAstNode src2 = operation.GetSource(1); + + return new OperationResult(AggregateType.Bool, emitB(context.TypeBool(), context.Get(AggregateType.Bool, src1), context.Get(AggregateType.Bool, src2))); + } + + private static OperationResult GenerateBinaryS32( + CodeGenContext context, + AstOperation operation, + Func emitS) + { + IAstNode src1 = operation.GetSource(0); + IAstNode src2 = operation.GetSource(1); + + return new OperationResult(AggregateType.S32, emitS(context.TypeS32(), context.GetS32(src1), context.GetS32(src2))); + } + + private static OperationResult GenerateBinaryU32( + CodeGenContext context, + AstOperation operation, + Func emitU) + { + IAstNode src1 = operation.GetSource(0); + IAstNode src2 = operation.GetSource(1); + + return new OperationResult(AggregateType.U32, emitU(context.TypeU32(), context.GetU32(src1), context.GetU32(src2))); + } + + private static OperationResult GenerateTernary( + CodeGenContext context, + AstOperation operation, + Func emitF, + Func emitI) + { + IAstNode src1 = operation.GetSource(0); + IAstNode src2 = operation.GetSource(1); + IAstNode src3 = operation.GetSource(2); + + if (operation.Inst.HasFlag(Instruction.FP64)) + { + SpvInstruction result = emitF(context.TypeFP64(), context.GetFP64(src1), context.GetFP64(src2), context.GetFP64(src3)); + + if (!context.HostCapabilities.ReducedPrecision || operation.ForcePrecise) + { + context.Decorate(result, Decoration.NoContraction); + } + + return new OperationResult(AggregateType.FP64, result); + } + else if (operation.Inst.HasFlag(Instruction.FP32)) + { + SpvInstruction result = emitF(context.TypeFP32(), context.GetFP32(src1), context.GetFP32(src2), context.GetFP32(src3)); + + if (!context.HostCapabilities.ReducedPrecision || operation.ForcePrecise) + { + context.Decorate(result, Decoration.NoContraction); + } + + return new OperationResult(AggregateType.FP32, result); + } + else + { + return new OperationResult(AggregateType.S32, emitI(context.TypeS32(), context.GetS32(src1), context.GetS32(src2), context.GetS32(src3))); + } + } + + private static OperationResult GenerateTernaryU32( + CodeGenContext context, + AstOperation operation, + Func emitU) + { + IAstNode src1 = operation.GetSource(0); + IAstNode src2 = operation.GetSource(1); + IAstNode src3 = operation.GetSource(2); + + return new OperationResult(AggregateType.U32, emitU( + context.TypeU32(), + context.GetU32(src1), + context.GetU32(src2), + context.GetU32(src3))); + } + + private static OperationResult GenerateBitfieldExtractS32( + CodeGenContext context, + AstOperation operation, + Func emitS) + { + IAstNode src1 = operation.GetSource(0); + IAstNode src2 = operation.GetSource(1); + IAstNode src3 = operation.GetSource(2); + + return new OperationResult(AggregateType.S32, emitS( + context.TypeS32(), + context.GetS32(src1), + context.GetU32(src2), + context.GetU32(src3))); + } + + private static OperationResult GenerateBitfieldInsert( + CodeGenContext context, + AstOperation operation, + Func emitS) + { + IAstNode src1 = operation.GetSource(0); + IAstNode src2 = operation.GetSource(1); + IAstNode src3 = operation.GetSource(2); + IAstNode src4 = operation.GetSource(3); + + return new OperationResult(AggregateType.U32, emitS( + context.TypeU32(), + context.GetU32(src1), + context.GetU32(src2), + context.GetU32(src3), + context.GetU32(src4))); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/IoMap.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/IoMap.cs new file mode 100644 index 0000000..2f991dd --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/IoMap.cs @@ -0,0 +1,124 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using System; +using static Spv.Specification; + +namespace Ryujinx.Graphics.Shader.CodeGen.Spirv +{ + static class IoMap + { + // At least 16 attributes are guaranteed by the spec. + private const int MaxAttributes = 16; + + public static (BuiltIn, AggregateType) GetSpirvBuiltIn(IoVariable ioVariable) + { + return ioVariable switch + { + IoVariable.BaseInstance => (BuiltIn.BaseInstance, AggregateType.S32), + IoVariable.BaseVertex => (BuiltIn.BaseVertex, AggregateType.S32), + IoVariable.ClipDistance => (BuiltIn.ClipDistance, AggregateType.Array | AggregateType.FP32), + IoVariable.CtaId => (BuiltIn.WorkgroupId, AggregateType.Vector3 | AggregateType.U32), + IoVariable.DrawIndex => (BuiltIn.DrawIndex, AggregateType.S32), + IoVariable.FragmentCoord => (BuiltIn.FragCoord, AggregateType.Vector4 | AggregateType.FP32), + IoVariable.FragmentOutputDepth => (BuiltIn.FragDepth, AggregateType.FP32), + IoVariable.FrontFacing => (BuiltIn.FrontFacing, AggregateType.Bool), + IoVariable.GlobalId => (BuiltIn.GlobalInvocationId, AggregateType.Vector3 | AggregateType.U32), + IoVariable.InstanceId => (BuiltIn.InstanceId, AggregateType.S32), + IoVariable.InstanceIndex => (BuiltIn.InstanceIndex, AggregateType.S32), + IoVariable.InvocationId => (BuiltIn.InvocationId, AggregateType.S32), + IoVariable.Layer => (BuiltIn.Layer, AggregateType.S32), + IoVariable.PatchVertices => (BuiltIn.PatchVertices, AggregateType.S32), + IoVariable.PointCoord => (BuiltIn.PointCoord, AggregateType.Vector2 | AggregateType.FP32), + IoVariable.PointSize => (BuiltIn.PointSize, AggregateType.FP32), + IoVariable.Position => (BuiltIn.Position, AggregateType.Vector4 | AggregateType.FP32), + IoVariable.PrimitiveId => (BuiltIn.PrimitiveId, AggregateType.S32), + IoVariable.SubgroupEqMask => (BuiltIn.SubgroupEqMask, AggregateType.Vector4 | AggregateType.U32), + IoVariable.SubgroupGeMask => (BuiltIn.SubgroupGeMask, AggregateType.Vector4 | AggregateType.U32), + IoVariable.SubgroupGtMask => (BuiltIn.SubgroupGtMask, AggregateType.Vector4 | AggregateType.U32), + IoVariable.SubgroupLaneId => (BuiltIn.SubgroupLocalInvocationId, AggregateType.U32), + IoVariable.SubgroupLeMask => (BuiltIn.SubgroupLeMask, AggregateType.Vector4 | AggregateType.U32), + IoVariable.SubgroupLtMask => (BuiltIn.SubgroupLtMask, AggregateType.Vector4 | AggregateType.U32), + IoVariable.TessellationCoord => (BuiltIn.TessCoord, AggregateType.Vector3 | AggregateType.FP32), + IoVariable.TessellationLevelInner => (BuiltIn.TessLevelInner, AggregateType.Array | AggregateType.FP32), + IoVariable.TessellationLevelOuter => (BuiltIn.TessLevelOuter, AggregateType.Array | AggregateType.FP32), + IoVariable.ThreadId => (BuiltIn.LocalInvocationId, AggregateType.Vector3 | AggregateType.U32), + IoVariable.ThreadKill => (BuiltIn.HelperInvocation, AggregateType.Bool), + IoVariable.VertexId => (BuiltIn.VertexId, AggregateType.S32), + IoVariable.VertexIndex => (BuiltIn.VertexIndex, AggregateType.S32), + IoVariable.ViewportIndex => (BuiltIn.ViewportIndex, AggregateType.S32), + IoVariable.ViewportMask => (BuiltIn.ViewportMaskNV, AggregateType.Array | AggregateType.S32), + _ => (default, AggregateType.Invalid), + }; + } + + public static int GetSpirvBuiltInArrayLength(IoVariable ioVariable) + { + return ioVariable switch + { + IoVariable.ClipDistance => 8, + IoVariable.TessellationLevelInner => 2, + IoVariable.TessellationLevelOuter => 4, + IoVariable.ViewportMask => 1, + IoVariable.UserDefined => MaxAttributes, + _ => 1, + }; + } + + public static bool IsPerVertex(IoVariable ioVariable, ShaderStage stage, bool isOutput) + { + switch (ioVariable) + { + case IoVariable.Layer: + case IoVariable.ViewportIndex: + case IoVariable.PointSize: + case IoVariable.Position: + case IoVariable.UserDefined: + case IoVariable.ClipDistance: + case IoVariable.PointCoord: + case IoVariable.ViewportMask: + return !isOutput && + stage is ShaderStage.TessellationControl or ShaderStage.TessellationEvaluation or ShaderStage.Geometry; + } + + return false; + } + + public static bool IsPerVertexBuiltIn(IoVariable ioVariable) + { + switch (ioVariable) + { + case IoVariable.Position: + case IoVariable.PointSize: + case IoVariable.ClipDistance: + return true; + } + + return false; + } + + public static bool IsPerVertexArrayBuiltIn(StorageKind storageKind, ShaderStage stage) + { + if (storageKind == StorageKind.Output) + { + return stage == ShaderStage.TessellationControl; + } + else + { + return stage is ShaderStage.TessellationControl or + ShaderStage.TessellationEvaluation or + ShaderStage.Geometry; + } + } + + public static int GetPerVertexStructFieldIndex(IoVariable ioVariable) + { + return ioVariable switch + { + IoVariable.Position => 0, + IoVariable.PointSize => 1, + IoVariable.ClipDistance => 2, + _ => throw new ArgumentException($"Invalid built-in variable {ioVariable}.") + }; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/OperationResult.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/OperationResult.cs new file mode 100644 index 0000000..d0381ec --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/OperationResult.cs @@ -0,0 +1,19 @@ +using Ryujinx.Graphics.Shader.Translation; +using Spv.Generator; + +namespace Ryujinx.Graphics.Shader.CodeGen.Spirv +{ + readonly struct OperationResult + { + public static OperationResult Invalid => new(AggregateType.Invalid, null); + + public AggregateType Type { get; } + public Instruction Value { get; } + + public OperationResult(AggregateType type, Instruction value) + { + Type = type; + Value = value; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/SamplerDeclaration.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/SamplerDeclaration.cs new file mode 100644 index 0000000..4ba282e --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/SamplerDeclaration.cs @@ -0,0 +1,27 @@ +using Spv.Generator; + +namespace Ryujinx.Graphics.Shader.CodeGen.Spirv +{ + readonly struct SamplerDeclaration + { + public readonly Instruction ImageType; + public readonly Instruction SampledImageType; + public readonly Instruction SampledImagePointerType; + public readonly Instruction Image; + public readonly bool IsIndexed; + + public SamplerDeclaration( + Instruction imageType, + Instruction sampledImageType, + Instruction sampledImagePointerType, + Instruction image, + bool isIndexed) + { + ImageType = imageType; + SampledImageType = sampledImageType; + SampledImagePointerType = sampledImagePointerType; + Image = image; + IsIndexed = isIndexed; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/SpirvDelegates.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/SpirvDelegates.cs new file mode 100644 index 0000000..3fbe5a4 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/SpirvDelegates.cs @@ -0,0 +1,228 @@ +using FuncBinaryInstruction = System.Func; +using FuncQuaternaryInstruction = System.Func; +using FuncTernaryInstruction = System.Func; +using FuncUnaryInstruction = System.Func; + +namespace Ryujinx.Graphics.Shader.CodeGen.Spirv +{ + /// + /// Delegate cache for SPIR-V instruction generators. Avoids delegate allocation when passing generators as arguments. + /// + internal readonly struct SpirvDelegates + { + // Unary + public readonly FuncUnaryInstruction GlslFAbs; + public readonly FuncUnaryInstruction GlslSAbs; + public readonly FuncUnaryInstruction GlslCeil; + public readonly FuncUnaryInstruction GlslCos; + public readonly FuncUnaryInstruction GlslExp2; + public readonly FuncUnaryInstruction GlslFloor; + public readonly FuncUnaryInstruction GlslLog2; + public readonly FuncUnaryInstruction FNegate; + public readonly FuncUnaryInstruction SNegate; + public readonly FuncUnaryInstruction GlslInverseSqrt; + public readonly FuncUnaryInstruction GlslRoundEven; + public readonly FuncUnaryInstruction GlslSin; + public readonly FuncUnaryInstruction GlslSqrt; + public readonly FuncUnaryInstruction GlslTrunc; + + // UnaryBool + public readonly FuncUnaryInstruction LogicalNot; + + // UnaryFP32 + public readonly FuncUnaryInstruction DPdx; + public readonly FuncUnaryInstruction DPdy; + + // UnaryS32 + public readonly FuncUnaryInstruction BitCount; + public readonly FuncUnaryInstruction BitReverse; + public readonly FuncUnaryInstruction Not; + + // Compare + public readonly FuncBinaryInstruction FOrdEqual; + public readonly FuncBinaryInstruction IEqual; + public readonly FuncBinaryInstruction FOrdGreaterThan; + public readonly FuncBinaryInstruction SGreaterThan; + public readonly FuncBinaryInstruction FOrdGreaterThanEqual; + public readonly FuncBinaryInstruction SGreaterThanEqual; + public readonly FuncBinaryInstruction FOrdLessThan; + public readonly FuncBinaryInstruction SLessThan; + public readonly FuncBinaryInstruction FOrdLessThanEqual; + public readonly FuncBinaryInstruction SLessThanEqual; + public readonly FuncBinaryInstruction FOrdNotEqual; + public readonly FuncBinaryInstruction INotEqual; + + // CompareU32 + public readonly FuncBinaryInstruction UGreaterThanEqual; + public readonly FuncBinaryInstruction UGreaterThan; + public readonly FuncBinaryInstruction ULessThanEqual; + public readonly FuncBinaryInstruction ULessThan; + + // Binary + public readonly FuncBinaryInstruction FAdd; + public readonly FuncBinaryInstruction IAdd; + public readonly FuncBinaryInstruction FDiv; + public readonly FuncBinaryInstruction SDiv; + public readonly FuncBinaryInstruction GlslFMax; + public readonly FuncBinaryInstruction GlslSMax; + public readonly FuncBinaryInstruction GlslFMin; + public readonly FuncBinaryInstruction GlslSMin; + public readonly FuncBinaryInstruction FMod; + public readonly FuncBinaryInstruction FMul; + public readonly FuncBinaryInstruction IMul; + public readonly FuncBinaryInstruction FSub; + public readonly FuncBinaryInstruction ISub; + + // BinaryBool + public readonly FuncBinaryInstruction LogicalAnd; + public readonly FuncBinaryInstruction LogicalNotEqual; + public readonly FuncBinaryInstruction LogicalOr; + + // BinaryS32 + public readonly FuncBinaryInstruction BitwiseAnd; + public readonly FuncBinaryInstruction BitwiseXor; + public readonly FuncBinaryInstruction BitwiseOr; + public readonly FuncBinaryInstruction ShiftLeftLogical; + public readonly FuncBinaryInstruction ShiftRightArithmetic; + public readonly FuncBinaryInstruction ShiftRightLogical; + + // BinaryU32 + public readonly FuncBinaryInstruction GlslUMax; + public readonly FuncBinaryInstruction GlslUMin; + + // AtomicMemoryBinary + public readonly FuncQuaternaryInstruction AtomicIAdd; + public readonly FuncQuaternaryInstruction AtomicAnd; + public readonly FuncQuaternaryInstruction AtomicSMin; + public readonly FuncQuaternaryInstruction AtomicUMin; + public readonly FuncQuaternaryInstruction AtomicSMax; + public readonly FuncQuaternaryInstruction AtomicUMax; + public readonly FuncQuaternaryInstruction AtomicOr; + public readonly FuncQuaternaryInstruction AtomicExchange; + public readonly FuncQuaternaryInstruction AtomicXor; + + // Ternary + public readonly FuncTernaryInstruction GlslFClamp; + public readonly FuncTernaryInstruction GlslSClamp; + public readonly FuncTernaryInstruction GlslFma; + + // TernaryS32 + public readonly FuncTernaryInstruction BitFieldSExtract; + public readonly FuncTernaryInstruction BitFieldUExtract; + + // TernaryU32 + public readonly FuncTernaryInstruction GlslUClamp; + + // QuaternaryS32 + public readonly FuncQuaternaryInstruction BitFieldInsert; + + public SpirvDelegates(CodeGenContext context) + { + // Unary + GlslFAbs = context.GlslFAbs; + GlslSAbs = context.GlslSAbs; + GlslCeil = context.GlslCeil; + GlslCos = context.GlslCos; + GlslExp2 = context.GlslExp2; + GlslFloor = context.GlslFloor; + GlslLog2 = context.GlslLog2; + FNegate = context.FNegate; + SNegate = context.SNegate; + GlslInverseSqrt = context.GlslInverseSqrt; + GlslRoundEven = context.GlslRoundEven; + GlslSin = context.GlslSin; + GlslSqrt = context.GlslSqrt; + GlslTrunc = context.GlslTrunc; + + // UnaryBool + LogicalNot = context.LogicalNot; + + // UnaryFP32 + DPdx = context.DPdx; + DPdy = context.DPdy; + + // UnaryS32 + BitCount = context.BitCount; + BitReverse = context.BitReverse; + Not = context.Not; + + // Compare + FOrdEqual = context.FOrdEqual; + IEqual = context.IEqual; + FOrdGreaterThan = context.FOrdGreaterThan; + SGreaterThan = context.SGreaterThan; + FOrdGreaterThanEqual = context.FOrdGreaterThanEqual; + SGreaterThanEqual = context.SGreaterThanEqual; + FOrdLessThan = context.FOrdLessThan; + SLessThan = context.SLessThan; + FOrdLessThanEqual = context.FOrdLessThanEqual; + SLessThanEqual = context.SLessThanEqual; + FOrdNotEqual = context.FOrdNotEqual; + INotEqual = context.INotEqual; + + // CompareU32 + UGreaterThanEqual = context.UGreaterThanEqual; + UGreaterThan = context.UGreaterThan; + ULessThanEqual = context.ULessThanEqual; + ULessThan = context.ULessThan; + + // Binary + FAdd = context.FAdd; + IAdd = context.IAdd; + FDiv = context.FDiv; + SDiv = context.SDiv; + GlslFMax = context.GlslFMax; + GlslSMax = context.GlslSMax; + GlslFMin = context.GlslFMin; + GlslSMin = context.GlslSMin; + FMod = context.FMod; + FMul = context.FMul; + IMul = context.IMul; + FSub = context.FSub; + ISub = context.ISub; + + // BinaryBool + LogicalAnd = context.LogicalAnd; + LogicalNotEqual = context.LogicalNotEqual; + LogicalOr = context.LogicalOr; + + // BinaryS32 + BitwiseAnd = context.BitwiseAnd; + BitwiseXor = context.BitwiseXor; + BitwiseOr = context.BitwiseOr; + ShiftLeftLogical = context.ShiftLeftLogical; + ShiftRightArithmetic = context.ShiftRightArithmetic; + ShiftRightLogical = context.ShiftRightLogical; + + // BinaryU32 + GlslUMax = context.GlslUMax; + GlslUMin = context.GlslUMin; + + // AtomicMemoryBinary + AtomicIAdd = context.AtomicIAdd; + AtomicAnd = context.AtomicAnd; + AtomicSMin = context.AtomicSMin; + AtomicUMin = context.AtomicUMin; + AtomicSMax = context.AtomicSMax; + AtomicUMax = context.AtomicUMax; + AtomicOr = context.AtomicOr; + AtomicExchange = context.AtomicExchange; + AtomicXor = context.AtomicXor; + + // Ternary + GlslFClamp = context.GlslFClamp; + GlslSClamp = context.GlslSClamp; + GlslFma = context.GlslFma; + + // TernaryS32 + BitFieldSExtract = context.BitFieldSExtract; + BitFieldUExtract = context.BitFieldUExtract; + + // TernaryU32 + GlslUClamp = context.GlslUClamp; + + // QuaternaryS32 + BitFieldInsert = context.BitFieldInsert; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/SpirvGenerator.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/SpirvGenerator.cs new file mode 100644 index 0000000..769ab89 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/SpirvGenerator.cs @@ -0,0 +1,425 @@ +using Ryujinx.Common; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.StructuredIr; +using Ryujinx.Graphics.Shader.Translation; +using System; +using System.Collections.Generic; +using System.Threading; +using static Spv.Specification; + +namespace Ryujinx.Graphics.Shader.CodeGen.Spirv +{ + using SpvInstruction = Spv.Generator.Instruction; + using SpvInstructionPool = Spv.Generator.GeneratorPool; + using SpvLiteralInteger = Spv.Generator.LiteralInteger; + using SpvLiteralIntegerPool = Spv.Generator.GeneratorPool; + + static class SpirvGenerator + { + // Resource pools for Spirv generation. Note: Increase count when more threads are being used. + private const int GeneratorPoolCount = 1; + private static readonly ObjectPool _instructionPool; + private static readonly ObjectPool _integerPool; + private static readonly Lock _poolLock = new(); + + static SpirvGenerator() + { + _instructionPool = new(() => new SpvInstructionPool(), GeneratorPoolCount); + _integerPool = new(() => new SpvLiteralIntegerPool(), GeneratorPoolCount); + } + + private const HelperFunctionsMask NeedsInvocationIdMask = HelperFunctionsMask.SwizzleAdd; + + public static byte[] Generate(StructuredProgramInfo info, CodeGenParameters parameters) + { + SpvInstructionPool instPool; + SpvLiteralIntegerPool integerPool; + + lock (_poolLock) + { + instPool = _instructionPool.Allocate(); + integerPool = _integerPool.Allocate(); + } + + CodeGenContext context = new(info, parameters, instPool, integerPool); + + context.AddCapability(Capability.Shader); + + context.SetMemoryModel(AddressingModel.Logical, MemoryModel.GLSL450); + + context.AddCapability(Capability.GroupNonUniformBallot); + context.AddCapability(Capability.GroupNonUniformShuffle); + context.AddCapability(Capability.GroupNonUniformVote); + context.AddCapability(Capability.ImageBuffer); + context.AddCapability(Capability.ImageGatherExtended); + context.AddCapability(Capability.ImageQuery); + context.AddCapability(Capability.SampledBuffer); + + if (parameters.HostCapabilities.SupportsShaderFloat64) + { + context.AddCapability(Capability.Float64); + } + + if (parameters.Definitions.TransformFeedbackEnabled && parameters.Definitions.LastInVertexPipeline) + { + context.AddCapability(Capability.TransformFeedback); + } + + if (parameters.Definitions.Stage == ShaderStage.Fragment) + { + if (context.Info.IoDefinitions.Contains(new IoDefinition(StorageKind.Input, IoVariable.Layer)) || + context.Info.IoDefinitions.Contains(new IoDefinition(StorageKind.Input, IoVariable.PrimitiveId))) + { + context.AddCapability(Capability.Geometry); + } + + if (context.HostCapabilities.SupportsFragmentShaderInterlock) + { + context.AddCapability(Capability.FragmentShaderPixelInterlockEXT); + context.AddExtension("SPV_EXT_fragment_shader_interlock"); + } + } + else if (parameters.Definitions.Stage == ShaderStage.Geometry) + { + context.AddCapability(Capability.Geometry); + + if (parameters.Definitions.GpPassthrough && context.HostCapabilities.SupportsGeometryShaderPassthrough) + { + context.AddExtension("SPV_NV_geometry_shader_passthrough"); + context.AddCapability(Capability.GeometryShaderPassthroughNV); + } + } + else if (parameters.Definitions.Stage is ShaderStage.TessellationControl or + ShaderStage.TessellationEvaluation) + { + context.AddCapability(Capability.Tessellation); + } + else if (parameters.Definitions.Stage == ShaderStage.Vertex) + { + context.AddCapability(Capability.DrawParameters); + } + + if (context.Definitions.Stage != ShaderStage.Fragment && + context.Definitions.Stage != ShaderStage.Geometry && + context.Definitions.Stage != ShaderStage.Compute && + (context.Info.IoDefinitions.Contains(new IoDefinition(StorageKind.Output, IoVariable.Layer)) || + context.Info.IoDefinitions.Contains(new IoDefinition(StorageKind.Output, IoVariable.ViewportIndex)))) + { + context.AddExtension("SPV_EXT_shader_viewport_index_layer"); + context.AddCapability(Capability.ShaderViewportIndexLayerEXT); + } + + if (context.Info.IoDefinitions.Contains(new IoDefinition(StorageKind.Output, IoVariable.ViewportMask))) + { + context.AddExtension("SPV_NV_viewport_array2"); + context.AddCapability(Capability.ShaderViewportMaskNV); + } + + if ((info.HelperFunctionsMask & NeedsInvocationIdMask) != 0) + { + info.IoDefinitions.Add(new IoDefinition(StorageKind.Input, IoVariable.SubgroupLaneId)); + } + + Declarations.DeclareAll(context, info); + + for (int funcIndex = 0; funcIndex < info.Functions.Count; funcIndex++) + { + StructuredFunction function = info.Functions[funcIndex]; + SpvInstruction retType = context.GetType(function.ReturnType); + + SpvInstruction[] funcArgs = new SpvInstruction[function.InArguments.Length + function.OutArguments.Length]; + + for (int argIndex = 0; argIndex < funcArgs.Length; argIndex++) + { + SpvInstruction argType = context.GetType(function.GetArgumentType(argIndex)); + SpvInstruction argPointerType = context.TypePointer(StorageClass.Function, argType); + funcArgs[argIndex] = argPointerType; + } + + SpvInstruction funcType = context.TypeFunction(retType, false, funcArgs); + SpvInstruction spvFunc = context.Function(retType, FunctionControlMask.MaskNone, funcType); + + context.DeclareFunction(funcIndex, function, spvFunc); + } + + for (int funcIndex = 0; funcIndex < info.Functions.Count; funcIndex++) + { + Generate(context, info, funcIndex); + } + + byte[] result = context.Generate(); + + lock (_poolLock) + { + _instructionPool.Release(instPool); + _integerPool.Release(integerPool); + } + + return result; + } + + private static void Generate(CodeGenContext context, StructuredProgramInfo info, int funcIndex) + { + (StructuredFunction function, SpvInstruction spvFunc) = context.GetFunction(funcIndex); + + context.CurrentFunction = function; + context.AddFunction(spvFunc); + context.StartFunction(isMainFunction: funcIndex == 0); + + Declarations.DeclareParameters(context, function); + + context.EnterBlock(function.MainBlock); + + Declarations.DeclareLocals(context, function); + + Generate(context, function.MainBlock); + + // Functions must always end with a return. + if (function.MainBlock.Last is not AstOperation operation || + (operation.Inst != Instruction.Return && operation.Inst != Instruction.Discard)) + { + context.Return(); + } + + context.FunctionEnd(); + + if (funcIndex == 0) + { + context.AddEntryPoint(context.Definitions.Stage.Convert(), spvFunc, "main", context.GetMainInterface()); + + if (context.Definitions.Stage == ShaderStage.TessellationControl) + { + context.AddExecutionMode(spvFunc, ExecutionMode.OutputVertices, (SpvLiteralInteger)context.Definitions.ThreadsPerInputPrimitive); + } + else if (context.Definitions.Stage == ShaderStage.TessellationEvaluation) + { + switch (context.Definitions.TessPatchType) + { + case TessPatchType.Isolines: + context.AddExecutionMode(spvFunc, ExecutionMode.Isolines); + break; + case TessPatchType.Triangles: + context.AddExecutionMode(spvFunc, ExecutionMode.Triangles); + break; + case TessPatchType.Quads: + context.AddExecutionMode(spvFunc, ExecutionMode.Quads); + break; + } + + switch (context.Definitions.TessSpacing) + { + case TessSpacing.EqualSpacing: + context.AddExecutionMode(spvFunc, ExecutionMode.SpacingEqual); + break; + case TessSpacing.FractionalEventSpacing: + context.AddExecutionMode(spvFunc, ExecutionMode.SpacingFractionalEven); + break; + case TessSpacing.FractionalOddSpacing: + context.AddExecutionMode(spvFunc, ExecutionMode.SpacingFractionalOdd); + break; + } + + bool tessCw = context.Definitions.TessCw; + + if (context.TargetApi == TargetApi.Vulkan) + { + // We invert the front face on Vulkan backend, so we need to do that here as well. + tessCw = !tessCw; + } + + if (tessCw) + { + context.AddExecutionMode(spvFunc, ExecutionMode.VertexOrderCw); + } + else + { + context.AddExecutionMode(spvFunc, ExecutionMode.VertexOrderCcw); + } + } + else if (context.Definitions.Stage == ShaderStage.Geometry) + { + context.AddExecutionMode(spvFunc, context.Definitions.InputTopology switch + { + InputTopology.Points => ExecutionMode.InputPoints, + InputTopology.Lines => ExecutionMode.InputLines, + InputTopology.LinesAdjacency => ExecutionMode.InputLinesAdjacency, + InputTopology.Triangles => ExecutionMode.Triangles, + InputTopology.TrianglesAdjacency => ExecutionMode.InputTrianglesAdjacency, + _ => throw new InvalidOperationException($"Invalid input topology \"{context.Definitions.InputTopology}\"."), + }); + + context.AddExecutionMode(spvFunc, ExecutionMode.Invocations, (SpvLiteralInteger)context.Definitions.ThreadsPerInputPrimitive); + + context.AddExecutionMode(spvFunc, context.Definitions.OutputTopology switch + { + OutputTopology.PointList => ExecutionMode.OutputPoints, + OutputTopology.LineStrip => ExecutionMode.OutputLineStrip, + OutputTopology.TriangleStrip => ExecutionMode.OutputTriangleStrip, + _ => throw new InvalidOperationException($"Invalid output topology \"{context.Definitions.OutputTopology}\"."), + }); + + context.AddExecutionMode(spvFunc, ExecutionMode.OutputVertices, (SpvLiteralInteger)context.Definitions.MaxOutputVertices); + } + else if (context.Definitions.Stage == ShaderStage.Fragment) + { + context.AddExecutionMode(spvFunc, context.Definitions.OriginUpperLeft + ? ExecutionMode.OriginUpperLeft + : ExecutionMode.OriginLowerLeft); + + if (context.Info.IoDefinitions.Contains(new IoDefinition(StorageKind.Output, IoVariable.FragmentOutputDepth))) + { + context.AddExecutionMode(spvFunc, ExecutionMode.DepthReplacing); + } + + if (context.Definitions.EarlyZForce) + { + context.AddExecutionMode(spvFunc, ExecutionMode.EarlyFragmentTests); + } + + if ((info.HelperFunctionsMask & HelperFunctionsMask.FSI) != 0 && + context.HostCapabilities.SupportsFragmentShaderInterlock) + { + context.AddExecutionMode(spvFunc, ExecutionMode.PixelInterlockOrderedEXT); + } + } + else if (context.Definitions.Stage == ShaderStage.Compute) + { + SpvLiteralInteger localSizeX = (SpvLiteralInteger)context.Definitions.ComputeLocalSizeX; + SpvLiteralInteger localSizeY = (SpvLiteralInteger)context.Definitions.ComputeLocalSizeY; + SpvLiteralInteger localSizeZ = (SpvLiteralInteger)context.Definitions.ComputeLocalSizeZ; + + context.AddExecutionMode( + spvFunc, + ExecutionMode.LocalSize, + localSizeX, + localSizeY, + localSizeZ); + } + + if (context.Definitions.TransformFeedbackEnabled && context.Definitions.LastInVertexPipeline) + { + context.AddExecutionMode(spvFunc, ExecutionMode.Xfb); + } + } + } + + private static void Generate(CodeGenContext context, AstBlock block) + { + AstBlockVisitor visitor = new(block); + + Dictionary loopTargets = new(); + + context.LoopTargets = loopTargets; + + visitor.BlockEntered += (sender, e) => + { + AstBlock mergeBlock = e.Block.Parent; + + if (e.Block.Type == AstBlockType.If) + { + AstBlock ifTrueBlock = e.Block; + AstBlock ifFalseBlock; + + if (AstHelper.Next(e.Block) is AstBlock nextBlock && nextBlock.Type == AstBlockType.Else) + { + ifFalseBlock = nextBlock; + } + else + { + ifFalseBlock = mergeBlock; + } + + SpvInstruction condition = context.Get(AggregateType.Bool, e.Block.Condition); + + context.SelectionMerge(context.GetNextLabel(mergeBlock), SelectionControlMask.MaskNone); + context.BranchConditional(condition, context.GetNextLabel(ifTrueBlock), context.GetNextLabel(ifFalseBlock)); + } + else if (e.Block.Type == AstBlockType.DoWhile) + { + SpvInstruction continueTarget = context.Label(); + + loopTargets.Add(e.Block, (context.NewBlock(), continueTarget)); + + context.LoopMerge(context.GetNextLabel(mergeBlock), continueTarget, LoopControlMask.MaskNone); + context.Branch(context.GetFirstLabel(e.Block)); + } + + context.EnterBlock(e.Block); + }; + + visitor.BlockLeft += (sender, e) => + { + if (e.Block.Parent != null) + { + if (e.Block.Type == AstBlockType.DoWhile) + { + // This is a loop, we need to jump back to the loop header + // if the condition is true. + AstBlock mergeBlock = e.Block.Parent; + + (SpvInstruction loopTarget, SpvInstruction continueTarget) = loopTargets[e.Block]; + + context.Branch(continueTarget); + context.AddLabel(continueTarget); + + SpvInstruction condition = context.Get(AggregateType.Bool, e.Block.Condition); + + context.BranchConditional(condition, loopTarget, context.GetNextLabel(mergeBlock)); + } + else + { + // We only need a branch if the last instruction didn't + // already cause the program to exit or jump elsewhere. + bool lastIsCf = e.Block.Last is AstOperation lastOp && + (lastOp.Inst == Instruction.Discard || + lastOp.Inst == Instruction.LoopBreak || + lastOp.Inst == Instruction.LoopContinue || + lastOp.Inst == Instruction.Return); + + if (!lastIsCf) + { + context.Branch(context.GetNextLabel(e.Block.Parent)); + } + } + + bool hasElse = AstHelper.Next(e.Block) is AstBlock nextBlock && + (nextBlock.Type == AstBlockType.Else || + nextBlock.Type == AstBlockType.ElseIf); + + // Re-enter the parent block. + if (e.Block.Parent != null && !hasElse) + { + context.EnterBlock(e.Block.Parent); + } + } + }; + + foreach (IAstNode node in visitor.Visit()) + { + if (node is AstAssignment assignment) + { + AstOperand dest = (AstOperand)assignment.Destination; + + if (dest.Type == OperandType.LocalVariable) + { + SpvInstruction source = context.Get(dest.VarType, assignment.Source); + context.Store(context.GetLocalPointer(dest), source); + } + else if (dest.Type == OperandType.Argument) + { + SpvInstruction source = context.Get(dest.VarType, assignment.Source); + context.Store(context.GetArgumentPointer(dest), source); + } + else + { + throw new NotImplementedException(dest.Type.ToString()); + } + } + else if (node is AstOperation operation) + { + Instructions.Generate(context, operation); + } + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Constants.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Constants.cs new file mode 100644 index 0000000..cf853f7 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Constants.cs @@ -0,0 +1,14 @@ +namespace Ryujinx.Graphics.Shader +{ + static class Constants + { + public const int ConstantBufferSize = 0x10000; // In bytes + + public const int MaxAttributes = 16; + public const int AllAttributesMask = (int)(uint.MaxValue >> (32 - MaxAttributes)); + + public const int NvnBaseVertexByteOffset = 0x640; + public const int NvnBaseInstanceByteOffset = 0x644; + public const int NvnDrawIndexByteOffset = 0x648; + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/Block.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/Block.cs new file mode 100644 index 0000000..1221af7 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/Block.cs @@ -0,0 +1,168 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System; +using System.Collections.Generic; +using System.Linq; + +namespace Ryujinx.Graphics.Shader.Decoders +{ + class PushOpInfo + { + public InstOp Op { get; } + public Dictionary Consumers; + + public PushOpInfo(InstOp op) + { + Op = op; + Consumers = new Dictionary(); + } + } + + readonly struct SyncTarget + { + public PushOpInfo PushOpInfo { get; } + public int PushOpId { get; } + + public SyncTarget(PushOpInfo pushOpInfo, int pushOpId) + { + PushOpInfo = pushOpInfo; + PushOpId = pushOpId; + } + } + + class Block + { + public ulong Address { get; set; } + public ulong EndAddress { get; set; } + + public List Predecessors { get; } + public List Successors { get; } + + public List OpCodes { get; } + public List PushOpCodes { get; } + public Dictionary SyncTargets { get; } + + public Block(ulong address) + { + Address = address; + + Predecessors = []; + Successors = []; + + OpCodes = []; + PushOpCodes = []; + SyncTargets = new Dictionary(); + } + + public void Split(Block rightBlock) + { + int splitIndex = BinarySearch(OpCodes, rightBlock.Address); + + if (OpCodes[splitIndex].Address < rightBlock.Address) + { + splitIndex++; + } + + int splitCount = OpCodes.Count - splitIndex; + if (splitCount <= 0) + { + throw new ArgumentException("Can't split at right block address."); + } + + rightBlock.EndAddress = EndAddress; + rightBlock.Successors.AddRange(Successors); + rightBlock.Predecessors.Add(this); + + EndAddress = rightBlock.Address; + + Successors.Clear(); + Successors.Add(rightBlock); + + // Move ops. + rightBlock.OpCodes.AddRange(OpCodes.GetRange(splitIndex, splitCount)); + + OpCodes.RemoveRange(splitIndex, splitCount); + + // Update push consumers that points to this block. + foreach (SyncTarget syncTarget in SyncTargets.Values) + { + PushOpInfo pushOpInfo = syncTarget.PushOpInfo; + + Operand local = pushOpInfo.Consumers[this]; + pushOpInfo.Consumers.Remove(this); + pushOpInfo.Consumers.Add(rightBlock, local); + } + + foreach ((ulong key, SyncTarget value) in SyncTargets) + { + rightBlock.SyncTargets.Add(key, value); + } + + SyncTargets.Clear(); + + // Move push ops. + for (int i = 0; i < PushOpCodes.Count; i++) + { + if (PushOpCodes[i].Op.Address >= rightBlock.Address) + { + int count = PushOpCodes.Count - i; + rightBlock.PushOpCodes.AddRange(PushOpCodes.Skip(i)); + PushOpCodes.RemoveRange(i, count); + break; + } + } + } + + private static int BinarySearch(List opCodes, ulong address) + { + int left = 0; + int middle = 0; + int right = opCodes.Count - 1; + + while (left <= right) + { + int size = right - left; + + middle = left + (size >> 1); + + InstOp opCode = opCodes[middle]; + + if (address == opCode.Address) + { + break; + } + + if (address < opCode.Address) + { + right = middle - 1; + } + else + { + left = middle + 1; + } + } + + return middle; + } + + public InstOp GetLastOp() + { + if (OpCodes.Count != 0) + { + return OpCodes[^1]; + } + + return default; + } + + public bool HasNext() + { + InstOp lastOp = GetLastOp(); + return OpCodes.Count != 0 && !Decoder.IsUnconditionalBranch(ref lastOp); + } + + public void AddPushOp(InstOp op) + { + PushOpCodes.Add(new PushOpInfo(op)); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/DecodedFunction.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/DecodedFunction.cs new file mode 100644 index 0000000..cfb1264 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/DecodedFunction.cs @@ -0,0 +1,48 @@ +using System; +using System.Collections.Generic; + +namespace Ryujinx.Graphics.Shader.Decoders +{ + class DecodedFunction + { + private readonly HashSet _callers; + + public bool IsCompilerGenerated => Type != FunctionType.User; + public FunctionType Type { get; set; } + public int Id { get; set; } + + public ulong Address { get; } + public Block[] Blocks { get; private set; } + + public DecodedFunction(ulong address) + { + Address = address; + _callers = []; + Type = FunctionType.User; + Id = -1; + } + + public void SetBlocks(Block[] blocks) + { + if (Blocks != null) + { + throw new InvalidOperationException("Blocks have already been set."); + } + + Blocks = blocks; + } + + public void AddCaller(DecodedFunction caller) + { + _callers.Add(caller); + } + + public void RemoveCaller(DecodedFunction caller) + { + if (_callers.Remove(caller) && _callers.Count == 0) + { + Type = FunctionType.Unused; + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/DecodedProgram.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/DecodedProgram.cs new file mode 100644 index 0000000..8b64b04 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/DecodedProgram.cs @@ -0,0 +1,78 @@ +using Ryujinx.Graphics.Shader.Translation; +using System; +using System.Collections; +using System.Collections.Generic; + +namespace Ryujinx.Graphics.Shader.Decoders +{ + readonly struct DecodedProgram : IEnumerable + { + public DecodedFunction MainFunction { get; } + private readonly IReadOnlyDictionary _functions; + private readonly List _functionsWithId; + public int FunctionsWithIdCount => _functionsWithId.Count; + + public AttributeUsage AttributeUsage { get; } + public FeatureFlags UsedFeatures { get; } + public byte ClipDistancesWritten { get; } + public int Cb1DataSize { get; } + + public DecodedProgram( + DecodedFunction mainFunction, + IReadOnlyDictionary functions, + AttributeUsage attributeUsage, + FeatureFlags usedFeatures, + byte clipDistancesWritten, + int cb1DataSize) + { + MainFunction = mainFunction; + _functions = functions; + _functionsWithId = []; + AttributeUsage = attributeUsage; + UsedFeatures = usedFeatures; + ClipDistancesWritten = clipDistancesWritten; + Cb1DataSize = cb1DataSize; + } + + public DecodedFunction GetFunctionByAddress(ulong address) + { + if (_functions.TryGetValue(address, out DecodedFunction function)) + { + return function; + } + + return null; + } + + public DecodedFunction GetFunctionById(int id) + { + if ((uint)id >= (uint)_functionsWithId.Count) + { + throw new ArgumentOutOfRangeException(nameof(id)); + } + + return _functionsWithId[id]; + } + + public void AddFunctionAndSetId(DecodedFunction function) + { + function.Id = _functionsWithId.Count; + _functionsWithId.Add(function); + } + + public IoUsage GetIoUsage() + { + return new IoUsage(UsedFeatures, ClipDistancesWritten, AttributeUsage.UsedOutputAttributes); + } + + public IEnumerator GetEnumerator() + { + return _functions.Values.GetEnumerator(); + } + + IEnumerator IEnumerable.GetEnumerator() + { + return GetEnumerator(); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/Decoder.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/Decoder.cs new file mode 100644 index 0000000..e0ee4c4 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/Decoder.cs @@ -0,0 +1,911 @@ +using Ryujinx.Graphics.Shader.Translation; +using System; +using System.Collections.Generic; +using System.Linq; +using System.Runtime.CompilerServices; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Decoders +{ + static class Decoder + { + private class Context + { + public AttributeUsage AttributeUsage { get; } + public FeatureFlags UsedFeatures { get; private set; } + public byte ClipDistancesWritten { get; private set; } + public int Cb1DataSize { get; private set; } + + private readonly IGpuAccessor _gpuAccessor; + + public Context(IGpuAccessor gpuAccessor) + { + _gpuAccessor = gpuAccessor; + AttributeUsage = new(gpuAccessor); + } + + public uint ConstantBuffer1Read(int offset) + { + if (Cb1DataSize < offset + 4) + { + Cb1DataSize = offset + 4; + } + + return _gpuAccessor.ConstantBuffer1Read(offset); + } + + public void SetUsedFeature(FeatureFlags flags) + { + UsedFeatures |= flags; + } + + public void SetClipDistanceWritten(int index) + { + ClipDistancesWritten |= (byte)(1 << index); + } + } + + public static DecodedProgram Decode(ShaderDefinitions definitions, IGpuAccessor gpuAccessor, ulong startAddress) + { + Context context = new(gpuAccessor); + Queue functionsQueue = new(); + Dictionary functionsVisited = new(); + + DecodedFunction EnqueueFunction(ulong address) + { + if (!functionsVisited.TryGetValue(address, out DecodedFunction function)) + { + functionsVisited.Add(address, function = new DecodedFunction(address)); + functionsQueue.Enqueue(function); + } + + return function; + } + + DecodedFunction mainFunction = EnqueueFunction(0); + + while (functionsQueue.TryDequeue(out DecodedFunction currentFunction)) + { + List blocks = []; + Queue workQueue = new(); + Dictionary visited = new(); + + Block GetBlock(ulong blkAddress) + { + if (!visited.TryGetValue(blkAddress, out Block block)) + { + block = new Block(blkAddress); + + workQueue.Enqueue(block); + visited.Add(blkAddress, block); + } + + return block; + } + + GetBlock(currentFunction.Address); + + bool hasNewTarget; + + do + { + while (workQueue.TryDequeue(out Block currBlock)) + { + // Check if the current block is inside another block. + if (BinarySearch(blocks, currBlock.Address, out int nBlkIndex)) + { + Block nBlock = blocks[nBlkIndex]; + + if (nBlock.Address == currBlock.Address) + { + throw new InvalidOperationException("Found duplicate block address on the list."); + } + + nBlock.Split(currBlock); + blocks.Insert(nBlkIndex + 1, currBlock); + + continue; + } + + // If we have a block after the current one, set the limit address. + ulong limitAddress = ulong.MaxValue; + + if (nBlkIndex != blocks.Count) + { + Block nBlock = blocks[nBlkIndex]; + + int nextIndex = nBlkIndex + 1; + + if (nBlock.Address < currBlock.Address && nextIndex < blocks.Count) + { + limitAddress = blocks[nextIndex].Address; + } + else if (nBlock.Address > currBlock.Address) + { + limitAddress = blocks[nBlkIndex].Address; + } + } + + FillBlock(definitions, gpuAccessor, context, currBlock, limitAddress, startAddress); + + if (currBlock.OpCodes.Count != 0) + { + // We should have blocks for all possible branch targets, + // including those from PBK/PCNT/SSY instructions. + foreach (PushOpInfo pushOp in currBlock.PushOpCodes) + { + GetBlock(pushOp.Op.GetAbsoluteAddress()); + } + + // Set child blocks. "Branch" is the block the branch instruction + // points to (when taken), "Next" is the block at the next address, + // executed when the branch is not taken. For Unconditional Branches + // or end of program, Next is null. + InstOp lastOp = currBlock.GetLastOp(); + + if (lastOp.Name == InstName.Cal) + { + EnqueueFunction(lastOp.GetAbsoluteAddress()).AddCaller(currentFunction); + } + else if (lastOp.Name == InstName.Bra) + { + Block succBlock = GetBlock(lastOp.GetAbsoluteAddress()); + currBlock.Successors.Add(succBlock); + succBlock.Predecessors.Add(currBlock); + } + + if (!IsUnconditionalBranch(ref lastOp)) + { + Block succBlock = GetBlock(currBlock.EndAddress); + currBlock.Successors.Insert(0, succBlock); + succBlock.Predecessors.Add(currBlock); + } + } + + // Insert the new block on the list (sorted by address). + if (blocks.Count != 0) + { + Block nBlock = blocks[nBlkIndex]; + + blocks.Insert(nBlkIndex + (nBlock.Address < currBlock.Address ? 1 : 0), currBlock); + } + else + { + blocks.Add(currBlock); + } + } + + // Propagate SSY/PBK addresses into their uses (SYNC/BRK). + foreach (Block block in blocks.Where(x => x.PushOpCodes.Count != 0)) + { + for (int pushOpIndex = 0; pushOpIndex < block.PushOpCodes.Count; pushOpIndex++) + { + PropagatePushOp(visited, block, pushOpIndex); + } + } + + // Try to find targets for BRX (indirect branch) instructions. + hasNewTarget = FindBrxTargets(context, blocks, GetBlock); + + // If we discovered new branch targets from the BRX instruction, + // we need another round of decoding to decode the new blocks. + // Additionally, we may have more SSY/PBK targets to propagate, + // and new BRX instructions. + } + while (hasNewTarget); + + currentFunction.SetBlocks(blocks.ToArray()); + } + + return new DecodedProgram( + mainFunction, + functionsVisited, + context.AttributeUsage, + context.UsedFeatures, + context.ClipDistancesWritten, + context.Cb1DataSize); + } + + private static bool BinarySearch(List blocks, ulong address, out int index) + { + index = 0; + + int left = 0; + int right = blocks.Count - 1; + + while (left <= right) + { + int size = right - left; + + int middle = left + (size >> 1); + + Block block = blocks[middle]; + + index = middle; + + if (address >= block.Address && address < block.EndAddress) + { + return true; + } + + if (address < block.Address) + { + right = middle - 1; + } + else + { + left = middle + 1; + } + } + + return false; + } + + private static void FillBlock( + ShaderDefinitions definitions, + IGpuAccessor gpuAccessor, + Context context, + Block block, + ulong limitAddress, + ulong startAddress) + { + ulong address = block.Address; + int bufferOffset = 0; + ReadOnlySpan buffer = ReadOnlySpan.Empty; + + InstOp op = default; + + do + { + if (address + 7 >= limitAddress) + { + break; + } + + // Ignore scheduling instructions, which are written every 32 bytes. + if ((address & 0x1f) == 0) + { + address += 8; + bufferOffset++; + continue; + } + + if (bufferOffset >= buffer.Length) + { + buffer = gpuAccessor.GetCode(startAddress + address, 8); + bufferOffset = 0; + } + + ulong opCode = buffer[bufferOffset++]; + + op = InstTable.GetOp(address, opCode); + + if (op.Props.HasFlag(InstProps.TexB)) + { + context.SetUsedFeature(FeatureFlags.Bindless); + } + + switch (op.Name) + { + case InstName.Ald: + case InstName.Ast: + case InstName.Ipa: + SetUserAttributeUses(definitions, context, op.Name, opCode); + break; + case InstName.Pbk: + case InstName.Pcnt: + case InstName.Ssy: + block.AddPushOp(op); + break; + case InstName.Shfl: + context.SetUsedFeature(FeatureFlags.Shuffle); + break; + case InstName.Ldl: + case InstName.Stl: + context.SetUsedFeature(FeatureFlags.LocalMemory); + break; + case InstName.Atoms: + case InstName.AtomsCas: + case InstName.Lds: + case InstName.Sts: + context.SetUsedFeature(FeatureFlags.SharedMemory); + break; + case InstName.Atom: + case InstName.AtomCas: + case InstName.Red: + case InstName.Stg: + case InstName.Suatom: + case InstName.SuatomB: + case InstName.SuatomB2: + case InstName.SuatomCas: + case InstName.SuatomCasB: + case InstName.Sured: + case InstName.SuredB: + case InstName.Sust: + case InstName.SustB: + case InstName.SustD: + case InstName.SustDB: + context.SetUsedFeature(FeatureFlags.Store); + break; + } + + block.OpCodes.Add(op); + + address += 8; + } + while (!op.Props.HasFlag(InstProps.Bra)); + + block.EndAddress = address; + } + + private static void SetUserAttributeUses(ShaderDefinitions definitions, Context context, InstName name, ulong opCode) + { + int offset; + int count = 1; + bool isStore = false; + bool indexed; + bool perPatch = false; + + if (name == InstName.Ast) + { + InstAst opAst = new(opCode); + count = (int)opAst.AlSize + 1; + offset = opAst.Imm11; + indexed = opAst.Phys; + perPatch = opAst.P; + isStore = true; + } + else if (name == InstName.Ald) + { + InstAld opAld = new(opCode); + count = (int)opAld.AlSize + 1; + offset = opAld.Imm11; + indexed = opAld.Phys; + perPatch = opAld.P; + isStore = opAld.O; + } + else /* if (name == InstName.Ipa) */ + { + InstIpa opIpa = new(opCode); + offset = opIpa.Imm10; + indexed = opIpa.Idx; + } + + if (indexed) + { + if (isStore) + { + context.AttributeUsage.SetAllOutputUserAttributes(); + definitions.EnableOutputIndexing(); + } + else + { + context.AttributeUsage.SetAllInputUserAttributes(); + definitions.EnableInputIndexing(); + } + } + else + { + for (int elemIndex = 0; elemIndex < count; elemIndex++) + { + int attr = offset + elemIndex * 4; + + if (perPatch) + { + if (attr is >= AttributeConsts.UserAttributePerPatchBase and < AttributeConsts.UserAttributePerPatchEnd) + { + int userAttr = attr - AttributeConsts.UserAttributePerPatchBase; + int index = userAttr / 16; + + if (isStore) + { + context.AttributeUsage.SetOutputUserAttributePerPatch(index); + } + else + { + context.AttributeUsage.SetInputUserAttributePerPatch(index); + } + } + } + else if (attr is >= AttributeConsts.UserAttributeBase and < AttributeConsts.UserAttributeEnd) + { + int userAttr = attr - AttributeConsts.UserAttributeBase; + int index = userAttr / 16; + + if (isStore) + { + context.AttributeUsage.SetOutputUserAttribute(index); + } + else + { + context.AttributeUsage.SetInputUserAttribute(index, (userAttr >> 2) & 3); + } + } + + if (!isStore && + (attr == AttributeConsts.FogCoord || + (attr >= AttributeConsts.FrontColorDiffuseR && attr < AttributeConsts.ClipDistance0) || + (attr >= AttributeConsts.TexCoordBase && attr < AttributeConsts.TexCoordEnd))) + { + context.SetUsedFeature(FeatureFlags.FixedFuncAttr); + } + else + { + if (isStore) + { + switch (attr) + { + case AttributeConsts.Layer: + if (definitions.Stage is not ShaderStage.Compute and not ShaderStage.Fragment) + { + context.SetUsedFeature(FeatureFlags.RtLayer); + } + + break; + case AttributeConsts.ViewportIndex: + if (definitions.Stage != ShaderStage.Fragment) + { + context.SetUsedFeature(FeatureFlags.ViewportIndex); + } + + break; + case AttributeConsts.ClipDistance0: + case AttributeConsts.ClipDistance1: + case AttributeConsts.ClipDistance2: + case AttributeConsts.ClipDistance3: + case AttributeConsts.ClipDistance4: + case AttributeConsts.ClipDistance5: + case AttributeConsts.ClipDistance6: + case AttributeConsts.ClipDistance7: + if (definitions.Stage.IsVtg) + { + context.SetClipDistanceWritten((attr - AttributeConsts.ClipDistance0) / 4); + } + + break; + case AttributeConsts.ViewportMask: + if (definitions.Stage != ShaderStage.Fragment) + { + context.SetUsedFeature(FeatureFlags.ViewportMask); + } + + break; + } + } + else + { + switch (attr) + { + case AttributeConsts.PositionX: + case AttributeConsts.PositionY: + if (definitions.Stage == ShaderStage.Fragment) + { + context.SetUsedFeature(FeatureFlags.FragCoordXY); + } + + break; + case AttributeConsts.InstanceId: + if (definitions.Stage == ShaderStage.Vertex) + { + context.SetUsedFeature(FeatureFlags.InstanceId); + } + + break; + } + } + } + } + } + } + + public static bool IsUnconditionalBranch(ref InstOp op) + { + return IsUnconditional(ref op) && op.Props.HasFlag(InstProps.Bra); + } + + private static bool IsUnconditional(ref InstOp op) + { + InstConditional condOp = new(op.RawOpCode); + + if ((op.Name == InstName.Bra || op.Name == InstName.Exit) && condOp.Ccc != Ccc.T) + { + return false; + } + + return condOp.Pred == RegisterConsts.PredicateTrueIndex && !condOp.PredInv; + } + + private static bool FindBrxTargets(Context context, IEnumerable blocks, Func getBlock) + { + bool hasNewTarget = false; + + foreach (Block block in blocks) + { + InstOp lastOp = block.GetLastOp(); + bool hasNext = block.HasNext(); + + if (lastOp.Name == InstName.Brx && block.Successors.Count == (hasNext ? 1 : 0)) + { + HashSet visited = []; + + InstBrx opBrx = new(lastOp.RawOpCode); + ulong baseOffset = lastOp.GetAbsoluteAddress(); + + // An indirect branch could go anywhere, + // try to get the possible target offsets from the constant buffer. + (int cbBaseOffset, int cbOffsetsCount) = FindBrxTargetRange(block, opBrx.SrcA); + + if (cbOffsetsCount != 0) + { + hasNewTarget = true; + } + + for (int i = 0; i < cbOffsetsCount; i++) + { + uint targetOffset = context.ConstantBuffer1Read(cbBaseOffset + i * 4); + ulong targetAddress = baseOffset + targetOffset; + + if (visited.Add(targetAddress)) + { + Block target = getBlock(targetAddress); + target.Predecessors.Add(block); + block.Successors.Add(target); + } + } + } + } + + return hasNewTarget; + } + + private static (int, int) FindBrxTargetRange(Block block, int brxReg) + { + // Try to match the following pattern: + // + // IMNMX.U32 Rx, Rx, UpperBound, PT + // SHL Rx, Rx, 0x2 + // LDC Rx, c[0x1][Rx+BaseOffset] + // + // Here, Rx is an arbitrary register, "UpperBound" and "BaseOffset" are constants. + // The above pattern is assumed to be generated by the compiler before BRX, + // as the instruction is usually used to implement jump tables for switch statement optimizations. + // On a successful match, "BaseOffset" is the offset in bytes where the jump offsets are + // located on the constant buffer, and "UpperBound" is the total number of offsets for the BRX, minus 1. + + HashSet visited = []; + + BlockLocation ldcLocation = FindFirstRegWrite(visited, new BlockLocation(block, block.OpCodes.Count - 1), brxReg); + if (ldcLocation.Block == null || ldcLocation.Block.OpCodes[ldcLocation.Index].Name != InstName.Ldc) + { + return (0, 0); + } + + GetOp(ldcLocation, out InstLdc opLdc); + + if (opLdc.CbufSlot != 1 || opLdc.AddressMode != 0) + { + return (0, 0); + } + + BlockLocation shlLocation = FindFirstRegWrite(visited, ldcLocation, opLdc.SrcA); + if (shlLocation.Block == null || !shlLocation.IsImmInst(InstName.Shl)) + { + return (0, 0); + } + + GetOp(shlLocation, out InstShlI opShl); + + if (opShl.Imm20 != 2) + { + return (0, 0); + } + + BlockLocation imnmxLocation = FindFirstRegWrite(visited, shlLocation, opShl.SrcA); + if (imnmxLocation.Block == null || !imnmxLocation.IsImmInst(InstName.Imnmx)) + { + return (0, 0); + } + + GetOp(imnmxLocation, out InstImnmxI opImnmx); + + if (opImnmx.Signed || opImnmx.SrcPred != RegisterConsts.PredicateTrueIndex || opImnmx.SrcPredInv) + { + return (0, 0); + } + + return (opLdc.CbufOffset, opImnmx.Imm20 + 1); + } + + private static void GetOp(BlockLocation location, out T op) where T : unmanaged + { + ulong rawOp = location.Block.OpCodes[location.Index].RawOpCode; + op = Unsafe.As(ref rawOp); + } + + private readonly struct BlockLocation + { + public Block Block { get; } + public int Index { get; } + + public BlockLocation(Block block, int index) + { + Block = block; + Index = index; + } + + public bool IsImmInst(InstName name) + { + InstOp op = Block.OpCodes[Index]; + return op.Name == name && op.Props.HasFlag(InstProps.Ib); + } + } + + private static BlockLocation FindFirstRegWrite(HashSet visited, BlockLocation location, int regIndex) + { + Queue toVisit = new(); + toVisit.Enqueue(location); + visited.Add(location.Block); + + while (toVisit.TryDequeue(out BlockLocation currentLocation)) + { + Block block = currentLocation.Block; + for (int i = currentLocation.Index - 1; i >= 0; i--) + { + if (WritesToRegister(block.OpCodes[i], regIndex)) + { + return new BlockLocation(block, i); + } + } + + foreach (Block predecessor in block.Predecessors) + { + if (visited.Add(predecessor)) + { + toVisit.Enqueue(new BlockLocation(predecessor, predecessor.OpCodes.Count)); + } + } + } + + return new BlockLocation(null, 0); + } + + private static bool WritesToRegister(InstOp op, int regIndex) + { + // Predicate instruction only ever writes to predicate, so we shouldn't check those. + if ((op.Props & (InstProps.Rd | InstProps.Rd2)) == 0) + { + return false; + } + + if (op.Props.HasFlag(InstProps.Rd2) && (byte)(op.RawOpCode >> 28) == regIndex) + { + return true; + } + + return (byte)op.RawOpCode == regIndex; + } + + private enum MergeType + { + Brk, + Cont, + Sync, + } + + private readonly struct PathBlockState + { + public Block Block { get; } + + private enum RestoreType + { + None, + PopPushOp, + PushBranchOp, + } + + private readonly RestoreType _restoreType; + + private readonly ulong _restoreValue; + private readonly MergeType _restoreMergeType; + + public bool ReturningFromVisit => _restoreType != RestoreType.None; + + public PathBlockState(Block block) + { + Block = block; + _restoreType = RestoreType.None; + _restoreValue = 0; + _restoreMergeType = default; + } + + public PathBlockState(int oldStackSize) + { + Block = null; + _restoreType = RestoreType.PopPushOp; + _restoreValue = (ulong)oldStackSize; + _restoreMergeType = default; + } + + public PathBlockState(ulong syncAddress, MergeType mergeType) + { + Block = null; + _restoreType = RestoreType.PushBranchOp; + _restoreValue = syncAddress; + _restoreMergeType = mergeType; + } + + public void RestoreStackState(Stack<(ulong, MergeType)> branchStack) + { + if (_restoreType == RestoreType.PushBranchOp) + { + branchStack.Push((_restoreValue, _restoreMergeType)); + } + else if (_restoreType == RestoreType.PopPushOp) + { + while (branchStack.Count > (uint)_restoreValue) + { + branchStack.Pop(); + } + } + } + } + + private static void PropagatePushOp(Dictionary blocks, Block currBlock, int pushOpIndex) + { + PushOpInfo pushOpInfo = currBlock.PushOpCodes[pushOpIndex]; + InstOp pushOp = pushOpInfo.Op; + + Block target = blocks[pushOp.GetAbsoluteAddress()]; + + Stack workQueue = new(); + HashSet visited = []; + Stack<(ulong, MergeType)> branchStack = new(); + + void Push(PathBlockState pbs) + { + // When block is null, this means we are pushing a restore operation. + // Restore operations are used to undo the work done inside a block + // when we return from it, for example it pops addresses pushed by + // SSY/PBK instructions inside the block, and pushes addresses poped + // by SYNC/BRK. + // For blocks, if it's already visited, we just ignore to avoid going + // around in circles and getting stuck here. + if (pbs.Block == null || !visited.Contains(pbs.Block)) + { + workQueue.Push(pbs); + } + } + + Push(new PathBlockState(currBlock)); + + while (workQueue.TryPop(out PathBlockState pbs)) + { + if (pbs.ReturningFromVisit) + { + pbs.RestoreStackState(branchStack); + + continue; + } + + Block current = pbs.Block; + + // If the block was already processed, we just ignore it, otherwise + // we would push the same child blocks of an already processed block, + // and go around in circles until memory is exhausted. + if (!visited.Add(current)) + { + continue; + } + + int pushOpsCount = current.PushOpCodes.Count; + if (pushOpsCount != 0) + { + Push(new PathBlockState(branchStack.Count)); + + for (int index = pushOpIndex; index < pushOpsCount; index++) + { + InstOp currentPushOp = current.PushOpCodes[index].Op; + MergeType pushMergeType = GetMergeTypeFromPush(currentPushOp.Name); + branchStack.Push((currentPushOp.GetAbsoluteAddress(), pushMergeType)); + } + } + + pushOpIndex = 0; + + bool hasNext = current.HasNext(); + if (hasNext) + { + Push(new PathBlockState(current.Successors[0])); + } + + InstOp lastOp = current.GetLastOp(); + if (IsPopBranch(lastOp.Name)) + { + MergeType popMergeType = GetMergeTypeFromPop(lastOp.Name); + + bool found = true; + ulong targetAddress = 0UL; + MergeType mergeType; + + do + { + if (branchStack.Count == 0) + { + found = false; + break; + } + + (targetAddress, mergeType) = branchStack.Pop(); + + // Push the target address (this will be used to push the address + // back into the PBK/PCNT/SSY stack when we return from that block), + Push(new PathBlockState(targetAddress, mergeType)); + } + while (mergeType != popMergeType); + + // Make sure we found the correct address, + // the push and pop instruction types must match, so: + // - BRK can only consume addresses pushed by PBK. + // - CONT can only consume addresses pushed by PCNT. + // - SYNC can only consume addresses pushed by SSY. + if (found) + { + if (branchStack.Count == 0) + { + // If the entire stack was consumed, then the current pop instruction + // just consumed the address from our push instruction. + if (current.SyncTargets.TryAdd(pushOp.Address, new SyncTarget(pushOpInfo, current.SyncTargets.Count))) + { + pushOpInfo.Consumers.Add(current, Local()); + target.Predecessors.Add(current); + current.Successors.Add(target); + } + } + else + { + // Push the block itself into the work queue for processing. + Push(new PathBlockState(blocks[targetAddress])); + } + } + } + else + { + // By adding them in descending order (sorted by address), we process the blocks + // in order (of ascending address), since we work with a LIFO. + foreach (Block possibleTarget in current.Successors.OrderByDescending(x => x.Address)) + { + if (!hasNext || possibleTarget != current.Successors[0]) + { + Push(new PathBlockState(possibleTarget)); + } + } + } + } + } + + public static bool IsPopBranch(InstName name) + { + return name is InstName.Brk or InstName.Cont or InstName.Sync; + } + + private static MergeType GetMergeTypeFromPush(InstName name) + { + return name switch + { + InstName.Pbk => MergeType.Brk, + InstName.Pcnt => MergeType.Cont, + _ => MergeType.Sync, + }; + } + + private static MergeType GetMergeTypeFromPop(InstName name) + { + return name switch + { + InstName.Brk => MergeType.Brk, + InstName.Cont => MergeType.Cont, + _ => MergeType.Sync, + }; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/FunctionType.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/FunctionType.cs new file mode 100644 index 0000000..efcdb79 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/FunctionType.cs @@ -0,0 +1,10 @@ +namespace Ryujinx.Graphics.Shader.Decoders +{ + enum FunctionType : byte + { + User, + Unused, + BuiltInFSIBegin, + BuiltInFSIEnd + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/InstDecoders.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/InstDecoders.cs new file mode 100644 index 0000000..3856174 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/InstDecoders.cs @@ -0,0 +1,5394 @@ +using System; + +namespace Ryujinx.Graphics.Shader.Decoders +{ + enum AlSize + { + _32 = 0, + _64 = 1, + _96 = 2, + _128 = 3, + } + + enum AtomSize + { + U32 = 0, + S32 = 1, + U64 = 2, + F32FtzRn = 3, + F16x2FtzRn = 4, + S64 = 5, + } + + enum AtomOp + { + Add = 0, + Min = 1, + Max = 2, + Inc = 3, + Dec = 4, + And = 5, + Or = 6, + Xor = 7, + Exch = 8, + Safeadd = 10, + } + + enum AtomsSize + { + U32 = 0, + S32 = 1, + U64 = 2, + S64 = 3, + } + + enum BarMode + { + Bar = 0, + Result = 1, + Warp = 2, + } + + enum BarOp + { + Sync = 0, + Arv = 1, + Red = 2, + Scan = 3, + SyncAll = 4, + } + + enum BarRedOp + { + Popc = 0, + And = 1, + Or = 2, + } + + enum Bpt + { + DrainIllegal = 0, + Cal = 1, + Pause = 2, + Trap = 3, + Int = 4, + Drain = 5, + } + + enum Ccc + { + F = 0, + Lt = 1, + Eq = 2, + Le = 3, + Gt = 4, + Ne = 5, + Ge = 6, + Num = 7, + Nan = 8, + Ltu = 9, + Equ = 10, + Leu = 11, + Gtu = 12, + Neu = 13, + Geu = 14, + T = 15, + Off = 16, + Lo = 17, + Sff = 18, + Ls = 19, + Hi = 20, + Sft = 21, + Hs = 22, + Oft = 23, + CsmTa = 24, + CsmTr = 25, + CsmMx = 26, + FcsmTa = 27, + FcsmTr = 28, + FcsmMx = 29, + Rle = 30, + Rgt = 31, + } + + enum CacheType + { + U = 1, + C = 2, + I = 3, + Crs = 4, + } + + enum CctlOp + { + Pf1 = 1, + Pf1_5 = 2, + Pf2 = 3, + Wb = 4, + Iv = 5, + Ivall = 6, + Rs = 7, + Rslb = 9, + } + + enum CctltOp + { + Ivth = 1, + } + + enum BoolOp + { + And = 0, + Or = 1, + Xor = 2, + } + + enum SReg + { + LaneId = 0, + Clock = 1, + VirtCfg = 2, + VirtId = 3, + Pm0 = 4, + Pm1 = 5, + Pm2 = 6, + Pm3 = 7, + Pm4 = 8, + Pm5 = 9, + Pm6 = 10, + Pm7 = 11, + OrderingTicket = 15, + PrimType = 16, + InvocationId = 17, + YDirection = 18, + ThreadKill = 19, + ShaderType = 20, + DirectCbeWriteAddressLow = 21, + DirectCbeWriteAddressHigh = 22, + DirectCbeWriteEnabled = 23, + MachineId0 = 24, + MachineId1 = 25, + MachineId2 = 26, + MachineId3 = 27, + Affinity = 28, + InvocationInfo = 29, + WScaleFactorXY = 30, + WScaleFactorZ = 31, + TId = 32, + TIdX = 33, + TIdY = 34, + TIdZ = 35, + CtaParam = 36, + CtaIdX = 37, + CtaIdY = 38, + CtaIdZ = 39, + Ntid = 40, + CirQueueIncrMinusOne = 41, + Nlatc = 42, + Swinlo = 48, + Swinsz = 49, + Smemsz = 50, + Smembanks = 51, + LWinLo = 52, + LWinSz = 53, + LMemLoSz = 54, + LMemHiOff = 55, + EqMask = 56, + LtMask = 57, + LeMask = 58, + GtMask = 59, + GeMask = 60, + RegAlloc = 61, + CtxAddr = 62, + GlobalErrorStatus = 64, + WarpErrorStatus = 66, + WarpErrorStatusClear = 67, + PmHi0 = 72, + PmHi1 = 73, + PmHi2 = 74, + PmHi3 = 75, + PmHi4 = 76, + PmHi5 = 77, + PmHi6 = 78, + PmHi7 = 79, + ClockLo = 80, + ClockHi = 81, + GlobalTimerLo = 82, + GlobalTimerHi = 83, + HwTaskId = 96, + CircularQueueEntryIndex = 97, + CircularQueueEntryAddressLow = 98, + CircularQueueEntryAddressHigh = 99, + } + + enum RoundMode + { + Rn = 0, + Rm = 1, + Rp = 2, + Rz = 3, + } + + enum FComp + { + F = 0, + Lt = 1, + Eq = 2, + Le = 3, + Gt = 4, + Ne = 5, + Ge = 6, + Num = 7, + Nan = 8, + Ltu = 9, + Equ = 10, + Leu = 11, + Gtu = 12, + Neu = 13, + Geu = 14, + T = 15, + } + + enum IntegerRound + { + Pass = 1, + Round = 4, + Floor = 5, + Ceil = 6, + Trunc = 7, + } + + enum IDstFmt + { + U16 = 1, + U32 = 2, + U64 = 3, + S16 = 5, + S32 = 6, + S64 = 7, + } + + enum ISrcFmt + { + U8 = 0, + U16 = 1, + U32 = 2, + U64 = 3, + S8 = 4, + S16 = 5, + S32 = 6, + S64 = 7, + } + + enum ISrcDstFmt + { + U8 = 0, + U16 = 1, + U32 = 2, + S8 = 4, + S16 = 5, + S32 = 6, + } + + enum RoundMode2 + { + Round = 0, + Floor = 1, + Ceil = 2, + Trunc = 3, + } + + enum ChkModeF + { + Divide = 0, + } + + enum Fmz + { + Ftz = 1, + Fmz = 2, + } + + enum MultiplyScale + { + NoScale = 0, + D2 = 1, + D4 = 2, + D8 = 3, + M8 = 4, + M4 = 5, + M2 = 6, + } + + enum OFmt + { + F16 = 0, + F32 = 1, + MrgH0 = 2, + MrgH1 = 3, + } + + enum HalfSwizzle + { + F16 = 0, + F32 = 1, + H0H0 = 2, + H1H1 = 3, + } + + enum ByteSel + { + B0 = 0, + B1 = 1, + B2 = 2, + B3 = 3, + } + + enum DstFmt + { + F16 = 1, + F32 = 2, + F64 = 3, + } + + enum AvgMode + { + NoNeg = 0, + NegB = 1, + NegA = 2, + PlusOne = 3, + } + + enum Lrs + { + None = 0, + RightShift = 1, + LeftShift = 2, + } + + enum HalfSelect + { + B32 = 0, + H0 = 1, + H1 = 2, + } + + enum IComp + { + F = 0, + Lt = 1, + Eq = 2, + Le = 3, + Gt = 4, + Ne = 5, + Ge = 6, + T = 7, + } + + enum XMode + { + Xlo = 1, + Xmed = 2, + Xhi = 3, + } + + enum IpaOp + { + Pass = 0, + Multiply = 1, + Constant = 2, + Sc = 3, + } + + enum IBase + { + Patch = 1, + Prim = 2, + Attr = 3, + } + + enum CacheOpLd + { + Ca = 0, + Cg = 1, + Ci = 2, + Cv = 3, + } + + enum CacheOpSt + { + Wb = 0, + Cg = 1, + Ci = 2, + Wt = 3, + } + + enum LsSize + { + U8 = 0, + S8 = 1, + U16 = 2, + S16 = 3, + B32 = 4, + B64 = 5, + B128 = 6, + UB128 = 7, + } + + enum LsSize2 + { + U8 = 0, + S8 = 1, + U16 = 2, + S16 = 3, + B32 = 4, + B64 = 5, + B128 = 6, + } + + enum AddressMode + { + Il = 1, + Is = 2, + Isl = 3, + } + + enum CacheOp2 + { + Lu = 1, + Ci = 2, + Cv = 3, + } + + enum PredicateOp + { + F = 0, + T = 1, + Z = 2, + Nz = 3, + } + + enum LogicOp + { + And = 0, + Or = 1, + Xor = 2, + PassB = 3, + } + + enum Membar + { + Cta = 0, + Gl = 1, + Sys = 2, + Vc = 3, + } + + enum Ivall + { + Ivalld = 1, + Ivallt = 2, + Ivalltd = 3, + } + + enum MufuOp + { + Cos = 0, + Sin = 1, + Ex2 = 2, + Lg2 = 3, + Rcp = 4, + Rsq = 5, + Rcp64h = 6, + Rsq64h = 7, + Sqrt = 8, + } + + enum OutType + { + Emit = 1, + Cut = 2, + EmitThenCut = 3, + } + + enum PixMode + { + Covmask = 1, + Covered = 2, + Offset = 3, + CentroidOffset = 4, + MyIndex = 5, + } + + enum PMode + { + F4e = 1, + B4e = 2, + Rc8 = 3, + Ecl = 4, + Ecr = 5, + Rc16 = 6, + } + + enum RedOp + { + Add = 0, + Min = 1, + Max = 2, + Inc = 3, + Dec = 4, + And = 5, + Or = 6, + Xor = 7, + } + + enum XModeShf + { + Hi = 1, + X = 2, + Xhi = 3, + } + + enum MaxShift + { + U64 = 2, + S64 = 3, + } + + enum ShflMode + { + Idx = 0, + Up = 1, + Down = 2, + Bfly = 3, + } + + enum Clamp + { + Ign = 0, + Trap = 2, + } + + enum SuatomSize + { + U32 = 0, + S32 = 1, + U64 = 2, + F32FtzRn = 3, + F16x2FtzRn = 4, + S64 = 5, + Sd32 = 6, + Sd64 = 7, + } + + enum SuDim + { + _1d = 0, + _1dBuffer = 1, + _1dArray = 2, + _2d = 3, + _2dArray = 4, + _3d = 5, + } + + enum SuatomOp + { + Add = 0, + Min = 1, + Max = 2, + Inc = 3, + Dec = 4, + And = 5, + Or = 6, + Xor = 7, + Exch = 8, + } + + enum SuSize + { + U8 = 0, + S8 = 1, + U16 = 2, + S16 = 3, + B32 = 4, + B64 = 5, + B128 = 6, + UB128 = 7, + } + + enum SuRgba + { + R = 1, + G = 2, + Rg = 3, + B = 4, + Rb = 5, + Gb = 6, + Rgb = 7, + A = 8, + Ra = 9, + Ga = 10, + Rga = 11, + Ba = 12, + Rba = 13, + Gba = 14, + Rgba = 15, + } + + enum Lod + { + Lz = 1, + Lb = 2, + Ll = 3, + Lba = 6, + Lla = 7, + } + + enum TexDim + { + _1d = 0, + Array1d = 1, + _2d = 2, + Array2d = 3, + _3d = 4, + Array3d = 5, + Cube = 6, + ArrayCube = 7, + } + + enum TexsTarget + { + Texture1DLodZero = 0, + Texture2D = 1, + Texture2DLodZero = 2, + Texture2DLodLevel = 3, + Texture2DDepthCompare = 4, + Texture2DLodLevelDepthCompare = 5, + Texture2DLodZeroDepthCompare = 6, + Texture2DArray = 7, + Texture2DArrayLodZero = 8, + Texture2DArrayLodZeroDepthCompare = 9, + Texture3D = 10, + Texture3DLodZero = 11, + TextureCube = 12, + TextureCubeLodLevel = 13, + } + + enum TldsTarget + { + Texture1DLodZero = 0x0, + Texture1DLodLevel = 0x1, + Texture2DLodZero = 0x2, + Texture2DLodZeroOffset = 0x4, + Texture2DLodLevel = 0x5, + Texture2DLodZeroMultisample = 0x6, + Texture3DLodZero = 0x7, + Texture2DArrayLodZero = 0x8, + Texture2DLodLevelOffset = 0xc, + } + + enum TexComp + { + R = 0, + G = 1, + B = 2, + A = 3, + } + + enum TexOffset + { + None = 0, + Aoffi = 1, + Ptp = 2, + } + + enum TexQuery + { + TexHeaderDimension = 1, + TexHeaderTextureType = 2, + TexHeaderSamplerPos = 5, + TexSamplerFilter = 16, + TexSamplerLod = 18, + TexSamplerWrap = 20, + TexSamplerBorderColor = 22, + } + + [Flags] + enum VectorSelect + { + U8B0 = 0, + U8B1 = 1, + U8B2 = 2, + U8B3 = 3, + U16H0 = 4, + U16H1 = 5, + U32 = 6, + S8B0 = 8, + S8B1 = 9, + S8B2 = 10, + S8B3 = 11, + S16H0 = 12, + S16H1 = 13, + S32 = 14, + } + + enum VideoOp + { + Mrg16h = 0, + Mrg16l = 1, + Mrg8b0 = 2, + Mrg8b2 = 3, + Acc = 4, + Min = 5, + Max = 6, + } + + enum VideoRed + { + Acc = 1, + } + + enum LaneMask4 + { + Z = 1, + W = 2, + Zw = 3, + X = 4, + Xz = 5, + Xw = 6, + Xzw = 7, + Y = 8, + Yz = 9, + Yw = 10, + Yzw = 11, + Xy = 12, + Xyz = 13, + Xyw = 14, + Xyzw = 15, + } + + enum ASelect4 + { + _0000 = 0, + _1111 = 1, + _2222 = 2, + _3333 = 3, + _3210 = 4, + _5432 = 6, + _6543 = 7, + _3201 = 8, + _3012 = 9, + _0213 = 10, + _3120 = 11, + _1230 = 12, + _2310 = 13, + } + + enum BSelect4 + { + _4444 = 0, + _5555 = 1, + _6666 = 2, + _7777 = 3, + _7654 = 4, + _5432 = 6, + _4321 = 7, + _4567 = 8, + _6745 = 9, + _5476 = 10, + } + + enum VideoScale + { + Shr7 = 1, + Shr15 = 2, + } + + enum VoteMode + { + All = 0, + Any = 1, + Eq = 2, + } + + enum XmadCop + { + Cfull = 0, + Clo = 1, + Chi = 2, + Csfu = 3, + Cbcc = 4, + } + + enum XmadCop2 + { + Cfull = 0, + Clo = 1, + Chi = 2, + Csfu = 3, + } + + enum ImadspASelect + { + U32 = 0, + S32 = 1, + U24 = 2, + S24 = 3, + U16h0 = 4, + S16h0 = 5, + U16h1 = 6, + S16h1 = 7, + } + + enum ImadspBSelect + { + U24 = 0, + S24 = 1, + U16h0 = 2, + S16h0 = 3, + } + + readonly struct InstConditional + { + private readonly ulong _opcode; + public InstConditional(ulong opcode) => _opcode = opcode; + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Ccc Ccc => (Ccc)(_opcode & 0x1F); + } + + readonly struct InstAl2p + { + private readonly ulong _opcode; + public InstAl2p(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public AlSize AlSize => (AlSize)((_opcode >> 47) & 0x3); + public bool Aio => (_opcode & 0x100000000) != 0; + public int Imm11 => (int)((_opcode >> 20) & 0x7FF); + public int DestPred => (int)((_opcode >> 44) & 0x7); + } + + readonly struct InstAld + { + private readonly ulong _opcode; + public InstAld(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int Imm11 => (int)((_opcode >> 20) & 0x7FF); + public bool P => (_opcode & 0x80000000) != 0; + public bool O => (_opcode & 0x100000000) != 0; + public AlSize AlSize => (AlSize)((_opcode >> 47) & 0x3); + public bool Phys => !P && Imm11 == 0 && SrcA != RegisterConsts.RegisterZeroIndex; + } + + readonly struct InstAst + { + private readonly ulong _opcode; + public InstAst(ulong opcode) => _opcode = opcode; + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)(_opcode & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int Imm11 => (int)((_opcode >> 20) & 0x7FF); + public bool P => (_opcode & 0x80000000) != 0; + public AlSize AlSize => (AlSize)((_opcode >> 47) & 0x3); + public bool Phys => !P && Imm11 == 0 && SrcA != RegisterConsts.RegisterZeroIndex; + } + + readonly struct InstAtom + { + private readonly ulong _opcode; + public InstAtom(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int Imm20 => (int)((_opcode >> 28) & 0xFFFFF); + public AtomSize Size => (AtomSize)((_opcode >> 49) & 0x7); + public AtomOp Op => (AtomOp)((_opcode >> 52) & 0xF); + public bool E => (_opcode & 0x1000000000000) != 0; + } + + readonly struct InstAtomCas + { + private readonly ulong _opcode; + public InstAtomCas(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int BcRz => (int)((_opcode >> 50) & 0x3); + public bool E => (_opcode & 0x1000000000000) != 0; + } + + readonly struct InstAtoms + { + private readonly ulong _opcode; + public InstAtoms(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int Imm22 => (int)((_opcode >> 30) & 0x3FFFFF); + public AtomsSize AtomsSize => (AtomsSize)((_opcode >> 28) & 0x3); + public AtomOp AtomOp => (AtomOp)((_opcode >> 52) & 0xF); + } + + readonly struct InstAtomsCas + { + private readonly ulong _opcode; + public InstAtomsCas(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int AtomsBcRz => (int)((_opcode >> 28) & 0x3); + } + + readonly struct InstB2r + { + private readonly ulong _opcode; + public InstB2r(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int DestPred => (int)((_opcode >> 45) & 0x7); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public BarMode Mode => (BarMode)((_opcode >> 32) & 0x3); + } + + readonly struct InstBar + { + private readonly ulong _opcode; + public InstBar(ulong opcode) => _opcode = opcode; + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm12 => (int)((_opcode >> 20) & 0xFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public BarOp BarOp => (BarOp)((_opcode >> 32) & 0x7); + public BarRedOp BarRedOp => (BarRedOp)((_opcode >> 35) & 0x3); + public bool AFixBar => (_opcode & 0x100000000000) != 0; + public bool BFixBar => (_opcode & 0x80000000000) != 0; + } + + readonly struct InstBfeR + { + private readonly ulong _opcode; + public InstBfeR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Signed => (_opcode & 0x1000000000000) != 0; + public bool Brev => (_opcode & 0x10000000000) != 0; + } + + readonly struct InstBfeI + { + private readonly ulong _opcode; + public InstBfeI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Signed => (_opcode & 0x1000000000000) != 0; + public bool Brev => (_opcode & 0x10000000000) != 0; + } + + readonly struct InstBfeC + { + private readonly ulong _opcode; + public InstBfeC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Signed => (_opcode & 0x1000000000000) != 0; + public bool Brev => (_opcode & 0x10000000000) != 0; + } + + readonly struct InstBfiR + { + private readonly ulong _opcode; + public InstBfiR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + } + + readonly struct InstBfiI + { + private readonly ulong _opcode; + public InstBfiI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + } + + readonly struct InstBfiC + { + private readonly ulong _opcode; + public InstBfiC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + } + + readonly struct InstBfiRc + { + private readonly ulong _opcode; + public InstBfiRc(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + } + + readonly struct InstBpt + { + private readonly ulong _opcode; + public InstBpt(ulong opcode) => _opcode = opcode; + public int Imm20 => (int)((_opcode >> 20) & 0xFFFFF); + public Bpt Bpt => (Bpt)((_opcode >> 6) & 0x7); + } + + readonly struct InstBra + { + private readonly ulong _opcode; + public InstBra(ulong opcode) => _opcode = opcode; + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Ccc Ccc => (Ccc)(_opcode & 0x1F); + public int Imm24 => (int)((_opcode >> 20) & 0xFFFFFF); + public bool Ca => (_opcode & 0x20) != 0; + public bool Lmt => (_opcode & 0x40) != 0; + public bool U => (_opcode & 0x80) != 0; + } + + readonly struct InstBrk + { + private readonly ulong _opcode; + public InstBrk(ulong opcode) => _opcode = opcode; + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Ccc Ccc => (Ccc)(_opcode & 0x1F); + } + + readonly struct InstBrx + { + private readonly ulong _opcode; + public InstBrx(ulong opcode) => _opcode = opcode; + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Ccc Ccc => (Ccc)(_opcode & 0x1F); + public int Imm24 => (int)((_opcode >> 20) & 0xFFFFFF); + public bool Ca => (_opcode & 0x20) != 0; + public bool Lmt => (_opcode & 0x40) != 0; + } + + readonly struct InstCal + { + private readonly ulong _opcode; + public InstCal(ulong opcode) => _opcode = opcode; + public bool Ca => (_opcode & 0x20) != 0; + public int Imm24 => (int)((_opcode >> 20) & 0xFFFFFF); + public bool Inc => (_opcode & 0x40) != 0; + } + + readonly struct InstCctl + { + private readonly ulong _opcode; + public InstCctl(ulong opcode) => _opcode = opcode; + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int Imm30 => (int)((_opcode >> 22) & 0x3FFFFFFF); + public bool E => (_opcode & 0x10000000000000) != 0; + public CacheType Cache => (CacheType)((_opcode >> 4) & 0x7); + public CctlOp CctlOp => (CctlOp)(_opcode & 0xF); + } + + readonly struct InstCctll + { + private readonly ulong _opcode; + public InstCctll(ulong opcode) => _opcode = opcode; + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int Imm22 => (int)((_opcode >> 22) & 0x3FFFFF); + public int Cache => (int)((_opcode >> 4) & 0x3); + public CctlOp CctlOp => (CctlOp)(_opcode & 0xF); + } + + readonly struct InstCctlt + { + private readonly ulong _opcode; + public InstCctlt(ulong opcode) => _opcode = opcode; + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int TsIdx13 => (int)((_opcode >> 36) & 0x1FFF); + public CctltOp CctltOp => (CctltOp)(_opcode & 0x3); + } + + readonly struct InstCctltR + { + private readonly ulong _opcode; + public InstCctltR(ulong opcode) => _opcode = opcode; + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public CctltOp CctltOp => (CctltOp)(_opcode & 0x3); + } + + readonly struct InstCont + { + private readonly ulong _opcode; + public InstCont(ulong opcode) => _opcode = opcode; + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Ccc Ccc => (Ccc)(_opcode & 0x1F); + } + + readonly struct InstCset + { + private readonly ulong _opcode; + public InstCset(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public Ccc Ccc => (Ccc)((_opcode >> 8) & 0x1F); + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public bool BVal => (_opcode & 0x100000000000) != 0; + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + } + + readonly struct InstCsetp + { + private readonly ulong _opcode; + public InstCsetp(ulong opcode) => _opcode = opcode; + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public Ccc Ccc => (Ccc)((_opcode >> 8) & 0x1F); + public int DestPred => (int)((_opcode >> 3) & 0x7); + public int DestPredInv => (int)(_opcode & 0x7); + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + } + + readonly struct InstCs2r + { + private readonly ulong _opcode; + public InstCs2r(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public SReg SReg => (SReg)((_opcode >> 20) & 0xFF); + } + + readonly struct InstDaddR + { + private readonly ulong _opcode; + public InstDaddR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool AbsA => (_opcode & 0x400000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public RoundMode RoundMode => (RoundMode)((_opcode >> 39) & 0x3); + } + + readonly struct InstDaddI + { + private readonly ulong _opcode; + public InstDaddI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool AbsA => (_opcode & 0x400000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public RoundMode RoundMode => (RoundMode)((_opcode >> 39) & 0x3); + } + + readonly struct InstDaddC + { + private readonly ulong _opcode; + public InstDaddC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool AbsA => (_opcode & 0x400000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public RoundMode RoundMode => (RoundMode)((_opcode >> 39) & 0x3); + } + + readonly struct InstDepbar + { + private readonly ulong _opcode; + public InstDepbar(ulong opcode) => _opcode = opcode; + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool Le => (_opcode & 0x20000000) != 0; + public int Sbid => (int)((_opcode >> 26) & 0x7); + public int PendCnt => (int)((_opcode >> 20) & 0x3F); + public int Imm6 => (int)(_opcode & 0x3F); + } + + readonly struct InstDfmaR + { + private readonly ulong _opcode; + public InstDfmaR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public RoundMode RoundMode => (RoundMode)((_opcode >> 50) & 0x3); + public bool NegC => (_opcode & 0x2000000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + } + + readonly struct InstDfmaI + { + private readonly ulong _opcode; + public InstDfmaI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public RoundMode RoundMode => (RoundMode)((_opcode >> 50) & 0x3); + public bool NegC => (_opcode & 0x2000000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + } + + readonly struct InstDfmaC + { + private readonly ulong _opcode; + public InstDfmaC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public RoundMode RoundMode => (RoundMode)((_opcode >> 50) & 0x3); + public bool NegC => (_opcode & 0x2000000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + } + + readonly struct InstDfmaRc + { + private readonly ulong _opcode; + public InstDfmaRc(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public RoundMode RoundMode => (RoundMode)((_opcode >> 50) & 0x3); + public bool NegC => (_opcode & 0x2000000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + } + + readonly struct InstDmnmxR + { + private readonly ulong _opcode; + public InstDmnmxR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool AbsA => (_opcode & 0x400000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + } + + readonly struct InstDmnmxI + { + private readonly ulong _opcode; + public InstDmnmxI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool AbsA => (_opcode & 0x400000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + } + + readonly struct InstDmnmxC + { + private readonly ulong _opcode; + public InstDmnmxC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool AbsA => (_opcode & 0x400000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + } + + readonly struct InstDmulR + { + private readonly ulong _opcode; + public InstDmulR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public RoundMode RoundMode => (RoundMode)((_opcode >> 39) & 0x3); + public bool NegA => (_opcode & 0x1000000000000) != 0; + } + + readonly struct InstDmulI + { + private readonly ulong _opcode; + public InstDmulI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public RoundMode RoundMode => (RoundMode)((_opcode >> 39) & 0x3); + public bool NegA => (_opcode & 0x1000000000000) != 0; + } + + readonly struct InstDmulC + { + private readonly ulong _opcode; + public InstDmulC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public RoundMode RoundMode => (RoundMode)((_opcode >> 39) & 0x3); + public bool NegA => (_opcode & 0x1000000000000) != 0; + } + + readonly struct InstDsetR + { + private readonly ulong _opcode; + public InstDsetR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool AbsA => (_opcode & 0x40000000000000) != 0; + public bool NegB => (_opcode & 0x20000000000000) != 0; + public bool BVal => (_opcode & 0x10000000000000) != 0; + public FComp FComp => (FComp)((_opcode >> 48) & 0xF); + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public bool AbsB => (_opcode & 0x100000000000) != 0; + public bool NegA => (_opcode & 0x80000000000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + } + + readonly struct InstDsetI + { + private readonly ulong _opcode; + public InstDsetI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool AbsA => (_opcode & 0x40000000000000) != 0; + public bool NegB => (_opcode & 0x20000000000000) != 0; + public bool BVal => (_opcode & 0x10000000000000) != 0; + public FComp FComp => (FComp)((_opcode >> 48) & 0xF); + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public bool AbsB => (_opcode & 0x100000000000) != 0; + public bool NegA => (_opcode & 0x80000000000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + } + + readonly struct InstDsetC + { + private readonly ulong _opcode; + public InstDsetC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool AbsA => (_opcode & 0x40000000000000) != 0; + public bool NegB => (_opcode & 0x20000000000000) != 0; + public bool BVal => (_opcode & 0x10000000000000) != 0; + public FComp FComp => (FComp)((_opcode >> 48) & 0xF); + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public bool AbsB => (_opcode & 0x100000000000) != 0; + public bool NegA => (_opcode & 0x80000000000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + } + + readonly struct InstDsetpR + { + private readonly ulong _opcode; + public InstDsetpR(ulong opcode) => _opcode = opcode; + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public FComp FComp => (FComp)((_opcode >> 48) & 0xF); + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public bool AbsB => (_opcode & 0x100000000000) != 0; + public bool NegA => (_opcode & 0x80000000000) != 0; + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool AbsA => (_opcode & 0x80) != 0; + public bool NegB => (_opcode & 0x40) != 0; + public int DestPred => (int)((_opcode >> 3) & 0x7); + public int DestPredInv => (int)(_opcode & 0x7); + } + + readonly struct InstDsetpI + { + private readonly ulong _opcode; + public InstDsetpI(ulong opcode) => _opcode = opcode; + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public FComp FComp => (FComp)((_opcode >> 48) & 0xF); + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public bool AbsB => (_opcode & 0x100000000000) != 0; + public bool NegA => (_opcode & 0x80000000000) != 0; + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool AbsA => (_opcode & 0x80) != 0; + public bool NegB => (_opcode & 0x40) != 0; + public int DestPred => (int)((_opcode >> 3) & 0x7); + public int DestPredInv => (int)(_opcode & 0x7); + } + + readonly struct InstDsetpC + { + private readonly ulong _opcode; + public InstDsetpC(ulong opcode) => _opcode = opcode; + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public FComp FComp => (FComp)((_opcode >> 48) & 0xF); + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public bool AbsB => (_opcode & 0x100000000000) != 0; + public bool NegA => (_opcode & 0x80000000000) != 0; + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool AbsA => (_opcode & 0x80) != 0; + public bool NegB => (_opcode & 0x40) != 0; + public int DestPred => (int)((_opcode >> 3) & 0x7); + public int DestPredInv => (int)(_opcode & 0x7); + } + + readonly struct InstExit + { + private readonly ulong _opcode; + public InstExit(ulong opcode) => _opcode = opcode; + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Ccc Ccc => (Ccc)(_opcode & 0x1F); + public bool KeepRefCnt => (_opcode & 0x20) != 0; + } + + readonly struct InstF2fR + { + private readonly ulong _opcode; + public InstF2fR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public bool Ftz => (_opcode & 0x100000000000) != 0; + public DstFmt DstFmt => (DstFmt)((_opcode >> 8) & 0x3); + public DstFmt SrcFmt => (DstFmt)((_opcode >> 10) & 0x3); + public IntegerRound RoundMode => (IntegerRound)((int)((_opcode >> 40) & 0x4) | (int)((_opcode >> 39) & 0x3)); + public bool Sh => (_opcode & 0x20000000000) != 0; + public bool Sat => (_opcode & 0x4000000000000) != 0; + } + + readonly struct InstF2fI + { + private readonly ulong _opcode; + public InstF2fI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public bool Ftz => (_opcode & 0x100000000000) != 0; + public DstFmt DstFmt => (DstFmt)((_opcode >> 8) & 0x3); + public DstFmt SrcFmt => (DstFmt)((_opcode >> 10) & 0x3); + public IntegerRound RoundMode => (IntegerRound)((int)((_opcode >> 40) & 0x4) | (int)((_opcode >> 39) & 0x3)); + public bool Sh => (_opcode & 0x20000000000) != 0; + public bool Sat => (_opcode & 0x4000000000000) != 0; + } + + readonly struct InstF2fC + { + private readonly ulong _opcode; + public InstF2fC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public bool Ftz => (_opcode & 0x100000000000) != 0; + public DstFmt DstFmt => (DstFmt)((_opcode >> 8) & 0x3); + public DstFmt SrcFmt => (DstFmt)((_opcode >> 10) & 0x3); + public IntegerRound RoundMode => (IntegerRound)((int)((_opcode >> 40) & 0x4) | (int)((_opcode >> 39) & 0x3)); + public bool Sh => (_opcode & 0x20000000000) != 0; + public bool Sat => (_opcode & 0x4000000000000) != 0; + } + + readonly struct InstF2iR + { + private readonly ulong _opcode; + public InstF2iR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public bool Ftz => (_opcode & 0x100000000000) != 0; + public bool Sh => (_opcode & 0x20000000000) != 0; + public IDstFmt IDstFmt => (IDstFmt)((int)((_opcode >> 10) & 0x4) | (int)((_opcode >> 8) & 0x3)); + public DstFmt SrcFmt => (DstFmt)((_opcode >> 10) & 0x3); + public RoundMode2 RoundMode => (RoundMode2)((_opcode >> 39) & 0x3); + } + + readonly struct InstF2iI + { + private readonly ulong _opcode; + public InstF2iI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public bool Ftz => (_opcode & 0x100000000000) != 0; + public bool Sh => (_opcode & 0x20000000000) != 0; + public IDstFmt IDstFmt => (IDstFmt)((int)((_opcode >> 10) & 0x4) | (int)((_opcode >> 8) & 0x3)); + public DstFmt SrcFmt => (DstFmt)((_opcode >> 10) & 0x3); + public RoundMode2 RoundMode => (RoundMode2)((_opcode >> 39) & 0x3); + } + + readonly struct InstF2iC + { + private readonly ulong _opcode; + public InstF2iC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public bool Ftz => (_opcode & 0x100000000000) != 0; + public bool Sh => (_opcode & 0x20000000000) != 0; + public IDstFmt IDstFmt => (IDstFmt)((int)((_opcode >> 10) & 0x4) | (int)((_opcode >> 8) & 0x3)); + public DstFmt SrcFmt => (DstFmt)((_opcode >> 10) & 0x3); + public RoundMode2 RoundMode => (RoundMode2)((_opcode >> 39) & 0x3); + } + + readonly struct InstFaddR + { + private readonly ulong _opcode; + public InstFaddR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Sat => (_opcode & 0x4000000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool AbsA => (_opcode & 0x400000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public bool Ftz => (_opcode & 0x100000000000) != 0; + public RoundMode RoundMode => (RoundMode)((_opcode >> 39) & 0x3); + } + + readonly struct InstFaddI + { + private readonly ulong _opcode; + public InstFaddI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Sat => (_opcode & 0x4000000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool AbsA => (_opcode & 0x400000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public bool Ftz => (_opcode & 0x100000000000) != 0; + public RoundMode RoundMode => (RoundMode)((_opcode >> 39) & 0x3); + } + + readonly struct InstFaddC + { + private readonly ulong _opcode; + public InstFaddC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Sat => (_opcode & 0x4000000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool AbsA => (_opcode & 0x400000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public bool Ftz => (_opcode & 0x100000000000) != 0; + public RoundMode RoundMode => (RoundMode)((_opcode >> 39) & 0x3); + } + + readonly struct InstFadd32i + { + private readonly ulong _opcode; + public InstFadd32i(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x10000000000000) != 0; + public int Imm32 => (int)(_opcode >> 20); + public bool AbsB => (_opcode & 0x200000000000000) != 0; + public bool NegA => (_opcode & 0x100000000000000) != 0; + public bool Ftz => (_opcode & 0x80000000000000) != 0; + public bool AbsA => (_opcode & 0x40000000000000) != 0; + public bool NegB => (_opcode & 0x20000000000000) != 0; + } + + readonly struct InstFchkR + { + private readonly ulong _opcode; + public InstFchkR(ulong opcode) => _opcode = opcode; + public int DestPred => (int)((_opcode >> 3) & 0x7); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool AbsA => (_opcode & 0x400000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public ChkModeF ChkModeF => (ChkModeF)((_opcode >> 39) & 0x3F); + } + + readonly struct InstFchkI + { + private readonly ulong _opcode; + public InstFchkI(ulong opcode) => _opcode = opcode; + public int DestPred => (int)((_opcode >> 3) & 0x7); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool AbsA => (_opcode & 0x400000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public ChkModeF ChkModeF => (ChkModeF)((_opcode >> 39) & 0x3F); + } + + readonly struct InstFchkC + { + private readonly ulong _opcode; + public InstFchkC(ulong opcode) => _opcode = opcode; + public int DestPred => (int)((_opcode >> 3) & 0x7); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool AbsA => (_opcode & 0x400000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public ChkModeF ChkModeF => (ChkModeF)((_opcode >> 39) & 0x3F); + } + + readonly struct InstFcmpR + { + private readonly ulong _opcode; + public InstFcmpR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public FComp FComp => (FComp)((_opcode >> 48) & 0xF); + public bool Ftz => (_opcode & 0x800000000000) != 0; + } + + readonly struct InstFcmpI + { + private readonly ulong _opcode; + public InstFcmpI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public FComp FComp => (FComp)((_opcode >> 48) & 0xF); + public bool Ftz => (_opcode & 0x800000000000) != 0; + } + + readonly struct InstFcmpC + { + private readonly ulong _opcode; + public InstFcmpC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public FComp FComp => (FComp)((_opcode >> 48) & 0xF); + public bool Ftz => (_opcode & 0x800000000000) != 0; + } + + readonly struct InstFcmpRc + { + private readonly ulong _opcode; + public InstFcmpRc(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public FComp FComp => (FComp)((_opcode >> 48) & 0xF); + public bool Ftz => (_opcode & 0x800000000000) != 0; + } + + readonly struct InstFfmaR + { + private readonly ulong _opcode; + public InstFfmaR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool NegC => (_opcode & 0x2000000000000) != 0; + public bool Sat => (_opcode & 0x4000000000000) != 0; + public RoundMode RoundMode => (RoundMode)((_opcode >> 51) & 0x3); + public Fmz Fmz => (Fmz)((_opcode >> 53) & 0x3); + } + + readonly struct InstFfmaI + { + private readonly ulong _opcode; + public InstFfmaI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool NegC => (_opcode & 0x2000000000000) != 0; + public bool Sat => (_opcode & 0x4000000000000) != 0; + public RoundMode RoundMode => (RoundMode)((_opcode >> 51) & 0x3); + public Fmz Fmz => (Fmz)((_opcode >> 53) & 0x3); + } + + readonly struct InstFfmaC + { + private readonly ulong _opcode; + public InstFfmaC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool NegC => (_opcode & 0x2000000000000) != 0; + public bool Sat => (_opcode & 0x4000000000000) != 0; + public RoundMode RoundMode => (RoundMode)((_opcode >> 51) & 0x3); + public Fmz Fmz => (Fmz)((_opcode >> 53) & 0x3); + } + + readonly struct InstFfmaRc + { + private readonly ulong _opcode; + public InstFfmaRc(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool NegC => (_opcode & 0x2000000000000) != 0; + public bool Sat => (_opcode & 0x4000000000000) != 0; + public RoundMode RoundMode => (RoundMode)((_opcode >> 51) & 0x3); + public Fmz Fmz => (Fmz)((_opcode >> 53) & 0x3); + } + + readonly struct InstFfma32i + { + private readonly ulong _opcode; + public InstFfma32i(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm32 => (int)(_opcode >> 20); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool NegC => (_opcode & 0x200000000000000) != 0; + public bool NegA => (_opcode & 0x100000000000000) != 0; + public bool Sat => (_opcode & 0x80000000000000) != 0; + public bool WriteCC => (_opcode & 0x10000000000000) != 0; + public Fmz Fmz => (Fmz)((_opcode >> 53) & 0x3); + } + + readonly struct InstFloR + { + private readonly ulong _opcode; + public InstFloR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Signed => (_opcode & 0x1000000000000) != 0; + public bool Sh => (_opcode & 0x20000000000) != 0; + public bool NegB => (_opcode & 0x10000000000) != 0; + } + + readonly struct InstFloI + { + private readonly ulong _opcode; + public InstFloI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Signed => (_opcode & 0x1000000000000) != 0; + public bool Sh => (_opcode & 0x20000000000) != 0; + public bool NegB => (_opcode & 0x10000000000) != 0; + } + + readonly struct InstFloC + { + private readonly ulong _opcode; + public InstFloC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Signed => (_opcode & 0x1000000000000) != 0; + public bool Sh => (_opcode & 0x20000000000) != 0; + public bool NegB => (_opcode & 0x10000000000) != 0; + } + + readonly struct InstFmnmxR + { + private readonly ulong _opcode; + public InstFmnmxR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool AbsA => (_opcode & 0x400000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public bool Ftz => (_opcode & 0x100000000000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + } + + readonly struct InstFmnmxI + { + private readonly ulong _opcode; + public InstFmnmxI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool AbsA => (_opcode & 0x400000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public bool Ftz => (_opcode & 0x100000000000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + } + + readonly struct InstFmnmxC + { + private readonly ulong _opcode; + public InstFmnmxC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool AbsA => (_opcode & 0x400000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public bool Ftz => (_opcode & 0x100000000000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + } + + readonly struct InstFmulR + { + private readonly ulong _opcode; + public InstFmulR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public RoundMode RoundMode => (RoundMode)((_opcode >> 39) & 0x3); + public Fmz Fmz => (Fmz)((_opcode >> 44) & 0x3); + public MultiplyScale Scale => (MultiplyScale)((_opcode >> 41) & 0x7); + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool Sat => (_opcode & 0x4000000000000) != 0; + } + + readonly struct InstFmulI + { + private readonly ulong _opcode; + public InstFmulI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public RoundMode RoundMode => (RoundMode)((_opcode >> 39) & 0x3); + public Fmz Fmz => (Fmz)((_opcode >> 44) & 0x3); + public MultiplyScale Scale => (MultiplyScale)((_opcode >> 41) & 0x7); + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool Sat => (_opcode & 0x4000000000000) != 0; + } + + readonly struct InstFmulC + { + private readonly ulong _opcode; + public InstFmulC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public RoundMode RoundMode => (RoundMode)((_opcode >> 39) & 0x3); + public Fmz Fmz => (Fmz)((_opcode >> 44) & 0x3); + public MultiplyScale Scale => (MultiplyScale)((_opcode >> 41) & 0x7); + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool Sat => (_opcode & 0x4000000000000) != 0; + } + + readonly struct InstFmul32i + { + private readonly ulong _opcode; + public InstFmul32i(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int Imm32 => (int)(_opcode >> 20); + public bool Sat => (_opcode & 0x80000000000000) != 0; + public Fmz Fmz => (Fmz)((_opcode >> 53) & 0x3); + public bool WriteCC => (_opcode & 0x10000000000000) != 0; + } + + readonly struct InstFsetR + { + private readonly ulong _opcode; + public InstFsetR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool NegA => (_opcode & 0x80000000000) != 0; + public bool NegB => (_opcode & 0x20000000000000) != 0; + public bool AbsA => (_opcode & 0x40000000000000) != 0; + public bool AbsB => (_opcode & 0x100000000000) != 0; + public FComp FComp => (FComp)((_opcode >> 48) & 0xF); + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public bool Ftz => (_opcode & 0x80000000000000) != 0; + public bool BVal => (_opcode & 0x10000000000000) != 0; + } + + readonly struct InstFsetC + { + private readonly ulong _opcode; + public InstFsetC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool NegA => (_opcode & 0x80000000000) != 0; + public bool NegB => (_opcode & 0x20000000000000) != 0; + public bool AbsA => (_opcode & 0x40000000000000) != 0; + public bool AbsB => (_opcode & 0x100000000000) != 0; + public FComp FComp => (FComp)((_opcode >> 48) & 0xF); + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public bool Ftz => (_opcode & 0x80000000000000) != 0; + public bool BVal => (_opcode & 0x10000000000000) != 0; + } + + readonly struct InstFsetI + { + private readonly ulong _opcode; + public InstFsetI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool NegA => (_opcode & 0x80000000000) != 0; + public bool NegB => (_opcode & 0x20000000000000) != 0; + public bool AbsA => (_opcode & 0x40000000000000) != 0; + public bool AbsB => (_opcode & 0x100000000000) != 0; + public FComp FComp => (FComp)((_opcode >> 48) & 0xF); + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public bool Ftz => (_opcode & 0x80000000000000) != 0; + public bool BVal => (_opcode & 0x10000000000000) != 0; + } + + readonly struct InstFsetpR + { + private readonly ulong _opcode; + public InstFsetpR(ulong opcode) => _opcode = opcode; + public int DestPred => (int)((_opcode >> 3) & 0x7); + public int DestPredInv => (int)(_opcode & 0x7); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool NegA => (_opcode & 0x80000000000) != 0; + public bool NegB => (_opcode & 0x40) != 0; + public bool AbsA => (_opcode & 0x80) != 0; + public bool AbsB => (_opcode & 0x100000000000) != 0; + public FComp FComp => (FComp)((_opcode >> 48) & 0xF); + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public bool Ftz => (_opcode & 0x800000000000) != 0; + } + + readonly struct InstFsetpI + { + private readonly ulong _opcode; + public InstFsetpI(ulong opcode) => _opcode = opcode; + public int DestPred => (int)((_opcode >> 3) & 0x7); + public int DestPredInv => (int)(_opcode & 0x7); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool NegA => (_opcode & 0x80000000000) != 0; + public bool NegB => (_opcode & 0x40) != 0; + public bool AbsA => (_opcode & 0x80) != 0; + public bool AbsB => (_opcode & 0x100000000000) != 0; + public FComp FComp => (FComp)((_opcode >> 48) & 0xF); + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public bool Ftz => (_opcode & 0x800000000000) != 0; + } + + readonly struct InstFsetpC + { + private readonly ulong _opcode; + public InstFsetpC(ulong opcode) => _opcode = opcode; + public int DestPred => (int)((_opcode >> 3) & 0x7); + public int DestPredInv => (int)(_opcode & 0x7); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool NegA => (_opcode & 0x80000000000) != 0; + public bool NegB => (_opcode & 0x40) != 0; + public bool AbsA => (_opcode & 0x80) != 0; + public bool AbsB => (_opcode & 0x100000000000) != 0; + public FComp FComp => (FComp)((_opcode >> 48) & 0xF); + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public bool Ftz => (_opcode & 0x800000000000) != 0; + } + + readonly struct InstFswzadd + { + private readonly ulong _opcode; + public InstFswzadd(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Ftz => (_opcode & 0x100000000000) != 0; + public RoundMode RoundMode => (RoundMode)((_opcode >> 39) & 0x3); + public bool Ndv => (_opcode & 0x4000000000) != 0; + public int PnWord => (int)((_opcode >> 28) & 0xFF); + } + + readonly struct InstGetcrsptr + { + private readonly ulong _opcode; + public InstGetcrsptr(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + } + + readonly struct InstGetlmembase + { + private readonly ulong _opcode; + public InstGetlmembase(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + } + + readonly struct InstHadd2R + { + private readonly ulong _opcode; + public InstHadd2R(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public OFmt OFmt => (OFmt)((_opcode >> 49) & 0x3); + public HalfSwizzle ASwizzle => (HalfSwizzle)((_opcode >> 47) & 0x3); + public HalfSwizzle BSwizzle => (HalfSwizzle)((_opcode >> 28) & 0x3); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool NegA => (_opcode & 0x80000000000) != 0; + public bool NegB => (_opcode & 0x80000000) != 0; + public bool AbsA => (_opcode & 0x100000000000) != 0; + public bool AbsB => (_opcode & 0x40000000) != 0; + public bool Sat => (_opcode & 0x100000000) != 0; + public bool Ftz => (_opcode & 0x8000000000) != 0; + } + + readonly struct InstHadd2I + { + private readonly ulong _opcode; + public InstHadd2I(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int BimmH0 => (int)((_opcode >> 20) & 0x3FF); + public int BimmH1 => (int)((_opcode >> 47) & 0x200) | (int)((_opcode >> 30) & 0x1FF); + public OFmt OFmt => (OFmt)((_opcode >> 49) & 0x3); + public HalfSwizzle ASwizzle => (HalfSwizzle)((_opcode >> 47) & 0x3); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool NegA => (_opcode & 0x80000000000) != 0; + public bool AbsA => (_opcode & 0x100000000000) != 0; + public bool Sat => (_opcode & 0x10000000000000) != 0; + public bool Ftz => (_opcode & 0x8000000000) != 0; + } + + readonly struct InstHadd2C + { + private readonly ulong _opcode; + public InstHadd2C(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public OFmt OFmt => (OFmt)((_opcode >> 49) & 0x3); + public HalfSwizzle ASwizzle => (HalfSwizzle)((_opcode >> 47) & 0x3); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool NegA => (_opcode & 0x80000000000) != 0; + public bool NegB => (_opcode & 0x100000000000000) != 0; + public bool AbsA => (_opcode & 0x100000000000) != 0; + public bool AbsB => (_opcode & 0x40000000000000) != 0; + public bool Sat => (_opcode & 0x10000000000000) != 0; + public bool Ftz => (_opcode & 0x8000000000) != 0; + } + + readonly struct InstHadd232i + { + private readonly ulong _opcode; + public InstHadd232i(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm => (int)(_opcode >> 20); + public HalfSwizzle ASwizzle => (HalfSwizzle)((_opcode >> 53) & 0x3); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool NegA => (_opcode & 0x100000000000000) != 0; + public bool Sat => (_opcode & 0x10000000000000) != 0; + public bool Ftz => (_opcode & 0x80000000000000) != 0; + } + + readonly struct InstHfma2R + { + private readonly ulong _opcode; + public InstHfma2R(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public OFmt OFmt => (OFmt)((_opcode >> 49) & 0x3); + public HalfSwizzle ASwizzle => (HalfSwizzle)((_opcode >> 47) & 0x3); + public HalfSwizzle BSwizzle => (HalfSwizzle)((_opcode >> 28) & 0x3); + public HalfSwizzle CSwizzle => (HalfSwizzle)((_opcode >> 35) & 0x3); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool NegA => (_opcode & 0x80000000) != 0; + public bool NegC => (_opcode & 0x40000000) != 0; + public bool Sat => (_opcode & 0x100000000) != 0; + public Fmz Fmz => (Fmz)((_opcode >> 37) & 0x3); + } + + readonly struct InstHfma2I + { + private readonly ulong _opcode; + public InstHfma2I(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int BimmH0 => (int)((_opcode >> 20) & 0x3FF); + public int BimmH1 => (int)((_opcode >> 47) & 0x200) | (int)((_opcode >> 30) & 0x1FF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public OFmt OFmt => (OFmt)((_opcode >> 49) & 0x3); + public HalfSwizzle ASwizzle => (HalfSwizzle)((_opcode >> 47) & 0x3); + public HalfSwizzle CSwizzle => (HalfSwizzle)((_opcode >> 53) & 0x3); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool NegC => (_opcode & 0x8000000000000) != 0; + public bool Sat => (_opcode & 0x10000000000000) != 0; + public Fmz Fmz => (Fmz)((_opcode >> 57) & 0x3); + } + + readonly struct InstHfma2C + { + private readonly ulong _opcode; + public InstHfma2C(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public OFmt OFmt => (OFmt)((_opcode >> 49) & 0x3); + public HalfSwizzle ASwizzle => (HalfSwizzle)((_opcode >> 47) & 0x3); + public HalfSwizzle CSwizzle => (HalfSwizzle)((_opcode >> 53) & 0x3); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool NegA => (_opcode & 0x100000000000000) != 0; + public bool NegC => (_opcode & 0x8000000000000) != 0; + public bool Sat => (_opcode & 0x10000000000000) != 0; + public Fmz Fmz => (Fmz)((_opcode >> 57) & 0x3); + } + + readonly struct InstHfma2Rc + { + private readonly ulong _opcode; + public InstHfma2Rc(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public OFmt OFmt => (OFmt)((_opcode >> 49) & 0x3); + public HalfSwizzle ASwizzle => (HalfSwizzle)((_opcode >> 47) & 0x3); + public HalfSwizzle CSwizzle => (HalfSwizzle)((_opcode >> 53) & 0x3); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool NegA => (_opcode & 0x100000000000000) != 0; + public bool NegC => (_opcode & 0x8000000000000) != 0; + public bool Sat => (_opcode & 0x10000000000000) != 0; + public Fmz Fmz => (Fmz)((_opcode >> 57) & 0x3); + } + + readonly struct InstHfma232i + { + private readonly ulong _opcode; + public InstHfma232i(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm => (int)(_opcode >> 20); + public HalfSwizzle ASwizzle => (HalfSwizzle)((_opcode >> 47) & 0x3); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool NegC => (_opcode & 0x8000000000000) != 0; + public Fmz Fmz => (Fmz)((_opcode >> 57) & 0x3); + } + + readonly struct InstHmul2R + { + private readonly ulong _opcode; + public InstHmul2R(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public OFmt OFmt => (OFmt)((_opcode >> 49) & 0x3); + public HalfSwizzle ASwizzle => (HalfSwizzle)((_opcode >> 47) & 0x3); + public HalfSwizzle BSwizzle => (HalfSwizzle)((_opcode >> 28) & 0x3); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool NegA => (_opcode & 0x80000000) != 0; + public bool AbsA => (_opcode & 0x100000000000) != 0; + public bool AbsB => (_opcode & 0x40000000) != 0; + public bool Sat => (_opcode & 0x100000000) != 0; + public Fmz Fmz => (Fmz)((_opcode >> 39) & 0x3); + } + + readonly struct InstHmul2I + { + private readonly ulong _opcode; + public InstHmul2I(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int BimmH0 => (int)((_opcode >> 20) & 0x3FF); + public int BimmH1 => (int)((_opcode >> 47) & 0x200) | (int)((_opcode >> 30) & 0x1FF); + public OFmt OFmt => (OFmt)((_opcode >> 49) & 0x3); + public HalfSwizzle ASwizzle => (HalfSwizzle)((_opcode >> 47) & 0x3); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool NegA => (_opcode & 0x80000000000) != 0; + public bool AbsA => (_opcode & 0x100000000000) != 0; + public bool Sat => (_opcode & 0x10000000000000) != 0; + public Fmz Fmz => (Fmz)((_opcode >> 39) & 0x3); + } + + readonly struct InstHmul2C + { + private readonly ulong _opcode; + public InstHmul2C(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public OFmt OFmt => (OFmt)((_opcode >> 49) & 0x3); + public HalfSwizzle ASwizzle => (HalfSwizzle)((_opcode >> 47) & 0x3); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool NegA => (_opcode & 0x80000000000) != 0; + public bool AbsA => (_opcode & 0x100000000000) != 0; + public bool AbsB => (_opcode & 0x40000000000000) != 0; + public bool Sat => (_opcode & 0x10000000000000) != 0; + public Fmz Fmz => (Fmz)((_opcode >> 39) & 0x3); + } + + readonly struct InstHmul232i + { + private readonly ulong _opcode; + public InstHmul232i(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm32 => (int)(_opcode >> 20); + public HalfSwizzle ASwizzle => (HalfSwizzle)((_opcode >> 53) & 0x3); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool Sat => (_opcode & 0x10000000000000) != 0; + public Fmz Fmz => (Fmz)((_opcode >> 55) & 0x3); + } + + readonly struct InstHset2R + { + private readonly ulong _opcode; + public InstHset2R(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public HalfSwizzle ASwizzle => (HalfSwizzle)((_opcode >> 47) & 0x3); + public HalfSwizzle BSwizzle => (HalfSwizzle)((_opcode >> 28) & 0x3); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool NegA => (_opcode & 0x80000000000) != 0; + public bool AbsA => (_opcode & 0x100000000000) != 0; + public bool NegB => (_opcode & 0x80000000) != 0; + public bool AbsB => (_opcode & 0x40000000) != 0; + public bool Bval => (_opcode & 0x2000000000000) != 0; + public FComp Cmp => (FComp)((_opcode >> 35) & 0xF); + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public bool Ftz => (_opcode & 0x4000000000000) != 0; + } + + readonly struct InstHset2I + { + private readonly ulong _opcode; + public InstHset2I(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int BimmH0 => (int)((_opcode >> 20) & 0x3FF); + public int BimmH1 => (int)((_opcode >> 47) & 0x200) | (int)((_opcode >> 30) & 0x1FF); + public HalfSwizzle ASwizzle => (HalfSwizzle)((_opcode >> 47) & 0x3); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool NegA => (_opcode & 0x80000000000) != 0; + public bool AbsA => (_opcode & 0x100000000000) != 0; + public bool Bval => (_opcode & 0x20000000000000) != 0; + public FComp Cmp => (FComp)((_opcode >> 49) & 0xF); + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public bool Ftz => (_opcode & 0x40000000000000) != 0; + } + + readonly struct InstHset2C + { + private readonly ulong _opcode; + public InstHset2C(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public HalfSwizzle ASwizzle => (HalfSwizzle)((_opcode >> 47) & 0x3); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool NegA => (_opcode & 0x80000000000) != 0; + public bool AbsA => (_opcode & 0x100000000000) != 0; + public bool NegB => (_opcode & 0x100000000000000) != 0; + public bool Bval => (_opcode & 0x20000000000000) != 0; + public FComp Cmp => (FComp)((_opcode >> 49) & 0xF); + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public bool Ftz => (_opcode & 0x40000000000000) != 0; + } + + readonly struct InstHsetp2R + { + private readonly ulong _opcode; + public InstHsetp2R(ulong opcode) => _opcode = opcode; + public int DestPred => (int)((_opcode >> 3) & 0x7); + public int DestPredInv => (int)(_opcode & 0x7); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool NegA => (_opcode & 0x80000000000) != 0; + public bool NegB => (_opcode & 0x80000000) != 0; + public bool AbsA => (_opcode & 0x100000000000) != 0; + public bool AbsB => (_opcode & 0x40000000) != 0; + public FComp FComp2 => (FComp)((_opcode >> 35) & 0xF); + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public bool Ftz => (_opcode & 0x40) != 0; + public bool HAnd => (_opcode & 0x2000000000000) != 0; + public HalfSwizzle ASwizzle => (HalfSwizzle)((_opcode >> 47) & 0x3); + public HalfSwizzle BSwizzle => (HalfSwizzle)((_opcode >> 28) & 0x3); + } + + readonly struct InstHsetp2I + { + private readonly ulong _opcode; + public InstHsetp2I(ulong opcode) => _opcode = opcode; + public int DestPred => (int)((_opcode >> 3) & 0x7); + public int DestPredInv => (int)(_opcode & 0x7); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int BimmH0 => (int)((_opcode >> 20) & 0x3FF); + public int BimmH1 => (int)((_opcode >> 47) & 0x200) | (int)((_opcode >> 30) & 0x1FF); + public bool NegA => (_opcode & 0x80000000000) != 0; + public bool AbsA => (_opcode & 0x100000000000) != 0; + public FComp FComp => (FComp)((_opcode >> 49) & 0xF); + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public bool Ftz => (_opcode & 0x40) != 0; + public bool HAnd => (_opcode & 0x20000000000000) != 0; + public HalfSwizzle ASwizzle => (HalfSwizzle)((_opcode >> 47) & 0x3); + } + + readonly struct InstHsetp2C + { + private readonly ulong _opcode; + public InstHsetp2C(ulong opcode) => _opcode = opcode; + public int DestPred => (int)((_opcode >> 3) & 0x7); + public int DestPredInv => (int)(_opcode & 0x7); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool NegA => (_opcode & 0x80000000000) != 0; + public bool NegB => (_opcode & 0x100000000000000) != 0; + public bool AbsA => (_opcode & 0x100000000000) != 0; + public bool AbsB => (_opcode & 0x40000000000000) != 0; + public FComp FComp => (FComp)((_opcode >> 49) & 0xF); + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public bool Ftz => (_opcode & 0x40) != 0; + public bool HAnd => (_opcode & 0x20000000000000) != 0; + public HalfSwizzle ASwizzle => (HalfSwizzle)((_opcode >> 47) & 0x3); + } + + readonly struct InstI2fR + { + private readonly ulong _opcode; + public InstI2fR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public ByteSel ByteSel => (ByteSel)((_opcode >> 41) & 0x3); + public RoundMode RoundMode => (RoundMode)((_opcode >> 39) & 0x3); + public ISrcFmt ISrcFmt => (ISrcFmt)((int)((_opcode >> 11) & 0x4) | (int)((_opcode >> 10) & 0x3)); + public DstFmt DstFmt => (DstFmt)((_opcode >> 8) & 0x3); + } + + readonly struct InstI2fI + { + private readonly ulong _opcode; + public InstI2fI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public ByteSel ByteSel => (ByteSel)((_opcode >> 41) & 0x3); + public RoundMode RoundMode => (RoundMode)((_opcode >> 39) & 0x3); + public ISrcFmt ISrcFmt => (ISrcFmt)((int)((_opcode >> 11) & 0x4) | (int)((_opcode >> 10) & 0x3)); + public DstFmt DstFmt => (DstFmt)((_opcode >> 8) & 0x3); + } + + readonly struct InstI2fC + { + private readonly ulong _opcode; + public InstI2fC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public ByteSel ByteSel => (ByteSel)((_opcode >> 41) & 0x3); + public RoundMode RoundMode => (RoundMode)((_opcode >> 39) & 0x3); + public ISrcFmt ISrcFmt => (ISrcFmt)((int)((_opcode >> 11) & 0x4) | (int)((_opcode >> 10) & 0x3)); + public DstFmt DstFmt => (DstFmt)((_opcode >> 8) & 0x3); + } + + readonly struct InstI2iR + { + private readonly ulong _opcode; + public InstI2iR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Sat => (_opcode & 0x4000000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public ByteSel ByteSel => (ByteSel)((_opcode >> 41) & 0x3); + public ISrcDstFmt IDstFmt => (ISrcDstFmt)((int)((_opcode >> 10) & 0x4) | (int)((_opcode >> 8) & 0x3)); + public ISrcDstFmt ISrcFmt => (ISrcDstFmt)((int)((_opcode >> 11) & 0x4) | (int)((_opcode >> 10) & 0x3)); + } + + readonly struct InstI2iI + { + private readonly ulong _opcode; + public InstI2iI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Sat => (_opcode & 0x4000000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public ByteSel ByteSel => (ByteSel)((_opcode >> 41) & 0x3); + public ISrcDstFmt IDstFmt => (ISrcDstFmt)((int)((_opcode >> 10) & 0x4) | (int)((_opcode >> 8) & 0x3)); + public ISrcDstFmt ISrcFmt => (ISrcDstFmt)((int)((_opcode >> 11) & 0x4) | (int)((_opcode >> 10) & 0x3)); + } + + readonly struct InstI2iC + { + private readonly ulong _opcode; + public InstI2iC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Sat => (_opcode & 0x4000000000000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public ByteSel ByteSel => (ByteSel)((_opcode >> 41) & 0x3); + public ISrcDstFmt IDstFmt => (ISrcDstFmt)((int)((_opcode >> 10) & 0x4) | (int)((_opcode >> 8) & 0x3)); + public ISrcDstFmt ISrcFmt => (ISrcDstFmt)((int)((_opcode >> 11) & 0x4) | (int)((_opcode >> 10) & 0x3)); + } + + readonly struct InstIaddR + { + private readonly ulong _opcode; + public InstIaddR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Sat => (_opcode & 0x4000000000000) != 0; + public AvgMode AvgMode => (AvgMode)((_opcode >> 48) & 0x3); + public bool X => (_opcode & 0x80000000000) != 0; + } + + readonly struct InstIaddI + { + private readonly ulong _opcode; + public InstIaddI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Sat => (_opcode & 0x4000000000000) != 0; + public AvgMode AvgMode => (AvgMode)((_opcode >> 48) & 0x3); + public bool X => (_opcode & 0x80000000000) != 0; + } + + readonly struct InstIaddC + { + private readonly ulong _opcode; + public InstIaddC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Sat => (_opcode & 0x4000000000000) != 0; + public AvgMode AvgMode => (AvgMode)((_opcode >> 48) & 0x3); + public bool X => (_opcode & 0x80000000000) != 0; + } + + readonly struct InstIadd32i + { + private readonly ulong _opcode; + public InstIadd32i(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int Imm32 => (int)(_opcode >> 20); + public AvgMode AvgMode => (AvgMode)((_opcode >> 55) & 0x3); + public bool Sat => (_opcode & 0x40000000000000) != 0; + public bool WriteCC => (_opcode & 0x10000000000000) != 0; + public bool X => (_opcode & 0x20000000000000) != 0; + } + + readonly struct InstIadd3R + { + private readonly ulong _opcode; + public InstIadd3R(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool NegA => (_opcode & 0x8000000000000) != 0; + public bool NegB => (_opcode & 0x4000000000000) != 0; + public bool NegC => (_opcode & 0x2000000000000) != 0; + public bool X => (_opcode & 0x1000000000000) != 0; + public Lrs Lrs => (Lrs)((_opcode >> 37) & 0x3); + public HalfSelect Apart => (HalfSelect)((_opcode >> 35) & 0x3); + public HalfSelect Bpart => (HalfSelect)((_opcode >> 33) & 0x3); + public HalfSelect Cpart => (HalfSelect)((_opcode >> 31) & 0x3); + } + + readonly struct InstIadd3I + { + private readonly ulong _opcode; + public InstIadd3I(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool NegA => (_opcode & 0x8000000000000) != 0; + public bool NegB => (_opcode & 0x4000000000000) != 0; + public bool NegC => (_opcode & 0x2000000000000) != 0; + public bool X => (_opcode & 0x1000000000000) != 0; + } + + readonly struct InstIadd3C + { + private readonly ulong _opcode; + public InstIadd3C(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool NegA => (_opcode & 0x8000000000000) != 0; + public bool NegB => (_opcode & 0x4000000000000) != 0; + public bool NegC => (_opcode & 0x2000000000000) != 0; + public bool X => (_opcode & 0x1000000000000) != 0; + } + + readonly struct InstIcmpR + { + private readonly ulong _opcode; + public InstIcmpR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public IComp IComp => (IComp)((_opcode >> 49) & 0x7); + public bool Signed => (_opcode & 0x1000000000000) != 0; + } + + readonly struct InstIcmpI + { + private readonly ulong _opcode; + public InstIcmpI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public IComp IComp => (IComp)((_opcode >> 49) & 0x7); + public bool Signed => (_opcode & 0x1000000000000) != 0; + } + + readonly struct InstIcmpC + { + private readonly ulong _opcode; + public InstIcmpC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public IComp IComp => (IComp)((_opcode >> 49) & 0x7); + public bool Signed => (_opcode & 0x1000000000000) != 0; + } + + readonly struct InstIcmpRc + { + private readonly ulong _opcode; + public InstIcmpRc(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public IComp IComp => (IComp)((_opcode >> 49) & 0x7); + public bool Signed => (_opcode & 0x1000000000000) != 0; + } + + readonly struct InstIde + { + private readonly ulong _opcode; + public InstIde(ulong opcode) => _opcode = opcode; + public int Imm16 => (int)((_opcode >> 20) & 0xFFFF); + public bool Di => (_opcode & 0x20) != 0; + } + + readonly struct InstIdpR + { + private readonly ulong _opcode; + public InstIdpR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool IsHi => (_opcode & 0x4000000000000) != 0; + public bool SrcASign => (_opcode & 0x2000000000000) != 0; + public bool IsDp => (_opcode & 0x1000000000000) != 0; + public bool SrcBSign => (_opcode & 0x800000000000) != 0; + } + + readonly struct InstIdpC + { + private readonly ulong _opcode; + public InstIdpC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool IsHi => (_opcode & 0x4000000000000) != 0; + public bool SrcASign => (_opcode & 0x2000000000000) != 0; + public bool IsDp => (_opcode & 0x1000000000000) != 0; + public bool SrcBSign => (_opcode & 0x800000000000) != 0; + } + + readonly struct InstImadR + { + private readonly ulong _opcode; + public InstImadR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Hilo => (_opcode & 0x40000000000000) != 0; + public bool BSigned => (_opcode & 0x20000000000000) != 0; + public AvgMode AvgMode => (AvgMode)((_opcode >> 51) & 0x3); + public bool Sat => (_opcode & 0x4000000000000) != 0; + public bool X => (_opcode & 0x2000000000000) != 0; + public bool ASigned => (_opcode & 0x1000000000000) != 0; + } + + readonly struct InstImadI + { + private readonly ulong _opcode; + public InstImadI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Hilo => (_opcode & 0x40000000000000) != 0; + public bool BSigned => (_opcode & 0x20000000000000) != 0; + public AvgMode AvgMode => (AvgMode)((_opcode >> 51) & 0x3); + public bool Sat => (_opcode & 0x4000000000000) != 0; + public bool X => (_opcode & 0x2000000000000) != 0; + public bool ASigned => (_opcode & 0x1000000000000) != 0; + } + + readonly struct InstImadC + { + private readonly ulong _opcode; + public InstImadC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Hilo => (_opcode & 0x40000000000000) != 0; + public bool BSigned => (_opcode & 0x20000000000000) != 0; + public AvgMode AvgMode => (AvgMode)((_opcode >> 51) & 0x3); + public bool Sat => (_opcode & 0x4000000000000) != 0; + public bool X => (_opcode & 0x2000000000000) != 0; + public bool ASigned => (_opcode & 0x1000000000000) != 0; + } + + readonly struct InstImadRc + { + private readonly ulong _opcode; + public InstImadRc(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Hilo => (_opcode & 0x40000000000000) != 0; + public bool BSigned => (_opcode & 0x20000000000000) != 0; + public AvgMode AvgMode => (AvgMode)((_opcode >> 51) & 0x3); + public bool Sat => (_opcode & 0x4000000000000) != 0; + public bool X => (_opcode & 0x2000000000000) != 0; + public bool ASigned => (_opcode & 0x1000000000000) != 0; + } + + readonly struct InstImad32i + { + private readonly ulong _opcode; + public InstImad32i(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int Imm32 => (int)(_opcode >> 20); + public bool BSigned => (_opcode & 0x200000000000000) != 0; + public AvgMode AvgMode => (AvgMode)((_opcode >> 55) & 0x3); + public bool ASigned => (_opcode & 0x40000000000000) != 0; + public bool WriteCC => (_opcode & 0x10000000000000) != 0; + public bool Hilo => (_opcode & 0x20000000000000) != 0; + } + + readonly struct InstImadspR + { + private readonly ulong _opcode; + public InstImadspR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public ImadspASelect ASelect => (ImadspASelect)((_opcode >> 48) & 0x7); + public ImadspBSelect BSelect => (ImadspBSelect)((_opcode >> 53) & 0x3); + public ImadspASelect CSelect => (ImadspASelect)((int)((_opcode >> 50) & 0x6) | (int)((_opcode >> 48) & 0x1)); + } + + readonly struct InstImadspI + { + private readonly ulong _opcode; + public InstImadspI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public ImadspASelect ASelect => (ImadspASelect)((_opcode >> 48) & 0x7); + public ImadspBSelect BSelect => (ImadspBSelect)((_opcode >> 53) & 0x3); + public ImadspASelect CSelect => (ImadspASelect)((int)((_opcode >> 50) & 0x6) | (int)((_opcode >> 48) & 0x1)); + } + + readonly struct InstImadspC + { + private readonly ulong _opcode; + public InstImadspC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public ImadspASelect ASelect => (ImadspASelect)((_opcode >> 48) & 0x7); + public ImadspBSelect BSelect => (ImadspBSelect)((_opcode >> 53) & 0x3); + public ImadspASelect CSelect => (ImadspASelect)((int)((_opcode >> 50) & 0x6) | (int)((_opcode >> 48) & 0x1)); + } + + readonly struct InstImadspRc + { + private readonly ulong _opcode; + public InstImadspRc(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public ImadspASelect ASelect => (ImadspASelect)((_opcode >> 48) & 0x7); + public ImadspBSelect BSelect => (ImadspBSelect)((_opcode >> 53) & 0x3); + public ImadspASelect CSelect => (ImadspASelect)((int)((_opcode >> 50) & 0x6) | (int)((_opcode >> 48) & 0x1)); + } + + readonly struct InstImnmxR + { + private readonly ulong _opcode; + public InstImnmxR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Signed => (_opcode & 0x1000000000000) != 0; + public XMode XMode => (XMode)((_opcode >> 43) & 0x3); + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + } + + readonly struct InstImnmxI + { + private readonly ulong _opcode; + public InstImnmxI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Signed => (_opcode & 0x1000000000000) != 0; + public XMode XMode => (XMode)((_opcode >> 43) & 0x3); + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + } + + readonly struct InstImnmxC + { + private readonly ulong _opcode; + public InstImnmxC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Signed => (_opcode & 0x1000000000000) != 0; + public XMode XMode => (XMode)((_opcode >> 43) & 0x3); + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + } + + readonly struct InstImulR + { + private readonly ulong _opcode; + public InstImulR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool ASigned => (_opcode & 0x10000000000) != 0; + public bool BSigned => (_opcode & 0x20000000000) != 0; + public bool Hilo => (_opcode & 0x8000000000) != 0; + } + + readonly struct InstImulI + { + private readonly ulong _opcode; + public InstImulI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool ASigned => (_opcode & 0x10000000000) != 0; + public bool BSigned => (_opcode & 0x20000000000) != 0; + public bool Hilo => (_opcode & 0x8000000000) != 0; + } + + readonly struct InstImulC + { + private readonly ulong _opcode; + public InstImulC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool ASigned => (_opcode & 0x10000000000) != 0; + public bool BSigned => (_opcode & 0x20000000000) != 0; + public bool Hilo => (_opcode & 0x8000000000) != 0; + } + + readonly struct InstImul32i + { + private readonly ulong _opcode; + public InstImul32i(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int Imm32 => (int)(_opcode >> 20); + public bool ASigned => (_opcode & 0x40000000000000) != 0; + public bool BSigned => (_opcode & 0x80000000000000) != 0; + public bool Hilo => (_opcode & 0x20000000000000) != 0; + public bool WriteCC => (_opcode & 0x10000000000000) != 0; + } + + readonly struct InstIpa + { + private readonly ulong _opcode; + public InstIpa(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public IpaOp IpaOp => (IpaOp)((_opcode >> 54) & 0x3); + public int Msi => (int)((_opcode >> 52) & 0x3); + public bool Sat => (_opcode & 0x8000000000000) != 0; + public bool Idx => (_opcode & 0x4000000000) != 0; + public int Imm10 => (int)((_opcode >> 28) & 0x3FF); + public int SrcPred => (int)((_opcode >> 47) & 0x7); + public bool SrcPredInv => (_opcode & 0x4000000000000) != 0; + } + + readonly struct InstIsberd + { + private readonly ulong _opcode; + public InstIsberd(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public AlSize AlSize => (AlSize)((_opcode >> 47) & 0x3); + public IBase IBase => (IBase)((_opcode >> 33) & 0x3); + public bool O => (_opcode & 0x100000000) != 0; + public bool P => (_opcode & 0x80000000) != 0; + } + + readonly struct InstIscaddR + { + private readonly ulong _opcode; + public InstIscaddR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public int Imm5 => (int)((_opcode >> 39) & 0x1F); + public AvgMode AvgMode => (AvgMode)((_opcode >> 48) & 0x3); + } + + readonly struct InstIscaddI + { + private readonly ulong _opcode; + public InstIscaddI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public int Imm5 => (int)((_opcode >> 39) & 0x1F); + public AvgMode AvgMode => (AvgMode)((_opcode >> 48) & 0x3); + } + + readonly struct InstIscaddC + { + private readonly ulong _opcode; + public InstIscaddC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public int Imm5 => (int)((_opcode >> 39) & 0x1F); + public AvgMode AvgMode => (AvgMode)((_opcode >> 48) & 0x3); + } + + readonly struct InstIscadd32i + { + private readonly ulong _opcode; + public InstIscadd32i(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int Imm32 => (int)(_opcode >> 20); + public bool WriteCC => (_opcode & 0x10000000000000) != 0; + public int Imm5 => (int)((_opcode >> 53) & 0x1F); + } + + readonly struct InstIsetR + { + private readonly ulong _opcode; + public InstIsetR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public IComp IComp => (IComp)((_opcode >> 49) & 0x7); + public bool Signed => (_opcode & 0x1000000000000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public bool BVal => (_opcode & 0x100000000000) != 0; + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public bool X => (_opcode & 0x80000000000) != 0; + } + + readonly struct InstIsetI + { + private readonly ulong _opcode; + public InstIsetI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public IComp IComp => (IComp)((_opcode >> 49) & 0x7); + public bool Signed => (_opcode & 0x1000000000000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public bool BVal => (_opcode & 0x100000000000) != 0; + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public bool X => (_opcode & 0x80000000000) != 0; + } + + readonly struct InstIsetC + { + private readonly ulong _opcode; + public InstIsetC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public IComp IComp => (IComp)((_opcode >> 49) & 0x7); + public bool Signed => (_opcode & 0x1000000000000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public bool BVal => (_opcode & 0x100000000000) != 0; + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public bool X => (_opcode & 0x80000000000) != 0; + } + + readonly struct InstIsetpR + { + private readonly ulong _opcode; + public InstIsetpR(ulong opcode) => _opcode = opcode; + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public IComp IComp => (IComp)((_opcode >> 49) & 0x7); + public bool Signed => (_opcode & 0x1000000000000) != 0; + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public bool X => (_opcode & 0x80000000000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public int DestPred => (int)((_opcode >> 3) & 0x7); + public int DestPredInv => (int)(_opcode & 0x7); + } + + readonly struct InstIsetpI + { + private readonly ulong _opcode; + public InstIsetpI(ulong opcode) => _opcode = opcode; + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public IComp IComp => (IComp)((_opcode >> 49) & 0x7); + public bool Signed => (_opcode & 0x1000000000000) != 0; + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public bool X => (_opcode & 0x80000000000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public int DestPred => (int)((_opcode >> 3) & 0x7); + public int DestPredInv => (int)(_opcode & 0x7); + } + + readonly struct InstIsetpC + { + private readonly ulong _opcode; + public InstIsetpC(ulong opcode) => _opcode = opcode; + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public IComp IComp => (IComp)((_opcode >> 49) & 0x7); + public bool Signed => (_opcode & 0x1000000000000) != 0; + public BoolOp Bop => (BoolOp)((_opcode >> 45) & 0x3); + public bool X => (_opcode & 0x80000000000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public int DestPred => (int)((_opcode >> 3) & 0x7); + public int DestPredInv => (int)(_opcode & 0x7); + } + + readonly struct InstJcal + { + private readonly ulong _opcode; + public InstJcal(ulong opcode) => _opcode = opcode; + public int Imm32 => (int)(_opcode >> 20); + public bool Ca => (_opcode & 0x20) != 0; + public bool Inc => (_opcode & 0x40) != 0; + } + + readonly struct InstJmp + { + private readonly ulong _opcode; + public InstJmp(ulong opcode) => _opcode = opcode; + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Ccc Ccc => (Ccc)(_opcode & 0x1F); + public bool Ca => (_opcode & 0x20) != 0; + public int Imm32 => (int)(_opcode >> 20); + public bool Lmt => (_opcode & 0x40) != 0; + public bool U => (_opcode & 0x80) != 0; + } + + readonly struct InstJmx + { + private readonly ulong _opcode; + public InstJmx(ulong opcode) => _opcode = opcode; + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Ccc Ccc => (Ccc)(_opcode & 0x1F); + public bool Ca => (_opcode & 0x20) != 0; + public int Imm32 => (int)(_opcode >> 20); + public bool Lmt => (_opcode & 0x40) != 0; + } + + readonly struct InstKil + { + private readonly ulong _opcode; + public InstKil(ulong opcode) => _opcode = opcode; + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Ccc Ccc => (Ccc)(_opcode & 0x1F); + } + + readonly struct InstLd + { + private readonly ulong _opcode; + public InstLd(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int SrcPred => (int)((_opcode >> 58) & 0x7); + public CacheOpLd CacheOp => (CacheOpLd)((_opcode >> 56) & 0x3); + public LsSize LsSize => (LsSize)((_opcode >> 53) & 0x7); + public bool E => (_opcode & 0x10000000000000) != 0; + public int Imm32 => (int)(_opcode >> 20); + } + + readonly struct InstLdc + { + private readonly ulong _opcode; + public InstLdc(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public LsSize2 LsSize => (LsSize2)((_opcode >> 48) & 0x7); + public AddressMode AddressMode => (AddressMode)((_opcode >> 44) & 0x3); + public int CbufSlot => (int)((_opcode >> 36) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0xFFFF); + } + + readonly struct InstLdg + { + private readonly ulong _opcode; + public InstLdg(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public LsSize LsSize => (LsSize)((_opcode >> 48) & 0x7); + public CacheOpLd CacheOp => (CacheOpLd)((_opcode >> 46) & 0x3); + public bool E => (_opcode & 0x200000000000) != 0; + public int Imm24 => (int)((_opcode >> 20) & 0xFFFFFF); + } + + readonly struct InstLdl + { + private readonly ulong _opcode; + public InstLdl(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public LsSize2 LsSize => (LsSize2)((_opcode >> 48) & 0x7); + public CacheOp2 CacheOp => (CacheOp2)((_opcode >> 44) & 0x3); + public int Imm24 => (int)((_opcode >> 20) & 0xFFFFFF); + } + + readonly struct InstLds + { + private readonly ulong _opcode; + public InstLds(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public LsSize2 LsSize => (LsSize2)((_opcode >> 48) & 0x7); + public bool U => (_opcode & 0x100000000000) != 0; + public int Imm24 => (int)((_opcode >> 20) & 0xFFFFFF); + } + + readonly struct InstLeaR + { + private readonly ulong _opcode; + public InstLeaR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool X => (_opcode & 0x400000000000) != 0; + public bool NegA => (_opcode & 0x200000000000) != 0; + public int ImmU5 => (int)((_opcode >> 39) & 0x1F); + public int DestPred => (int)((_opcode >> 48) & 0x7); + } + + readonly struct InstLeaI + { + private readonly ulong _opcode; + public InstLeaI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool X => (_opcode & 0x400000000000) != 0; + public bool NegA => (_opcode & 0x200000000000) != 0; + public int ImmU5 => (int)((_opcode >> 39) & 0x1F); + public int DestPred => (int)((_opcode >> 48) & 0x7); + } + + readonly struct InstLeaC + { + private readonly ulong _opcode; + public InstLeaC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool X => (_opcode & 0x400000000000) != 0; + public bool NegA => (_opcode & 0x200000000000) != 0; + public int ImmU5 => (int)((_opcode >> 39) & 0x1F); + public int DestPred => (int)((_opcode >> 48) & 0x7); + } + + readonly struct InstLeaHiR + { + private readonly ulong _opcode; + public InstLeaHiR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool X => (_opcode & 0x4000000000) != 0; + public bool NegA => (_opcode & 0x2000000000) != 0; + public int ImmU5 => (int)((_opcode >> 28) & 0x1F); + public int DestPred => (int)((_opcode >> 48) & 0x7); + } + + readonly struct InstLeaHiC + { + private readonly ulong _opcode; + public InstLeaHiC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool X => (_opcode & 0x200000000000000) != 0; + public bool NegA => (_opcode & 0x100000000000000) != 0; + public int ImmU5 => (int)((_opcode >> 51) & 0x1F); + public int DestPred => (int)((_opcode >> 48) & 0x7); + } + + readonly struct InstLepc + { +#pragma warning disable IDE0052 // Remove unread private member + private readonly ulong _opcode; +#pragma warning restore IDE0052 + public InstLepc(ulong opcode) => _opcode = opcode; + } + + readonly struct InstLongjmp + { + private readonly ulong _opcode; + public InstLongjmp(ulong opcode) => _opcode = opcode; + public Ccc Ccc => (Ccc)(_opcode & 0x1F); + } + + readonly struct InstLopR + { + private readonly ulong _opcode; + public InstLopR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public int DestPred => (int)((_opcode >> 48) & 0x7); + public PredicateOp PredicateOp => (PredicateOp)((_opcode >> 44) & 0x3); + public bool X => (_opcode & 0x80000000000) != 0; + public LogicOp Lop => (LogicOp)((_opcode >> 41) & 0x3); + public bool NegA => (_opcode & 0x8000000000) != 0; + public bool NegB => (_opcode & 0x10000000000) != 0; + } + + readonly struct InstLopI + { + private readonly ulong _opcode; + public InstLopI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public int DestPred => (int)((_opcode >> 48) & 0x7); + public PredicateOp PredicateOp => (PredicateOp)((_opcode >> 44) & 0x3); + public bool X => (_opcode & 0x80000000000) != 0; + public LogicOp LogicOp => (LogicOp)((_opcode >> 41) & 0x3); + public bool NegA => (_opcode & 0x8000000000) != 0; + public bool NegB => (_opcode & 0x10000000000) != 0; + } + + readonly struct InstLopC + { + private readonly ulong _opcode; + public InstLopC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public int DestPred => (int)((_opcode >> 48) & 0x7); + public PredicateOp PredicateOp => (PredicateOp)((_opcode >> 44) & 0x3); + public bool X => (_opcode & 0x80000000000) != 0; + public LogicOp LogicOp => (LogicOp)((_opcode >> 41) & 0x3); + public bool NegA => (_opcode & 0x8000000000) != 0; + public bool NegB => (_opcode & 0x10000000000) != 0; + } + + readonly struct InstLop3R + { + private readonly ulong _opcode; + public InstLop3R(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public int DestPred => (int)((_opcode >> 48) & 0x7); + public PredicateOp PredicateOp => (PredicateOp)((_opcode >> 36) & 0x3); + public bool X => (_opcode & 0x4000000000) != 0; + public int Imm => (int)((_opcode >> 28) & 0xFF); + } + + readonly struct InstLop3I + { + private readonly ulong _opcode; + public InstLop3I(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool X => (_opcode & 0x200000000000000) != 0; + public int Imm => (int)((_opcode >> 48) & 0xFF); + } + + readonly struct InstLop3C + { + private readonly ulong _opcode; + public InstLop3C(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool X => (_opcode & 0x100000000000000) != 0; + public int Imm => (int)((_opcode >> 48) & 0xFF); + } + + readonly struct InstLop32i + { + private readonly ulong _opcode; + public InstLop32i(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x10000000000000) != 0; + public int Imm32 => (int)(_opcode >> 20); + public bool X => (_opcode & 0x200000000000000) != 0; + public LogicOp LogicOp => (LogicOp)((_opcode >> 53) & 0x3); + public bool NegA => (_opcode & 0x80000000000000) != 0; + public bool NegB => (_opcode & 0x100000000000000) != 0; + } + + readonly struct InstMembar + { + private readonly ulong _opcode; + public InstMembar(ulong opcode) => _opcode = opcode; + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Membar Membar => (Membar)((_opcode >> 8) & 0x3); + public Ivall Ivall => (Ivall)(_opcode & 0x3); + } + + readonly struct InstMovR + { + private readonly ulong _opcode; + public InstMovR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int QuadMask => (int)((_opcode >> 39) & 0xF); + } + + readonly struct InstMovI + { + private readonly ulong _opcode; + public InstMovI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int QuadMask => (int)((_opcode >> 39) & 0xF); + } + + readonly struct InstMovC + { + private readonly ulong _opcode; + public InstMovC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int QuadMask => (int)((_opcode >> 39) & 0xF); + } + + readonly struct InstMov32i + { + private readonly ulong _opcode; + public InstMov32i(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int Imm32 => (int)(_opcode >> 20); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int QuadMask => (int)((_opcode >> 12) & 0xF); + } + + readonly struct InstMufu + { + private readonly ulong _opcode; + public InstMufu(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public MufuOp MufuOp => (MufuOp)((_opcode >> 20) & 0xF); + public bool AbsA => (_opcode & 0x400000000000) != 0; + public bool NegA => (_opcode & 0x1000000000000) != 0; + public bool Sat => (_opcode & 0x4000000000000) != 0; + } + + readonly struct InstNop + { + private readonly ulong _opcode; + public InstNop(ulong opcode) => _opcode = opcode; + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int Imm16 => (int)((_opcode >> 20) & 0xFFFF); + public bool Trig => (_opcode & 0x2000) != 0; + public Ccc Ccc => (Ccc)((_opcode >> 8) & 0x1F); + } + + readonly struct InstOutR + { + private readonly ulong _opcode; + public InstOutR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public OutType OutType => (OutType)((_opcode >> 39) & 0x3); + } + + readonly struct InstOutI + { + private readonly ulong _opcode; + public InstOutI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public OutType OutType => (OutType)((_opcode >> 39) & 0x3); + } + + readonly struct InstOutC + { + private readonly ulong _opcode; + public InstOutC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public OutType OutType => (OutType)((_opcode >> 39) & 0x3); + } + + readonly struct InstP2rR + { + private readonly ulong _opcode; + public InstP2rR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public ByteSel ByteSel => (ByteSel)((_opcode >> 41) & 0x3); + public bool Ccpr => (_opcode & 0x10000000000) != 0; + } + + readonly struct InstP2rI + { + private readonly ulong _opcode; + public InstP2rI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public ByteSel ByteSel => (ByteSel)((_opcode >> 41) & 0x3); + public bool Ccpr => (_opcode & 0x10000000000) != 0; + } + + readonly struct InstP2rC + { + private readonly ulong _opcode; + public InstP2rC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public ByteSel ByteSel => (ByteSel)((_opcode >> 41) & 0x3); + public bool Ccpr => (_opcode & 0x10000000000) != 0; + } + + readonly struct InstPbk + { + private readonly ulong _opcode; + public InstPbk(ulong opcode) => _opcode = opcode; + public int Imm24 => (int)((_opcode >> 20) & 0xFFFFFF); + public bool Ca => (_opcode & 0x20) != 0; + } + + readonly struct InstPcnt + { + private readonly ulong _opcode; + public InstPcnt(ulong opcode) => _opcode = opcode; + public int Imm24 => (int)((_opcode >> 20) & 0xFFFFFF); + public bool Ca => (_opcode & 0x20) != 0; + } + + readonly struct InstPexit + { + private readonly ulong _opcode; + public InstPexit(ulong opcode) => _opcode = opcode; + public int Imm24 => (int)((_opcode >> 20) & 0xFFFFFF); + } + + readonly struct InstPixld + { + private readonly ulong _opcode; + public InstPixld(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int DestPred => (int)((_opcode >> 45) & 0x7); + public PixMode PixMode => (PixMode)((_opcode >> 31) & 0x7); + public int Imm8 => (int)((_opcode >> 20) & 0xFF); + } + + readonly struct InstPlongjmp + { + private readonly ulong _opcode; + public InstPlongjmp(ulong opcode) => _opcode = opcode; + public int Imm24 => (int)((_opcode >> 20) & 0xFFFFFF); + public bool Ca => (_opcode & 0x20) != 0; + } + + readonly struct InstPopcR + { + private readonly ulong _opcode; + public InstPopcR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool NegB => (_opcode & 0x10000000000) != 0; + } + + readonly struct InstPopcI + { + private readonly ulong _opcode; + public InstPopcI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool NegB => (_opcode & 0x10000000000) != 0; + } + + readonly struct InstPopcC + { + private readonly ulong _opcode; + public InstPopcC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool NegB => (_opcode & 0x10000000000) != 0; + } + + readonly struct InstPret + { + private readonly ulong _opcode; + public InstPret(ulong opcode) => _opcode = opcode; + public bool Ca => (_opcode & 0x20) != 0; + public int Imm24 => (int)((_opcode >> 20) & 0xFFFFFF); + public bool Inc => (_opcode & 0x40) != 0; + } + + readonly struct InstPrmtR + { + private readonly ulong _opcode; + public InstPrmtR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public PMode PMode => (PMode)((_opcode >> 48) & 0xF); + } + + readonly struct InstPrmtI + { + private readonly ulong _opcode; + public InstPrmtI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public PMode PMode => (PMode)((_opcode >> 48) & 0xF); + } + + readonly struct InstPrmtC + { + private readonly ulong _opcode; + public InstPrmtC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public PMode PMode => (PMode)((_opcode >> 48) & 0xF); + } + + readonly struct InstPrmtRc + { + private readonly ulong _opcode; + public InstPrmtRc(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public PMode PMode => (PMode)((_opcode >> 48) & 0xF); + } + + readonly struct InstPset + { + private readonly ulong _opcode; + public InstPset(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public int Src2Pred => (int)((_opcode >> 12) & 0x7); + public bool Src2PredInv => (_opcode & 0x8000) != 0; + public int Src1Pred => (int)((_opcode >> 29) & 0x7); + public bool Src1PredInv => (_opcode & 0x100000000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public BoolOp BoolOpAB => (BoolOp)((_opcode >> 24) & 0x3); + public BoolOp BoolOpC => (BoolOp)((_opcode >> 45) & 0x3); + public bool BVal => (_opcode & 0x100000000000) != 0; + } + + readonly struct InstPsetp + { + private readonly ulong _opcode; + public InstPsetp(ulong opcode) => _opcode = opcode; + public int DestPred => (int)((_opcode >> 3) & 0x7); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int DestPredInv => (int)(_opcode & 0x7); + public int Src2Pred => (int)((_opcode >> 12) & 0x7); + public bool Src2PredInv => (_opcode & 0x8000) != 0; + public int Src1Pred => (int)((_opcode >> 29) & 0x7); + public bool Src1PredInv => (_opcode & 0x100000000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public BoolOp BoolOpAB => (BoolOp)((_opcode >> 24) & 0x3); + public BoolOp BoolOpC => (BoolOp)((_opcode >> 45) & 0x3); + } + + readonly struct InstR2b + { + private readonly ulong _opcode; + public InstR2b(ulong opcode) => _opcode = opcode; + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public BarMode Mode => (BarMode)((_opcode >> 32) & 0x3); + public int Name => (int)((_opcode >> 28) & 0xF); + } + + readonly struct InstR2pR + { + private readonly ulong _opcode; + public InstR2pR(ulong opcode) => _opcode = opcode; + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public ByteSel ByteSel => (ByteSel)((_opcode >> 41) & 0x3); + public bool Ccpr => (_opcode & 0x10000000000) != 0; + } + + readonly struct InstR2pI + { + private readonly ulong _opcode; + public InstR2pI(ulong opcode) => _opcode = opcode; + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public ByteSel ByteSel => (ByteSel)((_opcode >> 41) & 0x3); + public bool Ccpr => (_opcode & 0x10000000000) != 0; + } + + readonly struct InstR2pC + { + private readonly ulong _opcode; + public InstR2pC(ulong opcode) => _opcode = opcode; + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public ByteSel ByteSel => (ByteSel)((_opcode >> 41) & 0x3); + public bool Ccpr => (_opcode & 0x10000000000) != 0; + } + + readonly struct InstRam + { +#pragma warning disable IDE0052 // Remove unread private member + private readonly ulong _opcode; +#pragma warning restore IDE0052 + public InstRam(ulong opcode) => _opcode = opcode; + } + + readonly struct InstRed + { + private readonly ulong _opcode; + public InstRed(ulong opcode) => _opcode = opcode; + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)(_opcode & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int Imm20 => (int)((_opcode >> 28) & 0xFFFFF); + public AtomSize RedSize => (AtomSize)((_opcode >> 20) & 0x7); + public RedOp RedOp => (RedOp)((_opcode >> 23) & 0x7); + public bool E => (_opcode & 0x1000000000000) != 0; + } + + readonly struct InstRet + { + private readonly ulong _opcode; + public InstRet(ulong opcode) => _opcode = opcode; + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Ccc Ccc => (Ccc)(_opcode & 0x1F); + } + + readonly struct InstRroR + { + private readonly ulong _opcode; + public InstRroR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public bool RroOp => (_opcode & 0x8000000000) != 0; + } + + readonly struct InstRroI + { + private readonly ulong _opcode; + public InstRroI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public bool RroOp => (_opcode & 0x8000000000) != 0; + } + + readonly struct InstRroC + { + private readonly ulong _opcode; + public InstRroC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool AbsB => (_opcode & 0x2000000000000) != 0; + public bool NegB => (_opcode & 0x200000000000) != 0; + public bool RroOp => (_opcode & 0x8000000000) != 0; + } + + readonly struct InstRtt + { +#pragma warning disable IDE0052 // Remove unread private member + private readonly ulong _opcode; +#pragma warning restore IDE0052 + public InstRtt(ulong opcode) => _opcode = opcode; + } + + readonly struct InstS2r + { + private readonly ulong _opcode; + public InstS2r(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public SReg SReg => (SReg)((_opcode >> 20) & 0xFF); + } + + readonly struct InstSam + { +#pragma warning disable IDE0052 // Remove unread private member + private readonly ulong _opcode; +#pragma warning restore IDE0052 + public InstSam(ulong opcode) => _opcode = opcode; + } + + readonly struct InstSelR + { + private readonly ulong _opcode; + public InstSelR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + } + + readonly struct InstSelI + { + private readonly ulong _opcode; + public InstSelI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + } + + readonly struct InstSelC + { + private readonly ulong _opcode; + public InstSelC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + } + + readonly struct InstSetcrsptr + { + private readonly ulong _opcode; + public InstSetcrsptr(ulong opcode) => _opcode = opcode; + public int SrcA => (int)((_opcode >> 8) & 0xFF); + } + + readonly struct InstSetlmembase + { + private readonly ulong _opcode; + public InstSetlmembase(ulong opcode) => _opcode = opcode; + public int SrcA => (int)((_opcode >> 8) & 0xFF); + } + + readonly struct InstShfLR + { + private readonly ulong _opcode; + public InstShfLR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool M => (_opcode & 0x4000000000000) != 0; + public XModeShf XModeShf => (XModeShf)((_opcode >> 48) & 0x3); + public MaxShift MaxShift => (MaxShift)((_opcode >> 37) & 0x3); + } + + readonly struct InstShfRR + { + private readonly ulong _opcode; + public InstShfRR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool M => (_opcode & 0x4000000000000) != 0; + public XModeShf XModeShf => (XModeShf)((_opcode >> 48) & 0x3); + public MaxShift MaxShift => (MaxShift)((_opcode >> 37) & 0x3); + } + + readonly struct InstShfLI + { + private readonly ulong _opcode; + public InstShfLI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool M => (_opcode & 0x4000000000000) != 0; + public XModeShf XModeShf => (XModeShf)((_opcode >> 48) & 0x3); + public MaxShift MaxShift => (MaxShift)((_opcode >> 37) & 0x3); + public int Imm6 => (int)((_opcode >> 20) & 0x3F); + } + + readonly struct InstShfRI + { + private readonly ulong _opcode; + public InstShfRI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool M => (_opcode & 0x4000000000000) != 0; + public XModeShf XModeShf => (XModeShf)((_opcode >> 48) & 0x3); + public MaxShift MaxShift => (MaxShift)((_opcode >> 37) & 0x3); + public int Imm6 => (int)((_opcode >> 20) & 0x3F); + } + + readonly struct InstShfl + { + private readonly ulong _opcode; + public InstShfl(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int SrcBImm => (int)((_opcode >> 20) & 0x1F); + public int SrcCImm => (int)((_opcode >> 34) & 0x1FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public ShflMode ShflMode => (ShflMode)((_opcode >> 30) & 0x3); + public bool CFixShfl => (_opcode & 0x20000000) != 0; + public bool BFixShfl => (_opcode & 0x10000000) != 0; + public int DestPred => (int)((_opcode >> 48) & 0x7); + } + + readonly struct InstShlR + { + private readonly ulong _opcode; + public InstShlR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool X => (_opcode & 0x80000000000) != 0; + public bool M => (_opcode & 0x8000000000) != 0; + } + + readonly struct InstShlI + { + private readonly ulong _opcode; + public InstShlI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool X => (_opcode & 0x80000000000) != 0; + public bool M => (_opcode & 0x8000000000) != 0; + } + + readonly struct InstShlC + { + private readonly ulong _opcode; + public InstShlC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool X => (_opcode & 0x80000000000) != 0; + public bool M => (_opcode & 0x8000000000) != 0; + } + + readonly struct InstShrR + { + private readonly ulong _opcode; + public InstShrR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Signed => (_opcode & 0x1000000000000) != 0; + public XMode XMode => (XMode)((_opcode >> 43) & 0x3); + public bool Brev => (_opcode & 0x10000000000) != 0; + public bool M => (_opcode & 0x8000000000) != 0; + } + + readonly struct InstShrI + { + private readonly ulong _opcode; + public InstShrI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Imm20 => (int)((_opcode >> 37) & 0x80000) | (int)((_opcode >> 20) & 0x7FFFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Signed => (_opcode & 0x1000000000000) != 0; + public XMode XMode => (XMode)((_opcode >> 43) & 0x3); + public bool Brev => (_opcode & 0x10000000000) != 0; + public bool M => (_opcode & 0x8000000000) != 0; + } + + readonly struct InstShrC + { + private readonly ulong _opcode; + public InstShrC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Signed => (_opcode & 0x1000000000000) != 0; + public XMode XMode => (XMode)((_opcode >> 43) & 0x3); + public bool Brev => (_opcode & 0x10000000000) != 0; + public bool M => (_opcode & 0x8000000000) != 0; + } + + readonly struct InstSsy + { + private readonly ulong _opcode; + public InstSsy(ulong opcode) => _opcode = opcode; + public int Imm24 => (int)((_opcode >> 20) & 0xFFFFFF); + public bool Ca => (_opcode & 0x20) != 0; + } + + readonly struct InstSt + { + private readonly ulong _opcode; + public InstSt(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int SrcPred => (int)((_opcode >> 58) & 0x7); + public CacheOpSt CacheOp => (CacheOpSt)((_opcode >> 56) & 0x3); + public LsSize LsSize => (LsSize)((_opcode >> 53) & 0x7); + public bool E => (_opcode & 0x10000000000000) != 0; + public int Imm32 => (int)(_opcode >> 20); + } + + readonly struct InstStg + { + private readonly ulong _opcode; + public InstStg(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public LsSize2 LsSize => (LsSize2)((_opcode >> 48) & 0x7); + public CacheOpSt CacheOp => (CacheOpSt)((_opcode >> 46) & 0x3); + public bool E => (_opcode & 0x200000000000) != 0; + public int Imm24 => (int)((_opcode >> 20) & 0xFFFFFF); + } + + readonly struct InstStl + { + private readonly ulong _opcode; + public InstStl(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public LsSize2 LsSize => (LsSize2)((_opcode >> 48) & 0x7); + public CacheOpSt CacheOp => (CacheOpSt)((_opcode >> 44) & 0x3); + public int Imm24 => (int)((_opcode >> 20) & 0xFFFFFF); + } + + readonly struct InstStp + { + private readonly ulong _opcode; + public InstStp(ulong opcode) => _opcode = opcode; + public bool Wait => (_opcode & 0x80000000) != 0; + public int Imm8 => (int)((_opcode >> 20) & 0xFF); + } + + readonly struct InstSts + { + private readonly ulong _opcode; + public InstSts(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public LsSize2 LsSize => (LsSize2)((_opcode >> 48) & 0x7); + public int Imm24 => (int)((_opcode >> 20) & 0xFFFFFF); + } + + readonly struct InstSuatomB + { + private readonly ulong _opcode; + public InstSuatomB(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Clamp Clamp => (Clamp)((_opcode >> 49) & 0x3); + public SuatomSize Size => (SuatomSize)((_opcode >> 36) & 0x7); + public SuDim Dim => (SuDim)((_opcode >> 33) & 0x7); + public SuatomOp Op => (SuatomOp)((_opcode >> 29) & 0xF); + public bool Ba => (_opcode & 0x10000000) != 0; + } + + readonly struct InstSuatom + { + private readonly ulong _opcode; + public InstSuatom(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public SuatomSize Size => (SuatomSize)((_opcode >> 51) & 0x7); + public Clamp Clamp => (Clamp)((_opcode >> 49) & 0x3); + public int TidB => (int)((_opcode >> 36) & 0x1FFF); + public SuDim Dim => (SuDim)((_opcode >> 33) & 0x7); + public SuatomOp Op => (SuatomOp)((_opcode >> 29) & 0xF); + public bool Ba => (_opcode & 0x10000000) != 0; + } + + readonly struct InstSuatomB2 + { + private readonly ulong _opcode; + public InstSuatomB2(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int DestPred => (int)((_opcode >> 51) & 0x7); + public Clamp Clamp => (Clamp)((_opcode >> 49) & 0x3); + public SuatomSize Size => (SuatomSize)((_opcode >> 36) & 0x7); + public SuDim Dim => (SuDim)((_opcode >> 33) & 0x7); + public SuatomOp Op => (SuatomOp)((_opcode >> 29) & 0xF); + public bool Ba => (_opcode & 0x10000000) != 0; + } + + readonly struct InstSuatomCasB + { + private readonly ulong _opcode; + public InstSuatomCasB(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Clamp Clamp => (Clamp)((_opcode >> 49) & 0x3); + public SuatomSize Size => (SuatomSize)((_opcode >> 36) & 0x7); + public SuDim Dim => (SuDim)((_opcode >> 33) & 0x7); + public int DestPred => (int)((_opcode >> 30) & 0x7); + public bool Ba => (_opcode & 0x10000000) != 0; + } + + readonly struct InstSuatomCas + { + private readonly ulong _opcode; + public InstSuatomCas(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public SuatomSize Size => (SuatomSize)((_opcode >> 51) & 0x7); + public Clamp Clamp => (Clamp)((_opcode >> 49) & 0x3); + public int TidB => (int)((_opcode >> 36) & 0x1FFF); + public SuDim Dim => (SuDim)((_opcode >> 33) & 0x7); + public int DestPred => (int)((_opcode >> 30) & 0x7); + public bool Ba => (_opcode & 0x10000000) != 0; + } + + readonly struct InstSuldDB + { + private readonly ulong _opcode; + public InstSuldDB(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Clamp Clamp => (Clamp)((_opcode >> 49) & 0x3); + public SuDim Dim => (SuDim)((_opcode >> 33) & 0x7); + public int DestPred2 => (int)((_opcode >> 30) & 0x7); + public CacheOpLd CacheOp => (CacheOpLd)((_opcode >> 24) & 0x3); + public bool Ba => (_opcode & 0x800000) != 0; + public SuSize Size => (SuSize)((_opcode >> 20) & 0x7); + } + + readonly struct InstSuldD + { + private readonly ulong _opcode; + public InstSuldD(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Clamp Clamp => (Clamp)((_opcode >> 49) & 0x3); + public int TidB => (int)((_opcode >> 36) & 0x1FFF); + public SuDim Dim => (SuDim)((_opcode >> 33) & 0x7); + public int DestPred2 => (int)((_opcode >> 30) & 0x7); + public CacheOpLd CacheOp => (CacheOpLd)((_opcode >> 24) & 0x3); + public bool Ba => (_opcode & 0x800000) != 0; + public SuSize Size => (SuSize)((_opcode >> 20) & 0x7); + } + + readonly struct InstSuldB + { + private readonly ulong _opcode; + public InstSuldB(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Clamp Clamp => (Clamp)((_opcode >> 49) & 0x3); + public SuDim Dim => (SuDim)((_opcode >> 33) & 0x7); + public int DestPred2 => (int)((_opcode >> 30) & 0x7); + public CacheOpLd CacheOp => (CacheOpLd)((_opcode >> 24) & 0x3); + public SuRgba Rgba => (SuRgba)((_opcode >> 20) & 0xF); + } + + readonly struct InstSuld + { + private readonly ulong _opcode; + public InstSuld(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Clamp Clamp => (Clamp)((_opcode >> 49) & 0x3); + public int TidB => (int)((_opcode >> 36) & 0x1FFF); + public SuDim Dim => (SuDim)((_opcode >> 33) & 0x7); + public int DestPred2 => (int)((_opcode >> 30) & 0x7); + public CacheOpLd CacheOp => (CacheOpLd)((_opcode >> 24) & 0x3); + public SuRgba Rgba => (SuRgba)((_opcode >> 20) & 0xF); + } + + readonly struct InstSuredB + { + private readonly ulong _opcode; + public InstSuredB(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Clamp Clamp => (Clamp)((_opcode >> 49) & 0x3); + public SuDim Dim => (SuDim)((_opcode >> 33) & 0x7); + public RedOp Op => (RedOp)((_opcode >> 24) & 0x7); + public bool Ba => (_opcode & 0x800000) != 0; + public SuatomSize Size => (SuatomSize)((_opcode >> 20) & 0x7); + } + + readonly struct InstSured + { + private readonly ulong _opcode; + public InstSured(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Clamp Clamp => (Clamp)((_opcode >> 49) & 0x3); + public int TidB => (int)((_opcode >> 36) & 0x1FFF); + public SuDim Dim => (SuDim)((_opcode >> 33) & 0x7); + public RedOp Op => (RedOp)((_opcode >> 24) & 0x7); + public bool Ba => (_opcode & 0x800000) != 0; + public SuatomSize Size => (SuatomSize)((_opcode >> 20) & 0x7); + } + + readonly struct InstSustDB + { + private readonly ulong _opcode; + public InstSustDB(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Clamp Clamp => (Clamp)((_opcode >> 49) & 0x3); + public SuDim Dim => (SuDim)((_opcode >> 33) & 0x7); + public CacheOpSt CacheOp => (CacheOpSt)((_opcode >> 24) & 0x3); + public bool Ba => (_opcode & 0x800000) != 0; + public SuSize Size => (SuSize)((_opcode >> 20) & 0x7); + } + + readonly struct InstSustD + { + private readonly ulong _opcode; + public InstSustD(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Clamp Clamp => (Clamp)((_opcode >> 49) & 0x3); + public int TidB => (int)((_opcode >> 36) & 0x1FFF); + public SuDim Dim => (SuDim)((_opcode >> 33) & 0x7); + public CacheOpSt CacheOp => (CacheOpSt)((_opcode >> 24) & 0x3); + public bool Ba => (_opcode & 0x800000) != 0; + public SuSize Size => (SuSize)((_opcode >> 20) & 0x7); + } + + readonly struct InstSustB + { + private readonly ulong _opcode; + public InstSustB(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Clamp Clamp => (Clamp)((_opcode >> 49) & 0x3); + public SuDim Dim => (SuDim)((_opcode >> 33) & 0x7); + public CacheOpSt CacheOp => (CacheOpSt)((_opcode >> 24) & 0x3); + public SuRgba Rgba => (SuRgba)((_opcode >> 20) & 0xF); + } + + readonly struct InstSust + { + private readonly ulong _opcode; + public InstSust(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Clamp Clamp => (Clamp)((_opcode >> 49) & 0x3); + public int TidB => (int)((_opcode >> 36) & 0x1FFF); + public SuDim Dim => (SuDim)((_opcode >> 33) & 0x7); + public CacheOpSt CacheOp => (CacheOpSt)((_opcode >> 24) & 0x3); + public SuRgba Rgba => (SuRgba)((_opcode >> 20) & 0xF); + } + + readonly struct InstSync + { + private readonly ulong _opcode; + public InstSync(ulong opcode) => _opcode = opcode; + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public Ccc Ccc => (Ccc)(_opcode & 0x1F); + } + + readonly struct InstTex + { + private readonly ulong _opcode; + public InstTex(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool Lc => (_opcode & 0x400000000000000) != 0; + public int DestPred => (int)((_opcode >> 51) & 0x7); + public int TidB => (int)((_opcode >> 36) & 0x1FFF); + public Lod Lod => (Lod)((_opcode >> 55) & 0x7); + public bool Aoffi => (_opcode & 0x40000000000000) != 0; + public bool Dc => (_opcode & 0x4000000000000) != 0; + public bool Ndv => (_opcode & 0x800000000) != 0; + public TexDim Dim => (TexDim)((_opcode >> 28) & 0x7); + public int WMask => (int)((_opcode >> 31) & 0xF); + public bool Nodep => (_opcode & 0x2000000000000) != 0; + } + + readonly struct InstTexB + { + private readonly ulong _opcode; + public InstTexB(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool Lcb => (_opcode & 0x10000000000) != 0; + public int DestPred => (int)((_opcode >> 51) & 0x7); + public Lod Lodb => (Lod)((_opcode >> 37) & 0x7); + public bool Aoffib => (_opcode & 0x1000000000) != 0; + public bool Dc => (_opcode & 0x4000000000000) != 0; + public bool Ndv => (_opcode & 0x800000000) != 0; + public TexDim Dim => (TexDim)((_opcode >> 28) & 0x7); + public int WMask => (int)((_opcode >> 31) & 0xF); + public bool Nodep => (_opcode & 0x2000000000000) != 0; + } + + readonly struct InstTexs + { + private readonly ulong _opcode; + public InstTexs(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int TidB => (int)((_opcode >> 36) & 0x1FFF); + public TexsTarget Target => (TexsTarget)((_opcode >> 53) & 0xF); + public int WMask => (int)((_opcode >> 50) & 0x7); + public bool Nodep => (_opcode & 0x2000000000000) != 0; + public int Dest2 => (int)((_opcode >> 28) & 0xFF); + } + + readonly struct InstTld + { + private readonly ulong _opcode; + public InstTld(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int TidB => (int)((_opcode >> 36) & 0x1FFF); + public int WMask => (int)((_opcode >> 31) & 0xF); + public bool Lod => (_opcode & 0x80000000000000) != 0; + public bool Toff => (_opcode & 0x800000000) != 0; + public bool Ms => (_opcode & 0x4000000000000) != 0; + public bool Cl => (_opcode & 0x40000000000000) != 0; + public bool Nodep => (_opcode & 0x2000000000000) != 0; + public int DestPred => (int)((_opcode >> 51) & 0x7); + public TexDim Dim => (TexDim)((_opcode >> 28) & 0x7); + } + + readonly struct InstTldB + { + private readonly ulong _opcode; + public InstTldB(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int WMask => (int)((_opcode >> 31) & 0xF); + public bool Lod => (_opcode & 0x80000000000000) != 0; + public bool Toff => (_opcode & 0x800000000) != 0; + public bool Ms => (_opcode & 0x4000000000000) != 0; + public bool Cl => (_opcode & 0x40000000000000) != 0; + public bool Nodep => (_opcode & 0x2000000000000) != 0; + public int DestPred => (int)((_opcode >> 51) & 0x7); + public TexDim Dim => (TexDim)((_opcode >> 28) & 0x7); + } + + readonly struct InstTlds + { + private readonly ulong _opcode; + public InstTlds(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int TidB => (int)((_opcode >> 36) & 0x1FFF); + public TldsTarget Target => (TldsTarget)((_opcode >> 53) & 0xF); + public int WMask => (int)((_opcode >> 50) & 0x7); + public bool Nodep => (_opcode & 0x2000000000000) != 0; + public int Dest2 => (int)((_opcode >> 28) & 0xFF); + } + + readonly struct InstTld4 + { + private readonly ulong _opcode; + public InstTld4(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool Lc => (_opcode & 0x400000000000000) != 0; + public int DestPred => (int)((_opcode >> 51) & 0x7); + public int TidB => (int)((_opcode >> 36) & 0x1FFF); + public TexComp TexComp => (TexComp)((_opcode >> 56) & 0x3); + public TexOffset Toff => (TexOffset)((_opcode >> 54) & 0x3); + public bool Dc => (_opcode & 0x4000000000000) != 0; + public bool Ndv => (_opcode & 0x800000000) != 0; + public TexDim Dim => (TexDim)((_opcode >> 28) & 0x7); + public int WMask => (int)((_opcode >> 31) & 0xF); + public bool Nodep => (_opcode & 0x2000000000000) != 0; + } + + readonly struct InstTld4B + { + private readonly ulong _opcode; + public InstTld4B(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool Lc => (_opcode & 0x10000000000) != 0; + public int DestPred => (int)((_opcode >> 51) & 0x7); + public TexComp TexComp => (TexComp)((_opcode >> 38) & 0x3); + public TexOffset Toff => (TexOffset)((_opcode >> 36) & 0x3); + public bool Dc => (_opcode & 0x4000000000000) != 0; + public bool Ndv => (_opcode & 0x800000000) != 0; + public TexDim Dim => (TexDim)((_opcode >> 28) & 0x7); + public int WMask => (int)((_opcode >> 31) & 0xF); + public bool Nodep => (_opcode & 0x2000000000000) != 0; + } + + readonly struct InstTld4s + { + private readonly ulong _opcode; + public InstTld4s(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int TidB => (int)((_opcode >> 36) & 0x1FFF); + public TexComp TexComp => (TexComp)((_opcode >> 52) & 0x3); + public bool Aoffi => (_opcode & 0x8000000000000) != 0; + public bool Dc => (_opcode & 0x4000000000000) != 0; + public bool Nodep => (_opcode & 0x2000000000000) != 0; + public int Dest2 => (int)((_opcode >> 28) & 0xFF); + } + + readonly struct InstTmml + { + private readonly ulong _opcode; + public InstTmml(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool Nodep => (_opcode & 0x2000000000000) != 0; + public int TidB => (int)((_opcode >> 36) & 0x1FFF); + public bool Ndv => (_opcode & 0x800000000) != 0; + public int WMask => (int)((_opcode >> 31) & 0xF); + public TexDim Dim => (TexDim)((_opcode >> 28) & 0x7); + } + + readonly struct InstTmmlB + { + private readonly ulong _opcode; + public InstTmmlB(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool Nodep => (_opcode & 0x2000000000000) != 0; + public bool Ndv => (_opcode & 0x800000000) != 0; + public int WMask => (int)((_opcode >> 31) & 0xF); + public TexDim Dim => (TexDim)((_opcode >> 28) & 0x7); + } + + readonly struct InstTxa + { + private readonly ulong _opcode; + public InstTxa(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool Nodep => (_opcode & 0x2000000000000) != 0; + public int TidB => (int)((_opcode >> 36) & 0x1FFF); + public bool Ndv => (_opcode & 0x800000000) != 0; + public int WMask => (int)((_opcode >> 31) & 0xF); + } + + readonly struct InstTxd + { + private readonly ulong _opcode; + public InstTxd(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int DestPred => (int)((_opcode >> 51) & 0x7); + public bool Lc => (_opcode & 0x4000000000000) != 0; + public bool Nodep => (_opcode & 0x2000000000000) != 0; + public int TidB => (int)((_opcode >> 36) & 0x1FFF); + public bool Toff => (_opcode & 0x800000000) != 0; + public int WMask => (int)((_opcode >> 31) & 0xF); + public TexDim Dim => (TexDim)((_opcode >> 28) & 0x7); + } + + readonly struct InstTxdB + { + private readonly ulong _opcode; + public InstTxdB(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int DestPred => (int)((_opcode >> 51) & 0x7); + public bool Lc => (_opcode & 0x4000000000000) != 0; + public bool Nodep => (_opcode & 0x2000000000000) != 0; + public bool Toff => (_opcode & 0x800000000) != 0; + public int WMask => (int)((_opcode >> 31) & 0xF); + public TexDim Dim => (TexDim)((_opcode >> 28) & 0x7); + } + + readonly struct InstTxq + { + private readonly ulong _opcode; + public InstTxq(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool Nodep => (_opcode & 0x2000000000000) != 0; + public int TidB => (int)((_opcode >> 36) & 0x1FFF); + public int WMask => (int)((_opcode >> 31) & 0xF); + public TexQuery TexQuery => (TexQuery)((_opcode >> 22) & 0x3F); + } + + readonly struct InstTxqB + { + private readonly ulong _opcode; + public InstTxqB(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool Nodep => (_opcode & 0x2000000000000) != 0; + public int WMask => (int)((_opcode >> 31) & 0xF); + public TexQuery TexQuery => (TexQuery)((_opcode >> 22) & 0x3F); + } + + readonly struct InstVabsdiff + { + private readonly ulong _opcode; + public InstVabsdiff(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool DFormat => (_opcode & 0x40000000000000) != 0; + public VectorSelect ASelect => (VectorSelect)((int)((_opcode >> 45) & 0x8) | (int)((_opcode >> 36) & 0x7)); + public VectorSelect BSelect => (VectorSelect)((int)((_opcode >> 46) & 0x8) | (int)((_opcode >> 28) & 0x7)); + public bool Sat => (_opcode & 0x80000000000000) != 0; + public VideoOp VideoOp => (VideoOp)((_opcode >> 51) & 0x7); + public bool BVideo => (_opcode & 0x4000000000000) != 0; + } + + readonly struct InstVabsdiff4 + { + private readonly ulong _opcode; + public InstVabsdiff4(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public VideoRed VRed => (VideoRed)((_opcode >> 53) & 0x3); + public LaneMask4 LaneMask4 => (LaneMask4)((int)((_opcode >> 49) & 0xC) | (int)((_opcode >> 36) & 0x3)); + public bool Sat => (_opcode & 0x4000000000000) != 0; + public bool SrcBFmt => (_opcode & 0x2000000000000) != 0; + public bool SrcAFmt => (_opcode & 0x1000000000000) != 0; + public bool DFormat => (_opcode & 0x4000000000) != 0; + public ASelect4 Asel4 => (ASelect4)((_opcode >> 32) & 0xF); + public BSelect4 Bsel4 => (BSelect4)((_opcode >> 28) & 0xF); + } + + readonly struct InstVadd + { + private readonly ulong _opcode; + public InstVadd(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int Imm16 => (int)((_opcode >> 20) & 0xFFFF); + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public AvgMode AvgMode => (AvgMode)((_opcode >> 56) & 0x3); + public bool DFormat => (_opcode & 0x40000000000000) != 0; + public VectorSelect ASelect => (VectorSelect)((int)((_opcode >> 45) & 0x8) | (int)((_opcode >> 36) & 0x7)); + public VectorSelect BSelect => (VectorSelect)((int)((_opcode >> 46) & 0x8) | (int)((_opcode >> 28) & 0x7)); + public bool Sat => (_opcode & 0x80000000000000) != 0; + public VideoOp VideoOp => (VideoOp)((_opcode >> 51) & 0x7); + public bool BVideo => (_opcode & 0x4000000000000) != 0; + } + + readonly struct InstVmad + { + private readonly ulong _opcode; + public InstVmad(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int Imm16 => (int)((_opcode >> 20) & 0xFFFF); + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public VectorSelect ASelect => (VectorSelect)((int)((_opcode >> 45) & 0x8) | (int)((_opcode >> 36) & 0x7)); + public VectorSelect BSelect => (VectorSelect)((int)((_opcode >> 46) & 0x8) | (int)((_opcode >> 28) & 0x7)); + public bool Sat => (_opcode & 0x80000000000000) != 0; + public AvgMode AvgMode => (AvgMode)((_opcode >> 53) & 0x3); + public VideoScale VideoScale => (VideoScale)((_opcode >> 51) & 0x3); + public bool BVideo => (_opcode & 0x4000000000000) != 0; + } + + readonly struct InstVmnmx + { + private readonly ulong _opcode; + public InstVmnmx(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int Imm16 => (int)((_opcode >> 20) & 0xFFFF); + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool DFormat => (_opcode & 0x40000000000000) != 0; + public VectorSelect ASelect => (VectorSelect)((int)((_opcode >> 45) & 0x8) | (int)((_opcode >> 36) & 0x7)); + public VectorSelect BSelect => (VectorSelect)((int)((_opcode >> 46) & 0x8) | (int)((_opcode >> 28) & 0x7)); + public bool Sat => (_opcode & 0x80000000000000) != 0; + public VideoOp VideoOp => (VideoOp)((_opcode >> 51) & 0x7); + public bool Mn => (_opcode & 0x100000000000000) != 0; + public bool BVideo => (_opcode & 0x4000000000000) != 0; + } + + readonly struct InstVote + { + private readonly ulong _opcode; + public InstVote(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public VoteMode VoteMode => (VoteMode)((_opcode >> 48) & 0x3); + public int VpDest => (int)((_opcode >> 45) & 0x7); + } + + readonly struct InstVotevtg + { + private readonly ulong _opcode; + public InstVotevtg(ulong opcode) => _opcode = opcode; + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public VoteMode VoteMode => (VoteMode)((_opcode >> 48) & 0x3); + public int Imm28 => (int)((_opcode >> 20) & 0xFFFFFFF); + } + + readonly struct InstVset + { + private readonly ulong _opcode; + public InstVset(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public IComp VComp => (IComp)((_opcode >> 54) & 0x7); + public VectorSelect ASelect => (VectorSelect)((int)((_opcode >> 45) & 0x8) | (int)((_opcode >> 36) & 0x7)); + public VectorSelect BSelect => (VectorSelect)((int)((_opcode >> 46) & 0x8) | (int)((_opcode >> 28) & 0x7)); + public VideoOp VideoOp => (VideoOp)((_opcode >> 51) & 0x7); + public bool BVideo => (_opcode & 0x4000000000000) != 0; + } + + readonly struct InstVsetp + { + private readonly ulong _opcode; + public InstVsetp(ulong opcode) => _opcode = opcode; + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public int Imm16 => (int)((_opcode >> 20) & 0xFFFF); + public VectorSelect ASelect => (VectorSelect)((int)((_opcode >> 45) & 0x8) | (int)((_opcode >> 36) & 0x7)); + public VectorSelect BSelect => (VectorSelect)((int)((_opcode >> 46) & 0x8) | (int)((_opcode >> 28) & 0x7)); + public IComp VComp => (IComp)((int)((_opcode >> 45) & 0x4) | (int)((_opcode >> 43) & 0x3)); + public BoolOp BoolOp => (BoolOp)((_opcode >> 45) & 0x3); + public int SrcPred => (int)((_opcode >> 39) & 0x7); + public bool SrcPredInv => (_opcode & 0x40000000000) != 0; + public int DestPred => (int)((_opcode >> 3) & 0x7); + public int DestPredInv => (int)(_opcode & 0x7); + public bool BVideo => (_opcode & 0x4000000000000) != 0; + } + + readonly struct InstVshl + { + private readonly ulong _opcode; + public InstVshl(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Mv => (_opcode & 0x2000000000000) != 0; + public bool DFormat => (_opcode & 0x40000000000000) != 0; + public VectorSelect ASelect => (VectorSelect)((int)((_opcode >> 45) & 0x8) | (int)((_opcode >> 36) & 0x7)); + public VectorSelect BSelect => (VectorSelect)((_opcode >> 28) & 0x7); + public bool Sat => (_opcode & 0x80000000000000) != 0; + public VideoOp VideoOp => (VideoOp)((_opcode >> 51) & 0x7); + public bool BVideo => (_opcode & 0x4000000000000) != 0; + } + + readonly struct InstVshr + { + private readonly ulong _opcode; + public InstVshr(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Mv => (_opcode & 0x2000000000000) != 0; + public bool DFormat => (_opcode & 0x40000000000000) != 0; + public VectorSelect ASelect => (VectorSelect)((int)((_opcode >> 45) & 0x8) | (int)((_opcode >> 36) & 0x7)); + public VectorSelect BSelect => (VectorSelect)((_opcode >> 28) & 0x7); + public bool Sat => (_opcode & 0x80000000000000) != 0; + public VideoOp VideoOp => (VideoOp)((_opcode >> 51) & 0x7); + public bool BVideo => (_opcode & 0x4000000000000) != 0; + } + + readonly struct InstXmadR + { + private readonly ulong _opcode; + public InstXmadR(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcB => (int)((_opcode >> 20) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool HiloA => (_opcode & 0x20000000000000) != 0; + public XmadCop XmadCop => (XmadCop)((_opcode >> 50) & 0x7); + public bool BSigned => (_opcode & 0x2000000000000) != 0; + public bool ASigned => (_opcode & 0x1000000000000) != 0; + public bool X => (_opcode & 0x4000000000) != 0; + public bool Mrg => (_opcode & 0x2000000000) != 0; + public bool Psl => (_opcode & 0x1000000000) != 0; + public bool HiloB => (_opcode & 0x800000000) != 0; + } + + readonly struct InstXmadI + { + private readonly ulong _opcode; + public InstXmadI(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public int Imm16 => (int)((_opcode >> 20) & 0xFFFF); + public bool HiloA => (_opcode & 0x20000000000000) != 0; + public XmadCop XmadCop => (XmadCop)((_opcode >> 50) & 0x7); + public bool BSigned => (_opcode & 0x2000000000000) != 0; + public bool ASigned => (_opcode & 0x1000000000000) != 0; + public bool X => (_opcode & 0x4000000000) != 0; + public bool Mrg => (_opcode & 0x2000000000) != 0; + public bool Psl => (_opcode & 0x1000000000) != 0; + } + + readonly struct InstXmadC + { + private readonly ulong _opcode; + public InstXmadC(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool Mrg => (_opcode & 0x100000000000000) != 0; + public bool Psl => (_opcode & 0x80000000000000) != 0; + public bool X => (_opcode & 0x40000000000000) != 0; + public bool HiloA => (_opcode & 0x20000000000000) != 0; + public bool HiloB => (_opcode & 0x10000000000000) != 0; + public XmadCop2 XmadCop => (XmadCop2)((_opcode >> 50) & 0x3); + public bool BSigned => (_opcode & 0x2000000000000) != 0; + public bool ASigned => (_opcode & 0x1000000000000) != 0; + } + + readonly struct InstXmadRc + { + private readonly ulong _opcode; + public InstXmadRc(ulong opcode) => _opcode = opcode; + public int Dest => (int)(_opcode & 0xFF); + public int SrcA => (int)((_opcode >> 8) & 0xFF); + public int SrcC => (int)((_opcode >> 39) & 0xFF); + public int CbufSlot => (int)((_opcode >> 34) & 0x1F); + public int CbufOffset => (int)((_opcode >> 20) & 0x3FFF); + public int Pred => (int)((_opcode >> 16) & 0x7); + public bool PredInv => (_opcode & 0x80000) != 0; + public bool WriteCC => (_opcode & 0x800000000000) != 0; + public bool X => (_opcode & 0x40000000000000) != 0; + public bool HiloA => (_opcode & 0x20000000000000) != 0; + public bool HiloB => (_opcode & 0x10000000000000) != 0; + public XmadCop2 XmadCop => (XmadCop2)((_opcode >> 50) & 0x3); + public bool BSigned => (_opcode & 0x2000000000000) != 0; + public bool ASigned => (_opcode & 0x1000000000000) != 0; + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/InstName.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/InstName.cs new file mode 100644 index 0000000..f0de880 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/InstName.cs @@ -0,0 +1,188 @@ +namespace Ryujinx.Graphics.Shader.Decoders +{ + enum InstName : byte + { + Invalid = 0, + + Al2p, + Ald, + Ast, + Atom, + AtomCas, + Atoms, + AtomsCas, + B2r, + Bar, + Bfe, + Bfi, + Bpt, + Bra, + Brk, + Brx, + Cal, + Cctl, + Cctll, + Cctlt, + Cont, + Cset, + Csetp, + Cs2r, + Dadd, + Depbar, + Dfma, + Dmnmx, + Dmul, + Dset, + Dsetp, + Exit, + F2f, + F2i, + Fadd, + Fadd32i, + Fchk, + Fcmp, + Ffma, + Ffma32i, + Flo, + Fmnmx, + Fmul, + Fmul32i, + Fset, + Fsetp, + Fswzadd, + Getcrsptr, + Getlmembase, + Hadd2, + Hadd232i, + Hfma2, + Hmul2, + Hmul232i, + Hset2, + Hsetp2, + I2f, + I2i, + Iadd, + Iadd32i, + Iadd3, + Icmp, + Ide, + Idp, + Imad, + Imad32i, + Imadsp, + Imnmx, + Imul, + Imul32i, + Ipa, + Isberd, + Iscadd, + Iscadd32i, + Iset, + Isetp, + Jcal, + Jmp, + Jmx, + Kil, + Ld, + Ldc, + Ldg, + Ldl, + Lds, + Lea, + LeaHi, + Lepc, + Longjmp, + Lop, + Lop3, + Lop32i, + Membar, + Mov, + Mov32i, + Mufu, + Nop, + Out, + P2r, + Pbk, + Pcnt, + Pexit, + Pixld, + Plongjmp, + Popc, + Pret, + Prmt, + Pset, + Psetp, + R2b, + R2p, + Ram, + Red, + Ret, + Rro, + Rtt, + S2r, + Sam, + Sel, + Setcrsptr, + Setlmembase, + Shf, + Shf_2, + Shf_3, + Shf_4, + Shfl, + Shl, + Shr, + Ssy, + St, + Stg, + Stl, + Stp, + Sts, + SuatomB, + Suatom, + SuatomB2, + SuatomCasB, + SuatomCas, + SuldDB, + SuldD, + SuldB, + Suld, + SuredB, + Sured, + SustDB, + SustD, + SustB, + Sust, + Sync, + Tex, + TexB, + Texs, + TexsF16, + Tld, + TldB, + Tlds, + TldsF16, + Tld4, + Tld4B, + Tld4s, + Tld4sF16, + Tmml, + TmmlB, + Txa, + Txd, + TxdB, + Txq, + TxqB, + Vabsdiff, + Vabsdiff4, + Vadd, + Vmad, + Vmnmx, + Vote, + Votevtg, + Vset, + Vsetp, + Vshl, + Vshr, + Xmad, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/InstOp.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/InstOp.cs new file mode 100644 index 0000000..c034a55 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/InstOp.cs @@ -0,0 +1,27 @@ +using Ryujinx.Graphics.Shader.Instructions; + +namespace Ryujinx.Graphics.Shader.Decoders +{ + readonly struct InstOp + { + public readonly ulong Address; + public readonly ulong RawOpCode; + public readonly InstEmitter Emitter; + public readonly InstProps Props; + public readonly InstName Name; + + public InstOp(ulong address, ulong rawOpCode, InstName name, InstEmitter emitter, InstProps props) + { + Address = address; + RawOpCode = rawOpCode; + Name = name; + Emitter = emitter; + Props = props; + } + + public ulong GetAbsoluteAddress() + { + return (ulong)((long)Address + (((int)(RawOpCode >> 20) << 8) >> 8) + 8); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/InstProps.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/InstProps.cs new file mode 100644 index 0000000..a12dc3e --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/InstProps.cs @@ -0,0 +1,31 @@ +using System; + +namespace Ryujinx.Graphics.Shader.Decoders +{ + [Flags] + enum InstProps : ushort + { + None = 0, + Rd = 1 << 0, + Rd2 = 1 << 1, + Ra = 1 << 2, + Rb = 1 << 3, + Rb2 = 1 << 4, + Ib = 1 << 5, + Rc = 1 << 6, + + Pd = 1 << 7, + LPd = 2 << 7, + SPd = 3 << 7, + TPd = 4 << 7, + VPd = 5 << 7, + PdMask = 7 << 7, + + Pdn = 1 << 10, + Ps = 1 << 11, + Tex = 1 << 12, + TexB = 1 << 13, + Bra = 1 << 14, + NoPred = 1 << 15, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/InstTable.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/InstTable.cs new file mode 100644 index 0000000..12c051d --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/InstTable.cs @@ -0,0 +1,392 @@ +using Ryujinx.Graphics.Shader.Instructions; +using System; + +namespace Ryujinx.Graphics.Shader.Decoders +{ + static class InstTable + { + private const int EncodingBits = 14; + + private readonly struct TableEntry + { + public InstName Name { get; } + public InstEmitter Emitter { get; } + public InstProps Props { get; } + + public int XBits { get; } + + public TableEntry(InstName name, InstEmitter emitter, InstProps props, int xBits) + { + Name = name; + Emitter = emitter; + Props = props; + XBits = xBits; + } + } + + private static readonly TableEntry[] _opCodes; + + static InstTable() + { + _opCodes = new TableEntry[1 << EncodingBits]; + + #region Instructions +#pragma warning disable IDE0055 // Disable formatting + Add("1110111110100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Al2p, InstEmit.Al2p, InstProps.Rd | InstProps.Ra); + Add("1110111111011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Ald, InstEmit.Ald, InstProps.Rd | InstProps.Ra); + Add("1110111111110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Ast, InstEmit.Ast, InstProps.Ra | InstProps.Rb2 | InstProps.Rc); + Add("11101101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Atom, InstEmit.Atom, InstProps.Rd | InstProps.Ra | InstProps.Rb); + Add("111011101111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.AtomCas, InstEmit.AtomCas, InstProps.Rd | InstProps.Ra | InstProps.Rb); + Add("11101100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Atoms, InstEmit.Atoms, InstProps.Rd | InstProps.Ra | InstProps.Rb); + Add("111011100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.AtomsCas, InstEmit.AtomsCas, InstProps.Rd | InstProps.Ra | InstProps.Rb); + Add("1111000010111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.B2r, InstEmit.B2r, InstProps.Rd | InstProps.Ra | InstProps.VPd); + Add("1111000010101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Bar, InstEmit.Bar, InstProps.Ra | InstProps.Ps); + Add("0101110000000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Bfe, InstEmit.BfeR, InstProps.Rd | InstProps.Ra | InstProps.Rb); + Add("0011100x00000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Bfe, InstEmit.BfeI, InstProps.Rd | InstProps.Ra | InstProps.Ib); + Add("0100110000000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Bfe, InstEmit.BfeC, InstProps.Rd | InstProps.Ra); + Add("0101101111110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Bfi, InstEmit.BfiR, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("0011011x11110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Bfi, InstEmit.BfiI, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.Rc); + Add("0100101111110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Bfi, InstEmit.BfiC, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("0101001111110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Bfi, InstEmit.BfiRc, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("111000111010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Bpt, InstEmit.Bpt, InstProps.NoPred); + Add("111000100100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Bra, InstEmit.Bra, InstProps.Bra); + Add("111000110100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Brk, InstEmit.Brk, InstProps.Bra); + Add("111000100101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Brx, InstEmit.Brx, InstProps.Ra | InstProps.Bra); + Add("111000100110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Cal, InstEmit.Cal, InstProps.Bra | InstProps.NoPred); + Add("11101111011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Cctl, InstEmit.Cctl, InstProps.Ra); + Add("1110111110000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Cctll, InstEmit.Cctll, InstProps.Ra); + Add("1110101111110xx0000000000000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Cctlt, InstEmit.Cctlt); + Add("1110101111101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Cctlt, InstEmit.Cctlt, InstProps.Rc); + Add("111000110101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Cont, InstEmit.Cont, InstProps.Bra); + Add("0101000010011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Cset, InstEmit.Cset, InstProps.Rd | InstProps.Ps); + Add("0101000010100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Csetp, InstEmit.Csetp, InstProps.Pd | InstProps.Pdn | InstProps.Ps); + Add("0101000011001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Cs2r, InstEmit.Cs2r, InstProps.Rd); + Add("0101110001110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Dadd, InstEmit.DaddR, InstProps.Rd | InstProps.Ra | InstProps.Rb); + Add("0011100x01110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Dadd, InstEmit.DaddI, InstProps.Rd | InstProps.Ra | InstProps.Ib); + Add("0100110001110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Dadd, InstEmit.DaddC, InstProps.Rd | InstProps.Ra); + Add("1111000011110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Depbar, InstEmit.Depbar); + Add("010110110111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Dfma, InstEmit.DfmaR, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("0011011x0111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Dfma, InstEmit.DfmaI, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.Rc); + Add("010010110111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Dfma, InstEmit.DfmaC, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("010100110111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Dfma, InstEmit.DfmaRc, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("0101110001010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Dmnmx, InstEmit.DmnmxR, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Ps); + Add("0011100x01010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Dmnmx, InstEmit.DmnmxI, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.Ps); + Add("0100110001010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Dmnmx, InstEmit.DmnmxC, InstProps.Rd | InstProps.Ra | InstProps.Ps); + Add("0101110010000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Dmul, InstEmit.DmulR, InstProps.Rd | InstProps.Ra | InstProps.Rb); + Add("0011100x10000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Dmul, InstEmit.DmulI, InstProps.Rd | InstProps.Ra | InstProps.Ib); + Add("0100110010000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Dmul, InstEmit.DmulC, InstProps.Rd | InstProps.Ra); + Add("010110010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Dset, InstEmit.DsetR, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Ps); + Add("0011001x0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Dset, InstEmit.DsetI, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.Ps); + Add("010010010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Dset, InstEmit.DsetC, InstProps.Rd | InstProps.Ra | InstProps.Ps); + Add("010110111000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Dsetp, InstEmit.DsetpR, InstProps.Ra | InstProps.Rb | InstProps.Pd | InstProps.Pdn | InstProps.Ps); + Add("0011011x1000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Dsetp, InstEmit.DsetpI, InstProps.Ra | InstProps.Ib | InstProps.Pd | InstProps.Pdn | InstProps.Ps); + Add("010010111000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Dsetp, InstEmit.DsetpC, InstProps.Ra | InstProps.Pd | InstProps.Pdn | InstProps.Ps); + Add("111000110000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Exit, InstEmit.Exit, InstProps.Bra); + Add("0101110010101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.F2f, InstEmit.F2fR, InstProps.Rd | InstProps.Rb); + Add("0011100x10101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.F2f, InstEmit.F2fI, InstProps.Rd | InstProps.Ib); + Add("0100110010101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.F2f, InstEmit.F2fC, InstProps.Rd); + Add("0101110010110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.F2i, InstEmit.F2iR, InstProps.Rd | InstProps.Rb); + Add("0011100x10110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.F2i, InstEmit.F2iI, InstProps.Rd | InstProps.Ib); + Add("0100110010110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.F2i, InstEmit.F2iC, InstProps.Rd); + Add("0101110001011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fadd, InstEmit.FaddR, InstProps.Rd | InstProps.Ra | InstProps.Rb); + Add("0011100x01011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fadd, InstEmit.FaddI, InstProps.Rd | InstProps.Ra | InstProps.Ib); + Add("0100110001011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fadd, InstEmit.FaddC, InstProps.Rd | InstProps.Ra); + Add("000010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fadd32i, InstEmit.Fadd32i, InstProps.Rd | InstProps.Ra); + Add("0101110010001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fchk, InstEmit.FchkR, InstProps.Ra | InstProps.Rb | InstProps.Pd); + Add("0011100x10001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fchk, InstEmit.FchkI, InstProps.Ra | InstProps.Ib | InstProps.Pd); + Add("0100110010001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fchk, InstEmit.FchkC, InstProps.Ra | InstProps.Pd); + Add("010110111010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fcmp, InstEmit.FcmpR, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("0011011x1010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fcmp, InstEmit.FcmpI, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.Rc); + Add("010010111010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fcmp, InstEmit.FcmpC, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("010100111010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fcmp, InstEmit.FcmpRc, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("010110011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Ffma, InstEmit.FfmaR, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("0011001x1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Ffma, InstEmit.FfmaI, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.Rc); + Add("010010011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Ffma, InstEmit.FfmaC, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("010100011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Ffma, InstEmit.FfmaRc, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("000011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Ffma32i, InstEmit.Ffma32i, InstProps.Rd | InstProps.Ra); + Add("0101110000110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Flo, InstEmit.FloR, InstProps.Rd | InstProps.Rb); + Add("0011100x00110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Flo, InstEmit.FloI, InstProps.Rd | InstProps.Ib); + Add("0100110000110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Flo, InstEmit.FloC, InstProps.Rd); + Add("0101110001100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fmnmx, InstEmit.FmnmxR, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Ps); + Add("0011100x01100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fmnmx, InstEmit.FmnmxI, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.Ps); + Add("0100110001100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fmnmx, InstEmit.FmnmxC, InstProps.Rd | InstProps.Ra | InstProps.Ps); + Add("0101110001101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fmul, InstEmit.FmulR, InstProps.Rd | InstProps.Ra | InstProps.Rb); + Add("0011100x01101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fmul, InstEmit.FmulI, InstProps.Rd | InstProps.Ra | InstProps.Ib); + Add("0100110001101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fmul, InstEmit.FmulC, InstProps.Rd | InstProps.Ra); + Add("00011110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fmul32i, InstEmit.Fmul32i, InstProps.Rd | InstProps.Ra); + Add("01011000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fset, InstEmit.FsetR, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Ps); + Add("0011000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fset, InstEmit.FsetI, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.Ps); + Add("01001000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fset, InstEmit.FsetC, InstProps.Rd | InstProps.Ra | InstProps.Ps); + Add("010110111011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fsetp, InstEmit.FsetpR, InstProps.Ra | InstProps.Rb | InstProps.Pd | InstProps.Pdn | InstProps.Ps); + Add("0011011x1011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fsetp, InstEmit.FsetpI, InstProps.Ra | InstProps.Ib | InstProps.Pd | InstProps.Pdn | InstProps.Ps); + Add("010010111011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fsetp, InstEmit.FsetpC, InstProps.Ra | InstProps.Pd | InstProps.Pdn | InstProps.Ps); + Add("0101000011111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Fswzadd, InstEmit.Fswzadd, InstProps.Rd | InstProps.Ra | InstProps.Rb); + Add("111000101100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Getcrsptr, InstEmit.Getcrsptr, InstProps.Rd | InstProps.NoPred); + Add("111000101101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Getlmembase, InstEmit.Getlmembase, InstProps.Rd | InstProps.NoPred); + Add("0101110100010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Hadd2, InstEmit.Hadd2R, InstProps.Rd | InstProps.Ra); + Add("0111101x0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Hadd2, InstEmit.Hadd2I, InstProps.Rd | InstProps.Ra | InstProps.Ib); + Add("0111101x1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Hadd2, InstEmit.Hadd2C, InstProps.Rd | InstProps.Ra); + Add("0010110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Hadd232i, InstEmit.Hadd232i, InstProps.Rd | InstProps.Ra); + Add("0101110100000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Hfma2, InstEmit.Hfma2R, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("01110xxx0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Hfma2, InstEmit.Hfma2I, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.Rc); + Add("01110xxx1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Hfma2, InstEmit.Hfma2C, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("01100xxx1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Hfma2, InstEmit.Hfma2Rc, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("0010100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Hfma2, InstEmit.Hfma232i, InstProps.Rd | InstProps.Ra); + Add("0101110100001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Hmul2, InstEmit.Hmul2R, InstProps.Rd | InstProps.Ra | InstProps.Rb); + Add("0111100x0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Hmul2, InstEmit.Hmul2I, InstProps.Rd | InstProps.Ra | InstProps.Ib); + Add("0111100x1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Hmul2, InstEmit.Hmul2C, InstProps.Rd | InstProps.Ra); + Add("0010101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Hmul232i, InstEmit.Hmul232i, InstProps.Rd | InstProps.Ra); + Add("0101110100011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Hset2, InstEmit.Hset2R, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Ps); + Add("0111110x0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Hset2, InstEmit.Hset2I, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.Ps); + Add("0111110x1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Hset2, InstEmit.Hset2C, InstProps.Rd | InstProps.Ra | InstProps.Ps); + Add("0101110100100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Hsetp2, InstEmit.Hsetp2R, InstProps.Ra | InstProps.Rb | InstProps.Pd | InstProps.Pdn | InstProps.Ps); + Add("0111111x0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Hsetp2, InstEmit.Hsetp2I, InstProps.Ra | InstProps.Ib | InstProps.Pd | InstProps.Pdn | InstProps.Ps); + Add("0111111x1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Hsetp2, InstEmit.Hsetp2C, InstProps.Ra | InstProps.Pd | InstProps.Pdn | InstProps.Ps); + Add("0101110010111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.I2f, InstEmit.I2fR, InstProps.Rd | InstProps.Rb); + Add("0011100x10111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.I2f, InstEmit.I2fI, InstProps.Rd | InstProps.Ib); + Add("0100110010111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.I2f, InstEmit.I2fC, InstProps.Rd); + Add("0101110011100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.I2i, InstEmit.I2iR, InstProps.Rd | InstProps.Rb); + Add("0011100x11100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.I2i, InstEmit.I2iI, InstProps.Rd | InstProps.Ib); + Add("0100110011100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.I2i, InstEmit.I2iC, InstProps.Rd); + Add("0101110000010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Iadd, InstEmit.IaddR, InstProps.Rd | InstProps.Ra | InstProps.Rb); + Add("0011100x00010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Iadd, InstEmit.IaddI, InstProps.Rd | InstProps.Ra | InstProps.Ib); + Add("0100110000010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Iadd, InstEmit.IaddC, InstProps.Rd | InstProps.Ra); + Add("0001110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Iadd32i, InstEmit.Iadd32i, InstProps.Rd | InstProps.Ra); + Add("010111001100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Iadd3, InstEmit.Iadd3R, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("0011100x1100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Iadd3, InstEmit.Iadd3I, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.Rc); + Add("010011001100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Iadd3, InstEmit.Iadd3C, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("010110110100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Icmp, InstEmit.IcmpR, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("0011011x0100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Icmp, InstEmit.IcmpI, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.Rc); + Add("010010110100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Icmp, InstEmit.IcmpC, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("010100110100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Icmp, InstEmit.IcmpRc, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("111000111001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Ide, InstEmit.Ide, InstProps.NoPred); + Add("0101001111111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Idp, InstEmit.IdpR, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("0101001111011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Idp, InstEmit.IdpC, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("010110100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Imad, InstEmit.ImadR, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("0011010x0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Imad, InstEmit.ImadI, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.Rc); + Add("010010100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Imad, InstEmit.ImadC, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("010100100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Imad, InstEmit.ImadRc, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("000100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Imad32i, InstEmit.Imad32i, InstProps.Rd | InstProps.Ra); + Add("010110101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Imadsp, InstEmit.ImadspR, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("0011010x1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Imadsp, InstEmit.ImadspI, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.Rc); + Add("010010101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Imadsp, InstEmit.ImadspC, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("010100101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Imadsp, InstEmit.ImadspRc, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("0101110000100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Imnmx, InstEmit.ImnmxR, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Ps); + Add("0011100x00100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Imnmx, InstEmit.ImnmxI, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.Ps); + Add("0100110000100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Imnmx, InstEmit.ImnmxC, InstProps.Rd | InstProps.Ra | InstProps.Ps); + Add("0101110000111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Imul, InstEmit.ImulR, InstProps.Rd | InstProps.Ra | InstProps.Rb); + Add("0011100x00111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Imul, InstEmit.ImulI, InstProps.Rd | InstProps.Ra | InstProps.Ib); + Add("0100110000111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Imul, InstEmit.ImulC, InstProps.Rd | InstProps.Ra); + Add("00011111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Imul32i, InstEmit.Imul32i, InstProps.Rd | InstProps.Ra); + Add("11100000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Ipa, InstEmit.Ipa, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("1110111111010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Isberd, InstEmit.Isberd, InstProps.Rd | InstProps.Ra); + Add("0101110000011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Iscadd, InstEmit.IscaddR, InstProps.Rd | InstProps.Ra | InstProps.Rb); + Add("0011100x00011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Iscadd, InstEmit.IscaddI, InstProps.Rd | InstProps.Ra | InstProps.Ib); + Add("0100110000011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Iscadd, InstEmit.IscaddC, InstProps.Rd | InstProps.Ra); + Add("000101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Iscadd32i, InstEmit.Iscadd32i, InstProps.Rd | InstProps.Ra); + Add("010110110101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Iset, InstEmit.IsetR, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Ps); + Add("0011011x0101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Iset, InstEmit.IsetI, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.Ps); + Add("010010110101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Iset, InstEmit.IsetC, InstProps.Rd | InstProps.Ra | InstProps.Ps); + Add("010110110110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Isetp, InstEmit.IsetpR, InstProps.Ra | InstProps.Rb | InstProps.Pd | InstProps.Pdn | InstProps.Ps); + Add("0011011x0110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Isetp, InstEmit.IsetpI, InstProps.Ra | InstProps.Ib | InstProps.Pd | InstProps.Pdn | InstProps.Ps); + Add("010010110110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Isetp, InstEmit.IsetpC, InstProps.Ra | InstProps.Pd | InstProps.Pdn | InstProps.Ps); + Add("111000100010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Jcal, InstEmit.Jcal, InstProps.Bra); + Add("111000100001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Jmp, InstEmit.Jmp, InstProps.Ra | InstProps.Bra); + Add("111000100000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Jmx, InstEmit.Jmx, InstProps.Ra | InstProps.Bra); + Add("111000110011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Kil, InstEmit.Kil, InstProps.Bra); + Add("100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Ld, InstEmit.Ld, InstProps.Rd | InstProps.Ra); + Add("1110111110010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Ldc, InstEmit.Ldc, InstProps.Rd | InstProps.Ra); + Add("1110111011010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Ldg, InstEmit.Ldg, InstProps.Rd | InstProps.Ra); + Add("1110111101000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Ldl, InstEmit.Ldl, InstProps.Rd | InstProps.Ra); + Add("1110111101001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Lds, InstEmit.Lds, InstProps.Rd | InstProps.Ra); + Add("0101101111010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Lea, InstEmit.LeaR, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.LPd); + Add("0011011x11010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Lea, InstEmit.LeaI, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.LPd); + Add("0100101111010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Lea, InstEmit.LeaC, InstProps.Rd | InstProps.Ra | InstProps.LPd); + Add("0101101111011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.LeaHi, InstEmit.LeaHiR, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc | InstProps.LPd); + Add("000110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.LeaHi, InstEmit.LeaHiC, InstProps.Rd | InstProps.Ra | InstProps.Rc | InstProps.LPd); + Add("0101000011010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Lepc, InstEmit.Lepc); + Add("111000110001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Longjmp, InstEmit.Longjmp, InstProps.Bra); + Add("0101110001000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Lop, InstEmit.LopR, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.LPd); + Add("0011100x01000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Lop, InstEmit.LopI, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.LPd); + Add("0100110001000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Lop, InstEmit.LopC, InstProps.Rd | InstProps.Ra | InstProps.LPd); + Add("0101101111100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Lop3, InstEmit.Lop3R, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc | InstProps.LPd); + Add("001111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Lop3, InstEmit.Lop3I, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.Rc); + Add("0000001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Lop3, InstEmit.Lop3C, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("000001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Lop32i, InstEmit.Lop32i, InstProps.Rd | InstProps.Ra); + Add("1110111110011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Membar, InstEmit.Membar); + Add("0101110010011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Mov, InstEmit.MovR, InstProps.Rd | InstProps.Ra); + Add("0011100x10011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Mov, InstEmit.MovI, InstProps.Rd | InstProps.Ib); + Add("0100110010011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Mov, InstEmit.MovC, InstProps.Rd); + Add("000000010000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Mov32i, InstEmit.Mov32i, InstProps.Rd); + Add("0101000010000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Mufu, InstEmit.Mufu, InstProps.Rd | InstProps.Ra); + Add("0101000010110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Nop, InstEmit.Nop); + Add("1111101111100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Out, InstEmit.OutR, InstProps.Rd | InstProps.Ra | InstProps.Rb); + Add("1111011x11100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Out, InstEmit.OutI, InstProps.Rd | InstProps.Ra | InstProps.Ib); + Add("1110101111100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Out, InstEmit.OutC, InstProps.Rd | InstProps.Ra); + Add("0101110011101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.P2r, InstEmit.P2rR, InstProps.Rd | InstProps.Ra | InstProps.Rb); + Add("0011100x11101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.P2r, InstEmit.P2rI, InstProps.Rd | InstProps.Ra | InstProps.Ib); + Add("0100110011101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.P2r, InstEmit.P2rC, InstProps.Rd | InstProps.Ra); + Add("111000101010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Pbk, InstEmit.Pbk, InstProps.NoPred); + Add("111000101011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Pcnt, InstEmit.Pcnt, InstProps.NoPred); + Add("111000100011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Pexit, InstEmit.Pexit); + Add("1110111111101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Pixld, InstEmit.Pixld, InstProps.Rd | InstProps.Ra | InstProps.VPd); + Add("111000101000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Plongjmp, InstEmit.Plongjmp, InstProps.Bra | InstProps.NoPred); + Add("0101110000001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Popc, InstEmit.PopcR, InstProps.Rd | InstProps.Rb); + Add("0011100x00001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Popc, InstEmit.PopcI, InstProps.Rd | InstProps.Ib); + Add("0100110000001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Popc, InstEmit.PopcC, InstProps.Rd); + Add("111000100111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Pret, InstEmit.Pret, InstProps.NoPred); + Add("010110111100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Prmt, InstEmit.PrmtR, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("0011011x1100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Prmt, InstEmit.PrmtI, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.Rc); + Add("010010111100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Prmt, InstEmit.PrmtC, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("010100111100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Prmt, InstEmit.PrmtRc, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("0101000010001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Pset, InstEmit.Pset, InstProps.Rd | InstProps.Ps); + Add("0101000010010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Psetp, InstEmit.Psetp, InstProps.Pd | InstProps.Pdn | InstProps.Ps); + Add("1111000011000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.R2b, InstEmit.R2b, InstProps.Rb); + Add("0101110011110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.R2p, InstEmit.R2pR, InstProps.Ra | InstProps.Rb); + Add("0011100x11110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.R2p, InstEmit.R2pI, InstProps.Ra | InstProps.Ib); + Add("0100110011110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.R2p, InstEmit.R2pC, InstProps.Ra); + Add("111000111000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Ram, InstEmit.Ram, InstProps.NoPred); + Add("1110101111111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Red, InstEmit.Red, InstProps.Ra | InstProps.Rb2); + Add("111000110010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Ret, InstEmit.Ret, InstProps.Bra); + Add("0101110010010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Rro, InstEmit.RroR, InstProps.Rd | InstProps.Rb); + Add("0011100x10010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Rro, InstEmit.RroI, InstProps.Rd | InstProps.Ib); + Add("0100110010010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Rro, InstEmit.RroC, InstProps.Rd); + Add("111000110110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Rtt, InstEmit.Rtt, InstProps.NoPred); + Add("1111000011001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.S2r, InstEmit.S2r, InstProps.Rd); + Add("111000110111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Sam, InstEmit.Sam, InstProps.NoPred); + Add("0101110010100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Sel, InstEmit.SelR, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Ps); + Add("0011100x10100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Sel, InstEmit.SelI, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.Ps); + Add("0100110010100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Sel, InstEmit.SelC, InstProps.Rd | InstProps.Ra | InstProps.Ps); + Add("111000101110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Setcrsptr, InstEmit.Setcrsptr, InstProps.Ra | InstProps.NoPred); + Add("111000101111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Setlmembase, InstEmit.Setlmembase, InstProps.Ra | InstProps.NoPred); + Add("0101101111111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Shf, InstEmit.ShfLR, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("0101110011111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Shf, InstEmit.ShfRR, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("0011011x11111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Shf, InstEmit.ShfLI, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.Rc); + Add("0011100x11111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Shf, InstEmit.ShfRI, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.Rc); + Add("1110111100010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Shfl, InstEmit.Shfl, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc | InstProps.LPd); + Add("0101110001001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Shl, InstEmit.ShlR, InstProps.Rd | InstProps.Ra | InstProps.Rb); + Add("0011100x01001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Shl, InstEmit.ShlI, InstProps.Rd | InstProps.Ra | InstProps.Ib); + Add("0100110001001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Shl, InstEmit.ShlC, InstProps.Rd | InstProps.Ra); + Add("0101110000101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Shr, InstEmit.ShrR, InstProps.Rd | InstProps.Ra | InstProps.Rb); + Add("0011100x00101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Shr, InstEmit.ShrI, InstProps.Rd | InstProps.Ra | InstProps.Ib); + Add("0100110000101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Shr, InstEmit.ShrC, InstProps.Rd | InstProps.Ra); + Add("111000101001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Ssy, InstEmit.Ssy, InstProps.NoPred); + Add("101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.St, InstEmit.St, InstProps.Rd | InstProps.Ra); + Add("1110111011011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Stg, InstEmit.Stg, InstProps.Rd | InstProps.Ra); + Add("1110111101010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Stl, InstEmit.Stl, InstProps.Rd | InstProps.Ra); + Add("1110111010100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Stp, InstEmit.Stp, InstProps.NoPred); + Add("1110111101011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Sts, InstEmit.Sts, InstProps.Rd | InstProps.Ra); + Add("1110101001110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.SuatomB, InstEmit.SuatomB, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("11101010x0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Suatom, InstEmit.Suatom, InstProps.Rd | InstProps.Ra | InstProps.Rb); + Add("1110101110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.SuatomB2, InstEmit.SuatomB2, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("1110101011010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.SuatomCasB, InstEmit.SuatomCasB, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc | InstProps.SPd); + Add("1110101x1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.SuatomCas, InstEmit.SuatomCas, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.SPd); + Add("1110101100010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.SuldDB, InstEmit.SuldDB, InstProps.Rd | InstProps.Ra | InstProps.Rc | InstProps.SPd | InstProps.TexB); + Add("1110101100011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.SuldD, InstEmit.SuldD, InstProps.Rd | InstProps.Ra | InstProps.SPd | InstProps.Tex); + Add("1110101100000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.SuldB, InstEmit.SuldB, InstProps.Rd | InstProps.Ra | InstProps.Rc | InstProps.SPd | InstProps.TexB); + Add("1110101100001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Suld, InstEmit.Suld, InstProps.Rd | InstProps.Ra | InstProps.SPd | InstProps.Tex); + Add("1110101101010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.SuredB, InstEmit.SuredB, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("1110101101011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Sured, InstEmit.Sured, InstProps.Rd | InstProps.Ra); + Add("1110101100110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.SustDB, InstEmit.SustDB, InstProps.Rd | InstProps.Ra | InstProps.Rc | InstProps.TexB); + Add("1110101100111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.SustD, InstEmit.SustD, InstProps.Rd | InstProps.Ra | InstProps.Tex); + Add("1110101100100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.SustB, InstEmit.SustB, InstProps.Rd | InstProps.Ra | InstProps.Rc | InstProps.TexB); + Add("1110101100101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Sust, InstEmit.Sust, InstProps.Rd | InstProps.Ra | InstProps.Tex); + Add("1111000011111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Sync, InstEmit.Sync, InstProps.Bra); + Add("11000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Tex, InstEmit.Tex, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.TPd | InstProps.Tex); + Add("1101111010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.TexB, InstEmit.TexB, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.TPd | InstProps.TexB); + Add("1101100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Texs, InstEmit.Texs, InstProps.Rd | InstProps.Rd2 | InstProps.Ra | InstProps.Rb | InstProps.Tex); + Add("1101000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.TexsF16, InstEmit.TexsF16, InstProps.Rd | InstProps.Rd2 | InstProps.Ra | InstProps.Rb | InstProps.Tex); + Add("11011100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Tld, InstEmit.Tld, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.TPd | InstProps.Tex); + Add("11011101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.TldB, InstEmit.TldB, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.TPd | InstProps.TexB); + Add("1101101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Tlds, InstEmit.Tlds, InstProps.Rd | InstProps.Rd2 | InstProps.Ra | InstProps.Rb | InstProps.Tex); + Add("1101001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.TldsF16, InstEmit.TldsF16, InstProps.Rd | InstProps.Rd2 | InstProps.Ra | InstProps.Rb | InstProps.Tex); + Add("110010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Tld4, InstEmit.Tld4, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.TPd | InstProps.Tex); + Add("1101111011xxxxxxxxxxxxx0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Tld4B, InstEmit.Tld4B, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.TPd | InstProps.TexB); + Add("1101111100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Tld4s, InstEmit.Tld4s, InstProps.Rd | InstProps.Rd2 | InstProps.Ra | InstProps.Rb | InstProps.Tex); + Add("1101111110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Tld4sF16, InstEmit.Tld4sF16, InstProps.Rd | InstProps.Rd2 | InstProps.Ra | InstProps.Rb | InstProps.Tex); + Add("1101111101011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Tmml, InstEmit.Tmml, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Tex); + Add("1101111101100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.TmmlB, InstEmit.TmmlB, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.TexB); + Add("1101111101000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Txa, InstEmit.Txa, InstProps.Rd | InstProps.Ra | InstProps.Tex); + Add("110111100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Txd, InstEmit.Txd, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.TPd | InstProps.Tex); + Add("1101111001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.TxdB, InstEmit.TxdB, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.TPd | InstProps.TexB); + Add("1101111101001xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Txq, InstEmit.Txq, InstProps.Rd | InstProps.Ra | InstProps.Tex); + Add("1101111101010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.TxqB, InstEmit.TxqB, InstProps.Rd | InstProps.Ra | InstProps.TexB); + Add("01010100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Vabsdiff, InstEmit.Vabsdiff, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("010100000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Vabsdiff4, InstEmit.Vabsdiff4, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("001000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Vadd, InstEmit.Vadd, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("01011111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Vmad, InstEmit.Vmad, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("0011101xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Vmnmx, InstEmit.Vmnmx, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("0101000011011xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Vote, InstEmit.Vote, InstProps.Rd | InstProps.VPd | InstProps.Ps); + Add("0101000011100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Votevtg, InstEmit.Votevtg); + Add("0100000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Vset, InstEmit.Vset, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("0101000011110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Vsetp, InstEmit.Vsetp, InstProps.Ra | InstProps.Rb | InstProps.Pd | InstProps.Pdn | InstProps.Ps); + Add("01010111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Vshl, InstEmit.Vshl, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("01010110xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Vshr, InstEmit.Vshr, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("0101101100xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Xmad, InstEmit.XmadR, InstProps.Rd | InstProps.Ra | InstProps.Rb | InstProps.Rc); + Add("0011011x00xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Xmad, InstEmit.XmadI, InstProps.Rd | InstProps.Ra | InstProps.Ib | InstProps.Rc); + Add("0100111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Xmad, InstEmit.XmadC, InstProps.Rd | InstProps.Ra | InstProps.Rc); + Add("010100010xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", InstName.Xmad, InstEmit.XmadRc, InstProps.Rd | InstProps.Ra | InstProps.Rc); +#pragma warning restore IDE0055 + #endregion + } + + private static void Add(string encoding, InstName name, InstEmitter emitter, InstProps props = InstProps.None) + { + ReadOnlySpan encodingPart = encoding.AsSpan(0, EncodingBits); + + int bit = encodingPart.Length - 1; + int value = 0; + int xMask = 0; + int xBits = 0; + + int[] xPos = new int[encodingPart.Length]; + + for (int index = 0; index < encodingPart.Length; index++, bit--) + { + char chr = encodingPart[index]; + + if (chr == '1') + { + value |= 1 << bit; + } + else if (chr == 'x') + { + xMask |= 1 << bit; + + xPos[xBits++] = bit; + } + } + + xMask = ~xMask; + + TableEntry entry = new(name, emitter, props, xBits); + + for (int index = 0; index < (1 << xBits); index++) + { + value &= xMask; + + for (int x = 0; x < xBits; x++) + { + value |= ((index >> x) & 1) << xPos[x]; + } + + if (_opCodes[value].Emitter == null || _opCodes[value].XBits > xBits) + { + _opCodes[value] = entry; + } + } + } + + public static InstOp GetOp(ulong address, ulong opCode) + { + ref TableEntry entry = ref _opCodes[opCode >> (64 - EncodingBits)]; + + if (entry.Emitter != null) + { + return new InstOp(address, opCode, entry.Name, entry.Emitter, entry.Props); + } + + return new InstOp(address, opCode, InstName.Invalid, null, InstProps.None); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/Register.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/Register.cs new file mode 100644 index 0000000..29837fe --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/Register.cs @@ -0,0 +1,36 @@ +using System; + +namespace Ryujinx.Graphics.Shader.Decoders +{ + readonly struct Register : IEquatable + { + public int Index { get; } + + public RegisterType Type { get; } + + public bool IsRZ => Type == RegisterType.Gpr && Index == RegisterConsts.RegisterZeroIndex; + public bool IsPT => Type == RegisterType.Predicate && Index == RegisterConsts.PredicateTrueIndex; + + public Register(int index, RegisterType type) + { + Index = index; + Type = type; + } + + public override int GetHashCode() + { + return (ushort)Index | ((ushort)Type << 16); + } + + public override bool Equals(object obj) + { + return obj is Register reg && Equals(reg); + } + + public bool Equals(Register other) + { + return other.Index == Index && + other.Type == Type; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/RegisterConsts.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/RegisterConsts.cs new file mode 100644 index 0000000..25c62eb --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/RegisterConsts.cs @@ -0,0 +1,13 @@ +namespace Ryujinx.Graphics.Shader.Decoders +{ + static class RegisterConsts + { + public const int GprsCount = 255; + public const int PredsCount = 7; + public const int FlagsCount = 4; + public const int TotalCount = GprsCount + PredsCount + FlagsCount; + + public const int RegisterZeroIndex = GprsCount; + public const int PredicateTrueIndex = PredsCount; + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/RegisterType.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/RegisterType.cs new file mode 100644 index 0000000..f3fa194 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Decoders/RegisterType.cs @@ -0,0 +1,9 @@ +namespace Ryujinx.Graphics.Shader.Decoders +{ + enum RegisterType + { + Flag, + Gpr, + Predicate, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/GpuGraphicsState.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/GpuGraphicsState.cs new file mode 100644 index 0000000..a329792 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/GpuGraphicsState.cs @@ -0,0 +1,177 @@ +using Ryujinx.Common.Memory; + +namespace Ryujinx.Graphics.Shader +{ + /// + /// GPU graphics state that the shader depends on. + /// + public readonly struct GpuGraphicsState + { + /// + /// Early Z force enable. + /// + public readonly bool EarlyZForce; + + /// + /// Primitive topology of current draw. + /// + public readonly InputTopology Topology; + + /// + /// Tessellation winding order. + /// + public readonly bool TessCw; + + /// + /// Tessellation patch type. + /// + public readonly TessPatchType TessPatchType; + + /// + /// Tessellation spacing. + /// + public readonly TessSpacing TessSpacing; + + /// + /// Indicates whether alpha-to-coverage is enabled. + /// + public readonly bool AlphaToCoverageEnable; + + /// + /// Indicates whether alpha-to-coverage dithering is enabled. + /// + public readonly bool AlphaToCoverageDitherEnable; + + /// + /// Indicates whether the viewport transform is disabled. + /// + public readonly bool ViewportTransformDisable; + + /// + /// Depth mode zero to one or minus one to one. + /// + public readonly bool DepthMode; + + /// + /// Indicates if the point size is set on the shader or is fixed. + /// + public readonly bool ProgramPointSizeEnable; + + /// + /// Point size used if is false. + /// + public readonly float PointSize; + + /// + /// When alpha test is enabled, indicates the comparison that decides if the fragment should be discarded. + /// + public readonly AlphaTestOp AlphaTestCompare; + + /// + /// When alpha test is enabled, indicates the value to compare with the fragment output alpha. + /// + public readonly float AlphaTestReference; + + /// + /// Type of the vertex attributes consumed by the shader. + /// + public readonly Array32 AttributeTypes; + + /// + /// Indicates that the draw is writing the base vertex, base instance and draw index to Constant Buffer 0. + /// + public readonly bool HasConstantBufferDrawParameters; + + /// + /// Type of the fragment shader outputs. + /// + public readonly Array8 FragmentOutputTypes; + + /// + /// Indicates whether dual source blend is enabled. + /// + public readonly bool DualSourceBlendEnable; + + /// + /// Indicates if negation of the viewport Y axis is enabled. + /// + public readonly bool YNegateEnabled; + + /// + /// If true, indicates that the fragment origin is the upper left corner of the viewport, otherwise it is the lower left corner. + /// + public readonly bool OriginUpperLeft; + + /// + /// Indicates that the primitive ID values on the shader should be halved due to quad to triangles conversion. + /// + public readonly bool HalvePrimitiveId; + + /// + /// Creates a new GPU graphics state. + /// + /// Early Z force enable + /// Primitive topology + /// Tessellation winding order (clockwise or counter-clockwise) + /// Tessellation patch type + /// Tessellation spacing + /// Indicates whether alpha-to-coverage is enabled + /// Indicates whether alpha-to-coverage dithering is enabled + /// Indicates whether the viewport transform is disabled + /// Depth mode zero to one or minus one to one + /// Indicates if the point size is set on the shader or is fixed + /// Point size if not set from shader + /// When alpha test is enabled, indicates the comparison that decides if the fragment should be discarded + /// When alpha test is enabled, indicates the value to compare with the fragment output alpha + /// Type of the vertex attributes consumed by the shader + /// Indicates that the draw is writing the base vertex, base instance and draw index to Constant Buffer 0 + /// Type of the fragment shader outputs + /// Indicates whether dual source blend is enabled + /// Indicates if negation of the viewport Y axis is enabled + /// If true, indicates that the fragment origin is the upper left corner of the viewport, otherwise it is the lower left corner + /// Indicates that the primitive ID values on the shader should be halved due to quad to triangles conversion + public GpuGraphicsState( + bool earlyZForce, + InputTopology topology, + bool tessCw, + TessPatchType tessPatchType, + TessSpacing tessSpacing, + bool alphaToCoverageEnable, + bool alphaToCoverageDitherEnable, + bool viewportTransformDisable, + bool depthMode, + bool programPointSizeEnable, + float pointSize, + AlphaTestOp alphaTestCompare, + float alphaTestReference, + in Array32 attributeTypes, + bool hasConstantBufferDrawParameters, + in Array8 fragmentOutputTypes, + bool dualSourceBlendEnable, + bool yNegateEnabled, + bool originUpperLeft, + bool halvePrimitiveId) + { + EarlyZForce = earlyZForce; + Topology = topology; + TessCw = tessCw; + TessPatchType = tessPatchType; + TessSpacing = tessSpacing; + AlphaToCoverageEnable = alphaToCoverageEnable; + AlphaToCoverageDitherEnable = alphaToCoverageDitherEnable; + ViewportTransformDisable = viewportTransformDisable; + DepthMode = depthMode; + ProgramPointSizeEnable = programPointSizeEnable; + PointSize = pointSize; + AlphaTestCompare = alphaTestCompare; + AlphaTestReference = alphaTestReference; + AttributeTypes = attributeTypes; + HasConstantBufferDrawParameters = hasConstantBufferDrawParameters; + FragmentOutputTypes = fragmentOutputTypes; + DualSourceBlendEnable = dualSourceBlendEnable; + YNegateEnabled = yNegateEnabled; + OriginUpperLeft = originUpperLeft; + HalvePrimitiveId = halvePrimitiveId; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IGpuAccessor.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IGpuAccessor.cs new file mode 100644 index 0000000..24728f1 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IGpuAccessor.cs @@ -0,0 +1,502 @@ +using Ryujinx.Graphics.Shader.CodeGen; +using System; + +namespace Ryujinx.Graphics.Shader +{ + /// + /// GPU state access interface. + /// + public interface IGpuAccessor : ILogger + { + /// + /// Reads data from the constant buffer 1. + /// + /// Offset in bytes to read from + /// Value at the given offset + uint ConstantBuffer1Read(int offset) + { + return 0; + } + + /// + /// Gets a span of the specified memory location, containing shader code. + /// + /// GPU virtual address of the data + /// Minimum size that the returned span may have + /// Span of the memory location + ReadOnlySpan GetCode(ulong address, int minimumSize); + + /// + /// Gets the binding number of a constant buffer. + /// + /// Constant buffer index + /// Binding number + SetBindingPair CreateConstantBufferBinding(int index); + + /// + /// Gets the binding number of an image. + /// + /// For array of images, the number of elements of the array, otherwise it should be 1 + /// Indicates if the image is a buffer image + /// Binding number + SetBindingPair CreateImageBinding(int count, bool isBuffer); + + /// + /// Gets the binding number of a storage buffer. + /// + /// Storage buffer index + /// Binding number + SetBindingPair CreateStorageBufferBinding(int index); + + /// + /// Gets the binding number of a texture. + /// + /// For array of textures, the number of elements of the array, otherwise it should be 1 + /// Indicates if the texture is a buffer texture + /// Binding number + SetBindingPair CreateTextureBinding(int count, bool isBuffer); + + /// + /// Gets the set index for an additional set, or -1 if there's no extra set available. + /// + /// Extra set index, or -1 if not available + int CreateExtraSet() + { + return -1; + } + + /// + /// Queries Local Size X for compute shaders. + /// + /// Local Size X + int QueryComputeLocalSizeX() + { + return 1; + } + + /// + /// Queries Local Size Y for compute shaders. + /// + /// Local Size Y + int QueryComputeLocalSizeY() + { + return 1; + } + + /// + /// Queries Local Size Z for compute shaders. + /// + /// Local Size Z + int QueryComputeLocalSizeZ() + { + return 1; + } + + /// + /// Queries Local Memory size in bytes for compute shaders. + /// + /// Local Memory size in bytes + int QueryComputeLocalMemorySize() + { + return 0x1000; + } + + /// + /// Queries Shared Memory size in bytes for compute shaders. + /// + /// Shared Memory size in bytes + int QueryComputeSharedMemorySize() + { + return 0xc000; + } + + /// + /// Queries Constant Buffer usage information. + /// + /// A mask where each bit set indicates a bound constant buffer + uint QueryConstantBufferUse() + { + return 0; + } + + /// + /// Queries specialized GPU graphics state that the shader depends on. + /// + /// GPU graphics state + GpuGraphicsState QueryGraphicsState() + { + return new GpuGraphicsState( + false, + InputTopology.Points, + false, + TessPatchType.Triangles, + TessSpacing.EqualSpacing, + false, + false, + false, + false, + false, + 1f, + AlphaTestOp.Always, + 0f, + default, + true, + default, + false, + false, + false, + false); + } + + /// + /// Queries whenever the current draw has written the base vertex and base instance into Constant Buffer 0. + /// + /// True if the shader translator can assume that the constant buffer contains the base IDs, false otherwise + bool QueryHasConstantBufferDrawParameters() + { + return false; + } + + /// + /// Queries whenever the current draw uses unaligned storage buffer addresses. + /// + /// True if any storage buffer address is not aligned to 16 bytes, false otherwise + bool QueryHasUnalignedStorageBuffer() + { + return false; + } + + /// + /// Queries host's gather operation precision bits for biasing their coordinates. Zero means no bias. + /// + /// Bits of gather operation precision to use for coordinate bias + int QueryHostGatherBiasPrecision() + { + return 0; + } + + /// + /// Queries host about whether to reduce precision to improve performance. + /// + /// True if precision is limited to vertex position, false otherwise + bool QueryHostReducedPrecision() + { + return false; + } + + /// + /// Queries host about the presence of the FrontFacing built-in variable bug. + /// + /// True if the bug is present on the host device used, false otherwise + bool QueryHostHasFrontFacingBug() + { + return false; + } + + /// + /// Queries host about the presence of the vector indexing bug. + /// + /// True if the bug is present on the host device used, false otherwise + bool QueryHostHasVectorIndexingBug() + { + return false; + } + + /// + /// Queries host storage buffer alignment required. + /// + /// Host storage buffer alignment in bytes + int QueryHostStorageBufferOffsetAlignment() + { + return 16; + } + + /// + /// Queries host shader subgroup size. + /// + /// Host shader subgroup size in invocations + int QueryHostSubgroupSize() + { + return 32; + } + + /// + /// Queries host support for texture formats with BGRA component order (such as BGRA8). + /// + /// True if BGRA formats are supported, false otherwise + bool QueryHostSupportsBgraFormat() + { + return true; + } + + /// + /// Queries host support for fragment shader ordering critical sections on the shader code. + /// + /// True if fragment shader interlock is supported, false otherwise + bool QueryHostSupportsFragmentShaderInterlock() + { + return true; + } + + /// + /// Queries host support for fragment shader ordering scoped critical sections on the shader code. + /// + /// True if fragment shader ordering is supported, false otherwise + bool QueryHostSupportsFragmentShaderOrderingIntel() + { + return false; + } + + /// + /// Queries host GPU geometry shader support. + /// + /// True if the GPU and driver supports geometry shaders, false otherwise + bool QueryHostSupportsGeometryShader() + { + return true; + } + + /// + /// Queries host GPU geometry shader passthrough support. + /// + /// True if the GPU and driver supports geometry shader passthrough, false otherwise + bool QueryHostSupportsGeometryShaderPassthrough() + { + return true; + } + + /// + /// Queries host support for readable images without a explicit format declaration on the shader. + /// + /// True if formatted image load is supported, false otherwise + bool QueryHostSupportsImageLoadFormatted() + { + return true; + } + + /// + /// Queries host support for writes to the layer from vertex or tessellation shader stages. + /// + /// True if writes to the layer from vertex or tessellation are supported, false otherwise + bool QueryHostSupportsLayerVertexTessellation() + { + return true; + } + + /// + /// Queries host GPU non-constant texture offset support. + /// + /// True if the GPU and driver supports non-constant texture offsets, false otherwise + bool QueryHostSupportsNonConstantTextureOffset() + { + return true; + } + + /// + /// Queries host support scaled vertex formats, where a integer value is converted to floating-point. + /// + /// True if the host support scaled vertex formats, false otherwise + bool QueryHostSupportsScaledVertexFormats() + { + return true; + } + + /// + /// Queries host API support for separate textures and samplers. + /// + /// True if the API supports samplers and textures to be combined on the shader, false otherwise + bool QueryHostSupportsSeparateSampler() + { + return true; + } + + /// + /// Queries host GPU shader ballot support. + /// + /// True if the GPU and driver supports shader ballot, false otherwise + bool QueryHostSupportsShaderBallot() + { + return true; + } + + /// + /// Queries host GPU shader support for barrier instructions on divergent control flow paths. + /// + /// True if the GPU supports barriers on divergent control flow paths, false otherwise + bool QueryHostSupportsShaderBarrierDivergence() + { + return true; + } + + /// + /// Queries host GPU support for 64-bit floating point (double precision) operations on the shader. + /// + /// True if the GPU and driver supports double operations, false otherwise + bool QueryHostSupportsShaderFloat64() + { + return true; + } + bool QueryHostSupportsShaderNonUniformIndexing() + { + return false; + } + + /// + /// Queries host GPU support for signed normalized buffer texture formats. + /// + /// True if the GPU and driver supports the formats, false otherwise + bool QueryHostSupportsSnormBufferTextureFormat() + { + return true; + } + + /// + /// Queries host GPU texture gather with multiple offsets support. + /// + /// True if the GPU and driver supports texture gather offsets, false otherwise + bool QueryHostSupportsTextureGatherOffsets() + { + return true; + } + + /// + /// Queries host GPU texture shadow LOD support. + /// + /// True if the GPU and driver supports texture shadow LOD, false otherwise + bool QueryHostSupportsTextureShadowLod() + { + return true; + } + + /// + /// Queries host GPU transform feedback support. + /// + /// True if the GPU and driver supports transform feedback, false otherwise + bool QueryHostSupportsTransformFeedback() + { + return true; + } + + /// + /// Queries host support for writes to the viewport index from vertex or tessellation shader stages. + /// + /// True if writes to the viewport index from vertex or tessellation are supported, false otherwise + bool QueryHostSupportsViewportIndexVertexTessellation() + { + return true; + } + + /// + /// Queries host GPU shader viewport mask output support. + /// + /// True if the GPU and driver supports shader viewport mask output, false otherwise + bool QueryHostSupportsViewportMask() + { + return true; + } + + /// + /// Queries whether the host supports depth clip control. + /// + /// True if the GPU and driver supports depth clip control, false otherwise + bool QueryHostSupportsDepthClipControl() + { + return true; + } + + /// + /// Gets the maximum number of samplers that the bound texture pool may have. + /// + /// Maximum amount of samplers that the pool may have + int QuerySamplerArrayLengthFromPool(); + + /// + /// Queries sampler type information. + /// + /// Texture handle + /// Constant buffer slot for the texture handle + /// The sampler type value for the given handle + SamplerType QuerySamplerType(int handle, int cbufSlot = -1) + { + return SamplerType.Texture2D; + } + + /// + /// Gets the size in bytes of a bound constant buffer for the current shader stage. + /// + /// The number of the constant buffer to get the size from + /// Size in bytes + int QueryTextureArrayLengthFromBuffer(int slot); + + /// + /// Gets the maximum number of textures that the bound texture pool may have. + /// + /// Maximum amount of textures that the pool may have + int QueryTextureArrayLengthFromPool(); + + /// + /// Queries texture coordinate normalization information. + /// + /// Texture handle + /// Constant buffer slot for the texture handle + /// True if the coordinates are normalized, false otherwise + bool QueryTextureCoordNormalized(int handle, int cbufSlot = -1) + { + return true; + } + + /// + /// Queries texture format information, for shaders using image load or store. + /// + /// + /// This only returns non-compressed color formats. + /// If the format of the texture is a compressed, depth or unsupported format, then a default value is returned. + /// + /// Texture handle + /// Constant buffer slot for the texture handle + /// Color format of the non-compressed texture + TextureFormat QueryTextureFormat(int handle, int cbufSlot = -1) + { + return TextureFormat.R8G8B8A8Unorm; + } + + /// + /// Queries transform feedback enable state. + /// + /// True if the shader uses transform feedback, false otherwise + bool QueryTransformFeedbackEnabled() + { + return false; + } + + /// + /// Queries the varying locations that should be written to the transform feedback buffer. + /// + /// Index of the transform feedback buffer + /// Varying locations for the specified buffer + ReadOnlySpan QueryTransformFeedbackVaryingLocations(int bufferIndex) + { + return ReadOnlySpan.Empty; + } + + /// + /// Queries the stride (in bytes) of the per vertex data written into the transform feedback buffer. + /// + /// Index of the transform feedback buffer + /// Stride for the specified buffer + int QueryTransformFeedbackStride(int bufferIndex) + { + return 0; + } + + /// + /// Registers a texture used by the shader. + /// + /// Texture handle word offset + /// Constant buffer slot where the texture handle is located + void RegisterTexture(int handle, int cbufSlot) + { + // Only useful when recording information for a disk shader cache. + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/ILogger.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/ILogger.cs new file mode 100644 index 0000000..c3e0f68 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/ILogger.cs @@ -0,0 +1,17 @@ +namespace Ryujinx.Graphics.Shader.CodeGen +{ + /// + /// Shader code generation logging interface. + /// + public interface ILogger + { + /// + /// Prints a log message. + /// + /// Message to print + void Log(string message) + { + // No default log output. + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/InputTopology.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/InputTopology.cs new file mode 100644 index 0000000..615c94d --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/InputTopology.cs @@ -0,0 +1,47 @@ +namespace Ryujinx.Graphics.Shader +{ + public enum InputTopology : byte + { + Points, + Lines, + LinesAdjacency, + Triangles, + TrianglesAdjacency, + } + + static class InputTopologyExtensions + { + extension(InputTopology topology) + { + public string GlslString => topology switch + { + InputTopology.Points => "points", + InputTopology.Lines => "lines", + InputTopology.LinesAdjacency => "lines_adjacency", + InputTopology.Triangles => "triangles", + InputTopology.TrianglesAdjacency => "triangles_adjacency", + _ => "points", + }; + + public int InputVertexCount => topology switch + { + InputTopology.Points => 1, + InputTopology.Lines => 2, + InputTopology.LinesAdjacency => 4, + InputTopology.Triangles => 3, + InputTopology.TrianglesAdjacency => 6, + _ => 1, + }; + + public int InputVertexCountNoAdjacency => topology switch + { + InputTopology.Points => 1, + InputTopology.Lines or + InputTopology.LinesAdjacency => 2, + InputTopology.Triangles or + InputTopology.TrianglesAdjacency => 3, + _ => 1, + }; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/AttributeMap.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/AttributeMap.cs new file mode 100644 index 0000000..c417d66 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/AttributeMap.cs @@ -0,0 +1,351 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using System.Collections.Generic; +using System.Collections.ObjectModel; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static class AttributeMap + { + private enum StagesMask : byte + { + None = 0, + Compute = 1 << (int)ShaderStage.Compute, + Vertex = 1 << (int)ShaderStage.Vertex, + TessellationControl = 1 << (int)ShaderStage.TessellationControl, + TessellationEvaluation = 1 << (int)ShaderStage.TessellationEvaluation, + Geometry = 1 << (int)ShaderStage.Geometry, + Fragment = 1 << (int)ShaderStage.Fragment, + + Tessellation = TessellationControl | TessellationEvaluation, + VertexTessellationGeometry = Vertex | Tessellation | Geometry, + TessellationGeometryFragment = Tessellation | Geometry | Fragment, + AllGraphics = Vertex | Tessellation | Geometry | Fragment, + } + + private readonly struct AttributeEntry + { + public int BaseOffset { get; } + public AggregateType Type { get; } + public IoVariable IoVariable { get; } + public StagesMask InputMask { get; } + public StagesMask OutputMask { get; } + + public AttributeEntry( + int baseOffset, + AggregateType type, + IoVariable ioVariable, + StagesMask inputMask, + StagesMask outputMask) + { + BaseOffset = baseOffset; + Type = type; + IoVariable = ioVariable; + InputMask = inputMask; + OutputMask = outputMask; + } + } + + private static readonly ReadOnlyDictionary _attributes; + private static readonly ReadOnlyDictionary _attributesPerPatch; + + static AttributeMap() + { + _attributes = CreateMap(); + _attributesPerPatch = CreatePerPatchMap(); + } + + private static ReadOnlyDictionary CreateMap() + { + Dictionary map = new(); + + Add(map, 0x060, AggregateType.S32, IoVariable.PrimitiveId, StagesMask.TessellationGeometryFragment, StagesMask.Geometry); + Add(map, 0x064, AggregateType.S32, IoVariable.Layer, StagesMask.Fragment, StagesMask.VertexTessellationGeometry); + Add(map, 0x068, AggregateType.S32, IoVariable.ViewportIndex, StagesMask.Fragment, StagesMask.VertexTessellationGeometry); + Add(map, 0x06c, AggregateType.FP32, IoVariable.PointSize, StagesMask.None, StagesMask.VertexTessellationGeometry); + Add(map, 0x070, AggregateType.Vector4 | AggregateType.FP32, IoVariable.Position, StagesMask.TessellationGeometryFragment, StagesMask.VertexTessellationGeometry); + Add(map, 0x080, AggregateType.Vector4 | AggregateType.FP32, IoVariable.UserDefined, StagesMask.AllGraphics, StagesMask.VertexTessellationGeometry, 32); + Add(map, 0x280, AggregateType.Vector4 | AggregateType.FP32, IoVariable.FrontColorDiffuse, StagesMask.TessellationGeometryFragment, StagesMask.VertexTessellationGeometry); + Add(map, 0x290, AggregateType.Vector4 | AggregateType.FP32, IoVariable.FrontColorSpecular, StagesMask.TessellationGeometryFragment, StagesMask.VertexTessellationGeometry); + Add(map, 0x2a0, AggregateType.Vector4 | AggregateType.FP32, IoVariable.BackColorDiffuse, StagesMask.TessellationGeometryFragment, StagesMask.VertexTessellationGeometry); + Add(map, 0x2b0, AggregateType.Vector4 | AggregateType.FP32, IoVariable.BackColorSpecular, StagesMask.TessellationGeometryFragment, StagesMask.VertexTessellationGeometry); + Add(map, 0x2c0, AggregateType.Array | AggregateType.FP32, IoVariable.ClipDistance, StagesMask.TessellationGeometryFragment, StagesMask.VertexTessellationGeometry, 8); + Add(map, 0x2e0, AggregateType.Vector2 | AggregateType.FP32, IoVariable.PointCoord, StagesMask.Fragment, StagesMask.None); + Add(map, 0x2e8, AggregateType.FP32, IoVariable.FogCoord, StagesMask.TessellationGeometryFragment, StagesMask.VertexTessellationGeometry); + Add(map, 0x2f0, AggregateType.Vector2 | AggregateType.FP32, IoVariable.TessellationCoord, StagesMask.TessellationEvaluation, StagesMask.None); + Add(map, 0x2f8, AggregateType.S32, IoVariable.InstanceId, StagesMask.Vertex, StagesMask.None); + Add(map, 0x2fc, AggregateType.S32, IoVariable.VertexId, StagesMask.Vertex, StagesMask.None); + Add(map, 0x300, AggregateType.Vector4 | AggregateType.FP32, IoVariable.TextureCoord, StagesMask.TessellationGeometryFragment, StagesMask.VertexTessellationGeometry); + Add(map, 0x3a0, AggregateType.Array | AggregateType.S32, IoVariable.ViewportMask, StagesMask.Fragment, StagesMask.VertexTessellationGeometry); + Add(map, 0x3fc, AggregateType.Bool, IoVariable.FrontFacing, StagesMask.Fragment, StagesMask.None); + + return map.AsReadOnly(); + } + + private static ReadOnlyDictionary CreatePerPatchMap() + { + Dictionary map = new(); + + Add(map, 0x000, AggregateType.Vector4 | AggregateType.FP32, IoVariable.TessellationLevelOuter, StagesMask.TessellationEvaluation, StagesMask.TessellationControl); + Add(map, 0x010, AggregateType.Vector2 | AggregateType.FP32, IoVariable.TessellationLevelInner, StagesMask.TessellationEvaluation, StagesMask.TessellationControl); + Add(map, 0x018, AggregateType.Vector4 | AggregateType.FP32, IoVariable.UserDefined, StagesMask.TessellationEvaluation, StagesMask.TessellationControl, 31, 0x200); + + return map.AsReadOnly(); + } + + private static void Add( + Dictionary attributes, + int offset, + AggregateType type, + IoVariable ioVariable, + StagesMask inputMask, + StagesMask outputMask, + int count = 1, + int upperBound = 0x400) + { + int baseOffset = offset; + + int elementsCount = GetElementCount(type); + + for (int index = 0; index < count; index++) + { + for (int elementIndex = 0; elementIndex < elementsCount; elementIndex++) + { + attributes.Add(offset, new AttributeEntry(baseOffset, type, ioVariable, inputMask, outputMask)); + + offset += 4; + + if (offset >= upperBound) + { + return; + } + } + } + } + + public static Operand GenerateAttributeLoad(EmitterContext context, Operand primVertex, int offset, bool isOutput, bool isPerPatch) + { + if (!(isPerPatch ? _attributesPerPatch : _attributes).TryGetValue(offset, out AttributeEntry entry)) + { + context.TranslatorContext.GpuAccessor.Log($"Attribute offset 0x{offset:X} is not valid."); + return Const(0); + } + + StagesMask validUseMask = isOutput ? entry.OutputMask : entry.InputMask; + + if (((StagesMask)(1 << (int)context.TranslatorContext.Definitions.Stage) & validUseMask) == StagesMask.None) + { + context.TranslatorContext.GpuAccessor.Log($"Attribute offset 0x{offset:X} ({entry.IoVariable}) is not valid for stage {context.TranslatorContext.Definitions.Stage}."); + return Const(0); + } + + if (!IsSupportedByHost(context.TranslatorContext.GpuAccessor, context.TranslatorContext.Definitions.Stage, entry.IoVariable)) + { + context.TranslatorContext.GpuAccessor.Log($"Attribute offset 0x{offset:X} ({entry.IoVariable}) is not supported by the host for stage {context.TranslatorContext.Definitions.Stage}."); + return Const(0); + } + + if (HasInvocationId(context.TranslatorContext.Definitions.Stage, isOutput) && !isPerPatch) + { + primVertex = context.Load(StorageKind.Input, IoVariable.InvocationId); + } + + int innerOffset = offset - entry.BaseOffset; + int innerIndex = innerOffset / 4; + + StorageKind storageKind = isPerPatch + ? (isOutput ? StorageKind.OutputPerPatch : StorageKind.InputPerPatch) + : (isOutput ? StorageKind.Output : StorageKind.Input); + IoVariable ioVariable = GetIoVariable(context.TranslatorContext.Definitions.Stage, in entry); + AggregateType type = GetType(context.TranslatorContext.Definitions, isOutput, innerIndex, in entry); + int elementCount = GetElementCount(type); + + bool isArray = type.HasFlag(AggregateType.Array); + bool hasArrayIndex = isArray || context.TranslatorContext.Definitions.HasPerLocationInputOrOutput(ioVariable, isOutput); + + bool hasElementIndex = elementCount > 1; + + if (hasArrayIndex && hasElementIndex) + { + int arrayIndex = innerIndex / elementCount; + int elementIndex = innerIndex - (arrayIndex * elementCount); + + return primVertex == null || isArray + ? context.Load(storageKind, ioVariable, primVertex, Const(arrayIndex), Const(elementIndex)) + : context.Load(storageKind, ioVariable, Const(arrayIndex), primVertex, Const(elementIndex)); + } + else if (hasArrayIndex || hasElementIndex) + { + return primVertex == null || isArray || !hasArrayIndex + ? context.Load(storageKind, ioVariable, primVertex, Const(innerIndex)) + : context.Load(storageKind, ioVariable, Const(innerIndex), primVertex); + } + else + { + return context.Load(storageKind, ioVariable, primVertex); + } + } + + public static void GenerateAttributeStore(EmitterContext context, int offset, bool isPerPatch, Operand value) + { + if (!(isPerPatch ? _attributesPerPatch : _attributes).TryGetValue(offset, out AttributeEntry entry)) + { + context.TranslatorContext.GpuAccessor.Log($"Attribute offset 0x{offset:X} is not valid."); + return; + } + + if (((StagesMask)(1 << (int)context.TranslatorContext.Definitions.Stage) & entry.OutputMask) == StagesMask.None) + { + context.TranslatorContext.GpuAccessor.Log($"Attribute offset 0x{offset:X} ({entry.IoVariable}) is not valid for stage {context.TranslatorContext.Definitions.Stage}."); + return; + } + + if (!IsSupportedByHost(context.TranslatorContext.GpuAccessor, context.TranslatorContext.Definitions.Stage, entry.IoVariable)) + { + context.TranslatorContext.GpuAccessor.Log($"Attribute offset 0x{offset:X} ({entry.IoVariable}) is not supported by the host for stage {context.TranslatorContext.Definitions.Stage}."); + return; + } + + Operand invocationId = null; + + if (HasInvocationId(context.TranslatorContext.Definitions.Stage, isOutput: true) && !isPerPatch) + { + invocationId = context.Load(StorageKind.Input, IoVariable.InvocationId); + } + + int innerOffset = offset - entry.BaseOffset; + int innerIndex = innerOffset / 4; + + StorageKind storageKind = isPerPatch ? StorageKind.OutputPerPatch : StorageKind.Output; + IoVariable ioVariable = GetIoVariable(context.TranslatorContext.Definitions.Stage, in entry); + AggregateType type = GetType(context.TranslatorContext.Definitions, isOutput: true, innerIndex, in entry); + int elementCount = GetElementCount(type); + + bool isArray = type.HasFlag(AggregateType.Array); + bool hasArrayIndex = isArray || context.TranslatorContext.Definitions.HasPerLocationInputOrOutput(ioVariable, isOutput: true); + + bool hasElementIndex = elementCount > 1; + + if (hasArrayIndex && hasElementIndex) + { + int arrayIndex = innerIndex / elementCount; + int elementIndex = innerIndex - (arrayIndex * elementCount); + + if (invocationId == null || isArray) + { + context.Store(storageKind, ioVariable, invocationId, Const(arrayIndex), Const(elementIndex), value); + } + else + { + context.Store(storageKind, ioVariable, Const(arrayIndex), invocationId, Const(elementIndex), value); + } + } + else if (hasArrayIndex || hasElementIndex) + { + if (invocationId == null || isArray || !hasArrayIndex) + { + context.Store(storageKind, ioVariable, invocationId, Const(innerIndex), value); + } + else + { + context.Store(storageKind, ioVariable, Const(innerIndex), invocationId, value); + } + } + else + { + context.Store(storageKind, ioVariable, invocationId, value); + } + } + + private static bool IsSupportedByHost(IGpuAccessor gpuAccessor, ShaderStage stage, IoVariable ioVariable) + { + if (ioVariable == IoVariable.ViewportIndex && stage != ShaderStage.Geometry && stage != ShaderStage.Fragment) + { + return gpuAccessor.QueryHostSupportsViewportIndexVertexTessellation(); + } + else if (ioVariable == IoVariable.ViewportMask) + { + return gpuAccessor.QueryHostSupportsViewportMask(); + } + + return true; + } + + public static IoVariable GetIoVariable(ShaderDefinitions definitions, int offset, out int location) + { + location = 0; + + if (!_attributes.TryGetValue(offset, out AttributeEntry entry)) + { + return IoVariable.Invalid; + } + + if (((StagesMask)(1 << (int)definitions.Stage) & entry.OutputMask) == StagesMask.None) + { + return IoVariable.Invalid; + } + + if (definitions.HasPerLocationInputOrOutput(entry.IoVariable, isOutput: true)) + { + location = (offset - entry.BaseOffset) / 16; + } + + return GetIoVariable(definitions.Stage, in entry); + } + + private static IoVariable GetIoVariable(ShaderStage stage, in AttributeEntry entry) + { + if (entry.IoVariable == IoVariable.Position && stage == ShaderStage.Fragment) + { + return IoVariable.FragmentCoord; + } + + return entry.IoVariable; + } + + private static AggregateType GetType(ShaderDefinitions definitions, bool isOutput, int innerIndex, in AttributeEntry entry) + { + AggregateType type = entry.Type; + + if (entry.IoVariable == IoVariable.UserDefined) + { + type = definitions.GetUserDefinedType(innerIndex / 4, isOutput); + } + else if (entry.IoVariable == IoVariable.FragmentOutputColor) + { + type = definitions.GetFragmentOutputColorType(innerIndex / 4); + } + + return type; + } + + public static bool HasPrimitiveVertex(ShaderStage stage, bool isOutput) + { + if (isOutput) + { + return false; + } + + return stage is ShaderStage.TessellationControl or + ShaderStage.TessellationEvaluation or + ShaderStage.Geometry; + } + + public static bool HasInvocationId(ShaderStage stage, bool isOutput) + { + return isOutput && stage == ShaderStage.TessellationControl; + } + + private static int GetElementCount(AggregateType type) + { + return (type & AggregateType.ElementCountMask) switch + { + AggregateType.Vector2 => 2, + AggregateType.Vector3 => 3, + AggregateType.Vector4 => 4, + _ => 1, + }; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmit.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmit.cs new file mode 100644 index 0000000..7648ea1 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmit.cs @@ -0,0 +1,359 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.Translation; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void AtomCas(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction AtomCas is not implemented."); + } + + public static void AtomsCas(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction AtomsCas is not implemented."); + } + + public static void B2r(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction B2r is not implemented."); + } + + public static void Bpt(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Bpt is not implemented."); + } + + public static void Cctl(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Cctl is not implemented."); + } + + public static void Cctll(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Cctll is not implemented."); + } + + public static void Cctlt(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Cctlt is not implemented."); + } + + public static void Cs2r(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Cs2r is not implemented."); + } + + public static void FchkR(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction FchkR is not implemented."); + } + + public static void FchkI(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction FchkI is not implemented."); + } + + public static void FchkC(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction FchkC is not implemented."); + } + + public static void Getcrsptr(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Getcrsptr is not implemented."); + } + + public static void Getlmembase(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Getlmembase is not implemented."); + } + + public static void Ide(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Ide is not implemented."); + } + + public static void IdpR(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction IdpR is not implemented."); + } + + public static void IdpC(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction IdpC is not implemented."); + } + + public static void ImadspR(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction ImadspR is not implemented."); + } + + public static void ImadspI(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction ImadspI is not implemented."); + } + + public static void ImadspC(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction ImadspC is not implemented."); + } + + public static void ImadspRc(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction ImadspRc is not implemented."); + } + + public static void Jcal(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Jcal is not implemented."); + } + + public static void Jmp(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Jmp is not implemented."); + } + + public static void Jmx(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Jmx is not implemented."); + } + + public static void Ld(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Ld is not implemented."); + } + + public static void Lepc(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Lepc is not implemented."); + } + + public static void Longjmp(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Longjmp is not implemented."); + } + + public static void Pexit(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Pexit is not implemented."); + } + + public static void Pixld(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Pixld is not implemented."); + } + + public static void Plongjmp(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Plongjmp is not implemented."); + } + + public static void Pret(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Pret is not implemented."); + } + + public static void PrmtR(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction PrmtR is not implemented."); + } + + public static void PrmtI(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction PrmtI is not implemented."); + } + + public static void PrmtC(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction PrmtC is not implemented."); + } + + public static void PrmtRc(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction PrmtRc is not implemented."); + } + + public static void R2b(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction R2b is not implemented."); + } + + public static void Ram(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Ram is not implemented."); + } + + public static void Rtt(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Rtt is not implemented."); + } + + public static void Sam(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Sam is not implemented."); + } + + public static void Setcrsptr(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Setcrsptr is not implemented."); + } + + public static void Setlmembase(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Setlmembase is not implemented."); + } + + public static void St(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction St is not implemented."); + } + + public static void Stp(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Stp is not implemented."); + } + + public static void Txa(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Txa is not implemented."); + } + + public static void Vabsdiff(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Vabsdiff is not implemented."); + } + + public static void Vabsdiff4(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Vabsdiff4 is not implemented."); + } + + public static void Vadd(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Vadd is not implemented."); + } + + public static void Votevtg(EmitterContext context) + { + context.GetOp(); + + // This instruction is proprietary and will not be implemented. Commenting it out to avoid false reports. + //context.TranslatorContext.GpuAccessor.Log("Shader instruction Votevtg is not implemented."); + } + + public static void Vset(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Vset is not implemented."); + } + + public static void Vshl(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Vshl is not implemented."); + } + + public static void Vshr(EmitterContext context) + { + context.GetOp(); + + context.TranslatorContext.GpuAccessor.Log("Shader instruction Vshr is not implemented."); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitAluHelper.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitAluHelper.cs new file mode 100644 index 0000000..c9daa1f --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitAluHelper.cs @@ -0,0 +1,159 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using System; +using static Ryujinx.Graphics.Shader.Instructions.InstEmitHelper; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static class InstEmitAluHelper + { + public static long GetIntMin(IDstFmt type) + { + return type switch + { + IDstFmt.U16 => ushort.MinValue, + IDstFmt.S16 => short.MinValue, + IDstFmt.U32 => uint.MinValue, + IDstFmt.S32 => int.MinValue, + _ => throw new ArgumentException($"The type \"{type}\" is not a supported integer type."), + }; + } + + public static long GetIntMax(IDstFmt type) + { + return type switch + { + IDstFmt.U16 => ushort.MaxValue, + IDstFmt.S16 => short.MaxValue, + IDstFmt.U32 => uint.MaxValue, + IDstFmt.S32 => int.MaxValue, + _ => throw new ArgumentException($"The type \"{type}\" is not a supported integer type."), + }; + } + + public static long GetIntMin(ISrcDstFmt type) + { + return type switch + { + ISrcDstFmt.U8 => byte.MinValue, + ISrcDstFmt.S8 => sbyte.MinValue, + ISrcDstFmt.U16 => ushort.MinValue, + ISrcDstFmt.S16 => short.MinValue, + ISrcDstFmt.U32 => uint.MinValue, + ISrcDstFmt.S32 => int.MinValue, + _ => throw new ArgumentException($"The type \"{type}\" is not a supported integer type."), + }; + } + + public static long GetIntMax(ISrcDstFmt type) + { + return type switch + { + ISrcDstFmt.U8 => byte.MaxValue, + ISrcDstFmt.S8 => sbyte.MaxValue, + ISrcDstFmt.U16 => ushort.MaxValue, + ISrcDstFmt.S16 => short.MaxValue, + ISrcDstFmt.U32 => uint.MaxValue, + ISrcDstFmt.S32 => int.MaxValue, + _ => throw new ArgumentException($"The type \"{type}\" is not a supported integer type."), + }; + } + + public static Operand GetPredLogicalOp(EmitterContext context, BoolOp logicOp, Operand input, Operand pred) + { + return logicOp switch + { + BoolOp.And => context.BitwiseAnd(input, pred), + BoolOp.Or => context.BitwiseOr(input, pred), + BoolOp.Xor => context.BitwiseExclusiveOr(input, pred), + _ => input, + }; + } + + public static Operand Extend(EmitterContext context, Operand src, VectorSelect type) + { + return type switch + { + VectorSelect.U8B0 => ZeroExtendTo32(context, context.ShiftRightU32(src, Const(0)), 8), + VectorSelect.U8B1 => ZeroExtendTo32(context, context.ShiftRightU32(src, Const(8)), 8), + VectorSelect.U8B2 => ZeroExtendTo32(context, context.ShiftRightU32(src, Const(16)), 8), + VectorSelect.U8B3 => ZeroExtendTo32(context, context.ShiftRightU32(src, Const(24)), 8), + VectorSelect.U16H0 => ZeroExtendTo32(context, context.ShiftRightU32(src, Const(0)), 16), + VectorSelect.U16H1 => ZeroExtendTo32(context, context.ShiftRightU32(src, Const(16)), 16), + VectorSelect.S8B0 => SignExtendTo32(context, context.ShiftRightU32(src, Const(0)), 8), + VectorSelect.S8B1 => SignExtendTo32(context, context.ShiftRightU32(src, Const(8)), 8), + VectorSelect.S8B2 => SignExtendTo32(context, context.ShiftRightU32(src, Const(16)), 8), + VectorSelect.S8B3 => SignExtendTo32(context, context.ShiftRightU32(src, Const(24)), 8), + VectorSelect.S16H0 => SignExtendTo32(context, context.ShiftRightU32(src, Const(0)), 16), + VectorSelect.S16H1 => SignExtendTo32(context, context.ShiftRightU32(src, Const(16)), 16), + _ => src, + }; + } + + public static void SetZnFlags(EmitterContext context, Operand dest, bool setCC, bool extended = false) + { + if (!setCC) + { + return; + } + + if (extended) + { + // When the operation is extended, it means we are doing + // the operation on a long word with any number of bits, + // so we need to AND the zero flag from result with the + // previous result when extended is specified, to ensure + // we have ZF set only if all words are zero, and not just + // the last one. + Operand oldZF = GetZF(); + + Operand res = context.BitwiseAnd(context.ICompareEqual(dest, Const(0)), oldZF); + + context.Copy(GetZF(), res); + } + else + { + context.Copy(GetZF(), context.ICompareEqual(dest, Const(0))); + } + + context.Copy(GetNF(), context.ICompareLess(dest, Const(0))); + } + + public static void SetFPZnFlags(EmitterContext context, Operand dest, bool setCC, Instruction fpType = Instruction.FP32) + { + if (setCC) + { + Operand zero = ConstF(0); + + if (fpType == Instruction.FP64) + { + zero = context.FP32ConvertToFP64(zero); + } + + context.Copy(GetZF(), context.FPCompareEqual(dest, zero, fpType)); + context.Copy(GetNF(), context.FPCompareLess(dest, zero, fpType)); + } + } + + public static (Operand, Operand) NegateLong(EmitterContext context, Operand low, Operand high) + { + low = context.BitwiseNot(low); + high = context.BitwiseNot(high); + low = AddWithCarry(context, low, Const(1), out Operand carryOut); + high = context.IAdd(high, carryOut); + return (low, high); + } + + public static Operand AddWithCarry(EmitterContext context, Operand lhs, Operand rhs, out Operand carryOut) + { + Operand result = context.IAdd(lhs, rhs); + + // C = Rd < Rn + carryOut = context.INegate(context.ICompareLessUnsigned(result, lhs)); + + return result; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitAttribute.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitAttribute.cs new file mode 100644 index 0000000..761a386 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitAttribute.cs @@ -0,0 +1,409 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; + +using static Ryujinx.Graphics.Shader.Instructions.InstEmitHelper; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void Al2p(EmitterContext context) + { + InstAl2p op = context.GetOp(); + + context.Copy(GetDest(op.Dest), context.IAdd(GetSrcReg(context, op.SrcA), Const(op.Imm11))); + } + + public static void Ald(EmitterContext context) + { + InstAld op = context.GetOp(); + + // Some of those attributes are per invocation, + // so we should ignore any primitive vertex indexing for those. + bool hasPrimitiveVertex = AttributeMap.HasPrimitiveVertex(context.TranslatorContext.Definitions.Stage, op.O) && !op.P; + + if (!op.Phys) + { + hasPrimitiveVertex &= HasPrimitiveVertex(op.Imm11); + } + + Operand primVertex = hasPrimitiveVertex ? context.Copy(GetSrcReg(context, op.SrcB)) : null; + + for (int index = 0; index < (int)op.AlSize + 1; index++) + { + Register rd = new(op.Dest + index, RegisterType.Gpr); + + if (rd.IsRZ) + { + break; + } + + if (op.Phys) + { + Operand offset = context.ISubtract(GetSrcReg(context, op.SrcA), Const(AttributeConsts.UserAttributeBase)); + Operand vecIndex = context.ShiftRightU32(offset, Const(4)); + Operand elemIndex = context.BitwiseAnd(context.ShiftRightU32(offset, Const(2)), Const(3)); + + StorageKind storageKind = op.O ? StorageKind.Output : StorageKind.Input; + + context.Copy(Register(rd), context.Load(storageKind, IoVariable.UserDefined, primVertex, vecIndex, elemIndex)); + } + else if (op.SrcB == RegisterConsts.RegisterZeroIndex || op.P) + { + int offset = FixedFuncToUserAttribute(context.TranslatorContext, op.Imm11 + index * 4, op.O); + bool isOutput = op.O && CanLoadOutput(offset); + + if (!op.P && !isOutput && TryConvertIdToIndexForVulkan(context, offset, out Operand value)) + { + context.Copy(Register(rd), value); + } + else + { + value = AttributeMap.GenerateAttributeLoad(context, primVertex, offset, isOutput, op.P); + + if ((!context.TranslatorContext.Definitions.SupportsScaledVertexFormats || context.VertexAsCompute) && + context.TranslatorContext.Stage == ShaderStage.Vertex && + !op.O && + offset >= 0x80 && + offset < 0x280) + { + // The host does not support scaled vertex formats, + // the emulator should use a integer format, and + // we compensate here inserting the conversion to float. + + AttributeType type = context.TranslatorContext.Definitions.GetAttributeType((offset - 0x80) >> 4); + + if (type == AttributeType.Sscaled) + { + value = context.IConvertS32ToFP32(value); + } + else if (type == AttributeType.Uscaled) + { + value = context.IConvertU32ToFP32(value); + } + } + else if (offset == AttributeConsts.PrimitiveId && context.TranslatorContext.Definitions.HalvePrimitiveId) + { + value = context.ShiftRightS32(value, Const(1)); + } + + context.Copy(Register(rd), value); + } + } + else + { + int offset = FixedFuncToUserAttribute(context.TranslatorContext, op.Imm11 + index * 4, op.O); + bool isOutput = op.O && CanLoadOutput(offset); + + context.Copy(Register(rd), AttributeMap.GenerateAttributeLoad(context, primVertex, offset, isOutput, false)); + } + } + } + + public static void Ast(EmitterContext context) + { + InstAst op = context.GetOp(); + + for (int index = 0; index < (int)op.AlSize + 1; index++) + { + if (op.SrcB + index > RegisterConsts.RegisterZeroIndex) + { + break; + } + + Register rd = new(op.SrcB + index, RegisterType.Gpr); + + if (op.Phys) + { + Operand offset = context.ISubtract(GetSrcReg(context, op.SrcA), Const(AttributeConsts.UserAttributeBase)); + Operand vecIndex = context.ShiftRightU32(offset, Const(4)); + Operand elemIndex = context.BitwiseAnd(context.ShiftRightU32(offset, Const(2)), Const(3)); + Operand invocationId = AttributeMap.HasInvocationId(context.TranslatorContext.Definitions.Stage, isOutput: true) + ? context.Load(StorageKind.Input, IoVariable.InvocationId) + : null; + + context.Store(StorageKind.Output, IoVariable.UserDefined, invocationId, vecIndex, elemIndex, Register(rd)); + } + else + { + // TODO: Support indirect stores using Ra. + + int offset = op.Imm11 + index * 4; + + if (!context.TranslatorContext.AttributeUsage.IsUsedOutputAttribute(offset)) + { + return; + } + + offset = FixedFuncToUserAttribute(context.TranslatorContext, offset, isOutput: true); + AttributeMap.GenerateAttributeStore(context, offset, op.P, Register(rd)); + } + } + } + + public static void Ipa(EmitterContext context) + { + InstIpa op = context.GetOp(); + + Operand res; + + bool isFixedFunc = false; + + if (op.Idx) + { + Operand offset = context.ISubtract(GetSrcReg(context, op.SrcA), Const(AttributeConsts.UserAttributeBase)); + Operand vecIndex = context.ShiftRightU32(offset, Const(4)); + Operand elemIndex = context.BitwiseAnd(context.ShiftRightU32(offset, Const(2)), Const(3)); + + res = context.Load(StorageKind.Input, IoVariable.UserDefined, null, vecIndex, elemIndex); + res = context.FPMultiply(res, context.Load(StorageKind.Input, IoVariable.FragmentCoord, null, Const(3))); + } + else + { + isFixedFunc = TryFixedFuncToUserAttributeIpa(context, op.Imm10, out res); + + if (op.Imm10 is >= AttributeConsts.UserAttributeBase and < AttributeConsts.UserAttributeEnd) + { + int index = (op.Imm10 - AttributeConsts.UserAttributeBase) >> 4; + + if (context.TranslatorContext.Definitions.ImapTypes[index].GetFirstUsedType() == PixelImap.Perspective) + { + res = context.FPMultiply(res, context.Load(StorageKind.Input, IoVariable.FragmentCoord, null, Const(3))); + } + } + else if (op.Imm10 is AttributeConsts.PositionX or AttributeConsts.PositionY) + { + // FragCoord X/Y must be divided by the render target scale, if resolution scaling is active, + // because the shader code is not expecting scaled values. + res = context.FPDivide(res, context.Load(StorageKind.ConstantBuffer, SupportBuffer.Binding, Const((int)SupportBufferField.RenderScale), Const(0))); + + if (op.Imm10 == AttributeConsts.PositionY && context.TranslatorContext.Options.TargetApi != TargetApi.OpenGL) + { + // If YNegate is enabled, we need to flip the fragment coordinates vertically, unless + // the API supports changing the origin (only OpenGL does). + if (context.TranslatorContext.Definitions.YNegateEnabled) + { + Operand viewportHeight = context.Load(StorageKind.ConstantBuffer, 0, Const((int)SupportBufferField.ViewportSize), Const(1)); + + res = context.FPSubtract(viewportHeight, res); + } + } + } + else if (op.Imm10 == AttributeConsts.PrimitiveId && context.TranslatorContext.Definitions.HalvePrimitiveId) + { + // If quads are used, but the host does not support them, they need to be converted to triangles. + // Since each quad becomes 2 triangles, we need to compensate here and divide primitive ID by 2. + res = context.ShiftRightS32(res, Const(1)); + } + else if (op.Imm10 == AttributeConsts.FrontFacing && context.TranslatorContext.GpuAccessor.QueryHostHasFrontFacingBug()) + { + // gl_FrontFacing sometimes has incorrect (flipped) values depending how it is accessed on Intel GPUs. + // This weird trick makes it behave. + res = context.ICompareLess(context.INegate(context.FP32ConvertToS32(context.ConditionalSelect(res, ConstF(1f), ConstF(0f)))), Const(0)); + } + } + + if (op.IpaOp == IpaOp.Multiply && !isFixedFunc) + { + Operand srcB = GetSrcReg(context, op.SrcB); + + res = context.FPMultiply(res, srcB); + } + + res = context.FPSaturate(res, op.Sat); + + context.Copy(GetDest(op.Dest), res); + } + + public static void Isberd(EmitterContext context) + { + InstIsberd op = context.GetOp(); + + // This instruction performs a load from ISBE (Internal Stage Buffer Entry) memory. + // Here, we just propagate the offset, as the result from this instruction is usually + // used with ALD to perform vertex load on geometry or tessellation shaders. + // The offset is calculated as (PrimitiveIndex * VerticesPerPrimitive) + VertexIndex. + // Since we hardcode PrimitiveIndex to zero, then the offset will be just VertexIndex. + context.Copy(GetDest(op.Dest), GetSrcReg(context, op.SrcA)); + } + + public static void OutR(EmitterContext context) + { + InstOutR op = context.GetOp(); + + EmitOut(context, op.OutType.HasFlag(OutType.Emit), op.OutType.HasFlag(OutType.Cut)); + } + + public static void OutI(EmitterContext context) + { + InstOutI op = context.GetOp(); + + EmitOut(context, op.OutType.HasFlag(OutType.Emit), op.OutType.HasFlag(OutType.Cut)); + } + + public static void OutC(EmitterContext context) + { + InstOutC op = context.GetOp(); + + EmitOut(context, op.OutType.HasFlag(OutType.Emit), op.OutType.HasFlag(OutType.Cut)); + } + + private static void EmitOut(EmitterContext context, bool emit, bool cut) + { + if (!(emit || cut)) + { + context.TranslatorContext.GpuAccessor.Log("Invalid OUT encoding."); + } + + if (emit) + { + if (context.TranslatorContext.Definitions.LastInVertexPipeline) + { + context.PrepareForVertexReturn(out Operand tempXLocal, out Operand tempYLocal, out Operand tempZLocal); + + context.EmitVertex(); + + // Restore output position value before transformation. + + if (tempXLocal != null) + { + context.Copy(context.Load(StorageKind.Input, IoVariable.Position, null, Const(0)), tempXLocal); + } + + if (tempYLocal != null) + { + context.Copy(context.Load(StorageKind.Input, IoVariable.Position, null, Const(1)), tempYLocal); + } + + if (tempZLocal != null) + { + context.Copy(context.Load(StorageKind.Input, IoVariable.Position, null, Const(2)), tempZLocal); + } + } + else + { + context.EmitVertex(); + } + } + + if (cut) + { + context.EndPrimitive(); + } + } + + private static bool HasPrimitiveVertex(int attr) + { + return attr is not AttributeConsts.PrimitiveId and + not AttributeConsts.TessCoordX and + not AttributeConsts.TessCoordY; + } + + private static bool CanLoadOutput(int attr) + { + return attr is not AttributeConsts.TessCoordX and not AttributeConsts.TessCoordY; + } + + private static bool TryFixedFuncToUserAttributeIpa(EmitterContext context, int attr, out Operand selectedAttr) + { + if (attr is >= AttributeConsts.FrontColorDiffuseR and < AttributeConsts.BackColorDiffuseR) + { + // TODO: If two sided rendering is enabled, then this should return + // FrontColor if the fragment is front facing, and back color otherwise. + selectedAttr = GenerateIpaLoad(context, FixedFuncToUserAttribute(context.TranslatorContext, attr, isOutput: false)); + return true; + } + else if (attr == AttributeConsts.FogCoord) + { + // TODO: We likely need to emulate the fixed-function functionality for FogCoord here. + selectedAttr = GenerateIpaLoad(context, FixedFuncToUserAttribute(context.TranslatorContext, attr, isOutput: false)); + return true; + } + else if (attr is >= AttributeConsts.BackColorDiffuseR and < AttributeConsts.ClipDistance0) + { + selectedAttr = ConstF(((attr >> 2) & 3) == 3 ? 1f : 0f); + return true; + } + else if (attr is >= AttributeConsts.TexCoordBase and < AttributeConsts.TexCoordEnd) + { + selectedAttr = GenerateIpaLoad(context, FixedFuncToUserAttribute(context.TranslatorContext, attr, isOutput: false)); + return true; + } + + selectedAttr = GenerateIpaLoad(context, attr); + return false; + } + + private static Operand GenerateIpaLoad(EmitterContext context, int offset) + { + return AttributeMap.GenerateAttributeLoad(context, null, offset, isOutput: false, isPerPatch: false); + } + + private static int FixedFuncToUserAttribute(TranslatorContext translatorContext, int attr, bool isOutput) + { + bool supportsLayerFromVertexOrTess = translatorContext.GpuAccessor.QueryHostSupportsLayerVertexTessellation(); + int fixedStartAttr = supportsLayerFromVertexOrTess ? 0 : 1; + + if (attr == AttributeConsts.Layer && translatorContext.Definitions.Stage != ShaderStage.Geometry && !supportsLayerFromVertexOrTess) + { + attr = FixedFuncToUserAttribute(translatorContext, attr, AttributeConsts.Layer, 0, isOutput); + translatorContext.SetLayerOutputAttribute(attr); + } + else if (attr == AttributeConsts.FogCoord) + { + attr = FixedFuncToUserAttribute(translatorContext, attr, AttributeConsts.FogCoord, fixedStartAttr, isOutput); + } + else if (attr is >= AttributeConsts.FrontColorDiffuseR and < AttributeConsts.ClipDistance0) + { + attr = FixedFuncToUserAttribute(translatorContext, attr, AttributeConsts.FrontColorDiffuseR, fixedStartAttr + 1, isOutput); + } + else if (attr is >= AttributeConsts.TexCoordBase and < AttributeConsts.TexCoordEnd) + { + attr = FixedFuncToUserAttribute(translatorContext, attr, AttributeConsts.TexCoordBase, fixedStartAttr + 5, isOutput); + } + + return attr; + } + + private static int FixedFuncToUserAttribute(TranslatorContext translatorContext, int attr, int baseAttr, int baseIndex, bool isOutput) + { + int index = (attr - baseAttr) >> 4; + int userAttrIndex = translatorContext.AttributeUsage.GetFreeUserAttribute(isOutput, baseIndex + index); + + if ((uint)userAttrIndex < Constants.MaxAttributes) + { + attr = AttributeConsts.UserAttributeBase + userAttrIndex * 16 + (attr & 0xf); + } + else + { + translatorContext.GpuAccessor.Log($"No enough user attributes for fixed attribute offset 0x{attr:X}."); + } + + return attr; + } + + private static bool TryConvertIdToIndexForVulkan(EmitterContext context, int attr, out Operand value) + { + if (context.TranslatorContext.Options.TargetApi == TargetApi.Vulkan) + { + if (attr == AttributeConsts.InstanceId) + { + value = context.ISubtract( + context.Load(StorageKind.Input, IoVariable.InstanceIndex), + context.Load(StorageKind.Input, IoVariable.BaseInstance)); + return true; + } + else if (attr == AttributeConsts.VertexId) + { + value = context.Load(StorageKind.Input, IoVariable.VertexIndex); + return true; + } + } + + value = null; + return false; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitBarrier.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitBarrier.cs new file mode 100644 index 0000000..825cf9f --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitBarrier.cs @@ -0,0 +1,46 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.Translation; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void Bar(EmitterContext context) + { + InstBar op = context.GetOp(); + + // TODO: Support other modes. + if (op.BarOp == BarOp.Sync) + { + context.Barrier(); + } + else + { + context.TranslatorContext.GpuAccessor.Log($"Invalid barrier mode: {op.BarOp}."); + } + } + + public static void Depbar(EmitterContext context) + { +#pragma warning disable IDE0059 // Remove unnecessary value assignment + InstDepbar op = context.GetOp(); +#pragma warning restore IDE0059 + + // No operation. + } + + public static void Membar(EmitterContext context) + { + InstMembar op = context.GetOp(); + + if (op.Membar == Decoders.Membar.Cta) + { + context.GroupMemoryBarrier(); + } + else + { + context.MemoryBarrier(); + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitBitfield.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitBitfield.cs new file mode 100644 index 0000000..3146468 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitBitfield.cs @@ -0,0 +1,194 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; + +using static Ryujinx.Graphics.Shader.Instructions.InstEmitHelper; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void BfeR(EmitterContext context) + { + InstBfeR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + + EmitBfe(context, srcA, srcB, op.Dest, op.Brev, op.Signed); + } + + public static void BfeI(EmitterContext context) + { + InstBfeI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToSInt(op.Imm20)); + + EmitBfe(context, srcA, srcB, op.Dest, op.Brev, op.Signed); + } + + public static void BfeC(EmitterContext context) + { + InstBfeC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitBfe(context, srcA, srcB, op.Dest, op.Brev, op.Signed); + } + + public static void BfiR(EmitterContext context) + { + InstBfiR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitBfi(context, srcA, srcB, srcC, op.Dest); + } + + public static void BfiI(EmitterContext context) + { + InstBfiI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToSInt(op.Imm20)); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitBfi(context, srcA, srcB, srcC, op.Dest); + } + + public static void BfiC(EmitterContext context) + { + InstBfiC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitBfi(context, srcA, srcB, srcC, op.Dest); + } + + public static void BfiRc(EmitterContext context) + { + InstBfiRc op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcC); + Operand srcC = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitBfi(context, srcA, srcB, srcC, op.Dest); + } + + public static void FloR(EmitterContext context) + { + InstFloR op = context.GetOp(); + + EmitFlo(context, GetSrcReg(context, op.SrcB), op.Dest, op.NegB, op.Sh, op.Signed); + } + + public static void FloI(EmitterContext context) + { + InstFloI op = context.GetOp(); + + EmitFlo(context, GetSrcImm(context, Imm20ToSInt(op.Imm20)), op.Dest, op.NegB, op.Sh, op.Signed); + } + + public static void FloC(EmitterContext context) + { + InstFloC op = context.GetOp(); + + EmitFlo(context, GetSrcCbuf(context, op.CbufSlot, op.CbufOffset), op.Dest, op.NegB, op.Sh, op.Signed); + } + + public static void PopcR(EmitterContext context) + { + InstPopcR op = context.GetOp(); + + EmitPopc(context, GetSrcReg(context, op.SrcB), op.Dest, op.NegB); + } + + public static void PopcI(EmitterContext context) + { + InstPopcI op = context.GetOp(); + + EmitPopc(context, GetSrcImm(context, Imm20ToSInt(op.Imm20)), op.Dest, op.NegB); + } + + public static void PopcC(EmitterContext context) + { + InstPopcC op = context.GetOp(); + + EmitPopc(context, GetSrcCbuf(context, op.CbufSlot, op.CbufOffset), op.Dest, op.NegB); + } + + private static void EmitBfe( + EmitterContext context, + Operand srcA, + Operand srcB, + int rd, + bool bitReverse, + bool isSigned) + { + if (bitReverse) + { + srcA = context.BitfieldReverse(srcA); + } + + Operand position = context.BitwiseAnd(srcB, Const(0xff)); + + Operand size = context.BitfieldExtractU32(srcB, Const(8), Const(8)); + + Operand res = isSigned + ? context.BitfieldExtractS32(srcA, position, size) + : context.BitfieldExtractU32(srcA, position, size); + + context.Copy(GetDest(rd), res); + + // TODO: CC, X, corner cases. + } + + private static void EmitBfi(EmitterContext context, Operand srcA, Operand srcB, Operand srcC, int rd) + { + Operand position = context.BitwiseAnd(srcB, Const(0xff)); + + Operand size = context.BitfieldExtractU32(srcB, Const(8), Const(8)); + + Operand res = context.BitfieldInsert(srcC, srcA, position, size); + + context.Copy(GetDest(rd), res); + } + + private static void EmitFlo(EmitterContext context, Operand src, int rd, bool invert, bool sh, bool isSigned) + { + Operand srcB = context.BitwiseNot(src, invert); + + Operand res; + + if (sh) + { + res = context.FindLSB(context.BitfieldReverse(srcB)); + } + else + { + res = isSigned + ? context.FindMSBS32(srcB) + : context.FindMSBU32(srcB); + } + + context.Copy(GetDest(rd), res); + } + + private static void EmitPopc(EmitterContext context, Operand src, int rd, bool invert) + { + Operand srcB = context.BitwiseNot(src, invert); + + Operand res = context.BitCount(srcB); + + context.Copy(GetDest(rd), res); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitConditionCode.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitConditionCode.cs new file mode 100644 index 0000000..6858dd3 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitConditionCode.cs @@ -0,0 +1,86 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using static Ryujinx.Graphics.Shader.Instructions.InstEmitAluHelper; +using static Ryujinx.Graphics.Shader.Instructions.InstEmitHelper; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void Cset(EmitterContext context) + { + InstCset op = context.GetOp(); + + Operand res = GetCondition(context, op.Ccc); + Operand srcPred = GetPredicate(context, op.SrcPred, op.SrcPredInv); + + res = GetPredLogicalOp(context, op.Bop, res, srcPred); + + Operand dest = GetDest(op.Dest); + + if (op.BVal) + { + context.Copy(dest, context.ConditionalSelect(res, ConstF(1), Const(0))); + } + else + { + context.Copy(dest, res); + } + + // TODO: CC. + } + + public static void Csetp(EmitterContext context) + { + InstCsetp op = context.GetOp(); + + Operand p0Res = GetCondition(context, op.Ccc); + Operand p1Res = context.BitwiseNot(p0Res); + Operand srcPred = GetPredicate(context, op.SrcPred, op.SrcPredInv); + + p0Res = GetPredLogicalOp(context, op.Bop, p0Res, srcPred); + p1Res = GetPredLogicalOp(context, op.Bop, p1Res, srcPred); + + context.Copy(Register(op.DestPred, RegisterType.Predicate), p0Res); + context.Copy(Register(op.DestPredInv, RegisterType.Predicate), p1Res); + + // TODO: CC. + } + + private static Operand GetCondition(EmitterContext context, Ccc cond, int defaultCond = IrConsts.True) + { + return cond switch + { + Ccc.F => Const(IrConsts.False), + Ccc.Lt => context.BitwiseExclusiveOr(context.BitwiseAnd(GetNF(), context.BitwiseNot(GetZF())), GetVF()), + Ccc.Eq => context.BitwiseAnd(context.BitwiseNot(GetNF()), GetZF()), + Ccc.Le => context.BitwiseExclusiveOr(GetNF(), context.BitwiseOr(GetZF(), GetVF())), + Ccc.Gt => context.BitwiseNot(context.BitwiseOr(context.BitwiseExclusiveOr(GetNF(), GetVF()), GetZF())), + Ccc.Ne => context.BitwiseNot(GetZF()), + Ccc.Ge => context.BitwiseNot(context.BitwiseExclusiveOr(GetNF(), GetVF())), + Ccc.Num => context.BitwiseNot(context.BitwiseAnd(GetNF(), GetZF())), + Ccc.Nan => context.BitwiseAnd(GetNF(), GetZF()), + Ccc.Ltu => context.BitwiseExclusiveOr(GetNF(), GetVF()), + Ccc.Equ => GetZF(), + Ccc.Leu => context.BitwiseOr(context.BitwiseExclusiveOr(GetNF(), GetVF()), GetZF()), + Ccc.Gtu => context.BitwiseExclusiveOr(context.BitwiseNot(GetNF()), context.BitwiseOr(GetVF(), GetZF())), + Ccc.Neu => context.BitwiseOr(GetNF(), context.BitwiseNot(GetZF())), + Ccc.Geu => context.BitwiseExclusiveOr(context.BitwiseOr(context.BitwiseNot(GetNF()), GetZF()), GetVF()), + Ccc.T => Const(IrConsts.True), + Ccc.Off => context.BitwiseNot(GetVF()), + Ccc.Lo => context.BitwiseNot(GetCF()), + Ccc.Sff => context.BitwiseNot(GetNF()), + Ccc.Ls => context.BitwiseOr(GetZF(), context.BitwiseNot(GetCF())), + Ccc.Hi => context.BitwiseAnd(GetCF(), context.BitwiseNot(GetZF())), + Ccc.Sft => GetNF(), + Ccc.Hs => GetCF(), + Ccc.Oft => GetVF(), + Ccc.Rle => context.BitwiseOr(GetNF(), GetZF()), + Ccc.Rgt => context.BitwiseNot(context.BitwiseOr(GetNF(), GetZF())), + _ => Const(defaultCond), + }; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitConversion.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitConversion.cs new file mode 100644 index 0000000..7b8ce41 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitConversion.cs @@ -0,0 +1,424 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using System; +using static Ryujinx.Graphics.Shader.Instructions.InstEmitAluHelper; +using static Ryujinx.Graphics.Shader.Instructions.InstEmitHelper; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void F2fR(EmitterContext context) + { + InstF2fR op = context.GetOp(); + + Operand src = UnpackReg(context, op.SrcFmt, op.Sh, op.SrcB); + + EmitF2F(context, op.SrcFmt, op.DstFmt, op.RoundMode, src, op.Dest, op.AbsB, op.NegB, op.Sat); + } + + public static void F2fI(EmitterContext context) + { + InstF2fI op = context.GetOp(); + + Operand src = UnpackImm(context, op.SrcFmt, op.Sh, Imm20ToFloat(op.Imm20)); + + EmitF2F(context, op.SrcFmt, op.DstFmt, op.RoundMode, src, op.Dest, op.AbsB, op.NegB, op.Sat); + } + + public static void F2fC(EmitterContext context) + { + InstF2fC op = context.GetOp(); + + Operand src = UnpackCbuf(context, op.SrcFmt, op.Sh, op.CbufSlot, op.CbufOffset); + + EmitF2F(context, op.SrcFmt, op.DstFmt, op.RoundMode, src, op.Dest, op.AbsB, op.NegB, op.Sat); + } + + public static void F2iR(EmitterContext context) + { + InstF2iR op = context.GetOp(); + + Operand src = UnpackReg(context, op.SrcFmt, op.Sh, op.SrcB); + + EmitF2I(context, op.SrcFmt, op.IDstFmt, op.RoundMode, src, op.Dest, op.AbsB, op.NegB); + } + + public static void F2iI(EmitterContext context) + { + InstF2iI op = context.GetOp(); + + Operand src = UnpackImm(context, op.SrcFmt, op.Sh, Imm20ToFloat(op.Imm20)); + + EmitF2I(context, op.SrcFmt, op.IDstFmt, op.RoundMode, src, op.Dest, op.AbsB, op.NegB); + } + + public static void F2iC(EmitterContext context) + { + InstF2iC op = context.GetOp(); + + Operand src = UnpackCbuf(context, op.SrcFmt, op.Sh, op.CbufSlot, op.CbufOffset); + + EmitF2I(context, op.SrcFmt, op.IDstFmt, op.RoundMode, src, op.Dest, op.AbsB, op.NegB); + } + + public static void I2fR(EmitterContext context) + { + InstI2fR op = context.GetOp(); + + Operand src = GetSrcReg(context, op.SrcB); + + EmitI2F(context, op.ISrcFmt, op.DstFmt, src, op.ByteSel, op.Dest, op.AbsB, op.NegB); + } + + public static void I2fI(EmitterContext context) + { + InstI2fI op = context.GetOp(); + + Operand src = GetSrcImm(context, Imm20ToSInt(op.Imm20)); + + EmitI2F(context, op.ISrcFmt, op.DstFmt, src, op.ByteSel, op.Dest, op.AbsB, op.NegB); + } + + public static void I2fC(EmitterContext context) + { + InstI2fC op = context.GetOp(); + + Operand src = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitI2F(context, op.ISrcFmt, op.DstFmt, src, op.ByteSel, op.Dest, op.AbsB, op.NegB); + } + + public static void I2iR(EmitterContext context) + { + InstI2iR op = context.GetOp(); + + Operand src = GetSrcReg(context, op.SrcB); + + EmitI2I(context, op.ISrcFmt, op.IDstFmt, src, op.ByteSel, op.Dest, op.AbsB, op.NegB, op.Sat, op.WriteCC); + } + + public static void I2iI(EmitterContext context) + { + InstI2iI op = context.GetOp(); + + Operand src = GetSrcImm(context, Imm20ToSInt(op.Imm20)); + + EmitI2I(context, op.ISrcFmt, op.IDstFmt, src, op.ByteSel, op.Dest, op.AbsB, op.NegB, op.Sat, op.WriteCC); + } + + public static void I2iC(EmitterContext context) + { + InstI2iC op = context.GetOp(); + + Operand src = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitI2I(context, op.ISrcFmt, op.IDstFmt, src, op.ByteSel, op.Dest, op.AbsB, op.NegB, op.Sat, op.WriteCC); + } + + private static void EmitF2F( + EmitterContext context, + DstFmt srcType, + DstFmt dstType, + IntegerRound roundingMode, + Operand src, + int rd, + bool absolute, + bool negate, + bool saturate) + { + Operand srcB = context.FPAbsNeg(src, absolute, negate, srcType.ToInstFPType()); + + if (srcType == dstType) + { + srcB = roundingMode switch + { + IntegerRound.Round => context.FPRound(srcB, srcType.ToInstFPType()), + IntegerRound.Floor => context.FPFloor(srcB, srcType.ToInstFPType()), + IntegerRound.Ceil => context.FPCeiling(srcB, srcType.ToInstFPType()), + IntegerRound.Trunc => context.FPTruncate(srcB, srcType.ToInstFPType()), + _ => srcB, + }; + } + + // We don't need to handle conversions between FP16 <-> FP32 + // since we do FP16 operations as FP32 directly. + // FP16 <-> FP64 conversions are invalid. + if (srcType == DstFmt.F32 && dstType == DstFmt.F64) + { + srcB = context.FP32ConvertToFP64(srcB); + } + else if (srcType == DstFmt.F64 && dstType == DstFmt.F32) + { + srcB = context.FP64ConvertToFP32(srcB); + } + + srcB = context.FPSaturate(srcB, saturate, dstType.ToInstFPType()); + + WriteFP(context, dstType, srcB, rd); + + // TODO: CC. + } + + private static void EmitF2I( + EmitterContext context, + DstFmt srcType, + IDstFmt dstType, + RoundMode2 roundingMode, + Operand src, + int rd, + bool absolute, + bool negate) + { + if (dstType == IDstFmt.U64) + { + context.TranslatorContext.GpuAccessor.Log("Unimplemented 64-bits F2I."); + } + + Instruction fpType = srcType.ToInstFPType(); + + bool isSignedInt = dstType is IDstFmt.S16 or IDstFmt.S32 or IDstFmt.S64; + bool isSmallInt = dstType is IDstFmt.U16 or IDstFmt.S16; + + Operand srcB = context.FPAbsNeg(src, absolute, negate, fpType); + + srcB = roundingMode switch + { + RoundMode2.Round => context.FPRound(srcB, fpType), + RoundMode2.Floor => context.FPFloor(srcB, fpType), + RoundMode2.Ceil => context.FPCeiling(srcB, fpType), + RoundMode2.Trunc => context.FPTruncate(srcB, fpType), + _ => srcB, + }; + + if (!isSignedInt) + { + // Negative float to uint cast is undefined, so we clamp the value before conversion. + Operand c0 = srcType == DstFmt.F64 ? context.PackDouble2x32(0.0) : ConstF(0); + + srcB = context.FPMaximum(srcB, c0, fpType); + } + + if (srcType == DstFmt.F64) + { + srcB = isSignedInt + ? context.FP64ConvertToS32(srcB) + : context.FP64ConvertToU32(srcB); + } + else + { + srcB = isSignedInt + ? context.FP32ConvertToS32(srcB) + : context.FP32ConvertToU32(srcB); + } + + if (isSmallInt) + { + int min = (int)GetIntMin(dstType); + int max = (int)GetIntMax(dstType); + + srcB = isSignedInt + ? context.IClampS32(srcB, Const(min), Const(max)) + : context.IClampU32(srcB, Const(min), Const(max)); + } + + Operand dest = GetDest(rd); + + context.Copy(dest, srcB); + + // TODO: CC. + } + + private static void EmitI2F( + EmitterContext context, + ISrcFmt srcType, + DstFmt dstType, + Operand src, + ByteSel byteSelection, + int rd, + bool absolute, + bool negate) + { + bool isSignedInt = + srcType is ISrcFmt.S8 or + ISrcFmt.S16 or + ISrcFmt.S32 or + ISrcFmt.S64; + bool isSmallInt = + srcType is ISrcFmt.U16 or + ISrcFmt.S16 or + ISrcFmt.U8 or + ISrcFmt.S8; + + // TODO: Handle S/U64. + + Operand srcB = context.IAbsNeg(src, absolute, negate); + + if (isSmallInt) + { + int size = srcType is ISrcFmt.U16 or ISrcFmt.S16 ? 16 : 8; + + srcB = isSignedInt + ? context.BitfieldExtractS32(srcB, Const((int)byteSelection * 8), Const(size)) + : context.BitfieldExtractU32(srcB, Const((int)byteSelection * 8), Const(size)); + } + + if (dstType == DstFmt.F64) + { + srcB = isSignedInt + ? context.IConvertS32ToFP64(srcB) + : context.IConvertU32ToFP64(srcB); + } + else + { + srcB = isSignedInt + ? context.IConvertS32ToFP32(srcB) + : context.IConvertU32ToFP32(srcB); + } + + WriteFP(context, dstType, srcB, rd); + + // TODO: CC. + } + + private static void EmitI2I( + EmitterContext context, + ISrcDstFmt srcType, + ISrcDstFmt dstType, + Operand src, + ByteSel byteSelection, + int rd, + bool absolute, + bool negate, + bool saturate, + bool writeCC) + { + if ((srcType & ~ISrcDstFmt.S8) > ISrcDstFmt.U32 || (dstType & ~ISrcDstFmt.S8) > ISrcDstFmt.U32) + { + context.TranslatorContext.GpuAccessor.Log("Invalid I2I encoding."); + return; + } + + bool srcIsSignedInt = + srcType is ISrcDstFmt.S8 or + ISrcDstFmt.S16 or + ISrcDstFmt.S32; + bool dstIsSignedInt = + dstType is ISrcDstFmt.S8 or + ISrcDstFmt.S16 or + ISrcDstFmt.S32; + bool srcIsSmallInt = + srcType is ISrcDstFmt.U16 or + ISrcDstFmt.S16 or + ISrcDstFmt.U8 or + ISrcDstFmt.S8; + + if (srcIsSmallInt) + { + int size = srcType is ISrcDstFmt.U16 or ISrcDstFmt.S16 ? 16 : 8; + + src = srcIsSignedInt + ? context.BitfieldExtractS32(src, Const((int)byteSelection * 8), Const(size)) + : context.BitfieldExtractU32(src, Const((int)byteSelection * 8), Const(size)); + } + + src = context.IAbsNeg(src, absolute, negate); + + if (saturate) + { + int min = (int)GetIntMin(dstType); + int max = (int)GetIntMax(dstType); + + src = dstIsSignedInt + ? context.IClampS32(src, Const(min), Const(max)) + : context.IClampU32(src, Const(min), Const(max)); + } + + context.Copy(GetDest(rd), src); + + SetZnFlags(context, src, writeCC); + } + + private static Operand UnpackReg(EmitterContext context, DstFmt floatType, bool h, int reg) + { + if (floatType == DstFmt.F32) + { + return GetSrcReg(context, reg); + } + else if (floatType == DstFmt.F16) + { + return GetHalfUnpacked(context, GetSrcReg(context, reg), HalfSwizzle.F16)[h ? 1 : 0]; + } + else if (floatType == DstFmt.F64) + { + return GetSrcReg(context, reg, isFP64: true); + } + + throw new ArgumentException($"Invalid floating point type \"{floatType}\"."); + } + + private static Operand UnpackCbuf(EmitterContext context, DstFmt floatType, bool h, int cbufSlot, int cbufOffset) + { + if (floatType == DstFmt.F32) + { + return GetSrcCbuf(context, cbufSlot, cbufOffset); + } + else if (floatType == DstFmt.F16) + { + return GetHalfUnpacked(context, GetSrcCbuf(context, cbufSlot, cbufOffset), HalfSwizzle.F16)[h ? 1 : 0]; + } + else if (floatType == DstFmt.F64) + { + return GetSrcCbuf(context, cbufSlot, cbufOffset, isFP64: true); + } + + throw new ArgumentException($"Invalid floating point type \"{floatType}\"."); + } + + private static Operand UnpackImm(EmitterContext context, DstFmt floatType, bool h, int imm) + { + if (floatType == DstFmt.F32) + { + return GetSrcImm(context, imm); + } + else if (floatType == DstFmt.F16) + { + return GetHalfUnpacked(context, GetSrcImm(context, imm), HalfSwizzle.F16)[h ? 1 : 0]; + } + else if (floatType == DstFmt.F64) + { + return GetSrcImm(context, imm, isFP64: true); + } + + throw new ArgumentException($"Invalid floating point type \"{floatType}\"."); + } + + private static void WriteFP(EmitterContext context, DstFmt type, Operand srcB, int rd) + { + Operand dest = GetDest(rd); + + if (type == DstFmt.F32) + { + context.Copy(dest, srcB); + } + else if (type == DstFmt.F16) + { + context.Copy(dest, context.PackHalf2x16(srcB, ConstF(0))); + } + else /* if (type == FPType.FP64) */ + { + Operand dest2 = GetDest2(rd); + + context.Copy(dest, context.UnpackDouble2x32Low(srcB)); + context.Copy(dest2, context.UnpackDouble2x32High(srcB)); + } + } + + private static Instruction ToInstFPType(this DstFmt type) + { + return type == DstFmt.F64 ? Instruction.FP64 : Instruction.FP32; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitFloatArithmetic.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitFloatArithmetic.cs new file mode 100644 index 0000000..469054f --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitFloatArithmetic.cs @@ -0,0 +1,531 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using static Ryujinx.Graphics.Shader.Instructions.InstEmitAluHelper; +using static Ryujinx.Graphics.Shader.Instructions.InstEmitHelper; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void DaddR(EmitterContext context) + { + InstDaddR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA, isFP64: true); + Operand srcB = GetSrcReg(context, op.SrcB, isFP64: true); + + EmitFadd(context, Instruction.FP64, srcA, srcB, op.Dest, op.NegA, op.NegB, op.AbsA, op.AbsB, false, op.WriteCC); + } + + public static void DaddI(EmitterContext context) + { + InstDaddI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA, isFP64: true); + Operand srcB = GetSrcImm(context, Imm20ToFloat(op.Imm20), isFP64: true); + + EmitFadd(context, Instruction.FP64, srcA, srcB, op.Dest, op.NegA, op.NegB, op.AbsA, op.AbsB, false, op.WriteCC); + } + + public static void DaddC(EmitterContext context) + { + InstDaddC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA, isFP64: true); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset, isFP64: true); + + EmitFadd(context, Instruction.FP64, srcA, srcB, op.Dest, op.NegA, op.NegB, op.AbsA, op.AbsB, false, op.WriteCC); + } + + public static void DfmaR(EmitterContext context) + { + InstDfmaR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA, isFP64: true); + Operand srcB = GetSrcReg(context, op.SrcB, isFP64: true); + Operand srcC = GetSrcReg(context, op.SrcC, isFP64: true); + + EmitFfma(context, Instruction.FP64, srcA, srcB, srcC, op.Dest, op.NegA, op.NegC, false, op.WriteCC); + } + + public static void DfmaI(EmitterContext context) + { + InstDfmaI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA, isFP64: true); + Operand srcB = GetSrcImm(context, Imm20ToFloat(op.Imm20), isFP64: true); + Operand srcC = GetSrcReg(context, op.SrcC, isFP64: true); + + EmitFfma(context, Instruction.FP64, srcA, srcB, srcC, op.Dest, op.NegA, op.NegC, false, op.WriteCC); + } + + public static void DfmaC(EmitterContext context) + { + InstDfmaC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA, isFP64: true); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset, isFP64: true); + Operand srcC = GetSrcReg(context, op.SrcC, isFP64: true); + + EmitFfma(context, Instruction.FP64, srcA, srcB, srcC, op.Dest, op.NegA, op.NegC, false, op.WriteCC); + } + + public static void DfmaRc(EmitterContext context) + { + InstDfmaRc op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA, isFP64: true); + Operand srcB = GetSrcReg(context, op.SrcC, isFP64: true); + Operand srcC = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset, isFP64: true); + + EmitFfma(context, Instruction.FP64, srcA, srcB, srcC, op.Dest, op.NegA, op.NegC, false, op.WriteCC); + } + + public static void DmulR(EmitterContext context) + { + InstDmulR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA, isFP64: true); + Operand srcB = GetSrcReg(context, op.SrcB, isFP64: true); + + EmitFmul(context, Instruction.FP64, MultiplyScale.NoScale, srcA, srcB, op.Dest, op.NegA, false, op.WriteCC); + } + + public static void DmulI(EmitterContext context) + { + InstDmulI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA, isFP64: true); + Operand srcB = GetSrcImm(context, Imm20ToFloat(op.Imm20), isFP64: true); + + EmitFmul(context, Instruction.FP64, MultiplyScale.NoScale, srcA, srcB, op.Dest, op.NegA, false, op.WriteCC); + } + + public static void DmulC(EmitterContext context) + { + InstDmulC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA, isFP64: true); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset, isFP64: true); + + EmitFmul(context, Instruction.FP64, MultiplyScale.NoScale, srcA, srcB, op.Dest, op.NegA, false, op.WriteCC); + } + + public static void FaddR(EmitterContext context) + { + InstFaddR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + + EmitFadd(context, Instruction.FP32, srcA, srcB, op.Dest, op.NegA, op.NegB, op.AbsA, op.AbsB, op.Sat, op.WriteCC); + } + + public static void FaddI(EmitterContext context) + { + InstFaddI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToFloat(op.Imm20)); + + EmitFadd(context, Instruction.FP32, srcA, srcB, op.Dest, op.NegA, op.NegB, op.AbsA, op.AbsB, op.Sat, op.WriteCC); + } + + public static void FaddC(EmitterContext context) + { + InstFaddC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitFadd(context, Instruction.FP32, srcA, srcB, op.Dest, op.NegA, op.NegB, op.AbsA, op.AbsB, op.Sat, op.WriteCC); + } + + public static void Fadd32i(EmitterContext context) + { + InstFadd32i op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, op.Imm32); + + EmitFadd(context, Instruction.FP32, srcA, srcB, op.Dest, op.NegA, op.NegB, op.AbsA, op.AbsB, false, op.WriteCC); + } + + public static void FfmaR(EmitterContext context) + { + InstFfmaR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitFfma(context, Instruction.FP32, srcA, srcB, srcC, op.Dest, op.NegA, op.NegC, op.Sat, op.WriteCC); + } + + public static void FfmaI(EmitterContext context) + { + InstFfmaI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToFloat(op.Imm20)); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitFfma(context, Instruction.FP32, srcA, srcB, srcC, op.Dest, op.NegA, op.NegC, op.Sat, op.WriteCC); + } + + public static void FfmaC(EmitterContext context) + { + InstFfmaC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitFfma(context, Instruction.FP32, srcA, srcB, srcC, op.Dest, op.NegA, op.NegC, op.Sat, op.WriteCC); + } + + public static void FfmaRc(EmitterContext context) + { + InstFfmaRc op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcC); + Operand srcC = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitFfma(context, Instruction.FP32, srcA, srcB, srcC, op.Dest, op.NegA, op.NegC, op.Sat, op.WriteCC); + } + + public static void Ffma32i(EmitterContext context) + { + InstFfma32i op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, op.Imm32); + Operand srcC = GetSrcReg(context, op.Dest); + + EmitFfma(context, Instruction.FP32, srcA, srcB, srcC, op.Dest, op.NegA, op.NegC, op.Sat, op.WriteCC); + } + + public static void FmulR(EmitterContext context) + { + InstFmulR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + + EmitFmul(context, Instruction.FP32, op.Scale, srcA, srcB, op.Dest, op.NegA, op.Sat, op.WriteCC); + } + + public static void FmulI(EmitterContext context) + { + InstFmulI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToFloat(op.Imm20)); + + EmitFmul(context, Instruction.FP32, op.Scale, srcA, srcB, op.Dest, op.NegA, op.Sat, op.WriteCC); + } + + public static void FmulC(EmitterContext context) + { + InstFmulC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitFmul(context, Instruction.FP32, op.Scale, srcA, srcB, op.Dest, op.NegA, op.Sat, op.WriteCC); + } + + public static void Fmul32i(EmitterContext context) + { + InstFmul32i op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, op.Imm32); + + EmitFmul(context, Instruction.FP32, MultiplyScale.NoScale, srcA, srcB, op.Dest, false, op.Sat, op.WriteCC); + } + + public static void Hadd2R(EmitterContext context) + { + InstHadd2R op = context.GetOp(); + + Operand[] srcA = GetHalfSrc(context, op.ASwizzle, op.SrcA, op.NegA, op.AbsA); + Operand[] srcB = GetHalfSrc(context, op.BSwizzle, op.SrcB, op.NegB, op.AbsB); + + EmitHadd2Hmul2(context, op.OFmt, srcA, srcB, isAdd: true, op.Dest, op.Sat); + } + + public static void Hadd2I(EmitterContext context) + { + InstHadd2I op = context.GetOp(); + + Operand[] srcA = GetHalfSrc(context, op.ASwizzle, op.SrcA, op.NegA, op.AbsA); + Operand[] srcB = GetHalfSrc(context, op.BimmH0, op.BimmH1); + + EmitHadd2Hmul2(context, op.OFmt, srcA, srcB, isAdd: true, op.Dest, op.Sat); + } + + public static void Hadd2C(EmitterContext context) + { + InstHadd2C op = context.GetOp(); + + Operand[] srcA = GetHalfSrc(context, op.ASwizzle, op.SrcA, op.NegA, op.AbsA); + Operand[] srcB = GetHalfSrc(context, HalfSwizzle.F32, op.CbufSlot, op.CbufOffset, op.NegB, op.AbsB); + + EmitHadd2Hmul2(context, op.OFmt, srcA, srcB, isAdd: true, op.Dest, op.Sat); + } + + public static void Hadd232i(EmitterContext context) + { + InstHadd232i op = context.GetOp(); + + Operand[] srcA = GetHalfSrc(context, op.ASwizzle, op.SrcA, op.NegA, false); + Operand[] srcB = GetHalfSrc(context, op.Imm); + + EmitHadd2Hmul2(context, OFmt.F16, srcA, srcB, isAdd: true, op.Dest, op.Sat); + } + + public static void Hfma2R(EmitterContext context) + { + InstHfma2R op = context.GetOp(); + + Operand[] srcA = GetHalfSrc(context, op.ASwizzle, op.SrcA, false, false); + Operand[] srcB = GetHalfSrc(context, op.BSwizzle, op.SrcB, op.NegA, false); + Operand[] srcC = GetHalfSrc(context, op.CSwizzle, op.SrcC, op.NegC, false); + + EmitHfma2(context, op.OFmt, srcA, srcB, srcC, op.Dest, op.Sat); + } + + public static void Hfma2I(EmitterContext context) + { + InstHfma2I op = context.GetOp(); + + Operand[] srcA = GetHalfSrc(context, op.ASwizzle, op.SrcA, false, false); + Operand[] srcB = GetHalfSrc(context, op.BimmH0, op.BimmH1); + Operand[] srcC = GetHalfSrc(context, op.CSwizzle, op.SrcC, op.NegC, false); + + EmitHfma2(context, op.OFmt, srcA, srcB, srcC, op.Dest, op.Sat); + } + + public static void Hfma2C(EmitterContext context) + { + InstHfma2C op = context.GetOp(); + + Operand[] srcA = GetHalfSrc(context, op.ASwizzle, op.SrcA, false, false); + Operand[] srcB = GetHalfSrc(context, HalfSwizzle.F32, op.CbufSlot, op.CbufOffset, op.NegA, false); + Operand[] srcC = GetHalfSrc(context, op.CSwizzle, op.SrcC, op.NegC, false); + + EmitHfma2(context, op.OFmt, srcA, srcB, srcC, op.Dest, op.Sat); + } + + public static void Hfma2Rc(EmitterContext context) + { + InstHfma2Rc op = context.GetOp(); + + Operand[] srcA = GetHalfSrc(context, op.ASwizzle, op.SrcA, false, false); + Operand[] srcB = GetHalfSrc(context, op.CSwizzle, op.SrcC, op.NegA, false); + Operand[] srcC = GetHalfSrc(context, HalfSwizzle.F32, op.CbufSlot, op.CbufOffset, op.NegC, false); + + EmitHfma2(context, op.OFmt, srcA, srcB, srcC, op.Dest, op.Sat); + } + + public static void Hfma232i(EmitterContext context) + { + InstHfma232i op = context.GetOp(); + + Operand[] srcA = GetHalfSrc(context, op.ASwizzle, op.SrcA, false, false); + Operand[] srcB = GetHalfSrc(context, op.Imm); + Operand[] srcC = GetHalfSrc(context, HalfSwizzle.F16, op.Dest, op.NegC, false); + + EmitHfma2(context, OFmt.F16, srcA, srcB, srcC, op.Dest, saturate: false); + } + + public static void Hmul2R(EmitterContext context) + { + InstHmul2R op = context.GetOp(); + + Operand[] srcA = GetHalfSrc(context, op.ASwizzle, op.SrcA, false, op.AbsA); + Operand[] srcB = GetHalfSrc(context, op.BSwizzle, op.SrcB, op.NegA, op.AbsB); + + EmitHadd2Hmul2(context, op.OFmt, srcA, srcB, isAdd: false, op.Dest, op.Sat); + } + + public static void Hmul2I(EmitterContext context) + { + InstHmul2I op = context.GetOp(); + + Operand[] srcA = GetHalfSrc(context, op.ASwizzle, op.SrcA, op.NegA, op.AbsA); + Operand[] srcB = GetHalfSrc(context, op.BimmH0, op.BimmH1); + + EmitHadd2Hmul2(context, op.OFmt, srcA, srcB, isAdd: false, op.Dest, op.Sat); + } + + public static void Hmul2C(EmitterContext context) + { + InstHmul2C op = context.GetOp(); + + Operand[] srcA = GetHalfSrc(context, op.ASwizzle, op.SrcA, false, op.AbsA); + Operand[] srcB = GetHalfSrc(context, HalfSwizzle.F32, op.CbufSlot, op.CbufOffset, op.NegA, op.AbsB); + + EmitHadd2Hmul2(context, op.OFmt, srcA, srcB, isAdd: false, op.Dest, op.Sat); + } + + public static void Hmul232i(EmitterContext context) + { + InstHmul232i op = context.GetOp(); + + Operand[] srcA = GetHalfSrc(context, op.ASwizzle, op.SrcA, false, false); + Operand[] srcB = GetHalfSrc(context, op.Imm32); + + EmitHadd2Hmul2(context, OFmt.F16, srcA, srcB, isAdd: false, op.Dest, op.Sat); + } + + private static void EmitFadd( + EmitterContext context, + Instruction fpType, + Operand srcA, + Operand srcB, + int rd, + bool negateA, + bool negateB, + bool absoluteA, + bool absoluteB, + bool saturate, + bool writeCC) + { + bool isFP64 = fpType == Instruction.FP64; + + srcA = context.FPAbsNeg(srcA, absoluteA, negateA, fpType); + srcB = context.FPAbsNeg(srcB, absoluteB, negateB, fpType); + + Operand res = context.FPSaturate(context.FPAdd(srcA, srcB, fpType), saturate, fpType); + + SetDest(context, res, rd, isFP64); + + SetFPZnFlags(context, res, writeCC, fpType); + } + + private static void EmitFfma( + EmitterContext context, + Instruction fpType, + Operand srcA, + Operand srcB, + Operand srcC, + int rd, + bool negateB, + bool negateC, + bool saturate, + bool writeCC) + { + bool isFP64 = fpType == Instruction.FP64; + + srcB = context.FPNegate(srcB, negateB, fpType); + srcC = context.FPNegate(srcC, negateC, fpType); + + Operand res = context.FPSaturate(context.FPFusedMultiplyAdd(srcA, srcB, srcC, fpType), saturate, fpType); + + SetDest(context, res, rd, isFP64); + + SetFPZnFlags(context, res, writeCC, fpType); + } + + private static void EmitFmul( + EmitterContext context, + Instruction fpType, + MultiplyScale scale, + Operand srcA, + Operand srcB, + int rd, + bool negateB, + bool saturate, + bool writeCC) + { + bool isFP64 = fpType == Instruction.FP64; + + srcB = context.FPNegate(srcB, negateB, fpType); + + if (scale != MultiplyScale.NoScale) + { + Operand scaleConst = scale switch + { + MultiplyScale.D2 => ConstF(0.5f), + MultiplyScale.D4 => ConstF(0.25f), + MultiplyScale.D8 => ConstF(0.125f), + MultiplyScale.M2 => ConstF(2f), + MultiplyScale.M4 => ConstF(4f), + MultiplyScale.M8 => ConstF(8f), + _ => ConstF(1f), // Invalid, behave as if it had no scale. + }; + + if (scaleConst.AsFloat() == 1f) + { + context.TranslatorContext.GpuAccessor.Log($"Invalid FP multiply scale \"{scale}\"."); + } + + if (isFP64) + { + scaleConst = context.FP32ConvertToFP64(scaleConst); + } + + srcA = context.FPMultiply(srcA, scaleConst, fpType); + } + + Operand res = context.FPSaturate(context.FPMultiply(srcA, srcB, fpType), saturate, fpType); + + SetDest(context, res, rd, isFP64); + + SetFPZnFlags(context, res, writeCC, fpType); + } + + private static void EmitHadd2Hmul2( + EmitterContext context, + OFmt swizzle, + Operand[] srcA, + Operand[] srcB, + bool isAdd, + int rd, + bool saturate) + { + Operand[] res = new Operand[2]; + + for (int index = 0; index < res.Length; index++) + { + if (isAdd) + { + res[index] = context.FPAdd(srcA[index], srcB[index]); + } + else + { + res[index] = context.FPMultiply(srcA[index], srcB[index]); + } + + res[index] = context.FPSaturate(res[index], saturate); + } + + context.Copy(GetDest(rd), GetHalfPacked(context, swizzle, res, rd)); + } + + public static void EmitHfma2( + EmitterContext context, + OFmt swizzle, + Operand[] srcA, + Operand[] srcB, + Operand[] srcC, + int rd, + bool saturate) + { + Operand[] res = new Operand[2]; + + for (int index = 0; index < res.Length; index++) + { + res[index] = context.FPFusedMultiplyAdd(srcA[index], srcB[index], srcC[index]); + res[index] = context.FPSaturate(res[index], saturate); + } + + context.Copy(GetDest(rd), GetHalfPacked(context, swizzle, res, rd)); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitFloatComparison.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitFloatComparison.cs new file mode 100644 index 0000000..27649e2 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitFloatComparison.cs @@ -0,0 +1,570 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using System; +using static Ryujinx.Graphics.Shader.Instructions.InstEmitAluHelper; +using static Ryujinx.Graphics.Shader.Instructions.InstEmitHelper; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void DsetR(EmitterContext context) + { + InstDsetR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA, isFP64: true); + Operand srcB = GetSrcReg(context, op.SrcB, isFP64: true); + + EmitFset( + context, + op.FComp, + op.Bop, + srcA, + srcB, + op.SrcPred, + op.SrcPredInv, + op.Dest, + op.AbsA, + op.AbsB, + op.NegA, + op.NegB, + op.BVal, + op.WriteCC, + isFP64: true); + } + + public static void DsetI(EmitterContext context) + { + InstDsetI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA, isFP64: true); + Operand srcB = GetSrcImm(context, Imm20ToFloat(op.Imm20), isFP64: true); + + EmitFset( + context, + op.FComp, + op.Bop, + srcA, + srcB, + op.SrcPred, + op.SrcPredInv, + op.Dest, + op.AbsA, + op.AbsB, + op.NegA, + op.NegB, + op.BVal, + op.WriteCC, + isFP64: true); + } + + public static void DsetC(EmitterContext context) + { + InstDsetC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA, isFP64: true); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset, isFP64: true); + + EmitFset( + context, + op.FComp, + op.Bop, + srcA, + srcB, + op.SrcPred, + op.SrcPredInv, + op.Dest, + op.AbsA, + op.AbsB, + op.NegA, + op.NegB, + op.BVal, + op.WriteCC, + isFP64: true); + } + + public static void DsetpR(EmitterContext context) + { + InstDsetpR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA, isFP64: true); + Operand srcB = GetSrcReg(context, op.SrcB, isFP64: true); + + EmitFsetp( + context, + op.FComp, + op.Bop, + srcA, + srcB, + op.SrcPred, + op.SrcPredInv, + op.DestPred, + op.DestPredInv, + op.AbsA, + op.AbsB, + op.NegA, + op.NegB, + writeCC: false, + isFP64: true); + } + + public static void DsetpI(EmitterContext context) + { + InstDsetpI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA, isFP64: true); + Operand srcB = GetSrcImm(context, Imm20ToFloat(op.Imm20), isFP64: true); + + EmitFsetp( + context, + op.FComp, + op.Bop, + srcA, + srcB, + op.SrcPred, + op.SrcPredInv, + op.DestPred, + op.DestPredInv, + op.AbsA, + op.AbsB, + op.NegA, + op.NegB, + writeCC: false, + isFP64: true); + } + + public static void DsetpC(EmitterContext context) + { + InstDsetpC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA, isFP64: true); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset, isFP64: true); + + EmitFsetp( + context, + op.FComp, + op.Bop, + srcA, + srcB, + op.SrcPred, + op.SrcPredInv, + op.DestPred, + op.DestPredInv, + op.AbsA, + op.AbsB, + op.NegA, + op.NegB, + writeCC: false, + isFP64: true); + } + + public static void FcmpR(EmitterContext context) + { + InstFcmpR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitFcmp(context, op.FComp, srcA, srcB, srcC, op.Dest); + } + + public static void FcmpI(EmitterContext context) + { + InstFcmpI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToFloat(op.Imm20)); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitFcmp(context, op.FComp, srcA, srcB, srcC, op.Dest); + } + + public static void FcmpC(EmitterContext context) + { + InstFcmpC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitFcmp(context, op.FComp, srcA, srcB, srcC, op.Dest); + } + + public static void FcmpRc(EmitterContext context) + { + InstFcmpRc op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcC); + Operand srcC = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitFcmp(context, op.FComp, srcA, srcB, srcC, op.Dest); + } + + public static void FsetR(EmitterContext context) + { + InstFsetR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + + EmitFset(context, op.FComp, op.Bop, srcA, srcB, op.SrcPred, op.SrcPredInv, op.Dest, op.AbsA, op.AbsB, op.NegA, op.NegB, op.BVal, op.WriteCC); + } + + public static void FsetC(EmitterContext context) + { + InstFsetC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitFset(context, op.FComp, op.Bop, srcA, srcB, op.SrcPred, op.SrcPredInv, op.Dest, op.AbsA, op.AbsB, op.NegA, op.NegB, op.BVal, op.WriteCC); + } + + public static void FsetI(EmitterContext context) + { + InstFsetI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToFloat(op.Imm20)); + + EmitFset(context, op.FComp, op.Bop, srcA, srcB, op.SrcPred, op.SrcPredInv, op.Dest, op.AbsA, op.AbsB, op.NegA, op.NegB, op.BVal, op.WriteCC); + } + + public static void FsetpR(EmitterContext context) + { + InstFsetpR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + + EmitFsetp( + context, + op.FComp, + op.Bop, + srcA, + srcB, + op.SrcPred, + op.SrcPredInv, + op.DestPred, + op.DestPredInv, + op.AbsA, + op.AbsB, + op.NegA, + op.NegB, + op.WriteCC); + } + + public static void FsetpI(EmitterContext context) + { + InstFsetpI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToFloat(op.Imm20)); + + EmitFsetp( + context, + op.FComp, + op.Bop, + srcA, + srcB, + op.SrcPred, + op.SrcPredInv, + op.DestPred, + op.DestPredInv, + op.AbsA, + op.AbsB, + op.NegA, + op.NegB, + op.WriteCC); + } + + public static void FsetpC(EmitterContext context) + { + InstFsetpC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitFsetp( + context, + op.FComp, + op.Bop, + srcA, + srcB, + op.SrcPred, + op.SrcPredInv, + op.DestPred, + op.DestPredInv, + op.AbsA, + op.AbsB, + op.NegA, + op.NegB, + op.WriteCC); + } + + public static void Hset2R(EmitterContext context) + { + InstHset2R op = context.GetOp(); + + Operand[] srcA = GetHalfSrc(context, op.ASwizzle, op.SrcA, op.NegA, op.AbsA); + Operand[] srcB = GetHalfSrc(context, op.BSwizzle, op.SrcB, op.NegB, op.AbsB); + + EmitHset2(context, op.Cmp, op.Bop, srcA, srcB, op.SrcPred, op.SrcPredInv, op.Dest, op.Bval); + } + + public static void Hset2I(EmitterContext context) + { + InstHset2I op = context.GetOp(); + + Operand[] srcA = GetHalfSrc(context, op.ASwizzle, op.SrcA, op.NegA, op.AbsA); + Operand[] srcB = GetHalfSrc(context, op.BimmH0, op.BimmH1); + + EmitHset2(context, op.Cmp, op.Bop, srcA, srcB, op.SrcPred, op.SrcPredInv, op.Dest, op.Bval); + } + + public static void Hset2C(EmitterContext context) + { + InstHset2C op = context.GetOp(); + + Operand[] srcA = GetHalfSrc(context, op.ASwizzle, op.SrcA, op.NegA, op.AbsA); + Operand[] srcB = GetHalfSrc(context, HalfSwizzle.F32, op.CbufSlot, op.CbufOffset, op.NegB, false); + + EmitHset2(context, op.Cmp, op.Bop, srcA, srcB, op.SrcPred, op.SrcPredInv, op.Dest, op.Bval); + } + + public static void Hsetp2R(EmitterContext context) + { + InstHsetp2R op = context.GetOp(); + + Operand[] srcA = GetHalfSrc(context, op.ASwizzle, op.SrcA, op.NegA, op.AbsA); + Operand[] srcB = GetHalfSrc(context, op.BSwizzle, op.SrcB, op.NegB, op.AbsB); + + EmitHsetp2(context, op.FComp2, op.Bop, srcA, srcB, op.SrcPred, op.SrcPredInv, op.DestPred, op.DestPredInv, op.HAnd); + } + + public static void Hsetp2I(EmitterContext context) + { + InstHsetp2I op = context.GetOp(); + + Operand[] srcA = GetHalfSrc(context, op.ASwizzle, op.SrcA, op.NegA, op.AbsA); + Operand[] srcB = GetHalfSrc(context, op.BimmH0, op.BimmH1); + + EmitHsetp2(context, op.FComp, op.Bop, srcA, srcB, op.SrcPred, op.SrcPredInv, op.DestPred, op.DestPredInv, op.HAnd); + } + + public static void Hsetp2C(EmitterContext context) + { + InstHsetp2C op = context.GetOp(); + + Operand[] srcA = GetHalfSrc(context, op.ASwizzle, op.SrcA, op.NegA, op.AbsA); + Operand[] srcB = GetHalfSrc(context, HalfSwizzle.F32, op.CbufSlot, op.CbufOffset, op.NegB, op.AbsB); + + EmitHsetp2(context, op.FComp, op.Bop, srcA, srcB, op.SrcPred, op.SrcPredInv, op.DestPred, op.DestPredInv, op.HAnd); + } + + private static void EmitFcmp(EmitterContext context, FComp cmpOp, Operand srcA, Operand srcB, Operand srcC, int rd) + { + Operand cmpRes = GetFPComparison(context, cmpOp, srcC, ConstF(0)); + + Operand res = context.ConditionalSelect(cmpRes, srcA, srcB); + + context.Copy(GetDest(rd), res); + } + + private static void EmitFset( + EmitterContext context, + FComp cmpOp, + BoolOp logicOp, + Operand srcA, + Operand srcB, + int srcPred, + bool srcPredInv, + int rd, + bool absoluteA, + bool absoluteB, + bool negateA, + bool negateB, + bool boolFloat, + bool writeCC, + bool isFP64 = false) + { + Instruction fpType = isFP64 ? Instruction.FP64 : Instruction.FP32; + + srcA = context.FPAbsNeg(srcA, absoluteA, negateA, fpType); + srcB = context.FPAbsNeg(srcB, absoluteB, negateB, fpType); + + Operand res = GetFPComparison(context, cmpOp, srcA, srcB, fpType); + Operand pred = GetPredicate(context, srcPred, srcPredInv); + + res = GetPredLogicalOp(context, logicOp, res, pred); + + Operand dest = GetDest(rd); + + if (boolFloat) + { + res = context.ConditionalSelect(res, ConstF(1), Const(0)); + + context.Copy(dest, res); + + SetFPZnFlags(context, res, writeCC); + } + else + { + context.Copy(dest, res); + + SetZnFlags(context, res, writeCC, extended: false); + } + } + + private static void EmitFsetp( + EmitterContext context, + FComp cmpOp, + BoolOp logicOp, + Operand srcA, + Operand srcB, + int srcPred, + bool srcPredInv, + int destPred, + int destPredInv, + bool absoluteA, + bool absoluteB, + bool negateA, + bool negateB, + bool writeCC, + bool isFP64 = false) + { + Instruction fpType = isFP64 ? Instruction.FP64 : Instruction.FP32; + + srcA = context.FPAbsNeg(srcA, absoluteA, negateA, fpType); + srcB = context.FPAbsNeg(srcB, absoluteB, negateB, fpType); + + Operand p0Res = GetFPComparison(context, cmpOp, srcA, srcB, fpType); + Operand p1Res = context.BitwiseNot(p0Res); + Operand pred = GetPredicate(context, srcPred, srcPredInv); + + p0Res = GetPredLogicalOp(context, logicOp, p0Res, pred); + p1Res = GetPredLogicalOp(context, logicOp, p1Res, pred); + + context.Copy(Register(destPred, RegisterType.Predicate), p0Res); + context.Copy(Register(destPredInv, RegisterType.Predicate), p1Res); + } + + private static void EmitHset2( + EmitterContext context, + FComp cmpOp, + BoolOp logicOp, + Operand[] srcA, + Operand[] srcB, + int srcPred, + bool srcPredInv, + int rd, + bool boolFloat) + { + Operand[] res = new Operand[2]; + + res[0] = GetFPComparison(context, cmpOp, srcA[0], srcB[0]); + res[1] = GetFPComparison(context, cmpOp, srcA[1], srcB[1]); + + Operand pred = GetPredicate(context, srcPred, srcPredInv); + + res[0] = GetPredLogicalOp(context, logicOp, res[0], pred); + res[1] = GetPredLogicalOp(context, logicOp, res[1], pred); + + if (boolFloat) + { + res[0] = context.ConditionalSelect(res[0], ConstF(1), Const(0)); + res[1] = context.ConditionalSelect(res[1], ConstF(1), Const(0)); + + context.Copy(GetDest(rd), context.PackHalf2x16(res[0], res[1])); + } + else + { + Operand low = context.BitwiseAnd(res[0], Const(0xffff)); + Operand high = context.ShiftLeft(res[1], Const(16)); + + Operand packed = context.BitwiseOr(low, high); + + context.Copy(GetDest(rd), packed); + } + } + + private static void EmitHsetp2( + EmitterContext context, + FComp cmpOp, + BoolOp logicOp, + Operand[] srcA, + Operand[] srcB, + int srcPred, + bool srcPredInv, + int destPred, + int destPredInv, + bool hAnd) + { + Operand p0Res = GetFPComparison(context, cmpOp, srcA[0], srcB[0]); + Operand p1Res = GetFPComparison(context, cmpOp, srcA[1], srcB[1]); + + if (hAnd) + { + p0Res = context.BitwiseAnd(p0Res, p1Res); + p1Res = context.BitwiseNot(p0Res); + } + + Operand pred = GetPredicate(context, srcPred, srcPredInv); + + p0Res = GetPredLogicalOp(context, logicOp, p0Res, pred); + p1Res = GetPredLogicalOp(context, logicOp, p1Res, pred); + + context.Copy(Register(destPred, RegisterType.Predicate), p0Res); + context.Copy(Register(destPredInv, RegisterType.Predicate), p1Res); + } + + private static Operand GetFPComparison(EmitterContext context, FComp cond, Operand srcA, Operand srcB, Instruction fpType = Instruction.FP32) + { + Operand res; + + if (cond == FComp.T) + { + res = Const(IrConsts.True); + } + else if (cond == FComp.F) + { + res = Const(IrConsts.False); + } + else if (cond is FComp.Nan or FComp.Num) + { + res = context.BitwiseOr(context.IsNan(srcA, fpType), context.IsNan(srcB, fpType)); + + if (cond == FComp.Num) + { + res = context.BitwiseNot(res); + } + } + else + { + Instruction inst = (cond & ~FComp.Nan) switch + { + FComp.Lt => Instruction.CompareLess, + FComp.Eq => Instruction.CompareEqual, + FComp.Le => Instruction.CompareLessOrEqual, + FComp.Gt => Instruction.CompareGreater, + FComp.Ne => Instruction.CompareNotEqual, + FComp.Ge => Instruction.CompareGreaterOrEqual, + _ => throw new ArgumentException($"Unexpected condition \"{cond}\"."), + }; + res = context.Add(inst | fpType, Local(), srcA, srcB); + + if ((cond & FComp.Nan) != 0) + { + res = context.BitwiseOr(res, context.IsNan(srcA, fpType)); + res = context.BitwiseOr(res, context.IsNan(srcB, fpType)); + } + } + + return res; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitFloatMinMax.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitFloatMinMax.cs new file mode 100644 index 0000000..873a620 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitFloatMinMax.cs @@ -0,0 +1,106 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; + +using static Ryujinx.Graphics.Shader.Instructions.InstEmitAluHelper; +using static Ryujinx.Graphics.Shader.Instructions.InstEmitHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void DmnmxR(EmitterContext context) + { + InstDmnmxR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA, isFP64: true); + Operand srcB = GetSrcReg(context, op.SrcB, isFP64: true); + Operand srcPred = GetPredicate(context, op.SrcPred, op.SrcPredInv); + + EmitFmnmx(context, srcA, srcB, srcPred, op.Dest, op.AbsA, op.AbsB, op.NegA, op.NegB, op.WriteCC, isFP64: true); + } + + public static void DmnmxI(EmitterContext context) + { + InstDmnmxI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA, isFP64: true); + Operand srcB = GetSrcImm(context, Imm20ToFloat(op.Imm20), isFP64: true); + Operand srcPred = GetPredicate(context, op.SrcPred, op.SrcPredInv); + + EmitFmnmx(context, srcA, srcB, srcPred, op.Dest, op.AbsA, op.AbsB, op.NegA, op.NegB, op.WriteCC, isFP64: true); + } + + public static void DmnmxC(EmitterContext context) + { + InstDmnmxC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA, isFP64: true); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset, isFP64: true); + Operand srcPred = GetPredicate(context, op.SrcPred, op.SrcPredInv); + + EmitFmnmx(context, srcA, srcB, srcPred, op.Dest, op.AbsA, op.AbsB, op.NegA, op.NegB, op.WriteCC, isFP64: true); + } + + public static void FmnmxR(EmitterContext context) + { + InstFmnmxR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + Operand srcPred = GetPredicate(context, op.SrcPred, op.SrcPredInv); + + EmitFmnmx(context, srcA, srcB, srcPred, op.Dest, op.AbsA, op.AbsB, op.NegA, op.NegB, op.WriteCC); + } + + public static void FmnmxI(EmitterContext context) + { + InstFmnmxI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToFloat(op.Imm20)); + Operand srcPred = GetPredicate(context, op.SrcPred, op.SrcPredInv); + + EmitFmnmx(context, srcA, srcB, srcPred, op.Dest, op.AbsA, op.AbsB, op.NegA, op.NegB, op.WriteCC); + } + + public static void FmnmxC(EmitterContext context) + { + InstFmnmxC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + Operand srcPred = GetPredicate(context, op.SrcPred, op.SrcPredInv); + + EmitFmnmx(context, srcA, srcB, srcPred, op.Dest, op.AbsA, op.AbsB, op.NegA, op.NegB, op.WriteCC); + } + + private static void EmitFmnmx( + EmitterContext context, + Operand srcA, + Operand srcB, + Operand srcPred, + int rd, + bool absoluteA, + bool absoluteB, + bool negateA, + bool negateB, + bool writeCC, + bool isFP64 = false) + { + Instruction fpType = isFP64 ? Instruction.FP64 : Instruction.FP32; + + srcA = context.FPAbsNeg(srcA, absoluteA, negateA, fpType); + srcB = context.FPAbsNeg(srcB, absoluteB, negateB, fpType); + + Operand resMin = context.FPMinimum(srcA, srcB, fpType); + Operand resMax = context.FPMaximum(srcA, srcB, fpType); + + Operand res = context.ConditionalSelect(srcPred, resMin, resMax); + + SetDest(context, res, rd, isFP64); + + SetFPZnFlags(context, res, writeCC, fpType); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitFlowControl.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitFlowControl.cs new file mode 100644 index 0000000..a28e44d --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitFlowControl.cs @@ -0,0 +1,328 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using System.Collections.Generic; +using System.Linq; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void Bra(EmitterContext context) + { + context.GetOp(); + + EmitBranch(context, context.CurrBlock.Successors[^1].Address); + } + + public static void Brk(EmitterContext context) + { + context.GetOp(); + + EmitBrkContSync(context); + } + + public static void Brx(EmitterContext context) + { + InstBrx op = context.GetOp(); + InstOp currOp = context.CurrOp; + int startIndex = context.CurrBlock.HasNext() ? 1 : 0; + + if (context.CurrBlock.Successors.Count <= startIndex) + { + context.TranslatorContext.GpuAccessor.Log($"Failed to find targets for BRX instruction at 0x{currOp.Address:X}."); + return; + } + + int offset = (int)currOp.GetAbsoluteAddress(); + + Operand address = context.IAdd(Register(op.SrcA, RegisterType.Gpr), Const(offset)); + + IEnumerable targets = context.CurrBlock.Successors.Skip(startIndex); + + bool allTargetsSinglePred = true; + int total = context.CurrBlock.Successors.Count - startIndex; + int count = 0; + + foreach (Block target in targets.OrderBy(x => x.Address)) + { + if (++count < total && (target.Predecessors.Count > 1 || target.Address <= context.CurrBlock.Address)) + { + allTargetsSinglePred = false; + break; + } + } + + if (allTargetsSinglePred) + { + // Chain blocks, each target block will check if the BRX target address + // matches its own address, if not, it jumps to the next target which will do the same check, + // until it reaches the last possible target, which executed unconditionally. + // We can only do this if the BRX block is the only predecessor of all target blocks. + // Additionally, this is not supported for blocks located before the current block, + // since it will be too late to insert a label, but this is something that can be improved + // in the future if necessary. + + IOrderedEnumerable sortedTargets = targets.OrderBy(x => x.Address); + + Block currentTarget = null; + ulong firstTargetAddress = 0; + + foreach (Block nextTarget in sortedTargets) + { + if (currentTarget != null) + { + if (currentTarget.Address != nextTarget.Address) + { + context.SetBrxTarget(currentTarget.Address, address, (int)currentTarget.Address, nextTarget.Address); + } + } + else + { + firstTargetAddress = nextTarget.Address; + } + + currentTarget = nextTarget; + } + + context.Branch(context.GetLabel(firstTargetAddress)); + } + else + { + // Emit the branches sequentially. + // This generates slightly worse code, but should work for all cases. + + IOrderedEnumerable sortedTargets = targets.OrderByDescending(x => x.Address); + ulong lastTargetAddress = ulong.MaxValue; + + count = 0; + + foreach (Block target in sortedTargets) + { + Operand label = context.GetLabel(target.Address); + + if (++count < total) + { + if (target.Address != lastTargetAddress) + { + context.BranchIfTrue(label, context.ICompareEqual(address, Const((int)target.Address))); + } + + lastTargetAddress = target.Address; + } + else + { + context.Branch(label); + } + } + } + } + + public static void Cal(EmitterContext context) + { + context.GetOp(); + + DecodedFunction function = context.Program.GetFunctionByAddress(context.CurrOp.GetAbsoluteAddress()); + + if (function.IsCompilerGenerated) + { + switch (function.Type) + { + case FunctionType.BuiltInFSIBegin: + context.FSIBegin(); + break; + case FunctionType.BuiltInFSIEnd: + context.FSIEnd(); + break; + } + } + else + { + context.Call(function.Id, false); + } + } + + public static void Cont(EmitterContext context) + { + context.GetOp(); + + EmitBrkContSync(context); + } + + public static void Exit(EmitterContext context) + { + InstExit op = context.GetOp(); + + if (context.IsNonMain) + { + context.TranslatorContext.GpuAccessor.Log("Invalid exit on non-main function."); + return; + } + + if (op.Ccc == Ccc.T) + { + if (context.PrepareForReturn()) + { + context.Return(); + } + } + else + { + Operand cond = GetCondition(context, op.Ccc, IrConsts.False); + + // If the condition is always false, we don't need to do anything. + if (cond.Type != OperandType.Constant || cond.Value != IrConsts.False) + { + Operand lblSkip = Label(); + context.BranchIfFalse(lblSkip, cond); + + if (context.PrepareForReturn()) + { + context.Return(); + } + + context.MarkLabel(lblSkip); + } + } + } + + public static void Kil(EmitterContext context) + { + context.GetOp(); + + context.Discard(); + } + + public static void Pbk(EmitterContext context) + { + context.GetOp(); + + EmitPbkPcntSsy(context); + } + + public static void Pcnt(EmitterContext context) + { + context.GetOp(); + + EmitPbkPcntSsy(context); + } + + public static void Ret(EmitterContext context) + { + context.GetOp(); + + if (context.IsNonMain) + { + context.Return(); + } + else + { + context.TranslatorContext.GpuAccessor.Log("Invalid return on main function."); + } + } + + public static void Ssy(EmitterContext context) + { + context.GetOp(); + + EmitPbkPcntSsy(context); + } + + public static void Sync(EmitterContext context) + { + context.GetOp(); + + EmitBrkContSync(context); + } + + private static void EmitPbkPcntSsy(EmitterContext context) + { + Dictionary consumers = context.CurrBlock.PushOpCodes.First(x => x.Op.Address == context.CurrOp.Address).Consumers; + + foreach (KeyValuePair kv in consumers) + { + Block consumerBlock = kv.Key; + Operand local = kv.Value; + + int id = consumerBlock.SyncTargets[context.CurrOp.Address].PushOpId; + + context.Copy(local, Const(id)); + } + } + + private static void EmitBrkContSync(EmitterContext context) + { + Dictionary targets = context.CurrBlock.SyncTargets; + + if (targets.Count == 1) + { + // If we have only one target, then the SSY/PBK is basically + // a branch, we can produce better codegen for this case. + EmitBranch(context, targets.Values.First().PushOpInfo.Op.GetAbsoluteAddress()); + } + else + { + // TODO: Support CC here as well (condition). + foreach (SyncTarget target in targets.Values) + { + PushOpInfo pushOpInfo = target.PushOpInfo; + + Operand label = context.GetLabel(pushOpInfo.Op.GetAbsoluteAddress()); + Operand local = pushOpInfo.Consumers[context.CurrBlock]; + + context.BranchIfTrue(label, context.ICompareEqual(local, Const(target.PushOpId))); + } + } + } + + private static void EmitBranch(EmitterContext context, ulong address) + { + InstOp op = context.CurrOp; + InstConditional opCond = new(op.RawOpCode); + + // If we're branching to the next instruction, then the branch + // is useless and we can ignore it. + if (address == op.Address + 8) + { + return; + } + + Operand label = context.GetLabel(address); + + Operand pred = Register(opCond.Pred, RegisterType.Predicate); + + if (opCond.Ccc != Ccc.T) + { + Operand cond = GetCondition(context, opCond.Ccc); + + if (opCond.Pred == RegisterConsts.PredicateTrueIndex) + { + pred = cond; + } + else if (opCond.PredInv) + { + pred = context.BitwiseAnd(context.BitwiseNot(pred), cond); + } + else + { + pred = context.BitwiseAnd(pred, cond); + } + + context.BranchIfTrue(label, pred); + } + else if (opCond.Pred == RegisterConsts.PredicateTrueIndex) + { + context.Branch(label); + } + else if (opCond.PredInv) + { + context.BranchIfFalse(label, pred); + } + else + { + context.BranchIfTrue(label, pred); + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitHelper.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitHelper.cs new file mode 100644 index 0000000..774c9d1 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitHelper.cs @@ -0,0 +1,260 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using System; +using System.Runtime.CompilerServices; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static class InstEmitHelper + { + public static Operand GetZF() + { + return Register(0, RegisterType.Flag); + } + + public static Operand GetNF() + { + return Register(1, RegisterType.Flag); + } + + public static Operand GetCF() + { + return Register(2, RegisterType.Flag); + } + + public static Operand GetVF() + { + return Register(3, RegisterType.Flag); + } + + public static Operand GetDest(int rd) + { + return Register(rd, RegisterType.Gpr); + } + + public static Operand GetDest2(int rd) + { + return Register(rd | 1, RegisterType.Gpr); + } + + public static Operand GetSrcCbuf(EmitterContext context, int cbufSlot, int cbufOffset, bool isFP64 = false) + { + if (isFP64) + { + return context.PackDouble2x32( + Cbuf(cbufSlot, cbufOffset), + Cbuf(cbufSlot, cbufOffset + 1)); + } + else + { + return Cbuf(cbufSlot, cbufOffset); + } + } + + public static Operand GetSrcImm(EmitterContext context, int imm, bool isFP64 = false) + { + if (isFP64) + { + return context.PackDouble2x32(Const(0), Const(imm)); + } + else + { + return Const(imm); + } + } + + public static Operand GetSrcReg(EmitterContext context, int reg, bool isFP64 = false) + { + if (isFP64) + { + return context.PackDouble2x32(Register(reg, RegisterType.Gpr), Register(reg | 1, RegisterType.Gpr)); + } + else + { + return Register(reg, RegisterType.Gpr); + } + } + + public static Operand[] GetHalfSrc( + EmitterContext context, + HalfSwizzle swizzle, + int ra, + bool negate, + bool absolute) + { + Operand[] operands = GetHalfUnpacked(context, GetSrcReg(context, ra), swizzle); + + return FPAbsNeg(context, operands, absolute, negate); + } + + public static Operand[] GetHalfSrc( + EmitterContext context, + HalfSwizzle swizzle, + int cbufSlot, + int cbufOffset, + bool negate, + bool absolute) + { + Operand[] operands = GetHalfUnpacked(context, GetSrcCbuf(context, cbufSlot, cbufOffset), swizzle); + + return FPAbsNeg(context, operands, absolute, negate); + } + + public static Operand[] GetHalfSrc(EmitterContext context, int immH0, int immH1) + { + ushort low = (ushort)(immH0 << 6); + ushort high = (ushort)(immH1 << 6); + + return + [ + ConstF((float)Unsafe.As(ref low)), + ConstF((float)Unsafe.As(ref high)) + ]; + } + + public static Operand[] GetHalfSrc(EmitterContext context, int imm32) + { + ushort low = (ushort)imm32; + ushort high = (ushort)(imm32 >> 16); + + return + [ + ConstF((float)Unsafe.As(ref low)), + ConstF((float)Unsafe.As(ref high)) + ]; + } + + public static Operand[] FPAbsNeg(EmitterContext context, Operand[] operands, bool abs, bool neg) + { + for (int index = 0; index < operands.Length; index++) + { + operands[index] = context.FPAbsNeg(operands[index], abs, neg); + } + + return operands; + } + + public static Operand[] GetHalfUnpacked(EmitterContext context, Operand src, HalfSwizzle swizzle) + { + return swizzle switch + { + HalfSwizzle.F16 => + [ + context.UnpackHalf2x16Low (src), + context.UnpackHalf2x16High(src) + ], + HalfSwizzle.F32 => [src, src], + HalfSwizzle.H0H0 => + [ + context.UnpackHalf2x16Low(src), + context.UnpackHalf2x16Low(src) + ], + HalfSwizzle.H1H1 => + [ + context.UnpackHalf2x16High(src), + context.UnpackHalf2x16High(src) + ], + _ => throw new ArgumentException($"Invalid swizzle \"{swizzle}\"."), + }; + } + + public static Operand GetHalfPacked(EmitterContext context, OFmt swizzle, Operand[] results, int rd) + { + switch (swizzle) + { + case OFmt.F16: + return context.PackHalf2x16(results[0], results[1]); + + case OFmt.F32: + return results[0]; + + case OFmt.MrgH0: + { + Operand h1 = GetHalfDest(context, rd, isHigh: true); + + return context.PackHalf2x16(results[0], h1); + } + + case OFmt.MrgH1: + { + Operand h0 = GetHalfDest(context, rd, isHigh: false); + + return context.PackHalf2x16(h0, results[1]); + } + } + + throw new ArgumentException($"Invalid swizzle \"{swizzle}\"."); + } + + public static Operand GetHalfDest(EmitterContext context, int rd, bool isHigh) + { + if (isHigh) + { + return context.UnpackHalf2x16High(GetDest(rd)); + } + else + { + return context.UnpackHalf2x16Low(GetDest(rd)); + } + } + + public static Operand GetPredicate(EmitterContext context, int pred, bool not) + { + Operand local = Register(pred, RegisterType.Predicate); + + if (not) + { + local = context.BitwiseNot(local); + } + + return local; + } + + public static void SetDest(EmitterContext context, Operand value, int rd, bool isFP64) + { + if (isFP64) + { + context.Copy(GetDest(rd), context.UnpackDouble2x32Low(value)); + context.Copy(GetDest2(rd), context.UnpackDouble2x32High(value)); + } + else + { + context.Copy(GetDest(rd), value); + } + } + + public static int Imm16ToSInt(int imm16) + { + return (short)imm16; + } + + public static int Imm20ToFloat(int imm20) + { + return imm20 << 12; + } + + public static int Imm20ToSInt(int imm20) + { + return (imm20 << 12) >> 12; + } + + public static int Imm24ToSInt(int imm24) + { + return (imm24 << 8) >> 8; + } + + public static Operand SignExtendTo32(EmitterContext context, Operand src, int srcBits) + { + return context.BitfieldExtractS32(src, Const(0), Const(srcBits)); + } + + public static Operand ZeroExtendTo32(EmitterContext context, Operand src, int srcBits) + { + int mask = (int)(uint.MaxValue >> (32 - srcBits)); + + return context.BitwiseAnd(src, Const(mask)); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitIntegerArithmetic.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitIntegerArithmetic.cs new file mode 100644 index 0000000..9ac5a25 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitIntegerArithmetic.cs @@ -0,0 +1,701 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; + +using static Ryujinx.Graphics.Shader.Instructions.InstEmitAluHelper; +using static Ryujinx.Graphics.Shader.Instructions.InstEmitHelper; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void IaddR(EmitterContext context) + { + InstIaddR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + + EmitIadd(context, srcA, srcB, op.Dest, op.AvgMode, op.X, op.WriteCC); + } + + public static void IaddI(EmitterContext context) + { + InstIaddI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToSInt(op.Imm20)); + + EmitIadd(context, srcA, srcB, op.Dest, op.AvgMode, op.X, op.WriteCC); + } + + public static void IaddC(EmitterContext context) + { + InstIaddC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitIadd(context, srcA, srcB, op.Dest, op.AvgMode, op.X, op.WriteCC); + } + + public static void Iadd32i(EmitterContext context) + { + InstIadd32i op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, op.Imm32); + + EmitIadd(context, srcA, srcB, op.Dest, op.AvgMode, op.X, op.WriteCC); + } + + public static void Iadd3R(EmitterContext context) + { + InstIadd3R op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitIadd3(context, op.Lrs, srcA, srcB, srcC, op.Apart, op.Bpart, op.Cpart, op.Dest, op.NegA, op.NegB, op.NegC); + } + + public static void Iadd3I(EmitterContext context) + { + InstIadd3I op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToSInt(op.Imm20)); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitIadd3(context, Lrs.None, srcA, srcB, srcC, HalfSelect.B32, HalfSelect.B32, HalfSelect.B32, op.Dest, op.NegA, op.NegB, op.NegC); + } + + public static void Iadd3C(EmitterContext context) + { + InstIadd3C op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitIadd3(context, Lrs.None, srcA, srcB, srcC, HalfSelect.B32, HalfSelect.B32, HalfSelect.B32, op.Dest, op.NegA, op.NegB, op.NegC); + } + + public static void ImadR(EmitterContext context) + { + InstImadR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitImad(context, srcA, srcB, srcC, op.Dest, op.AvgMode, op.ASigned, op.BSigned, op.Hilo); + } + + public static void ImadI(EmitterContext context) + { + InstImadI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToSInt(op.Imm20)); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitImad(context, srcA, srcB, srcC, op.Dest, op.AvgMode, op.ASigned, op.BSigned, op.Hilo); + } + + public static void ImadC(EmitterContext context) + { + InstImadC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitImad(context, srcA, srcB, srcC, op.Dest, op.AvgMode, op.ASigned, op.BSigned, op.Hilo); + } + + public static void ImadRc(EmitterContext context) + { + InstImadRc op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcC); + Operand srcC = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitImad(context, srcA, srcB, srcC, op.Dest, op.AvgMode, op.ASigned, op.BSigned, op.Hilo); + } + + public static void Imad32i(EmitterContext context) + { + InstImad32i op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, op.Imm32); + Operand srcC = GetSrcReg(context, op.Dest); + + EmitImad(context, srcA, srcB, srcC, op.Dest, op.AvgMode, op.ASigned, op.BSigned, op.Hilo); + } + + public static void ImulR(EmitterContext context) + { + InstImulR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + + EmitImad(context, srcA, srcB, Const(0), op.Dest, AvgMode.NoNeg, op.ASigned, op.BSigned, op.Hilo); + } + + public static void ImulI(EmitterContext context) + { + InstImulI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToSInt(op.Imm20)); + + EmitImad(context, srcA, srcB, Const(0), op.Dest, AvgMode.NoNeg, op.ASigned, op.BSigned, op.Hilo); + } + + public static void ImulC(EmitterContext context) + { + InstImulC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitImad(context, srcA, srcB, Const(0), op.Dest, AvgMode.NoNeg, op.ASigned, op.BSigned, op.Hilo); + } + + public static void Imul32i(EmitterContext context) + { + InstImul32i op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, op.Imm32); + + EmitImad(context, srcA, srcB, Const(0), op.Dest, AvgMode.NoNeg, op.ASigned, op.BSigned, op.Hilo); + } + + public static void IscaddR(EmitterContext context) + { + InstIscaddR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + + EmitIscadd(context, srcA, srcB, op.Dest, op.Imm5, op.AvgMode, op.WriteCC); + } + + public static void IscaddI(EmitterContext context) + { + InstIscaddI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToSInt(op.Imm20)); + + EmitIscadd(context, srcA, srcB, op.Dest, op.Imm5, op.AvgMode, op.WriteCC); + } + + public static void IscaddC(EmitterContext context) + { + InstIscaddC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitIscadd(context, srcA, srcB, op.Dest, op.Imm5, op.AvgMode, op.WriteCC); + } + + public static void Iscadd32i(EmitterContext context) + { + InstIscadd32i op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, op.Imm32); + + EmitIscadd(context, srcA, srcB, op.Dest, op.Imm5, AvgMode.NoNeg, op.WriteCC); + } + + public static void LeaR(EmitterContext context) + { + InstLeaR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + + EmitLea(context, srcA, srcB, op.Dest, op.NegA, op.ImmU5); + } + + public static void LeaI(EmitterContext context) + { + InstLeaI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToSInt(op.Imm20)); + + EmitLea(context, srcA, srcB, op.Dest, op.NegA, op.ImmU5); + } + + public static void LeaC(EmitterContext context) + { + InstLeaC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitLea(context, srcA, srcB, op.Dest, op.NegA, op.ImmU5); + } + + public static void LeaHiR(EmitterContext context) + { + InstLeaHiR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitLeaHi(context, srcA, srcB, srcC, op.Dest, op.NegA, op.ImmU5); + } + + public static void LeaHiC(EmitterContext context) + { + InstLeaHiC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitLeaHi(context, srcA, srcB, srcC, op.Dest, op.NegA, op.ImmU5); + } + + public static void XmadR(EmitterContext context) + { + InstXmadR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitXmad(context, op.XmadCop, srcA, srcB, srcC, op.Dest, op.ASigned, op.BSigned, op.HiloA, op.HiloB, op.Psl, op.Mrg, op.X, op.WriteCC); + } + + public static void XmadI(EmitterContext context) + { + InstXmadI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, op.Imm16); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitXmad(context, op.XmadCop, srcA, srcB, srcC, op.Dest, op.ASigned, op.BSigned, op.HiloA, false, op.Psl, op.Mrg, op.X, op.WriteCC); + } + + public static void XmadC(EmitterContext context) + { + InstXmadC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitXmad(context, op.XmadCop, srcA, srcB, srcC, op.Dest, op.ASigned, op.BSigned, op.HiloA, op.HiloB, op.Psl, op.Mrg, op.X, op.WriteCC); + } + + public static void XmadRc(EmitterContext context) + { + InstXmadRc op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcC); + Operand srcC = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitXmad(context, op.XmadCop, srcA, srcB, srcC, op.Dest, op.ASigned, op.BSigned, op.HiloA, op.HiloB, false, false, op.X, op.WriteCC); + } + + private static void EmitIadd( + EmitterContext context, + Operand srcA, + Operand srcB, + int rd, + AvgMode avgMode, + bool extended, + bool writeCC) + { + srcA = context.INegate(srcA, avgMode == AvgMode.NegA); + srcB = context.INegate(srcB, avgMode == AvgMode.NegB); + + Operand res = context.IAdd(srcA, srcB); + + if (extended) + { + res = context.IAdd(res, context.BitwiseAnd(GetCF(), Const(1))); + } + + SetIaddFlags(context, res, srcA, srcB, writeCC, extended); + + // TODO: SAT. + + context.Copy(GetDest(rd), res); + } + + private static void EmitIadd3( + EmitterContext context, + Lrs mode, + Operand srcA, + Operand srcB, + Operand srcC, + HalfSelect partA, + HalfSelect partB, + HalfSelect partC, + int rd, + bool negateA, + bool negateB, + bool negateC) + { + Operand Extend(Operand src, HalfSelect part) + { + if (part == HalfSelect.B32) + { + return src; + } + + if (part == HalfSelect.H0) + { + return context.BitwiseAnd(src, Const(0xffff)); + } + else if (part == HalfSelect.H1) + { + return context.ShiftRightU32(src, Const(16)); + } + else + { + context.TranslatorContext.GpuAccessor.Log($"Iadd3 has invalid component selection {part}."); + } + + return src; + } + + srcA = context.INegate(Extend(srcA, partA), negateA); + srcB = context.INegate(Extend(srcB, partB), negateB); + srcC = context.INegate(Extend(srcC, partC), negateC); + + Operand res = context.IAdd(srcA, srcB); + + if (mode != Lrs.None) + { + if (mode == Lrs.LeftShift) + { + res = context.ShiftLeft(res, Const(16)); + } + else if (mode == Lrs.RightShift) + { + res = context.ShiftRightU32(res, Const(16)); + } + else + { + // TODO: Warning. + } + } + + res = context.IAdd(res, srcC); + + context.Copy(GetDest(rd), res); + + // TODO: CC, X, corner cases. + } + + private static void EmitImad( + EmitterContext context, + Operand srcA, + Operand srcB, + Operand srcC, + int rd, + AvgMode avgMode, + bool signedA, + bool signedB, + bool high) + { + srcB = context.INegate(srcB, avgMode == AvgMode.NegA); + srcC = context.INegate(srcC, avgMode == AvgMode.NegB); + + Operand res; + + if (high) + { + if (signedA && signedB) + { + res = context.MultiplyHighS32(srcA, srcB); + } + else + { + res = context.MultiplyHighU32(srcA, srcB); + + if (signedA) + { + res = context.IAdd(res, context.IMultiply(srcB, context.ShiftRightS32(srcA, Const(31)))); + } + else if (signedB) + { + res = context.IAdd(res, context.IMultiply(srcA, context.ShiftRightS32(srcB, Const(31)))); + } + } + } + else + { + res = context.IMultiply(srcA, srcB); + } + + if (srcC.Type != OperandType.Constant || srcC.Value != 0) + { + res = context.IAdd(res, srcC); + } + + // TODO: CC, X, SAT, and more? + + context.Copy(GetDest(rd), res); + } + + private static void EmitIscadd( + EmitterContext context, + Operand srcA, + Operand srcB, + int rd, + int shift, + AvgMode avgMode, + bool writeCC) + { + srcA = context.ShiftLeft(srcA, Const(shift)); + + srcA = context.INegate(srcA, avgMode == AvgMode.NegA); + srcB = context.INegate(srcB, avgMode == AvgMode.NegB); + + Operand res = context.IAdd(srcA, srcB); + + SetIaddFlags(context, res, srcA, srcB, writeCC, false); + + context.Copy(GetDest(rd), res); + } + + public static void EmitLea(EmitterContext context, Operand srcA, Operand srcB, int rd, bool negateA, int shift) + { + srcA = context.ShiftLeft(srcA, Const(shift)); + srcA = context.INegate(srcA, negateA); + + Operand res = context.IAdd(srcA, srcB); + + context.Copy(GetDest(rd), res); + + // TODO: CC, X. + } + + private static void EmitLeaHi( + EmitterContext context, + Operand srcA, + Operand srcB, + Operand srcC, + int rd, + bool negateA, + int shift) + { + Operand aLow = context.ShiftLeft(srcA, Const(shift)); + Operand aHigh = shift == 0 ? Const(0) : context.ShiftRightU32(srcA, Const(32 - shift)); + aHigh = context.BitwiseOr(aHigh, context.ShiftLeft(srcC, Const(shift))); + + if (negateA) + { + // Perform 64-bit negation by doing bitwise not of the value, + // then adding 1 and carrying over from low to high. + aLow = context.BitwiseNot(aLow); + aHigh = context.BitwiseNot(aHigh); + +#pragma warning disable IDE0059 // Remove unnecessary value assignment + aLow = AddWithCarry(context, aLow, Const(1), out Operand aLowCOut); +#pragma warning restore IDE0059 + aHigh = context.IAdd(aHigh, aLowCOut); + } + + Operand res = context.IAdd(aHigh, srcB); + + context.Copy(GetDest(rd), res); + + // TODO: CC, X. + } + + public static void EmitXmad( + EmitterContext context, + XmadCop2 mode, + Operand srcA, + Operand srcB, + Operand srcC, + int rd, + bool signedA, + bool signedB, + bool highA, + bool highB, + bool productShiftLeft, + bool merge, + bool extended, + bool writeCC) + { + XmadCop modeConv; + switch (mode) + { + case XmadCop2.Cfull: + modeConv = XmadCop.Cfull; + break; + case XmadCop2.Clo: + modeConv = XmadCop.Clo; + break; + case XmadCop2.Chi: + modeConv = XmadCop.Chi; + break; + case XmadCop2.Csfu: + modeConv = XmadCop.Csfu; + break; + default: + context.TranslatorContext.GpuAccessor.Log($"Invalid XMAD mode \"{mode}\"."); + return; + } + + EmitXmad(context, modeConv, srcA, srcB, srcC, rd, signedA, signedB, highA, highB, productShiftLeft, merge, extended, writeCC); + } + + public static void EmitXmad( + EmitterContext context, + XmadCop mode, + Operand srcA, + Operand srcB, + Operand srcC, + int rd, + bool signedA, + bool signedB, + bool highA, + bool highB, + bool productShiftLeft, + bool merge, + bool extended, + bool writeCC) + { + Operand srcBUnmodified = srcB; + + Operand Extend16To32(Operand src, bool high, bool signed) + { + if (signed && high) + { + return context.ShiftRightS32(src, Const(16)); + } + else if (signed) + { + return context.BitfieldExtractS32(src, Const(0), Const(16)); + } + else if (high) + { + return context.ShiftRightU32(src, Const(16)); + } + else + { + return context.BitwiseAnd(src, Const(0xffff)); + } + } + + srcA = Extend16To32(srcA, highA, signedA); + srcB = Extend16To32(srcB, highB, signedB); + + Operand res = context.IMultiply(srcA, srcB); + + if (productShiftLeft) + { + res = context.ShiftLeft(res, Const(16)); + } + + switch (mode) + { + case XmadCop.Cfull: + break; + + case XmadCop.Clo: + srcC = Extend16To32(srcC, high: false, signed: false); + break; + case XmadCop.Chi: + srcC = Extend16To32(srcC, high: true, signed: false); + break; + + case XmadCop.Cbcc: + srcC = context.IAdd(srcC, context.ShiftLeft(srcBUnmodified, Const(16))); + break; + + case XmadCop.Csfu: + Operand signAdjustA = context.ShiftLeft(context.ShiftRightU32(srcA, Const(31)), Const(16)); + Operand signAdjustB = context.ShiftLeft(context.ShiftRightU32(srcB, Const(31)), Const(16)); + + srcC = context.ISubtract(srcC, context.IAdd(signAdjustA, signAdjustB)); + break; + + default: + context.TranslatorContext.GpuAccessor.Log($"Invalid XMAD mode \"{mode}\"."); + return; + } + + Operand product = res; + + if (extended) + { + // Add with carry. + res = context.IAdd(res, context.BitwiseAnd(GetCF(), Const(1))); + } + else + { + // Add (no carry in). + res = context.IAdd(res, srcC); + } + + SetIaddFlags(context, res, product, srcC, writeCC, extended); + + if (merge) + { + res = context.BitwiseAnd(res, Const(0xffff)); + res = context.BitwiseOr(res, context.ShiftLeft(srcBUnmodified, Const(16))); + } + + context.Copy(GetDest(rd), res); + } + + private static void SetIaddFlags(EmitterContext context, Operand res, Operand srcA, Operand srcB, bool setCC, bool extended) + { + if (!setCC) + { + return; + } + + if (extended) + { + // C = (d == a && CIn) || d < a + Operand tempC0 = context.ICompareEqual(res, srcA); + Operand tempC1 = context.ICompareLessUnsigned(res, srcA); + + tempC0 = context.BitwiseAnd(tempC0, GetCF()); + + context.Copy(GetCF(), context.BitwiseOr(tempC0, tempC1)); + } + else + { + // C = d < a + context.Copy(GetCF(), context.ICompareLessUnsigned(res, srcA)); + } + + // V = (d ^ a) & ~(a ^ b) < 0 + Operand tempV0 = context.BitwiseExclusiveOr(res, srcA); + Operand tempV1 = context.BitwiseExclusiveOr(srcA, srcB); + + tempV1 = context.BitwiseNot(tempV1); + + Operand tempV = context.BitwiseAnd(tempV0, tempV1); + + context.Copy(GetVF(), context.ICompareLess(tempV, Const(0))); + + SetZnFlags(context, res, setCC: true, extended: extended); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitIntegerComparison.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitIntegerComparison.cs new file mode 100644 index 0000000..87dbd99 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitIntegerComparison.cs @@ -0,0 +1,319 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using System; +using static Ryujinx.Graphics.Shader.Instructions.InstEmitAluHelper; +using static Ryujinx.Graphics.Shader.Instructions.InstEmitHelper; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void IcmpR(EmitterContext context) + { + InstIcmpR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitIcmp(context, op.IComp, srcA, srcB, srcC, op.Dest, op.Signed); + } + + public static void IcmpI(EmitterContext context) + { + InstIcmpI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToSInt(op.Imm20)); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitIcmp(context, op.IComp, srcA, srcB, srcC, op.Dest, op.Signed); + } + + public static void IcmpC(EmitterContext context) + { + InstIcmpC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitIcmp(context, op.IComp, srcA, srcB, srcC, op.Dest, op.Signed); + } + + public static void IcmpRc(EmitterContext context) + { + InstIcmpRc op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcC); + Operand srcC = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitIcmp(context, op.IComp, srcA, srcB, srcC, op.Dest, op.Signed); + } + + public static void IsetR(EmitterContext context) + { + InstIsetR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + + EmitIset(context, op.IComp, op.Bop, srcA, srcB, op.SrcPred, op.SrcPredInv, op.Dest, op.BVal, op.Signed, op.X, op.WriteCC); + } + + public static void IsetI(EmitterContext context) + { + InstIsetI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToSInt(op.Imm20)); + + EmitIset(context, op.IComp, op.Bop, srcA, srcB, op.SrcPred, op.SrcPredInv, op.Dest, op.BVal, op.Signed, op.X, op.WriteCC); + } + + public static void IsetC(EmitterContext context) + { + InstIsetC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitIset(context, op.IComp, op.Bop, srcA, srcB, op.SrcPred, op.SrcPredInv, op.Dest, op.BVal, op.Signed, op.X, op.WriteCC); + } + + public static void IsetpR(EmitterContext context) + { + InstIsetpR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + + EmitIsetp(context, op.IComp, op.Bop, srcA, srcB, op.SrcPred, op.SrcPredInv, op.DestPred, op.DestPredInv, op.Signed, op.X); + } + + public static void IsetpI(EmitterContext context) + { + InstIsetpI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToSInt(op.Imm20)); + + EmitIsetp(context, op.IComp, op.Bop, srcA, srcB, op.SrcPred, op.SrcPredInv, op.DestPred, op.DestPredInv, op.Signed, op.X); + } + + public static void IsetpC(EmitterContext context) + { + InstIsetpC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitIsetp(context, op.IComp, op.Bop, srcA, srcB, op.SrcPred, op.SrcPredInv, op.DestPred, op.DestPredInv, op.Signed, op.X); + } + + private static void EmitIcmp( + EmitterContext context, + IComp cmpOp, + Operand srcA, + Operand srcB, + Operand srcC, + int rd, + bool isSigned) + { + Operand cmpRes = GetIntComparison(context, cmpOp, srcC, Const(0), isSigned); + + Operand res = context.ConditionalSelect(cmpRes, srcA, srcB); + + context.Copy(GetDest(rd), res); + } + + private static void EmitIset( + EmitterContext context, + IComp cmpOp, + BoolOp logicOp, + Operand srcA, + Operand srcB, + int srcPred, + bool srcPredInv, + int rd, + bool boolFloat, + bool isSigned, + bool extended, + bool writeCC) + { + Operand res = GetIntComparison(context, cmpOp, srcA, srcB, isSigned, extended); + Operand pred = GetPredicate(context, srcPred, srcPredInv); + + res = GetPredLogicalOp(context, logicOp, res, pred); + + Operand dest = GetDest(rd); + + if (boolFloat) + { + res = context.ConditionalSelect(res, ConstF(1), Const(0)); + + context.Copy(dest, res); + + SetFPZnFlags(context, res, writeCC); + } + else + { + context.Copy(dest, res); + + SetZnFlags(context, res, writeCC, extended); + } + } + + private static void EmitIsetp( + EmitterContext context, + IComp cmpOp, + BoolOp logicOp, + Operand srcA, + Operand srcB, + int srcPred, + bool srcPredInv, + int destPred, + int destPredInv, + bool isSigned, + bool extended) + { + Operand p0Res = GetIntComparison(context, cmpOp, srcA, srcB, isSigned, extended); + Operand p1Res = context.BitwiseNot(p0Res); + Operand pred = GetPredicate(context, srcPred, srcPredInv); + + p0Res = GetPredLogicalOp(context, logicOp, p0Res, pred); + p1Res = GetPredLogicalOp(context, logicOp, p1Res, pred); + + context.Copy(Register(destPred, RegisterType.Predicate), p0Res); + context.Copy(Register(destPredInv, RegisterType.Predicate), p1Res); + } + + private static Operand GetIntComparison( + EmitterContext context, + IComp cond, + Operand srcA, + Operand srcB, + bool isSigned, + bool extended) + { + return extended + ? GetIntComparisonExtended(context, cond, srcA, srcB, isSigned) + : GetIntComparison(context, cond, srcA, srcB, isSigned); + } + + private static Operand GetIntComparisonExtended(EmitterContext context, IComp cond, Operand srcA, Operand srcB, bool isSigned) + { + Operand res; + + if (cond == IComp.T) + { + res = Const(IrConsts.True); + } + else if (cond == IComp.F) + { + res = Const(IrConsts.False); + } + else + { + res = context.ISubtract(srcA, srcB); +#pragma warning disable IDE0059 // Remove unnecessary value assignment + res = context.IAdd(res, context.BitwiseNot(GetCF())); +#pragma warning restore IDE0059 + + switch (cond) + { + case IComp.Eq: // r = xh == yh && xl == yl + res = context.BitwiseAnd(context.ICompareEqual(srcA, srcB), GetZF()); + break; + case IComp.Lt: // r = xh < yh || (xh == yh && xl < yl) + Operand notC = context.BitwiseNot(GetCF()); + Operand prevLt = context.BitwiseAnd(context.ICompareEqual(srcA, srcB), notC); + res = isSigned + ? context.BitwiseOr(context.ICompareLess(srcA, srcB), prevLt) + : context.BitwiseOr(context.ICompareLessUnsigned(srcA, srcB), prevLt); + break; + case IComp.Le: // r = xh < yh || (xh == yh && xl <= yl) + Operand zOrNotC = context.BitwiseOr(GetZF(), context.BitwiseNot(GetCF())); + Operand prevLe = context.BitwiseAnd(context.ICompareEqual(srcA, srcB), zOrNotC); + res = isSigned + ? context.BitwiseOr(context.ICompareLess(srcA, srcB), prevLe) + : context.BitwiseOr(context.ICompareLessUnsigned(srcA, srcB), prevLe); + break; + case IComp.Gt: // r = xh > yh || (xh == yh && xl > yl) + Operand notZAndC = context.BitwiseAnd(context.BitwiseNot(GetZF()), GetCF()); + Operand prevGt = context.BitwiseAnd(context.ICompareEqual(srcA, srcB), notZAndC); + res = isSigned + ? context.BitwiseOr(context.ICompareGreater(srcA, srcB), prevGt) + : context.BitwiseOr(context.ICompareGreaterUnsigned(srcA, srcB), prevGt); + break; + case IComp.Ge: // r = xh > yh || (xh == yh && xl >= yl) + Operand prevGe = context.BitwiseAnd(context.ICompareEqual(srcA, srcB), GetCF()); + res = isSigned + ? context.BitwiseOr(context.ICompareGreater(srcA, srcB), prevGe) + : context.BitwiseOr(context.ICompareGreaterUnsigned(srcA, srcB), prevGe); + break; + case IComp.Ne: // r = xh != yh || xl != yl + res = context.BitwiseOr(context.ICompareNotEqual(srcA, srcB), context.BitwiseNot(GetZF())); + break; + default: + throw new ArgumentException($"Unexpected condition \"{cond}\"."); + } + } + + return res; + } + + private static Operand GetIntComparison(EmitterContext context, IComp cond, Operand srcA, Operand srcB, bool isSigned) + { + Operand res; + + if (cond == IComp.T) + { + res = Const(IrConsts.True); + } + else if (cond == IComp.F) + { + res = Const(IrConsts.False); + } + else + { + Instruction inst = cond switch + { + IComp.Lt => Instruction.CompareLessU32, + IComp.Eq => Instruction.CompareEqual, + IComp.Le => Instruction.CompareLessOrEqualU32, + IComp.Gt => Instruction.CompareGreaterU32, + IComp.Ne => Instruction.CompareNotEqual, + IComp.Ge => Instruction.CompareGreaterOrEqualU32, + _ => throw new InvalidOperationException($"Unexpected condition \"{cond}\"."), + }; + + if (isSigned) + { + switch (cond) + { + case IComp.Lt: + inst = Instruction.CompareLess; + break; + case IComp.Le: + inst = Instruction.CompareLessOrEqual; + break; + case IComp.Gt: + inst = Instruction.CompareGreater; + break; + case IComp.Ge: + inst = Instruction.CompareGreaterOrEqual; + break; + } + } + + res = context.Add(inst, Local(), srcA, srcB); + } + + return res; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitIntegerLogical.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitIntegerLogical.cs new file mode 100644 index 0000000..769c052 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitIntegerLogical.cs @@ -0,0 +1,166 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using static Ryujinx.Graphics.Shader.Instructions.InstEmitAluHelper; +using static Ryujinx.Graphics.Shader.Instructions.InstEmitHelper; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + private const int PT = RegisterConsts.PredicateTrueIndex; + + public static void LopR(EmitterContext context) + { + InstLopR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + + EmitLop(context, op.Lop, op.PredicateOp, srcA, srcB, op.Dest, op.DestPred, op.NegA, op.NegB, op.X, op.WriteCC); + } + + public static void LopI(EmitterContext context) + { + InstLopI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToSInt(op.Imm20)); + + EmitLop(context, op.LogicOp, op.PredicateOp, srcA, srcB, op.Dest, op.DestPred, op.NegA, op.NegB, op.X, op.WriteCC); + } + + public static void LopC(EmitterContext context) + { + InstLopC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitLop(context, op.LogicOp, op.PredicateOp, srcA, srcB, op.Dest, op.DestPred, op.NegA, op.NegB, op.X, op.WriteCC); + } + + public static void Lop32i(EmitterContext context) + { + InstLop32i op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, op.Imm32); + + EmitLop(context, op.LogicOp, PredicateOp.F, srcA, srcB, op.Dest, PT, op.NegA, op.NegB, op.X, op.WriteCC); + } + + public static void Lop3R(EmitterContext context) + { + InstLop3R op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitLop3(context, op.Imm, op.PredicateOp, srcA, srcB, srcC, op.Dest, op.DestPred, op.X, op.WriteCC); + } + + public static void Lop3I(EmitterContext context) + { + InstLop3I op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToSInt(op.Imm20)); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitLop3(context, op.Imm, PredicateOp.F, srcA, srcB, srcC, op.Dest, PT, false, op.WriteCC); + } + + public static void Lop3C(EmitterContext context) + { + InstLop3C op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitLop3(context, op.Imm, PredicateOp.F, srcA, srcB, srcC, op.Dest, PT, false, op.WriteCC); + } + + private static void EmitLop( + EmitterContext context, + LogicOp logicOp, + PredicateOp predOp, + Operand srcA, + Operand srcB, + int rd, + int destPred, + bool invertA, + bool invertB, + bool extended, + bool writeCC) + { + srcA = context.BitwiseNot(srcA, invertA); + srcB = context.BitwiseNot(srcB, invertB); + + Operand res = logicOp switch + { + LogicOp.And => context.BitwiseAnd(srcA, srcB), + LogicOp.Or => context.BitwiseOr(srcA, srcB), + LogicOp.Xor => context.BitwiseExclusiveOr(srcA, srcB), + _ => srcB, + }; + + EmitLopPredWrite(context, res, predOp, destPred); + + context.Copy(GetDest(rd), res); + + SetZnFlags(context, res, writeCC, extended); + } + + private static void EmitLop3( + EmitterContext context, + int truthTable, + PredicateOp predOp, + Operand srcA, + Operand srcB, + Operand srcC, + int rd, + int destPred, + bool extended, + bool writeCC) + { + Operand res = Lop3Expression.GetFromTruthTable(context, srcA, srcB, srcC, truthTable); + + EmitLopPredWrite(context, res, predOp, destPred); + + context.Copy(GetDest(rd), res); + + SetZnFlags(context, res, writeCC, extended); + } + + private static void EmitLopPredWrite(EmitterContext context, Operand result, PredicateOp predOp, int pred) + { + if (pred != RegisterConsts.PredicateTrueIndex) + { + Operand pRes; + + if (predOp == PredicateOp.F) + { + pRes = Const(IrConsts.False); + } + else if (predOp == PredicateOp.T) + { + pRes = Const(IrConsts.True); + } + else if (predOp == PredicateOp.Z) + { + pRes = context.ICompareEqual(result, Const(0)); + } + else /* if (predOp == Pop.Nz) */ + { + pRes = context.ICompareNotEqual(result, Const(0)); + } + + context.Copy(Register(pred, RegisterType.Predicate), pRes); + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitIntegerMinMax.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitIntegerMinMax.cs new file mode 100644 index 0000000..45fbd28 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitIntegerMinMax.cs @@ -0,0 +1,71 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; + +using static Ryujinx.Graphics.Shader.Instructions.InstEmitAluHelper; +using static Ryujinx.Graphics.Shader.Instructions.InstEmitHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void ImnmxR(EmitterContext context) + { + InstImnmxR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + Operand srcPred = GetPredicate(context, op.SrcPred, op.SrcPredInv); + + EmitImnmx(context, srcA, srcB, srcPred, op.Dest, op.Signed, op.WriteCC); + } + + public static void ImnmxI(EmitterContext context) + { + InstImnmxI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToSInt(op.Imm20)); + Operand srcPred = GetPredicate(context, op.SrcPred, op.SrcPredInv); + + EmitImnmx(context, srcA, srcB, srcPred, op.Dest, op.Signed, op.WriteCC); + } + + public static void ImnmxC(EmitterContext context) + { + InstImnmxC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + Operand srcPred = GetPredicate(context, op.SrcPred, op.SrcPredInv); + + EmitImnmx(context, srcA, srcB, srcPred, op.Dest, op.Signed, op.WriteCC); + } + + private static void EmitImnmx( + EmitterContext context, + Operand srcA, + Operand srcB, + Operand srcPred, + int rd, + bool isSignedInt, + bool writeCC) + { + Operand resMin = isSignedInt + ? context.IMinimumS32(srcA, srcB) + : context.IMinimumU32(srcA, srcB); + + Operand resMax = isSignedInt + ? context.IMaximumS32(srcA, srcB) + : context.IMaximumU32(srcA, srcB); + + Operand res = context.ConditionalSelect(srcPred, resMin, resMax); + + context.Copy(GetDest(rd), res); + + SetZnFlags(context, res, writeCC); + + // TODO: X flags. + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitMemory.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitMemory.cs new file mode 100644 index 0000000..fcb8f24 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitMemory.cs @@ -0,0 +1,604 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using System.Numerics; +using static Ryujinx.Graphics.Shader.Instructions.InstEmitHelper; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void Atom(EmitterContext context) + { + InstAtom op = context.GetOp(); + + int sOffset = (op.Imm20 << 12) >> 12; + + (Operand addrLow, Operand addrHigh) = Get40BitsAddress(context, new Register(op.SrcA, RegisterType.Gpr), op.E, sOffset); + + Operand value = GetSrcReg(context, op.SrcB); + + Operand res = EmitAtomicOp(context, StorageKind.GlobalMemory, op.Op, op.Size, addrLow, addrHigh, value); + + context.Copy(GetDest(op.Dest), res); + } + + public static void Atoms(EmitterContext context) + { + if (context.TranslatorContext.Definitions.Stage != ShaderStage.Compute) + { + context.TranslatorContext.GpuAccessor.Log($"Atoms instruction is not valid on \"{context.TranslatorContext.Definitions.Stage}\" stage."); + return; + } + + InstAtoms op = context.GetOp(); + + Operand offset = context.ShiftRightU32(GetSrcReg(context, op.SrcA), Const(2)); + + int sOffset = (op.Imm22 << 10) >> 10; + + offset = context.IAdd(offset, Const(sOffset)); + + Operand value = GetSrcReg(context, op.SrcB); + + AtomSize size = op.AtomsSize switch + { + AtomsSize.S32 => AtomSize.S32, + AtomsSize.U64 => AtomSize.U64, + AtomsSize.S64 => AtomSize.S64, + _ => AtomSize.U32, + }; + + Operand id = Const(context.ResourceManager.SharedMemoryId); + Operand res = EmitAtomicOp(context, StorageKind.SharedMemory, op.AtomOp, size, id, offset, value); + + context.Copy(GetDest(op.Dest), res); + } + + public static void Ldc(EmitterContext context) + { + InstLdc op = context.GetOp(); + + if (op.LsSize > LsSize2.B64) + { + context.TranslatorContext.GpuAccessor.Log($"Invalid LDC size: {op.LsSize}."); + return; + } + + bool isSmallInt = op.LsSize < LsSize2.B32; + + int count = op.LsSize == LsSize2.B64 ? 2 : 1; + + Operand slot = Const(op.CbufSlot); + Operand srcA = GetSrcReg(context, op.SrcA); + + if (op.AddressMode is AddressMode.Is or AddressMode.Isl) + { + slot = context.IAdd(slot, context.BitfieldExtractU32(srcA, Const(16), Const(16))); + srcA = context.BitwiseAnd(srcA, Const(0xffff)); + } + + Operand addr = context.IAdd(srcA, Const(Imm16ToSInt(op.CbufOffset))); + Operand wordOffset = context.ShiftRightU32(addr, Const(2)); + + for (int index = 0; index < count; index++) + { + Register dest = new(op.Dest + index, RegisterType.Gpr); + + if (dest.IsRZ) + { + break; + } + + Operand offset = context.IAdd(wordOffset, Const(index)); + Operand value = EmitLoadConstant(context, slot, offset); + + if (isSmallInt) + { + value = ExtractSmallInt(context, (LsSize)op.LsSize, GetBitOffset(context, addr), value); + } + + context.Copy(Register(dest), value); + } + } + + public static void Ldg(EmitterContext context) + { + InstLdg op = context.GetOp(); + + EmitLdg(context, op.LsSize, op.SrcA, op.Dest, Imm24ToSInt(op.Imm24), op.E); + } + + public static void Ldl(EmitterContext context) + { + InstLdl op = context.GetOp(); + + EmitLoad(context, StorageKind.LocalMemory, op.LsSize, GetSrcReg(context, op.SrcA), op.Dest, Imm24ToSInt(op.Imm24)); + } + + public static void Lds(EmitterContext context) + { + if (context.TranslatorContext.Definitions.Stage != ShaderStage.Compute) + { + context.TranslatorContext.GpuAccessor.Log($"Lds instruction is not valid on \"{context.TranslatorContext.Definitions.Stage}\" stage."); + return; + } + + InstLds op = context.GetOp(); + + EmitLoad(context, StorageKind.SharedMemory, op.LsSize, GetSrcReg(context, op.SrcA), op.Dest, Imm24ToSInt(op.Imm24)); + } + + public static void Red(EmitterContext context) + { + InstRed op = context.GetOp(); + + (Operand addrLow, Operand addrHigh) = Get40BitsAddress(context, new Register(op.SrcA, RegisterType.Gpr), op.E, op.Imm20); + + EmitAtomicOp(context, StorageKind.GlobalMemory, (AtomOp)op.RedOp, op.RedSize, addrLow, addrHigh, GetDest(op.SrcB)); + } + + public static void Stg(EmitterContext context) + { + InstStg op = context.GetOp(); + + EmitStg(context, op.LsSize, op.SrcA, op.Dest, Imm24ToSInt(op.Imm24), op.E); + } + + public static void Stl(EmitterContext context) + { + InstStl op = context.GetOp(); + + EmitStore(context, StorageKind.LocalMemory, op.LsSize, GetSrcReg(context, op.SrcA), op.Dest, Imm24ToSInt(op.Imm24)); + } + + public static void Sts(EmitterContext context) + { + if (context.TranslatorContext.Definitions.Stage != ShaderStage.Compute) + { + context.TranslatorContext.GpuAccessor.Log($"Sts instruction is not valid on \"{context.TranslatorContext.Definitions.Stage}\" stage."); + return; + } + + InstSts op = context.GetOp(); + + EmitStore(context, StorageKind.SharedMemory, op.LsSize, GetSrcReg(context, op.SrcA), op.Dest, Imm24ToSInt(op.Imm24)); + } + + private static Operand EmitLoadConstant(EmitterContext context, Operand slot, Operand offset) + { + Operand vecIndex = context.ShiftRightU32(offset, Const(2)); + Operand elemIndex = context.BitwiseAnd(offset, Const(3)); + + if (slot.Type == OperandType.Constant) + { + int binding = context.ResourceManager.GetConstantBufferBinding(slot.Value); + return context.Load(StorageKind.ConstantBuffer, binding, Const(0), vecIndex, elemIndex); + } + else + { + Operand value = Const(0); + + uint cbUseMask = context.TranslatorContext.GpuAccessor.QueryConstantBufferUse(); + + while (cbUseMask != 0) + { + int cbIndex = BitOperations.TrailingZeroCount(cbUseMask); + int binding = context.ResourceManager.GetConstantBufferBinding(cbIndex); + + Operand isCurrent = context.ICompareEqual(slot, Const(cbIndex)); + Operand currentValue = context.Load(StorageKind.ConstantBuffer, binding, Const(0), vecIndex, elemIndex); + + value = context.ConditionalSelect(isCurrent, currentValue, value); + + cbUseMask &= ~(1u << cbIndex); + } + + return value; + } + } + + private static Operand EmitAtomicOp( + EmitterContext context, + StorageKind storageKind, + AtomOp op, + AtomSize type, + Operand e0, + Operand e1, + Operand value) + { + Operand res = Const(0); + + switch (op) + { + case AtomOp.Add: + if (type is AtomSize.S32 or AtomSize.U32) + { + res = context.AtomicAdd(storageKind, e0, e1, value); + } + else + { + context.TranslatorContext.GpuAccessor.Log($"Invalid reduction type: {type}."); + } + + break; + case AtomOp.Min: + if (type == AtomSize.S32) + { + res = context.AtomicMinS32(storageKind, e0, e1, value); + } + else if (type == AtomSize.U32) + { + res = context.AtomicMinU32(storageKind, e0, e1, value); + } + else + { + context.TranslatorContext.GpuAccessor.Log($"Invalid reduction type: {type}."); + } + + break; + case AtomOp.Max: + if (type == AtomSize.S32) + { + res = context.AtomicMaxS32(storageKind, e0, e1, value); + } + else if (type == AtomSize.U32) + { + res = context.AtomicMaxU32(storageKind, e0, e1, value); + } + else + { + context.TranslatorContext.GpuAccessor.Log($"Invalid reduction type: {type}."); + } + + break; + case AtomOp.And: + if (type is AtomSize.S32 or AtomSize.U32) + { + res = context.AtomicAnd(storageKind, e0, e1, value); + } + else + { + context.TranslatorContext.GpuAccessor.Log($"Invalid reduction type: {type}."); + } + + break; + case AtomOp.Or: + if (type is AtomSize.S32 or AtomSize.U32) + { + res = context.AtomicOr(storageKind, e0, e1, value); + } + else + { + context.TranslatorContext.GpuAccessor.Log($"Invalid reduction type: {type}."); + } + + break; + case AtomOp.Xor: + if (type is AtomSize.S32 or AtomSize.U32) + { + res = context.AtomicXor(storageKind, e0, e1, value); + } + else + { + context.TranslatorContext.GpuAccessor.Log($"Invalid reduction type: {type}."); + } + + break; + case AtomOp.Exch: + if (type is AtomSize.S32 or AtomSize.U32) + { + res = context.AtomicSwap(storageKind, e0, e1, value); + } + else + { + context.TranslatorContext.GpuAccessor.Log($"Invalid reduction type: {type}."); + } + + break; + default: + context.TranslatorContext.GpuAccessor.Log($"Invalid atomic operation: {op}."); + break; + } + + return res; + } + + private static void EmitLoad( + EmitterContext context, + StorageKind storageKind, + LsSize2 size, + Operand srcA, + int rd, + int offset) + { + if (size > LsSize2.B128) + { + context.TranslatorContext.GpuAccessor.Log($"Invalid load size: {size}."); + return; + } + + int id = storageKind == StorageKind.LocalMemory + ? context.ResourceManager.LocalMemoryId + : context.ResourceManager.SharedMemoryId; + bool isSmallInt = size < LsSize2.B32; + + int count = size switch + { + LsSize2.B64 => 2, + LsSize2.B128 => 4, + _ => 1, + }; + + Operand baseOffset = context.Copy(srcA); + + for (int index = 0; index < count; index++) + { + Register dest = new(rd + index, RegisterType.Gpr); + + if (dest.IsRZ) + { + break; + } + + Operand byteOffset = context.IAdd(baseOffset, Const(offset + index * 4)); + Operand wordOffset = context.ShiftRightU32(byteOffset, Const(2)); // Word offset = byte offset / 4 (one word = 4 bytes). + Operand bitOffset = GetBitOffset(context, byteOffset); + Operand value = context.Load(storageKind, id, wordOffset); + + if (isSmallInt) + { + value = ExtractSmallInt(context, (LsSize)size, bitOffset, value); + } + + context.Copy(Register(dest), value); + } + } + + private static void EmitLdg( + EmitterContext context, + LsSize size, + int ra, + int rd, + int offset, + bool extended) + { + int count = GetVectorCount(size); + StorageKind storageKind = GetStorageKind(size); + + (_, Operand addrHigh) = Get40BitsAddress(context, new Register(ra, RegisterType.Gpr), extended, offset); + + Operand srcA = context.Copy(new Operand(new Register(ra, RegisterType.Gpr))); + + for (int index = 0; index < count; index++) + { + Register dest = new(rd + index, RegisterType.Gpr); + + if (dest.IsRZ) + { + break; + } + + Operand value = context.Load(storageKind, context.IAdd(srcA, Const(offset + index * 4)), addrHigh); + + context.Copy(Register(dest), value); + } + } + + private static void EmitStore( + EmitterContext context, + StorageKind storageKind, + LsSize2 size, + Operand srcA, + int rd, + int offset) + { + if (size > LsSize2.B128) + { + context.TranslatorContext.GpuAccessor.Log($"Invalid store size: {size}."); + return; + } + + int id = storageKind == StorageKind.LocalMemory + ? context.ResourceManager.LocalMemoryId + : context.ResourceManager.SharedMemoryId; + bool isSmallInt = size < LsSize2.B32; + + int count = size switch + { + LsSize2.B64 => 2, + LsSize2.B128 => 4, + _ => 1, + }; + + Operand baseOffset = context.Copy(srcA); + + for (int index = 0; index < count; index++) + { + bool isRz = rd + index >= RegisterConsts.RegisterZeroIndex; + + Operand value = Register(isRz ? rd : rd + index, RegisterType.Gpr); + Operand byteOffset = context.IAdd(baseOffset, Const(offset + index * 4)); + Operand wordOffset = context.ShiftRightU32(byteOffset, Const(2)); + Operand bitOffset = GetBitOffset(context, byteOffset); + + if (isSmallInt && storageKind == StorageKind.LocalMemory) + { + Operand word = context.Load(storageKind, id, wordOffset); + + value = InsertSmallInt(context, (LsSize)size, bitOffset, word, value); + } + + if (storageKind == StorageKind.LocalMemory) + { + context.Store(storageKind, id, wordOffset, value); + } + else if (storageKind == StorageKind.SharedMemory) + { + switch (size) + { + case LsSize2.U8: + case LsSize2.S8: + context.Store(StorageKind.SharedMemory8, id, byteOffset, value); + break; + case LsSize2.U16: + case LsSize2.S16: + context.Store(StorageKind.SharedMemory16, id, byteOffset, value); + break; + default: + context.Store(storageKind, id, wordOffset, value); + break; + } + } + } + } + + private static void EmitStg( + EmitterContext context, + LsSize2 size, + int ra, + int rd, + int offset, + bool extended) + { + if (size > LsSize2.B128) + { + context.TranslatorContext.GpuAccessor.Log($"Invalid store size: {size}."); + return; + } + + int count = GetVectorCount((LsSize)size); + StorageKind storageKind = GetStorageKind((LsSize)size); + + (_, Operand addrHigh) = Get40BitsAddress(context, new Register(ra, RegisterType.Gpr), extended, offset); + + Operand srcA = context.Copy(new Operand(new Register(ra, RegisterType.Gpr))); + + for (int index = 0; index < count; index++) + { + bool isRz = rd + index >= RegisterConsts.RegisterZeroIndex; + + Operand value = Register(isRz ? rd : rd + index, RegisterType.Gpr); + + Operand addrLowOffset = context.IAdd(srcA, Const(offset + index * 4)); + + context.Store(storageKind, addrLowOffset, addrHigh, value); + } + } + + private static StorageKind GetStorageKind(LsSize size) + { + return size switch + { + LsSize.U8 => StorageKind.GlobalMemoryU8, + LsSize.S8 => StorageKind.GlobalMemoryS8, + LsSize.U16 => StorageKind.GlobalMemoryU16, + LsSize.S16 => StorageKind.GlobalMemoryS16, + _ => StorageKind.GlobalMemory, + }; + } + + private static int GetVectorCount(LsSize size) + { + return size switch + { + LsSize.B64 => 2, + LsSize.B128 or LsSize.UB128 => 4, + _ => 1, + }; + } + + private static (Operand, Operand) Get40BitsAddress( + EmitterContext context, + Register ra, + bool extended, + int offset) + { + Operand addrLow = Register(ra); + Operand addrHigh; + + if (extended && !ra.IsRZ) + { + addrHigh = Register(ra.Index + 1, RegisterType.Gpr); + } + else + { + addrHigh = Const(0); + } + + Operand offs = Const(offset); + + addrLow = context.IAdd(addrLow, offs); + + if (extended) + { + Operand carry = context.ICompareLessUnsigned(addrLow, offs); + + addrHigh = context.IAdd(addrHigh, context.ConditionalSelect(carry, Const(1), Const(0))); + } + + return (addrLow, addrHigh); + } + + private static Operand GetBitOffset(EmitterContext context, Operand baseOffset) + { + // Note: bit offset = (baseOffset & 0b11) * 8. + // Addresses should be always aligned to the integer type, + // so we don't need to take unaligned addresses into account. + return context.ShiftLeft(context.BitwiseAnd(baseOffset, Const(3)), Const(3)); + } + + private static Operand ExtractSmallInt( + EmitterContext context, + LsSize size, + Operand bitOffset, + Operand value) + { + value = context.ShiftRightU32(value, bitOffset); + + switch (size) + { + case LsSize.U8: + value = ZeroExtendTo32(context, value, 8); + break; + case LsSize.U16: + value = ZeroExtendTo32(context, value, 16); + break; + case LsSize.S8: + value = SignExtendTo32(context, value, 8); + break; + case LsSize.S16: + value = SignExtendTo32(context, value, 16); + break; + } + + return value; + } + + private static Operand InsertSmallInt( + EmitterContext context, + LsSize size, + Operand bitOffset, + Operand word, + Operand value) + { + switch (size) + { + case LsSize.U8: + case LsSize.S8: + value = context.BitwiseAnd(value, Const(0xff)); + value = context.BitfieldInsert(word, value, bitOffset, Const(8)); + break; + + case LsSize.U16: + case LsSize.S16: + value = context.BitwiseAnd(value, Const(0xffff)); + value = context.BitfieldInsert(word, value, bitOffset, Const(16)); + break; + } + + return value; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitMove.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitMove.cs new file mode 100644 index 0000000..9f3df63 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitMove.cs @@ -0,0 +1,278 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; + +using static Ryujinx.Graphics.Shader.Instructions.InstEmitHelper; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void MovR(EmitterContext context) + { + InstMovR op = context.GetOp(); + + context.Copy(GetDest(op.Dest), GetSrcReg(context, op.SrcA)); + } + + public static void MovI(EmitterContext context) + { + InstMovI op = context.GetOp(); + + context.Copy(GetDest(op.Dest), GetSrcImm(context, op.Imm20)); + } + + public static void MovC(EmitterContext context) + { + InstMovC op = context.GetOp(); + + context.Copy(GetDest(op.Dest), GetSrcCbuf(context, op.CbufSlot, op.CbufOffset)); + } + + public static void Mov32i(EmitterContext context) + { + InstMov32i op = context.GetOp(); + + context.Copy(GetDest(op.Dest), GetSrcImm(context, op.Imm32)); + } + + public static void R2pR(EmitterContext context) + { + InstR2pR op = context.GetOp(); + + Operand value = GetSrcReg(context, op.SrcA); + Operand mask = GetSrcReg(context, op.SrcB); + + EmitR2p(context, value, mask, op.ByteSel, op.Ccpr); + } + + public static void R2pI(EmitterContext context) + { + InstR2pI op = context.GetOp(); + + Operand value = GetSrcReg(context, op.SrcA); + Operand mask = GetSrcImm(context, Imm20ToSInt(op.Imm20)); + + EmitR2p(context, value, mask, op.ByteSel, op.Ccpr); + } + + public static void R2pC(EmitterContext context) + { + InstR2pC op = context.GetOp(); + + Operand value = GetSrcReg(context, op.SrcA); + Operand mask = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitR2p(context, value, mask, op.ByteSel, op.Ccpr); + } + + public static void S2r(EmitterContext context) + { + InstS2r op = context.GetOp(); + + Operand src; + + switch (op.SReg) + { + case SReg.LaneId: + src = EmitLoadSubgroupLaneId(context); + break; + + case SReg.InvocationId: + src = context.Load(StorageKind.Input, IoVariable.InvocationId); + break; + + case SReg.YDirection: + src = ConstF(1); // TODO: Use value from Y direction GPU register. + break; + + case SReg.ThreadKill: + src = context.TranslatorContext.Definitions.Stage == ShaderStage.Fragment ? context.Load(StorageKind.Input, IoVariable.ThreadKill) : Const(0); + break; + + case SReg.InvocationInfo: + if (context.TranslatorContext.Definitions.Stage is not ShaderStage.Compute and not ShaderStage.Fragment) + { + // Note: Lowest 8-bits seems to contain some primitive index, + // but it seems to be NVIDIA implementation specific as it's only used + // to calculate ISBE offsets, so we can just keep it as zero. + + if (context.TranslatorContext.Definitions.Stage is ShaderStage.TessellationControl or + ShaderStage.TessellationEvaluation) + { + src = context.ShiftLeft(context.Load(StorageKind.Input, IoVariable.PatchVertices), Const(16)); + } + else + { + src = Const(context.TranslatorContext.Definitions.InputTopology.InputVertexCount << 16); + } + } + else + { + src = Const(0); + } + + break; + + case SReg.TId: + Operand tidX = context.Load(StorageKind.Input, IoVariable.ThreadId, null, Const(0)); + Operand tidY = context.Load(StorageKind.Input, IoVariable.ThreadId, null, Const(1)); + Operand tidZ = context.Load(StorageKind.Input, IoVariable.ThreadId, null, Const(2)); + + tidY = context.ShiftLeft(tidY, Const(16)); + tidZ = context.ShiftLeft(tidZ, Const(26)); + + src = context.BitwiseOr(tidX, context.BitwiseOr(tidY, tidZ)); + break; + + case SReg.TIdX: + src = context.Load(StorageKind.Input, IoVariable.ThreadId, null, Const(0)); + break; + case SReg.TIdY: + src = context.Load(StorageKind.Input, IoVariable.ThreadId, null, Const(1)); + break; + case SReg.TIdZ: + src = context.Load(StorageKind.Input, IoVariable.ThreadId, null, Const(2)); + break; + + case SReg.CtaIdX: + src = context.Load(StorageKind.Input, IoVariable.CtaId, null, Const(0)); + break; + case SReg.CtaIdY: + src = context.Load(StorageKind.Input, IoVariable.CtaId, null, Const(1)); + break; + case SReg.CtaIdZ: + src = context.Load(StorageKind.Input, IoVariable.CtaId, null, Const(2)); + break; + + case SReg.EqMask: + src = EmitLoadSubgroupMask(context, IoVariable.SubgroupEqMask); + break; + case SReg.LtMask: + src = EmitLoadSubgroupMask(context, IoVariable.SubgroupLtMask); + break; + case SReg.LeMask: + src = EmitLoadSubgroupMask(context, IoVariable.SubgroupLeMask); + break; + case SReg.GtMask: + src = EmitLoadSubgroupMask(context, IoVariable.SubgroupGtMask); + break; + case SReg.GeMask: + src = EmitLoadSubgroupMask(context, IoVariable.SubgroupGeMask); + break; + + default: + src = Const(0); + break; + } + + context.Copy(GetDest(op.Dest), src); + } + + private static Operand EmitLoadSubgroupLaneId(EmitterContext context) + { + if (context.TranslatorContext.GpuAccessor.QueryHostSubgroupSize() <= 32) + { + return context.Load(StorageKind.Input, IoVariable.SubgroupLaneId); + } + + return context.BitwiseAnd(context.Load(StorageKind.Input, IoVariable.SubgroupLaneId), Const(0x1f)); + } + + private static Operand EmitLoadSubgroupMask(EmitterContext context, IoVariable ioVariable) + { + int subgroupSize = context.TranslatorContext.GpuAccessor.QueryHostSubgroupSize(); + + if (subgroupSize <= 32) + { + return context.Load(StorageKind.Input, ioVariable, null, Const(0)); + } + else if (subgroupSize == 64) + { + Operand laneId = context.Load(StorageKind.Input, IoVariable.SubgroupLaneId); + Operand low = context.Load(StorageKind.Input, ioVariable, null, Const(0)); + Operand high = context.Load(StorageKind.Input, ioVariable, null, Const(1)); + + return context.ConditionalSelect(context.BitwiseAnd(laneId, Const(32)), high, low); + } + else + { + Operand laneId = context.Load(StorageKind.Input, IoVariable.SubgroupLaneId); + Operand element = context.ShiftRightU32(laneId, Const(5)); + + Operand res = context.Load(StorageKind.Input, ioVariable, null, Const(0)); + res = context.ConditionalSelect( + context.ICompareEqual(element, Const(1)), + context.Load(StorageKind.Input, ioVariable, null, Const(1)), res); + res = context.ConditionalSelect( + context.ICompareEqual(element, Const(2)), + context.Load(StorageKind.Input, ioVariable, null, Const(2)), res); + res = context.ConditionalSelect( + context.ICompareEqual(element, Const(3)), + context.Load(StorageKind.Input, ioVariable, null, Const(3)), res); + + return res; + } + } + + public static void SelR(EmitterContext context) + { + InstSelR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + Operand srcPred = GetPredicate(context, op.SrcPred, op.SrcPredInv); + + EmitSel(context, srcA, srcB, srcPred, op.Dest); + } + + public static void SelI(EmitterContext context) + { + InstSelI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToSInt(op.Imm20)); + Operand srcPred = GetPredicate(context, op.SrcPred, op.SrcPredInv); + + EmitSel(context, srcA, srcB, srcPred, op.Dest); + } + + public static void SelC(EmitterContext context) + { + InstSelC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + Operand srcPred = GetPredicate(context, op.SrcPred, op.SrcPredInv); + + EmitSel(context, srcA, srcB, srcPred, op.Dest); + } + + private static void EmitR2p(EmitterContext context, Operand value, Operand mask, ByteSel byteSel, bool ccpr) + { + Operand Test(Operand value, int bit) + { + return context.ICompareNotEqual(context.BitwiseAnd(value, Const(1 << bit)), Const(0)); + } + + int count = ccpr ? RegisterConsts.FlagsCount : RegisterConsts.PredsCount; + RegisterType type = ccpr ? RegisterType.Flag : RegisterType.Predicate; + int shift = (int)byteSel * 8; + + for (int bit = 0; bit < count; bit++) + { + Operand flag = Register(bit, type); + Operand res = context.ConditionalSelect(Test(mask, bit), Test(value, bit + shift), flag); + context.Copy(flag, res); + } + } + + private static void EmitSel(EmitterContext context, Operand srcA, Operand srcB, Operand srcPred, int rd) + { + Operand res = context.ConditionalSelect(srcPred, srcA, srcB); + + context.Copy(GetDest(rd), res); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitMultifunction.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitMultifunction.cs new file mode 100644 index 0000000..2ea88f9 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitMultifunction.cs @@ -0,0 +1,97 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; + +using static Ryujinx.Graphics.Shader.Instructions.InstEmitHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void RroR(EmitterContext context) + { + InstRroR op = context.GetOp(); + + EmitRro(context, GetSrcReg(context, op.SrcB), op.Dest, op.AbsB, op.NegB); + } + + public static void RroI(EmitterContext context) + { + InstRroI op = context.GetOp(); + + EmitRro(context, GetSrcImm(context, Imm20ToFloat(op.Imm20)), op.Dest, op.AbsB, op.NegB); + } + + public static void RroC(EmitterContext context) + { + InstRroC op = context.GetOp(); + + EmitRro(context, GetSrcCbuf(context, op.CbufSlot, op.CbufOffset), op.Dest, op.AbsB, op.NegB); + } + + public static void Mufu(EmitterContext context) + { + InstMufu op = context.GetOp(); + + Operand res = context.FPAbsNeg(GetSrcReg(context, op.SrcA), op.AbsA, op.NegA); + + switch (op.MufuOp) + { + case MufuOp.Cos: + res = context.FPCosine(res); + break; + + case MufuOp.Sin: + res = context.FPSine(res); + break; + + case MufuOp.Ex2: + res = context.FPExponentB2(res); + break; + + case MufuOp.Lg2: + res = context.FPLogarithmB2(res); + break; + + case MufuOp.Rcp: + res = context.FPReciprocal(res); + break; + + case MufuOp.Rsq: + res = context.FPReciprocalSquareRoot(res); + break; + + case MufuOp.Rcp64h: + res = context.PackDouble2x32(OperandHelper.Const(0), res); + res = context.UnpackDouble2x32High(context.FPReciprocal(res, Instruction.FP64)); + break; + + case MufuOp.Rsq64h: + res = context.PackDouble2x32(OperandHelper.Const(0), res); + res = context.UnpackDouble2x32High(context.FPReciprocalSquareRoot(res, Instruction.FP64)); + break; + + case MufuOp.Sqrt: + res = context.FPSquareRoot(res); + break; + + default: + context.TranslatorContext.GpuAccessor.Log($"Invalid MUFU operation \"{op.MufuOp}\"."); + break; + } + + context.Copy(GetDest(op.Dest), context.FPSaturate(res, op.Sat)); + } + + private static void EmitRro(EmitterContext context, Operand srcB, int rd, bool absB, bool negB) + { + // This is the range reduction operator, + // we translate it as a simple move, as it + // should be always followed by a matching + // MUFU instruction. + srcB = context.FPAbsNeg(srcB, absB, negB); + + context.Copy(GetDest(rd), srcB); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitNop.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitNop.cs new file mode 100644 index 0000000..743bfce --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitNop.cs @@ -0,0 +1,15 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.Translation; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void Nop(EmitterContext context) + { + context.GetOp(); + + // No operation. + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitPredicate.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitPredicate.cs new file mode 100644 index 0000000..645a464 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitPredicate.cs @@ -0,0 +1,116 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using static Ryujinx.Graphics.Shader.Instructions.InstEmitAluHelper; +using static Ryujinx.Graphics.Shader.Instructions.InstEmitHelper; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void Pset(EmitterContext context) + { + InstPset op = context.GetOp(); + + Operand srcA = context.BitwiseNot(Register(op.Src2Pred, RegisterType.Predicate), op.Src2PredInv); + Operand srcB = context.BitwiseNot(Register(op.Src1Pred, RegisterType.Predicate), op.Src1PredInv); + Operand srcC = context.BitwiseNot(Register(op.SrcPred, RegisterType.Predicate), op.SrcPredInv); + + Operand res = GetPredLogicalOp(context, op.BoolOpAB, srcA, srcB); + res = GetPredLogicalOp(context, op.BoolOpC, res, srcC); + + Operand dest = GetDest(op.Dest); + + if (op.BVal) + { + context.Copy(dest, context.ConditionalSelect(res, ConstF(1), ConstF(0))); + } + else + { + context.Copy(dest, res); + } + } + + public static void Psetp(EmitterContext context) + { + InstPsetp op = context.GetOp(); + + Operand srcA = context.BitwiseNot(Register(op.Src2Pred, RegisterType.Predicate), op.Src2PredInv); + Operand srcB = context.BitwiseNot(Register(op.Src1Pred, RegisterType.Predicate), op.Src1PredInv); + + Operand p0Res = GetPredLogicalOp(context, op.BoolOpAB, srcA, srcB); + Operand p1Res = context.BitwiseNot(p0Res); + Operand srcPred = GetPredicate(context, op.SrcPred, op.SrcPredInv); + + p0Res = GetPredLogicalOp(context, op.BoolOpC, p0Res, srcPred); + p1Res = GetPredLogicalOp(context, op.BoolOpC, p1Res, srcPred); + + context.Copy(Register(op.DestPred, RegisterType.Predicate), p0Res); + context.Copy(Register(op.DestPredInv, RegisterType.Predicate), p1Res); + } + + public static void P2rC(EmitterContext context) + { + InstP2rC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand dest = GetSrcReg(context, op.Dest); + Operand mask = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitP2r(context, srcA, dest, mask, op.ByteSel, op.Ccpr); + } + + public static void P2rI(EmitterContext context) + { + InstP2rI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand dest = GetSrcReg(context, op.Dest); + Operand mask = GetSrcImm(context, op.Imm20); + + EmitP2r(context, srcA, dest, mask, op.ByteSel, op.Ccpr); + } + + public static void P2rR(EmitterContext context) + { + InstP2rR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand dest = GetSrcReg(context, op.Dest); + Operand mask = GetSrcReg(context, op.SrcB); + + EmitP2r(context, srcA, dest, mask, op.ByteSel, op.Ccpr); + } + + private static void EmitP2r( + EmitterContext context, + Operand srcA, + Operand dest, + Operand mask, + ByteSel byteSel, + bool ccpr) + { + int count = ccpr ? RegisterConsts.FlagsCount : RegisterConsts.PredsCount; + int shift = (int)byteSel * 8; + mask = context.BitwiseAnd(mask, Const(0xff)); + + Operand insert = Const(0); + for (int i = 0; i < count; i++) + { + Operand condition = ccpr + ? Register(i, RegisterType.Flag) + : Register(i, RegisterType.Predicate); + + Operand bit = context.ConditionalSelect(condition, Const(1 << (i + shift)), Const(0)); + insert = context.BitwiseOr(insert, bit); + } + + Operand maskShifted = context.ShiftLeft(mask, Const(shift)); + Operand masked = context.BitwiseAnd(srcA, context.BitwiseNot(maskShifted)); + Operand res = context.BitwiseOr(masked, context.BitwiseAnd(insert, maskShifted)); + + context.Copy(dest, res); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitShift.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitShift.cs new file mode 100644 index 0000000..100ad95 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitShift.cs @@ -0,0 +1,249 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; + +using static Ryujinx.Graphics.Shader.Instructions.InstEmitHelper; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void ShfLR(EmitterContext context) + { + InstShfLR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitShf(context, op.MaxShift, srcA, srcB, srcC, op.Dest, op.M, left: true, op.WriteCC); + } + + public static void ShfRR(EmitterContext context) + { + InstShfRR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitShf(context, op.MaxShift, srcA, srcB, srcC, op.Dest, op.M, left: false, op.WriteCC); + } + + public static void ShfLI(EmitterContext context) + { + InstShfLI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = Const(op.Imm6); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitShf(context, op.MaxShift, srcA, srcB, srcC, op.Dest, op.M, left: true, op.WriteCC); + } + + public static void ShfRI(EmitterContext context) + { + InstShfRI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = Const(op.Imm6); + Operand srcC = GetSrcReg(context, op.SrcC); + + EmitShf(context, op.MaxShift, srcA, srcB, srcC, op.Dest, op.M, left: false, op.WriteCC); + } + + public static void ShlR(EmitterContext context) + { + InstShlR op = context.GetOp(); + + EmitShl(context, GetSrcReg(context, op.SrcA), GetSrcReg(context, op.SrcB), op.Dest, op.M); + } + + public static void ShlI(EmitterContext context) + { + InstShlI op = context.GetOp(); + + EmitShl(context, GetSrcReg(context, op.SrcA), GetSrcImm(context, Imm20ToSInt(op.Imm20)), op.Dest, op.M); + } + + public static void ShlC(EmitterContext context) + { + InstShlC op = context.GetOp(); + + EmitShl(context, GetSrcReg(context, op.SrcA), GetSrcCbuf(context, op.CbufSlot, op.CbufOffset), op.Dest, op.M); + } + + public static void ShrR(EmitterContext context) + { + InstShrR op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + + EmitShr(context, srcA, srcB, op.Dest, op.M, op.Brev, op.Signed); + } + + public static void ShrI(EmitterContext context) + { + InstShrI op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcImm(context, Imm20ToSInt(op.Imm20)); + + EmitShr(context, srcA, srcB, op.Dest, op.M, op.Brev, op.Signed); + } + + public static void ShrC(EmitterContext context) + { + InstShrC op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset); + + EmitShr(context, srcA, srcB, op.Dest, op.M, op.Brev, op.Signed); + } + + private static void EmitShf( + EmitterContext context, + MaxShift maxShift, + Operand srcA, + Operand srcB, + Operand srcC, + int rd, + bool mask, + bool left, + bool writeCC) + { + bool isLongShift = maxShift is MaxShift.U64 or MaxShift.S64; + bool signedShift = maxShift == MaxShift.S64; + int maxShiftConst = isLongShift ? 64 : 32; + + if (mask) + { + srcB = context.BitwiseAnd(srcB, Const(maxShiftConst - 1)); + } + + Operand res; + + if (left) + { + // res = (C << B) | (A >> (32 - B)) + res = context.ShiftLeft(srcC, srcB); + res = context.BitwiseOr(res, context.ShiftRightU32(srcA, context.ISubtract(Const(32), srcB))); + + if (isLongShift) + { + // res = B >= 32 ? A << (B - 32) : res + Operand lowerShift = context.ShiftLeft(srcA, context.ISubtract(srcB, Const(32))); + + Operand shiftGreaterThan31 = context.ICompareGreaterOrEqualUnsigned(srcB, Const(32)); + res = context.ConditionalSelect(shiftGreaterThan31, lowerShift, res); + } + } + else + { + // res = (A >> B) | (C << (32 - B)) + res = context.ShiftRightU32(srcA, srcB); + res = context.BitwiseOr(res, context.ShiftLeft(srcC, context.ISubtract(Const(32), srcB))); + + if (isLongShift) + { + // res = B >= 32 ? C >> (B - 32) : res + Operand upperShift = signedShift + ? context.ShiftRightS32(srcC, context.ISubtract(srcB, Const(32))) + : context.ShiftRightU32(srcC, context.ISubtract(srcB, Const(32))); + + Operand shiftGreaterThan31 = context.ICompareGreaterOrEqualUnsigned(srcB, Const(32)); + res = context.ConditionalSelect(shiftGreaterThan31, upperShift, res); + } + } + + if (!mask) + { + // Clamped shift value. + Operand isLessThanMax = context.ICompareLessUnsigned(srcB, Const(maxShiftConst)); + + res = context.ConditionalSelect(isLessThanMax, res, Const(0)); + } + + context.Copy(GetDest(rd), res); + + if (writeCC) + { + InstEmitAluHelper.SetZnFlags(context, res, writeCC); + } + + // TODO: X. + } + + private static void EmitShl(EmitterContext context, Operand srcA, Operand srcB, int rd, bool mask) + { + if (mask) + { + srcB = context.BitwiseAnd(srcB, Const(0x1f)); + } + + Operand res = context.ShiftLeft(srcA, srcB); + + if (!mask) + { + // Clamped shift value. + Operand isLessThan32 = context.ICompareLessUnsigned(srcB, Const(32)); + + res = context.ConditionalSelect(isLessThan32, res, Const(0)); + } + + // TODO: X, CC. + + context.Copy(GetDest(rd), res); + } + + private static void EmitShr( + EmitterContext context, + Operand srcA, + Operand srcB, + int rd, + bool mask, + bool bitReverse, + bool isSigned) + { + if (bitReverse) + { + srcA = context.BitfieldReverse(srcA); + } + + if (mask) + { + srcB = context.BitwiseAnd(srcB, Const(0x1f)); + } + + Operand res = isSigned + ? context.ShiftRightS32(srcA, srcB) + : context.ShiftRightU32(srcA, srcB); + + if (!mask) + { + // Clamped shift value. + Operand resShiftBy32; + + if (isSigned) + { + resShiftBy32 = context.ShiftRightS32(srcA, Const(31)); + } + else + { + resShiftBy32 = Const(0); + } + + Operand isLessThan32 = context.ICompareLessUnsigned(srcB, Const(32)); + + res = context.ConditionalSelect(isLessThan32, res, resShiftBy32); + } + + // TODO: X, CC. + + context.Copy(GetDest(rd), res); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitSurface.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitSurface.cs new file mode 100644 index 0000000..3583e3a --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitSurface.cs @@ -0,0 +1,799 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.StructuredIr; +using Ryujinx.Graphics.Shader.Translation; +using System; +using System.Collections.Generic; +using System.Numerics; +using static Ryujinx.Graphics.Shader.Instructions.InstEmitHelper; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void SuatomB(EmitterContext context) + { + InstSuatomB op = context.GetOp(); + + EmitSuatom( + context, + op.Dim, + op.Op, + op.Size, + 0, + op.SrcA, + op.SrcB, + op.SrcC, + op.Dest, + op.Ba, + isBindless: true, + compareAndSwap: false); + } + + public static void Suatom(EmitterContext context) + { + InstSuatom op = context.GetOp(); + + EmitSuatom( + context, + op.Dim, + op.Op, + op.Size, + op.TidB, + op.SrcA, + op.SrcB, + 0, + op.Dest, + op.Ba, + isBindless: false, + compareAndSwap: false); + } + + public static void SuatomB2(EmitterContext context) + { + InstSuatomB2 op = context.GetOp(); + + EmitSuatom( + context, + op.Dim, + op.Op, + op.Size, + 0, + op.SrcA, + op.SrcB, + op.SrcC, + op.Dest, + op.Ba, + isBindless: true, + compareAndSwap: false); + } + + public static void SuatomCasB(EmitterContext context) + { + InstSuatomCasB op = context.GetOp(); + + EmitSuatom( + context, + op.Dim, + 0, + op.Size, + 0, + op.SrcA, + op.SrcB, + op.SrcC, + op.Dest, + op.Ba, + isBindless: true, + compareAndSwap: true); + } + + public static void SuatomCas(EmitterContext context) + { + InstSuatomCas op = context.GetOp(); + + EmitSuatom( + context, + op.Dim, + 0, + op.Size, + op.TidB, + op.SrcA, + op.SrcB, + 0, + op.Dest, + op.Ba, + isBindless: false, + compareAndSwap: true); + } + + public static void SuldDB(EmitterContext context) + { + InstSuldDB op = context.GetOp(); + + EmitSuld(context, op.CacheOp, op.Dim, op.Size, 0, 0, op.SrcA, op.Dest, op.SrcC, useComponents: false, op.Ba, isBindless: true); + } + + public static void SuldD(EmitterContext context) + { + InstSuldD op = context.GetOp(); + + EmitSuld(context, op.CacheOp, op.Dim, op.Size, op.TidB, 0, op.SrcA, op.Dest, 0, useComponents: false, op.Ba, isBindless: false); + } + + public static void SuldB(EmitterContext context) + { + InstSuldB op = context.GetOp(); + + EmitSuld(context, op.CacheOp, op.Dim, 0, 0, op.Rgba, op.SrcA, op.Dest, op.SrcC, useComponents: true, false, isBindless: true); + } + + public static void Suld(EmitterContext context) + { + InstSuld op = context.GetOp(); + + EmitSuld(context, op.CacheOp, op.Dim, 0, op.TidB, op.Rgba, op.SrcA, op.Dest, 0, useComponents: true, false, isBindless: false); + } + + public static void SuredB(EmitterContext context) + { + InstSuredB op = context.GetOp(); + + EmitSured(context, op.Dim, op.Op, op.Size, 0, op.SrcA, op.Dest, op.SrcC, op.Ba, isBindless: true); + } + + public static void Sured(EmitterContext context) + { + InstSured op = context.GetOp(); + + EmitSured(context, op.Dim, op.Op, op.Size, op.TidB, op.SrcA, op.Dest, 0, op.Ba, isBindless: false); + } + + public static void SustDB(EmitterContext context) + { + InstSustDB op = context.GetOp(); + + EmitSust(context, op.CacheOp, op.Dim, op.Size, 0, 0, op.SrcA, op.Dest, op.SrcC, useComponents: false, op.Ba, isBindless: true); + } + + public static void SustD(EmitterContext context) + { + InstSustD op = context.GetOp(); + + EmitSust(context, op.CacheOp, op.Dim, op.Size, op.TidB, 0, op.SrcA, op.Dest, 0, useComponents: false, op.Ba, isBindless: false); + } + + public static void SustB(EmitterContext context) + { + InstSustB op = context.GetOp(); + + EmitSust(context, op.CacheOp, op.Dim, 0, 0, op.Rgba, op.SrcA, op.Dest, op.SrcC, useComponents: true, false, isBindless: true); + } + + public static void Sust(EmitterContext context) + { + InstSust op = context.GetOp(); + + EmitSust(context, op.CacheOp, op.Dim, 0, op.TidB, op.Rgba, op.SrcA, op.Dest, 0, useComponents: true, false, isBindless: false); + } + + private static void EmitSuatom( + EmitterContext context, + SuDim dimensions, + SuatomOp atomicOp, + SuatomSize size, + int imm, + int srcA, + int srcB, + int srcC, + int dest, + bool byteAddress, + bool isBindless, + bool compareAndSwap) + { + SamplerType type = ConvertSamplerType(dimensions); + + if (type == SamplerType.None) + { + context.TranslatorContext.GpuAccessor.Log("Invalid image atomic sampler type."); + return; + } + + Operand Ra() + { + if (srcA > RegisterConsts.RegisterZeroIndex) + { + return Const(0); + } + + return context.Copy(Register(srcA++, RegisterType.Gpr)); + } + + Operand Rb() + { + if (srcB > RegisterConsts.RegisterZeroIndex) + { + return Const(0); + } + + return context.Copy(Register(srcB++, RegisterType.Gpr)); + } + + Operand d = Register(dest, RegisterType.Gpr); + + List sourcesList = []; + + if (isBindless) + { + sourcesList.Add(context.Copy(GetSrcReg(context, srcC))); + } + + int coordsCount = type.Dimensions; + + for (int index = 0; index < coordsCount; index++) + { + sourcesList.Add(Ra()); + } + + if (Sample1DAs2D && (type & SamplerType.Mask) == SamplerType.Texture1D) + { + sourcesList.Add(Const(0)); + + type &= ~SamplerType.Mask; + type |= SamplerType.Texture2D; + } + + if (type.HasFlag(SamplerType.Array)) + { + sourcesList.Add(Ra()); + + type |= SamplerType.Array; + } + + if (byteAddress) + { + int xIndex = isBindless ? 1 : 0; + + sourcesList[xIndex] = context.ShiftRightS32(sourcesList[xIndex], Const(GetComponentSizeInBytesLog2(size))); + } + + // TODO: FP and 64-bit formats. + TextureFormat format = size is SuatomSize.Sd32 or SuatomSize.Sd64 + ? (isBindless ? TextureFormat.Unknown : ShaderProperties.GetTextureFormatAtomic(context.TranslatorContext.GpuAccessor, imm)) + : GetTextureFormat(size); + + if (compareAndSwap) + { + sourcesList.Add(Rb()); + } + + sourcesList.Add(Rb()); + + Operand[] sources = sourcesList.ToArray(); + + TextureFlags flags = compareAndSwap ? TextureFlags.CAS : GetAtomicOpFlags(atomicOp); + + if (isBindless) + { + flags |= TextureFlags.Bindless; + } + + SetBindingPair setAndBinding = isBindless ? default : context.ResourceManager.GetTextureOrImageBinding( + Instruction.ImageAtomic, + type, + format, + flags, + TextureOperation.DefaultCbufSlot, + imm); + + Operand res = context.ImageAtomic(type, format, flags, setAndBinding, sources); + + context.Copy(d, res); + } + + private static void EmitSuld( + EmitterContext context, + CacheOpLd cacheOp, + SuDim dimensions, + SuSize size, + int imm, + SuRgba componentMask, + int srcA, + int srcB, + int srcC, + bool useComponents, + bool byteAddress, + bool isBindless) + { + if (srcB == RegisterConsts.RegisterZeroIndex) + { + return; + } + + SamplerType type = ConvertSamplerType(dimensions); + + if (type == SamplerType.None) + { + context.TranslatorContext.GpuAccessor.Log("Invalid image store sampler type."); + return; + } + + Operand Ra() + { + if (srcA > RegisterConsts.RegisterZeroIndex) + { + return Const(0); + } + + return context.Copy(Register(srcA++, RegisterType.Gpr)); + } + + List sourcesList = []; + + if (isBindless) + { + sourcesList.Add(context.Copy(Register(srcC, RegisterType.Gpr))); + } + + int coordsCount = type.Dimensions; + + for (int index = 0; index < coordsCount; index++) + { + sourcesList.Add(Ra()); + } + + if (Sample1DAs2D && (type & SamplerType.Mask) == SamplerType.Texture1D) + { + sourcesList.Add(Const(0)); + + type &= ~SamplerType.Mask; + type |= SamplerType.Texture2D; + } + + if (type.HasFlag(SamplerType.Array)) + { + sourcesList.Add(Ra()); + } + + Operand[] sources = sourcesList.ToArray(); + + int handle = imm; + + TextureFlags flags = isBindless ? TextureFlags.Bindless : TextureFlags.None; + + if (cacheOp == CacheOpLd.Cg) + { + flags |= TextureFlags.Coherent; + } + + if (useComponents) + { + Operand[] dests = new Operand[BitOperations.PopCount((uint)componentMask)]; + + int outputIndex = 0; + + for (int i = 0; i < dests.Length; i++) + { + if (srcB + i >= RegisterConsts.RegisterZeroIndex) + { + break; + } + + dests[outputIndex++] = Register(srcB + i, RegisterType.Gpr); + } + + if (outputIndex != dests.Length) + { + Array.Resize(ref dests, outputIndex); + } + + TextureFormat format = isBindless ? TextureFormat.Unknown : ShaderProperties.GetTextureFormat(context.TranslatorContext.GpuAccessor, handle); + + SetBindingPair setAndBinding = isBindless ? default : context.ResourceManager.GetTextureOrImageBinding( + Instruction.ImageLoad, + type, + format, + flags, + TextureOperation.DefaultCbufSlot, + handle); + + context.ImageLoad(type, format, flags, setAndBinding, (int)componentMask, dests, sources); + } + else + { + if (byteAddress) + { + int xIndex = isBindless ? 1 : 0; + + sources[xIndex] = context.ShiftRightS32(sources[xIndex], Const(GetComponentSizeInBytesLog2(size))); + } + + int components = GetComponents(size); + int compMask = (1 << components) - 1; + + Operand[] dests = new Operand[components]; + + int outputIndex = 0; + + for (int i = 0; i < dests.Length; i++) + { + if (srcB + i >= RegisterConsts.RegisterZeroIndex) + { + break; + } + + dests[outputIndex++] = Register(srcB + i, RegisterType.Gpr); + } + + if (outputIndex != dests.Length) + { + Array.Resize(ref dests, outputIndex); + } + + TextureFormat format = GetTextureFormat(size); + + SetBindingPair setAndBinding = isBindless ? default : context.ResourceManager.GetTextureOrImageBinding( + Instruction.ImageLoad, + type, + format, + flags, + TextureOperation.DefaultCbufSlot, + handle); + + context.ImageLoad(type, format, flags, setAndBinding, compMask, dests, sources); + + switch (size) + { + case SuSize.U8: + context.Copy(dests[0], ZeroExtendTo32(context, dests[0], 8)); + break; + case SuSize.U16: + context.Copy(dests[0], ZeroExtendTo32(context, dests[0], 16)); + break; + case SuSize.S8: + context.Copy(dests[0], SignExtendTo32(context, dests[0], 8)); + break; + case SuSize.S16: + context.Copy(dests[0], SignExtendTo32(context, dests[0], 16)); + break; + } + } + } + + private static void EmitSured( + EmitterContext context, + SuDim dimensions, + RedOp atomicOp, + SuatomSize size, + int imm, + int srcA, + int srcB, + int srcC, + bool byteAddress, + bool isBindless) + { + SamplerType type = ConvertSamplerType(dimensions); + + if (type == SamplerType.None) + { + context.TranslatorContext.GpuAccessor.Log("Invalid image reduction sampler type."); + return; + } + + Operand Ra() + { + if (srcA > RegisterConsts.RegisterZeroIndex) + { + return Const(0); + } + + return context.Copy(Register(srcA++, RegisterType.Gpr)); + } + + Operand Rb() + { + if (srcB > RegisterConsts.RegisterZeroIndex) + { + return Const(0); + } + + return context.Copy(Register(srcB++, RegisterType.Gpr)); + } + + List sourcesList = []; + + if (isBindless) + { + sourcesList.Add(context.Copy(GetSrcReg(context, srcC))); + } + + int coordsCount = type.Dimensions; + + for (int index = 0; index < coordsCount; index++) + { + sourcesList.Add(Ra()); + } + + if (Sample1DAs2D && (type & SamplerType.Mask) == SamplerType.Texture1D) + { + sourcesList.Add(Const(0)); + + type &= ~SamplerType.Mask; + type |= SamplerType.Texture2D; + } + + if (type.HasFlag(SamplerType.Array)) + { + sourcesList.Add(Ra()); + + type |= SamplerType.Array; + } + + if (byteAddress) + { + int xIndex = isBindless ? 1 : 0; + + sourcesList[xIndex] = context.ShiftRightS32(sourcesList[xIndex], Const(GetComponentSizeInBytesLog2(size))); + } + + // TODO: FP and 64-bit formats. + TextureFormat format = size is SuatomSize.Sd32 or SuatomSize.Sd64 + ? (isBindless ? TextureFormat.Unknown : ShaderProperties.GetTextureFormatAtomic(context.TranslatorContext.GpuAccessor, imm)) + : GetTextureFormat(size); + + sourcesList.Add(Rb()); + + Operand[] sources = sourcesList.ToArray(); + + TextureFlags flags = GetAtomicOpFlags((SuatomOp)atomicOp); + + if (isBindless) + { + flags |= TextureFlags.Bindless; + } + + SetBindingPair setAndBinding = isBindless ? default : context.ResourceManager.GetTextureOrImageBinding( + Instruction.ImageAtomic, + type, + format, + flags, + TextureOperation.DefaultCbufSlot, + imm); + + context.ImageAtomic(type, format, flags, setAndBinding, sources); + } + + private static void EmitSust( + EmitterContext context, + CacheOpSt cacheOp, + SuDim dimensions, + SuSize size, + int imm, + SuRgba componentMask, + int srcA, + int srcB, + int srcC, + bool useComponents, + bool byteAddress, + bool isBindless) + { + SamplerType type = ConvertSamplerType(dimensions); + + if (type == SamplerType.None) + { + context.TranslatorContext.GpuAccessor.Log("Invalid image store sampler type."); + return; + } + + Operand Ra() + { + if (srcA > RegisterConsts.RegisterZeroIndex) + { + return Const(0); + } + + return context.Copy(Register(srcA++, RegisterType.Gpr)); + } + + Operand Rb() + { + if (srcB > RegisterConsts.RegisterZeroIndex) + { + return Const(0); + } + + return context.Copy(Register(srcB++, RegisterType.Gpr)); + } + + List sourcesList = []; + + if (isBindless) + { + sourcesList.Add(context.Copy(Register(srcC, RegisterType.Gpr))); + } + + int coordsCount = type.Dimensions; + + for (int index = 0; index < coordsCount; index++) + { + sourcesList.Add(Ra()); + } + + if (Sample1DAs2D && (type & SamplerType.Mask) == SamplerType.Texture1D) + { + sourcesList.Add(Const(0)); + + type &= ~SamplerType.Mask; + type |= SamplerType.Texture2D; + } + + if (type.HasFlag(SamplerType.Array)) + { + sourcesList.Add(Ra()); + } + + TextureFormat format = TextureFormat.Unknown; + + if (useComponents) + { + for (int compMask = (int)componentMask, compIndex = 0; compMask != 0; compMask >>= 1, compIndex++) + { + if ((compMask & 1) != 0) + { + sourcesList.Add(Rb()); + } + } + + if (!isBindless) + { + format = ShaderProperties.GetTextureFormat(context.TranslatorContext.GpuAccessor, imm); + } + } + else + { + if (byteAddress) + { + int xIndex = isBindless ? 1 : 0; + + sourcesList[xIndex] = context.ShiftRightS32(sourcesList[xIndex], Const(GetComponentSizeInBytesLog2(size))); + } + + int components = GetComponents(size); + + for (int compIndex = 0; compIndex < components; compIndex++) + { + sourcesList.Add(Rb()); + } + + format = GetTextureFormat(size); + } + + Operand[] sources = sourcesList.ToArray(); + + int handle = imm; + + TextureFlags flags = isBindless ? TextureFlags.Bindless : TextureFlags.None; + + if (cacheOp == CacheOpSt.Cg) + { + flags |= TextureFlags.Coherent; + } + + SetBindingPair setAndBinding = isBindless ? default : context.ResourceManager.GetTextureOrImageBinding( + Instruction.ImageStore, + type, + format, + flags, + TextureOperation.DefaultCbufSlot, + handle); + + context.ImageStore(type, format, flags, setAndBinding, sources); + } + + private static int GetComponentSizeInBytesLog2(SuatomSize size) + { + return size switch + { + SuatomSize.U32 => 2, + SuatomSize.S32 => 2, + SuatomSize.U64 => 3, + SuatomSize.F32FtzRn => 2, + SuatomSize.F16x2FtzRn => 2, + SuatomSize.S64 => 3, + SuatomSize.Sd32 => 2, + SuatomSize.Sd64 => 3, + _ => 2, + }; + } + + private static TextureFormat GetTextureFormat(SuatomSize size) + { + return size switch + { + SuatomSize.U32 => TextureFormat.R32Uint, + SuatomSize.S32 => TextureFormat.R32Sint, + SuatomSize.U64 => TextureFormat.R32G32Uint, + SuatomSize.F32FtzRn => TextureFormat.R32Float, + SuatomSize.F16x2FtzRn => TextureFormat.R16G16Float, + SuatomSize.S64 => TextureFormat.R32G32Uint, + SuatomSize.Sd32 => TextureFormat.R32Uint, + SuatomSize.Sd64 => TextureFormat.R32G32Uint, + _ => TextureFormat.R32Uint, + }; + } + + private static TextureFlags GetAtomicOpFlags(SuatomOp op) + { + return op switch + { + SuatomOp.Add => TextureFlags.Add, + SuatomOp.Min => TextureFlags.Minimum, + SuatomOp.Max => TextureFlags.Maximum, + SuatomOp.Inc => TextureFlags.Increment, + SuatomOp.Dec => TextureFlags.Decrement, + SuatomOp.And => TextureFlags.BitwiseAnd, + SuatomOp.Or => TextureFlags.BitwiseOr, + SuatomOp.Xor => TextureFlags.BitwiseXor, + SuatomOp.Exch => TextureFlags.Swap, + _ => TextureFlags.Add, + }; + } + + private static int GetComponents(SuSize size) + { + return size switch + { + SuSize.B64 => 2, + SuSize.B128 => 4, + SuSize.UB128 => 4, + _ => 1, + }; + } + + private static int GetComponentSizeInBytesLog2(SuSize size) + { + return size switch + { + SuSize.U8 => 0, + SuSize.S8 => 0, + SuSize.U16 => 1, + SuSize.S16 => 1, + SuSize.B32 => 2, + SuSize.B64 => 3, + SuSize.B128 => 4, + SuSize.UB128 => 4, + _ => 2, + }; + } + + private static TextureFormat GetTextureFormat(SuSize size) + { + return size switch + { + SuSize.U8 => TextureFormat.R8Uint, + SuSize.S8 => TextureFormat.R8Sint, + SuSize.U16 => TextureFormat.R16Uint, + SuSize.S16 => TextureFormat.R16Sint, + SuSize.B32 => TextureFormat.R32Uint, + SuSize.B64 => TextureFormat.R32G32Uint, + SuSize.B128 => TextureFormat.R32G32B32A32Uint, + SuSize.UB128 => TextureFormat.R32G32B32A32Uint, + _ => TextureFormat.R32Uint, + }; + } + + private static SamplerType ConvertSamplerType(SuDim target) + { + return target switch + { + SuDim._1d => SamplerType.Texture1D, + SuDim._1dBuffer => SamplerType.TextureBuffer, + SuDim._1dArray => SamplerType.Texture1D | SamplerType.Array, + SuDim._2d => SamplerType.Texture2D, + SuDim._2dArray => SamplerType.Texture2D | SamplerType.Array, + SuDim._3d => SamplerType.Texture3D, + _ => SamplerType.None, + }; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitTexture.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitTexture.cs new file mode 100644 index 0000000..6b24018 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitTexture.cs @@ -0,0 +1,1336 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using System; +using System.Collections.Generic; +using System.Numerics; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + private static readonly int[][] _maskLut = new int[][] + { + [0b0001, 0b0010, 0b0100, 0b1000, 0b0011, 0b1001, 0b1010, 0b1100], [0b0111, 0b1011, 0b1101, 0b1110, 0b1111, 0b0000, 0b0000, 0b0000 + ], + }; + + public const bool Sample1DAs2D = true; + + private enum TexsType + { + Texs, + Tlds, + Tld4s, + } + + public static void Tex(EmitterContext context) + { + InstTex op = context.GetOp(); + + EmitTex(context, TextureFlags.None, op.Dim, op.Lod, op.TidB, op.WMask, op.SrcA, op.SrcB, op.Dest, false, op.Dc, op.Aoffi); + } + + public static void TexB(EmitterContext context) + { + InstTexB op = context.GetOp(); + + EmitTex(context, TextureFlags.Bindless, op.Dim, op.Lodb, 0, op.WMask, op.SrcA, op.SrcB, op.Dest, false, op.Dc, op.Aoffib); + } + + public static void Texs(EmitterContext context) + { + InstTexs op = context.GetOp(); + + EmitTexs(context, TexsType.Texs, op.TidB, op.WMask, op.SrcA, op.SrcB, op.Dest, op.Dest2, isF16: false); + } + + public static void TexsF16(EmitterContext context) + { + InstTexs op = context.GetOp(); + + EmitTexs(context, TexsType.Texs, op.TidB, op.WMask, op.SrcA, op.SrcB, op.Dest, op.Dest2, isF16: true); + } + + public static void Tld(EmitterContext context) + { + InstTld op = context.GetOp(); + + Lod lod = op.Lod ? Lod.Ll : Lod.Lz; + + EmitTex(context, TextureFlags.IntCoords, op.Dim, lod, op.TidB, op.WMask, op.SrcA, op.SrcB, op.Dest, op.Ms, false, op.Toff); + } + + public static void TldB(EmitterContext context) + { + InstTldB op = context.GetOp(); + + TextureFlags flags = TextureFlags.IntCoords | TextureFlags.Bindless; + Lod lod = op.Lod ? Lod.Ll : Lod.Lz; + + EmitTex(context, flags, op.Dim, lod, 0, op.WMask, op.SrcA, op.SrcB, op.Dest, op.Ms, false, op.Toff); + } + + public static void Tlds(EmitterContext context) + { + InstTlds op = context.GetOp(); + + EmitTexs(context, TexsType.Tlds, op.TidB, op.WMask, op.SrcA, op.SrcB, op.Dest, op.Dest2, isF16: false); + } + + public static void TldsF16(EmitterContext context) + { + InstTlds op = context.GetOp(); + + EmitTexs(context, TexsType.Tlds, op.TidB, op.WMask, op.SrcA, op.SrcB, op.Dest, op.Dest2, isF16: true); + } + + public static void Tld4(EmitterContext context) + { + InstTld4 op = context.GetOp(); + + EmitTld4(context, op.Dim, op.TexComp, op.TidB, op.WMask, op.SrcA, op.SrcB, op.Dest, op.Toff, op.Dc, isBindless: false); + } + + public static void Tld4B(EmitterContext context) + { + InstTld4B op = context.GetOp(); + + EmitTld4(context, op.Dim, op.TexComp, 0, op.WMask, op.SrcA, op.SrcB, op.Dest, op.Toff, op.Dc, isBindless: true); + } + + public static void Tld4s(EmitterContext context) + { + InstTld4s op = context.GetOp(); + + EmitTexs(context, TexsType.Tld4s, op.TidB, 4, op.SrcA, op.SrcB, op.Dest, op.Dest2, isF16: false); + } + + public static void Tld4sF16(EmitterContext context) + { + InstTld4s op = context.GetOp(); + + EmitTexs(context, TexsType.Tld4s, op.TidB, 4, op.SrcA, op.SrcB, op.Dest, op.Dest2, isF16: true); + } + + public static void Tmml(EmitterContext context) + { + InstTmml op = context.GetOp(); + + EmitTmml(context, op.Dim, op.TidB, op.WMask, op.SrcA, op.SrcB, op.Dest, isBindless: false); + } + + public static void TmmlB(EmitterContext context) + { + InstTmmlB op = context.GetOp(); + + EmitTmml(context, op.Dim, 0, op.WMask, op.SrcA, op.SrcB, op.Dest, isBindless: true); + } + + public static void Txd(EmitterContext context) + { + InstTxd op = context.GetOp(); + + EmitTxd(context, op.Dim, op.TidB, op.WMask, op.SrcA, op.SrcB, op.Dest, op.Toff, isBindless: false); + } + + public static void TxdB(EmitterContext context) + { + InstTxdB op = context.GetOp(); + + EmitTxd(context, op.Dim, 0, op.WMask, op.SrcA, op.SrcB, op.Dest, op.Toff, isBindless: true); + } + + public static void Txq(EmitterContext context) + { + InstTxq op = context.GetOp(); + + EmitTxq(context, op.TexQuery, op.TidB, op.WMask, op.SrcA, op.Dest, isBindless: false); + } + + public static void TxqB(EmitterContext context) + { + InstTxqB op = context.GetOp(); + + EmitTxq(context, op.TexQuery, 0, op.WMask, op.SrcA, op.Dest, isBindless: true); + } + + private static void EmitTex( + EmitterContext context, + TextureFlags flags, + TexDim dimensions, + Lod lodMode, + int imm, + int componentMask, + int raIndex, + int rbIndex, + int rdIndex, + bool isMultisample, + bool hasDepthCompare, + bool hasOffset) + { + if (rdIndex == RegisterConsts.RegisterZeroIndex) + { + return; + } + + Operand Ra() + { + if (raIndex > RegisterConsts.RegisterZeroIndex) + { + return Const(0); + } + + return context.Copy(Register(raIndex++, RegisterType.Gpr)); + } + + Operand Rb() + { + if (rbIndex > RegisterConsts.RegisterZeroIndex) + { + return Const(0); + } + + return context.Copy(Register(rbIndex++, RegisterType.Gpr)); + } + + SamplerType type = ConvertSamplerType(dimensions); + + bool isArray = type.HasFlag(SamplerType.Array); + bool isBindless = flags.HasFlag(TextureFlags.Bindless); + + Operand arrayIndex = isArray ? Ra() : null; + + List sourcesList = []; + + if (isBindless) + { + sourcesList.Add(Rb()); + } + + bool hasLod = lodMode > Lod.Lz; + + if (type == SamplerType.Texture1D && (flags & ~TextureFlags.Bindless) == TextureFlags.IntCoords && !( + hasLod || + hasDepthCompare || + hasOffset || + isArray || + isMultisample)) + { + // For bindless, we don't have any way to know the texture type, + // so we assume it's texture buffer when the sampler type is 1D, since that's more common. + bool isTypeBuffer = isBindless || context.TranslatorContext.GpuAccessor.QuerySamplerType(imm) == SamplerType.TextureBuffer; + if (isTypeBuffer) + { + type = SamplerType.TextureBuffer; + } + } + + int coordsCount = type.Dimensions; + + for (int index = 0; index < coordsCount; index++) + { + sourcesList.Add(Ra()); + } + + bool is1DTo2D = false; + + if (Sample1DAs2D && (type & SamplerType.Mask) == SamplerType.Texture1D) + { + sourcesList.Add(ConstF(0)); + + type = SamplerType.Texture2D | (type & SamplerType.Array); + is1DTo2D = true; + } + + if (isArray) + { + sourcesList.Add(arrayIndex); + } + + Operand lodValue = hasLod ? Rb() : ConstF(0); + + Operand packedOffs = hasOffset ? Rb() : null; + + if (hasDepthCompare) + { + sourcesList.Add(Rb()); + + type |= SamplerType.Shadow; + } + + if ((lodMode == Lod.Lz || + lodMode == Lod.Ll || + lodMode == Lod.Lla) && !isMultisample && type != SamplerType.TextureBuffer) + { + sourcesList.Add(lodValue); + + flags |= TextureFlags.LodLevel; + } + + if (hasOffset) + { + for (int index = 0; index < coordsCount; index++) + { + sourcesList.Add(context.BitfieldExtractS32(packedOffs, Const(index * 4), Const(4))); + } + + if (is1DTo2D) + { + sourcesList.Add(Const(0)); + } + + flags |= TextureFlags.Offset; + } + + if (lodMode is Lod.Lb or Lod.Lba) + { + sourcesList.Add(lodValue); + + flags |= TextureFlags.LodBias; + } + + if (isMultisample) + { + sourcesList.Add(Rb()); + + type |= SamplerType.Multisample; + } + + Operand[] sources = sourcesList.ToArray(); + Operand[] dests = new Operand[BitOperations.PopCount((uint)componentMask)]; + + int outputIndex = 0; + + for (int i = 0; i < dests.Length; i++) + { + if (rdIndex + i >= RegisterConsts.RegisterZeroIndex) + { + break; + } + + dests[outputIndex++] = Register(rdIndex + i, RegisterType.Gpr); + } + + if (outputIndex != dests.Length) + { + Array.Resize(ref dests, outputIndex); + } + + int handle = !isBindless ? imm : 0; + + EmitTextureSample(context, type, flags, handle, componentMask, dests, sources); + } + + private static void EmitTexs( + EmitterContext context, + TexsType texsType, + int imm, + int writeMask, + int srcA, + int srcB, + int dest, + int dest2, + bool isF16) + { + if (dest == RegisterConsts.RegisterZeroIndex && dest2 == RegisterConsts.RegisterZeroIndex) + { + return; + } + + List sourcesList = []; + + Operand Ra() + { + if (srcA > RegisterConsts.RegisterZeroIndex) + { + return Const(0); + } + + return context.Copy(Register(srcA++, RegisterType.Gpr)); + } + + Operand Rb() + { + if (srcB > RegisterConsts.RegisterZeroIndex) + { + return Const(0); + } + + return context.Copy(Register(srcB++, RegisterType.Gpr)); + } + + void AddTextureOffset(int coordsCount, int stride, int size) + { + Operand packedOffs = Rb(); + + for (int index = 0; index < coordsCount; index++) + { + sourcesList.Add(context.BitfieldExtractS32(packedOffs, Const(index * stride), Const(size))); + } + } + + SamplerType type; + TextureFlags flags; + + if (texsType == TexsType.Texs) + { + InstTexs texsOp = context.GetOp(); + + type = ConvertSamplerType(texsOp.Target); + + if (type == SamplerType.None) + { + context.TranslatorContext.GpuAccessor.Log("Invalid texture sampler type."); + return; + } + + flags = ConvertTextureFlags(texsOp.Target); + + // We don't need to handle 1D -> Buffer conversions here as + // only texture sample with integer coordinates can ever use buffer targets. + + if ((type & SamplerType.Array) != 0) + { + Operand arrayIndex = Ra(); + + sourcesList.Add(Ra()); + sourcesList.Add(Rb()); + + sourcesList.Add(arrayIndex); + + if ((type & SamplerType.Shadow) != 0) + { + sourcesList.Add(Rb()); + } + + if ((flags & TextureFlags.LodLevel) != 0) + { + sourcesList.Add(ConstF(0)); + } + } + else + { + switch (texsOp.Target) + { + case TexsTarget.Texture1DLodZero: + sourcesList.Add(Ra()); + + if (Sample1DAs2D) + { + sourcesList.Add(ConstF(0)); + + type &= ~SamplerType.Mask; + type |= SamplerType.Texture2D; + } + + sourcesList.Add(ConstF(0)); + break; + + case TexsTarget.Texture2D: + sourcesList.Add(Ra()); + sourcesList.Add(Rb()); + break; + + case TexsTarget.Texture2DLodZero: + sourcesList.Add(Ra()); + sourcesList.Add(Rb()); + sourcesList.Add(ConstF(0)); + break; + + case TexsTarget.Texture2DLodLevel: + case TexsTarget.Texture2DDepthCompare: + case TexsTarget.Texture3D: + case TexsTarget.TextureCube: + sourcesList.Add(Ra()); + sourcesList.Add(Ra()); + sourcesList.Add(Rb()); + break; + + case TexsTarget.Texture2DLodZeroDepthCompare: + case TexsTarget.Texture3DLodZero: + sourcesList.Add(Ra()); + sourcesList.Add(Ra()); + sourcesList.Add(Rb()); + sourcesList.Add(ConstF(0)); + break; + + case TexsTarget.Texture2DLodLevelDepthCompare: + case TexsTarget.TextureCubeLodLevel: + sourcesList.Add(Ra()); + sourcesList.Add(Ra()); + sourcesList.Add(Rb()); + sourcesList.Add(Rb()); + break; + } + } + } + else if (texsType == TexsType.Tlds) + { + InstTlds tldsOp = context.GetOp(); + + type = ConvertSamplerType(tldsOp.Target); + + if (type == SamplerType.None) + { + context.TranslatorContext.GpuAccessor.Log("Invalid texel fetch sampler type."); + return; + } + + flags = ConvertTextureFlags(tldsOp.Target) | TextureFlags.IntCoords; + + if (tldsOp.Target == TldsTarget.Texture1DLodZero && + context.TranslatorContext.GpuAccessor.QuerySamplerType(tldsOp.TidB) == SamplerType.TextureBuffer) + { + type = SamplerType.TextureBuffer; + flags &= ~TextureFlags.LodLevel; + } + + switch (tldsOp.Target) + { + case TldsTarget.Texture1DLodZero: + sourcesList.Add(Ra()); + + if (type != SamplerType.TextureBuffer) + { + if (Sample1DAs2D) + { + sourcesList.Add(ConstF(0)); + + type &= ~SamplerType.Mask; + type |= SamplerType.Texture2D; + } + + sourcesList.Add(ConstF(0)); + } + + break; + + case TldsTarget.Texture1DLodLevel: + sourcesList.Add(Ra()); + + if (Sample1DAs2D) + { + sourcesList.Add(ConstF(0)); + + type &= ~SamplerType.Mask; + type |= SamplerType.Texture2D; + } + + sourcesList.Add(Rb()); + break; + + case TldsTarget.Texture2DLodZero: + sourcesList.Add(Ra()); + sourcesList.Add(Rb()); + sourcesList.Add(Const(0)); + break; + + case TldsTarget.Texture2DLodZeroOffset: + sourcesList.Add(Ra()); + sourcesList.Add(Ra()); + sourcesList.Add(Const(0)); + break; + + case TldsTarget.Texture2DLodZeroMultisample: + case TldsTarget.Texture2DLodLevel: + case TldsTarget.Texture2DLodLevelOffset: + sourcesList.Add(Ra()); + sourcesList.Add(Ra()); + sourcesList.Add(Rb()); + break; + + case TldsTarget.Texture3DLodZero: + sourcesList.Add(Ra()); + sourcesList.Add(Ra()); + sourcesList.Add(Rb()); + sourcesList.Add(Const(0)); + break; + + case TldsTarget.Texture2DArrayLodZero: + sourcesList.Add(Rb()); + sourcesList.Add(Rb()); + sourcesList.Add(Ra()); + sourcesList.Add(Const(0)); + break; + } + + if ((flags & TextureFlags.Offset) != 0) + { + AddTextureOffset(type.Dimensions, 4, 4); + } + } + else if (texsType == TexsType.Tld4s) + { + InstTld4s tld4sOp = context.GetOp(); + + if (!(tld4sOp.Dc || tld4sOp.Aoffi)) + { + sourcesList.Add(Ra()); + sourcesList.Add(Rb()); + } + else + { + sourcesList.Add(Ra()); + sourcesList.Add(Ra()); + } + + type = SamplerType.Texture2D; + flags = TextureFlags.Gather; + + int depthCompareIndex = sourcesList.Count; + + if (tld4sOp.Aoffi) + { + AddTextureOffset(type.Dimensions, 8, 6); + + flags |= TextureFlags.Offset; + } + + if (tld4sOp.Dc) + { + sourcesList.Insert(depthCompareIndex, Rb()); + + type |= SamplerType.Shadow; + } + else + { + sourcesList.Add(Const((int)tld4sOp.TexComp)); + } + } + else + { + throw new ArgumentException($"Invalid TEXS type \"{texsType}\"."); + } + + Operand[] sources = sourcesList.ToArray(); + + Operand[] rd0 = [ConstF(0), ConstF(0)]; + Operand[] rd1 = [ConstF(0), ConstF(0)]; + + int handle = imm; + int componentMask = _maskLut[dest2 == RegisterConsts.RegisterZeroIndex ? 0 : 1][writeMask]; + + int componentsCount = BitOperations.PopCount((uint)componentMask); + + Operand[] dests = new Operand[componentsCount]; + + int outputIndex = 0; + + for (int i = 0; i < componentsCount; i++) + { + int high = i >> 1; + int low = i & 1; + + if (isF16) + { + dests[outputIndex++] = high != 0 + ? (rd1[low] = Local()) + : (rd0[low] = Local()); + } + else + { + int rdIndex = high != 0 ? dest2 : dest; + + if (rdIndex < RegisterConsts.RegisterZeroIndex) + { + rdIndex += low; + } + + dests[outputIndex++] = Register(rdIndex, RegisterType.Gpr); + } + } + + if (outputIndex != dests.Length) + { + Array.Resize(ref dests, outputIndex); + } + + EmitTextureSample(context, type, flags, handle, componentMask, dests, sources); + + if (isF16) + { + context.Copy(Register(dest, RegisterType.Gpr), context.PackHalf2x16(rd0[0], rd0[1])); + context.Copy(Register(dest2, RegisterType.Gpr), context.PackHalf2x16(rd1[0], rd1[1])); + } + } + + private static void EmitTld4( + EmitterContext context, + TexDim dimensions, + TexComp component, + int imm, + int componentMask, + int srcA, + int srcB, + int dest, + TexOffset offset, + bool hasDepthCompare, + bool isBindless) + { + if (dest == RegisterConsts.RegisterZeroIndex) + { + return; + } + + Operand Ra() + { + if (srcA > RegisterConsts.RegisterZeroIndex) + { + return Const(0); + } + + return context.Copy(Register(srcA++, RegisterType.Gpr)); + } + + Operand Rb() + { + if (srcB > RegisterConsts.RegisterZeroIndex) + { + return Const(0); + } + + return context.Copy(Register(srcB++, RegisterType.Gpr)); + } + + bool isArray = + dimensions is TexDim.Array1d or + TexDim.Array2d or + TexDim.Array3d or + TexDim.ArrayCube; + + Operand arrayIndex = isArray ? Ra() : null; + + List sourcesList = []; + + SamplerType type = ConvertSamplerType(dimensions); + TextureFlags flags = TextureFlags.Gather; + + if (isBindless) + { + sourcesList.Add(Rb()); + + flags |= TextureFlags.Bindless; + } + + int coordsCount = type.Dimensions; + + for (int index = 0; index < coordsCount; index++) + { + sourcesList.Add(Ra()); + } + + bool is1DTo2D = Sample1DAs2D && (type & SamplerType.Mask) == SamplerType.Texture1D; + + if (is1DTo2D) + { + sourcesList.Add(ConstF(0)); + + type = SamplerType.Texture2D | (type & SamplerType.Array); + } + + if (isArray) + { + sourcesList.Add(arrayIndex); + } + + Operand[] packedOffs = new Operand[2]; + + bool hasAnyOffset = offset is TexOffset.Aoffi or TexOffset.Ptp; + + packedOffs[0] = hasAnyOffset ? Rb() : null; + packedOffs[1] = offset == TexOffset.Ptp ? Rb() : null; + + if (hasDepthCompare) + { + sourcesList.Add(Rb()); + + type |= SamplerType.Shadow; + } + + if (hasAnyOffset) + { + int offsetTexelsCount = offset == TexOffset.Ptp ? 4 : 1; + + for (int index = 0; index < coordsCount * offsetTexelsCount; index++) + { + Operand packed = packedOffs[(index >> 2) & 1]; + + sourcesList.Add(context.BitfieldExtractS32(packed, Const((index & 3) * 8), Const(6))); + } + + if (is1DTo2D) + { + for (int index = 0; index < offsetTexelsCount; index++) + { + sourcesList.Add(Const(0)); + } + } + + flags |= offset == TexOffset.Ptp ? TextureFlags.Offsets : TextureFlags.Offset; + } + + if (!hasDepthCompare) + { + sourcesList.Add(Const((int)component)); + } + + Operand[] sources = sourcesList.ToArray(); + Operand[] dests = new Operand[BitOperations.PopCount((uint)componentMask)]; + + int outputIndex = 0; + + for (int i = 0; i < dests.Length; i++) + { + if (dest + i >= RegisterConsts.RegisterZeroIndex) + { + break; + } + + dests[outputIndex++] = Register(dest + i, RegisterType.Gpr); + } + + if (outputIndex != dests.Length) + { + Array.Resize(ref dests, outputIndex); + } + + EmitTextureSample(context, type, flags, imm, componentMask, dests, sources); + } + + private static void EmitTmml( + EmitterContext context, + TexDim dimensions, + int imm, + int componentMask, + int srcA, + int srcB, + int dest, + bool isBindless) + { + if (dest == RegisterConsts.RegisterZeroIndex) + { + return; + } + + Operand Ra() + { + if (srcA > RegisterConsts.RegisterZeroIndex) + { + return Const(0); + } + + return context.Copy(Register(srcA++, RegisterType.Gpr)); + } + + Operand Rb() + { + if (srcB > RegisterConsts.RegisterZeroIndex) + { + return Const(0); + } + + return context.Copy(Register(srcB++, RegisterType.Gpr)); + } + + TextureFlags flags = TextureFlags.None; + + List sourcesList = []; + + if (isBindless) + { + sourcesList.Add(Rb()); + + flags |= TextureFlags.Bindless; + } + + SamplerType type = ConvertSamplerType(dimensions); + + int coordsCount = type.Dimensions; + + bool isArray = + dimensions is TexDim.Array1d or + TexDim.Array2d or + TexDim.Array3d or + TexDim.ArrayCube; + + Operand arrayIndex = isArray ? Ra() : null; + + for (int index = 0; index < coordsCount; index++) + { + sourcesList.Add(Ra()); + } + + if (Sample1DAs2D && (type & SamplerType.Mask) == SamplerType.Texture1D) + { + sourcesList.Add(ConstF(0)); + + type = SamplerType.Texture2D | (type & SamplerType.Array); + } + + if (isArray) + { + sourcesList.Add(arrayIndex); + } + + Operand[] sources = sourcesList.ToArray(); + + Operand GetDest() + { + if (dest >= RegisterConsts.RegisterZeroIndex) + { + return null; + } + + return Register(dest++, RegisterType.Gpr); + } + + SetBindingPair setAndBinding = isBindless ? default : context.ResourceManager.GetTextureOrImageBinding( + Instruction.Lod, + type, + TextureFormat.Unknown, + flags, + TextureOperation.DefaultCbufSlot, + imm); + + for (int compMask = componentMask, compIndex = 0; compMask != 0; compMask >>= 1, compIndex++) + { + if ((compMask & 1) != 0) + { + Operand d = GetDest(); + + if (d == null) + { + break; + } + + // Components z and w aren't standard, we return 0 in this case and add a comment. + if (compIndex >= 2) + { + context.Add(new CommentNode("Unsupported component z or w found")); + context.Copy(d, Const(0)); + } + else + { + // The instruction component order is the inverse of GLSL's. + Operand res = context.Lod(type, flags, setAndBinding, compIndex ^ 1, sources); + + res = context.FPMultiply(res, ConstF(256.0f)); + + Operand fixedPointValue = context.FP32ConvertToS32(res); + + context.Copy(d, fixedPointValue); + } + } + } + } + + private static void EmitTxd( + EmitterContext context, + TexDim dimensions, + int imm, + int componentMask, + int srcA, + int srcB, + int dest, + bool hasOffset, + bool isBindless) + { + if (dest == RegisterConsts.RegisterZeroIndex) + { + return; + } + + TextureFlags flags = TextureFlags.Derivatives; + + List sourcesList = []; + + if (isBindless) + { + sourcesList.Add(Ra()); + + flags |= TextureFlags.Bindless; + } + + SamplerType type = ConvertSamplerType(dimensions); + + int coordsCount = type.Dimensions; + + for (int index = 0; index < coordsCount; index++) + { + sourcesList.Add(Ra()); + } + + bool is1DTo2D = Sample1DAs2D && (type & SamplerType.Mask) == SamplerType.Texture1D; + + if (is1DTo2D) + { + sourcesList.Add(ConstF(0)); + + type = SamplerType.Texture2D | (type & SamplerType.Array); + } + + Operand packedParams = Ra(); + + bool isArray = + dimensions is TexDim.Array1d or + TexDim.Array2d or + TexDim.Array3d or + TexDim.ArrayCube; + + if (isArray) + { + sourcesList.Add(context.BitwiseAnd(packedParams, Const(0xffff))); + } + + // Derivatives (X and Y). + for (int dIndex = 0; dIndex < 2 * coordsCount; dIndex++) + { + sourcesList.Add(Rb()); + + if (is1DTo2D) + { + sourcesList.Add(ConstF(0)); + } + } + + if (hasOffset) + { + for (int index = 0; index < coordsCount; index++) + { + sourcesList.Add(context.BitfieldExtractS32(packedParams, Const(16 + index * 4), Const(4))); + } + + if (is1DTo2D) + { + sourcesList.Add(Const(0)); + } + + flags |= TextureFlags.Offset; + } + + Operand[] sources = sourcesList.ToArray(); + Operand[] dests = new Operand[BitOperations.PopCount((uint)componentMask)]; + + int outputIndex = 0; + + for (int i = 0; i < dests.Length; i++) + { + if (dest + i >= RegisterConsts.RegisterZeroIndex) + { + break; + } + + dests[outputIndex++] = Register(dest + i, RegisterType.Gpr); + } + + if (outputIndex != dests.Length) + { + Array.Resize(ref dests, outputIndex); + } + + EmitTextureSample(context, type, flags, imm, componentMask, dests, sources); + + return; + + Operand Ra() + { + if (srcA > RegisterConsts.RegisterZeroIndex) + { + return Const(0); + } + + return context.Copy(Register(srcA++, RegisterType.Gpr)); + } + + Operand Rb() + { + if (srcB > RegisterConsts.RegisterZeroIndex) + { + return Const(0); + } + + return context.Copy(Register(srcB++, RegisterType.Gpr)); + } + } + + private static void EmitTxq( + EmitterContext context, + TexQuery query, + int imm, + int componentMask, + int srcA, + int dest, + bool isBindless) + { + if (dest == RegisterConsts.RegisterZeroIndex) + { + return; + } + + Operand Ra() + { + if (srcA > RegisterConsts.RegisterZeroIndex) + { + return Const(0); + } + + return context.Copy(Register(srcA++, RegisterType.Gpr)); + } + + List sourcesList = []; + + if (isBindless) + { + sourcesList.Add(Ra()); + } + + sourcesList.Add(Ra()); + + Operand[] sources = sourcesList.ToArray(); + + Operand GetDest() + { + if (dest >= RegisterConsts.RegisterZeroIndex) + { + return null; + } + + return Register(dest++, RegisterType.Gpr); + } + + SamplerType type; + + if (isBindless) + { + if (query == TexQuery.TexHeaderTextureType) + { + type = SamplerType.Texture2D | SamplerType.Multisample; + } + else + { + type = (componentMask & 4) != 0 ? SamplerType.Texture3D : SamplerType.Texture2D; + } + } + else + { + type = context.TranslatorContext.GpuAccessor.QuerySamplerType(imm); + } + + TextureFlags flags = isBindless ? TextureFlags.Bindless : TextureFlags.None; + SetBindingPair setAndBinding; + + switch (query) + { + case TexQuery.TexHeaderDimension: + setAndBinding = isBindless ? default : context.ResourceManager.GetTextureOrImageBinding( + Instruction.TextureQuerySize, + type, + TextureFormat.Unknown, + flags, + TextureOperation.DefaultCbufSlot, + imm); + + for (int compMask = componentMask, compIndex = 0; compMask != 0; compMask >>= 1, compIndex++) + { + if ((compMask & 1) != 0) + { + Operand d = GetDest(); + + if (d == null) + { + break; + } + + context.Copy(d, context.TextureQuerySize(type, flags, setAndBinding, compIndex, sources)); + } + } + + break; + + case TexQuery.TexHeaderTextureType: + setAndBinding = isBindless ? default : context.ResourceManager.GetTextureOrImageBinding( + Instruction.TextureQuerySamples, + type, + TextureFormat.Unknown, + flags, + TextureOperation.DefaultCbufSlot, + imm); + + if ((componentMask & 4) != 0) + { + // Skip first 2 components if necessary. + if ((componentMask & 1) != 0) + { + GetDest(); + } + + if ((componentMask & 2) != 0) + { + GetDest(); + } + + Operand d = GetDest(); + + if (d != null) + { + context.Copy(d, context.TextureQuerySamples(type, flags, setAndBinding, sources)); + } + } + + break; + + default: + context.TranslatorContext.GpuAccessor.Log($"Invalid or unsupported query type \"{query}\"."); + break; + } + } + + private static void EmitTextureSample( + EmitterContext context, + SamplerType type, + TextureFlags flags, + int handle, + int componentMask, + Operand[] dests, + Operand[] sources) + { + SetBindingPair setAndBinding = flags.HasFlag(TextureFlags.Bindless) ? default : context.ResourceManager.GetTextureOrImageBinding( + Instruction.TextureSample, + type, + TextureFormat.Unknown, + flags, + TextureOperation.DefaultCbufSlot, + handle); + + context.TextureSample(type, flags, setAndBinding, componentMask, dests, sources); + } + + private static SamplerType ConvertSamplerType(TexDim dimensions) + { + return dimensions switch + { + TexDim._1d => SamplerType.Texture1D, + TexDim.Array1d => SamplerType.Texture1D | SamplerType.Array, + TexDim._2d => SamplerType.Texture2D, + TexDim.Array2d => SamplerType.Texture2D | SamplerType.Array, + TexDim._3d => SamplerType.Texture3D, + TexDim.Array3d => SamplerType.Texture3D | SamplerType.Array, + TexDim.Cube => SamplerType.TextureCube, + TexDim.ArrayCube => SamplerType.TextureCube | SamplerType.Array, + _ => throw new ArgumentException($"Invalid texture dimensions \"{dimensions}\"."), + }; + } + + private static SamplerType ConvertSamplerType(TexsTarget type) + { + switch (type) + { + case TexsTarget.Texture1DLodZero: + return SamplerType.Texture1D; + + case TexsTarget.Texture2D: + case TexsTarget.Texture2DLodZero: + case TexsTarget.Texture2DLodLevel: + return SamplerType.Texture2D; + + case TexsTarget.Texture2DDepthCompare: + case TexsTarget.Texture2DLodLevelDepthCompare: + case TexsTarget.Texture2DLodZeroDepthCompare: + return SamplerType.Texture2D | SamplerType.Shadow; + + case TexsTarget.Texture2DArray: + case TexsTarget.Texture2DArrayLodZero: + return SamplerType.Texture2D | SamplerType.Array; + + case TexsTarget.Texture2DArrayLodZeroDepthCompare: + return SamplerType.Texture2D | SamplerType.Array | SamplerType.Shadow; + + case TexsTarget.Texture3D: + case TexsTarget.Texture3DLodZero: + return SamplerType.Texture3D; + + case TexsTarget.TextureCube: + case TexsTarget.TextureCubeLodLevel: + return SamplerType.TextureCube; + } + + return SamplerType.None; + } + + private static SamplerType ConvertSamplerType(TldsTarget type) + { + switch (type) + { + case TldsTarget.Texture1DLodZero: + case TldsTarget.Texture1DLodLevel: + return SamplerType.Texture1D; + + case TldsTarget.Texture2DLodZero: + case TldsTarget.Texture2DLodZeroOffset: + case TldsTarget.Texture2DLodLevel: + case TldsTarget.Texture2DLodLevelOffset: + return SamplerType.Texture2D; + + case TldsTarget.Texture2DLodZeroMultisample: + return SamplerType.Texture2D | SamplerType.Multisample; + + case TldsTarget.Texture3DLodZero: + return SamplerType.Texture3D; + + case TldsTarget.Texture2DArrayLodZero: + return SamplerType.Texture2D | SamplerType.Array; + } + + return SamplerType.None; + } + + private static TextureFlags ConvertTextureFlags(TexsTarget type) + { + switch (type) + { + case TexsTarget.Texture1DLodZero: + case TexsTarget.Texture2DLodZero: + case TexsTarget.Texture2DLodLevel: + case TexsTarget.Texture2DLodLevelDepthCompare: + case TexsTarget.Texture2DLodZeroDepthCompare: + case TexsTarget.Texture2DArrayLodZero: + case TexsTarget.Texture2DArrayLodZeroDepthCompare: + case TexsTarget.Texture3DLodZero: + case TexsTarget.TextureCubeLodLevel: + return TextureFlags.LodLevel; + + case TexsTarget.Texture2D: + case TexsTarget.Texture2DDepthCompare: + case TexsTarget.Texture2DArray: + case TexsTarget.Texture3D: + case TexsTarget.TextureCube: + return TextureFlags.None; + } + + return TextureFlags.None; + } + + private static TextureFlags ConvertTextureFlags(TldsTarget type) + { + switch (type) + { + case TldsTarget.Texture1DLodZero: + case TldsTarget.Texture1DLodLevel: + case TldsTarget.Texture2DLodZero: + case TldsTarget.Texture2DLodLevel: + case TldsTarget.Texture2DLodZeroMultisample: + case TldsTarget.Texture3DLodZero: + case TldsTarget.Texture2DArrayLodZero: + return TextureFlags.LodLevel; + + case TldsTarget.Texture2DLodZeroOffset: + case TldsTarget.Texture2DLodLevelOffset: + return TextureFlags.LodLevel | TextureFlags.Offset; + } + + return TextureFlags.None; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitVideoArithmetic.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitVideoArithmetic.cs new file mode 100644 index 0000000..1ec80e9 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitVideoArithmetic.cs @@ -0,0 +1,117 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using static Ryujinx.Graphics.Shader.Instructions.InstEmitHelper; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void Vmad(EmitterContext context) + { + InstVmad op = context.GetOp(); + + bool aSigned = (op.ASelect & VectorSelect.S8B0) != 0; + bool bSigned = (op.BSelect & VectorSelect.S8B0) != 0; + + Operand srcA = InstEmitAluHelper.Extend(context, GetSrcReg(context, op.SrcA), op.ASelect); + Operand srcC = context.INegate(GetSrcReg(context, op.SrcC), op.AvgMode == AvgMode.NegB); + Operand srcB; + + if (op.BVideo) + { + srcB = InstEmitAluHelper.Extend(context, GetSrcReg(context, op.SrcB), op.BSelect); + } + else + { + int imm = op.Imm16; + + if (bSigned) + { + imm = (imm << 16) >> 16; + } + + srcB = Const(imm); + } + + Operand productLow = context.IMultiply(srcA, srcB); + Operand productHigh; + + if (aSigned == bSigned) + { + productHigh = aSigned + ? context.MultiplyHighS32(srcA, srcB) + : context.MultiplyHighU32(srcA, srcB); + } + else + { + Operand temp = aSigned + ? context.IMultiply(srcB, context.ShiftRightS32(srcA, Const(31))) + : context.IMultiply(srcA, context.ShiftRightS32(srcB, Const(31))); + + productHigh = context.IAdd(temp, context.MultiplyHighU32(srcA, srcB)); + } + + if (op.AvgMode == AvgMode.NegA) + { + (productLow, productHigh) = InstEmitAluHelper.NegateLong(context, productLow, productHigh); + } + + Operand resLow = InstEmitAluHelper.AddWithCarry(context, productLow, srcC, out Operand sumCarry); + Operand resHigh = context.IAdd(productHigh, sumCarry); + + if (op.AvgMode == AvgMode.PlusOne) + { + resLow = InstEmitAluHelper.AddWithCarry(context, resLow, Const(1), out Operand poCarry); + resHigh = context.IAdd(resHigh, poCarry); + } + + bool resSigned = op.ASelect == VectorSelect.S32 || + op.BSelect == VectorSelect.S32 || + op.AvgMode == AvgMode.NegB || + op.AvgMode == AvgMode.NegA; + + int shift = op.VideoScale switch + { + VideoScale.Shr7 => 7, + VideoScale.Shr15 => 15, + _ => 0, + }; + + if (shift != 0) + { + // Low = (Low >> Shift) | (High << (32 - Shift)) + // High >>= Shift + resLow = context.ShiftRightU32(resLow, Const(shift)); + resLow = context.BitwiseOr(resLow, context.ShiftLeft(resHigh, Const(32 - shift))); + resHigh = resSigned + ? context.ShiftRightS32(resHigh, Const(shift)) + : context.ShiftRightU32(resHigh, Const(shift)); + } + + Operand res = resLow; + + if (op.Sat) + { + Operand sign = context.ShiftRightS32(resHigh, Const(31)); + + if (resSigned) + { + Operand overflow = context.ICompareNotEqual(resHigh, context.ShiftRightS32(resLow, Const(31))); + Operand clampValue = context.ConditionalSelect(sign, Const(int.MinValue), Const(int.MaxValue)); + res = context.ConditionalSelect(overflow, clampValue, resLow); + } + else + { + Operand overflow = context.ICompareNotEqual(resHigh, Const(0)); + res = context.ConditionalSelect(overflow, context.BitwiseNot(sign), resLow); + } + } + + context.Copy(GetDest(op.Dest), res); + + // TODO: CC. + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitVideoMinMax.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitVideoMinMax.cs new file mode 100644 index 0000000..aee565b --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitVideoMinMax.cs @@ -0,0 +1,183 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; + +using static Ryujinx.Graphics.Shader.Instructions.InstEmitHelper; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void Vmnmx(EmitterContext context) + { + InstVmnmx op = context.GetOp(); + + Operand srcA = InstEmitAluHelper.Extend(context, GetSrcReg(context, op.SrcA), op.ASelect); + Operand srcC = GetSrcReg(context, op.SrcC); + Operand srcB; + + if (op.BVideo) + { + srcB = InstEmitAluHelper.Extend(context, GetSrcReg(context, op.SrcB), op.BSelect); + } + else + { + int imm = op.Imm16; + + if ((op.BSelect & VectorSelect.S8B0) != 0) + { + imm = (imm << 16) >> 16; + } + + srcB = Const(imm); + } + + Operand res; + + bool resSigned; + + if ((op.ASelect & VectorSelect.S8B0) != (op.BSelect & VectorSelect.S8B0)) + { + // Signedness is different, but for max, result will always fit a U32, + // since one of the inputs can't be negative, and the result is the one + // with highest value. For min, it will always fit on a S32, since + // one of the input can't be greater than INT_MAX and we want the lowest value. + resSigned = !op.Mn; + + res = op.Mn ? context.IMaximumU32(srcA, srcB) : context.IMinimumS32(srcA, srcB); + + if ((op.ASelect & VectorSelect.S8B0) != 0) + { + Operand isBGtIntMax = context.ICompareLess(srcB, Const(0)); + + res = context.ConditionalSelect(isBGtIntMax, srcB, res); + } + else + { + Operand isAGtIntMax = context.ICompareLess(srcA, Const(0)); + + res = context.ConditionalSelect(isAGtIntMax, srcA, res); + } + } + else + { + // Ra and Rb have the same signedness, so doesn't matter which one we test. + resSigned = (op.ASelect & VectorSelect.S8B0) != 0; + + if (op.Mn) + { + res = resSigned + ? context.IMaximumS32(srcA, srcB) + : context.IMaximumU32(srcA, srcB); + } + else + { + res = resSigned + ? context.IMinimumS32(srcA, srcB) + : context.IMinimumU32(srcA, srcB); + } + } + + if (op.Sat) + { + if (op.DFormat && !resSigned) + { + res = context.IMinimumU32(res, Const(int.MaxValue)); + } + else if (!op.DFormat && resSigned) + { + res = context.IMaximumS32(res, Const(0)); + } + } + + switch (op.VideoOp) + { + case VideoOp.Acc: + res = context.IAdd(res, srcC); + break; + case VideoOp.Max: + res = op.DFormat ? context.IMaximumS32(res, srcC) : context.IMaximumU32(res, srcC); + break; + case VideoOp.Min: + res = op.DFormat ? context.IMinimumS32(res, srcC) : context.IMinimumU32(res, srcC); + break; + case VideoOp.Mrg16h: + res = context.BitfieldInsert(srcC, res, Const(16), Const(16)); + break; + case VideoOp.Mrg16l: + res = context.BitfieldInsert(srcC, res, Const(0), Const(16)); + break; + case VideoOp.Mrg8b0: + res = context.BitfieldInsert(srcC, res, Const(0), Const(8)); + break; + case VideoOp.Mrg8b2: + res = context.BitfieldInsert(srcC, res, Const(16), Const(8)); + break; + } + + context.Copy(GetDest(op.Dest), res); + } + + public static void Vsetp(EmitterContext context) + { + InstVsetp op = context.GetOp(); + + Operand srcA = InstEmitAluHelper.Extend(context, GetSrcReg(context, op.SrcA), op.ASelect); + Operand srcB; + + if (op.BVideo) + { + srcB = InstEmitAluHelper.Extend(context, GetSrcReg(context, op.SrcB), op.BSelect); + } + else + { + int imm = op.Imm16; + + if ((op.BSelect & VectorSelect.S8B0) != 0) + { + imm = (imm << 16) >> 16; + } + + srcB = Const(imm); + } + + Operand p0Res; + + bool signedA = (op.ASelect & VectorSelect.S8B0) != 0; + bool signedB = (op.BSelect & VectorSelect.S8B0) != 0; + + if (signedA != signedB) + { + bool a32 = (op.ASelect & ~VectorSelect.S8B0) == VectorSelect.U32; + bool b32 = (op.BSelect & ~VectorSelect.S8B0) == VectorSelect.U32; + + if (!a32 && !b32) + { + // Both values are extended small integer and can always fit in a S32, just do a signed comparison. + p0Res = GetIntComparison(context, op.VComp, srcA, srcB, isSigned: true, extended: false); + } + else + { + // TODO: Mismatching sign case. + p0Res = Const(0); + } + } + else + { + // Sign matches, just do a regular comparison. + p0Res = GetIntComparison(context, op.VComp, srcA, srcB, signedA, extended: false); + } + + Operand p1Res = context.BitwiseNot(p0Res); + + Operand pred = GetPredicate(context, op.SrcPred, op.SrcPredInv); + + p0Res = InstEmitAluHelper.GetPredLogicalOp(context, op.BoolOp, p0Res, pred); + p1Res = InstEmitAluHelper.GetPredLogicalOp(context, op.BoolOp, p1Res, pred); + + context.Copy(Register(op.DestPred, RegisterType.Predicate), p0Res); + context.Copy(Register(op.DestPredInv, RegisterType.Predicate), p1Res); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitWarp.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitWarp.cs new file mode 100644 index 0000000..249df05 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitWarp.cs @@ -0,0 +1,144 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using static Ryujinx.Graphics.Shader.Instructions.InstEmitHelper; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static partial class InstEmit + { + public static void Fswzadd(EmitterContext context) + { + InstFswzadd op = context.GetOp(); + + Operand srcA = GetSrcReg(context, op.SrcA); + Operand srcB = GetSrcReg(context, op.SrcB); + Operand dest = GetDest(op.Dest); + + context.Copy(dest, context.FPSwizzleAdd(srcA, srcB, op.PnWord)); + + InstEmitAluHelper.SetFPZnFlags(context, dest, op.WriteCC); + } + + public static void Shfl(EmitterContext context) + { + InstShfl op = context.GetOp(); + + Operand pred = Register(op.DestPred, RegisterType.Predicate); + + Operand srcA = GetSrcReg(context, op.SrcA); + + Operand srcB = op.BFixShfl ? Const(op.SrcBImm) : GetSrcReg(context, op.SrcB); + Operand srcC = op.CFixShfl ? Const(op.SrcCImm) : GetSrcReg(context, op.SrcC); + + (Operand res, Operand valid) = op.ShflMode switch + { + ShflMode.Idx => context.Shuffle(srcA, srcB, srcC), + ShflMode.Up => context.ShuffleUp(srcA, srcB, srcC), + ShflMode.Down => context.ShuffleDown(srcA, srcB, srcC), + ShflMode.Bfly => context.ShuffleXor(srcA, srcB, srcC), + _ => (null, null), + }; + + context.Copy(GetDest(op.Dest), res); + context.Copy(pred, valid); + } + + public static void Vote(EmitterContext context) + { + InstVote op = context.GetOp(); + + Operand pred = GetPredicate(context, op.SrcPred, op.SrcPredInv); + Operand res = EmitVote(context, op.VoteMode, pred); + + if (res != null) + { + context.Copy(Register(op.VpDest, RegisterType.Predicate), res); + } + else + { + context.TranslatorContext.GpuAccessor.Log($"Invalid vote operation: {op.VoteMode}."); + } + + if (op.Dest != RegisterConsts.RegisterZeroIndex) + { + context.Copy(GetDest(op.Dest), EmitBallot(context, pred)); + } + } + + private static Operand EmitVote(EmitterContext context, VoteMode voteMode, Operand pred) + { + int subgroupSize = context.TranslatorContext.GpuAccessor.QueryHostSubgroupSize(); + + if (subgroupSize <= 32) + { + return voteMode switch + { + VoteMode.All => context.VoteAll(pred), + VoteMode.Any => context.VoteAny(pred), + VoteMode.Eq => context.VoteAllEqual(pred), + _ => null, + }; + } + + // Emulate vote with ballot masks. + // We do that when the GPU thread count is not 32, + // since the shader code assumes it is 32. + // allInvocations => ballot(pred) == ballot(true), + // anyInvocation => ballot(pred) != 0, + // allInvocationsEqual => ballot(pred) == balot(true) || ballot(pred) == 0 + Operand ballotMask = EmitBallot(context, pred); + + Operand AllTrue() => context.ICompareEqual(ballotMask, EmitBallot(context, Const(IrConsts.True))); + + return voteMode switch + { + VoteMode.All => AllTrue(), + VoteMode.Any => context.ICompareNotEqual(ballotMask, Const(0)), + VoteMode.Eq => context.BitwiseOr(AllTrue(), context.ICompareEqual(ballotMask, Const(0))), + _ => null, + }; + } + + private static Operand EmitBallot(EmitterContext context, Operand pred) + { + int subgroupSize = context.TranslatorContext.GpuAccessor.QueryHostSubgroupSize(); + + if (subgroupSize <= 32) + { + return context.Ballot(pred, 0); + } + else if (subgroupSize == 64) + { + // TODO: Add support for vector destination and do that with a single operation. + + Operand laneId = context.Load(StorageKind.Input, IoVariable.SubgroupLaneId); + Operand low = context.Ballot(pred, 0); + Operand high = context.Ballot(pred, 1); + + return context.ConditionalSelect(context.BitwiseAnd(laneId, Const(32)), high, low); + } + else + { + // TODO: Add support for vector destination and do that with a single operation. + + Operand laneId = context.Load(StorageKind.Input, IoVariable.SubgroupLaneId); + Operand element = context.ShiftRightU32(laneId, Const(5)); + + Operand res = context.Ballot(pred, 0); + res = context.ConditionalSelect( + context.ICompareEqual(element, Const(1)), + context.Ballot(pred, 1), res); + res = context.ConditionalSelect( + context.ICompareEqual(element, Const(2)), + context.Ballot(pred, 2), res); + res = context.ConditionalSelect( + context.ICompareEqual(element, Const(3)), + context.Ballot(pred, 3), res); + + return res; + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitter.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitter.cs new file mode 100644 index 0000000..4e486ca --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/InstEmitter.cs @@ -0,0 +1,6 @@ +using Ryujinx.Graphics.Shader.Translation; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + delegate void InstEmitter(EmitterContext context); +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/Lop3Expression.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/Lop3Expression.cs new file mode 100644 index 0000000..d201db9 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Instructions/Lop3Expression.cs @@ -0,0 +1,142 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Instructions +{ + static class Lop3Expression + { + private enum TruthTable : byte + { + False = 0x00, // false + True = 0xff, // true + In = 0xf0, // a + And2 = 0xc0, // a & b + Or2 = 0xfc, // a | b + Xor2 = 0x3c, // a ^ b + And3 = 0x80, // a & b & c + Or3 = 0xfe, // a | b | c + XorAnd = 0x60, // a & (b ^ c) + XorOr = 0xf6, // a | (b ^ c) + OrAnd = 0xe0, // a & (b | c) + AndOr = 0xf8, // a | (b & c) + Onehot = 0x16, // (a & !b & !c) | (!a & b & !c) | (!a & !b & c) - Only one value is true. + Majority = 0xe8, // Popcount(a, b, c) >= 2 + Gamble = 0x81, // (a & b & c) | (!a & !b & !c) - All on or all off + InverseGamble = 0x7e, // Inverse of Gamble + Dot = 0x1a, // a ^ (c | (a & b)) + Mux = 0xca, // a ? b : c + AndXor = 0x78, // a ^ (b & c) + OrXor = 0x1e, // a ^ (b | c) + Xor3 = 0x96, // a ^ b ^ c + } + + public static Operand GetFromTruthTable(EmitterContext context, Operand srcA, Operand srcB, Operand srcC, int imm) + { + for (int i = 0; i < 0x40; i++) + { + TruthTable currImm = (TruthTable)imm; + + Operand x = srcA; + Operand y = srcB; + Operand z = srcC; + + if ((i & 0x01) != 0) + { + (x, y) = (y, x); + currImm = PermuteTable(currImm, 7, 6, 3, 2, 5, 4, 1, 0); + } + + if ((i & 0x02) != 0) + { + (x, z) = (z, x); + currImm = PermuteTable(currImm, 7, 3, 5, 1, 6, 2, 4, 0); + } + + if ((i & 0x04) != 0) + { + (y, z) = (z, y); + currImm = PermuteTable(currImm, 7, 5, 6, 4, 3, 1, 2, 0); + } + + if ((i & 0x08) != 0) + { + x = context.BitwiseNot(x); + currImm = PermuteTable(currImm, 3, 2, 1, 0, 7, 6, 5, 4); + } + + if ((i & 0x10) != 0) + { + y = context.BitwiseNot(y); + currImm = PermuteTable(currImm, 5, 4, 7, 6, 1, 0, 3, 2); + } + + if ((i & 0x20) != 0) + { + z = context.BitwiseNot(z); + currImm = PermuteTable(currImm, 6, 7, 4, 5, 2, 3, 0, 1); + } + + Operand result = GetExpr(currImm, context, x, y, z); + if (result != null) + { + return result; + } + + Operand notResult = GetExpr((TruthTable)((~(int)currImm) & 0xff), context, x, y, z); + if (notResult != null) + { + return context.BitwiseNot(notResult); + } + } + + return null; + } + + private static Operand GetExpr(TruthTable imm, EmitterContext context, Operand x, Operand y, Operand z) + { + return imm switch + { +#pragma warning disable IDE0055 // Disable formatting + TruthTable.False => Const(0), + TruthTable.True => Const(-1), + TruthTable.In => x, + TruthTable.And2 => context.BitwiseAnd(x, y), + TruthTable.Or2 => context.BitwiseOr(x, y), + TruthTable.Xor2 => context.BitwiseExclusiveOr(x, y), + TruthTable.And3 => context.BitwiseAnd(x, context.BitwiseAnd(y, z)), + TruthTable.Or3 => context.BitwiseOr(x, context.BitwiseOr(y, z)), + TruthTable.XorAnd => context.BitwiseAnd(x, context.BitwiseExclusiveOr(y, z)), + TruthTable.XorOr => context.BitwiseOr(x, context.BitwiseExclusiveOr(y, z)), + TruthTable.OrAnd => context.BitwiseAnd(x, context.BitwiseOr(y, z)), + TruthTable.AndOr => context.BitwiseOr(x, context.BitwiseAnd(y, z)), + TruthTable.Onehot => context.BitwiseExclusiveOr(context.BitwiseOr(x, y), context.BitwiseOr(z, context.BitwiseAnd(x, y))), + TruthTable.Majority => context.BitwiseAnd(context.BitwiseOr(x, y), context.BitwiseOr(z, context.BitwiseAnd(x, y))), + TruthTable.InverseGamble => context.BitwiseOr(context.BitwiseExclusiveOr(x, y), context.BitwiseExclusiveOr(x, z)), + TruthTable.Dot => context.BitwiseAnd(context.BitwiseExclusiveOr(x, z), context.BitwiseOr(context.BitwiseNot(y), z)), + TruthTable.Mux => context.BitwiseOr(context.BitwiseAnd(x, y), context.BitwiseAnd(context.BitwiseNot(x), z)), + TruthTable.AndXor => context.BitwiseExclusiveOr(x, context.BitwiseAnd(y, z)), + TruthTable.OrXor => context.BitwiseExclusiveOr(x, context.BitwiseOr(y, z)), + TruthTable.Xor3 => context.BitwiseExclusiveOr(x, context.BitwiseExclusiveOr(y, z)), + _ => null, +#pragma warning restore IDE0055 + }; + } + + private static TruthTable PermuteTable(TruthTable imm, int bit7, int bit6, int bit5, int bit4, int bit3, int bit2, int bit1, int bit0) + { + int result = 0; + + result |= (((int)imm >> 0) & 1) << bit0; + result |= (((int)imm >> 1) & 1) << bit1; + result |= (((int)imm >> 2) & 1) << bit2; + result |= (((int)imm >> 3) & 1) << bit3; + result |= (((int)imm >> 4) & 1) << bit4; + result |= (((int)imm >> 5) & 1) << bit5; + result |= (((int)imm >> 6) & 1) << bit6; + result |= (((int)imm >> 7) & 1) << bit7; + + return (TruthTable)result; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/BasicBlock.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/BasicBlock.cs new file mode 100644 index 0000000..fdda409 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/BasicBlock.cs @@ -0,0 +1,91 @@ +using System.Collections.Generic; + +namespace Ryujinx.Graphics.Shader.IntermediateRepresentation +{ + class BasicBlock + { + public int Index { get; set; } + + public LinkedList Operations { get; } + + private BasicBlock _next; + private BasicBlock _branch; + + public BasicBlock Next + { + get => _next; + set => _next = AddSuccessor(_next, value); + } + + public BasicBlock Branch + { + get => _branch; + set => _branch = AddSuccessor(_branch, value); + } + + public bool HasBranch => _branch != null; + public bool Reachable => Index == 0 || Predecessors.Count != 0; + + public List Predecessors { get; } + + public HashSet DominanceFrontiers { get; } + + public BasicBlock ImmediateDominator { get; set; } + + public BasicBlock() + { + Operations = []; + + Predecessors = []; + + DominanceFrontiers = []; + } + + public BasicBlock(int index) : this() + { + Index = index; + } + + private BasicBlock AddSuccessor(BasicBlock oldBlock, BasicBlock newBlock) + { + oldBlock?.Predecessors.Remove(this); + newBlock?.Predecessors.Add(this); + + return newBlock; + } + + public INode GetLastOp() + { + return Operations.Last?.Value; + } + + public void Append(INode node) + { + INode lastOp = GetLastOp(); + + if (lastOp is Operation operation && IsControlFlowInst(operation.Inst)) + { + Operations.AddBefore(Operations.Last, node); + } + else + { + Operations.AddLast(node); + } + } + + private static bool IsControlFlowInst(Instruction inst) + { + switch (inst) + { + case Instruction.Branch: + case Instruction.BranchIfFalse: + case Instruction.BranchIfTrue: + case Instruction.Discard: + case Instruction.Return: + return true; + default: + return false; + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/CommentNode.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/CommentNode.cs new file mode 100644 index 0000000..404bf36 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/CommentNode.cs @@ -0,0 +1,12 @@ +namespace Ryujinx.Graphics.Shader.IntermediateRepresentation +{ + class CommentNode : Operation + { + public string Comment { get; } + + public CommentNode(string comment) : base(Instruction.Comment, null) + { + Comment = comment; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/Function.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/Function.cs new file mode 100644 index 0000000..0d9497c --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/Function.cs @@ -0,0 +1,23 @@ +namespace Ryujinx.Graphics.Shader.IntermediateRepresentation +{ + class Function + { + public BasicBlock[] Blocks { get; } + + public string Name { get; } + + public bool ReturnsValue { get; } + + public int InArgumentsCount { get; } + public int OutArgumentsCount { get; } + + public Function(BasicBlock[] blocks, string name, bool returnsValue, int inArgumentsCount, int outArgumentsCount) + { + Blocks = blocks; + Name = name; + ReturnsValue = returnsValue; + InArgumentsCount = inArgumentsCount; + OutArgumentsCount = outArgumentsCount; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/INode.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/INode.cs new file mode 100644 index 0000000..11f76e9 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/INode.cs @@ -0,0 +1,15 @@ +namespace Ryujinx.Graphics.Shader.IntermediateRepresentation +{ + interface INode + { + Operand Dest { get; set; } + + int DestsCount { get; } + int SourcesCount { get; } + + Operand GetDest(int index); + Operand GetSource(int index); + + void SetSource(int index, Operand operand); + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/Instruction.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/Instruction.cs new file mode 100644 index 0000000..9df9f15 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/Instruction.cs @@ -0,0 +1,197 @@ +using System; + +namespace Ryujinx.Graphics.Shader.IntermediateRepresentation +{ + [Flags] + enum Instruction + { + Absolute = 1, + Add, + AtomicAdd, + AtomicAnd, + AtomicCompareAndSwap, + AtomicMinS32, + AtomicMinU32, + AtomicMaxS32, + AtomicMaxU32, + AtomicOr, + AtomicSwap, + AtomicXor, + Ballot, + Barrier, + BitCount, + BitfieldExtractS32, + BitfieldExtractU32, + BitfieldInsert, + BitfieldReverse, + BitwiseAnd, + BitwiseExclusiveOr, + BitwiseNot, + BitwiseOr, + Branch, + BranchIfFalse, + BranchIfTrue, + Call, + Ceiling, + Clamp, + ClampU32, + Comment, + CompareEqual, + CompareGreater, + CompareGreaterOrEqual, + CompareGreaterOrEqualU32, + CompareGreaterU32, + CompareLess, + CompareLessOrEqual, + CompareLessOrEqualU32, + CompareLessU32, + CompareNotEqual, + ConditionalSelect, + ConvertFP32ToFP64, + ConvertFP64ToFP32, + ConvertFP32ToS32, + ConvertFP32ToU32, + ConvertFP64ToS32, + ConvertFP64ToU32, + ConvertS32ToFP32, + ConvertS32ToFP64, + ConvertU32ToFP32, + ConvertU32ToFP64, + Copy, + Cosine, + Ddx, + Ddy, + Discard, + Divide, + EmitVertex, + EndPrimitive, + ExponentB2, + FSIBegin, + FSIEnd, + FindLSB, + FindMSBS32, + FindMSBU32, + Floor, + FusedMultiplyAdd, + GroupMemoryBarrier, + ImageLoad, + ImageStore, + ImageAtomic, + IsNan, + Load, + Lod, + LogarithmB2, + LogicalAnd, + LogicalExclusiveOr, + LogicalNot, + LogicalOr, + LoopBreak, + LoopContinue, + MarkLabel, + Maximum, + MaximumU32, + MemoryBarrier, + Minimum, + MinimumU32, + Modulo, + Multiply, + MultiplyHighS32, + MultiplyHighU32, + Negate, + PackDouble2x32, + PackHalf2x16, + ReciprocalSquareRoot, + Return, + Round, + ShiftLeft, + ShiftRightS32, + ShiftRightU32, + Shuffle, + ShuffleDown, + ShuffleUp, + ShuffleXor, + Sine, + SquareRoot, + Store, + Subtract, + SwizzleAdd, + TextureSample, + TextureQuerySamples, + TextureQuerySize, + Truncate, + UnpackDouble2x32, + UnpackHalf2x16, + VectorExtract, + VoteAll, + VoteAllEqual, + VoteAny, + + Count, + + FP32 = 1 << 16, + FP64 = 1 << 17, + + Mask = 0xffff, + } + + static class InstructionExtensions + { + public static bool IsAtomic(this Instruction inst) + { + switch (inst & Instruction.Mask) + { + case Instruction.AtomicAdd: + case Instruction.AtomicAnd: + case Instruction.AtomicCompareAndSwap: + case Instruction.AtomicMaxS32: + case Instruction.AtomicMaxU32: + case Instruction.AtomicMinS32: + case Instruction.AtomicMinU32: + case Instruction.AtomicOr: + case Instruction.AtomicSwap: + case Instruction.AtomicXor: + return true; + } + + return false; + } + + public static bool IsComparison(this Instruction inst) + { + switch (inst & Instruction.Mask) + { + case Instruction.CompareEqual: + case Instruction.CompareGreater: + case Instruction.CompareGreaterOrEqual: + case Instruction.CompareGreaterOrEqualU32: + case Instruction.CompareGreaterU32: + case Instruction.CompareLess: + case Instruction.CompareLessOrEqual: + case Instruction.CompareLessOrEqualU32: + case Instruction.CompareLessU32: + case Instruction.CompareNotEqual: + return true; + } + + return false; + } + + public static bool IsTextureQuery(this Instruction inst) + { + inst &= Instruction.Mask; + return inst is Instruction.Lod or Instruction.TextureQuerySamples or Instruction.TextureQuerySize; + } + + public static bool IsImage(this Instruction inst) + { + inst &= Instruction.Mask; + return inst is Instruction.ImageAtomic or Instruction.ImageLoad or Instruction.ImageStore; + } + + public static bool IsImageStore(this Instruction inst) + { + inst &= Instruction.Mask; + return inst is Instruction.ImageAtomic or Instruction.ImageStore; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/IoVariable.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/IoVariable.cs new file mode 100644 index 0000000..e1bbead --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/IoVariable.cs @@ -0,0 +1,49 @@ +namespace Ryujinx.Graphics.Shader.IntermediateRepresentation +{ + enum IoVariable + { + Invalid, + + BackColorDiffuse, + BackColorSpecular, + BaseInstance, + BaseVertex, + ClipDistance, + CtaId, + DrawIndex, + FogCoord, + FragmentCoord, + FragmentOutputColor, + FragmentOutputDepth, + FrontColorDiffuse, + FrontColorSpecular, + FrontFacing, + GlobalId, + InstanceId, + InstanceIndex, + InvocationId, + Layer, + PatchVertices, + PointCoord, + PointSize, + Position, + PrimitiveId, + SubgroupEqMask, + SubgroupGeMask, + SubgroupGtMask, + SubgroupLaneId, + SubgroupLeMask, + SubgroupLtMask, + TessellationCoord, + TessellationLevelInner, + TessellationLevelOuter, + TextureCoord, + ThreadId, + ThreadKill, + UserDefined, + VertexId, + VertexIndex, + ViewportIndex, + ViewportMask, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/IrConsts.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/IrConsts.cs new file mode 100644 index 0000000..80b978a --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/IrConsts.cs @@ -0,0 +1,8 @@ +namespace Ryujinx.Graphics.Shader.IntermediateRepresentation +{ + static class IrConsts + { + public const int False = 0; + public const int True = -1; + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/Operand.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/Operand.cs new file mode 100644 index 0000000..bcd88ab --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/Operand.cs @@ -0,0 +1,79 @@ +using Ryujinx.Graphics.Shader.Decoders; +using System; +using System.Collections.Generic; + +namespace Ryujinx.Graphics.Shader.IntermediateRepresentation +{ + class Operand + { + private const int CbufSlotBits = 5; + private const int CbufSlotLsb = 32 - CbufSlotBits; + private const int CbufSlotMask = (1 << CbufSlotBits) - 1; + + public OperandType Type { get; } + + public int Value { get; } + + public INode AsgOp { get; set; } + + public HashSet UseOps { get; } + + private Operand() + { + UseOps = []; + } + + public Operand(OperandType type) : this() + { + Type = type; + } + + public Operand(OperandType type, int value) : this() + { + Type = type; + Value = value; + } + + public Operand(Register reg) : this() + { + Type = OperandType.Register; + Value = PackRegInfo(reg.Index, reg.Type); + } + + public Operand(int slot, int offset) : this() + { + Type = OperandType.ConstantBuffer; + Value = PackCbufInfo(slot, offset); + } + + private static int PackCbufInfo(int slot, int offset) + { + return (slot << CbufSlotLsb) | offset; + } + + private static int PackRegInfo(int index, RegisterType type) + { + return ((int)type << 24) | index; + } + + public int GetCbufSlot() + { + return (Value >> CbufSlotLsb) & CbufSlotMask; + } + + public int GetCbufOffset() + { + return Value & ~(CbufSlotMask << CbufSlotLsb); + } + + public Register GetRegister() + { + return new Register(Value & 0xffffff, (RegisterType)(Value >> 24)); + } + + public float AsFloat() + { + return BitConverter.Int32BitsToSingle(Value); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/OperandHelper.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/OperandHelper.cs new file mode 100644 index 0000000..fa540ad --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/OperandHelper.cs @@ -0,0 +1,62 @@ +using Ryujinx.Graphics.Shader.Decoders; +using System; + +namespace Ryujinx.Graphics.Shader.IntermediateRepresentation +{ + static class OperandHelper + { + public static Operand Argument(int value) + { + return new Operand(OperandType.Argument, value); + } + + public static Operand Cbuf(int slot, int offset) + { + return new Operand(slot, offset); + } + + public static Operand Const(int value) + { + return new Operand(OperandType.Constant, value); + } + + public static Operand ConstF(float value) + { + return new Operand(OperandType.Constant, BitConverter.SingleToInt32Bits(value)); + } + + public static Operand Label() + { + return new Operand(OperandType.Label); + } + + public static Operand Local() + { + return new Operand(OperandType.LocalVariable); + } + + public static Operand Register(int index, RegisterType type) + { + return Register(new Register(index, type)); + } + + public static Operand Register(Register reg) + { + if (reg.IsRZ) + { + return Const(0); + } + else if (reg.IsPT) + { + return Const(IrConsts.True); + } + + return new Operand(reg); + } + + public static Operand Undef() + { + return new Operand(OperandType.Undefined); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/OperandType.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/OperandType.cs new file mode 100644 index 0000000..36c2f46 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/OperandType.cs @@ -0,0 +1,13 @@ +namespace Ryujinx.Graphics.Shader.IntermediateRepresentation +{ + enum OperandType + { + Argument, + Constant, + ConstantBuffer, + Label, + LocalVariable, + Register, + Undefined, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/Operation.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/Operation.cs new file mode 100644 index 0000000..4ab9491 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/Operation.cs @@ -0,0 +1,294 @@ +using System; +using System.Diagnostics; + +namespace Ryujinx.Graphics.Shader.IntermediateRepresentation +{ + class Operation : INode + { + public Instruction Inst { get; private set; } + public StorageKind StorageKind { get; } + + public bool ForcePrecise { get; set; } + + private Operand[] _dests; + + public Operand Dest + { + get + { + return _dests.Length != 0 ? _dests[0] : null; + } + set + { + if (value != null) + { + if (value.Type == OperandType.LocalVariable) + { + value.AsgOp = this; + } + + _dests = [value]; + } + else + { + _dests = []; + } + } + } + + public int DestsCount => _dests.Length; + + private Operand[] _sources; + + public int SourcesCount => _sources.Length; + + public int Index { get; } + + private Operation(Operand[] sources) + { + // The array may be modified externally, so we store a copy. + _sources = (Operand[])sources.Clone(); + + for (int index = 0; index < _sources.Length; index++) + { + Operand source = _sources[index]; + + if (source.Type == OperandType.LocalVariable) + { + source.UseOps.Add(this); + } + } + } + + public Operation(Instruction inst, int index, Operand[] dests, Operand[] sources) : this(sources) + { + Inst = inst; + Index = index; + + if (dests != null) + { + // The array may be modified externally, so we store a copy. + _dests = (Operand[])dests.Clone(); + + for (int dstIndex = 0; dstIndex < dests.Length; dstIndex++) + { + Operand dest = dests[dstIndex]; + + if (dest != null && dest.Type == OperandType.LocalVariable) + { + dest.AsgOp = this; + } + } + } + else + { + _dests = []; + } + } + + public Operation(Instruction inst, Operand dest, params Operand[] sources) : this(sources) + { + Inst = inst; + + if (dest != null) + { + dest.AsgOp = this; + + _dests = [dest]; + } + else + { + _dests = []; + } + } + + public Operation(Instruction inst, StorageKind storageKind, Operand dest, params Operand[] sources) : this(sources) + { + Inst = inst; + StorageKind = storageKind; + + if (dest != null) + { + dest.AsgOp = this; + + _dests = [dest]; + } + else + { + _dests = []; + } + } + + public Operation(Instruction inst, int index, Operand dest, params Operand[] sources) : this(inst, dest, sources) + { + Index = index; + } + + public void AppendDests(Operand[] operands) + { + int startIndex = _dests.Length; + + Array.Resize(ref _dests, startIndex + operands.Length); + + for (int index = 0; index < operands.Length; index++) + { + Operand dest = operands[index]; + + if (dest != null && dest.Type == OperandType.LocalVariable) + { + Debug.Assert(dest.AsgOp == null); + dest.AsgOp = this; + } + + _dests[startIndex + index] = dest; + } + } + + public void AppendSources(Operand[] operands) + { + int startIndex = _sources.Length; + + Array.Resize(ref _sources, startIndex + operands.Length); + + for (int index = 0; index < operands.Length; index++) + { + Operand source = operands[index]; + + if (source.Type == OperandType.LocalVariable) + { + source.UseOps.Add(this); + } + + _sources[startIndex + index] = source; + } + } + + public Operand GetDest(int index) + { + return _dests[index]; + } + + public Operand GetSource(int index) + { + return _sources[index]; + } + + public void SetDest(int index, Operand dest) + { + Operand oldDest = _dests[index]; + + if (oldDest != null && oldDest.Type == OperandType.LocalVariable) + { + oldDest.AsgOp = null; + } + + if (dest != null && dest.Type == OperandType.LocalVariable) + { + dest.AsgOp = this; + } + + _dests[index] = dest; + } + + public void SetSource(int index, Operand source) + { + Operand oldSrc = _sources[index]; + + if (oldSrc != null && oldSrc.Type == OperandType.LocalVariable) + { + oldSrc.UseOps.Remove(this); + } + + if (source != null && source.Type == OperandType.LocalVariable) + { + source.UseOps.Add(this); + } + + _sources[index] = source; + } + + public void InsertSource(int index, Operand source) + { + Operand[] newSources = new Operand[_sources.Length + 1]; + + Array.Copy(_sources, 0, newSources, 0, index); + Array.Copy(_sources, index, newSources, index + 1, _sources.Length - index); + + newSources[index] = source; + + if (source != null && source.Type == OperandType.LocalVariable) + { + source.UseOps.Add(this); + } + + _sources = newSources; + } + + protected void RemoveSource(int index) + { + SetSource(index, null); + + Operand[] newSources = new Operand[_sources.Length - 1]; + + Array.Copy(_sources, 0, newSources, 0, index); + Array.Copy(_sources, index + 1, newSources, index, _sources.Length - (index + 1)); + + _sources = newSources; + } + + public void TurnIntoCopy(Operand source) + { + TurnInto(Instruction.Copy, source); + } + + public void TurnInto(Instruction newInst, Operand source) + { + Inst = newInst; + + foreach (Operand oldSrc in _sources) + { + if (oldSrc != null && oldSrc.Type == OperandType.LocalVariable) + { + oldSrc.UseOps.Remove(this); + } + } + + if (source.Type == OperandType.LocalVariable) + { + source.UseOps.Add(this); + } + + _sources = [source]; + } + + public void TurnDoubleIntoFloat() + { + if ((Inst & ~Instruction.Mask) == Instruction.FP64) + { + Inst = (Inst & Instruction.Mask) | Instruction.FP32; + } + else + { + switch (Inst) + { + case Instruction.ConvertFP32ToFP64: + case Instruction.ConvertFP64ToFP32: + Inst = Instruction.Copy; + break; + case Instruction.ConvertFP64ToS32: + Inst = Instruction.ConvertFP32ToS32; + break; + case Instruction.ConvertFP64ToU32: + Inst = Instruction.ConvertFP32ToU32; + break; + case Instruction.ConvertS32ToFP64: + Inst = Instruction.ConvertS32ToFP32; + break; + case Instruction.ConvertU32ToFP64: + Inst = Instruction.ConvertU32ToFP32; + break; + } + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/PhiNode.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/PhiNode.cs new file mode 100644 index 0000000..719ce2a --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/PhiNode.cs @@ -0,0 +1,104 @@ +using System; +using System.Collections.Generic; + +namespace Ryujinx.Graphics.Shader.IntermediateRepresentation +{ + class PhiNode : INode + { + private Operand _dest; + + public Operand Dest + { + get => _dest; + set => _dest = AssignDest(value); + } + + public int DestsCount => _dest != null ? 1 : 0; + + private readonly HashSet _blocks; + + private class PhiSource + { + public BasicBlock Block { get; } + public Operand Operand { get; set; } + + public PhiSource(BasicBlock block, Operand operand) + { + Block = block; + Operand = operand; + } + } + + private readonly List _sources; + + public int SourcesCount => _sources.Count; + + public PhiNode(Operand dest) + { + _blocks = []; + + _sources = []; + + dest.AsgOp = this; + + Dest = dest; + } + + private Operand AssignDest(Operand dest) + { + if (dest != null && dest.Type == OperandType.LocalVariable) + { + dest.AsgOp = this; + } + + return dest; + } + + public void AddSource(BasicBlock block, Operand operand) + { + if (_blocks.Add(block)) + { + if (operand.Type == OperandType.LocalVariable) + { + operand.UseOps.Add(this); + } + + _sources.Add(new PhiSource(block, operand)); + } + } + + public Operand GetDest(int index) + { + ArgumentOutOfRangeException.ThrowIfNotEqual(index, 0); + + return _dest; + } + + public Operand GetSource(int index) + { + return _sources[index].Operand; + } + + public BasicBlock GetBlock(int index) + { + return _sources[index].Block; + } + + public void SetSource(int index, Operand source) + { + Operand oldSrc = _sources[index].Operand; + + if (oldSrc != null && oldSrc.Type == OperandType.LocalVariable) + { + oldSrc.UseOps.Remove(this); + } + + if (source.Type == OperandType.LocalVariable) + { + source.UseOps.Add(this); + } + + _sources[index].Operand = source; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/StorageKind.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/StorageKind.cs new file mode 100644 index 0000000..5be1180 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/StorageKind.cs @@ -0,0 +1,45 @@ +namespace Ryujinx.Graphics.Shader.IntermediateRepresentation +{ + enum StorageKind + { + None, + Input, + InputPerPatch, + Output, + OutputPerPatch, + ConstantBuffer, + StorageBuffer, + LocalMemory, + SharedMemory, + SharedMemory8, // TODO: Remove this and store type as a field on the Operation class itself. + SharedMemory16, // TODO: Remove this and store type as a field on the Operation class itself. + GlobalMemory, + GlobalMemoryS8, // TODO: Remove this and store type as a field on the Operation class itself. + GlobalMemoryS16, // TODO: Remove this and store type as a field on the Operation class itself. + GlobalMemoryU8, // TODO: Remove this and store type as a field on the Operation class itself. + GlobalMemoryU16, // TODO: Remove this and store type as a field on the Operation class itself. + } + + static class StorageKindExtensions + { + public static bool IsInputOrOutput(this StorageKind storageKind) + { + return storageKind is StorageKind.Input or + StorageKind.InputPerPatch or + StorageKind.Output or + StorageKind.OutputPerPatch; + } + + public static bool IsOutput(this StorageKind storageKind) + { + return storageKind is StorageKind.Output or + StorageKind.OutputPerPatch; + } + + public static bool IsPerPatch(this StorageKind storageKind) + { + return storageKind is StorageKind.InputPerPatch or + StorageKind.OutputPerPatch; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/TextureFlags.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/TextureFlags.cs new file mode 100644 index 0000000..56ab4b5 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/TextureFlags.cs @@ -0,0 +1,32 @@ +using System; + +namespace Ryujinx.Graphics.Shader.IntermediateRepresentation +{ + [Flags] + enum TextureFlags + { + None = 0, + Bindless = 1 << 0, + Gather = 1 << 1, + Derivatives = 1 << 2, + IntCoords = 1 << 3, + LodBias = 1 << 4, + LodLevel = 1 << 5, + Offset = 1 << 6, + Offsets = 1 << 7, + Coherent = 1 << 8, + + AtomicMask = 15 << 16, + + Add = 0 << 16, + Minimum = 1 << 16, + Maximum = 2 << 16, + Increment = 3 << 16, + Decrement = 4 << 16, + BitwiseAnd = 5 << 16, + BitwiseOr = 6 << 16, + BitwiseXor = 7 << 16, + Swap = 8 << 16, + CAS = 9 << 16, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/TextureOperation.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/TextureOperation.cs new file mode 100644 index 0000000..728b1b4 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/IntermediateRepresentation/TextureOperation.cs @@ -0,0 +1,69 @@ +namespace Ryujinx.Graphics.Shader.IntermediateRepresentation +{ + class TextureOperation : Operation + { + public const int DefaultCbufSlot = -1; + + public SamplerType Type { get; set; } + public TextureFormat Format { get; set; } + public TextureFlags Flags { get; private set; } + + public int Set { get; private set; } + public int Binding { get; private set; } + public int SamplerSet { get; private set; } + public int SamplerBinding { get; private set; } + + public TextureOperation( + Instruction inst, + SamplerType type, + TextureFormat format, + TextureFlags flags, + int set, + int binding, + int compIndex, + Operand[] dests, + Operand[] sources) : base(inst, compIndex, dests, sources) + { + Type = type; + Format = format; + Flags = flags; + Set = set; + Binding = binding; + SamplerSet = -1; + SamplerBinding = -1; + } + + public void TurnIntoArray(SetBindingPair setAndBinding) + { + Flags &= ~TextureFlags.Bindless; + Set = setAndBinding.SetIndex; + Binding = setAndBinding.Binding; + } + + public void TurnIntoArray(SetBindingPair textureSetAndBinding, SetBindingPair samplerSetAndBinding) + { + TurnIntoArray(textureSetAndBinding); + + SamplerSet = samplerSetAndBinding.SetIndex; + SamplerBinding = samplerSetAndBinding.Binding; + } + + public void SetBinding(SetBindingPair setAndBinding) + { + if ((Flags & TextureFlags.Bindless) != 0) + { + Flags &= ~TextureFlags.Bindless; + + RemoveSource(0); + } + + Set = setAndBinding.SetIndex; + Binding = setAndBinding.Binding; + } + + public void SetLodLevelFlag() + { + Flags |= TextureFlags.LodLevel; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/OutputTopology.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/OutputTopology.cs new file mode 100644 index 0000000..0a8634b --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/OutputTopology.cs @@ -0,0 +1,24 @@ +namespace Ryujinx.Graphics.Shader +{ + enum OutputTopology + { + PointList = 1, + LineStrip = 6, + TriangleStrip = 7, + } + + static class OutputTopologyExtensions + { + + extension(OutputTopology topology) + { + public string GlslString => topology switch + { + OutputTopology.LineStrip => "line_strip", + OutputTopology.PointList => "points", + OutputTopology.TriangleStrip => "triangle_strip", + _ => "points", + }; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/ResourceReservationCounts.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/ResourceReservationCounts.cs new file mode 100644 index 0000000..15870f3 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/ResourceReservationCounts.cs @@ -0,0 +1,22 @@ +using Ryujinx.Graphics.Shader.Translation; + +namespace Ryujinx.Graphics.Shader +{ + public readonly struct ResourceReservationCounts + { + public readonly int ReservedConstantBuffers { get; } + public readonly int ReservedStorageBuffers { get; } + public readonly int ReservedTextures { get; } + public readonly int ReservedImages { get; } + + public ResourceReservationCounts(bool isTransformFeedbackEmulated, bool vertexAsCompute) + { + ResourceReservations reservations = new(isTransformFeedbackEmulated, vertexAsCompute); + + ReservedConstantBuffers = reservations.ReservedConstantBuffers; + ReservedStorageBuffers = reservations.ReservedStorageBuffers; + ReservedTextures = reservations.ReservedTextures; + ReservedImages = reservations.ReservedImages; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Ryujinx.Graphics.Shader.csproj b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Ryujinx.Graphics.Shader.csproj new file mode 100644 index 0000000..f152502 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Ryujinx.Graphics.Shader.csproj @@ -0,0 +1,18 @@ + + + + $(DefaultItemExcludes);._* + + + + + + + + + + + + + + diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/SamplerType.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/SamplerType.cs new file mode 100644 index 0000000..197a674 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/SamplerType.cs @@ -0,0 +1,168 @@ +using Ryujinx.Graphics.Shader.Translation; +using System; + +namespace Ryujinx.Graphics.Shader +{ + [Flags] + public enum SamplerType + { + None = 0, + Texture1D, + TextureBuffer, + Texture2D, + Texture3D, + TextureCube, + + Mask = 0xff, + + Array = 1 << 8, + Multisample = 1 << 9, + Shadow = 1 << 10, + } + + static class SamplerTypeExtensions + { + extension(SamplerType type) + { + public int Dimensions => (type & SamplerType.Mask) switch + { + SamplerType.Texture1D => 1, + SamplerType.TextureBuffer => 1, + SamplerType.Texture2D => 2, + SamplerType.Texture3D => 3, + SamplerType.TextureCube => 3, + _ => throw new ArgumentException($"Invalid sampler type \"{type}\"."), + }; + + public string ShortTypeName + { + get + { + string typeName = (type & SamplerType.Mask) switch + { + SamplerType.Texture1D => "1d", + SamplerType.TextureBuffer => "b", + SamplerType.Texture2D => "2d", + SamplerType.Texture3D => "3d", + SamplerType.TextureCube => "cube", + _ => throw new ArgumentException($"Invalid sampler type \"{type}\"."), + }; + + if ((type & SamplerType.Multisample) != 0) + { + typeName += "ms"; + } + + if ((type & SamplerType.Array) != 0) + { + typeName += "a"; + } + + if ((type & SamplerType.Shadow) != 0) + { + typeName += "s"; + } + + return typeName; + } + } + + public string GlslSamplerTypeName + { + get + { + string typeName = (type & SamplerType.Mask) switch + { + SamplerType.None => "sampler", + SamplerType.Texture1D => "sampler1D", + SamplerType.TextureBuffer => "samplerBuffer", + SamplerType.Texture2D => "sampler2D", + SamplerType.Texture3D => "sampler3D", + SamplerType.TextureCube => "samplerCube", + _ => throw new ArgumentException($"Invalid sampler type \"{type}\"."), + }; + + if ((type & SamplerType.Multisample) != 0) + { + typeName += "MS"; + } + + if ((type & SamplerType.Array) != 0) + { + typeName += "Array"; + } + + if ((type & SamplerType.Shadow) != 0) + { + typeName += "Shadow"; + } + + return typeName; + } + } + + public string GlslTextureTypeName + { + get + { + string typeName = (type & SamplerType.Mask) switch + { + SamplerType.Texture1D => "texture1D", + SamplerType.TextureBuffer => "textureBuffer", + SamplerType.Texture2D => "texture2D", + SamplerType.Texture3D => "texture3D", + SamplerType.TextureCube => "textureCube", + _ => throw new ArgumentException($"Invalid texture type \"{type}\"."), + }; + + if ((type & SamplerType.Multisample) != 0) + { + typeName += "MS"; + } + + if ((type & SamplerType.Array) != 0) + { + typeName += "Array"; + } + + return typeName; + } + } + + public string GetGlslImageTypeName(AggregateType componentType) + { + string typeName = (type & SamplerType.Mask) switch + { + SamplerType.Texture1D => "image1D", + SamplerType.TextureBuffer => "imageBuffer", + SamplerType.Texture2D => "image2D", + SamplerType.Texture3D => "image3D", + SamplerType.TextureCube => "imageCube", + _ => throw new ArgumentException($"Invalid sampler type \"{type}\"."), + }; + + if ((type & SamplerType.Multisample) != 0) + { + typeName += "MS"; + } + + if ((type & SamplerType.Array) != 0) + { + typeName += "Array"; + } + + switch (componentType) + { + case AggregateType.U32: + typeName = 'u' + typeName; + break; + case AggregateType.S32: + typeName = 'i' + typeName; + break; + } + + return typeName; + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/SetBindingPair.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/SetBindingPair.cs new file mode 100644 index 0000000..782bb50 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/SetBindingPair.cs @@ -0,0 +1,4 @@ +namespace Ryujinx.Graphics.Shader +{ + public readonly record struct SetBindingPair(int SetIndex, int Binding); +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/ShaderProgram.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/ShaderProgram.cs new file mode 100644 index 0000000..59316eb --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/ShaderProgram.cs @@ -0,0 +1,35 @@ +using Ryujinx.Graphics.Shader.Translation; +using System; + +namespace Ryujinx.Graphics.Shader +{ + public class ShaderProgram + { + public ShaderProgramInfo Info { get; } + public TargetLanguage Language { get; } + + public string Code { get; private set; } + public byte[] BinaryCode { get; } + + private ShaderProgram(ShaderProgramInfo info, TargetLanguage language) + { + Info = info; + Language = language; + } + + public ShaderProgram(ShaderProgramInfo info, TargetLanguage language, string code) : this(info, language) + { + Code = code; + } + + public ShaderProgram(ShaderProgramInfo info, TargetLanguage language, byte[] binaryCode) : this(info, language) + { + BinaryCode = binaryCode; + } + + public void Prepend(string line) + { + Code = line + Environment.NewLine + Code; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/ShaderProgramInfo.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/ShaderProgramInfo.cs new file mode 100644 index 0000000..72c17fd --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/ShaderProgramInfo.cs @@ -0,0 +1,57 @@ +using System; +using System.Collections.ObjectModel; + +namespace Ryujinx.Graphics.Shader +{ + public class ShaderProgramInfo + { + public ReadOnlyCollection CBuffers { get; } + public ReadOnlyCollection SBuffers { get; } + public ReadOnlyCollection Textures { get; } + public ReadOnlyCollection Images { get; } + + public ShaderStage Stage { get; } + public int GeometryVerticesPerPrimitive { get; } + public int GeometryMaxOutputVertices { get; } + public int ThreadsPerInputPrimitive { get; } + public bool UsesFragCoord { get; } + public bool UsesInstanceId { get; } + public bool UsesDrawParameters { get; } + public bool UsesRtLayer { get; } + public byte ClipDistancesWritten { get; } + public int FragmentOutputMap { get; } + + public ShaderProgramInfo( + BufferDescriptor[] cBuffers, + BufferDescriptor[] sBuffers, + TextureDescriptor[] textures, + TextureDescriptor[] images, + ShaderStage stage, + int geometryVerticesPerPrimitive, + int geometryMaxOutputVertices, + int threadsPerInputPrimitive, + bool usesFragCoord, + bool usesInstanceId, + bool usesDrawParameters, + bool usesRtLayer, + byte clipDistancesWritten, + int fragmentOutputMap) + { + CBuffers = Array.AsReadOnly(cBuffers); + SBuffers = Array.AsReadOnly(sBuffers); + Textures = Array.AsReadOnly(textures); + Images = Array.AsReadOnly(images); + + Stage = stage; + GeometryVerticesPerPrimitive = geometryVerticesPerPrimitive; + GeometryMaxOutputVertices = geometryMaxOutputVertices; + ThreadsPerInputPrimitive = threadsPerInputPrimitive; + UsesFragCoord = usesFragCoord; + UsesInstanceId = usesInstanceId; + UsesDrawParameters = usesDrawParameters; + UsesRtLayer = usesRtLayer; + ClipDistancesWritten = clipDistancesWritten; + FragmentOutputMap = fragmentOutputMap; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/ShaderStage.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/ShaderStage.cs new file mode 100644 index 0000000..7925ab0 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/ShaderStage.cs @@ -0,0 +1,36 @@ +namespace Ryujinx.Graphics.Shader +{ + public enum ShaderStage : byte + { + Compute, + Vertex, + TessellationControl, + TessellationEvaluation, + Geometry, + Fragment, + + Count, + } + + public static class ShaderStageExtensions + { + extension(ShaderStage shaderStage) + { + /// + /// Checks if the shader stage supports render scale. + /// + public bool SupportsRenderScale => + shaderStage is ShaderStage.Vertex or ShaderStage.Fragment or ShaderStage.Compute; + + /// + /// Checks if the shader stage is vertex, tessellation or geometry. + /// + public bool IsVtg => + shaderStage is ShaderStage.Vertex or + ShaderStage.TessellationControl or + ShaderStage.TessellationEvaluation or + ShaderStage.Geometry; + } + + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstAssignment.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstAssignment.cs new file mode 100644 index 0000000..1f5969d --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstAssignment.cs @@ -0,0 +1,35 @@ +using static Ryujinx.Graphics.Shader.StructuredIr.AstHelper; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + class AstAssignment : AstNode + { + public IAstNode Destination { get; } + + private IAstNode _source; + + public IAstNode Source + { + get + { + return _source; + } + set + { + RemoveUse(_source, this); + + AddUse(value, this); + + _source = value; + } + } + + public AstAssignment(IAstNode destination, IAstNode source) + { + Destination = destination; + Source = source; + + AddDef(destination, this); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstBlock.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstBlock.cs new file mode 100644 index 0000000..b2d7c2e --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstBlock.cs @@ -0,0 +1,117 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System; +using System.Collections; +using System.Collections.Generic; + +using static Ryujinx.Graphics.Shader.StructuredIr.AstHelper; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + class AstBlock : AstNode, IEnumerable + { + public AstBlockType Type { get; private set; } + + private IAstNode _condition; + + public IAstNode Condition + { + get + { + return _condition; + } + set + { + RemoveUse(_condition, this); + + AddUse(value, this); + + _condition = value; + } + } + + private readonly LinkedList _nodes; + + public IAstNode First => _nodes.First?.Value; + public IAstNode Last => _nodes.Last?.Value; + + public int Count => _nodes.Count; + + public AstBlock(AstBlockType type, IAstNode condition = null) + { + Type = type; + Condition = condition; + + _nodes = []; + } + + public void Add(IAstNode node) + { + Add(node, _nodes.AddLast(node)); + } + + public void AddFirst(IAstNode node) + { + Add(node, _nodes.AddFirst(node)); + } + + public void AddBefore(IAstNode next, IAstNode node) + { + Add(node, _nodes.AddBefore(next.LLNode, node)); + } + + public void AddAfter(IAstNode prev, IAstNode node) + { + Add(node, _nodes.AddAfter(prev.LLNode, node)); + } + + private void Add(IAstNode node, LinkedListNode newNode) + { + if (node.Parent != null) + { + throw new ArgumentException("Node already belongs to a block."); + } + + node.Parent = this; + node.LLNode = newNode; + } + + public void Remove(IAstNode node) + { + _nodes.Remove(node.LLNode); + + node.Parent = null; + node.LLNode = null; + } + + public void AndCondition(IAstNode cond) + { + Condition = new AstOperation(Instruction.LogicalAnd, Condition, cond); + } + + public void OrCondition(IAstNode cond) + { + Condition = new AstOperation(Instruction.LogicalOr, Condition, cond); + } + public void TurnIntoIf(IAstNode cond) + { + Condition = cond; + + Type = AstBlockType.If; + } + + public void TurnIntoElseIf() + { + Type = AstBlockType.ElseIf; + } + + public IEnumerator GetEnumerator() + { + return _nodes.GetEnumerator(); + } + + IEnumerator IEnumerable.GetEnumerator() + { + return GetEnumerator(); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstBlockType.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstBlockType.cs new file mode 100644 index 0000000..47d94ab --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstBlockType.cs @@ -0,0 +1,12 @@ +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + enum AstBlockType + { + DoWhile, + If, + Else, + ElseIf, + Main, + While, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstBlockVisitor.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstBlockVisitor.cs new file mode 100644 index 0000000..afeb88a --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstBlockVisitor.cs @@ -0,0 +1,68 @@ +using System; +using System.Collections.Generic; + +using static Ryujinx.Graphics.Shader.StructuredIr.AstHelper; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + class AstBlockVisitor + { + public AstBlock Block { get; private set; } + + public class BlockVisitationEventArgs : EventArgs + { + public AstBlock Block { get; } + + public BlockVisitationEventArgs(AstBlock block) + { + Block = block; + } + } + + public event EventHandler BlockEntered; + public event EventHandler BlockLeft; + + public AstBlockVisitor(AstBlock mainBlock) + { + Block = mainBlock; + } + + public IEnumerable Visit() + { + IAstNode node = Block.First; + + while (node != null) + { + // We reached a child block, visit the nodes inside. + while (node is AstBlock childBlock) + { + Block = childBlock; + + node = childBlock.First; + + BlockEntered?.Invoke(this, new BlockVisitationEventArgs(Block)); + } + + // Node may be null, if the block is empty. + if (node != null) + { + IAstNode next = Next(node); + + yield return node; + + node = next; + } + + // We reached the end of the list, go up on tree to the parent blocks. + while (node == null && Block.Type != AstBlockType.Main) + { + BlockLeft?.Invoke(this, new BlockVisitationEventArgs(Block)); + + node = Next(Block); + + Block = Block.Parent; + } + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstComment.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstComment.cs new file mode 100644 index 0000000..7002c2e --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstComment.cs @@ -0,0 +1,12 @@ +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + class AstComment : AstNode + { + public string Comment { get; } + + public AstComment(string comment) + { + Comment = comment; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstHelper.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstHelper.cs new file mode 100644 index 0000000..df6fb7d --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstHelper.cs @@ -0,0 +1,75 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + static class AstHelper + { + public static void AddUse(IAstNode node, IAstNode parent) + { + if (node is AstOperand operand && operand.Type == OperandType.LocalVariable) + { + operand.Uses.Add(parent); + } + } + + public static void AddDef(IAstNode node, IAstNode parent) + { + if (node is AstOperand operand && operand.Type == OperandType.LocalVariable) + { + operand.Defs.Add(parent); + } + } + + public static void RemoveUse(IAstNode node, IAstNode parent) + { + if (node is AstOperand operand && operand.Type == OperandType.LocalVariable) + { + operand.Uses.Remove(parent); + } + } + + public static void RemoveDef(IAstNode node, IAstNode parent) + { + if (node is AstOperand operand && operand.Type == OperandType.LocalVariable) + { + operand.Defs.Remove(parent); + } + } + + public static AstAssignment Assign(IAstNode destination, IAstNode source) + { + return new AstAssignment(destination, source); + } + + public static AstOperand Const(int value) + { + return new AstOperand(OperandType.Constant, value); + } + + public static AstOperand Local(AggregateType type) + { + AstOperand local = new(OperandType.LocalVariable) + { + VarType = type, + }; + + return local; + } + + public static IAstNode InverseCond(IAstNode cond) + { + return new AstOperation(Instruction.LogicalNot, cond); + } + + public static IAstNode Next(IAstNode node) + { + return node.LLNode.Next?.Value; + } + + public static IAstNode Previous(IAstNode node) + { + return node.LLNode.Previous?.Value; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstNode.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstNode.cs new file mode 100644 index 0000000..70fdbd4 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstNode.cs @@ -0,0 +1,11 @@ +using System.Collections.Generic; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + class AstNode : IAstNode + { + public AstBlock Parent { get; set; } + + public LinkedListNode LLNode { get; set; } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstOperand.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstOperand.cs new file mode 100644 index 0000000..370c8cc --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstOperand.cs @@ -0,0 +1,38 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using System.Collections.Generic; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + class AstOperand : AstNode + { + public HashSet Defs { get; } + public HashSet Uses { get; } + + public OperandType Type { get; } + + public AggregateType VarType { get; set; } + + public int Value { get; } + + private AstOperand() + { + Defs = []; + Uses = []; + + VarType = AggregateType.S32; + } + + public AstOperand(Operand operand) : this() + { + Type = operand.Type; + Value = operand.Value; + } + + public AstOperand(OperandType type, int value = 0) : this() + { + Type = type; + Value = value; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstOperation.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstOperation.cs new file mode 100644 index 0000000..2095cf2 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstOperation.cs @@ -0,0 +1,94 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using System.Numerics; + +using static Ryujinx.Graphics.Shader.StructuredIr.AstHelper; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + class AstOperation : AstNode + { + public Instruction Inst { get; } + public StorageKind StorageKind { get; } + public bool ForcePrecise { get; } + + public int Index { get; } + + private readonly IAstNode[] _sources; + + public int SourcesCount => _sources.Length; + + public AstOperation(Instruction inst, StorageKind storageKind, bool forcePrecise, IAstNode[] sources, int sourcesCount) + { + Inst = inst; + StorageKind = storageKind; + ForcePrecise = forcePrecise; + _sources = sources; + + for (int index = 0; index < sources.Length; index++) + { + if (index < sourcesCount) + { + AddUse(sources[index], this); + } + else + { + AddDef(sources[index], this); + } + } + + Index = 0; + } + + public AstOperation( + Instruction inst, + StorageKind storageKind, + bool forcePrecise, + int index, + IAstNode[] sources, + int sourcesCount) : this(inst, storageKind, forcePrecise, sources, sourcesCount) + { + Index = index; + } + + public AstOperation(Instruction inst, params IAstNode[] sources) : this(inst, StorageKind.None, false, sources, sources.Length) + { + } + + public IAstNode GetSource(int index) + { + return _sources[index]; + } + + public void SetSource(int index, IAstNode source) + { + RemoveUse(_sources[index], this); + + AddUse(source, this); + + _sources[index] = source; + } + + public AggregateType GetVectorType(AggregateType scalarType) + { + int componentsCount = BitOperations.PopCount((uint)Index); + + AggregateType type = scalarType; + + switch (componentsCount) + { + case 2: + type |= AggregateType.Vector2; + break; + case 3: + type |= AggregateType.Vector3; + break; + case 4: + type |= AggregateType.Vector4; + break; + } + + return type; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstOptimizer.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstOptimizer.cs new file mode 100644 index 0000000..4a267b6 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstOptimizer.cs @@ -0,0 +1,196 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System.Collections.Generic; +using System.Linq; + +using static Ryujinx.Graphics.Shader.StructuredIr.AstHelper; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + static class AstOptimizer + { + public static void Optimize(StructuredProgramContext context) + { + AstBlock mainBlock = context.CurrentFunction.MainBlock; + + // When debug mode is enabled, we disable expression propagation + // (this makes comparison with the disassembly easier). + if (!context.DebugMode) + { + AstBlockVisitor visitor = new(mainBlock); + + foreach (IAstNode node in visitor.Visit()) + { + if (node is AstAssignment assignment && assignment.Destination is AstOperand propVar) + { + // TOTK (2026-10-16): an atomic / call / image atomic whose result is unused (Maxwell RED, ATOM with + // an RZ destination) stays a statement in place: propagating it would move the side effect to its + // use, and removing the assignment removed the operation itself (the reduction kernel c7830fac + // lost its atomicMin on 2026-10-13; every translation ran in DebugMode since, to avoid this). + bool sideEffects = HasSideEffects(assignment.Source); + bool isWorthPropagating = !sideEffects && (propVar.Uses.Count == 1 || IsWorthPropagating(assignment.Source)); + + // TOTK: SPIR-V passes call arguments by pointer to a variable (StructuredProgram copies them + // into temporaries); folding an expression into a call argument breaks GenerateCall. + bool usedByCall = propVar.Uses.Any(u => u is AstOperation op && op.Inst == Instruction.Call); + + if (propVar.Defs.Count == 1 && isWorthPropagating && !usedByCall) + { + PropagateExpression(propVar, assignment.Source); + } + + if (propVar.Type == OperandType.LocalVariable && propVar.Uses.Count == 0 && !sideEffects) + { + visitor.Block.Remove(assignment); + + context.CurrentFunction.Locals.Remove(propVar); + } + } + } + } + + RemoveEmptyBlocks(mainBlock); + } + + private static bool HasSideEffects(IAstNode source) + { + if (source is AstOperation op) + { + switch (op.Inst & Instruction.Mask) + { + case Instruction.AtomicAdd: + case Instruction.AtomicAnd: + case Instruction.AtomicCompareAndSwap: + case Instruction.AtomicMaxS32: + case Instruction.AtomicMaxU32: + case Instruction.AtomicMinS32: + case Instruction.AtomicMinU32: + case Instruction.AtomicOr: + case Instruction.AtomicSwap: + case Instruction.AtomicXor: + case Instruction.Call: + case Instruction.ImageAtomic: + return true; + } + } + + return false; + } + + private static bool IsWorthPropagating(IAstNode source) + { + if (source is not AstOperation srcOp) + { + return false; + } + + if (!InstructionInfo.IsUnary(srcOp.Inst)) + { + return false; + } + + return srcOp.GetSource(0) is AstOperand || srcOp.Inst == Instruction.Copy; + } + + private static void PropagateExpression(AstOperand propVar, IAstNode source) + { + IAstNode[] uses = propVar.Uses.ToArray(); + + foreach (IAstNode useNode in uses) + { + if (useNode is AstBlock useBlock) + { + useBlock.Condition = source; + } + else if (useNode is AstOperation useOperation) + { + for (int srcIndex = 0; srcIndex < useOperation.SourcesCount; srcIndex++) + { + if (useOperation.GetSource(srcIndex) == propVar) + { + useOperation.SetSource(srcIndex, source); + } + } + } + else if (useNode is AstAssignment useAssignment) + { + useAssignment.Source = source; + } + } + } + + private const bool ElseIfFlattening = false; + + private static void RemoveEmptyBlocks(AstBlock mainBlock) + { + Queue pending = new(); + + pending.Enqueue(mainBlock); + + while (pending.TryDequeue(out AstBlock block)) + { + foreach (IAstNode node in block) + { + if (node is AstBlock childBlock) + { + pending.Enqueue(childBlock); + } + } + + AstBlock parent = block.Parent; + + if (parent == null) + { + continue; + } + + AstBlock nextBlock = Next(block) as AstBlock; + + bool hasElse = nextBlock != null && nextBlock.Type == AstBlockType.Else; + + bool isIf = block.Type == AstBlockType.If; + + if (block.Count == 0) + { + if (isIf) + { + if (hasElse) + { + nextBlock.TurnIntoIf(InverseCond(block.Condition)); + } + + parent.Remove(block); + } + else if (block.Type == AstBlockType.Else) + { + parent.Remove(block); + } + } + // TOTK (2026-10-16): the else-if flattening (`else { if (c) A else B }` -> `else if (c) A else B`) is + // disabled. The SPIR-V backend gives the else-if the same merge block as the if before it, and + // SPIRV-Cross then drops the trailing else of the chain: TOTK's fragment shader 1870878a lost three + // loops with an atomic max (a feedback write) and a discard (scripts/diag/msl_invariants.py found 12 + // variants losing atomics and 83 changing discards). The nested form is what DebugMode always produced, + // because its comment nodes keep every block non-empty and this branch never ran. + else if (ElseIfFlattening && isIf && parent.Type == AstBlockType.Else && parent.Count == (hasElse ? 2 : 1)) + { + AstBlock parentOfParent = parent.Parent; + + parent.Remove(block); + + parentOfParent.AddAfter(parent, block); + + if (hasElse) + { + parent.Remove(nextBlock); + + parentOfParent.AddAfter(block, nextBlock); + } + + parentOfParent.Remove(parent); + + block.TurnIntoElseIf(); + } + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstTextureOperation.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstTextureOperation.cs new file mode 100644 index 0000000..f136166 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/AstTextureOperation.cs @@ -0,0 +1,49 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + class AstTextureOperation : AstOperation + { + public SamplerType Type { get; } + public TextureFormat Format { get; } + public TextureFlags Flags { get; } + + public int Set { get; } + public int Binding { get; } + public int SamplerSet { get; } + public int SamplerBinding { get; } + + public bool IsSeparate => SamplerBinding >= 0; + + public AstTextureOperation( + Instruction inst, + SamplerType type, + TextureFormat format, + TextureFlags flags, + int set, + int binding, + int samplerSet, + int samplerBinding, + int index, + params IAstNode[] sources) : base(inst, StorageKind.None, false, index, sources, sources.Length) + { + Type = type; + Format = format; + Flags = flags; + Set = set; + Binding = binding; + SamplerSet = samplerSet; + SamplerBinding = samplerBinding; + } + + public SetBindingPair GetTextureSetAndBinding() + { + return new SetBindingPair(Set, Binding); + } + + public SetBindingPair GetSamplerSetAndBinding() + { + return new SetBindingPair(SamplerSet, SamplerBinding); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/BufferDefinition.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/BufferDefinition.cs new file mode 100644 index 0000000..14cc1af --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/BufferDefinition.cs @@ -0,0 +1,20 @@ +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + readonly struct BufferDefinition + { + public BufferLayout Layout { get; } + public int Set { get; } + public int Binding { get; } + public string Name { get; } + public StructureType Type { get; } + + public BufferDefinition(BufferLayout layout, int set, int binding, string name, StructureType type) + { + Layout = layout; + Set = set; + Binding = binding; + Name = name; + Type = type; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/BufferLayout.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/BufferLayout.cs new file mode 100644 index 0000000..9c941a2 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/BufferLayout.cs @@ -0,0 +1,8 @@ +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + enum BufferLayout + { + Std140, + Std430, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/GotoElimination.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/GotoElimination.cs new file mode 100644 index 0000000..f572202 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/GotoElimination.cs @@ -0,0 +1,458 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System; +using System.Collections.Generic; +using static Ryujinx.Graphics.Shader.StructuredIr.AstHelper; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + static class GotoElimination + { + // This is a modified version of the algorithm presented on the paper + // "Taming Control Flow: A Structured Approach to Eliminating Goto Statements". + public static void Eliminate(GotoStatement[] gotos) + { + for (int index = gotos.Length - 1; index >= 0; index--) + { + GotoStatement stmt = gotos[index]; + + AstBlock gBlock = ParentBlock(stmt.Goto); + AstBlock lBlock = ParentBlock(stmt.Label); + + int gLevel = Level(gBlock); + int lLevel = Level(lBlock); + + if (IndirectlyRelated(gBlock, lBlock, gLevel, lLevel)) + { + AstBlock drBlock = gBlock; + + int drLevel = gLevel; + + do + { + drBlock = drBlock.Parent; + + drLevel--; + } + while (!DirectlyRelated(drBlock, lBlock, drLevel, lLevel)); + + MoveOutward(stmt, gLevel, drLevel); + + gBlock = drBlock; + gLevel = drLevel; + + if (Previous(stmt.Goto) is AstBlock elseBlock && elseBlock.Type == AstBlockType.Else) + { + // It's possible that the label was enclosed inside an else block, + // in this case we need to update the block and level. + // We also need to set the IsLoop for the case when the label is + // now before the goto, due to the newly introduced else block. + lBlock = ParentBlock(stmt.Label); + + lLevel = Level(lBlock); + + if (!IndirectlyRelated(elseBlock, lBlock, gLevel + 1, lLevel)) + { + stmt.IsLoop = true; + } + } + } + + if (DirectlyRelated(gBlock, lBlock, gLevel, lLevel)) + { + if (gLevel > lLevel) + { + MoveOutward(stmt, gLevel, lLevel); + } + else + { + if (stmt.IsLoop) + { + Lift(stmt); + } + + MoveInward(stmt); + } + } + + gBlock = ParentBlock(stmt.Goto); + + if (stmt.IsLoop) + { + EncloseDoWhile(stmt, gBlock, stmt.Label); + } + else + { + Enclose(gBlock, AstBlockType.If, stmt.Condition, Next(stmt.Goto), stmt.Label); + } + + gBlock.Remove(stmt.Goto); + } + } + + private static bool IndirectlyRelated(AstBlock lBlock, AstBlock rBlock, int lLevel, int rlevel) + { + return !(lBlock == rBlock || DirectlyRelated(lBlock, rBlock, lLevel, rlevel)); + } + + private static bool DirectlyRelated(AstBlock lBlock, AstBlock rBlock, int lLevel, int rLevel) + { + // If the levels are equal, they can be either siblings or indirectly related. + if (lLevel == rLevel) + { + return false; + } + + AstNode block; + AstNode other; + + int blockLvl, otherLvl; + + if (lLevel > rLevel) + { + block = lBlock; + blockLvl = lLevel; + other = rBlock; + otherLvl = rLevel; + } + else /* if (rLevel > lLevel) */ + { + block = rBlock; + blockLvl = rLevel; + other = lBlock; + otherLvl = lLevel; + } + + while (blockLvl >= otherLvl) + { + if (block == other) + { + return true; + } + + block = block.Parent; + + blockLvl--; + } + + return false; + } + + private static void Lift(GotoStatement stmt) + { + AstBlock block = ParentBlock(stmt.Goto); + + AstBlock[] path = BackwardsPath(block, ParentBlock(stmt.Label)); + + AstBlock loopFirstStmt = path[^1]; + + if (loopFirstStmt.Type == AstBlockType.Else) + { + loopFirstStmt = Previous(loopFirstStmt) as AstBlock; + + if (loopFirstStmt == null || loopFirstStmt.Type != AstBlockType.If) + { + throw new InvalidOperationException("Found an else without a matching if."); + } + } + + AstBlock newBlock = EncloseDoWhile(stmt, block, loopFirstStmt); + + block.Remove(stmt.Goto); + + newBlock.AddFirst(stmt.Goto); + + stmt.IsLoop = false; + } + + private static void MoveOutward(GotoStatement stmt, int gLevel, int lLevel) + { + AstBlock origin = ParentBlock(stmt.Goto); + + AstBlock block = origin; + + // Check if a loop is enclosing the goto, and the block that is + // directly related to the label is above the loop block. + // In that case, we need to introduce a break to get out of the loop. + AstBlock loopBlock = origin; + + int loopLevel = gLevel; + + while (loopLevel > lLevel) + { + AstBlock child = loopBlock; + + loopBlock = loopBlock.Parent; + + loopLevel--; + + if (child.Type == AstBlockType.DoWhile) + { + EncloseSingleInst(stmt, Instruction.LoopBreak); + + block.Remove(stmt.Goto); + + loopBlock.AddAfter(child, stmt.Goto); + + block = loopBlock; + gLevel = loopLevel; + } + } + + // Insert ifs to skip the parts that shouldn't be executed due to the goto. + bool tryInsertElse = stmt.IsUnconditional && origin.Type == AstBlockType.If; + + while (gLevel > lLevel) + { + Enclose(block, AstBlockType.If, stmt.Condition, Next(stmt.Goto)); + + block.Remove(stmt.Goto); + + AstBlock child = block; + + // We can't move the goto in the middle of a if and a else block, in + // this case we need to move it after the else. + // IsLoop may need to be updated if the label is inside the else, as + // introducing a loop is the only way to ensure the else will be executed. + if (Next(child) is AstBlock elseBlock && elseBlock.Type == AstBlockType.Else) + { + child = elseBlock; + } + + block = block.Parent; + + block.AddAfter(child, stmt.Goto); + + gLevel--; + + if (tryInsertElse && child == origin) + { + AstBlock lBlock = ParentBlock(stmt.Label); + + IAstNode last = block == lBlock && !stmt.IsLoop ? stmt.Label : null; + + AstBlock newBlock = Enclose(block, AstBlockType.Else, null, Next(stmt.Goto), last); + + if (newBlock != null) + { + block.Remove(stmt.Goto); + + block.AddAfter(newBlock, stmt.Goto); + } + } + } + } + + private static void MoveInward(GotoStatement stmt) + { + AstBlock block = ParentBlock(stmt.Goto); + + AstBlock[] path = BackwardsPath(block, ParentBlock(stmt.Label)); + + for (int index = path.Length - 1; index >= 0; index--) + { + AstBlock child = path[index]; + AstBlock last = child; + + if (child.Type == AstBlockType.If) + { + // Modify the if condition to allow it to be entered by the goto. + if (!ContainsCondComb(child.Condition, Instruction.LogicalOr, stmt.Condition)) + { + child.OrCondition(stmt.Condition); + } + } + else if (child.Type == AstBlockType.Else) + { + // Modify the matching if condition to force the else to be entered by the goto. + if (Previous(child) is not AstBlock ifBlock || ifBlock.Type != AstBlockType.If) + { + throw new InvalidOperationException("Found an else without a matching if."); + } + + IAstNode cond = InverseCond(stmt.Condition); + + if (!ContainsCondComb(ifBlock.Condition, Instruction.LogicalAnd, cond)) + { + ifBlock.AndCondition(cond); + } + + last = ifBlock; + } + + Enclose(block, AstBlockType.If, stmt.Condition, Next(stmt.Goto), last); + + block.Remove(stmt.Goto); + + child.AddFirst(stmt.Goto); + + block = child; + } + } + + private static bool ContainsCondComb(IAstNode node, Instruction inst, IAstNode newCond) + { + while (node is AstOperation operation && operation.SourcesCount == 2) + { + if (operation.Inst == inst && IsSameCond(operation.GetSource(1), newCond)) + { + return true; + } + + node = operation.GetSource(0); + } + + return false; + } + + private static AstBlock EncloseDoWhile(GotoStatement stmt, AstBlock block, IAstNode first) + { + if (block.Type == AstBlockType.DoWhile && first == block.First) + { + // We only need to insert the continue if we're not at the end of the loop, + // or if our condition is different from the loop condition. + if (Next(stmt.Goto) != null || block.Condition != stmt.Condition) + { + EncloseSingleInst(stmt, Instruction.LoopContinue); + } + + // Modify the do-while condition to allow it to continue. + if (!ContainsCondComb(block.Condition, Instruction.LogicalOr, stmt.Condition)) + { + block.OrCondition(stmt.Condition); + } + + return block; + } + + return Enclose(block, AstBlockType.DoWhile, stmt.Condition, first, stmt.Goto); + } + + private static void EncloseSingleInst(GotoStatement stmt, Instruction inst) + { + AstBlock block = ParentBlock(stmt.Goto); + + AstBlock newBlock = new(AstBlockType.If, stmt.Condition); + + block.AddAfter(stmt.Goto, newBlock); + + newBlock.AddFirst(new AstOperation(inst)); + } + + private static AstBlock Enclose( + AstBlock block, + AstBlockType type, + IAstNode cond, + IAstNode first, + IAstNode last = null) + { + if (first == last) + { + return null; + } + + if (type == AstBlockType.If) + { + cond = InverseCond(cond); + } + + // Do a quick check, if we are enclosing a single block, + // and the block type/condition matches the one we're going + // to create, then we don't need a new block, we can just + // return the old one. + bool hasSingleNode = Next(first) == last; + + if (hasSingleNode && BlockMatches(first, type, cond)) + { + return first as AstBlock; + } + + AstBlock newBlock = new(type, cond); + + block.AddBefore(first, newBlock); + + while (first != last) + { + IAstNode next = Next(first); + + block.Remove(first); + + newBlock.Add(first); + + first = next; + } + + return newBlock; + } + + private static bool BlockMatches(IAstNode node, AstBlockType type, IAstNode cond) + { + if (node is not AstBlock block) + { + return false; + } + + return block.Type == type && IsSameCond(block.Condition, cond); + } + + private static bool IsSameCond(IAstNode lCond, IAstNode rCond) + { + if (lCond is AstOperation lCondOp && lCondOp.Inst == Instruction.LogicalNot) + { + if (rCond is not AstOperation rCondOp || rCondOp.Inst != lCondOp.Inst) + { + return false; + } + + lCond = lCondOp.GetSource(0); + rCond = rCondOp.GetSource(0); + } + + return lCond == rCond; + } + + private static AstBlock ParentBlock(IAstNode node) + { + if (node is AstBlock block) + { + return block.Parent; + } + + while (node is not AstBlock) + { + node = node.Parent; + } + + return node as AstBlock; + } + + private static AstBlock[] BackwardsPath(AstBlock top, AstBlock bottom) + { + AstBlock block = bottom; + + List path = []; + + while (block != top) + { + path.Add(block); + + block = block.Parent; + } + + return path.ToArray(); + } + + private static int Level(AstNode node) + { + int level = 0; + + while (node != null) + { + level++; + + node = node.Parent; + } + + return level; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/GotoStatement.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/GotoStatement.cs new file mode 100644 index 0000000..409a97e --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/GotoStatement.cs @@ -0,0 +1,23 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + class GotoStatement + { + public AstOperation Goto { get; } + public AstAssignment Label { get; } + + public IAstNode Condition => Label.Destination; + + public bool IsLoop { get; set; } + + public bool IsUnconditional => Goto.Inst == Instruction.Branch; + + public GotoStatement(AstOperation branch, AstAssignment label, bool isLoop) + { + Goto = branch; + Label = label; + IsLoop = isLoop; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/HelperFunctionsMask.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/HelperFunctionsMask.cs new file mode 100644 index 0000000..88c0e0b --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/HelperFunctionsMask.cs @@ -0,0 +1,13 @@ +using System; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + [Flags] + enum HelperFunctionsMask + { + MultiplyHighS32 = 1 << 2, + MultiplyHighU32 = 1 << 3, + SwizzleAdd = 1 << 10, + FSI = 1 << 11, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/IAstNode.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/IAstNode.cs new file mode 100644 index 0000000..0d727a0 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/IAstNode.cs @@ -0,0 +1,11 @@ +using System.Collections.Generic; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + interface IAstNode + { + AstBlock Parent { get; set; } + + LinkedListNode LLNode { get; set; } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/InstructionInfo.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/InstructionInfo.cs new file mode 100644 index 0000000..527a879 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/InstructionInfo.cs @@ -0,0 +1,208 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using System; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + static class InstructionInfo + { + private readonly struct InstInfo + { + public AggregateType DestType { get; } + + public AggregateType[] SrcTypes { get; } + + public InstInfo(AggregateType destType, params AggregateType[] srcTypes) + { + DestType = destType; + SrcTypes = srcTypes; + } + } + + private static readonly InstInfo[] _infoTbl; + + static InstructionInfo() + { + _infoTbl = new InstInfo[(int)Instruction.Count]; + +#pragma warning disable IDE0055 // Disable formatting + // Inst Destination type Source 1 type Source 2 type Source 3 type Source 4 type + Add(Instruction.AtomicAdd, AggregateType.U32, AggregateType.S32, AggregateType.S32, AggregateType.U32); + Add(Instruction.AtomicAnd, AggregateType.U32, AggregateType.S32, AggregateType.S32, AggregateType.U32); + Add(Instruction.AtomicCompareAndSwap, AggregateType.U32, AggregateType.S32, AggregateType.S32, AggregateType.U32, AggregateType.U32); + Add(Instruction.AtomicMaxS32, AggregateType.S32, AggregateType.S32, AggregateType.S32, AggregateType.S32); + Add(Instruction.AtomicMaxU32, AggregateType.U32, AggregateType.S32, AggregateType.S32, AggregateType.U32); + Add(Instruction.AtomicMinS32, AggregateType.S32, AggregateType.S32, AggregateType.S32, AggregateType.S32); + Add(Instruction.AtomicMinU32, AggregateType.U32, AggregateType.S32, AggregateType.S32, AggregateType.U32); + Add(Instruction.AtomicOr, AggregateType.U32, AggregateType.S32, AggregateType.S32, AggregateType.U32); + Add(Instruction.AtomicSwap, AggregateType.U32, AggregateType.S32, AggregateType.S32, AggregateType.U32); + Add(Instruction.AtomicXor, AggregateType.U32, AggregateType.S32, AggregateType.S32, AggregateType.U32); + Add(Instruction.Absolute, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.Add, AggregateType.Scalar, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.Ballot, AggregateType.U32, AggregateType.Bool); + Add(Instruction.BitCount, AggregateType.S32, AggregateType.S32); + Add(Instruction.BitfieldExtractS32, AggregateType.S32, AggregateType.S32, AggregateType.S32, AggregateType.S32); + Add(Instruction.BitfieldExtractU32, AggregateType.U32, AggregateType.U32, AggregateType.S32, AggregateType.S32); + Add(Instruction.BitfieldInsert, AggregateType.S32, AggregateType.S32, AggregateType.S32, AggregateType.S32, AggregateType.S32); + Add(Instruction.BitfieldReverse, AggregateType.S32, AggregateType.S32); + Add(Instruction.BitwiseAnd, AggregateType.S32, AggregateType.S32, AggregateType.S32); + Add(Instruction.BitwiseExclusiveOr, AggregateType.S32, AggregateType.S32, AggregateType.S32); + Add(Instruction.BitwiseNot, AggregateType.S32, AggregateType.S32); + Add(Instruction.BitwiseOr, AggregateType.S32, AggregateType.S32, AggregateType.S32); + Add(Instruction.BranchIfTrue, AggregateType.Void, AggregateType.Bool); + Add(Instruction.BranchIfFalse, AggregateType.Void, AggregateType.Bool); + Add(Instruction.Call, AggregateType.Scalar); + Add(Instruction.Ceiling, AggregateType.Scalar, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.Clamp, AggregateType.Scalar, AggregateType.Scalar, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.ClampU32, AggregateType.U32, AggregateType.U32, AggregateType.U32, AggregateType.U32); + Add(Instruction.CompareEqual, AggregateType.Bool, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.CompareGreater, AggregateType.Bool, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.CompareGreaterOrEqual, AggregateType.Bool, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.CompareGreaterOrEqualU32, AggregateType.Bool, AggregateType.U32, AggregateType.U32); + Add(Instruction.CompareGreaterU32, AggregateType.Bool, AggregateType.U32, AggregateType.U32); + Add(Instruction.CompareLess, AggregateType.Bool, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.CompareLessOrEqual, AggregateType.Bool, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.CompareLessOrEqualU32, AggregateType.Bool, AggregateType.U32, AggregateType.U32); + Add(Instruction.CompareLessU32, AggregateType.Bool, AggregateType.U32, AggregateType.U32); + Add(Instruction.CompareNotEqual, AggregateType.Bool, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.ConditionalSelect, AggregateType.Scalar, AggregateType.Bool, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.ConvertFP32ToFP64, AggregateType.FP64, AggregateType.FP32); + Add(Instruction.ConvertFP64ToFP32, AggregateType.FP32, AggregateType.FP64); + Add(Instruction.ConvertFP32ToS32, AggregateType.S32, AggregateType.FP32); + Add(Instruction.ConvertFP32ToU32, AggregateType.U32, AggregateType.FP32); + Add(Instruction.ConvertFP64ToS32, AggregateType.S32, AggregateType.FP64); + Add(Instruction.ConvertFP64ToU32, AggregateType.U32, AggregateType.FP64); + Add(Instruction.ConvertS32ToFP32, AggregateType.FP32, AggregateType.S32); + Add(Instruction.ConvertS32ToFP64, AggregateType.FP64, AggregateType.S32); + Add(Instruction.ConvertU32ToFP32, AggregateType.FP32, AggregateType.U32); + Add(Instruction.ConvertU32ToFP64, AggregateType.FP64, AggregateType.U32); + Add(Instruction.Cosine, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.Ddx, AggregateType.FP32, AggregateType.FP32); + Add(Instruction.Ddy, AggregateType.FP32, AggregateType.FP32); + Add(Instruction.Divide, AggregateType.Scalar, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.ExponentB2, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.FindLSB, AggregateType.S32, AggregateType.S32); + Add(Instruction.FindMSBS32, AggregateType.S32, AggregateType.S32); + Add(Instruction.FindMSBU32, AggregateType.S32, AggregateType.U32); + Add(Instruction.Floor, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.FusedMultiplyAdd, AggregateType.Scalar, AggregateType.Scalar, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.ImageLoad, AggregateType.FP32); + Add(Instruction.ImageStore, AggregateType.Void); + Add(Instruction.ImageAtomic, AggregateType.S32); + Add(Instruction.IsNan, AggregateType.Bool, AggregateType.Scalar); + Add(Instruction.Load, AggregateType.FP32); + Add(Instruction.Lod, AggregateType.FP32); + Add(Instruction.LogarithmB2, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.LogicalAnd, AggregateType.Bool, AggregateType.Bool, AggregateType.Bool); + Add(Instruction.LogicalExclusiveOr, AggregateType.Bool, AggregateType.Bool, AggregateType.Bool); + Add(Instruction.LogicalNot, AggregateType.Bool, AggregateType.Bool); + Add(Instruction.LogicalOr, AggregateType.Bool, AggregateType.Bool, AggregateType.Bool); + Add(Instruction.Maximum, AggregateType.Scalar, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.MaximumU32, AggregateType.U32, AggregateType.U32, AggregateType.U32); + Add(Instruction.Minimum, AggregateType.Scalar, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.MinimumU32, AggregateType.U32, AggregateType.U32, AggregateType.U32); + Add(Instruction.Modulo, AggregateType.Scalar, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.Multiply, AggregateType.Scalar, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.MultiplyHighS32, AggregateType.S32, AggregateType.S32, AggregateType.S32); + Add(Instruction.MultiplyHighU32, AggregateType.U32, AggregateType.U32, AggregateType.U32); + Add(Instruction.Negate, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.PackDouble2x32, AggregateType.FP64, AggregateType.U32, AggregateType.U32); + Add(Instruction.PackHalf2x16, AggregateType.U32, AggregateType.FP32, AggregateType.FP32); + Add(Instruction.ReciprocalSquareRoot, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.Return, AggregateType.Void, AggregateType.U32); + Add(Instruction.Round, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.ShiftLeft, AggregateType.S32, AggregateType.S32, AggregateType.S32); + Add(Instruction.ShiftRightS32, AggregateType.S32, AggregateType.S32, AggregateType.S32); + Add(Instruction.ShiftRightU32, AggregateType.U32, AggregateType.U32, AggregateType.S32); + Add(Instruction.Shuffle, AggregateType.FP32, AggregateType.FP32, AggregateType.U32); + Add(Instruction.ShuffleDown, AggregateType.FP32, AggregateType.FP32, AggregateType.U32); + Add(Instruction.ShuffleUp, AggregateType.FP32, AggregateType.FP32, AggregateType.U32); + Add(Instruction.ShuffleXor, AggregateType.FP32, AggregateType.FP32, AggregateType.U32); + Add(Instruction.Sine, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.SquareRoot, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.Store, AggregateType.Void); + Add(Instruction.Subtract, AggregateType.Scalar, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.SwizzleAdd, AggregateType.FP32, AggregateType.FP32, AggregateType.FP32, AggregateType.S32); + Add(Instruction.TextureSample, AggregateType.FP32); + Add(Instruction.TextureQuerySamples, AggregateType.S32, AggregateType.S32); + Add(Instruction.TextureQuerySize, AggregateType.S32, AggregateType.S32, AggregateType.S32); + Add(Instruction.Truncate, AggregateType.Scalar, AggregateType.Scalar); + Add(Instruction.UnpackDouble2x32, AggregateType.U32, AggregateType.FP64); + Add(Instruction.UnpackHalf2x16, AggregateType.FP32, AggregateType.U32); + Add(Instruction.VectorExtract, AggregateType.Scalar, AggregateType.Vector4, AggregateType.S32); + Add(Instruction.VoteAll, AggregateType.Bool, AggregateType.Bool); + Add(Instruction.VoteAllEqual, AggregateType.Bool, AggregateType.Bool); + Add(Instruction.VoteAny, AggregateType.Bool, AggregateType.Bool); +#pragma warning restore IDE0055 + } + + private static void Add(Instruction inst, AggregateType destType, params AggregateType[] srcTypes) + { + _infoTbl[(int)inst] = new InstInfo(destType, srcTypes); + } + + public static AggregateType GetDestVarType(Instruction inst) + { + return GetFinalVarType(_infoTbl[(int)(inst & Instruction.Mask)].DestType, inst); + } + + public static AggregateType GetSrcVarType(Instruction inst, int index) + { + // TODO: Return correct type depending on source index, + // that can improve the decompiler output. + if (inst is Instruction.ImageLoad or + Instruction.ImageStore or + Instruction.ImageAtomic or + Instruction.Lod or + Instruction.TextureSample) + { + return AggregateType.FP32; + } + else if (inst == Instruction.Call || inst == Instruction.Load || inst == Instruction.Store || inst.IsAtomic()) + { + return AggregateType.S32; + } + + return GetFinalVarType(_infoTbl[(int)(inst & Instruction.Mask)].SrcTypes[index], inst); + } + + private static AggregateType GetFinalVarType(AggregateType type, Instruction inst) + { + if (type == AggregateType.Scalar) + { + if ((inst & Instruction.FP32) != 0) + { + return AggregateType.FP32; + } + else if ((inst & Instruction.FP64) != 0) + { + return AggregateType.FP64; + } + else + { + return AggregateType.S32; + } + } + else if (type == AggregateType.Void) + { + throw new ArgumentException($"Invalid operand for instruction \"{inst}\"."); + } + + return type; + } + + public static bool IsUnary(Instruction inst) + { + if (inst == Instruction.Copy) + { + return true; + } + else if (inst == Instruction.TextureSample) + { + return false; + } + + return _infoTbl[(int)(inst & Instruction.Mask)].SrcTypes.Length == 1; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/IoDefinition.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/IoDefinition.cs new file mode 100644 index 0000000..5763431 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/IoDefinition.cs @@ -0,0 +1,44 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + readonly struct IoDefinition : IEquatable + { + public StorageKind StorageKind { get; } + public IoVariable IoVariable { get; } + public int Location { get; } + public int Component { get; } + + public IoDefinition(StorageKind storageKind, IoVariable ioVariable, int location = 0, int component = 0) + { + StorageKind = storageKind; + IoVariable = ioVariable; + Location = location; + Component = component; + } + + public override bool Equals(object other) + { + return other is IoDefinition ioDefinition && Equals(ioDefinition); + } + + public bool Equals(IoDefinition other) + { + return StorageKind == other.StorageKind && + IoVariable == other.IoVariable && + Location == other.Location && + Component == other.Component; + } + + public override int GetHashCode() + { + return (int)StorageKind | ((int)IoVariable << 8) | (Location << 16) | (Component << 24); + } + + public override string ToString() + { + return $"{StorageKind}.{IoVariable}.{Location}.{Component}"; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/MemoryDefinition.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/MemoryDefinition.cs new file mode 100644 index 0000000..61f0d82 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/MemoryDefinition.cs @@ -0,0 +1,18 @@ +using Ryujinx.Graphics.Shader.Translation; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + readonly struct MemoryDefinition + { + public string Name { get; } + public AggregateType Type { get; } + public int ArrayLength { get; } + + public MemoryDefinition(string name, AggregateType type, int arrayLength = 1) + { + Name = name; + Type = type; + ArrayLength = arrayLength; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/OperandInfo.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/OperandInfo.cs new file mode 100644 index 0000000..15d274c --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/OperandInfo.cs @@ -0,0 +1,32 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using System; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + static class OperandInfo + { + public static AggregateType GetVarType(AstOperand operand) + { + if (operand.Type == OperandType.LocalVariable) + { + return operand.VarType; + } + else + { + return GetVarType(operand.Type); + } + } + + public static AggregateType GetVarType(OperandType type) + { + return type switch + { + OperandType.Argument => AggregateType.S32, + OperandType.Constant => AggregateType.S32, + OperandType.Undefined => AggregateType.S32, + _ => throw new ArgumentException($"Invalid operand type \"{type}\"."), + }; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/PhiFunctions.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/PhiFunctions.cs new file mode 100644 index 0000000..fd737e8 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/PhiFunctions.cs @@ -0,0 +1,49 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System.Collections.Generic; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + static class PhiFunctions + { + public static void Remove(BasicBlock[] blocks) + { + for (int blkIndex = 0; blkIndex < blocks.Length; blkIndex++) + { + BasicBlock block = blocks[blkIndex]; + + LinkedListNode node = block.Operations.First; + + while (node != null) + { + LinkedListNode nextNode = node.Next; + + if (node.Value is not PhiNode phi) + { + node = nextNode; + + continue; + } + + Operand temp = OperandHelper.Local(); + + for (int index = 0; index < phi.SourcesCount; index++) + { + Operand src = phi.GetSource(index); + BasicBlock srcBlock = phi.GetBlock(index); + + Operation copyOp = new(Instruction.Copy, temp, src); + + srcBlock.Append(copyOp); + } + + Operation copyOp2 = new(Instruction.Copy, phi.Dest, temp); + + nextNode = block.Operations.AddAfter(node, copyOp2).Next; + block.Operations.Remove(node); + + node = nextNode; + } + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/ShaderProperties.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/ShaderProperties.cs new file mode 100644 index 0000000..3f1f88f --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/ShaderProperties.cs @@ -0,0 +1,110 @@ +using System.Collections.Generic; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + class ShaderProperties + { + private readonly Dictionary _constantBuffers; + private readonly Dictionary _storageBuffers; + private readonly Dictionary _textures; + private readonly Dictionary _images; + private readonly Dictionary _localMemories; + private readonly Dictionary _sharedMemories; + + public IReadOnlyDictionary ConstantBuffers => _constantBuffers; + public IReadOnlyDictionary StorageBuffers => _storageBuffers; + public IReadOnlyDictionary Textures => _textures; + public IReadOnlyDictionary Images => _images; + public IReadOnlyDictionary LocalMemories => _localMemories; + public IReadOnlyDictionary SharedMemories => _sharedMemories; + + public ShaderProperties() + { + _constantBuffers = new Dictionary(); + _storageBuffers = new Dictionary(); + _textures = new Dictionary(); + _images = new Dictionary(); + _localMemories = new Dictionary(); + _sharedMemories = new Dictionary(); + } + + public void AddOrUpdateConstantBuffer(BufferDefinition definition) + { + _constantBuffers[definition.Binding] = definition; + } + + public void AddOrUpdateStorageBuffer(BufferDefinition definition) + { + _storageBuffers[definition.Binding] = definition; + } + + public void AddOrUpdateTexture(TextureDefinition definition) + { + _textures[new(definition.Set, definition.Binding)] = definition; + } + + public void AddOrUpdateImage(TextureDefinition definition) + { + _images[new(definition.Set, definition.Binding)] = definition; + } + + public int AddLocalMemory(MemoryDefinition definition) + { + int id = _localMemories.Count; + _localMemories.Add(id, definition); + + return id; + } + + public int AddSharedMemory(MemoryDefinition definition) + { + int id = _sharedMemories.Count; + _sharedMemories.Add(id, definition); + + return id; + } + + public static TextureFormat GetTextureFormat(IGpuAccessor gpuAccessor, int handle, int cbufSlot = -1) + { + // When the formatted load extension is supported, we don't need to + // specify a format, we can just declare it without a format and the GPU will handle it. + if (gpuAccessor.QueryHostSupportsImageLoadFormatted()) + { + return TextureFormat.Unknown; + } + + TextureFormat format = gpuAccessor.QueryTextureFormat(handle, cbufSlot); + + if (format == TextureFormat.Unknown) + { + gpuAccessor.Log($"Unknown format for texture {handle}."); + + format = TextureFormat.R8G8B8A8Unorm; + } + + return format; + } + + private static bool FormatSupportsAtomic(TextureFormat format) + { + return format is TextureFormat.R32Sint or TextureFormat.R32Uint; + } + + public static TextureFormat GetTextureFormatAtomic(IGpuAccessor gpuAccessor, int handle, int cbufSlot = -1) + { + // Atomic image instructions do not support GL_EXT_shader_image_load_formatted, + // and must have a type specified. Default to R32Sint if not available. + + TextureFormat format = gpuAccessor.QueryTextureFormat(handle, cbufSlot); + + if (!FormatSupportsAtomic(format)) + { + gpuAccessor.Log($"Unsupported format for texture {handle}: {format}."); + + format = TextureFormat.R32Sint; + } + + return format; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/StructureType.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/StructureType.cs new file mode 100644 index 0000000..9f13a38 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/StructureType.cs @@ -0,0 +1,28 @@ +using Ryujinx.Graphics.Shader.Translation; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + readonly struct StructureField + { + public AggregateType Type { get; } + public string Name { get; } + public int ArrayLength { get; } + + public StructureField(AggregateType type, string name, int arrayLength = 1) + { + Type = type; + Name = name; + ArrayLength = arrayLength; + } + } + + class StructureType + { + public StructureField[] Fields { get; } + + public StructureType(StructureField[] fields) + { + Fields = fields; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/StructuredFunction.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/StructuredFunction.cs new file mode 100644 index 0000000..f251356 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/StructuredFunction.cs @@ -0,0 +1,42 @@ +using Ryujinx.Graphics.Shader.Translation; +using System.Collections.Generic; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + class StructuredFunction + { + public AstBlock MainBlock { get; } + + public string Name { get; } + + public AggregateType ReturnType { get; } + + public AggregateType[] InArguments { get; } + public AggregateType[] OutArguments { get; } + + public HashSet Locals { get; } + + public StructuredFunction( + AstBlock mainBlock, + string name, + AggregateType returnType, + AggregateType[] inArguments, + AggregateType[] outArguments) + { + MainBlock = mainBlock; + Name = name; + ReturnType = returnType; + InArguments = inArguments; + OutArguments = outArguments; + + Locals = []; + } + + public AggregateType GetArgumentType(int index) + { + return index >= InArguments.Length + ? OutArguments[index - InArguments.Length] + : InArguments[index]; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/StructuredProgram.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/StructuredProgram.cs new file mode 100644 index 0000000..5bd9f8e --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/StructuredProgram.cs @@ -0,0 +1,467 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Numerics; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + static class StructuredProgram + { + // TODO: Eventually it should be possible to specify the parameter types for the function instead of using S32 for everything. + private const AggregateType FuncParameterType = AggregateType.S32; + + public static StructuredProgramInfo MakeStructuredProgram( + IReadOnlyList functions, + AttributeUsage attributeUsage, + ShaderDefinitions definitions, + ResourceManager resourceManager, + TargetLanguage targetLanguage, + bool debugMode) + { + StructuredProgramContext context = new(attributeUsage, definitions, resourceManager, debugMode); + + for (int funcIndex = 0; funcIndex < functions.Count; funcIndex++) + { + Function function = functions[funcIndex]; + + BasicBlock[] blocks = function.Blocks; + + AggregateType returnType = function.ReturnsValue ? FuncParameterType : AggregateType.Void; + + AggregateType[] inArguments = new AggregateType[function.InArgumentsCount]; + AggregateType[] outArguments = new AggregateType[function.OutArgumentsCount]; + + for (int i = 0; i < inArguments.Length; i++) + { + inArguments[i] = FuncParameterType; + } + + for (int i = 0; i < outArguments.Length; i++) + { + outArguments[i] = FuncParameterType; + } + + context.EnterFunction(blocks.Length, function.Name, returnType, inArguments, outArguments); + + PhiFunctions.Remove(blocks); + + for (int blkIndex = 0; blkIndex < blocks.Length; blkIndex++) + { + BasicBlock block = blocks[blkIndex]; + + context.EnterBlock(block); + + for (LinkedListNode opNode = block.Operations.First; opNode != null; opNode = opNode.Next) + { + Operation operation = (Operation)opNode.Value; + + if (IsBranchInst(operation.Inst)) + { + context.LeaveBlock(block, operation); + } + else + { + AddOperation(context, operation, targetLanguage, functions); + } + } + } + + GotoElimination.Eliminate(context.GetGotos()); + + AstOptimizer.Optimize(context); + + context.LeaveFunction(); + } + + return context.Info; + } + + private static void AddOperation(StructuredProgramContext context, Operation operation, TargetLanguage targetLanguage, IReadOnlyList functions) + { + Instruction inst = operation.Inst; + StorageKind storageKind = operation.StorageKind; + + if (inst is Instruction.Load or Instruction.Store) + { + if (storageKind.IsInputOrOutput()) + { + IoVariable ioVariable = (IoVariable)operation.GetSource(0).Value; + bool isOutput = storageKind.IsOutput(); + int location = 0; + int component = 0; + + if (context.Definitions.HasPerLocationInputOrOutput(ioVariable, isOutput)) + { + location = operation.GetSource(1).Value; + + if (operation.SourcesCount > 2 && + operation.GetSource(2).Type == OperandType.Constant && + context.Definitions.HasPerLocationInputOrOutputComponent(ioVariable, location, operation.GetSource(2).Value, isOutput)) + { + component = operation.GetSource(2).Value; + } + } + + context.Info.IoDefinitions.Add(new IoDefinition(storageKind, ioVariable, location, component)); + } + else if (storageKind == StorageKind.ConstantBuffer && operation.GetSource(0).Type == OperandType.Constant) + { + context.ResourceManager.SetUsedConstantBufferBinding(operation.GetSource(0).Value); + } + } + + bool vectorDest = IsVectorDestInst(inst); + + int sourcesCount = operation.SourcesCount; + int outDestsCount = operation.DestsCount != 0 && !vectorDest ? operation.DestsCount - 1 : 0; + + IAstNode[] sources = new IAstNode[sourcesCount + outDestsCount]; + + if (inst == Instruction.Call && targetLanguage == TargetLanguage.Spirv) + { + // SPIR-V requires that all function parameters are copied to a local variable before the call + // (or at least that's what the Khronos compiler does). + + // First one is the function index. + Operand funcIndexOperand = operation.GetSource(0); + Debug.Assert(funcIndexOperand.Type == OperandType.Constant); + int funcIndex = funcIndexOperand.Value; + + sources[0] = new AstOperand(OperandType.Constant, funcIndex); + + int inArgsCount = functions[funcIndex].InArgumentsCount; + + // Remaining ones are parameters, copy them to a temp local variable. + for (int index = 1; index < operation.SourcesCount; index++) + { + IAstNode source = context.GetOperandOrCbLoad(operation.GetSource(index)); + + if (index - 1 < inArgsCount) + { + AstOperand argTemp = context.NewTemp(FuncParameterType); + context.AddNode(new AstAssignment(argTemp, source)); + sources[index] = argTemp; + } + else + { + sources[index] = source; + } + } + } + else + { + for (int index = 0; index < operation.SourcesCount; index++) + { + sources[index] = context.GetOperandOrCbLoad(operation.GetSource(index)); + } + } + + for (int index = 0; index < outDestsCount; index++) + { + AstOperand oper = context.GetOperand(operation.GetDest(1 + index)); + + oper.VarType = InstructionInfo.GetSrcVarType(inst, sourcesCount + index); + + sources[sourcesCount + index] = oper; + } + + AstTextureOperation GetAstTextureOperation(TextureOperation texOp) + { + return new AstTextureOperation( + inst, + texOp.Type, + texOp.Format, + texOp.Flags, + texOp.Set, + texOp.Binding, + texOp.SamplerSet, + texOp.SamplerBinding, + texOp.Index, + sources); + } + + int componentsCount = BitOperations.PopCount((uint)operation.Index); + + if (vectorDest && componentsCount > 1) + { + AggregateType destType = InstructionInfo.GetDestVarType(inst); + + IAstNode source; + + if (operation is TextureOperation texOp) + { + if (texOp.Inst == Instruction.ImageLoad) + { + destType = texOp.Format.GetComponentType(); + } + + source = GetAstTextureOperation(texOp); + } + else + { + source = new AstOperation( + inst, + operation.StorageKind, + operation.ForcePrecise, + operation.Index, + sources, + operation.SourcesCount); + } + + AggregateType destElemType = destType; + + switch (componentsCount) + { + case 2: + destType |= AggregateType.Vector2; + break; + case 3: + destType |= AggregateType.Vector3; + break; + case 4: + destType |= AggregateType.Vector4; + break; + } + + AstOperand destVec = context.NewTemp(destType); + + context.AddNode(new AstAssignment(destVec, source)); + + for (int i = 0; i < operation.DestsCount; i++) + { + AstOperand dest = context.GetOperand(operation.GetDest(i)); + AstOperand index = new(OperandType.Constant, i); + + dest.VarType = destElemType; + + context.AddNode(new AstAssignment(dest, new AstOperation(Instruction.VectorExtract, StorageKind.None, false, + [destVec, index], 2))); + } + } + else if (operation.Dest != null) + { + AstOperand dest = context.GetOperand(operation.Dest); + + // If all the sources are bool, it's better to use short-circuiting + // logical operations, rather than forcing a cast to int and doing + // a bitwise operation with the value, as it is likely to be used as + // a bool in the end. + if (IsBitwiseInst(inst) && AreAllSourceTypesEqual(sources, AggregateType.Bool)) + { + inst = GetLogicalFromBitwiseInst(inst); + } + + bool isCondSel = inst == Instruction.ConditionalSelect; + bool isCopy = inst == Instruction.Copy; + + if (isCondSel || isCopy) + { + AggregateType type = GetVarTypeFromUses(operation.Dest); + + if (isCondSel && type == AggregateType.FP32) + { + inst |= Instruction.FP32; + } + + dest.VarType = type; + } + else + { + dest.VarType = InstructionInfo.GetDestVarType(inst); + } + + IAstNode source; + + if (operation is TextureOperation texOp) + { + if (texOp.Inst == Instruction.ImageLoad) + { + dest.VarType = texOp.Format.GetComponentType(); + } + + source = GetAstTextureOperation(texOp); + } + else if (!isCopy) + { + source = new AstOperation( + inst, + operation.StorageKind, + operation.ForcePrecise, + operation.Index, + sources, + operation.SourcesCount); + } + else + { + source = sources[0]; + } + + context.AddNode(new AstAssignment(dest, source)); + } + else if (operation.Inst == Instruction.Comment) + { + context.AddNode(new AstComment(((CommentNode)operation).Comment)); + } + else if (operation is TextureOperation texOp) + { + AstTextureOperation astTexOp = GetAstTextureOperation(texOp); + + context.AddNode(astTexOp); + } + else + { + context.AddNode(new AstOperation( + inst, + operation.StorageKind, + operation.ForcePrecise, + operation.Index, + sources, + operation.SourcesCount)); + } + + // Those instructions needs to be emulated by using helper functions, + // because they are NVIDIA specific. Those flags helps the backend to + // decide which helper functions are needed on the final generated code. + switch (operation.Inst) + { + case Instruction.MultiplyHighS32: + context.Info.HelperFunctionsMask |= HelperFunctionsMask.MultiplyHighS32; + break; + case Instruction.MultiplyHighU32: + context.Info.HelperFunctionsMask |= HelperFunctionsMask.MultiplyHighU32; + break; + case Instruction.SwizzleAdd: + context.Info.HelperFunctionsMask |= HelperFunctionsMask.SwizzleAdd; + break; + case Instruction.FSIBegin: + case Instruction.FSIEnd: + context.Info.HelperFunctionsMask |= HelperFunctionsMask.FSI; + break; + } + } + + private static AggregateType GetVarTypeFromUses(Operand dest) + { + HashSet visited = []; + + Queue pending = new(); + + bool Enqueue(Operand operand) + { + if (visited.Add(operand)) + { + pending.Enqueue(operand); + + return true; + } + + return false; + } + + Enqueue(dest); + + while (pending.TryDequeue(out Operand operand)) + { + foreach (INode useNode in operand.UseOps) + { + if (useNode is not Operation operation) + { + continue; + } + + if (operation.Inst == Instruction.Copy) + { + if (operation.Dest.Type == OperandType.LocalVariable) + { + if (Enqueue(operation.Dest)) + { + break; + } + } + else + { + return OperandInfo.GetVarType(operation.Dest.Type); + } + } + else + { + for (int index = 0; index < operation.SourcesCount; index++) + { + if (operation.GetSource(index) == operand) + { + return InstructionInfo.GetSrcVarType(operation.Inst, index); + } + } + } + } + } + + return AggregateType.S32; + } + + private static bool AreAllSourceTypesEqual(IAstNode[] sources, AggregateType type) + { + foreach (IAstNode node in sources) + { + if (node is not AstOperand operand) + { + return false; + } + + if (operand.VarType != type) + { + return false; + } + } + + return true; + } + + private static bool IsVectorDestInst(Instruction inst) + { + return inst switch + { + Instruction.ImageLoad or + Instruction.TextureSample => true, + _ => false, + }; + } + + private static bool IsBranchInst(Instruction inst) + { + return inst switch + { + Instruction.Branch or + Instruction.BranchIfFalse or + Instruction.BranchIfTrue => true, + _ => false, + }; + } + + private static bool IsBitwiseInst(Instruction inst) + { + return inst switch + { + Instruction.BitwiseAnd or + Instruction.BitwiseExclusiveOr or + Instruction.BitwiseNot or + Instruction.BitwiseOr => true, + _ => false, + }; + } + + private static Instruction GetLogicalFromBitwiseInst(Instruction inst) + { + return inst switch + { + Instruction.BitwiseAnd => Instruction.LogicalAnd, + Instruction.BitwiseExclusiveOr => Instruction.LogicalExclusiveOr, + Instruction.BitwiseNot => Instruction.LogicalNot, + Instruction.BitwiseOr => Instruction.LogicalOr, + _ => throw new ArgumentException($"Unexpected instruction \"{inst}\"."), + }; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/StructuredProgramContext.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/StructuredProgramContext.cs new file mode 100644 index 0000000..36a3e9e --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/StructuredProgramContext.cs @@ -0,0 +1,354 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation; +using System.Collections.Generic; +using System.Linq; +using System.Numerics; +using static Ryujinx.Graphics.Shader.StructuredIr.AstHelper; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + class StructuredProgramContext + { + private HashSet _loopTails; + + private Stack<(AstBlock Block, int CurrEndIndex, int LoopEndIndex)> _blockStack; + + private Dictionary _localsMap; + + private Dictionary _gotoTempAsgs; + + private List _gotos; + + private AstBlock _currBlock; + + private int _currEndIndex; + private int _loopEndIndex; + + public StructuredFunction CurrentFunction { get; private set; } + + public StructuredProgramInfo Info { get; } + + public ShaderDefinitions Definitions { get; } + public ResourceManager ResourceManager { get; } + public bool DebugMode { get; } + + public StructuredProgramContext( + AttributeUsage attributeUsage, + ShaderDefinitions definitions, + ResourceManager resourceManager, + bool debugMode) + { + Info = new StructuredProgramInfo(); + + Definitions = definitions; + ResourceManager = resourceManager; + DebugMode = debugMode; + + if (definitions.GpPassthrough) + { + int passthroughAttributes = attributeUsage.PassthroughAttributes; + while (passthroughAttributes != 0) + { + int index = BitOperations.TrailingZeroCount(passthroughAttributes); + + Info.IoDefinitions.Add(new IoDefinition(StorageKind.Input, IoVariable.UserDefined, index)); + + passthroughAttributes &= ~(1 << index); + } + + Info.IoDefinitions.Add(new IoDefinition(StorageKind.Input, IoVariable.Position)); + Info.IoDefinitions.Add(new IoDefinition(StorageKind.Input, IoVariable.PointSize)); + Info.IoDefinitions.Add(new IoDefinition(StorageKind.Input, IoVariable.ClipDistance)); + } + } + + public void EnterFunction( + int blocksCount, + string name, + AggregateType returnType, + AggregateType[] inArguments, + AggregateType[] outArguments) + { + _loopTails = []; + + _blockStack = new Stack<(AstBlock, int, int)>(); + + _localsMap = new Dictionary(); + + _gotoTempAsgs = new Dictionary(); + + _gotos = []; + + _currBlock = new AstBlock(AstBlockType.Main); + + _currEndIndex = blocksCount; + _loopEndIndex = blocksCount; + + CurrentFunction = new StructuredFunction(_currBlock, name, returnType, inArguments, outArguments); + } + + public void LeaveFunction() + { + Info.Functions.Add(CurrentFunction); + } + + public void EnterBlock(BasicBlock block) + { + while (_currEndIndex == block.Index) + { + (_currBlock, _currEndIndex, _loopEndIndex) = _blockStack.Pop(); + } + + if (_gotoTempAsgs.TryGetValue(block.Index, out AstAssignment gotoTempAsg)) + { + AddGotoTempReset(block, gotoTempAsg); + } + + LookForDoWhileStatements(block); + } + + public void LeaveBlock(BasicBlock block, Operation branchOp) + { + LookForIfStatements(block, branchOp); + } + + private void LookForDoWhileStatements(BasicBlock block) + { + // Check if we have any predecessor whose index is greater than the + // current block, this indicates a loop. + bool done = false; + + foreach (BasicBlock predecessor in block.Predecessors.OrderByDescending(x => x.Index)) + { + // If not a loop, break. + if (predecessor.Index < block.Index) + { + break; + } + + // Check if we can create a do-while loop here (only possible if the loop end + // falls inside the current scope), if not add a goto instead. + if (predecessor.Index < _currEndIndex && !done) + { + // Create do-while loop block. We must avoid inserting a goto at the end + // of the loop later, when the tail block is processed. So we add the predecessor + // to a list of loop tails to prevent it from being processed later. + Operation branchOp = (Operation)predecessor.GetLastOp(); + + NewBlock(AstBlockType.DoWhile, branchOp, predecessor.Index + 1); + + _loopTails.Add(predecessor); + + done = true; + } + else + { + // Failed to create loop. Since this block is the loop head, we reset the + // goto condition variable here. The variable is always reset on the jump + // target, and this block is the jump target for some loop. + AddGotoTempReset(block, GetGotoTempAsg(block.Index)); + + break; + } + } + } + + private void LookForIfStatements(BasicBlock block, Operation branchOp) + { + if (block.Branch == null) + { + return; + } + + // We can only enclose the "if" when the branch lands before + // the end of the current block. If the current enclosing block + // is not a loop, then we can also do so if the branch lands + // right at the end of the current block. When it is a loop, + // this is not valid as the loop condition would be evaluated, + // and it could erroneously jump back to the start of the loop. + bool inRange = + block.Branch.Index < _currEndIndex || + (block.Branch.Index == _currEndIndex && block.Branch.Index < _loopEndIndex); + + bool isLoop = block.Branch.Index <= block.Index; + + if (inRange && !isLoop) + { + NewBlock(AstBlockType.If, branchOp, block.Branch.Index); + } + else if (!_loopTails.Contains(block)) + { + AstAssignment gotoTempAsg = GetGotoTempAsg(block.Branch.Index); + + // We use DoWhile type here, as the condition should be true for + // unconditional branches, or it should jump if the condition is true otherwise. + IAstNode cond = GetBranchCond(AstBlockType.DoWhile, branchOp); + + AddNode(Assign(gotoTempAsg.Destination, cond)); + + AstOperation branch = new(branchOp.Inst); + + AddNode(branch); + + GotoStatement gotoStmt = new(branch, gotoTempAsg, isLoop); + + _gotos.Add(gotoStmt); + } + } + + private AstAssignment GetGotoTempAsg(int index) + { + if (_gotoTempAsgs.TryGetValue(index, out AstAssignment gotoTempAsg)) + { + return gotoTempAsg; + } + + AstOperand gotoTemp = NewTemp(AggregateType.Bool); + + gotoTempAsg = Assign(gotoTemp, Const(IrConsts.False)); + + _gotoTempAsgs.Add(index, gotoTempAsg); + + return gotoTempAsg; + } + + private void AddGotoTempReset(BasicBlock block, AstAssignment gotoTempAsg) + { + // If it was already added, we don't need to add it again. + if (gotoTempAsg.Parent != null) + { + return; + } + + AddNode(gotoTempAsg); + + // For block 0, we don't need to add the extra "reset" at the beginning, + // because it is already the first node to be executed on the shader, + // so it is reset to false by the "local" assignment anyway. + if (block.Index != 0) + { + CurrentFunction.MainBlock.AddFirst(Assign(gotoTempAsg.Destination, Const(IrConsts.False))); + } + } + + private void NewBlock(AstBlockType type, Operation branchOp, int endIndex) + { + NewBlock(type, GetBranchCond(type, branchOp), endIndex); + } + + private void NewBlock(AstBlockType type, IAstNode cond, int endIndex) + { + AstBlock childBlock = new(type, cond); + + AddNode(childBlock); + + _blockStack.Push((_currBlock, _currEndIndex, _loopEndIndex)); + + _currBlock = childBlock; + _currEndIndex = endIndex; + + if (type == AstBlockType.DoWhile) + { + _loopEndIndex = endIndex; + } + } + + private IAstNode GetBranchCond(AstBlockType type, Operation branchOp) + { + IAstNode cond; + + if (branchOp.Inst == Instruction.Branch) + { + // If the branch is not conditional, the condition is a constant. + // For if it's false (always jump over, if block never executed). + // For loops it's always true (always loop). + cond = Const(type == AstBlockType.If ? IrConsts.False : IrConsts.True); + } + else + { + cond = GetOperand(branchOp.GetSource(0)); + + Instruction invInst = type == AstBlockType.If + ? Instruction.BranchIfTrue + : Instruction.BranchIfFalse; + + if (branchOp.Inst == invInst) + { + cond = new AstOperation(Instruction.LogicalNot, cond); + } + } + + return cond; + } + + public void AddNode(IAstNode node) + { + _currBlock.Add(node); + } + + public GotoStatement[] GetGotos() + { + return _gotos.ToArray(); + } + + public AstOperand NewTemp(AggregateType type) + { + AstOperand newTemp = Local(type); + + CurrentFunction.Locals.Add(newTemp); + + return newTemp; + } + + public IAstNode GetOperandOrCbLoad(Operand operand) + { + if (operand.Type == OperandType.ConstantBuffer) + { + int cbufSlot = operand.GetCbufSlot(); + int cbufOffset = operand.GetCbufOffset(); + + int binding = ResourceManager.GetConstantBufferBinding(cbufSlot); + int vecIndex = cbufOffset >> 2; + int elemIndex = cbufOffset & 3; + + ResourceManager.SetUsedConstantBufferBinding(binding); + + IAstNode[] sources = + [ + new AstOperand(OperandType.Constant, binding), + new AstOperand(OperandType.Constant, 0), + new AstOperand(OperandType.Constant, vecIndex), + new AstOperand(OperandType.Constant, elemIndex) + ]; + + return new AstOperation(Instruction.Load, StorageKind.ConstantBuffer, false, sources, sources.Length); + } + + return GetOperand(operand); + } + + public AstOperand GetOperand(Operand operand) + { + if (operand == null) + { + return null; + } + + if (operand.Type != OperandType.LocalVariable) + { + return new AstOperand(operand); + } + + if (!_localsMap.TryGetValue(operand, out AstOperand astOperand)) + { + astOperand = new AstOperand(operand); + + _localsMap.Add(operand, astOperand); + + CurrentFunction.Locals.Add(astOperand); + } + + return astOperand; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/StructuredProgramInfo.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/StructuredProgramInfo.cs new file mode 100644 index 0000000..a47eae7 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/StructuredProgramInfo.cs @@ -0,0 +1,20 @@ +using System.Collections.Generic; + +namespace Ryujinx.Graphics.Shader.StructuredIr +{ + class StructuredProgramInfo + { + public List Functions { get; } + + public HashSet IoDefinitions { get; } + + public HelperFunctionsMask HelperFunctionsMask { get; set; } + + public StructuredProgramInfo() + { + Functions = new List(); + + IoDefinitions = new HashSet(); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/TextureDefinition.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/TextureDefinition.cs new file mode 100644 index 0000000..894f0ea --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/StructuredIr/TextureDefinition.cs @@ -0,0 +1,43 @@ +namespace Ryujinx.Graphics.Shader +{ + readonly struct TextureDefinition + { + public int Set { get; } + public int Binding { get; } + public int ArrayLength { get; } + public bool Separate { get; } + public string Name { get; } + public SamplerType Type { get; } + public TextureFormat Format { get; } + public TextureUsageFlags Flags { get; } + + public TextureDefinition( + int set, + int binding, + int arrayLength, + bool separate, + string name, + SamplerType type, + TextureFormat format, + TextureUsageFlags flags) + { + Set = set; + Binding = binding; + ArrayLength = arrayLength; + Separate = separate; + Name = name; + Type = type; + Format = format; + Flags = flags; + } + + public TextureDefinition(int set, int binding, string name, SamplerType type) : this(set, binding, 1, false, name, type, TextureFormat.Unknown, TextureUsageFlags.None) + { + } + + public TextureDefinition SetFlag(TextureUsageFlags flag) + { + return new TextureDefinition(Set, Binding, ArrayLength, Separate, Name, Type, Format, Flags | flag); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/SupportBuffer.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/SupportBuffer.cs new file mode 100644 index 0000000..7b3b225 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/SupportBuffer.cs @@ -0,0 +1,99 @@ +using Ryujinx.Common.Memory; +using Ryujinx.Graphics.Shader.StructuredIr; +using Ryujinx.Graphics.Shader.Translation; +using System.Runtime.CompilerServices; + +namespace Ryujinx.Graphics.Shader +{ + public struct Vector4 + { + public T X; + public T Y; + public T Z; + public T W; + } + + enum SupportBufferField + { + // Must match the order of the fields on the struct. + FragmentAlphaTest, + FragmentIsBgra, + ViewportInverse, + ViewportSize, + FragmentRenderScaleCount, + RenderScale, + TfeOffset, + TfeVertexCount, + } + + public struct SupportBuffer + { + public const int Binding = 0; + + public static readonly int FieldSize; + public static readonly int RequiredSize; + + public static readonly int FragmentAlphaTestOffset; + public static readonly int FragmentIsBgraOffset; + public static readonly int ViewportInverseOffset; + public static readonly int ViewportSizeOffset; + public static readonly int FragmentRenderScaleCountOffset; + public static readonly int GraphicsRenderScaleOffset; + public static readonly int ComputeRenderScaleOffset; + public static readonly int TfeOffsetOffset; + public static readonly int TfeVertexCountOffset; + + public const int FragmentIsBgraCount = 8; + // One for the render target, 64 for the textures, and 8 for the images. + public const int RenderScaleMaxCount = 1 + 64 + 8; + + private static int OffsetOf(ref SupportBuffer storage, ref T target) + { + return (int)Unsafe.ByteOffset(ref Unsafe.As(ref storage), ref target); + } + + static SupportBuffer() + { + FieldSize = Unsafe.SizeOf>(); + RequiredSize = Unsafe.SizeOf(); + + SupportBuffer instance = new(); + + FragmentAlphaTestOffset = OffsetOf(ref instance, ref instance.FragmentAlphaTest); + FragmentIsBgraOffset = OffsetOf(ref instance, ref instance.FragmentIsBgra); + ViewportInverseOffset = OffsetOf(ref instance, ref instance.ViewportInverse); + ViewportSizeOffset = OffsetOf(ref instance, ref instance.ViewportSize); + FragmentRenderScaleCountOffset = OffsetOf(ref instance, ref instance.FragmentRenderScaleCount); + GraphicsRenderScaleOffset = OffsetOf(ref instance, ref instance.RenderScale); + ComputeRenderScaleOffset = GraphicsRenderScaleOffset + FieldSize; + TfeOffsetOffset = OffsetOf(ref instance, ref instance.TfeOffset); + TfeVertexCountOffset = OffsetOf(ref instance, ref instance.TfeVertexCount); + } + + internal static StructureType GetStructureType() + { + return new StructureType([ + new StructureField(AggregateType.U32, "alpha_test"), + new StructureField(AggregateType.Array | AggregateType.U32, "is_bgra", FragmentIsBgraCount), + new StructureField(AggregateType.Vector4 | AggregateType.FP32, "viewport_inverse"), + new StructureField(AggregateType.Vector4 | AggregateType.FP32, "viewport_size"), + new StructureField(AggregateType.S32, "frag_scale_count"), + new StructureField(AggregateType.Array | AggregateType.FP32, "render_scale", RenderScaleMaxCount), + new StructureField(AggregateType.Vector4 | AggregateType.S32, "tfe_offset"), + new StructureField(AggregateType.S32, "tfe_vertex_count") + ]); + } + + public Vector4 FragmentAlphaTest; + public Array8> FragmentIsBgra; + public Vector4 ViewportInverse; + public Vector4 ViewportSize; + public Vector4 FragmentRenderScaleCount; + + // Render scale max count: 1 + 64 + 8. First scale is fragment output scale, others are textures/image inputs. + public Array73> RenderScale; + + public Vector4 TfeOffset; + public Vector4 TfeVertexCount; + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/TessPatchType.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/TessPatchType.cs new file mode 100644 index 0000000..c2d75de --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/TessPatchType.cs @@ -0,0 +1,22 @@ +namespace Ryujinx.Graphics.Shader +{ + public enum TessPatchType + { + Isolines = 0, + Triangles = 1, + Quads = 2, + } + + static class TessPatchTypeExtensions + { + extension(TessPatchType patchType) + { + public string Glsl => patchType switch + { + TessPatchType.Isolines => "isolines", + TessPatchType.Quads => "quads", + _ => "triangles", + }; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/TessSpacing.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/TessSpacing.cs new file mode 100644 index 0000000..9189a60 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/TessSpacing.cs @@ -0,0 +1,22 @@ +namespace Ryujinx.Graphics.Shader +{ + public enum TessSpacing + { + EqualSpacing = 0, + FractionalEventSpacing = 1, + FractionalOddSpacing = 2, + } + + static class TessSpacingExtensions + { + extension(TessSpacing spacing) + { + public string Glsl => spacing switch + { + TessSpacing.FractionalEventSpacing => "fractional_even_spacing", + TessSpacing.FractionalOddSpacing => "fractional_odd_spacing", + _ => "equal_spacing", + }; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/TextureDescriptor.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/TextureDescriptor.cs new file mode 100644 index 0000000..0059a02 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/TextureDescriptor.cs @@ -0,0 +1,43 @@ +namespace Ryujinx.Graphics.Shader +{ + public readonly struct TextureDescriptor + { + // New fields should be added to the end of the struct to keep disk shader cache compatibility. + + public readonly int Set; + public readonly int Binding; + + public readonly SamplerType Type; + public readonly TextureFormat Format; + + public readonly int CbufSlot; + public readonly int HandleIndex; + public readonly int ArrayLength; + + public readonly bool Separate; + + public readonly TextureUsageFlags Flags; + + public TextureDescriptor( + int set, + int binding, + SamplerType type, + TextureFormat format, + int cbufSlot, + int handleIndex, + int arrayLength, + bool separate, + TextureUsageFlags flags) + { + Set = set; + Binding = binding; + Type = type; + Format = format; + CbufSlot = cbufSlot; + HandleIndex = handleIndex; + ArrayLength = arrayLength; + Separate = separate; + Flags = flags; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/TextureFormat.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/TextureFormat.cs new file mode 100644 index 0000000..13281a4 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/TextureFormat.cs @@ -0,0 +1,130 @@ +using Ryujinx.Graphics.Shader.Translation; + +namespace Ryujinx.Graphics.Shader +{ + public enum TextureFormat + { + Unknown, + R8Unorm, + R8Snorm, + R8Uint, + R8Sint, + R16Float, + R16Unorm, + R16Snorm, + R16Uint, + R16Sint, + R32Float, + R32Uint, + R32Sint, + R8G8Unorm, + R8G8Snorm, + R8G8Uint, + R8G8Sint, + R16G16Float, + R16G16Unorm, + R16G16Snorm, + R16G16Uint, + R16G16Sint, + R32G32Float, + R32G32Uint, + R32G32Sint, + R8G8B8A8Unorm, + R8G8B8A8Snorm, + R8G8B8A8Uint, + R8G8B8A8Sint, + R16G16B16A16Float, + R16G16B16A16Unorm, + R16G16B16A16Snorm, + R16G16B16A16Uint, + R16G16B16A16Sint, + R32G32B32A32Float, + R32G32B32A32Uint, + R32G32B32A32Sint, + R10G10B10A2Unorm, + R10G10B10A2Uint, + R11G11B10Float, + } + + static class TextureFormatExtensions + { + public static string ToGlslFormat(this TextureFormat format) + { + return format switch + { +#pragma warning disable IDE0055 // Disable formatting + TextureFormat.R8Unorm => "r8", + TextureFormat.R8Snorm => "r8_snorm", + TextureFormat.R8Uint => "r8ui", + TextureFormat.R8Sint => "r8i", + TextureFormat.R16Float => "r16f", + TextureFormat.R16Unorm => "r16", + TextureFormat.R16Snorm => "r16_snorm", + TextureFormat.R16Uint => "r16ui", + TextureFormat.R16Sint => "r16i", + TextureFormat.R32Float => "r32f", + TextureFormat.R32Uint => "r32ui", + TextureFormat.R32Sint => "r32i", + TextureFormat.R8G8Unorm => "rg8", + TextureFormat.R8G8Snorm => "rg8_snorm", + TextureFormat.R8G8Uint => "rg8ui", + TextureFormat.R8G8Sint => "rg8i", + TextureFormat.R16G16Float => "rg16f", + TextureFormat.R16G16Unorm => "rg16", + TextureFormat.R16G16Snorm => "rg16_snorm", + TextureFormat.R16G16Uint => "rg16ui", + TextureFormat.R16G16Sint => "rg16i", + TextureFormat.R32G32Float => "rg32f", + TextureFormat.R32G32Uint => "rg32ui", + TextureFormat.R32G32Sint => "rg32i", + TextureFormat.R8G8B8A8Unorm => "rgba8", + TextureFormat.R8G8B8A8Snorm => "rgba8_snorm", + TextureFormat.R8G8B8A8Uint => "rgba8ui", + TextureFormat.R8G8B8A8Sint => "rgba8i", + TextureFormat.R16G16B16A16Float => "rgba16f", + TextureFormat.R16G16B16A16Unorm => "rgba16", + TextureFormat.R16G16B16A16Snorm => "rgba16_snorm", + TextureFormat.R16G16B16A16Uint => "rgba16ui", + TextureFormat.R16G16B16A16Sint => "rgba16i", + TextureFormat.R32G32B32A32Float => "rgba32f", + TextureFormat.R32G32B32A32Uint => "rgba32ui", + TextureFormat.R32G32B32A32Sint => "rgba32i", + TextureFormat.R10G10B10A2Unorm => "rgb10_a2", + TextureFormat.R10G10B10A2Uint => "rgb10_a2ui", + TextureFormat.R11G11B10Float => "r11f_g11f_b10f", + _ => string.Empty, +#pragma warning restore IDE0055 + }; + } + + public static AggregateType GetComponentType(this TextureFormat format) + { + switch (format) + { + case TextureFormat.R8Uint: + case TextureFormat.R16Uint: + case TextureFormat.R32Uint: + case TextureFormat.R8G8Uint: + case TextureFormat.R16G16Uint: + case TextureFormat.R32G32Uint: + case TextureFormat.R8G8B8A8Uint: + case TextureFormat.R16G16B16A16Uint: + case TextureFormat.R32G32B32A32Uint: + case TextureFormat.R10G10B10A2Uint: + return AggregateType.U32; + case TextureFormat.R8Sint: + case TextureFormat.R16Sint: + case TextureFormat.R32Sint: + case TextureFormat.R8G8Sint: + case TextureFormat.R16G16Sint: + case TextureFormat.R32G32Sint: + case TextureFormat.R8G8B8A8Sint: + case TextureFormat.R16G16B16A16Sint: + case TextureFormat.R32G32B32A32Sint: + return AggregateType.S32; + } + + return AggregateType.FP32; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/TextureHandle.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/TextureHandle.cs new file mode 100644 index 0000000..46f0741 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/TextureHandle.cs @@ -0,0 +1,125 @@ +using System; +using System.Runtime.CompilerServices; + +namespace Ryujinx.Graphics.Shader +{ + public enum TextureHandleType + { + CombinedSampler = 0, // Must be 0. + SeparateSamplerHandle = 1, + SeparateSamplerId = 2, + SeparateConstantSamplerHandle = 3, + Direct = 4, + } + + public static class TextureHandle + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int PackSlots(int cbufSlot0, int cbufSlot1) + { + return cbufSlot0 | ((cbufSlot1 + 1) << 16); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static (int, int) UnpackSlots(int slots, int defaultTextureBufferIndex) + { + int textureBufferIndex; + int samplerBufferIndex; + + if (slots < 0) + { + textureBufferIndex = defaultTextureBufferIndex; + samplerBufferIndex = textureBufferIndex; + } + else + { + uint high = (uint)slots >> 16; + + textureBufferIndex = (ushort)slots; + samplerBufferIndex = high != 0 ? (int)high - 1 : textureBufferIndex; + } + + return (textureBufferIndex, samplerBufferIndex); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int PackOffsets(int cbufOffset0, int cbufOffset1, TextureHandleType type) + { + return cbufOffset0 | (cbufOffset1 << 14) | ((int)type << 28); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static (int, int, TextureHandleType) UnpackOffsets(int handle) + { + return (handle & 0x3fff, (handle >> 14) & 0x3fff, (TextureHandleType)((uint)handle >> 28)); + } + + /// + /// Unpacks the texture ID from the real texture handle. + /// + /// The real texture handle + /// The texture ID + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int UnpackTextureId(int packedId) + { + return packedId & 0xfffff; + } + + /// + /// Unpacks the sampler ID from the real texture handle. + /// + /// The real texture handle + /// The sampler ID + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int UnpackSamplerId(int packedId) + { + return (packedId >> 20) & 0xfff; + } + + /// + /// Reads a packed texture and sampler ID (basically, the real texture handle) + /// from a given texture/sampler constant buffer. + /// + /// A word offset of the handle on the buffer (the "fake" shader handle) + /// The constant buffer to fetch texture IDs from + /// The constant buffer to fetch sampler IDs from + /// The packed texture and sampler ID (the real texture handle) + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int ReadPackedId(int wordOffset, ReadOnlySpan cachedTextureBuffer, ReadOnlySpan cachedSamplerBuffer) + { + (int textureWordOffset, int samplerWordOffset, TextureHandleType handleType) = UnpackOffsets(wordOffset); + + int handle = textureWordOffset < cachedTextureBuffer.Length ? cachedTextureBuffer[textureWordOffset] : 0; + + // The "wordOffset" (which is really the immediate value used on texture instructions on the shader) + // is a 13-bit value. However, in order to also support separate samplers and textures (which uses + // bindless textures on the shader), we extend it with another value on the higher 16 bits with + // another offset for the sampler. + // The shader translator has code to detect separate texture and sampler uses with a bindless texture, + // turn that into a regular texture access and produce those special handles with values on the higher 16 bits. + if (handleType != TextureHandleType.CombinedSampler) + { + int samplerHandle; + + if (handleType != TextureHandleType.SeparateConstantSamplerHandle) + { + samplerHandle = samplerWordOffset < cachedSamplerBuffer.Length ? cachedSamplerBuffer[samplerWordOffset] : 0; + } + else + { + samplerHandle = samplerWordOffset; + } + + if (handleType is TextureHandleType.SeparateSamplerId or + TextureHandleType.SeparateConstantSamplerHandle) + { + samplerHandle <<= 20; + } + + handle |= samplerHandle; + } + + return handle; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/TextureUsageFlags.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/TextureUsageFlags.cs new file mode 100644 index 0000000..9abe551 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/TextureUsageFlags.cs @@ -0,0 +1,19 @@ +using System; + +namespace Ryujinx.Graphics.Shader +{ + /// + /// Flags that indicate how a texture will be used in a shader. + /// + [Flags] + public enum TextureUsageFlags + { + None = 0, + + // Integer sampled textures must be noted for resolution scaling. + ResScaleUnsupported = 1 << 0, + NeedsScaleValue = 1 << 1, + ImageStore = 1 << 2, + ImageCoherent = 1 << 3, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/AggregateType.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/AggregateType.cs new file mode 100644 index 0000000..3665578 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/AggregateType.cs @@ -0,0 +1,59 @@ +using System; + +namespace Ryujinx.Graphics.Shader.Translation +{ + [Flags] + enum AggregateType + { + Invalid, + Void, + Bool, + FP32, + FP64, + S32, + U32, + + ElementTypeMask = 0xff, + + ElementCountShift = 8, + ElementCountMask = 3 << ElementCountShift, + + Scalar = 0 << ElementCountShift, + Vector2 = 1 << ElementCountShift, + Vector3 = 2 << ElementCountShift, + Vector4 = 3 << ElementCountShift, + + Array = 1 << 10, + } + + static class AggregateTypeExtensions + { + public static int GetSizeInBytes(this AggregateType type) + { + int elementSize = (type & AggregateType.ElementTypeMask) switch + { + AggregateType.Bool or + AggregateType.FP32 or + AggregateType.S32 or + AggregateType.U32 => 4, + AggregateType.FP64 => 8, + _ => 0, + }; + + switch (type & AggregateType.ElementCountMask) + { + case AggregateType.Vector2: + elementSize *= 2; + break; + case AggregateType.Vector3: + elementSize *= 3; + break; + case AggregateType.Vector4: + elementSize *= 4; + break; + } + + return elementSize; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/AttributeConsts.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/AttributeConsts.cs new file mode 100644 index 0000000..b28fd0e --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/AttributeConsts.cs @@ -0,0 +1,38 @@ +namespace Ryujinx.Graphics.Shader.Translation +{ + static class AttributeConsts + { + public const int PrimitiveId = 0x060; + public const int Layer = 0x064; + public const int ViewportIndex = 0x068; + public const int PositionX = 0x070; + public const int PositionY = 0x074; + public const int FrontColorDiffuseR = 0x280; + public const int BackColorDiffuseR = 0x2a0; + public const int ClipDistance0 = 0x2c0; + public const int ClipDistance1 = 0x2c4; + public const int ClipDistance2 = 0x2c8; + public const int ClipDistance3 = 0x2cc; + public const int ClipDistance4 = 0x2d0; + public const int ClipDistance5 = 0x2d4; + public const int ClipDistance6 = 0x2d8; + public const int ClipDistance7 = 0x2dc; + public const int FogCoord = 0x2e8; + public const int TessCoordX = 0x2f0; + public const int TessCoordY = 0x2f4; + public const int InstanceId = 0x2f8; + public const int VertexId = 0x2fc; + public const int TexCoordCount = 10; + public const int TexCoordBase = 0x300; + public const int TexCoordEnd = TexCoordBase + TexCoordCount * 16; + public const int ViewportMask = 0x3a0; + public const int FrontFacing = 0x3fc; + + public const int UserAttributesCount = 32; + public const int UserAttributeBase = 0x80; + public const int UserAttributeEnd = UserAttributeBase + UserAttributesCount * 16; + + public const int UserAttributePerPatchBase = 0x18; + public const int UserAttributePerPatchEnd = 0x200; + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/AttributeUsage.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/AttributeUsage.cs new file mode 100644 index 0000000..89377cc --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/AttributeUsage.cs @@ -0,0 +1,168 @@ +using System; +using System.Collections.Generic; +using System.Numerics; + +namespace Ryujinx.Graphics.Shader.Translation +{ + class AttributeUsage + { + public bool NextUsesFixedFuncAttributes { get; private set; } + public int UsedInputAttributes { get; private set; } + public int UsedOutputAttributes { get; private set; } + public HashSet UsedInputAttributesPerPatch { get; } + public HashSet UsedOutputAttributesPerPatch { get; } + public HashSet NextUsedInputAttributesPerPatch { get; private set; } + public int PassthroughAttributes { get; private set; } + private int _nextUsedInputAttributes; + private int _thisUsedInputAttributes; + private Dictionary _perPatchAttributeLocations; + private readonly IGpuAccessor _gpuAccessor; + + public UInt128 NextInputAttributesComponents { get; private set; } + public UInt128 ThisInputAttributesComponents { get; private set; } + + public AttributeUsage(IGpuAccessor gpuAccessor) + { + _gpuAccessor = gpuAccessor; + + UsedInputAttributesPerPatch = []; + UsedOutputAttributesPerPatch = []; + } + + public void SetInputUserAttribute(int index, int component) + { + int mask = 1 << index; + + UsedInputAttributes |= mask; + _thisUsedInputAttributes |= mask; + ThisInputAttributesComponents |= UInt128.One << (index * 4 + component); + } + + public void SetInputUserAttributePerPatch(int index) + { + UsedInputAttributesPerPatch.Add(index); + } + + public void SetOutputUserAttribute(int index) + { + UsedOutputAttributes |= 1 << index; + } + + public void SetOutputUserAttributePerPatch(int index) + { + UsedOutputAttributesPerPatch.Add(index); + } + + public void MergeFromtNextStage(bool gpPassthrough, bool nextUsesFixedFunctionAttributes, AttributeUsage nextStage) + { + NextInputAttributesComponents = nextStage.ThisInputAttributesComponents; + NextUsedInputAttributesPerPatch = nextStage.UsedInputAttributesPerPatch; + NextUsesFixedFuncAttributes = nextUsesFixedFunctionAttributes; + MergeOutputUserAttributes(gpPassthrough, nextStage.UsedInputAttributes, nextStage.UsedInputAttributesPerPatch); + + if (UsedOutputAttributesPerPatch.Count != 0) + { + // Regular and per-patch input/output locations can't overlap, + // so we must assign on our location using unused regular input/output locations. + + Dictionary locationsMap = new(); + + int freeMask = ~UsedOutputAttributes; + + foreach (int attr in UsedOutputAttributesPerPatch) + { + int location = BitOperations.TrailingZeroCount(freeMask); + if (location == 32) + { + _gpuAccessor.Log($"No enough free locations for patch input/output 0x{attr:X}."); + break; + } + + locationsMap.Add(attr, location); + freeMask &= ~(1 << location); + } + + // Both stages must agree on the locations, so use the same "map" for both. + _perPatchAttributeLocations = locationsMap; + nextStage._perPatchAttributeLocations = locationsMap; + } + } + + private void MergeOutputUserAttributes(bool gpPassthrough, int mask, IEnumerable perPatch) + { + _nextUsedInputAttributes = mask; + + if (gpPassthrough) + { + PassthroughAttributes = mask & ~UsedOutputAttributes; + } + else + { + UsedOutputAttributes |= mask; + UsedOutputAttributesPerPatch.UnionWith(perPatch); + } + } + + public int GetPerPatchAttributeLocation(int index) + { + if (_perPatchAttributeLocations == null || !_perPatchAttributeLocations.TryGetValue(index, out int location)) + { + return index; + } + + return location; + } + + public bool IsUsedOutputAttribute(int attr) + { + // The check for fixed function attributes on the next stage is conservative, + // returning false if the output is just not used by the next stage is also valid. + if (NextUsesFixedFuncAttributes && + attr >= AttributeConsts.UserAttributeBase && + attr < AttributeConsts.UserAttributeEnd) + { + int index = (attr - AttributeConsts.UserAttributeBase) >> 4; + return (_nextUsedInputAttributes & (1 << index)) != 0; + } + + return true; + } + + public int GetFreeUserAttribute(bool isOutput, int index) + { + int useMask = isOutput ? _nextUsedInputAttributes : _thisUsedInputAttributes; + int bit = -1; + + while (useMask != -1) + { + bit = BitOperations.TrailingZeroCount(~useMask); + + if (bit == 32) + { + bit = -1; + break; + } + else if (index < 1) + { + break; + } + + useMask |= 1 << bit; + index--; + } + + return bit; + } + + public void SetAllInputUserAttributes() + { + UsedInputAttributes |= Constants.AllAttributesMask; + ThisInputAttributesComponents |= ~UInt128.Zero >> (128 - Constants.MaxAttributes * 4); + } + + public void SetAllOutputUserAttributes() + { + UsedOutputAttributes |= Constants.AllAttributesMask; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/ControlFlowGraph.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/ControlFlowGraph.cs new file mode 100644 index 0000000..bc3c00d --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/ControlFlowGraph.cs @@ -0,0 +1,176 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System.Collections.Generic; + +namespace Ryujinx.Graphics.Shader.Translation +{ + class ControlFlowGraph + { + public BasicBlock[] Blocks { get; } + public BasicBlock[] PostOrderBlocks { get; } + public int[] PostOrderMap { get; } + + public ControlFlowGraph(BasicBlock[] blocks) + { + Blocks = blocks; + + HashSet visited = []; + + Stack blockStack = new(); + + List postOrderBlocks = new(blocks.Length); + + PostOrderMap = new int[blocks.Length]; + + visited.Add(blocks[0]); + + blockStack.Push(blocks[0]); + + while (blockStack.TryPop(out BasicBlock block)) + { + if (block.Next != null && visited.Add(block.Next)) + { + blockStack.Push(block); + blockStack.Push(block.Next); + } + else if (block.Branch != null && visited.Add(block.Branch)) + { + blockStack.Push(block); + blockStack.Push(block.Branch); + } + else + { + PostOrderMap[block.Index] = postOrderBlocks.Count; + + postOrderBlocks.Add(block); + } + } + + PostOrderBlocks = postOrderBlocks.ToArray(); + } + + public static ControlFlowGraph Create(Operation[] operations) + { + Dictionary labels = new(); + + List blocks = []; + + BasicBlock currentBlock = null; + + void NextBlock(BasicBlock nextBlock) + { + if (currentBlock != null && !EndsWithUnconditionalInst(currentBlock.GetLastOp())) + { + currentBlock.Next = nextBlock; + } + + currentBlock = nextBlock; + } + + void NewNextBlock() + { + BasicBlock block = new(blocks.Count); + + blocks.Add(block); + + NextBlock(block); + } + + bool needsNewBlock = true; + + for (int index = 0; index < operations.Length; index++) + { + Operation operation = operations[index]; + + if (operation.Inst == Instruction.MarkLabel) + { + Operand label = operation.Dest; + + if (labels.TryGetValue(label, out BasicBlock nextBlock)) + { + nextBlock.Index = blocks.Count; + + blocks.Add(nextBlock); + + NextBlock(nextBlock); + } + else + { + NewNextBlock(); + + labels.Add(label, currentBlock); + } + } + else + { + if (needsNewBlock) + { + NewNextBlock(); + } + + currentBlock.Operations.AddLast(operation); + } + + needsNewBlock = operation.Inst is Instruction.Branch or + Instruction.BranchIfTrue or + Instruction.BranchIfFalse; + + if (needsNewBlock) + { + Operand label = operation.Dest; + + if (!labels.TryGetValue(label, out BasicBlock branchBlock)) + { + branchBlock = new BasicBlock(); + + labels.Add(label, branchBlock); + } + + currentBlock.Branch = branchBlock; + } + } + + // Remove unreachable blocks. + bool hasUnreachable; + + do + { + hasUnreachable = false; + + for (int blkIndex = 1; blkIndex < blocks.Count; blkIndex++) + { + BasicBlock block = blocks[blkIndex]; + + if (block.Predecessors.Count == 0) + { + block.Next = null; + block.Branch = null; + blocks.RemoveAt(blkIndex--); + hasUnreachable = true; + } + else + { + block.Index = blkIndex; + } + } + } while (hasUnreachable); + + return new ControlFlowGraph(blocks.ToArray()); + } + + private static bool EndsWithUnconditionalInst(INode node) + { + if (node is Operation operation) + { + switch (operation.Inst) + { + case Instruction.Branch: + case Instruction.Discard: + case Instruction.Return: + return true; + } + } + + return false; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Dominance.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Dominance.cs new file mode 100644 index 0000000..8b4d9fc --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Dominance.cs @@ -0,0 +1,94 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; + +namespace Ryujinx.Graphics.Shader.Translation +{ + static class Dominance + { + // Those methods are an implementation of the algorithms on "A Simple, Fast Dominance Algorithm". + // https://www.cs.rice.edu/~keith/EMBED/dom.pdf + public static void FindDominators(ControlFlowGraph cfg) + { + BasicBlock Intersect(BasicBlock block1, BasicBlock block2) + { + while (block1 != block2) + { + while (cfg.PostOrderMap[block1.Index] < cfg.PostOrderMap[block2.Index]) + { + block1 = block1.ImmediateDominator; + } + + while (cfg.PostOrderMap[block2.Index] < cfg.PostOrderMap[block1.Index]) + { + block2 = block2.ImmediateDominator; + } + } + + return block1; + } + + cfg.Blocks[0].ImmediateDominator = cfg.Blocks[0]; + + bool modified; + + do + { + modified = false; + + for (int blkIndex = cfg.PostOrderBlocks.Length - 2; blkIndex >= 0; blkIndex--) + { + BasicBlock block = cfg.PostOrderBlocks[blkIndex]; + + BasicBlock newIDom = null; + + foreach (BasicBlock predecessor in block.Predecessors) + { + if (predecessor.ImmediateDominator != null) + { + if (newIDom != null) + { + newIDom = Intersect(predecessor, newIDom); + } + else + { + newIDom = predecessor; + } + } + } + + if (block.ImmediateDominator != newIDom) + { + block.ImmediateDominator = newIDom; + + modified = true; + } + } + } + while (modified); + } + + public static void FindDominanceFrontiers(BasicBlock[] blocks) + { + for (int blkIndex = 0; blkIndex < blocks.Length; blkIndex++) + { + BasicBlock block = blocks[blkIndex]; + + if (block.Predecessors.Count < 2) + { + continue; + } + + for (int pBlkIndex = 0; pBlkIndex < block.Predecessors.Count; pBlkIndex++) + { + BasicBlock current = block.Predecessors[pBlkIndex]; + + while (current != block.ImmediateDominator) + { + current.DominanceFrontiers.Add(block); + + current = current.ImmediateDominator; + } + } + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/EmitterContext.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/EmitterContext.cs new file mode 100644 index 0000000..a287de1 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/EmitterContext.cs @@ -0,0 +1,628 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Numerics; +using System.Runtime.CompilerServices; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Translation +{ + class EmitterContext + { + public DecodedProgram Program { get; } + public TranslatorContext TranslatorContext { get; } + public ResourceManager ResourceManager { get; } + + public bool VertexAsCompute { get; } + + public bool IsNonMain { get; } + + public Block CurrBlock { get; set; } + public InstOp CurrOp { get; set; } + + public int OperationsCount => _operations.Count; + + private readonly struct BrxTarget + { + public readonly Operand Selector; + public readonly int ExpectedValue; + public readonly ulong NextTargetAddress; + + public BrxTarget(Operand selector, int expectedValue, ulong nextTargetAddress) + { + Selector = selector; + ExpectedValue = expectedValue; + NextTargetAddress = nextTargetAddress; + } + } + + private class BlockLabel + { + public readonly Operand Label; + public BrxTarget BrxTarget; + + public BlockLabel(Operand label) + { + Label = label; + } + } + + private readonly List _operations; + private readonly Dictionary _labels; + + public EmitterContext() + { + _operations = []; + _labels = new Dictionary(); + } + + public EmitterContext( + TranslatorContext translatorContext, + ResourceManager resourceManager, + DecodedProgram program, + bool vertexAsCompute, + bool isNonMain) : this() + { + TranslatorContext = translatorContext; + ResourceManager = resourceManager; + Program = program; + VertexAsCompute = vertexAsCompute; + IsNonMain = isNonMain; + + EmitStart(); + } + + private void EmitStart() + { + if (TranslatorContext.Options.Flags.HasFlag(TranslationFlags.VertexA)) + { + return; + } + + // Vulkan requires the point size to be always written on the shader if the primitive topology is points. + // OpenGL requires the point size to be always written on the shader if PROGRAM_POINT_SIZE is set. + if (TranslatorContext.Definitions.Stage == ShaderStage.Vertex) + { + this.Store(StorageKind.Output, IoVariable.PointSize, null, ConstF(TranslatorContext.Definitions.PointSize)); + } + + if (VertexAsCompute) + { + int vertexInfoCbBinding = ResourceManager.Reservations.VertexInfoConstantBufferBinding; + int countFieldIndex = TranslatorContext.Stage == ShaderStage.Vertex + ? (int)VertexInfoBufferField.VertexCounts + : (int)VertexInfoBufferField.GeometryCounts; + + Operand outputVertexOffset = this.Load(StorageKind.Input, IoVariable.GlobalId, Const(0)); + Operand vertexCount = this.Load(StorageKind.ConstantBuffer, vertexInfoCbBinding, Const(countFieldIndex), Const(0)); + Operand isVertexOob = this.ICompareGreaterOrEqualUnsigned(outputVertexOffset, vertexCount); + + Operand lblVertexInBounds = Label(); + + this.BranchIfFalse(lblVertexInBounds, isVertexOob); + this.Return(); + this.MarkLabel(lblVertexInBounds); + + Operand outputInstanceOffset = this.Load(StorageKind.Input, IoVariable.GlobalId, Const(1)); + Operand instanceCount = this.Load(StorageKind.ConstantBuffer, vertexInfoCbBinding, Const((int)VertexInfoBufferField.VertexCounts), Const(1)); + Operand firstVertex = this.Load(StorageKind.ConstantBuffer, vertexInfoCbBinding, Const((int)VertexInfoBufferField.VertexCounts), Const(2)); + Operand firstInstance = this.Load(StorageKind.ConstantBuffer, vertexInfoCbBinding, Const((int)VertexInfoBufferField.VertexCounts), Const(3)); + Operand ibBaseOffset = this.Load(StorageKind.ConstantBuffer, vertexInfoCbBinding, Const((int)VertexInfoBufferField.GeometryCounts), Const(3)); + Operand isInstanceOob = this.ICompareGreaterOrEqualUnsigned(outputInstanceOffset, instanceCount); + + Operand lblInstanceInBounds = Label(); + + this.BranchIfFalse(lblInstanceInBounds, isInstanceOob); + this.Return(); + this.MarkLabel(lblInstanceInBounds); + + if (TranslatorContext.Stage == ShaderStage.Vertex) + { + Operand vertexIndexVr = Local(); + + this.TextureSample( + SamplerType.TextureBuffer, + TextureFlags.IntCoords, + ResourceManager.Reservations.GetIndexBufferTextureSetAndBinding(), + 1, + [vertexIndexVr], + [this.IAdd(ibBaseOffset, outputVertexOffset)]); + + this.Store(StorageKind.LocalMemory, ResourceManager.LocalVertexIndexVertexRateMemoryId, this.IAdd(firstVertex, vertexIndexVr)); + this.Store(StorageKind.LocalMemory, ResourceManager.LocalVertexIndexInstanceRateMemoryId, this.IAdd(firstInstance, outputInstanceOffset)); + } + else if (TranslatorContext.Stage == ShaderStage.Geometry) + { + int inputVertices = TranslatorContext.Definitions.InputTopology.InputVertexCount; + + Operand baseVertex = this.IMultiply(outputVertexOffset, Const(inputVertices)); + + for (int index = 0; index < inputVertices; index++) + { + Operand vertexIndex = Local(); + + this.TextureSample( + SamplerType.TextureBuffer, + TextureFlags.IntCoords, + ResourceManager.Reservations.GetTopologyRemapBufferTextureSetAndBinding(), + 1, + [vertexIndex], + [this.IAdd(baseVertex, Const(index))]); + + this.Store(StorageKind.LocalMemory, ResourceManager.LocalTopologyRemapMemoryId, Const(index), vertexIndex); + } + + this.Store(StorageKind.LocalMemory, ResourceManager.LocalGeometryOutputVertexCountMemoryId, Const(0)); + this.Store(StorageKind.LocalMemory, ResourceManager.LocalGeometryOutputIndexCountMemoryId, Const(0)); + } + } + } + + public T GetOp() where T : unmanaged + { + Debug.Assert(Unsafe.SizeOf() == sizeof(ulong)); + ulong op = CurrOp.RawOpCode; + return Unsafe.As(ref op); + } + + public Operand Add(Instruction inst, Operand dest = null, params Operand[] sources) + { + Operation operation = new(inst, dest, sources); + + _operations.Add(operation); + + return dest; + } + + public Operand Add(Instruction inst, StorageKind storageKind, Operand dest = null, params Operand[] sources) + { + Operation operation = new(inst, storageKind, dest, sources); + + _operations.Add(operation); + + return dest; + } + + public (Operand, Operand) Add(Instruction inst, (Operand, Operand) dest, params Operand[] sources) + { + Operand[] dests = [dest.Item1, dest.Item2]; + + Operation operation = new(inst, 0, dests, sources); + + Add(operation); + + return dest; + } + + public void Add(Operation operation) + { + _operations.Add(operation); + } + + public void MarkLabel(Operand label) + { + Add(Instruction.MarkLabel, label); + } + + public Operand GetLabel(ulong address) + { + return EnsureBlockLabel(address).Label; + } + + public void SetBrxTarget(ulong address, Operand selector, int targetValue, ulong nextTargetAddress) + { + BlockLabel blockLabel = EnsureBlockLabel(address); + Debug.Assert(blockLabel.BrxTarget.Selector == null); + blockLabel.BrxTarget = new BrxTarget(selector, targetValue, nextTargetAddress); + } + + public void EnterBlock(ulong address) + { + BlockLabel blockLabel = EnsureBlockLabel(address); + + MarkLabel(blockLabel.Label); + + BrxTarget brxTarget = blockLabel.BrxTarget; + + if (brxTarget.Selector != null) + { + this.BranchIfFalse(GetLabel(brxTarget.NextTargetAddress), this.ICompareEqual(brxTarget.Selector, Const(brxTarget.ExpectedValue))); + } + } + + private BlockLabel EnsureBlockLabel(ulong address) + { + if (!_labels.TryGetValue(address, out BlockLabel blockLabel)) + { + blockLabel = new BlockLabel(Label()); + + _labels.Add(address, blockLabel); + } + + return blockLabel; + } + + public void PrepareForVertexReturn() + { + // TODO: Support transform feedback emulation on stages other than vertex. + // Those stages might produce more primitives, so it needs a way to "compact" the output after it is written. + + if (!TranslatorContext.GpuAccessor.QueryHostSupportsTransformFeedback() && + TranslatorContext.GpuAccessor.QueryTransformFeedbackEnabled() && + TranslatorContext.Stage == ShaderStage.Vertex) + { + Operand vertexCount = this.Load(StorageKind.ConstantBuffer, SupportBuffer.Binding, Const((int)SupportBufferField.TfeVertexCount)); + + for (int tfbIndex = 0; tfbIndex < ResourceReservations.TfeBuffersCount; tfbIndex++) + { + ReadOnlySpan locations = TranslatorContext.GpuAccessor.QueryTransformFeedbackVaryingLocations(tfbIndex); + int stride = TranslatorContext.GpuAccessor.QueryTransformFeedbackStride(tfbIndex); + + Operand baseOffset = this.Load(StorageKind.ConstantBuffer, SupportBuffer.Binding, Const((int)SupportBufferField.TfeOffset), Const(tfbIndex)); + Operand baseVertex = this.Load(StorageKind.Input, IoVariable.BaseVertex); + Operand baseInstance = this.Load(StorageKind.Input, IoVariable.BaseInstance); + Operand vertexIndex = this.Load(StorageKind.Input, IoVariable.VertexIndex); + Operand instanceIndex = this.Load(StorageKind.Input, IoVariable.InstanceIndex); + + Operand outputVertexOffset = this.ISubtract(vertexIndex, baseVertex); + Operand outputInstanceOffset = this.ISubtract(instanceIndex, baseInstance); + + Operand outputBaseVertex = this.IMultiply(outputInstanceOffset, vertexCount); + + Operand vertexOffset = this.IMultiply(this.IAdd(outputBaseVertex, outputVertexOffset), Const(stride / 4)); + baseOffset = this.IAdd(baseOffset, vertexOffset); + + for (int j = 0; j < locations.Length; j++) + { + byte location = locations[j]; + if (location == 0xff) + { + continue; + } + + Operand offset = this.IAdd(baseOffset, Const(j)); + Operand value = Instructions.AttributeMap.GenerateAttributeLoad(this, null, location * 4, isOutput: true, isPerPatch: false); + + int binding = ResourceManager.Reservations.GetTfeBufferStorageBufferBinding(tfbIndex); + + this.Store(StorageKind.StorageBuffer, binding, Const(0), offset, value); + } + } + } + + if (TranslatorContext.Definitions.ViewportTransformDisable) + { + Operand x = this.Load(StorageKind.Output, IoVariable.Position, null, Const(0)); + Operand y = this.Load(StorageKind.Output, IoVariable.Position, null, Const(1)); + Operand xScale = this.Load(StorageKind.ConstantBuffer, SupportBuffer.Binding, Const((int)SupportBufferField.ViewportInverse), Const(0)); + Operand yScale = this.Load(StorageKind.ConstantBuffer, SupportBuffer.Binding, Const((int)SupportBufferField.ViewportInverse), Const(1)); + Operand negativeOne = ConstF(-1.0f); + + this.Store(StorageKind.Output, IoVariable.Position, null, Const(0), this.FPFusedMultiplyAdd(x, xScale, negativeOne)); + this.Store(StorageKind.Output, IoVariable.Position, null, Const(1), this.FPFusedMultiplyAdd(y, yScale, negativeOne)); + } + + if (TranslatorContext.Definitions.DepthMode && !TranslatorContext.GpuAccessor.QueryHostSupportsDepthClipControl()) + { + Operand z = this.Load(StorageKind.Output, IoVariable.Position, null, Const(2)); + Operand w = this.Load(StorageKind.Output, IoVariable.Position, null, Const(3)); + Operand halfW = this.FPMultiply(w, ConstF(0.5f)); + + this.Store(StorageKind.Output, IoVariable.Position, null, Const(2), this.FPFusedMultiplyAdd(z, ConstF(0.5f), halfW)); + } + } + + public void PrepareForVertexReturn(out Operand oldXLocal, out Operand oldYLocal, out Operand oldZLocal) + { + if (TranslatorContext.Definitions.ViewportTransformDisable) + { + oldXLocal = Local(); + this.Copy(oldXLocal, this.Load(StorageKind.Output, IoVariable.Position, null, Const(0))); + oldYLocal = Local(); + this.Copy(oldYLocal, this.Load(StorageKind.Output, IoVariable.Position, null, Const(1))); + } + else + { + oldXLocal = null; + oldYLocal = null; + } + + if (TranslatorContext.Definitions.DepthMode && !TranslatorContext.GpuAccessor.QueryHostSupportsDepthClipControl()) + { + oldZLocal = Local(); + this.Copy(oldZLocal, this.Load(StorageKind.Output, IoVariable.Position, null, Const(2))); + } + else + { + oldZLocal = null; + } + + PrepareForVertexReturn(); + } + + public bool PrepareForReturn() + { + if (IsNonMain) + { + return true; + } + + if (TranslatorContext.Definitions.LastInVertexPipeline && + (TranslatorContext.Definitions.Stage == ShaderStage.Vertex || TranslatorContext.Definitions.Stage == ShaderStage.TessellationEvaluation) && + (TranslatorContext.Options.Flags & TranslationFlags.VertexA) == 0) + { + PrepareForVertexReturn(); + } + else if (TranslatorContext.Definitions.Stage == ShaderStage.Geometry) + { + void WritePositionOutput(int primIndex) + { + Operand x = this.Load(StorageKind.Input, IoVariable.Position, Const(primIndex), Const(0)); + Operand y = this.Load(StorageKind.Input, IoVariable.Position, Const(primIndex), Const(1)); + Operand z = this.Load(StorageKind.Input, IoVariable.Position, Const(primIndex), Const(2)); + Operand w = this.Load(StorageKind.Input, IoVariable.Position, Const(primIndex), Const(3)); + + this.Store(StorageKind.Output, IoVariable.Position, null, Const(0), x); + this.Store(StorageKind.Output, IoVariable.Position, null, Const(1), y); + this.Store(StorageKind.Output, IoVariable.Position, null, Const(2), z); + this.Store(StorageKind.Output, IoVariable.Position, null, Const(3), w); + } + + void WriteUserDefinedOutput(int index, int primIndex) + { + Operand x = this.Load(StorageKind.Input, IoVariable.UserDefined, Const(index), Const(primIndex), Const(0)); + Operand y = this.Load(StorageKind.Input, IoVariable.UserDefined, Const(index), Const(primIndex), Const(1)); + Operand z = this.Load(StorageKind.Input, IoVariable.UserDefined, Const(index), Const(primIndex), Const(2)); + Operand w = this.Load(StorageKind.Input, IoVariable.UserDefined, Const(index), Const(primIndex), Const(3)); + + this.Store(StorageKind.Output, IoVariable.UserDefined, null, Const(index), Const(0), x); + this.Store(StorageKind.Output, IoVariable.UserDefined, null, Const(index), Const(1), y); + this.Store(StorageKind.Output, IoVariable.UserDefined, null, Const(index), Const(2), z); + this.Store(StorageKind.Output, IoVariable.UserDefined, null, Const(index), Const(3), w); + } + + if (TranslatorContext.Definitions.GpPassthrough && !TranslatorContext.GpuAccessor.QueryHostSupportsGeometryShaderPassthrough()) + { + int inputStart, inputEnd, inputStep; + + InputTopology topology = TranslatorContext.Definitions.InputTopology; + + if (topology == InputTopology.LinesAdjacency) + { + inputStart = 1; + inputEnd = 3; + inputStep = 1; + } + else if (topology == InputTopology.TrianglesAdjacency) + { + inputStart = 0; + inputEnd = 6; + inputStep = 2; + } + else + { + inputStart = 0; + inputEnd = topology.InputVertexCountNoAdjacency; + inputStep = 1; + } + + for (int primIndex = inputStart; primIndex < inputEnd; primIndex += inputStep) + { + WritePositionOutput(primIndex); + + int passthroughAttributes = TranslatorContext.AttributeUsage.PassthroughAttributes; + while (passthroughAttributes != 0) + { + int index = BitOperations.TrailingZeroCount(passthroughAttributes); + WriteUserDefinedOutput(index, primIndex); + passthroughAttributes &= ~(1 << index); + } + + this.EmitVertex(); + } + + this.EndPrimitive(); + } + } + else if (TranslatorContext.Definitions.Stage == ShaderStage.Fragment) + { + GenerateAlphaToCoverageDitherDiscard(); + + bool supportsBgra = TranslatorContext.GpuAccessor.QueryHostSupportsBgraFormat(); + + if (TranslatorContext.Definitions.OmapDepth) + { + Operand src = Register(TranslatorContext.GetDepthRegister(), RegisterType.Gpr); + + this.Store(StorageKind.Output, IoVariable.FragmentOutputDepth, null, src); + } + + AlphaTestOp alphaTestOp = TranslatorContext.Definitions.AlphaTestCompare; + + if (alphaTestOp != AlphaTestOp.Always) + { + if (alphaTestOp == AlphaTestOp.Never) + { + this.Discard(); + } + else if ((TranslatorContext.Definitions.OmapTargets & 8) != 0) + { + Instruction comparator = alphaTestOp switch + { + AlphaTestOp.Equal => Instruction.CompareEqual, + AlphaTestOp.Greater => Instruction.CompareGreater, + AlphaTestOp.GreaterOrEqual => Instruction.CompareGreaterOrEqual, + AlphaTestOp.Less => Instruction.CompareLess, + AlphaTestOp.LessOrEqual => Instruction.CompareLessOrEqual, + AlphaTestOp.NotEqual => Instruction.CompareNotEqual, + _ => 0, + }; + + Debug.Assert(comparator != 0, $"Invalid alpha test operation \"{alphaTestOp}\"."); + + Operand alpha = Register(3, RegisterType.Gpr); + Operand alphaRef = ConstF(TranslatorContext.Definitions.AlphaTestReference); + Operand alphaPass = Add(Instruction.FP32 | comparator, Local(), alpha, alphaRef); + Operand alphaPassLabel = Label(); + + this.BranchIfTrue(alphaPassLabel, alphaPass); + this.Discard(); + this.MarkLabel(alphaPassLabel); + } + } + + // We don't need to output anything if alpha test always fails. + if (alphaTestOp == AlphaTestOp.Never) + { + return false; + } + + int regIndexBase = 0; + + for (int rtIndex = 0; rtIndex < 8; rtIndex++) + { + for (int component = 0; component < 4; component++) + { + bool componentEnabled = (TranslatorContext.Definitions.OmapTargets & (1 << (rtIndex * 4 + component))) != 0; + if (!componentEnabled) + { + continue; + } + + Operand src = Register(regIndexBase + component, RegisterType.Gpr); + + // Perform B <-> R swap if needed, for BGRA formats (not supported on OpenGL). + if (!supportsBgra && (component == 0 || component == 2)) + { + Operand isBgra = this.Load(StorageKind.ConstantBuffer, SupportBuffer.Binding, Const((int)SupportBufferField.FragmentIsBgra), Const(rtIndex)); + + Operand lblIsBgra = Label(); + Operand lblEnd = Label(); + + this.BranchIfTrue(lblIsBgra, isBgra); + + this.Store(StorageKind.Output, IoVariable.FragmentOutputColor, null, Const(rtIndex), Const(component), src); + this.Branch(lblEnd); + + MarkLabel(lblIsBgra); + + this.Store(StorageKind.Output, IoVariable.FragmentOutputColor, null, Const(rtIndex), Const(2 - component), src); + + MarkLabel(lblEnd); + } + else + { + this.Store(StorageKind.Output, IoVariable.FragmentOutputColor, null, Const(rtIndex), Const(component), src); + } + } + + bool targetEnabled = (TranslatorContext.Definitions.OmapTargets & (0xf << (rtIndex * 4))) != 0; + if (targetEnabled) + { + regIndexBase += 4; + } + } + } + + if (VertexAsCompute) + { + if (TranslatorContext.Stage == ShaderStage.Vertex) + { + int vertexInfoCbBinding = ResourceManager.Reservations.VertexInfoConstantBufferBinding; + int vertexOutputSbBinding = ResourceManager.Reservations.VertexOutputStorageBufferBinding; + int stride = ResourceManager.Reservations.OutputSizePerInvocation; + + Operand vertexCount = this.Load(StorageKind.ConstantBuffer, vertexInfoCbBinding, Const((int)VertexInfoBufferField.VertexCounts), Const(0)); + + Operand outputVertexOffset = this.Load(StorageKind.Input, IoVariable.GlobalId, Const(0)); + Operand outputInstanceOffset = this.Load(StorageKind.Input, IoVariable.GlobalId, Const(1)); + + Operand outputBaseVertex = this.IMultiply(outputInstanceOffset, vertexCount); + + Operand baseOffset = this.IMultiply(this.IAdd(outputBaseVertex, outputVertexOffset), Const(stride)); + + for (int offset = 0; offset < stride; offset++) + { + Operand vertexOffset = this.IAdd(baseOffset, Const(offset)); + Operand value = this.Load(StorageKind.LocalMemory, ResourceManager.LocalVertexDataMemoryId, Const(offset)); + + this.Store(StorageKind.StorageBuffer, vertexOutputSbBinding, Const(0), vertexOffset, value); + } + } + else if (TranslatorContext.Stage == ShaderStage.Geometry) + { + Operand lblLoopHead = Label(); + Operand lblExit = Label(); + + this.MarkLabel(lblLoopHead); + + Operand writtenIndices = this.Load(StorageKind.LocalMemory, ResourceManager.LocalGeometryOutputIndexCountMemoryId); + + int maxIndicesPerPrimitiveInvocation = TranslatorContext.Definitions.GetGeometryOutputIndexBufferStridePerInstance(); + int maxIndicesPerPrimitive = maxIndicesPerPrimitiveInvocation * TranslatorContext.Definitions.ThreadsPerInputPrimitive; + + this.BranchIfTrue(lblExit, this.ICompareGreaterOrEqualUnsigned(writtenIndices, Const(maxIndicesPerPrimitiveInvocation))); + + int vertexInfoCbBinding = ResourceManager.Reservations.VertexInfoConstantBufferBinding; + + Operand primitiveIndex = this.Load(StorageKind.Input, IoVariable.GlobalId, Const(0)); + Operand instanceIndex = this.Load(StorageKind.Input, IoVariable.GlobalId, Const(1)); + Operand invocationId = this.Load(StorageKind.Input, IoVariable.GlobalId, Const(2)); + Operand vertexCount = this.Load(StorageKind.ConstantBuffer, vertexInfoCbBinding, Const((int)VertexInfoBufferField.VertexCounts), Const(0)); + Operand primitiveId = this.IAdd(this.IMultiply(instanceIndex, vertexCount), primitiveIndex); + Operand ibOffset = this.IMultiply(primitiveId, Const(maxIndicesPerPrimitive)); + ibOffset = this.IAdd(ibOffset, this.IMultiply(invocationId, Const(maxIndicesPerPrimitiveInvocation))); + ibOffset = this.IAdd(ibOffset, writtenIndices); + + this.Store(StorageKind.StorageBuffer, ResourceManager.Reservations.GeometryIndexOutputStorageBufferBinding, Const(0), ibOffset, Const(-1)); + this.Store(StorageKind.LocalMemory, ResourceManager.LocalGeometryOutputIndexCountMemoryId, this.IAdd(writtenIndices, Const(1))); + + this.Branch(lblLoopHead); + + this.MarkLabel(lblExit); + } + } + + return true; + } + + private void GenerateAlphaToCoverageDitherDiscard() + { + // If the feature is disabled, or alpha is not written, then we're done. + if (!TranslatorContext.Definitions.AlphaToCoverageDitherEnable || (TranslatorContext.Definitions.OmapTargets & 8) == 0) + { + return; + } + + // 11 11 11 10 10 10 10 00 + // 11 01 01 01 01 00 00 00 + Operand ditherMask = Const(unchecked((int)0xfbb99110u)); + + Operand fragCoordX = this.Load(StorageKind.Input, IoVariable.FragmentCoord, null, Const(0)); + Operand fragCoordY = this.Load(StorageKind.Input, IoVariable.FragmentCoord, null, Const(1)); + + Operand x = this.BitwiseAnd(this.FP32ConvertToU32(fragCoordX), Const(1)); + Operand y = this.BitwiseAnd(this.FP32ConvertToU32(fragCoordY), Const(1)); + Operand xy = this.BitwiseOr(x, this.ShiftLeft(y, Const(1))); + + Operand alpha = Register(3, RegisterType.Gpr); + Operand scaledAlpha = this.FPMultiply(this.FPSaturate(alpha), ConstF(8)); + Operand quantizedAlpha = this.IMinimumU32(this.FP32ConvertToU32(scaledAlpha), Const(7)); + Operand shift = this.BitwiseOr(this.ShiftLeft(quantizedAlpha, Const(2)), xy); + Operand opaque = this.BitwiseAnd(this.ShiftRightU32(ditherMask, shift), Const(1)); + + Operand a2cDitherEndLabel = Label(); + + this.BranchIfTrue(a2cDitherEndLabel, opaque); + this.Discard(); + this.MarkLabel(a2cDitherEndLabel); + } + + public Operation[] GetOperations() + { + return _operations.ToArray(); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/EmitterContextInsts.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/EmitterContextInsts.cs new file mode 100644 index 0000000..ed92f5d --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/EmitterContextInsts.cs @@ -0,0 +1,1045 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System; + +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Translation +{ + static class EmitterContextInsts + { + public static Operand AtomicAdd(this EmitterContext context, StorageKind storageKind, Operand a, Operand b, Operand c) + { + return context.Add(Instruction.AtomicAdd, storageKind, Local(), a, b, c); + } + + public static Operand AtomicAnd(this EmitterContext context, StorageKind storageKind, Operand a, Operand b, Operand c) + { + return context.Add(Instruction.AtomicAnd, storageKind, Local(), a, b, c); + } + + public static Operand AtomicCompareAndSwap(this EmitterContext context, StorageKind storageKind, Operand a, Operand b, Operand c, Operand d) + { + return context.Add(Instruction.AtomicCompareAndSwap, storageKind, Local(), a, b, c, d); + } + + public static Operand AtomicMaxS32(this EmitterContext context, StorageKind storageKind, Operand a, Operand b, Operand c) + { + return context.Add(Instruction.AtomicMaxS32, storageKind, Local(), a, b, c); + } + + public static Operand AtomicMaxU32(this EmitterContext context, StorageKind storageKind, Operand a, Operand b, Operand c) + { + return context.Add(Instruction.AtomicMaxU32, storageKind, Local(), a, b, c); + } + + public static Operand AtomicMinS32(this EmitterContext context, StorageKind storageKind, Operand a, Operand b, Operand c) + { + return context.Add(Instruction.AtomicMinS32, storageKind, Local(), a, b, c); + } + + public static Operand AtomicMinU32(this EmitterContext context, StorageKind storageKind, Operand a, Operand b, Operand c) + { + return context.Add(Instruction.AtomicMinU32, storageKind, Local(), a, b, c); + } + + public static Operand AtomicOr(this EmitterContext context, StorageKind storageKind, Operand a, Operand b, Operand c) + { + return context.Add(Instruction.AtomicOr, storageKind, Local(), a, b, c); + } + + public static Operand AtomicSwap(this EmitterContext context, StorageKind storageKind, Operand a, Operand b, Operand c) + { + return context.Add(Instruction.AtomicSwap, storageKind, Local(), a, b, c); + } + + public static Operand AtomicXor(this EmitterContext context, StorageKind storageKind, Operand a, Operand b, Operand c) + { + return context.Add(Instruction.AtomicXor, storageKind, Local(), a, b, c); + } + + public static Operand AtomicAdd(this EmitterContext context, StorageKind storageKind, int binding, Operand e0, Operand e1, Operand value) + { + return context.Add(Instruction.AtomicAdd, storageKind, Local(), Const(binding), e0, e1, value); + } + + public static Operand AtomicAnd(this EmitterContext context, StorageKind storageKind, int binding, Operand e0, Operand e1, Operand value) + { + return context.Add(Instruction.AtomicAnd, storageKind, Local(), Const(binding), e0, e1, value); + } + + public static Operand AtomicCompareAndSwap(this EmitterContext context, StorageKind storageKind, int binding, Operand e0, Operand compare, Operand value) + { + return context.Add(Instruction.AtomicCompareAndSwap, storageKind, Local(), Const(binding), e0, compare, value); + } + + public static Operand AtomicCompareAndSwap(this EmitterContext context, StorageKind storageKind, int binding, Operand e0, Operand e1, Operand compare, Operand value) + { + return context.Add(Instruction.AtomicCompareAndSwap, storageKind, Local(), Const(binding), e0, e1, compare, value); + } + + public static Operand AtomicMaxS32(this EmitterContext context, StorageKind storageKind, int binding, Operand e0, Operand e1, Operand value) + { + return context.Add(Instruction.AtomicMaxS32, storageKind, Local(), Const(binding), e0, e1, value); + } + + public static Operand AtomicMaxU32(this EmitterContext context, StorageKind storageKind, int binding, Operand e0, Operand e1, Operand value) + { + return context.Add(Instruction.AtomicMaxU32, storageKind, Local(), Const(binding), e0, e1, value); + } + + public static Operand AtomicMinS32(this EmitterContext context, StorageKind storageKind, int binding, Operand e0, Operand e1, Operand value) + { + return context.Add(Instruction.AtomicMinS32, storageKind, Local(), Const(binding), e0, e1, value); + } + + public static Operand AtomicMinU32(this EmitterContext context, StorageKind storageKind, int binding, Operand e0, Operand e1, Operand value) + { + return context.Add(Instruction.AtomicMinU32, storageKind, Local(), Const(binding), e0, e1, value); + } + + public static Operand AtomicOr(this EmitterContext context, StorageKind storageKind, int binding, Operand e0, Operand e1, Operand value) + { + return context.Add(Instruction.AtomicOr, storageKind, Local(), Const(binding), e0, e1, value); + } + + public static Operand AtomicSwap(this EmitterContext context, StorageKind storageKind, int binding, Operand e0, Operand e1, Operand value) + { + return context.Add(Instruction.AtomicSwap, storageKind, Local(), Const(binding), e0, e1, value); + } + + public static Operand AtomicXor(this EmitterContext context, StorageKind storageKind, int binding, Operand e0, Operand e1, Operand value) + { + return context.Add(Instruction.AtomicXor, storageKind, Local(), Const(binding), e0, e1, value); + } + + public static Operand Ballot(this EmitterContext context, Operand a, int index) + { + Operand dest = Local(); + + context.Add(new Operation(Instruction.Ballot, index, dest, a)); + + return dest; + } + + public static Operand Barrier(this EmitterContext context) + { + return context.Add(Instruction.Barrier); + } + + public static Operand BitCount(this EmitterContext context, Operand a) + { + return context.Add(Instruction.BitCount, Local(), a); + } + + public static Operand BitfieldExtractS32(this EmitterContext context, Operand a, Operand b, Operand c) + { + return context.Add(Instruction.BitfieldExtractS32, Local(), a, b, c); + } + + public static Operand BitfieldExtractU32(this EmitterContext context, Operand a, Operand b, Operand c) + { + return context.Add(Instruction.BitfieldExtractU32, Local(), a, b, c); + } + + public static Operand BitfieldInsert(this EmitterContext context, Operand a, Operand b, Operand c, Operand d) + { + return context.Add(Instruction.BitfieldInsert, Local(), a, b, c, d); + } + + public static Operand BitfieldReverse(this EmitterContext context, Operand a) + { + return context.Add(Instruction.BitfieldReverse, Local(), a); + } + + public static Operand BitwiseAnd(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.BitwiseAnd, Local(), a, b); + } + + public static Operand BitwiseExclusiveOr(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.BitwiseExclusiveOr, Local(), a, b); + } + + public static Operand BitwiseNot(this EmitterContext context, Operand a, bool invert) + { + if (invert) + { + a = context.BitwiseNot(a); + } + + return a; + } + + public static Operand BitwiseNot(this EmitterContext context, Operand a) + { + return context.Add(Instruction.BitwiseNot, Local(), a); + } + + public static Operand BitwiseOr(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.BitwiseOr, Local(), a, b); + } + + public static Operand Branch(this EmitterContext context, Operand d) + { + return context.Add(Instruction.Branch, d); + } + + public static Operand BranchIfFalse(this EmitterContext context, Operand d, Operand a) + { + return context.Add(Instruction.BranchIfFalse, d, a); + } + + public static Operand BranchIfTrue(this EmitterContext context, Operand d, Operand a) + { + return context.Add(Instruction.BranchIfTrue, d, a); + } + + public static Operand Call(this EmitterContext context, int funcId, bool returns, params Operand[] args) + { + Operand[] args2 = new Operand[args.Length + 1]; + + args2[0] = Const(funcId); + args.CopyTo(args2, 1); + + return context.Add(Instruction.Call, returns ? Local() : null, args2); + } + + public static Operand ConditionalSelect(this EmitterContext context, Operand a, Operand b, Operand c) + { + return context.Add(Instruction.ConditionalSelect, Local(), a, b, c); + } + + public static Operand Copy(this EmitterContext context, Operand a) + { + return context.Add(Instruction.Copy, Local(), a); + } + + public static void Copy(this EmitterContext context, Operand d, Operand a) + { + if (d.Type == OperandType.Constant) + { + return; + } + + context.Add(Instruction.Copy, d, a); + } + + public static Operand Discard(this EmitterContext context) + { + return context.Add(Instruction.Discard); + } + + public static Operand EmitVertex(this EmitterContext context) + { + return context.Add(Instruction.EmitVertex); + } + + public static Operand EndPrimitive(this EmitterContext context) + { + return context.Add(Instruction.EndPrimitive); + } + + public static Operand FindLSB(this EmitterContext context, Operand a) + { + return context.Add(Instruction.FindLSB, Local(), a); + } + + public static Operand FindMSBS32(this EmitterContext context, Operand a) + { + return context.Add(Instruction.FindMSBS32, Local(), a); + } + + public static Operand FindMSBU32(this EmitterContext context, Operand a) + { + return context.Add(Instruction.FindMSBU32, Local(), a); + } + + public static Operand FP32ConvertToFP64(this EmitterContext context, Operand a) + { + return context.Add(Instruction.ConvertFP32ToFP64, Local(), a); + } + + public static Operand FP64ConvertToFP32(this EmitterContext context, Operand a) + { + return context.Add(Instruction.ConvertFP64ToFP32, Local(), a); + } + + public static Operand FPAbsNeg(this EmitterContext context, Operand a, bool abs, bool neg, Instruction fpType = Instruction.FP32) + { + return context.FPNegate(context.FPAbsolute(a, abs, fpType), neg, fpType); + } + + public static Operand FPAbsolute(this EmitterContext context, Operand a, bool abs, Instruction fpType = Instruction.FP32) + { + if (abs) + { + a = context.FPAbsolute(a, fpType); + } + + return a; + } + + public static Operand FPAbsolute(this EmitterContext context, Operand a, Instruction fpType = Instruction.FP32) + { + return context.Add(fpType | Instruction.Absolute, Local(), a); + } + + public static Operand FPAdd(this EmitterContext context, Operand a, Operand b, Instruction fpType = Instruction.FP32) + { + return context.Add(fpType | Instruction.Add, Local(), a, b); + } + + public static Operand FPCeiling(this EmitterContext context, Operand a, Instruction fpType = Instruction.FP32) + { + return context.Add(fpType | Instruction.Ceiling, Local(), a); + } + + public static Operand FPCompareEqual(this EmitterContext context, Operand a, Operand b, Instruction fpType = Instruction.FP32) + { + return context.Add(fpType | Instruction.CompareEqual, Local(), a, b); + } + + public static Operand FPCompareLess(this EmitterContext context, Operand a, Operand b, Instruction fpType = Instruction.FP32) + { + return context.Add(fpType | Instruction.CompareLess, Local(), a, b); + } + + public static Operand FP32ConvertToS32(this EmitterContext context, Operand a) + { + return context.Add(Instruction.ConvertFP32ToS32, Local(), a); + } + + public static Operand FP32ConvertToU32(this EmitterContext context, Operand a) + { + return context.Add(Instruction.ConvertFP32ToU32, Local(), a); + } + + public static Operand FP64ConvertToS32(this EmitterContext context, Operand a) + { + return context.Add(Instruction.ConvertFP64ToS32, Local(), a); + } + + public static Operand FP64ConvertToU32(this EmitterContext context, Operand a) + { + return context.Add(Instruction.ConvertFP64ToU32, Local(), a); + } + + public static Operand FPCosine(this EmitterContext context, Operand a) + { + return context.Add(Instruction.FP32 | Instruction.Cosine, Local(), a); + } + + public static Operand FPDivide(this EmitterContext context, Operand a, Operand b, Instruction fpType = Instruction.FP32) + { + return context.Add(fpType | Instruction.Divide, Local(), a, b); + } + + public static Operand FPExponentB2(this EmitterContext context, Operand a) + { + return context.Add(Instruction.FP32 | Instruction.ExponentB2, Local(), a); + } + + public static Operand FPFloor(this EmitterContext context, Operand a, Instruction fpType = Instruction.FP32) + { + return context.Add(fpType | Instruction.Floor, Local(), a); + } + + public static Operand FPFusedMultiplyAdd(this EmitterContext context, Operand a, Operand b, Operand c, Instruction fpType = Instruction.FP32) + { + return context.Add(fpType | Instruction.FusedMultiplyAdd, Local(), a, b, c); + } + + public static Operand FPLogarithmB2(this EmitterContext context, Operand a) + { + return context.Add(Instruction.FP32 | Instruction.LogarithmB2, Local(), a); + } + + public static Operand FPMaximum(this EmitterContext context, Operand a, Operand b, Instruction fpType = Instruction.FP32) + { + return context.Add(fpType | Instruction.Maximum, Local(), a, b); + } + + public static Operand FPMinimum(this EmitterContext context, Operand a, Operand b, Instruction fpType = Instruction.FP32) + { + return context.Add(fpType | Instruction.Minimum, Local(), a, b); + } + + public static Operand FPModulo(this EmitterContext context, Operand a, Operand b, Instruction fpType = Instruction.FP32) + { + return context.Add(fpType | Instruction.Modulo, Local(), a, b); + } + + public static Operand FPMultiply(this EmitterContext context, Operand a, Operand b, Instruction fpType = Instruction.FP32) + { + return context.Add(fpType | Instruction.Multiply, Local(), a, b); + } + + public static Operand FPNegate(this EmitterContext context, Operand a, bool neg, Instruction fpType = Instruction.FP32) + { + if (neg) + { + a = context.FPNegate(a, fpType); + } + + return a; + } + + public static Operand FPNegate(this EmitterContext context, Operand a, Instruction fpType = Instruction.FP32) + { + return context.Add(fpType | Instruction.Negate, Local(), a); + } + + public static Operand FPReciprocal(this EmitterContext context, Operand a, Instruction fpType = Instruction.FP32) + { + return context.FPDivide(fpType == Instruction.FP64 ? context.PackDouble2x32(1.0) : ConstF(1), a, fpType); + } + + public static Operand FPReciprocalSquareRoot(this EmitterContext context, Operand a, Instruction fpType = Instruction.FP32) + { + return context.Add(fpType | Instruction.ReciprocalSquareRoot, Local(), a); + } + + public static Operand FPRound(this EmitterContext context, Operand a, Instruction fpType = Instruction.FP32) + { + return context.Add(fpType | Instruction.Round, Local(), a); + } + + public static Operand FPSaturate(this EmitterContext context, Operand a, bool sat, Instruction fpType = Instruction.FP32) + { + if (sat) + { + a = context.FPSaturate(a, fpType); + } + + return a; + } + + public static Operand FPSaturate(this EmitterContext context, Operand a, Instruction fpType = Instruction.FP32) + { + return fpType == Instruction.FP64 + ? context.Add(fpType | Instruction.Clamp, Local(), a, context.PackDouble2x32(0.0), context.PackDouble2x32(1.0)) + : context.Add(fpType | Instruction.Clamp, Local(), a, ConstF(0), ConstF(1)); + } + + public static Operand FPSine(this EmitterContext context, Operand a) + { + return context.Add(Instruction.FP32 | Instruction.Sine, Local(), a); + } + + public static Operand FPSquareRoot(this EmitterContext context, Operand a) + { + return context.Add(Instruction.FP32 | Instruction.SquareRoot, Local(), a); + } + + public static Operand FPSubtract(this EmitterContext context, Operand a, Operand b, Instruction fpType = Instruction.FP32) + { + return context.Add(fpType | Instruction.Subtract, Local(), a, b); + } + + public static Operand FPTruncate(this EmitterContext context, Operand a, Instruction fpType = Instruction.FP32) + { + return context.Add(fpType | Instruction.Truncate, Local(), a); + } + + public static Operand FPSwizzleAdd(this EmitterContext context, Operand a, Operand b, int mask) + { + return context.Add(Instruction.SwizzleAdd, Local(), a, b, Const(mask)); + } + + public static void FSIBegin(this EmitterContext context) + { + context.Add(Instruction.FSIBegin); + } + + public static void FSIEnd(this EmitterContext context) + { + context.Add(Instruction.FSIEnd); + } + + public static Operand GroupMemoryBarrier(this EmitterContext context) + { + return context.Add(Instruction.GroupMemoryBarrier); + } + + public static Operand IAbsNeg(this EmitterContext context, Operand a, bool abs, bool neg) + { + return context.INegate(context.IAbsolute(a, abs), neg); + } + + public static Operand IAbsolute(this EmitterContext context, Operand a, bool abs) + { + if (abs) + { + a = context.IAbsolute(a); + } + + return a; + } + + public static Operand IAbsolute(this EmitterContext context, Operand a) + { + return context.Add(Instruction.Absolute, Local(), a); + } + + public static Operand IAdd(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.Add, Local(), a, b); + } + + public static Operand IClampS32(this EmitterContext context, Operand a, Operand b, Operand c) + { + return context.Add(Instruction.Clamp, Local(), a, b, c); + } + + public static Operand IClampU32(this EmitterContext context, Operand a, Operand b, Operand c) + { + return context.Add(Instruction.ClampU32, Local(), a, b, c); + } + + public static Operand ICompareEqual(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.CompareEqual, Local(), a, b); + } + + public static Operand ICompareGreater(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.CompareGreater, Local(), a, b); + } + + public static Operand ICompareGreaterOrEqual(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.CompareGreaterOrEqual, Local(), a, b); + } + + public static Operand ICompareGreaterOrEqualUnsigned(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.CompareGreaterOrEqualU32, Local(), a, b); + } + + public static Operand ICompareGreaterUnsigned(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.CompareGreaterU32, Local(), a, b); + } + + public static Operand ICompareLess(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.CompareLess, Local(), a, b); + } + + public static Operand ICompareLessOrEqual(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.CompareLessOrEqual, Local(), a, b); + } + + public static Operand ICompareLessOrEqualUnsigned(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.CompareLessOrEqualU32, Local(), a, b); + } + + public static Operand ICompareLessUnsigned(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.CompareLessU32, Local(), a, b); + } + + public static Operand ICompareNotEqual(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.CompareNotEqual, Local(), a, b); + } + + public static Operand IConvertS32ToFP32(this EmitterContext context, Operand a) + { + return context.Add(Instruction.ConvertS32ToFP32, Local(), a); + } + + public static Operand IConvertS32ToFP64(this EmitterContext context, Operand a) + { + return context.Add(Instruction.ConvertS32ToFP64, Local(), a); + } + + public static Operand IConvertU32ToFP32(this EmitterContext context, Operand a) + { + return context.Add(Instruction.ConvertU32ToFP32, Local(), a); + } + + public static Operand IConvertU32ToFP64(this EmitterContext context, Operand a) + { + return context.Add(Instruction.ConvertU32ToFP64, Local(), a); + } + + public static Operand IMaximumS32(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.Maximum, Local(), a, b); + } + + public static Operand IMaximumU32(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.MaximumU32, Local(), a, b); + } + + public static Operand IMinimumS32(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.Minimum, Local(), a, b); + } + + public static Operand IMinimumU32(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.MinimumU32, Local(), a, b); + } + + public static Operand IMultiply(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.Multiply, Local(), a, b); + } + + public static Operand INegate(this EmitterContext context, Operand a, bool neg) + { + if (neg) + { + a = context.INegate(a); + } + + return a; + } + + public static Operand INegate(this EmitterContext context, Operand a) + { + return context.Add(Instruction.Negate, Local(), a); + } + + public static Operand ISubtract(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.Subtract, Local(), a, b); + } + + public static Operand ImageAtomic( + this EmitterContext context, + SamplerType type, + TextureFormat format, + TextureFlags flags, + SetBindingPair setAndBinding, + Operand[] sources) + { + Operand dest = Local(); + + context.Add(new TextureOperation( + Instruction.ImageAtomic, + type, + format, + flags, + setAndBinding.SetIndex, + setAndBinding.Binding, + 0, + [dest], + sources)); + + return dest; + } + + public static void ImageLoad( + this EmitterContext context, + SamplerType type, + TextureFormat format, + TextureFlags flags, + SetBindingPair setAndBinding, + int compMask, + Operand[] dests, + Operand[] sources) + { + context.Add(new TextureOperation( + Instruction.ImageLoad, + type, + format, + flags, + setAndBinding.SetIndex, + setAndBinding.Binding, + compMask, + dests, + sources)); + } + + public static void ImageStore( + this EmitterContext context, + SamplerType type, + TextureFormat format, + TextureFlags flags, + SetBindingPair setAndBinding, + Operand[] sources) + { + context.Add(new TextureOperation( + Instruction.ImageStore, + type, + format, + flags, + setAndBinding.SetIndex, + setAndBinding.Binding, + 0, + null, + sources)); + } + + public static Operand IsNan(this EmitterContext context, Operand a, Instruction fpType = Instruction.FP32) + { + return context.Add(fpType | Instruction.IsNan, Local(), a); + } + + public static Operand Load(this EmitterContext context, StorageKind storageKind, Operand e0, Operand e1) + { + return context.Add(Instruction.Load, storageKind, Local(), e0, e1); + } + + public static Operand Load(this EmitterContext context, StorageKind storageKind, int binding) + { + return context.Add(Instruction.Load, storageKind, Local(), Const(binding)); + } + + public static Operand Load(this EmitterContext context, StorageKind storageKind, int binding, Operand e0) + { + return context.Add(Instruction.Load, storageKind, Local(), Const(binding), e0); + } + + public static Operand Load(this EmitterContext context, StorageKind storageKind, int binding, Operand e0, Operand e1) + { + return context.Add(Instruction.Load, storageKind, Local(), Const(binding), e0, e1); + } + + public static Operand Load(this EmitterContext context, StorageKind storageKind, int binding, Operand e0, Operand e1, Operand e2) + { + return context.Add(Instruction.Load, storageKind, Local(), Const(binding), e0, e1, e2); + } + + public static Operand Load(this EmitterContext context, StorageKind storageKind, IoVariable ioVariable, Operand primVertex = null) + { + return primVertex != null + ? context.Load(storageKind, (int)ioVariable, primVertex) + : context.Load(storageKind, (int)ioVariable); + } + + public static Operand Load( + this EmitterContext context, + StorageKind storageKind, + IoVariable ioVariable, + Operand primVertex, + Operand elemIndex) + { + return primVertex != null + ? context.Load(storageKind, (int)ioVariable, primVertex, elemIndex) + : context.Load(storageKind, (int)ioVariable, elemIndex); + } + + public static Operand Load( + this EmitterContext context, + StorageKind storageKind, + IoVariable ioVariable, + Operand primVertex, + Operand arrayIndex, + Operand elemIndex) + { + return primVertex != null + ? context.Load(storageKind, (int)ioVariable, primVertex, arrayIndex, elemIndex) + : context.Load(storageKind, (int)ioVariable, arrayIndex, elemIndex); + } + + public static Operand Lod( + this EmitterContext context, + SamplerType type, + TextureFlags flags, + SetBindingPair setAndBinding, + int compIndex, + Operand[] sources) + { + Operand dest = Local(); + + context.Add(new TextureOperation( + Instruction.Lod, + type, + TextureFormat.Unknown, + flags, + setAndBinding.SetIndex, + setAndBinding.Binding, + compIndex, + [dest], + sources)); + + return dest; + } + + public static Operand MemoryBarrier(this EmitterContext context) + { + return context.Add(Instruction.MemoryBarrier); + } + + public static Operand MultiplyHighS32(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.MultiplyHighS32, Local(), a, b); + } + + public static Operand MultiplyHighU32(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.MultiplyHighU32, Local(), a, b); + } + + public static Operand PackDouble2x32(this EmitterContext context, double value) + { + long valueAsLong = BitConverter.DoubleToInt64Bits(value); + + return context.Add(Instruction.PackDouble2x32, Local(), Const((int)valueAsLong), Const((int)(valueAsLong >> 32))); + } + + public static Operand PackDouble2x32(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.PackDouble2x32, Local(), a, b); + } + + public static Operand PackHalf2x16(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.PackHalf2x16, Local(), a, b); + } + + public static void Return(this EmitterContext context) + { + context.Add(Instruction.Return); + } + + public static void Return(this EmitterContext context, Operand returnValue) + { + context.Add(Instruction.Return, null, returnValue); + } + + public static Operand ShiftLeft(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.ShiftLeft, Local(), a, b); + } + + public static Operand ShiftRightS32(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.ShiftRightS32, Local(), a, b); + } + + public static Operand ShiftRightU32(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.ShiftRightU32, Local(), a, b); + } + + public static Operand Shuffle(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.Shuffle, Local(), a, b); + } + + public static (Operand, Operand) Shuffle(this EmitterContext context, Operand a, Operand b, Operand c) + { + return context.Add(Instruction.Shuffle, (Local(), Local()), a, b, c); + } + + public static Operand ShuffleDown(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.ShuffleDown, Local(), a, b); + } + + public static (Operand, Operand) ShuffleDown(this EmitterContext context, Operand a, Operand b, Operand c) + { + return context.Add(Instruction.ShuffleDown, (Local(), Local()), a, b, c); + } + + public static Operand ShuffleUp(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.ShuffleUp, Local(), a, b); + } + + public static (Operand, Operand) ShuffleUp(this EmitterContext context, Operand a, Operand b, Operand c) + { + return context.Add(Instruction.ShuffleUp, (Local(), Local()), a, b, c); + } + + public static Operand ShuffleXor(this EmitterContext context, Operand a, Operand b) + { + return context.Add(Instruction.ShuffleXor, Local(), a, b); + } + + public static (Operand, Operand) ShuffleXor(this EmitterContext context, Operand a, Operand b, Operand c) + { + return context.Add(Instruction.ShuffleXor, (Local(), Local()), a, b, c); + } + + public static Operand Store(this EmitterContext context, StorageKind storageKind, Operand e0, Operand e1, Operand value) + { + return context.Add(Instruction.Store, storageKind, null, e0, e1, value); + } + + public static Operand Store(this EmitterContext context, StorageKind storageKind, int binding, Operand value) + { + return context.Add(Instruction.Store, storageKind, null, Const(binding), value); + } + + public static Operand Store(this EmitterContext context, StorageKind storageKind, int binding, Operand e0, Operand value) + { + return context.Add(Instruction.Store, storageKind, null, Const(binding), e0, value); + } + + public static Operand Store(this EmitterContext context, StorageKind storageKind, int binding, Operand e0, Operand e1, Operand value) + { + return context.Add(Instruction.Store, storageKind, null, Const(binding), e0, e1, value); + } + + public static Operand Store( + this EmitterContext context, + StorageKind storageKind, + IoVariable ioVariable, + Operand invocationId, + Operand value) + { + return invocationId != null + ? context.Add(Instruction.Store, storageKind, null, Const((int)ioVariable), invocationId, value) + : context.Add(Instruction.Store, storageKind, null, Const((int)ioVariable), value); + } + + public static Operand Store( + this EmitterContext context, + StorageKind storageKind, + IoVariable ioVariable, + Operand invocationId, + Operand elemIndex, + Operand value) + { + return invocationId != null + ? context.Add(Instruction.Store, storageKind, null, Const((int)ioVariable), invocationId, elemIndex, value) + : context.Add(Instruction.Store, storageKind, null, Const((int)ioVariable), elemIndex, value); + } + + public static Operand Store( + this EmitterContext context, + StorageKind storageKind, + IoVariable ioVariable, + Operand invocationId, + Operand arrayIndex, + Operand elemIndex, + Operand value) + { + return invocationId != null + ? context.Add(Instruction.Store, storageKind, null, Const((int)ioVariable), invocationId, arrayIndex, elemIndex, value) + : context.Add(Instruction.Store, storageKind, null, Const((int)ioVariable), arrayIndex, elemIndex, value); + } + + public static void TextureSample( + this EmitterContext context, + SamplerType type, + TextureFlags flags, + SetBindingPair setAndBinding, + int compMask, + Operand[] dests, + Operand[] sources) + { + context.Add(new TextureOperation( + Instruction.TextureSample, + type, + TextureFormat.Unknown, + flags, + setAndBinding.SetIndex, + setAndBinding.Binding, + compMask, + dests, + sources)); + } + + public static Operand TextureQuerySamples( + this EmitterContext context, + SamplerType type, + TextureFlags flags, + SetBindingPair setAndBinding, + Operand[] sources) + { + Operand dest = Local(); + + context.Add(new TextureOperation( + Instruction.TextureQuerySamples, + type, + TextureFormat.Unknown, + flags, + setAndBinding.SetIndex, + setAndBinding.Binding, + 0, + [dest], + sources)); + + return dest; + } + + public static Operand TextureQuerySize( + this EmitterContext context, + SamplerType type, + TextureFlags flags, + SetBindingPair setAndBinding, + int compIndex, + Operand[] sources) + { + Operand dest = Local(); + + context.Add(new TextureOperation( + Instruction.TextureQuerySize, + type, + TextureFormat.Unknown, + flags, + setAndBinding.SetIndex, + setAndBinding.Binding, + compIndex, + [dest], + sources)); + + return dest; + } + + public static Operand UnpackDouble2x32High(this EmitterContext context, Operand a) + { + return UnpackDouble2x32(context, a, 1); + } + + public static Operand UnpackDouble2x32Low(this EmitterContext context, Operand a) + { + return UnpackDouble2x32(context, a, 0); + } + + private static Operand UnpackDouble2x32(this EmitterContext context, Operand a, int index) + { + Operand dest = Local(); + + context.Add(new Operation(Instruction.UnpackDouble2x32, index, dest, a)); + + return dest; + } + + public static Operand UnpackHalf2x16High(this EmitterContext context, Operand a) + { + return UnpackHalf2x16(context, a, 1); + } + + public static Operand UnpackHalf2x16Low(this EmitterContext context, Operand a) + { + return UnpackHalf2x16(context, a, 0); + } + + private static Operand UnpackHalf2x16(this EmitterContext context, Operand a, int index) + { + Operand dest = Local(); + + context.Add(new Operation(Instruction.UnpackHalf2x16, index, dest, a)); + + return dest; + } + + public static Operand VoteAll(this EmitterContext context, Operand a) + { + return context.Add(Instruction.VoteAll, Local(), a); + } + + public static Operand VoteAllEqual(this EmitterContext context, Operand a) + { + return context.Add(Instruction.VoteAllEqual, Local(), a); + } + + public static Operand VoteAny(this EmitterContext context, Operand a) + { + return context.Add(Instruction.VoteAny, Local(), a); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/FeatureFlags.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/FeatureFlags.cs new file mode 100644 index 0000000..07e711f --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/FeatureFlags.cs @@ -0,0 +1,30 @@ +using System; + +namespace Ryujinx.Graphics.Shader.Translation +{ + /// + /// Features used by the shader that are important for the code generator to know in advance. + /// These typically change the declarations in the shader header. + /// + [Flags] + public enum FeatureFlags + { + None = 0, + + // Affected by resolution scaling. + FragCoordXY = 1 << 1, + + Bindless = 1 << 2, + InstanceId = 1 << 3, + DrawParameters = 1 << 4, + RtLayer = 1 << 5, + Shuffle = 1 << 6, + ViewportIndex = 1 << 7, + ViewportMask = 1 << 8, + FixedFuncAttr = 1 << 9, + LocalMemory = 1 << 10, + SharedMemory = 1 << 11, + Store = 1 << 12, + VtgAsCompute = 1 << 13, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/FunctionMatch.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/FunctionMatch.cs new file mode 100644 index 0000000..95b71ed --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/FunctionMatch.cs @@ -0,0 +1,865 @@ +using Ryujinx.Graphics.Shader.Decoders; +using System; +using System.Collections.Generic; +using System.Runtime.CompilerServices; + +namespace Ryujinx.Graphics.Shader.Translation +{ + static class FunctionMatch + { + private static readonly IPatternTreeNode[] _fsiGetAddressTree = PatternTrees.GetFsiGetAddress(); + private static readonly IPatternTreeNode[] _fsiGetAddressV2Tree = PatternTrees.GetFsiGetAddressV2(); + private static readonly IPatternTreeNode[] _fsiIsLastWarpThreadPatternTree = PatternTrees.GetFsiIsLastWarpThread(); + private static readonly IPatternTreeNode[] _fsiBeginPatternTree = PatternTrees.GetFsiBeginPattern(); + private static readonly IPatternTreeNode[] _fsiEndPatternTree = PatternTrees.GetFsiEndPattern(); + + public static void RunPass(DecodedProgram program) + { + byte[] externalRegs = new byte[4]; + bool hasGetAddress = false; + + foreach (DecodedFunction function in program) + { + if (function == program.MainFunction) + { + continue; + } + + int externalReg4 = 0; + + TreeNode[] functionTree = BuildTree(function.Blocks); + + if (Matches(_fsiGetAddressTree, functionTree)) + { + externalRegs[1] = functionTree[0].GetRd(); + externalRegs[2] = functionTree[2].GetRd(); + externalRegs[3] = functionTree[1].GetRd(); + externalReg4 = functionTree[3].GetRd(); + } + else if (Matches(_fsiGetAddressV2Tree, functionTree)) + { + externalRegs[1] = functionTree[2].GetRd(); + externalRegs[2] = functionTree[1].GetRd(); + externalRegs[3] = functionTree[0].GetRd(); + externalReg4 = functionTree[3].GetRd(); + } + + // Ensure the register allocation is valid. + // If so, then we have a match. + if (externalRegs[1] != externalRegs[2] && + externalRegs[2] != externalRegs[3] && + externalRegs[1] != externalRegs[3] && + externalRegs[1] + 1 != externalRegs[2] && + externalRegs[1] + 1 != externalRegs[3] && + externalRegs[1] + 1 == externalReg4 && + externalRegs[2] != RegisterConsts.RegisterZeroIndex && + externalRegs[3] != RegisterConsts.RegisterZeroIndex && + externalReg4 != RegisterConsts.RegisterZeroIndex) + { + hasGetAddress = true; + function.Type = FunctionType.Unused; + break; + } + } + + foreach (DecodedFunction function in program) + { + if (function.IsCompilerGenerated || function == program.MainFunction) + { + continue; + } + + if (hasGetAddress) + { + TreeNode[] functionTree = BuildTree(function.Blocks); + + if (MatchesFsi(_fsiBeginPatternTree, program, function, functionTree, externalRegs)) + { + function.Type = FunctionType.BuiltInFSIBegin; + continue; + } + else if (MatchesFsi(_fsiEndPatternTree, program, function, functionTree, externalRegs)) + { + function.Type = FunctionType.BuiltInFSIEnd; + continue; + } + } + } + } + + private readonly struct TreeNodeUse + { + public TreeNode Node { get; } + public int Index { get; } + public bool Inverted { get; } + + private TreeNodeUse(int index, bool inverted, TreeNode node) + { + Index = index; + Inverted = inverted; + Node = node; + } + + public TreeNodeUse(int index, TreeNode node) : this(index, false, node) + { + } + + public TreeNodeUse Flip() + { + return new TreeNodeUse(Index, !Inverted, Node); + } + } + + private enum TreeNodeType : byte + { + Op, + Label, + } + + private class TreeNode + { + public readonly InstOp Op; + public readonly List Uses; + public TreeNodeType Type { get; } + public byte Order { get; } + + public TreeNode(byte order) + { + Type = TreeNodeType.Label; + Order = order; + } + + public TreeNode(InstOp op, byte order) + { + Op = op; + Uses = []; + Type = TreeNodeType.Op; + Order = order; + } + + public byte GetPd() + { + return (byte)((Op.RawOpCode >> 3) & 7); + } + + public byte GetRd() + { + return (byte)Op.RawOpCode; + } + } + + private static TreeNode[] BuildTree(Block[] blocks) + { + List nodes = []; + + Dictionary labels = new(); + + TreeNodeUse[] predDefs = new TreeNodeUse[RegisterConsts.PredsCount]; + TreeNodeUse[] gprDefs = new TreeNodeUse[RegisterConsts.GprsCount]; + + void DefPred(byte predIndex, int index, TreeNode node) + { + if (predIndex != RegisterConsts.PredicateTrueIndex) + { + predDefs[predIndex] = new TreeNodeUse(index, node); + } + } + + void DefGpr(byte regIndex, int index, TreeNode node) + { + if (regIndex != RegisterConsts.RegisterZeroIndex) + { + gprDefs[regIndex] = new TreeNodeUse(index, node); + } + } + + TreeNodeUse UsePred(byte predIndex, bool predInv) + { + if (predIndex != RegisterConsts.PredicateTrueIndex) + { + TreeNodeUse use = predDefs[predIndex]; + + if (use.Node != null) + { + nodes.Remove(use.Node); + } + else + { + use = new TreeNodeUse(-(predIndex + 2), null); + } + + return predInv ? use.Flip() : use; + } + + return new TreeNodeUse(-1, null); + } + + TreeNodeUse UseGpr(byte regIndex) + { + if (regIndex != RegisterConsts.RegisterZeroIndex) + { + TreeNodeUse use = gprDefs[regIndex]; + + if (use.Node != null) + { + nodes.Remove(use.Node); + } + else + { + use = new TreeNodeUse(-(regIndex + 2), null); + } + + return use; + } + + return new TreeNodeUse(-1, null); + } + + byte order = 0; + + for (int index = 0; index < blocks.Length; index++) + { + Block block = blocks[index]; + + if (block.Predecessors.Count > 1) + { + TreeNode label = new(order++); + nodes.Add(label); + labels.Add(block.Address, label); + } + + for (int opIndex = 0; opIndex < block.OpCodes.Count; opIndex++) + { + InstOp op = block.OpCodes[opIndex]; + + TreeNode node = new(op, IsOrderDependant(op.Name) ? order : (byte)0); + + // Add uses. + + if (!op.Props.HasFlag(InstProps.NoPred)) + { + byte predIndex = (byte)((op.RawOpCode >> 16) & 7); + bool predInv = (op.RawOpCode & 0x80000) != 0; + node.Uses.Add(UsePred(predIndex, predInv)); + } + + if (op.Props.HasFlag(InstProps.Ps)) + { + byte predIndex = (byte)((op.RawOpCode >> 39) & 7); + bool predInv = (op.RawOpCode & 0x40000000000) != 0; + node.Uses.Add(UsePred(predIndex, predInv)); + } + + if (op.Props.HasFlag(InstProps.Ra)) + { + byte ra = (byte)(op.RawOpCode >> 8); + node.Uses.Add(UseGpr(ra)); + } + + if ((op.Props & (InstProps.Rb | InstProps.Rb2)) != 0) + { + byte rb = op.Props.HasFlag(InstProps.Rb2) ? (byte)op.RawOpCode : (byte)(op.RawOpCode >> 20); + node.Uses.Add(UseGpr(rb)); + } + + if (op.Props.HasFlag(InstProps.Rc)) + { + byte rc = (byte)(op.RawOpCode >> 39); + node.Uses.Add(UseGpr(rc)); + } + + if (op.Name == InstName.Bra && labels.TryGetValue(op.GetAbsoluteAddress(), out TreeNode label)) + { + node.Uses.Add(new TreeNodeUse(0, label)); + } + + // Make definitions. + + int defIndex = 0; + + InstProps pdType = op.Props & InstProps.PdMask; + + if (pdType != 0) + { + int bit = pdType switch + { + InstProps.Pd => 3, + InstProps.LPd => 48, + InstProps.SPd => 30, + InstProps.TPd => 51, + InstProps.VPd => 45, + _ => throw new InvalidOperationException($"Table has unknown predicate destination {pdType}."), + }; + + byte predIndex = (byte)((op.RawOpCode >> bit) & 7); + DefPred(predIndex, defIndex++, node); + } + + if (op.Props.HasFlag(InstProps.Rd)) + { + byte rd = (byte)op.RawOpCode; + DefGpr(rd, defIndex++, node); + } + + nodes.Add(node); + } + } + + return nodes.ToArray(); + } + + private static bool IsOrderDependant(InstName name) + { + switch (name) + { + case InstName.Atom: + case InstName.AtomCas: + case InstName.Atoms: + case InstName.AtomsCas: + case InstName.Ld: + case InstName.Ldg: + case InstName.Ldl: + case InstName.Lds: + case InstName.Suatom: + case InstName.SuatomB: + case InstName.SuatomB2: + case InstName.SuatomCas: + case InstName.SuatomCasB: + case InstName.Suld: + case InstName.SuldB: + case InstName.SuldD: + case InstName.SuldDB: + return true; + } + + return false; + } + + private interface IPatternTreeNode + { + List Uses { get; } + InstName Name { get; } + TreeNodeType Type { get; } + byte Order { get; } + bool IsImm { get; } + bool Matches(in InstOp opInfo); + } + + private readonly struct PatternTreeNodeUse + { + public IPatternTreeNode Node { get; } + public int Index { get; } + public bool Inverted { get; } + public PatternTreeNodeUse Inv => new(Index, !Inverted, Node); + + private PatternTreeNodeUse(int index, bool inverted, IPatternTreeNode node) + { + Index = index; + Inverted = inverted; + Node = node; + } + + public PatternTreeNodeUse(int index, IPatternTreeNode node) : this(index, false, node) + { + } + } + + private class PatternTreeNode : IPatternTreeNode + { + public List Uses { get; } + private readonly Func _match; + + public InstName Name { get; } + public TreeNodeType Type { get; } + public byte Order { get; } + public bool IsImm { get; } + public PatternTreeNodeUse Out => new(0, this); + + public PatternTreeNode(InstName name, Func match, TreeNodeType type = TreeNodeType.Op, byte order = 0, bool isImm = false) + { + Name = name; + _match = match; + Type = type; + Order = order; + IsImm = isImm; + Uses = []; + } + + public PatternTreeNode Use(PatternTreeNodeUse use) + { + Uses.Add(use); + return this; + } + + public PatternTreeNodeUse OutAt(int index) + { + return new PatternTreeNodeUse(index, this); + } + + public bool Matches(in InstOp opInfo) + { + if (opInfo.Name != Name) + { + return false; + } + + ulong rawOp = opInfo.RawOpCode; + T op = Unsafe.As(ref rawOp); + + if (!_match(op)) + { + return false; + } + + return true; + } + } + + private static bool MatchesFsi( + IPatternTreeNode[] pattern, + DecodedProgram program, + DecodedFunction function, + TreeNode[] functionTree, + byte[] externalRegs) + { + if (function.Blocks.Length == 0) + { + return false; + } + + InstOp callOp = function.Blocks[0].GetLastOp(); + + if (callOp.Name != InstName.Cal) + { + return false; + } + + DecodedFunction callTarget = program.GetFunctionByAddress(callOp.GetAbsoluteAddress()); + TreeNode[] callTargetTree; + + if (callTarget == null || !Matches(_fsiIsLastWarpThreadPatternTree, callTargetTree = BuildTree(callTarget.Blocks))) + { + return false; + } + + externalRegs[0] = callTargetTree[0].GetPd(); + + if (Matches(pattern, functionTree, externalRegs)) + { + callTarget.RemoveCaller(function); + return true; + } + + return false; + } + + private static bool Matches(IPatternTreeNode[] pTree, TreeNode[] cTree, byte[] externalRegs = null) + { + if (pTree.Length != cTree.Length) + { + return false; + } + + for (int index = 0; index < pTree.Length; index++) + { + if (!Matches(pTree[index], cTree[index], externalRegs)) + { + return false; + } + } + + return true; + } + + private static bool Matches(IPatternTreeNode pTreeNode, TreeNode cTreeNode, byte[] externalRegs) + { + if (!pTreeNode.Matches(in cTreeNode.Op) || + pTreeNode.Type != cTreeNode.Type || + pTreeNode.Order != cTreeNode.Order || + pTreeNode.IsImm != cTreeNode.Op.Props.HasFlag(InstProps.Ib)) + { + return false; + } + + if (pTreeNode.Type == TreeNodeType.Op) + { + if (pTreeNode.Uses.Count != cTreeNode.Uses.Count) + { + return false; + } + + for (int index = 0; index < pTreeNode.Uses.Count; index++) + { + PatternTreeNodeUse pUse = pTreeNode.Uses[index]; + TreeNodeUse cUse = cTreeNode.Uses[index]; + + if (pUse.Index <= -2) + { + if (externalRegs[-pUse.Index - 2] != (-cUse.Index - 2)) + { + return false; + } + } + else if (pUse.Index != cUse.Index) + { + return false; + } + + if (pUse.Inverted != cUse.Inverted || (pUse.Node == null) != (cUse.Node == null)) + { + return false; + } + + if (pUse.Node != null && !Matches(pUse.Node, cUse.Node, externalRegs)) + { + return false; + } + } + } + + return true; + } + + private static class PatternTrees + { + public static IPatternTreeNode[] GetFsiGetAddress() + { + PatternTreeNodeUse affinityValue = S2r(SReg.Affinity).Use(PT).Out; + PatternTreeNodeUse orderingTicketValue = S2r(SReg.OrderingTicket).Use(PT).Out; + + return + [ + Iscadd(cc: true, 2, 0, 404) + .Use(PT) + .Use(Iscadd(cc: false, 8) + .Use(PT) + .Use(Lop32i(LogicOp.And, 0xff) + .Use(PT) + .Use(affinityValue).Out) + .Use(Lop32i(LogicOp.And, 0xff) + .Use(PT) + .Use(orderingTicketValue).Out).Out), + ShrU32W(16) + .Use(PT) + .Use(orderingTicketValue), + Iadd32i(0x200) + .Use(PT) + .Use(Lop32i(LogicOp.And, 0xfe00) + .Use(PT) + .Use(orderingTicketValue).Out), + Iadd(x: true, 0, 405).Use(PT).Use(RZ), + Ret().Use(PT) + ]; + } + + public static IPatternTreeNode[] GetFsiGetAddressV2() + { + PatternTreeNodeUse affinityValue = S2r(SReg.Affinity).Use(PT).Out; + PatternTreeNodeUse orderingTicketValue = S2r(SReg.OrderingTicket).Use(PT).Out; + + return + [ + ShrU32W(16) + .Use(PT) + .Use(orderingTicketValue), + Iadd32i(0x200) + .Use(PT) + .Use(Lop32i(LogicOp.And, 0xfe00) + .Use(PT) + .Use(orderingTicketValue).Out), + Iscadd(cc: true, 2, 0, 404) + .Use(PT) + .Use(Bfi(0x808) + .Use(PT) + .Use(affinityValue) + .Use(Lop32i(LogicOp.And, 0xff) + .Use(PT) + .Use(orderingTicketValue).Out).Out), + Iadd(x: true, 0, 405).Use(PT).Use(RZ), + Ret().Use(PT) + ]; + } + + public static IPatternTreeNode[] GetFsiIsLastWarpThread() + { + PatternTreeNodeUse threadKillValue = S2r(SReg.ThreadKill).Use(PT).Out; + PatternTreeNodeUse laneIdValue = S2r(SReg.LaneId).Use(PT).Out; + + return + [ + IsetpU32(IComp.Eq) + .Use(PT) + .Use(PT) + .Use(FloU32() + .Use(PT) + .Use(Vote(VoteMode.Any) + .Use(PT) + .Use(IsetpU32(IComp.Ne) + .Use(PT) + .Use(PT) + .Use(Lop(negB: true, LogicOp.PassB) + .Use(PT) + .Use(RZ) + .Use(threadKillValue).OutAt(1)) + .Use(RZ).Out).OutAt(1)).Out) + .Use(laneIdValue), + Ret().Use(PT) + ]; + } + + public static IPatternTreeNode[] GetFsiBeginPattern() + { + PatternTreeNodeUse addressLowValue = CallArg(1); + + static PatternTreeNodeUse HighU16Equals(PatternTreeNodeUse x) + { + PatternTreeNodeUse expectedValue = CallArg(3); + + return IsetpU32(IComp.Eq) + .Use(PT) + .Use(PT) + .Use(ShrU32W(16).Use(PT).Use(x).Out) + .Use(expectedValue).Out; + } + + PatternTreeNode label; + + return + [ + Cal(), + Ret().Use(CallArg(0).Inv), + Ret() + .Use(HighU16Equals(LdgE(CacheOpLd.Cg, LsSize.B32) + .Use(PT) + .Use(addressLowValue).Out)), + label = Label(), + Bra() + .Use(HighU16Equals(LdgE(CacheOpLd.Cg, LsSize.B32, 1) + .Use(PT) + .Use(addressLowValue).Out).Inv) + .Use(label.Out), + Ret().Use(PT) + ]; + } + + public static IPatternTreeNode[] GetFsiEndPattern() + { + PatternTreeNodeUse voteResult = Vote(VoteMode.All).Use(PT).Use(PT).OutAt(1); + PatternTreeNodeUse popcResult = Popc().Use(PT).Use(voteResult).Out; + PatternTreeNodeUse threadKillValue = S2r(SReg.ThreadKill).Use(PT).Out; + PatternTreeNodeUse laneIdValue = S2r(SReg.LaneId).Use(PT).Out; + + PatternTreeNodeUse addressLowValue = CallArg(1); + PatternTreeNodeUse incrementValue = CallArg(2); + + return + [ + Cal(), + Ret().Use(CallArg(0).Inv), + Membar(Decoders.Membar.Vc).Use(PT), + Ret().Use(IsetpU32(IComp.Ne) + .Use(PT) + .Use(PT) + .Use(threadKillValue) + .Use(RZ).Out), + RedE(RedOp.Add, AtomSize.U32) + .Use(IsetpU32(IComp.Eq) + .Use(PT) + .Use(PT) + .Use(FloU32() + .Use(PT) + .Use(voteResult).Out) + .Use(laneIdValue).Out) + .Use(addressLowValue) + .Use(Xmad(XmadCop.Cbcc, psl: true, hiloA: true, hiloB: true) + .Use(PT) + .Use(incrementValue) + .Use(Xmad(XmadCop.Cfull, mrg: true, hiloB: true) + .Use(PT) + .Use(incrementValue) + .Use(popcResult) + .Use(RZ).Out) + .Use(Xmad(XmadCop.Cfull) + .Use(PT) + .Use(incrementValue) + .Use(popcResult) + .Use(RZ).Out).Out), + Ret().Use(PT) + ]; + } + + private static PatternTreeNode Bfi(int imm) + { + return new(InstName.Bfi, (op) => !op.WriteCC && op.Imm20 == imm, isImm: true); + } + + private static PatternTreeNode Bra() + { + return new(InstName.Bra, (op) => op.Ccc == Ccc.T && !op.Ca); + } + + private static PatternTreeNode Cal() + { + return new(InstName.Cal, (op) => !op.Ca && op.Inc); + } + + private static PatternTreeNode FloU32() + { + return new(InstName.Flo, (op) => !op.Signed && !op.Sh && !op.NegB && !op.WriteCC); + } + + private static PatternTreeNode Iadd(bool x, int cbufSlot, int cbufOffset) + { + return new(InstName.Iadd, (op) => + !op.Sat && + !op.WriteCC && + op.X == x && + op.AvgMode == AvgMode.NoNeg && + op.CbufSlot == cbufSlot && + op.CbufOffset == cbufOffset); + } + + private static PatternTreeNode Iadd32i(int imm) + { + return new(InstName.Iadd32i, (op) => !op.Sat && !op.WriteCC && !op.X && op.AvgMode == AvgMode.NoNeg && op.Imm32 == imm); + } + + private static PatternTreeNode Iscadd(bool cc, int imm) + { + return new(InstName.Iscadd, (op) => op.WriteCC == cc && op.AvgMode == AvgMode.NoNeg && op.Imm5 == imm); + } + + private static PatternTreeNode Iscadd(bool cc, int imm, int cbufSlot, int cbufOffset) + { + return new(InstName.Iscadd, (op) => + op.WriteCC == cc && + op.AvgMode == AvgMode.NoNeg && + op.Imm5 == imm && + op.CbufSlot == cbufSlot && + op.CbufOffset == cbufOffset); + } + + private static PatternTreeNode IsetpU32(IComp comp) + { + return new(InstName.Isetp, (op) => !op.Signed && op.IComp == comp && op.Bop == BoolOp.And); + } + + private static PatternTreeNode Label() + { + return new(InstName.Invalid, (op) => true, type: TreeNodeType.Label); + } + + private static PatternTreeNode Lop(bool negB, LogicOp logicOp) + { + return new(InstName.Lop, (op) => !op.NegA && op.NegB == negB && !op.WriteCC && !op.X && op.Lop == logicOp && op.PredicateOp == PredicateOp.F); + } + + private static PatternTreeNode Lop32i(LogicOp logicOp, int imm) + { + return new(InstName.Lop32i, (op) => !op.NegA && !op.NegB && !op.X && !op.WriteCC && op.LogicOp == logicOp && op.Imm32 == imm); + } + + private static PatternTreeNode Membar(Membar membar) + { + return new(InstName.Membar, (op) => op.Membar == membar); + } + + private static PatternTreeNode Popc() + { + return new(InstName.Popc, (op) => !op.NegB); + } + + private static PatternTreeNode Ret() + { + return new(InstName.Ret, (op) => op.Ccc == Ccc.T); + } + + private static PatternTreeNode S2r(SReg reg) + { + return new(InstName.S2r, (op) => op.SReg == reg); + } + + private static PatternTreeNode ShrU32W(int imm) + { + return new(InstName.Shr, (op) => !op.Signed && !op.Brev && op.M && op.XMode == 0 && op.Imm20 == imm, isImm: true); + } + + private static PatternTreeNode LdgE(CacheOpLd cacheOp, LsSize size, byte order = 0) + { + return new(InstName.Ldg, (op) => op.E && op.CacheOp == cacheOp && op.LsSize == size, order: order); + } + + private static PatternTreeNode RedE(RedOp redOp, AtomSize size, byte order = 0) + { + return new(InstName.Red, (op) => op.E && op.RedOp == redOp && op.RedSize == size, order: order); + } + + private static PatternTreeNode Vote(VoteMode mode) + { + return new(InstName.Vote, (op) => op.VoteMode == mode); + } + + private static PatternTreeNode Xmad(XmadCop cop, bool psl = false, bool mrg = false, bool hiloA = false, bool hiloB = false) + { + return new(InstName.Xmad, (op) => op.XmadCop == cop && op.Psl == psl && op.Mrg == mrg && op.HiloA == hiloA && op.HiloB == hiloB); + } + + private static PatternTreeNodeUse PT => PTOrRZ(); + private static PatternTreeNodeUse RZ => PTOrRZ(); + + private static PatternTreeNodeUse CallArg(int index) + { + return new PatternTreeNodeUse(-(index + 2), null); + } + + private static PatternTreeNodeUse PTOrRZ() + { + return new PatternTreeNodeUse(-1, null); + } + } + + private static void PrintTreeNode(TreeNode node, string indentation) + { + Console.WriteLine($" {node.Op.Name}"); + + for (int i = 0; i < node.Uses.Count; i++) + { + TreeNodeUse use = node.Uses[i]; + bool last = i == node.Uses.Count - 1; + char separator = last ? '`' : '|'; + + if (use.Node != null) + { + Console.Write($"{indentation} {separator}- ({(use.Inverted ? "INV " : string.Empty)}{use.Index})"); + PrintTreeNode(use.Node, indentation + (last ? " " : " | ")); + } + else + { + Console.WriteLine($"{indentation} {separator}- ({(use.Inverted ? "INV " : string.Empty)}{use.Index}) NULL"); + } + } + } + + private static void PrintTreeNode(IPatternTreeNode node, string indentation) + { + Console.WriteLine($" {node.Name}"); + + for (int i = 0; i < node.Uses.Count; i++) + { + PatternTreeNodeUse use = node.Uses[i]; + bool last = i == node.Uses.Count - 1; + char separator = last ? '`' : '|'; + + if (use.Node != null) + { + Console.Write($"{indentation} {separator}- ({(use.Inverted ? "INV " : string.Empty)}{use.Index})"); + PrintTreeNode(use.Node, indentation + (last ? " " : " | ")); + } + else + { + Console.WriteLine($"{indentation} {separator}- ({(use.Inverted ? "INV " : string.Empty)}{use.Index}) NULL"); + } + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/HelperFunctionManager.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/HelperFunctionManager.cs new file mode 100644 index 0000000..2031cfb --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/HelperFunctionManager.cs @@ -0,0 +1,474 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System; +using System.Collections.Generic; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Translation +{ + class HelperFunctionManager + { + private readonly List _functionList; + private readonly Dictionary _functionIds; + private readonly ShaderStage _stage; + + public HelperFunctionManager(List functionList, ShaderStage stage) + { + _functionList = functionList; + _functionIds = new Dictionary(); + _stage = stage; + } + + public int AddFunction(Function function) + { + int functionId = _functionList.Count; + _functionList.Add(function); + + return functionId; + } + + public int GetOrCreateFunctionId(HelperFunctionName functionName) + { + if (_functionIds.TryGetValue((int)functionName, out int functionId)) + { + return functionId; + } + + Function function = GenerateFunction(functionName); + functionId = AddFunction(function); + _functionIds.Add((int)functionName, functionId); + + return functionId; + } + + public int GetOrCreateFunctionId(HelperFunctionName functionName, int id) + { + int key = (int)functionName | (id << 16); + + if (_functionIds.TryGetValue(key, out int functionId)) + { + return functionId; + } + + Function function = GenerateFunction(functionName, id); + functionId = AddFunction(function); + _functionIds.Add(key, functionId); + + return functionId; + } + + public int GetOrCreateShuffleFunctionId(HelperFunctionName functionName, int subgroupSize) + { + if (_functionIds.TryGetValue((int)functionName, out int functionId)) + { + return functionId; + } + + Function function = GenerateShuffleFunction(functionName, subgroupSize); + functionId = AddFunction(function); + _functionIds.Add((int)functionName, functionId); + + return functionId; + } + + private Function GenerateFunction(HelperFunctionName functionName) + { + return functionName switch + { + HelperFunctionName.ConvertDoubleToFloat => GenerateConvertDoubleToFloatFunction(), + HelperFunctionName.ConvertFloatToDouble => GenerateConvertFloatToDoubleFunction(), + HelperFunctionName.TexelFetchScale => GenerateTexelFetchScaleFunction(), + HelperFunctionName.TextureSizeUnscale => GenerateTextureSizeUnscaleFunction(), + _ => throw new ArgumentException($"Invalid function name {functionName}"), + }; + } + + private static Function GenerateConvertDoubleToFloatFunction() + { + EmitterContext context = new(); + + Operand valueLow = Argument(0); + Operand valueHigh = Argument(1); + + Operand mantissaLow = context.BitwiseAnd(valueLow, Const(((1 << 22) - 1))); + Operand mantissa = context.ShiftRightU32(valueLow, Const(22)); + + mantissa = context.BitwiseOr(mantissa, context.ShiftLeft(context.BitwiseAnd(valueHigh, Const(0xfffff)), Const(10))); + mantissa = context.BitwiseOr(mantissa, context.ConditionalSelect(mantissaLow, Const(1), Const(0))); + + Operand exp = context.BitwiseAnd(context.ShiftRightU32(valueHigh, Const(20)), Const(0x7ff)); + Operand sign = context.ShiftRightS32(valueHigh, Const(31)); + + Operand resultSign = context.ShiftLeft(sign, Const(31)); + + Operand notZero = context.BitwiseOr(mantissa, exp); + + Operand lblNotZero = Label(); + + context.BranchIfTrue(lblNotZero, notZero); + + context.Return(resultSign); + + context.MarkLabel(lblNotZero); + + Operand notNaNOrInf = context.ICompareNotEqual(exp, Const(0x7ff)); + + mantissa = context.BitwiseOr(mantissa, Const(0x40000000)); + exp = context.ISubtract(exp, Const(0x381)); + + // Note: Overflow cases are not handled here and might produce incorrect results. + + Operand roundBits = context.BitwiseAnd(mantissa, Const(0x7f)); + Operand roundBitsXor64 = context.BitwiseExclusiveOr(roundBits, Const(0x40)); + mantissa = context.ShiftRightU32(context.IAdd(mantissa, Const(0x40)), Const(7)); + mantissa = context.BitwiseAnd(mantissa, context.ConditionalSelect(roundBitsXor64, Const(~0), Const(~1))); + + exp = context.ConditionalSelect(mantissa, exp, Const(0)); + exp = context.ConditionalSelect(notNaNOrInf, exp, Const(0xff)); + + Operand result = context.IAdd(context.IAdd(mantissa, context.ShiftLeft(exp, Const(23))), resultSign); + + context.Return(result); + + return new Function(ControlFlowGraph.Create(context.GetOperations()).Blocks, "ConvertDoubleToFloat", true, 2, 0); + } + + private static Function GenerateConvertFloatToDoubleFunction() + { + EmitterContext context = new(); + + Operand value = Argument(0); + + Operand mantissa = context.BitwiseAnd(value, Const(0x7fffff)); + Operand exp = context.BitwiseAnd(context.ShiftRightU32(value, Const(23)), Const(0xff)); + Operand sign = context.ShiftRightS32(value, Const(31)); + + Operand notNaNOrInf = context.ICompareNotEqual(exp, Const(0xff)); + Operand expNotZero = context.ICompareNotEqual(exp, Const(0)); + Operand notDenorm = context.BitwiseOr(expNotZero, context.ICompareEqual(mantissa, Const(0))); + + exp = context.IAdd(exp, Const(0x380)); + + Operand shiftDist = context.ISubtract(Const(32), context.FindMSBU32(mantissa)); + Operand normExp = context.ISubtract(context.ISubtract(Const(1), shiftDist), Const(1)); + Operand normMant = context.ShiftLeft(mantissa, shiftDist); + + exp = context.ConditionalSelect(notNaNOrInf, exp, Const(0x7ff)); + exp = context.ConditionalSelect(notDenorm, exp, normExp); + mantissa = context.ConditionalSelect(expNotZero, mantissa, normMant); + + Operand resultLow = context.ShiftLeft(mantissa, Const(29)); + Operand resultHigh = context.ShiftRightU32(mantissa, Const(3)); + + resultHigh = context.IAdd(resultHigh, context.ShiftLeft(exp, Const(20))); + resultHigh = context.IAdd(resultHigh, context.ShiftLeft(sign, Const(31))); + + context.Copy(Argument(1), resultLow); + context.Copy(Argument(2), resultHigh); + context.Return(); + + return new Function(ControlFlowGraph.Create(context.GetOperations()).Blocks, "ConvertFloatToDouble", false, 1, 2); + } + + private static Function GenerateFunction(HelperFunctionName functionName, int id) + { + return functionName switch + { + HelperFunctionName.SharedAtomicMaxS32 => GenerateSharedAtomicSigned(id, isMin: false), + HelperFunctionName.SharedAtomicMinS32 => GenerateSharedAtomicSigned(id, isMin: true), + HelperFunctionName.SharedStore8 => GenerateSharedStore8(id), + HelperFunctionName.SharedStore16 => GenerateSharedStore16(id), + _ => throw new ArgumentException($"Invalid function name {functionName}"), + }; + } + + private static Function GenerateSharedAtomicSigned(int id, bool isMin) + { + EmitterContext context = new(); + + Operand wordOffset = Argument(0); + Operand value = Argument(1); + + Operand result = GenerateSharedAtomicCasLoop(context, wordOffset, id, (memValue) => + { + return isMin + ? context.IMinimumS32(memValue, value) + : context.IMaximumS32(memValue, value); + }); + + context.Return(result); + + return new Function(ControlFlowGraph.Create(context.GetOperations()).Blocks, $"SharedAtomic{(isMin ? "Min" : "Max")}_{id}", true, 2, 0); + } + + private static Function GenerateSharedStore8(int id) + { + return GenerateSharedStore(id, 8); + } + + private static Function GenerateSharedStore16(int id) + { + return GenerateSharedStore(id, 16); + } + + private static Function GenerateSharedStore(int id, int bitSize) + { + EmitterContext context = new(); + + Operand offset = Argument(0); + Operand value = Argument(1); + + Operand wordOffset = context.ShiftRightU32(offset, Const(2)); + Operand bitOffset = GetBitOffset(context, offset); + + GenerateSharedAtomicCasLoop(context, wordOffset, id, (memValue) => + { + return context.BitfieldInsert(memValue, value, bitOffset, Const(bitSize)); + }); + + context.Return(); + + return new Function(ControlFlowGraph.Create(context.GetOperations()).Blocks, $"SharedStore{bitSize}_{id}", false, 2, 0); + } + + private static Function GenerateShuffleFunction(HelperFunctionName functionName, int subgroupSize) + { + return functionName switch + { + HelperFunctionName.Shuffle => GenerateShuffle(subgroupSize), + HelperFunctionName.ShuffleDown => GenerateShuffleDown(subgroupSize), + HelperFunctionName.ShuffleUp => GenerateShuffleUp(subgroupSize), + HelperFunctionName.ShuffleXor => GenerateShuffleXor(subgroupSize), + _ => throw new ArgumentException($"Invalid function name {functionName}"), + }; + } + + private static Function GenerateShuffle(int subgroupSize) + { + EmitterContext context = new(); + + Operand value = Argument(0); + Operand index = Argument(1); + Operand mask = Argument(2); + + Operand clamp = context.BitwiseAnd(mask, Const(0x1f)); + Operand segMask = context.BitwiseAnd(context.ShiftRightU32(mask, Const(8)), Const(0x1f)); + Operand minThreadId = context.BitwiseAnd(GenerateLoadSubgroupLaneId(context, subgroupSize), segMask); + Operand maxThreadId = context.BitwiseOr(context.BitwiseAnd(clamp, context.BitwiseNot(segMask)), minThreadId); + Operand srcThreadId = context.BitwiseOr(context.BitwiseAnd(index, context.BitwiseNot(segMask)), minThreadId); + Operand valid = context.ICompareLessOrEqualUnsigned(srcThreadId, maxThreadId); + + context.Copy(Argument(3), valid); + + Operand result = context.Shuffle(value, GenerateSubgroupShuffleIndex(context, srcThreadId, subgroupSize)); + + context.Return(context.ConditionalSelect(valid, result, value)); + + return new Function(ControlFlowGraph.Create(context.GetOperations()).Blocks, "Shuffle", true, 3, 1); + } + + private static Function GenerateShuffleDown(int subgroupSize) + { + EmitterContext context = new(); + + Operand value = Argument(0); + Operand index = Argument(1); + Operand mask = Argument(2); + + Operand clamp = context.BitwiseAnd(mask, Const(0x1f)); + Operand segMask = context.BitwiseAnd(context.ShiftRightU32(mask, Const(8)), Const(0x1f)); + Operand laneId = GenerateLoadSubgroupLaneId(context, subgroupSize); + Operand minThreadId = context.BitwiseAnd(laneId, segMask); + Operand maxThreadId = context.BitwiseOr(context.BitwiseAnd(clamp, context.BitwiseNot(segMask)), minThreadId); + Operand srcThreadId = context.IAdd(laneId, index); + Operand valid = context.ICompareLessOrEqualUnsigned(srcThreadId, maxThreadId); + + context.Copy(Argument(3), valid); + + Operand result = context.Shuffle(value, GenerateSubgroupShuffleIndex(context, srcThreadId, subgroupSize)); + + context.Return(context.ConditionalSelect(valid, result, value)); + + return new Function(ControlFlowGraph.Create(context.GetOperations()).Blocks, "ShuffleDown", true, 3, 1); + } + + private static Function GenerateShuffleUp(int subgroupSize) + { + EmitterContext context = new(); + + Operand value = Argument(0); + Operand index = Argument(1); + Operand mask = Argument(2); + + Operand segMask = context.BitwiseAnd(context.ShiftRightU32(mask, Const(8)), Const(0x1f)); + Operand laneId = GenerateLoadSubgroupLaneId(context, subgroupSize); + Operand minThreadId = context.BitwiseAnd(laneId, segMask); + Operand srcThreadId = context.ISubtract(laneId, index); + Operand valid = context.ICompareGreaterOrEqual(srcThreadId, minThreadId); + + context.Copy(Argument(3), valid); + + Operand result = context.Shuffle(value, GenerateSubgroupShuffleIndex(context, srcThreadId, subgroupSize)); + + context.Return(context.ConditionalSelect(valid, result, value)); + + return new Function(ControlFlowGraph.Create(context.GetOperations()).Blocks, "ShuffleUp", true, 3, 1); + } + + private static Function GenerateShuffleXor(int subgroupSize) + { + EmitterContext context = new(); + + Operand value = Argument(0); + Operand index = Argument(1); + Operand mask = Argument(2); + + Operand clamp = context.BitwiseAnd(mask, Const(0x1f)); + Operand segMask = context.BitwiseAnd(context.ShiftRightU32(mask, Const(8)), Const(0x1f)); + Operand laneId = GenerateLoadSubgroupLaneId(context, subgroupSize); + Operand minThreadId = context.BitwiseAnd(laneId, segMask); + Operand maxThreadId = context.BitwiseOr(context.BitwiseAnd(clamp, context.BitwiseNot(segMask)), minThreadId); + Operand srcThreadId = context.BitwiseExclusiveOr(laneId, index); + Operand valid = context.ICompareLessOrEqualUnsigned(srcThreadId, maxThreadId); + + context.Copy(Argument(3), valid); + + Operand result = context.Shuffle(value, GenerateSubgroupShuffleIndex(context, srcThreadId, subgroupSize)); + + context.Return(context.ConditionalSelect(valid, result, value)); + + return new Function(ControlFlowGraph.Create(context.GetOperations()).Blocks, "ShuffleXor", true, 3, 1); + } + + private static Operand GenerateLoadSubgroupLaneId(EmitterContext context, int subgroupSize) + { + if (subgroupSize <= 32) + { + return context.Load(StorageKind.Input, IoVariable.SubgroupLaneId); + } + + return context.BitwiseAnd(context.Load(StorageKind.Input, IoVariable.SubgroupLaneId), Const(0x1f)); + } + + private static Operand GenerateSubgroupShuffleIndex(EmitterContext context, Operand srcThreadId, int subgroupSize) + { + if (subgroupSize <= 32) + { + return srcThreadId; + } + + return context.BitwiseOr( + context.BitwiseAnd(context.Load(StorageKind.Input, IoVariable.SubgroupLaneId), Const(0x60)), + srcThreadId); + } + + private Function GenerateTexelFetchScaleFunction() + { + EmitterContext context = new(); + + Operand input = Argument(0); + Operand samplerIndex = Argument(1); + Operand index = GetScaleIndex(context, samplerIndex); + + Operand scale = context.Load(StorageKind.ConstantBuffer, 0, Const((int)SupportBufferField.RenderScale), index); + + Operand scaleIsOne = context.FPCompareEqual(scale, ConstF(1f)); + Operand lblScaleNotOne = Label(); + + context.BranchIfFalse(lblScaleNotOne, scaleIsOne); + context.Return(input); + context.MarkLabel(lblScaleNotOne); + + int inArgumentsCount; + + if (_stage == ShaderStage.Fragment) + { + Operand scaleIsLessThanZero = context.FPCompareLess(scale, ConstF(0f)); + Operand lblScaleGreaterOrEqualZero = Label(); + + context.BranchIfFalse(lblScaleGreaterOrEqualZero, scaleIsLessThanZero); + + Operand negScale = context.FPNegate(scale); + Operand inputScaled = context.FPMultiply(context.IConvertS32ToFP32(input), negScale); + Operand fragCoordX = context.Load(StorageKind.Input, IoVariable.FragmentCoord, null, Const(0)); + Operand fragCoordY = context.Load(StorageKind.Input, IoVariable.FragmentCoord, null, Const(1)); + Operand fragCoord = context.ConditionalSelect(Argument(2), fragCoordY, fragCoordX); + Operand inputBias = context.FPModulo(fragCoord, negScale); + Operand inputWithBias = context.FPAdd(inputScaled, inputBias); + + context.Return(context.FP32ConvertToS32(inputWithBias)); + context.MarkLabel(lblScaleGreaterOrEqualZero); + + inArgumentsCount = 3; + } + else + { + inArgumentsCount = 2; + } + + Operand inputScaled2 = context.FPMultiply(context.IConvertS32ToFP32(input), scale); + + context.Return(context.FP32ConvertToS32(inputScaled2)); + + return new Function(ControlFlowGraph.Create(context.GetOperations()).Blocks, "TexelFetchScale", true, inArgumentsCount, 0); + } + + private Function GenerateTextureSizeUnscaleFunction() + { + EmitterContext context = new(); + + Operand input = Argument(0); + Operand samplerIndex = Argument(1); + Operand index = GetScaleIndex(context, samplerIndex); + + Operand scale = context.FPAbsolute(context.Load(StorageKind.ConstantBuffer, 0, Const((int)SupportBufferField.RenderScale), index)); + + Operand scaleIsOne = context.FPCompareEqual(scale, ConstF(1f)); + Operand lblScaleNotOne = Label(); + + context.BranchIfFalse(lblScaleNotOne, scaleIsOne); + context.Return(input); + context.MarkLabel(lblScaleNotOne); + + Operand inputUnscaled = context.FPDivide(context.IConvertS32ToFP32(input), scale); + + context.Return(context.FP32ConvertToS32(inputUnscaled)); + + return new Function(ControlFlowGraph.Create(context.GetOperations()).Blocks, "TextureSizeUnscale", true, 2, 0); + } + + private Operand GetScaleIndex(EmitterContext context, Operand index) + { + switch (_stage) + { + case ShaderStage.Vertex: + Operand fragScaleCount = context.Load(StorageKind.ConstantBuffer, 0, Const((int)SupportBufferField.FragmentRenderScaleCount)); + return context.IAdd(Const(1), context.IAdd(index, fragScaleCount)); + default: + return context.IAdd(Const(1), index); + } + } + + public static Operand GetBitOffset(EmitterContext context, Operand offset) + { + return context.ShiftLeft(context.BitwiseAnd(offset, Const(3)), Const(3)); + } + + private static Operand GenerateSharedAtomicCasLoop(EmitterContext context, Operand wordOffset, int id, Func opCallback) + { + Operand lblLoopHead = Label(); + + context.MarkLabel(lblLoopHead); + + Operand oldValue = context.Load(StorageKind.SharedMemory, id, wordOffset); + Operand newValue = opCallback(oldValue); + + Operand casResult = context.AtomicCompareAndSwap(StorageKind.SharedMemory, id, wordOffset, oldValue, newValue); + + Operand casFail = context.ICompareNotEqual(casResult, oldValue); + + context.BranchIfTrue(lblLoopHead, casFail); + + return oldValue; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/HelperFunctionName.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/HelperFunctionName.cs new file mode 100644 index 0000000..7da2f34 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/HelperFunctionName.cs @@ -0,0 +1,20 @@ +namespace Ryujinx.Graphics.Shader.Translation +{ + enum HelperFunctionName + { + Invalid, + + ConvertDoubleToFloat, + ConvertFloatToDouble, + SharedAtomicMaxS32, + SharedAtomicMinS32, + SharedStore8, + SharedStore16, + Shuffle, + ShuffleDown, + ShuffleUp, + ShuffleXor, + TexelFetchScale, + TextureSizeUnscale, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/HostCapabilities.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/HostCapabilities.cs new file mode 100644 index 0000000..e3c76f5 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/HostCapabilities.cs @@ -0,0 +1,40 @@ +namespace Ryujinx.Graphics.Shader.Translation +{ + class HostCapabilities + { + public readonly bool ReducedPrecision; + public readonly bool SupportsFragmentShaderInterlock; + public readonly bool SupportsFragmentShaderOrderingIntel; + public readonly bool SupportsGeometryShaderPassthrough; + public readonly bool SupportsShaderBallot; + public readonly bool SupportsShaderBarrierDivergence; + public readonly bool SupportsShaderFloat64; + public readonly bool SupportsShaderNonUniformIndexing; + public readonly bool SupportsTextureShadowLod; + public readonly bool SupportsViewportMask; + + public HostCapabilities( + bool reducedPrecision, + bool supportsFragmentShaderInterlock, + bool supportsFragmentShaderOrderingIntel, + bool supportsGeometryShaderPassthrough, + bool supportsShaderBallot, + bool supportsShaderBarrierDivergence, + bool supportsShaderFloat64, + bool supportsShaderNonUniformIndexing, + bool supportsTextureShadowLod, + bool supportsViewportMask) + { + ReducedPrecision = reducedPrecision; + SupportsFragmentShaderInterlock = supportsFragmentShaderInterlock; + SupportsFragmentShaderOrderingIntel = supportsFragmentShaderOrderingIntel; + SupportsGeometryShaderPassthrough = supportsGeometryShaderPassthrough; + SupportsShaderBallot = supportsShaderBallot; + SupportsShaderBarrierDivergence = supportsShaderBarrierDivergence; + SupportsShaderFloat64 = supportsShaderFloat64; + SupportsShaderNonUniformIndexing = supportsShaderNonUniformIndexing; + SupportsTextureShadowLod = supportsTextureShadowLod; + SupportsViewportMask = supportsViewportMask; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/IoUsage.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/IoUsage.cs new file mode 100644 index 0000000..f342773 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/IoUsage.cs @@ -0,0 +1,28 @@ +namespace Ryujinx.Graphics.Shader.Translation +{ + readonly struct IoUsage + { + private readonly FeatureFlags _usedFeatures; + + public readonly bool UsesRtLayer => _usedFeatures.HasFlag(FeatureFlags.RtLayer); + public readonly bool UsesViewportIndex => _usedFeatures.HasFlag(FeatureFlags.ViewportIndex); + public readonly bool UsesViewportMask => _usedFeatures.HasFlag(FeatureFlags.ViewportMask); + public readonly byte ClipDistancesWritten { get; } + public readonly int UserDefinedMap { get; } + + public IoUsage(FeatureFlags usedFeatures, byte clipDistancesWritten, int userDefinedMap) + { + _usedFeatures = usedFeatures; + ClipDistancesWritten = clipDistancesWritten; + UserDefinedMap = userDefinedMap; + } + + public readonly IoUsage Combine(IoUsage other) + { + return new IoUsage( + _usedFeatures | other._usedFeatures, + (byte)(ClipDistancesWritten | other.ClipDistancesWritten), + UserDefinedMap | other.UserDefinedMap); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/BindlessElimination.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/BindlessElimination.cs new file mode 100644 index 0000000..d3c1397 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/BindlessElimination.cs @@ -0,0 +1,494 @@ +using Ryujinx.Graphics.Shader.Instructions; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.StructuredIr; +using System; +using System.Collections.Generic; + +namespace Ryujinx.Graphics.Shader.Translation.Optimizations +{ + class BindlessElimination + { + public static void RunPass(BasicBlock block, ResourceManager resourceManager, IGpuAccessor gpuAccessor) + { + // We can turn a bindless into regular access by recognizing the pattern + // produced by the compiler for separate texture and sampler. + // We check for the following conditions: + // - The handle is a constant buffer value. + // - The handle is the result of a bitwise OR logical operation. + // - Both sources of the OR operation comes from a constant buffer. + LinkedListNode nextNode; + + for (LinkedListNode node = block.Operations.First; node != null; node = nextNode) + { + nextNode = node.Next; + + if (node.Value is not TextureOperation texOp) + { + continue; + } + + if ((texOp.Flags & TextureFlags.Bindless) == 0) + { + continue; + } + + if (!TryConvertBindless(block, resourceManager, gpuAccessor, texOp) && + !GenerateBindlessAccess(block, resourceManager, gpuAccessor, texOp, node)) + { + // If we can't do bindless elimination, remove the texture operation. + // Set any destination variables to zero. + + string typeName = texOp.Inst.IsImage() + ? texOp.Type.GetGlslImageTypeName(texOp.Format.GetComponentType()) + : texOp.Type.GlslTextureTypeName; + + gpuAccessor.Log($"Failed to find handle source for bindless access of type \"{typeName}\"."); + + for (int destIndex = 0; destIndex < texOp.DestsCount; destIndex++) + { + block.Operations.AddBefore(node, new Operation(Instruction.Copy, texOp.GetDest(destIndex), OperandHelper.Const(0))); + } + + Utils.DeleteNode(node, texOp); + } + } + } + + private static bool GenerateBindlessAccess( + BasicBlock block, + ResourceManager resourceManager, + IGpuAccessor gpuAccessor, + TextureOperation texOp, + LinkedListNode node) + { + if (!gpuAccessor.QueryHostSupportsSeparateSampler()) + { + // We depend on combining samplers and textures in the shader being supported for this. + + return false; + } + + Operand bindlessHandle = texOp.GetSource(0); + + if (bindlessHandle.AsgOp is PhiNode phi) + { + for (int srcIndex = 0; srcIndex < phi.SourcesCount; srcIndex++) + { + Operand phiSource = phi.GetSource(srcIndex); + + if (phiSource.AsgOp is not PhiNode && !IsBindlessAccessAllowed(phiSource)) + { + return false; + } + } + } + else if (!IsBindlessAccessAllowed(bindlessHandle)) + { + return false; + } + + Operand textureHandle = OperandHelper.Local(); + Operand samplerHandle = OperandHelper.Local(); + Operand textureIndex = OperandHelper.Local(); + + block.Operations.AddBefore(node, new Operation(Instruction.BitwiseAnd, textureHandle, bindlessHandle, OperandHelper.Const(0xfffff))); + block.Operations.AddBefore(node, new Operation(Instruction.ShiftRightU32, samplerHandle, bindlessHandle, OperandHelper.Const(20))); + + int texturePoolLength = Math.Max(BindlessToArray.MinimumArrayLength, gpuAccessor.QueryTextureArrayLengthFromPool()); + + block.Operations.AddBefore(node, new Operation(Instruction.MinimumU32, textureIndex, textureHandle, OperandHelper.Const(texturePoolLength - 1))); + + texOp.SetSource(0, textureIndex); + + bool hasSampler = !texOp.Inst.IsImage(); + + SetBindingPair textureSetAndBinding = resourceManager.GetTextureOrImageBinding( + texOp.Inst, + texOp.Type, + texOp.Format, + texOp.Flags & ~TextureFlags.Bindless, + 0, + TextureHandle.PackOffsets(0, 0, TextureHandleType.Direct), + texturePoolLength, + hasSampler); + + if (hasSampler) + { + Operand samplerIndex = OperandHelper.Local(); + + int samplerPoolLength = Math.Max(BindlessToArray.MinimumArrayLength, gpuAccessor.QuerySamplerArrayLengthFromPool()); + + block.Operations.AddBefore(node, new Operation(Instruction.MinimumU32, samplerIndex, samplerHandle, OperandHelper.Const(samplerPoolLength - 1))); + + texOp.InsertSource(1, samplerIndex); + + SetBindingPair samplerSetAndBinding = resourceManager.GetTextureOrImageBinding( + texOp.Inst, + SamplerType.None, + texOp.Format, + TextureFlags.None, + 0, + TextureHandle.PackOffsets(0, 0, TextureHandleType.Direct), + samplerPoolLength); + + texOp.TurnIntoArray(textureSetAndBinding, samplerSetAndBinding); + } + else + { + texOp.TurnIntoArray(textureSetAndBinding); + } + + return true; + } + + private static bool IsBindlessAccessAllowed(Operand bindlessHandle) + { + if (bindlessHandle.Type == OperandType.ConstantBuffer) + { + // Bindless access with handles from constant buffer is allowed. + + return true; + } + + if (bindlessHandle.AsgOp is not Operation handleOp || + handleOp.Inst != Instruction.Load || + (handleOp.StorageKind != StorageKind.Input && handleOp.StorageKind != StorageKind.StorageBuffer)) + { + // Right now, we only allow bindless access when the handle comes from a shader input or storage buffer. + // This is an artificial limitation to prevent it from being used in cases where it + // would have a large performance impact of loading all textures in the pool. + // It might be removed in the future, if we can mitigate the performance impact. + + return false; + } + + return true; + } + + private static bool TryConvertBindless(BasicBlock block, ResourceManager resourceManager, IGpuAccessor gpuAccessor, TextureOperation texOp) + { + if (texOp.Inst == Instruction.TextureSample || texOp.Inst.IsTextureQuery()) + { + Operand bindlessHandle = texOp.GetSource(0); + + // In some cases the compiler uses a shuffle operation to get the handle, + // for some textureGrad implementations. In those cases, we can skip the shuffle. + if (bindlessHandle.AsgOp is Operation shuffleOp && shuffleOp.Inst == Instruction.Shuffle) + { + bindlessHandle = shuffleOp.GetSource(0); + } + + bindlessHandle = Utils.FindLastOperation(bindlessHandle, block); + + // Some instructions do not encode an accurate sampler type: + // - Most instructions uses the same type for 1D and Buffer. + // - Query instructions may not have any type. + // For those cases, we need to try getting the type from current GPU state, + // as long bindless elimination is successful and we know where the texture descriptor is located. + bool rewriteSamplerType = + texOp.Type == SamplerType.TextureBuffer || + texOp.Inst == Instruction.TextureQuerySamples || + texOp.Inst == Instruction.TextureQuerySize; + + if (bindlessHandle.Type == OperandType.ConstantBuffer) + { + SetHandle( + resourceManager, + gpuAccessor, + texOp, + bindlessHandle.GetCbufOffset(), + bindlessHandle.GetCbufSlot(), + rewriteSamplerType, + isImage: false); + + return true; + } + + if (!TryGetOperation(bindlessHandle.AsgOp, out Operation handleCombineOp)) + { + return false; + } + + if (handleCombineOp.Inst != Instruction.BitwiseOr) + { + return false; + } + + Operand src0 = Utils.FindLastOperation(handleCombineOp.GetSource(0), block); + Operand src1 = Utils.FindLastOperation(handleCombineOp.GetSource(1), block); + + // For cases where we have a constant, ensure that the constant is always + // the second operand. + // Since this is a commutative operation, both are fine, + // and having a "canonical" representation simplifies some checks below. + if (src0.Type == OperandType.Constant && src1.Type != OperandType.Constant) + { + (src0, src1) = (src1, src0); + } + + TextureHandleType handleType = TextureHandleType.SeparateSamplerHandle; + + // Try to match the following patterns: + // Masked pattern: + // - samplerHandle = samplerHandle & 0xFFF00000; + // - textureHandle = textureHandle & 0xFFFFF; + // - combinedHandle = samplerHandle | textureHandle; + // Where samplerHandle and textureHandle comes from a constant buffer. + // Shifted pattern: + // - samplerHandle = samplerId << 20; + // - combinedHandle = samplerHandle | textureHandle; + // Where samplerId and textureHandle comes from a constant buffer. + // Constant pattern: + // - combinedHandle = samplerHandleConstant | textureHandle; + // Where samplerHandleConstant is a constant value, and textureHandle comes from a constant buffer. + if (src0.AsgOp is Operation src0AsgOp) + { + if (src1.AsgOp is Operation src1AsgOp && + src0AsgOp.Inst == Instruction.BitwiseAnd && + src1AsgOp.Inst == Instruction.BitwiseAnd) + { + src0 = GetSourceForMaskedHandle(src0AsgOp, 0xFFFFF); + src1 = GetSourceForMaskedHandle(src1AsgOp, 0xFFF00000); + + // The OR operation is commutative, so we can also try to swap the operands to get a match. + if (src0 == null || src1 == null) + { + src0 = GetSourceForMaskedHandle(src1AsgOp, 0xFFFFF); + src1 = GetSourceForMaskedHandle(src0AsgOp, 0xFFF00000); + } + + if (src0 == null || src1 == null) + { + return false; + } + } + else if (src0AsgOp.Inst == Instruction.ShiftLeft) + { + Operand shift = src0AsgOp.GetSource(1); + + if (shift.Type == OperandType.Constant && shift.Value == 20) + { + src0 = src1; + src1 = src0AsgOp.GetSource(0); + handleType = TextureHandleType.SeparateSamplerId; + } + } + } + else if (src1.AsgOp is Operation src1AsgOp && src1AsgOp.Inst == Instruction.ShiftLeft) + { + Operand shift = src1AsgOp.GetSource(1); + + if (shift.Type == OperandType.Constant && shift.Value == 20) + { + src1 = src1AsgOp.GetSource(0); + handleType = TextureHandleType.SeparateSamplerId; + } + } + else if (src1.Type == OperandType.Constant && (src1.Value & 0xfffff) == 0) + { + handleType = TextureHandleType.SeparateConstantSamplerHandle; + } + + if (src0.Type != OperandType.ConstantBuffer) + { + return false; + } + + if (handleType == TextureHandleType.SeparateConstantSamplerHandle) + { + SetHandle( + resourceManager, + gpuAccessor, + texOp, + TextureHandle.PackOffsets(src0.GetCbufOffset(), (src1.Value >> 20) & 0xfff, handleType), + TextureHandle.PackSlots(src0.GetCbufSlot(), 0), + rewriteSamplerType, + isImage: false); + + return true; + } + else if (src1.Type == OperandType.ConstantBuffer) + { + SetHandle( + resourceManager, + gpuAccessor, + texOp, + TextureHandle.PackOffsets(src0.GetCbufOffset(), src1.GetCbufOffset(), handleType), + TextureHandle.PackSlots(src0.GetCbufSlot(), src1.GetCbufSlot()), + rewriteSamplerType, + isImage: false); + + return true; + } + } + else if (texOp.Inst.IsImage()) + { + Operand src0 = Utils.FindLastOperation(texOp.GetSource(0), block); + + if (src0.Type == OperandType.ConstantBuffer) + { + int cbufOffset = src0.GetCbufOffset(); + int cbufSlot = src0.GetCbufSlot(); + + if (texOp.Format == TextureFormat.Unknown) + { + if (texOp.Inst == Instruction.ImageAtomic) + { + texOp.Format = ShaderProperties.GetTextureFormatAtomic(gpuAccessor, cbufOffset, cbufSlot); + } + else + { + texOp.Format = ShaderProperties.GetTextureFormat(gpuAccessor, cbufOffset, cbufSlot); + } + } + + bool rewriteSamplerType = texOp.Type == SamplerType.TextureBuffer; + + SetHandle(resourceManager, gpuAccessor, texOp, cbufOffset, cbufSlot, rewriteSamplerType, isImage: true); + + return true; + } + } + + return false; + } + + private static bool TryGetOperation(INode asgOp, out Operation outOperation) + { + if (asgOp is PhiNode phi) + { + // If we have a phi, let's check if all inputs are effectively the same value. + // If so, we can "see through" the phi and pick any of the inputs (since they are all the same). + + Operand firstSrc = phi.GetSource(0); + + for (int index = 1; index < phi.SourcesCount; index++) + { + if (!IsSameOperand(firstSrc, phi.GetSource(index))) + { + outOperation = null; + + return false; + } + } + + asgOp = firstSrc.AsgOp; + } + + if (asgOp is Operation operation) + { + outOperation = operation; + + return true; + } + + outOperation = null; + + return false; + } + + private static bool IsSameOperand(Operand x, Operand y) + { + if (x.Type == y.Type && x.Type == OperandType.LocalVariable) + { + return x.AsgOp is Operation xOp && + y.AsgOp is Operation yOp && + xOp.Inst == Instruction.BitwiseOr && + yOp.Inst == Instruction.BitwiseOr && + AreBothEqualConstantBuffers(xOp.GetSource(0), yOp.GetSource(0)) && + AreBothEqualConstantBuffers(xOp.GetSource(1), yOp.GetSource(1)); + } + + return false; + } + + private static bool AreBothEqualConstantBuffers(Operand x, Operand y) + { + return x.Type == y.Type && x.Value == y.Value && x.Type == OperandType.ConstantBuffer; + } + + private static Operand GetSourceForMaskedHandle(Operation asgOp, uint mask) + { + // Assume it was already checked that the operation is bitwise AND. + + Operand src0 = asgOp.GetSource(0); + Operand src1 = asgOp.GetSource(1); + + if (src0.Type == OperandType.ConstantBuffer && src1.Type == OperandType.ConstantBuffer) + { + // We can't check if the mask matches here as both operands are from a constant buffer. + // Be optimistic and assume it matches. Avoid constant buffer 1 as official drivers + // uses this one to store compiler constants. + + return src0.GetCbufSlot() == 1 ? src1 : src0; + } + else if (src0.Type == OperandType.ConstantBuffer && src1.Type == OperandType.Constant) + { + if ((uint)src1.Value == mask) + { + return src0; + } + } + else if (src0.Type == OperandType.Constant && src1.Type == OperandType.ConstantBuffer) + { + if ((uint)src0.Value == mask) + { + return src1; + } + } + + return null; + } + + private static void SetHandle( + ResourceManager resourceManager, + IGpuAccessor gpuAccessor, + TextureOperation texOp, + int cbufOffset, + int cbufSlot, + bool rewriteSamplerType, + bool isImage) + { + if (rewriteSamplerType) + { + SamplerType newType = gpuAccessor.QuerySamplerType(cbufOffset, cbufSlot); + + if (texOp.Inst.IsTextureQuery()) + { + texOp.Type = newType; + } + else if (texOp.Type == SamplerType.TextureBuffer && newType == SamplerType.Texture1D) + { + int coordsCount = 2; + + if (InstEmit.Sample1DAs2D) + { + newType = SamplerType.Texture2D; + texOp.InsertSource(coordsCount++, OperandHelper.Const(0)); + } + + if (!isImage && + (texOp.Flags & TextureFlags.IntCoords) != 0 && + (texOp.Flags & TextureFlags.LodLevel) == 0) + { + // IntCoords textures must always have explicit LOD. + texOp.SetLodLevelFlag(); + texOp.InsertSource(coordsCount, OperandHelper.Const(0)); + } + + texOp.Type = newType; + } + } + + SetBindingPair setAndBinding = resourceManager.GetTextureOrImageBinding( + texOp.Inst, + texOp.Type, + texOp.Format, + texOp.Flags & ~TextureFlags.Bindless, + cbufSlot, + cbufOffset); + + texOp.SetBinding(setAndBinding); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/BindlessToArray.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/BindlessToArray.cs new file mode 100644 index 0000000..16c42b1 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/BindlessToArray.cs @@ -0,0 +1,238 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System; +using System.Collections.Generic; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Translation.Optimizations +{ + static class BindlessToArray + { + private const int NvnTextureBufferIndex = 2; + private const int HardcodedArrayLengthOgl = 4; + + // 1 and 0 elements are not considered arrays anymore. + public const int MinimumArrayLength = 2; + + public static void RunPassOgl(BasicBlock block, ResourceManager resourceManager) + { + // We can turn a bindless texture access into a indexed access, + // as long the following conditions are true: + // - The handle is loaded using a LDC instruction. + // - The handle is loaded from the constant buffer with the handles (CB2 for NVN). + // - The load has a constant offset. + // The base offset of the array of handles on the constant buffer is the constant offset. + for (LinkedListNode node = block.Operations.First; node != null; node = node.Next) + { + if (node.Value is not TextureOperation texOp) + { + continue; + } + + if ((texOp.Flags & TextureFlags.Bindless) == 0) + { + continue; + } + + if (texOp.GetSource(0).AsgOp is not Operation handleAsgOp) + { + continue; + } + + if (handleAsgOp.Inst != Instruction.Load || + handleAsgOp.StorageKind != StorageKind.ConstantBuffer || + handleAsgOp.SourcesCount != 4) + { + continue; + } + + Operand ldcSrc0 = handleAsgOp.GetSource(0); + + if (ldcSrc0.Type != OperandType.Constant || + !resourceManager.TryGetConstantBufferSlot(ldcSrc0.Value, out int src0CbufSlot) || + src0CbufSlot != NvnTextureBufferIndex) + { + continue; + } + + Operand ldcSrc1 = handleAsgOp.GetSource(1); + + // We expect field index 0 to be accessed. + if (ldcSrc1.Type != OperandType.Constant || ldcSrc1.Value != 0) + { + continue; + } + + Operand ldcSrc2 = handleAsgOp.GetSource(2); + + // FIXME: This is missing some checks, for example, a check to ensure that the shift value is 2. + // Might be not worth fixing since if that doesn't kick in, the result will be no texture + // to access anyway which is also wrong. + // Plus this whole transform is fundamentally flawed as-is since we have no way to know the array size. + // Eventually, this should be entirely removed in favor of a implementation that supports true bindless + // texture access. + if (ldcSrc2.AsgOp is not Operation shrOp || shrOp.Inst != Instruction.ShiftRightU32) + { + continue; + } + + if (shrOp.GetSource(0).AsgOp is not Operation shrOp2 || shrOp2.Inst != Instruction.ShiftRightU32) + { + continue; + } + + if (shrOp2.GetSource(0).AsgOp is not Operation addOp || addOp.Inst != Instruction.Add) + { + continue; + } + + Operand addSrc1 = addOp.GetSource(1); + + if (addSrc1.Type != OperandType.Constant) + { + continue; + } + + TurnIntoArray(resourceManager, texOp, NvnTextureBufferIndex, addSrc1.Value / 4, HardcodedArrayLengthOgl); + + Operand index = Local(); + + Operand source = addOp.GetSource(0); + + Operation shrBy3 = new(Instruction.ShiftRightU32, index, source, Const(3)); + + block.Operations.AddBefore(node, shrBy3); + + texOp.SetSource(0, index); + } + } + + public static void RunPass(BasicBlock block, ResourceManager resourceManager, IGpuAccessor gpuAccessor) + { + // We can turn a bindless texture access into a indexed access, + // as long the following conditions are true: + // - The handle is loaded using a LDC instruction. + // - The handle is loaded from the constant buffer with the handles (CB2 for NVN). + // - The load has a constant offset. + // The base offset of the array of handles on the constant buffer is the constant offset. + for (LinkedListNode node = block.Operations.First; node != null; node = node.Next) + { + if (node.Value is not TextureOperation texOp) + { + continue; + } + + if ((texOp.Flags & TextureFlags.Bindless) == 0) + { + continue; + } + + Operand bindlessHandle = Utils.FindLastOperation(texOp.GetSource(0), block); + + if (bindlessHandle.AsgOp is not Operation handleAsgOp) + { + continue; + } + + int secondaryCbufSlot = 0; + int secondaryCbufOffset = 0; + bool hasSecondaryHandle = false; + + if (handleAsgOp.Inst == Instruction.BitwiseOr) + { + Operand src0 = Utils.FindLastOperation(handleAsgOp.GetSource(0), block); + Operand src1 = Utils.FindLastOperation(handleAsgOp.GetSource(1), block); + + if (src0.Type == OperandType.ConstantBuffer && src1.AsgOp is Operation) + { + handleAsgOp = src1.AsgOp as Operation; + secondaryCbufSlot = src0.GetCbufSlot(); + secondaryCbufOffset = src0.GetCbufOffset(); + hasSecondaryHandle = true; + } + else if (src0.AsgOp is Operation && src1.Type == OperandType.ConstantBuffer) + { + handleAsgOp = src0.AsgOp as Operation; + secondaryCbufSlot = src1.GetCbufSlot(); + secondaryCbufOffset = src1.GetCbufOffset(); + hasSecondaryHandle = true; + } + } + + if (handleAsgOp.Inst != Instruction.Load || + handleAsgOp.StorageKind != StorageKind.ConstantBuffer || + handleAsgOp.SourcesCount != 4) + { + continue; + } + + Operand ldcSrc0 = handleAsgOp.GetSource(0); + + if (ldcSrc0.Type != OperandType.Constant || + !resourceManager.TryGetConstantBufferSlot(ldcSrc0.Value, out int src0CbufSlot)) + { + continue; + } + + Operand ldcSrc1 = handleAsgOp.GetSource(1); + + // We expect field index 0 to be accessed. + if (ldcSrc1.Type != OperandType.Constant || ldcSrc1.Value != 0) + { + continue; + } + + Operand ldcVecIndex = handleAsgOp.GetSource(2); + Operand ldcElemIndex = handleAsgOp.GetSource(3); + + if (ldcVecIndex.Type != OperandType.LocalVariable || ldcElemIndex.Type != OperandType.LocalVariable) + { + continue; + } + + int cbufSlot; + int handleIndex; + + if (hasSecondaryHandle) + { + cbufSlot = TextureHandle.PackSlots(src0CbufSlot, secondaryCbufSlot); + handleIndex = TextureHandle.PackOffsets(0, secondaryCbufOffset, TextureHandleType.SeparateSamplerHandle); + } + else + { + cbufSlot = src0CbufSlot; + handleIndex = 0; + } + + int length = Math.Max(MinimumArrayLength, gpuAccessor.QueryTextureArrayLengthFromBuffer(src0CbufSlot)); + + TurnIntoArray(resourceManager, texOp, cbufSlot, handleIndex, length); + + Operand vecIndex = Local(); + Operand elemIndex = Local(); + Operand index = Local(); + Operand indexMin = Local(); + + block.Operations.AddBefore(node, new Operation(Instruction.ShiftLeft, vecIndex, ldcVecIndex, Const(1))); + block.Operations.AddBefore(node, new Operation(Instruction.ShiftRightU32, elemIndex, ldcElemIndex, Const(1))); + block.Operations.AddBefore(node, new Operation(Instruction.Add, index, vecIndex, elemIndex)); + block.Operations.AddBefore(node, new Operation(Instruction.MinimumU32, indexMin, index, Const(length - 1))); + + texOp.SetSource(0, indexMin); + } + } + + private static void TurnIntoArray(ResourceManager resourceManager, TextureOperation texOp, int cbufSlot, int handleIndex, int length) + { + SetBindingPair setAndBinding = resourceManager.GetTextureOrImageBinding( + texOp.Inst, + texOp.Type, + texOp.Format, + texOp.Flags & ~TextureFlags.Bindless, + cbufSlot, + handleIndex, + length); + + texOp.TurnIntoArray(setAndBinding); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/BranchElimination.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/BranchElimination.cs new file mode 100644 index 0000000..1e1b576 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/BranchElimination.cs @@ -0,0 +1,60 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System; + +namespace Ryujinx.Graphics.Shader.Translation.Optimizations +{ + static class BranchElimination + { + public static bool RunPass(BasicBlock block) + { + if (block.HasBranch && IsRedundantBranch((Operation)block.GetLastOp(), Next(block))) + { + block.Branch = null; + + return true; + } + + return false; + } + + private static bool IsRedundantBranch(Operation current, BasicBlock nextBlock) + { + // Here we check that: + // - The current block ends with a branch. + // - The next block only contains a branch. + // - The branch on the next block is unconditional. + // - Both branches are jumping to the same location. + // In this case, the branch on the current block can be removed, + // as the next block is going to jump to the same place anyway. + + if (nextBlock?.Operations.First?.Value is not Operation next) + { + return false; + } + + if (next.Inst != Instruction.Branch) + { + return false; + } + + return current.Dest == next.Dest; + } + + private static BasicBlock Next(BasicBlock block) + { + block = block.Next; + + while (block != null && block.Operations.Count == 0) + { + if (block.HasBranch) + { + throw new InvalidOperationException("Found a bogus empty block that \"ends with a branch\"."); + } + + block = block.Next; + } + + return block; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/ConstantFolding.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/ConstantFolding.cs new file mode 100644 index 0000000..f17d0b1 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/ConstantFolding.cs @@ -0,0 +1,352 @@ +using Ryujinx.Common.Utilities; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Translation.Optimizations +{ + static class ConstantFolding + { + public static void RunPass(ResourceManager resourceManager, Operation operation) + { + if (!AreAllSourcesConstant(operation)) + { + return; + } + + switch (operation.Inst) + { + case Instruction.Add: + EvaluateBinary(operation, (x, y) => x + y); + break; + + case Instruction.BitCount: + EvaluateUnary(operation, (x) => BitCount(x)); + break; + + case Instruction.BitwiseAnd: + EvaluateBinary(operation, (x, y) => x & y); + break; + + case Instruction.BitwiseExclusiveOr: + EvaluateBinary(operation, (x, y) => x ^ y); + break; + + case Instruction.BitwiseNot: + EvaluateUnary(operation, (x) => ~x); + break; + + case Instruction.BitwiseOr: + EvaluateBinary(operation, (x, y) => x | y); + break; + + case Instruction.BitfieldExtractS32: + BitfieldExtractS32(operation); + break; + + case Instruction.BitfieldExtractU32: + BitfieldExtractU32(operation); + break; + + case Instruction.Clamp: + EvaluateTernary(operation, (x, y, z) => Math.Clamp(x, y, z)); + break; + + case Instruction.ClampU32: + EvaluateTernary(operation, (x, y, z) => (int)Math.Clamp((uint)x, (uint)y, (uint)z)); + break; + + case Instruction.CompareEqual: + EvaluateBinary(operation, (x, y) => x == y); + break; + + case Instruction.CompareGreater: + EvaluateBinary(operation, (x, y) => x > y); + break; + + case Instruction.CompareGreaterOrEqual: + EvaluateBinary(operation, (x, y) => x >= y); + break; + + case Instruction.CompareGreaterOrEqualU32: + EvaluateBinary(operation, (x, y) => (uint)x >= (uint)y); + break; + + case Instruction.CompareGreaterU32: + EvaluateBinary(operation, (x, y) => (uint)x > (uint)y); + break; + + case Instruction.CompareLess: + EvaluateBinary(operation, (x, y) => x < y); + break; + + case Instruction.CompareLessOrEqual: + EvaluateBinary(operation, (x, y) => x <= y); + break; + + case Instruction.CompareLessOrEqualU32: + EvaluateBinary(operation, (x, y) => (uint)x <= (uint)y); + break; + + case Instruction.CompareLessU32: + EvaluateBinary(operation, (x, y) => (uint)x < (uint)y); + break; + + case Instruction.CompareNotEqual: + EvaluateBinary(operation, (x, y) => x != y); + break; + + case Instruction.Divide: + EvaluateBinary(operation, (x, y) => y != 0 ? x / y : 0); + break; + + case Instruction.FP32 | Instruction.Add: + EvaluateFPBinary(operation, (x, y) => x + y); + break; + + case Instruction.FP32 | Instruction.Clamp: + EvaluateFPTernary(operation, (x, y, z) => Math.Clamp(x, y, z)); + break; + + case Instruction.FP32 | Instruction.CompareEqual: + EvaluateFPBinary(operation, (x, y) => x == y); + break; + + case Instruction.FP32 | Instruction.CompareGreater: + EvaluateFPBinary(operation, (x, y) => x > y); + break; + + case Instruction.FP32 | Instruction.CompareGreaterOrEqual: + EvaluateFPBinary(operation, (x, y) => x >= y); + break; + + case Instruction.FP32 | Instruction.CompareLess: + EvaluateFPBinary(operation, (x, y) => x < y); + break; + + case Instruction.FP32 | Instruction.CompareLessOrEqual: + EvaluateFPBinary(operation, (x, y) => x <= y); + break; + + case Instruction.FP32 | Instruction.CompareNotEqual: + EvaluateFPBinary(operation, (x, y) => x != y); + break; + + case Instruction.FP32 | Instruction.Divide: + EvaluateFPBinary(operation, (x, y) => x / y); + break; + + case Instruction.FP32 | Instruction.Multiply: + EvaluateFPBinary(operation, (x, y) => x * y); + break; + + case Instruction.FP32 | Instruction.Negate: + EvaluateFPUnary(operation, (x) => -x); + break; + + case Instruction.FP32 | Instruction.Subtract: + EvaluateFPBinary(operation, (x, y) => x - y); + break; + + case Instruction.IsNan: + EvaluateFPUnary(operation, (x) => float.IsNaN(x)); + break; + + case Instruction.Load: + if (operation.StorageKind == StorageKind.ConstantBuffer && operation.SourcesCount == 4) + { + int binding = operation.GetSource(0).Value; + int fieldIndex = operation.GetSource(1).Value; + + if (resourceManager.TryGetConstantBufferSlot(binding, out int cbufSlot) && fieldIndex == 0) + { + int vecIndex = operation.GetSource(2).Value; + int elemIndex = operation.GetSource(3).Value; + int cbufOffset = vecIndex * 4 + elemIndex; + + operation.TurnIntoCopy(Cbuf(cbufSlot, cbufOffset)); + } + } + + break; + + case Instruction.Maximum: + EvaluateBinary(operation, (x, y) => Math.Max(x, y)); + break; + + case Instruction.MaximumU32: + EvaluateBinary(operation, (x, y) => (int)Math.Max((uint)x, (uint)y)); + break; + + case Instruction.Minimum: + EvaluateBinary(operation, (x, y) => Math.Min(x, y)); + break; + + case Instruction.MinimumU32: + EvaluateBinary(operation, (x, y) => (int)Math.Min((uint)x, (uint)y)); + break; + + case Instruction.Multiply: + EvaluateBinary(operation, (x, y) => x * y); + break; + + case Instruction.Negate: + EvaluateUnary(operation, (x) => -x); + break; + + case Instruction.ShiftLeft: + EvaluateBinary(operation, (x, y) => x << y); + break; + + case Instruction.ShiftRightS32: + EvaluateBinary(operation, (x, y) => x >> y); + break; + + case Instruction.ShiftRightU32: + EvaluateBinary(operation, (x, y) => (int)((uint)x >> y)); + break; + + case Instruction.Subtract: + EvaluateBinary(operation, (x, y) => x - y); + break; + + case Instruction.UnpackHalf2x16: + UnpackHalf2x16(operation); + break; + } + } + + private static bool AreAllSourcesConstant(Operation operation) + { + for (int index = 0; index < operation.SourcesCount; index++) + { + if (operation.GetSource(index).Type != OperandType.Constant) + { + return false; + } + } + + return true; + } + + private static int BitCount(int value) + { + int count = 0; + + for (int bit = 0; bit < 32; bit++) + { + if (value.Extract(bit)) + { + count++; + } + } + + return count; + } + + private static void BitfieldExtractS32(Operation operation) + { + int value = GetBitfieldExtractValue(operation); + + int shift = 32 - operation.GetSource(2).Value; + + value = (value << shift) >> shift; + + operation.TurnIntoCopy(Const(value)); + } + + private static void BitfieldExtractU32(Operation operation) + { + operation.TurnIntoCopy(Const(GetBitfieldExtractValue(operation))); + } + + private static int GetBitfieldExtractValue(Operation operation) + { + int value = operation.GetSource(0).Value; + int lsb = operation.GetSource(1).Value; + int length = operation.GetSource(2).Value; + + return value.Extract(lsb, length); + } + + private static void UnpackHalf2x16(Operation operation) + { + int value = operation.GetSource(0).Value; + + value = (value >> operation.Index * 16) & 0xffff; + + operation.TurnIntoCopy(ConstF((float)BitConverter.UInt16BitsToHalf((ushort)value))); + } + + private static void EvaluateUnary(Operation operation, Func op) + { + int x = operation.GetSource(0).Value; + + operation.TurnIntoCopy(Const(op(x))); + } + + private static void EvaluateFPUnary(Operation operation, Func op) + { + float x = operation.GetSource(0).AsFloat(); + + operation.TurnIntoCopy(ConstF(op(x))); + } + + private static void EvaluateFPUnary(Operation operation, Func op) + { + float x = operation.GetSource(0).AsFloat(); + + operation.TurnIntoCopy(Const(op(x) ? IrConsts.True : IrConsts.False)); + } + + private static void EvaluateBinary(Operation operation, Func op) + { + int x = operation.GetSource(0).Value; + int y = operation.GetSource(1).Value; + + operation.TurnIntoCopy(Const(op(x, y))); + } + + private static void EvaluateBinary(Operation operation, Func op) + { + int x = operation.GetSource(0).Value; + int y = operation.GetSource(1).Value; + + operation.TurnIntoCopy(Const(op(x, y) ? IrConsts.True : IrConsts.False)); + } + + private static void EvaluateFPBinary(Operation operation, Func op) + { + float x = operation.GetSource(0).AsFloat(); + float y = operation.GetSource(1).AsFloat(); + + operation.TurnIntoCopy(ConstF(op(x, y))); + } + + private static void EvaluateFPBinary(Operation operation, Func op) + { + float x = operation.GetSource(0).AsFloat(); + float y = operation.GetSource(1).AsFloat(); + + operation.TurnIntoCopy(Const(op(x, y) ? IrConsts.True : IrConsts.False)); + } + + private static void EvaluateTernary(Operation operation, Func op) + { + int x = operation.GetSource(0).Value; + int y = operation.GetSource(1).Value; + int z = operation.GetSource(2).Value; + + operation.TurnIntoCopy(Const(op(x, y, z))); + } + + private static void EvaluateFPTernary(Operation operation, Func op) + { + float x = operation.GetSource(0).AsFloat(); + float y = operation.GetSource(1).AsFloat(); + float z = operation.GetSource(2).AsFloat(); + + operation.TurnIntoCopy(ConstF(op(x, y, z))); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/DoubleToFloat.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/DoubleToFloat.cs new file mode 100644 index 0000000..d83a146 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/DoubleToFloat.cs @@ -0,0 +1,70 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System.Collections.Generic; + +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Translation.Optimizations +{ + static class DoubleToFloat + { + public static void RunPass(HelperFunctionManager hfm, BasicBlock block) + { + for (LinkedListNode node = block.Operations.First; node != null; node = node.Next) + { + if (node.Value is not Operation) + { + continue; + } + + node = InsertSoftFloat64(hfm, node); + } + } + + private static LinkedListNode InsertSoftFloat64(HelperFunctionManager hfm, LinkedListNode node) + { + Operation operation = (Operation)node.Value; + + if (operation.Inst == Instruction.PackDouble2x32) + { + int functionId = hfm.GetOrCreateFunctionId(HelperFunctionName.ConvertDoubleToFloat); + + Operand[] callArgs = [Const(functionId), operation.GetSource(0), operation.GetSource(1)]; + + Operand floatValue = operation.Dest; + + operation.Dest = null; + + LinkedListNode newNode = node.List.AddBefore(node, new Operation(Instruction.Call, 0, floatValue, callArgs)); + + Utils.DeleteNode(node, operation); + + return newNode; + } + else if (operation.Inst == Instruction.UnpackDouble2x32) + { + int functionId = hfm.GetOrCreateFunctionId(HelperFunctionName.ConvertFloatToDouble); + + // TODO: Allow UnpackDouble2x32 to produce two outputs and get rid of "operation.Index". + + Operand resultLow = operation.Index == 0 ? operation.Dest : Local(); + Operand resultHigh = operation.Index == 1 ? operation.Dest : Local(); + + operation.Dest = null; + + Operand[] callArgs = [Const(functionId), operation.GetSource(0), resultLow, resultHigh]; + + LinkedListNode newNode = node.List.AddBefore(node, new Operation(Instruction.Call, 0, (Operand)null, callArgs)); + + Utils.DeleteNode(node, operation); + + return newNode; + } + else + { + operation.TurnDoubleIntoFloat(); + + return node; + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/GlobalToStorage.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/GlobalToStorage.cs new file mode 100644 index 0000000..772ad25 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/GlobalToStorage.cs @@ -0,0 +1,1177 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Translation.Optimizations +{ + static class GlobalToStorage + { + private const int DriverReservedCb = 0; + + enum LsMemoryType + { + Local, + Shared, + } + + private class GtsContext + { + private readonly struct Entry + { + public readonly int FunctionId; + public readonly Instruction Inst; + public readonly StorageKind StorageKind; + public readonly bool IsMultiTarget; + public readonly IReadOnlyList TargetCbs; + + public Entry( + int functionId, + Instruction inst, + StorageKind storageKind, + bool isMultiTarget, + IReadOnlyList targetCbs) + { + FunctionId = functionId; + Inst = inst; + StorageKind = storageKind; + IsMultiTarget = isMultiTarget; + TargetCbs = targetCbs; + } + } + + private readonly struct LsKey : IEquatable + { + public readonly Operand BaseOffset; + public readonly int ConstOffset; + public readonly LsMemoryType Type; + + public LsKey(Operand baseOffset, int constOffset, LsMemoryType type) + { + BaseOffset = baseOffset; + ConstOffset = constOffset; + Type = type; + } + + public override int GetHashCode() + { + return HashCode.Combine(BaseOffset, ConstOffset, Type); + } + + public override bool Equals(object obj) + { + return obj is LsKey other && Equals(other); + } + + public bool Equals(LsKey other) + { + return other.BaseOffset == BaseOffset && other.ConstOffset == ConstOffset && other.Type == Type; + } + } + + private readonly List _entries; + private readonly Dictionary> _sharedEntries; + private readonly HelperFunctionManager _hfm; + + public GtsContext(HelperFunctionManager hfm) + { + _entries = []; + _sharedEntries = new Dictionary>(); + _hfm = hfm; + } + + public int AddFunction(Operation baseOp, bool isMultiTarget, IReadOnlyList targetCbs, Function function) + { + int functionId = _hfm.AddFunction(function); + + _entries.Add(new Entry(functionId, baseOp.Inst, baseOp.StorageKind, isMultiTarget, targetCbs)); + + return functionId; + } + + public bool TryGetFunctionId(Operation baseOp, bool isMultiTarget, List targetCbs, out int functionId) + { + foreach (Entry entry in _entries) + { + if (entry.Inst != baseOp.Inst || + entry.StorageKind != baseOp.StorageKind || + entry.IsMultiTarget != isMultiTarget || + entry.TargetCbs.Count != targetCbs.Count) + { + continue; + } + + bool allEqual = true; + + for (int index = 0; index < targetCbs.Count; index++) + { + if (targetCbs[index] != entry.TargetCbs[index]) + { + allEqual = false; + break; + } + } + + if (allEqual) + { + functionId = entry.FunctionId; + return true; + } + } + + functionId = -1; + return false; + } + + public void AddMemoryTargetCb(LsMemoryType type, Operand baseOffset, int constOffset, uint targetCb, SearchResult result) + { + LsKey key = new(baseOffset, constOffset, type); + + if (!_sharedEntries.TryGetValue(key, out Dictionary targetCbs)) + { + // No entry with this base offset, create a new one. + + targetCbs = new Dictionary() { { targetCb, result } }; + + _sharedEntries.Add(key, targetCbs); + } + else if (targetCbs.TryGetValue(targetCb, out SearchResult existingResult)) + { + // If our entry already exists, but does not match the new result, + // we set the offset to null to indicate there are multiple possible offsets. + // This will be used on the multi-target access that does not need to know the offset. + + if (existingResult.Offset != null && + (existingResult.Offset != result.Offset || + existingResult.ConstOffset != result.ConstOffset)) + { + targetCbs[targetCb] = new SearchResult(result.SbCbSlot, result.SbCbOffset); + } + } + else + { + // An entry for this base offset already exists, but not for the specified + // constant buffer region where the storage buffer base address and size + // comes from. + + targetCbs.Add(targetCb, result); + } + } + + public bool TryGetMemoryTargetCb(LsMemoryType type, Operand baseOffset, int constOffset, out SearchResult result) + { + LsKey key = new(baseOffset, constOffset, type); + + if (_sharedEntries.TryGetValue(key, out Dictionary targetCbs) && targetCbs.Count == 1) + { + SearchResult candidateResult = targetCbs.Values.First(); + + if (candidateResult.Found) + { + result = candidateResult; + + return true; + } + } + + result = default; + + return false; + } + } + + private readonly struct SearchResult + { + public static SearchResult NotFound => new(-1, 0); + public bool Found => SbCbSlot != -1; + public int SbCbSlot { get; } + public int SbCbOffset { get; } + public Operand Offset { get; } + public int ConstOffset { get; } + + public SearchResult(int sbCbSlot, int sbCbOffset) + { + SbCbSlot = sbCbSlot; + SbCbOffset = sbCbOffset; + } + + public SearchResult(int sbCbSlot, int sbCbOffset, Operand offset, int constOffset = 0) + { + SbCbSlot = sbCbSlot; + SbCbOffset = sbCbOffset; + Offset = offset; + ConstOffset = constOffset; + } + } + + public static void RunPass( + HelperFunctionManager hfm, + BasicBlock[] blocks, + ResourceManager resourceManager, + IGpuAccessor gpuAccessor, + TargetLanguage targetLanguage) + { + GtsContext gtsContext = new(hfm); + + foreach (BasicBlock block in blocks) + { + for (LinkedListNode node = block.Operations.First; node != null; node = node.Next) + { + if (node.Value is not Operation operation) + { + continue; + } + + if (IsGlobalMemory(operation.StorageKind)) + { + LinkedListNode nextNode = ReplaceGlobalMemoryWithStorage( + gtsContext, + resourceManager, + gpuAccessor, + targetLanguage, + block, + node); + + if (nextNode == null) + { + // The returned value being null means that the global memory replacement failed, + // so we just make loads read 0 and stores do nothing. + + gpuAccessor.Log($"Failed to reserve storage buffer for global memory operation \"{operation.Inst}\"."); + + if (operation.Dest != null) + { + operation.TurnIntoCopy(Const(0)); + } + else + { + Utils.DeleteNode(node, operation); + } + } + else + { + node = nextNode; + } + } + else if (operation.Inst == Instruction.Store && + (operation.StorageKind == StorageKind.SharedMemory || + operation.StorageKind == StorageKind.LocalMemory)) + { + // The NVIDIA compiler can sometimes use shared or local memory as temporary + // storage to place the base address and size on, so we need + // to be able to find such information stored in memory too. + + if (TryGetMemoryOffsets(operation, out LsMemoryType type, out Operand baseOffset, out int constOffset)) + { + Operand value = operation.GetSource(operation.SourcesCount - 1); + + SearchResult result = FindUniqueBaseAddressCb(gtsContext, block, value, needsOffset: false); + if (result.Found) + { + uint targetCb = PackCbSlotAndOffset(result.SbCbSlot, result.SbCbOffset); + gtsContext.AddMemoryTargetCb(type, baseOffset, constOffset, targetCb, result); + } + } + } + } + } + } + + private static bool IsGlobalMemory(StorageKind storageKind) + { + return storageKind is StorageKind.GlobalMemory or + StorageKind.GlobalMemoryS8 or + StorageKind.GlobalMemoryS16 or + StorageKind.GlobalMemoryU8 or + StorageKind.GlobalMemoryU16; + } + + private static bool IsSmallInt(StorageKind storageKind) + { + return storageKind is StorageKind.GlobalMemoryS8 or + StorageKind.GlobalMemoryS16 or + StorageKind.GlobalMemoryU8 or + StorageKind.GlobalMemoryU16; + } + + private static LinkedListNode ReplaceGlobalMemoryWithStorage( + GtsContext gtsContext, + ResourceManager resourceManager, + IGpuAccessor gpuAccessor, + TargetLanguage targetLanguage, + BasicBlock block, + LinkedListNode node) + { + Operation operation = node.Value as Operation; + Operand globalAddress = operation.GetSource(0); + SearchResult result = FindUniqueBaseAddressCb(gtsContext, block, globalAddress, needsOffset: true); + + if (result.Found) + { + // We found the storage buffer that is being accessed. + // There are two possible paths here, if the operation is simple enough, + // we just generate the storage access code inline. + // Otherwise, we generate a function call (and the function if necessary). + + Operand offset = result.Offset; + + bool storageUnaligned = gpuAccessor.QueryHasUnalignedStorageBuffer(); + + if (storageUnaligned) + { + Operand baseAddress = Cbuf(result.SbCbSlot, result.SbCbOffset); + + Operand baseAddressMasked = Local(); + Operand hostOffset = Local(); + + int alignment = gpuAccessor.QueryHostStorageBufferOffsetAlignment(); + + Operation maskOp = new(Instruction.BitwiseAnd, baseAddressMasked, baseAddress, Const(-alignment)); + Operation subOp = new(Instruction.Subtract, hostOffset, globalAddress, baseAddressMasked); + + node.List.AddBefore(node, maskOp); + node.List.AddBefore(node, subOp); + + offset = hostOffset; + } + else if (result.ConstOffset != 0) + { + Operand newOffset = Local(); + + Operation addOp = new(Instruction.Add, newOffset, offset, Const(result.ConstOffset)); + + node.List.AddBefore(node, addOp); + + offset = newOffset; + } + + if (CanUseInlineStorageOp(operation, targetLanguage)) + { + return GenerateInlineStorageOp(resourceManager, node, operation, offset, result); + } + else + { + if (!TryGenerateSingleTargetStorageOp( + gtsContext, + resourceManager, + targetLanguage, + operation, + result, + out int functionId)) + { + return null; + } + + return GenerateCallStorageOp(node, operation, offset, functionId); + } + } + else + { + // Failed to find the storage buffer directly. + // Try to walk through Phi chains and find all possible constant buffers where + // the base address might be stored. + // Generate a helper function that will check all possible storage buffers and use the right one. + + if (!TryGenerateMultiTargetStorageOp( + gtsContext, + resourceManager, + gpuAccessor, + targetLanguage, + block, + operation, + out int functionId)) + { + return null; + } + + return GenerateCallStorageOp(node, operation, null, functionId); + } + } + + private static bool CanUseInlineStorageOp(Operation operation, TargetLanguage targetLanguage) + { + if (operation.StorageKind != StorageKind.GlobalMemory) + { + return false; + } + + return (operation.Inst != Instruction.AtomicMaxS32 && + operation.Inst != Instruction.AtomicMinS32) || targetLanguage == TargetLanguage.Spirv; + } + + private static LinkedListNode GenerateInlineStorageOp( + ResourceManager resourceManager, + LinkedListNode node, + Operation operation, + Operand offset, + SearchResult result) + { + bool isStore = operation.Inst == Instruction.Store || operation.Inst.IsAtomic(); + if (!resourceManager.TryGetStorageBufferBinding(result.SbCbSlot, result.SbCbOffset, isStore, out int binding)) + { + return null; + } + + Operand wordOffset = Local(); + + Operand[] sources; + + if (operation.Inst == Instruction.AtomicCompareAndSwap) + { + sources = + [ + Const(binding), + Const(0), + wordOffset, + operation.GetSource(operation.SourcesCount - 2), + operation.GetSource(operation.SourcesCount - 1) + ]; + } + else if (isStore) + { + sources = [Const(binding), Const(0), wordOffset, operation.GetSource(operation.SourcesCount - 1)]; + } + else + { + sources = [Const(binding), Const(0), wordOffset]; + } + + Operation shiftOp = new(Instruction.ShiftRightU32, wordOffset, offset, Const(2)); + Operation storageOp = new(operation.Inst, StorageKind.StorageBuffer, operation.Dest, sources); + + node.List.AddBefore(node, shiftOp); + LinkedListNode newNode = node.List.AddBefore(node, storageOp); + + Utils.DeleteNode(node, operation); + + return newNode; + } + + private static LinkedListNode GenerateCallStorageOp(LinkedListNode node, Operation operation, Operand offset, int functionId) + { + // Generate call to a helper function that will perform the storage buffer operation. + + Operand[] sources = new Operand[operation.SourcesCount - 1 + (offset == null ? 2 : 1)]; + + sources[0] = Const(functionId); + + if (offset != null) + { + // If the offset was supplised, we use that and skip the global address. + + sources[1] = offset; + + for (int srcIndex = 2; srcIndex < operation.SourcesCount; srcIndex++) + { + sources[srcIndex] = operation.GetSource(srcIndex); + } + } + else + { + // Use the 64-bit global address which is split in 2 32-bit arguments. + + for (int srcIndex = 0; srcIndex < operation.SourcesCount; srcIndex++) + { + sources[srcIndex + 1] = operation.GetSource(srcIndex); + } + } + + bool returnsValue = operation.Dest != null; + Operand returnValue = returnsValue ? Local() : null; + + Operation callOp = new(Instruction.Call, returnValue, sources); + + LinkedListNode newNode = node.List.AddBefore(node, callOp); + + if (returnsValue) + { + operation.TurnIntoCopy(returnValue); + + return node; + } + else + { + Utils.DeleteNode(node, operation); + + return newNode; + } + } + + private static bool TryGenerateSingleTargetStorageOp( + GtsContext gtsContext, + ResourceManager resourceManager, + TargetLanguage targetLanguage, + Operation operation, + SearchResult result, + out int functionId) + { + List targetCbs = [PackCbSlotAndOffset(result.SbCbSlot, result.SbCbOffset)]; + + if (gtsContext.TryGetFunctionId(operation, isMultiTarget: false, targetCbs, out functionId)) + { + return true; + } + + int inArgumentsCount = 1; + + if (operation.Inst == Instruction.AtomicCompareAndSwap) + { + inArgumentsCount = 3; + } + else if (operation.Inst == Instruction.Store || operation.Inst.IsAtomic()) + { + inArgumentsCount = 2; + } + + EmitterContext context = new(); + + Operand offset = Argument(0); + Operand compare = null; + Operand value = null; + + if (inArgumentsCount == 3) + { + compare = Argument(1); + value = Argument(2); + } + else if (inArgumentsCount == 2) + { + value = Argument(1); + } + + if (!TryGenerateStorageOp( + resourceManager, + targetLanguage, + context, + operation.Inst, + operation.StorageKind, + offset, + compare, + value, + result, + out Operand resultValue)) + { + functionId = 0; + return false; + } + + bool returnsValue = resultValue != null; + + if (returnsValue) + { + context.Return(resultValue); + } + else + { + context.Return(); + } + + string functionName = GetFunctionName(operation, isMultiTarget: false, targetCbs); + + Function function = new( + ControlFlowGraph.Create(context.GetOperations()).Blocks, + functionName, + returnsValue, + inArgumentsCount, + 0); + + functionId = gtsContext.AddFunction(operation, isMultiTarget: false, targetCbs, function); + + return true; + } + + private static bool TryGenerateMultiTargetStorageOp( + GtsContext gtsContext, + ResourceManager resourceManager, + IGpuAccessor gpuAccessor, + TargetLanguage targetLanguage, + BasicBlock block, + Operation operation, + out int functionId) + { + Queue phis = new(); + HashSet visited = []; + List targetCbs = []; + + Operand globalAddress = operation.GetSource(0); + + if (globalAddress.AsgOp is Operation addOp && addOp.Inst == Instruction.Add) + { + Operand src1 = addOp.GetSource(0); + Operand src2 = addOp.GetSource(1); + + if (src1.Type == OperandType.Constant && src2.Type == OperandType.LocalVariable) + { + globalAddress = src2; + } + else if (src1.Type == OperandType.LocalVariable && src2.Type == OperandType.Constant) + { + globalAddress = src1; + } + } + + if (globalAddress.AsgOp is PhiNode phi && visited.Add(phi)) + { + phis.Enqueue(phi); + } + else + { + SearchResult result = FindUniqueBaseAddressCb(gtsContext, block, operation.GetSource(0), needsOffset: false); + + if (result.Found) + { + targetCbs.Add(PackCbSlotAndOffset(result.SbCbSlot, result.SbCbOffset)); + } + } + + while (phis.TryDequeue(out phi)) + { + for (int srcIndex = 0; srcIndex < phi.SourcesCount; srcIndex++) + { + BasicBlock phiBlock = phi.GetBlock(srcIndex); + Operand phiSource = phi.GetSource(srcIndex); + + SearchResult result = FindUniqueBaseAddressCb(gtsContext, phiBlock, phiSource, needsOffset: false); + + if (result.Found) + { + uint targetCb = PackCbSlotAndOffset(result.SbCbSlot, result.SbCbOffset); + + if (!targetCbs.Contains(targetCb)) + { + targetCbs.Add(targetCb); + } + } + else if (phiSource.AsgOp is PhiNode phi2 && visited.Add(phi2)) + { + phis.Enqueue(phi2); + } + } + } + + targetCbs.Sort(); + + if (targetCbs.Count == 0) + { + gpuAccessor.Log($"Failed to find storage buffer for global memory operation \"{operation.Inst}\"."); + } + + if (gtsContext.TryGetFunctionId(operation, isMultiTarget: true, targetCbs, out functionId)) + { + return true; + } + + int inArgumentsCount = 2; + + if (operation.Inst == Instruction.AtomicCompareAndSwap) + { + inArgumentsCount = 4; + } + else if (operation.Inst == Instruction.Store || operation.Inst.IsAtomic()) + { + inArgumentsCount = 3; + } + + EmitterContext context = new(); + + Operand globalAddressLow = Argument(0); + Operand globalAddressHigh = Argument(1); + + foreach (uint targetCb in targetCbs) + { + (int sbCbSlot, int sbCbOffset) = UnpackCbSlotAndOffset(targetCb); + + Operand baseAddrLow = Cbuf(sbCbSlot, sbCbOffset); + Operand baseAddrHigh = Cbuf(sbCbSlot, sbCbOffset + 1); + Operand size = Cbuf(sbCbSlot, sbCbOffset + 2); + + Operand offset = context.ISubtract(globalAddressLow, baseAddrLow); + Operand borrow = context.ICompareLessUnsigned(globalAddressLow, baseAddrLow); + + Operand inRangeLow = context.ICompareLessUnsigned(offset, size); + + Operand addrHighBorrowed = context.IAdd(globalAddressHigh, borrow); + + Operand inRangeHigh = context.ICompareEqual(addrHighBorrowed, baseAddrHigh); + + Operand inRange = context.BitwiseAnd(inRangeLow, inRangeHigh); + + Operand lblSkip = Label(); + context.BranchIfFalse(lblSkip, inRange); + + Operand compare = null; + Operand value = null; + + if (inArgumentsCount == 4) + { + compare = Argument(2); + value = Argument(3); + } + else if (inArgumentsCount == 3) + { + value = Argument(2); + } + + SearchResult result = new(sbCbSlot, sbCbOffset); + + int alignment = gpuAccessor.QueryHostStorageBufferOffsetAlignment(); + + Operand baseAddressMasked = context.BitwiseAnd(baseAddrLow, Const(-alignment)); + Operand hostOffset = context.ISubtract(globalAddressLow, baseAddressMasked); + + if (!TryGenerateStorageOp( + resourceManager, + targetLanguage, + context, + operation.Inst, + operation.StorageKind, + hostOffset, + compare, + value, + result, + out Operand resultValue)) + { + functionId = 0; + return false; + } + + if (resultValue != null) + { + context.Return(resultValue); + } + else + { + context.Return(); + } + + context.MarkLabel(lblSkip); + } + + bool returnsValue = operation.Dest != null; + + if (returnsValue) + { + context.Return(Const(0)); + } + else + { + context.Return(); + } + + string functionName = GetFunctionName(operation, isMultiTarget: true, targetCbs); + + Function function = new( + ControlFlowGraph.Create(context.GetOperations()).Blocks, + functionName, + returnsValue, + inArgumentsCount, + 0); + + functionId = gtsContext.AddFunction(operation, isMultiTarget: true, targetCbs, function); + + return true; + } + + private static uint PackCbSlotAndOffset(int cbSlot, int cbOffset) + { + return (uint)((ushort)cbSlot | ((ushort)cbOffset << 16)); + } + + private static (int, int) UnpackCbSlotAndOffset(uint packed) + { + return ((ushort)packed, (ushort)(packed >> 16)); + } + + private static string GetFunctionName(Operation baseOp, bool isMultiTarget, IReadOnlyList targetCbs) + { + StringBuilder nameBuilder = new(); + nameBuilder.Append(baseOp.Inst.ToString()); + + nameBuilder.Append(baseOp.StorageKind switch + { + StorageKind.GlobalMemoryS8 => "S8", + StorageKind.GlobalMemoryS16 => "S16", + StorageKind.GlobalMemoryU8 => "U8", + StorageKind.GlobalMemoryU16 => "U16", + _ => string.Empty, + }); + + if (isMultiTarget) + { + nameBuilder.Append("Multi"); + } + + foreach (uint targetCb in targetCbs) + { + (int sbCbSlot, int sbCbOffset) = UnpackCbSlotAndOffset(targetCb); + + nameBuilder.Append($"_c{sbCbSlot}o{sbCbOffset}"); + } + + return nameBuilder.ToString(); + } + + private static bool TryGenerateStorageOp( + ResourceManager resourceManager, + TargetLanguage targetLanguage, + EmitterContext context, + Instruction inst, + StorageKind storageKind, + Operand offset, + Operand compare, + Operand value, + SearchResult result, + out Operand resultValue) + { + resultValue = null; + bool isStore = inst.IsAtomic() || inst == Instruction.Store; + + if (!resourceManager.TryGetStorageBufferBinding(result.SbCbSlot, result.SbCbOffset, isStore, out int binding)) + { + return false; + } + + Operand wordOffset = context.ShiftRightU32(offset, Const(2)); + + if (inst.IsAtomic()) + { + if (IsSmallInt(storageKind)) + { + throw new NotImplementedException(); + } + + switch (inst) + { + case Instruction.AtomicAdd: + resultValue = context.AtomicAdd(StorageKind.StorageBuffer, binding, Const(0), wordOffset, value); + break; + case Instruction.AtomicAnd: + resultValue = context.AtomicAnd(StorageKind.StorageBuffer, binding, Const(0), wordOffset, value); + break; + case Instruction.AtomicCompareAndSwap: + resultValue = context.AtomicCompareAndSwap(StorageKind.StorageBuffer, binding, Const(0), wordOffset, compare, value); + break; + case Instruction.AtomicMaxS32: + if (targetLanguage == TargetLanguage.Spirv) + { + resultValue = context.AtomicMaxS32(StorageKind.StorageBuffer, binding, Const(0), wordOffset, value); + } + else + { + resultValue = GenerateAtomicCasLoop(context, wordOffset, binding, (memValue) => + { + return context.IMaximumS32(memValue, value); + }); + } + + break; + case Instruction.AtomicMaxU32: + resultValue = context.AtomicMaxU32(StorageKind.StorageBuffer, binding, Const(0), wordOffset, value); + break; + case Instruction.AtomicMinS32: + if (targetLanguage == TargetLanguage.Spirv) + { + resultValue = context.AtomicMinS32(StorageKind.StorageBuffer, binding, Const(0), wordOffset, value); + } + else + { + resultValue = GenerateAtomicCasLoop(context, wordOffset, binding, (memValue) => + { + return context.IMinimumS32(memValue, value); + }); + } + + break; + case Instruction.AtomicMinU32: + resultValue = context.AtomicMinU32(StorageKind.StorageBuffer, binding, Const(0), wordOffset, value); + break; + case Instruction.AtomicOr: + resultValue = context.AtomicOr(StorageKind.StorageBuffer, binding, Const(0), wordOffset, value); + break; + case Instruction.AtomicSwap: + resultValue = context.AtomicSwap(StorageKind.StorageBuffer, binding, Const(0), wordOffset, value); + break; + case Instruction.AtomicXor: + resultValue = context.AtomicXor(StorageKind.StorageBuffer, binding, Const(0), wordOffset, value); + break; + } + } + else if (inst == Instruction.Store) + { + int bitSize = storageKind switch + { + StorageKind.GlobalMemoryS8 or + StorageKind.GlobalMemoryU8 => 8, + StorageKind.GlobalMemoryS16 or + StorageKind.GlobalMemoryU16 => 16, + _ => 32, + }; + + if (bitSize < 32) + { + Operand bitOffset = HelperFunctionManager.GetBitOffset(context, offset); + + GenerateAtomicCasLoop(context, wordOffset, binding, (memValue) => + { + return context.BitfieldInsert(memValue, value, bitOffset, Const(bitSize)); + }); + } + else + { + context.Store(StorageKind.StorageBuffer, binding, Const(0), wordOffset, value); + } + } + else + { + value = context.Load(StorageKind.StorageBuffer, binding, Const(0), wordOffset); + + if (IsSmallInt(storageKind)) + { + Operand bitOffset = HelperFunctionManager.GetBitOffset(context, offset); + + switch (storageKind) + { + case StorageKind.GlobalMemoryS8: + value = context.ShiftRightS32(value, bitOffset); + value = context.BitfieldExtractS32(value, Const(0), Const(8)); + break; + case StorageKind.GlobalMemoryS16: + value = context.ShiftRightS32(value, bitOffset); + value = context.BitfieldExtractS32(value, Const(0), Const(16)); + break; + case StorageKind.GlobalMemoryU8: + value = context.ShiftRightU32(value, bitOffset); + value = context.BitwiseAnd(value, Const(byte.MaxValue)); + break; + case StorageKind.GlobalMemoryU16: + value = context.ShiftRightU32(value, bitOffset); + value = context.BitwiseAnd(value, Const(ushort.MaxValue)); + break; + } + } + + resultValue = value; + } + + return true; + } + + private static Operand GenerateAtomicCasLoop(EmitterContext context, Operand wordOffset, int binding, Func opCallback) + { + Operand lblLoopHead = Label(); + + context.MarkLabel(lblLoopHead); + + Operand oldValue = context.Load(StorageKind.StorageBuffer, binding, Const(0), wordOffset); + Operand newValue = opCallback(oldValue); + + Operand casResult = context.AtomicCompareAndSwap( + StorageKind.StorageBuffer, + binding, + Const(0), + wordOffset, + oldValue, + newValue); + + Operand casFail = context.ICompareNotEqual(casResult, oldValue); + + context.BranchIfTrue(lblLoopHead, casFail); + + return oldValue; + } + + private static SearchResult FindUniqueBaseAddressCb(GtsContext gtsContext, BasicBlock block, Operand globalAddress, bool needsOffset) + { + globalAddress = Utils.FindLastOperation(globalAddress, block); + + if (globalAddress.Type == OperandType.ConstantBuffer) + { + return GetBaseAddressCbWithOffset(globalAddress, Const(0), 0); + } + + Operation operation = globalAddress.AsgOp as Operation; + + if (operation == null || operation.Inst != Instruction.Add) + { + return FindBaseAddressCbFromMemory(gtsContext, operation, 0, needsOffset); + } + + Operand src1 = operation.GetSource(0); + Operand src2 = operation.GetSource(1); + + int constOffset = 0; + + if ((src1.Type == OperandType.LocalVariable && src2.Type == OperandType.Constant) || + (src2.Type == OperandType.LocalVariable && src1.Type == OperandType.Constant)) + { + Operand baseAddr; + Operand offset; + + if (src1.Type == OperandType.LocalVariable) + { + baseAddr = Utils.FindLastOperation(src1, block); + offset = src2; + } + else + { + baseAddr = Utils.FindLastOperation(src2, block); + offset = src1; + } + + SearchResult result = GetBaseAddressCbWithOffset(baseAddr, offset, 0); + if (result.Found) + { + return result; + } + + constOffset = offset.Value; + operation = baseAddr.AsgOp as Operation; + + if (operation == null || operation.Inst != Instruction.Add) + { + return FindBaseAddressCbFromMemory(gtsContext, operation, constOffset, needsOffset); + } + } + + src1 = operation.GetSource(0); + src2 = operation.GetSource(1); + + // If we have two possible results, we give preference to the ones from + // the driver reserved constant buffer, as those are the ones that + // contains the base address. + + // If both are constant buffer, give preference to the second operand, + // because constant buffer are always encoded as the second operand, + // so the second operand will always be the one from the last instruction. + + if (src1.Type != OperandType.ConstantBuffer || + (src1.Type == OperandType.ConstantBuffer && src2.Type == OperandType.ConstantBuffer) || + (src2.Type == OperandType.ConstantBuffer && src2.GetCbufSlot() == DriverReservedCb)) + { + return GetBaseAddressCbWithOffset(src2, src1, constOffset); + } + + return GetBaseAddressCbWithOffset(src1, src2, constOffset); + } + + private static SearchResult FindBaseAddressCbFromMemory(GtsContext gtsContext, Operation operation, int constOffset, bool needsOffset) + { + if (operation != null) + { + if (TryGetMemoryOffsets(operation, out LsMemoryType type, out Operand bo, out int co) && + gtsContext.TryGetMemoryTargetCb(type, bo, co, out SearchResult result) && + (result.Offset != null || !needsOffset)) + { + if (constOffset != 0) + { + return new SearchResult( + result.SbCbSlot, + result.SbCbOffset, + result.Offset, + result.ConstOffset + constOffset); + } + + return result; + } + } + + return SearchResult.NotFound; + } + + private static SearchResult GetBaseAddressCbWithOffset(Operand baseAddress, Operand offset, int constOffset) + { + if (baseAddress.Type == OperandType.ConstantBuffer) + { + int sbCbSlot = baseAddress.GetCbufSlot(); + int sbCbOffset = baseAddress.GetCbufOffset(); + + // We require the offset to be aligned to 1 word (64 bits), + // since the address size is 64-bit and the GPU only supports aligned memory access. + if ((sbCbOffset & 1) == 0) + { + return new SearchResult(sbCbSlot, sbCbOffset, offset, constOffset); + } + } + + return SearchResult.NotFound; + } + + private static bool TryGetMemoryOffsets(Operation operation, out LsMemoryType type, out Operand baseOffset, out int constOffset) + { + baseOffset = null; + + if (operation.Inst is Instruction.Load or Instruction.Store) + { + if (operation.StorageKind == StorageKind.SharedMemory) + { + type = LsMemoryType.Shared; + return TryGetSharedMemoryOffsets(operation, out baseOffset, out constOffset); + } + else if (operation.StorageKind == StorageKind.LocalMemory) + { + type = LsMemoryType.Local; + return TryGetLocalMemoryOffset(operation, out constOffset); + } + } + + type = default; + constOffset = 0; + return false; + } + + private static bool TryGetSharedMemoryOffsets(Operation operation, out Operand baseOffset, out int constOffset) + { + baseOffset = null; + constOffset = 0; + + // The byte offset is right shifted by 2 to get the 32-bit word offset, + // so we want to get the byte offset back, since each one of those word + // offsets are a new "local variable" which will not match. + + if (operation.GetSource(1).AsgOp is Operation shiftRightOp && + shiftRightOp.Inst == Instruction.ShiftRightU32 && + shiftRightOp.GetSource(1).Type == OperandType.Constant && + shiftRightOp.GetSource(1).Value == 2) + { + baseOffset = shiftRightOp.GetSource(0); + } + + // Check if we have a constant offset being added to the base offset. + + if (baseOffset?.AsgOp is Operation addOp && addOp.Inst == Instruction.Add) + { + Operand src1 = addOp.GetSource(0); + Operand src2 = addOp.GetSource(1); + + if (src1.Type == OperandType.Constant && src2.Type == OperandType.LocalVariable) + { + constOffset = src1.Value; + baseOffset = src2; + } + else if (src1.Type == OperandType.LocalVariable && src2.Type == OperandType.Constant) + { + baseOffset = src1; + constOffset = src2.Value; + } + } + + return baseOffset != null && baseOffset.Type == OperandType.LocalVariable; + } + + private static bool TryGetLocalMemoryOffset(Operation operation, out int constOffset) + { + Operand offset = operation.GetSource(1); + + if (offset.Type == OperandType.Constant) + { + constOffset = offset.Value; + return true; + } + + constOffset = 0; + return false; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/Optimizer.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/Optimizer.cs new file mode 100644 index 0000000..96a235d --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/Optimizer.cs @@ -0,0 +1,467 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System.Collections.Generic; +using System.Diagnostics; +using System.Linq; + +namespace Ryujinx.Graphics.Shader.Translation.Optimizations +{ + static class Optimizer + { + public static void RunPass(TransformContext context) + { + for (int blkIndex = 0; blkIndex < context.Blocks.Length; blkIndex++) + { + XmadOptimizer.RunPass(context.Blocks[blkIndex]); + } + + RunOptimizationPasses(context.Blocks, context.ResourceManager); + + // TODO: Some of those are not optimizations and shouldn't be here. + + GlobalToStorage.RunPass(context.Hfm, context.Blocks, context.ResourceManager, context.GpuAccessor, context.TargetLanguage); + + bool hostSupportsShaderFloat64 = context.GpuAccessor.QueryHostSupportsShaderFloat64(); + + // Those passes are looking for specific patterns and only needs to run once. + for (int blkIndex = 0; blkIndex < context.Blocks.Length; blkIndex++) + { + if (context.TargetApi == TargetApi.OpenGL) + { + BindlessToArray.RunPassOgl(context.Blocks[blkIndex], context.ResourceManager); + } + else + { + BindlessToArray.RunPass(context.Blocks[blkIndex], context.ResourceManager, context.GpuAccessor); + } + + BindlessElimination.RunPass(context.Blocks[blkIndex], context.ResourceManager, context.GpuAccessor); + + // FragmentCoord only exists on fragment shaders, so we don't need to check other stages. + if (context.Stage == ShaderStage.Fragment) + { + EliminateMultiplyByFragmentCoordW(context.Blocks[blkIndex]); + } + + // If the host does not support double operations, we need to turn them into float operations. + if (!hostSupportsShaderFloat64) + { + DoubleToFloat.RunPass(context.Hfm, context.Blocks[blkIndex]); + } + } + + // Run optimizations one last time to remove any code that is now optimizable after above passes. + RunOptimizationPasses(context.Blocks, context.ResourceManager); + } + + private static void RunOptimizationPasses(BasicBlock[] blocks, ResourceManager resourceManager) + { + bool modified; + + do + { + modified = false; + + for (int blkIndex = 0; blkIndex < blocks.Length; blkIndex++) + { + BasicBlock block = blocks[blkIndex]; + + LinkedListNode node = block.Operations.First; + + while (node != null) + { + LinkedListNode nextNode = node.Next; + + bool isUnused = IsUnused(node.Value); + + if (node.Value is not Operation operation || isUnused) + { + if (node.Value is PhiNode phi && !isUnused) + { + isUnused = PropagatePhi(phi); + } + + if (isUnused) + { + RemoveNode(block, node); + + modified = true; + } + + node = nextNode; + + continue; + } + + ConstantFolding.RunPass(resourceManager, operation); + Simplification.RunPass(operation); + + if (DestIsLocalVar(operation)) + { + if (operation.Inst == Instruction.Copy) + { + PropagateCopy(operation); + + RemoveNode(block, node); + + modified = true; + } + else if ((operation.Inst == Instruction.PackHalf2x16 && PropagatePack(operation)) || + (operation.Inst == Instruction.ShuffleXor && MatchDdxOrDdy(operation))) + { + if (DestHasNoUses(operation)) + { + RemoveNode(block, node); + } + + modified = true; + } + } + + node = nextNode; + } + + if (BranchElimination.RunPass(block)) + { + RemoveNode(block, block.Operations.Last); + + modified = true; + } + } + } + while (modified); + } + + private static void PropagateCopy(Operation copyOp) + { + // Propagate copy source operand to all uses of + // the destination operand. + + Operand dest = copyOp.Dest; + Operand src = copyOp.GetSource(0); + + INode[] uses = dest.UseOps.ToArray(); + + foreach (INode useNode in uses) + { + for (int index = 0; index < useNode.SourcesCount; index++) + { + if (useNode.GetSource(index) == dest) + { + useNode.SetSource(index, src); + } + } + } + } + + private static bool PropagatePhi(PhiNode phi) + { + // If all phi sources are the same, we can propagate it and remove the phi. + + if (!Utils.AreAllSourcesTheSameOperand(phi)) + { + return false; + } + + // All sources are equal, we can propagate the value. + + Operand firstSrc = phi.GetSource(0); + Operand dest = phi.Dest; + + INode[] uses = dest.UseOps.ToArray(); + + foreach (INode useNode in uses) + { + for (int index = 0; index < useNode.SourcesCount; index++) + { + if (useNode.GetSource(index) == dest) + { + useNode.SetSource(index, firstSrc); + } + } + } + + return true; + } + + private static bool PropagatePack(Operation packOp) + { + // Propagate pack source operands to uses by unpack + // instruction. The source depends on the unpack instruction. + bool modified = false; + + Operand dest = packOp.Dest; + Operand src0 = packOp.GetSource(0); + Operand src1 = packOp.GetSource(1); + + INode[] uses = dest.UseOps.ToArray(); + + foreach (INode useNode in uses) + { + if (useNode is not Operation operation || operation.Inst != Instruction.UnpackHalf2x16) + { + continue; + } + + if (operation.GetSource(0) == dest) + { + operation.TurnIntoCopy(operation.Index == 1 ? src1 : src0); + + modified = true; + } + } + + return modified; + } + + public static bool MatchDdxOrDdy(Operation operation) + { + // It's assumed that "operation.Inst" is ShuffleXor, + // that should be checked before calling this method. + Debug.Assert(operation.Inst == Instruction.ShuffleXor); + + bool modified = false; + + Operand src2 = operation.GetSource(1); + Operand src3 = operation.GetSource(2); + + if (src2.Type != OperandType.Constant || (src2.Value != 1 && src2.Value != 2)) + { + return false; + } + + if (src3.Type != OperandType.Constant || src3.Value != 0x1c03) + { + return false; + } + + bool isDdy = src2.Value == 2; + bool isDdx = !isDdy; + + // We can replace any use by a FSWZADD with DDX/DDY, when + // the following conditions are true: + // - The mask should be 0b10100101 for DDY, or 0b10011001 for DDX. + // - The first source operand must be the shuffle output. + // - The second source operand must be the shuffle first source operand. + INode[] uses = operation.Dest.UseOps.ToArray(); + + foreach (INode use in uses) + { + if (use is not Operation test) + { + continue; + } + + if (use is not Operation useOp || useOp.Inst != Instruction.SwizzleAdd) + { + continue; + } + + Operand fswzaddSrc1 = useOp.GetSource(0); + Operand fswzaddSrc2 = useOp.GetSource(1); + Operand fswzaddSrc3 = useOp.GetSource(2); + + if (fswzaddSrc1 != operation.Dest) + { + continue; + } + + if (fswzaddSrc2 != operation.GetSource(0)) + { + continue; + } + + if (fswzaddSrc3.Type != OperandType.Constant) + { + continue; + } + + int mask = fswzaddSrc3.Value; + + if ((isDdx && mask != 0b10011001) || + (isDdy && mask != 0b10100101)) + { + continue; + } + + useOp.TurnInto(isDdx ? Instruction.Ddx : Instruction.Ddy, fswzaddSrc2); + + modified = true; + } + + return modified; + } + + private static void EliminateMultiplyByFragmentCoordW(BasicBlock block) + { + foreach (INode node in block.Operations) + { + if (node is Operation operation) + { + EliminateMultiplyByFragmentCoordW(operation); + } + } + } + + private static void EliminateMultiplyByFragmentCoordW(Operation operation) + { + // We're looking for the pattern: + // y = x * gl_FragCoord.w + // v = y * (1.0 / gl_FragCoord.w) + // Then we transform it into: + // v = x + // This pattern is common on fragment shaders due to the way how perspective correction is done. + + // We are expecting a multiplication by the reciprocal of gl_FragCoord.w. + if (operation.Inst != (Instruction.FP32 | Instruction.Multiply)) + { + return; + } + + Operand lhs = operation.GetSource(0); + Operand rhs = operation.GetSource(1); + + // Check LHS of the main multiplication operation. We expect an input being multiplied by gl_FragCoord.w. + if (lhs.AsgOp is not Operation attrMulOp || attrMulOp.Inst != (Instruction.FP32 | Instruction.Multiply)) + { + return; + } + + Operand attrMulLhs = attrMulOp.GetSource(0); + Operand attrMulRhs = attrMulOp.GetSource(1); + + // LHS should be any input, RHS should be exactly gl_FragCoord.w. + if (!Utils.IsInputLoad(attrMulLhs.AsgOp) || !Utils.IsInputLoad(attrMulRhs.AsgOp, IoVariable.FragmentCoord, 3)) + { + return; + } + + // RHS of the main multiplication should be a reciprocal operation (1.0 / x). + if (rhs.AsgOp is not Operation reciprocalOp || reciprocalOp.Inst != (Instruction.FP32 | Instruction.Divide)) + { + return; + } + + Operand reciprocalLhs = reciprocalOp.GetSource(0); + Operand reciprocalRhs = reciprocalOp.GetSource(1); + + // Check if the divisor is a constant equal to 1.0. + if (reciprocalLhs.Type != OperandType.Constant || reciprocalLhs.AsFloat() != 1.0f) + { + return; + } + + // Check if the dividend is gl_FragCoord.w. + if (!Utils.IsInputLoad(reciprocalRhs.AsgOp, IoVariable.FragmentCoord, 3)) + { + return; + } + + // If everything matches, we can replace the operation with the input load result. + operation.TurnIntoCopy(attrMulLhs); + } + + public static void RemoveNode(BasicBlock block, LinkedListNode llNode) + { + // Remove a node from the nodes list, and also remove itself + // from all the use lists on the operands that this node uses. + block.Operations.Remove(llNode); + + Queue nodes = new(); + + nodes.Enqueue(llNode.Value); + + while (nodes.TryDequeue(out INode node)) + { + for (int index = 0; index < node.SourcesCount; index++) + { + Operand src = node.GetSource(index); + + if (src.Type != OperandType.LocalVariable) + { + continue; + } + + if (src.UseOps.Remove(node) && src.UseOps.Count == 0) + { + Debug.Assert(src.AsgOp != null); + + // TOTK (2026-10-16, from Ryujinx_Optime 01a1e37f): only follow into a producer that will itself be + // removed. An operation with side effects (a call, an atomic) stays in the block even when its + // result is unused, so its sources are still live: stripping their use lists here made the next + // DCE pass delete the whole chain feeding a global-memory atomic turned into a helper call, which + // then read an uninitialised value (TOTK's sun-visibility kernel; our c7830fac lost its atomicMin). + if (!HasSideEffects(src.AsgOp)) + { + nodes.Enqueue(src.AsgOp); + } + } + } + } + } + + private static bool IsUnused(INode node) + { + return !HasSideEffects(node) && DestIsLocalVar(node) && DestHasNoUses(node); + } + + private static bool HasSideEffects(INode node) + { + if (node is Operation operation) + { + switch (operation.Inst & Instruction.Mask) + { + case Instruction.AtomicAdd: + case Instruction.AtomicAnd: + case Instruction.AtomicCompareAndSwap: + case Instruction.AtomicMaxS32: + case Instruction.AtomicMaxU32: + case Instruction.AtomicMinS32: + case Instruction.AtomicMinU32: + case Instruction.AtomicOr: + case Instruction.AtomicSwap: + case Instruction.AtomicXor: + case Instruction.Call: + case Instruction.ImageAtomic: + return true; + } + } + + return false; + } + + private static bool DestIsLocalVar(INode node) + { + if (node.DestsCount == 0) + { + return false; + } + + for (int index = 0; index < node.DestsCount; index++) + { + Operand dest = node.GetDest(index); + + if (dest != null && dest.Type != OperandType.LocalVariable) + { + return false; + } + } + + return true; + } + + private static bool DestHasNoUses(INode node) + { + for (int index = 0; index < node.DestsCount; index++) + { + Operand dest = node.GetDest(index); + + if (dest != null && dest.UseOps.Count != 0) + { + return false; + } + } + + return true; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/Simplification.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/Simplification.cs new file mode 100644 index 0000000..b885096 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/Simplification.cs @@ -0,0 +1,236 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; + +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Translation.Optimizations +{ + static class Simplification + { + private const int AllOnes = ~0; + + public static void RunPass(Operation operation) + { + switch (operation.Inst) + { + case Instruction.Add: + TryEliminateBinaryOpCommutative(operation, 0); + break; + + case Instruction.BitwiseAnd: + TryEliminateBitwiseAnd(operation); + break; + + case Instruction.BitwiseExclusiveOr: + if (!TryEliminateXorSwap(operation)) + { + TryEliminateBinaryOpCommutative(operation, 0); + } + + break; + + case Instruction.BitwiseOr: + TryEliminateBitwiseOr(operation); + break; + + case Instruction.CompareNotEqual: + TryEliminateCompareNotEqual(operation); + break; + + case Instruction.ConditionalSelect: + TryEliminateConditionalSelect(operation); + break; + + case Instruction.Divide: + TryEliminateBinaryOpY(operation, 1); + break; + + case Instruction.Multiply: + TryEliminateBinaryOpCommutative(operation, 1); + break; + + case Instruction.ShiftLeft: + case Instruction.ShiftRightS32: + case Instruction.ShiftRightU32: + case Instruction.Subtract: + TryEliminateBinaryOpY(operation, 0); + break; + } + } + + private static void TryEliminateBitwiseAnd(Operation operation) + { + // Try to recognize and optimize those 3 patterns (in order): + // x & 0xFFFFFFFF == x, 0xFFFFFFFF & y == y, + // x & 0x00000000 == 0x00000000, 0x00000000 & y == 0x00000000 + + Operand x = operation.GetSource(0); + Operand y = operation.GetSource(1); + + if (IsConstEqual(x, AllOnes)) + { + operation.TurnIntoCopy(y); + } + else if (IsConstEqual(y, AllOnes)) + { + operation.TurnIntoCopy(x); + } + else if (IsConstEqual(x, 0) || IsConstEqual(y, 0)) + { + operation.TurnIntoCopy(Const(0)); + } + } + + private static bool TryEliminateXorSwap(Operation xCopyOp) + { + // Try to recognize XOR swap pattern: + // x = x ^ y + // y = x ^ y + // x = x ^ y + // Or, in SSA: + // x2 = x ^ y + // y2 = x2 ^ y + // x3 = x2 ^ y2 + // Transform it into something more sane: + // temp = y + // y = x + // x = temp + + // Note that because XOR is commutative, there are actually + // multiple possible combinations of this pattern, for + // simplicity this only catches one of them. + + Operand x = xCopyOp.GetSource(0); + Operand y = xCopyOp.GetSource(1); + + if (x.AsgOp is not Operation tCopyOp || tCopyOp.Inst != Instruction.BitwiseExclusiveOr || + y.AsgOp is not Operation yCopyOp || yCopyOp.Inst != Instruction.BitwiseExclusiveOr) + { + return false; + } + + if (tCopyOp == yCopyOp) + { + return false; + } + + if (yCopyOp.GetSource(0) != x || + yCopyOp.GetSource(1) != tCopyOp.GetSource(1) || + x.UseOps.Count != 2) + { + return false; + } + + x = tCopyOp.GetSource(0); + y = tCopyOp.GetSource(1); + + tCopyOp.TurnIntoCopy(y); // Temp = Y + yCopyOp.TurnIntoCopy(x); // Y = X + xCopyOp.TurnIntoCopy(tCopyOp.Dest); // X = Temp + + return true; + } + + private static void TryEliminateBitwiseOr(Operation operation) + { + // Try to recognize and optimize those 3 patterns (in order): + // x | 0x00000000 == x, 0x00000000 | y == y, + // x | 0xFFFFFFFF == 0xFFFFFFFF, 0xFFFFFFFF | y == 0xFFFFFFFF + + Operand x = operation.GetSource(0); + Operand y = operation.GetSource(1); + + if (IsConstEqual(x, 0)) + { + operation.TurnIntoCopy(y); + } + else if (IsConstEqual(y, 0)) + { + operation.TurnIntoCopy(x); + } + else if (IsConstEqual(x, AllOnes) || IsConstEqual(y, AllOnes)) + { + operation.TurnIntoCopy(Const(AllOnes)); + } + } + + private static void TryEliminateBinaryOpY(Operation operation, int comparand) + { + Operand x = operation.GetSource(0); + Operand y = operation.GetSource(1); + + if (IsConstEqual(y, comparand)) + { + operation.TurnIntoCopy(x); + } + } + + private static void TryEliminateBinaryOpCommutative(Operation operation, int comparand) + { + Operand x = operation.GetSource(0); + Operand y = operation.GetSource(1); + + if (IsConstEqual(x, comparand)) + { + operation.TurnIntoCopy(y); + } + else if (IsConstEqual(y, comparand)) + { + operation.TurnIntoCopy(x); + } + } + + private static void TryEliminateCompareNotEqual(Operation operation) + { + // Comparison instruction returns 0 if the result is false, and -1 if true. + // Doing a not equal zero comparison on the result is redundant, so we can just copy the first result in this case. + + Operand lhs = operation.GetSource(0); + Operand rhs = operation.GetSource(1); + + if (lhs.Type == OperandType.Constant) + { + (lhs, rhs) = (rhs, lhs); + } + + if (rhs.Type != OperandType.Constant || rhs.Value != 0) + { + return; + } + + if (lhs.AsgOp is not Operation compareOp || !compareOp.Inst.IsComparison()) + { + return; + } + + operation.TurnIntoCopy(lhs); + } + + private static void TryEliminateConditionalSelect(Operation operation) + { + Operand cond = operation.GetSource(0); + + if (cond.Type != OperandType.Constant) + { + return; + } + + // The condition is constant, we can turn it into a copy, and select + // the source based on the condition value. + int srcIndex = cond.Value != 0 ? 1 : 2; + + Operand source = operation.GetSource(srcIndex); + + operation.TurnIntoCopy(source); + } + + private static bool IsConstEqual(Operand operand, int comparand) + { + if (operand.Type != OperandType.Constant) + { + return false; + } + + return operand.Value == comparand; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/Utils.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/Utils.cs new file mode 100644 index 0000000..0f82671 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/Utils.cs @@ -0,0 +1,202 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System.Collections.Generic; + +namespace Ryujinx.Graphics.Shader.Translation.Optimizations +{ + static class Utils + { + public static bool IsInputLoad(INode node) + { + return (node is Operation operation) && + operation.Inst == Instruction.Load && + operation.StorageKind == StorageKind.Input; + } + + public static bool IsInputLoad(INode node, IoVariable ioVariable, int elemIndex) + { + if (node is not Operation operation || + operation.Inst != Instruction.Load || + operation.StorageKind != StorageKind.Input || + operation.SourcesCount != 2) + { + return false; + } + + Operand ioVariableSrc = operation.GetSource(0); + + if (ioVariableSrc.Type != OperandType.Constant || (IoVariable)ioVariableSrc.Value != ioVariable) + { + return false; + } + + Operand elemIndexSrc = operation.GetSource(1); + + return elemIndexSrc.Type == OperandType.Constant && elemIndexSrc.Value == elemIndex; + } + + private static bool IsSameOperand(Operand x, Operand y) + { + if (x.Type != y.Type || x.Value != y.Value) + { + return false; + } + + // TODO: Handle Load operations with the same storage and the same constant parameters. + return x == y || x.Type == OperandType.Constant || x.Type == OperandType.ConstantBuffer; + } + + private static bool AreAllSourcesEqual(Operation node, Operation otherNode) + { + if (node.SourcesCount != otherNode.SourcesCount) + { + return false; + } + + for (int index = 0; index < node.SourcesCount; index++) + { + if (!IsSameOperand(node.GetSource(index), otherNode.GetSource(index))) + { + return false; + } + } + + return true; + } + + public static bool AreAllSourcesTheSameOperand(INode node) + { + Operand firstSrc = node.GetSource(0); + + for (int index = 1; index < node.SourcesCount; index++) + { + if (!IsSameOperand(firstSrc, node.GetSource(index))) + { + return false; + } + } + + return true; + } + + private static Operation FindBranchSource(BasicBlock block) + { + foreach (BasicBlock sourceBlock in block.Predecessors) + { + if (sourceBlock.Operations.Count > 0) + { + if (sourceBlock.GetLastOp() is Operation lastOp && IsConditionalBranch(lastOp.Inst) && sourceBlock.Next == block) + { + return lastOp; + } + } + } + + return null; + } + + private static bool IsConditionalBranch(Instruction inst) + { + return inst is Instruction.BranchIfFalse or Instruction.BranchIfTrue; + } + + private static bool IsSameCondition(Operand currentCondition, Operand queryCondition) + { + if (currentCondition == queryCondition) + { + return true; + } + + return currentCondition.AsgOp is Operation currentOperation && + queryCondition.AsgOp is Operation queryOperation && + currentOperation.Inst == queryOperation.Inst && + AreAllSourcesEqual(currentOperation, queryOperation); + } + + private static bool BlockConditionsMatch(BasicBlock currentBlock, BasicBlock queryBlock) + { + // Check if all the conditions for the query block are satisfied by the current block. + // Just checks the top-most conditional for now. + + Operation currentBranch = FindBranchSource(currentBlock); + Operation queryBranch = FindBranchSource(queryBlock); + + Operand currentCondition = currentBranch?.GetSource(0); + Operand queryCondition = queryBranch?.GetSource(0); + + // The condition should be the same operand instance. + + return currentBranch != null && queryBranch != null && + currentBranch.Inst == queryBranch.Inst && + IsSameCondition(currentCondition, queryCondition); + } + + public static Operand FindLastOperation(Operand source, BasicBlock block, bool recurse = true) + { + if (source.AsgOp is PhiNode phiNode) + { + // This source can have a different value depending on a previous branch. + // Ensure that conditions met for that branch are also met for the current one. + // Prefer the latest sources for the phi node. + + int undefCount = 0; + + for (int i = phiNode.SourcesCount - 1; i >= 0; i--) + { + BasicBlock phiBlock = phiNode.GetBlock(i); + Operand phiSource = phiNode.GetSource(i); + + if (BlockConditionsMatch(block, phiBlock)) + { + return phiSource; + } + else if (recurse && phiSource.AsgOp is PhiNode) + { + // Phi source is another phi. + // Let's check if that phi has a block that matches our condition. + + Operand match = FindLastOperation(phiSource, block, false); + + if (match != phiSource) + { + return match; + } + } + else if (phiSource.Type == OperandType.Undefined) + { + undefCount++; + } + } + + // If all sources but one are undefined, we can assume that the one + // that is not undefined is the right one. + + if (undefCount == phiNode.SourcesCount - 1) + { + for (int i = phiNode.SourcesCount - 1; i >= 0; i--) + { + Operand phiSource = phiNode.GetSource(i); + + if (phiSource.Type != OperandType.Undefined) + { + return phiSource; + } + } + } + } + + return source; + } + + public static void DeleteNode(LinkedListNode node, Operation operation) + { + node.List.Remove(node); + + for (int srcIndex = 0; srcIndex < operation.SourcesCount; srcIndex++) + { + operation.SetSource(srcIndex, null); + } + + operation.Dest = null; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/XmadOptimizer.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/XmadOptimizer.cs new file mode 100644 index 0000000..19f3191 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Optimizations/XmadOptimizer.cs @@ -0,0 +1,342 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System.Collections.Generic; + +namespace Ryujinx.Graphics.Shader.Translation.Optimizations +{ + static class XmadOptimizer + { + public static void RunPass(BasicBlock block) + { + for (LinkedListNode node = block.Operations.First; node != null; node = node.Next) + { + if (node.Value is not Operation operation) + { + continue; + } + + if (TryMatchXmadPattern(operation, out Operand x, out Operand y, out Operand addend)) + { + LinkedListNode nextNode; + + if (addend != null) + { + Operand temp = OperandHelper.Local(); + + nextNode = block.Operations.AddAfter(node, new Operation(Instruction.Multiply, temp, x, y)); + nextNode = block.Operations.AddAfter(nextNode, new Operation(Instruction.Add, operation.Dest, temp, addend)); + } + else + { + nextNode = block.Operations.AddAfter(node, new Operation(Instruction.Multiply, operation.Dest, x, y)); + } + + Optimizer.RemoveNode(block, node); + node = nextNode; + } + } + } + + private static bool TryMatchXmadPattern(Operation operation, out Operand x, out Operand y, out Operand addend) + { + return TryMatchXmad32x32Pattern(operation, out x, out y, out addend) || + TryMatchXmad32x16Pattern(operation, out x, out y, out addend); + } + + private static bool TryMatchXmad32x32Pattern(Operation operation, out Operand x, out Operand y, out Operand addend) + { + x = null; + y = null; + addend = null; + + if (operation.Inst != Instruction.Add) + { + return false; + } + + Operand src1 = operation.GetSource(0); + Operand src2 = operation.GetSource(1); + + if (src2.AsgOp is not Operation addOp || addOp.Inst != Instruction.Add) + { + return false; + } + + Operand lowTimesLowResult = GetCopySource(addOp.GetSource(0)); + + if (lowTimesLowResult.AsgOp is not Operation lowTimesLowOp) + { + return false; + } + + if (!TryMatchLowTimesLow(lowTimesLowOp, out x, out y, out addend)) + { + return false; + } + + Operand lowTimesHighResult = GetCopySource(GetShifted16Source(addOp.GetSource(1), Instruction.ShiftLeft)); + + if (lowTimesHighResult.AsgOp is not Operation lowTimesHighOp) + { + return false; + } + + if (!TryMatchLowTimesHigh(lowTimesHighOp, x, y)) + { + return false; + } + + if (src1.AsgOp is not Operation highTimesHighOp) + { + return false; + } + + if (!TryMatchHighTimesHigh(highTimesHighOp, x, lowTimesHighResult)) + { + return false; + } + + return true; + } + + private static bool TryMatchXmad32x16Pattern(Operation operation, out Operand x, out Operand y, out Operand addend) + { + x = null; + y = null; + addend = null; + + if (operation.Inst != Instruction.Add) + { + return false; + } + + Operand src1 = operation.GetSource(0); + Operand src2 = operation.GetSource(1); + + Operand lowTimesLowResult = GetCopySource(src2); + + if (lowTimesLowResult.AsgOp is not Operation lowTimesLowOp) + { + return false; + } + + if (!TryMatchLowTimesLow(lowTimesLowOp, out x, out y, out addend)) + { + return false; + } + + Operand highTimesLowResult = src1; + + if (highTimesLowResult.AsgOp is not Operation highTimesLowOp) + { + return false; + } + + if (!TryMatchHighTimesLow(highTimesLowOp, x, y)) + { + return false; + } + + return y.Type == OperandType.Constant && (ushort)y.Value == y.Value; + } + + private static bool TryMatchLowTimesLow(Operation operation, out Operand x, out Operand y, out Operand addend) + { + // x = x & 0xFFFF + // y = y & 0xFFFF + // lowTimesLow = x * y + + x = null; + y = null; + addend = null; + + if (operation.Inst == Instruction.Add) + { + if (operation.GetSource(0).AsgOp is not Operation mulOp) + { + return false; + } + + addend = operation.GetSource(1); + operation = mulOp; + } + + if (operation.Inst != Instruction.Multiply) + { + return false; + } + + Operand src1 = GetMasked16Source(operation.GetSource(0)); + Operand src2 = GetMasked16Source(operation.GetSource(1)); + + if (src1 == null || src2 == null) + { + return false; + } + + x = src1; + y = src2; + + return true; + } + + private static bool TryMatchLowTimesHigh(Operation operation, Operand x, Operand y) + { + // xLow = x & 0xFFFF + // yHigh = y >> 16 + // lowTimesHigh = xLow * yHigh + // result = (lowTimesHigh & 0xFFFF) | (y << 16) + + if (operation.Inst != Instruction.BitwiseOr) + { + return false; + } + + Operand mulResult = GetMasked16Source(operation.GetSource(0)); + + if (mulResult == null) + { + return false; + } + + mulResult = GetCopySource(mulResult); + + if (mulResult.AsgOp is not Operation mulOp || mulOp.Inst != Instruction.Multiply) + { + return false; + } + + if (GetMasked16Source(mulOp.GetSource(0)) != x) + { + return false; + } + + if (GetShifted16Source(mulOp.GetSource(1), Instruction.ShiftRightU32) != y) + { + return false; + } + + if (GetShifted16Source(operation.GetSource(1), Instruction.ShiftLeft) != y) + { + return false; + } + + return true; + } + + private static bool TryMatchHighTimesLow(Operation operation, Operand x, Operand y) + { + // xHigh = x >> 16 + // yLow = y & 0xFFFF + // highTimesLow = xHigh * yLow + // result = highTimesLow << 16 + + if (operation.Inst != Instruction.ShiftLeft || !IsConst(operation.GetSource(1), 16)) + { + return false; + } + + Operand mulResult = operation.GetSource(0); + + if (mulResult.AsgOp is not Operation mulOp || mulOp.Inst != Instruction.Multiply) + { + return false; + } + + if (GetShifted16Source(mulOp.GetSource(0), Instruction.ShiftRightU32) != x) + { + return false; + } + + Operand src2 = GetMasked16Source(mulOp.GetSource(1)); + + if (src2.Type != y.Type || src2.Value != y.Value) + { + return false; + } + + return true; + } + + private static bool TryMatchHighTimesHigh(Operation operation, Operand x, Operand lowTimesHighResult) + { + // xHigh = x >> 16 + // lowTimesHighResultHigh = lowTimesHighResult >> 16 + // highTimesHigh = xHigh * lowTimesHighResultHigh + // result = highTimesHigh << 16 + + if (operation.Inst != Instruction.ShiftLeft || !IsConst(operation.GetSource(1), 16)) + { + return false; + } + + Operand mulResult = operation.GetSource(0); + + if (mulResult.AsgOp is not Operation mulOp || mulOp.Inst != Instruction.Multiply) + { + return false; + } + + if (GetShifted16Source(mulOp.GetSource(0), Instruction.ShiftRightU32) != x) + { + return false; + } + + if (GetCopySource(GetShifted16Source(mulOp.GetSource(1), Instruction.ShiftRightU32)) != lowTimesHighResult) + { + return false; + } + + return true; + } + + private static Operand GetMasked16Source(Operand value) + { + if (value.AsgOp is not Operation maskOp) + { + return null; + } + + if (maskOp.Inst != Instruction.BitwiseAnd || !IsConst(maskOp.GetSource(1), ushort.MaxValue)) + { + return null; + } + + return maskOp.GetSource(0); + } + + private static Operand GetShifted16Source(Operand value, Instruction shiftInst) + { + if (value.AsgOp is not Operation shiftOp) + { + return null; + } + + if (shiftOp.Inst != shiftInst || !IsConst(shiftOp.GetSource(1), 16)) + { + return null; + } + + return shiftOp.GetSource(0); + } + + private static Operand GetCopySource(Operand value) + { + while (value.AsgOp is Operation operation && IsCopy(operation)) + { + value = operation.GetSource(0); + } + + return value; + } + + private static bool IsCopy(Operation operation) + { + return operation.Inst == Instruction.Copy || (operation.Inst == Instruction.Add && IsConst(operation.GetSource(1), 0)); + } + + private static bool IsConst(Operand operand, int value) + { + return operand.Type == OperandType.Constant && operand.Value == value; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/RegisterUsage.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/RegisterUsage.cs new file mode 100644 index 0000000..ca43520 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/RegisterUsage.cs @@ -0,0 +1,491 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Numerics; + +namespace Ryujinx.Graphics.Shader.Translation +{ + static class RegisterUsage + { + private const int RegsCount = 256; + private const int RegsMask = RegsCount - 1; + + private const int GprMasks = 4; + private const int PredMasks = 1; + private const int FlagMasks = 1; + private const int TotalMasks = GprMasks + PredMasks + FlagMasks; + + private struct RegisterMask : IEquatable + { + public long GprMask0 { get; set; } + public long GprMask1 { get; set; } + public long GprMask2 { get; set; } + public long GprMask3 { get; set; } + public long PredMask { get; set; } + public long FlagMask { get; set; } + + public RegisterMask(long gprMask0, long gprMask1, long gprMask2, long gprMask3, long predMask, long flagMask) + { + GprMask0 = gprMask0; + GprMask1 = gprMask1; + GprMask2 = gprMask2; + GprMask3 = gprMask3; + PredMask = predMask; + FlagMask = flagMask; + } + + public readonly long GetMask(int index) + { + return index switch + { + 0 => GprMask0, + 1 => GprMask1, + 2 => GprMask2, + 3 => GprMask3, + 4 => PredMask, + 5 => FlagMask, + _ => throw new ArgumentOutOfRangeException(nameof(index)), + }; + } + + public static RegisterMask operator &(RegisterMask x, RegisterMask y) + { + return new RegisterMask( + x.GprMask0 & y.GprMask0, + x.GprMask1 & y.GprMask1, + x.GprMask2 & y.GprMask2, + x.GprMask3 & y.GprMask3, + x.PredMask & y.PredMask, + x.FlagMask & y.FlagMask); + } + + public static RegisterMask operator |(RegisterMask x, RegisterMask y) + { + return new RegisterMask( + x.GprMask0 | y.GprMask0, + x.GprMask1 | y.GprMask1, + x.GprMask2 | y.GprMask2, + x.GprMask3 | y.GprMask3, + x.PredMask | y.PredMask, + x.FlagMask | y.FlagMask); + } + + public static RegisterMask operator ~(RegisterMask x) + { + return new RegisterMask( + ~x.GprMask0, + ~x.GprMask1, + ~x.GprMask2, + ~x.GprMask3, + ~x.PredMask, + ~x.FlagMask); + } + + public static bool operator ==(RegisterMask x, RegisterMask y) + { + return x.Equals(y); + } + + public static bool operator !=(RegisterMask x, RegisterMask y) + { + return !x.Equals(y); + } + + public readonly override bool Equals(object obj) + { + return obj is RegisterMask regMask && Equals(regMask); + } + + public readonly bool Equals(RegisterMask other) + { + return GprMask0 == other.GprMask0 && + GprMask1 == other.GprMask1 && + GprMask2 == other.GprMask2 && + GprMask3 == other.GprMask3 && + PredMask == other.PredMask && + FlagMask == other.FlagMask; + } + + public readonly override int GetHashCode() + { + return HashCode.Combine(GprMask0, GprMask1, GprMask2, GprMask3, PredMask, FlagMask); + } + } + + public readonly struct FunctionRegisterUsage + { + public Register[] InArguments { get; } + public Register[] OutArguments { get; } + + public FunctionRegisterUsage(Register[] inArguments, Register[] outArguments) + { + InArguments = inArguments; + OutArguments = outArguments; + } + } + + public static FunctionRegisterUsage RunPass(ControlFlowGraph cfg) + { + List inArguments = []; + List outArguments = []; + + // Compute local register inputs and outputs used inside blocks. + RegisterMask[] localInputs = new RegisterMask[cfg.Blocks.Length]; + RegisterMask[] localOutputs = new RegisterMask[cfg.Blocks.Length]; + + foreach (BasicBlock block in cfg.Blocks) + { + for (LinkedListNode node = block.Operations.First; node != null; node = node.Next) + { + Operation operation = node.Value as Operation; + + for (int srcIndex = 0; srcIndex < operation.SourcesCount; srcIndex++) + { + Operand source = operation.GetSource(srcIndex); + + if (source.Type != OperandType.Register) + { + continue; + } + + Register register = source.GetRegister(); + + localInputs[block.Index] |= GetMask(register) & ~localOutputs[block.Index]; + } + + for (int dstIndex = 0; dstIndex < operation.DestsCount; dstIndex++) + { + Operand dest = operation.GetDest(dstIndex); + + if (dest != null && dest.Type == OperandType.Register) + { + localOutputs[block.Index] |= GetMask(dest.GetRegister()); + } + } + } + } + + // Compute global register inputs and outputs used across blocks. + RegisterMask[] globalCmnOutputs = new RegisterMask[cfg.Blocks.Length]; + + RegisterMask[] globalInputs = new RegisterMask[cfg.Blocks.Length]; + RegisterMask[] globalOutputs = new RegisterMask[cfg.Blocks.Length]; + + RegisterMask allOutputs = new(); + RegisterMask allCmnOutputs = new(-1L, -1L, -1L, -1L, -1L, -1L); + + bool modified; + + bool firstPass = true; + + do + { + modified = false; + + // Compute register outputs. + for (int index = cfg.PostOrderBlocks.Length - 1; index >= 0; index--) + { + BasicBlock block = cfg.PostOrderBlocks[index]; + + if (block.Predecessors.Count != 0) + { + BasicBlock predecessor = block.Predecessors[0]; + + RegisterMask cmnOutputs = localOutputs[predecessor.Index] | globalCmnOutputs[predecessor.Index]; + + RegisterMask outputs = globalOutputs[predecessor.Index]; + + for (int pIndex = 1; pIndex < block.Predecessors.Count; pIndex++) + { + predecessor = block.Predecessors[pIndex]; + + cmnOutputs &= localOutputs[predecessor.Index] | globalCmnOutputs[predecessor.Index]; + + outputs |= globalOutputs[predecessor.Index]; + } + + globalInputs[block.Index] |= outputs & ~cmnOutputs; + + if (!firstPass) + { + cmnOutputs &= globalCmnOutputs[block.Index]; + } + + if (EndsWithReturn(block)) + { + allCmnOutputs &= cmnOutputs | localOutputs[block.Index]; + } + + if (Exchange(globalCmnOutputs, block.Index, cmnOutputs)) + { + modified = true; + } + + outputs |= localOutputs[block.Index]; + + if (Exchange(globalOutputs, block.Index, globalOutputs[block.Index] | outputs)) + { + allOutputs |= outputs; + modified = true; + } + } + else if (Exchange(globalOutputs, block.Index, localOutputs[block.Index])) + { + allOutputs |= localOutputs[block.Index]; + modified = true; + } + } + + // Compute register inputs. + for (int index = 0; index < cfg.PostOrderBlocks.Length; index++) + { + BasicBlock block = cfg.PostOrderBlocks[index]; + + RegisterMask inputs = localInputs[block.Index]; + + if (block.Next != null) + { + inputs |= globalInputs[block.Next.Index]; + } + + if (block.Branch != null) + { + inputs |= globalInputs[block.Branch.Index]; + } + + inputs &= ~globalCmnOutputs[block.Index]; + + if (Exchange(globalInputs, block.Index, globalInputs[block.Index] | inputs)) + { + modified = true; + } + } + + firstPass = false; + } + while (modified); + + // Insert load and store context instructions where needed. + foreach (BasicBlock block in cfg.Blocks) + { + // The only block without any predecessor should be the entry block. + // It always needs a context load as it is the first block to run. + if (block.Predecessors.Count == 0) + { + RegisterMask inputs = globalInputs[block.Index] | (allOutputs & ~allCmnOutputs); + + LoadLocals(block, inputs, inArguments); + } + + if (EndsWithReturn(block)) + { + StoreLocals(block, allOutputs, inArguments.Count, outArguments); + } + } + + return new FunctionRegisterUsage(inArguments.ToArray(), outArguments.ToArray()); + } + + public static void FixupCalls(BasicBlock[] blocks, FunctionRegisterUsage[] frus) + { + foreach (BasicBlock block in blocks) + { + for (LinkedListNode node = block.Operations.First; node != null; node = node.Next) + { + Operation operation = node.Value as Operation; + + if (operation.Inst == Instruction.Call) + { + Operand funcId = operation.GetSource(0); + + Debug.Assert(funcId.Type == OperandType.Constant); + + FunctionRegisterUsage fru = frus[funcId.Value]; + + Operand[] inRegs = new Operand[fru.InArguments.Length]; + + for (int i = 0; i < fru.InArguments.Length; i++) + { + inRegs[i] = OperandHelper.Register(fru.InArguments[i]); + } + + operation.AppendSources(inRegs); + + Operand[] outRegs = new Operand[1 + fru.OutArguments.Length]; + + for (int i = 0; i < fru.OutArguments.Length; i++) + { + outRegs[1 + i] = OperandHelper.Register(fru.OutArguments[i]); + } + + operation.AppendDests(outRegs); + } + } + } + } + + private static bool StartsWith(BasicBlock block, Instruction inst) + { + if (block.Operations.Count == 0) + { + return false; + } + + return block.Operations.First.Value is Operation operation && operation.Inst == inst; + } + + private static bool EndsWith(BasicBlock block, Instruction inst) + { + if (block.Operations.Count == 0) + { + return false; + } + + return block.Operations.Last.Value is Operation operation && operation.Inst == inst; + } + + private static RegisterMask GetMask(Register register) + { + Span gprMasks = stackalloc long[4]; + long predMask = 0; + long flagMask = 0; + + switch (register.Type) + { + case RegisterType.Gpr: + gprMasks[register.Index >> 6] = 1L << (register.Index & 0x3f); + break; + case RegisterType.Predicate: + predMask = 1L << register.Index; + break; + case RegisterType.Flag: + flagMask = 1L << register.Index; + break; + } + + return new RegisterMask(gprMasks[0], gprMasks[1], gprMasks[2], gprMasks[3], predMask, flagMask); + } + + private static bool Exchange(RegisterMask[] masks, int blkIndex, RegisterMask value) + { + RegisterMask oldValue = masks[blkIndex]; + + masks[blkIndex] = value; + + return oldValue != value; + } + + private static void LoadLocals(BasicBlock block, RegisterMask masks, List inArguments) + { + bool fillArgsList = inArguments.Count == 0; + LinkedListNode node = null; + int argIndex = 0; + + for (int i = 0; i < TotalMasks; i++) + { + (RegisterType regType, int baseRegIndex) = GetRegTypeAndBaseIndex(i); + long mask = masks.GetMask(i); + + while (mask != 0) + { + int bit = BitOperations.TrailingZeroCount(mask); + + mask &= ~(1L << bit); + + Register register = new(baseRegIndex + bit, regType); + + if (fillArgsList) + { + inArguments.Add(register); + } + + Operation copyOp = new(Instruction.Copy, OperandHelper.Register(register), OperandHelper.Argument(argIndex++)); + + if (node == null) + { + node = block.Operations.AddFirst(copyOp); + } + else + { + node = block.Operations.AddAfter(node, copyOp); + } + } + } + + Debug.Assert(argIndex <= inArguments.Count); + } + + private static void StoreLocals(BasicBlock block, RegisterMask masks, int inArgumentsCount, List outArguments) + { + LinkedListNode node = null; + int argIndex = inArgumentsCount; + bool fillArgsList = outArguments.Count == 0; + + for (int i = 0; i < TotalMasks; i++) + { + (RegisterType regType, int baseRegIndex) = GetRegTypeAndBaseIndex(i); + long mask = masks.GetMask(i); + + while (mask != 0) + { + int bit = BitOperations.TrailingZeroCount(mask); + + mask &= ~(1L << bit); + + Register register = new(baseRegIndex + bit, regType); + + if (fillArgsList) + { + outArguments.Add(register); + } + + Operation copyOp = new(Instruction.Copy, OperandHelper.Argument(argIndex++), OperandHelper.Register(register)); + + if (node == null) + { + node = block.Operations.AddBefore(block.Operations.Last, copyOp); + } + else + { + node = block.Operations.AddAfter(node, copyOp); + } + } + } + + Debug.Assert(argIndex <= inArgumentsCount + outArguments.Count); + } + + private static (RegisterType RegType, int BaseRegIndex) GetRegTypeAndBaseIndex(int i) + { + RegisterType regType = RegisterType.Gpr; + int baseRegIndex = 0; + + if (i < GprMasks) + { + baseRegIndex = i * sizeof(long) * 8; + } + else if (i == GprMasks) + { + regType = RegisterType.Predicate; + } + else + { + regType = RegisterType.Flag; + } + + return (regType, baseRegIndex); + } + + private static bool EndsWithReturn(BasicBlock block) + { + if (block.GetLastOp() is not Operation operation) + { + return false; + } + + return operation.Inst == Instruction.Return; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/ResourceManager.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/ResourceManager.cs new file mode 100644 index 0000000..af359a6 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/ResourceManager.cs @@ -0,0 +1,649 @@ +using Ryujinx.Common; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.StructuredIr; +using System; +using System.Collections.Generic; +using System.Globalization; + +namespace Ryujinx.Graphics.Shader.Translation +{ + class ResourceManager + { + // Those values are used if the shader as local or shared memory access, + // but for some reason the supplied size was 0. + private const int DefaultLocalMemorySize = 128; + private const int DefaultSharedMemorySize = 4096; + + private static readonly string[] _stagePrefixes = new string[] { "cp", "vp", "tcp", "tep", "gp", "fp" }; + + private readonly IGpuAccessor _gpuAccessor; + private readonly ShaderStage _stage; + private readonly string _stagePrefix; + + private readonly SetBindingPair[] _cbSlotToBindingMap; + private readonly SetBindingPair[] _sbSlotToBindingMap; + private uint _sbSlotWritten; + + private readonly Dictionary _sbSlots; + private readonly Dictionary _sbSlotsReverse; + + private readonly HashSet _usedConstantBufferBindings; + + private readonly record struct TextureInfo(int CbufSlot, int Handle, int ArrayLength, bool Separate, SamplerType Type, TextureFormat Format); + + private struct TextureMeta + { + public int Set; + public int Binding; + public bool AccurateType; + public SamplerType Type; + public TextureUsageFlags UsageFlags; + } + + private readonly Dictionary _usedTextures; + private readonly Dictionary _usedImages; + + public int LocalMemoryId { get; private set; } + public int SharedMemoryId { get; private set; } + + public int LocalVertexDataMemoryId { get; private set; } + public int LocalTopologyRemapMemoryId { get; private set; } + public int LocalVertexIndexVertexRateMemoryId { get; private set; } + public int LocalVertexIndexInstanceRateMemoryId { get; private set; } + public int LocalGeometryOutputVertexCountMemoryId { get; private set; } + public int LocalGeometryOutputIndexCountMemoryId { get; private set; } + + public ShaderProperties Properties { get; } + + public ResourceReservations Reservations { get; } + + public ResourceManager(ShaderStage stage, IGpuAccessor gpuAccessor, ResourceReservations reservations = null) + { + _gpuAccessor = gpuAccessor; + Properties = new(); + Reservations = reservations; + _stage = stage; + _stagePrefix = GetShaderStagePrefix(stage); + + _cbSlotToBindingMap = new SetBindingPair[18]; + _sbSlotToBindingMap = new SetBindingPair[16]; + _cbSlotToBindingMap.AsSpan().Fill(new(-1, -1)); + _sbSlotToBindingMap.AsSpan().Fill(new(-1, -1)); + + _sbSlots = new(); + _sbSlotsReverse = new(); + + _usedConstantBufferBindings = new(); + + _usedTextures = new(); + _usedImages = new(); + + Properties.AddOrUpdateConstantBuffer(new(BufferLayout.Std140, 0, SupportBuffer.Binding, "support_buffer", SupportBuffer.GetStructureType())); + + LocalMemoryId = -1; + SharedMemoryId = -1; + } + + public void SetCurrentLocalMemory(int size, bool isUsed) + { + if (isUsed) + { + if (size <= 0) + { + size = DefaultLocalMemorySize; + } + + MemoryDefinition lmem = new MemoryDefinition("local_memory", AggregateType.Array | AggregateType.U32, BitUtils.DivRoundUp(size, sizeof(uint))); + + LocalMemoryId = Properties.AddLocalMemory(lmem); + } + else + { + LocalMemoryId = -1; + } + } + + public void SetCurrentSharedMemory(int size, bool isUsed) + { + if (isUsed) + { + if (size <= 0) + { + size = DefaultSharedMemorySize; + } + + MemoryDefinition smem = new MemoryDefinition("shared_memory", AggregateType.Array | AggregateType.U32, BitUtils.DivRoundUp(size, sizeof(uint))); + + SharedMemoryId = Properties.AddSharedMemory(smem); + } + else + { + SharedMemoryId = -1; + } + } + + public void SetVertexAsComputeLocalMemories(ShaderStage stage, InputTopology inputTopology) + { + LocalVertexDataMemoryId = AddMemoryDefinition("local_vertex_data", AggregateType.Array | AggregateType.FP32, Reservations.OutputSizePerInvocation); + + if (stage == ShaderStage.Vertex) + { + LocalVertexIndexVertexRateMemoryId = AddMemoryDefinition("local_vertex_index_vr", AggregateType.U32); + LocalVertexIndexInstanceRateMemoryId = AddMemoryDefinition("local_vertex_index_ir", AggregateType.U32); + } + else if (stage == ShaderStage.Geometry) + { + LocalTopologyRemapMemoryId = AddMemoryDefinition("local_topology_remap", AggregateType.Array | AggregateType.U32, inputTopology.InputVertexCount); + + LocalGeometryOutputVertexCountMemoryId = AddMemoryDefinition("local_geometry_output_vertex", AggregateType.U32); + LocalGeometryOutputIndexCountMemoryId = AddMemoryDefinition("local_geometry_output_index", AggregateType.U32); + } + } + + private int AddMemoryDefinition(string name, AggregateType type, int arrayLength = 1) + { + return Properties.AddLocalMemory(new MemoryDefinition(name, type, arrayLength)); + } + + public int GetConstantBufferBinding(int slot) + { + SetBindingPair setAndBinding = _cbSlotToBindingMap[slot]; + if (setAndBinding.Binding < 0) + { + setAndBinding = _gpuAccessor.CreateConstantBufferBinding(slot); + _cbSlotToBindingMap[slot] = setAndBinding; + string slotNumber = slot.ToString(CultureInfo.InvariantCulture); + AddNewConstantBuffer(setAndBinding.SetIndex, setAndBinding.Binding, $"{_stagePrefix}_c{slotNumber}"); + } + + return setAndBinding.Binding; + } + + public bool TryGetStorageBufferBinding(int sbCbSlot, int sbCbOffset, bool write, out int binding) + { + if (!TryGetSbSlot((byte)sbCbSlot, (ushort)sbCbOffset, out int slot)) + { + binding = 0; + return false; + } + + SetBindingPair setAndBinding = _sbSlotToBindingMap[slot]; + + if (setAndBinding.Binding < 0) + { + setAndBinding = _gpuAccessor.CreateStorageBufferBinding(slot); + _sbSlotToBindingMap[slot] = setAndBinding; + string slotNumber = slot.ToString(CultureInfo.InvariantCulture); + AddNewStorageBuffer(setAndBinding.SetIndex, setAndBinding.Binding, $"{_stagePrefix}_s{slotNumber}"); + } + + if (write) + { + _sbSlotWritten |= 1u << slot; + } + + binding = setAndBinding.Binding; + return true; + } + + private bool TryGetSbSlot(byte sbCbSlot, ushort sbCbOffset, out int slot) + { + int key = PackSbCbInfo(sbCbSlot, sbCbOffset); + + if (!_sbSlots.TryGetValue(key, out slot)) + { + slot = _sbSlots.Count; + + if (slot >= _sbSlotToBindingMap.Length) + { + return false; + } + + _sbSlots.Add(key, slot); + _sbSlotsReverse.Add(slot, key); + } + + return true; + } + + public bool TryGetConstantBufferSlot(int binding, out int slot) + { + for (slot = 0; slot < _cbSlotToBindingMap.Length; slot++) + { + if (_cbSlotToBindingMap[slot].Binding == binding) + { + return true; + } + } + + slot = 0; + return false; + } + + public SetBindingPair GetTextureOrImageBinding( + Instruction inst, + SamplerType type, + TextureFormat format, + TextureFlags flags, + int cbufSlot, + int handle, + int arrayLength = 1, + bool separate = false) + { + inst &= Instruction.Mask; + bool isImage = inst.IsImage(); + bool isWrite = inst.IsImageStore(); + bool accurateType = !inst.IsTextureQuery(); + bool intCoords = isImage || flags.HasFlag(TextureFlags.IntCoords) || inst == Instruction.TextureQuerySize; + bool coherent = flags.HasFlag(TextureFlags.Coherent); + + if (!isImage) + { + format = TextureFormat.Unknown; + } + + SetBindingPair setAndBinding = GetTextureOrImageBinding( + cbufSlot, + handle, + arrayLength, + type, + format, + isImage, + intCoords, + isWrite, + accurateType, + coherent, + separate); + + _gpuAccessor.RegisterTexture(handle, cbufSlot); + + return setAndBinding; + } + + private SetBindingPair GetTextureOrImageBinding( + int cbufSlot, + int handle, + int arrayLength, + SamplerType type, + TextureFormat format, + bool isImage, + bool intCoords, + bool write, + bool accurateType, + bool coherent, + bool separate) + { + int dimensions = type == SamplerType.None ? 0 : type.Dimensions; + Dictionary dict = isImage ? _usedImages : _usedTextures; + + TextureUsageFlags usageFlags = TextureUsageFlags.None; + + if (intCoords) + { + usageFlags |= TextureUsageFlags.NeedsScaleValue; + + bool canScale = _stage.SupportsRenderScale && arrayLength == 1 && !write && dimensions == 2; + + if (!canScale) + { + // Resolution scaling cannot be applied to this texture right now. + // Flag so that we know to blacklist scaling on related textures when binding them. + usageFlags |= TextureUsageFlags.ResScaleUnsupported; + } + } + + if (write) + { + usageFlags |= TextureUsageFlags.ImageStore; + } + + if (coherent) + { + usageFlags |= TextureUsageFlags.ImageCoherent; + } + + // For array textures, we also want to use type as key, + // since we may have texture handles stores in the same buffer, but for textures with different types. + SamplerType keyType = arrayLength > 1 ? type : SamplerType.None; + TextureInfo info = new TextureInfo(cbufSlot, handle, arrayLength, separate, keyType, format); + TextureMeta meta = new TextureMeta() + { + AccurateType = accurateType, + Type = type, + UsageFlags = usageFlags, + }; + + int setIndex; + int binding; + + if (dict.TryGetValue(info, out TextureMeta existingMeta)) + { + dict[info] = MergeTextureMeta(meta, existingMeta); + setIndex = existingMeta.Set; + binding = existingMeta.Binding; + } + else + { + if (arrayLength > 1 && (setIndex = _gpuAccessor.CreateExtraSet()) >= 0) + { + // We reserved an "extra set" for the array. + // In this case the binding is always the first one (0). + // Using separate sets for array is better as we need to do less descriptor set updates. + + binding = 0; + } + else + { + bool isBuffer = (type & SamplerType.Mask) == SamplerType.TextureBuffer; + + SetBindingPair setAndBinding = isImage + ? _gpuAccessor.CreateImageBinding(arrayLength, isBuffer) + : _gpuAccessor.CreateTextureBinding(arrayLength, isBuffer); + + setIndex = setAndBinding.SetIndex; + binding = setAndBinding.Binding; + } + + meta.Set = setIndex; + meta.Binding = binding; + + dict.Add(info, meta); + } + + string nameSuffix; + string prefix = isImage ? "i" : "t"; + + if (arrayLength != 1 && type != SamplerType.None) + { + prefix += type.ShortTypeName; + } + + if (isImage) + { + nameSuffix = cbufSlot < 0 + ? $"{prefix}_tcb_{handle:X}_{format.ToGlslFormat()}" + : $"{prefix}_cb{cbufSlot}_{handle:X}_{format.ToGlslFormat()}"; + } + else if (type == SamplerType.None) + { + nameSuffix = cbufSlot < 0 ? $"s_tcb_{handle:X}" : $"s_cb{cbufSlot}_{handle:X}"; + } + else + { + nameSuffix = cbufSlot < 0 ? $"{prefix}_tcb_{handle:X}" : $"{prefix}_cb{cbufSlot}_{handle:X}"; + } + + TextureDefinition definition = new TextureDefinition( + setIndex, + binding, + arrayLength, + separate, + $"{_stagePrefix}_{nameSuffix}", + meta.Type, + info.Format, + meta.UsageFlags); + + if (isImage) + { + Properties.AddOrUpdateImage(definition); + } + else + { + Properties.AddOrUpdateTexture(definition); + } + + return new SetBindingPair(setIndex, binding); + } + + private static TextureMeta MergeTextureMeta(TextureMeta meta, TextureMeta existingMeta) + { + meta.Set = existingMeta.Set; + meta.Binding = existingMeta.Binding; + meta.UsageFlags |= existingMeta.UsageFlags; + + // If the texture we have has inaccurate type information, then + // we prefer the most accurate one. + if (existingMeta.AccurateType) + { + meta.AccurateType = true; + meta.Type = existingMeta.Type; + } + + return meta; + } + + public void SetUsageFlagsForTextureQuery(int binding, SamplerType type) + { + TextureInfo selectedInfo = default; + TextureMeta selectedMeta = default; + bool found = false; + + foreach ((TextureInfo info, TextureMeta meta) in _usedTextures) + { + if (meta.Binding == binding) + { + selectedInfo = info; + selectedMeta = meta; + found = true; + break; + } + } + + if (found) + { + selectedMeta.UsageFlags |= TextureUsageFlags.NeedsScaleValue; + + int dimensions = type.Dimensions; + bool canScale = _stage.SupportsRenderScale && selectedInfo.ArrayLength == 1 && dimensions == 2; + + if (!canScale) + { + // Resolution scaling cannot be applied to this texture right now. + // Flag so that we know to blacklist scaling on related textures when binding them. + selectedMeta.UsageFlags |= TextureUsageFlags.ResScaleUnsupported; + } + + _usedTextures[selectedInfo] = selectedMeta; + } + } + + public void SetUsedConstantBufferBinding(int binding) + { + _usedConstantBufferBindings.Add(binding); + } + + public BufferDescriptor[] GetConstantBufferDescriptors() + { + BufferDescriptor[] descriptors = new BufferDescriptor[_usedConstantBufferBindings.Count]; + + int descriptorIndex = 0; + + for (int slot = 0; slot < _cbSlotToBindingMap.Length; slot++) + { + SetBindingPair setAndBinding = _cbSlotToBindingMap[slot]; + + if (setAndBinding.Binding >= 0 && _usedConstantBufferBindings.Contains(setAndBinding.Binding)) + { + descriptors[descriptorIndex++] = new BufferDescriptor(setAndBinding.SetIndex, setAndBinding.Binding, slot); + } + } + + if (descriptors.Length != descriptorIndex) + { + Array.Resize(ref descriptors, descriptorIndex); + } + + return descriptors; + } + + public BufferDescriptor[] GetStorageBufferDescriptors() + { + BufferDescriptor[] descriptors = new BufferDescriptor[_sbSlots.Count]; + + int descriptorIndex = 0; + + foreach ((int key, int slot) in _sbSlots) + { + SetBindingPair setAndBinding = _sbSlotToBindingMap[slot]; + + if (setAndBinding.Binding >= 0) + { + (int sbCbSlot, int sbCbOffset) = UnpackSbCbInfo(key); + BufferUsageFlags flags = (_sbSlotWritten & (1u << slot)) != 0 ? BufferUsageFlags.Write : BufferUsageFlags.None; + descriptors[descriptorIndex++] = new BufferDescriptor(setAndBinding.SetIndex, setAndBinding.Binding, slot, sbCbSlot, sbCbOffset, flags); + } + } + + if (descriptors.Length != descriptorIndex) + { + Array.Resize(ref descriptors, descriptorIndex); + } + + return descriptors; + } + + public TextureDescriptor[] GetTextureDescriptors(bool includeArrays = true) + { + return GetDescriptors(_usedTextures, includeArrays); + } + + public TextureDescriptor[] GetImageDescriptors(bool includeArrays = true) + { + return GetDescriptors(_usedImages, includeArrays); + } + + private static TextureDescriptor[] GetDescriptors(IReadOnlyDictionary usedResources, bool includeArrays) + { + List descriptors = new(); + + bool hasAnyArray = false; + + foreach ((TextureInfo info, TextureMeta meta) in usedResources) + { + if (info.ArrayLength > 1) + { + hasAnyArray = true; + continue; + } + + descriptors.Add(new TextureDescriptor( + meta.Set, + meta.Binding, + meta.Type, + info.Format, + info.CbufSlot, + info.Handle, + info.ArrayLength, + info.Separate, + meta.UsageFlags)); + } + + if (hasAnyArray && includeArrays) + { + foreach ((TextureInfo info, TextureMeta meta) in usedResources) + { + if (info.ArrayLength <= 1) + { + continue; + } + + descriptors.Add(new TextureDescriptor( + meta.Set, + meta.Binding, + meta.Type, + info.Format, + info.CbufSlot, + info.Handle, + info.ArrayLength, + info.Separate, + meta.UsageFlags)); + } + } + + return descriptors.ToArray(); + } + + public bool TryGetCbufSlotAndHandleForTexture(int binding, out int cbufSlot, out int handle) + { + foreach ((TextureInfo info, TextureMeta meta) in _usedTextures) + { + if (meta.Binding == binding) + { + cbufSlot = info.CbufSlot; + handle = info.Handle; + + return true; + } + } + + cbufSlot = 0; + handle = 0; + return false; + } + + private static int FindDescriptorIndex(TextureDescriptor[] array, int binding) + { + return Array.FindIndex(array, x => x.Binding == binding); + } + + public int FindTextureDescriptorIndex(int binding) + { + return FindDescriptorIndex(GetTextureDescriptors(), binding); + } + + public int FindImageDescriptorIndex(int binding) + { + return FindDescriptorIndex(GetImageDescriptors(), binding); + } + + public bool IsArrayOfTexturesOrImages(int binding, bool isImage) + { + foreach ((TextureInfo info, TextureMeta meta) in isImage ? _usedImages : _usedTextures) + { + if (meta.Binding == binding) + { + return info.ArrayLength != 1; + } + } + + return false; + } + + private void AddNewConstantBuffer(int setIndex, int binding, string name) + { + StructureType type = new(new[] + { + new StructureField(AggregateType.Array | AggregateType.Vector4 | AggregateType.FP32, "data", Constants.ConstantBufferSize / 16), + }); + + Properties.AddOrUpdateConstantBuffer(new(BufferLayout.Std140, setIndex, binding, name, type)); + } + + private void AddNewStorageBuffer(int setIndex, int binding, string name) + { + StructureType type = new(new[] + { + new StructureField(AggregateType.Array | AggregateType.U32, "data", 0), + }); + + Properties.AddOrUpdateStorageBuffer(new(BufferLayout.Std430, setIndex, binding, name, type)); + } + + public static string GetShaderStagePrefix(ShaderStage stage) + { + uint index = (uint)stage; + + return index >= _stagePrefixes.Length ? "invalid" : _stagePrefixes[index]; + } + + private static int PackSbCbInfo(int sbCbSlot, int sbCbOffset) + { + return sbCbOffset | (sbCbSlot << 16); + } + + private static (int, int) UnpackSbCbInfo(int key) + { + return ((byte)(key >> 16), (ushort)key); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/ResourceReservations.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/ResourceReservations.cs new file mode 100644 index 0000000..cf0fa18 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/ResourceReservations.cs @@ -0,0 +1,203 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.StructuredIr; +using System.Collections.Generic; +using System.Numerics; + +namespace Ryujinx.Graphics.Shader.Translation +{ + public class ResourceReservations + { + public const int TfeBuffersCount = 4; + + public const int MaxVertexBufferTextures = 32; + + private const int TextureSetIndex = 2; // TODO: Get from GPU accessor. + + public int VertexInfoConstantBufferBinding { get; } + public int VertexOutputStorageBufferBinding { get; } + public int GeometryVertexOutputStorageBufferBinding { get; } + public int GeometryIndexOutputStorageBufferBinding { get; } + public int IndexBufferTextureBinding { get; } + public int TopologyRemapBufferTextureBinding { get; } + + public int ReservedConstantBuffers { get; } + public int ReservedStorageBuffers { get; } + public int ReservedTextures { get; } + public int ReservedImages { get; } + public int InputSizePerInvocation { get; } + public int OutputSizePerInvocation { get; } + public int OutputSizeInBytesPerInvocation => OutputSizePerInvocation * sizeof(uint); + + private readonly int _tfeBufferSbBaseBinding; + private readonly int _vertexBufferTextureBaseBinding; + + private readonly Dictionary _offsets; + internal IReadOnlyDictionary Offsets => _offsets; + + internal ResourceReservations(bool isTransformFeedbackEmulated, bool vertexAsCompute) + { + // All stages reserves the first constant buffer binding for the support buffer. + ReservedConstantBuffers = 1; + ReservedStorageBuffers = 0; + ReservedTextures = 0; + ReservedImages = 0; + + if (isTransformFeedbackEmulated) + { + // Transform feedback emulation currently always uses 4 storage buffers. + _tfeBufferSbBaseBinding = ReservedStorageBuffers; + ReservedStorageBuffers = TfeBuffersCount; + } + + if (vertexAsCompute) + { + // One constant buffer reserved for vertex related state. + VertexInfoConstantBufferBinding = ReservedConstantBuffers++; + + // One storage buffer for the output vertex data. + VertexOutputStorageBufferBinding = ReservedStorageBuffers++; + + // One storage buffer for the output geometry vertex data. + GeometryVertexOutputStorageBufferBinding = ReservedStorageBuffers++; + + // One storage buffer for the output geometry index data. + GeometryIndexOutputStorageBufferBinding = ReservedStorageBuffers++; + + // Enough textures reserved for all vertex attributes, plus the index buffer. + IndexBufferTextureBinding = ReservedTextures; + TopologyRemapBufferTextureBinding = ReservedTextures + 1; + _vertexBufferTextureBaseBinding = ReservedTextures + 2; + ReservedTextures += 2 + MaxVertexBufferTextures; + } + } + + internal ResourceReservations( + IGpuAccessor gpuAccessor, + bool isTransformFeedbackEmulated, + bool vertexAsCompute, + IoUsage? vacInput, + IoUsage vacOutput) : this(isTransformFeedbackEmulated, vertexAsCompute) + { + if (vertexAsCompute) + { + _offsets = new(); + + if (vacInput.HasValue) + { + InputSizePerInvocation = FillIoOffsetMap(gpuAccessor, StorageKind.Input, vacInput.Value); + } + + OutputSizePerInvocation = FillIoOffsetMap(gpuAccessor, StorageKind.Output, vacOutput); + } + } + + private int FillIoOffsetMap(IGpuAccessor gpuAccessor, StorageKind storageKind, IoUsage vacUsage) + { + int offset = 0; + + for (int c = 0; c < 4; c++) + { + _offsets.Add(new IoDefinition(storageKind, IoVariable.Position, 0, c), offset++); + } + + _offsets.Add(new IoDefinition(storageKind, IoVariable.PointSize), offset++); + + int clipDistancesWrittenMap = vacUsage.ClipDistancesWritten; + + while (clipDistancesWrittenMap != 0) + { + int index = BitOperations.TrailingZeroCount(clipDistancesWrittenMap); + + _offsets.Add(new IoDefinition(storageKind, IoVariable.ClipDistance, 0, index), offset++); + + clipDistancesWrittenMap &= ~(1 << index); + } + + if (vacUsage.UsesRtLayer) + { + _offsets.Add(new IoDefinition(storageKind, IoVariable.Layer), offset++); + } + + if (vacUsage.UsesViewportIndex && gpuAccessor.QueryHostSupportsViewportIndexVertexTessellation()) + { + _offsets.Add(new IoDefinition(storageKind, IoVariable.VertexIndex), offset++); + } + + if (vacUsage.UsesViewportMask && gpuAccessor.QueryHostSupportsViewportMask()) + { + _offsets.Add(new IoDefinition(storageKind, IoVariable.ViewportMask), offset++); + } + + int usedDefinedMap = vacUsage.UserDefinedMap; + + while (usedDefinedMap != 0) + { + int location = BitOperations.TrailingZeroCount(usedDefinedMap); + + for (int c = 0; c < 4; c++) + { + _offsets.Add(new IoDefinition(storageKind, IoVariable.UserDefined, location, c), offset++); + } + + usedDefinedMap &= ~(1 << location); + } + + return offset; + } + + internal static bool IsVectorOrArrayVariable(IoVariable variable) + { + return variable switch + { + IoVariable.ClipDistance or + IoVariable.Position => true, + _ => false, + }; + } + + public int GetTfeBufferStorageBufferBinding(int bufferIndex) + { + return _tfeBufferSbBaseBinding + bufferIndex; + } + + public int GetVertexBufferTextureBinding(int vaLocation) + { + return _vertexBufferTextureBaseBinding + vaLocation; + } + + public SetBindingPair GetVertexBufferTextureSetAndBinding(int vaLocation) + { + return new SetBindingPair(TextureSetIndex, GetVertexBufferTextureBinding(vaLocation)); + } + + public SetBindingPair GetIndexBufferTextureSetAndBinding() + { + return new SetBindingPair(TextureSetIndex, IndexBufferTextureBinding); + } + + public SetBindingPair GetTopologyRemapBufferTextureSetAndBinding() + { + return new SetBindingPair(TextureSetIndex, TopologyRemapBufferTextureBinding); + } + + internal bool TryGetOffset(StorageKind storageKind, int location, int component, out int offset) + { + return _offsets.TryGetValue(new IoDefinition(storageKind, IoVariable.UserDefined, location, component), out offset); + } + + internal bool TryGetOffset(StorageKind storageKind, IoVariable ioVariable, int location, int component, out int offset) + { + return _offsets.TryGetValue(new IoDefinition(storageKind, ioVariable, location, component), out offset); + } + + internal bool TryGetOffset(StorageKind storageKind, IoVariable ioVariable, int component, out int offset) + { + return _offsets.TryGetValue(new IoDefinition(storageKind, ioVariable, 0, component), out offset); + } + + internal bool TryGetOffset(StorageKind storageKind, IoVariable ioVariable, out int offset) + { + return _offsets.TryGetValue(new IoDefinition(storageKind, ioVariable, 0, 0), out offset); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/ShaderDefinitions.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/ShaderDefinitions.cs new file mode 100644 index 0000000..ed278d5 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/ShaderDefinitions.cs @@ -0,0 +1,355 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System; +using System.Collections.Generic; +using System.Numerics; + +namespace Ryujinx.Graphics.Shader.Translation +{ + class ShaderDefinitions + { + private readonly GpuGraphicsState _graphicsState; + + public ShaderStage Stage { get; } + + public int ComputeLocalSizeX { get; } + public int ComputeLocalSizeY { get; } + public int ComputeLocalSizeZ { get; } + + public bool TessCw => _graphicsState.TessCw; + public TessPatchType TessPatchType => _graphicsState.TessPatchType; + public TessSpacing TessSpacing => _graphicsState.TessSpacing; + + public bool AlphaToCoverageDitherEnable => _graphicsState.AlphaToCoverageEnable && _graphicsState.AlphaToCoverageDitherEnable; + public bool ViewportTransformDisable => _graphicsState.ViewportTransformDisable; + + public bool DepthMode => _graphicsState.DepthMode; + + public float PointSize => _graphicsState.PointSize; + + public AlphaTestOp AlphaTestCompare => _graphicsState.AlphaTestCompare; + public float AlphaTestReference => _graphicsState.AlphaTestReference; + + public bool GpPassthrough { get; } + public bool LastInVertexPipeline { get; set; } + + public int ThreadsPerInputPrimitive { get; private set; } + + public InputTopology InputTopology => _graphicsState.Topology; + public OutputTopology OutputTopology { get; } + + public int MaxOutputVertices { get; } + + public bool DualSourceBlend => _graphicsState.DualSourceBlendEnable; + public bool EarlyZForce => _graphicsState.EarlyZForce; + + public bool YNegateEnabled => _graphicsState.YNegateEnabled; + public bool OriginUpperLeft => _graphicsState.OriginUpperLeft; + + public bool HalvePrimitiveId => _graphicsState.HalvePrimitiveId; + + public ImapPixelType[] ImapTypes { get; } + public bool IaIndexing { get; private set; } + public bool OaIndexing { get; private set; } + + public int OmapTargets { get; } + public bool OmapSampleMask { get; } + public bool OmapDepth { get; } + + public bool SupportsScaledVertexFormats { get; } + + public bool TransformFeedbackEnabled { get; } + + private readonly TransformFeedbackOutput[] _transformFeedbackOutputs; + + readonly struct TransformFeedbackVariable : IEquatable + { + public IoVariable IoVariable { get; } + public int Location { get; } + public int Component { get; } + + public TransformFeedbackVariable(IoVariable ioVariable, int location = 0, int component = 0) + { + IoVariable = ioVariable; + Location = location; + Component = component; + } + + public override bool Equals(object other) + { + return other is TransformFeedbackVariable tfbVar && Equals(tfbVar); + } + + public bool Equals(TransformFeedbackVariable other) + { + return IoVariable == other.IoVariable && + Location == other.Location && + Component == other.Component; + } + + public override int GetHashCode() + { + return (int)IoVariable | (Location << 8) | (Component << 16); + } + + public override string ToString() + { + return $"{IoVariable}.{Location}.{Component}"; + } + } + + private readonly Dictionary _transformFeedbackDefinitions; + + public ShaderDefinitions(ShaderStage stage, ulong transformFeedbackVecMap, TransformFeedbackOutput[] transformFeedbackOutputs) + { + Stage = stage; + TransformFeedbackEnabled = transformFeedbackOutputs != null; + _transformFeedbackOutputs = transformFeedbackOutputs; + _transformFeedbackDefinitions = new(); + + PopulateTransformFeedbackDefinitions(transformFeedbackVecMap, transformFeedbackOutputs); + } + + public ShaderDefinitions( + ShaderStage stage, + int computeLocalSizeX, + int computeLocalSizeY, + int computeLocalSizeZ) + { + Stage = stage; + ComputeLocalSizeX = computeLocalSizeX; + ComputeLocalSizeY = computeLocalSizeY; + ComputeLocalSizeZ = computeLocalSizeZ; + } + + public ShaderDefinitions( + ShaderStage stage, + GpuGraphicsState graphicsState, + bool gpPassthrough, + int threadsPerInputPrimitive, + OutputTopology outputTopology, + int maxOutputVertices) + { + Stage = stage; + _graphicsState = graphicsState; + GpPassthrough = gpPassthrough; + ThreadsPerInputPrimitive = threadsPerInputPrimitive; + OutputTopology = outputTopology; + MaxOutputVertices = maxOutputVertices; + } + + public ShaderDefinitions( + ShaderStage stage, + GpuGraphicsState graphicsState, + bool gpPassthrough, + int threadsPerInputPrimitive, + OutputTopology outputTopology, + int maxOutputVertices, + ImapPixelType[] imapTypes, + int omapTargets, + bool omapSampleMask, + bool omapDepth, + bool supportsScaledVertexFormats, + ulong transformFeedbackVecMap, + TransformFeedbackOutput[] transformFeedbackOutputs) + { + Stage = stage; + _graphicsState = graphicsState; + GpPassthrough = gpPassthrough; + ThreadsPerInputPrimitive = threadsPerInputPrimitive; + OutputTopology = outputTopology; + MaxOutputVertices = gpPassthrough ? graphicsState.Topology.InputVertexCountNoAdjacency : maxOutputVertices; + ImapTypes = imapTypes; + OmapTargets = omapTargets; + OmapSampleMask = omapSampleMask; + OmapDepth = omapDepth; + LastInVertexPipeline = stage < ShaderStage.Fragment; + SupportsScaledVertexFormats = supportsScaledVertexFormats; + TransformFeedbackEnabled = transformFeedbackOutputs != null; + _transformFeedbackOutputs = transformFeedbackOutputs; + _transformFeedbackDefinitions = new(); + + PopulateTransformFeedbackDefinitions(transformFeedbackVecMap, transformFeedbackOutputs); + } + + private void PopulateTransformFeedbackDefinitions(ulong transformFeedbackVecMap, TransformFeedbackOutput[] transformFeedbackOutputs) + { + while (transformFeedbackVecMap != 0) + { + int vecIndex = BitOperations.TrailingZeroCount(transformFeedbackVecMap); + + for (int subIndex = 0; subIndex < 4; subIndex++) + { + int wordOffset = vecIndex * 4 + subIndex; + int byteOffset = wordOffset * 4; + + if (transformFeedbackOutputs[wordOffset].Valid) + { + IoVariable ioVariable = Instructions.AttributeMap.GetIoVariable(this, byteOffset, out int location); + int component = 0; + + if (HasPerLocationInputOrOutputComponent(ioVariable, location, subIndex, isOutput: true)) + { + component = subIndex; + } + + TransformFeedbackVariable transformFeedbackVariable = new(ioVariable, location, component); + _transformFeedbackDefinitions.TryAdd(transformFeedbackVariable, transformFeedbackOutputs[wordOffset]); + } + } + + transformFeedbackVecMap &= ~(1UL << vecIndex); + } + } + + public void EnableInputIndexing() + { + IaIndexing = true; + } + + public void EnableOutputIndexing() + { + OaIndexing = true; + } + + public bool TryGetTransformFeedbackOutput(IoVariable ioVariable, int location, int component, out TransformFeedbackOutput transformFeedbackOutput) + { + if (!HasTransformFeedbackOutputs()) + { + transformFeedbackOutput = default; + return false; + } + + TransformFeedbackVariable transformFeedbackVariable = new(ioVariable, location, component); + return _transformFeedbackDefinitions.TryGetValue(transformFeedbackVariable, out transformFeedbackOutput); + } + + private bool HasTransformFeedbackOutputs() + { + return TransformFeedbackEnabled && (LastInVertexPipeline || Stage == ShaderStage.Fragment); + } + + public bool HasTransformFeedbackOutputs(bool isOutput) + { + return TransformFeedbackEnabled && ((isOutput && LastInVertexPipeline) || (!isOutput && Stage == ShaderStage.Fragment)); + } + + public bool HasPerLocationInputOrOutput(IoVariable ioVariable, bool isOutput) + { + if (ioVariable == IoVariable.UserDefined) + { + return (!isOutput && !IaIndexing) || (isOutput && !OaIndexing); + } + + return ioVariable == IoVariable.FragmentOutputColor; + } + + public bool HasPerLocationInputOrOutputComponent(IoVariable ioVariable, int location, int component, bool isOutput) + { + if (ioVariable != IoVariable.UserDefined || !HasTransformFeedbackOutputs(isOutput)) + { + return false; + } + + return GetTransformFeedbackOutputComponents(location, component) == 1; + } + + public TransformFeedbackOutput GetTransformFeedbackOutput(int wordOffset) + { + return _transformFeedbackOutputs[wordOffset]; + } + + public TransformFeedbackOutput GetTransformFeedbackOutput(int location, int component) + { + return GetTransformFeedbackOutput((AttributeConsts.UserAttributeBase / 4) + location * 4 + component); + } + + public int GetTransformFeedbackOutputComponents(int location, int component) + { + int baseIndex = (AttributeConsts.UserAttributeBase / 4) + location * 4; + int index = baseIndex + component; + int count = 1; + + for (; count < 4; count++) + { + ref TransformFeedbackOutput prev = ref _transformFeedbackOutputs[baseIndex + count - 1]; + ref TransformFeedbackOutput curr = ref _transformFeedbackOutputs[baseIndex + count]; + + int prevOffset = prev.Offset; + int currOffset = curr.Offset; + + if (!prev.Valid || !curr.Valid || prevOffset + 4 != currOffset) + { + break; + } + } + + if (baseIndex + count <= index) + { + return 1; + } + + return count; + } + + public AggregateType GetFragmentOutputColorType(int location) + { + return AggregateType.Vector4 | _graphicsState.FragmentOutputTypes[location].Aggregate; + } + + public AggregateType GetUserDefinedType(int location, bool isOutput) + { + if ((!isOutput && IaIndexing) || (isOutput && OaIndexing)) + { + return AggregateType.Array | AggregateType.Vector4 | AggregateType.FP32; + } + + AggregateType type = AggregateType.Vector4; + + if (Stage == ShaderStage.Vertex && !isOutput) + { + type |= _graphicsState.AttributeTypes[location].AsAggregate(SupportsScaledVertexFormats); + } + else + { + type |= AggregateType.FP32; + } + + return type; + } + + public AttributeType GetAttributeType(int location) + { + return _graphicsState.AttributeTypes[location]; + } + + public bool IsAttributeSint(int location) + { + return (_graphicsState.AttributeTypes[location] & ~AttributeType.AnyPacked) == AttributeType.Sint; + } + + public bool IsAttributePacked(int location) + { + return _graphicsState.AttributeTypes[location].HasFlag(AttributeType.Packed); + } + + public bool IsAttributePackedRgb10A2Signed(int location) + { + return _graphicsState.AttributeTypes[location].HasFlag(AttributeType.PackedRgb10A2Signed); + } + + public int GetGeometryOutputIndexBufferStridePerInstance() + { + return MaxOutputVertices + OutputTopology switch + { + OutputTopology.LineStrip => MaxOutputVertices / 2, + OutputTopology.TriangleStrip => MaxOutputVertices / 3, + _ => MaxOutputVertices, + }; + } + + public int GetGeometryOutputIndexBufferStride() + { + return GetGeometryOutputIndexBufferStridePerInstance() * ThreadsPerInputPrimitive; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/ShaderHeader.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/ShaderHeader.cs new file mode 100644 index 0000000..85579dc --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/ShaderHeader.cs @@ -0,0 +1,170 @@ +using Ryujinx.Common.Utilities; +using System; +using System.Runtime.InteropServices; + +namespace Ryujinx.Graphics.Shader.Translation +{ + enum PixelImap + { + Unused = 0, + Constant = 1, + Perspective = 2, + ScreenLinear = 3, + } + + readonly struct ImapPixelType + { + public PixelImap X { get; } + public PixelImap Y { get; } + public PixelImap Z { get; } + public PixelImap W { get; } + + public ImapPixelType(PixelImap x, PixelImap y, PixelImap z, PixelImap w) + { + X = x; + Y = y; + Z = z; + W = w; + } + + public PixelImap GetFirstUsedType() + { + if (X != PixelImap.Unused) + { + return X; + } + + if (Y != PixelImap.Unused) + { + return Y; + } + + if (Z != PixelImap.Unused) + { + return Z; + } + + return W; + } + } + + class ShaderHeader + { + public int SphType { get; } + public int Version { get; } + + public ShaderStage Stage { get; } + + public bool MrtEnable { get; } + + public bool KillsPixels { get; } + + public bool DoesGlobalStore { get; } + + public int SassVersion { get; } + + public bool GpPassthrough { get; } + + public bool DoesLoadOrStore { get; } + public bool DoesFp64 { get; } + + public int StreamOutMask { get; } + + public int ShaderLocalMemoryLowSize { get; } + + public int PerPatchAttributeCount { get; } + + public int ShaderLocalMemoryHighSize { get; } + + public int ThreadsPerInputPrimitive { get; } + + public int ShaderLocalMemoryCrsSize { get; } + + public OutputTopology OutputTopology { get; } + + public int MaxOutputVertexCount { get; } + + public int StoreReqStart { get; } + public int StoreReqEnd { get; } + + public ImapPixelType[] ImapTypes { get; } + + public int OmapTargets { get; } + public bool OmapSampleMask { get; } + public bool OmapDepth { get; } + + public ShaderHeader(IGpuAccessor gpuAccessor, ulong address) + { + ReadOnlySpan header = MemoryMarshal.Cast(gpuAccessor.GetCode(address, 0x50)); + + int commonWord0 = header[0]; + int commonWord1 = header[1]; + int commonWord2 = header[2]; + int commonWord3 = header[3]; + int commonWord4 = header[4]; + + SphType = commonWord0.Extract(0, 5); + Version = commonWord0.Extract(5, 5); + + Stage = (ShaderStage)commonWord0.Extract(10, 4); + + // Invalid. + if (Stage == ShaderStage.Compute) + { + Stage = ShaderStage.Vertex; + } + + MrtEnable = commonWord0.Extract(14); + + KillsPixels = commonWord0.Extract(15); + + DoesGlobalStore = commonWord0.Extract(16); + + SassVersion = commonWord0.Extract(17, 4); + + GpPassthrough = commonWord0.Extract(24); + + DoesLoadOrStore = commonWord0.Extract(26); + DoesFp64 = commonWord0.Extract(27); + + StreamOutMask = commonWord0.Extract(28, 4); + + ShaderLocalMemoryLowSize = commonWord1.Extract(0, 24); + + PerPatchAttributeCount = commonWord1.Extract(24, 8); + + ShaderLocalMemoryHighSize = commonWord2.Extract(0, 24); + + ThreadsPerInputPrimitive = commonWord2.Extract(24, 8); + + ShaderLocalMemoryCrsSize = commonWord3.Extract(0, 24); + + OutputTopology = (OutputTopology)commonWord3.Extract(24, 4); + + MaxOutputVertexCount = commonWord4.Extract(0, 12); + + StoreReqStart = commonWord4.Extract(12, 8); + StoreReqEnd = commonWord4.Extract(24, 8); + + ImapTypes = new ImapPixelType[32]; + + for (int i = 0; i < 32; i++) + { + byte imap = (byte)(header[6 + (i >> 2)] >> ((i & 3) * 8)); + + ImapTypes[i] = new ImapPixelType( + (PixelImap)((imap >> 0) & 3), + (PixelImap)((imap >> 2) & 3), + (PixelImap)((imap >> 4) & 3), + (PixelImap)((imap >> 6) & 3)); + } + + int type2OmapTarget = header[18]; + int type2Omap = header[19]; + + OmapTargets = type2OmapTarget; + OmapSampleMask = type2Omap.Extract(0); + OmapDepth = type2Omap.Extract(1); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Ssa.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Ssa.cs new file mode 100644 index 0000000..17e6f76 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Ssa.cs @@ -0,0 +1,375 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System.Collections.Generic; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Translation +{ + static class Ssa + { + private const int GprsAndPredsCount = RegisterConsts.GprsCount + RegisterConsts.PredsCount; + + private class DefMap + { + private readonly Dictionary _map; + + private readonly long[] _phiMasks; + + public DefMap() + { + _map = new Dictionary(); + + _phiMasks = new long[(RegisterConsts.TotalCount + 63) / 64]; + } + + public bool TryAddOperand(Register reg, Operand operand) + { + return _map.TryAdd(reg, operand); + } + + public bool TryGetOperand(Register reg, out Operand operand) + { + return _map.TryGetValue(reg, out operand); + } + + public bool AddPhi(Register reg) + { + int key = GetKeyFromRegister(reg); + + int index = key / 64; + int bit = key & 63; + + long mask = 1L << bit; + + if ((_phiMasks[index] & mask) != 0) + { + return false; + } + + _phiMasks[index] |= mask; + + return true; + } + + public bool HasPhi(Register reg) + { + int key = GetKeyFromRegister(reg); + + int index = key / 64; + int bit = key & 63; + + return (_phiMasks[index] & (1L << bit)) != 0; + } + } + + private class LocalDefMap + { + private readonly Operand[] _map; + private readonly int[] _uses; + public int UseCount { get; private set; } + + public LocalDefMap() + { + _map = new Operand[RegisterConsts.TotalCount]; + _uses = new int[RegisterConsts.TotalCount]; + } + + public Operand Get(int key) + { + return _map[key]; + } + + public void Add(int key, Operand operand) + { + if (_map[key] == null) + { + _uses[UseCount++] = key; + } + + _map[key] = operand; + } + + public Operand GetUse(int index, out int key) + { + key = _uses[index]; + + return _map[key]; + } + + public void Clear() + { + for (int i = 0; i < UseCount; i++) + { + _map[_uses[i]] = null; + } + + UseCount = 0; + } + } + + private readonly struct Definition + { + public BasicBlock Block { get; } + public Operand Local { get; } + + public Definition(BasicBlock block, Operand local) + { + Block = block; + Local = local; + } + } + + public static void Rename(BasicBlock[] blocks) + { + DefMap[] globalDefs = new DefMap[blocks.Length]; + LocalDefMap localDefs = new(); + + for (int blkIndex = 0; blkIndex < blocks.Length; blkIndex++) + { + globalDefs[blkIndex] = new DefMap(); + } + + Queue dfPhiBlocks = new(); + + // First pass, get all defs and locals uses. + for (int blkIndex = 0; blkIndex < blocks.Length; blkIndex++) + { + Operand RenameLocal(Operand operand) + { + if (operand != null && operand.Type == OperandType.Register) + { + Operand local = localDefs.Get(GetKeyFromRegister(operand.GetRegister())); + + operand = local ?? operand; + } + + return operand; + } + + BasicBlock block = blocks[blkIndex]; + + LinkedListNode node = block.Operations.First; + + while (node != null) + { + if (node.Value is Operation operation) + { + for (int index = 0; index < operation.SourcesCount; index++) + { + operation.SetSource(index, RenameLocal(operation.GetSource(index))); + } + + for (int index = 0; index < operation.DestsCount; index++) + { + Operand dest = operation.GetDest(index); + + if (dest != null && dest.Type == OperandType.Register) + { + Operand local = Local(); + + localDefs.Add(GetKeyFromRegister(dest.GetRegister()), local); + + operation.SetDest(index, local); + } + } + } + + node = node.Next; + } + + int localUses = localDefs.UseCount; + for (int index = 0; index < localUses; index++) + { + Operand local = localDefs.GetUse(index, out int key); + + Register reg = GetRegisterFromKey(key); + + globalDefs[block.Index].TryAddOperand(reg, local); + + dfPhiBlocks.Enqueue(block); + + while (dfPhiBlocks.TryDequeue(out BasicBlock dfPhiBlock)) + { + foreach (BasicBlock domFrontier in dfPhiBlock.DominanceFrontiers) + { + if (globalDefs[domFrontier.Index].AddPhi(reg)) + { + dfPhiBlocks.Enqueue(domFrontier); + } + } + } + } + + localDefs.Clear(); + } + + // Second pass, rename variables with definitions on different blocks. + for (int blkIndex = 0; blkIndex < blocks.Length; blkIndex++) + { + BasicBlock block = blocks[blkIndex]; + + Operand RenameGlobal(Operand operand) + { + if (operand != null && operand.Type == OperandType.Register) + { + int key = GetKeyFromRegister(operand.GetRegister()); + + Operand local = localDefs.Get(key); + + if (local != null) + { + return local; + } + + operand = FindDefinitionForCurr(globalDefs, block, operand.GetRegister()); + + localDefs.Add(key, operand); + } + + return operand; + } + + for (LinkedListNode node = block.Operations.First; node != null; node = node.Next) + { + if (node.Value is Operation operation) + { + for (int index = 0; index < operation.SourcesCount; index++) + { + operation.SetSource(index, RenameGlobal(operation.GetSource(index))); + } + } + } + + if (blkIndex < blocks.Length - 1) + { + localDefs.Clear(); + } + } + } + + private static Operand FindDefinitionForCurr(DefMap[] globalDefs, BasicBlock current, Register reg) + { + if (globalDefs[current.Index].HasPhi(reg)) + { + return InsertPhi(globalDefs, current, reg); + } + + if (current != current.ImmediateDominator) + { + return FindDefinition(globalDefs, current.ImmediateDominator, reg).Local; + } + + return Undef(); + } + + private static Definition FindDefinition(DefMap[] globalDefs, BasicBlock current, Register reg) + { + foreach (BasicBlock block in SelfAndImmediateDominators(current)) + { + DefMap defMap = globalDefs[block.Index]; + + if (defMap.TryGetOperand(reg, out Operand lastDef)) + { + return new Definition(block, lastDef); + } + + if (defMap.HasPhi(reg)) + { + return new Definition(block, InsertPhi(globalDefs, block, reg)); + } + } + + return new Definition(current, Undef()); + } + + private static IEnumerable SelfAndImmediateDominators(BasicBlock block) + { + while (block != block.ImmediateDominator) + { + yield return block; + + block = block.ImmediateDominator; + } + + yield return block; + } + + private static Operand InsertPhi(DefMap[] globalDefs, BasicBlock block, Register reg) + { + // This block has a Phi that has not been materialized yet, but that + // would define a new version of the variable we're looking for. We need + // to materialize the Phi, add all the block/operand pairs into the Phi, and + // then use the definition from that Phi. + Operand local = Local(); + + PhiNode phi = new(local); + + AddPhi(block, phi); + + globalDefs[block.Index].TryAddOperand(reg, local); + + foreach (BasicBlock predecessor in block.Predecessors) + { + Definition def = FindDefinition(globalDefs, predecessor, reg); + + phi.AddSource(def.Block, def.Local); + } + + return local; + } + + private static void AddPhi(BasicBlock block, PhiNode phi) + { + LinkedListNode node = block.Operations.First; + + if (node != null) + { + while (node.Next?.Value is PhiNode) + { + node = node.Next; + } + } + + if (node?.Value is PhiNode) + { + block.Operations.AddAfter(node, phi); + } + else + { + block.Operations.AddFirst(phi); + } + } + + private static int GetKeyFromRegister(Register reg) + { + if (reg.Type == RegisterType.Gpr) + { + return reg.Index; + } + else if (reg.Type == RegisterType.Predicate) + { + return RegisterConsts.GprsCount + reg.Index; + } + else /* if (reg.Type == RegisterType.Flag) */ + { + return GprsAndPredsCount + reg.Index; + } + } + + private static Register GetRegisterFromKey(int key) + { + if (key < RegisterConsts.GprsCount) + { + return new Register(key, RegisterType.Gpr); + } + else if (key < GprsAndPredsCount) + { + return new Register(key - RegisterConsts.GprsCount, RegisterType.Predicate); + } + else /* if (key < RegisterConsts.TotalCount) */ + { + return new Register(key - GprsAndPredsCount, RegisterType.Flag); + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/TargetApi.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/TargetApi.cs new file mode 100644 index 0000000..1b130b8 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/TargetApi.cs @@ -0,0 +1,8 @@ +namespace Ryujinx.Graphics.Shader.Translation +{ + public enum TargetApi + { + OpenGL, + Vulkan, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/TargetLanguage.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/TargetLanguage.cs new file mode 100644 index 0000000..30ba02d --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/TargetLanguage.cs @@ -0,0 +1,9 @@ +namespace Ryujinx.Graphics.Shader.Translation +{ + public enum TargetLanguage + { + Glsl, + Spirv, + Arb, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/TransformContext.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/TransformContext.cs new file mode 100644 index 0000000..461edaa --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/TransformContext.cs @@ -0,0 +1,39 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; + +namespace Ryujinx.Graphics.Shader.Translation +{ + readonly ref struct TransformContext + { + public readonly HelperFunctionManager Hfm; + public readonly BasicBlock[] Blocks; + public readonly ShaderDefinitions Definitions; + public readonly ResourceManager ResourceManager; + public readonly IGpuAccessor GpuAccessor; + public readonly TargetApi TargetApi; + public readonly TargetLanguage TargetLanguage; + public readonly ShaderStage Stage; + public readonly ref FeatureFlags UsedFeatures; + + public TransformContext( + HelperFunctionManager hfm, + BasicBlock[] blocks, + ShaderDefinitions definitions, + ResourceManager resourceManager, + IGpuAccessor gpuAccessor, + TargetApi targetApi, + TargetLanguage targetLanguage, + ShaderStage stage, + ref FeatureFlags usedFeatures) + { + Hfm = hfm; + Blocks = blocks; + Definitions = definitions; + ResourceManager = resourceManager; + GpuAccessor = gpuAccessor; + TargetApi = targetApi; + TargetLanguage = targetLanguage; + Stage = stage; + UsedFeatures = ref usedFeatures; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/TransformFeedbackOutput.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/TransformFeedbackOutput.cs new file mode 100644 index 0000000..46622ec --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/TransformFeedbackOutput.cs @@ -0,0 +1,18 @@ +namespace Ryujinx.Graphics.Shader.Translation +{ + readonly struct TransformFeedbackOutput + { + public readonly bool Valid; + public readonly int Buffer; + public readonly int Offset; + public readonly int Stride; + + public TransformFeedbackOutput(int buffer, int offset, int stride) + { + Valid = true; + Buffer = buffer; + Offset = offset; + Stride = stride; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/DrawParametersReplace.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/DrawParametersReplace.cs new file mode 100644 index 0000000..41ab70a --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/DrawParametersReplace.cs @@ -0,0 +1,93 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System.Collections.Generic; + +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Translation.Transforms +{ + class DrawParametersReplace : ITransformPass + { + public static bool IsEnabled(IGpuAccessor gpuAccessor, ShaderStage stage, TargetLanguage targetLanguage, FeatureFlags usedFeatures) + { + return stage == ShaderStage.Vertex; + } + + public static LinkedListNode RunPass(TransformContext context, LinkedListNode node) + { + Operation operation = (Operation)node.Value; + + if (context.GpuAccessor.QueryHasConstantBufferDrawParameters()) + { + if (ReplaceConstantBufferWithDrawParameters(node, operation)) + { + context.UsedFeatures |= FeatureFlags.DrawParameters; + } + } + else if (HasConstantBufferDrawParameters(operation)) + { + context.UsedFeatures |= FeatureFlags.DrawParameters; + } + + return node; + } + + private static bool ReplaceConstantBufferWithDrawParameters(LinkedListNode node, Operation operation) + { + Operand GenerateLoad(IoVariable ioVariable) + { + Operand value = Local(); + node.List.AddBefore(node, new Operation(Instruction.Load, StorageKind.Input, value, Const((int)ioVariable))); + return value; + } + + bool modified = false; + + for (int srcIndex = 0; srcIndex < operation.SourcesCount; srcIndex++) + { + Operand src = operation.GetSource(srcIndex); + + if (src.Type == OperandType.ConstantBuffer && src.GetCbufSlot() == 0) + { + switch (src.GetCbufOffset()) + { + case Constants.NvnBaseVertexByteOffset / 4: + operation.SetSource(srcIndex, GenerateLoad(IoVariable.BaseVertex)); + modified = true; + break; + case Constants.NvnBaseInstanceByteOffset / 4: + operation.SetSource(srcIndex, GenerateLoad(IoVariable.BaseInstance)); + modified = true; + break; + case Constants.NvnDrawIndexByteOffset / 4: + operation.SetSource(srcIndex, GenerateLoad(IoVariable.DrawIndex)); + modified = true; + break; + } + } + } + + return modified; + } + + private static bool HasConstantBufferDrawParameters(Operation operation) + { + for (int srcIndex = 0; srcIndex < operation.SourcesCount; srcIndex++) + { + Operand src = operation.GetSource(srcIndex); + + if (src.Type == OperandType.ConstantBuffer && src.GetCbufSlot() == 0) + { + switch (src.GetCbufOffset()) + { + case Constants.NvnBaseVertexByteOffset / 4: + case Constants.NvnBaseInstanceByteOffset / 4: + case Constants.NvnDrawIndexByteOffset / 4: + return true; + } + } + } + + return false; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/ForcePreciseEnable.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/ForcePreciseEnable.cs new file mode 100644 index 0000000..b0d6ea8 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/ForcePreciseEnable.cs @@ -0,0 +1,36 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System.Collections.Generic; + +namespace Ryujinx.Graphics.Shader.Translation.Transforms +{ + class ForcePreciseEnable : ITransformPass + { + public static bool IsEnabled(IGpuAccessor gpuAccessor, ShaderStage stage, TargetLanguage targetLanguage, FeatureFlags usedFeatures) + { + return stage == ShaderStage.Fragment && gpuAccessor.QueryHostReducedPrecision(); + } + + public static LinkedListNode RunPass(TransformContext context, LinkedListNode node) + { + // There are some cases where a small bias is added to values to prevent division by zero. + // When operating with reduced precision, it is possible for this bias to get rounded to 0 + // and cause a division by zero. + // To prevent that, we force those operations to be precise even if the host wants + // imprecise operations for performance. + + Operation operation = (Operation)node.Value; + + if (operation.Inst == (Instruction.FP32 | Instruction.Divide) && + operation.GetSource(0).Type == OperandType.Constant && + operation.GetSource(0).AsFloat() == 1f && + operation.GetSource(1).AsgOp is Operation addOp && + addOp.Inst == (Instruction.FP32 | Instruction.Add) && + addOp.GetSource(1).Type == OperandType.Constant) + { + addOp.ForcePrecise = true; + } + + return node; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/GeometryToCompute.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/GeometryToCompute.cs new file mode 100644 index 0000000..c07e004 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/GeometryToCompute.cs @@ -0,0 +1,380 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation.Optimizations; +using System.Collections.Generic; + +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Translation.Transforms +{ + class GeometryToCompute : ITransformPass + { + public static bool IsEnabled(IGpuAccessor gpuAccessor, ShaderStage stage, TargetLanguage targetLanguage, FeatureFlags usedFeatures) + { + return usedFeatures.HasFlag(FeatureFlags.VtgAsCompute); + } + + public static LinkedListNode RunPass(TransformContext context, LinkedListNode node) + { + if (context.Definitions.Stage != ShaderStage.Geometry) + { + return node; + } + + Operation operation = (Operation)node.Value; + + LinkedListNode newNode = node; + + switch (operation.Inst) + { + case Instruction.EmitVertex: + newNode = GenerateEmitVertex(context.Definitions, context.ResourceManager, node); + break; + case Instruction.EndPrimitive: + newNode = GenerateEndPrimitive(context.Definitions, context.ResourceManager, node); + break; + case Instruction.Load: + if (operation.StorageKind == StorageKind.Input) + { + IoVariable ioVariable = (IoVariable)operation.GetSource(0).Value; + + if (TryGetOffset(context.ResourceManager, operation, StorageKind.Input, out int inputOffset)) + { + Operand primVertex = ioVariable == IoVariable.UserDefined + ? operation.GetSource(2) + : operation.GetSource(1); + + Operand vertexElemOffset = GenerateVertexOffset(context.ResourceManager, node, inputOffset, primVertex); + + newNode = node.List.AddBefore(node, new Operation( + Instruction.Load, + StorageKind.StorageBuffer, + operation.Dest, + new[] { Const(context.ResourceManager.Reservations.VertexOutputStorageBufferBinding), Const(0), vertexElemOffset })); + } + else + { + switch (ioVariable) + { + case IoVariable.InvocationId: + newNode = GenerateInvocationId(node, operation.Dest); + break; + case IoVariable.PrimitiveId: + newNode = GeneratePrimitiveId(context.ResourceManager, node, operation.Dest); + break; + case IoVariable.GlobalId: + case IoVariable.SubgroupEqMask: + case IoVariable.SubgroupGeMask: + case IoVariable.SubgroupGtMask: + case IoVariable.SubgroupLaneId: + case IoVariable.SubgroupLeMask: + case IoVariable.SubgroupLtMask: + // Those are valid or expected for geometry shaders. + break; + default: + context.GpuAccessor.Log($"Invalid input \"{ioVariable}\"."); + break; + } + } + } + else if (operation.StorageKind == StorageKind.Output) + { + if (TryGetOffset(context.ResourceManager, operation, StorageKind.Output, out int outputOffset)) + { + newNode = node.List.AddBefore(node, new Operation( + Instruction.Load, + StorageKind.LocalMemory, + operation.Dest, + new[] { Const(context.ResourceManager.LocalVertexDataMemoryId), Const(outputOffset) })); + } + else + { + context.GpuAccessor.Log($"Invalid output \"{(IoVariable)operation.GetSource(0).Value}\"."); + } + } + + break; + case Instruction.Store: + if (operation.StorageKind == StorageKind.Output) + { + if (TryGetOffset(context.ResourceManager, operation, StorageKind.Output, out int outputOffset)) + { + Operand value = operation.GetSource(operation.SourcesCount - 1); + + newNode = node.List.AddBefore(node, new Operation( + Instruction.Store, + StorageKind.LocalMemory, + (Operand)null, + new[] { Const(context.ResourceManager.LocalVertexDataMemoryId), Const(outputOffset), value })); + } + else + { + context.GpuAccessor.Log($"Invalid output \"{(IoVariable)operation.GetSource(0).Value}\"."); + } + } + + break; + } + + if (newNode != node) + { + Utils.DeleteNode(node, operation); + } + + return newNode; + } + + private static LinkedListNode GenerateEmitVertex(ShaderDefinitions definitions, ResourceManager resourceManager, LinkedListNode node) + { + int vbOutputBinding = resourceManager.Reservations.GeometryVertexOutputStorageBufferBinding; + int ibOutputBinding = resourceManager.Reservations.GeometryIndexOutputStorageBufferBinding; + int stride = resourceManager.Reservations.OutputSizePerInvocation; + + Operand outputPrimVertex = IncrementLocalMemory(node, resourceManager.LocalGeometryOutputVertexCountMemoryId); + Operand baseVertexOffset = GenerateBaseOffset( + resourceManager, + node, + definitions.MaxOutputVertices * definitions.ThreadsPerInputPrimitive, + definitions.ThreadsPerInputPrimitive); + Operand outputBaseVertex = Local(); + node.List.AddBefore(node, new Operation(Instruction.Add, outputBaseVertex, new[] { baseVertexOffset, outputPrimVertex })); + + Operand outputPrimIndex = IncrementLocalMemory(node, resourceManager.LocalGeometryOutputIndexCountMemoryId); + Operand baseIndexOffset = GenerateBaseOffset( + resourceManager, + node, + definitions.GetGeometryOutputIndexBufferStride(), + definitions.ThreadsPerInputPrimitive); + Operand outputBaseIndex = Local(); + node.List.AddBefore(node, new Operation(Instruction.Add, outputBaseIndex, new[] { baseIndexOffset, outputPrimIndex })); + + node.List.AddBefore(node, new Operation( + Instruction.Store, + StorageKind.StorageBuffer, + null, + new[] { Const(ibOutputBinding), Const(0), outputBaseIndex, outputBaseVertex })); + + Operand baseOffset = Local(); + node.List.AddBefore(node, new Operation(Instruction.Multiply, baseOffset, new[] { outputBaseVertex, Const(stride) })); + + LinkedListNode newNode = node; + + for (int offset = 0; offset < stride; offset++) + { + Operand vertexOffset; + + if (offset > 0) + { + vertexOffset = Local(); + node.List.AddBefore(node, new Operation(Instruction.Add, vertexOffset, new[] { baseOffset, Const(offset) })); + } + else + { + vertexOffset = baseOffset; + } + + Operand value = Local(); + node.List.AddBefore(node, new Operation( + Instruction.Load, + StorageKind.LocalMemory, + value, + new[] { Const(resourceManager.LocalVertexDataMemoryId), Const(offset) })); + + newNode = node.List.AddBefore(node, new Operation( + Instruction.Store, + StorageKind.StorageBuffer, + null, + new[] { Const(vbOutputBinding), Const(0), vertexOffset, value })); + } + + return newNode; + } + + private static LinkedListNode GenerateEndPrimitive(ShaderDefinitions definitions, ResourceManager resourceManager, LinkedListNode node) + { + int ibOutputBinding = resourceManager.Reservations.GeometryIndexOutputStorageBufferBinding; + + Operand outputPrimIndex = IncrementLocalMemory(node, resourceManager.LocalGeometryOutputIndexCountMemoryId); + Operand baseIndexOffset = GenerateBaseOffset( + resourceManager, + node, + definitions.GetGeometryOutputIndexBufferStride(), + definitions.ThreadsPerInputPrimitive); + Operand outputBaseIndex = Local(); + node.List.AddBefore(node, new Operation(Instruction.Add, outputBaseIndex, new[] { baseIndexOffset, outputPrimIndex })); + + return node.List.AddBefore(node, new Operation( + Instruction.Store, + StorageKind.StorageBuffer, + null, + new[] { Const(ibOutputBinding), Const(0), outputBaseIndex, Const(-1) })); + } + + private static Operand GenerateBaseOffset(ResourceManager resourceManager, LinkedListNode node, int stride, int threadsPerInputPrimitive) + { + Operand primitiveId = Local(); + GeneratePrimitiveId(resourceManager, node, primitiveId); + + Operand baseOffset = Local(); + node.List.AddBefore(node, new Operation(Instruction.Multiply, baseOffset, new[] { primitiveId, Const(stride) })); + + Operand invocationId = Local(); + GenerateInvocationId(node, invocationId); + + Operand invocationOffset = Local(); + node.List.AddBefore(node, new Operation(Instruction.Multiply, invocationOffset, new[] { invocationId, Const(stride / threadsPerInputPrimitive) })); + + Operand combinedOffset = Local(); + node.List.AddBefore(node, new Operation(Instruction.Add, combinedOffset, new[] { baseOffset, invocationOffset })); + + return combinedOffset; + } + + private static Operand IncrementLocalMemory(LinkedListNode node, int memoryId) + { + Operand oldValue = Local(); + node.List.AddBefore(node, new Operation( + Instruction.Load, + StorageKind.LocalMemory, + oldValue, + new[] { Const(memoryId) })); + + Operand newValue = Local(); + node.List.AddBefore(node, new Operation(Instruction.Add, newValue, new[] { oldValue, Const(1) })); + + node.List.AddBefore(node, new Operation(Instruction.Store, StorageKind.LocalMemory, null, new[] { Const(memoryId), newValue })); + + return oldValue; + } + + private static Operand GenerateVertexOffset( + ResourceManager resourceManager, + LinkedListNode node, + int elementOffset, + Operand primVertex) + { + int vertexInfoCbBinding = resourceManager.Reservations.VertexInfoConstantBufferBinding; + + Operand vertexCount = Local(); + node.List.AddBefore(node, new Operation( + Instruction.Load, + StorageKind.ConstantBuffer, + vertexCount, + new[] { Const(vertexInfoCbBinding), Const((int)VertexInfoBufferField.VertexCounts), Const(0) })); + + Operand primInputVertex = Local(); + node.List.AddBefore(node, new Operation( + Instruction.Load, + StorageKind.LocalMemory, + primInputVertex, + new[] { Const(resourceManager.LocalTopologyRemapMemoryId), primVertex })); + + Operand instanceIndex = Local(); + node.List.AddBefore(node, new Operation( + Instruction.Load, + StorageKind.Input, + instanceIndex, + new[] { Const((int)IoVariable.GlobalId), Const(1) })); + + Operand baseVertex = Local(); + node.List.AddBefore(node, new Operation(Instruction.Multiply, baseVertex, new[] { instanceIndex, vertexCount })); + + Operand vertexIndex = Local(); + node.List.AddBefore(node, new Operation(Instruction.Add, vertexIndex, new[] { baseVertex, primInputVertex })); + + Operand vertexBaseOffset = Local(); + node.List.AddBefore(node, new Operation( + Instruction.Multiply, + vertexBaseOffset, + new[] { vertexIndex, Const(resourceManager.Reservations.InputSizePerInvocation) })); + + Operand vertexElemOffset; + + if (elementOffset != 0) + { + vertexElemOffset = Local(); + + node.List.AddBefore(node, new Operation(Instruction.Add, vertexElemOffset, new[] { vertexBaseOffset, Const(elementOffset) })); + } + else + { + vertexElemOffset = vertexBaseOffset; + } + + return vertexElemOffset; + } + + private static LinkedListNode GeneratePrimitiveId(ResourceManager resourceManager, LinkedListNode node, Operand dest) + { + int vertexInfoCbBinding = resourceManager.Reservations.VertexInfoConstantBufferBinding; + + Operand vertexCount = Local(); + node.List.AddBefore(node, new Operation( + Instruction.Load, + StorageKind.ConstantBuffer, + vertexCount, + new[] { Const(vertexInfoCbBinding), Const((int)VertexInfoBufferField.VertexCounts), Const(0) })); + + Operand vertexIndex = Local(); + node.List.AddBefore(node, new Operation( + Instruction.Load, + StorageKind.Input, + vertexIndex, + new[] { Const((int)IoVariable.GlobalId), Const(0) })); + + Operand instanceIndex = Local(); + node.List.AddBefore(node, new Operation( + Instruction.Load, + StorageKind.Input, + instanceIndex, + new[] { Const((int)IoVariable.GlobalId), Const(1) })); + + Operand baseVertex = Local(); + node.List.AddBefore(node, new Operation(Instruction.Multiply, baseVertex, new[] { instanceIndex, vertexCount })); + + return node.List.AddBefore(node, new Operation(Instruction.Add, dest, new[] { baseVertex, vertexIndex })); + } + + private static LinkedListNode GenerateInvocationId(LinkedListNode node, Operand dest) + { + return node.List.AddBefore(node, new Operation( + Instruction.Load, + StorageKind.Input, + dest, + new[] { Const((int)IoVariable.GlobalId), Const(2) })); + } + + private static bool TryGetOffset(ResourceManager resourceManager, Operation operation, StorageKind storageKind, out int outputOffset) + { + bool isStore = operation.Inst == Instruction.Store; + + IoVariable ioVariable = (IoVariable)operation.GetSource(0).Value; + + bool isValidOutput; + + if (ioVariable == IoVariable.UserDefined) + { + int lastIndex = operation.SourcesCount - (isStore ? 2 : 1); + + int location = operation.GetSource(1).Value; + int component = operation.GetSource(lastIndex).Value; + + isValidOutput = resourceManager.Reservations.TryGetOffset(storageKind, location, component, out outputOffset); + } + else + { + if (ResourceReservations.IsVectorOrArrayVariable(ioVariable)) + { + int component = operation.GetSource(operation.SourcesCount - (isStore ? 2 : 1)).Value; + + isValidOutput = resourceManager.Reservations.TryGetOffset(storageKind, ioVariable, component, out outputOffset); + } + else + { + isValidOutput = resourceManager.Reservations.TryGetOffset(storageKind, ioVariable, out outputOffset); + } + } + + return isValidOutput; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/ITransformPass.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/ITransformPass.cs new file mode 100644 index 0000000..792f0a6 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/ITransformPass.cs @@ -0,0 +1,11 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System.Collections.Generic; + +namespace Ryujinx.Graphics.Shader.Translation.Transforms +{ + interface ITransformPass + { + abstract static bool IsEnabled(IGpuAccessor gpuAccessor, ShaderStage stage, TargetLanguage targetLanguage, FeatureFlags usedFeatures); + abstract static LinkedListNode RunPass(TransformContext context, LinkedListNode node); + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/SharedAtomicSignedCas.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/SharedAtomicSignedCas.cs new file mode 100644 index 0000000..2385995 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/SharedAtomicSignedCas.cs @@ -0,0 +1,58 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation.Optimizations; +using System.Collections.Generic; +using System.Diagnostics; + +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Translation.Transforms +{ + class SharedAtomicSignedCas : ITransformPass + { + public static bool IsEnabled(IGpuAccessor gpuAccessor, ShaderStage stage, TargetLanguage targetLanguage, FeatureFlags usedFeatures) + { + return targetLanguage != TargetLanguage.Spirv && stage == ShaderStage.Compute && usedFeatures.HasFlag(FeatureFlags.SharedMemory); + } + + public static LinkedListNode RunPass(TransformContext context, LinkedListNode node) + { + Operation operation = (Operation)node.Value; + HelperFunctionName name; + + if (operation.Inst == Instruction.AtomicMaxS32) + { + name = HelperFunctionName.SharedAtomicMaxS32; + } + else if (operation.Inst == Instruction.AtomicMinS32) + { + name = HelperFunctionName.SharedAtomicMinS32; + } + else + { + return node; + } + + if (operation.StorageKind != StorageKind.SharedMemory) + { + return node; + } + + Operand result = operation.Dest; + Operand memoryId = operation.GetSource(0); + Operand byteOffset = operation.GetSource(1); + Operand value = operation.GetSource(2); + + Debug.Assert(memoryId.Type == OperandType.Constant); + + int functionId = context.Hfm.GetOrCreateFunctionId(name, memoryId.Value); + + Operand[] callArgs = [Const(functionId), byteOffset, value]; + + LinkedListNode newNode = node.List.AddBefore(node, new Operation(Instruction.Call, 0, result, callArgs)); + + Utils.DeleteNode(node, operation); + + return newNode; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/SharedStoreSmallIntCas.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/SharedStoreSmallIntCas.cs new file mode 100644 index 0000000..57b62f9 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/SharedStoreSmallIntCas.cs @@ -0,0 +1,57 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation.Optimizations; +using System.Collections.Generic; +using System.Diagnostics; + +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Translation.Transforms +{ + class SharedStoreSmallIntCas : ITransformPass + { + public static bool IsEnabled(IGpuAccessor gpuAccessor, ShaderStage stage, TargetLanguage targetLanguage, FeatureFlags usedFeatures) + { + return stage == ShaderStage.Compute && usedFeatures.HasFlag(FeatureFlags.SharedMemory); + } + + public static LinkedListNode RunPass(TransformContext context, LinkedListNode node) + { + Operation operation = (Operation)node.Value; + HelperFunctionName name; + + if (operation.StorageKind == StorageKind.SharedMemory8) + { + name = HelperFunctionName.SharedStore8; + } + else if (operation.StorageKind == StorageKind.SharedMemory16) + { + name = HelperFunctionName.SharedStore16; + } + else + { + return node; + } + + if (operation.Inst != Instruction.Store) + { + return node; + } + + Operand memoryId = operation.GetSource(0); + Operand byteOffset = operation.GetSource(1); + Operand value = operation.GetSource(2); + + Debug.Assert(memoryId.Type == OperandType.Constant); + + int functionId = context.Hfm.GetOrCreateFunctionId(name, memoryId.Value); + + Operand[] callArgs = [Const(functionId), byteOffset, value]; + + LinkedListNode newNode = node.List.AddBefore(node, new Operation(Instruction.Call, 0, (Operand)null, callArgs)); + + Utils.DeleteNode(node, operation); + + return newNode; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/ShufflePass.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/ShufflePass.cs new file mode 100644 index 0000000..4ee7761 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/ShufflePass.cs @@ -0,0 +1,52 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation.Optimizations; +using System.Collections.Generic; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Translation.Transforms +{ + class ShufflePass : ITransformPass + { + public static bool IsEnabled(IGpuAccessor gpuAccessor, ShaderStage stage, TargetLanguage targetLanguage, FeatureFlags usedFeatures) + { + return usedFeatures.HasFlag(FeatureFlags.Shuffle); + } + + public static LinkedListNode RunPass(TransformContext context, LinkedListNode node) + { + Operation operation = (Operation)node.Value; + + HelperFunctionName functionName = operation.Inst switch + { + Instruction.Shuffle => HelperFunctionName.Shuffle, + Instruction.ShuffleDown => HelperFunctionName.ShuffleDown, + Instruction.ShuffleUp => HelperFunctionName.ShuffleUp, + Instruction.ShuffleXor => HelperFunctionName.ShuffleXor, + _ => HelperFunctionName.Invalid, + }; + + if (functionName == HelperFunctionName.Invalid || operation.SourcesCount != 3 || operation.DestsCount != 2) + { + return node; + } + + int functionId = context.Hfm.GetOrCreateShuffleFunctionId(functionName, context.GpuAccessor.QueryHostSubgroupSize()); + + Operand result = operation.GetDest(0); + Operand valid = operation.GetDest(1); + Operand value = operation.GetSource(0); + Operand index = operation.GetSource(1); + Operand mask = operation.GetSource(2); + + operation.Dest = null; + + Operand[] callArgs = [Const(functionId), value, index, mask, valid]; + + LinkedListNode newNode = node.List.AddBefore(node, new Operation(Instruction.Call, 0, result, callArgs)); + + Utils.DeleteNode(node, operation); + + return newNode; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/TexturePass.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/TexturePass.cs new file mode 100644 index 0000000..399f5e3 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/TexturePass.cs @@ -0,0 +1,749 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System.Collections.Generic; +using System.Linq; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Translation.Transforms +{ + class TexturePass : ITransformPass + { + public static bool IsEnabled(IGpuAccessor gpuAccessor, ShaderStage stage, TargetLanguage targetLanguage, FeatureFlags usedFeatures) + { + return true; + } + + public static LinkedListNode RunPass(TransformContext context, LinkedListNode node) + { + if (node.Value is TextureOperation texOp) + { + node = InsertTexelFetchScale(context.Hfm, node, context.ResourceManager, context.Stage); + node = InsertTextureSizeUnscale(context.Hfm, node, context.ResourceManager, context.Stage); + + if (texOp.Inst == Instruction.TextureSample) + { + node = InsertCoordNormalization(context.Hfm, node, context.ResourceManager, context.GpuAccessor, context.Stage); + node = InsertCoordGatherBias(node, context.ResourceManager, context.GpuAccessor); + node = InsertConstOffsets(node, context.ResourceManager, context.GpuAccessor, context.Stage); + + if (texOp.Type == SamplerType.TextureBuffer && !context.GpuAccessor.QueryHostSupportsSnormBufferTextureFormat()) + { + node = InsertSnormNormalization(node, context.ResourceManager, context.GpuAccessor); + } + } + } + + return node; + } + + private static LinkedListNode InsertTexelFetchScale( + HelperFunctionManager hfm, + LinkedListNode node, + ResourceManager resourceManager, + ShaderStage stage) + { + TextureOperation texOp = (TextureOperation)node.Value; + + bool isBindless = (texOp.Flags & TextureFlags.Bindless) != 0; + bool intCoords = (texOp.Flags & TextureFlags.IntCoords) != 0; + + bool isImage = IsImageInstructionWithScale(texOp.Inst); + bool isIndexed = resourceManager.IsArrayOfTexturesOrImages(texOp.Binding, isImage); + + if ((texOp.Inst == Instruction.TextureSample || isImage) && + (intCoords || isImage) && + !isBindless && + !isIndexed && + stage.SupportsRenderScale && + TypeSupportsScale(texOp.Type)) + { + int functionId = hfm.GetOrCreateFunctionId(HelperFunctionName.TexelFetchScale); + int samplerIndex = isImage + ? resourceManager.GetTextureDescriptors(includeArrays: false).Length + resourceManager.FindImageDescriptorIndex(texOp.Binding) + : resourceManager.FindTextureDescriptorIndex(texOp.Binding); + + int coordsCount = texOp.Type.Dimensions; + int coordsIndex = isBindless ? 1 : 0; + + for (int index = 0; index < coordsCount; index++) + { + Operand scaledCoord = Local(); + Operand[] callArgs; + + if (stage == ShaderStage.Fragment) + { + callArgs = [Const(functionId), texOp.GetSource(coordsIndex + index), Const(samplerIndex), Const(index) + ]; + } + else + { + callArgs = [Const(functionId), texOp.GetSource(coordsIndex + index), Const(samplerIndex)]; + } + + node.List.AddBefore(node, new Operation(Instruction.Call, 0, scaledCoord, callArgs)); + + texOp.SetSource(coordsIndex + index, scaledCoord); + } + } + + return node; + } + + private static LinkedListNode InsertTextureSizeUnscale( + HelperFunctionManager hfm, + LinkedListNode node, + ResourceManager resourceManager, + ShaderStage stage) + { + TextureOperation texOp = (TextureOperation)node.Value; + + bool isBindless = (texOp.Flags & TextureFlags.Bindless) != 0; + bool isIndexed = resourceManager.IsArrayOfTexturesOrImages(texOp.Binding, isImage: false); + + if (texOp.Inst == Instruction.TextureQuerySize && + texOp.Index < 2 && + !isBindless && + !isIndexed && + stage.SupportsRenderScale && + TypeSupportsScale(texOp.Type)) + { + int functionId = hfm.GetOrCreateFunctionId(HelperFunctionName.TextureSizeUnscale); + int samplerIndex = resourceManager.FindTextureDescriptorIndex(texOp.Binding); + + for (int index = texOp.DestsCount - 1; index >= 0; index--) + { + Operand dest = texOp.GetDest(index); + + Operand unscaledSize = Local(); + + // Replace all uses with the unscaled size value. + // This must be done before the call is added, since it also is a use of the original size. + foreach (INode useOp in dest.UseOps) + { + for (int srcIndex = 0; srcIndex < useOp.SourcesCount; srcIndex++) + { + if (useOp.GetSource(srcIndex) == dest) + { + useOp.SetSource(srcIndex, unscaledSize); + } + } + } + + Operand[] callArgs = [Const(functionId), dest, Const(samplerIndex)]; + + node.List.AddAfter(node, new Operation(Instruction.Call, 0, unscaledSize, callArgs)); + } + } + + return node; + } + + private static LinkedListNode InsertCoordNormalization( + HelperFunctionManager hfm, + LinkedListNode node, + ResourceManager resourceManager, + IGpuAccessor gpuAccessor, + ShaderStage stage) + { + // Emulate non-normalized coordinates by normalizing the coordinates on the shader. + // Without normalization, the coordinates are expected to the in the [0, W or H] range, + // and otherwise, it is expected to be in the [0, 1] range. + // We normalize by dividing the coords by the texture size. + + TextureOperation texOp = (TextureOperation)node.Value; + + bool isBindless = (texOp.Flags & TextureFlags.Bindless) != 0; + bool isIndexed = resourceManager.IsArrayOfTexturesOrImages(texOp.Binding, isImage: false); + + if (isBindless || isIndexed || !resourceManager.TryGetCbufSlotAndHandleForTexture(texOp.Binding, out int cbufSlot, out int handle)) + { + return node; + } + + bool intCoords = (texOp.Flags & TextureFlags.IntCoords) != 0; + + bool isCoordNormalized = gpuAccessor.QueryTextureCoordNormalized(handle, cbufSlot); + + if (isCoordNormalized || intCoords) + { + return node; + } + + int coordsCount = texOp.Type.Dimensions; + + int normCoordsCount = (texOp.Type & SamplerType.Mask) == SamplerType.TextureCube ? 2 : coordsCount; + + for (int index = 0; index < normCoordsCount; index++) + { + Operand coordSize = Local(); + + Operand[] texSizeSources = [Const(0)]; + + LinkedListNode textureSizeNode = node.List.AddBefore(node, new TextureOperation( + Instruction.TextureQuerySize, + texOp.Type, + texOp.Format, + texOp.Flags, + texOp.Set, + texOp.Binding, + index, + [coordSize], + texSizeSources)); + + resourceManager.SetUsageFlagsForTextureQuery(texOp.Binding, texOp.Type); + + Operand source = texOp.GetSource(index); + + Operand coordNormalized = Local(); + + node.List.AddBefore(node, new Operation(Instruction.FP32 | Instruction.Divide, coordNormalized, source, GenerateI2f(node, coordSize))); + + texOp.SetSource(index, coordNormalized); + + InsertTextureSizeUnscale(hfm, textureSizeNode, resourceManager, stage); + } + + return node; + } + + private static LinkedListNode InsertCoordGatherBias(LinkedListNode node, ResourceManager resourceManager, IGpuAccessor gpuAccessor) + { + // The gather behavior when the coordinate sits right in the middle of two texels is not well defined. + // To ensure the correct texel is sampled, we add a small bias value to the coordinate. + // This value is calculated as the minimum value required to change the texel it will sample from, + // and is 0 if the host does not require the bias. + + TextureOperation texOp = (TextureOperation)node.Value; + + bool isBindless = (texOp.Flags & TextureFlags.Bindless) != 0; + bool isGather = (texOp.Flags & TextureFlags.Gather) != 0; + + int gatherBiasPrecision = gpuAccessor.QueryHostGatherBiasPrecision(); + + if (!isGather || gatherBiasPrecision == 0) + { + return node; + } + + bool isIndexed = resourceManager.IsArrayOfTexturesOrImages(texOp.Binding, isImage: false); + + int coordsCount = texOp.Type.Dimensions; + int coordsIndex = isBindless || isIndexed ? 1 : 0; + + int normCoordsCount = (texOp.Type & SamplerType.Mask) == SamplerType.TextureCube ? 2 : coordsCount; + + for (int index = 0; index < normCoordsCount; index++) + { + Operand coordSize = Local(); + Operand scaledSize = Local(); + Operand bias = Local(); + + Operand[] texSizeSources; + + if (isBindless || isIndexed) + { + texSizeSources = [texOp.GetSource(0), Const(0)]; + } + else + { + texSizeSources = [Const(0)]; + } + + node.List.AddBefore(node, new TextureOperation( + Instruction.TextureQuerySize, + texOp.Type, + texOp.Format, + texOp.Flags, + texOp.Set, + texOp.Binding, + index, + [coordSize], + texSizeSources)); + + node.List.AddBefore(node, new Operation( + Instruction.FP32 | Instruction.Multiply, + scaledSize, + GenerateI2f(node, coordSize), + ConstF((float)(1 << (gatherBiasPrecision + 1))))); + node.List.AddBefore(node, new Operation(Instruction.FP32 | Instruction.Divide, bias, ConstF(1f), scaledSize)); + + Operand source = texOp.GetSource(coordsIndex + index); + + Operand coordBiased = Local(); + + node.List.AddBefore(node, new Operation(Instruction.FP32 | Instruction.Add, coordBiased, source, bias)); + + texOp.SetSource(coordsIndex + index, coordBiased); + } + + return node; + } + + private static LinkedListNode InsertConstOffsets(LinkedListNode node, ResourceManager resourceManager, IGpuAccessor gpuAccessor, ShaderStage stage) + { + // Non-constant texture offsets are not allowed (according to the spec), + // however some GPUs does support that. + // For GPUs where it is not supported, we can replace the instruction with the following: + // For texture*Offset, we replace it by texture*, and add the offset to the P coords. + // The offset can be calculated as offset / textureSize(lod), where lod = textureQueryLod(coords). + // For texelFetchOffset, we replace it by texelFetch and add the offset to the P coords directly. + // For textureGatherOffset, we split the operation into up to 4 operations, one for each component + // that is accessed, where each textureGather operation has a different offset for each pixel. + + TextureOperation texOp = (TextureOperation)node.Value; + + bool hasOffset = (texOp.Flags & TextureFlags.Offset) != 0; + bool hasOffsets = (texOp.Flags & TextureFlags.Offsets) != 0; + + bool needsOffsetsEmulation = hasOffsets && !gpuAccessor.QueryHostSupportsTextureGatherOffsets(); + + bool hasInvalidOffset = needsOffsetsEmulation || ((hasOffset || hasOffsets) && !gpuAccessor.QueryHostSupportsNonConstantTextureOffset()); + + bool isBindless = (texOp.Flags & TextureFlags.Bindless) != 0; + + if (!hasInvalidOffset) + { + return node; + } + + bool isGather = (texOp.Flags & TextureFlags.Gather) != 0; + bool hasDerivatives = (texOp.Flags & TextureFlags.Derivatives) != 0; + bool intCoords = (texOp.Flags & TextureFlags.IntCoords) != 0; + bool hasLodBias = (texOp.Flags & TextureFlags.LodBias) != 0; + bool hasLodLevel = (texOp.Flags & TextureFlags.LodLevel) != 0; + + bool isArray = (texOp.Type & SamplerType.Array) != 0; + bool isMultisample = (texOp.Type & SamplerType.Multisample) != 0; + bool isShadow = (texOp.Type & SamplerType.Shadow) != 0; + + int coordsCount = texOp.Type.Dimensions; + + int offsetsCount; + + if (hasOffsets) + { + offsetsCount = coordsCount * 4; + } + else if (hasOffset) + { + offsetsCount = coordsCount; + } + else + { + offsetsCount = 0; + } + + bool isIndexed = resourceManager.IsArrayOfTexturesOrImages(texOp.Binding, isImage: false); + + Operand[] offsets = new Operand[offsetsCount]; + Operand[] sources = new Operand[texOp.SourcesCount - offsetsCount]; + + int copyCount = 0; + + if (isBindless || isIndexed) + { + copyCount++; + } + + Operand[] lodSources = new Operand[copyCount + coordsCount]; + + for (int index = 0; index < lodSources.Length; index++) + { + lodSources[index] = texOp.GetSource(index); + } + + copyCount += coordsCount; + + if (isArray) + { + copyCount++; + } + + if (isShadow) + { + copyCount++; + } + + if (hasDerivatives) + { + copyCount += coordsCount * 2; + } + + if (isMultisample) + { + copyCount++; + } + else if (hasLodLevel) + { + copyCount++; + } + + int srcIndex = 0; + int dstIndex = 0; + + for (int index = 0; index < copyCount; index++) + { + sources[dstIndex++] = texOp.GetSource(srcIndex++); + } + + bool areAllOffsetsConstant = true; + + for (int index = 0; index < offsetsCount; index++) + { + Operand offset = texOp.GetSource(srcIndex++); + + areAllOffsetsConstant &= offset.Type == OperandType.Constant; + + offsets[index] = offset; + } + + if (!needsOffsetsEmulation) + { + hasInvalidOffset &= !areAllOffsetsConstant; + + if (!hasInvalidOffset) + { + return node; + } + } + + if (hasLodBias) + { + sources[dstIndex++] = texOp.GetSource(srcIndex++); + } + + if (isGather && !isShadow) + { + sources[dstIndex++] = texOp.GetSource(srcIndex++); + } + + int coordsIndex = isBindless || isIndexed ? 1 : 0; + + int componentIndex = texOp.Index; + + Operand[] dests = new Operand[texOp.DestsCount]; + + for (int i = 0; i < texOp.DestsCount; i++) + { + dests[i] = texOp.GetDest(i); + } + + Operand bindlessHandle = isBindless || isIndexed ? sources[0] : null; + + LinkedListNode oldNode = node; + + if (isGather && !isShadow && hasOffsets) + { + Operand[] newSources = new Operand[sources.Length]; + + sources.CopyTo(newSources, 0); + + Operand[] texSizes = InsertTextureBaseSize(node, texOp, bindlessHandle, coordsCount); + + int destIndex = 0; + + for (int compIndex = 0; compIndex < 4; compIndex++) + { + if (((texOp.Index >> compIndex) & 1) == 0) + { + continue; + } + + for (int index = 0; index < coordsCount; index++) + { + Operand offset = Local(); + + Operand intOffset = offsets[index + compIndex * coordsCount]; + + node.List.AddBefore(node, new Operation( + Instruction.FP32 | Instruction.Divide, + offset, + GenerateI2f(node, intOffset), + GenerateI2f(node, texSizes[index]))); + + Operand source = sources[coordsIndex + index]; + + Operand coordPlusOffset = Local(); + + node.List.AddBefore(node, new Operation(Instruction.FP32 | Instruction.Add, coordPlusOffset, source, offset)); + + newSources[coordsIndex + index] = coordPlusOffset; + } + + TextureOperation newTexOp = new( + Instruction.TextureSample, + texOp.Type, + texOp.Format, + texOp.Flags & ~(TextureFlags.Offset | TextureFlags.Offsets), + texOp.Set, + texOp.Binding, + 1 << 3, // W component: i=0, j=0 + [dests[destIndex++]], + newSources); + + node = node.List.AddBefore(node, newTexOp); + } + } + else + { + if (intCoords) + { + for (int index = 0; index < coordsCount; index++) + { + Operand source = sources[coordsIndex + index]; + + Operand coordPlusOffset = Local(); + + node.List.AddBefore(node, new Operation(Instruction.Add, coordPlusOffset, source, offsets[index])); + + sources[coordsIndex + index] = coordPlusOffset; + } + } + else + { + Operand[] texSizes = isGather + ? InsertTextureBaseSize(node, texOp, bindlessHandle, coordsCount) + : InsertTextureLod(node, texOp, lodSources, bindlessHandle, coordsCount, stage); + + for (int index = 0; index < coordsCount; index++) + { + Operand offset = Local(); + + Operand intOffset = offsets[index]; + + node.List.AddBefore(node, new Operation( + Instruction.FP32 | Instruction.Divide, + offset, + GenerateI2f(node, intOffset), + GenerateI2f(node, texSizes[index]))); + + Operand source = sources[coordsIndex + index]; + + Operand coordPlusOffset = Local(); + + node.List.AddBefore(node, new Operation(Instruction.FP32 | Instruction.Add, coordPlusOffset, source, offset)); + + sources[coordsIndex + index] = coordPlusOffset; + } + } + + TextureOperation newTexOp = new( + Instruction.TextureSample, + texOp.Type, + texOp.Format, + texOp.Flags & ~(TextureFlags.Offset | TextureFlags.Offsets), + texOp.Set, + texOp.Binding, + componentIndex, + dests, + sources); + + node = node.List.AddBefore(node, newTexOp); + } + + node.List.Remove(oldNode); + + for (int index = 0; index < texOp.SourcesCount; index++) + { + texOp.SetSource(index, null); + } + + return node; + } + + private static Operand[] InsertTextureBaseSize( + LinkedListNode node, + TextureOperation texOp, + Operand bindlessHandle, + int coordsCount) + { + Operand[] texSizes = new Operand[coordsCount]; + + for (int index = 0; index < coordsCount; index++) + { + texSizes[index] = Local(); + + Operand[] texSizeSources; + + if (bindlessHandle != null) + { + texSizeSources = [bindlessHandle, Const(0)]; + } + else + { + texSizeSources = [Const(0)]; + } + + node.List.AddBefore(node, new TextureOperation( + Instruction.TextureQuerySize, + texOp.Type, + texOp.Format, + texOp.Flags, + texOp.Set, + texOp.Binding, + index, + [texSizes[index]], + texSizeSources)); + } + + return texSizes; + } + + private static Operand[] InsertTextureLod( + LinkedListNode node, + TextureOperation texOp, + Operand[] lodSources, + Operand bindlessHandle, + int coordsCount, + ShaderStage stage) + { + Operand[] texSizes = new Operand[coordsCount]; + + Operand lod; + + if (stage == ShaderStage.Fragment) + { + lod = Local(); + + node.List.AddBefore(node, new TextureOperation( + Instruction.Lod, + texOp.Type, + texOp.Format, + texOp.Flags, + texOp.Set, + texOp.Binding, + 0, + [lod], + lodSources)); + } + else + { + lod = Const(0); + } + + for (int index = 0; index < coordsCount; index++) + { + texSizes[index] = Local(); + + Operand[] texSizeSources; + + if (bindlessHandle != null) + { + texSizeSources = [bindlessHandle, GenerateF2i(node, lod)]; + } + else + { + texSizeSources = [GenerateF2i(node, lod)]; + } + + node.List.AddBefore(node, new TextureOperation( + Instruction.TextureQuerySize, + texOp.Type, + texOp.Format, + texOp.Flags, + texOp.Set, + texOp.Binding, + index, + [texSizes[index]], + texSizeSources)); + } + + return texSizes; + } + + private static LinkedListNode InsertSnormNormalization(LinkedListNode node, ResourceManager resourceManager, IGpuAccessor gpuAccessor) + { + TextureOperation texOp = (TextureOperation)node.Value; + + // We can't query the format of a bindless texture, + // because the handle is unknown, it can have any format. + if (texOp.Flags.HasFlag(TextureFlags.Bindless) || !resourceManager.TryGetCbufSlotAndHandleForTexture(texOp.Binding, out int cbufSlot, out int handle)) + { + return node; + } + + TextureFormat format = gpuAccessor.QueryTextureFormat(handle, cbufSlot); + + int maxPositive = format switch + { + TextureFormat.R8Snorm => sbyte.MaxValue, + TextureFormat.R8G8Snorm => sbyte.MaxValue, + TextureFormat.R8G8B8A8Snorm => sbyte.MaxValue, + TextureFormat.R16Snorm => short.MaxValue, + TextureFormat.R16G16Snorm => short.MaxValue, + TextureFormat.R16G16B16A16Snorm => short.MaxValue, + _ => 0, + }; + + // The value being 0 means that the format is not a SNORM format, + // so there's nothing to do here. + if (maxPositive == 0) + { + return node; + } + + // Do normalization. We assume SINT formats are being used + // as replacement for SNORM (which is not supported). + for (int i = 0; i < texOp.DestsCount; i++) + { + Operand dest = texOp.GetDest(i); + + INode[] uses = dest.UseOps.ToArray(); + + Operation convOp = new(Instruction.ConvertS32ToFP32, Local(), dest); + Operation normOp = new(Instruction.FP32 | Instruction.Multiply, Local(), convOp.Dest, ConstF(1f / maxPositive)); + + node = node.List.AddAfter(node, convOp); + node = node.List.AddAfter(node, normOp); + + foreach (INode useOp in uses) + { + if (useOp is not Operation op) + { + continue; + } + + // Replace all uses of the texture pixel value with the normalized value. + for (int index = 0; index < op.SourcesCount; index++) + { + if (op.GetSource(index) == dest) + { + op.SetSource(index, normOp.Dest); + } + } + } + } + + return node; + } + + private static Operand GenerateI2f(LinkedListNode node, Operand value) + { + Operand res = Local(); + + node.List.AddBefore(node, new Operation(Instruction.ConvertS32ToFP32, res, value)); + + return res; + } + + private static Operand GenerateF2i(LinkedListNode node, Operand value) + { + Operand res = Local(); + + node.List.AddBefore(node, new Operation(Instruction.ConvertFP32ToS32, res, value)); + + return res; + } + + private static bool IsImageInstructionWithScale(Instruction inst) + { + // Currently, we don't support scaling images that are modified, + // so we only need to care about the load instruction. + return inst == Instruction.ImageLoad; + } + + private static bool TypeSupportsScale(SamplerType type) + { + return (type & SamplerType.Mask) == SamplerType.Texture2D; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/TransformPasses.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/TransformPasses.cs new file mode 100644 index 0000000..b7bae7e --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/TransformPasses.cs @@ -0,0 +1,44 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System.Collections.Generic; + +namespace Ryujinx.Graphics.Shader.Translation.Transforms +{ + static class TransformPasses + { + public static void RunPass(TransformContext context) + { + RunPass(context); + RunPass(context); + RunPass(context); + RunPass(context); + RunPass(context); + RunPass(context); + RunPass(context); + RunPass(context); + RunPass(context); + } + + private static void RunPass(TransformContext context) where T : ITransformPass + { + if (!T.IsEnabled(context.GpuAccessor, context.Stage, context.TargetLanguage, context.UsedFeatures)) + { + return; + } + + for (int blkIndex = 0; blkIndex < context.Blocks.Length; blkIndex++) + { + BasicBlock block = context.Blocks[blkIndex]; + + for (LinkedListNode node = block.Operations.First; node != null; node = node.Next) + { + if (node.Value is not Operation) + { + continue; + } + + node = T.RunPass(context, node); + } + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/VectorComponentSelect.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/VectorComponentSelect.cs new file mode 100644 index 0000000..d763e07 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/VectorComponentSelect.cs @@ -0,0 +1,96 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.StructuredIr; +using System.Collections.Generic; + +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Translation.Transforms +{ + class VectorComponentSelect : ITransformPass + { + public static bool IsEnabled(IGpuAccessor gpuAccessor, ShaderStage stage, TargetLanguage targetLanguage, FeatureFlags usedFeatures) + { + return gpuAccessor.QueryHostHasVectorIndexingBug(); + } + + public static LinkedListNode RunPass(TransformContext context, LinkedListNode node) + { + Operation operation = (Operation)node.Value; + + if (operation.Inst != Instruction.Load || + operation.StorageKind != StorageKind.ConstantBuffer || + operation.SourcesCount < 3) + { + return node; + } + + Operand bindingIndex = operation.GetSource(0); + Operand fieldIndex = operation.GetSource(1); + Operand elemIndex = operation.GetSource(operation.SourcesCount - 1); + + if (bindingIndex.Type != OperandType.Constant || + fieldIndex.Type != OperandType.Constant || + elemIndex.Type == OperandType.Constant) + { + return node; + } + + BufferDefinition buffer = context.ResourceManager.Properties.ConstantBuffers[bindingIndex.Value]; + StructureField field = buffer.Type.Fields[fieldIndex.Value]; + + int elemCount = (field.Type & AggregateType.ElementCountMask) switch + { + AggregateType.Vector2 => 2, + AggregateType.Vector3 => 3, + AggregateType.Vector4 => 4, + _ => 1 + }; + + if (elemCount == 1) + { + return node; + } + + Operand result = null; + + for (int i = 0; i < elemCount; i++) + { + Operand value = Local(); + Operand[] inputs = new Operand[operation.SourcesCount]; + + for (int srcIndex = 0; srcIndex < inputs.Length - 1; srcIndex++) + { + inputs[srcIndex] = operation.GetSource(srcIndex); + } + + inputs[^1] = Const(i); + + Operation loadOp = new(Instruction.Load, StorageKind.ConstantBuffer, value, inputs); + + node.List.AddBefore(node, loadOp); + + if (i == 0) + { + result = value; + } + else + { + Operand isCurrentIndex = Local(); + Operand selection = Local(); + + Operation compareOp = new(Instruction.CompareEqual, isCurrentIndex, new Operand[] { elemIndex, Const(i) }); + Operation selectOp = new(Instruction.ConditionalSelect, selection, new Operand[] { isCurrentIndex, value, result }); + + node.List.AddBefore(node, compareOp); + node.List.AddBefore(node, selectOp); + + result = selection; + } + } + + operation.TurnIntoCopy(result); + + return node; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/VertexToCompute.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/VertexToCompute.cs new file mode 100644 index 0000000..85ad2e8 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Transforms/VertexToCompute.cs @@ -0,0 +1,369 @@ +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.Translation.Optimizations; +using System.Collections.Generic; + +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Translation.Transforms +{ + class VertexToCompute : ITransformPass + { + public static bool IsEnabled(IGpuAccessor gpuAccessor, ShaderStage stage, TargetLanguage targetLanguage, FeatureFlags usedFeatures) + { + return usedFeatures.HasFlag(FeatureFlags.VtgAsCompute); + } + + public static LinkedListNode RunPass(TransformContext context, LinkedListNode node) + { + if (context.Definitions.Stage != ShaderStage.Vertex) + { + return node; + } + + Operation operation = (Operation)node.Value; + + LinkedListNode newNode = node; + + if (operation.Inst == Instruction.Load && operation.StorageKind == StorageKind.Input) + { + Operand dest = operation.Dest; + + switch ((IoVariable)operation.GetSource(0).Value) + { + case IoVariable.BaseInstance: + newNode = GenerateBaseInstanceLoad(context.ResourceManager, node, dest); + break; + case IoVariable.BaseVertex: + newNode = GenerateBaseVertexLoad(context.ResourceManager, node, dest); + break; + case IoVariable.InstanceId: + newNode = GenerateInstanceIdLoad(node, dest); + break; + case IoVariable.InstanceIndex: + newNode = GenerateInstanceIndexLoad(context.ResourceManager, node, dest); + break; + case IoVariable.VertexId: + case IoVariable.VertexIndex: + newNode = GenerateVertexIndexLoad(context.ResourceManager, node, dest); + break; + case IoVariable.UserDefined: + int location = operation.GetSource(1).Value; + int component = operation.GetSource(2).Value; + + if (context.Definitions.IsAttributePacked(location)) + { + bool needsSextNorm = context.Definitions.IsAttributePackedRgb10A2Signed(location); + + SetBindingPair setAndBinding = context.ResourceManager.Reservations.GetVertexBufferTextureSetAndBinding(location); + Operand temp = needsSextNorm ? Local() : dest; + Operand vertexElemOffset = GenerateVertexOffset(context.ResourceManager, node, location, 0); + + newNode = node.List.AddBefore(node, new TextureOperation( + Instruction.TextureSample, + SamplerType.TextureBuffer, + TextureFormat.Unknown, + TextureFlags.IntCoords, + setAndBinding.SetIndex, + setAndBinding.Binding, + 1 << component, + [temp], + [vertexElemOffset])); + + if (needsSextNorm) + { + bool sint = context.Definitions.IsAttributeSint(location); + CopySignExtendedNormalized(node, component == 3 ? 2 : 10, !sint, dest, temp); + } + } + else + { + SetBindingPair setAndBinding = context.ResourceManager.Reservations.GetVertexBufferTextureSetAndBinding(location); + Operand temp = component > 0 ? Local() : dest; + Operand vertexElemOffset = GenerateVertexOffset(context.ResourceManager, node, location, component); + + newNode = node.List.AddBefore(node, new TextureOperation( + Instruction.TextureSample, + SamplerType.TextureBuffer, + TextureFormat.Unknown, + TextureFlags.IntCoords, + setAndBinding.SetIndex, + setAndBinding.Binding, + 1, + [temp], + [vertexElemOffset])); + + if (component > 0) + { + newNode = CopyMasked(context.ResourceManager, newNode, location, component, dest, temp); + } + } + + break; + case IoVariable.GlobalId: + case IoVariable.SubgroupEqMask: + case IoVariable.SubgroupGeMask: + case IoVariable.SubgroupGtMask: + case IoVariable.SubgroupLaneId: + case IoVariable.SubgroupLeMask: + case IoVariable.SubgroupLtMask: + // Those are valid or expected for vertex shaders. + break; + default: + context.GpuAccessor.Log($"Invalid input \"{(IoVariable)operation.GetSource(0).Value}\"."); + break; + } + } + else if (operation.Inst == Instruction.Load && operation.StorageKind == StorageKind.Output) + { + if (TryGetOutputOffset(context.ResourceManager, operation, out int outputOffset)) + { + newNode = node.List.AddBefore(node, new Operation( + Instruction.Load, + StorageKind.LocalMemory, + operation.Dest, + new[] { Const(context.ResourceManager.LocalVertexDataMemoryId), Const(outputOffset) })); + } + else + { + context.GpuAccessor.Log($"Invalid output \"{(IoVariable)operation.GetSource(0).Value}\"."); + } + } + else if (operation.Inst == Instruction.Store && operation.StorageKind == StorageKind.Output) + { + if (TryGetOutputOffset(context.ResourceManager, operation, out int outputOffset)) + { + Operand value = operation.GetSource(operation.SourcesCount - 1); + + newNode = node.List.AddBefore(node, new Operation( + Instruction.Store, + StorageKind.LocalMemory, + (Operand)null, + new[] { Const(context.ResourceManager.LocalVertexDataMemoryId), Const(outputOffset), value })); + } + else + { + context.GpuAccessor.Log($"Invalid output \"{(IoVariable)operation.GetSource(0).Value}\"."); + } + } + + if (newNode != node) + { + Utils.DeleteNode(node, operation); + } + + return newNode; + } + + private static Operand GenerateVertexOffset(ResourceManager resourceManager, LinkedListNode node, int location, int component) + { + int vertexInfoCbBinding = resourceManager.Reservations.VertexInfoConstantBufferBinding; + + Operand vertexIdVr = Local(); + GenerateVertexIdVertexRateLoad(resourceManager, node, vertexIdVr); + + Operand vertexIdIr = Local(); + GenerateVertexIdInstanceRateLoad(resourceManager, node, vertexIdIr); + + Operand attributeOffset = Local(); + node.List.AddBefore(node, new Operation( + Instruction.Load, + StorageKind.ConstantBuffer, + attributeOffset, + new[] { Const(vertexInfoCbBinding), Const((int)VertexInfoBufferField.VertexOffsets), Const(location), Const(0) })); + + Operand isInstanceRate = Local(); + node.List.AddBefore(node, new Operation( + Instruction.Load, + StorageKind.ConstantBuffer, + isInstanceRate, + new[] { Const(vertexInfoCbBinding), Const((int)VertexInfoBufferField.VertexOffsets), Const(location), Const(1) })); + + Operand vertexId = Local(); + node.List.AddBefore(node, new Operation( + Instruction.ConditionalSelect, + vertexId, + new[] { isInstanceRate, vertexIdIr, vertexIdVr })); + + Operand vertexStride = Local(); + node.List.AddBefore(node, new Operation( + Instruction.Load, + StorageKind.ConstantBuffer, + vertexStride, + new[] { Const(vertexInfoCbBinding), Const((int)VertexInfoBufferField.VertexStrides), Const(location), Const(0) })); + + Operand vertexBaseOffset = Local(); + node.List.AddBefore(node, new Operation(Instruction.Multiply, vertexBaseOffset, new[] { vertexId, vertexStride })); + + Operand vertexOffset = Local(); + node.List.AddBefore(node, new Operation(Instruction.Add, vertexOffset, new[] { attributeOffset, vertexBaseOffset })); + + Operand vertexElemOffset; + + if (component != 0) + { + vertexElemOffset = Local(); + + node.List.AddBefore(node, new Operation(Instruction.Add, vertexElemOffset, new[] { vertexOffset, Const(component) })); + } + else + { + vertexElemOffset = vertexOffset; + } + + return vertexElemOffset; + } + + private static LinkedListNode CopySignExtendedNormalized(LinkedListNode node, int bits, bool normalize, Operand dest, Operand src) + { + Operand leftShifted = Local(); + node = node.List.AddAfter(node, new Operation( + Instruction.ShiftLeft, + leftShifted, + new[] { src, Const(32 - bits) })); + + Operand rightShifted = normalize ? Local() : dest; + node = node.List.AddAfter(node, new Operation( + Instruction.ShiftRightS32, + rightShifted, + new[] { leftShifted, Const(32 - bits) })); + + if (normalize) + { + Operand asFloat = Local(); + node = node.List.AddAfter(node, new Operation(Instruction.ConvertS32ToFP32, asFloat, new[] { rightShifted })); + node = node.List.AddAfter(node, new Operation( + Instruction.FP32 | Instruction.Multiply, + dest, + new[] { asFloat, ConstF(1f / (1 << (bits - 1))) })); + } + + return node; + } + + private static LinkedListNode CopyMasked( + ResourceManager resourceManager, + LinkedListNode node, + int location, + int component, + Operand dest, + Operand src) + { + Operand componentExists = Local(); + int vertexInfoCbBinding = resourceManager.Reservations.VertexInfoConstantBufferBinding; + node = node.List.AddAfter(node, new Operation( + Instruction.Load, + StorageKind.ConstantBuffer, + componentExists, + new[] { Const(vertexInfoCbBinding), Const((int)VertexInfoBufferField.VertexStrides), Const(location), Const(component) })); + + return node.List.AddAfter(node, new Operation( + Instruction.ConditionalSelect, + dest, + new[] { componentExists, src, ConstF(component == 3 ? 1f : 0f) })); + } + + private static LinkedListNode GenerateBaseVertexLoad(ResourceManager resourceManager, LinkedListNode node, Operand dest) + { + int vertexInfoCbBinding = resourceManager.Reservations.VertexInfoConstantBufferBinding; + + return node.List.AddBefore(node, new Operation( + Instruction.Load, + StorageKind.ConstantBuffer, + dest, + new[] { Const(vertexInfoCbBinding), Const((int)VertexInfoBufferField.VertexCounts), Const(2) })); + } + + private static LinkedListNode GenerateBaseInstanceLoad(ResourceManager resourceManager, LinkedListNode node, Operand dest) + { + int vertexInfoCbBinding = resourceManager.Reservations.VertexInfoConstantBufferBinding; + + return node.List.AddBefore(node, new Operation( + Instruction.Load, + StorageKind.ConstantBuffer, + dest, + new[] { Const(vertexInfoCbBinding), Const((int)VertexInfoBufferField.VertexCounts), Const(3) })); + } + + private static LinkedListNode GenerateVertexIndexLoad(ResourceManager resourceManager, LinkedListNode node, Operand dest) + { + Operand baseVertex = Local(); + Operand vertexId = Local(); + + GenerateBaseVertexLoad(resourceManager, node, baseVertex); + GenerateVertexIdVertexRateLoad(resourceManager, node, vertexId); + + return node.List.AddBefore(node, new Operation(Instruction.Add, dest, new[] { baseVertex, vertexId })); + } + + private static LinkedListNode GenerateInstanceIndexLoad(ResourceManager resourceManager, LinkedListNode node, Operand dest) + { + Operand baseInstance = Local(); + Operand instanceId = Local(); + + GenerateBaseInstanceLoad(resourceManager, node, baseInstance); + + node.List.AddBefore(node, new Operation( + Instruction.Load, + StorageKind.Input, + instanceId, + new[] { Const((int)IoVariable.GlobalId), Const(1) })); + + return node.List.AddBefore(node, new Operation(Instruction.Add, dest, new[] { baseInstance, instanceId })); + } + + private static LinkedListNode GenerateVertexIdVertexRateLoad(ResourceManager resourceManager, LinkedListNode node, Operand dest) + { + Operand[] sources = [Const(resourceManager.LocalVertexIndexVertexRateMemoryId)]; + + return node.List.AddBefore(node, new Operation(Instruction.Load, StorageKind.LocalMemory, dest, sources)); + } + + private static LinkedListNode GenerateVertexIdInstanceRateLoad(ResourceManager resourceManager, LinkedListNode node, Operand dest) + { + Operand[] sources = [Const(resourceManager.LocalVertexIndexInstanceRateMemoryId)]; + + return node.List.AddBefore(node, new Operation(Instruction.Load, StorageKind.LocalMemory, dest, sources)); + } + + private static LinkedListNode GenerateInstanceIdLoad(LinkedListNode node, Operand dest) + { + Operand[] sources = [Const((int)IoVariable.GlobalId), Const(1)]; + + return node.List.AddBefore(node, new Operation(Instruction.Load, StorageKind.Input, dest, sources)); + } + + private static bool TryGetOutputOffset(ResourceManager resourceManager, Operation operation, out int outputOffset) + { + bool isStore = operation.Inst == Instruction.Store; + + IoVariable ioVariable = (IoVariable)operation.GetSource(0).Value; + + bool isValidOutput; + + if (ioVariable == IoVariable.UserDefined) + { + int lastIndex = operation.SourcesCount - (isStore ? 2 : 1); + + int location = operation.GetSource(1).Value; + int component = operation.GetSource(lastIndex).Value; + + isValidOutput = resourceManager.Reservations.TryGetOffset(StorageKind.Output, location, component, out outputOffset); + } + else + { + if (ResourceReservations.IsVectorOrArrayVariable(ioVariable)) + { + int component = operation.GetSource(operation.SourcesCount - (isStore ? 2 : 1)).Value; + + isValidOutput = resourceManager.Reservations.TryGetOffset(StorageKind.Output, ioVariable, component, out outputOffset); + } + else + { + isValidOutput = resourceManager.Reservations.TryGetOffset(StorageKind.Output, ioVariable, out outputOffset); + } + } + + return isValidOutput; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/TranslationFlags.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/TranslationFlags.cs new file mode 100644 index 0000000..145bfe2 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/TranslationFlags.cs @@ -0,0 +1,14 @@ +using System; + +namespace Ryujinx.Graphics.Shader.Translation +{ + [Flags] + public enum TranslationFlags + { + None = 0, + + VertexA = 1 << 0, + Compute = 1 << 1, + DebugMode = 1 << 2, + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/TranslationOptions.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/TranslationOptions.cs new file mode 100644 index 0000000..a1e17ac --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/TranslationOptions.cs @@ -0,0 +1,16 @@ +namespace Ryujinx.Graphics.Shader.Translation +{ + public readonly struct TranslationOptions + { + public TargetLanguage TargetLanguage { get; } + public TargetApi TargetApi { get; } + public TranslationFlags Flags { get; } + + public TranslationOptions(TargetLanguage targetLanguage, TargetApi targetApi, TranslationFlags flags) + { + TargetLanguage = targetLanguage; + TargetApi = targetApi; + Flags = flags; + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Translator.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Translator.cs new file mode 100644 index 0000000..fccbc60 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/Translator.cs @@ -0,0 +1,397 @@ +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using System; +using System.Linq; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; + +namespace Ryujinx.Graphics.Shader.Translation +{ + public static class Translator + { + private const int ThreadsPerWarp = 32; + private const int HeaderSize = 0x50; + + internal readonly struct FunctionCode + { + public Operation[] Code { get; } + + public FunctionCode(Operation[] code) + { + Code = code; + } + } + + public static TranslatorContext CreateContext(ulong address, IGpuAccessor gpuAccessor, TranslationOptions options) + { + return DecodeShader(address, gpuAccessor, options); + } + + private static TranslatorContext DecodeShader(ulong address, IGpuAccessor gpuAccessor, TranslationOptions options) + { + int localMemorySize; + ShaderDefinitions definitions; + DecodedProgram program; + + if (options.Flags.HasFlag(TranslationFlags.Compute)) + { + definitions = CreateComputeDefinitions(gpuAccessor); + localMemorySize = gpuAccessor.QueryComputeLocalMemorySize(); + + program = Decoder.Decode(definitions, gpuAccessor, address); + } + else + { + ShaderHeader header = new(gpuAccessor, address); + + definitions = CreateGraphicsDefinitions(gpuAccessor, header); + localMemorySize = GetLocalMemorySize(header); + + program = Decoder.Decode(definitions, gpuAccessor, address + HeaderSize); + } + + ulong maxEndAddress = 0; + + foreach (DecodedFunction function in program) + { + foreach (Block block in function.Blocks) + { + if (maxEndAddress < block.EndAddress) + { + maxEndAddress = block.EndAddress; + } + } + } + + int size = (int)maxEndAddress + (options.Flags.HasFlag(TranslationFlags.Compute) ? 0 : HeaderSize); + + return new TranslatorContext(address, size, localMemorySize, definitions, gpuAccessor, options, program); + } + + private static ShaderDefinitions CreateComputeDefinitions(IGpuAccessor gpuAccessor) + { + return new ShaderDefinitions( + ShaderStage.Compute, + gpuAccessor.QueryComputeLocalSizeX(), + gpuAccessor.QueryComputeLocalSizeY(), + gpuAccessor.QueryComputeLocalSizeZ()); + } + + private static ShaderDefinitions CreateGraphicsDefinitions(IGpuAccessor gpuAccessor, ShaderHeader header) + { + TransformFeedbackOutput[] transformFeedbackOutputs = GetTransformFeedbackOutputs(gpuAccessor, out ulong transformFeedbackVecMap); + + return new ShaderDefinitions( + header.Stage, + gpuAccessor.QueryGraphicsState(), + header.Stage == ShaderStage.Geometry && header.GpPassthrough, + header.ThreadsPerInputPrimitive, + header.OutputTopology, + header.MaxOutputVertexCount, + header.ImapTypes, + header.OmapTargets, + header.OmapSampleMask, + header.OmapDepth, + gpuAccessor.QueryHostSupportsScaledVertexFormats(), + transformFeedbackVecMap, + transformFeedbackOutputs); + } + + internal static TransformFeedbackOutput[] GetTransformFeedbackOutputs(IGpuAccessor gpuAccessor, out ulong transformFeedbackVecMap) + { + bool transformFeedbackEnabled = + gpuAccessor.QueryTransformFeedbackEnabled() && + gpuAccessor.QueryHostSupportsTransformFeedback(); + TransformFeedbackOutput[] transformFeedbackOutputs = null; + transformFeedbackVecMap = 0UL; + + if (transformFeedbackEnabled) + { + transformFeedbackOutputs = new TransformFeedbackOutput[0xc0]; + + for (int tfbIndex = 0; tfbIndex < 4; tfbIndex++) + { + ReadOnlySpan locations = gpuAccessor.QueryTransformFeedbackVaryingLocations(tfbIndex); + int stride = gpuAccessor.QueryTransformFeedbackStride(tfbIndex); + + for (int i = 0; i < locations.Length; i++) + { + byte wordOffset = locations[i]; + if (wordOffset < 0xc0) + { + transformFeedbackOutputs[wordOffset] = new TransformFeedbackOutput(tfbIndex, i * 4, stride); + transformFeedbackVecMap |= 1UL << (wordOffset / 4); + } + } + } + } + + return transformFeedbackOutputs; + } + + private static int GetLocalMemorySize(ShaderHeader header) + { + return header.ShaderLocalMemoryLowSize + header.ShaderLocalMemoryHighSize + (header.ShaderLocalMemoryCrsSize / ThreadsPerWarp); + } + + internal static FunctionCode[] EmitShader( + TranslatorContext translatorContext, + ResourceManager resourceManager, + DecodedProgram program, + bool vertexAsCompute, + bool initializeOutputs, + out int initializationOperations) + { + initializationOperations = 0; + + FunctionMatch.RunPass(program); + + foreach (DecodedFunction function in program.Where(x => !x.IsCompilerGenerated).OrderBy(x => x.Address)) + { + program.AddFunctionAndSetId(function); + } + + FunctionCode[] functions = new FunctionCode[program.FunctionsWithIdCount]; + + for (int index = 0; index < functions.Length; index++) + { + EmitterContext context = new(translatorContext, resourceManager, program, vertexAsCompute, index != 0); + + if (initializeOutputs && index == 0) + { + EmitOutputsInitialization(context, translatorContext.AttributeUsage, translatorContext.GpuAccessor, translatorContext.Stage); + initializationOperations = context.OperationsCount; + } + + DecodedFunction function = program.GetFunctionById(index); + + foreach (Block block in function.Blocks) + { + context.CurrBlock = block; + + context.EnterBlock(block.Address); + + EmitOps(context, block); + } + + functions[index] = new(context.GetOperations()); + } + + return functions; + } + + private static void EmitOutputsInitialization(EmitterContext context, AttributeUsage attributeUsage, IGpuAccessor gpuAccessor, ShaderStage stage) + { + // Compute has no output attributes, so we + // don't need to initialize outputs on that stage. + if (stage == ShaderStage.Compute) + { + return; + } + + if (stage == ShaderStage.Fragment) + { + // Fragment is the last stage, so we don't need to + // initialize outputs unless we're using DSB, in which + // we need to make sure the ouput has a valid value. + if (gpuAccessor.QueryGraphicsState().DualSourceBlendEnable) + { + for (int i = 0; i < 4; i++) + { + context.Store(StorageKind.Output, IoVariable.FragmentOutputColor, null, Const(1), Const(i), ConstF(0)); + } + } + + return; + } + + if (stage == ShaderStage.Vertex) + { + InitializeVertexOutputs(context); + } + + UInt128 usedAttributes = context.TranslatorContext.AttributeUsage.NextInputAttributesComponents; + while (usedAttributes != UInt128.Zero) + { + int index = (int)UInt128.TrailingZeroCount(usedAttributes); + int vecIndex = index / 4; + + usedAttributes &= ~(UInt128.One << index); + + // We don't need to initialize passthrough attributes. + if ((context.TranslatorContext.AttributeUsage.PassthroughAttributes & (1 << vecIndex)) != 0) + { + continue; + } + + InitializeOutputComponent(context, vecIndex, index & 3, perPatch: false); + } + + if (context.TranslatorContext.AttributeUsage.NextUsedInputAttributesPerPatch != null) + { + foreach (int vecIndex in context.TranslatorContext.AttributeUsage.NextUsedInputAttributesPerPatch.Order()) + { + InitializeOutput(context, vecIndex, perPatch: true); + } + } + + if (attributeUsage.NextUsesFixedFuncAttributes) + { + bool supportsLayerFromVertexOrTess = gpuAccessor.QueryHostSupportsLayerVertexTessellation(); + int fixedStartAttr = supportsLayerFromVertexOrTess ? 0 : 1; + + for (int i = fixedStartAttr; i < fixedStartAttr + 5 + AttributeConsts.TexCoordCount; i++) + { + int index = attributeUsage.GetFreeUserAttribute(isOutput: true, i); + if (index < 0) + { + break; + } + + InitializeOutput(context, index, perPatch: false); + } + } + } + + private static void InitializeVertexOutputs(EmitterContext context) + { + for (int c = 0; c < 4; c++) + { + context.Store(StorageKind.Output, IoVariable.Position, null, Const(c), ConstF(c == 3 ? 1f : 0f)); + } + + if (context.Program.ClipDistancesWritten != 0) + { + for (int i = 0; i < 8; i++) + { + context.Store(StorageKind.Output, IoVariable.ClipDistance, null, Const(i), ConstF(0f)); + } + } + } + + private static void InitializeOutput(EmitterContext context, int location, bool perPatch) + { + for (int c = 0; c < 4; c++) + { + InitializeOutputComponent(context, location, c, perPatch); + } + } + + private static void InitializeOutputComponent(EmitterContext context, int location, int c, bool perPatch) + { + StorageKind storageKind = perPatch ? StorageKind.OutputPerPatch : StorageKind.Output; + + if (context.TranslatorContext.Definitions.OaIndexing) + { + Operand invocationId = null; + + if (context.TranslatorContext.Definitions.Stage == ShaderStage.TessellationControl && !perPatch) + { + invocationId = context.Load(StorageKind.Input, IoVariable.InvocationId); + } + + int index = location * 4 + c; + + context.Store(storageKind, IoVariable.UserDefined, invocationId, Const(index), ConstF(c == 3 ? 1f : 0f)); + } + else + { + if (context.TranslatorContext.Definitions.Stage == ShaderStage.TessellationControl && !perPatch) + { + Operand invocationId = context.Load(StorageKind.Input, IoVariable.InvocationId); + context.Store(storageKind, IoVariable.UserDefined, Const(location), invocationId, Const(c), ConstF(c == 3 ? 1f : 0f)); + } + else + { + context.Store(storageKind, IoVariable.UserDefined, null, Const(location), Const(c), ConstF(c == 3 ? 1f : 0f)); + } + } + } + + private static void EmitOps(EmitterContext context, Block block) + { + for (int opIndex = 0; opIndex < block.OpCodes.Count; opIndex++) + { + InstOp op = block.OpCodes[opIndex]; + + if (context.TranslatorContext.Options.Flags.HasFlag(TranslationFlags.DebugMode)) + { + string instName; + + if (op.Emitter != null) + { + instName = op.Name.ToString(); + } + else + { + instName = "???"; + + context.TranslatorContext.GpuAccessor.Log($"Invalid instruction at 0x{op.Address:X6} (0x{op.RawOpCode:X16})."); + } + + string dbgComment = $"0x{op.Address:X6}: 0x{op.RawOpCode:X16} {instName}"; + + context.Add(new CommentNode(dbgComment)); + } + + InstConditional opConditional = new(op.RawOpCode); + + bool noPred = op.Props.HasFlag(InstProps.NoPred); + if (!noPred && opConditional.Pred == RegisterConsts.PredicateTrueIndex && opConditional.PredInv) + { + continue; + } + + Operand predSkipLbl = null; + + if (Decoder.IsPopBranch(op.Name)) + { + // If the instruction is a SYNC or BRK instruction with only one + // possible target address, then the instruction is basically + // just a simple branch, we can generate code similar to branch + // instructions, with the condition check on the branch itself. + noPred = block.SyncTargets.Count <= 1; + } + else if (op.Name == InstName.Bra) + { + noPred = true; + } + + if (!(opConditional.Pred == RegisterConsts.PredicateTrueIndex || noPred)) + { + Operand label; + + if (opIndex == block.OpCodes.Count - 1 && block.HasNext()) + { + label = context.GetLabel(block.Successors[0].Address); + } + else + { + label = Label(); + + predSkipLbl = label; + } + + Operand pred = Register(opConditional.Pred, RegisterType.Predicate); + + if (opConditional.PredInv) + { + context.BranchIfTrue(label, pred); + } + else + { + context.BranchIfFalse(label, pred); + } + } + + context.CurrOp = op; + + op.Emitter?.Invoke(context); + + if (predSkipLbl != null) + { + context.MarkLabel(predSkipLbl); + } + } + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/TranslatorContext.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/TranslatorContext.cs new file mode 100644 index 0000000..9d9a587 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/Translation/TranslatorContext.cs @@ -0,0 +1,674 @@ +using Ryujinx.Graphics.Shader.CodeGen; +using Ryujinx.Graphics.Shader.CodeGen.Glsl; +using Ryujinx.Graphics.Shader.CodeGen.Spirv; +using Ryujinx.Graphics.Shader.Decoders; +using Ryujinx.Graphics.Shader.IntermediateRepresentation; +using Ryujinx.Graphics.Shader.StructuredIr; +using Ryujinx.Graphics.Shader.Translation.Optimizations; +using Ryujinx.Graphics.Shader.Translation.Transforms; +using System; +using System.Collections.Generic; +using System.Numerics; +using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper; +using static Ryujinx.Graphics.Shader.Translation.Translator; + +namespace Ryujinx.Graphics.Shader.Translation +{ + public class TranslatorContext + { + private readonly DecodedProgram _program; + private readonly int _localMemorySize; + private IoUsage _vertexOutput; + + public ulong Address { get; } + public int Size { get; } + public int Cb1DataSize => _program.Cb1DataSize; + + internal AttributeUsage AttributeUsage => _program.AttributeUsage; + + internal ShaderDefinitions Definitions { get; } + + public ShaderStage Stage => Definitions.Stage; + + internal IGpuAccessor GpuAccessor { get; } + + internal TranslationOptions Options { get; } + + private bool IsTransformFeedbackEmulated => !GpuAccessor.QueryHostSupportsTransformFeedback() && GpuAccessor.QueryTransformFeedbackEnabled(); + public bool HasStore => _program.UsedFeatures.HasFlag(FeatureFlags.Store) || (IsTransformFeedbackEmulated && Definitions.LastInVertexPipeline); + + public bool LayerOutputWritten { get; private set; } + public int LayerOutputAttribute { get; private set; } + + internal TranslatorContext( + ulong address, + int size, + int localMemorySize, + ShaderDefinitions definitions, + IGpuAccessor gpuAccessor, + TranslationOptions options, + DecodedProgram program) + { + Address = address; + Size = size; + _program = program; + _localMemorySize = localMemorySize; + _vertexOutput = new IoUsage(FeatureFlags.None, 0, -1); + Definitions = definitions; + GpuAccessor = gpuAccessor; + Options = options; + } + + private static bool IsLoadUserDefined(Operation operation) + { + // TODO: Check if sources count match and all sources are constant. + return operation.Inst == Instruction.Load && (IoVariable)operation.GetSource(0).Value == IoVariable.UserDefined; + } + + private static bool IsStoreUserDefined(Operation operation) + { + // TODO: Check if sources count match and all sources are constant. + return operation.Inst == Instruction.Store && (IoVariable)operation.GetSource(0).Value == IoVariable.UserDefined; + } + + private static FunctionCode[] Combine(FunctionCode[] a, FunctionCode[] b, int aStart) + { + // Here we combine two shaders. + // For shader A: + // - All user attribute stores on shader A are turned into copies to a + // temporary variable. It's assumed that shader B will consume them. + // - All return instructions are turned into branch instructions, the + // branch target being the start of the shader B code. + // For shader B: + // - All user attribute loads on shader B are turned into copies from a + // temporary variable, as long that attribute is written by shader A. + FunctionCode[] output = new FunctionCode[a.Length + b.Length - 1]; + + List ops = new(a.Length + b.Length); + + Operand[] temps = new Operand[AttributeConsts.UserAttributesCount * 4]; + + Operand lblB = Label(); + + for (int index = aStart; index < a[0].Code.Length; index++) + { + Operation operation = a[0].Code[index]; + + if (IsStoreUserDefined(operation)) + { + int tIndex = operation.GetSource(1).Value * 4 + operation.GetSource(2).Value; + + Operand temp = temps[tIndex]; + + if (temp == null) + { + temp = Local(); + + temps[tIndex] = temp; + } + + operation.Dest = temp; + operation.TurnIntoCopy(operation.GetSource(operation.SourcesCount - 1)); + } + + if (operation.Inst == Instruction.Return) + { + ops.Add(new Operation(Instruction.Branch, lblB)); + } + else + { + ops.Add(operation); + } + } + + ops.Add(new Operation(Instruction.MarkLabel, lblB)); + + for (int index = 0; index < b[0].Code.Length; index++) + { + Operation operation = b[0].Code[index]; + + if (IsLoadUserDefined(operation)) + { + int tIndex = operation.GetSource(1).Value * 4 + operation.GetSource(2).Value; + + Operand temp = temps[tIndex]; + + if (temp != null) + { + operation.TurnIntoCopy(temp); + } + } + + ops.Add(operation); + } + + output[0] = new FunctionCode(ops.ToArray()); + + for (int i = 1; i < a.Length; i++) + { + output[i] = a[i]; + } + + for (int i = 1; i < b.Length; i++) + { + output[a.Length + i - 1] = b[i]; + } + + return output; + } + + internal int GetDepthRegister() + { + // The depth register is always two registers after the last color output. + return BitOperations.PopCount((uint)Definitions.OmapTargets) + 1; + } + + public void SetLayerOutputAttribute(int attr) + { + LayerOutputWritten = true; + LayerOutputAttribute = attr; + } + + public void SetLastInVertexPipeline() + { + Definitions.LastInVertexPipeline = true; + } + + public void SetNextStage(TranslatorContext nextStage) + { + AttributeUsage.MergeFromtNextStage( + Definitions.GpPassthrough, + nextStage._program.UsedFeatures.HasFlag(FeatureFlags.FixedFuncAttr), + nextStage.AttributeUsage); + + // We don't consider geometry shaders using the geometry shader passthrough feature + // as being the last because when this feature is used, it can't actually modify any of the outputs, + // so the stage that comes before it is the last one that can do modifications. + if (nextStage.Definitions.Stage != ShaderStage.Fragment && + (nextStage.Definitions.Stage != ShaderStage.Geometry || !nextStage.Definitions.GpPassthrough)) + { + Definitions.LastInVertexPipeline = false; + } + } + + public ShaderProgram Translate(bool asCompute = false) + { + ResourceManager resourceManager = CreateResourceManager(asCompute); + + bool usesLocalMemory = _program.UsedFeatures.HasFlag(FeatureFlags.LocalMemory); + + resourceManager.SetCurrentLocalMemory(_localMemorySize, usesLocalMemory); + + if (Stage == ShaderStage.Compute) + { + bool usesSharedMemory = _program.UsedFeatures.HasFlag(FeatureFlags.SharedMemory); + + resourceManager.SetCurrentSharedMemory(GpuAccessor.QueryComputeSharedMemorySize(), usesSharedMemory); + } + + FunctionCode[] code = EmitShader(this, resourceManager, _program, asCompute, initializeOutputs: true, out _); + + return Translate(code, resourceManager, _program.UsedFeatures, _program.ClipDistancesWritten, asCompute); + } + + public ShaderProgram Translate(TranslatorContext other, bool asCompute = false) + { + ResourceManager resourceManager = CreateResourceManager(asCompute); + + bool usesLocalMemory = _program.UsedFeatures.HasFlag(FeatureFlags.LocalMemory); + resourceManager.SetCurrentLocalMemory(_localMemorySize, usesLocalMemory); + + FunctionCode[] code = EmitShader(this, resourceManager, _program, asCompute, initializeOutputs: false, out _); + + bool otherUsesLocalMemory = other._program.UsedFeatures.HasFlag(FeatureFlags.LocalMemory); + resourceManager.SetCurrentLocalMemory(other._localMemorySize, otherUsesLocalMemory); + + FunctionCode[] otherCode = EmitShader(other, resourceManager, other._program, asCompute, initializeOutputs: true, out int aStart); + + code = Combine(otherCode, code, aStart); + + return Translate( + code, + resourceManager, + _program.UsedFeatures | other._program.UsedFeatures, + (byte)(_program.ClipDistancesWritten | other._program.ClipDistancesWritten), + asCompute); + } + + private ShaderProgram Translate(FunctionCode[] functions, ResourceManager resourceManager, FeatureFlags usedFeatures, byte clipDistancesWritten, bool asCompute) + { + if (asCompute) + { + usedFeatures |= FeatureFlags.VtgAsCompute; + } + + ControlFlowGraph[] cfgs = new ControlFlowGraph[functions.Length]; + RegisterUsage.FunctionRegisterUsage[] frus = new RegisterUsage.FunctionRegisterUsage[functions.Length]; + + for (int i = 0; i < functions.Length; i++) + { + cfgs[i] = ControlFlowGraph.Create(functions[i].Code); + + if (i != 0) + { + frus[i] = RegisterUsage.RunPass(cfgs[i]); + } + } + + List funcs = new(functions.Length); + + for (int i = 0; i < functions.Length; i++) + { + funcs.Add(null); + } + + HelperFunctionManager hfm = new(funcs, Definitions.Stage); + + for (int i = 0; i < functions.Length; i++) + { + ControlFlowGraph cfg = cfgs[i]; + + int inArgumentsCount = 0; + int outArgumentsCount = 0; + + if (i != 0) + { + RegisterUsage.FunctionRegisterUsage fru = frus[i]; + + inArgumentsCount = fru.InArguments.Length; + outArgumentsCount = fru.OutArguments.Length; + } + + if (cfg.Blocks.Length != 0) + { + RegisterUsage.FixupCalls(cfg.Blocks, frus); + + Dominance.FindDominators(cfg); + Dominance.FindDominanceFrontiers(cfg.Blocks); + + Ssa.Rename(cfg.Blocks); + + TransformContext context = new( + hfm, + cfg.Blocks, + Definitions, + resourceManager, + GpuAccessor, + Options.TargetApi, + Options.TargetLanguage, + Definitions.Stage, + ref usedFeatures); + + Optimizer.RunPass(context); + TransformPasses.RunPass(context); + } + + funcs[i] = new Function(cfg.Blocks, $"fun{i}", false, inArgumentsCount, outArgumentsCount); + } + + return Generate( + funcs, + AttributeUsage, + GetDefinitions(asCompute), + Definitions, + resourceManager, + usedFeatures, + clipDistancesWritten); + } + + private ShaderProgram Generate( + IReadOnlyList funcs, + AttributeUsage attributeUsage, + ShaderDefinitions definitions, + ShaderDefinitions originalDefinitions, + ResourceManager resourceManager, + FeatureFlags usedFeatures, + byte clipDistancesWritten) + { + StructuredProgramInfo sInfo = StructuredProgram.MakeStructuredProgram( + funcs, + attributeUsage, + definitions, + resourceManager, + Options.TargetLanguage, + Options.Flags.HasFlag(TranslationFlags.DebugMode)); + + int geometryVerticesPerPrimitive = Definitions.OutputTopology switch + { + OutputTopology.LineStrip => 2, + OutputTopology.TriangleStrip => 3, + _ => 1 + }; + + ShaderProgramInfo info = new ShaderProgramInfo( + resourceManager.GetConstantBufferDescriptors(), + resourceManager.GetStorageBufferDescriptors(), + resourceManager.GetTextureDescriptors(), + resourceManager.GetImageDescriptors(), + originalDefinitions.Stage, + geometryVerticesPerPrimitive, + originalDefinitions.MaxOutputVertices, + originalDefinitions.ThreadsPerInputPrimitive, + usedFeatures.HasFlag(FeatureFlags.FragCoordXY), + usedFeatures.HasFlag(FeatureFlags.InstanceId), + usedFeatures.HasFlag(FeatureFlags.DrawParameters), + usedFeatures.HasFlag(FeatureFlags.RtLayer), + clipDistancesWritten, + originalDefinitions.OmapTargets); + + HostCapabilities hostCapabilities = new HostCapabilities( + GpuAccessor.QueryHostReducedPrecision(), + GpuAccessor.QueryHostSupportsFragmentShaderInterlock(), + GpuAccessor.QueryHostSupportsFragmentShaderOrderingIntel(), + GpuAccessor.QueryHostSupportsGeometryShaderPassthrough(), + GpuAccessor.QueryHostSupportsShaderBallot(), + GpuAccessor.QueryHostSupportsShaderBarrierDivergence(), + GpuAccessor.QueryHostSupportsShaderFloat64(), + GpuAccessor.QueryHostSupportsShaderNonUniformIndexing(), + GpuAccessor.QueryHostSupportsTextureShadowLod(), + GpuAccessor.QueryHostSupportsViewportMask()); + + CodeGenParameters parameters = new CodeGenParameters(attributeUsage, definitions, resourceManager.Properties, hostCapabilities, GpuAccessor, Options.TargetApi); + + return Options.TargetLanguage switch + { + TargetLanguage.Glsl => new ShaderProgram(info, TargetLanguage.Glsl, GlslGenerator.Generate(sInfo, parameters)), + TargetLanguage.Spirv => new ShaderProgram(info, TargetLanguage.Spirv, SpirvGenerator.Generate(sInfo, parameters)), + _ => throw new NotImplementedException(Options.TargetLanguage.ToString()), + }; + } + + private ResourceManager CreateResourceManager(bool vertexAsCompute) + { + ResourceManager resourceManager = new(Definitions.Stage, GpuAccessor, GetResourceReservations()); + + if (IsTransformFeedbackEmulated) + { + StructureType tfeDataStruct = new(new StructureField[] + { + new(AggregateType.Array | AggregateType.U32, "data", 0) + }); + + for (int i = 0; i < ResourceReservations.TfeBuffersCount; i++) + { + int binding = resourceManager.Reservations.GetTfeBufferStorageBufferBinding(i); + BufferDefinition tfeDataBuffer = new(BufferLayout.Std430, 1, binding, $"tfe_data{i}", tfeDataStruct); + resourceManager.Properties.AddOrUpdateStorageBuffer(tfeDataBuffer); + } + } + + if (vertexAsCompute) + { + int vertexInfoCbBinding = resourceManager.Reservations.VertexInfoConstantBufferBinding; + BufferDefinition vertexInfoBuffer = new(BufferLayout.Std140, 0, vertexInfoCbBinding, "vb_info", VertexInfoBuffer.GetStructureType()); + resourceManager.Properties.AddOrUpdateConstantBuffer(vertexInfoBuffer); + + StructureType vertexOutputStruct = new(new StructureField[] + { + new(AggregateType.Array | AggregateType.FP32, "data", 0) + }); + + int vertexOutputSbBinding = resourceManager.Reservations.VertexOutputStorageBufferBinding; + BufferDefinition vertexOutputBuffer = new(BufferLayout.Std430, 1, vertexOutputSbBinding, "vertex_output", vertexOutputStruct); + resourceManager.Properties.AddOrUpdateStorageBuffer(vertexOutputBuffer); + + if (Stage == ShaderStage.Vertex) + { + SetBindingPair ibSetAndBinding = resourceManager.Reservations.GetIndexBufferTextureSetAndBinding(); + TextureDefinition indexBuffer = new(ibSetAndBinding.SetIndex, ibSetAndBinding.Binding, "ib_data", SamplerType.TextureBuffer); + resourceManager.Properties.AddOrUpdateTexture(indexBuffer); + + int inputMap = _program.AttributeUsage.UsedInputAttributes; + + while (inputMap != 0) + { + int location = BitOperations.TrailingZeroCount(inputMap); + SetBindingPair setAndBinding = resourceManager.Reservations.GetVertexBufferTextureSetAndBinding(location); + TextureDefinition vaBuffer = new(setAndBinding.SetIndex, setAndBinding.Binding, $"vb_data{location}", SamplerType.TextureBuffer); + resourceManager.Properties.AddOrUpdateTexture(vaBuffer); + + inputMap &= ~(1 << location); + } + } + else if (Stage == ShaderStage.Geometry) + { + SetBindingPair trbSetAndBinding = resourceManager.Reservations.GetTopologyRemapBufferTextureSetAndBinding(); + TextureDefinition remapBuffer = new(trbSetAndBinding.SetIndex, trbSetAndBinding.Binding, "trb_data", SamplerType.TextureBuffer); + resourceManager.Properties.AddOrUpdateTexture(remapBuffer); + + int geometryVbOutputSbBinding = resourceManager.Reservations.GeometryVertexOutputStorageBufferBinding; + BufferDefinition geometryVbOutputBuffer = new(BufferLayout.Std430, 1, geometryVbOutputSbBinding, "geometry_vb_output", vertexOutputStruct); + resourceManager.Properties.AddOrUpdateStorageBuffer(geometryVbOutputBuffer); + + StructureType geometryIbOutputStruct = new(new StructureField[] + { + new(AggregateType.Array | AggregateType.U32, "data", 0) + }); + + int geometryIbOutputSbBinding = resourceManager.Reservations.GeometryIndexOutputStorageBufferBinding; + BufferDefinition geometryIbOutputBuffer = new(BufferLayout.Std430, 1, geometryIbOutputSbBinding, "geometry_ib_output", geometryIbOutputStruct); + resourceManager.Properties.AddOrUpdateStorageBuffer(geometryIbOutputBuffer); + } + + resourceManager.SetVertexAsComputeLocalMemories(Definitions.Stage, Definitions.InputTopology); + } + + return resourceManager; + } + + private ShaderDefinitions GetDefinitions(bool vertexAsCompute) + { + if (vertexAsCompute) + { + return new ShaderDefinitions(ShaderStage.Compute, 32, 32, 1); + } + else + { + return Definitions; + } + } + + public ResourceReservations GetResourceReservations() + { + IoUsage ioUsage = _program.GetIoUsage(); + + if (Definitions.GpPassthrough) + { + ioUsage = ioUsage.Combine(_vertexOutput); + } + + return new ResourceReservations(GpuAccessor, IsTransformFeedbackEmulated, vertexAsCompute: true, _vertexOutput, ioUsage); + } + + public void SetVertexOutputMapForGeometryAsCompute(TranslatorContext vertexContext) + { + _vertexOutput = vertexContext._program.GetIoUsage(); + } + + public ShaderProgram GenerateVertexPassthroughForCompute() + { + AttributeUsage attributeUsage = new AttributeUsage(GpuAccessor); + ResourceManager resourceManager = new ResourceManager(ShaderStage.Vertex, GpuAccessor); + + ResourceReservations reservations = GetResourceReservations(); + + int vertexInfoCbBinding = reservations.VertexInfoConstantBufferBinding; + + if (Stage == ShaderStage.Vertex) + { + BufferDefinition vertexInfoBuffer = new(BufferLayout.Std140, 0, vertexInfoCbBinding, "vb_info", VertexInfoBuffer.GetStructureType()); + resourceManager.Properties.AddOrUpdateConstantBuffer(vertexInfoBuffer); + } + + StructureType vertexInputStruct = new(new StructureField[] + { + new(AggregateType.Array | AggregateType.FP32, "data", 0) + }); + + int vertexDataSbBinding = reservations.VertexOutputStorageBufferBinding; + BufferDefinition vertexOutputBuffer = new(BufferLayout.Std430, 1, vertexDataSbBinding, "vb_input", vertexInputStruct); + resourceManager.Properties.AddOrUpdateStorageBuffer(vertexOutputBuffer); + + EmitterContext context = new EmitterContext(); + + Operand vertexIndex = Options.TargetApi == TargetApi.OpenGL + ? context.Load(StorageKind.Input, IoVariable.VertexId) + : context.Load(StorageKind.Input, IoVariable.VertexIndex); + + if (Stage == ShaderStage.Vertex) + { + Operand vertexCount = context.Load(StorageKind.ConstantBuffer, vertexInfoCbBinding, Const((int)VertexInfoBufferField.VertexCounts), Const(0)); + + // Base instance will be always zero when this shader is used, so which one we use here doesn't really matter. + Operand instanceId = Options.TargetApi == TargetApi.OpenGL + ? context.Load(StorageKind.Input, IoVariable.InstanceId) + : context.Load(StorageKind.Input, IoVariable.InstanceIndex); + + vertexIndex = context.IAdd(context.IMultiply(instanceId, vertexCount), vertexIndex); + } + + Operand baseOffset = context.IMultiply(vertexIndex, Const(reservations.OutputSizePerInvocation)); + + foreach ((IoDefinition ioDefinition, int inputOffset) in reservations.Offsets) + { + if (ioDefinition.StorageKind != StorageKind.Output) + { + continue; + } + + Operand vertexOffset = inputOffset != 0 ? context.IAdd(baseOffset, Const(inputOffset)) : baseOffset; + Operand value = context.Load(StorageKind.StorageBuffer, vertexDataSbBinding, Const(0), vertexOffset); + + if (ioDefinition.IoVariable == IoVariable.UserDefined) + { + context.Store(StorageKind.Output, ioDefinition.IoVariable, null, Const(ioDefinition.Location), Const(ioDefinition.Component), value); + attributeUsage.SetOutputUserAttribute(ioDefinition.Location); + } + else if (ResourceReservations.IsVectorOrArrayVariable(ioDefinition.IoVariable)) + { + context.Store(StorageKind.Output, ioDefinition.IoVariable, null, Const(ioDefinition.Component), value); + } + else + { + context.Store(StorageKind.Output, ioDefinition.IoVariable, null, value); + } + } + + Operation[] operations = context.GetOperations(); + ControlFlowGraph cfg = ControlFlowGraph.Create(operations); + Function function = new Function(cfg.Blocks, "main", false, 0, 0); + + TransformFeedbackOutput[] transformFeedbackOutputs = GetTransformFeedbackOutputs(GpuAccessor, out ulong transformFeedbackVecMap); + + ShaderDefinitions definitions = new ShaderDefinitions(ShaderStage.Vertex, transformFeedbackVecMap, transformFeedbackOutputs) + { + LastInVertexPipeline = true + }; + + return Generate( + new[] { function }, + attributeUsage, + definitions, + definitions, + resourceManager, + FeatureFlags.None, + 0); + } + + public ShaderProgram GenerateGeometryPassthrough() + { + int outputAttributesMask = AttributeUsage.UsedOutputAttributes; + int layerOutputAttr = LayerOutputAttribute; + + if (LayerOutputWritten) + { + outputAttributesMask |= 1 << ((layerOutputAttr - AttributeConsts.UserAttributeBase) / 16); + } + + OutputTopology outputTopology; + int maxOutputVertices; + + switch (Definitions.InputTopology) + { + case InputTopology.Points: + outputTopology = OutputTopology.PointList; + maxOutputVertices = 1; + break; + case InputTopology.Lines: + case InputTopology.LinesAdjacency: + outputTopology = OutputTopology.LineStrip; + maxOutputVertices = 2; + break; + default: + outputTopology = OutputTopology.TriangleStrip; + maxOutputVertices = 3; + break; + } + + AttributeUsage attributeUsage = new AttributeUsage(GpuAccessor); + ResourceManager resourceManager = new ResourceManager(ShaderStage.Geometry, GpuAccessor); + + EmitterContext context = new EmitterContext(); + + for (int v = 0; v < maxOutputVertices; v++) + { + int outAttrsMask = outputAttributesMask; + + while (outAttrsMask != 0) + { + int attrIndex = BitOperations.TrailingZeroCount(outAttrsMask); + + outAttrsMask &= ~(1 << attrIndex); + + for (int c = 0; c < 4; c++) + { + int attr = AttributeConsts.UserAttributeBase + attrIndex * 16 + c * 4; + + Operand value = context.Load(StorageKind.Input, IoVariable.UserDefined, Const(attrIndex), Const(v), Const(c)); + + if (attr == layerOutputAttr) + { + context.Store(StorageKind.Output, IoVariable.Layer, null, value); + } + else + { + context.Store(StorageKind.Output, IoVariable.UserDefined, null, Const(attrIndex), Const(c), value); + } + } + } + + for (int c = 0; c < 4; c++) + { + Operand value = context.Load(StorageKind.Input, IoVariable.Position, Const(v), Const(c)); + + context.Store(StorageKind.Output, IoVariable.Position, null, Const(c), value); + } + + context.EmitVertex(); + } + + context.EndPrimitive(); + + Operation[] operations = context.GetOperations(); + ControlFlowGraph cfg = ControlFlowGraph.Create(operations); + Function function = new Function(cfg.Blocks, "main", false, 0, 0); + + ShaderDefinitions definitions = new ShaderDefinitions( + ShaderStage.Geometry, + GpuAccessor.QueryGraphicsState(), + false, + 1, + outputTopology, + maxOutputVertices); + + return Generate( + new[] { function }, + attributeUsage, + definitions, + definitions, + resourceManager, + FeatureFlags.RtLayer, + 0); + } + } +} diff --git a/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/VertexInfoBuffer.cs b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/VertexInfoBuffer.cs new file mode 100644 index 0000000..7493262 --- /dev/null +++ b/third_party/maxwell_shader/src/Ryujinx.Graphics.Shader/VertexInfoBuffer.cs @@ -0,0 +1,58 @@ +using Ryujinx.Common.Memory; +using Ryujinx.Graphics.Shader.StructuredIr; +using Ryujinx.Graphics.Shader.Translation; +using System.Runtime.CompilerServices; + +namespace Ryujinx.Graphics.Shader +{ + enum VertexInfoBufferField + { + // Must match the order of the fields on the struct. + VertexCounts, + GeometryCounts, + VertexStrides, + VertexOffsets, + } + + public struct VertexInfoBuffer + { + public static readonly int RequiredSize; + + public static readonly int VertexCountsOffset; + public static readonly int GeometryCountsOffset; + public static readonly int VertexStridesOffset; + public static readonly int VertexOffsetsOffset; + + private static int OffsetOf(ref VertexInfoBuffer storage, ref T target) + { + return (int)Unsafe.ByteOffset(ref Unsafe.As(ref storage), ref target); + } + + static VertexInfoBuffer() + { + RequiredSize = Unsafe.SizeOf(); + + VertexInfoBuffer instance = new(); + + VertexCountsOffset = OffsetOf(ref instance, ref instance.VertexCounts); + GeometryCountsOffset = OffsetOf(ref instance, ref instance.GeometryCounts); + VertexStridesOffset = OffsetOf(ref instance, ref instance.VertexStrides); + VertexOffsetsOffset = OffsetOf(ref instance, ref instance.VertexOffsets); + } + + internal static StructureType GetStructureType() + { + return new StructureType([ + new StructureField(AggregateType.Vector4 | AggregateType.U32, "vertex_counts"), + new StructureField(AggregateType.Vector4 | AggregateType.U32, "geometry_counts"), + new StructureField(AggregateType.Array | AggregateType.Vector4 | AggregateType.U32, "vertex_strides", ResourceReservations.MaxVertexBufferTextures), + new StructureField(AggregateType.Array | AggregateType.Vector4 | AggregateType.U32, "vertex_offsets", ResourceReservations.MaxVertexBufferTextures) + ]); + } + + public Vector4 VertexCounts; + public Vector4 GeometryCounts; + public Array32> VertexStrides; + public Array32> VertexOffsets; + } +} diff --git a/third_party/maxwell_shader/src/Spv.Generator/Autogenerated/CoreGrammar.cs b/third_party/maxwell_shader/src/Spv.Generator/Autogenerated/CoreGrammar.cs new file mode 100644 index 0000000..594f3fb --- /dev/null +++ b/third_party/maxwell_shader/src/Spv.Generator/Autogenerated/CoreGrammar.cs @@ -0,0 +1,5387 @@ +// AUTOGENERATED: DO NOT EDIT +// Last update date: 2021-01-06 23:02:26.837899 +#region Grammar License +// Copyright (c) 2014-2020 The Khronos Group Inc. +// +// Permission is hereby granted, free of charge, to any person obtaining a copy +// of this software and/or associated documentation files (the "Materials"), +// to deal in the Materials without restriction, including without limitation +// the rights to use, copy, modify, merge, publish, distribute, sublicense, +// and/or sell copies of the Materials, and to permit persons to whom the +// Materials are furnished to do so, subject to the following conditions: +// +// The above copyright notice and this permission notice shall be included in +// all copies or substantial portions of the Materials. +// +// MODIFICATIONS TO THIS FILE MAY MEAN IT NO LONGER ACCURATELY REFLECTS KHRONOS +// STANDARDS. THE UNMODIFIED, NORMATIVE VERSIONS OF KHRONOS SPECIFICATIONS AND +// HEADER INFORMATION ARE LOCATED AT https://www.khronos.org/registry/ +// +// THE MATERIALS ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS +// OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +// THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM,OUT OF OR IN CONNECTION WITH THE MATERIALS OR THE USE OR OTHER DEALINGS +// IN THE MATERIALS. +#endregion + +using System; +using static Spv.Specification; + +namespace Spv.Generator +{ + public partial class Module + { + // Miscellaneous + + public Instruction Nop() + { + Instruction result = NewInstruction(Op.OpNop); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction Undef(Instruction resultType) + { + Instruction result = NewInstruction(Op.OpUndef, GetNewId(), resultType); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SizeOf(Instruction resultType, Instruction pointer) + { + Instruction result = NewInstruction(Op.OpSizeOf, GetNewId(), resultType); + + result.AddOperand(pointer); + AddToFunctionDefinitions(result); + + return result; + } + + // Debug + + public Instruction SourceContinued(string continuedSource) + { + Instruction result = NewInstruction(Op.OpSourceContinued); + + result.AddOperand(continuedSource); + AddDebug(result); + + return result; + } + + public Instruction Source(SourceLanguage sourceLanguage, LiteralInteger version, Instruction file = null, string source = null) + { + Instruction result = NewInstruction(Op.OpSource); + + result.AddOperand(sourceLanguage); + result.AddOperand(version); + if (file != null) + { + result.AddOperand(file); + } + + if (source != null) + { + result.AddOperand(source); + } + + AddDebug(result); + + return result; + } + + public Instruction SourceExtension(string extension) + { + Instruction result = NewInstruction(Op.OpSourceExtension); + + result.AddOperand(extension); + AddDebug(result); + + return result; + } + + public Instruction Name(Instruction target, string name) + { + Instruction result = NewInstruction(Op.OpName); + + result.AddOperand(target); + result.AddOperand(name); + AddDebug(result); + + return result; + } + + public Instruction MemberName(Instruction type, LiteralInteger member, string name) + { + Instruction result = NewInstruction(Op.OpMemberName); + + result.AddOperand(type); + result.AddOperand(member); + result.AddOperand(name); + AddDebug(result); + + return result; + } + + public Instruction String(string str) + { + Instruction result = NewInstruction(Op.OpString, GetNewId()); + + result.AddOperand(str); + AddDebug(result); + + return result; + } + + public Instruction Line(Instruction file, LiteralInteger line, LiteralInteger column) + { + Instruction result = NewInstruction(Op.OpLine); + + result.AddOperand(file); + result.AddOperand(line); + result.AddOperand(column); + AddDebug(result); + + return result; + } + + public Instruction NoLine() + { + Instruction result = NewInstruction(Op.OpNoLine); + + AddDebug(result); + + return result; + } + + public Instruction ModuleProcessed(string process) + { + Instruction result = NewInstruction(Op.OpModuleProcessed); + + result.AddOperand(process); + AddDebug(result); + + return result; + } + + // Annotation + + public Instruction Decorate(Instruction target, Decoration decoration) + { + Instruction result = NewInstruction(Op.OpDecorate); + + result.AddOperand(target); + result.AddOperand(decoration); + AddAnnotation(result); + + return result; + } + + public Instruction Decorate(Instruction target, Decoration decoration, IOperand parameter) + { + Instruction result = NewInstruction(Op.OpDecorate); + + result.AddOperand(target); + result.AddOperand(decoration); + result.AddOperand(parameter); + AddAnnotation(result); + + return result; + } + + public Instruction Decorate(Instruction target, Decoration decoration, params ReadOnlySpan parameters) + { + Instruction result = NewInstruction(Op.OpDecorate); + + result.AddOperand(target); + result.AddOperand(decoration); + result.AddOperand(parameters); + AddAnnotation(result); + + return result; + } + + public Instruction MemberDecorate(Instruction structureType, LiteralInteger member, Decoration decoration) + { + Instruction result = NewInstruction(Op.OpMemberDecorate); + + result.AddOperand(structureType); + result.AddOperand(member); + result.AddOperand(decoration); + AddAnnotation(result); + + return result; + } + + public Instruction MemberDecorate(Instruction structureType, LiteralInteger member, Decoration decoration, IOperand parameter) + { + Instruction result = NewInstruction(Op.OpMemberDecorate); + + result.AddOperand(structureType); + result.AddOperand(member); + result.AddOperand(decoration); + result.AddOperand(parameter); + AddAnnotation(result); + + return result; + } + + public Instruction MemberDecorate(Instruction structureType, LiteralInteger member, Decoration decoration, params ReadOnlySpan parameters) + { + Instruction result = NewInstruction(Op.OpMemberDecorate); + + result.AddOperand(structureType); + result.AddOperand(member); + result.AddOperand(decoration); + result.AddOperand(parameters); + AddAnnotation(result); + + return result; + } + + public Instruction DecorationGroup() + { + Instruction result = NewInstruction(Op.OpDecorationGroup, GetNewId()); + + AddAnnotation(result); + + return result; + } + + public Instruction GroupDecorate(Instruction decorationGroup, params ReadOnlySpan targets) + { + Instruction result = NewInstruction(Op.OpGroupDecorate); + + result.AddOperand(decorationGroup); + result.AddOperand(targets); + AddAnnotation(result); + + return result; + } + + public Instruction GroupMemberDecorate(Instruction decorationGroup, params ReadOnlySpan targets) + { + Instruction result = NewInstruction(Op.OpGroupMemberDecorate); + + result.AddOperand(decorationGroup); + result.AddOperand(targets); + AddAnnotation(result); + + return result; + } + + public Instruction DecorateId(Instruction target, Decoration decoration, params ReadOnlySpan parameters) + { + Instruction result = NewInstruction(Op.OpDecorateId); + + result.AddOperand(target); + result.AddOperand(decoration); + result.AddOperand(parameters); + AddAnnotation(result); + + return result; + } + + public Instruction DecorateString(Instruction target, Decoration decoration, params ReadOnlySpan parameters) + { + Instruction result = NewInstruction(Op.OpDecorateString); + + result.AddOperand(target); + result.AddOperand(decoration); + result.AddOperand(parameters); + AddAnnotation(result); + + return result; + } + + public Instruction DecorateStringGOOGLE(Instruction target, Decoration decoration, params ReadOnlySpan parameters) + { + Instruction result = NewInstruction(Op.OpDecorateStringGOOGLE); + + result.AddOperand(target); + result.AddOperand(decoration); + result.AddOperand(parameters); + AddAnnotation(result); + + return result; + } + + public Instruction MemberDecorateString(Instruction structType, LiteralInteger member, Decoration decoration, params ReadOnlySpan parameters) + { + Instruction result = NewInstruction(Op.OpMemberDecorateString); + + result.AddOperand(structType); + result.AddOperand(member); + result.AddOperand(decoration); + result.AddOperand(parameters); + AddAnnotation(result); + + return result; + } + + public Instruction MemberDecorateStringGOOGLE(Instruction structType, LiteralInteger member, Decoration decoration, params ReadOnlySpan parameters) + { + Instruction result = NewInstruction(Op.OpMemberDecorateStringGOOGLE); + + result.AddOperand(structType); + result.AddOperand(member); + result.AddOperand(decoration); + result.AddOperand(parameters); + AddAnnotation(result); + + return result; + } + + // Type-Declaration + + public Instruction TypeVoid(bool forceIdAllocation = false) + { + Instruction result = NewInstruction(Op.OpTypeVoid); + + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + public Instruction TypeBool(bool forceIdAllocation = false) + { + Instruction result = NewInstruction(Op.OpTypeBool); + + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + public Instruction TypeInt(LiteralInteger width, LiteralInteger signedness, bool forceIdAllocation = false) + { + Instruction result = NewInstruction(Op.OpTypeInt); + + result.AddOperand(width); + result.AddOperand(signedness); + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + public Instruction TypeFloat(LiteralInteger width, bool forceIdAllocation = false) + { + Instruction result = NewInstruction(Op.OpTypeFloat); + + result.AddOperand(width); + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + public Instruction TypeVector(Instruction componentType, LiteralInteger componentCount, bool forceIdAllocation = false) + { + Instruction result = NewInstruction(Op.OpTypeVector); + + result.AddOperand(componentType); + result.AddOperand(componentCount); + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + public Instruction TypeMatrix(Instruction columnType, LiteralInteger columnCount, bool forceIdAllocation = false) + { + Instruction result = NewInstruction(Op.OpTypeMatrix); + + result.AddOperand(columnType); + result.AddOperand(columnCount); + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + public Instruction TypeImage(Instruction sampledType, Dim dim, LiteralInteger depth, LiteralInteger arrayed, LiteralInteger mS, LiteralInteger sampled, ImageFormat imageFormat, AccessQualifier accessQualifier = (AccessQualifier)int.MaxValue, bool forceIdAllocation = false) + { + Instruction result = NewInstruction(Op.OpTypeImage); + + result.AddOperand(sampledType); + result.AddOperand(dim); + result.AddOperand(depth); + result.AddOperand(arrayed); + result.AddOperand(mS); + result.AddOperand(sampled); + result.AddOperand(imageFormat); + if (accessQualifier != (AccessQualifier)int.MaxValue) + { + result.AddOperand(accessQualifier); + } + + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + public Instruction TypeSampler(bool forceIdAllocation = false) + { + Instruction result = NewInstruction(Op.OpTypeSampler); + + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + public Instruction TypeSampledImage(Instruction imageType, bool forceIdAllocation = false) + { + Instruction result = NewInstruction(Op.OpTypeSampledImage); + + result.AddOperand(imageType); + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + public Instruction TypeArray(Instruction elementType, Instruction length, bool forceIdAllocation = false) + { + Instruction result = NewInstruction(Op.OpTypeArray); + + result.AddOperand(elementType); + result.AddOperand(length); + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + public Instruction TypeRuntimeArray(Instruction elementType, bool forceIdAllocation = false) + { + Instruction result = NewInstruction(Op.OpTypeRuntimeArray); + + result.AddOperand(elementType); + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + public Instruction TypeStruct(bool forceIdAllocation, params ReadOnlySpan parameters) + { + Instruction result = NewInstruction(Op.OpTypeStruct); + + result.AddOperand(parameters); + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + public Instruction TypeOpaque(string thenameoftheopaquetype, bool forceIdAllocation = false) + { + Instruction result = NewInstruction(Op.OpTypeOpaque); + + result.AddOperand(thenameoftheopaquetype); + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + public Instruction TypePointer(StorageClass storageClass, Instruction type, bool forceIdAllocation = false) + { + Instruction result = NewInstruction(Op.OpTypePointer); + + result.AddOperand(storageClass); + result.AddOperand(type); + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + public Instruction TypeFunction(Instruction returnType, bool forceIdAllocation, params ReadOnlySpan parameters) + { + Instruction result = NewInstruction(Op.OpTypeFunction); + + result.AddOperand(returnType); + result.AddOperand(parameters); + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + public Instruction TypeEvent(bool forceIdAllocation = false) + { + Instruction result = NewInstruction(Op.OpTypeEvent); + + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + public Instruction TypeDeviceEvent(bool forceIdAllocation = false) + { + Instruction result = NewInstruction(Op.OpTypeDeviceEvent); + + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + public Instruction TypeReserveId(bool forceIdAllocation = false) + { + Instruction result = NewInstruction(Op.OpTypeReserveId); + + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + public Instruction TypeQueue(bool forceIdAllocation = false) + { + Instruction result = NewInstruction(Op.OpTypeQueue); + + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + public Instruction TypePipe(AccessQualifier qualifier, bool forceIdAllocation = false) + { + Instruction result = NewInstruction(Op.OpTypePipe); + + result.AddOperand(qualifier); + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + public Instruction TypeForwardPointer(Instruction pointerType, StorageClass storageClass, bool forceIdAllocation = false) + { + Instruction result = NewInstruction(Op.OpTypeForwardPointer); + + result.AddOperand(pointerType); + result.AddOperand(storageClass); + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + public Instruction TypePipeStorage(bool forceIdAllocation = false) + { + Instruction result = NewInstruction(Op.OpTypePipeStorage); + + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + public Instruction TypeNamedBarrier(bool forceIdAllocation = false) + { + Instruction result = NewInstruction(Op.OpTypeNamedBarrier); + + AddTypeDeclaration(result, forceIdAllocation); + + return result; + } + + // Constant-Creation + + public Instruction ConstantTrue(Instruction resultType) + { + Instruction result = NewInstruction(Op.OpConstantTrue, Instruction.InvalidId, resultType); + + AddConstant(result); + + return result; + } + + public Instruction ConstantFalse(Instruction resultType) + { + Instruction result = NewInstruction(Op.OpConstantFalse, Instruction.InvalidId, resultType); + + AddConstant(result); + + return result; + } + + public Instruction Constant(Instruction resultType, LiteralInteger value) + { + Instruction result = NewInstruction(Op.OpConstant, Instruction.InvalidId, resultType); + + result.AddOperand(value); + AddConstant(result); + + return result; + } + + public Instruction ConstantComposite(Instruction resultType, params ReadOnlySpan constituents) + { + Instruction result = NewInstruction(Op.OpConstantComposite, Instruction.InvalidId, resultType); + + result.AddOperand(constituents); + AddConstant(result); + + return result; + } + + public Instruction ConstantSampler(Instruction resultType, SamplerAddressingMode samplerAddressingMode, LiteralInteger param, SamplerFilterMode samplerFilterMode) + { + Instruction result = NewInstruction(Op.OpConstantSampler, Instruction.InvalidId, resultType); + + result.AddOperand(samplerAddressingMode); + result.AddOperand(param); + result.AddOperand(samplerFilterMode); + AddConstant(result); + + return result; + } + + public Instruction ConstantNull(Instruction resultType) + { + Instruction result = NewInstruction(Op.OpConstantNull, Instruction.InvalidId, resultType); + + AddConstant(result); + + return result; + } + + public Instruction SpecConstantTrue(Instruction resultType) + { + Instruction result = NewInstruction(Op.OpSpecConstantTrue, GetNewId(), resultType); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SpecConstantFalse(Instruction resultType) + { + Instruction result = NewInstruction(Op.OpSpecConstantFalse, GetNewId(), resultType); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SpecConstant(Instruction resultType, LiteralInteger value) + { + Instruction result = NewInstruction(Op.OpSpecConstant, GetNewId(), resultType); + + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SpecConstantComposite(Instruction resultType, params ReadOnlySpan constituents) + { + Instruction result = NewInstruction(Op.OpSpecConstantComposite, GetNewId(), resultType); + + result.AddOperand(constituents); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SpecConstantOp(Instruction resultType, LiteralInteger opcode) + { + Instruction result = NewInstruction(Op.OpSpecConstantOp, GetNewId(), resultType); + + result.AddOperand(opcode); + AddToFunctionDefinitions(result); + + return result; + } + + // Memory + + public Instruction Variable(Instruction resultType, StorageClass storageClass, Instruction initializer = null) + { + Instruction result = NewInstruction(Op.OpVariable, GetNewId(), resultType); + + result.AddOperand(storageClass); + if (initializer != null) + { + result.AddOperand(initializer); + } + + return result; + } + + public Instruction ImageTexelPointer(Instruction resultType, Instruction image, Instruction coordinate, Instruction sample) + { + Instruction result = NewInstruction(Op.OpImageTexelPointer, GetNewId(), resultType); + + result.AddOperand(image); + result.AddOperand(coordinate); + result.AddOperand(sample); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction Load(Instruction resultType, Instruction pointer, MemoryAccessMask memoryAccess = (MemoryAccessMask)int.MaxValue) + { + Instruction result = NewInstruction(Op.OpLoad, GetNewId(), resultType); + + result.AddOperand(pointer); + if (memoryAccess != (MemoryAccessMask)int.MaxValue) + { + result.AddOperand(memoryAccess); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction Store(Instruction pointer, Instruction obj, MemoryAccessMask memoryAccess = (MemoryAccessMask)int.MaxValue) + { + Instruction result = NewInstruction(Op.OpStore); + + result.AddOperand(pointer); + result.AddOperand(obj); + if (memoryAccess != (MemoryAccessMask)int.MaxValue) + { + result.AddOperand(memoryAccess); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction CopyMemory(Instruction target, Instruction source, MemoryAccessMask memoryAccess0 = (MemoryAccessMask)int.MaxValue, MemoryAccessMask memoryAccess1 = (MemoryAccessMask)int.MaxValue) + { + Instruction result = NewInstruction(Op.OpCopyMemory); + + result.AddOperand(target); + result.AddOperand(source); + if (memoryAccess0 != (MemoryAccessMask)int.MaxValue) + { + result.AddOperand(memoryAccess0); + } + + if (memoryAccess1 != (MemoryAccessMask)int.MaxValue) + { + result.AddOperand(memoryAccess1); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction CopyMemorySized(Instruction target, Instruction source, Instruction size, MemoryAccessMask memoryAccess0 = (MemoryAccessMask)int.MaxValue, MemoryAccessMask memoryAccess1 = (MemoryAccessMask)int.MaxValue) + { + Instruction result = NewInstruction(Op.OpCopyMemorySized); + + result.AddOperand(target); + result.AddOperand(source); + result.AddOperand(size); + if (memoryAccess0 != (MemoryAccessMask)int.MaxValue) + { + result.AddOperand(memoryAccess0); + } + + if (memoryAccess1 != (MemoryAccessMask)int.MaxValue) + { + result.AddOperand(memoryAccess1); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AccessChain(Instruction resultType, Instruction baseObj, Instruction index) + { + Instruction result = NewInstruction(Op.OpAccessChain, GetNewId(), resultType); + + result.AddOperand(baseObj); + result.AddOperand(index); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AccessChain(Instruction resultType, Instruction baseObj, Instruction index0, Instruction index1) + { + Instruction result = NewInstruction(Op.OpAccessChain, GetNewId(), resultType); + + result.AddOperand(baseObj); + result.AddOperand(index0); + result.AddOperand(index1); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AccessChain(Instruction resultType, Instruction baseObj, Instruction index0, Instruction index1, Instruction index2) + { + Instruction result = NewInstruction(Op.OpAccessChain, GetNewId(), resultType); + + result.AddOperand(baseObj); + result.AddOperand(index0); + result.AddOperand(index1); + result.AddOperand(index2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AccessChain(Instruction resultType, Instruction baseObj, params ReadOnlySpan indexes) + { + Instruction result = NewInstruction(Op.OpAccessChain, GetNewId(), resultType); + + result.AddOperand(baseObj); + result.AddOperand(indexes); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction InBoundsAccessChain(Instruction resultType, Instruction baseObj, params ReadOnlySpan indexes) + { + Instruction result = NewInstruction(Op.OpInBoundsAccessChain, GetNewId(), resultType); + + result.AddOperand(baseObj); + result.AddOperand(indexes); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction PtrAccessChain(Instruction resultType, Instruction baseObj, Instruction element, params ReadOnlySpan indexes) + { + Instruction result = NewInstruction(Op.OpPtrAccessChain, GetNewId(), resultType); + + result.AddOperand(baseObj); + result.AddOperand(element); + result.AddOperand(indexes); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ArrayLength(Instruction resultType, Instruction structure, LiteralInteger arraymember) + { + Instruction result = NewInstruction(Op.OpArrayLength, GetNewId(), resultType); + + result.AddOperand(structure); + result.AddOperand(arraymember); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GenericPtrMemSemantics(Instruction resultType, Instruction pointer) + { + Instruction result = NewInstruction(Op.OpGenericPtrMemSemantics, GetNewId(), resultType); + + result.AddOperand(pointer); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction InBoundsPtrAccessChain(Instruction resultType, Instruction baseObj, Instruction element, params ReadOnlySpan indexes) + { + Instruction result = NewInstruction(Op.OpInBoundsPtrAccessChain, GetNewId(), resultType); + + result.AddOperand(baseObj); + result.AddOperand(element); + result.AddOperand(indexes); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction PtrEqual(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpPtrEqual, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction PtrNotEqual(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpPtrNotEqual, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction PtrDiff(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpPtrDiff, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + // Function + + public Instruction Function(Instruction resultType, FunctionControlMask functionControl, Instruction functionType) + { + Instruction result = NewInstruction(Op.OpFunction, GetNewId(), resultType); + + result.AddOperand(functionControl); + result.AddOperand(functionType); + + return result; + } + + public void AddFunction(Instruction function) + { + AddToFunctionDefinitions(function); + } + + public Instruction FunctionParameter(Instruction resultType) + { + Instruction result = NewInstruction(Op.OpFunctionParameter, GetNewId(), resultType); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FunctionEnd() + { + Instruction result = NewInstruction(Op.OpFunctionEnd); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FunctionCall(Instruction resultType, Instruction function, params ReadOnlySpan parameters) + { + Instruction result = NewInstruction(Op.OpFunctionCall, GetNewId(), resultType); + + result.AddOperand(function); + result.AddOperand(parameters); + AddToFunctionDefinitions(result); + + return result; + } + + // Image + + public Instruction SampledImage(Instruction resultType, Instruction image, Instruction sampler) + { + Instruction result = NewInstruction(Op.OpSampledImage, GetNewId(), resultType); + + result.AddOperand(image); + result.AddOperand(sampler); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageSampleImplicitLod(Instruction resultType, Instruction sampledImage, Instruction coordinate, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageSampleImplicitLod, GetNewId(), resultType); + + result.AddOperand(sampledImage); + result.AddOperand(coordinate); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperands); + } + + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageSampleExplicitLod(Instruction resultType, Instruction sampledImage, Instruction coordinate, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageSampleExplicitLod, GetNewId(), resultType); + + result.AddOperand(sampledImage); + result.AddOperand(coordinate); + result.AddOperand(imageOperands); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageSampleDrefImplicitLod(Instruction resultType, Instruction sampledImage, Instruction coordinate, Instruction dRef, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageSampleDrefImplicitLod, GetNewId(), resultType); + + result.AddOperand(sampledImage); + result.AddOperand(coordinate); + result.AddOperand(dRef); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperands); + } + + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageSampleDrefExplicitLod(Instruction resultType, Instruction sampledImage, Instruction coordinate, Instruction dRef, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageSampleDrefExplicitLod, GetNewId(), resultType); + + result.AddOperand(sampledImage); + result.AddOperand(coordinate); + result.AddOperand(dRef); + result.AddOperand(imageOperands); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageSampleProjImplicitLod(Instruction resultType, Instruction sampledImage, Instruction coordinate, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageSampleProjImplicitLod, GetNewId(), resultType); + + result.AddOperand(sampledImage); + result.AddOperand(coordinate); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperands); + } + + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageSampleProjExplicitLod(Instruction resultType, Instruction sampledImage, Instruction coordinate, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageSampleProjExplicitLod, GetNewId(), resultType); + + result.AddOperand(sampledImage); + result.AddOperand(coordinate); + result.AddOperand(imageOperands); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageSampleProjDrefImplicitLod(Instruction resultType, Instruction sampledImage, Instruction coordinate, Instruction dRef, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageSampleProjDrefImplicitLod, GetNewId(), resultType); + + result.AddOperand(sampledImage); + result.AddOperand(coordinate); + result.AddOperand(dRef); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperands); + } + + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageSampleProjDrefExplicitLod(Instruction resultType, Instruction sampledImage, Instruction coordinate, Instruction dRef, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageSampleProjDrefExplicitLod, GetNewId(), resultType); + + result.AddOperand(sampledImage); + result.AddOperand(coordinate); + result.AddOperand(dRef); + result.AddOperand(imageOperands); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageFetch(Instruction resultType, Instruction image, Instruction coordinate, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageFetch, GetNewId(), resultType); + + result.AddOperand(image); + result.AddOperand(coordinate); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperands); + } + + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageGather(Instruction resultType, Instruction sampledImage, Instruction coordinate, Instruction component, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageGather, GetNewId(), resultType); + + result.AddOperand(sampledImage); + result.AddOperand(coordinate); + result.AddOperand(component); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperands); + } + + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageDrefGather(Instruction resultType, Instruction sampledImage, Instruction coordinate, Instruction dRef, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageDrefGather, GetNewId(), resultType); + + result.AddOperand(sampledImage); + result.AddOperand(coordinate); + result.AddOperand(dRef); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperands); + } + + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageRead(Instruction resultType, Instruction image, Instruction coordinate, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageRead, GetNewId(), resultType); + + result.AddOperand(image); + result.AddOperand(coordinate); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperands); + } + + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageWrite(Instruction image, Instruction coordinate, Instruction texel, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageWrite); + + result.AddOperand(image); + result.AddOperand(coordinate); + result.AddOperand(texel); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperands); + } + + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction Image(Instruction resultType, Instruction sampledImage) + { + Instruction result = NewInstruction(Op.OpImage, GetNewId(), resultType); + + result.AddOperand(sampledImage); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageQueryFormat(Instruction resultType, Instruction image) + { + Instruction result = NewInstruction(Op.OpImageQueryFormat, GetNewId(), resultType); + + result.AddOperand(image); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageQueryOrder(Instruction resultType, Instruction image) + { + Instruction result = NewInstruction(Op.OpImageQueryOrder, GetNewId(), resultType); + + result.AddOperand(image); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageQuerySizeLod(Instruction resultType, Instruction image, Instruction levelofDetail) + { + Instruction result = NewInstruction(Op.OpImageQuerySizeLod, GetNewId(), resultType); + + result.AddOperand(image); + result.AddOperand(levelofDetail); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageQuerySize(Instruction resultType, Instruction image) + { + Instruction result = NewInstruction(Op.OpImageQuerySize, GetNewId(), resultType); + + result.AddOperand(image); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageQueryLod(Instruction resultType, Instruction sampledImage, Instruction coordinate) + { + Instruction result = NewInstruction(Op.OpImageQueryLod, GetNewId(), resultType); + + result.AddOperand(sampledImage); + result.AddOperand(coordinate); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageQueryLevels(Instruction resultType, Instruction image) + { + Instruction result = NewInstruction(Op.OpImageQueryLevels, GetNewId(), resultType); + + result.AddOperand(image); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageQuerySamples(Instruction resultType, Instruction image) + { + Instruction result = NewInstruction(Op.OpImageQuerySamples, GetNewId(), resultType); + + result.AddOperand(image); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageSparseSampleImplicitLod(Instruction resultType, Instruction sampledImage, Instruction coordinate, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageSparseSampleImplicitLod, GetNewId(), resultType); + + result.AddOperand(sampledImage); + result.AddOperand(coordinate); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperands); + } + + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageSparseSampleExplicitLod(Instruction resultType, Instruction sampledImage, Instruction coordinate, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageSparseSampleExplicitLod, GetNewId(), resultType); + + result.AddOperand(sampledImage); + result.AddOperand(coordinate); + result.AddOperand(imageOperands); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageSparseSampleDrefImplicitLod(Instruction resultType, Instruction sampledImage, Instruction coordinate, Instruction dRef, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageSparseSampleDrefImplicitLod, GetNewId(), resultType); + + result.AddOperand(sampledImage); + result.AddOperand(coordinate); + result.AddOperand(dRef); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperands); + } + + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageSparseSampleDrefExplicitLod(Instruction resultType, Instruction sampledImage, Instruction coordinate, Instruction dRef, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageSparseSampleDrefExplicitLod, GetNewId(), resultType); + + result.AddOperand(sampledImage); + result.AddOperand(coordinate); + result.AddOperand(dRef); + result.AddOperand(imageOperands); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageSparseSampleProjImplicitLod(Instruction resultType, Instruction sampledImage, Instruction coordinate, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageSparseSampleProjImplicitLod, GetNewId(), resultType); + + result.AddOperand(sampledImage); + result.AddOperand(coordinate); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperands); + } + + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageSparseSampleProjExplicitLod(Instruction resultType, Instruction sampledImage, Instruction coordinate, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageSparseSampleProjExplicitLod, GetNewId(), resultType); + + result.AddOperand(sampledImage); + result.AddOperand(coordinate); + result.AddOperand(imageOperands); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageSparseSampleProjDrefImplicitLod(Instruction resultType, Instruction sampledImage, Instruction coordinate, Instruction dRef, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageSparseSampleProjDrefImplicitLod, GetNewId(), resultType); + + result.AddOperand(sampledImage); + result.AddOperand(coordinate); + result.AddOperand(dRef); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperands); + } + + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageSparseSampleProjDrefExplicitLod(Instruction resultType, Instruction sampledImage, Instruction coordinate, Instruction dRef, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageSparseSampleProjDrefExplicitLod, GetNewId(), resultType); + + result.AddOperand(sampledImage); + result.AddOperand(coordinate); + result.AddOperand(dRef); + result.AddOperand(imageOperands); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageSparseFetch(Instruction resultType, Instruction image, Instruction coordinate, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageSparseFetch, GetNewId(), resultType); + + result.AddOperand(image); + result.AddOperand(coordinate); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperands); + } + + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageSparseGather(Instruction resultType, Instruction sampledImage, Instruction coordinate, Instruction component, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageSparseGather, GetNewId(), resultType); + + result.AddOperand(sampledImage); + result.AddOperand(coordinate); + result.AddOperand(component); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperands); + } + + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageSparseDrefGather(Instruction resultType, Instruction sampledImage, Instruction coordinate, Instruction dRef, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageSparseDrefGather, GetNewId(), resultType); + + result.AddOperand(sampledImage); + result.AddOperand(coordinate); + result.AddOperand(dRef); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperands); + } + + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageSparseTexelsResident(Instruction resultType, Instruction residentCode) + { + Instruction result = NewInstruction(Op.OpImageSparseTexelsResident, GetNewId(), resultType); + + result.AddOperand(residentCode); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageSparseRead(Instruction resultType, Instruction image, Instruction coordinate, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageSparseRead, GetNewId(), resultType); + + result.AddOperand(image); + result.AddOperand(coordinate); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperands); + } + + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ImageSampleFootprintNV(Instruction resultType, Instruction sampledImage, Instruction coordinate, Instruction granularity, Instruction coarse, ImageOperandsMask imageOperands, params ReadOnlySpan imageOperandIds) + { + Instruction result = NewInstruction(Op.OpImageSampleFootprintNV, GetNewId(), resultType); + + result.AddOperand(sampledImage); + result.AddOperand(coordinate); + result.AddOperand(granularity); + result.AddOperand(coarse); + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperands); + } + + if (imageOperands != (ImageOperandsMask)int.MaxValue) + { + result.AddOperand(imageOperandIds); + } + + AddToFunctionDefinitions(result); + + return result; + } + + // Conversion + + public Instruction ConvertFToU(Instruction resultType, Instruction floatValue) + { + Instruction result = NewInstruction(Op.OpConvertFToU, GetNewId(), resultType); + + result.AddOperand(floatValue); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ConvertFToS(Instruction resultType, Instruction floatValue) + { + Instruction result = NewInstruction(Op.OpConvertFToS, GetNewId(), resultType); + + result.AddOperand(floatValue); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ConvertSToF(Instruction resultType, Instruction signedValue) + { + Instruction result = NewInstruction(Op.OpConvertSToF, GetNewId(), resultType); + + result.AddOperand(signedValue); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ConvertUToF(Instruction resultType, Instruction unsignedValue) + { + Instruction result = NewInstruction(Op.OpConvertUToF, GetNewId(), resultType); + + result.AddOperand(unsignedValue); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction UConvert(Instruction resultType, Instruction unsignedValue) + { + Instruction result = NewInstruction(Op.OpUConvert, GetNewId(), resultType); + + result.AddOperand(unsignedValue); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SConvert(Instruction resultType, Instruction signedValue) + { + Instruction result = NewInstruction(Op.OpSConvert, GetNewId(), resultType); + + result.AddOperand(signedValue); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FConvert(Instruction resultType, Instruction floatValue) + { + Instruction result = NewInstruction(Op.OpFConvert, GetNewId(), resultType); + + result.AddOperand(floatValue); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction QuantizeToF16(Instruction resultType, Instruction value) + { + Instruction result = NewInstruction(Op.OpQuantizeToF16, GetNewId(), resultType); + + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ConvertPtrToU(Instruction resultType, Instruction pointer) + { + Instruction result = NewInstruction(Op.OpConvertPtrToU, GetNewId(), resultType); + + result.AddOperand(pointer); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SatConvertSToU(Instruction resultType, Instruction signedValue) + { + Instruction result = NewInstruction(Op.OpSatConvertSToU, GetNewId(), resultType); + + result.AddOperand(signedValue); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SatConvertUToS(Instruction resultType, Instruction unsignedValue) + { + Instruction result = NewInstruction(Op.OpSatConvertUToS, GetNewId(), resultType); + + result.AddOperand(unsignedValue); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ConvertUToPtr(Instruction resultType, Instruction integerValue) + { + Instruction result = NewInstruction(Op.OpConvertUToPtr, GetNewId(), resultType); + + result.AddOperand(integerValue); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction PtrCastToGeneric(Instruction resultType, Instruction pointer) + { + Instruction result = NewInstruction(Op.OpPtrCastToGeneric, GetNewId(), resultType); + + result.AddOperand(pointer); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GenericCastToPtr(Instruction resultType, Instruction pointer) + { + Instruction result = NewInstruction(Op.OpGenericCastToPtr, GetNewId(), resultType); + + result.AddOperand(pointer); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GenericCastToPtrExplicit(Instruction resultType, Instruction pointer, StorageClass storage) + { + Instruction result = NewInstruction(Op.OpGenericCastToPtrExplicit, GetNewId(), resultType); + + result.AddOperand(pointer); + result.AddOperand(storage); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction Bitcast(Instruction resultType, Instruction operand) + { + Instruction result = NewInstruction(Op.OpBitcast, GetNewId(), resultType); + + result.AddOperand(operand); + AddToFunctionDefinitions(result); + + return result; + } + + // Composite + + public Instruction VectorExtractDynamic(Instruction resultType, Instruction vector, Instruction index) + { + Instruction result = NewInstruction(Op.OpVectorExtractDynamic, GetNewId(), resultType); + + result.AddOperand(vector); + result.AddOperand(index); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction VectorInsertDynamic(Instruction resultType, Instruction vector, Instruction component, Instruction index) + { + Instruction result = NewInstruction(Op.OpVectorInsertDynamic, GetNewId(), resultType); + + result.AddOperand(vector); + result.AddOperand(component); + result.AddOperand(index); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction VectorShuffle(Instruction resultType, Instruction vector1, Instruction vector2, params ReadOnlySpan components) + { + Instruction result = NewInstruction(Op.OpVectorShuffle, GetNewId(), resultType); + + result.AddOperand(vector1); + result.AddOperand(vector2); + result.AddOperand(components); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction CompositeConstruct(Instruction resultType, params ReadOnlySpan constituents) + { + Instruction result = NewInstruction(Op.OpCompositeConstruct, GetNewId(), resultType); + + result.AddOperand(constituents); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction CompositeExtract(Instruction resultType, Instruction composite, params ReadOnlySpan indexes) + { + Instruction result = NewInstruction(Op.OpCompositeExtract, GetNewId(), resultType); + + result.AddOperand(composite); + result.AddOperand(indexes); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction CompositeInsert(Instruction resultType, Instruction obj, Instruction composite, params ReadOnlySpan indexes) + { + Instruction result = NewInstruction(Op.OpCompositeInsert, GetNewId(), resultType); + + result.AddOperand(obj); + result.AddOperand(composite); + result.AddOperand(indexes); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction CopyObject(Instruction resultType, Instruction operand) + { + Instruction result = NewInstruction(Op.OpCopyObject, GetNewId(), resultType); + + result.AddOperand(operand); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction Transpose(Instruction resultType, Instruction matrix) + { + Instruction result = NewInstruction(Op.OpTranspose, GetNewId(), resultType); + + result.AddOperand(matrix); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction CopyLogical(Instruction resultType, Instruction operand) + { + Instruction result = NewInstruction(Op.OpCopyLogical, GetNewId(), resultType); + + result.AddOperand(operand); + AddToFunctionDefinitions(result); + + return result; + } + + // Arithmetic + + public Instruction SNegate(Instruction resultType, Instruction operand) + { + Instruction result = NewInstruction(Op.OpSNegate, GetNewId(), resultType); + + result.AddOperand(operand); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FNegate(Instruction resultType, Instruction operand) + { + Instruction result = NewInstruction(Op.OpFNegate, GetNewId(), resultType); + + result.AddOperand(operand); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction IAdd(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpIAdd, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FAdd(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpFAdd, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ISub(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpISub, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FSub(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpFSub, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction IMul(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpIMul, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FMul(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpFMul, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction UDiv(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpUDiv, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SDiv(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpSDiv, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FDiv(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpFDiv, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction UMod(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpUMod, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SRem(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpSRem, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SMod(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpSMod, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FRem(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpFRem, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FMod(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpFMod, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction VectorTimesScalar(Instruction resultType, Instruction vector, Instruction scalar) + { + Instruction result = NewInstruction(Op.OpVectorTimesScalar, GetNewId(), resultType); + + result.AddOperand(vector); + result.AddOperand(scalar); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction MatrixTimesScalar(Instruction resultType, Instruction matrix, Instruction scalar) + { + Instruction result = NewInstruction(Op.OpMatrixTimesScalar, GetNewId(), resultType); + + result.AddOperand(matrix); + result.AddOperand(scalar); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction VectorTimesMatrix(Instruction resultType, Instruction vector, Instruction matrix) + { + Instruction result = NewInstruction(Op.OpVectorTimesMatrix, GetNewId(), resultType); + + result.AddOperand(vector); + result.AddOperand(matrix); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction MatrixTimesVector(Instruction resultType, Instruction matrix, Instruction vector) + { + Instruction result = NewInstruction(Op.OpMatrixTimesVector, GetNewId(), resultType); + + result.AddOperand(matrix); + result.AddOperand(vector); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction MatrixTimesMatrix(Instruction resultType, Instruction leftMatrix, Instruction rightMatrix) + { + Instruction result = NewInstruction(Op.OpMatrixTimesMatrix, GetNewId(), resultType); + + result.AddOperand(leftMatrix); + result.AddOperand(rightMatrix); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction OuterProduct(Instruction resultType, Instruction vector1, Instruction vector2) + { + Instruction result = NewInstruction(Op.OpOuterProduct, GetNewId(), resultType); + + result.AddOperand(vector1); + result.AddOperand(vector2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction Dot(Instruction resultType, Instruction vector1, Instruction vector2) + { + Instruction result = NewInstruction(Op.OpDot, GetNewId(), resultType); + + result.AddOperand(vector1); + result.AddOperand(vector2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction IAddCarry(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpIAddCarry, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ISubBorrow(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpISubBorrow, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction UMulExtended(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpUMulExtended, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SMulExtended(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpSMulExtended, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + // Bit + + public Instruction ShiftRightLogical(Instruction resultType, Instruction baseObj, Instruction shift) + { + Instruction result = NewInstruction(Op.OpShiftRightLogical, GetNewId(), resultType); + + result.AddOperand(baseObj); + result.AddOperand(shift); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ShiftRightArithmetic(Instruction resultType, Instruction baseObj, Instruction shift) + { + Instruction result = NewInstruction(Op.OpShiftRightArithmetic, GetNewId(), resultType); + + result.AddOperand(baseObj); + result.AddOperand(shift); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ShiftLeftLogical(Instruction resultType, Instruction baseObj, Instruction shift) + { + Instruction result = NewInstruction(Op.OpShiftLeftLogical, GetNewId(), resultType); + + result.AddOperand(baseObj); + result.AddOperand(shift); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction BitwiseOr(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpBitwiseOr, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction BitwiseXor(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpBitwiseXor, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction BitwiseAnd(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpBitwiseAnd, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction Not(Instruction resultType, Instruction operand) + { + Instruction result = NewInstruction(Op.OpNot, GetNewId(), resultType); + + result.AddOperand(operand); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction BitFieldInsert(Instruction resultType, Instruction baseObj, Instruction insert, Instruction offset, Instruction count) + { + Instruction result = NewInstruction(Op.OpBitFieldInsert, GetNewId(), resultType); + + result.AddOperand(baseObj); + result.AddOperand(insert); + result.AddOperand(offset); + result.AddOperand(count); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction BitFieldSExtract(Instruction resultType, Instruction baseObj, Instruction offset, Instruction count) + { + Instruction result = NewInstruction(Op.OpBitFieldSExtract, GetNewId(), resultType); + + result.AddOperand(baseObj); + result.AddOperand(offset); + result.AddOperand(count); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction BitFieldUExtract(Instruction resultType, Instruction baseObj, Instruction offset, Instruction count) + { + Instruction result = NewInstruction(Op.OpBitFieldUExtract, GetNewId(), resultType); + + result.AddOperand(baseObj); + result.AddOperand(offset); + result.AddOperand(count); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction BitReverse(Instruction resultType, Instruction baseObj) + { + Instruction result = NewInstruction(Op.OpBitReverse, GetNewId(), resultType); + + result.AddOperand(baseObj); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction BitCount(Instruction resultType, Instruction baseObj) + { + Instruction result = NewInstruction(Op.OpBitCount, GetNewId(), resultType); + + result.AddOperand(baseObj); + AddToFunctionDefinitions(result); + + return result; + } + + // Relational_and_Logical + + public Instruction Any(Instruction resultType, Instruction vector) + { + Instruction result = NewInstruction(Op.OpAny, GetNewId(), resultType); + + result.AddOperand(vector); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction All(Instruction resultType, Instruction vector) + { + Instruction result = NewInstruction(Op.OpAll, GetNewId(), resultType); + + result.AddOperand(vector); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction IsNan(Instruction resultType, Instruction x) + { + Instruction result = NewInstruction(Op.OpIsNan, GetNewId(), resultType); + + result.AddOperand(x); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction IsInf(Instruction resultType, Instruction x) + { + Instruction result = NewInstruction(Op.OpIsInf, GetNewId(), resultType); + + result.AddOperand(x); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction IsFinite(Instruction resultType, Instruction x) + { + Instruction result = NewInstruction(Op.OpIsFinite, GetNewId(), resultType); + + result.AddOperand(x); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction IsNormal(Instruction resultType, Instruction x) + { + Instruction result = NewInstruction(Op.OpIsNormal, GetNewId(), resultType); + + result.AddOperand(x); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SignBitSet(Instruction resultType, Instruction x) + { + Instruction result = NewInstruction(Op.OpSignBitSet, GetNewId(), resultType); + + result.AddOperand(x); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction LessOrGreater(Instruction resultType, Instruction x, Instruction y) + { + Instruction result = NewInstruction(Op.OpLessOrGreater, GetNewId(), resultType); + + result.AddOperand(x); + result.AddOperand(y); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction Ordered(Instruction resultType, Instruction x, Instruction y) + { + Instruction result = NewInstruction(Op.OpOrdered, GetNewId(), resultType); + + result.AddOperand(x); + result.AddOperand(y); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction Unordered(Instruction resultType, Instruction x, Instruction y) + { + Instruction result = NewInstruction(Op.OpUnordered, GetNewId(), resultType); + + result.AddOperand(x); + result.AddOperand(y); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction LogicalEqual(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpLogicalEqual, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction LogicalNotEqual(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpLogicalNotEqual, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction LogicalOr(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpLogicalOr, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction LogicalAnd(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpLogicalAnd, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction LogicalNot(Instruction resultType, Instruction operand) + { + Instruction result = NewInstruction(Op.OpLogicalNot, GetNewId(), resultType); + + result.AddOperand(operand); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction Select(Instruction resultType, Instruction condition, Instruction object1, Instruction object2) + { + Instruction result = NewInstruction(Op.OpSelect, GetNewId(), resultType); + + result.AddOperand(condition); + result.AddOperand(object1); + result.AddOperand(object2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction IEqual(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpIEqual, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction INotEqual(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpINotEqual, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction UGreaterThan(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpUGreaterThan, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SGreaterThan(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpSGreaterThan, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction UGreaterThanEqual(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpUGreaterThanEqual, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SGreaterThanEqual(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpSGreaterThanEqual, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ULessThan(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpULessThan, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SLessThan(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpSLessThan, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ULessThanEqual(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpULessThanEqual, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SLessThanEqual(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpSLessThanEqual, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FOrdEqual(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpFOrdEqual, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FUnordEqual(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpFUnordEqual, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FOrdNotEqual(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpFOrdNotEqual, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FUnordNotEqual(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpFUnordNotEqual, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FOrdLessThan(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpFOrdLessThan, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FUnordLessThan(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpFUnordLessThan, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FOrdGreaterThan(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpFOrdGreaterThan, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FUnordGreaterThan(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpFUnordGreaterThan, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FOrdLessThanEqual(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpFOrdLessThanEqual, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FUnordLessThanEqual(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpFUnordLessThanEqual, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FOrdGreaterThanEqual(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpFOrdGreaterThanEqual, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FUnordGreaterThanEqual(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpFUnordGreaterThanEqual, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + // Derivative + + public Instruction DPdx(Instruction resultType, Instruction p) + { + Instruction result = NewInstruction(Op.OpDPdx, GetNewId(), resultType); + + result.AddOperand(p); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction DPdy(Instruction resultType, Instruction p) + { + Instruction result = NewInstruction(Op.OpDPdy, GetNewId(), resultType); + + result.AddOperand(p); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction Fwidth(Instruction resultType, Instruction p) + { + Instruction result = NewInstruction(Op.OpFwidth, GetNewId(), resultType); + + result.AddOperand(p); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction DPdxFine(Instruction resultType, Instruction p) + { + Instruction result = NewInstruction(Op.OpDPdxFine, GetNewId(), resultType); + + result.AddOperand(p); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction DPdyFine(Instruction resultType, Instruction p) + { + Instruction result = NewInstruction(Op.OpDPdyFine, GetNewId(), resultType); + + result.AddOperand(p); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FwidthFine(Instruction resultType, Instruction p) + { + Instruction result = NewInstruction(Op.OpFwidthFine, GetNewId(), resultType); + + result.AddOperand(p); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction DPdxCoarse(Instruction resultType, Instruction p) + { + Instruction result = NewInstruction(Op.OpDPdxCoarse, GetNewId(), resultType); + + result.AddOperand(p); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction DPdyCoarse(Instruction resultType, Instruction p) + { + Instruction result = NewInstruction(Op.OpDPdyCoarse, GetNewId(), resultType); + + result.AddOperand(p); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FwidthCoarse(Instruction resultType, Instruction p) + { + Instruction result = NewInstruction(Op.OpFwidthCoarse, GetNewId(), resultType); + + result.AddOperand(p); + AddToFunctionDefinitions(result); + + return result; + } + + // Control-Flow + + public Instruction Phi(Instruction resultType, params ReadOnlySpan parameters) + { + Instruction result = NewInstruction(Op.OpPhi, GetNewId(), resultType); + + result.AddOperand(parameters); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction LoopMerge(Instruction mergeBlock, Instruction continueTarget, LoopControlMask loopControl) + { + Instruction result = NewInstruction(Op.OpLoopMerge); + + result.AddOperand(mergeBlock); + result.AddOperand(continueTarget); + result.AddOperand(loopControl); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SelectionMerge(Instruction mergeBlock, SelectionControlMask selectionControl) + { + Instruction result = NewInstruction(Op.OpSelectionMerge); + + result.AddOperand(mergeBlock); + result.AddOperand(selectionControl); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction Label() + { + Instruction result = NewInstruction(Op.OpLabel); + + return result; + } + + public Instruction Branch(Instruction targetLabel) + { + Instruction result = NewInstruction(Op.OpBranch); + + result.AddOperand(targetLabel); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction BranchConditional(Instruction condition, Instruction trueLabel, Instruction falseLabel, params ReadOnlySpan branchweights) + { + Instruction result = NewInstruction(Op.OpBranchConditional); + + result.AddOperand(condition); + result.AddOperand(trueLabel); + result.AddOperand(falseLabel); + result.AddOperand(branchweights); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction Switch(Instruction selector, Instruction defaultObj, params ReadOnlySpan target) + { + Instruction result = NewInstruction(Op.OpSwitch); + + result.AddOperand(selector); + result.AddOperand(defaultObj); + result.AddOperand(target); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction Kill() + { + Instruction result = NewInstruction(Op.OpKill); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction Return() + { + Instruction result = NewInstruction(Op.OpReturn); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ReturnValue(Instruction value) + { + Instruction result = NewInstruction(Op.OpReturnValue); + + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction Unreachable() + { + Instruction result = NewInstruction(Op.OpUnreachable); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction LifetimeStart(Instruction pointer, LiteralInteger size) + { + Instruction result = NewInstruction(Op.OpLifetimeStart); + + result.AddOperand(pointer); + result.AddOperand(size); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction LifetimeStop(Instruction pointer, LiteralInteger size) + { + Instruction result = NewInstruction(Op.OpLifetimeStop); + + result.AddOperand(pointer); + result.AddOperand(size); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction TerminateInvocation() + { + Instruction result = NewInstruction(Op.OpTerminateInvocation); + + AddToFunctionDefinitions(result); + + return result; + } + + // Atomic + + public Instruction AtomicLoad(Instruction resultType, Instruction pointer, Instruction memory, Instruction semantics) + { + Instruction result = NewInstruction(Op.OpAtomicLoad, GetNewId(), resultType); + + result.AddOperand(pointer); + result.AddOperand(memory); + result.AddOperand(semantics); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AtomicStore(Instruction pointer, Instruction memory, Instruction semantics, Instruction value) + { + Instruction result = NewInstruction(Op.OpAtomicStore); + + result.AddOperand(pointer); + result.AddOperand(memory); + result.AddOperand(semantics); + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AtomicExchange(Instruction resultType, Instruction pointer, Instruction memory, Instruction semantics, Instruction value) + { + Instruction result = NewInstruction(Op.OpAtomicExchange, GetNewId(), resultType); + + result.AddOperand(pointer); + result.AddOperand(memory); + result.AddOperand(semantics); + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AtomicCompareExchange(Instruction resultType, Instruction pointer, Instruction memory, Instruction equal, Instruction unequal, Instruction value, Instruction comparator) + { + Instruction result = NewInstruction(Op.OpAtomicCompareExchange, GetNewId(), resultType); + + result.AddOperand(pointer); + result.AddOperand(memory); + result.AddOperand(equal); + result.AddOperand(unequal); + result.AddOperand(value); + result.AddOperand(comparator); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AtomicCompareExchangeWeak(Instruction resultType, Instruction pointer, Instruction memory, Instruction equal, Instruction unequal, Instruction value, Instruction comparator) + { + Instruction result = NewInstruction(Op.OpAtomicCompareExchangeWeak, GetNewId(), resultType); + + result.AddOperand(pointer); + result.AddOperand(memory); + result.AddOperand(equal); + result.AddOperand(unequal); + result.AddOperand(value); + result.AddOperand(comparator); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AtomicIIncrement(Instruction resultType, Instruction pointer, Instruction memory, Instruction semantics) + { + Instruction result = NewInstruction(Op.OpAtomicIIncrement, GetNewId(), resultType); + + result.AddOperand(pointer); + result.AddOperand(memory); + result.AddOperand(semantics); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AtomicIDecrement(Instruction resultType, Instruction pointer, Instruction memory, Instruction semantics) + { + Instruction result = NewInstruction(Op.OpAtomicIDecrement, GetNewId(), resultType); + + result.AddOperand(pointer); + result.AddOperand(memory); + result.AddOperand(semantics); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AtomicIAdd(Instruction resultType, Instruction pointer, Instruction memory, Instruction semantics, Instruction value) + { + Instruction result = NewInstruction(Op.OpAtomicIAdd, GetNewId(), resultType); + + result.AddOperand(pointer); + result.AddOperand(memory); + result.AddOperand(semantics); + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AtomicISub(Instruction resultType, Instruction pointer, Instruction memory, Instruction semantics, Instruction value) + { + Instruction result = NewInstruction(Op.OpAtomicISub, GetNewId(), resultType); + + result.AddOperand(pointer); + result.AddOperand(memory); + result.AddOperand(semantics); + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AtomicSMin(Instruction resultType, Instruction pointer, Instruction memory, Instruction semantics, Instruction value) + { + Instruction result = NewInstruction(Op.OpAtomicSMin, GetNewId(), resultType); + + result.AddOperand(pointer); + result.AddOperand(memory); + result.AddOperand(semantics); + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AtomicUMin(Instruction resultType, Instruction pointer, Instruction memory, Instruction semantics, Instruction value) + { + Instruction result = NewInstruction(Op.OpAtomicUMin, GetNewId(), resultType); + + result.AddOperand(pointer); + result.AddOperand(memory); + result.AddOperand(semantics); + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AtomicSMax(Instruction resultType, Instruction pointer, Instruction memory, Instruction semantics, Instruction value) + { + Instruction result = NewInstruction(Op.OpAtomicSMax, GetNewId(), resultType); + + result.AddOperand(pointer); + result.AddOperand(memory); + result.AddOperand(semantics); + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AtomicUMax(Instruction resultType, Instruction pointer, Instruction memory, Instruction semantics, Instruction value) + { + Instruction result = NewInstruction(Op.OpAtomicUMax, GetNewId(), resultType); + + result.AddOperand(pointer); + result.AddOperand(memory); + result.AddOperand(semantics); + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AtomicAnd(Instruction resultType, Instruction pointer, Instruction memory, Instruction semantics, Instruction value) + { + Instruction result = NewInstruction(Op.OpAtomicAnd, GetNewId(), resultType); + + result.AddOperand(pointer); + result.AddOperand(memory); + result.AddOperand(semantics); + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AtomicOr(Instruction resultType, Instruction pointer, Instruction memory, Instruction semantics, Instruction value) + { + Instruction result = NewInstruction(Op.OpAtomicOr, GetNewId(), resultType); + + result.AddOperand(pointer); + result.AddOperand(memory); + result.AddOperand(semantics); + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AtomicXor(Instruction resultType, Instruction pointer, Instruction memory, Instruction semantics, Instruction value) + { + Instruction result = NewInstruction(Op.OpAtomicXor, GetNewId(), resultType); + + result.AddOperand(pointer); + result.AddOperand(memory); + result.AddOperand(semantics); + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AtomicFlagTestAndSet(Instruction resultType, Instruction pointer, Instruction memory, Instruction semantics) + { + Instruction result = NewInstruction(Op.OpAtomicFlagTestAndSet, GetNewId(), resultType); + + result.AddOperand(pointer); + result.AddOperand(memory); + result.AddOperand(semantics); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AtomicFlagClear(Instruction pointer, Instruction memory, Instruction semantics) + { + Instruction result = NewInstruction(Op.OpAtomicFlagClear); + + result.AddOperand(pointer); + result.AddOperand(memory); + result.AddOperand(semantics); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AtomicFAddEXT(Instruction resultType, Instruction pointer, Instruction memory, Instruction semantics, Instruction value) + { + Instruction result = NewInstruction(Op.OpAtomicFAddEXT, GetNewId(), resultType); + + result.AddOperand(pointer); + result.AddOperand(memory); + result.AddOperand(semantics); + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + // Primitive + + public Instruction EmitVertex() + { + Instruction result = NewInstruction(Op.OpEmitVertex); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction EndPrimitive() + { + Instruction result = NewInstruction(Op.OpEndPrimitive); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction EmitStreamVertex(Instruction stream) + { + Instruction result = NewInstruction(Op.OpEmitStreamVertex); + + result.AddOperand(stream); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction EndStreamPrimitive(Instruction stream) + { + Instruction result = NewInstruction(Op.OpEndStreamPrimitive); + + result.AddOperand(stream); + AddToFunctionDefinitions(result); + + return result; + } + + // Barrier + + public Instruction ControlBarrier(Instruction execution, Instruction memory, Instruction semantics) + { + Instruction result = NewInstruction(Op.OpControlBarrier); + + result.AddOperand(execution); + result.AddOperand(memory); + result.AddOperand(semantics); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction MemoryBarrier(Instruction memory, Instruction semantics) + { + Instruction result = NewInstruction(Op.OpMemoryBarrier); + + result.AddOperand(memory); + result.AddOperand(semantics); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction NamedBarrierInitialize(Instruction resultType, Instruction subgroupCount) + { + Instruction result = NewInstruction(Op.OpNamedBarrierInitialize, GetNewId(), resultType); + + result.AddOperand(subgroupCount); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction MemoryNamedBarrier(Instruction namedBarrier, Instruction memory, Instruction semantics) + { + Instruction result = NewInstruction(Op.OpMemoryNamedBarrier); + + result.AddOperand(namedBarrier); + result.AddOperand(memory); + result.AddOperand(semantics); + AddToFunctionDefinitions(result); + + return result; + } + + // Group + + public Instruction GroupAsyncCopy(Instruction resultType, Instruction execution, Instruction destination, Instruction source, Instruction numElements, Instruction stride, Instruction eventObj) + { + Instruction result = NewInstruction(Op.OpGroupAsyncCopy, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(destination); + result.AddOperand(source); + result.AddOperand(numElements); + result.AddOperand(stride); + result.AddOperand(eventObj); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupWaitEvents(Instruction execution, Instruction numEvents, Instruction eventsList) + { + Instruction result = NewInstruction(Op.OpGroupWaitEvents); + + result.AddOperand(execution); + result.AddOperand(numEvents); + result.AddOperand(eventsList); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupAll(Instruction resultType, Instruction execution, Instruction predicate) + { + Instruction result = NewInstruction(Op.OpGroupAll, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(predicate); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupAny(Instruction resultType, Instruction execution, Instruction predicate) + { + Instruction result = NewInstruction(Op.OpGroupAny, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(predicate); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupBroadcast(Instruction resultType, Instruction execution, Instruction value, Instruction localId) + { + Instruction result = NewInstruction(Op.OpGroupBroadcast, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(value); + result.AddOperand(localId); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupIAdd(Instruction resultType, Instruction execution, GroupOperation operation, Instruction x) + { + Instruction result = NewInstruction(Op.OpGroupIAdd, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(x); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupFAdd(Instruction resultType, Instruction execution, GroupOperation operation, Instruction x) + { + Instruction result = NewInstruction(Op.OpGroupFAdd, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(x); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupFMin(Instruction resultType, Instruction execution, GroupOperation operation, Instruction x) + { + Instruction result = NewInstruction(Op.OpGroupFMin, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(x); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupUMin(Instruction resultType, Instruction execution, GroupOperation operation, Instruction x) + { + Instruction result = NewInstruction(Op.OpGroupUMin, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(x); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupSMin(Instruction resultType, Instruction execution, GroupOperation operation, Instruction x) + { + Instruction result = NewInstruction(Op.OpGroupSMin, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(x); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupFMax(Instruction resultType, Instruction execution, GroupOperation operation, Instruction x) + { + Instruction result = NewInstruction(Op.OpGroupFMax, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(x); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupUMax(Instruction resultType, Instruction execution, GroupOperation operation, Instruction x) + { + Instruction result = NewInstruction(Op.OpGroupUMax, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(x); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupSMax(Instruction resultType, Instruction execution, GroupOperation operation, Instruction x) + { + Instruction result = NewInstruction(Op.OpGroupSMax, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(x); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SubgroupBallotKHR(Instruction resultType, Instruction predicate) + { + Instruction result = NewInstruction(Op.OpSubgroupBallotKHR, GetNewId(), resultType); + + result.AddOperand(predicate); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SubgroupFirstInvocationKHR(Instruction resultType, Instruction value) + { + Instruction result = NewInstruction(Op.OpSubgroupFirstInvocationKHR, GetNewId(), resultType); + + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SubgroupAllKHR(Instruction resultType, Instruction predicate) + { + Instruction result = NewInstruction(Op.OpSubgroupAllKHR, GetNewId(), resultType); + + result.AddOperand(predicate); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SubgroupAnyKHR(Instruction resultType, Instruction predicate) + { + Instruction result = NewInstruction(Op.OpSubgroupAnyKHR, GetNewId(), resultType); + + result.AddOperand(predicate); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SubgroupAllEqualKHR(Instruction resultType, Instruction predicate) + { + Instruction result = NewInstruction(Op.OpSubgroupAllEqualKHR, GetNewId(), resultType); + + result.AddOperand(predicate); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SubgroupReadInvocationKHR(Instruction resultType, Instruction value, Instruction index) + { + Instruction result = NewInstruction(Op.OpSubgroupReadInvocationKHR, GetNewId(), resultType); + + result.AddOperand(value); + result.AddOperand(index); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupIAddNonUniformAMD(Instruction resultType, Instruction execution, GroupOperation operation, Instruction x) + { + Instruction result = NewInstruction(Op.OpGroupIAddNonUniformAMD, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(x); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupFAddNonUniformAMD(Instruction resultType, Instruction execution, GroupOperation operation, Instruction x) + { + Instruction result = NewInstruction(Op.OpGroupFAddNonUniformAMD, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(x); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupFMinNonUniformAMD(Instruction resultType, Instruction execution, GroupOperation operation, Instruction x) + { + Instruction result = NewInstruction(Op.OpGroupFMinNonUniformAMD, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(x); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupUMinNonUniformAMD(Instruction resultType, Instruction execution, GroupOperation operation, Instruction x) + { + Instruction result = NewInstruction(Op.OpGroupUMinNonUniformAMD, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(x); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupSMinNonUniformAMD(Instruction resultType, Instruction execution, GroupOperation operation, Instruction x) + { + Instruction result = NewInstruction(Op.OpGroupSMinNonUniformAMD, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(x); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupFMaxNonUniformAMD(Instruction resultType, Instruction execution, GroupOperation operation, Instruction x) + { + Instruction result = NewInstruction(Op.OpGroupFMaxNonUniformAMD, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(x); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupUMaxNonUniformAMD(Instruction resultType, Instruction execution, GroupOperation operation, Instruction x) + { + Instruction result = NewInstruction(Op.OpGroupUMaxNonUniformAMD, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(x); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupSMaxNonUniformAMD(Instruction resultType, Instruction execution, GroupOperation operation, Instruction x) + { + Instruction result = NewInstruction(Op.OpGroupSMaxNonUniformAMD, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(x); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SubgroupShuffleINTEL(Instruction resultType, Instruction data, Instruction invocationId) + { + Instruction result = NewInstruction(Op.OpSubgroupShuffleINTEL, GetNewId(), resultType); + + result.AddOperand(data); + result.AddOperand(invocationId); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SubgroupShuffleDownINTEL(Instruction resultType, Instruction current, Instruction next, Instruction delta) + { + Instruction result = NewInstruction(Op.OpSubgroupShuffleDownINTEL, GetNewId(), resultType); + + result.AddOperand(current); + result.AddOperand(next); + result.AddOperand(delta); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SubgroupShuffleUpINTEL(Instruction resultType, Instruction previous, Instruction current, Instruction delta) + { + Instruction result = NewInstruction(Op.OpSubgroupShuffleUpINTEL, GetNewId(), resultType); + + result.AddOperand(previous); + result.AddOperand(current); + result.AddOperand(delta); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SubgroupShuffleXorINTEL(Instruction resultType, Instruction data, Instruction value) + { + Instruction result = NewInstruction(Op.OpSubgroupShuffleXorINTEL, GetNewId(), resultType); + + result.AddOperand(data); + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SubgroupBlockReadINTEL(Instruction resultType, Instruction ptr) + { + Instruction result = NewInstruction(Op.OpSubgroupBlockReadINTEL, GetNewId(), resultType); + + result.AddOperand(ptr); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SubgroupBlockWriteINTEL(Instruction ptr, Instruction data) + { + Instruction result = NewInstruction(Op.OpSubgroupBlockWriteINTEL); + + result.AddOperand(ptr); + result.AddOperand(data); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SubgroupImageBlockReadINTEL(Instruction resultType, Instruction image, Instruction coordinate) + { + Instruction result = NewInstruction(Op.OpSubgroupImageBlockReadINTEL, GetNewId(), resultType); + + result.AddOperand(image); + result.AddOperand(coordinate); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SubgroupImageBlockWriteINTEL(Instruction image, Instruction coordinate, Instruction data) + { + Instruction result = NewInstruction(Op.OpSubgroupImageBlockWriteINTEL); + + result.AddOperand(image); + result.AddOperand(coordinate); + result.AddOperand(data); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SubgroupImageMediaBlockReadINTEL(Instruction resultType, Instruction image, Instruction coordinate, Instruction width, Instruction height) + { + Instruction result = NewInstruction(Op.OpSubgroupImageMediaBlockReadINTEL, GetNewId(), resultType); + + result.AddOperand(image); + result.AddOperand(coordinate); + result.AddOperand(width); + result.AddOperand(height); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SubgroupImageMediaBlockWriteINTEL(Instruction image, Instruction coordinate, Instruction width, Instruction height, Instruction data) + { + Instruction result = NewInstruction(Op.OpSubgroupImageMediaBlockWriteINTEL); + + result.AddOperand(image); + result.AddOperand(coordinate); + result.AddOperand(width); + result.AddOperand(height); + result.AddOperand(data); + AddToFunctionDefinitions(result); + + return result; + } + + // Device-Side_Enqueue + + public Instruction EnqueueMarker(Instruction resultType, Instruction queue, Instruction numEvents, Instruction waitEvents, Instruction retEvent) + { + Instruction result = NewInstruction(Op.OpEnqueueMarker, GetNewId(), resultType); + + result.AddOperand(queue); + result.AddOperand(numEvents); + result.AddOperand(waitEvents); + result.AddOperand(retEvent); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction EnqueueKernel(Instruction resultType, Instruction queue, Instruction flags, Instruction nDRange, Instruction numEvents, Instruction waitEvents, Instruction retEvent, Instruction invoke, Instruction param, Instruction paramSize, Instruction paramAlign, params ReadOnlySpan localSize) + { + Instruction result = NewInstruction(Op.OpEnqueueKernel, GetNewId(), resultType); + + result.AddOperand(queue); + result.AddOperand(flags); + result.AddOperand(nDRange); + result.AddOperand(numEvents); + result.AddOperand(waitEvents); + result.AddOperand(retEvent); + result.AddOperand(invoke); + result.AddOperand(param); + result.AddOperand(paramSize); + result.AddOperand(paramAlign); + result.AddOperand(localSize); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GetKernelNDrangeSubGroupCount(Instruction resultType, Instruction nDRange, Instruction invoke, Instruction param, Instruction paramSize, Instruction paramAlign) + { + Instruction result = NewInstruction(Op.OpGetKernelNDrangeSubGroupCount, GetNewId(), resultType); + + result.AddOperand(nDRange); + result.AddOperand(invoke); + result.AddOperand(param); + result.AddOperand(paramSize); + result.AddOperand(paramAlign); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GetKernelNDrangeMaxSubGroupSize(Instruction resultType, Instruction nDRange, Instruction invoke, Instruction param, Instruction paramSize, Instruction paramAlign) + { + Instruction result = NewInstruction(Op.OpGetKernelNDrangeMaxSubGroupSize, GetNewId(), resultType); + + result.AddOperand(nDRange); + result.AddOperand(invoke); + result.AddOperand(param); + result.AddOperand(paramSize); + result.AddOperand(paramAlign); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GetKernelWorkGroupSize(Instruction resultType, Instruction invoke, Instruction param, Instruction paramSize, Instruction paramAlign) + { + Instruction result = NewInstruction(Op.OpGetKernelWorkGroupSize, GetNewId(), resultType); + + result.AddOperand(invoke); + result.AddOperand(param); + result.AddOperand(paramSize); + result.AddOperand(paramAlign); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GetKernelPreferredWorkGroupSizeMultiple(Instruction resultType, Instruction invoke, Instruction param, Instruction paramSize, Instruction paramAlign) + { + Instruction result = NewInstruction(Op.OpGetKernelPreferredWorkGroupSizeMultiple, GetNewId(), resultType); + + result.AddOperand(invoke); + result.AddOperand(param); + result.AddOperand(paramSize); + result.AddOperand(paramAlign); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RetainEvent(Instruction eventObj) + { + Instruction result = NewInstruction(Op.OpRetainEvent); + + result.AddOperand(eventObj); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ReleaseEvent(Instruction eventObj) + { + Instruction result = NewInstruction(Op.OpReleaseEvent); + + result.AddOperand(eventObj); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction CreateUserEvent(Instruction resultType) + { + Instruction result = NewInstruction(Op.OpCreateUserEvent, GetNewId(), resultType); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction IsValidEvent(Instruction resultType, Instruction eventObj) + { + Instruction result = NewInstruction(Op.OpIsValidEvent, GetNewId(), resultType); + + result.AddOperand(eventObj); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction SetUserEventStatus(Instruction eventObj, Instruction status) + { + Instruction result = NewInstruction(Op.OpSetUserEventStatus); + + result.AddOperand(eventObj); + result.AddOperand(status); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction CaptureEventProfilingInfo(Instruction eventObj, Instruction profilingInfo, Instruction value) + { + Instruction result = NewInstruction(Op.OpCaptureEventProfilingInfo); + + result.AddOperand(eventObj); + result.AddOperand(profilingInfo); + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GetDefaultQueue(Instruction resultType) + { + Instruction result = NewInstruction(Op.OpGetDefaultQueue, GetNewId(), resultType); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction BuildNDRange(Instruction resultType, Instruction globalWorkSize, Instruction localWorkSize, Instruction globalWorkOffset) + { + Instruction result = NewInstruction(Op.OpBuildNDRange, GetNewId(), resultType); + + result.AddOperand(globalWorkSize); + result.AddOperand(localWorkSize); + result.AddOperand(globalWorkOffset); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GetKernelLocalSizeForSubgroupCount(Instruction resultType, Instruction subgroupCount, Instruction invoke, Instruction param, Instruction paramSize, Instruction paramAlign) + { + Instruction result = NewInstruction(Op.OpGetKernelLocalSizeForSubgroupCount, GetNewId(), resultType); + + result.AddOperand(subgroupCount); + result.AddOperand(invoke); + result.AddOperand(param); + result.AddOperand(paramSize); + result.AddOperand(paramAlign); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GetKernelMaxNumSubgroups(Instruction resultType, Instruction invoke, Instruction param, Instruction paramSize, Instruction paramAlign) + { + Instruction result = NewInstruction(Op.OpGetKernelMaxNumSubgroups, GetNewId(), resultType); + + result.AddOperand(invoke); + result.AddOperand(param); + result.AddOperand(paramSize); + result.AddOperand(paramAlign); + AddToFunctionDefinitions(result); + + return result; + } + + // Pipe + + public Instruction ReadPipe(Instruction resultType, Instruction pipe, Instruction pointer, Instruction packetSize, Instruction packetAlignment) + { + Instruction result = NewInstruction(Op.OpReadPipe, GetNewId(), resultType); + + result.AddOperand(pipe); + result.AddOperand(pointer); + result.AddOperand(packetSize); + result.AddOperand(packetAlignment); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction WritePipe(Instruction resultType, Instruction pipe, Instruction pointer, Instruction packetSize, Instruction packetAlignment) + { + Instruction result = NewInstruction(Op.OpWritePipe, GetNewId(), resultType); + + result.AddOperand(pipe); + result.AddOperand(pointer); + result.AddOperand(packetSize); + result.AddOperand(packetAlignment); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ReservedReadPipe(Instruction resultType, Instruction pipe, Instruction reserveId, Instruction index, Instruction pointer, Instruction packetSize, Instruction packetAlignment) + { + Instruction result = NewInstruction(Op.OpReservedReadPipe, GetNewId(), resultType); + + result.AddOperand(pipe); + result.AddOperand(reserveId); + result.AddOperand(index); + result.AddOperand(pointer); + result.AddOperand(packetSize); + result.AddOperand(packetAlignment); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ReservedWritePipe(Instruction resultType, Instruction pipe, Instruction reserveId, Instruction index, Instruction pointer, Instruction packetSize, Instruction packetAlignment) + { + Instruction result = NewInstruction(Op.OpReservedWritePipe, GetNewId(), resultType); + + result.AddOperand(pipe); + result.AddOperand(reserveId); + result.AddOperand(index); + result.AddOperand(pointer); + result.AddOperand(packetSize); + result.AddOperand(packetAlignment); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ReserveReadPipePackets(Instruction resultType, Instruction pipe, Instruction numPackets, Instruction packetSize, Instruction packetAlignment) + { + Instruction result = NewInstruction(Op.OpReserveReadPipePackets, GetNewId(), resultType); + + result.AddOperand(pipe); + result.AddOperand(numPackets); + result.AddOperand(packetSize); + result.AddOperand(packetAlignment); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ReserveWritePipePackets(Instruction resultType, Instruction pipe, Instruction numPackets, Instruction packetSize, Instruction packetAlignment) + { + Instruction result = NewInstruction(Op.OpReserveWritePipePackets, GetNewId(), resultType); + + result.AddOperand(pipe); + result.AddOperand(numPackets); + result.AddOperand(packetSize); + result.AddOperand(packetAlignment); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction CommitReadPipe(Instruction pipe, Instruction reserveId, Instruction packetSize, Instruction packetAlignment) + { + Instruction result = NewInstruction(Op.OpCommitReadPipe); + + result.AddOperand(pipe); + result.AddOperand(reserveId); + result.AddOperand(packetSize); + result.AddOperand(packetAlignment); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction CommitWritePipe(Instruction pipe, Instruction reserveId, Instruction packetSize, Instruction packetAlignment) + { + Instruction result = NewInstruction(Op.OpCommitWritePipe); + + result.AddOperand(pipe); + result.AddOperand(reserveId); + result.AddOperand(packetSize); + result.AddOperand(packetAlignment); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction IsValidReserveId(Instruction resultType, Instruction reserveId) + { + Instruction result = NewInstruction(Op.OpIsValidReserveId, GetNewId(), resultType); + + result.AddOperand(reserveId); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GetNumPipePackets(Instruction resultType, Instruction pipe, Instruction packetSize, Instruction packetAlignment) + { + Instruction result = NewInstruction(Op.OpGetNumPipePackets, GetNewId(), resultType); + + result.AddOperand(pipe); + result.AddOperand(packetSize); + result.AddOperand(packetAlignment); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GetMaxPipePackets(Instruction resultType, Instruction pipe, Instruction packetSize, Instruction packetAlignment) + { + Instruction result = NewInstruction(Op.OpGetMaxPipePackets, GetNewId(), resultType); + + result.AddOperand(pipe); + result.AddOperand(packetSize); + result.AddOperand(packetAlignment); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupReserveReadPipePackets(Instruction resultType, Instruction execution, Instruction pipe, Instruction numPackets, Instruction packetSize, Instruction packetAlignment) + { + Instruction result = NewInstruction(Op.OpGroupReserveReadPipePackets, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(pipe); + result.AddOperand(numPackets); + result.AddOperand(packetSize); + result.AddOperand(packetAlignment); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupReserveWritePipePackets(Instruction resultType, Instruction execution, Instruction pipe, Instruction numPackets, Instruction packetSize, Instruction packetAlignment) + { + Instruction result = NewInstruction(Op.OpGroupReserveWritePipePackets, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(pipe); + result.AddOperand(numPackets); + result.AddOperand(packetSize); + result.AddOperand(packetAlignment); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupCommitReadPipe(Instruction execution, Instruction pipe, Instruction reserveId, Instruction packetSize, Instruction packetAlignment) + { + Instruction result = NewInstruction(Op.OpGroupCommitReadPipe); + + result.AddOperand(execution); + result.AddOperand(pipe); + result.AddOperand(reserveId); + result.AddOperand(packetSize); + result.AddOperand(packetAlignment); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupCommitWritePipe(Instruction execution, Instruction pipe, Instruction reserveId, Instruction packetSize, Instruction packetAlignment) + { + Instruction result = NewInstruction(Op.OpGroupCommitWritePipe); + + result.AddOperand(execution); + result.AddOperand(pipe); + result.AddOperand(reserveId); + result.AddOperand(packetSize); + result.AddOperand(packetAlignment); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ConstantPipeStorage(Instruction resultType, LiteralInteger packetSize, LiteralInteger packetAlignment, LiteralInteger capacity) + { + Instruction result = NewInstruction(Op.OpConstantPipeStorage, GetNewId(), resultType); + + result.AddOperand(packetSize); + result.AddOperand(packetAlignment); + result.AddOperand(capacity); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction CreatePipeFromPipeStorage(Instruction resultType, Instruction pipeStorage) + { + Instruction result = NewInstruction(Op.OpCreatePipeFromPipeStorage, GetNewId(), resultType); + + result.AddOperand(pipeStorage); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ReadPipeBlockingINTEL(Instruction resultType, Instruction packetSize, Instruction packetAlignment) + { + Instruction result = NewInstruction(Op.OpReadPipeBlockingINTEL, GetNewId(), resultType); + + result.AddOperand(packetSize); + result.AddOperand(packetAlignment); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction WritePipeBlockingINTEL(Instruction resultType, Instruction packetSize, Instruction packetAlignment) + { + Instruction result = NewInstruction(Op.OpWritePipeBlockingINTEL, GetNewId(), resultType); + + result.AddOperand(packetSize); + result.AddOperand(packetAlignment); + AddToFunctionDefinitions(result); + + return result; + } + + // Non-Uniform + + public Instruction GroupNonUniformElect(Instruction resultType, Instruction execution) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformElect, GetNewId(), resultType); + + result.AddOperand(execution); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformAll(Instruction resultType, Instruction execution, Instruction predicate) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformAll, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(predicate); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformAny(Instruction resultType, Instruction execution, Instruction predicate) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformAny, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(predicate); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformAllEqual(Instruction resultType, Instruction execution, Instruction value) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformAllEqual, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformBroadcast(Instruction resultType, Instruction execution, Instruction value, Instruction id) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformBroadcast, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(value); + result.AddOperand(id); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformBroadcastFirst(Instruction resultType, Instruction execution, Instruction value) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformBroadcastFirst, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformBallot(Instruction resultType, Instruction execution, Instruction predicate) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformBallot, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(predicate); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformInverseBallot(Instruction resultType, Instruction execution, Instruction value) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformInverseBallot, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformBallotBitExtract(Instruction resultType, Instruction execution, Instruction value, Instruction index) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformBallotBitExtract, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(value); + result.AddOperand(index); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformBallotBitCount(Instruction resultType, Instruction execution, GroupOperation operation, Instruction value) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformBallotBitCount, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformBallotFindLSB(Instruction resultType, Instruction execution, Instruction value) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformBallotFindLSB, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformBallotFindMSB(Instruction resultType, Instruction execution, Instruction value) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformBallotFindMSB, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformShuffle(Instruction resultType, Instruction execution, Instruction value, Instruction id) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformShuffle, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(value); + result.AddOperand(id); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformShuffleXor(Instruction resultType, Instruction execution, Instruction value, Instruction mask) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformShuffleXor, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(value); + result.AddOperand(mask); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformShuffleUp(Instruction resultType, Instruction execution, Instruction value, Instruction delta) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformShuffleUp, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(value); + result.AddOperand(delta); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformShuffleDown(Instruction resultType, Instruction execution, Instruction value, Instruction delta) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformShuffleDown, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(value); + result.AddOperand(delta); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformIAdd(Instruction resultType, Instruction execution, GroupOperation operation, Instruction value, Instruction clusterSize = null) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformIAdd, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(value); + if (clusterSize != null) + { + result.AddOperand(clusterSize); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformFAdd(Instruction resultType, Instruction execution, GroupOperation operation, Instruction value, Instruction clusterSize = null) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformFAdd, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(value); + if (clusterSize != null) + { + result.AddOperand(clusterSize); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformIMul(Instruction resultType, Instruction execution, GroupOperation operation, Instruction value, Instruction clusterSize = null) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformIMul, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(value); + if (clusterSize != null) + { + result.AddOperand(clusterSize); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformFMul(Instruction resultType, Instruction execution, GroupOperation operation, Instruction value, Instruction clusterSize = null) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformFMul, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(value); + if (clusterSize != null) + { + result.AddOperand(clusterSize); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformSMin(Instruction resultType, Instruction execution, GroupOperation operation, Instruction value, Instruction clusterSize = null) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformSMin, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(value); + if (clusterSize != null) + { + result.AddOperand(clusterSize); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformUMin(Instruction resultType, Instruction execution, GroupOperation operation, Instruction value, Instruction clusterSize = null) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformUMin, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(value); + if (clusterSize != null) + { + result.AddOperand(clusterSize); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformFMin(Instruction resultType, Instruction execution, GroupOperation operation, Instruction value, Instruction clusterSize = null) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformFMin, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(value); + if (clusterSize != null) + { + result.AddOperand(clusterSize); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformSMax(Instruction resultType, Instruction execution, GroupOperation operation, Instruction value, Instruction clusterSize = null) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformSMax, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(value); + if (clusterSize != null) + { + result.AddOperand(clusterSize); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformUMax(Instruction resultType, Instruction execution, GroupOperation operation, Instruction value, Instruction clusterSize = null) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformUMax, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(value); + if (clusterSize != null) + { + result.AddOperand(clusterSize); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformFMax(Instruction resultType, Instruction execution, GroupOperation operation, Instruction value, Instruction clusterSize = null) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformFMax, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(value); + if (clusterSize != null) + { + result.AddOperand(clusterSize); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformBitwiseAnd(Instruction resultType, Instruction execution, GroupOperation operation, Instruction value, Instruction clusterSize = null) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformBitwiseAnd, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(value); + if (clusterSize != null) + { + result.AddOperand(clusterSize); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformBitwiseOr(Instruction resultType, Instruction execution, GroupOperation operation, Instruction value, Instruction clusterSize = null) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformBitwiseOr, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(value); + if (clusterSize != null) + { + result.AddOperand(clusterSize); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformBitwiseXor(Instruction resultType, Instruction execution, GroupOperation operation, Instruction value, Instruction clusterSize = null) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformBitwiseXor, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(value); + if (clusterSize != null) + { + result.AddOperand(clusterSize); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformLogicalAnd(Instruction resultType, Instruction execution, GroupOperation operation, Instruction value, Instruction clusterSize = null) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformLogicalAnd, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(value); + if (clusterSize != null) + { + result.AddOperand(clusterSize); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformLogicalOr(Instruction resultType, Instruction execution, GroupOperation operation, Instruction value, Instruction clusterSize = null) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformLogicalOr, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(value); + if (clusterSize != null) + { + result.AddOperand(clusterSize); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformLogicalXor(Instruction resultType, Instruction execution, GroupOperation operation, Instruction value, Instruction clusterSize = null) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformLogicalXor, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(operation); + result.AddOperand(value); + if (clusterSize != null) + { + result.AddOperand(clusterSize); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformQuadBroadcast(Instruction resultType, Instruction execution, Instruction value, Instruction index) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformQuadBroadcast, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(value); + result.AddOperand(index); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformQuadSwap(Instruction resultType, Instruction execution, Instruction value, Instruction direction) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformQuadSwap, GetNewId(), resultType); + + result.AddOperand(execution); + result.AddOperand(value); + result.AddOperand(direction); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction GroupNonUniformPartitionNV(Instruction resultType, Instruction value) + { + Instruction result = NewInstruction(Op.OpGroupNonUniformPartitionNV, GetNewId(), resultType); + + result.AddOperand(value); + AddToFunctionDefinitions(result); + + return result; + } + + // Reserved + + public Instruction TraceRayKHR(Instruction accel, Instruction rayFlags, Instruction cullMask, Instruction sBTOffset, Instruction sBTStride, Instruction missIndex, Instruction rayOrigin, Instruction rayTmin, Instruction rayDirection, Instruction rayTmax, Instruction payload) + { + Instruction result = NewInstruction(Op.OpTraceRayKHR); + + result.AddOperand(accel); + result.AddOperand(rayFlags); + result.AddOperand(cullMask); + result.AddOperand(sBTOffset); + result.AddOperand(sBTStride); + result.AddOperand(missIndex); + result.AddOperand(rayOrigin); + result.AddOperand(rayTmin); + result.AddOperand(rayDirection); + result.AddOperand(rayTmax); + result.AddOperand(payload); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ExecuteCallableKHR(Instruction sBTIndex, Instruction callableData) + { + Instruction result = NewInstruction(Op.OpExecuteCallableKHR); + + result.AddOperand(sBTIndex); + result.AddOperand(callableData); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ConvertUToAccelerationStructureKHR(Instruction resultType, Instruction accel) + { + Instruction result = NewInstruction(Op.OpConvertUToAccelerationStructureKHR, GetNewId(), resultType); + + result.AddOperand(accel); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction IgnoreIntersectionKHR() + { + Instruction result = NewInstruction(Op.OpIgnoreIntersectionKHR); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction TerminateRayKHR() + { + Instruction result = NewInstruction(Op.OpTerminateRayKHR); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction TypeRayQueryKHR() + { + Instruction result = NewInstruction(Op.OpTypeRayQueryKHR, GetNewId()); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryInitializeKHR(Instruction rayQuery, Instruction accel, Instruction rayFlags, Instruction cullMask, Instruction rayOrigin, Instruction rayTMin, Instruction rayDirection, Instruction rayTMax) + { + Instruction result = NewInstruction(Op.OpRayQueryInitializeKHR); + + result.AddOperand(rayQuery); + result.AddOperand(accel); + result.AddOperand(rayFlags); + result.AddOperand(cullMask); + result.AddOperand(rayOrigin); + result.AddOperand(rayTMin); + result.AddOperand(rayDirection); + result.AddOperand(rayTMax); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryTerminateKHR(Instruction rayQuery) + { + Instruction result = NewInstruction(Op.OpRayQueryTerminateKHR); + + result.AddOperand(rayQuery); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryGenerateIntersectionKHR(Instruction rayQuery, Instruction hitT) + { + Instruction result = NewInstruction(Op.OpRayQueryGenerateIntersectionKHR); + + result.AddOperand(rayQuery); + result.AddOperand(hitT); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryConfirmIntersectionKHR(Instruction rayQuery) + { + Instruction result = NewInstruction(Op.OpRayQueryConfirmIntersectionKHR); + + result.AddOperand(rayQuery); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryProceedKHR(Instruction resultType, Instruction rayQuery) + { + Instruction result = NewInstruction(Op.OpRayQueryProceedKHR, GetNewId(), resultType); + + result.AddOperand(rayQuery); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryGetIntersectionTypeKHR(Instruction resultType, Instruction rayQuery, Instruction intersection) + { + Instruction result = NewInstruction(Op.OpRayQueryGetIntersectionTypeKHR, GetNewId(), resultType); + + result.AddOperand(rayQuery); + result.AddOperand(intersection); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FragmentMaskFetchAMD(Instruction resultType, Instruction image, Instruction coordinate) + { + Instruction result = NewInstruction(Op.OpFragmentMaskFetchAMD, GetNewId(), resultType); + + result.AddOperand(image); + result.AddOperand(coordinate); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FragmentFetchAMD(Instruction resultType, Instruction image, Instruction coordinate, Instruction fragmentIndex) + { + Instruction result = NewInstruction(Op.OpFragmentFetchAMD, GetNewId(), resultType); + + result.AddOperand(image); + result.AddOperand(coordinate); + result.AddOperand(fragmentIndex); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ReadClockKHR(Instruction resultType, Instruction execution) + { + Instruction result = NewInstruction(Op.OpReadClockKHR, GetNewId(), resultType); + + result.AddOperand(execution); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction WritePackedPrimitiveIndices4x8NV(Instruction indexOffset, Instruction packedIndices) + { + Instruction result = NewInstruction(Op.OpWritePackedPrimitiveIndices4x8NV); + + result.AddOperand(indexOffset); + result.AddOperand(packedIndices); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ReportIntersectionNV(Instruction resultType, Instruction hit, Instruction hitKind) + { + Instruction result = NewInstruction(Op.OpReportIntersectionNV, GetNewId(), resultType); + + result.AddOperand(hit); + result.AddOperand(hitKind); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ReportIntersectionKHR(Instruction resultType, Instruction hit, Instruction hitKind) + { + Instruction result = NewInstruction(Op.OpReportIntersectionKHR, GetNewId(), resultType); + + result.AddOperand(hit); + result.AddOperand(hitKind); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction IgnoreIntersectionNV() + { + Instruction result = NewInstruction(Op.OpIgnoreIntersectionNV); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction TerminateRayNV() + { + Instruction result = NewInstruction(Op.OpTerminateRayNV); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction TraceNV(Instruction accel, Instruction rayFlags, Instruction cullMask, Instruction sBTOffset, Instruction sBTStride, Instruction missIndex, Instruction rayOrigin, Instruction rayTmin, Instruction rayDirection, Instruction rayTmax, Instruction payloadId) + { + Instruction result = NewInstruction(Op.OpTraceNV); + + result.AddOperand(accel); + result.AddOperand(rayFlags); + result.AddOperand(cullMask); + result.AddOperand(sBTOffset); + result.AddOperand(sBTStride); + result.AddOperand(missIndex); + result.AddOperand(rayOrigin); + result.AddOperand(rayTmin); + result.AddOperand(rayDirection); + result.AddOperand(rayTmax); + result.AddOperand(payloadId); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction TypeAccelerationStructureNV() + { + Instruction result = NewInstruction(Op.OpTypeAccelerationStructureNV, GetNewId()); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction TypeAccelerationStructureKHR() + { + Instruction result = NewInstruction(Op.OpTypeAccelerationStructureKHR, GetNewId()); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ExecuteCallableNV(Instruction sBTIndex, Instruction callableDataId) + { + Instruction result = NewInstruction(Op.OpExecuteCallableNV); + + result.AddOperand(sBTIndex); + result.AddOperand(callableDataId); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction TypeCooperativeMatrixNV(Instruction componentType, Instruction execution, Instruction rows, Instruction columns) + { + Instruction result = NewInstruction(Op.OpTypeCooperativeMatrixNV, GetNewId()); + + result.AddOperand(componentType); + result.AddOperand(execution); + result.AddOperand(rows); + result.AddOperand(columns); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction CooperativeMatrixLoadNV(Instruction resultType, Instruction pointer, Instruction stride, Instruction columnMajor, MemoryAccessMask memoryAccess = (MemoryAccessMask)int.MaxValue) + { + Instruction result = NewInstruction(Op.OpCooperativeMatrixLoadNV, GetNewId(), resultType); + + result.AddOperand(pointer); + result.AddOperand(stride); + result.AddOperand(columnMajor); + if (memoryAccess != (MemoryAccessMask)int.MaxValue) + { + result.AddOperand(memoryAccess); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction CooperativeMatrixStoreNV(Instruction pointer, Instruction obj, Instruction stride, Instruction columnMajor, MemoryAccessMask memoryAccess = (MemoryAccessMask)int.MaxValue) + { + Instruction result = NewInstruction(Op.OpCooperativeMatrixStoreNV); + + result.AddOperand(pointer); + result.AddOperand(obj); + result.AddOperand(stride); + result.AddOperand(columnMajor); + if (memoryAccess != (MemoryAccessMask)int.MaxValue) + { + result.AddOperand(memoryAccess); + } + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction CooperativeMatrixMulAddNV(Instruction resultType, Instruction a, Instruction b, Instruction c) + { + Instruction result = NewInstruction(Op.OpCooperativeMatrixMulAddNV, GetNewId(), resultType); + + result.AddOperand(a); + result.AddOperand(b); + result.AddOperand(c); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction CooperativeMatrixLengthNV(Instruction resultType, Instruction type) + { + Instruction result = NewInstruction(Op.OpCooperativeMatrixLengthNV, GetNewId(), resultType); + + result.AddOperand(type); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction BeginInvocationInterlockEXT() + { + Instruction result = NewInstruction(Op.OpBeginInvocationInterlockEXT); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction EndInvocationInterlockEXT() + { + Instruction result = NewInstruction(Op.OpEndInvocationInterlockEXT); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction DemoteToHelperInvocationEXT() + { + Instruction result = NewInstruction(Op.OpDemoteToHelperInvocationEXT); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction IsHelperInvocationEXT(Instruction resultType) + { + Instruction result = NewInstruction(Op.OpIsHelperInvocationEXT, GetNewId(), resultType); + + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction UCountLeadingZerosINTEL(Instruction resultType, Instruction operand) + { + Instruction result = NewInstruction(Op.OpUCountLeadingZerosINTEL, GetNewId(), resultType); + + result.AddOperand(operand); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction UCountTrailingZerosINTEL(Instruction resultType, Instruction operand) + { + Instruction result = NewInstruction(Op.OpUCountTrailingZerosINTEL, GetNewId(), resultType); + + result.AddOperand(operand); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AbsISubINTEL(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpAbsISubINTEL, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction AbsUSubINTEL(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpAbsUSubINTEL, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction IAddSatINTEL(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpIAddSatINTEL, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction UAddSatINTEL(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpUAddSatINTEL, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction IAverageINTEL(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpIAverageINTEL, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction UAverageINTEL(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpUAverageINTEL, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction IAverageRoundedINTEL(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpIAverageRoundedINTEL, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction UAverageRoundedINTEL(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpUAverageRoundedINTEL, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction ISubSatINTEL(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpISubSatINTEL, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction USubSatINTEL(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpUSubSatINTEL, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction IMul32x16INTEL(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpIMul32x16INTEL, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction UMul32x16INTEL(Instruction resultType, Instruction operand1, Instruction operand2) + { + Instruction result = NewInstruction(Op.OpUMul32x16INTEL, GetNewId(), resultType); + + result.AddOperand(operand1); + result.AddOperand(operand2); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction LoopControlINTEL(params ReadOnlySpan loopControlParameters) + { + Instruction result = NewInstruction(Op.OpLoopControlINTEL); + + result.AddOperand(loopControlParameters); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction FPGARegINTEL(Instruction resultType, Instruction resultObj, Instruction input) + { + Instruction result = NewInstruction(Op.OpFPGARegINTEL, GetNewId(), resultType); + + result.AddOperand(resultObj); + result.AddOperand(input); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryGetRayTMinKHR(Instruction resultType, Instruction rayQuery) + { + Instruction result = NewInstruction(Op.OpRayQueryGetRayTMinKHR, GetNewId(), resultType); + + result.AddOperand(rayQuery); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryGetRayFlagsKHR(Instruction resultType, Instruction rayQuery) + { + Instruction result = NewInstruction(Op.OpRayQueryGetRayFlagsKHR, GetNewId(), resultType); + + result.AddOperand(rayQuery); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryGetIntersectionTKHR(Instruction resultType, Instruction rayQuery, Instruction intersection) + { + Instruction result = NewInstruction(Op.OpRayQueryGetIntersectionTKHR, GetNewId(), resultType); + + result.AddOperand(rayQuery); + result.AddOperand(intersection); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryGetIntersectionInstanceCustomIndexKHR(Instruction resultType, Instruction rayQuery, Instruction intersection) + { + Instruction result = NewInstruction(Op.OpRayQueryGetIntersectionInstanceCustomIndexKHR, GetNewId(), resultType); + + result.AddOperand(rayQuery); + result.AddOperand(intersection); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryGetIntersectionInstanceIdKHR(Instruction resultType, Instruction rayQuery, Instruction intersection) + { + Instruction result = NewInstruction(Op.OpRayQueryGetIntersectionInstanceIdKHR, GetNewId(), resultType); + + result.AddOperand(rayQuery); + result.AddOperand(intersection); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryGetIntersectionInstanceShaderBindingTableRecordOffsetKHR(Instruction resultType, Instruction rayQuery, Instruction intersection) + { + Instruction result = NewInstruction(Op.OpRayQueryGetIntersectionInstanceShaderBindingTableRecordOffsetKHR, GetNewId(), resultType); + + result.AddOperand(rayQuery); + result.AddOperand(intersection); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryGetIntersectionGeometryIndexKHR(Instruction resultType, Instruction rayQuery, Instruction intersection) + { + Instruction result = NewInstruction(Op.OpRayQueryGetIntersectionGeometryIndexKHR, GetNewId(), resultType); + + result.AddOperand(rayQuery); + result.AddOperand(intersection); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryGetIntersectionPrimitiveIndexKHR(Instruction resultType, Instruction rayQuery, Instruction intersection) + { + Instruction result = NewInstruction(Op.OpRayQueryGetIntersectionPrimitiveIndexKHR, GetNewId(), resultType); + + result.AddOperand(rayQuery); + result.AddOperand(intersection); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryGetIntersectionBarycentricsKHR(Instruction resultType, Instruction rayQuery, Instruction intersection) + { + Instruction result = NewInstruction(Op.OpRayQueryGetIntersectionBarycentricsKHR, GetNewId(), resultType); + + result.AddOperand(rayQuery); + result.AddOperand(intersection); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryGetIntersectionFrontFaceKHR(Instruction resultType, Instruction rayQuery, Instruction intersection) + { + Instruction result = NewInstruction(Op.OpRayQueryGetIntersectionFrontFaceKHR, GetNewId(), resultType); + + result.AddOperand(rayQuery); + result.AddOperand(intersection); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryGetIntersectionCandidateAABBOpaqueKHR(Instruction resultType, Instruction rayQuery) + { + Instruction result = NewInstruction(Op.OpRayQueryGetIntersectionCandidateAABBOpaqueKHR, GetNewId(), resultType); + + result.AddOperand(rayQuery); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryGetIntersectionObjectRayDirectionKHR(Instruction resultType, Instruction rayQuery, Instruction intersection) + { + Instruction result = NewInstruction(Op.OpRayQueryGetIntersectionObjectRayDirectionKHR, GetNewId(), resultType); + + result.AddOperand(rayQuery); + result.AddOperand(intersection); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryGetIntersectionObjectRayOriginKHR(Instruction resultType, Instruction rayQuery, Instruction intersection) + { + Instruction result = NewInstruction(Op.OpRayQueryGetIntersectionObjectRayOriginKHR, GetNewId(), resultType); + + result.AddOperand(rayQuery); + result.AddOperand(intersection); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryGetWorldRayDirectionKHR(Instruction resultType, Instruction rayQuery) + { + Instruction result = NewInstruction(Op.OpRayQueryGetWorldRayDirectionKHR, GetNewId(), resultType); + + result.AddOperand(rayQuery); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryGetWorldRayOriginKHR(Instruction resultType, Instruction rayQuery) + { + Instruction result = NewInstruction(Op.OpRayQueryGetWorldRayOriginKHR, GetNewId(), resultType); + + result.AddOperand(rayQuery); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryGetIntersectionObjectToWorldKHR(Instruction resultType, Instruction rayQuery, Instruction intersection) + { + Instruction result = NewInstruction(Op.OpRayQueryGetIntersectionObjectToWorldKHR, GetNewId(), resultType); + + result.AddOperand(rayQuery); + result.AddOperand(intersection); + AddToFunctionDefinitions(result); + + return result; + } + + public Instruction RayQueryGetIntersectionWorldToObjectKHR(Instruction resultType, Instruction rayQuery, Instruction intersection) + { + Instruction result = NewInstruction(Op.OpRayQueryGetIntersectionWorldToObjectKHR, GetNewId(), resultType); + + result.AddOperand(rayQuery); + result.AddOperand(intersection); + AddToFunctionDefinitions(result); + + return result; + } + } +} diff --git a/third_party/maxwell_shader/src/Spv.Generator/Autogenerated/GlslStd450Grammar.cs b/third_party/maxwell_shader/src/Spv.Generator/Autogenerated/GlslStd450Grammar.cs new file mode 100644 index 0000000..7199969 --- /dev/null +++ b/third_party/maxwell_shader/src/Spv.Generator/Autogenerated/GlslStd450Grammar.cs @@ -0,0 +1,438 @@ +// AUTOGENERATED: DO NOT EDIT +// Last update date: 2021-01-06 23:02:26.955269 +#region Grammar License +// Copyright (c) 2014-2016 The Khronos Group Inc. +// +// Permission is hereby granted, free of charge, to any person obtaining a copy +// of this software and/or associated documentation files (the "Materials"), +// to deal in the Materials without restriction, including without limitation +// the rights to use, copy, modify, merge, publish, distribute, sublicense, +// and/or sell copies of the Materials, and to permit persons to whom the +// Materials are furnished to do so, subject to the following conditions: +// +// The above copyright notice and this permission notice shall be included in +// all copies or substantial portions of the Materials. +// +// MODIFICATIONS TO THIS FILE MAY MEAN IT NO LONGER ACCURATELY REFLECTS KHRONOS +// STANDARDS. THE UNMODIFIED, NORMATIVE VERSIONS OF KHRONOS SPECIFICATIONS AND +// HEADER INFORMATION ARE LOCATED AT https://www.khronos.org/registry/ +// +// THE MATERIALS ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS +// OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +// THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM,OUT OF OR IN CONNECTION WITH THE MATERIALS OR THE USE OR OTHER DEALINGS +// IN THE MATERIALS. +#endregion + +namespace Spv.Generator +{ + public partial class Module + { + public Instruction GlslRound(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 1, x); + } + + public Instruction GlslRoundEven(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 2, x); + } + + public Instruction GlslTrunc(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 3, x); + } + + public Instruction GlslFAbs(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 4, x); + } + + public Instruction GlslSAbs(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 5, x); + } + + public Instruction GlslFSign(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 6, x); + } + + public Instruction GlslSSign(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 7, x); + } + + public Instruction GlslFloor(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 8, x); + } + + public Instruction GlslCeil(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 9, x); + } + + public Instruction GlslFract(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 10, x); + } + + public Instruction GlslRadians(Instruction resultType, Instruction degrees) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 11, degrees); + } + + public Instruction GlslDegrees(Instruction resultType, Instruction radians) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 12, radians); + } + + public Instruction GlslSin(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 13, x); + } + + public Instruction GlslCos(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 14, x); + } + + public Instruction GlslTan(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 15, x); + } + + public Instruction GlslAsin(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 16, x); + } + + public Instruction GlslAcos(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 17, x); + } + + public Instruction GlslAtan(Instruction resultType, Instruction y_over_x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 18, y_over_x); + } + + public Instruction GlslSinh(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 19, x); + } + + public Instruction GlslCosh(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 20, x); + } + + public Instruction GlslTanh(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 21, x); + } + + public Instruction GlslAsinh(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 22, x); + } + + public Instruction GlslAcosh(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 23, x); + } + + public Instruction GlslAtanh(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 24, x); + } + + public Instruction GlslAtan2(Instruction resultType, Instruction y, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 25, y, x); + } + + public Instruction GlslPow(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 26, x, y); + } + + public Instruction GlslExp(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 27, x); + } + + public Instruction GlslLog(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 28, x); + } + + public Instruction GlslExp2(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 29, x); + } + + public Instruction GlslLog2(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 30, x); + } + + public Instruction GlslSqrt(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 31, x); + } + + public Instruction GlslInverseSqrt(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 32, x); + } + + public Instruction GlslDeterminant(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 33, x); + } + + public Instruction GlslMatrixInverse(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 34, x); + } + + public Instruction GlslModf(Instruction resultType, Instruction x, Instruction i) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 35, x, i); + } + + public Instruction GlslModfStruct(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 36, x); + } + + public Instruction GlslFMin(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 37, x, y); + } + + public Instruction GlslUMin(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 38, x, y); + } + + public Instruction GlslSMin(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 39, x, y); + } + + public Instruction GlslFMax(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 40, x, y); + } + + public Instruction GlslUMax(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 41, x, y); + } + + public Instruction GlslSMax(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 42, x, y); + } + + public Instruction GlslFClamp(Instruction resultType, Instruction x, Instruction minVal, Instruction maxVal) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 43, x, minVal, maxVal); + } + + public Instruction GlslUClamp(Instruction resultType, Instruction x, Instruction minVal, Instruction maxVal) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 44, x, minVal, maxVal); + } + + public Instruction GlslSClamp(Instruction resultType, Instruction x, Instruction minVal, Instruction maxVal) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 45, x, minVal, maxVal); + } + + public Instruction GlslFMix(Instruction resultType, Instruction x, Instruction y, Instruction a) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 46, x, y, a); + } + + public Instruction GlslIMix(Instruction resultType, Instruction x, Instruction y, Instruction a) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 47, x, y, a); + } + + public Instruction GlslStep(Instruction resultType, Instruction edge, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 48, edge, x); + } + + public Instruction GlslSmoothStep(Instruction resultType, Instruction edge0, Instruction edge1, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 49, edge0, edge1, x); + } + + public Instruction GlslFma(Instruction resultType, Instruction a, Instruction b, Instruction c) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 50, a, b, c); + } + + public Instruction GlslFrexp(Instruction resultType, Instruction x, Instruction exp) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 51, x, exp); + } + + public Instruction GlslFrexpStruct(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 52, x); + } + + public Instruction GlslLdexp(Instruction resultType, Instruction x, Instruction exp) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 53, x, exp); + } + + public Instruction GlslPackSnorm4x8(Instruction resultType, Instruction v) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 54, v); + } + + public Instruction GlslPackUnorm4x8(Instruction resultType, Instruction v) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 55, v); + } + + public Instruction GlslPackSnorm2x16(Instruction resultType, Instruction v) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 56, v); + } + + public Instruction GlslPackUnorm2x16(Instruction resultType, Instruction v) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 57, v); + } + + public Instruction GlslPackHalf2x16(Instruction resultType, Instruction v) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 58, v); + } + + public Instruction GlslPackDouble2x32(Instruction resultType, Instruction v) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 59, v); + } + + public Instruction GlslUnpackSnorm2x16(Instruction resultType, Instruction p) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 60, p); + } + + public Instruction GlslUnpackUnorm2x16(Instruction resultType, Instruction p) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 61, p); + } + + public Instruction GlslUnpackHalf2x16(Instruction resultType, Instruction v) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 62, v); + } + + public Instruction GlslUnpackSnorm4x8(Instruction resultType, Instruction p) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 63, p); + } + + public Instruction GlslUnpackUnorm4x8(Instruction resultType, Instruction p) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 64, p); + } + + public Instruction GlslUnpackDouble2x32(Instruction resultType, Instruction v) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 65, v); + } + + public Instruction GlslLength(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 66, x); + } + + public Instruction GlslDistance(Instruction resultType, Instruction p0, Instruction p1) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 67, p0, p1); + } + + public Instruction GlslCross(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 68, x, y); + } + + public Instruction GlslNormalize(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 69, x); + } + + public Instruction GlslFaceForward(Instruction resultType, Instruction n, Instruction i, Instruction nref) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 70, n, i, nref); + } + + public Instruction GlslReflect(Instruction resultType, Instruction i, Instruction n) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 71, i, n); + } + + public Instruction GlslRefract(Instruction resultType, Instruction i, Instruction n, Instruction eta) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 72, i, n, eta); + } + + public Instruction GlslFindILsb(Instruction resultType, Instruction value) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 73, value); + } + + public Instruction GlslFindSMsb(Instruction resultType, Instruction value) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 74, value); + } + + public Instruction GlslFindUMsb(Instruction resultType, Instruction value) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 75, value); + } + + public Instruction GlslInterpolateAtCentroid(Instruction resultType, Instruction interpolant) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 76, interpolant); + } + + public Instruction GlslInterpolateAtSample(Instruction resultType, Instruction interpolant, Instruction sample) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 77, interpolant, sample); + } + + public Instruction GlslInterpolateAtOffset(Instruction resultType, Instruction interpolant, Instruction offset) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 78, interpolant, offset); + } + + public Instruction GlslNMin(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 79, x, y); + } + + public Instruction GlslNMax(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 80, x, y); + } + + public Instruction GlslNClamp(Instruction resultType, Instruction x, Instruction minVal, Instruction maxVal) + { + return ExtInst(resultType, AddExtInstImport("GLSL.std.450"), 81, x, minVal, maxVal); + } + } +} diff --git a/third_party/maxwell_shader/src/Spv.Generator/Autogenerated/OpenClGrammar.cs b/third_party/maxwell_shader/src/Spv.Generator/Autogenerated/OpenClGrammar.cs new file mode 100644 index 0000000..aa88283 --- /dev/null +++ b/third_party/maxwell_shader/src/Spv.Generator/Autogenerated/OpenClGrammar.cs @@ -0,0 +1,840 @@ +// AUTOGENERATED: DO NOT EDIT +// Last update date: 2021-01-06 23:02:27.020534 +#region Grammar License +// Copyright (c) 2014-2016 The Khronos Group Inc. +// +// Permission is hereby granted, free of charge, to any person obtaining a copy +// of this software and/or associated documentation files (the "Materials"), +// to deal in the Materials without restriction, including without limitation +// the rights to use, copy, modify, merge, publish, distribute, sublicense, +// and/or sell copies of the Materials, and to permit persons to whom the +// Materials are furnished to do so, subject to the following conditions: +// +// The above copyright notice and this permission notice shall be included in +// all copies or substantial portions of the Materials. +// +// MODIFICATIONS TO THIS FILE MAY MEAN IT NO LONGER ACCURATELY REFLECTS KHRONOS +// STANDARDS. THE UNMODIFIED, NORMATIVE VERSIONS OF KHRONOS SPECIFICATIONS AND +// HEADER INFORMATION ARE LOCATED AT https://www.khronos.org/registry/ +// +// THE MATERIALS ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS +// OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +// THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM,OUT OF OR IN CONNECTION WITH THE MATERIALS OR THE USE OR OTHER DEALINGS +// IN THE MATERIALS. +#endregion + +using static Spv.Specification; + +namespace Spv.Generator +{ + public partial class Module + { + public Instruction OpenClAcos(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 0, x); + } + + public Instruction OpenClAcosh(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 1, x); + } + + public Instruction OpenClAcospi(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 2, x); + } + + public Instruction OpenClAsin(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 3, x); + } + + public Instruction OpenClAsinh(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 4, x); + } + + public Instruction OpenClAsinpi(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 5, x); + } + + public Instruction OpenClAtan(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 6, x); + } + + public Instruction OpenClAtan2(Instruction resultType, Instruction y, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 7, y, x); + } + + public Instruction OpenClAtanh(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 8, x); + } + + public Instruction OpenClAtanpi(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 9, x); + } + + public Instruction OpenClAtan2pi(Instruction resultType, Instruction y, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 10, y, x); + } + + public Instruction OpenClCbrt(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 11, x); + } + + public Instruction OpenClCeil(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 12, x); + } + + public Instruction OpenClCopysign(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 13, x, y); + } + + public Instruction OpenClCos(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 14, x); + } + + public Instruction OpenClCosh(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 15, x); + } + + public Instruction OpenClCospi(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 16, x); + } + + public Instruction OpenClErfc(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 17, x); + } + + public Instruction OpenClErf(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 18, x); + } + + public Instruction OpenClExp(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 19, x); + } + + public Instruction OpenClExp2(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 20, x); + } + + public Instruction OpenClExp10(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 21, x); + } + + public Instruction OpenClExpm1(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 22, x); + } + + public Instruction OpenClFabs(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 23, x); + } + + public Instruction OpenClFdim(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 24, x, y); + } + + public Instruction OpenClFloor(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 25, x); + } + + public Instruction OpenClFma(Instruction resultType, Instruction a, Instruction b, Instruction c) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 26, a, b, c); + } + + public Instruction OpenClFmax(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 27, x, y); + } + + public Instruction OpenClFmin(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 28, x, y); + } + + public Instruction OpenClFmod(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 29, x, y); + } + + public Instruction OpenClFract(Instruction resultType, Instruction x, Instruction ptr) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 30, x, ptr); + } + + public Instruction OpenClFrexp(Instruction resultType, Instruction x, Instruction exp) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 31, x, exp); + } + + public Instruction OpenClHypot(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 32, x, y); + } + + public Instruction OpenClIlogb(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 33, x); + } + + public Instruction OpenClLdexp(Instruction resultType, Instruction x, Instruction k) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 34, x, k); + } + + public Instruction OpenClLgamma(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 35, x); + } + + public Instruction OpenClLgamma_r(Instruction resultType, Instruction x, Instruction signp) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 36, x, signp); + } + + public Instruction OpenClLog(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 37, x); + } + + public Instruction OpenClLog2(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 38, x); + } + + public Instruction OpenClLog10(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 39, x); + } + + public Instruction OpenClLog1p(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 40, x); + } + + public Instruction OpenClLogb(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 41, x); + } + + public Instruction OpenClMad(Instruction resultType, Instruction a, Instruction b, Instruction c) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 42, a, b, c); + } + + public Instruction OpenClMaxmag(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 43, x, y); + } + + public Instruction OpenClMinmag(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 44, x, y); + } + + public Instruction OpenClModf(Instruction resultType, Instruction x, Instruction iptr) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 45, x, iptr); + } + + public Instruction OpenClNan(Instruction resultType, Instruction nancode) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 46, nancode); + } + + public Instruction OpenClNextafter(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 47, x, y); + } + + public Instruction OpenClPow(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 48, x, y); + } + + public Instruction OpenClPown(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 49, x, y); + } + + public Instruction OpenClPowr(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 50, x, y); + } + + public Instruction OpenClRemainder(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 51, x, y); + } + + public Instruction OpenClRemquo(Instruction resultType, Instruction x, Instruction y, Instruction quo) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 52, x, y, quo); + } + + public Instruction OpenClRint(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 53, x); + } + + public Instruction OpenClRootn(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 54, x, y); + } + + public Instruction OpenClRound(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 55, x); + } + + public Instruction OpenClRsqrt(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 56, x); + } + + public Instruction OpenClSin(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 57, x); + } + + public Instruction OpenClSincos(Instruction resultType, Instruction x, Instruction cosval) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 58, x, cosval); + } + + public Instruction OpenClSinh(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 59, x); + } + + public Instruction OpenClSinpi(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 60, x); + } + + public Instruction OpenClSqrt(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 61, x); + } + + public Instruction OpenClTan(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 62, x); + } + + public Instruction OpenClTanh(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 63, x); + } + + public Instruction OpenClTanpi(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 64, x); + } + + public Instruction OpenClTgamma(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 65, x); + } + + public Instruction OpenClTrunc(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 66, x); + } + + public Instruction OpenClHalf_cos(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 67, x); + } + + public Instruction OpenClHalf_divide(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 68, x, y); + } + + public Instruction OpenClHalf_exp(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 69, x); + } + + public Instruction OpenClHalf_exp2(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 70, x); + } + + public Instruction OpenClHalf_exp10(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 71, x); + } + + public Instruction OpenClHalf_log(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 72, x); + } + + public Instruction OpenClHalf_log2(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 73, x); + } + + public Instruction OpenClHalf_log10(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 74, x); + } + + public Instruction OpenClHalf_powr(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 75, x, y); + } + + public Instruction OpenClHalf_recip(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 76, x); + } + + public Instruction OpenClHalf_rsqrt(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 77, x); + } + + public Instruction OpenClHalf_sin(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 78, x); + } + + public Instruction OpenClHalf_sqrt(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 79, x); + } + + public Instruction OpenClHalf_tan(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 80, x); + } + + public Instruction OpenClNative_cos(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 81, x); + } + + public Instruction OpenClNative_divide(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 82, x, y); + } + + public Instruction OpenClNative_exp(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 83, x); + } + + public Instruction OpenClNative_exp2(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 84, x); + } + + public Instruction OpenClNative_exp10(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 85, x); + } + + public Instruction OpenClNative_log(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 86, x); + } + + public Instruction OpenClNative_log2(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 87, x); + } + + public Instruction OpenClNative_log10(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 88, x); + } + + public Instruction OpenClNative_powr(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 89, x, y); + } + + public Instruction OpenClNative_recip(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 90, x); + } + + public Instruction OpenClNative_rsqrt(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 91, x); + } + + public Instruction OpenClNative_sin(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 92, x); + } + + public Instruction OpenClNative_sqrt(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 93, x); + } + + public Instruction OpenClNative_tan(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 94, x); + } + + public Instruction OpenClS_abs(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 141, x); + } + + public Instruction OpenClS_abs_diff(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 142, x, y); + } + + public Instruction OpenClS_add_sat(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 143, x, y); + } + + public Instruction OpenClU_add_sat(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 144, x, y); + } + + public Instruction OpenClS_hadd(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 145, x, y); + } + + public Instruction OpenClU_hadd(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 146, x, y); + } + + public Instruction OpenClS_rhadd(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 147, x, y); + } + + public Instruction OpenClU_rhadd(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 148, x, y); + } + + public Instruction OpenClS_clamp(Instruction resultType, Instruction x, Instruction minval, Instruction maxval) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 149, x, minval, maxval); + } + + public Instruction OpenClU_clamp(Instruction resultType, Instruction x, Instruction minval, Instruction maxval) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 150, x, minval, maxval); + } + + public Instruction OpenClClz(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 151, x); + } + + public Instruction OpenClCtz(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 152, x); + } + + public Instruction OpenClS_mad_hi(Instruction resultType, Instruction a, Instruction b, Instruction c) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 153, a, b, c); + } + + public Instruction OpenClU_mad_sat(Instruction resultType, Instruction x, Instruction y, Instruction z) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 154, x, y, z); + } + + public Instruction OpenClS_mad_sat(Instruction resultType, Instruction x, Instruction y, Instruction z) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 155, x, y, z); + } + + public Instruction OpenClS_max(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 156, x, y); + } + + public Instruction OpenClU_max(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 157, x, y); + } + + public Instruction OpenClS_min(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 158, x, y); + } + + public Instruction OpenClU_min(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 159, x, y); + } + + public Instruction OpenClS_mul_hi(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 160, x, y); + } + + public Instruction OpenClRotate(Instruction resultType, Instruction v, Instruction i) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 161, v, i); + } + + public Instruction OpenClS_sub_sat(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 162, x, y); + } + + public Instruction OpenClU_sub_sat(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 163, x, y); + } + + public Instruction OpenClU_upsample(Instruction resultType, Instruction hi, Instruction lo) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 164, hi, lo); + } + + public Instruction OpenClS_upsample(Instruction resultType, Instruction hi, Instruction lo) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 165, hi, lo); + } + + public Instruction OpenClPopcount(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 166, x); + } + + public Instruction OpenClS_mad24(Instruction resultType, Instruction x, Instruction y, Instruction z) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 167, x, y, z); + } + + public Instruction OpenClU_mad24(Instruction resultType, Instruction x, Instruction y, Instruction z) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 168, x, y, z); + } + + public Instruction OpenClS_mul24(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 169, x, y); + } + + public Instruction OpenClU_mul24(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 170, x, y); + } + + public Instruction OpenClU_abs(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 201, x); + } + + public Instruction OpenClU_abs_diff(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 202, x, y); + } + + public Instruction OpenClU_mul_hi(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 203, x, y); + } + + public Instruction OpenClU_mad_hi(Instruction resultType, Instruction a, Instruction b, Instruction c) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 204, a, b, c); + } + + public Instruction OpenClFclamp(Instruction resultType, Instruction x, Instruction minval, Instruction maxval) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 95, x, minval, maxval); + } + + public Instruction OpenClDegrees(Instruction resultType, Instruction radians) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 96, radians); + } + + public Instruction OpenClFmax_common(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 97, x, y); + } + + public Instruction OpenClFmin_common(Instruction resultType, Instruction x, Instruction y) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 98, x, y); + } + + public Instruction OpenClMix(Instruction resultType, Instruction x, Instruction y, Instruction a) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 99, x, y, a); + } + + public Instruction OpenClRadians(Instruction resultType, Instruction degrees) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 100, degrees); + } + + public Instruction OpenClStep(Instruction resultType, Instruction edge, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 101, edge, x); + } + + public Instruction OpenClSmoothstep(Instruction resultType, Instruction edge0, Instruction edge1, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 102, edge0, edge1, x); + } + + public Instruction OpenClSign(Instruction resultType, Instruction x) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 103, x); + } + + public Instruction OpenClCross(Instruction resultType, Instruction p0, Instruction p1) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 104, p0, p1); + } + + public Instruction OpenClDistance(Instruction resultType, Instruction p0, Instruction p1) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 105, p0, p1); + } + + public Instruction OpenClLength(Instruction resultType, Instruction p) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 106, p); + } + + public Instruction OpenClNormalize(Instruction resultType, Instruction p) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 107, p); + } + + public Instruction OpenClFast_distance(Instruction resultType, Instruction p0, Instruction p1) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 108, p0, p1); + } + + public Instruction OpenClFast_length(Instruction resultType, Instruction p) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 109, p); + } + + public Instruction OpenClFast_normalize(Instruction resultType, Instruction p) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 110, p); + } + + public Instruction OpenClBitselect(Instruction resultType, Instruction a, Instruction b, Instruction c) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 186, a, b, c); + } + + public Instruction OpenClSelect(Instruction resultType, Instruction a, Instruction b, Instruction c) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 187, a, b, c); + } + + public Instruction OpenClVloadn(Instruction resultType, Instruction offset, Instruction p, LiteralInteger n) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 171, offset, p, n); + } + + public Instruction OpenClVstoren(Instruction resultType, Instruction data, Instruction offset, Instruction p) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 172, data, offset, p); + } + + public Instruction OpenClVload_half(Instruction resultType, Instruction offset, Instruction p) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 173, offset, p); + } + + public Instruction OpenClVload_halfn(Instruction resultType, Instruction offset, Instruction p, LiteralInteger n) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 174, offset, p, n); + } + + public Instruction OpenClVstore_half(Instruction resultType, Instruction data, Instruction offset, Instruction p) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 175, data, offset, p); + } + + public Instruction OpenClVstore_half_r(Instruction resultType, Instruction data, Instruction offset, Instruction p, FPRoundingMode mode) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 176, data, offset, p, LiteralInteger.CreateForEnum(mode)); + } + + public Instruction OpenClVstore_halfn(Instruction resultType, Instruction data, Instruction offset, Instruction p) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 177, data, offset, p); + } + + public Instruction OpenClVstore_halfn_r(Instruction resultType, Instruction data, Instruction offset, Instruction p, FPRoundingMode mode) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 178, data, offset, p, LiteralInteger.CreateForEnum(mode)); + } + + public Instruction OpenClVloada_halfn(Instruction resultType, Instruction offset, Instruction p, LiteralInteger n) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 179, offset, p, n); + } + + public Instruction OpenClVstorea_halfn(Instruction resultType, Instruction data, Instruction offset, Instruction p) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 180, data, offset, p); + } + + public Instruction OpenClVstorea_halfn_r(Instruction resultType, Instruction data, Instruction offset, Instruction p, FPRoundingMode mode) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 181, data, offset, p, LiteralInteger.CreateForEnum(mode)); + } + + public Instruction OpenClShuffle(Instruction resultType, Instruction x, Instruction shufflemask) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 182, x, shufflemask); + } + + public Instruction OpenClShuffle2(Instruction resultType, Instruction x, Instruction y, Instruction shufflemask) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 183, x, y, shufflemask); + } + + public Instruction OpenClPrefetch(Instruction resultType, Instruction ptr, Instruction numelements) + { + return ExtInst(resultType, AddExtInstImport("OpenCL.std"), 185, ptr, numelements); + } + } +} diff --git a/third_party/maxwell_shader/src/Spv.Generator/ConstantKey.cs b/third_party/maxwell_shader/src/Spv.Generator/ConstantKey.cs new file mode 100644 index 0000000..e844f6f --- /dev/null +++ b/third_party/maxwell_shader/src/Spv.Generator/ConstantKey.cs @@ -0,0 +1,30 @@ +using System; +using System.Diagnostics.CodeAnalysis; + +namespace Spv.Generator +{ + internal readonly struct ConstantKey : IEquatable + { + private readonly Instruction _constant; + + public ConstantKey(Instruction constant) + { + _constant = constant; + } + + public override int GetHashCode() + { + return HashCode.Combine(_constant.Opcode, _constant.GetHashCodeContent(), _constant.GetHashCodeResultType()); + } + + public bool Equals(ConstantKey other) + { + return _constant.Opcode == other._constant.Opcode && _constant.EqualsContent(other._constant) && _constant.EqualsResultType(other._constant); + } + + public override bool Equals([NotNullWhen(true)] object obj) + { + return obj is ConstantKey key && Equals(key); + } + } +} diff --git a/third_party/maxwell_shader/src/Spv.Generator/DeterministicHashCode.cs b/third_party/maxwell_shader/src/Spv.Generator/DeterministicHashCode.cs new file mode 100644 index 0000000..567d86e --- /dev/null +++ b/third_party/maxwell_shader/src/Spv.Generator/DeterministicHashCode.cs @@ -0,0 +1,109 @@ +using System; +using System.Numerics; +using System.Runtime.CompilerServices; + +namespace Spv.Generator +{ + /// + /// Similar to System.HashCode, but without introducing random values. + /// The same primes and shifts are used. + /// + internal static class DeterministicHashCode + { + private const uint Prime1 = 2654435761U; + private const uint Prime2 = 2246822519U; + private const uint Prime3 = 3266489917U; + private const uint Prime4 = 668265263U; + + public static int GetHashCode(string value) + { + uint hash = (uint)value.Length + Prime1; + + for (int i = 0; i < value.Length; i++) + { + hash += (hash << 7) ^ value[i]; + } + + return (int)MixFinal(hash); + } + + public static int Combine(ReadOnlySpan values) + { + uint hashCode = Prime2; + hashCode += 4 * (uint)values.Length; + + foreach (T value in values) + { + uint hc = (uint)(value?.GetHashCode() ?? 0); + hashCode = MixStep(hashCode, hc); + } + + return (int)MixFinal(hashCode); + } + + public static int Combine(T1 value1, T2 value2) + { + uint hc1 = (uint)(value1?.GetHashCode() ?? 0); + uint hc2 = (uint)(value2?.GetHashCode() ?? 0); + + uint hash = Prime2; + hash += 8; + + hash = MixStep(hash, hc1); + hash = MixStep(hash, hc2); + + return (int)MixFinal(hash); + } + + public static int Combine(T1 value1, T2 value2, T3 value3) + { + uint hc1 = (uint)(value1?.GetHashCode() ?? 0); + uint hc2 = (uint)(value2?.GetHashCode() ?? 0); + uint hc3 = (uint)(value3?.GetHashCode() ?? 0); + + uint hash = Prime2; + hash += 12; + + hash = MixStep(hash, hc1); + hash = MixStep(hash, hc2); + hash = MixStep(hash, hc3); + + return (int)MixFinal(hash); + } + + public static int Combine(T1 value1, T2 value2, T3 value3, T4 value4) + { + uint hc1 = (uint)(value1?.GetHashCode() ?? 0); + uint hc2 = (uint)(value2?.GetHashCode() ?? 0); + uint hc3 = (uint)(value3?.GetHashCode() ?? 0); + uint hc4 = (uint)(value4?.GetHashCode() ?? 0); + + uint hash = Prime2; + hash += 16; + + hash = MixStep(hash, hc1); + hash = MixStep(hash, hc2); + hash = MixStep(hash, hc3); + hash = MixStep(hash, hc4); + + return (int)MixFinal(hash); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint MixStep(uint hashCode, uint mixValue) + { + return BitOperations.RotateLeft(hashCode + mixValue * Prime3, 17) * Prime4; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint MixFinal(uint hash) + { + hash ^= hash >> 15; + hash *= Prime2; + hash ^= hash >> 13; + hash *= Prime3; + hash ^= hash >> 16; + return hash; + } + } +} diff --git a/third_party/maxwell_shader/src/Spv.Generator/DeterministicStringKey.cs b/third_party/maxwell_shader/src/Spv.Generator/DeterministicStringKey.cs new file mode 100644 index 0000000..0897553 --- /dev/null +++ b/third_party/maxwell_shader/src/Spv.Generator/DeterministicStringKey.cs @@ -0,0 +1,29 @@ +using System; + +namespace Spv.Generator +{ + internal class DeterministicStringKey : IEquatable + { + private readonly string _value; + + public DeterministicStringKey(string value) + { + _value = value; + } + + public override int GetHashCode() + { + return DeterministicHashCode.GetHashCode(_value); + } + + public bool Equals(DeterministicStringKey other) + { + return _value == other?._value; + } + + public override bool Equals(object obj) + { + return obj is DeterministicStringKey key && Equals(key); + } + } +} diff --git a/third_party/maxwell_shader/src/Spv.Generator/GeneratorPool.cs b/third_party/maxwell_shader/src/Spv.Generator/GeneratorPool.cs new file mode 100644 index 0000000..63aef8a --- /dev/null +++ b/third_party/maxwell_shader/src/Spv.Generator/GeneratorPool.cs @@ -0,0 +1,56 @@ +using System.Collections.Generic; + +namespace Spv.Generator +{ + public class GeneratorPool where T : class, new() + { + private readonly List _pool; + private int _chunkIndex = -1; + private int _poolIndex = -1; + private readonly int _poolSizeIncrement; + + public GeneratorPool() : this(1000, 200) { } + + public GeneratorPool(int chunkSizeLimit, int poolSizeIncrement) + { + _poolSizeIncrement = poolSizeIncrement; + + _pool = new(chunkSizeLimit * 2); + + AddChunkIfNeeded(); + } + + public T Allocate() + { + if (++_poolIndex >= _poolSizeIncrement) + { + AddChunkIfNeeded(); + + _poolIndex = 0; + } + + return _pool[_chunkIndex][_poolIndex]; + } + + private void AddChunkIfNeeded() + { + if (++_chunkIndex >= _pool.Count) + { + T[] pool = new T[_poolSizeIncrement]; + + for (int i = 0; i < _poolSizeIncrement; i++) + { + pool[i] = new T(); + } + + _pool.Add(pool); + } + } + + public void Clear() + { + _chunkIndex = 0; + _poolIndex = -1; + } + } +} diff --git a/third_party/maxwell_shader/src/Spv.Generator/IOperand.cs b/third_party/maxwell_shader/src/Spv.Generator/IOperand.cs new file mode 100644 index 0000000..102e80d --- /dev/null +++ b/third_party/maxwell_shader/src/Spv.Generator/IOperand.cs @@ -0,0 +1,14 @@ +using System; +using System.IO; + +namespace Spv.Generator +{ + public interface IOperand : IEquatable + { + OperandType Type { get; } + + ushort WordCount { get; } + + void WriteOperand(BinaryWriter writer); + } +} diff --git a/third_party/maxwell_shader/src/Spv.Generator/Instruction.cs b/third_party/maxwell_shader/src/Spv.Generator/Instruction.cs new file mode 100644 index 0000000..e380cdc --- /dev/null +++ b/third_party/maxwell_shader/src/Spv.Generator/Instruction.cs @@ -0,0 +1,247 @@ +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.IO; + +namespace Spv.Generator +{ + public sealed class Instruction : IOperand, IEquatable + { + public const uint InvalidId = uint.MaxValue; + + public Specification.Op Opcode { get; private set; } + private Instruction _resultType; + private InstructionOperands _operands; + + public uint Id { get; set; } + + public Instruction() { } + + public void Set(Specification.Op opcode, uint id = InvalidId, Instruction resultType = null) + { + Opcode = opcode; + Id = id; + _resultType = resultType; + + _operands = new InstructionOperands(); + } + + public void SetId(uint id) + { + Id = id; + } + + public OperandType Type => OperandType.Instruction; + + public ushort GetTotalWordCount() + { + ushort result = WordCount; + + if (Id != InvalidId) + { + result++; + } + + if (_resultType != null) + { + result += _resultType.WordCount; + } + + Span operands = _operands.AsSpan(); + for (int i = 0; i < operands.Length; i++) + { + result += operands[i].WordCount; + } + + return result; + } + + public ushort WordCount => 1; + + public void AddOperand(IOperand value) + { + Debug.Assert(value != null); + _operands.Add(value); + } + + public void AddOperand(ReadOnlySpan value) + { + foreach (IOperand instruction in value) + { + AddOperand(instruction); + } + } + + public void AddOperand(ReadOnlySpan value) + { + foreach (LiteralInteger instruction in value) + { + AddOperand(instruction); + } + } + + public void AddOperand(LiteralInteger value) + { + AddOperand((IOperand)value); + } + + public void AddOperand(ReadOnlySpan value) + { + foreach (Instruction instruction in value) + { + AddOperand(instruction); + } + } + + public void AddOperand(Instruction value) + { + AddOperand((IOperand)value); + } + + public void AddOperand(string value) + { + AddOperand(new LiteralString(value)); + } + + public void AddOperand(T value) where T : Enum + { + AddOperand(LiteralInteger.CreateForEnum(value)); + } + + public void Write(BinaryWriter writer) + { + // Word 0 + writer.Write((ushort)Opcode); + writer.Write(GetTotalWordCount()); + + _resultType?.WriteOperand(writer); + + if (Id != InvalidId) + { + writer.Write(Id); + } + + Span operands = _operands.AsSpan(); + for (int i = 0; i < operands.Length; i++) + { + operands[i].WriteOperand(writer); + } + } + + public void WriteOperand(BinaryWriter writer) + { + Debug.Assert(Id != InvalidId); + + if (Id == InvalidId) + { + string methodToCall; + + if (Opcode == Specification.Op.OpVariable) + { + methodToCall = "AddLocalVariable or AddGlobalVariable"; + } + else if (Opcode == Specification.Op.OpLabel) + { + methodToCall = "AddLabel"; + } + else + { + throw new InvalidOperationException("Internal error"); + } + + throw new InvalidOperationException($"Id wasn't bound to the module, please make sure to call {methodToCall}"); + } + + writer.Write(Id); + } + + public override bool Equals(object obj) + { + return obj is Instruction instruction && Equals(instruction); + } + + public bool Equals(Instruction cmpObj) + { + bool result = Type == cmpObj.Type && Id == cmpObj.Id; + + if (result) + { + if (_resultType != null && cmpObj._resultType != null) + { + result &= _resultType.Equals(cmpObj._resultType); + } + else if (_resultType != null || cmpObj._resultType != null) + { + return false; + } + } + + if (result) + { + result &= EqualsContent(cmpObj); + } + + return result; + } + + public bool EqualsContent(Instruction cmpObj) + { + Span thisOperands = _operands.AsSpan(); + Span cmpOperands = cmpObj._operands.AsSpan(); + + if (thisOperands.Length != cmpOperands.Length) + { + return false; + } + + for (int i = 0; i < thisOperands.Length; i++) + { + if (!thisOperands[i].Equals(cmpOperands[i])) + { + return false; + } + } + + return true; + } + + public bool EqualsResultType(Instruction cmpObj) + { + return _resultType.Opcode == cmpObj._resultType.Opcode && _resultType.EqualsContent(cmpObj._resultType); + } + + public int GetHashCodeContent() + { + return DeterministicHashCode.Combine(_operands.AsSpan()); + } + + public int GetHashCodeResultType() + { + return DeterministicHashCode.Combine(_resultType.Opcode, _resultType.GetHashCodeContent()); + } + + public override int GetHashCode() + { + return DeterministicHashCode.Combine(Opcode, Id, _resultType, DeterministicHashCode.Combine(_operands.AsSpan())); + } + + public bool Equals(IOperand obj) + { + return obj is Instruction instruction && Equals(instruction); + } + + private static readonly Dictionary _operandLabels = new() + { + { Specification.Op.OpConstant, ["Value"] }, + { Specification.Op.OpTypeInt, ["Width", "Signed"] }, + { Specification.Op.OpTypeFloat, ["Width"] }, + }; + + public override string ToString() + { + string[] labels = _operandLabels.TryGetValue(Opcode, out string[] opLabels) ? opLabels : []; + string result = _resultType == null ? string.Empty : $"{_resultType} "; + return $"{result}{Opcode}{_operands.ToString(labels)}"; + } + } +} diff --git a/third_party/maxwell_shader/src/Spv.Generator/InstructionOperands.cs b/third_party/maxwell_shader/src/Spv.Generator/InstructionOperands.cs new file mode 100644 index 0000000..1a4846a --- /dev/null +++ b/third_party/maxwell_shader/src/Spv.Generator/InstructionOperands.cs @@ -0,0 +1,72 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Runtime.InteropServices; + +namespace Spv.Generator +{ + public struct InstructionOperands + { + private const int InternalCount = 5; + + public int Count; + public IOperand Operand1; + public IOperand Operand2; + public IOperand Operand3; + public IOperand Operand4; + public IOperand Operand5; + public IOperand[] Overflow; + + public Span AsSpan() + { + if (Count > InternalCount) + { + return MemoryMarshal.CreateSpan(ref this.Overflow[0], Count); + } + else + { + return MemoryMarshal.CreateSpan(ref this.Operand1, Count); + } + } + + public void Add(IOperand operand) + { + if (Count < InternalCount) + { + MemoryMarshal.CreateSpan(ref this.Operand1, Count + 1)[Count] = operand; + Count++; + } + else + { + if (Overflow == null) + { + Overflow = new IOperand[InternalCount * 2]; + MemoryMarshal.CreateSpan(ref this.Operand1, InternalCount).CopyTo(Overflow.AsSpan()); + } + else if (Count == Overflow.Length) + { + Array.Resize(ref Overflow, Overflow.Length * 2); + } + + Overflow[Count++] = operand; + } + } + + private readonly IEnumerable AllOperands => new[] { Operand1, Operand2, Operand3, Operand4, Operand5 } + .Concat(Overflow ?? []) + .Take(Count); + + public readonly override string ToString() + { + return $"({string.Join(", ", AllOperands)})"; + } + + public readonly string ToString(string[] labels) + { + IEnumerable labeledParams = AllOperands.Zip(labels, (op, label) => $"{label}: {op}"); + IEnumerable unlabeledParams = AllOperands.Skip(labels.Length).Select(op => op.ToString()); + IEnumerable paramsToPrint = labeledParams.Concat(unlabeledParams); + return $"({string.Join(", ", paramsToPrint)})"; + } + } +} diff --git a/third_party/maxwell_shader/src/Spv.Generator/LICENSE b/third_party/maxwell_shader/src/Spv.Generator/LICENSE new file mode 100644 index 0000000..458723b --- /dev/null +++ b/third_party/maxwell_shader/src/Spv.Generator/LICENSE @@ -0,0 +1,23 @@ +Permission is hereby granted, free of charge, to any +person obtaining a copy of this software and associated +documentation files (the "Software"), to deal in the +Software without restriction, including without +limitation the rights to use, copy, modify, merge, +publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software +is furnished to do so, subject to the following +conditions: + +The above copyright notice and this permission notice +shall be included in all copies or substantial portions +of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF +ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED +TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT +SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR +IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. diff --git a/third_party/maxwell_shader/src/Spv.Generator/LiteralInteger.cs b/third_party/maxwell_shader/src/Spv.Generator/LiteralInteger.cs new file mode 100644 index 0000000..1f6e943 --- /dev/null +++ b/third_party/maxwell_shader/src/Spv.Generator/LiteralInteger.cs @@ -0,0 +1,105 @@ +using System; +using System.IO; + +namespace Spv.Generator +{ + public class LiteralInteger : IOperand, IEquatable + { + [ThreadStatic] + private static GeneratorPool _pool; + + internal static void RegisterPool(GeneratorPool pool) + { + _pool = pool; + } + + internal static void UnregisterPool() + { + _pool = null; + } + + public OperandType Type => OperandType.Number; + + private enum IntegerType + { + UInt32, + Int32, + UInt64, + Int64, + Float32, + Float64, + } + + private IntegerType _integerType; + private ulong _data; + + public ushort WordCount { get; private set; } + + public LiteralInteger() { } + + private static LiteralInteger New() + { + return _pool.Allocate(); + } + + private LiteralInteger Set(ulong data, IntegerType integerType, ushort wordCount) + { + _data = data; + _integerType = integerType; + + WordCount = wordCount; + + return this; + } + + public static implicit operator LiteralInteger(int value) => New().Set((ulong)value, IntegerType.Int32, 1); + public static implicit operator LiteralInteger(uint value) => New().Set(value, IntegerType.UInt32, 1); + public static implicit operator LiteralInteger(long value) => New().Set((ulong)value, IntegerType.Int64, 2); + public static implicit operator LiteralInteger(ulong value) => New().Set(value, IntegerType.UInt64, 2); + public static implicit operator LiteralInteger(float value) => New().Set(BitConverter.SingleToUInt32Bits(value), IntegerType.Float32, 1); + public static implicit operator LiteralInteger(double value) => New().Set(BitConverter.DoubleToUInt64Bits(value), IntegerType.Float64, 2); + public static implicit operator LiteralInteger(Enum value) => New().Set((ulong)(int)(object)value, IntegerType.Int32, 1); + + // NOTE: this is not in the standard, but this is some syntax sugar useful in some instructions (TypeInt ect) + public static implicit operator LiteralInteger(bool value) => New().Set(Convert.ToUInt64(value), IntegerType.Int32, 1); + + public static LiteralInteger CreateForEnum(T value) where T : Enum + { + return value; + } + + public void WriteOperand(BinaryWriter writer) + { + if (WordCount == 1) + { + writer.Write((uint)_data); + } + else + { + writer.Write(_data); + } + } + + public override bool Equals(object obj) + { + return obj is LiteralInteger literalInteger && Equals(literalInteger); + } + + public bool Equals(LiteralInteger cmpObj) + { + return Type == cmpObj.Type && _integerType == cmpObj._integerType && _data == cmpObj._data; + } + + public override int GetHashCode() + { + return DeterministicHashCode.Combine(Type, _data); + } + + public bool Equals(IOperand obj) + { + return obj is LiteralInteger literalInteger && Equals(literalInteger); + } + + public override string ToString() => $"{_integerType} {_data}"; + } +} diff --git a/third_party/maxwell_shader/src/Spv.Generator/LiteralString.cs b/third_party/maxwell_shader/src/Spv.Generator/LiteralString.cs new file mode 100644 index 0000000..6e6918b --- /dev/null +++ b/third_party/maxwell_shader/src/Spv.Generator/LiteralString.cs @@ -0,0 +1,54 @@ +using System; +using System.IO; +using System.Text; + +namespace Spv.Generator +{ + public class LiteralString : IOperand, IEquatable + { + public OperandType Type => OperandType.String; + + private readonly string _value; + + public LiteralString(string value) + { + _value = value; + } + + public ushort WordCount => (ushort)(_value.Length / 4 + 1); + + public void WriteOperand(BinaryWriter writer) + { + writer.Write(_value.AsSpan()); + + // String must be padded to the word size (which is 4 bytes). + int paddingSize = 4 - (Encoding.ASCII.GetByteCount(_value) % 4); + + Span padding = stackalloc byte[paddingSize]; + + writer.Write(padding); + } + + public override bool Equals(object obj) + { + return obj is LiteralString literalString && Equals(literalString); + } + + public bool Equals(LiteralString cmpObj) + { + return Type == cmpObj.Type && _value.Equals(cmpObj._value); + } + + public override int GetHashCode() + { + return DeterministicHashCode.Combine(Type, DeterministicHashCode.GetHashCode(_value)); + } + + public bool Equals(IOperand obj) + { + return obj is LiteralString literalString && Equals(literalString); + } + + public override string ToString() => _value; + } +} diff --git a/third_party/maxwell_shader/src/Spv.Generator/Module.cs b/third_party/maxwell_shader/src/Spv.Generator/Module.cs new file mode 100644 index 0000000..a9c0e55 --- /dev/null +++ b/third_party/maxwell_shader/src/Spv.Generator/Module.cs @@ -0,0 +1,368 @@ +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.IO; +using static Spv.Specification; + +namespace Spv.Generator +{ + public partial class Module + { + // TODO: Register on SPIR-V registry. + private const int GeneratorId = 0; + + private readonly uint _version; + + private uint _bound; + + // Follow spec order here while keeping it as simple as possible. + private readonly List _capabilities; + private readonly List _extensions; + private readonly Dictionary _extInstImports; + private AddressingModel _addressingModel; + private MemoryModel _memoryModel; + + private readonly List _entrypoints; + private readonly List _executionModes; + private readonly List _debug; + private readonly List _annotations; + + // In the declaration block. + private readonly Dictionary _typeDeclarations; + private readonly List _typeDeclarationsList; + // In the declaration block. + private readonly List _globals; + // In the declaration block. + private readonly Dictionary _constants; + // In the declaration block, for function that aren't defined in the module. + private readonly List _functionsDeclarations; + + private readonly List _functionsDefinitions; + + private readonly GeneratorPool _instPool; + private readonly GeneratorPool _integerPool; + + public Module(uint version, GeneratorPool instPool = null, GeneratorPool integerPool = null) + { + _version = version; + _bound = 1; + _capabilities = []; + _extensions = []; + _extInstImports = new Dictionary(); + _addressingModel = AddressingModel.Logical; + _memoryModel = MemoryModel.Simple; + _entrypoints = []; + _executionModes = []; + _debug = []; + _annotations = []; + _typeDeclarations = new Dictionary(); + _typeDeclarationsList = []; + _constants = new Dictionary(); + _globals = []; + _functionsDeclarations = []; + _functionsDefinitions = []; + + _instPool = instPool ?? new GeneratorPool(); + _integerPool = integerPool ?? new GeneratorPool(); + + LiteralInteger.RegisterPool(_integerPool); + } + + private uint GetNewId() + { + return _bound++; + } + + public void AddCapability(Capability capability) + { + _capabilities.Add(capability); + } + + public void AddExtension(string extension) + { + _extensions.Add(extension); + } + + public Instruction NewInstruction(Op opcode, uint id = Instruction.InvalidId, Instruction resultType = null) + { + Instruction result = _instPool.Allocate(); + result.Set(opcode, id, resultType); + + return result; + } + + public Instruction AddExtInstImport(string import) + { + DeterministicStringKey key = new(import); + + if (_extInstImports.TryGetValue(key, out Instruction extInstImport)) + { + // Update the duplicate instance to use the good id so it ends up being encoded correctly. + return extInstImport; + } + + Instruction instruction = NewInstruction(Op.OpExtInstImport); + instruction.AddOperand(import); + + instruction.SetId(GetNewId()); + + _extInstImports.Add(key, instruction); + + return instruction; + } + + private void AddTypeDeclaration(Instruction instruction, bool forceIdAllocation) + { + TypeDeclarationKey key = new(instruction); + + if (!forceIdAllocation) + { + if (_typeDeclarations.TryGetValue(key, out Instruction typeDeclaration)) + { + // Update the duplicate instance to use the good id so it ends up being encoded correctly. + + instruction.SetId(typeDeclaration.Id); + + return; + } + } + + instruction.SetId(GetNewId()); + + _typeDeclarations[key] = instruction; + _typeDeclarationsList.Add(instruction); + } + + public void AddEntryPoint(ExecutionModel executionModel, Instruction function, string name, params ReadOnlySpan interfaces) + { + Debug.Assert(function.Opcode == Op.OpFunction); + + Instruction entryPoint = NewInstruction(Op.OpEntryPoint); + + entryPoint.AddOperand(executionModel); + entryPoint.AddOperand(function); + entryPoint.AddOperand(name); + entryPoint.AddOperand(interfaces); + + _entrypoints.Add(entryPoint); + } + + public void AddExecutionMode(Instruction function, ExecutionMode mode, params ReadOnlySpan parameters) + { + Debug.Assert(function.Opcode == Op.OpFunction); + + Instruction executionModeInstruction = NewInstruction(Op.OpExecutionMode); + + executionModeInstruction.AddOperand(function); + executionModeInstruction.AddOperand(mode); + executionModeInstruction.AddOperand(parameters); + + _executionModes.Add(executionModeInstruction); + } + + private void AddToFunctionDefinitions(Instruction instruction) + { + Debug.Assert(instruction.Opcode != Op.OpTypeInt); + _functionsDefinitions.Add(instruction); + } + + private void AddAnnotation(Instruction annotation) + { + _annotations.Add(annotation); + } + + private void AddDebug(Instruction debug) + { + _debug.Add(debug); + } + + public void AddLabel(Instruction label) + { + Debug.Assert(label.Opcode == Op.OpLabel); + + label.SetId(GetNewId()); + + AddToFunctionDefinitions(label); + } + + public void AddLocalVariable(Instruction variable) + { + // TODO: Ensure it has the local modifier. + Debug.Assert(variable.Opcode == Op.OpVariable); + + variable.SetId(GetNewId()); + + AddToFunctionDefinitions(variable); + } + + public void AddGlobalVariable(Instruction variable) + { + // TODO: Ensure it has the global modifier. + // TODO: All constants opcodes (OpSpecXXX and the rest of the OpConstantXXX). + Debug.Assert(variable.Opcode == Op.OpVariable); + + variable.SetId(GetNewId()); + + _globals.Add(variable); + } + + private void AddConstant(Instruction constant) + { + Debug.Assert(constant.Opcode is Op.OpConstant or + Op.OpConstantFalse or + Op.OpConstantTrue or + Op.OpConstantNull or + Op.OpConstantComposite); + + ConstantKey key = new(constant); + + if (_constants.TryGetValue(key, out Instruction global)) + { + // Update the duplicate instance to use the good id so it ends up being encoded correctly. + constant.SetId(global.Id); + + return; + } + + constant.SetId(GetNewId()); + + _constants.Add(key, constant); + } + + public Instruction ExtInst(Instruction resultType, Instruction set, LiteralInteger instruction, params ReadOnlySpan parameters) + { + Instruction result = NewInstruction(Op.OpExtInst, GetNewId(), resultType); + + result.AddOperand(set); + result.AddOperand(instruction); + result.AddOperand(parameters); + AddToFunctionDefinitions(result); + + return result; + } + + public void SetMemoryModel(AddressingModel addressingModel, MemoryModel memoryModel) + { + _addressingModel = addressingModel; + _memoryModel = memoryModel; + } + + // TODO: Find a way to make the auto generate one used. + public Instruction OpenClPrintf(Instruction resultType, Instruction format, params ReadOnlySpan additionalarguments) + { + Instruction result = NewInstruction(Op.OpExtInst, GetNewId(), resultType); + + result.AddOperand(AddExtInstImport("OpenCL.std")); + result.AddOperand((LiteralInteger)184); + result.AddOperand(format); + result.AddOperand(additionalarguments); + AddToFunctionDefinitions(result); + + return result; + } + + public byte[] Generate() + { + // Estimate the size needed for the generated code, to avoid expanding the MemoryStream. + int sizeEstimate = 1024 + _functionsDefinitions.Count * 32; + + using MemoryStream stream = new(sizeEstimate); + + BinaryWriter writer = new(stream, System.Text.Encoding.ASCII); + + // Header + writer.Write(MagicNumber); + writer.Write(_version); + writer.Write(GeneratorId); + writer.Write(_bound); + writer.Write(0u); + + // 1. + foreach (Capability capability in _capabilities) + { + Instruction capabilityInstruction = NewInstruction(Op.OpCapability); + + capabilityInstruction.AddOperand(capability); + capabilityInstruction.Write(writer); + } + + // 2. + foreach (string extension in _extensions) + { + Instruction extensionInstruction = NewInstruction(Op.OpExtension); + + extensionInstruction.AddOperand(extension); + extensionInstruction.Write(writer); + } + + // 3. + foreach (Instruction extInstImport in _extInstImports.Values) + { + extInstImport.Write(writer); + } + + // 4. + Instruction memoryModelInstruction = NewInstruction(Op.OpMemoryModel); + memoryModelInstruction.AddOperand(_addressingModel); + memoryModelInstruction.AddOperand(_memoryModel); + memoryModelInstruction.Write(writer); + + // 5. + foreach (Instruction entrypoint in _entrypoints) + { + entrypoint.Write(writer); + } + + // 6. + foreach (Instruction executionMode in _executionModes) + { + executionMode.Write(writer); + } + + // 7. + // TODO: Order debug information correctly. + foreach (Instruction debug in _debug) + { + debug.Write(writer); + } + + // 8. + foreach (Instruction annotation in _annotations) + { + annotation.Write(writer); + } + + // Ensure that everything is in the right order in the declarations section. + List declarations = []; + declarations.AddRange(_typeDeclarationsList); + declarations.AddRange(_globals); + declarations.AddRange(_constants.Values); + declarations.Sort((x, y) => x.Id.CompareTo(y.Id)); + + // 9. + foreach (Instruction declaration in declarations) + { + declaration.Write(writer); + } + + // 10. + foreach (Instruction functionDeclaration in _functionsDeclarations) + { + functionDeclaration.Write(writer); + } + + // 11. + foreach (Instruction functionDefinition in _functionsDefinitions) + { + functionDefinition.Write(writer); + } + + _instPool.Clear(); + _integerPool.Clear(); + + LiteralInteger.UnregisterPool(); + + return stream.ToArray(); + } + } +} diff --git a/third_party/maxwell_shader/src/Spv.Generator/OperandType.cs b/third_party/maxwell_shader/src/Spv.Generator/OperandType.cs new file mode 100644 index 0000000..e9607f4 --- /dev/null +++ b/third_party/maxwell_shader/src/Spv.Generator/OperandType.cs @@ -0,0 +1,10 @@ +namespace Spv.Generator +{ + public enum OperandType + { + Invalid, + Number, + String, + Instruction, + } +} diff --git a/third_party/maxwell_shader/src/Spv.Generator/Spv.Generator.csproj b/third_party/maxwell_shader/src/Spv.Generator/Spv.Generator.csproj new file mode 100644 index 0000000..5ea1b76 --- /dev/null +++ b/third_party/maxwell_shader/src/Spv.Generator/Spv.Generator.csproj @@ -0,0 +1,7 @@ + + + + $(DefaultItemExcludes);._* + + + diff --git a/third_party/maxwell_shader/src/Spv.Generator/TypeDeclarationKey.cs b/third_party/maxwell_shader/src/Spv.Generator/TypeDeclarationKey.cs new file mode 100644 index 0000000..a4eb8d2 --- /dev/null +++ b/third_party/maxwell_shader/src/Spv.Generator/TypeDeclarationKey.cs @@ -0,0 +1,30 @@ +using System; +using System.Diagnostics.CodeAnalysis; + +namespace Spv.Generator +{ + internal readonly struct TypeDeclarationKey : IEquatable + { + private readonly Instruction _typeDeclaration; + + public TypeDeclarationKey(Instruction typeDeclaration) + { + _typeDeclaration = typeDeclaration; + } + + public override int GetHashCode() + { + return DeterministicHashCode.Combine(_typeDeclaration.Opcode, _typeDeclaration.GetHashCodeContent()); + } + + public bool Equals(TypeDeclarationKey other) + { + return _typeDeclaration.Opcode == other._typeDeclaration.Opcode && _typeDeclaration.EqualsContent(other._typeDeclaration); + } + + public override bool Equals([NotNullWhen(true)] object obj) + { + return obj is TypeDeclarationKey key && Equals(key); + } + } +} diff --git a/third_party/maxwell_shader/src/Spv.Generator/spirv.cs b/third_party/maxwell_shader/src/Spv.Generator/spirv.cs new file mode 100644 index 0000000..fce7f16 --- /dev/null +++ b/third_party/maxwell_shader/src/Spv.Generator/spirv.cs @@ -0,0 +1,1624 @@ +// Copyright (c) 2014-2020 The Khronos Group Inc. +// +// Permission is hereby granted, free of charge, to any person obtaining a copy +// of this software and/or associated documentation files (the "Materials"), +// to deal in the Materials without restriction, including without limitation +// the rights to use, copy, modify, merge, publish, distribute, sublicense, +// and/or sell copies of the Materials, and to permit persons to whom the +// Materials are furnished to do so, subject to the following conditions: +// +// The above copyright notice and this permission notice shall be included in +// all copies or substantial portions of the Materials. +// +// MODIFICATIONS TO THIS FILE MAY MEAN IT NO LONGER ACCURATELY REFLECTS KHRONOS +// STANDARDS. THE UNMODIFIED, NORMATIVE VERSIONS OF KHRONOS SPECIFICATIONS AND +// HEADER INFORMATION ARE LOCATED AT https://www.khronos.org/registry/ +// +// THE MATERIALS ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS +// OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +// THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM,OUT OF OR IN CONNECTION WITH THE MATERIALS OR THE USE OR OTHER DEALINGS +// IN THE MATERIALS. + +// This header is automatically generated by the same tool that creates +// the Binary Section of the SPIR-V specification. + +// Enumeration tokens for SPIR-V, in various styles: +// C, C++, C++11, JSON, Lua, Python, C#, D +// +// - C will have tokens with a "Spv" prefix, e.g.: SpvSourceLanguageGLSL +// - C++ will have tokens in the "spv" name space, e.g.: spv::SourceLanguageGLSL +// - C++11 will use enum classes in the spv namespace, e.g.: spv::SourceLanguage::GLSL +// - Lua will use tables, e.g.: spv.SourceLanguage.GLSL +// - Python will use dictionaries, e.g.: spv['SourceLanguage']['GLSL'] +// - C# will use enum classes in the Specification class located in the "Spv" namespace, +// e.g.: Spv.Specification.SourceLanguage.GLSL +// - D will have tokens under the "spv" module, e.g: spv.SourceLanguage.GLSL +// +// Some tokens act like mask values, which can be OR'd together, +// while others are mutually exclusive. The mask-like ones have +// "Mask" in their name, and a parallel enum that has the shift +// amount (1 << x) for each corresponding enumerant. + +namespace Spv +{ + + public static class Specification + { + public const uint MagicNumber = 0x07230203; + public const uint Version = 0x00010500; + public const uint Revision = 4; + public const uint OpCodeMask = 0xffff; + public const uint WordCountShift = 16; + + public enum SourceLanguage + { + Unknown = 0, + ESSL = 1, + GLSL = 2, + OpenCL_C = 3, + OpenCL_CPP = 4, + HLSL = 5, + } + + public enum ExecutionModel + { + Vertex = 0, + TessellationControl = 1, + TessellationEvaluation = 2, + Geometry = 3, + Fragment = 4, + GLCompute = 5, + Kernel = 6, + TaskNV = 5267, + MeshNV = 5268, + RayGenerationKHR = 5313, + RayGenerationNV = RayGenerationKHR, + IntersectionKHR = 5314, + IntersectionNV = IntersectionKHR, + AnyHitKHR = 5315, + AnyHitNV = AnyHitKHR, + ClosestHitKHR = 5316, + ClosestHitNV = ClosestHitKHR, + MissKHR = 5317, + MissNV = MissKHR, + CallableKHR = 5318, + CallableNV = CallableKHR, + } + + public enum AddressingModel + { + Logical = 0, + Physical32 = 1, + Physical64 = 2, + PhysicalStorageBuffer64 = 5348, + PhysicalStorageBuffer64EXT = PhysicalStorageBuffer64, + } + + public enum MemoryModel + { + Simple = 0, + GLSL450 = 1, + OpenCL = 2, + Vulkan = 3, + VulkanKHR = Vulkan, + } + + public enum ExecutionMode + { + Invocations = 0, + SpacingEqual = 1, + SpacingFractionalEven = 2, + SpacingFractionalOdd = 3, + VertexOrderCw = 4, + VertexOrderCcw = 5, + PixelCenterInteger = 6, + OriginUpperLeft = 7, + OriginLowerLeft = 8, + EarlyFragmentTests = 9, + PointMode = 10, + Xfb = 11, + DepthReplacing = 12, + DepthGreater = 14, + DepthLess = 15, + DepthUnchanged = 16, + LocalSize = 17, + LocalSizeHint = 18, + InputPoints = 19, + InputLines = 20, + InputLinesAdjacency = 21, + Triangles = 22, + InputTrianglesAdjacency = 23, + Quads = 24, + Isolines = 25, + OutputVertices = 26, + OutputPoints = 27, + OutputLineStrip = 28, + OutputTriangleStrip = 29, + VecTypeHint = 30, + ContractionOff = 31, + Initializer = 33, + Finalizer = 34, + SubgroupSize = 35, + SubgroupsPerWorkgroup = 36, + SubgroupsPerWorkgroupId = 37, + LocalSizeId = 38, + LocalSizeHintId = 39, + PostDepthCoverage = 4446, + DenormPreserve = 4459, + DenormFlushToZero = 4460, + SignedZeroInfNanPreserve = 4461, + RoundingModeRTE = 4462, + RoundingModeRTZ = 4463, + StencilRefReplacingEXT = 5027, + OutputLinesNV = 5269, + OutputPrimitivesNV = 5270, + DerivativeGroupQuadsNV = 5289, + DerivativeGroupLinearNV = 5290, + OutputTrianglesNV = 5298, + PixelInterlockOrderedEXT = 5366, + PixelInterlockUnorderedEXT = 5367, + SampleInterlockOrderedEXT = 5368, + SampleInterlockUnorderedEXT = 5369, + ShadingRateInterlockOrderedEXT = 5370, + ShadingRateInterlockUnorderedEXT = 5371, + MaxWorkgroupSizeINTEL = 5893, + MaxWorkDimINTEL = 5894, + NoGlobalOffsetINTEL = 5895, + NumSIMDWorkitemsINTEL = 5896, + } + + public enum StorageClass + { + UniformConstant = 0, + Input = 1, + Uniform = 2, + Output = 3, + Workgroup = 4, + CrossWorkgroup = 5, + Private = 6, + Function = 7, + Generic = 8, + PushConstant = 9, + AtomicCounter = 10, + Image = 11, + StorageBuffer = 12, + CallableDataKHR = 5328, + CallableDataNV = CallableDataKHR, + IncomingCallableDataKHR = 5329, + IncomingCallableDataNV = IncomingCallableDataKHR, + RayPayloadKHR = 5338, + RayPayloadNV = RayPayloadKHR, + HitAttributeKHR = 5339, + HitAttributeNV = HitAttributeKHR, + IncomingRayPayloadKHR = 5342, + IncomingRayPayloadNV = IncomingRayPayloadKHR, + ShaderRecordBufferKHR = 5343, + ShaderRecordBufferNV = ShaderRecordBufferKHR, + PhysicalStorageBuffer = 5349, + PhysicalStorageBufferEXT = PhysicalStorageBuffer, + CodeSectionINTEL = 5605, + } + + public enum Dim + { + Dim1D = 0, + Dim2D = 1, + Dim3D = 2, + Cube = 3, + Rect = 4, + Buffer = 5, + SubpassData = 6, + } + + public enum SamplerAddressingMode + { + None = 0, + ClampToEdge = 1, + Clamp = 2, + Repeat = 3, + RepeatMirrored = 4, + } + + public enum SamplerFilterMode + { + Nearest = 0, + Linear = 1, + } + + public enum ImageFormat + { + Unknown = 0, + Rgba32f = 1, + Rgba16f = 2, + R32f = 3, + Rgba8 = 4, + Rgba8Snorm = 5, + Rg32f = 6, + Rg16f = 7, + R11fG11fB10f = 8, + R16f = 9, + Rgba16 = 10, + Rgb10A2 = 11, + Rg16 = 12, + Rg8 = 13, + R16 = 14, + R8 = 15, + Rgba16Snorm = 16, + Rg16Snorm = 17, + Rg8Snorm = 18, + R16Snorm = 19, + R8Snorm = 20, + Rgba32i = 21, + Rgba16i = 22, + Rgba8i = 23, + R32i = 24, + Rg32i = 25, + Rg16i = 26, + Rg8i = 27, + R16i = 28, + R8i = 29, + Rgba32ui = 30, + Rgba16ui = 31, + Rgba8ui = 32, + R32ui = 33, + Rgb10a2ui = 34, + Rg32ui = 35, + Rg16ui = 36, + Rg8ui = 37, + R16ui = 38, + R8ui = 39, + R64ui = 40, + R64i = 41, + } + + public enum ImageChannelOrder + { + R = 0, + A = 1, + RG = 2, + RA = 3, + RGB = 4, + RGBA = 5, + BGRA = 6, + ARGB = 7, + Intensity = 8, + Luminance = 9, + Rx = 10, + RGx = 11, + RGBx = 12, + Depth = 13, + DepthStencil = 14, + sRGB = 15, + sRGBx = 16, + sRGBA = 17, + sBGRA = 18, + ABGR = 19, + } + + public enum ImageChannelDataType + { + SnormInt8 = 0, + SnormInt16 = 1, + UnormInt8 = 2, + UnormInt16 = 3, + UnormShort565 = 4, + UnormShort555 = 5, + UnormInt101010 = 6, + SignedInt8 = 7, + SignedInt16 = 8, + SignedInt32 = 9, + UnsignedInt8 = 10, + UnsignedInt16 = 11, + UnsignedInt32 = 12, + HalfFloat = 13, + Float = 14, + UnormInt24 = 15, + UnormInt101010_2 = 16, + } + + public enum ImageOperandsShift + { + Bias = 0, + Lod = 1, + Grad = 2, + ConstOffset = 3, + Offset = 4, + ConstOffsets = 5, + Sample = 6, + MinLod = 7, + MakeTexelAvailable = 8, + MakeTexelAvailableKHR = MakeTexelAvailable, + MakeTexelVisible = 9, + MakeTexelVisibleKHR = MakeTexelVisible, + NonPrivateTexel = 10, + NonPrivateTexelKHR = NonPrivateTexel, + VolatileTexel = 11, + VolatileTexelKHR = VolatileTexel, + SignExtend = 12, + ZeroExtend = 13, + } + + public enum ImageOperandsMask + { + MaskNone = 0, + Bias = 0x00000001, + Lod = 0x00000002, + Grad = 0x00000004, + ConstOffset = 0x00000008, + Offset = 0x00000010, + ConstOffsets = 0x00000020, + Sample = 0x00000040, + MinLod = 0x00000080, + MakeTexelAvailable = 0x00000100, + MakeTexelAvailableKHR = MakeTexelAvailable, + MakeTexelVisible = 0x00000200, + MakeTexelVisibleKHR = MakeTexelVisible, + NonPrivateTexel = 0x00000400, + NonPrivateTexelKHR = NonPrivateTexel, + VolatileTexel = 0x00000800, + VolatileTexelKHR = VolatileTexel, + SignExtend = 0x00001000, + ZeroExtend = 0x00002000, + Offsets = 0x00010000, + } + + public enum FPFastMathModeShift + { + NotNaN = 0, + NotInf = 1, + NSZ = 2, + AllowRecip = 3, + Fast = 4, + } + + public enum FPFastMathModeMask + { + MaskNone = 0, + NotNaN = 0x00000001, + NotInf = 0x00000002, + NSZ = 0x00000004, + AllowRecip = 0x00000008, + Fast = 0x00000010, + } + + public enum FPRoundingMode + { + RTE = 0, + RTZ = 1, + RTP = 2, + RTN = 3, + } + + public enum LinkageType + { + Export = 0, + Import = 1, + } + + public enum AccessQualifier + { + ReadOnly = 0, + WriteOnly = 1, + ReadWrite = 2, + } + + public enum FunctionParameterAttribute + { + Zext = 0, + Sext = 1, + ByVal = 2, + Sret = 3, + NoAlias = 4, + NoCapture = 5, + NoWrite = 6, + NoReadWrite = 7, + } + + public enum Decoration + { + RelaxedPrecision = 0, + SpecId = 1, + Block = 2, + BufferBlock = 3, + RowMajor = 4, + ColMajor = 5, + ArrayStride = 6, + MatrixStride = 7, + GLSLShared = 8, + GLSLPacked = 9, + CPacked = 10, + BuiltIn = 11, + NoPerspective = 13, + Flat = 14, + Patch = 15, + Centroid = 16, + Sample = 17, + Invariant = 18, + Restrict = 19, + Aliased = 20, + Volatile = 21, + Constant = 22, + Coherent = 23, + NonWritable = 24, + NonReadable = 25, + Uniform = 26, + UniformId = 27, + SaturatedConversion = 28, + Stream = 29, + Location = 30, + Component = 31, + Index = 32, + Binding = 33, + DescriptorSet = 34, + Offset = 35, + XfbBuffer = 36, + XfbStride = 37, + FuncParamAttr = 38, + FPRoundingMode = 39, + FPFastMathMode = 40, + LinkageAttributes = 41, + NoContraction = 42, + InputAttachmentIndex = 43, + Alignment = 44, + MaxByteOffset = 45, + AlignmentId = 46, + MaxByteOffsetId = 47, + NoSignedWrap = 4469, + NoUnsignedWrap = 4470, + ExplicitInterpAMD = 4999, + OverrideCoverageNV = 5248, + PassthroughNV = 5250, + ViewportRelativeNV = 5252, + SecondaryViewportRelativeNV = 5256, + PerPrimitiveNV = 5271, + PerViewNV = 5272, + PerTaskNV = 5273, + PerVertexNV = 5285, + NonUniform = 5300, + NonUniformEXT = NonUniform, + RestrictPointer = 5355, + RestrictPointerEXT = RestrictPointer, + AliasedPointer = 5356, + AliasedPointerEXT = AliasedPointer, + ReferencedIndirectlyINTEL = 5602, + CounterBuffer = 5634, + HlslCounterBufferGOOGLE = CounterBuffer, + HlslSemanticGOOGLE = 5635, + UserSemantic = HlslSemanticGOOGLE, + UserTypeGOOGLE = 5636, + RegisterINTEL = 5825, + MemoryINTEL = 5826, + NumbanksINTEL = 5827, + BankwidthINTEL = 5828, + MaxPrivateCopiesINTEL = 5829, + SinglepumpINTEL = 5830, + DoublepumpINTEL = 5831, + MaxReplicatesINTEL = 5832, + SimpleDualPortINTEL = 5833, + MergeINTEL = 5834, + BankBitsINTEL = 5835, + ForcePow2DepthINTEL = 5836, + } + + public enum BuiltIn + { + Position = 0, + PointSize = 1, + ClipDistance = 3, + CullDistance = 4, + VertexId = 5, + InstanceId = 6, + PrimitiveId = 7, + InvocationId = 8, + Layer = 9, + ViewportIndex = 10, + TessLevelOuter = 11, + TessLevelInner = 12, + TessCoord = 13, + PatchVertices = 14, + FragCoord = 15, + PointCoord = 16, + FrontFacing = 17, + SampleId = 18, + SamplePosition = 19, + SampleMask = 20, + FragDepth = 22, + HelperInvocation = 23, + NumWorkgroups = 24, + WorkgroupSize = 25, + WorkgroupId = 26, + LocalInvocationId = 27, + GlobalInvocationId = 28, + LocalInvocationIndex = 29, + WorkDim = 30, + GlobalSize = 31, + EnqueuedWorkgroupSize = 32, + GlobalOffset = 33, + GlobalLinearId = 34, + SubgroupSize = 36, + SubgroupMaxSize = 37, + NumSubgroups = 38, + NumEnqueuedSubgroups = 39, + SubgroupId = 40, + SubgroupLocalInvocationId = 41, + VertexIndex = 42, + InstanceIndex = 43, + SubgroupEqMask = 4416, + SubgroupEqMaskKHR = SubgroupEqMask, + SubgroupGeMask = 4417, + SubgroupGeMaskKHR = SubgroupGeMask, + SubgroupGtMask = 4418, + SubgroupGtMaskKHR = SubgroupGtMask, + SubgroupLeMask = 4419, + SubgroupLeMaskKHR = SubgroupLeMask, + SubgroupLtMask = 4420, + SubgroupLtMaskKHR = SubgroupLtMask, + BaseVertex = 4424, + BaseInstance = 4425, + DrawIndex = 4426, + PrimitiveShadingRateKHR = 4432, + DeviceIndex = 4438, + ViewIndex = 4440, + ShadingRateKHR = 4444, + BaryCoordNoPerspAMD = 4992, + BaryCoordNoPerspCentroidAMD = 4993, + BaryCoordNoPerspSampleAMD = 4994, + BaryCoordSmoothAMD = 4995, + BaryCoordSmoothCentroidAMD = 4996, + BaryCoordSmoothSampleAMD = 4997, + BaryCoordPullModelAMD = 4998, + FragStencilRefEXT = 5014, + ViewportMaskNV = 5253, + SecondaryPositionNV = 5257, + SecondaryViewportMaskNV = 5258, + PositionPerViewNV = 5261, + ViewportMaskPerViewNV = 5262, + FullyCoveredEXT = 5264, + TaskCountNV = 5274, + PrimitiveCountNV = 5275, + PrimitiveIndicesNV = 5276, + ClipDistancePerViewNV = 5277, + CullDistancePerViewNV = 5278, + LayerPerViewNV = 5279, + MeshViewCountNV = 5280, + MeshViewIndicesNV = 5281, + BaryCoordNV = 5286, + BaryCoordNoPerspNV = 5287, + FragSizeEXT = 5292, + FragmentSizeNV = FragSizeEXT, + FragInvocationCountEXT = 5293, + InvocationsPerPixelNV = FragInvocationCountEXT, + LaunchIdKHR = 5319, + LaunchIdNV = LaunchIdKHR, + LaunchSizeKHR = 5320, + LaunchSizeNV = LaunchSizeKHR, + WorldRayOriginKHR = 5321, + WorldRayOriginNV = WorldRayOriginKHR, + WorldRayDirectionKHR = 5322, + WorldRayDirectionNV = WorldRayDirectionKHR, + ObjectRayOriginKHR = 5323, + ObjectRayOriginNV = ObjectRayOriginKHR, + ObjectRayDirectionKHR = 5324, + ObjectRayDirectionNV = ObjectRayDirectionKHR, + RayTminKHR = 5325, + RayTminNV = RayTminKHR, + RayTmaxKHR = 5326, + RayTmaxNV = RayTmaxKHR, + InstanceCustomIndexKHR = 5327, + InstanceCustomIndexNV = InstanceCustomIndexKHR, + ObjectToWorldKHR = 5330, + ObjectToWorldNV = ObjectToWorldKHR, + WorldToObjectKHR = 5331, + WorldToObjectNV = WorldToObjectKHR, + HitTNV = 5332, + HitKindKHR = 5333, + HitKindNV = HitKindKHR, + IncomingRayFlagsKHR = 5351, + IncomingRayFlagsNV = IncomingRayFlagsKHR, + RayGeometryIndexKHR = 5352, + WarpsPerSMNV = 5374, + SMCountNV = 5375, + WarpIDNV = 5376, + SMIDNV = 5377, + } + + public enum SelectionControlShift + { + Flatten = 0, + DontFlatten = 1, + } + + public enum SelectionControlMask + { + MaskNone = 0, + Flatten = 0x00000001, + DontFlatten = 0x00000002, + } + + public enum LoopControlShift + { + Unroll = 0, + DontUnroll = 1, + DependencyInfinite = 2, + DependencyLength = 3, + MinIterations = 4, + MaxIterations = 5, + IterationMultiple = 6, + PeelCount = 7, + PartialCount = 8, + InitiationIntervalINTEL = 16, + MaxConcurrencyINTEL = 17, + DependencyArrayINTEL = 18, + PipelineEnableINTEL = 19, + LoopCoalesceINTEL = 20, + MaxInterleavingINTEL = 21, + SpeculatedIterationsINTEL = 22, + } + + public enum LoopControlMask + { + MaskNone = 0, + Unroll = 0x00000001, + DontUnroll = 0x00000002, + DependencyInfinite = 0x00000004, + DependencyLength = 0x00000008, + MinIterations = 0x00000010, + MaxIterations = 0x00000020, + IterationMultiple = 0x00000040, + PeelCount = 0x00000080, + PartialCount = 0x00000100, + InitiationIntervalINTEL = 0x00010000, + MaxConcurrencyINTEL = 0x00020000, + DependencyArrayINTEL = 0x00040000, + PipelineEnableINTEL = 0x00080000, + LoopCoalesceINTEL = 0x00100000, + MaxInterleavingINTEL = 0x00200000, + SpeculatedIterationsINTEL = 0x00400000, + } + + public enum FunctionControlShift + { + Inline = 0, + DontInline = 1, + Pure = 2, + Const = 3, + } + + public enum FunctionControlMask + { + MaskNone = 0, + Inline = 0x00000001, + DontInline = 0x00000002, + Pure = 0x00000004, + Const = 0x00000008, + } + + public enum MemorySemanticsShift + { + Acquire = 1, + Release = 2, + AcquireRelease = 3, + SequentiallyConsistent = 4, + UniformMemory = 6, + SubgroupMemory = 7, + WorkgroupMemory = 8, + CrossWorkgroupMemory = 9, + AtomicCounterMemory = 10, + ImageMemory = 11, + OutputMemory = 12, + OutputMemoryKHR = OutputMemory, + MakeAvailable = 13, + MakeAvailableKHR = MakeAvailable, + MakeVisible = 14, + MakeVisibleKHR = MakeVisible, + Volatile = 15, + } + + public enum MemorySemanticsMask + { + MaskNone = 0, + Acquire = 0x00000002, + Release = 0x00000004, + AcquireRelease = 0x00000008, + SequentiallyConsistent = 0x00000010, + UniformMemory = 0x00000040, + SubgroupMemory = 0x00000080, + WorkgroupMemory = 0x00000100, + CrossWorkgroupMemory = 0x00000200, + AtomicCounterMemory = 0x00000400, + ImageMemory = 0x00000800, + OutputMemory = 0x00001000, + OutputMemoryKHR = OutputMemory, + MakeAvailable = 0x00002000, + MakeAvailableKHR = MakeAvailable, + MakeVisible = 0x00004000, + MakeVisibleKHR = MakeVisible, + Volatile = 0x00008000, + } + + public enum MemoryAccessShift + { + Volatile = 0, + Aligned = 1, + Nontemporal = 2, + MakePointerAvailable = 3, + MakePointerAvailableKHR = MakePointerAvailable, + MakePointerVisible = 4, + MakePointerVisibleKHR = MakePointerVisible, + NonPrivatePointer = 5, + NonPrivatePointerKHR = NonPrivatePointer, + } + + public enum MemoryAccessMask + { + MaskNone = 0, + Volatile = 0x00000001, + Aligned = 0x00000002, + Nontemporal = 0x00000004, + MakePointerAvailable = 0x00000008, + MakePointerAvailableKHR = MakePointerAvailable, + MakePointerVisible = 0x00000010, + MakePointerVisibleKHR = MakePointerVisible, + NonPrivatePointer = 0x00000020, + NonPrivatePointerKHR = NonPrivatePointer, + } + + public enum Scope + { + CrossDevice = 0, + Device = 1, + Workgroup = 2, + Subgroup = 3, + Invocation = 4, + QueueFamily = 5, + QueueFamilyKHR = QueueFamily, + ShaderCallKHR = 6, + } + + public enum GroupOperation + { + Reduce = 0, + InclusiveScan = 1, + ExclusiveScan = 2, + ClusteredReduce = 3, + PartitionedReduceNV = 6, + PartitionedInclusiveScanNV = 7, + PartitionedExclusiveScanNV = 8, + } + + public enum KernelEnqueueFlags + { + NoWait = 0, + WaitKernel = 1, + WaitWorkGroup = 2, + } + + public enum KernelProfilingInfoShift + { + CmdExecTime = 0, + } + + public enum KernelProfilingInfoMask + { + MaskNone = 0, + CmdExecTime = 0x00000001, + } + + public enum Capability + { + Matrix = 0, + Shader = 1, + Geometry = 2, + Tessellation = 3, + Addresses = 4, + Linkage = 5, + Kernel = 6, + Vector16 = 7, + Float16Buffer = 8, + Float16 = 9, + Float64 = 10, + Int64 = 11, + Int64Atomics = 12, + ImageBasic = 13, + ImageReadWrite = 14, + ImageMipmap = 15, + Pipes = 17, + Groups = 18, + DeviceEnqueue = 19, + LiteralSampler = 20, + AtomicStorage = 21, + Int16 = 22, + TessellationPointSize = 23, + GeometryPointSize = 24, + ImageGatherExtended = 25, + StorageImageMultisample = 27, + UniformBufferArrayDynamicIndexing = 28, + SampledImageArrayDynamicIndexing = 29, + StorageBufferArrayDynamicIndexing = 30, + StorageImageArrayDynamicIndexing = 31, + ClipDistance = 32, + CullDistance = 33, + ImageCubeArray = 34, + SampleRateShading = 35, + ImageRect = 36, + SampledRect = 37, + GenericPointer = 38, + Int8 = 39, + InputAttachment = 40, + SparseResidency = 41, + MinLod = 42, + Sampled1D = 43, + Image1D = 44, + SampledCubeArray = 45, + SampledBuffer = 46, + ImageBuffer = 47, + ImageMSArray = 48, + StorageImageExtendedFormats = 49, + ImageQuery = 50, + DerivativeControl = 51, + InterpolationFunction = 52, + TransformFeedback = 53, + GeometryStreams = 54, + StorageImageReadWithoutFormat = 55, + StorageImageWriteWithoutFormat = 56, + MultiViewport = 57, + SubgroupDispatch = 58, + NamedBarrier = 59, + PipeStorage = 60, + GroupNonUniform = 61, + GroupNonUniformVote = 62, + GroupNonUniformArithmetic = 63, + GroupNonUniformBallot = 64, + GroupNonUniformShuffle = 65, + GroupNonUniformShuffleRelative = 66, + GroupNonUniformClustered = 67, + GroupNonUniformQuad = 68, + ShaderLayer = 69, + ShaderViewportIndex = 70, + FragmentShadingRateKHR = 4422, + SubgroupBallotKHR = 4423, + DrawParameters = 4427, + SubgroupVoteKHR = 4431, + StorageBuffer16BitAccess = 4433, + StorageUniformBufferBlock16 = StorageBuffer16BitAccess, + StorageUniform16 = 4434, + UniformAndStorageBuffer16BitAccess = StorageUniform16, + StoragePushConstant16 = 4435, + StorageInputOutput16 = 4436, + DeviceGroup = 4437, + MultiView = 4439, + VariablePointersStorageBuffer = 4441, + VariablePointers = 4442, + AtomicStorageOps = 4445, + SampleMaskPostDepthCoverage = 4447, + StorageBuffer8BitAccess = 4448, + UniformAndStorageBuffer8BitAccess = 4449, + StoragePushConstant8 = 4450, + DenormPreserve = 4464, + DenormFlushToZero = 4465, + SignedZeroInfNanPreserve = 4466, + RoundingModeRTE = 4467, + RoundingModeRTZ = 4468, + RayQueryProvisionalKHR = 4471, + RayQueryKHR = 4472, + RayTraversalPrimitiveCullingKHR = 4478, + RayTracingKHR = 4479, + Float16ImageAMD = 5008, + ImageGatherBiasLodAMD = 5009, + FragmentMaskAMD = 5010, + StencilExportEXT = 5013, + ImageReadWriteLodAMD = 5015, + Int64ImageEXT = 5016, + ShaderClockKHR = 5055, + SampleMaskOverrideCoverageNV = 5249, + GeometryShaderPassthroughNV = 5251, + ShaderViewportIndexLayerEXT = 5254, + ShaderViewportIndexLayerNV = ShaderViewportIndexLayerEXT, + ShaderViewportMaskNV = 5255, + ShaderStereoViewNV = 5259, + PerViewAttributesNV = 5260, + FragmentFullyCoveredEXT = 5265, + MeshShadingNV = 5266, + ImageFootprintNV = 5282, + FragmentBarycentricNV = 5284, + ComputeDerivativeGroupQuadsNV = 5288, + FragmentDensityEXT = 5291, + ShadingRateNV = FragmentDensityEXT, + GroupNonUniformPartitionedNV = 5297, + ShaderNonUniform = 5301, + ShaderNonUniformEXT = ShaderNonUniform, + RuntimeDescriptorArray = 5302, + RuntimeDescriptorArrayEXT = RuntimeDescriptorArray, + InputAttachmentArrayDynamicIndexing = 5303, + InputAttachmentArrayDynamicIndexingEXT = InputAttachmentArrayDynamicIndexing, + UniformTexelBufferArrayDynamicIndexing = 5304, + UniformTexelBufferArrayDynamicIndexingEXT = UniformTexelBufferArrayDynamicIndexing, + StorageTexelBufferArrayDynamicIndexing = 5305, + StorageTexelBufferArrayDynamicIndexingEXT = StorageTexelBufferArrayDynamicIndexing, + UniformBufferArrayNonUniformIndexing = 5306, + UniformBufferArrayNonUniformIndexingEXT = UniformBufferArrayNonUniformIndexing, + SampledImageArrayNonUniformIndexing = 5307, + SampledImageArrayNonUniformIndexingEXT = SampledImageArrayNonUniformIndexing, + StorageBufferArrayNonUniformIndexing = 5308, + StorageBufferArrayNonUniformIndexingEXT = StorageBufferArrayNonUniformIndexing, + StorageImageArrayNonUniformIndexing = 5309, + StorageImageArrayNonUniformIndexingEXT = StorageImageArrayNonUniformIndexing, + InputAttachmentArrayNonUniformIndexing = 5310, + InputAttachmentArrayNonUniformIndexingEXT = InputAttachmentArrayNonUniformIndexing, + UniformTexelBufferArrayNonUniformIndexing = 5311, + UniformTexelBufferArrayNonUniformIndexingEXT = UniformTexelBufferArrayNonUniformIndexing, + StorageTexelBufferArrayNonUniformIndexing = 5312, + StorageTexelBufferArrayNonUniformIndexingEXT = StorageTexelBufferArrayNonUniformIndexing, + RayTracingNV = 5340, + VulkanMemoryModel = 5345, + VulkanMemoryModelKHR = VulkanMemoryModel, + VulkanMemoryModelDeviceScope = 5346, + VulkanMemoryModelDeviceScopeKHR = VulkanMemoryModelDeviceScope, + PhysicalStorageBufferAddresses = 5347, + PhysicalStorageBufferAddressesEXT = PhysicalStorageBufferAddresses, + ComputeDerivativeGroupLinearNV = 5350, + RayTracingProvisionalKHR = 5353, + CooperativeMatrixNV = 5357, + FragmentShaderSampleInterlockEXT = 5363, + FragmentShaderShadingRateInterlockEXT = 5372, + ShaderSMBuiltinsNV = 5373, + FragmentShaderPixelInterlockEXT = 5378, + DemoteToHelperInvocationEXT = 5379, + SubgroupShuffleINTEL = 5568, + SubgroupBufferBlockIOINTEL = 5569, + SubgroupImageBlockIOINTEL = 5570, + SubgroupImageMediaBlockIOINTEL = 5579, + IntegerFunctions2INTEL = 5584, + FunctionPointersINTEL = 5603, + IndirectReferencesINTEL = 5604, + SubgroupAvcMotionEstimationINTEL = 5696, + SubgroupAvcMotionEstimationIntraINTEL = 5697, + SubgroupAvcMotionEstimationChromaINTEL = 5698, + FPGAMemoryAttributesINTEL = 5824, + UnstructuredLoopControlsINTEL = 5886, + FPGALoopControlsINTEL = 5888, + KernelAttributesINTEL = 5892, + FPGAKernelAttributesINTEL = 5897, + BlockingPipesINTEL = 5945, + FPGARegINTEL = 5948, + AtomicFloat32AddEXT = 6033, + AtomicFloat64AddEXT = 6034, + } + + public enum RayFlagsShift + { + OpaqueKHR = 0, + NoOpaqueKHR = 1, + TerminateOnFirstHitKHR = 2, + SkipClosestHitShaderKHR = 3, + CullBackFacingTrianglesKHR = 4, + CullFrontFacingTrianglesKHR = 5, + CullOpaqueKHR = 6, + CullNoOpaqueKHR = 7, + SkipTrianglesKHR = 8, + SkipAABBsKHR = 9, + } + + public enum RayFlagsMask + { + MaskNone = 0, + OpaqueKHR = 0x00000001, + NoOpaqueKHR = 0x00000002, + TerminateOnFirstHitKHR = 0x00000004, + SkipClosestHitShaderKHR = 0x00000008, + CullBackFacingTrianglesKHR = 0x00000010, + CullFrontFacingTrianglesKHR = 0x00000020, + CullOpaqueKHR = 0x00000040, + CullNoOpaqueKHR = 0x00000080, + SkipTrianglesKHR = 0x00000100, + SkipAABBsKHR = 0x00000200, + } + + public enum RayQueryIntersection + { + RayQueryCandidateIntersectionKHR = 0, + RayQueryCommittedIntersectionKHR = 1, + } + + public enum RayQueryCommittedIntersectionType + { + RayQueryCommittedIntersectionNoneKHR = 0, + RayQueryCommittedIntersectionTriangleKHR = 1, + RayQueryCommittedIntersectionGeneratedKHR = 2, + } + + public enum RayQueryCandidateIntersectionType + { + RayQueryCandidateIntersectionTriangleKHR = 0, + RayQueryCandidateIntersectionAABBKHR = 1, + } + + public enum FragmentShadingRateShift + { + Vertical2Pixels = 0, + Vertical4Pixels = 1, + Horizontal2Pixels = 2, + Horizontal4Pixels = 3, + } + + public enum FragmentShadingRateMask + { + MaskNone = 0, + Vertical2Pixels = 0x00000001, + Vertical4Pixels = 0x00000002, + Horizontal2Pixels = 0x00000004, + Horizontal4Pixels = 0x00000008, + } + + public enum Op + { + OpNop = 0, + OpUndef = 1, + OpSourceContinued = 2, + OpSource = 3, + OpSourceExtension = 4, + OpName = 5, + OpMemberName = 6, + OpString = 7, + OpLine = 8, + OpExtension = 10, + OpExtInstImport = 11, + OpExtInst = 12, + OpMemoryModel = 14, + OpEntryPoint = 15, + OpExecutionMode = 16, + OpCapability = 17, + OpTypeVoid = 19, + OpTypeBool = 20, + OpTypeInt = 21, + OpTypeFloat = 22, + OpTypeVector = 23, + OpTypeMatrix = 24, + OpTypeImage = 25, + OpTypeSampler = 26, + OpTypeSampledImage = 27, + OpTypeArray = 28, + OpTypeRuntimeArray = 29, + OpTypeStruct = 30, + OpTypeOpaque = 31, + OpTypePointer = 32, + OpTypeFunction = 33, + OpTypeEvent = 34, + OpTypeDeviceEvent = 35, + OpTypeReserveId = 36, + OpTypeQueue = 37, + OpTypePipe = 38, + OpTypeForwardPointer = 39, + OpConstantTrue = 41, + OpConstantFalse = 42, + OpConstant = 43, + OpConstantComposite = 44, + OpConstantSampler = 45, + OpConstantNull = 46, + OpSpecConstantTrue = 48, + OpSpecConstantFalse = 49, + OpSpecConstant = 50, + OpSpecConstantComposite = 51, + OpSpecConstantOp = 52, + OpFunction = 54, + OpFunctionParameter = 55, + OpFunctionEnd = 56, + OpFunctionCall = 57, + OpVariable = 59, + OpImageTexelPointer = 60, + OpLoad = 61, + OpStore = 62, + OpCopyMemory = 63, + OpCopyMemorySized = 64, + OpAccessChain = 65, + OpInBoundsAccessChain = 66, + OpPtrAccessChain = 67, + OpArrayLength = 68, + OpGenericPtrMemSemantics = 69, + OpInBoundsPtrAccessChain = 70, + OpDecorate = 71, + OpMemberDecorate = 72, + OpDecorationGroup = 73, + OpGroupDecorate = 74, + OpGroupMemberDecorate = 75, + OpVectorExtractDynamic = 77, + OpVectorInsertDynamic = 78, + OpVectorShuffle = 79, + OpCompositeConstruct = 80, + OpCompositeExtract = 81, + OpCompositeInsert = 82, + OpCopyObject = 83, + OpTranspose = 84, + OpSampledImage = 86, + OpImageSampleImplicitLod = 87, + OpImageSampleExplicitLod = 88, + OpImageSampleDrefImplicitLod = 89, + OpImageSampleDrefExplicitLod = 90, + OpImageSampleProjImplicitLod = 91, + OpImageSampleProjExplicitLod = 92, + OpImageSampleProjDrefImplicitLod = 93, + OpImageSampleProjDrefExplicitLod = 94, + OpImageFetch = 95, + OpImageGather = 96, + OpImageDrefGather = 97, + OpImageRead = 98, + OpImageWrite = 99, + OpImage = 100, + OpImageQueryFormat = 101, + OpImageQueryOrder = 102, + OpImageQuerySizeLod = 103, + OpImageQuerySize = 104, + OpImageQueryLod = 105, + OpImageQueryLevels = 106, + OpImageQuerySamples = 107, + OpConvertFToU = 109, + OpConvertFToS = 110, + OpConvertSToF = 111, + OpConvertUToF = 112, + OpUConvert = 113, + OpSConvert = 114, + OpFConvert = 115, + OpQuantizeToF16 = 116, + OpConvertPtrToU = 117, + OpSatConvertSToU = 118, + OpSatConvertUToS = 119, + OpConvertUToPtr = 120, + OpPtrCastToGeneric = 121, + OpGenericCastToPtr = 122, + OpGenericCastToPtrExplicit = 123, + OpBitcast = 124, + OpSNegate = 126, + OpFNegate = 127, + OpIAdd = 128, + OpFAdd = 129, + OpISub = 130, + OpFSub = 131, + OpIMul = 132, + OpFMul = 133, + OpUDiv = 134, + OpSDiv = 135, + OpFDiv = 136, + OpUMod = 137, + OpSRem = 138, + OpSMod = 139, + OpFRem = 140, + OpFMod = 141, + OpVectorTimesScalar = 142, + OpMatrixTimesScalar = 143, + OpVectorTimesMatrix = 144, + OpMatrixTimesVector = 145, + OpMatrixTimesMatrix = 146, + OpOuterProduct = 147, + OpDot = 148, + OpIAddCarry = 149, + OpISubBorrow = 150, + OpUMulExtended = 151, + OpSMulExtended = 152, + OpAny = 154, + OpAll = 155, + OpIsNan = 156, + OpIsInf = 157, + OpIsFinite = 158, + OpIsNormal = 159, + OpSignBitSet = 160, + OpLessOrGreater = 161, + OpOrdered = 162, + OpUnordered = 163, + OpLogicalEqual = 164, + OpLogicalNotEqual = 165, + OpLogicalOr = 166, + OpLogicalAnd = 167, + OpLogicalNot = 168, + OpSelect = 169, + OpIEqual = 170, + OpINotEqual = 171, + OpUGreaterThan = 172, + OpSGreaterThan = 173, + OpUGreaterThanEqual = 174, + OpSGreaterThanEqual = 175, + OpULessThan = 176, + OpSLessThan = 177, + OpULessThanEqual = 178, + OpSLessThanEqual = 179, + OpFOrdEqual = 180, + OpFUnordEqual = 181, + OpFOrdNotEqual = 182, + OpFUnordNotEqual = 183, + OpFOrdLessThan = 184, + OpFUnordLessThan = 185, + OpFOrdGreaterThan = 186, + OpFUnordGreaterThan = 187, + OpFOrdLessThanEqual = 188, + OpFUnordLessThanEqual = 189, + OpFOrdGreaterThanEqual = 190, + OpFUnordGreaterThanEqual = 191, + OpShiftRightLogical = 194, + OpShiftRightArithmetic = 195, + OpShiftLeftLogical = 196, + OpBitwiseOr = 197, + OpBitwiseXor = 198, + OpBitwiseAnd = 199, + OpNot = 200, + OpBitFieldInsert = 201, + OpBitFieldSExtract = 202, + OpBitFieldUExtract = 203, + OpBitReverse = 204, + OpBitCount = 205, + OpDPdx = 207, + OpDPdy = 208, + OpFwidth = 209, + OpDPdxFine = 210, + OpDPdyFine = 211, + OpFwidthFine = 212, + OpDPdxCoarse = 213, + OpDPdyCoarse = 214, + OpFwidthCoarse = 215, + OpEmitVertex = 218, + OpEndPrimitive = 219, + OpEmitStreamVertex = 220, + OpEndStreamPrimitive = 221, + OpControlBarrier = 224, + OpMemoryBarrier = 225, + OpAtomicLoad = 227, + OpAtomicStore = 228, + OpAtomicExchange = 229, + OpAtomicCompareExchange = 230, + OpAtomicCompareExchangeWeak = 231, + OpAtomicIIncrement = 232, + OpAtomicIDecrement = 233, + OpAtomicIAdd = 234, + OpAtomicISub = 235, + OpAtomicSMin = 236, + OpAtomicUMin = 237, + OpAtomicSMax = 238, + OpAtomicUMax = 239, + OpAtomicAnd = 240, + OpAtomicOr = 241, + OpAtomicXor = 242, + OpPhi = 245, + OpLoopMerge = 246, + OpSelectionMerge = 247, + OpLabel = 248, + OpBranch = 249, + OpBranchConditional = 250, + OpSwitch = 251, + OpKill = 252, + OpReturn = 253, + OpReturnValue = 254, + OpUnreachable = 255, + OpLifetimeStart = 256, + OpLifetimeStop = 257, + OpGroupAsyncCopy = 259, + OpGroupWaitEvents = 260, + OpGroupAll = 261, + OpGroupAny = 262, + OpGroupBroadcast = 263, + OpGroupIAdd = 264, + OpGroupFAdd = 265, + OpGroupFMin = 266, + OpGroupUMin = 267, + OpGroupSMin = 268, + OpGroupFMax = 269, + OpGroupUMax = 270, + OpGroupSMax = 271, + OpReadPipe = 274, + OpWritePipe = 275, + OpReservedReadPipe = 276, + OpReservedWritePipe = 277, + OpReserveReadPipePackets = 278, + OpReserveWritePipePackets = 279, + OpCommitReadPipe = 280, + OpCommitWritePipe = 281, + OpIsValidReserveId = 282, + OpGetNumPipePackets = 283, + OpGetMaxPipePackets = 284, + OpGroupReserveReadPipePackets = 285, + OpGroupReserveWritePipePackets = 286, + OpGroupCommitReadPipe = 287, + OpGroupCommitWritePipe = 288, + OpEnqueueMarker = 291, + OpEnqueueKernel = 292, + OpGetKernelNDrangeSubGroupCount = 293, + OpGetKernelNDrangeMaxSubGroupSize = 294, + OpGetKernelWorkGroupSize = 295, + OpGetKernelPreferredWorkGroupSizeMultiple = 296, + OpRetainEvent = 297, + OpReleaseEvent = 298, + OpCreateUserEvent = 299, + OpIsValidEvent = 300, + OpSetUserEventStatus = 301, + OpCaptureEventProfilingInfo = 302, + OpGetDefaultQueue = 303, + OpBuildNDRange = 304, + OpImageSparseSampleImplicitLod = 305, + OpImageSparseSampleExplicitLod = 306, + OpImageSparseSampleDrefImplicitLod = 307, + OpImageSparseSampleDrefExplicitLod = 308, + OpImageSparseSampleProjImplicitLod = 309, + OpImageSparseSampleProjExplicitLod = 310, + OpImageSparseSampleProjDrefImplicitLod = 311, + OpImageSparseSampleProjDrefExplicitLod = 312, + OpImageSparseFetch = 313, + OpImageSparseGather = 314, + OpImageSparseDrefGather = 315, + OpImageSparseTexelsResident = 316, + OpNoLine = 317, + OpAtomicFlagTestAndSet = 318, + OpAtomicFlagClear = 319, + OpImageSparseRead = 320, + OpSizeOf = 321, + OpTypePipeStorage = 322, + OpConstantPipeStorage = 323, + OpCreatePipeFromPipeStorage = 324, + OpGetKernelLocalSizeForSubgroupCount = 325, + OpGetKernelMaxNumSubgroups = 326, + OpTypeNamedBarrier = 327, + OpNamedBarrierInitialize = 328, + OpMemoryNamedBarrier = 329, + OpModuleProcessed = 330, + OpExecutionModeId = 331, + OpDecorateId = 332, + OpGroupNonUniformElect = 333, + OpGroupNonUniformAll = 334, + OpGroupNonUniformAny = 335, + OpGroupNonUniformAllEqual = 336, + OpGroupNonUniformBroadcast = 337, + OpGroupNonUniformBroadcastFirst = 338, + OpGroupNonUniformBallot = 339, + OpGroupNonUniformInverseBallot = 340, + OpGroupNonUniformBallotBitExtract = 341, + OpGroupNonUniformBallotBitCount = 342, + OpGroupNonUniformBallotFindLSB = 343, + OpGroupNonUniformBallotFindMSB = 344, + OpGroupNonUniformShuffle = 345, + OpGroupNonUniformShuffleXor = 346, + OpGroupNonUniformShuffleUp = 347, + OpGroupNonUniformShuffleDown = 348, + OpGroupNonUniformIAdd = 349, + OpGroupNonUniformFAdd = 350, + OpGroupNonUniformIMul = 351, + OpGroupNonUniformFMul = 352, + OpGroupNonUniformSMin = 353, + OpGroupNonUniformUMin = 354, + OpGroupNonUniformFMin = 355, + OpGroupNonUniformSMax = 356, + OpGroupNonUniformUMax = 357, + OpGroupNonUniformFMax = 358, + OpGroupNonUniformBitwiseAnd = 359, + OpGroupNonUniformBitwiseOr = 360, + OpGroupNonUniformBitwiseXor = 361, + OpGroupNonUniformLogicalAnd = 362, + OpGroupNonUniformLogicalOr = 363, + OpGroupNonUniformLogicalXor = 364, + OpGroupNonUniformQuadBroadcast = 365, + OpGroupNonUniformQuadSwap = 366, + OpCopyLogical = 400, + OpPtrEqual = 401, + OpPtrNotEqual = 402, + OpPtrDiff = 403, + OpTerminateInvocation = 4416, + OpSubgroupBallotKHR = 4421, + OpSubgroupFirstInvocationKHR = 4422, + OpSubgroupAllKHR = 4428, + OpSubgroupAnyKHR = 4429, + OpSubgroupAllEqualKHR = 4430, + OpSubgroupReadInvocationKHR = 4432, + OpTraceRayKHR = 4445, + OpExecuteCallableKHR = 4446, + OpConvertUToAccelerationStructureKHR = 4447, + OpIgnoreIntersectionKHR = 4448, + OpTerminateRayKHR = 4449, + OpTypeRayQueryKHR = 4472, + OpRayQueryInitializeKHR = 4473, + OpRayQueryTerminateKHR = 4474, + OpRayQueryGenerateIntersectionKHR = 4475, + OpRayQueryConfirmIntersectionKHR = 4476, + OpRayQueryProceedKHR = 4477, + OpRayQueryGetIntersectionTypeKHR = 4479, + OpGroupIAddNonUniformAMD = 5000, + OpGroupFAddNonUniformAMD = 5001, + OpGroupFMinNonUniformAMD = 5002, + OpGroupUMinNonUniformAMD = 5003, + OpGroupSMinNonUniformAMD = 5004, + OpGroupFMaxNonUniformAMD = 5005, + OpGroupUMaxNonUniformAMD = 5006, + OpGroupSMaxNonUniformAMD = 5007, + OpFragmentMaskFetchAMD = 5011, + OpFragmentFetchAMD = 5012, + OpReadClockKHR = 5056, + OpImageSampleFootprintNV = 5283, + OpGroupNonUniformPartitionNV = 5296, + OpWritePackedPrimitiveIndices4x8NV = 5299, + OpReportIntersectionKHR = 5334, + OpReportIntersectionNV = OpReportIntersectionKHR, + OpIgnoreIntersectionNV = 5335, + OpTerminateRayNV = 5336, + OpTraceNV = 5337, + OpTypeAccelerationStructureKHR = 5341, + OpTypeAccelerationStructureNV = OpTypeAccelerationStructureKHR, + OpExecuteCallableNV = 5344, + OpTypeCooperativeMatrixNV = 5358, + OpCooperativeMatrixLoadNV = 5359, + OpCooperativeMatrixStoreNV = 5360, + OpCooperativeMatrixMulAddNV = 5361, + OpCooperativeMatrixLengthNV = 5362, + OpBeginInvocationInterlockEXT = 5364, + OpEndInvocationInterlockEXT = 5365, + OpDemoteToHelperInvocationEXT = 5380, + OpIsHelperInvocationEXT = 5381, + OpSubgroupShuffleINTEL = 5571, + OpSubgroupShuffleDownINTEL = 5572, + OpSubgroupShuffleUpINTEL = 5573, + OpSubgroupShuffleXorINTEL = 5574, + OpSubgroupBlockReadINTEL = 5575, + OpSubgroupBlockWriteINTEL = 5576, + OpSubgroupImageBlockReadINTEL = 5577, + OpSubgroupImageBlockWriteINTEL = 5578, + OpSubgroupImageMediaBlockReadINTEL = 5580, + OpSubgroupImageMediaBlockWriteINTEL = 5581, + OpUCountLeadingZerosINTEL = 5585, + OpUCountTrailingZerosINTEL = 5586, + OpAbsISubINTEL = 5587, + OpAbsUSubINTEL = 5588, + OpIAddSatINTEL = 5589, + OpUAddSatINTEL = 5590, + OpIAverageINTEL = 5591, + OpUAverageINTEL = 5592, + OpIAverageRoundedINTEL = 5593, + OpUAverageRoundedINTEL = 5594, + OpISubSatINTEL = 5595, + OpUSubSatINTEL = 5596, + OpIMul32x16INTEL = 5597, + OpUMul32x16INTEL = 5598, + OpFunctionPointerINTEL = 5600, + OpFunctionPointerCallINTEL = 5601, + OpDecorateString = 5632, + OpDecorateStringGOOGLE = OpDecorateString, + OpMemberDecorateString = 5633, + OpMemberDecorateStringGOOGLE = OpMemberDecorateString, + OpVmeImageINTEL = 5699, + OpTypeVmeImageINTEL = 5700, + OpTypeAvcImePayloadINTEL = 5701, + OpTypeAvcRefPayloadINTEL = 5702, + OpTypeAvcSicPayloadINTEL = 5703, + OpTypeAvcMcePayloadINTEL = 5704, + OpTypeAvcMceResultINTEL = 5705, + OpTypeAvcImeResultINTEL = 5706, + OpTypeAvcImeResultSingleReferenceStreamoutINTEL = 5707, + OpTypeAvcImeResultDualReferenceStreamoutINTEL = 5708, + OpTypeAvcImeSingleReferenceStreaminINTEL = 5709, + OpTypeAvcImeDualReferenceStreaminINTEL = 5710, + OpTypeAvcRefResultINTEL = 5711, + OpTypeAvcSicResultINTEL = 5712, + OpSubgroupAvcMceGetDefaultInterBaseMultiReferencePenaltyINTEL = 5713, + OpSubgroupAvcMceSetInterBaseMultiReferencePenaltyINTEL = 5714, + OpSubgroupAvcMceGetDefaultInterShapePenaltyINTEL = 5715, + OpSubgroupAvcMceSetInterShapePenaltyINTEL = 5716, + OpSubgroupAvcMceGetDefaultInterDirectionPenaltyINTEL = 5717, + OpSubgroupAvcMceSetInterDirectionPenaltyINTEL = 5718, + OpSubgroupAvcMceGetDefaultIntraLumaShapePenaltyINTEL = 5719, + OpSubgroupAvcMceGetDefaultInterMotionVectorCostTableINTEL = 5720, + OpSubgroupAvcMceGetDefaultHighPenaltyCostTableINTEL = 5721, + OpSubgroupAvcMceGetDefaultMediumPenaltyCostTableINTEL = 5722, + OpSubgroupAvcMceGetDefaultLowPenaltyCostTableINTEL = 5723, + OpSubgroupAvcMceSetMotionVectorCostFunctionINTEL = 5724, + OpSubgroupAvcMceGetDefaultIntraLumaModePenaltyINTEL = 5725, + OpSubgroupAvcMceGetDefaultNonDcLumaIntraPenaltyINTEL = 5726, + OpSubgroupAvcMceGetDefaultIntraChromaModeBasePenaltyINTEL = 5727, + OpSubgroupAvcMceSetAcOnlyHaarINTEL = 5728, + OpSubgroupAvcMceSetSourceInterlacedFieldPolarityINTEL = 5729, + OpSubgroupAvcMceSetSingleReferenceInterlacedFieldPolarityINTEL = 5730, + OpSubgroupAvcMceSetDualReferenceInterlacedFieldPolaritiesINTEL = 5731, + OpSubgroupAvcMceConvertToImePayloadINTEL = 5732, + OpSubgroupAvcMceConvertToImeResultINTEL = 5733, + OpSubgroupAvcMceConvertToRefPayloadINTEL = 5734, + OpSubgroupAvcMceConvertToRefResultINTEL = 5735, + OpSubgroupAvcMceConvertToSicPayloadINTEL = 5736, + OpSubgroupAvcMceConvertToSicResultINTEL = 5737, + OpSubgroupAvcMceGetMotionVectorsINTEL = 5738, + OpSubgroupAvcMceGetInterDistortionsINTEL = 5739, + OpSubgroupAvcMceGetBestInterDistortionsINTEL = 5740, + OpSubgroupAvcMceGetInterMajorShapeINTEL = 5741, + OpSubgroupAvcMceGetInterMinorShapeINTEL = 5742, + OpSubgroupAvcMceGetInterDirectionsINTEL = 5743, + OpSubgroupAvcMceGetInterMotionVectorCountINTEL = 5744, + OpSubgroupAvcMceGetInterReferenceIdsINTEL = 5745, + OpSubgroupAvcMceGetInterReferenceInterlacedFieldPolaritiesINTEL = 5746, + OpSubgroupAvcImeInitializeINTEL = 5747, + OpSubgroupAvcImeSetSingleReferenceINTEL = 5748, + OpSubgroupAvcImeSetDualReferenceINTEL = 5749, + OpSubgroupAvcImeRefWindowSizeINTEL = 5750, + OpSubgroupAvcImeAdjustRefOffsetINTEL = 5751, + OpSubgroupAvcImeConvertToMcePayloadINTEL = 5752, + OpSubgroupAvcImeSetMaxMotionVectorCountINTEL = 5753, + OpSubgroupAvcImeSetUnidirectionalMixDisableINTEL = 5754, + OpSubgroupAvcImeSetEarlySearchTerminationThresholdINTEL = 5755, + OpSubgroupAvcImeSetWeightedSadINTEL = 5756, + OpSubgroupAvcImeEvaluateWithSingleReferenceINTEL = 5757, + OpSubgroupAvcImeEvaluateWithDualReferenceINTEL = 5758, + OpSubgroupAvcImeEvaluateWithSingleReferenceStreaminINTEL = 5759, + OpSubgroupAvcImeEvaluateWithDualReferenceStreaminINTEL = 5760, + OpSubgroupAvcImeEvaluateWithSingleReferenceStreamoutINTEL = 5761, + OpSubgroupAvcImeEvaluateWithDualReferenceStreamoutINTEL = 5762, + OpSubgroupAvcImeEvaluateWithSingleReferenceStreaminoutINTEL = 5763, + OpSubgroupAvcImeEvaluateWithDualReferenceStreaminoutINTEL = 5764, + OpSubgroupAvcImeConvertToMceResultINTEL = 5765, + OpSubgroupAvcImeGetSingleReferenceStreaminINTEL = 5766, + OpSubgroupAvcImeGetDualReferenceStreaminINTEL = 5767, + OpSubgroupAvcImeStripSingleReferenceStreamoutINTEL = 5768, + OpSubgroupAvcImeStripDualReferenceStreamoutINTEL = 5769, + OpSubgroupAvcImeGetStreamoutSingleReferenceMajorShapeMotionVectorsINTEL = 5770, + OpSubgroupAvcImeGetStreamoutSingleReferenceMajorShapeDistortionsINTEL = 5771, + OpSubgroupAvcImeGetStreamoutSingleReferenceMajorShapeReferenceIdsINTEL = 5772, + OpSubgroupAvcImeGetStreamoutDualReferenceMajorShapeMotionVectorsINTEL = 5773, + OpSubgroupAvcImeGetStreamoutDualReferenceMajorShapeDistortionsINTEL = 5774, + OpSubgroupAvcImeGetStreamoutDualReferenceMajorShapeReferenceIdsINTEL = 5775, + OpSubgroupAvcImeGetBorderReachedINTEL = 5776, + OpSubgroupAvcImeGetTruncatedSearchIndicationINTEL = 5777, + OpSubgroupAvcImeGetUnidirectionalEarlySearchTerminationINTEL = 5778, + OpSubgroupAvcImeGetWeightingPatternMinimumMotionVectorINTEL = 5779, + OpSubgroupAvcImeGetWeightingPatternMinimumDistortionINTEL = 5780, + OpSubgroupAvcFmeInitializeINTEL = 5781, + OpSubgroupAvcBmeInitializeINTEL = 5782, + OpSubgroupAvcRefConvertToMcePayloadINTEL = 5783, + OpSubgroupAvcRefSetBidirectionalMixDisableINTEL = 5784, + OpSubgroupAvcRefSetBilinearFilterEnableINTEL = 5785, + OpSubgroupAvcRefEvaluateWithSingleReferenceINTEL = 5786, + OpSubgroupAvcRefEvaluateWithDualReferenceINTEL = 5787, + OpSubgroupAvcRefEvaluateWithMultiReferenceINTEL = 5788, + OpSubgroupAvcRefEvaluateWithMultiReferenceInterlacedINTEL = 5789, + OpSubgroupAvcRefConvertToMceResultINTEL = 5790, + OpSubgroupAvcSicInitializeINTEL = 5791, + OpSubgroupAvcSicConfigureSkcINTEL = 5792, + OpSubgroupAvcSicConfigureIpeLumaINTEL = 5793, + OpSubgroupAvcSicConfigureIpeLumaChromaINTEL = 5794, + OpSubgroupAvcSicGetMotionVectorMaskINTEL = 5795, + OpSubgroupAvcSicConvertToMcePayloadINTEL = 5796, + OpSubgroupAvcSicSetIntraLumaShapePenaltyINTEL = 5797, + OpSubgroupAvcSicSetIntraLumaModeCostFunctionINTEL = 5798, + OpSubgroupAvcSicSetIntraChromaModeCostFunctionINTEL = 5799, + OpSubgroupAvcSicSetBilinearFilterEnableINTEL = 5800, + OpSubgroupAvcSicSetSkcForwardTransformEnableINTEL = 5801, + OpSubgroupAvcSicSetBlockBasedRawSkipSadINTEL = 5802, + OpSubgroupAvcSicEvaluateIpeINTEL = 5803, + OpSubgroupAvcSicEvaluateWithSingleReferenceINTEL = 5804, + OpSubgroupAvcSicEvaluateWithDualReferenceINTEL = 5805, + OpSubgroupAvcSicEvaluateWithMultiReferenceINTEL = 5806, + OpSubgroupAvcSicEvaluateWithMultiReferenceInterlacedINTEL = 5807, + OpSubgroupAvcSicConvertToMceResultINTEL = 5808, + OpSubgroupAvcSicGetIpeLumaShapeINTEL = 5809, + OpSubgroupAvcSicGetBestIpeLumaDistortionINTEL = 5810, + OpSubgroupAvcSicGetBestIpeChromaDistortionINTEL = 5811, + OpSubgroupAvcSicGetPackedIpeLumaModesINTEL = 5812, + OpSubgroupAvcSicGetIpeChromaModeINTEL = 5813, + OpSubgroupAvcSicGetPackedSkcLumaCountThresholdINTEL = 5814, + OpSubgroupAvcSicGetPackedSkcLumaSumThresholdINTEL = 5815, + OpSubgroupAvcSicGetInterRawSadsINTEL = 5816, + OpLoopControlINTEL = 5887, + OpReadPipeBlockingINTEL = 5946, + OpWritePipeBlockingINTEL = 5947, + OpFPGARegINTEL = 5949, + OpRayQueryGetRayTMinKHR = 6016, + OpRayQueryGetRayFlagsKHR = 6017, + OpRayQueryGetIntersectionTKHR = 6018, + OpRayQueryGetIntersectionInstanceCustomIndexKHR = 6019, + OpRayQueryGetIntersectionInstanceIdKHR = 6020, + OpRayQueryGetIntersectionInstanceShaderBindingTableRecordOffsetKHR = 6021, + OpRayQueryGetIntersectionGeometryIndexKHR = 6022, + OpRayQueryGetIntersectionPrimitiveIndexKHR = 6023, + OpRayQueryGetIntersectionBarycentricsKHR = 6024, + OpRayQueryGetIntersectionFrontFaceKHR = 6025, + OpRayQueryGetIntersectionCandidateAABBOpaqueKHR = 6026, + OpRayQueryGetIntersectionObjectRayDirectionKHR = 6027, + OpRayQueryGetIntersectionObjectRayOriginKHR = 6028, + OpRayQueryGetWorldRayDirectionKHR = 6029, + OpRayQueryGetWorldRayOriginKHR = 6030, + OpRayQueryGetIntersectionObjectToWorldKHR = 6031, + OpRayQueryGetIntersectionWorldToObjectKHR = 6032, + OpAtomicFAddEXT = 6035, + } + } +} diff --git a/third_party/shader_translator/LICENSE b/third_party/shader_translator/LICENSE new file mode 100644 index 0000000..afd57b3 --- /dev/null +++ b/third_party/shader_translator/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 jhaizhou + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/third_party/shader_translator/Program.cs b/third_party/shader_translator/Program.cs new file mode 100644 index 0000000..1f71589 --- /dev/null +++ b/third_party/shader_translator/Program.cs @@ -0,0 +1,215 @@ +// TOTK adaptation (2026-10-12): adds a --serve mode (one job per stdin line) and records which texture/image +// slots the translator queried, so the host can key runtime translations by the answers actually used. +// Based on D2R Mobile dm/gfx/tools/dmsh_xlate/Program.cs (MIT, see LICENSE). +// Switch 原版着色器（Maxwell 机器码）→ SPIR-V（Ryujinx.Graphics.Shader 翻译器），供 B 路线离线出 metallib。 +// 与 Ryujinx.ShaderTools 的区别：翻译器的「查询」按真实情况作答，而不是一律缺省值—— +// 贴图类型 / 图像格式 / 顶点属性类型 / 片元输出类型 / 计算线程组大小取自作业文件（由映射表配对的 PC 反射给出，同槽号 = 同资源）； +// 窗口原点左上、深度 0..1（游戏实测 nvnDeviceSetWindowOriginMode / SetDepthMode）。 +// 用法：dmsh_xlate <作业.json> <输出目录> +// 作业：[{ "key", "stage": "vs|ps|cs", "code": 机器码文件（图形 = 0x50 字节 SPH + 指令，计算 = 指令）， +// "tex": { "槽": "2d|2darray|cube|cubearray|3d|buffer|1d|1darray|2dms|2dmsarray", … }, "img": { "槽": { "type", "fmt" } }, +// "attr": { "位置": "float|sint|uint" }, "fout": { "目标": "float|sint|uint" }, "local": [x, y, z] }] +// 输出：.spv 与 .res.json（存储缓冲槽在 c0 的偏移等，贴图句柄 → 槽号的解码在出 metallib 的脚本里做） +// Original Switch shaders (Maxwell machine code) → SPIR-V (the Ryujinx.Graphics.Shader translator), for route B's offline metallib build. +// Difference from Ryujinx.ShaderTools: the translator's "queries" are answered truthfully rather than always with defaults - +// texture type / image format / vertex attribute type / fragment output type / compute thread group size come from the job file (given by the PC reflection paired via the mapping table; same slot number = same resource); +// window origin upper-left, depth 0..1 (measured from the game: nvnDeviceSetWindowOriginMode / SetDepthMode). +// Usage: dmsh_xlate +// Job: [{ "key", "stage": "vs|ps|cs", "code": machine code file (graphics = 0x50-byte SPH + instructions, compute = instructions), +// "tex": { "slot": "2d|2darray|cube|cubearray|3d|buffer|1d|1darray|2dms|2dmsarray", ... }, "img": { "slot": { "type", "fmt" } }, +// "attr": { "location": "float|sint|uint" }, "fout": { "target": "float|sint|uint" }, "local": [x, y, z] }] +// Output: .spv and .res.json (offsets of storage buffer slots in c0, etc.; decoding texture handle → slot number is done in the script that builds the metallib) +using Ryujinx.Common.Memory; +using Ryujinx.Graphics.Shader; +using Ryujinx.Graphics.Shader.Translation; +using System; +using System.Collections.Generic; +using System.Linq; +using System.IO; +using System.Runtime.InteropServices; +using System.Text.Json; +using System.Text.Json.Nodes; + +static class SwshXlate +{ + const int TexBase = 0x168, SampBase = 0x568; // NVN 分离贴图 / 采样器句柄在 c[2] 的字节基址（每槽 8 字节；dm/gfx/tools/maxwell_refs.py 全量实测） | byte base of NVN separate texture / sampler handles in c[2] (8 bytes per slot; full measurement in dm/gfx/tools/maxwell_refs.py) + + sealed class Accessor : IGpuAccessor + { + readonly byte[] _code; readonly JsonObject _job; int _binding = 1; public readonly List Notes = new(); public readonly SortedSet TexQueried = new(), ImgQueried = new(); + readonly byte[] _c1; + public Accessor(byte[] code, JsonObject job) { _code = code; _job = job; _c1 = job["c1"] is JsonNode h ? Convert.FromHexString((string)h) : Array.Empty(); } + // c1 常数段（着色器包控制段前 0x100 字节）：间接跳转 BRX 的跳转表在这里（LDC c[1]）；不提供时翻译器解不出跳转目标、控制流出错（粒子着色器死循环） + // c1 constant segment (the 0x100 bytes before the control segment in the shader package): the jump table of indirect jump BRX lives here (LDC c[1]); without it the translator cannot resolve jump targets and control flow goes wrong (particle shaders loop forever) + public uint ConstantBuffer1Read(int offset) { if (offset >= 0 && offset + 4 <= _c1.Length) return BitConverter.ToUInt32(_c1, offset); Notes.Add($"c1 读越界 +{offset:X}（c1 长 {_c1.Length}）"); return 0; } + // 全部资源放进第 0 组，各占不同绑定号（spirv-cross 出成一个参数缓冲）；常量段号 index 的缓冲由翻译器给 binding = index+1 的惯例不依赖 + // All resources go into set 0, each with a distinct binding number (spirv-cross emits them as one argument buffer); the translator's convention of giving the buffer of constant segment number index binding = index+1 is not relied on + public SetBindingPair CreateConstantBufferBinding(int index) => new(0, _binding++); + public SetBindingPair CreateImageBinding(int count, bool isBuffer) { int b = _binding; _binding += count; return new(0, b); } + public SetBindingPair CreateStorageBufferBinding(int index) => new(0, _binding++); + public SetBindingPair CreateTextureBinding(int count, bool isBuffer) { int b = _binding; _binding += count; return new(0, b); } + public ReadOnlySpan GetCode(ulong address, int minimumSize) => MemoryMarshal.Cast(new ReadOnlySpan(_code)[(int)address..]); + public int QuerySamplerArrayLengthFromPool() => 32; + public int QueryTextureArrayLengthFromBuffer(int slot) => 32; + public int QueryTextureArrayLengthFromPool() => 32; + + string Get(string table, int slot) => _job[table] is JsonObject t && t[slot.ToString()] is JsonNode n ? (n is JsonObject o ? (string)o["type"] : (string)n) : null; + int TexSlot(int handle, int cbufSlot) + { + (int off0, _, _) = TextureHandle.UnpackOffsets(handle); + int bytes = off0 * 4; + return (cbufSlot < 0 || (cbufSlot & 0xffff) == 2) && bytes >= TexBase && bytes < SampBase && (bytes - TexBase) % 8 == 0 ? (bytes - TexBase) / 8 : -1; + } + public SamplerType QuerySamplerType(int handle, int cbufSlot = -1) + { + int slot = TexSlot(handle, cbufSlot); + if (slot >= 0) TexQueried.Add(slot); + string t = slot >= 0 ? Get("tex", slot) ?? Get("img", slot) : null; + if (t == null) { Notes.Add($"贴图句柄 {handle:X}（常量段 {cbufSlot}，槽 {slot}）无 PC 类型，按 2D"); return SamplerType.Texture2D; } + if (t.EndsWith("_depth")) t = t.Substring(0, t.Length - 6); // depth-format texture: same sampler type here; the MSL declaration is rewritten to depth2d by shader_service.py + return t switch + { + "1d" => SamplerType.Texture1D, "1darray" => SamplerType.Texture1D | SamplerType.Array, + "buffer" => SamplerType.TextureBuffer, "2d" => SamplerType.Texture2D, "2darray" => SamplerType.Texture2D | SamplerType.Array, + "2dms" => SamplerType.Texture2D | SamplerType.Multisample, "2dmsarray" => SamplerType.Texture2D | SamplerType.Multisample | SamplerType.Array, + "3d" => SamplerType.Texture3D, "cube" => SamplerType.TextureCube, "cubearray" => SamplerType.TextureCube | SamplerType.Array, + _ => SamplerType.Texture2D, + }; + } + public TextureFormat QueryTextureFormat(int handle, int cbufSlot = -1) + { + int slot = TexSlot(handle, cbufSlot); + if (slot >= 0) ImgQueried.Add(slot); + string f = _job["img"] is JsonObject t && slot >= 0 && t[slot.ToString()] is JsonObject o ? (string)o["fmt"] : null; + if (f == null) { Notes.Add($"图像句柄 {handle:X} 槽 {slot} 无 PC 格式，按 R8G8B8A8Unorm"); return TextureFormat.R8G8B8A8Unorm; } + return Enum.TryParse(f, true, out TextureFormat tf) ? tf : TextureFormat.R8G8B8A8Unorm; + } + int Local(int i) => _job["local"] is JsonArray a && a.Count > i ? (int)a[i] : 1; + public int QueryComputeLocalSizeX() => Local(0); + public int QueryComputeLocalSizeY() => Local(1); + public int QueryComputeLocalSizeZ() => Local(2); + public int QueryComputeSharedMemorySize() => _job["shared"] is JsonNode n ? (int)n : 0; // 配对 PC 的 groupshared 声明之和 | sum of the paired PC shader's groupshared declarations + static AttributeType Attr(string s) => s switch { "sint" => AttributeType.Sint, "uint" => AttributeType.Uint, _ => AttributeType.Float }; + public GpuGraphicsState QueryGraphicsState() + { + Array32 attr = default; Array8 fout = default; + if (_job["attr"] is JsonObject a) foreach (var kv in a) attr[int.Parse(kv.Key)] = Attr((string)kv.Value); + if (_job["fout"] is JsonObject f) foreach (var kv in f) fout[int.Parse(kv.Key)] = Attr((string)kv.Value); + // TOTK (2026-10-13): the game calls nvnDeviceSetDepthMode(NEAR_IS_MINUS_W) (GL clip range -w..w); a frame-800 uniform dump + // showed the UI projection producing clip z = -1, which Metal (0..w) clips entirely, so no draw ever produced a fragment. + // The job says "depth_minus_w": true for that mode and the translator then emits z = z * 0.5 + w * 0.5 (Metal has no + // depth clip control, see QueryHostSupportsDepthClipControl below). + bool depthMinusW = _job["depth_minus_w"] is JsonValue dm && (dm.TryGetValue(out bool db) ? db : dm.TryGetValue(out int di) && di != 0); + return new GpuGraphicsState(false, InputTopology.Triangles, false, TessPatchType.Triangles, TessSpacing.EqualSpacing, + false, false, false, depthMinusW, false, 1f, AlphaTestOp.Always, 0f, attr, false, fout, false, + false /* 不做 Y 取反 | no Y flip */, true /* 原点左上 | origin upper-left */, false); + } + public bool QueryHostSupportsDepthClipControl() => false; // Metal clips z to 0..w only: a -w..w depth mode must be converted in the shader + // Ryujinx's DoubleToFloat pass lowers FP64 operations and pack/unpack + // helpers for Metal, which has no shader double type. Enabled only on + // a retry for shaders that actually emit FP64 in the first pass. + public bool QueryHostSupportsShaderFloat64() => _job["softfloat64"] is not JsonValue v || !(v.TryGetValue(out bool b) && b); + public bool QueryHostSupportsTextureShadowLod() => true; + // 作业 reduced = true：放宽精度——翻译器只给强制精确的运算打 NoContraction（缺省全部打：保持 Maxwell 分开舍入的逐位结果，代价是 MSL 里大量禁优化的 spvFMul 等） + // Job reduced = true: relaxed precision - the translator marks NoContraction only on operations that must be exact (the default marks all: keeps the bit-exact results of Maxwell's separate rounding, at the cost of many optimization-disabled spvFMul etc. in the MSL) + public bool QueryHostReducedPrecision() => _job["reduced"] is JsonNode n && (bool)n; + } + + + internal static string Translate(JsonObject job, string outDir) // also called by the --service mode (Service.cs) + { + string key = (string)job["key"]; + try + { + byte[] code = File.ReadAllBytes((string)job["code"]); + var acc = new Accessor(code, job); + // 作业 debug = false：关 DebugMode（DebugMode 会跳过 AST 优化器，产物满是临时变量；不改语义） + // Job debug = false: turn off DebugMode (DebugMode skips the AST optimizer and the output is full of temporaries; semantics unchanged) + var flags = (job["debug"] is JsonNode dbg && !(bool)dbg ? 0 : TranslationFlags.DebugMode) | ((string)job["stage"] == "cs" ? TranslationFlags.Compute : 0); + var ctx = Translator.CreateContext(0, acc, new TranslationOptions(TargetLanguage.Spirv, TargetApi.Vulkan, flags)); + ShaderProgram p = ctx.Translate(); + File.WriteAllBytes(Path.Combine(outDir, key + ".spv"), + VertexCounterLowering.Apply(code, (string)job["stage"], p.BinaryCode, acc.Notes)); + var res = new JsonObject + { + ["key"] = key, ["stage"] = (string)job["stage"], ["notes"] = new JsonArray(acc.Notes.ConvertAll(n => (JsonNode)n).ToArray()), + ["cbufs"] = new JsonArray(), ["sbufs"] = new JsonArray(), ["textures"] = new JsonArray(), ["images"] = new JsonArray(), + ["uses_frag_coord"] = p.Info.UsesFragCoord, ["frag_out_map"] = p.Info.FragmentOutputMap, + ["tex_queried"] = new JsonArray(acc.TexQueried.Select(x => (JsonNode)x).ToArray()), ["img_queried"] = new JsonArray(acc.ImgQueried.Select(x => (JsonNode)x).ToArray()), + }; + foreach (var b in p.Info.CBuffers) res["cbufs"].AsArray().Add((JsonNode)new JsonObject { ["binding"] = b.Binding, ["slot"] = b.Slot }); + foreach (var b in p.Info.SBuffers) res["sbufs"].AsArray().Add((JsonNode)new JsonObject { ["binding"] = b.Binding, ["slot"] = b.Slot, ["cb_slot"] = b.SbCbSlot, ["cb_offset"] = b.SbCbOffset, ["flags"] = (int)b.Flags }); + foreach (var (list, name) in new[] { (p.Info.Textures, "textures"), (p.Info.Images, "images") }) + foreach (var t in list) res[name].AsArray().Add((JsonNode)new JsonObject { ["binding"] = t.Binding, ["type"] = (int)t.Type, ["format"] = t.Format.ToString(), ["cbuf_slot"] = t.CbufSlot, ["handle"] = t.HandleIndex, ["array"] = t.ArrayLength, ["separate"] = t.Separate, ["flags"] = (int)t.Flags }); + File.WriteAllText(Path.Combine(outDir, key + ".res.json"), res.ToJsonString(new JsonSerializerOptions { WriteIndented = true })); + return null; + } + catch (Exception e) { if (Environment.GetEnvironmentVariable("DMSH_STACK") != null) Console.Error.WriteLine(e.StackTrace); return $"{e.GetType().Name}: {e.Message.Split('\n')[0]}"; } + } + + // --disasm [vs|ps|cs]: print the decoded Maxwell instruction stream (address, name, raw opcode) with the + // constant-buffer slot / offset of LDC and of cbuf-operand ALU forms, and the address register of global loads. + static int Disasm(string codePath, string stage) + { + // reads the translator's private decoder state by reflection, which a NativeAOT build does not keep + if (!System.Runtime.CompilerServices.RuntimeFeature.IsDynamicCodeSupported) { Console.Error.WriteLine("--disasm needs the JIT build: tooling/dotnet/dotnet build/totk_xlate/totk_xlate.dll --disasm ..."); return 2; } + byte[] code = File.ReadAllBytes(codePath); + var job = new JsonObject { ["key"] = "disasm", ["stage"] = stage, ["c1"] = "", ["debug"] = true, ["reduced"] = true }; + var acc = new Accessor(code, job); + var flags = TranslationFlags.DebugMode | (stage == "cs" ? TranslationFlags.Compute : 0); + var ctx = Translator.CreateContext(0, acc, new TranslationOptions(TargetLanguage.Spirv, TargetApi.Vulkan, flags)); + var f = typeof(TranslatorContext).GetField("_program", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance); + var program = (System.Collections.IEnumerable)f.GetValue(ctx); + foreach (object func in program) + { + var blocks = (Array)func.GetType().GetProperty("Blocks").GetValue(func); + foreach (object block in blocks) + { + var ops = (System.Collections.IEnumerable)block.GetType().GetProperty("OpCodes").GetValue(block); + foreach (object op in ops) + { + ulong addr = (ulong)op.GetType().GetField("Address").GetValue(op); + ulong raw = (ulong)op.GetType().GetField("RawOpCode").GetValue(op); + string name = op.GetType().GetField("Name").GetValue(op).ToString(); + string extra = ""; + if (name == "Ldc") extra = $" c[{(raw >> 36) & 0x1f}][{(raw >> 20) & 0xffff:x}] + R{(raw >> 8) & 0xff} -> R{raw & 0xff} size={(raw >> 48) & 7} mode={(raw >> 44) & 3}"; + else if (name == "Ldg" || name == "Ld" || name == "Stg" || name == "St") extra = $" [R{(raw >> 8) & 0xff} + {(raw >> 20) & 0xffffff:x}] R{raw & 0xff} size={(raw >> 48) & 7} e={(raw >> 45) & 1}"; + else if (name.EndsWith("C")) extra = $" c[{(raw >> 34) & 0x1f}][{((raw >> 20) & 0x3fff) * 4:x}] rA=R{(raw >> 8) & 0xff} -> R{raw & 0xff}"; + else if (name.EndsWith("R") || name.EndsWith("I")) extra = $" rA=R{(raw >> 8) & 0xff} rB=R{(raw >> 20) & 0xff} -> R{raw & 0xff}"; + Console.WriteLine($"{addr:x4}: {name,-12} {raw:x16}{extra}"); + } + } + } + return 0; + } + + static int Main(string[] args) + { + if (args.Length >= 2 && args[0] == "--disasm") return Disasm(args[1], args.Length > 2 ? args[2] : "vs"); + if (args.Length >= 1 && args[0] == "--service") return ShaderService.Run(args); // the runtime's translation service (Service.cs) + if (args.Length == 1 && args[0] == "--serve") + { + // One request per line: {"job": {...}, "out": ""}; one reply line "ok " / "fail ". + string line; + while ((line = Console.ReadLine()) != null) + { + if (line.Length == 0) continue; + var req = JsonNode.Parse(line).AsObject(); var job = req["job"].AsObject(); string o = (string)req["out"]; + Directory.CreateDirectory(o); + string err = Translate(job, o); + Console.WriteLine(err == null ? $"ok {(string)job["key"]}" : $"fail {(string)job["key"]} {err}"); + Console.Out.Flush(); + } + return 0; + } + var jobs = JsonNode.Parse(File.ReadAllText(args[0])).AsArray(); string outDir = args[1]; Directory.CreateDirectory(outDir); + int ok = 0, bad = 0; + foreach (JsonObject job in jobs) + { + string err = Translate(job, outDir); + if (err == null) ok++; else { bad++; Console.Error.WriteLine($"【红】{(string)job["key"]}：{err}"); } + } + Console.WriteLine($"翻译 {ok} 成功，{bad} 失败"); + return bad == 0 ? 0 : 1; + } +} diff --git a/third_party/shader_translator/Service.cs b/third_party/shader_translator/Service.cs new file mode 100644 index 0000000..e960826 --- /dev/null +++ b/third_party/shader_translator/Service.cs @@ -0,0 +1,468 @@ +// TOTK (2026-10-17): the runtime shader translation service as one native program: totk_xlate --service. +// Same request / reply lines and the same cache files as scripts/shader_service.py, which it replaces when +// build/shader-service/totk_xlate exists (totk_shader_rt.m); the translation runs in this process instead of a second +// "dotnet totk_xlate.dll --serve" process, and published with NativeAOT (make shader-service) it needs neither Python nor +// a .NET runtime. This is a line-by-line port of shader_service.py: keep the two in step, and check a change with +// scripts/diag/retranslate_cache.py --service build/shader-service/totk_xlate --compare (every MSL byte for byte, +// every reflection file as JSON). +// Request: {"variant", "stage": "vs|ps|cs", "code": path, "c1": hex, "tex": {slot: type}, "img": {slot: {type, fmt}}, +// "attr": {loc: float|sint|uint}, "fout": {idx: float|sint|uint}, "local": [x, y, z], "depth_minus_w": bool} +// Reply: {"ok": true} or {"ok": false, "error": "..."}; outputs /_b.metal and _b.json. +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Globalization; +using System.IO; +using System.Linq; +using System.Text; +using System.Text.Json; +using System.Text.Json.Nodes; +using System.Text.RegularExpressions; + +static class ShaderService +{ + sealed class RuntimeError : Exception { public RuntimeError(string message) : base(message) { } } + + const int TexBase = 0x168, SampBase = 0x568, ImgBase = 0x120; + // TOTK (NVN 55): combined texture handles bound with nvnCommandBufferBindTexture live at c2+0x20, 32 slots of 8 bytes + // (measured over 300 pixel and 400 vertex shaders of the game); separate textures stay at 0x168, samplers at 0x568. + const int CombinedBase = 0x20, CombinedSlots = 32; + static readonly string[] Xc = { "--msl", "--msl-version", "20300", "--msl-argument-buffers", "--msl-texture-buffer-native", "--msl-pad-fragment-output" }; + // Driver-segment words the replay fills, per stage (nvn_mtl.m drv_fillable / encode_args): c0 storage-buffer table of the + // stage, compute thread-group counts (0x200) and uniform-buffer table (0x210), vertex point-size limit (0x610); c2 handles. + static readonly Dictionary DrvFillC0 = new() + { + ["vs"] = new[] { (0x110L, 0x210L), (0x610L, 0x620L) }, ["ps"] = new[] { (0x510L, 0x610L) }, ["cs"] = new[] { (0x200L, 0x410L) }, + }; + static readonly (long lo, long hi)[] DrvFillC2 = { (0x20, 0x120), (0x168, 0x568), (0x568, 0x968) }; + static readonly Dictionary StagePrefix = new() { ["vs"] = "vp", ["ps"] = "fp", ["cs"] = "cp" }; + // Depth-format textures sampled through plain texture slots: Metal returns nothing for a depth texture read as texture2d, + // so the declaration becomes depth2d / depth2d_array / depthcube (same for _ms) and every .sample / .read on it is wrapped + // so the scalar depth becomes (d, 0, 0, 1) like a depth texture fetch on NVN. .gather already returns float4 on depth. + static readonly Dictionary DepthTypes = new() + { + ["texture2d"] = "depth2d", ["texture2d_array"] = "depth2d_array", ["texturecube"] = "depthcube", + ["texturecube_array"] = "depthcube_array", ["texture2d_ms"] = "depth2d_ms", ["texture2d_ms_array"] = "depth2d_ms_array", + }; + static readonly Regex Fp64 = new(@"\bdouble\b|unsupported_GLSLstd450(?:Pack|Unpack)Double2x32"); + + static string spirvCross, outDir, chain; + static int debugMode; + + // ---- small helpers with Python's meaning ---- + static long ParseInt(string s) => long.Parse(s, NumberStyles.None, CultureInfo.InvariantCulture); + static long ParseHex(string s) => long.Parse(s, NumberStyles.AllowHexSpecifier, CultureInfo.InvariantCulture); + static long FloorDiv(long a, long b) => a / b - ((a % b != 0) && ((a < 0) != (b < 0)) ? 1 : 0); // Python // + static string Slice(string s, int start, int stop) // Python s[start:stop] for 0 <= start + { + stop = Math.Min(stop, s.Length); start = Math.Min(start, s.Length); + return stop <= start ? "" : s.Substring(start, stop - start); + } + static string Cut(string s, int n) => s.Length > n ? s.Substring(0, n) : s; + static bool Truthy(JsonNode n) => n switch + { + null => false, + JsonObject o => o.Count > 0, + JsonArray a => a.Count > 0, + JsonValue v => v.GetValueKind() switch + { + JsonValueKind.True => true, JsonValueKind.False => false, + JsonValueKind.String => v.GetValue().Length > 0, + JsonValueKind.Number => v.TryGetValue(out long l) ? l != 0 : v.GetValue() != 0, + _ => false, + }, + _ => false, + }; + static JsonNode Get(JsonObject o, string key) => o.TryGetPropertyValue(key, out JsonNode v) ? v : null; + static JsonObject GetObj(JsonObject o, string key) => Get(o, key) as JsonObject ?? new JsonObject(); // rq.get(key, {}) + static string GetStr(JsonObject o, string key, string fallback) => o.TryGetPropertyValue(key, out JsonNode v) ? (string)v : fallback; + static JsonNode GetOr(JsonObject o, string key, JsonNode fallback) => o.TryGetPropertyValue(key, out JsonNode v) ? v?.DeepClone() : fallback; + static JsonNode Num(long? v) => v is long l ? JsonValue.Create(l) : null; + static long Long(JsonNode n) => n.AsValue().TryGetValue(out long l) ? l : (long)n.GetValue(); + + // ---- binding names -> NVN slots (shader_service.py decode_tex / tex_slot_of_name) ---- + static long? Slot(long word, long @base, long cb) => cb == 2 && word * 4 >= @base && (word * 4 - @base) % 8 == 0 ? (word * 4 - @base) / 8 : null; + + static (long? t, long? s, long ty)? DecodeTex(string name) + { + Match m = Regex.Match(name, @"^[a-z]+_[ti]\w*?_(?:cb(\d+)|tcb)_([0-9A-F]+)"); + if (!m.Success) return null; + long slots = m.Groups[1].Success ? ParseInt(m.Groups[1].Value) : -1; + long h = ParseHex(m.Groups[2].Value); + long tb = slots >= 0 ? (slots & 0xffff) : 2; + long sb = slots >= 0 && (slots >> 16) != 0 ? (slots >> 16) - 1 : tb; + long o0 = h & 0x3fff, o1 = (h >> 14) & 0x3fff, ty = h >> 28; + long[] words = ty != 0 ? new[] { o0, o1 } : new[] { o0 }; + if (Regex.IsMatch(name, @"^[a-z]+_i")) + return (words.Where(x => ImgBase <= x * 4 && x * 4 < TexBase && Slot(x, ImgBase, tb) != null).Select(x => Slot(x, ImgBase, tb)).FirstOrDefault(), null, ty); + long? t = words.Where(x => Slot(x, TexBase, tb) != null && x * 4 < SampBase).Select(x => Slot(x, TexBase, tb)).FirstOrDefault(); + if (t == null) + t = words.Where(x => CombinedBase <= x * 4 && x * 4 < CombinedBase + 8 * CombinedSlots && Slot(x, CombinedBase, tb) != null).Select(x => Slot(x, CombinedBase, tb)).FirstOrDefault(); + long? s = words.Where(x => x * 4 >= SampBase && Slot(x, SampBase, sb) != null).Select(x => Slot(x, SampBase, sb)).FirstOrDefault(); + return (t, s, ty); + } + + static long TexSlotOfName(string name) + { + Match m = Regex.Match(name, @"^\w+_t_tcb_([0-9A-Fa-f]+)$"); + return m.Success ? FloorDiv(ParseHex(m.Groups[1].Value) - 8, 2) : -1; + } + + static int CloseParen(string src, int i) // index just past the ')' closing a call whose '(' ends at i (or src.Length) + { + int depth = 1; + while (i < src.Length && depth != 0) { depth += src[i] == '(' ? 1 : src[i] == ')' ? -1 : 0; i++; } + return depth == 0 ? i : -i - 1; + } + + static string RewriteDepthTextures(string src, HashSet depthSlots) + { + var names = new List<(string ty, string name)>(); + foreach (Match m in Regex.Matches(src, @"\b(texture\w+)\s+(\w+)\s*\[\[id\((\d+)\)\]\]")) + { + string ty = m.Groups[1].Value, name = m.Groups[2].Value; + if (!depthSlots.Contains(TexSlotOfName(name))) continue; + if (!DepthTypes.ContainsKey(ty)) throw new RuntimeError($"depth texture {name} declared as {ty}: no depth equivalent"); + names.Add((ty, name)); + } + foreach (var (ty, name) in names) + { + string decl = DepthTypes[ty] + " " + name; + src = new Regex(@"\b" + ty + @"\s+" + name + @"\b").Replace(src, _ => decl, 1); + // compare sampling: spirv-cross casts the texture2d to depth2d itself; the declaration is now depth2d, drop the cast + src = Regex.Replace(src, @"spvDepthCast\(((?:\w+\.)*" + name + @")\)", m => m.Groups[1].Value); + // depth2d.gather takes no component argument (only the depth component exists) + var gpat = new Regex(@"((?:\w+\.)*" + name + @")\.gather\("); + int pos = 0; + while (pos <= src.Length) + { + Match m = gpat.Match(src, pos); + if (!m.Success) break; + int end = m.Index + m.Length, c = CloseParen(src, end), i = c >= 0 ? c : -c - 1; + string call = Slice(src, end, i - 1); + if (Regex.IsMatch(call, @"component::[yzw]")) throw new RuntimeError($"gather of a non-x component on depth texture {name}"); + string call2 = Regex.Replace(call, @",\s*component::x", ""); + src = src.Substring(0, end) + call2 + src.Substring(i - 1); + pos = end + call2.Length + 1; + } + var pat = new Regex(@"((?:\w+\.)*" + name + @")\.(sample|read)\("); + pos = 0; + while (pos <= src.Length) + { + Match m = pat.Match(src, pos); + if (!m.Success) break; + int start = m.Groups[1].Index, c = CloseParen(src, m.Index + m.Length); + if (c < 0) throw new RuntimeError($"unbalanced call on depth texture {name}"); + src = src.Substring(0, start) + "totk_depth_fetch(" + src.Substring(start, c - start) + ")" + src.Substring(c); + pos = c + "totk_depth_fetch()".Length; + } + } + if (names.Count > 0) + { + const string ns = "using namespace metal;"; + int at = src.IndexOf(ns, StringComparison.Ordinal); + if (at >= 0) src = src.Substring(0, at) + ns + "\nstatic inline float4 totk_depth_fetch(float d) { return float4(d, 0.0, 0.0, 1.0); }" + src.Substring(at + ns.Length); + } + return src; + } + + /// True when the MSL stores or does atomics through storage binding `name` (same rule as nvn_mtl.m msl_stores_through). + static bool MslStoresThrough(string msl, string name) + { + if (msl.Contains($"&(*spvDescriptorSet0.{name})", StringComparison.Ordinal)) return true; + return Regex.IsMatch(msl, @"\(\*spvDescriptorSet0\." + Regex.Escape(name) + @"\)\._m0\[(?:[^\[\]]|\[[^\[\]]*\])*\](?:\.[xyzw]+)?\s*=(?!=)"); + } + + static JsonArray Reflect(string stage, string msl, JsonObject res) + { + Match body = Regex.Match(msl, @"struct spvDescriptorSetBuffer0\s*\{(.*?)\};", RegexOptions.Singleline); + var binds = new List(); + foreach (Match m in Regex.Matches(body.Success ? body.Groups[1].Value : "", @"^\s*(.+?)\s+(\w+)\s*\[\[id\((\d+)\)\]\];", RegexOptions.Multiline)) + { + string ty = m.Groups[1].Value.Trim(), nm = m.Groups[2].Value; + var e = new JsonObject { ["msl_name"] = nm, ["msl_type"] = ty, ["arg_id"] = ParseInt(m.Groups[3].Value) }; + Match fm; + if (nm == "support_buffer") e["kind"] = "support"; + else if ((fm = Regex.Match(nm, @"^[a-z]+_c(\d+)\z")).Success) + { + long n = ParseInt(fm.Groups[1].Value); + if (3 <= n && n <= 16) { e["kind"] = "cbuf"; e["reg"] = n - 3; } + else { e["kind"] = n == 1 ? "c1" : "drvcb"; e["bank"] = n; } + } + else if ((fm = Regex.Match(nm, @"^[a-z]+_s(\d+)\z")).Success) { e["kind"] = "sbuf"; e["index"] = ParseInt(fm.Groups[1].Value); } + else if (nm.EndsWith("Smplr", StringComparison.Ordinal)) + { + var d = DecodeTex(nm.Substring(0, nm.Length - 5)) ?? throw new RuntimeError($"sampler {nm}: no texture handle in the name"); + if (d.ty != 0) { e["kind"] = "samp"; e["reg"] = Num(d.s); } else { e["kind"] = "samp_tex"; e["reg"] = Num(d.t); } + } + else + { + var d = DecodeTex(nm); + if (d == null) e["kind"] = "?"; + else { e["kind"] = Regex.IsMatch(nm, @"^[a-z]+_i") ? "img" : (ty.Contains("texture_buffer") ? "tbuf" : "tex"); e["reg"] = Num(d.Value.t); } + } + binds.Add(e); + } + long? @base = stage switch { "vs" => 0x110, "cs" => 0x310, "ps" => 0x510, _ => null }; + List sbSorted = res["sbufs"].AsArray().Select(b => b.AsObject()).OrderBy(b => Long(b["binding"])).ToList(); // stable, as sorted() + foreach (JsonObject e in binds) + { + if ((string)e["kind"] != "sbuf") continue; + long index = Long(e["index"]); + JsonObject b = index < sbSorted.Count ? sbSorted[(int)index] : null; + long cbSlot = b != null ? Long(b["cb_slot"]) : 0, cbOffset = b != null ? Long(b["cb_offset"]) : 0; + long? off = b != null && cbSlot == 0 ? cbOffset * 4 : null; + if (stage == "cs" && off != null && 0x210 <= off && off < 0x310 && (off - 0x210) % 16 == 0) { e["kind"] = "ubo_ptr"; e["reg"] = (off - 0x210) / 16; } + else if (b != null && cbSlot != 0 && 3 <= cbSlot && cbSlot <= 16) + { + // the shader itself loads this storage buffer's address from a user uniform block (NVN uniform slot cb_slot-3, + // byte offset cb_offset*4); the replay reads that pointer at bind time + e["kind"] = "sbuf_ubo"; e["reg"] = cbSlot - 3; e["off"] = cbOffset * 4; + } + else e["reg"] = off != null && @base != null && 0 <= off - @base && off - @base < 0x100 && (off - @base) % 16 == 0 ? Num((off - @base) / 16) : null; + // Ryujinx BufferUsageFlags.Write is 1 << 0; the MSL is the other witness: a store or an atomic through the binding + e["rw"] = (b != null && (Long(b["flags"]) & 1) != 0) || MslStoresThrough(msl, (string)e["msl_name"]); + } + return new JsonArray(binds.Select(x => (JsonNode)x).ToArray()); + } + + static JsonObject DrvReads(string src, string stage) + { + var dr = new JsonObject(); + foreach (int bank in new[] { 0, 2 }) + { + var offs = new SortedSet(); bool dyn = false; + foreach (Match m in Regex.Matches(src, StagePrefix[stage] + "_c" + bank.ToString(CultureInfo.InvariantCulture) + @"\)\._m0\[([^\]]+)\]")) + { + string idx = m.Groups[1].Value.TrimEnd('u'); + if (idx.Length > 0 && idx.All(c => c >= '0' && c <= '9')) offs.Add(ParseInt(idx) * 16); else dyn = true; + } + if (offs.Count > 0 || dyn) + dr[bank.ToString(CultureInfo.InvariantCulture)] = new JsonObject { ["vec4_offsets"] = new JsonArray(offs.Select(o => (JsonNode)o).ToArray()), ["dynamic"] = dyn }; + } + return dr; + } + + static JsonArray Unfillable(JsonObject dr, string stage) + { + var bad = new JsonArray(); + foreach (var kv in dr) + { + JsonObject d = kv.Value.AsObject(); + if ((bool)d["dynamic"]) bad.Add((JsonNode)$"c{kv.Key} dynamic index"); + long bank = ParseInt(kv.Key); + var ranges = bank == 2 ? DrvFillC2 : bank == 0 ? DrvFillC0[stage] : Array.Empty<(long lo, long hi)>(); + foreach (JsonNode o in d["vec4_offsets"].AsArray()) + { + long ov = Long(o); + if (!ranges.Any(r => r.lo <= ov && ov < r.hi)) bad.Add((JsonNode)$"c{kv.Key}+0x{ov.ToString("x", CultureInfo.InvariantCulture)}"); + } + } + return bad; + } + + static void SpirvCross(string spv, string msl, string what) + { + var psi = new ProcessStartInfo(spirvCross) { UseShellExecute = false, RedirectStandardOutput = true, RedirectStandardError = true }; + foreach (string a in Xc) psi.ArgumentList.Add(a); + psi.ArgumentList.Add(spv); psi.ArgumentList.Add("--output"); psi.ArgumentList.Add(msl); + using Process p = Process.Start(psi) ?? throw new RuntimeError(what + "could not start " + spirvCross); + var err = p.StandardError.ReadToEndAsync(); var outp = p.StandardOutput.ReadToEndAsync(); + p.WaitForExit(); + if (p.ExitCode != 0) throw new RuntimeError(what + Cut(err.Result.Trim(), 300)); + _ = outp.Result; + } + + static void Translate(JsonObject job) + { + string err = SwshXlate.Translate(job, outDir); + if (err != null) throw new RuntimeError(Cut($"fail {(string)job["key"]} {err}".Trim(), 300)); + } + + static JsonObject ReadRes(string variant) => JsonNode.Parse(File.ReadAllText(Path.Combine(outDir, variant + ".res.json"))).AsObject(); + + static void Handle(JsonObject rq) + { + string variant = (string)rq["variant"], stage = (string)rq["stage"]; + if (stage != "vs" && stage != "ps" && stage != "cs") throw new RuntimeError($"stage {stage} not supported by the translation chain"); + if (stage == "cs" && !Truthy(Get(rq, "local"))) throw new RuntimeError("compute shader without thread group size"); + // DebugMode (AST optimizer off) since 2026-10-13: --debug-mode / TOTK_SHADER_DEBUGMODE, as in shader_service.py + var job = new JsonObject + { + ["key"] = variant, ["stage"] = stage, ["code"] = (string)rq["code"], ["c1"] = GetStr(rq, "c1", ""), ["debug"] = debugMode != 0, ["reduced"] = true, + ["tex"] = GetObj(rq, "tex").DeepClone(), ["img"] = GetObj(rq, "img").DeepClone(), ["attr"] = GetObj(rq, "attr").DeepClone(), ["fout"] = GetObj(rq, "fout").DeepClone(), + }; + if (Truthy(Get(rq, "local"))) job["local"] = rq["local"].DeepClone(); + job["depth_minus_w"] = Truthy(Get(rq, "depth_minus_w")); // nvnDeviceSetDepthMode(NEAR_IS_MINUS_W): clip z converted to Metal's 0..w + Translate(job); + JsonObject res = ReadRes(variant); + string spv = Path.Combine(outDir, variant + ".spv"), msl = Path.Combine(outDir, variant + "_b.metal"); + SpirvCross(spv, msl, "spirv-cross: "); + string src = File.ReadAllText(msl); + bool softfloat64 = Fp64.IsMatch(src); + if (softfloat64) + { + // Metal has no FP64 in shaders: Ryujinx's DoubleToFloat lowering for this variant only + job["softfloat64"] = true; + Translate(job); + res = ReadRes(variant); + SpirvCross(spv, msl, "spirv-cross FP64 fallback: "); + src = File.ReadAllText(msl); + if (Fp64.IsMatch(src)) throw new RuntimeError("Ryujinx FP64 fallback still emitted Metal-unsupported double"); + } + JsonObject tex = GetObj(rq, "tex"), img = GetObj(rq, "img"), attr = GetObj(rq, "attr"), fout = GetObj(rq, "fout"); + var depthSlots = new HashSet(tex.Where(kv => kv.Value is JsonValue v && v.GetValueKind() == JsonValueKind.String && v.GetValue().EndsWith("_depth", StringComparison.Ordinal)) + .Select(kv => ParseInt(kv.Key))); + if (depthSlots.Count > 0) + { + src = RewriteDepthTextures(src, depthSlots); + File.WriteAllText(msl, src); + } + Match entry = Regex.Match(src, @"(vertex|fragment|kernel)\s+\S+\s+(\w+)\s*\("); + JsonObject dr = DrvReads(src, stage); + JsonArray unfillable = Unfillable(dr, stage); + var vertexInputs = new JsonArray(Regex.Matches(src, @"(\w+)\s+\w+\s*\[\[attribute\((\d+)\)\]\]") + .Select(m => (JsonNode)new JsonObject { ["location"] = ParseInt(m.Groups[2].Value), ["msl_type"] = m.Groups[1].Value }).ToArray()); + Match outStruct = Regex.Match(src, @"struct main0_out\s*\{(.*?)\};", RegexOptions.Singleline); + var colors = new SortedSet(Regex.Matches(outStruct.Success ? outStruct.Groups[1].Value : "", @"\[\[color\((\d+)\)\]\]").Select(m => ParseInt(m.Groups[1].Value))); + // Answers this translation depends on: queried texture / image slots, consumed vertex inputs, written colour outputs; + // every texture slot the MSL binds is a dependency too (a depth-format texture there changes the MSL). + var usedSlots = new HashSet(Regex.Matches(src, @"\b(texture\w+|depth\w+)\s+(\w+)\s*\[\[id\(\d+\)\]\]").Select(m => TexSlotOfName(m.Groups[2].Value))); + usedSlots.Remove(-1); + var deps = new JsonObject { ["tex"] = new JsonObject(), ["img"] = new JsonObject(), ["attr"] = new JsonObject(), ["fout"] = new JsonObject() }; + IEnumerable Ints(string key) => res.TryGetPropertyValue(key, out JsonNode a) && a is JsonArray arr ? arr.Select(Long) : Enumerable.Empty(); + foreach (long s in new SortedSet(Ints("tex_queried").Concat(usedSlots))) + deps["tex"][s.ToString(CultureInfo.InvariantCulture)] = GetOr(tex, s.ToString(CultureInfo.InvariantCulture), "2d"); + foreach (long s in Ints("img_queried")) + deps["img"][s.ToString(CultureInfo.InvariantCulture)] = GetOr(img, s.ToString(CultureInfo.InvariantCulture), new JsonObject()); + if (stage == "vs") + foreach (JsonNode v in vertexInputs) { string loc = Long(v["location"]).ToString(CultureInfo.InvariantCulture); deps["attr"][loc] = GetOr(attr, loc, "float"); } + if (stage == "ps") + foreach (long c in colors) { string k = c.ToString(CultureInfo.InvariantCulture); deps["fout"][k] = GetOr(fout, k, "float"); } + var refl = new JsonObject + { + ["key"] = variant + "_b", ["route"] = "B", ["chain"] = chain, ["stage"] = stage, + ["entry"] = entry.Success ? entry.Groups[2].Value : "main0", ["msl_entry_kind"] = entry.Success ? entry.Groups[1].Value : null, + ["threadgroup_size"] = GetOr(rq, "local", null), ["c1"] = GetStr(rq, "c1", ""), ["notes"] = res["notes"].DeepClone(), + ["drv_reads"] = dr, ["drv_unfillable"] = unfillable, ["bindings"] = Reflect(stage, src, res), + ["vertex_inputs"] = vertexInputs, ["raster"] = new JsonObject(), ["deps"] = deps, ["softfloat64"] = softfloat64, + }; + string tmp = Path.Combine(outDir, variant + "_b.json.tmp"); + File.WriteAllText(tmp, PyJson(refl, 1, false)); + File.Move(tmp, Path.Combine(outDir, variant + "_b.json"), true); + } + + // ---- JSON text as Python's json.dump writes it (indent >= 0: indented, < 0: one line; ascii: ensure_ascii) ---- + static string PyJson(JsonNode n, int indent, bool ascii) { var sb = new StringBuilder(); PyJson(sb, n, indent, ascii, 0); return sb.ToString(); } + static void PyJson(StringBuilder sb, JsonNode n, int indent, bool ascii, int level) + { + switch (n) + { + case null: sb.Append("null"); return; + case JsonObject o: + if (o.Count == 0) { sb.Append("{}"); return; } + sb.Append('{'); + int i = 0; + foreach (var kv in o) + { + if (i++ > 0) sb.Append(indent < 0 ? ", " : ","); + if (indent >= 0) sb.Append('\n').Append(' ', indent * (level + 1)); + PyStr(sb, kv.Key, ascii); sb.Append(": "); PyJson(sb, kv.Value, indent, ascii, level + 1); + } + if (indent >= 0) sb.Append('\n').Append(' ', indent * level); + sb.Append('}'); return; + case JsonArray a: + if (a.Count == 0) { sb.Append("[]"); return; } + sb.Append('['); + for (int j = 0; j < a.Count; j++) + { + if (j > 0) sb.Append(indent < 0 ? ", " : ","); + if (indent >= 0) sb.Append('\n').Append(' ', indent * (level + 1)); + PyJson(sb, a[j], indent, ascii, level + 1); + } + if (indent >= 0) sb.Append('\n').Append(' ', indent * level); + sb.Append(']'); return; + case JsonValue v: + switch (v.GetValueKind()) + { + case JsonValueKind.String: PyStr(sb, v.GetValue(), ascii); return; + case JsonValueKind.True: sb.Append("true"); return; + case JsonValueKind.False: sb.Append("false"); return; + case JsonValueKind.Number: + if (v.TryGetValue(out int iv)) sb.Append(iv.ToString(CultureInfo.InvariantCulture)); + else if (v.TryGetValue(out long lv)) sb.Append(lv.ToString(CultureInfo.InvariantCulture)); + else if (v.TryGetValue(out ulong uv)) sb.Append(uv.ToString(CultureInfo.InvariantCulture)); + else sb.Append(v.ToJsonString()); + return; + default: sb.Append("null"); return; + } + } + } + static void PyStr(StringBuilder sb, string s, bool ascii) + { + sb.Append('"'); + foreach (char c in s) + { + switch (c) + { + case '"': sb.Append("\\\""); break; + case '\\': sb.Append("\\\\"); break; + case '\n': sb.Append("\\n"); break; + case '\r': sb.Append("\\r"); break; + case '\t': sb.Append("\\t"); break; + case '\b': sb.Append("\\b"); break; + case '\f': sb.Append("\\f"); break; + default: + if (c < 0x20 || (ascii && c > 0x7e)) sb.Append("\\u").Append(((int)c).ToString("x4", CultureInfo.InvariantCulture)); + else sb.Append(c); + break; + } + } + sb.Append('"'); + } + + // totk_xlate --service --spirv-cross --out --chain [--debug-mode 0|1] + public static int Run(string[] args) + { + string dm = Environment.GetEnvironmentVariable("TOTK_SHADER_DEBUGMODE"); + debugMode = string.IsNullOrEmpty(dm) ? 1 : (int)ParseInt(dm.Trim()); + for (int i = 1; i < args.Length; i++) + { + string v = i + 1 < args.Length ? args[i + 1] : null; + switch (args[i]) + { + case "--spirv-cross": spirvCross = v; i++; break; + case "--out": outDir = v; i++; break; + case "--chain": chain = v; i++; break; + case "--debug-mode": debugMode = (int)ParseInt(v); i++; break; + default: Console.Error.WriteLine($"totk_xlate --service: unknown argument {args[i]}"); return 2; + } + } + if (spirvCross == null || outDir == null || chain == null) + { + Console.Error.WriteLine("usage: totk_xlate --service --spirv-cross --out --chain [--debug-mode 0|1]"); + return 2; + } + Directory.CreateDirectory(outDir); + var stdout = Console.Out; + string line; + while ((line = Console.In.ReadLine()) != null) + { + if (line.Trim().Length == 0) continue; + string reply; + try + { + Handle(JsonNode.Parse(line).AsObject()); + reply = "{\"ok\": true}"; + } + catch (Exception e) // reported to the host, which stops with the message + { + string kind = e is RuntimeError ? "RuntimeError" : e.GetType().Name; + reply = PyJson(new JsonObject { ["ok"] = false, ["error"] = Cut($"{kind}: {e.Message}", 500) }, -1, true); + } + stdout.Write(reply + "\n"); + stdout.Flush(); + } + return 0; + } +} diff --git a/third_party/shader_translator/VertexCounterLowering.cs b/third_party/shader_translator/VertexCounterLowering.cs new file mode 100644 index 0000000..b3bea76 --- /dev/null +++ b/third_party/shader_translator/VertexCounterLowering.cs @@ -0,0 +1,61 @@ +using System; +using System.Collections.Generic; +using System.Security.Cryptography; + +// TOTK 1.1.2 / 1.4.3 visibility counter: each invocation passing its input test +// participates in a ballot; the highest active lane adds popcount(ballot) to s1[0]. +// The atomic's return value is unused and no other invocation observes it here. +// Singleton groups therefore preserve the final modulo-2^32 sum exactly. Metal's +// vertex stage cannot expose the subgroup builtins through our SPIRV-Cross path. +// This is deliberately restricted to the analysed Maxwell bytes, NOT a general +// subgroup fallback (shuffle, vote-dependent output, and atomic allocation differ). +static class VertexCounterLowering +{ + const string VertexCodeHash = "EABE9FDD6FC737B8414E94CC3E14EC0031AB3A156511953ABE5E3A04128524AD"; + // 1.4.3 reads s0[114 + (instance_id - base_instance)*4] instead of + // 1.1.2's s0[8304 + vertex_id*4]. The guarded ballot/popcount/atomic tail + // has the same reduction: one increment for every passing invocation. + const string InstanceCodeHash = "F5F4D23149565ACD852F6B7091F64CC8CB3AA019CCB33BA3F88CED62C6D1FF7B"; + internal static byte[] Apply(byte[] code, string stage, byte[] spirv, List notes) + { + if (stage != "vs") return spirv; + string codeHash=Convert.ToHexString(SHA256.HashData(code)); + if(codeHash!=VertexCodeHash&&codeHash!=InstanceCodeHash)return spirv; + uint[] w = new uint[spirv.Length / 4]; Buffer.BlockCopy(spirv, 0, w, 0, spirv.Length); + uint lane = 0, uintType = 0, vectorType = 0, trueId = 0; + int ballots = 0, loads = 0; + for (int i = 5; i < w.Length; i += (int)(w[i] >> 16)) + { + uint op = w[i] & 65535; + if (op == 71 && w[i + 2] == 11 && w[i + 3] == 41) lane = w[i + 1]; // BuiltIn SubgroupLocalInvocationId + if (op == 21 && w[i + 2] == 32 && w[i + 3] == 0) uintType = w[i + 1]; + if (op == 41) trueId = w[i + 2]; // OpConstantTrue + if (op == 339) { vectorType = w[i + 1]; ballots++; if (w[i + 4] != trueId) throw new InvalidOperationException("counter ballot predicate changed"); } + } + if (lane == 0 || uintType == 0 || vectorType == 0 || ballots != 1) + throw new InvalidOperationException("analysed vertex counter SPIR-V shape changed"); + uint zero = w[3]++, one = w[3]++, mask = w[3]++; + var result = new List(w.Length + 16); result.AddRange(w.AsSpan(0, 5).ToArray()); + bool constants = false; + for (int i = 5; i < w.Length; i += (int)(w[i] >> 16)) + { + uint op = w[i] & 65535; + if (op == 54 && !constants) + { + result.AddRange(new uint[] { (4u << 16) | 43, uintType, zero, 0, + (4u << 16) | 43, uintType, one, 1, + (7u << 16) | 44, vectorType, mask, one, zero, zero, zero }); + constants = true; + } + if (op == 61 && w[i + 3] == lane) // load lane -> 0 + { result.AddRange(new uint[] { (4u << 16) | 83, w[i + 1], w[i + 2], zero }); loads++; } + else if (op == 339) // ballot(true) -> (1,0,0,0) + result.AddRange(new uint[] { (4u << 16) | 83, w[i + 1], w[i + 2], mask }); + else result.AddRange(w.AsSpan(i, (int)(w[i] >> 16)).ToArray()); + } + if (loads != 1 || !constants) throw new InvalidOperationException("counter lane use changed"); + notes.Add("verified vertex visibility counter: singleton reduction (one atomic per passing vertex)"); + byte[] output = new byte[result.Count * 4]; Buffer.BlockCopy(result.ToArray(), 0, output, 0, output.Length); + return output; + } +} diff --git a/third_party/shader_translator/patches/0001-keep-call-arguments-in-variables.patch b/third_party/shader_translator/patches/0001-keep-call-arguments-in-variables.patch new file mode 100644 index 0000000..ce0bc97 --- /dev/null +++ b/third_party/shader_translator/patches/0001-keep-call-arguments-in-variables.patch @@ -0,0 +1,15 @@ +--- a/src/Ryujinx.Graphics.Shader/StructuredIr/AstOptimizer.cs ++++ b/src/Ryujinx.Graphics.Shader/StructuredIr/AstOptimizer.cs +@@ -24,7 +24,11 @@ + { + bool isWorthPropagating = propVar.Uses.Count == 1 || IsWorthPropagating(assignment.Source); + +- if (propVar.Defs.Count == 1 && isWorthPropagating) ++ // TOTK: SPIR-V passes call arguments by pointer to a variable (StructuredProgram copies them ++ // into temporaries); folding an expression into a call argument breaks GenerateCall. ++ bool usedByCall = propVar.Uses.Any(u => u is AstOperation op && op.Inst == Instruction.Call); ++ ++ if (propVar.Defs.Count == 1 && isWorthPropagating && !usedByCall) + { + PropagateExpression(propVar, assignment.Source); + } diff --git a/third_party/shader_translator/patches/0002-keep-live-call-sources-when-removing-dead-node.patch b/third_party/shader_translator/patches/0002-keep-live-call-sources-when-removing-dead-node.patch new file mode 100644 index 0000000..3849bfd --- /dev/null +++ b/third_party/shader_translator/patches/0002-keep-live-call-sources-when-removing-dead-node.patch @@ -0,0 +1,20 @@ +--- a/src/Ryujinx.Graphics.Shader/Translation/Optimizations/Optimizer.cs ++++ b/src/Ryujinx.Graphics.Shader/Translation/Optimizations/Optimizer.cs +@@ -384,7 +384,16 @@ + if (src.UseOps.Remove(node) && src.UseOps.Count == 0) + { + Debug.Assert(src.AsgOp != null); +- nodes.Enqueue(src.AsgOp); ++ ++ // TOTK (2026-10-16, from Ryujinx_Optime 01a1e37f): only follow into a producer that will itself be ++ // removed. An operation with side effects (a call, an atomic) stays in the block even when its ++ // result is unused, so its sources are still live: stripping their use lists here made the next ++ // DCE pass delete the whole chain feeding a global-memory atomic turned into a helper call, which ++ // then read an uninitialised value (TOTK's sun-visibility kernel; our c7830fac lost its atomicMin). ++ if (!HasSideEffects(src.AsgOp)) ++ { ++ nodes.Enqueue(src.AsgOp); ++ } + } + } + } diff --git a/third_party/shader_translator/patches/0003-keep-side-effect-assignments-in-the-ast-optimizer.patch b/third_party/shader_translator/patches/0003-keep-side-effect-assignments-in-the-ast-optimizer.patch new file mode 100644 index 0000000..014faed --- /dev/null +++ b/third_party/shader_translator/patches/0003-keep-side-effect-assignments-in-the-ast-optimizer.patch @@ -0,0 +1,57 @@ +--- a/src/Ryujinx.Graphics.Shader/StructuredIr/AstOptimizer.cs ++++ b/src/Ryujinx.Graphics.Shader/StructuredIr/AstOptimizer.cs +@@ -22,7 +22,12 @@ + { + if (node is AstAssignment assignment && assignment.Destination is AstOperand propVar) + { +- bool isWorthPropagating = propVar.Uses.Count == 1 || IsWorthPropagating(assignment.Source); ++ // TOTK (2026-10-16): an atomic / call / image atomic whose result is unused (Maxwell RED, ATOM with ++ // an RZ destination) stays a statement in place: propagating it would move the side effect to its ++ // use, and removing the assignment removed the operation itself (the reduction kernel c7830fac ++ // lost its atomicMin on 2026-10-13; every translation ran in DebugMode since, to avoid this). ++ bool sideEffects = HasSideEffects(assignment.Source); ++ bool isWorthPropagating = !sideEffects && (propVar.Uses.Count == 1 || IsWorthPropagating(assignment.Source)); + + // TOTK: SPIR-V passes call arguments by pointer to a variable (StructuredProgram copies them + // into temporaries); folding an expression into a call argument breaks GenerateCall. +@@ -33,7 +38,7 @@ + PropagateExpression(propVar, assignment.Source); + } + +- if (propVar.Type == OperandType.LocalVariable && propVar.Uses.Count == 0) ++ if (propVar.Type == OperandType.LocalVariable && propVar.Uses.Count == 0 && !sideEffects) + { + visitor.Block.Remove(assignment); + +@@ -46,6 +51,31 @@ + RemoveEmptyBlocks(mainBlock); + } + ++ private static bool HasSideEffects(IAstNode source) ++ { ++ if (source is AstOperation op) ++ { ++ switch (op.Inst & Instruction.Mask) ++ { ++ case Instruction.AtomicAdd: ++ case Instruction.AtomicAnd: ++ case Instruction.AtomicCompareAndSwap: ++ case Instruction.AtomicMaxS32: ++ case Instruction.AtomicMaxU32: ++ case Instruction.AtomicMinS32: ++ case Instruction.AtomicMinU32: ++ case Instruction.AtomicOr: ++ case Instruction.AtomicSwap: ++ case Instruction.AtomicXor: ++ case Instruction.Call: ++ case Instruction.ImageAtomic: ++ return true; ++ } ++ } ++ ++ return false; ++ } ++ + private static bool IsWorthPropagating(IAstNode source) + { + if (source is not AstOperation srcOp) diff --git a/third_party/shader_translator/patches/0004-no-else-if-flattening.patch b/third_party/shader_translator/patches/0004-no-else-if-flattening.patch new file mode 100644 index 0000000..742c0c3 --- /dev/null +++ b/third_party/shader_translator/patches/0004-no-else-if-flattening.patch @@ -0,0 +1,26 @@ +--- a/src/Ryujinx.Graphics.Shader/StructuredIr/AstOptimizer.cs ++++ b/src/Ryujinx.Graphics.Shader/StructuredIr/AstOptimizer.cs +@@ -118,6 +118,8 @@ + } + } + ++ private const bool ElseIfFlattening = false; ++ + private static void RemoveEmptyBlocks(AstBlock mainBlock) + { + Queue pending = new(); +@@ -163,7 +165,13 @@ + parent.Remove(block); + } + } +- else if (isIf && parent.Type == AstBlockType.Else && parent.Count == (hasElse ? 2 : 1)) ++ // TOTK (2026-10-16): the else-if flattening (`else { if (c) A else B }` -> `else if (c) A else B`) is ++ // disabled. The SPIR-V backend gives the else-if the same merge block as the if before it, and ++ // SPIRV-Cross then drops the trailing else of the chain: TOTK's fragment shader 1870878a lost three ++ // loops with an atomic max (a feedback write) and a discard (scripts/diag/msl_invariants.py found 12 ++ // variants losing atomics and 83 changing discards). The nested form is what DebugMode always produced, ++ // because its comment nodes keep every block non-empty and this branch never ran. ++ else if (ElseIfFlattening && isIf && parent.Type == AstBlockType.Else && parent.Count == (hasElse ? 2 : 1)) + { + AstBlock parentOfParent = parent.Parent; + diff --git a/third_party/shader_translator/ultrs-shaders.csproj b/third_party/shader_translator/ultrs-shaders.csproj new file mode 100644 index 0000000..ee47857 --- /dev/null +++ b/third_party/shader_translator/ultrs-shaders.csproj @@ -0,0 +1,14 @@ + + + + Exe + net10.0 + disable + true + $(MSBuildThisFileDirectory)../../tooling/d2r-mobile/dm/third_party/ryujinx + + + + + + diff --git a/tools/bc7_build.py b/tools/bc7_build.py new file mode 100644 index 0000000..0624bf9 --- /dev/null +++ b/tools/bc7_build.py @@ -0,0 +1,81 @@ +"""Fill the BC7 cache from the player's RomFS (tools/bc7build/bc7build.c): every ASTC texture of the TexToGo (.txtg) and +BNTX (.bntx / .bntx.zs: the interface, some effects) files, encoded once. + +usage: python tools/bc7_build.py [--romfs data/romfs] [--cache data/textures] [--threads 8] [--quality 2] + [--limit N] [--max-block-area 144] +--max-block-area: ASTC blocks larger than this many texels are left out (decoded when met instead). Blocks larger than +100 texels (the interface's 12x12 maps, 3000x3000) become BC1 when opaque, else BC7 (the user's choice, 2026-10-25). +Runs on --threads logical CPUs at below-normal priority (the rest of the PC stays free), as tools/build_game.py does. +The cache is the player's own derived data: never in git or a build (README.md, Legal). +""" +import argparse +import json +import os +import pathlib +import subprocess +import sys +import time + +ROOT = pathlib.Path(__file__).resolve().parent.parent +sys.path.insert(0, str(ROOT / 'tools')) +from toolchain import CLANG64, env as tool_env # noqa: E402 +from build_game import run_limited # noqa: E402 +from romfs import RomFS # noqa: E402 + +SOURCES_C = ['tools/bc7build/bc7build.c', 'src/gfx/tex/astc.c', 'src/gfx/tex/bc7key.c'] +SOURCES_CXX = ['src/gfx/tex/bc7.cpp', 'src/gfx/tex/bc7enc/bc7enc.cpp', 'src/gfx/tex/bc7enc/rgbcx.cpp'] + + +def build_tool(env): + exe = ROOT / 'build/bc7build.exe' + newest = max((ROOT / s).stat().st_mtime for s in SOURCES_C + SOURCES_CXX) + if exe.exists() and exe.stat().st_mtime > newest: + return exe + objs = [] + (ROOT / 'build/obj').mkdir(parents=True, exist_ok=True) + for s in SOURCES_C + SOURCES_CXX: + cxx = s.endswith('.cpp') + obj = ROOT / 'build/obj' / ('bc7build_' + s.replace('/', '_').rsplit('.', 1)[0] + '.o') + cmd = [str(CLANG64 / ('clang++.exe' if cxx else 'clang.exe')), '-O2', '-march=x86-64-v3', '-w', '-c', str(ROOT / s), '-o', str(obj)] + if cxx: + cmd += ['-std=c++17', '-fno-exceptions', '-fno-rtti'] + subprocess.run(cmd, check=True, env=env) + objs.append(str(obj)) + subprocess.run([str(CLANG64 / 'clang.exe'), '-fuse-ld=lld', '-o', str(exe)] + objs + ['-static', '-pthread', '-lzstd', '-lc++'], check=True, env=env) + return exe + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument('--romfs', default='data/romfs') + ap.add_argument('--cache', default='data/textures') + ap.add_argument('--threads', type=int, default=8) + ap.add_argument('--quality', type=int, default=2) + ap.add_argument('--limit', type=int, default=0) + ap.add_argument('--max-block-area', type=int, default=144) + a = ap.parse_args() + rom = RomFS(ROOT / a.romfs) # a folder of plain files or the development romfs.bin + files = [dict(path=p, size=n, offset=rom.offset(p)) for p, n in rom.files.items() + if (p.startswith('TexToGo/') and p.endswith('.txtg')) or p.endswith('.bntx.zs') or p.endswith('.bntx')] + files.sort(key=lambda f: -f['size']) # large ones first: the threads finish together + if a.limit: + files = files[:a.limit] + cache = ROOT / a.cache + cache.mkdir(parents=True, exist_ok=True) + lst = cache / 'texture-list.txt' + with open(lst, 'w', encoding='utf-8', newline='\n') as out: + for f in files: + out.write(f"{f['offset']} {f['size']} {f['path']}\n") + env = tool_env() + exe = build_tool(env) + print(f'{len(files)} texture files -> {cache}', flush=True) + t0 = time.time() + rc = run_limited([str(exe), str(rom.image or rom.root), str(lst), str(cache), str(a.threads), str(a.quality), + str(a.max_block_area)], env, + (1 << a.threads) - 1) + print(f'{time.time() - t0:.0f} s', flush=True) + sys.exit(rc) + + +if __name__ == '__main__': + main() diff --git a/tools/bc7build/bc7build.c b/tools/bc7build/bc7build.c new file mode 100644 index 0000000..c6b5496 --- /dev/null +++ b/tools/bc7build/bc7build.c @@ -0,0 +1,323 @@ +// Fills the BC7 cache (src/gfx/vkb_bc7.c) from the player's RomFS before the game runs: every ASTC surface of the +// TexToGo textures (.txtg) is decoded and encoded to BC7 here, so the game uploads BC7 instead of decoding ASTC. +// Development and the player's own machine only: it reads the player's unpacked RomFS; nothing from the game goes into +// the repository or a build (README.md, Legal). +// +// usage: bc7build [threads] [quality] [max block area] +// list: lines " " (tools/bc7_build.py writes it): the offset in romfs.bin, or 0 with a folder of +// plain files (the player's dump), where each file is opened by its path +// [max block area]: ASTC with larger blocks is left out (12x12 = 144: the interface's maps, 9x the bytes as BC7) +// +// TexToGo (facts of the format as described by Switch Toolbox's TXTG.cs and TxtgSharp; this is the port's own code): +// 0x00 u16 header size, 0x02 u16 version, 0x04 "6PK0", 0x08 u16 width, 0x0A u16 height, 0x0C u16 array layers, +// 0x0E u8 mip levels, 0x3C u16 format, 0x44 u32 block info (ASTC: 0x7F00 | (block width - 1) << 4 | block height - 1). +// At the header size: per surface {u16 layer, u8 level, u8 1}, then per surface {u32 size, u32 6}, then the surfaces, +// each a zstd frame of block-linear (Tegra X1) data cut at its last byte. ASTC formats: 0x1xx (sRGB: 0x105, 0x109). +// BNTX (Nintendo's texture container, .bntx / .bntx.zs: the interface and some effects): "NX " at 0x20 with the texture +// count (u32 0x24) and the offset of its BRTI pointers (u64 0x28); a BRTI: mip levels u16 0x16, format u32 0x1C +// (ASTC: 0x2D..0x3A << 8 | 1 UNORM / 6 sRGB), width 0x24, height 0x28, depth 0x2C, array layers 0x30, block height +// log2 in the low 3 bits of 0x34, image size u32 0x50, mip offsets u64[] at the offset in 0x70 (layer 0; each layer +// image size / layers further). Data block-linear with that block height (smaller for small mips, as the GPU does). +// The key of a surface is that of the game's upload (tex/bc7key.c): its blocks in rows, its size and block size. +// Textures with large ASTC blocks (more than 100 texels: the interface's 12x12 maps) become BC1 when none of their +// texels is transparent (half the bytes of BC7; such low-rate ASTC holds no detail BC1 would lose), else BC7. +#include +#include +#include +#include +#include +#include +#include + +#include "../../src/gfx/tex/astc.h" +#include "../../src/gfx/tex/bc7.h" +#include "../../src/gfx/tex/bc7key.h" + +typedef struct { uint64_t off, size; char path[192]; } file_t; +static file_t *g_files; +static int g_nfiles, g_quality = 2, g_max_area = 144; // block area at most (12x12 = 144): larger ones skipped +enum { BC1_AREA = 100, BC1_LEVEL = 10 }; // larger ASTC blocks: BC1 when opaque (rgbcx level) +static _Atomic uint64_t g_bc1; +static const char *g_romfs; +static int g_folder; // g_romfs is a folder of plain files, not romfs.bin +static _Atomic int g_next; +static _Atomic uint64_t g_surfaces, g_astc_files, g_encoded, g_had, g_bad, g_texels; + +// ---- the cache files (the same records as vkb_bc7.c) +typedef struct { uint64_t a, b, off; uint32_t len, fmt; } rec_t; // fmt: 0 BC7, 1 BC1 +static FILE *g_pack, *g_idx; +static rec_t *g_tab; +static size_t g_cap, g_n; +static pthread_mutex_t g_mu = PTHREAD_MUTEX_INITIALIZER; +static rec_t *slot(uint64_t a, uint64_t b) { + size_t h = (size_t)(a ^ (b * 0x9E3779B97F4A7C15ull)) & (g_cap - 1); + while ((g_tab[h].a || g_tab[h].b) && !(g_tab[h].a == a && g_tab[h].b == b)) h = (h + 1) & (g_cap - 1); + return &g_tab[h]; +} +static void put(const rec_t *r) { + if ((g_n + 1) * 2 > g_cap) { + rec_t *old = g_tab; + size_t n = g_cap; + g_cap = g_cap ? g_cap * 2 : 1 << 16; + g_tab = calloc(g_cap, sizeof *g_tab); + if (!g_tab) { fprintf(stderr, "out of memory\n"); exit(1); } + for (size_t i = 0; i < n; i++) if (old[i].a || old[i].b) *slot(old[i].a, old[i].b) = old[i]; + free(old); + } + rec_t *s = slot(r->a, r->b); + if (!s->a && !s->b) g_n++; + *s = *r; +} +static int claim(bc7_key k) { // 1: not in the cache yet and now reserved for this thread (len 0 until written) + pthread_mutex_lock(&g_mu); + int mine = !(g_cap && (slot(k.a, k.b)->a || slot(k.a, k.b)->b)); + if (mine) { rec_t r = {k.a, k.b, 0, 0, 0}; put(&r); } + pthread_mutex_unlock(&g_mu); + return mine; +} +static void store(bc7_key k, const uint8_t *data, size_t n, int fmt) { + pthread_mutex_lock(&g_mu); + _fseeki64(g_pack, 0, SEEK_END); + rec_t r = {k.a, k.b, (uint64_t)_ftelli64(g_pack), (uint32_t)n, (uint32_t)fmt}; + if (fwrite(data, 1, n, g_pack) == n) { + fflush(g_pack); + _fseeki64(g_idx, 0, SEEK_END); + fwrite(&r, sizeof r, 1, g_idx); + fflush(g_idx); + put(&r); + } + pthread_mutex_unlock(&g_mu); +} +static void open_cache(const char *dir) { + char path[512]; + snprintf(path, sizeof path, "%s/bc7.pack", dir); + if (!(g_pack = fopen(path, "r+b")) && !(g_pack = fopen(path, "w+b"))) { fprintf(stderr, "cannot open %s\n", path); exit(1); } + _fseeki64(g_pack, 0, SEEK_END); + int64_t pack_size = _ftelli64(g_pack); + snprintf(path, sizeof path, "%s/bc7.idx", dir); + if (!(g_idx = fopen(path, "r+b")) && !(g_idx = fopen(path, "w+b"))) { fprintf(stderr, "cannot open %s\n", path); exit(1); } + rec_t r; + int64_t good = 0; + while (fread(&r, sizeof r, 1, g_idx) == 1 && (int64_t)(r.off + r.len) <= pack_size) { put(&r); good += sizeof r; } + _fseeki64(g_idx, good, SEEK_SET); + printf("cache %s: %zu surfaces, %.1f MiB\n", dir, g_n, pack_size / 1048576.0); +} + +// ---- Tegra X1 block-linear: a GOB is 64 bytes x 8 rows; GOBs stack `bhm` high +static uint32_t block_height0(uint32_t hb) { // the base level's GOBs per block, from its height in blocks + uint32_t x = hb + hb / 2; + return x >= 128 ? 16 : x >= 64 ? 8 : x >= 32 ? 4 : x >= 16 ? 2 : 1; +} +static uint32_t pow2_at_least(uint32_t v) { uint32_t p = 1; while (p < v) p <<= 1; return p; } +static int deswizzle(const uint8_t *src, size_t len, uint32_t wb, uint32_t hb, uint32_t bpb, uint32_t bhm, uint8_t *dst) { + uint32_t row = wb * bpb, gobs = (row + 63) / 64; + for (uint32_t y = 0; y < hb; y++) + for (uint32_t x = 0; x < row; x += bpb) { + size_t o = (size_t)(y / (8 * bhm)) * (512u * bhm * gobs) + (size_t)(x / 64) * (512u * bhm) + ((y % (8 * bhm)) / 8) * 512u + + ((x % 64) / 32) * 256u + ((y % 8) / 2) * 64u + ((x % 32) / 16) * 32u + (y % 2) * 16u + (x % 16); + if (o + bpb > len) return 0; + memcpy(dst + (size_t)y * row + x, src + o, bpb); + } + return 1; +} + +// The surfaces of one texture: decoded as they come; with large blocks, kept until the texture's format is known. +typedef struct { bc7_key k; uint8_t *rgba; uint32_t lw, lh; } dec_t; +typedef struct { dec_t *d; int n, cap, opaque, srgb, large; } tex_t; +static void encode_one(const dec_t *d, int fmt, int srgb) { + size_t n = fmt ? bc1_size(d->lw, d->lh, 1) : bc7_size(d->lw, d->lh, 1); + uint8_t *out = malloc(n); + if (!out) return; + if (fmt) { bc1_encode(d->rgba, d->lw, d->lh, 1, BC1_LEVEL, out); g_bc1++; } + else bc7_encode(d->rgba, d->lw, d->lh, 1, srgb, g_quality, out); + store(d->k, out, n, fmt); + g_encoded++; + g_texels += (uint64_t)d->lw * d->lh; + free(out); +} +static void tex_end(tex_t *t) { // a texture with large blocks: BC1 if no texel of it is transparent, else BC7 + for (int i = 0; i < t->n; i++) { encode_one(&t->d[i], t->opaque ? 1 : 0, t->srgb); free(t->d[i].rgba); } + t->n = 0; +} +// One ASTC surface in block-linear form: deswizzled, keyed, decoded and encoded unless the cache has it. +static void do_surface(tex_t *t, const uint8_t *sw, size_t len, uint32_t lw, uint32_t lh, uint32_t bw, uint32_t bh, uint32_t bhm, int srgb) { + uint32_t wb = (lw + bw - 1) / bw, hb = (lh + bh - 1) / bh; + uint8_t *rows = malloc((size_t)wb * hb * 16); + if (!rows || !deswizzle(sw, len, wb, hb, 16, bhm, rows)) { g_bad++; free(rows); return; } + g_surfaces++; + bc7_key k = bc7_key_of(rows, (size_t)wb * hb * 16, lw, lh, 1, bw, bh); + if (claim(k)) { + dec_t d = {k, malloc((size_t)lw * lh * 4), lw, lh}; + if (d.rgba) { + astc_decode_ldr(rows, lw, lh, bw, bh, srgb, d.rgba); + if (!t->large) { encode_one(&d, 0, srgb); free(d.rgba); } + else { + for (size_t i = 3; i < (size_t)lw * lh * 4 && t->opaque; i += 4) t->opaque = d.rgba[i] == 255; + if (t->n == t->cap) { t->cap = t->cap ? t->cap * 2 : 16; t->d = realloc(t->d, sizeof *t->d * (size_t)t->cap); } + t->d[t->n++] = d; + } + } + } else g_had++; + free(rows); +} +static void tex_begin(tex_t *t, uint32_t bw, uint32_t bh, int srgb) { t->n = 0; t->opaque = 1; t->srgb = srgb; t->large = bw * bh > BC1_AREA; } +static uint32_t mip_block_height(uint32_t hb, uint32_t bh0) { // GOBs per block of a level: as the GPU shrinks it for small levels + uint32_t bhm = pow2_at_least((hb + 7) / 8); + if (bhm > 16) bhm = 16; + return bhm > bh0 ? bh0 : bhm; +} + +static const uint8_t g_astc_dims[14][2] = {{4, 4}, {5, 4}, {5, 5}, {6, 5}, {6, 6}, {8, 5}, {8, 6}, {8, 8}, {10, 5}, {10, 6}, {10, 8}, {10, 10}, {12, 10}, {12, 12}}; +static void do_bntx(const uint8_t *d, size_t n) { + if (n < 0x30 || memcmp(d, "BNTX", 4) || memcmp(d + 0x20, "NX ", 4)) { g_bad++; return; } + uint32_t count; uint64_t arr; + memcpy(&count, d + 0x24, 4); memcpy(&arr, d + 0x28, 8); + for (uint32_t t = 0; t < count && arr + 8 * (t + 1) <= n; t++) { + uint64_t o; memcpy(&o, d + arr + 8 * t, 8); + if (o + 0x78 > n || memcmp(d + o, "BRTI", 4)) { g_bad++; continue; } + uint16_t mips; uint32_t fmt, layout, image; int32_t w, h, depth, layers; uint64_t ptrs; + memcpy(&mips, d + o + 0x16, 2); memcpy(&fmt, d + o + 0x1C, 4); memcpy(&w, d + o + 0x24, 4); memcpy(&h, d + o + 0x28, 4); + memcpy(&depth, d + o + 0x2C, 4); memcpy(&layers, d + o + 0x30, 4); memcpy(&layout, d + o + 0x34, 4); + memcpy(&image, d + o + 0x50, 4); memcpy(&ptrs, d + o + 0x70, 8); + uint32_t type = fmt >> 8; + if (type < 0x2D || type > 0x3A || depth > 1 || w <= 0 || h <= 0 || ptrs + 8ull * mips > n) continue; // not 2D ASTC + uint32_t bw = g_astc_dims[type - 0x2D][0], bh = g_astc_dims[type - 0x2D][1]; + if (bw * bh > (uint32_t)g_max_area) continue; + int srgb = (fmt & 0xFF) == 6; + if (layers < 1) layers = 1; + g_astc_files++; + tex_t tx = {0}; + tex_begin(&tx, bw, bh, srgb); + uint32_t bh0 = 1u << (layout & 7); + uint64_t stride = (uint64_t)image / (uint32_t)layers; + for (int32_t layer = 0; layer < layers; layer++) + for (uint32_t m = 0; m < mips; m++) { + uint64_t mo; memcpy(&mo, d + ptrs + 8 * m, 8); + mo += stride * (uint64_t)layer; + if (mo >= n) { g_bad++; continue; } + uint32_t lw = (uint32_t)w >> m ? (uint32_t)w >> m : 1, lh = (uint32_t)h >> m ? (uint32_t)h >> m : 1; + uint32_t hb = (lh + bh - 1) / bh; + do_surface(&tx, d + mo, n - mo, lw, lh, bw, bh, mip_block_height(hb, bh0), srgb); + } + tex_end(&tx); + free(tx.d); + } +} + +// The file's bytes (f->size of them) into buf: from the image at its offset, or from its own file in the folder. +static int read_file(FILE *rf, const file_t *f, uint8_t *buf) { + if (!g_folder) { + _fseeki64(rf, (int64_t)f->off, SEEK_SET); + return fread(buf, 1, f->size, rf) == f->size; + } + char path[4096]; + snprintf(path, sizeof path, "%s/%s", g_romfs, f->path); + FILE *ff = fopen(path, "rb"); + if (!ff) return 0; + int ok = fread(buf, 1, f->size, ff) == f->size; + fclose(ff); + return ok; +} +static void do_file(FILE *rf, const file_t *f, uint8_t **buf, size_t *cap) { + size_t pl = strlen(f->path); + int bntx = pl > 5 && (!strcmp(f->path + pl - 5, ".bntx") || (pl > 8 && !strcmp(f->path + pl - 8, ".bntx.zs"))); + if (bntx) { + if (*cap < f->size) { free(*buf); *cap = f->size; if (!(*buf = malloc(*cap))) { fprintf(stderr, "out of memory\n"); exit(1); } } + if (!read_file(rf, f, *buf)) { g_bad++; return; } + if (f->path[pl - 1] != 's') { do_bntx(*buf, f->size); return; } + unsigned long long raw = ZSTD_getFrameContentSize(*buf, f->size); + if (raw == ZSTD_CONTENTSIZE_ERROR || raw == ZSTD_CONTENTSIZE_UNKNOWN || raw > (1ull << 32)) { g_bad++; return; } + uint8_t *d = malloc(raw ? raw : 1); + if (!d || ZSTD_isError(ZSTD_decompress(d, raw, *buf, f->size))) { g_bad++; free(d); return; } + do_bntx(d, raw); + free(d); + return; + } + if (f->size < 0x50) return; + if (*cap < f->size) { free(*buf); *cap = f->size; if (!(*buf = malloc(*cap))) { fprintf(stderr, "out of memory\n"); exit(1); } } + uint8_t *b = *buf; + if (!read_file(rf, f, b) || memcmp(b + 4, "6PK0", 4)) { g_bad++; return; } + uint16_t hs, w, h, layers, fmt; uint8_t mips; uint32_t info; + memcpy(&hs, b, 2); memcpy(&w, b + 8, 2); memcpy(&h, b + 0xA, 2); memcpy(&layers, b + 0xC, 2); mips = b[0xE]; + memcpy(&fmt, b + 0x3C, 2); memcpy(&info, b + 0x44, 4); + if ((fmt >> 8) != 1 || (info & 0xFFFFFF00u) != 0x7F00) return; // not ASTC: the GPU samples it as it is + uint32_t bw = ((info >> 4) & 15) + 1, bh = (info & 15) + 1; + if (bw * bh > (uint32_t)g_max_area) return; + int srgb = fmt == 0x105 || fmt == 0x109; + uint32_t ns = (uint32_t)mips * (layers ? layers : 1); + if (!ns || hs + (size_t)ns * 12 > f->size) { g_bad++; return; } + g_astc_files++; + tex_t tx = {0}; + tex_begin(&tx, bw, bh, srgb); + const uint8_t *ent = b + hs, *sizes = ent + ns * 4, *data = sizes + ns * 8; + uint32_t bh0 = block_height0((h + bh - 1) / bh); + for (uint32_t i = 0; i < ns; i++) { + uint16_t layer; memcpy(&layer, ent + i * 4, 2); + uint8_t level = ent[i * 4 + 2]; + uint32_t zs; memcpy(&zs, sizes + i * 8, 4); + if (data + zs > b + f->size) { g_bad++; break; } + const uint8_t *z = data; + data += zs; + (void)layer; + uint32_t lw = w >> level ? w >> level : 1, lh = h >> level ? h >> level : 1; + uint32_t hb = (lh + bh - 1) / bh; + unsigned long long raw = ZSTD_getFrameContentSize(z, zs); + if (raw == ZSTD_CONTENTSIZE_ERROR || raw == ZSTD_CONTENTSIZE_UNKNOWN || raw > (1ull << 31)) { g_bad++; continue; } + uint8_t *sw = malloc(raw ? raw : 1); + if (!sw || ZSTD_isError(ZSTD_decompress(sw, raw, z, zs))) { g_bad++; free(sw); continue; } + do_surface(&tx, sw, raw, lw, lh, bw, bh, mip_block_height(hb, bh0), srgb); + free(sw); + } + tex_end(&tx); + free(tx.d); +} + +static void *worker(void *unused) { + (void)unused; + FILE *rf = NULL; // the image; with a folder, files are opened one by one (read_file) + if (!g_folder && !(rf = fopen(g_romfs, "rb"))) { fprintf(stderr, "cannot open %s\n", g_romfs); exit(1); } + uint8_t *buf = NULL; + size_t cap = 0; + for (int i; (i = atomic_fetch_add(&g_next, 1)) < g_nfiles;) { + do_file(rf, &g_files[i], &buf, &cap); + if (i % 500 == 0) { + printf("%d / %d files, %llu ASTC surfaces, %llu encoded, %.0f Mtexels\n", i, g_nfiles, (unsigned long long)g_surfaces, + (unsigned long long)g_encoded, g_texels / 1e6); + fflush(stdout); + } + } + free(buf); + if (rf) fclose(rf); + return NULL; +} + +int main(int argc, char **argv) { + if (argc < 4) { fprintf(stderr, "usage: bc7build [threads] [quality]\n"); return 2; } + g_romfs = argv[1]; + int threads = argc > 4 ? atoi(argv[4]) : 8; + if (argc > 5) g_quality = atoi(argv[5]); + if (argc > 6) g_max_area = atoi(argv[6]); + FILE *lf = fopen(argv[2], "r"); + if (!lf) { fprintf(stderr, "cannot open %s\n", argv[2]); return 1; } + int cap = 0; + file_t f; + struct stat st; + g_folder = !stat(g_romfs, &st) && S_ISDIR(st.st_mode); + while (fscanf(lf, "%llu %llu %191[^\n]", (unsigned long long *)&f.off, (unsigned long long *)&f.size, f.path) == 3) { + if (g_nfiles == cap) { cap = cap ? cap * 2 : 4096; g_files = realloc(g_files, sizeof *g_files * (size_t)cap); } + g_files[g_nfiles++] = f; + } + fclose(lf); + open_cache(argv[3]); + printf("%d files, %d threads, quality %d\n", g_nfiles, threads, g_quality); + pthread_t t[64]; + if (threads < 1) threads = 1; + if (threads > 64) threads = 64; + for (int i = 0; i < threads; i++) pthread_create(&t[i], NULL, worker, NULL); + for (int i = 0; i < threads; i++) pthread_join(t[i], NULL); + printf("done: %llu ASTC files, %llu surfaces (%llu encoded, of them %llu BC1; %llu already there), %llu unreadable, %.0f Mtexels encoded; cache %zu surfaces\n", + (unsigned long long)g_astc_files, (unsigned long long)g_surfaces, (unsigned long long)g_encoded, (unsigned long long)g_bc1, (unsigned long long)g_had, + (unsigned long long)g_bad, g_texels / 1e6, g_n); + fclose(g_pack); fclose(g_idx); + return 0; +} diff --git a/tools/build_game.py b/tools/build_game.py new file mode 100644 index 0000000..e121c5e --- /dev/null +++ b/tools/build_game.py @@ -0,0 +1,181 @@ +"""Write build/build.ninja for the runtime + the generated game code, and run ninja (MSYS2 CLANG64 clang/ninja). + +usage: python tools/build_game.py [--generated data/generated] [--target probe_cpu] [-j N] [--cpus N] +The ultratears target links UltraTears.exe in the game folder itself (its debug symbols: build/UltraTears.pdb). +Run in the game folder (tools/setup.py does): the generated C is in data/ (derived from the game), objects and +executables go to build/; neither is ever in git. +""" +import argparse +import os +import pathlib +import subprocess +import sys + +ROOT = pathlib.Path(__file__).resolve().parent.parent +sys.path.insert(0, str(ROOT / 'tools')) +from toolchain import CLANG64, env as tool_env # noqa: E402 +import version # noqa: E402 + +RUNTIME = ['src/runtime/dispatch.c', 'src/runtime/loader.c', 'src/runtime/platform/windows.c'] +TARGETS = {'probe_cpu': ['src/probe/probe_cpu.c'], + 'ultratears': ['src/host/boot.c', 'src/host/guest.c', 'src/host/svc.c', 'src/host/threads.c', 'src/host/ipc.c', + 'src/host/services_lm.c', 'src/host/overrides_oe.c', + 'src/host/overrides_time.c', + 'src/host/fs.c', 'src/host/account.c', + 'src/host/overrides_applet.c', 'src/host/overrides_input.c', + 'src/host/audio_renderer.c', 'src/host/audio_mixer.c', 'src/host/audio_effects.c', + 'src/host/audio_reverb.c', 'src/host/opus_codec.c', 'src/host/movie_host.c', 'src/host/hid_feedback.c', + 'src/host/shared_font.c', + 'src/host/window_sdl.c', 'src/vk/vk.c', 'src/vk/present.c', 'src/host/motion_fusion.c', + 'src/host/profiler.c', 'src/host/watch.c']} +# The graphics front end (MacNativeGames' NVN layer) with its macOS stand-ins: own include paths, own rule +GFX = ['src/gfx/nvn/hle_nvn.c', 'src/gfx/nvn/nvn_tex.c', 'src/gfx/nvn/totk_nvn_ext.c', 'src/gfx/nvn/nvn_registry.c', + 'src/gfx/nvn/nvn_thunks.gen.c', 'src/gfx/nvn/nvn_dirty_win.c', 'src/gfx/nvn/totk_timeline_win.c', + 'src/gfx/nvn/nvn_names.gen.c', 'src/gfx/backend_null.c', 'src/gfx/gfx_bridge.c', + 'src/gfx/front_win.c', 'src/gfx/nvn/totk_vpool_win.c', 'src/gfx/backend_vk.c', 'src/gfx/vkb_tex.c', 'src/gfx/vkb_shader.c', 'src/gfx/vkb_spv.c', 'src/gfx/vkb_audit.c', 'src/gfx/vkb_copy.c', 'src/gfx/vkb_desc.c', 'src/gfx/vkb_shadow.c', 'src/gfx/vkb_ucap.c', + 'src/gfx/vkb_draw.c', 'src/gfx/tex/nvn_deswizzle.c', 'src/gfx/tex/astc.c', 'src/gfx/vkb_bc7.c', + 'src/gfx/tex/bc7.cpp', 'src/gfx/tex/bc7enc/bc7enc.cpp', 'src/gfx/tex/bc7enc/rgbcx.cpp', 'src/gfx/tex/bc7key.c'] +GFX_INCLUDES = '-Isrc/gfx -Isrc/gfx/nvn -Isrc/gfx/nvn/include -Isrc/gfx/compat' +LIBS = '-static -Wl,--error-limit=0 -Wl,--disable-high-entropy-va -pthread -ldbghelp -lwinmm -lsynchronization -lshell32 -lopus -lvpx -lSDL3 -liconv' \ + ' -luser32 -lgdi32 -limm32 -lole32 -loleaut32 -lversion -luuid -ladvapi32 -lsetupapi -ldinput8 -lbcrypt -lc++' +CFLAGS = '-O2 -march=x86-64-v3 -g -gcodeview -fno-strict-aliasing -D_FILE_OFFSET_BITS=64' +GAME_CFLAGS = CFLAGS + ' -w -Isrc/runtime' + + +def rel(p): + return pathlib.Path(os.path.relpath(p, ROOT)).as_posix() + + +def write_if_changed(path, text): # an unchanged file keeps its time: nothing rebuilds + if not path.is_file() or path.read_text(encoding='utf-8') != text: + path.write_text(text, encoding='utf-8') + + +def version_files(build): + """build/version.h (the version and commit, tools/version.py) and the exe's resources: icon and version info.""" + v, commit = version.current() + n = ','.join(str(x) for x in version.numbers(v)) + write_if_changed(build / 'version.h', f'#define ULTRS_VERSION "{v}"\n#define ULTRS_COMMIT "{commit}"\n') + write_if_changed(build / 'ultratears.rc', f'''1 ICON "assets/icon.ico" +1 VERSIONINFO +FILEVERSION {n} +PRODUCTVERSION {n} +BEGIN + BLOCK "StringFileInfo" + BEGIN + BLOCK "040904B0" + BEGIN + VALUE "CompanyName", "UltraTears contributors" + VALUE "FileDescription", "UltraTears" + VALUE "FileVersion", "{v} ({commit})" + VALUE "LegalCopyright", "GPL-3.0-or-later. Not affiliated with Nintendo." + VALUE "OriginalFilename", "UltraTears.exe" + VALUE "ProductName", "UltraTears" + VALUE "ProductVersion", "{v}" + END + END + BLOCK "VarFileInfo" + BEGIN + VALUE "Translation", 0x409, 1200 + END +END +''') + return v + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument('--generated', default='data/generated') + ap.add_argument('--target', default='probe_cpu') + ap.add_argument('-j', type=int, default=0, help='parallel jobs (default: --cpus)') + ap.add_argument('--cpus', type=int, default=8, help='logical CPUs the build may use (affinity); the rest stay free for the user') + a = ap.parse_args() + a.j = a.j or a.cpus + gen = ROOT / a.generated + build = ROOT / 'build' + build.mkdir(exist_ok=True) + lines = [ + 'builddir = build', # ninja's own state (.ninja_log, .ninja_deps) in build/, not beside the sources + f'cflags = {CFLAGS} -Wall -Wno-unused-function -Ibuild', # build/version.h (the game's code: not, so a new version rebuilds nothing of it) + f'gameflags = {GAME_CFLAGS}', + 'rule cc', + ' command = clang -MD -MF $out.d $cflags -c $in -o $out', + ' depfile = $out.d', + ' deps = gcc', + f'gfxflags = {CFLAGS} -w {GFX_INCLUDES}', + 'rule gfxcc', + ' command = clang -MD -MF $out.d $gfxflags -c $in -o $out', + ' depfile = $out.d', + ' deps = gcc', + 'rule gfxcxx', # C++ (the BC7 encoder): no exceptions or RTTI, static libc++ at the link + ' command = clang++ -MD -MF $out.d $gfxflags -std=c++17 -fno-exceptions -fno-rtti -c $in -o $out', + ' depfile = $out.d', + ' deps = gcc', + 'rule gamecc', + ' command = clang -MD -MF $out.d $gameflags -c $in -o $out', # runtime headers (cpu.h, simd*.h) are inlined + ' depfile = $out.d', + ' deps = gcc', + 'rule rc', + ' command = llvm-windres --target=pe-x86-64 -I . $in -o $out', + 'rule link', + f' command = clang -fuse-ld=lld -g -Wl,--pdb=$pdb -o $out @$out.rsp {LIBS}', + ' rspfile = $out.rsp', + ' rspfile_content = $in', + '', + ] + objs = [] + for src in RUNTIME + TARGETS[a.target]: + obj = f'build/obj/{src[:-2].replace("/", "_")}.o' + lines.append(f'build {obj}: cc {src}') + objs.append(obj) + if a.target == 'ultratears': + # the port's icon (assets/icon.ico, the author's own drawing: README.md, Legal) and version in the resources + print(f'UltraTears {version_files(build)}', flush=True) + lines.append('build build/obj/ultratears_res.o: rc build/ultratears.rc | assets/icon.ico') + objs.append('build/obj/ultratears_res.o') + for src in GFX: + cxx = src.endswith('.cpp') + obj = f'build/obj/{src.rsplit(".", 1)[0].replace("/", "_")}.o' + lines.append(f'build {obj}: {"gfxcxx" if cxx else "gfxcc"} {src}') + objs.append(obj) + game_objs = [] + for module in ('main', 'subsdk0', 'sdk'): + for c in sorted((gen / module).glob('*.c')): + obj = f'build/game/{module}/{c.stem}.o' + lines.append(f'build {obj}: gamecc {rel(c)}') + text = c.read_text(errors='replace') + # [temporary] a function with an unresolved computed branch is one switch over every + # instruction; with thousands of labels clang -O1/-O2 takes from minutes to hours per file (main_376: + # 20+ min at -O1), -O0 15 s. Such functions run unoptimized until their tables resolve. + if 'L_dispatch:' in text and text.count(chr(10) + 'L_') > 2000: + lines.append(' gameflags = ' + GAME_CFLAGS.replace('-O2', '-O0')) + game_objs.append(obj) + if not game_objs: + sys.exit(f'no generated C in {gen}: run tools/gen_game.py first') + exe = 'UltraTears.exe' if a.target == 'ultratears' else f'build/{a.target}.exe' # the game: in the game folder + lines.append(f'build {exe}: link {" ".join(objs + game_objs)}') + lines.append(f' pdb = build/{pathlib.Path(exe).stem}.pdb') + lines.append(f'default {exe}') + (build / 'build.ninja').write_text('\n'.join(lines) + '\n') + env = tool_env() + print(f'{len(game_objs)} generated files, {len(objs)} runtime files', flush=True) + # The user works on this PC during builds: ninja starts suspended, gets the first --cpus logical CPUs and + # below-normal priority, then runs; every clang it starts inherits both, so the other CPUs stay free. + ninja = [str(CLANG64 / 'ninja.exe'), '-C', str(ROOT), '-f', 'build/build.ninja', f'-j{a.j}', '-k', '0'] + sys.exit(run_limited(ninja, env, (1 << a.cpus) - 1)) + + +def run_limited(cmd, env, mask): + import ctypes + CREATE_SUSPENDED, BELOW_NORMAL_PRIORITY_CLASS = 0x4, 0x4000 + p = subprocess.Popen(cmd, env=env, creationflags=CREATE_SUSPENDED | BELOW_NORMAL_PRIORITY_CLASS) + handle = int(p._handle) + if not ctypes.windll.kernel32.SetProcessAffinityMask(handle, ctypes.c_size_t(mask)): + p.kill() + sys.exit('SetProcessAffinityMask failed') + ctypes.windll.ntdll.NtResumeProcess(handle) + return p.wait() + + +if __name__ == '__main__': + main() diff --git a/tools/build_xlate.py b/tools/build_xlate.py new file mode 100644 index 0000000..6ef638d --- /dev/null +++ b/tools/build_xlate.py @@ -0,0 +1,44 @@ +"""Build the shader translator ultrs-shaders (MacNativeGames' totk_xlate service around the Ryujinx 1.3.2 shader +translator in third_party/maxwell_shader, both MIT) as one self-contained Windows program: build/shaders/ultrs-shaders.exe, which needs no .NET +installed to run (dev-local/build/shaders; tools/deploy.py and tools/make_package.py take it from there). It turns the game's Switch GPU shaders (Maxwell machine code) into SPIR-V for Vulkan. + +usage: python tools/build_xlate.py [--dotnet dotnet] +Sources: third_party/shader_translator (the service; its patches/ are already applied to the copy in third_party/maxwell_shader, +kept as the record of what differs from Ryujinx 1.3.2). Needs the .NET 10 SDK and, the first time, network access for +the NuGet packages; everything it fetches stays inside the project (dev-local/nuget, dev-local/dotnet-home). +The builder package carries the result, so players never run this. +""" +import argparse +import os +import pathlib +import shutil +import subprocess +import sys + +ROOT = pathlib.Path(__file__).resolve().parent.parent + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument('--dotnet', default='dotnet') + a = ap.parse_args() + out = ROOT / 'dev-local/build/shaders' # a development tool's build: no game data, not in git + env = dict(os.environ, NUGET_PACKAGES=str(ROOT / 'dev-local/nuget'), DOTNET_CLI_HOME=str(ROOT / 'dev-local/dotnet-home'), + DOTNET_CLI_TELEMETRY_OPTOUT='1', DOTNET_NOLOGO='1', DOTNET_SKIP_FIRST_TIME_EXPERIENCE='1') + proj = ROOT / 'third_party/shader_translator/ultrs-shaders.csproj' + rc = subprocess.run([a.dotnet, 'publish', str(proj), '-c', 'Release', '-r', 'win-x64', '--self-contained', 'true', + '-p:PublishSingleFile=true', '-p:InvariantGlobalization=true', + '-p:IncludeNativeLibrariesForSelfExtract=true', + f'-p:RyuSrc={ROOT / "third_party/maxwell_shader"}', + '-o', str(out), '-nologo', '-v:q', '-maxcpucount:4'], env=env).returncode + if rc: + sys.exit(rc) + exe = out / 'ultrs-shaders.exe' + for f in out.iterdir(): # one program; the debug symbols are not needed + if f.suffix == '.pdb': + f.unlink() + print(f'built {exe} ({exe.stat().st_size / 1e6:.0f} MB)') + + +if __name__ == '__main__': + main() diff --git a/tools/compare_spirv_opt.py b/tools/compare_spirv_opt.py new file mode 100644 index 0000000..5ca080d --- /dev/null +++ b/tools/compare_spirv_opt.py @@ -0,0 +1,66 @@ +"""Compare two translators' SPIR-V after spirv-opt -O on a sample of shaders both translated validly. + +usage: python tools/compare_spirv_opt.py work/shaders/v143 [--sample 1000] +Prints the raw and optimized size totals per translator and per stage (instruction words, from the module size). +""" +import argparse +import collections +import ctypes +import pathlib +import random +import subprocess +import tempfile +import threading + +OPT = r'C:\VulkanSDK\1.4.363.0\Bin\spirv-opt.exe' + + +def valid(base, name): + rows = (base / f'{name}-spirv-val.tsv').read_text().splitlines()[1:] + return {r.split('\t')[0] for r in rows if r.split('\t')[1] == 'ok'} + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument('dir', type=pathlib.Path) + ap.add_argument('--sample', type=int, default=1000) + a = ap.parse_args() + stage = {l.split('\t')[0]: l.split('\t')[1] for l in (a.dir / 'shaders.tsv').read_text().splitlines()[1:]} + keys = sorted(valid(a.dir, 'ryujinx') & valid(a.dir, 'eden')) + random.Random(1).shuffle(keys) + keys = keys[:a.sample] + tmp = pathlib.Path(tempfile.mkdtemp(dir=a.dir)) + sizes = collections.defaultdict(lambda: [0, 0, 0]) # (translator, stage) -> raw, optimized, count + lock = threading.Lock() + failed = collections.Counter() + + def worker(chunk): + for key in chunk: + for name in ('ryujinx', 'eden'): + src = a.dir / name / f'{key}.spv' + dst = tmp / f'{name}-{key}.spv' + p = subprocess.Popen([OPT, '-O', '--target-env=vulkan1.3', '--scalar-block-layout', str(src), '-o', str(dst)], + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, creationflags=0x4 | 0x4000) + ctypes.windll.kernel32.SetProcessAffinityMask(int(p._handle), ctypes.c_size_t(0xFF)) + ctypes.windll.ntdll.NtResumeProcess(int(p._handle)) + if p.wait() or not dst.exists(): + with lock: failed[name] += 1 + continue + with lock: + s = sizes[(name, stage[key])] + s[0] += src.stat().st_size; s[1] += dst.stat().st_size; s[2] += 1 + dst.unlink() + threads = [threading.Thread(target=worker, args=(keys[i::8],)) for i in range(8)] + for t in threads: t.start() + for t in threads: t.join() + tmp.rmdir() + print(f'sample: {len(keys)} shaders valid in both; spirv-opt failures: {dict(failed)}') + for st in ('vs', 'ps', 'cs'): + r, e = sizes[('ryujinx', st)], sizes[('eden', st)] + if r[2] and e[2]: + print(f'{st}: n={r[2]:5d} raw KB/shader ryujinx {r[0]/r[2]/1e3:6.1f} eden {e[0]/e[2]/1e3:6.1f} ' + f'optimized ryujinx {r[1]/r[2]/1e3:6.1f} eden {e[1]/e[2]/1e3:6.1f} (eden/ryujinx {e[1]/r[1]:.2f})') + + +if __name__ == '__main__': + main() diff --git a/tools/deploy.py b/tools/deploy.py new file mode 100644 index 0000000..0a3b013 --- /dev/null +++ b/tools/deploy.py @@ -0,0 +1,95 @@ +"""Development: bring the game folder (default ..\\release-test, the player's layout from the builder package) up to this +working tree, then build there with its own setup.py, as a player updating the port would. + +usage: python tools/deploy.py [--home ..\\release-test] [--cpus 8] [--no-setup] + - the source: every file of the public tree (tools/public_tree.py: what git tracks, without the internal notes), as + it is in this working tree (uncommitted edits too), is copied where it differs; files a previous deploy put there + and the tree no longer has are removed (the list is kept in \\deploy-files.txt); + - the shader translator: dev-local\\build\\shaders\\ultrs-shaders.exe, if built (tools/build_xlate.py), replaces + \\deps\\shaders\\ultrs-shaders.exe when it differs; + - then \\deps\\python\\python.exe tools\\setup.py --cpus N in (incremental: seconds to minutes). +Never touches the game folder's game\\, data\\, build\\, userdata\\, diag\\ or the rest of deps\\. The source folder holds +no game data at all: everything made from the game lives in the game folder. +""" +import argparse +import filecmp +import os +import pathlib +import shutil +import subprocess +import sys + +ROOT = pathlib.Path(__file__).resolve().parent.parent +sys.path.insert(0, str(ROOT / 'tools')) +import public_tree # noqa: E402 +import version # noqa: E402 + +KEEP = {'game', 'data', 'build', 'userdata', 'diag', 'deps'} # the game folder's own: never written by a deploy +# made by tools/make_package.py or setup.py, not source files: a deploy leaves them +PACKAGE_OWN = {'setup.bat', 'VERSION.txt', 'THIRD-PARTY-NOTICES.md', 'licenses', 'ultratears.cfg', 'UltraTears.exe', + 'UltraTears.pdb', 'deploy-files.txt'} + + +def public_files(): + """Paths of the public tree that exist in the working tree (tracked files, PRIVATE left out).""" + out = subprocess.run(['git', '-C', str(ROOT), 'ls-files', '-z'], check=True, capture_output=True).stdout + files = [f for f in out.decode('utf-8').split('\0') if f] + return [f for f in files if f.split('/')[0] not in public_tree.PRIVATE and (ROOT / f).is_file()] + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument('--home', type=pathlib.Path, default=ROOT.parent / 'release-test') + ap.add_argument('--cpus', type=int, default=8) + ap.add_argument('--no-setup', action='store_true') + a = ap.parse_args() + home = a.home.resolve() + if not (home / 'deps/python/python.exe').is_file(): + sys.exit(f'{home} is not a game folder: unpack a builder package there first (deps\\python is missing)') + files = public_files() + if any(f.split('/')[0] in KEEP for f in files): + sys.exit('a tracked path would overwrite the game folder\'s own data: refusing') + changed = 0 + for f in files: + src, dst = ROOT / f, home / f + if dst.is_file() and filecmp.cmp(src, dst, shallow=False): + continue + dst.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(src, dst) + changed += 1 + listing = home / 'deploy-files.txt' + if listing.is_file(): + old = set(listing.read_text(encoding='utf-8').splitlines()) + else: # the first deploy over an unpacked package: its source files are the old ones + old = {p.relative_to(home).as_posix() for p in home.rglob('*') if p.is_file() and + p.relative_to(home).parts[0] not in KEEP | PACKAGE_OWN} + removed = 0 + for f in sorted(old - set(files)): + p = home / f + if f.split('/')[0] not in KEEP | PACKAGE_OWN and p.is_file(): # only source files + p.unlink() + removed += 1 + for d in p.parents: # folders left empty go too + if d == home or any(d.iterdir()): + break + d.rmdir() + listing.write_text('\n'.join(files) + '\n', encoding='utf-8') + v_txt = version.version_txt(*version.current()) # the game folder has no .git: the version goes with the files + if not (home / 'VERSION.txt').is_file() or (home / 'VERSION.txt').read_text(encoding='utf-8') != v_txt: + (home / 'VERSION.txt').write_text(v_txt, encoding='utf-8') + xlate = ROOT / 'dev-local/build/shaders/ultrs-shaders.exe' + dst = home / 'deps/shaders/ultrs-shaders.exe' + if xlate.is_file() and not (dst.is_file() and filecmp.cmp(xlate, dst, shallow=False)): + dst.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(xlate, dst) + changed += 1 + print(f'deploy -> {home}: {changed} files updated, {removed} removed', flush=True) + if a.no_setup: + return + rc = subprocess.run([str(home / 'deps/python/python.exe'), str(home / 'tools/setup.py'), '--cpus', str(a.cpus)], + cwd=home, stdin=subprocess.DEVNULL, env=dict(os.environ, PYTHONNOUSERSITE='1')).returncode + sys.exit(rc) + + +if __name__ == '__main__': + main() diff --git a/tools/dev_glsl_tree.py b/tools/dev_glsl_tree.py new file mode 100644 index 0000000..8bbb97b --- /dev/null +++ b/tools/dev_glsl_tree.py @@ -0,0 +1,39 @@ +"""Development: the expression tree of a variable in SPIRV-Cross GLSL output (which inputs feed an output). + +usage: python tools/dev_glsl_tree.py [depth] +Follows `float _N = expr;` / `_64.x = _N;` assignments upward; uniforms (fp_cN_1._m0[i].c), texture samples and +inputs are leaves. +""" +import re +import sys + + +def main(): + path, root = sys.argv[1], sys.argv[2] + depth = int(sys.argv[3]) if len(sys.argv) > 3 else 12 + defs = {} + for line in open(path, encoding="utf-8"): + m = re.match(r"\s*(?:precise\s+)?(?:float|int|uint|bool|vec[234]|ivec[234])?\s*([_\w.]+)\s*=\s*(.+);\s*$", line) + if m: + defs.setdefault(m.group(1), []).append(m.group(2)) + seen = set() + + def show(var, d, indent): + exprs = defs.get(var) + if not exprs: + print(f"{indent}{var} (input)") + return + expr = exprs[-1] if len(exprs) == 1 else " | ".join(exprs) + print(f"{indent}{var} = {expr}") + if d == 0 or var in seen: + return + seen.add(var) + for ref in sorted(set(re.findall(r"\b_\d+\b(?:\.[xyzw])?", expr))): + if ref != var: + show(ref, d - 1, indent + " ") + + show(root, depth, "") + + +if __name__ == "__main__": + main() diff --git a/tools/dev_lum.py b/tools/dev_lum.py new file mode 100644 index 0000000..5493bf0 --- /dev/null +++ b/tools/dev_lum.py @@ -0,0 +1,33 @@ +"""Development: mean linear colour of HDR targets in a TOTK_DUMP folder (the dump stores float formats Reinhard-mapped). + +usage: python tools/dev_lum.py [name filter] +""" +import glob +import os +import sys + +from PIL import Image + + +def linear(t): + t = t / 255.0 + return t / (1 - t) if t < 0.999 else 99.0 + + +def main(): + folder = sys.argv[1] + filt = sys.argv[2] if len(sys.argv) > 2 else "" + for f in sorted(glob.glob(os.path.join(folder, "*.bmp"))): + name = os.path.basename(f) + if filt not in name: + continue + fmt = name.split("_")[1] + im = Image.open(f).convert("RGB").resize((64, 36)) + px = list(im.get_flattened_data()) if hasattr(im, "get_flattened_data") else list(im.getdata()) + conv = linear if fmt in ("3f", "29") else (lambda v: v / 255.0) + m = [sum(conv(p[i]) for p in px) / len(px) for i in range(3)] + print(f"{name:28s} mean {m[0]:7.3f} {m[1]:7.3f} {m[2]:7.3f}") + + +if __name__ == "__main__": + main() diff --git a/tools/dev_player_dump.py b/tools/dev_player_dump.py new file mode 100644 index 0000000..cd3a271 --- /dev/null +++ b/tools/dev_player_dump.py @@ -0,0 +1,41 @@ +"""Development: lay out the development extraction as a player's dump (a folder with exefs/ and romfs/ of plain files), +to test tools/setup.py and the RomFS folder mode as a player would meet them. + +usage: python tools/dev_player_dump.py --src [--out ../release-test/game] +Game-derived data: written outside the source folder (the game folder's game/), never in git. +""" +import argparse +import json +import pathlib +import shutil + +ROOT = pathlib.Path(__file__).resolve().parent.parent + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument('--src', required=True) + ap.add_argument('--out', default=str(ROOT.parent / 'release-test/game')) + a = ap.parse_args() + src, out = ROOT / a.src, ROOT / a.out + (out / 'exefs').mkdir(parents=True, exist_ok=True) + for f in (src / 'exefs').iterdir(): + shutil.copyfile(f, out / 'exefs' / f.name) + files = json.loads((src / 'romfs/index.json').read_text())['files'] + done = 0 + with open(src / 'romfs/romfs.bin', 'rb') as img: + for f in files: + dst = out / 'romfs' / f['path'] + if dst.is_file() and dst.stat().st_size == f['size']: + continue + dst.parent.mkdir(parents=True, exist_ok=True) + img.seek(f['offset']) + dst.write_bytes(img.read(f['size'])) + done += 1 + if done % 20000 == 0: + print(f'{done} files', flush=True) + print(f'{len(files)} files in {out / "romfs"} ({done} written)') + + +if __name__ == '__main__': + main() diff --git a/tools/dev_timeline.py b/tools/dev_timeline.py new file mode 100644 index 0000000..e413b24 --- /dev/null +++ b/tools/dev_timeline.py @@ -0,0 +1,68 @@ +"""Read a TOTK_TIMELINE file (src/gfx/nvn/totk_timeline_win.c) and show who waits for whom in each frame. + +usage: python tools/dev_timeline.py [frame] +Without a frame: per frame, the time of each kind of span (game thread submit / present / sync waits, the replay of +submissions on the translation thread, GPU spans). With a frame: that frame's events in order, relative to its first. +""" +import collections +import sys + +SPANS = {'sub_in': ('sub_out', 'submit'), 'pres_in': ('pres_out', 'present'), 'wait_in': ('wait_out', 'sync wait'), + 'acq_in': ('acq_out', 'acquire'), 'rep_begin': ('rep_end', 'replay'), 'finish_in': ('finish_out', 'finish')} + + +def load(path): + ev = [] + for line in open(path, encoding='utf-8'): + if line.startswith('#'): + continue + t, kind, tid, frame, a, b = line.rstrip('\n').split('\t') + ev.append((int(t), kind, int(tid), int(frame), int(a), int(b))) + ev.sort() + return ev + + +def spans(ev): + """(start, end, label, tid, frame) for every paired span and every GPU span.""" + out, open_ = [], {} + for t, kind, tid, frame, a, b in ev: + if kind in SPANS: + open_[(tid, kind)] = (t, frame) + elif kind == 'gpu': + out.append((t, b, 'GPU', 0, frame)) + else: + for start, (end, label) in SPANS.items(): + if kind == end and (tid, start) in open_: + t0, f0 = open_.pop((tid, start)) + out.append((t0, t, label, tid, f0)) + return out + + +def main(): + ev = load(sys.argv[1]) + sp = spans(ev) + pres = [t for t, k, *_ in ev if k == 'pres_in'] + if len(sys.argv) > 2: + f = int(sys.argv[2]) + t0 = min(t for t, k, tid, fr, a, b in ev if fr == f) + for s, e, label, tid, fr in sorted(sp): + if fr == f or (label == 'GPU' and t0 - 50e6 <= s <= t0 + 150e6): + print(f'{(s - t0) / 1e6:8.2f} .. {(e - t0) / 1e6:8.2f} ms {e - s and (e - s) / 1e6:7.2f} ms {label:10} thread {tid}') + return + per = collections.defaultdict(lambda: collections.defaultdict(float)) + threads = collections.defaultdict(lambda: collections.defaultdict(float)) + for s, e, label, tid, fr in sp: + per[fr][label] += (e - s) / 1e6 + threads[label][tid] += (e - s) / 1e6 + frames = sorted(f for f in per if f) + gaps = [(b - a) / 1e6 for a, b in zip(pres, pres[1:])] + print(f'presents: {len(pres)}, frame time (present to present) mean {sum(gaps) / max(1, len(gaps)):.1f} ms') + labels = ['submit', 'present', 'acquire', 'sync wait', 'finish', 'replay', 'GPU'] + print('frame ' + ''.join(f'{l:>11}' for l in labels)) + for f in frames: + print(f'{f:5} ' + ''.join(f'{per[f].get(l, 0):11.1f}' for l in labels)) + print('threads per span kind:', {k: {t: round(v, 1) for t, v in d.items()} for k, d in threads.items()}) + + +if __name__ == '__main__': + main() diff --git a/tools/eden_xlate/CMakeLists.txt b/tools/eden_xlate/CMakeLists.txt new file mode 100644 index 0000000..01a37d4 --- /dev/null +++ b/tools/eden_xlate/CMakeLists.txt @@ -0,0 +1,35 @@ +# Offline Eden shader translator (src/shader_recompiler, SPIR-V backend only) for comparing it with Ryujinx's. +# Sources come from refs/eden (GPL-3.0-or-later / GPL-2.0-or-later) and refs/sirit; common/settings.h, logging.h and +# assert.h are replaced by the stand-ins in shim/. +cmake_minimum_required(VERSION 3.20) +project(eden_xlate CXX) +set(CMAKE_CXX_STANDARD 20) +set(CMAKE_CXX_STANDARD_REQUIRED ON) + +set(REFS ${CMAKE_CURRENT_LIST_DIR}/../../refs) +set(EDEN ${REFS}/eden/src) + +set(SPIRV-Headers_DIR ${CMAKE_CURRENT_LIST_DIR}/cmake) +set(SIRIT_USE_SYSTEM_SPIRV_HEADERS ON CACHE BOOL "" FORCE) +set(SIRIT_TESTS OFF CACHE BOOL "" FORCE) +add_subdirectory(${REFS}/sirit sirit EXCLUDE_FROM_ALL) + +find_package(fmt REQUIRED) +find_package(Boost REQUIRED) + +file(GLOB_RECURSE RECOMPILER_SOURCES ${EDEN}/shader_recompiler/*.cpp) +list(FILTER RECOMPILER_SOURCES EXCLUDE REGEX "/backend/(glsl|glasm)/") + +add_executable(eden_xlate eden_xlate.cpp ${RECOMPILER_SOURCES}) +target_include_directories(eden_xlate BEFORE PRIVATE ${CMAKE_CURRENT_LIST_DIR}/shim) +target_include_directories(eden_xlate PRIVATE ${EDEN}) +target_link_libraries(eden_xlate PRIVATE sirit::sirit fmt::fmt Boost::headers) +target_compile_options(eden_xlate PRIVATE -O2 -march=x86-64-v3 -fbracket-depth=1024 -w -include ${CMAKE_CURRENT_LIST_DIR}/shim/eden_fmt_enums.h) + +# sirit v1.0.5 uses std::abort without (MSYS2 CLANG64's libc++ does not pull it in) and builds with +# -Werror; both are adjusted from here instead of editing the reference clone. +foreach(t siritobj sirit) + if(TARGET ${t}) + target_compile_options(${t} PRIVATE -include cstdlib -Wno-error) + endif() +endforeach() diff --git a/tools/eden_xlate/cmake/SPIRV-HeadersConfig.cmake b/tools/eden_xlate/cmake/SPIRV-HeadersConfig.cmake new file mode 100644 index 0000000..193d744 --- /dev/null +++ b/tools/eden_xlate/cmake/SPIRV-HeadersConfig.cmake @@ -0,0 +1,7 @@ +# The SPIR-V headers MSYS2 CLANG64 installs (include/spirv) without a CMake package: what sirit's +# find_package(SPIRV-Headers) needs. +if(NOT TARGET SPIRV-Headers::SPIRV-Headers) + add_library(SPIRV-Headers::SPIRV-Headers INTERFACE IMPORTED) + set_target_properties(SPIRV-Headers::SPIRV-Headers PROPERTIES INTERFACE_INCLUDE_DIRECTORIES "C:/msys64/clang64/include") +endif() +set(SPIRV-Headers_FOUND TRUE) diff --git a/tools/eden_xlate/eden_xlate.cpp b/tools/eden_xlate/eden_xlate.cpp new file mode 100644 index 0000000..aba8b94 --- /dev/null +++ b/tools/eden_xlate/eden_xlate.cpp @@ -0,0 +1,190 @@ +// Offline translation of TOTK's Maxwell shaders to SPIR-V with Eden's shader recompiler (src/shader_recompiler), +// for comparing it with the Ryujinx translator on the same shaders and the same default answers. +// Modeled on mk8-recomp's offline_nvn_shader.cpp (GPL-3.0-or-later), which drives suyu's recompiler the same way. +// +// usage: eden_xlate +// raw dir: .ctl (0x880-byte NVN control section) and .bin (the code as the game hands it over) +// writes /.spv and prints one line per shader: key, ok|fail, seconds, SPIR-V bytes, message +// Default answers, as MacNativeGames gives the Ryujinx translator for the default bind state: every texture 2D +// RGBA8 float, vertex inputs and colour outputs float, viewport transform on. +#include +#include +#include +#include +#include +#include +#include +#include + +#include "shader_recompiler/backend/spirv/emit_spirv.h" +#include "shader_recompiler/environment.h" +#include "shader_recompiler/frontend/maxwell/control_flow.h" +#include "shader_recompiler/frontend/maxwell/translate_program.h" +#include "shader_recompiler/host_translate_info.h" +#include "shader_recompiler/object_pool.h" +#include "shader_recompiler/profile.h" +#include "shader_recompiler/runtime_info.h" + +namespace { + +std::vector read_file(const std::string& path) { + std::ifstream f(path, std::ios::binary | std::ios::ate); + if (!f) throw std::runtime_error("cannot read " + path); + std::vector b(static_cast(f.tellg())); + f.seekg(0); + f.read(b.data(), static_cast(b.size())); + return b; +} + +u32 word(const std::vector& b, size_t at) { + if (at + 4 > b.size()) throw std::runtime_error("read out of bounds"); + u32 v; + std::memcpy(&v, b.data() + at, 4); + return v; +} + +class NvnEnvironment final : public Shader::Environment { +public: + NvnEnvironment(const std::vector& ctl, const std::vector& bin, bool compute) : code(bin) { + program_start = word(ctl, 0x708); + program_size = word(ctl, 0x6f8); + c1_start = word(ctl, 0x700); + c1_size = word(ctl, 0x6fc); + if (program_start > code.size() || program_size > code.size() - program_start) throw std::runtime_error("program extent"); + if (c1_start > code.size() || c1_size > code.size() - c1_start) throw std::runtime_error("c1 extent"); + start_address = program_start; + is_proprietary_driver = true; + if (compute) { + stage = Shader::Stage::Compute; + } else { + std::memcpy(&sph, code.data() + program_start, sizeof sph); + const u32 type = sph.common0.shader_type; + stage = type == 1 ? Shader::Stage::VertexB : type == 5 ? Shader::Stage::Fragment + : type == 4 ? Shader::Stage::Geometry : type == 2 ? Shader::Stage::TessellationControl + : type == 3 ? Shader::Stage::TessellationEval : throw std::runtime_error("unknown shader type"); + } + } + u64 ReadInstruction(u32 address) override { + if (address + 8 > code.size()) throw std::runtime_error("instruction read out of bounds"); + u64 v; + std::memcpy(&v, code.data() + address, 8); + return v; + } + u32 ReadCbufValue(u32 index, u32 offset) override { + if (index == 1 && offset + 4 <= c1_size) return word(code, c1_start + offset); + return 0; + } + Shader::TextureType ReadTextureType(u32) override { return Shader::TextureType::Color2D; } + Shader::TexturePixelFormat ReadTexturePixelFormat(u32) override { return Shader::TexturePixelFormat::A8B8G8R8_UNORM; } + bool IsTexturePixelFormatInteger(u32) override { return false; } + u32 ReadViewportTransformState() override { return 1; } + u32 TextureBoundBuffer() const override { return 2; } // NVN binds texture handles in c[2] + u32 LocalMemorySize() const override { return stage == Shader::Stage::Compute ? 0x800 : static_cast(sph.LocalMemorySize()); } + u32 SharedMemorySize() const override { return 0xc000; } // Maxwell's maximum: the control's own value is not decoded yet + std::array WorkgroupSize() const override { return {1, 1, 1}; } + bool HasHLEMacroState() const override { return false; } + std::optional GetReplaceConstBuffer(u32, u32) override { return std::nullopt; } + void Dump(u64, u64) override {} + + u32 program_start{}, program_size{}, c1_start{}, c1_size{}; + +private: + const std::vector& code; +}; + +Shader::Profile desktop_profile() { + // A current desktop GPU (RTX 30-series class) through Vulkan 1.3, as Eden's renderer would report it. + Shader::Profile p{}; + p.supported_spirv = 0x00010600; + p.unified_descriptor_binding = true; + p.support_descriptor_aliasing = true; + p.support_int8 = p.support_uniform_and_storage_buffer_8bit = p.support_storage_buffer_8bit = true; + p.support_int16 = p.support_uniform_and_storage_buffer_16bit = p.support_storage_buffer_16bit = true; + p.support_int64 = true; + p.support_float_controls = true; + p.support_separate_denorm_behavior = p.support_separate_rounding_mode = true; + p.support_fp16_denorm_preserve = p.support_fp32_denorm_preserve = true; + p.support_fp16_denorm_flush = p.support_fp32_denorm_flush = true; + p.support_fp16_signed_zero_nan_preserve = p.support_fp32_signed_zero_nan_preserve = p.support_fp64_signed_zero_nan_preserve = true; + p.support_quad_shuffles = p.support_vote = p.support_shuffle_relative = true; + p.support_viewport_index_layer_non_geometry = true; + p.support_typeless_image_loads = true; + p.support_demote_to_helper_invocation = true; + p.support_int64_atomics = p.support_shared_int64_atomics = true; + p.support_derivative_control = true; + p.support_native_ndc = true; + p.support_scaled_attributes = true; + p.support_multi_viewport = true; + p.support_geometry_streams = true; + p.support_sampled_image_array_nonuniform_indexing = p.support_storage_image_array_nonuniform_indexing = true; + p.support_uniform_texel_buffer_array_nonuniform_indexing = p.support_storage_texel_buffer_array_nonuniform_indexing = true; + p.warp_size_potentially_larger_than_guest = false; + p.need_declared_frag_colors = true; + p.max_subgroup_size = 32; + p.min_ssbo_alignment = 16; + p.max_user_clip_distances = 8; + return p; +} + +} // namespace + +int main(int argc, char** argv) { + if (argc != 6) { + std::cerr << "usage: eden_xlate \n"; + return 2; + } + const std::string raw = argv[2], out = argv[3]; + const int shard = std::stoi(argv[4]), shards = std::stoi(argv[5]); + std::ifstream table(argv[1]); + std::string line; + std::getline(table, line); // header + const Shader::Profile profile = desktop_profile(); + Shader::HostTranslateInfo host{}; + host.support_float64 = host.support_float16 = host.support_int64 = true; + host.support_snorm_render_buffer = host.support_viewport_index_layer = host.support_conditional_barrier = true; + host.min_ssbo_alignment = 16; + host.ApplyDescriptorLimitPolicy(); + for (int row = 0; std::getline(table, line); ++row) { + if (row % shards != shard) continue; + std::istringstream fields(line); + std::string key, stage; + std::getline(fields, key, '\t'); + std::getline(fields, stage, '\t'); + const auto t0 = std::chrono::steady_clock::now(); + std::string result = "ok", message; + size_t bytes = 0; + try { + const auto ctl = read_file(raw + "/" + key + ".ctl"); + const auto bin = read_file(raw + "/" + key + ".bin"); + const bool compute = stage == "cs"; + NvnEnvironment env(ctl, bin, compute); + Shader::ObjectPool inst; + Shader::ObjectPool blocks; + Shader::ObjectPool flow; + Shader::Maxwell::Flow::CFG cfg(env, flow, env.StartAddress() + (compute ? 0 : 0x50), compute); + auto program = Shader::Maxwell::TranslateProgram(inst, blocks, env, cfg, host); + Shader::RuntimeInfo runtime{}; + runtime.input_topology = Shader::InputTopology::Triangles; + runtime.generic_input_types.fill(Shader::AttributeType::Float); + runtime.color_output_types.fill(Shader::AttributeType::Float); + runtime.previous_stage_stores.mask.set(); + if (!compute) Shader::Maxwell::ConvertLegacyToGeneric(program, runtime); + Shader::Backend::Bindings bindings{}; + const auto spv = Shader::Backend::SPIRV::EmitSPIRV(profile, runtime, program, bindings); + std::ofstream o(out + "/" + key + ".spv", std::ios::binary); + o.write(reinterpret_cast(spv.data()), static_cast(spv.size() * 4)); + bytes = spv.size() * 4; + } catch (const std::exception& e) { + result = "fail"; + message = e.what(); + } catch (...) { + result = "fail"; + message = "unknown exception"; + } + const double s = std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); + for (char& c : message) if (c == '\t' || c == '\n') c = ' '; + std::printf("%s\t%s\t%.4f\t%zu\t%.200s\n", key.c_str(), result.c_str(), s, bytes, message.c_str()); + std::fflush(stdout); + } + return 0; +} diff --git a/tools/eden_xlate/shim/common/assert.h b/tools/eden_xlate/shim/common/assert.h new file mode 100644 index 0000000..7f954f3 --- /dev/null +++ b/tools/eden_xlate/shim/common/assert.h @@ -0,0 +1,16 @@ +// Stand-in for Eden's common/assert.h for the offline shader tool: a failed assertion throws, so one bad shader +// fails its own translation instead of ending the batch. +#pragma once +#include +#include +#include "common/logging.h" + +#define ASSERT(cond) do { if (!(cond)) throw std::logic_error("assertion failed: " #cond); } while (0) +#define ASSERT_MSG(cond, ...) do { if (!(cond)) throw std::logic_error(std::string("assertion failed: " #__VA_ARGS__)); } while (0) +#define DEBUG_ASSERT(cond) do {} while (0) +#define DEBUG_ASSERT_MSG(cond, ...) do {} while (0) +#define UNREACHABLE() throw std::logic_error("unreachable") +#define UNREACHABLE_MSG(...) throw std::logic_error(std::string("unreachable: " #__VA_ARGS__)) +#define UNIMPLEMENTED() throw std::logic_error("unimplemented") +#define UNIMPLEMENTED_MSG(...) throw std::logic_error(std::string("unimplemented: " #__VA_ARGS__)) +#define ASSERT_OR_EXECUTE(cond, body) do { if (!(cond)) { body } } while (0) diff --git a/tools/eden_xlate/shim/common/logging.h b/tools/eden_xlate/shim/common/logging.h new file mode 100644 index 0000000..16d1aff --- /dev/null +++ b/tools/eden_xlate/shim/common/logging.h @@ -0,0 +1,19 @@ +// Stand-in for Eden's common/logging.h for the offline shader tool. The arguments are not formatted: the message +// is printed as written in the source, and only when EDEN_XLATE_LOG is set; debug and trace output is dropped. +#pragma once +#include +#include +#include +#include + +namespace EdenXlate { +inline bool LogOn() { static const bool on = std::getenv("EDEN_XLATE_LOG") != nullptr; return on; } +} +#define EDEN_XLATE_LOG_AT(level, cls, ...) \ + do { if (::EdenXlate::LogOn()) std::fprintf(stderr, "[%s] %s\n", level, #__VA_ARGS__); } while (0) +#define LOG_TRACE(cls, ...) do {} while (0) +#define LOG_DEBUG(cls, ...) do {} while (0) +#define LOG_INFO(cls, ...) EDEN_XLATE_LOG_AT("info", cls, __VA_ARGS__) +#define LOG_WARNING(cls, ...) EDEN_XLATE_LOG_AT("warning", cls, __VA_ARGS__) +#define LOG_ERROR(cls, ...) EDEN_XLATE_LOG_AT("error", cls, __VA_ARGS__) +#define LOG_CRITICAL(cls, ...) EDEN_XLATE_LOG_AT("critical", cls, __VA_ARGS__) diff --git a/tools/eden_xlate/shim/common/settings.h b/tools/eden_xlate/shim/common/settings.h new file mode 100644 index 0000000..94e3545 --- /dev/null +++ b/tools/eden_xlate/shim/common/settings.h @@ -0,0 +1,28 @@ +// Stand-in for Eden's common/settings.h for the offline shader tool: only the values the shader recompiler reads, +// fixed to Eden's defaults (no resolution scaling, loop safety checks on, no bloom fix, no renderer debug). +#pragma once +#include "common/common_types.h" + +namespace Settings { +template +struct Setting { + T value{}; + T GetValue() const { return value; } + operator T() const { return value; } +}; +struct ResolutionScalingInfo { + u32 up_scale{1}; + u32 down_shift{0}; + f32 up_factor{1.0f}; + f32 down_factor{1.0f}; + bool active{}; + bool downscale{}; +}; +struct Values { + ResolutionScalingInfo resolution_info{}; + Setting fix_bloom_effects{}; + Setting renderer_debug{}; + Setting disable_shader_loop_safety_checks{}; +}; +inline Values values{}; +} // namespace Settings diff --git a/tools/eden_xlate/shim/eden_fmt_enums.h b/tools/eden_xlate/shim/eden_fmt_enums.h new file mode 100644 index 0000000..49ccaa4 --- /dev/null +++ b/tools/eden_xlate/shim/eden_fmt_enums.h @@ -0,0 +1,12 @@ +// Eden formats its enums with fmt (exception messages, logs) as the older fmt it pins did implicitly; current fmt +// (MSYS2's) no longer converts enums. Format any enum without its own formatter as its underlying value. +#pragma once +#include +#include + +template +struct fmt::formatter>> : fmt::formatter, char> { + auto format(E e, fmt::format_context& ctx) const { + return fmt::formatter, char>::format(static_cast>(e), ctx); + } +}; diff --git a/tools/export_shaders.py b/tools/export_shaders.py new file mode 100644 index 0000000..a3d037c --- /dev/null +++ b/tools/export_shaders.py @@ -0,0 +1,49 @@ +"""Export every shader of the game (tools/shaders/inventory.py) for offline translation. + +usage: python tools/export_shaders.py + romfs: a folder of plain files (the player's dump) or the development extraction (romfs.bin + index.json); + exefs: the NSO modules (main, subsdk0, sdk): they embed engine shaders too (35 in 1.4.3, among them the title + screen's full-screen passes). +Writes /raw/.ctl (the 0x880-byte NVN control section) and .bin (the code as the game hands it over), +and /jobs.jsonl: one totk_xlate --serve request per shader for the default bind state (as MacNativeGames' +prepare_shader_seed: graphics code without the 0x30-byte NVN header, c1 = the code's tail from the control's c1 offset). +Game-derived data: stays out of git. +""" +import json +import pathlib +import struct +import sys + +ROOT = pathlib.Path(__file__).resolve().parent.parent +sys.path.insert(0, str(ROOT / 'tools')) +sys.path.insert(0, str(ROOT / 'tools/lifter')) +from nso import Nso # noqa: E402 +from romfs import RomFS # noqa: E402 +from shaders.inventory import inventory # noqa: E402 + + +def main(): + romfs, out, exefs = pathlib.Path(sys.argv[1]), pathlib.Path(sys.argv[2]), pathlib.Path(sys.argv[3]) + modules = {name: bytes(Nso(exefs / name).image) for name in ('main', 'subsdk0', 'sdk')} + raw = out / 'raw' + code_dir = out / 'ryujinx-code' + raw.mkdir(parents=True, exist_ok=True) + code_dir.mkdir(parents=True, exist_ok=True) + shaders, startup = inventory(RomFS(romfs), modules) + with (out / 'jobs.jsonl').open('w') as jobs, (out / 'shaders.tsv').open('w') as table: + table.write('key\tstage\tstartup\tcode_bytes\tsource\n') + for key, (stage, control, code, source) in sorted(shaders.items()): + (raw / f'{key}.ctl').write_bytes(control) + (raw / f'{key}.bin').write_bytes(code) + c1, length = struct.unpack_from(' +Bases must match src/runtime/rt.h (MAIN_BASE, SUBSDK0_BASE, SDK_BASE). +""" +import pathlib +import subprocess +import sys +import time + +ROOT = pathlib.Path(__file__).resolve().parent.parent +MODULES = {'main': 0x0800000000, 'subsdk0': 0x0804000000, 'sdk': 0x0805000000} + + +def main(): + exefs, out = pathlib.Path(sys.argv[1]), pathlib.Path(sys.argv[2]) + t0 = time.time() + procs = [] + for name, base in MODULES.items(): + (out / name).mkdir(parents=True, exist_ok=True) + log = open(out / f'{name}.log', 'w') + procs.append((name, log, subprocess.Popen( + [sys.executable, str(ROOT / 'tools/lifter/gen.py'), name, str(exefs / name), hex(base), str(out / name), + '--simd', 'c'], stdout=log, stderr=subprocess.STDOUT))) + failed = False + for name, log, p in procs: + code = p.wait() + log.close() + summary = [l for l in (out / f'{name}.log').read_text().splitlines() if 'progress' not in l] + print(f'{name}: exit {code}; ' + ' | '.join(summary[-2:])) + failed |= code != 0 + print(f'{time.time() - t0:.0f}s') + sys.exit(1 if failed else 0) + + +if __name__ == '__main__': + main() diff --git a/tools/gen_nvn_thunks.py b/tools/gen_nvn_thunks.py new file mode 100644 index 0000000..c834ae0 --- /dev/null +++ b/tools/gen_nvn_thunks.py @@ -0,0 +1,135 @@ +"""Generate guest-register thunks for the NVN implementations (src/gfx/nvn/nvn_thunks.gen.c). + +The game calls NVN through function pointers it gets from nvnBootstrapLoader / nvnDeviceGetProcAddress. Each pointer is +a fake guest address of a host function on the guest registers (src/runtime: hle_register). This script reads the +implementations' own declarations (NVN / NVN_NOPROTO / NVN_EXTRA in src/gfx/nvn/*.c, MacNativeGames' macros) and the +types of include/nvn.h, and writes for each one a thunk that unpacks AAPCS64 arguments: integers, pointers, enums and +handles from x0..x7, float/double from v0..v7; the result goes to x0 (or v0 for float/double). +A structure passed or returned by value stops the generator: AAPCS64 passes those differently. + +usage: python tools/gen_nvn_thunks.py +""" +import pathlib +import re +import sys + +ROOT = pathlib.Path(__file__).resolve().parent.parent +NVN = ROOT / 'src/gfx/nvn' +DECL = re.compile(r'^(NVN|NVN_NOPROTO|NVN_EXTRA)\((\w+),\s*([^,]+?),\s*(.*?)\)\s*\{', re.M | re.S) + + +SMALL = {} # struct type -> True when it is a single integer field of at most 8 bytes (AAPCS64: one x register) + + +def struct_types(): + text = (NVN / 'include/nvn.h').read_text(encoding='utf-8', errors='replace') + names = set() + for body, name in re.findall(r'typedef struct\s*\w*\s*\{([^}]*)\}\s*(\w+)\s*;', text): + names.add(name) + fields = [f.strip() for f in body.split(';') if f.strip() and not f.strip().startswith('//')] + if len(fields) == 1 and re.fullmatch(r'(u?int(8|16|32|64)_t)\s+\w+', fields[0]): + SMALL[name] = True + return names + + +def split_params(s): + out, depth, cur = [], 0, '' + for ch in s: + if ch == '(': + depth += 1 + elif ch == ')': + depth -= 1 + if ch == ',' and depth == 0: + out.append(cur.strip()); cur = '' + else: + cur += ch + if cur.strip(): + out.append(cur.strip()) + return [p for p in out if p and p != 'void'] + + +def param_type(p): + m = re.match(r'^(.*?)(\w+)$', p.strip()) + if not m or not m.group(1).strip(): + sys.exit(f'cannot parse parameter {p!r}') + return m.group(1).strip() + + +def kind(t, structs): + base = t.replace('const', '').strip() + if '*' in base: + return 'ptr' + if base in ('float', 'double'): + return base + if base in SMALL: + return 'small' + if base in structs: + return 'struct' + return 'int' + + +def main(): + structs = struct_types() + decls = {} + for src in sorted(NVN.glob('*.c')): + text = src.read_text(encoding='utf-8', errors='replace') + for m in DECL.finditer(text): + macro, name, ret, params = m.group(1), m.group(2), m.group(3).strip(), split_params(m.group(4)) + if name in decls and decls[name][1:] != (ret, params): + sys.exit(f'{name}: two different declarations') + decls[name] = (macro, ret, params) + out = ['// Generated by tools/gen_nvn_thunks.py from the NVN implementations\' declarations. Do not edit.', + '#include ', '#include "include/nvn.h"', '#include "../../runtime/cpu.h"', + 'void *nvn_impl_fn(const char *name, int extra);', ''] + table = [] + for name, (macro, ret, params) in sorted(decls.items()): + rk = kind(ret, structs) + if rk == 'struct': + sys.exit(f'{name}: returns a structure by value') + types = [param_type(p) for p in params] + nx = nv = 0 + args = [] + for t in types: + k = kind(t, structs) + if k == 'struct': + sys.exit(f'{name}: structure parameter by value ({t})') + if k in ('float', 'double'): + if nv >= 8: sys.exit(f'{name}: more than 8 FP arguments') + args.append(f'({{ {k} v_; memcpy(&v_, &c->v[{nv}], sizeof v_); v_; }})') + nv += 1 + else: + if nx >= 8: sys.exit(f'{name}: more than 8 integer arguments (stack arguments are not handled)') + if k == 'small': + args.append(f'({{ {t} v_; memcpy(&v_, &c->x[{nx}], sizeof v_); v_; }})') + else: + args.append(f'(void *)(uintptr_t)c->x[{nx}]' if k == 'ptr' else f'({t})c->x[{nx}]') + nx += 1 + # pointers are passed as void * (the same in both ABIs): extension types need not be visible here + ptypes = ['void *' if kind(t, structs) == 'ptr' else t for t in types] + rtype = 'void *' if rk == 'ptr' and ret not in ('PFNNVNGENERICFUNCPTRPROC',) else ret + fnty = f'{rtype} (*)({", ".join(ptypes) or "void"})' + call = f'(({fnty})f)({", ".join(args)})' + out.append(f'static void T_{name}(Cpu *c) {{') + out.append(f' static void *f; if (!f) f = nvn_impl_fn("{name}", {int(macro == "NVN_EXTRA")});') + if ret == 'void': + out.append(f' {call};') + elif rk in ('float', 'double'): + out.append(f' {ret} r_ = {call}; memset(&c->v[0], 0, sizeof c->v[0]); memcpy(&c->v[0], &r_, sizeof r_);') + elif rk == 'ptr': + out.append(f' c->x[0] = (uint64_t)(uintptr_t){call};') + elif rk == 'small': # a single-integer-field structure: returned in x0 (AAPCS64) / RAX (Win64) + out.append(f' {ret} r_ = {call}; uint64_t v_ = 0; memcpy(&v_, &r_, sizeof r_); c->x[0] = v_;') + else: + out.append(f' c->x[0] = (uint64_t){call};') + out.append('}') + table.append(f' {{"{name}", T_{name}}},') + out.append('const struct { const char *name; void (*fn)(Cpu *); } nvn_thunks[] = {') + out += table + out.append('};') + out.append(f'const unsigned nvn_thunk_count = {len(table)};') + (NVN / 'nvn_thunks.gen.c').write_text('\n'.join(out) + '\n', encoding='utf-8') + print(f'{len(table)} thunks -> {NVN / "nvn_thunks.gen.c"}') + + +if __name__ == '__main__': + main() diff --git a/tools/lifter/LICENSE-LASRecomp b/tools/lifter/LICENSE-LASRecomp new file mode 100644 index 0000000..b0977fd --- /dev/null +++ b/tools/lifter/LICENSE-LASRecomp @@ -0,0 +1,24 @@ +MIT License + +Copyright (c) 2026 Peiikai + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +This license covers LAS Recomp's own code. It does not cover The Legend of Zelda: Link's Awakening or any other +Nintendo property, nor the third-party components in third_party/, which keep their own licenses. diff --git a/tools/lifter/README.md b/tools/lifter/README.md new file mode 100644 index 0000000..ff5e801 --- /dev/null +++ b/tools/lifter/README.md @@ -0,0 +1,8 @@ +# ARM64 -> C lifter + +Copied from [LASRecomp](https://github.com/peiikai/LASRecomp) `tools/` at commit `da35b83a9611d046a9a681aae668d7085954d37f` +(MIT, `LICENSE-LASRecomp`): `gen.py`, `lifter.py`, `simd_c.py`, `analyze.py`, `nso.py`. + +TOTK changes: +- `nso.py`: DT_RELR packed relative relocations and DT_GNU_HASH symbol counts (TOTK 1.4.3 modules). +- `gen.py`: Link's Awakening randomizer variants removed; `patches.py` is TOTK's own (empty for now). diff --git a/tools/lifter/analyze.py b/tools/lifter/analyze.py new file mode 100644 index 0000000..f9a73d0 --- /dev/null +++ b/tools/lifter/analyze.py @@ -0,0 +1,105 @@ +"""Function discovery and instruction census for an NSO module.""" +import struct +import sys +from collections import Counter + +import capstone +from nso import Nso, R_AARCH64_RELATIVE + + +def read_encoded(img, off, enc, pc_base): + fmt = enc & 0x0F + app = enc & 0x70 + if fmt == 0x0B: # sdata4 + v, size = struct.unpack_from('= len(img): + return [] + ver, ptr_enc, cnt_enc, tbl_enc = img[hdr:hdr + 4] + off = hdr + 4 + _, sz = read_encoded(img, off, ptr_enc, hdr) + off += sz + count, sz = read_encoded(img, off, cnt_enc, hdr) + off += sz + starts = [] + for _ in range(count): + loc, sz = read_encoded(img, off, tbl_enc, hdr) + off += sz + _, sz = read_encoded(img, off, tbl_enc, hdr) + off += sz + starts.append(loc) + return starts + + +def discover(n): + text_lo, text_hi = n.text[0], n.text[0] + n.text[1] + in_text = lambda a: text_lo <= a < text_hi and a % 4 == 0 + funcs = {} + for a in eh_frame_functions(n): + if in_text(a): + funcs[a] = 'eh' + for name, info, shndx, value, size in n.syms: + if shndx and in_text(value) and (info & 0xF) == 2: + funcs.setdefault(value, 'sym') + # Code pointers stored in data (vtables, callbacks) + ptr_targets = set() + for off, typ, sym, add in n.relocs: + if typ == R_AARCH64_RELATIVE and in_text(add): + ptr_targets.add(add) + for a in ptr_targets: + funcs.setdefault(a, 'ptr') + # BL targets + code = bytes(n.image[text_lo:text_hi]) + words = struct.unpack(f'<{len(code) // 4}I', code) + for i, w in enumerate(words): + if (w & 0xFC000000) == 0x94000000: + imm = w & 0x03FFFFFF + if imm & 0x02000000: + imm -= 0x04000000 + t = text_lo + i * 4 + imm * 4 + if in_text(t): + funcs.setdefault(t, 'bl') + return funcs, words + + +def census(n, words): + md = capstone.Cs(capstone.CS_ARCH_ARM64, capstone.CS_MODE_ARM) + c = Counter() + bad = 0 + code = struct.pack(f'<{len(words)}I', *words) + off = 0 + while off < len(code): + got = False + for ins in md.disasm_lite(code[off:], n.text[0] + off): + c[ins[2]] += 1 + off += 4 + got = True + if off < len(code): # undecodable word: skip it + bad += 1 + off += 4 + return c, bad + + +if __name__ == '__main__': + n = Nso(sys.argv[1]) + funcs, words = discover(n) + print(f'functions: {len(funcs)} by source: {dict(Counter(funcs.values()))}') + if '--census' in sys.argv: + c, bad = census(n, words) + print(f'instructions: {sum(c.values())} undecodable words: {bad} distinct mnemonics: {len(c)}') + print(' '.join(f'{m}:{k}' for m, k in c.most_common())) diff --git a/tools/lifter/gen.py b/tools/lifter/gen.py new file mode 100644 index 0000000..705b0fc --- /dev/null +++ b/tools/lifter/gen.py @@ -0,0 +1,757 @@ +"""Generate C for a whole NSO module. + +From LASRecomp tools/gen.py @da35b83 (MIT, LICENSE-LASRecomp); TOTK changes: no randomizer variants, +TOTK patch table (patches.py). + +Usage: gen.py [--hle hle_symbols.txt] [--chunk N] [--simd asm|c] + +--simd: FP/SIMD/CRC32 as AArch64 inline asm (default on arm64 hosts) or portable C (default elsewhere). +""" +import argparse +import bisect +import os +import platform +import re +import struct +import sys +import time +from collections import Counter + +import unicorn as uc +from unicorn import arm64_const as A + +from analyze import eh_frame_functions +from lifter import Insn, Lifter, Unsupported, bits, emit_asm_batch, hx, sext +from nso import Nso, R_AARCH64_RELATIVE + +SCRATCH_MEM = 0x7300000000 + + +def sext26(w): + v = w & 0x03FFFFFF + return v - 0x04000000 if v & 0x02000000 else v + + +def fname(a): + return f'f_{a:x}' + + +def sanitize(sym): + return 'hle_' + re.sub(r'[^A-Za-z0-9_]', '_', sym) + + +class Module: + def __init__(self, name, path, base, simd='asm'): + self.name = name + self.nso = Nso(path) + self.base = base + n = self.nso + self.lo = base + n.text[0] + self.hi = base + n.text[0] + n.text[1] + self.words = struct.unpack(f'<{n.text[1] // 4}I', bytes(n.image[n.text[0]:n.text[0] + n.text[1]])) + self.lifter = Lifter(base, simd) + self.cache = {} + self.stats = Counter() + self.mu = None + + def in_text(self, a): + return self.lo <= a < self.hi and a % 4 == 0 + + def word(self, a): + return self.words[(a - self.lo) // 4] + + def lift(self, a): + ins = self.cache.get(a) + if ins is None: + try: + ins = self.lifter.lift(self.word(a), a) + except Unsupported as e: + self.stats['unsupported'] += 1 + ins = Insn('trap', f'rt_udf(c, {hx(a)}); /* {e} */') + self.cache[a] = ins + return ins + + def boundaries(self): + n, b = self.nso, self.base + starts = set() + for a in eh_frame_functions(n): + starts.add(b + a) + for name, info, shndx, value, size in n.syms: + if shndx and (info & 0xF) == 2: + starts.add(b + value) + for off, typ, sym, add in n.relocs: + if typ == R_AARCH64_RELATIVE: + starts.add(b + add) + starts = sorted(a for a in starts if self.in_text(a)) + return starts + + # ------------------------------------------------------------ jump tables + + def emu(self): + if self.mu is None: + mu = uc.Uc(uc.UC_ARCH_ARM64, uc.UC_MODE_ARM) + n = self.nso + mu.mem_map(self.base, len(n.image)) + mu.mem_write(self.base, bytes(n.image)) + mu.mem_map(SCRATCH_MEM, 0x100000) + mu.reg_write(A.UC_ARM64_REG_CPACR_EL1, 0x300000) + + # [TOTK] Loads unrelated to the table between the bounds check and the br (e.g. `ldr q29, [x8, x13]` + # with both registers holding the scratch address) used to fault and drop the whole table (798 + # functions in main fell back to an all-instruction switch). Unmapped pages now read as zeros. + def unmapped(uc_, access, address, size, value, data): + page = address & ~0xFFF + if page >= 1 << 48: + return False + try: + uc_.mem_map(page, 0x1000) + except uc.UcError: + return False + return True + mu.hook_add(uc.UC_HOOK_MEM_READ_UNMAPPED | uc.UC_HOOK_MEM_WRITE_UNMAPPED, unmapped) + self.mu = mu + return self.mu + + def jump_table(self, br_pc, entry, end): + """Resolve `br xN` targets from a bounds-checked jump table, or None.""" + r = self._jump_table_fallthrough(br_pc, entry, end) + if r is None: + r = self._jump_table_taken(br_pc, entry, end) + # [TOTK] The emulation sets the register the bounds check compares; when the load indexes the table with + # another register holding the same value (main 0x80209aa08: `cmp w22, #0xb` ... `ldrh w10, [x8, x28, lsl #1]`) + # every index gives the same target (a table read at a fixed scratch address). Such a single-target result is + # replaced by the entries read from the image. (Adding the scanned entries to every table instead put + # trailing data that looks like entries into 1619 of main's 2367 tables.) + if r is not None and len(set(r)) > 1: + return r + s = self._jump_table_scan(br_pc, entry, end) + if r is not None and s is not None: + self.stats['jumptables_single_target_rescanned'] += 1 + return sorted(set(r) | set(s)) + return r if r is not None else s + + def _jump_table_scan(self, br_pc, entry, end): + """[TOTK] A table whose bounds check the compiler hoisted far from the br (main: functions with thousands of + cases): `adr xB, base; adrp/add xT, table; ldr{b,h,sw} wO, [xT, xI, ...]; add xD, xB, wO, {lsl,sxt*} #k; br xD`. + The entries are read from the image until one points outside the function or is misaligned; trailing bytes + that happen to look like entries only add labels (an index the table never had is never taken).""" + import capstone + if not hasattr(self, '_cs'): + self._cs = capstone.Cs(capstone.CS_ARCH_ARM64, capstone.CS_MODE_ARM) + self._cs.detail = True + cs = self._cs + window = [] + a = br_pc + while a >= entry and br_pc - a <= 4 * 24: + window.append(a) + a -= 4 + insns = {} + for x in window: + got = list(cs.disasm(struct.pack(' #k + add = load = None + for x in window[1:]: + i = insns[x] + if i.mnemonic == 'add' and len(i.operands) == 3 and i.reg_name(i.operands[0].reg) == f'x{rn}': + add = i + break + if self.lift(x).kind not in ('op', 'asm'): + return None + if add is None or add.operands[2].type != capstone.arm64.ARM64_OP_REG: + return None + base_reg = add.reg_name(add.operands[1].reg) + off_reg = add.reg_name(add.operands[2].reg) + shift = add.operands[2].shift.value if add.operands[2].shift.type else 0 + values = {} # register -> constant from adr / adrp+add before the add + + def scan_back(reg, before): + for x in window: + if x >= before: + continue + i = insns[x] + if not i.operands or i.operands[0].type != capstone.arm64.ARM64_OP_REG: + continue + if i.reg_name(i.operands[0].reg) != reg: + continue + return x, i + return None, None + x, i = scan_back(base_reg, add.address) + if i is None or i.mnemonic != 'adr': + return None + base = i.operands[1].imm + # the load of the offset + x, i = scan_back(off_reg.replace('x', 'w'), add.address) + if i is None: + x, i = scan_back(off_reg.replace('w', 'x'), add.address) + if i is None or i.mnemonic not in ('ldrb', 'ldrh', 'ldrsb', 'ldrsh', 'ldrsw', 'ldr'): + return None + mem = i.operands[1] + if mem.type != capstone.arm64.ARM64_OP_MEM or not mem.mem.index: + return None + size = {'ldrb': 1, 'ldrsb': 1, 'ldrh': 2, 'ldrsh': 2, 'ldrsw': 4, 'ldr': 4}[i.mnemonic] + signed = i.mnemonic.startswith('ldrs') + table_reg = i.reg_name(mem.mem.base) + # table address: adrp + add on table_reg, near the br or (hoisted out of a loop) anywhere earlier in the + # function: the closest earlier write of table_reg must be that add, and the write before it the adrp. + def writer_before(reg, before): + for y in range(before - 4, entry - 4, -4): + got = list(cs.disasm(struct.pack(' hi or at < lo: + break + raw = int.from_bytes(self.nso.image[at - self.base:at - self.base + size], 'little', signed=signed) + t = base + (raw << shift) + if not (entry <= t < end) or t % 4: + break + targets.add(t) + if len(targets) < 2: + return None + self.stats['jumptables_scanned'] += 1 + return sorted(targets) + + def _index_regs(self, start, at, idx): + """The index register plus registers holding the same value in start..at: sources it was + copied/masked from, and copies of those (the table lookup sometimes recomputes the index + from another copy after the bounds check).""" + def copy_src(w): + if (w & 0x7F800000) == 0x12000000: # AND (imm) + return bits(w, 5, 5) + if (w & 0x7FE0FFE0) == 0x2A0003E0: # MOV (ORR with zr) + return bits(w, 16, 5) + if (w & 0x7F800000) in (0x53000000, 0x13000000): # UBFM/SBFM (uxtb, uxth, sxtw...) + return bits(w, 5, 5) + return None + regs = {idx} + changed = True + while changed: + changed = False + for a in range(start, at, 4): + w = self.word(a) + rd, src = w & 31, copy_src(w) + if src is None or src >= 31 or rd >= 31: + continue + for x, y in ((rd, src), (src, rd)): + if x in regs and y not in regs: + regs.add(y) + changed = True + return sorted(regs) + + def _set_index(self, mu, regs, i): + for r in regs: + mu.reg_write(A.UC_ARM64_REG_X0 + r if r < 29 else A.UC_ARM64_REG_X29, i) + + def _emulate_table(self, br_pc, entry, end, start, idx, count, set_at): + """Run start..br for each index value (set at address set_at), collect targets.""" + mu = self.emu() + rn = bits(self.word(br_pc), 5, 5) + + def step_to(frm, to): + pc = frm + for _ in range(64): + if pc == to: + return True + mu.emu_start(pc, 0, count=1) + pc = mu.reg_read(A.UC_ARM64_REG_PC) + return False + + targets = [] + try: + for i in range(count): + for r in range(29): + mu.reg_write(A.UC_ARM64_REG_X0 + r, SCRATCH_MEM + 0x80000) + mu.reg_write(A.UC_ARM64_REG_SP, SCRATCH_MEM + 0x80000) + a2 = entry + while a2 < start: + w2 = self.word(a2) + if (w2 & 0x1F000000) == 0x10000000 or (w2 & 0xFF800000) == 0x91000000: + mu.emu_start(a2, 0, count=1) + a2 += 4 + if set_at != start and not step_to(start, set_at): + return None + self._set_index(mu, [idx], i) + if not step_to(set_at, br_pc): + return None + t = mu.reg_read(A.UC_ARM64_REG_X0 + rn if rn < 29 else + (A.UC_ARM64_REG_X29 if rn == 29 else A.UC_ARM64_REG_X30)) + if not (entry <= t < end) or t % 4: + return None + targets.append(t) + except uc.UcError: + return None + return sorted(set(targets)) + + def _jump_table_taken(self, br_pc, entry, end): + """Form: cmp idx, #N ; b.ls/b.lo table ... table:
; br""" + # start of the block holding the br + s = br_pc + while s - 4 >= entry and br_pc - (s - 4) <= 4 * 12 and self.lift(s - 4).kind in ('op', 'asm'): + s -= 4 + for g in range(entry, end, 4): + w = self.word(g) + if (w & 0xFF000010) != 0x54000000 or bits(w, 0, 4) not in (9, 3): + continue + t = g + sext(bits(w, 5, 19), 19) * 4 + if t != s: + continue + cw = self.word(g - 4) if g - 4 >= entry else 0 + if (cw & 0x7F80001F) != 0x7100001F or bits(cw, 22, 1): + continue + limit = bits(cw, 10, 12) + count = limit + 1 if bits(w, 0, 4) == 9 else limit + if count <= 0 or count > 4096: + continue + return self._emulate_table(br_pc, entry, end, s, bits(cw, 5, 5), count, s) + return None + + def _jump_table_fallthrough(self, br_pc, entry, end): + rn = bits(self.word(br_pc), 5, 5) + # find the guarding b.hi / b.hs and its cmp, scanning backwards + a = br_pc - 4 + guard = cmp = None + while a >= entry and br_pc - a <= 4 * 16: + w = self.word(a) + if (w & 0xFF000010) == 0x54000000 and bits(w, 0, 4) in (8, 2): + guard = a + break + if self.lift(a).kind in ('b', 'bl', 'blr', 'br', 'ret', 'trap'): + return None + a -= 4 + if guard is None: + return None + a = guard - 4 + while a >= entry and guard - a <= 4 * 8: + w = self.word(a) + # SUBS (imm) with Rd = zr, i.e. CMP #imm + if (w & 0x7F80001F) == 0x7100001F: + cmp = a + break + if self.lift(a).kind != 'op': + return None + a -= 4 + if cmp is None: + return None + w = self.word(cmp) + if bits(w, 22, 1): + return None + limit = bits(w, 10, 12) + count = limit + 1 if bits(self.word(guard), 0, 4) == 8 else limit + if count <= 0 or count > 4096: + return None + idx = bits(w, 5, 5) + # window start: back from cmp to the first control-flow instruction + start = cmp + while start - 4 >= entry and cmp - (start - 4) <= 4 * 12 and self.lift(start - 4).kind in ('op', 'asm'): + start -= 4 + idx_regs = self._index_regs(start, cmp, idx) + mu = self.emu() + targets = [] + + def step_to(frm, to): + # single-step: Unicorn's `until` is unreliable mid translation-block + pc = frm + for _ in range(64): + if pc == to: + return True + mu.emu_start(pc, 0, count=1) + pc = mu.reg_read(A.UC_ARM64_REG_PC) + return False + + try: + for i in range(count): + for r in range(29): + mu.reg_write(A.UC_ARM64_REG_X0 + r, SCRATCH_MEM + 0x80000) + mu.reg_write(A.UC_ARM64_REG_SP, SCRATCH_MEM + 0x80000) + # replay address materialization (ADRP/ADR/ADD-imm) from the function entry, so table + # bases computed before loops or branches are known + a2 = entry + while a2 < start: + w2 = self.word(a2) + if (w2 & 0x1F000000) == 0x10000000 or (w2 & 0xFF800000) == 0x91000000: + mu.emu_start(a2, 0, count=1) + a2 += 4 + if not step_to(start, cmp): + return None + self._set_index(mu, idx_regs, i) + if not step_to(cmp, br_pc): + return None + t = mu.reg_read(A.UC_ARM64_REG_X0 + rn if rn < 29 else + (A.UC_ARM64_REG_X29 if rn == 29 else A.UC_ARM64_REG_X30)) + if not (entry <= t < end) or t % 4: + return None + targets.append(t) + except uc.UcError: + return None + return sorted(set(targets)) + + # ------------------------------------------------------------ functions + + def computed_target(self, br_pc, entry): + """True if the br register comes from address arithmetic (a table) rather than a load.""" + rn = bits(self.word(br_pc), 5, 5) + a = br_pc - 4 + while a >= entry and br_pc - a <= 4 * 16: + w = self.word(a) + if self.lift(a).kind not in ('op', 'asm'): + return False + if (w & 31) == rn: + if (w & 0x1F000000) == 0x0B000000: # ADD/SUB (shifted or extended register) + return True + if (w & 0x9F000000) == 0x10000000: # ADR + return True + return False + a -= 4 + return False + + def scan(self, entry, end): + """Collect labels and external entries referenced by a function.""" + labels, ext, tables = {entry}, set(), {} + terminated = False + for a in range(entry, end, 4): + ins = self.lift(a) + if ins.kind in ('b', 'bcond'): + t = ins.target + if entry <= t < end: + labels.add(t) + elif self.in_text(t): + ext.add(t) + elif ins.kind == 'bl': + if self.in_text(ins.target): + ext.add(ins.target) + elif ins.kind == 'br': + jt = self.jump_table(a, entry, end) + if jt: + tables[a] = jt + labels.update(jt) + self.stats['jumptables'] += 1 + elif self.computed_target(a, entry): + # unresolved table jump: allow jumping to any instruction of the function + tables[a] = 'all' + self.stats['br_all_labels'] += 1 + else: + self.stats['br_dispatch'] += 1 + terminated = ins.kind in ('b', 'br', 'ret', 'trap') + if not terminated and end < self.hi: + ext.add(end) + return labels, ext, tables + + def render_single(self, ins, a, jump, refs): + """C for one lifted instruction (no batching); used for patch variants.""" + k = ins.kind + if k == 'asm': + return [emit_asm_batch([ins.asm])] + if k in ('op', 'trap'): + return [ins.code] + if k == 'b': + return [jump(ins.target)] + if k == 'bcond': + return [f'if ({ins.cond}) {{ {jump(ins.target)} }}'] + if k == 'bl': + t = ins.target + if self.in_text(t): + refs.add(t) + return [f'c->x[30] = {hx(a + 4)}; {fname(t)}(c);'] + return [f'c->x[30] = {hx(a + 4)}; c->pc = {hx(t)}; rt_dispatch(c);'] + if k == 'blr': + return [f'c->pc = {ins.target}; c->x[30] = {hx(a + 4)}; rt_dispatch(c);'] + if k == 'br': + return [f'c->pc = {ins.target}; __attribute__((musttail)) return rt_dispatch(c);'] + if k == 'ret': + return ['return;'] + raise Unsupported(k) + + def emit_function(self, entry, end, labels, tables, hle): + out = [f'void {fname(entry)}(Cpu *c) {{'] + if entry in hle: + out.append(f' {sanitize(hle[entry])}(c);') + out.append('}') + return out + out.append(f' FN_ENTER({hx(entry)});') + batch = [] + refs = set() + + def flush(): + if batch: + out.append(' ' + emit_asm_batch(batch)) + batch.clear() + + def jump(t): + if entry <= t < end: + return f'goto L_{t:x};' + if not self.in_text(t): + return f'c->pc = {hx(t)}; __attribute__((musttail)) return rt_dispatch(c);' + refs.add(t) + return f'__attribute__((musttail)) return {fname(t)}(c);' + + # [TOTK] Only functions with an unresolved computed branch ('all') get the all-instruction fallback. LASRecomp + # gave it to every function with a jump table: thousands of labels made clang take hours on single functions + # (main_127.c: 15 880 labels, >2.5 h at -O2) and kept it from optimizing them. A resolved table's targets are + # complete (each index was emulated); an index outside them reaches rt_dispatch and fails loudly. + all_labels = any(v == 'all' for v in tables.values()) + if all_labels: + labels = set(range(entry, end, 4)) + last = None + for a in range(entry, end, 4): + ins = self.lift(a) + if a in labels: + flush() + out.append(f'L_{a:x}:;') + if a in PATCHES: # hand-written replacement for one instruction (tools/patches.py) + flush() + out.append(' ' + PATCHES[a](self, a)) + last = 'op' + continue + k = ins.kind + if k == 'asm': + if len(batch) >= 24: + flush() + batch.append(ins.asm) + try: + emit_asm_batch(batch) + except Unsupported: + batch.pop() + flush() + batch.append(ins.asm) + last = k + continue + flush() + if k == 'op' or k == 'trap': + out.append(' ' + ins.code) + elif k == 'b': + out.append(' ' + jump(ins.target)) + elif k == 'bcond': + out.append(f' if ({ins.cond}) {{ {jump(ins.target)} }}') + elif k == 'bl': + t = ins.target + if self.in_text(t): + refs.add(t) + out.append(f' c->x[30] = {hx(a + 4)}; {fname(t)}(c);') + else: + out.append(f' c->x[30] = {hx(a + 4)}; c->pc = {hx(t)}; rt_dispatch(c);') + elif k == 'blr': + out.append(f' c->pc = {ins.target}; c->x[30] = {hx(a + 4)}; rt_dispatch(c);') + elif k == 'br': + jt = tables.get(a) + if jt == 'all': + out.append(f' c->pc = {ins.target}; goto L_dispatch;') + continue + if jt: + cases = ' '.join(f'case {hx(t)}: goto L_{t:x};' for t in jt) + out.append(f' switch ({ins.target}) {{ {cases} default: break; }}') + if all_labels: + out.append(f' c->pc = {ins.target}; goto L_dispatch;') + else: + out.append(f' c->pc = {ins.target}; __attribute__((musttail)) return rt_dispatch(c);') + continue + out.append(f' c->pc = {ins.target}; __attribute__((musttail)) return rt_dispatch(c);') + elif k == 'ret': + out.append(' return;') + last = k + flush() + if last not in ('b', 'br', 'ret', 'trap'): + out.append(' ' + jump(end) if end < self.hi else f' rt_udf(c, {hx(end)});') + if all_labels: + out.append('L_dispatch:') + cases = ' '.join(f'case {hx(t)}: goto L_{t:x};' for t in range(entry, end, 4)) + out.append(f' switch (c->pc) {{ {cases} default: break; }}') + out.append(' __attribute__((musttail)) return rt_dispatch(c);') + out.append('}') + self._refs = refs + return out + + +from patches import PATCHES + + +def table_base(m, i, reg, window=40): + """True if the ADR at word i loads the base of a jump table: within a few instructions an ADD (register) uses + `reg` and a BR jumps to the sum. [TOTK] The compiler schedules other address setup between them (main + 0x800d2aab0: the br is 17 instructions after the adr), so the window ends at an unconditional branch or return + rather than after 16 instructions.""" + regs, added = {reg}, False + for j in range(i + 1, min(i + 1 + window, len(m.words))): + w = m.words[j] + if (w & 0xFC000000) == 0x14000000 or w == 0xD65F03C0: # B / RET: the block ends without the br + return False + if (w & 0xFFFFFC1F) == 0xD61F0000: # BR Xn + return added and ((w >> 5) & 31) in regs + if (w & 0x7F200000) in (0x0B000000, 0x0B200000) and (((w >> 5) & 31) in regs or ((w >> 16) & 31) in regs): + regs.add(w & 31) # ADD (shifted/extended register) of the base + added = True + return False + + +def write_if_changed(path, text): + """Leaves unchanged chunks untouched so make only recompiles what a patch affected.""" + try: + with open(path) as f: + if f.read() == text: + return + except FileNotFoundError: + pass + with open(path, 'w') as f: + f.write(text) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument('name') + ap.add_argument('nso') + ap.add_argument('base', type=lambda s: int(s, 0)) + ap.add_argument('outdir') + ap.add_argument('--hle') + ap.add_argument('--chunk', type=int, default=1500) # functions per file at most + ap.add_argument('--chunk-bytes', type=int, default=1500000) # and about this much C (even compile times) + ap.add_argument('--big-fn', type=int, default=100000) # bytes of C from which a function gets its own file + ap.add_argument('--limit', type=int, default=0) + ap.add_argument('--simd', choices=('asm', 'c'), + default='asm' if platform.machine().lower() in ('arm64', 'aarch64') else 'c') + args = ap.parse_args() + + t0 = time.time() + m = Module(args.name, args.nso, args.base, args.simd) + starts = m.boundaries() + hle = {} + if args.hle and os.path.exists(args.hle): + want = {l.strip() for l in open(args.hle) if l.strip() and not l.startswith('#')} + for name, info, shndx, value, size in m.nso.syms: + if shndx and name in want: + hle[args.base + value] = name + + # Partition text into functions; any branch/call target outside its function becomes a new + # boundary (splitting the containing function) until a fixpoint is reached. + start_set = set(starts) + for i, w in enumerate(m.words): + if (w & 0xFC000000) == 0x94000000: + t = m.lo + i * 4 + sext26(w) * 4 + if m.in_text(t): + start_set.add(t) + # Code addresses materialized in registers (callbacks, vtables built at runtime): ADR, or + # ADRP followed shortly by ADD on the same register. + adrp_regs = {} + for i, w in enumerate(m.words): + pc = m.lo + i * 4 + if (w & 0x9F000000) == 0x10000000: # ADR + imm = ((w >> 5) & 0x7FFFF) << 2 | ((w >> 29) & 3) + if imm & (1 << 20): + imm -= 1 << 21 + # [TOTK] an ADR that is the base of a jump table (adr xT; ...; add xD, xT, wI, ...; br xD) is not a + # function: as a start it split the switch function there and made the table's targets unreachable. + if m.in_text(pc + imm) and not table_base(m, i, w & 31): + start_set.add(pc + imm) + elif (w & 0x9F000000) == 0x90000000: # ADRP + imm = ((w >> 5) & 0x7FFFF) << 2 | ((w >> 29) & 3) + if imm & (1 << 20): + imm -= 1 << 21 + adrp_regs[w & 31] = ((pc & ~0xFFF) + (imm << 12), i) + elif (w & 0xFFC00000) == 0x91000000: # ADD (imm, 64-bit, no shift) + rn, rd = (w >> 5) & 31, w & 31 + if rn in adrp_regs and i - adrp_regs[rn][1] <= 8: + t = adrp_regs[rn][0] + ((w >> 10) & 0xFFF) + if m.in_text(t): + start_set.add(t) + scanned = {} + rounds = 0 + while True: + rounds += 1 + order = sorted(start_set) + new = set() + for i, a in enumerate(order): + end = order[i + 1] if i + 1 < len(order) else m.hi + key = (a, end) + if key not in scanned: + scanned[key] = m.scan(a, end) + new |= scanned[key][1] - start_set + if not new: + break + start_set |= new + entries = {} + for i, a in enumerate(order): + end = order[i + 1] if i + 1 < len(order) else m.hi + labels, ext, tables = scanned[(a, end)] + entries[a] = (end, labels, tables) + print(f'{args.name}: {len(starts)} boundaries, {len(entries)} functions, {rounds} rounds, {dict(m.stats)}, ' + f'{time.time() - t0:.0f}s', file=sys.stderr) + + os.makedirs(args.outdir, exist_ok=True) + order = sorted(entries) + if args.limit: + order = order[:args.limit] + nchunks = 0 + ci = 0 + pending = None # (address, lines, refs) of a big function that starts the next file + while ci < len(order) or pending: + body, refs, size, fns = [], set(), 0, [] + while (pending or ci < len(order)) and len(fns) < args.chunk and size < args.chunk_bytes: + if pending: + a, lines, r = pending + pending = None + else: + a = order[ci] + ci += 1 + end, labels, tables = entries[a] + lines = m.emit_function(a, end, labels, tables, hle) + r = getattr(m, '_refs', set()) + m._refs = set() + n = sum(len(l) + 1 for l in lines) + if n > args.big_fn and fns: # (a big function gets a file of its own: they compile in parallel) + pending = (a, lines, r) + break + body += lines + size += n + fns.append(a) + refs |= r + if n > args.big_fn: + break + path = os.path.join(args.outdir, f'{args.name}_{nchunks:03d}.c') + text = ['#include "cpu.h"\n' + ('#include "simd.h"\n' if m.lifter.sc else '')] + text += [f'void {fname(r)}(Cpu *c);\n' for r in sorted(refs)] + text += [f'void {sanitize(hle[a])}(Cpu *c);\n' for a in sorted(set(hle) & set(fns))] + text += ['\n'.join(body), '\n'] + write_if_changed(path, ''.join(text)) + nchunks += 1 + # (the start screen's setup reads these for its progress bar: runtime/gamebuild.c) + print(f'{args.name}: progress {ci}/{len(order)} files {nchunks}', file=sys.stderr, flush=True) + for fn in os.listdir(args.outdir): # an earlier run's extra chunks + mm = re.fullmatch(re.escape(args.name) + r'_(\d+)\.c', fn) + if mm and int(mm.group(1)) >= nchunks: + os.remove(os.path.join(args.outdir, fn)) + import io + with io.StringIO() as f: + f.write('#include "cpu.h"\n') + for a in order: + f.write(f'void {fname(a)}(Cpu *c);\n') + f.write(f'const uint64_t {args.name}_fn_addrs[] = {{\n') + f.write(',\n'.join(hx(a) for a in order)) + f.write('};\n') + f.write(f'const GuestFn {args.name}_fn_ptrs[] = {{\n') + f.write(',\n'.join(fname(a) for a in order)) + f.write('};\n') + f.write(f'const uint64_t {args.name}_fn_count = {len(order)};\n') + write_if_changed(os.path.join(args.outdir, f'{args.name}_table.c'), f.getvalue()) + print(f'{args.name}: wrote {nchunks} chunks, {time.time() - t0:.0f}s total', file=sys.stderr) + + +if __name__ == '__main__': + main() diff --git a/tools/lifter/lifter.py b/tools/lifter/lifter.py new file mode 100644 index 0000000..88e7a16 --- /dev/null +++ b/tools/lifter/lifter.py @@ -0,0 +1,728 @@ +"""AArch64 -> C static recompiler. + +Integer, memory, and control flow are translated to C. FP/SIMD instructions (and CRC32) are +either re-emitted as inline assembly (simd='asm', AArch64 hosts), with guest V registers +loaded from / stored to the Cpu struct around each batch, so their semantics are +bit-exact by construction, or lowered to portable C (simd='c', tools/simd_c.py + runtime/simd.h). +""" +import struct + +# Cpu struct offsets (must match runtime/cpu.h) +OFF_X = 0 +OFF_SP = 256 +OFF_PC = 264 +OFF_NZCV = 272 +OFF_V = 512 + +COND_NAMES = ['eq', 'ne', 'cs', 'cc', 'mi', 'pl', 'vs', 'vc', 'hi', 'ls', 'ge', 'lt', 'gt', 'le', 'al', 'nv'] +SCRATCH = [9, 10, 11, 12, 13, 14, 15, 16, 17] + + +class Unsupported(Exception): + pass + + +def bits(w, lo, n): + return (w >> lo) & ((1 << n) - 1) + + +def sext(v, n): + return v - (1 << n) if v & (1 << (n - 1)) else v + + +def ones(n): + return (1 << n) - 1 + + +def ror(v, r, size): + r %= size + return ((v >> r) | (v << (size - r))) & ones(size) if r else v + + +def replicate(v, esize, size): + out = 0 + for i in range(0, size, esize): + out |= v << i + return out + + +def decode_bit_masks(N, imms, immr, size, immediate): + x = (N << 6) | (~imms & 0x3F) + if x == 0: + raise Unsupported('reserved bitmask') + length = x.bit_length() - 1 + if length < 1: + raise Unsupported('reserved bitmask') + levels = ones(length) + if immediate and (imms & levels) == levels: + raise Unsupported('reserved bitmask') + S = imms & levels + R = immr & levels + diff = (S - R) & 0x3F + esize = 1 << length + d = diff & ones(length) + welem = ones(S + 1) + telem = ones(d + 1) + wmask = replicate(ror(welem, R, esize), esize, size) + tmask = replicate(telem, esize, size) + return wmask, tmask + + +def hx(v): + return f'0x{v:x}ull' + + +# ---------------------------------------------------------------- register helpers + +def X(r): + return '0ull' if r == 31 else f'c->x[{r}]' + + +def XS(r): + return 'c->sp' if r == 31 else f'c->x[{r}]' + + +def W(r): + return '0u' if r == 31 else f'(uint32_t)c->x[{r}]' + + +def WS(r): + return '(uint32_t)c->sp' if r == 31 else f'(uint32_t)c->x[{r}]' + + +def R(sf, r, sp=False): + if sf: + return XS(r) if sp else X(r) + return WS(r) if sp else W(r) + + +def set_reg(sf, r, expr, sp=False): + """Statement writing expr to Rd (zero-extending 32-bit results).""" + if r == 31 and not sp: + return f'(void)({expr});' + dst = 'c->sp' if r == 31 else f'c->x[{r}]' + if sf: + return f'{dst} = (uint64_t)({expr});' + return f'{dst} = (uint32_t)({expr});' + + +def shift_expr(sf, val, typ, amt): + if amt == 0 and typ != 3: + return val + if typ == 0: + return f'({val} << {amt})' + if typ == 1: + return f'({val} >> {amt})' + if typ == 2: + return f'(uint64_t)((int64_t){val} >> {amt})' if sf else f'(uint32_t)((int32_t){val} >> {amt})' + return f'ROR{64 if sf else 32}({val}, {amt})' + + +def extend_expr(r, option, shift): + base = X(r) + e = { + 0: f'(uint64_t)(uint8_t){base}', + 1: f'(uint64_t)(uint16_t){base}', + 2: f'(uint64_t)(uint32_t){base}', + 3: base, + 4: f'(uint64_t)(int64_t)(int8_t){base}', + 5: f'(uint64_t)(int64_t)(int16_t){base}', + 6: f'(uint64_t)(int64_t)(int32_t){base}', + 7: base, + }[option] + return f'({e} << {shift})' if shift else e + + +# ---------------------------------------------------------------- instruction classes + +class Insn: + """Result of lifting one instruction.""" + __slots__ = ('kind', 'code', 'target', 'cond', 'asm') + + def __init__(self, kind, code='', target=None, cond=None, asm=None): + self.kind = kind # 'op', 'asm', 'b', 'bcond', 'bl', 'blr', 'br', 'ret', 'trap' + self.code = code # C statements (for 'op'), or condition expr for 'bcond' + self.target = target # branch target (int) or register expr + self.cond = cond + self.asm = asm # AsmInsn for 'asm' + + +class AsmInsn: + """An FP/SIMD instruction to pass through as inline asm.""" + __slots__ = ('word', 'vregs', 'gprs', 'flags', 'consts') + + def __init__(self, word, vregs, gprs, flags=False, consts=None): + self.word = word + self.vregs = vregs # set of V register numbers touched + self.gprs = gprs # list of (bit position, guest reg or 'sp') + self.flags = flags + self.consts = consts or [] # list of (bit position, constant) loaded into scratch + + +def is_fp_simd_dp(w): + return bits(w, 25, 3) == 0b111 + + +def is_ldst(w): + return bits(w, 25, 1) == 0 and bits(w, 27, 1) == 1 + + +class Lifter: + def __init__(self, base, simd='asm'): + self.base = base # absolute load address of the module (guest == host) + if simd not in ('asm', 'c'): + raise ValueError(f'simd={simd!r}') + self.sc = None + if simd == 'c': + from simd_c import SimdC + self.sc = SimdC() + + def lift(self, w, pc): + """pc is absolute. Returns an Insn.""" + op0 = bits(w, 25, 4) + if (op0 & 0b1110) == 0b1000: + return self.dp_imm(w, pc) + if (op0 & 0b1110) == 0b1010: + return self.branch_sys(w, pc) + if (op0 & 0b0101) == 0b0100: + return self.ldst(w, pc) + if (op0 & 0b0111) == 0b0101: + return self.dp_reg(w, pc) + if (op0 & 0b0111) == 0b0111: + return self.fp_simd(w, pc) + # padding, trap words (e7ffdefe), header data: trap if ever executed + return Insn('trap', f'rt_udf(c, {hx(pc)});') + + # ------------------------------------------------------------ data processing (imm) + + def dp_imm(self, w, pc): + op = bits(w, 23, 3) + sf = bits(w, 31, 1) + rd, rn = bits(w, 0, 5), bits(w, 5, 5) + if op in (0, 1): # PC-relative + imm = sext((bits(w, 5, 19) << 2) | bits(w, 29, 2), 21) + if bits(w, 31, 1): + val = (pc & ~0xFFF) + (imm << 12) + else: + val = pc + imm + return Insn('op', set_reg(1, rd, hx(val & ones(64)))) + if op == 2: # add/sub imm + opc, S, sh = bits(w, 30, 1), bits(w, 29, 1), bits(w, 22, 1) + imm = bits(w, 10, 12) << (12 if sh else 0) + a = R(sf, rn, sp=True) + if S: + fn = ('subs' if opc else 'adds') + ('64' if sf else '32') + return Insn('op', set_reg(sf, rd, f'{fn}(c, {a}, {hx(imm)})')) + return Insn('op', set_reg(sf, rd, f'{a} {"-" if opc else "+"} {hx(imm)}', sp=True)) + if op == 4: # logical imm + opc, N = bits(w, 29, 2), bits(w, 22, 1) + if not sf and N: + raise Unsupported('logical imm N=1 32-bit') + size = 64 if sf else 32 + imm, _ = decode_bit_masks(N, bits(w, 10, 6), bits(w, 16, 6), size, True) + a = R(sf, rn) + if opc == 0: + return Insn('op', set_reg(sf, rd, f'{a} & {hx(imm)}', sp=True)) + if opc == 1: + return Insn('op', set_reg(sf, rd, f'{a} | {hx(imm)}', sp=True)) + if opc == 2: + return Insn('op', set_reg(sf, rd, f'{a} ^ {hx(imm)}', sp=True)) + return Insn('op', set_reg(sf, rd, f'logic{size}(c, {a} & {hx(imm)})')) + if op == 5: # move wide + opc, hw = bits(w, 29, 2), bits(w, 21, 2) + if not sf and hw > 1: + raise Unsupported('movw hw') + imm = bits(w, 5, 16) << (hw * 16) + if opc == 0: + return Insn('op', set_reg(sf, rd, hx(~imm & ones(64 if sf else 32)))) + if opc == 2: + return Insn('op', set_reg(sf, rd, hx(imm))) + if opc == 3: + mask = ~(0xFFFF << (hw * 16)) & ones(64) + return Insn('op', set_reg(sf, rd, f'({X(rd)} & {hx(mask)}) | {hx(imm)}')) + raise Unsupported('movw opc') + if op == 6: # bitfield + opc, N = bits(w, 29, 2), bits(w, 22, 1) + immr, imms = bits(w, 16, 6), bits(w, 10, 6) + size = 64 if sf else 32 + if N != sf: + raise Unsupported('bitfield N') + wmask, tmask = decode_bit_masks(N, imms, immr, size, False) + src = R(sf, rn) + rot = f'ROR{size}({src}, {immr})' if immr else src + if opc == 2: # UBFM + return Insn('op', set_reg(sf, rd, f'{rot} & {hx(wmask & tmask)}')) + if opc == 0: # SBFM + top = f'(uint{size}_t)(-(int{size}_t)(({src} >> {imms}) & 1))' + return Insn('op', set_reg( + sf, rd, f'({top} & {hx(~tmask & ones(size))}) | ({rot} & {hx(wmask & tmask)})')) + if opc == 1: # BFM + dst = R(sf, rd) + bot = f'(({dst} & {hx(~wmask & ones(size))}) | ({rot} & {hx(wmask)}))' + return Insn('op', set_reg( + sf, rd, f'({dst} & {hx(~tmask & ones(size))}) | ({bot} & {hx(tmask)})')) + raise Unsupported('bitfield opc') + if op == 7: # EXTR + rm, lsb = bits(w, 16, 5), bits(w, 10, 6) + size = 64 if sf else 32 + if lsb == 0: + return Insn('op', set_reg(sf, rd, R(sf, rm))) + return Insn('op', set_reg( + sf, rd, f'({R(sf, rm)} >> {lsb}) | ((uint{size}_t){R(sf, rn)} << {size - lsb})')) + raise Unsupported(f'dp_imm {w:08x}') + + # ------------------------------------------------------------ branches / system + + def branch_sys(self, w, pc): + if bits(w, 26, 5) == 0b00101: # B / BL + t = (pc + sext(bits(w, 0, 26), 26) * 4) & ones(64) + return Insn('bl' if bits(w, 31, 1) else 'b', target=t) + if bits(w, 25, 7) == 0b0101010 and not bits(w, 4, 1): # B.cond + t = pc + sext(bits(w, 5, 19), 19) * 4 + cond = bits(w, 0, 4) + if cond >= 14: + return Insn('b', target=t) + return Insn('bcond', target=t, cond=f'COND_{COND_NAMES[cond].upper()}(c)') + if bits(w, 25, 6) == 0b011010: # CBZ/CBNZ + sf, nz, rt = bits(w, 31, 1), bits(w, 24, 1), bits(w, 0, 5) + t = pc + sext(bits(w, 5, 19), 19) * 4 + return Insn('bcond', target=t, cond=f'{R(sf, rt)} {"!=" if nz else "=="} 0') + if bits(w, 25, 6) == 0b011011: # TBZ/TBNZ + b = (bits(w, 31, 1) << 5) | bits(w, 19, 5) + nz, rt = bits(w, 24, 1), bits(w, 0, 5) + t = pc + sext(bits(w, 5, 14), 14) * 4 + return Insn('bcond', target=t, cond=f'(({X(rt)} >> {b}) & 1) {"!=" if nz else "=="} 0') + if bits(w, 25, 7) == 0b1101011: # branch register + opc, rn = bits(w, 21, 4), bits(w, 5, 5) + if bits(w, 16, 5) != 31 or bits(w, 10, 6) or bits(w, 0, 5): + raise Unsupported('branch reg variant') + if opc == 0: + return Insn('br', target=X(rn)) + if opc == 1: + return Insn('blr', target=X(rn)) + if opc == 2: + return Insn('ret') + raise Unsupported('branch reg opc') + if bits(w, 24, 8) == 0b11010100: # exception generation + opc, ll, imm = bits(w, 21, 3), bits(w, 0, 5), bits(w, 5, 16) + if opc == 0 and ll == 1: + return Insn('op', f'rt_svc(c, {imm}, {hx(pc)});') + if opc == 1 and ll == 0: + return Insn('trap', f'rt_brk(c, {imm}, {hx(pc)});') + raise Unsupported('exception') + if bits(w, 22, 10) == 0b1101010100: + return self.system(w, pc) + raise Unsupported(f'branch_sys {w:08x}') + + def system(self, w, pc): + L = bits(w, 21, 1) + op0, op1, crn, crm, op2, rt = (bits(w, 19, 2), bits(w, 16, 3), bits(w, 12, 4), + bits(w, 8, 4), bits(w, 5, 3), bits(w, 0, 5)) + if op0 == 0 and not L: + if crn == 2 and op1 == 3 and rt == 31: # hints + return Insn('op', '/* hint */') + if crn == 3 and op1 == 3 and rt == 31: # barriers + if op2 == 2: + return Insn('op', 'c->excl_addr = ~0ull;') + return Insn('op', '__atomic_thread_fence(__ATOMIC_SEQ_CST);') + raise Unsupported(f'pstate/system {w:08x}') + if op0 == 1: # SYS + if not L and op1 == 3 and crn == 7 and crm == 4 and op2 == 1: # DC ZVA + return Insn('op', f'memset((void*)(uintptr_t)({X(rt)} & ~63ull), 0, 64);') + return Insn('op', '/* cache maintenance */') + key = (op0, op1, crn, crm, op2) + if L: # MRS + src = { + (3, 3, 13, 0, 3): 'c->tpidrro', + (3, 3, 13, 0, 2): 'c->tpidr', + (3, 3, 14, 0, 0): '19200000ull', + (3, 3, 14, 0, 1): 'rt_ticks()', + (3, 3, 14, 0, 2): 'rt_ticks()', + (3, 3, 4, 2, 0): 'c->nzcv', + (3, 3, 4, 4, 0): 'c->fpcr' if self.sc else 'rt_get_fpcr()', + (3, 3, 4, 4, 1): 'c->fpsr' if self.sc else 'rt_get_fpsr()', + (3, 3, 0, 0, 7): '4ull', + (3, 3, 0, 0, 1): '0x8444c004ull', + }.get(key) + if src is None: + raise Unsupported(f'mrs {key}') + return Insn('op', set_reg(1, rt, src)) + dst = { + (3, 3, 13, 0, 2): 'c->tpidr = {v};', + (3, 3, 4, 2, 0): 'c->nzcv = {v} & 0xf0000000ull;', + (3, 3, 4, 4, 0): 'rt_fpcr_write(c, {v});' if self.sc else 'rt_set_fpcr({v});', # [TOTK] FZ -> host MXCSR (simd.h) + (3, 3, 4, 4, 1): 'c->fpsr = {v} & 0xf800009full;' if self.sc else 'rt_set_fpsr({v});', + }.get(key) + if dst is None: + raise Unsupported(f'msr {key}') + return Insn('op', dst.format(v=X(rt))) + + # ------------------------------------------------------------ loads / stores + + def ldst(self, w, pc): + V = bits(w, 26, 1) + if V: + if self.sc: + return Insn('op', self.sc.ldst(w, pc)) + return self.ldst_simd(w, pc) + if bits(w, 24, 6) == 0b001000: + return self.ldst_excl(w, pc) + if bits(w, 27, 3) == 0b011 and bits(w, 24, 2) == 0: # literal + opc, rt = bits(w, 30, 2), bits(w, 0, 5) + addr = pc + sext(bits(w, 5, 19), 19) * 4 + if opc == 0: + return Insn('op', set_reg(0, rt, f'LD32({hx(addr)})')) + if opc == 1: + return Insn('op', set_reg(1, rt, f'LD64({hx(addr)})')) + if opc == 2: + return Insn('op', set_reg(1, rt, f'(int64_t)(int32_t)LD32({hx(addr)})')) + return Insn('op', '/* prfm */') + if bits(w, 27, 3) == 0b101: + return self.ldst_pair(w) + if bits(w, 27, 3) == 0b111: + return self.ldst_reg(w) + raise Unsupported(f'ldst {w:08x}') + + def ldst_excl(self, w, pc): + size, o2, L, o1, rs, o0, rn, rt = (bits(w, 30, 2), bits(w, 23, 1), bits(w, 22, 1), bits(w, 21, 1), + bits(w, 16, 5), bits(w, 15, 1), bits(w, 5, 5), bits(w, 0, 5)) + nbits = 8 << size + addr = XS(rn) + if o1: + raise Unsupported('exclusive pair / CAS') + if o2 == 0: + if L: + return Insn('op', set_reg(1, rt, f'ldx{nbits}(c, {addr})')) + return Insn('op', f'{{ uint32_t st = stx{nbits}(c, {addr}, {X(rt)}); {set_reg(0, rs, "st")} }}') + if L: + return Insn('op', set_reg(1, rt, f'lda{nbits}({addr})')) + return Insn('op', f'stl{nbits}({addr}, {X(rt)});') + + def ldst_pair(self, w): + opc, mode, L = bits(w, 30, 2), bits(w, 23, 2), bits(w, 22, 1) + imm7, rt2, rn, rt = sext(bits(w, 15, 7), 7), bits(w, 10, 5), bits(w, 5, 5), bits(w, 0, 5) + if opc == 3: + raise Unsupported('pair opc 3') + scale = 3 if opc == 2 else 2 + off = imm7 << scale + base = XS(rn) + post = mode == 1 + wb = mode in (1, 3) + ea = base if post else (f'{base} + {off}' if off >= 0 else f'{base} - {-off}') + lines = [f'uint64_t ea = {ea};'] + if L: + if opc == 2: + lines.append('uint64_t a = LD64(ea), b = LD64(ea + 8);') + elif opc == 1: # LDPSW + lines.append('uint64_t a = (int64_t)(int32_t)LD32(ea), b = (int64_t)(int32_t)LD32(ea + 4);') + else: + lines.append('uint64_t a = LD32(ea), b = LD32(ea + 4);') + else: + if opc == 2: + lines.append(f'ST64(ea, {X(rt)}); ST64(ea + 8, {X(rt2)});') + else: + lines.append(f'ST32(ea, {W(rt)}); ST32(ea + 4, {W(rt2)});') + if wb: + dst = 'c->sp' if rn == 31 else f'c->x[{rn}]' + if not post: + lines.append(f'{dst} = ea;') + else: + lines.append(f'{dst} = ea + {off};' if off >= 0 else f'{dst} = ea - {-off};') + if L: + lines.append(set_reg(1, rt, 'a')) + lines.append(set_reg(1, rt2, 'b')) + return Insn('op', '{ ' + ' '.join(lines) + ' }') + + def ldst_reg(self, w): + size, opc, rn, rt = bits(w, 30, 2), bits(w, 22, 2), bits(w, 5, 5), bits(w, 0, 5) + base = XS(rn) + wb = None + if bits(w, 24, 2) == 1: + ea = f'{base} + {hx(bits(w, 10, 12) << size)}' + else: + kind = bits(w, 10, 2) + if bits(w, 21, 1): + if kind != 2: + raise Unsupported('ldst reg atomic/pac') + rm, option, S = bits(w, 16, 5), bits(w, 13, 3), bits(w, 12, 1) + if option not in (2, 3, 6, 7): + raise Unsupported('ldst reg option') + ea = f'{base} + {extend_expr(rm, option, size if S else 0)}' + else: + imm9 = sext(bits(w, 12, 9), 9) + sign_imm = f'+ {imm9}' if imm9 >= 0 else f'- {-imm9}' + if kind in (0, 2): # unscaled, unprivileged + ea = f'{base} {sign_imm}' + elif kind == 1: # post + ea = base + wb = f'{base} {sign_imm}' + else: # pre + ea = f'{base} {sign_imm}' + wb = 'ea' + nbits = 8 << size + if opc == 0: + parts = [f'ST{nbits}(ea, {X(rt) if size == 3 else W(rt)});'] + load = None + elif opc == 1: + parts = [f'uint64_t v = LD{nbits}(ea);'] + load = set_reg(1, rt, 'v') + elif size == 3: + return Insn('op', '/* prfm */') + elif size == 2 and opc == 3: + raise Unsupported('ldst size2 opc3') + else: + parts = [f'uint64_t v = (uint64_t)(int64_t)(int{nbits}_t)LD{nbits}(ea);'] + load = set_reg(1 if opc == 2 else 0, rt, 'v') + parts.insert(0, f'uint64_t ea = {ea};') + if wb: + # writeback before the register load, so a load into rn wins + parts.append(f'{"c->sp" if rn == 31 else f"c->x[{rn}]"} = {wb};') + if load: + parts.append(load) + return Insn('op', '{ ' + ' '.join(parts) + ' }') + + def ldst_simd(self, w, pc): + """FP/SIMD loads and stores are passed through as asm; GPR address regs are remapped.""" + rt = bits(w, 0, 5) + if bits(w, 27, 3) == 0b011 and bits(w, 24, 2) == 0: # literal -> LDR (imm, offset 0) via scratch + opc = bits(w, 30, 2) + size, o = {0: (2, 1), 1: (3, 1), 2: (0, 3)}[opc] + addr = pc + sext(bits(w, 5, 19), 19) * 4 + new = (size << 30) | 0x3D000000 | (o << 22) | (31 << 5) | rt + return Insn('asm', asm=AsmInsn(new, {rt}, [], consts=[(5, addr)])) + if bits(w, 31, 1) == 0 and bits(w, 24, 6) in (0b001100, 0b001101): # structures + post = bits(w, 23, 1) + rn, rm = bits(w, 5, 5), bits(w, 16, 5) + if bits(w, 24, 1) == 0: + n = {0: 4, 2: 4, 4: 3, 6: 3, 7: 1, 8: 2, 10: 2}.get(bits(w, 12, 4)) + if n is None: + raise Unsupported('ldst multiple opcode') + else: + n = ((bits(w, 13, 1) << 1) | bits(w, 21, 1)) + 1 + vregs = {(rt + i) % 32 for i in range(n)} + gprs = [(5, 'sp' if rn == 31 else rn)] + if post and rm != 31: + gprs.append((16, rm)) + return Insn('asm', asm=AsmInsn(w, vregs, gprs)) + rn = bits(w, 5, 5) + gprs = [(5, 'sp' if rn == 31 else rn)] + vregs = {rt} + if bits(w, 27, 3) == 0b101: # pair + vregs.add(bits(w, 10, 5)) + elif bits(w, 27, 3) == 0b111 and bits(w, 24, 2) == 0 and bits(w, 21, 1) and bits(w, 10, 2) == 2: + rm = bits(w, 16, 5) + if rm != 31: + gprs.append((16, rm)) + return Insn('asm', asm=AsmInsn(w, vregs, gprs)) + + # ------------------------------------------------------------ data processing (reg) + + def dp_reg(self, w, pc): + sf = bits(w, 31, 1) + rd, rn, rm = bits(w, 0, 5), bits(w, 5, 5), bits(w, 16, 5) + size = 64 if sf else 32 + op1 = bits(w, 28, 1) + if not op1: + if bits(w, 24, 4) == 0b1010: # logical shifted + opc, sh, N, amt = bits(w, 29, 2), bits(w, 22, 2), bits(w, 21, 1), bits(w, 10, 6) + if not sf and amt >= 32: + raise Unsupported('logical shift amount') + b = shift_expr(sf, R(sf, rm), sh, amt) + if N: + b = f'(uint{size}_t)~{b}' + a = R(sf, rn) + expr = {0: f'{a} & {b}', 1: f'{a} | {b}', 2: f'{a} ^ {b}', 3: f'{a} & {b}'}[opc] + if opc == 3: + return Insn('op', set_reg(sf, rd, f'logic{size}(c, (uint{size}_t)({expr}))')) + return Insn('op', set_reg(sf, rd, expr)) + if bits(w, 24, 4) == 0b1011: # add/sub shifted or extended + op, S = bits(w, 30, 1), bits(w, 29, 1) + if bits(w, 21, 1) == 0: + sh, amt = bits(w, 22, 2), bits(w, 10, 6) + if sh == 3: + raise Unsupported('add shift ror') + a = R(sf, rn) + b = shift_expr(sf, R(sf, rm), sh, amt) + sp_ok = False + else: + if bits(w, 22, 2): + raise Unsupported('add ext opt') + option, amt = bits(w, 13, 3), bits(w, 10, 3) + a = R(sf, rn, sp=True) + b = extend_expr(rm, option, amt) + if not sf: + b = f'(uint32_t)({b})' + sp_ok = not S + if S: + fn = ('subs' if op else 'adds') + str(size) + return Insn('op', set_reg(sf, rd, f'{fn}(c, {a}, (uint{size}_t)({b}))')) + return Insn('op', set_reg(sf, rd, f'{a} {"-" if op else "+"} {b}', sp=sp_ok)) + raise Unsupported(f'dp_reg {w:08x}') + op2 = bits(w, 21, 4) + if bits(w, 24, 4) == 0b1011: # 3-source + op54, op31, o0, ra = bits(w, 29, 2), bits(w, 21, 3), bits(w, 15, 1), bits(w, 10, 5) + if op54: + raise Unsupported('dp3 op54') + sgn = '-' if o0 else '+' + if op31 == 0: + return Insn('op', set_reg(sf, rd, f'{R(sf, ra)} {sgn} {R(sf, rn)} * {R(sf, rm)}')) + if op31 == 1: + return Insn('op', set_reg(1, rd, f'{X(ra)} {sgn} (uint64_t)((int64_t)(int32_t){W(rn)} * (int64_t)(int32_t){W(rm)})')) + if op31 == 5: + return Insn('op', set_reg(1, rd, f'{X(ra)} {sgn} (uint64_t){W(rn)} * (uint64_t){W(rm)}')) + if op31 == 2: + return Insn('op', set_reg( + 1, rd, f'(uint64_t)(((__int128)(int64_t){X(rn)} * (__int128)(int64_t){X(rm)}) >> 64)')) + if op31 == 6: + return Insn('op', set_reg(1, rd, f'(uint64_t)(((unsigned __int128){X(rn)} * {X(rm)}) >> 64)')) + raise Unsupported('dp3') + if op2 == 0: # add/sub with carry + op, S = bits(w, 30, 1), bits(w, 29, 1) + fn = ('sbc' if op else 'adc') + ('s' if S else '') + str(size) + return Insn('op', set_reg(sf, rd, f'{fn}(c, {R(sf, rn)}, {R(sf, rm)})')) + if op2 == 2: # conditional compare + op, imm_form, cond, nzcv = bits(w, 30, 1), bits(w, 11, 1), bits(w, 12, 4), bits(w, 0, 4) + b = hx(rm) if imm_form else R(sf, rm) + fn = ('subs' if op else 'adds') + str(size) + return Insn('op', f'if ({cond_expr(cond)}) (void){fn}(c, {R(sf, rn)}, {b}); ' + f'else c->nzcv = {hx(nzcv << 28)};') + if op2 == 4: # conditional select + op, o2, cond = bits(w, 30, 1), bits(w, 10, 2), bits(w, 12, 4) + b = R(sf, rm) + alt = {(0, 0): b, (0, 1): f'{b} + 1', (1, 0): f'~{b}', (1, 1): f'-{b}'}.get((op, o2)) + if alt is None: + raise Unsupported('csel variant') + return Insn('op', set_reg(sf, rd, f'({cond_expr(cond)}) ? {R(sf, rn)} : (uint{size}_t)({alt})')) + if op2 == 6: + opcode = bits(w, 10, 6) + if bits(w, 30, 1): # 1-source + if bits(w, 16, 5): + raise Unsupported('dp1 opcode2') + a = R(sf, rn) + expr = { + 0: f'rbit{size}({a})', + 1: f'rev16_{size}({a})', + 2: f'__builtin_bswap32({a})' if not sf else f'rev32_64({a})', + 3: f'__builtin_bswap64({a})' if sf else None, + 4: f'clz{size}({a})', + 5: f'cls{size}({a})', + }.get(opcode) + if expr is None: + raise Unsupported('dp1') + return Insn('op', set_reg(sf, rd, expr)) + if 16 <= opcode <= 23: # CRC32: host has the same instruction + if self.sc: + return Insn('op', self.sc.crc(w)) + gprs = [(p, r) for p, r in ((0, rd), (5, rn), (16, rm)) if r != 31] + return Insn('asm', asm=AsmInsn(w, set(), gprs)) + a, b = R(sf, rn), R(sf, rm) + expr = { + 2: f'udiv{size}({a}, {b})', + 3: f'sdiv{size}({a}, {b})', + 8: f'{a} << ({b} & {size - 1})', + 9: f'{a} >> ({b} & {size - 1})', + 10: f'(uint{size}_t)((int{size}_t){a} >> ({b} & {size - 1}))', + 11: f'ROR{size}({a}, {b} & {size - 1})', + }.get(opcode) + if expr is None: + raise Unsupported('dp2') + return Insn('op', set_reg(sf, rd, expr)) + raise Unsupported(f'dp_reg {w:08x}') + + # ------------------------------------------------------------ FP / SIMD data processing + + def fp_simd(self, w, pc): + if self.sc: + return Insn('op', self.sc.dp(w, pc)) + rd, rn, rm, ra = bits(w, 0, 5), bits(w, 5, 5), bits(w, 16, 5), bits(w, 10, 5) + vregs = {rd, rn, rm, ra} + gprs = [] + flags = False + if (w & 0x5F20FC00) == 0x1E200000: # FP <-> integer + opcode = bits(w, 16, 3) + if opcode in (0, 1, 4, 5, 6): + gprs = [(0, rd)] + vregs = {rn} + else: + gprs = [(5, rn)] + vregs = {rd} + elif (w & 0x5F200000) == 0x1E000000: # FP <-> fixed point + opcode = bits(w, 16, 3) + if opcode in (0, 1): + gprs = [(0, rd)] + vregs = {rn} + else: + gprs = [(5, rn)] + vregs = {rd} + elif (w & 0x9FE08400) == 0x0E000400 and bits(w, 29, 1) == 0: # AdvSIMD copy + imm4 = bits(w, 11, 4) + if imm4 in (1, 3): + gprs = [(5, rn)] + vregs = {rd} + elif imm4 in (5, 7): + gprs = [(0, rd)] + vregs = {rn} + else: + vregs = {rd, rn} + elif (w & 0xBFE08C00) == 0x0E000000: # TBL/TBX + n = bits(w, 13, 2) + 1 + vregs = {rd, rm} | {(rn + i) % 32 for i in range(n)} + if (w & 0x5F203C00) == 0x1E202000 or (w & 0x5F200C00) in (0x1E200400, 0x1E200C00): + flags = True + gprs = [(p, r) for p, r in gprs if r != 31] # register 31 is zr in these encodings + return Insn('asm', asm=AsmInsn(w, vregs, gprs, flags)) + + +def cond_expr(cond): + return f'COND_{COND_NAMES[cond].upper()}(c)' if cond < 14 else '1' + + +def emit_asm_batch(batch): + """Emit one inline-asm block for a run of FP/SIMD instructions.""" + vregs = set() + gmap = {} + flags = False + consts = [] + words = [] + for a in batch: + vregs |= a.vregs + flags |= a.flags + w = a.word + for pos, g in a.gprs: + if g not in gmap: + if len(gmap) + len(consts) >= len(SCRATCH): + raise Unsupported('scratch exhausted') + gmap[g] = SCRATCH[len(gmap) + len(consts)] + w = (w & ~(31 << pos)) | (gmap[g] << pos) + for pos, val in a.consts: + s = SCRATCH[len(gmap) + len(consts)] + consts.append((s, val)) + w = (w & ~(31 << pos)) | (s << pos) + words.append(w) + lines = [] + ins = ['"r"(c)'] + for g, s in gmap.items(): + off = OFF_SP if g == 'sp' else OFF_X + 8 * g + lines.append(f'ldr x{s}, [%0, #{off}]') + for i, (s, val) in enumerate(consts): + ins.append(f'"r"({hx(val)})') + lines.append(f'mov x{s}, %{len(ins) - 1}') + for v in sorted(vregs): + lines.append(f'ldr q{v}, [%0, #{OFF_V + 16 * v}]') + if flags: + lines.append(f'ldr x16, [%0, #{OFF_NZCV}]') + lines.append('msr nzcv, x16') + lines += [f'.inst 0x{x:08x}' for x in words] + if flags: + lines.append('mrs x16, nzcv') + lines.append(f'str x16, [%0, #{OFF_NZCV}]') + for v in sorted(vregs): + lines.append(f'str q{v}, [%0, #{OFF_V + 16 * v}]') + for g, s in gmap.items(): + off = OFF_SP if g == 'sp' else OFF_X + 8 * g + lines.append(f'str x{s}, [%0, #{off}]') + clob = [f'"v{v}"' for v in sorted(vregs)] + clob += [f'"x{s}"' for s in sorted(set(gmap.values()) | {s for s, _ in consts} | ({16} if flags else set()))] + clob += ['"cc"', '"memory"'] + body = '\\n\\t'.join(lines) + return f'__asm__ volatile("{body}" :: {", ".join(ins)} : {", ".join(clob)});' diff --git a/tools/lifter/nso.py b/tools/lifter/nso.py new file mode 100644 index 0000000..ccc3e38 --- /dev/null +++ b/tools/lifter/nso.py @@ -0,0 +1,130 @@ +"""NSO loader: decompress segments, build the memory image, parse MOD0/dynamic info.""" +import struct +import sys +import lz4.block + +DT_NULL, DT_NEEDED, DT_PLTRELSZ, DT_HASH, DT_STRTAB, DT_SYMTAB, DT_RELA, DT_RELASZ = 0, 1, 2, 4, 5, 6, 7, 8 +DT_STRSZ, DT_JMPREL, DT_RELACOUNT = 10, 23, 0x6FFFFFF9 +DT_RELRSZ, DT_RELR, DT_GNU_HASH = 35, 36, 0x6FFFFEF5 +R_AARCH64_ABS64, R_AARCH64_GLOB_DAT, R_AARCH64_JUMP_SLOT, R_AARCH64_RELATIVE = 257, 1025, 1026, 1027 + + +class Nso: + def __init__(self, path): + raw = open(path, 'rb').read() + assert raw[:4] == b'NSO0', path + flags = struct.unpack_from('> 32, r_add)) + # Packed relative relocations (DT_RELR, TOTK 1.4.3): the addend is the word stored at the location. + if DT_RELR in dyn: + b, n = dyn[DT_RELR], dyn[DT_RELRSZ] // 8 + where = 0 + for i in range(n): + entry = self.u64(b + i * 8) + if entry & 1 == 0: + self.relocs.append((entry, R_AARCH64_RELATIVE, 0, self.u64(entry))) + where = entry + 8 + else: + bitmap = entry >> 1 + for bit in range(63): + if bitmap >> bit & 1: + at = where + bit * 8 + self.relocs.append((at, R_AARCH64_RELATIVE, 0, self.u64(at))) + where += 63 * 8 + + def _gnu_hash_count(self, off): + nbuckets, symoffset, bloom_size, _ = struct.unpack_from(' function(gen, addr) returning C code. + +Every patch checks the original instruction word first, so lifting another game version (or a different build) +fails loudly instead of changing the wrong code. Runtime switches live in src/gfx/nvn/hle_nvn.c. +""" + + +def expect(gen, addr, word, what): + got = gen.word(addr) + if got != word: + raise SystemExit(f'patch {what} at {addr:#x}: expected {word:#010x}, found {got:#010x} (not TOTK 1.4.3?)') + + +def drs_off(gen, addr): + """Dynamic resolution off (1.4.3). main+0x584300 is the scene resolution controller: each frame it fits the + recent frame times and returns {scale (float), state} to the renderer (main+0x581968), which turns the scale into + the scene size (+0xccc). Its first test is the controller's enable bit (+0x110 bit 0); with the bit clear it + takes its own disabled path (main+0x584c3c): scale 1.0, history untouched. The patch takes that path unless + TOTK_DRS=1. Why: the game flips the scene targets between 1600x896 and 1432x808 over the same memory and our + per-size images then hand stale content to the other size (terrain gone, clouds over the scene); + image quality is better at a fixed size and upscaling will come from FSR.""" + expect(gen, addr, 0x360042e8, 'drs_off') # tbz w8, #0, 0x800584c3c + return ('{ extern int totk_drs_enabled(void); ' + 'if (!totk_drs_enabled() || ((c->x[8] >> 0) & 1) == 0) { goto L_800584c3c; } }') + + +PATCHES = { + 0x8005843e0: drs_off, +} diff --git a/tools/lifter/simd_c.py b/tools/lifter/simd_c.py new file mode 100644 index 0000000..5d88b30 --- /dev/null +++ b/tools/lifter/simd_c.py @@ -0,0 +1,1156 @@ +"""Portable C lowering of AArch64 FP / AdvSIMD / crypto / CRC32 instructions (lifter --simd=c). + +Every encoding is decoded here at lift time and turned into C over the Cpu struct, calling the static +inline helpers in runtime/simd.h with constant fields (sizes, lanes, ops), so clang folds the helpers' +switches. Covers ARMv8.0-A + Crypto + CRC32 (the Switch's Cortex-A57); unallocated or later-architecture +encodings (FP16 arithmetic, RDM, dot product, FCMA, JSCVT, ...) raise Unsupported. +""" +from lifter import COND_NAMES, Unsupported, W, X, XS, bits, extend_expr, hx, set_reg, sext + + +def lop(name, u=0): + """C name of a lane op (enum in runtime/simd.h), with the unsigned flag.""" + return f'LOP_{name}' + (' | LOP_U' if u else '') + + +def vu(sz): + return ('vu8', 'vu16', 'vu32', 'vu64')[sz] + + +def vs(sz): + return ('vs8', 'vs16', 'vs32', 'vs64')[sz] + + +def ut(sz): + return ('uint8_t', 'uint16_t', 'uint32_t', 'uint64_t')[sz] + + +def VR(T, n): + return f'VREG({T}, {n})' + + +def VQ(T, Q, n): + return f'VREG({T}, {n})' if Q else f'VQ({T}, 0, {n})' + + +def VWr(T, Q, d, e): + return f'VW({T}, {Q}, {d}, {e});' + + +def cond(c): + return f'COND_{COND_NAMES[c].upper()}(c)' if c < 14 else '1' + + +def gen(op, sz, ne, d, szd, offd, keep, n, szn, offn, m=-1, szm=0, offm=0, bidx=-1, bimm=0): + return (f'sl_gen(c, {op}, {sz}, {ne}, {d}, {szd}, {offd}, {keep}, {n}, {szn}, {offn}, {m}, {szm}, {offm}, ' + f'{bidx}, {hx(bimm)});') + + +def same(op, sz, ne, d, n, m=-1, bimm=0, bidx=-1): + """Lane-wise op, all operands the same size.""" + return gen(op, sz, ne, d, sz, 0, 0, n, sz, 0, m, sz, 0, bidx, bimm) + + +def lanes(Q, sz): + return (8 << Q) >> sz + + +def shuf(T, a, b, idx): + return f'SHUF({a}, {b}, {", ".join(str(i) for i in idx)})' + + +def u128(v): + lo, hi = v & ((1 << 64) - 1), v >> 64 + return f'((u128){hx(hi)} << 64 | {hx(lo)})' if hi else hx(lo) + + +FT = {0: ('s', 'vf32', 'vu32', 'float', 'VSF32', 'VS32', 'FP_SETS'), + 1: ('d', 'vf64', 'vu64', 'double', 'VSF64', 'VS64', 'FP_SETD')} + +FOP3 = { # vector three-same FP: (U, a, opcode) -> FOP + (0, 0, 0b11000): 'FOP_MAXNM', (0, 0, 0b11010): 'FOP_ADD', (0, 0, 0b11011): 'FOP_MULX', + (0, 0, 0b11110): 'FOP_MAX', (0, 0, 0b11111): 'FOP_RECPS', + (0, 1, 0b11000): 'FOP_MINNM', (0, 1, 0b11010): 'FOP_SUB', (0, 1, 0b11110): 'FOP_MIN', (0, 1, 0b11111): 'FOP_RSQRTS', + (1, 0, 0b11011): 'FOP_MUL', (1, 0, 0b11111): 'FOP_DIV', (1, 1, 0b11010): 'FOP_ABD', +} +FPAIRV = {(1, 0, 0b11000): 'FOP_MAXNM', (1, 1, 0b11000): 'FOP_MINNM', (1, 0, 0b11010): 'FOP_ADD', + (1, 0, 0b11110): 'FOP_MAX', (1, 1, 0b11110): 'FOP_MIN'} +FCMP3 = {(0, 0, 0b11100): 0, (1, 0, 0b11100): 1, (1, 1, 0b11100): 2, (1, 0, 0b11101): 3, (1, 1, 0b11101): 4} + + +def vfp_expand_imm(imm8, N): + E = 8 if N == 32 else 11 + F = N - E - 1 + sign = imm8 >> 7 + b6 = (imm8 >> 6) & 1 + exp = ((b6 ^ 1) << (E - 1)) | ((((1 << (E - 3)) - 1) if b6 else 0) << 2) | ((imm8 >> 4) & 3) + frac = (imm8 & 15) << (F - 4) + return (sign << (N - 1)) | (exp << F) | frac + + +def simd_expand_imm(op, cmode, imm8): + c = cmode >> 1 + if c == 0: + imm = (imm8 << 32) | imm8 + elif c == 1: + imm = ((imm8 << 8) << 32) | (imm8 << 8) + elif c == 2: + imm = ((imm8 << 16) << 32) | (imm8 << 16) + elif c == 3: + imm = ((imm8 << 24) << 32) | (imm8 << 24) + elif c == 4: + imm = sum(imm8 << s for s in range(0, 64, 16)) + elif c == 5: + imm = sum((imm8 << 8) << s for s in range(0, 64, 16)) + elif c == 6: + v = ((imm8 << 8) | 0xFF) if not (cmode & 1) else ((imm8 << 16) | 0xFFFF) + imm = (v << 32) | v + else: + if not (cmode & 1) and not op: + imm = sum(imm8 << s for s in range(0, 64, 8)) + elif not (cmode & 1) and op: + imm = 0 + for i in range(8): + if (imm8 >> i) & 1: + imm |= 0xFF << (8 * i) + elif (cmode & 1) and not op: + v = vfp_expand_imm(imm8, 32) + imm = (v << 32) | v + else: + imm = vfp_expand_imm(imm8, 64) + return imm + + +class SimdC: + """FP/SIMD decoder emitting portable C. Methods return C statements or raise Unsupported.""" + + # ------------------------------------------------------------ top level + + def dp(self, w, pc): + op0, op1, op2, op3 = bits(w, 28, 4), bits(w, 23, 2), bits(w, 19, 4), bits(w, 10, 9) + if op0 == 0b0100 and not (op1 & 2) and (op2 & 7) == 5 and (op3 & 0b110000011) == 0b10: + return self.crypto_aes(w) + if op0 == 0b0101 and not (op1 & 2) and not (op2 & 4) and (op3 & 0b000100011) == 0: + return self.crypto_sha3(w) + if op0 == 0b0101 and not (op1 & 2) and (op2 & 7) == 5 and (op3 & 0b110000011) == 0b10: + return self.crypto_sha2(w) + if (op0 & 0b1101) == 0b0101: # AdvSIMD scalar + if op1 == 0 and not (op2 & 0b1100) and (op3 & 0b000100001) == 1: + return self.scalar_copy(w) + if not (op1 & 2) and (op2 & 7) == 4 and (op3 & 0b110000011) == 0b10: + return self.misc2(w, scalar=True) + if not (op1 & 2) and (op2 & 7) == 6 and (op3 & 0b110000011) == 0b10: + return self.scalar_pairwise(w) + if not (op1 & 2) and (op2 & 4) and (op3 & 3) == 0: + return self.three_diff(w, scalar=True) + if not (op1 & 2) and (op2 & 4) and (op3 & 1): + return self.three_same(w, scalar=True) + if op1 == 2 and (op3 & 1): + return self.shift_imm(w, scalar=True) + if (op1 & 2) and not (op3 & 1): + return self.indexed(w, scalar=True) + raise Unsupported(f'simd scalar {w:08x}') + if (op0 & 0b1011) == 0 and not (op1 & 2) and not (op2 & 4) and (op3 & 0b000100011) == 0: + return self.tbl(w) + if (op0 & 0b1011) == 0 and not (op1 & 2) and not (op2 & 4) and (op3 & 0b000100011) == 2: + return self.permute(w) + if (op0 & 0b1011) == 0b0010 and not (op1 & 2) and not (op2 & 4) and (op3 & 0b000100001) == 0: + return self.ext(w) + if (op0 & 0b1001) == 0: # AdvSIMD vector + if op1 == 0 and not (op2 & 0b1100) and (op3 & 0b000100001) == 1: + return self.copy(w) + if not (op1 & 2) and (op2 & 4) and (op3 & 1): + return self.three_same(w) + if not (op1 & 2) and (op2 & 4) and (op3 & 3) == 0: + return self.three_diff(w) + if not (op1 & 2) and (op2 & 7) == 4 and (op3 & 0b110000011) == 0b10: + return self.misc2(w) + if not (op1 & 2) and (op2 & 7) == 6 and (op3 & 0b110000011) == 0b10: + return self.across(w) + if op1 == 2 and op2 == 0 and (op3 & 1): + return self.modimm(w) + if op1 == 2 and op2 and (op3 & 1): + return self.shift_imm(w) + if (op1 & 2) and not (op3 & 1): + return self.indexed(w) + raise Unsupported(f'simd vector {w:08x}') + if (op0 & 0b0101) == 0b0001: # scalar FP + if bits(w, 29, 1) or (bits(w, 31, 1) and (op1 & 2 or (op2 & 4 and op3 & 0b111111))): + raise Unsupported('fp M/S') + if op1 & 2: + return self.fp_dp3(w) + if not (op2 & 4): + return self.fp_fixed(w) + if (op3 & 0b111111) == 0: + return self.fp_int(w) + if (op3 & 0b11111) == 0b10000: + return self.fp_dp1(w) + if (op3 & 0b1111) == 0b1000: + return self.fp_cmp(w) + if (op3 & 0b111) == 0b100: + return self.fp_imm(w) + if (op3 & 3) == 1: + return self.fp_ccmp(w) + if (op3 & 3) == 2: + return self.fp_dp2(w) + if (op3 & 3) == 3: + return self.fp_csel(w) + raise Unsupported('fp op3') + raise Unsupported(f'fp/simd {w:08x}') + + # ------------------------------------------------------------ crypto + + def crypto_aes(self, w): + if bits(w, 22, 2): + raise Unsupported('aes size') + opc = bits(w, 12, 5) + if opc not in (4, 5, 6, 7): + raise Unsupported('aes opcode') + return f'sv_aes(c, {opc - 4}, {bits(w, 0, 5)}, {bits(w, 5, 5)});' + + def crypto_sha3(self, w): + if bits(w, 22, 2): + raise Unsupported('sha size') + opc, d, n, m = bits(w, 12, 3), bits(w, 0, 5), bits(w, 5, 5), bits(w, 16, 5) + if opc < 3: + return f'sv_sha1hash(c, {opc}, {d}, {n}, {m});' + if opc == 3: + return f'sv_sha_misc(c, 0, {d}, {n}, {m});' + if opc in (4, 5): + return f'sv_sha256hash(c, {int(opc == 4)}, {d}, {n}, {m});' + if opc == 6: + return f'sv_sha_misc(c, 3, {d}, {n}, {m});' + raise Unsupported('sha3 opcode') + + def crypto_sha2(self, w): + if bits(w, 22, 2): + raise Unsupported('sha size') + opc, d, n = bits(w, 12, 5), bits(w, 0, 5), bits(w, 5, 5) + op = {0: 4, 1: 1, 2: 2}.get(opc) + if op is None: + raise Unsupported('sha2 opcode') + return f'sv_sha_misc(c, {op}, {d}, {n}, 0);' + + # ------------------------------------------------------------ copy / permute / table + + @staticmethod + def _imm5(imm5): + if not imm5 & 15: + raise Unsupported('copy imm5') + sz = (imm5 & -imm5).bit_length() - 1 + return sz, imm5 >> (sz + 1) + + def scalar_copy(self, w): + if bits(w, 29, 1) or bits(w, 11, 4): + raise Unsupported('scalar copy') + sz, idx = self._imm5(bits(w, 16, 5)) + if sz > 3: + raise Unsupported('scalar dup size') + d, n = bits(w, 0, 5), bits(w, 5, 5) + return f'c->v[{d}] = (({("uint8_t", "sa16", "sa32", "sa64")[sz]} *)&c->v[{n}])[{idx}];' + + def copy(self, w): + Q, op, imm5, imm4 = bits(w, 30, 1), bits(w, 29, 1), bits(w, 16, 5), bits(w, 11, 4) + d, n = bits(w, 0, 5), bits(w, 5, 5) + sz, idx = self._imm5(imm5) + if sz > 3: + raise Unsupported('copy size') + P = ('uint8_t', 'sa16', 'sa32', 'sa64')[sz] + if op: + if not Q: + raise Unsupported('ins elem Q') + idx2 = imm4 >> sz + return f'(({P} *)&c->v[{d}])[{idx}] = (({P} *)&c->v[{n}])[{idx2}];' + if imm4 == 0: # DUP (element) + if sz == 3 and not Q: + raise Unsupported('dup 1d') + nl = 16 >> sz + return VWr(vu(sz), Q, d, shuf(vu(sz), VR(vu(sz), n), VR(vu(sz), n), [idx] * nl)) + if imm4 == 1: # DUP (general) + if sz == 3 and not Q: + raise Unsupported('dup 1d') + return VWr(vu(sz), Q, d, f'({vu(sz)}){{}} + ({ut(sz)}){X(n)}') + if imm4 == 3: # INS (general) + if not Q: + raise Unsupported('ins Q') + return f'(({P} *)&c->v[{d}])[{idx}] = ({ut(sz)}){X(n)};' + if imm4 == 5: # SMOV + if sz >= 2 + Q: + raise Unsupported('smov size') + st = ('int8_t', 'int16_t', 'int32_t')[sz] + return set_reg(Q, d, f'(uint64_t)(int64_t)({st})(({P} *)&c->v[{n}])[{idx}]') + if imm4 == 7: # UMOV + if (Q and sz != 3) or (not Q and sz == 3): + raise Unsupported('umov size') + return set_reg(Q, d, f'(({P} *)&c->v[{n}])[{idx}]') + raise Unsupported('copy imm4') + + def permute(self, w): + Q, sz, m, opc, n, d = bits(w, 30, 1), bits(w, 22, 2), bits(w, 16, 5), bits(w, 12, 3), bits(w, 5, 5), bits(w, 0, 5) + if opc in (0, 4) or (sz == 3 and not Q): + raise Unsupported('permute') + ne, NL = lanes(Q, sz), 16 >> sz + part = opc >> 2 + kind = opc & 3 + idx = [] + for i in range(ne): + if kind == 1: # UZP + j = 2 * i + part + idx.append(j if j < ne else NL + j - ne) + elif kind == 2: # TRN + p = i & ~1 + idx.append(p + part if not (i & 1) else NL + p + part) + else: # ZIP + base = part * ne // 2 + idx.append(base + i // 2 if not (i & 1) else NL + base + i // 2) + idx += [0] * (NL - ne) + return VWr(vu(sz), Q, d, shuf(vu(sz), VR(vu(sz), n), VR(vu(sz), m), idx)) + + def ext(self, w): + Q, m, imm4, n, d = bits(w, 30, 1), bits(w, 16, 5), bits(w, 11, 4), bits(w, 5, 5), bits(w, 0, 5) + if bits(w, 22, 2) or (not Q and imm4 >= 8): + raise Unsupported('ext') + if Q: + idx = [imm4 + i for i in range(16)] + else: + idx = [(imm4 + i) if imm4 + i < 8 else 16 + imm4 + i - 8 for i in range(8)] + [0] * 8 + return VWr('vu8', Q, d, shuf('vu8', VR('vu8', n), VR('vu8', m), idx)) + + def tbl(self, w): + Q, m, ln, op, n, d = bits(w, 30, 1), bits(w, 16, 5), bits(w, 13, 2), bits(w, 12, 1), bits(w, 5, 5), bits(w, 0, 5) + if bits(w, 22, 2): + raise Unsupported('tbl op2') + return f'sv_tbl(c, {Q}, {ln}, {op}, {d}, {n}, {m});' + + # ------------------------------------------------------------ three same + + def three_same(self, w, scalar=False): + Q, U, sz, m, opc, n, d = (bits(w, 30, 1), bits(w, 29, 1), bits(w, 22, 2), bits(w, 16, 5), bits(w, 11, 5), + bits(w, 5, 5), bits(w, 0, 5)) + if opc >= 0b11000: + return self.three_same_fp(w, scalar) + if scalar: + return self.three_same_scalar(U, sz, opc, d, n, m) + if sz == 3 and not Q and opc != 3: + raise Unsupported('3same 1d') + ne = lanes(Q, sz) + T, S = vu(sz), vs(sz) + a, b, o = VR(T, n), VR(T, m), VR(T, d) + sa, sb = VR(S, n), VR(S, m) + x, y = (a, b) if U else (sa, sb) + if opc in (0, 2, 4, 14, 15, 12, 13) and sz == 3: + raise Unsupported('3same size') + if opc == 0: + return same(lop('HADD', U), sz, ne, d, n, m) + if opc == 1: + return VWr(T, Q, d, f'({T})__builtin_elementwise_{"add"}_sat({x}, {y})') + if opc == 2: + return same(lop('RHADD', U), sz, ne, d, n, m) + if opc == 3: + a64, b64, o64 = VR('vu64', n), VR('vu64', m), VR('vu64', d) + e = [[f'{a64} & {b64}', f'{a64} & ~{b64}', f'{a64} | {b64}', f'{a64} | ~{b64}'], + [f'{a64} ^ {b64}', f'({o64} & {a64}) | (~{o64} & {b64})', f'({o64} & ~{b64}) | ({a64} & {b64})', + f'({o64} & {b64}) | ({a64} & ~{b64})']][U][sz] + return VWr('vu64', Q, d, e) + if opc == 4: + return same(lop('HSUB', U), sz, ne, d, n, m) + if opc == 5: + return VWr(T, Q, d, f'({T})__builtin_elementwise_sub_sat({x}, {y})') + if opc == 6: + return VWr(T, Q, d, f'({T})({x} > {y})') + if opc == 7: + return VWr(T, Q, d, f'({T})({x} >= {y})') + if 8 <= opc <= 11: + return same(lop(('SHL', 'QSHL', 'RSHL', 'QRSHL')[opc - 8], U), sz, ne, d, n, m) + if opc == 12: + return VWr(T, Q, d, f'({T})__builtin_elementwise_max({x}, {y})') + if opc == 13: + return VWr(T, Q, d, f'({T})__builtin_elementwise_min({x}, {y})') + if opc in (14, 15): + e = f'({T})__builtin_elementwise_max({x}, {y}) - ({T})__builtin_elementwise_min({x}, {y})' + return VWr(T, Q, d, f'{o} + ({e})' if opc == 15 else e) + if opc == 16: + return VWr(T, Q, d, f'{a} {"-" if U else "+"} {b}') + if opc == 17: + return VWr(T, Q, d, f'({T})({a} == {b})' if U else f'({T})(({a} & {b}) != 0)') + if sz == 3 and opc != 23: + raise Unsupported('3same size') + if opc == 18: + return VWr(T, Q, d, f'{o} {"-" if U else "+"} {a} * {b}') + if opc == 19: + if U: + if sz: + raise Unsupported('pmul size') + return same(lop('PMUL'), 0, ne, d, n, m) + return VWr(T, Q, d, f'{a} * {b}') + if opc in (20, 21, 23): + if opc == 23 and U: + raise Unsupported('addp U') + ev, od = self._pair_idx(Q, sz) + A, B = (a, b) if (U or opc == 23) else (sa, sb) + L, R = shuf(T, A, B, ev), shuf(T, A, B, od) + if opc == 23: + return VWr(T, Q, d, f'({T})({L} + {R})') + return VWr(T, Q, d, f'({T})__builtin_elementwise_{"max" if opc == 20 else "min"}({L}, {R})') + if opc == 22: + if sz not in (1, 2): + raise Unsupported('sqdmulh size') + return same(lop('SQRDMULH' if U else 'SQDMULH'), sz, ne, d, n, m) + raise Unsupported('3same opcode') + + @staticmethod + def _pair_idx(Q, sz): + """Lane indices (into SHUF(a, b)) of the left / right operands of a pairwise op; for Q=0 the + padding lanes point at a's upper half.""" + ne, NL = lanes(Q, sz), 16 >> sz + src = list(range(ne)) + [NL + i for i in range(ne)] + ev = [src[2 * k] for k in range(ne)] + [ne % NL] * (NL - ne) + od = [src[2 * k + 1] for k in range(ne)] + [ne % NL] * (NL - ne) + return ev, od + + def three_same_scalar(self, U, sz, opc, d, n, m): + if opc in (1, 5): + return same(lop('UQADD' if U else 'SQADD') if opc == 1 else lop('UQSUB' if U else 'SQSUB'), sz, 1, d, n, m) + if opc in (9, 11): + return same(lop('QSHL' if opc == 9 else 'QRSHL', U), sz, 1, d, n, m) + if opc == 22: + if sz not in (1, 2): + raise Unsupported('sqdmulh size') + return same(lop('SQRDMULH' if U else 'SQDMULH'), sz, 1, d, n, m) + if sz != 3: + raise Unsupported('scalar 3same size') + a, b = f'VS64({n})', f'VS64({m})' + sa, sb = f'(int64_t)VS64({n})', f'(int64_t)VS64({m})' + x, y = (a, b) if U else (sa, sb) + if opc == 6: + return f'c->v[{d}] = ({x} > {y}) ? ~0ull : 0;' + if opc == 7: + return f'c->v[{d}] = ({x} >= {y}) ? ~0ull : 0;' + if opc in (8, 10): + return same(lop('SHL' if opc == 8 else 'RSHL', U), 3, 1, d, n, m) + if opc == 16: + return f'c->v[{d}] = (uint64_t)({a} {"-" if U else "+"} {b});' + if opc == 17: + return f'c->v[{d}] = ({a} == {b}) ? ~0ull : 0;' if U else f'c->v[{d}] = ({a} & {b}) ? ~0ull : 0;' + raise Unsupported('scalar 3same opcode') + + def three_same_fp(self, w, scalar): + Q, U, a_, sz, m, opc, n, d = (bits(w, 30, 1), bits(w, 29, 1), bits(w, 23, 1), bits(w, 22, 1), bits(w, 16, 5), + bits(w, 11, 5), bits(w, 5, 5), bits(w, 0, 5)) + if not scalar and sz and not Q: + raise Unsupported('fp 2d Q') + F, VT, VU, T, SF, SU, SET = FT[sz] + key = (U, a_, opc) + if key in FCMP3: + cop = FCMP3[key] + if scalar: + return (f'c->v[{d}] = ({ut(2 + sz)})vcmp_{F}(c, {cop}, ({VT}){{{SF}({n})}}, ' + f'({VT}){{{SF}({m})}})[0];') + return VWr(VU, Q, d, f'vcmp_{F}(c, {cop}, {VQ(VT, Q, n)}, {VQ(VT, Q, m)})') + if scalar: + if key not in ((0, 0, 0b11011), (0, 0, 0b11111), (0, 1, 0b11111), (1, 1, 0b11010)): + raise Unsupported('scalar fp 3same') + fop = FOP3[key] + x = f'neg_{F}({SF}({n}))' if fop in ('FOP_RECPS', 'FOP_RSQRTS') else f'{SF}({n})' + return f'{SET}({d}, op_{F}(c, {fop}, {x}, {SF}({m}), 0));' + A, B = VQ(VT, Q, n), VQ(VT, Q, m) + if key in FPAIRV: + ev, od = self._pair_idx(Q, 2 + sz) + return f'SV_FOP_{F.upper()}({FPAIRV[key]}, {Q}, {d}, {shuf(VT, A, B, ev)}, {shuf(VT, A, B, od)}, ({VT}){{}});' + if U == 0 and opc == 0b11001: # FMLA / FMLS: the multiplicand is negated for FMLS + return f'SV_FOP_{F.upper()}(FOP_MADD, {Q}, {d}, {VQ(VT, Q, d)}, {f"vneg_{F}({A})" if a_ else A}, {B});' + fop = FOP3.get(key) + if fop is None: + raise Unsupported('fp 3same opcode') + if fop in ('FOP_RECPS', 'FOP_RSQRTS'): + A = f'vneg_{F}({A})' + return f'SV_FOP_{F.upper()}({fop}, {Q}, {d}, {A}, {B}, ({VT}){{}});' + + # ------------------------------------------------------------ three different + + def three_diff(self, w, scalar=False): + Q, U, sz, m, opc, n, d = (bits(w, 30, 1), bits(w, 29, 1), bits(w, 22, 2), bits(w, 16, 5), bits(w, 12, 4), + bits(w, 5, 5), bits(w, 0, 5)) + if scalar: + if U or opc not in (9, 11, 13) or sz not in (1, 2): + raise Unsupported('scalar 3diff') + op = {9: 'SQDMLAL', 11: 'SQDMLSL', 13: 'SQDMULL'}[opc] + return gen(lop(op), sz, 1, d, sz + 1, 0, 0, n, sz, 0, m, sz, 0) + if opc == 14: # PMULL + if U: + raise Unsupported('pmull U') + if sz == 3: + return f'sl_pmull64(c, {Q}, {d}, {n}, {m});' + if sz: + raise Unsupported('pmull size') + if sz == 3 or opc == 15: + raise Unsupported('3diff size') + ne = 8 >> sz + off = Q * ne + if opc in (4, 6): # ADDHN/RADDHN/SUBHN/RSUBHN: narrow into the low or high half + op = ('ADDHN', 'RADDHN', 'SUBHN', 'RSUBHN')[(opc == 6) * 2 + U] + return gen(lop(op), sz, ne, d, sz, off, Q, n, sz + 1, 0, m, sz + 1, 0) + if opc in (1, 3): # wide + op = 'ADDW' if opc == 1 else 'SUBW' + return gen(lop(op, U), sz, ne, d, sz + 1, 0, 0, n, sz + 1, 0, m, sz, off) + op = {0: 'ADDL', 2: 'SUBL', 5: 'ABAL', 7: 'ABDL', 8: 'MLAL', 10: 'MLSL', 12: 'MULL', + 9: 'SQDMLAL', 11: 'SQDMLSL', 13: 'SQDMULL', 14: 'PMULL'}[opc] + if op.startswith('SQD'): + if U or sz not in (1, 2): + raise Unsupported('sqdmull') + return gen(lop(op, U if not op.startswith(('SQD', 'PM')) else 0), sz, ne, d, sz + 1, 0, 0, n, sz, off, + m, sz, off) + + # ------------------------------------------------------------ two-register misc + + def misc2(self, w, scalar=False): + Q, U, sz, opc, n, d = bits(w, 30, 1), bits(w, 29, 1), bits(w, 22, 2), bits(w, 12, 5), bits(w, 5, 5), bits(w, 0, 5) + if opc >= 0b11000 or opc in (0b10110, 0b10111) or 0b01100 <= opc <= 0b01111: + return self.misc2_fp(w, scalar) + if scalar: + return self.misc2_scalar(U, sz, opc, d, n) + ne = lanes(Q, sz) + T, S = vu(sz), vs(sz) + if opc == 0 or (opc == 1 and not U): # REV64 / REV32 / REV16 + gsz = {(0, 0): 3, (1, 0): 2, (0, 1): 1}.get((U, opc)) + if gsz is None or sz >= gsz: + raise Unsupported('rev') + es = 1 << sz + gb = 1 << gsz + idx = [] + for i in range(16): + g, o = i // gb * gb, i % gb + e, b = o // es, o % es + idx.append(g + (gb // es - 1 - e) * es + b) + return VWr('vu8', Q, d, shuf('vu8', VR('vu8', n), VR('vu8', n), idx)) + if sz == 3 and not Q and opc not in (0b10010, 0b10100, 0b10011): + raise Unsupported('misc 1d') + if opc in (2, 6): + if sz == 3: + raise Unsupported('addlp size') + return f'sl_addlp(c, {U}, {int(opc == 6)}, {Q}, {sz}, {d}, {n});' + if opc == 3: + return same(lop('USQADD' if U else 'SUQADD'), sz, ne, d, n) + if opc == 4: + if sz == 3: + raise Unsupported('clz size') + return same(lop('CLZ' if U else 'CLS'), sz, ne, d, n) + if opc == 5: + if not U: + if sz: + raise Unsupported('cnt size') + return same(lop('CNT'), 0, ne, d, n) + if sz == 0: + return VWr('vu64', Q, d, f'~{VR("vu64", n)}') + if sz == 1: + return same(lop('RBIT'), 0, lanes(Q, 0), d, n) + raise Unsupported('not/rbit size') + if opc == 7: + return same(lop('SQNEG' if U else 'SQABS'), sz, ne, d, n) + a, sa = VR(T, n), VR(S, n) + if opc == 8: + return VWr(T, Q, d, f'({T})({sa} {">=" if U else ">"} 0)') + if opc == 9: + return VWr(T, Q, d, f'({T})({sa} {"<=" if U else "=="} 0)') + if opc == 10 and not U: + return VWr(T, Q, d, f'({T})({sa} < 0)') + if opc == 11: + if U: + return VWr(T, Q, d, f'-{a}') + return VWr(T, Q, d, f'(({T}){sa} ^ ({T})({sa} >> {(8 << sz) - 1})) - ({T})({sa} >> {(8 << sz) - 1})') + if opc in (0b10010, 0b10100): # XTN / SQXTUN / SQXTN / UQXTN + if sz == 3: + raise Unsupported('xtn size') + op = {(0b10010, 0): 'XTN', (0b10010, 1): 'SQXTUN', (0b10100, 0): 'SQXTN', (0b10100, 1): 'UQXTN'}[(opc, U)] + ne = 8 >> sz + return gen(lop(op), sz, ne, d, sz, Q * ne, Q, n, sz + 1, 0) + if opc == 0b10011 and U: # SHLL + if sz == 3: + raise Unsupported('shll size') + ne = 8 >> sz + return gen(lop('SHLL'), sz, ne, d, sz + 1, 0, 0, n, sz, Q * ne) + raise Unsupported('misc2 opcode') + + def misc2_scalar(self, U, sz, opc, d, n): + if opc == 3: + return same(lop('USQADD' if U else 'SUQADD'), sz, 1, d, n) + if opc == 7: + return same(lop('SQNEG' if U else 'SQABS'), sz, 1, d, n) + if opc in (0b10010, 0b10100): + if sz == 3 or (opc == 0b10010 and not U): + raise Unsupported('scalar xtn') + op = {(0b10010, 1): 'SQXTUN', (0b10100, 0): 'SQXTN', (0b10100, 1): 'UQXTN'}[(opc, U)] + return gen(lop(op), sz, 1, d, sz, 0, 0, n, sz + 1, 0) + if sz != 3: + raise Unsupported('scalar misc size') + a, sa = f'VS64({n})', f'(int64_t)VS64({n})' + if opc == 8: + return f'c->v[{d}] = ({sa} {">=" if U else ">"} 0) ? ~0ull : 0;' + if opc == 9: + return f'c->v[{d}] = ({sa} {"<=" if U else "=="} 0) ? ~0ull : 0;' + if opc == 10 and not U: + return f'c->v[{d}] = ({sa} < 0) ? ~0ull : 0;' + if opc == 11: + return f'c->v[{d}] = (uint64_t)-{a};' if U else f'c->v[{d}] = ({sa} < 0) ? (uint64_t)-{a} : {a};' + raise Unsupported('scalar misc opcode') + + def misc2_fp(self, w, scalar): + """FP two-register misc, vector and scalar.""" + Q, U, a_, sz, opc, n, d = (bits(w, 30, 1), bits(w, 29, 1), bits(w, 23, 1), bits(w, 22, 1), bits(w, 12, 5), + bits(w, 5, 5), bits(w, 0, 5)) + F, VT, VU, T, SF, SU, SET = FT[sz] + if opc in (0b10110, 0b10111): # FCVTN / FCVTL / FCVTXN + if a_: + raise Unsupported('fcvtn a') + if opc == 0b10110 and U: + if not sz: + raise Unsupported('fcvtxn size') + if scalar: + return f'FP_SETS({d}, simd_d2s_odd(c, VSF64({n})));' + return f'sv_fcvtn(c, 1, {Q}, 1, {d}, {n});' + if scalar or U: + raise Unsupported('fcvtn/l') + if opc == 0b10110: + return f'sv_fcvtn(c, {sz}, {Q}, 0, {d}, {n});' + return f'sv_fcvtl(c, {sz}, {Q}, {d}, {n});' + if not scalar and sz and not Q: + raise Unsupported('fp misc 2d Q') + x = VQ(VT, Q, n) + if 0b01100 <= opc <= 0b01111: + if not a_: + raise Unsupported('fp16 misc') + if opc == 0b01111: # FABS / FNEG + if scalar: + raise Unsupported('scalar fabs') + mask = '0x80000000u' if not sz else '0x8000000000000000ull' + return VWr(VU, Q, d, f'{VR(VU, n)} ^ {mask}' if U else f'{VR(VU, n)} & ~{mask}') + cop = {(0b01100, 0): (2, 0), (0b01100, 1): (1, 0), (0b01101, 0): (0, 0), (0b01101, 1): (1, 1), + (0b01110, 0): (2, 1)}.get((opc, U)) + if cop is None: + raise Unsupported('fcm zero') + op, swap = cop + if scalar: + a, b = f'({VT}){{{SF}({n})}}', f'({VT}){{}}' + if swap: + a, b = b, a + return f'c->v[{d}] = ({ut(2 + sz)})vcmp_{F}(c, {op}, {a}, {b})[0];' + a, b = x, f'({VT}){{}}' + if swap: + a, b = b, a + return VWr(VU, Q, d, f'vcmp_{F}(c, {op}, {a}, {b})') + if opc == 0b11111: + if not a_: + raise Unsupported('fp misc 11111') + if U: + if scalar: + raise Unsupported('scalar fsqrt') + return f'SV_FOP_{F.upper()}(FOP_SQRT, {Q}, {d}, {x}, {x}, ({VT}){{}});' + if not scalar: + raise Unsupported('vector frecpx') + return f'{SET}({d}, recpx_{F}(c, {SF}({n})));' + if opc in (0b11000, 0b11001): # FRINT* + if scalar: + raise Unsupported('scalar frint') + rm = {(0, 0, 0): 0, (0, 0, 1): 2, (0, 1, 0): 1, (0, 1, 1): 3, (1, 0, 0): 4, (1, 0, 1): 5, (1, 1, 1): 6}.get( + (U, a_, opc & 1)) + if rm is None: + raise Unsupported('frint') + return VWr(VT, Q, d, f'vfrint_{F}(c, {x}, {rm})') + if opc in (0b11010, 0b11011) or (opc == 0b11100 and not a_): # FCVT[NPMZA][SU] + rm = {(0b11010, 0): 0, (0b11010, 1): 1, (0b11011, 0): 2, (0b11011, 1): 3, (0b11100, 0): 4}[(opc, a_)] + if scalar: + return f'c->v[{d}] = ({ut(2 + sz)})toint_{F}(c, {SF}({n}), {rm}, 0, {32 << sz}, {U});' + return VWr(VU, Q, d, f'vtoint_{F}(c, {x}, {rm}, 0, {U})') + if opc == 0b11100: # URECPE / URSQRTE + if scalar or sz: + raise Unsupported('urecpe') + return same(lop('URSQRTE' if U else 'URECPE'), 2, lanes(Q, 2), d, n) + if opc == 0b11101: + if not a_: # SCVTF / UCVTF + if scalar: + return f'{SET}({d}, fromint_{F}(c, {SU}({n}), {32 << sz}, {U}, 0));' + return VWr(VT, Q, d, f'vfromint_{F}(c, {VR(VU, n)}, 0, {U})') + if scalar: # FRECPE / FRSQRTE + return f'{SET}({d}, {"rsqrte" if U else "recpe"}_{F}(c, {SF}({n})));' + return VWr(VT, Q, d, f'v{"rsqrte" if U else "recpe"}_{F}(c, {x})') + raise Unsupported('fp misc opcode') + + # ------------------------------------------------------------ across lanes / pairwise + + def across(self, w): + Q, U, sz, opc, n, d = bits(w, 30, 1), bits(w, 29, 1), bits(w, 22, 2), bits(w, 12, 5), bits(w, 5, 5), bits(w, 0, 5) + if opc in (0b01100, 0b01111): + if not U: + raise Unsupported('fp16 across') + if not Q or (sz & 1): + raise Unsupported('fp across size') + fop = {(0b01100, 0): 'FOP_MAXNM', (0b01100, 1): 'FOP_MINNM', (0b01111, 0): 'FOP_MAX', + (0b01111, 1): 'FOP_MIN'}[(opc, sz >> 1)] + return f'sv_fred_s(c, {fop}, {d}, {n});' + op = {0b00011: 1, 0b01010: 2, 0b11010: 3, 0b11011: 0}.get(opc) + if op is None or (op == 0 and U) or sz == 3 or (sz == 2 and not Q): + raise Unsupported('across') + return f'sl_across(c, {op}, {U}, {Q}, {sz}, {d}, {n});' + + def scalar_pairwise(self, w): + U, sz, opc, n, d = bits(w, 29, 1), bits(w, 22, 2), bits(w, 12, 5), bits(w, 5, 5), bits(w, 0, 5) + if not U: + if opc == 0b11011 and sz == 3: + return f'c->v[{d}] = (uint64_t)(VS64({n}) + VHI64({n}));' + raise Unsupported('scalar pairwise') + fop = {(0b01100, 0): 'FOP_MAXNM', (0b01100, 1): 'FOP_MINNM', (0b01101, 0): 'FOP_ADD', + (0b01111, 0): 'FOP_MAX', (0b01111, 1): 'FOP_MIN'}.get((opc, sz >> 1)) + if fop is None: + raise Unsupported('scalar fp pairwise') + F, VT, VU, T, SF, SU, SET = FT[sz & 1] + return f'{SET}({d}, pair_{F}(c, {fop}, {VR(VT, n)}[0], {VR(VT, n)}[1]));' + + # ------------------------------------------------------------ modified immediate + + def modimm(self, w): + Q, op, cmode, o2, d = bits(w, 30, 1), bits(w, 29, 1), bits(w, 12, 4), bits(w, 11, 1), bits(w, 0, 5) + imm8 = (bits(w, 16, 3) << 5) | bits(w, 5, 5) + if o2: + raise Unsupported('fmov half imm') + if cmode == 15 and op and not Q: + raise Unsupported('fmov 1d') + imm = simd_expand_imm(op, cmode, imm8) + c4 = cmode + if (c4 < 8 and c4 & 1) or (8 <= c4 < 12 and c4 & 1): # ORR / BIC + if op: + imm = ~imm & ((1 << 64) - 1) + return VWr('vu64', Q, d, f'{VR("vu64", d)} & {hx(imm)}') + return VWr('vu64', Q, d, f'{VR("vu64", d)} | {hx(imm)}') + if op and not (c4 == 14 or c4 == 15): # MVNI + imm = ~imm & ((1 << 64) - 1) + v = imm | (imm << 64) if Q else imm + return f'c->v[{d}] = {u128(v)};' + + # ------------------------------------------------------------ shift by immediate + + def shift_imm(self, w, scalar=False): + Q, U, immh, immb, opc, n, d = (bits(w, 30, 1), bits(w, 29, 1), bits(w, 19, 4), bits(w, 16, 3), bits(w, 11, 5), + bits(w, 5, 5), bits(w, 0, 5)) + if not immh: + raise Unsupported('shift immh') + sz = immh.bit_length() - 1 + es = 8 << sz + ih = (immh << 3) | immb + rsh, lsh = 2 * es - ih, ih - es + if scalar: + Q = 1 + ne = 1 if scalar else lanes(Q, sz) + narrow = 0b10000 <= opc <= 0b10011 + if opc in (0b11100, 0b11111): # fixed-point conversions + if sz < 2: + raise Unsupported('fp16 fixed') + if sz == 3 and not Q: + raise Unsupported('fixed 1d') + F, VT, VU, T, SF, SU, SET = FT[sz - 2] + fb = rsh + if opc == 0b11100: + if scalar: + return f'{SET}({d}, fromint_{F}(c, {SU}({n}), {es}, {U}, {fb}));' + return VWr(VT, Q, d, f'vfromint_{F}(c, {VR(VU, n)}, {fb}, {U})') + if scalar: + return f'c->v[{d}] = ({ut(sz)})toint_{F}(c, {SF}({n}), 3, {fb}, {es}, {U});' + return VWr(VU, Q, d, f'vtoint_{F}(c, {VQ(VT, Q, n)}, 3, {fb}, {U})') + if narrow: + if sz == 3: + raise Unsupported('narrow shift size') + ne = 1 if scalar else 8 >> sz + op = {(0b10000, 0): ('SHRN', 0), (0b10001, 0): ('RSHRN', 0), (0b10000, 1): ('QSHRUN', 0), + (0b10001, 1): ('QRSHRUN', 0), (0b10010, 0): ('QSHRN', 0), (0b10010, 1): ('QSHRN', 1), + (0b10011, 0): ('QRSHRN', 0), (0b10011, 1): ('QRSHRN', 1)}[(opc, U)] + if scalar and op[0] in ('SHRN', 'RSHRN'): + raise Unsupported('scalar shrn') + part = 0 if scalar else Q + return gen(lop(*op), sz, ne, d, sz, part * ne, part, n, sz + 1, 0, -1, 0, 0, -1, rsh) + if opc == 0b10100: # SSHLL / USHLL + if scalar or sz == 3: + raise Unsupported('sshll') + ne = 8 >> sz + return gen(lop('SHLLI', U), sz, ne, d, sz + 1, 0, 0, n, sz, Q * ne, -1, 0, 0, -1, lsh) + if sz == 3 and not Q: + raise Unsupported('shift 1d') + if scalar and sz != 3 and opc not in (0b01100, 0b01110): + raise Unsupported('scalar shift size') + T, S = vu(sz), vs(sz) + if opc in (0, 2): # SSHR/USHR, SSRA/USRA + if U: + e = f'({T}){{}}' if rsh >= es else f'({VR(T, n)} >> {rsh})' + else: + e = f'({T})({VR(S, n)} >> {min(rsh, es - 1)})' + if opc == 2: + e = f'{VR(T, d)} + {e}' + return VWr(T, Q, d, e) if not scalar else f'c->v[{d}] = (uint64_t)(({e})[0]);' + if opc in (4, 6): + return same(lop('RSHRI' if opc == 4 else 'RSRAI', U), sz, ne, d, n, -1, rsh) + if opc == 8: + if not U: + raise Unsupported('sri U') + return same(lop('SRI'), sz, ne, d, n, -1, rsh) + if opc == 10: + if U: + return same(lop('SLI'), sz, ne, d, n, -1, lsh) + e = f'{VR(T, n)} << {lsh}' + return VWr(T, Q, d, e) if not scalar else f'c->v[{d}] = (uint64_t)(({e})[0]);' + if opc == 12: + if not U: + raise Unsupported('sqshlu U') + return same(lop('QSHLUI'), sz, ne, d, n, -1, lsh) + if opc == 14: + return same(lop('QSHLI', U), sz, ne, d, n, -1, lsh) + raise Unsupported('shift imm opcode') + + # ------------------------------------------------------------ indexed element + + def indexed(self, w, scalar=False): + Q, U, sz, L, M, rm, opc, H, n, d = (bits(w, 30, 1), bits(w, 29, 1), bits(w, 22, 2), bits(w, 21, 1), + bits(w, 20, 1), bits(w, 16, 4), bits(w, 12, 4), bits(w, 11, 1), + bits(w, 5, 5), bits(w, 0, 5)) + fp = opc in (1, 5, 9) and not (U and opc != 9) + if fp: + if sz < 2: + raise Unsupported('fp16 indexed') + fsz = sz & 1 + if fsz: + if L: + raise Unsupported('indexed L') + idx = H + else: + idx = (H << 1) | L + m = (M << 4) | rm + if not scalar and fsz and not Q: + raise Unsupported('indexed 2d Q') + F, VT, VU, T, SF, SU, SET = FT[fsz] + if scalar: + el = f'{VR(VT, m)}[{idx}]' + if opc == 9: + return f'{SET}({d}, op_{F}(c, {"FOP_MULX" if U else "FOP_MUL"}, {SF}({n}), {el}, 0));' + x = f'{SF}({n})' if opc == 1 else f'neg_{F}({SF}({n}))' + return f'{SET}({d}, op_{F}(c, FOP_MADD, {SF}({d}), {x}, {el}));' + el = f'vsplat_{F}({VR(VT, m)}, {idx}, {Q})' + if opc == 9: + return f'SV_FOP_{F.upper()}({"FOP_MULX" if U else "FOP_MUL"}, {Q}, {d}, {VQ(VT, Q, n)}, {el}, ({VT}){{}});' + x = VQ(VT, Q, n) if opc == 1 else f'vneg_{F}({VQ(VT, Q, n)})' + return f'SV_FOP_{F.upper()}(FOP_MADD, {Q}, {d}, {VQ(VT, Q, d)}, {x}, {el});' + if sz == 1: + idx, m = (H << 2) | (L << 1) | M, rm + elif sz == 2: + idx, m = (H << 1) | L, (M << 4) | rm + else: + raise Unsupported('indexed size') + T = vu(sz) + if scalar: + op = {(0, 3): 'SQDMLAL', (0, 7): 'SQDMLSL', (0, 11): 'SQDMULL', (0, 12): 'SQDMULH', + (0, 13): 'SQRDMULH'}.get((U, opc)) + if op is None: + raise Unsupported('scalar indexed') + if op in ('SQDMULH', 'SQRDMULH'): + return gen(lop(op), sz, 1, d, sz, 0, 0, n, sz, 0, m, sz, 0, idx) + return gen(lop(op), sz, 1, d, sz + 1, 0, 0, n, sz, 0, m, sz, 0, idx) + ne = lanes(Q, sz) + el = shuf(T, VR(T, m), VR(T, m), [idx] * (16 >> sz)) + if (U, opc) == (0, 8): + return VWr(T, Q, d, f'{VR(T, n)} * {el}') + if (U, opc) in ((1, 0), (1, 4)): + return VWr(T, Q, d, f'{VR(T, d)} {"+" if opc == 0 else "-"} {VR(T, n)} * {el}') + if (U, opc) in ((0, 12), (0, 13)): + return same(lop('SQDMULH' if opc == 12 else 'SQRDMULH'), sz, ne, d, n, m, 0, idx) + lng = {2: 'MLAL', 6: 'MLSL', 10: 'MULL', 3: 'SQDMLAL', 7: 'SQDMLSL', 11: 'SQDMULL'}.get(opc) + if lng is None or (U and lng.startswith('SQ')): + raise Unsupported('indexed opcode') + ne = 8 >> sz + return gen(lop(lng, U if not lng.startswith('SQ') else 0), sz, ne, d, sz + 1, 0, 0, n, sz, Q * ne, m, sz, 0, idx) + + # ------------------------------------------------------------ scalar FP + + @staticmethod + def _ftype(w): + t = bits(w, 22, 2) + if t > 1: + raise Unsupported('fp type') + return t + + def fp_dp3(self, w): + t = self._ftype(w) + o1, o0 = bits(w, 21, 1), bits(w, 15, 1) + d, n, m, a = bits(w, 0, 5), bits(w, 5, 5), bits(w, 16, 5), bits(w, 10, 5) + F = FT[t][0] + na, nn = o1, o1 ^ o0 # FMADD 00, FMSUB 01 (-n), FNMADD 10 (-a, -n), FNMSUB 11 (-a) + return f'fp_fma_{F}(c, {na}, {nn}, {d}, {n}, {m}, {a});' + + def fp_dp2(self, w): + t = self._ftype(w) + opc, d, n, m = bits(w, 12, 4), bits(w, 0, 5), bits(w, 5, 5), bits(w, 16, 5) + fop = ('FOP_MUL', 'FOP_DIV', 'FOP_ADD', 'FOP_SUB', 'FOP_MAX', 'FOP_MIN', 'FOP_MAXNM', 'FOP_MINNM', + 'FOP_NMUL')[opc] if opc <= 8 else None + if fop is None: + raise Unsupported('fp dp2') + return f'fp_op_{FT[t][0]}(c, {fop}, {d}, {n}, {m});' + + def fp_dp1(self, w): + t = bits(w, 22, 2) + opc, d, n = bits(w, 15, 6), bits(w, 0, 5), bits(w, 5, 5) + if t == 3: # half source: only FCVT + if opc == 4: + return f'FP_SETS({d}, simd_h2s(c, VS16({n})));' + if opc == 5: + return f'FP_SETD({d}, simd_h2d(c, VS16({n})));' + raise Unsupported('fp16 dp1') + if t == 2: + raise Unsupported('fp type') + F, VT, VU, T, SF, SU, SET = FT[t] + sign = '0x80000000u' if not t else '0x8000000000000000ull' + if opc == 0: + return f'c->v[{d}] = {SU}({n});' + if opc == 1: + return f'c->v[{d}] = {SU}({n}) & ~{sign};' + if opc == 2: + return f'c->v[{d}] = {SU}({n}) ^ {sign};' + if opc == 3: + return f'fp_op_{F}(c, FOP_SQRT, {d}, {n}, {n});' + if opc in (4, 5, 7): + to = {4: 0, 5: 1, 7: 3}[opc] + if to == t: + raise Unsupported('fcvt same') + if to == 3: + return f'c->v[{d}] = {"simd_s2h" if not t else "simd_dd2h"}(c, {SF}({n}));' + return f'FP_SETD({d}, simd_s2d(c, VSF32({n})));' if to else f'FP_SETS({d}, simd_d2s(c, VSF64({n})));' + rm = {8: 0, 9: 1, 10: 2, 11: 3, 12: 4, 14: 5, 15: 6}.get(opc) + if rm is None: + raise Unsupported('fp dp1 opcode') + return f'{SET}({d}, frint_{F}(c, {SF}({n}), {rm}));' + + def fp_cmp(self, w): + t = self._ftype(w) + if bits(w, 14, 2) or bits(w, 0, 3): + raise Unsupported('fcmp op') + F, VT, VU, T, SF, SU, SET = FT[t] + n, m, zero = bits(w, 5, 5), bits(w, 16, 5), bits(w, 3, 1) + b = '0' if zero else f'{SF}({m})' + return f'c->nzcv = cmp_{F}(c, {SF}({n}), {b});' + + def fp_ccmp(self, w): + t = self._ftype(w) + F, VT, VU, T, SF, SU, SET = FT[t] + n, m, cnd, nzcv = bits(w, 5, 5), bits(w, 16, 5), bits(w, 12, 4), bits(w, 0, 4) + return f'c->nzcv = {cond(cnd)} ? cmp_{F}(c, {SF}({n}), {SF}({m})) : {hx(nzcv << 28)};' + + def fp_csel(self, w): + t = self._ftype(w) + SU = FT[t][5] + d, n, m, cnd = bits(w, 0, 5), bits(w, 5, 5), bits(w, 16, 5), bits(w, 12, 4) + return f'c->v[{d}] = {cond(cnd)} ? {SU}({n}) : {SU}({m});' + + def fp_imm(self, w): + t = self._ftype(w) + if bits(w, 5, 5): + raise Unsupported('fmov imm5') + imm8, d = bits(w, 13, 8), bits(w, 0, 5) + return f'c->v[{d}] = {hx(vfp_expand_imm(imm8, 32 << t))};' + + def fp_fixed(self, w): + sf, t, rmode, opc, scale, n, d = (bits(w, 31, 1), bits(w, 22, 2), bits(w, 19, 2), bits(w, 16, 3), bits(w, 10, 6), + bits(w, 5, 5), bits(w, 0, 5)) + if t > 1: + raise Unsupported('fixed type') + if not sf and scale < 32: + raise Unsupported('fixed scale') + F, VT, VU, T, SF, SU, SET = FT[t] + fb, ib = 64 - scale, 64 if sf else 32 + if rmode == 0 and opc in (2, 3): + src = X(n) if sf else W(n) + return f'{SET}({d}, fromint_{F}(c, {src}, {ib}, {opc & 1}, {fb}));' + if rmode == 3 and opc in (0, 1): + return set_reg(sf, d, f'toint_{F}(c, {SF}({n}), 3, {fb}, {ib}, {opc})') + raise Unsupported('fixed opcode') + + def fp_int(self, w): + sf, t, rmode, opc, n, d = bits(w, 31, 1), bits(w, 22, 2), bits(w, 19, 2), bits(w, 16, 3), bits(w, 5, 5), bits(w, 0, 5) + if opc in (6, 7): # FMOV + if t == 2 and sf and rmode == 1: + return set_reg(1, d, f'VHI64({n})') if opc == 6 else f'VHI64({d}) = {X(n)};' + if rmode or t != sf or t > 1: + raise Unsupported('fmov general') + if opc == 6: + return set_reg(sf, d, f'VS64({n})' if sf else f'VS32({n})') + return f'c->v[{d}] = {X(n) if sf else W(n)};' + if t > 1: + raise Unsupported('fp int type') + F, VT, VU, T, SF, SU, SET = FT[t] + ib = 64 if sf else 32 + if opc in (2, 3): + if rmode: + raise Unsupported('cvtf rmode') + return f'{SET}({d}, fromint_{F}(c, {X(n) if sf else W(n)}, {ib}, {opc & 1}, 0));' + if opc in (4, 5): + if rmode: + raise Unsupported('fcvta rmode') + rm = 4 + else: + rm = (0, 1, 2, 3)[rmode] + return set_reg(sf, d, f'toint_{F}(c, {SF}({n}), {rm}, 0, {ib}, {opc & 1})') + + # ------------------------------------------------------------ loads / stores + + def ldst(self, w, pc): + rt = bits(w, 0, 5) + if bits(w, 27, 3) == 0b011 and bits(w, 24, 2) == 0: # literal + opc = bits(w, 30, 2) + if opc == 3: + raise Unsupported('ldr literal opc') + addr = (pc + sext(bits(w, 5, 19), 19) * 4) & ((1 << 64) - 1) + return f'c->v[{rt}] = {("LD32", "LD64", "LD128")[opc]}({hx(addr)});' + if bits(w, 31, 1) == 0 and bits(w, 24, 6) in (0b001100, 0b001101): + return self.ldst_struct(w) + if bits(w, 27, 3) == 0b101: + return self.ldst_pair(w) + if bits(w, 27, 3) == 0b111: + return self.ldst_reg(w) + raise Unsupported(f'ldst simd {w:08x}') + + def ldst_pair(self, w): + opc, mode, L = bits(w, 30, 2), bits(w, 23, 2), bits(w, 22, 1) + imm7, rt2, rn, rt = sext(bits(w, 15, 7), 7), bits(w, 10, 5), bits(w, 5, 5), bits(w, 0, 5) + if opc == 3: + raise Unsupported('pair opc 3') + scale = 2 + opc + off = imm7 << scale + base = XS(rn) + post, wb = mode == 1, mode in (1, 3) + ea = base if post else (f'{base} + {off}' if off >= 0 else f'{base} - {-off}') + ld, st = {2: ('LD32', 'ST32'), 3: ('LD64', 'ST64'), 4: ('LD128', 'ST128')}[scale] + sz = 1 << scale + lines = [f'uint64_t ea = {ea};'] + if L: + lines.append(f'u128 a = {ld}(ea), b = {ld}(ea + {sz});') + lines.append(f'c->v[{rt}] = a; c->v[{rt2}] = b;') + else: + cast = {2: 'VS32', 3: 'VS64', 4: 'c->v'}[scale] + v1, v2 = (f'c->v[{rt}]', f'c->v[{rt2}]') if scale == 4 else (f'{cast}({rt})', f'{cast}({rt2})') + lines.append(f'{st}(ea, {v1}); {st}(ea + {sz}, {v2});') + if wb: + dst = 'c->sp' if rn == 31 else f'c->x[{rn}]' + if not post: + lines.append(f'{dst} = ea;') + else: + lines.append(f'{dst} = ea + {off};' if off >= 0 else f'{dst} = ea - {-off};') + return '{ ' + ' '.join(lines) + ' }' + + def ldst_reg(self, w): + size, opc, rn, rt = bits(w, 30, 2), bits(w, 22, 2), bits(w, 5, 5), bits(w, 0, 5) + if opc & 2: + if size: + raise Unsupported('ldst simd size') + scale = 4 + else: + scale = size + L = opc & 1 + base = XS(rn) + wb = None + if bits(w, 24, 2) == 1: + ea = f'{base} + {hx(bits(w, 10, 12) << scale)}' + elif bits(w, 24, 2) == 0: + kind = bits(w, 10, 2) + if bits(w, 21, 1): + if kind != 2: + raise Unsupported('ldst simd reg kind') + rm, option, S = bits(w, 16, 5), bits(w, 13, 3), bits(w, 12, 1) + if option not in (2, 3, 6, 7): + raise Unsupported('ldst simd reg option') + ea = f'{base} + {extend_expr(rm, option, scale if S else 0)}' + else: + imm9 = sext(bits(w, 12, 9), 9) + si = f'+ {imm9}' if imm9 >= 0 else f'- {-imm9}' + if kind == 0: + ea = f'{base} {si}' + elif kind == 1: + ea, wb = base, f'{base} {si}' + elif kind == 3: + ea, wb = f'{base} {si}', 'ea' + else: + raise Unsupported('ldst simd unprivileged') + else: + raise Unsupported('ldst simd') + parts = [f'uint64_t ea = {ea};'] + if L: + parts.append(f'c->v[{rt}] = {("LD8", "LD16", "LD32", "LD64", "LD128")[scale]}(ea);') + else: + src = (f'VS8({rt})', f'VS16({rt})', f'VS32({rt})', f'VS64({rt})', f'c->v[{rt}]')[scale] + parts.append(f'{("ST8", "ST16", "ST32", "ST64", "ST128")[scale]}(ea, {src});') + if wb: + parts.append(f'{"c->sp" if rn == 31 else f"c->x[{rn}]"} = {wb};') + return '{ ' + ' '.join(parts) + ' }' + + def ldst_struct(self, w): + Q, L, post, rm, rn, rt, sz = (bits(w, 30, 1), bits(w, 22, 1), bits(w, 23, 1), bits(w, 16, 5), bits(w, 5, 5), + bits(w, 0, 5), bits(w, 10, 2)) + if not post and rm: + raise Unsupported('ldst struct rm') + base = XS(rn) + if bits(w, 24, 1) == 0: # multiple structures + if bits(w, 21, 1): + raise Unsupported('ldst multiple bit21') + opc = bits(w, 12, 4) + sel = {0: (4, 1), 2: (1, 4), 4: (3, 1), 6: (1, 3), 7: (1, 1), 8: (2, 1), 10: (1, 2)}.get(opc) + if sel is None: + raise Unsupported('ldst multiple opcode') + selem, rpt = sel + if sz == 3 and not Q and selem > 1: + raise Unsupported('ldst multiple 1d') + total = (8 << Q) * selem * rpt + if selem == 1: # LD1/ST1: whole registers + regs = [(rt + i) % 32 for i in range(rpt)] + step = 16 if Q else 8 + if L: + acc = ' '.join(f'c->v[{r}] = {"LD128" if Q else "LD64"}(ea + {i * step});' for i, r in enumerate(regs)) + else: + acc = ' '.join(f'{"ST128" if Q else "ST64"}(ea + {i * step}, {"c->v[%d]" % r if Q else "VS64(%d)" % r});' + for i, r in enumerate(regs)) + else: + acc = f'sl_{"ldm" if L else "stm"}(c, {Q}, {sz}, {selem}, {rpt}, {rt}, ea);' + else: # single structure + opc, S, R = bits(w, 13, 3), bits(w, 12, 1), bits(w, 21, 1) + scale = opc >> 1 + selem = ((opc & 1) << 1 | R) + 1 + if scale == 3: + if not L or S: + raise Unsupported('ldnr') + total = selem << sz + acc = f'sl_ldr(c, {Q}, {sz}, {selem}, {rt}, ea);' + else: + if scale == 0: + idx = (Q << 3) | (S << 2) | sz + elif scale == 1: + if sz & 1: + raise Unsupported('ld1 h size') + idx = (Q << 2) | (S << 1) | (sz >> 1) + else: + if sz & 2: + raise Unsupported('ld1 s size') + if sz & 1: + if S: + raise Unsupported('ld1 d S') + idx, scale = Q, 3 + else: + idx = (Q << 1) | S + total = selem << scale + acc = f'sl_{"lds" if L else "sts"}(c, {scale}, {selem}, {idx}, {rt}, ea);' + parts = [f'uint64_t ea = {base};', acc] + if post: + dst = 'c->sp' if rn == 31 else f'c->x[{rn}]' + parts.append(f'{dst} = ea + {total};' if rm == 31 else f'{dst} = ea + {X(rm)};') + return '{ ' + ' '.join(parts) + ' }' + + # ------------------------------------------------------------ CRC32 + + @staticmethod + def crc(w): + sf, rm, C, sz, rn, rd = bits(w, 31, 1), bits(w, 16, 5), bits(w, 12, 1), bits(w, 10, 2), bits(w, 5, 5), bits(w, 0, 5) + if (sz == 3) != bool(sf): + raise Unsupported('crc size') + val = X(rm) if sz == 3 else W(rm) + return set_reg(0, rd, f'{"CRC32C" if C else "CRC32"}({W(rn)}, {val}, {1 << sz})') + diff --git a/tools/make_package.py b/tools/make_package.py new file mode 100644 index 0000000..b201bca --- /dev/null +++ b/tools/make_package.py @@ -0,0 +1,243 @@ +"""Make the builder package for a release: dist/UltraTears-builder--windows-x86_64.zip. + +usage: python tools/make_package.py [--home ../release-test] [--python ] [--no-zip] +The version is the tag v on HEAD (tools/version.py): tag the release commit first. +Run on the developer's machine after a complete build in the game folder --home (tools/deploy.py: its build/build.ninja +and objects tell which parts of clang64 the build reads) and tools/build_xlate.py (dev-local/build/shaders): + - the source: the public tree of HEAD (tools/public_tree.py: what git tracks, without the internal notes), so nothing + made from the game can slip in; + - deps/clang64: the part of MSYS2 CLANG64 the build uses, found by measuring, not guessing: the headers ninja + recorded for every compiled file (and those of tools/bc7build), the libraries and objects lld reads when it links + UltraTears.exe and bc7build.exe (its --verbose "Reading" lines), the programs (clang, clang++, lld, llvm-windres, ninja) and every DLL + they load (llvm-objdump -p, followed recursively), and the licenses of the MSYS2 packages these come from; + - deps/python: Python 3.14 (the given install, default the one running this) without its tests, docs and Tk, with + the packages the build imports (lz4, capstone, unicorn) from this project's .venv; + - deps/shaders/ultrs-shaders.exe, the shader translator (tools/build_xlate.py); + - setup.bat, VERSION.txt and THIRD-PARTY-NOTICES.md. +Refuses a dirty working tree: the package must be the commit it names. +""" +import argparse +import io +import os +import pathlib +import re +import shutil +import subprocess +import sys +import tarfile +import zipfile + +ROOT = pathlib.Path(__file__).resolve().parent.parent +sys.path.insert(0, str(ROOT / 'tools')) +from toolchain import CLANG64, env as tool_env # noqa: E402 +import public_tree # noqa: E402 +import version # noqa: E402 + +PREFIX = CLANG64.parent if CLANG64 else None # .../clang64 +PROGRAMS = ['clang.exe', 'clang++.exe', 'ld.lld.exe', 'lld.exe', 'llvm-windres.exe', 'llvm-rc.exe', 'ninja.exe'] +# MSYS2 package (share/licenses/) of each part of clang64 the package carries +LICENSES = ['clang', 'clang-libs', 'compiler-rt', 'crt', 'headers', 'libc++', 'libunwind', 'libwinpthread', 'winpthreads', + 'lld', 'llvm', 'llvm-libs', 'llvm-tools', 'ninja', 'SDL3', 'opus', 'libvpx', 'zstd', 'libiconv', 'zlib', 'xz', + 'libffi', 'libxml2', 'vulkan-headers'] +PY_PACKAGES = ['lz4', 'capstone', 'unicorn'] +PY_SKIP = {'test', 'idlelib', 'tkinter', 'turtledemo', 'ensurepip', 'lib2to3', '__pycache__'} + +SETUP_BAT = r'''@echo off +rem UltraTears: build the game from your own dump. Put your extracted exefs and romfs folders into game\ here and +rem run this file (or drag a dump folder kept elsewhere onto it). Run it again to resume, or after an update. +cd /d "%~dp0" +if "%~1"=="" ( + "deps\python\python.exe" tools\setup.py +) else ( + "deps\python\python.exe" tools\setup.py --dump "%~1" +) +echo. +pause +''' + + +GAME_TXT = '''Put your own extracted copy of The Legend of Zelda: Tears of the Kingdom, update 1.4.3 applied, here: + + game\\exefs\\ main, subsdk0, sdk, main.npdm, ... + game\\romfs\\ Actor\\, Pack\\, Shader\\, ... (about 304,000 files, 16 GB) + +then run setup.bat. After the build the game reads data\\romfs, so game\\romfs may be deleted. README.md has the details. + +Положите сюда свою извлечённую копию The Legend of Zelda: Tears of the Kingdom с обновлением 1.4.3: + + game\\exefs\\ main, subsdk0, sdk, main.npdm, ... + game\\romfs\\ Actor\\, Pack\\, Shader\\, ... (около 304 000 файлов, 16 ГБ) + +и запустите setup.bat. После сборки игра читает data\\romfs, папку game\\romfs можно удалить. Подробнее — README.ru.md. +''' + + +NOTICES = '''# Third-party notices (builder package) + +UltraTears itself is GPL-3.0-or-later (LICENSE). Its source's third-party code is listed in third_party/README.md. +This package also carries tools to build it; each keeps its license, copied into licenses/: + +| Part | What | License | Text | +|---|---|---|---| +| deps/clang64 | Parts of MSYS2 CLANG64: LLVM clang, lld, libc++, libunwind, compiler-rt (Apache-2.0 WITH LLVM-exception); mingw-w64 headers, CRT and winpthreads (ZPL / MIT / public domain); ninja (Apache-2.0); SDL3 (zlib); opus, libvpx, zstd (BSD); libiconv (LGPL-2.1+, linked statically: the objects to relink are made on your PC by the build itself); zlib, xz, libffi, libxml2 (their DLLs, loaded by clang); Vulkan headers (Apache-2.0 / MIT) | as listed | licenses/msys2// | +| deps/python | Python 3.14 (PSF License) with lz4 (BSD), capstone (BSD), unicorn (GPL-2.0) | as listed | licenses/Python.txt; each package's *.dist-info in deps/python/Lib/site-packages | +| deps/shaders | ultrs-shaders: totk_xlate (MacNativeGames, MIT) around the Ryujinx 1.3.2 shader translator (MIT), published self-contained with the .NET runtime (MIT) | MIT | licenses/Ryujinx.txt, third_party/shader_translator/LICENSE | + +Sources: MSYS2 packages https://packages.msys2.org (CLANG64), Python https://www.python.org, Ryujinx +https://github.com/Ryubing, MacNativeGames (third_party/README.md). +''' + + +def git(*args): + return subprocess.run(['git', '-C', str(ROOT)] + list(args), check=True, capture_output=True, text=True).stdout + + +def clang64_files_used(home): + """Headers, libraries and objects of clang64 that the build reads, measured.""" + used = set() + deps = subprocess.run([str(CLANG64 / 'ninja.exe'), '-C', str(home), '-f', 'build/build.ninja', '-t', 'deps'], + capture_output=True, text=True, env=tool_env()).stdout + # the game folder's build may have used its own deps/clang64 (a previous package): the same files of PREFIX + prefixes = [str(PREFIX).replace('\\', '/').lower(), str(home / 'deps/clang64').replace('\\', '/').lower()] + for line in deps.splitlines(): + p = line.strip().replace('\\', '/') + for pre in prefixes: + if p.lower().startswith(pre + '/'): + used.add(PREFIX / p[len(pre) + 1:]) + tmp = ROOT / 'dev-local/build/package-probe' + tmp.mkdir(parents=True, exist_ok=True) + for src, cxx in (('tools/bc7build/bc7build.c', False), ('src/gfx/tex/astc.c', False), ('src/gfx/tex/bc7.cpp', True), + ('src/gfx/tex/bc7enc/bc7enc.cpp', True), ('src/gfx/tex/bc7enc/rgbcx.cpp', True)): + cmd = [str(CLANG64 / ('clang++.exe' if cxx else 'clang.exe')), '-M', str(ROOT / src)] + if cxx: + cmd.append('-std=c++17') + out = subprocess.run(cmd, capture_output=True, text=True, env=tool_env()).stdout + for p in re.split(r'\s+', out.replace('\\\n', ' ')): + p = p.replace('\\', '/') + if p.lower().startswith(str(PREFIX).replace('\\', '/').lower()): + used.add(pathlib.Path(p)) + # libraries: what lld reads in the two links (--verbose) + ninja = (home / 'build/build.ninja').read_text() + rsp = re.search(r'^build UltraTears\.exe: link (.*)$', ninja, re.M).group(1).split() # ninja deletes the .rsp + link = re.search(r'command = clang (.*) -o \$out @\$out.rsp (.*)', ninja) + flags = link.group(1).split() + link.group(2).split() + for objs, libs in ((rsp, flags), ([], ['-static', '-pthread', '-lzstd', '-lc++'])): + probe = tmp / 'probe.c' + probe.write_text('int main(void) { return 0; }\n') + cmd = [str(CLANG64 / 'clang.exe'), '-fuse-ld=lld', '-Wl,--verbose', '-o', str(tmp / 'probe.exe')] + rsp_file = tmp / 'probe.rsp' # 1500 objects: past Windows' command-line limit + rsp_file.write_text('\n'.join((home / o).as_posix() for o in objs) if objs else probe.as_posix()) + cmd += ['@' + str(rsp_file)] + [f for f in libs if f not in ('-g',)] + cmd = [c for c in cmd if not c.startswith('-Wl,--pdb')] + r = subprocess.run(cmd, capture_output=True, text=True, env=tool_env(), cwd=home) + if r.returncode: + sys.exit('the probe link failed:\n' + (r.stdout + r.stderr)[-2000:]) + for line in (r.stdout + r.stderr).splitlines(): + m = re.match(r'ld\.lld: Reading (.+)$', line.strip()) + if not m: + continue + p = m.group(1).replace('\\', '/') + if p.lower().startswith(str(PREFIX).replace('\\', '/').lower()) and os.path.isfile(p): + used.add(pathlib.Path(p)) + shutil.rmtree(tmp, ignore_errors=True) + return used + + +def dll_closure(programs): + seen, todo = set(), [CLANG64 / p for p in programs] + while todo: + f = todo.pop() + if f in seen or not f.is_file(): + continue + seen.add(f) + out = subprocess.run([str(CLANG64 / 'llvm-objdump.exe'), '-p', str(f)], capture_output=True, text=True).stdout + for dll in re.findall(r'DLL Name: (\S+)', out): + cand = CLANG64 / dll + if cand.is_file(): + todo.append(cand) + return seen + + +def copy_python(src, dst): + for item in src.iterdir(): + if item.name in ('python.exe', 'pythonw.exe', 'python3.dll', 'python314.dll', 'vcruntime140.dll', + 'vcruntime140_1.dll', 'LICENSE.txt'): + shutil.copy2(item, dst / item.name) + shutil.copytree(src / 'DLLs', dst / 'DLLs', ignore=shutil.ignore_patterns('_tkinter*', 'tcl*', 'tk*')) + shutil.copytree(src / 'Lib', dst / 'Lib', ignore=lambda d, names: [n for n in names if n in PY_SKIP or n == 'site-packages']) + site = dst / 'Lib/site-packages' + site.mkdir(parents=True, exist_ok=True) + venv_site = ROOT / '.venv/Lib/site-packages' + for name in os.listdir(venv_site): + base = name.split('-')[0].split('.')[0] + if base.lower() in {p.lower() for p in PY_PACKAGES} or (name.endswith('.dist-info') and + name.split('-')[0].lower() in ('lz4', 'capstone', 'unicorn')): + s = venv_site / name + (shutil.copytree(s, site / name, ignore=shutil.ignore_patterns('__pycache__')) if s.is_dir() + else shutil.copy2(s, site / name)) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument('--python', type=pathlib.Path, default=pathlib.Path(sys.base_prefix)) + ap.add_argument('--no-zip', action='store_true') + ap.add_argument('--home', type=pathlib.Path, default=ROOT.parent / 'release-test') + a = ap.parse_args() + if git('status', '--porcelain', '--untracked-files=no').strip(): + sys.exit('the working tree has uncommitted changes: commit first, the package must be the commit it names') + a.version = version.release() + if not a.version: + sys.exit('HEAD has no release tag: git tag v (the package is named after it)') + home, xlate = a.home.resolve(), ROOT / 'dev-local/build/shaders/ultrs-shaders.exe' + for need in (home / 'UltraTears.exe', home / 'build/build.ninja', xlate): + if not need.is_file(): + sys.exit(f'{need} is missing: build first (tools/deploy.py, tools/build_xlate.py)') + name = f'UltraTears-builder-{a.version}-windows-x86_64' + out = ROOT / 'dev-local/dist' / name + if out.exists(): + shutil.rmtree(out) + out.mkdir(parents=True) + # the source: the public tree of HEAD + tar = subprocess.run(['git', '-C', str(ROOT), 'archive', '--format=tar', public_tree.tree()], check=True, + capture_output=True).stdout + with tarfile.open(fileobj=io.BytesIO(tar)) as t: + t.extractall(out, filter='data') + commit = git('rev-parse', '--short', 'HEAD').strip() + # deps/clang64 + used = clang64_files_used(home) | dll_closure(PROGRAMS) + for f in sorted(used): + rel = f.relative_to(PREFIX) if f.is_absolute() else f + dst = out / 'deps/clang64' / rel + dst.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(f, dst) + lic = out / 'licenses' + for pkg in LICENSES: + d = PREFIX / 'share/licenses' / pkg + if d.is_dir(): + shutil.copytree(d, lic / 'msys2' / pkg) + # deps/python, deps/shaders + py = out / 'deps/python' + py.mkdir(parents=True) + copy_python(a.python, py) + (out / 'deps/shaders').mkdir(parents=True) + shutil.copy2(xlate, out / 'deps/shaders/ultrs-shaders.exe') + shutil.copy2(ROOT / 'third_party/maxwell_shader/LICENSE.txt', lic / 'Ryujinx.txt') + shutil.copy2(a.python / 'LICENSE.txt', lic / 'Python.txt') + (out / 'setup.bat').write_text(SETUP_BAT.replace('\n', '\r\n')) + (out / 'THIRD-PARTY-NOTICES.md').write_text(NOTICES) + (out / 'game').mkdir() + (out / 'game/PUT-YOUR-GAME-HERE.txt').write_text(GAME_TXT.replace('\n', '\r\n'), encoding='utf-8') + (out / 'VERSION.txt').write_text(version.version_txt(a.version, commit)) + size = sum(f.stat().st_size for f in out.rglob('*') if f.is_file()) + print(f'{out}: {size / 2**20:.0f} MiB, {len(used)} files of clang64') + if not a.no_zip: + z = out.parent / (name + '.zip') + with zipfile.ZipFile(z, 'w', zipfile.ZIP_DEFLATED, compresslevel=6) as zf: + for f in sorted(out.rglob('*')): + if f.is_file(): + zf.write(f, pathlib.Path(name) / f.relative_to(out)) + print(f'{z}: {z.stat().st_size / 2**20:.0f} MiB') + + +if __name__ == '__main__': + main() diff --git a/tools/make_romfs_index.py b/tools/make_romfs_index.py new file mode 100644 index 0000000..9b10837 --- /dev/null +++ b/tools/make_romfs_index.py @@ -0,0 +1,32 @@ +"""Write the runtime's RomFS index (MacNativeGames' runtime-index.bin layout, "TOTKRFS1") from the verified index.json. + +usage: python tools/make_romfs_index.py work/extracted/v143/romfs +Layout: 8-byte magic, u64 file count, u64 strings offset, then per file (sorted by path, byte order) +{u64 offset in romfs.bin, u64 size, u32 name offset, u32 name length including the NUL}, then the NUL-terminated paths. +""" +import json +import pathlib +import struct +import sys + + +def main(): + romfs = pathlib.Path(sys.argv[1]) + index = json.loads((romfs / 'index.json').read_text()) + if index.get('status') != 'pass': + sys.exit('RomFS has not passed integrity verification') + files = sorted(index['files'], key=lambda f: f['path'].encode('utf-8')) + strings = bytearray() + entries = bytearray() + for f in files: + name = f['path'].encode('utf-8') + b'\0' + entries += struct.pack(' {out}') + + +if __name__ == '__main__': + main() diff --git a/tools/pack_romfs.py b/tools/pack_romfs.py new file mode 100644 index 0000000..d0f3bef --- /dev/null +++ b/tools/pack_romfs.py @@ -0,0 +1,84 @@ +"""Pack the player's RomFS folder (plain files) into one image, as the development extraction has it: romfs.bin (the files +back to back in path order), index.json and runtime-index.bin (tools/make_romfs_index.py). + +usage: python tools/pack_romfs.py [--threads 16] +Why one file: opening a file costs ~5 ms cold on Windows (measured on 304 243 files of TOTK 1.4.3: 17 MiB/s one by one, +110 MiB/s with 16 threads, one image read 446 MiB/s), so the game and the build tools read one image instead of +hundreds of thousands of files. The folder is read with --threads readers, in order, a batch at a time. +Prints the content hash: SHA-256 over the SHA-256 of every file in path order (tools/setup.py compares it with the +version the port is made for). Game-derived data: stays out of git. +""" +import argparse +import hashlib +import json +import pathlib +import subprocess +import sys +import time +from concurrent.futures import ThreadPoolExecutor + +ROOT = pathlib.Path(__file__).resolve().parent.parent +sys.path.insert(0, str(ROOT / 'tools')) +from romfs import RomFS # noqa: E402 + +BATCH = 2000 +ALIGN = 16 + + +def pack(folder, out, threads=16, progress=print): + """-> (file count, bytes, content hash). Writes /romfs.bin, index.json, runtime-index.bin.""" + rom = RomFS(folder) + if rom.image is not None: + raise ValueError(f'{folder} is already an image') + names = sorted(rom.files) # byte order of the paths, as the runtime's index wants + out.mkdir(parents=True, exist_ok=True) + rows, total, offset = [], sum(rom.files.values()), 0 + content = hashlib.sha256() + t0 = last = time.time() + + def read(p): + data = rom.read(p) + return data, hashlib.sha256(data).digest() + + part = out / 'romfs.bin.part' + with open(part, 'wb') as img, ThreadPoolExecutor(threads) as ex: + for b in range(0, len(names), BATCH): + batch = names[b:b + BATCH] + for p, (data, digest) in zip(batch, ex.map(read, batch)): + if len(data) != rom.files[p]: + raise ValueError(f'{p}: size changed while packing') + pad = -offset % ALIGN + if pad: + img.write(b'\0' * pad) + offset += pad + rows.append(dict(path=p, offset=offset, size=len(data))) + img.write(data) + offset += len(data) + content.update(digest) + if time.time() - last > 10: + last = time.time() + done = sum(r['size'] for r in rows) + progress(f' RomFS: {len(rows)} / {len(names)} files, {done / 2**30:.1f} / {total / 2**30:.1f} GiB, ' + f'{time.time() - t0:.0f} s') + (out / 'romfs.bin').unlink(missing_ok=True) + part.rename(out / 'romfs.bin') + (out / 'index.json').write_text(json.dumps(dict(status='pass', source='folder', content_sha256=content.hexdigest(), + files=rows))) + subprocess.run([sys.executable, str(ROOT / 'tools/make_romfs_index.py'), str(out)], check=True, + stdout=subprocess.DEVNULL) + return len(rows), offset, content.hexdigest() + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument('folder', type=pathlib.Path) + ap.add_argument('out', type=pathlib.Path) + ap.add_argument('--threads', type=int, default=16) + a = ap.parse_args() + t0 = time.time() + n, size, h = pack(a.folder, a.out, a.threads) + print(f'{n} files, {size / 2**30:.1f} GiB, content {h}, {time.time() - t0:.0f} s') + + +if __name__ == '__main__': + main() diff --git a/tools/public_tree.py b/tools/public_tree.py new file mode 100644 index 0000000..8b317c1 --- /dev/null +++ b/tools/public_tree.py @@ -0,0 +1,64 @@ +"""What of the repository is public: the GitHub branch `public` and the builder package carry the files git tracks, +except PRIVATE (the internal working notes, not published for now: the user's decision of 2026-10-25). + +usage: python tools/public_tree.py --publish (moves the branch `public` to a new commit named after the tag v on + HEAD; pushing stays a separate, explicit step: git push origin public:main) +The local history (master) keeps everything; `public` is one commit per release on top of the previous one, so only +what a release holds reaches GitHub. +""" +import argparse +import os +import pathlib +import subprocess +import sys +import tempfile + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)) +import version # noqa: E402 + +ROOT = pathlib.Path(__file__).resolve().parent.parent +PRIVATE = ['docs'] + + +def git(*args, env=None): + return subprocess.run(['git', '-C', str(ROOT)] + list(args), check=True, capture_output=True, text=True, + env=env).stdout.strip() + + +def tree(ref='HEAD'): + """The id of the public tree of ref: its tree without PRIVATE.""" + with tempfile.TemporaryDirectory() as d: + env = dict(os.environ, GIT_INDEX_FILE=str(pathlib.Path(d) / 'index')) + git('read-tree', ref, env=env) + git('rm', '-r', '-q', '--cached', '--ignore-unmatch', '--', *PRIVATE, env=env) + return git('write-tree', env=env) + + +def publish(version): + t = tree() + parent = subprocess.run(['git', '-C', str(ROOT), 'rev-parse', '-q', '--verify', 'refs/heads/public'], + capture_output=True, text=True).stdout.strip() + if parent and git('rev-parse', parent + '^{tree}') == t: + return parent # nothing changed since the last release + msg = (f'UltraTears {version}\n\nAn unofficial native PC port of The Legend of Zelda: Tears of the Kingdom (Switch 1.4.3)' + ' by static recompilation, built on the player\'s PC from their own dump. See README.md.\n\n' + 'Co-Authored-By: Claude Opus 5.5 ') + commit = git('commit-tree', t, *(['-p', parent] if parent else []), '-m', msg) + git('branch', '-f', 'public', commit) + return commit + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument('--publish', action='store_true', required=True) + a = ap.parse_args() + if git('status', '--porcelain', '--untracked-files=no'): + raise SystemExit('commit first: the public branch is made from HEAD') + v = version.release() + if not v: + raise SystemExit('HEAD has no release tag: git tag v') + print(f'public {v} -> {publish(v)}') + + +if __name__ == '__main__': + main() diff --git a/tools/recon_code.py b/tools/recon_code.py new file mode 100644 index 0000000..fc3cf72 --- /dev/null +++ b/tools/recon_code.py @@ -0,0 +1,156 @@ +"""Code recon of the extracted NSO modules (analysis ELFs from MacNativeGames' prepare_modules.py). + +Per module: segment sizes, function count from .eh_frame_hdr (via MOD0), and an instruction histogram +(capstone, no detail) grouped into classes that matter for ARM64 -> x86-64 translation. + +usage: python tools/recon_code.py work/extracted/v143/elf --output work/analysis/v143-code.json +""" +import argparse +import collections +import json +import multiprocessing +import pathlib +import struct + +import capstone + +CHUNK = 1 << 20 # bytes of .text per worker task + +ATOMIC_EXCL = {'ldxr', 'ldxrb', 'ldxrh', 'ldaxr', 'ldaxrb', 'ldaxrh', 'stxr', 'stxrb', 'stxrh', 'stlxr', 'stlxrb', + 'stlxrh', 'ldxp', 'ldaxp', 'stxp', 'stlxp', 'clrex'} +LSE_PREFIX = ('cas', 'swp', 'ldadd', 'ldclr', 'ldeor', 'ldset', 'ldsmax', 'ldsmin', 'ldumax', 'ldumin', + 'stadd', 'stclr', 'steor', 'stset', 'stsmax', 'stsmin', 'stumax', 'stumin') +BARRIERS = {'dmb', 'dsb', 'isb'} +ACQREL = {'ldar', 'ldarb', 'ldarh', 'stlr', 'stlrb', 'stlrh', 'ldapr', 'ldaprb', 'ldaprh'} + + +def classify(mnemonic, ops): + m = mnemonic.split('.')[0] + if m in ('svc', 'hvc', 'brk', 'udf', 'hlt'): + return 'system:' + m + if m in ('mrs', 'msr'): + reg = ops.split(',')[1 if m == 'mrs' else 0].strip().lower() + return f'system:{m} {reg}' + if m in ATOMIC_EXCL: + return 'atomic:exclusive' + if m.startswith(LSE_PREFIX): + return 'atomic:lse' + if m in BARRIERS: + return 'barrier' + if m in ACQREL: + return 'atomic:acq_rel' + if m in ('br', 'braa', 'brab', 'braaz', 'brabz'): + return 'branch:indirect_br' + if m in ('blr', 'blraa', 'blrab', 'blraaz', 'blrabz'): + return 'branch:indirect_call' + if m in ('ret', 'retaa', 'retab'): + return 'branch:ret' + if m == 'bl': + return 'branch:call' + if m in ('b', 'cbz', 'cbnz', 'tbz', 'tbnz') or mnemonic.startswith('b.'): + return 'branch:direct' + vector = ' v' in ' ' + ops.replace('{', ' ').replace('[', ' ') and any( + t.strip().startswith(('v', '{v')) for t in ops.split(',')) + if vector: + return 'simd:vector' + regs = [t.strip() for t in ops.split(',')] + if any(r[:1] in ('s', 'd', 'h', 'q', 'b') and r[1:].isdigit() for r in regs if r): + if m.startswith(('ld', 'st')): + return 'fp:load_store' + return 'fp:scalar' + if m.startswith(('ld', 'st', 'prfm')): + return 'int:load_store' + if m == 'nop': + return 'nop' + return 'int:alu' + + +def scan(args): + data, base = args + md = capstone.Cs(capstone.CS_ARCH_ARM64, capstone.CS_MODE_ARM) + md.skipdata = True + classes = collections.Counter() + mnemonics = collections.Counter() + x18 = 0 + for _, _, mnemonic, ops in md.disasm_lite(data, base): + if mnemonic == '.byte': + classes['data_in_text'] += 1 + continue + mnemonics[mnemonic] += 1 + classes[classify(mnemonic, ops)] += 1 + if 'x18' in ops or 'w18' in ops: + x18 += 1 + return classes, mnemonics, x18 + + +def eh_frame_fdes(elf, segments): + """Function count from the binary-search table of .eh_frame_hdr (MOD0 points to it).""" + text = segments['text'] + mod0_ptr = struct.unpack_from(' a_) + tasks = [(text[i:i + CHUNK], t['addr'] + i) for i in range(0, len(text), CHUNK)] + classes, mnemonics, x18 = collections.Counter(), collections.Counter(), 0 + for c, m, x in pool.imap_unordered(scan, tasks): + classes.update(c); mnemonics.update(m); x18 += x + total = sum(v for k, v in classes.items() if k != 'data_in_text') + report[module] = dict( + segments={k: dict(addr=hex(v['addr']), size=v['size'], memsz=v['memsz']) for k, v in seg.items()}, + instructions=total, eh_frame=fdes, + function_size_bytes=dict(median=sizes[len(sizes) // 2], p90=sizes[int(len(sizes) * .9)], + max=sizes[-1]) if sizes else None, + classes={k: dict(count=v, share=round(v / total, 4)) for k, v in classes.most_common()}, + x18_operand_instructions=x18, + top_mnemonics=mnemonics.most_common(60)) + print(module, total, 'insns', fdes and fdes.get('fde_count'), 'FDEs', flush=True) + a.output.parent.mkdir(parents=True, exist_ok=True) + a.output.write_text(json.dumps(report, indent=1)) + + +if __name__ == '__main__': + main() diff --git a/tools/recon_romfs.py b/tools/recon_romfs.py new file mode 100644 index 0000000..2801b56 --- /dev/null +++ b/tools/recon_romfs.py @@ -0,0 +1,54 @@ +"""RomFS recon: file types and sizes, plus the shader inventory from MacNativeGames' prepare_shader_seed. + +usage: python tools/recon_romfs.py work/extracted/v143/romfs --output work/analysis/v143-romfs.json [--no-shaders] +""" +import argparse +import collections +import json +import pathlib +import sys + +ROOT = pathlib.Path(__file__).resolve().parent.parent +sys.path.insert(0, str(ROOT / 'refs/MacNativeGames/games/totk/scripts')) + + +def kind(path): + name = path.rsplit('/', 1)[-1] + if name.endswith('.zs'): + name = name[:-3] + parts = name.split('.') + return '.'.join(parts[-2:]) if len(parts) > 2 else (parts[-1] if len(parts) > 1 else '(none)') + + +def main(): + p = argparse.ArgumentParser() + p.add_argument('romfs', type=pathlib.Path) + p.add_argument('--output', type=pathlib.Path, required=True) + p.add_argument('--no-shaders', action='store_true') + a = p.parse_args() + index = json.loads((a.romfs / 'index.json').read_text()) + files = index['files'] + by_kind = collections.defaultdict(lambda: [0, 0]) + by_top = collections.defaultdict(lambda: [0, 0]) + for f in files: + k = by_kind[kind(f['path'])]; k[0] += 1; k[1] += f['size'] + t = by_top[f['path'].split('/', 1)[0]]; t[0] += 1; t[1] += f['size'] + total = sum(f['size'] for f in files) + report = dict(files=len(files), bytes=total, romfs_sha256=index['romfs_sha256'], + by_top_dir={k: dict(files=v[0], mib=round(v[1] / 2**20, 1)) for k, v in + sorted(by_top.items(), key=lambda kv: -kv[1][1])}, + by_type={k: dict(files=v[0], mib=round(v[1] / 2**20, 1)) for k, v in + sorted(by_kind.items(), key=lambda kv: -kv[1][1])[:40]}) + if not a.no_shaders: + import prepare_shader_seed as seed + _, shaders, startup, archives = seed.inventory(a.romfs) + stages = collections.Counter(stage for stage, *_ in shaders.values()) + report['shaders'] = dict(unique=len(shaders), default_bind_state=len(startup), archives=len(archives), + by_stage=dict(stages)) + a.output.parent.mkdir(parents=True, exist_ok=True) + a.output.write_text(json.dumps(report, indent=1)) + print(json.dumps({k: v for k, v in report.items() if k not in ('by_type',)}, indent=1)[:4000]) + + +if __name__ == '__main__': + main() diff --git a/tools/romfs.py b/tools/romfs.py new file mode 100644 index 0000000..ac4cc29 --- /dev/null +++ b/tools/romfs.py @@ -0,0 +1,44 @@ +"""Read-only access to a RomFS in either form the build meets: + - a folder of plain files (the player's dump: romfs/ extracted by their own tools); + - the packed image: romfs.bin with index.json (tools/pack_romfs.py). +files: {path: size}, paths with '/' as on the console; read(path) -> bytes; image: the romfs.bin path or None. +""" +import json +import os +import pathlib + + +class RomFS: + def __init__(self, root): + self.root = pathlib.Path(root) + bin_path, index = self.root / 'romfs.bin', self.root / 'index.json' + if bin_path.is_file() and index.is_file(): + self.image = bin_path + rows = json.loads(index.read_text())['files'] + self.offsets = {r['path']: r['offset'] for r in rows} + self.files = {r['path']: r['size'] for r in rows} + self._stream = None + else: + self.image = None + self.offsets = None + self.files = {} + base = len(str(self.root)) + 1 + for dirpath, _, names in os.walk(self.root): + for n in names: + full = os.path.join(dirpath, n) + self.files[full[base:].replace('\\', '/')] = os.path.getsize(full) + + def offset(self, path): + """The file's offset in romfs.bin (0 for a folder: tools then open it by its path).""" + return self.offsets[path] if self.offsets else 0 + + def read(self, path): + if self.image is None: + return (self.root / path).read_bytes() + if self._stream is None: + self._stream = open(self.image, 'rb') + self._stream.seek(self.offsets[path]) + data = self._stream.read(self.files[path]) + if len(data) != self.files[path]: + raise ValueError(f'short RomFS read: {path}') + return data diff --git a/tools/run_eden_xlate.py b/tools/run_eden_xlate.py new file mode 100644 index 0000000..76e22b2 --- /dev/null +++ b/tools/run_eden_xlate.py @@ -0,0 +1,48 @@ +"""Translate the exported shaders with eden_xlate (Eden's shader recompiler), 8 shards on at most 8 logical CPUs. + +usage: python tools/run_eden_xlate.py work/shaders/v143 [--shards 8] [--only N] +Writes /eden/.spv and /eden-results.tsv (key, ok/fail, seconds, spv bytes, message). +""" +import argparse +import ctypes +import os +import pathlib +import subprocess +import time + +ROOT = pathlib.Path(__file__).resolve().parent.parent + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument('dir', type=pathlib.Path) + ap.add_argument('--shards', type=int, default=8) + ap.add_argument('--only', type=int, default=0, help='run shard 0 of N only (a quick check)') + a = ap.parse_args() + (a.dir / 'eden').mkdir(exist_ok=True) + env = dict(os.environ, PATH=r'C:\msys64\clang64\bin' + os.pathsep + os.environ['PATH']) + exe = ROOT / 'build/eden_xlate/eden_xlate.exe' + shards = a.only or a.shards + t0 = time.time() + procs = [] + for s in (range(1) if a.only else range(shards)): + p = subprocess.Popen([str(exe), str(a.dir / 'shaders.tsv'), str(a.dir / 'raw'), str(a.dir / 'eden'), str(s), str(shards)], + env=env, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True, creationflags=0x4 | 0x4000) + ctypes.windll.kernel32.SetProcessAffinityMask(int(p._handle), ctypes.c_size_t(0xFF)) + ctypes.windll.ntdll.NtResumeProcess(int(p._handle)) + procs.append(p) + rows = [] + for p in procs: + out, _ = p.communicate() + rows += [l for l in out.splitlines() if l.strip()] + if p.returncode: + rows.append(f'\tfail\t0\t0\texit code {p.returncode}') + with (a.dir / 'eden-results.tsv').open('w') as f: + f.write('key\tresult\tseconds\tspv_bytes\tmessage\n') + f.write('\n'.join(sorted(rows)) + '\n') + ok = sum(r.split('\t')[1] == 'ok' for r in rows) + print(f'eden: {ok}/{len(rows)} ok, {time.time() - t0:.0f}s wall') + + +if __name__ == '__main__': + main() diff --git a/tools/run_limited.py b/tools/run_limited.py new file mode 100644 index 0000000..6b40575 --- /dev/null +++ b/tools/run_limited.py @@ -0,0 +1,30 @@ +"""Run a heavy command on at most 8 logical CPUs at below-normal priority (the user works on this PC meanwhile). + +usage: python tools/run_limited.py [--cpus N] -- [args...] +The process starts suspended, gets the affinity mask and priority, then runs; its children inherit both. +MSYS2 CLANG64's bin directory is put first on PATH. +""" +import ctypes +import os +import subprocess +import sys + + +def main(): + args = sys.argv[1:] + cpus = 8 + if args[:1] == ['--cpus']: + cpus, args = int(args[1]), args[2:] + if args[:1] == ['--']: + args = args[1:] + env = dict(os.environ, PATH=r'C:\msys64\clang64\bin' + os.pathsep + os.environ['PATH']) + p = subprocess.Popen(args, env=env, creationflags=0x4 | 0x4000) # CREATE_SUSPENDED | BELOW_NORMAL_PRIORITY_CLASS + if not ctypes.windll.kernel32.SetProcessAffinityMask(int(p._handle), ctypes.c_size_t((1 << cpus) - 1)): + p.kill() + sys.exit('SetProcessAffinityMask failed') + ctypes.windll.ntdll.NtResumeProcess(int(p._handle)) + sys.exit(p.wait()) + + +if __name__ == '__main__': + main() diff --git a/tools/run_ryujinx_xlate.py b/tools/run_ryujinx_xlate.py new file mode 100644 index 0000000..199e107 --- /dev/null +++ b/tools/run_ryujinx_xlate.py @@ -0,0 +1,80 @@ +"""Translate the exported shaders with totk_xlate --serve (Ryujinx), in parallel on at most 8 logical CPUs. + +usage: python tools/run_ryujinx_xlate.py data/shaders [--workers 8] +Writes /ryujinx/.spv and /ryujinx-results.tsv (key, ok/fail, seconds, spv bytes, message). +""" +import argparse +import ctypes +import json +import os +import pathlib +import subprocess +import sys +import threading +import time + +ROOT = pathlib.Path(__file__).resolve().parent.parent +sys.path.insert(0, str(ROOT / 'tools')) +from toolchain import XLATE # noqa: E402 + + +def start_pinned(cmd, mask, env): + CREATE_SUSPENDED, BELOW_NORMAL = 0x4, 0x4000 + p = subprocess.Popen(cmd, env=env, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, + text=True, creationflags=CREATE_SUSPENDED | BELOW_NORMAL) + ctypes.windll.kernel32.SetProcessAffinityMask(int(p._handle), ctypes.c_size_t(mask)) + ctypes.windll.ntdll.NtResumeProcess(int(p._handle)) + return p + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument('dir', type=pathlib.Path) + ap.add_argument('--workers', type=int, default=8) + ap.add_argument('--new-only', action='store_true', help='translate only keys without a .spv') + ap.add_argument('--quiet', action='store_true') + a = ap.parse_args() + lines = (a.dir / 'jobs.jsonl').read_text().splitlines() + if a.new_only: # only shaders without a translation yet (the comparison's outputs stay) + lines = [l for l in lines if not (a.dir / 'ryujinx' / (json.loads(l)['job']['key'] + '.spv')).exists()] + (a.dir / 'ryujinx').mkdir(exist_ok=True) + env = dict(os.environ, DOTNET_CLI_TELEMETRY_OPTOUT='1', DOTNET_NOLOGO='1', + DOTNET_CLI_HOME=str(ROOT / 'third_party/dotnet-home')) + if XLATE is None: + sys.exit('the shader translator ultrs-shaders.exe is missing: python tools/build_xlate.py') + cmd = [str(XLATE)] # tools/build_xlate.py, or the builder package's deps/shaders + results, lock = [], threading.Lock() + shards = [lines[i::a.workers] for i in range(a.workers)] + t0 = time.time() + + def worker(shard): + p = start_pinned(cmd + ['--serve'], (1 << a.workers) - 1, env) + for line in shard: + key = json.loads(line)['job']['key'] + s = time.perf_counter() + p.stdin.write(line + '\n'); p.stdin.flush() + reply = p.stdout.readline().strip() + dt = time.perf_counter() - s + spv = a.dir / 'ryujinx' / f'{key}.spv' + ok = reply.startswith('ok ') + size = spv.stat().st_size if ok and spv.exists() else 0 + msg = '' if ok else (reply[len('fail ' + key) + 1:] if reply.startswith('fail') else reply or 'no reply') + with lock: + results.append((key, 'ok' if ok else 'fail', dt, size, msg.replace('\t', ' ')[:300])) + if len(results) % 2000 == 0 and not a.quiet: + print(f'{len(results)}/{len(lines)} {time.time() - t0:.0f}s', flush=True) + p.stdin.close(); p.wait() + + threads = [threading.Thread(target=worker, args=(s,)) for s in shards] + for t in threads: t.start() + for t in threads: t.join() + with (a.dir / ('ryujinx-results-new.tsv' if a.new_only else 'ryujinx-results.tsv')).open('w') as f: # --new-only keeps the full table + f.write('key\tresult\tseconds\tspv_bytes\tmessage\n') + for r in sorted(results): + f.write(f'{r[0]}\t{r[1]}\t{r[2]:.4f}\t{r[3]}\t{r[4]}\n') + ok = sum(r[1] == 'ok' for r in results) + print(f'ryujinx: {ok}/{len(results)} ok, {time.time() - t0:.0f}s wall, {sum(r[2] for r in results):.0f}s busy') + + +if __name__ == '__main__': + main() diff --git a/tools/setup.py b/tools/setup.py new file mode 100644 index 0000000..fca7f43 --- /dev/null +++ b/tools/setup.py @@ -0,0 +1,250 @@ +"""Build UltraTears from the player's own dump, in one command: every step is checked and can be resumed. + +usage: python tools/setup.py [--dump ] [--cpus N] [--only ] +The dump is game\\ in this folder unless --dump names another (remembered for later runs); run it again to resume, or to +rebuild what changed after an update of the port. +The dump: the ExeFS and RomFS of The Legend of Zelda: Tears of the Kingdom with update 1.4.3 applied, as plain files, +extracted by the player from their own console. Encrypted packages (.nsp, .xci, .nca, ...) are refused: the port +decrypts nothing and reads no keys (README.md, Legal). Optional: \\sysdata\\fonts\\*.bfttf, the console's shared fonts. + +Steps (data\\ holds everything made from the dump; it never leaves this PC): + check the tools, the dump's version (SHA-256 of the code modules) and its files + romfs the RomFS folder packed into one image, data\\romfs (its content hash checked against 1.4.3) + lift the game's code translated to C, data\\generated + compile the C compiled and linked: UltraTears.exe (objects in build\\) + shaders every shader of the game found and translated to SPIR-V, data\\shaders + textures the ASTC textures re-encoded for PC graphics cards (BC7 / BC1), data\\textures + finish ultratears.cfg +""" +import argparse +import hashlib +import json +import os +import pathlib +import shutil +import subprocess +import sys +import time + +ROOT = pathlib.Path(__file__).resolve().parent.parent +sys.path.insert(0, str(ROOT / 'tools')) +import toolchain # noqa: E402 + +DATA = ROOT / 'data' +STATE = DATA / 'setup.json' +# The version the port is made for: TOTK 1.4.3 (facts about the player's files, no game data). +EXEFS = { + 'main': 'e84a90d551fed0824b4b74933b9c6d043c458521363a7cea58b2faca1822dc9e', + 'subsdk0': '767c35bcd7d23e78f9063c69b638290b8b83f91e839beb10799e6576960179b1', + 'sdk': '1d187f29784cd0af46e9e123fc2a62c77b3b07c5a39c8633d90d6c80de64e4ad', +} +ROMFS_FILES = 304243 +ROMFS_BYTES = 17026725026 +ROMFS_CONTENT = 'b2a2e966a06158e098c03e564f77fd72809bc57275f04eb0725bcde14d522056' +ENCRYPTED = ('.nsp', '.nsz', '.xci', '.xcz', '.nca', '.ncz') + + +class Fail(Exception): + pass + + +def say(msg): + print(msg, flush=True) + + +def sha256(path): + h = hashlib.sha256() + with open(path, 'rb') as f: + for block in iter(lambda: f.read(1 << 20), b''): + h.update(block) + return h.hexdigest() + + +def tree_hash(paths): + """A stamp of source files: when it changes, the step that uses them runs again.""" + h = hashlib.sha256() + for p in sorted(paths): + h.update(str(p.relative_to(ROOT)).encode() + b'\0' + p.read_bytes()) + return h.hexdigest()[:16] + + +def run(cmd, what): + t0 = time.time() + rc = subprocess.run([str(c) for c in cmd], cwd=ROOT).returncode + if rc: + raise Fail(f'{what} failed (exit code {rc}); the lines above say why') + return time.time() - t0 + + +def load_state(): + try: + return json.loads(STATE.read_text()) + except (OSError, ValueError): + return {} + + +def save_state(state): + DATA.mkdir(exist_ok=True) + STATE.write_text(json.dumps(state, indent=1)) + + +# ---------------------------------------------------------------- steps +def step_check(dump, state): + if sys.version_info < (3, 14): + raise Fail(f'Python 3.14 or newer is needed (this is {sys.version.split()[0]})') + for mod in ('lz4', 'capstone', 'unicorn'): + try: + __import__(mod) + except ImportError: + raise Fail(f'the Python package {mod} is missing: python -m pip install lz4 capstone unicorn') + toolchain.env() + if toolchain.XLATE is None: + raise Fail('the shader translator ultrs-shaders.exe is missing: build it with python tools/build_xlate.py ' + '(needs the .NET 10 SDK) or take deps\\shaders from the builder package') + if packed(state) and all(ok_module(DATA / 'exefs' / n, h) for n, h in EXEFS.items()): + return 'Tears of the Kingdom 1.4.3, already taken in (data\\exefs, data\\romfs): the dump is no longer needed' + if not dump.is_dir(): + raise Fail(f'{dump} does not exist: put your extracted game there (game\\exefs and game\\romfs), ' + 'or give its folder with --dump') + for p in list(dump.iterdir()) + list((dump / 'exefs').glob('*') if (dump / 'exefs').is_dir() else []): + if p.suffix.lower() in ENCRYPTED: + raise Fail(f'{p.name}: encrypted packages are not accepted. Give the extracted exefs and romfs folders ' + '(this project decrypts nothing and reads no keys)') + exefs, romfs = dump / 'exefs', dump / 'romfs' + if not exefs.is_dir() or not romfs.is_dir(): + raise Fail(f'{dump} must hold the folders exefs and romfs (extracted, as plain files)') + for name, want in EXEFS.items(): + f = exefs / name + if not f.is_file(): + raise Fail(f'exefs\\{name} is missing') + if sha256(f) != want: + raise Fail(f'exefs\\{name} is not the one of Tears of the Kingdom 1.4.3. Extract the ExeFS again with ' + 'update 1.4.3 applied (the base game 1.0.0 and other updates are not supported)') + if not (romfs / 'Pack/ZsDic.pack.zs').is_file(): + raise Fail(f'{romfs} does not look like the RomFS of Tears of the Kingdom (Pack\\ZsDic.pack.zs is missing)') + return 'Tears of the Kingdom 1.4.3 (the code modules match)' + + +def packed(state): + out = DATA / 'romfs' + return state.get('romfs') == ROMFS_CONTENT and (out / 'romfs.bin').is_file() and (out / 'runtime-index.bin').is_file() + + +def ok_module(path, want): + return path.is_file() and sha256(path) == want + + +def step_romfs(dump, state, cpus): + out = DATA / 'romfs' + if packed(state): + return 'already packed' + from pack_romfs import pack + n, size, content = pack(dump / 'romfs', out, threads=16, progress=say) + if n != ROMFS_FILES or size < ROMFS_BYTES or content != ROMFS_CONTENT: + raise Fail(f'the RomFS is not the one of Tears of the Kingdom 1.4.3 ({n} files, expected {ROMFS_FILES}; ' + 'content differs). Extract the RomFS again with update 1.4.3 applied; mods do not go here') + state['romfs'] = content + return f'{n} files, {size / 2**30:.1f} GiB, content verified' + + +def step_lift(dump, state, cpus): + exefs = DATA / 'exefs' + exefs.mkdir(parents=True, exist_ok=True) + for name in list(EXEFS) + ['main.npdm']: # the code modules are taken in, so the dump can go after the build + src = dump / 'exefs' / name + if src.is_file() and (not (exefs / name).is_file() or (exefs / name).stat().st_size != src.stat().st_size): + shutil.copyfile(src, exefs / name) + stamp = tree_hash(list((ROOT / 'tools/lifter').glob('*.py')) + [ROOT / 'tools/gen_game.py']) + if state.get('lift') == stamp and (DATA / 'generated/main').is_dir(): + return 'up to date' + run([sys.executable, 'tools/gen_game.py', exefs, DATA / 'generated'], 'translating the code') + state['lift'] = stamp + return 'done' + + +def step_compile(dump, state, cpus): + run([sys.executable, 'tools/build_game.py', '--generated', DATA / 'generated', '--target', 'ultratears', '--cpus', cpus], + 'compiling') + return 'UltraTears.exe' + + +def step_shaders(dump, state, cpus): + out = DATA / 'shaders' + stamp = tree_hash([ROOT / 'tools/shaders/inventory.py', ROOT / 'tools/export_shaders.py']) + if state.get('shaders-export') != stamp or not (out / 'jobs.jsonl').is_file(): + run([sys.executable, 'tools/export_shaders.py', DATA / 'romfs', out, DATA / 'exefs'], 'finding the shaders') + state['shaders-export'] = stamp + save_state(state) + xlate = sha256(toolchain.XLATE)[:16] + new_only = ['--new-only'] if state.get('shaders-xlate') == xlate else [] + run([sys.executable, 'tools/run_ryujinx_xlate.py', out, '--workers', min(cpus, 32), '--quiet'] + new_only, + 'translating the shaders') + state['shaders-xlate'] = xlate + return 'done' + + +def step_textures(dump, state, cpus): + run([sys.executable, 'tools/bc7_build.py', '--romfs', DATA / 'romfs', '--cache', DATA / 'textures', '--threads', + min(cpus, 64)], 're-encoding the textures') + return 'done' + + +def step_finish(dump, state, cpus): + fonts = dump / 'sysdata/fonts' + lines = ['# Written by tools/setup.py: where UltraTears finds its data (relative to this folder).', + 'exefs=data/exefs', 'romfs=data/romfs', 'shaders=data/shaders/ryujinx', 'textures=data/textures', + 'saves=userdata'] + if fonts.is_dir() and list(fonts.glob('*.bfttf')): + shutil.copytree(fonts, DATA / 'fonts', dirs_exist_ok=True) + lines.append('fonts=data/fonts') + (ROOT / 'ultratears.cfg').write_text('\n'.join(lines) + '\n') + (ROOT / 'build/UltraTears.exe').unlink(missing_ok=True) # where builds before 0.1.0 linked it + if (dump / 'romfs').is_dir(): + say(f' The game reads data\\romfs and data\\exefs now: {dump} (16 GB) is no longer needed.') + if sys.stdin.isatty() and dump.resolve() == (ROOT / 'game').resolve(): + answer = input(' Delete game\\exefs and game\\romfs to free 16 GB? [y/N] ').strip().lower() + if answer in ('y', 'yes', 'д', 'да'): + for sub in ('exefs', 'romfs'): + shutil.rmtree(dump / sub, ignore_errors=True) + say(' deleted') + return 'ultratears.cfg ready' + + +STEPS = [('check', step_check), ('romfs', step_romfs), ('lift', step_lift), ('compile', step_compile), + ('shaders', step_shaders), ('textures', step_textures), ('finish', step_finish)] + + +def main(): + ap = argparse.ArgumentParser(description='Build UltraTears from your own dump of the game.') + ap.add_argument('--dump', type=pathlib.Path, + help='the folder with exefs\\ and romfs\\ (default: game\\ here; remembered for later runs)') + ap.add_argument('--cpus', type=int, default=os.cpu_count() or 4, + help='logical CPUs the build may use (default: all; it runs at low priority)') + ap.add_argument('--only', choices=[s for s, _ in STEPS], help='run one step only') + a = ap.parse_args() + state = load_state() + if a.dump: + state['dump'] = str(a.dump.resolve()) + if not state.get('dump'): + state['dump'] = str(ROOT / 'game') + dump = pathlib.Path(state['dump']) + say(f'UltraTears setup: dump {dump}, {a.cpus} CPUs') + t_all = time.time() + for i, (name, fn) in enumerate(STEPS, 1): + if a.only and name != a.only and name != 'check': + continue + say(f'[{i}/{len(STEPS)}] {name}') + t0 = time.time() + try: + result = fn(dump, state) if name == 'check' else fn(dump, state, a.cpus) + except Fail as e: + say(f'\nERROR in step "{name}": {e}') + save_state(state) + sys.exit(1) + save_state(state) + say(f' {result or "ok"} ({time.time() - t0:.0f} s)') + say(f'\nDone in {(time.time() - t_all) / 60:.0f} min. Start the game: UltraTears.exe in {ROOT}') + + +if __name__ == '__main__': + main() diff --git a/tools/shaders/inventory.py b/tools/shaders/inventory.py new file mode 100644 index 0000000..df3c39d --- /dev/null +++ b/tools/shaders/inventory.py @@ -0,0 +1,175 @@ +"""Every shader of the game: the NVN shaders in the RomFS (Shader/ packages, SDK and boot packages, effects and UI layouts) +and the few the game modules embed, as (stage, control, code, source) by the runtime's key (SHA-256 of control + code). + +From MacNativeGames games/totk/scripts/prepare_shader_seed.py @b75be8c (MIT, (c) 2026 MacNativeGames contributors; +license: third_party/licenses/MacNativeGames.txt): digest, checked, sarc, bnsh_shaders, sharcb_shaders and +startup_package unchanged; inventory() reads the RomFS through tools/romfs.py (a folder of plain files or the +development romfs.bin) and takes the modules as images (bytes), not as files. +""" +import hashlib +import struct + +ZSTD = b'\x28\xb5\x2f\xfd' + + +def digest(data): + return hashlib.sha256(data).hexdigest() + + +def checked(data, offset, size): + if offset < 0 or size < 0 or offset + size > len(data): + raise ValueError('shader container range outside file') + return data[offset:offset+size] + + +def sarc(data): + if data[:4] != b'SARC': + raise ValueError('expected SARC') + endian = '<' if data[6:8] == b'\xff\xfe' else '>' + header, size, start = struct.unpack_from(endian+'HxxII', data, 4) + if size != len(data) or header < 20 or checked(data, header, 4) != b'SFAT': + raise ValueError('invalid SARC header') + count = struct.unpack_from(endian+'H', data, header+6)[0] + names = header+12+count*16 + if checked(data, names, 4) != b'SFNT': + raise ValueError('invalid SARC string table') + names += 8 + result = {} + for i in range(count): + _, attribute, lo, hi = struct.unpack_from(endian+'4I', checked(data, header+12+i*16, 16)) + off = names+(attribute & 0xffffff)*4 + if attribute >> 24 != 1 or not names <= off < start or hi < lo or start+lo < start: + raise ValueError('invalid SARC member') + name = data[off:data.index(0, off, start)].decode('utf-8') + if name.startswith('/') or '\\' in name or any(p in ('', '.', '..') for p in name.split('/')) or name in result: + raise ValueError('unsafe or duplicate SARC name') + result[name] = checked(data, start+lo, hi-lo) + return result + + +def bnsh_shaders(data): + """BNSH absolute offsets are relative to each embedded BNSH, not BFSHA.""" + if data[:4] != b'BNSH' or checked(data, 12, 2) != b'\xff\xfe' or checked(data, 96, 4) != b'grsc': + raise ValueError('unsupported BNSH header') + size = struct.unpack_from('; a record + may reuse an earlier record's control. Control (its first 0x880 bytes) and code hash to the runtime's key.""" + if len(data) < 32 or data[:4] != b'BAHS': + raise ValueError('expected an agl shader package') + start, pool_size, pool = struct.unpack_from('<3I', data, 20) + if not 32 <= start <= pool or pool + pool_size > len(data): + raise ValueError('agl shader package layout outside the file') + seen = set() + for size in (b'\x80\x08\x00\x00', b'\x8c\x08\x00\x00'): + at = start + while (at := data.find(size, at, pool)) >= 0: + record, at = at - 8, at + 4 + if record < start or record % 4: + continue + code_off = struct.unpack_from('= pool_size: + continue + control = data[control_at:control_at+0x880] + c1, length = struct.unpack_from(' length: + continue + code = data[pool+code_off:pool+code_off+length] + key = digest(control+code) + if key not in seen: + seen.add(key) + yield key, stage, control, code + + +def startup_package(name): + """The packages the startup shaders come from; effects and UI layouts carry more.""" + return (name == 'Lib/NintendoSDK/TextureShader.bnsh' or name == 'Pack/Bootup.Nin_NX_NVN.pack.zs' or + name.startswith('Shader/') and name.endswith('.zs')) + + +def inventory(rom, modules=None): + """rom: a tools/romfs.RomFS; modules: {name: image bytes} of the game modules (main, subsdk0, sdk). + -> shaders {key: (stage, control, code, source path)}, startup (keys translated for the default bind state).""" + import compression.zstd as zstd + shaders, startup = {}, set() + dictionaries = {} + for raw in sarc(zstd.decompress(rom.read('Pack/ZsDic.pack.zs'))).values(): + d = zstd.ZstdDict(raw) + dictionaries[d.dict_id] = d + + def decode(raw): + info = zstd.get_frame_info(raw) + if info.decompressed_size is None or info.decompressed_size > 512 << 20: + raise ValueError('unbounded shader package') + return zstd.decompress(raw, zstd_dict=dictionaries[info.dictionary_id] if info.dictionary_id else None) + + def visit(name, data, keys, depth=0): + if data[:4] == b'BAHS': # agl: the engine's own shaders (title screen, post effects, lighting) + for key, stage, control, code in sharcb_shaders(data): + shaders.setdefault(key, (stage, control, code, name)) + keys.add(key); startup.add(key) + return + if data[:4] == b'SARC': + if depth > 4: + return + for member, raw in sarc(data).items(): + visit(name+'/'+member, decode(raw) if raw[:4] == ZSTD else raw, keys, depth+1) + return + # BFSHA and BNSH files, and effect / layout resources, hold complete, independently addressed BNSH files. + pos = 0 + while True: + pos = data.find(b'BNSH\0\0\0\0', pos) + if pos < 0: + break + size = struct.unpack_from('/-spirv-val.tsv (key, ok/invalid, first error line). +""" +import ctypes +import pathlib +import subprocess +import sys +import threading + +SPIRV_VAL = r'C:\VulkanSDK\1.4.363.0\Bin\spirv-val.exe' + + +def main(): + base, name = pathlib.Path(sys.argv[1]), sys.argv[2] + files = sorted((base / name).glob('*.spv')) + results, lock = [], threading.Lock() + + def worker(chunk): + for f in chunk: + p = subprocess.Popen([SPIRV_VAL, '--target-env', 'vulkan1.3', '--scalar-block-layout', str(f)], stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, text=True, creationflags=0x4 | 0x4000) + ctypes.windll.kernel32.SetProcessAffinityMask(int(p._handle), ctypes.c_size_t(0xFF)) + ctypes.windll.ntdll.NtResumeProcess(int(p._handle)) + out, _ = p.communicate() + first = next((l for l in out.splitlines() if l.strip()), '') + with lock: + results.append((f.stem, 'ok' if p.returncode == 0 else 'invalid', first.replace('\t', ' ')[:200])) + threads = [threading.Thread(target=worker, args=(files[i::8],)) for i in range(8)] + for t in threads: t.start() + for t in threads: t.join() + with (base / f'{name}-spirv-val.tsv').open('w') as out: + out.write('key\tresult\tmessage\n') + for r in sorted(results): + out.write('\t'.join(r) + '\n') + bad = [r for r in results if r[1] != 'ok'] + print(f'{name}: {len(results) - len(bad)}/{len(results)} valid') + + +if __name__ == '__main__': + main() diff --git a/tools/version.py b/tools/version.py new file mode 100644 index 0000000..ac49089 --- /dev/null +++ b/tools/version.py @@ -0,0 +1,55 @@ +"""The port's version: one source, the git tag v on the release commit. + + - a release: the tag on HEAD (tools/make_package.py and tools/public_tree.py refuse an untagged HEAD); + - a builder package or a game folder made by tools/deploy.py has no .git: VERSION.txt at its root says it + ("UltraTears (commit )", written by make_package.py / deploy.py); + - a clone: git describe, e.g. 0.1.0-3-gabc1234-dirty between releases. +tools/build_game.py puts it into the exe (build/version.h: the first log line, the file's properties in Windows). +""" +import pathlib +import re +import subprocess + +ROOT = pathlib.Path(__file__).resolve().parent.parent +TAG = re.compile(r'^v(\d+\.\d+\.\d+)$') + + +def _git(*args): + r = subprocess.run(['git', '-C', str(ROOT)] + list(args), capture_output=True, text=True) + return r.stdout.strip() if r.returncode == 0 else None + + +def release(): + """The version of the tag v on HEAD, or None.""" + for t in (_git('tag', '--points-at', 'HEAD') or '').split(): + m = TAG.match(t) + if m: + return m.group(1) + return None + + +def current(): + """(version, commit) of this tree.""" + f = ROOT / 'VERSION.txt' + if f.is_file(): + m = re.match(r'UltraTears (\S+) \(commit (\w+)\)', f.read_text(encoding='utf-8')) + if m: + return m.group(1), m.group(2) + d = _git('describe', '--tags', '--match', 'v[0-9]*', '--dirty') + commit = _git('rev-parse', '--short', 'HEAD') or 'unknown' + if d: + return d[1:], commit + if _git('rev-parse', 'HEAD'): # no release tag yet + dirty = '-dirty' if _git('status', '--porcelain', '--untracked-files=no') else '' + return f'0.0.0-dev{dirty}', commit + return 'unknown', 'unknown' + + +def version_txt(version, commit): + return f'UltraTears {version} (commit {commit})\n' + + +def numbers(version): + """Four numbers for the Windows version resource: x.y.z and the commits since the tag (0.1.0-3-g... -> 0,1,0,3).""" + m = re.match(r'(\d+)\.(\d+)\.(\d+)(?:-(\d+)-g)?', version) + return tuple(int(x or 0) for x in m.groups()) if m else (0, 0, 0, 0)