Overhaul the build system (#234)
* wip
* fix
* add disassembler
* Disasm builds OK
* Variable addends
* More wip
* Rodata migration implemented
* Cleanup old tools
* Try fix submodule -> subrepo merge
* git subrepo pull --force --remote=https://github.com/zeldaret/ZAPD.git tools/ZAPD
subrepo:
subdir: "tools/ZAPD"
merged: "602e609"
upstream:
origin: "https://github.com/zeldaret/ZAPD.git"
branch: "master"
commit: "602e609"
git-subrepo:
version: "0.4.3"
origin: "https://github.com/ingydotnet/git-subrepo"
commit: "2f68596"
* Builds again but assets are totally broken
* git subrepo pull --force tools/asm-processor
subrepo:
subdir: "tools/asm-processor"
merged: "1ffdb08a"
upstream:
origin: "https://github.com/simonlindholm/asm-processor.git"
branch: "master"
commit: "1ffdb08a"
git-subrepo:
version: "0.4.3"
origin: "https://github.com/ingydotnet/git-subrepo"
commit: "2f68596"
* More cleanup, move functions.txt and variables.txt to tools/disasm and rm tables
* rm z64compress in preparation for subrepo
* git subrepo clone (merge) https://github.com/z64me/z64compress.git tools/z64compress
subrepo:
subdir: "tools/z64compress"
merged: "eb11085c"
upstream:
origin: "https://github.com/z64me/z64compress.git"
branch: "main"
commit: "eb11085c"
git-subrepo:
version: "0.4.3"
origin: "https://github.com/ingydotnet/git-subrepo"
commit: "2f68596"
* Fix asset extraction
* Fix diff-init make rule
* Split code bss
* Split assumed linker bug padding from assembly files
* add filelists for mm.us.rev1
* Maybe working, but I'm not sure
* add overlays to spec
* Add rodata to actos
* Everything compiles
* Make a lot of C files for code
* Add almost every file in code to spec
* whoops
* 3 code files left
* add scenes to spec
* More progress on progress.py
* Fix skelanime in spec
* audio files!
* Fix merge issues
* Fix some C files in code
* Fix remaining code files
* Use existing O1 C files in spec
* reorder boot order in spec
* update spec
* fault.c
* Convert relocs on completed actors, fixbaserom uses current rom name
* more boot files
* Add VT macros and script
* finish already existing boot files
* most of libultra
* fix 64bits libultra files
* Use C files for libultra, wrap some functions in NON_MATCHING
* Remove duplicate of OS_CLOCK_RATE from fault.c
* C files for fbdemos
* delete dumb files
* bootstrap C files, still need to add them to the spec
* update fixbaserom
* boot OK?
* I forgot to commit the spec
* C for gamestates
* C for kaleido
* Change all includes to ""
* copy actor sizes script from oot
* I forgot to delete those files
* Basic C files for effects
* Add effects initvars names
* Remove mislabelled boot functions from header/txt
* Begin porting bootstrap_fx, some sizes
* Fix <>
* Fix enum
* Fix diff.py
* fix libultra stuff
* update regconvert
* update setup warnings
* add some missing ;
* Fix some makefile stuff and other fixes on some non_matching functions
* add executable flag in extract_baserom and fixbaserom
* fix relative path
* copy assist from oot
* fix map path
* another assist path fix
* Delete C files for handwritten files
* add code_801A51F0 to spec
* add gfxbuffers to spec
* Move rodata to top of each file when possible
* UNK_TYPEs for func_801A51F0
* Remove kaleido rodata from spec
* Update spec and undefined_syms for recent merge
* GCC warnings and fix errors in nonmatchings,
* round percentage numbers
* progress script: format changes
* progress: error on non-existing files
* fix warning in z_scene_table
* Match 2 nonmatchings in z_actor
* Warnings in lightswitch and invadepoh
* Fix warning in z_actor_dlftbls
* I though I fixed this one
* whoops
* Comment out CC_CHECK
* Removed redundant ultra64.h includes
* Update asm_processor, sorted boot_O1 into other folders, completed the fbdemo bootstrap, cleaned up undefined_syms
* Completed gamestates bootstrap
* Split kaleido_scope
* Remove section.h and segment.h, move keep object externs to a common location in variables.h
* Completed effects bootstrap
* Segmented address externs for effects, fbdemos, gamestates and kaleido
* Move actor data externs out of the if 0
* Segmented address externs for actors
* Prepare actionfunc detection
* fix script, how did it even work before
* Fix actionfunc script again, re-introduce some more intermediate prints to the disassembler
* Automated actionFunc detection in actors
* Segmented addresses from player .text
* rm old segment addrs script and fix build
* Move sizes folder to tools
* Make build.py executable
* New Jenkinsfile Prayge
* Remove numpy dependencies
* Add warnings_disasm_current.txt
* my bad
* Update spec and undefined_syms
* Add z_eff_ss_hahen to pametfrog
* git subrepo pull (merge) --force tools/z64compress
subrepo:
subdir: "tools/z64compress"
merged: "163ca2af"
upstream:
origin: "https://github.com/z64me/z64compress.git"
branch: "main"
commit: "163ca2af"
git-subrepo:
version: "0.4.3"
origin: "https://github.com/ingydotnet/git-subrepo"
commit: "2f68596"
* Make z64compress print to stdout
* sneeky commit to update warnings tooling
* test
* Another test
* Mark fixing overlay reloc generating as a TODO
* Update warnings stuff
* Communicate the return code from running z64compress back to the Makefile through the wrapper
* Run formatter, remove extra commented copy of function
* Re-fix some includes
* Convert atan to hex to conform to decided style
* Some tidying up, remove c for fp and the other two handwritten code files
* BSS in z_collision_check & z_scene_proc
* add static back in
* Fix timerintr bss, add file to spec, some cleanup
* Remove externs
* Newline
* Readd enums
* Typo
* Colours
* Comments for hitmark enum values
Co-authored-by: EllipticEllipsis <73679967+EllipticEllipsis@users.noreply.github.com>
* Improvements and suggestions
* Organize and remove unused imports and use env for python3 scripts, delete unused overlay.py
Co-authored-by: angie <angheloalf95@gmail.com>
Co-authored-by: Elliptic Ellipsis <elliptic.ellipsis@gmail.com>
Co-authored-by: engineer124 <engineer124engineer124@gmail.com>
Co-authored-by: EllipticEllipsis <73679967+EllipticEllipsis@users.noreply.github.com>
2021-08-04 03:21:31 +00:00
# z64compress
`z64compress` is a program for compressing Zelda 64 roms: be they retail, hacked traditionally, or custom-built from the [`Ocarina of Time` ](https://github.com/zeldaret/oot ) or [`Majora's Mask` ](https://github.com/zeldaret/mm ) reverse engineering projects. It is written in highly efficient C and leverages the power of multithreading to make compression as fast as possible. To reduce overhead on subsequent compressions, an optional cache directory can be specified.
In addition to the default `yaz` , it supports some faster and more compact algorithms such as `lzo` , `ucl` , and `aplib` . In order to use these, grab patches or code from my [`z64enc` repository ](https://github.com/z64me/z64enc ).
If you add an algorithm, please make sure `valgrind` reports no memory leaks or other errors before making a pull request. Thank you!
2021-09-22 00:13:03 +00:00
(By the way, `valgrind` works better without the `-march=native -mtune=native` optimizations, so turn those off when testing `valgrind` .)
Overhaul the build system (#234)
* wip
* fix
* add disassembler
* Disasm builds OK
* Variable addends
* More wip
* Rodata migration implemented
* Cleanup old tools
* Try fix submodule -> subrepo merge
* git subrepo pull --force --remote=https://github.com/zeldaret/ZAPD.git tools/ZAPD
subrepo:
subdir: "tools/ZAPD"
merged: "602e609"
upstream:
origin: "https://github.com/zeldaret/ZAPD.git"
branch: "master"
commit: "602e609"
git-subrepo:
version: "0.4.3"
origin: "https://github.com/ingydotnet/git-subrepo"
commit: "2f68596"
* Builds again but assets are totally broken
* git subrepo pull --force tools/asm-processor
subrepo:
subdir: "tools/asm-processor"
merged: "1ffdb08a"
upstream:
origin: "https://github.com/simonlindholm/asm-processor.git"
branch: "master"
commit: "1ffdb08a"
git-subrepo:
version: "0.4.3"
origin: "https://github.com/ingydotnet/git-subrepo"
commit: "2f68596"
* More cleanup, move functions.txt and variables.txt to tools/disasm and rm tables
* rm z64compress in preparation for subrepo
* git subrepo clone (merge) https://github.com/z64me/z64compress.git tools/z64compress
subrepo:
subdir: "tools/z64compress"
merged: "eb11085c"
upstream:
origin: "https://github.com/z64me/z64compress.git"
branch: "main"
commit: "eb11085c"
git-subrepo:
version: "0.4.3"
origin: "https://github.com/ingydotnet/git-subrepo"
commit: "2f68596"
* Fix asset extraction
* Fix diff-init make rule
* Split code bss
* Split assumed linker bug padding from assembly files
* add filelists for mm.us.rev1
* Maybe working, but I'm not sure
* add overlays to spec
* Add rodata to actos
* Everything compiles
* Make a lot of C files for code
* Add almost every file in code to spec
* whoops
* 3 code files left
* add scenes to spec
* More progress on progress.py
* Fix skelanime in spec
* audio files!
* Fix merge issues
* Fix some C files in code
* Fix remaining code files
* Use existing O1 C files in spec
* reorder boot order in spec
* update spec
* fault.c
* Convert relocs on completed actors, fixbaserom uses current rom name
* more boot files
* Add VT macros and script
* finish already existing boot files
* most of libultra
* fix 64bits libultra files
* Use C files for libultra, wrap some functions in NON_MATCHING
* Remove duplicate of OS_CLOCK_RATE from fault.c
* C files for fbdemos
* delete dumb files
* bootstrap C files, still need to add them to the spec
* update fixbaserom
* boot OK?
* I forgot to commit the spec
* C for gamestates
* C for kaleido
* Change all includes to ""
* copy actor sizes script from oot
* I forgot to delete those files
* Basic C files for effects
* Add effects initvars names
* Remove mislabelled boot functions from header/txt
* Begin porting bootstrap_fx, some sizes
* Fix <>
* Fix enum
* Fix diff.py
* fix libultra stuff
* update regconvert
* update setup warnings
* add some missing ;
* Fix some makefile stuff and other fixes on some non_matching functions
* add executable flag in extract_baserom and fixbaserom
* fix relative path
* copy assist from oot
* fix map path
* another assist path fix
* Delete C files for handwritten files
* add code_801A51F0 to spec
* add gfxbuffers to spec
* Move rodata to top of each file when possible
* UNK_TYPEs for func_801A51F0
* Remove kaleido rodata from spec
* Update spec and undefined_syms for recent merge
* GCC warnings and fix errors in nonmatchings,
* round percentage numbers
* progress script: format changes
* progress: error on non-existing files
* fix warning in z_scene_table
* Match 2 nonmatchings in z_actor
* Warnings in lightswitch and invadepoh
* Fix warning in z_actor_dlftbls
* I though I fixed this one
* whoops
* Comment out CC_CHECK
* Removed redundant ultra64.h includes
* Update asm_processor, sorted boot_O1 into other folders, completed the fbdemo bootstrap, cleaned up undefined_syms
* Completed gamestates bootstrap
* Split kaleido_scope
* Remove section.h and segment.h, move keep object externs to a common location in variables.h
* Completed effects bootstrap
* Segmented address externs for effects, fbdemos, gamestates and kaleido
* Move actor data externs out of the if 0
* Segmented address externs for actors
* Prepare actionfunc detection
* fix script, how did it even work before
* Fix actionfunc script again, re-introduce some more intermediate prints to the disassembler
* Automated actionFunc detection in actors
* Segmented addresses from player .text
* rm old segment addrs script and fix build
* Move sizes folder to tools
* Make build.py executable
* New Jenkinsfile Prayge
* Remove numpy dependencies
* Add warnings_disasm_current.txt
* my bad
* Update spec and undefined_syms
* Add z_eff_ss_hahen to pametfrog
* git subrepo pull (merge) --force tools/z64compress
subrepo:
subdir: "tools/z64compress"
merged: "163ca2af"
upstream:
origin: "https://github.com/z64me/z64compress.git"
branch: "main"
commit: "163ca2af"
git-subrepo:
version: "0.4.3"
origin: "https://github.com/ingydotnet/git-subrepo"
commit: "2f68596"
* Make z64compress print to stdout
* sneeky commit to update warnings tooling
* test
* Another test
* Mark fixing overlay reloc generating as a TODO
* Update warnings stuff
* Communicate the return code from running z64compress back to the Makefile through the wrapper
* Run formatter, remove extra commented copy of function
* Re-fix some includes
* Convert atan to hex to conform to decided style
* Some tidying up, remove c for fp and the other two handwritten code files
* BSS in z_collision_check & z_scene_proc
* add static back in
* Fix timerintr bss, add file to spec, some cleanup
* Remove externs
* Newline
* Readd enums
* Typo
* Colours
* Comments for hitmark enum values
Co-authored-by: EllipticEllipsis <73679967+EllipticEllipsis@users.noreply.github.com>
* Improvements and suggestions
* Organize and remove unused imports and use env for python3 scripts, delete unused overlay.py
Co-authored-by: angie <angheloalf95@gmail.com>
Co-authored-by: Elliptic Ellipsis <elliptic.ellipsis@gmail.com>
Co-authored-by: engineer124 <engineer124engineer124@gmail.com>
Co-authored-by: EllipticEllipsis <73679967+EllipticEllipsis@users.noreply.github.com>
2021-08-04 03:21:31 +00:00
## Usage
This is a command line application. Learn from these common examples and adapt the arguments to your needs:
```
compressing oot debug
--in "path/to/in.z64"
--out "path/to/out.z64"
--mb 32
--codec yaz
--cache "path/to/cache"
--dma "0x12F70,1548"
--compress "9-14,28-END"
--threads 4
compressing oot ntsc 1.0
--in "path/to/in.z64"
--out "path/to/out.z64"
--mb 32
--codec yaz
--cache "path/to/cache"
--dma "0x7430,1526"
--compress "10-14,27-END"
--threads 4
compressing mm usa
--in "path/to/in.z64"
--out "path/to/out.z64"
--mb 32
--codec yaz
--cache "path/to/cache"
--dma "0x1A500,1568"
--compress "10-14,23,24,31-END"
--skip "1127"
--repack "15-20,22"
--threads 4
```
## Arguments
```
--in uncompressed input rom
--out compressed output rom
--matching attempt matching compression at the cost of
some optimizations and reduced performance
--mb how many mb the compressed rom should be
--codec currently supported codecs
yaz
ucl
lzo
aplib
* to use non-yaz codecs, find patches
and code on my z64enc repo
--cache is optional and won't be created if
no path is specified (having a cache
makes subsequent compressions faster)
* pro-tip: linux users who don't want a
cache to persist across power cycles
can use the path "/tmp/z64compress"
--dma specify dmadata address and count
--compress enable compression on specified files
--skip disable compression on specified files
--repack handles Majora's Mask archives
--threads optional multithreading;
exclude this argument to disable it
--only-stdout reserve stderr for errors and print
everything else to stdout
arguments are executed as they
are parsed, so order matters!
```
## Building
I have included shell scripts for building Linux and Windows binaries. Windows binaries are built using a cross compiler ([I recommend `MXE` ](https://mxe.cc/)).
Alternatively, a Makefile-based build system is provided. Choose the target platform with `make TARGET=linux64|linux32|win32` , default is linux64. If building for windows with a cross compiler, specify the compiler executable with `make TARGET=win32 CC=/path/to/executable` .