mm/tools/z64compress_wrapper.py
Tharo 7743e5a2c4
Overhaul the build system (#234)
* wip

* fix

* add disassembler

* Disasm builds OK

* Variable addends

* More wip

* Rodata migration implemented

* Cleanup old tools

* Try fix submodule -> subrepo merge

* git subrepo pull --force --remote=https://github.com/zeldaret/ZAPD.git tools/ZAPD

subrepo:
  subdir:   "tools/ZAPD"
  merged:   "602e609"
upstream:
  origin:   "https://github.com/zeldaret/ZAPD.git"
  branch:   "master"
  commit:   "602e609"
git-subrepo:
  version:  "0.4.3"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "2f68596"

* Builds again but assets are totally broken

* git subrepo pull --force tools/asm-processor

subrepo:
  subdir:   "tools/asm-processor"
  merged:   "1ffdb08a"
upstream:
  origin:   "https://github.com/simonlindholm/asm-processor.git"
  branch:   "master"
  commit:   "1ffdb08a"
git-subrepo:
  version:  "0.4.3"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "2f68596"

* More cleanup, move functions.txt and variables.txt to tools/disasm and rm tables

* rm z64compress in preparation for subrepo

* git subrepo clone (merge) https://github.com/z64me/z64compress.git tools/z64compress

subrepo:
  subdir:   "tools/z64compress"
  merged:   "eb11085c"
upstream:
  origin:   "https://github.com/z64me/z64compress.git"
  branch:   "main"
  commit:   "eb11085c"
git-subrepo:
  version:  "0.4.3"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "2f68596"

* Fix asset extraction

* Fix diff-init make rule

* Split code bss

* Split assumed linker bug padding from assembly files

* add filelists for mm.us.rev1

* Maybe working, but I'm not sure

* add overlays to spec

* Add rodata to actos

* Everything compiles

* Make a lot of C files for code

* Add almost every file in code to spec

* whoops

* 3 code files left

* add scenes to spec

* More progress on progress.py

* Fix skelanime in spec

* audio files!

* Fix merge issues

* Fix some C files in code

* Fix remaining code files

* Use existing O1 C files in spec

* reorder boot order in spec

* update spec

* fault.c

* Convert relocs on completed actors, fixbaserom uses current rom name

* more boot files

* Add VT macros and script

* finish already existing boot files

* most of  libultra

* fix 64bits libultra files

* Use C files for libultra, wrap some functions in NON_MATCHING

* Remove duplicate of OS_CLOCK_RATE from fault.c

* C files for fbdemos

* delete dumb files

* bootstrap C files, still need to add them to the spec

* update fixbaserom

* boot OK?

* I forgot to commit the spec

* C for gamestates

* C for kaleido

* Change all includes to ""

* copy actor sizes script from oot

* I forgot to delete those files

* Basic C files for effects

* Add effects initvars names

* Remove mislabelled boot functions from header/txt

* Begin porting bootstrap_fx, some sizes

* Fix <>

* Fix enum

* Fix diff.py

* fix libultra stuff

* update regconvert

* update setup warnings

* add some missing ;

* Fix some makefile stuff and other fixes on some non_matching functions

* add executable flag in extract_baserom and fixbaserom

* fix relative path

* copy assist from oot

* fix map path

* another assist path fix

* Delete C files for handwritten files

* add code_801A51F0 to spec

* add gfxbuffers to spec

* Move rodata to top of each file when possible

* UNK_TYPEs for func_801A51F0

* Remove kaleido rodata from spec

* Update spec and undefined_syms for recent merge

* GCC warnings and fix errors in nonmatchings,

* round percentage numbers

* progress script: format changes

* progress: error on non-existing files

* fix warning in z_scene_table

* Match 2 nonmatchings in z_actor

* Warnings in lightswitch and invadepoh

* Fix warning in z_actor_dlftbls

* I though I fixed this one

* whoops

* Comment out CC_CHECK

* Removed redundant ultra64.h includes

* Update asm_processor, sorted boot_O1 into other folders, completed the fbdemo bootstrap, cleaned up undefined_syms

* Completed gamestates bootstrap

* Split kaleido_scope

* Remove section.h and segment.h, move keep object externs to a common location in variables.h

* Completed effects bootstrap

* Segmented address externs for effects, fbdemos, gamestates and kaleido

* Move actor data externs out of the if 0

* Segmented address externs for actors

* Prepare actionfunc detection

* fix script, how did it even work before

* Fix actionfunc script again, re-introduce some more intermediate prints to the disassembler

* Automated actionFunc detection in actors

* Segmented addresses from player .text

* rm old segment addrs script and fix build

* Move sizes folder to tools

* Make build.py executable

* New Jenkinsfile Prayge

* Remove numpy dependencies

* Add warnings_disasm_current.txt

* my bad

* Update spec and undefined_syms

* Add z_eff_ss_hahen to pametfrog

* git subrepo pull (merge) --force tools/z64compress

subrepo:
  subdir:   "tools/z64compress"
  merged:   "163ca2af"
upstream:
  origin:   "https://github.com/z64me/z64compress.git"
  branch:   "main"
  commit:   "163ca2af"
git-subrepo:
  version:  "0.4.3"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "2f68596"

* Make z64compress print to stdout

* sneeky commit to update warnings tooling

* test

* Another test

* Mark fixing overlay reloc generating as a TODO

* Update warnings stuff

* Communicate the return code from running z64compress back to the Makefile through the wrapper

* Run formatter, remove extra commented copy of function

* Re-fix some includes

* Convert atan to hex to conform to decided style

* Some tidying up, remove c for fp and the other two handwritten code files

* BSS in z_collision_check & z_scene_proc

* add static back in

* Fix timerintr bss, add file to spec, some cleanup

* Remove externs

* Newline

* Readd enums

* Typo

* Colours

* Comments for hitmark enum values

Co-authored-by: EllipticEllipsis <73679967+EllipticEllipsis@users.noreply.github.com>

* Improvements and suggestions

* Organize and remove unused imports and use env for python3 scripts, delete unused overlay.py

Co-authored-by: angie <angheloalf95@gmail.com>
Co-authored-by: Elliptic Ellipsis <elliptic.ellipsis@gmail.com>
Co-authored-by: engineer124 <engineer124engineer124@gmail.com>
Co-authored-by: EllipticEllipsis <73679967+EllipticEllipsis@users.noreply.github.com>
2021-08-03 23:21:31 -04:00

105 lines
3.5 KiB
Python

#!/usr/bin/env python3
#
# z64compress wrapper for decomp projects
# Arguments: <rom in> <rom out> <elf> <spec> [cache directory] [num threads]
#
import argparse, itertools, subprocess, sys
from elftools.elf.elffile import ELFFile
from elftools.elf.sections import SymbolTableSection
# Args from command line
parser = argparse.ArgumentParser(description="Compress rom produced by the OoT Decomp project")
parser.add_argument("in_rom", help="uncompressed input rom filename")
parser.add_argument("out_rom", help="compressed output rom filename")
parser.add_argument("elf", help="path to the uncompressed rom elf file")
parser.add_argument("spec", help="path to processed spec file")
parser.add_argument("--cache", help="cache directory")
parser.add_argument("--threads", help="number of threads to run compression on, 0 disables multithreading")
parser.add_argument("--mb", help="compressed rom size in MB, default 32")
parser.add_argument("--matching", help="matching compression, forfeits some useful optimizations", action="store_true")
parser.add_argument("--stderr", help="z64compress will write its output messages to stderr instead of stdout", action="store_true")
args = parser.parse_args()
IN_ROM = args.in_rom
OUT_ROM = args.out_rom
STDOUT = not args.stderr
elf_path = args.elf
CACHE_DIR = args.cache
N_THREADS = int(args.threads or 0)
MB = int(args.mb or 32)
matching = args.matching
# Get segments to compress
spec = ""
with open(args.spec, "r") as infile:
spec = infile.read()
def ranges(i):
for _, b in itertools.groupby(enumerate(i), lambda pair: pair[1] - pair[0]):
b = list(b)
yield b[0][1], b[-1][1]
compress_segments = []
shift = 0
for i,segment in enumerate(spec.split("beginseg\n")[1:],0):
directives = segment.split("endseg")[0].split("\n")
for directive in directives:
directive = directive.strip()
if directive.startswith("flags") and "NOLOAD" in directive:
shift += 1
elif directive.startswith("compress"):
compress_segments.append(i - shift)
compress_segments = list(ranges(compress_segments))
COMPRESS_INDICES = ",".join([f"{start}-{end}" if start != end else f"{start}" for start,end in compress_segments])
# Find dmadata
def get_dmadata_start_len():
dmadata_start = -1
dmadata_end = -1
with open(elf_path, "rb") as elf_file:
elf = ELFFile(elf_file)
for section in elf.iter_sections():
if not isinstance(section, SymbolTableSection):
continue
for sym in section.iter_symbols():
if sym.name == "_dmadataSegmentRomStart":
dmadata_start = sym['st_value']
elif sym.name == "_dmadataSegmentRomEnd":
dmadata_end = sym['st_value']
if dmadata_start != -1 and dmadata_end != -1:
break
assert dmadata_start != -1
assert dmadata_end != -1
return dmadata_start, (dmadata_end - dmadata_start)//0x10
DMADATA_ADDR, DMADATA_COUNT = get_dmadata_start_len()
# Run
cmd = f"./tools/z64compress/z64compress --in {IN_ROM} --out {OUT_ROM}{' --matching' if matching else ''} \
--mb {MB} --codec yaz{f' --cache {CACHE_DIR}' if CACHE_DIR is not None else ''} --dma 0x{DMADATA_ADDR:X},{DMADATA_COUNT} \
--compress {COMPRESS_INDICES}{f' --threads {N_THREADS}' if N_THREADS > 0 else ''}{f' --only-stdout' if STDOUT else ''}"
print(cmd)
try:
subprocess.check_call(cmd, shell=True)
except subprocess.CalledProcessError as e:
sys.exit(e.returncode)