retdec/CHANGELOG.md

Changelog

dev

• Fix: Handle Intel MPX instructions (#1154, #1148, #1135).
• Fix: Make RetDec compilable by the new gcc-13 (#1149, #1153).

v5.0 (2022-12-08)

The major change:

• New Feature: RetDec is now a library (#779).
  • Related changes are the removal of retdec-decompiler.py (it is now a binary, e.g. retdec-decompiler.exe on Windows), retdec-bin2llvmir, retdec-llvmir2hll, and some other supportive functionality.
  • See an example in src/retdectool, or an actual implementation of RetDec executable in src/retdec-decompiler, to find out how to use RetDec library.

Other changes:

• New feature: Use GitHub Actions Continuous integration (#1053, #1125).
• New feature: Add checking for damaged (unloadable) ELF files (#1036, regression-tests #113).
• New feature: Parse various PE timestamps and make them available in Fileinfo (#1035, regression-tests #112).
• New feature: Generate ELF (import) symbol-related hashes, including VirusTotal compatible telfhash (#286, #936).
• New Feature: retdec-fileinfo can be configured via JSON file. See --fileinfo-config option for more details.
• Enhancement: Use Capstone v5 instead of v4 (#1059, #1124).
• Enhancement: Add prototypes for dynamically-linked functions without headers (#1092).
• Enhancement: Handle Procedure Linkage calls for 32bit x86 from gcc (#1088).
• Enhancement: Add DLL name from export directory to output (#1060).
• Enhancement: Updated YARA to v4.2.0-rc1 (#1061).
• Enhancement: Use Authenticode parser library instead of RetDec's own implementation (#1027, regression-tests #110).
• Enhancement: Remove --backend-aggressive-opts option and all the related code (#1016, #1032).
• Enhancement: Add SECURITY.md (#1018, #1025).
• Enhancement: Improve PE's .NET parsing - make it more aligned with parsing in YARA (#997, regression tests #106).
• Enhancement: Add signatureVerified flag to PE's digital signature entries (#994, regression tests #102).
• Enhancement: Add YARA signature for InnoSetup 6.1.0 (#989).
• Enhancement: Provide one-line-style values for digital signature's subjects and issuers (#956, #976, regression tests #92).
• Enhancement: Compute hashes of decrypted PE rich headers (#621, #945).
• Enhancement: Unified logging on stdout/stderr. Added option --silent. Printed text is colored only when output is a terminal (#791.
• Enhancement: Support all the CMake build types (i.e. Debug, Release, RelWithDebInfo and MinSizeRel) on all systems (#774).
• Enhancement: YARA updated to version 4.0.1 (#758), fixed Mach-O parsing issue (#283).
• Enhancement: Improved detection of many packers/installers/compilers in retdec-fileinfo, including Armadillo (#733), VMProtect (#734, #778), Petite (#735), Enigma (#741), ASPack (#743), Eziriz (#746), PyInstaller (#748), Astrum InstallWizard (#753), AutoHotKey (#756), AutoIt (#757), BAT to PE-EXE script compilers (#761), Bero (#764), CExe (#781), MoleBox (#815), Blizzard Protector (#845), CreateInstall installer (#852), FlyStudio installer (#853), Gentee installer (#855), Ghost installer and InnoSetup (#857, #899), InstallCreator (#804), Quick Batch installer (#864), Wise installer (#865), Viseman installer (#868), Setup Factory (#869), Xenocode Application Launcher #870, SmartInstall Maker (#871), and other improvements (#804, #831).
• Enhancement: Enable .NET module in RetDec's YARA (#747).
• Enhancement: Require OpenSSL as a prerequisite. It is no longer built by RetDec (#807).
• Enhancement: Replace RetDec's FilesystemPath implementation with C++ Filesystem library (#806).
• Enhancement: Added support for Ninja CMake generator (#8, #830).
• Enhancement: Removed copyrights from RetDec's outputs (#843).
• Enhancement: Add --version program option to all executable RetDec apps and add this info to retdec-fileinfo's verbose output as well (#926).
• Enhancement: Added support for new ELF UPX unpacking stubs (versions 3.93 - 3.96) (#929).
• Enhancement: Improved YARA rules for detection of the SHA-512 algorithm (#935).
• Enhancement: Improved PE Authenticode parsing (#902, #380).
• Fix: Fix parameter and return types for dynamically called functions (#1085).
• Fix: Disable memory-limiting capabilities on macOS because there is currently no working way of doing so (#1074, #1045, #379).
• Fix: Add OpenSSL 3.0 support (#1040, #1041).
• Fix: ImageLoader::Save() properly saves PE's Rich Header and section data (#1028, #1029).
• Fix: Check if data is not empty in .NET integer decoding functions (#1030).
• Fix: Stricter validation of PE signatures - they need to be outside of the image to be considered valid (#972, #986, regression tests #108).
• Fix: Do not provide entry point offset in case it doesn't exist (#962, #975, regression tests #101).
• Fix: Fix PE resource parsing issues (#963, #982, regression tests #105).
• Fix: Fix PE imports parsing issues (#1003, regression tests #107).
• Fix: Accept PDB info only if IMAGE_DEBUG_TYPE_CODEVIEW flag is set (#1004).
• Fix: Prevent PE delayed import parser to load garbage data (#981).
• Fix: Don't detect .NET structures that do not belong to the binary itself, but to the embedded binary (#967, #970, regression tests #90).
• Fix: Fixed handling of escaped characters in PE section names (#958, #979, regression tests #94).
• Fix: Fixed .NET's TypeLib ID parsing - add Parent relationship check (#966, #983, regression tests #96).
• Fix: Make error handling for PE resource directory in sync with YARA - i.e. behave as YARA does (#988, regression tests #98).
• Fix: Fixed memory leak in Fileformat library (#951, #984).
• Fix: Raise max length limit applied to PE symbol names (#957, #978, regression tests #93).
• Fix: Fixed parsing of junk data from PE resource table's type entry (#959, #974).
• Fix: Fixed PE rich header analysis algorithm (#973, #960, #965, regression tests #91).
• Fix: Arithmetic shift is no longer converted to signed division as these operations provide different output with negative numbers (#724).
• Fix: Fixed infinite looping during the copy-propagation optimization in llvmir2hll (#876).
• Fix: Fixed analyzed calling convention on MIPS architecture. Register F0 is used for floating point function return (#656).
• Fix: Fixed filtration to better handle functions with no arguments and therefore to reduce noise in output (#155).
• Fix: Fixed build on some systems by adding missing includes of <limits> into retdec-fileinfo (#745).
• Fix: Fixed two type errors in scripts/retdec-archive-decompiler.py (#759).
• Fix: Fixed runtime and memory use of retdec-fileinfo on PE samples having corrupted relocations (#872, #873).
• Fix: Fixed a corruption check for PE samples with invalid import thunks (#897, #917).
• Fix: Fixed recognition of very corrupted PE samples (#921).
• Fix: Fixed the recognition of the "RVA of the import name is invalid" corruption in PE samples (#1063).
• Fix: Fixed parsing of corrupted resources in retdec-fileinfo (#907, #911).
• Fix: MPRESS unpacker will now correctly copy resources, exports and other non-packed sections correctly.
• Fix: retdec-fileinfo.py is now usable even when decompiler is not installed.
• ... and many others.

v4.0 (2020-04-07)

• New Feature: Added support for decompilation of 64-bit ARM binaries (#268, #533, #550).
• New Feature: Added presentation of section and overlay entropy in retdec-fileinfo (#502, #507).
• New Feature: Added presentation of version info from PE file in retdec-fileinfo (#408, #519).
• New Feature: Added presentation of thread-local storage directory from PE file in retdec-fileinfo (#417, #523).
• New Feature: Added presentation of missing dependencies of PE files in retdec-fileinfo (#585).
• New Feature: Added presentation of anomalies of PE files in retdec-fileinfo (#415, #570).
• New Feature: Added heuristic detection of StarForce, SecuROM, SafeDisc, MPRMMGVA, ActiveMark, Petite, and RLPack (#600, #607, #615).
• New Feature: Added control flow related information to RetDec config (#646).
• New Feature: Added option to generate the decompilation results as JSON (JSON output file format). This output contains additional meta-information and can be conveniently consumed by 3rd-party tools.
• New Feature: Added a new library called retdec that lets you decompile the input into both LLVM IR module and structured (i.e. functions and basic blocks) Capstone disassembly. See the retdectool demo application.
• Enhancement: Improved handling of ELF object files and ELF thunks (implemented in PR #577, solved issues #184, #480, and partially solved #201).
• Enhancement: Demangler rewritten (#95).
• Enhancement: Added macOS and Linux (Ubuntu, Debian, Fedora) release builds (#526).
• Enhancement: Added support for using a local repository clone for RetDec external dependencies (#279).
• Enhancement: Parallelized compilation of YARA rules during installation (#540).
• Enhancement: Updated LLVM to version 8.0.0 (#110).
• Enhancement: Updated YARA to version 3.9 (#527).
• Enhancement: Updated OpenSSL to version 1.1.1c (#601). This fixes build of OpenSSL on macOS Mojave (#439).
• Enhancement: Added support for relocations that pair multiple R_MIPS_LO16 against a single R_MIPS_HI16 (#627, #628).
• Enhancement: Added handling of all x86 FPU instructions in assembly to LLVM IR translation (#394, #643).
• Enhancement: All registers are localized - i.e. transformed from global variables to local variables (#652). This significantly (20% on average) speeds up the decompilation process and greatly reduces noise in output.
• Enhancement: Added CMake options to build and install only specific targets (#510).
• Enhancement: Switched from C++14 to C++17 (#650).
• Enhancement: Replaced uses of mpark::variant from deps/variant with standard C++17 std::variant. Removed the variant dependency.
• Enhancement: Updated Yaramod to version v3.0.0 (#680). RetDec no longer requires Flex and Bison. This fixes #103.
• Enhancement: Take out most of the types from config library and place them to a separate common library that could be used across an entire RetDec source base (#686).
• Enhancement: retdec-fileinfo is now able to produce human-readable representation of a product name and VS version from Rich header (#691).
• Enhancement: Added a new corruption check into retdec-fileinfo that detects cut or zeroed digital signature (#719).
• Enhancement: Reduced RetDec's external dependencies:
  • The sources of the following 3rd-party projects were moved from their own repositories directly to the main RetDec repository (to /deps/): ELFIO, RapidJSON, TinyXML2.
  • The sources of the following Avast projects were moved from their own repositories directly to the main RetDec repository (to /src/): Yaracpp, PeLib.
  • The following 3rd-party dependencies use upstream project repositories, not modified Avast forks as before: Capstone, Yara.
  • The following dependencies are no longer needed: JsonCpp, Libdwarf, Libelf.
• Enhancement: Implemented proper RetDec installation (#648). It is now possible to easily use RetDec components in other CMake projects.
• Enhancement: Some optimizations in critical RetDec components (#731). It is however still often very slow on big inputs.
• Fix: Increased the limit for the number of entries in import directory when deciding whether a PE file is corrupted or not (avast/pelib#13).
• Fix: Fixed build on BSD systems (#598).
• Fix: Resources which are located in the different section than resource tree are now properly parsed (#596).
• Fix: Version information which contained strings shorter than reported are now properly parsed (#596).
• Fix: Fixed crashes of retdec-fileinfo when analyzing ELF samples containing invalid ranges (#521).
• Fix: Fixed crashes of retdec-unpacker when trying to unpack corrupted ELF samples having incorrect size of additional data (#582).
• Fix: Fixed several Mach-O parsing crashes (#581, #561, #568).
• Fix: Fixed import table hashes computation - hashes are no longer produced from empty strings (#460).
• Fix: Fixed build on Microsoft Windows via MSYS2 (#606).
• Fix: Fixed build on macOS Mojave by updating OpenSSL and using xcrun (#439).
• Fix: Fixed computation of the "RVA of the import name is invalid" loader error (avast/pelib#11).
• Fix: Fixed computation of the "Import directory is cut" loader error (avast/pelib#17).
• Fix: Export ordinals are now correctly calculated as relative to the base (#612, avast/pelib#10).
• Fix: Fixed crash in the decoding phase (#637, #641).
• Fix: Fixed global variable naming issue (#636, #645).
• Fix: Fixed binary to LLVM IR translation of some MIPS instructions (#633), and made the translation process less error prone altogether (#672).
• Fix: Fixed incorrect translation of PHI nodes in llvmir2hll (#658).
• Fix: Fixed the build of LLVM when having OCaml installed in your system (#701).

v3.3 (2019-03-18)

• New Feature: Added basic support of 64-bit x86 architecture (#9, #513).
• New Feature: Added presentation of imported types and TypeRef hashes for .NET binaries (#363, #364, #428).
• New Feature: Added presentation of metadata from binaries written in Visual Basic and detection of P-code (#138, #440).
• New Feature: Added computation and presentation of icon hashes for exact and also similarity matching in PE files (#339).
• Enhancement: Distribute YARA rules in a text form in the RetDec support package (retdec-support #3).
• Enhancement: Updated YARA to version 3.8.1 (#218).
• Enhancement: Made --generate-log option of retdec-decompiler.py work on macOS (#383, #450).
• Enhancement: Replaced recursion with iterative implementation in x87 FPU analysis in retdec-bin2llvmir (#450).
• Enhancement: The new LLVM IR to BIR converter is now the default (and only) back-end's converter. In most cases, this improves code structure and significantly speeds up decompilations (#211, #508, #509).
• Enhancement: The fileformat library, and all its object file modules, accept both std::istream and (data, size) pair, in addition to the original input file path.
• Enhancement: Reduced the needed stack space in retdec-llvmir2hll (#492, #495).
• Enhancement: Added support for build and run on FreeBSD and potentially on other BSD OSes (#476).
• Enhancement: It is possible to use local PeLib directory instead of remote revision via CMake variable PELIB_LOCAL_DIR. This is convenient when modifying both PeLib and RetDec at the same time.
• Enhancement: Improved detection of needed libraries and imported/external functions in ELF binaries (#457).
• Enhancement: Added semantics for more MIPS instructions.
• Enhancement: Capstone2LlvmIr library refactoring (#115).
• Enhancement: Removed the build and runtime dependency on ncurses/libtinfo (#409).
• Enhancement: Add a check into our scripts that they are run from an installation directory and not from the scripts directory (#418).
• Enhancement: Improved corruption checks in retdec-fileinfo to recognize cut PE files which are still loadable (#463).
• Enhancement: Redesign output files naming scheme (#132).
• Fix: Fixed false COFF file format detections (#421, #431).
• Fix: Fixed LLVM IR syntax error: Global variable initializer type does not match global variable type (#436).
• Fix: Fixed translation of x86 sbb instruction (#401).
• Fix: Fixed fileinfo crash during Asn1Sequence initialization when parsing PE certificates (#256).
• Fix: Fixed fileinfo crash during reconstruction of .NET types (#458, #511).
• Fix: Fixed generation of MIPS branch instructions (#88).
• Fix: Fixed generation of empty if blocks in C output (#83).
• Fix: Fixed decompilation of simple x86 system calls (#24).
• Fix: Fixed potential infinite looping in llvmir2hll's copy propagation pass (#479).
• Fix: Fixed FilesystemPath::isFile() (#490, #491).
• Fix: Fixed retdec-signature-from-library-creator.py when there is a lot of input files by adding an option to retdec-bin2pat to have the objects list passed through a text file instead of via program arguments (#472, #484).
• Fix: Stricter rules for PE section names (#451).
• Fix: Fixed incorrect return code of bin2pat that caused signature-from-library-creator.py to silently fail on error (#473, #474).
• Fix: Fixed installation when Python is in a path containing spaces (#441).
• Fix: Fixed handling of pointers with segment overrides, including loads/stores from/to zero (null) pointers (#41, #169, #347, #376, #391).
• Fix: Fixed translation of x86 FPU instructions to LLVM IR (#293).

v3.2 (2018-08-16)

• New Feature: Converted shell scripts to Python scripts so that Windows users no longer have to install MSYS2 in order to run RetDec (#338, #147).
• New Feature: Added generation of export-table hashes into retdec-fileinfo (#121, #321).
• New Feature: Automatically generate and publicly host an up-to-date Doxygen documentation (#20).
• Enhancement: Suppress superfluous ranlib warnings about static libraries having no symbols on macOS (#271, #349).
• Fix: Fixed assertions in statically linked code recognition (#333).
• Fix: Fixed aborts due to assertions during decoding of some MIPS binaries (#335).
• Fix: Fixed a memory leak when parsing Mach-O files (#331).

v3.1 (2018-06-07)

• New Feature: retdec-fileinfo is now able to detect when a PE file is corrupted and cannot be loaded (#281).
• New Feature: Added a new tool: retdec-getsig. It can be used for creating signatures of packers, compilers, and other tools.
• New Feature: The number of bytes read from the input file's entry point by retdec-fileinfo is now configurable with the --ep-bytes option.
• Enhancement: Complete rewrite of binary to LLVM IR decoding phase (#116).
• Enhancement: Added reference checks to statically linked code detection (#113).
• Enhancement: Speeded up RetDec rebuild and installation by disabling forced reconfiguration of LLVM (#294).
• Enhancement: Added new OS/ABI and tool detections for ELF files (#244).
• Enhancement: Improved support for analysis of ELF core files by retdec-fileinfo (#142).
• Enhancement: Added support for limiting overall memory when running decompilations and tools (#270, #290). By default, decompilations are now run with limited memory (half of system RAM) to prevent "black screens" (mostly on Windows). Use --no-memory-limit to override.
• Enhancement: On macOS, you no longer need to have GNU coreutils in PATH to build and install RetDec. GNU coreutils are still needed to run RetDec, though.
• Enhancement: Import-table hashes generated by retdec-fileinfo are now compatible with import-table hashes from YARA/pefile (#246).
• Enhancement: Tool retdec-macho-extractor can now extract objects from non-archive Mach-O universal binaries (#125).
• Enhancement: References to ticket numbers from our internal issue tracking system were replaced by short descriptions in the retdec-regression-tests repository (retdec-regression-tests #1).
• Enhancement: Added a missing license for the retdec-support repository (retdec-support #1).
• Enhancement: Better detection of tools: new signatures and heuristics. YARA signatures are compiled now.
• Enhancement: Added Travis and AppVeyor continuous integration builds (#2).
• Enhancement: Build with -std=c++14 instead of -std=gnu++14 with GCC on Linux (#76).
• Enhancement: Speeded up build by skipping compilation of unnecessary dependencies (e.g. unused LLVM libraries, tools, and examples).
• Enhancement: OpenSSL is now automatically built only if it is not found in your system.
• Enhancement: Added support for a system-wide installation (#94).
• Enhancement: Prefixed all the installed binaries and scripts with retdec- (#70). Also, some tools were renamed to make their names more uniform.
• Enhancement: Got rid of all git submodules (#92, #93). Moved sources of all RetDec-related repositories to this main repository. Third-party dependencies are downloaded and built via CMake's external projects. This allows us to have e.g. only a single copy of LLVM (#14) and not require a recursive clone (#48, #68, #72).
• Enhancement: Set a proper rpath during installation on Linux and macOS (#77, #100). This allows us to move the installation directory after the installation into another location.
• Enhancement: Added community support for building and running RetDec inside Docker (#60).
• Enhancement: Decrease the default timeout when downloading the support package during installation (#6).
• Enhancement: Any shell can be used to install the decompiler, not just Bash.
• Enhancement: Added unofficial support for macOS build (#7).
• Enhancement: Allow 32b versions of bin2llvmir and llvmir2hll on Windows access more memory (#7).
• Enhancement: Added a method in loader::Image to obtain segment content as a raw data pointer.
• Enhancement: retdec-fileinfo now prints raw bytes of Rich Header in the JSON format (#288).
• Enhancement: Delayed imports in PE files are now distinguished from non-delayed imports in the output from retdec-fileinfo by a boolean flag (#287).
• Fix: Add a missing .c extension to files generated by retdec-archive-decompiler.sh (#261.
• Fix: Fixed build of yaracpp on 32b Unix-like operating systems (#299).
• Fix: Fixed parsing of PE files having corrupted import tables (#101).
• Fix: Fixed parsing of delayed imports by ordinals in PE files (#282).
• Fix: Fixed ordering of detected tools (e.g. compilers and packers) on systems whose std::sort() is not stable (#262).
• Fix: When running RetDec on macOS, gtimeout is now used instead of timeout (#260). This fixes the following runtime error when running retdec-archive-decompiler.sh: The timeout command is required but it is not available.
• Fix: When running RetDec on macOS, greadlink is now used instead of readlink. This fixes runtime errors of the form readlink: illegal option -- e.
• Fix: retdec-decompiler.sh on macOS no longer requires the GNU version of sed (#257).
• Fix: #!/usr/bin/env bash is now used instead of #!/bin/bash to run our scripts (#258).
• Fix: Fixed retdec-fileinfo crashes when verifying digital signature of PE files (#87).
• Fix: Fixed infinite loop in COFF word length detection for rare cases (#242).
• Fix: Fixed several ELF bugs causing crashes (#239, #240, #241, #248).
• Fix: Fixed unit-tests discovery in retdec-tests-runner.sh on macOS (#238).
• Fix: Non-printable characters in ELF .dynamic section output are now replaced with hexadecimal codes (#82).
• Fix: Fix for several segmentation faults in ELF parsing module (#89).
• Fix: Added a workaround for a GCC 5 compilation bug (#231).
• Fix: Fix LLVM (and therefore RetDec) build on systems with architecture other than x86 (llvm #3).
• Fix: Valid Mach-O x64 relocations are no longer ignored.
• Fix: Only a single copy of LLVM (and all other components) is kept (#14).
• Fix: RetDec works even if it is installed to a directory which have whitespaces in its path.
• Fix: Reduced the length of build paths to external projects (#61).
• Fix: Build of googletest with VS 2017 (#55).
• Fix: Build of retdec-config when two different compilers are employed (#52).
• Fix: Build of the llvm submodule with VS 2017 when DIA SDK is installed (#61).
• Fix: Ordering of compiler detections (#39).
• Fix: Remove duplicate lib prefix when installing libdwarf libraries (#31).
• Fix: When installing the decompiler, do not remove the entire share directory (#12).
• Fix: Improve OS type detection when installing the decompiler.
• Fix: Remove useless OS type detection when running decompilations (#10).
• Fix: Filesystem path in utils now returns correct information when it is appended with another path.
• Fix: Plain output of fileinfo now escapes non-printable characters in subject/issuer name/organization of PE certificates (#253).
• Fix: Invalid dates are no longer shown in the output of fileinfo (#251).
• Fix: Fixed crash of fileinfo when accessing slightly corrupted security directory (#255, #250).
• Fix: Delayed imports are now ignored when calculating import-table hashes for PE files (#287).
• Fix: Import-table hashes for Mach-O binaries are now always generated even if commands for library loading are not ordered (#285).
• Fix: OpenSSL can now be built on ARM architectures (Linux and Windows) and other non-recognized architectures (Linux only) (#299).
• Fix: Decompilation in raw mode (--mode raw) no longer removes the original input file when cleanup option is used (--cleanup) (#309).
• Fix: Retdec can now be cross-compiled (yaracpp #2).

v3.0 (2017-12-13)

Initial public release.