Commit Graph

26 Commits

Author SHA1 Message Date
Peter Matula
81b0fd26bb deps/capstone: fix lib/lib64 controversy 2020-07-30 13:48:23 +02:00
Matula Peter
7a9ddedcd1 deps: refactor to enable Ninja build 2020-07-30 13:48:23 +02:00
Peter Matula
1771ddebf4
Issue 774 (#785)
* deps/cmake: fix #774 using generator expressions

The problem was that our previous solution could handle only Release or Debug config. The new solution can handle any configuration and is simpler.

* deps/yara: allow only Debug or Release config

* CHANGELOG.md: add entry for #774

* Revert "cmake: replace set_property() with target_{link,compile}_options()"

This reverts commit 7fa4753a9a.

* llvmir2hll: replace set_property() with target_compile_options()
2020-06-12 10:09:24 +02:00
Peter Matula
5281b06dc8
Retdec is a library now (#779)
* llvmir-emul: do not include llvm/Support/PluginLoader.h

This must not be included in a library.

* bin2llvmirtool: simplify

* llvmir2hlltool: do not include llvm/Support/PluginLoader.h

It is not needed and it is dangerous because it contains "load" option which can be included only once.

* bin2llvmir/providers/config: use params output file to generate config

* config/parameters: add more params

* bin2llvmir: add config-generator pass

* retdec/retdec: add super ugly decompilation API

* stacofin: stricter signature filtering

This is still very shitty. A better solution would be using some metadata in signature files - not the current signature-file-path-based filtering.

* progress: all test from "integration.ack" work

* progress

* progress

* do not manually create passes in code, use just the pass list

* create LlvmIr2Hll pass in llvmir2hll lib

* progress

* add decompiler-config.json

* aggregate LLVM passes

* add -k option

* bin2llvmir/config: fix Config::empty()

* bin2llvmir/unreahable_funcs: use params to disable opt

* retdec-decompiler: add more command line options

* progress

* all regression tests pass

* src/configtool: remove, not needed anymore

* config: remove isFromIda flag

* config: remove unused exceptions

* configL fix exceptions

* config: remove frontend version entry

* config: remove some duplicate values

* config: refactor

* config: refactor

* bin2llvmir: fix #301, fix #750, proper removal of artificial undef functions

* deps/llvm: update ref to fix gcc 10 compilation error

* deps/llvm: enable exeptions and RTTI

* progress

* remove debug msgs

* tests/debugformat_tests: fix compilation

* replace retdec-decompiler.py with retdec-decompiler

* retdec-decompiler: return decompilation error code

* tests/bin2llvmir/unreachable_funcs: fix JSON configs

* progress

* llvmir2hll: remove code specific for Python output HLL

* llvmir2hll: fix JSON output generation

* progress

* progress

* progress

* remove bin2llvmirtool and llvmir2hlltool

* refactor

* tests/bin2llvmir/x87_fpu: fix compilation

* unpackertool: do not build unpaker plugins separatelly

* scripts: return retdec-fileinfo.py back, some reg tests need it

* bin2llvmir: fix doxygen warnings

* set CMAKE_POSITION_INDEPENDENT_CODE and propagate it to deps

* Win: macOS: link llvmir2hll to decompiler target

* bin2llvmir/lti: fix pat filtering on windows

* retdec-decompiler: increase windows stack size

Co-authored-by: Peter Kubov <peter.kubov@avast.com>
2020-06-05 07:42:46 +02:00
Peter Matula
9594cef158 deps: install inludes and libs to the same location as native retdec components 2020-04-06 20:26:21 +02:00
Peter Matula
8fea90b2f7
New build system (#730)
* something working

* progress

* progress

* progress

* cmake: fix common and add ctypes

* src/serdes: new build system

* new build system for pdbparser

* new build system for yaracpp

* new build system for crypto component

* new build system for config

* refactor cmake for retdec-configtool

* new build system for idr2pat

* new build system for ar-extractor and its LLVM dependency

* refactor the new build system

* new build for ar-extractortool

* new build system for macho-extractor and macho-extractortool

* new build for ctypesparser, demangler, and demanglertool

* new build for llvm-support and llvmir-emul

* new build for capstone2llvmir and capstone2llvmirtool

* new build for fileformat, loader, cpdetect, fileinfo

* new build for most remaining modules

* cmake/options.cmake: fix debugformat enable settings

* some build system fixes

* cmake/options.cmake: make RETDEC_ENABLE_ALL into option

this allows us to set it to ODD and build only documentation

* tests: new build system for unit tests

* pelib: fix doxygen comment

* retdec/retdec: remove unused include of llvm-support/diagnostics.h

* fileformat: do not include openssl in headers

* cmake: make install paths relative.

When they are absolute, then all the paths in instaled cmake scripts are absolute - install dir could not be moved.

* deps/yara: refactor cmake to properly install libs

* deps/yara: small cosmetic changes

* deps/llvm: refactor cmake to properly install libs

* deps/capstone: refactor cmake to properly install libs

* deps: refactor cmake files

* deps: refactor cmake files

* deps/yaramod: refactor cmake to properly install libs

* CMakeLists.txt: fix files installation

* config/retdec-config-config.cmake: fix required component order

Looks like this may matter (on some machines or cmake version).

* deps/llvm: fix include installation

* fileformat/cmake: do not use openssl-crypto

For some reasons, this can cause linking problems on some machines.

* deps/yaramod: fix link library order

* deps/googletests: fix target link library order

* rtti-finder: make deps PUBLIC.

I have no idea why it doesn't work with PRIVATE.

* deps/yaramod: fix formatting

* cmake: more refactorization...

1. Protect against including the same target file multiple times. It looks like this was not a problem, but who knows.
2. Use find_package() instead of find_dependency(). It looks like the later does not work correctly for components on some CMake versions.

* cmake: do not create version files for all components

Only the main RetDec version file is needed.

* cmake: propagate current project version to package config files

It is used to find the same version retdec components.

* CMakeLists.txt: replace AnyNewerVersion -> ExactVersion in compatibility settings

Since we probably will make breaking changes, make retdec installation compatible only with the same version.

* cmake: remove all COMPONENT options, these are not really needed

* cmake: move dependency finders inside double target protection conditions

* cmake: prefix all deps targets with retdec::deps::

* fileinfo: add openssl-crypto dep

* Revert "fileinfo: add openssl-crypto dep"

This reverts commit e1645d7fd5.

* remove RETDEC_FORCE_OPENSSL_BUILD option, always build OpenSSL

* cmake: refactorization

* deps/whereami: fix include paths

* deps/tinyxml+whereami: fix included dirs

* deps/yaramod: fix pog_fmt lib installation

* deps/whereami: properly install target, not as part of utils.

Previous version should work, but there is a CMake bug in older versions which screws it up on Windows and macOS.

* crypto: link retdec::deps::openssl-crypto as PUBLIC

* bin2llvmir/providers/debugformat: fix segfault

Co-authored-by: Peter Matula <imatula@fit.vutbr.cz>
2020-04-05 16:50:56 +02:00
Peter Matula
9280a7558f cmake: add RETDEC_MSVC_STATIC_RUNTIME optioni
Default = OFF, if ON whole RetDec is build against windows static
runtime libs.
2020-03-23 10:01:03 +01:00
Matula Peter
cb019aa1f5 deps: move deps' URL and hashes to separate cmake file. 2020-01-27 10:13:41 +01:00
Matula Peter
07b938c114 deps/capstone: use the upstream Capstone repo, not avast fork. 2020-01-22 13:02:53 +01:00
Petr Zemek
0456a3d0b3 Remove trailing whitespace from all source files.
Reasons (in Czech): https://cs-blog.petrzemek.net/2011-09-09-proc-se-zbavovat-nadbytecnych-bilych-znaku
2019-05-30 13:39:34 +02:00
Peter Matula
fd3ee25205 Merge branch 'master' into arm64
# Conflicts:
#	deps/capstone/CMakeLists.txt
2019-05-27 10:28:42 +02:00
Peter Matula
658bb9d545 deps: fix #279, support for using local repository clones of RetDec deps 2019-04-25 14:08:13 +02:00
MatejKastak
3ab9797b53 Arm64 (#550)
* Arm64: System registers + bariers + Pseudo for vector registers

* Arm64: Neg instruction can take d registers for some reason

* Arm64: Helper functions to extract value from vector registers

- those are used in simple mov instructions or sometimes in pseudo

* Arm64: Regular ADD can take FP operands + test

- include function decl in header from last commit

* Arm64: PSTATE operands + More systemregs

- fixed FMinMax and Movi vector variants to generate psuedo

* Arm64: Create missing register type

* Arm64: Generate all conditional codes

- Fixed generation of AL and NV conditions

* Arm64: Cond branch tests

- altered the definition of conditional instruction to be only true if
  ARM64_CC_INVALID, and generate allways true for AL and NV

* Arm64: Missing B16 register

* Arm64: Temporary solution for Msl shift and stopped ignoring DMB ins

* Arm64: UXTX + SXTX extensions fix

* Arm64: try to solve unhandled regs better

* Arm64: Add and sub can have fp registers as operands

For some reason this is valid and the operation is addition integerwise

* Arm64: Removed debug output
2019-04-17 11:50:05 +02:00
Petr Zemek
5b478c72df Fix links to repositories after the move from avast-tl to avast.
We have moved all of our repositories in https://github.com/avast-tl to
https://github.com/avast.
2019-04-17 09:09:58 +02:00
MatejKastak
f07407f006 Aarch64 decompilation (#533)
* Capstone2llvmirtool default basic modes for architectures

Run tool with reasonable Capstone basic modes for specified architecture.
Default values are as follows:
-a arm   : CS_MODE_ARM
-a arm64 : CS_MODE_ARM [looks like keystone doesn't like this]
-a mips  : CS_MODE_MIPS32
-a x86   : CS_MODE_32
-a ppc   : CS_MODE_32
-a <rest>: CS_MODE_LITTLE_ENDIAN

* Base for the ARM64 translator

- register maps(_reg2type)
- instructions map(_i2fm)
Modified ARM Translator unit, Work in progress.

* Fix the cs_reg_name

- register name could not be found because of the wrong cs_arch in constructor

* Add ARM64 support for capstone dependency

- capstone was configured without the ARM64 support, this caused
  cs_open to fail

* Temporary solution to call translate function

* Status register and program counter added to environment

- flags from status register added to arm64 env
- program counter added to arm64 env

* Methods store/load registers/operands skeletons + add instruction

- basic implementation of functions needed for loading and storing operands
- translateAdd is for testing purposes

* Store instruction base

- started implementation of MEM operand type
- Store register instruction translation method
e.g. retdec-capstone2llvmir -a arm64 -t 'str x0, [x1]'

* Operand shifts ported from ARM and MOV instruction tranlation

- MOV, MVN and MOVZ instructions
- operand shift functions moved and changed for ARM64
- instructions like 'movz x0, #3 LSL 16' work now

* Arm64 - tests ported from Arm

- test framework capstone2llvmirtranslator
- first INS_ADD test
- cmake compilation

* Basic MOV tests

- MOV, MOVZ

* Test for STR instruction and test header comments

* STP instruction + tests, pc in new enum, get op addr function

- Store pair instruction{pre-index, post-index, signed-offset}
- test for all cases except 32bit operands
- pc moved to its own enum
- generateGetOperandAddr to generate address from instruction operand

* LDR + STR, LDR tests from ARM, LDP stub

- LDR{pre-index, post-index, signed-offset} instruction implemented
- STR{pre-index, post-index, signed-offset} instruction implemented
- LDR tests ported from ARM
- LDP todo

* Implemented parent register handling

- Register parent map
- Storing registers
- Loading registers
- Headers

- Need more changes to conversions, I think 'mov w0, #3' zeroes out
  the upper 32bits of x0 register. But need to investigate further.

* LLVM data layout modified for ARM64

- taken from uname -a in qemu arm64 machine
Linux debian-aarch64 4.9.0-4-arm64 #1 SMP Debian
4.9.65-3+deb9u1 (2017-12-23) aarch64 GNU/Linux

* Removed useless debug output

* getCarryRegister for ARM64 fixed

* Store register ZEXT_TRUNC, 32bit tests baseline + tests

- when writing value to 32bit reg the 64bit, the value is zero
  extended to the vhole register
- parent register mapping enabled in tests
- 32bit version of tests

* Zero extension tests for ADD and MOV 32bit variants

* Implemented BL instruction

- added tests for label and imm branch

* Implemented RET instruction

- added tests

* Implemented LDP instruction

- added tests for instruction

* Implemeneted ADRP instruction

- real binary testing is needed
- without tests

* enable arm64 in decompiler.py and add arm64 architecture

in Architecture::setArch() ARM64 needs to be set before ARM
because "arm" from ARM matches the "arm aarch64" from ARM64

* Arm64 ABI implementation

* Arm64 decoder ported from Arm

* Arm64 imm operand shifts should not update flags by default.

- Added the option to switch this behaviour
- add one ADD test with shift

* Operand register extension generator + 64bit variant extension tests

- Arm supports the extension of operand e.g. 'add x0, x1, w2, SXTW'
  will sign-extend the w2 register to 64 bit and after that add the values
- test for 64bit variant implemented
- need to check the optional imm(shift VM outputs weird values)

* Arm64 Zero/Sign extension 32bit variant tests

* Implemented SUB instruction

- added tests for instruction

* Implemented BR instruction

- added tests for instruction

* Arm64 syscall id register is X8

* Specified call and return instruction ID for implemented instruction

- BL Branch link is hinting the function call
- RET is hinting the function return

* Fixed compilation after merge

- new methods added isOperandRegister, getOperandAccess
- loadOpTernaryop1op2 probably changed to loadOpBinaryOrTernaryOp1Op2
- made sure all unit tests passed
- TODO: implement new conventions from master

* Generate pseudoasm instruction when translation routine is not found

- Function to generate condition code

* Check preconditions in implemented arm64 instructions

* Changed register generation to match other modules.

* LDR instruction all 3 formats + tests

- register
- imm
- literal (label)

* Binaries can now be decompiled

- jumpTargetDryRun updated

* Generate condition codes for conditional instructions.

* ARM64: strb, strh instructions + tests

* Arm64: conditional and unconditional branch instruction + tests

- removed the generation of conditional code in translate instruction function,
  this is not necessary because condition is generated in body of given
  instruction and arm64 support only specific instruction to be conditional.

* Arm64: Instruction ret can have optional register operand + test

* Arm64: BLR instruction + test

* Arm64: CBNZ, CBZ instruction + test

* Arm64: TBNZ, TBZ implementation + tests

* Arm64: LDR different size variants, sign/zero extend + tests

* Arm64: LDPSW instruction + tests

- minor warning fix in STR instruction

* Arm64: ADC instruction + tests

- including flag setting for ADC and ADD instructions
- ADDS tests

* Arm64: ADCS 32bit tests for flags

* Arm64: ADR, ADRP instruction + tests

* Arm64: AND, ANDS instruction + tests

* Arm64: ASR instruction + tests

- ASRV variant

* Arm64: LSL, LSR, ROR instructions + tests

- all major shifts implemented

* Arm64: SUB, SBC flags + tests

- changed asserts to exceptions

* Arm64: CMP, CMN instructions + tests

* Arm64: CSEL instruction + tests

* Arm64: CSET, CSETM instruction + tests

* Arm64: MUL instruction + tests

* Arm64: MADD instruction + tests

- 32bit tests for MUL

* Arm64: MSUB instruction + tests

* Arm64: MNEG instruction + tests

* Arm64: NEG, NEGS instruction + tests

* Arm64: NGC, NGCS initial implementation + tests

- Check the carry flags + add tests

* Arm64: SDIV, UDIV instruction + tests

* Arm64: Fix correct semantics for SBC and NEG instructions

* Arm64: SMADDL, UMADDL instruction + tests

* Arm64: UMSUBL, SMSUBL instruction + tests

* Arm64: SMNEG, UMNEG instruction + tests

* Arm64: UMULL, SMULL, UMULH, SMULH instruction + tests

* Arm64: Conditional select operation instruction + tests

* Arm64: CINC, CINV, CNEG tests

* Arm64: EON, EOR instruction + tests

* Arm64: ORN, ORR instruction + tests

* Arm64: TST instruction + tests

- fixed the AND instruction to set carry and overflow flags to zero

* Arm64: EXTR instruction + tests

* Arm64: Extend instructions + tests

* Arm64: CCMN, CCMP instruction + tests

* Arm64: NOP instruction + tests

* Arm64: REV, RBIT, CLZ instructions + tests

* Arm64: BIC instruction + tests

* Arm64: Unpriviledged loads/stores instructions + tests

* Arm64: Load/Store exclusive instructions + tests

* ARM64: LDAXR instruction variants + tests

* Arm64: LDAR instruction variants + tests

* Arm64, llvmir-emul: don't lower bitreverse intrinsic

- updated tests to check if the correct intrinsic functions was
  called

* Arm64: FP environment + basic unary and binary operations + tests

* Arm64: FMIN, FMINNM, FMAX, FMAXNM instruction + tests

* Arm64: FCMP, FCCMP, FCVT, {U, S}CVTF instructions + tests

* Arm64: FCVTZS, FCVTZU instructions + tests

- let's start testing

* Arm64, bin2llvmir: Decoder should not analyse stack.

* Arm64: MOVK instruction + tests

* Arm64: MOVN instructions + tests

* Merge master with arm-prep

* Architecture: Change arm architectures to account for arm64

-> isArmOrThumb renamed to isArm32OrThumb
-> added isArm32 method
-> thumb is now set with a flag _thumbFlag

* Architecture: Removed the wrong architecture types

Now the enum eArch represents only general architecture and all
subtypes of architecture are checked to getBitSize() or _thumbFlag.

The function isArm() return true for every type of subarchitecture
e.g. {arm32, arm64 or thumb}

* Arm64: XZR loads zero and discards result when written

- Added some instruction IDs to branch types

* Arm64: STR and LDR instructions now determine correct register size

- For example 'str w0, [sp]' should store only 4bytes to stack pointer

* Arm64: Syscall optimalization and detection

Replace svc #0 with corresponding syscall decoded from previous assignments.

* Arm64: MOVI instructions + tests, Vector and half register

Generate Vector registers so in case the pseudo instructions with them
as operands is generated we don't crash. For the similar purpose I
changed the f16 in ARM64_REG_H* to i16 since half type in not
supported and we wan't to be able to at least generate pseudo instructions.

* Arm64: STR and LDR tests

Those tests target loading and storing floating point values.

* Arm64: Removed zero division semantics from llvmir

- Zero division is NOW undefined behaviour
- This caused problems in modulo idiom detection
- Also removed coresponding tests

* Arm64: FMOV instruction with immediate values

- Correctly handle imm values as operands of this instruction

* Revert "Arm64, bin2llvmir: Decoder should not analyse stack."

This reverts commit 7b88475280.
This change caused other tests to fail.

* Arm64: Simplified and documented some code

- Removed unused code from decoder/arm64.cpp
- Fixed insnWrittesPcArm64 to work better
- Fixed Cond branch tests

* Arm64: Fixed documentation build
2019-03-28 11:44:26 +01:00
Peter Matula
296ebfe5fd Merge branch 'master' into faster-build
# Conflicts:
#	deps/capstone/CMakeLists.txt
#	deps/elfio/CMakeLists.txt
#	deps/llvm/CMakeLists.txt
2018-01-22 16:32:05 +01:00
Peter Matula
a991b17a7f deps: propagate C/C++ compilers to all deps. 2018-01-22 16:20:10 +01:00
Peter Matula
361e42b5b0 deps/capstone: disable AT&T asm syntax. 2018-01-19 13:13:04 +01:00
Peter Matula
b659bc6c62 deps/capstone: build only what is actually needed at the moment. 2018-01-19 13:11:12 +01:00
Petr Zemek
405e21d96f Add DOWNLOAD_NAME for each external project.
Reasons:
- Downloaded archives will have more meaningful names (e.g. capstone.zip
  instead of 27c713fe4f6eaf9721785932d850b6291a6073fe.zip).
- This prevents collisions between projects (e.g. both yaracpp and yaramod are
  in version 1.0.1 at the moment, so they were both downloaded as v1.0.1.zip).
2018-01-19 11:57:50 +01:00
Petr Zemek
39b74bf554 Replace uses of CMAKE_FIND_LIBRARY_PREFIXES with CMAKE_STATIC_LIBRARY_PREFIX.
From https://cmake.org/cmake/help/latest/variable/CMAKE_FIND_LIBRARY_PREFIXES.html:

    This specifies what prefixes to add to library names when the
    find_library() command looks for libraries.

What we want is CMAKE_STATIC_LIBRARY_PREFIX:

    The prefix to use for the name of a static library, lib on UNIX.

Also, we are already using CMAKE_STATIC_LIBRARY_SUFFIX, so this change also
makes the used variables consistent.
2018-01-12 12:25:13 +01:00
Petr Zemek
13365143f9 Remove useless cmake_minimum_required() command from nested CMakeLists.txt files.
We just need this command in the top-level CMakeLists.txt.
2018-01-12 09:20:33 +01:00
Petr Zemek
9ef11c96d2 Unify formatting in CMakeLists.txt files.
* Remove redundant first empty line from some files.
* Convert spaces to tabs.
* Remove redundant comments.
2018-01-12 09:00:52 +01:00
Petr Zemek
1e8294c264 Download external projects via URL instead of GIT_REPOSITORY.
The use of URL is safer than GIT_REPOSITORY as CMake does not update the cloned
repository when GIT_TAG changes. On the other hand, it downloads new sources
when URL changes.
2018-01-12 08:33:07 +01:00
Peter Matula
30a9f172a1 deps: remove target guards from CMake files. 2018-01-10 16:33:15 +01:00
Peter Matula
7cad5076c0 all in one repo, no submodules, deps via cmake external project 2018-01-05 11:26:55 +01:00