Eric Hennenfent 1208759e55 WASM Support MVP (#1495)
The WASM branch is ready to be included in the rest of Manticore. It doesn't include several advanced features, including a symbolic memory model, symbolic floating point support, or a standard library implementation (WASI/emscripten), but it does have full symbolic semantics for all of the integer instructions. 

### Commit log:
* Empty WASM Module

Will run on a .wasm file and do nothing forever

* Read and parse sections

/shrug

* Can partially allocate modules

Just pushing for the weekend

* Blacken

* Can allocate with exports now

* Static initialization

Doesn't evaluate any of the starting expressions, but that's okay

* Make type hints a little more pycharm-friendly

* First part of invocation

* Blacken...

* Barebones execution poc

Can execute the following snippet:
```
main(){
return 42;
}
```

* blacken

* Add some loads and stores

* Add global operations

* Move float methods further down

* Support internal function calls

```
18:20:42 (mc) ehennenfent@nessie:~/wasm ➜ cat loop_check.c
#include <stdio.h>
#include <stdbool.h>

bool check(int arg){
  if (((arg << 2) ^ 16) == 36) {
    return true;
  }
  else{
    return false;
  }
}

int main(){
    printf("Hello world!\n");

    int g = getchar();

    printf("Got: %d\n", g);

    int y = 0;
    for (int i = 0; i < 20; i++){
      y++;

      if (y > 18){
        return check(g) ? 0 : -1;
      }
    }

    return -1;
}
18:20:54 (mc) ehennenfent@nessie:~/wasm ➜ manticore loop_check.wasm
65: i32.const (16)
11: end
65: i32.const (32)
11: end
Initialization Complete
65: i32.const (0)
65: i32.const (0)
40: i32.load (Offset 4)
65: i32.const (16)
107: i32.sub
34: tee_local (Local 1)
54: i32.store (Offset 4)
65: i32.const (32)
16: call (Func Idx 2)
Called stub function with args: (32,)
HostFunc returned [13]
26: drop
32: get_local (Local 1)
16: call (Func Idx 0)
Called stub function with args: ()
HostFunc returned [13]
34: tee_local (Local 0)
54: i32.store (Offset 0)
65: i32.const (16)
32: get_local (Local 1)
16: call (Func Idx 1)
Called stub function with args: (16, 16)
HostFunc returned [13]
26: drop
65: i32.const (0)
32: get_local (Local 1)
65: i32.const (16)
106: i32.add
54: i32.store (Offset 4)
65: i32.const (-1)
65: i32.const (0)
32: get_local (Local 0)
65: i32.const (1073741823)
113: i32.and
65: i32.const (13)
71: i32.ne
27: select
11: end
[0]
2019-07-25 18:20:58,138: [13842] m.c.manticore:INFO: Generated testcase No. 0 - test
2019-07-25 18:20:58,143: [13842] m.c.manticore:INFO: Results in /home/ehennenfent/wasm/mcore_1evptpip
```

* Improve pickle-friendliness

* Convert wasm module instructions into internal type s

* Fix class naming conflict

Thanks Pickle

* Support bitvecs as an integer type

* Refactor execution to handle single instructions at a time

* Add Atomic Stacks

It's a very inefficient implementation, but we'll make it better

* Handle Concretize Exceptions

Needs a lot of fine tuning but it's a start

* Propagate symbols all the way to return

* Add test generation infrastructure

Still very basic but it's a start

* Attempt to make Travis happy

* Fix missing generation step

* Ignore stack exhaustsion and invalid modules

* Add basic shift instructions

* Add lots of arithmetic instructions

* Fix global initialization

* Add optimization for fixed size left shift

* Make Travis tests less verbose

* Blacken

* Properly handle modulo in shifting

* Add several control flow instructions

* Blacken

* Partially Fix return instruction

* Fix missing return value from look_forward

* Update look_forward to handle nested blocks

* Fix type sigs and executor copy/paste errors, deduplicate test names

* Fix an instruction exit issue

* Fix local ordering

* Blacken

* test_address now passes for wasm
Fixed issue with json2mc.py issue with testname uniqueness
Started some work on floating point

* i hate git

* i hate git

* Adding linenumber to testcase function name
minor float progress

* f32_cmp passed for wasm

* f64_cmp passes for wasm

* f32_bitwise passes except for the problem with assertEqual not working with nan

* f64_bitwise passes except for the problem with assertEqual not working with nan

* Fixed the tests to handle [nan] comparisons

* better handling of nans and infinites

* added integer backing value for f64

* Blacken

* Blacken

* f64.nearest

* Fix argument ordering and implement N-bit memory stores

* Use subtests

* Switch print to logger

* Modify control flow instructions to match spec

"Continuation" != the thing that goes inside the Label

* Teporarily disable call instructions to prevent infinite loop

Still need to figure out control flow for nested function calls

* Blacken

* Fixed infinite loop in _call via block depth tracking

The behavior of `end` needs to depend on how many labels and activations are on the stack. Every time we push or pop a label, we increment or decrement the counter for the current call frame.

* Blacken

* Initial `br` implementation

Probably still bugged

* Support if blocks without an else instruction

* Add call_indirect

* Add grow_memory

* i32_wrap_i64

* Add several arithmetic operators

* Copy i32.ctz impl from x86 TZCNT

* Fix off-by-one in br

I think this is the correct fix (it makes all the _br tests pass) but I'm leaving the TODO there just in case.

* Blacken...

* Fix lingering issue with if block splitting

* Fix bad args to sextend in i64_extend_s_i32

* Fix indirect call null check and Float binop return types

* Improve br_table indexing

See comment

* Remove vestigial argument popping in invoke

* Fix looking forward in _return and _block

* Blacken

* Generate trap and action tests

* Fix extraneous call to exec_for_test

* Fix missing traps in load instructions

* Fix c1=0 case in ctz/clz

Hopefully this also fixes the fact that the original CTZ implementations seems to have been intended for little-endian architectures

* Handle traps in-stride during test execution

* Treat zero division as trap

* Use ctypes to convert integers

* Blacken

* Apparently I was wrong about the CTZ endianness thing

* Better range checks for offsets

This really hammers home why checking whether ea + size is _greater than_ len(mem.data) is strange if mem.data is 0-indexed, but idk, that's what the spec says.

* Make stub functions return the correct number of results

* ctypes can't handle floats

* Add option to run the start function

* Correctly handle functions with unacceptable names

* Blacken

* Treat NotImplemented as a Skip

Also counts individual subtests directly instead of screwing around with trying to grep for them in the log.

* Fix unhandled negative indices

* Fixed some integer instruction bugs

i64/32:
div_s/u
shr_s/u
shl
clz

* Fix global and memory imports

* CC

* Make default loglevel debug

* Rework conversion from int

* Make float_store work

* _actually_ fix float_store

* Restore old float creation method

* Blacken

* Add range checks to float load/store

* Force cast to unsigned in _u instructions

* Fix yet another I32 -> I64 copy paste error

How many times do we have to teach you this lesson, ~old~ young man?

* Skip problematic float tests for now

* Handle floating point inaccuracies when dividing large ints

* Skip conversion and endianness tests

* Rotl, rotl, popcnt

* Blacken

* Add some conversion stubs

* Maybe fix conversions

* Improve reinterpret and float demotion

* Re-enable endianness tests

* Include WASM logging in verbosity controls

* Propagate constraints to executor

* Blacken

* Add prototype symbolic tests

* Add api for invocation to Manticore object

* Temporarily disable native/evm tests

There's no reason to tie up Travis for 40 minutes every time I push a commit.

* Fix wasmworld import

* Concretize br_if

* Pass arg generator to invoke

* Download updated WABT

It's fixed now!

* Generate Symbolic tests (partial)

* Follow m.run format

* Blacken

* Reinitialize Manticore every 50 tests

This should cut down on the performance issues we've been seeing

* Revert "Temporarily disable native/evm tests"

This reverts commit 7abd6c1333.

* I32 and I64 pass Symbolic

* Create new MC on reinit

* Blacken

* Fix return handling in test generation

* Concretize operand in `if_`

* Concretize memory addresses before access

In the long run, we should have a symbolic memory model

* Better memory access concretization (and br_table)

* Concretize float converions and indirect call operands

* Run start method

* Handle size errors in wrap+i64 and select

* Add timeout (and skip extra-problematic tests)

* Blacken

* Aggressively concretize floats

* Recast ints that should have been floats

* Better trap handling

* Unstash float concretization

* Fix log-lived 'trap' attr

* Blacken

* Concretize F32 when promoting to F64

_shouldn't_ strictly be necesssary, but the lazy conversion means that one of the tests fails. Until we get symbolic floating point, this will have to do.

* Docstrings for platforms/wasm.py

* Docstrings for ManticoreWASM

* Add docs for the module structure

* Add WASM to RTD

* missing docstring for Module itself

* Fix sphinx errors

* Bump RTD python version

I'm sorry and I swear I'll fix it later

* Reformat docstrings for sphinx

* Pacify RTD by adding newlines before params

* Missed a few

* Document the types

* Document executor

* A few more type docstrings

* Partial runtime_structure docs

* Finish documenting runtime structure

* Fix trailing whitespace

* Add collatz conjecture example

* Include undocumented members in RTD

* Add a bunch of events

* Appropriately connect event publishers

* Generate useful testcases

* Blacken

* Attempt at 3.6 backporting

* Optimize AtomicStack

* Sort solve_n output in test_solve_n

Very unclear why this is happening, but it's breaking a travis build
https://travis-ci.com/trailofbits/manticore/jobs/246451527

* Roll back to trusty?

Unsure if this is what's causing the EVM issues

* Create custom trap types

* Lazily evaluate overflows

May need to revert this if it slows down the Travis tests. It's 0.7% slower in the example I'm looking at, but that may not carry over.

* Blacken

* Address Dominik's feedback, remove stale TODOs

* Expand & Test Examples

* Fix relative path

* Fix relative path?

* capitalize Path

* Fix relative path for real

* Blacken...

* Test finalize

* Add supplemental tests for coverage

* Use dedicated helper for store access

* Handle quoted strings in imported module names

* Allow WASMWorld to hold multiple modules

Necessary for getting imports/exports to work, but still needs some changes before it passes the tests.

* Actually use export maps

* Fix most of the control flow issues with import tests

* Use available cores for test generation

* Escape module name

* Don't reinit modules that have already been initialized

* Use addresses for imports instead of copying

* Fix removed timeout argument

* Interleave registration and aliases

* Fix broken "None" check for address

* Blacken

* Make manual exports work with address syntax

* Import all registered modules

Only necessary for elem, but it's tricky to make changes just for one test, so *shrug emoji*

* Remove stringified annotations

Turns out those only work in the same file...

* Fix easy type issues in all but runtime structure

* Fix easy type issues in runtime_structure

* Fix* everything but imports

* Blacken

* Blacken 2

* Typo

* Merge module structure and runtime structure

* Fix instance vs type confusion

* Fix mypy and concrete tests

Still an issue with the symbolic tests

* Still generate symbolic tests though

I keep forgetting not to commit that file...

* Attempt to fix env for symbolic tests

* Blacken/mypy

* Fix missing supplemental environment variables arg

* Fix docstrings for new import style

* CC

* Bump timeout duration to 70 minutes

I don't like it, but adding more jobs to Travis isn't going to make our builds any faster

* Fix pycharm type, import, and docstring errors

* Explain type: ignore

* Missed two type:ignore's

* Fix CLI so it doesn't throw up on arbitrary modules
2019-11-08 07:58:24 -08:00
2019-11-08 07:58:24 -08:00
2019-11-08 07:58:24 -08:00
2019-11-08 07:58:24 -08:00
2019-11-08 07:58:24 -08:00
2019-11-08 07:58:24 -08:00
2019-05-23 15:19:24 -05:00
2018-08-15 16:49:40 -04:00
2018-07-25 10:10:44 -04:00
2019-11-08 07:58:24 -08:00
2019-08-06 17:59:01 -05:00
2019-10-31 14:55:05 -07:00
2018-09-01 12:28:22 -03:00
2019-11-08 07:58:24 -08:00
2019-08-01 16:10:34 -05:00
2019-11-08 07:58:24 -08:00
2019-11-08 07:58:24 -08:00
2019-05-23 15:19:24 -05:00

Manticore


Build Status PyPI version Slack Status Documentation Status Maintainability Test Coverage

Manticore is a symbolic execution tool for analysis of smart contracts and binaries.

Note: Beginning with version 0.2.0, Python 3.6+ is required.

Features

  • Input Generation: Manticore automatically generates inputs that trigger unique code paths
  • Error Discovery: Manticore discovers bugs and produces inputs required to trigger them
  • Execution Tracing: Manticore records an instruction-level trace of execution for each generated input
  • Programmatic Interface: Manticore exposes programmatic access to its analysis engine via a Python API

Manticore can analyze the following types of programs:

  • Ethereum smart contracts (EVM bytecode)
  • Linux ELF binaries (x86, x86_64 and ARMv7)

Usage

CLI

Manticore has a command line interface which can be used to easily symbolically execute a supported program or smart contract. Analysis results will be placed into a new directory beginning with mcore_.

Use the CLI to explore possible states in Ethereum smart contracts. Manticore includes detectors that flag potentially vulnerable code in discovered states; output from them will be written to stdout and the results directory. Solidity smart contracts must have a .sol extension for analysis by Manticore. See a demo.

$ manticore ./path/to/contract.sol  # runs, and creates a mcore_* directory with analysis results

The command line can also be used to simply explore a Linux binary:

$ manticore ./path/to/binary        # runs, and creates a mcore_* directory with analysis results
$ manticore ./path/to/binary ab cd  # use concrete strings "ab", "cd" as program arguments
$ manticore ./path/to/binary ++ ++  # use two symbolic strings of length two as program arguments

API

Manticore has a Python programming interface which can be used to implement custom analyses.

For Ethereum smart contracts, it can be used for detailed verification of arbitrary contract properties. Set starting conditions, execute symbolic transactions, then review discovered states to ensure invariants for your contract hold.

from manticore.ethereum import ManticoreEVM
contract_src="""
contract Adder {
    function incremented(uint value) public returns (uint){
        if (value == 1)
            revert();
        return value + 1;
    }
}
"""
m = ManticoreEVM()

user_account = m.create_account(balance=1000)
contract_account = m.solidity_create_contract(contract_src,
                                              owner=user_account,
                                              balance=0)
value = m.make_symbolic_value()

contract_account.incremented(value)

for state in m.ready_states:
    print("can value be 1? {}".format(state.can_be_true(value == 1)))
    print("can value be 200? {}".format(state.can_be_true(value == 200)))

It is also possible to use the API to create custom analysis tools for Linux binaries.

# example Manticore script
from manticore.native import Manticore

hook_pc = 0x400ca0

m = Manticore.linux('./path/to/binary')

@m.hook(hook_pc)
def hook(state):
  cpu = state.cpu
  print('eax', cpu.EAX)
  print(cpu.read_int(cpu.ESP))

  m.kill()  # tell Manticore to stop

m.run()

Requirements

  • Manticore is supported on Linux and requires Python 3.6+.
  • Ubuntu 18.04 is strongly recommended.
  • Ethereum smart contract analysis requires the solc program in your $PATH.
  • Increased stack size is recommended; this can be done by ulimit -s 100000 or by passing --ulimit stack=100000000:100000000 to docker run if docker is used.

Quickstart

Install and try Manticore in a few shell commands:

# Install system dependencies
sudo apt-get update && sudo apt-get install python3 python3-pip -y

# Install Manticore and its dependencies
sudo pip3 install manticore[native]

# Download the examples
git clone https://github.com/trailofbits/manticore.git && cd manticore/examples/linux

# Build the examples
make

# Use the Manticore CLI
manticore basic
cat mcore_*/*0.stdin | ./basic
cat mcore_*/*1.stdin | ./basic

# Use the Manticore API
cd ../script
python3 count_instructions.py ../linux/helloworld

You can also use Docker to quickly install and try Manticore:

# Run container with a shared examples/ directory
# Note that `--rm` will make the container be deleted if you exit it
# (if you want to persist data from the container, use docker volumes)
# (we need to increase maximum stack size, so we use ulimit for that)
$ docker run --rm -it --ulimit stack=100000000:100000000 trailofbits/manticore bash

# Change to examples directory
manticore@8d456f662d0f:~$ cd manticore/examples/linux/

# Build the examples
manticore@8d456f662d0f:~/manticore/examples/linux$ make

# Use the Manticore CLI
manticore@8d456f662d0f:~/manticore/examples/linux$ manticore basic


manticore@8d456f662d0f:~/manticore/examples/linux$ cat mcore_*/*0.stdin | ./basic
manticore@8d456f662d0f:~/manticore/examples/linux$ cat mcore_*/*1.stdin | ./basic

# Use the Manticore API
manticore@8d456f662d0f:~/manticore/examples/linux$ cd ../script
manticore@8d456f662d0f:~/manticore/examples/script$ python3 count_instructions.py ../linux/helloworld

Installation

NOTE: For native binary analysis, Manticore requires additional dependencies that are not installed by default. To install these also, substitute manticore[native] for manticore in any pip command.

Option 1: Perform a user install (requires ~/.local/bin in your PATH).

echo "PATH=\$PATH:~/.local/bin" >> ~/.profile
source ~/.profile
pip3 install --user manticore

Option 2: Use a virtual environment (requires virtualenvwrapper or similar).

sudo pip3 install virtualenvwrapper
echo "source /usr/local/bin/virtualenvwrapper.sh" >> ~/.profile
source ~/.profile
mkvirtualenv manticore
sudo ./manticore/bin/pip3 install manticore

Option 3: Perform a system install.

sudo pip3 install manticore

Option 4: Install via Docker.

docker pull trailofbits/manticore

Once installed, the manticore CLI tool and Python API will be available.

For installing a development version of Manticore, see our wiki.

If you use Mac OS X you may need to install dependencies manually:

brew install capstone
export MACOS_UNIVERSAL=no && pip install capstone

brew install unicorn
UNICORN_QEMU_FLAGS="--python=`whereis python`" pip install unicorn

Solidity Versions

Note that we're still in the process of implementing full support for the EVM Constantinople instruction semantics, so certain opcodes may not be supported. You may want to consider using a version of solc that's less likely to generate these opcodes (eg pre-0.5.0).

Getting Help

Feel free to stop by our #manticore slack channel in Empire Hacking for help using or extending Manticore.

Documentation is available in several places:

  • The wiki contains some basic information about getting started with Manticore and contributing

  • The examples directory has some very minimal examples that showcase API features

  • The API reference has more thorough and in-depth documentation on our API

  • The manticore-examples repository has some more involved examples, for instance solving real CTF problems

License

Manticore is licensed and distributed under the AGPLv3 license. Contact us if you're looking for an exception to the terms.

Languages
Python 99.4%
Solidity 0.3%
C 0.1%