Commit Graph
92 Commits
Author SHA1 Message Date
Eric Hennenfent 904f336670 Switch to stable Black (#2536)
* Switch to stable Black version

`black` has recently stabilized on version 22, which means that we can loosen the version restrictions a bit to get upstream bugfixes, without needing to worry about our formatting breaking unexpectedly. In 2023, they may issue a new release with breaking changes, but we won't be automatically upgraded.

There have been a large number of formatting changes since 20.8.b1, which is the last version we standardized on. This will likely require rebase of several of the existing PRs, but with relatively few of them currently active, I think it's as good a time as any.

More info about black's further development in their [stability policy](https://black.readthedocs.io/en/latest/the_black_code_style/index.html#stability-policy)

* Add new formatting changes

Mostly docstrings and exponentiation
2022-03-01 09:36:01 -08:00
Eric Hennenfent 36b3024ff0 Switch to Python 3.7 (#2472)
* Add version info to installations

* Let's try a newer version?

* Use fixed yices release

* Really? No relative path support?

* Can't handle newer C compilers

Ugh we should really do something about that

* Let's try 3.8

* Back to 3.7

3.6 is EOL soon, so we can justify jumping up to a more recent version, but 3.8 might still be too new for some people.

* Find & Replace 3.6 -> 3.7

* Bump setup-python version

* Fix additional 3.6 usage
2021-08-04 15:36:25 -07:00
Eric Kilmer 5205fa3e02 Fix incorrect comparison for symbolic file wildcards (#2454)
* Add tests for symbolic file wildcard handling

* Add example script and assertions for fileio.c in `examples` directory
2021-06-02 09:30:22 -04:00
feliam 089e089e55 Attempt to allow symbolic balances from the start (#1818) 2021-01-05 09:02:54 -08:00
feliam 6325e481f9 Bump black and mypy (#1824) 2020-12-07 20:13:54 -03:00
Eric HennenfentandEric Kilmer f46f78b69b State Introspection API (#1775)
* Create state transition events

Should make it possible to track movements between state lists

* Plug new events into context

This will break the state merging plugin (but I'll fix it eventually)

* move most enums to their own module

* Blacken

* Add DaemonThread from TUI branch

* Add interface for registering daemon threads

* Timestamp StateDescriptor upon updates

* Capture return value

* Blacken

* Add solver wrapper to StateBase

* Add `solve` events to all instances of SelectedSolver.instance()

* Remove executor constraints from WASM

* Add solve events to memory.py

* Add intermittent execution event

* Be more generous with states whose initialization we missed

* Add Native callback for updating state descriptor

* Fix state killing

* Blacken

* codecov: Remove outdated 'yml' entry in CI

From these commits
https://github.com/codecov/codecov-action/commit/ebea5cacdf7d0b843f66bcb1ae27d8f27b758e81
https://github.com/codecov/codecov-action/commit/49c86d6a5fd072b05c31a42baa67b8fb2e87c8f7

* Add solve event to evm

Make warning messages better

Debug GH actions

Revert "Debug GH actions"

This reverts commit f575eea3c3a09dbf2cd2b8e81940fc4297fdf039.

Fix some pycharm-detected problems

Make symbolic function error message more verbose

Add solve to published events

Loud errors in callbacks by default

Trying to find out what's killing truffle

Revert "Trying to find out what's killing truffle"

This reverts commit 8bd02245ccc2ce54c9161072eba231880c084874.

Revert "Make symbolic function error message more verbose"

This reverts commit bd3e90cdb0dfabf0ac874badad723be50998f63b.

Debugging Truffle

Restore introspector

Add try_except on every callback

Unconditionally print error message

Add traceback

Update event.py

Debug subscriptions

Debug arguments to callbacks

Different debug msg

1ast arg

Print statement debugging...

Pass in `None` as state

Revert "Add try_except on every callback"

This reverts commit 1c689dd43c619ba322cda5e7283686308c8db4e0.

* Drop solve events outside of a state context

Forgot did_solve

Remove traceback

* Fix must/cannot_be_null usage

* Fix missing solve event

* Partially restore old did_fork_state ABI

* Called internally

* Clone iterators instead of creating a list

* Use isgenerator instead of checking if iterable

* Fix snapshot restoration

* Slightly improve Unicorn test API usage

* Temporarily disable property verifier tests

* improper skip arg

* Add simple tests for introspection API

* Add test for custom introspector, improve base introspection test

* Add intermittent update timestamp

* Only allow daemon registration and introspection registration at initialization

* Add docs to manticore.py

* Add docs for plugin, add update_state_descriptor to EVM

* Fix renamed will_start_run --> will_run

* Docstrings for DaemonThread and EventSolver

* Docs for enums

* Improve pretty printer, add some mypy fixes

* Don't run daemon threads if run is called multiple times

* If at first you don't succeed, destroy all the evidence you tried.

* Test the pretty printer

* Add StateDescriptor to RTD

* Add newlines for RTD parsing

* Make info logs debug logs

* Apply suggestions from code review

Newlines for doc comments

Co-authored-by: Eric Kilmer <[email protected]>

* Add some type hints to manticore.py

* Add some type hints to plugin.py

* Fix type hint for get_state

* Add termination message from TUI PR

* Add example script

* Add docstrings to the example script

* Pass introspection plugin type as an argument

* Unskip property verifier tests

* Add mypy-requests type hints

* Remove itertools.tee

The problem with usign tee is that only the first callback to use the iterator can write to it. In `ready_states`, the `save_state` after the `yield` statement is ignored for all others.

* Make generator cloning a little bit more robust

Now Manticore will give up and return the original argument instead of blowing up if it can't clone the generator

* Clean up invalidated unit tests

We now fire `introspect` for the first time before we have any states

* Debug missing Truffle & Examples coverage

* Merge coverage from XML file

* Switch coverage to JSON, ignore debug logging and NotImplemented code

* Fix copy commands

* Move .coverage files directly

* Set examples to append coverage

* FLAG_NAME doesn't work the way we'd like

* Use plugin dict to store introspector

* Appease mypy

* Fix missing property on unique name

* Grab EVM PC

* Blacken

* Run black on all files if the git diff command fails

Co-authored-by: Eric Kilmer <[email protected]>
2020-09-09 08:32:46 -07:00
099dc47b69 Fix constant folding & constraint set slicing (#1706)
* Add ONE failing test

* Fix test

* Try fix get-related

* Move regression to other

* blkn

* blkn

* Move get related

* CC

* fix concolic

* lint

* DivByZero default to zero

* blkn

* Update manticore/core/smtlib/solver.py

Co-authored-by: Eric Kilmer <[email protected]>

* Update manticore/core/smtlib/constraints.py

Co-authored-by: Eric Kilmer <[email protected]>

* Update manticore/core/smtlib/constraints.py

Co-authored-by: Eric Kilmer <[email protected]>

* remove odd string

* lint

* mypy lint

* Update manticore/core/smtlib/visitors.py

Co-authored-by: Eric Kilmer <[email protected]>

* lint

* Add Docs

* Replace modulo with masks

* Blacken

* blkn

* fix mypy

* fix mypy

* Add tests for signed LT behavior

* New test

* Fix constant folding

* lint

* lint

* lint

* Unittesting power

* Permisive read_buffer

* Preserve precision in constant folding

* Strip left-in print debugging

* fix wasm

* Fix

* REmove get_related from the default path and fix arm test

* blkn

* fix related to tests

* blkn

* Fix bug in test and disable debug messages in Solver

* smtlib config to disable multiple check-sat in newer z3

* Disable log test and fix merging poc vs variable migration

* blkn

* Avoid exception in some callbacks

* can_raise at did_will

* slightly better z3 onfiguration

* lint

Co-authored-by: Eric Kilmer <[email protected]>
Co-authored-by: Eric Hennenfent <[email protected]>
2020-06-16 21:59:32 -03:00
Eric Hennenfent bf79b805ab Update README, Add Issue Templates (#1685)
* Add canned issue templates

* Fix URL params

* Update the README

* Add CLA

* Update Youtube URL

* Remove sudo from example readme

* Reword dependencies in main readme

* Replace mkvirtualenv with default version
2020-05-15 11:28:01 -07:00
Brad Larsen cc77acee41 Overhaul Linux file emulation (#1673)
Overhaul Linux file emulation.

Previously, open file descriptors could be mapped to any of the `File`, `Directory`, `SocketDesc`, `Socket`, and `ProcSelfMaps` classes, which didn't share a common supertype, and which didn't have consistent interfaces.

Now, each of these classes implements the `FdLike` abstract base class, which specifies a number of methods.

This overhaul additionally fixes several issues with the Linux support code:

- A program that tried to ioctl on a socket would crash Manticore
- In several places, the error code returned by a failing syscall was incorrect
- Many file descriptor operations could crash Manticore when used on an already-closed descriptor
- Manticore state serialization / deserialization was incorrect for file descriptor types other than File or Socket
- In certain cases, `sys_open` / `sys_openat` could crash Manticore when trying to log a debug message
- In certain cases, `sys_readv` could crash Manticore, instead of correctly returning a negative int value
- `sys_sendfile` was broken due to a variable name typo
- In several places in Manticore’s Linux support code, incorrect type hints were present
- In some places, an exception would attempt to be raised, but that itself would crash due to a missing import.

This commit adds two more example / integration test programs.

Additionally, this adds a helper for working with Python abstract base classes.  This helper lets you declare on a class, via a decorator, that the class is expected to have no unimplemented abstract methods.  If this is not true at runtime, a `TypeError` is thrown.  Without this decorator, you will only discover that a class has unimplemented abstract methods when you try to _instantiate_ that class. This may be much, much later than at class creation time, or perhaps not at all, depending on your code coverage.

Thanks to Eric Kilmer <[email protected]> for review!
2020-05-08 13:57:41 -04:00
Brad Larsen c8a6fc8ae7 Rework WASM Imports and Fix Typos (#1666)
* Cleanup: add missing space in a description string

* Cleanup: ignore a derived example file

* Cleanup: sort WASM type import lists
2020-04-16 10:14:58 -07:00
Brad Larsen d059a0370c Fix 2 problems in Linux sys_open support & add type hints (#1657)
* Add type hints to some Linux syscalls

* Fix a type error & add a type hint in Linux `sys_open`

* Add an example program that can exercise symbolic file support

* Add a test case based on the `fileio` example

Note, The `fileio_linux_amd64` test binary was manually copied from the
examples directory and added to Git.  Maybe one day we will come up with
a more principled mechanism for this.

Also note, this test case currently fails, even with the `sys_open` type
fixes.  Multiple errors are revealed by this test case.

* Add a type hint

* Add type hints to manticore workspace code

* Make an optional parameter non-optional

This function never seems to be used without specifying the formerly
optional value.

* Fix choice of abbreviation in comments

* Allow keys with `/` in them in `FilesystemStore`.

This is an expedient fix -- a more proper fix would nail down the API of
which characters are allowed in keys, maximum length etc, at the level
of the `Store` class.

* Have `fileio` exit with 0 on incorrect usage

Our GitHub CI has one step where it builds each of the example programs,
and then attempts to run them with no arguments and no standard input,
checking that they exit with code 0.  Rather than rework this approach
to support more general invocation patterns for example programs, it's
easier to change the `fileio` example I added.

* Blacken manticore/core/workspace.py
2020-04-08 13:58:50 -07:00
Brad Larsen 2527fa5c40 Add a native example program & test case for #1604 (#1606)
* Organize examples in Makefile

* .gitignore binaries created in examples/linux

* Add a new Linux example using fclose

* Add a test case for the fclose example binary

See also #1602 and #1604.

* Add a comment for the change from #1604

* Delete an obsolete comment

* Add type hints to native integration tests

* Do not squelch command output from test

* Use `--worksapce` for `test_logger_verbosity`

This should prevent extra mcore_* directories from hanging around after testing.

* Reformat test_integration_native.py via `black`
2020-02-04 09:22:53 -08:00
Eric HennenfentandHaxMeadroom 83325f95b9 WASM QoL Improvements (#1578)
* Add some loads and stores

* Add global operations

* Move float methods further down

* Support internal function calls

```
18:20:42 (mc) ehennenfent@nessie:~/wasm ➜ cat loop_check.c
#include <stdio.h>
#include <stdbool.h>

bool check(int arg){
  if (((arg << 2) ^ 16) == 36) {
    return true;
  }
  else{
    return false;
  }
}

int main(){
    printf("Hello world!\n");

    int g = getchar();

    printf("Got: %d\n", g);

    int y = 0;
    for (int i = 0; i < 20; i++){
      y++;

      if (y > 18){
        return check(g) ? 0 : -1;
      }
    }

    return -1;
}
18:20:54 (mc) ehennenfent@nessie:~/wasm ➜ manticore loop_check.wasm
65: i32.const (16)
11: end
65: i32.const (32)
11: end
Initialization Complete
65: i32.const (0)
65: i32.const (0)
40: i32.load (Offset 4)
65: i32.const (16)
107: i32.sub
34: tee_local (Local 1)
54: i32.store (Offset 4)
65: i32.const (32)
16: call (Func Idx 2)
Called stub function with args: (32,)
HostFunc returned [13]
26: drop
32: get_local (Local 1)
16: call (Func Idx 0)
Called stub function with args: ()
HostFunc returned [13]
34: tee_local (Local 0)
54: i32.store (Offset 0)
65: i32.const (16)
32: get_local (Local 1)
16: call (Func Idx 1)
Called stub function with args: (16, 16)
HostFunc returned [13]
26: drop
65: i32.const (0)
32: get_local (Local 1)
65: i32.const (16)
106: i32.add
54: i32.store (Offset 4)
65: i32.const (-1)
65: i32.const (0)
32: get_local (Local 0)
65: i32.const (1073741823)
113: i32.and
65: i32.const (13)
71: i32.ne
27: select
11: end
[0]
2019-07-25 18:20:58,138: [13842] m.c.manticore:INFO: Generated testcase No. 0 - test
2019-07-25 18:20:58,143: [13842] m.c.manticore:INFO: Results in /home/ehennenfent/wasm/mcore_1evptpip
```

* Improve pickle-friendliness

* Convert wasm module instructions into internal type s

* Fix class naming conflict

Thanks Pickle

* Support bitvecs as an integer type

* Refactor execution to handle single instructions at a time

* Add Atomic Stacks

It's a very inefficient implementation, but we'll make it better

* Handle Concretize Exceptions

Needs a lot of fine tuning but it's a start

* Propagate symbols all the way to return

* Add test generation infrastructure

Still very basic but it's a start

* Attempt to make Travis happy

* Fix missing generation step

* Ignore stack exhaustsion and invalid modules

* Add basic shift instructions

* Add lots of arithmetic instructions

* Fix global initialization

* Add optimization for fixed size left shift

* Make Travis tests less verbose

* Blacken

* Properly handle modulo in shifting

* Add several control flow instructions

* Blacken

* Partially Fix return instruction

* Fix missing return value from look_forward

* Update look_forward to handle nested blocks

* Fix type sigs and executor copy/paste errors, deduplicate test names

* Fix an instruction exit issue

* Fix local ordering

* Blacken

* test_address now passes for wasm
Fixed issue with json2mc.py issue with testname uniqueness
Started some work on floating point

* i hate git

* i hate git

* Adding linenumber to testcase function name
minor float progress

* f32_cmp passed for wasm

* f64_cmp passes for wasm

* f32_bitwise passes except for the problem with assertEqual not working with nan

* f64_bitwise passes except for the problem with assertEqual not working with nan

* Fixed the tests to handle [nan] comparisons

* better handling of nans and infinites

* added integer backing value for f64

* Blacken

* Blacken

* f64.nearest

* Fix argument ordering and implement N-bit memory stores

* Use subtests

* Switch print to logger

* Modify control flow instructions to match spec

"Continuation" != the thing that goes inside the Label

* Teporarily disable call instructions to prevent infinite loop

Still need to figure out control flow for nested function calls

* Blacken

* Fixed infinite loop in _call via block depth tracking

The behavior of `end` needs to depend on how many labels and activations are on the stack. Every time we push or pop a label, we increment or decrement the counter for the current call frame.

* Blacken

* Initial `br` implementation

Probably still bugged

* Support if blocks without an else instruction

* Add call_indirect

* Add grow_memory

* i32_wrap_i64

* Add several arithmetic operators

* Copy i32.ctz impl from x86 TZCNT

* Fix off-by-one in br

I think this is the correct fix (it makes all the _br tests pass) but I'm leaving the TODO there just in case.

* Blacken...

* Fix lingering issue with if block splitting

* Fix bad args to sextend in i64_extend_s_i32

* Fix indirect call null check and Float binop return types

* Improve br_table indexing

See comment

* Remove vestigial argument popping in invoke

* Fix looking forward in _return and _block

* Blacken

* Generate trap and action tests

* Fix extraneous call to exec_for_test

* Fix missing traps in load instructions

* Fix c1=0 case in ctz/clz

Hopefully this also fixes the fact that the original CTZ implementations seems to have been intended for little-endian architectures

* Handle traps in-stride during test execution

* Treat zero division as trap

* Use ctypes to convert integers

* Blacken

* Apparently I was wrong about the CTZ endianness thing

* Better range checks for offsets

This really hammers home why checking whether ea + size is _greater than_ len(mem.data) is strange if mem.data is 0-indexed, but idk, that's what the spec says.

* Make stub functions return the correct number of results

* ctypes can't handle floats

* Add option to run the start function

* Correctly handle functions with unacceptable names

* Blacken

* Treat NotImplemented as a Skip

Also counts individual subtests directly instead of screwing around with trying to grep for them in the log.

* Fix unhandled negative indices

* Fixed some integer instruction bugs

i64/32:
div_s/u
shr_s/u
shl
clz

* Fix global and memory imports

* CC

* Make default loglevel debug

* Rework conversion from int

* Make float_store work

* _actually_ fix float_store

* Restore old float creation method

* Blacken

* Add range checks to float load/store

* Force cast to unsigned in _u instructions

* Fix yet another I32 -> I64 copy paste error

How many times do we have to teach you this lesson, ~old~ young man?

* Skip problematic float tests for now

* Handle floating point inaccuracies when dividing large ints

* Skip conversion and endianness tests

* Rotl, rotl, popcnt

* Blacken

* Add some conversion stubs

* Maybe fix conversions

* Improve reinterpret and float demotion

* Re-enable endianness tests

* Include WASM logging in verbosity controls

* Propagate constraints to executor

* Blacken

* Add prototype symbolic tests

* Add api for invocation to Manticore object

* Temporarily disable native/evm tests

There's no reason to tie up Travis for 40 minutes every time I push a commit.

* Fix wasmworld import

* Concretize br_if

* Pass arg generator to invoke

* Download updated WABT

It's fixed now!

* Generate Symbolic tests (partial)

* Follow m.run format

* Blacken

* Reinitialize Manticore every 50 tests

This should cut down on the performance issues we've been seeing

* Revert "Temporarily disable native/evm tests"

This reverts commit 7abd6c1333.

* I32 and I64 pass Symbolic

* Create new MC on reinit

* Blacken

* Fix return handling in test generation

* Concretize operand in `if_`

* Concretize memory addresses before access

In the long run, we should have a symbolic memory model

* Better memory access concretization (and br_table)

* Concretize float converions and indirect call operands

* Run start method

* Handle size errors in wrap+i64 and select

* Add timeout (and skip extra-problematic tests)

* Blacken

* Aggressively concretize floats

* Recast ints that should have been floats

* Better trap handling

* Unstash float concretization

* Fix log-lived 'trap' attr

* Blacken

* Concretize F32 when promoting to F64

_shouldn't_ strictly be necesssary, but the lazy conversion means that one of the tests fails. Until we get symbolic floating point, this will have to do.

* Docstrings for platforms/wasm.py

* Docstrings for ManticoreWASM

* Add docs for the module structure

* Add WASM to RTD

* missing docstring for Module itself

* Fix sphinx errors

* Bump RTD python version

I'm sorry and I swear I'll fix it later

* Reformat docstrings for sphinx

* Pacify RTD by adding newlines before params

* Missed a few

* Document the types

* Document executor

* A few more type docstrings

* Partial runtime_structure docs

* Finish documenting runtime structure

* Fix trailing whitespace

* Add collatz conjecture example

* Include undocumented members in RTD

* Add a bunch of events

* Appropriately connect event publishers

* Generate useful testcases

* Blacken

* Attempt at 3.6 backporting

* Optimize AtomicStack

* Sort solve_n output in test_solve_n

Very unclear why this is happening, but it's breaking a travis build
https://travis-ci.com/trailofbits/manticore/jobs/246451527

* Roll back to trusty?

Unsure if this is what's causing the EVM issues

* Create custom trap types

* Lazily evaluate overflows

May need to revert this if it slows down the Travis tests. It's 0.7% slower in the example I'm looking at, but that may not carry over.

* Blacken

* Address Dominik's feedback, remove stale TODOs

* Expand & Test Examples

* Fix relative path

* Fix relative path?

* capitalize Path

* Fix relative path for real

* Blacken...

* Test finalize

* Add supplemental tests for coverage

* Use dedicated helper for store access

* Handle quoted strings in imported module names

* Allow WASMWorld to hold multiple modules

Necessary for getting imports/exports to work, but still needs some changes before it passes the tests.

* Actually use export maps

* Fix most of the control flow issues with import tests

* Use available cores for test generation

* Escape module name

* Don't reinit modules that have already been initialized

* Use addresses for imports instead of copying

* Fix removed timeout argument

* Interleave registration and aliases

* Fix broken "None" check for address

* Blacken

* Make manual exports work with address syntax

* Import all registered modules

Only necessary for elem, but it's tricky to make changes just for one test, so *shrug emoji*

* Remove stringified annotations

Turns out those only work in the same file...

* Fix easy type issues in all but runtime structure

* Fix easy type issues in runtime_structure

* Fix* everything but imports

* Blacken

* Blacken 2

* Typo

* Merge module structure and runtime structure

* Fix instance vs type confusion

* Fix mypy and concrete tests

Still an issue with the symbolic tests

* Still generate symbolic tests though

I keep forgetting not to commit that file...

* Attempt to fix env for symbolic tests

* Blacken/mypy

* Fix missing supplemental environment variables arg

* Fix docstrings for new import style

* CC

* Bump timeout duration to 70 minutes

I don't like it, but adding more jobs to Travis isn't going to make our builds any faster

* Fix pycharm type, import, and docstring errors

* Explain type: ignore

* Missed two type:ignore's

* Fix CLI so it doesn't throw up on arbitrary modules

* Add getters for locals and globals to state

* Make host functions take the state instead of the constriant set

* Support __getattr__ style function calling

* Implicitly run Manticore when invoked via __getattr__

* Automatically reinitialize before __getattr__ style function calls

* Fix docstring

* Fix mypy?

* Fix test getchar impl

* Add tests for implicit call and main function

* Fix wrong memory callback

* Okay fine I'll check the output

* Add missing jinja dep

Sphinx includes it, but we should do so as well

* Bump spec tests version

* Test multiple implicit calls

* Split symbolic/concrete WASM tests

* Add some missing docstrings

* Hopefully fix wasm test split?

* Properly split symbolic tests

* Fix fallthrough

Okay I swear it's fixed this time

* Add if_check example

* Blacken

* Add WASM Example to README

Co-authored-by: HaxMeadroom <[email protected]>
2020-01-03 11:55:22 -08:00
Eric Kilmer 1c67927f1d Fix deprecation warnings related to setting verbosity (#1574) 2019-12-16 08:23:56 -08:00
Paul Kehrer 6b77189d6e switch to github actions (take two) (#1568)
* switch to github actions

* badges, better lint dep setup

* remove meaningless comment
2019-12-04 11:30:54 -06:00
Eric Hennenfent 1208759e55 WASM Support MVP (#1495)
The WASM branch is ready to be included in the rest of Manticore. It doesn't include several advanced features, including a symbolic memory model, symbolic floating point support, or a standard library implementation (WASI/emscripten), but it does have full symbolic semantics for all of the integer instructions. 

### Commit log:
* Empty WASM Module

Will run on a .wasm file and do nothing forever

* Read and parse sections

/shrug

* Can partially allocate modules

Just pushing for the weekend

* Blacken

* Can allocate with exports now

* Static initialization

Doesn't evaluate any of the starting expressions, but that's okay

* Make type hints a little more pycharm-friendly

* First part of invocation

* Blacken...

* Barebones execution poc

Can execute the following snippet:
```
main(){
return 42;
}
```

* blacken

* Add some loads and stores

* Add global operations

* Move float methods further down

* Support internal function calls

```
18:20:42 (mc) ehennenfent@nessie:~/wasm ➜ cat loop_check.c
#include <stdio.h>
#include <stdbool.h>

bool check(int arg){
  if (((arg << 2) ^ 16) == 36) {
    return true;
  }
  else{
    return false;
  }
}

int main(){
    printf("Hello world!\n");

    int g = getchar();

    printf("Got: %d\n", g);

    int y = 0;
    for (int i = 0; i < 20; i++){
      y++;

      if (y > 18){
        return check(g) ? 0 : -1;
      }
    }

    return -1;
}
18:20:54 (mc) ehennenfent@nessie:~/wasm ➜ manticore loop_check.wasm
65: i32.const (16)
11: end
65: i32.const (32)
11: end
Initialization Complete
65: i32.const (0)
65: i32.const (0)
40: i32.load (Offset 4)
65: i32.const (16)
107: i32.sub
34: tee_local (Local 1)
54: i32.store (Offset 4)
65: i32.const (32)
16: call (Func Idx 2)
Called stub function with args: (32,)
HostFunc returned [13]
26: drop
32: get_local (Local 1)
16: call (Func Idx 0)
Called stub function with args: ()
HostFunc returned [13]
34: tee_local (Local 0)
54: i32.store (Offset 0)
65: i32.const (16)
32: get_local (Local 1)
16: call (Func Idx 1)
Called stub function with args: (16, 16)
HostFunc returned [13]
26: drop
65: i32.const (0)
32: get_local (Local 1)
65: i32.const (16)
106: i32.add
54: i32.store (Offset 4)
65: i32.const (-1)
65: i32.const (0)
32: get_local (Local 0)
65: i32.const (1073741823)
113: i32.and
65: i32.const (13)
71: i32.ne
27: select
11: end
[0]
2019-07-25 18:20:58,138: [13842] m.c.manticore:INFO: Generated testcase No. 0 - test
2019-07-25 18:20:58,143: [13842] m.c.manticore:INFO: Results in /home/ehennenfent/wasm/mcore_1evptpip
```

* Improve pickle-friendliness

* Convert wasm module instructions into internal type s

* Fix class naming conflict

Thanks Pickle

* Support bitvecs as an integer type

* Refactor execution to handle single instructions at a time

* Add Atomic Stacks

It's a very inefficient implementation, but we'll make it better

* Handle Concretize Exceptions

Needs a lot of fine tuning but it's a start

* Propagate symbols all the way to return

* Add test generation infrastructure

Still very basic but it's a start

* Attempt to make Travis happy

* Fix missing generation step

* Ignore stack exhaustsion and invalid modules

* Add basic shift instructions

* Add lots of arithmetic instructions

* Fix global initialization

* Add optimization for fixed size left shift

* Make Travis tests less verbose

* Blacken

* Properly handle modulo in shifting

* Add several control flow instructions

* Blacken

* Partially Fix return instruction

* Fix missing return value from look_forward

* Update look_forward to handle nested blocks

* Fix type sigs and executor copy/paste errors, deduplicate test names

* Fix an instruction exit issue

* Fix local ordering

* Blacken

* test_address now passes for wasm
Fixed issue with json2mc.py issue with testname uniqueness
Started some work on floating point

* i hate git

* i hate git

* Adding linenumber to testcase function name
minor float progress

* f32_cmp passed for wasm

* f64_cmp passes for wasm

* f32_bitwise passes except for the problem with assertEqual not working with nan

* f64_bitwise passes except for the problem with assertEqual not working with nan

* Fixed the tests to handle [nan] comparisons

* better handling of nans and infinites

* added integer backing value for f64

* Blacken

* Blacken

* f64.nearest

* Fix argument ordering and implement N-bit memory stores

* Use subtests

* Switch print to logger

* Modify control flow instructions to match spec

"Continuation" != the thing that goes inside the Label

* Teporarily disable call instructions to prevent infinite loop

Still need to figure out control flow for nested function calls

* Blacken

* Fixed infinite loop in _call via block depth tracking

The behavior of `end` needs to depend on how many labels and activations are on the stack. Every time we push or pop a label, we increment or decrement the counter for the current call frame.

* Blacken

* Initial `br` implementation

Probably still bugged

* Support if blocks without an else instruction

* Add call_indirect

* Add grow_memory

* i32_wrap_i64

* Add several arithmetic operators

* Copy i32.ctz impl from x86 TZCNT

* Fix off-by-one in br

I think this is the correct fix (it makes all the _br tests pass) but I'm leaving the TODO there just in case.

* Blacken...

* Fix lingering issue with if block splitting

* Fix bad args to sextend in i64_extend_s_i32

* Fix indirect call null check and Float binop return types

* Improve br_table indexing

See comment

* Remove vestigial argument popping in invoke

* Fix looking forward in _return and _block

* Blacken

* Generate trap and action tests

* Fix extraneous call to exec_for_test

* Fix missing traps in load instructions

* Fix c1=0 case in ctz/clz

Hopefully this also fixes the fact that the original CTZ implementations seems to have been intended for little-endian architectures

* Handle traps in-stride during test execution

* Treat zero division as trap

* Use ctypes to convert integers

* Blacken

* Apparently I was wrong about the CTZ endianness thing

* Better range checks for offsets

This really hammers home why checking whether ea + size is _greater than_ len(mem.data) is strange if mem.data is 0-indexed, but idk, that's what the spec says.

* Make stub functions return the correct number of results

* ctypes can't handle floats

* Add option to run the start function

* Correctly handle functions with unacceptable names

* Blacken

* Treat NotImplemented as a Skip

Also counts individual subtests directly instead of screwing around with trying to grep for them in the log.

* Fix unhandled negative indices

* Fixed some integer instruction bugs

i64/32:
div_s/u
shr_s/u
shl
clz

* Fix global and memory imports

* CC

* Make default loglevel debug

* Rework conversion from int

* Make float_store work

* _actually_ fix float_store

* Restore old float creation method

* Blacken

* Add range checks to float load/store

* Force cast to unsigned in _u instructions

* Fix yet another I32 -> I64 copy paste error

How many times do we have to teach you this lesson, ~old~ young man?

* Skip problematic float tests for now

* Handle floating point inaccuracies when dividing large ints

* Skip conversion and endianness tests

* Rotl, rotl, popcnt

* Blacken

* Add some conversion stubs

* Maybe fix conversions

* Improve reinterpret and float demotion

* Re-enable endianness tests

* Include WASM logging in verbosity controls

* Propagate constraints to executor

* Blacken

* Add prototype symbolic tests

* Add api for invocation to Manticore object

* Temporarily disable native/evm tests

There's no reason to tie up Travis for 40 minutes every time I push a commit.

* Fix wasmworld import

* Concretize br_if

* Pass arg generator to invoke

* Download updated WABT

It's fixed now!

* Generate Symbolic tests (partial)

* Follow m.run format

* Blacken

* Reinitialize Manticore every 50 tests

This should cut down on the performance issues we've been seeing

* Revert "Temporarily disable native/evm tests"

This reverts commit 7abd6c1333.

* I32 and I64 pass Symbolic

* Create new MC on reinit

* Blacken

* Fix return handling in test generation

* Concretize operand in `if_`

* Concretize memory addresses before access

In the long run, we should have a symbolic memory model

* Better memory access concretization (and br_table)

* Concretize float converions and indirect call operands

* Run start method

* Handle size errors in wrap+i64 and select

* Add timeout (and skip extra-problematic tests)

* Blacken

* Aggressively concretize floats

* Recast ints that should have been floats

* Better trap handling

* Unstash float concretization

* Fix log-lived 'trap' attr

* Blacken

* Concretize F32 when promoting to F64

_shouldn't_ strictly be necesssary, but the lazy conversion means that one of the tests fails. Until we get symbolic floating point, this will have to do.

* Docstrings for platforms/wasm.py

* Docstrings for ManticoreWASM

* Add docs for the module structure

* Add WASM to RTD

* missing docstring for Module itself

* Fix sphinx errors

* Bump RTD python version

I'm sorry and I swear I'll fix it later

* Reformat docstrings for sphinx

* Pacify RTD by adding newlines before params

* Missed a few

* Document the types

* Document executor

* A few more type docstrings

* Partial runtime_structure docs

* Finish documenting runtime structure

* Fix trailing whitespace

* Add collatz conjecture example

* Include undocumented members in RTD

* Add a bunch of events

* Appropriately connect event publishers

* Generate useful testcases

* Blacken

* Attempt at 3.6 backporting

* Optimize AtomicStack

* Sort solve_n output in test_solve_n

Very unclear why this is happening, but it's breaking a travis build
https://travis-ci.com/trailofbits/manticore/jobs/246451527

* Roll back to trusty?

Unsure if this is what's causing the EVM issues

* Create custom trap types

* Lazily evaluate overflows

May need to revert this if it slows down the Travis tests. It's 0.7% slower in the example I'm looking at, but that may not carry over.

* Blacken

* Address Dominik's feedback, remove stale TODOs

* Expand & Test Examples

* Fix relative path

* Fix relative path?

* capitalize Path

* Fix relative path for real

* Blacken...

* Test finalize

* Add supplemental tests for coverage

* Use dedicated helper for store access

* Handle quoted strings in imported module names

* Allow WASMWorld to hold multiple modules

Necessary for getting imports/exports to work, but still needs some changes before it passes the tests.

* Actually use export maps

* Fix most of the control flow issues with import tests

* Use available cores for test generation

* Escape module name

* Don't reinit modules that have already been initialized

* Use addresses for imports instead of copying

* Fix removed timeout argument

* Interleave registration and aliases

* Fix broken "None" check for address

* Blacken

* Make manual exports work with address syntax

* Import all registered modules

Only necessary for elem, but it's tricky to make changes just for one test, so *shrug emoji*

* Remove stringified annotations

Turns out those only work in the same file...

* Fix easy type issues in all but runtime structure

* Fix easy type issues in runtime_structure

* Fix* everything but imports

* Blacken

* Blacken 2

* Typo

* Merge module structure and runtime structure

* Fix instance vs type confusion

* Fix mypy and concrete tests

Still an issue with the symbolic tests

* Still generate symbolic tests though

I keep forgetting not to commit that file...

* Attempt to fix env for symbolic tests

* Blacken/mypy

* Fix missing supplemental environment variables arg

* Fix docstrings for new import style

* CC

* Bump timeout duration to 70 minutes

I don't like it, but adding more jobs to Travis isn't going to make our builds any faster

* Fix pycharm type, import, and docstring errors

* Explain type: ignore

* Missed two type:ignore's

* Fix CLI so it doesn't throw up on arbitrary modules
2019-11-08 07:58:24 -08:00
feliam 7142116fe1 Unsound symbolication - A general approach to handle symbolic imprecisions (#1526)
* Unsound symbolication and asorted fixes

* Add test for ULE

* More operators tests

* remove TT

* remove TT

* remove TT

* Add test for simple sha3 concretization

* Add test for simple sha3 concretization

* Add coverage to integration tests

* Add coverage to integration tests

* Added timeout to sha3 matching

* 2 files reformatted

* Review fixes

* black

* Fix KeepOnlyIfStorageChanges test

* remove dead code

* cleanups

* remove unused code

* Blacken

* Better concretize and calldataload

* black

* blkn

* mulmod fix

* Blacken

* fix calldata rel tests

* blk

* Better doc- review

* Better doc. Cleanups. sha3timeout changed

* sha3 - 42mins

* blkn

* Fix spelling issues, duplicate code, and add type hints

* Rm commented code

Done as a separate commit for easy reversion

* review

* blkn

* Remove hardcoded conf

* Fix expression bool test

* Fix sha3concrete test
2019-10-31 15:05:28 -03:00
Samuel E. Moelius a773d4a955 Eliminate bogus generate_testcase argument. (#1517) 2019-08-30 22:59:31 +02:00
feliam 0f1a4f90d0 Initial master merge into experimental state merging for native (#1482)
* Initial master merge into experimental st merging for native

* black

* Add C file and fix arguments

* Push (randomly failing) rudimentary state merging test

Fails nondeterministically, so like, that's a problem.

The issue I think lies in the fact that we pull states from the disk in a random order. Fix that so it's the right order, and this will hopefully work.

* Blacken

* Add basic_state_merging binary

* Restore seed parameter to random number generator

* Handle issues with instantiating Manitcore when multiple instances run simultaneously

* Replace print with logging
2019-08-05 17:02:22 -05:00
Artur Cygan f2099bd263 Update crackme example to match the description (#1502) 2019-08-01 16:10:49 -05:00
Luke Biery aa728d8e54 Update evm examples (#1486)
* Update syntax for refactor

* Update syntax for refactor

* Update syntax for refactor

* Update syntax for refactor
2019-07-22 18:14:40 -04:00
Luke Biery 97ef58acf9 Dev apidocs refactor (#1469)
* initial api docs refactor

change sphinx subsectioning and fix python docstring warnings

* Fix nostdlib compilation

* Update Native docs to include more about state

* Add ManticoreBase.__init__

Also reorganize other sections

* Try to force sphinx 2.0 on RTD

* Fix sphinx codeblock errors

* Blacken files

* Add newline before :param where necessary

* Blacken files

* Exclude evm instructions

* Move syminput to native

* Update symbolic header weight

* Begin adding documentation for callbacks

* Revert debugging print in MOVHPD

* Revert "Revert debugging print in MOVHPD"

This reverts commit b6a41326c9.

* Remove MOVHPD debug without formatting

* Update method paths in symbolic input

* Initial plugin skeletons
2019-06-28 17:10:39 -04:00
Eric Hennenfent ddb833dfa3 Restore old init hook behavior (#1447)
Now only takes the current state as a callback
2019-05-24 14:01:04 -05:00
Eric Hennenfent 451965f03a Blacken Manticore (#1438)
* Add black to CI tests

I'd rather have blackening be an optional check, but this will work for now

* Yep, messed up Travis

Again, sorry about the commit spam

* Bump accepted line length to 120

Going to auto-format at 100 but will allow 20% overruns

* Blacken

* Cap line length at 100

rip my undisciplined coding habits

* update black on the fly if necessary

* Ignore CC errors

* Only check modified files

* Only python files

* Re-blacken post merge

* Re-Blacken

* Fix deleted files

* Explicitly invoke python

The +x must not have gotten commited

* Forgot to blacken again

Hoisted on my own petard

* Re-blacken
2019-05-23 15:19:24 -05:00
Nikita Karetnikov 91bb222837 aarch64: do not use the 'procs' argument (#1439) 2019-05-17 16:51:48 -05:00
Eric Hennenfent 37aeae86b0 System Call Audit (#1384)
* Move unimplemented syscall logging to decorator

Adds [wrapt module](https://wrapt.readthedocs.io/en/latest/) from pypi

* Move unimplemented calls further down

* Add stubs for unimplemented system calls

* Move most stubs back to stub file

A few have been left in linux.py because they're necessary for other projects

* Codeclimate

* Fix discrepancies in syscall ABI

* Removed extraneous void argument for getpid and gettid
* Added missing sigsetsize parameter for rt_sigaction
* Removed incorrect 'flags' parameter on accept (should be on accept4
* Fixed 'SocketDesc' has no attribute 'close' error
* (Partially) Fixed duplicate constraint error with socket file descriptors

* Add gettimeofday and directory handling

* Hacky pipe implementation

Pretend that sockets are pipes and pipes are sockets

* Better support for MSR writing

* Default to success on directory operations

* Codeclimate

* More verbose unimplemented syscall printing

* Fix line overrun before CC calls me out

* Add ftruncate support

* Add link, unlink, and nanosleep

* Add getdents

* Add tests for mkdir and time functions

* Fix codeclimate and force rebuild

* Add pre and post hooks for syscalls

* Fix lingering single quotes

* Handle dumb invocations

* Type-o

* Make emulator follow logging behavior

* CC

* Fix pipe tests

* Fix shadowed name

* Add test for ftruncate

* Add link tests

* Add chmod test

* Rewrite docstrings

Best not to risk licensing issues

* Replace with simple returns

* Strip out unique implementations

Should bring the line count down a bit

* Check that unimplemented syscalls exhibit the proper behavior

* Fix missing keyword 'solver'

* Pass initial states to will_run callback

* Restore debug printouts after run

* Fix existing `init` calls

* Restore old exception behavior

* Move save_run_data to manticore native

* CC

* Refactor finalization call

* Fix save_run_data

I think it makes sense now

* Adjust expected output line counts

* CC [ci skip]

* Rename terminate state on exception to kill state

There's gotta be a better pun about 'overthrowing the state' in there somewhere

* Move time started to manticore_native
2019-05-16 14:45:32 -05:00
feliam a527e81830 ManticoreBase refactor (#1385)
* [WIP][WIP][WIP] Moving executor functionality to ManticoreBase and refactor concurrency shared data

* Workspace locks

* Concurrency flavor configurable from commandline

* Asserts and refactorrrrrrrs

* Remove unused callback

* Some CC

* Some CC

* Some CC

* Fix solver vs Z3Solver

* Make solver a singleton based on tid/pid. REfactor m._save. Fix some tests

* typo and evm bugfix

* Fix some tests referecing global solver

* Fix concolic tests and more global solver refs

* Fix tests

* CC fixes

* Fix tests. Fix testcase id generation

* Move profiling to a plugin and fix tests

* Add solver intance ref to mem test

* Fix mem workspace tests

* Fix output checking tests

* Fix z3solver ref

* Relax verbosity/log tests

* Moved Workers to its own file

* Relax output tests

* Relax output tests

* Fix profiling test

* merge

* Fix more tests

* Default multiprocessing

* Try to clean mcore __del__

* Change Worker life span

* Fix Single mode

* CC

* Merge branch 'master' into dev-executor-refactor

* revert verbosity travis

* CC and solver ref fix

* Relax ouput checking tests and some bugfixes

* running -> ready

* Fixing teeests

* add weak cache to _load

* del debug prints

* Adding config.py support for Enums

* Try/Remove generate_testcase event as it never occurs online. Fix tests

* Fix CC

* Kill the cache when start/stop run. Remove debugprints. clean tests

* Fix travis test _other_

* Fix native tests and timeout

* Fix state.must_be_true

* fix CC

* Merge and fix more tests

* Changing fstat tets...:
~'

* LLLLLLLLinux tests

* Skip unicorn concrete test for now

* Try fix CodeClimate

* Try fix CodeClimate

* Update evm examples to newest solidity

* Complete transformation of consts.mprocessing to enum

* Add blank line (codeclimate)

* Using the enum instead of the string

* Merge branch 'dev-executor-refactor' of github.com:trailofbits/manticore into dev-executor-refactor

* Using the enum instead of the string

* Merge branch 'master' into dev-executor-refactor

* Fix solver instance reference
2019-05-09 20:35:13 -03:00
Nikita Karetnikov 04a1cd3a31 aarch64: add everything except instructions (#1418)
* aarch64: add everything except instructions

Based on disconnect3d/aarch64/skeleton.

* fixup: do not use relative imports

Otherwise, 'scripts/travis_test.sh' fails to work.

* fixup: remove x86-specific code from '_reg_name'

* review: rename the file

* review: avoid wildcard imports

* Revert "fixup: remove x86-specific code from '_reg_name'"

* fixup: support ARM64 in '_reg_name'
2019-04-30 14:57:41 -05:00
Mark Mossberg efdac08147 Remove all broad excepts (#1405) 2019-04-18 13:59:28 -04:00
feliam 3a460d528d Load truffle json artifacts (#1376)
* WIP load truffle json artifacts

* No 'signature'

* Better truffle -artifact support

* CC

* Add basic test for truffle artifacs and update solidity linking

* Missing data file for truffle test

* CC

* Add full fledged example for truffle json

* Clean argument dosc for truffle

* Update manticore/ethereum/manticore.py

Co-Authored-By: feliam <[email protected]>

* Update manticore/ethereum/manticore.py

Necessary for 9a93cc5

* Add check for byte type and fix f string

* New lib linking bugfix

* Ability to pass constructor arguments to a Truffle JSON contract (#1376)

* Added sampling capabilities to RETURN.size

* Update manticore/ethereum/manticore.py

Co-Authored-By: feliam <[email protected]>

* Update manticore/ethereum/manticore.py

Co-Authored-By: feliam <[email protected]>
2019-03-18 15:50:56 -03:00
William Woodruff 71b8d2ba7d examples/linux: Refactor, fix crackme.py (#1378)
* examples/linux: Refactor crackme.py

* Dockerfile: Re-add LANG environment variable

See: #1300
See: #1137
2019-02-27 01:47:55 +01:00
Disconnect3d 580b358678 Refactor generate_testcase events and fix eth verbose trace plugin (#1305)
* WIP

* Add test

* Add missing test data file

* Fix missing testcase msg and assert it in test

* Remove Executor.generate_testcase - just publish event

* Remove redundant import

* Add ManticoreBase._last_run_stats

* Review fixes

* Remove redundant assertions in test_basic_arm

* Elapsed time calc fix
2018-12-17 15:04:00 +01:00
feliam e8a6321707 Add new auto CONCRETE test generator for eth vmTests (#1160)
* New auto evm test generator [WIP]

* New auto test generator for EVM

* New VMTest mcore translator

* Allow to auto-make single tests

* New concrete auto test for vmTests snad tests

* MAke bytecode bytes or array in tests too

* Make bytecode bytes or array in tests too more

* initial timestamp fix

* Fix jmpdest gas test and evm

* New tests style and some fixes

* __init__ in test folders

* Fix VMTest to unitests. Fix Array/ArraySlice default values

* Added rlp to setup.py

* Remove old deprecated tests

* Debugging storage/memory default values

* avoid gas consumption of empty reads

* CC

* Fix .written list for sliced arrays

* Move blocknumber/timestamp out of evmworld

* CC

* Several gas related fix. Frontier concrete pass

* Fix ArrayProxy tests

* Debug travis

* Split the regresion test so it prints . (dots)

* If value is in the cache then it is a known index. optimization

* add main to some unittest

* FIx test maker

* CC and skip slow test

* Skipp other slow test for now

* initial_timestamp -> timestamp

* more skipping

* Fix/mitigate expression copying issue

* Try fix concolic

* Fix simplification so it does not loose taint

* Fix no-testcases and verbosity in tests

* Extra visitor tests

* Extra visitor tests

* Better array slicing test. Code review. Basic fork support. VMTest concrete added

* bump pyevmasm version

* Remove pyevmasm version pinning

* Pyevmasm from github for now

* Setup.py crazyness

* Pyevmasm version bump - again

* Update manticore/platforms/evm.py

Co-Authored-By: feliam <[email protected]>

* Update manticore/platforms/evm.py

Co-Authored-By: feliam <[email protected]>

* Update manticore/platforms/evm.py

Co-Authored-By: feliam <[email protected]>

* Update manticore/platforms/evm.py

Co-Authored-By: feliam <[email protected]>

* Update manticore/platforms/evm.py

Co-Authored-By: feliam <[email protected]>

* Remove cache oddity and fix auto tests

* Add some type checks

* A fix
2018-12-11 15:58:33 -03:00
Disconnect3d 169904561a Make Manticore installable for separate targets (#1257)
* WIP

* WIP

* WIP

* Optimize imports

* WIP

* WIP

* WIP

* Fix setup.py comment

* Move abitypes to ethereum

* WIP: sth works lol

* Set verbosity from args

* WIP

* Fix evm logging + minor refactor

* Move ply dependency to evm setup

* NOTE: Removed Manticore.evm

* Check deps on setup, other improvements

* Proper ethereum/__init__ external imports

* Fix logging

* Fix imports in tests

* Fix scripts imports

* Fix imports in example scripts

* Fix tests?

* Remove debug print

* Fix circular import

* Fix imports in tests

* Fix verbosity test

* Fix typo

* Fix typo in test

* Fix measure cov

* Small review fixes

* Reviewfix: move STDIN_INPUT~ to consts.stdin_size

* Reviewfix: dont alias evm/native main funcs

* Fix missing consts.stdin_size

* Remove redundant list() call

* Fix types in docstrings

* Use relative imports

* Fix types in docstring

* Move invoke_model back to native State

* Split only to manticore, manticore[native]

* Fix dev and dev-noks setup extras

* Review fixes

* Move core/cpu -> native/cpu

* Update Changelog

* Import fixes

* Fix test imports

* Fix cpu automatic tests imports

* Fix imports in tests

* Fix imports & deprecated warning

* Fix imports

* Fix imports

* Fix imports

* Fix travis measure_cov path

* Add comment about native deps

* Use ManticoreBase.verbosity as static method

* Fix docs build

* Fix loger test

* Move imports: from manticore import issymbolic, istainted

* Docs fixes

* Fix issymbolic import in test
2018-12-03 21:20:55 +01:00
Stephan Tolksdorf a7bf228251 Fix all ResourceWarnings triggered by the test suite (#1224)
* Prevent ResourceWarnings in Z3Solver._stop_proc

* Wrap temporary uses of open in with statement

* Close program file in platforms.linux.Linux destructor

* Close files opened in tests

This commit also ensures that the tempdir created in _create_openat_state in test_linux.py is deleted at the end of the test.

* Fix binary file open modes

* Enable ResourceWarnings in Travis build

* Fix codeclimate issue: bad indent

* Suppress harmless ResourceWarning emitted during the initial abitypes parser table generation
2018-10-27 01:18:48 +02:00
Mark Mossberg 2a634b707c Eth updates (#1195)
* Add offset param to interface for extract a uint of variable with

* add leniency for g

* Rename attr

* other change

* bugs

* Automatically truncate sym vars that are too big

This is opinionated. Users use m.make_symbolic_value() to pass symbolic
arguments to contract functions. this creates a 256 bitvec by default.
if they do this for an argument that is not 256 bits, an `address` for
example, this would previously cause an AssertionError in ZEXTEND
because we're trying to ZEXTEND down to a smaller size (160 bits).

Manticore has all the information it needs to do the right thing here,
and automatically truncate the symbolic variable, so this implements
that.

* Fix logic bug in _readBE

* make padding default False because it's never used?

* h tx

* clean comment

* clearer comment

* fix bug, size should always be nbytes

* move type forgiveness up to the serialize() level

* fix logic bug - correctly loop

* update callsites correctly

* better err msg

* fix

* add tests for str args to ctor
2018-10-24 13:58:28 +08:00
Disconnect3d 5c44a34374 Convert format strings to literal string interpolation [rebased] (#1194)
* Convert format strings to literal string interpolation
This commit converts most (only a couple of exceptions) of the formatted strings in Manticore to follow PEP-0498, literal string interpolation.

* Converted most format strings to f-strings, except for some in tests

* Fixing codeclimate issues

* Updated the new format strings that appeared after merging, fixed a flipped set of parameters

* Fixed another incorrect quote typo

* fixed extra quote typo
2018-10-18 11:03:19 +02:00
redyoshi49q cab7a7f1f7 A massive amount of spellchecking (#1141) 2018-09-18 19:14:44 -04:00
Jay Little 501da0d267 python3 update to linux example makefile (#1122)
Ubuntu 18.04 doesn't alias python to python3
2018-09-05 12:01:14 +02:00
defunct a0c2f76311 Port remaining examples to py3 (#1042)
* port use_def

* port some scripts, cleanup

* ported `scripts/gdb.py` - untested

* misc
2018-08-10 15:28:41 -04:00
defunct 0d843457a9 Merge pull request #990 from defunctio/py3-optimization-pass
Python 3; optimization / stylization pass
2018-08-02 10:35:35 -04:00
defunct 822cc889e9 cleanup examples (#1010) 2018-08-01 18:19:59 -03:00
defunct e1b6f1a5bd super() 2018-07-27 23:59:06 +00:00
defunct e18016aa4a py3 yolo_master (feature / bugfixes) (#994)
* DAO detector + bugfixes

* The actual benchmark tests

* The actual benchmark tests

* CC

* Experiment reporting the finding at a JUMPI

* Fix taint. Detect returned overflowded data

* DAO -> Reentrancy

* DAO -> reentrancy, C -> Benchmark

* DAO -> reentrancy, C -> Benchmark

* Allow function names to have numbers

* Fix contract names in benchmark

* Fix contract names in benchmark

* Move default plugin registration

* Better regexp

* Fix minimal_bytecode example

* Fix Array Slice and test

* add tests

* correct other bug

* implement bytesM

* BROKEN partial progress

* need bytearray here

* rm cmt

* add basic tests for bytesM and bytes symbolic

* correct bytes symbolic test

* Refactor, clean bytesM handling

* Add initial symbolic 'bytes' handling

* refactor tests

* Unify symbolic/concrete bytes handling in bytesM/bytes

* Rm import

* Rm debug assert

* cc

* Visitor/migrate/simplify fixes to make the seth refactor pass

* Fix concolic?

* Fix concolic?

* CC

* bytesM fix

* Fix address and caller concretization on symb tx

* Fix/refactor symbolic address/caller concretization

* Fix caller concretization

* Fix expression visiting

* Fix account policy refactor

* Accept numbers in function names abitypes

* Simplify installation instructions to recommend install manticore only for the current user

* Run some tests in parallel (#970)

This PR splits the current test runner into three environments: 

1. Linux examples
2. Ethereum tests
3. Remaining tests

to faster complete each testing run. Ethereum tests include a number of integration tests that execute scripts to completion, which takes a while. We run them concurrently with other tests to save on execution time. The split is done by naming Ethereum tests differently (`eth_*.py` vs `test_*.py`) and updating what pattern unittest's `discover` uses.

This change also updates the installation script and chooses to forego installing Keystone for EVM tests as it takes a while, and it adds a `setup.cfg` config file so that Nose finds the eth tests as well by default.

* Be less verbose when testing

* Fix slicing wrongly reference to proxyArray. Fix #912

* Only export human/external tx in the testcase (#972)

* Make ManticoreEVM.make_symbolic_value size adjustable (#974)

* Make size adjustable

* Default to 256

* Dev evm yolo fix gas (#975)

* Fix gas stipend on CALL and check dao

* Add order dependence 1

* missing files

* 985

* formatting fixes; codeclimate

* review changes
2018-07-27 14:12:09 -04:00
defunct 664e53bff0 Python 3 (#968)
* Initial commit for Python3 branch that targets porting the following;
* Core manticore functionality
* x86/x64/arm architectures
* linux platform
* functioning unit tests for the above

* WIP ethereum support
notes:
* eth unit tests passing
* decree disabled
* ManticoreEVM.__init__ procs=1
* no iterpickle; see eth tests for setting stack size/recursion

* EVMAccount;
  add __hash__
  remove redefinition of __eq__

port/update EVM examples

* update stacksize / recursion limit

* pyevmasm dep updates, port linux examples

* update docs, tox/travis configs, port missing x86 unit test

* disable decree tests

* update example, update pyevmasm dep_links

* disable testArgumentsAssertions to speed up tests

* py35/36 json.loads compat fix

* fix summary output encoding

* disable eth binary test 808

* PEP479 - Python 3.7 support

* travisci; fix code coverage reporting

* port decree and tests

* codeclimate

* review changes;
	* cleanup
	* reset version
	* remove EVMAccount.__hash__, fix EVMWorld.get_balance()

* switch to official pyevmasm

* z3 version check - py3.5 fix

* pyevmasm; remove version requirement

* review changes;
	* remove redundant locking
	* shutdown executor manager on deletion

* review changes;
  * cleanup PY3FIX comments
  * _run_solc version ascii to utf encoding
  * save_summary - use filter instead of comprehension
  * comments

* __get_related - set literal mistake

* review changes;
  * remove redundant []
  * cleanup unused code, params, vars, imports
  * additional comments

* review changes;
  * fix symbolic const instruction decoding and add unit test
  * misc cleanup

* fix verbose logging output

* Update manticore defaults / requirements;
  * Changes docs for 18.04
  * Python >= 3.6

* minor review changes

* minor review changes, codeclimate
2018-07-25 10:10:44 -04:00
feliam 5c14e47d8b Better temporary expression handling (#944)
* Rearrange Detectors closer to ethereum. Fix Detector reporting for each state.

* codeclimate

* codeclimate

* Move detectors back to __main__. Commandline argument added

* Codeclimate

* Findings and global_findings for each detectot. Move global findings to its own file on reports

* Codeclimate

* Better commandline arguments. Better report on failing constructors

* Codeclimate. All assertion bench done

* Convenience evm stack function. Doc improvements

* Explicit reference in ded_evm_execute_instruction

* Assembler to solidity line number fixed

* Better variable names

* CC

* CC

* Regression tests

* Better int overflow detector (expensive)

* Added a few more IO reg tests

* CC

* CC

* CC

* Fix io tests. Remove dead code. Improve Transaction.set_result

* Account names, New ABI, user level constraining, state persistence

* merge bugfix

* Merge fix fix

* fix docstr

* Better type checks

* Fix minimal.py example

* CC and symbolic address

* CC

* CC

* CC

* CC

* Add bytearray-Array concatenation/ Fix ABI tests

* CC

* CC

* This fixes EVERYTHING!

* Fixing truly everything, maybe

* Review 1/10

* A few extra ABI serialization tests

* Better default arguments for constructor in multitx

* CC

* CC

* Exception EthereumError

* review

* Improve normal/contract_account filters. Move some ABI stuff to auxiliary funcs(CC)

* CC

* ABI.serialize smallfixes

* Catch parsing exceptions (review)

* Add comment to BitVec.cast()

* manticore/platforms/evm.py

Bugfixes and review

* manticore/platforms/evm.py

Bugfixes and review

* CC

* cs.migrate and bette temporary handling

* CC

* typo

* A Bugfix(tm)
2018-06-22 20:08:44 -03:00
feliam e0f5bce7ad Account names, ABI, and state persistence... (#926)
* Rearrange Detectors closer to ethereum. Fix Detector reporting for each state.

* codeclimate

* codeclimate

* Move detectors back to __main__. Commandline argument added

* Codeclimate

* Findings and global_findings for each detectot. Move global findings to its own file on reports

* Codeclimate

* Better commandline arguments. Better report on failing constructors

* Codeclimate. All assertion bench done

* Convenience evm stack function. Doc improvements

* Explicit reference in ded_evm_execute_instruction

* Assembler to solidity line number fixed

* Better variable names

* CC

* CC

* Regression tests

* Better int overflow detector (expensive)

* Added a few more IO reg tests

* CC

* CC

* CC

* Fix io tests. Remove dead code. Improve Transaction.set_result

* Account names, New ABI, user level constraining, state persistence

* merge bugfix

* Merge fix fix

* fix docstr

* Better type checks

* Fix minimal.py example

* CC and symbolic address

* CC

* CC

* CC

* CC

* Add bytearray-Array concatenation/ Fix ABI tests

* CC

* CC

* This fixes EVERYTHING!

* Fixing truly everything, maybe

* Review 1/10

* A few extra ABI serialization tests

* Better default arguments for constructor in multitx

* CC

* CC

* Exception EthereumError

* review

* Improve normal/contract_account filters. Move some ABI stuff to auxiliary funcs(CC)

* CC

* ABI.serialize smallfixes

* Catch parsing exceptions (review)

* Add comment to BitVec.cast()

* manticore/platforms/evm.py

Bugfixes and review

* manticore/platforms/evm.py

Bugfixes and review

* CC
2018-06-22 17:58:49 -03:00
feliam c29c3cc7e8 [WIP] Large EVM refactor (#843)
* Array value configurable. 8,16,32..256 bits (not prefixed to 8)

* Test Array256->256

* Remove old 8 bit assumptions

* Add arrayProxy copy constructors and remove old 8 bit assumptions

* remove old 8 bit assumptions

* remove old 8 bit assumptions

* Better ArrayProxy special case handling

* WiP-research Use raw no-optimized smtlib generic arrays as memory and storage

* remove pdebug print:

* Fix tests

* Review fixes

* review fix

* fix gas variable

* Review fixes

* git hell

* fix asm_to_smtlib

* Fix vm.gas access

* getvalue hack

* Improve internal API

* solver.can_be_true(cs, True) must check that cs is feasible

* Simpler callstack code

* Fix merge and tests

* Fix copy.copy

* Move state ids to private

* New internal add_transaction api

* Fix tx.return_data

* Remove comments

* Do not rely on .last_exception

* Fix evm tests

* peping

* peping

* Peping +test fixes

* Peping +test fixes

* Peping +test fixes

* Fix endTx pop vm

* Improve speed and simplifications. Remove internal_transaction mess

* Change test_eth.EthTests test a bit

* Fix some tests and CALLDATALOAD

* Add CALLCODE and reachability test frame

* Fix calldataload, msize. Add reachability test frame

* Fixlog

* Clean reachability test

* Fixe CODECOPY concrete

* Some style sdded

* peping

* Remove origin and price from EVM as it can be retrieved from human tx

* Added current_human_transaction and last_human_transaction

* peping

* Refactor tests

* Increased argument count  allowed CC

* Initial automatic library loading (no toposort)

* pep8

* Accepts any number of libaries on any ordering

* pep8 and relaxed codeclimate

* Simple library test

* merge libraries in

* Solve https://capturetheether.com/challenges/math/mapping/

* Remove debug print

* More robust path creation for test files

* DELEGATECALL support

* generate testcases only at finalize()

* Add functools32 as dependency for lru in py2

* Manticore Library PoC

* Favor bytearray

* Favor bytearray

* config

* Fix concolic and sha3 arraybyte-> str

* Fix concolic

* Fix typo

* remove absolute path from tests so it fits in link placeholders

* Shrink path spec for solc

* Hopefuly Fix solc invocation bug

* FFFFFFFFFIIIIIIIIIIIIX ITTTTTTTTTTTTTTT

* Better performance on visitors cache, and new ArraySlice

* better os joining, evm library link, and tx based events

* Fitering solidity functions plugin

* add itereator capabilities to ConstarintSet

* Some solver functions exported to state

* bye bye metaclass dynamic check

* extra simplification

* Improved symbolic gas and other bugfixes

* Better visit_BitVecExtract

* Default initial timestamp and block number

* 99 typos

* Re-enable account.named_func() trick

* typo

* Better stronger

* Added default symbolic arguments to constructor..

* Remove constant marked functions from multi_tx

* Add default 16G memory cap to z3

* concretized_args now supports concretizing address arguments to all possible accounts

* Fix coverage calc

* reset fails sometimes

* PEP8 ROA (#900)

* pep8 formatting

* Code Climate updates

* cyclocomplexity--

* Refunds moved from the instruction to the the exeption catching

* merge

* merge

* accept both bytearray and str

* doc

* blockhash changed and returns a symbolic value now, so these tests are wrong

* A bugfix

* pep8/complexity updates

* remove more questionable tests

same thing as with the blockhash thing. we're doing things a little more symbolically now so comparing to concrete will not work

* Accomodate our chubby State class

* Keep moving the refunds

* Disble reachability test for now

* Remove double declaring log for now

* Fix coverage calculation

was prev creating a generator, which you can't do `in` on??

* require 1 tx for int overflow test

* Fix path tests

* Remove debug print in tests

* Only enable gas when necessary

* codeclimate fixes

* Fix concretize_args

* only 1 tx

* Disable max/min support in solver
2018-05-25 19:22:49 -04:00
feliam 68c4716872 Remove solidity in minimal.py example (#895)
* Remove all needs for solidity in minimal.py example

* Separate evm only minimal example
2018-05-18 18:14:34 -03:00
feliam efbcd084fd Rollback to old mini assembler only example (#889)
* Rollback to old mini assembler only example

* Rollback reordering and remove comment

* Ah no. We need to keep the states in the lists until the end

* Update ethereum.py

* Use calculate coverage funciton
2018-05-16 11:39:23 -03:00