Commit Graph
775 Commits
Author SHA1 Message Date
Eric Hennenfent 3ffafd57ec Revert Unicorn Dependency Update (#1459)
* Partial revert for Unicorn Dependency Update

* re-blacken

Okay fine I fixed it. Happy?
0.3.0
2019-06-06 17:54:52 -04:00
Eric Hennenfent a0cca2f09c Manticore 0.3.0 (#1448)
* Manticore 0.3.0

* Explicitly note lack of Constantinople support 

Will hopefully head-off people with issues regarding unimplemented instructions and incorrect behavior

* Add blurb about solver

* Link to state machine diagram

* Fix typo

* Fix duplicated section

And fix date
2019-06-06 11:02:28 -04:00
Eric Hennenfent 05890c180e Install native deps on rtd (#1457) 2019-06-04 17:45:10 -04:00
Eric Kilmer b20b019e6d Fix check for symbolic syscall argument (#1452) 2019-06-04 15:11:06 -04:00
Eric Kilmer ebdf2eb4cc Use deque instead of list for FIFO queue in Linux Socket buffer (#1453) 2019-06-04 15:10:42 -04:00
Eric Hennenfent ddb833dfa3 Restore old init hook behavior (#1447)
Now only takes the current state as a callback
2019-05-24 14:01:04 -05:00
Eric Hennenfent 95f223ff4c Fix invalid pip argument (#1445)
It's --upgrade (or -U), not --update

I'm not sure why, but I guess Travis' build cache meant we didn't catch this ahead of time
2019-05-23 16:13:18 -05:00
Eric Hennenfent 451965f03a Blacken Manticore (#1438)
* Add black to CI tests

I'd rather have blackening be an optional check, but this will work for now

* Yep, messed up Travis

Again, sorry about the commit spam

* Bump accepted line length to 120

Going to auto-format at 100 but will allow 20% overruns

* Blacken

* Cap line length at 100

rip my undisciplined coding habits

* update black on the fly if necessary

* Ignore CC errors

* Only check modified files

* Only python files

* Re-blacken post merge

* Re-Blacken

* Fix deleted files

* Explicitly invoke python

The +x must not have gotten commited

* Forgot to blacken again

Hoisted on my own petard

* Re-blacken
2019-05-23 15:19:24 -05:00
Eric Hennenfent 581f43d144 Relegate coverage file to plugin (#1442)
Thanks [@dnivra](https://empirehacking.slack.com/archives/C3PTWK7UM/p1558401830048200)!
2019-05-23 14:04:20 -05:00
William WoodruffandEric Hennenfent 7de6371983 x86: movhps support (#1444)
* x86: MOVHPS support

Closes #1432.

* Update manticore/native/cpu/x86.py

Co-Authored-By: Eric Hennenfent <[email protected]>

* tests: Add movhps tests

* tests: Fix tests

* tests: Remove relative import

Works locally, fails on CI.

* tests: Fix movhps shellcode

Was using qword and not qword ptr.

* tests: Swap assertions
2019-05-23 11:41:39 -05:00
Eric Hennenfent d72d556365 Delete statically generated tests (#1443)
Tests are green and we discussed this on slack, so I'm going to go ahead and merge
2019-05-22 16:31:34 -05:00
Eric Hennenfent 6867e4ecb9 Hardcode expected matching concrete registers (#1437)
Tests are green and we discussed this at the dev meeting, so I'm going to go ahead and merge

* Hardcode the set of registers we expect to pass

* More verbose error message

* Remove vector regs
2019-05-22 16:30:33 -05:00
feliamandEric Hennenfent 39a23a559d Symbolic tests rf otf (#1441)
* Add assertions to auto test gen

* Add symbolic tests

* Make calldata symbolic

* EVM: Support exp aka pow (#1361)

* EVM: Add support for EXP with concrete/solvable exponent

Adds support for EXP aka POW, with concrete or solvable exponent, fixing https://github.com/trailofbits/manticore/issues/1005 effectively.

Not sure if we can do it any better - it seems z3 doesn't have exp/pow for bitvectors.

* Update operators.py

* Update evm.py

* Use concretized_args

* Move Operators.POW to EVM._exp

* Extend travis wait for output to 30m

* Extend travis build to 60m...

* Fix Operators.ITE -> Operators.ITEBV

* Split ethereum travis job to two jobs

* EVM.EXP: concretize base=SAMPLED

* Fix concrete tests: use to_constant

* Fix set storage in concrete tests

* Split ethereum_vm tests into concrete and symbolic

* Fix travis tests

* Split symbolic tests into two jobs

* Split VMTests even more

* More tests split

* [WIP][WIP][WIP] Moving executor functionality to ManticoreBase and refactor concurrency shared data

* Workspace locks

* Concurrency flavor configurable from commandline

* Asserts and refactorrrrrrrs

* Remove unused callback

* Some CC

* Some CC

* Some CC

* Fix solver vs Z3Solver

* Make solver a singleton based on tid/pid. REfactor m._save. Fix some tests

* typo and evm bugfix

* Fix some tests referecing global solver

* Fix concolic tests and more global solver refs

* Fix tests

* CC fixes

* Fix tests. Fix testcase id generation

* Move profiling to a plugin and fix tests

* Add solver intance ref to mem test

* Fix mem workspace tests

* Fix output checking tests

* Fix z3solver ref

* Relax verbosity/log tests

* Moved Workers to its own file

* Relax output tests

* Relax output tests

* Fix profiling test

* Fix more tests

* Default multiprocessing

* Try to clean mcore __del__

* Change Worker life span

* Fix Single mode

* CC

* revert verbosity travis

* CC and solver ref fix

* Relax ouput checking tests and some bugfixes

* running -> ready

* Fixing teeests

* add weak cache to _load

* del debug prints

* Adding config.py support for Enums

* Try/Remove generate_testcase event as it never occurs online. Fix tests

* Fix CC

* Kill the cache when start/stop run. Remove debugprints. clean tests

* Fix travis test _other_

* Fix native tests and timeout

* Fix state.must_be_true

* fix CC

* Changing fstat tets...:
~'

* LLLLLLLLinux tests

* Skip unicorn concrete test for now

* Try fix CodeClimate

* Try fix CodeClimate

* Update evm examples to newest solidity

* Complete transformation of consts.mprocessing to enum

* Add blank line (codeclimate)

* Using the enum instead of the string

* Using the enum instead of the string

* Merge and fix

* CC and debug print

* Move fee consumption to checkpoint so it is not done twice. Fix frontier test generator

* Fix Job Count (and force travis rebuild)

```
0.02s$ ./cc-test-reporter sum-coverage --output - --parts $JOB_COUNT coverage/codeclimate.*.json | ./cc-test-reporter upload-coverage --input -
Error: expected 3 parts, received 4 parts
```

* Add tx number to testcase log

* Attempt to do vmtests on the fly

* Del test verbosity

* Update .travis.yml

Co-Authored-By: Eric Hennenfent <[email protected]>
2019-05-21 17:07:47 -05:00
Eric Kilmer 43b9a0c957 Unicorn dependency update (#1440)
As noted, we should roll this back whenever Unicorn pushes to PyPi

* Remove write to Unicorn x86 FS register

In latest version of unicorn (778171fc9546c1fc3d1341ff1151eab379848ea0)
a write to FS will cause segfault or hang if we try to write to it.

Issue #1434

* Update unicorn dependency to latest commit on master

Latest Unicorn master commit is 778171fc9546c1fc3d1341ff1151eab379848ea0

Issue #1434
2019-05-21 15:49:45 -05:00
Nikita Karetnikov 91bb222837 aarch64: do not use the 'procs' argument (#1439) 2019-05-17 16:51:48 -05:00
feliam 00d551d40f Symbolic tests rf (#1431)
* Add assertions to auto test gen

* Add symbolic tests

* Make calldata symbolic

* EVM: Support exp aka pow (#1361)

* EVM: Add support for EXP with concrete/solvable exponent

Adds support for EXP aka POW, with concrete or solvable exponent, fixing https://github.com/trailofbits/manticore/issues/1005 effectively.

Not sure if we can do it any better - it seems z3 doesn't have exp/pow for bitvectors.

* Update operators.py

* Update evm.py

* Use concretized_args

* Move Operators.POW to EVM._exp

* Extend travis wait for output to 30m

* Extend travis build to 60m...

* Fix Operators.ITE -> Operators.ITEBV

* Split ethereum travis job to two jobs

* EVM.EXP: concretize base=SAMPLED

* Fix concrete tests: use to_constant

* Fix set storage in concrete tests

* Split ethereum_vm tests into concrete and symbolic

* Fix travis tests

* Split symbolic tests into two jobs

* Split VMTests even more

* More tests split

* [WIP][WIP][WIP] Moving executor functionality to ManticoreBase and refactor concurrency shared data

* Workspace locks

* Concurrency flavor configurable from commandline

* Asserts and refactorrrrrrrs

* Remove unused callback

* Some CC

* Some CC

* Some CC

* Fix solver vs Z3Solver

* Make solver a singleton based on tid/pid. REfactor m._save. Fix some tests

* typo and evm bugfix

* Fix some tests referecing global solver

* Fix concolic tests and more global solver refs

* Fix tests

* CC fixes

* Fix tests. Fix testcase id generation

* Move profiling to a plugin and fix tests

* Add solver intance ref to mem test

* Fix mem workspace tests

* Fix output checking tests

* Fix z3solver ref

* Relax verbosity/log tests

* Moved Workers to its own file

* Relax output tests

* Relax output tests

* Fix profiling test

* Fix more tests

* Default multiprocessing

* Try to clean mcore __del__

* Change Worker life span

* Fix Single mode

* CC

* revert verbosity travis

* CC and solver ref fix

* Relax ouput checking tests and some bugfixes

* running -> ready

* Fixing teeests

* add weak cache to _load

* del debug prints

* Adding config.py support for Enums

* Try/Remove generate_testcase event as it never occurs online. Fix tests

* Fix CC

* Kill the cache when start/stop run. Remove debugprints. clean tests

* Fix travis test _other_

* Fix native tests and timeout

* Fix state.must_be_true

* fix CC

* Changing fstat tets...:
~'

* LLLLLLLLinux tests

* Skip unicorn concrete test for now

* Try fix CodeClimate

* Try fix CodeClimate

* Update evm examples to newest solidity

* Complete transformation of consts.mprocessing to enum

* Add blank line (codeclimate)

* Using the enum instead of the string

* Using the enum instead of the string

* Merge and fix

* CC and debug print

* Move fee consumption to checkpoint so it is not done twice. Fix frontier test generator

* Fix Job Count (and force travis rebuild)

```
0.02s$ ./cc-test-reporter sum-coverage --output - --parts $JOB_COUNT coverage/codeclimate.*.json | ./cc-test-reporter upload-coverage --input -
Error: expected 3 parts, received 4 parts
```

* Add tx number to testcase log

* Del test verbosity
2019-05-17 16:49:26 -05:00
Nikita Karetnikov 65b73145dd AArch64: fix ldrb size (#1433)
* aarch64: test strb followed by ldrb (immediate)

* aarch64: use the right size in '_ldr_str_immediate'

Discovered due to 'ldrb w11, [x9]' failing in symbolic mode.
2019-05-16 14:46:52 -05:00
Eric Hennenfent 37aeae86b0 System Call Audit (#1384)
* Move unimplemented syscall logging to decorator

Adds [wrapt module](https://wrapt.readthedocs.io/en/latest/) from pypi

* Move unimplemented calls further down

* Add stubs for unimplemented system calls

* Move most stubs back to stub file

A few have been left in linux.py because they're necessary for other projects

* Codeclimate

* Fix discrepancies in syscall ABI

* Removed extraneous void argument for getpid and gettid
* Added missing sigsetsize parameter for rt_sigaction
* Removed incorrect 'flags' parameter on accept (should be on accept4
* Fixed 'SocketDesc' has no attribute 'close' error
* (Partially) Fixed duplicate constraint error with socket file descriptors

* Add gettimeofday and directory handling

* Hacky pipe implementation

Pretend that sockets are pipes and pipes are sockets

* Better support for MSR writing

* Default to success on directory operations

* Codeclimate

* More verbose unimplemented syscall printing

* Fix line overrun before CC calls me out

* Add ftruncate support

* Add link, unlink, and nanosleep

* Add getdents

* Add tests for mkdir and time functions

* Fix codeclimate and force rebuild

* Add pre and post hooks for syscalls

* Fix lingering single quotes

* Handle dumb invocations

* Type-o

* Make emulator follow logging behavior

* CC

* Fix pipe tests

* Fix shadowed name

* Add test for ftruncate

* Add link tests

* Add chmod test

* Rewrite docstrings

Best not to risk licensing issues

* Replace with simple returns

* Strip out unique implementations

Should bring the line count down a bit

* Check that unimplemented syscalls exhibit the proper behavior

* Fix missing keyword 'solver'

* Pass initial states to will_run callback

* Restore debug printouts after run

* Fix existing `init` calls

* Restore old exception behavior

* Move save_run_data to manticore native

* CC

* Refactor finalization call

* Fix save_run_data

I think it makes sense now

* Adjust expected output line counts

* CC [ci skip]

* Rename terminate state on exception to kill state

There's gotta be a better pun about 'overthrowing the state' in there somewhere

* Move time started to manticore_native
2019-05-16 14:45:32 -05:00
feliam a527e81830 ManticoreBase refactor (#1385)
* [WIP][WIP][WIP] Moving executor functionality to ManticoreBase and refactor concurrency shared data

* Workspace locks

* Concurrency flavor configurable from commandline

* Asserts and refactorrrrrrrs

* Remove unused callback

* Some CC

* Some CC

* Some CC

* Fix solver vs Z3Solver

* Make solver a singleton based on tid/pid. REfactor m._save. Fix some tests

* typo and evm bugfix

* Fix some tests referecing global solver

* Fix concolic tests and more global solver refs

* Fix tests

* CC fixes

* Fix tests. Fix testcase id generation

* Move profiling to a plugin and fix tests

* Add solver intance ref to mem test

* Fix mem workspace tests

* Fix output checking tests

* Fix z3solver ref

* Relax verbosity/log tests

* Moved Workers to its own file

* Relax output tests

* Relax output tests

* Fix profiling test

* merge

* Fix more tests

* Default multiprocessing

* Try to clean mcore __del__

* Change Worker life span

* Fix Single mode

* CC

* Merge branch 'master' into dev-executor-refactor

* revert verbosity travis

* CC and solver ref fix

* Relax ouput checking tests and some bugfixes

* running -> ready

* Fixing teeests

* add weak cache to _load

* del debug prints

* Adding config.py support for Enums

* Try/Remove generate_testcase event as it never occurs online. Fix tests

* Fix CC

* Kill the cache when start/stop run. Remove debugprints. clean tests

* Fix travis test _other_

* Fix native tests and timeout

* Fix state.must_be_true

* fix CC

* Merge and fix more tests

* Changing fstat tets...:
~'

* LLLLLLLLinux tests

* Skip unicorn concrete test for now

* Try fix CodeClimate

* Try fix CodeClimate

* Update evm examples to newest solidity

* Complete transformation of consts.mprocessing to enum

* Add blank line (codeclimate)

* Using the enum instead of the string

* Merge branch 'dev-executor-refactor' of github.com:trailofbits/manticore into dev-executor-refactor

* Using the enum instead of the string

* Merge branch 'master' into dev-executor-refactor

* Fix solver instance reference
2019-05-09 20:35:13 -03:00
Eric Hennenfent df072fbab9 Add missing checks for ARM boundaries (#1429)
Previously only supported ARM64 and x86. Now adds support for ARMv7
2019-05-06 13:19:44 -05:00
Eric Hennenfent 5f37fb6fa6 Merge pull request #1424 from trailofbits/merge-aarch64
Final AArch64 Merge (unsquashed)
2019-04-30 17:26:33 -05:00
Nikita Karetnikov a6b2f51b61 aarch64: add instruction tests: T-U (#1423)
* aarch64: add instruction tests: A-B

* aarch64: add instruction tests: C-L

* aarch64: add instruction tests: M-S

* aarch64: add instruction tests: T-U

* fixup: do not use relative imports

Otherwise, 'scripts/travis_test.sh' fails to work.

* fixup: remove x86-specific code from '_reg_name'

* review: rename the file

* review: avoid wildcard imports

* Revert "fixup: remove x86-specific code from '_reg_name'"

* fixup: support ARM64 in '_reg_name'
2019-04-30 15:08:36 -05:00
Nikita Karetnikov 50340c9b14 aarch64: add instruction tests: M-S (#1422)
* aarch64: add instruction tests: A-B

* aarch64: add instruction tests: C-L

* aarch64: add instruction tests: M-S

* fixup: do not use relative imports

Otherwise, 'scripts/travis_test.sh' fails to work.

* fixup: remove x86-specific code from '_reg_name'

* review: rename the file

* review: avoid wildcard imports

* Revert "fixup: remove x86-specific code from '_reg_name'"

* fixup: support ARM64 in '_reg_name'
2019-04-30 15:07:29 -05:00
Nikita Karetnikov 4c406b1776 aarch64: add instruction tests: C-L (#1421)
* aarch64: add instruction tests: A-B

* aarch64: add instruction tests: C-L

* fixup: do not use relative imports

Otherwise, 'scripts/travis_test.sh' fails to work.

* fixup: remove x86-specific code from '_reg_name'

* review: rename the file

* review: avoid wildcard imports

* Revert "fixup: remove x86-specific code from '_reg_name'"

* fixup: support ARM64 in '_reg_name'
2019-04-30 15:05:56 -05:00
Nikita Karetnikov dd6e56575e aarch64: add instruction tests: A-B (#1420)
* aarch64: add everything except instructions

Based on disconnect3d/aarch64/skeleton.

* aarch64: add instructions

* aarch64: add instruction tests: A-B

* fixup: do not use relative imports

Otherwise, 'scripts/travis_test.sh' fails to work.

* fixup: remove x86-specific code from '_reg_name'

* review: rename the file

* review: avoid wildcard imports

* Revert "fixup: remove x86-specific code from '_reg_name'"

* fixup: support ARM64 in '_reg_name'
2019-04-30 15:03:04 -05:00
Eric Hennenfent b590c41339 Fix conflicts 2019-04-30 15:02:31 -05:00
Eric Hennenfent ff4d419971 merge into merge-aarch64 2019-04-30 15:01:14 -05:00
Nikita Karetnikov 04a1cd3a31 aarch64: add everything except instructions (#1418)
* aarch64: add everything except instructions

Based on disconnect3d/aarch64/skeleton.

* fixup: do not use relative imports

Otherwise, 'scripts/travis_test.sh' fails to work.

* fixup: remove x86-specific code from '_reg_name'

* review: rename the file

* review: avoid wildcard imports

* Revert "fixup: remove x86-specific code from '_reg_name'"

* fixup: support ARM64 in '_reg_name'
2019-04-30 14:57:41 -05:00
Nikita Karetnikov a01593f41a fixup: support ARM64 in '_reg_name' 2019-04-30 20:21:41 +03:00
Nikita Karetnikov f6db31cc24 Revert "fixup: remove x86-specific code from '_reg_name'" 2019-04-30 20:21:41 +03:00
Nikita Karetnikov e33f165bd2 review: avoid wildcard imports 2019-04-30 16:11:32 +03:00
Nikita Karetnikov 8769389202 review: rename the file 2019-04-30 16:11:32 +03:00
Nikita Karetnikov b5dcbb7fa0 fixup: remove x86-specific code from '_reg_name' 2019-04-30 14:52:10 +03:00
Nikita Karetnikov ab70743da9 fixup: do not use relative imports
Otherwise, 'scripts/travis_test.sh' fails to work.
2019-04-29 15:57:37 +03:00
feliam 54c5a15b11 Generates a more sensible symbolic default for constructor arguments (#1414)
* Fix #1412

* Add basic test

* CC
2019-04-29 09:37:24 -03:00
Nikita Karetnikov 229d2d6925 aarch64: add instructions 2019-04-29 13:46:14 +03:00
Nikita Karetnikov 0002b20731 aarch64: add everything except instructions
Based on disconnect3d/aarch64/skeleton.
2019-04-29 13:45:32 +03:00
Eric Hennenfent 2a1c341a0c Switches the Travis-CI badge from .org to .com (#1416) 2019-04-25 17:58:17 -05:00
Catena cyber 4d5d5b8f51 Performance optimization : use set instead of list (#1415)
Better average performance for `ìn` operation : O(1) instead of O(n)
2019-04-25 17:57:57 -05:00
Eric Hennenfent 22cdf85ff1 Fix DECREE Magic Pages (#1413)
Travis is almost done and the relevant parts have passed
2019-04-23 19:42:02 -05:00
Alan aadd7a06e7 Migrate sandshrew concolic executor as Manticore script (#1396)
* Add consolidated sandshrew concolic testing script

* Migrate repository content into one-file script
* Reorganize for code style consistency

* Optimizations to sandshrew script

* Add sandshrew example test cases

* Move test cases over from standalone repo
* Add Makefile

* Add sandshrew documentation files
2019-04-22 16:15:19 -05:00
Eric Hennenfent 9c9b619d92 Fix Travis Coverage (#1409)
* Fix JOB_COUNT env var

That took way longer than it should have to figure out

* Fix incorrect comment

Whoops...
2019-04-19 16:18:14 -05:00
Eric Hennenfent 3fc2faa588 Fix CodeClimate Master (#1408)
Fixes a few CC bugs that made it into master, and marks the "too many functions" issues that we always ignore as wontfix.

Travis is green so I'm just going to merge this.
2019-04-19 13:37:08 -05:00
Mark Mossberg efdac08147 Remove all broad excepts (#1405) 2019-04-18 13:59:28 -04:00
Logan Campos 220b9499bf PEP-257: Change docstrings using single quotes to double quotes (#1401)
* PEP-257: change docustrings using single quotes to double quotes

Signed-off-by: binaryflesh <[email protected]>

* fix pseudocode in docstring

Signed-off-by: binaryflesh <[email protected]>
2019-04-17 11:53:01 +02:00
Logan Campos d0a5ec2f67 Add bash syntax highlighting to shell examples in README (#1402)
Signed-off-by: binaryflesh <[email protected]>
2019-04-17 11:49:43 +02:00
Gustavo Grieco 67c1d66e26 Update README.md (#1397) 2019-04-04 02:27:08 +02:00
Evan Sultanik 913c2583c6 Concretize pc if possible to resolve issue #1394 (#1395) 2019-03-28 17:23:50 -03:00
Eric Kilmer a1f8fb3e31 Fix logic for solc relative path (#1393)
This allows solc-select (which runs in docker) to correctly access the
relative path of a sol file.
2019-03-26 16:14:25 +01:00
Catena cyber f4582b386c Only publish events when there is one subscriber (#1388)
* Only publish events when there is one subscriber

Performance optimization

* Move sub_events to init.

* Revert "Move sub_events to init."

This reverts commit 0859a63020.

* Update event.py
2019-03-26 12:14:02 -03:00