Only use a lower bound (not sure if the lower bound actually means
anything with regards to what Manticore _requires_, but better keep it
around just in case)
Protobuf introduced major version 4.21.0, which contains breaking changes
This is a quick fix which pins protobuf to 3.20.1 before we eventually migrate the codebase to proto4
Co-authored-by: kokrui <[email protected]>
* Use latest tagged capstone==5.0.0rc2
* Fix aarch64
* Fix ARM
* New x86 instrution test
Co-authored-by: feliam <[email protected]>
Co-authored-by: Boyan MILANOV <[email protected]>
* Switch to stable Black version
`black` has recently stabilized on version 22, which means that we can loosen the version restrictions a bit to get upstream bugfixes, without needing to worry about our formatting breaking unexpectedly. In 2023, they may issue a new release with breaking changes, but we won't be automatically upgraded.
There have been a large number of formatting changes since 20.8.b1, which is the last version we standardized on. This will likely require rebase of several of the existing PRs, but with relatively few of them currently active, I think it's as good a time as any.
More info about black's further development in their [stability policy](https://black.readthedocs.io/en/latest/the_black_code_style/index.html#stability-policy)
* Add new formatting changes
Mostly docstrings and exponentiation
* Update to crytic-compile 0.2.2
This PR bumps our required version of `crytic-compile` to the latest release (0.2.2). Tweaks the initialization routine in `ethereum/manticore.py` in order to account for breaking changes in https://github.com/crytic/crytic-compile/pull/223
* Delete accidental files
This reverts commit e5fe245d06.
* Fix variable name reuse
* Add version info to installations
* Let's try a newer version?
* Use fixed yices release
* Really? No relative path support?
* Can't handle newer C compilers
Ugh we should really do something about that
* Let's try 3.8
* Back to 3.7
3.6 is EOL soon, so we can justify jumping up to a more recent version, but 3.8 might still be too new for some people.
* Find & Replace 3.6 -> 3.7
* Bump setup-python version
* Fix additional 3.6 usage
* Manticore 0.3.6
* Create release.yml
* Bump point release number for development releases
* Don't include coveralls in release run
* Update nightly build notice
* Update CHANGELOG.md
* Add changelog line for #2420
* Disable write back once we hit our stop in Unicorn emulation
* Make emulator reinitialize after write backs are disabled
* Fix linting on test_general
Extremely strange that my local black==19.10b0 doesn't catch this, but the one on GH actions does
* Add a Rust/Unicorn resumption test
* Make rusticorn binary actually check behavior
* Delete duplicated test method
* Fix addresses and improve error handling
* Fix Unicorn resume
* Update unicorn to latest 1.0.2
* Update capstone to latest 4.0.2
* Correctly process memory maps
* Synchronize data from manticore to unicorn upon resume
* Fix some issues with FS register and segments
* Fix some synchronization with CPUID instruction as compared to Unicorn
* X86 syscall instruction breakout for semantics
Co-authored-by: Eric Hennenfent <[email protected]>
* Support for TUI (#1605)
* Update worker thread for server creation
* Add necessary files for TUI connectivity
* Add necessary files for TUI connectivity
* Update MonitorWorker
* Update protocol
* Blacken
* Update setup.py dependencies
* Remove state debugging messages
* Update setup.py to build protobuf protocol upon install
* Remove previously generated state_pb2.py
* Change subprocess.Popen to subprocess.check_output
* Remove extraneous output
* First attempt at fixing protobuf installation
It might work, it might not. We'll let the CI sort it out.
* Can't forget the f-string
* Error on missing protoc
* Disable auto-generation of protobuf file
* Ignore pb2_errors
* Disable monitor start
See if this makes the EVM tests pass
Co-authored-by: Eric Hennenfent <[email protected]>
* Add log monitoring
* Log monitoring via TCP
* Swittch to rendering state lists directly
* Extraneous line
* Switch log buffer to multiprocessing queue
* Create state transition events
Should make it possible to track movements between state lists
* Plug new events into context
This will break the state merging plugin (but I'll fix it eventually)
* move most enums to their own module
* Blacken
* Add DaemonThread from TUI branch
* Add interface for registering daemon threads
* Timestamp StateDescriptor upon updates
* Capture return value
* Blacken
* Add solver wrapper to StateBase
* Add `solve` events to all instances of SelectedSolver.instance()
* Remove executor constraints from WASM
* Add solve events to memory.py
* Add intermittent execution event
* Be more generous with states whose initialization we missed
* Add Native callback for updating state descriptor
* Fix state killing
* Blacken
* codecov: Remove outdated 'yml' entry in CI
From these commits
https://github.com/codecov/codecov-action/commit/ebea5cacdf7d0b843f66bcb1ae27d8f27b758e81https://github.com/codecov/codecov-action/commit/49c86d6a5fd072b05c31a42baa67b8fb2e87c8f7
* Add solve event to evm
Make warning messages better
Debug GH actions
Revert "Debug GH actions"
This reverts commit f575eea3c3a09dbf2cd2b8e81940fc4297fdf039.
Fix some pycharm-detected problems
Make symbolic function error message more verbose
Add solve to published events
Loud errors in callbacks by default
Trying to find out what's killing truffle
Revert "Trying to find out what's killing truffle"
This reverts commit 8bd02245ccc2ce54c9161072eba231880c084874.
Revert "Make symbolic function error message more verbose"
This reverts commit bd3e90cdb0dfabf0ac874badad723be50998f63b.
Debugging Truffle
Restore introspector
Add try_except on every callback
Unconditionally print error message
Add traceback
Update event.py
Debug subscriptions
Debug arguments to callbacks
Different debug msg
1ast arg
Print statement debugging...
Pass in `None` as state
Revert "Add try_except on every callback"
This reverts commit 1c689dd43c619ba322cda5e7283686308c8db4e0.
* Drop solve events outside of a state context
Forgot did_solve
Remove traceback
* Fix must/cannot_be_null usage
* Fix missing solve event
* Partially restore old did_fork_state ABI
* Called internally
* Clone iterators instead of creating a list
* Use isgenerator instead of checking if iterable
* Fix snapshot restoration
* Slightly improve Unicorn test API usage
* Temporarily disable property verifier tests
* improper skip arg
* Add simple tests for introspection API
* Add test for custom introspector, improve base introspection test
* Add intermittent update timestamp
* Only allow daemon registration and introspection registration at initialization
* Add docs to manticore.py
* Add docs for plugin, add update_state_descriptor to EVM
* Fix renamed will_start_run --> will_run
* Docstrings for DaemonThread and EventSolver
* Docs for enums
* Improve pretty printer, add some mypy fixes
* Don't run daemon threads if run is called multiple times
* If at first you don't succeed, destroy all the evidence you tried.
* Test the pretty printer
* Add StateDescriptor to RTD
* Add newlines for RTD parsing
* Update to work with new state introspection API
* Add termination messages
* Also capture killed state messages
* Make info logs debug logs
* Apply suggestions from code review
Newlines for doc comments
Co-authored-by: Eric Kilmer <[email protected]>
* Add some type hints to manticore.py
* Add some type hints to plugin.py
* Fix type hint for get_state
* Add termination message from TUI PR
* Add example script
* Add docstrings to the example script
* Pass introspection plugin type as an argument
* Unskip property verifier tests
* Add mypy-requests type hints
* Remove itertools.tee
The problem with usign tee is that only the first callback to use the iterator can write to it. In `ready_states`, the `save_state` after the `yield` statement is ignored for all others.
* Make generator cloning a little bit more robust
Now Manticore will give up and return the original argument instead of blowing up if it can't clone the generator
* Clean up invalidated unit tests
We now fire `introspect` for the first time before we have any states
* Debug missing Truffle & Examples coverage
* Merge coverage from XML file
* Switch coverage to JSON, ignore debug logging and NotImplemented code
* Fix copy commands
* Move .coverage files directly
* Set examples to append coverage
* FLAG_NAME doesn't work the way we'd like
* Use plugin dict to store introspector
* Appease mypy
* Fix missing property on unique name
* Grab EVM PC
* Blacken
* Run black on all files if the git diff command fails
* Fix mypy errors
* Make plugin logging even less verbose
* Move log capture and state monitoring to daemon threads
* Use the config module for host & port
* Fix worker configuration and add test for TUI API
* Fix log messages breaking native tests
* Split up base Manticore tests and logging tests
The verbosity changes seem to be taking hold when they shouldn't
* Merge LogTCPHandler and MonitorTCPHandler
* Confirm that logging tests return to base level
* Fix mypy
* Switch back to using a deque for log buffering in the default case
* Fix deque API
* Update state_pb2.py
* Reformat programatically generated files
* Drop max verbosity in logging tests
Haven't been able to figure out why, but somehow other loggers get "stuck" at this high verbosity and the integration tests try to print out the values of every single register.
* Fix duplicated code from bad merge
* Remove is_main from state_monitor
* Add comment about log buffer size
* Remove vestigial is_main
* Blacken
Co-authored-by: Philip Wang <[email protected]>
Co-authored-by: Eric Kilmer <[email protected]>
* Update changelog and version numbers
* Update CHANGELOG.md
* Fix missing link to #1816
Definitely worth noting that we don't track gas anymore by default.
* Fix date
* Revert 1e23f7ab
Remove psutil as a dependency
* Bump version numb, update changelog
* Re-do updated model not available message
* Update for latest PRs
* Update dates
* Add ONE failing test
* Fix test
* Try fix get-related
* Snapshots
* CC
* Allow snapshooting just from main()
* A test
* Test and fix is_main
* CC
* review
* ismain vs is_running
* is main to is main script
* Remove unused member variable
* initial import manticore verifier
* Fix verifier installation
* clear ready states
* CC and smoke test
* CC
* Increase cov
* Move regression to other
* blkn
* blkn
* Move get related
* CC
* fix concolic
* lint
* DivByZero default to zero
* blkn
* Update manticore/core/smtlib/solver.py
Co-authored-by: Eric Kilmer <[email protected]>
* Update manticore/core/smtlib/constraints.py
Co-authored-by: Eric Kilmer <[email protected]>
* Update manticore/core/smtlib/constraints.py
Co-authored-by: Eric Kilmer <[email protected]>
* remove odd string
* lint
* mypy lint
* Update manticore/core/smtlib/visitors.py
Co-authored-by: Eric Kilmer <[email protected]>
* lint
* Add Docs
* blkn
* blkn
* Replace modulo with masks
* Blacken
* blkn
* fix mypy
* fix mypy
* Add tests for signed LT behavior
* New test
* Fix constant folding
* lint
* blkn
* lint
* Fixes...
* lint
* Unittesting power
* Permisive read_buffer
* Fix optimize
* Preserve precision in constant folding
* Strip left-in print debugging
* blkn
* remove wasm_sym temporarily
* Better simplification for constants
* Add json
* blkn
* Better commmandline args
* blkn
* fix wasm
* Fix
* better commanlining
* REmove get_related from the default path and fix arm test
* blkn
* fix related to tests
* blkn
* Fix bug in test and disable debug messages in Solver
* smtlib config to disable multiple check-sat in newer z3
* Disable log test and fix merging poc vs variable migration
* blkn
* Avoid exception in some callbacks
* can_rais at did_will
* yikes!
* blkn
* Yices found one more states in truffle
* Add a config constant to ignore symbolic balances
* Add a config constant to ignore symbolic balances 2
* Relax truffle state count
* Relax truffle state count 2
* Change cli name and test
* Clear redy states in verifier workspace
* Clear ready states in verifier workspace
* Relative ws path on verifier output. Clean CREATE testcase example
* Blkn
* Improve coverage and funcid solving
* lint
* Better testcase generation at verifier
* Better testcase generation at verifier and typos
* Better output and default re
* Test must revert
* Better output and default re
* move generate_testcase_ex to private method
Co-authored-by: Eric Kilmer <[email protected]>
Co-authored-by: Eric Hennenfent <[email protected]>
* Rework syscall invocation for proper behavior under typeguard
Previously, using Typeguard with Manticore would break several emulated
syscalls. With this commit, it does not.
Some background information:
When a syscall is made from an emulated binary, Manticore uses the
syscall number to look up the appropriate Python method that models that
syscall, and then uses Python introspection to massage arguments to that
syscall model function as deemed appropriate.
Previously, this mechanism used the deprecated `inspect.getfullargspec`
to determine the number of arguments to the model function, and whether
or not it takes varargs.
However, `inspect.getfullargspec` doesn't look through wrapper
functions; it looks only at exactly the function object it is given.
This is an issue, however, when trying to inspect _decorated_ functions.
(The use of a decorator in Python introduces a _new_ function object
that wraps the decorated item.)
How did that break Manticore when using Typeguard? It turns out that
When using Typeguard via the `--typeguard-packages=manticore` option to
`pytest`, the Typeguard plugin implicitly adds a `@typeguard.check_types`
decorator on _every_ function & method in the `manticore` package.
In this way, each syscall implementation function in Manticore ends up
with a wrapper around it, and the syscall invocation mechanism based on
`inspect.getfullargspec` would somewhat quietly cause syscalls to break.
Now, instead of using `inspect.getfullargspec`, Manticore's syscall
invocation mechansim uses the non-deprecated `inspect.signature` API to
get the needed information. This API _does_ look through wrapper
functions. Additionally, it allows us to get rid of some conditional
logic, about whether `self` appears in a function's parameter list or
not.
* Update manticore/native/cpu/abstractcpu.py
* Get rid of the `wrapt` library
We only were using `wrapt` in a single place -- to implement the
`unimplemented` syscall decorator. That dependency was added in #1384,
so that the old syscall mechanism could work with the decorator.
Now, with the rework of Manticore's syscall mechanism, this is no longer
necessary, and a "regular" Python decorator implemented using
`functools.wraps` should work just fine.
* Fix rewrite of `unimplemented`
* Rework platform `unimplemented` decorator, take 3
Additionally, add some extra tests for the decorator, to get better
coverage from where it is used.
* Clean up `test_sycalls.py` imports
* Fix an unbound variable error
(found with `mypy --check-untyped-defs` and some grep)
* Get rid of some unused imports
* Add some loads and stores
* Add global operations
* Move float methods further down
* Support internal function calls
```
18:20:42 (mc) ehennenfent@nessie:~/wasm ➜ cat loop_check.c
#include <stdio.h>
#include <stdbool.h>
bool check(int arg){
if (((arg << 2) ^ 16) == 36) {
return true;
}
else{
return false;
}
}
int main(){
printf("Hello world!\n");
int g = getchar();
printf("Got: %d\n", g);
int y = 0;
for (int i = 0; i < 20; i++){
y++;
if (y > 18){
return check(g) ? 0 : -1;
}
}
return -1;
}
18:20:54 (mc) ehennenfent@nessie:~/wasm ➜ manticore loop_check.wasm
65: i32.const (16)
11: end
65: i32.const (32)
11: end
Initialization Complete
65: i32.const (0)
65: i32.const (0)
40: i32.load (Offset 4)
65: i32.const (16)
107: i32.sub
34: tee_local (Local 1)
54: i32.store (Offset 4)
65: i32.const (32)
16: call (Func Idx 2)
Called stub function with args: (32,)
HostFunc returned [13]
26: drop
32: get_local (Local 1)
16: call (Func Idx 0)
Called stub function with args: ()
HostFunc returned [13]
34: tee_local (Local 0)
54: i32.store (Offset 0)
65: i32.const (16)
32: get_local (Local 1)
16: call (Func Idx 1)
Called stub function with args: (16, 16)
HostFunc returned [13]
26: drop
65: i32.const (0)
32: get_local (Local 1)
65: i32.const (16)
106: i32.add
54: i32.store (Offset 4)
65: i32.const (-1)
65: i32.const (0)
32: get_local (Local 0)
65: i32.const (1073741823)
113: i32.and
65: i32.const (13)
71: i32.ne
27: select
11: end
[0]
2019-07-25 18:20:58,138: [13842] m.c.manticore:INFO: Generated testcase No. 0 - test
2019-07-25 18:20:58,143: [13842] m.c.manticore:INFO: Results in /home/ehennenfent/wasm/mcore_1evptpip
```
* Improve pickle-friendliness
* Convert wasm module instructions into internal type s
* Fix class naming conflict
Thanks Pickle
* Support bitvecs as an integer type
* Refactor execution to handle single instructions at a time
* Add Atomic Stacks
It's a very inefficient implementation, but we'll make it better
* Handle Concretize Exceptions
Needs a lot of fine tuning but it's a start
* Propagate symbols all the way to return
* Add test generation infrastructure
Still very basic but it's a start
* Attempt to make Travis happy
* Fix missing generation step
* Ignore stack exhaustsion and invalid modules
* Add basic shift instructions
* Add lots of arithmetic instructions
* Fix global initialization
* Add optimization for fixed size left shift
* Make Travis tests less verbose
* Blacken
* Properly handle modulo in shifting
* Add several control flow instructions
* Blacken
* Partially Fix return instruction
* Fix missing return value from look_forward
* Update look_forward to handle nested blocks
* Fix type sigs and executor copy/paste errors, deduplicate test names
* Fix an instruction exit issue
* Fix local ordering
* Blacken
* test_address now passes for wasm
Fixed issue with json2mc.py issue with testname uniqueness
Started some work on floating point
* i hate git
* i hate git
* Adding linenumber to testcase function name
minor float progress
* f32_cmp passed for wasm
* f64_cmp passes for wasm
* f32_bitwise passes except for the problem with assertEqual not working with nan
* f64_bitwise passes except for the problem with assertEqual not working with nan
* Fixed the tests to handle [nan] comparisons
* better handling of nans and infinites
* added integer backing value for f64
* Blacken
* Blacken
* f64.nearest
* Fix argument ordering and implement N-bit memory stores
* Use subtests
* Switch print to logger
* Modify control flow instructions to match spec
"Continuation" != the thing that goes inside the Label
* Teporarily disable call instructions to prevent infinite loop
Still need to figure out control flow for nested function calls
* Blacken
* Fixed infinite loop in _call via block depth tracking
The behavior of `end` needs to depend on how many labels and activations are on the stack. Every time we push or pop a label, we increment or decrement the counter for the current call frame.
* Blacken
* Initial `br` implementation
Probably still bugged
* Support if blocks without an else instruction
* Add call_indirect
* Add grow_memory
* i32_wrap_i64
* Add several arithmetic operators
* Copy i32.ctz impl from x86 TZCNT
* Fix off-by-one in br
I think this is the correct fix (it makes all the _br tests pass) but I'm leaving the TODO there just in case.
* Blacken...
* Fix lingering issue with if block splitting
* Fix bad args to sextend in i64_extend_s_i32
* Fix indirect call null check and Float binop return types
* Improve br_table indexing
See comment
* Remove vestigial argument popping in invoke
* Fix looking forward in _return and _block
* Blacken
* Generate trap and action tests
* Fix extraneous call to exec_for_test
* Fix missing traps in load instructions
* Fix c1=0 case in ctz/clz
Hopefully this also fixes the fact that the original CTZ implementations seems to have been intended for little-endian architectures
* Handle traps in-stride during test execution
* Treat zero division as trap
* Use ctypes to convert integers
* Blacken
* Apparently I was wrong about the CTZ endianness thing
* Better range checks for offsets
This really hammers home why checking whether ea + size is _greater than_ len(mem.data) is strange if mem.data is 0-indexed, but idk, that's what the spec says.
* Make stub functions return the correct number of results
* ctypes can't handle floats
* Add option to run the start function
* Correctly handle functions with unacceptable names
* Blacken
* Treat NotImplemented as a Skip
Also counts individual subtests directly instead of screwing around with trying to grep for them in the log.
* Fix unhandled negative indices
* Fixed some integer instruction bugs
i64/32:
div_s/u
shr_s/u
shl
clz
* Fix global and memory imports
* CC
* Make default loglevel debug
* Rework conversion from int
* Make float_store work
* _actually_ fix float_store
* Restore old float creation method
* Blacken
* Add range checks to float load/store
* Force cast to unsigned in _u instructions
* Fix yet another I32 -> I64 copy paste error
How many times do we have to teach you this lesson, ~old~ young man?
* Skip problematic float tests for now
* Handle floating point inaccuracies when dividing large ints
* Skip conversion and endianness tests
* Rotl, rotl, popcnt
* Blacken
* Add some conversion stubs
* Maybe fix conversions
* Improve reinterpret and float demotion
* Re-enable endianness tests
* Include WASM logging in verbosity controls
* Propagate constraints to executor
* Blacken
* Add prototype symbolic tests
* Add api for invocation to Manticore object
* Temporarily disable native/evm tests
There's no reason to tie up Travis for 40 minutes every time I push a commit.
* Fix wasmworld import
* Concretize br_if
* Pass arg generator to invoke
* Download updated WABT
It's fixed now!
* Generate Symbolic tests (partial)
* Follow m.run format
* Blacken
* Reinitialize Manticore every 50 tests
This should cut down on the performance issues we've been seeing
* Revert "Temporarily disable native/evm tests"
This reverts commit 7abd6c1333.
* I32 and I64 pass Symbolic
* Create new MC on reinit
* Blacken
* Fix return handling in test generation
* Concretize operand in `if_`
* Concretize memory addresses before access
In the long run, we should have a symbolic memory model
* Better memory access concretization (and br_table)
* Concretize float converions and indirect call operands
* Run start method
* Handle size errors in wrap+i64 and select
* Add timeout (and skip extra-problematic tests)
* Blacken
* Aggressively concretize floats
* Recast ints that should have been floats
* Better trap handling
* Unstash float concretization
* Fix log-lived 'trap' attr
* Blacken
* Concretize F32 when promoting to F64
_shouldn't_ strictly be necesssary, but the lazy conversion means that one of the tests fails. Until we get symbolic floating point, this will have to do.
* Docstrings for platforms/wasm.py
* Docstrings for ManticoreWASM
* Add docs for the module structure
* Add WASM to RTD
* missing docstring for Module itself
* Fix sphinx errors
* Bump RTD python version
I'm sorry and I swear I'll fix it later
* Reformat docstrings for sphinx
* Pacify RTD by adding newlines before params
* Missed a few
* Document the types
* Document executor
* A few more type docstrings
* Partial runtime_structure docs
* Finish documenting runtime structure
* Fix trailing whitespace
* Add collatz conjecture example
* Include undocumented members in RTD
* Add a bunch of events
* Appropriately connect event publishers
* Generate useful testcases
* Blacken
* Attempt at 3.6 backporting
* Optimize AtomicStack
* Sort solve_n output in test_solve_n
Very unclear why this is happening, but it's breaking a travis build
https://travis-ci.com/trailofbits/manticore/jobs/246451527
* Roll back to trusty?
Unsure if this is what's causing the EVM issues
* Create custom trap types
* Lazily evaluate overflows
May need to revert this if it slows down the Travis tests. It's 0.7% slower in the example I'm looking at, but that may not carry over.
* Blacken
* Address Dominik's feedback, remove stale TODOs
* Expand & Test Examples
* Fix relative path
* Fix relative path?
* capitalize Path
* Fix relative path for real
* Blacken...
* Test finalize
* Add supplemental tests for coverage
* Use dedicated helper for store access
* Handle quoted strings in imported module names
* Allow WASMWorld to hold multiple modules
Necessary for getting imports/exports to work, but still needs some changes before it passes the tests.
* Actually use export maps
* Fix most of the control flow issues with import tests
* Use available cores for test generation
* Escape module name
* Don't reinit modules that have already been initialized
* Use addresses for imports instead of copying
* Fix removed timeout argument
* Interleave registration and aliases
* Fix broken "None" check for address
* Blacken
* Make manual exports work with address syntax
* Import all registered modules
Only necessary for elem, but it's tricky to make changes just for one test, so *shrug emoji*
* Remove stringified annotations
Turns out those only work in the same file...
* Fix easy type issues in all but runtime structure
* Fix easy type issues in runtime_structure
* Fix* everything but imports
* Blacken
* Blacken 2
* Typo
* Merge module structure and runtime structure
* Fix instance vs type confusion
* Fix mypy and concrete tests
Still an issue with the symbolic tests
* Still generate symbolic tests though
I keep forgetting not to commit that file...
* Attempt to fix env for symbolic tests
* Blacken/mypy
* Fix missing supplemental environment variables arg
* Fix docstrings for new import style
* CC
* Bump timeout duration to 70 minutes
I don't like it, but adding more jobs to Travis isn't going to make our builds any faster
* Fix pycharm type, import, and docstring errors
* Explain type: ignore
* Missed two type:ignore's
* Fix CLI so it doesn't throw up on arbitrary modules
* Add getters for locals and globals to state
* Make host functions take the state instead of the constriant set
* Support __getattr__ style function calling
* Implicitly run Manticore when invoked via __getattr__
* Automatically reinitialize before __getattr__ style function calls
* Fix docstring
* Fix mypy?
* Fix test getchar impl
* Add tests for implicit call and main function
* Fix wrong memory callback
* Okay fine I'll check the output
* Add missing jinja dep
Sphinx includes it, but we should do so as well
* Bump spec tests version
* Test multiple implicit calls
* Split symbolic/concrete WASM tests
* Add some missing docstrings
* Hopefully fix wasm test split?
* Properly split symbolic tests
* Fix fallthrough
Okay I swear it's fixed this time
* Add if_check example
* Blacken
* Add WASM Example to README
Co-authored-by: HaxMeadroom <[email protected]>
The WASM branch is ready to be included in the rest of Manticore. It doesn't include several advanced features, including a symbolic memory model, symbolic floating point support, or a standard library implementation (WASI/emscripten), but it does have full symbolic semantics for all of the integer instructions.
### Commit log:
* Empty WASM Module
Will run on a .wasm file and do nothing forever
* Read and parse sections
/shrug
* Can partially allocate modules
Just pushing for the weekend
* Blacken
* Can allocate with exports now
* Static initialization
Doesn't evaluate any of the starting expressions, but that's okay
* Make type hints a little more pycharm-friendly
* First part of invocation
* Blacken...
* Barebones execution poc
Can execute the following snippet:
```
main(){
return 42;
}
```
* blacken
* Add some loads and stores
* Add global operations
* Move float methods further down
* Support internal function calls
```
18:20:42 (mc) ehennenfent@nessie:~/wasm ➜ cat loop_check.c
#include <stdio.h>
#include <stdbool.h>
bool check(int arg){
if (((arg << 2) ^ 16) == 36) {
return true;
}
else{
return false;
}
}
int main(){
printf("Hello world!\n");
int g = getchar();
printf("Got: %d\n", g);
int y = 0;
for (int i = 0; i < 20; i++){
y++;
if (y > 18){
return check(g) ? 0 : -1;
}
}
return -1;
}
18:20:54 (mc) ehennenfent@nessie:~/wasm ➜ manticore loop_check.wasm
65: i32.const (16)
11: end
65: i32.const (32)
11: end
Initialization Complete
65: i32.const (0)
65: i32.const (0)
40: i32.load (Offset 4)
65: i32.const (16)
107: i32.sub
34: tee_local (Local 1)
54: i32.store (Offset 4)
65: i32.const (32)
16: call (Func Idx 2)
Called stub function with args: (32,)
HostFunc returned [13]
26: drop
32: get_local (Local 1)
16: call (Func Idx 0)
Called stub function with args: ()
HostFunc returned [13]
34: tee_local (Local 0)
54: i32.store (Offset 0)
65: i32.const (16)
32: get_local (Local 1)
16: call (Func Idx 1)
Called stub function with args: (16, 16)
HostFunc returned [13]
26: drop
65: i32.const (0)
32: get_local (Local 1)
65: i32.const (16)
106: i32.add
54: i32.store (Offset 4)
65: i32.const (-1)
65: i32.const (0)
32: get_local (Local 0)
65: i32.const (1073741823)
113: i32.and
65: i32.const (13)
71: i32.ne
27: select
11: end
[0]
2019-07-25 18:20:58,138: [13842] m.c.manticore:INFO: Generated testcase No. 0 - test
2019-07-25 18:20:58,143: [13842] m.c.manticore:INFO: Results in /home/ehennenfent/wasm/mcore_1evptpip
```
* Improve pickle-friendliness
* Convert wasm module instructions into internal type s
* Fix class naming conflict
Thanks Pickle
* Support bitvecs as an integer type
* Refactor execution to handle single instructions at a time
* Add Atomic Stacks
It's a very inefficient implementation, but we'll make it better
* Handle Concretize Exceptions
Needs a lot of fine tuning but it's a start
* Propagate symbols all the way to return
* Add test generation infrastructure
Still very basic but it's a start
* Attempt to make Travis happy
* Fix missing generation step
* Ignore stack exhaustsion and invalid modules
* Add basic shift instructions
* Add lots of arithmetic instructions
* Fix global initialization
* Add optimization for fixed size left shift
* Make Travis tests less verbose
* Blacken
* Properly handle modulo in shifting
* Add several control flow instructions
* Blacken
* Partially Fix return instruction
* Fix missing return value from look_forward
* Update look_forward to handle nested blocks
* Fix type sigs and executor copy/paste errors, deduplicate test names
* Fix an instruction exit issue
* Fix local ordering
* Blacken
* test_address now passes for wasm
Fixed issue with json2mc.py issue with testname uniqueness
Started some work on floating point
* i hate git
* i hate git
* Adding linenumber to testcase function name
minor float progress
* f32_cmp passed for wasm
* f64_cmp passes for wasm
* f32_bitwise passes except for the problem with assertEqual not working with nan
* f64_bitwise passes except for the problem with assertEqual not working with nan
* Fixed the tests to handle [nan] comparisons
* better handling of nans and infinites
* added integer backing value for f64
* Blacken
* Blacken
* f64.nearest
* Fix argument ordering and implement N-bit memory stores
* Use subtests
* Switch print to logger
* Modify control flow instructions to match spec
"Continuation" != the thing that goes inside the Label
* Teporarily disable call instructions to prevent infinite loop
Still need to figure out control flow for nested function calls
* Blacken
* Fixed infinite loop in _call via block depth tracking
The behavior of `end` needs to depend on how many labels and activations are on the stack. Every time we push or pop a label, we increment or decrement the counter for the current call frame.
* Blacken
* Initial `br` implementation
Probably still bugged
* Support if blocks without an else instruction
* Add call_indirect
* Add grow_memory
* i32_wrap_i64
* Add several arithmetic operators
* Copy i32.ctz impl from x86 TZCNT
* Fix off-by-one in br
I think this is the correct fix (it makes all the _br tests pass) but I'm leaving the TODO there just in case.
* Blacken...
* Fix lingering issue with if block splitting
* Fix bad args to sextend in i64_extend_s_i32
* Fix indirect call null check and Float binop return types
* Improve br_table indexing
See comment
* Remove vestigial argument popping in invoke
* Fix looking forward in _return and _block
* Blacken
* Generate trap and action tests
* Fix extraneous call to exec_for_test
* Fix missing traps in load instructions
* Fix c1=0 case in ctz/clz
Hopefully this also fixes the fact that the original CTZ implementations seems to have been intended for little-endian architectures
* Handle traps in-stride during test execution
* Treat zero division as trap
* Use ctypes to convert integers
* Blacken
* Apparently I was wrong about the CTZ endianness thing
* Better range checks for offsets
This really hammers home why checking whether ea + size is _greater than_ len(mem.data) is strange if mem.data is 0-indexed, but idk, that's what the spec says.
* Make stub functions return the correct number of results
* ctypes can't handle floats
* Add option to run the start function
* Correctly handle functions with unacceptable names
* Blacken
* Treat NotImplemented as a Skip
Also counts individual subtests directly instead of screwing around with trying to grep for them in the log.
* Fix unhandled negative indices
* Fixed some integer instruction bugs
i64/32:
div_s/u
shr_s/u
shl
clz
* Fix global and memory imports
* CC
* Make default loglevel debug
* Rework conversion from int
* Make float_store work
* _actually_ fix float_store
* Restore old float creation method
* Blacken
* Add range checks to float load/store
* Force cast to unsigned in _u instructions
* Fix yet another I32 -> I64 copy paste error
How many times do we have to teach you this lesson, ~old~ young man?
* Skip problematic float tests for now
* Handle floating point inaccuracies when dividing large ints
* Skip conversion and endianness tests
* Rotl, rotl, popcnt
* Blacken
* Add some conversion stubs
* Maybe fix conversions
* Improve reinterpret and float demotion
* Re-enable endianness tests
* Include WASM logging in verbosity controls
* Propagate constraints to executor
* Blacken
* Add prototype symbolic tests
* Add api for invocation to Manticore object
* Temporarily disable native/evm tests
There's no reason to tie up Travis for 40 minutes every time I push a commit.
* Fix wasmworld import
* Concretize br_if
* Pass arg generator to invoke
* Download updated WABT
It's fixed now!
* Generate Symbolic tests (partial)
* Follow m.run format
* Blacken
* Reinitialize Manticore every 50 tests
This should cut down on the performance issues we've been seeing
* Revert "Temporarily disable native/evm tests"
This reverts commit 7abd6c1333.
* I32 and I64 pass Symbolic
* Create new MC on reinit
* Blacken
* Fix return handling in test generation
* Concretize operand in `if_`
* Concretize memory addresses before access
In the long run, we should have a symbolic memory model
* Better memory access concretization (and br_table)
* Concretize float converions and indirect call operands
* Run start method
* Handle size errors in wrap+i64 and select
* Add timeout (and skip extra-problematic tests)
* Blacken
* Aggressively concretize floats
* Recast ints that should have been floats
* Better trap handling
* Unstash float concretization
* Fix log-lived 'trap' attr
* Blacken
* Concretize F32 when promoting to F64
_shouldn't_ strictly be necesssary, but the lazy conversion means that one of the tests fails. Until we get symbolic floating point, this will have to do.
* Docstrings for platforms/wasm.py
* Docstrings for ManticoreWASM
* Add docs for the module structure
* Add WASM to RTD
* missing docstring for Module itself
* Fix sphinx errors
* Bump RTD python version
I'm sorry and I swear I'll fix it later
* Reformat docstrings for sphinx
* Pacify RTD by adding newlines before params
* Missed a few
* Document the types
* Document executor
* A few more type docstrings
* Partial runtime_structure docs
* Finish documenting runtime structure
* Fix trailing whitespace
* Add collatz conjecture example
* Include undocumented members in RTD
* Add a bunch of events
* Appropriately connect event publishers
* Generate useful testcases
* Blacken
* Attempt at 3.6 backporting
* Optimize AtomicStack
* Sort solve_n output in test_solve_n
Very unclear why this is happening, but it's breaking a travis build
https://travis-ci.com/trailofbits/manticore/jobs/246451527
* Roll back to trusty?
Unsure if this is what's causing the EVM issues
* Create custom trap types
* Lazily evaluate overflows
May need to revert this if it slows down the Travis tests. It's 0.7% slower in the example I'm looking at, but that may not carry over.
* Blacken
* Address Dominik's feedback, remove stale TODOs
* Expand & Test Examples
* Fix relative path
* Fix relative path?
* capitalize Path
* Fix relative path for real
* Blacken...
* Test finalize
* Add supplemental tests for coverage
* Use dedicated helper for store access
* Handle quoted strings in imported module names
* Allow WASMWorld to hold multiple modules
Necessary for getting imports/exports to work, but still needs some changes before it passes the tests.
* Actually use export maps
* Fix most of the control flow issues with import tests
* Use available cores for test generation
* Escape module name
* Don't reinit modules that have already been initialized
* Use addresses for imports instead of copying
* Fix removed timeout argument
* Interleave registration and aliases
* Fix broken "None" check for address
* Blacken
* Make manual exports work with address syntax
* Import all registered modules
Only necessary for elem, but it's tricky to make changes just for one test, so *shrug emoji*
* Remove stringified annotations
Turns out those only work in the same file...
* Fix easy type issues in all but runtime structure
* Fix easy type issues in runtime_structure
* Fix* everything but imports
* Blacken
* Blacken 2
* Typo
* Merge module structure and runtime structure
* Fix instance vs type confusion
* Fix mypy and concrete tests
Still an issue with the symbolic tests
* Still generate symbolic tests though
I keep forgetting not to commit that file...
* Attempt to fix env for symbolic tests
* Blacken/mypy
* Fix missing supplemental environment variables arg
* Fix docstrings for new import style
* CC
* Bump timeout duration to 70 minutes
I don't like it, but adding more jobs to Travis isn't going to make our builds any faster
* Fix pycharm type, import, and docstring errors
* Explain type: ignore
* Missed two type:ignore's
* Fix CLI so it doesn't throw up on arbitrary modules
* linux: Add return 0 for success on sys_pipe2
* mypy: Initial config file to ignore modules without typing info
* mypy: types manticore/platforms/linux.py
* mypy: manticore/ethereum/manticore.py
* Ignore error because mypy doesn't support decorated @property's
* mypy: manticore/ethereum/solidity.py
* Ignore error because mypy doesn't support decorated @property's
* mypy: manticore/ethereum/detectors.py
* Use Python's Enum class for DetectorClassification
* mypy: manticore/platforms/cgcrandom.py
* Set as byte string
* mypy: manticore/binary/binary.py
* mypy: manticore/utils/event.py
* mypy: manticore/ethereum/abitypes
* Use actual ResourceWarning class instead of string name
* mypy: manticore/utils/config.py
* mypy: manticore/core/workspace.py
* Support both Python 3.6 and Python 3.7 with nullcontext. Ignore the
mypy error because it checks against Python 3.6 and Manticore does not
explicitly claim support for Python 3.7
* mypy: manticore/core/manticore
* Ignore mypy error on sync, at_running, at_not_running:
Self argument missing for a non-static method (or an invalid type for
self)
* Ignore type info for unsupported decorators on @property fields
* mypy: manticore/platforms/evm.py
* Ignore error of already defined variable when setting member variable
of class transaction
* Remove redundant get_nonce call
* Move property setter next to property definition
* mypy: manticore/core/smtlib/solver.py
* Fix instance where no match was found in regex
* mypy: Add mypy to travis and setup.py
* Lock to specific mypy and black tool versions
* Rename "format" stage to "lint"
* Add text about mypy to CONTRIBUTING.md
* Revert black tool upgrade to 19.3b0
* Fix typing issues introduced by SHA3 branch
* Manticore 0.3.0
* Explicitly note lack of Constantinople support
Will hopefully head-off people with issues regarding unimplemented instructions and incorrect behavior
* Add blurb about solver
* Link to state machine diagram
* Fix typo
* Fix duplicated section
And fix date
* Add black to CI tests
I'd rather have blackening be an optional check, but this will work for now
* Yep, messed up Travis
Again, sorry about the commit spam
* Bump accepted line length to 120
Going to auto-format at 100 but will allow 20% overruns
* Blacken
* Cap line length at 100
rip my undisciplined coding habits
* update black on the fly if necessary
* Ignore CC errors
* Only check modified files
* Only python files
* Re-blacken post merge
* Re-Blacken
* Fix deleted files
* Explicitly invoke python
The +x must not have gotten commited
* Forgot to blacken again
Hoisted on my own petard
* Re-blacken
As noted, we should roll this back whenever Unicorn pushes to PyPi
* Remove write to Unicorn x86 FS register
In latest version of unicorn (778171fc9546c1fc3d1341ff1151eab379848ea0)
a write to FS will cause segfault or hang if we try to write to it.
Issue #1434
* Update unicorn dependency to latest commit on master
Latest Unicorn master commit is 778171fc9546c1fc3d1341ff1151eab379848ea0
Issue #1434
* Move unimplemented syscall logging to decorator
Adds [wrapt module](https://wrapt.readthedocs.io/en/latest/) from pypi
* Move unimplemented calls further down
* Add stubs for unimplemented system calls
* Move most stubs back to stub file
A few have been left in linux.py because they're necessary for other projects
* Codeclimate
* Fix discrepancies in syscall ABI
* Removed extraneous void argument for getpid and gettid
* Added missing sigsetsize parameter for rt_sigaction
* Removed incorrect 'flags' parameter on accept (should be on accept4
* Fixed 'SocketDesc' has no attribute 'close' error
* (Partially) Fixed duplicate constraint error with socket file descriptors
* Add gettimeofday and directory handling
* Hacky pipe implementation
Pretend that sockets are pipes and pipes are sockets
* Better support for MSR writing
* Default to success on directory operations
* Codeclimate
* More verbose unimplemented syscall printing
* Fix line overrun before CC calls me out
* Add ftruncate support
* Add link, unlink, and nanosleep
* Add getdents
* Add tests for mkdir and time functions
* Fix codeclimate and force rebuild
* Add pre and post hooks for syscalls
* Fix lingering single quotes
* Handle dumb invocations
* Type-o
* Make emulator follow logging behavior
* CC
* Fix pipe tests
* Fix shadowed name
* Add test for ftruncate
* Add link tests
* Add chmod test
* Rewrite docstrings
Best not to risk licensing issues
* Replace with simple returns
* Strip out unique implementations
Should bring the line count down a bit
* Check that unimplemented syscalls exhibit the proper behavior
* Fix missing keyword 'solver'
* Pass initial states to will_run callback
* Restore debug printouts after run
* Fix existing `init` calls
* Restore old exception behavior
* Move save_run_data to manticore native
* CC
* Refactor finalization call
* Fix save_run_data
I think it makes sense now
* Adjust expected output line counts
* CC [ci skip]
* Rename terminate state on exception to kill state
There's gotta be a better pun about 'overthrowing the state' in there somewhere
* Move time started to manticore_native
Eric Kilmer