Having Dir is Type is handy, but forces us to duplicate lots of types.
E.g. if a struct is referenced as both in and out, then we need to
have 2 copies and 2 copies of structs/types it includes.
If also prevents us from having the struct type as struct identity
(because we can have up to 3 of them).
Revert to the old way we used to do it: propagate Dir as we walk
syscall arguments. This moves lots of dir passing from pkg/compiler
to prog package.
Now Arg contains the dir, so once we build the tree, we can use dirs
as before.
Reduces size of sys/linux/gen/amd64.go from 6058336 to 5661150 (-6.6%).
Update #1580
We will need a wrapper for target.SanitizeCall that will do more
than just calling the target-provided function. To avoid confusion
and potential mistakes, give the target function and prog function
different names. Prog package will continue to call this "sanitize",
which will include target's "neutralize" + more.
Also refactor API a bit: we need a helper function that sanitizes
the whole program because that's needed most of the time.
Fixes#477Fixes#502
Strings with enumerated values are frequently file names
or have complete enumeration of relevant values.
Mutating complete enumeration if not very profitable.
Mutating file names leads to escaping paths and
fuzzer messing with things it is not supposed to mess with as in:
r0 = openat$apparmor_task_exec(0xffffffffffffff9c, &(0x7f0000000440)='/proc/self//exe\x00', 0x3, 0x0)
Non-determinism is bad:
- it leads to flaky coverage reports
- it makes test failures non-reproducible
Remove 4 sources of non-determinism related to maps:
- file name generation
- string generation
- resource generation
- hints generation
All a test that ensures all main operations are fully deterministic.
Hints mutation could produce unsanitized calls.
Sanitize calls after hints mutation.
Also sanitize on load (in validate), because bad programs
can already be in corpuses. And it's just the right thing
to do because sanitization rules can change over time.
Use big-endian match/replace for both blobs and ints.
Sometimes we have unmarked blobs (no little/big-endian info);
for ANYBLOBs we intentionally lose all marking;
but even for marked ints we may need this too.
Consider that kernel code does not convert the data
(i.e. not ntohs(pkt->proto) == ETH_P_BATMAN),
but instead converts the constant (i.e. pkt->proto == htons(ETH_P_BATMAN)).
In such case we will see dynamic operand that does not
match what we have in the program.
Squash complex structs into flat byte array and mutate this array
with generic blob mutations. This allows to mutate what we currently
consider as paddings and add/remove paddings from structs, etc.
1. mmap all memory always, without explicit mmap calls in the program.
This makes lots of things much easier and removes lots of code.
Makes mmap not a special syscall and allows to fuzz without mmap enabled.
2. Change address assignment algorithm.
Current algorithm allocates unmapped addresses too frequently
and allows collisions between arguments of a single syscall.
The new algorithm analyzes actual allocations in the program
and places new arguments at unused locations.
Make Foreach* callback accept the arg and a context struct
that can contain lots of aux info.
This (1) removes lots of unuser base/parent args,
(2) provides foundation for stopping recursion,
(3) allows to merge foreachSubargOffset.
Fixes#188
We now will write just ""/1000 to denote a 1000-byte output buffer.
Also we now don't store 1000-byte buffer in memory just to denote size.
Old format is still parsed.
Clone program only once.
Preallocate slices in clone.
Remove the clone full mode.
Always mutate args in place.
Allocate replacers map lazily.
Don't allocate res map at all (calculate valus on the go).
Remove sliceToUint64, pad.
benchmark old ns/op new ns/op delta
BenchmarkHints 122100048 7466013 -93.89%
The race initially showed up on the new benchmark (see race report below).
The race indicated a wrong call passed to replaceArg,
as the result we sanitized the wrong call and left the new call un-sanitized.
Fix this.
Add test that exposes this.
Run benchmarks in race mode during presubmit
(benchmarks have higher chances of uncovering races than tests).
WARNING: DATA RACE
Write at 0x00c42000d3f0 by goroutine 18:
github.com/google/syzkaller/sys/linux.(*arch).sanitizeCall()
sys/linux/init.go:155 +0x256
github.com/google/syzkaller/sys/linux.(*arch).(github.com/google/syzkaller/sys/linux.sanitizeCall)-fm()
sys/linux/init.go:42 +0x4b
github.com/google/syzkaller/prog.(*Prog).replaceArg()
prog/prog.go:357 +0x239
github.com/google/syzkaller/prog.generateHints.func2()
prog/hints.go:105 +0x124
github.com/google/syzkaller/prog.checkConstArg()
prog/hints.go:128 +0xf3
github.com/google/syzkaller/prog.generateHints()
prog/hints.go:120 +0x495
github.com/google/syzkaller/prog.(*Prog).MutateWithHints.func1()
prog/hints.go:72 +0x67
github.com/google/syzkaller/prog.foreachSubargImpl.func1()
prog/analysis.go:86 +0x9f
github.com/google/syzkaller/prog.foreachSubargImpl()
prog/analysis.go:104 +0xc8
github.com/google/syzkaller/prog.foreachArgArray()
prog/analysis.go:113 +0x89
github.com/google/syzkaller/prog.foreachArg()
prog/analysis.go:121 +0x50
github.com/google/syzkaller/prog.(*Prog).MutateWithHints()
prog/hints.go:71 +0x18e
github.com/google/syzkaller/prog.BenchmarkHints.func1()
prog/hints_test.go:477 +0x77
testing.(*B).RunParallel.func1()
testing/benchmark.go:626 +0x156
Previous read at 0x00c42000d3f0 by goroutine 17:
github.com/google/syzkaller/prog.clone()
prog/clone.go:38 +0xbaa
github.com/google/syzkaller/prog.(*Prog).cloneImpl()
prog/clone.go:21 +0x17f
github.com/google/syzkaller/prog.generateHints()
prog/hints.go:95 +0xd0
github.com/google/syzkaller/prog.(*Prog).MutateWithHints.func1()
prog/hints.go:72 +0x67
github.com/google/syzkaller/prog.foreachSubargImpl.func1()
prog/analysis.go:86 +0x9f
github.com/google/syzkaller/prog.foreachSubargImpl()
prog/analysis.go:104 +0xc8
github.com/google/syzkaller/prog.foreachArgArray()
prog/analysis.go:113 +0x89
github.com/google/syzkaller/prog.foreachArg()
prog/analysis.go:121 +0x50
github.com/google/syzkaller/prog.(*Prog).MutateWithHints()
prog/hints.go:71 +0x18e
github.com/google/syzkaller/prog.BenchmarkHints.func1()
prog/hints_test.go:477 +0x77
testing.(*B).RunParallel.func1()
testing/benchmark.go:626 +0x156
Hints for data args don't work.
We do all the work, but at the final stage we patch
arg in the _old_ program, not in the _new_ one.
So programs passed to the callback are all the same
and don't contain any mutations.
Tests did not catch this because they work right before that point
(don't test the actual interface function MutateWithHints).
Fix that and add a test that catches this.
During smashing we know what call gave new coverage,
so we can concentrate just on it.
This helps to reduce amount of hints generated (we have too many of them).
Now each prog function accepts the desired target explicitly.
No global, implicit state involved.
This is much cleaner and allows cross-OS/arch testing, etc.
Large overhaul moves syscalls and arg types from sys to prog.
Sys package now depends on prog and contains only generated
descriptions of syscalls.
Introduce prog.Target type that encapsulates all targer properties,
like syscall list, ptr/page size, etc. Also moves OS-dependent pieces
like mmap call generation from prog to sys.
Update #191
A hint is basically a tuple consisting of a pointer to an argument
in one of the syscalls of a program and a value, which should be
assigned to that argument.
A simplified version of hints workflow looks like this:
1. Fuzzer launches a program and collects all the comparisons' data
for every syscall in the program.
2. Next it tries to match the obtained comparison operands' values
vs. the input arguments' values.
3. For every such match the fuzzer mutates the program by
replacing the pointed argument with the saved value.
4. If a valid program is obtained, then fuzzer launches it and
checks if new coverage is obtained.
This commit includes:
1. All the code related to hints generation, parsing and mutations.
2. Fuzzer functions to launch the process.
3. Some new stats gathered by fuzzer and manager, related to hints.
4. An updated version of execprog to test the hints process.
