gometalinter says:
pkg/compiler/consts.go:192:⚠️ internal error: no range for "n" (vetshadow)
pkg/compiler/consts.go:197:⚠️ internal error: no range for "n" (vetshadow)
prog/encoding.go:862:⚠️ declaration of "v" shadows declaration at prog/encoding.go:852 (vetshadow)
This somehow happens only with Go1.11 but not 1.12 so wasn't detected locally.
The prog warnings looks legit.
The pkg/compiler warning was amusingly introduced to please golangci-lint checker,
revert that fix for now.
C's \xHH hex constants in strings accept any number of hex digits
(not just 2 or 4). So later non-hex escaped chars glue to the \x construct.
Use \OOO instead as it accepts at most 3 octal digits.
AUTO arguments can be used for:
- consts
- lens
- pointers
For const's and len's AUTO is replaced with the natural value,
addresses for AUTO pointers are allocated linearly.
This greatly simplifies writing test programs by hand
as most of the time we want these natural values.
Update tests to use AUTO.
Add bulk of checks for strict parsing mode.
Probably not complete, but we can extend then in future as needed.
Turns out we can't easily use it for serialized programs
as they omit default args and during deserialization it looks like missing args.
Over time we relaxed parsing to handle all kinds of invalid programs
(excessive/missing args, wrong types, etc).
This is useful when reading old programs from corpus.
But this is harmful for e.g. reading test inputs as they can become arbitrary outdated.
For runtests which creates additional problem of executing not
what is actually written in the test (or at least what author meant).
Add strict parsing mode that does not tolerate any errors.
For now it just checks excessive syscall arguments.
Move target and vars into parser and make all
parsing functions methods of the parser.
This reduces number of args that we need to pass around
and eases adding more state that needs to be passed around.
Currently we only generate either valid user-space pointers or NULL.
Extend NULL to a set of special pointers that we will use in programs.
All targets now contain 3 special values:
- NULL
- 0xfffffffffffffff (invalid kernel pointer)
- 0x999999999999999 (non-canonical address)
Each target can add additional special pointers on top of this.
Also generate NULL/special pointers for non-opt ptr's.
This restriction was always too restrictive. We may want to generate
them with very low probability, but we do want to generate them.
Also change pointers to NULL/special during mutation
(but still not in the opposite direction).
Handle most of type changes, e.g. const is changed to struct,
or struct to pointers. In all these cases we create default args.
They may not give the coverage anymore, but still better than
losing them right away.
We now always create net namespace for testing,
so socket ports and other IDs do not overlap between
different test processes.
Proc types play badly with squashing packets to ANYBLOB.
To squash into a block we need concrete value, but it depends
on process id.
Removing proc also makes tun setup and address descriptions simpler.
Squash complex structs into flat byte array and mutate this array
with generic blob mutations. This allows to mutate what we currently
consider as paddings and add/remove paddings from structs, etc.
1. mmap all memory always, without explicit mmap calls in the program.
This makes lots of things much easier and removes lots of code.
Makes mmap not a special syscall and allows to fuzz without mmap enabled.
2. Change address assignment algorithm.
Current algorithm allocates unmapped addresses too frequently
and allows collisions between arguments of a single syscall.
The new algorithm analyzes actual allocations in the program
and places new arguments at unused locations.
This reduces size of a corpus in half.
We store corpus on manager and on hub,
so this will reduce their memory consumption.
But also makes large programs more readable.
Introduce isUsed(arg) helper, use it in several places.
Move method definitions closer to their types.
Simplify presence check for ArgUsed.Used() in several places.
Fixes#188
We now will write just ""/1000 to denote a 1000-byte output buffer.
Also we now don't store 1000-byte buffer in memory just to denote size.
Old format is still parsed.
Fixes#460
File names, crypto algorithm names, etc in programs are completely unreadable:
bind$alg(r0, &(0x7f0000408000)={0x26, "6861736800000000000000000000",
0x0, 0x0, "6d6435000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000
00000000000"}, 0x58)
Introduce another format for printable strings.
New args are denoted by '' ("" for old args).
New format is enabled for printable chars, \x00
and \t, \r, \n.
Example:
`serialize(&(0x7f0000408000)={"6861736800000000000000000000", "4849000000"})`,
vs:
`serialize(&(0x7f0000408000)={'hash\x00', 'HI\x00'})`,
For string[N] we successfully deserialize a string of any length.
Similarly for a fixed-size array[T, N] we successfully deserialize
an array of any size.
Such programs later crash in foreachSubargOffset because static size
Type.Size() does not match what we've calculated iterating over fields.
The crash happens only in SerializeForExec in syz-fuzzer,
which is especially bad.
Fix this from both sides:
1. Validate sizes of arrays/buffers in Validate.
2. Repair incorrect sizes in Deserialize.
Now each prog function accepts the desired target explicitly.
No global, implicit state involved.
This is much cleaner and allows cross-OS/arch testing, etc.
Large overhaul moves syscalls and arg types from sys to prog.
Sys package now depends on prog and contains only generated
descriptions of syscalls.
Introduce prog.Target type that encapsulates all targer properties,
like syscall list, ptr/page size, etc. Also moves OS-dependent pieces
like mmap call generation from prog to sys.
Update #191
We currently use uintptr for all values.
This won't work for 32-bit archs.
Moreover in some cases we use uintptr but assume
that it is always 64-bits (e.g. in encodingexec).
Switch everything to uint64.
Update #324
After a change in syscall description the number of syscall arguments
might change and some of the programs in corpus get invalidated.
This change makes syzkaller to generate missing arguments when decoding a
program as an attempt to fix and keep more programs from corpus.
Right now Arg is a huge struct (160 bytes), which has many different fields
used for different arg kinds. Since most of the args we see in a typical
corpus are ArgConst, this results in a significant memory overuse.
This change:
- makes Arg an interface instead of a struct
- adds a SomethingArg struct for each arg kind we have
- converts all *Arg pointers into just Arg, since interface variable by
itself contains a pointer to the actual data
- removes ArgPageSize, now ConstArg is used instead
- consolidates correspondence between arg kinds and types, see comments
before each SomethingArg struct definition
- now LenType args that denote the length of VmaType args are serialized as
"0x1000" instead of "(0x1000)"; to preserve backwards compatibility
syzkaller is able to parse the old format for now
- multiple small changes all over to make the above work
After this change syzkaller uses twice less memory after deserializing a
typical corpus.
The optimization change removed validation too aggressively.
We do need program validation during deserialization,
because we can get bad programs from corpus or hub.
Restore program validation after deserialization.
