3127 Commits

Author SHA1 Message Date
Dmitry Vyukov
c138f09215 tools/syz-trace2syz/proggen: remove currentStraceArg
It's used only by 2 functions: genSockaddrNetlink and genIfrIfru.
Majority of functions just accept the straceType as argument,
which looks like a much more appropriate way to pass an argument to a function.
Amusingly, both functions already accept and use the straceType as argument.
2018-12-07 14:28:12 +01:00
Dmitry Vyukov
276faf74b2 tools/syz-trace2syz/proggen: unexport and refactor Context
1. Unexport Context, it's not meant for callers.
2. Unexport all Context fields.
3. Make all function Context methods.
2018-12-07 14:23:58 +01:00
Dmitry Vyukov
c9f43ce698 tools/syz-trace2syz/proggen: tidy up shouldSkip 2018-12-07 14:08:56 +01:00
Dmitry Vyukov
4f39cef6c2 tools/syz-trace2syz/proggen: convert tests to table format
This has number of advantages:
1. Tests are readable and writable.
   The current checks [1] are neither.
2. Tests are much more compact.
3. Tests verify all aspects rather than just
   1 aspect of the resulting program.
4. Tests are much less fragile.
5. Any diffs in the results will be more clearly visible.

[1]
switch a := p.Calls[1].Args[0].(type) {
case *prog.ResultArg:
	if a.Res != p.Calls[0].Ret {

switch a := p.Calls[1].Args[0].(type) {
case *prog.ResultArg:
	pipeSecondFd := p.Calls[0].Args[0].(*prog.PointerArg).Res.(*prog.GroupArg).Inner[1]
	if a.Res != pipeSecondFd {

write := p.Calls[len(p.Calls)-2]
inotifyRmWatch := p.Calls[len(p.Calls)-1]
switch a := write.Args[0].Type().(type) {
case *prog.ResourceType:
	if a.TypeName != "fd" {
		t.Fatalf("expected first argument of write to have type fd, got: %s", a.TypeName)
	}
default:
	t.Fatalf("first argument of write is not resource type: %s", a.Name())
}
switch a := inotifyRmWatch.Args[1].(type) {
case *prog.ResultArg:
	b := a.Type().(*prog.ResourceType)
	if b.TypeName != "inotifydesc" {
		t.Fatalf("expected second argument of inotify_rm_watch to have type inoitfydesc, got: %s", b.TypeName)
	}
	if a.Res != p.Calls[2].Ret {
		t.Fatalf("inotify_rm_watch's second argument should match the result of inotify_add_watch.")
	}
}

sockaddr, ok := a.(*prog.PointerArg).Res.(*prog.GroupArg)
if !ok {
	t.Fatalf("%s", a.Type().Name())
}
ipv4Addr, ok := sockaddr.Inner[2].(*prog.UnionArg)
if !ok {
	t.Fatalf("expected 3rd argument to be unionArg, got %s", sockaddr.Inner[2].Type().Name())
}
optName := ipv4Addr.Option.Type().FieldName()
if !strings.Contains(optName, "rand") {
	t.Fatalf("expected ip option to be random opt, got: %s", optName)
}
ip, ok := ipv4Addr.Option.(*prog.ConstArg)
if !ok {
	t.Fatalf("ipv4Addr option is not IntType")
}
if ip.Val != expectedIp {
	t.Fatalf("parsed != expected, %d != %d", ip.Val, expectedIp)
}
2018-12-07 14:01:26 +01:00
Dmitry Vyukov
eada53b810 tools/syz-trace2syz/proggen: fix vma allocation
There are 2 bugs:
1. We always allocate 1 page, even if use more.
2. VMA addresses are not aligned, so most mmap-like functions fail with EINVAL.
The added test currently panics with "unaligned vma address".
2018-12-07 12:56:38 +01:00
Dmitry Vyukov
9e8a45fe27 tools/syz-trace2syz/proggen: replace memoryTracker with prog.memAlloc 2018-12-07 12:44:45 +01:00
Dmitry Vyukov
413e414738 tools/syz-trace2syz: add go-fuzz fuzzer
Inputs like "2__R" or "3_F	T.3.3l" make
traze2syz hang infinitely consuming all machine memory.
Need to fix all crashes over time.
2018-12-07 12:12:27 +01:00
Dmitry Vyukov
742f85bb22 tools/syz-trace2syz: start adding proper error handling
log.Fatal is not the proper way to handle errors.
It does not allow to write good tests, fuzzers
and utilities that crash all the time.
2018-12-07 12:05:43 +01:00
Dmitry Vyukov
8056889866 tools/syz-trace2syz/proggen: add ParseFile function
Current code structuring has 2 problems:

1. parsing anything with proggen requires complex multistep dance including
 - parsing data with parser
 - walking the resulting tree manually and calling proggen on each
 - then for each context
   - calling FillOutMemory (unclear why it's not part of parsing)
   - calling prog.Finalize
   - checking is the program is not too large
All of this duplicated across trace2syz and tests.
And any new tests or fuzzers we will write will need to duplicate
all of this logic too.

2. As the result of this structuring, lots of proggen guts
and implementation details are exposed.
While none of the callers are actually intersted in Context details,
they are not interested in Context itself whatsoever.

What every caller wants is "here is data to parse, give me programs".
Add such function.
2018-12-07 11:30:13 +01:00
Dmitry Vyukov
840b5cc058 tools/syz-trace2syz/parser: remove Filename from TraceTree
We already printed file name of the trace in parseTraces,
no need to print it again and again.
Consequently we don't need Filename in TraceTree.
If needed, caller can always log it before parsing,
or pass along with the TraceTree.
2018-12-07 10:43:27 +01:00
Dmitry Vyukov
5cdc1f5f6d tools/syz-trace2syz: use short variable declaration syntax
Use short variable declaration syntax where possible.
Move declarations closer to usages.
2018-12-07 10:35:42 +01:00
Dmitry Vyukov
ae17c862c9 tools/syz-trace2syz/parser: use []byte instead of string for file contents
If we are handling whole files, it's more efficient to use []byte.
string is not really meant to hold large amounts of data.
2018-12-07 10:24:24 +01:00
Shankara Pailoor
1eb6a7e433 tools/syz-trace2syz: adding missing copyright headers
Adding missing copyright headers to return_cache.go and context.go
2018-12-07 09:50:45 +01:00
Greg Steuck
610352d5d7 pkg/build/openbsd: require kernel config files as configuration data
Previously the config was generated directly, but testing multiple
configurations makes this cumbersome going forward. This makes
kernel_config a mandatory parameter.
2018-12-07 09:50:12 +01:00
Anton Lindqvist
f1c702a86e sys/targets: some syscalls on OpenBSD does need defines
Some syscalls on OpenBSD violates the ordinary SYS_ prefix convention. This is
an exhaustive enumeration of the deviations.

Regression introduced in commit 88746fdf ("pkg/csource: use defines from
sys/syscall.h on *bsd").
2018-12-07 09:45:14 +01:00
Anton Lindqvist
8bff832f48 sys/openbsd: correct semctl syscall number 2018-12-07 09:45:14 +01:00
Greg Steuck
dcf836b12d tools/syz-trace2syz/proggen/return_cache.go: format string mismatch 2018-12-06 20:33:02 +01:00
Dmitry Vyukov
ab4b148b34 prog: add Prog.Finalize
Prog.Finalize combines assignSizesCall, SanitizeCall and validate.
Intended for users who build own programs,
so that we don't need to expose all individual methods.
2018-12-06 18:56:08 +01:00
Dmitry Vyukov
ceaec61a83 prog: export Type.DefaultArg
It's effectively exported anyway.
So export it the proper way.
2018-12-06 18:55:46 +01:00
Dmitry Vyukov
f40330afce tools/syz-trace2syz: skip 2 more syscalls
These set_robust_list and set_tid_address are issued by glibc
for every process/thread start.
Normal programs don't use them and it's unlikely we build
something interesting with them (e.g. we won't get real robust list in memory).
Skip them.
2018-12-06 17:33:05 +01:00
Dmitry Vyukov
df8657a90f tools/syz-trace2syz: tidy up code
Lots of assorted changes mainly converting code to idiomatic Go
and replacing code with equivalent, but shorter code.
2018-12-06 17:28:09 +01:00
Dmitry Vyukov
c1641491e4 pkg/db: provide helper function for database creation
This is needed for both tools/syz-db and tools/syz-trace2syz.
Also, remove code to resolve SHA1 collisions.
Also, don't set db version as we actually want to minimize
and smash these programs like anything else
(not minimizing nor smashing them is only useful during tool testing).
2018-12-06 16:49:37 +01:00
Dmitry Vyukov
d68400a8d1 tools/syz-trace2syz: merge config package into proggen
Since we now have only single variable there,
it does not seem to deserve a separate package.
2018-12-06 16:30:14 +01:00
shankarapailoor
6a60a19530 tools/syz-trace2syz: add tool to convert strace output to programs
* fixing weird merge error

* fixing presubmit

* fixing presubmit

* removing parsing code because of -Xraw option

* fix presubmit

* update

* deleting vma_call_handlers as we are currently skipping most vma calls. This simplifies memory_tracker as we don't need to keep track of vma allocations

* removing custom handling of bpf_instruction union

* removing ifconf parsing

* update

* removed all expression types and replaced them with constant types. removing ipv6_addr parsing while -Xraw is getting fixed. Removing constants.go

* removing ipv6 parsing

* presubmit

* moving direction check from ipv4_addr out to genUnion

* removing code that parses kcov

* removing redundant test

* removing custom code in generate unions to fill ipv4_addr

* proggen: changing order of imports to make external packages import first

fixing presubmit

* changing log messages to lower case to be consistent with other packages.

* removing pointer type and simplifying memory_tracker

removing comment

* moving context and return_cache to seaparate files

* deleting default argument generation when we should probably throw an error
2018-12-06 16:25:37 +01:00
Dmitry Vyukov
3ab38479ab
Update syzbot.md
add newer clang compiler
2018-12-06 09:38:08 +01:00
Dmitry Vyukov
9858b52e7e
Update found_bugs.md 2018-12-06 07:45:08 +01:00
Dmitry Vyukov
764b42c46b pkg/csource: exclude linux/arm64 tests
I think I misinterpreted the error that episodically happens on ci:

collect2: error: ld terminated with signal 11 [Segmentation fault], core dumped
compiler invocation: aarch64-linux-gnu-gcc [-Wall -Werror -O1 -g -o /tmp/syz-executor570589071 -pthread -DGOOS_linux=1 -DGOARCH_arm64=1 -x c - -static]

as OOM, but they all involve aarch64-linux-gnu-gcc:

https://travis-ci.org/google/syzkaller/jobs/461827347
https://travis-ci.org/google/syzkaller/jobs/460226110
https://travis-ci.org/google/syzkaller/jobs/463564291

So I guess the problem can be with the arm64 toolchain that just crashes randomly.
2018-12-05 13:59:01 +01:00
Michael Tüxen
c64cb0567e sys/freebsd: remove linux specific include files for ICMP 2018-12-05 10:51:02 +00:00
Michael Tüxen
ac6c05788b sys/freebsd: improve TCP tests
* sys/freebsd: improve TCP tests

Add missing TCP socket options for FreeBSD.

* sys/freebsd: improve TCP tests

Add socket option description for TCP_FASTOPEN.
2018-12-05 06:57:20 +00:00
Michael Tuexen
f162ad97ac sys/freebsd: improve udp tests
Add support for the UDP_ENCAP. Add required includes and
remove the Linux specific ones.
2018-12-04 14:15:42 +00:00
Michael Tüxen
6ad0ae6171 sys: remove socketpair for AF_INET and AF_INET6
* OpebBSD: remove socketpair() for AF_INET and AF_INET6.

socketpair() is only supported on AF_UNIX.

* NetBSD: remove socketpair() for AF_INET and AF_INET6.

socketpair() is only supported for AF_UNIX.

* FreeBSD: remove socketpair() for AF_INET and AF_INET6.

socketpair() only supports AF_UNIX.

* Linux: remove socketpair for AF_INET and AF_INET6.

socketpair only supports AF_UNIX.

* Autogenerated files.

These are manually generated for all platforms you are not
running on. FreeBSD in this case.

* executor: rebase.

* sys/freebsd: rebase.

* sys/linux: use AF_UNIX based socketpair for nbd.

This was suggested by Dmitry.

Fixes #845
2018-12-04 08:58:22 +00:00
Greg Steuck
49e1764c75 tools/create-openbsd-vmm-worker.sh: shut off pagination in ddb 2018-12-04 08:55:04 +00:00
Michael Tüxen
03f94a4556 sys/freebsd: add UDP-Lite descriptions 2018-12-03 13:52:09 +00:00
Dmitry Vyukov
2192790481 pkg/csource: reduce short tests
pkg/csource test gets OOM-killed on travis:
https://travis-ci.org/google/syzkaller/jobs/461827347
https://travis-ci.org/google/syzkaller/jobs/460226110

Add several measures:
 - set GOMAXPROCS=1 to restrict parallel processes
 - remove -g from compiler invocation
 - reduce set of tests run in short mode to compensate for GOMAXPROCS=1
 - also reduce set of tests in full mode as they timeout now
2018-12-03 13:29:57 +01:00
Dmitry Vyukov
c3ff1810cf Makefile: constrain gometalinter memory usage
gometalinter gets OOM-killed on travis:
https://travis-ci.org/google/syzkaller/jobs/462546388

I've run 4 experiments locally and this combination
seems to make memory usage slightly better:

GOGC=50 GOMAXPROCS=1 TIME="%e %M" time gometalinter.v2 ./...
114.35 8274556
122.54 8293580
117.85 8349960

GOGC=50 GOMAXPROCS=1 TIME="%e %M" time gometalinter.v2 ./...
147.45 7512512
150.13 6820408
149.26 7909636

GOGC=100 GOMAXPROCS=2 TIME="%e %M" time gometalinter.v2 ./...
109.73 8332476
114.91 8563776
114.54 8478368

GOGC=50 GOMAXPROCS=2 TIME="%e %M" time gometalinter.v2 ./...
145.43 6726144
147.40 8247952
155.56 7815656
2018-12-03 13:29:57 +01:00
Dmitry Vyukov
819002b081
docs: update contributing.md
Clarify commit description guidelines.
2018-12-03 10:36:40 +01:00
Dmitry Vyukov
31677db602 sys/targets: use g++ as preprocessor on freebsd
As per:
https://github.com/google/syzkaller/pull/844#issuecomment-443509014
2018-12-03 10:02:43 +01:00
Dmitry Vyukov
7dcaeaf322 vm/gce: close old consolew in Run
Run can be executed several times on a VM.
2018-12-02 13:23:51 +00:00
Greg Steuck
7a0edfbe7c vm/gce: use openbsd console diagnostic code for both vmm and gce
* openbsd: use console diagnostic code for both vmm and gce.

* gometalinter wants less indentation and more stuff in scope

* Comment no longer applies
2018-12-02 13:22:10 +00:00
Anton Lindqvist
048d09b205 docs/openbsd: update found_bugs.md 2018-12-02 13:20:29 +00:00
Greg Steuck
88746fdf89 pkg/csource: use defines from sys/syscall.h on *bsd
Fixes #841
2018-12-02 13:18:07 +00:00
Greg Steuck
e0d8c853f6 tools: set openbsd hostname to instance name for serviceability
* openbsd images: set hostname to instance name for serviceability

* openbsd startup scripts insist on EOL characters
2018-12-02 09:53:06 +00:00
Michael Tüxen
5a58167323 executor: add support for tap interfaces on FreeBSD 2018-12-01 10:22:39 +00:00
Greg Steuck
d8988561c0 openbsd: repair pkg/csource_test 2018-11-30 19:55:51 +00:00
Dmitry Vyukov
f0d4c650a3 pkg/report: suppress another gvisor OOM 2018-11-30 19:17:05 +00:00
Michael Tüxen
271b354c51 vm/qemu: improve debug output
* vm/qemu: Improve debug output.

When running in debug mode, the number of VMs is reduced to 1.
State this in the debug output.

* vm/qemu: Don't start debug output with a capital letter.

As requested by Dimitry.

* vm: Provide debug message when reduing number of VMs.

Apply this change to all affected platforms for consistency.
Suggested by Dmitry.

* Add myself to AUTHORS/CONTRIBUTORS files.

* vm: Fix compilation issues missed in earlier commit.

* vm: Use logging to write debug message.
2018-11-30 17:12:03 +00:00
Shankara Pailoor
bc6b598a2d removing trace2syz constants and moving rand_addr as the first ipv4_addr option 2018-11-30 15:24:55 +00:00
Michael Tüxen
115a1379eb FreeBSD: Fix make extract
* FreeBSD: Fix make extract.

A header was missing...

* FreeBSD: Fix make extract

Changes in generated files.
2018-11-30 15:13:30 +00:00
Michael Tuexen
8b13c56200 executor: Use correct macros.
Suggested by Dmitry Vyukov.
2018-11-30 13:48:23 +00:00
Michael Tuexen
9656dc4c83 executor: Fix compilation on FreeBSD.
This is an autogenerated file, but it seem that changes to it
resulting from changes in executor/common_bsd.h must also be
committed.
e
2018-11-30 13:48:23 +00:00