1. Always append diagnosis output at the end.
Don't intermix it with kernel output. It's confusing and not useful.
2. Don't include diagnosis output into Report.
It's too verbose and is not the crash. Keep it only in the Output.
The new manager config argument workdir_template refers to a directory. Optional.
Each VM will get a recursive copy of the files that are present in workdir_template.
VM config can then use these private copies as needed. The copy directory
can be referenced with "{{TEMPLATE}}" string. This is different from using
the files directly in that each instance will get own clean, private,
scratch copy of the files. Currently supported only for qemu_args argument
of qemu VM type. Use example:
Create a template dir with necessary files:
$ mkdir /mytemplatedir
$ truncate -s 64K /mytemplatedir/fd
Then specify the dir in the manager config:
"workdir_template": "/mytemplatedir"
Then use these files in VM config:
"qemu_args": "-fda {{TEMPLATE}}/fd"
We've got a case when "ODEBUG:" was incorrectly detected as crash.
That was caused by a flaw in matchPos logic. Fix that.
See the added test for details.
We duplicated the no output timeout in the repro package,
and it got out of sync. It's not 3 mins now, but 5 mins.
Remove the duplication and fix this.
* vm: add bhyve support
bhyve is FreeBSD's native hypervisor. Because it is missing snapshot
support and user networking, some additional configuration on the host
is required. However, unlike QEMU on FreeBSD, bhyve can make use of
hardware virtualization features and is thus faster.
* docs/freebsd: document bhyve support
Currently we try to ssh into the machine for 10 minutes
even if it crashed right away. Make qemu exit on kernel panic
and stop ssh'ing when qemu exits.
Handling bad kernels fast is actually important for bisection.
Update #501
Currently we only support canExit flag.
However there are actually 3 separate conditions:
- program can exit normally
- program can timeout (e.g. fuzzer test or runtest can't)
- program can exit with error (e.g. C test can)
Allow to specify these 3 conditions separately.
Rather than writing the diagnosis to the kernel console, Diagnose can
now directly return the extra debugging info, which will be appended ot
the kernel console log.
This gives almost 100% coverage for MonitorExecution.
Test all corner cases like lost connection, no output,
diagnose, exiting/non-exiting programs, etc.
Update #875
* build/openbsd: minor cleanup (use tuples instead of maps)
* Grammar nits in comments.
* Simplify openbsd.Create, will defer when there's more than one error exit.
* pkg/build: Support copying kernel into GCE image
* Simple test for openbsd image copy build.
* Cleanup in case something failed before.
* Support multi-processor VMs on GCE.
* More debug
* Reformat
* OpenBSD gce image needs to be raw.
* GC
* Force format to GNU directly on Go 1.10 or newer.
* Use vmType passed as a parameter inside openbsd.go
* gofmt
* more fmt
* Can't use GENERIC.mp just yet.
* capitalize
* Copyright
We currently have this list in multiple places (somewhat diverged).
Specify this "overcommit" property in VM implementations.
In particular, we also want to allow overcommit for "vmm" type.
Update #712
mgrconfig was used only by syz-manager initially,
but now it's used by a dozen of packages and it's
weird to import from under a binary dir.
pkg/ is much more reasonable dir for a widely used
helper package.
Some kernel bugs don't stop kernel.
For such bugs whiel vm.MonitorExecution waits for kernel output for 10 secs,
fuzzer continues running programs and produces tons of output
after the kernel bug message. Kill fuzzer once MonitorExecution
detects a kernel bug.
See #516 for description of the problem.
The new scheme is:
1. RCU stalls the highest priority.
CONFIG_RCU_CPU_STALL_TIMEOUT=100
which results in stalls detected after 100-101 secs.
2. Then softlockup detector.
kernel.watchdog_thresh = 55 (sysctl)
which surprisingly detects stalls after 110-132 secs.
3. Then hung tasks and workqueue stalls.
Unfortunately we can't separate them because that would
require setting "no output" timeout to 10+ minutes.
workqueue.watchdog_thresh=140 (cmdline)
CONFIG_DEFAULT_HUNG_TASK_TIMEOUT=140
Both are detected after 140-280 secs.
4. Finally, "no output" crashes.
Detected by vm.MonitorExecution after 300 secs.
Fixes#516
Currently all (linux-specific) suppressions are hardcoded in mgrconfig.
This is very wrong. Move them to pkg/report and allow to specify per OS.
Add gvisor-specific suppressions.
This required a bit of refactoring. Introduce mgrconfig.KernelObj finally.
Make report.NewReporter and vm.Create accept mgrconfig directly
instead of passing it as multiple scattered args.
Remove tools/syz-parse and it always did the same as tools/syz-symbolize.
Simplify global vars in syz-manager/cover.go.
Create reporter eagerly in manager. Use sort.Slice more.
Overall -90 lines removed.
In pkg/report we add up to 5 lines of kernel output before the report.
However, MonitorExecution leaves only up to 128 bytes of preceeding output,
so frequently preceeding lines are not included in the report.
Increase the context to 512 bytes.
Try extracting report from console output only first. If that doesn't work,
try extracting it from the whole log.
Add regexp for executor printed BUGs.
Optimize regexps for rcu detected stalls.
Update rep.StartPos and rep.EndPos in vm/vm.go as well as rep.Output.
Make it possible to monitor health and operation
of all managers from dashboard.
1. Notify dashboard about internal syz-ci errors
(currently we don't know when/if they happen).
2. Send statistics from managers to dashboard.
Boot and minimally test images before declaring them as good
and switching to using them.
If image build/boot/test fails, upload report about this to dashboard.
Currently getting a complete report requires a complex,
multi-step dance (including getting information that
external users are not interested in -- guilty file).
Simplify interface down to 2 functions: Parse and Symbolize.
Parse does what it did before, Symbolize symbolizes report
and fills in maintainers. This simplifies both implementations
of Reporter interface and all users of the interface.
Potentially we could get this down to 1 function Parse
that does everything. However, (1) Symbolize can fail,
while Parse cannot, (2) usually we want to ignore (log)
Symbolize errors, but otherwise proceed with the report,
(3) repro does not need symbolization for all but the
last report.
Whole raw output is indivisble part of Report,
currently we always pass Output separately along with Report.
Make Output a Report field.
Then, put whole Report into manager Crash and repro context and Result.
There is little point in passing Report as aa bunch of separate fields.
This allows callers to get access to Report.Corrupted.
Better than adding 6-th return value and will allow
to pipe other report properties if necessary.
We currently have several names for crash attributes, which is disturbing.
E.g. crash title is called "Title" or "Desc". Name them consistently.
Title - single line bug identity.
Report - whole crash text.
Log - whole fuzzer/kernel output.
Frequently it's the same condition.
In one case there is just a stray error message on console
that turns the crash into "not executing programs".
While in another case there is no stray message,
and then it's detected as "no output".
Add a new isolated VM for machines that you cannot easily manage. It
assumes the machine is only available through SSH and create a reverse
proxy to ensure the machine can connect back to syz-manager.
Signed-off-by: Thomas Garnier <thgarnie@google.com>
Currently we have unix permissions for new files/dirs
hardcoded throughout the code base. Some places use 0644,
some - 0640, some - 0600 and a variety of other constants.
Introduce osutil.MkdirAll/WriteFile that use the default
permissions and use them throughout the code base.
This makes permissions consistent and also allows to easily
change the permissions later if we change our minds.
Also merge pkg/fileutil into pkg/osutil as they become
dependent on each other. The line between them was poorly
defined anyway as both operate on files.
VM infrastructure currently has several problems:
- Config struct is complete mess with a superset of params for all VM types
- verification of Config is mess spread across several places
- there is no place where VM code could do global initialization
like creating GCE connection, uploading GCE image to GCS,
matching adb devices with consoles, etc
- it hard to add private VM implementations
such impl would need to add code to config package
which would lead to constant merge conflicts
- interface for VM implementation is mixed with interface for VM users
this does not allow to provide best interface for both of them
- there is no way to add common code for all VM implementations
This change solves these problems by:
- splitting VM interface for users (vm package) and VM interface
for VM implementations (vmimpl pacakge), this in turn allows
to add common code
- adding Pool concept that allows to do global initialization
and config checking at the right time
- decoupling manager config from VM-specific config
each VM type now defines own config
Note: manager configs need to be changed after this change:
VM-specific parts are moved to own "vm" subobject.
Note: this change also drops "local" VM type.
Its story was long unclear and there is now syz-stress which solves the same problem.
This commit adds Odroid C2 support to syzkaller.
It's now possible to specify "type": "odroid" in manager config.
Documentation on how to setup fuzzing with Odroid C2 board is here:
https://github.com/google/syzkaller/wiki/Setup:-Odroid-C2
Note, that after this change libusb-1.0-0-dev package should be
installed to build syzkaller.
syz-fuzzer never exits (normally) so this does not affect syz-manager.
But during reproduction we can run a short running program (no repeat mode)
and currently VMs treat premature exit as an error.
Properly detect when a program exits and let callers decide what to do with it.
Add new config parameter "ignores" which contains list of regexp expressions.
If one of the expressions is matched against oops line,
crash report is not saved and VM is not restarted.
