Currently we have them duplicated in docs/configuration.md
and the actual source in pkg/mgrconfig/config.go.
Documentation is missing in one place or another,
some is outdated, some is phrased slightly differently.
Move all docs to pkg/mgrconfig/config.go and reference it
from docs/configuration.md instead.
We duplicated the no output timeout in the repro package,
and it got out of sync. It's not 3 mins now, but 5 mins.
Remove the duplication and fix this.
* vm: add bhyve support
bhyve is FreeBSD's native hypervisor. Because it is missing snapshot
support and user networking, some additional configuration on the host
is required. However, unlike QEMU on FreeBSD, bhyve can make use of
hardware virtualization features and is thus faster.
* docs/freebsd: document bhyve support
We are seeing some flakes during bisection and image testing.
Hard to tell what's the root cause because they are episodic.
But using non-preemptible VMs for bisection and image testing
looks good on all fronts.
Update #501
Currently we try to ssh into the machine for 10 minutes
even if it crashed right away. Make qemu exit on kernel panic
and stop ssh'ing when qemu exits.
Handling bad kernels fast is actually important for bisection.
Update #501
* Modified pkg/build/netbsd.go
1. Made kernel build incremental
2. Added code to copy kernel to disk image
* Fix formating issues
* Fixed basic issues
* Fixed copy kernel to disk
1. Added CPU option to default
2. Added snapshot option to vm/qemu/qemu.go
3. Hacky solution to get the build working
* Fixed issues and added vm.MonitorExecution
* Added sync instead of poweroff
The console and root that we use are specific for 386/amd64.
arm/arm64 usually use a different console and root device.
Use separate settings for 386/amd64 and arm/arm64.
A line length of 79 in the ddb output does not necessarily imply that the
following line is a continuation of the current line. Since there's no way
to distinguish between ordinary and continuation lines, it could end up
corrupting the report by joining two lines that are disjoint[1].
Instead, disable line wrapping in ddb. If we want some kind of wrapping in
the future it's easier done by pkg/report.
[1] https://syzkaller.appspot.com/bug?extid=03f7377a9848d7d008c9
* pkg/report: symbolize witness trace on OpenBSD
* vm/vmimpl: show witness locks on panic
* vm/vmimpl: show memory stats on panic
* fixup! pkg/report: symbolize witness trace on OpenBSD
Disable qemu VGA output on ppc64le. The current pseries console code in
the kernel has a bunch of bugs, and if you don't disable VGA syzkaller
won't see console output.
Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
ppc64le supports KVM, let's use it.
This obviously won't work on an x86 host, but at present there are issues
using syzkaller on ppc64le with qemu in TCG mode anyway.
Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Currently we only support canExit flag.
However there are actually 3 separate conditions:
- program can exit normally
- program can timeout (e.g. fuzzer test or runtest can't)
- program can exit with error (e.g. C test can)
Allow to specify these 3 conditions separately.
Rather than writing the diagnosis to the kernel console, Diagnose can
now directly return the extra debugging info, which will be appended ot
the kernel console log.
This gives almost 100% coverage for MonitorExecution.
Test all corner cases like lost connection, no output,
diagnose, exiting/non-exiting programs, etc.
Update #875
Diagnose currently sends the panic signal to generate a traceback for
additional context.
However, Diagnose is also called in otherwise successful scenarios
(vm.Instance.MonitorExecution -> vm.monitor.extractError). Triggering a
panic will make this successful scenario look like a failure.
We could simply suppress this panic, but 1) that means we never shutdown
cleanly (not important, but ugly), and 2) we're less likely to detect
delayed crashes since we kill the sandbox immediately (that's what
MonitorExecution is checking for).
Instead, switch from -panic-signal to -trace-signal, which simply logs a
traceback without exiting. This option was added to runsc in
24c1158b9c.
The other uses of Diagnose will always generate a report regardless of
an additional panic, so we're not losing any reports.
* vm/qemu: Improve debug output.
When running in debug mode, the number of VMs is reduced to 1.
State this in the debug output.
* vm/qemu: Don't start debug output with a capital letter.
As requested by Dimitry.
* vm: Provide debug message when reduing number of VMs.
Apply this change to all affected platforms for consistency.
Suggested by Dmitry.
* Add myself to AUTHORS/CONTRIBUTORS files.
* vm: Fix compilation issues missed in earlier commit.
* vm: Use logging to write debug message.
Allow setting qemu_args to "" in the config file. This is needed
when running qemu from the qemu-devel package on FreeBSD, which
does not support the -enable-kvm option.
Without this patch, an entry "" is added to the list of command
line parameters, which breaks the starting of the qemu instances.
* build/openbsd: minor cleanup (use tuples instead of maps)
* Grammar nits in comments.
* Simplify openbsd.Create, will defer when there's more than one error exit.
* pkg/build: Support copying kernel into GCE image
* Simple test for openbsd image copy build.
* Cleanup in case something failed before.
* Support multi-processor VMs on GCE.
* More debug
* Reformat
* OpenBSD gce image needs to be raw.
* GC
* Force format to GNU directly on Go 1.10 or newer.
* Use vmType passed as a parameter inside openbsd.go
* gofmt
* more fmt
* Can't use GENERIC.mp just yet.
* capitalize
* Copyright
The ddb(4) debugger defaults to showing 24 lines at a time, the next chunk of
lines will be displayed only after receiving keyboard input. Setting maxlines to
0 disables pagination completely.
As of commit 3f053259, gVisor sentry panics are no longer sent to the
stderr for "runsc run" by default, as that stderr belongs exclusively to
the application.
As a result, syzbot never sees the gVisor panic stack trace, and is only
reporting errors that occur when waiting for a dead sandbox.
Passing the "-debug" flag to runsc will make the sentry panics visible
to syzbot again.
As a result, the boot time is significantly improved since there's no longer any
need to copy the complete disk.
This feature was recently committed to OpenBSD-current. Any existing base image
used must be recreated, this time using the qcow2 disk format.
vmctl start periodically fails with:
vmctl: start vm command failed: Operation already in progress
So try to sleep for a bit after vmctl stop.
And detect when vmctl start terminates prematurely
to avoid 10 minute timeout for ip extraction.
vmctl console fails from time to time with:
vmctl: console not found
Probably there is some race (most of these things assume
that there is a human typing commands with delays).
Also, vmctl start can connect to console itself with -c flag.
So use that because it both solves the console race and
also makes code much more similar to other VM implementations (qemu, gvisor).
This also eliminates 3 additional goroutines per VM.
A dozen of vmm's running on a GCE machine can be really slow to boot.
Timeouts have only single goal: preventing complete system stalls
when/if external commands episodically hang. There is no value
in keeping them as close as possible to expected durations.
This can only lead to various flakes. Increase timeouts by an
order of magnitude.
The goroutine sends on bootOutputStop to notify about its completion,
but the main goroutine is not receiving from the chan on success
and since the chan in unbuffered, the output reading goroutine
hangs on the send forever.
I am getting:
failed to run vmctl -name syzkaller-ci-openbsd-main-test-0
vmctl: name too long
The name is auto-generated from parts which ensure that it is unique.
We can't easily name it shorter. So strip the syzkaller prefix,
which is not strictly necessary.
