Commit Graph

98 Commits

Author SHA1 Message Date
Dmitry Vyukov
c1fd2f3b4a syz-ci: fix formatting of error message
PrependContext prepends the string rather than format.
2019-03-17 18:06:44 +01:00
Dmitry Vyukov
41c6dd359b syz-ci: add flag that allows to not start managers
For bisection testing.

Update #501
2019-03-17 18:06:44 +01:00
Dmitry Vyukov
80765640f0 syz-ci: ignore job errors on shutdown
Ctrl+C can kill a child process which will cause an error.
Ignore such errors, it's better to retry after restart then
to report a false failure.

Update #501
2019-03-17 18:06:44 +01:00
Dmitry Vyukov
8fb18735b7 syz-ci: move GOPATH/GOROOT setup to main
GOPATH/GOROOT are also needed for jobs.
Updater can be disabled with -autoupdate=0,
so move GOPATH/GOROOT setup to main.

Update #501
2019-03-17 18:06:44 +01:00
Marco Vanotti
12365b99ce syz-ci: fix error reporting for syzupdater.
syzupdater fails to report build errors because it is constructing the
dashapi client with the wrong parameters (it is mixing dashboardAddr
with dashboardClient).

This commit swaps the order of those parameters in syzupdater's
uploadBuildError function.

Fixes #1044
2019-03-08 08:07:09 +01:00
Dmitry Vyukov
a8f92d1c95 syz-ci: enhance autoupdate=0 mode
If autoupdate=0 build syzkaller binaries once if they are missing.
Even without autoupdate syzkaller binaries are required for operation.
2019-02-26 16:27:33 +01:00
Dmitry Vyukov
18107ce07f syz-ci: make some manager creation errors non-fatal
If only part of the managers failed to initialize
we still can continue running the rest.
2019-02-23 08:45:42 +01:00
Dmitry Vyukov
9b05a0a131 syz-ci: add a flag to disable auto-update
Useful for local testing.
With -autoupdate=0 syz-ci does not need syzkaller repo,
will not poll, build and update itself.
So a binary with local changes can be tested without
pushing changes to some git repo.
2019-02-22 18:40:51 +01:00
Dmitry Vyukov
59f36113ee syz-ci: don't extract fixing commits if no report email
There can be no ReportEmail is email reporting is not used,
don't poll fixing commits in such case.
2019-02-17 15:46:16 +01:00
Dmitry Vyukov
3e98cc3080 dashboard/app: poll commits info
This implements 2 features:
 - syz-ci polls a set of additional repos to discover fixing commits sooner
   (e.g. it can now discover a fixing commit in netfilter tree before
   it reaches any of the tested trees).
 - syz-ci uploads info about commits to dashboard.
   For example, a user marks a bug as fixed by commit "foo: bar".
   syz-ci will find this commit in the main namespace repo
   and upload commmit hash/date/author to dashboard. This in turn
   allows to show links to fixing commits.

Fixes #691
Fixes #610
2019-02-17 15:08:45 +01:00
Julia Hansbrough
e3bc2f2b5a syz-ci: fix error log output
This eases debugging when running syz-ci locally.
2019-01-23 11:02:31 +01:00
Dmitry Vyukov
9a689e2d3d syz-ci: extend logging in uploadBuildError 2019-01-23 10:19:27 +01:00
Dmitry Vyukov
1e30dd27ee syz-ci: run syzkaller tests without sandbox
pkg/vcs git tests fail with sandboxing as part of git
invocations is sandboxed and part is not.
2018-12-18 10:48:38 +01:00
Dmitry Vyukov
bd8b4220d5 syz-ci: upload syzkaller build errors to dashboard
Fixes #793
2018-12-18 10:33:05 +01:00
Dmitry Vyukov
057f344bb7 syz-ci: add kernel crash title to boot errors
Fixes #829
2018-12-18 09:50:08 +01:00
Dmitry Vyukov
a9cc88c19f pkg/osutil: provide better Rename
os.Rename fails for cross-device renaming (e.g. to/from tmpfs).
This is quite unpleasant. Provide own version that falls back to copying.
2018-12-17 19:09:10 +01:00
Dmitry Vyukov
2d8316b8fb syz-ci: default kernel branch to master 2018-11-21 07:17:13 +01:00
Julia Hansbrough
b08ee62aec [fuchsia] SSH fix and build invocation fix.
SSH keys are now included at the fx clean-build config.

A proper escape sequence looked weird so use a string literal to pass
that config.

Fixed some typos I found while debugging.
2018-11-16 11:58:29 -08:00
Dmitry Vyukov
07a35107a4 syz-ci: split Manager.loop function
gometalinter says the function is too complex:

syz-ci/manager.go:155:⚠️ cyclomatic complexity 30 of function (*Manager).loop() is high (> 24) (gocyclo)

Split into 2 functions.
2018-09-21 10:44:49 +02:00
Dmitry Vyukov
6cee973cb5 syz-ci: fix manager http addr
Manager http addr is usually just ":port".
Prepend "127.0.0.1" in such case.
2018-09-20 19:58:12 +02:00
Dmitry Vyukov
9a6f6af645 syz-ci: upload coverage reports to GCS
Upload coverage reports from all managers to GCS
after 6 hours of runtime (to make it more apples-to-apples).
2018-09-20 18:29:38 +02:00
Dmitry Vyukov
0415891efa syz-ci: enable net/http/pprof
Useful to have for all servers.
2018-09-20 11:22:33 +02:00
Dmitry Vyukov
3805b76483 syz-ci: make goroot optional
If not set, syz-ci will use the system go.

Update #712
2018-09-17 10:51:17 +02:00
Dmitry Vyukov
19e9088bb7 syz-ci: use gmake instead of make on openbsd
Update #712
2018-09-13 12:13:54 +02:00
Dmitry Vyukov
ff2962ae80 syz-ci: de-hardcode kernel object file names
We used to have this list in syz-ci and syz-managers.
Now we have kernel object in sys/targets, so use it instead.

Update #712
2018-09-11 15:43:55 +02:00
Dmitry Vyukov
a54c2b7b92 syz-ci: de-hardcode list of VMs that support overcommit
We currently have this list in multiple places (somewhat diverged).
Specify this "overcommit" property in VM implementations.
In particular, we also want to allow overcommit for "vmm" type.

Update #712
2018-09-11 15:33:45 +02:00
Dmitry Vyukov
5509d25c1d syz-ci: allow manager http address auto-assignment
Manually assigning http addresses is tedious and error-prone.
2018-09-06 11:15:56 +02:00
Dmitry Vyukov
67669d1190 syz-ci: wait for current builds before stopping managers
If update comes in the middle of a long build (bisection),
we will stop all other managers prematurely (bisection can take a day).
So wait for current builds to finish before starting shutdown.

Update #501
2018-09-04 17:23:15 +02:00
Dmitry Vyukov
fbedd425b5 pkg/mgrconfig: move from syz-manager/mgrconfig
mgrconfig was used only by syz-manager initially,
but now it's used by a dozen of packages and it's
weird to import from under a binary dir.
pkg/ is much more reasonable dir for a widely used
helper package.
2018-08-02 16:57:32 +02:00
Dmitry Vyukov
8ef497b652 gometalinter: clean up vetshadow
This just cleans up existing warnings.
vetshadow is not enabled yet because it crashes.

Update #538
2018-07-31 20:38:33 +02:00
Dmitry Vyukov
8cc079c3f6 syz-ci: fix poll of fixing commits 2018-07-20 20:48:34 +02:00
Dmitry Vyukov
6d5bd5b5eb syz-ci: fix arch passed to build.Clean 2018-07-17 17:55:52 +02:00
Dmitry Vyukov
9fa03fa5c9 syz-ci: don't put dynamic data into boot/test errors
We append underlying error to the title of boot/test errors.
The error can come from anywhere and can contain dynamic data,
which can cause duplication of bugs.

Put the underlying error into report body instead.
2018-07-10 10:19:48 +02:00
Dmitry Vyukov
84277f259c syz-ci: unset GOPATH
syz-ci always passes own GOPATH,
if GOPATH is already present in env, it causes havoc.
Unset it.
2018-07-05 13:11:55 +02:00
Dmitry Vyukov
4906b5987e syz-ci: copy akaros object file 2018-07-05 13:11:37 +02:00
Dmitry Vyukov
68ce63c468 pkg/build: support fuchsia builds 2018-06-28 13:42:07 +02:00
Dmitry Vyukov
58e8587f64 pkg/vcs: pave way for multi-vcs support
Wrap current git interface in abstract interface.
Provide constructor that create repo interface for the given os/vm.
2018-06-27 13:57:21 +02:00
Dmitry Vyukov
43da5e3a1b pkg/vcs: move from pkg/git
Rename pkg/git to pkg/vcs because we need to support not only git.
2018-06-27 13:07:03 +02:00
Dmitry Vyukov
21caae5df6 syz-ci: don't report transient errors during patch testing
If no instances crashed and at least one booted and completed
testing successfully, report success.

Fixes #641
2018-06-26 19:17:45 +02:00
Dmitry Vyukov
802897bc33 pkg/instance: say that gvisor can create instances out of thin air 2018-06-23 10:12:35 +02:00
Dmitry Vyukov
c97f0d7a86 pkg/build: add gvisor support 2018-06-22 16:40:45 +02:00
Dmitry Vyukov
ea804a7120 pkg/build: pave way for multi-OS support
Unify kernel and image build, that distinction is really uninteresting.
Define interface that each OS needs to implement.
Add gvisor stub.
2018-06-22 16:40:45 +02:00
Dmitry Vyukov
94ef62054c pkg/build: move from pkg/kernel
Rename pkg/kernel to pkg/build and prepare for multi-OS support.
2018-06-22 16:40:45 +02:00
Dmitry Vyukov
2a075d57ab pkg/report: allow to specify suppressions per OS
Currently all (linux-specific) suppressions are hardcoded in mgrconfig.
This is very wrong. Move them to pkg/report and allow to specify per OS.
Add gvisor-specific suppressions.
This required a bit of refactoring. Introduce mgrconfig.KernelObj finally.
Make report.NewReporter and vm.Create accept mgrconfig directly
instead of passing it as multiple scattered args.
Remove tools/syz-parse and it always did the same as tools/syz-symbolize.
Simplify global vars in syz-manager/cover.go.
Create reporter eagerly in manager. Use sort.Slice more.
Overall -90 lines removed.
2018-06-22 16:40:45 +02:00
Dmitry Vyukov
6595937c34 tools/create-gce-image.sh: support both nbd and loop
Pass target OS/arch and VM type to kernel.CreateImage.
Use nbd for gce and loop for qemu VM type.
2018-05-17 14:50:18 +02:00
Dmitry Vyukov
477728038c syz-ci: fix manager config creation
SyzFuzzerBin/SyzExecprogBin/SyzExecutorBin are ended up being empty.
Fix this.
2018-05-14 12:17:34 +02:00
Dmitry Vyukov
2407e7407a pkg/instance: add package for testing of images/patches/bisection
Move helper image/patch testing code from syz-ci/testing.go
to a separate package so that it can be reused during bisection.

Update #501
2018-05-14 11:32:56 +02:00
Dmitry Vyukov
2a287a4dfc pkg/kernel: accept kernel config as byte slice
We usually have the config in memory,
so saving it to disk is an unnecessary step.
Accept byte slice directly.

Update #501
2018-05-14 11:17:23 +02:00
Dmitry Vyukov
4fce9bf152 syz-manager: add concept of partial configs
syz-ci uses partial (incomplete) manager config in several places.
Currently it is implemented in some ugly way.
Provide better support and unexport DefaultValues and SplitTarget.

Update #501
2018-05-14 11:16:57 +02:00
Dmitry Vyukov
9e0846e8a4 all: get rid of underscores in identifiers
Underscores are against Go coding style.

Update #538
2018-05-07 17:59:06 +02:00