syzkaller/pkg
Dmitry Vyukov bcbe71c80a dashboard/app, syz-ci: bisection support
This adds bulk of support for bisection to dashboard/app and syz-ci:
- APIs to send bisection jobs and accept results
- syz-ci logic to execute bisection jobs
- formatting of emails with results
- showing of results on dashboard

Some difficulties we have to overcome:
- since linux is frequently build/boot broken, lots of bisections are inconclusive,
  need to present such results too
- git bisect is poorly suitable for automation, have to resort to output parsing (is output stable?)
- git bisect turns out to fail (exit with non-0 status) when bisection is inconclusive
  (multiple potential cause commits)
- older syzkaller revisions can't be built with newer (broken) kernel header, e.g.:
  ebtables.h:197:19: error: invalid conversion from ‘void*’ to ‘ebt_entry_target*’
- newer compilers produce more warnings and break old syzkaller builds, e.g.:
  kvm.S.h:6:12: error: ‘kvm_asm64_vm86’ defined but not used [-Werror=unused-const-variable=]
- figuring relevant emails to CC from a commit is non-trivial:
  besides commit author, there can be some emails in commit tags, or not,
  which tags to use is an interesting question (some may include irrelevant emails)
  we can also run get_maintainers.pl on the commit, but this can produce too wide
  list if commit touches lots of files, it can also produce too small list,
  and then we need to resort to blame
- for inconclusive bisection we probably don't need to include emails referenced
  in the commits (there can be too many of these commits)
- need to be careful to exclude own syzbot email from commit CC list,
  now syzbot emails are referenced in some commits (Reported-by/Tested-by/etc)
  (can cause some kind of infinite recursion)
- lots of commits reference stable mailing list,
  we should not include it in CC because it's referenced for backports rather then bug reports
- since we add new Bug entity fields which we use in queries,
  whole datastore need to be upgrades to add the new field to index
- we must not discard the crash that was used for bisection
  (treat it as a reported crash)
- bisection results need 2 forms of reports:
  one when we add bisection results to already reported bug
  another when we report a bug first time with bisection results
- when reporting a bug with bisection results we need to use the crash
  that was used for bisection
- some fraction of bisections will probably fail with various errors
  and we will need some mechanism to retry bisection after the root cause is resolved
  this is not implemented yet
- linux-next is problematic for 2 reasons:
  fix bisection can't possibly run on linux-next as commits are not reachable from HEAD
  lots of commits are missing in linux-next (even in linux-next-history)
  e.g. we have some c63e9e91a254a52 which is now missing in linux-next/linux-next-history
- older kernels can't be build with fresh gcc/binutils/perl/make/glibc
  for now we have to stop at v3.9 (this only requires switching gcc several times along the way)
- kernels past v4.11 do not build with gcc 7 and 8 (undefined reference to `____ilog2_NaN')
- v4.1 and back have only compiler-gcc5.h
- v3.17 and back have only compiler-gcc4.h
- v3.6 and back do not have make olddefconfig
- compat socket calls can't be bisected past "x86/entry/syscalls: Wire up 32-bit
  direct socket calls" (v4.10) because of
  https://syzkaller.appspot.com/bug?id=b5b150e322d5f48c869bcf1528cdbee08d1421cb
- v2.6.28 and below does not work with modern make:
  *** mixed implicit and normal rules: deprecated syntax
- v3.8 build fails:
  Can't use 'defined(@array)' (Maybe you should just omit the defined()?) at kernel/timeconst.pl line 373.
  kernel/Makefile:134: recipe for target 'kernel/timeconst.h' failed
- make 3.81 works for v2.6.28.
  3.81 almost works with current HEAD, you need to run make twice because first run spuriously fails with:
- v2.6.28 with gcc-4.9.4 broken with:
  include/linux/kvm.h:240:9: error: duplicate member ‘padding’
- but even defconfig fails:
  VDSO    arch/x86/vdso/vdso.so.dbg
  gcc: error: elf_x86_64: No such file or directory
  gcc: error: unrecognized command line option ‘-m’
  It seems that we also need old binutils.
- for v3.8 and below we need perl-5.14.4.
  Unfortunately this or any manually built perl doesn't work for later kernels:
  Can't locate strict.pm in @INC
- kernels starting from 4.14 and older are boot broken:
  https://lkml.org/lkml/2018/9/7/648
- kernels older than 4.12 are broken during netdev setup
  (fixed by commit 675c8da049fd6556eb2d6cdd745fe812752f07a8)

Update #501
2019-03-17 18:06:44 +01:00
..
ast sys/linux: extend AX25/ROSE/NETROM descriptions 2018-12-24 18:56:06 +01:00
bisect pkg/bisect: various improvements 2019-03-17 18:06:44 +01:00
build all: switch to Go 1.12 2019-03-14 14:55:59 +01:00
compiler pkg/compiler: fix potential nil deref 2019-03-06 16:54:38 +01:00
config pkg/mgrconfig: do not serialize optional fields 2019-01-16 14:49:47 +01:00
cover syz-manager: move coverage report code to pkg/cover 2018-12-10 16:37:01 +01:00
csource pkg/runtest: make tests pass on freebsd 2019-03-07 20:37:43 +01:00
db pkg/osutil: provide better Rename 2018-12-17 19:09:10 +01:00
email pkg/email, pkg/vcs: test that OpenBSD-style commits are handled 2018-10-11 19:47:32 +02:00
gce .gometalinter.json: enable gofmt 2018-07-31 12:16:54 +02:00
gcs syz-ci: upload coverage reports to GCS 2018-09-20 18:29:38 +02:00
hash pkg/hash: allow to hash multiple byte slices 2017-07-21 10:06:46 +02:00
host pkg/host: don't fail if CONFIG_FAIL_FUTEX is not enabled 2019-02-11 18:35:46 +01:00
html dashboard/app, syz-ci: bisection support 2019-03-17 18:06:44 +01:00
ifuzz pkg/ifuzz/generated: change package name to match path 2019-02-19 15:23:15 +01:00
instance pkg/instance: fix boot error detection 2019-03-17 18:06:44 +01:00
ipc execprog, stress, prog2c: unify flags to enable additional features 2019-03-05 14:30:10 +01:00
kd pkg/kd: add KD protocol decoder 2017-09-28 16:53:58 +02:00
log pkg/log: add VerboseWriter 2019-03-17 18:06:44 +01:00
mgrconfig pkg/mgrconfig: do not serialize optional fields 2019-01-16 14:49:47 +01:00
osutil pkg/osutil: return output with errors from Run 2019-03-17 18:06:44 +01:00
report pkg/report: improve page fault detection on OpenBSD 2019-03-15 09:08:49 +01:00
repro execprog, stress, prog2c: unify flags to enable additional features 2019-03-05 14:30:10 +01:00
rpctype syz-manager: factor out rpc serving part 2018-12-26 10:58:23 +01:00
runtest pkg/runtest: make tests pass on freebsd 2019-03-07 20:37:43 +01:00
serializer pkg/serializer: refactor 2018-08-02 16:57:31 +02:00
signal prog: add stronger fallback signal 2018-06-29 20:34:43 +02:00
symbolizer pkg/symbolizer: favor portable nm(1) format 2018-09-24 10:12:38 +02:00
vcs pkg/vcs: reset repo more 2019-03-17 18:06:44 +01:00