syzkaller - linux syscall fuzzer

syzkaller is a distributed, unsupervised, coverage-guided Linux syscall fuzzer. It is meant to be used with KASAN (CONFIG_KASAN=y), KTSAN (CONFIG_KTSAN=y), or [KUBSAN] (http://developerblog.redhat.com/2014/10/16/gcc-undefined-behavior-sanitizer-ubsan/) (patch).

Project mailing list: syzkaller@googlegroups.com, which you can subscribe to either with an google account or by sending an email to syzkaller+subscribe@googlegroups.com.

List of found bugs.

This is work-in-progress, some things may not work yet.

Usage

Various components are needed to build and run syzkaller.

• C compiler with coverage support
• Linux kernel with coverage additions
• QEMU and disk image
• The syzkaller components

Setting each of these up is discussed in the following sections.

C Compiler

Syzkaller is a coverage-guided fuzzer and so needs the kernel to be built with coverage support. Therefore, a recent upstream version of GCC is needed. Coverage support is submitted to gcc in revision 231296, released in gcc6.

Linux Kernel

As well as adding coverage support to the C compiler, the Linux kernel itself needs to be modified to:

• add support in the build system for the coverage options (under CONFIG_KCOV)
• add extra instrumentation on system call entry/exit (for a CONFIG_KCOV build)
• add code to track and report per-task coverage information.

KCOV is upstreamed in linux 4.6. For older kernels you need to backport commit 5c9a8750a6409c63a0f01d51a9024861022f6593. The kernel should be configured with CONFIG_KCOV plus CONFIG_KASAN or CONFIG_KTSAN.

(Note that if the kernel under test does not include support for all namespaces, the dropprivs configuration value should be set to false.)

QEMU Setup

Syzkaller runs its fuzzer processes inside QEMU virtual machines, so a working QEMU system is needed – see QEMU docs for details.

In particular:

• The fuzzing processes communicate with the outside world, so the VM image needs to include networking support.
• The program files for the fuzzer processes are transmitted into the VM using SSH, so the VM image needs a running SSH server.
• The VM's SSH configuration should be set up to allow root access for the identity that is included in the syz-manager's configuration. In other words, you should be able to do ssh -i $SSHID -p $PORT root@localhost without being prompted for a password (where SSHID is the SSH identification file and PORT is the port that are specified in the syz-manager configuration file).
• The kernel exports coverage information via a debugfs entry, so the VM image needs to mount the debugfs filesystem at /sys/kernel/debug.

create-image.sh script can be used to create a suitable Linux image.

TODO: Describe how to support other types of VM other than QEMU.

Syzkaller

The syzkaller tools are written in Go, so a Go compiler (>= 1.4) is needed to build them. Build with make, which generates compiled binaries in the bin/ folder.

Configuration

The operation of the syzkaller syz-manager process is governed by a configuration file, passed at invocation time with the -config option. This configuration can be based on the syz-manager/example.cfg; the file is in JSON format with the following keys in its top-level object:

• http: URL that will display information about the running syz-manager process.
• workdir: Location of a working directory for the syz-manager process. Outputs here include:
  • <workdir>/instance-x: per VM instance temporary files
  • <workdir>/crashes/crashN-T: crash output files
  • <workdir>/corpus/*: corpus with interesting programs
• syzkaller: Location of the syzkaller checkout.
• vmlinux: Location of the vmlinux file that corresponds to the kernel being tested.
• type: Type of virtual machine to use, e.g. qemu or kvm.
• count: Number of VMs to run in parallel.
• procs: Number of parallel test processes in each VM (4 or 8 would be a reasonable number).
• leak: Detect memory leaks with kmemleak (very slow).
• kernel: Location of the bzImage file for the kernel to be tested; this is passed as the -kernel option to qemu-system-x86_64.
• cmdline: Additional command line options for the booting kernel, for example root=/dev/sda1.
• image: Location of the disk image file for the QEMU instance; a copy of this file is passed as the -hda option to qemu-system-x86_64.
• sshkey: Location (on the host machine) of an SSH identity to use for communicating with the virtual machine.
• cpu: Number of CPUs to simulate in the VM (not currently used).
• mem: Amount of memory (in MiB) for the VM; this is passed as the -m option to qemu-system-x86_64.
• dropprivs : Whether the executor program should try to use namespaces to drop privileges before executing (requires a kernel built with CONFIG_NAMESPACES, CONFIG_UTS_NS, CONFIG_USER_NS, CONFIG_PID_NS and CONFIG_NET_NS).
• enable_syscalls: List of syscalls to test (optional).
• disable_syscalls: List of system calls that should be treated as disabled (optional).
• suppressions: List of regexps for known bugs.

Running syzkaller

Start the syz-manager process as:

./bin/syz-manager -config my.cfg

The -config command line option gives the location of the configuration file described above.

The syz-manager process will wind up qemu virtual machines and start fuzzing in them. It also reports some statistics on the HTTP address.

Process Structure

The process structure for the syzkaller system is shown in the following diagram; red labels indicate corresponding configuration options.

The syz-manager process starts, monitors and restarts several VM instances (support for physical machines is not implemented yet), and starts a syz-fuzzer process inside of the VMs. It is responsible for persistent corpus and crash storage. As opposed to syz-fuzzer processes, it runs on a host with stable kernel which does not experience white-noise fuzzer load.

The syz-fuzzer process runs inside of presumably unstable VMs (or physical machines under test). The syz-fuzzer guides fuzzing process itself (input generation, mutation, minimization, etc) and sends inputs that trigger new coverage back to the syz-manager process via RPC. It also starts transient syz-executor processes.

Each syz-executor process executes a single input (a sequence of syscalls). It accepts the program to execute from the syz-fuzzer process and sends results back. It is designed to be as simple as possible (to not interfere with fuzzing process), written in C++, compiled as static binary and uses shared memory for communication.

Syscall description

syzkaller uses declarative description of syscalls to generate, mutate, minimize, serialize and deserialize programs (sequences of syscalls). Below you can see (hopefully self-explanatory) excerpt from the description:

open(file filename, flags flags[open_flags], mode flags[open_mode]) fd
read(fd fd, buf buffer[out], count len[buf]) len[buf]
close(fd fd)
open_mode = S_IRUSR, S_IWUSR, S_IXUSR, S_IRGRP, S_IWGRP, S_IXGRP, S_IROTH, S_IWOTH, S_IXOTH

The description is contained in sys/sys.txt file.

Troubleshooting

Here are some things to check if there are problems running syzkaller.

• Check that QEMU can successfully boot the virtual machine. For example, if IMAGE is set to the VM's disk image (as per the image config value) and KERNEL is set to the test kernel (as per the kernel config value) then something like the following command should start the VM successfully:

  ```qemu-system-x86_64 -hda $IMAGE -m 256 -net nic -net user,host=10.0.2.10,hostfwd=tcp::23505-:22 -enable-kvm -kernel $KERNEL -append root=/dev/sda```
  

• Check that inbound SSH to the running virtual machine works. For example, with a VM running and with SSHKEY set to the SSH identity (as per the sshkey config value) the following command should connect:

  ```ssh -i $SSHKEY -p 23505 root@localhost```
  

• Check that the CONFIG_KCOV option is available inside the VM:

  • ls /sys/kernel/debug # Check debugfs mounted
  • ls /sys/kernel/debug/kcov # Check kcov enabled
  • Build the test program from Documentation/kcov.txt and run it inside the VM.

• Check that debug information (from the CONFIG_DEBUG_INFO option) is available

  • Pass the hex output from the kcov test program to addr2line -a -i -f -e $VMLINUX (where VMLINUX is the vmlinux file, as per the vmlinux config value), to confirm that symbols for the kernel are available.

• Use the -v N command line option to increase the amount of logging output, from both the syz-manager top-level program and the syz-fuzzer instances (which go to the output files in the crashes subdirectory of the working directory). Higher values of N give more output.

• If logging indicates problems with the executor program (e.g. executor failure), try manually running a short sequence of system calls:

  • Build additional tools with make all-tools
  • Copy syz-executor and syz-execprog into a running VM.
  • In the VM run ./syz-execprog -executor ./syz-executor -debug sampleprog where sampleprog is a simple system call script (e.g. just containing getpid()).
  • For example, if this reports that clone has failed, this probably indicates that the test kernel does not include support for all of the required namespaces. In this case, running the syz-execprog test with the -nobody=0 option fixes the problem, so the main configuration needs to be updated to set dropprivs to false.

Fuzzing new system calls

This section describes how to extend syzkaller to allow fuzz testing of a new system call; this is particularly useful for kernel developers who are proposing new system calls.

First, add a declarative description of the new system call to the appropriate file:

• Various sys/<subsystem>.txt files hold system calls for particular kernel subsystems, for example bpf or socket.
• sys/sys.txt holds descriptions for more general system calls.
• An entirely new subsystem can be added as a new sys/<new>.txt file, but needs the generate target in the Makefile to be updated to include it.

The description format is described above and in the master sys/sys.txt file.

Next, run make LINUX=$KSRC generate with KSRC set to the location of a kernel source tree (for up to date kernel headers); if the kernel was built into a separate directory (with make O=...) then also set LINUXBLD=$KBLD to the location of the build directory.

This will re-create the following source code files:

• sys/sys.go: Code to initialize a Go data structure with information about all of the available system calls.
• prog/consts.go: Constant definitions for all the named constants that are mentioned in the system call descriptions.
• sys/sys_<ARCH>.go: Data structure to map syzkaller internal syscall IDs to (per-architecture) kernel syscall numbers.
• executor/syscalls.h: Constant definitions (in C) for all system call numbers.

If there are problems with this step, run bin/syz-sysgen directly and add the use -v=5 flag to show more details of the generation process.

Rebuild syzkaller (make clean all) to force use of the new system call definitions.

Finally, adjust the enable_syscalls configuration value for syzkaller to specifically target the new system calls.

Disclaimer

This is not an official Google product.