This is useful for integrating into Fuchsia's build system, where we
need to be able to run syz-sysgen with a read-only source directory,
and emit the output files elsewhere.
Currently ANY implementation fabricates new types dynamically.
This is something we don't do anywhere else, generally types
come from compiler and all are static.
Dynamic types will conflict with use of Ref in Arg optimization.
Move ANY types creation into compiler.
Update #1580
Remove StructDesc, KeyedStruct, StructKey and all associated
logic/complexity in prog and pkg/compiler.
We can now handle recursion more generically with the Ref type,
and Dir/FieldName are not a part of the type anymore.
This makes StructType/UnionType simpler and more natural.
Reduces size of sys/linux/gen/amd64.go from 5201321 to 4180861 (-20%).
Update #1580
Add prog.Ref Type that serves as a proxy for real types
and allows to deduplicate Types in generated descriptions.
The Ref type is effectively an index in an array of types.
Just before serialization pkg/compiler replaces real types
with the Ref types and prepares corresponding array of real types.
When a Target is registered in prog package, we do the opposite
operation and replace Ref's with the corresponding real types.
This brings improvements across the board:
compiler memory consumption is reduced by 15%,
test building time by 25%, descriptions size by 33%.
Before:
$ du -h sys/linux/gen
54M sys/linux/gen
$ time GOMAXPROCS=1 go test -p=1 -c ./prog
real 0m54.200s
real 0m53.883s
$ time GOMAXPROCS=1 go install -p=1 ./tools/syz-execprog
real 0m27.911s
real 0m27.767s
$ TIME="%e %P %M" GOMAXPROCS=1 time go tool compile ./sys/linux/gen
20.59 100% 3200016
20.97 100% 3445976
20.25 100% 3209684
After:
$ du -h sys/linux/gen
36M sys/linux/gen
$ time GOMAXPROCS=1 go test -p=1 -c ./prog
real 0m42.290s
real 0m43.230s
$ time GOMAXPROCS=1 go install -p=1 ./tools/syz-execprog
real 0m24.337s
real 0m24.727s
$ TIME="%e %P %M" GOMAXPROCS=1 time go tool compile ./sys/linux/gen
19.11 100% 2764952
19.66 100% 2787624
19.35 100% 2749376
Update #1580
Add common infrastructure for syscall attributes.
Add few attributes we want, but they are not implemented for now
(don't affect behavior, this will follow).
Ensure that we don't have conflicting sizes for the same argument
of the same syscall, e.g.:
foo$1(a int16)
foo$2(a int32)
This is useful for several reasons:
- we will be able avoid morphing syscalls into other syscalls
- we will be able to figure out more precise sizes for args
(lots of them are implicitly intptr, which is the largest
type on most important arches)
- found few bugs in linux descriptions
Update #477
Update #502
Currently we print them as part of `make genereate`,
but nobody reads them, too much output each time.
Don't print them in `make generate` and instead
print in syz-check, the warn files are a good mechanism
to handle "known warnings".
Using a build tag to exclude files for golangci-lint
reduces memory consumption (it does not parse them).
The naive attempt with skip-dirs did not work.
So add codeanalysis build tag and use it in auto-generated files.
Update #977
Currently target binaries contain support for all OS/arch combinations.
However, obviously a fuchsia target binary won't test windows.
For target binaries we need support only for a single target
(with the exception of 386/arm target in amd64/arm64 binaries).
So compile in only _the_ target into target binaries.
This reduces akaros/amd64 fuzzer binary from 33 to 7 MB
and execprog from 28 to 2 MB.
Make as much code as possible shared between all OSes.
In particular main is now common across all OSes.
Make more code shared between executor and csource
(in particular, loop function and threaded execution logic).
Also make loop and threaded logic shared across all OSes.
Make more posix/unix code shared across OSes
(e.g. signal handling, pthread creation, etc).
Plus other changes along similar lines.
Also support test OS in executor (based on portable posix)
and add 4 arches that cover all execution modes
(fork server/no fork server, shmem/no shmem).
This change paves way for testing of executor code
and allows to preserve consistency across OSes and executor/csource.
We have fallback coverage implmentation for freebsd.
1. It's broken after some recent changes.
2. We need it for fuchsia, windows, akaros, linux too.
3. It's painful to work with C code.
Move fallback coverage to ipc package,
fix it and provide for all OSes.
We see some crashes that suggest corruption of the syscall number:
invalid command number 1296 (errno 11)
invalid command number 107 (errno 110)
Make the table and the number constant to prevent corruption.
Move generated files to gen subdir. This allows to:
1. Rebuild init.go without rebuilding generated code.
2. Excluding generated files from gometalinter checking.
This makes faster and consume less memory.
Update #538
1. mmap all memory always, without explicit mmap calls in the program.
This makes lots of things much easier and removes lots of code.
Makes mmap not a special syscall and allows to fuzz without mmap enabled.
2. Change address assignment algorithm.
Current algorithm allocates unmapped addresses too frequently
and allows collisions between arguments of a single syscall.
The new algorithm analyzes actual allocations in the program
and places new arguments at unused locations.
This breaks circular dependency between:
sysgen -> sys/linux -> sys -> sysgen
With this circular dependency it is very difficult to
update format of generated descriptions because sysgen does not build.
Info about targets (like C arch/CFLAGS) is required in multiple places:
extract.sh
syz-extract
syz-sysgen
csource
vm/qemu
...
Consolidate it in one place and use that in syz-sysgen.
Large overhaul moves syscalls and arg types from sys to prog.
Sys package now depends on prog and contains only generated
descriptions of syscalls.
Introduce prog.Target type that encapsulates all targer properties,
like syscall list, ptr/page size, etc. Also moves OS-dependent pieces
like mmap call generation from prog to sys.
Update #191
All other types have optional underlying type as the last argument.
Proc have it as first. This creates very irregular syntax and
complicates parsing. Make type the last argument.
Note: this is a breaking change if you have descriptions on the side.
