Checking in the generated descriptions files makes few things simpler,
but causes pain for pull requests: (1) PRs that touch descriptions
_always_ conflict, (2) PRs are large and harder to review,
(3) people sometimes forget to add auto-generated files.
The proposed way does not require us to hardcode lots of dependencies
in the Makefile (which is nice) and seem to work.
Let's see how it works.
The main contributor-visible consequence is that the auto-generated
files do not need to be checked-in now.
Credit for figuring the Makefile magic goes to @melver.
Fixes#1291
Add prog.Ref Type that serves as a proxy for real types
and allows to deduplicate Types in generated descriptions.
The Ref type is effectively an index in an array of types.
Just before serialization pkg/compiler replaces real types
with the Ref types and prepares corresponding array of real types.
When a Target is registered in prog package, we do the opposite
operation and replace Ref's with the corresponding real types.
This brings improvements across the board:
compiler memory consumption is reduced by 15%,
test building time by 25%, descriptions size by 33%.
Before:
$ du -h sys/linux/gen
54M sys/linux/gen
$ time GOMAXPROCS=1 go test -p=1 -c ./prog
real 0m54.200s
real 0m53.883s
$ time GOMAXPROCS=1 go install -p=1 ./tools/syz-execprog
real 0m27.911s
real 0m27.767s
$ TIME="%e %P %M" GOMAXPROCS=1 time go tool compile ./sys/linux/gen
20.59 100% 3200016
20.97 100% 3445976
20.25 100% 3209684
After:
$ du -h sys/linux/gen
36M sys/linux/gen
$ time GOMAXPROCS=1 go test -p=1 -c ./prog
real 0m42.290s
real 0m43.230s
$ time GOMAXPROCS=1 go install -p=1 ./tools/syz-execprog
real 0m24.337s
real 0m24.727s
$ TIME="%e %P %M" GOMAXPROCS=1 time go tool compile ./sys/linux/gen
19.11 100% 2764952
19.66 100% 2787624
19.35 100% 2749376
Update #1580
Introduce common infrastructure for describing and parsing attribute
instead of custom per-attribute code scattered across several locations.
Change align attribute syntax from the weird align_N to align[N].
This also allows to use literal constants as N.
Introduce notion of builtin constants.
Currently we have only PTR_SIZE, which is needed to replace
align_ptr with align[PTR_SIZE].
Make MakeMmap return more than 1 call.
This is a preparation for future changes.
Also remove addr/size as they are effectively
always the same and can be inferred from the target
(will also conflict with the future changes).
Also rename to MakeDataMmap to better represent
the new purpose: it's just some arbitrary mmap,
but rather mapping of the data segment.
If we have:
ioctl(fd fd, cmd int32)
ioctl$FOO(fd fd, cmd const[FOO])
Currently we assume that cmd size in ioctl$FOO is sizeof(void*).
However, we know that in ioctl it's specified as int32,
so we can infer that the actual syscall size is 4.
This massively reduces sizes of socket/setsockopt/getsockopt/ioctl
and some other syscalls, which is good because we now use physical
size in mutation/hints and some other places.
This will also enable not morphing ioctl's into other ioctl's.
Update #477
Update #502
If we are going to write all values, don't write field names.
This only increases size of generated files.
The change reduces size of generated files by 5.8%
(62870496-59410354=3460142 bytes saved).
They can't be a bitmask. This fixes important cases
of "0, 1" and "0, 1, 2" flags. Fix some descriptions
that added 0 to bitmasks explicitly (we should do it
automatically instead).
Will simplify runtime analysis of flags.
Also just no reason to make it more deterministic
and avoid unnecessary diffs in future if values are reordered.
Generate const[0] for flags without values and for flags
with a single value which is 0.
This is the intention in all existing cases (e.g. an enum with types
of something, but there is really only 1 type exists).
The port-based exception APIs have been deprecated on Fuchsia and will
be removed shortly. Delete them from the syscall definitions and
modify the Fuchsia executor to use the new channel-based APIs instead.
* sys/fuchsia: update zx_clock_get.
zx_clock_get was deprecated and replaced by zx_clock_get_new. In a
recent CL[0], they replaced the zx_clock_get by zx_clock_get_new and
moved all client. This commit updates syzkaller to use the new function.
[0]: https://fuchsia-review.googlesource.com/c/fuchsia/+/298575
* run make extract && make generate
This commit removes the fuchsia-power fidl definitions. The interface
doesn't have a service implementing it in fuchsia, and it is causing
issues in make extract.
Consolidating FIDL library build mappings in one place, so that it can
be used from extraction, and script invoking fidlgen. This also makes
code clearer, and provides a more natural path for evolutions / dealing
with oddities.
Minor doc update post fx command changes.
Using a build tag to exclude files for golangci-lint
reduces memory consumption (it does not parse them).
The naive attempt with skip-dirs did not work.
So add codeanalysis build tag and use it in auto-generated files.
Update #977
* sys/fuchsia: update all syscalls.
This commit modifies all the existing syscalls definitions to match more
closely the documentation in the Fuchsia repo.
* run make extract && make generate
Ptr type has special handling of direction (pointers are always input).
But buffer type missed this special case all the time.
Make buffer less special by aliasing to the ptr[array[int8]] type.
As the result buffer type can't have optional trailing "opt" attribute
because we don't have such support for templates yet.
Change such cases to use ptr type directly.
Fixes#1097
Apparently, sysroot/include doesn't exist in the build-zircon path
anymore. I changed the path in sys-extract to make it point to the
exported sdk include path.
I also ran make extract and make generate to add new fidl definitions.
TEST=ran make extract and make generate.
Recently the fuchsia team decided to merge all their subrepos into one
big git repo. This meant that the "zircon" repo doesn't exist anymore.
Instead almost everything is under the fuchsia repo.
This change updates all the reference I could find in the code that
point to the zircon and docs repo to make them point to the new fuchsia
repo.
Garnet-layer FIDL descriptions are in a new subdirectory.
This CL changes it to the proper directory, and performed a "make
extract" shortly after.
This should fix some of the errors on the Syzkaller hub.
* sys/fuchsia: fix fidlgen issues.
This change modifies fidlgen removing fuchsia.mediacodec and renaming
zircon-ethernet to fuchsia-hardware-ethernet.
fuchsia.mediacodec has most of its functionality merged into
fuchsia.media and it's not included in the default amd64 builds.
zircon-ethernet has been renamed to fuchsia-hardware-ethernet
recently:
https://fuchsia-review.googlesource.com/c/zircon/+/237178
* docs/fuchsia: Add troubleshooting sections
This commit adds a troubleshooting section in the fuchsia syzkaller docs
that mentions how to solve fidlgen issues when fidl interfaces are
removed or renamed.
* sys/fuchsia: remove stale autogenerated files.
* sys/fucsia run make extract && make generate.
Change TODO comments that refer to missing description
language features to NEED comments as that it's easier to grep for them.
Remove some outdated comments. Fix 2 easy fixable TODOs.
This commit fixes two issues related to the task syscalls.
The zx_task_resume syscall has been recently removed from zircon[0]. It
has been deprecated for some time already. This commit removes the
syscall.
The `ZX_EXCEPTION_PORT_UNBIND_QUIETLY` option for the syscall
`zx_task_bind_exception_port` has been removed recently as well[1]. This
commit removes that option from tasks.txt.
To test this change, I followed the procedure for building syzkaller for
fuchsia:
```shell
$ make extract TARGETOS=fuchsia SOURCEDIR=${FUCHSIADIR}
$ make generate
$ make TARGETOS=fuchsia TARGETARCH=amd64 SOURCEDIR=${FUCHSIADIR}
```
I excluded the changes from make extract. This commit only has the
generated files from make generate.
[0]: https://fuchsia-review.googlesource.com/c/zircon/+/228712
[1]: https://fuchsia-review.googlesource.com/c/zircon/+/228658
* sys/fuchsia: update vmar syscalls.
In a previous zircon commit[0], the vmar related syscalls (like
`zx_vmar_map`, `zx_vmar_protect` and `zx_vmar_allocate`) changed the
order of their parameters, making putting the flags parameter as the
second parameter, and renaming it to "options".
This commit modifies vmars.txt so that it reflects the latest state of
the syscalls in zircon. I also modified the usage in
`executor/common_fuchsia.h`
I ran make extract, make generate and compiled syzkaller to test this
change.
[0]: https://fuchsia-review.googlesource.com/c/zircon/+/168060
* sys/fuchsia run make generate
This commit is just the result of running make generate after its
parent. This regenerates the definitions for the modified VMAR syscalls.
This commit is just a run of make extract + make generate with an up to
date fuchsia tree. This commit only updates the fidl definitions, there
are some other syscalls that had changed and need to be manually updated
(that will come in a future commit).
* pkg/compiler: Add error handler in `CollectUnused`.
This commit adds an error handler for the `CollectUnused` function. The
error handler just panics on any error, but is useful for debugging.
The error handler is used any time `comp` finds an error, and if it's
missing, it will panic due to a `nil` pointer dereference. At least now
we get a better understanding of the errors.
The only user of `CollectUnused` is `sys/fuchsia/fidlgen`, which is
failing now and will be fixed in a future commit.
The output message looks like this:
```
panic: could not collect unused nodes. fidl_net-stack.txt:110:15:
unknown type zx_chan_zircon_ethernet_Device_client
```
* pkg/compiler Better error handling in CollectUnused
This commit changes the default error handler for compiler to
`ast.LoggingHandler`, meaning that if `nil` is passed as an error
handler, `LoggingHandler` will be used instead.
`CollectUnused` now returns an error if any of the subfunctions produce errors.
`fidlgen` is the only caller of `CollectUnused`, and now checks for errors
as well.
* pkg/compiler Add tests for CollectUnused
This commit adds basic tests for the CollectUnused function. There's one
test that checks that it returns the right nodes, and another one that
makes sure that it returns errors when needed.
To make the test clearer, I had to add the error handler as an explicit
parameter in `CollectUnunsed`, instead of using the default one. This
avoid printing garbage in the logs. The `TestCollectUnusedError` function
uses a nopErrorHandler to avoid printing anything.
* pkg/compiler fix presubmit warnings
This commits adds "zircon-ethernet" to the list of fidl libraries that
fidlgen would look at. This is required because recently, a change in
fuchsia[0] added a new method to the netstack interface, that allows for
adding an EthernetInterface. One of the parameters is a
zircon.ethernet.Device, so we need to import that type for fidlgen to be
able to understand fidl_net-stack.txt.
I tested this by running:
```shell
$ make extract TARGETOS=fuchsia SOURCEDIR=${FUCHSIADIR}
$ make generate
```
[0]: https://fuchsia-review.googlesource.com/c/zircon/+/220580
Anytime we're running `make extract/generate` with Fuchsia as a target, we
probably want to update FIDL descriptions too, so we don't fall behind
what upstream is up to.
This includes generate_fidl as part of the `make extract` workflow.
Also, corrects fidlgenPath to properly find the path for ARM targets (we
only have host_x64 right now).
After generating syscall description for fidl files using fidlgen, prune
all unused structs using the exact same mechanism used by the compiler's
check for unused structs. This allows the FIDL compiler to support
modular compilation; it does not need to have global knowledge of
whether each struct is used or not.
Add simple fuchsia program, the one that is run during image testing.
Fix csource errno printing for fuchsia.
Fix creation of executable files (chmod is not implemented on fuchsia).
Check that we get signal/coverage from all syscalls.
* create a response handle structure for each FIDL method
* generate zx_channel_write for requests having no response
* generate zx_channel_read for events (no request but just response)
* translate FIDL union types to syzkaller union types
(see fuchsia_io_ObjectInfoHandles)
