syzkaller/sys/fuchsia
Dmitry Vyukov 0ce7569ee7 pkg/compiler: deduplicate Types in descriptions
Add prog.Ref Type that serves as a proxy for real types
and allows to deduplicate Types in generated descriptions.
The Ref type is effectively an index in an array of types.
Just before serialization pkg/compiler replaces real types
with the Ref types and prepares corresponding array of real types.
When a Target is registered in prog package, we do the opposite
operation and replace Ref's with the corresponding real types.

This brings improvements across the board:
compiler memory consumption is reduced by 15%,
test building time by 25%, descriptions size by 33%.

Before:
$ du -h sys/linux/gen
54M	sys/linux/gen

$ time GOMAXPROCS=1 go test -p=1 -c ./prog
real	0m54.200s
real	0m53.883s

$ time GOMAXPROCS=1 go install -p=1 ./tools/syz-execprog
real	0m27.911s
real	0m27.767s

$ TIME="%e %P %M" GOMAXPROCS=1 time go tool compile ./sys/linux/gen
20.59 100% 3200016
20.97 100% 3445976
20.25 100% 3209684

After:
$ du -h sys/linux/gen
36M	sys/linux/gen

$ time GOMAXPROCS=1 go test -p=1 -c ./prog
real	0m42.290s
real	0m43.230s

$ time GOMAXPROCS=1 go install -p=1 ./tools/syz-execprog
real	0m24.337s
real	0m24.727s

$ TIME="%e %P %M" GOMAXPROCS=1 time go tool compile ./sys/linux/gen
19.11 100% 2764952
19.66 100% 2787624
19.35 100% 2749376

Update #1580
2020-04-26 05:58:31 +02:00
..
fidlgen sys: Updating Fuchsia extraction scripts 2019-05-21 14:55:36 -07:00
gen pkg/compiler: deduplicate Types in descriptions 2020-04-26 05:58:31 +02:00
layout sys/fuchsia: fix Fuchsia extract/generate 2019-11-14 16:21:01 -08:00
test sys/fuchsia: Add cprng and vmo tests. 2020-03-21 07:02:15 +01:00
channels_amd64.const sys/fuchsia: describe more syscalls 2017-09-25 08:47:48 +02:00
channels_arm64.const sys/fuchsia: describe more syscalls 2017-09-25 08:47:48 +02:00
channels.txt Update syscalls (#1116) 2019-04-22 11:08:29 -07:00
ddk_amd64.const sys/fuchsia: Update Fuchsia syscalls. 2018-07-07 13:56:36 +02:00
ddk_arm64.const sys/fuchsia: Update Fuchsia syscalls. 2018-07-07 13:56:36 +02:00
ddk.txt tools: add script that checks copyright headers 2020-02-18 16:05:10 +01:00
events.txt sys/fuchsia: fix Fuchsia extract/generate 2019-11-14 16:21:01 -08:00
fidl_amd64.const sys/fuchsia: semi-automatically generated FIDL launcher description 2018-08-17 14:31:53 -07:00
fidl_arm64.const sys/fuchsia: semi-automatically generated FIDL launcher description 2018-08-17 14:31:53 -07:00
fidl_cobalt.syz_amd64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fidl_cobalt.syz_arm64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fidl_devicesettings.syz_amd64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fidl_devicesettings.syz_arm64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fidl_hardware-ethernet.syz_amd64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fidl_hardware-ethernet.syz_arm64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fidl_io.syz_amd64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fidl_io.syz_arm64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fidl_ldsvc.syz_amd64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fidl_ldsvc.syz_arm64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fidl_net-stack.syz_amd64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fidl_net-stack.syz_arm64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fidl_net.syz_amd64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fidl_net.syz_arm64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fidl_power.syz_amd64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fidl_power.syz_arm64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fidl_process.syz_amd64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fidl_process.syz_arm64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fidl_scpi.syz_amd64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fidl_scpi.syz_arm64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fidl_timezone.syz_amd64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fidl_timezone.syz_arm64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fidl.txt pkg/compiler: refactor attribute handling 2020-04-19 10:26:57 +02:00
fifos.txt Update syscalls (#1116) 2019-04-22 11:08:29 -07:00
fuchsia_cobalt.syz_amd64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fuchsia_cobalt.syz_arm64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fuchsia_cobalt.syz.txt sys/fuchsia: fix Fuchsia extract/generate 2019-11-14 16:21:01 -08:00
fuchsia_devicesettings.syz_amd64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fuchsia_devicesettings.syz_arm64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fuchsia_devicesettings.syz.txt sys/fuchsia: fix Fuchsia extract/generate 2019-11-14 16:21:01 -08:00
fuchsia_hardware_ethernet.syz_amd64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fuchsia_hardware_ethernet.syz_arm64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fuchsia_hardware_ethernet.syz.txt sys/fuchsia: fix Fuchsia extract/generate 2019-11-14 16:21:01 -08:00
fuchsia_io.syz_amd64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fuchsia_io.syz_arm64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fuchsia_io.syz.txt sys/fuchsia: fix Fuchsia extract/generate 2019-11-14 16:21:01 -08:00
fuchsia_ldsvc.syz_amd64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fuchsia_ldsvc.syz_arm64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fuchsia_ldsvc.syz.txt sys/fuchsia: fix Fuchsia extract/generate 2019-11-14 16:21:01 -08:00
fuchsia_mem.syz.txt sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fuchsia_net_stack.syz_amd64.const sys/fuchsia: fix Fuchsia extract/generate 2019-11-14 16:21:01 -08:00
fuchsia_net_stack.syz_arm64.const sys/fuchsia: fix Fuchsia extract/generate 2019-11-14 16:21:01 -08:00
fuchsia_net_stack.syz.txt sys/fuchsia: fix Fuchsia extract/generate 2019-11-14 16:21:01 -08:00
fuchsia_net.syz_amd64.const run make extract and make generate for fuchsia 2019-07-16 16:34:29 +02:00
fuchsia_net.syz_arm64.const run make extract and make generate for fuchsia 2019-07-16 16:34:29 +02:00
fuchsia_net.syz.txt sys/fuchsia: fix Fuchsia extract/generate 2019-11-14 16:21:01 -08:00
fuchsia_process.syz_amd64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fuchsia_process.syz_arm64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fuchsia_process.syz.txt sys/fuchsia: fix Fuchsia extract/generate 2019-11-14 16:21:01 -08:00
fuchsia_scpi.syz_amd64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fuchsia_scpi.syz_arm64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fuchsia_scpi.syz.txt sys/fuchsia: fix Fuchsia extract/generate 2019-11-14 16:21:01 -08:00
fuchsia_timezone.syz_amd64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fuchsia_timezone.syz_arm64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
fuchsia_timezone.syz.txt sys/fuchsia: fix Fuchsia extract/generate 2019-11-14 16:21:01 -08:00
futexes.txt Update syscalls (#1116) 2019-04-22 11:08:29 -07:00
handles_amd64.const sys/fuchsia: describe more syscalls 2017-09-25 08:47:48 +02:00
handles_arm64.const sys/fuchsia: describe more syscalls 2017-09-25 08:47:48 +02:00
handles.txt Update syscalls (#1116) 2019-04-22 11:08:29 -07:00
hypervisor_guests_amd64.const sys/fuchsia: Update Fuchsia syscalls. 2018-07-07 13:56:36 +02:00
hypervisor_guests_arm64.const sys/fuchsia: Update Fuchsia syscalls. 2018-07-07 13:56:36 +02:00
hypervisor_guests.txt tools: add script that checks copyright headers 2020-02-18 16:05:10 +01:00
init.go prog: refactor target.MakeMmap 2020-04-18 14:35:45 +02:00
jobs_amd64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
jobs_arm64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
jobs.txt sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
log_amd64.const sys/fuchsia: describe more syscalls 2017-09-25 08:47:48 +02:00
log_arm64.const sys/fuchsia: describe more syscalls 2017-09-25 08:47:48 +02:00
log.txt prog: add ignore_return and breaks_returns call attribtues 2020-04-19 10:26:57 +02:00
objects_amd64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
objects_arm64.const sys/fuchsia: Updating Fuchsia definitions 2019-05-21 14:55:36 -07:00
objects.txt sys/fuchsia: remove deprecated exception APIs 2019-11-14 16:21:01 -08:00
ports_amd64.const sys/fuchsia: describe more syscalls 2017-09-25 08:47:48 +02:00
ports_arm64.const sys/fuchsia: describe more syscalls 2017-09-25 08:47:48 +02:00
ports.txt Update syscalls (#1116) 2019-04-22 11:08:29 -07:00
posix_amd64.const sys/fuchsia: Update Fuchsia syscalls. 2018-07-07 13:56:36 +02:00
posix_arm64.const sys/fuchsia: Update Fuchsia syscalls. 2018-07-07 13:56:36 +02:00
posix.txt pkg/compiler: check for unused declarations 2018-06-30 19:34:41 +02:00
processes.txt Update syscalls (#1116) 2019-04-22 11:08:29 -07:00
sockets_amd64.const sys/fuchsia: update socket syscalls 2019-08-14 09:34:25 +02:00
sockets_arm64.const sys/fuchsia: update socket syscalls 2019-08-14 09:34:25 +02:00
sockets.txt sys/fuchsia: update socket syscalls 2019-08-14 09:34:25 +02:00
sys.txt Update syscalls (#1116) 2019-04-22 11:08:29 -07:00
tasks_amd64.const sys/fuchsia: remove deprecated exception APIs 2019-11-14 16:21:01 -08:00
tasks_arm64.const sys/fuchsia: remove deprecated exception APIs 2019-11-14 16:21:01 -08:00
tasks.txt sys/fuchsia: remove deprecated exception APIs 2019-11-14 16:21:01 -08:00
threads.txt Update syscalls (#1116) 2019-04-22 11:08:29 -07:00
time_amd64.const sys/fuchsia: Remove read_status from zx_channel_call args, and reflect zx_time_t type change (unsigned -> signed) 2018-08-24 20:01:29 -07:00
time_arm64.const sys/fuchsia: description for fuchsia.io FIDL interfaces 2018-08-27 14:01:34 -07:00
time.txt prog: add ignore_return and breaks_returns call attribtues 2020-04-19 10:26:57 +02:00
timers_amd64.const sys/fuchsia: describe more syscalls 2017-09-25 08:47:48 +02:00
timers_arm64.const sys/fuchsia: describe more syscalls 2017-09-25 08:47:48 +02:00
timers.txt Update syscalls (#1116) 2019-04-22 11:08:29 -07:00
vmars_amd64.const sys/fuchsia: add more descriptions 2017-10-02 10:53:28 +02:00
vmars_arm64.const sys/fuchsia: add more descriptions 2017-10-02 10:53:28 +02:00
vmars.txt sys/fuchsia: update VMAR syscalls 2018-11-21 05:56:43 +01:00
vmos_amd64.const run make extract and make generate for fuchsia 2019-07-16 16:34:29 +02:00
vmos_arm64.const run make extract and make generate for fuchsia 2019-07-16 16:34:29 +02:00
vmos.txt sys/fuchsia: remove RESIZEABLE flags from vmo ops. 2019-07-16 16:34:29 +02:00