For Stylo development, LLVM packages are required due to Stylo's
extensive use of bindgen--generating Rust bindings to Gecko's C++ code.
While people can install LLVM via their system package manager, we've
opted to download the LLVM packages used on Mozilla infrastructure for
building Gecko. Using Mozilla's packages for LLVM ensures that they
work, and also makes it easier/trivial to integrate support for other
things (e.g. Mozilla's static checkers) into `mach bootstrap`.
The way it works today, compiler warnings are logged to build output
twice. The compiler's raw output is logged. Then, if mach's compiler
warning parser detects a warning, a structured log message with info
about that warning is printed as well. Because of the order in which
callbacks fire, mach's warning message is delivered to the logger
first. So build output looks something like:
0:04.63 Warning: -Wsign-compare in /home/gps/src/firefox/security/nss/lib/dev/ckhelper.c: comparison of integers of different signs: 'CK_ULONG' (aka 'unsigned long') and 'int'
0:04.63 /home/gps/src/firefox/security/nss/lib/dev/ckhelper.c:135:45: warning: comparison of integers of different signs: 'CK_ULONG' (aka 'unsigned long') and 'int' [-Wsign-compare]
0:04.63 (obj_template[i].ulValueLen == -1)) {
0:04.63 ~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~
That first line is our structured message formatted to plain text.
Subsequent lines are Clang.
The output here is redundant. But having the structured compiler warning
in the JSON logs is nice so downstream systems don't have to reinvent
the parsing wheel.
Also today, at the end of a clobber build we print a count of the
total number of compiler warnings.
Compiler warnings represent potential bugs. So we want to encourage
people to look at them. However, with build system output often spanning
several hundred lines and compiler warnings scattered in no
deterministic order because build system execution is non-deterministic,
it is easy to lose sight of compiler warnings as they go by in build
output. And, it can be difficult to find a warning in files you care
about because you don't know where in the log to look for that file.
This commit attempts to improve the visibility of compiler warnings
by printing a sorted list of warnings at the end of the build. The list
of warnings it prints are only those seen during the current build
operation. We /could/ print a list of all warnings in the persisted
database. But I think this would be annoying, particularly for
incremental or partial builds.
The structured log message previously emitted at compiler invocation
time has been removed. So now the build backend output doesn't have a
redundant "Warning: " line summarizing the warning next to the compiler
output itself. This is less confusing.
That structured log message has been reborn at the end of the build
and reformatted slightly. If there are any consumers assuming that the
log entry near this structured message is compiler output itself,
they will break. I don't think any such consumers exist. It is also
possible the structured log may not contain warning messages if a
process exit occurs. I'm fine with this: if the `mach` process dies,
we have bigger problems to worry about.
The new output looks something like this:
18:07.97 warning: gfx/angle/src/compiler/translator/util.cpp:216:15 [-Wimplicit-fallthrough] unannotated fall-through between switch labels
18:07.97 warning: gfx/angle/src/compiler/translator/util.cpp:225:15 [-Wimplicit-fallthrough] unannotated fall-through between switch labels
18:07.97 warning: gfx/angle/src/compiler/translator/util.cpp:234:15 [-Wimplicit-fallthrough] unannotated fall-through between switch labels
18:07.97 warning: gfx/cairo/libpixman/src/pixman-bits-image.c:268:32 [-Wshift-negative-value] shifting a negative signed value is undefined
18:07.97 warning: gfx/cairo/libpixman/src/pixman-linear-gradient.c:395:6 [-Wunreachable-code] code will never be executed
18:07.97 warning: gfx/cairo/libpixman/src/pixman-x86.c:80:5 [-Wexpansion-to-defined] macro expansion producing 'defined' has undefined behavior
Note the sorting of files, including by line number and column. If we
wanted to, we could even print that line. But that's for another day
(especially since compilers themselves tend to do this already).
This change has the potential to annoy people. That's because instead
of the parsed warning summary lines being spread out over the build
(where they tend not to be noticed), they will be lumped at the end of
the build log, which people do tend to notice. In my build, I have
~363 warnings/lines. Some could perceive this as excessive and
redundant. But the recourse for this is simple: eliminate compiler
warnings. My hope is that by having the compiler warnings exposed
clearly at the end of the build log that people will notice them
and will take action to eliminate them. Sunlight is a disinfectant.
MozReview-Commit-ID: 9VHXsPhAYmr
--HG--
extra : rebase_source : 6a2da05f794a54ea54a48167a36d130dfbf34e59
Currently, the build monitor has a single compiler warnings database
that unions warnings from previous runs with warnings from the current
invocation. I want to introduce functionality that treats warnings
seen during the current invocation differently from "all warnings."
To facilitate this, this commit introduces a 2nd, non-persisted
warnings database to record warnings just for the current invocation.
MozReview-Commit-ID: FIY0GiarDmr
--HG--
extra : rebase_source : b2002e1c248ea65b2c0ee45a78b1e74d61a26f3c
Currently, the WarningsCollector (which parses warnings from line inputs)
accepts a WarningsDatabase (an object representing a collection of
warnings) and populates that instance as a new warning is parsed. In
an upcoming commit I want to introduce a 2nd WarningsDatabase.
Rather that add it to WarningsCollector, I figure it will be easier
to decouple WarningsDatabase from WarningsCollector.
This commit refactors WarningsCollector to call a callback when a
warning is parsed. This allows consumers to do anything they want
with a warning, including potentially write it to multiple databases.
MozReview-Commit-ID: 7Z9x4FMwyof
--HG--
extra : rebase_source : b4844b5c2b1926840d37e46ead38c8c358762ba8
mozharness is currently making a manual `make -k` invocation. We don't
want automation calling `make` directly. So teach `mach build` to
accept a --keep-going argument that results in `make -k`.
MozReview-Commit-ID: H3lJ4r8S4vj
--HG--
extra : rebase_source : 9feb7bcaeb855254c53c5fa9d49177c5133f2773
This patch adds a copy of vswhere.exe to build/win32, downloaded from the
current latest release (1.0.62):
https://github.com/Microsoft/vswhere/releases/download/1.0.62/vswhere.exe
It changes toolchain.configure to invoke vswhere.exe instead of reading
the registry, since that no longer works for VS2017 (but vswhere can locate
VS2015). It also removes a layer of complexity in that code by dropping
support for non-64-bit host systems, since we don't really support building
on 32-bit Windows anymore anyway.
There's a little bit of fixup in windows.configure where some LIB paths
have changed in 2017.
MozReview-Commit-ID: 5XLWjidS6W4
--HG--
extra : rebase_source : 90f79b6f4a2d8d925dd20eb0bf6ab96262c227d5
rustc generates .lib files for its libraries when compiling for Windows
(even using MinGW on Linux). But MinGW expects .a files. So we add in
rust-specific prefix and suffixes so MinGW builds can find the libs that
rustc generates. (And the RUST_LIB- variables default to the same vales
as the LIB_ variables otherwise.)
MozReview-Commit-ID: ClsA0YuJaxh
--HG--
extra : rebase_source : 7b46460c94ceb34b7a5a302ce91d3f1dca600041
Everything depending on the widget being gonk can go away, as well as
everything depending on MOZ_AUDIO_CHANNEL_MANAGER, which was only
defined on gonk builds under b2g/ (which goes away in bug 1357326).
--HG--
extra : rebase_source : 9f0aeeb7eea8417fa4e06d662d566d67ecaf2a24
There can be cases where there is simply nothing to download, especially
with the --for-job argument. So we just stop erroring out when nothing
was downloaded.
However, if the user explicitly requested a particular file(s) via the
command line and there is nothing to download, we still emit an error
code.
--HG--
extra : rebase_source : 143c15c9711bbfbbfdc110da14f3738132d53ed4
This patch does a few things:
a) Adds the resources location from the .app directory to the read whitelist
b) When it's a non-packaged build, mach run (and various mach tests) set an environment variable for the repo location which we allow reads from.
r=haik,froydnj
MozReview-Commit-ID: KNvAoUs5Ati
--HG--
extra : rebase_source : 81ba8bfee0ca96979cf8e30d75cdd47f06bc10ea
This patch does a few things:
a) Adds the resources location from the .app directory to the read whitelist
b) When it's a non-packaged build, mach run (and various mach tests) set an environment variable for the repo location which we allow reads from.
r=haik,froydnj
MozReview-Commit-ID: KNvAoUs5Ati
--HG--
extra : rebase_source : f637acff32fc8582732de932503dd696abc57877
Now that we have automated build jobs that produce toolchains, we want
to avoid the burden of uploading them to tooltool and then update the
tooltool manifests. But we don't have build jobs for all the possible
toolchains, so we allow `mach artifact toolchain` to get a mix of
tooltool and taskcluster artifacts.
For taskcluster artifacts, we can give a list of job names (conveniently
automatically normalized to begin with 'toolchain-' and end with '/opt')
for which the artifacts will be downloaded, in place of any tooltool
package with the same name (if a tooltool manifest is given).
The taskcluster artifacts that we download are the ones matching the
contents of the tree the command is run with, per the resources declared
for the corresponding toolchain build job (in
taskcluster/ci/toolchain*.yml)
So for example, a linux64 build could call the following command:
mach artifact toolchain --tooltool-manifest \
browser/config/tooltool-manifests/linux64/releng.manifest \
--from-build linux64-gcc
and get the right gcc corresponding to the build-gcc script in tree,
along with the other non-gcc files from the tooltool manifest.
Things are however planned to be even more convenient, but some commands
can already benefit from this form (even without a tooltool manifest).
See e.g. bug 1328454.
--HG--
extra : rebase_source : b12ed77bef529eb8d67476aceac0166bdfe2eeed
We're going to potentially use the same download manager for tooltool
and taskcluster artifacts, and we don't want to send the tooltool
authentication header to the taskcluster requests.
--HG--
extra : rebase_source : 79a0afdbf06cb05d7792f413ab1f6715b2a9fe2e
beb43155b7a6 changed WPT items to be 3-tuples instead of 2-tuples.
This broke test_defaults_for_path().
MozReview-Commit-ID: 7M0RcQ7bOIU
--HG--
extra : rebase_source : 28e44e5206abb7939d540ba809ec71325ead341c
At the same time, make the artifact cache directory not indexed, and
move the directory creation to the base classes that actually use it.
--HG--
extra : rebase_source : 62994499afceb5166c0041148dcc702f87166fdc
The ultimate goal is to have a generic command that pulls relevant
toolchains from either tooltool or taskcluster artifacts.
This introduces the command and makes it work to allow to wrap tooltool
in most places where it's used currently, with the ability to replace
tooltool_wrapper.sh as well.
--HG--
extra : rebase_source : f63178282f00f8698a148db484f0276c99d61f7b
Profiling revealed that mozpath.relpath() accounted for a lot of CPU
time when operating on an input of ~42,000 paths.
Due to the nature of the paths we're operating on, we don't need the
full power of mozpath.relpath() here. Instead, we can implement a
specialized version that works given already normalized paths and the
knowledge that context paths must be ancestors of the current path
being examined.
This change drops execution time of a mach command feeding ~42,000
paths to this function from ~90s to ~24s. On an input with 9131 paths,
execution time dropped from ~8.8s to ~3.7s.
MozReview-Commit-ID: EGLiJa10Zj2
--HG--
extra : rebase_source : 8e9d7a957ea3d865a6f18a3b99cca2b22f7a0385
Calling self.test_defaults_for_path() from files_info() with tens
of thousands of paths resulted in a CPU explosion in various path
normalization functions. I don't think it was so much the complexity
of the operations as much as the volume.
For an input with 9131 elements, this reduces execution time of a
mach command from ~25.7s to ~8.8s. With ~42,000 inputs, execution time
drops from <it took too long and I gave up> to ~90s.
MozReview-Commit-ID: pjQQByi2Bc
--HG--
extra : rebase_source : b8307a92e494f128ba3cf29348bc7996cfea4221
With ~42,000 entries in relpaths, this change drops execution time
of this loop from ~23s to ~0.01s.
MozReview-Commit-ID: Afm245tjWUQ
--HG--
extra : rebase_source : 86ee525b0539c80e47e4326a3f111a3fc84d0a6e
For people working on Rust code, compiling in debug mode (Cargo's "dev"
profile) is convenient: debug assertions are turned on, optimization is
turned off, and parallel compilation inside of rustc itself can be
used. These things make the build faster and the debugging experience
more pleasant.
To obtain that currently, one needs to --enable-debug at the Gecko
toplevel, which turns on debug assertions for the entire browser, which
makes things run unreasonably slowly. So it would be desirable to be
able to turn *off* debug mode for the entirety of the browser, but turn
on debug mode for the Rust code only.
Hence this added switch, --enable-rust-debug, which does what it
suggests and defaults to the value of --enable-debug. For our own
sanity and because we judge it a non-existent use case, we do not
support --enable-debug --disable-rust-debug.
