If we had an unsuccessful build, then duplicating the warnings at the
end of the compilation job would just be burying the actual error in a
blizzard of noise.
Several years ago there was a single zip file for all test files. Clients
would only extract the files they needed. Thus, zip was a reasonable
archive format because it allowed direct access to members without
having to decompress the entirety of the stream.
We have since split up that monolithic archive into separate,
domain-specific archives. e.g. 1 archive for mochitests and one
for xpcshell tests. This drastically cut down on network I/O
required on testers because they only fetched archives/data that
was relevant. It also enabled parallel generation of test archives,
we shaved dozens of seconds off builds due to compression being
a long pole.
Despite the architectural changes to test archive management, we
still used zip files. This is not ideal because we no longer access
specific files in test archives and thus don't care about single/partial
member access performance.
This commit implements support for generating tar.gz test archives.
And it switches the web-platform archive to a tar.gz file.
The performance implications for archive generation are significant:
before: 48,321,250 bytes; 6.05s
after: 31,844,267 bytes; 4.57s
The size is reduced because we have a single compression context
so data from 1 file can benefit compression in a subsequent file.
CPU usage is reduced because the compressor has to work less with
1 context than it does with N. While I didn't measure it, decompression
performance should also be improved for the same reasons. And of course
network I/O will be reduced.
mozharness consumers use a generic method for handling unarchiving.
This method automagically handles multiple file extensions. So as long
as downstream consumers aren't hard coding ".zip" this change should
"just work."
MozReview-Commit-ID: LQa5MIHLsms
--HG--
extra : rebase_source : cd029cdbbcccc1d16f03d63a5f1fdf60be5db4fd
extra : source : a0e257e346ccf3c1db332ec5903241f4eeb9a7ee
This allows us to write files coming from a finder or other source
that isn't directly the filesystem.
MozReview-Commit-ID: KhPSD0JYzsQ
--HG--
extra : rebase_source : 24db84974b54a714ba82dfad7ff68fd1a5bf656a
extra : source : ae8bce278626bc84914063f93292ac5e825eec36
Some manifest entries (e.g. skin or locale) have an attached identifier,
and there can be different entries with different identifiers for the
same chrome name. The change from bug 1366169 would consider those as
errors, while they are the expected configuration.
--HG--
extra : rebase_source : ceb08da909121a2ac0a2cdaba7970e4594dde09f
Several years ago there was a single zip file for all test files. Clients
would only extract the files they needed. Thus, zip was a reasonable
archive format because it allowed direct access to members without
having to decompress the entirety of the stream.
We have since split up that monolithic archive into separate,
domain-specific archives. e.g. 1 archive for mochitests and one
for xpcshell tests. This drastically cut down on network I/O
required on testers because they only fetched archives/data that
was relevant. It also enabled parallel generation of test archives,
we shaved dozens of seconds off builds due to compression being
a long pole.
Despite the architectural changes to test archive management, we
still used zip files. This is not ideal because we no longer access
specific files in test archives and thus don't care about single/partial
member access performance.
This commit implements support for generating tar.gz test archives.
And it switches the web-platform archive to a tar.gz file.
The performance implications for archive generation are significant:
before: 48,321,250 bytes; 6.05s
after: 31,844,267 bytes; 4.57s
The size is reduced because we have a single compression context
so data from 1 file can benefit compression in a subsequent file.
CPU usage is reduced because the compressor has to work less with
1 context than it does with N. While I didn't measure it, decompression
performance should also be improved for the same reasons. And of course
network I/O will be reduced.
mozharness consumers use a generic method for handling unarchiving.
This method automagically handles multiple file extensions. So as long
as downstream consumers aren't hard coding ".zip" this change should
"just work."
MozReview-Commit-ID: LQa5MIHLsms
--HG--
extra : rebase_source : 19ec875917546abc147b234a815e1a64c204b92a
This allows us to write files coming from a finder or other source
that isn't directly the filesystem.
MozReview-Commit-ID: KhPSD0JYzsQ
--HG--
extra : rebase_source : 10d494fc910982c3e34f4744592edca906d3a85d
We're getting an intermittent failure running `hg manifest` in CI. I
have no clue why.
What I do know is that we now have the mozversioncontrol Python package
for containing utility code for interacting with version control. It is
slightly more robust and I'm willing to support it more than I am
check_utils.py.
This commit adds a new API to our abstract repository class to obtain the
files in the working directory by querying version control.
Since I suspect cwd was coming into play in automation, I've also
added a utility function to mozversioncontrol to attempt to find
a version control checkout from the current working directory. It
simply traces ancestor paths looking for a .hg or .git directory.
Finally, I've ported all callers of the now-deleted API to the new
one. The old code had some "../.." paths in it, meaning it only
worked when cwd was just right. Since we resolve the absolute path
to the checkout and store it on the repo object, I've updated the
code so it should work no matter what cwd is as long as a repo can
be found. I'm not 100% confident I found all consumers assuming cwd.
But it's a start.
I'm not 100% confident this will fix the intermittent issues in CI. But
at least we should get a better error message and at least we'll be
running less hacky code.
MozReview-Commit-ID: AmCraHXcTEX
--HG--
extra : rebase_source : 815ae369776577ad374333920fd645d412a55148
Windows repackaging for complete mar files will also need to pull a
different value out of the application.ini file, so this code should be
shareable.
MozReview-Commit-ID: CzCoNRYcBPX
--HG--
extra : rebase_source : a5e4b31ba876d811436a7d8d15462fa85f0762f8
The chmod permissions need to be in octal format to get the expected
permissions settings. This only seems to affect the output if we run the
repackaging command on Linux, but it should still be fixed.
MozReview-Commit-ID: to4v7dkSBl
--HG--
extra : rebase_source : 9d2289511ebd05aea8a4c6d37136f258e2463dee
Python can run these files with 'python -m 7z_exe_foo', but using
'import 7z_exe_foo' is not allowed because the module begins with a
number.
See also: https://stackoverflow.com/a/17487228
MozReview-Commit-ID: 97iDdXlZJ1a
--HG--
rename : python/mozbuild/mozbuild/action/7z_exe_archive.py => python/mozbuild/mozbuild/action/exe_7z_archive.py
rename : python/mozbuild/mozbuild/action/7z_exe_extract.py => python/mozbuild/mozbuild/action/exe_7z_extract.py
extra : rebase_source : 1941986a7e6e56305b742710c73b90a3f7b5226b
It makes more sense to have the repackage commands in a separate
directory, since Windows repackaging will add several new types.
MozReview-Commit-ID: 1wA7F7k4NXf
--HG--
rename : python/mozbuild/mozbuild/repackage.py => python/mozbuild/mozbuild/repackaging/dmg.py
extra : rebase_source : b4b4f2fd5c45900aaf125928c144c15cfa1e115e
As described in changeset c94e87a18096, chrome manifest entries are
reordered and don't necessarily appear in the order they have in jar.mn.
And in some cases, the order does matter: when entries with flags are
followed by entries with more broad flags or no flags at all. Nobody
should be writing entries in that order on purpose, so error out during
packaging when we detect the pattern.
--HG--
extra : rebase_source : d9617bbcbd8560503c532a13c10c8afb0fd49411
To make things simpler in configure code, as well as to allow the linter
to skip bugging about some --help dependencies, we make the following
work:
something.some_attr
where the result is equivalent to, currently:
delayed_getattr(something, 'some_attr')
Similar to how they can be combined with "|", we now allow using "&".
As for "|", it would have been better if it were "and", but it's not
possible to override "and" in python ; __and__ is for "&".
CommandLineHandler() expects argv to be like sys.argv, containing the
command name in argv[0], but various tests weren't doing that, in some
cases even leading to ignored arguments passed as argv[0].
In turn, that made lint.py only test browser/moz.configure instead of
all the project moz.configures as intended.
--HG--
extra : rebase_source : 8a87216edaa4a2fd27abb9ef74d38a254a2bbeed
The base bootstrap class has code to conditionally upgrade Mercurial
depending on its version. It is kinda broken. This commit overrides
that code in the MozillaBuild bootstrapper to always run `pip`, which
will ensure the latest stable Mercurial release is installed.
MozReview-Commit-ID: 2O1Ff7dAp4o
--HG--
extra : rebase_source : 98cba0abbebf13e8fe93673d5b46eb16e6a3c147
Previously, `mach bootstrap` would unconditionally prompt to help
configure Mercurial in most scenarios. I agree with Ehsan's observation
in a mailing list post that this behavior doesn't make sense when
running from a Git checkout, as the user probably doesn't care about
Mercurial if they are using Git.
This change doesn't completely ignore Mercurial for Git users. For
example, we still unconditionally run code that verifies that Mercurial
is installed and reasonably up to date. Changing this would be a bit
of work. But even if we wanted to change it, git-cinnabar users
would benefit from having a modern Mercurial installed. So it isn't
straightforward for Git users to ignore Mercurial completely.
MozReview-Commit-ID: 8ncHRgCsjz
--HG--
extra : rebase_source : 7945e3bf3d5283105bac517885f794fc5d7bba6d
Doing something like "not foo" when foo is a @depends function is never
going to do what the user expects, while not necessarily leading to an
error (like, when used in set_config).
It is better to have an error in those cases where it's expected not to
work, at the expense of making templates a little more verbose, rather
than silently do something the user is not expecting.
--HG--
extra : rebase_source : 87ba80eccc5606322f6dd185d5cb5fc88471e51b
Artifact builds were relying on disabling Rust source file inclusion.
The RustLibrary mozbuild class wants to know cargo's output directory
when initializing itself, but when no compile environment is available
rust.configure isn't included and the corresponding config keys
aren't available.
Skip inializing the dependent fields in that case. Since the artifact
build never tries to compile any of the rust libraries, leaving
these properties undefined works ok.
MozReview-Commit-ID: 8IzTsweSygn
--HG--
extra : rebase_source : a59fc01483fbc85766ff4445c5db7ddb1e49b87c
mozconfig_loader is invoked with /bin/sh, which may not be bash. We could
force mozconfigs to be run with bash instead, but that might be disruptive
for existing mozconfigs, and the sole bashism used in mozconfig_loader is
rather straightforward to workaround by namespacing the variables.
Fixes a 403 error running `./mach bootstrap` with stylo enabled.
Bug 1359968 updated the tooltool manifests to point to a
publicly-available clang package. Update the macOS hash
used by mozboot to install the same package for local
use by developers who need rust-bindgen.
MozReview-Commit-ID: FtkATaLwc7n
--HG--
extra : rebase_source : 61390c729e694589b0351026aac0263440a9a3c5
The Python print() function's does not take a format string, so the extraneous %s here inadvertently prints "Downloading clang package from %s https://..." instead of "Downloading clang package from https://...".
MozReview-Commit-ID: 5Is5TK3X34e
--HG--
extra : rebase_source : 02e625fbdaab536d1a787c33edd6dfda9c1039c6
We've traditionally had per-directory 'update' scripts for
third-party media codec implementations. Instead, leverage
the new 'mach vendor' command to centralize the import and
build file generation, placing the upstream source in
third_party/aom.
Note this includes another copy of gtest and other dependencies
which we don't use, but they're required by the upstream build
process we use to generate our own build description, so I've
left them in for now.
MozReview-Commit-ID: CnWcSwvQZEh
--HG--
extra : rebase_source : 28172a41332e920c9ea4a475a6990d43ebf8185f
Add AArch64/ARM64 support to the script that produces android version
codes. Use the same scheme for AArch64 as x86, since the two
architectures don't overlap, and AArch64 should override ARM just like
x86 should override ARM.
For Stylo development, LLVM packages are required due to Stylo's
extensive use of bindgen--generating Rust bindings to Gecko's C++ code.
While people can install LLVM via their system package manager, we've
opted to download the LLVM packages used on Mozilla infrastructure for
building Gecko. Using Mozilla's packages for LLVM ensures that they
work, and also makes it easier/trivial to integrate support for other
things (e.g. Mozilla's static checkers) into `mach bootstrap`.
The way it works today, compiler warnings are logged to build output
twice. The compiler's raw output is logged. Then, if mach's compiler
warning parser detects a warning, a structured log message with info
about that warning is printed as well. Because of the order in which
callbacks fire, mach's warning message is delivered to the logger
first. So build output looks something like:
0:04.63 Warning: -Wsign-compare in /home/gps/src/firefox/security/nss/lib/dev/ckhelper.c: comparison of integers of different signs: 'CK_ULONG' (aka 'unsigned long') and 'int'
0:04.63 /home/gps/src/firefox/security/nss/lib/dev/ckhelper.c:135:45: warning: comparison of integers of different signs: 'CK_ULONG' (aka 'unsigned long') and 'int' [-Wsign-compare]
0:04.63 (obj_template[i].ulValueLen == -1)) {
0:04.63 ~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~
That first line is our structured message formatted to plain text.
Subsequent lines are Clang.
The output here is redundant. But having the structured compiler warning
in the JSON logs is nice so downstream systems don't have to reinvent
the parsing wheel.
Also today, at the end of a clobber build we print a count of the
total number of compiler warnings.
Compiler warnings represent potential bugs. So we want to encourage
people to look at them. However, with build system output often spanning
several hundred lines and compiler warnings scattered in no
deterministic order because build system execution is non-deterministic,
it is easy to lose sight of compiler warnings as they go by in build
output. And, it can be difficult to find a warning in files you care
about because you don't know where in the log to look for that file.
This commit attempts to improve the visibility of compiler warnings
by printing a sorted list of warnings at the end of the build. The list
of warnings it prints are only those seen during the current build
operation. We /could/ print a list of all warnings in the persisted
database. But I think this would be annoying, particularly for
incremental or partial builds.
The structured log message previously emitted at compiler invocation
time has been removed. So now the build backend output doesn't have a
redundant "Warning: " line summarizing the warning next to the compiler
output itself. This is less confusing.
That structured log message has been reborn at the end of the build
and reformatted slightly. If there are any consumers assuming that the
log entry near this structured message is compiler output itself,
they will break. I don't think any such consumers exist. It is also
possible the structured log may not contain warning messages if a
process exit occurs. I'm fine with this: if the `mach` process dies,
we have bigger problems to worry about.
The new output looks something like this:
18:07.97 warning: gfx/angle/src/compiler/translator/util.cpp:216:15 [-Wimplicit-fallthrough] unannotated fall-through between switch labels
18:07.97 warning: gfx/angle/src/compiler/translator/util.cpp:225:15 [-Wimplicit-fallthrough] unannotated fall-through between switch labels
18:07.97 warning: gfx/angle/src/compiler/translator/util.cpp:234:15 [-Wimplicit-fallthrough] unannotated fall-through between switch labels
18:07.97 warning: gfx/cairo/libpixman/src/pixman-bits-image.c:268:32 [-Wshift-negative-value] shifting a negative signed value is undefined
18:07.97 warning: gfx/cairo/libpixman/src/pixman-linear-gradient.c:395:6 [-Wunreachable-code] code will never be executed
18:07.97 warning: gfx/cairo/libpixman/src/pixman-x86.c:80:5 [-Wexpansion-to-defined] macro expansion producing 'defined' has undefined behavior
Note the sorting of files, including by line number and column. If we
wanted to, we could even print that line. But that's for another day
(especially since compilers themselves tend to do this already).
This change has the potential to annoy people. That's because instead
of the parsed warning summary lines being spread out over the build
(where they tend not to be noticed), they will be lumped at the end of
the build log, which people do tend to notice. In my build, I have
~363 warnings/lines. Some could perceive this as excessive and
redundant. But the recourse for this is simple: eliminate compiler
warnings. My hope is that by having the compiler warnings exposed
clearly at the end of the build log that people will notice them
and will take action to eliminate them. Sunlight is a disinfectant.
MozReview-Commit-ID: 9VHXsPhAYmr
--HG--
extra : rebase_source : 6a2da05f794a54ea54a48167a36d130dfbf34e59
Currently, the build monitor has a single compiler warnings database
that unions warnings from previous runs with warnings from the current
invocation. I want to introduce functionality that treats warnings
seen during the current invocation differently from "all warnings."
To facilitate this, this commit introduces a 2nd, non-persisted
warnings database to record warnings just for the current invocation.
MozReview-Commit-ID: FIY0GiarDmr
--HG--
extra : rebase_source : b2002e1c248ea65b2c0ee45a78b1e74d61a26f3c
Currently, the WarningsCollector (which parses warnings from line inputs)
accepts a WarningsDatabase (an object representing a collection of
warnings) and populates that instance as a new warning is parsed. In
an upcoming commit I want to introduce a 2nd WarningsDatabase.
Rather that add it to WarningsCollector, I figure it will be easier
to decouple WarningsDatabase from WarningsCollector.
This commit refactors WarningsCollector to call a callback when a
warning is parsed. This allows consumers to do anything they want
with a warning, including potentially write it to multiple databases.
MozReview-Commit-ID: 7Z9x4FMwyof
--HG--
extra : rebase_source : b4844b5c2b1926840d37e46ead38c8c358762ba8
mozharness is currently making a manual `make -k` invocation. We don't
want automation calling `make` directly. So teach `mach build` to
accept a --keep-going argument that results in `make -k`.
MozReview-Commit-ID: H3lJ4r8S4vj
--HG--
extra : rebase_source : 9feb7bcaeb855254c53c5fa9d49177c5133f2773
