Some manifest entries (e.g. skin or locale) have an attached identifier,
and there can be different entries with different identifiers for the
same chrome name. The change from bug 1366169 would consider those as
errors, while they are the expected configuration.
--HG--
extra : rebase_source : ceb08da909121a2ac0a2cdaba7970e4594dde09f
Several years ago there was a single zip file for all test files. Clients
would only extract the files they needed. Thus, zip was a reasonable
archive format because it allowed direct access to members without
having to decompress the entirety of the stream.
We have since split up that monolithic archive into separate,
domain-specific archives. e.g. 1 archive for mochitests and one
for xpcshell tests. This drastically cut down on network I/O
required on testers because they only fetched archives/data that
was relevant. It also enabled parallel generation of test archives,
we shaved dozens of seconds off builds due to compression being
a long pole.
Despite the architectural changes to test archive management, we
still used zip files. This is not ideal because we no longer access
specific files in test archives and thus don't care about single/partial
member access performance.
This commit implements support for generating tar.gz test archives.
And it switches the web-platform archive to a tar.gz file.
The performance implications for archive generation are significant:
before: 48,321,250 bytes; 6.05s
after: 31,844,267 bytes; 4.57s
The size is reduced because we have a single compression context
so data from 1 file can benefit compression in a subsequent file.
CPU usage is reduced because the compressor has to work less with
1 context than it does with N. While I didn't measure it, decompression
performance should also be improved for the same reasons. And of course
network I/O will be reduced.
mozharness consumers use a generic method for handling unarchiving.
This method automagically handles multiple file extensions. So as long
as downstream consumers aren't hard coding ".zip" this change should
"just work."
MozReview-Commit-ID: LQa5MIHLsms
--HG--
extra : rebase_source : 19ec875917546abc147b234a815e1a64c204b92a
This allows us to write files coming from a finder or other source
that isn't directly the filesystem.
MozReview-Commit-ID: KhPSD0JYzsQ
--HG--
extra : rebase_source : 10d494fc910982c3e34f4744592edca906d3a85d
Windows repackaging for complete mar files will also need to pull a
different value out of the application.ini file, so this code should be
shareable.
MozReview-Commit-ID: CzCoNRYcBPX
--HG--
extra : rebase_source : a5e4b31ba876d811436a7d8d15462fa85f0762f8
The chmod permissions need to be in octal format to get the expected
permissions settings. This only seems to affect the output if we run the
repackaging command on Linux, but it should still be fixed.
MozReview-Commit-ID: to4v7dkSBl
--HG--
extra : rebase_source : 9d2289511ebd05aea8a4c6d37136f258e2463dee
Python can run these files with 'python -m 7z_exe_foo', but using
'import 7z_exe_foo' is not allowed because the module begins with a
number.
See also: https://stackoverflow.com/a/17487228
MozReview-Commit-ID: 97iDdXlZJ1a
--HG--
rename : python/mozbuild/mozbuild/action/7z_exe_archive.py => python/mozbuild/mozbuild/action/exe_7z_archive.py
rename : python/mozbuild/mozbuild/action/7z_exe_extract.py => python/mozbuild/mozbuild/action/exe_7z_extract.py
extra : rebase_source : 1941986a7e6e56305b742710c73b90a3f7b5226b
It makes more sense to have the repackage commands in a separate
directory, since Windows repackaging will add several new types.
MozReview-Commit-ID: 1wA7F7k4NXf
--HG--
rename : python/mozbuild/mozbuild/repackage.py => python/mozbuild/mozbuild/repackaging/dmg.py
extra : rebase_source : b4b4f2fd5c45900aaf125928c144c15cfa1e115e
As described in changeset c94e87a18096, chrome manifest entries are
reordered and don't necessarily appear in the order they have in jar.mn.
And in some cases, the order does matter: when entries with flags are
followed by entries with more broad flags or no flags at all. Nobody
should be writing entries in that order on purpose, so error out during
packaging when we detect the pattern.
--HG--
extra : rebase_source : d9617bbcbd8560503c532a13c10c8afb0fd49411
To make things simpler in configure code, as well as to allow the linter
to skip bugging about some --help dependencies, we make the following
work:
something.some_attr
where the result is equivalent to, currently:
delayed_getattr(something, 'some_attr')
Similar to how they can be combined with "|", we now allow using "&".
As for "|", it would have been better if it were "and", but it's not
possible to override "and" in python ; __and__ is for "&".
CommandLineHandler() expects argv to be like sys.argv, containing the
command name in argv[0], but various tests weren't doing that, in some
cases even leading to ignored arguments passed as argv[0].
In turn, that made lint.py only test browser/moz.configure instead of
all the project moz.configures as intended.
--HG--
extra : rebase_source : 8a87216edaa4a2fd27abb9ef74d38a254a2bbeed
Doing something like "not foo" when foo is a @depends function is never
going to do what the user expects, while not necessarily leading to an
error (like, when used in set_config).
It is better to have an error in those cases where it's expected not to
work, at the expense of making templates a little more verbose, rather
than silently do something the user is not expecting.
--HG--
extra : rebase_source : 87ba80eccc5606322f6dd185d5cb5fc88471e51b
Artifact builds were relying on disabling Rust source file inclusion.
The RustLibrary mozbuild class wants to know cargo's output directory
when initializing itself, but when no compile environment is available
rust.configure isn't included and the corresponding config keys
aren't available.
Skip inializing the dependent fields in that case. Since the artifact
build never tries to compile any of the rust libraries, leaving
these properties undefined works ok.
MozReview-Commit-ID: 8IzTsweSygn
--HG--
extra : rebase_source : a59fc01483fbc85766ff4445c5db7ddb1e49b87c
mozconfig_loader is invoked with /bin/sh, which may not be bash. We could
force mozconfigs to be run with bash instead, but that might be disruptive
for existing mozconfigs, and the sole bashism used in mozconfig_loader is
rather straightforward to workaround by namespacing the variables.
We've traditionally had per-directory 'update' scripts for
third-party media codec implementations. Instead, leverage
the new 'mach vendor' command to centralize the import and
build file generation, placing the upstream source in
third_party/aom.
Note this includes another copy of gtest and other dependencies
which we don't use, but they're required by the upstream build
process we use to generate our own build description, so I've
left them in for now.
MozReview-Commit-ID: CnWcSwvQZEh
--HG--
extra : rebase_source : 28172a41332e920c9ea4a475a6990d43ebf8185f
Add AArch64/ARM64 support to the script that produces android version
codes. Use the same scheme for AArch64 as x86, since the two
architectures don't overlap, and AArch64 should override ARM just like
x86 should override ARM.
The way it works today, compiler warnings are logged to build output
twice. The compiler's raw output is logged. Then, if mach's compiler
warning parser detects a warning, a structured log message with info
about that warning is printed as well. Because of the order in which
callbacks fire, mach's warning message is delivered to the logger
first. So build output looks something like:
0:04.63 Warning: -Wsign-compare in /home/gps/src/firefox/security/nss/lib/dev/ckhelper.c: comparison of integers of different signs: 'CK_ULONG' (aka 'unsigned long') and 'int'
0:04.63 /home/gps/src/firefox/security/nss/lib/dev/ckhelper.c:135:45: warning: comparison of integers of different signs: 'CK_ULONG' (aka 'unsigned long') and 'int' [-Wsign-compare]
0:04.63 (obj_template[i].ulValueLen == -1)) {
0:04.63 ~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~
That first line is our structured message formatted to plain text.
Subsequent lines are Clang.
The output here is redundant. But having the structured compiler warning
in the JSON logs is nice so downstream systems don't have to reinvent
the parsing wheel.
Also today, at the end of a clobber build we print a count of the
total number of compiler warnings.
Compiler warnings represent potential bugs. So we want to encourage
people to look at them. However, with build system output often spanning
several hundred lines and compiler warnings scattered in no
deterministic order because build system execution is non-deterministic,
it is easy to lose sight of compiler warnings as they go by in build
output. And, it can be difficult to find a warning in files you care
about because you don't know where in the log to look for that file.
This commit attempts to improve the visibility of compiler warnings
by printing a sorted list of warnings at the end of the build. The list
of warnings it prints are only those seen during the current build
operation. We /could/ print a list of all warnings in the persisted
database. But I think this would be annoying, particularly for
incremental or partial builds.
The structured log message previously emitted at compiler invocation
time has been removed. So now the build backend output doesn't have a
redundant "Warning: " line summarizing the warning next to the compiler
output itself. This is less confusing.
That structured log message has been reborn at the end of the build
and reformatted slightly. If there are any consumers assuming that the
log entry near this structured message is compiler output itself,
they will break. I don't think any such consumers exist. It is also
possible the structured log may not contain warning messages if a
process exit occurs. I'm fine with this: if the `mach` process dies,
we have bigger problems to worry about.
The new output looks something like this:
18:07.97 warning: gfx/angle/src/compiler/translator/util.cpp:216:15 [-Wimplicit-fallthrough] unannotated fall-through between switch labels
18:07.97 warning: gfx/angle/src/compiler/translator/util.cpp:225:15 [-Wimplicit-fallthrough] unannotated fall-through between switch labels
18:07.97 warning: gfx/angle/src/compiler/translator/util.cpp:234:15 [-Wimplicit-fallthrough] unannotated fall-through between switch labels
18:07.97 warning: gfx/cairo/libpixman/src/pixman-bits-image.c:268:32 [-Wshift-negative-value] shifting a negative signed value is undefined
18:07.97 warning: gfx/cairo/libpixman/src/pixman-linear-gradient.c:395:6 [-Wunreachable-code] code will never be executed
18:07.97 warning: gfx/cairo/libpixman/src/pixman-x86.c:80:5 [-Wexpansion-to-defined] macro expansion producing 'defined' has undefined behavior
Note the sorting of files, including by line number and column. If we
wanted to, we could even print that line. But that's for another day
(especially since compilers themselves tend to do this already).
This change has the potential to annoy people. That's because instead
of the parsed warning summary lines being spread out over the build
(where they tend not to be noticed), they will be lumped at the end of
the build log, which people do tend to notice. In my build, I have
~363 warnings/lines. Some could perceive this as excessive and
redundant. But the recourse for this is simple: eliminate compiler
warnings. My hope is that by having the compiler warnings exposed
clearly at the end of the build log that people will notice them
and will take action to eliminate them. Sunlight is a disinfectant.
MozReview-Commit-ID: 9VHXsPhAYmr
--HG--
extra : rebase_source : 6a2da05f794a54ea54a48167a36d130dfbf34e59
Currently, the build monitor has a single compiler warnings database
that unions warnings from previous runs with warnings from the current
invocation. I want to introduce functionality that treats warnings
seen during the current invocation differently from "all warnings."
To facilitate this, this commit introduces a 2nd, non-persisted
warnings database to record warnings just for the current invocation.
MozReview-Commit-ID: FIY0GiarDmr
--HG--
extra : rebase_source : b2002e1c248ea65b2c0ee45a78b1e74d61a26f3c
Currently, the WarningsCollector (which parses warnings from line inputs)
accepts a WarningsDatabase (an object representing a collection of
warnings) and populates that instance as a new warning is parsed. In
an upcoming commit I want to introduce a 2nd WarningsDatabase.
Rather that add it to WarningsCollector, I figure it will be easier
to decouple WarningsDatabase from WarningsCollector.
This commit refactors WarningsCollector to call a callback when a
warning is parsed. This allows consumers to do anything they want
with a warning, including potentially write it to multiple databases.
MozReview-Commit-ID: 7Z9x4FMwyof
--HG--
extra : rebase_source : b4844b5c2b1926840d37e46ead38c8c358762ba8
mozharness is currently making a manual `make -k` invocation. We don't
want automation calling `make` directly. So teach `mach build` to
accept a --keep-going argument that results in `make -k`.
MozReview-Commit-ID: H3lJ4r8S4vj
--HG--
extra : rebase_source : 9feb7bcaeb855254c53c5fa9d49177c5133f2773
This patch adds a copy of vswhere.exe to build/win32, downloaded from the
current latest release (1.0.62):
https://github.com/Microsoft/vswhere/releases/download/1.0.62/vswhere.exe
It changes toolchain.configure to invoke vswhere.exe instead of reading
the registry, since that no longer works for VS2017 (but vswhere can locate
VS2015). It also removes a layer of complexity in that code by dropping
support for non-64-bit host systems, since we don't really support building
on 32-bit Windows anymore anyway.
There's a little bit of fixup in windows.configure where some LIB paths
have changed in 2017.
MozReview-Commit-ID: 5XLWjidS6W4
--HG--
extra : rebase_source : 90f79b6f4a2d8d925dd20eb0bf6ab96262c227d5
rustc generates .lib files for its libraries when compiling for Windows
(even using MinGW on Linux). But MinGW expects .a files. So we add in
rust-specific prefix and suffixes so MinGW builds can find the libs that
rustc generates. (And the RUST_LIB- variables default to the same vales
as the LIB_ variables otherwise.)
MozReview-Commit-ID: ClsA0YuJaxh
--HG--
extra : rebase_source : 7b46460c94ceb34b7a5a302ce91d3f1dca600041
Everything depending on the widget being gonk can go away, as well as
everything depending on MOZ_AUDIO_CHANNEL_MANAGER, which was only
defined on gonk builds under b2g/ (which goes away in bug 1357326).
--HG--
extra : rebase_source : 9f0aeeb7eea8417fa4e06d662d566d67ecaf2a24
There can be cases where there is simply nothing to download, especially
with the --for-job argument. So we just stop erroring out when nothing
was downloaded.
However, if the user explicitly requested a particular file(s) via the
command line and there is nothing to download, we still emit an error
code.
--HG--
extra : rebase_source : 143c15c9711bbfbbfdc110da14f3738132d53ed4
This patch does a few things:
a) Adds the resources location from the .app directory to the read whitelist
b) When it's a non-packaged build, mach run (and various mach tests) set an environment variable for the repo location which we allow reads from.
r=haik,froydnj
MozReview-Commit-ID: KNvAoUs5Ati
--HG--
extra : rebase_source : 81ba8bfee0ca96979cf8e30d75cdd47f06bc10ea
This patch does a few things:
a) Adds the resources location from the .app directory to the read whitelist
b) When it's a non-packaged build, mach run (and various mach tests) set an environment variable for the repo location which we allow reads from.
r=haik,froydnj
MozReview-Commit-ID: KNvAoUs5Ati
--HG--
extra : rebase_source : f637acff32fc8582732de932503dd696abc57877
Now that we have automated build jobs that produce toolchains, we want
to avoid the burden of uploading them to tooltool and then update the
tooltool manifests. But we don't have build jobs for all the possible
toolchains, so we allow `mach artifact toolchain` to get a mix of
tooltool and taskcluster artifacts.
For taskcluster artifacts, we can give a list of job names (conveniently
automatically normalized to begin with 'toolchain-' and end with '/opt')
for which the artifacts will be downloaded, in place of any tooltool
package with the same name (if a tooltool manifest is given).
The taskcluster artifacts that we download are the ones matching the
contents of the tree the command is run with, per the resources declared
for the corresponding toolchain build job (in
taskcluster/ci/toolchain*.yml)
So for example, a linux64 build could call the following command:
mach artifact toolchain --tooltool-manifest \
browser/config/tooltool-manifests/linux64/releng.manifest \
--from-build linux64-gcc
and get the right gcc corresponding to the build-gcc script in tree,
along with the other non-gcc files from the tooltool manifest.
Things are however planned to be even more convenient, but some commands
can already benefit from this form (even without a tooltool manifest).
See e.g. bug 1328454.
--HG--
extra : rebase_source : b12ed77bef529eb8d67476aceac0166bdfe2eeed
We're going to potentially use the same download manager for tooltool
and taskcluster artifacts, and we don't want to send the tooltool
authentication header to the taskcluster requests.
--HG--
extra : rebase_source : 79a0afdbf06cb05d7792f413ab1f6715b2a9fe2e
beb43155b7a6 changed WPT items to be 3-tuples instead of 2-tuples.
This broke test_defaults_for_path().
MozReview-Commit-ID: 7M0RcQ7bOIU
--HG--
extra : rebase_source : 28e44e5206abb7939d540ba809ec71325ead341c
At the same time, make the artifact cache directory not indexed, and
move the directory creation to the base classes that actually use it.
--HG--
extra : rebase_source : 62994499afceb5166c0041148dcc702f87166fdc
The ultimate goal is to have a generic command that pulls relevant
toolchains from either tooltool or taskcluster artifacts.
This introduces the command and makes it work to allow to wrap tooltool
in most places where it's used currently, with the ability to replace
tooltool_wrapper.sh as well.
--HG--
extra : rebase_source : f63178282f00f8698a148db484f0276c99d61f7b
Profiling revealed that mozpath.relpath() accounted for a lot of CPU
time when operating on an input of ~42,000 paths.
Due to the nature of the paths we're operating on, we don't need the
full power of mozpath.relpath() here. Instead, we can implement a
specialized version that works given already normalized paths and the
knowledge that context paths must be ancestors of the current path
being examined.
This change drops execution time of a mach command feeding ~42,000
paths to this function from ~90s to ~24s. On an input with 9131 paths,
execution time dropped from ~8.8s to ~3.7s.
MozReview-Commit-ID: EGLiJa10Zj2
--HG--
extra : rebase_source : 8e9d7a957ea3d865a6f18a3b99cca2b22f7a0385
Calling self.test_defaults_for_path() from files_info() with tens
of thousands of paths resulted in a CPU explosion in various path
normalization functions. I don't think it was so much the complexity
of the operations as much as the volume.
For an input with 9131 elements, this reduces execution time of a
mach command from ~25.7s to ~8.8s. With ~42,000 inputs, execution time
drops from <it took too long and I gave up> to ~90s.
MozReview-Commit-ID: pjQQByi2Bc
--HG--
extra : rebase_source : b8307a92e494f128ba3cf29348bc7996cfea4221
With ~42,000 entries in relpaths, this change drops execution time
of this loop from ~23s to ~0.01s.
MozReview-Commit-ID: Afm245tjWUQ
--HG--
extra : rebase_source : 86ee525b0539c80e47e4326a3f111a3fc84d0a6e
For people working on Rust code, compiling in debug mode (Cargo's "dev"
profile) is convenient: debug assertions are turned on, optimization is
turned off, and parallel compilation inside of rustc itself can be
used. These things make the build faster and the debugging experience
more pleasant.
To obtain that currently, one needs to --enable-debug at the Gecko
toplevel, which turns on debug assertions for the entire browser, which
makes things run unreasonably slowly. So it would be desirable to be
able to turn *off* debug mode for the entirety of the browser, but turn
on debug mode for the Rust code only.
Hence this added switch, --enable-rust-debug, which does what it
suggests and defaults to the value of --enable-debug. For our own
sanity and because we judge it a non-existent use case, we do not
support --enable-debug --disable-rust-debug.
This commit moves symbol dumping to the compile tier, to be run via "syms"
targets. Tracking files are used for the sake of incremental builds, because
dump_syms may genearate multiple outputs whose paths are not known ahead of
time.
Minimal changes to symbolstore.py are made here. More extensive
simplifications will be made in a future commit on the basis of symbolstore.py
handling one file at a time.
MozReview-Commit-ID: 3mOP8A6Y7iM
--HG--
extra : rebase_source : bfe97afcdfc05b9e79f01577701c83e8b00eb4e9
This creates "syms" targets that depend on the corresponding "target" for
directories containing shared libraries or programs. These targets are added
to the main compile graph in automation, and can be invoked through a special
"symbols" target. A future commit will use these targets to dump symbols for
shared libraries and programs during the compile tier.
MozReview-Commit-ID: KLuvmqsK4Zj
--HG--
extra : rebase_source : 8d76b999cb6fac8f11168ac6ebfb58774dfc2d3c
These variables are ignored in favour or reading the same values
from the passed in libdef object.
MozReview-Commit-ID: 8Xkkd68clNN
--HG--
extra : rebase_source : 767c15f1b59a6f48d09411c4f2e64eff9f8b4cc6
This adds a unit test for the expected behavior, and adds the modules
from mach_bootstrap.py that are missing in the virtualenv for the test
to run.
--HG--
extra : rebase_source : e34d0474cfb6c8c5ce9cd847b96de88906191923
In most cases, the HTTP response for the download will contain the
content-length, but if some error happens (e.g. authentication error),
there might not be one, and the download fails with a TypeError for a
division by None, instead of failing with a more friendly error message
about the HTTP error.
--HG--
extra : rebase_source : f179142e69c9ca09b05ad2ff942753fd84da6a69
This is the initial support of 'mach repackage', which can take an
existing tarball and create a DMG on either an OSX host or on a Linux
host with cross-OSX tools. Configure is needed in order to find the
tools necessary to create the DMG. On a Linux cross-compiled environment
with tooltool, this can be as simple as:
export MKFSHFS=$topsrcdir/hfsplus-tools/newfs_hfs
export DMG_TOOL=$topsrcdir/dmg/dmg
export HFS_TOOL=$topsrcdir/dmg/hfsplus
ac_add_options --disable-compile-environment
MozReview-Commit-ID: 6t2rlXpwUvu
--HG--
extra : rebase_source : 39f2f9ecac9d7e5af197f1b5dea70cd307acf488
In bug 1335309, FileFinder was made to default to not find executables,
and zip.py was made to use the default instead. Which made sense for
most uses of zip.py, except for the jsshell package.
So we add a flag to make zip.py able to strip executable (which happens
when the FileFinder is made to find them), and use that flag for the
jsshell package only.
--HG--
extra : rebase_source : 0202f9acd5e6175d3790aaef026e18c6913cf0c6
EmptyConfig objects set JS_STANDALONE=1 by default. However, test tasks that need to run without an objdir
need to be behind an "if not CONFIG['JS_STANDALONE']" condition to avoid causing bustage to sm-pkg task (js
packaging). This patch explicitly deletes that default value, only when generating the TestManifestBackend.
Ideally, the js/src packaging should have their own moz.build instead of re-using the root moz.build. But this
is an easier fix in the short term to get the marionette-harness tests working again.
MozReview-Commit-ID: 26lHLY6WlZK
--HG--
extra : rebase_source : 9c2ffdd938f2f2d6ead7d2aead610a7028e18d97
This imports two modules from mozregression in the tree to do so. They
are imported from current trunk on github, rather than the version we
were getting from pypi.
Note we take six from testing/web-platform/tests/tools/six) instead of
moving it to python/six because it's there by coming from a copy of
https://github.com/w3c/wpt-tools, which contains it as a submodule, and
moving it would make updates there harder.
--HG--
extra : rebase_source : 16619d1c3f38f6493fb490a64361b3f4e8fecc1f
Cargo hashes various compilation settings into the dependency graph for
dependent libraries. So if the compilation settings for gkrust and
gkrust-gtest are different, their dependencies will likewise be
different. The setup we've created in the previous patches depends on
the compilation settings being identical, so we should enforce that at
the moz.build level.
Rust libraries can set RUST_LIBRARY_TARGET_DIR so that they can share
compilation artifacts with other libraries. This setting needs to be
propagated to the backend so it can be communicated to Cargo.
The "unsupported code directive" is added to the 'ccache -s' output in
b6d7cf5502
We need to teach our parser for it.
MozReview-Commit-ID: IrrJv7I7BVa
--HG--
extra : rebase_source : 025167f75ce8486287d408ccdb3d8113450354cb
The "unsupported code directive" is added to the 'ccache -s' output in
b6d7cf5502
We need to teach our parser for it.
MozReview-Commit-ID: IrrJv7I7BVa
--HG--
extra : rebase_source : b3bb6de344e26e7a62fc59f899c45168bafca4ef
This removes the UNIFY_DIST and UNIFIED_BUILD variables, as well as the
--unify flag from the packager and UnifiedBuildFinder from mozpack. As a
result the STAGEPATH variable is never defined anymore, so its uses can
be removed as well.
test_unify.py is currently the only mozbuild/mozpack test that fails
without running configure first, and there isn't much point in fixing
tests for things that we don't actually use anymore.
MozReview-Commit-ID: F5q1FPW3Did
--HG--
extra : rebase_source : cadbd237f51c23ea1983135294521d628d16f0df
This helps us avoid recursing over every directory when we only need to
run 'make check' in a select few.
MozReview-Commit-ID: BJ3hJBOneIz
--HG--
extra : rebase_source : 2493f924b9ccba3c779e512d7a8b7a2c26f43797
This replaces the 'run-tests-deps' make target with a python function that will directly
read moz.build files, emit them with TestManifestEmitter, then consume them with
TestManifestBackend. Because the TestResolver is the only place that actually reads the
test metadata files, we can remove this logic from the CommonBackend as well.
MozReview-Commit-ID: DXgMoeH5dKf
MozReview-Commit-ID: HstZ57qkqf2
--HG--
extra : rebase_source : f377fa6863ef66d3adb86ed64f844e346686862f
Currently, the only way to emit objects after reading moz.build, is to emit everything. Though, sometimes
it may be desirable to only emit certain types of objects. This adds a new argument that allows consumers
to specify a custom emitter function. This gives them the flexibility to do whatever they want.
This will be used when resolving tests, so only TestManifest objects are emitted.
MozReview-Commit-ID: DPGgNmn2JvE
--HG--
extra : rebase_source : 044f8c8cad1bf7fd1dfbed05c89fbd114508182a
This is a drive by fix that is not relevant to the rest of the commit series.
MozReview-Commit-ID: Bwrb74o3Qh8
--HG--
extra : rebase_source : 34706a6640a6c704f9c03f24aede71e4c1c5ac1c
Currently the CommonBackend is responsible for processing TestManifest objects and using them to generate
the test metadata files (e.g all-tests.pkl et al). This patch pulls that logic out into a partial backend
specifically for test manifests.
This patch is solely a refactoring and shouldn't change any build behaviour. CommonBackend has a
TestManifestBackend instance and calls consume_object directly on it. However, this is just a temporary
measure to avoid checking in a broken commit.
This commit also adds a test for the 'test-defaults.pkl' file which was previously missing.
MozReview-Commit-ID: HOr2QVT8CJ1
--HG--
extra : rebase_source : e4b9cb982937381346c0821971191277173b165b
When vendoring third-party files, we'd like an explicit notice/review
when said files contain a "large file". This commit adds such checks
for files vendored via `mach vendor rust`.
As we don't yet have a server-side hook in place to prevent large files
from being added, we just have a command-line flag that people are
expected to use, on the honor system, to permit large files to be added
when vendoring.
Without this, clang-tidy cannot check any diffs including such files.
In the future, when checking the entire tree using clang-tidy, we will
ignore these entries in the compilation database.
The way directory traversal is computed relies on the
RecursiveMakeTraversal class, which is used to reproduce the old
traversal order from the old entirely-in-make traversal with DIRS,
PARALLEL_DIRS, etc. because of the undeclared intra-directory
dependencies that are looming here and there.
It's fed through DirectoryTraversal objects emitted by the frontend.
Normally, DirectoryTraversal objects are emitted for a directory,
possibly giving the subdirectories defined in DIRS/TEST_DIRS its
moz.build. But in the case of gyp processing, nothing places the gyp
objdirs in some virtual DIRS of some parent moz.build since bug 1308982.
As a consequence, the corresponding entries in the
RecursiveMakeTraversal instance attached to the backend are not attached
to any parent directory. When subsequently traversing the tree from the
root, they are never found, and end up being skipped, irregarding of
their actual _no_skip status.
It would probably be possible to revert the changes from bug 1308992,
but we might as well not rely on remains from the old ways. So instead,
we make the RecursiveMakeTraversal consider directories without a
declared parent attached directly to the root directory. They don't need
to depend on any other directory anyways.
--HG--
extra : rebase_source : 17403922322a71d490fdea8db0ff16b04983ed7a
CLOSED TREE
Backed out changeset 158233bce738 (bug 1197325)
Backed out changeset b5ac3fa0bbe7 (bug 1197325)
Backed out changeset 55a8ad127517 (bug 1197325)
We had been installing them to dist/plugins with the rest of the test plugins,
but tests are actually expecting them to be in dist/bin.
MozReview-Commit-ID: 9qYFgZA4Fni
--HG--
extra : rebase_source : 52a4cbf70ffe7998cfe81ab5e508b9f802dd3a35
Remove the option to build without rust code. We are not testing
this configuration and expect to land non-optional rust code in
the near future, so it doesn't make sense to maintain this option.
MozReview-Commit-ID: CwTlMXGvr5n
--HG--
extra : rebase_source : 080a9df5b4828c66aa2452ad1c16a503bcd5e689
This updates the client artifact code to locate test support files in the
common test archive and populate the objdir with these files appropriately.
MozReview-Commit-ID: GuXjwUtsl
--HG--
extra : rebase_source : 3560efee22533f60be1e7394fa0841005991cca6
Generated support files for specific tests generally end up in a harness specific
test archive, but artifact builds, which might not want to generate these files
themselves, get all of their test related files from the common.tests.zip. This
effectively moves a small number of these support files from harness specific
archives to the common test archive, making it possible to run tests that
depend on these files against artifact builds.
MozReview-Commit-ID: BvEj1ot1OTw
--HG--
extra : rebase_source : 06e958adc7b5667c50ca5ddbf06c1bf878d75737
Install manifests influence test packaging, allowing artifact builds to consume
generated test support files and run tests that depend on them. This commit
tracks binaries with an install target under "_tests" so they will be correctly
installed in artifact builds.
This could similarly be achieved by installing the binaries via TEST_HARNESS_FILES
rather than setting FINAL_TARGET, however, PGO builds will fail attempting to install
the files during the profile generation step, because SIMPLE_PROGRAMS are not built
for this step.
MozReview-Commit-ID: ES4bTxOoqMN
--HG--
extra : rebase_source : f41d8c47711fc15247fa336e2e9cbdedf57c2daf
Back when the class was written, for the packaging code, it made sense
that the default was True. But now that it's used all over the place,
and that the vast majority of uses are with find_executables=False, it
makes more sense for that to be the default.
--HG--
extra : rebase_source : ff813735fc0d53093f348f20eb77ee03e9b09d4e
And make it an error not to give it. While the default is True, we do
pass a value of False wherever it makes sense, which happens to be, in
most places.
This is an intermediate step to flip the default from True to False,
ensuring that we don't unwantedly switch some calls to False.
--HG--
extra : rebase_source : 432e03f032fef378af482581685583262e5d2661
We want to avoid giving --help dependencies to host and target, so that
they we don't spawn config.guess and config.sub when running configure
--help, and don't need to reach out to the which module to find a
suitable shell to execute them.
So, when --help is given, we return a fake host/target namespace, and
avoid the config.guess/config.sub-invoking code being executed.
Then, by giving the --help option to the linter, it can properly find
that the config.guess/config.sub-invoking code doesn't need the
dependency on --help.
This effectively unbreaks configure --help after bug 1313306.
--HG--
extra : rebase_source : 9ed4cbe5a81121b139d0d86aff6af542c2dff53b
Ideally, it would have been better if it were "or", but it's not
possible to override "or" in python ; __or__ is for "|".
This does feel magic, but it's also shorter than adding something like
@depends_any(), and while we're only adding "|" as of this change, we
can add other operations such as "&" in the future, or __getattr__ for
things like milestone.is_nightly.
An alternative form in moz.configure could require the @depends function
to be called, e.g. "a() | b()" instead of "a | b", but I'm not
particularly convinced that one is less magic than the other.
This feature is hooked up such that b is not resolved if a is true,
although in practice, it will still be resolved in Sandbox.run... but
not when --help is passed. In the long run, the forced resolution of
@depends functions will be removed from Sandbox.run.
--HG--
extra : rebase_source : 2389153acfbc3930786d5495ba4a04640325613e
Adding those dependencies, retrospectively, only worked around the poor
handling of --help requirements by the linter, that we fixed a few
commits ago. This is now not necessary anymore.
--HG--
extra : rebase_source : 0ba95ab2d246315be097fed7897db1c20666060a
Several things were wrong with the wrapping:
- the equality test on functions was actually comparing the memoized
functions, which have a type memoize, which inherits from dict. So
they weren't comparing actual functions, but the dict used to store
the cache of their invocation.
- each CombinedDependsFunction created for the same combination function
used a different wrapped function, so even if the dict problem wasn't
there, the equality test still wouldn't work, except if the function
wrapping itself was memoized.
- the memoization was not particularly useful.
Also, for upcoming changes, we'd actually like the combination function to
take an iterable instead of a variable argument list, so that items of
the iterable can be skipped.
--HG--
extra : rebase_source : 2c91889315b49695a2e2b709a9264de9ff237598
We're going to change the function signature for CombinedDependsFunction,
so make it visible in the API that the function member is not meant to
be used directly. The linter still does, though, because it needs to
look in their guts.
At the same time, avoid setting DependsFunction names via the function
name itself, because in upcoming changes, it will not be modifiable in
some cases.
--HG--
extra : rebase_source : a46ec8d3d2e2511dbb44a48cf2c02ab7ad190510
Options with a `when` argument (either directly, or inherited through
only_when() or an include) require --help per _value_for_option, but
that code path is not exercised during a lint pass.
With this change, along the previous one, we now correctly detect that
bug 1316957 was not supposed to work as is.
--HG--
extra : rebase_source : 0961c132bb200a90814b948aff7d2e80a1b21a30
Bug 1313306 relaxed the --help dependency requirement in some cases, but
while doing so, the requirement was also removed in other, unexpected
cases. Specifically, the --help dependency ended up not being required
on indirect dependencies that should have had it, had the --help
dependency been explicit on the direct dependency.
--HG--
extra : rebase_source : 81755988754798590482dfb297b7ddfc14ee88c7
It turns out there are shockingly few cases of manifestparser manifests that actually use the ';'
character as a comment. Because we will soon allow inline comments, deprecating the use of ';' will
ensure that values are allowed to have semicolons in them.
Even without inline comments, might as well enforce consistency across manifests.
MozReview-Commit-ID: AEPPQFdNXG0
--HG--
extra : rebase_source : 3540fa385f328bffb020c0a6debc4d2b3a90ed39
We're going to remove the xpcom glue, so there is no need to check that
nothing depends on it anymore.
MozReview-Commit-ID: IoTo2b9DtiL
--HG--
extra : rebase_source : 03d1a01af802a7d0833f2536e1dcc0180696148c
extra : source : ad4e531c7070874770201b5af956f3f8045c973b
We're going to remove the xpcom glue, so there is no need to check that
nothing depends on it anymore.
--HG--
extra : rebase_source : 4323d9c388c60258c581771ac3bc5aa2100ea699
Here, we also modify APKOpen to use the XPCOM glue loading process
instead of custom symbol resolution, so that the Bootstrap API can be
used in a more straightforward manner.
--HG--
extra : rebase_source : 55037ba30ca66a090b73923a3ce8df5b054bf47a
This should fetch the artifacts necessary to run tests and extract
them correctly.
While we're here, clean up some other regular expressions which
weren't escaped consistently.
Also, if you're going to use backslashes in literal strings because of
regular expressions, r-strings are probably better. (Otherwise you
should double all the backslashes.)
All the same, try to stop short of going full PEP8 without express
permission.
MozReview-Commit-ID: 8S1PVHX9xEZ
--HG--
extra : rebase_source : a011f4bf99842bcb897b405ea7810143e93acec9
We're going to remove the xpcom glue, so there is no need to check that
nothing depends on it anymore.
--HG--
extra : rebase_source : 4323d9c388c60258c581771ac3bc5aa2100ea699
Originally landed from https://reviewboard.mozilla.org/r/88220/#review88512 on date. With the following message:
Bug 1312585 - Vendor chunkify r=gps r=bhearsum
for - Set l10n repacks to pass in explicit list of locales, rather than having mozharness do chunking
I'm vendoring from bhearsum's repo, extra r=him so he is aware of its in-tree home.
MozReview-Commit-ID: 50Ub24kyTHX
--HG--
extra : rebase_source : c07870fe309b71ca77b8f1fd3598c6f6677acf88
extra : source : b0a0539d9d55b3b54fd024afbadbf687616a9443
This allows developers to manually upgrade packages as desired by running
|cargo update -p <package>| or |cargo update| on gkrust and gkrust-gtest.
Previously this update would happen implicitly as part of |mach vendor rust|,
and made it hard to upgrade specific packages without upgrading everything.
MozReview-Commit-ID: 91GOdtGmwX0
MacOS doesn't include openssl any more, but the cargo-vendor
build expects a system version. Special case using a copy
in /usr/local/opt/openssl, where homebrew typically installs it.
MozReview-Commit-ID: DbdnfVdEhWV
--HG--
extra : rebase_source : c793cf239c238f817ba43798e759b8afe0fd5b9c
This was a mistake from the beginning. I'm removing it now so that I
can easily generate across objdirs. While we transition from
moz.build to Gradle, I want all the build logic to be in
mobile/android/base but the outputs to be split across Gradle project
locations. That's hard to do when generated/ is automatically
prepended to generated_sources paths.
MozReview-Commit-ID: L07ZZBTsNw5
--HG--
extra : rebase_source : 1fbd131f1f28dfd67465f7ecda40985abd530ae0
Make the symlinks in the dist directory relative instead of absolute.
MozReview-Commit-ID: HS7KL4JwSbV
--HG--
extra : rebase_source : 5dca673cc17423d47e6707d8800f7ee9693a9c48
The test results were updated to match current behaviour. The
TestDummyAudioWithTransport and TestDummyVideoWithTransports are disabled due
to shutdown crashes and intermittent failures that show up in automation.
A follow up bug has been filed to fix these. The GMP test was removed
completely as it seems unlikely that it will be practical to test that from a
gtest.
MozReview-Commit-ID: 2pOb7u2Qp7v
--HG--
rename : media/webrtc/signaling/test/mediaconduit_unittests.cpp => media/webrtc/signaling/gtest/mediaconduit_unittests.cpp
extra : rebase_source : 992330f83e0a6a57810f1c5f0b4ea77f2512cd92
This commit also adds an overdue test for plain Rust libraries in the
recursivemake backend, but said test also serves to ensure that we don't
emit features for a library if none were defined in moz.build.
It turns out that, in practice, the LD variable is not used by the build
system, except on Windows, where it's used to feed the default for LINK,
which is then re-injected as LD.
The upcoming changes are going to normalize the use of LD/LINK.
--HG--
extra : rebase_source : 2a1a9924e963e082e119ff3874e8ff24247f4f94
We're going to need this functionality for Rust programs as well as Rust
libraries, so we might as well move the functionality out of RustLibrary
into a separate function.
The pip function `check_if_exists` returns True if any version of the named
package exists, and sets attributes on the requirement object based on
whether the correct version exists. Prior to this patch, we were interpreting
a `True` result from `check_if_exists` to mean the named package did not need
to be installed or updated at all, causing the build system virtualenv to use
incorrect dependency versions and fail in some cases. This was found in the
case of attempting an artifact build on a system where mozregression was
installed locally, resulting in an incompatible version of the taskcluster
client being used by the artifact code.
MozReview-Commit-ID: 5Q54PsUIrKR
--HG--
extra : rebase_source : f6ae53957401af13cf2eae2780783094f99f8136
Ideally, we'd shell-quote more arguments, but that would also require
different quoting for the make dependencies and the command line
arguments, so go for the immediate need.
--HG--
extra : rebase_source : c24c782e4faec41ccb18575dce4d0cf513f8649a
While the recursive make backend needs to handle ObjDir*Files separately
for now, other backends can handle them just as FinalTarget*Files instead,
simplifying their implementation.
--HG--
extra : rebase_source : 3201fc755e07489b0f440839f02e4052fb3dff4b
This change is mostly straightforward, except for the following.
- It removes all the printing from the do_check_* macros because gtest macros
do appropriate printing.
- test_StatementCache.cpp needs some special gtest magic for the type
parameterization.
- It merges the four tests in test_unlock_notify.cpp because they rely on being
executed in order, and so aren't independent.
- storage_test_harness_tail.h is no longer necessary because gtest provides the
test looping functionality.
- It uses #include and the preprocessor to remove the duplication between
test_deadlock_detector.cpp and xpcom/tests/DeadlockDetector.cpp.
- It makes the test in test_service_init_background_thread.cpp a death test to
force it to be the first storage gtest, because it fails otherwise.
- It adds code to undo the SQLite mutex hooking as necessary, so that tests
don't interfere with each other.
- It de-virtualizes Spinner's destructor (as identified in bug 1318282).
--HG--
rename : storage/test/storage_test_harness.h => storage/test/gtest/storage_test_harness.h
rename : storage/test/test_AsXXX_helpers.cpp => storage/test/gtest/test_AsXXX_helpers.cpp
rename : storage/test/test_StatementCache.cpp => storage/test/gtest/test_StatementCache.cpp
rename : storage/test/test_asyncStatementExecution_transaction.cpp => storage/test/gtest/test_asyncStatementExecution_transaction.cpp
rename : storage/test/test_async_callbacks_with_spun_event_loops.cpp => storage/test/gtest/test_async_callbacks_with_spun_event_loops.cpp
rename : storage/test/test_binding_params.cpp => storage/test/gtest/test_binding_params.cpp
rename : storage/test/test_deadlock_detector.cpp => storage/test/gtest/test_deadlock_detector.cpp
rename : storage/test/test_file_perms.cpp => storage/test/gtest/test_file_perms.cpp
rename : storage/test/test_mutex.cpp => storage/test/gtest/test_mutex.cpp
rename : storage/test/test_service_init_background_thread.cpp => storage/test/gtest/test_service_init_background_thread.cpp
rename : storage/test/test_statement_scoper.cpp => storage/test/gtest/test_statement_scoper.cpp
rename : storage/test/test_transaction_helper.cpp => storage/test/gtest/test_transaction_helper.cpp
rename : storage/test/test_true_async.cpp => storage/test/gtest/test_true_async.cpp
rename : storage/test/test_unlock_notify.cpp => storage/test/gtest/test_unlock_notify.cpp
extra : rebase_source : dbb695c112564efa1945116be1a8435988982e74
The current way they are generated makes it so that it's not trivial for
the backends to figure that one depends on the other. While we should
eventually make things such that using FINAL_TARGET_PP_FILES works for
that, it's currently simpler to just use GENERATED_FILES.
--HG--
rename : build/application.ini => build/application.ini.in
extra : rebase_source : cad3460b65d99bdae3e6f1d9dd611e0a602ebfed
This will allow tests, etc to conditionally run depending on Stylo's
presence.
MozReview-Commit-ID: 3WHxNawP1qC
--HG--
extra : rebase_source : d4d983d37d3bfac1f9d465a3023b5bd5df52cd56
extra : source : 6baa88f7b1f4183ea57b9f8928556975042cad29
This deprecates PYTHON_UNIT_TESTS and replaces it with PYTHON_UNITTEST_MANIFESTS.
In the build system, this means python unittests will be treated the same as all
other test suites that use manifestparser. New manifests called 'python.ini' have
been created for all test directories containing python unittests.
MozReview-Commit-ID: IBHG7Thif2D
--HG--
extra : rebase_source : 11a92a2bc544d067946bbd774975140147458caa
This patch does a few things:
1) Change all the in-tree tooltool manifests to contain sccache2 instead of the existing Python sccache
2) Change mozconfig.cache to point at sccache.
3) Lightly tweak the --with-cccache configure option to support sccache, and detect whether we're using ccache or sccache and set an option appropriately.
4) Add a MOZ_SCCACHE_VERBOSE_STATS option, and add a target in the top-level Makefile to make sccache spit out its stats at the end of the build. This is useful to see the cache hits/errors until we get something better.
5) Add MOZ_USING_SCCACHE to the build telemetry. Not that I think it will be immediately useful, but for future use.
MozReview-Commit-ID: 9lrdLwNj5Bm
--HG--
extra : rebase_source : d323457df10d0ee0ac5811940e518d9422a7e070
In order for VS2017 to recognize solution and project files as
belonging to VS2017 and not some earlier version, we need to bump
a few versions strings.
The added code path can't be hit yet because MSVS_VERSION cannot
be "2017" until configure detects VS2017. I hacked up MSVS_VERSION
locally and verified that solutions produced with the new code load
in VS2017.
FWIW, the solution loads significantly faster in VS2017 compared to
VS2015!
MozReview-Commit-ID: 2vYAgd6wOsG
--HG--
extra : rebase_source : a29b956bb6eef9071cb46d30460a42ef17658026
