This formats the marionette-harness python tests to be a regular |mach python-test| suite. Though
we add subsuite=marionette, this is just for automation purposes. The new preferred way to run the
marionette harness tests locally is:
./mach python-test testing/marionette
They will also run if running the full suite.
The mozbase packages.txt file modifies mozlog to use 'setup.py' instead of 'pth'. The reason for
this is that the marionette-harness tests use the pytest_mozlog pytest plugin for formatting
their results (converts pytest format into something resembling the standard tbpl logging format).
In order for this plugin to get picked up however, mozlog's setup.py file needs to be processed.
MozReview-Commit-ID: Ata99evHxbd
--HG--
extra : rebase_source : 16ed70edd38a53c3279d8632d7cba3df4d5216c3
The "unsupported code directive" is added to the 'ccache -s' output in
b6d7cf5502
We need to teach our parser for it.
MozReview-Commit-ID: IrrJv7I7BVa
--HG--
extra : rebase_source : 025167f75ce8486287d408ccdb3d8113450354cb
The "unsupported code directive" is added to the 'ccache -s' output in
b6d7cf5502
We need to teach our parser for it.
MozReview-Commit-ID: IrrJv7I7BVa
--HG--
extra : rebase_source : b3bb6de344e26e7a62fc59f899c45168bafca4ef
This removes the UNIFY_DIST and UNIFIED_BUILD variables, as well as the
--unify flag from the packager and UnifiedBuildFinder from mozpack. As a
result the STAGEPATH variable is never defined anymore, so its uses can
be removed as well.
test_unify.py is currently the only mozbuild/mozpack test that fails
without running configure first, and there isn't much point in fixing
tests for things that we don't actually use anymore.
MozReview-Commit-ID: F5q1FPW3Did
--HG--
extra : rebase_source : cadbd237f51c23ea1983135294521d628d16f0df
This helps us avoid recursing over every directory when we only need to
run 'make check' in a select few.
MozReview-Commit-ID: BJ3hJBOneIz
--HG--
extra : rebase_source : 2493f924b9ccba3c779e512d7a8b7a2c26f43797
This adds the ability to use manifestparser subsuites to |mach python-test|.
Subsuites are based on the premise of a "default" set that gets run when no
subsuites are explicitly specified. When a test is labelled with a subsuite,
that test is removed from the default set and will only run if that subsuite
is explicitly specified. This will allow us to chunk python unittests out of
'make check' piecemeal. The default set will run in 'make check', and
individual tasks (e.g mozbase), will specify a subsuite explicitly.
The |mach python-test| implementation is slightly different. By default,
subsuites are not considered if developers do not pass in --subsuite. This
means running |mach python-test| without arguments will still run the full set
of tests, and similarly, passing in test paths will *just work*.
If for some reason a developer needs to actually run the default set, a special
"default" subsuite has been create, so they can use
|mach python-test --subsuite default|. This default subsuite is also what 'make
check' will explicitly invoke.
MozReview-Commit-ID: FaHb4nvuoK9
--HG--
extra : rebase_source : 2ecbc902bb6bafa7cd78ac0e380a10bad7c14351
This replaces the 'run-tests-deps' make target with a python function that will directly
read moz.build files, emit them with TestManifestEmitter, then consume them with
TestManifestBackend. Because the TestResolver is the only place that actually reads the
test metadata files, we can remove this logic from the CommonBackend as well.
MozReview-Commit-ID: DXgMoeH5dKf
MozReview-Commit-ID: HstZ57qkqf2
--HG--
extra : rebase_source : f377fa6863ef66d3adb86ed64f844e346686862f
Currently, the only way to emit objects after reading moz.build, is to emit everything. Though, sometimes
it may be desirable to only emit certain types of objects. This adds a new argument that allows consumers
to specify a custom emitter function. This gives them the flexibility to do whatever they want.
This will be used when resolving tests, so only TestManifest objects are emitted.
MozReview-Commit-ID: DPGgNmn2JvE
--HG--
extra : rebase_source : 044f8c8cad1bf7fd1dfbed05c89fbd114508182a
This is a drive by fix that is not relevant to the rest of the commit series.
MozReview-Commit-ID: Bwrb74o3Qh8
--HG--
extra : rebase_source : 34706a6640a6c704f9c03f24aede71e4c1c5ac1c
Currently the CommonBackend is responsible for processing TestManifest objects and using them to generate
the test metadata files (e.g all-tests.pkl et al). This patch pulls that logic out into a partial backend
specifically for test manifests.
This patch is solely a refactoring and shouldn't change any build behaviour. CommonBackend has a
TestManifestBackend instance and calls consume_object directly on it. However, this is just a temporary
measure to avoid checking in a broken commit.
This commit also adds a test for the 'test-defaults.pkl' file which was previously missing.
MozReview-Commit-ID: HOr2QVT8CJ1
--HG--
extra : rebase_source : e4b9cb982937381346c0821971191277173b165b
When vendoring third-party files, we'd like an explicit notice/review
when said files contain a "large file". This commit adds such checks
for files vendored via `mach vendor rust`.
As we don't yet have a server-side hook in place to prevent large files
from being added, we just have a command-line flag that people are
expected to use, on the honor system, to permit large files to be added
when vendoring.
This command is useful for users who wish to perform something like the
following:
1. Add/remove a bunch of files;
2. Examine the additions/removals/modifications for interesting changes;
3. Reject the add/remove if it doesn't meet some set of conditions.
The FileNotFoundError built-in exception is only present in
python 3. Emulate its behaviour in python 2 with a conditional
OSError.
MozReview-Commit-ID: 4b8THPG7jph
--HG--
extra : rebase_source : 718bf3659f14bd349d052d43bf3197dfbb4a016f
Bump the minimum version of the rust toolchain we require to
build. The 1.15 release includes support for custom #[derive]
directives, letting us use the serde serialization crate without
checking in a lot of generated code.
This is primarily motivated by webrender and the audio remoting
work, and lets us drop the heavy syntex dependency.
MozReview-Commit-ID: 6IObHhouPAn
--HG--
extra : rebase_source : 4be8b148fb653a48f6df4309811ab1d8755f7edf
Without this, clang-tidy cannot check any diffs including such files.
In the future, when checking the entire tree using clang-tidy, we will
ignore these entries in the compilation database.
The way directory traversal is computed relies on the
RecursiveMakeTraversal class, which is used to reproduce the old
traversal order from the old entirely-in-make traversal with DIRS,
PARALLEL_DIRS, etc. because of the undeclared intra-directory
dependencies that are looming here and there.
It's fed through DirectoryTraversal objects emitted by the frontend.
Normally, DirectoryTraversal objects are emitted for a directory,
possibly giving the subdirectories defined in DIRS/TEST_DIRS its
moz.build. But in the case of gyp processing, nothing places the gyp
objdirs in some virtual DIRS of some parent moz.build since bug 1308982.
As a consequence, the corresponding entries in the
RecursiveMakeTraversal instance attached to the backend are not attached
to any parent directory. When subsequently traversing the tree from the
root, they are never found, and end up being skipped, irregarding of
their actual _no_skip status.
It would probably be possible to revert the changes from bug 1308992,
but we might as well not rely on remains from the old ways. So instead,
we make the RecursiveMakeTraversal consider directories without a
declared parent attached directly to the root directory. They don't need
to depend on any other directory anyways.
--HG--
extra : rebase_source : 17403922322a71d490fdea8db0ff16b04983ed7a
CLOSED TREE
Backed out changeset 158233bce738 (bug 1197325)
Backed out changeset b5ac3fa0bbe7 (bug 1197325)
Backed out changeset 55a8ad127517 (bug 1197325)
We had been installing them to dist/plugins with the rest of the test plugins,
but tests are actually expecting them to be in dist/bin.
MozReview-Commit-ID: 9qYFgZA4Fni
--HG--
extra : rebase_source : 52a4cbf70ffe7998cfe81ab5e508b9f802dd3a35
Remove the option to build without rust code. We are not testing
this configuration and expect to land non-optional rust code in
the near future, so it doesn't make sense to maintain this option.
MozReview-Commit-ID: CwTlMXGvr5n
--HG--
extra : rebase_source : 080a9df5b4828c66aa2452ad1c16a503bcd5e689
This updates the client artifact code to locate test support files in the
common test archive and populate the objdir with these files appropriately.
MozReview-Commit-ID: GuXjwUtsl
--HG--
extra : rebase_source : 3560efee22533f60be1e7394fa0841005991cca6
Generated support files for specific tests generally end up in a harness specific
test archive, but artifact builds, which might not want to generate these files
themselves, get all of their test related files from the common.tests.zip. This
effectively moves a small number of these support files from harness specific
archives to the common test archive, making it possible to run tests that
depend on these files against artifact builds.
MozReview-Commit-ID: BvEj1ot1OTw
--HG--
extra : rebase_source : 06e958adc7b5667c50ca5ddbf06c1bf878d75737
Install manifests influence test packaging, allowing artifact builds to consume
generated test support files and run tests that depend on them. This commit
tracks binaries with an install target under "_tests" so they will be correctly
installed in artifact builds.
This could similarly be achieved by installing the binaries via TEST_HARNESS_FILES
rather than setting FINAL_TARGET, however, PGO builds will fail attempting to install
the files during the profile generation step, because SIMPLE_PROGRAMS are not built
for this step.
MozReview-Commit-ID: ES4bTxOoqMN
--HG--
extra : rebase_source : f41d8c47711fc15247fa336e2e9cbdedf57c2daf
Back when the class was written, for the packaging code, it made sense
that the default was True. But now that it's used all over the place,
and that the vast majority of uses are with find_executables=False, it
makes more sense for that to be the default.
--HG--
extra : rebase_source : ff813735fc0d53093f348f20eb77ee03e9b09d4e
And make it an error not to give it. While the default is True, we do
pass a value of False wherever it makes sense, which happens to be, in
most places.
This is an intermediate step to flip the default from True to False,
ensuring that we don't unwantedly switch some calls to False.
--HG--
extra : rebase_source : 432e03f032fef378af482581685583262e5d2661
We want to avoid giving --help dependencies to host and target, so that
they we don't spawn config.guess and config.sub when running configure
--help, and don't need to reach out to the which module to find a
suitable shell to execute them.
So, when --help is given, we return a fake host/target namespace, and
avoid the config.guess/config.sub-invoking code being executed.
Then, by giving the --help option to the linter, it can properly find
that the config.guess/config.sub-invoking code doesn't need the
dependency on --help.
This effectively unbreaks configure --help after bug 1313306.
--HG--
extra : rebase_source : 9ed4cbe5a81121b139d0d86aff6af542c2dff53b
Ideally, it would have been better if it were "or", but it's not
possible to override "or" in python ; __or__ is for "|".
This does feel magic, but it's also shorter than adding something like
@depends_any(), and while we're only adding "|" as of this change, we
can add other operations such as "&" in the future, or __getattr__ for
things like milestone.is_nightly.
An alternative form in moz.configure could require the @depends function
to be called, e.g. "a() | b()" instead of "a | b", but I'm not
particularly convinced that one is less magic than the other.
This feature is hooked up such that b is not resolved if a is true,
although in practice, it will still be resolved in Sandbox.run... but
not when --help is passed. In the long run, the forced resolution of
@depends functions will be removed from Sandbox.run.
--HG--
extra : rebase_source : 2389153acfbc3930786d5495ba4a04640325613e
Adding those dependencies, retrospectively, only worked around the poor
handling of --help requirements by the linter, that we fixed a few
commits ago. This is now not necessary anymore.
--HG--
extra : rebase_source : 0ba95ab2d246315be097fed7897db1c20666060a
Several things were wrong with the wrapping:
- the equality test on functions was actually comparing the memoized
functions, which have a type memoize, which inherits from dict. So
they weren't comparing actual functions, but the dict used to store
the cache of their invocation.
- each CombinedDependsFunction created for the same combination function
used a different wrapped function, so even if the dict problem wasn't
there, the equality test still wouldn't work, except if the function
wrapping itself was memoized.
- the memoization was not particularly useful.
Also, for upcoming changes, we'd actually like the combination function to
take an iterable instead of a variable argument list, so that items of
the iterable can be skipped.
--HG--
extra : rebase_source : 2c91889315b49695a2e2b709a9264de9ff237598
We're going to change the function signature for CombinedDependsFunction,
so make it visible in the API that the function member is not meant to
be used directly. The linter still does, though, because it needs to
look in their guts.
At the same time, avoid setting DependsFunction names via the function
name itself, because in upcoming changes, it will not be modifiable in
some cases.
--HG--
extra : rebase_source : a46ec8d3d2e2511dbb44a48cf2c02ab7ad190510
Options with a `when` argument (either directly, or inherited through
only_when() or an include) require --help per _value_for_option, but
that code path is not exercised during a lint pass.
With this change, along the previous one, we now correctly detect that
bug 1316957 was not supposed to work as is.
--HG--
extra : rebase_source : 0961c132bb200a90814b948aff7d2e80a1b21a30
Bug 1313306 relaxed the --help dependency requirement in some cases, but
while doing so, the requirement was also removed in other, unexpected
cases. Specifically, the --help dependency ended up not being required
on indirect dependencies that should have had it, had the --help
dependency been explicit on the direct dependency.
--HG--
extra : rebase_source : 81755988754798590482dfb297b7ddfc14ee88c7
It turns out there are shockingly few cases of manifestparser manifests that actually use the ';'
character as a comment. Because we will soon allow inline comments, deprecating the use of ';' will
ensure that values are allowed to have semicolons in them.
Even without inline comments, might as well enforce consistency across manifests.
MozReview-Commit-ID: AEPPQFdNXG0
--HG--
extra : rebase_source : 3540fa385f328bffb020c0a6debc4d2b3a90ed39
This is an educated guess in an attempt to solve a hard to reproduce intermittent. This
also makes sure the original SIGINT handler is restored after running the linters.
MozReview-Commit-ID: HRo0GRlSN5L
--HG--
extra : rebase_source : 7b8bf4b5818d741b5ab6a38c8c56f4eb53a801c8
If we have rustup installed, use it to check the available
target platforms and install 32-bit windows support if we're
on the (default 64-bit) windows platform.
This catches systems where the mozillabuild bootstrapper was
run before it installed this, so rustup is available, but the
i686 target library isn't.
MozReview-Commit-ID: 9bE2OQnmvxs
--HG--
extra : rebase_source : 0915fef85755718c5524b13b6b7d0d8b6dbe05b4
Windows devs often want to target 32-bit windows. Make this
easier by installing the target for them at mozboot time.
MozReview-Commit-ID: 6gFbFBOqMz8
--HG--
extra : rebase_source : 12933c6cb7895cef859c9022efa87c62f321219e
We need to mangle the path for both parts of the text printed
but rust_path_advice on windows. Otherwise the report of where
the rust binary was found ends up a mixed pathname even when
the suggested shell command is correct.
MozReview-Commit-ID: FDtP5HY8tJ1
--HG--
extra : rebase_source : ad3671be04751ece8966fa19267239b9f1614551
Adjust newlines so prompt and result information is easier to read.
MozReview-Commit-ID: BbJldqZ6G4
--HG--
extra : rebase_source : 50781f672c452e97c47d4ae07aa7f8b67a96ec50
We're going to remove the xpcom glue, so there is no need to check that
nothing depends on it anymore.
MozReview-Commit-ID: IoTo2b9DtiL
--HG--
extra : rebase_source : 03d1a01af802a7d0833f2536e1dcc0180696148c
extra : source : ad4e531c7070874770201b5af956f3f8045c973b
We're going to remove the xpcom glue, so there is no need to check that
nothing depends on it anymore.
--HG--
extra : rebase_source : 4323d9c388c60258c581771ac3bc5aa2100ea699
Here, we also modify APKOpen to use the XPCOM glue loading process
instead of custom symbol resolution, so that the Bootstrap API can be
used in a more straightforward manner.
--HG--
extra : rebase_source : 55037ba30ca66a090b73923a3ce8df5b054bf47a
This should fetch the artifacts necessary to run tests and extract
them correctly.
While we're here, clean up some other regular expressions which
weren't escaped consistently.
Also, if you're going to use backslashes in literal strings because of
regular expressions, r-strings are probably better. (Otherwise you
should double all the backslashes.)
All the same, try to stop short of going full PEP8 without express
permission.
MozReview-Commit-ID: 8S1PVHX9xEZ
--HG--
extra : rebase_source : a011f4bf99842bcb897b405ea7810143e93acec9
We're going to remove the xpcom glue, so there is no need to check that
nothing depends on it anymore.
--HG--
extra : rebase_source : 4323d9c388c60258c581771ac3bc5aa2100ea699
Originally landed from https://reviewboard.mozilla.org/r/88220/#review88512 on date. With the following message:
Bug 1312585 - Vendor chunkify r=gps r=bhearsum
for - Set l10n repacks to pass in explicit list of locales, rather than having mozharness do chunking
I'm vendoring from bhearsum's repo, extra r=him so he is aware of its in-tree home.
MozReview-Commit-ID: 50Ub24kyTHX
--HG--
extra : rebase_source : c07870fe309b71ca77b8f1fd3598c6f6677acf88
extra : source : b0a0539d9d55b3b54fd024afbadbf687616a9443
Work around missing redhat-hardened-cc1 when building psutil_linux
on Fedora by installing the redhat-rpm-config package.
This fixed build warnings with several mach commands, and errors
with others like `./mach mochitest` and `./mach install` for fennec
builds.
MozReview-Commit-ID: G9jn4abUEtp
--HG--
extra : rebase_source : 98337b820fff52e2efd2368e89f7ff488b36f1ff
Should be pretty safe as Java compa is usually very good.
This will fix the issue on Debian testing not having openjdk-7-jdk and
current Debian stable having it.
(same with Ubuntu)
MozReview-Commit-ID: Alxg4K4PwQ4
--HG--
extra : rebase_source : 920cdb1618ba587a4776e33ef7857415a59c53b9
In bug 1308472 we are seeing 'make -k check' fail intermittently with
the only apparent error message something like:
make: *** [check] Error 245
This debug should make it more clear which test (if any) is responsible
for setting the return code in 'mach python-test', and whether or not
'mach python-test' is actually reaching the end of the function.
MozReview-Commit-ID: 6IQrZQqs8ij
--HG--
extra : rebase_source : 4d2d4b8b139d7f16bda2b22ce79ab2c84e8fe4c7
On windows, python generally returns windows-style paths,
with drive letters and backslash for a separator. However,
when we offer advice for updating the PATH variable, we're
talking about the msys environment which uses unix-style
paths. Convert to avoid confusion.
This is intended to turn c:\Users\mozilla\ into /c/Users/mozilla/.
MozReview-Commit-ID: FdB8FvjeCV1
--HG--
extra : rebase_source : 6d9e87b460417254bbe2eb5af3813e22f2126fb1
This provides build-completion notification from mach.
We already do this on debian-based distros.
MozReview-Commit-ID: Jl3OWa9MpZ6
--HG--
extra : rebase_source : e97c02d2924f888b33593ad5209cedaccceba633
Update the version number and checksums of the rustup
installer we download to 1.0.0. This had a first stable
release alongside rust 1.15.
This is the result of running `python mozboot/rust.py --update`
and applying the resulting output.
MozReview-Commit-ID: 1gzMLHZuhNx
--HG--
extra : rebase_source : b9d0f95f17e76a32e17e82d05505cf07a21c5e66
