Also removes InstallManifestNoSymlinks which can be more simply expressed by
passing link_policy='copy' to InstallManifest.populate_registry.
MozReview-Commit-ID: Bkjc2hIub4A
--HG--
extra : rebase_source : df05080fef3baf7e6c60ff9a468e71f0b2c67462
In bug 1181040, we added ${var}_IS_SET variables for
mk_add_options-defined variables. In the two years since, that has never
been used for anything else than MOZ_PGO_IS_SET, and the only use for
that has now been removed, so remove those ${var}_IS_SET variables.
--HG--
extra : rebase_source : 2fc9abe0c3badbf06f3858fcf326237e67891fee
Chain of trust validation will require to know what the inputs for a
given build are, and mach artifact toolchain fetches such inputs.
So we make it output a manifest that the chain of trust validation
process will be able to use and correlate with other information, such
as the one from bug 1383993.
At the same time, we make the produced manifest contain information
about tooltool-downloaded packages, which will allow to track the
progress in the migration from tooltool to TC artifacts.
--HG--
extra : rebase_source : 5b3fc32a9fd641cc7edc57865d2b60aaa6ffcbed
Adding PROG_IS_C_ONLY seems like a good point to add tests, and once we
have tests for that, adding tests for the existing library support is
not too difficult.
Similar to the existing LIB_IS_C_ONLY variable, these variables indicate
that the program in question has only C sources and so can be linked by
the C compiler rather than the C++ compiler. We need to add a little
more information to BaseProgram so we can avoid emitting periods into
Makefile variables.
Bug 1374940 adds a MOZ_TOOLCHAINS environment variable with a list of
path@task-id strings, where task-id is corresponding to the (possibly
optimized) toolchain job, and path corresponding to the
toolchain-artifact defined for that toolchain job.
We want to use that to pull artifacts instead of tooltool packages.
--HG--
extra : rebase_source : 277daa2c83d6d197975cb4ef36ee131176afa992
We now store HG or GIT in substs. We don't need to search for
binary paths.
MozReview-Commit-ID: 8sSgPNLok9M
--HG--
extra : rebase_source : bc51087bcb9f2a723e27f240dd06a88540f6d8a8
We now have a variable in config.status for recording the checkout
type. These helper functions for determining if we're Mercurial or Git
can now be one-liners.
As a bonus, we no longer do I/O as part of this function.
MozReview-Commit-ID: HT9sbOhDEkf
--HG--
extra : rebase_source : 8b53b5f50d14c0bdd4ef3dc7b190314af80a76f0
We now store HG or GIT in substs. We don't need to search for
binary paths.
MozReview-Commit-ID: 8sSgPNLok9M
--HG--
extra : rebase_source : bc51087bcb9f2a723e27f240dd06a88540f6d8a8
We now have a variable in config.status for recording the checkout
type. These helper functions for determining if we're Mercurial or Git
can now be one-liners.
As a bonus, we no longer do I/O as part of this function.
MozReview-Commit-ID: HT9sbOhDEkf
--HG--
extra : rebase_source : 8b53b5f50d14c0bdd4ef3dc7b190314af80a76f0
Having Rust dump a stack on panic seems like a useful on-by-default
feature.
MozReview-Commit-ID: ABYTArsMTFh
--HG--
extra : rebase_source : 7d15f44a9ffd14db475db9e5c2964aca31bf0f70
This patch changes the name of the keys that are in the 'linked-files-map.json' that is produced in the code coverage build and are used to map symbolic links to their source files. The new key names (which are the paths to the symbolic links) are now the entire absolute path to each of the files.
MozReview-Commit-ID: 4x1dfk9h2Ov
--HG--
extra : rebase_source : 7d424bbbf1d026ea67c66b743c8c43ea75185733
This is similar to bug 1301751, where something in rust seems to trigger errors
running dsymutil to generate debug symbols in OSX. We can set debuginfo=1 for
these builds as a temporary workaround for now, while we work on a more
permanent solution in rust and/or dsymutil. debuginfo=1 still gives us enough
info for stack traces, although without line info. debuginfo=2 would be useful
for debugging, but is irrelevant to crash reports.
MozReview-Commit-ID: DdA00GzVfWg
--HG--
extra : amend_source : 47d3573042098194a07f9b473e4a02c86a1eba7c
The rigid type comparison added in 51a92a22d6d1 (bug 1375231) was
too rigid. This broke at least one consumer that was comparing an
empty PositiveOptionValue/NegativeOptionValue against a string.
Since comparisons against empty OptionValue are convenient and
don't violate the spirit of the type checking previously added,
this commit relaxes the type equivalence check in cases where the
OptionValue is empty.
MozReview-Commit-ID: UllTrzCjj
--HG--
extra : rebase_source : 2c41428d1be667edecdab0947d006a1a6a533569
OptionValue and its derived classes are exposed to moz.configure
files. As the previous bug fix showed, it is really easy to
accidentally assume the type is a simple string value and do a
string compare against it.
To prevent this class of bugs, this commit adds additional type
awareness to OptionValue.__eq__.
We first check that the argument is a tuple (including any OptionValue
types). If not, we raise a TypeError because the comparison is
invalid. This is arguably a violation of __eq__. But since OptionValue
is exposed to the moz.configure sandbox and typing '==' will invoke
__eq__, we have to do something to prevent this foot gun.
The change also changes the comparison logic.
If we compare against a non-derived tuple instance, we do a tuple
compare. Otherwise, we fall back to the previous logic of
requiring an identical type then doing a tuple compare.
MozReview-Commit-ID: 7IVSL2Asg9j
--HG--
extra : rebase_source : edab573834da240df9ad7c2fc78c85d6159a6ef9
Leaving off this argument makes `mach vendor rust` with large files fall
over with a Python error. While the user still gets a semi-useful error
message prior to this failing, it would be better to not fail here at all.
Cargo recently introduced the `cargo check` command for shortening the
edit-compile cycle when working on large programs. Since we don't
really support invoking `cargo` directly, let's wire up this command to
`mach`. Gecko developers can then `mach cargo check` to ensure their
changes typecheck.
We have a flag set on all Linkables, cxx_link, denoting whether there's
anything being linked into them that requires C++. We do this even when
we link against shared libraries that required C++. But if these
libraries don't export C++ interfaces, there's no reason that the things
linking against them should require C++. Therefore, ignore shared
libraries when making the determination of whether an object requires
C++ or not.
This patch renames the mozinfo flag 'coverage' to 'ccov' to avoid ambiguity in whether a test is being skipped for linux64-ccov or for linux64-jsdcov. It also removes the 'runtests.py' mozinfo hack and renames all occurrences of 'coverage' that are used for skipping tests in linux64-ccov.
MozReview-Commit-ID: IF2640bDQP7
--HG--
extra : rebase_source : 614020325e30d1ca9e01aaf08479b8a4ffaec888
This patch adds a flag to the 'mozinfo.json' that can be used to disable tests when they are running on linux64-ccov. Then, this flag is used to prevent the marionnette test 'test_crash.py' from running on linux64-ccov.
MozReview-Commit-ID: 9IHMiZHxcMK
--HG--
extra : rebase_source : ec690cb3ffa27d3e88d2c0b8c5d510e72a5c5079
The -fsanitize=integer analysis from UBSan can be helpful to detect signed and unsigned integer overflows in the codebase. Unfortunately, those occur very frequently, making it impossible to test anything with it without the use of a huge blacklist. This patch includes a blacklist that is broad enough to silence everything that would drain performance too much. But even with this blacklist, neither tests nor fuzzing is "clean". We can however in the future combine this with static analysis to limit ourselves to interesting places to look at, or improve the dynamic analysis to omit typical benign overflows.
It also adds another attribute that can be used on functions. It is not used right now because it was initially easier to add things to the compile-time blacklist to get started.
Finally, it includes a runtime suppression list and patches various parts in the test harnesses to support that. It is currently empty and it should not be used on frequent overflows because it is expensive. However, it has the advantage that it can be used to differentiate between signed and unsigned overflows while the compile-time blacklist cannot do that. So it can be used to e.g. silence unsigned integer overflows on a file or function while still reporting signed issues. We can also use this suppression list for any other UBSan related suppressions, should we ever want to use other features from that sanitizer.
MozReview-Commit-ID: C5ofhfJdpCS
--HG--
extra : rebase_source : 952043a441b41b2f58ec4abc51ac15fa71fc142f
Support OSX Signed nightlies (in the complete.mar too)
MozReview-Commit-ID: HXiFGE14wYJ
--HG--
extra : rebase_source : 1d02b4714c8fafe6cdcd74e6d9b5612c44dcb3b4
- Set the extensions.legacy.enabled pref for mochitests etc
- Skip a plugin-inside-xpi test for now if legacy extensions
are force-disabled. That test can just be removed once we
get to 57.
MozReview-Commit-ID: As9LtkQTcTS
--HG--
extra : rebase_source : fcc84daef95c453e893cc3b98498fdb87f54b1bb
This adds a py_action invocation wrapping aapt and implements a hacky
implementation of the Gradle build system's resource merging
algorithm; once we have the moz.build and Gradle resources identical,
we'll be one big step closer to producing bit-identical builds and
flipping the switch in favour of Gradle. With this, the R.txt
produced by the aapt invocation is the same as the R.txt produced by
the py_action invocation.
Originally I wrote this to use GENERATED_FILES, but it produced a
world of pain. Since Android's aapt tool is fundamentally directory
oriented, not file oriented, it required adding support for FORCE to
GENERATED_FILES and required directory crawling and FileAvoidWrite in
the wrapper. After getting that working I was eventually stymied by
the arcane requirements of the Android re-packaging system, which
interacts with the l10n system. I would have required support for
building GENERATED_FILES in the libs tier rather than the misc tier.
After that realization I gave up and turned to py_action: the
dependencies on branding are just too entangled with l10n to use
GENERATED_FILES.
And, in the not-so-distant future, all of this moz.build and
Makefile.in chicanery will be deleted in favour of invoking Gradle at
the appropriate points!
MozReview-Commit-ID: 4ueVNa7gzgs
--HG--
extra : rebase_source : dab092a188bc735ef819a4be0ad13387e85c87e2
extra : source : a05bd87b09ee5e4cff20fe84c2e75ac3b2a494a1
This prevents us from redundantly installing httpd.js and httpd.manifest
from the test package during an artifact build, which interferes with
the Tup backend's handling of these files as symlinks.
MozReview-Commit-ID: LuMurUc1P36
--HG--
extra : rebase_source : 1aabd788ff71ae28434a4076d5304f611ada5d92
As mozprocess doesn't have any special handling of stderr, should cargo
operations fail their output is dropped. Switch to subprocess.check_call to
ensure cargo errors are displayed to the caller.
If we don't do this, various bits of test infrastructure will turn on
when stylo is merely built, not enabled, which will cause no end of
orange and unhappiness.
Before, MSVS was set in old-configure and could only be unset or
"2015." We move the definition of the variable to moz.configure
and support defining its value as "2017" when VS2017 is being used.
As part of this, I discovered that GYP barfs with a "2017" value.
This is likely a limitation of the legacy version of GYP we have
vendored. Rather than go down the rabbit hole of upgrading GYP,
I added code to convert the value to "2015." This preserves existing
behavior and unblocks us from setting MSVS_VERSION properly. A
warning is emitted to remind us to remove this hack once GYP is
upgraded.
After this commit, we now generate native VS2017 solutions and
projects when building with VS2017.
MozReview-Commit-ID: BvNJX3F8qCn
--HG--
extra : rebase_source : 13a495856a83d9ca7afbb4770ebab1cc7f651cae
This script uses the info generated by the UrlMap backend to replace chrome urls
found in lcov files with source-paths. There are several other cases a script
might not correspond to a source url, such as "data:" uris, which are handled
by omitting those sections in the produced lcov files. The set of files included by a JS source (if it is preprocessed) is also calculated, but not consumed
at this time. A more extensive lcov rewriting mechanism will be added in the
future to re-construct source information in produced coverage reports.
MozReview-Commit-ID: DDcreOdEbam
--HG--
extra : rebase_source : a8cff598d8ed6a27838bf4e5f83c664b58f867c5
The -fsanitize=integer analysis from UBSan can be helpful to detect signed and unsigned integer overflows in the codebase. Unfortunately, those occur very frequently, making it impossible to test anything with it without the use of a huge blacklist. This patch includes a blacklist that is broad enough to silence everything that would drain performance too much. But even with this blacklist, neither tests nor fuzzing is "clean". We can however in the future combine this with static analysis to limit ourselves to interesting places to look at, or improve the dynamic analysis to omit typical benign overflows.
It also adds another attribute that can be used on functions. It is not used right now because it was initially easier to add things to the compile-time blacklist to get started.
Finally, it includes a runtime suppression list and patches various parts in the test harnesses to support that. It is currently empty and it should not be used on frequent overflows because it is expensive. However, it has the advantage that it can be used to differentiate between signed and unsigned overflows while the compile-time blacklist cannot do that. So it can be used to e.g. silence unsigned integer overflows on a file or function while still reporting signed issues. We can also use this suppression list for any other UBSan related suppressions, should we ever want to use other features from that sanitizer.
MozReview-Commit-ID: C5ofhfJdpCS
--HG--
extra : rebase_source : 64aa804965d24bb90b103c00c692a2ac6859e408
This happen to uncover the fact that mingw clang was not handled
properly after bug 1372987. This will allow us to substantially modify
the function that handles them and avoid regressions.
--HG--
extra : rebase_source : 007257caecf7917480ef6b0a834c304768f77591
This adds a mach repackage command to create a complete mar file from an
input package and mar host executable. The standard make_full_update.sh
script actually does the work of creating the complete mar.
--HG--
extra : rebase_source : 9e84e8d22d2347be459cc5cbd3d94286462dff0d
These commands are similar to 'mach repackage dmg', except the installer
and stub installer require multiple inputs. The regular installer needs
a package, tag file, and setup.exe file; the stub installer needs just a
tag file and setup.exe file. The output is a self-extracting .exe file.
The archive_exe() function originally just supported the regular
installer, but by supporting an optional package, the function can
support the stub installer as well.
--HG--
extra : rebase_source : 2c6fe21534ec60bb03e4431bccf524704e64a978
With multiple types of repackaging, it will be simpler to manage the
various commands if they are organized as sub-commands. This way the
different repackage types can take varying arguments, and we don't have
to guess on the repackage type based on the output filename.
MozReview-Commit-ID: BknRPAwFG5H
--HG--
extra : rebase_source : e74a9776a73791e41ada4038a8bb2ddd243d0de8
This was not strictly necessary because they would appear as
dependencies of other things in the compile graph, but it makes it
easier to iterate the compile graph for all the node.
Incidentally, this makes the rust libraries on par with rust programs.
--HG--
extra : rebase_source : 3c672308478d3ef3dd38b9036f7b1862abcf3f97
This was not strictly necessary because they would appear as
dependencies of other things in the compile graph, but it makes it
easier to iterate the compile graph for all the node.
Incidentally, this makes the rust libraries on par with rust programs.
--HG--
extra : rebase_source : 3c672308478d3ef3dd38b9036f7b1862abcf3f97
The MacOS client artifact code relies on its own bookkeeping to install
binaries to their final locations and assumes that files installed to
'Contents/MacOS' will generally be installed to dist/bin by the build system.
This commit extends this logic to account for minidump-analyzer, which is
installed to directly to crashreporter.app/Contents/MacOS by setting DIST_SUBDIR
in moz.build.
Ideally we would rely on build system metadata to install binaries in artifact
builds, however this is non-trivial as the relevant moz.build data isn't
read in non-compile-environment builds and originating builds do not expose it
directly.
MozReview-Commit-ID: BMmhtx1HKn7
--HG--
extra : rebase_source : ec02e7664dcd9fea4f035483e8856fa62966c1cb