Having `rustc` be `rustup`'s wrapper for `rustc` means that we may
silently honor `rustup`'s override mechanisms. We noticed this first on
OS X, where we use the "real" `cargo` but `rustup`'s `rustc` wrapper,
and problems ensued when `cargo` thought it was using one version of
`rustc`, but actually wound up using something different.
It seems better to avoid silently interposing `rustup`'s toolchain
override mechanisms everywhere, rather than having to special-case OS
X. So let's factor out a general mechanism for removing the wrappers
`rustup` provides and use that for both `rustc` and `cargo`. The tests
need adjusting because we weren't triggering the unwrapping cases
before; we don't yet test the case where we really do need to unwrap.
That test can be left for a future patch.
Differential Revision: https://phabricator.services.mozilla.com/D29531
--HG--
extra : moz-landing-system : lando
We've not been checking the clang-cl version in use. This lack of
checking is bad, for a couple of reasons:
* Released versions of clang-cl differ drastically in their robustness;
* Only the most recent version of clang-cl supports aarch64.
We should check for a minimum version of clang-cl, just like our other
supported compilers. As a bonus, we can then start depending on
features that we know appear in the particular minimum clang-cl
version. (The current patch is motivated by `/clang:` command-line
support, but one could pick other things.)
Differential Revision: https://phabricator.services.mozilla.com/D30723
--HG--
extra : moz-landing-system : lando
Also for checker `modernize-avoid-bind` export the respective reliability index from config.yaml
Differential Revision: https://phabricator.services.mozilla.com/D30904
--HG--
extra : moz-landing-system : lando
(adds __build_class__ to the builtin sandbox and allow CombinedDependsFunction to be hashed)
Differential Revision: https://phabricator.services.mozilla.com/D28114
--HG--
extra : rebase_source : 967a394d10d41e3da322005c84757c001a3b74cb
extra : intermediate-source : 2be63fc923997e6865ab9abc4fa7fb728e55d532
extra : source : aed41a0c652ae3f8e4b4f76bfaa0c56130515a75
Also adjust 'basestring' usage in some places.
Differential Revision: https://phabricator.services.mozilla.com/D28102
--HG--
extra : rebase_source : f0fecd6b040336b0cf2f055e266fe37eea5a9a48
extra : intermediate-source : 712718250f0ba5c286c1fb6b6c7f370b713c7225
extra : source : 6b24ce2da518d8158e13627adb11c9fcad19c92a
shellutil.py needed special handling since it is invoked earlier than the
vendor directory adding we do, so it does not have six available.
Differential Revision: https://phabricator.services.mozilla.com/D28096
--HG--
extra : rebase_source : c5c611d280dc821747c9db736e4943c2cc421557
extra : intermediate-source : 30fb268b41fdc45937a22ab446a8805fb099c1f0
extra : source : d102292b6b5b812e7df165116410488d18042bd3
0.1.21 mishandles cargo package renames, which are a required
feature for Rust 2018 support. The latest version fixes this.
Differential Revision: https://phabricator.services.mozilla.com/D29946
--HG--
extra : moz-landing-system : lando
Under the hood, browsertime invokes a certain `visualmetrics.py`
script. That script depends on `ffmpeg` and ImageMagick's `convert`,
`compare`, and `mogrify` commands. It also depends on certain Python
packages.
So this installs those dependencies, and then wires up the evaluation
environment such that `./mach browsertime` can find the dependencies.
It also adds a `./mach visualmetrics` command for processing a
captured MP4 file in the same way that browsertime processes such a
file.
In order to avoid downloading dependencies multiple time, the existing
artifact cache is extracted. This is a small first step towards [Bug
1526021](https://bugzilla.mozilla.org/show_bug.cgi?id=1526021), which
might want to use this artifact cache as well.
At this time, hashes and filesizes are not verified. During
development, the upstream files changed multiple times, and it's not
worth being completely locked down while experimenting with this
functionality. If we start running this code in automation or in more
sensitive environments, we can build fetch tasks and TC indexes to
streamline the artifact gathering process.
It is expected that a future mach command will want to invoke
browsertime without suffering the overhead of invoking Python (and
mach, which is itself bulky) so a nod is given to exposing the
relevant environment pieces.
During testing, it was discovered that [MozillaBuild doesn't ship
git](https://bugzilla.mozilla.org/show_bug.cgi?id=1503028), so that
git repositories can't be used out-of-the-box on Windows. So instead
we use a [tarball link from github.com/$USER/$REPO/tarball/$COMMIT-LIKE](https://github.blog/2008-03-03-tarball-downloads/).
Differential Revision: https://phabricator.services.mozilla.com/D29442
--HG--
rename : python/mozbuild/mozbuild/test/test_artifacts.py => python/mozbuild/mozbuild/test/test_artifact_cache.py
extra : moz-landing-system : lando
Some media libraries use gas syntax in their assembly files. Rather than
converting these arm assembly syntax files for aarch64, we can use clang-cl
to build them directly.
Differential Revision: https://phabricator.services.mozilla.com/D27785
--HG--
extra : moz-landing-system : lando
Lint python/mozbuild/{mozbuild,mozpack}.
This makes sure we use byte strings (type(b'')) in many places.
This also has a few places where we know we want unicode, by enforcing it.
This code also has a few places where we call into windows API's which expect either all args to be unicode,
or all args to be bytestrings (not-unicode) so make sure those calls align.
This is the most risky of the stack, since there are some mach commands and codepaths that are neither excercised in automation
nor excercised in tests.
Differential Revision: https://phabricator.services.mozilla.com/D26645
--HG--
extra : rebase_source : 1ee367576bede0318a2db05644693d4f440a250a
extra : intermediate-source : 2516c350cba6f27eb204f0087728721abbb0e085
extra : source : 6351a29c5765c13c39ef6186f9e5e911e00420e7
Leaving one fix for an explicit review.
Lint python/mozbuild/{mozbuild,mozpack}. r=#build
Differential Revision: https://phabricator.services.mozilla.com/D26641
--HG--
extra : rebase_source : e14d7762c7802d98e24ee8f86b0167d96666d288
extra : intermediate-source : 79b0d1678dd2f6aadf60f480d902ddeca4a52c3a
extra : source : faadf440e7d7b0a8c473e2dee3fc90b27023814a
Lint python/mozbuild/{mozbuild,mozpack}.
ran './mach lint -l py2 -l flake8 -l shellcheck -l codespell -l yaml python/mozbuild/{mozbuild,mozpack}/ --fix' in order
to undo some black changes and get closer to making this folder able to be validated on every lint run
Differential Revision: https://phabricator.services.mozilla.com/D26640
--HG--
extra : rebase_source : 6b69b6ebcac73835b752607f7b5b4429de7f95cf
extra : intermediate-source : 114355f83d36188fac592c8c6497242c71b27ad6
extra : source : 51fbdf63bbce8f75fac407de305f826dc775fcb5
With this change tooltool.py also supports taskcluster credentials to be passed
(in json format) to --authentication-file option. RelengAPI tokens are still
working with this patch, just additional authentication is added.
Differential Revision: https://phabricator.services.mozilla.com/D27881
--HG--
extra : moz-landing-system : lando
The noatime option doesn't appear to have a significant impact on no-op
build times, even though the build system does a significant number of
file reads. Since macs have atime enabled by default, we should just
disable this check rather than give developers unhelpful error messages.
Differential Revision: https://phabricator.services.mozilla.com/D28906
--HG--
extra : moz-landing-system : lando
For some reason, using `tar -Jxf` with Windows paths fails, and that makes
`mach artifact toolchain` unable to pull .tar.xz archives on Windows.
However, executing `xz -d -c` works, and we can feed its output to
python's tarfile.
Differential Revision: https://phabricator.services.mozilla.com/D28779
--HG--
extra : moz-landing-system : lando
With this change tooltool.py also supports taskcluster credentials to be passed
(in json format) to --authentication-file option. RelengAPI tokens are still
working with this patch, just additional authentication is added.
Differential Revision: https://phabricator.services.mozilla.com/D27881
--HG--
extra : moz-landing-system : lando
Add basic support for 'mach gtest' on Android.
Handling of Android-only and desktop-only options is awkward; I hope to
re-visit this after bug 1519369.
Differential Revision: https://phabricator.services.mozilla.com/D28129
--HG--
extra : moz-landing-system : lando
`caskroom/versions` was replaced with `homebrew/cask-versions` in 2018.
Tap `caskroom/versions` instead of the old one.
If you have two taps, remove the old one using:
brew untap caskroom/versions
Differential Revision: https://phabricator.services.mozilla.com/D27839
--HG--
extra : moz-landing-system : lando
If the user runs mach clang-format from the builddir, it was finding and using
unified files.
Differential Revision: https://phabricator.services.mozilla.com/D27750
--HG--
extra : moz-landing-system : lando
The TestManifestBackend always has a dependency on config.status, even
if configure hasn't been run yet. If configure is run later, this is
intended to cause the TestManifestBackend to be regenerated so that it
can use the configured values instead of a potentially incomplete setup.
Mozbuild's backend_out_of_date() logic treats a missing file as always
being out of date, which means if configure hasn't run, the
TestManifestBackend will always be regenerated.
Instead we can create a stub config.status file that will fix the
out-of-date check, and can be treated as a
BuildEnvironmentNotFoundException as if the file didn't exist at all.
Differential Revision: https://phabricator.services.mozilla.com/D27195
--HG--
extra : moz-landing-system : lando
Bug 1537574 moved some code that uses errno from building.py to base.py,
but the latter didn't already import errno.
Differential Revision: https://phabricator.services.mozilla.com/D27125
--HG--
extra : moz-landing-system : lando
If mozbuild parsing fails due to a missing file (eg: a file not existing
in UNIFIED_SOURCES), then no Makefiles are written out, but
config.status exists. This would cause mozbuild to think that configure
doesn't need to run, and rely on make to perform the backend-out-of-date
check in rebuild-backend.mk. Unfortunately since no Makefiles were
written, the make command fails immediately and no attempt is made to
re-create the backend. Note that this is only a problem if the first
mozbuild parsing from a clobber build fails, otherwise there is
typically a top-level Makefile from a previous build to call into (at
which point make can determine it is out-of-date, and re-invoke itself).
The fix is to have the RecursiveMake backend re-use the same logic that
was introduced into mozbuild for alternate backends, and remove
rebuild-backend.mk. This way, mozbuild can always determine if the
backend needs to be regenerated, even if the initial parsing failed.
Test code was also relying on rebuild-backend.mk to generate the
TestBackend, but moving backend_out_of_date() into MozbuildObject allows
this code to be shared.
Differential Revision: https://phabricator.services.mozilla.com/D26262
--HG--
rename : build/gen_test_backend.py => python/mozbuild/mozbuild/gen_test_backend.py
extra : moz-landing-system : lando
If there are a large number of untracked files in the working directory, hg
will attempt to print them all out with the default pager. This does not
interact very will with commands that are built atop this functionality. We
set HGPLAIN=1 so that the underlying hg will not attempt to use a pager.
Differential Revision: https://phabricator.services.mozilla.com/D26607
--HG--
extra : moz-landing-system : lando
Before bug 938437, we had a rather large and error-prone
nsStaticXULComponents.cpp used to register all modules. That was
replaced with clever use of the linker, which allowed to avoid the mess
that maintaining that file was.
Fast forward to now, where after bug 1524687 and other work that
preceded it, we have a much smaller number of remaining static xpcom
components, registered via this linker hack, and don't expect to add
any new ones. The list should eventually go down to zero.
Within that context, it seems to be the right time to get rid of the
magic, and with it the problems it causes on its own.
Some of those components could probably be trivially be converted to
static registration via .conf files, but I didn't want to deal with the
possible need to increase the number of dummy modules in XPCOMInit.cpp.
They can still be converted as a followup.
Differential Revision: https://phabricator.services.mozilla.com/D26076
--HG--
extra : moz-landing-system : lando
If mozbuild parsing fails due to a missing file (eg: a file not existing
in UNIFIED_SOURCES), then no Makefiles are written out, but
config.status exists. This would cause mozbuild to think that configure
doesn't need to run, and rely on make to perform the backend-out-of-date
check in rebuild-backend.mk. Unfortunately since no Makefiles were
written, the make command fails immediately and no attempt is made to
re-create the backend. Note that this is only a problem if the first
mozbuild parsing from a clobber build fails, otherwise there is
typically a top-level Makefile from a previous build to call into (at
which point make can determine it is out-of-date, and re-invoke itself).
The fix is to have the RecursiveMake backend re-use the same logic that
was introduced into mozbuild for alternate backends, and remove
rebuild-backend.mk. This way, mozbuild can always determine if the
backend needs to be regenerated, even if the initial parsing failed.
Test code was also relying on rebuild-backend.mk to generate the
TestBackend, but moving backend_out_of_date() into MozbuildObject allows
this code to be shared.
Differential Revision: https://phabricator.services.mozilla.com/D26262
--HG--
rename : build/gen_test_backend.py => python/mozbuild/mozbuild/gen_test_backend.py
extra : moz-landing-system : lando
This is the only <radiogroup> and <radio> consumer in browser.xul.
By making the DOM get constructed on demand, we can avoid connecting
radios at startup, and it also gives us a chance to migrate the
strings to Fluent.
Differential Revision: https://phabricator.services.mozilla.com/D24914
--HG--
extra : moz-landing-system : lando
This patch introduces a new marionette media test along
with a Youtube test.
To run the Youtube streaming test locally:
./mach marionette-test dom/media/test/marionette/test_youtube.py -vv --gecko-log -
Differential Revision: https://phabricator.services.mozilla.com/D23644
--HG--
extra : moz-landing-system : lando
Keep a checksum to avoid re-extracting artifacts over and over.
In the future we can also check the checksum to avoid downloading the artifact
altogether, maybe.
Differential Revision: https://phabricator.services.mozilla.com/D25542
--HG--
extra : moz-landing-system : lando
Bootstrap is happy with rust version 1.32, but that version of rust
doesn't support the thumbv7neon target that we're trying to install
since bug 1539005. So install the armv7 target when rust is version
1.32.
Differential Revision: https://phabricator.services.mozilla.com/D25683
--HG--
extra : moz-landing-system : lando
This patch introduces a new marionette media test along
with a Youtube test.
To run the Youtube streaming test locally:
./mach marionette-test dom/media/test/marionette/test_youtube.py -vv --gecko-log -
Differential Revision: https://phabricator.services.mozilla.com/D23644
--HG--
extra : moz-landing-system : lando
Bootstrap is happy with rust version 1.32, but that version of rust
doesn't support the thumbv7neon target that we're trying to install
since bug 1539005. So install the armv7 target when rust is version
1.32.
Differential Revision: https://phabricator.services.mozilla.com/D25683
--HG--
extra : moz-landing-system : lando
We produce two types of build symbol archives in automation:
- "crashreporter-symbols.zip" contains Breakpad-format .sym files
- "crashreporter-symbols-full.zip" contains Breakpad-format .sym files
and compressed ELF debug symbol .dbg.gz files
Right now, `--enable-artifact-build-symbols` from Bug 1305502
downloads only "crashreporter-symbols.zip".
The Android Studio version of lldb, currently 7.0.0, doesn't support
Breakpad-format .sym files. It does support (uncompressed) ELF debug
symbols. (Note that gdb isn't supported on Android and hasn't been for
some time.)
This makes ` --enable-artifact-build-symbols` download the full
symbols for Android builds that aren't in automation, to be useful for
debugging Android builds with lldb locally.
Differential Revision: https://phabricator.services.mozilla.com/D19091
--HG--
extra : moz-landing-system : lando
It's not helpful to have a DEBUG log level, 'cuz the -v flag to
`./mach artifact install` doesn't turn on DEBUG logging.
Differential Revision: https://phabricator.services.mozilla.com/D19090
--HG--
extra : moz-landing-system : lando
This patch removes the packages which are obviously unnecessary, either
because they're entirely unused (e.g., imake), or because they're built
in-tree (e.g., nss, icu, hunspell) by default.
xorg-server-xvfb is arguably also not necessary, but is probably useful enough
to keep. Some of the others I'm unsure about, so I've left as well.
Differential Revision: https://phabricator.services.mozilla.com/D28140
--HG--
extra : rebase_source : 948a9056091f3004da02fe110de3930ee454f1ac
extra : histedit_source : 44320ee6197a2ca5702021fe21663ad2bc9f15f7%2Ce09f30ffc4d432587fef610d6d219b4151a010f5
When an added file is not under an app or addon directory, its base
directory is ''. The code added in bug 1539355 would skip doing its
thing in that case, which it shouldn't.
Also revert the workaround from bug 1525762.
We have some convoluted packaging logic to automatically package any directory
with a manifest.json file as an XPI. Unfortunately, that logic also applies to
extensions that are supposed to be part of omni.ja.
It would be nice to have a cleaner way to filter out any resources which we've
already flagged to be part of omni.ja, but this patch takes the simpler
approach of just including anything that's in a modules/ directory, which is
whitelisted for omni.ja elsewhere.
--HG--
extra : source : d30967b60a03f8f4286c26f14342d50e03a59f9d
extra : amend_source : 2ddab7bd7ce2fb666d5335bf9941328d9dd7d66f
Moved tab context menu out of browser.dtd to browser.xul except for sendPageToDevice, sendLinkToDevice, moveTabOptions, moveSelectedTabOptions, undoCloseTab. Not sure if tabbrowser.js and tabbrowser.xul are working as intended.
Differential Revision: https://phabricator.services.mozilla.com/D19312
--HG--
extra : moz-landing-system : lando
We shouldn't pass those flags when building C. It doesn't matter /too/
much currently, but will in a subsequent change, which will introduce
a C++-only flag in stlport_cppflags.
Differential Revision: https://phabricator.services.mozilla.com/D25018
--HG--
extra : moz-landing-system : lando
Practically speaking, it makes no difference, as only the calls to
add_base from tests weren't already ordered.
This allows a simpler approach for next change.
Differential Revision: https://phabricator.services.mozilla.com/D25036
--HG--
extra : moz-landing-system : lando
This demonstrates how addons positioned under a resource directory (a
directory that would normally go in omni.ja) is currently treated.
Differential Revision: https://phabricator.services.mozilla.com/D25035
--HG--
extra : moz-landing-system : lando
Bug 1533425 makes Gecko try to load from $ARCH/greprefs.js, etc on
Android. This patch teaches the packager to put preferences into
those architecture-specific locations for that code to find.
Differential Revision: https://phabricator.services.mozilla.com/D24984
--HG--
extra : moz-landing-system : lando
Last time we updated the requirements was in bug 1278456, after we
switched to GTK+3.
While it might be worth checking what would be a reasonable requirement,
the immediate problem is that updating ANGLE hits a snag with a
libstdc++ 4.7 symbol, and the only main distro release that bumping to
that version would make us not support anymore is Ubuntu 12.04 LTS,
EOLed 2 years ago (April 2017).
Bumping to some even newer version would need more consideration.
Desupporting Ubuntu 12.04 LTS at this point is almost a no-brainer.
Differential Revision: https://phabricator.services.mozilla.com/D23979
--HG--
extra : moz-landing-system : lando
Newer versions of rust come with a specialized arm target that matches
more closely our armv7 targets (with neon and thumb2), so use that when
possible.
Depends on D24324
Differential Revision: https://phabricator.services.mozilla.com/D24325
--HG--
extra : moz-landing-system : lando
Changed the following:
- Use (Yn) instead of [Y/n] to align with mercurial-setup that is using
mercurial's ui class
- Use (Yn) prompt instead of int prompt where the options are Yes/No
- Remove empty lines around options to align with the first question
about Firefox {Desktop,Mobile} {,non-}Artifact
- Move the question `Would you like to enable build system telemetry?`
to the last line
- Use `Your choice:` propmt for int prompt to align with the first question
Differential Revision: https://phabricator.services.mozilla.com/D23457
--HG--
extra : moz-landing-system : lando
- Do not use `ERROR` when user enters possibly parent-directory when
choosing the directory to clone mozilla-unified
- Show the path of pre-existing mozilla-unified path on error
Differential Revision: https://phabricator.services.mozilla.com/D23456
--HG--
extra : moz-landing-system : lando
Sometimes tools install pypi at runtime via mach (e.g self.install_pip_package
/ self.install_pip_requirements). It's difficult to test these modules with
pytest because we usually won't be going through mach.
This gives tests the ability to depend on external pypi packages the same way
they might get installed when running via mach.
Note, I only added support for requirements.txt here because
python/mozbuild/mozbuild/virtualenv.py's 'install_pip_package' function is
completely busted with modern pip. And the pip used with |mach python-test| is
more modern than the one used with the regular build venv due to pipenv. We'll
need to fix this eventually, but that's another bug for another day.
Differential Revision: https://phabricator.services.mozilla.com/D22784
--HG--
extra : moz-landing-system : lando
Bug 1528123 added a way to specify a task id to influence mach
artifact. In the case of upcoming EME-enabled win64-aarch64 builds, we
need to be able to specify two different task ids, which a single
environment variable doesn't really allow.
So extend the scheme to check environment variables with the build
type included in the variable name.
Differential Revision: https://phabricator.services.mozilla.com/D23101
--HG--
extra : moz-landing-system : lando
Instead of over-estimating the number of items in a batch, do the opposite:
slightly under-estimate the number of items, then dispatch outstanding items
(by just adding one item to each batch).
Differential Revision: https://phabricator.services.mozilla.com/D23141
--HG--
extra : moz-landing-system : lando
Parts of this patch were taken from the original work of :bbouvier in Bug 1521772.
We needed to revert Bug 1521772 since it broken Windows compatibility.
Differential Revision: https://phabricator.services.mozilla.com/D23100
--HG--
extra : moz-landing-system : lando
Flake8 ignores the 'exclude' section of the .flake8.yml if you pass in a direct
path to a file. To get around this we have some custom logic to handle these
exclusions for us, but this custom logic didn't account for globs.
Differential Revision: https://phabricator.services.mozilla.com/D23145
--HG--
extra : moz-landing-system : lando
It's occasionally useful to know what versions are being discovered for
these variables. It's also convenient for logs from automation to
contain all the relevant information, rather than setting up a loaner
and running all the commands yourself.
Differential Revision: https://phabricator.services.mozilla.com/D22385
--HG--
extra : moz-landing-system : lando
The jar log is used for optimization of the packaged jar files according
to their usage patterns during a profile run. The current content of the
file currently come with 2 caveats:
- it contains entries for jar archives that aren't relevant to
packaging, which is not a problem in itself, but see below.
- it contains full paths for jar archives that may not correspond to the
location of the packaged directory (on e.g. Android, where the build
almost certainly doesn't happen in the same directory on the host as
Fennec runs in the emulator/on the device).
The current JarLog code does somehow handle the various ways paths are
currently presented, but it's clearly missing code to map the paths in
the log to packaged paths. Instead of requiring manual work and extra
build options to handle this mapping, and considering the caveats above,
it's just simpler to log archive paths as if they were relative to the
packaged application directory in a build, and use that during
packaging.
Depends on D21655
Differential Revision: https://phabricator.services.mozilla.com/D21656
--HG--
extra : moz-landing-system : lando
The jar log is used for optimization of the packaged jar files according
to their usage patterns during a profile run. The current content of the
file currently come with 2 caveats:
- it contains entries for jar archives that aren't relevant to
packaging, which is not a problem in itself, but see below.
- it contains full paths for jar archives that may not correspond to the
location of the packaged directory (on e.g. Android, where the build
almost certainly doesn't happen in the same directory on the host as
Fennec runs in the emulator/on the device).
The current JarLog code does somehow handle the various ways paths are
currently presented, but it's clearly missing code to map the paths in
the log to packaged paths. Instead of requiring manual work and extra
build options to handle this mapping, and considering the caveats above,
it's just simpler to log archive paths as if they were relative to the
packaged application directory in a build, and use that during
packaging.
Depends on D21655
Differential Revision: https://phabricator.services.mozilla.com/D21656
--HG--
extra : moz-landing-system : lando
Before this change linting python/mozbuild brought about many errors. Some of this errors could be fixed using |mach lint --fix|. The remaining errors where fixed in this bug.
Differential Revision: https://phabricator.services.mozilla.com/D20138
--HG--
extra : rebase_source : 8bd3d622d2221b981f08b8a080c1198b002cdc49
extra : amend_source : 3f6a70c1b7104a7c2d83e11fe911942771e331ea
Fennec has a value of OMNIJAR_NAME that contains a directory, contrary
to other platforms, and relies in post-packaging, pre-unpacking steps to
accommodate with the difference.
With this change, we just make the packaging and unpacking steps aware
of this setup, and make allow them to pack/unpack resources in foo/
under foo/$OMNIJAR_NAME, whether $OMNIJAR_NAME is a file name or a path.
This will, further down the road, allow the packager code to handle jar
logs from PGO instrumentation without munging them.
Differential Revision: https://phabricator.services.mozilla.com/D21654
--HG--
extra : moz-landing-system : lando