Lint python/mozbuild/{mozbuild,mozpack}. r=#build
Add 'from __future__ import absolute_import' to all missing files in python/mozbuild/{mozbuild,mozpack}/**/*.py
Differential Revision: https://phabricator.services.mozilla.com/D26643
--HG--
extra : moz-landing-system : lando
Leaving one fix for an explicit review.
Lint python/mozbuild/{mozbuild,mozpack}. r=#build
Differential Revision: https://phabricator.services.mozilla.com/D26641
--HG--
extra : moz-landing-system : lando
Lint python/mozbuild/{mozbuild,mozpack}.
ran './mach lint -l py2 -l flake8 -l shellcheck -l codespell -l yaml python/mozbuild/{mozbuild,mozpack}/ --fix' in order
to undo some black changes and get closer to making this folder able to be validated on every lint run
Differential Revision: https://phabricator.services.mozilla.com/D26640
--HG--
extra : moz-landing-system : lando
We're moving to IR-level PGO instrumentation for clang-cl. We've also
moved to using static linker ordering files, which was the primary
application of the previous style of PGO instrumentation. We therefore
we no longer need this code.
Differential Revision: https://phabricator.services.mozilla.com/D31134
--HG--
extra : moz-landing-system : lando
We used to support specifying single Rust files for compilation,
emphasis on "used to".
Differential Revision: https://phabricator.services.mozilla.com/D32314
--HG--
extra : moz-landing-system : lando
Bug 1533425 makes Gecko try to load from $ARCH/greprefs.js, etc on
Android. This patch teaches the packager to put certain preferences
into those architecture-specific locations for that code to find.
Differential Revision: https://phabricator.services.mozilla.com/D24984
--HG--
extra : moz-landing-system : lando
This way compilation backends don't all have to figure out the right way
to handle symbol files.
Differential Revision: https://phabricator.services.mozilla.com/D29673
--HG--
extra : moz-landing-system : lando
Our current OS X builds use `--target=x86_64-darwin11` (which
corresponds to OS X 10.7). This target is problematic for two reasons:
* We're actually targeting for OS X 10.9 (`MACOSX_DEPLOYMENT_TARGET`);
* It's slightly different from the default Rust target.
Let's address these problems in reverse order: differences from the Rust
target are bad, because the `--target` we provide to `clang` and the
Rust target find their way into LLVM bitcode files and the linker will
refuse to link together bitcode files that have incompatible targets.
Why are the two incompatible? The current `--target` doesn't have a
"vendor" in triple-speak, whereas the Rust one has "apple" as the
vendor (`x86_64-apple-darwin`) We therefore need to change the
`--target` we pass to `clang` to have a vendor of "apple".
This need is behind the {init,toolchain}.configure changes,
but it has ramifications elsewhere, because `clang` looks for
`--target`-prefixed build tools. So we have to change the `--target`
for cctools to get the right tool prefixes and we have to change the
`--target` for building clang ourselves so that *those* builds can find
the newly renamed cctools.
Once we've done, that's really enough; we don't *need to address the
first problem: While the `--target` might be `x86_64-apple-darwin11`,
both `clang` and `rustc` will dynamically choose the target triple that
eventually lands in LLVM bitcode files based on
`MACOSX_DEPLOYMENT_TARGET`, which we set in all builds. But the current
target is slightly misleading, and the cctools don't need to be prefixed
with a particular Darwin version, since they work for all Darwin
targets. Let's just drop the "11" from the `--target` and eliminate a
little bit of confusion.
Differential Revision: https://phabricator.services.mozilla.com/D31128
--HG--
extra : moz-landing-system : lando
Our current OS X builds use `--target=x86_64-darwin11` (which
corresponds to OS X 10.7). This target is problematic for two reasons:
* We're actually targeting for OS X 10.9 (`MACOSX_DEPLOYMENT_TARGET`);
* It's slightly different from the default Rust target.
Let's address these problems in reverse order: differences from the Rust
target are bad, because the `--target` we provide to `clang` and the
Rust target find their way into LLVM bitcode files and the linker will
refuse to link together bitcode files that have incompatible targets.
Why are the two incompatible? The current `--target` doesn't have a
"vendor" in triple-speak, whereas the Rust one has "apple" as the
vendor (`x86_64-apple-darwin`) We therefore need to change the
`--target` we pass to `clang` to have a vendor of "apple".
This need is behind the {init,toolchain}.configure changes,
but it has ramifications elsewhere, because `clang` looks for
`--target`-prefixed build tools. So we have to change the `--target`
for cctools to get the right tool prefixes and we have to change the
`--target` for building clang ourselves so that *those* builds can find
the newly renamed cctools.
Once we've done, that's really enough; we don't *need to address the
first problem: While the `--target` might be `x86_64-apple-darwin11`,
both `clang` and `rustc` will dynamically choose the target triple that
eventually lands in LLVM bitcode files based on
`MACOSX_DEPLOYMENT_TARGET`, which we set in all builds. But the current
target is slightly misleading, and the cctools don't need to be prefixed
with a particular Darwin version, since they work for all Darwin
targets. Let's just drop the "11" from the `--target` and eliminate a
little bit of confusion.
Differential Revision: https://phabricator.services.mozilla.com/D31128
--HG--
extra : moz-landing-system : lando
The zipfile.is_zipfile() function is overly lenient, in that non-zip
files that contain the four bytes "PK\005\006" near the end of the file
are reported as zip files even if the zip structure is not valid.
Occasionally, our target.tar.bz2 files randomly contain those 4 bytes at
the expected location, which causes the mar repackaging code to try to
process the package as a zip file instead of a tar file.
The tarfile.is_tarfile() logic looks a little more robust in that it
actually tries to open the file, so if we try tar first before falling
back to zip we should be a little more resilient.
Differential Revision: https://phabricator.services.mozilla.com/D31908
--HG--
extra : moz-landing-system : lando
Our current OS X builds use `--target=x86_64-darwin11` (which
corresponds to OS X 10.7). This target is problematic for two reasons:
* We're actually targeting for OS X 10.9 (`MACOSX_DEPLOYMENT_TARGET`);
* It's slightly different from the default Rust target.
Let's address these problems in reverse order: differences from the Rust
target are bad, because the `--target` we provide to `clang` and the
Rust target find their way into LLVM bitcode files and the linker will
refuse to link together bitcode files that have incompatible targets.
Why are the two incompatible? The current `--target` doesn't have a
"vendor" in triple-speak, whereas the Rust one has "apple" as the
vendor (`x86_64-apple-darwin`) We therefore need to change the
`--target` we pass to `clang` to have a vendor of "apple".
This need is behind the {init,toolchain}.configure changes,
but it has ramifications elsewhere, because `clang` looks for
`--target`-prefixed build tools. So we have to change the `--target`
for cctools to get the right tool prefixes and we have to change the
`--target` for building clang ourselves so that *those* builds can find
the newly renamed cctools.
Once we've done, that's really enough; we don't *need to address the
first problem: While the `--target` might be `x86_64-apple-darwin11`,
both `clang` and `rustc` will dynamically choose the target triple that
eventually lands in LLVM bitcode files based on
`MACOSX_DEPLOYMENT_TARGET`, which we set in all builds. But the current
target is slightly misleading, and the cctools don't need to be prefixed
with a particular Darwin version, since they work for all Darwin
targets. Let's just drop the "11" from the `--target` and eliminate a
little bit of confusion.
Differential Revision: https://phabricator.services.mozilla.com/D31128
--HG--
extra : moz-landing-system : lando
This just separates out the Android definitions into
mobile/android/mach_commands.py. There was vestigial support for
running on Android with debuggers, but it was for wiring up JimDB,
which is no longer supported and in fact hasn't worked on actual
devices for a very long time. (The new flow for running on Android
under a debugger goes through the Android Studio hybrid debugger.)
Differential Revision: https://phabricator.services.mozilla.com/D18291
--HG--
extra : moz-landing-system : lando
Having `rustc` be `rustup`'s wrapper for `rustc` means that we may
silently honor `rustup`'s override mechanisms. We noticed this first on
OS X, where we use the "real" `cargo` but `rustup`'s `rustc` wrapper,
and problems ensued when `cargo` thought it was using one version of
`rustc`, but actually wound up using something different.
It seems better to avoid silently interposing `rustup`'s toolchain
override mechanisms everywhere, rather than having to special-case OS
X. So let's factor out a general mechanism for removing the wrappers
`rustup` provides and use that for both `rustc` and `cargo`. The tests
need adjusting because we weren't triggering the unwrapping cases
before; we don't yet test the case where we really do need to unwrap.
That test can be left for a future patch.
Differential Revision: https://phabricator.services.mozilla.com/D29531
--HG--
extra : moz-landing-system : lando
We've not been checking the clang-cl version in use. This lack of
checking is bad, for a couple of reasons:
* Released versions of clang-cl differ drastically in their robustness;
* Only the most recent version of clang-cl supports aarch64.
We should check for a minimum version of clang-cl, just like our other
supported compilers. As a bonus, we can then start depending on
features that we know appear in the particular minimum clang-cl
version. (The current patch is motivated by `/clang:` command-line
support, but one could pick other things.)
Differential Revision: https://phabricator.services.mozilla.com/D30723
--HG--
extra : moz-landing-system : lando
Also for checker `modernize-avoid-bind` export the respective reliability index from config.yaml
Differential Revision: https://phabricator.services.mozilla.com/D30904
--HG--
extra : moz-landing-system : lando
(adds __build_class__ to the builtin sandbox and allow CombinedDependsFunction to be hashed)
Differential Revision: https://phabricator.services.mozilla.com/D28114
--HG--
extra : rebase_source : 967a394d10d41e3da322005c84757c001a3b74cb
extra : intermediate-source : 2be63fc923997e6865ab9abc4fa7fb728e55d532
extra : source : aed41a0c652ae3f8e4b4f76bfaa0c56130515a75
Also adjust 'basestring' usage in some places.
Differential Revision: https://phabricator.services.mozilla.com/D28102
--HG--
extra : rebase_source : f0fecd6b040336b0cf2f055e266fe37eea5a9a48
extra : intermediate-source : 712718250f0ba5c286c1fb6b6c7f370b713c7225
extra : source : 6b24ce2da518d8158e13627adb11c9fcad19c92a
shellutil.py needed special handling since it is invoked earlier than the
vendor directory adding we do, so it does not have six available.
Differential Revision: https://phabricator.services.mozilla.com/D28096
--HG--
extra : rebase_source : c5c611d280dc821747c9db736e4943c2cc421557
extra : intermediate-source : 30fb268b41fdc45937a22ab446a8805fb099c1f0
extra : source : d102292b6b5b812e7df165116410488d18042bd3
0.1.21 mishandles cargo package renames, which are a required
feature for Rust 2018 support. The latest version fixes this.
Differential Revision: https://phabricator.services.mozilla.com/D29946
--HG--
extra : moz-landing-system : lando
Under the hood, browsertime invokes a certain `visualmetrics.py`
script. That script depends on `ffmpeg` and ImageMagick's `convert`,
`compare`, and `mogrify` commands. It also depends on certain Python
packages.
So this installs those dependencies, and then wires up the evaluation
environment such that `./mach browsertime` can find the dependencies.
It also adds a `./mach visualmetrics` command for processing a
captured MP4 file in the same way that browsertime processes such a
file.
In order to avoid downloading dependencies multiple time, the existing
artifact cache is extracted. This is a small first step towards [Bug
1526021](https://bugzilla.mozilla.org/show_bug.cgi?id=1526021), which
might want to use this artifact cache as well.
At this time, hashes and filesizes are not verified. During
development, the upstream files changed multiple times, and it's not
worth being completely locked down while experimenting with this
functionality. If we start running this code in automation or in more
sensitive environments, we can build fetch tasks and TC indexes to
streamline the artifact gathering process.
It is expected that a future mach command will want to invoke
browsertime without suffering the overhead of invoking Python (and
mach, which is itself bulky) so a nod is given to exposing the
relevant environment pieces.
During testing, it was discovered that [MozillaBuild doesn't ship
git](https://bugzilla.mozilla.org/show_bug.cgi?id=1503028), so that
git repositories can't be used out-of-the-box on Windows. So instead
we use a [tarball link from github.com/$USER/$REPO/tarball/$COMMIT-LIKE](https://github.blog/2008-03-03-tarball-downloads/).
Differential Revision: https://phabricator.services.mozilla.com/D29442
--HG--
rename : python/mozbuild/mozbuild/test/test_artifacts.py => python/mozbuild/mozbuild/test/test_artifact_cache.py
extra : moz-landing-system : lando
Some media libraries use gas syntax in their assembly files. Rather than
converting these arm assembly syntax files for aarch64, we can use clang-cl
to build them directly.
Differential Revision: https://phabricator.services.mozilla.com/D27785
--HG--
extra : moz-landing-system : lando
Lint python/mozbuild/{mozbuild,mozpack}.
This makes sure we use byte strings (type(b'')) in many places.
This also has a few places where we know we want unicode, by enforcing it.
This code also has a few places where we call into windows API's which expect either all args to be unicode,
or all args to be bytestrings (not-unicode) so make sure those calls align.
This is the most risky of the stack, since there are some mach commands and codepaths that are neither excercised in automation
nor excercised in tests.
Differential Revision: https://phabricator.services.mozilla.com/D26645
--HG--
extra : rebase_source : 1ee367576bede0318a2db05644693d4f440a250a
extra : intermediate-source : 2516c350cba6f27eb204f0087728721abbb0e085
extra : source : 6351a29c5765c13c39ef6186f9e5e911e00420e7
Leaving one fix for an explicit review.
Lint python/mozbuild/{mozbuild,mozpack}. r=#build
Differential Revision: https://phabricator.services.mozilla.com/D26641
--HG--
extra : rebase_source : e14d7762c7802d98e24ee8f86b0167d96666d288
extra : intermediate-source : 79b0d1678dd2f6aadf60f480d902ddeca4a52c3a
extra : source : faadf440e7d7b0a8c473e2dee3fc90b27023814a
Lint python/mozbuild/{mozbuild,mozpack}.
ran './mach lint -l py2 -l flake8 -l shellcheck -l codespell -l yaml python/mozbuild/{mozbuild,mozpack}/ --fix' in order
to undo some black changes and get closer to making this folder able to be validated on every lint run
Differential Revision: https://phabricator.services.mozilla.com/D26640
--HG--
extra : rebase_source : 6b69b6ebcac73835b752607f7b5b4429de7f95cf
extra : intermediate-source : 114355f83d36188fac592c8c6497242c71b27ad6
extra : source : 51fbdf63bbce8f75fac407de305f826dc775fcb5
With this change tooltool.py also supports taskcluster credentials to be passed
(in json format) to --authentication-file option. RelengAPI tokens are still
working with this patch, just additional authentication is added.
Differential Revision: https://phabricator.services.mozilla.com/D27881
--HG--
extra : moz-landing-system : lando
The noatime option doesn't appear to have a significant impact on no-op
build times, even though the build system does a significant number of
file reads. Since macs have atime enabled by default, we should just
disable this check rather than give developers unhelpful error messages.
Differential Revision: https://phabricator.services.mozilla.com/D28906
--HG--
extra : moz-landing-system : lando
For some reason, using `tar -Jxf` with Windows paths fails, and that makes
`mach artifact toolchain` unable to pull .tar.xz archives on Windows.
However, executing `xz -d -c` works, and we can feed its output to
python's tarfile.
Differential Revision: https://phabricator.services.mozilla.com/D28779
--HG--
extra : moz-landing-system : lando
With this change tooltool.py also supports taskcluster credentials to be passed
(in json format) to --authentication-file option. RelengAPI tokens are still
working with this patch, just additional authentication is added.
Differential Revision: https://phabricator.services.mozilla.com/D27881
--HG--
extra : moz-landing-system : lando
Add basic support for 'mach gtest' on Android.
Handling of Android-only and desktop-only options is awkward; I hope to
re-visit this after bug 1519369.
Differential Revision: https://phabricator.services.mozilla.com/D28129
--HG--
extra : moz-landing-system : lando
If the user runs mach clang-format from the builddir, it was finding and using
unified files.
Differential Revision: https://phabricator.services.mozilla.com/D27750
--HG--
extra : moz-landing-system : lando
The TestManifestBackend always has a dependency on config.status, even
if configure hasn't been run yet. If configure is run later, this is
intended to cause the TestManifestBackend to be regenerated so that it
can use the configured values instead of a potentially incomplete setup.
Mozbuild's backend_out_of_date() logic treats a missing file as always
being out of date, which means if configure hasn't run, the
TestManifestBackend will always be regenerated.
Instead we can create a stub config.status file that will fix the
out-of-date check, and can be treated as a
BuildEnvironmentNotFoundException as if the file didn't exist at all.
Differential Revision: https://phabricator.services.mozilla.com/D27195
--HG--
extra : moz-landing-system : lando
Bug 1537574 moved some code that uses errno from building.py to base.py,
but the latter didn't already import errno.
Differential Revision: https://phabricator.services.mozilla.com/D27125
--HG--
extra : moz-landing-system : lando
If mozbuild parsing fails due to a missing file (eg: a file not existing
in UNIFIED_SOURCES), then no Makefiles are written out, but
config.status exists. This would cause mozbuild to think that configure
doesn't need to run, and rely on make to perform the backend-out-of-date
check in rebuild-backend.mk. Unfortunately since no Makefiles were
written, the make command fails immediately and no attempt is made to
re-create the backend. Note that this is only a problem if the first
mozbuild parsing from a clobber build fails, otherwise there is
typically a top-level Makefile from a previous build to call into (at
which point make can determine it is out-of-date, and re-invoke itself).
The fix is to have the RecursiveMake backend re-use the same logic that
was introduced into mozbuild for alternate backends, and remove
rebuild-backend.mk. This way, mozbuild can always determine if the
backend needs to be regenerated, even if the initial parsing failed.
Test code was also relying on rebuild-backend.mk to generate the
TestBackend, but moving backend_out_of_date() into MozbuildObject allows
this code to be shared.
Differential Revision: https://phabricator.services.mozilla.com/D26262
--HG--
rename : build/gen_test_backend.py => python/mozbuild/mozbuild/gen_test_backend.py
extra : moz-landing-system : lando
Before bug 938437, we had a rather large and error-prone
nsStaticXULComponents.cpp used to register all modules. That was
replaced with clever use of the linker, which allowed to avoid the mess
that maintaining that file was.
Fast forward to now, where after bug 1524687 and other work that
preceded it, we have a much smaller number of remaining static xpcom
components, registered via this linker hack, and don't expect to add
any new ones. The list should eventually go down to zero.
Within that context, it seems to be the right time to get rid of the
magic, and with it the problems it causes on its own.
Some of those components could probably be trivially be converted to
static registration via .conf files, but I didn't want to deal with the
possible need to increase the number of dummy modules in XPCOMInit.cpp.
They can still be converted as a followup.
Differential Revision: https://phabricator.services.mozilla.com/D26076
--HG--
extra : moz-landing-system : lando
If mozbuild parsing fails due to a missing file (eg: a file not existing
in UNIFIED_SOURCES), then no Makefiles are written out, but
config.status exists. This would cause mozbuild to think that configure
doesn't need to run, and rely on make to perform the backend-out-of-date
check in rebuild-backend.mk. Unfortunately since no Makefiles were
written, the make command fails immediately and no attempt is made to
re-create the backend. Note that this is only a problem if the first
mozbuild parsing from a clobber build fails, otherwise there is
typically a top-level Makefile from a previous build to call into (at
which point make can determine it is out-of-date, and re-invoke itself).
The fix is to have the RecursiveMake backend re-use the same logic that
was introduced into mozbuild for alternate backends, and remove
rebuild-backend.mk. This way, mozbuild can always determine if the
backend needs to be regenerated, even if the initial parsing failed.
Test code was also relying on rebuild-backend.mk to generate the
TestBackend, but moving backend_out_of_date() into MozbuildObject allows
this code to be shared.
Differential Revision: https://phabricator.services.mozilla.com/D26262
--HG--
rename : build/gen_test_backend.py => python/mozbuild/mozbuild/gen_test_backend.py
extra : moz-landing-system : lando
This patch introduces a new marionette media test along
with a Youtube test.
To run the Youtube streaming test locally:
./mach marionette-test dom/media/test/marionette/test_youtube.py -vv --gecko-log -
Differential Revision: https://phabricator.services.mozilla.com/D23644
--HG--
extra : moz-landing-system : lando
Keep a checksum to avoid re-extracting artifacts over and over.
In the future we can also check the checksum to avoid downloading the artifact
altogether, maybe.
Differential Revision: https://phabricator.services.mozilla.com/D25542
--HG--
extra : moz-landing-system : lando
This patch introduces a new marionette media test along
with a Youtube test.
To run the Youtube streaming test locally:
./mach marionette-test dom/media/test/marionette/test_youtube.py -vv --gecko-log -
Differential Revision: https://phabricator.services.mozilla.com/D23644
--HG--
extra : moz-landing-system : lando
We produce two types of build symbol archives in automation:
- "crashreporter-symbols.zip" contains Breakpad-format .sym files
- "crashreporter-symbols-full.zip" contains Breakpad-format .sym files
and compressed ELF debug symbol .dbg.gz files
Right now, `--enable-artifact-build-symbols` from Bug 1305502
downloads only "crashreporter-symbols.zip".
The Android Studio version of lldb, currently 7.0.0, doesn't support
Breakpad-format .sym files. It does support (uncompressed) ELF debug
symbols. (Note that gdb isn't supported on Android and hasn't been for
some time.)
This makes ` --enable-artifact-build-symbols` download the full
symbols for Android builds that aren't in automation, to be useful for
debugging Android builds with lldb locally.
Differential Revision: https://phabricator.services.mozilla.com/D19091
--HG--
extra : moz-landing-system : lando
It's not helpful to have a DEBUG log level, 'cuz the -v flag to
`./mach artifact install` doesn't turn on DEBUG logging.
Differential Revision: https://phabricator.services.mozilla.com/D19090
--HG--
extra : moz-landing-system : lando
When an added file is not under an app or addon directory, its base
directory is ''. The code added in bug 1539355 would skip doing its
thing in that case, which it shouldn't.
Also revert the workaround from bug 1525762.
We have some convoluted packaging logic to automatically package any directory
with a manifest.json file as an XPI. Unfortunately, that logic also applies to
extensions that are supposed to be part of omni.ja.
It would be nice to have a cleaner way to filter out any resources which we've
already flagged to be part of omni.ja, but this patch takes the simpler
approach of just including anything that's in a modules/ directory, which is
whitelisted for omni.ja elsewhere.
--HG--
extra : source : d30967b60a03f8f4286c26f14342d50e03a59f9d
extra : amend_source : 2ddab7bd7ce2fb666d5335bf9941328d9dd7d66f
We shouldn't pass those flags when building C. It doesn't matter /too/
much currently, but will in a subsequent change, which will introduce
a C++-only flag in stlport_cppflags.
Differential Revision: https://phabricator.services.mozilla.com/D25018
--HG--
extra : moz-landing-system : lando
Practically speaking, it makes no difference, as only the calls to
add_base from tests weren't already ordered.
This allows a simpler approach for next change.
Differential Revision: https://phabricator.services.mozilla.com/D25036
--HG--
extra : moz-landing-system : lando
This demonstrates how addons positioned under a resource directory (a
directory that would normally go in omni.ja) is currently treated.
Differential Revision: https://phabricator.services.mozilla.com/D25035
--HG--
extra : moz-landing-system : lando
Bug 1533425 makes Gecko try to load from $ARCH/greprefs.js, etc on
Android. This patch teaches the packager to put preferences into
those architecture-specific locations for that code to find.
Differential Revision: https://phabricator.services.mozilla.com/D24984
--HG--
extra : moz-landing-system : lando
Summary: Since we add host parameter to java_search_paths, test_java_tool_checks is always failure due to no host function. So I need to set dummy host function.
Reviewers: nalexander
Bug #: 1527796
Differential Revision: https://phabricator.services.mozilla.com/D24873
--HG--
extra : rebase_source : b1fba1e6c108f39f078f67c21923b606cca87d98
Last time we updated the requirements was in bug 1278456, after we
switched to GTK+3.
While it might be worth checking what would be a reasonable requirement,
the immediate problem is that updating ANGLE hits a snag with a
libstdc++ 4.7 symbol, and the only main distro release that bumping to
that version would make us not support anymore is Ubuntu 12.04 LTS,
EOLed 2 years ago (April 2017).
Bumping to some even newer version would need more consideration.
Desupporting Ubuntu 12.04 LTS at this point is almost a no-brainer.
Differential Revision: https://phabricator.services.mozilla.com/D23979
--HG--
extra : moz-landing-system : lando
Newer versions of rust come with a specialized arm target that matches
more closely our armv7 targets (with neon and thumb2), so use that when
possible.
Depends on D24324
Differential Revision: https://phabricator.services.mozilla.com/D24325
--HG--
extra : moz-landing-system : lando
Bug 1528123 added a way to specify a task id to influence mach
artifact. In the case of upcoming EME-enabled win64-aarch64 builds, we
need to be able to specify two different task ids, which a single
environment variable doesn't really allow.
So extend the scheme to check environment variables with the build
type included in the variable name.
Differential Revision: https://phabricator.services.mozilla.com/D23101
--HG--
extra : moz-landing-system : lando
Instead of over-estimating the number of items in a batch, do the opposite:
slightly under-estimate the number of items, then dispatch outstanding items
(by just adding one item to each batch).
Differential Revision: https://phabricator.services.mozilla.com/D23141
--HG--
extra : moz-landing-system : lando
Parts of this patch were taken from the original work of :bbouvier in Bug 1521772.
We needed to revert Bug 1521772 since it broken Windows compatibility.
Differential Revision: https://phabricator.services.mozilla.com/D23100
--HG--
extra : moz-landing-system : lando
It's occasionally useful to know what versions are being discovered for
these variables. It's also convenient for logs from automation to
contain all the relevant information, rather than setting up a loaner
and running all the commands yourself.
Differential Revision: https://phabricator.services.mozilla.com/D22385
--HG--
extra : moz-landing-system : lando
The jar log is used for optimization of the packaged jar files according
to their usage patterns during a profile run. The current content of the
file currently come with 2 caveats:
- it contains entries for jar archives that aren't relevant to
packaging, which is not a problem in itself, but see below.
- it contains full paths for jar archives that may not correspond to the
location of the packaged directory (on e.g. Android, where the build
almost certainly doesn't happen in the same directory on the host as
Fennec runs in the emulator/on the device).
The current JarLog code does somehow handle the various ways paths are
currently presented, but it's clearly missing code to map the paths in
the log to packaged paths. Instead of requiring manual work and extra
build options to handle this mapping, and considering the caveats above,
it's just simpler to log archive paths as if they were relative to the
packaged application directory in a build, and use that during
packaging.
Depends on D21655
Differential Revision: https://phabricator.services.mozilla.com/D21656
--HG--
extra : moz-landing-system : lando
The jar log is used for optimization of the packaged jar files according
to their usage patterns during a profile run. The current content of the
file currently come with 2 caveats:
- it contains entries for jar archives that aren't relevant to
packaging, which is not a problem in itself, but see below.
- it contains full paths for jar archives that may not correspond to the
location of the packaged directory (on e.g. Android, where the build
almost certainly doesn't happen in the same directory on the host as
Fennec runs in the emulator/on the device).
The current JarLog code does somehow handle the various ways paths are
currently presented, but it's clearly missing code to map the paths in
the log to packaged paths. Instead of requiring manual work and extra
build options to handle this mapping, and considering the caveats above,
it's just simpler to log archive paths as if they were relative to the
packaged application directory in a build, and use that during
packaging.
Depends on D21655
Differential Revision: https://phabricator.services.mozilla.com/D21656
--HG--
extra : moz-landing-system : lando
Before this change linting python/mozbuild brought about many errors. Some of this errors could be fixed using |mach lint --fix|. The remaining errors where fixed in this bug.
Differential Revision: https://phabricator.services.mozilla.com/D20138
--HG--
extra : rebase_source : 8bd3d622d2221b981f08b8a080c1198b002cdc49
extra : amend_source : 3f6a70c1b7104a7c2d83e11fe911942771e331ea
Fennec has a value of OMNIJAR_NAME that contains a directory, contrary
to other platforms, and relies in post-packaging, pre-unpacking steps to
accommodate with the difference.
With this change, we just make the packaging and unpacking steps aware
of this setup, and make allow them to pack/unpack resources in foo/
under foo/$OMNIJAR_NAME, whether $OMNIJAR_NAME is a file name or a path.
This will, further down the road, allow the packager code to handle jar
logs from PGO instrumentation without munging them.
Differential Revision: https://phabricator.services.mozilla.com/D21654
--HG--
extra : moz-landing-system : lando
The underlying backends only support directories in LOCAL_INCLUDES
(since these ultimately translate to -I arguments to the compiler). We
should disallow filenames here, in case a developer accidentally
specifies a header file instead of a directory.
Differential Revision: https://phabricator.services.mozilla.com/D21698
--HG--
extra : moz-landing-system : lando
In cases like those in the added unit test, explicit options on the
command line could end up being silently ignored. So instead of that happening,
error out. Unfortunately, the error message is not entirely accurate,
but it's better than nothing. It's rare anyways (I only stumbled upon it
because I was trying to do something fishy), and correlation between the
error message and the corresponding changes should make it clear what's
going on.
Depends on D20822
Differential Revision: https://phabricator.services.mozilla.com/D20823
--HG--
extra : moz-landing-system : lando
Currently, when a dependency loop involve imply_option, it is possible
to end up with an error message saying the implied option is unknown,
when it fact it is. So instead of bailing out with a weird error
message, try to make things work (if the implied value is not different
from what's known), or bail with a more accurate message.
Differential Revision: https://phabricator.services.mozilla.com/D20822
--HG--
extra : moz-landing-system : lando
Optimizing jars without preloading/reordering data only moves the
jar central directory to the beginning of the file, which, without
preloading information, is not very useful. Let's just stop doing it if
there's not going to be preloading/reordering information at all.
Differential Revision: https://phabricator.services.mozilla.com/D21170
--HG--
extra : moz-landing-system : lando
On Android we apparently try to include the cpufeatures.c file from the
NDK as part of the generated-sources tarball. While this works, it makes
tar drop the bogus entry for safety reasons when unpacking the archive.
It also makes tar return an error, which is undesirable for searchfox.
It's better to just skip the entry when building the tarball.
Differential Revision: https://phabricator.services.mozilla.com/D20942
--HG--
extra : moz-landing-system : lando
This patch adds detection for when icecream is in use to build telemetry.
icecream is commonly enabled in two ways: by either setting CC/CXX to point
to icecream's cc/c++ symlinks, or by setting adding
mk_add_options 'export CCACHE_PREFIX=icecc' to a mozconfig when also using
ccache. For the former, this patch adds a simple configure check to see
if CXX is a symlink to a file named 'icecc'. For the latter this patch adds
CCACHE_PREFIX as a configure subst to capture the value.
We don't currently have a facility for writing telemetry tests that depend on
configure values. Local manual testing shows that it does work as expected.
Differential Revision: https://phabricator.services.mozilla.com/D18138
--HG--
extra : moz-landing-system : lando
Instead of having cache misses for all variables that may vary when
getting job details. While the currently used variables are mostly
constants, we're going to introduce some flexibility in an subsequent
change.
Depends on D20443
Differential Revision: https://phabricator.services.mozilla.com/D20444
--HG--
extra : moz-landing-system : lando
Once all tests regexp are gone, we come to realize that all the package
regexp are the same for each ArtifactJob subclass, some we move the
definitions to the subclasses.
Depends on D20442
Differential Revision: https://phabricator.services.mozilla.com/D20443
--HG--
extra : moz-landing-system : lando
Once the firefox-* and fennec-* regexp are gone, we come to realize
that all jobs have the same regexp for tests, so we can just
use that across the board and remove the definition from JOB_DETAILS.
Depends on D20441
Differential Revision: https://phabricator.services.mozilla.com/D20442
--HG--
extra : moz-landing-system : lando
The filters were added back when buildbot was still used, and some
artifacts were still using file names beginning with "firefox" or
"fennec".
That is not true now that all builds are on taskcluster, and for all of
them, they are now target.$ext.
Differential Revision: https://phabricator.services.mozilla.com/D20441
--HG--
extra : moz-landing-system : lando
On CI, Windows builds start from different directories on every build,
except when sccache is enabled. This affects many build types, such as
l10n repacks, and the preprocessor likes to put full paths in its
output, which means it includes those different directories, making the
builds non reproducible.
This changes the preprocessor to replace the source and object
directories with generic strings.
Differential Revision: https://phabricator.services.mozilla.com/D20421
--HG--
extra : moz-landing-system : lando
This will avoid trying to generate them during artifact builds.
Depends on D20587
Differential Revision: https://phabricator.services.mozilla.com/D20436
--HG--
extra : moz-landing-system : lando
In bug 1522354, we changed host and target detection to not invoke
config.sub, assuming the output from config.guess would satisfy our
needs in split_target.
It turns out that on some plaforms, that doesn't work out, so, while we
still skip config.sub, we now catch errors from split_target when doing
so, and try again after running config.sub when split_target fails.
Differential Revision: https://phabricator.services.mozilla.com/D19937
Since bug 1522788, mach invokes a part of configure as part of getting
mozconfig and target information for its own purpose. The problem is
that the sandboxed code may sys.exit(), which then makes mach silently
exit.
So when the sandboxed configure code does sys.exit(), instead of
silently failing, we catch the exit call, and print out what we
captured.
Differential Revision: https://phabricator.services.mozilla.com/D19936
Currently, artifact builds pull from some task they determine from the
job type, and finding a pushhead.
Sometimes that latter fails, most notably on automation. But automation
can already know the right task to use without guesswork.
This change allows artifact builds to pull from a specific taskid given
through an environment variable, and make tasks from the artifact-build
kind (not the --artifact builds from try) use that.
Remove the workaround from bug 1382982 because it's now dead code.
Differential Revision: https://phabricator.services.mozilla.com/D19881
Turns out hg diff works differently, and when Miko tried to use this
option last week he realized it didn't work quite as expected.
Differential Revision: https://phabricator.services.mozilla.com/D20102
--HG--
extra : moz-landing-system : lando
Consequently, this removes:
- MOZ_LIBPRIO, which is now always enabled.
- non_msvc_compiler, which is now always true.
- The cl.py wrapper, since it's not used anymore.
- CL_INCLUDES_PREFIX, which was only used for the cl.py wrapper.
- NONASCII, which was only there to ensure CL_INCLUDES_PREFIX still
worked in non-ASCII cases.
This however keeps a large part of detecting and configuring for MSVC,
because we still do need it for at least headers, libraries, and midl.
Depends on D19614
Differential Revision: https://phabricator.services.mozilla.com/D19615
--HG--
extra : moz-landing-system : lando
There are cases that can be easily detected where an undefined variable
is used, and such mistakes seem to happen more often than they should,
as all the errors fixed in the previous patches (that this new lint
caught).
Depends on D19112
Differential Revision: https://phabricator.services.mozilla.com/D19113
--HG--
extra : moz-landing-system : lando
Currently, running the configure lint will raise errors with little
context, if any, which can make finding the cause of the errors tedious.
So instead of raising exceptions directly, we use a hack to make the
exception appear as if they had been thrown at the code location an
issue is reported for, which makes the test harness print more useful
context.
With more context, printed out, some exception messages can be made
lighter.
The added _raise_from method does more than is necessary here, but will
be fully used in a new check shortly.
Depends on D19111
Differential Revision: https://phabricator.services.mozilla.com/D19112
--HG--
extra : moz-landing-system : lando
Using the `assert` keyword actually generates bytecode that uses the
`AssertionError` class under the hood. Ideally, we should probably not
use asserts in python configure, but there are currently so many that
it's easier to allow it, until a better strategy is decided upon.
Depends on D19110
Differential Revision: https://phabricator.services.mozilla.com/D19111
--HG--
extra : moz-landing-system : lando
In upcoming changes, we're going to use the normal compiler flags for
bindgen instead of having a separate logic for essentially the same
flags (and there's not much reason not to use the same flags after all).
One hiccup, though is that for some reason, --target=i686-linux-gnu
doesn't work with bindgen while it works with clang in the same setup.
But -m32 does.
Considering -m32 and -m64 are standard flags and that we're using them
in many cases, it doesn't hurt to use them instead of --target with
clang.
While we're doing that, we might as well refactor a little to avoid the
multiple branches handling the use of -m32/-m64, and fix the theoretical
compile-x86_64-with-x86-clang-cl case, as well as the weird check for
aarch64.
And because only two cases actually require -Xclang, only one of which
requires a condition, and the control flow is not too complex to avoid
flag duplication, we just remove the append_flag function; it was too
confusing, with all the cases we skipped it when -Xclang was not wanted.
Differential Revision: https://phabricator.services.mozilla.com/D18316
--HG--
extra : moz-landing-system : lando
mozboot.util.get_state_dir() returns a tuple of (<path>, <bool). The bool
denotes whether or not the state dir came from an environment variable.
But this value is only used in a single place, and is very easy to test for
anyway. It's not worth the added complexity it imposes on all other consumers
of this function. Let's just make this function return the path.
Differential Revision: https://phabricator.services.mozilla.com/D15723
--HG--
extra : moz-landing-system : lando
Also use armv7a as the default when no target is given at all.
Also change bootstrap to create a simpler mozconfig. The downside is
that the resulting mozconfig would not work when building older
revisions.
Add unit tests for this as well as the simplications added in bug
1523341.
Differential Revision: https://phabricator.services.mozilla.com/D17906
--HG--
extra : moz-landing-system : lando
This only affects the compiler version in the build system; not the
version reported by the compiler.
--HG--
extra : histedit_source : 37fc1a2090d872ca5e95d022e9f93b620b2d8dbb
I always forget to run clang-format before-hand. This allows me to run it on a
particular commit of my tree where the formatting doesn't match the canonical
one.
Differential Revision: https://phabricator.services.mozilla.com/D17955
--HG--
extra : moz-landing-system : lando
We can't run dump_syms without the DIA SDK binary directory in $PATH
because dump_syms requires the DIA dll from there.
Obviously, the corresponding test can't run if the DIA SDK binary
directory is not known (rather than when the dia2.h header is not found,
since the build system currently relies on WIN_DIA_SDK_BIN_DIR being
given manually).
Differential Revision: https://phabricator.services.mozilla.com/D17892
--HG--
extra : moz-landing-system : lando
In fact, "simply" use whatever python configure does to find a shell to
execute config.guess and config.sub, and get both the mozconfig content
and the real, canonicalized target alias. This has the side effect of
making builds with --target=$cpu use a complete obj-$cpu-$os default
objdir instead of obj-$cpu. This will also allow to change the
host-guessing logic without having to duplicate code.
Differential Revision: https://phabricator.services.mozilla.com/D17618
--HG--
extra : moz-landing-system : lando
We reject MSVC compilers < 2017 already, there's no point checking for
smaller versions after that.
Differential Revision: https://phabricator.services.mozilla.com/D17770
--HG--
extra : moz-landing-system : lando
