The motivating change here was wpt-sync try pushes failing because
`libgraphitewasm.dylib` wasn't getting packaged in Mac artifact builds.
We add `libosclientcerts.dylib` because packaging was complaining about
that library as well.
Differential Revision: https://phabricator.services.mozilla.com/D63705
--HG--
extra : moz-landing-system : lando
Without PYTHON3 defined, we can't actually run any GENERATED_FILES
scripts in the fastermake backend.
Differential Revision: https://phabricator.services.mozilla.com/D63437
--HG--
extra : moz-landing-system : lando
Without PYTHON3 defined, we can't actually run any GENERATED_FILES
scripts in the fastermake backend.
Differential Revision: https://phabricator.services.mozilla.com/D63437
--HG--
extra : moz-landing-system : lando
GENERATED_FILES now defaults to python3 unless py2=True is specified as
an argument. All existing GENERATED_FILES scripts and GeneratedFile
templates have the py2=True attribute added, so this patch should
effectively be a no-op.
Going forward, individual scripts can be converted to python3 and their
corresponding py2=True attribute can be deleted. In effect, this patch
will be backed out in pieces until all scripts run in python3, at which
point the py2 attribute itself can be removed.
Differential Revision: https://phabricator.services.mozilla.com/D60919
--HG--
extra : moz-landing-system : lando
We're going to enable this on Mac, and it won't do to have configure
assert when we actually do so.
Differential Revision: https://phabricator.services.mozilla.com/D62799
--HG--
extra : moz-landing-system : lando
NodeJS 8.x is End-of-Lifed and is no longer receiving security fixes. 10.19.0 is now the oldest Long Term Support version of NodeJS, and it has just been released with several HTTP security fixes.
Differential Revision: https://phabricator.services.mozilla.com/D62781
--HG--
extra : moz-landing-system : lando
GENERATED_FILES now defaults to python3 unless py2=True is specified as
an argument. All existing GENERATED_FILES scripts and GeneratedFile
templates have the py2=True attribute added, so this patch should
effectively be a no-op.
Going forward, individual scripts can be converted to python3 and their
corresponding py2=True attribute can be deleted. In effect, this patch
will be backed out in pieces until all scripts run in python3, at which
point the py2 attribute itself can be removed.
Differential Revision: https://phabricator.services.mozilla.com/D60919
--HG--
extra : moz-landing-system : lando
We're going to enable this on Mac, and it won't do to have configure
assert when we actually do so.
Differential Revision: https://phabricator.services.mozilla.com/D62799
--HG--
extra : moz-landing-system : lando
GENERATED_FILES now defaults to python3 unless py2=True is specified as
an argument. All existing GENERATED_FILES scripts and GeneratedFile
templates have the py2=True attribute added, so this patch should
effectively be a no-op.
Going forward, individual scripts can be converted to python3 and their
corresponding py2=True attribute can be deleted. In effect, this patch
will be backed out in pieces until all scripts run in python3, at which
point the py2 attribute itself can be removed.
Differential Revision: https://phabricator.services.mozilla.com/D60919
--HG--
extra : moz-landing-system : lando
This change adds `-ferror-limit=0` to the "command" lines in the
compile_commands.json file that's built by mach.
Differential Revision: https://phabricator.services.mozilla.com/D61392
--HG--
extra : moz-landing-system : lando
Changes:
Build string using the `format()` instead of `%` and relocate the `expandtabs` call to not trigger a AttributeError exception.
Differential Revision: https://phabricator.services.mozilla.com/D60939
--HG--
extra : moz-landing-system : lando
Changes:
Use compatibility layer provided by six for `iteritems` and `itervalues`.
Make `urlparse` import compatible with both 2/3.
Differential Revision: https://phabricator.services.mozilla.com/D60944
--HG--
extra : moz-landing-system : lando
Instead of showing percentages, show actual numbers for memory usage,
with caveats:
- On macOS, psutil doesn't seem to be getting anything that would sum up
to the total it reports, so we keep the "percent".
- While the sum of all the fields shown on Linux does sum up to the total,
the visible usage doesn't quite match what the "percent" look like for
the same dataset.
- On Windows, only "used" and "free" are available. They do sum up to
the total.
Differential Revision: https://phabricator.services.mozilla.com/D60226
--HG--
extra : moz-landing-system : lando
The original code had unfinished code to switch between different
categories, but I found it more useful to be able to see both CPU and
memory at the same time.
Differential Revision: https://phabricator.services.mozilla.com/D60111
--HG--
extra : moz-landing-system : lando
Two of these no longer have any misc:: rules associated with them, and
can be removed outright.
The remaining misc:: rule in toolkit/components/telemetry is already
traversed because it contains EXTRA_COMPONENTS, which are processed
during misc in the RecursiveMake backend. So we can remove HAS_MISC_RULE
entirely even though we still have a custom misc rule to process addons
(until bug 988938 is fixed).
Differential Revision: https://phabricator.services.mozilla.com/D59425
--HG--
extra : moz-landing-system : lando
Legacy extensions are no longer loaded, so we can drop the build config for it. We
still need flags for handling experimental APIs since what we require differs between builds
and distributions.
Differential Revision: https://phabricator.services.mozilla.com/D57413
--HG--
extra : moz-landing-system : lando
This gets rid of the last use of the 'which' module left in the tree. So not
only will this help 'configure' become a little more Python 3 compatible, but
we can now remove 'third_party/python/which'.
Differential Revision: https://phabricator.services.mozilla.com/D37427
--HG--
extra : moz-landing-system : lando
This gets rid of the last use of the 'which' module left in the tree. So not
only will this help 'configure' become a little more Python 3 compatible, but
we can now remove 'third_party/python/which'.
Differential Revision: https://phabricator.services.mozilla.com/D37427
--HG--
extra : moz-landing-system : lando
Since we have a limit of the number of arguments length that the shell can support we have to
pass the files for the analysis to `run-clang-tidy.py` in chuncks, calculated in proportion
to the number or logical cpu cores.
Differential Revision: https://phabricator.services.mozilla.com/D58971
--HG--
extra : moz-landing-system : lando
Many of the utility classes very heavily rely on multiple inheritance which is broadly unncessary and made debugging difficult, so I ripped it out wherever it was necessary. The indented_repr() function is also more or less a re-implementation of pprint, so to support Python 3 I've instead made it a small wrapper around pprint. (This results in the output of the function being slightly different than we're used to, as pprint does sorting/indentation/line breaking differently than indented_repr() currently does, but it'll be nice to not have to maintain this code that is slightly different than pprint for no real reason.)
Differential Revision: https://phabricator.services.mozilla.com/D58306
--HG--
extra : moz-landing-system : lando
Warning flags might have been generated for a different compiler, so we
can't use them here. The same argument applies to coverage flags.
Depends on D58070
Differential Revision: https://phabricator.services.mozilla.com/D58071
--HG--
extra : moz-landing-system : lando
The optimization flags that may have been chosen for our target compiler
are not necessarily appropriate for our wasm compiler.
Differential Revision: https://phabricator.services.mozilla.com/D58070
--HG--
extra : moz-landing-system : lando
There was only a single use of 'install-to-subdir' in all our manifests (which
was removed in the previous commit). All the comments around it suggest that it
was a giant hack. Removing it cleans up a lot of complexity and saves me the
trouble of needing to support it in the new sourcedir based TestResolver.
Differential Revision: https://phabricator.services.mozilla.com/D57170
--HG--
extra : moz-landing-system : lando
By default we create both the RecursiveMake and FasterMake backends, but
building.py is only checking if the RecursiveMake backend is out of date
to decide if the backend needs to be updated. Since the FasterMake
backend may have different input files, we also need to check if that's
out of date, otherwise renaming certain files can cause './mach build
faster' to break.
Differential Revision: https://phabricator.services.mozilla.com/D58010
--HG--
extra : moz-landing-system : lando
Change import order.
Enable python3 testing for `test_copier.py`.
In the test methods, ensure the returned file names are decoded as UTF-8 by default.
In the test methods, check for byte-string by default; in python2, this would be treated like the standard string.
Differential Revision: https://phabricator.services.mozilla.com/D55187
--HG--
extra : moz-landing-system : lando
This patch causes the newest version of NodeJS 8.17.0 to be used to build the mozilla toolchains installed on the CI machines as well as by `mach bootstrap`, which in turn gets an important security fix by updating to npm 6.13.4.
This patch also causes mach commands which call npm to abort if the version in use is older than 6.13.4 to avoid (especially) exploits on local developers that exfiltrate their data.
Differential Revision: https://phabricator.services.mozilla.com/D57674
--HG--
extra : moz-landing-system : lando
Currently BuildReader._relevant_mozbuild_finder attempted to exclude objdirs by
ignoring the pattern 'obj*'. However this can cause problems for developers who
don't prefix their objdirs with the string 'obj'. This attempts to ignore all
objdirs by looking for 'config.status' files in the topsrcdir.
This wasn't a huge issue before because BuildReader._relevant_mozbuild_finder
was only used in a handful of obscure edge cases. But now it's being invoked by
the taskgraph and consequently |mach try|. So we'll see this issue pop up for
people more frequently.
Differential Revision: https://phabricator.services.mozilla.com/D57113
--HG--
extra : moz-landing-system : lando
This change modifies all tests that use key3/cert8 to use the new files. It
removes test_sdr_upgraded_with_password, as without the upgrade part that is now
the same test as test_sdr_preexisting_with_password.
Differential Revision: https://phabricator.services.mozilla.com/D55708
--HG--
rename : security/manager/ssl/tests/unit/test_sdr_preexisting/key4.db => security/manager/ssl/tests/unit/test_broken_fips/key4.db
extra : moz-landing-system : lando
This change modifies all tests that use key3/cert8 to use the new files. It
removes test_sdr_upgraded_with_password, as without the upgrade part that is now
the same test as test_sdr_preexisting_with_password.
Differential Revision: https://phabricator.services.mozilla.com/D55708
--HG--
rename : security/manager/ssl/tests/unit/test_sdr_preexisting/key4.db => security/manager/ssl/tests/unit/test_broken_fips/key4.db
extra : moz-landing-system : lando
This change removes the legacy libnssdbm database that we migrated away from since Firefox 60.
This change modifies all tests that use key3/cert8 to use the new files. It
removes test_sdr_upgraded_with_password, as without the upgrade part that is now
the same test as test_sdr_preexisting_with_password. It otherwise removes support for libnssdbm everywhere in Gecko.
Differential Revision: https://phabricator.services.mozilla.com/D55708
--HG--
rename : security/manager/ssl/tests/unit/test_sdr_preexisting/key4.db => security/manager/ssl/tests/unit/test_broken_fips/key4.db
extra : moz-landing-system : lando
This optionally uses the rust-based `dump_syms` toolchain if it has been
installed, otherwise it uses the locally built version.
Differential Revision: https://phabricator.services.mozilla.com/D53320
--HG--
extra : moz-landing-system : lando
Bug 1295542 added empty chrome.manifest files to prevent malware from abusing
them. This workaround is no longer necessary because Firefox stopped reading
chrome.manifest outside omni.ja since bug 1543761.
Differential Revision: https://phabricator.services.mozilla.com/D55954
--HG--
extra : moz-landing-system : lando
Make the $PYTHON3 build var point to a full virtualenv bootstrapped with
the same libraries as the $PYTHON Python 2 build var. This allows us to
upgrade build tasks from $PYTHON to $PYTHON3.
This patch adds some debug logging and documentation to the Python
2 virtualenv so that it is easier to diagnose issues that may arise
from running two different Python interpreters in re-entrant
multiprocess routines.
Differential Revision: https://phabricator.services.mozilla.com/D50819
--HG--
extra : moz-landing-system : lando
This optionally uses the rust-based `dump_syms` toolchain if it has been
installed, otherwise it uses the locally built version.
Differential Revision: https://phabricator.services.mozilla.com/D53320
--HG--
extra : moz-landing-system : lando
In 1-tier PGO builds that shared the objdir between the instrumented and
profile-use builds, the instrumented build objects used a different
suffix (.i_o) to separate them from the profile-use build (which uses
the default .o suffix). These builds are now always in separate objdirs,
and don't need special suffix rules anymore.
As a bonus, this helps fix an issue with buildid.cpp continually
rebuilding because libxul_so.list always lists the inputs as *.o, which
don't exist if we're using a .i_o suffix. Make would always re-create
buildid.cpp and therefore libxul.so in the instrumented build even when
nothing has changed.
Differential Revision: https://phabricator.services.mozilla.com/D56115
--HG--
extra : moz-landing-system : lando
Allow running the jsshell using ./mach run. This copies the debugger
parameter handling from desktop configurations.
Differential Revision: https://phabricator.services.mozilla.com/D56296
--HG--
extra : moz-landing-system : lando
Local PGO builds now use 3-tier machinery under the hood. Instead of a
single object directory that gets cleaned in between the instrumented
and profile-use builds, now the instrumented build uses a separate
'${objdir}/instrumented' directory. This makes it easier to handle
within mach since we can drive the two builds with environment variables
and keep all build artifacts separate, without needing to do manual
cleanup in between.
Differential Revision: https://phabricator.services.mozilla.com/D50098
--HG--
extra : moz-landing-system : lando
This patch addresses the following issues which prevent building the
documentation on a Windows system:
1. check_jsdoc attempts to launch a file by the name of jsdoc using
subprocess.check_output(). But, the file jsdoc is a Bash script,
so it can't be launched this way on Windows. We should find the
correct file name to use instead (likely jsdoc.cmd).
2. The Windows CopyFile API will fail if the destination directory does
not exist. The procedure that creates the staging directory for Sphinx
attempts to use this API before any directories are created, so most calls
to it fail and most files don't get staged. These errors are ignored, so
the result is that Sphinx succeeds in generating a set of documentation
that is mostly empty.
3. Several instances where manually constructed paths that contain both back
and forward slashes are compared to or otherwise used with normalized paths,
resulting in incorrect behaviors.
Differential Revision: https://phabricator.services.mozilla.com/D54549
--HG--
extra : moz-landing-system : lando
Remove the "clever" logic from patch D53070, which was meant to reduce code duplication, and replace with easier-to-understand logic. Copy-and-paste the previous version of the CompileFlags class before patch D53070 to ensure that we stamp out the regression.
Differential Revision: https://phabricator.services.mozilla.com/D55472
--HG--
extra : moz-landing-system : lando
This change does not fully enable C++17, as we still need standard
library support from some platforms.
Differential Revision: https://phabricator.services.mozilla.com/D54202
--HG--
extra : moz-landing-system : lando
This change does not fully enable C++17, as we still need standard
library support from some platforms.
Differential Revision: https://phabricator.services.mozilla.com/D54202
--HG--
extra : moz-landing-system : lando
Ensure that environment variables with unicode values are handled correctly
when setting up a virtualenv.
Differential Revision: https://phabricator.services.mozilla.com/D55019
--HG--
extra : moz-landing-system : lando
Add backend stuff to build sandboxed wasm libraries. (Don't actually update any moz.build files to consume this yet.)
Differential Revision: https://phabricator.services.mozilla.com/D54152
--HG--
extra : moz-landing-system : lando
Make the $PYTHON3 build var point to a full virtualenv bootstrapped with
the same libraries as the $PYTHON Python 2 build var. This allows us to
upgrade build tasks from $PYTHON to $PYTHON3.
This patch adds some debug logging and documentation to the Python
2 virtualenv so that it is easier to diagnose issues that may arise
from running two different Python interpreters in re-entrant
multiprocess routines.
Differential Revision: https://phabricator.services.mozilla.com/D50819
--HG--
extra : moz-landing-system : lando
Changes:
Remove `environment` from the python3 blacklist.
Remove the option to specify `configure` for the formatting argument, since it has never been supported (the method has never been implemented).
Differential Revision: https://phabricator.services.mozilla.com/D54395
--HG--
extra : moz-landing-system : lando
The FileAvoidWrite class does a bunch of stuff with bytes and strings
that doesn't work in Python 3. Make sure the object is handling only
bytes or only strings under the hood so Python 3 is happy.
The FileAvoidWrite unit tests written with MockedOpen don't work in
Python 3 either. Swap them out for vanilla pytest tests without
the MockedOpen dependency.
Differential Revision: https://phabricator.services.mozilla.com/D51341
--HG--
extra : moz-landing-system : lando
`os.path.abspath` on Windows doesn't return a consistent value -- the capitalization can vary based on environmental factors that I don't fully understand. Regardless, normalize case where this function is called in config_status.
Differential Revision: https://phabricator.services.mozilla.com/D53889
--HG--
extra : moz-landing-system : lando
If configure is invoked manually, clobber.py is bypassed and the CLOBBER
file doesn't get touched. The clobber check in Makefile.in gets
triggered causing the build to stop.
Moving the objdir/CLOBBER file creation into configure.py should cause
it to be created regardless of how configure is invoked.
Depends on D53290
Differential Revision: https://phabricator.services.mozilla.com/D53291
--HG--
extra : moz-landing-system : lando
Bug 1596868 was triggered by forgetting a comma, turning an intended
tuple into a string, which Python dutifully iterated over as a sequence
of single characters. Let's add some basic typechecking so these sort
of mistakes don't happen again.
Differential Revision: https://phabricator.services.mozilla.com/D53815
--HG--
extra : moz-landing-system : lando
The minidump-analyzer tool was originally conceived to be used from the crash
report client and as such was installed in the crash reporter client
application bundle on macOS. It was later adapted to work from Firefox itself
but this caused linking problems when invoked from the Firefox app bundle.
This patch moves the minidump-analyzer into the Firefox app bundle and adapts
the relevant code to find it there.
The minidump-analyzer was also not signed like the rest of our executables and
this patch addresses that issue too.
Differential Revision: https://phabricator.services.mozilla.com/D52910
--HG--
extra : moz-landing-system : lando
We need this for "full" C++17 support (everything is supported, but some
C++17 features still have bugs) and this change also brings Linux into
parity with our Mac requirements.
MANUAL PUSH: build toolchains on inbound to avoid clogging autoland
Differential Revision: https://phabricator.services.mozilla.com/D51450
This loader uses 'reader.find_variables_from_ast' to parse all *_MANIFESTS variables from
moz.build files using the abstract syntax tree. This means it will find all such variables
regardless of the current buildconfig.
Differential Revision: https://phabricator.services.mozilla.com/D51834
--HG--
extra : moz-landing-system : lando
Despite what the comment says, the finder *does* pick up the root moz.build. So
we end up processing it twice. This was never caught before because we only ever
used this function to read Sphinx related variables, of which the root moz.build
contains none.
Differential Revision: https://phabricator.services.mozilla.com/D51832
--HG--
extra : moz-landing-system : lando
Changes:
- use regular strings instead of byte strings when matching
- encode files in UTF-8 prior to hashing
Depends on D51414
Differential Revision: https://phabricator.services.mozilla.com/D51728
--HG--
extra : moz-landing-system : lando
Changes:
- change `urlparse` import to `urllib.parse` to make switch over to python3
Depends on D51414
Differential Revision: https://phabricator.services.mozilla.com/D51648
--HG--
extra : moz-landing-system : lando
Changes:
- update sections of `generate_sources_mozbuild.py` and `cmakeparser.py` to be python3 compatible
- change import of `urlparse` to be python3 compatible
Depends on D51414
Differential Revision: https://phabricator.services.mozilla.com/D51692
--HG--
extra : moz-landing-system : lando
Add test annotations to the ReftestManifest and TestResolver. For example, a
test description from the TestResolver might now contain 'skip-if': 'skiaContent';
similar to the content provided for manifestparser tests; this will allow
'mach test-info report' to filter tests based on reftest manifest test
annotations.
Also add the referenced-test field which identifies the test file associated
with test entries for reference files; this will allow test-verify to
run the correct reftest when only the reference file has been modified.
Differential Revision: https://phabricator.services.mozilla.com/D51113
--HG--
extra : moz-landing-system : lando
This allows subclasses of MozbuildObject to define additional instance
arguments and still use 'from_environment'.
Differential Revision: https://phabricator.services.mozilla.com/D51174
--HG--
extra : moz-landing-system : lando
On Windows, './mach bootstrap' downloads 8 artifacts taking up 1GiB+ of
space. Given the current limits of 6 artifacts and 1GiB, this means a
full run of './mach bootstrap' doesn't actually use anything from the
cache. Each new artifact is over the limits, so the oldest artifact gets
removed from the cache before downloading the next one. Doubling the
limits here should give us some space to work with.
Depends on D50676
Differential Revision: https://phabricator.services.mozilla.com/D50677
--HG--
extra : moz-landing-system : lando
Artifacts are fetched to a local cache, and if the artifact in the cache
exists it won't be re-downloaded. However, the messaging suggested that
artifacts were always being downloaded when they were just re-used from
the cache, leading to confusion.
Differential Revision: https://phabricator.services.mozilla.com/D50676
--HG--
extra : moz-landing-system : lando
Separating these changes out into a separate patch/bug makes them a
little easier to verify. Turning on C++17 support should then be just a
matter of updating expected results.
Differential Revision: https://phabricator.services.mozilla.com/D47324
--HG--
extra : moz-landing-system : lando
Changes:
- remove the `unicode_literals` import from `test_archive.py` as it was causing failures when dealing with file paths
- mark failing tests with `xfail` annotations, to be investigated at a later date
Differential Revision: https://phabricator.services.mozilla.com/D49498
--HG--
extra : moz-landing-system : lando
Add code to clean up Git and Hg repositories, and invoke that where `./mach vendor rust` would throw an error. Unfortunately, `cargo vendor` also updates the repo's root `Cargo.lock` file in-place and while we could `git checkout`/`hg revert` that file for the user, `Cargo.lock` may have had pre-existing changes that would be overwritten by such a change. Instead of a potentially destructive update in the error case, I've opted to add an extra error message to tell you how to deal with that one file.
Differential Revision: https://phabricator.services.mozilla.com/D49494
--HG--
extra : moz-landing-system : lando
Changes:
- remove `uuid` from the python3 blacklist in `mach`
- enable `test_telemetry.py` for python3
- adjust test outcome expectation for Windows + python3, suspect `mozpack.path` is not filtering Windows path correctly in python3 environment
- switch file read mode between `r` and `rb` depending on version of python
Differential Revision: https://phabricator.services.mozilla.com/D45903
--HG--
extra : moz-landing-system : lando
The BuildReader's 'find_sphinx_variables' function is hardcoded to look for the
SPHINX_TREES and SPHINX_PYTHON_PACKAGE_DIRS variables. But it's otherwise
implemented to find any arbitrary variable (as long as they are a simple list
or dict).
This change simply moves the variable name(s) to the function call. The comment
about possibly wanted to use a higher level AST library to parse other kinds of
variables still applies, but for now this change is good enough to suit my
needs.
Differential Revision: https://phabricator.services.mozilla.com/D48424
--HG--
extra : moz-landing-system : lando
Fennec is removed, but mach install still tries to install Fennec. It should
install GVE instead.
Differential Revision: https://phabricator.services.mozilla.com/D48811
--HG--
extra : moz-landing-system : lando
Some generated files are missing in generated-sources.json so just add them.
Differential Revision: https://phabricator.services.mozilla.com/D48085
--HG--
extra : moz-landing-system : lando
This leaves the testharness files, because they are used in various mochitests.
Differential Revision: https://phabricator.services.mozilla.com/D49132
--HG--
extra : moz-landing-system : lando
This leaves the testharness files, because they are used in mochitest-chrome
tests in dom/animation/test.
Differential Revision: https://phabricator.services.mozilla.com/D49132
--HG--
extra : moz-landing-system : lando
This leaves the testharness files, because they are used in mochitest-chrome
tests in dom/animation/test.
Differential Revision: https://phabricator.services.mozilla.com/D49132
--HG--
extra : moz-landing-system : lando
Adds a way for mochitest, reftest, and crashtests to skip XBL related
tests when XBL is disabled. Also, add an app constant so JS can
check whether XBL is enabled.
Depends on D45614
Differential Revision: https://phabricator.services.mozilla.com/D45615
--HG--
extra : moz-landing-system : lando
