The motivating change here was wpt-sync try pushes failing because
`libgraphitewasm.dylib` wasn't getting packaged in Mac artifact builds.
We add `libosclientcerts.dylib` because packaging was complaining about
that library as well.
Differential Revision: https://phabricator.services.mozilla.com/D63705
--HG--
extra : moz-landing-system : lando
Without PYTHON3 defined, we can't actually run any GENERATED_FILES
scripts in the fastermake backend.
Differential Revision: https://phabricator.services.mozilla.com/D63437
--HG--
extra : moz-landing-system : lando
Without PYTHON3 defined, we can't actually run any GENERATED_FILES
scripts in the fastermake backend.
Differential Revision: https://phabricator.services.mozilla.com/D63437
--HG--
extra : moz-landing-system : lando
GENERATED_FILES now defaults to python3 unless py2=True is specified as
an argument. All existing GENERATED_FILES scripts and GeneratedFile
templates have the py2=True attribute added, so this patch should
effectively be a no-op.
Going forward, individual scripts can be converted to python3 and their
corresponding py2=True attribute can be deleted. In effect, this patch
will be backed out in pieces until all scripts run in python3, at which
point the py2 attribute itself can be removed.
Differential Revision: https://phabricator.services.mozilla.com/D60919
--HG--
extra : moz-landing-system : lando
We're going to enable this on Mac, and it won't do to have configure
assert when we actually do so.
Differential Revision: https://phabricator.services.mozilla.com/D62799
--HG--
extra : moz-landing-system : lando
NodeJS 8.x is End-of-Lifed and is no longer receiving security fixes. 10.19.0 is now the oldest Long Term Support version of NodeJS, and it has just been released with several HTTP security fixes.
Differential Revision: https://phabricator.services.mozilla.com/D62781
--HG--
extra : moz-landing-system : lando
GENERATED_FILES now defaults to python3 unless py2=True is specified as
an argument. All existing GENERATED_FILES scripts and GeneratedFile
templates have the py2=True attribute added, so this patch should
effectively be a no-op.
Going forward, individual scripts can be converted to python3 and their
corresponding py2=True attribute can be deleted. In effect, this patch
will be backed out in pieces until all scripts run in python3, at which
point the py2 attribute itself can be removed.
Differential Revision: https://phabricator.services.mozilla.com/D60919
--HG--
extra : moz-landing-system : lando
We're going to enable this on Mac, and it won't do to have configure
assert when we actually do so.
Differential Revision: https://phabricator.services.mozilla.com/D62799
--HG--
extra : moz-landing-system : lando
GENERATED_FILES now defaults to python3 unless py2=True is specified as
an argument. All existing GENERATED_FILES scripts and GeneratedFile
templates have the py2=True attribute added, so this patch should
effectively be a no-op.
Going forward, individual scripts can be converted to python3 and their
corresponding py2=True attribute can be deleted. In effect, this patch
will be backed out in pieces until all scripts run in python3, at which
point the py2 attribute itself can be removed.
Differential Revision: https://phabricator.services.mozilla.com/D60919
--HG--
extra : moz-landing-system : lando
This change adds `-ferror-limit=0` to the "command" lines in the
compile_commands.json file that's built by mach.
Differential Revision: https://phabricator.services.mozilla.com/D61392
--HG--
extra : moz-landing-system : lando
Changes:
Build string using the `format()` instead of `%` and relocate the `expandtabs` call to not trigger a AttributeError exception.
Differential Revision: https://phabricator.services.mozilla.com/D60939
--HG--
extra : moz-landing-system : lando
Changes:
Use compatibility layer provided by six for `iteritems` and `itervalues`.
Make `urlparse` import compatible with both 2/3.
Differential Revision: https://phabricator.services.mozilla.com/D60944
--HG--
extra : moz-landing-system : lando
Instead of showing percentages, show actual numbers for memory usage,
with caveats:
- On macOS, psutil doesn't seem to be getting anything that would sum up
to the total it reports, so we keep the "percent".
- While the sum of all the fields shown on Linux does sum up to the total,
the visible usage doesn't quite match what the "percent" look like for
the same dataset.
- On Windows, only "used" and "free" are available. They do sum up to
the total.
Differential Revision: https://phabricator.services.mozilla.com/D60226
--HG--
extra : moz-landing-system : lando
The original code had unfinished code to switch between different
categories, but I found it more useful to be able to see both CPU and
memory at the same time.
Differential Revision: https://phabricator.services.mozilla.com/D60111
--HG--
extra : moz-landing-system : lando
Two of these no longer have any misc:: rules associated with them, and
can be removed outright.
The remaining misc:: rule in toolkit/components/telemetry is already
traversed because it contains EXTRA_COMPONENTS, which are processed
during misc in the RecursiveMake backend. So we can remove HAS_MISC_RULE
entirely even though we still have a custom misc rule to process addons
(until bug 988938 is fixed).
Differential Revision: https://phabricator.services.mozilla.com/D59425
--HG--
extra : moz-landing-system : lando
Legacy extensions are no longer loaded, so we can drop the build config for it. We
still need flags for handling experimental APIs since what we require differs between builds
and distributions.
Differential Revision: https://phabricator.services.mozilla.com/D57413
--HG--
extra : moz-landing-system : lando
This gets rid of the last use of the 'which' module left in the tree. So not
only will this help 'configure' become a little more Python 3 compatible, but
we can now remove 'third_party/python/which'.
Differential Revision: https://phabricator.services.mozilla.com/D37427
--HG--
extra : moz-landing-system : lando
This gets rid of the last use of the 'which' module left in the tree. So not
only will this help 'configure' become a little more Python 3 compatible, but
we can now remove 'third_party/python/which'.
Differential Revision: https://phabricator.services.mozilla.com/D37427
--HG--
extra : moz-landing-system : lando
Since we have a limit of the number of arguments length that the shell can support we have to
pass the files for the analysis to `run-clang-tidy.py` in chuncks, calculated in proportion
to the number or logical cpu cores.
Differential Revision: https://phabricator.services.mozilla.com/D58971
--HG--
extra : moz-landing-system : lando
Many of the utility classes very heavily rely on multiple inheritance which is broadly unncessary and made debugging difficult, so I ripped it out wherever it was necessary. The indented_repr() function is also more or less a re-implementation of pprint, so to support Python 3 I've instead made it a small wrapper around pprint. (This results in the output of the function being slightly different than we're used to, as pprint does sorting/indentation/line breaking differently than indented_repr() currently does, but it'll be nice to not have to maintain this code that is slightly different than pprint for no real reason.)
Differential Revision: https://phabricator.services.mozilla.com/D58306
--HG--
extra : moz-landing-system : lando
Warning flags might have been generated for a different compiler, so we
can't use them here. The same argument applies to coverage flags.
Depends on D58070
Differential Revision: https://phabricator.services.mozilla.com/D58071
--HG--
extra : moz-landing-system : lando
The optimization flags that may have been chosen for our target compiler
are not necessarily appropriate for our wasm compiler.
Differential Revision: https://phabricator.services.mozilla.com/D58070
--HG--
extra : moz-landing-system : lando
There was only a single use of 'install-to-subdir' in all our manifests (which
was removed in the previous commit). All the comments around it suggest that it
was a giant hack. Removing it cleans up a lot of complexity and saves me the
trouble of needing to support it in the new sourcedir based TestResolver.
Differential Revision: https://phabricator.services.mozilla.com/D57170
--HG--
extra : moz-landing-system : lando
By default we create both the RecursiveMake and FasterMake backends, but
building.py is only checking if the RecursiveMake backend is out of date
to decide if the backend needs to be updated. Since the FasterMake
backend may have different input files, we also need to check if that's
out of date, otherwise renaming certain files can cause './mach build
faster' to break.
Differential Revision: https://phabricator.services.mozilla.com/D58010
--HG--
extra : moz-landing-system : lando
Change import order.
Enable python3 testing for `test_copier.py`.
In the test methods, ensure the returned file names are decoded as UTF-8 by default.
In the test methods, check for byte-string by default; in python2, this would be treated like the standard string.
Differential Revision: https://phabricator.services.mozilla.com/D55187
--HG--
extra : moz-landing-system : lando
This patch causes the newest version of NodeJS 8.17.0 to be used to build the mozilla toolchains installed on the CI machines as well as by `mach bootstrap`, which in turn gets an important security fix by updating to npm 6.13.4.
This patch also causes mach commands which call npm to abort if the version in use is older than 6.13.4 to avoid (especially) exploits on local developers that exfiltrate their data.
Differential Revision: https://phabricator.services.mozilla.com/D57674
--HG--
extra : moz-landing-system : lando
Currently BuildReader._relevant_mozbuild_finder attempted to exclude objdirs by
ignoring the pattern 'obj*'. However this can cause problems for developers who
don't prefix their objdirs with the string 'obj'. This attempts to ignore all
objdirs by looking for 'config.status' files in the topsrcdir.
This wasn't a huge issue before because BuildReader._relevant_mozbuild_finder
was only used in a handful of obscure edge cases. But now it's being invoked by
the taskgraph and consequently |mach try|. So we'll see this issue pop up for
people more frequently.
Differential Revision: https://phabricator.services.mozilla.com/D57113
--HG--
extra : moz-landing-system : lando
This change modifies all tests that use key3/cert8 to use the new files. It
removes test_sdr_upgraded_with_password, as without the upgrade part that is now
the same test as test_sdr_preexisting_with_password.
Differential Revision: https://phabricator.services.mozilla.com/D55708
--HG--
rename : security/manager/ssl/tests/unit/test_sdr_preexisting/key4.db => security/manager/ssl/tests/unit/test_broken_fips/key4.db
extra : moz-landing-system : lando
This change modifies all tests that use key3/cert8 to use the new files. It
removes test_sdr_upgraded_with_password, as without the upgrade part that is now
the same test as test_sdr_preexisting_with_password.
Differential Revision: https://phabricator.services.mozilla.com/D55708
--HG--
rename : security/manager/ssl/tests/unit/test_sdr_preexisting/key4.db => security/manager/ssl/tests/unit/test_broken_fips/key4.db
extra : moz-landing-system : lando
This change removes the legacy libnssdbm database that we migrated away from since Firefox 60.
This change modifies all tests that use key3/cert8 to use the new files. It
removes test_sdr_upgraded_with_password, as without the upgrade part that is now
the same test as test_sdr_preexisting_with_password. It otherwise removes support for libnssdbm everywhere in Gecko.
Differential Revision: https://phabricator.services.mozilla.com/D55708
--HG--
rename : security/manager/ssl/tests/unit/test_sdr_preexisting/key4.db => security/manager/ssl/tests/unit/test_broken_fips/key4.db
extra : moz-landing-system : lando
This optionally uses the rust-based `dump_syms` toolchain if it has been
installed, otherwise it uses the locally built version.
Differential Revision: https://phabricator.services.mozilla.com/D53320
--HG--
extra : moz-landing-system : lando
Bug 1295542 added empty chrome.manifest files to prevent malware from abusing
them. This workaround is no longer necessary because Firefox stopped reading
chrome.manifest outside omni.ja since bug 1543761.
Differential Revision: https://phabricator.services.mozilla.com/D55954
--HG--
extra : moz-landing-system : lando
Make the $PYTHON3 build var point to a full virtualenv bootstrapped with
the same libraries as the $PYTHON Python 2 build var. This allows us to
upgrade build tasks from $PYTHON to $PYTHON3.
This patch adds some debug logging and documentation to the Python
2 virtualenv so that it is easier to diagnose issues that may arise
from running two different Python interpreters in re-entrant
multiprocess routines.
Differential Revision: https://phabricator.services.mozilla.com/D50819
--HG--
extra : moz-landing-system : lando
This optionally uses the rust-based `dump_syms` toolchain if it has been
installed, otherwise it uses the locally built version.
Differential Revision: https://phabricator.services.mozilla.com/D53320
--HG--
extra : moz-landing-system : lando
In 1-tier PGO builds that shared the objdir between the instrumented and
profile-use builds, the instrumented build objects used a different
suffix (.i_o) to separate them from the profile-use build (which uses
the default .o suffix). These builds are now always in separate objdirs,
and don't need special suffix rules anymore.
As a bonus, this helps fix an issue with buildid.cpp continually
rebuilding because libxul_so.list always lists the inputs as *.o, which
don't exist if we're using a .i_o suffix. Make would always re-create
buildid.cpp and therefore libxul.so in the instrumented build even when
nothing has changed.
Differential Revision: https://phabricator.services.mozilla.com/D56115
--HG--
extra : moz-landing-system : lando
Allow running the jsshell using ./mach run. This copies the debugger
parameter handling from desktop configurations.
Differential Revision: https://phabricator.services.mozilla.com/D56296
--HG--
extra : moz-landing-system : lando
Local PGO builds now use 3-tier machinery under the hood. Instead of a
single object directory that gets cleaned in between the instrumented
and profile-use builds, now the instrumented build uses a separate
'${objdir}/instrumented' directory. This makes it easier to handle
within mach since we can drive the two builds with environment variables
and keep all build artifacts separate, without needing to do manual
cleanup in between.
Differential Revision: https://phabricator.services.mozilla.com/D50098
--HG--
extra : moz-landing-system : lando
This patch addresses the following issues which prevent building the
documentation on a Windows system:
1. check_jsdoc attempts to launch a file by the name of jsdoc using
subprocess.check_output(). But, the file jsdoc is a Bash script,
so it can't be launched this way on Windows. We should find the
correct file name to use instead (likely jsdoc.cmd).
2. The Windows CopyFile API will fail if the destination directory does
not exist. The procedure that creates the staging directory for Sphinx
attempts to use this API before any directories are created, so most calls
to it fail and most files don't get staged. These errors are ignored, so
the result is that Sphinx succeeds in generating a set of documentation
that is mostly empty.
3. Several instances where manually constructed paths that contain both back
and forward slashes are compared to or otherwise used with normalized paths,
resulting in incorrect behaviors.
Differential Revision: https://phabricator.services.mozilla.com/D54549
--HG--
extra : moz-landing-system : lando
Remove the "clever" logic from patch D53070, which was meant to reduce code duplication, and replace with easier-to-understand logic. Copy-and-paste the previous version of the CompileFlags class before patch D53070 to ensure that we stamp out the regression.
Differential Revision: https://phabricator.services.mozilla.com/D55472
--HG--
extra : moz-landing-system : lando
This change does not fully enable C++17, as we still need standard
library support from some platforms.
Differential Revision: https://phabricator.services.mozilla.com/D54202
--HG--
extra : moz-landing-system : lando
This change does not fully enable C++17, as we still need standard
library support from some platforms.
Differential Revision: https://phabricator.services.mozilla.com/D54202
--HG--
extra : moz-landing-system : lando
Ensure that environment variables with unicode values are handled correctly
when setting up a virtualenv.
Differential Revision: https://phabricator.services.mozilla.com/D55019
--HG--
extra : moz-landing-system : lando
Add backend stuff to build sandboxed wasm libraries. (Don't actually update any moz.build files to consume this yet.)
Differential Revision: https://phabricator.services.mozilla.com/D54152
--HG--
extra : moz-landing-system : lando
Make the $PYTHON3 build var point to a full virtualenv bootstrapped with
the same libraries as the $PYTHON Python 2 build var. This allows us to
upgrade build tasks from $PYTHON to $PYTHON3.
This patch adds some debug logging and documentation to the Python
2 virtualenv so that it is easier to diagnose issues that may arise
from running two different Python interpreters in re-entrant
multiprocess routines.
Differential Revision: https://phabricator.services.mozilla.com/D50819
--HG--
extra : moz-landing-system : lando
Changes:
Remove `environment` from the python3 blacklist.
Remove the option to specify `configure` for the formatting argument, since it has never been supported (the method has never been implemented).
Differential Revision: https://phabricator.services.mozilla.com/D54395
--HG--
extra : moz-landing-system : lando
The FileAvoidWrite class does a bunch of stuff with bytes and strings
that doesn't work in Python 3. Make sure the object is handling only
bytes or only strings under the hood so Python 3 is happy.
The FileAvoidWrite unit tests written with MockedOpen don't work in
Python 3 either. Swap them out for vanilla pytest tests without
the MockedOpen dependency.
Differential Revision: https://phabricator.services.mozilla.com/D51341
--HG--
extra : moz-landing-system : lando
`os.path.abspath` on Windows doesn't return a consistent value -- the capitalization can vary based on environmental factors that I don't fully understand. Regardless, normalize case where this function is called in config_status.
Differential Revision: https://phabricator.services.mozilla.com/D53889
--HG--
extra : moz-landing-system : lando
If configure is invoked manually, clobber.py is bypassed and the CLOBBER
file doesn't get touched. The clobber check in Makefile.in gets
triggered causing the build to stop.
Moving the objdir/CLOBBER file creation into configure.py should cause
it to be created regardless of how configure is invoked.
Depends on D53290
Differential Revision: https://phabricator.services.mozilla.com/D53291
--HG--
extra : moz-landing-system : lando
Bug 1596868 was triggered by forgetting a comma, turning an intended
tuple into a string, which Python dutifully iterated over as a sequence
of single characters. Let's add some basic typechecking so these sort
of mistakes don't happen again.
Differential Revision: https://phabricator.services.mozilla.com/D53815
--HG--
extra : moz-landing-system : lando