Most HG commands use subprocesses, even if a context manager (and therefore an
hglib client) has been created. There are only two commands that make use of
the client, but they *only* work inside a context manager. I don't think
there are any technical reason these two commands *need* to use the context
manager.
This patch merges the HgRepository._run_in_client function with
HgRepository._run(). If a client exists, that will be used, otherwise a
subprocess will be used.
Differential Revision: https://phabricator.services.mozilla.com/D1809
--HG--
extra : moz-landing-system : lando
This adds just enough host shared library support for this one use case,
but also takes shortcuts, because fully supporting host shared library
is a deep rabbit hole I'm not ready to take just to fix --enable-lto
--enable-clang-plugin on mac builds.
One downside is that one my machine the plugin now takes > 80s to build,
instead of 15s before, thanks to the lack of unified sources.
--HG--
extra : rebase_source : bf52a72a01d4e3eb77cf52b646b19734b9273075
The test checks the modules imported, which means it will fail if we import mozbuild.base to determine the topsrcdir. Rather than include the additional modules, which may change in the future, I have narrowed the scope of the test to the mozbuild/tests module.
MozReview-Commit-ID: GiaPzp19adE
--HG--
extra : rebase_source : 9d79ad79bef2b45e3f62190200bc46f21d57f90d
Because we have no linking configure test in python configure (yet), we
just make old-configure tests use LINKER_LDFLAGS, and make those flags
added to LDFLAGS by old-configure at the same time.
--HG--
extra : rebase_source : 80ab7c5021c7ddd1b53d58ef76cd4372a524d3cb
vendor_rust.py checks all dirs in third_party/rust/ for license information, but it wasn't distinguishing between subdirs and files when listing the contents of that directory, and the script earlier causes macOS to create a .DS_Store file in the directory, which the script then treats as a subdir, causing an IOError "not a directory". This change excludes the .DS_Store file (and other non-dirs) from the list of subdirs of third_party/rust/ to check for license info.
MozReview-Commit-ID: 4CrJH9tZLnG
--HG--
extra : rebase_source : 464ad17f4c9b76303f9024d8eaad475b1fc248f7
Added `./mach python-safety`, distinct from python-test so it doesn't have
to be run on every CI job - its errors may not depend on the area the push has changed.
Added the python/safety directory to ensure a different Pipfile is used, avoiding
conflicts with python-test.
Differential Revision: https://phabricator.services.mozilla.com/D1825
--HG--
extra : moz-landing-system : lando
We're well overdue for an upgrade of the rust compiler requirements.
Now that we're building with 1.28 (albeit a beta, due to be bumped when
it's released), we can bump the requirement away from 1.24 which is now
old. 1.27 is too new, though, so settle for the older 1.26.
--HG--
extra : rebase_source : a17aa496bf3d4af4d1349d69a637c686c6817d0f
This will make sure that when running |mach python-test --python 3| locally,
we only run the tests that also run in CI with python 3 (and therefore pass
presumably).
MozReview-Commit-ID: 3OBr9yLSlSq
--HG--
extra : rebase_source : 456340d0ecdddf1078f2b5b4ebb1eddf3813b26a
By making the archive URL dynamic, we can fetch an old version of the
bootstrap files. This will make it easier to test the bootstrapper in
CI.
Differential Revision: https://phabricator.services.mozilla.com/D1698
--HG--
extra : rebase_source : 9ba582cf3c138dba433e2bb354650f14b3f16aa7
extra : amend_source : 8a515bb755187e7f0d87b90a25a99f3803ea9e0f
extra : source : 1dcd43dd2a7b04e2bb714349033a456ea5158f3e
The previously listed server wasn't working. This has likely been
broken for years (I initially authored this commit in November 2016).
Differential Revision: https://phabricator.services.mozilla.com/D1697
We're well overdue for an upgrade of the rust compiler requirements.
Now that we're building with 1.28 (albeit a beta, due to be bumped when
it's released), we can bump the requirement away from 1.24 which is now
old. 1.27 is too new, though, so settle for the older 1.26.
--HG--
extra : rebase_source : c788ef4f7da9949b81df2f0577af6f6039ea63d8
Overall, this makes the whole setup less fragile, and make it work with
LTO in more situations.
--HG--
extra : rebase_source : de968c61dc4ef337fdc28745c202334ac41763cd
We perform, on the binaries we build, a series of check, that are
implemented as half-baked make commands, invoked after linking them.
- check libstdc++ symbol versions to ensure binary compatibility with
a baseline.
- check glibc symbol versions to ensure binary compatibility with a
baseline.
- check that target binaries don't contain text relocations.
- check that libmozglue is linked before libc on android.
- on libxul, check that NSModules are laid out correctly.
- on libxul, check that there is more than one PT_LOAD segment.
Those checks happen to work where they matter, but their setup is
unreliable. For example, the checks for symbol versions are supposed to
work for libclang-plugin on cross osx builds, but in fact, don't,
because the readelf path doesn't exist, and the command doesn't fail in
that case.
So move them all to a standalone script, performing the checks more
thoroughly (especially the NSModules one, where we now also check that
they are all adjacent), and more verbosely.
--HG--
extra : rebase_source : 7072e622e95f363d4a6c3a8e272d3445d998b592
yaml.load() is unsafe and can lead to arbitrary code execution via
syntax like `!!python/object/apply:os.system`. yaml.safe_load() is
more reasonable.
Differential Revision: https://phabricator.services.mozilla.com/D1738
--HG--
extra : rebase_source : 597c07b3c1538dc27ad6f46e01cdb7f48755d0bc
extra : histedit_source : 131d570f8ac1ee047487cba54822dbf20abf6681
The bitstream is frozen and we're updating to v1.0.0. There is no longer any need
to indicate which revision we're using in the mimetype.
--HG--
extra : rebase_source : 5f5bf8649bd21610ebf04661e8f80bacbb69ca09
Some recently introduced gn-configs contain duplicate flags/includes, which
aren't handled correctly and introduce non-determinism into the gn-moz.build
generation. This patch makes the moz.build generator faithfully reproduce
duplicated flags, usually to no effect, which is unfortunate in these cases,
but a reasonable approach for the moz.build generator in general.
MozReview-Commit-ID: 6PvobD9JRwN
--HG--
extra : rebase_source : 50c9e92bb03400e6b41908de1a1bfadaced1ad91
When we do a './mach build-backend --dry-run', no files in the objdir
should be written or touched. Updating the mtime of the backend file
during dry-run causes a future './mach build' to think that the backend
is up-to-date when it is not, which causes an incorrect build.
MozReview-Commit-ID: 2NA0rcSGrvL
--HG--
extra : amend_source : f694503264dc6275568c453de59b7f13983091f7
This patch allows us to enable Python 3 tests and skip any tests that fail so that we can work on adding support for Python 3 without risking regressing any existing support. It will also eventually allow us to skip tests from running against Python 2, when we decide to drop support for it. To skip a test against Python 3, add "skip-if = python == 3" to the [DEFAULT] or test file section of a manifest file.
MozReview-Commit-ID: KYzjW6PQw2Q
--HG--
extra : rebase_source : 9e0670efe237b8953aab96e95cfa3c9ae678351c
Bug 1467041 changed the default for --enable-release such that it is
affected by the milestone. Test both possible cases for milestones.
--HG--
extra : rebase_source : 5bfeadf4cd1e8c672cf78e4922ebb91bdafecad8
90dca0906337 accidentally broke `mach artifact toolchain --from-build`
because that code is attempting to load toolchain tasks in isolation.
The new "use_fetches" transform added to toolchain tasks requires
that "fetch" tasks are already processed and their references are
available to toolchain tasks.
This commit adds a mechanism to effectively disable the "use_fetches"
transform when called by `mach artifact toolchain`. It is a hack. I
suspect future planned work around artifacts/fetches will necessitate
additional changes to the `mach artifact toolchain` code. But this
can be deferred to a later day: this commit unbusts `mach artifact
toolchain` and isn't super hacky, so it seems more reasonable than
backing out fetch tasks completely.
Differential Revision: https://phabricator.services.mozilla.com/D1588
This populates $OBJDIR/dist/host/bin as part of |mach artifact install|.
Conceptually, the mar and mbsdiff utilities should be grouped (in the
same way that the test-related binaries are grouped). However, it's
difficult to achieve that with the current structure of the code, so
this fetches mar and mbsdiff and produces $HASH-mar.processed.jar and
$HASH-mbsdiff.processed.jar files.
MozReview-Commit-ID: 3ks5xsUEKp5
--HG--
extra : rebase_source : 5fcf186decc95537cbaa90ffedb86774eab050d2
This upgrades sphinx to version 1.7.5, which contained a couple backwards
incompatible changes that needed fixing.
This also leaves sphinx-js at version 2.1 as upgrading that to 2.5 seems to
introduce an intermittent in the Doc task.
MozReview-Commit-ID: FRUTcXs5yzb
--HG--
extra : rebase_source : e874a2e9c637b7cec710203f75f4dd989a5681a1
This patch allows executing |mach python-test| against Python 3 by specifying the optional |--three| command line option. When this option is present, pipenv will be used to manage a virtual environment using Python 3 and attempt to run the tests. When it is not present, pipenv will not be used, and everything will work as it did before this patch.
My original plan was to use pipenv regardless of the target version of Python, however I encountered several issues running some of the tests against our Python packages. Once all tests have been patched to run against Python 3, then we should be able to use pipenv when running them against Python 2.
Note that this patch allows tests to run against Python 3, but there are plenty of issues preventing them from passing. With this patch in place we can start to add Python 3 support to our packages and have the tests running in CI to ensure we don't regress back to just supporting Python 2.
MozReview-Commit-ID: BuU5gZK83hL
IHG: changed taskcluster/ci/source-test/python.yml
--HG--
extra : rebase_source : ca2b15d905f7a5c895a2fd8916144841f5d205de
This patch allows executing |mach python-test| against Python 3 by specifying the optional |--three| command line option. When this option is present, pipenv will be used to manage a virtual environment using Python 3 and attempt to run the tests. When it is not present, pipenv will not be used, and everything will work as it did before this patch.
My original plan was to use pipenv regardless of the target version of Python, however I encountered several issues running some of the tests against our Python packages. Once all tests have been patched to run against Python 3, then we should be able to use pipenv when running them against Python 2.
Note that this patch allows tests to run against Python 3, but there are plenty of issues preventing them from passing. With this patch in place we can start to add Python 3 support to our packages and have the tests running in CI to ensure we don't regress back to just supporting Python 2.
MozReview-Commit-ID: BuU5gZK83hL
IHG: changed taskcluster/ci/source-test/python.yml
--HG--
extra : rebase_source : e1a64c0ffa8fe5cce71a041579601d4a72e37779
Creates a standard library for working with moz.yaml files, and adds a `mach
vendor manifest --verify` command that loads and verifies manifest schema.
The list of permitted licenses is one I derived from about:license, pending an
authoritative list from legal.
Differential Revision: https://phabricator.services.mozilla.com/D1208
The crash reporter symbol files are the easiest cross-platform way to
find static initializers. While some types of static initializers (e.g.
__attribute__(constructor) functions) don't appear there in a notable
way, the static initializers we do care the most about for tracking do
(static initializers from C++ globals). As a matter of fact, there is
only a difference of 2 compared to the currently reported count of 125
on a linux64 build, so this is a good enough approximation. And allows
us to easily track the count on Android, OSX and Windows builds, which
we currently don't do.
The tricky part is that the symbol files are in
dist/crashreporter-symbols/$lib/$fileid/$lib.sym, and $fileid is hard to
figure out. There is a `fileid` tool in testing/tools, but it is a
target binary, meaning it's not available on cross builds (OSX,
Android).
So the simplest is just to gather the data while creating the symbol
files, which unfortunately requires to go through some hoops to make it
happen for just the files we care about.
--HG--
extra : rebase_source : 458fed1ffd6f9294eefef61f10ff7a284af0d986
This is slightly ugly, but is unfortunately necessary due to do the nature of
l10n repacks. Hopefully this can go away once we move to bundling lancpack
add-ons rather than repacking in the future.
MozReview-Commit-ID: JZUblVsEbZI
--HG--
extra : rebase_source : 60c9ced2184a52f52c7f2a8820021b14b1a66abf
This one looks to be pretty straight-forward. It irritates me that
the jar.mn entry doesn't explicitly say that the result is coming from
the object directory, like
locale/browser/bookmarks.html (!bookmarks.html)
but that's for another day.
MozReview-Commit-ID: Cw8E0VJhSxv
--HG--
extra : rebase_source : a1045a5b564b0094b562729bc7234e69ec7a786d
To vendor a Python package, run ``mach vendor python [PACKAGE]``, where
``[PACKAGE]`` is one or more package names along with a version number in the
format ``pytest==3.5.1``. The package will be installed, transient dependencies
will be determined, and a ``requirements.txt`` file will be generated with the
full list of dependencies. The requirements file is then used with ``pip`` to
download and extract the source distributions of all packages into the
``third_party/python`` directory.
If you're familiar with ``Pipfile`` you can also directly modify this in the in
the top source directory and then run ``mach vendor python`` for your changes
to take effect. This allows advanced options such as specifying alternative
package indexed (see below), and
`PEP 508 specifiers <https://www.python.org/dev/peps/pep-0508/>`_.§
MozReview-Commit-ID: CRWoFamUy7V
--HG--
extra : rebase_source : e71ab6ecc74a168faa9df7cc4c2273cd8bf9e0d2
To vendor a Python package, run ``mach vendor python [PACKAGE]``, where
``[PACKAGE]`` is one or more package names along with a version number in the
format ``pytest==3.5.1``. The package will be installed, transient dependencies
will be determined, and a ``requirements.txt`` file will be generated with the
full list of dependencies. The requirements file is then used with ``pip`` to
download and extract the source distributions of all packages into the
``third_party/python`` directory.
If you're familiar with ``Pipfile`` you can also directly modify this in the in
the top source directory and then run ``mach vendor python`` for your changes
to take effect. This allows advanced options such as specifying alternative
package indexed (see below), and
`PEP 508 specifiers <https://www.python.org/dev/peps/pep-0508/>`_.§
MozReview-Commit-ID: CRWoFamUy7V
--HG--
extra : rebase_source : e71ab6ecc74a168faa9df7cc4c2273cd8bf9e0d2
This relies on the fact that providing multiple --version-script
combines them all, so we effectively create a new symbol version
that has no global symbol, but hides the std:🧵:_M_start_thread
symbols.
This version script trick happens to work with BFD ld, gold, and lld.
The downside is that when providing multiple --version-script's, ld
doesn't want any of them to have no version at all. So for the libraries
that do already have a version script (through SYMBOLS_FILE), we use a
version where there used to be none, using the library name as the
version. Practically speaking, this binds the libraries a little closer
than they used to be, kind of non-flat namespace on OSX (which is the
default there), meaning the dynamic linker will actively want to use
symbols from those libraries instead of a system library that might
happen to have the same symbol name.
--HG--
extra : rebase_source : a7f672c35609d993849385ddb874ba791b34f929
We wrap STL headers so as to ensure our inline infallible operator new
definitions are used, but for some reason, STL wrappers were disabled
in whatever code we used gyp for first, and that ended up spreading as
more things used gyp, and now with gn, whose handling cargo culted
STL disabling from the gyp code.
We remove all this disabling, because it turns out it causes no build
problem anymore.
--HG--
extra : rebase_source : 0dd5dc3dbf613a519db74072ebda04d05ea1214c
This relies on the fact that providing multiple --version-script
combines them all, so we effectively create a new symbol version
that has no global symbol, but hides as much std::* stuff as possible.
The added symbol script could use `extern "C++"` syntax and demangled
symbols but there is no guarantee the demangled symbols won't change.
Plus, it's not possible to match demangled symbols that have a return
type: they contain a space, and the only way to match that is to use
double quotes, which doesn't allow globs at the same time.
This version script trick happens to work with BFD ld, gold, and lld.
The downside is that when providing multiple --version-script's, ld
doesn't want any of them to have no version at all. So for the libraries
that do already have a version script (through SYMBOLS_FILE), we use a
version where there used to be none, using the library name as the
version. Practically speaking, this binds the libraries a little closer
than they used to be, kind of non-flat namespace on OSX (which is the
default there), meaning the dynamic linker will actively want to use
symbols from those libraries instead of a system library that might
happen to have the same symbol name.
--HG--
extra : rebase_source : 78adb64b90e75ebad203b8a647b305c9d7198d16
Reduce the amount of text so that the options are more likely to be
visible and people are more likely to read it.
--HG--
extra : rebase_source : 95eacc8b6b09a82dfb1bec0e837bc70057c5cef1
This also removes the workerbootstrap test extension, which is no longer used,
and contains the last references to the Worker and ChromeWorker bootstrap
globals.
MozReview-Commit-ID: 8YWReXMqX5W
--HG--
extra : rebase_source : b0aa59b2b5e6a08f4be803e828bd507f894e4a19
AMO needs to enter the application version for every Firefox release at this time, and in doing so they don't usually enter .sec versions, as these versions are exposed to addon devs in UX where they can specify outside of the xpi what versions of Firefox they are compatible with.
Language packs however set min version to things like 59.0.2 which AMO doesn't know about.
AMO will also fail to validate an .xpi with an unknown min version.
This code logic is slightly compounded by the fact that SeaMonkey uses these codepaths as well, so we need to account for it here.
Differential Revision: https://phabricator.services.mozilla.com/D1112
--HG--
extra : rebase_source : 5bf74d235bdf651714984b7fbe0e79d2d3f61b6e
extra : histedit_source : 91a49a4c0452f217d4e3de534bfdd817dd2350cc
The previous version was removed from Gentoo's portage repository making it
impossible to bootstrap correctly.
MozReview-Commit-ID: HTao6D3g61L
--HG--
extra : rebase_source : 57be7946b105289e662dc2f687bb1b2b9056a3f2
Since MozbuildObject.from_environment() reads from mozinfo.json, tup
picks up that file as a dependency for anything that imports buildconfig
(eg: all generated files). Using FileAvoidWrite when creating
mozinfo.json will help avoid unnecessary work after re-running configure
in the tup backend.
MozReview-Commit-ID: EEOPQYJA1MV
--HG--
extra : rebase_source : 136a0579090776dc55ea5cee870574f13cc27c58
The build system knows at build-backend time where to find each IDL
file; making xpidl-process.py rediscover this by requiring
xpidl-process.py to search through directories to find input IDL files
is silly. To rememdy this, we're going to modify things so full paths
are passed into the script. Those paths can then be used directly, with
no searching.
The previous patch required us to pass a single -I argument pointing at
$(DIST)/idl so IDL include statements would work correctly. This patch
lifts that limitation and explicitly points xpidl-process.py at the
locations of all the IDL source directories to search for included IDL
files. Invocations of xpidl-process.py no longer depend on IDL files
being copied to the objdir.
Building on the last patch, we can change the build process to pass in
the directories where the input IDL files can be found. It is
convenient to pass in just the relative source directory paths, to
encourage people to not look in the object directory and to make the
command lines slightly shorter.
xpidl-process.py still assumes that included IDL files can be found by
looking in a single directory. We add a single -I argument to the
invocation of xpidl-process.py to accommodate this short-sightedness.
The current IDL build setup assumes that all IDL files can be found in a
single directory. This setup requires that all IDL files be copied to a
single directory, which is suboptimal in terms of disk I/O and also
complicates things like generating IDL files at build time.
As a first step in moving away from this state of affairs,
xpidl-process.py needs to be taught that the input IDL files could
potentially be found in multiple directories. The current setup can
just specify $(DIST)/idl as the lone directory to examine. Future
patches will change this to examine multiple directories.
This method is only called in one place, and it doesn't pass
allow_existing. Whatever ugly thing this keyword was working around
doesn't exist anymore, so let's get rid of it.
We no longer want to update mtimes of FileAvoidWrites so that downstream
rules aren't triggered if the files aren't changed. Since the .stub file
target of GENERATED_FILES are always touched, make won't continually
rebuild them.
MozReview-Commit-ID: GxrFgCJTYk
--HG--
extra : rebase_source : f4412af1dc29142b76f7695627ba3354baf84edd
The make backend was treating the first output of a GENERATED_FILES rule
specially, since it was the target of the rule containing the script
invocation. We want the outputs of GENERATED_FILES rules to be
FileAvoidWrite so that we avoid triggering downstream rules if the
outputs are unchanged, but if the target of the script invocation is
FileAvoidWrite, then make may continually re-run the script during a
no-op build.
The solution here is to use a stub file as the target of the script
invocation which will always be touched when the script runs. Since
nothing else in the build depends on the stub, we don't need to
FileAvoidWrite it. All actual outputs of the script can be
FileAvoidWrite, and make can properly avoid work for files that haven't
changed.
MozReview-Commit-ID: 3GejZw2tpqu
--HG--
extra : rebase_source : 2b9be82f893e89a4c2f254f05b1e8b9a0f9c631b
Some GENERATED_FILES entries don't have .scripts associated with them
(notably midl on Windows builds). In this case, we don't want to
generate dependencies automatically since they will be handled by the
Makefiles.
MozReview-Commit-ID: AXmN2Unk9AY
--HG--
extra : rebase_source : 1f06672add87c46ae199189fcae27b721e008f9e
Some commands produce a large number of output files, such as
make-system-wrappers.py, which has over 1000 outputs. The GeneratedFile
handler in the tup backend displayed all the outputs, which makes the
build output unreadable, and breaks 'tup graph'. This patch displays
only the first 3 outputs and truncates the rest.
MozReview-Commit-ID: 5AnrmMe0Nyx
--HG--
extra : rebase_source : 1a6766be36aef4603c1e5333cfc13af006369966
Files returned from version control (i.e via --outgoing or --workdir), are currently joined to
cwd. This will cause failures if |mach lint| is run from anywhere other than topsrcdir.
However we *do* want to join manually specified paths to cwd so things like:
cd devtools && mach lint client
continue to work. This patch makes sure we join the proper kind of path to the proper place.
MozReview-Commit-ID: EQmRhAr3Oog
--HG--
extra : rebase_source : 2629cc27f79059e44369d46d4f8278f83923582c
This gets raptor to use the newly created "perf" profile that talos
also uses. There is a single pref that raptor sets that we can't set
in talos. To that end, this also creates a "raptor" specific profile.
This means to set a pref in talos and raptor, edit:
testing/profiles/perf/user.js
To set a pref in raptor only, edit:
testing/profiles/raptor/user.js
The performance of extensions can now be tested by dropping the
extension into:
testing/profiles/perf/extensions
MozReview-Commit-ID: LEJeytmmiFF
--HG--
extra : rebase_source : 0d2a6b18868f8cc6ff198ef868ad0324b57b1dc2
This moves all of the global prefs that were previously defined
in testing/talos/talos/config.py, into a new "perf" profile under
testing/profiles/perf/user.js.
This perf profile will be shared with raptor, so changes to one
framework will result in changes to the other.
MozReview-Commit-ID: JRxZEDlPu6b
--HG--
extra : rebase_source : 38f61eb6f9dd3e8dd9e0425ffe32dbdf845fcf65
