This pulls a new nom version, which is slightly unfortunate, but I do want some
of the fixes upstream, and it's build-only, so I think it's not a huge deal.
Differential Revision: https://phabricator.services.mozilla.com/D9362
Now that we're outputing a stub file for each build script we can index these
commands by their output to de-duplicate them in the tup backend.
Differential Revision: https://phabricator.services.mozilla.com/D9042
--HG--
extra : moz-landing-system : lando
The desired outcome of this change is that we'll set
-Wl,--version-script based on linker kind and not on the output of
$LINKER -v.
This is a cheap way to address a simple problem that has a complicated
ideal solution. The underlying issue is that in some situations, when
targeting Android, a macOS system ld is interrogated to determine if
a cross-compiling linker "is GNU ld" and a particular linker feature
is set in that situation. The macOS system ld doesn't pass the "is
GNU ld" test, and the linker feature isn't set; that causes link
failures, even though the actual linker has nothing to do with the
system ld.
The ideal solution is to test for linker capabilities dynamically. We
do a lot of that in old-configure.in, and we don't do any of that in
toolchain.configure. Rather than start testing in
toolchain.configure, we hard-code: a cheap solution to the immediate
problem.
MinGW suffers somewhat from the opposite problem: the linker "is GNU
ld" (compatible), but the linker checks don't happen at all. We hard-code
for MinGW based on the C compiler instead.
Differential Revision: https://phabricator.services.mozilla.com/D8471
--HG--
extra : moz-landing-system : lando
Bug 1438688 made it so that XPT information is compiled directly into
the binary instead of being shipped separately in interface
files. This means that manifests are no longer necessary for JS
components, which means the manifest check in emitter.py can be
removed.
That check is the only use of NO_JS_MANIFEST, so that can in turn be
removed entirely.
Differential Revision: https://phabricator.services.mozilla.com/D8885
--HG--
extra : moz-landing-system : lando
This addresses a related issue along the way: a build that results in running
configure would not update the value of self.config_environment (and therefore
self.substs) as seen from the build driver, so out of date values would have
been used.
The changes to Makefile.in and client.mk made exploit the assumption that by
he time anything in the Make build is running, config.status is up to date.
Users running "make" without the benefit of "mach" will need to manually run
configure when necessary in order to take this into account.
Differential Revision: https://phabricator.services.mozilla.com/D8473
--HG--
extra : moz-landing-system : lando
We were currently adding exclude paths to the "discard" set if the path contains
the include, but we weren't adding them if the include contains the path.
Depends on D5863
Differential Revision: https://phabricator.services.mozilla.com/D8845
--HG--
extra : moz-landing-system : lando
Right now there are excludes defined in the linter definition (via the .yml
files), as well as excludes defined in lintargs (via the mach_commands.py).
This is a minor simplification that extends each linter definition's local
excludes with the global ones right off the bat. This just makes it a bit
easier to keep track of.
Depends on D5863
Differential Revision: https://phabricator.services.mozilla.com/D8844
--HG--
extra : moz-landing-system : lando
Currently pathutils.filterpaths computes some extra paths that the underlying linter
should try to exclude if it can. It sets these on lintargs['exclude'], and relies on
the fact that it was passed by reference to work.
However, it seems like pytest does some magic under the hood that prevents this value
from being propagated back. It will also fail if 'filterpaths' was invoked in a
subprocess.
It's much simpler and cleaner to pass this value back directly, rather than rely on a
reference.
Differential Revision: https://phabricator.services.mozilla.com/D8850
--HG--
extra : moz-landing-system : lando
This fixes a latent bug that is currently not being hit (by sheer luck). Basically
the 'ignore' argument of a FileFinder object needs all paths to be relative to the
base. As luck would have it, most of the time it worked out that way if you were
running |mach lint| from the root of the repo.
However there are edge cases where this will cause an 'exclude' directive to get
missed. Plus this bug is about to be exposed 100% of the time in the next commit :).
Depends on D8842
Differential Revision: https://phabricator.services.mozilla.com/D8843
--HG--
extra : moz-landing-system : lando
I missed an edge case where the computed base itself was specified in the
paths input.
Differential Revision: https://phabricator.services.mozilla.com/D8842
--HG--
extra : moz-landing-system : lando
This patch also changes how pdbs for the ASAN job are copied:
we relax restrictions so that pdbs if present) are always copied out
and add an environment variable MOZ_COPY_PDBS to indicate when we
want to produce pdbs for copying.
The desired outcome of this change is that we'll set
`-Wl,--version-script` based on linker kind and not on the output of
`$LINKER -v`.
This is a cheap way to address a simple problem that has a complicated
ideal solution. The underlying issue is that in some situations, when
targeting Android, a macOS system `ld` is interrogated to determine if
a cross-compiling linker "is GNU ld" and a particular linker feature
is set in that situation. The macOS system `ld` doesn't pass the "is
GNU ld" test, and the linker feature isn't set; that causes link
failures, even though the actual linker has nothing to do with the
system `ld`.
The ideal solution is to test for linker capabilities dynamically. We
do a lot of that in old-configure.in, and we don't do any of that in
toolchain.configure. Rather than start testing in
toolchain.configure, we hard-code: a cheap solution to the immediate
problem.
Differential Revision: https://phabricator.services.mozilla.com/D8471
--HG--
extra : moz-landing-system : lando
While we do have some uses of @depends-function comparison in some
templaces, related to host/target, we ought to be using `is` comparisons
rather than `==` anyways, so we switch those, and prevent other kinds of
comparisons being used at all.
This unveils the one noted in
https://phabricator.services.mozilla.com/D7713?id=21357#inline-30414
(and surprisingly only that one), that we remove entirely since it was
doing nothing in practice. Bug 1492305 will have to add it back in a
proper form.
Differential Revision: https://phabricator.services.mozilla.com/D8501
--HG--
extra : moz-landing-system : lando
Often we specify globs in our exclude patterns, e.g:
exclude:
- **/node_modules
- obj*
However, these globs get expanded out to *every* file that matches them. This
can sometimes be thousands or even tens of thousands of files.
We then pass these paths on to the underlying linters and tell them to
exclude them all. This causes a lot of overhead and slows down performance.
This commit implements a "collapse" function. Given a set of paths, it'll
collapse them into the smallest set of parent directories that contain the
original set, and that don't contain any extra files.
For example, given a directory structure like this:
a
-- foo.txt
-- b
-- bar.txt
-- baz.txt
-- c
-- ham.txt
-- d
-- spam.txt
Then the following will happen:
>>> collapse(['a/foo.txt', 'a/b/bar.txt', 'a/c/ham.txt', 'a/c/d/spam.txt'])
['a/foo.txt', 'b/bar.txt', 'c']
Since all files under directory 'c' are specified by the original set (both
'c/ham.txt' and 'c/d/spam.txt'), we can collapse it down to just 'c'. However
not all files under 'b' are specified (we're missing 'a/b/baz.txt'), so we
can't collapse 'b' (and by extension also can't collapse 'a').
If we had included 'a/b/baz.txt':
>>> collapse(['a/foo.txt', 'a/b/bar.txt', 'a/b/baz.txt', 'a/c/ham.txt', 'a/c/d/spam.txt'])
['a']
In both cases, the smallest set of paths that contains the original set (and
only the original set) is computed.
The collapse function has a little bit of overhead but it's not too bad.
For example collapsing all files matched by '**/node_modules' takes ~0.015s.
Collapsing two full objdirs, takes ~0.6s. But a follow up commit is planned to
make sure we stop using 'obj*' to reduce that overhead.
Depends on D7738
Differential Revision: https://phabricator.services.mozilla.com/D7739
--HG--
extra : moz-landing-system : lando
If scandir is already present on the system the attempt to import the
c helper library will currently find the c helper from the system
install which may well be an outdated verion, so causing mach to
break. To solve this this patch does two things:
* Stops importing scandir in files that are run unconditionally when
invoking mach. This is generally considered good for performance
reasons.
* Installs the vendored scandir into the virtualenv for `mach lint`
rather than trying to import it directly from the source tree and so
not getting the c helper library.
Differential Revision: https://phabricator.services.mozilla.com/D8379
--HG--
extra : moz-landing-system : lando
At least one Python package that got vendored via `mach vendor python`
contains a symlink, which hg.mozilla.org rejects. This change makes it so
symlinks get replaced with the contents of the file they point to.
Differential Revision: https://phabricator.services.mozilla.com/D4135
--HG--
extra : moz-landing-system : lando
This patch adds two new actions to virtualenv_packages.txt processing:
windows and !windows. The former processes the rest of the action only on
Windows, and the latter processes it only on non-Windows.
These new features are used in virtualenv_packages.txt to use the
path to the unpacked Windows psutil wheel when on Windows, and build psutil
from source and use that path on other platforms.
This fixes the long-standing problem of not having psutil available on most
Windows systems (since they don't have the right set of Visual C++ build tools).
Differential Revision: https://phabricator.services.mozilla.com/D3437
--HG--
extra : moz-landing-system : lando
This option is very single-purpose: it's intended to let us vendor an unpacked
wheel for psutil on Windows. To that end the mach command will error if you
try to use it for anything but vendoring a single package. The mach command
will vendor source packages as it currently does, and then run `pip download`
again with some hardcoded parameters to fetch the right wheel for Python 2.7
on win64 and unpack it to a `package-platform` directory under
`third_party/python`.
I don't expect this to be used for anything but psutil, but it should make life
simpler for anyone that wants to update our vendored copy of psutil in the
future.
Differential Revision: https://phabricator.services.mozilla.com/D3435
--HG--
extra : moz-landing-system : lando
Often we specify globs in our exclude patterns, e.g:
exclude:
- **/node_modules
- obj*
However, these globs get expanded out to *every* file that matches them. This
can sometimes be thousands or even tens of thousands of files.
We then pass these paths on to the underlying linters and tell them to
exclude them all. This causes a lot of overhead and slows down performance.
This commit implements a "collapse" function. Given a set of paths, it'll
collapse them into the smallest set of parent directories that contain the
original set, and that don't contain any extra files.
For example, given a directory structure like this:
a
-- foo.txt
-- b
-- bar.txt
-- baz.txt
-- c
-- ham.txt
-- d
-- spam.txt
Then the following will happen:
>>> collapse(['a/foo.txt', 'a/b/bar.txt', 'a/c/ham.txt', 'a/c/d/spam.txt'])
['a/foo.txt', 'b/bar.txt', 'c']
Since all files under directory 'c' are specified by the original set (both
'c/ham.txt' and 'c/d/spam.txt'), we can collapse it down to just 'c'. However
not all files under 'b' are specified (we're missing 'a/b/baz.txt'), so we
can't collapse 'b' (and by extension also can't collapse 'a').
If we had included 'a/b/baz.txt':
>>> collapse(['a/foo.txt', 'a/b/bar.txt', 'a/b/baz.txt', 'a/c/ham.txt', 'a/c/d/spam.txt'])
['a']
In both cases, the smallest set of paths that contains the original set (and
only the original set) is computed.
The collapse function has a little bit of overhead but it's not too bad.
For example collapsing all files matched by '**/node_modules' takes ~0.015s.
Collapsing two full objdirs, takes ~0.6s. But a follow up commit is planned to
make sure we stop using 'obj*' to reduce that overhead.
Depends on D7738
Differential Revision: https://phabricator.services.mozilla.com/D7739
--HG--
extra : moz-landing-system : lando
