Currently line linters (linters that open a file and process it line by line,
by applying a regex for example), don't handle directories. If a directory is
passed in, it will try to 'open' it, which fails. Directories can get hit if
the linter has a directory in its include directive or if the user passes in
--no-filter.
This patch modifies LineLinters so that if a directory is detected, we search
for all relevant files under that directory. If 'extensions' is used, we'll
look for only files with appropriate extensions. Otherwise we assume the
linter wants every file.
MozReview-Commit-ID: D9lzTNuQTob
--HG--
extra : rebase_source : 0b952c06eae28b67b687813ff7e75b231b2dd4d3
The MozillaBuildBootstrapper specific rust install code in not needed as
mozbase already includes genertic code to achieve the same outcome. The
mozilla-build specific code also leads to issues where it tries to add already
existing targets and fails the bootstrap. This changeset removes the
mozilla-build specific step.
MozReview-Commit-ID: G0BqKZrF40A
--HG--
extra : rebase_source : 60e9638afff744c937a9665d6fd5830187835ea4
The code for obtaining a BuildReader for evaluating moz.build files
is generic and non-trivial. We already had a custom implementation
for `mach file-info` that implemented support for Mercurial
integration. Bug 1383880 will introduce a second consumer.
So this commit factors out the "obtain a BuildReader" logic into
a reusable function on our base MozbuildObject class. This makes
it easily available to various parts of the build system and mach
commands.
As part of the change, we detect when ``.`` is being used as the
revision and verify the working directory is clean. This behavior
can be disabled via argument if unwanted. But it's useful by default
to ensure consumers aren't expecting to read uncommitted changes.
MozReview-Commit-ID: LeYFqAb3HAe
--HG--
extra : rebase_source : d5ed4e4f5570a58a68853188de2225cd4e64ab3a
This is generally useful functionality to have. A consume will be
introduced in an upcoming commit.
MozReview-Commit-ID: 4arTMfJSiEC
--HG--
extra : rebase_source : 4bcf70f58b57b79b8dcb7a6eed633e1c7e42aca3
It seems reasonable to expose this outside of the BuildReader.
MozReview-Commit-ID: 4paDbYl9dEd
--HG--
extra : rebase_source : 86bb559500952a40fc9afbf958be5706dd9f858e
This also passes the '--noreplace' option to all the emerge invocations thus
preventing already installed packages from being rebuilt from scratch.
MozReview-Commit-ID: 4JBuptmgS3Y
--HG--
extra : rebase_source : e581607d4a2e997e7d79c7c4496d13b8e9b10e50
To construct an empty set, we need to use the `set()` notation. In order
to do that, we need to expose `set` to the moz.build sandbox.
MozReview-Commit-ID: DMyKnF0FEx2
--HG--
extra : rebase_source : 5cfe8080ec333a1eca70cd3edba2aaaff6406820
And convert consumers to context managers because hglib requires that.
MozReview-Commit-ID: Ckf1yBYeUlm
--HG--
extra : rebase_source : 985220032bced1a7077fd9b04ca8ad6de822c887
Because hglib spawns a persistent process, we introduce a context
manager for Repository. It no-ops by default. On HgRepository it
controls the lifetime of the persistent hg process.
A helper method for running an hg command via hglib has been added.
We can't transition existing methods to hglib because hglib
requires a context manager, which no consumer is using yet.
MozReview-Commit-ID: 8z0fcGFeAm5
--HG--
extra : rebase_source : 1cbfe44a5aafe838c25bcde91cc1deaf5e7367de
This ensures we use forward slashes, even if Mercurial emits
backslashes (which it can do on Windows).
MozReview-Commit-ID: 2dnWAEvytwn
--HG--
extra : rebase_source : 9ebb454bc9ad11b3eba334e412685e529573a0a1
These tests weren't running in automation because hglib wasn't
available. An upcoming commit will vendor hglib. This exposed that the
tests can fail if ui.username isn't set.
In this commit, we introduce a helper function to obtain an hglib
client with ui.username set. We also convert tests to use the context
manager form of the client so resources are cleaned up immediately
without relying on refcounting or garbage collection.
MozReview-Commit-ID: HRSBDlYgqpC
--HG--
extra : rebase_source : 83deb56a0c2efefa883d6df104cd67194a811907
If configure has defined VCS binaries, we should use those.
MozReview-Commit-ID: DVnsSaJC8eN
--HG--
extra : rebase_source : 6a940b6b4f7986eece350ab692242701dfbf41dd
Some callers already have a build config object. Let's not
make them call a function that imports buildconfig.
MozReview-Commit-ID: J22HhyVma9y
--HG--
extra : rebase_source : fe5151925aee76508d26a9368e7a2ebec7e23ddb
This allows .flake8 files to override one another, and fixes a pretty bad known
bug with our flake8 implementation. For example, say we have a .flake8 file at:
/foo/.flake8
Before this patch, if we ran |mach lint foo/bar|, the configuration defined in
that .flake8 file wouldn't get picked up. It would only work if running the
specific directory that contains it, e.g |mach lint foo|.
This change additionally allows multiple .flake8 files to be used. So if
there's one defined at both:
/.flake8
/foo/.flake8
Then running |mach lint foo/bar| will first apply the root .flake8, then the
one under /foo (overriding earlier configuration).
This bug still doesn't make flake8 configuration perfect though. Any directory
containing a .flake8 file still needs to be explicitly listed in the "include"
section of /tools/lint/flake8.yml. Otherwise in the example above, if running
|mach lint /|, it wouldn't be able to find /foo/.flake8. This is a hard problem
and is likely best solved by fixing flake8's upstream configuration handling.
Unfortunately this means we still can't switch from a whitelist to a blacklist.
MozReview-Commit-ID: 3DZAi1QHYYo
--HG--
extra : rebase_source : 51298c5847f6c2792581d9b312c87b70fa716ee1
The install manifest with the .track files uses os.path.exists() to
determine if a previously tracked file is no longer installed and needs
to be removed from the system. However, exists() returns False for
broken symlinks, so as far as the manifest is concerned, there is no
file in the filesystem that needs to be removed. We should use lexists()
so we know that the broken symlink still exists in the system so
that it can be removed when the install manifest is processed.
MozReview-Commit-ID: 6v7CYOKzjGs
--HG--
extra : rebase_source : 8aeeef59e644613f34c8458bd30a83d8299585ea
There's a persistent test failure in automation that seems to have to do
with shutting down the `multiprocessing.Manager` that's used to get a
`Queue` to submit jobs to worker processes. After toying around with fixing
that I decided it would be simpler to just use concurrent.futures here,
since we already have it in-tree and it fits the use case here better
than using raw multiprocessing.
MozReview-Commit-ID: 8DdSvs2qp0q
--HG--
extra : rebase_source : 0b1cbb96bd3016778e4974a311722a8882f87216
Before, the "relevant" moz.build files were based strictly on filename.
In reality, there are some moz.build files that we wish to ignore.
The previous commit introduced a Finder that knows how to ignore
moz.build files that should be ignored. In this commit, we hook
it up to our low-level function for determining the set of relevant
moz.build files for a path.
The main benefit of this change is that paths in the moz.build
test directory no longer say test moz.build files are relevant.
Previously, we would return these test moz.build files. Some of
these are invalid and would cause execution to fail. So, commands
like `mach file-info` will no longer attempt to evaluate moz.build
files they weren't supposed to and will stop erroring.
Another benefit is that the function returns faster. When passing
in every file in the repo (>230,000 files), execution time dropped
from ~8.03s to ~6.16s. This is probably due to fewer path operations.
MozReview-Commit-ID: J2d25ZtxjFt
--HG--
extra : rebase_source : 26105de85c49e061a720b54019b4f16b6425748e
Not every moz.build file in the repo is a normal moz.build file.
Some moz.build files are used for testing moz.build files. Others
may exist in directories that should be ignored.
all_mozbuild_path() already knew how to filter out moz.build files
that should be ignored. Let's extract the Finder for doing this
into an instance attribute so it can be used elsewhere.
MozReview-Commit-ID: 9PaZQAbjIZO
--HG--
extra : rebase_source : 5bfd27b5a9ab6b24b9e3aa3c8cc286d64c3ebd3c
This passes `python3 -mcompileall`. Changes:
* use `0o` prefix for octal literals
* print as a function
* except .. as
* use six.reraise to replace a multi-argument raise statement
* use six.string_types and six.moves.configparser
* remove uses of `L` suffix for long integers
MozReview-Commit-ID: KLaYRNHGpay
--HG--
extra : rebase_source : 6ca1b5447cd28eff8d9f2805add6a0f07e8b4c63
Bug 1355661 added support for brotli streams in "jar" files handled by
Gecko, and bug 1355671 made us build the `bro` command line utility that
allows to compress and decompress brotli streams.
This change uses the `bro` command line utility in the packager so that
it can create and handle "jar" files using brotli streams.
However, the `bro` command line utility is not available to l10n
repacks. As, at the moment, we're only hoping that the outcome of using
brotli will be good, we avoid doing all the work to make those work and
just hook things enough to enable brotli, while ensuring l10n repacks
don't break. This involves forcing some files to be deflated, and to
disable some optimizations from the packager.
Things will need to be figured out more properly if the experiment
proves brotli to be worthwhile.
--HG--
extra : rebase_source : a2e0cff67dcaed465fd441ed5d2a7de94b6351c5
Change webextensions experiments test to use the shimmed certficiate DB
instead of the extensions.legacy.enabled pref.
In builds that don't honor the extensions.legacy.enabled pref, disable
test_legacy.js since that tests that flipping that preference works properly.
Finally, remove a now doubly-obsolete test of plugins embedded in xpis.
MozReview-Commit-ID: JiRdgCXyjKR
--HG--
extra : rebase_source : f0c7672b0755993bd20f9fc84e242eb76cb949ef
This preserves ./mach settings' --list option. If --list is passed in, we call splitlines()
on the description and print only the first line. The full multi-line description will be
printed otherwise.
This also displays the type and/or choices of the option as appropriate.
MozReview-Commit-ID: 7UMsN9qslWt
--HG--
extra : rebase_source : 4bc9554d8652e02e290c6a190634f1a72cdbadc3
