Currently line linters (linters that open a file and process it line by line,
by applying a regex for example), don't handle directories. If a directory is
passed in, it will try to 'open' it, which fails. Directories can get hit if
the linter has a directory in its include directive or if the user passes in
--no-filter.
This patch modifies LineLinters so that if a directory is detected, we search
for all relevant files under that directory. If 'extensions' is used, we'll
look for only files with appropriate extensions. Otherwise we assume the
linter wants every file.
MozReview-Commit-ID: D9lzTNuQTob
--HG--
extra : rebase_source : 0b952c06eae28b67b687813ff7e75b231b2dd4d3
The MozillaBuildBootstrapper specific rust install code in not needed as
mozbase already includes genertic code to achieve the same outcome. The
mozilla-build specific code also leads to issues where it tries to add already
existing targets and fails the bootstrap. This changeset removes the
mozilla-build specific step.
MozReview-Commit-ID: G0BqKZrF40A
--HG--
extra : rebase_source : 60e9638afff744c937a9665d6fd5830187835ea4
The code for obtaining a BuildReader for evaluating moz.build files
is generic and non-trivial. We already had a custom implementation
for `mach file-info` that implemented support for Mercurial
integration. Bug 1383880 will introduce a second consumer.
So this commit factors out the "obtain a BuildReader" logic into
a reusable function on our base MozbuildObject class. This makes
it easily available to various parts of the build system and mach
commands.
As part of the change, we detect when ``.`` is being used as the
revision and verify the working directory is clean. This behavior
can be disabled via argument if unwanted. But it's useful by default
to ensure consumers aren't expecting to read uncommitted changes.
MozReview-Commit-ID: LeYFqAb3HAe
--HG--
extra : rebase_source : d5ed4e4f5570a58a68853188de2225cd4e64ab3a
This is generally useful functionality to have. A consume will be
introduced in an upcoming commit.
MozReview-Commit-ID: 4arTMfJSiEC
--HG--
extra : rebase_source : 4bcf70f58b57b79b8dcb7a6eed633e1c7e42aca3
It seems reasonable to expose this outside of the BuildReader.
MozReview-Commit-ID: 4paDbYl9dEd
--HG--
extra : rebase_source : 86bb559500952a40fc9afbf958be5706dd9f858e
This also passes the '--noreplace' option to all the emerge invocations thus
preventing already installed packages from being rebuilt from scratch.
MozReview-Commit-ID: 4JBuptmgS3Y
--HG--
extra : rebase_source : e581607d4a2e997e7d79c7c4496d13b8e9b10e50
To construct an empty set, we need to use the `set()` notation. In order
to do that, we need to expose `set` to the moz.build sandbox.
MozReview-Commit-ID: DMyKnF0FEx2
--HG--
extra : rebase_source : 5cfe8080ec333a1eca70cd3edba2aaaff6406820
And convert consumers to context managers because hglib requires that.
MozReview-Commit-ID: Ckf1yBYeUlm
--HG--
extra : rebase_source : 985220032bced1a7077fd9b04ca8ad6de822c887
Because hglib spawns a persistent process, we introduce a context
manager for Repository. It no-ops by default. On HgRepository it
controls the lifetime of the persistent hg process.
A helper method for running an hg command via hglib has been added.
We can't transition existing methods to hglib because hglib
requires a context manager, which no consumer is using yet.
MozReview-Commit-ID: 8z0fcGFeAm5
--HG--
extra : rebase_source : 1cbfe44a5aafe838c25bcde91cc1deaf5e7367de
This ensures we use forward slashes, even if Mercurial emits
backslashes (which it can do on Windows).
MozReview-Commit-ID: 2dnWAEvytwn
--HG--
extra : rebase_source : 9ebb454bc9ad11b3eba334e412685e529573a0a1
These tests weren't running in automation because hglib wasn't
available. An upcoming commit will vendor hglib. This exposed that the
tests can fail if ui.username isn't set.
In this commit, we introduce a helper function to obtain an hglib
client with ui.username set. We also convert tests to use the context
manager form of the client so resources are cleaned up immediately
without relying on refcounting or garbage collection.
MozReview-Commit-ID: HRSBDlYgqpC
--HG--
extra : rebase_source : 83deb56a0c2efefa883d6df104cd67194a811907
If configure has defined VCS binaries, we should use those.
MozReview-Commit-ID: DVnsSaJC8eN
--HG--
extra : rebase_source : 6a940b6b4f7986eece350ab692242701dfbf41dd
Some callers already have a build config object. Let's not
make them call a function that imports buildconfig.
MozReview-Commit-ID: J22HhyVma9y
--HG--
extra : rebase_source : fe5151925aee76508d26a9368e7a2ebec7e23ddb
This allows .flake8 files to override one another, and fixes a pretty bad known
bug with our flake8 implementation. For example, say we have a .flake8 file at:
/foo/.flake8
Before this patch, if we ran |mach lint foo/bar|, the configuration defined in
that .flake8 file wouldn't get picked up. It would only work if running the
specific directory that contains it, e.g |mach lint foo|.
This change additionally allows multiple .flake8 files to be used. So if
there's one defined at both:
/.flake8
/foo/.flake8
Then running |mach lint foo/bar| will first apply the root .flake8, then the
one under /foo (overriding earlier configuration).
This bug still doesn't make flake8 configuration perfect though. Any directory
containing a .flake8 file still needs to be explicitly listed in the "include"
section of /tools/lint/flake8.yml. Otherwise in the example above, if running
|mach lint /|, it wouldn't be able to find /foo/.flake8. This is a hard problem
and is likely best solved by fixing flake8's upstream configuration handling.
Unfortunately this means we still can't switch from a whitelist to a blacklist.
MozReview-Commit-ID: 3DZAi1QHYYo
--HG--
extra : rebase_source : 51298c5847f6c2792581d9b312c87b70fa716ee1
The install manifest with the .track files uses os.path.exists() to
determine if a previously tracked file is no longer installed and needs
to be removed from the system. However, exists() returns False for
broken symlinks, so as far as the manifest is concerned, there is no
file in the filesystem that needs to be removed. We should use lexists()
so we know that the broken symlink still exists in the system so
that it can be removed when the install manifest is processed.
MozReview-Commit-ID: 6v7CYOKzjGs
--HG--
extra : rebase_source : 8aeeef59e644613f34c8458bd30a83d8299585ea
There's a persistent test failure in automation that seems to have to do
with shutting down the `multiprocessing.Manager` that's used to get a
`Queue` to submit jobs to worker processes. After toying around with fixing
that I decided it would be simpler to just use concurrent.futures here,
since we already have it in-tree and it fits the use case here better
than using raw multiprocessing.
MozReview-Commit-ID: 8DdSvs2qp0q
--HG--
extra : rebase_source : 0b1cbb96bd3016778e4974a311722a8882f87216
Before, the "relevant" moz.build files were based strictly on filename.
In reality, there are some moz.build files that we wish to ignore.
The previous commit introduced a Finder that knows how to ignore
moz.build files that should be ignored. In this commit, we hook
it up to our low-level function for determining the set of relevant
moz.build files for a path.
The main benefit of this change is that paths in the moz.build
test directory no longer say test moz.build files are relevant.
Previously, we would return these test moz.build files. Some of
these are invalid and would cause execution to fail. So, commands
like `mach file-info` will no longer attempt to evaluate moz.build
files they weren't supposed to and will stop erroring.
Another benefit is that the function returns faster. When passing
in every file in the repo (>230,000 files), execution time dropped
from ~8.03s to ~6.16s. This is probably due to fewer path operations.
MozReview-Commit-ID: J2d25ZtxjFt
--HG--
extra : rebase_source : 26105de85c49e061a720b54019b4f16b6425748e
Not every moz.build file in the repo is a normal moz.build file.
Some moz.build files are used for testing moz.build files. Others
may exist in directories that should be ignored.
all_mozbuild_path() already knew how to filter out moz.build files
that should be ignored. Let's extract the Finder for doing this
into an instance attribute so it can be used elsewhere.
MozReview-Commit-ID: 9PaZQAbjIZO
--HG--
extra : rebase_source : 5bfd27b5a9ab6b24b9e3aa3c8cc286d64c3ebd3c
This passes `python3 -mcompileall`. Changes:
* use `0o` prefix for octal literals
* print as a function
* except .. as
* use six.reraise to replace a multi-argument raise statement
* use six.string_types and six.moves.configparser
* remove uses of `L` suffix for long integers
MozReview-Commit-ID: KLaYRNHGpay
--HG--
extra : rebase_source : 6ca1b5447cd28eff8d9f2805add6a0f07e8b4c63
Bug 1355661 added support for brotli streams in "jar" files handled by
Gecko, and bug 1355671 made us build the `bro` command line utility that
allows to compress and decompress brotli streams.
This change uses the `bro` command line utility in the packager so that
it can create and handle "jar" files using brotli streams.
However, the `bro` command line utility is not available to l10n
repacks. As, at the moment, we're only hoping that the outcome of using
brotli will be good, we avoid doing all the work to make those work and
just hook things enough to enable brotli, while ensuring l10n repacks
don't break. This involves forcing some files to be deflated, and to
disable some optimizations from the packager.
Things will need to be figured out more properly if the experiment
proves brotli to be worthwhile.
--HG--
extra : rebase_source : a2e0cff67dcaed465fd441ed5d2a7de94b6351c5
Change webextensions experiments test to use the shimmed certficiate DB
instead of the extensions.legacy.enabled pref.
In builds that don't honor the extensions.legacy.enabled pref, disable
test_legacy.js since that tests that flipping that preference works properly.
Finally, remove a now doubly-obsolete test of plugins embedded in xpis.
MozReview-Commit-ID: JiRdgCXyjKR
--HG--
extra : rebase_source : f0c7672b0755993bd20f9fc84e242eb76cb949ef
This preserves ./mach settings' --list option. If --list is passed in, we call splitlines()
on the description and print only the first line. The full multi-line description will be
printed otherwise.
This also displays the type and/or choices of the option as appropriate.
MozReview-Commit-ID: 7UMsN9qslWt
--HG--
extra : rebase_source : 4bc9554d8652e02e290c6a190634f1a72cdbadc3
We've been recording the commit id from the last vendor in
README_MOZILLA inside the various media directories. Since
we now support a --repo switch to pull from forks, record
this info as well, to make it easier to find contiguous
upstream source.
MozReview-Commit-ID: 1RanpkWfAeC
--HG--
extra : rebase_source : b6bd16b56626a871802822385be6f3a24db6cd50
Add a --repo switch to `mach vendor aom` to allow specifying
an alternate repository url.
Update our vendor script to support commit query and
snapshot download from github as well as upstream's
gitiles instance.
This lets us work with experimental branches for testing.
Also cleans up some naming and checks the passed url for
one of the two supported sites. We could fall back to
doing a complete clone and query the local repository,
but this covers most use cases.
--HG--
extra : rebase_source : 4ecc095db4539b86de4e82a853d5b28ac66c7f1d
This is a quick and dirty hack to get treeherder to show pytest failures. Long term, we might
want to investigate using something like pytest-mozlog. But the benefit of this approach is
we get to keep pytest's fantastic default log format.
MozReview-Commit-ID: Gcsz6z8MeOi
--HG--
extra : rebase_source : 00ee7973eadf86c081b548d5e79c48ca951e25a6
``print()`` has no business being in library code like this. It was
a holdover from this code being copied from bootstrap. So remove it.
While we're here, replace the generic exception with a specific one.
We don't want to be swallowing bugs via ``except Exception``.
MozReview-Commit-ID: 49goUstfPBz
--HG--
extra : rebase_source : a821159bd12a449ed1a0edf21a1f9eb29711ad95
We currently raise if we detect a command has both the `parser` attribute and a subcommand
at the same time, but as far as I can tell, there's no good reason to do this. Handling a
parser + subcommands seems to work exactly how you would expect. Furthermore, it isn't an
error to have subcommands + @CommandArgument, so it doesn't make sense that we're allowing
one but not the other.
This change solves an (admittedly unique) use case I'm trying to build into |mach try|. There
are N subcommands that all support a --save and --load style argument. So, e.g, we might have:
./mach try syntax --save foo
./mach try fuzzy --save bar
The main command will have the ability to detect which subcommand a saved value was generated
from and automatically dispatch to it. So this will work:
./mach try --load foo # dispatches to the syntax subcommand
./mach try --load bar # dispatches to the fuzzy subcommand
In order to share the --save/--load arguments across the main command + subcommands, we need
to set the parser attribute.
MozReview-Commit-ID: KmXRj8TBvYK
--HG--
extra : rebase_source : cbf1e402a080913709a34430274ae3191821dd72
This will facilitate handling this error condition specially.
Some unused imports were also removed.
MozReview-Commit-ID: 4zxVDgE7NxU
--HG--
extra : rebase_source : c4d0dd96629e028ce8a83215564a8d865e4b4b3d
The Repository interface gains a new method to determine if a sparse
checkout is present. Mercurial's implementation is somewhat crude,
but it should work (Mercurial's sparse support is still experimental
and I only intend to support sparse checkout in Firefox CI until it
is less experimental). Git's always returns False (for now).
To prove it works and to expose the information more widely, we hook
it up to moz.configure. We do this by first implementing a function
that returns a Repository instance. Then we simply call a function on
it to resolve the sparse checkout flag.
MozReview-Commit-ID: AlsT5LdSPdZ
--HG--
extra : rebase_source : f1e9aaa7d15f11c7c5e8d268d4ad82468732103b
This will remove the need to sniff class types. The 1 in-tree
consumer doing this has been converted.
MozReview-Commit-ID: I8cUa8J54VE
--HG--
extra : rebase_source : 4c24adaf7eb9d62678ac78604e819a7376d4073b
The --track flag provides a more accurate accounting of what files were
installed by the manifest, so they can be appropriately removed. For
example, test files are now removed from _tests if an entry in a test
file is deleted.
The --no-remove flag is removed as an alternative, and the --track flag
is now mandatory.
MozReview-Commit-ID: Wiup4Gzwkb
--HG--
extra : rebase_source : 4a44c7fe066ba9b5f1e37ec682464f7f4f6cb2cf
This patch removes the ability to select which protocols you want
included in necko, a wholly untested configuration that is broken in
practice. We have no need of this kind of configurability in necko.
In addition, this removes the final vestiges of rtsp support, which was
originally removed in bug 1295885 but still had some stuff hanging
around behind some ifdefs (that were never true).
MozReview-Commit-ID: KOEaDmit2IL
--HG--
extra : rebase_source : f6c2fdb972aaba46e922cda801252dc953550b94
This switches most tests over to use pytest as the runner instead of unittest (taking
advantage of the fact that pytest can run unittest based tests).
There were a couple tests that had failures when swithing to pytest:
config/tests/unit-expandlibs.py
xpcom/idl-parser/xpidl/runtests.py
For these tests, I added a runwith='unittest' argument so that they still run the
same way as before. Once we fix them to use pytest, the unittest logic in mozunit.py
can be deleted.
MozReview-Commit-ID: Gcsz6z8MeOi
--HG--
extra : rebase_source : 3c762422ce0af54cbbe7d9fc20085a2d1ebe7057
This change adds an upload-generated-sources task kind that runs after nightly
builds, fetches their `target.generated-files.tar.gz` artifact, and uploads
all the contained files to an S3 bucket. For actual nightly and release builds
on SCM level 3 trees, the S3 bucket is configured to be publicly accessible,
so that tools like Socorro will be able to fetch generated source files that
appear in crash reports, and debuggers will be able to fetch generated sources
when they show up while debugging Nightly or Release builds.
There are also level-2 and level-1 S3 buckets configured for builds happening
on trees of other levels such as try. They are not configured as publicly
accessible, but they exist so that these tasks can be tested in try.
MozReview-Commit-ID: Js1HRftbtep
--HG--
extra : rebase_source : b1172c9cc8b8be437d3b94a6bf0ff6b2f7d3508b
extra : source : 73bf88110b3821d62a3d393e85b56896a12f2930
This change makes us upload an `$(PKG_BASENAME).generated-files.tar.gz` archive
alongside other build artifacts which contains all the generated source files
from the build. A change after this will introduce an `upload-generated-sources`
task to take this artifact and upload the individual files to an S3 bucket.
This will be used to provide links to generated source files when they appear
in stack traces in crash reports.
MozReview-Commit-ID: 6yQAdlZ5q3O
--HG--
extra : rebase_source : d92fb17ae737d1360e9724997f6688e29bedef12
extra : source : 14d18d7cf454c4c3d0f6d49d1d01660e06e4be4b
Ensure stream only has a single LZMA block.
Ensure that dictionary size is always 8 MiB.
MozReview-Commit-ID: A0CV6M3LIf9
--HG--
extra : rebase_source : e5e35a33dd34a9eebae46f94c1c0fabf74038946
Tooltool manifests contain digests that have been used to validate
tooltool downloads. Toolchain artifacts don't benefit from that, and as
a result, an incomplete download can be considered as finished, and
unpack fail after that, without retrying, even with --retry.
Fortunately, the chain of trust artifacts do contains digests for
taskcluster artifacts, as long as the jobs that created the artifacts
have chain of trust enabled.
As of now, the goal is not cryptographic validation of the download, but
to ensure that we got the complete file, and to trigger a new download
if we haven't.
--HG--
extra : rebase_source : cdf4b4ec0c99db1f671db799f3941804f2bcbaf9
We define extra_toolchain_flags for passing extra flags to the target
compiler during configure. But the way things are currently set up, we
pass those flags to the host compiler during configure as well. This
behavior is incorrect, and we should only be passing the flags from
extra_toolchain_flags if we're compiling for the target.
When the tooltool manifest contains e.g. clang.tar.bz2 and it's also
provided by some toolchain job artifact, we still download both. In
fact, depending how things go, the tooltool one could take precedence.
In practice, this hasn't caused problems so far because we've removed
tooltool manifest items early on, and when we didn't, it was still the
same version as provided by toolchain jobs.
It is still necessary to fix for the future, though.
--HG--
extra : rebase_source : 1d187333ee6149f72e3f9ed91eb1a6a78ad9197f
We correctly worked around problems for `hg pull` but not for
`hg update`. Make the workaround consistent.
MozReview-Commit-ID: 7A4dgANO0ip
--HG--
extra : rebase_source : fd4b90200ab3ce07a3921a02ae28f5a7a0c3d83c
Other related changes:
* Only target.{zip,tar.gz,tar.bz} get the widevine signature (not the various installers).
* Linux{32,64}-nightly are now repackaged. Their mar files are not signed during signed step anymore. It now happens after the repackage.
* As a consequence, funsize routes for linux are now set to repackage-signing (instead of signing)
* Signed upstream artifacts are now defined in a dedicated module (to avoid duplication)
* Platforms defined in beetmover_repackage now allow regex (to reduce duplication too)
* Mozharness configs: Delete unused (and misleading) `src_mozconfig` for windows. This value is actually not used when `run_configure` (in the same dict) is set to False.
MozReview-Commit-ID: COKqevW9Mzn
--HG--
extra : source : ffc2e43aa834e05f0d51d68dfb36317c1b408b08
Various people want to start experimenting with Python 3 in the build
system and in related tools (like mach).
We want to make it easy to find and use an appropriate Python 3
binary.
This commit introduces a generic function for finding a Python 3
binary and resolving its version.
We use the new code in configure to set PYTHON3 and PYTHON3_VERSION
subst entries for later consultation.
We also expose a cached attribute on the base class used by many
mach and build system types to return a Python 3 executable's info.
By default, we only find Python 3.5+. From my experience, Python 3.5
was the first Python 3 where it was reasonable to write code that
supports both Python 2 and 3 (mainly due to the restoration of the
% operator on bytes types). We could probably support Python 3.4
in the build system. But for now I'd like to see if we can get
away with 3.5+.
MozReview-Commit-ID: BlwCJ3kkjY9
--HG--
extra : rebase_source : b00464972183ef1a97a0b5d888520be425717ae7
A recent mercurial update changed the order of revisions returned by the
"last" revset. The expected revisions are still output, so the artifact
code is updated in this change to impose its own order based on the local
revision order to accommodate any output order.
MozReview-Commit-ID: 7Zka0kQtxJO
--HG--
extra : rebase_source : 415077a806053c0452e4f9dec997a0e40934e51d
Introduce a new make command to produce new type of language packs based
on web extensions.
MozReview-Commit-ID: EltYtzT7ZRR
--HG--
extra : rebase_source : 6477be07c747e90992f18d8c7bff93fd48965200
While --fix previously worked with eslint, it is now more official (will show
up in the |mach lint --help|). ESlint is still the only thing that implements
it, but we can implement it for flake8 using the `autopep8` module.
--edit is a new concept that will open an editor for each failing file to let
you fix the errors manually. For now it is very naive (just opens the file),
and is only really useful if you have an editor integration for the linter(s).
But in the future I'd like to have editor-specific implementations for this.
For example, with vim, we can use -q to pass in an error file that will start
the editor pre-populated with a list of all errors that can then be easily
jumped to. Other editors may just open up to the line containing the error.
--fix and --edit can be used in conjunction with one another. Doing that means
only errors that can't be fixed automatically will show up in your editor.
MozReview-Commit-ID: 5JJJhMIrMIB
--HG--
extra : rebase_source : 2f78a77a91133d7fcc5620ecd5caa500decbce1b
While --fix previously worked with eslint, it is now more official (will show
up in the |mach lint --help|). ESlint is still the only thing that implements
it, but we can implement it for flake8 using the `autopep8` module.
--edit is a new concept that will open an editor for each failing file to let
you fix the errors manually. For now it is very naive (just opens the file),
and is only really useful if you have an editor integration for the linter(s).
But in the future I'd like to have editor-specific implementations for this.
For example, with vim, we can use -q to pass in an error file that will start
the editor pre-populated with a list of all errors that can then be easily
jumped to. Other editors may just open up to the line containing the error.
--fix and --edit can be used in conjunction with one another. Doing that means
only errors that can't be fixed automatically will show up in your editor.
MozReview-Commit-ID: 5JJJhMIrMIB
--HG--
extra : rebase_source : 889c825dcde5e950e35bc64539b299988f82ac68
We should have CI Lint YAML files in the tree.
MozReview-Commit-ID: IMOKGhxKFJW
--HG--
extra : rebase_source : 8c695dcb3a81b6683bc76ba663feeacb3126691e
We should have CI Lint YAML files in the tree.
MozReview-Commit-ID: IMOKGhxKFJW
--HG--
extra : rebase_source : a2e47f5c5517e999fe21bac4f86491611bd0cece
We should have CI Lint YAML files in the tree.
MozReview-Commit-ID: IMOKGhxKFJW
--HG--
extra : rebase_source : 0ef71f24141a531833d48f2a305183dd808f00f5
This patch introduces a new registry for localization resources to replace
ChromeRegistry. It uses asynchronous I/O and iterators to generate
permutations of possible sources of language resources for use in the new
Localization API.
In the initial form the API handles packages resources and has API for
interacting with the AddonsManager which will report language packs.
MozReview-Commit-ID: LfERDYMROh
--HG--
extra : rebase_source : 68a664c2c59a82b4dfcae66542c315a00ddae565
This patch introduces a new registry for localization resources to replace
ChromeRegistry. It uses asynchronous I/O and iterators to generate
permutations of possible sources of language resources for use in the new
Localization API.
In the initial form the API handles packages resources and has API for
interacting with the AddonsManager which will report language packs.
MozReview-Commit-ID: LfERDYMROh
--HG--
extra : rebase_source : a6e5b790142e5afb1ce750478190e5ad87012f8d
This mechanically replaces nsILocalFile with nsIFile in
*.js, *.jsm, *.sjs, *.html, *.xul, *.xml, and *.py.
MozReview-Commit-ID: 4ecl3RZhOwC
--HG--
extra : rebase_source : 412880ea27766118c38498d021331a3df6bccc70
Update scripts to support both lzma and bzip2
Update unused python script to support lzma. This also adds python 3.0 support to the script while still supporting pythin 2.7
Update test scripts to support lzma
f497b6194e9f bumped the minimum Java version requirement in configure
from 1.7 to 1.8. It forgot to update tests that were pinning the
Java version at 1.7. The tests then failed.
We bump the versions in tests to restore order.
CLOSED TREE
MozReview-Commit-ID: 1zm5L1QQy7a
--HG--
extra : rebase_source : 8868224b5a1a8ffbb633b638df9485760f84fd8a
extra : amend_source : 3faa435ab5478c62aefb3dc18de19256ce0d906e
There's a natural follow-on that I haven't time to explore right now:
I want the faster make backend to also write a "unified chrome
manifest" that maps outputs
(browser/chrome/browser/content/browser/ext-utils.js) to chrome:// or
resource:// URLs (chrome://content/browser/ext-utils.js or similar).
MozReview-Commit-ID: LDQmm8KD57I
--HG--
extra : rebase_source : 13dbeac2fbfa78741be3718fd5305a8ae0d698a8
extra : source : 2af5df7741147769a51da78770c308b370a0cded
Prefs can be set with `./mach run --setpref foo=bar` or in the ~/.mozbuild/machrc file as:
[runprefs]
foo=bar
MozReview-Commit-ID: HO3tdFi9ffi
--HG--
extra : rebase_source : 3c2e6f33567448c19defafb656e6cb9f3a729391
Capture the list of generated source files derived from moz.build data
and save it in a generated-sources.json in the objdir so that we can upload
generated source files for use in crash reports and when debugging release
builds.
MozReview-Commit-ID: FrHcyFo0rBF
--HG--
extra : rebase_source : 09a5a38d22430e9a2c8121ddc8d47fabdd3be705
This is necessary because the existing manifests don't expose full
dependency information. I needed to avoid the existing dependency
files because those code paths need to know the output destination of
the manifest in order to parse the Make dependency files; trying to
adapt this system is more complicated than just preprocessing each
file to extract dependency information directly.
MozReview-Commit-ID: 5m0SEqmhJMM
--HG--
extra : rebase_source : 1ff6a1a51bc76efa78eb564cd8e572777dace0f6
Also removes InstallManifestNoSymlinks which can be more simply expressed by
passing link_policy='copy' to InstallManifest.populate_registry.
MozReview-Commit-ID: Bkjc2hIub4A
--HG--
extra : rebase_source : df05080fef3baf7e6c60ff9a468e71f0b2c67462
In bug 1181040, we added ${var}_IS_SET variables for
mk_add_options-defined variables. In the two years since, that has never
been used for anything else than MOZ_PGO_IS_SET, and the only use for
that has now been removed, so remove those ${var}_IS_SET variables.
--HG--
extra : rebase_source : 2fc9abe0c3badbf06f3858fcf326237e67891fee
Chain of trust validation will require to know what the inputs for a
given build are, and mach artifact toolchain fetches such inputs.
So we make it output a manifest that the chain of trust validation
process will be able to use and correlate with other information, such
as the one from bug 1383993.
At the same time, we make the produced manifest contain information
about tooltool-downloaded packages, which will allow to track the
progress in the migration from tooltool to TC artifacts.
--HG--
extra : rebase_source : 5b3fc32a9fd641cc7edc57865d2b60aaa6ffcbed
Hacky, but it works -- until Google updates its license hashes. This
looks ahead to using |mach bootstrap| to build docker images.
MozReview-Commit-ID: DF23v8tr8SW
--HG--
extra : rebase_source : 5a80bdd5ddfb551b374464f42c3783fef5a71fc3
This looks ahead to using |mach bootstrap| when building docker
images.
MozReview-Commit-ID: LoMU7ji5T0x
--HG--
extra : rebase_source : 2885195855497d69b1cbba02a943964dd3286b93
This was a regression introduced in Bug 1344244. Google changed the
layout of their packages. It used to be that the Android SDK archive
had a top-level 'android-sdk-$OS_NAME' directory; that's no longer the
case. It would be cleaner to unpack to 'android-sdk' without the OS
name, but that complicates the logic that detects existing installs.
MozReview-Commit-ID: 4B2Rt1AM2ky
--HG--
extra : rebase_source : 3cd1fef88cbb47fc9cd4e47a2a4bc2277c576c58
Consider Rust toolchains prior to 1.19.0 old enough to update
by the `mach bootstrap` command, in preparation for requiring
that release during Firefox 57 nightly development.
MozReview-Commit-ID: JaaM3sPDmzn
--HG--
extra : rebase_source : 3835103a822cfc05bde92ee920b344fe7b68d61c
Adding PROG_IS_C_ONLY seems like a good point to add tests, and once we
have tests for that, adding tests for the existing library support is
not too difficult.
Similar to the existing LIB_IS_C_ONLY variable, these variables indicate
that the program in question has only C sources and so can be linked by
the C compiler rather than the C++ compiler. We need to add a little
more information to BaseProgram so we can avoid emitting periods into
Makefile variables.
Bug 1374940 adds a MOZ_TOOLCHAINS environment variable with a list of
path@task-id strings, where task-id is corresponding to the (possibly
optimized) toolchain job, and path corresponding to the
toolchain-artifact defined for that toolchain job.
We want to use that to pull artifacts instead of tooltool packages.
--HG--
extra : rebase_source : 277daa2c83d6d197975cb4ef36ee131176afa992
For reasons I can't explain, Windows builds are failing intermittently
because they are unable to locate the `hg` binary when running
some SpiderMonkey test processes. These processes use
mozversioncontrol.get_repository_from_env() to locate the
current repository.
We now store VCS info in configure. This makes it available to anything
running in a build system context.
This commit teaches mozversioncontrol.get_repository_from_env()
to import the "buildconfig" module to locate VCS info. If the module
can be imported, it is the sole source of VCS info. Otherwise, we
fall back to the existing detection mechanisms.
This should get rid of the intermittent failure. If it doesn't,
it is still a step in the right direction because it will allow
build system processes to consistently use a well-defined VCS
binary.
MozReview-Commit-ID: DMxXheJLRqH
--HG--
extra : rebase_source : a9c599934c8c08da1fbb92a9105f5c7cba0867b3
We now store HG or GIT in substs. We don't need to search for
binary paths.
MozReview-Commit-ID: 8sSgPNLok9M
--HG--
extra : rebase_source : bc51087bcb9f2a723e27f240dd06a88540f6d8a8
We now have a variable in config.status for recording the checkout
type. These helper functions for determining if we're Mercurial or Git
can now be one-liners.
As a bonus, we no longer do I/O as part of this function.
MozReview-Commit-ID: HT9sbOhDEkf
--HG--
extra : rebase_source : 8b53b5f50d14c0bdd4ef3dc7b190314af80a76f0
For reasons I can't explain, Windows builds are failing intermittently
because they are unable to locate the `hg` binary when running
some SpiderMonkey test processes. These processes use
mozversioncontrol.get_repository_from_env() to locate the
current repository.
We now store VCS info in configure. This makes it available to anything
running in a build system context.
This commit teaches mozversioncontrol.get_repository_from_env()
to import the "buildconfig" module to locate VCS info. If the module
can be imported, it is the sole source of VCS info. Otherwise, we
fall back to the existing detection mechanisms.
This should get rid of the intermittent failure. If it doesn't,
it is still a step in the right direction because it will allow
build system processes to consistently use a well-defined VCS
binary.
MozReview-Commit-ID: DMxXheJLRqH
--HG--
extra : rebase_source : a9c599934c8c08da1fbb92a9105f5c7cba0867b3
We now store HG or GIT in substs. We don't need to search for
binary paths.
MozReview-Commit-ID: 8sSgPNLok9M
--HG--
extra : rebase_source : bc51087bcb9f2a723e27f240dd06a88540f6d8a8
We now have a variable in config.status for recording the checkout
type. These helper functions for determining if we're Mercurial or Git
can now be one-liners.
As a bonus, we no longer do I/O as part of this function.
MozReview-Commit-ID: HT9sbOhDEkf
--HG--
extra : rebase_source : 8b53b5f50d14c0bdd4ef3dc7b190314af80a76f0
Having Rust dump a stack on panic seems like a useful on-by-default
feature.
MozReview-Commit-ID: ABYTArsMTFh
--HG--
extra : rebase_source : 7d15f44a9ffd14db475db9e5c2964aca31bf0f70
This allows subcommands to use external argument parsers.
MozReview-Commit-ID: 7TkbTff0Tv8
--HG--
extra : rebase_source : a1c245efa7ac7b28b974534b4cd2727c96f9219d
After this patch, the following will all display the subcommand help where they previously displayed
the command help:
$ mach help <command> <subcommand>
$ mach <command> --help <subcommand>
$ mach <command> <subcommand> --help
The command help will still be shown for:
$ mach help <command>
$ mach <command> --help
MozReview-Commit-ID: 7EsVblnCaFM
--HG--
extra : rebase_source : 2a1d289d56164366ce140fa653adec93f56be067
There's no need to install the caskroom bits for Homebrew now,
allowing to fix Java installation on Mac OS X and simplify the code at
the same time.
MozReview-Commit-ID: 1Ssjm4YRrPQ
--HG--
extra : rebase_source : 3414d1fbe2bdb693cae1f5b1379d8d9335f1e7f4
This is the real fix. Google has replaced the |android --no-ui ...|
tool with a simpler |sdkmanager| tool, which makes it easier to
install packages with particular major versions. (Minor versions
still can't be controlled; for example, the m2repository extras are
constantly rolling forward.)
|sdkmanager| fails if the required packages aren't installed and can't
be installed, so there's no need to search for missing packages, etc,
simplifying the code considerably.
I don't see an easy way to upgrade outdated Android SDK installations
-- it's not clear that unpacking over top of an existing SDK
installation succeeds -- so I've included a message about moving or
removing outdated installations. This will punish folks who have
added additional Android platforms, or download emulator images using
the Android toolchain (but not those downloaded using |mach
emulator|). C'est la vie.
MozReview-Commit-ID: GLhKyuq701k
--HG--
extra : rebase_source : 26578c5ef4dcc6a29809630add2232a98407ec54
This refactoring unifies similar code that has been copy-pasted and
subsequently diverged.
MozReview-Commit-ID: EuVQBR4gsDo
--HG--
extra : rebase_source : bda66ef9001a1cddf75417aaeebd9dcecd05a6b7
This patch changes the name of the keys that are in the 'linked-files-map.json' that is produced in the code coverage build and are used to map symbolic links to their source files. The new key names (which are the paths to the symbolic links) are now the entire absolute path to each of the files.
MozReview-Commit-ID: 4x1dfk9h2Ov
--HG--
extra : rebase_source : 7d424bbbf1d026ea67c66b743c8c43ea75185733
This is similar to bug 1301751, where something in rust seems to trigger errors
running dsymutil to generate debug symbols in OSX. We can set debuginfo=1 for
these builds as a temporary workaround for now, while we work on a more
permanent solution in rust and/or dsymutil. debuginfo=1 still gives us enough
info for stack traces, although without line info. debuginfo=2 would be useful
for debugging, but is irrelevant to crash reports.
MozReview-Commit-ID: DdA00GzVfWg
--HG--
extra : amend_source : 47d3573042098194a07f9b473e4a02c86a1eba7c
The rigid type comparison added in 51a92a22d6d1 (bug 1375231) was
too rigid. This broke at least one consumer that was comparing an
empty PositiveOptionValue/NegativeOptionValue against a string.
Since comparisons against empty OptionValue are convenient and
don't violate the spirit of the type checking previously added,
this commit relaxes the type equivalence check in cases where the
OptionValue is empty.
MozReview-Commit-ID: UllTrzCjj
--HG--
extra : rebase_source : 2c41428d1be667edecdab0947d006a1a6a533569
OptionValue and its derived classes are exposed to moz.configure
files. As the previous bug fix showed, it is really easy to
accidentally assume the type is a simple string value and do a
string compare against it.
To prevent this class of bugs, this commit adds additional type
awareness to OptionValue.__eq__.
We first check that the argument is a tuple (including any OptionValue
types). If not, we raise a TypeError because the comparison is
invalid. This is arguably a violation of __eq__. But since OptionValue
is exposed to the moz.configure sandbox and typing '==' will invoke
__eq__, we have to do something to prevent this foot gun.
The change also changes the comparison logic.
If we compare against a non-derived tuple instance, we do a tuple
compare. Otherwise, we fall back to the previous logic of
requiring an identical type then doing a tuple compare.
MozReview-Commit-ID: 7IVSL2Asg9j
--HG--
extra : rebase_source : edab573834da240df9ad7c2fc78c85d6159a6ef9
Leaving off this argument makes `mach vendor rust` with large files fall
over with a Python error. While the user still gets a semi-useful error
message prior to this failing, it would be better to not fail here at all.
Cargo recently introduced the `cargo check` command for shortening the
edit-compile cycle when working on large programs. Since we don't
really support invoking `cargo` directly, let's wire up this command to
`mach`. Gecko developers can then `mach cargo check` to ensure their
changes typecheck.
We have a flag set on all Linkables, cxx_link, denoting whether there's
anything being linked into them that requires C++. We do this even when
we link against shared libraries that required C++. But if these
libraries don't export C++ interfaces, there's no reason that the things
linking against them should require C++. Therefore, ignore shared
libraries when making the determination of whether an object requires
C++ or not.
This patch renames the mozinfo flag 'coverage' to 'ccov' to avoid ambiguity in whether a test is being skipped for linux64-ccov or for linux64-jsdcov. It also removes the 'runtests.py' mozinfo hack and renames all occurrences of 'coverage' that are used for skipping tests in linux64-ccov.
MozReview-Commit-ID: IF2640bDQP7
--HG--
extra : rebase_source : 614020325e30d1ca9e01aaf08479b8a4ffaec888
With configure now being able to auto-detect the presence of a `mach
bootstrap`-installed clang and libclang and the upcoming
build-by-defaultness of Stylo, we can start downloading these packages
all the time.
LLVM/clang is needed for Stylo's bindgen, and Apple's clang is unusable
for such purposes. For other platforms, we have installed LLVM/clang
from our tooltool archive on the supposition that those packages are
definitely known to work, as we use said packages in automation. For
Mac, however, we haven't been able to generate packages for tooltool
that successfully build Stylo, and even if we had, those packages would
solely be used for developer builds of Stylo and would not be used in
automation. The case for downloading LLVM/clang for Mac from tooltool,
therefore, is not as strong as for other platforms.
As a result, we'll rely on the installed package manager for LLVM/clang,
which many people may have installed anyway.
In passing, also delete some old code for OS X versions < 10.7; such
platforms are no longer supported for running or building Firefox.
This patch adds a flag to the 'mozinfo.json' that can be used to disable tests when they are running on linux64-ccov. Then, this flag is used to prevent the marionnette test 'test_crash.py' from running on linux64-ccov.
MozReview-Commit-ID: 9IHMiZHxcMK
--HG--
extra : rebase_source : ec690cb3ffa27d3e88d2c0b8c5d510e72a5c5079
When determining the path of a possible mozilla-central checkout,
bootstrap currently considers the existence of a `.git` directory as
sufficient proof that the containing directory is a git checkout.
Unfortunately, if you happen to execute standalone bootstrap from a git
checkout of something else, you're gonna have a bad time.
To prevent this, check for the existence of a moz.configure file. This
is not an ideal proof, but it is better than what we currently have.
The -fsanitize=integer analysis from UBSan can be helpful to detect signed and unsigned integer overflows in the codebase. Unfortunately, those occur very frequently, making it impossible to test anything with it without the use of a huge blacklist. This patch includes a blacklist that is broad enough to silence everything that would drain performance too much. But even with this blacklist, neither tests nor fuzzing is "clean". We can however in the future combine this with static analysis to limit ourselves to interesting places to look at, or improve the dynamic analysis to omit typical benign overflows.
It also adds another attribute that can be used on functions. It is not used right now because it was initially easier to add things to the compile-time blacklist to get started.
Finally, it includes a runtime suppression list and patches various parts in the test harnesses to support that. It is currently empty and it should not be used on frequent overflows because it is expensive. However, it has the advantage that it can be used to differentiate between signed and unsigned overflows while the compile-time blacklist cannot do that. So it can be used to e.g. silence unsigned integer overflows on a file or function while still reporting signed issues. We can also use this suppression list for any other UBSan related suppressions, should we ever want to use other features from that sanitizer.
MozReview-Commit-ID: C5ofhfJdpCS
--HG--
extra : rebase_source : 952043a441b41b2f58ec4abc51ac15fa71fc142f
Support OSX Signed nightlies (in the complete.mar too)
MozReview-Commit-ID: HXiFGE14wYJ
--HG--
extra : rebase_source : 1d02b4714c8fafe6cdcd74e6d9b5612c44dcb3b4
- Set the extensions.legacy.enabled pref for mochitests etc
- Skip a plugin-inside-xpi test for now if legacy extensions
are force-disabled. That test can just be removed once we
get to 57.
MozReview-Commit-ID: As9LtkQTcTS
--HG--
extra : rebase_source : fcc84daef95c453e893cc3b98498fdb87f54b1bb
This adds a py_action invocation wrapping aapt and implements a hacky
implementation of the Gradle build system's resource merging
algorithm; once we have the moz.build and Gradle resources identical,
we'll be one big step closer to producing bit-identical builds and
flipping the switch in favour of Gradle. With this, the R.txt
produced by the aapt invocation is the same as the R.txt produced by
the py_action invocation.
Originally I wrote this to use GENERATED_FILES, but it produced a
world of pain. Since Android's aapt tool is fundamentally directory
oriented, not file oriented, it required adding support for FORCE to
GENERATED_FILES and required directory crawling and FileAvoidWrite in
the wrapper. After getting that working I was eventually stymied by
the arcane requirements of the Android re-packaging system, which
interacts with the l10n system. I would have required support for
building GENERATED_FILES in the libs tier rather than the misc tier.
After that realization I gave up and turned to py_action: the
dependencies on branding are just too entangled with l10n to use
GENERATED_FILES.
And, in the not-so-distant future, all of this moz.build and
Makefile.in chicanery will be deleted in favour of invoking Gradle at
the appropriate points!
MozReview-Commit-ID: 4ueVNa7gzgs
--HG--
extra : rebase_source : dab092a188bc735ef819a4be0ad13387e85c87e2
extra : source : a05bd87b09ee5e4cff20fe84c2e75ac3b2a494a1
Create a test for version control related functionality.
MozReview-Commit-ID: GXd27O69GNg
--HG--
extra : rebase_source : 56ce4a38b591fd62f05fbaed0ff05d56ec127422
The underlying commands to mozlint's vcs.py had a few problems. This:
1. Ensures deleted files aren't considered in both hg and git
2. Automatically determines the default remote repo and branch when using --outgoing with git
3. Excludes metadata from output of the git command used with --outgoing
4. Adds --cached to the git command for --workdir, which lints all staged files
MozReview-Commit-ID: EBtM3MCiCDs
--HG--
extra : rebase_source : 982b24acd81c86e383b28b8a71bcd51a041e60f4
The old method had a bunch of 'if vcs == git' statements and the like. This
made it hard to follow code paths for a given vcs system. I think this refactor
makes things cleaner to read. It shouldn't be changing any functionality.
MozReview-Commit-ID: EBtM3MCiCDs
--HG--
extra : rebase_source : 9c0fda3f4f824351bae852af25bcd2240b1b1024
The rev option is inherently broken. It does let you lint files touched by any
revision, but it doesn't update those files to that revision first. Instead,
they get linted at whatever the working directory is and their results are
bogus. Even if we did some magic to update the files to the proper revision
with in-memory version control magic, the config files would still be out of
date.
Plus, the new --outgoing option does pretty much the only thing --rev was good
for. Rather than cause confusion, I think it's better to just remove the
option.
MozReview-Commit-ID: 2y2UnfIkvsR
--HG--
extra : rebase_source : 9b5c142270c98905d71ebb89d1620e91914c0b47
This prevents us from redundantly installing httpd.js and httpd.manifest
from the test package during an artifact build, which interferes with
the Tup backend's handling of these files as symlinks.
MozReview-Commit-ID: LuMurUc1P36
--HG--
extra : rebase_source : 1aabd788ff71ae28434a4076d5304f611ada5d92
As mozprocess doesn't have any special handling of stderr, should cargo
operations fail their output is dropped. Switch to subprocess.check_call to
ensure cargo errors are displayed to the caller.
This was a regression from bug 1288432. The 'extensions' config in mozlint required a
leading period, but eslint requires them without the period (and this got copied over
to the linter definition). The result was mozlint filtering out any files (not dirs)
that were passed in.
This just modifies mozlint to strip out the period so both are acceptable.
MozReview-Commit-ID: CbNynYzrbGz
--HG--
extra : rebase_source : 51c740cb1d2febaee3ae46784f83381cda5e5eaa
libGL package was consolidated into mesa-libs but quarterly set still
uses the old name. Since gtk3 (via libepoxy) and gtk2 (via cairo)
already indirectly depend on mesa-libs take advantage of it to avoid
churn on updates.
MozReview-Commit-ID: F5LBOOthAMc
--HG--
extra : rebase_source : 4e3e30e817187c3ffef30e280554b12b02f44568
Bump the target version and checksums for the rustup installers
we use for the latest release so reduce the variance with manual
installs.
MozReview-Commit-ID: E5O4UOu1wLr
--HG--
extra : rebase_source : a0745515957667787929bc5df05a66adb29cbd66
If we don't do this, various bits of test infrastructure will turn on
when stylo is merely built, not enabled, which will cause no end of
orange and unhappiness.
