6881 Commits

Author SHA1 Message Date
Nick Alexander
cba669901b Bug 1713173 - Add --profile-command flag for profiling mach commands. r=firefox-build-system-reviewers,mhentges
I chose to do this at the level of the outer Python invocation because:

1. `python -m cProfile ...` handles writing the file and some other
   details.  It's possible to rebuild the functionality -- the tools
   are there -- but the APIs are awkward.

2. this allows to profile `mach` internals, instead of just the
   invoked command's implementation.

This uses the return code of the `get_command` subshell to transmit
the single bit of information "is the flag present".

The Python-level argument is required in order to have `--help` know
about the option and to avoid the `mach` shell script having to filter
arguments.

Differential Revision: https://phabricator.services.mozilla.com/D116151
2021-06-02 21:28:56 +00:00
Nick Alexander
e11d5cd04e Bug 1714159 - Handle c:/... and C:/... when making relative paths in mozlint. r=ahal
Depends on D116151

Differential Revision: https://phabricator.services.mozilla.com/D116637
2021-06-02 21:27:40 +00:00
Alexandru Michis
fd44fe5ee5 Bug 1714156 - Fix lint failures in test_manifest.py
CLOSED TREE
2021-06-02 21:25:07 +03:00
june wilde
bece79b87e Bug 1714156 - Disallow 0 weeks in frequency in moz.yaml schema; r=tjr
Differential Revision: https://phabricator.services.mozilla.com/D116626
2021-06-02 18:05:49 +00:00
Mike Hommey
2733dbe0dc Bug 1713733 - Remove WASM_LDFLAGS. r=firefox-build-system-reviewers,mhentges
It's not practically make a difference (the flags we end up setting it
to don't do anything when linking all the wasm files into one).

Differential Revision: https://phabricator.services.mozilla.com/D116437
2021-06-02 04:34:20 +00:00
Butkovits Atila
1b7d8eac8e Backed out changeset a9562036bedd (bug 1713818) for causing wrench bustages. 2021-06-02 02:19:35 +03:00
Mitchell Hentges
271c19459a Bug 1713818: Require Python 3.6+ to run Mach r=ahal
This will both:
* Ensure that developers aren't accidentally using Mach with Python 2
  in the year 2021.
* Confirm that CI tasks aren't still using Python 2 with Mach.
* Remove support for Python 3.5, which is EOL'd.

Differential Revision: https://phabricator.services.mozilla.com/D116484
2021-06-01 20:03:38 +00:00
Nick Alexander
a8f3540069 Bug 1713178 - Make black linter setup avoid updating the virtualenv if it's installed. r=ahal
This could be easily generalized to avoid the updating the virtualenv
more generally, but I think this functionality should be in the
`virtualenv_manager` -- something like `install_pip_binary(...)` -- so
I've just done the most impactful example here.

Differential Revision: https://phabricator.services.mozilla.com/D116148
2021-05-31 21:44:05 +00:00
Alex Lopez
1ee8086c02 Bug 1696251 - Refactor constructors in MachCommandBase subclasses to remove them. r=mhentges,remote-protocol-reviewers
Another step towards avoiding the need for classes in mach commands;
here we are removing constructors either by changing them into helpers
or by simple refactorings.

Differential Revision: https://phabricator.services.mozilla.com/D112434
2021-05-28 18:30:03 +00:00
Steve Fink
6497489f40 Bug 1713235 - Use python3-devel for CentOS 8 r=firefox-build-system-reviewers,mhentges
Differential Revision: https://phabricator.services.mozilla.com/D116184
2021-05-28 14:32:07 +00:00
Mitchell Hentges
06a214573a Bug 1712819: Avoid pip's "outdated" warning in virtualenvs r=ahal
Now, when running mach commands that invoke `pip`, it will no longer
inform the user that it needs an update.

We reach into `distutils` to determine the "site-packages" directory
pattern, then apply it to our virtualenv.

Differential Revision: https://phabricator.services.mozilla.com/D115940
2021-05-27 23:04:06 +00:00
Mitchell Hentges
79c56399e1 Bug 1712819: Fix VirtualenvManager not being expandable in debugger r=ahal
At least in PyCharm, expanding a `VirtualenvManager` instance means
resolving all the properties and fields of the instance.
However, if that property is doing non-trivial work, the debugger
wouldn't run that subprocess while we're stopped at a breakpoint.
So, the instance would sit there with the "Collecting data..." text.

Differential Revision: https://phabricator.services.mozilla.com/D115935
2021-05-27 23:04:06 +00:00
Mike Hommey
95e6d3cd22 Bug 1712350 - Stop guessing where make is in python code. r=firefox-build-system-reviewers,mhentges
This was necessary before because we first started the build by invoking
make with client.mk, which would then invoke configure. But that changed
in bug 1671424, and we now never need make before configure has run
(except in tests, for which we implement a fallback).

Differential Revision: https://phabricator.services.mozilla.com/D116064
2021-05-27 22:27:25 +00:00
Butkovits Atila
6bb47ac424 Backed out changeset 4b889750da57 (bug 1696251) for causing lint failures. CLOSED TREE 2021-05-28 01:43:11 +03:00
Alex Lopez
2e8613720d Bug 1696251 - Refactor constructors in MachCommandBase subclasses to remove them. r=mhentges,remote-protocol-reviewers
Another step towards avoiding the need for classes in mach commands;
here we are removing constructors either by changing them into helpers
or by simple refactorings.

Differential Revision: https://phabricator.services.mozilla.com/D112434
2021-05-27 21:03:46 +00:00
Mitchell Hentges
c329fa104b Bug 1713060: Don't override "substs" to fetch it fallibly r=ahal
A clobber must be able to run, even if `substs` (configure output)
is not available.
However, when this behaviour was implemented, it was set up to replace
all usages.

Since we're trying to make mach commands no longer objects, this
kind of overriding won't be possible in the future. Fortunately, it's
not needed: only `Clobberer` needs a fallible substs, the other
consumers don't!

Differential Revision: https://phabricator.services.mozilla.com/D116128
2021-05-27 21:00:42 +00:00
Mitchell Hentges
78fcd81e02 Bug 1712382: ./mach clobber gradle works if directory already gone r=nalexander
Back-to-back `./mach clobber gradle` invocations were failing
because `shutil.rmtree(..)` would complain that the target directory
didn't exist.

Differential Revision: https://phabricator.services.mozilla.com/D116129
2021-05-27 18:04:32 +00:00
Mitchell Hentges
de10b8114b Bug 1712382: Includes virtualenvs when clobbering Python r=glandium
No longer includes `python` in the `./mach clobber` defaults since
python cache files should no longer be affecting `./mach artifact`.

Removes `third_party/python` purging from `./mach clobber python`
since we don't build native Python modules there.

Moves virtualenv-purging from `./mach clobber objdir` to
`./mach clobber python`.

Differential Revision: https://phabricator.services.mozilla.com/D115728
2021-05-27 18:04:32 +00:00
Dorel Luca
61aa3325c5 Backed out 2 changesets (bug 1712819) for Python failures. CLOSED TREE
Backed out changeset f51d72e5f0b8 (bug 1712819)
Backed out changeset 40c109bb5f9a (bug 1712819)
2021-05-27 19:05:15 +03:00
Mitchell Hentges
20d4059a6f Bug 1712133: Simplify virtualenv "sitecustomize" writing r=ahal
Child `handle_package(...)` invocations don't need to modify
`sitecustomize.py`, so don't pass it to them.

Differential Revision: https://phabricator.services.mozilla.com/D115924
2021-05-27 15:18:37 +00:00
Mitchell Hentges
17f68a38c4 Bug 1712133: Use if/else chain instead of early-return in handle_package r=ahal
Accidentally missing a `return` in a code path could mean that
`handle_package(...)` would accidentally do an action _and_
raise the "Unknown action" error.

This change resolves that, and it simplifies the code a bit.

Differential Revision: https://phabricator.services.mozilla.com/D115923
2021-05-27 15:18:37 +00:00
Mitchell Hentges
0d1d6ed813 Bug 1712804: Change comm action to be specific to Thunderbird r=rjl,ahal
Avoid the "Error processing command" warning when building virtualenvs
in a Firefox checkout

Differential Revision: https://phabricator.services.mozilla.com/D115922
2021-05-27 15:18:37 +00:00
Mitchell Hentges
9522e92323 Bug 1712382: Remove "set-variable" action support from virtualenvs r=ahal
`MACH_VIRTUALENV` was never used, and `MOZBUILD_VIRTUALENV` was never
set (the virtualenv was always activated during the build, rather
than before).

Differential Revision: https://phabricator.services.mozilla.com/D115921
2021-05-27 15:18:36 +00:00
Mitchell Hentges
04a6f9736c Bug 1712382: Remove "setup.py" action support from virtualenvs r=ahal
The `setup.py` action is no longer used, so we can safely remove it.

Depends on D115641

Differential Revision: https://phabricator.services.mozilla.com/D115913
2021-05-27 15:18:36 +00:00
Mitchell Hentges
5eb07cd570 Bug 1712133: Remove build VIRTUALENV_NAME customization r=glandium
This was originally set up so that tests wouldn't "create a new
`virtualenv` for no reason." However, virtual environments now will have
different packages installed, and therefore the separation is necessary.

So, for the virtual environment used for builds (regular or for tests):
* We want it to be able to reuse the build venv, if it already exists.
* We don't want to pollute a `pytest` virtualenv with build-specific
  packages.

Differential Revision: https://phabricator.services.mozilla.com/D115641
2021-05-27 15:18:35 +00:00
Mitchell Hentges
846e25c93b Bug 1712133: Rename default virtualenv to "common" r=ahal
We had split up `init` from `init_py3` because `mach` had
traditionally been invoked by either Python 2 or Python 3, and
the two couldn't share the same virtualenv.

Now that the same context isn't shared by both Python 2 and 3
3
(developers always use Python 3, and the remaining Python 2
usages are CI jobs that never reuse the objdir with Python 3),
We can centralize on a single default virtualenv.

I've called this "common" instead of "init" to clarify its
existing position as the virtualenv that's used by many different
commands. As we associate virtualenvs with requirement definitions,
it'll also make the file less confusing: it's a "common" requirement
definition as opposed to an "init" one.

Differential Revision: https://phabricator.services.mozilla.com/D115635
2021-05-27 15:18:35 +00:00
Mitchell Hentges
d41a0b2774 Bug 1712819: Avoid pip's "outdated" warning in virtualenvs r=ahal
Now, when running mach commands that invoke `pip`, it will no longer
inform the user that it needs an update.

We reach into `distutils` to determine the "site-packages" directory
pattern, then apply it to our virtualenv.

Differential Revision: https://phabricator.services.mozilla.com/D115940
2021-05-27 15:16:16 +00:00
Mitchell Hentges
f8ce33772c Bug 1712819: Fix VirtualenvManager not being expandable in debugger r=ahal
At least in PyCharm, expanding a `VirtualenvManager` instance means
resolving all the properties and fields of the instance.
However, if that property is doing non-trivial work, the debugger
wouldn't run that subprocess while we're stopped at a breakpoint.
So, the instance would sit there with the "Collecting data..." text.

Differential Revision: https://phabricator.services.mozilla.com/D115935
2021-05-27 15:16:16 +00:00
Tom Ritter
2b1afa9921 Bug 1712815 - Fix the 'remove a file from a moz.build file' logic r=jewilde
We were raising an exception as a result of a python error
which was masked by a catch statement that thought we didn't
know how to remove the file. But we did.

Fix the python error, and add an exception at the end of the
function to be raised and then caught when we really don't know
how to remove a file.

Differential Revision: https://phabricator.services.mozilla.com/D115936
2021-05-27 14:19:25 +00:00
Tom Ritter
8bcc518a54 Bug 1712953 - Add a frequency attribute to Updatebot tasks r=jewilde
Differential Revision: https://phabricator.services.mozilla.com/D116020
2021-05-26 20:36:13 +00:00
Alex Lopez
07cf832a8a Bug 1696251 - Turn all properties in MachCommandBase subclasses into methods. r=mhentges,perftest-reviewers,sparky
As a step towards moving mach commands outside of classes, this converts all
properties into methods so that they can later become top-level helper functions.

Differential Revision: https://phabricator.services.mozilla.com/D112196
2021-05-26 18:08:02 +00:00
Henrik Skupin
fa4fbb78b4 Bug 1695031 - Combine build flags --disable-marionette and --enable-cdp as --disable-webdriver. r=firefox-build-system-reviewers,Gijs,smaug,keeler,jdescottes,glandium
Differential Revision: https://phabricator.services.mozilla.com/D115583
2021-05-25 09:13:28 +00:00
Julien Cristau
1c67430110 Bug 1712150 - take branches into account in release-started email notification. r=releng-reviewers,aki
Instead of using the pushlog, grab the hg log using a revset to avoid
being confused by branches: we're looking for revisions that are
ancestors of the new revision but not of the previous tag, i.e.
"current % previous".

Differential Revision: https://phabricator.services.mozilla.com/D115624
2021-05-21 12:51:41 +00:00
Iulian Moraru
57426f492a Backed out changeset 67422384cb3e (bug 1712150) for causing release python failures on test_buglist_creator.py::test_get_bugs_in_changeset. CLOSED TREE 2021-05-21 14:22:14 +03:00
Julien Cristau
627434975b Bug 1712150 - take branches into account in release-started email notification. r=releng-reviewers,aki DONTBUILD
Instead of using the pushlog, grab the hg log using a revset to avoid
being confused by branches: we're looking for revisions that are
ancestors of the new revision but not of the previous tag, i.e.
"current % previous".

Differential Revision: https://phabricator.services.mozilla.com/D115624
2021-05-21 09:36:08 +00:00
Mike Hommey
c3d194a44b Bug 1712023 - Avoid displaying truncated symbols in check_binary_compat. r=firefox-build-system-reviewers,mhentges
Differential Revision: https://phabricator.services.mozilla.com/D115561
2021-05-20 22:32:10 +00:00
Ed Lee
851ab05cae Bug 1710955 - Change MR1 upgrade onboarding to Pin then Default then Theme screens r=fluent-reviewers,flod,pdahiya,k88hudson
Copy / migrate strings from onboarding/defaultBrowserNotification. Add updated ltr/rtl images and show them for pin and pin+default. Dynamically adjust steps and record telemetry to identify what was shown. Handle button actions based on string ids. Alias upgradeDialog to aboutwelcome for now. Avoid oddness with scrollbars by hiding horizontal scroll.

Differential Revision: https://phabricator.services.mozilla.com/D115142
2021-05-18 03:16:34 +00:00
Alex Lopez
455d9a088b Bug 1696251 - Pass MachCommandBase object as first argument for Mach Commands. r=mhentges,remote-protocol-reviewers,marionette-reviewers,webdriver-reviewers,perftest-reviewers
As an intermediate step to allow mach commands as standalone functions, the MachCommandBase
subclass instance that currently corresponds to self has to be made available as a separate
argument (named command_context).

Differential Revision: https://phabricator.services.mozilla.com/D109650
2021-05-17 16:15:58 +00:00
Gregory Mierzwinski
e8e2ef66d3 Bug 1710578 - Add missing python packages to search paths for mozperftest. r=perftest-reviewers,Bebe
This patch fixes a regression caused by bug 1698616 which made modifications to third-party/python packages.

Differential Revision: https://phabricator.services.mozilla.com/D114869
2021-05-12 14:22:28 +00:00
Mike Hommey
d9c58f174d Bug 1707829 - Go back to building http3server as a rust program. r=necko-reviewers,kershaw
This mostly revert bug 1616238.

Differential Revision: https://phabricator.services.mozilla.com/D113485
2021-05-07 20:22:56 +00:00
Sandor Molnar
93e6afe93a Backed out changeset bd2c9d3fa4b0 (bug 1707829) for causing xpc failures. CLOSED TREE 2021-05-07 01:58:43 +03:00
Mike Hommey
eae232636b Bug 1707829 - Go back to building http3server as a rust program. r=necko-reviewers,kershaw
This mostly revert bug 1616238.

Differential Revision: https://phabricator.services.mozilla.com/D113485
2021-05-06 21:37:01 +00:00
Andi-Bogdan Postelnicu
d6010d98f2 Bug 1702172 - collect if shell is opened via vscode and if it's a remote connection through ssh. r=firefox-build-system-reviewers,mhentges
Added two fields:

1. Collect if the current shell is opened via vscode.
2. Connect if it's a remote ssh connection n the current shell.

Differential Revision: https://phabricator.services.mozilla.com/D114323
2021-05-05 14:56:32 +00:00
Marco Bonardo
f1454c9308 Bug 1706334 - Improving the accessible text on Add Engine items. r=adw,Jamie,fluent-reviewers,flod
Differential Revision: https://phabricator.services.mozilla.com/D113146
2021-05-04 19:52:52 +00:00
Narcis Beleuzu
6371054f62 Bug 1709401 - Fix lint failure a=lint-fix . CLOSED TREE 2021-05-04 20:59:02 +03:00
Tom Ritter
b5376d204b Bug 1709401 - Shim the python 3.8 API get_source_segment for ./mach vendor r=jewilde
Differential Revision: https://phabricator.services.mozilla.com/D114254
2021-05-04 17:33:17 +00:00
Mitchell Hentges
e170f04fe1 Bug 1707546: Support git worktrees when finding ".arcconfig" r=firefox-build-system-reviewers,andi,tcampbell,glandium
A `.arcconfig` may exist in the `.git` directory. Within worktrees,
the `.git` folder isn't at `<topsrcdir>/.git`, so we have to resolve it.

Differential Revision: https://phabricator.services.mozilla.com/D113320
2021-05-04 15:06:42 +00:00
Steve Fink
c0568910c7 Bug 1708048 - Ensure that bootstrap-generated mozconfigs end with newline r=mhentges
Differential Revision: https://phabricator.services.mozilla.com/D113877
2021-05-03 20:39:09 +00:00
Steve Fink
852ab33676 Bug 1708048 - Implement mach bootstrap --app=js for SpiderMonkey shell development r=firefox-build-system-reviewers,mhentges
Differential Revision: https://phabricator.services.mozilla.com/D113598
2021-05-03 20:39:08 +00:00
Mike Hommey
f2dcdd5390 Bug 1708592 - Get tools from MOZ_FETCHES_DIR on automation. r=firefox-build-system-reviewers,mhentges
While here, remove the unused clippyProcess.config. And because we need
to manipulate `$PATH` before running `cargo`, use the same wrapping
function in `get_clippy_version`.

Differential Revision: https://phabricator.services.mozilla.com/D113902
2021-04-30 21:09:11 +00:00