Commit Graph

58 Commits

Author SHA1 Message Date
Ricky Stewart
02a7b4ebdf Bug 1654103: Standardize on Black for Python code in mozilla-central.
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.

To produce this patch I did all of the following:

1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.

2. Run ./mach lint --linter black --fix

3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.

4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.

5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).

# ignore-this-changeset

Differential Revision: https://phabricator.services.mozilla.com/D94045
2020-10-26 18:34:53 +00:00
Bogdan Tara
da1098d4aa Backed out 10 changesets (bug 1654103, bug 1672023, bug 1518999) for PanZoomControllerTest.touchEventForResult gv-junit failures CLOSED TREE
Backed out changeset ff3fb0b4a512 (bug 1672023)
Backed out changeset e7834b600201 (bug 1654103)
Backed out changeset 807893ca8069 (bug 1518999)
Backed out changeset 13e6b92440e9 (bug 1518999)
Backed out changeset 8b2ac5a6c98a (bug 1518999)
Backed out changeset 575748295752 (bug 1518999)
Backed out changeset 65f07ce7b39b (bug 1518999)
Backed out changeset 4bb80556158d (bug 1518999)
Backed out changeset 8ac8461d7bd7 (bug 1518999)
Backed out changeset e8ba13ee17f5 (bug 1518999)
2020-10-24 03:36:18 +03:00
Ricky Stewart
c0cea3b0fa Bug 1654103: Standardize on Black for Python code in mozilla-central. r=remote-protocol-reviewers,marionette-reviewers,webdriver-reviewers,perftest-reviewers,devtools-backward-compat-reviewers,jgilbert,preferences-reviewers,sylvestre,maja_zf,webcompat-reviewers,denschub,ntim,whimboo,sparky
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.

To produce this patch I did all of the following:

1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.

2. Run ./mach lint --linter black --fix

3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.

4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.

5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).

# ignore-this-changeset

Differential Revision: https://phabricator.services.mozilla.com/D94045
2020-10-23 20:40:42 +00:00
Dorel Luca
1ff59cb7a3 Backed out changeset 7558c8821a07 (bug 1654103) for multiple failures. CLOSED TREE 2020-10-22 03:51:06 +03:00
Ricky Stewart
50762dacab Bug 1654103: Standardize on Black for Python code in mozilla-central. r=remote-protocol-reviewers,marionette-reviewers,webdriver-reviewers,perftest-reviewers,devtools-backward-compat-reviewers,jgilbert,preferences-reviewers,sylvestre,maja_zf,webcompat-reviewers,denschub,ntim,whimboo,sparky
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.

To produce this patch I did all of the following:

1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.

2. Run ./mach lint --linter black --fix

3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.

4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).

# ignore-this-changeset

Differential Revision: https://phabricator.services.mozilla.com/D94045
2020-10-21 21:27:27 +00:00
Mike Hommey
54f6141287 Bug 1671424 - Move configure execution from client.mk to mach configure. r=firefox-build-system-reviewers,rstewart
`mach configure` currently runs the equivalent to `make -f client.mk`.
This is history, and essentially does the following:
- Create `configure` and `js/src/configure` from `configure.in` and
`js/src/configure.in` respectively.
- Create the objdir.
- Run `configure` from the objdir.

The `configure` script is, nowadays, only really used as a means to set
OLD_CONFIGURE (and also for people who want to run `configure`,
literally, as in the `configure; make` workflow). `mach configure`
actually doesn't need it. Neither does recursing into `js/src` require
`js/src/configure`, since bug 1520340 (and now as of bug 1669633, we
don't even recurse).

Because configure.py can actually derive OLD_CONFIGURE on its own
(except for `js/src/configure`, but `mach configure` doesn't run that),
we don't really need `configure` for `mach configure`.

So all in all, we're at a point in history where it's straightforward to
just initiate configure.py from mach configure, so we just do that.

And in the hypothetical case where the `mach configure` code is somehow
running in python2, we get the mach virtualenv python3 and use it to
execute `configure.py`.

Differential Revision: https://phabricator.services.mozilla.com/D93741
2020-10-20 20:41:52 +00:00
Narcis Beleuzu
9252175982 Backed out changeset e34634758f51 (bug 1671424) for bustages on configure.py . CLOSED TREE 2020-10-20 00:16:22 +03:00
Mike Hommey
fcd16177c6 Bug 1671424 - Move configure execution from client.mk to mach configure. r=firefox-build-system-reviewers,rstewart
`mach configure` currently runs the equivalent to `make -f client.mk`.
This is history, and essentially does the following:
- Create `configure` and `js/src/configure` from `configure.in` and
`js/src/configure.in` respectively.
- Create the objdir.
- Run `configure` from the objdir.

The `configure` script is, nowadays, only really used as a means to set
OLD_CONFIGURE (and also for people who want to run `configure`,
literally, as in the `configure; make` workflow). `mach configure`
actually doesn't need it. Neither does recursing into `js/src` require
`js/src/configure`, since bug 1520340 (and now as of bug 1669633, we
don't even recurse).

Because configure.py can actually derive OLD_CONFIGURE on its own
(except for `js/src/configure`, but `mach configure` doesn't run that),
we don't really need `configure` for `mach configure`.

So all in all, we're at a point in history where it's straightforward to
just initiate configure.py from mach configure, so we just do that.

And in the hypothetical case where the `mach configure` code is somehow
running in python2, we get the mach virtualenv python3 and use it to
execute `configure.py`.

Differential Revision: https://phabricator.services.mozilla.com/D93741
2020-10-19 16:24:34 +00:00
Mike Hommey
9ebbe2c5bc Bug 1669633 - Don't recurse into js/src for the python-part of configure. r=firefox-build-system-reviewers,rstewart
Instead, we now run js/src/old-configure from the top-level configure
after having run old-configure and extracted a few variables to inherit
from it.

Because we're now running from the top-level, $_objdir is always the
top-level objdir, which simplifies some things. The topobjdir in
js/src/config.status, however, needs to stay in js/src because of the
build frontend expecting it there.

When running js/src/old-configure, we used to need some special
treatment for a large number of variables for historic reasons, where
we'd take values from the assigned values before running old-configure
for some, or from AC_SUBSTs after running old-configure.

Now that both old-configure and js/src/old-configure get the same
assignments from old-configure.vars, we don't need anything special for
the former. And only a few remaining variables still need manual work
for the latter.

One notable difference, though, is that the new code doesn't try to
avoid running js subconfigure, which added complexity, and was actually
error-prone.

Differential Revision: https://phabricator.services.mozilla.com/D92725
2020-10-07 21:13:19 +00:00
Mike Hommey
e555ff8024 Bug 1669633 - Normalize sanitized_config earlier. r=firefox-build-system-reviewers,rstewart
It wasn't normalized early enough to end up in config.status. It is
surprising this hasn't caused problems earlier.

Differential Revision: https://phabricator.services.mozilla.com/D92719
2020-10-07 17:58:21 +00:00
Mike Hommey
1b0b2b298e Bug 1669633 - Don't set AC_SUBSTs and AC_DEFINEs from old-configure directly. r=firefox-build-system-reviewers,rstewart
In order to be able to run both old-configure and js/src/old-configure
from the same python configure run, we need to stop setting the items
set by old-configure into the global sandbox config, and instead store
them to be later handled by configure.py.

Differential Revision: https://phabricator.services.mozilla.com/D92718
2020-10-07 17:59:51 +00:00
Mike Hommey
b297cc22e4 Bug 1664077 - Remove _NON_GLOBAL_ACDEFINES. r=firefox-build-system-reviewers,mhentges,rstewart
I think they're remnants from the past that we don't really need anymore.
And they're making things more complicated for some pending work of mine.

Differential Revision: https://phabricator.services.mozilla.com/D89687
2020-09-15 14:25:14 +00:00
Razvan Maries
f46af2adfc Backed out changeset ee6dbeaf8852 (bug 1664077) for build bustages on gfxFontUtils.h. CLOSED TREE 2020-09-15 04:02:01 +03:00
Mike Hommey
3fd47263f8 Bug 1664077 - Remove _NON_GLOBAL_ACDEFINES. r=firefox-build-system-reviewers,mhentges,rstewart
I think they're remnants from the past that we don't really need anymore.
And they're making things more complicated for some pending work of mine.

Differential Revision: https://phabricator.services.mozilla.com/D89687
2020-09-14 16:31:31 +00:00
Ricky Stewart
6983088e37 Bug 1649901 - Improve reproducibility of config_status_deps.in files r=froydnj
Differential Revision: https://phabricator.services.mozilla.com/D81925
2020-07-01 22:55:52 +00:00
Mitchell Hentges
ed90a04ad1 Bug 1632874: |mach bootstrap| should write mozconfig file if it doesn't exist r=rstewart
Rather than always printing instructions at the end of the bootstrap phase, we will now create a mozconfig
file if one doesn't exist and there's configuration to be written.

Differential Revision: https://phabricator.services.mozilla.com/D78417
2020-06-08 19:36:56 +00:00
Mike Hommey
7dff6bcc31 Bug 1631633 - Generate config.status deterministically. r=rstewart
Differential Revision: https://phabricator.services.mozilla.com/D71675
2020-04-21 21:32:30 +00:00
Mike Hommey
f805480ab5 Bug 1627163 - Switch python configure to python 3. r=firefox-build-system-reviewers,rstewart
This also does a few remaining python 2 incompatible changes to
.configure files.

Differential Revision: https://phabricator.services.mozilla.com/D69538

--HG--
extra : moz-landing-system : lando
2020-04-07 18:31:56 +00:00
Mike Hommey
21a4f4b925 Bug 1627163 - Do not rely on write_indented_repr serializing list-like items as lists. r=firefox-build-system-reviewers,rstewart
Many values we get out from configure are of types that look like
lists/tuples, that write_indented_repr will serialize as lists... but
only in python 2, because the alternative implementation for python 3
is not doing that. So sanitize first.

Differential Revision: https://phabricator.services.mozilla.com/D69949

--HG--
extra : moz-landing-system : lando
2020-04-07 22:08:48 +00:00
Ricky Stewart
e52b331537 Bug 1623982 - Fix performance of indented_repr in Python 3 r=ahal
Differential Revision: https://phabricator.services.mozilla.com/D67688

--HG--
extra : moz-landing-system : lando
2020-03-25 19:40:06 +00:00
Anthony Ramine
b3cefc98c5 Bug 1624190 - Try importing from collections.abc first; r=rstewart
Differential Revision: https://phabricator.services.mozilla.com/D67773

--HG--
extra : moz-landing-system : lando
2020-03-23 16:33:07 +00:00
Ricky Stewart
bd01350f97 Bug 1619788 - Ensure files generated by ./mach configure don't have CRLF line endings, even under Python 3 r=glandium
Differential Revision: https://phabricator.services.mozilla.com/D65354

--HG--
extra : moz-landing-system : lando
2020-03-06 02:19:21 +00:00
Mike Shal
3ce3aba1c7 Bug 1593948 - Touch clobber in configure.py instead of clobber.py; r=firefox-build-system-reviewers,chmanchester
If configure is invoked manually, clobber.py is bypassed and the CLOBBER
file doesn't get touched. The clobber check in Makefile.in gets
triggered causing the build to stop.

Moving the objdir/CLOBBER file creation into configure.py should cause
it to be created regardless of how configure is invoked.

Depends on D53290

Differential Revision: https://phabricator.services.mozilla.com/D53291

--HG--
extra : moz-landing-system : lando
2019-11-19 21:49:13 +00:00
Mike Hommey
ad930b523d Bug 844509 - Always encode config.status as utf-8. r=nalexander
Now that there is no bytes strings in it, we don't need to store
config.status in the system encoding to keep those valid. Moreover, the
system encoding is lossy, which utf-8 is not.

Differential Revision: https://phabricator.services.mozilla.com/D42731

--HG--
extra : moz-landing-system : lando
2019-08-21 21:26:32 +00:00
Mike Hommey
621ee253e1 Bug 844509 - Don't encode the configuration that configure passes to config_status. r=nalexander
With bug 1575135, we ensure nothing gets out of the configure sandbox
as a bytes string. We can thus now avoid the encode() pass in
configure.py. We still need, however, to normalize the configuration
so that it doesn't contain unexpected types, and conformning to what
indented_repr does to the configuration in config.status.

While here, convert some obj.iteritems() to six.iteritems(obj).
And remove the now unused encode function.

Differential Revision: https://phabricator.services.mozilla.com/D42630

--HG--
extra : moz-landing-system : lando
2019-08-20 22:22:14 +00:00
Mike Hommey
d097494bcc Bug 1575135 - Make configure complain on non-unicode strings. r=nalexander
Make it a hard error when the sandbox returns non-unicode strings.
This should help quickly catch any remaining non-unicode string that
are not caught by automation.

Differential Revision: https://phabricator.services.mozilla.com/D42607

--HG--
extra : moz-landing-system : lando
2019-08-20 21:31:34 +00:00
Mike Hommey
cc52fc2e1a Bug 1575135 - Define whether the system encoding is mbcs or utf-8 once. r=nalexander
Differential Revision: https://phabricator.services.mozilla.com/D42601

--HG--
extra : moz-landing-system : lando
2019-08-20 16:40:24 +00:00
Justin Wood
7a37d64346 Bug 1559975 - Make configure.py lint with 'py2' and 'py3' r=catlee
Depends on D36090

Differential Revision: https://phabricator.services.mozilla.com/D36091

--HG--
extra : moz-landing-system : lando
2019-07-08 17:34:09 +00:00
Justin Wood
38c14ef720 Bug 1547730 - Add six to path when running configure. r=glandium
This is necessary for any attempt at @import('six') or doing any similar py2+py3 work
in the configure sandbox.

Differential Revision: https://phabricator.services.mozilla.com/D29604

--HG--
extra : moz-landing-system : lando
2019-05-28 14:23:35 +00:00
Mike Hommey
2ead86b7c1 Bug 1528241 - Add a trace mode to python configure that logs internal values. r=chmanchester
Differential Revision: https://phabricator.services.mozilla.com/D19940
2019-02-20 07:46:22 +09:00
Mike Hommey
889bc79f27 Bug 1520394 - Don't invoke js subconfigure as a separate process. r=nalexander
Since js configure is also python configure, we can actually create
a ConfigureSandbox directly, with the right environment and arguments.

Depends on D16668

Differential Revision: https://phabricator.services.mozilla.com/D16669

--HG--
extra : moz-landing-system : lando
2019-01-16 23:42:12 +00:00
Chris Manchester
636b4fc4c0 Bug 1461836 - Write out complete configure dependencies from configure for consumption by make and non-make backends. r=mshal
MozReview-Commit-ID: 792seCZ2rs1

--HG--
extra : rebase_source : e6450cd2947e2fd55085a2af469535421bc07bfb
2018-05-21 14:01:50 -07:00
Mike Shal
642a32a948 Bug 1461795 - Use FileAvoidWrite when writing mozinfo.json; r=chmanchester
Since MozbuildObject.from_environment() reads from mozinfo.json, tup
picks up that file as a dependency for anything that imports buildconfig
(eg: all generated files). Using FileAvoidWrite when creating
mozinfo.json will help avoid unnecessary work after re-running configure
in the tup backend.

MozReview-Commit-ID: EEOPQYJA1MV

--HG--
extra : rebase_source : 136a0579090776dc55ea5cee870574f13cc27c58
2018-05-14 17:30:21 -04:00
Gregory Szorc
0bd6f6c942 Bug 1412460 - Don't write make variables in configure.d; r=mshal
So the file can be more easily consumed without this variable defined.

MozReview-Commit-ID: DF3ASwx4SZP

--HG--
extra : rebase_source : 7d760a71c8f0fa389c2f31f1b383a1f298220545
2017-11-01 15:02:11 -07:00
Sylvestre Ledru
358507fde2 Bug 1406650 - Make build/*.py and a few other files flake8 compatible and add them to the list of files to check r=chmanchester
MozReview-Commit-ID: icmFJtbWdN

--HG--
extra : rebase_source : 7b2ef783822bcad7f955bad8cbd6cea6df588f92
2017-10-07 16:45:22 +02:00
Mike Shal
10d4a372fd Bug 1402012 - Create config.statusd directory; r=glandium
The config.statusd directory is created alongside config.status, which
contains the same information but is split across many files instead of
all in a single file. This allows the build system to track dependencies
on individual configure values.

MozReview-Commit-ID: 2DbwKCJuNSX

--HG--
extra : rebase_source : 8b6257fd9c75cff3e4b6513d69048c0e3fdda5f4
2017-08-18 10:41:50 -04:00
Chris Manchester
2d2292267f Bug 1316140 - Allow use of multiprocessing from config.status on windows. r=mshal
This generalizes the monkey-patching of the main module added
in bug 914563 to allow multiprocessing to be used from config.status
(which triggers the same bug as when it's used with `mach` as main).

MozReview-Commit-ID: AdOdpKzmbsp
2016-12-21 16:28:28 -08:00
Chris Manchester
eb8660f21f Bug 1317778 - Emit a depfile with python configure dependencies so Make will know when to re-run configure. r=glandium
MozReview-Commit-ID: AuTHadY7KqO
2016-12-02 10:05:57 -08:00
Mike Hommey
f97add6317 Bug 1296503 - Switch config.status to unicode literals. r=ted
Ironically, the first thing we do with those unicode literals is convert
them to byte strings because the build backends don't like them yet.
2016-08-24 07:32:52 +09:00
Mike Hommey
ce060d48bf Bug 1257049 - Stop spawning a separate process for config.status from configure.py. r=gps
--HG--
extra : rebase_source : c3a76c32eaca7a9a192e239ac143174d599fb28c
2016-08-19 11:11:57 +09:00
Elias Probst
fa561f244a Bug 1293234 - Use octal notation for permission modes in the build system; r=gps
As of Python 3, decimal notations of octal values for permission modes
are no longer permitted and will result in a `SyntaxError` exception
(`invalid token`).

Using the proper octal notation which is also Python 2.7 compatible will
fix this issue.

--HG--
extra : rebase_source : 2e897c51f04ad0ee69071f84b98df224f3af72d3
2016-08-08 13:45:17 +02:00
Gregory Szorc
fedd3f734d Bug 1285860 - Add python/mozbuild at beginning of sys.path; r=glandium
We want local source directory paths to take precedence over system
paths so system installed packages don't get used.

MozReview-Commit-ID: FIZ0RxqZj78

--HG--
extra : rebase_source : e8d484ea0c8a0bdb1df10375317f6b2c349b9162
2016-07-11 10:42:11 -07:00
Mike Hommey
9607db6e29 Bug 1283052 - Remove some of the magic around mozconfig detection. r=gps
The mozconfig detection logic has bitten us on many occasions in the
past. The following changes are made to tentatively improve the
situation:

- The API is modified such that autodetection of the mozconfig has
  to be a conscious decision made by the caller, and not triggered
  any time there is no mozconfig given, which could be a conscious
  decision of the opposite.

- mozinfo.json now stores the actual mozconfig (or lack thereof) used
  during configure.

--HG--
extra : rebase_source : c7a632afd414f25daf7bbe7e1a66c3736c26e039
2016-07-08 08:43:17 +09:00
Mike Hommey
8a3b88f341 Bug 1278415 - Also write a mozinfo for js standalone builds. r=gps
Historically, a mozinfo for js standalone build has not been necessary,
but with the move towards a) having things work with mach and b)
buildconfig using the MozbuildObject.from_environment in next patch,
mozinfo becomes necessary (it's required for
MozbuildObject.from_environment to find the directory is an objdir).

Interestingly, hazard builds do both a js standalone build and a desktop
Firefox build at the same time, both of which are done with MOZCONFIG
set in the environment... with the Firefox mozconfig. The result of now
writing a mozinfo for js standalone builds is that in that case, they
end up with a reference to the mozconfig, which the build system then
reads, and finds a MOZ_OBJDIR, which then doesn't match the js
standalone build objdir. So for those builds, reset MOZCONFIG.
2016-06-16 11:43:23 +09:00
Mike Hommey
c4eadb8455 Bug 1258618 - Serialize substs/configs and defines bools as '1' or '' in config.status. r=nalexander
This allows to use True and False as values given to
set_config/set_define in moz.configure files, while postponing having to
deal with the long tail of things depending on the values from substs
and defines.

Ideally, everything would handle the bools just fine, but there are too
many things involved to deal with this right now: scripts using
buildconfig.{substs,defines}, scripts using ConfigEnvironment (e.g.
process_define_files.py), ConfigEnvironment itself, etc.
2016-03-23 09:24:47 +09:00
Mike Hommey
cec6eb6fbb Bug 1257468 - Replace tests on BUILDING_JS with tests on MOZ_BUILD_APP==js. r=nalexander
Now that MOZ_BUILD_APP is set to js when building js/src, we can
distinguish those builds with MOZ_BUILD_APP==js instead of BUILDING_JS.
Consequently, remove BUILDING_JS.
2016-03-22 11:09:42 +09:00
Mike Hommey
0ea268dc18 Backout changesets f5090987b7c6, 48a94777837d and a63437a74f7a (bug 1257468) for SM bustage on a CLOSED TREE 2016-03-22 08:27:28 +09:00
Mike Hommey
37d7b30422 Bug 1257468 - Replace tests on BUILDING_JS with tests on MOZ_BUILD_APP==js. r=nalexander
Now that MOZ_BUILD_APP is set to js when building js/src, we can
distinguish those builds with MOZ_BUILD_APP==js instead of BUILDING_JS.
Consequently, remove BUILDING_JS.
2016-03-22 08:05:00 +09:00
Mike Hommey
c15884c85a Backout changeset 37f3e53ede1f (bug 1257049) because of bustage. 2016-03-16 11:22:43 +09:00
Mike Hommey
2891514755 Bug 1257049 - Stop spawning a separate process for config.status from configure.py. r=gps 2016-03-16 10:55:10 +09:00