The mach driver will now run all misspelled commands with Python 3. That means
we can't automatically execute the suggested command anymore, as it may need to
run against Python 2.
Ideally we could figure out a way to check the command against the 'mach'
whitelist, but until then, let's just disable automatic execution. Worst case
scenario we can turn it back on after the migration has finished.
Differential Revision: https://phabricator.services.mozilla.com/D44282
--HG--
extra : moz-landing-system : lando
Some commands use external argument parsers, so invoking |mach help <command>| will import
external modules (which may only be Python 2 compatible).
This makes sure that we detect the actual subcommand we're generating help for
and use the proper Python.
A much simpler solution would have been to run |mach help| with Python 2 all
the time. However, as we convert things to Python 3 this would have meant that
Python 3 only code would blow up. This would have forced us to continue
supporting Python 2, even for Python 3-only commands.
Differential Revision: https://phabricator.services.mozilla.com/D43989
--HG--
extra : moz-landing-system : lando
This fixes an edge case where a task needs to invoke the TestManifest backend from a source dir that doesn't have an objdir created yet.
Depends on D43513
Differential Revision: https://phabricator.services.mozilla.com/D43514
--HG--
extra : moz-landing-system : lando
Changes:
- remove binary encoding when the subdirectories are being collected
- do not open file in binary mode when reading from `telemetry_client_id.json`
Differential Revision: https://phabricator.services.mozilla.com/D44057
--HG--
extra : moz-landing-system : lando
MOZ_FULL_PRODUCT_VERSION is only used here, while tools/update-packaging/make_full_update.sh expects MOZ_PRODUCT_VERSION.
Depends on D44089
Differential Revision: https://phabricator.services.mozilla.com/D44090
--HG--
extra : moz-landing-system : lando
This removes these functions: bind, getaddrinfo, recvfrom, sendto, setsockopt,
socket from the check_networking test to allow for their use by the Rust mDNS
responder.
Differential Revision: https://phabricator.services.mozilla.com/D38488
--HG--
extra : moz-landing-system : lando
This removes these functions: bind, getaddrinfo, recvfrom, sendto, setsockopt,
socket from the check_networking test to allow for their use by the Rust mDNS
responder.
Differential Revision: https://phabricator.services.mozilla.com/D38488
--HG--
extra : moz-landing-system : lando
Fix some library calls and syntax that are required to support Python 3.
Depends on D39364
Differential Revision: https://phabricator.services.mozilla.com/D39365
--HG--
extra : moz-landing-system : lando
Make bootstrap.py compatible with both Python 3.5+ and Python 2.7. Remove
Python 2.6 support as Python 2.6 is no longer supported in the Firefox source
tree.
Differential Revision: https://phabricator.services.mozilla.com/D39364
--HG--
extra : moz-landing-system : lando
Add an option to run bootstrap.py with '--debug'. This will print full
tracebacks if the bootstrap process encounters an uncaught exception. It should
be useful for developers who are working on the bootstrap code as well as users
when writing bug reports.
Depends on D39362
Differential Revision: https://phabricator.services.mozilla.com/D39363
--HG--
extra : moz-landing-system : lando
Support installing the android development toolchain with Python 3 as well as
Python 2.7.
Depends on D39361
Differential Revision: https://phabricator.services.mozilla.com/D39362
--HG--
extra : moz-landing-system : lando
Add support for Python 3 and Python 2.7 to the Debian-based linux distro
bootstrap routines.
Differential Revision: https://phabricator.services.mozilla.com/D39361
--HG--
extra : moz-landing-system : lando
Add support for both Python 3 and Python 2.7 to the mozboot.base and
mozboot.bootstrap modules. Remove legacy Python 2.6 code or mark it for later
removal.
Depends on D39359
Differential Revision: https://phabricator.services.mozilla.com/D39360
--HG--
extra : moz-landing-system : lando
Add unicode_literals to all mozboot module __future__ statements to support
running the modules under Python 3. Remove comments about unicode_literals and
Python 2.6 support as Python 2.6 is no longer supported in tree.
Differential Revision: https://phabricator.services.mozilla.com/D39359
--HG--
extra : moz-landing-system : lando
Credit: mars for making the shell POSIX compliant
This embeds a blacklist of every mach command that needs to run with Python 2
directly in the mach driver itself. Initially this is every mach command. We
then use a bit of shell to determine whether the command being run needs Python
2 or 3.
While this approach may seem a bit hacky, it has several benefits:
1. No need to add complex machinery in mach's registration code.
2. No need to spawn two separate Python interpreters in the event a different
Python from the original interpreter is needed.
3. Perf impact is negligible.
4. New commands are Python 3 by default.
It is also only a temporary hack. Once all commands are running with Python 3,
we can revert back to the original mach driver.
Differential Revision: https://phabricator.services.mozilla.com/D36103
--HG--
extra : moz-landing-system : lando
Use of 'BaseException.message` was deprecated in Python 2.6 (and removed in
Python 3). We should rely on the exception's '__str__' instead.
We also need to ensure the mozconfig subprocess' output is text when formatting
it into the message with Py3.
Differential Revision: https://phabricator.services.mozilla.com/D42843
--HG--
extra : moz-landing-system : lando
Maybe back when .cargo/config.in was added, the directory indicated for
vendored crates needed to be absolute. That is at least not the case
with the current supported versions of rust.
The current setup has a few caveats:
- .cargo/config.in has shown to become stale (it currently contains
multiple unused entries)
- non-gecko build tasks have to generate a .cargo/config on their own if
they want to use vendored crates
- in turn, non-gecko build tasks that don't, may unknowingly get their
dependencies from crates.io (see the recent attempt at moving
geckodriver builds to a separate task).
By checking in a .cargo/config file, we can alleviate the last two, but
that comes at the price of `cargo update` not wanting to act when
.cargo/config exists, because of the source replacement configuration.
But rust vendor gently generates a suitable configuration on its own, so
we can use that to generate a .cargo/config automatically. Which
addresses the first caveat of the current setup. That leaves us with
`cargo update` not working out of the box, but that just requires people
running it to manually remove .cargo/config first. Which is arguably
what rust wants you to do in the first place. It's kind of incidental
that we started with a .cargo/config.in rather than .cargo/config.
Now, while a simple .cargo/config works, that's not enough for the case
where the objdir doesn't live inside the source directory. In that case
cargo looks for the configuration from the objdir, and fails to find it.
So we still need a .cargo/config.in, which we generate with a little
trick.
Differential Revision: https://phabricator.services.mozilla.com/D43012
--HG--
rename : .cargo/config.in => .cargo/config
extra : moz-landing-system : lando
We'll require preprocessing that configure subst files don't allow in
the next change, so prepare for that.
Differential Revision: https://phabricator.services.mozilla.com/D43011
--HG--
extra : moz-landing-system : lando
Straight-forward addition of the --no-install option for reftest, just like
the existing option for mochitest. For gtest, I alos noticed the mach command
help needed some cleanup.
Differential Revision: https://phabricator.services.mozilla.com/D43299
--HG--
extra : moz-landing-system : lando
This time, existing config.status trying to import it will throw an exception
that will trigger a re-configure.
Differential Revision: https://phabricator.services.mozilla.com/D43215
--HG--
extra : moz-landing-system : lando
As seen in bug 1575774/1575959, things can go badly when config.status
doesn't match expectations, especially when most mach commands try to
read it, starting with mach clobber, which is supposed to be the one
commant to get away from most problems.
The idea here is to throw a recognizable exception so that callers can
react accordingly. While not obvious from the patch, the result is that
running e.g. mach build with a broken config.status will automatically
run configure (because the relevant caller actually handles the rethrown
exception that way).
There are other calls to from_config_status in the mozbuild reader, but
that can't be reached before MozbuildObject.config_environment.
Differential Revision: https://phabricator.services.mozilla.com/D43214
--HG--
extra : moz-landing-system : lando
Older config.status laying around in old trees are read by mach whenever
it runs, including when running mach clobber. Those files still try to
include mozbuild.util.encode, which is not here anymore. So we restore
the function for now to unbreak those.
Differential Revision: https://phabricator.services.mozilla.com//D43132
--HG--
extra : histedit_source : 3eb10cc8e18cc9094887061b00705afb6e705b54
Note that cargo vendor always uses the native version even when
cargo-vendor was already installed, so there is no concern wrt that.
Differential Revision: https://phabricator.services.mozilla.com/D43007
--HG--
extra : moz-landing-system : lando
Bug 1522788 changed how MozbuildObject gets a value for @CONFIG_GUESS@
in mozconfigs, and it's not entirely reliable to manually call
config.guess in test_base. As a matter of fact, in some cases,
config.guess is not even invoked by MozbuildObject (like, on Windows).
Plus, the config.guess invocation in the test currently doesn't work on
Windows on automation, for some reason.
Differential Revision: https://phabricator.services.mozilla.com/D42755
--HG--
extra : moz-landing-system : lando
Python 3.7 changed what it escapes with re.escape. Adapt mozpath.match
to account for this.
Differential Revision: https://phabricator.services.mozilla.com/D42750
--HG--
extra : moz-landing-system : lando
Now that the configuration comes in without bytes strings, there is no
need to convert it anymore.
Differential Revision: https://phabricator.services.mozilla.com/D42631
--HG--
extra : moz-landing-system : lando
With bug 1575135, we ensure nothing gets out of the configure sandbox
as a bytes string. We can thus now avoid the encode() pass in
configure.py. We still need, however, to normalize the configuration
so that it doesn't contain unexpected types, and conformning to what
indented_repr does to the configuration in config.status.
While here, convert some obj.iteritems() to six.iteritems(obj).
And remove the now unused encode function.
Differential Revision: https://phabricator.services.mozilla.com/D42630
--HG--
extra : moz-landing-system : lando
The configure sandbox has wrapped subprocess methods to add its own
encoded environment if none is provided, since bug 1520394. It only
makes sense that it normalizes the environment that comes in too,
avoiding caller in the configure sandbox to have to do it themselves.
OTOH, and while we're here, none of get_cmd_output, old_configure or the
sandbox were actually using the right encoding for this conversion, so
fix the configure sandbox to use the right one, and make it stop using
encode(), which does deep recursion that is not necessary here, and that
I'm trying to remove entirely.
Also while here, remove an unused import of encode().
Differential Revision: https://phabricator.services.mozilla.com/D42608
--HG--
extra : moz-landing-system : lando
