This removes these functions: bind, getaddrinfo, recvfrom, sendto, setsockopt,
socket from the check_networking test to allow for their use by the Rust mDNS
responder.
Differential Revision: https://phabricator.services.mozilla.com/D38488
--HG--
extra : moz-landing-system : lando
Fix some library calls and syntax that are required to support Python 3.
Depends on D39364
Differential Revision: https://phabricator.services.mozilla.com/D39365
--HG--
extra : moz-landing-system : lando
Make bootstrap.py compatible with both Python 3.5+ and Python 2.7. Remove
Python 2.6 support as Python 2.6 is no longer supported in the Firefox source
tree.
Differential Revision: https://phabricator.services.mozilla.com/D39364
--HG--
extra : moz-landing-system : lando
Add an option to run bootstrap.py with '--debug'. This will print full
tracebacks if the bootstrap process encounters an uncaught exception. It should
be useful for developers who are working on the bootstrap code as well as users
when writing bug reports.
Depends on D39362
Differential Revision: https://phabricator.services.mozilla.com/D39363
--HG--
extra : moz-landing-system : lando
Support installing the android development toolchain with Python 3 as well as
Python 2.7.
Depends on D39361
Differential Revision: https://phabricator.services.mozilla.com/D39362
--HG--
extra : moz-landing-system : lando
Add support for Python 3 and Python 2.7 to the Debian-based linux distro
bootstrap routines.
Differential Revision: https://phabricator.services.mozilla.com/D39361
--HG--
extra : moz-landing-system : lando
Add support for both Python 3 and Python 2.7 to the mozboot.base and
mozboot.bootstrap modules. Remove legacy Python 2.6 code or mark it for later
removal.
Depends on D39359
Differential Revision: https://phabricator.services.mozilla.com/D39360
--HG--
extra : moz-landing-system : lando
Add unicode_literals to all mozboot module __future__ statements to support
running the modules under Python 3. Remove comments about unicode_literals and
Python 2.6 support as Python 2.6 is no longer supported in tree.
Differential Revision: https://phabricator.services.mozilla.com/D39359
--HG--
extra : moz-landing-system : lando
Credit: mars for making the shell POSIX compliant
This embeds a blacklist of every mach command that needs to run with Python 2
directly in the mach driver itself. Initially this is every mach command. We
then use a bit of shell to determine whether the command being run needs Python
2 or 3.
While this approach may seem a bit hacky, it has several benefits:
1. No need to add complex machinery in mach's registration code.
2. No need to spawn two separate Python interpreters in the event a different
Python from the original interpreter is needed.
3. Perf impact is negligible.
4. New commands are Python 3 by default.
It is also only a temporary hack. Once all commands are running with Python 3,
we can revert back to the original mach driver.
Differential Revision: https://phabricator.services.mozilla.com/D36103
--HG--
extra : moz-landing-system : lando
Use of 'BaseException.message` was deprecated in Python 2.6 (and removed in
Python 3). We should rely on the exception's '__str__' instead.
We also need to ensure the mozconfig subprocess' output is text when formatting
it into the message with Py3.
Differential Revision: https://phabricator.services.mozilla.com/D42843
--HG--
extra : moz-landing-system : lando
Maybe back when .cargo/config.in was added, the directory indicated for
vendored crates needed to be absolute. That is at least not the case
with the current supported versions of rust.
The current setup has a few caveats:
- .cargo/config.in has shown to become stale (it currently contains
multiple unused entries)
- non-gecko build tasks have to generate a .cargo/config on their own if
they want to use vendored crates
- in turn, non-gecko build tasks that don't, may unknowingly get their
dependencies from crates.io (see the recent attempt at moving
geckodriver builds to a separate task).
By checking in a .cargo/config file, we can alleviate the last two, but
that comes at the price of `cargo update` not wanting to act when
.cargo/config exists, because of the source replacement configuration.
But rust vendor gently generates a suitable configuration on its own, so
we can use that to generate a .cargo/config automatically. Which
addresses the first caveat of the current setup. That leaves us with
`cargo update` not working out of the box, but that just requires people
running it to manually remove .cargo/config first. Which is arguably
what rust wants you to do in the first place. It's kind of incidental
that we started with a .cargo/config.in rather than .cargo/config.
Now, while a simple .cargo/config works, that's not enough for the case
where the objdir doesn't live inside the source directory. In that case
cargo looks for the configuration from the objdir, and fails to find it.
So we still need a .cargo/config.in, which we generate with a little
trick.
Differential Revision: https://phabricator.services.mozilla.com/D43012
--HG--
rename : .cargo/config.in => .cargo/config
extra : moz-landing-system : lando
We'll require preprocessing that configure subst files don't allow in
the next change, so prepare for that.
Differential Revision: https://phabricator.services.mozilla.com/D43011
--HG--
extra : moz-landing-system : lando
Straight-forward addition of the --no-install option for reftest, just like
the existing option for mochitest. For gtest, I alos noticed the mach command
help needed some cleanup.
Differential Revision: https://phabricator.services.mozilla.com/D43299
--HG--
extra : moz-landing-system : lando
This time, existing config.status trying to import it will throw an exception
that will trigger a re-configure.
Differential Revision: https://phabricator.services.mozilla.com/D43215
--HG--
extra : moz-landing-system : lando
As seen in bug 1575774/1575959, things can go badly when config.status
doesn't match expectations, especially when most mach commands try to
read it, starting with mach clobber, which is supposed to be the one
commant to get away from most problems.
The idea here is to throw a recognizable exception so that callers can
react accordingly. While not obvious from the patch, the result is that
running e.g. mach build with a broken config.status will automatically
run configure (because the relevant caller actually handles the rethrown
exception that way).
There are other calls to from_config_status in the mozbuild reader, but
that can't be reached before MozbuildObject.config_environment.
Differential Revision: https://phabricator.services.mozilla.com/D43214
--HG--
extra : moz-landing-system : lando
Older config.status laying around in old trees are read by mach whenever
it runs, including when running mach clobber. Those files still try to
include mozbuild.util.encode, which is not here anymore. So we restore
the function for now to unbreak those.
Differential Revision: https://phabricator.services.mozilla.com//D43132
--HG--
extra : histedit_source : 3eb10cc8e18cc9094887061b00705afb6e705b54
Note that cargo vendor always uses the native version even when
cargo-vendor was already installed, so there is no concern wrt that.
Differential Revision: https://phabricator.services.mozilla.com/D43007
--HG--
extra : moz-landing-system : lando
Bug 1522788 changed how MozbuildObject gets a value for @CONFIG_GUESS@
in mozconfigs, and it's not entirely reliable to manually call
config.guess in test_base. As a matter of fact, in some cases,
config.guess is not even invoked by MozbuildObject (like, on Windows).
Plus, the config.guess invocation in the test currently doesn't work on
Windows on automation, for some reason.
Differential Revision: https://phabricator.services.mozilla.com/D42755
--HG--
extra : moz-landing-system : lando
Python 3.7 changed what it escapes with re.escape. Adapt mozpath.match
to account for this.
Differential Revision: https://phabricator.services.mozilla.com/D42750
--HG--
extra : moz-landing-system : lando
Now that the configuration comes in without bytes strings, there is no
need to convert it anymore.
Differential Revision: https://phabricator.services.mozilla.com/D42631
--HG--
extra : moz-landing-system : lando
With bug 1575135, we ensure nothing gets out of the configure sandbox
as a bytes string. We can thus now avoid the encode() pass in
configure.py. We still need, however, to normalize the configuration
so that it doesn't contain unexpected types, and conformning to what
indented_repr does to the configuration in config.status.
While here, convert some obj.iteritems() to six.iteritems(obj).
And remove the now unused encode function.
Differential Revision: https://phabricator.services.mozilla.com/D42630
--HG--
extra : moz-landing-system : lando
The configure sandbox has wrapped subprocess methods to add its own
encoded environment if none is provided, since bug 1520394. It only
makes sense that it normalizes the environment that comes in too,
avoiding caller in the configure sandbox to have to do it themselves.
OTOH, and while we're here, none of get_cmd_output, old_configure or the
sandbox were actually using the right encoding for this conversion, so
fix the configure sandbox to use the right one, and make it stop using
encode(), which does deep recursion that is not necessary here, and that
I'm trying to remove entirely.
Also while here, remove an unused import of encode().
Differential Revision: https://phabricator.services.mozilla.com/D42608
--HG--
extra : moz-landing-system : lando
There are a few problems with the strategy currently used to find the
rust target. For example, we don't find a target for arm freebsd, and we
pick the wrong target for armel linux. Both are related to how things
currently work when multiple targets have the same (cpu, endianness,
os).
So, to derive the rust target, we now use a more fine-grained approach.
Differential Revision: https://phabricator.services.mozilla.com/D41481
--HG--
extra : moz-landing-system : lando
The order that PerSourceFlags are emitted can vary if the srcdir is
changed. By sorting them we can make sure that the backend.mk files are
consistent regardless of srcdir.
Differential Revision: https://phabricator.services.mozilla.com/D42965
--HG--
extra : moz-landing-system : lando
Make it a hard error when the sandbox returns non-unicode strings.
This should help quickly catch any remaining non-unicode string that
are not caught by automation.
Differential Revision: https://phabricator.services.mozilla.com/D42607
--HG--
extra : moz-landing-system : lando
As a consequence, we can replace the encoded_open function that did the
same in an opt-in manner.
Differential Revision: https://phabricator.services.mozilla.com/D42605
--HG--
extra : moz-landing-system : lando
It only happens when things go badly, which is why it doesn't cause
problems, but when moving things around, triggering the error, we
currently get a formatting error rather than the actual error.
Differential Revision: https://phabricator.services.mozilla.com/D42602
--HG--
extra : moz-landing-system : lando
Fix some library calls and syntax that are required to support Python 3.
Depends on D39364
Differential Revision: https://phabricator.services.mozilla.com/D39365
--HG--
extra : moz-landing-system : lando
Make bootstrap.py compatible with both Python 3.5+ and Python 2.7. Remove
Python 2.6 support as Python 2.6 is no longer supported in the Firefox source
tree.
Differential Revision: https://phabricator.services.mozilla.com/D39364
--HG--
extra : moz-landing-system : lando
Add an option to run bootstrap.py with '--debug'. This will print full
tracebacks if the bootstrap process encounters an uncaught exception. It should
be useful for developers who are working on the bootstrap code as well as users
when writing bug reports.
Depends on D39362
Differential Revision: https://phabricator.services.mozilla.com/D39363
--HG--
extra : moz-landing-system : lando
Support installing the android development toolchain with Python 3 as well as
Python 2.7.
Depends on D39361
Differential Revision: https://phabricator.services.mozilla.com/D39362
--HG--
extra : moz-landing-system : lando
Add support for Python 3 and Python 2.7 to the Debian-based linux distro
bootstrap routines.
Differential Revision: https://phabricator.services.mozilla.com/D39361
--HG--
extra : moz-landing-system : lando
Add support for both Python 3 and Python 2.7 to the mozboot.base and
mozboot.bootstrap modules. Remove legacy Python 2.6 code or mark it for later
removal.
Depends on D39359
Differential Revision: https://phabricator.services.mozilla.com/D39360
--HG--
extra : moz-landing-system : lando
Add unicode_literals to all mozboot module __future__ statements to support
running the modules under Python 3. Remove comments about unicode_literals and
Python 2.6 support as Python 2.6 is no longer supported in tree.
Differential Revision: https://phabricator.services.mozilla.com/D39359
--HG--
extra : moz-landing-system : lando
Make the target and host targets depend on those, and flatten the
dependencies.
That is, instead of chains like:
browser/app/target: mozglue/build/target
mozglue/build/target: memory/build/target
we now have:
browser/app/target: browser/app/target-objects \
mozglue/build/target-objects \
memory/build/target-objects
Which means Make can feel free to build the object files in browser/app
before building other dependencies.
Differential Revision: https://phabricator.services.mozilla.com/D42252
--HG--
extra : moz-landing-system : lando
With the addition of toolkit/library/build because of the rust
shenanigans, bug 1573314 and bug 1572046 don't do anything useful
anymore. We're going to do something better anyways.
Differential Revision: https://phabricator.services.mozilla.com/D42251
--HG--
extra : moz-landing-system : lando
HostLibrary is always an expand lib. HostRustLibrary is always a
non-expand lib. But we currently rely on type checking to figure that
out, rather than an attribute, like for StaticLibrary. Doing that
simplifies existing and upcoming code.
Differential Revision: https://phabricator.services.mozilla.com/D42250
--HG--
extra : moz-landing-system : lando
Apart from the need to link them last, they don't actually need to be
treated any different from normal static libraries.
Differential Revision: https://phabricator.services.mozilla.com/D42249
--HG--
extra : moz-landing-system : lando
So that isinstance(foo, StaticLibrary) doesn't end up being true for
HostRustLibrary. Instead, it now inherits from HostLibrary.
Differential Revision: https://phabricator.services.mozilla.com/D42248
--HG--
extra : moz-landing-system : lando
The existing check is no thorough enough: it only checks that multiple
Rust libraries are not directly linked to the same binary, but that's
not enough. In fact, until the change prior to this one, this was
happening to xul-gtest, and without that change, this is what configure
would now have to say:
Cannot link the following Rust libraries into the "xul-gtest" library:
- gkrust-gtest
- gkrust
Only one is allowed.
This only ended up not being a problem because the backend somehow works
around the problem, which it shouldn't have to do.
Differential Revision: https://phabricator.services.mozilla.com/D42247
--HG--
extra : moz-landing-system : lando
The Puppeteer test flavour are functional tests for the CDP-based
Puppeteer library from Google, that we want to run against our
implementation of CDP for Firefox.
They are distinct from the Firefox Puppeteer tests based on Marionette.
Differential Revision: https://phabricator.services.mozilla.com/D37012
--HG--
extra : moz-landing-system : lando
The Puppeteer test flavour are functional tests for the CDP-based
Puppeteer library from Google, that we want to run against our
implementation of CDP for Firefox.
They are distinct from the Firefox Puppeteer tests based on Marionette.
Differential Revision: https://phabricator.services.mozilla.com/D37012
--HG--
extra : moz-landing-system : lando
Basic test infrastructure for Fluent migration recipes, automate creating
a reference from recipe data, and running the recipe dry and wet.
There are no hard failures for diffs, as it might just be OK.
Differential Revision: https://phabricator.services.mozilla.com/D40200
--HG--
extra : moz-landing-system : lando
When a directory, like toolkit/library, builds both a static and a
shared library, and another, like toolkit/library/gtest, depends on the
static part, it currently needs to wait for the shared library to be
finished building, preventing both libraries being built in parallel.
By separating shared libraries to a different target, we allow more
parallelism to the build.
Differential Revision: https://phabricator.services.mozilla.com/D41099
--HG--
extra : moz-landing-system : lando
This patch makes BuildEnvironmentNotFoundException a subclass of AttributeError as well, because hasattr in
python3 no longer catches all tracebacks but only AttributeError, and we use both hasattr and
BuildEnvironmentNotFoundException to guard against a handful of buildconfig variables in a few places
where it is OK to not have a buildenvironment.
We also use universal_newlines in real_host in init.configure (since I found
that fix while working on the AttributeError one) so that we get the right string type back from the process call
Lastly this patch also uses BytesIO for calling into a ReducedConfigureSandbox as its stdout and stderr pipes,
This ensures that related code handling the sandbox doesn't complain about getbuffer() missing in StringIO in py3.
Differential Revision: https://phabricator.services.mozilla.com/D36605
--HG--
extra : moz-landing-system : lando
The build docker images need python-dev installed to build psutil, used
by the build resources monitor.
Differential Revision: https://phabricator.services.mozilla.com/D40781
--HG--
rename : python/mozbuild/mozbuild/resources/html-build-viewer/index.html => python/mozbuild/mozbuild/resources/html-build-viewer/build_resources.html
extra : moz-landing-system : lando
The longer version is that this changes how the HTML viewer looks up
files. Instead of looking at a list file that contains a list of
build resources data, it now looks at a build_resources.json file that
either contains data directly, or a list of files containing data.
Differential Revision: https://phabricator.services.mozilla.com/D40780
--HG--
extra : moz-landing-system : lando
This patch adds a global lint that only runs when a file or directory
that matches their configuration (via `extensions` and `exclude`) has
been modified or specified.
Global lints never shard into chunks; they are, by definition global
(i.e., across the entire source tree) and act on all inputs in a
single invocation. It's up to the global lint to manage command line
sizes, etc. Since batching is handled by the lint type but sharding
is handled by the lint roller, there's a little abstraction leak so
that the lint type can control how its invocation is sharded: the
existing `batch` member is generalized from the existing `True` and
`False` to add a new `"global"` value which disables sharding.
Differential Revision: https://phabricator.services.mozilla.com/D35275
--HG--
extra : moz-landing-system : lando
This allows lints to "condition" themselves on having a build
environment or a specific build application. It also adds the "name"
parameter, so that setup functions can be shared across lints.
`MozbuildObject` cannot be used as parameters to functions distributed
via multiprocessing, since they cannot be pickled (due, currently, to
internal terminal handles). Therefore we extract just a few key
parts of the environment to expose.
Differential Revision: https://phabricator.services.mozilla.com/D35274
--HG--
extra : moz-landing-system : lando
On Windows in Python 2, the subprocess module requires the use of bytes with
the 'env' argument. For that reason, we would sometimes use byte strings with
'os.environ' like so:
os.environ[b"FOO"] = b"bar"
However, this is a failure with Python 3 as 'os.environ' must only be used with
the text type. This patch creates a new 'setenv' helper that ensures we create
new environment with 'bytes' on Python 2, and 'text' on Python 3.
Differential Revision: https://phabricator.services.mozilla.com/D38237
--HG--
extra : moz-landing-system : lando
It causes confusion because it's the mode used to _read_ the overwritten
file. Make that more obvious by renaming to `readmode`.
Differential Revision: https://phabricator.services.mozilla.com/D39445
--HG--
extra : moz-landing-system : lando
There is less incentive to keep things building with older versions of
clang for OSX builds, and we're going to require an objective-C feature
that was added in clang 5.
Differential Revision: https://phabricator.services.mozilla.com/D38581
--HG--
extra : moz-landing-system : lando
The preprocessed install manifest code uses a Makefile dependency file
to list inputs. If one of the inputs was missing (due to a file removal
or rename), an incremental build would fail until clobbered. We should
treat a missing input as simply being out-of-date and continue
processing.
Differential Revision: https://phabricator.services.mozilla.com/D38576
--HG--
extra : moz-landing-system : lando
Rather than relying on the mar-channel-id set in the `mar` binary, set the channel
explicitly from taskcluster. This allows us to re-use the `mar` binary between
builds/channels.
Differential Revision: https://phabricator.services.mozilla.com/D37481
--HG--
extra : moz-landing-system : lando
Credit: Callek for figuring out an issue in 'make check' making the binary
absolute in mozbuild.base.
Differential Revision: https://phabricator.services.mozilla.com/D37319
--HG--
extra : moz-landing-system : lando
When building Gecko/Android/aarch64 on Windows, `--target` parameter may not be incorrect value. Although `check_compiler`'s `info` is target compiler, clang on Windows is always detected as `clang-cl`, not `clang`.
```
c:/Users/mkato/.mozbuild/clang/bin/clang.exe -E -dM - < /dev/null
...
#define _MSC_VER 1916
```
So even if using clang on Windows, not clang-cl, we should detect as 'clang' correctly
Differential Revision: https://phabricator.services.mozilla.com/D36422
--HG--
extra : moz-landing-system : lando
By globally importing PackageFrontend from the globe-analysis module we break the logger for
the PackageFrontend package.
Differential Revision: https://phabricator.services.mozilla.com/D36887
--HG--
extra : moz-landing-system : lando
The SDK headers may not be installed in /usr/include. The usual response
has been to have people run e.g. `open
/Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg`
which is not really sustainable.
This makes builds that happen on a macOS host try to detect their SDK
and use that as a default for --with-macos-sdk, which has the side
effect of enabling the SDK version check in that configuration.
Differential Revision: https://phabricator.services.mozilla.com/D36558
--HG--
extra : moz-landing-system : lando
This test is working with minimal effort. Let's get it running to prevent
future regressions.
Differential Revision: https://phabricator.services.mozilla.com/D36098
--HG--
extra : moz-landing-system : lando
Configure should just be able to find the right one. If it doesn't, that
should be fixed in configure rather than with suggestions in bootstrap.
Differential Revision: https://phabricator.services.mozilla.com/D36562
--HG--
extra : moz-landing-system : lando
Configure should just be able to find the right one. If it doesn't, that
should be fixed in configure rather than with suggestions in bootstrap.
Differential Revision: https://phabricator.services.mozilla.com/D36562
--HG--
extra : moz-landing-system : lando
Bug 1554987 made `mach try` use a transient remote, but that causes
problems with existing setups that happen to use the same remote name,
because of a combination of not-quite-as-documented-as-it-should
behavior of git.
- `git -c foo.bar=qux` doesn't override the value of `foo.bar` from the
git configuration when `foo.bar` is an item that can take several
values.
- `remote.$remote.url` and `remote.$remote.pushurl` take several values,
allowing to give several URLs.
The combination of both means that if the git configuration already has
`remote.try.url` set, that value takes precedence (because git push
tries them one after the other, and takes the one from the command line
last)
One way we could increase the chances of things working out fine would
be to use `remote.try.pushurl`, which if already set, is more likely to
be right than an existing `remote.try.url`.
OTOH, it turns out, after more investigation, that bug 1554987 requires
a footgunny setup to happen in the first place. Namely, it requires
having run `git lfs install` from a git-cinnabar clone.
so we just go back to the previous status quo.
Differential Revision: https://phabricator.services.mozilla.com/D36149
--HG--
extra : moz-landing-system : lando
