Brew's Python 3.10 causes `virtualenv==20.7.2` to produce a wonky folder
structure (`$venv/opt/homebrew/lib/python3.10/site-packages`?).
This is likely fixed with newer `virtualenv`, but the simpler workaround
here is to use `venv` instead now that Python 3 is always used.
Adds `python3-venv` to docker image so that tests and debian-based tasks
can leverage it.
Differential Revision: https://phabricator.services.mozilla.com/D144872
Messages used by the `about:addons` page are migrated to `aboutAddons.ftl`.
Unused messages are dropped, as is the `get_string()` helper from `test/browser/head.js`.
Differential Revision: https://phabricator.services.mozilla.com/D131793
Virtualenv-based objdir detection doesn't work for instrumented builds,
as instrumentation-specific objdirs aren't created.
Resolve this by:
* Removing redundant cwd-based topsrcdir resolution - the
`__file__`-based code path below covers this case.
* Only using virtualenv-based resolving if topsrcdir isn't resolved.
* Documenting that virtualenv-based resolution is problematic for
instrumented builds.
Differential Revision: https://phabricator.services.mozilla.com/D144030
The default download folder prefs are now counterintuitively used
whether browser.download.useDownloadDir is enabled or not, since we no
longer save downloads to the "temp" folder. But the display for these
prefs in about:preferences makes it seem as though the configured path
will not be used if the "Always ask..." option is selected. This patch
removes the radiogroup and replaces it with a checkbox, so that the
download folder path can be changed irrespective of useDownloadDir's
value (unless browser.download.dir is disabled by policy).
Differential Revision: https://phabricator.services.mozilla.com/D143555
Virtualenv-based objdir detection doesn't work for instrumented builds,
as instrumentation-specific objdirs aren't created.
Resolve this by:
* Removing redundant cwd-based topsrcdir resolution - the
`__file__`-based code path below covers this case.
* Only using virtualenv-based resolving if topsrcdir isn't resolved.
* Documenting that virtualenv-based resolution is problematic for
instrumented builds.
Differential Revision: https://phabricator.services.mozilla.com/D144030
This allows to remove the requirement on "--help" for dependable(),
which avoids dependable() implying the execution of `when` when it's
within a `with only_when`.
Differential Revision: https://phabricator.services.mozilla.com/D144404
1. add the MOZ_X11 config flag in build/gn.mozbuild and set the gn_vars
accordingly.
2. create the new gn-config/mozconfig files and delete previous ones
dom/media/webrtc/third_party_build/gn-configs/**.mozconfig with
--enable-default-toolkit=cairo-gtk3-wayland-only for the non X11 version.
New toolkit nmae is required to force disable X11 detection as
cairo-gtk3-wayland will auto-detect X11 and make generate-gn-build-files.sh
fail.
3. Add the MOZ_X11 config flag in python/mozbuild/mozbuild/gn_processor.py
Then run
dom/media/webrtc/third_party_build/gn-configs/generate-gn-build-files.sh
Differential Revision: https://phabricator.services.mozilla.com/D142904
Until clang 14, the default was -ffp-contract=fast, but some reason
didn't really enable FP contraction in practice. GCC has the same
default, but does, in fact enable FP contraction*. This leads to more
inconsistencies in FP results between builds than would normally be
expected, and has an effect on e.g. fingerprinting resistance in fdlibm
trigonometric functions, or some other places (e.g. bug 1765436).
As far as the Firefox versions Mozilla ships are concerned, FP
contraction only really affects aarch64, but this change should in
practice make no difference with the current version of clang used (13),
and would preserve our now new default behavior with clang 14.
(*) We never noticed because we neither build with GCC for arm64 nor run
tests with GCC builds.
Differential Revision: https://phabricator.services.mozilla.com/D144136
The `test_vendor.py` test needs access to `poetry` to make its
assertions. This requirement was previously satisfied by adding `poetry`
to the `python-test` site.
Unfortunately, this caused conflicts when running the `mozlint` tests,
as they use the ini "requirements" feature to install their
dependencies. Importantly, the `mozlint` dependencies aren't compatible
with the version of `poetry` we need.
Fortunately, this was a good reminder: since `poetry` is a test-specific
dependency, leverage the existing ini "requirements" feature as it
represents exactly that: the extra package dependencies of a specific
test.
Note that I had to doctor the `dataclass` line in
`vendor_requirements.txt` to include an environment marker, otherwise it
would fail to install with Python >= 3.7.
Differential Revision: https://phabricator.services.mozilla.com/D143376
1. add the MOZ_X11 config flag in build/gn.mozbuild and set the gn_vars
accordingly.
2. create the new gn-config/mozconfig files and delete previous ones
dom/media/webrtc/third_party_build/gn-configs/**.mozconfig with
--enable-default-toolkit=cairo-gtk3-wayland-only for the non X11 version.
New toolkit nmae is required to force disable X11 detection as
cairo-gtk3-wayland will auto-detect X11 and make generate-gn-build-files.sh
fail.
3. Add the MOZ_X11 config flag in python/mozbuild/mozbuild/gn_processor.py
Then run
dom/media/webrtc/third_party_build/gn-configs/generate-gn-build-files.sh
Differential Revision: https://phabricator.services.mozilla.com/D142904
This makes some ./mach vendor improvements needed:
- Adds a replace-in-file-regex action
- Allows you to skip copying the temporary extract directory
so that a script can do it
- Removes 'commit' from the look-in-moz.yaml-for-release match
Differential Revision: https://phabricator.services.mozilla.com/D142720
This change does a few things
- Moves get_full_path from a function-local function to
a class member
- Makes yaml_file a class member so I don't need to pass
it everywhere
- Creates a new moz.yaml file directive 'generated'
- Adds that directive to libdav1d
- Detects if any files (other than generated ones) were
modified by an update; and if not - do not update the
moz.yaml. Exit with -2 so Updatebot can detect it.
Differential Revision: https://phabricator.services.mozilla.com/D141994
The `upload_geneated_sources.py` script needs `boto3`, but was:
* Running in CI with Mach having a site-packages-source of `None`
* Using the `common` site (which inherits Mach's site-packages-source)
* Installing `boto3` outside of the knowledge of the centralized
dependency system.
Make a `upload-generated-sources` site, and move `boto3` to it.
Differential Revision: https://phabricator.services.mozilla.com/D143715
* There is no `--duration` argument, but there //is// `--durations`
* `pytest` behaves better when called via its main entry points (either
`bin/pytest`, or `-m pytest`)
Differential Revision: https://phabricator.services.mozilla.com/D143609
De-duping with `set` causes order to no longer be maintained when the
items are distinct. Use the `OrderedDict` technique to de-dupe
consistently instead.
Differential Revision: https://phabricator.services.mozilla.com/D143604
I believe this is safe because from what I've been able to find
SafeLoader is FullLoader but without the bad stuff that lets you
execute arbitrary python code. We don't need (nor want) the more
comlpicated data types like 'dates' and 'numbers' so we can use
BaseLoader which does nothing except treat everything as a string.
Depends on D142125
Differential Revision: https://phabricator.services.mozilla.com/D142126
Both the dtd and properties files strings are moved to the existing storage.ftl.
A migration file was added to keep the localized strings.
Some of the strings were migrated to a declarative approach, setting `data-l10n-id` and
`data-l10n-args` attributes on elements. But in other cases, this was not easily doable.
To accomodate with the asynchronicity of `formatValue`, we translate all the strings that
don't have dynamic parameters during the initialization of the panel, and store them in
a Map for easy retrieval.
We also took this opportunity to cleanup the strings key names and have them all prefixed
with `storage-*`. Some functions where refactored so it should be easier to search the
code for a specific l10n string.
Finally, the `table.headers.indexedDB.*` strings weren't migrated as they were never read
in the UI (they are referenced in `NON_L10N_STRINGS`, which bypass localization).
Differential Revision: https://phabricator.services.mozilla.com/D143012
Verify that activating mach and command sites in various configurations
affects the `sys.path` accurately.
Also verify that the `sys.path` is correct between the "activated
process" and fresh processes spawned from created virtualenvs.
Finally, tweak existing path-management behaviour to be more consistent:
* Don't unnecessarily sort `pths` from requirement definitions - it's
more consistent (and more dependable) to keep the provided order.
* Continue removing unwanted `venv` entries from the `sys.path`, but
don't add them back (such as the `$prefix` entry)
* Always remove `venv` entries from the `sys.path` when the venv isn't
being populated. Though this doesn't affect behaviour, it makes
assertions easier and simplifies the runtime state a bit.
Differential Revision: https://phabricator.services.mozilla.com/D143201
Previously, when using the system Python packages, Mach would reuse the
values already existing in the `sys.path`. This had two benefits:
1. We didn't have to do work to calculate which paths the "system
Python" specifically referred to.
2. This allowed us to support nested Mach calls
(such as `./mach --virtualenv psutil ./mach build`).
However, it came with its own headaches, specifically around "consistent
imports" and Python subprocesses, such as in the following example
1. Mach runs "using system Python"
2. The "build" site (for example) would be activated. The current
`sys.path` is included in the virtualenv's `mach.pth` file so that
subprocesses will have access to the same packages as the primary
Mach process
3. //Something// adds something to the `sys.path`
4. The build virtualenv is redundantly re-activated (such as due to
generic setup code for some module). Though we don't physically
re-activate the virtualenv, we //do// check that it's up-to-date to
validate assumptions - **and it's not!** It's missing the
//something// that was added to the `sys.path`.
5. We can't re-create the build virtualenv because it's already active,
so:
6. An error is raised so that unexpected out-of-date-ness doesn't fly
under the radar.
-----
This patch solves this by calculating the "system Python" paths in a
deterministic way. Then, even when using the system Python, the Mach and
command sites use define themselves in a consistent way.
This means that we can't do `./mach --virtualenv <venv> ./mach ...` to
shares `venv`'s packages with the whole new Mach process. Fortunately,
this has been replaced with moving such packages into the nested Mach
command's requirements definition instead.
TL;DR: more consistent, less failures. 👍
-----
One more detail, this time around the `sys_path()` function:
Ideally, we could calculate the standard library paths *and* the
"system" paths with only one Python subprocess. Unfortunately,
that's not the case:
* If `-S` isn't provided, then it's tricky to separate the standard
library paths from the system paths
* If `-S` is provided, then doing `site.getsitepackages()` in a
virtualenv returns the *system* site packages, not the virtualenv's.
To work around this, we call the external python twice, and in
parallel. This allows resolving the information we need while avoiding
performance costs.
Differential Revision: https://phabricator.services.mozilla.com/D143200
Use `poetry` instead of `pip-compile`, then leverage `pip` to assert
that there's no conflicts between environments.
This enables simplifying `requirements.in`, since `colorama` is now
automatically picked up, as vendoring is no longer linux-py3.6-specific.
`poetry==1.2.0a2` was chosen because it was the first version that would
include `pip`/`setuptools`/`wheel` in the `requirements.txt`, yet it was
also the last version that supports Python 3.6 (as still used in CI in
some spots).
Sorry about the `requirements.txt` noise, that's because poetry doesn't
sort hashes yet [1]. Fortunately, you can be confident in
`requirements.txt` changes by checking that this patch doesn't include
changes to vendored packages themselves (I'm including the results of a
`./mach vendor python`).
[1] https://github.com/python-poetry/poetry/issues/5408
Differential Revision: https://phabricator.services.mozilla.com/D142884
The PGO-specific objdir is meant for the PGO build artifacts.
The python virtualenvs just exist for the host, so there's no need to
scope them. Besides, the existing virtualenv infrastructure doesn't know
how to distinguish between "main 'build' site" and "PGO 'build' site",
which is why it raised this "out-of-date" error.
Differential Revision: https://phabricator.services.mozilla.com/D142373
Now that we are gleaning a site packages source in a process-global way
(that is *not* conditional on the site that's being managed), we've
removed an edge case from occurring: the case where Mach would use
`SitePackagesSource.NONE`, but the command site would use
`SitePackagesSource.SYSTEM`. This would've occurred if none of Mach's
dependencies were found in the system Python, but instead once the
command site was initialized, one of *its* dependencies were located.
Since that can no longer happen:
* Command sites that don't populate their VENV will *always* use the
same site packages source as Mach
* `pthfile` generation is simplified (the priority of the system paths
is no longer variable)
* We no longer need to track `site_packages_source` in metadata, since
we can use `mach_site_packages_source` instead.
While here, I moved the "`PIP_NETWORK_INSTALL_RESTRICTED_VIRTUALENVS`
sites shouldn't have `pypi_requirements`" check to a test, since there's
no need for it to happen at runtime.
Differential Revision: https://phabricator.services.mozilla.com/D140668
Simplify Mach's decision-making in CI for where to find native Python
packages: unless `MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE=system` is
provided, always ignore system packages.
Removes optional dependency on system's `pip`, instead leaning on
vendored `pip` copy to see what package versions are installed.
Differential Revision: https://phabricator.services.mozilla.com/D140258
Unfortunately, the dependencies of l10nregistry-rs require tokio 1.0. This
allows this third duplicated instance. I think it makes more sense to try to
move the other uses in tree to a more modern version than to try to downgrade
the version of tokio required by l10nregistry-rs and dependencies.
Differential Revision: https://phabricator.services.mozilla.com/D139854
The dataReportingNotification is the last place where the .properties
vendorShortName is used. As notifications already support Fluent,
porting this to use Fluent as well seems pretty strqaightforward.
Differential Revision: https://phabricator.services.mozilla.com/D141860
The Mach site is asserted to be compatible with all command sites, but
command sites are deliberately //not// asserted to be compatible with
each other - they're //supposed// to have the flexibility of being
able to be incompatible.
Accordingly, let's fail loudly if code tries to activate from one
command site to another.
Note that, due to the Mach site being a superset of common, it's safe to
activate from "common" to "<other command site>". This is needed for
cases like `./mach python --virtualenv ...`.
This required updating the `mozproxy` tests, who would deliberately
activate the `common` site so they could call a `mozproxy` entry point
script. These tests were fixed by instead invoking `mozproxy` as a
module (`-m`) of the current `python-test` site.
Differential Revision: https://phabricator.services.mozilla.com/D141659
This is needed for a few reasons:
* All mach commands can use virtualenvs, not just `build`-related
commands, so the files don't make sense to be in `build/`.
* When locking is added, more files associated with virtualenvs will be
added, and this will change will ease the related directory structure
setup.
* This removes the need for a redundant "_virtualenv_packages" keyword
as part of the manifest filenames.
Differential Revision: https://phabricator.services.mozilla.com/D140382
The intention is to remove all the files in the vendoring directory
(which may be the yaml directory or may not be) except those
specified in 'keep'.
However convert_patterns_to_paths expects a list of patterns, and
we were providing it with only one. This resulted in it iterating
over '**' as a string (one iteration for '*' and a second for '*')
This prevented it from traversing recursively. Turning it into a
list fixes it.
Depends on D141901
Differential Revision: https://phabricator.services.mozilla.com/D141902
This creates a way to override the 'tracking: tag' mechnaism
of a moz.yaml file without having to edit it. Useful to
test a library update when there is no new tag available.
Differential Revision: https://phabricator.services.mozilla.com/D141901
Update DSN for reporting exceptions to the new Sentry instance, and
update CI job to create releases on the new Sentry as well.
Differential Revision: https://phabricator.services.mozilla.com/D141122
Since bug 1717104 changed `./mach vendor python` to no longer have a
manual virtualenv activation, we can remove the test mocking that
softly handled the activation.
Differential Revision: https://phabricator.services.mozilla.com/D141674
All commands declaring a virtualenv will have them activated before the
command executes. Removes all now-redundant manual activations of
declared virtualenvs.
Commands that don't declare a virtualenv will still implicitly be
associated with the "common" virtualenv, but unlike explicit
virtualenv declarations it'll have to be activated manually, just
like it was before this patch.
To smooth the migration with existing usages, virtualenv activation
behaviour was changed slightly: if attempting to activate a new
virtualenv, but the source venv is already command venv, then raise an
exception. (In the future, we should improve testability of
virtualenv scaffolding logic so that tests can be added for this
sort of thing.) This did cause some issues with some tests, which
will be solved more cleanly with bug 1724273. In the meantime,
minimal modifications were made to failing tests to keep them green:
* `test_command_line.py` was activating the `common` virtualenv so
that it could install `mozproxy`, and use its CLI. Instead, I
modified the test to use `mozproxy` using the "module" interface
(`python -m mozproxy ...`). At that point, `MozbuildObject` was
unnecessary and usages were replaced with simpler variants.
* `test_vendor.py` needed its explicit `activate_virtualenv()` call
patched out. It still needs to use a virtualenv's Python
executable, but due to `sys.executable` now being kept up-to-date
as of bug 1717051, it could be used directly.
Differential Revision: https://phabricator.services.mozilla.com/D122892
Creates/updates virtualenvs for some mach commands, replacing
their ad-hoc usage of `install_pip_package()`, `pip install`,
and `sys.path` modifications.
Note: The `docs` virtualenv has `Sphinx==1.1.3` installed, even
though a more modern version of `Sphinx` is used when
`./mach doc` is run. This is ok for now, since `./mach doc` will
just install the newer `Sphinx` over top of the old one. Secondarily,
when we port `./mach doc` to use the centralized system, we'll
be incentivized to make the different `doc` commands use synchonized
versions of the same packages. Success!
Also, note that manual installation of `html5lib` and `requests`
isn't ported to the `wpt` site: this is because they're already
provided by the inherited Mach site.
Differential Revision: https://phabricator.services.mozilla.com/D122902
There are cases in CI where sites are being considered out-of-date,
despite the situation being unexpected - for example, the
"site out-of-date even though already activated" issue described in
this patch's associated bug.
Return a `reason` string from `_is_venv_up_to_date()`, and print it in
cases where a virtualenv being out-of-date is erroneous.
Differential Revision: https://phabricator.services.mozilla.com/D140855
The Mach site management has been significantly simplified
in `mach_initialize.py`, so less of the `MachSiteManager`
API is used. Privatize such now-externally-unused functions,
and no longer return unneeded values.
Differential Revision: https://phabricator.services.mozilla.com/D140854
There are cases in CI where sites are being considered out-of-date,
despite the situation being unexpected - for example, the
"site out-of-date even though already activated" issue described in
this patch's associated bug.
Return a `reason` string from `_is_venv_up_to_date()`, and print it in
cases where a virtualenv being out-of-date is erroneous.
Differential Revision: https://phabricator.services.mozilla.com/D140855
The Mach site management has been significantly simplified
in `mach_initialize.py`, so less of the `MachSiteManager`
API is used. Privatize such now-externally-unused functions,
and no longer return unneeded values.
Differential Revision: https://phabricator.services.mozilla.com/D140854
Previously, when trying to determine the minimal `sys.path` that just
includes the standard library (but no `pip`-installed packages), we
would resolve the `sys.path` and remove the system and user
site-packages. However, this "removal" step didn't work as-is for
`brew`'s Python, because its "system site-packages" is //different//
from its `site.getsitepackages()`.
Stepping back though, there's an easier solution: run `python` with the
`-S` flag, and no "site" paths or custom initialization logic will add
anything extra to the stdlib.
Differential Revision: https://phabricator.services.mozilla.com/D141508
Now that we're using `sysconfig` instead of `distutils`, our `purelib`
and `platlib` paths should correctly point to python environment paths
without a spurious `local/` component at their roots.
Differential Revision: https://phabricator.services.mozilla.com/D140871
Depending on the subprocesses created (such as if
`--with-ccache=sccache` is set), `./mach configure` may hang.
More details about why this is is documented in bug 1753797.
The workaround is to lean on the standard library to stream-and-format
output, rather than our existing `ProcessHandler` code.
Though this still streams both `stdout` and `stderr` in real-time, I saw
some ordering differences between the two streams locally. I don't yet
have a reason to believe that these differences are harmful or otherwise
incorrect.
Differential Revision: https://phabricator.services.mozilla.com/D141028
On newer systems, Python is intelligently populating pure-python and
platform-specific packages into //two different directories//.
Traditionally, these directories would be symlinked - or only one would
exist. However, to support these newer systems (and be better integrated
with Python conventions), we should handle both of these directories
properly.
The solution here is, when calculating the `site_packages_dir()`,
calculate all of them, and handle the multitude at all call sites.
The one use case where we want only one path (the resolving the location
for the `mach.pth` pthfile), I've opted for the `platlib`, since both
pure python dependencies //and// platform-specific dependencies are
referenced by the pthfile.
Differential Revision: https://phabricator.services.mozilla.com/D140870
I'm guessing that the reason we haven't been building the Mach
virtualenv in CI so far is because:
* The default location in `~/.mozbuild` may not be cleared between jobs
* There's a performance cost to `pip install`-ing packages, especially
if they're not needed
* In the past, it was tricky to have some jobs use a Mach virtualenv and
others not.
This led to weird workarounds where a virtualenv would be created,
populated, then masqueraded as the "system Python" so that Mach would
be able to run with access to its dependencies without needing to
"create a Mach virtualenv".
This patch addresses the first point about the Mach virtualenv location:
In CI, if `WORKSPACE` is set, the Mach venv will be created in
`$WORKSPACE/mach_virtualenv`.
The other two points are solved by explicit configuration using the
`MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE` environment variable.
Differential Revision: https://phabricator.services.mozilla.com/D140256
There are cases in CI where we're using the system Python, but we
still want to create a virtualenv. For example, build jobs use the
system Python packages, but Mozharness wants to have access to Mach
packages. So, a virtualenv needs to be created (the `"common"`
virtualenv), whose associated `python` binary is used to invoke
Mozharness.
So, just like for the `build` site, allow creating the `common`
site even when the "native package source" is the system Python.
Since `mach_virtualenv_requirements.txt` already depends on
`common_virtualenv_requirements.txt`, this restriction should not cause
validation breakage.
Differential Revision: https://phabricator.services.mozilla.com/D140255
Adds support and documentation for the
`MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE` environment variable.
The "fallback" behaviour here should be backwards-compatible, which will
enable us to piece-by-piece explicitly use the `"system"` where
needed, then remove the backwards-compatibility shim afterwards.
A restriction added by this change is that it's no longer possible to
have the combination of:
* Have Mach use the system Python, and
* Have the active command site do `pip install`.
This is because this case shouldn't be necessary (if `pip install` is
allowed, why use the system?) and will allow us to enforce that all
non-build command sites always use their lockfiles, when they're set up.
When referring to behaviour in CI, I've opted to only refer to the
upcoming behaviour (of `MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"`)
because the current heuristic behaviour feels complex enough that it
should be understood by checking the code, rather than docs. Besides,
the heuristic will be removed pretty soon (right? ;)
Differential Revision: https://phabricator.services.mozilla.com/D138932
I'm guessing that the reason we haven't been building the Mach
virtualenv in CI so far is because:
* The default location in `~/.mozbuild` may not be cleared between jobs
* There's a performance cost to `pip install`-ing packages, especially
if they're not needed
* In the past, it was tricky to have some jobs use a Mach virtualenv and
others not.
This led to weird workarounds where a virtualenv would be created,
populated, then masqueraded as the "system Python" so that Mach would
be able to run with access to its dependencies without needing to
"create a Mach virtualenv".
This patch addresses the first point about the Mach virtualenv location:
In CI, if `WORKSPACE` is set, the Mach venv will be created in
`$WORKSPACE/mach_virtualenv`.
The other two points are solved by explicit configuration using the
`MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE` environment variable.
Differential Revision: https://phabricator.services.mozilla.com/D140256
There are cases in CI where we're using the system Python, but we
still want to create a virtualenv. For example, build jobs use the
system Python packages, but Mozharness wants to have access to Mach
packages. So, a virtualenv needs to be created (the `"common"`
virtualenv), whose associated `python` binary is used to invoke
Mozharness.
So, just like for the `build` site, allow creating the `common`
site even when the "native package source" is the system Python.
Since `mach_virtualenv_requirements.txt` already depends on
`common_virtualenv_requirements.txt`, this restriction should not cause
validation breakage.
Differential Revision: https://phabricator.services.mozilla.com/D140255
Adds support and documentation for the
`MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE` environment variable.
The "fallback" behaviour here should be backwards-compatible, which will
enable us to piece-by-piece explicitly use the `"system"` where
needed, then remove the backwards-compatibility shim afterwards.
A restriction added by this change is that it's no longer possible to
have the combination of:
* Have Mach use the system Python, and
* Have the active command site do `pip install`.
This is because this case shouldn't be necessary (if `pip install` is
allowed, why use the system?) and will allow us to enforce that all
non-build command sites always use their lockfiles, when they're set up.
When referring to behaviour in CI, I've opted to only refer to the
upcoming behaviour (of `MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"`)
because the current heuristic behaviour feels complex enough that it
should be understood by checking the code, rather than docs. Besides,
the heuristic will be removed pretty soon (right? ;)
Differential Revision: https://phabricator.services.mozilla.com/D138932
We want `_pthfile_lines()` to consistently and accurately represent a
virtualenv's `sys.path` modifications.
However, this becomes tricky when comparing expected `pthfile_lines`
//before// activating a virtualenv versus //afterwards//, especially
since Python implicitly adds some paths (such as the path to
`site-packages`).
The current solution made this scrubbing only happen if we were
`pip install`-ing into the site. Unfortunately, it //doesn't// work for
the "use system Python for the `build` site", because:
* Pre-activation result: `site-packages` isn't added, because we aren't
using it
* Post-activation result, implicitly-added `site-packages` isn't
scrubbed, because **scrubbing only happened for
`SitePackagesSource.VENV`**
Stepping back, the solution here is:
* `pthfile_lines` only represents Mach modifications //on top// of
implicit virtualenv behaviour: since `site-packages` is always added
by default, it shouldn't be in our explicit `pthfile_lines`.
* The only time when implicit `sys.path` additions throws us off is when
`SitePackagesSource.SYSTEM`: so, move the scrubbing to happen in that
case instead.
Finally refactor the "conditional deprioritization" comment to be more
useful and more accurate: we've implemented the "nontrivial complexity"
of purging site-packages, and the other piece about nothing being
`pip install`-ed feels self-explanatory enough.
Differential Revision: https://phabricator.services.mozilla.com/D140580
So far, we've been using `virtualenv`'s `activate_this.py` script.
However, unlike earlier expectations, it adds its `sys.path` additions
to the //front//, not the back! This breaks our prioritization
requirements, such as:
* When using any package from the system environment, import *all
possible* packages from the system to avoid compatibility issues.
* Use vendored packages instead of virtualenv-installed packages
wherever possible, because it more-closely matches developer
expectations ("why is this package vendored if it's not used?")
Define an `activate_virtualenv()` function that replicates the logic
of `activate_this.py` [1], except for three differences:
* Don't modify `sys.real_prefix`, since it's a non-standard property of
`sys`.
* Only add seen-with-`venv`-module paths to the `sys.path` (`$prefix`,
`$prefix/.../$site_packages_dir`) - don't do the paths in-between.
* And, of course, append instead of prepend `sys.path` entries.
As an aside, this is one of the few remaining blockers from allowing
us to fully embrace `venv` instead of `virtualenv` - the last piece is
waiting on the fix for bug 1697833 to propagate.
[1]
https://github.com/pypa/virtualenv/blob/20.7.2/src/virtualenv/activation/python/activate_this.py
Differential Revision: https://phabricator.services.mozilla.com/D140579
Unless either the site's requirements or its on-disk virtualenv
changes, it shouldn't become out-of-date.
This will also catch situations where sites are incorrectly
being marked out-of-date due to `sys.path` mismanagement.
Differential Revision: https://phabricator.services.mozilla.com/D140578
When calling Mach, it was prefixed with the `srcdir`, even though the
`srcdir` was also the working directory of the invocation. This caused
the invocation to fail.
Differential Revision: https://phabricator.services.mozilla.com/D140400
Inconsistency confuses some of our tools. As part of this, I:
* Updated some docs to point to rust-minidump
* Added a fallback to mozcrash.py to try both versions
* Make mozcrash.py use --brief output when the local mdsw is used
* Remove the renaming hack from build-minidump-stackwalk.sh
This isn't as simple as a sed because we still have breakpad in tree
for minidump-analyzer. I did my best to replace the right strings.
Differential Revision: https://phabricator.services.mozilla.com/D138971
Updates parser documentation to clarify that, if no `paths` are
provided, then only those discovered by `--outgoing` and `--workdir`
are linted.
Differential Revision: https://phabricator.services.mozilla.com/D139158
Tie into `code_analysis`'s `get_clang_tools()` functionality to
intelligently bootstrap clang if it either doesn't exist or is
out-of-date.
This required exposing `command_context` to the linting logic, as it's
needed to call `artifact_toolchain(...)`.
Note that this means that the standalone `runcli.py` file won't be able
to support bootstrapping `clang-format`, or other linters that lean on
`command_context` in the future.
Finally, `substs.get("HOST_BIN_SUFFIX")` was replaced with a
windows-specific `binary += ".exe"`, because not all contexts where
the tests are run will have access to populated `substs` data.
Note that this worked before without the extension because it was
only used for starting a process, in which context Windows automatically
tries all `PATHEXT` options. Since we're now doing an `isfile()` check
(to enable more intelligent failure cases when `clang-format` doesn't
exist), we need the path to be fully correct.
Differential Revision: https://phabricator.services.mozilla.com/D137335
This unifies the ZIP permissions in omnijars and internal XPIs
produced by the packager directly and those repacked by l10n
single-repacks.
Differential Revision: https://phabricator.services.mozilla.com/D137340
Before, clang tools would only be bootstrapped if they didn't exist.
Now, bootstrapping also occurs if the version doesn't meet requirements.
Differential Revision: https://phabricator.services.mozilla.com/D137331
* Restructure "Using third-party Python packages" page to focus on the
"Mach commands"/"adding a Python package" use case since that's why
most people will be looking at these docs.
* Document the `<site>_virtualenv_packages.txt` behaviour and how it
relates to a Mach command's definition.
* Simplify the information around using a non-PyPI index to reference
the RelEng docs directly. It's a shame that the existing docs don't
explain how to identify tasks that need to use the internal mirror,
because I'm not sure either. There's existing cases of ad-hoc `pip`
installs //not// using the mirror, but the pattern isn't clear to me.
* Remove the "specify hashes" information, since the centralized
solution (will) automatically manage this internally.
* Arguably, it's still beneficial instructions for ad-hoc
`pip install` usages, but those are frowned upon today anyways - use
the centralized solution!
Differential Revision: https://phabricator.services.mozilla.com/D138931
This change allows IPDL sources to respect FINAL_LIBRARY when building, which
is important for allowing us to build gtest-only IPDL_SOURCES files.
Differential Revision: https://phabricator.services.mozilla.com/D137167
This patch adds support for generated files to be placed in
UNIFIED_SOURCES. This will be used in part 3 to build .cpp files
generated by IPDL_SOURCES by adding them to IPDL's UNIFIED_SOURCES under
the hood. Using a unified build for IPDL files is important, as a
significant build time regression was observed locally when using
non-unified sources to build ipdl sources, due to the large number of
generated files.
Differential Revision: https://phabricator.services.mozilla.com/D137166
This replaces all of the functions in gecko_taskgraph/util/taskcluster.py with
the ones from the vendored taskgraph if they are identical.
Differential Revision: https://phabricator.services.mozilla.com/D138458
This replaces all of the functions in gecko_taskgraph/util/taskcluster.py with
the ones from the vendored taskgraph if they are identical.
Differential Revision: https://phabricator.services.mozilla.com/D138458
`coverage` has native code, so the vendored version was only used as
source code from which the actual package could be built.
Since its always used in a context where we can `pip install` over the
network, let's do that. This cleans up our tree a bit and allows us to
leverage the `coverage` wheels.
Differential Revision: https://phabricator.services.mozilla.com/D138816
Following the pytest "deprecations and removals" docs [1], this patch:
* Replaces `@pytest.yield_fixture` with `@pytest.fixture`.
* Replaces `.funcargnames` with `.fixturenames`.
* Uses `Pathlib` parameter instead of the `py.path.local` one in
associated `pytest_*` hooks.
* Replaces `--strict` with `--strict-markers`
[1] https://docs.pytest.org/en/latest/deprecations.html
Differential Revision: https://phabricator.services.mozilla.com/D138815
If previous config file paths were generated with a different
driveletter casing than the current context, then each file's contents
are replaced with `null`.
MozillaBuild 3.4 used a lowercase drive letter, and MozillaBuild 4.0
will use an uppercase drive letter.
So, using 3.4's existing objdir, and doing an incremental build with
4.0 will cause the `null` contents issue.
Adjust the files to be their `Path().resolve()`'d counterparts, so
they're string-comparable and the subset-calculation works correctly.
Differential Revision: https://phabricator.services.mozilla.com/D137807
* The python2/python3 lookup logic is obsolete, we can always depend on
`sys.executable` since Mach starts with a Python version check
* `MozbuildObject`'s `python3` property was unused, which means that all
of this removed code was dead anyways.
Differential Revision: https://phabricator.services.mozilla.com/D138824
It seems sfv 0.8.0 is used in the tree, while neqo is using the newer 0.9.1.
Updating http-sfv to use sfv 0.9.1 should remove the code duplication.
Differential Revision: https://phabricator.services.mozilla.com/D139027
This will give additional feedback to the user. Now they will have
immediate feedback in regards to the progress of the test run, instead
of being unsure of what's happening until the first test run completes.
Differential Revision: https://phabricator.services.mozilla.com/D137925
Since stdout is being piped, redirecting stderr to stdout causes it
to end up in the same pipe interleaved, which is what we want.
Also added a non-zero return code to be set when there is no test output.
Differential Revision: https://phabricator.services.mozilla.com/D138481
Logs from CI tasks have each line prefixed with date and time. Tasks like
Python linting log another timestamp how long the script has been running. This
should be omitted in automation to align it with log formats of other tasks
like mochitests which integrates it better with expectations of Treeherder's
log parser and bug filer..
Differential Revision: https://phabricator.services.mozilla.com/D138444
If there's no files to lint in CI, that's a smell that something is
misconfigured.
Resolve this by raising an exception accordingly if `MOZ_AUTOMATION` is
set (and we aren't running the tests for the linting code itself).
Automatically recommend specifying `*`, as (assuming the use of a
shell) that will resolve to "all files/dirs at $topsrcdir`", which
matches existing behaviour if `--outgoing` and `--workdir` aren't
provided.
Differential Revision: https://phabricator.services.mozilla.com/D138172
This patch adds the --proxy-perftest-page option. With this option, we'll be able to specify which test pages we want to record rather than modifying the pageload_sites.json. Note that the login fields will not be taken into consideration with this option.
You can use the flag as follows (seperate multiple pages with a comma): --proxy-perftest-page microsft,linkedin,netflix
Furthermore, some changes were made to strengthen the conditions around when a login site can be tested (only when RAPTOR_LOGINS is defined locally, or if we are in CI).
Differential Revision: https://phabricator.services.mozilla.com/D137468
There's a bug with `ProcessHandler` on Windows that sometimes causes
tests to hang for an additional 3 minutes. This is a workaround that
just replaces `ProcessHandler` with a standard Python3 approach that
achieves the same thing.
Also added some logging output as the threads are created as some
additional feedback. Prior to this, the tests silently started
running without any indication.
Differential Revision: https://phabricator.services.mozilla.com/D137822
The new `mozlint` behaviour will always lean on the VCS
to only lint changed files.
However, in tests, the VCS is going to be slow and also provide
inconsistent data between tests. Additionally, in some contexts,
the VCS won't be available.
Resolve this by hardcoding a file to lint for each case.
Differential Revision: https://phabricator.services.mozilla.com/D138033
Python 3.10 prints a warning when it sees `distutils` used, so let's
avoid using `distutils` in the "common Mach code" that runs on every
Mach call.
Note that this change will also fix the `SetuptoolsDeprecationWarning`
that shipped on `setuptools>=58.3.0`, since our usage of `distutils` was
calling a `setup.py install` under the hood.
Differential Revision: https://phabricator.services.mozilla.com/D137499
Tweaks `./mach lint` behaviour to always fall back to only linting files
that have changed according to VCS - previously, this only happened if
no linter was provided.
Adjusts "am I at $topsrcdir" check to use `pathlib` to avoid mismatches
due to inconsistent capitalization or slash direction.
Updates CI references to explicitly provide `*` as the path to avoid
the only-lint-files-changed restriction.
Differential Revision: https://phabricator.services.mozilla.com/D137870
The main difference between `OSXBootstrapperLight` and `OSXBootstrapper`
is that the "light" one doesn't install utilities via `brew`.
However, these utilities are all optional for builds (`watchman`,
`terminal-notifier`, etc).
Since arm macOS + Rosetta + brew installs are failing (because brew is
complaining that the "ARM prefix" is being used in a Rosetta context),
the solution provided by this patch is to avoid using `brew` in this
case.
Differential Revision: https://phabricator.services.mozilla.com/D137661
In addition to showing the `pip check` failure, also show:
* The `pip install --requirements ...` output
* The output of `pip list -v`
* The path to the `requirements.txt` file and the affected site.
To get sufficient detail of `pip install --requirements` to display
if the subsequent `pip check` call failed, I had to remove the
`--quiet` parameter. It was replicated by hiding _all_ output
if `quiet=True` and `pip install` succeeds - this is different to
the previous behaviour, where warnings would still poke through,
even if `quiet=True` and the install succeeded.
This was the only way to make it work such that all output would
be available without having to run `pip install` twice (which would
likely get different output the second time). Besides, there weren't
many warnings triggering this edge case, so the change in behaviour is
probably acceptable.
Additionally, in both places doing `pip check`
(post-install-requirements and when doing system-package compatibility
checking), move the `pip check` output to happen last, so that users
don't need to scroll to the top of the `pip list` wall of text to find
the reason for their command failure.
Differential Revision: https://phabricator.services.mozilla.com/D137206
This reduces some code duplication and paves the way for modifying how
Mach loads modules.
The plan is to follow this up by making this function take a more fleshed out 'spec'
that maps command names to the files they're defined at and possibly more metadata.
Since this may affect how Mach works internally (e.g. with lazy loading in the roadmap),
it makes sense to move this logic inside the Mach class.
Differential Revision: https://phabricator.services.mozilla.com/D136789
Though not a panacea, this patch addresses existing usages of
`get_repository_from_build_config()` to fall back to
`get_repository_object()` if `MissingVCSTool` was encountered.
Ideally, we'd update `substs["HG"]` when we detect that it's changed,
but we don't have infrastructure to incrementally change configure
output.
So, other issues may creep up that will still warrant a fresh
`./mach configure`.
Differential Revision: https://phabricator.services.mozilla.com/D137675
Implements keep key in moz.yaml for keeping pre-existing files in the vendor directory
Implements include key in moz.yaml for forcing inclusion of files and folders from upstream
Differential Revision: https://phabricator.services.mozilla.com/D124799
The py.path allowed adding arbitrary fields to it, while a pathlib.Path
does not. To replicate the behavior with as few changes as possible, I
made a test fixture wrapper class (RepoTestFixture) to add the same
arbitrary fields to, while now containing a Pathlib path instead.
Also added a utility functions to make the running of tests a
bit cleaner ('execute_next_step')
Differential Revision: https://phabricator.services.mozilla.com/D136302
Note: In `test_commit.py` For date formatting, the `%s` did not work on Windows.
On Linux it gives the time in seconds. The switch to the rfc822date format for
both hg and git `%aD` made the expected output in the check later work for both
git and hg, and made it platform agnostic.
Differential Revision: https://phabricator.services.mozilla.com/D136209
The compression options appear to be the same: `mozjar` defaults to
level 9 compression. The `mozjar` code is definitely slower than the
native ZIP executable, but not, I think, by enough to not do this.
This is the sole consumer of `make_zip.py` in the tree, so we remove
the file entirely.
Differential Revision: https://phabricator.services.mozilla.com/D118528
This keeps the shape of the package produced by `mach package` in the
artifact build like that of the input `Bs` package.
Differential Revision: https://phabricator.services.mozilla.com/D136671
We support splitting `MOZ_ARTIFACT_FILE` by `os.pathsep` and splitting
`MOZ_ARTIFACT_URL` by space (`" "`) to allow both the package and the
tests.common archives to be specified.
Sadly, `file://` URLs are not (yet) accepted, so there's need for the
sibling environment variables.
Differential Revision: https://phabricator.services.mozilla.com/D135939
Currently, developers don't have a way to have the Mach virtualenv
re-attempt to install optional dependencies (such as `glean`).
As part of `./mach bootstrap` (the general catch-all "re-create my dev
environment" command), we should retry installing optional dependencies.
This also matches the "glean isn't installed" error message
recommendation.
Note: This doesn't address the case in which command virtualenvs
need their optional dependencies attempted to be reinstalled.
However, since we don't have any such cases yet, I'm satisfied with
deferring that work.
Differential Revision: https://phabricator.services.mozilla.com/D123242
Our platform-specific `bootstrap` code transitively depends on
the standard library's `distutils` module, but Debian Linux and derived
distros generally split `distutils` into a separate installable OS
package from `python3` itself.
So, Python 3 being able to run isn't a sufficient guarantee that
`distutils` is available.
To catch this case, add a `distutils` check at the very beginning of
Mach initialization.
-----
Also remove an obsolete comment claiming that `mach bootstrap` doesn't
need `distutils`, which was obsoleted by Bug 1717051.
Differential Revision: https://phabricator.services.mozilla.com/D136639
What this function does can easily be a one-liner now that we're using
Pathlib, so both calls to it have been replaced by said one-liner.
Differential Revision: https://phabricator.services.mozilla.com/D135263
When installing from Python.org, the `python` command isn't available by
default (in favour of `py`). Support using `py` instead.
There were issues running `hg` when `python` isn't added to the `PATH`,
so the docs are being updated accordingly.
Of course, it's still worth supporting `py`-instead-of-`python`
workflows for `git` users who won't be impacted by that bug.
Differential Revision: https://phabricator.services.mozilla.com/D136286
The main thread is blocking (for an unknown reason) without piping the entirety of the log
file to `less` (unless the user manually scrolls to almost the bottom). Since the piping
cannot always complete without a user terminating, we must hide the errors that arise
from using the `stdin` stream for the `less` process after the process is terminated.
Differential Revision: https://phabricator.services.mozilla.com/D135979
