1. add the MOZ_X11 config flag in build/gn.mozbuild and set the gn_vars
accordingly.
2. create the new gn-config/mozconfig files and delete previous ones
dom/media/webrtc/third_party_build/gn-configs/**.mozconfig with
--enable-default-toolkit=cairo-gtk3-wayland-only for the non X11 version.
New toolkit nmae is required to force disable X11 detection as
cairo-gtk3-wayland will auto-detect X11 and make generate-gn-build-files.sh
fail.
3. Add the MOZ_X11 config flag in python/mozbuild/mozbuild/gn_processor.py
Then run
dom/media/webrtc/third_party_build/gn-configs/generate-gn-build-files.sh
Differential Revision: https://phabricator.services.mozilla.com/D142904
Until clang 14, the default was -ffp-contract=fast, but some reason
didn't really enable FP contraction in practice. GCC has the same
default, but does, in fact enable FP contraction*. This leads to more
inconsistencies in FP results between builds than would normally be
expected, and has an effect on e.g. fingerprinting resistance in fdlibm
trigonometric functions, or some other places (e.g. bug 1765436).
As far as the Firefox versions Mozilla ships are concerned, FP
contraction only really affects aarch64, but this change should in
practice make no difference with the current version of clang used (13),
and would preserve our now new default behavior with clang 14.
(*) We never noticed because we neither build with GCC for arm64 nor run
tests with GCC builds.
Differential Revision: https://phabricator.services.mozilla.com/D144136
The `test_vendor.py` test needs access to `poetry` to make its
assertions. This requirement was previously satisfied by adding `poetry`
to the `python-test` site.
Unfortunately, this caused conflicts when running the `mozlint` tests,
as they use the ini "requirements" feature to install their
dependencies. Importantly, the `mozlint` dependencies aren't compatible
with the version of `poetry` we need.
Fortunately, this was a good reminder: since `poetry` is a test-specific
dependency, leverage the existing ini "requirements" feature as it
represents exactly that: the extra package dependencies of a specific
test.
Note that I had to doctor the `dataclass` line in
`vendor_requirements.txt` to include an environment marker, otherwise it
would fail to install with Python >= 3.7.
Differential Revision: https://phabricator.services.mozilla.com/D143376
1. add the MOZ_X11 config flag in build/gn.mozbuild and set the gn_vars
accordingly.
2. create the new gn-config/mozconfig files and delete previous ones
dom/media/webrtc/third_party_build/gn-configs/**.mozconfig with
--enable-default-toolkit=cairo-gtk3-wayland-only for the non X11 version.
New toolkit nmae is required to force disable X11 detection as
cairo-gtk3-wayland will auto-detect X11 and make generate-gn-build-files.sh
fail.
3. Add the MOZ_X11 config flag in python/mozbuild/mozbuild/gn_processor.py
Then run
dom/media/webrtc/third_party_build/gn-configs/generate-gn-build-files.sh
Differential Revision: https://phabricator.services.mozilla.com/D142904
This makes some ./mach vendor improvements needed:
- Adds a replace-in-file-regex action
- Allows you to skip copying the temporary extract directory
so that a script can do it
- Removes 'commit' from the look-in-moz.yaml-for-release match
Differential Revision: https://phabricator.services.mozilla.com/D142720
This change does a few things
- Moves get_full_path from a function-local function to
a class member
- Makes yaml_file a class member so I don't need to pass
it everywhere
- Creates a new moz.yaml file directive 'generated'
- Adds that directive to libdav1d
- Detects if any files (other than generated ones) were
modified by an update; and if not - do not update the
moz.yaml. Exit with -2 so Updatebot can detect it.
Differential Revision: https://phabricator.services.mozilla.com/D141994
The `upload_geneated_sources.py` script needs `boto3`, but was:
* Running in CI with Mach having a site-packages-source of `None`
* Using the `common` site (which inherits Mach's site-packages-source)
* Installing `boto3` outside of the knowledge of the centralized
dependency system.
Make a `upload-generated-sources` site, and move `boto3` to it.
Differential Revision: https://phabricator.services.mozilla.com/D143715
* There is no `--duration` argument, but there //is// `--durations`
* `pytest` behaves better when called via its main entry points (either
`bin/pytest`, or `-m pytest`)
Differential Revision: https://phabricator.services.mozilla.com/D143609
De-duping with `set` causes order to no longer be maintained when the
items are distinct. Use the `OrderedDict` technique to de-dupe
consistently instead.
Differential Revision: https://phabricator.services.mozilla.com/D143604
I believe this is safe because from what I've been able to find
SafeLoader is FullLoader but without the bad stuff that lets you
execute arbitrary python code. We don't need (nor want) the more
comlpicated data types like 'dates' and 'numbers' so we can use
BaseLoader which does nothing except treat everything as a string.
Depends on D142125
Differential Revision: https://phabricator.services.mozilla.com/D142126
Both the dtd and properties files strings are moved to the existing storage.ftl.
A migration file was added to keep the localized strings.
Some of the strings were migrated to a declarative approach, setting `data-l10n-id` and
`data-l10n-args` attributes on elements. But in other cases, this was not easily doable.
To accomodate with the asynchronicity of `formatValue`, we translate all the strings that
don't have dynamic parameters during the initialization of the panel, and store them in
a Map for easy retrieval.
We also took this opportunity to cleanup the strings key names and have them all prefixed
with `storage-*`. Some functions where refactored so it should be easier to search the
code for a specific l10n string.
Finally, the `table.headers.indexedDB.*` strings weren't migrated as they were never read
in the UI (they are referenced in `NON_L10N_STRINGS`, which bypass localization).
Differential Revision: https://phabricator.services.mozilla.com/D143012
Verify that activating mach and command sites in various configurations
affects the `sys.path` accurately.
Also verify that the `sys.path` is correct between the "activated
process" and fresh processes spawned from created virtualenvs.
Finally, tweak existing path-management behaviour to be more consistent:
* Don't unnecessarily sort `pths` from requirement definitions - it's
more consistent (and more dependable) to keep the provided order.
* Continue removing unwanted `venv` entries from the `sys.path`, but
don't add them back (such as the `$prefix` entry)
* Always remove `venv` entries from the `sys.path` when the venv isn't
being populated. Though this doesn't affect behaviour, it makes
assertions easier and simplifies the runtime state a bit.
Differential Revision: https://phabricator.services.mozilla.com/D143201
Previously, when using the system Python packages, Mach would reuse the
values already existing in the `sys.path`. This had two benefits:
1. We didn't have to do work to calculate which paths the "system
Python" specifically referred to.
2. This allowed us to support nested Mach calls
(such as `./mach --virtualenv psutil ./mach build`).
However, it came with its own headaches, specifically around "consistent
imports" and Python subprocesses, such as in the following example
1. Mach runs "using system Python"
2. The "build" site (for example) would be activated. The current
`sys.path` is included in the virtualenv's `mach.pth` file so that
subprocesses will have access to the same packages as the primary
Mach process
3. //Something// adds something to the `sys.path`
4. The build virtualenv is redundantly re-activated (such as due to
generic setup code for some module). Though we don't physically
re-activate the virtualenv, we //do// check that it's up-to-date to
validate assumptions - **and it's not!** It's missing the
//something// that was added to the `sys.path`.
5. We can't re-create the build virtualenv because it's already active,
so:
6. An error is raised so that unexpected out-of-date-ness doesn't fly
under the radar.
-----
This patch solves this by calculating the "system Python" paths in a
deterministic way. Then, even when using the system Python, the Mach and
command sites use define themselves in a consistent way.
This means that we can't do `./mach --virtualenv <venv> ./mach ...` to
shares `venv`'s packages with the whole new Mach process. Fortunately,
this has been replaced with moving such packages into the nested Mach
command's requirements definition instead.
TL;DR: more consistent, less failures. 👍
-----
One more detail, this time around the `sys_path()` function:
Ideally, we could calculate the standard library paths *and* the
"system" paths with only one Python subprocess. Unfortunately,
that's not the case:
* If `-S` isn't provided, then it's tricky to separate the standard
library paths from the system paths
* If `-S` is provided, then doing `site.getsitepackages()` in a
virtualenv returns the *system* site packages, not the virtualenv's.
To work around this, we call the external python twice, and in
parallel. This allows resolving the information we need while avoiding
performance costs.
Differential Revision: https://phabricator.services.mozilla.com/D143200
Use `poetry` instead of `pip-compile`, then leverage `pip` to assert
that there's no conflicts between environments.
This enables simplifying `requirements.in`, since `colorama` is now
automatically picked up, as vendoring is no longer linux-py3.6-specific.
`poetry==1.2.0a2` was chosen because it was the first version that would
include `pip`/`setuptools`/`wheel` in the `requirements.txt`, yet it was
also the last version that supports Python 3.6 (as still used in CI in
some spots).
Sorry about the `requirements.txt` noise, that's because poetry doesn't
sort hashes yet [1]. Fortunately, you can be confident in
`requirements.txt` changes by checking that this patch doesn't include
changes to vendored packages themselves (I'm including the results of a
`./mach vendor python`).
[1] https://github.com/python-poetry/poetry/issues/5408
Differential Revision: https://phabricator.services.mozilla.com/D142884
The PGO-specific objdir is meant for the PGO build artifacts.
The python virtualenvs just exist for the host, so there's no need to
scope them. Besides, the existing virtualenv infrastructure doesn't know
how to distinguish between "main 'build' site" and "PGO 'build' site",
which is why it raised this "out-of-date" error.
Differential Revision: https://phabricator.services.mozilla.com/D142373
Now that we are gleaning a site packages source in a process-global way
(that is *not* conditional on the site that's being managed), we've
removed an edge case from occurring: the case where Mach would use
`SitePackagesSource.NONE`, but the command site would use
`SitePackagesSource.SYSTEM`. This would've occurred if none of Mach's
dependencies were found in the system Python, but instead once the
command site was initialized, one of *its* dependencies were located.
Since that can no longer happen:
* Command sites that don't populate their VENV will *always* use the
same site packages source as Mach
* `pthfile` generation is simplified (the priority of the system paths
is no longer variable)
* We no longer need to track `site_packages_source` in metadata, since
we can use `mach_site_packages_source` instead.
While here, I moved the "`PIP_NETWORK_INSTALL_RESTRICTED_VIRTUALENVS`
sites shouldn't have `pypi_requirements`" check to a test, since there's
no need for it to happen at runtime.
Differential Revision: https://phabricator.services.mozilla.com/D140668
