Commit Graph
29 Commits
Author SHA1 Message Date
Alex Ionescu 904f44227c Bug 1789229 - Add side-by-side docs r=perftest-reviewers,sparky
Add docs and fix a local bug that prevents the command from running.

Differential Revision: https://phabricator.services.mozilla.com/D157920
2022-09-23 14:33:21 +00:00
Johan Lorenzo 2aa213630b Bug 1784232 - Bump taskgraph to v3.2.0 r=ahal
Differential Revision: https://phabricator.services.mozilla.com/D155978
2022-09-20 09:11:45 +00:00
Jan-Erik Rediger 1f85ca4b91 Bug 1788040 - Update to Glean v51.2.0. r=chutten,supply-chain-reviewers
Differential Revision: https://phabricator.services.mozilla.com/D156937
2022-09-13 12:57:58 +00:00
Jan-Erik Rediger 03c4ad04b1 Bug 1783754 - Update to Glean v51.1.0. r=TravisLong,supply-chain-reviewers
Differential Revision: https://phabricator.services.mozilla.com/D154088
2022-08-09 15:58:51 +00:00
Jan-Erik Rediger 85dcae1bb7 Bug 1781030 - Upgrade to Glean v51.0.1. r=perry.mcmanis,supply-chain-reviewers
Differential Revision: https://phabricator.services.mozilla.com/D152626
2022-08-01 10:50:47 +00:00
ahochheiden 02d22d0bc3 Bug 1777414 - Vendor standalone 'looseversion' at version 1.0.1 r=firefox-build-system-reviewers,glandium
This will be used to replace the `LooseVersion` within `distutils`.
`StrictVersion` from `distutils` will need something else, as swapping
usages of `StrictVersion` with `LooseVersion` does not result in the
desired behavior.

Differential Revision: https://phabricator.services.mozilla.com/D151062
2022-07-29 02:03:37 +00:00
Jan-Erik Rediger 84f999cd3c Bug 1781085 - Update to Glean v50.1.3 r=chutten,supply-chain-reviewers
Differential Revision: https://phabricator.services.mozilla.com/D152770
2022-07-27 10:24:48 +00:00
criss 2d3106819f Backed out changeset 581d7209c242 (bug 1781085) for causing build bustages. CLOSED TREE 2022-07-27 12:19:52 +03:00
Jan-Erik Rediger ba68651b23 Bug 1781085 - Update to Glean v50.1.3 r=chutten,supply-chain-reviewers
Differential Revision: https://phabricator.services.mozilla.com/D152770
2022-07-27 08:58:11 +00:00
Travis Long 3248e25004 Bug 1772132 - Update Glean and remove cargo metadata replacement. r=janerik,supply-chain-reviewers
Differential Revision: https://phabricator.services.mozilla.com/D151499
2022-07-14 13:11:13 +00:00
Norisz Fay cb23b29e60 Backed out changeset ce93635100b6 (bug 1772132) for causing webrender build bustage CLOSED TREE 2022-07-13 22:08:19 +03:00
Travis Long 45b430b958 Bug 1772132 - Update Glean and remove cargo metadata replacement. r=janerik,supply-chain-reviewers
Differential Revision: https://phabricator.services.mozilla.com/D151499
2022-07-13 18:00:35 +00:00
Bogdan Szekely 3f3ff21488 Backed out changeset 1ff9bb6e4a8c (bug 1772132) for causing build bustage. CLOSED TREE 2022-07-13 16:17:53 +03:00
Travis Long 464b4a3db3 Bug 1772132 - Update Glean and remove cargo metadata replacement. r=janerik,supply-chain-reviewers
Differential Revision: https://phabricator.services.mozilla.com/D151499
2022-07-13 12:59:16 +00:00
smolnar 03f912e7ed Backed out changeset ab4790cd94eb (bug 1772132) for causing wrench bustage CLOSED TREE 2022-07-13 12:04:13 +03:00
Travis Long 458cd6f54c Bug 1772132 - Update Glean and remove cargo metadata replacement. r=janerik,supply-chain-reviewers
Differential Revision: https://phabricator.services.mozilla.com/D151499
2022-07-13 08:43:41 +00:00
Jan-Erik Rediger e447a2597f Bug 1773255 - Update to Glean v50.1.0 r=chutten,supply-chain-reviewers
Including glean_parser 6.1.1

Two important things in there:
* glean_parser: [data-review] Include extra keys' names and descriptions in data review template
* Glean: Derive `serde::{Deserialize, Serialize}` on `Lifetime` and `CommonMetricData`

Differential Revision: https://phabricator.services.mozilla.com/D149381
2022-06-15 15:39:33 +00:00
Jan-Erik Rediger 4ca6841988 Bug 1768834 - Switch to Glean with UniFFI integration r=glandium,chutten
Upgrades to Glean v50.0.1, which comes with a rewritten core and
UniFFI-powered bindings.
Glean has some API changes, so we swap it over to that. Mostly mechanical changes.
Also upgrades to inherent v1.0 in fog.
This matches what Glean uses internally and gets rid of one duplicated crate.

Also upgrades to glean-parser==6.0.1

One crate duplication now (change in `python/mozbuild/mozbuild/vendor/vendor_rust.py` required).
Some new crates now vendored.
These are transitive dependencies of Glean dependencies, all with valid
licenses and already used in other products (mobile).

Differential Revision: https://phabricator.services.mozilla.com/D146062
2022-06-07 12:37:20 +00:00
Mitchell Hentges 1ad3706fbc Bug 1763695: Move python-test poetry to ini "requirements" r=ahal
The `test_vendor.py` test needs access to `poetry` to make its
assertions. This requirement was previously satisfied by adding `poetry`
to the `python-test` site.

Unfortunately, this caused conflicts when running the `mozlint` tests,
as they use the ini "requirements" feature to install their
dependencies. Importantly, the `mozlint` dependencies aren't compatible
with the version of `poetry` we need.

Fortunately, this was a good reminder: since `poetry` is a test-specific
dependency, leverage the existing ini "requirements" feature as it
represents exactly that: the extra package dependencies of a specific
test.

Note that I had to doctor the `dataclass` line in
`vendor_requirements.txt` to include an environment marker, otherwise it
would fail to install with Python >= 3.7.

Differential Revision: https://phabricator.services.mozilla.com/D143376
2022-04-19 19:10:38 +00:00
Chris H-C ad3310802e Bug 1764549 - Update Glean SDK to v44.1.1 r=TravisLong
Differential Revision: https://phabricator.services.mozilla.com/D143749
2022-04-14 20:35:08 +00:00
Mitchell Hentges f80023da60 Bug 1764711: Move boto3 to requirement file r=ahal
The `upload_geneated_sources.py` script needs `boto3`, but was:
* Running in CI with Mach having a site-packages-source of `None`
* Using the `common` site (which inherits Mach's site-packages-source)
* Installing `boto3` outside of the knowledge of the centralized
  dependency system.

Make a `upload-generated-sources` site, and move `boto3` to it.

Differential Revision: https://phabricator.services.mozilla.com/D143715
2022-04-14 18:00:30 +00:00
Mike Hommey 47fdd6b7f0 Bug 1289641 - Vendor the vsdownload script. r=firefox-build-system-reviewers,mhentges
Differential Revision: https://phabricator.services.mozilla.com/D143125
2022-04-12 22:52:53 +00:00
Butkovits Atila 4e5f6bf343 Backed out 7 changesets (bug 1289641) for causing Python failures. CLOSED TREE
Backed out changeset 0c1cedebf915 (bug 1289641)
Backed out changeset 575732020bd2 (bug 1289641)
Backed out changeset 55b68a71e03a (bug 1289641)
Backed out changeset 0b16a05aa405 (bug 1289641)
Backed out changeset acbff13b91b3 (bug 1289641)
Backed out changeset 2645367da7cb (bug 1289641)
Backed out changeset 8b22a8ec44fe (bug 1289641)
2022-04-12 22:23:27 +03:00
Mitchell Hentges 0adecbdafd Bug 1761150: Mach should use fresh system python, don't inherit sys.path r=ahal
Previously, when using the system Python packages, Mach would reuse the
values already existing in the `sys.path`. This had two benefits:
1. We didn't have to do work to calculate which paths the "system
   Python" specifically referred to.
2. This allowed us to support nested Mach calls
   (such as `./mach --virtualenv psutil ./mach build`).

However, it came with its own headaches, specifically around "consistent
imports" and Python subprocesses, such as in the following example
1. Mach runs "using system Python"
2. The "build" site (for example) would be activated. The current
   `sys.path` is included in the virtualenv's `mach.pth` file so that
   subprocesses will have access to the same packages as the primary
   Mach process
3. //Something// adds something to the `sys.path`
4. The build virtualenv is redundantly re-activated (such as due to
   generic setup code for some module). Though we don't physically
   re-activate the virtualenv, we //do// check that it's up-to-date to
   validate assumptions - **and it's not!** It's missing the
   //something// that was added to the `sys.path`.
5. We can't re-create the build virtualenv because it's already active,
   so:
6. An error is raised so that unexpected out-of-date-ness doesn't fly
   under the radar.

-----

This patch solves this by calculating the "system Python" paths in a
deterministic way. Then, even when using the system Python, the Mach and
command sites use define themselves in a consistent way.

This means that we can't do `./mach --virtualenv <venv> ./mach ...` to
shares `venv`'s packages with the whole new Mach process. Fortunately,
this has been replaced with  moving such packages into the nested Mach
command's requirements  definition instead.

TL;DR: more consistent, less failures. 👍

-----

One more detail, this time around the `sys_path()` function:
Ideally, we could calculate the standard library paths *and* the
"system" paths with only one Python subprocess. Unfortunately,
that's not the case:
* If `-S` isn't provided, then it's tricky to separate the standard
  library paths from the system paths
* If `-S` is provided, then doing `site.getsitepackages()` in a
  virtualenv returns the *system* site packages, not the virtualenv's.

To work around this, we call the external python twice, and in
parallel. This allows resolving the information we need while avoiding
performance costs.

Differential Revision: https://phabricator.services.mozilla.com/D143200
2022-04-12 18:40:14 +00:00
Mike Hommey 520d40c3d2 Bug 1289641 - Vendor the vsdownload script. r=firefox-build-system-reviewers,mhentges
Differential Revision: https://phabricator.services.mozilla.com/D143125
2022-04-11 23:00:08 +00:00
Jan-Erik Rediger 05d0f00624 Bug 1763602 - Update to Glean v44.1.0. r=chutten
Differential Revision: https://phabricator.services.mozilla.com/D143162
2022-04-11 08:25:00 +00:00
Mitchell Hentges d62f3b8a4e Bug 1659593: Make ./mach vendor python cross-platform r=ahal
Use `poetry` instead of `pip-compile`, then leverage `pip` to assert
that there's no conflicts between environments.

This enables simplifying `requirements.in`, since `colorama` is now
automatically picked up, as vendoring is no longer linux-py3.6-specific.

`poetry==1.2.0a2` was chosen because it was the first version that would
include `pip`/`setuptools`/`wheel` in the `requirements.txt`, yet it was
also the last version that supports Python 3.6 (as still used in CI in
some spots).

Sorry about the `requirements.txt` noise, that's because poetry doesn't
sort hashes yet [1]. Fortunately, you can be confident in
`requirements.txt` changes by checking that this patch doesn't include
changes to vendored packages themselves (I'm including the results of a
`./mach vendor python`).

[1] https://github.com/python-poetry/poetry/issues/5408

Differential Revision: https://phabricator.services.mozilla.com/D142884
2022-04-07 18:00:28 +00:00
Mitchell Hentges bcba18107d Bug 1732795: Update pytest to 7.0.1 r=ahal
`4.6.6` was 3 years old. There's been many helpful features and
ergonomics improvements in the years since.

Differential Revision: https://phabricator.services.mozilla.com/D140679
2022-04-04 17:03:33 +00:00
Mitchell Hentges eb3080a220 Bug 1724274: Move virtualenv dependency manifests to python/virtualenvs r=ahal
This is needed for a few reasons:
* All mach commands can use virtualenvs, not just `build`-related
  commands, so the files don't make sense to be in `build/`.
* When locking is added, more files associated with virtualenvs will be
  added, and this will change will ease the related directory structure
  setup.
* This removes the need for a redundant "_virtualenv_packages" keyword
  as part of the manifest filenames.

Differential Revision: https://phabricator.services.mozilla.com/D140382
2022-03-24 14:04:34 +00:00