This adds the `-E/--use-existing-tasks` flag to the `fuzzy` or `chooser`
selectors. For example:
./mach try fuzzy -E
This queries the Treeherder API to find your latest try push, then any tasks
that exist on that push, but aren't explicitly selected this time, will be
replaced with those older tasks.
You can also run:
./mach try fuzzy -E task-id=<task id>
Where `<task id>` is the id of the Decision task from the push you would like
to use as the base.
Differential Revision: https://phabricator.services.mozilla.com/D200380
This adds the `-E/--use-existing-tasks` flag to the `fuzzy` or `chooser`
selectors. For example:
./mach try fuzzy -E
This queries the Treeherder API to find your latest try push, then any tasks
that exist on that push, but aren't explicitly selected this time, will be
replaced with those older tasks.
You can also run:
./mach try fuzzy -E task-id=<task id>
Where `<task id>` is the id of the Decision task from the push you would like
to use as the base.
Differential Revision: https://phabricator.services.mozilla.com/D200380
If this is not done, the next person to vendor will also make all of the changes in this commit, which is unnecessary noise.
Differential Revision: https://phabricator.services.mozilla.com/D183450
If this is not done, the next person to vendor will also make all of the changes in this commit, which is unnecessary noise.
Differential Revision: https://phabricator.services.mozilla.com/D183450
Add support for submitting stacks of commits to Lando for queueing
on Try, instead of pushing to hg.mozilla.org directly. This patch
implements the Device Code Authorization flow for Auth0, simple changeset
discovery and patch gathering, and submissing to Lando via HTTP POST.
Add a `try.txt` virtualenv site that contains packages from the common
virtualenv as well as the `auth0-python` package for verifying Auth0
JWTs. Use this new virtualenv for `mach try` and related subcommands.
Add a `--push-to-lando` flag that controls whether the push will be made
via the Lando API or using the VCS to hg.mozilla.org directly.
Create a `lando.py` module in the `tryselect` package that handles the
details around submitting to Lando. Authentication is handled by the
Device Code Authorization flow, and the returned access token is saved
to the mozbuild state directory. Auth0 details are added to the `.lando.ini`
file in the repo root, and a `LANDO_TRY_USE_DEV` environment variable can
be set to control submitting to the prod or dev Lando environments.
This module also includes patch stack gathering and discovery via `mozversioncontrol`.
mozversioncontrol's `Repository` subclass is extended with helper functions
to gather patch files from Mercurial and Git. We also add a `try_config_commit`
context manager that creates a temporary commit to hold try syntax commit
messages and/or a `try_task_config.json` file, which is removed from version
control on submission.
The `mach try` cram tests now use a separate virtualenv that must be built
before running, causing unexpected output in the tests. Run `mach try --help`
in the test setup to force the virtualenv to be built before running any
test.
`mach try chooser` would previously install packages for a small web application
from a `requirements.txt` file when run. Install these via the new `try.txt`
site environment instead. The required dependencies are also added to the
`python-test.txt` site since they will cause some tests to fail due to the
missing dependencies.
Differential Revision: https://phabricator.services.mozilla.com/D187641
D180501 made the original change that this patch undoes. An earlier
change in that stack inadvertently silently disabled Telemetry, which
hid this issue.
The problem is that `glean` (which we optionally install in the `mach`
site also needs `jsonschema` (and `jsonschema` depends on `pyrsistent`
and `pkgutil_resolve_name`). Since our vendored version was no longer
specified in the `mach.txt` site, it pulled a different version
(`4.19.1`) from the vendored version (`4.17.3`) which causes issues in
the `wpt` site.
Since glean explicitly needs `jsonschema` it never should have been
taken out of the `mach` site, so this patch corrects that mistake.
Differential Revision: https://phabricator.services.mozilla.com/D189553
Update `aiohttp` to version 3.8.5 and `requests` to version 2.31.0,
and vendor their respective dependencies. Add all the new dependencies
to the various required site virtualenv requirements files.
Differential Revision: https://phabricator.services.mozilla.com/D188904
ManifestParser will read TOML files, if present, when use_toml=True
Added tomlkit as a third_party python package
Added poetry-core and tomlkit to pypi (separately as Bug 1845383, Bug 1844787)
Adds TOML test coverage
Adds tomlkit as a dependency of mozharness (in test_archive.py)
Added tomlkit to virtualenv_modules in testing/mozharness/configs/unittests
Removes dependency on six
testing/tools/mach_test_package_initialize.py
- Corrected SEARCH_PATHS
testing/mozharness/mozharness/mozilla/testing/per_test_base.py
- moved `from manifestparser import TestManifest` into function call
to avoid harness inability to locate the internal artifact
- Removed linter warnings
testing/mozbase/manifestparser/manifestparser/manifestparser.py
- Removed linter warnings
- Updated logger usage pattern
- Simplifed _read logic, refactored get_fp_filename()
- Improve context for `include:` logging message
- Defer `import mozlog` until the point of use
testing/mozbase/manifestparser/manifestparser/toml.py
- Removed linter warnings
- Removed unused logger
- Improved readability of read_toml()
testing/mozbase/manifestparser/manifestparser/ini.py
- Removed linter warnings
- Removed unused logger
testing/mozbase/manifestparser/manifestparser/filters.py
- Removed linter warnings
testing/mozbase/manifestparser/tests/test_chunking.py
- Removed linter warnings
Bumped manifestparser version to 2.2.31
Differential Revision: https://phabricator.services.mozilla.com/D184020
Now that we selectively load command modules, and activate the command
virtualenv much earlier in the mach process, a lot of the module
dependencies specified in `mach.txt` are no longer necessary there. With
their removal from `mach.txt` they will no longer be automatically
inherited by every site, which reduces potential dependency conflicts
for specific sites.
The `common` site still effectively has the same set of dependencies.
This is the default site that all commands use unless otherwise
specified. Most commands use this site, and going through every command
and seeing if a dependency is or isn't needed, then deciding if or if
not to create a new site for that command made sense was too time
consuming to do here.
Essentially the idea here going forward is that if you're trying to
add/update a new dependency to a command that is currently defaulting to
the `common` site and there is a conflict with one of the dependencies
in `common` you can move your command to a new site specifically for
your command, and you will have the minimal possible set of dependencies
a mach command can have, improving the odds that you can add the
module(s) you need for your command.
Depends on D180500
Differential Revision: https://phabricator.services.mozilla.com/D180501
Now that we selectively load command modules, and activate the command
virtualenv much earlier in the mach process, a lot of the module
dependencies specified in `mach.txt` are no longer necessary there. With
their removal from `mach.txt` they will no longer be automatically
inherited by every site, which reduces potential dependency conflicts
for specific sites.
The `common` site still effectively has the same set of dependencies.
This is the default site that all commands use unless otherwise
specified. Most commands use this site, and going through every command
and seeing if a dependency is or isn't needed, then deciding if or if
not to create a new site for that command made sense was too time
consuming to do here.
Essentially the idea here going forward is that if you're trying to
add/update a new dependency to a command that is currently defaulting to
the `common` site and there is a conflict with one of the dependencies
in `common` you can move your command to a new site specifically for
your command, and you will have the minimal possible set of dependencies
a mach command can have, improving the odds that you can add the
module(s) you need for your command.
Depends on D180500
Differential Revision: https://phabricator.services.mozilla.com/D180501
Now that we selectively load command modules, and activate the command
virtualenv much earlier in the mach process, a lot of the module
dependencies specified in `mach.txt` are no longer necessary there. With
their removal from `mach.txt` they will no longer be automatically
inherited by every site, which reduces potential dependency conflicts
for specific sites.
The `common` site still effectively has the same set of dependencies.
This is the default site that all commands use unless otherwise
specified. Most commands use this site, and going through every command
and seeing if a dependency is or isn't needed, then deciding if or if
not to create a new site for that command made sense was too time
consuming to do here.
Essentially the idea here going forward is that if you're trying to
add/update a new dependency to a command that is currently defaulting to
the `common` site and there is a conflict with one of the dependencies
in `common` you can move your command to a new site specifically for
your command, and you will have the minimal possible set of dependencies
a mach command can have, improving the odds that you can add the
module(s) you need for your command.
Depends on D180500
Differential Revision: https://phabricator.services.mozilla.com/D180501
The `test_vendor.py` test needs access to `poetry` to make its
assertions. This requirement was previously satisfied by adding `poetry`
to the `python-test` site.
Unfortunately, this caused conflicts when running the `mozlint` tests,
as they use the ini "requirements" feature to install their
dependencies. Importantly, the `mozlint` dependencies aren't compatible
with the version of `poetry` we need.
Fortunately, this was a good reminder: since `poetry` is a test-specific
dependency, leverage the existing ini "requirements" feature as it
represents exactly that: the extra package dependencies of a specific
test.
Note that I had to doctor the `dataclass` line in
`vendor_requirements.txt` to include an environment marker, otherwise it
would fail to install with Python >= 3.7.
Differential Revision: https://phabricator.services.mozilla.com/D143376
Use `poetry` instead of `pip-compile`, then leverage `pip` to assert
that there's no conflicts between environments.
This enables simplifying `requirements.in`, since `colorama` is now
automatically picked up, as vendoring is no longer linux-py3.6-specific.
`poetry==1.2.0a2` was chosen because it was the first version that would
include `pip`/`setuptools`/`wheel` in the `requirements.txt`, yet it was
also the last version that supports Python 3.6 (as still used in CI in
some spots).
Sorry about the `requirements.txt` noise, that's because poetry doesn't
sort hashes yet [1]. Fortunately, you can be confident in
`requirements.txt` changes by checking that this patch doesn't include
changes to vendored packages themselves (I'm including the results of a
`./mach vendor python`).
[1] https://github.com/python-poetry/poetry/issues/5408
Differential Revision: https://phabricator.services.mozilla.com/D142884
This is needed for a few reasons:
* All mach commands can use virtualenvs, not just `build`-related
commands, so the files don't make sense to be in `build/`.
* When locking is added, more files associated with virtualenvs will be
added, and this will change will ease the related directory structure
setup.
* This removes the need for a redundant "_virtualenv_packages" keyword
as part of the manifest filenames.
Differential Revision: https://phabricator.services.mozilla.com/D140382
Andrew Halberstadt