It turns out subprocess functions have the annoying behavior of changing
line endings in `input` when using `text=True` or `universal_newlines=True`,
or even `encoding="utf-8"`. And git fast-import doesn't like `\r\n`.
Differential Revision: https://phabricator.services.mozilla.com/D215158
Currently, the caller potentially prepares files, add them to the VCS,
and then calls the function to create the commit, does whatever it wants
to do with it, then undoes it.
With the ultimate goal of not doing local changes to the work tree to
act on that try commit (for a push), this change in API makes it so
that everything except what do do with the commit is handled by one
function, that now takes paths and file contents for the files that were
previously created by the caller.
Differential Revision: https://phabricator.services.mozilla.com/D214214
The previous implementation used days since Jan 1 2000 for the last
16-bit segment. This was not unique enough and caused issues with
Antivirus software if two different channels were built on the same day.
The new approach uses hours since the last milestone bump and uses the
VCS to determine how long ago that was relative to the build time. This
means it will always reset when a new cycle begins, but still be unique
since the digits in the first 3 segments have incremented.
We also now use two of the 16-bits to encode the channel (nightly, beta,
ESR, and release). So two channels built within the same hour will still
be unique.
Using only 14-bits to store the 'hours since version bump', we have
about ~682 days from a version bump before we reach the maximum value we
can store. If a build is done after that point, the segment value will
always be the maximum value for that channel.
Differential Revision: https://phabricator.services.mozilla.com/D200989
The previous implementation used days since Jan 1 2000 for the last
16-bit segment. This was not unique enough and caused issues with
Antivirus software if two different channels were built on the same day.
The new approach uses hours since the last milestone bump and uses the
VCS to determine how long ago that was relative to the build time. This
means it will always reset when a new cycle begins, but still be unique
since the digits in the first 3 segments have incremented.
We also now use two of the 16-bits to encode the channel (nightly, beta,
ESR, and release). So two channels built within the same hour will still
be unique.
Using only 14-bits to store the 'hours since version bump', we have
about ~682 days from a version bump before we reach the maximum value we
can store. If a build is done after that point, the segment value will
always be the maximum value for that channel.
Differential Revision: https://phabricator.services.mozilla.com/D200989
The previous implementation used days since Jan 1 2000 for the last
16-bit segment. This was not unique enough and caused issues with
Antivirus software if two different channels were built on the same day.
The new approach uses hours since the last milestone bump and uses the
VCS to determine how long ago that was relative to the build time. This
means it will always reset when a new cycle begins, but still be unique
since the digits in the first 3 segments have incremented.
We also now use two of the 16-bits to encode the channel (nightly, beta,
ESR, and release). So two channels built within the same hour will still
be unique.
Using only 14-bits to store the 'hours since version bump', we have
about ~682 days from a version bump before we reach the maximum value we
can store. If a build is done after that point, the segment value will
always be the maximum value for that channel.
Differential Revision: https://phabricator.services.mozilla.com/D200989
Adds a CLI option `--l10n-git` that uses the firefox-l10n-source git repo instead of the gecko-strings hg repo when linting localization files.
When we complete the migration, the option and mercurial repo support should be removed.
Differential Revision: https://phabricator.services.mozilla.com/D199866
A previous patch added parsing `git remote -v`, preferring
`mozilla-unified` as the official upstream remote and falling
back to the first official-looking remote if unified was not
found. We assumed that a developer is either using mozilla-unified,
or is using a single-headed repo like central. This overlooks
the fact that it is possible to clone from central, and then
pull other repos in, creating a repo with multiple official
remotes.
Update the `get_upstream_remote` function to find all official
looking remotes instead of a single official remote, and change
the `get_remote_arg` function to return a list of `--remotes`
arguments to be passed to various Git commands. This allows
Git to take all official remotes into consideration and more
precisely find the commits which are not present on any official
upsteams.
Add a test for `get_mozilla_remote_args` while we are here.
Differential Revision: https://phabricator.services.mozilla.com/D199637
In bug 1866395 I added a function to return the canonical Firefox
remote and use it when determining the base ref. I overlooked
that this also needs to be used for `get_branch_nodes` to avoid
returning a set of branch nodes that doesn't begin from the same
base ref. Move the building of the `--remotes` arg to a separate
function and use it in `get_branch_nodes`.
Differential Revision: https://phabricator.services.mozilla.com/D198032
Add a `get_mozilla_upstream_remote` method for `GitRepository` that
scans the output of `git remote -v` for an official Mozilla URL. Use
this URL in `base_ref` to limit the remotes considered for remote
branches, defaulting to `--remotes` without an argument if one can't
be found.
Differential Revision: https://phabricator.services.mozilla.com/D194881
Git versions before 2.41 don't display headers for empty commits
in `git format-patch`. Force headers to display with `--always`,
so Lando can properly apply the patch when pushing to try.
Differential Revision: https://phabricator.services.mozilla.com/D194911
`mozversioncontrol` runs currently use `universal_newlines`, which
is an alias for `text`. This forces the command output file object
to be opened in text mode using the platform's default encoding. On
Windows the default encoding is usually a single-byte encoding such
as `cp-1252`, which can cause issues when parsing patches that include
multi-byte sequences. Add an `encoding` option to `_run` with `utf-8`
as the default option and change the `check_output` call to use it.
Callers can revert to the previous behaviour quite easily by setting
`encoding=None` if necessary.
Differential Revision: https://phabricator.services.mozilla.com/D190710
Add support for submitting stacks of commits to Lando for queueing
on Try, instead of pushing to hg.mozilla.org directly. This patch
implements the Device Code Authorization flow for Auth0, simple changeset
discovery and patch gathering, and submissing to Lando via HTTP POST.
Add a `try.txt` virtualenv site that contains packages from the common
virtualenv as well as the `auth0-python` package for verifying Auth0
JWTs. Use this new virtualenv for `mach try` and related subcommands.
Add a `--push-to-lando` flag that controls whether the push will be made
via the Lando API or using the VCS to hg.mozilla.org directly.
Create a `lando.py` module in the `tryselect` package that handles the
details around submitting to Lando. Authentication is handled by the
Device Code Authorization flow, and the returned access token is saved
to the mozbuild state directory. Auth0 details are added to the `.lando.ini`
file in the repo root, and a `LANDO_TRY_USE_DEV` environment variable can
be set to control submitting to the prod or dev Lando environments.
This module also includes patch stack gathering and discovery via `mozversioncontrol`.
mozversioncontrol's `Repository` subclass is extended with helper functions
to gather patch files from Mercurial and Git. We also add a `try_config_commit`
context manager that creates a temporary commit to hold try syntax commit
messages and/or a `try_task_config.json` file, which is removed from version
control on submission.
The `mach try` cram tests now use a separate virtualenv that must be built
before running, causing unexpected output in the tests. Run `mach try --help`
in the test setup to force the virtualenv to be built before running any
test.
`mach try chooser` would previously install packages for a small web application
from a `requirements.txt` file when run. Install these via the new `try.txt`
site environment instead. The required dependencies are also added to the
`python-test.txt` site since they will cause some tests to fail due to the
missing dependencies.
Differential Revision: https://phabricator.services.mozilla.com/D187641
This results in some changes from our current `isort` configuration. I'm
unclear if it's because ruff isn't at 100% parity with isort, they choose
different defaults or if I missed some configuration.
Either way, the changes all look reasonable to me (see child commit), so I'm
inclined to just accept the new import format it imposes.
Differential Revision: https://phabricator.services.mozilla.com/D172348
This results in some changes from our current `isort` configuration. I'm
unclear if it's because ruff isn't at 100% parity with isort, they choose
different defaults or if I missed some configuration.
Either way, the changes all look reasonable to me (see child commit), so I'm
inclined to just accept the new import format it imposes.
Differential Revision: https://phabricator.services.mozilla.com/D172348
This patch adds a `capture_log` flag that will allow a user to capture the logs produced when running the `push_to_try` methods. Currently, we use `subprocess.check_call` which causes hg to hang when we attempt to redirect and gather stdout. Using `subprocess.run` is better as it lets us capture the logs, but the log output is very slow and can easily lead people to believe that hg is hanging when it's not. This results in corrupted repos. Using Popen, the logs are output slower than `check_call`, but faster than `run` so you know something is happening. The speed at which these logs are printed is also why I have this log capturing behind a flag.
This functionality will be used in the `./mach try perf` selector to capture the child revisions produced for the try task config file changes and allow us to produce a PerfCompare link to provide the user.
Differential Revision: https://phabricator.services.mozilla.com/D155980
Following the pytest "deprecations and removals" docs [1], this patch:
* Replaces `@pytest.yield_fixture` with `@pytest.fixture`.
* Replaces `.funcargnames` with `.fixturenames`.
* Uses `Pathlib` parameter instead of the `py.path.local` one in
associated `pytest_*` hooks.
* Replaces `--strict` with `--strict-markers`
[1] https://docs.pytest.org/en/latest/deprecations.html
Differential Revision: https://phabricator.services.mozilla.com/D138815
Though not a panacea, this patch addresses existing usages of
`get_repository_from_build_config()` to fall back to
`get_repository_object()` if `MissingVCSTool` was encountered.
Ideally, we'd update `substs["HG"]` when we detect that it's changed,
but we don't have infrastructure to incrementally change configure
output.
So, other issues may creep up that will still warrant a fresh
`./mach configure`.
Differential Revision: https://phabricator.services.mozilla.com/D137675
Stanca Serban