It turns out subprocess functions have the annoying behavior of changing
line endings in `input` when using `text=True` or `universal_newlines=True`,
or even `encoding="utf-8"`. And git fast-import doesn't like `\r\n`.
Differential Revision: https://phabricator.services.mozilla.com/D215158
Currently, the caller potentially prepares files, add them to the VCS,
and then calls the function to create the commit, does whatever it wants
to do with it, then undoes it.
With the ultimate goal of not doing local changes to the work tree to
act on that try commit (for a push), this change in API makes it so
that everything except what do do with the commit is handled by one
function, that now takes paths and file contents for the files that were
previously created by the caller.
Differential Revision: https://phabricator.services.mozilla.com/D214214
The previous implementation used days since Jan 1 2000 for the last
16-bit segment. This was not unique enough and caused issues with
Antivirus software if two different channels were built on the same day.
The new approach uses hours since the last milestone bump and uses the
VCS to determine how long ago that was relative to the build time. This
means it will always reset when a new cycle begins, but still be unique
since the digits in the first 3 segments have incremented.
We also now use two of the 16-bits to encode the channel (nightly, beta,
ESR, and release). So two channels built within the same hour will still
be unique.
Using only 14-bits to store the 'hours since version bump', we have
about ~682 days from a version bump before we reach the maximum value we
can store. If a build is done after that point, the segment value will
always be the maximum value for that channel.
Differential Revision: https://phabricator.services.mozilla.com/D200989
The previous implementation used days since Jan 1 2000 for the last
16-bit segment. This was not unique enough and caused issues with
Antivirus software if two different channels were built on the same day.
The new approach uses hours since the last milestone bump and uses the
VCS to determine how long ago that was relative to the build time. This
means it will always reset when a new cycle begins, but still be unique
since the digits in the first 3 segments have incremented.
We also now use two of the 16-bits to encode the channel (nightly, beta,
ESR, and release). So two channels built within the same hour will still
be unique.
Using only 14-bits to store the 'hours since version bump', we have
about ~682 days from a version bump before we reach the maximum value we
can store. If a build is done after that point, the segment value will
always be the maximum value for that channel.
Differential Revision: https://phabricator.services.mozilla.com/D200989
The previous implementation used days since Jan 1 2000 for the last
16-bit segment. This was not unique enough and caused issues with
Antivirus software if two different channels were built on the same day.
The new approach uses hours since the last milestone bump and uses the
VCS to determine how long ago that was relative to the build time. This
means it will always reset when a new cycle begins, but still be unique
since the digits in the first 3 segments have incremented.
We also now use two of the 16-bits to encode the channel (nightly, beta,
ESR, and release). So two channels built within the same hour will still
be unique.
Using only 14-bits to store the 'hours since version bump', we have
about ~682 days from a version bump before we reach the maximum value we
can store. If a build is done after that point, the segment value will
always be the maximum value for that channel.
Differential Revision: https://phabricator.services.mozilla.com/D200989
A previous patch added parsing `git remote -v`, preferring
`mozilla-unified` as the official upstream remote and falling
back to the first official-looking remote if unified was not
found. We assumed that a developer is either using mozilla-unified,
or is using a single-headed repo like central. This overlooks
the fact that it is possible to clone from central, and then
pull other repos in, creating a repo with multiple official
remotes.
Update the `get_upstream_remote` function to find all official
looking remotes instead of a single official remote, and change
the `get_remote_arg` function to return a list of `--remotes`
arguments to be passed to various Git commands. This allows
Git to take all official remotes into consideration and more
precisely find the commits which are not present on any official
upsteams.
Add a test for `get_mozilla_remote_args` while we are here.
Differential Revision: https://phabricator.services.mozilla.com/D199637
Add a `get_mozilla_upstream_remote` method for `GitRepository` that
scans the output of `git remote -v` for an official Mozilla URL. Use
this URL in `base_ref` to limit the remotes considered for remote
branches, defaulting to `--remotes` without an argument if one can't
be found.
Differential Revision: https://phabricator.services.mozilla.com/D194881
Add support for submitting stacks of commits to Lando for queueing
on Try, instead of pushing to hg.mozilla.org directly. This patch
implements the Device Code Authorization flow for Auth0, simple changeset
discovery and patch gathering, and submissing to Lando via HTTP POST.
Add a `try.txt` virtualenv site that contains packages from the common
virtualenv as well as the `auth0-python` package for verifying Auth0
JWTs. Use this new virtualenv for `mach try` and related subcommands.
Add a `--push-to-lando` flag that controls whether the push will be made
via the Lando API or using the VCS to hg.mozilla.org directly.
Create a `lando.py` module in the `tryselect` package that handles the
details around submitting to Lando. Authentication is handled by the
Device Code Authorization flow, and the returned access token is saved
to the mozbuild state directory. Auth0 details are added to the `.lando.ini`
file in the repo root, and a `LANDO_TRY_USE_DEV` environment variable can
be set to control submitting to the prod or dev Lando environments.
This module also includes patch stack gathering and discovery via `mozversioncontrol`.
mozversioncontrol's `Repository` subclass is extended with helper functions
to gather patch files from Mercurial and Git. We also add a `try_config_commit`
context manager that creates a temporary commit to hold try syntax commit
messages and/or a `try_task_config.json` file, which is removed from version
control on submission.
The `mach try` cram tests now use a separate virtualenv that must be built
before running, causing unexpected output in the tests. Run `mach try --help`
in the test setup to force the virtualenv to be built before running any
test.
`mach try chooser` would previously install packages for a small web application
from a `requirements.txt` file when run. Install these via the new `try.txt`
site environment instead. The required dependencies are also added to the
`python-test.txt` site since they will cause some tests to fail due to the
missing dependencies.
Differential Revision: https://phabricator.services.mozilla.com/D187641
Following the pytest "deprecations and removals" docs [1], this patch:
* Replaces `@pytest.yield_fixture` with `@pytest.fixture`.
* Replaces `.funcargnames` with `.fixturenames`.
* Uses `Pathlib` parameter instead of the `py.path.local` one in
associated `pytest_*` hooks.
* Replaces `--strict` with `--strict-markers`
[1] https://docs.pytest.org/en/latest/deprecations.html
Differential Revision: https://phabricator.services.mozilla.com/D138815
The py.path allowed adding arbitrary fields to it, while a pathlib.Path
does not. To replicate the behavior with as few changes as possible, I
made a test fixture wrapper class (RepoTestFixture) to add the same
arbitrary fields to, while now containing a Pathlib path instead.
Also added a utility functions to make the running of tests a
bit cleaner ('execute_next_step')
Differential Revision: https://phabricator.services.mozilla.com/D136302
Note: In `test_commit.py` For date formatting, the `%s` did not work on Windows.
On Linux it gives the time in seconds. The switch to the rfc822date format for
both hg and git `%aD` made the expected output in the check later work for both
git and hg, and made it platform agnostic.
Differential Revision: https://phabricator.services.mozilla.com/D136209
Unless an upstream is specified, we should be able to detect which files
are changed on our local branch without hitting the network.
Differential Revision: https://phabricator.services.mozilla.com/D130684
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
In the process, fixed a few bugs:
- the template eg `{file_adds % "\n{file}"}` produced a leading blank line,
which led to everything being linted.
- 'd' was replaced with 'r' in diff_filters, but the replacement was discarded.
- as a result of the above, `hg status -d` was being used ("show only deleted (but tracked) files") and now it is `hg status -r` ("show only removed files"). I *think* this is what was intended?
Differential Revision: https://phabricator.services.mozilla.com/D66714
Bug 1554987 made `mach try` use a transient remote, but that causes
problems with existing setups that happen to use the same remote name,
because of a combination of not-quite-as-documented-as-it-should
behavior of git.
- `git -c foo.bar=qux` doesn't override the value of `foo.bar` from the
git configuration when `foo.bar` is an item that can take several
values.
- `remote.$remote.url` and `remote.$remote.pushurl` take several values,
allowing to give several URLs.
The combination of both means that if the git configuration already has
`remote.try.url` set, that value takes precedence (because git push
tries them one after the other, and takes the one from the command line
last)
One way we could increase the chances of things working out fine would
be to use `remote.try.pushurl`, which if already set, is more likely to
be right than an existing `remote.try.url`.
OTOH, it turns out, after more investigation, that bug 1554987 requires
a footgunny setup to happen in the first place. Namely, it requires
having run `git lfs install` from a git-cinnabar clone.
so we just go back to the previous status quo.
Differential Revision: https://phabricator.services.mozilla.com/D36149
The git version shipped in some versions of OSX is patched by apple in a
way such that doing `git push hg::ssh://...` fails with an error message
like `Invalid remote name "hg::ssh://...`.
So instead, we define a named remote via inline configuration, and use
that remote's name for the push.
Differential Revision: https://phabricator.services.mozilla.com/D35351
The doc string for the "working_directory_clean" function states:
> By default, untracked and ignored files are not considered.
But the git implementation for this function used to run:
git status --porcelain
Which *does* consider untracked files by default. Instead, we need to run:
git status --porcelain --untracked-files=no
Differential Revision: https://phabricator.services.mozilla.com/D2134
Most HG commands use subprocesses, even if a context manager (and therefore an
hglib client) has been created. There are only two commands that make use of
the client, but they *only* work inside a context manager. I don't think
there are any technical reason these two commands *need* to use the context
manager.
This patch merges the HgRepository._run_in_client function with
HgRepository._run(). If a client exists, that will be used, otherwise a
subprocess will be used.
Differential Revision: https://phabricator.services.mozilla.com/D1809
Most HG commands use subprocesses, even if a context manager (and therefore an
hglib client) has been created. There are only two commands that make use of
the client, but they *only* work inside a context manager. I don't think
there are any technical reason these two commands *need* to use the context
manager.
This patch merges the HgRepository._run_in_client function with
HgRepository._run(). If a client exists, that will be used, otherwise a
subprocess will be used.
Differential Revision: https://phabricator.services.mozilla.com/D1809
Stanca Serban