Add code to clean up Git and Hg repositories, and invoke that where `./mach vendor rust` would throw an error. Unfortunately, `cargo vendor` also updates the repo's root `Cargo.lock` file in-place and while we could `git checkout`/`hg revert` that file for the user, `Cargo.lock` may have had pre-existing changes that would be overwritten by such a change. Instead of a potentially destructive update in the error case, I've opted to add an extra error message to tell you how to deal with that one file.
Differential Revision: https://phabricator.services.mozilla.com/D49494
--HG--
extra : moz-landing-system : lando
Environment should be bytes on Python 2 (to avoid Windows errors) and text on
Python 3. The 'ensure_subprocess' env utility function handles this.
Differential Revision: https://phabricator.services.mozilla.com/D48115
--HG--
extra : moz-landing-system : lando
Bug 1554987 made `mach try` use a transient remote, but that causes
problems with existing setups that happen to use the same remote name,
because of a combination of not-quite-as-documented-as-it-should
behavior of git.
- `git -c foo.bar=qux` doesn't override the value of `foo.bar` from the
git configuration when `foo.bar` is an item that can take several
values.
- `remote.$remote.url` and `remote.$remote.pushurl` take several values,
allowing to give several URLs.
The combination of both means that if the git configuration already has
`remote.try.url` set, that value takes precedence (because git push
tries them one after the other, and takes the one from the command line
last)
One way we could increase the chances of things working out fine would
be to use `remote.try.pushurl`, which if already set, is more likely to
be right than an existing `remote.try.url`.
OTOH, it turns out, after more investigation, that bug 1554987 requires
a footgunny setup to happen in the first place. Namely, it requires
having run `git lfs install` from a git-cinnabar clone.
so we just go back to the previous status quo.
Differential Revision: https://phabricator.services.mozilla.com/D36149
--HG--
extra : moz-landing-system : lando
The git version shipped in some versions of OSX is patched by apple in a
way such that doing `git push hg::ssh://...` fails with an error message
like `Invalid remote name "hg::ssh://...`.
So instead, we define a named remote via inline configuration, and use
that remote's name for the push.
Differential Revision: https://phabricator.services.mozilla.com/D35351
--HG--
extra : moz-landing-system : lando
The git version shipped in some versions of OSX is patched by apple in a
way such that doing `git push hg::ssh://...` fails with an error message
like `Invalid remote name "hg::ssh://...`.
So instead, we define a named remote via inline configuration, and use
that remote's name for the push.
Differential Revision: https://phabricator.services.mozilla.com/D35351
--HG--
extra : moz-landing-system : lando
If there are a large number of untracked files in the working directory, hg
will attempt to print them all out with the default pager. This does not
interact very will with commands that are built atop this functionality. We
set HGPLAIN=1 so that the underlying hg will not attempt to use a pager.
Differential Revision: https://phabricator.services.mozilla.com/D26607
--HG--
extra : moz-landing-system : lando
This also enables the py2 linter which will help maintain compatibility
with both 2 and 3.
Differential Revision: https://phabricator.services.mozilla.com/D1884
--HG--
extra : moz-landing-system : lando
a35b188d0e44 inadvertently regressed behavior in the case where
the Git status.showUntrackedFiles config option was set and
we want to purge untracked files.
Differential Revision: https://phabricator.services.mozilla.com/D2141
--HG--
extra : moz-landing-system : lando
The doc string for the "working_directory_clean" function states:
> By default, untracked and ignored files are not considered.
But the git implementation for this function used to run:
git status --porcelain
Which *does* consider untracked files by default. Instead, we need to run:
git status --porcelain --untracked-files=no
Differential Revision: https://phabricator.services.mozilla.com/D2134
--HG--
extra : moz-landing-system : lando
Most HG commands use subprocesses, even if a context manager (and therefore an
hglib client) has been created. There are only two commands that make use of
the client, but they *only* work inside a context manager. I don't think
there are any technical reason these two commands *need* to use the context
manager.
This patch merges the HgRepository._run_in_client function with
HgRepository._run(). If a client exists, that will be used, otherwise a
subprocess will be used.
Differential Revision: https://phabricator.services.mozilla.com/D1809
--HG--
extra : moz-landing-system : lando
Most HG commands use subprocesses, even if a context manager (and therefore an
hglib client) has been created. There are only two commands that make use of
the client, but they *only* work inside a context manager. I don't think
there are any technical reason these two commands *need* to use the context
manager.
This patch merges the HgRepository._run_in_client function with
HgRepository._run(). If a client exists, that will be used, otherwise a
subprocess will be used.
Differential Revision: https://phabricator.services.mozilla.com/D1809
--HG--
extra : moz-landing-system : lando
This will make sure that when running |mach python-test --python 3| locally,
we only run the tests that also run in CI with python 3 (and therefore pass
presumably).
MozReview-Commit-ID: 3OBr9yLSlSq
--HG--
extra : rebase_source : 456340d0ecdddf1078f2b5b4ebb1eddf3813b26a
This patch allows executing |mach python-test| against Python 3 by specifying the optional |--three| command line option. When this option is present, pipenv will be used to manage a virtual environment using Python 3 and attempt to run the tests. When it is not present, pipenv will not be used, and everything will work as it did before this patch.
My original plan was to use pipenv regardless of the target version of Python, however I encountered several issues running some of the tests against our Python packages. Once all tests have been patched to run against Python 3, then we should be able to use pipenv when running them against Python 2.
Note that this patch allows tests to run against Python 3, but there are plenty of issues preventing them from passing. With this patch in place we can start to add Python 3 support to our packages and have the tests running in CI to ensure we don't regress back to just supporting Python 2.
MozReview-Commit-ID: BuU5gZK83hL
IHG: changed taskcluster/ci/source-test/python.yml
--HG--
extra : rebase_source : ca2b15d905f7a5c895a2fd8916144841f5d205de
This patch allows executing |mach python-test| against Python 3 by specifying the optional |--three| command line option. When this option is present, pipenv will be used to manage a virtual environment using Python 3 and attempt to run the tests. When it is not present, pipenv will not be used, and everything will work as it did before this patch.
My original plan was to use pipenv regardless of the target version of Python, however I encountered several issues running some of the tests against our Python packages. Once all tests have been patched to run against Python 3, then we should be able to use pipenv when running them against Python 2.
Note that this patch allows tests to run against Python 3, but there are plenty of issues preventing them from passing. With this patch in place we can start to add Python 3 support to our packages and have the tests running in CI to ensure we don't regress back to just supporting Python 2.
MozReview-Commit-ID: BuU5gZK83hL
IHG: changed taskcluster/ci/source-test/python.yml
--HG--
extra : rebase_source : e1a64c0ffa8fe5cce71a041579601d4a72e37779
Detecting the upstream repository isn't always possible. Using 'git merge-base'
seems to be a more reliable way of detecting outgoing files.
MozReview-Commit-ID: 3zOdTRIcggw
--HG--
extra : rebase_source : 56dfd16bbddca38195f6a833dc5696cf509f3fa8
In addition to help fix this bug, having the base revision will allow us
to download the most likely relevant artifacts from CI. For example, in
|mach try| we can search for a 'full-taskgraph.json' artifact from the
most relevant decision task, even if not updated to the latest central.
The ability to find the base revision will also be needed when merging
tools/tryselect/vcs.py into mozversioncontrol.
MozReview-Commit-ID: 2WRRs4AoXcK
--HG--
extra : rebase_source : 28d759ffaece14dd260402fa730136288ab8111b
This also migrates the vcs.py test to mozversioncontrol and adds a new task for
it.
MozReview-Commit-ID: 9jTRkjNupVA
--HG--
extra : rebase_source : 400f27498e00ea45234ad7c951770b098e916b8e
Sometimes commands return non-zero even though everything is ok. For example,
'hg outgoing' returns 1 if there are no outgoing files. This adds a way for
specific function calls not to abort if something goes wrong. Instead, stderr
will be printed (if any) and an empty string is returned.
MozReview-Commit-ID: E089djeHrmr
--HG--
extra : rebase_source : be351f357705e09c3fe876cecefa7ddd9c48987e
This adds 'get_outgoing_files'. First it automatically attempts to find the
upstream remote the current change is based on, then returns all files changed
in the local branch.
If an upstream remote can't be detected, it raises MissingUpstreamRepo
MozReview-Commit-ID: 9zSB9EdwVU8
--HG--
extra : rebase_source : e352d6d471644ef9eda76b972b789d6b54449471
There's currently a function for getting added files (A) and modified files
(M). We'll also eventually need the ability to get deleted files (D) and any
combination of the above, e.g (AM). Rather than creating a new function for
each possible case, let's have a single function where you can pass in which
modifiers you are interested in. With this patch, if you want all modified and
added files, you can do:
get_changed_files('AM')
By default 'ADM' is used.
This also adds a 'mode' option for git. This allows consumers to return staged
files, unstaged files or both. The default ('unstaged') keeps the current
behaviour in tact.
MozReview-Commit-ID: 9IA1bxaJS80
--HG--
extra : rebase_source : 160f650220ca9a35b4b116bc9fa13f28d84419fa
This is generally useful functionality to have. A consume will be
introduced in an upcoming commit.
MozReview-Commit-ID: 4arTMfJSiEC
--HG--
extra : rebase_source : 4bcf70f58b57b79b8dcb7a6eed633e1c7e42aca3
And convert consumers to context managers because hglib requires that.
MozReview-Commit-ID: Ckf1yBYeUlm
--HG--
extra : rebase_source : 985220032bced1a7077fd9b04ca8ad6de822c887
Because hglib spawns a persistent process, we introduce a context
manager for Repository. It no-ops by default. On HgRepository it
controls the lifetime of the persistent hg process.
A helper method for running an hg command via hglib has been added.
We can't transition existing methods to hglib because hglib
requires a context manager, which no consumer is using yet.
MozReview-Commit-ID: 8z0fcGFeAm5
--HG--
extra : rebase_source : 1cbfe44a5aafe838c25bcde91cc1deaf5e7367de
Some callers already have a build config object. Let's not
make them call a function that imports buildconfig.
MozReview-Commit-ID: J22HhyVma9y
--HG--
extra : rebase_source : fe5151925aee76508d26a9368e7a2ebec7e23ddb
``print()`` has no business being in library code like this. It was
a holdover from this code being copied from bootstrap. So remove it.
While we're here, replace the generic exception with a specific one.
We don't want to be swallowing bugs via ``except Exception``.
MozReview-Commit-ID: 49goUstfPBz
--HG--
extra : rebase_source : a821159bd12a449ed1a0edf21a1f9eb29711ad95
And convert consumers to context managers because hglib requires that.
MozReview-Commit-ID: Ckf1yBYeUlm
--HG--
extra : rebase_source : e76b2690bd02bc6889dbe0f4524d61669ba82367
Because hglib spawns a persistent process, we introduce a context
manager for Repository. It no-ops by default. On HgRepository it
controls the lifetime of the persistent hg process.
A helper method for running an hg command via hglib has been added.
We can't transition existing methods to hglib because hglib
requires a context manager, which no consumer is using yet.
MozReview-Commit-ID: 8z0fcGFeAm5
--HG--
extra : rebase_source : 914322f3647fb29406a22203811ac8cc0ac82420
Some callers already have a build config object. Let's not
make them call a function that imports buildconfig.
MozReview-Commit-ID: J22HhyVma9y
--HG--
extra : rebase_source : fa013dc0d8f85d1767e5666107ea656a17d137a2
The Repository interface gains a new method to determine if a sparse
checkout is present. Mercurial's implementation is somewhat crude,
but it should work (Mercurial's sparse support is still experimental
and I only intend to support sparse checkout in Firefox CI until it
is less experimental). Git's always returns False (for now).
To prove it works and to expose the information more widely, we hook
it up to moz.configure. We do this by first implementing a function
that returns a Repository instance. Then we simply call a function on
it to resolve the sparse checkout flag.
MozReview-Commit-ID: AlsT5LdSPdZ
--HG--
extra : rebase_source : f1e9aaa7d15f11c7c5e8d268d4ad82468732103b
This will remove the need to sniff class types. The 1 in-tree
consumer doing this has been converted.
MozReview-Commit-ID: I8cUa8J54VE
--HG--
extra : rebase_source : 4c24adaf7eb9d62678ac78604e819a7376d4073b
For reasons I can't explain, Windows builds are failing intermittently
because they are unable to locate the `hg` binary when running
some SpiderMonkey test processes. These processes use
mozversioncontrol.get_repository_from_env() to locate the
current repository.
We now store VCS info in configure. This makes it available to anything
running in a build system context.
This commit teaches mozversioncontrol.get_repository_from_env()
to import the "buildconfig" module to locate VCS info. If the module
can be imported, it is the sole source of VCS info. Otherwise, we
fall back to the existing detection mechanisms.
This should get rid of the intermittent failure. If it doesn't,
it is still a step in the right direction because it will allow
build system processes to consistently use a well-defined VCS
binary.
MozReview-Commit-ID: DMxXheJLRqH
--HG--
extra : rebase_source : a9c599934c8c08da1fbb92a9105f5c7cba0867b3
For reasons I can't explain, Windows builds are failing intermittently
because they are unable to locate the `hg` binary when running
some SpiderMonkey test processes. These processes use
mozversioncontrol.get_repository_from_env() to locate the
current repository.
We now store VCS info in configure. This makes it available to anything
running in a build system context.
This commit teaches mozversioncontrol.get_repository_from_env()
to import the "buildconfig" module to locate VCS info. If the module
can be imported, it is the sole source of VCS info. Otherwise, we
fall back to the existing detection mechanisms.
This should get rid of the intermittent failure. If it doesn't,
it is still a step in the right direction because it will allow
build system processes to consistently use a well-defined VCS
binary.
MozReview-Commit-ID: DMxXheJLRqH
--HG--
extra : rebase_source : a9c599934c8c08da1fbb92a9105f5c7cba0867b3
