a35b188d0e44 inadvertently regressed behavior in the case where
the Git status.showUntrackedFiles config option was set and
we want to purge untracked files.
Differential Revision: https://phabricator.services.mozilla.com/D2141
The doc string for the "working_directory_clean" function states:
> By default, untracked and ignored files are not considered.
But the git implementation for this function used to run:
git status --porcelain
Which *does* consider untracked files by default. Instead, we need to run:
git status --porcelain --untracked-files=no
Differential Revision: https://phabricator.services.mozilla.com/D2134
Most HG commands use subprocesses, even if a context manager (and therefore an
hglib client) has been created. There are only two commands that make use of
the client, but they *only* work inside a context manager. I don't think
there are any technical reason these two commands *need* to use the context
manager.
This patch merges the HgRepository._run_in_client function with
HgRepository._run(). If a client exists, that will be used, otherwise a
subprocess will be used.
Differential Revision: https://phabricator.services.mozilla.com/D1809
Most HG commands use subprocesses, even if a context manager (and therefore an
hglib client) has been created. There are only two commands that make use of
the client, but they *only* work inside a context manager. I don't think
there are any technical reason these two commands *need* to use the context
manager.
This patch merges the HgRepository._run_in_client function with
HgRepository._run(). If a client exists, that will be used, otherwise a
subprocess will be used.
Differential Revision: https://phabricator.services.mozilla.com/D1809
This will make sure that when running |mach python-test --python 3| locally,
we only run the tests that also run in CI with python 3 (and therefore pass
presumably).
MozReview-Commit-ID: 3OBr9yLSlSq
This patch allows executing |mach python-test| against Python 3 by specifying the optional |--three| command line option. When this option is present, pipenv will be used to manage a virtual environment using Python 3 and attempt to run the tests. When it is not present, pipenv will not be used, and everything will work as it did before this patch.
My original plan was to use pipenv regardless of the target version of Python, however I encountered several issues running some of the tests against our Python packages. Once all tests have been patched to run against Python 3, then we should be able to use pipenv when running them against Python 2.
Note that this patch allows tests to run against Python 3, but there are plenty of issues preventing them from passing. With this patch in place we can start to add Python 3 support to our packages and have the tests running in CI to ensure we don't regress back to just supporting Python 2.
MozReview-Commit-ID: BuU5gZK83hL
IHG: changed taskcluster/ci/source-test/python.yml
This patch allows executing |mach python-test| against Python 3 by specifying the optional |--three| command line option. When this option is present, pipenv will be used to manage a virtual environment using Python 3 and attempt to run the tests. When it is not present, pipenv will not be used, and everything will work as it did before this patch.
My original plan was to use pipenv regardless of the target version of Python, however I encountered several issues running some of the tests against our Python packages. Once all tests have been patched to run against Python 3, then we should be able to use pipenv when running them against Python 2.
Note that this patch allows tests to run against Python 3, but there are plenty of issues preventing them from passing. With this patch in place we can start to add Python 3 support to our packages and have the tests running in CI to ensure we don't regress back to just supporting Python 2.
MozReview-Commit-ID: BuU5gZK83hL
IHG: changed taskcluster/ci/source-test/python.yml
Detecting the upstream repository isn't always possible. Using 'git merge-base'
seems to be a more reliable way of detecting outgoing files.
MozReview-Commit-ID: 3zOdTRIcggw
In addition to help fix this bug, having the base revision will allow us
to download the most likely relevant artifacts from CI. For example, in
|mach try| we can search for a 'full-taskgraph.json' artifact from the
most relevant decision task, even if not updated to the latest central.
The ability to find the base revision will also be needed when merging
tools/tryselect/vcs.py into mozversioncontrol.
MozReview-Commit-ID: 2WRRs4AoXcK
Sometimes commands return non-zero even though everything is ok. For example,
'hg outgoing' returns 1 if there are no outgoing files. This adds a way for
specific function calls not to abort if something goes wrong. Instead, stderr
will be printed (if any) and an empty string is returned.
MozReview-Commit-ID: E089djeHrmr
This adds 'get_outgoing_files'. First it automatically attempts to find the
upstream remote the current change is based on, then returns all files changed
in the local branch.
If an upstream remote can't be detected, it raises MissingUpstreamRepo
MozReview-Commit-ID: 9zSB9EdwVU8
There's currently a function for getting added files (A) and modified files
(M). We'll also eventually need the ability to get deleted files (D) and any
combination of the above, e.g (AM). Rather than creating a new function for
each possible case, let's have a single function where you can pass in which
modifiers you are interested in. With this patch, if you want all modified and
added files, you can do:
get_changed_files('AM')
By default 'ADM' is used.
This also adds a 'mode' option for git. This allows consumers to return staged
files, unstaged files or both. The default ('unstaged') keeps the current
behaviour in tact.
MozReview-Commit-ID: 9IA1bxaJS80
Because hglib spawns a persistent process, we introduce a context
manager for Repository. It no-ops by default. On HgRepository it
controls the lifetime of the persistent hg process.
A helper method for running an hg command via hglib has been added.
We can't transition existing methods to hglib because hglib
requires a context manager, which no consumer is using yet.
MozReview-Commit-ID: 8z0fcGFeAm5
``print()`` has no business being in library code like this. It was
a holdover from this code being copied from bootstrap. So remove it.
While we're here, replace the generic exception with a specific one.
We don't want to be swallowing bugs via ``except Exception``.
MozReview-Commit-ID: 49goUstfPBz
Because hglib spawns a persistent process, we introduce a context
manager for Repository. It no-ops by default. On HgRepository it
controls the lifetime of the persistent hg process.
A helper method for running an hg command via hglib has been added.
We can't transition existing methods to hglib because hglib
requires a context manager, which no consumer is using yet.
MozReview-Commit-ID: 8z0fcGFeAm5
The Repository interface gains a new method to determine if a sparse
checkout is present. Mercurial's implementation is somewhat crude,
but it should work (Mercurial's sparse support is still experimental
and I only intend to support sparse checkout in Firefox CI until it
is less experimental). Git's always returns False (for now).
To prove it works and to expose the information more widely, we hook
it up to moz.configure. We do this by first implementing a function
that returns a Repository instance. Then we simply call a function on
it to resolve the sparse checkout flag.
MozReview-Commit-ID: AlsT5LdSPdZ
For reasons I can't explain, Windows builds are failing intermittently
because they are unable to locate the `hg` binary when running
some SpiderMonkey test processes. These processes use
mozversioncontrol.get_repository_from_env() to locate the
current repository.
We now store VCS info in configure. This makes it available to anything
running in a build system context.
This commit teaches mozversioncontrol.get_repository_from_env()
to import the "buildconfig" module to locate VCS info. If the module
can be imported, it is the sole source of VCS info. Otherwise, we
fall back to the existing detection mechanisms.
This should get rid of the intermittent failure. If it doesn't,
it is still a step in the right direction because it will allow
build system processes to consistently use a well-defined VCS
binary.
MozReview-Commit-ID: DMxXheJLRqH
For reasons I can't explain, Windows builds are failing intermittently
because they are unable to locate the `hg` binary when running
some SpiderMonkey test processes. These processes use
mozversioncontrol.get_repository_from_env() to locate the
current repository.
We now store VCS info in configure. This makes it available to anything
running in a build system context.
This commit teaches mozversioncontrol.get_repository_from_env()
to import the "buildconfig" module to locate VCS info. If the module
can be imported, it is the sole source of VCS info. Otherwise, we
fall back to the existing detection mechanisms.
This should get rid of the intermittent failure. If it doesn't,
it is still a step in the right direction because it will allow
build system processes to consistently use a well-defined VCS
binary.
MozReview-Commit-ID: DMxXheJLRqH
We're getting an intermittent failure running `hg manifest` in CI. I
have no clue why.
What I do know is that we now have the mozversioncontrol Python package
for containing utility code for interacting with version control. It is
slightly more robust and I'm willing to support it more than I am
check_utils.py.
This commit adds a new API to our abstract repository class to obtain the
files in the working directory by querying version control.
Since I suspect cwd was coming into play in automation, I've also
added a utility function to mozversioncontrol to attempt to find
a version control checkout from the current working directory. It
simply traces ancestor paths looking for a .hg or .git directory.
Finally, I've ported all callers of the now-deleted API to the new
one. The old code had some "../.." paths in it, meaning it only
worked when cwd was just right. Since we resolve the absolute path
to the checkout and store it on the repo object, I've updated the
code so it should work no matter what cwd is as long as a repo can
be found. I'm not 100% confident I found all consumers assuming cwd.
But it's a start.
I'm not 100% confident this will fix the intermittent issues in CI. But
at least we should get a better error message and at least we'll be
running less hacky code.
MozReview-Commit-ID: AmCraHXcTEX
This command is useful for users who wish to perform something like the
following:
1. Add/remove a bunch of files;
2. Examine the additions/removals/modifications for interesting changes;
3. Reject the add/remove if it doesn't meet some set of conditions.
I wanted to be able to do some VCS interaction from a mach command, and we
didn't have anything suitable, so I tore up mozversioncontrol and replaced
it with a framework to hang new features off of. I've only implemented
the bits I need currently (get_modified_files and add_remove_files),
but it should be straightforward to add more functionality there.
This patch also adds a `repository` property to `MozbuildObject`, which will
return a `Repository` object for the topsrcdir to make using these helpers
even easier for `MozbuildObject`-derived classes.
MozReview-Commit-ID: Gw6Ixp1ltiN
Running old extensions with newer versions of Mercurial may crash `hg`
due to the old extension accessing something or doing something that has
been changed in the new release.
To minimize the risk of this happening, we disable common 3rd party
extensions when cloning or pulling as part of `mach mercurial-setup`. We
don't want to disable everything because some extensions (like
remotenames) provide features the user may want enabled as part of the
clone/update. This leaves the door open for more failures. Hopefully
this approach is sufficient. We can always revisit later.
We want to make it turnkey for people to use reviewboard. So, we add
reviewboard and related functionality to |mach mercurial-setup|.
Since the reviewboard extension only works in Mercurial 3.0 and newer,
we add some version detection for the Mercurial version. This should
have been done months ago. We now have it.
I also took the opportunity to inform |mach bootstrap| that Mercurial
2.x is no longer modern.
I also updated the messaging around mq to encourage fewer new users to
use it. You may find this controversial. People can always ignore the
message.
Finally, I also added a histedit prompt to the mix, since a lot of
people don't know about that and many find it useful.
I could have broken this into multiple patches. Meh.
arthur.iakab