When do matching inside of `ignore` or `globset`, we sometimes need a
reusable mutable scratch space for matching. To do this, we use a `Pool`
abstraction from `regex-automata` to do this heavy lifting for us.
It turns out that this can be a limiting factor in some scenarios when
gitignore matching dominates. For example, in my checkout of the
Chromium repository:
```
$ hyperfine 'rg-master -j32 --files' 'rg-bigger-pool -j32 --files' 'rg-master -j32 --files --no-ignore'
Benchmark 1: rg-master -j32 --files
Time (mean ± σ): 156.3 ms ± 5.5 ms [User: 2356.4 ms, System: 380.8 ms]
Range (min … max): 147.6 ms … 164.5 ms 19 runs
Benchmark 2: rg-bigger-pool -j32 --files
Time (mean ± σ): 131.6 ms ± 2.1 ms [User: 1628.2 ms, System: 310.1 ms]
Range (min … max): 128.8 ms … 136.3 ms 22 runs
Benchmark 3: rg-master -j32 --files --no-ignore
Time (mean ± σ): 107.4 ms ± 1.0 ms [User: 664.7 ms, System: 304.7 ms]
Range (min … max): 104.9 ms … 109.6 ms 28 runs
Summary
rg-master -j32 --files --no-ignore ran
1.22 ± 0.02 times faster than rg-bigger-pool -j32 --files
1.45 ± 0.05 times faster than rg-master -j32 --files
```
The `--no-ignore` benchmark is our baseline. It doesn't process
gitignore files, so it's quite a bit faster. But with this change,
processing gitignore files gets faster than what is currently on master.
Specifically, re-running the above without the baseline:
```
$ hyperfine 'rg-master -j32 --files' 'rg-bigger-pool -j32 --files'
Benchmark 1: rg-master -j32 --files
Time (mean ± σ): 154.1 ms ± 5.1 ms [User: 2287.4 ms, System: 378.2 ms]
Range (min … max): 144.1 ms … 163.9 ms 19 runs
Benchmark 2: rg-bigger-pool -j32 --files
Time (mean ± σ): 131.7 ms ± 2.1 ms [User: 1618.6 ms, System: 314.1 ms]
Range (min … max): 128.4 ms … 135.3 ms 22 runs
Summary
rg-bigger-pool -j32 --files ran
1.17 ± 0.04 times faster than rg-master -j32 --files
```
I've used `-j32` here because this change is best observed as improving
the *scaling* properties. (And my system has 32 cores, so it's the best
case scenario.) If we just let ripgrep use its default thread heuristic
(which caps itself at a lower value), then the improvement here is a
bit more modest:
```
$ hyperfine 'rg-master --files' 'rg-bigger-pool --files'
Benchmark 1: rg-master --files
Time (mean ± σ): 150.0 ms ± 8.5 ms [User: 1139.6 ms, System: 237.7 ms]
Range (min … max): 131.1 ms … 162.4 ms 22 runs
Benchmark 2: rg-bigger-pool --files
Time (mean ± σ): 136.0 ms ± 12.1 ms [User: 1034.4 ms, System: 236.0 ms]
Range (min … max): 117.7 ms … 157.6 ms 22 runs
Summary
rg-bigger-pool --files ran
1.10 ± 0.12 times faster than rg-master --files
```
This improves scaling because the underlying change here now uses a
cache line for each available thread instead of limiting itself to a
fixed value of `8`. The cost is that ripgrep may use a bit more memory
storing cached values per thread.
It looks like the location of executables is changing in Cargo's test
runner, so use the blessed way of discovering an executable. That is,
for ripgrep, consult the `CARGO_BIN_EXE_rg` environment variable.
When parallelism isn't used (e.g., when sorting results), then
it's possible in some cases for the `ignore` crate to still try and
load gitignore files inside of directories that are never
descended into. This can occur when `--max-depth` is used, or
even just when a directory is skipped for some other reason.
Fixes#2565, PR #3496
The indexing feature will be born with a lot of restrictions on it in
order to bootstrap and end-to-end working system. For example, getting
the `--pre` flag (and similar) working correctly is somewhat of a
hazard, since you'd need to specify the same transformation during both
indexing and not.
We disable pretty much any kind of filtering or transform related flags
other than what the default provides. Again, we'll relax at least most
of these in the future. But let's start here.
This just contains some prototype code that I wrote years ago at
present. But it sets a base on which to build. It also establishes a
semver boundary between ripgrep's core indexing and a library-version of
that indexing.
... and add some of the initial flags, spec'd out in #1497, for adding
indexing to ripgrep. These don't do anything yet.
I chose to go this route because indexing will be a big feature, and it
would be better to develop it on master instead of maintaining a
long-lived branch for it.
This adds a new increment checker for ignore files, overrides and file
type selection. This effectively makes it possible to check individual
file paths as to whether they are ignored or not by any ignore file in
the directory tree (including parent gitignores). This is particularly
useful in the context of implementing a file watcher, where one wants to
make a decision about whether a newly added file should be included
*without* doing a full re-traversal of the directory tree.
This implementation reuses pretty much all of the existing gitignore
logic without making any changes to it.
This does the more sensible thing of putting the implementation within
the routine instead of repeating the docs.
We also rename this routine since we're going to want a path variant for
incremental checking.
The old OPUS download.php link for the v2018 English monolingual
corpus now 404s. Point the README benchmark section at the live
object-storage URL for the same corpus.
Fixes#3471
When a visitor panics and the workers shutdown, we indicate as such and
quit everything. This effectively acts as a sort of poison that will
cause all other workers to stop even when in their idle loop.
Fixes#3009
`gitconfig_excludes_path()` resolves the global gitignore
by reading `core.excludesFile` from `$HOME/.gitconfig` and
`$XDG_CONFIG_HOME/git/config`, but git 2.32 (June 2021) introduced
`GIT_CONFIG_GLOBAL` and `GIT_CONFIG_SYSTEM` environment variables that
override these locations. The `ignore` crate doesn't check either of
them, so users who point `GIT_CONFIG_GLOBAL` at a non-standard path
(e.g. NixOS managing git config at `/etc/git/config`) have their global
gitignore silently ignored by ripgrep while `git` itself respects it.
This commit checks `GIT_CONFIG_GLOBAL` before the hardcoded paths,
matching git's semantics where it replaces both `$HOME/.gitconfig` and
the XDG config. It also adds system-level config (`GIT_CONFIG_SYSTEM`,
falling back to `/etc/gitconfig`) as a lowest-priority source for
`core.excludesFile`, which the crate previously didn't consult at all.
Fixes#3275, Closes#3276
Note that we continue to disable memory maps on macOS.
Note also that I've generally been unable to observe a performance
improvement from a change like this. However, I think it's fine to do
this for reasons of "good sense."
Closes#3246
This used to be Coq, but the language got renamed:
https://rocq-prover.org/about#Name
Interestingly, there is no mention of the pronunciation of the name in
English speaking countries.
Closes#3357
Make sure that we correctly check that all globs are matched, even if
multiple globs use the same strategy.
Previously we only checked that all strategies matched, but didn't
check with the strategies themselves.
This now adds a `matches_all` method to each strategy to check all
globs within a strategy.
This avoids allocating for tracking the number of matches.
Fixes#3290, Closes#3302
Andrew Gallant