This one was a bit hard to swallow because it involved copying a
fairly short but not terribly simple function for normalizing property
names/values. But the code is so small, changes rarely, and is easily
tested, that it's just not worth bringing in a whole dependency for it
given how big regex-syntax already is.
This commit brings the utf8-ranges crate into regex-syntax as a utf8
sub-module.
This was done because it was observed that utf8-ranges is effectively
unused outside the context of regex-syntax. It is a very small amount of
code, and fits alongside the rest of regex-syntax. In particular, anyone
building a regex engine using regex-syntax will likely need this code
anyway.
Rust 1.28 is almost a year old by this point, and there were a number of
nice stabilizations between 1.24 and 1.28. Notably, vendor intrinsics were
stabilized in Rust 1.26, so we no longer need a build script.
The problem with putting it in the regex crate proper is that it
requires the regex crate to bump its minimal regex-syntax crate version.
While this isn't necessarily an issue, since we can't enable Cargo's
minimal version check because of the `rand` dependency, this winds up
being a hazard. Plus, having it in the regex crate doesn't buy us too
much. It's just as well to have the tests in regex-syntax.
Fixes#593
This fixes yet another bug with our handling of (?flags) directives in
the regex. This time, we try to be a bit more principled and
specifically treat a (?flags) directive as a valid empty sub-expression.
While this means we could remove errors reported from previous fixes for
things like `(?i)+`, we retain those for now since they are a bit weird.
Although `((?i))+` is now allowed, which is equivalent. We should
probably allow `(?i)+` in the future for consistency sake.
Fixes#527
This fixes a bug where the HIR translator would panic on regexes such as
`(?i){1}` since it assumes that every repetition operator has a valid
sub-expression, and `(?i)` is not actually a sub-expression (but is more
like a directive instead).
Previously, we fixed this same bug for *uncounted* repetitions in commit
17764ffe (for bug #465), but we did not fix it for counted repetitions.
We apply the same fix here.
Fixes#555
This adds a couple new methods on HIR expressions for determining whether
they are literals or not. This is useful for determining whether to apply
optimizations such as Aho-Corasick without re-analyzing the syntax.
This commit adds two new predicates to `Hir` values that permit querying
whether an expression is *line* anchored at the start or end.
This was motivated by a desire to tweak the offsets of a match when
enabling --crlf mode in ripgrep.
This commit adds several emoji properties such as Emoji and
Extended_Pictographic. We also add support for the Grapheme_Cluster_Break,
Word_Break and Sentence_Break enumeration properties.
Ensure `[[:blank:]]` only matches `[ \t]`. It appears that there was
a transcription error when `regex-syntax` was rewritten such that
`[[:blank:]]` ended up matching more than it was supposed to.
Fixes#533
This updates the documentation on `allow_invalid_utf8` to reflect the
current behavior of the translator. The old documentation was describing
the behavior of regex-syntax 0.5, but it was changed in regex-syntax
0.6.
This adds `scripts/generate.py`, and uses it to regenerate all tables
with data from Unicode 11.0.0. This also restores the character tests
that were first added in #400, with a new one for 11.
The issue with the ASCII version of \B is that it can match between code
units of UTF-8, which means it can cause match indices reported to be on
invalid UTF-8 boundaries. Therefore, similar to things like `(?-u:\xFF)`,
we ban negated ASCII word boundaries from Unicode regular expressions.
Normal ASCII word boundaries remain accessible from Unicode regular
expressions.
See #457
This commit removes our explicit implementations of encode_utf8 and
replaces them with uses of `char::encode_utf8`, which was added to the
standard library in Rust 1.15.
This commit fixes a bug with the handling of `(?flags)` sub-expressions
in the parser. Previously, the parser read `(?flags)`, added it to the
current concatenation, and then treat that as a valid sub-expression for
repetition operators, as in `(?i)*`. This in turn caused the translator
to panic on a failed assumption: that witnessing a repetition operator
necessarily implies a preceding sub-expression. But `(?i)` has no
explicit represents in the HIR, so there is no sub-expression.
There are two legitimate ways to fix this:
1. Ban such constructions in the parser.
2. Remove the assumption in the translator, and/or always translate a
`(?i)` into an empty sub-expression, which should generally be a
no-op.
This commit chooses (1) because it is more conservative. That is, it
turns a panic into an error, which gives us flexibility in the future to
choose (2) if necessary.
Fixes#465
This re-generates the Unicode table for property name aliases after fixing
a bug in property name canonicalization. Namely, the 'isc' alias of the
'ISO_Comment' property was being canonicalized to 'c', which is actually
an alias of the 'Other' general category. This is a result of the
canonicalization procedure ignoring 'is' prefixes, as permitted by UTS#18.
Fixes#466
This commit adds a new type of error message that is used whenever a
character class escape sequence is used as the start or end of a
character class range.
Fixes#461
This fixes an off-by-one bug in the error formatter. Namely, if a regex
ends with a literal `\n` *and* an error is reported that contains a span
at the end of the regex, then this trips a bug in the formatter because
its line count ends up being wrong. We fix this by tweaking the line
count. The actual error message is still a little wonky, but given the
literal `\n`, it's hard not to make it wonky.
Fixes#464
This adds a printer for the high-level intermediate representation. The
regex it prints is valid, and can be used as a way to turn it into a
regex::Regex.
Previously, we had some inconsistencies in how we were handling ASCII
word boundaries. In particular, the translator was accepting a negated
ASCII word boundary even if the caller didn't disable the UTF-8 invariant.
This is wrong, since a negated ASCII word boundary can match between any
two arbitrary bytes. However, fixing this is a breaking change, so for
now we document the bug. We plan to fix it with regex 1.0. See #457.
Additionally, we were incorrectly declaring that an ASCII word boundary
matched invalid UTF-8 via the Hir::is_always_utf8 property. An ASCII word
boundary must always match an ASCII byte on one side, which implies a
valid UTF-8 position.
This fixes a bug in the parser where a regex like `(?x)[ / - ]` would
fail to parse. In particular, since whitespace insensitive mode is
enabled, this regex should be equivalent to `[/-]`, where the `-` is
treated as a literal `-` instead of a range since it is the last
character in the class. However, the parser did not account for
whitespace insensitive mode, so it didn't see the `-` in `(?x)[ / - ]`
as trailing, and therefore reported an unclosed character class (since
the `]` was treated as part of the range).
We fix that in this commit by accounting for whitespace insensitive
mode, which we do by adding a `peek` method that skips over whitespace.
Fixes#455
