This commit represents a ground up rewrite of the regex-syntax crate.
This commit is also an intermediate state. That is, it adds a new
regex-syntax-2 crate without making any serious changes to any other
code. Subsequent commits will cover the integration of the rewrite and
the removal of the old crate.
The rewrite is intended to be the first phase in an effort to overhaul
the entire regex crate. To that end, this rewrite takes steps in that
direction:
* The principle change in the public API is an explicit split between a
regular expression's abstract syntax (AST) and a high-level
intermediate representation (HIR) that is easier to analyze. The old
version of this crate mixes these two concepts, but leaned heavily
towards an HIR. The AST in the rewrite has a much closer
correspondence with the concrete syntax than the old `Expr` type does.
The new HIR embraces its role; all flags are now compiled away
(including the `i` flag), which will simplify subsequent passes,
including literal detection and the compiler. ASTs are produced by
ast::parse and HIR is produced by hir::translate. A top-level parser
is provided that combines these so that callers can skip straight from
concrete syntax to HIR.
* Error messages are vastly improved thanks to the span information that
is now embedded in the AST. In addition to better formatting, error
messages now also include helpful hints when trying to use features
that aren't supported (like backreferences and look-around). In
particular, octal support is now an opt-in option. (Octal support
will continue to be enabled in regex proper to support backwards
compatibility, but will be disabled in 1.0.)
* More robust support for Unicode Level 1 as described in UTS#18.
In particular, we now fully support Unicode character classes
including set notation (difference, intersection, symmetric
difference) and correct support for named general categories, scripts,
script extensions and age. That is, `\p{scx:Hira}` and `p{age:3.0}`
now work. To make this work, we introduce an internal interval set
data structure.
* With the exception of literal extraction (which will be overhauled in
a later phase), all code in the rewrite uses constant stack space,
even while performing analysis that requires structural induction over
the AST or HIR. This is done by pushing the call stack onto the heap,
and is abstracted by the `ast::Visitor` and `hir::Visitor` traits.
The point of this method is to eliminate stack overflows in the
general case.
* Empty sub-expressions are now properly supported. Expressions like
`()`, `|`, `a|` and `b|()+` are now valid syntax.
The principle downsides of these changes are parse time and binary size.
Both seemed to have increased (slower and bigger) by about 1.5x. Parse
time is generally peanuts compared to the compiler, so we mostly don't
care about that. Binary size is mildly unfortunate, and if it becomes a
serious issue, it should be possible to introduce a feature that
disables some level of Unicode support and/or work on compressing the
Unicode tables. Compile times have increased slightly, but are still a
very small fraction of the overall time it takes to compile `regex`.
Fixes#174, Fixes#424
This implements parts of UTS#18 RL1.3, namely:
* Nested character classes, e.g.: `[a[b-c]]`
* Intersections in classes, e.g.: `[\w&&\p{Greek}]`
They can be combined to do things like `[\w&&[^a]]` to get all word
characters except `a`.
Fixes#341
Instead of ignoring space in all the bump/peek methods (as proposed in
pull request #349), have an explicit `ignore_space` method that can be
used in places where space/comments should be allowed.
This makes parsing a bit stricter than before as well.
When doing literal extraction, a non-empty concatenation should always be
cut when a `^` (for prefixes) or a `$` (for suffixes) is seen. If a counted
repetition is used, e.g., `${2}`, then the cut detection fails. We add in
a special case to handle it.
Fixes#321
Writing all of the testing scripts inside the .travis.yml file was
becoming painful, and parts of it were wrong by allowing for some
commands to fail without failing the entire build.
This also fixes the Github token (again).
The escaping of &, - and ~ is only required when the characters are
repeated adjacently, which should be quite rare. Escaping of [ is always
required, unless it appear in the second position of a range.
These rules enable us to add character class sets as described in
UTS#18 RL1.3 in a backward compatible way.
For example, the regex `[:upper:]` used to correspond to the `upper`
ASCII character class, but it now corresponds to the character class
containing the characters `:upper:`.
Forms like `[[:upper:][:blank:]]` are still accepted.
Fixes#175
For `[^\x00-\xff]`, while it is still treated as a full Unicode
character class, it is not empty. For instance `≥` would still be
matched.
However, when `CharClass::to_byte_class` is called on it (as is done
when using `regex::bytes::Regex::new` rather than `regex::Regex::new`),
it _is_ now empty, since it excludes all possible bytes.
This commit adds a test asserting that `regex::bytes::Regex::new`
returns `Err` for this case (in accordance with
https://github.com/rust-lang-nursery/regex/issues/106) and adds an
`is_empty` check to the result of calling `CharClass::to_byte_class`,
which allows the test to pass.
Specifically, given the strings ABCX, CDAX and BCX, it was reporting
the unambiguous set as A, BCX and CDAX, which is wrong since A is a
substring of CDAX.
unambiguous_prefixes is now quite a bit of a mess, but so is the rest
of the literal extraction code. The only thing it has going for it is
a massive test suite.
Fixes#289
In particular, if a range started or ended with `-`, then the pretty
printer would naively print `-` as if it were a normal character. For
example, `[--/]` corresponds to the class `[-/]` since the first `-` is
treated as a literal `-` and not part of any range.
There are a couple ways to fix this. For now, we fix the pretty printer
so that it never prints a `-` as part of a range.
This bug was found through ripgrep:
https://github.com/BurntSushi/ripgrep/issues/156
The specific problem here is that our literal search doesn't know about
anchors, so it will try to search all of the detected literals in a regex.
In a regex like `a|^b`, the literal `b` should only be searched for at the
beginning of the haystack and in no other place.
The right way to fix this is probably to make the literal detector smarter,
but the literal detector is already too complex. Instead, this commit
detects whether a regex is partially anchored (that is, when the regex has
at least one matchable sub-expression that is anchored), and if so,
disables the literal engine.
Note that this doesn't disable all literal optimizations, just the
optimization that opts out of regex engines entirely. Both the DFA and the
NFA will still use literal prefixes to search. Namely, if it searches and
finds a literal that needs to be anchored but isn't in the haystack, then
the regex engine rules it out as a false positive.
Fixes#268.
This commit fixes a bug where matching (?-u:\B) (that is, "not an ASCII
word boundary") in the NFA engines could produce match positions at invalid
UTF-8 sequence boundaries. The specific problem is that determining whether
(?-u:\B) matches or not relies on knowing whether we must report matches
only at UTF-8 boundaries, and this wasn't actually being taken into
account. (Instead, we prefer to enforce this invariant in the compiler, so
that the matching engines mostly don't have to care about it.) But of
course, the zero-width assertions are kind of a special case all around,
so we need to handle ASCII word boundaries differently depending on
whether we require valid UTF-8.
This bug was noticed because the DFA actually handles this correctly (by
encoding ASCII word boundaries into the state machine itself, which in turn
guarantees the valid UTF-8 invariant) while the NFAs don't, leading to an
inconsistency.
Fix#241.
When Unicode mode is disabled, we also disable the use of Unicode literals
in the regular expression, since it can lead to unintuitive behavior. In
this case, Unicode literals in character classes were not disallowed, and
subsequent code filtered them out, which resulted in an empty character
class. The compiler assumes that empty character classes are not allowed,
and so this causes an assert to trigger.
Fixes#250.
The compiler in particular assumes that it never gets an empty character
class. The current parser is pretty paranoid about rejecting empty classes,
but a few tricky cases made it through. In particular, one can write
`[^\d\D]` to correspond to "match nothing." This commit now looks for
empty classes explicitly, and if one is found, returns an error.
Interestingly, other regex engines allow this particular idiosyncrasy and
interpret it as "never match." Even more interesting, expressions like
`a{0}` are also allowed (including by this regex library) and are
interpreted as "always match the empty string." Both seem semantically the
same. In any case, we forbid empty character classes, primarily because
that seems like the sensible thing to do but secondarily because it's the
conservative choice.
It seems plausible that such a construct could be occasionally useful if
one were machine generating regexes, because it could be used to indicate
"never match." If we do want to support that use case, we'll need to add
a new opcode to the regex matching engines. One can still achieve
that today using something like `(a|[^a])`.
Fixes#257, where using such a form caused an assert to trip in the
compiler. A new, more explicit assert has been added.
When popping the parse stack at the end of parsing, we weren't actually
checking that the stack was empty. If the stack isn't empty, then that
indicates an unclosed parenthesis.
Fixes#239.
