The issue with the ASCII version of \B is that it can match between code
units of UTF-8, which means it can cause match indices reported to be on
invalid UTF-8 boundaries. Therefore, similar to things like `(?-u:\xFF)`,
we ban negated ASCII word boundaries from Unicode regular expressions.
Normal ASCII word boundaries remain accessible from Unicode regular
expressions.
See #457
This commit removes our explicit implementations of encode_utf8 and
replaces them with uses of `char::encode_utf8`, which was added to the
standard library in Rust 1.15.
This commit fixes a bug with the handling of `(?flags)` sub-expressions
in the parser. Previously, the parser read `(?flags)`, added it to the
current concatenation, and then treat that as a valid sub-expression for
repetition operators, as in `(?i)*`. This in turn caused the translator
to panic on a failed assumption: that witnessing a repetition operator
necessarily implies a preceding sub-expression. But `(?i)` has no
explicit represents in the HIR, so there is no sub-expression.
There are two legitimate ways to fix this:
1. Ban such constructions in the parser.
2. Remove the assumption in the translator, and/or always translate a
`(?i)` into an empty sub-expression, which should generally be a
no-op.
This commit chooses (1) because it is more conservative. That is, it
turns a panic into an error, which gives us flexibility in the future to
choose (2) if necessary.
Fixes#465
This re-generates the Unicode table for property name aliases after fixing
a bug in property name canonicalization. Namely, the 'isc' alias of the
'ISO_Comment' property was being canonicalized to 'c', which is actually
an alias of the 'Other' general category. This is a result of the
canonicalization procedure ignoring 'is' prefixes, as permitted by UTS#18.
Fixes#466
This commit adds a new type of error message that is used whenever a
character class escape sequence is used as the start or end of a
character class range.
Fixes#461
This fixes an off-by-one bug in the error formatter. Namely, if a regex
ends with a literal `\n` *and* an error is reported that contains a span
at the end of the regex, then this trips a bug in the formatter because
its line count ends up being wrong. We fix this by tweaking the line
count. The actual error message is still a little wonky, but given the
literal `\n`, it's hard not to make it wonky.
Fixes#464
This adds a printer for the high-level intermediate representation. The
regex it prints is valid, and can be used as a way to turn it into a
regex::Regex.
Previously, we had some inconsistencies in how we were handling ASCII
word boundaries. In particular, the translator was accepting a negated
ASCII word boundary even if the caller didn't disable the UTF-8 invariant.
This is wrong, since a negated ASCII word boundary can match between any
two arbitrary bytes. However, fixing this is a breaking change, so for
now we document the bug. We plan to fix it with regex 1.0. See #457.
Additionally, we were incorrectly declaring that an ASCII word boundary
matched invalid UTF-8 via the Hir::is_always_utf8 property. An ASCII word
boundary must always match an ASCII byte on one side, which implies a
valid UTF-8 position.
This fixes a bug in the parser where a regex like `(?x)[ / - ]` would
fail to parse. In particular, since whitespace insensitive mode is
enabled, this regex should be equivalent to `[/-]`, where the `-` is
treated as a literal `-` instead of a range since it is the last
character in the class. However, the parser did not account for
whitespace insensitive mode, so it didn't see the `-` in `(?x)[ / - ]`
as trailing, and therefore reported an unclosed character class (since
the `]` was treated as part of the range).
We fix that in this commit by accounting for whitespace insensitive
mode, which we do by adding a `peek` method that skips over whitespace.
Fixes#455
This commit fixes an embarrassing bug where the depth in the nest limit
checker was never decremented during postorder traversal, which means
long but shallow regexes would incorrectly trip the nest limit. We fix
that in this commit and add two regression tests.
Fixes#454
This commit does the mechanical changes necessary to remove the old
regex-syntax crate and replace it with the rewrite. The rewrite now
subsumes the `regex-syntax` crate name, and gets a semver bump to 0.5.0.
This commit represents a ground up rewrite of the regex-syntax crate.
This commit is also an intermediate state. That is, it adds a new
regex-syntax-2 crate without making any serious changes to any other
code. Subsequent commits will cover the integration of the rewrite and
the removal of the old crate.
The rewrite is intended to be the first phase in an effort to overhaul
the entire regex crate. To that end, this rewrite takes steps in that
direction:
* The principle change in the public API is an explicit split between a
regular expression's abstract syntax (AST) and a high-level
intermediate representation (HIR) that is easier to analyze. The old
version of this crate mixes these two concepts, but leaned heavily
towards an HIR. The AST in the rewrite has a much closer
correspondence with the concrete syntax than the old `Expr` type does.
The new HIR embraces its role; all flags are now compiled away
(including the `i` flag), which will simplify subsequent passes,
including literal detection and the compiler. ASTs are produced by
ast::parse and HIR is produced by hir::translate. A top-level parser
is provided that combines these so that callers can skip straight from
concrete syntax to HIR.
* Error messages are vastly improved thanks to the span information that
is now embedded in the AST. In addition to better formatting, error
messages now also include helpful hints when trying to use features
that aren't supported (like backreferences and look-around). In
particular, octal support is now an opt-in option. (Octal support
will continue to be enabled in regex proper to support backwards
compatibility, but will be disabled in 1.0.)
* More robust support for Unicode Level 1 as described in UTS#18.
In particular, we now fully support Unicode character classes
including set notation (difference, intersection, symmetric
difference) and correct support for named general categories, scripts,
script extensions and age. That is, `\p{scx:Hira}` and `p{age:3.0}`
now work. To make this work, we introduce an internal interval set
data structure.
* With the exception of literal extraction (which will be overhauled in
a later phase), all code in the rewrite uses constant stack space,
even while performing analysis that requires structural induction over
the AST or HIR. This is done by pushing the call stack onto the heap,
and is abstracted by the `ast::Visitor` and `hir::Visitor` traits.
The point of this method is to eliminate stack overflows in the
general case.
* Empty sub-expressions are now properly supported. Expressions like
`()`, `|`, `a|` and `b|()+` are now valid syntax.
The principle downsides of these changes are parse time and binary size.
Both seemed to have increased (slower and bigger) by about 1.5x. Parse
time is generally peanuts compared to the compiler, so we mostly don't
care about that. Binary size is mildly unfortunate, and if it becomes a
serious issue, it should be possible to introduce a feature that
disables some level of Unicode support and/or work on compressing the
Unicode tables. Compile times have increased slightly, but are still a
very small fraction of the overall time it takes to compile `regex`.
Fixes#174, Fixes#424
This implements parts of UTS#18 RL1.3, namely:
* Nested character classes, e.g.: `[a[b-c]]`
* Intersections in classes, e.g.: `[\w&&\p{Greek}]`
They can be combined to do things like `[\w&&[^a]]` to get all word
characters except `a`.
Fixes#341
Instead of ignoring space in all the bump/peek methods (as proposed in
pull request #349), have an explicit `ignore_space` method that can be
used in places where space/comments should be allowed.
This makes parsing a bit stricter than before as well.
When doing literal extraction, a non-empty concatenation should always be
cut when a `^` (for prefixes) or a `$` (for suffixes) is seen. If a counted
repetition is used, e.g., `${2}`, then the cut detection fails. We add in
a special case to handle it.
Fixes#321
Writing all of the testing scripts inside the .travis.yml file was
becoming painful, and parts of it were wrong by allowing for some
commands to fail without failing the entire build.
This also fixes the Github token (again).
The escaping of &, - and ~ is only required when the characters are
repeated adjacently, which should be quite rare. Escaping of [ is always
required, unless it appear in the second position of a range.
These rules enable us to add character class sets as described in
UTS#18 RL1.3 in a backward compatible way.
For example, the regex `[:upper:]` used to correspond to the `upper`
ASCII character class, but it now corresponds to the character class
containing the characters `:upper:`.
Forms like `[[:upper:][:blank:]]` are still accepted.
Fixes#175
For `[^\x00-\xff]`, while it is still treated as a full Unicode
character class, it is not empty. For instance `≥` would still be
matched.
However, when `CharClass::to_byte_class` is called on it (as is done
when using `regex::bytes::Regex::new` rather than `regex::Regex::new`),
it _is_ now empty, since it excludes all possible bytes.
This commit adds a test asserting that `regex::bytes::Regex::new`
returns `Err` for this case (in accordance with
https://github.com/rust-lang-nursery/regex/issues/106) and adds an
`is_empty` check to the result of calling `CharClass::to_byte_class`,
which allows the test to pass.
