
A lazy DFA is much faster than executing an NFA because it doesn't repeat the work of following epsilon transitions over and and over. Instead, it computes states during search and caches them for reuse. We avoid exponential state blow up by bounding the cache in size. When the DFA isn't powerful enough to fulfill the caller's request (e.g., return sub-capture locations), it still runs to find the boundaries of the match and then falls back to NFA execution on the matched region. The lazy DFA can otherwise execute on every regular expression *except* for regular expressions that contain word boundary assertions (`\b` or `\B`). (They are tricky to implement in the lazy DFA because they are Unicode aware and therefore require multi-byte look-behind/ahead.) The implementation in this PR is based on the implementation in Google's RE2 library. Adding a lazy DFA was a substantial change and required several modifications: 1. The compiler can now produce both Unicode based programs (still used by the NFA engines) and byte based programs (required by the lazy DFA, but possible to use in the NFA engines too). In byte based programs, UTF-8 decoding is built into the automaton. 2. A new `Exec` type was introduced to implement the logic for compiling and choosing the right engine to use on each search. 3. Prefix literal detection was rewritten to work on bytes. 4. Benchmarks were overhauled and new ones were added to more carefully track the impact of various optimizations. 5. A new `HACKING.md` guide has been added that gives a high-level design overview of this crate. Other changes in this commit include: 1. Protection against stack overflows. All places that once required recursion have now either acquired a bound or have been converted to using a stack on the heap. 2. Update the Aho-Corasick dependency, which includes `memchr2` and `memchr3` optimizations. 3. Add PCRE benchmarks using the Rust `pcre` bindings. Closes #66, #146.
regex
A Rust library for parsing, compiling, and executing regular expressions. This particular implementation of regular expressions guarantees execution in linear time with respect to the size of the regular expression and search text by using finite automata. In particular, it makes use of both NFAs and DFAs when matching. Much of the syntax and implementation is inspired by RE2.
Documentation
Module documentation with examples. The module documentation also include a comprehensive description of the syntax supported.
Documentation with examples for the various matching functions and iterators
can be found on the
Regex
type.
Usage
Add this to your Cargo.toml
:
[dependencies]
regex = "0.1"
and this to your crate root:
extern crate regex;
Here's a simple example that matches a date in YYYY-MM-DD format and prints the year, month and day:
extern crate regex;
use regex::Regex;
fn main() {
let re = Regex::new(r"(?x)
(?P<year>\d{4}) # the year
-
(?P<month>\d{2}) # the month
-
(?P<day>\d{2}) # the day
").unwrap();
let caps = re.captures("2010-03-14").unwrap();
assert_eq!("2010", caps.name("year").unwrap());
assert_eq!("03", caps.name("month").unwrap());
assert_eq!("14", caps.name("day").unwrap());
}
If you have lots of dates in text that you'd like to iterate over, then it's easy to adapt the above example with an iterator:
extern crate regex;
use regex::Regex;
const TO_SEARCH: &'static str = "
On 2010-03-14, foo happened. On 2014-10-14, bar happened.
";
fn main() {
let re = Regex::new(r"(\d{4})-(\d{2})-(\d{2})").unwrap();
for caps in re.captures_iter(TO_SEARCH) {
// Note that all of the unwraps are actually OK for this regex
// because the only way for the regex to match is if all of the
// capture groups match. This is not true in general though!
println!("year: {}, month: {}, day: {}",
caps.at(1).unwrap(),
caps.at(2).unwrap(),
caps.at(3).unwrap());
}
}
This example outputs:
year: 2010, month: 03, day: 14
year: 2014, month: 10, day: 14
Usage: Avoid compiling the same regex in a loop
It is an anti-pattern to compile the same regular expression in a loop since compilation is typically expensive. (It takes anywhere from a few microseconds to a few milliseconds depending on the size of the regex.) Not only is compilation itself expensive, but this also prevents optimizations that reuse allocations internally to the matching engines.
In Rust, it can sometimes be a pain to pass regular expressions around if
they're used from inside a helper function. Instead, we recommend using the
lazy_static
crate to ensure that
regular expressions are compiled exactly once.
For example:
#[macro_use] extern crate lazy_static;
extern crate regex;
use regex::Regex;
fn some_helper_function(text: &str) -> bool {
lazy_static! {
static ref RE: Regex = Regex::new("...").unwrap();
}
RE.is_match(text)
}
Specifically, in this example, the regex will be compiled when it is used for the first time. On subsequent uses, it will reuse the previous compilation.
Usage: regex!
compiler plugin
The regex!
compiler plugin will compile your regexes at compile time. This
only works with a nightly compiler.
The
documentation explains the trade
offs.
Here is a small example:
#![feature(plugin)]
#![plugin(regex_macros)]
extern crate regex;
fn main() {
let re = regex!(r"(\d{4})-(\d{2})-(\d{2})");
let caps = re.captures("2010-03-14").unwrap();
assert_eq!("2010", caps.at(1).unwrap());
assert_eq!("03", caps.at(2).unwrap());
assert_eq!("14", caps.at(3).unwrap());
}
Notice that we never unwrap
the result of regex!
. This is because your
program won't compile if the regex doesn't compile. (Try regex!("(")
.)
Due to recent optimizations in the regex
crate, the normal "dynamic" regex
created via Regex::new(...)
is faster in almost all cases than regex!(...)
.
In theory, this should be temporary, but the path to fixing it isn't quite
clear yet.
Usage: a regular expression parser
This repository contains a crate that provides a well tested regular expression parser and abstract syntax. It provides no facilities for compilation or execution. This may be useful if you're implementing your own regex engine or otherwise need to do analysis on the syntax of a regular expression. It is otherwise not recommended for general use.
Documentation for regex-syntax
with
examples.
License
regex
is primarily distributed under the terms of both the MIT license and
the Apache License (Version 2.0), with portions covered by various BSD-like
licenses.
See LICENSE-APACHE, and LICENSE-MIT for details.