Commit Graph
31 Commits
Author SHA1 Message Date
Martin Potter 1af8016a5c Regex fixes for substring parsing. 2014-08-04 14:42:35 -07:00
Bojan Rajkovic 1bbb0f0880 Revert "Bug 2663: Regular Expression Bug Fixes"
This reverts commit dc6efdab29. This
patch completely regressed performance for regexes in the same class
as the regexes it fixed. Xamarin bug #6198 details the regressions,
and I've confirmed that reverting this commit fixes the 2 order of
magnitude performance drop (with the patch, the test takes almost 7
seconds, without it, it takes .07 seconds). The bug still needs to be
fixed, but this is not the right fix.
2012-10-17 12:35:57 -04:00
Marek Safar 5d659d3500 Fixes a warning 2012-06-19 10:54:29 +01:00
Rob Wilkens 60a463c6f3 Bug 2663: Regular Expression Bug Fixes
What this fixes:
IF you were checking a regex which contained something like "(A|AB)C" and then you
checked something like "ABC" against it, it would fail previously, because its
first trip thru the regex it would match "A" first and short circuit out of the
"or" (|)  then next try to match "C" but find "B" and fail.  In this case, it should
be obvious that it needs to, after it fails, go back and try "AB" rather than "A"
and that's where this patch comes in.

Brief summary of each change in order that "git diff --staged" is giving them to
me:

This first one summarizes the background of goal of project:
New Function: EmitJumpTest
For when we are evaluating Alternatives (or's) we now want to be able to go back
and try or paths we haven't already tried if and only if we fail.  This alternate to
EmitJump instead of jumping to target if we have a match immediately, will
on first attempt do the jump, but later if we have already tried this jump
combination, it will jump to ElseTarget, which is the equivalent of this test
failing on subsequent attempts with the same combination of 'or's passing.

RxCompiler:
EmitJumpTest isn't implemented for RxCompiler because i didn't have sample
code to test it on, so it falls back to a regular jump, nothing changes in
RxCompiler in other words other than having this required fucntion front.

arch.cs:
New OpCode added for JumpTest (see EmitJumpTest above)

compiler.cs:
EmitJumpTest added here for regular compiler, it now takes two arguments (two
items on stack) first was the standard where to jump to address, the second
argument is where to jump to in case where we have already tried this
combination)

interpreter.cs:

JumpTestEntry - What we keep track of to know if we've been here before, for
now i presumed just 'pc' (program counter') and 'ptr' (pointer - where we are)
were enough to tell us if we've been to this particular spot before.

Eval has been moved into Eval_Real

Eval if it gets called with parameters indicating we've already been here
(i.e we're being called recursively) just passes the call directly to eval_real
as-was.

Otherwise eval will loop on eval_real while we fail and haven't tried every
possible path through all the or's. (Note: The special case of where we're
running a 'test' (i.e. checking !(a|b|c) rather than just matching (a|b|c)),
and if we're running a test, we turn off JumpTestTracking while it's in its
recursive section.  The reason for this is because while we can rightly check
all the options of (a|ab|abc)  first for a then try again with ab then try again
with abc, we cannot do the same for a not (i.e. !(something or something else)
is really (not something AND not somtehing else), and it is not (not something
or not something else))

JumpTestList tracks our current run's JumpTest calls...
TriedCombo's tracks all the combos (all the JumpTestList results) that we've
already tried.

All calls to anything which calls Eval were modified to call eval passing
the optional new parameters that we now need, and remember if we call
eval with those parameters having a value, it will just forward the call
to Eval_Real.

This included modifying TryMatch because it calls Eval.

OpCode.JumpTest handler was added, what it does is first check if we need
to ByPass to jump to 'else' right away, which again happens when we're in
a test recursive call (Checking not, for example).
Next, it will create and populate a new JumpTestEntry with the current values.
It then calls CheckComboMatch() which sees if we match a combo already on file.
If we don't have a Match we jump to the first label (at pc+1) which is the
regular jump call.  If we have a match, though, it will jump to the else label
which is now at (pc+2).

CheckComboMatch had to be broken up into two functions because it was going
too deep in terms of nesting.  What it does by now should be somewhat understood
it will see if we have a match between our current run and previous runs, and
if we have a match, it returns true.  If we don't have a match, it returns
false.  I have some doubts about whether this function is 100% correct, there
may remain a bug or two, but it does pass all the "System" unit tests which
includes all the regular expression tests, plus it works for all the tests
in the bug report, so i have little reason to mess with it.

syntax.cs:
Here's where i modified so that when evaluating alternatives ("or"s or "|"s)
we instead call EmitJumpTest rather than just a regular EmitJump.

RegExBugs.cs:
This is the unit test (a bunch in one) which fails on mono without this patch
but succeeds with it
commit 48027ac2bdd9b60d18985ec2537eb188607f8164 1 parent 91c4180d55

This patch was originally submitted under another branch which had a bunch of
unrelated commits in it.  I'm hoping if i did this right, these changes stand
alone.
2012-06-18 07:38:57 -04:00
Raja R Harinath 56c2a1dc07 Mitigate a possible source of unbounded memory growth
* interpreter.cs (Backtrack): Move mark_start back, and invalidate
discarded marks.

svn path=/trunk/mcs/; revision=159647
2010-06-29 07:20:12 +00:00
Zoltan Varga f1e948d1b0 2008-02-10 Zoltan Varga <[email protected]>
* replace.cs (NeedsGroupsOrCaptures): New property. Determines whenever the
	replacement process needs access to the groups or captures properties of the
	Match object.

	* Group.cs: Add an internal ctor which avoids allocations.

	* Match.cs: Ditto.

	* BaseMachine.cs interpreter.cs: Add an optimization to avoid fully creating
	Match objects during Replace () if the replacement string is simple.

svn path=/trunk/mcs/; revision=95394
2008-02-10 18:08:27 +00:00
Zoltan Varga 36b055d414 2008-02-10 Zoltan Varga <[email protected]>
* interpreter.cs (Eval): Remove a needless string allocation.

svn path=/trunk/mcs/; revision=95353
2008-02-10 02:16:24 +00:00
Arina Itkes 61d27ad7fc - Refactoring of Interpreter with extracting of base abstract class that executes some methods that were moved from Regex and Match classes.
- Added a field that maps group numbers to group names in Regex for improvement of performance of GroupNameFromNumber method.

svn path=/trunk/mcs/; revision=88065
2007-10-24 14:18:57 +00:00
Raja R Harinath b27e2a88ae * interpreter.cs (Eval) <OpCode.Reference>: Distribute for loop inside if.
for () if (a) s1; else s2; => if (a) for () s1; else for () s2;

svn path=/trunk/mcs/; revision=75393
2007-04-04 14:46:59 +00:00
Andrew Skiba 692e5cf88b revert unnecessary TARGET_JVM
svn path=/trunk/mcs/; revision=66184
2006-10-03 12:50:13 +00:00
Andrew Skiba 7f0db39bd5 TARGET_JVM
svn path=/trunk/mcs/; revision=66033
2006-09-28 13:45:23 +00:00
Raja R Harinath 083ccb79d8 In System.Text.RegularExpressions:
Fix #78278
	Remove 65535-limit on number of repetitions matched by a pattern.
	We still have a 65535 limit on the length of a pattern and the
	number of groups in a pattern.
	* compiler.cs (PatternCompiler.EmitCount): New.  Emits an int as
	two ushorts into the program stream.
	(EmitInfo, EmitRepeat, EmitFastRepeat): Use it to emit integers
	rather than shorts.
	* interpreter.cs (Intepreter.ReadProgramCount): Read an int
	emitted into the program stream.
	(Interpreter): Use it.  Update counts.
	(Interpreter.Eval) [OpCode.Repeat, OpCode.FastRepeat]: Likewise.
	* parser.cs (ParseGroup): Pass 0x7ffffff as the max value for '*'
	and '+' repetition patterns.
	* arch.cs (Info, Repeat, FastRepeat): Update description.

In Test/System.Text.RegularExpressions:
	* RegexBugs.cs (Kill65535_1, Kill65535_2, No65535Limit): New tests
	based on #78278.

svn path=/trunk/mcs/; revision=60401
2006-05-08 13:44:23 +00:00
Raja R Harinath d96a4c42b2 In System.Text.RegularExpressions:
Fix #78007
	* interpreter.cs (Interpreter.Eval) [Anchor, Position.StartOfString]:
	Don't reset 'ptr' to 0 during forward scan.

In Test/System.Text.RegularExpressions:
	* RegexBugs.cs (Bug78007): New test from #78007.

svn path=/trunk/mcs/; revision=59177
2006-04-07 13:44:55 +00:00
Raja R Harinath 4d94dda2bf In System.Text.RegularExpressions:
Fix #76345
	* interpreter.cs (Interpreter.FastEval) [FastRepeat]: If the first
	tail operation has a 'negate' flag, avoid the "match next char"
	optimization.

In Test/System.Text.RegularExpressions:
	* RegexBugs.cs (Bug76345): New test from #76345.

svn path=/trunk/mcs/; revision=59174
2006-04-07 12:01:44 +00:00
Raja R Harinath aeacac9505 In System.Text.RegularExpressions:
Fix #69269
	* arch.cs (OpCode.NotCategory): New.  Stands for matching a
	character _not_ from the given category.
	* debug.cs (DisassembleBlock): Handle it.
	* compiler.cs (ICompiler.EmitNotCategory): New.
	(Compiler.EmitNotCategory): New.  Emit OpCode.NotCategory.
	* syntax.cs (CharacterClass.Compile): Don't conflate negation of
	the character class and negation of the category.  Use
	EmitNotCategory.
	* interpreter.cs (Interpreter.Eval): Pass OpCode.NotCategory to
	EvalChar.
	(Interpreter.EvalChar): Handle it.

In Test/System.Text.RegularExpressions:
	* RegexBugs.cs (Bug69269): New test from #69269.

svn path=/trunk/mcs/; revision=59173
2006-04-07 10:00:23 +00:00
Raja R Harinath 91300ea498 Fix #77487
* System.Text.RegularExpressions/interpreter.cs (Eval) [Until, FastUntil]:
Set 'deep' to null when evaluating the tail.  Ensure that backtracks don't
confuse the recursion vs. iteration detector.
* Test/System.Text.RegularExpressions/RegexBugs.cs (Bug77487):
New test based on #77487.

svn path=/trunk/mcs/; revision=59114
2006-04-06 14:29:06 +00:00
Raja R Harinath 5c74d8a989 * interpreter.cs (Eval) [Until, lazy]: Avoid extra evaluation on a
degenerate match.

svn path=/trunk/mcs/; revision=58954
2006-04-03 14:04:53 +00:00
Raja R Harinath 955798252a * interpreter.cs (Interpreter.Eval) [OpCode.Until]: Invert the
sense of a test to reflect the code re-organization.

svn path=/trunk/mcs/; revision=50566
2005-09-23 09:34:08 +00:00
Raja R Harinath 26594d40db Fix #72152, #72989.
* interpreter.cs (Interpreter.Eval) [OpCode.Until]: Avoid some
cases of recursion when dealing with eager quantifiers too.

svn path=/trunk/mcs/; revision=50491
2005-09-22 15:06:55 +00:00
Raja R Harinath 89ef4f6585 Handle degenerate matches better.
svn path=/trunk/mcs/; revision=50370
2005-09-21 12:42:48 +00:00
Raja R Harinath 1a3f0b79ee * interpreter.cs (Interpreter.Eval) [OpCode.Until]: Avoid some
cases of recursion when dealing with the minimum count and lazy
quantifiers.

svn path=/trunk/mcs/; revision=50369
2005-09-21 12:24:12 +00:00
Raja R Harinath bcc7b46795 * interpreter.cs (Interpreter.GenerateMatch): Avoid allocating two
intermediate arrays to build the final result.
(Interpreter.GetGroupInfo, Interpreter.PopulateGroup): New helper
functions.
* CaptureCollection.cs (list): Change from ArrayList to list.
(SetValue): New internal helper, used by Interpreter.PopulateGroup.
(Enumerator): Remove helper class.
(IEnumerator.GetEnumerator): Just use list.GetEnumerator.
* GroupCollection.cs: Likewise.
* Group.cs (Group): Move responsibility of populating 'Captures'
to Interpreter.PopulateGroup.
* Match.cs (Match): Move responsibility of populating 'Groups' to
Interpreter.GenerateMatch.

This reduced the time for 100000 iterations of the 'regex'[1] benchmark
from 63s to 53s.

svn path=/trunk/mcs/; revision=45757
2005-06-10 13:54:07 +00:00
Gonzalo Paniagua Javier 1d9c9e2441 Fix warnings
svn path=/trunk/mcs/; revision=43368
2005-04-21 00:47:25 +00:00
Duncan Mak 24f5ef2c44 Add MIT license to System.dll
svn path=/trunk/mcs/; revision=30054
2004-06-21 18:02:11 +00:00
Gonzalo Paniagua Javier 3917ace10c 2004-04-19 Gonzalo Paniagua Javier <[email protected]>
* System.Text.RegularExpressions/arch.cs:
	* System.Text.RegularExpressions/compiler.cs:
	* System.Text.RegularExpressions/interpreter.cs:
	* System.Text.RegularExpressions/parser.cs:
	* System.Text.RegularExpressions/syntax.cs:
	* Test/System.Text.RegularExpressions/PerlTrials.cs:
	Patch by Eric Durand Tremblay.
	1) Capture inner group when named.
	2) Resolved parse error caused by not capturing inner group
	3) Resolved incorrect capture group
	4) Now, not capturing anything when unnamed ( correct behavior)

svn path=/trunk/mcs/; revision=25691
2004-04-19 17:07:46 +00:00
Gonzalo Paniagua Javier 2e689c7335 2004-04-19 Gonzalo Paniagua Javier <[email protected]>
* arch.cs:
	* compiler.cs:
	* interpreter.cs:
	* parser.cs:
	* syntax.cs: converted to unix line endings.

svn path=/trunk/mcs/; revision=25688
2004-04-19 16:54:36 +00:00
Gonzalo Paniagua Javier 3015c8a6b3 2004-03-17 Francois Beauchemin <[email protected]>
* syntax.cs, interpreter.cs, quicksearch.cs, regex.cs, compiler.cs :
 		Revised support for RigthToLeft.
 		quicksearch has now an reverse option.
 		This fixes bug #54537

  	* regex.cs, compiler.cs :
 	 	Some code to support CILCompiler.
 	* regex.cs :
 		Added some undocumented of MS.

svn path=/trunk/mcs/; revision=24249
2004-03-17 20:21:43 +00:00
Gonzalo Paniagua Javier 1b248b38fb 2003-11-27 Gonzalo Paniagua Javier <[email protected]>
* System.Text.RegularExpressions/interpreter.cs: when evaluating a
	degenerate match, restore the RepeatContext if fail. Fixes bug #42529.

	* Test/System.Text.RegularExpressions/RegexBugs.cs: added tests from
	bug #42529.

svn path=/trunk/mcs/; revision=20536
2003-11-27 15:31:59 +00:00
Dan Lewis 7a7c18e47c 2002-05-08 Dan Lewis <[email protected]>
* interpreter.cs: Moved to an array-based stack representation
	for faster captures.

	* match.cs, collections.cs: Decoupled capture representation from
	interpreter internals.

	* cache.cs: Changed Key type from struct to class for speed.

svn path=/trunk/mcs/; revision=4415
2002-05-08 12:42:49 +00:00
Dan Lewis a25f3cbe52 Branch bug was affecting FastRepeat: a(?:b|c|d)*(.) threw a stack overflow.
Added note about (?(X)...) capture/expression disambiguation.

svn path=/trunk/mcs/; revision=2215
2002-01-31 18:15:22 +00:00
Dan Lewis ad375c8180 Regex classes, initial revision.
svn path=/trunk/mcs/; revision=2205
2002-01-31 08:00:16 +00:00