* Added warning on unknown escape sequences
* fixed memory leaks and different output when using regex as operand with strict-escape parameter
* Remove some unwanted characters from strict escaping, add testcases
With this change the yr_scanner_scan_mem_blocks function can return ERROR_BLOCK_NOT_READY when the YR_MEMORY_BLOCK_ITERATOR that provides memory blocks to the function is not ready to return the next block. After this error the caller can retry the call to yr_scanner_scan_mem_blocks multiple times until the function succeeds or fails with some other error code.
Co-authored-by: Simon Hardy-Francis <[email protected]>
* tests: Disable process scanning test on Windows
There's no sys/wait.h and no fork, so let's just #ifdef it out.
* tests: Open sample files in binary mode
Relevant on Windows.
When we hit the maximum match limit on a string we currently abort all scanning,
which can be problematic since it isn't a good reason to stop all scanning, and
is better off as a warning to the user.
This commit makes it so that when the string match limit is hit a warning is
printed and scanning will continue. The warning message indicates that the
results may be invalid. It is hard to know for certain if they are invalid
because it depends entirely upon the condition. If you use the # or @ operator
then your condition _MAY_ be invalid since the total number of matches and
locations is not known anymore. Just checking that the string exists won't be an
invalid state since we have confirmed the existence of _AT LEAST_ the maximum
number of occurrences, which clearly satisfies the existence of a single
instance of your string.
I decided to implement this as a callback so the user of libyara has some choice
on the default behavior. If they want to consider this a "stop all scanning"
error they are free to return CALLBACK_ERROR in their callback.
* Review all calls to YR_DEBUG_FPRINTF and make the following changes:
* Use PRIx64 and PRIi64 for printing 64-bits values, which is more portable.
* Use %zu for printing size_t types.
* Remove the empty {} at the end of printed messages, as they don't provide that much information and clutter the format string
* Don't use ' in format strings for thousand separators as this is not very portable (https://stackoverflow.com/questions/10234766/warning-using-icc-and-printf-with-thousands-grouping-format-apostrophe)
* Re-introduce the curly braces in YR_DEBUG_FPRINTF messages.
The callback called by the compiler to report errors and warnings now received a pointer to the rule causing the error/warning. This allows richer error messages.
* Implement improved xor modifier.
* Fix breakage when xor(...) came before another modifier.
* Add documentation for xor improvements.
* Fixes from review by Victor.
- No more changes to lexer.
- Duplicate modifiers are now errors.
- Do not rely on undocumented behavior in bison.
* Apply fix to deal with undocumented bison feature we shouldn't rely on.
* Make xor work properly with other modifiers.
This commit now makes it so that "xor(1-255) wide" will no longer search for the
plaintext wide version of the string, but "xor wide" will because it implies
xor(0-255).
* Add support for xor(X) and tests.
* Add test case.
* Fix issue with regexps and hex strings that were not splitted correctly.
The function yr_re_ast_split_at_chaining_point was not splitting correctly regexps and hex strings that contained more than one large jump (a.k.a gap). For example, { 01 02 03 [0-1000] 04 05 06 [500-2000] 07 08 09 } was not splitted in three parts as expected, but only in two parts: { 01 02 03 } and { 04 05 06 [500-2000] 07 08 09 } . This didn't affected the correctness of the matches, but it caused ERROR_TOO_MANY_RE_FIBERS in some cases while trying to match { 04 05 06 [500-2000] 07 08 09 } because of the large jump in the pattern. The occurrence of the error depended on the data being scanned.
This commit fixes the aforementioned issue, improves the documentation of related functions and simplifies some portions of the code.
* Avoid putting the same string multiple times in matching_strings_arena.
This fixes a small issue introduced in #1096. The same string could be put more than once in the matching_strings_arena, this doesn't causes any major error nor affect the correctness of the matching, but it could have a negative impact on performance.
* Add more comments.
* Improve documentation for _yr_scan_verify_chained_string_match.
* YR_TRYCATCH: Remove duplicate initialization of oldmask
* Add test for handling exceptions
With
#define YR_TRYCATCH(_try_clause_, _catch_clause_) {_try_clause_}
this will cause a bus error on Linux -- as it should.
* tests: Add assert_*_rule_blob_size, make count_matches usable from test code
* test-exception: Block and send SIGUSR1 before 2nd yr_rules_scan_mem call
This seems to reproduce something similar to the symptom described in
$ ./test-exception
Scanning for "aaaa"...
err = 4, matches = 0
Sending blocked SIGUSR1 to ourselves...
Scanning for {00 00 00 00}...
User defined signal 1
* exception (UNIX): Remove unneeded pthread_sigmask() calls
sigsetjmp() already saves the signal mask for us.
Also, setting the signal mask using the (empty) act.sa_mask was wrong
and led to the behavior described in #544.
We still want delivery of all signals to be blocked during execution of
the signal handler.
* Document test-exception
YR_MATCH was keeping a pointer to the original scanned data. With the introduction of iterator logic the original data is not guaranteed to be in memory by the time YR_MATCH is passed to library clients. Now a copy of the matching string is kept alongside the YR_MATCH structure.
The test code for parsing and behavior of the rule engine belongs with
Yara itself rather than the the Python bindings that have been split
off. Now the tests can be run again when the library is built.
Victor M. Alvarez