The hex string and regexp lexers parse jump lengths and repeat intervals
with atoi, which is undefined for numbers that don't fit in an int. glibc
hands back the low 32 bits, so a value that wraps to a small positive
number passes every bounds check and silently changes the pattern:
rule r1 { strings: $a = /a{4294967296}/ condition: $a }
rule r2 { strings: $b = { 61 [0-4294967297] 62 } condition: $b }
Before this change r1 compiled as a{0} and matched at every offset, and
r2 compiled as { 61 [0-1] 62 }. MSVC saturates atoi at INT_MAX instead,
so the same rule behaves differently there, which is what #1791 reports.
Parse with strtoll instead. Numbers too large for a long long are clamped
to LLONG_MAX, which still fails the range check, so "repeat interval too
large" now fires for everything above RE_MAX_RANGE on any platform. Hex
jumps get an explicit INT_MAX check because RE_NODE stores start and end
as ints, and the grammar already casts to int.
The hi_bound < 0 and lo_bound < 0 checks are gone. They were there to
catch atoi wrapping negative and are unreachable now that the only thing
the pattern matches is digits.
Fixes#1791.
Constant folding in `grammar.y` hits UB at the `INT64_MIN` boundary, reachable from rule text:
- `*`: `llabs(INT64_MIN)` is UB and returns a negative value, so the overflow guard is bypassed and folds to a garbage constant instead of `ERROR_INTEGER_OVERFLOW`. Now compares magnitudes in unsigned space.
- `\`: `INT64_MIN / -1` is overflow UB and raises SIGFPE on x86. Now rejected with `ERROR_INTEGER_OVERFLOW`.
- unary `-`: `-INT64_MIN` is negation UB. Now negated in unsigned space (wraps to `INT64_MIN`, unchanged behavior).
Reproduce `condition` expr (each reported an UBSan error):
`~9223372036854775807 * 9223372036854775807`
`~9223372036854775807 \ -1`
`-(~9223372036854775807)`
Example Logs:
```
libyara/grammar.y:2813:30: runtime error: negation of -9223372036854775808 cannot be represented in type 'int64_t' (aka 'long'); cast to an unsigned type to negate this value to itself
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior libyara/grammar.y:2813:30
```
Note: The VM ops in `exec.c` and the `%` rule likely share this UB.
Disclose: assisted with AI
In regular expressions, repetition operators`{0,0}` and `{,0}` are valid, but they always match the zero-length string. For instance, `a{0,0}` and `[a-z]{0,0}` both match the zero-length string.
When the whole regular expression consists in one of these repetitions it caused a segfault during the evaluation of the regular expression because the `forward_code_ref` field for the root atom in the atom's tree was null.
Closes#2084.
For instance, `x{n, m}`, `x{n ,m}` and `x{n , m}` are all equivalent to `x{n,m}`. This is done for compatibility with YARA-X, but I've seen real-life regular expressions in that contain such spaces, for example:
```
/C:\\Users\\qqq\\.{30, 50}\.pdb/
```
The `matches` operator was returning false if the left operand was an empty strings. So, `"" matches <some regexp>` was returning false regardless of the regexp used. However, some regexp *do* match the empty strings, like for example `/foo|/`.
Regexps with quantifiers `{0}` and `{0,0}` were considered syntactically invalid. These quantifiers are not actually useful, but they are now allowed for compatibility with other regexp engines.
Closes#2037.
Some regexps like `/(abc|)/` where not matching the empty string even if they should. Matching an empty string is not very useful, as such regexp will match on every offset of every file, but this is fixed for correctness.
Also fixes another issue with the `\b` metacharacter in regexps, which was exposed after fixing the main issue.
PR #1927 didn't take into account that as a side-effect of skipping the evaluation of global rules, the corresponding namespace was not flagged as unsatisfied (i.e: a global rule that evaluates to false prevents all other rules in the namespace from evaluating to true). The issue was not caught by any test case because the test case for global rules were using rules that didn't required any strings to match, and therefore were unaffected by the changes introduced in #1927.
This change makes sure that any global rule that is skipped is considered false, and therefore its namespace is flagged as unsatisfied.
Closes#2022.
* Added warning on unknown escape sequences
* fixed memory leaks and different output when using regex as operand with strict-escape parameter
* Remove some unwanted characters from strict escaping, add testcases
As briefly discussed in #1937, this change will make it so that any string
identifier that starts with '_' can be unreferenced. Any anonymous strings
must still be referenced.
While testing this out I realized that an unreferenced string still had the
STRING_FLAG_FIXED_OFFSET set, which meant any unreferenced string would have a
fixed_offset of YR_UNDEFINED. To deal with this when we are reducing the rule
we unset STRING_FLAG_FIXED_OFFSET if the string is unreferenced.
This is now illegal syntax:
for 3.14159 in (1): (1)
It doesn't make sense to allow floating point in the primary expression used in
for expressions. This is an extension of the work done in a5903e1d which banned
negative integers, strings and regexp from being used here too.
* Add warnings for edge cases.
When using "all of them at 0" it is now a warning if you have more than 1 string
defined in "them".
"N of them" (where N > 1) is also a warning.
* Use IS_UNDEFINED when checking integer expression.
* Add OP_OF_FOUND_AT.
Add support for "any of them at 0" constructs to the language. This allows users
to avoid using long or chains like "$a at 0 or $b at 0" and also is a nicer way
to write "for any of them: ($ at 0)".
* Add short blurb to docs about new string set at offset.
* Implement text string sets.
Add support for text string sets into the grammar. They look like this:
for any s in ("a", "b"): (pe.imphash() == s)
This requires changing integer_set and integer_enumeration to just be set and
enumeration, and adding a new type (YR_ENUMERATION) that tracks the type of
enumeration (integer or otherwise) and the number of items in the enumeration.
The enumeration now checks that each type of the item in the enumeration is
consistent and will cause a compiler error if they are not. For example, this is
an error:
for any s in ("a", 0): (s)
Also, fix the build when using --with-debug-verbose option as it was missing the
assert.h include.
* Add docs and adjust constants layout.
Before, this fix, this rule was true:
```
import "tests"
rule a {
strings:
$foo = "foo"
condition:
not defined ($foo in (0..tests.integer_array[40])) and
defined ($foo at tests.integer_array[40])
}
```
where `tests.integer_array[40]` is just a way to get an undefined value.
This is surprising, OP_FOUND_AT is the only operation of this kind that
is not propagating the undefined value. This commit fixes this behavior.
The x86 platform is very forgiving when you access a 16-bits variable stored at a memory address that is not aligned to a 2-bytes boundary, or a 32-bits variable that is not aligned to 4-bytes boundary, and so on. Other platforms, like ARM or Sparc are not that flexible, and accessing a value by dereferencing a pointer that is not aligned to the size of the value causes a processor fault.
This fixes multiple issues caused by pointers that are not aligned to the size of the value.
Fixes: #1700
* Fix another compiler edge case.
When compiling and we encounter a range that has values we can determine at
compile time we should error out if the lower bound is less than 0.
* Fix OP_COUNT_IN paranoid check.
The OP_COUNT_IN operation does not need the paranoid check on r1 as it is the
lower bound of the range, and thus a constant and not a pointer somewhere. It
should, instead, be checking r3.
Fixes#1721 (the first half of it).
test_process_scan launches a process and tests YARA's process scanning capabilities on it. This process was maintained alive for 10 minutes, which caused `make -j32 check` to hang during all that time before returning. This change reduces the time to 5 seconds, which should be enough for running the test cases.
Closes#1667 .
* Fix some compiler edge cases.
Add a warning when "0 of them" is used. Point to some new documentation that
explains why "none of them" is preferred.
Handle some edge cases, which are all now errors:
* -1 of them
* "foo" of them
* /foo/ of them
And this one is also an error if the external variable "x" is a negative integer
or a string:
* x of them
I've also made sure this works with a construct like "0 + x of them" where x is
defined as a negative integer.
Add test cases for as many of these as I can. I don't have test cases for the
external variable case, but it does work as expected. Here is the output of a
rule using external variables where the variable is defined to be 1, 0, -1 and
foo:
```
wxs@mbp yara % cat rules/test.yara
rule a {
strings:
$a = "FreeBSD"
condition:
x of them
}
wxs@mbp yara % ./yara -d x=1 rules/test.yara /bin/ls
a /bin/ls
wxs@mbp yara % ./yara -d x=0 rules/test.yara /bin/ls
warning: rule "a" in rules/test.yara(5): Consider using "none" keyword, it is less ambiguous. Please see https://yara.readthedocs.io/en/stable/writingrules.html#sets-of-strings-1 for an explanation.
wxs@mbp yara % ./yara -d x=-1 rules/test.yara /bin/ls
error: rule "a" in rules/test.yara(5): invalid value in condition: "-1"
wxs@mbp yara % ./yara -d x=foo rules/test.yara /bin/ls
error: rule "a" in rules/test.yara(5): invalid value in condition: "string in for_expression is invalid"
wxs@mbp yara %
```
* More compiler warnings and errors.
If you specify "2 of ($a)" it is now a warning because it will always evaluate
to false, and is quite possibly not what you wanted to write. This is true for
string sets, rule sets and string sets in a range like "2 of ($a) in (0..10)".
If you are using an integer range it is now an error if the lower bound is
greater than the upper bound.
To support rule sets I needed to modify yr_parser_emit_pushes_for_rules() to
use a count parameter, which it will update with the number of pushed rules.
This is identical to how yr_parser_emit_pushes_for_strings() works.
While I'm here, remove the link from the warning message for the "none" keyword.
As discussed in https://github.com/VirusTotal/yara/issues/1695, there is an
issue with some expressions involving "0" (and by extension "none"). If you
said "0 of them" and one of them matched we would evaluate that expression to
true because we never checked for the special 0 case.
Turns out this bug has existed for a while now but was likely never triggered
because not many people would say "0 of them" before the "none" keyword came
around. However, the author of the issue is correct that this should have been
exposed with better tests when I implemented the none keyword.
Fixes#1695.
Regular expressions with a character class followed by a dash (-) failed with "bad character range". For example, /[0-9]-2/ was not accepted as a valid regexp.
* Add not operator (~) for bytes in hex strings.
* Describe not hexstring not operator in documentation.
* Basic tests for the hextring NOT operator
* Fix not operator documentation ordering
There is an issue where the OP_OF_FOUND_IN opcode is broken when using "all of".
This is because the UNDEF that is pushed on the stack to indicate "all" is
compared to the number of matched strings as signed integers. If there are no
matched strings the condition still evaluates to true because 0 > (signed)
UNDEF.
Fix it by checking if we are using an UNDEF and compare it to the number of
strings checked if that is the case.
Fix issue #1646 by generalizing the solution provided in #1647. Atoms with a higher number of unique bytes are prioritized over atoms that have repeated characters.
Also fixes issue #1654, which was discovered in a debug session, but has been there for a long time.
This adds the ability to use rule sets just like we do with string sets.
Specifically, it enables conditions like the following:
5 of (a, b)
50% of (a*)
where "a" is an existing rule and "b*" is a wildcard rule identifier, which is
expanded out precisely as wildcard string identifiers are.
I chose to make OP_OF and OP_OF_PERCENT emit an argument which is the "type" of
set it is expected to match on. The two acceptable values are OF_STRING_SET and
OF_RULE_SET. When OF_STRING_SET is used then the references on the stack are
expected to be strings and OF_RULE_SET will check if the rules on the stack
match.
I added a yr_parser_emit_pushes_for_rules() which behaves similarly to the
string counterpart. The one big exception is that I can't use yr_rules_foreach
macro since that needs the terminating "null" rule which is not yet added, so
instead I have to iterate the rules manually using yr_arena_get_ptr() to walk
all the currently compiled rules.
* Stricter ordering of rule set in compiler.
Make sure that when we compile a new rule that any existing rule sets do not
match the new rule. Specifically we are making the following a compiler error:
rule a1 { condition: true }
rule b { condition: 1 of (a*) }
rule a2 { condition: true }
This is a compiler error because "a2" is defined after the usage of (a*) in rule
b.
There is still an edge case here. These rules compile:
rule a { condition 1 of (a) }
rule b { condition: 1 of (b*) }
In both cases those rules evaluate to false, which is in line with the behavior
of this rule:
rule c { condition: c }
Given that the behavior around rule sets matches the current behavior I think
this is acceptable.
* Add docs for rule sets.
* Implement hash table iterators.
* Remove old comment about linked list.
* Move variable declaration.
* Iterate over rules better.
YARA didn't support any expression which would check value against YR_UNDEFINED constant so checking for non-existence of something can be very problematic. But we run multiple times into issue of people asking us for checking of absence of something.
Because of this we propose operator defined which would be unary operator (like not) which works like this:
defined x evaluates to
true - whenever x != YR_UNDEFINED
false - whenever x == YR_UNDEFINED
Related to issue #835
* Implement OP_COUNT_IN.
Add a new instruction (OP_COUNT_IN) which is used to check the number of
occurrences of a string in a range. You can use this with the following syntax:
#a in (0..100) == 2
This will check that the $a string matches exactly 2 times in the range
provided.
* Add docs for counting strings in a range.
Cole Munz