There is an issue where the OP_OF_FOUND_IN opcode is broken when using "all of".
This is because the UNDEF that is pushed on the stack to indicate "all" is
compared to the number of matched strings as signed integers. If there are no
matched strings the condition still evaluates to true because 0 > (signed)
UNDEF.
Fix it by checking if we are using an UNDEF and compare it to the number of
strings checked if that is the case.
This adds the ability to use rule sets just like we do with string sets.
Specifically, it enables conditions like the following:
5 of (a, b)
50% of (a*)
where "a" is an existing rule and "b*" is a wildcard rule identifier, which is
expanded out precisely as wildcard string identifiers are.
I chose to make OP_OF and OP_OF_PERCENT emit an argument which is the "type" of
set it is expected to match on. The two acceptable values are OF_STRING_SET and
OF_RULE_SET. When OF_STRING_SET is used then the references on the stack are
expected to be strings and OF_RULE_SET will check if the rules on the stack
match.
I added a yr_parser_emit_pushes_for_rules() which behaves similarly to the
string counterpart. The one big exception is that I can't use yr_rules_foreach
macro since that needs the terminating "null" rule which is not yet added, so
instead I have to iterate the rules manually using yr_arena_get_ptr() to walk
all the currently compiled rules.
* Stricter ordering of rule set in compiler.
Make sure that when we compile a new rule that any existing rule sets do not
match the new rule. Specifically we are making the following a compiler error:
rule a1 { condition: true }
rule b { condition: 1 of (a*) }
rule a2 { condition: true }
This is a compiler error because "a2" is defined after the usage of (a*) in rule
b.
There is still an edge case here. These rules compile:
rule a { condition 1 of (a) }
rule b { condition: 1 of (b*) }
In both cases those rules evaluate to false, which is in line with the behavior
of this rule:
rule c { condition: c }
Given that the behavior around rule sets matches the current behavior I think
this is acceptable.
* Add docs for rule sets.
* Implement hash table iterators.
* Remove old comment about linked list.
* Move variable declaration.
* Iterate over rules better.
This is a more comprehensive fix than #1617, it adds new functions to the API for getting/setting uint32 and uint64 settings. Using these functions is preferable over calling `yr_(get|set)_configuration` directly.
YARA didn't support any expression which would check value against YR_UNDEFINED constant so checking for non-existence of something can be very problematic. But we run multiple times into issue of people asking us for checking of absence of something.
Because of this we propose operator defined which would be unary operator (like not) which works like this:
defined x evaluates to
true - whenever x != YR_UNDEFINED
false - whenever x == YR_UNDEFINED
Related to issue #835
* Implement OP_COUNT_IN.
Add a new instruction (OP_COUNT_IN) which is used to check the number of
occurrences of a string in a range. You can use this with the following syntax:
#a in (0..100) == 2
This will check that the $a string matches exactly 2 times in the range
provided.
* Add docs for counting strings in a range.
This commit implements the "%" operator and extends the grammar to
recognize it for string sets.
The grammar now recognizes "50% of them" or variations on that. The
percentage number must be between 1 and 100 (inclusive) or the compiler will
return an error.
The implementation uses a new opcode (OP_OF_PERCENT) and this is because the
stack for these looks different from a typical "for_expression _OF_ string set"
rule. In the "for_expression" case (which uses OP_OF) the stack looks like this
(assuming two strings $a and $b):
$a
$b
undefined <-- end of list
2 <-- for_expression
The OP_OF operation checks each of the strings until it hits the end of list and
then compares the number found to whatever is left on the stack (2 in this
case).
The new grammar causes a stack at runtime which looks like this:
50 <-- percent number
$b
$a
undefined <-- end of list
The OP_OF operator won't work in this case because the percent number is sitting
on the top of the stack compared to the for_expression sitting at the bottom of
the stack.
I also cleaned up a minor oversight in compiler.c when printing the
ERROR_INVALID_MODIFIER message. It is the same as the other compiler errors so I
just moved it into the same set of fallthrough case statements.
Lastly, add some test cases to make sure this is working as intended.
Co-authored-by: Wesley Shields <[email protected]>
This is a combination of "OP_OF" and "OP_FOUND_IN" that allows you to express logic like "all of them in (0..100)". This came out of the discussion in #1483 and likely makes part of that pull request redundant.
With this change the yr_scanner_scan_mem_blocks function can return ERROR_BLOCK_NOT_READY when the YR_MEMORY_BLOCK_ITERATOR that provides memory blocks to the function is not ready to return the next block. After this error the caller can retry the call to yr_scanner_scan_mem_blocks multiple times until the function succeeds or fails with some other error code.
Co-authored-by: Simon Hardy-Francis <[email protected]>
The new names are rules_table, strings_table and ext_vars_table respectively. Now that those fields are not linked list the previous names were somewhat confusing. Document the fields in YR_RULES a little bit better.
This cuts the condition evaluation program generated from the
signature-base[1] ruleset from 429053 to 329558 bytes, saving 23.1%
[1] https://github.com/Neo23x0/signature-base
This does't occur often, as this kind of errors are catch during compilation. However, this may occur when you try to use rules that were compiled with a version of YARA that has support for OpenSSL with another version that doesn't have it. That's because YARA with OpenSSL supports some fields in the "pe" module that are not allowed by YARA without OpenSSL. The compiled rules are valid for YARA with OpenSSL, but the fields won't be found when they are used without OpenSSL support.
Closes#1288
Some unaligned data access patterns are not entirely OK on Linux/ARM.
Depending on kernel configuration, it may cause bus errors or "only"
lead to massive slowdowns as the kernel works around faulting
instructions and perhaps even emits log messages. (This option does
not seem to exist for 32 bit code running on an ARM64 kernel, though.)
Most of this patch which was originally developed by Steve Langasek
has been part of the Debian package for a while.
This changes the way in which comparison operators, and also the "contains" and "matches" operators handle undefined operands. Before this change the result of such operations was undefined, after this change the result is false.
For loops like "for all i in (1..some_undefined_value)" were returning true in all cases. If the loop was like "for all i in (some_undefined_value..100)", it was it was executed too many times.
Instead of measuring the time spent in verifying all atom matches, do it only for a fraction of them. This way the cost of measuring the time is a smaller percentage of the time spent doing the actual verification.
Instead of measuring the time spent in verifying all atom matches, do it only for a fraction of them. This way the cost of measuring the time is a smaller percentage of the time spent doing the actual verification.
Frequency has been reduced to a check every 10 instructions to a check every 100 instructions. This improves performance at the expense of timeout accuracy.
Victor M. Alvarez