This commit implements the "%" operator and extends the grammar to
recognize it for string sets.
The grammar now recognizes "50% of them" or variations on that. The
percentage number must be between 1 and 100 (inclusive) or the compiler will
return an error.
The implementation uses a new opcode (OP_OF_PERCENT) and this is because the
stack for these looks different from a typical "for_expression _OF_ string set"
rule. In the "for_expression" case (which uses OP_OF) the stack looks like this
(assuming two strings $a and $b):
$a
$b
undefined <-- end of list
2 <-- for_expression
The OP_OF operation checks each of the strings until it hits the end of list and
then compares the number found to whatever is left on the stack (2 in this
case).
The new grammar causes a stack at runtime which looks like this:
50 <-- percent number
$b
$a
undefined <-- end of list
The OP_OF operator won't work in this case because the percent number is sitting
on the top of the stack compared to the for_expression sitting at the bottom of
the stack.
I also cleaned up a minor oversight in compiler.c when printing the
ERROR_INVALID_MODIFIER message. It is the same as the other compiler errors so I
just moved it into the same set of fallthrough case statements.
Lastly, add some test cases to make sure this is working as intended.
Co-authored-by: Wesley Shields <[email protected]>
This is a combination of "OP_OF" and "OP_FOUND_IN" that allows you to express logic like "all of them in (0..100)". This came out of the discussion in #1483 and likely makes part of that pull request redundant.
Reading memory pages from /proc/$PID/mem has the unpleasant side
effect of forcing otherwise unused process memory pages into the
target process VM, considerably increasing the target process'
resident set size.
As a typical example, many programs only make use of a subset of code
from shared libraries that they are linked against. The solution
consists of replicating memory mappings mappings from files if they
are directly available through the filesystem. All information needed
to find the file (and to determine if it is indeed identical to the
file mapped by the target process) is available from /proc/$PID/maps.
Only pages that may have been changed according to /proc/$PID/pagemap
are then read via /proc/$PID/mem to overwrite the local mapping.
Mappings whose underlying filesystem object has changed read from
/proc/$PID/mem as before.
Mappings that are not backed by a regular file (block devices, stack
or heap sections, etc.) are assumed to behave like zeroed-out files
and otherwise treated the same.
* Minor re-styling.
* Roll our own is_alnum() equivalent.
Use yr_isalnum() when checking if the match is fullword. See
https://github.com/VirusTotal/yara-python/issues/184 for details of what is
happening here, as this only happens on MacOS python AFAICT.
* Fix word character in RE code.
Use yr_isalnum() in RE code for handling \w.
* Fix styling.
* Roll our own is_alnum() equivalent.
Use yr_isalnum() when checking if the match is fullword. See
https://github.com/VirusTotal/yara-python/issues/184 for details of what is
happening here, as this only happens on MacOS python AFAICT.
* Fix word character in RE code.
Use yr_isalnum() in RE code for handling \w.
* Fix styling.
Co-authored-by: Victor M. Alvarez <[email protected]>
Add a --with-debug-verbose option to the configure script which takes a number
greater than 0 as an argument. This gives us the flexibility to make verbosity
of this useful debugging information greater than it's current maximum (2)
without needing to modify the configure script.
Currently the maximum is 2, which gets you a lot of useful runtime debugging
information. A value of 1 is only used in tests. It may be worth it to consider
reworking this system so the numbers describe how much information you want.
For example, 1 could just be function entry and exit, 2 could add arguments to
it. I'll leave changing that up to a future diff if it is desired.
While here, fix a warning in the scanner from using an incorrect format string:
scan.c:644:7: warning: format specifies type 'long long' but the argument has type 'size_t' (aka 'unsigned long') [-Wformat]
match_offset,
^~~~~~~~~~~~~
./include/yara/globals.h:100:29: note: expanded from macro 'YR_DEBUG_FPRINTF'
fprintf(STREAM, FORMAT, __VA_ARGS__); \
~~~~~~ ^~~~~~~~~~~
1 warning generated.
after the first null
There are certain samples like 0043812838495a45449a0ac61a81b9c16eddca1ad249fb4f7fdb1c4505e9bb34
which contain section names with additional characters after the first
null. This is an useful information which can be used to detect such
anomalies. YARA is not able to process those even though VirusTotal web interface is able to detect them.
PE specification states that section name is "An 8-byte, null-padded UTF-8 encoded string. If the string is exactly 8 characters long, there is no terminating null.".
If you however consider the first null as the terminator then
null-padding requirement from PE specification is not adhered. If you
what we are doing in this commit then you are also going against the
specification because we are not honoring the fact that those strings
are null-terminated. We however found that having more information is
more useful than having less information, hence this PR.
The change is made with backward compatibility in mind so for vast
majority of samples, nothing changes. Their section names will remain
the same. It only affect such weird samples like the one mentioned above
so even though it might break some rules on some specific samples, it
shouldn't really affect anyone that much.
Fix an issue with the TOO_MANY_MATCHES warning where it would set the
STRING_FLAGS_DISABLED bit on the string, but it would never be reset when the
scanner was used to scan another file. This meant that a string that caused
TOO_MANY_MATCHES would be disabled permanently for the lifetime of that scanner,
when the intention was to only disable it for that one scan. A future scan
should have the string re-enabled until it matches too many times again.
I decided to fix this by using a bitmask for each string in the rules. This
allows me to very quickly reset them all without having to touch every string to
unset the STRING_FLAGS_DISABLED flag.
This also returns the STRING_FLAGS_DISABLED flag to it's original meaning, which
was that the string was disabled permanently because it is causing serious
performance problems.
I added a test for all this too.
yr_re_fast_exec used a very naive approach based on backtracking which had extremely bad performance with certain pathological cases. For example the following pattern could take more than a minute when applied to a small file containing the sequence 0xAA 0xAA 0xAA 0xAA three times, with around 100 zeroes following each of those sequences:
{ AA AA AA AA [1-60] 00 [1-60] 00 [1-60] AA AA AA AA [1-60] 00 [1-60] 00 [1-60] AA AA AA AA [1-60] 00 [1-60] BB }
With the new implementation it takes a fraction of a second.
There are certain samples where Rich signature starts immediatelly after
MZ header. This check just allows this offset to be inspected during
the detection loop.
If the file_size function is NULL the filesize keyword in a YARA rule will evaluate to undefined. This makes the _undefined_file_size function in proc.c unnecessary.
Until now the release version has been used as the ABI version, but that's not recommended at all. Start tracking the ABI version with its own version number, starting with 8 in order to make clear that this number is not tied to the release version.
