* Fix issue #1443.
As pointed out in the bug report the way of aligning the win_cert pointer to the next 8-bytes boundary is not correct. If the previous certificates ends at 0x1234000A, the next one should start at 0x12340010, not at 0x1234000c which is not aligned to 8 bytes.
When we hit the maximum match limit on a string we currently abort all scanning,
which can be problematic since it isn't a good reason to stop all scanning, and
is better off as a warning to the user.
This commit makes it so that when the string match limit is hit a warning is
printed and scanning will continue. The warning message indicates that the
results may be invalid. It is hard to know for certain if they are invalid
because it depends entirely upon the condition. If you use the # or @ operator
then your condition _MAY_ be invalid since the total number of matches and
locations is not known anymore. Just checking that the string exists won't be an
invalid state since we have confirmed the existence of _AT LEAST_ the maximum
number of occurrences, which clearly satisfies the existence of a single
instance of your string.
I decided to implement this as a callback so the user of libyara has some choice
on the default behavior. If they want to consider this a "stop all scanning"
error they are free to return CALLBACK_ERROR in their callback.
The current Mach-O parsing code attempts to call macho_rva_to_offset
while parsing load commands and before it has set number_of_segments.
Since macho_rva_to_offset uses number_of_segments this can result
in it using a number_of_segments that is uninitialized and is just a random
large number. In turn yara runs a really long time.
This adds a test binary based on this post
https://codegolf.stackexchange.com/questions/102471/smallest-possible-runnable-mach-o-executable/154685#154685
It has two load commands. The first is a LC_UNIXTHREAD command and
the second is a LC_SEGMENT command. It's a valid 386 Mach-O binary.
It reproduces this error.
UPX packed files will cause export parsing to stop parsing because the function
address is not a valid RVA, and I was refusing to parse that export in that
case.
This is a regression from how we used to do parsing (we were never checking the
export RVA so it was never considered in prior releases). This is causing rules
which used to work to no longer work on packed samples. Specifically, checking
for a DLL which is UPX packed using pe.exports("foo") is likely to not work in
4.0.0 when it used to work in 3.11.0.
Apologies for missing this, but to make sure it doesn't happen again in the
future I've added a test case for this (just packed an existing DLL that is in
the repo).
Add test for invalid use of base64 and fullword together
Skip second base64 encoding of single ascii byte
Add test for correct base64 encoding of single ascii byte
Add two new modifiers: base64 and widebase64.
These modifiers take the given text string and generate 3 different strings
being careful to trim off the bytes which are dependent upon leading or
trailing bytes in the larger search space.
I've implemented it by slightly cheating. I generate all the search strings in
a list and then creating one large string suitable for the RE compiler to deal
with. For example, the string "This program cannot" generates these three
base64 encoded strings:
VGhpcyBwcm9ncmFtIGNhbm5vd
RoaXMgcHJvZ3JhbSBjYW5ub3
UaGlzIHByb2dyYW0gY2Fubm90
Those three strings are then transformed into a RE that looks like this:
(VGhpcyBwcm9ncmFtIGNhbm5vd|RoaXMgcHJvZ3JhbSBjYW5ub3|UaGlzIHByb2dyYW0gY2Fubm90)
That string is then passed to the RE compiler for parsing and AST generation.
The AST is then emitted into the appropriate spot and YARA believes it was
given a regex from that point on.
I've also implemented support for specifying custom alphabets:
base64("!@#$%^&*(){}[].,|ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstu")
This means that I have to be careful to escape any special RE characters in the
string passed to the compiler. The base64 alphabet has to be 64 bytes long, and
does support escaped bytes properly too.
To avoid the need to deal with escaping I had a first implementation which
attempted to generate the AST by hand, which was mostly working but was very
cumbersome to maintain. In doing this I ended up improving yr_re_print_node()
so that it indents the tree properly, which made debugging that attempt easier.
* Fix Rich header parsing.
The rich header doesn't always start at 0x80. The proper way to find the rich
header is to start at the NT header and work backwards one dword at a time
(technically the NT header is supposed to be 8 byte aligned, if someone has a
sample which isn't I'd love to see it) looking for Rich. Once you find it you
can get the XOR key which follows it and then continue to look backwards for
the DanS (after applying the XOR).
Decent documentation on this is at
http://bytepointer.com/articles/the_microsoft_rich_header.htm.
Also, fun fact which we may want to consider implementing is that the XOR key is
actually derived from the DOS header, so it may be an interesting way to tell if
the rich header has been tampered with if they do not properly recalculate the
key.
Fixes#1149.
* Split Rich signature walking into two loops.
While here, add a test that makes sure the clear version of the rich signature
is properly calculated.
* Fixes for xor modifier.
This commit makes the xor modifier no longer include "plaintext" versions of the
strings. In particular if you have:
$a = "This program cannot" xor
Then it will no longer match on the plaintext version of the string. It will
only match on the xor versions. If you want it to match on plaintext or xor then
you can use:
$a = "This program cannot" xor ascii
The same is true for wide.
* Make 'xor nocase' a compiler error.
* Add newline between ascii and wide strings.
Add some tests for the dotnet module, which uses the smallest .NET binary I had
laying around that exercises a fair bit of the code. In particular it has a
resource, guids, assembly information and a couple of other things that make for
good tests.
Add a bunch of tests which exercise improved DLL parsing. The DLL I'm using for
testing was taken from a Windows 10 VM I had laying around and was chosen
because it was the smallest DLL I had which has at least one local function and
one forwarded function.
Also, I'm glad I wrote these tests because they exposed a copy/paste mistake in
exports_index_regex where I was returning 1 and not the index where the regex
matched.
