* Fix negative indexing in dex module.
When attempting to call dex_get_integer() or dex_get_string() with a negative
index we would eventually land in the assert() at
https://github.com/VirusTotal/yara/blob/master/libyara/object.c#L497 failing.
Instead of doing that let's check for negative values before going any further,
which will at least allow the module to continue processing.
* YR_UNDEFINED is < 0 already. Simplify the logic when checking for negative index.
* Revert "YR_UNDEFINED is < 0 already. Simplify the logic when checking for negative index."
This reverts commit 38af38fb8d.
* Fix compiler warnings with dex debug mode.
* Fix crashes in dex module.
This commit fixes a few crashes in the dex module. There are actually three of
them:
The first is incorrect usage of "struct_fits_in_dex" caused by passing
"sizeof(code_item_t)" instead of just "code_item_t" as the third argument. In
the test case the pointer for code_item started in the bounds of the dex but
only the first 8 bytes were within bounds, and since
"sizeof(sizeof(code_item_t))" is less than 8 the check was passing. The fix here
is to pass just the struct type as the third argument.
The second crash was an off-by-one error when parsing a string. The check
ensured the string fits in the dex but was not including an extra byte which was
copied in the call to set_sized_string. Just like before, this was a case of a
string falling right on the end of a dex file.
The third crash was due to a missing "struct_fits_in_dex" check. We ended up
with a pointer to a map_item_t which was off the ends of the dex bounds.
With this commit all the test cases provided in the report are now passing. I
did a quick sweep of the module to make sure there were no other cases where we
were incorrectly using "struct_fits_in_dex" and didn't find any.
These were all documented at a private report via huntr.dev
(https://huntr.dev/bounties/007a7784-c211-4847-9cc3-aec38e7d5157/)
Found by @sudhackar.
Fixes#1726.
The x86 platform is very forgiving when you access a 16-bits variable stored at a memory address that is not aligned to a 2-bytes boundary, or a 32-bits variable that is not aligned to 4-bytes boundary, and so on. Other platforms, like ARM or Sparc are not that flexible, and accessing a value by dereferencing a pointer that is not aligned to the size of the value causes a processor fault.
This fixes multiple issues caused by pointers that are not aligned to the size of the value.
Fixes: #1700
* Fix another compiler edge case.
When compiling and we encounter a range that has values we can determine at
compile time we should error out if the lower bound is less than 0.
* Fix OP_COUNT_IN paranoid check.
The OP_COUNT_IN operation does not need the paranoid check on r1 as it is the
lower bound of the range, and thus a constant and not a pointer somewhere. It
should, instead, be checking r3.
Fixes#1721 (the first half of it).
* test: Relax pe32 condition in magic tests
Turns out different versions of libmagic return different file
identification for the PE32 sample in magic tests. Relax this condition
to the common part of the identification to make tests less reliant on
the testing platform.
* test: Relax macho condition in magic tests
Trying to run `autoreconf -fiv` on a Windows system using wsl causes the following error:
```
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --force -I m4
autoreconf: configure.ac: tracing
autoreconf: running: libtoolize --copy --force
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, 'build-aux'.
libtoolize: copying file 'build-aux/ltmain.sh'
.ibtoolize: error: AC_CONFIG_MACRO_DIRS([m4]) conflicts with ACLOCAL_AMFLAGS=-I m4
autoreconf: libtoolize failed with exit status: 1
```
This is likely due to a CRLF error in the makefile. This commit applies the following fix:
https://pete.akeo.ie/2010/12/that-darn-libtoolize-acconfigmacrodirm4.html
test_process_scan launches a process and tests YARA's process scanning capabilities on it. This process was maintained alive for 10 minutes, which caused `make -j32 check` to hang during all that time before returning. This change reduces the time to 5 seconds, which should be enough for running the test cases.
Closes#1667 .
test_process_scan launches a process and tests YARA's process scanning capabilities on it. This process was maintained alive for 10 minutes, which caused `make -j32 check` to hang during all that time before returning. This change reduces the time to 5 seconds, which should be enough for running the test cases.
Closes#1683.
* Fix some compiler edge cases.
Add a warning when "0 of them" is used. Point to some new documentation that
explains why "none of them" is preferred.
Handle some edge cases, which are all now errors:
* -1 of them
* "foo" of them
* /foo/ of them
And this one is also an error if the external variable "x" is a negative integer
or a string:
* x of them
I've also made sure this works with a construct like "0 + x of them" where x is
defined as a negative integer.
Add test cases for as many of these as I can. I don't have test cases for the
external variable case, but it does work as expected. Here is the output of a
rule using external variables where the variable is defined to be 1, 0, -1 and
foo:
```
wxs@mbp yara % cat rules/test.yara
rule a {
strings:
$a = "FreeBSD"
condition:
x of them
}
wxs@mbp yara % ./yara -d x=1 rules/test.yara /bin/ls
a /bin/ls
wxs@mbp yara % ./yara -d x=0 rules/test.yara /bin/ls
warning: rule "a" in rules/test.yara(5): Consider using "none" keyword, it is less ambiguous. Please see https://yara.readthedocs.io/en/stable/writingrules.html#sets-of-strings-1 for an explanation.
wxs@mbp yara % ./yara -d x=-1 rules/test.yara /bin/ls
error: rule "a" in rules/test.yara(5): invalid value in condition: "-1"
wxs@mbp yara % ./yara -d x=foo rules/test.yara /bin/ls
error: rule "a" in rules/test.yara(5): invalid value in condition: "string in for_expression is invalid"
wxs@mbp yara %
```
* More compiler warnings and errors.
If you specify "2 of ($a)" it is now a warning because it will always evaluate
to false, and is quite possibly not what you wanted to write. This is true for
string sets, rule sets and string sets in a range like "2 of ($a) in (0..10)".
If you are using an integer range it is now an error if the lower bound is
greater than the upper bound.
To support rule sets I needed to modify yr_parser_emit_pushes_for_rules() to
use a count parameter, which it will update with the number of pushed rules.
This is identical to how yr_parser_emit_pushes_for_strings() works.
While I'm here, remove the link from the warning message for the "none" keyword.
"%define parse.error detailed" is changed to "%define parse.error verbose" as the former is not supported by Bison versions older than 3.6, while the latter produces good enough syntax error messages.
...short-circuit evaluation and usage of none with them.
`for .. in` and `for .. of` loops now use the same codegen for their
bytecode using 2 instructions `ITER_CONDITION` and `ITER_END`.
`ITER_CONDITION` evaluates each iteration whether the condition of the
loop was satisfied prematurely or whether the loop is still satisfiable,
so essentially making sure that loops are properly short-circuited.
`ITER_END` is used for final evaluation of the loop, whether it is
satisfied or not, after the iterator got exhausted or the loop was
short-circuited.
In order to be able to use single codegen for both iterators and string
sets, new iterator type was instroduced - `YR_ITERATOR_STRING_SET`. When
`ITER_START_STRING_SET` is executed, it is capable of consuming all
strings pushed on the stack and construct `YR_ITERATOR_STRING_SET`
object which can then be used as a regular iterator.
As discussed in https://github.com/VirusTotal/yara/issues/1695, there is an
issue with some expressions involving "0" (and by extension "none"). If you
said "0 of them" and one of them matched we would evaluate that expression to
true because we never checked for the special 0 case.
Turns out this bug has existed for a while now but was likely never triggered
because not many people would say "0 of them" before the "none" keyword came
around. However, the author of the issue is correct that this should have been
exposed with better tests when I implemented the none keyword.
Fixes#1695.
In ARM the pointer returned by yr_notebook_alloc was being rounded up to a 4-bytes boundary, but the size of the buffer was not increased accordingly. This means the caller of yr_notebook_alloc could receive a buffer that is 1 to 3 bytes smaller than the requested sized.
Regular expressions with a character class followed by a dash (-) failed with "bad character range". For example, /[0-9]-2/ was not accepted as a valid regexp.
Until now this macro was aborting the parsing with YYERROR, independently of the kind of error. YYERROR aborts the parsing in a recoverable way, allowing the parsing to continue with the next rule. However, some errors like ERROR_INSUFFICIENT_MEMORY may leave the compiler in an inconsistent state and we should not continue with the parsing.
* Add not operator (~) for bytes in hex strings.
* Describe not hexstring not operator in documentation.
* Basic tests for the hextring NOT operator
* Fix not operator documentation ordering
iterator->last_error is now initialized to ERROR_SUCCESS in yr_process_open_iterator. Additionally if yr_process_get_first_memory_block returns, NULL, iterator->last_error is set to ERROR_COULD_NOT_READ_PROCESS_MEMORY.
HoundThe