The x86 platform is very forgiving when you access a 16-bits variable stored at a memory address that is not aligned to a 2-bytes boundary, or a 32-bits variable that is not aligned to 4-bytes boundary, and so on. Other platforms, like ARM or Sparc are not that flexible, and accessing a value by dereferencing a pointer that is not aligned to the size of the value causes a processor fault.
This fixes multiple issues caused by pointers that are not aligned to the size of the value.
Fixes: #1700
With this change the yr_scanner_scan_mem_blocks function can return ERROR_BLOCK_NOT_READY when the YR_MEMORY_BLOCK_ITERATOR that provides memory blocks to the function is not ready to return the next block. After this error the caller can retry the call to yr_scanner_scan_mem_blocks multiple times until the function succeeds or fails with some other error code.
Co-authored-by: Simon Hardy-Francis <[email protected]>
This removes 3 unused bytes at the end of each SIZED_STRING. Those 3 bytes are not initialized by alloc_sized_string and cause MemorySanitizer to fail with use-of-uninitialized-value.
* Add private modifier for strings.
This commit adds a private modifier to the grammar, which can be used on any
string. When this modifier is used the string is treated as normal except it is
never reported back in match objects. It is still searched for normally and
still used normally in condition evaluation but it will never be seen in the
match callback.
This is a useful feature when you have strings that you want to use but they
match a lot of times and you don't want them polluting your log results. The
best example that demonstrates this is the technique described at [1]. In this
example the local file header magic is going to match a lot of times, and it is
only needed to look through the local file header for filenames at offsets
relative to the matches. We don't really care where the offsets of the local
file header are, just where the matching filenames are.
If you are logging where all the matches are for every string and there are
10,000 local file headers that's a lot of log data being generated which is not
helpful at all.
[1]: https://blog.korelogic.com/blog/2015/08/19/yara-ctf-2
* Optimize VM execution code for private strings.
Rather than checking the matches list and only if none are found then check for
private matches, it's better to check only the appropriate list after using
STRING_IS_PRIVATE.
* Add private strings in various tests.
This should cover all the cases of operators on strings that also need to check
for private strings.
* Add docs for private strings.
* Fixup documentation.
Private strings apply to all string types, not just text strings.
* Fix private modifier for chained strings. Add a test case for it too.
In 3cceccd8b6 the increment of profiling counters was changed to an atomic operation. This fixed race-conditions but introduced a noticeable contention while using the same YR_RULES structure with multiple threads. This commit fixes this problem by using one counter per thread, which are aggregated later, outside the critical path.
With this new feature you can pass an atom prevalence table to YARA that will be used for computing the quality of each atom based in real-life data instead of using heuristics. The atom prevalence table consists in a list of atoms, sorted lexicographically and accompanied by a score between 0-255 that indicates how frequent is that atom in the data you use to scan with YARA. With that information YARA can make better selection of the atoms.
This commit implements two new opcodes for regular expressions and replaces _yr_scan_fast_re_exec with a cleaner implementation in yr_re_fast_scan, as the existing implementation used some hacks that were hard to maintain.
* Travis: Show test-suite.log on failure
* Fix alignment tests to account for change made in 7ff47de
* Travis: Configure 32-bit build without support for cuckoo, magic
YR_MATCH was keeping a pointer to the original scanned data. With the introduction of iterator logic the original data is not guaranteed to be in memory by the time YR_MATCH is passed to library clients. Now a copy of the matching string is kept alongside the YR_MATCH structure.
Victor M. Alvarez