The `matches` operator was returning false if the left operand was an empty strings. So, `"" matches <some regexp>` was returning false regardless of the regexp used. However, some regexp *do* match the empty strings, like for example `/foo|/`.
File e64e806c6df76f245c9b1288f59198bc9478329cfdd2d7ddcb9490a57d0b3cf4 was not properly parsed because `NumberOfRvaAndSizes` overflowed and became negative.
Some PE files have a NumberOfRvaAndSizes equal to zero. Even if these files are not accepted by the Windows loader, they can be parsed, and useful information can be extracted from them. By relaxing the sanity check in pe_get_directory_entry we are no able to parse such files (like 0aa527fd26c8e7fa7f3476b6d7eb8f22c071ff4c0fd3ed3797fa90ceb80f24ce).
.NET resources can be found outside the current file. Before this change those resources were being ignored, but now they are included in the `resources` array. The `name` and `length` fields are kept undefined though. So, external resources have only the `name` field set.
This function was checking the first two bytes at the entrypoint of .NET files, if the bytes were not `FF 25` the file was not considered a .NET file. This check was overly restrictive, as some files like `8fa7fe73a65296e9ca8301734a0deaa298cda0b7a65f6b7d2ef6d1c8bbb8cd7a` don't have such files at the entrypoint.
This function was returning a wrong RVA for files like `67508ec9cbd9f82c6aea8a93ca27fd3684d368664470397755f8d70934d481fb`, when a section's virtual size is smaller than its physical size.
Failing to parse some resource directory does not mean that we can't keep parsing other directories. For instance, with file `0a65e22cdff348be11739122906355af728f6247fb849f98d69a6f6d51103071` YARA was aborting the parsing too early and extracted a single resource, while the file has more resources that could be parsed correctly.
YARA was making the assumption that the 12 bytes following the `DanS` header contained 3 copies of the 32-bits XOR key, but these bytes are actually padding. As these bytes were usually filled with zeroes before applying the XOR key, they appear to contain the key itself after the XOR operation. However, files like `043066108b68b30fc2c475eae8edfafc080be7d451600eaa283d2c750bddbceb` don't contain zeroes in those bytes, and therefore were not satisfying YARA's assumption.
YARA-X was right: https://github.com/VirusTotal/yara-x/blob/6ada059b9e4b328e2341b42304765c5614fc89af/lib/src/modules/pe/parser.rs#L687-L693
Upon fixing the latest issue regarding parsing CMS, we figured out we
are not properly extracting all certificates from Microsoft
countersignatures. This should now properly expose them.
Also fixes one memory leak for countersignature parsing.
As discussed in https://github.com/VirusTotal/yara-python/issues/247, the example using `StringIO` worked fine in Python 2, but ti doesn't work in Python 3. Now that Python 2 is mostly dead, update the example.
This is redundant and already handled by TS_VERIFY_CTX_new().
Additionally it causes build failures with LibreSSL >= 3.8.0 which
removed this function since it serves no purpose with opaque
TS_VERIFY_CTX where it will zero out an already zero'd out ctx or cause
a leak.
See: https://github.com/libressl/portable/issues/1006
This is backward compatible change that affects the `magic` field. After this change the value in the `magic` field looks exactly as it looks in the file regardless of the endianness of the current platform, if the file starts with `CA FE BA BE` the value in magic is `0xCAFEBABE`, not `0xBEBAFECA` as it used to be in little-endian architectures.
Regexps with quantifiers `{0}` and `{0,0}` were considered syntactically invalid. These quantifiers are not actually useful, but they are now allowed for compatibility with other regexp engines.
Closes#2037.
Until now, `pe.is_signed` was undefined for every PE file that was not signed, making impossible to use it in negated conditions, like `pe.is_pe and not pe.is_signed`. With this change, `pe.is_signed` is false for every PE file that is not signed.
Closes#2040
The pointer returned from `libmagic` was passed directly to `yr_strdup` (a wrapper of `strdup`), but this function has undefined behavior when the argument is null.
Instead of downloading the latest master branch, use the code in the current branch. Until know, this workflow was always operating wit the latest commit in the master branch.
The process scanning on macos was broken since 4.2.0 in this commit:
605b2edf07
Logic was added to properly iterate on memory chunks if a memory region
was too big, but this completely broke the iteration. To retrieve a
memory region, the code basically does:
```
current_begin = 0;
for (;;) {
vm_address_t addr = current_begin;
mach_vm_region(..., &addr, &size, ...);
size_t chunk_size = size - (current_begin - addr);
...
}
```
It assumes that `addr` is smaller than `current_begin`, which is never
the case outside of chunking. This leads to an iteration on completely
wrong regions, leading to read that either fail because the starting
addr or length is wrong, or that succeed, but do not cover the whole
region.
For example, with a process memory that looks like:
```
1065a6000-1065f2000
1065f2000-1065f6000
1065f6000-1065fa000
...
600000000000-600008000000
600008000000-600010000000
...
7fe537f00000-7fe538000000
7fe538000000-7fe538800000
```
The very first call would have `current_begin=0` and `addr=0x1065a6000`,
leading to underflow when computing `chunk_size`, and trying to iterate
on the region `0x00000000-0x40000000` (as the max memory chunk is 1GB
per default). this will fail to fetch, and will then iterate on
`0x40000000-0x80000000`, then `0x80000000-0xC0000000`, then
`0xC0000000-0x100000000`. We will finally reach the very first real
memory region, but fail to properly compute the region size with another
underflow, etc.
This commit fixes the issue by distinguishing the two cases:
- either `addr > current_begin` : `current_begin` is not in any memory
region, and we must fast forward.
- or `addr <= current_begin`: we can compute the current chunk by
diffing the two values.
This was not tested because the test_process_scan was not running on
macos: trying to scan the sh process always fails even when root, and
the test was thus skipped. I hacked the test locally to check if it
worked by replacing the sh process by something else that we can attach
to, and it showed the scanning fails. With this commit, the test does
pass.
I did not commit any modification to the test as I used a custom binary
for this and it needs to be run as root, but it would probably be a good
idea to adapt it.
This updates authenticode parser to the latest versions which contains
sereral fixes which accumulated over time.
* Several time_t variables replaced with int64_t
* RFC5652 countersignatures are now correctly parsed
* Memory corruption related fixes where authenticode-parser would crash otherwise
Some regexps like `/(abc|)/` where not matching the empty string even if they should. Matching an empty string is not very useful, as such regexp will match on every offset of every file, but this is fixed for correctness.
Also fixes another issue with the `\b` metacharacter in regexps, which was exposed after fixing the main issue.
Some modules functions (from hash and math) can access multiple memory
blocks, as long as those are contiguous. For example, when hashing
between offsets 0..1000, blocks that cover this range will be fetched.
This logic however is a bit buggy on the boundary: if the fetched range
ends exactly at the end of a block, the iteration keeps going instead of
ending. This can result in two situations:
- The next block is contiguous: it is then fetched.
- if the fetch fails, the function fails (and return undefined)
- if the fetch succeeds, we read 0 bytes from it and
end the iteration, so a useless fetch was done.
- The next block is not contiguous: the function fails (and return
undefined).
This is fixed by modifying the condition to detect if there are still
more bytes to fetch.
While porting the `dotnet` module to YARA-X I discovered the the implementation of `read_blob_signed` had a subtle bug. As `first_byte` was declared as `uint8_t` instead of `int8_t`, sign extension when casting the value to a `int32_t` was not done, leading to incorrect values.
PR #1927 didn't take into account that as a side-effect of skipping the evaluation of global rules, the corresponding namespace was not flagged as unsatisfied (i.e: a global rule that evaluates to false prevents all other rules in the namespace from evaluating to true). The issue was not caught by any test case because the test case for global rules were using rules that didn't required any strings to match, and therefore were unaffected by the changes introduced in #1927.
This change makes sure that any global rule that is skipped is considered false, and therefore its namespace is flagged as unsatisfied.
Closes#2022.
This reverts commit 2d103d3ce3. The previous fix introduces shit-reduce and reduce-reduce conflicts in the grammar. The grammar.y file was not correctly processed by `bison`.
When scanning multiple memory blocks, we should stop as soon as a valid
elf is found. This was broken as the "break" applies to the switch, and
not the outer loop. Fix this by reworking the code to remove the switch.
This bug had two repercutions:
- instead of parsing the first memory block that contains an elf, it
returned the last one, or actually even worse, a mix of all elfs
depending on which values are set again or not. This however should
not manifest in normal situations, notably during process scanning, as
I suppose only one memory region would contain an elf executable.
- the module caused the fetch of all memory blocks instead of stopping
as soon as one was found, which is worse performance wise.
* fixed validation error in grammar.y
rule_modifiers 'private' and 'global' can no longer be used more than once in a rule definition. They can be used exclusively or together in any order.
* Update grammar.y
* feat: prevent race conditions on signal handler
If multiple threads were running in parallel (e.g. each using a
YR_SCANNER), race conditions could cause the original signal handler
to not be restored correctly.
Add a mutex that controls access to the (process wide) signal handler
to ensure that the YARA signal handler is only installed once and
only removed when it is no longer needed.
* feat: forward signal to old signal handler
If a signal is received that does not originate from YARA (which is
quite possible in multi-threaded applications using YARA), it is
currently ignored. Especially for SIGSEGV, this may cause unexpected
behaviour in the caller, even in threads that do not use YARA.
Add logic to call the original signal handler to handle this case
more cleanly. It is not perfect - with only a process wide signal
handler, it's probably impossible to have a perfect solution -
but it's better than the current behaviour.
* fix: make exception test more verbose
* fix: only handle signals in memory block data
Some callbacks from YARA code are done from YR_TRYCATCH. Currently,
YARA assumes that SIGBUS / SIGSEGV from these callbacks are due to
the scanned memory range, but this is not necessarily true.
Add logic to identify the currently scanned memory block and only
catch the signal if the SIGBUS / SIGSEGV originated from it.
* chore: simplify signal handler forwarding
* feat: reduce caught signals to required for known OS
* fix: check nullity of first block in math module
This module retrieves the first block of memory to do some validation on
parameters. However, contrary to for example the hash module, the block
value is not checked. This can lead, if for example scanning a process
and having the process die during the scan, to a SIGSEGV/Null access
violation, as the first block call will return NULL.
This bug manifests in surprising ways. Since yara runs by default
evaluation of condition in a try catch block, this bug is actually
caught by the try catch block, and a surprising "ERROR_COULD_NOT_MAP_FILE"
error is then returned.
In addition, since the try catch bypasses the whole stack, this leads to
a memory leak that can add up as more instances of this bug is triggered.
* fix: check first block in magic module
For the magic module, this bug (see previous commit) is less reachable,
since if the SCAN_FLAGS_PROCESS_MEMORY is set, we do not reach this
code. However, with a custom block iterator and without this flag, the
block value could be NULL.
* doc: update doc example on how to retrieve first memory block
per POSIX semaphore, sem_init should be paired with sem_destroy (unnamed
semaphore), sem_close should be paired with sem_open (named one).
While glibc seems forgiving about this, this will cause hangs or
segfaults with musl (for example, #1887).
Victor M. Alvarez