As discussed in https://github.com/VirusTotal/yara-python/issues/247, the example using `StringIO` worked fine in Python 2, but ti doesn't work in Python 3. Now that Python 2 is mostly dead, update the example.
This is redundant and already handled by TS_VERIFY_CTX_new().
Additionally it causes build failures with LibreSSL >= 3.8.0 which
removed this function since it serves no purpose with opaque
TS_VERIFY_CTX where it will zero out an already zero'd out ctx or cause
a leak.
See: https://github.com/libressl/portable/issues/1006
This is backward compatible change that affects the `magic` field. After this change the value in the `magic` field looks exactly as it looks in the file regardless of the endianness of the current platform, if the file starts with `CA FE BA BE` the value in magic is `0xCAFEBABE`, not `0xBEBAFECA` as it used to be in little-endian architectures.
Regexps with quantifiers `{0}` and `{0,0}` were considered syntactically invalid. These quantifiers are not actually useful, but they are now allowed for compatibility with other regexp engines.
Closes#2037.
Until now, `pe.is_signed` was undefined for every PE file that was not signed, making impossible to use it in negated conditions, like `pe.is_pe and not pe.is_signed`. With this change, `pe.is_signed` is false for every PE file that is not signed.
Closes#2040
The pointer returned from `libmagic` was passed directly to `yr_strdup` (a wrapper of `strdup`), but this function has undefined behavior when the argument is null.
Instead of downloading the latest master branch, use the code in the current branch. Until know, this workflow was always operating wit the latest commit in the master branch.
The process scanning on macos was broken since 4.2.0 in this commit:
605b2edf07
Logic was added to properly iterate on memory chunks if a memory region
was too big, but this completely broke the iteration. To retrieve a
memory region, the code basically does:
```
current_begin = 0;
for (;;) {
vm_address_t addr = current_begin;
mach_vm_region(..., &addr, &size, ...);
size_t chunk_size = size - (current_begin - addr);
...
}
```
It assumes that `addr` is smaller than `current_begin`, which is never
the case outside of chunking. This leads to an iteration on completely
wrong regions, leading to read that either fail because the starting
addr or length is wrong, or that succeed, but do not cover the whole
region.
For example, with a process memory that looks like:
```
1065a6000-1065f2000
1065f2000-1065f6000
1065f6000-1065fa000
...
600000000000-600008000000
600008000000-600010000000
...
7fe537f00000-7fe538000000
7fe538000000-7fe538800000
```
The very first call would have `current_begin=0` and `addr=0x1065a6000`,
leading to underflow when computing `chunk_size`, and trying to iterate
on the region `0x00000000-0x40000000` (as the max memory chunk is 1GB
per default). this will fail to fetch, and will then iterate on
`0x40000000-0x80000000`, then `0x80000000-0xC0000000`, then
`0xC0000000-0x100000000`. We will finally reach the very first real
memory region, but fail to properly compute the region size with another
underflow, etc.
This commit fixes the issue by distinguishing the two cases:
- either `addr > current_begin` : `current_begin` is not in any memory
region, and we must fast forward.
- or `addr <= current_begin`: we can compute the current chunk by
diffing the two values.
This was not tested because the test_process_scan was not running on
macos: trying to scan the sh process always fails even when root, and
the test was thus skipped. I hacked the test locally to check if it
worked by replacing the sh process by something else that we can attach
to, and it showed the scanning fails. With this commit, the test does
pass.
I did not commit any modification to the test as I used a custom binary
for this and it needs to be run as root, but it would probably be a good
idea to adapt it.
This updates authenticode parser to the latest versions which contains
sereral fixes which accumulated over time.
* Several time_t variables replaced with int64_t
* RFC5652 countersignatures are now correctly parsed
* Memory corruption related fixes where authenticode-parser would crash otherwise
Some regexps like `/(abc|)/` where not matching the empty string even if they should. Matching an empty string is not very useful, as such regexp will match on every offset of every file, but this is fixed for correctness.
Also fixes another issue with the `\b` metacharacter in regexps, which was exposed after fixing the main issue.
Some modules functions (from hash and math) can access multiple memory
blocks, as long as those are contiguous. For example, when hashing
between offsets 0..1000, blocks that cover this range will be fetched.
This logic however is a bit buggy on the boundary: if the fetched range
ends exactly at the end of a block, the iteration keeps going instead of
ending. This can result in two situations:
- The next block is contiguous: it is then fetched.
- if the fetch fails, the function fails (and return undefined)
- if the fetch succeeds, we read 0 bytes from it and
end the iteration, so a useless fetch was done.
- The next block is not contiguous: the function fails (and return
undefined).
This is fixed by modifying the condition to detect if there are still
more bytes to fetch.
While porting the `dotnet` module to YARA-X I discovered the the implementation of `read_blob_signed` had a subtle bug. As `first_byte` was declared as `uint8_t` instead of `int8_t`, sign extension when casting the value to a `int32_t` was not done, leading to incorrect values.
PR #1927 didn't take into account that as a side-effect of skipping the evaluation of global rules, the corresponding namespace was not flagged as unsatisfied (i.e: a global rule that evaluates to false prevents all other rules in the namespace from evaluating to true). The issue was not caught by any test case because the test case for global rules were using rules that didn't required any strings to match, and therefore were unaffected by the changes introduced in #1927.
This change makes sure that any global rule that is skipped is considered false, and therefore its namespace is flagged as unsatisfied.
Closes#2022.
This reverts commit 2d103d3ce3. The previous fix introduces shit-reduce and reduce-reduce conflicts in the grammar. The grammar.y file was not correctly processed by `bison`.
When scanning multiple memory blocks, we should stop as soon as a valid
elf is found. This was broken as the "break" applies to the switch, and
not the outer loop. Fix this by reworking the code to remove the switch.
This bug had two repercutions:
- instead of parsing the first memory block that contains an elf, it
returned the last one, or actually even worse, a mix of all elfs
depending on which values are set again or not. This however should
not manifest in normal situations, notably during process scanning, as
I suppose only one memory region would contain an elf executable.
- the module caused the fetch of all memory blocks instead of stopping
as soon as one was found, which is worse performance wise.
* fixed validation error in grammar.y
rule_modifiers 'private' and 'global' can no longer be used more than once in a rule definition. They can be used exclusively or together in any order.
* Update grammar.y
* feat: prevent race conditions on signal handler
If multiple threads were running in parallel (e.g. each using a
YR_SCANNER), race conditions could cause the original signal handler
to not be restored correctly.
Add a mutex that controls access to the (process wide) signal handler
to ensure that the YARA signal handler is only installed once and
only removed when it is no longer needed.
* feat: forward signal to old signal handler
If a signal is received that does not originate from YARA (which is
quite possible in multi-threaded applications using YARA), it is
currently ignored. Especially for SIGSEGV, this may cause unexpected
behaviour in the caller, even in threads that do not use YARA.
Add logic to call the original signal handler to handle this case
more cleanly. It is not perfect - with only a process wide signal
handler, it's probably impossible to have a perfect solution -
but it's better than the current behaviour.
* fix: make exception test more verbose
* fix: only handle signals in memory block data
Some callbacks from YARA code are done from YR_TRYCATCH. Currently,
YARA assumes that SIGBUS / SIGSEGV from these callbacks are due to
the scanned memory range, but this is not necessarily true.
Add logic to identify the currently scanned memory block and only
catch the signal if the SIGBUS / SIGSEGV originated from it.
* chore: simplify signal handler forwarding
* feat: reduce caught signals to required for known OS
* fix: check nullity of first block in math module
This module retrieves the first block of memory to do some validation on
parameters. However, contrary to for example the hash module, the block
value is not checked. This can lead, if for example scanning a process
and having the process die during the scan, to a SIGSEGV/Null access
violation, as the first block call will return NULL.
This bug manifests in surprising ways. Since yara runs by default
evaluation of condition in a try catch block, this bug is actually
caught by the try catch block, and a surprising "ERROR_COULD_NOT_MAP_FILE"
error is then returned.
In addition, since the try catch bypasses the whole stack, this leads to
a memory leak that can add up as more instances of this bug is triggered.
* fix: check first block in magic module
For the magic module, this bug (see previous commit) is less reachable,
since if the SCAN_FLAGS_PROCESS_MEMORY is set, we do not reach this
code. However, with a custom block iterator and without this flag, the
block value could be NULL.
* doc: update doc example on how to retrieve first memory block
per POSIX semaphore, sem_init should be paired with sem_destroy (unnamed
semaphore), sem_close should be paired with sem_open (named one).
While glibc seems forgiving about this, this will cause hangs or
segfaults with musl (for example, #1887).
The commit dfd6f7b5f brought large file detection in the autoconf file
to fix process scanning on linux in x86 architecture. This is however a
bit flaky, because:
- process scanning won't work on systems that do not have LFS, which
does not make sense (although I don't really know how autoconf detects
it or not).
- the fix does not work if the compilation is done without the use of
autoconf, and `_FILE_OFFSET_BITS` is not defined. This is for example
the case for yara-rust, where the process scanning does not work on
linux 32bits for this reason.
This breaks process scanning because without it, the pread function
on the mem file will fail if provided with an offset bigger than
INT32_MAX, which will always happen for processes.
A better fix is to use the pread64 syscall, which always takes a 64-bits
value for the offset. This is the actual linux syscall, so it is
completely safe.
See for example the exact same fix on chromium:
https://groups.google.com/a/chromium.org/g/crashpad-dev/c/IJLo8M2AQs0/m/Edyyau2LAwAJ
When the file imported DLLs that contained a dot (.) in its name, like `glib-2.0.dll`, the file extension was not properly removed, causing the the imphash to be wrong.
With this change, when the PE imports are parsed, any imported DLL with a name longer than MAX_IMPORT_DLL_NAME_LENGTH (256) is ignored. The limit is large enough to accommodate real life DLL names, but prevents corrupted files from being parsed.
For instance, file 601ef2a4cdbb757050f33f8ebcb0ef642025a89f8ae5eca988722cc073266001 was causing the `imphash` function to take forever due to very large DLL names that were completely invalid, but the parser was accepting them as valid. Those DLL names were filled with `0xff` bytes, so the another alternative was limiting DLL names to be ASCII only, but that's a more restrictive solution that can reject imports where the DLL name contains some non-ascii byte.
Closes#2001.
Commit 1a85738 fixed one bug but introduced a new one. The
`pe.is_signed` field is now properly set (true if any signature is
valid), but the `signatures[i].verified` is now incorrectly set: instead
of being set to whether this signature is verified, it is set to
true if any previous signature is verified.
Unfortunately, there is no test file that is triggering this behavior.
This would require having a signature that is invalid appearing after a
signature that is valid.
This fixes a compiler crash while parsing the invalid YARA rules described in #1992. The root of the issue was the incorrect initialization of a `YR_EXPRESSION` structure where the type was set to `EXPRESSION_TYPE_STRING` but the value was initialized as an integer and set to `YR_UNDEFINED`. As the value is an union type, this undefined integer was later interpreted as an arena reference via the `sized_string_ref` union field.
Closes#1992.
As is done for PE, the entrypoint computed during process memory
scanning should be added to the base address of the block, so that the
returned value is the process address of the entrypoint.
* fix: check mmap result
Check if mmap failed (return value MAP_FAILED) and, if yes, exit.
* feat: fallback to pread() if file mmap failed
* fix: initialize zero buffer for uninitialized mapping
* fix bug in entrypoint computation on process mem scan
When using the PROCESS_MEMORY flag and the macho module, the LC_MAIN
load command was badly handled, the computation of the end of a segment
was invalid.
* add base address to entrypoint when scanning proc memory
As is done for other file analysis modules, the base address of the
scanned region must be added to the entrypoint when scanning with the
PROCESS_MEMORY flag set.
* do not set invalid entrypoint value if command is too small
If the unixthread command is too small, the entrypoint value is not
retrieved. The function still continues executing however, and
considers 0 to be the entrypoint value. This is invalid, and the
entrypoint should stay undefined in those cases.
Victor M. Alvarez