For instance, `x{n, m}`, `x{n ,m}` and `x{n , m}` are all equivalent to `x{n,m}`. This is done for compatibility with YARA-X, but I've seen real-life regular expressions in that contain such spaces, for example:
```
/C:\\Users\\qqq\\.{30, 50}\.pdb/
```
Several bugs were present in the parsing and generation of a string to
represent an array type, notably when the "lo_bound" value is set.
- The "read_blob_signed" was buggy for values outside the
[-2**6, 2**6-1], the sign edition did not use the right bitmask,
and the type used was unsigned.
- The display when lo_bound != 0 was buggy:
- size=5, lobound=0 => should be `5`, this was ok
- size=5, lobound=1 => should be `1...5`, this was buggy and displayed
`1...6`. The "range format" is inclusive.
- 0 should still be displayed if size is 0. Only when size is unset
should it be left out. So an array declared as `[5,0,3]` should be
displayed the same way, and not as `[5,,3]`.
* fix signature and magic strings in dex module
Several bytestring values in the dex module were not set properly,
and were cut short due to the presence of a nul byte.
This happened on:
- all the dex.DEX_FILE_MAGIC_* constants, which were cut short by one
byte (the last one is the nul byte).
- the magic and signature field in the "header" object of the module.
For all of those, the size is fixed and known, so use the right length
and do not cut it short if a nul byte is present.
* fix declared fields in the dex module
There was some mismatch between the declared fields and the ones filled
by the module:
- `dex.field[*].static` and `dex.field[*].instance` were defined, but
not declared, making their use impossible. They are now properly
declared.
- several fields in `dex.method[*].code_item` were declared but never
defined: `padding`, `tries` and `handlers`. Those are removed since
they couldn't have been used.
The `pe_parse_delayed_imports` function didn't take into account that `pe_rva_to_offset` could return -1 when the RVA can't be translated into a file offset.
Until now, these two fields were populated only when the offset and offset + length were within the file boundaries. This meant that the fields were undefined for some files that were truncated. For instance, 3323480fc4e35f73a8e93c4a0658c818d7e517df7ba15ba4cbb5cd2b6058cb0f.
The `matches` operator was returning false if the left operand was an empty strings. So, `"" matches <some regexp>` was returning false regardless of the regexp used. However, some regexp *do* match the empty strings, like for example `/foo|/`.
File e64e806c6df76f245c9b1288f59198bc9478329cfdd2d7ddcb9490a57d0b3cf4 was not properly parsed because `NumberOfRvaAndSizes` overflowed and became negative.
Some PE files have a NumberOfRvaAndSizes equal to zero. Even if these files are not accepted by the Windows loader, they can be parsed, and useful information can be extracted from them. By relaxing the sanity check in pe_get_directory_entry we are no able to parse such files (like 0aa527fd26c8e7fa7f3476b6d7eb8f22c071ff4c0fd3ed3797fa90ceb80f24ce).
.NET resources can be found outside the current file. Before this change those resources were being ignored, but now they are included in the `resources` array. The `name` and `length` fields are kept undefined though. So, external resources have only the `name` field set.
This function was checking the first two bytes at the entrypoint of .NET files, if the bytes were not `FF 25` the file was not considered a .NET file. This check was overly restrictive, as some files like `8fa7fe73a65296e9ca8301734a0deaa298cda0b7a65f6b7d2ef6d1c8bbb8cd7a` don't have such files at the entrypoint.
This function was returning a wrong RVA for files like `67508ec9cbd9f82c6aea8a93ca27fd3684d368664470397755f8d70934d481fb`, when a section's virtual size is smaller than its physical size.
Failing to parse some resource directory does not mean that we can't keep parsing other directories. For instance, with file `0a65e22cdff348be11739122906355af728f6247fb849f98d69a6f6d51103071` YARA was aborting the parsing too early and extracted a single resource, while the file has more resources that could be parsed correctly.
YARA was making the assumption that the 12 bytes following the `DanS` header contained 3 copies of the 32-bits XOR key, but these bytes are actually padding. As these bytes were usually filled with zeroes before applying the XOR key, they appear to contain the key itself after the XOR operation. However, files like `043066108b68b30fc2c475eae8edfafc080be7d451600eaa283d2c750bddbceb` don't contain zeroes in those bytes, and therefore were not satisfying YARA's assumption.
YARA-X was right: https://github.com/VirusTotal/yara-x/blob/6ada059b9e4b328e2341b42304765c5614fc89af/lib/src/modules/pe/parser.rs#L687-L693
Upon fixing the latest issue regarding parsing CMS, we figured out we
are not properly extracting all certificates from Microsoft
countersignatures. This should now properly expose them.
Also fixes one memory leak for countersignature parsing.
As discussed in https://github.com/VirusTotal/yara-python/issues/247, the example using `StringIO` worked fine in Python 2, but ti doesn't work in Python 3. Now that Python 2 is mostly dead, update the example.
This is redundant and already handled by TS_VERIFY_CTX_new().
Additionally it causes build failures with LibreSSL >= 3.8.0 which
removed this function since it serves no purpose with opaque
TS_VERIFY_CTX where it will zero out an already zero'd out ctx or cause
a leak.
See: https://github.com/libressl/portable/issues/1006
This is backward compatible change that affects the `magic` field. After this change the value in the `magic` field looks exactly as it looks in the file regardless of the endianness of the current platform, if the file starts with `CA FE BA BE` the value in magic is `0xCAFEBABE`, not `0xBEBAFECA` as it used to be in little-endian architectures.
Regexps with quantifiers `{0}` and `{0,0}` were considered syntactically invalid. These quantifiers are not actually useful, but they are now allowed for compatibility with other regexp engines.
Closes#2037.
Until now, `pe.is_signed` was undefined for every PE file that was not signed, making impossible to use it in negated conditions, like `pe.is_pe and not pe.is_signed`. With this change, `pe.is_signed` is false for every PE file that is not signed.
Closes#2040
Victor M. Alvarez