YR_MAJOR_VERSION, YR_MINOR_VERSION and YR_MICRO_VERSION are fixed when the
caller is compiled, so a program linked against a shared libyara has no way
to tell which version it actually loaded. Distros ship libyara.so in its own
package precisely so it can be updated independently of its consumers, which
is exactly the case the macros cannot describe.
Add yr_get_version(), which reports the version of the library that is
actually loaded. Any argument may be NULL if the caller does not need that
component, and since the function touches no global state it can be called
before yr_initialize.
Closes#2136
---------
Co-authored-by: Victor M. Alvarez <[email protected]>
elf.machine can hold any value from the ELF header, but the module only
named 14 of them, so a rule targeting anything else had to hardcode the
number or drop the module and match the header bytes by hand.
Add the architectures from the issue: PARISC, SPARC32PLUS, S390, MCORE,
SH, SPARCV9, ARC_COMPACT, BPF and LOONGARCH.
0x27 is exposed under both EM_MCORE and EM_RCE. EM_RCE is the name it was
originally registered under and still what some toolchains emit, so a rule
written against either name works.
Closes#2149
Fixes#2194.
The regexp engines cap how much data they scan at `YR_RE_SCAN_LIMIT` (4096 bytes, defined in `libyara/include/yara/limits.h`), so a match that needs more than that simply isn't reported. @plusvic pointed at the constant in the issue thread, but the limit isn't mentioned anywhere in the docs, which was the reporter's actual complaint: rules with wide wildcards fail silently and there's nothing in the manual explaining why.
This adds a warning box to the "Regular expressions" section of writingrules.rst stating the default limit, the silent-miss behavior, and that it can be raised at compile time.
Checked against HEAD (fa372eb) before writing the note:
- built yara and ran `/foo.*bar/` over a file with `foo` and `bar` 4000 bytes apart (matches) and one with them 5000 bytes apart (no match, no error, exit 0)
- rebuilt with `CPPFLAGS='-DYR_RE_SCAN_LIMIT=8192'`: the 5000-byte gap now matches and a 9000-byte one doesn't, so the compile-time override wording is accurate
- sphinx-build runs with no new warnings and the box renders at the end of the regexp section
As discussed in https://github.com/VirusTotal/yara-python/issues/247, the example using `StringIO` worked fine in Python 2, but ti doesn't work in Python 3. Now that Python 2 is mostly dead, update the example.
* fix: check nullity of first block in math module
This module retrieves the first block of memory to do some validation on
parameters. However, contrary to for example the hash module, the block
value is not checked. This can lead, if for example scanning a process
and having the process die during the scan, to a SIGSEGV/Null access
violation, as the first block call will return NULL.
This bug manifests in surprising ways. Since yara runs by default
evaluation of condition in a try catch block, this bug is actually
caught by the try catch block, and a surprising "ERROR_COULD_NOT_MAP_FILE"
error is then returned.
In addition, since the try catch bypasses the whole stack, this leads to
a memory leak that can add up as more instances of this bug is triggered.
* fix: check first block in magic module
For the magic module, this bug (see previous commit) is less reachable,
since if the SCAN_FLAGS_PROCESS_MEMORY is set, we do not reach this
code. However, with a custom block iterator and without this flag, the
block value could be NULL.
* doc: update doc example on how to retrieve first memory block
* LNK module initial commit
* Update atom logic to properly score [A-Z]
The documentation says `Bytes [a-zA-Z] contribute 18 points each`, but it looks like in the code that only `[a-z]` is given 18 points, whereas `[A-Z]` is given 20 points. This commit will make sure these ranges have the proper scoring as expected.
* Add test for comparing atom quality ABCD == abcd
Add test entry which compares the atom quality of "ABCD" and "abcd" and asserts that they are equal.
* Update test-atoms.c
* Revert "Update atom logic to properly score [A-Z]"
This reverts commit bf7130ff0b.
* Revert "Add test for comparing atom quality ABCD == abcd"
This reverts commit 181dbf2326.
* Revert "Update test-atoms.c"
This reverts commit 81239034b3.
* Test
* Revert "Test"
This reverts commit 1c53212b63.
* Add objects for LinkFlags parsing
Also fixed `DWORD` compiling error, and replaced with `uint32_t` types.
* Add file attribute flags
* Start hotkey parsing code
* Add the rest of the hotkey values
* Add initial test case for LNK module
Using the sample LNK from the MS standard for LNK files for testing, and putting in an initial test:
https://winprotocoldoc.blob.core.windows.net/productionwindowsarchives/MS-SHLLINK/%5bMS-SHLLINK%5d.pdf
* Add initial LNK tests
* Add has_hotkey variable
* Formatting
* Fix timestamps set
Previous version always seemed to be an hour (3600 seconds) off what it should have been. This commit adds a test to make sure it's getting the right value.
I don't know why its always an hour off, but this should fix it!
* Refactor hotkey char select into its own function
* Add a couple of comments
* Add min size for parsing LNK
All the mandatory LNK header bytes add up to 76 bytes, and as such we won't parse an LNK file unless it is at least 76 bytes in length
* Add LinkTargetIDList parsing
* Begin working on code for LinkInfo parsing
* Refactor code into header + util files
Moving a lot of the definitions over to a separate header file to keep the main code clean.
Also following the PE module's structure of having a separate `lnk_utils` file to deal with some of the convenient functions it provides.
* Starting VolumeID parsing
* Adding free statement
* Add some VolumeID tests and fix free error
* Finish parsing VolumeID
* Parse LocalBasePath
* Refactor to prevent unsafe memory copying
Start taking into account the remaining size in the block being scanned, to make sure we don't break anything when dereferencing pointers.
* Parse CommonPathSuffix
Also create empty function for when we need to parse CommonNetworkRelativeLink
* Parse LocalBasePathUnicode and CommonPathSuffixUnicode
NOTE: these will need some testing before this will be production ready, as I'm making a lot of assumptions here.
* Return size of LinkInfo section
* Add StringData parsing
* Add initial code for parsing ExtraData section
Only parsing `TrackerDataBlock` at the moment, but setting up values in advance of setting up the other structures/parsers
* Add the remaining ExtraData structures
In some cases I don't make a structure where the parsing function will just handle the data (e.g. where there is variable data).
I will likely also ignore PropertyStore for now as that has separate documentation to parse it.
* More specific parsing of Droid GUIDs
* Move code out of VolumeID parsing indent
The rest of the LinkInfo section wasn't parsing unless a VolumeID was present, which shouldn't be how it works.
* Remove incorrect CommonNetworkRelativeLinkOffset setting
Also add structure for CommonNetworkRelativeLink parsing
* Implement CommonNetworkRelativeLink parsing
And add a couple of test cases for it
* Formatting
* Make LinkInfo parsing memory safe
* Remove unnecessary constant
* Make StringData parsing memory safe
Also convert `block_data_size_remaining` to `int` from `size_t`. This is because we don't want to underflow the `size_t` type, and assume we have more data left than we actually do.
* Fix up block data size remaining calculations in ExtraData
* Add more constants
* Add in overlay variable
* Make current ExtraData function memory safe
* Setup switch/case for ExtraData parsing
* ConsoleDataBlock parsing
* Add ConsoleFedData parsing
* Convert tracker data to struct and add more fields
* Add extra fields to current ExtraData structures
* Refactor structures
* Rename sample LNKs
* Refactor into parse_volume_id
* Parse DarwinData and fixed up block struct entries
* Add is_malformed entry
Checks whether a LNK has failed parsing due to it being malformed.
* Add overlay test case
* Parse EnvironmentVariableDataBlock
* Parse IconEnvironmentDataBlock
* Add KnownFolderDataBlock parsing
* Add placeholder parsing for PropertyDataStoreBlock
* Parse ShimDataBlock
* Parse SpecialFolderDataBlock
* Remove id_list_ptr var
* Refactor into its own function to parse IDList
* Parse VistaAndAboveIDListDataBlock
* Remove printf
* Refactor item_id_list into structure
* Add LinkInfoFlags as external variables
* Only process relevant link info if flags set
* Update StringData to also accept ASCII
I assumed it would only be ASCII, but the `IsUnicode` flag can help indicate what type of string it is.
* Only parse ExtraData if there is data left
* Retype to strings
* Fix environment var data parsing
I thought wchar_t was 16 bits, but turns out it can also be 32 bits... I'll need to change up other routines to take this into account as well
* Move inside if statement to prevent setting
* Fix icon env unicode issue
* Convert remaining wchar_t struct entries
* Add ExtraData test cases
* Add ExtraData tests (2)
* Make sure timestamps are UTC
* Start LNK module documentation
* Add ShowCommand constants
* Finish documenting LNK headers
* Change test to flag comparison
* Correct structure names
* Update test-lnk.c
* Document LinkInfo section
* Document StringData
* Add more constants
* Document ConsoleData
* Update lnk.rst
* Finish documenting LNK module
* Some extra comments
* Update lnk.rst
* Apply clang formatting
* Removed test for now
* Add more flags and renamed console_fe_data
* Formatting
* Update comment on PropertyStoreData
* Change variable names due to potential clashes
Changed the variables used in header files:
- Reserved1
- Reserved2
- Unused1
- Unused2
The CI/CD seems to fail with these values as a default, so I'm hoping changing them fixes things
* Change packing to address misaligned pointer error
* Add LNK module to VS Projects
* Changed `block_data_size_remaining` to `unsigned int`
Make variable `unsigned int` instead of just `int` to prevent the value being negative.
* Begin removing static of strings
Starting with the ones I can test, by dynamically allocating memory after doing the check, I'm removing the possibility of there being a buffer overflow on this data.
* Dynamically allocate `volume_id_data`
* Remove more statically typed char arrays
* Dynamically allocate unicode strings
Previously were statically set in length, now dynamically allocated
* Add clearer comment
* Add todo
* Add `yr_` prefix to necessary functions
Due to #1762, needed to add `yr_` as a prefix to functions like `set_string`
* Convert 'unsigned int' to `uint32_t`
* Make constants have consistent casing
* Add extra checks on block size remaining
I found a case where a function returned a section size that was greater than the remaining block size, which caused a crash. This patch checks each value returned to make sure the next stage doesn't fail to parse.
* Fix docs for icon_location
Add `lnk.` prefix to `icon_location` variable
* Add LNK module back in
* Correct LNK module source path
* Add lnk_utils to modules list
And update other libyara auto changes
* Add LNK module tests
* Fix up LNK test in Makefile
* Fix typo in test LNK
* Attempt to fix bazel build
* Testing to see why Bazel build fails
* Revert removing tests
It seems even one test gives the error `There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.`
* Include LNK module in BUILD.bazel
* Add missing deps line
* Add lnk.h to Makefile
* Add lnk.h to yara.bzl
* Add lnk_utils.h to yara.bzl
* Revert automatic changes
Change some C/header files that got automatically changed while compiling YARA to the same that they are on the main branch
* Include lnk_utils in Makefile
* Convert some console data variables to signed and update docs
Some variables in the console data section are actually signed variables. So I've converted them to `int16_t` instead of `uint16_t`, added some test cases to make sure these values are properly parsed, and added some examples in the docs.
* Fix typo for window_size_y
* Add flags for LNK module
* Add extra time tests
Add tests for access time and write time
* Temporary update for debugging
Attempting to update this temporarily in order to test big endian issues on this branch - revert before merging
* Revert "Temporary update for debugging"
This reverts commit 2120c7e2e5.
* Temporary change for debugging
* Change order
* Temporarily disable most tests
* Initial big endian test fix
* LNK header endian fix
* Update test-lnk.c
* Fix up time conversion function calls
* Re-enable item ID tests
* Focus on item_id_list_size test
* Fix endianness of parse_link_target_id_list
* Fix endianness of parse_id_list
* Fix endianness of link_info
* Finish applying endianness fixes
Enabled all tests now
* Roll back to test console data and string data
* Add missing endianness conversion
* Re-enable all tests
* Revert back to original state
* LNK module initial commit
* Update atom logic to properly score [A-Z]
The documentation says `Bytes [a-zA-Z] contribute 18 points each`, but it looks like in the code that only `[a-z]` is given 18 points, whereas `[A-Z]` is given 20 points. This commit will make sure these ranges have the proper scoring as expected.
* Add test for comparing atom quality ABCD == abcd
Add test entry which compares the atom quality of "ABCD" and "abcd" and asserts that they are equal.
* Update test-atoms.c
* Revert "Update atom logic to properly score [A-Z]"
This reverts commit bf7130ff0b.
* Revert "Add test for comparing atom quality ABCD == abcd"
This reverts commit 181dbf2326.
* Revert "Update test-atoms.c"
This reverts commit 81239034b3.
* Test
* Revert "Test"
This reverts commit 1c53212b63.
* Add objects for LinkFlags parsing
Also fixed `DWORD` compiling error, and replaced with `uint32_t` types.
* Add file attribute flags
* Start hotkey parsing code
* Add the rest of the hotkey values
* Add initial test case for LNK module
Using the sample LNK from the MS standard for LNK files for testing, and putting in an initial test:
https://winprotocoldoc.blob.core.windows.net/productionwindowsarchives/MS-SHLLINK/%5bMS-SHLLINK%5d.pdf
* Add initial LNK tests
* Add has_hotkey variable
* Formatting
* Fix timestamps set
Previous version always seemed to be an hour (3600 seconds) off what it should have been. This commit adds a test to make sure it's getting the right value.
I don't know why its always an hour off, but this should fix it!
* Refactor hotkey char select into its own function
* Add a couple of comments
* Add min size for parsing LNK
All the mandatory LNK header bytes add up to 76 bytes, and as such we won't parse an LNK file unless it is at least 76 bytes in length
* Add LinkTargetIDList parsing
* Begin working on code for LinkInfo parsing
* Refactor code into header + util files
Moving a lot of the definitions over to a separate header file to keep the main code clean.
Also following the PE module's structure of having a separate `lnk_utils` file to deal with some of the convenient functions it provides.
* Starting VolumeID parsing
* Adding free statement
* Add some VolumeID tests and fix free error
* Finish parsing VolumeID
* Parse LocalBasePath
* Refactor to prevent unsafe memory copying
Start taking into account the remaining size in the block being scanned, to make sure we don't break anything when dereferencing pointers.
* Parse CommonPathSuffix
Also create empty function for when we need to parse CommonNetworkRelativeLink
* Parse LocalBasePathUnicode and CommonPathSuffixUnicode
NOTE: these will need some testing before this will be production ready, as I'm making a lot of assumptions here.
* Return size of LinkInfo section
* Add StringData parsing
* Add initial code for parsing ExtraData section
Only parsing `TrackerDataBlock` at the moment, but setting up values in advance of setting up the other structures/parsers
* Add the remaining ExtraData structures
In some cases I don't make a structure where the parsing function will just handle the data (e.g. where there is variable data).
I will likely also ignore PropertyStore for now as that has separate documentation to parse it.
* More specific parsing of Droid GUIDs
* Move code out of VolumeID parsing indent
The rest of the LinkInfo section wasn't parsing unless a VolumeID was present, which shouldn't be how it works.
* Remove incorrect CommonNetworkRelativeLinkOffset setting
Also add structure for CommonNetworkRelativeLink parsing
* Implement CommonNetworkRelativeLink parsing
And add a couple of test cases for it
* Formatting
* Make LinkInfo parsing memory safe
* Remove unnecessary constant
* Make StringData parsing memory safe
Also convert `block_data_size_remaining` to `int` from `size_t`. This is because we don't want to underflow the `size_t` type, and assume we have more data left than we actually do.
* Fix up block data size remaining calculations in ExtraData
* Add more constants
* Add in overlay variable
* Make current ExtraData function memory safe
* Setup switch/case for ExtraData parsing
* ConsoleDataBlock parsing
* Add ConsoleFedData parsing
* Convert tracker data to struct and add more fields
* Add extra fields to current ExtraData structures
* Refactor structures
* Rename sample LNKs
* Refactor into parse_volume_id
* Parse DarwinData and fixed up block struct entries
* Add is_malformed entry
Checks whether a LNK has failed parsing due to it being malformed.
* Add overlay test case
* Parse EnvironmentVariableDataBlock
* Parse IconEnvironmentDataBlock
* Add KnownFolderDataBlock parsing
* Add placeholder parsing for PropertyDataStoreBlock
* Parse ShimDataBlock
* Parse SpecialFolderDataBlock
* Remove id_list_ptr var
* Refactor into its own function to parse IDList
* Parse VistaAndAboveIDListDataBlock
* Remove printf
* Refactor item_id_list into structure
* Add LinkInfoFlags as external variables
* Only process relevant link info if flags set
* Update StringData to also accept ASCII
I assumed it would only be ASCII, but the `IsUnicode` flag can help indicate what type of string it is.
* Only parse ExtraData if there is data left
* Retype to strings
* Fix environment var data parsing
I thought wchar_t was 16 bits, but turns out it can also be 32 bits... I'll need to change up other routines to take this into account as well
* Move inside if statement to prevent setting
* Fix icon env unicode issue
* Convert remaining wchar_t struct entries
* Add ExtraData test cases
* Add ExtraData tests (2)
* Make sure timestamps are UTC
* Start LNK module documentation
* Add ShowCommand constants
* Finish documenting LNK headers
* Change test to flag comparison
* Correct structure names
* Update test-lnk.c
* Document LinkInfo section
* Document StringData
* Add more constants
* Document ConsoleData
* Update lnk.rst
* Finish documenting LNK module
* Some extra comments
* Update lnk.rst
* Apply clang formatting
* Removed test for now
* Add more flags and renamed console_fe_data
* Formatting
* Update comment on PropertyStoreData
* Change variable names due to potential clashes
Changed the variables used in header files:
- Reserved1
- Reserved2
- Unused1
- Unused2
The CI/CD seems to fail with these values as a default, so I'm hoping changing them fixes things
* Change packing to address misaligned pointer error
* Add LNK module to VS Projects
* Changed `block_data_size_remaining` to `unsigned int`
Make variable `unsigned int` instead of just `int` to prevent the value being negative.
* Begin removing static of strings
Starting with the ones I can test, by dynamically allocating memory after doing the check, I'm removing the possibility of there being a buffer overflow on this data.
* Dynamically allocate `volume_id_data`
* Remove more statically typed char arrays
* Dynamically allocate unicode strings
Previously were statically set in length, now dynamically allocated
* Add clearer comment
* Add todo
* Add `yr_` prefix to necessary functions
Due to #1762, needed to add `yr_` as a prefix to functions like `set_string`
* Convert 'unsigned int' to `uint32_t`
* Make constants have consistent casing
* Add extra checks on block size remaining
I found a case where a function returned a section size that was greater than the remaining block size, which caused a crash. This patch checks each value returned to make sure the next stage doesn't fail to parse.
* Fix docs for icon_location
Add `lnk.` prefix to `icon_location` variable
* Add LNK module back in
* Correct LNK module source path
* Add lnk_utils to modules list
And update other libyara auto changes
* Add LNK module tests
* Fix up LNK test in Makefile
* Fix typo in test LNK
* Attempt to fix bazel build
* Testing to see why Bazel build fails
* Revert removing tests
It seems even one test gives the error `There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.`
* Include LNK module in BUILD.bazel
* Add missing deps line
* Add lnk.h to Makefile
* Add lnk.h to yara.bzl
* Add lnk_utils.h to yara.bzl
* Revert automatic changes
Change some C/header files that got automatically changed while compiling YARA to the same that they are on the main branch
* Include lnk_utils in Makefile
* Convert some console data variables to signed and update docs
Some variables in the console data section are actually signed variables. So I've converted them to `int16_t` instead of `uint16_t`, added some test cases to make sure these values are properly parsed, and added some examples in the docs.
* Fix typo for window_size_y
* Add flags for LNK module
* Add extra time tests
Add tests for access time and write time
- Add CLI flag that will disable the console module from printing:
Short: -q
Log: --disable-console-logs
- Update documentation with new CLI flags
Closes#1912
* Add support for version_data in rich header.
Pull out just the version data (every other dword) into it's own string. This is
quite useful if you want to hash just the version fields of the rich header,
which makes for a more robust signature because it is now independent of the
count field.
Hashing this value is a useful signature. ;)
* Add docs for version_data.
* fix error handling in string.to_int()
Error handling in the to_int method of the new string module was not
properly done.
- errno was not reset prior to the call to strtoll, so YR_UNDEFINED
could be returned if a prior call failed, but not the current one.
- overflow/underflow was not properly checked, as it returns LLONG_MIN
or LLONG_MAX and not 0.
Instead, do what is recommended in the man page for strtol:
- set errno to 0 prior to the call
- check errno after the call
* Flesh out invalid cases for string.to_int
Ensure the behavior of string.to_int is the least surprising and
consistent across all OSes:
- return an error when the string is not entirely parsed (so "10p"
fails to parse).
- return an error when the string has no digit whatsoever (avoiding
parsing "foo" as 0).
- remove the evaluation differences on different libc implementations.
Fixes#1843
* validate value of base in string.to_int(string, base)
There seems to be implementation differences on invalid base values,
especially with cygwin. So validate that the base has a valid value
first.
* Fix `pe_rva_to_offset`
* Fix: checking if RVA is inside section
* Fix: real pointer to raw data is aligned down to sector size
* Add pe.import_rva() functions.
Add pe.import_rva("foo.dll", "func1") which returns the RVA of the imported
function. Also add pe.import_rva("foo.dll", 1) which does the same but the
import is done by ordinal.
* Implement delayed import RVA and add docs.
* Add math.length()
Add a math.length() which will return the length of the sequence of bytes,
including any NULL bytes.
Fixes#1778.
* Add string module.
Move the math.to_int() functions and math.length() over to the new string
module. I decided to move the to_int() because it seems logical to convert from
a string to an integer using a string module rather than the math module.
You still use math.to_string() to convert an integer to a string, and use
string.to_int() to convert a string to an integer.
Add tests and docs for string module. Move the appropriate tests from the math
module over to the new string tests.
While here, also add the console module to the bazel build as it was apparently
missing (this is untested).
* Remove unused include.
* Add tests for https://github.com/VirusTotal/yara/pull/1561/
* Fix copyright year in string module.
Co-authored-by: Peter Babka <[email protected]>
* Add OP_OF_FOUND_AT.
Add support for "any of them at 0" constructs to the language. This allows users
to avoid using long or chains like "$a at 0 or $b at 0" and also is a nicer way
to write "for any of them: ($ at 0)".
* Add short blurb to docs about new string set at offset.
* Implement text string sets.
Add support for text string sets into the grammar. They look like this:
for any s in ("a", "b"): (pe.imphash() == s)
This requires changing integer_set and integer_enumeration to just be set and
enumeration, and adding a new type (YR_ENUMERATION) that tracks the type of
enumeration (integer or otherwise) and the number of items in the enumeration.
The enumeration now checks that each type of the item in the enumeration is
consistent and will cause a compiler error if they are not. For example, this is
an error:
for any s in ("a", 0): (s)
Also, fix the build when using --with-debug-verbose option as it was missing the
assert.h include.
* Add docs and adjust constants layout.
