* LNK module initial commit
* Update atom logic to properly score [A-Z]
The documentation says `Bytes [a-zA-Z] contribute 18 points each`, but it looks like in the code that only `[a-z]` is given 18 points, whereas `[A-Z]` is given 20 points. This commit will make sure these ranges have the proper scoring as expected.
* Add test for comparing atom quality ABCD == abcd
Add test entry which compares the atom quality of "ABCD" and "abcd" and asserts that they are equal.
* Update test-atoms.c
* Revert "Update atom logic to properly score [A-Z]"
This reverts commit bf7130ff0b.
* Revert "Add test for comparing atom quality ABCD == abcd"
This reverts commit 181dbf2326.
* Revert "Update test-atoms.c"
This reverts commit 81239034b3.
* Test
* Revert "Test"
This reverts commit 1c53212b63.
* Add objects for LinkFlags parsing
Also fixed `DWORD` compiling error, and replaced with `uint32_t` types.
* Add file attribute flags
* Start hotkey parsing code
* Add the rest of the hotkey values
* Add initial test case for LNK module
Using the sample LNK from the MS standard for LNK files for testing, and putting in an initial test:
https://winprotocoldoc.blob.core.windows.net/productionwindowsarchives/MS-SHLLINK/%5bMS-SHLLINK%5d.pdf
* Add initial LNK tests
* Add has_hotkey variable
* Formatting
* Fix timestamps set
Previous version always seemed to be an hour (3600 seconds) off what it should have been. This commit adds a test to make sure it's getting the right value.
I don't know why its always an hour off, but this should fix it!
* Refactor hotkey char select into its own function
* Add a couple of comments
* Add min size for parsing LNK
All the mandatory LNK header bytes add up to 76 bytes, and as such we won't parse an LNK file unless it is at least 76 bytes in length
* Add LinkTargetIDList parsing
* Begin working on code for LinkInfo parsing
* Refactor code into header + util files
Moving a lot of the definitions over to a separate header file to keep the main code clean.
Also following the PE module's structure of having a separate `lnk_utils` file to deal with some of the convenient functions it provides.
* Starting VolumeID parsing
* Adding free statement
* Add some VolumeID tests and fix free error
* Finish parsing VolumeID
* Parse LocalBasePath
* Refactor to prevent unsafe memory copying
Start taking into account the remaining size in the block being scanned, to make sure we don't break anything when dereferencing pointers.
* Parse CommonPathSuffix
Also create empty function for when we need to parse CommonNetworkRelativeLink
* Parse LocalBasePathUnicode and CommonPathSuffixUnicode
NOTE: these will need some testing before this will be production ready, as I'm making a lot of assumptions here.
* Return size of LinkInfo section
* Add StringData parsing
* Add initial code for parsing ExtraData section
Only parsing `TrackerDataBlock` at the moment, but setting up values in advance of setting up the other structures/parsers
* Add the remaining ExtraData structures
In some cases I don't make a structure where the parsing function will just handle the data (e.g. where there is variable data).
I will likely also ignore PropertyStore for now as that has separate documentation to parse it.
* More specific parsing of Droid GUIDs
* Move code out of VolumeID parsing indent
The rest of the LinkInfo section wasn't parsing unless a VolumeID was present, which shouldn't be how it works.
* Remove incorrect CommonNetworkRelativeLinkOffset setting
Also add structure for CommonNetworkRelativeLink parsing
* Implement CommonNetworkRelativeLink parsing
And add a couple of test cases for it
* Formatting
* Make LinkInfo parsing memory safe
* Remove unnecessary constant
* Make StringData parsing memory safe
Also convert `block_data_size_remaining` to `int` from `size_t`. This is because we don't want to underflow the `size_t` type, and assume we have more data left than we actually do.
* Fix up block data size remaining calculations in ExtraData
* Add more constants
* Add in overlay variable
* Make current ExtraData function memory safe
* Setup switch/case for ExtraData parsing
* ConsoleDataBlock parsing
* Add ConsoleFedData parsing
* Convert tracker data to struct and add more fields
* Add extra fields to current ExtraData structures
* Refactor structures
* Rename sample LNKs
* Refactor into parse_volume_id
* Parse DarwinData and fixed up block struct entries
* Add is_malformed entry
Checks whether a LNK has failed parsing due to it being malformed.
* Add overlay test case
* Parse EnvironmentVariableDataBlock
* Parse IconEnvironmentDataBlock
* Add KnownFolderDataBlock parsing
* Add placeholder parsing for PropertyDataStoreBlock
* Parse ShimDataBlock
* Parse SpecialFolderDataBlock
* Remove id_list_ptr var
* Refactor into its own function to parse IDList
* Parse VistaAndAboveIDListDataBlock
* Remove printf
* Refactor item_id_list into structure
* Add LinkInfoFlags as external variables
* Only process relevant link info if flags set
* Update StringData to also accept ASCII
I assumed it would only be ASCII, but the `IsUnicode` flag can help indicate what type of string it is.
* Only parse ExtraData if there is data left
* Retype to strings
* Fix environment var data parsing
I thought wchar_t was 16 bits, but turns out it can also be 32 bits... I'll need to change up other routines to take this into account as well
* Move inside if statement to prevent setting
* Fix icon env unicode issue
* Convert remaining wchar_t struct entries
* Add ExtraData test cases
* Add ExtraData tests (2)
* Make sure timestamps are UTC
* Start LNK module documentation
* Add ShowCommand constants
* Finish documenting LNK headers
* Change test to flag comparison
* Correct structure names
* Update test-lnk.c
* Document LinkInfo section
* Document StringData
* Add more constants
* Document ConsoleData
* Update lnk.rst
* Finish documenting LNK module
* Some extra comments
* Update lnk.rst
* Apply clang formatting
* Removed test for now
* Add more flags and renamed console_fe_data
* Formatting
* Update comment on PropertyStoreData
* Change variable names due to potential clashes
Changed the variables used in header files:
- Reserved1
- Reserved2
- Unused1
- Unused2
The CI/CD seems to fail with these values as a default, so I'm hoping changing them fixes things
* Change packing to address misaligned pointer error
* Add LNK module to VS Projects
* Changed `block_data_size_remaining` to `unsigned int`
Make variable `unsigned int` instead of just `int` to prevent the value being negative.
* Begin removing static of strings
Starting with the ones I can test, by dynamically allocating memory after doing the check, I'm removing the possibility of there being a buffer overflow on this data.
* Dynamically allocate `volume_id_data`
* Remove more statically typed char arrays
* Dynamically allocate unicode strings
Previously were statically set in length, now dynamically allocated
* Add clearer comment
* Add todo
* Add `yr_` prefix to necessary functions
Due to #1762, needed to add `yr_` as a prefix to functions like `set_string`
* Convert 'unsigned int' to `uint32_t`
* Make constants have consistent casing
* Add extra checks on block size remaining
I found a case where a function returned a section size that was greater than the remaining block size, which caused a crash. This patch checks each value returned to make sure the next stage doesn't fail to parse.
* Fix docs for icon_location
Add `lnk.` prefix to `icon_location` variable
* Add LNK module back in
* Correct LNK module source path
* Add lnk_utils to modules list
And update other libyara auto changes
* Add LNK module tests
* Fix up LNK test in Makefile
* Fix typo in test LNK
* Attempt to fix bazel build
* Testing to see why Bazel build fails
* Revert removing tests
It seems even one test gives the error `There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.`
* Include LNK module in BUILD.bazel
* Add missing deps line
* Add lnk.h to Makefile
* Add lnk.h to yara.bzl
* Add lnk_utils.h to yara.bzl
* Revert automatic changes
Change some C/header files that got automatically changed while compiling YARA to the same that they are on the main branch
* Include lnk_utils in Makefile
* Convert some console data variables to signed and update docs
Some variables in the console data section are actually signed variables. So I've converted them to `int16_t` instead of `uint16_t`, added some test cases to make sure these values are properly parsed, and added some examples in the docs.
* Fix typo for window_size_y
* Add flags for LNK module
* Add extra time tests
Add tests for access time and write time
* Temporary update for debugging
Attempting to update this temporarily in order to test big endian issues on this branch - revert before merging
* Revert "Temporary update for debugging"
This reverts commit 2120c7e2e5.
* Temporary change for debugging
* Change order
* Temporarily disable most tests
* Initial big endian test fix
* LNK header endian fix
* Update test-lnk.c
* Fix up time conversion function calls
* Re-enable item ID tests
* Focus on item_id_list_size test
* Fix endianness of parse_link_target_id_list
* Fix endianness of parse_id_list
* Fix endianness of link_info
* Finish applying endianness fixes
Enabled all tests now
* Roll back to test console data and string data
* Add missing endianness conversion
* Re-enable all tests
* Revert back to original state
* LNK module initial commit
* Update atom logic to properly score [A-Z]
The documentation says `Bytes [a-zA-Z] contribute 18 points each`, but it looks like in the code that only `[a-z]` is given 18 points, whereas `[A-Z]` is given 20 points. This commit will make sure these ranges have the proper scoring as expected.
* Add test for comparing atom quality ABCD == abcd
Add test entry which compares the atom quality of "ABCD" and "abcd" and asserts that they are equal.
* Update test-atoms.c
* Revert "Update atom logic to properly score [A-Z]"
This reverts commit bf7130ff0b.
* Revert "Add test for comparing atom quality ABCD == abcd"
This reverts commit 181dbf2326.
* Revert "Update test-atoms.c"
This reverts commit 81239034b3.
* Test
* Revert "Test"
This reverts commit 1c53212b63.
* Add objects for LinkFlags parsing
Also fixed `DWORD` compiling error, and replaced with `uint32_t` types.
* Add file attribute flags
* Start hotkey parsing code
* Add the rest of the hotkey values
* Add initial test case for LNK module
Using the sample LNK from the MS standard for LNK files for testing, and putting in an initial test:
https://winprotocoldoc.blob.core.windows.net/productionwindowsarchives/MS-SHLLINK/%5bMS-SHLLINK%5d.pdf
* Add initial LNK tests
* Add has_hotkey variable
* Formatting
* Fix timestamps set
Previous version always seemed to be an hour (3600 seconds) off what it should have been. This commit adds a test to make sure it's getting the right value.
I don't know why its always an hour off, but this should fix it!
* Refactor hotkey char select into its own function
* Add a couple of comments
* Add min size for parsing LNK
All the mandatory LNK header bytes add up to 76 bytes, and as such we won't parse an LNK file unless it is at least 76 bytes in length
* Add LinkTargetIDList parsing
* Begin working on code for LinkInfo parsing
* Refactor code into header + util files
Moving a lot of the definitions over to a separate header file to keep the main code clean.
Also following the PE module's structure of having a separate `lnk_utils` file to deal with some of the convenient functions it provides.
* Starting VolumeID parsing
* Adding free statement
* Add some VolumeID tests and fix free error
* Finish parsing VolumeID
* Parse LocalBasePath
* Refactor to prevent unsafe memory copying
Start taking into account the remaining size in the block being scanned, to make sure we don't break anything when dereferencing pointers.
* Parse CommonPathSuffix
Also create empty function for when we need to parse CommonNetworkRelativeLink
* Parse LocalBasePathUnicode and CommonPathSuffixUnicode
NOTE: these will need some testing before this will be production ready, as I'm making a lot of assumptions here.
* Return size of LinkInfo section
* Add StringData parsing
* Add initial code for parsing ExtraData section
Only parsing `TrackerDataBlock` at the moment, but setting up values in advance of setting up the other structures/parsers
* Add the remaining ExtraData structures
In some cases I don't make a structure where the parsing function will just handle the data (e.g. where there is variable data).
I will likely also ignore PropertyStore for now as that has separate documentation to parse it.
* More specific parsing of Droid GUIDs
* Move code out of VolumeID parsing indent
The rest of the LinkInfo section wasn't parsing unless a VolumeID was present, which shouldn't be how it works.
* Remove incorrect CommonNetworkRelativeLinkOffset setting
Also add structure for CommonNetworkRelativeLink parsing
* Implement CommonNetworkRelativeLink parsing
And add a couple of test cases for it
* Formatting
* Make LinkInfo parsing memory safe
* Remove unnecessary constant
* Make StringData parsing memory safe
Also convert `block_data_size_remaining` to `int` from `size_t`. This is because we don't want to underflow the `size_t` type, and assume we have more data left than we actually do.
* Fix up block data size remaining calculations in ExtraData
* Add more constants
* Add in overlay variable
* Make current ExtraData function memory safe
* Setup switch/case for ExtraData parsing
* ConsoleDataBlock parsing
* Add ConsoleFedData parsing
* Convert tracker data to struct and add more fields
* Add extra fields to current ExtraData structures
* Refactor structures
* Rename sample LNKs
* Refactor into parse_volume_id
* Parse DarwinData and fixed up block struct entries
* Add is_malformed entry
Checks whether a LNK has failed parsing due to it being malformed.
* Add overlay test case
* Parse EnvironmentVariableDataBlock
* Parse IconEnvironmentDataBlock
* Add KnownFolderDataBlock parsing
* Add placeholder parsing for PropertyDataStoreBlock
* Parse ShimDataBlock
* Parse SpecialFolderDataBlock
* Remove id_list_ptr var
* Refactor into its own function to parse IDList
* Parse VistaAndAboveIDListDataBlock
* Remove printf
* Refactor item_id_list into structure
* Add LinkInfoFlags as external variables
* Only process relevant link info if flags set
* Update StringData to also accept ASCII
I assumed it would only be ASCII, but the `IsUnicode` flag can help indicate what type of string it is.
* Only parse ExtraData if there is data left
* Retype to strings
* Fix environment var data parsing
I thought wchar_t was 16 bits, but turns out it can also be 32 bits... I'll need to change up other routines to take this into account as well
* Move inside if statement to prevent setting
* Fix icon env unicode issue
* Convert remaining wchar_t struct entries
* Add ExtraData test cases
* Add ExtraData tests (2)
* Make sure timestamps are UTC
* Start LNK module documentation
* Add ShowCommand constants
* Finish documenting LNK headers
* Change test to flag comparison
* Correct structure names
* Update test-lnk.c
* Document LinkInfo section
* Document StringData
* Add more constants
* Document ConsoleData
* Update lnk.rst
* Finish documenting LNK module
* Some extra comments
* Update lnk.rst
* Apply clang formatting
* Removed test for now
* Add more flags and renamed console_fe_data
* Formatting
* Update comment on PropertyStoreData
* Change variable names due to potential clashes
Changed the variables used in header files:
- Reserved1
- Reserved2
- Unused1
- Unused2
The CI/CD seems to fail with these values as a default, so I'm hoping changing them fixes things
* Change packing to address misaligned pointer error
* Add LNK module to VS Projects
* Changed `block_data_size_remaining` to `unsigned int`
Make variable `unsigned int` instead of just `int` to prevent the value being negative.
* Begin removing static of strings
Starting with the ones I can test, by dynamically allocating memory after doing the check, I'm removing the possibility of there being a buffer overflow on this data.
* Dynamically allocate `volume_id_data`
* Remove more statically typed char arrays
* Dynamically allocate unicode strings
Previously were statically set in length, now dynamically allocated
* Add clearer comment
* Add todo
* Add `yr_` prefix to necessary functions
Due to #1762, needed to add `yr_` as a prefix to functions like `set_string`
* Convert 'unsigned int' to `uint32_t`
* Make constants have consistent casing
* Add extra checks on block size remaining
I found a case where a function returned a section size that was greater than the remaining block size, which caused a crash. This patch checks each value returned to make sure the next stage doesn't fail to parse.
* Fix docs for icon_location
Add `lnk.` prefix to `icon_location` variable
* Add LNK module back in
* Correct LNK module source path
* Add lnk_utils to modules list
And update other libyara auto changes
* Add LNK module tests
* Fix up LNK test in Makefile
* Fix typo in test LNK
* Attempt to fix bazel build
* Testing to see why Bazel build fails
* Revert removing tests
It seems even one test gives the error `There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.`
* Include LNK module in BUILD.bazel
* Add missing deps line
* Add lnk.h to Makefile
* Add lnk.h to yara.bzl
* Add lnk_utils.h to yara.bzl
* Revert automatic changes
Change some C/header files that got automatically changed while compiling YARA to the same that they are on the main branch
* Include lnk_utils in Makefile
* Convert some console data variables to signed and update docs
Some variables in the console data section are actually signed variables. So I've converted them to `int16_t` instead of `uint16_t`, added some test cases to make sure these values are properly parsed, and added some examples in the docs.
* Fix typo for window_size_y
* Add flags for LNK module
* Add extra time tests
Add tests for access time and write time
* Import authenticode-parser into Yara. Rewrite PE signature processing using the authenticode-parser. Update bazel OpenSSL dependency to 1.1.1 from 1.1.0
* Forgot to include structs.c
* Add Avast to authors and myself to contributors
* Set forgotten algorithm_oid
* Fix incorrect indentation
* Fix race condition when initializing openssl objects
* Don't malloc ByteArray with length 0
* Auth. parser update - bug fixes, openssl3.0 support
* Update authenticode parser
Co-authored-by: Victor M. Alvarez <[email protected]>
The root cause of this issue was that the size of YR_AC_MATCH structure was 36 instead of 49 bytes when compiling for 32-bits with Visual Studio. This caused that rules compiled with a 64-bit version of YARA, or even a 32-bit version for Linux, were invalid when loaded with the 32-bits Windows version.
* Fix race conditions when using the same YR_RULES struct from different threads with PROFILING_ENABLED.
* Make time_cost signed as InterlockedAdd64 expects a signed integer.
Closes#918.
For compiling YARA in Windows as a DLL you must define YR_BUILDING_DLL. For using it that DLL in your project you must define YR_IMPORTING_DLL. This allows to use the same header files in both cases. If YR_BUILDING_DLL is defined, YR_API translates to __declspec(dllexport), if YR_IMPORTING_DLL is defined YR_API translates to __declspec(dllimport).
This commit speeds up the generation of the Aho-Corasick automaton and changes the type of each item in the automaton transition table from uint64_t to uin32_t. This also speeds-up scanning because the more compact improves the CPU cache usage.
* Improve rules profiling. The most relevant changes are:
* Stop using time() and difftime() for measuring time. Use yr_stopwatch_elapsed_ns instead.
* clock_ticks field in YR_RULE and YR_STRING structures has been renamed to time_cost.
* When the time_cost for some YR_STRING structure is incremented, it's incremented for its associated YR_RULE structure too. Iterating over the YR_STRING structures is not necessary for computing the total cost for some YR_STRING.
* A time_cost field was added to the YR_RULES structure. The ratio between YR_RULES structure's time_cost and YR_RULE structure's time_cost offers a better measure of how costly a rule is.
A new set of functions yr_scanner_* has been added to the API. The purpose of these functions is providing a way for using the same YR_RULES object with different sets of values for external variables. With the existing yr_rules_scan_* functions all clients sharing a YR_RULES object were forced to use the same set of values for externals variables, because the external variables were associated to the YR_RULES object. With the new API each client can create a separate YR_SCANNER to be used with the same YR_RULES object. Each YR_SCANNER can have its own set of external variables. The existing API keeps working as usual.
Roman