deps/yaracpp: move yaracpp to retdec repository, add YARA dependency.

This commit is contained in:
Matula Peter 2020-01-08 12:33:01 +01:00
parent 0f61b0fc2a
commit eaf4c5d42c
26 changed files with 1188 additions and 134 deletions

View File

@ -26,6 +26,7 @@
* Enhancement: Replace uses of `mpark::variant` from `deps/variant` with standard C++17 `std::variant`. Remove the `variant` dependency.
* Enhancement: Updated Yaramod to version v3.0.0 ([#680](https://github.com/avast/retdec/pull/680)). RetDec no longer requires Flex and Bison. This fixes [#103](https://github.com/avast/retdec/issues/103).
* Enhancement: Take out most of the types from `config` library and place them to a separate `common` library that could be used across an entire RetDec source base ([#686](https://github.com/avast/retdec/issues/686)).
* Enhancement: Move Yaracpp from its own repository to the main RetDec repository as `retdec-yaracpp` target. YARA becomes a direct RetDec dependency. This reduces the complexity of RetDec dependencies and flattens the dependency tree to a single level.
* Fix: Increased the limit for the number of entries in import directory when deciding whether a PE file is corrupted or not ([avast/pelib#13](https://github.com/avast/pelib/pull/13)).
* Fix: Fixed build on BSD systems ([#598](https://github.com/avast/retdec/pull/598)).
* Fix: Resources which are located in the different section than resource tree are now properly parsed ([#596](https://github.com/avast/retdec/pull/596)).

View File

@ -19,7 +19,7 @@ RetDec uses the following third-party libraries or other resources:
11) RapidJSON: https://github.com/Tencent/rapidjson
12) TinyXML-2: https://github.com/leethomason/tinyxml2
13) whereami: https://github.com/gpakosz/whereami
14) yaracpp: https://github.com/avast/yaracpp
14) yara: https://virustotal.github.io/yara/
15) yaramod: https://github.com/avast/yaramod
These third-party libraries or other resources are licensed under the
@ -1886,30 +1886,35 @@ Permission is granted to anyone to use this software for any purpose, including
2. Montesqieu et camembert, vive la France, zut alors!
===============================================================================
14) yaracpp
14) YARA
===============================================================================
The MIT License (MIT)
Copyright (c) 2007-2016. The YARA Authors. All Rights Reserved.
Copyright (c) 2017 Avast Software
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
of the Software, and to permit persons to whom the Software is furnished to do
so, subject to the following conditions:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation and/or
other materials provided with the distribution.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
3. Neither the name of the copyright holder nor the names of its contributors
may be used to endorse or promote products derived from this software without
specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
===============================================================================
15) yaramod

View File

@ -242,7 +242,7 @@ You can pass the following additional parameters to `cmake`:
* `-DRETDEC_COMPILE_YARA=OFF` to disable YARA rules compilation at installation step (enabled by default).
* `-DCMAKE_BUILD_TYPE=Debug` to build with debugging information, which is useful during development. By default, the project is built in the `Release` mode. This has no effect on Windows, but the same thing can be achieved by running `cmake --build .` with the `--config Debug` parameter.
* `-DCMAKE_PROGRAM_PATH=<path>` to use Perl at `<path>` (probably useful only on Windows).
* `-D<dep>_LOCAL_DIR=<path>` where `<dep>` is from `{CAPSTONE, ELFIO, GOOGLETEST, JSONCPP, KEYSTONE, LIBDWARF, LLVM, PELIB, RAPIDJSON, TINYXML, YARACPP, YARAMOD}` (e.g. `-DCAPSTONE_LOCAL_DIR=<path>`), to use the local repository clone at `<path>` for RetDec dependency instead of downloading a fresh copy at build time. Multiple such options may be used at the same time.
* `-D<dep>_LOCAL_DIR=<path>` where `<dep>` is from `{CAPSTONE, ELFIO, GOOGLETEST, JSONCPP, KEYSTONE, LIBDWARF, LLVM, PELIB, RAPIDJSON, TINYXML, YARA, YARAMOD}` (e.g. `-DCAPSTONE_LOCAL_DIR=<path>`), to use the local repository clone at `<path>` for RetDec dependency instead of downloading a fresh copy at build time. Multiple such options may be used at the same time.
* `-DRETDEC_ENABLE_<component>=ON` to build only the specified component(s) (multiple such options can be used at once), and its (theirs) dependencies. By default, all the components are built. If at least one component is enabled via this mechanism, all the other components that were not explicitly enabled (and are not needed as dependencies of enabled components) are not built. See [cmake/options.cmake](https://github.com/avast/retdec/blob/master/cmake/options.cmake) for all the available component options.
* `-DRETDEC_ENABLE_ALL=ON` can be used to (re-)enable all the components.
* Alternatively, `-DRETDEC_ENABLE=<comma-separated component list>` can be used instead of `-DRETDEC_ENABLE_<component>=ON` (e.g. `-DRETDEC_ENABLE=fileformat,loader,ctypesparser` is equivalent to `-DRETDEC_ENABLE_FILEFORMAT=ON -DRETDEC_ENABLE_LOADER=ON -DRETDEC_ENABLE_CTYPESPARSER=ON`).
@ -313,6 +313,7 @@ This repository contains the following libraries:
* `stacofin` - static code finder library.
* `unpacker` - collection of unpacking functions.
* `utils` - general C++ utility library.
* `yaracpp` - C++ wrapper for YARA.
This repository contains the following tools:
* `ar-extractortool` - frontend for the ar-extractor library (installed as `retdec-ar-extractor`).

View File

@ -50,6 +50,7 @@ option(RETDEC_ENABLE_STACOFINTOOL "" OFF)
option(RETDEC_ENABLE_UNPACKER "" OFF)
option(RETDEC_ENABLE_UNPACKERTOOL "" OFF)
option(RETDEC_ENABLE_UTILS "" OFF)
option(RETDEC_ENABLE_YARACPP "" OFF)
# Default: ALL build is enabled.
#
@ -105,6 +106,7 @@ foreach(t ${RETDEC_ENABLE})
set_if_equal(${t} "unpacker" RETDEC_ENABLE_UNPACKER)
set_if_equal(${t} "unpackertool" RETDEC_ENABLE_UNPACKERTOOL)
set_if_equal(${t} "utils" RETDEC_ENABLE_UTILS)
set_if_equal(${t} "yaracpp" RETDEC_ENABLE_YARACPP)
endforeach()
# If at least one RETDEC_ENABLE_<component> is set, disable RETDEC_ENABLE_ALL.
@ -149,7 +151,8 @@ if (RETDEC_ENABLE_AR_EXTRACTOR
OR RETDEC_ENABLE_STACOFINTOOL
OR RETDEC_ENABLE_UNPACKER
OR RETDEC_ENABLE_UNPACKERTOOL
OR RETDEC_ENABLE_UTILS)
OR RETDEC_ENABLE_UTILS
OR RETDEC_ENABLE_YARACPP)
set(RETDEC_ENABLE_ALL OFF)
endif()
@ -396,12 +399,18 @@ set_if_at_least_one_set(RETDEC_ENABLE_UTILS
RETDEC_ENABLE_LOADER
RETDEC_ENABLE_MACHO_EXTRACTOR
RETDEC_ENABLE_MACHO_EXTRACTORTOOL
RETDEC_ENABLE_PAT2YARA
RETDEC_ENABLE_PRETDEC_ENABLE_CPDETECTAT2YARA
RETDEC_ENABLE_PATTERNGEN
RETDEC_ENABLE_RTTI_FINDER
RETDEC_ENABLE_STACOFIN
RETDEC_ENABLE_UNPACKERTOOL)
set_if_at_least_one_set(RETDEC_ENABLE_YARACPP
RETDEC_ENABLE_ALL
RETDEC_ENABLE_CPDETECT
RETDEC_ENABLE_FILEINFO
RETDEC_ENABLE_STACOFIN)
# tests
set_if_all_set(RETDEC_ENABLE_BIN2LLVMIR_TESTS
RETDEC_TESTS
@ -522,10 +531,8 @@ set_if_at_least_one_set(RETDEC_ENABLE_VARIANT
set_if_at_least_one_set(RETDEC_ENABLE_WHEREAMI
RETDEC_ENABLE_UTILS)
set_if_at_least_one_set(RETDEC_ENABLE_YARACPP
RETDEC_ENABLE_CPDETECT
RETDEC_ENABLE_FILEINFO
RETDEC_ENABLE_STACOFIN)
set_if_at_least_one_set(RETDEC_ENABLE_YARA
RETDEC_ENABLE_YARACPP)
set_if_at_least_one_set(RETDEC_ENABLE_YARAMOD
RETDEC_ENABLE_BIN2PAT

2
deps/CMakeLists.txt vendored
View File

@ -15,5 +15,5 @@ cond_add_subdirectory(pelib RETDEC_ENABLE_PELIB)
cond_add_subdirectory(rapidjson RETDEC_ENABLE_RAPIDJSON)
cond_add_subdirectory(tinyxml2 RETDEC_ENABLE_TINYXML2)
cond_add_subdirectory(whereami RETDEC_ENABLE_WHEREAMI)
cond_add_subdirectory(yaracpp RETDEC_ENABLE_YARACPP)
cond_add_subdirectory(yara RETDEC_ENABLE_YARA)
cond_add_subdirectory(yaramod RETDEC_ENABLE_YARAMOD)

114
deps/yara/CMakeLists.txt vendored Normal file
View File

@ -0,0 +1,114 @@
include(ProcessorCount)
include(ExternalProject)
find_package(Threads REQUIRED)
ProcessorCount(CPUS)
set(YARA_DIR ${CMAKE_CURRENT_BINARY_DIR}/yara/src/yara)
set(YARA_INCLUDE_DIR ${YARA_DIR}/libyara/include)
set(YARA_LIBRARY_NAME "libyara")
if(MSVC)
if(CMAKE_SIZEOF_VOID_P EQUAL 8)
set(PLATFORM "x64")
set(YARA_LIBRARY_NAME "${YARA_LIBRARY_NAME}64")
set(YARAC_NAME "yarac64.exe" CACHE STRING "Yarac binary name" FORCE)
else()
set(PLATFORM "x86")
set(YARA_LIBRARY_NAME "${YARA_LIBRARY_NAME}32")
set(YARAC_NAME "yarac32.exe" CACHE STRING "Yarac binary name" FORCE)
endif()
set(YARA_WIN_DIR ${YARA_DIR}/windows/vs2015)
set(YARA_SOLUTION ${YARA_WIN_DIR}/yara.sln)
set(YARA_LIBRARY_DIR ${YARA_WIN_DIR}/libyara/$<CONFIGURATION>)
set(YARA_BUILD_COMMAND "${CMAKE_VS_MSBUILD_COMMAND}"
/m:${CPUS}
/p:Platform=${PLATFORM}
/p:Configuration=$<CONFIGURATION>
/p:PlatformToolset=${CMAKE_VS_PLATFORM_TOOLSET}
/p:WindowsTargetPlatformVersion=${CMAKE_VS_WINDOWS_TARGET_PLATFORM_VERSION}
${YARA_SOLUTION})
set(YARAC_PATH ${YARA_WIN_DIR}/\${CMAKE_INSTALL_CONFIG_NAME}/${YARAC_NAME})
else()
set(YARA_LIBRARY_DIR ${YARA_DIR}/libyara/.libs)
set(YARA_BUILD_COMMAND make -j${CPUS})
set(YARAC_PATH ${YARA_DIR}/yarac)
endif()
if(YARA_LOCAL_DIR)
message(STATUS "YARA: using local YARA directory.")
ExternalProject_Add(yara
PREFIX "yara"
DOWNLOAD_COMMAND ""
SOURCE_DIR "${YARA_LOCAL_DIR}"
DOWNLOAD_NAME yara.zip
BUILD_IN_SOURCE 1
CONFIGURE_COMMAND ""
BUILD_COMMAND ""
${YARA_BUILD_COMMAND}
INSTALL_COMMAND ""
)
force_configure_step(yara)
else()
message(STATUS "YARA: using remote YARA revision.")
ExternalProject_Add(yara
PREFIX "yara"
URL https://github.com/avast/yara/archive/e3ae3848d66cdb551e149f1a87a45bcc87578f9c.zip
URL_HASH SHA256=3acc49dc9c7f892b0d20c380727ac6f893e10d8e756924bb158db0531cac36fe
DOWNLOAD_NAME yara.zip
BUILD_IN_SOURCE 1
CONFIGURE_COMMAND ""
BUILD_COMMAND ""
${YARA_BUILD_COMMAND}
INSTALL_COMMAND ""
)
endif()
check_if_variable_changed(YARA_LOCAL_DIR CHANGED)
if(CHANGED)
ExternalProject_Get_Property(yara binary_dir)
message(STATUS "YARA: path to YARA directory changed -> cleaning CMake files in ${binary_dir}.")
clean_cmake_files(${binary_dir})
endif()
# https://github.com/avast-tl/retdec/issues/439
if(APPLE)
execute_process(
COMMAND xcrun --show-sdk-path
OUTPUT_VARIABLE MACOS_SDK_PATH
)
list(APPEND CMAKE_C_FLAGS "--sysroot=${MACOS_SDK_PATH}")
endif()
if(NOT MSVC)
ExternalProject_Add_Step(yara configure-linux
WORKING_DIRECTORY ${YARA_DIR}
DEPENDEES configure
DEPENDERS build
COMMAND ./bootstrap.sh
COMMAND ./configure
--prefix=${YARA_INSTALL_DIR}
--enable-macho
--disable-shared
--without-crypto
CC=${CMAKE_C_COMPILER}
CFLAGS=${CMAKE_C_FLAGS}
)
endif()
set(LIBRARIES
${YARA_LIBRARY_DIR}/${YARA_LIBRARY_NAME}${CMAKE_STATIC_LIBRARY_SUFFIX}
${CMAKE_THREAD_LIBS_INIT}
)
add_library(libyara INTERFACE)
add_dependencies(libyara yara)
set_target_properties(libyara PROPERTIES INTERFACE_LINK_LIBRARIES "${LIBRARIES}")
target_include_directories(libyara INTERFACE ${YARA_INCLUDE_DIR})
# Install yarac application - we may need it to compile YARA files.
install(PROGRAMS "${YARAC_PATH}" DESTINATION bin RENAME "retdec-yarac${CMAKE_EXECUTABLE_SUFFIX}")

View File

@ -1,100 +0,0 @@
include(ExternalProject)
if(CMAKE_C_COMPILER)
set(CMAKE_C_COMPILER_OPTION "-DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}")
endif()
if(CMAKE_CXX_COMPILER)
set(CMAKE_CXX_COMPILER_OPTION "-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}")
endif()
if(YARACPP_LOCAL_DIR)
message(STATUS "YaraCpp: using local YaraCpp directory.")
ExternalProject_Add(yaracpp-project
DOWNLOAD_COMMAND ""
SOURCE_DIR "${YARACPP_LOCAL_DIR}"
CMAKE_ARGS
# This does not work on MSVC, but may be useful on Linux.
-DCMAKE_BUILD_TYPE=Release
# Force the use of the same compiler as used to build the top-level
# project. Otherwise, the external project may pick up a different
# compiler, which may result in link errors.
"${CMAKE_C_COMPILER_OPTION}"
"${CMAKE_CXX_COMPILER_OPTION}"
# Disable the update step.
UPDATE_COMMAND ""
# Disable the install step.
INSTALL_COMMAND ""
)
force_configure_step(yaracpp-project)
else()
message(STATUS "YaraCpp: using remote YaraCpp revision.")
ExternalProject_Add(yaracpp-project
URL https://github.com/avast/yaracpp/archive/3417c9e855ebb06096571f54175e33c5efb0f774.zip
URL_HASH SHA256=5f37f7a3638351bd4ba24d1e7fcaa2aa74ad3c41151377ede99ccbd5fc5e079b
DOWNLOAD_NAME yaracpp.zip
CMAKE_ARGS
# This does not work on MSVC, but may be useful on Linux.
-DCMAKE_BUILD_TYPE=Release
# Force the use of the same compiler as used to build the top-level
# project. Otherwise, the external project may pick up a different
# compiler, which may result in link errors.
"${CMAKE_C_COMPILER_OPTION}"
"${CMAKE_CXX_COMPILER_OPTION}"
# Disable the update step.
UPDATE_COMMAND ""
# Disable the install step.
INSTALL_COMMAND ""
LOG_DOWNLOAD ON
LOG_CONFIGURE ON
LOG_BUILD ON
)
endif()
check_if_variable_changed(YARACPP_LOCAL_DIR CHANGED)
if(CHANGED)
ExternalProject_Get_Property(yaracpp-project binary_dir)
message(STATUS "YaraCpp: path to YaraCpp directory changed -> cleaning CMake files in ${binary_dir}.")
clean_cmake_files(${binary_dir})
endif()
ExternalProject_Get_Property(yaracpp-project source_dir)
ExternalProject_Get_Property(yaracpp-project binary_dir)
set(YARA_DIR ${binary_dir}/deps/yara/src/yara)
set(YARA_INCLUDE_DIR ${YARA_DIR}/libyara/include)
set(YARA_LIBRARY_NAME "libyara")
if(MSVC)
if(CMAKE_SIZEOF_VOID_P EQUAL 8)
set(YARA_LIBRARY_NAME "${YARA_LIBRARY_NAME}64")
set(YARAC_NAME "yarac64.exe" CACHE STRING "Yarac binary name" FORCE)
else()
set(YARA_LIBRARY_NAME "${YARA_LIBRARY_NAME}32")
set(YARAC_NAME "yarac32.exe" CACHE STRING "Yarac binary name" FORCE)
endif()
set(YARA_WIN_DIR ${YARA_DIR}/windows/vs2015)
set(YARA_LIBRARY_DIR ${YARA_WIN_DIR}/libyara/$<CONFIGURATION>)
set(YARAC_PATH ${YARA_WIN_DIR}/\${CMAKE_INSTALL_CONFIG_NAME}/${YARAC_NAME})
elseif(MSYS)
set(YARA_LIBRARY_DIR ${YARA_DIR}/libyara/.libs)
set(YARAC_PATH ${YARA_DIR}/yarac.exe)
else()
set(YARA_LIBRARY_DIR ${YARA_DIR}/libyara/.libs)
set(YARAC_PATH ${YARA_DIR}/yarac)
endif()
# Add libraries.
add_library(yaracpp INTERFACE)
add_dependencies(yaracpp yaracpp-project)
target_include_directories(yaracpp SYSTEM INTERFACE ${source_dir}/include)
target_include_directories(yaracpp SYSTEM INTERFACE ${YARA_INCLUDE_DIR})
target_link_libraries(yaracpp INTERFACE debug ${binary_dir}/src/${DEBUG_DIR}${CMAKE_STATIC_LIBRARY_PREFIX}yaracpp${CMAKE_STATIC_LIBRARY_SUFFIX})
target_link_libraries(yaracpp INTERFACE debug ${YARA_LIBRARY_DIR}/${YARA_LIBRARY_NAME}${CMAKE_STATIC_LIBRARY_SUFFIX})
target_link_libraries(yaracpp INTERFACE optimized ${binary_dir}/src/${RELEASE_DIR}${CMAKE_STATIC_LIBRARY_PREFIX}yaracpp${CMAKE_STATIC_LIBRARY_SUFFIX})
target_link_libraries(yaracpp INTERFACE optimized ${YARA_LIBRARY_DIR}/${YARA_LIBRARY_NAME}${CMAKE_STATIC_LIBRARY_SUFFIX})
# Install yarac application - we may need it to compile YARA files.
install(PROGRAMS "${YARAC_PATH}" DESTINATION bin RENAME "retdec-yarac${CMAKE_EXECUTABLE_SUFFIX}")

View File

@ -0,0 +1,11 @@
/**
* @file include/yaracpp/doxygen.h
* @brief Doxygen documentation of the yaracpp namespace.
* @copyright (c) 2017 Avast Software, licensed under the MIT license
*/
// As there is no better place to comment this namespace, we do this in the
// present file.
/// @namespace yaracpp A library providing parsing and detection of
/// YARA rules.

View File

@ -0,0 +1,43 @@
/**
* @file include/yaracpp/types/yara_match.h
* @brief Library representation of one YARA match.
* @copyright (c) 2017 Avast Software, licensed under the MIT license
*/
#pragma once
#include <cstdint>
#include <vector>
namespace yaracpp
{
/**
* Representation of one match
*/
class YaraMatch
{
private:
std::size_t offset; ///< offset of match detection
std::vector<std::uint8_t> data; ///< data
public:
/// @name Getters
/// @{
std::size_t getOffset() const;
std::size_t getDataSize() const;
const std::vector<std::uint8_t>& getData() const;
/// @}
/// @name Setters
/// @{
void setOffset(std::size_t offsetValue);
void setData(const std::uint8_t* dataBuffer, std::size_t dataLength);
/// @}
/// @name Other methods
/// @{
void addByte(std::uint8_t byte);
/// @}
};
} // namespace yaracpp

View File

@ -0,0 +1,54 @@
/**
* @file include/yaracpp/types/yara_meta.h
* @brief Library representation of one YARA meta.
* @copyright (c) 2017 Avast Software, licensed under the MIT license
*/
#pragma once
#include <string>
namespace yaracpp
{
/**
* Representation of metadata
*/
class YaraMeta
{
public:
enum class Type
{
String,
Int
};
private:
std::string id; ///< name of meta
Type type; ///< type of meta
std::string strValue; ///< string value of meta
std::uint64_t intValue; ///< int value of meta
public:
/// @name Const getters
/// @{
const std::string& getId() const;
YaraMeta::Type getType() const;
const std::string& getStringValue() const;
const std::uint64_t& getIntValue() const;
/// @}
/// @name Getters
/// @{
std::string& getStringValue();
std::uint64_t& getIntValue();
/// @}
/// @name Setters
/// @{
void setId(const std::string &metaId);
void setType(YaraMeta::Type metaType);
void setStringValue(const std::string &metaValue);
void setIntValue(std::uint64_t metaValue);
/// @}
};
} // namespace yaracpp

View File

@ -0,0 +1,64 @@
/**
* @file include/yaracpp/types/yara_rule.h
* @brief Library representation of one YARA rule.
* @copyright (c) 2017 Avast Software, licensed under the MIT license
*/
#pragma once
#include <iosfwd>
#include <vector>
#include "retdec/yaracpp/types/yara_match.h"
#include "retdec/yaracpp/types/yara_meta.h"
namespace yaracpp
{
/**
* Representation of one YARA rule
*/
class YaraRule
{
private:
std::string name; ///< name of rule
std::vector<YaraMeta> metas; ///< all meta-data related to rule
std::vector<YaraMatch> matches; ///< all matches of rule
public:
/// @name Const getters
/// @{
const std::string &getName() const;
const YaraMeta* getMeta(const std::string &id) const;
const YaraMatch* getMatch(std::size_t index) const;
const YaraMatch* getFirstMatch() const;
const std::vector<YaraMeta>& getMetas() const;
const std::vector<YaraMatch>& getMatches() const;
std::size_t getNumberOfMetas() const;
std::size_t getNumberOfMatches() const;
/// @}
/// @name Getters
/// @{
YaraMeta* getMeta(const std::string &id);
YaraMatch* getMatch(std::size_t index);
YaraMatch* getFirstMatch();
/// @}
/// @name Setters
/// @{
void setName(const std::string &ruleName);
/// @}
/// @name Other methods
/// @{
void addMeta(const YaraMeta &meta);
void addMatch(const YaraMatch &match);
/// @}
// @name Utility operator overloads
// @{
friend std::ostream& operator<<(std::ostream& o, const YaraRule& rule);
// @}
};
} // namespace yaracpp

View File

@ -0,0 +1,98 @@
/**
* @file include/yaracpp/yara_detector/yara_detector.h
* @brief Interpret of YARA rules.
* @copyright (c) 2017 Avast Software, licensed under the MIT license
*/
#pragma once
#include <string>
#include <unordered_map>
#include <vector>
#include <yara/compiler.h>
#include <yara/types.h>
#include "retdec/yaracpp/types/yara_rule.h"
namespace yaracpp
{
/**
* Interpret of YARA rules
*/
class YaraDetector
{
public:
/**
* Structure for callback function
*/
class CallbackSettings
{
private:
bool storeAll; ///< set to @c true if you want store all rules (not only detected)
std::vector<YaraRule> &storedDetected; ///< link to detected rules
std::vector<YaraRule> &storedUndetected; ///< link to undetected rules
public:
CallbackSettings(bool cStoreAll, std::vector<YaraRule> &cDetected, std::vector<YaraRule> &cUndetected);
~CallbackSettings();
/// @name Other methods
/// @{
void addDetected(YaraRule &rule);
void addUndetected(YaraRule &rule);
bool storeAllRules() const;
/// @}
};
struct RuleFile
{
RuleFile(const std::string& pathToFile_, bool precompiled_, FILE* handle_)
: pathToFile(pathToFile_), precompiled(precompiled_), handle(handle_) {}
std::string pathToFile;
bool precompiled;
FILE* handle;
};
private:
YR_COMPILER *compiler; ///< compiler or text rules
std::vector<FILE*> files; ///< representation of files with rules
std::vector<YaraRule> detectedRules; ///< representation of detected rules
std::vector<YaraRule> undetectedRules; ///< representation of undetected rules
YR_RULES* textFilesRules; ///< rules from input text files
std::vector<YR_RULES*> precompiledRules; ///< rules from precompiled files
bool stateIsValid; ///< internal state of instance
bool needsRecompilation; ///< indicates whether text files need recompilation
/// @name Static auxiliary methods
/// @{
static int yaraCallback(int message, void *messageData, void *userData);
/// @}
/// @name Auxiliary detection methods
/// @{
template <typename T> bool analyzeWithScan(T&& value, bool storeAllRules = false);
YR_RULES* getCompiledRules();
/// @}
public:
YaraDetector();
~YaraDetector();
/// @name Other methods
/// @{
bool addRules(const char *string);
bool addRuleFile(const std::string &pathToFile, const std::string &nameSpace = std::string());
bool isInValidState() const;
/// @}
/// @name Detection methods
/// @{
bool analyze(const std::string &pathToInputFile, bool storeAllRules = false);
bool analyze(std::vector<std::uint8_t> &bytes, bool storeAllRules = false);
const std::vector<YaraRule>& getDetectedRules() const;
const std::vector<YaraRule>& getUndetectedRules() const;
/// @}
};
} // namespace yaracpp

View File

@ -0,0 +1,9 @@
/**
* @file include/yaracpp/yaracpp.h
* @brief Interface to yaracpp library.
* @copyright (c) 2017 Avast Software, licensed under the MIT license
*/
#pragma once
#include "retdec/yaracpp/yara_detector/yara_detector.h"

View File

@ -39,3 +39,4 @@ cond_add_subdirectory(stacofintool RETDEC_ENABLE_STACOFINTOOL)
cond_add_subdirectory(unpacker RETDEC_ENABLE_UNPACKER)
cond_add_subdirectory(unpackertool RETDEC_ENABLE_UNPACKERTOOL)
cond_add_subdirectory(utils RETDEC_ENABLE_UTILS)
cond_add_subdirectory(yaracpp RETDEC_ENABLE_YARACPP)

View File

@ -20,5 +20,9 @@ set(CPDETECT_SOURCES
)
add_library(retdec-cpdetect STATIC ${CPDETECT_SOURCES})
target_link_libraries(retdec-cpdetect libdwarf retdec-fileformat yaracpp tinyxml2)
target_link_libraries(retdec-cpdetect
libdwarf
retdec-fileformat
retdec-yaracpp
tinyxml2)
target_include_directories(retdec-cpdetect PUBLIC ${PROJECT_SOURCE_DIR}/include/)

View File

@ -12,7 +12,7 @@
#include "retdec/cpdetect/compiler_detector/compiler_detector.h"
#include "retdec/cpdetect/settings.h"
#include "retdec/cpdetect/utils/version_solver.h"
#include "yaracpp/yara_detector/yara_detector.h"
#include "retdec/yaracpp/yara_detector/yara_detector.h"
using namespace retdec::fileformat;
using namespace retdec::utils;

View File

@ -116,7 +116,7 @@ target_link_libraries(retdec-fileinfo
retdec-ar-extractor
retdec-fileformat
retdec-cpdetect
yaracpp
retdec-yaracpp
retdec-utils
retdec-common
retdec-config

View File

@ -10,7 +10,7 @@
#include "retdec/utils/filesystem_path.h"
#include "retdec/utils/string.h"
#include "fileinfo/pattern_detector/pattern_detector.h"
#include "yaracpp/yara_detector/yara_detector.h"
#include "retdec/yaracpp/yara_detector/yara_detector.h"
using namespace retdec::utils;
using namespace yaracpp;

View File

@ -7,7 +7,7 @@ target_link_libraries(retdec-stacofin
retdec-config
retdec-common
retdec-utils
yaracpp
retdec-yaracpp
capstone
)
target_include_directories(retdec-stacofin PUBLIC ${PROJECT_SOURCE_DIR}/include/)

View File

@ -9,7 +9,7 @@
#include <string>
#include "retdec/stacofin/stacofin.h"
#include "yaracpp/yara_detector/yara_detector.h"
#include "retdec/yaracpp/yara_detector/yara_detector.h"
#include "retdec/loader/loader/image.h"
#include "retdec/utils/string.h"

View File

@ -0,0 +1,9 @@
add_library(retdec-yaracpp STATIC
types/yara_match.cpp
types/yara_meta.cpp
types/yara_rule.cpp
yara_detector/yara_detector.cpp
)
target_include_directories(retdec-yaracpp PUBLIC ${PROJECT_SOURCE_DIR}/include/)
target_link_libraries(retdec-yaracpp libyara)

View File

@ -0,0 +1,67 @@
/**
* @file src/types/yara_match.cpp
* @brief Library representation of one YARA match.
* @copyright (c) 2017 Avast Software, licensed under the MIT license
*/
#include "retdec/yaracpp/types/yara_match.h"
namespace yaracpp
{
/**
* Get offset of detection
* @return Offset of detection
*/
std::size_t YaraMatch::getOffset() const
{
return offset;
}
/**
* Get size of stored bytes
* @return size of stored bytes
*/
std::size_t YaraMatch::getDataSize() const
{
return data.size();
}
/**
* Get stored bytes
* @return Byte representation of match
*/
const std::vector<std::uint8_t>& YaraMatch::getData() const
{
return data;
}
/**
* Set offset of detection
* @param offsetValue Offset of detection
*/
void YaraMatch::setOffset(std::size_t offsetValue)
{
offset = offsetValue;
}
/**
* Set detected data
* @param dataBuffer Pointer to the beginning of the data
* @param dataLength Length of the buffer
*/
void YaraMatch::setData(const std::uint8_t* dataBuffer, std::size_t dataLength)
{
data.assign(dataBuffer, dataBuffer + dataLength);
}
/**
* Add detected byte
* @param byte Value of byte
*/
void YaraMatch::addByte(std::uint8_t byte)
{
data.push_back(byte);
}
} // namespace yaracpp

View File

@ -0,0 +1,110 @@
/**
* @file src/types/yara_meta.cpp
* @brief Library representation of one YARA meta.
* @copyright (c) 2017 Avast Software, licensed under the MIT license
*/
#include <cassert>
#include "retdec/yaracpp/types/yara_meta.h"
namespace yaracpp
{
/**
* Get name of meta
* @return Name of meta
*/
const std::string& YaraMeta::getId() const
{
return id;
}
/**
* Get type of meta
* @return Type of meta
*/
YaraMeta::Type YaraMeta::getType() const
{
return type;
}
/**
* Get string value of meta
* @return String value of meta
*/
const std::string& YaraMeta::getStringValue() const
{
assert(type == Type::String);
return strValue;
}
/**
* Get int value of meta
* @return Int value of meta
*/
const std::uint64_t& YaraMeta::getIntValue() const
{
assert(type == Type::Int);
return intValue;
}
/**
* Get string value of meta
* @return String value of meta
*/
std::string& YaraMeta::getStringValue()
{
assert(type == Type::String);
return strValue;
}
/**
* Get int value of meta
* @return Int value of meta
*/
std::uint64_t& YaraMeta::getIntValue()
{
assert(type == Type::Int);
return intValue;
}
/**
* Set name of meta
* @param metaId Name of meta
*/
void YaraMeta::setId(const std::string &metaId)
{
id = metaId;
}
/**
* Set type of meta
* @param metaType Type of meta
*/
void YaraMeta::setType(YaraMeta::Type metaType)
{
type = metaType;
}
/**
* Set string value of meta
* @param metaValue String value of meta
*/
void YaraMeta::setStringValue(const std::string &metaValue)
{
assert(type == Type::String);
strValue = metaValue;
}
/**
* Set int value of meta
* @param metaValue Int value of meta
*/
void YaraMeta::setIntValue(std::uint64_t metaValue)
{
assert(type == Type::Int);
intValue = metaValue;
}
} // namespace yaracpp

View File

@ -0,0 +1,164 @@
/**
* @file src/types/yara_rule.cpp
* @brief Library representation of one YARA rule.
* @copyright (c) 2017 Avast Software, licensed under the MIT license
*/
#include "retdec/yaracpp/types/yara_rule.h"
#include <ostream>
namespace yaracpp
{
/**
* Get name related to this rule
* @return Name of rule
*/
const std::string &YaraRule::getName() const
{
return name;
}
/**
* Get selected meta related to this rule
* @param id Name of selected meta
* @return Pointer to selected meta or @c nullptr if such meta is not found
*/
const YaraMeta* YaraRule::getMeta(const std::string &id) const
{
for(const auto &meta : metas)
{
if(meta.getId() == id)
{
return &meta;
}
}
return nullptr;
}
/**
* Get selected match of this rule
* @param index Index of selected match (indexed from 0)
* @return Pointer to selected match or @c nullptr if such match is not found
*/
const YaraMatch* YaraRule::getMatch(std::size_t index) const
{
return ((index < matches.size()) ? &matches[index] : nullptr);
}
/**
* Get first match of this rule
* @return Pointer to first match or @c nullptr if rule has no matches
*/
const YaraMatch* YaraRule::getFirstMatch() const
{
return getMatch(0);
}
/**
* Get all metas
* @return All metas related to this rule
*/
const std::vector<YaraMeta>& YaraRule::getMetas() const
{
return metas;
}
/**
* Get all matches
* @return All matches related to this rule
*/
const std::vector<YaraMatch>& YaraRule::getMatches() const
{
return matches;
}
/**
* Get number of stored metas
* @return Number of stored metas
*/
std::size_t YaraRule::getNumberOfMetas() const
{
return metas.size();
}
/**
* Get number of stored matches
* @return Number of stored matches
*/
std::size_t YaraRule::getNumberOfMatches() const
{
return matches.size();
}
/**
* Get selected meta related to this rule
* @param id Name of selected meta
* @return Pointer to selected meta or @c nullptr if such meta is not found
*/
YaraMeta* YaraRule::getMeta(const std::string &id)
{
return const_cast<YaraMeta*>(static_cast<const YaraRule*>(this)->getMeta(id));
}
/**
* Get selected match of this rule
* @param index Index of selected match (indexed from 0)
* @return Pointer to selected match or @c nullptr if such match is not found
*/
YaraMatch* YaraRule::getMatch(std::size_t index)
{
return ((index < matches.size()) ? &matches[index] : nullptr);
}
/**
* Get first match of this rule
* @return Pointer to first match or @c nullptr if rule has no matches
*/
YaraMatch* YaraRule::getFirstMatch()
{
return const_cast<YaraMatch*>(static_cast<const YaraRule*>(this)->getFirstMatch());
}
/**
* Set name of rule
* @param ruleName Name of rule
*/
void YaraRule::setName(const std::string &ruleName)
{
name = ruleName;
}
/**
* Add meta
* @param meta Meta related to this rule
*/
void YaraRule::addMeta(const YaraMeta &meta)
{
metas.push_back(meta);
}
/**
* Add match
* @param match Match related to this rule
*/
void YaraRule::addMatch(const YaraMatch &match)
{
matches.push_back(match);
}
/**
* Overload to print rule's name
* @param o output stream
* @param rule rule being printed
* @return output stream for chaining operators
*/
std::ostream& operator<<(std::ostream& o, const YaraRule& rule)
{
o << rule.name;
return o;
}
} // namespace yaracpp

View File

@ -0,0 +1,392 @@
/**
* @file src/yara_detector/yara_detector.cpp
* @brief Interpret of YARA rules.
* @copyright (c) 2017 Avast Software, licensed under the MIT license
*/
#include <yara.h>
#include "retdec/yaracpp/yara_detector/yara_detector.h"
namespace yaracpp
{
namespace
{
/**
* Interface for YARA scanning interface. Uses template specialization
* to decide whether to scan file or memory buffer.
*/
template <typename T>
struct Scanner {};
/**
* Specialization for scanning files.
*/
template <>
struct Scanner<std::string>
{
static bool scan(YR_RULES* rules, YR_CALLBACK_FUNC callback, YaraDetector::CallbackSettings& settings, const std::string& pathToFile)
{
return yr_rules_scan_file(rules, pathToFile.c_str(), 0, callback, &settings, 0) == ERROR_SUCCESS;
}
};
/**
* Specialization for scanning memory buffers.
*/
template <>
struct Scanner<std::vector<std::uint8_t>>
{
static bool scan(YR_RULES* rules, YR_CALLBACK_FUNC callback, YaraDetector::CallbackSettings& settings, const std::vector<std::uint8_t>& buffer)
{
return yr_rules_scan_mem(rules, const_cast<uint8_t*>(buffer.data()), buffer.size(), 0, callback, &settings, 0) == ERROR_SUCCESS;
}
};
/**
* Interface for Scanner. Provides template type deduction and
* always passes correct type into Scanner template.
*/
template <typename T>
bool scan(YR_RULES* rules, YR_CALLBACK_FUNC callback, YaraDetector::CallbackSettings& settings, T&& value)
{
return Scanner<std::decay_t<T>>::scan(rules, callback, settings, std::forward<T>(value));
}
}
/**
* Constructor
*/
YaraDetector::YaraDetector() : compiler(nullptr), files(), detectedRules(), undetectedRules(), textFilesRules(nullptr),
precompiledRules(), stateIsValid(true), needsRecompilation(true)
{
stateIsValid = ((yr_initialize() == ERROR_SUCCESS) && (yr_compiler_create(&compiler) == ERROR_SUCCESS));
}
/**
* Destructor
*/
YaraDetector::~YaraDetector()
{
for (auto* file : files)
{
if (file)
fclose(file);
}
files.clear();
detectedRules.clear();
undetectedRules.clear();
if (compiler)
{
yr_compiler_destroy(compiler);
}
if (textFilesRules)
yr_rules_destroy(textFilesRules);
for (auto* rules : precompiledRules)
{
if (rules)
yr_rules_destroy(rules);
}
yr_finalize();
}
/**
* Constructor of settings class
* @param cStoreAll If this parameter is set to @c true, all rules will be
* stored (not only detected rules)
* @param cDetected Into this variable detected rules will be stored
* @param cUndetected Into this variable undetected rules will be stored
*/
YaraDetector::CallbackSettings::CallbackSettings(bool cStoreAll, std::vector<YaraRule> &cDetected, std::vector<YaraRule> &cUndetected) :
storeAll(cStoreAll), storedDetected(cDetected), storedUndetected(cUndetected)
{
}
/**
* Destructor of settings class
*/
YaraDetector::CallbackSettings::~CallbackSettings()
{
}
/**
* Add detected rule
* @param rule Rule to store
*/
void YaraDetector::CallbackSettings::addDetected(YaraRule &rule)
{
storedDetected.push_back(rule);
}
/**
* Add undetected rule
* @param rule Rule to store
*/
void YaraDetector::CallbackSettings::addUndetected(YaraRule &rule)
{
storedUndetected.push_back(rule);
}
/**
* Check if storing of all rules (not only detected) is set
* @return @c true if storing of all rules is set
*/
bool YaraDetector::CallbackSettings::storeAllRules() const
{
return storeAll;
}
/**
* Callback function for scanning of input file
* @param message Type of message from libyara
* @param messageData Content of message
* @param userData @c Pointer for save information about detected rules
* @return Instruction for the next scan
*
* Read libyara documentation for more detailed information about callback function
*/
int YaraDetector::yaraCallback(int message, void *messageData, void *userData)
{
if(message == CALLBACK_MSG_IMPORT_MODULE || message == CALLBACK_MSG_MODULE_IMPORTED)
{
return CALLBACK_CONTINUE;
}
else if(message == CALLBACK_MSG_SCAN_FINISHED)
{
return CALLBACK_ABORT;
}
else if(message != CALLBACK_MSG_RULE_MATCHING && message != CALLBACK_MSG_RULE_NOT_MATCHING)
{
return CALLBACK_ERROR;
}
auto *settings = static_cast<CallbackSettings*>(userData);
if(!settings)
{
return CALLBACK_ERROR;
}
else if(!settings->storeAllRules() && message == CALLBACK_MSG_RULE_NOT_MATCHING)
{
return CALLBACK_CONTINUE;
}
auto *actRule = static_cast<YR_RULE*>(messageData);
if(!actRule)
{
return CALLBACK_ERROR;
}
YaraRule actual;
actual.setName(actRule->identifier);
YR_META *meta;
yr_rule_metas_foreach(actRule, meta)
{
if(meta && meta->type != META_TYPE_NULL)
{
YaraMeta yaralMeta;
yaralMeta.setId(meta->identifier);
if(meta->type == META_TYPE_STRING)
{
yaralMeta.setType(YaraMeta::Type::String);
yaralMeta.setStringValue(meta->string);
}
else
{
yaralMeta.setType(YaraMeta::Type::Int);
yaralMeta.setIntValue(meta->integer);
}
actual.addMeta(yaralMeta);
}
}
if(message == CALLBACK_MSG_RULE_MATCHING)
{
YR_STRING *string;
yr_rule_strings_foreach(actRule, string)
{
if(string)
{
YR_MATCH *match;
yr_string_matches_foreach(string, match)
{
if(match)
{
YaraMatch yaralMatch;
yaralMatch.setOffset(match->base + match->offset);
yaralMatch.setData(match->data, match->data_length);
actual.addMatch(yaralMatch);
}
}
}
}
settings->addDetected(actual);
}
else
{
settings->addUndetected(actual);
}
return CALLBACK_CONTINUE;
}
/**
* Add text rules to compiler
* @param string YARA rules to add
*/
bool YaraDetector::addRules(const char *string)
{
const auto result = yr_compiler_add_string(compiler, string, nullptr);
needsRecompilation = (result == 0);
return needsRecompilation;
}
/**
* Add external file with text rules
* @param pathToFile Path to rule file
* @param nameSpace Namespace to use for the given rule file. If the file is
* already compiled, this has no effect. If it is a text file,
* this allows to have multiple rules with the same ID across
* multiple rule files.
*/
bool YaraDetector::addRuleFile(const std::string &pathToFile, const std::string &nameSpace)
{
// AT first, try to load the files as precompiled file
YR_RULES* rules = nullptr;
if (yr_rules_load(pathToFile.c_str(), &rules) == ERROR_SUCCESS)
{
precompiledRules.push_back(rules);
}
// If we didn't succeeded consider it as text file
else
{
auto file = fopen(pathToFile.c_str(), "r");
if (!file)
return false;
const char* ns = nameSpace.empty() ? nullptr : nameSpace.c_str();
if (yr_compiler_add_file(compiler, file, ns, nullptr) != 0)
{
fclose(file);
return false;
}
files.push_back(file);
needsRecompilation = true;
}
return true;
}
/**
* Getter for state of instance
* @return @c true if all is OK, @c false otherwise
*/
bool YaraDetector::isInValidState() const
{
return stateIsValid;
}
/**
* Analyze input file
* @param pathToInputFile Path to input file
* @param storeAllRules If this parameter is set to @c true, store all rules (not only detected)
* @return @c true if analysis completed without any error, otherwise @c false.
*/
bool YaraDetector::analyze(const std::string &pathToInputFile, bool storeAllRules)
{
return analyzeWithScan(pathToInputFile, storeAllRules);
}
/**
* Analyze input bytes
* @param bytes Vector of input bytes
* @param storeAllRules If this parameter is set to @c true, store all rules (not only detected)
* @return @c true if analysis completed without any error, otherwise @c false.
*/
bool YaraDetector::analyze(std::vector<std::uint8_t> &bytes, bool storeAllRules)
{
return analyzeWithScan(bytes, storeAllRules);
}
/**
* Get detected rules
* @return Detected rules
*/
const std::vector<YaraRule>& YaraDetector::getDetectedRules() const
{
return detectedRules;
}
/**
* Get undetected rules
* @return Undetected rules
*/
const std::vector<YaraRule>& YaraDetector::getUndetectedRules() const
{
return undetectedRules;
}
/**
* Analyze input sequence
* @param value Value to analyze
* @param storeAllRules If this parameter is set to @c true, store all rules (not only detected)
* @return @c true if analysis completed without any error, otherwise @c false.
*/
template <typename T>
bool YaraDetector::analyzeWithScan(T&& value, bool storeAllRules)
{
auto settings = CallbackSettings(storeAllRules, detectedRules, undetectedRules);
auto rules = getCompiledRules();
if (!(rules))
return false;
if (!scan(rules, yaraCallback, settings, std::forward<T>(value)))
return false;
for (auto* rules : precompiledRules)
{
if (!scan(rules, yaraCallback, settings, std::forward<T>(value)))
return false;
}
return true;
}
/**
* Returns the compiled rules from text files.
* @return Compiled rules.
*/
YR_RULES* YaraDetector::getCompiledRules()
{
// File is text file and needs to be compiled first
// All text files are compiled into single YR_RULES structure and we shouldn't compile it twice if it's not needed
// analyze() called for the first time or the file was added since the last analyze() call
if (needsRecompilation)
{
YR_RULES* rules = nullptr;
if (yr_compiler_get_rules(compiler, &rules) != ERROR_SUCCESS)
return nullptr;
if (textFilesRules)
yr_rules_destroy(textFilesRules);
textFilesRules = rules;
needsRecompilation = false;
}
return textFilesRules;
}
} // namespace yaracpp

View File

@ -1 +1 @@
Compilation and installation of YARA rules is done in deps/yaracpp.
Compilation and installation of YARA rules is done in deps/yara.