Instead of measuring the time spent in verifying all atom matches, do it only for a fraction of them. This way the cost of measuring the time is a smaller percentage of the time spent doing the actual verification.
Instead of measuring the time spent in verifying all atom matches, do it only for a fraction of them. This way the cost of measuring the time is a smaller percentage of the time spent doing the actual verification.
The callback called by the compiler to report errors and warnings now received a pointer to the rule causing the error/warning. This allows richer error messages.
Frequency has been reduced to a check every 10 instructions to a check every 100 instructions. This improves performance at the expense of timeout accuracy.
Added instructions JLE_P and JFALSE_P, which are similar to their counterparts JLE and JFALSE, but pop their arguments from the stack instead of leaving the stack unmodified. These new instructions allow generating tighter loops and simplify the code a little bit.
* Add additional check in OP_COUNT for making sure that the string pointer is not a fake one.
* Initialize scratch memory in order to avoid maliciously crafted YARA rules from reading values left in the stack.
* Reduce stack footprint by allowing concatenation nodes in regexp's AST to have more than one children.
Previously all nodes in a regexp AST has two children (left and right), including nodes of type RE_NODE_CONCAT. Consequently, the AST for long regexps and hex strings was a very deep tree, which required a lot of stack space for traversing it recursively. Now regexp's AST nodes have a linked list of children, allowing multiple children per node, which reduces the AST's depth.
* Fix memory leak.
In 3cceccd8b6 the increment of profiling counters was changed to an atomic operation. This fixed race-conditions but introduced a noticeable contention while using the same YR_RULES structure with multiple threads. This commit fixes this problem by using one counter per thread, which are aggregated later, outside the critical path.
* Fix race conditions when using the same YR_RULES struct from different threads with PROFILING_ENABLED.
* Make time_cost signed as InterlockedAdd64 expects a signed integer.
Closes#918.
For compiling YARA in Windows as a DLL you must define YR_BUILDING_DLL. For using it that DLL in your project you must define YR_IMPORTING_DLL. This allows to use the same header files in both cases. If YR_BUILDING_DLL is defined, YR_API translates to __declspec(dllexport), if YR_IMPORTING_DLL is defined YR_API translates to __declspec(dllimport).
* Unify compiler's last_result and last_error fields into a single last_error field.
* Rename RE_LEX_ENVIRONMENT's last_error_code to last_error, for coherency with YR_COMPILER.
With this new feature you can pass an atom prevalence table to YARA that will be used for computing the quality of each atom based in real-life data instead of using heuristics. The atom prevalence table consists in a list of atoms, sorted lexicographically and accompanied by a score between 0-255 that indicates how frequent is that atom in the data you use to scan with YARA. With that information YARA can make better selection of the atoms.
A number of measures has been taken in order to prevent exploitation hand-crafted compiled rules files. The measures include:
* Validate bounds while reading/writing from/to scratch memory.
* Additional checks for preventing an out-of-bound access to the "stack" buffer in OP_INT_TO_DBL and a stack overflow in OP_CALL.
* Ensure that pointers point to somewhere within the arena after relocation.
* Add a canary value to objects for preventing maliciously hand-crafted objects to be used by yr_execute_code.
These measures can be disabled by defining PARANOID_EXEC as 0, but they are enabled by default.
This commit speeds up the generation of the Aho-Corasick automaton and changes the type of each item in the automaton transition table from uint64_t to uin32_t. This also speeds-up scanning because the more compact improves the CPU cache usage.
* Improve rules profiling. The most relevant changes are:
* Stop using time() and difftime() for measuring time. Use yr_stopwatch_elapsed_ns instead.
* clock_ticks field in YR_RULE and YR_STRING structures has been renamed to time_cost.
* When the time_cost for some YR_STRING structure is incremented, it's incremented for its associated YR_RULE structure too. Iterating over the YR_STRING structures is not necessary for computing the total cost for some YR_STRING.
* A time_cost field was added to the YR_RULES structure. The ratio between YR_RULES structure's time_cost and YR_RULE structure's time_cost offers a better measure of how costly a rule is.
This commit reduces YARA's memory footprint by:
* Stripping relocation information from those arena don't need to be relocated. This apply particularly to arenas created by yr_arena_load_stream, which implies that those arenas can't be saved back with yr_arena_save_stream.
* Reduce the default amount of matching data hold in YR_MATCH structures.
* Do not use thread local storage for storing fiber pools in regexp engine.
The regexp engine maintains a fiber pool for each thread that executes the yr_re_exec function. The fiber pool was stored using thread local storage (TLS), but this design as proven to be a bad decision, as each thread was responsible for freeing its own fiber pool and this was problematic in languages like Go, where the runtime can create multiple threads but the programmer don't have full control of such threads. Now the fiber pool is stored in the YR_SCAN_CONTEXT structure.
* Remove re_match macro and make calls to yr_re_match explicit in modules.
A new set of functions yr_scanner_* has been added to the API. The purpose of these functions is providing a way for using the same YR_RULES object with different sets of values for external variables. With the existing yr_rules_scan_* functions all clients sharing a YR_RULES object were forced to use the same set of values for externals variables, because the external variables were associated to the YR_RULES object. With the new API each client can create a separate YR_SCANNER to be used with the same YR_RULES object. Each YR_SCANNER can have its own set of external variables. The existing API keeps working as usual.
A memory leak occurred when parsing a hex string or regexp exceeding the RE_MAX_AST_LEVELS limit. In those cases the parsing was aborted without releasing some RE_NODE structs.
* Full structure implemented, callback available. TODO: actually use the callback's result to import rules files.
* Importing files via external callback fully implemented and working
* Removing useless comments
* Adding documentation
* Updating doc for include callbacks
* Fixing compiler_set_include_callback's signature
* Fixing issues according to @plusvic 's advice
* Limited lines to 80 chars
* include callback is now using its own user_data and does not conflict with other callbacks
* Copying include callback result to the stack before pushing it into the flex scanner
* yara's memory management functions are now exposed through the API
* Making the lexer reponsible for freeing the memory allocated by the include callback
* Updating docs accordingly
* * fixing memory management issues in callbacks
* adding a callback to allow external code to free the memory used to return the include callback's result
* API: un-exposing yara's memory management functions
* making include callbacks the only way ton include files
* adding a default include callback to open files from the disk (mimicking original behavior)
* updating docs accordingly
* fixing missing pointer cast
* re-generating lexer.c
* removing imports ununsed since commit 92877db48e
* fixing default file read callback
* Fixing line numbers in errors
* Fix some issues related to includes callback.
* Remove allow_includes field from YR_COMPILER. Includes now can be disallowed by setting the include callback to NULL.
* Fix memory and file handle leaks.
* Calculate size of error message buffer correctly.
* Add test case.
DECLARE_REFERENCE force variables to be 64bit long even when YARA is compiled in 32bit mode, but in that case the higher 4 bytes are unused. Those 4 bytes weren't being initialized and random values from the stack were saved into compiled rules files. This was actually innocuous unless you used rules files compiled by 32-bit yarac with 64-bit yara.
pe.overlay.size was undefined for files not having overlaid data. Conditions like math.entropy(0, filesize - pe.overlay.size) >= 7.0 were failing because of this, even if they make perfect sense. Now for PE files not having overlaid data pe.overlay.size is set to zero. The value is still undefined for non-PE files or malformed PEs.
Function yr_re_exec was returning with error after RE_MAX_FIBERS was reached, but the currently running fibers weren’t put back into the fiber pool, causing a memory leak. Additionally, if yr_re_exec was called again by the same thread it returned immediately with ERROR_TOO_MANY_RE_FIBERS, as no ready-to run thread was found in the pool.
The zero-sized repeat in regexps like /a{0}/ and /a{0,0}/ prevented the code for the regexp from being generated. Many regexp engines accept 0 repeats, but they are useless and we are forbidding them in YARA.
If a function name is truncated at the end of the new code still adds the truncated name it to the imported functions list. But we can live with that in favor of a simpler code.
plusvic