This commit introduces and improves a number of things related to static
analysis of malicious Microsoft Office files. In particular:
* Extend Extractor logic to allow for static-only analysis.
* Enhance Yara integration allowing it to take external variables,
perform on in-memory buffers, and work correctly with Office archives.
* Improve parsing logic of cmd.exe in our Scripting engine.
* More Extracted Artefacts logic and handling as well as versioning.
* Various tweaks and improvements.
This commit is mostly complete, however, it could certainly use more
error/exception handling. We should also more generically catch
exceptions thrown in Extractor plugins.
From now on we'll force our reports to contain a meta.description for
each Yara match, regardless of whether said Yara rule has a description
or not. This simplifies the patches that would be required for #1687.
Any and all failing reports should 'simply' be reprocessed now.
Rather than creating the index_*.yar files during "cuckoo -d" and then
loading those when running "cuckoo process", we'll from now on always
enumerate the available Yara rules recursively and load 'em on-the-go.
This simplifies life when updating Yara rules and re-running "cuckoo
process -r". Furthermore this commit adds recursive walking support as
well as following symbolic links.
Currently there's a limitation in the usage of Yara rules, though, as at
the moment it isn't possible to use "include" statements in the rules
(with relative paths, anyway). We may be able to fix that by temporarily
os.chdir()'ing into the yara/category/ directory.
This commit resolves#773 and provides a stepping stone for #776 & #512.
Note that the os.path.realpath() is required since Mac OS X symlinks
/var to /private/var and the "sample" filepath points to /private/var
whereas the CWD points to /var, causing unit testing issues.
Thanks to Victor Manuel Alvarez for working on properly releasing the
yara-python package (and Yara in general, of course).
Implements the ability to recursively extract core elements from other
extracted elements (and in the future perhaps other items as well?)
For now solely depends on Yara rules to start the recursive extraction
process. Introduces a new Extractor class for which rules may be placed
in $CWD/signatures/extractor/.
* Make Static handle archives transparently.
* Take the "exe" package for granted just as the others.
* Move get_keys() to File where it's more appropriate.
In order to show proper information about the file that has been
analyzed, these two need each other. From now on the reports will be
able to share information (size, hashes, etc) about the actual file
being analyzed rather than the archive around it.
Introduces the concept of temporary files that are removed after the
object expires / is no longer used.
By compiling all of the Yara rules at Cuckoo's startup we avoid the
issue that we've been having up until now: finding out about Yara
problems at runtime. Naturally this simplifies some Yara-related code,
resolves an issue where, in case one of the referenced Yara rules has a
syntax error, no Yara rule matches are performed & get_yara() returns
None (i.e., silently not doing any Yara magic & later on crashing hard
due to the silent thing), and some accompanying unit tests.
In a couple of weeks there will be a new Yara release which we can
properly include as a dependency. With that release we'll be able to
bind Cuckoo to our version of choice, i.e., the latest version.
The new offset field will allow us to quickly locate where the strings
were found in the binary / memory / etc.
This probably breaks backwards compatibility with older versions of
Yara, but if no Yara release is around in time, we can just do a soft
error on versions that are not 3.5.0.
Also disables some kernel driver stuff that's not yet ready for
deployment.
Also gets rid of older behavior and backwards compatibility with another
version of the libmagic library. Note that removing such older code from
now on requires proper unittests to ensure that code remains working.
Also reintroduce the capability to provide a custom connection string to
the Database (which is possible now that making the connection is no
longer part of __init__).
Jurriaan Bremer