This commit passes through tlsmaster.txt secrets to httpreplay such that
it can prepare the correct .mitm file through mitmproxy and also
installs the mitmproxy certificate as part of replay. Replay now
supports both HTTP and HTTPS.
Implements an Auxiliary module for setting up mitmdump (part of
mitmproxy), version 3.0 or higher, for replaying pre-captured HTTP
requests.
Bumps httpreplay for decoupling the pcap2mitm method from the click
entry point and does some other code cleanup. Also makes Cuckoo
Rooter less quiet, somewhat simplifying debugging.
In the future we might want also want to invoke drop_enable and
drop_disable Cuckoo Rooter commands, ensuring no network traffic is
propagated outside of the host machine.
Introduces the concept of Yara rules for extracted scripts. We'll likely
also have a category for extracted shellcode soon. First to work on
recursive extraction and allowing 3rd party plugins to, e.g., extract a
shellcode stream from a PowerShell script (statically).
While checking the configuration settings we want to be able to do some
console logging in case something went wrong. Improve our existing logic
to initialize console logging rather than printing to stderr directly.
For unknown reasons nobody ran into this issue before the 2.0.0 release,
however, on a clean Cuckoo setup with newly created configuration, the
machine.options field would always be set to None rather than an empty
string or list, resulting in various exceptions around the Core code.
This commit more improves the handling of this 'special' list type and
ensures the machine.options field will always be at least an empty list.
This counts for both the data retrieved from the configuration as well
as the database handling part.
Thanks to CSIRTUK and pinksawtooth for reporting and doomedraven for
providing an ad hoc solution.
* Code de-duplication with regards to get_section_types() in Config.
* Make cast() & read_kv_conf() less strict about star sections.
* Don't overwrite given values for default star items, e.g., "cuckoo1".
* While writing away the configuration, the "section" variable would be
overwritten, causing an exception for the second star entry.
* Improve config2() to accept star items and return a Dictionary.
* Identify and report Host <> Guest connection issues and drop hints to
the user in both the command-line logs as well as the Web Interface.
* Provide FAQ entry for these issues.
* Only keep unique errors & actions in the report.
* Some other documentation tweaks (regarding formatting etc).
* Due to legacy configuration, the "bpf" entry may not exist yet.
* Mapping from actual version numbers to configuration migration ones.
* Add support for bugfix release Cuckoo Sandbox 1.1.1.
* Unittesting for configuration from Cuckoo 1.1.
* Get rid of the "encoding" configuration value. This should be latin-1
as otherwise we'll get unicode-related issues. If there are actually
other possible encodings formats we can always consider changing to
one of those if they meet our requirements (i.e., unittests).
* Bump file I/O buffering size to 1 MB, this may improve performance
slightly (as compared to the default 8 KB value).
* Get rid of UnicodeError & use "indent" value the way you expect it.
Especially the removal of codecs.open() is a great performance boost,
reducing the required time by almost half in some cases.
It's most certainly not necessary to read in configuration files once or
multiple times for each processed analysis. We strip down on re-reading
configuration files by, yet again, moving away from Config() usage to
the config() and config2() variants (which cache results).
* Use config2() to obtain cached versions of the configuration values
for running Auxiliary, Processing, and Reporting modules.
* Get rid of Config() usage in some processing modules.
* Unittests for all of the above, including a unittest which ensures
that Config() is only called twice when processing multiple analyses;
once for reading in processing.conf and once for reporting.conf.
Without these changes, it would be possible to have a dozens (although
for larger analyses this could be much, much more) configuration reads
per processed analysis.
So having a whitespace in front of the line means the data belongs to
the line before it. If this is the first line in a section, then there's
no line before it, and ConfigParser throws a ParsingError exception. We
should just probably write our own .ini-style configuration parser..
Also updates the previously-broken Moloch integration to something that
should work (although it hasn't been tested or unit tested).
Conflicts:
cuckoo/data-private/cwd/conf/qemu.conf
cuckoo/data-private/cwd/conf/reporting.conf
web/analysis/views.py
web/web/settings.py
This is probably a feature on ConfigParser's side to support multi-line
values for certain keys, but it seems to have broken the Cuckoo setup
for at least a handful of people so far. So let's raise an accurate
exception early on to prevent those people from running into related
issues later on at runtime.
In no particular order:
* Sort emitted analysis options by alphabat.
* Allow assignment of analysis options by string and dict.
* Allow assignment of analysis tags by string and tuple/list.
* Further analysis options <> Web Interface options translation.
* Simplify and improve correctness of analysis import functionality.
* Remove plenty of outdated code / templates / etc.
* Attempt to pass analysis options & errors to /filetree.
* Reimplement resubmit & reboot analysis using new submit.
* Low/medium/high priority are {1..3}, not {0..2} in Web Interface.
* Fixed up various "missing variable" templating issues.
* Made /submit/api/submit input & unit tests up-to-date.
* Code refactoring, cleanup, and unit tests.
Following the one-liner to more properly indent & sort dictionary keys
alphabetically for the /submit input JSON data (so that future changes
are more clear than our current changes - which not only made the input
data more up-to-date but also altered some fields).
import json, sys
print json.dumps(json.load(sys.stdin), indent=4, sort_keys=True)
* Required free diskspace from 64 MB to 1 GB.
* Maximum dropped file size from 10 MB to 128 MB.
* Maximum .bson behavioral analysis size from 100 MB to 128 MB.
First work on abstraction layers for MongoDB/GridFS and ElasticSearch
usage, making the code easier on the eye and less complex to unit test.
* Introduce init_once() method for all plugins/modules.
* Use global mongo & elastic instance so only one connection is
required, instead of a connection per processed analysis.
* Configuration migration for MongoDB user authentication.
* Proper configuration support for ElasticSearch's multiple hosts.
* Get rid of "settings.MONGO" & "settings.ELASTIC" usage.
* Include elasticsearch==2.2.0 dependency (to be upgraded by #1196).
* Some unit tests to back up the above, but not full coverage.
* Code cleanup & Screenshots unit test fix.
Note that as per their documentation, both the MongoDB and ElasticSearch
Python API's are thread-safe. Other than that, let's hope the untested
changes are fine :-)
Nex