Since version 0.3.4 sflock features 64-bit Windows support for magic(5).
Now that we no longer need this compatibility layer (which was
equivalent to the one in sflock) we simply get rid of it.
It was required earlier because Cuckoo wasn't capable of requiring
updated libraries in the pre-Python package era.
Rather than creating the index_*.yar files during "cuckoo -d" and then
loading those when running "cuckoo process", we'll from now on always
enumerate the available Yara rules recursively and load 'em on-the-go.
This simplifies life when updating Yara rules and re-running "cuckoo
process -r". Furthermore this commit adds recursive walking support as
well as following symbolic links.
Currently there's a limitation in the usage of Yara rules, though, as at
the moment it isn't possible to use "include" statements in the rules
(with relative paths, anyway). We may be able to fix that by temporarily
os.chdir()'ing into the yara/category/ directory.
This commit resolves#773 and provides a stepping stone for #776 & #512.
Resolves issue where it is impossible to add a Sample to the database on
64-bit Windows environments as support for magic filetypes is missing in
that environment.
This somewhat works around that, however, unless a package is provided
during submission, in some cases not much will happen during the actual
analysis as the file type guessing won't work and Cuckoo has to rely
solely on the sample file extension.
For unknown reasons nobody ran into this issue before the 2.0.0 release,
however, on a clean Cuckoo setup with newly created configuration, the
machine.options field would always be set to None rather than an empty
string or list, resulting in various exceptions around the Core code.
This commit more improves the handling of this 'special' list type and
ensures the machine.options field will always be at least an empty list.
This counts for both the data retrieved from the configuration as well
as the database handling part.
Thanks to CSIRTUK and pinksawtooth for reporting and doomedraven for
providing an ad hoc solution.
When submitting, e.g., 100 samples as part of just one submit, then the
postsubmit page would make never-ending XHR's. Move away from X
view_task() queries to a singular view_tasks() with X task IDs.
By compiling all of the Yara rules at Cuckoo's startup we avoid the
issue that we've been having up until now: finding out about Yara
problems at runtime. Naturally this simplifies some Yara-related code,
resolves an issue where, in case one of the referenced Yara rules has a
syntax error, no Yara rule matches are performed & get_yara() returns
None (i.e., silently not doing any Yara magic & later on crashing hard
due to the silent thing), and some accompanying unit tests.
* Gets rid of unused action.json log.
* Introduces special handling & error logging for "critical timeout"
issues that occur way too often among our users.
* Add "action" column to the Error table so we can instruct users to
undertake certain actions.
* More properly pass along errors & action information upon analysis
export and correctly handle those when importing an analysis.
* When an action is found, the Web Interface will now show possible
solutions and/or workarounds that a user should follow.
Since we're also using the submit table for reboot analysis, analysis
resubmits, and importing existing analyses it's preferable to make the
submit table a little bit less strict. Also changes an existing database
migration, which is not so nice, but that's fine for now.
In no particular order:
* Sort emitted analysis options by alphabat.
* Allow assignment of analysis options by string and dict.
* Allow assignment of analysis tags by string and tuple/list.
* Further analysis options <> Web Interface options translation.
* Simplify and improve correctness of analysis import functionality.
* Remove plenty of outdated code / templates / etc.
* Attempt to pass analysis options & errors to /filetree.
* Reimplement resubmit & reboot analysis using new submit.
* Low/medium/high priority are {1..3}, not {0..2} in Web Interface.
* Fixed up various "missing variable" templating issues.
* Made /submit/api/submit input & unit tests up-to-date.
* Code refactoring, cleanup, and unit tests.
Following the one-liner to more properly indent & sort dictionary keys
alphabetically for the /submit input JSON data (so that future changes
are more clear than our current changes - which not only made the input
data more up-to-date but also altered some fields).
import json, sys
print json.dumps(json.load(sys.stdin), indent=4, sort_keys=True)
Before the postsubmit page would take a list of task IDs in the URL (as
part of the GET request). Aside from the fact that this is rather ugly,
this also doesn't scale in the sense that URLs are limited to 2-4kb in
length. Fix this by adding a relationship between Task and Submit and
showing that relation in the postsubmit page.
It's quite likely that we're going to require these changes soon enough
with, at the very least, our new build matrix, but most likely also for
other use-cases.
With these changes it's now possible to invoke "cuckoo process -r 1,2,3"
to reprocess analysis 1 up to 3. Similarly one may use a range of tasks
as follows "cuckoo process -r 1-3" to achieve that same goal.
So the database migration for tasks/tags table were missing from our 0.6
release towards the 1.1 release, this doesn't mean, however, that we
should still apply those migrations for later setups (i.e., 1.1 and
upwards). Also note that Cuckoo 1.1 was shipping the incorrect schema
version (the one for 0.6 instead of 1.1) so some shuffling around of
revision nummbers was required.
Reimplements earlier archive-based submission logic on top of the
regular file submission logic. Provides additional unit test and
improves some code here and there as well.
There was a time, before Cuckoo 1.1'ish, that we didn't have the
alembic_version table. This commit ensures that Cuckoo won't
automatically create it and set it to the latest available version
if it's missing - preventing migrations from 0.6 to go through.
Let's no longer do a drop database unit test, as PostgreSQL and MySQL
don't like that so much - and why would we anyway. Also note that we're
not installing PostgreSQL as that's already installed on Travis CI by
default.
Yes, indeed, there is an issue in error handling where errors that are
too long are not logged due to string length constraints in the error
table of our database. To be properly solved with a database migration.
Also reintroduce the capability to provide a custom connection string to
the Database (which is possible now that making the connection is no
longer part of __init__).
Ricardo van Zutphen