args_parse zeroed every option's value pointer right before returning, so args_free's NULL check always skipped the free. Even without that, args_free was freeing the address of the caller's static variable (e.g. &ext_vars), not the string it points to.
On Windows (_UNICODE builds) each ARGS_OPT_STRING value is a heap copy from unicode_to_ansi(), so -d, -x, -i, atom-quality-table and module-data args leaked on every run. On non-Windows the value just aliases argv, nothing to free.
Removed the value-clearing loop in args_parse (nothing reads .value after parsing except args_free) and rewrote args_free to walk options->count entries and free the actual string pointers, gated on _UNICODE so it's a no-op elsewhere.
scan_dir() prints "skipping <file> ... because it's larger than N
bytes" unconditionally on both the POSIX and Windows paths, ignoring
ignore_warnings even though every other warning in this file checks
it. Guard both prints the same way the other warnings already are.
Fixes#2137
per POSIX semaphore, sem_init should be paired with sem_destroy (unnamed
semaphore), sem_close should be paired with sem_open (named one).
While glibc seems forgiving about this, this will cause hangs or
segfaults with musl (for example, #1887).
* Added warning on unknown escape sequences
* fixed memory leaks and different output when using regex as operand with strict-escape parameter
* Remove some unwanted characters from strict escaping, add testcases
- Add CLI flag that will disable the console module from printing:
Short: -q
Log: --disable-console-logs
- Update documentation with new CLI flags
Closes#1912
* Make -X print the plaintext.
Make the -X argument print the plaintext along with the xor key.
Output looks like this:
```
wxs@mbp yara % cat rules/xor.yara
rule a {
strings:
$a = "This program cannot"
$b = "This program cannot" xor(1-255)
condition:
any of them
}
wxs@mbp yara % ./yara -s -X rules/xor.yara tests/data/xorwideandascii.out | head -5
a tests/data/xorwideandascii.out
0x4:$a:xor(0x00,This program cannot): This program cannot
0x1c:$b:xor(0x01,This program cannot): Uihr!qsnfs`l!b`oonu
0x34:$b:xor(0x02,This program cannot): Vjkq"rpmepco"acllmv
0x4c:$b:xor(0x03,This program cannot): Wkjp#sqldqbn#`bmmlw
wxs@mbp yara % ./yara -s -X rules/xor.yara tests/data/xorwideandascii.out | tail -5
0x1878:$b:xor(0xfb,This program cannot): \xAF\x93\x92\x88\xDB\x8B\x89\x94\x9C\x89\x9A\x96\xDB\x98\x9A\x95\x95\x94\x8F
0x1891:$b:xor(0xfc,This program cannot): \xA8\x94\x95\x8F\xDC\x8C\x8E\x93\x9B\x8E\x9D\x91\xDC\x9F\x9D\x92\x92\x93\x88
0x18aa:$b:xor(0xfd,This program cannot): \xA9\x95\x94\x8E\xDD\x8D\x8F\x92\x9A\x8F\x9C\x90\xDD\x9E\x9C\x93\x93\x92\x89
0x18c3:$b:xor(0xfe,This program cannot): \xAA\x96\x97\x8D\xDE\x8E\x8C\x91\x99\x8C\x9F\x93\xDE\x9D\x9F\x90\x90\x91\x8A
0x18dc:$b:xor(0xff,This program cannot): \xAB\x97\x96\x8C\xDF\x8F\x8D\x90\x98\x8D\x9E\x92\xDF\x9C\x9E\x91\x91\x90\x8B
wxs@mbp yara %
```
* Update argument description.
* Fix#1851.
This is actually a combination of two bugs. The first bug is that if a string
fits in an atom we never calculated the xor key. The second bug is that if you
specify the xor modifier we were NOT setting the ascii flag on the string, so we
could not calculate the xor key properly one we fixed the first bug.
Tested with:
wxs@mbp yara % cat rules/xor.yara
rule a {
strings:
$c = "This" xor
condition:
any of them
}
wxs@mbp yara % ./yara -s -X rules/xor.yara tests/data/xorwideandascii.out | head -5
a tests/data/xorwideandascii.out
0x4:$c:xor(0x00,This): This
0x1c:$c:xor(0x01,This): Uihr
0x34:$c:xor(0x02,This): Vjkq
0x4c:$c:xor(0x03,This): Wkjp
wxs@mbp yara %
An interesting side effect of this bug is that you could also trigger it with a
"xor wide" combination of modifiers if the string was short enough to fit in an
atom, which is now also fixed.
wxs@mbp yara % cat rules/xor.yara
rule a {
strings:
$c = "Th" xor wide
condition:
any of them
}
wxs@mbp yara % ./yara -s -X rules/xor.yara tests/data/xorwideandascii.out | head -5
a tests/data/xorwideandascii.out
0x18f4:$c:xor(0x00,T\x00h\x00): T\x00h\x00
0x191f:$c:xor(0x01,T\x00h\x00): U\x01i\x01
0x194a:$c:xor(0x02,T\x00h\x00): V\x02j\x02
0x1975:$c:xor(0x03,T\x00h\x00): W\x03k\x03
wxs@mbp yara %
Noticed by @melomac, who provided a nice writeup to help point out that this
only happens on "short" strings (which led me to realize it was only on strings
that fit in an atom).
* Add test case for #1851.
* Print module names.
Add a -M option to the cli which will print the module names to stdout. As the
number of modules has grown it can become confusing to know what modules are
available, so this option will display them on stdout.
NOTE: I'm not sure I like how I implemented this. It seems hackish to expose an
API that just prints things on stdout but it felt even more hackish to walk the
yr_modules_table array from outside of the module code. ;)
Fixes#737.
* Expose access to modules in the API.
This adds a yr_modules_get_table to the libyara API so callers can get access to
the modules. This is currently only used for printing the available module names
in the yara cli. With this change it is now possible to expose this information
in yara-python too.
According to the documentation of
[_setmode](https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/setmode?view=msvc-170#remarks),
stream should be flushed before it is switched to another mode. This was
causing issues when you redirected output of YARA on Windows to a pipe
or a file. stdout is line buffered when terminal is attached but
once the output is redirected, it's usually buffered by a certain buffer
size. That's how systems usually do it.
* XOR Values.
When the xor modifier is used we have not displayed (or even kept) the xor key.
This diff adds a -X option to the CLI that will display the xor key. To do this
I am recording the xor key in _yr_scan_xor_compare() and _yr_scan_xor_wcompare()
and then populating that in the YR_MATCH structure. This way it is available to
the consumers of libyara to handle how they see fit.
I'll be adding support for exposing this in yara-python if this PR is accepted.
* xor key fixes.
Per suggestion from Victor, always display the xor key when -X is specified,
even if the string is not an xor string. This makes it more consistent to parse
with common tools because the number of fields will always be the same.
Also, specify that it is an xor key using "xor(0x01)" format.
* cli/yara: Replace printf with _tprintf in console output callback
This unbreaks output on MSVC-based builds as discussed in #1594
* cli/yara: Fix MSVC warning about incompatible pointers
There is an issue with argument handling that is causing issues when scanning
using certain argument combinations. I believe this is similar to
https://github.com/VirusTotal/yara/pull/927. Here's an example of it broken in
master:
wxs@wxs-mbp yara % cat rules/always_true.yara
rule a { condition: true }
wxs@wxs-mbp yara % ./yara -p 1 rules/always_true.yara /bin
wxs@wxs-mbp yara %
And the same thing with this fix:
wxs@wxs-mbp yara % ./yara -p 1 rules/always_true.yara /bin
a /bin/cat
a /bin/echo
a /bin/launchctl
a /bin/df
<snip>
Issue spotted by: Conor Quigley (https://github.com/schrodyn)
Add a console module which can be used to print things during condition
execution. This is especially useful for debugging complex conditions where
you're following multiple offsets. For example, people sometimes want to express
a PE by doing this:
uint32(uint32(0x3c)) == 0x50450000
What they really want is:
uint32be(uint32(0x3c)) == 0x50450000
Or they want:
uint32(uint32(0x3c)) == 0x00004550
In either case it would be helpful if they could do:
console.hex(uint32(uint32(0x3c)))
Another example of where this could be useful is being able to know what YARA
thinks an imphash is just by doing:
console.log(pe.imphash())
* Make console module a default module.
* Add support for optional string messages.
Override each function so that it can also take an optional string that is
printed at the start of the message. Tested with the following:
import "pe"
import "math"
import "console"
rule a {
condition:
console.hex("0x3c: ", uint32(uint32(0x3c))) and
console.hex(uint32(uint32(0x3c))) and
console.log("String: ", "foo\x00bar\x01baz") and
console.log("foo\x00bar\x01baz") and
console.log("Some integer: ", 987654321) and
console.log(987654321) and
console.log("Some float: ", math.MEAN_BYTES) and
console.log(math.MEAN_BYTES) and
console.log("Imphash: ", pe.imphash()) and
console.log(pe.imphash())
}
* Remove duplicate macro definitions.
* Make console functions use callback.
Add a CALLBACK_MSG_CONSOLE_LOG which the console module uses to send a message
to the main callback. Make the console module functions all generate a single
string which is then passed to the callback. This keeps the callback handler
simple: it just prints a string.
* Fix newlines.
* Add docs for console functionality.
* Add docs for console_callback to yara-python.
* Address comments from Victor.
This is a more comprehensive fix than #1617, it adds new functions to the API for getting/setting uint32 and uint64 settings. Using these functions is preferable over calling `yr_(get|set)_configuration` directly.
A common use case (I've run into this many times, as have some friends) is to
run yara against a file named after the hash of the content. If the hash happens
to start with a number yara.c treats it as a PID, and will fail. The workaround
is to prefix the file with "./" (or whatever the equivalent is on Windows) so
that the strol() check fails.
This diff switches the behavior so it treats the argument as a file first, and
only if that can be opened is it then checked for a PID. Further, to ensure the
PID check is accurate I extended it so that the entire argument must be
converted. If there are any characters left after the conversion then it is not
a valid PID and we will not attempt to scan it as it is (by definition) not a
valid file nor a valid PID.
The warning was caused by the use of strcmp with a string returned by readlink, which doesn't guarantee that the string is null-terminated. In this case the use of strcmp is not required as we are checking only one character.
1. skip all symlinks pointing to . so you do "yara -r test.yar /usr/bin" on debian without -N (/usr/bin/X11 points to . )
2. Show 64 instead of 32 hex chars (regex are shown in full so there's enough clutter anyway)
Includes the following changes:
* Print the rule identifier in warnings due to strings matching too much. The string identifier is not enough, common identifiers like $a are not very informative.
* Use the same format for compile time warnings and scan time warnings.
* --no-warnings disables the "too many matches" warning.
* --fail-on-warnings makes yara fail with the "too many matches" warning.
* Use the same message format in yara and yarac.
When we hit the maximum match limit on a string we currently abort all scanning,
which can be problematic since it isn't a good reason to stop all scanning, and
is better off as a warning to the user.
This commit makes it so that when the string match limit is hit a warning is
printed and scanning will continue. The warning message indicates that the
results may be invalid. It is hard to know for certain if they are invalid
because it depends entirely upon the condition. If you use the # or @ operator
then your condition _MAY_ be invalid since the total number of matches and
locations is not known anymore. Just checking that the string exists won't be an
invalid state since we have confirmed the existence of _AT LEAST_ the maximum
number of occurrences, which clearly satisfies the existence of a single
instance of your string.
I decided to implement this as a callback so the user of libyara has some choice
on the default behavior. If they want to consider this a "stop all scanning"
error they are free to return CALLBACK_ERROR in their callback.
Apparently, the C standard library in older Windows versions does not
contain strtok_s (see discussiion for PR #1381). Since strtok_s is
only used in the single-threaded code of YARA itself, reentrancy is
not really needed.
Cole Munz