Problem: When Windows event logs are cleared or deleted,
Windows.EventLogs.Cleared can show that a log was cleared, but the
events themselves are lost because parse_evtx() only reads intact log
files. The old data usually stays on disk, in unallocated space, the
pagefile or memory, until it is overwritten.
Solution:
This PR adds a new carve_evtx() VQL plugin. It scans any file or device
for EVTX chunk signatures at any offset and parses each 64KB chunk on
its own.
- Rows have the same structure as parse_evtx(), plus ChunkOffset,
ChunkHeaderValid, ChunkDataValid, ChunkTruncated and ChunkCompressed.
- NTFS-compressed chunks (the Windows default for event logs) are
decompressed.
- Damaged chunks are flagged, not dropped.
- With output=, the recovered events are also written to a new .evtx
file that Event Viewer can open.
Example:
SELECT * FROM carve_evtx(filename='''\\.\C:''', accessor="raw_file")
WHERE System.Channel = "Application"
Known limitation
Old records in chunk slack are not recovered yet. This could be a
follow-up.
---------
Co-authored-by: Mike Cohen <[email protected]>
1. Remove NewArtifactPathManager to Require to artifact mode to be
specified by the called. This avoids possible confusion of the artifact
types.
2. SSH Accessor: Added a hostkey parameter and log the host key when not
provided.
3. Refactor FlowStorageManager to split deletion function into a
DeletionManager.
4. AppendJsonlItem now rejects invalid JSONL lines when it is unable to
add the specified item.
The queue manager is used to pass messages between parts of
Velociraptor. Previously, the queue name corresponded with the artifact
name that defines the queue. However, this allowed messages intended to
client monitoring to be passed to server monitoring artifacts under some
conditions.
This refactor changes the queue name to also contain the artifact type.
This change better keeps track of messages and which queues they are
sent to.
Additionally, the journal manager now records two records in each
monitoring row:
1. The _Source records who authorized the row.
2. The _From records who actually sent the row in the first place.
ACL decisions are now made based on the _Source only making it more
consistent to determine if a row can be sent on a particular queue.
This PR changes the build toolchains to use the hermetic mingw clang
toolchains so we do not depend on the OS installed ones.
The build will now automatically download and unpack the toolchain
inside the build directory and use it for reproducible builds.
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Adds commandline params to create .rpm or .deb setup scripts with custom user/group
``` bash
./velociraptor-v0.77.2-linux-amd64 \
--config server.config.yaml debian server \
--binary velociraptor-v0.77.2-linux-amd64 \
"--server_user" myExistingUser "--server_group" myExistingGroup
```
* defaults unchanged: still executing as velociraptor:velociraptor when
not supplying --server_user
* supplying only --server_user will set the group to be the same as the
user (user:user)
* supplying only --server_group will error out
* supplying both --server_user and --server_group allows to set
different values
# Why
Most companies define their user-naming-scheme and might even have
service users automatically set up. A default
`velociraptor:velociraptor` breaks this contract, and will have to be
adapted after installing the .deb or .rpm
---------
Co-authored-by: Mike Cohen <[email protected]>
Co-authored-by: evilele <[email protected]>
The download_transforms option specified a transform the server can take
after downloading the file and before storing in the inventory.
Currently we only support "gunzip" the gzip compressed data will be
unzipped before being stored in the inventory.
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Client monitoring artifacts with required permissions were not
enforced. This PR ensures the user that calls
SetClientMonitoringState() has all the required permissions for each
artifact in the client monitoring set.
Also:
* Prevent template escapes when restoring notebooks from backups.
* Ensure a TTL is set for any use of ttlcache - due to a bug in
ttlcache not setting a TTL causes expiry of most recent rather than
least recently use.
When the context is cancelled, the syslog watcher when given a query
would spin on it instead of exiting the loop. This PR fixes this
condition detecting when the context is cancelled and breaking out of
the loop.
Also fixes:
- Crashes for malformed NTFS images
- Crashes for binary parser with negative index.
- Artifacts with quick_links did not have default notebook built.
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Version 0.77.1 separates the hunt request from the main hunt object.
This helps avoid expensive operations when the hunt parameters are very
large.
For many operations we do not need to load the entire request - for
example start or stop a hunt. In these case the ModifyHuntObject()
method takes an option that specifies the hunt request should not be
loaded fully, instead only a summary is provided.
The bug was that this summary was synced back to the disk overriding the
full request.
Fixes: #4912
Additionally fix various bugs:
* Use html template for logoff screen
* Escape CVS values that look like formulas to prevent excel from
executing them.
* Enforce DELETE_RESULTS permissions for hunt delete
* Fix crash in tool upload handler
* Fix Dead disk artifact when building the remapping file from a
directory.
* Fixed bug in rsyslog() VQL function.
* Fixed divide by zero crash when downloading certain sparse files from
the GUI
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Some bug fixes for the GUI
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This allows the request to be very large without affecting performance
of flow processing. A large request might include e.g. large Yara rule
sets of complex VQL queries.
Previously these very large collection requests would break the GUI due
to large RPC connections resulting in RPC errors.
This is especially bad in master/minion setup where the minion much
fetch flow objects from the master while processing client data. Making
the request live outside the flow object should make this processing
much faster.
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Implemented persistent Bash, Powershell, CMD and VQL shells
Added full screen and transcript download. The transcript allows the
user to download a textual transcript of the
interactive session.
Also Update evtx parser to support preferred message language
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This can now be used as needed using a VQL query but it should normally
not be needed.
The housekeeping thread generates a lot of IO and takes a long time. It
is only used to tally up the hunt stats in the overview page so it is
not super important. Normally the hunt stats should be accurate but if
the server is hard exited or crashes before it is able to flush data out
to disk, the totals may be out.
Also change the code to ensure that totals are always incremental - this
avoids an issue where the totals can be reset during the hunt recuitment
stage (when a lot of clients are added quickly). This was observed in
#4799 where a race condition was observed - by the time the flows were
calculated from disk, the totals were much higher so then when the stats
were updated they appeared to go backwards.
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Also various bugfixes:
- Support arbitrary format of org id (Fixes#4720)
- Fixed bug in uploader in the main dashboard
- Filtering column in timeline does not produce filter indicators like
in normal tables (Fixes#4743)
We now use the same code for forwarding logs to syslog:
1. Syslog now uses octet counting transport
2. Syslog log forwarding now works on Windows as well.
The new process tracker is more efficient:
1. Process Children are stored in the tracked process entry avoiding the
need to sweep across the entire tracker when constructing the process
tree
2. The tracker now uses links to maintain correspondence between process
ID (which is heavily reused) and internal unique ID (based on the start
time). This avoids the need to sweep through the tracker when we detect
a PID was reused.
3. The tracker can use an on disk sqlite based LRU which is a bit slower
but reduces memory use and allows tracking much larger number of
processes.
Many CLI commands are actually just wrappers around artifact
collections. As we move towards more powerful artifacts it is possible
to convert more and more functionality to simple artifacts.
However currently the `artifact collect` command is a bit clumsy and
more difficult to use.
This PR presents an alternate CLI parser which converts into the
standard `artifact collect` command but it is simpler to use.
The genreral format is:
```
velociraptor[<common flags> ...] -r Windows.Triage.Targets [<artifact params> ...]
```
Where the common flags are standard args to velociraptor (e.g. output,
debug etc).
When the new parser sees the `-r` flag, it interprets the next token as
the name of an artifact to collect. All flags after that are interpreted
as parameter values.
For example:
```
velociraptor.exe -v --config server.config.yaml
--output test.zip -r Windows.Triage.Targets --HighLevelTargets _Live
```
To view the help you may add -h or --help after the -r flag to print a
details help message about the artifact.
```
velociraptor.exe -v --config server.config.yaml
--output test.zip -r Windows.Triage.Targets -h
```
The help output lists all artifact parameters and their defaults and
types.
Note that for custom artifacts and many server artifacts, the config
parameter is required so we can load the artifacts before collecting
them. These flags must be provided before the `-r` flag for common
flags.
Additionally for `multichoice` parameters which normally expect a json
encoded list, we now also support command separated values as these are
easier to type on the Windows shell.
With allowed_file_accessor_prefix and denied_file_accessor_prefix
together it is possible to mask off different areas of the filesystem
with sufficient flexibility.
Previously the interactions were not correct causing more files to be
rejected than needed.
This allows automatic backup of global notebooks.
Only the cell queries are backed up not the output. Notebooks will need
to be recalculated when restored.
On Windows the filesystems are case insensitive so the upload() plugin
may encounter various versions of the same filename. This PR takes it
into account and deduplicates based on the canonical lower cased name.
Also some minor bug fixes:
- ACE editor: disable web workers to prevent invalid http request
(#4006)
- Fixed some bugs in artifacts: Windows.Forensics.Prefetch and
Windows.System.TaskScheduler
- Offline collector now uses Hostname in default name rather than FQDN.
Added profile tracker for nanny and ring buffers.
Added fault injector to emulate stuck http connections - these seem to
be present sometimes after suspend.
Nanny has a mechanism now to warn clients they are out of compliance and
allow them an opportunity to rectify the issue. This is now used in the
send and receive HTTP connections to abort the current connection and
restart it.
This allows us to validate offline spec files and reject invalid files.
Also added an upload to the Server.Utils.CreateCollector that contains
the spec file for ease of use. Users now simply can download that spec
file and use it on the CLI builder.
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
When the disk fills up, the server switches to read only data store and
returns a 503 error to the clients. This bug caused the clients to retry
too quickly causing a lot of un-necessary network traffic.
1. When verifying artifacts, reported error line number now accounts
for the yaml structure.
2. Verify new case: Defining functions in LET and calling them with
incorrect parameters.
3. Identify calling undefined plugins or functions (previously this
was not a warning because it could have been defined by LET
definitions).
4. Resolving import and export sections when verifying artifacts now
warn about misuse of imported functions which may have changed
their definitions.
5. Uploads are now reported as Dicts instead of an upload object.
6. Dict sorting now implemented.
7. Lots of artifact bugs found by previous static analysis
8. Updated to go 1.24.0
---------
Co-authored-by: snyk-bot <[email protected]>
Custom deletion code for result sets, bulk files and better testing
ensure we do not leave files behind when deleting flows.
Also pass standard http client (with proxy support) to azure_upload()
and s3_upload()
This allows execve to be used with a secret specifying the prefix
command line.
Combined with security.vql_must_use_secrets this mechanism creates a way
to lock down execve() on the server to only be able to run certain
approved programs controlled by a secret.
The PR refactors the accessor API to add a descriptor for each accessor,
like vql plugins and functions. This allows a more structured approach
to documentation.
Additionally, the descriptor can specify the type of permissions
required to use the accessor, just like plugins. Some accessrs are
configured through scope variables, and this now adds a self documenting
args descriptor just like plugins.
This PR also introduces a new permission NETWORK. This permission
declares that the user is able to access the network and it is enforced
by various plugins/functions and accessors that access the network. This
additional permission provide finer grained control over user activity.
On the server, VQL can access the entire filesystem, either with the
"file" accessor or the "fs" accessor. This allows a VQL query in the
notebook to read any internal files in the file store as well as other
sensitive files on the server
This PR restricts the file and fs accessors into a set of allowable,
preconfigured directories. Additionally VQL plugins that are allowed to
write to the filesystem also honor this restriction.
This PR also introduces a Data Encryption Key for secrets. This ensures
that even if the secret files are exposed the secrets themselves can not
be unlocked without access to the DEK.
The DEK can be specified in the config file or in an environment
variable.
This PR introduces resumable uploads. These uploads are opt-in set by
setting a VQL variable.
When the variable is set, uploads will complete immediately:
- Schedule an upload transaction and inform the server
- VQL queries return an upload id immediately and the VQL query
continues (ie uploads are asynchronous).
- Begin uploading transactions on the client
If the client is interrupted or crashes, the transactions can be resumed
on the server.
One of the common issues with Velociraptor is managing disk space. Up to
now the server would store all files in uncompressed clear text form.
While this is handy because all data is just a simple file, it is
wasteful of disk space and often causes disk issues.
This PR allows the data to be stored in a compressed form on the disk:
1. Clients are requested to compress they payload (either VQLResponse
which is the json output from a VQL query, or FileBuffer which is bulk
file data).
2. The server simply writes the already compressed data in chunks on the
disk.
This allows the server to offload compression to the clients, and
actually allows the server to go much faster due to reduced IO, by
simply writing already compressed chunks.
When reading the data back out, the server is able to transparently
decompress the data using the "fs" accessor. This allows compression to
be handled transparently.
Some metrics:
- A Windows.NTFS.MFT JSON output is 960Mb uncompressed and 80mb
compressed. JSON is highly compressible and this helps a lot.
