Client monitoring artifacts were recently added to honor their
required_permissions field but this was not added to the artifact
verifier.
This PR adds this check and also propagates required_permissions from
sub artifacts to caller artifacts.
Additionally this `artifacts verify` command now fails by default on any
warning.
1. Remove NewArtifactPathManager to Require to artifact mode to be
specified by the called. This avoids possible confusion of the artifact
types.
2. SSH Accessor: Added a hostkey parameter and log the host key when not
provided.
3. Refactor FlowStorageManager to split deletion function into a
DeletionManager.
4. AppendJsonlItem now rejects invalid JSONL lines when it is unable to
add the specified item.
The queue manager is used to pass messages between parts of
Velociraptor. Previously, the queue name corresponded with the artifact
name that defines the queue. However, this allowed messages intended to
client monitoring to be passed to server monitoring artifacts under some
conditions.
This refactor changes the queue name to also contain the artifact type.
This change better keeps track of messages and which queues they are
sent to.
Additionally, the journal manager now records two records in each
monitoring row:
1. The _Source records who authorized the row.
2. The _From records who actually sent the row in the first place.
ACL decisions are now made based on the _Source only making it more
consistent to determine if a row can be sent on a particular queue.
This was recently broken and I am still not sure the pool helps that
much. Lets go back to the simple gRPC client and see if there are
performance issues.
Fixes various quirks and minor bugs uncovered during [Playwright
testing](https://github.com/predictiple/velociraptor/blob/a35d55e10c1e87e47f19f16960c06988c4a52223/gui/velociraptor/TEST-MATRIX.md):
- flows-list: "Show only my hunts"/"Show all hunts" sr-only text now
says "collections" (copy-paste bug)
- notebooks-list: Copy Notebook button's sr-only text was "New Notebook"
- new-notebook: wizard step 1 title was "New Notebook: Configure
Parameters"
- add_user: dialog title had a double space ("Add a new User")
- events: URL-encode artifact names in the route so slash-named
artifacts (Server.Monitor.Health/Prometheus) deep-link correctly;
react-router v5 does not decode params, so decode on read
- notebook: detect same-second cell edits by also comparing
latest_cell_id and cell count (modified_time is second-granularity)
- user-inspector: keep the ACL visible while refreshing after role
changes (no blank/flicker)
- secrets: EditSecretDialog copies the secret prop into state so the
heading shows the secret name
- notebook store: nanosecond-precision version tracking so a delete in
the same second as the last index write is not masked by
GetSharedNotebooks
The post-fix behaviours have been verified against the Playwright test
suite and the relevant [tests updated
accordingly](https://github.com/predictiple/velociraptor/commit/a3c5d2a52e3bb7714e434e9b1e185f60725c1829#diff-a29f2f6162037a2ad0f598a4a7ec5f4130fbe2693ff159ab114237131404d12a).
---------
Co-authored-by: Mike Cohen <[email protected]>
This PR changes the client info manager's Modify() function from taking
a big lock on the entire service (and therefore blocking all clients) to
finer locks on client records. This allows faster updates under load.
## Completions
- Argument names now complete inside unclosed calls (`pslist(`) and
behind trigger dots (`parse_lines(.`), each carrying a `textEdit` so the
editor replaces any typed prefix cleanly.
- `(` triggers argument completion for queries that don't parse yet;
signature help stays silent there so the popups never overlap.
- Positional filtering keeps functions out of `FROM` positions; partial
typing (`parse_lines(fi`) filters args by name instead of matching them
all equally.
## gRPC proxy resilience
- The proxy caches open documents and replays them if the backend loses
state, so frontend restarts no longer silently break completions until
the file is reopened.
- A watchdog pings the backend every 10s; a stuck connection is
hot-swapped for a fresh dial rather than black-holing editor requests.
- Forwarded calls carry a 30s deadline, and the previously missing
shutdown/exit lifecycle handlers mean the language server stops cleanly
instead of being hard-killed after a timeout.
All verified end-to-end with scripted VS Code sessions (keystroke
injection) plus wire-level regression checks.
---------
Co-authored-by: Mike Cohen <[email protected]>
Co-authored-by: Mike Cohen <[email protected]>
The download_transforms option specified a transform the server can take
after downloading the file and before storing in the inventory.
Currently we only support "gunzip" the gzip compressed data will be
unzipped before being stored in the inventory.
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Also:
* Do not override config file in docker container if it already exists
* Use temporary repository for verify() plugin
* Allow user_grant to set the user sub if it is already known.
* Do not allow compiled_collector_args to be set in hunts
* Fixed deadlock in user manager
Velociraptor uses the email address as a unique username identifier.
However, for some IdP implementations, the email address is not unique
and can not be reliably used to identify the user. Instead, these
implementations use the "sub" claim as a unique immutable identifier for
the email address. Some IdP implementations allow the user to switch
email addresses, therefore they are vulnerable to account takeover.
Usually Velociraptor uses the "email_verified" claim but this may not be
reliable enough because:
- Some IdP like Azure ADFS do not use this claim.
- Emails may be verified at one point in time, but then be reassigned
later without the IdP's knowledge.
This PR pins the email used to login with the OIDC "sub" or "oid" claim.
The first time a user logs in we record the "oid" and then if the same
email logs in with a different oid, the login is rejected.
This change stops the same email from being used across different IdP
when using the "multi" authenticator, so it may break some use cases. A
configuration option is provided to disable this pin.
Also ensure it is only checked against the root org. The org_admin
permission is only meaningful when attached to the root org as it allows
the user to perform operations across orgs.
Also:
* Fixed persistence in stacking GUI - This was a regression when the
stack table was reset to the start after showing it again from the main
table.
* Restrict URLs to http and https method.
Version 0.77.1 separates the hunt request from the main hunt object.
This helps avoid expensive operations when the hunt parameters are very
large.
For many operations we do not need to load the entire request - for
example start or stop a hunt. In these case the ModifyHuntObject()
method takes an option that specifies the hunt request should not be
loaded fully, instead only a summary is provided.
The bug was that this summary was synced back to the disk overriding the
full request.
Fixes: #4912
Additionally fix various bugs:
* Use html template for logoff screen
* Escape CVS values that look like formulas to prevent excel from
executing them.
* Enforce DELETE_RESULTS permissions for hunt delete
* Fix crash in tool upload handler
* Fix Dead disk artifact when building the remapping file from a
directory.
* Fixed bug in rsyslog() VQL function.
* Fixed divide by zero crash when downloading certain sparse files from
the GUI
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Recent FontAwesome switched the spinner off when the OS setting
prefers-reduced-motion is set to reduced. This causes the application to
appear broken and unresponsive without the spinner feedback. This PR
forces the spinner to work even if the prefers-reduced-motion is set.
Also this PR fixes the substitution of the custom Generic.Client.Info
from the GUI interrogate screen.
This is needed now that the hunt dispatcher does not automatically
refresh the stats periodically.
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This allows a link to encode a complete collections:
- the artifat to be collected
- the parameters to pass
Also added a button GUI element - this allows notebooks to create clear
buttons as distinct from simple links.
Velociraptor's datastore abstration assumes that datastore objects are
reasonably small so they can be read and written atomically. However,
sometimes, these can become very large. This happens especially for:
1. Artifacts that contain a lot of data internally for example
`Windows.Hayabusa.Rules` contains all the Sigma rules inside the
artifact.
2. Artifacts that can accept very large parameters, for example
`Generic.Detection.Yara.Glob` can include large yara rules as a
parameter.
Both these situations cause the size of the compiled artifact to be
large:
1. The Flow object request to the client becomes very large.
2. Tasts scheduled for the clients are very large
3. Hunt requests become very large.
This makes it worse with a distributed master/minion architecture which
requires the minions to fetch large objects from the master for
scheduling clients. This can sometimes lead to gRPC errors due to the
large requests.
This PR limits the size of the datastore objects to enforce a reasonable
limit. This ensures that performance remains reasonable and prevents
grpc `received message larger than max` errors.
However, the downside is that very large artifacts will not be able to
schedule any more. These artifacts can be redesigned to work more
efficiently:
* For artifacts that include large data blobs internally, they can be
compressed or extracted into a tool (see
https://docs.velociraptor.app/docs/artifacts/tools/ ).
* Artifacts that include large parameters (e.g. Yara rulesets) can be
changed to include a URL to the ruleset which can be fetched by the
client (for example `Generic.Detection.Yara.Glob` has the `YaraUrl`
parameter instead).
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
The request may be very large and storing it in the hunt object may
cause problems. This PR separates out the request into a different
datastore member. Since it is not used that much we can avoid loading it
into memory most of the time keeping the hunt objects very small.
This allows the request to be very large without affecting performance
of flow processing. A large request might include e.g. large Yara rule
sets of complex VQL queries.
Previously these very large collection requests would break the GUI due
to large RPC connections resulting in RPC errors.
This is especially bad in master/minion setup where the minion much
fetch flow objects from the master while processing client data. Making
the request live outside the flow object should make this processing
much faster.
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Velociraptor's shell interface is not a true shell - each interaction
evaluates a commmand in a separate flow, and the shell interface
displays the output in a separate UI cell for each command
This is not how most people want to use the shell. Most users expect the
shell to consist of a "session" where each command builds on the
previous one and they are all visible in the same UI element.
This PR introduces a new concept - relaunching a collection. This
feature allows the user to relaunch an existing collection with the same
flow id as a previous collection. The client will run the new collection
and the results will be appended to the old collection.
When used in the shell, this allows each command to be appended to the
previous ones inside a flow. This keeps all shell commands in the same
flow. The new UI shows them all as belonging to the same session.
Currently there is no client side state kept so it is not a real
session, but this is the first step.
Also Update evtx parser to support preferred message language
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This can now be used as needed using a VQL query but it should normally
not be needed.
The housekeeping thread generates a lot of IO and takes a long time. It
is only used to tally up the hunt stats in the overview page so it is
not super important. Normally the hunt stats should be accurate but if
the server is hard exited or crashes before it is able to flush data out
to disk, the totals may be out.
Also change the code to ensure that totals are always incremental - this
avoids an issue where the totals can be reset during the hunt recuitment
stage (when a lot of clients are added quickly). This was observed in
#4799 where a race condition was observed - by the time the flows were
calculated from disk, the totals were much higher so then when the stats
were updated they appeared to go backwards.
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This refactor tags each event with the caller's ID. This enabled
listeners to determine who sent the message and decide if they trust
it based on that.
This avoids situations where a user can send an event to a privileged
queue which normally only expects messages from the server itself.
* More efficient discovery of artifact types
* More rigorous checks for forwarding messages from the source client
to the type of artifacts allowed.
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This allows artifacts to be collected remotely and their results
fetched over the API. For example:
```
velociraptor --api_config api.config.yaml -r Server.Utils.CreateMSI -o /tmp/test.zip
```
This can also work on client collections:
```
velociraptor -v --api_config api.config.yaml -r Generic.Client.Info -o /tmp/test.zip --client_id C.c892ae5478a6cda2
```
Additionally added the `artifacts fetch` command which allows remote
artifacts to be fetched:
```
velociraptor -v --api_config api.config.yaml artifacts fetch --client_id C.c892ae5478a6cda2
--flow_id F.D6JCVA49O524O --output /tmp/test.zip
Also various bugfixes:
- Support arbitrary format of org id (Fixes#4720)
- Fixed bug in uploader in the main dashboard
- Filtering column in timeline does not produce filter indicators like
in normal tables (Fixes#4743)
Most of the time people want the OIDC automated roles to be able to
remove access from users, which was not the way it was implemented
previously (roles were set by OIDC as the minimum role but users could
be given higher roles).
This PR also adds claims from the ID token which can be used on Azure to
add custom claims in addition to te access token.
There are a couple of very large libraries with huge API surface, making
the binary extremely large:
1. The Elastic library is now handled via a fork to isolate just the
Bulk upload API - reducing the binary size by 16mb
2. The next largest library is the AWS client library used by
s3_upload() increasing the binary size by 6mb.
3. The Google cloud client library is also huge at around 5mb
This PR introduces a new build tag `sumo` which includes these large
libraries. However for regular builds:
* Use the minio S3 library to connect to AWS - this library is much much
smaller and easier to use.
* Remove Google cloud dependencies: pubsub is a rarely used feature, and
with Google cloud storage (GCS) we can always enable S3 compatible mode
so there is no real need for specific GCS access.
* Remove the starlark() plugin since it is not really very often used.
By enabling `sumo` build tags (i.e. `make linux_sumo`) it is possible to
build a larger binary with the full AWS and Google client libraries.
Also added a CI test to ensure Version is kept in sync with plugin arg
modifications.
Also removed old config options from config file in order to trigger
syntax errors if people try to use them.
This shows the last time a user was seen and the remote IP they were
seen from. This feature helps to spot unused user accounts so they can
be removed.
The information is also available in gui_users() plugin which allows
management via VQL.
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: snyk-bot <[email protected]>
Sometimes when the server crashes or due to various reasons, the hunt
stats become out of step with the actual data. This shows up as a 0 in
scheduled clients, or the numbers are not quite right.
This PR performs occasional sweeps over the hunts to fix up the stats
from the flows themselves. This allows us to keep high level stats like:
- Errored clients
- Total bytes/rows transferred
- Scheduled clients
etc.
We now use the same code for forwarding logs to syslog:
1. Syslog now uses octet counting transport
2. Syslog log forwarding now works on Windows as well.
The new process tracker is more efficient:
1. Process Children are stored in the tracked process entry avoiding the
need to sweep across the entire tracker when constructing the process
tree
2. The tracker now uses links to maintain correspondence between process
ID (which is heavily reused) and internal unique ID (based on the start
time). This avoids the need to sweep through the tracker when we detect
a PID was reused.
3. The tracker can use an on disk sqlite based LRU which is a bit slower
but reduces memory use and allows tracking much larger number of
processes.
Mike Cohen