While the pool client is designed for _server_ load testing, it has
shown itself to be useful for testing/learning/development scenarios too
(for example, testing hunt behaviours).
This PR preserves the load-testing aspects but adds some features that
make the pool client more human-friendly and therefore more useful for
lab and learning environments.
Each pool client now presents a unique hostname and FQDN (e.g.
"quantum-phoenix.example.com") to the server instead of the real host's
hostname suffixed with the client id. The FQDN reuses the domain of the
host running the pool client so the emulated clients appear to belong to
the same network. Hostnames are adjective-noun combinations (e.g.
"quantum-phoenix") - a numeric suffix is only appended when the number
of clients is almost ten thousand. Name collisions are possible if
running pool clients simultaneously on multiple hosts connected to the
same server, but the server is designed to expect name collisions in the
real world, so it's not an issue.
Identities are generated upfront for all clients and applied to the
`BasicInformation` and `DetailedInfo` sources of the
`Generic.Client.Info` interrogation, as well as the periodic client info
messages.
Writebacks now default to in-memory only (`memory:// scheme`) so no
writeback files are written to disk and each client's identity is
ephemeral. This makes it easy to spin up clients -> run some tests ->
stop the clients -> delete the data -> repeat, without needing to go
clean up hundreds of writeback files each time. This also makes it easy
to create massive amounts of hunt data in batches (create hunt -> add
10,000 clients -> hunt collections run -> stop clients -> add 10,000 new
clients -> hunt collections run -> stop clients -> repeat) for
server-side query performance testing.
The writeback location is now configured for all platforms - previously
Darwin was left unset and fell back to the config file's path, causing
all clients on macOS to share a single writeback file. The root config
loader no longer requires a writeback file, but it is still available if
you need persistent pool client identities.
Screenshots:
<img width="1600" height="800" alt="pool_client"
src="https://github.com/user-attachments/assets/53d14917-4252-4924-857c-acee5c6708eb"
/>
<img width="1200" height="637" alt="pool_client2"
src="https://github.com/user-attachments/assets/c42300ec-4869-465b-b4d6-2141dcbca6bd"
/>
The pool client's hostname transform silently no-oped for newer clients
because the launcher compresses VQL responses by default
(FlowRequest_ZLIB). The payload lives in CompressedJsonResponse while
JSONLResponse is empty, so the transform parsed an empty string and
returned the response unchanged.
Decompress the payload before transforming and re-compress the result,
preserving the original compression state.
History:
- The bug has existed since July 2025, when an unrelated filestore
change (#4361) made compressed flow requests the default: the pool
client's hostname transform, written in 2023 when responses were
uncompressed, silently stopped working because it only read
`JSONLResponse` while the payload now lived in `CompressedJsonResponse`.
The failure was completely invisible - parsing the empty string returns
no error, so the transform passed every response through unchanged
without a single log line - and the symptom (all virtual clients showing
the same hostname) was indistinguishable from expected behavior, since
the pool client's entire premise is many simulated clients running on
one real host. Combined with the fact that the pool client is a
load-testing tool whose operators watch server metrics rather than
client hostnames, the `-<id>` suffix was a purely cosmetic nicety whose
absence looked normal, and the only alternative hostname path (the
periodic `GetClientInfo()` update) was disabled by default, so nothing
ever surfaced the discrepancy.
It surfaced now because I'm experimenting with giving pool clients
unique hostnames, so that the feature can also be useful in training and
development environments 😎
Also ensure it is only checked against the root org. The org_admin
permission is only meaningful when attached to the root org as it allows
the user to perform operations across orgs.
Also:
* Fixed persistence in stacking GUI - This was a regression when the
stack table was reset to the start after showing it again from the main
table.
* Restrict URLs to http and https method.
Recent FontAwesome switched the spinner off when the OS setting
prefers-reduced-motion is set to reduced. This causes the application to
appear broken and unresponsive without the spinner feedback. This PR
forces the spinner to work even if the prefers-reduced-motion is set.
Also this PR fixes the substitution of the custom Generic.Client.Info
from the GUI interrogate screen.
This is needed now that the hunt dispatcher does not automatically
refresh the stats periodically.
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Also some minor bug fixes:
- ACE editor: disable web workers to prevent invalid http request
(#4006)
- Fixed some bugs in artifacts: Windows.Forensics.Prefetch and
Windows.System.TaskScheduler
- Offline collector now uses Hostname in default name rather than FQDN.
Added profile tracker for nanny and ring buffers.
Added fault injector to emulate stuck http connections - these seem to
be present sometimes after suspend.
Nanny has a mechanism now to warn clients they are out of compliance and
allow them an opportunity to rectify the issue. This is now used in the
send and receive HTTP connections to abort the current connection and
restart it.
This allows us to validate offline spec files and reject invalid files.
Also added an upload to the Server.Utils.CreateCollector that contains
the spec file for ease of use. Users now simply can download that spec
file and use it on the CLI builder.
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
1. When verifying artifacts, reported error line number now accounts
for the yaml structure.
2. Verify new case: Defining functions in LET and calling them with
incorrect parameters.
3. Identify calling undefined plugins or functions (previously this
was not a warning because it could have been defined by LET
definitions).
4. Resolving import and export sections when verifying artifacts now
warn about misuse of imported functions which may have changed
their definitions.
5. Uploads are now reported as Dicts instead of an upload object.
6. Dict sorting now implemented.
7. Lots of artifact bugs found by previous static analysis
8. Updated to go 1.24.0
---------
Co-authored-by: snyk-bot <[email protected]>
This PR introduces resumable uploads. These uploads are opt-in set by
setting a VQL variable.
When the variable is set, uploads will complete immediately:
- Schedule an upload transaction and inform the server
- VQL queries return an upload id immediately and the VQL query
continues (ie uploads are asynchronous).
- Begin uploading transactions on the client
If the client is interrupted or crashes, the transactions can be resumed
on the server.
Prevously the flow state was set to error as soon as a log was issued at
level ERROR however the query continued running until the end. This
caused the GUI to disable the cancel button so it was impossible to
cancel the flow.
This PR Keeps the flow at the IN_PROGRESS state until completion while
tagging the error message in the flow. This allows the flow to be
cancelled after the error by the GUI. After completion the flow will be
switched to the ERROR state.
Also:
1. Fix Registry accessor parsing of MULTI_SZ fields.
2. Fix Generic.Client.Profile artifact
3. Refactor tests to use the global mock clock
The client will fail a collection when there is a log message at ERROR
level. Previously this tagged the collection as completed (with an
error). This means no further status was sent to the server. However,
the VQL continues running and sending back results.
This leaves the collection in an inconsistent state - the total count
of rows is incorrect, as more rows are still actually collected by the
client. The GUI may not show the correct results collected.
This PR does not consider an error to be represent the completion of
the collection - stat updates continue to be sent until the collection
is really finished.
---------
Co-authored-by: snyk-bot <[email protected]>
1. Throttler is now a global service making checking the cpu utilization
more efficient because the same information can be shared between all
queries.
2. Added a tracker for the CPU throttler so we can see which queries are
being blocked.
3. Made checking for cpu load more efficient - on windows retain the
handle across calls avoiding repeated open process calls. On linux issue
the times() system call instead of parsing /proc/
4. Evaluate limites lazily, if no queries care about iops we dont look
at it.
This helps for very large notebooks with many cells. Previous behavior
was to load all the cells immediately but this can increase load on the
server and cause the connections to timeout.
This change allows cells to be loaded only when they are scrolled into
the view port.
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: snyk-bot <[email protected]>
The responses were not forwarded causing the server to constantly
query about the state of the flows.
Also added debug view into directly connected clients. Added tags to
timeouts so we can identify the cause of timeout cancellations better.
This is a workaround to the introduction of GetTempPath2 in Windows. The
Golang standard library switched to this API in recent Go version
https://github.com/golang/go/issues/56899
The GetTempPath2 API ignores the TEMP environment variable and always
returns C:\Windows\SystemTemp for system level programs. This causes the
os.TempDir() API to ignore our configuration of the correct temp
location.
Since practically, we can not allow list the SystemTemp directory in
e.g. EDR or other security products, this bug forces Velociraptor to
write temp files in this common area ignoring the configuration file.
To work around this issue we wrap all calls to temporary files and
ignore the os.TempDir() API completely.
Currently flows can be in the running, completed or errored states. This
is not enough granularity leaving users wondering what is happening to
their flows in case the client crashes or reboots for some reason, or
may become unresponsive. In such cases sometimes flows remain in the
running state so it is not easy to retry them.
This PR adds a server side mechanism to actively check up on the
progress of the flows. This ensures that we can keep track of running
flows better and terminate them when the client reboots. Flows now move
through more fine grained states:
1. RUNNING or Scheduled - in this state the flow is scheduled for the
client which may not be online.
2. WAITING - The flow is waiting on the client to run - usually clients
have a concurrency limit which blocks too many queries from running at
the same time. Additional queries will be blocked in the WAITING state
until they can begin running.
3. IN_PROGRESS - this state indicates that the query is running on the
client. Periodic status updates will be sent to the server to make sure
the query is still alive.
4. UNRESPONSIVE - if no periodic updates arrived for the query for a
while, the flow goes into the UNRESPONSIVE state. This could happen if
the client crashed or disappeared
5. FINISHED - This indicates the flow is completes successfully.
6. ERROR - The flow is in error.
If a flow has not been seen for a while, and the client connects again,
the server will now send a special request to the client asking it to
update the status of those in flight flows. The flows will go from the
UNRESPONSIVE state to the IN_PROGRESS state if the flow is still
running. But in the more common case where the client restarted and lot
progress on the flow, the client will send an ERROR message to the flow
causing the in flight flow to be properly closed off with an error.
This is a more robust mechanism which should ensure that flows are not
left in the running state without being finalized one way or the other.
The current parsing breaks if there are any options present in the
authorized_keys file. Options are not too uncommon, especially in
systems that restricts sessions with "command", "restrict", "no-pty"
etc. The new regex parser accepts any options, with optional arguments,
also with quoted space support. The options are included in the output
in the new column "Opts".
The new parser also allows key types other than just "*ssh*". *man 8
sshd*
lists the following valid formats:
- [email protected]
- ecdsa-sha2-nistp256
- ecdsa-sha2-nistp384
- ecdsa-sha2-nistp521
- [email protected]
- ssh-ed25519
- ssh-dss
- ssh-rsa
In particular, ssh-ed25519 is a popular modern alternative to ssh-rsa.
In addition, tests are added. **HOWEVER**, they are not fully
functional. I've added an .out file that matches the expected results,
but the test itself produces a lot of duplicate results. I'm probably
messing up the output with my mocking (first time I'm using it). Help
needed!
---------
Co-authored-by: Mike Cohen <[email protected]>
Since WSS connections are never re-established the nanny will timeout if
no messages were sent within the time (by default 10 min). This will
cause a hard exit.
---------
Co-authored-by: rob-wilco <[email protected]>
1. Add release binaries for Linux/arm64
2. Have Server.Utils.ImportCollection trigger System.Flow.Completion
3. Overuse of red color when no errors occur
4. Deprecate flag --config_override
5. Start Velociraptor service after ClientSysVRPM install
6. Boolean artifact parameters are not checked when their default state
is true
Fixes: #2811, #2751, #2742, #2721, #2792, #2798
The gopsutils library shells out to various programs (like ps, lsof)
under some situations. This makes it unpredictable how expensive an
operation is.
This PR wraps gopsutils carefully ensuring that no expensive operations
occur.
Fixes: #2822
By including custom event artifacts in the config it is possible to
implement client initialization actions that can take place even before
the client contacts the server.
This PR creates a checkpoint before running a flow that will be removed
when the flow is completed. If the client crashes while collecting the
flow, the client will send a message to the server once it starts to
ensure that the server can record the crash.
Updated Windows.Search.Yara to use new OSPath
Also updated log() function to deduplicate multiple messages at the
same time. Each message will be throttled in time separately so it is
now easy to make progress logs for large queries.
Event monitoing and client side flow manager were globals for the
client which caused problems when running multiple clients in the same
process (e.g. `velociraptor gui` command)
The refactor creates separate instaces that live within the executor
allowing multiple different instances to exist simultaneously.
* Also reduced permission requirements for manipulating client event
tables to COLLECT_CLIENT (normally given to the investigator role).
A trace file is a zip file containing important trace information
about the execution of the client. This PR allows users to easily
trace a collection by collecting and uploading a trace file every few
seconds to the server.
Now FlowStats are sent for all queries even after they are
teminated. This makes it much easier to track which query terminated,
and what the status is. The ArtifactCollectorContext is now made up
from two parts:
1. A list of QueryStatus items one for each query (including stats for
each query and final status code).
2. Derived fields from the above which represent the collection wide
totals.
Added more tests.
Server Artifact service is now refactored to use the same
FlowRequest/FlowStats messages as the client.
This message combines all collections in the same flow into a single
message, reducing the total number of task files in message
queues. This should help to reduce IO operations and increase
throughput
predictiple