This is needed now that the hunt dispatcher does not automatically
refresh the stats periodically.
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Also added a CI test to ensure Version is kept in sync with plugin arg
modifications.
Also removed old config options from config file in order to trigger
syntax errors if people try to use them.
The new process tracker is more efficient:
1. Process Children are stored in the tracked process entry avoiding the
need to sweep across the entire tracker when constructing the process
tree
2. The tracker now uses links to maintain correspondence between process
ID (which is heavily reused) and internal unique ID (based on the start
time). This avoids the need to sweep through the tracker when we detect
a PID was reused.
3. The tracker can use an on disk sqlite based LRU which is a bit slower
but reduces memory use and allows tracking much larger number of
processes.
This allows us to validate offline spec files and reject invalid files.
Also added an upload to the Server.Utils.CreateCollector that contains
the spec file for ease of use. Users now simply can download that spec
file and use it on the CLI builder.
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
1. When verifying artifacts, reported error line number now accounts
for the yaml structure.
2. Verify new case: Defining functions in LET and calling them with
incorrect parameters.
3. Identify calling undefined plugins or functions (previously this
was not a warning because it could have been defined by LET
definitions).
4. Resolving import and export sections when verifying artifacts now
warn about misuse of imported functions which may have changed
their definitions.
5. Uploads are now reported as Dicts instead of an upload object.
6. Dict sorting now implemented.
7. Lots of artifact bugs found by previous static analysis
8. Updated to go 1.24.0
---------
Co-authored-by: snyk-bot <[email protected]>
Also added:
* Tests for Various ShellBags
* Delve scripts to make debugging better - support Velociraptor specific
pretty printer.
* Update Linux.Network.Netstat to support IPv6 sockets.
Previously we used artifact name prefix to distinguish and group
artifacts. This is useful for broad artifacts like `Windows`, `Linux`
etc but proved to be problematic for `Exchange` artifacts.
If an arbitrary name prefix is used when importing exchange artifacts,
artifacts that import other artifacts may break due to the name
changing.
This PR introduces a "Tag" on artifacts. Tags allow grouping artifacts
into distinct groups. The artifacts viewer allows searching for tags
automatically (unique tags are added to the end of the search selector).
Tags may be attached to the artifact using artifact_set_metadata() or
directly as part of artifact_set().
This PR also adds `Server.Import.Extras` artifact to replace the
distinct import artifacts from the various semi-official sources. All
external sources are now imported using this artifact.
This means we can also remove these artifacts from being built in:
* Generic.Collectors.SQLECmd
* Generic.Forensic.SQLiteHunter
* Server.Import.CuratedSigma
* Server.Import.Rapid7Labs
* Windows.KapeFiles.Targets
---------
Co-authored-by: snyk-bot <[email protected]>
This PR introduces resumable uploads. These uploads are opt-in set by
setting a VQL variable.
When the variable is set, uploads will complete immediately:
- Schedule an upload transaction and inform the server
- VQL queries return an upload id immediately and the VQL query
continues (ie uploads are asynchronous).
- Begin uploading transactions on the client
If the client is interrupted or crashes, the transactions can be resumed
on the server.
This accomodates complex routing configurations which allow base path to
depend on arbitrary things (like org name).
Also:
* fixed copy notebook cell bug
* revert gorilla csrf to unbroken version
* remove unused pst artifacts
* Added audit log on jwt exchange
* Allow markdown block quotes to be rendered in artifact descriptions
* Add tool tips to column headers
* Move the default "LIMIT 50" vql into a config option
---------
Co-authored-by: snyk-bot <[email protected]>
Fixed spelling in name and metadata
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Matthew Green <[email protected]>
Co-authored-by: Mike Cohen <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: snyk-bot <[email protected]>
* Reflect user preference timezone
* Shows times in RFC3339 format
* Enter time in RFC3339 format
* Dropdown calendar
* Quick settings - next week, prev week etc.
This is a workaround to the introduction of GetTempPath2 in Windows. The
Golang standard library switched to this API in recent Go version
https://github.com/golang/go/issues/56899
The GetTempPath2 API ignores the TEMP environment variable and always
returns C:\Windows\SystemTemp for system level programs. This causes the
os.TempDir() API to ignore our configuration of the correct temp
location.
Since practically, we can not allow list the SystemTemp directory in
e.g. EDR or other security products, this bug forces Velociraptor to
write temp files in this common area ignoring the configuration file.
To work around this issue we wrap all calls to temporary files and
ignore the os.TempDir() API completely.
The Windows.KapeFiles.Targets artifact allows to collect many bulk
forensic artifacts like registry hives etc. People often use it to
collect offline collections for preservation of hosts.
Although best practice is to **also** collect parsing artifacts as the
same time, sometimes this is left out. In this case the user needs to
parse the collected raw files (for example collecting $MFT then needing
to apply `Windows.NTFS.MFT` to parse it). This is not always easy to do.
This PR adds a notebook suggestion which applies remapping on the
KapeFiles collection in such as way that some regular artifacts designed
to run on the live system can work to some extent off the raw
collection. The additional parsing will be added to new collections in
the same (possibly virtual) client.
Also: Bugfix for zip_nocase accessor failed to work in some cases.
Since we now use remapping there is no need for the
Linux.KapeFiles.CollectFromDirectory (which was basically the same as
the Windows.KapeFiles.Targets but applies to a mounted directory). That
artifact is now removed.
This was previously more like flatten() which had some surprising
results. This PR changes array() to be more like an array constructor
creating an array from a variety of objects
In the distant past we used to use url encoding to encode recursive
pathspecs but this is problematic because url encoding is just not
reliable enough. We have since moved to an OSPath object that properly
encodes paths as JSON objects but there is still some backwards
compatibility code in there that tries to detect a URL style path by
looking for # (the fragment component of the url).
This is probably a good time to remove this compatibility code because
we dont use urls for probably about 4-5 releases now.
Also sync latest KapeFiles Targets and fixed tests
Fixes: #2906
This makes VSS analysis a lot simpler since the VQL just needs to use
the 'ntfs_vss' accessor and will suddenly see distinct files from all
the VSS returned by a glob in the same directory. Further analysis of
these artifacts will use the ntfs_vss to open these files transparenly.
New script makes it easier to manage new translations:
1. Script automatically creates a json file with newer translations
2. User manually copies and pastes into google translate to get
translations. Paste to the automated json file.
3. Language files merge translations at runtime.
When a human is able to review translations they can move them into the
primary file once vetted.
* When copying large files it is useful to get some kind of feedback.
* Also implemented less strict escaping for zip filenames - previously
this was reusing the same rules as the filestore which mutated
things like the .db extension.
* Updated pathspec() function to accept an accessor
* Start client's CA Pool from the OS root store, then adds built in
certs. (Issue #2330)
* Updated KapeFiles to latest.
The NTFS context gets reset periodically and this causes a read to
return EOF on closed file handle. When this happens we need to retry
the read which will force it to recreate caches and parse MFT etc.
This is usually not needed for single shot collections, so this PR
also disables it from the KapeTargets artifact
This allows controlling the maximum size of the line buffer (default
is 64kb).
Also various cleanup tasks
* sync KapeFiles.Targets
* remove SDS target from KapeFiles and add a special collector for it.
* Refacted Windows.KapeFiles.Extract - Use uploads table directly to
find file in container.
Fixes: #1142Fixes: #1051
Sometimes users need to host tools on external resources (e.g. S3
bucket). This change allows this url to be configured in the GUI so
all endpoints can download the tool from the specified URL.
Also this change includes some fixes to hunt and flow view refresh and
a fix to Windows.KapeFiles.Target that broke due to typed parameters.
The Memory Ring Buffer implementation suffers from a potential
deadlock that may occur when the buffer is full causing calls to
AddResponse() to block. Additionally, the memory ring buffer was not
properly filtering out cancelled messages (like the file ring buffer).
This creates problems in cancelling large hunts and could potentially
lead to client deadlocks.
The lazy_ntfs accessor is faster because it only looks at $I30
records. The lazy_ntfs accessor was calling record.IsValid() to make
sure the records are valid. However in practice $I30 records are not
always fully populated in particular $LogFile has wrong timestamps.
This change does not force this check any more.
Removed unused commands:
- console (completely replaced by notebooks).
- artifacts acquire (Not very different from artifacts collect)
- inspect (not really used any more)
- report (superseded by notebooks)
It is possible to bake in labels into client configs. There was a bug
in syncing those when interrogating the client. Also added a test for
this.
Re-synced KapeFiles repository - they started using %user% when they
really meant *.
Custom collector builder now adds custom artifacts into the collector
configuration.
