mirror of
https://github.com/Velocidex/velociraptor.git
synced 2026-10-12 02:09:59 +00:00
Mike Cohen
e16721ee09
One of the common issues with Velociraptor is managing disk space. Up to now the server would store all files in uncompressed clear text form. While this is handy because all data is just a simple file, it is wasteful of disk space and often causes disk issues. This PR allows the data to be stored in a compressed form on the disk: 1. Clients are requested to compress they payload (either VQLResponse which is the json output from a VQL query, or FileBuffer which is bulk file data). 2. The server simply writes the already compressed data in chunks on the disk. This allows the server to offload compression to the clients, and actually allows the server to go much faster due to reduced IO, by simply writing already compressed chunks. When reading the data back out, the server is able to transparently decompress the data using the "fs" accessor. This allows compression to be handled transparently. Some metrics: - A Windows.NTFS.MFT JSON output is 960Mb uncompressed and 80mb compressed. JSON is highly compressible and this helps a lot.
20 lines
386 B
Go
20 lines
386 B
Go
package utils
|
|
|
|
import (
|
|
"context"
|
|
"testing"
|
|
|
|
"www.velocidex.com/golang/velociraptor/vtesting/assert"
|
|
)
|
|
|
|
func TestCompression(t *testing.T) {
|
|
test_str := []byte("hello world")
|
|
buffer, err := Compress(test_str)
|
|
assert.NoError(t, err)
|
|
|
|
uncompressed, err := Uncompress(context.Background(), buffer)
|
|
assert.NoError(t, err)
|
|
|
|
assert.Equal(t, string(test_str), string(uncompressed))
|
|
}
|