Compare commits

...
Author SHA1 Message Date
Mzack9999 40321dd68a Merge dev 2026-10-27 16:09:42 +01:00
Mzack9999 abcdcf19a4 Merge pull request #7940 from mazzz3r/feat/info-placeholders
feat(templates): fill placeholders in info description, impact and remediation
2026-10-27 15:21:09 +01:00
Michael Rodger 9dc004926d fix(trackers): gitea reporter pagination (#7945) 2026-10-27 14:11:01 +07:00
Mzack9999 3d2f411e1d Merge pull request #7925 from projectdiscovery/dwisiswant0/fix/offlinehttp/read-status-lines-by-position-not-text-search
fix(offlinehttp): read status lines by position, not text search
2026-10-26 21:38:57 +01:00
Mzack9999 763e490f42 merge dev 2026-10-26 21:10:10 +01:00
Mzack9999 421ec70686 Merge pull request #7947 from projectdiscovery/dwisiswant0/fix/trackers/follow-x-next-page-in-gitlab-duplicate-search
fix(trackers): follow X-Next-Page in gitlab duplicate search
2026-10-26 20:47:55 +01:00
Mzack9999 55d17f5a29 Pin baseline 2026-10-26 16:29:39 +01:00
Mzack9999 b7e66ae57c Fix test 2026-10-26 15:57:03 +01:00
Dwi Siswanto aac1427cdc fix(trackers): follow X-Next-Page in gitlab duplicate search
The GitLab tracker ended the duplicate search at the first page holding
fewer issues than duplicate-issue-page-size. GitLab caps per_page at
100, so with a page size above 100 every page looked like the last one.
The tracker read only the first 100 search results and opened a new
issue when the match came later.

The client library already parses X-Next-Page into Response.NextPage.
Follow it instead of guessing from the page length. This also drops the
empty-page request the old check sent when the last page was full.

Fixes: #7946
Refs: #7944
Signed-off-by: Dwi Siswanto <[email protected]>
2026-10-26 21:10:51 +07:00
Mzack9999 9aef358094 Score llm 2026-10-26 12:26:43 +01:00
Mzack9999 b62180c641 Merge dev 2026-10-26 12:04:12 +01:00
Mzack9999 1d3f1f5bc2 Gate interactsh 2026-10-26 11:32:06 +01:00
Mzack9999 4b3c1633ee Merge dev 2026-10-26 01:00:32 +01:00
Mzack9999 d2f862edf5 Merge origin 2026-10-26 00:33:21 +01:00
mazzz3r c4552c6011 feat(templates): fill placeholders in info description, impact and remediation
Description, impact and remediation can reference values of the finding
with the usual {{...}} syntax: named extractors, payload values of the
matching request, variables, DSL helpers, and the result fields host,
port, ip, url, scheme, path, matched-at, matcher-name and
extractor-name, which win over values of the same name. Name, severity,
tags and classification stay static, since filtering, grouping and
issue trackers key on them.

MakeDefaultResultEvent renders each result's copy of the info, so every
protocol gets it and the template never changes. Each field goes
through render.Render without scan options, like request and matcher
text, so nested placeholders and helpers behave the same and network
helpers stay denied. A field whose expression fails keeps the
template's text.

When placeholders change the text, the result keeps the written info in
TemplateInfo (json:"-" bson:"-"):
- the writer runs the -redact patterns over the changed fields, as it
  does over the raw request and response;
- SARIF builds the shared rule from the written info and puts each
  finding's filled text, capped at 4096 characters, in its result
  message.

Info is filled only from a template's own event. Templates with info
placeholders are not clustered, since a cluster's shared event holds the
first template's variables and extracted values, and a global matcher
fills its info from the result fields alone.
2026-10-25 19:37:42 +00:00
Dwi Siswanto 4331cd0f7a fix(offlinehttp): read status lines by position, not text search
Passive mode added a minor version to status lines such as "HTTP/2 200
OK" by rewriting the first "HTTP/<digit> " anywhere in the raw data. In
a valid HTTP/1.1 response that text could sit in a header, the reason
phrase, or the body, which was then altered: a Content-Length body came
out truncated and a chunked one failed to decode. Only the protocol
token at the start of the response is normalized now.

When the data starts with a request, the response is now read from
where the request body ends, with the same normalization. That path
used to reject HTTP/2 status lines and read a request body as the
status line, then fall back to the last "HTTP/" in the data, which
fails whenever the response body contains one. A dump that keeps a
request's Content-Length but leaves out its body is retried right after
the headers. Two exchanges back to back now yield the first response
for HTTP/2, as they already did for HTTP/1.1.

Up to five interim 1xx responses are skipped to reach the final one,
and an interim response is returned only when no final response is
found. A response read after a request carries that request. Data that
cannot be read this way still uses the last "HTTP/" search; httpx
-store-chain output, which puts a newline after the first request,
relies on it to return the chain's final response.

This supersedes #7922, which anchors the version pattern and so fixes

Fixes: #7921
Fixes: #7924
Reported-by: @MasterchiefGalaxy
Signed-off-by: Dwi Siswanto <[email protected]>
2026-10-25 13:16:36 +01:00
Mzack9999 717062fd0e fix confidence 2026-09-23 21:58:12 +02:00
Mzack9999 6c2ea4edcb merge dev 2026-09-23 20:23:17 +02:00
Mzack9999 1898b4aec4 merge dev 2026-09-14 22:34:30 +02:00
Mzack9999 e3864b60f2 Resolve conflicts 2026-07-15 19:02:06 +04:00
Mzack9999 187d855145 allowlist openvas 2026-06-26 01:48:22 +02:00
Mzack9999 1f276227ff report confidence 2026-06-26 01:31:23 +02:00
Mzack9999 e0fd39a862 protocol parts 2026-06-26 01:24:31 +02:00
Mzack9999 e0ed203765 negative evidence 2026-06-26 01:14:08 +02:00
Mzack9999 3ae786749d tune scoring 2026-06-26 01:08:50 +02:00
Mzack9999 b3881248ca refine scoring 2026-06-26 01:00:51 +02:00
Mzack9999 1ddd00efef baseline detection 2026-06-26 00:57:58 +02:00
Mzack9999 bc776d29a8 score confidence 2026-06-26 00:57:58 +02:00
37 changed files with 2538 additions and 102 deletions

No files matched your search

+18
View File
@@ -530,6 +530,16 @@ Description of the template.
You can go in-depth here on what the template actually does.
Description, impact and remediation may hold {{placeholders}} filled for
each finding: named extractors, payloads, variables and DSL helpers
(network helpers excluded), and the result fields host, port, ip, url,
scheme, path, matched-at, matcher-name and extractor-name, which win over
values of the same name. Use join(", ", name) for an extractor with
several values, and brackets for a hyphenated name inside a helper:
to_upper([matcher-name]). Unknown placeholders stay as written, and so
does a field whose expression fails. Global matcher templates match on
other templates' responses, so only the result fields fill their text.
Examples:
@@ -543,6 +553,10 @@ description: Bower is a package manager which stores package information in the
description: Subversion ALM for the enterprise before 8.8.2 allows reflected XSS at multiple locations
```
```yaml
description: Redis {{version}} on {{host}}:{{port}} accepts the default password
```
</div>
@@ -559,6 +573,8 @@ Impact of the template.
You can go in-depth here on impact of the template.
Supports the same placeholders as description.
Examples:
@@ -665,6 +681,8 @@ Remediation steps for the template.
You can go in-depth here on how to mitigate the problem found by this template.
Supports the same placeholders as description.
Examples:
+2
View File
@@ -23,6 +23,8 @@ extend-exclude = [
]
[default.extend-identifiers]
# Proper noun (OpenVAS scanner, source of the QoD confidence model)
OpenVAS = "OpenVAS"
# Go identifiers that cannot be renamed without breaking API
MisMatched = "MisMatched"
IsMisMatchedCert = "IsMisMatchedCert" # tlsx clients API
+2 -1
View File
@@ -57,7 +57,8 @@
# severity: low
# # duplicate-issue-check (optional) flag to enable duplicate tracking issue check
# duplicate-issue-check: false
# # duplicate-issue-page-size (optional) controls how many issues to fetch per page when searching for duplicates
# # duplicate-issue-page-size (optional) sets the number of issues to request per page when searching for duplicates
# # (GitLab returns at most 100 issues per page)
# duplicate-issue-page-size: 100
# # duplicate-issue-max-pages (optional) limits how many pages to fetch when searching for duplicates (0 = no limit)
# duplicate-issue-max-pages: 0
+1
View File
@@ -469,6 +469,7 @@ on extensive configurability, massive extensibility and ease of use.`)
flagSet.BoolVarP(&options.DisableHTTPProbe, "no-httpx", "nh", false, "disable httpx probing for non-url input"),
flagSet.BoolVar(&options.PreflightPortScan, "preflight-portscan", false, "run preflight resolve + TCP portscan and filter targets before scanning (disabled by default)"),
flagSet.BoolVar(&options.DisableStdin, "no-stdin", false, "disable stdin processing"),
flagSet.BoolVarP(&options.ConfidenceBaseline, "confidence-baseline", "cob", false, "send a per-host control request to lower confidence on catch-all (false-positive) matches"),
flagSet.BoolVar(&options.EnableHTTPCache, "http-cache", false, "enable HTTP cache (RFC 9111) for HTTP requests"),
)
+5
View File
@@ -55,6 +55,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/projectfile"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/automaticscan"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/baseline"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/globalmatchers"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/honeypotdetector"
@@ -695,6 +696,10 @@ func (r *Runner) RunEnumeration() error {
executorOpts.ExportReqURLPattern = true
}
if r.options.ConfidenceBaseline {
executorOpts.BaselineCache = baseline.New()
}
if len(r.options.SecretsFile) > 0 && !r.options.Validate {
// Clone options so GetAuthTmplStore can modify them without affecting the original
authOptions := r.options.Copy()
+63
View File
@@ -1,9 +1,13 @@
package model
import (
"slices"
"strings"
"github.com/invopop/jsonschema"
"github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
"github.com/projectdiscovery/nuclei/v3/pkg/model/types/stringslice"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/marker"
)
type schemaMetadata struct {
@@ -47,15 +51,28 @@ type Info struct {
//
// You can go in-depth here on what the template actually does.
//
// Description, impact and remediation may hold {{placeholders}} filled for
// each finding: named extractors, payloads, variables and DSL helpers
// (network helpers excluded), and the result fields host, port, ip, url,
// scheme, path, matched-at, matcher-name and extractor-name, which win over
// values of the same name. Use join(", ", name) for an extractor with
// several values, and brackets for a hyphenated name inside a helper:
// to_upper([matcher-name]). Unknown placeholders stay as written, and so
// does a field whose expression fails. Global matcher templates match on
// other templates' responses, so only the result fields fill their text.
//
// examples:
// - value: "\"Bower is a package manager which stores package information in the bower.json file\""
// - value: "\"Subversion ALM for the enterprise before 8.8.2 allows reflected XSS at multiple locations\""
// - value: "\"Redis {{version}} on {{host}}:{{port}} accepts the default password\""
Description string `json:"description,omitempty" yaml:"description,omitempty" jsonschema:"title=description of the template,description=In-depth explanation on what the template does,type=string,example=Bower is a package manager which stores package information in the bower.json file"`
// description: |
// Impact of the template.
//
// You can go in-depth here on impact of the template.
//
// Supports the same placeholders as description.
//
// examples:
// - value: "\"Successful exploitation of this vulnerability could allow an attacker to execute arbitrary SQL queries, potentially leading to unauthorized access, data leakage, or data manipulation.\""
// - value: "\"Successful exploitation of this vulnerability could allow an attacker to execute arbitrary script code in the context of the victim's browser, potentially leading to session hijacking, defacement, or theft of sensitive information.\""
@@ -89,11 +106,57 @@ type Info struct {
//
// You can go in-depth here on how to mitigate the problem found by this template.
//
// Supports the same placeholders as description.
//
// examples:
// - value: "\"Change the default administrative username and password of Apache ActiveMQ by editing the file jetty-realm.properties\""
Remediation string `json:"remediation,omitempty" yaml:"remediation,omitempty" jsonschema:"title=remediation steps for the template,description=In-depth explanation on how to fix the issues found by the template,example=Change the default administrative username and password of Apache ActiveMQ by editing the file jetty-realm.properties,type=string"`
}
// TextField is a free-text field of the info, labeled as reports show it.
type TextField struct {
Label string
Value *string
}
// PlaceholderFields returns the free-text fields that may hold {{placeholders}}
// filled for each finding. Rendering, redaction and the sarif export all walk
// this one list, so a field added here is handled by every one of them.
func (i *Info) PlaceholderFields() []TextField {
return []TextField{
{Label: "Description", Value: &i.Description},
{Label: "Impact", Value: &i.Impact},
{Label: "Remediation", Value: &i.Remediation},
}
}
// HasPlaceholders reports whether the field holds a {{...}} or §...§ marker
// the renderer could fill.
func (f TextField) HasPlaceholders() bool {
return strings.Contains(*f.Value, marker.ParenthesisOpen) || strings.Contains(*f.Value, marker.General)
}
// HasPlaceholders reports whether any placeholder field holds a marker.
func (i *Info) HasPlaceholders() bool {
return slices.ContainsFunc(i.PlaceholderFields(), TextField.HasPlaceholders)
}
// FilledFields returns the placeholder fields whose text differs from written,
// the info as the template wrote it. A nil written means nothing was filled.
func (i *Info) FilledFields(written *Info) []TextField {
if written == nil {
return nil
}
writtenFields := written.PlaceholderFields()
var filled []TextField
for n, field := range i.PlaceholderFields() {
if *field.Value != *writtenFields[n].Value {
filled = append(filled, field)
}
}
return filled
}
// JSONSchemaProperty returns the JSON schema property for the Info object.
func (i Info) JSONSchemaExtend(base *jsonschema.Schema) {
// since we are re-using a stringslice and rawStringSlice everywhere, we can extend/edit the schema here
+18
View File
@@ -135,3 +135,21 @@ func TestUnmarshal(t *testing.T) {
info2 := assertUnmarshalledTemplateInfo(t, yamlPayload2)
require.Equal(t, info1, info2)
}
func TestInfoFilledFields(t *testing.T) {
written := Info{Description: "{{host}} is exposed", Impact: "Data loss"}
filled := written
filled.Description = "example.com is exposed"
require.Nil(t, filled.FilledFields(nil))
fields := filled.FilledFields(&written)
require.Len(t, fields, 1)
require.Equal(t, "Description", fields[0].Label)
require.Equal(t, "example.com is exposed", *fields[0].Value)
}
func TestInfoHasPlaceholders(t *testing.T) {
require.False(t, (&Info{Description: "Redis is exposed"}).HasPlaceholders())
require.True(t, (&Info{Impact: "{{host}} is exposed"}).HasPlaceholders())
require.True(t, (&Info{Remediation: "rotate §password§"}).HasPlaceholders())
}
+300
View File
@@ -0,0 +1,300 @@
package operators
import (
"strings"
"github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
)
// Confidence tiers reported alongside a result. They express how reliable the
// detection is, independent of severity (how bad the issue is). The split
// follows the Burp "tentative/firm/certain" and OpenVAS QoD models.
const (
ConfidenceLow = "low"
ConfidenceMedium = "medium"
ConfidenceHigh = "high"
)
// confidence is reported on a 0-100 scale modelled on OpenVAS "Quality of
// Detection" (QoD). Tier thresholds align with the scale; OpenVAS uses 70 as its
// default trustworthy cutoff, which sits inside the medium band here.
const (
confidenceMediumThreshold = 50
confidenceHighThreshold = 80
// ConfidenceCertain is the score for out-of-band confirmed detections
// (e.g. an interactsh interaction): unambiguous proof, so the maximum.
ConfidenceCertain = 100
// ConfidenceCatchAll is the clamped score for matches that also fire against
// the host's catch-all baseline: a strong false-positive signal.
ConfidenceCatchAll = 10
)
// Base reliability of each evidence class. The ordering follows the black-box
// scanner literature and QoD detection methods: matching on a status code or
// response size is the most false-positive prone form of inference (an HTTP 500
// can be any error, a banner can be backported), matching content with a
// word is moderate, and matching content with a regex/dsl/binary/xpath is the
// strongest in-band signal. Out-of-band confirmation (handled by the caller)
// outranks all of these.
const (
weightStatusOrSize = 25
// weightLLM scores an llm matcher. A model verdict is a judgement rather
// than a deterministic check of the response, so it sits below word
// matching and needs deterministic matchers to reach a higher tier.
weightLLM = 40
weightWordHeader = 45
weightStrongHeader = 55
weightWordContent = 60
weightStrongContent = 75
// corroboration adds confidence only across DISTINCT, AND-combined evidence
// classes (independent confirmation), with diminishing returns.
corroborationPerClass = 10
corroborationCap = 20
negativeMatcherBonus = 5
extractorBonus = 3
// multiPatternBonus rewards a single matcher that requires several distinct
// patterns to all be present (condition: and / match-all). Needing many
// independent strings is far harder to trigger by accident than one.
multiPatternBonus = 8
)
// dslWeakIndicators are DSL variables that only inspect response metadata.
// dslContentIndicators signal that a DSL expression actually inspects the
// response body/headers, which is a far stronger detection method.
var (
dslWeakIndicators = []string{"status_code", "content_length", "duration"}
dslContentIndicators = []string{"body", "header", "all_headers", "contains", "regex", "len(", "tolower", "toupper", "title", "hash", "md5", "sha", "base64", "html", "response", "data"}
)
// dslWeakOnly reports whether a DSL matcher only inspects status/size metadata.
// Such a matcher is a "strong" matcher type but carries status-level reliability
// (an HTTP 500 can be any error), so it must not be scored as body evidence.
func dslWeakOnly(m *matchers.Matcher) bool {
if len(m.DSL) == 0 {
return false
}
weak := false
for _, expr := range m.DSL {
e := strings.ToLower(expr)
for _, c := range dslContentIndicators {
if strings.Contains(e, c) {
return false
}
}
for _, w := range dslWeakIndicators {
if strings.Contains(e, w) {
weak = true
}
}
}
return weak
}
// requiresAllPatterns reports whether every pattern in the matcher must be
// present (condition: and or match-all), rather than any single one.
func requiresAllPatterns(m *matchers.Matcher) bool {
return m.MatchAll || strings.EqualFold(m.Condition, "and")
}
// patternCount counts the independent content patterns a matcher carries.
// Status/size codes are intentionally excluded: matching any of several status
// codes broadens rather than narrows a match.
func patternCount(m *matchers.Matcher) int {
return len(m.Words) + len(m.Regex) + len(m.Binary) + len(m.DSL) + len(m.XPath)
}
// metadataParts are response parts that carry weak, false-positive-prone signal
// across every protocol: protocol metadata (status/size/rcode/duration), HTTP
// response headers (the generic part and the common header fields used directly)
// and request-side echoes. Matching these is more backportable / accidental than
// matching real response content. Everything not listed here (response bodies,
// DNS answer/ns/extra sections, TLS certificate fields, command stdout/stderr,
// websocket/whois/javascript payloads, ...) is treated as content, so non-HTTP
// protocols are scored on the strength of what they actually inspect.
var metadataParts = map[string]struct{}{
"header": {}, "all_headers": {}, "headers": {},
"status": {}, "status_code": {}, "size": {}, "content_length": {}, "duration": {},
"rcode": {},
"request": {}, "host": {}, "ip": {}, "port": {}, "matched": {}, "type": {},
// common HTTP response header fields that templates match on directly
"content_type": {}, "content-type": {}, "server": {}, "location": {},
"set_cookie": {}, "set-cookie": {}, "www_authenticate": {}, "www-authenticate": {},
"x_powered_by": {}, "x-powered-by": {}, "cache_control": {}, "cache-control": {},
}
// stripIndexSuffix removes the _<n> suffix that flow / multi-request templates
// append to response parts (body_2, header_3, content_type_2) so they classify
// the same as their base part.
func stripIndexSuffix(p string) string {
i := strings.LastIndexByte(p, '_')
if i <= 0 || i == len(p)-1 {
return p
}
for j := i + 1; j < len(p); j++ {
if p[j] < '0' || p[j] > '9' {
return p
}
}
return p[:i]
}
// isContentPart reports whether a matcher part inspects response content rather
// than protocol metadata or headers. Content is the default: only the known
// weak metadata parts are excluded. Content matches are harder to trigger by
// accident, so they carry higher confidence weight.
func isContentPart(part string) bool {
p := stripIndexSuffix(strings.ToLower(strings.TrimSpace(part)))
_, meta := metadataParts[p]
return !meta
}
// evidenceClass identifies an independent line of evidence. Corroboration is
// rewarded across distinct classes, not redundant repeats of the same weak
// matcher, so three status matchers do not masquerade as strong evidence.
type evidenceClass struct {
kind string
content bool
}
// classify returns the evidence class and base reliability weight for a matcher.
func classify(m *matchers.Matcher) (evidenceClass, int) {
content := isContentPart(m.Part)
switch m.Type.MatcherType {
case matchers.StatusMatcher:
return evidenceClass{kind: "status"}, weightStatusOrSize
case matchers.SizeMatcher:
return evidenceClass{kind: "size"}, weightStatusOrSize
case matchers.WordsMatcher:
if content {
return evidenceClass{kind: "word", content: true}, weightWordContent
}
return evidenceClass{kind: "word"}, weightWordHeader
case matchers.LLMMatcher:
// one class whatever part it reads: several llm matchers are not
// independent evidence, but an llm matcher AND-combined with a
// deterministic matcher still counts as corroboration.
return evidenceClass{kind: "llm"}, weightLLM
case matchers.DSLMatcher:
if dslWeakOnly(m) {
// a dsl that only checks status_code/content_length is metadata
// inference, not body evidence, despite being a "strong" type.
return evidenceClass{kind: "status"}, weightStatusOrSize
}
if content {
return evidenceClass{kind: "strong", content: true}, weightStrongContent
}
return evidenceClass{kind: "strong"}, weightStrongHeader
default: // regex, binary, xpath
if content {
return evidenceClass{kind: "strong", content: true}, weightStrongContent
}
return evidenceClass{kind: "strong"}, weightStrongHeader
}
}
// Confidence computes a static, deterministic detection-confidence score (0-100)
// and tier from the template's matcher composition. It rewards strong matcher
// techniques, content-level matching, independent corroboration across distinct
// AND-combined evidence classes and negative matchers, while capping redundant
// weak matchers. Hidden (internal) helper matchers are ignored. The score is
// derived from the template definition alone and sends no additional traffic;
// out-of-band confirmation is folded in by the caller.
func (operators *Operators) Confidence() (int, string) {
if operators == nil || len(operators.Matchers) == 0 {
// extraction-only or matcher-less detections carry the weakest signal
return 0, ConfidenceLow
}
// positive (direct) evidence and negative (absence) evidence are tracked
// separately: a negative matcher behaves as a false-positive guard when it
// accompanies positive evidence, but is the detection itself when it stands
// alone (e.g. token-spray validity checks, "setting reverted" CVE steps).
classes := make(map[evidenceClass]struct{})
best := 0
hasMultiPattern := false
negClasses := make(map[evidenceClass]struct{})
negBest := 0
negMultiPattern := false
negCount := 0
for _, m := range operators.Matchers {
if m == nil || m.Internal {
// internal matchers feed dynamic extraction chains and are hidden
// from output, so they are not evidence for the user-facing match.
continue
}
class, weight := classify(m)
// a single matcher that requires several distinct content patterns to
// all hold is highly specific. Status/size and weak-dsl matchers are
// excluded (weight gate) since matching any of several codes broadens.
// An llm matcher gets no bonus: its inputs are not independent patterns.
multi := weight > weightStatusOrSize && class.kind != "llm" && requiresAllPatterns(m) && patternCount(m) > 1
if m.Negative {
negCount++
negClasses[class] = struct{}{}
if weight > negBest {
negBest = weight
}
if multi {
negMultiPattern = true
}
continue
}
classes[class] = struct{}{}
if weight > best {
best = weight
}
if multi {
hasMultiPattern = true
}
}
// a negative matcher is a guard only when it backs up positive evidence.
negativeAsGuard := best > 0 && negCount > 0
if best == 0 {
// no positive evidence: the absence assertions are the detection. Score
// them by their own weight rather than discarding the match entirely.
best, classes, hasMultiPattern = negBest, negClasses, negMultiPattern
}
if best == 0 {
return 0, ConfidenceLow
}
score := best
if hasMultiPattern {
score += multiPatternBonus
}
// Independent corroboration raises confidence, but only when every matcher
// must hold (AND) and only across distinct evidence classes.
if operators.matchersCondition == matchers.ANDCondition && len(classes) > 1 {
bonus := (len(classes) - 1) * corroborationPerClass
if bonus > corroborationCap {
bonus = corroborationCap
}
score += bonus
}
if negativeAsGuard {
score += negativeMatcherBonus
}
if len(operators.Extractors) > 0 {
score += extractorBonus
}
if score > ConfidenceCertain {
score = ConfidenceCertain
}
return score, ScoreToTier(score)
}
// ScoreToTier maps a 0-100 confidence score to its tier.
func ScoreToTier(score int) string {
switch {
case score >= confidenceHighThreshold:
return ConfidenceHigh
case score >= confidenceMediumThreshold:
return ConfidenceMedium
default:
return ConfidenceLow
}
}
+470
View File
@@ -0,0 +1,470 @@
package operators
import (
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
"github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
"github.com/stretchr/testify/require"
)
func matcher(t matchers.MatcherType, part string) *matchers.Matcher {
return &matchers.Matcher{Type: matchers.MatcherTypeHolder{MatcherType: t}, Part: part}
}
func TestConfidenceTier(t *testing.T) {
tests := []struct {
name string
operators *Operators
wantTier string
}{
{
name: "nil operators",
operators: nil,
wantTier: ConfidenceLow,
},
{
name: "no matchers",
operators: &Operators{},
wantTier: ConfidenceLow,
},
{
name: "single status matcher is low",
operators: &Operators{
Matchers: []*matchers.Matcher{matcher(matchers.StatusMatcher, "")},
},
wantTier: ConfidenceLow,
},
{
name: "redundant status matchers stay low",
operators: &Operators{
matchersCondition: matchers.ANDCondition,
Matchers: []*matchers.Matcher{
matcher(matchers.StatusMatcher, ""),
matcher(matchers.StatusMatcher, ""),
matcher(matchers.StatusMatcher, ""),
},
},
wantTier: ConfidenceLow,
},
{
name: "single body word is medium",
operators: &Operators{
Matchers: []*matchers.Matcher{matcher(matchers.WordsMatcher, "body")},
},
wantTier: ConfidenceMedium,
},
{
name: "single body regex is medium",
operators: &Operators{
Matchers: []*matchers.Matcher{matcher(matchers.RegexMatcher, "body")},
},
wantTier: ConfidenceMedium,
},
{
name: "or condition takes strongest matcher",
operators: &Operators{
matchersCondition: matchers.ORCondition,
Matchers: []*matchers.Matcher{
matcher(matchers.StatusMatcher, ""),
matcher(matchers.DSLMatcher, "body"),
},
},
wantTier: ConfidenceMedium,
},
{
name: "distinct and-combined classes reach high",
operators: &Operators{
matchersCondition: matchers.ANDCondition,
Matchers: []*matchers.Matcher{
matcher(matchers.WordsMatcher, "body"),
matcher(matchers.RegexMatcher, "body"),
},
},
wantTier: ConfidenceHigh,
},
{
name: "status with extractor stays low",
operators: &Operators{
Matchers: []*matchers.Matcher{matcher(matchers.StatusMatcher, "")},
Extractors: []*extractors.Extractor{{}},
},
wantTier: ConfidenceLow,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
_, tier := tt.operators.Confidence()
require.Equal(t, tt.wantTier, tier)
})
}
}
// TestConfidenceCorroboration verifies that confidence rises only with distinct
// independent evidence, not with redundant repeats of the same matcher class.
func TestConfidenceCorroboration(t *testing.T) {
distinct := &Operators{
matchersCondition: matchers.ANDCondition,
Matchers: []*matchers.Matcher{
matcher(matchers.RegexMatcher, "body"),
matcher(matchers.WordsMatcher, "header"),
matcher(matchers.StatusMatcher, ""),
},
}
redundant := &Operators{
matchersCondition: matchers.ANDCondition,
Matchers: []*matchers.Matcher{
matcher(matchers.RegexMatcher, "body"),
matcher(matchers.RegexMatcher, "body"),
matcher(matchers.RegexMatcher, "body"),
},
}
distinctScore, _ := distinct.Confidence()
redundantScore, _ := redundant.Confidence()
require.Greater(t, distinctScore, redundantScore, "distinct evidence classes must outscore redundant repeats of one class")
}
// TestConfidenceNegativeMatcherBonus verifies a negative matcher (explicit
// exclusion of generic/error noise) raises confidence over an identical match
// without it.
func TestConfidenceNegativeMatcherBonus(t *testing.T) {
base := &Operators{Matchers: []*matchers.Matcher{matcher(matchers.RegexMatcher, "body")}}
withNegative := &Operators{
matchersCondition: matchers.ANDCondition,
Matchers: []*matchers.Matcher{
matcher(matchers.RegexMatcher, "body"),
{Type: matchers.MatcherTypeHolder{MatcherType: matchers.WordsMatcher}, Part: "body", Negative: true},
},
}
baseScore, _ := base.Confidence()
negScore, _ := withNegative.Confidence()
require.Greater(t, negScore, baseScore, "negative matcher should add a specificity bonus")
}
// TestConfidenceNegativeMatcherIsNotPositiveEvidence verifies that a negative
// (absence) guard does not contribute its matcher weight as positive evidence.
// A status-only match guarded by "body must not contain an error string" is
// still only a status code match and must stay low confidence.
func TestConfidenceNegativeMatcherIsNotPositiveEvidence(t *testing.T) {
guarded := &Operators{
matchersCondition: matchers.ANDCondition,
Matchers: []*matchers.Matcher{
matcher(matchers.StatusMatcher, ""),
{Type: matchers.MatcherTypeHolder{MatcherType: matchers.RegexMatcher}, Part: "body", Negative: true},
},
}
score, tier := guarded.Confidence()
require.Equal(t, ConfidenceLow, tier, "a strong negative guard must not lift a status-only match out of low")
require.LessOrEqual(t, score, weightStatusOrSize+negativeMatcherBonus, "negative matcher weight must not count as positive evidence")
}
// TestConfidenceIgnoresInternalMatchers verifies hidden helper matchers used for
// dynamic extraction do not count as user-facing evidence.
func TestConfidenceIgnoresInternalMatchers(t *testing.T) {
ops := &Operators{
matchersCondition: matchers.ANDCondition,
Matchers: []*matchers.Matcher{
matcher(matchers.StatusMatcher, ""),
{Type: matchers.MatcherTypeHolder{MatcherType: matchers.DSLMatcher}, Part: "body", Internal: true},
},
}
score, tier := ops.Confidence()
require.Equal(t, weightStatusOrSize, score, "internal matcher must be ignored, leaving only the status weight")
require.Equal(t, ConfidenceLow, tier)
}
// neg builds a negative matcher of the given type/part.
func neg(t matchers.MatcherType, part string) *matchers.Matcher {
return &matchers.Matcher{Type: matchers.MatcherTypeHolder{MatcherType: t}, Part: part, Negative: true}
}
// TestConfidenceRealWorldScenarios pins the score band and tier for matcher
// compositions that mirror real nuclei templates, so the scorer stays sensible
// as it evolves.
func TestConfidenceRealWorldScenarios(t *testing.T) {
tests := []struct {
name string
operators *Operators
minScore int
maxScore int
wantTier string
}{
{
name: "host-up status probe",
operators: &Operators{
Matchers: []*matchers.Matcher{matcher(matchers.StatusMatcher, "")},
},
minScore: 1, maxScore: weightStatusOrSize, wantTier: ConfidenceLow,
},
{
name: "tech detection single header banner",
operators: &Operators{
Matchers: []*matchers.Matcher{matcher(matchers.WordsMatcher, "header")},
},
minScore: weightWordHeader, maxScore: weightWordHeader, wantTier: ConfidenceLow,
},
{
name: "exposed panel title word",
operators: &Operators{
Matchers: []*matchers.Matcher{matcher(matchers.WordsMatcher, "body")},
},
minScore: weightWordContent, maxScore: weightWordContent, wantTier: ConfidenceMedium,
},
{
name: "favicon hash dsl",
operators: &Operators{
Matchers: []*matchers.Matcher{matcher(matchers.DSLMatcher, "")},
},
minScore: weightStrongContent, maxScore: weightStrongContent, wantTier: ConfidenceMedium,
},
{
name: "cve body regex and status with extractor",
operators: &Operators{
matchersCondition: matchers.ANDCondition,
Matchers: []*matchers.Matcher{
matcher(matchers.RegexMatcher, "body"),
matcher(matchers.StatusMatcher, ""),
},
Extractors: []*extractors.Extractor{{}},
},
minScore: confidenceHighThreshold, maxScore: ConfidenceCertain, wantTier: ConfidenceHigh,
},
{
name: "strong body match guarded by negative error page",
operators: &Operators{
matchersCondition: matchers.ANDCondition,
Matchers: []*matchers.Matcher{
matcher(matchers.WordsMatcher, "body"),
neg(matchers.WordsMatcher, "body"),
},
},
minScore: weightWordContent, maxScore: weightWordContent + negativeMatcherBonus, wantTier: ConfidenceMedium,
},
{
name: "fingerprint or any of several strong signals",
operators: &Operators{
matchersCondition: matchers.ORCondition,
Matchers: []*matchers.Matcher{
matcher(matchers.RegexMatcher, "body"),
matcher(matchers.DSLMatcher, "body"),
matcher(matchers.WordsMatcher, "header"),
},
},
minScore: weightStrongContent, maxScore: weightStrongContent, wantTier: ConfidenceMedium,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
score, tier := tt.operators.Confidence()
require.GreaterOrEqual(t, score, tt.minScore, "score below expected band")
require.LessOrEqual(t, score, tt.maxScore, "score above expected band")
require.Equal(t, tt.wantTier, tier)
})
}
}
// TestConfidenceNegativeOnlyIsEvidence verifies that when a detection has no
// positive matchers, a sole negative (absence) matcher is scored as the
// detection itself rather than discarded. This mirrors token-spray validity
// checks and "setting reverted" CVE steps.
func TestConfidenceNegativeOnlyIsEvidence(t *testing.T) {
negativeOnly := &Operators{Matchers: []*matchers.Matcher{
{Type: matchers.MatcherTypeHolder{MatcherType: matchers.WordsMatcher}, Part: "body", Words: []string{"Access Token Invalid"}, Negative: true},
}}
positive := &Operators{Matchers: []*matchers.Matcher{matcher(matchers.WordsMatcher, "body")}}
negScore, negTier := negativeOnly.Confidence()
posScore, _ := positive.Confidence()
require.Equal(t, weightWordContent, negScore, "a sole negative body matcher should score as a body match")
require.Equal(t, ConfidenceMedium, negTier)
require.Equal(t, posScore, negScore, "negative-only evidence should match the equivalent positive matcher weight")
}
// TestConfidenceMultiPatternBonus verifies a single matcher requiring several
// distinct content patterns (condition: and) outscores the same matcher with a
// single pattern, while a broad any-of (or) matcher gets no bonus.
func TestConfidenceMultiPatternBonus(t *testing.T) {
single := &Operators{Matchers: []*matchers.Matcher{
{Type: matchers.MatcherTypeHolder{MatcherType: matchers.WordsMatcher}, Part: "body", Words: []string{"a"}},
}}
multiAnd := &Operators{Matchers: []*matchers.Matcher{
{Type: matchers.MatcherTypeHolder{MatcherType: matchers.WordsMatcher}, Part: "body", Words: []string{"a", "b", "c"}, Condition: "and"},
}}
multiOr := &Operators{Matchers: []*matchers.Matcher{
{Type: matchers.MatcherTypeHolder{MatcherType: matchers.WordsMatcher}, Part: "body", Words: []string{"a", "b", "c"}, Condition: "or"},
}}
matchAll := &Operators{Matchers: []*matchers.Matcher{
{Type: matchers.MatcherTypeHolder{MatcherType: matchers.WordsMatcher}, Part: "body", Words: []string{"a", "b"}, MatchAll: true},
}}
singleScore, _ := single.Confidence()
multiAndScore, _ := multiAnd.Confidence()
multiOrScore, _ := multiOr.Confidence()
matchAllScore, _ := matchAll.Confidence()
require.Greater(t, multiAndScore, singleScore, "all-of multi-pattern matcher should outscore single pattern")
require.Equal(t, singleScore, multiOrScore, "any-of multi-pattern matcher should not earn the bonus")
require.Greater(t, matchAllScore, singleScore, "match-all matcher should earn the multi-pattern bonus")
}
// TestConfidenceDSLWeakOnly verifies a dsl matcher that only inspects status or
// size metadata is scored as weak as a status match, while a dsl that inspects
// the body keeps strong-content reliability.
func TestConfidenceDSLWeakOnly(t *testing.T) {
weak := &Operators{Matchers: []*matchers.Matcher{
{Type: matchers.MatcherTypeHolder{MatcherType: matchers.DSLMatcher}, DSL: []string{"status_code == 200"}},
}}
weakCombo := &Operators{Matchers: []*matchers.Matcher{
{Type: matchers.MatcherTypeHolder{MatcherType: matchers.DSLMatcher}, DSL: []string{"status_code == 200 && content_length > 10"}},
}}
strong := &Operators{Matchers: []*matchers.Matcher{
{Type: matchers.MatcherTypeHolder{MatcherType: matchers.DSLMatcher}, DSL: []string{"contains(body, 'SecretMarker') && status_code == 200"}},
}}
weakScore, weakTier := weak.Confidence()
weakComboScore, _ := weakCombo.Confidence()
strongScore, strongTier := strong.Confidence()
require.Equal(t, weightStatusOrSize, weakScore, "metadata-only dsl must score as a status match")
require.Equal(t, ConfidenceLow, weakTier)
require.Equal(t, weightStatusOrSize, weakComboScore, "status+size-only dsl is still metadata inference")
require.Equal(t, weightStrongContent, strongScore, "body-inspecting dsl keeps strong-content weight")
require.Equal(t, ConfidenceMedium, strongTier)
}
// TestIsContentPartAcrossProtocols verifies the part classifier recognizes
// content vs metadata parts for every protocol, including numbered suffixes
// emitted by multi-request / flow templates.
func TestIsContentPartAcrossProtocols(t *testing.T) {
content := []string{
"", "body", "all", "response", "raw", "data", // http / generic
"body_2", "body_5", "response_2", // multi-request numbered
"answer", "question", "ns", "extra", "trace", "a", "aaaa", "cname", // dns
"stderr", "code_1_response", // code
"subject_dn", "issuer_cn", "fingerprint", "cipher", "domains", // ssl
"history", // headless
"domain", "registrar", // whois
}
metadata := []string{
"header", "all_headers", "header_2", "header_3", // http headers
"content_type", "content_type_2", "server", "location", "set_cookie", // header fields
"status", "status_code", "size", "content_length", "duration", // http metadata
"rcode", // dns metadata
"request", "host", "ip", "port", "matched", "type", // request-side
}
for _, p := range content {
require.True(t, isContentPart(p), "part %q should be content", p)
}
for _, p := range metadata {
require.False(t, isContentPart(p), "part %q should be metadata", p)
}
}
// TestConfidenceNonHTTPContentParts verifies that content matches on non-HTTP
// protocols (dns answer, code stderr, tls cert fields) are scored as content,
// not weak metadata.
func TestConfidenceNonHTTPContentParts(t *testing.T) {
for _, part := range []string{"answer", "stderr", "subject_dn", "code_1_response"} {
ops := &Operators{Matchers: []*matchers.Matcher{matcher(matchers.WordsMatcher, part)}}
score, tier := ops.Confidence()
require.Equal(t, weightWordContent, score, "word match on %q should score as content", part)
require.Equal(t, ConfidenceMedium, tier)
}
}
func TestScoreToTier(t *testing.T) {
require.Equal(t, ConfidenceLow, ScoreToTier(0))
require.Equal(t, ConfidenceLow, ScoreToTier(confidenceMediumThreshold-1))
require.Equal(t, ConfidenceMedium, ScoreToTier(confidenceMediumThreshold))
require.Equal(t, ConfidenceHigh, ScoreToTier(confidenceHighThreshold))
require.Equal(t, ConfidenceHigh, ScoreToTier(ConfidenceCertain))
}
func TestConfidenceLLMMatcher(t *testing.T) {
llm := func() *matchers.Matcher {
m := matcher(matchers.LLMMatcher, "body")
m.Prompt = "is this an exposed admin panel?"
return m
}
llmMulti := func() *matchers.Matcher {
m := llm()
m.Condition = "and"
m.MatchAll = true
m.Words = []string{"admin", "panel"}
return m
}
tests := []struct {
name string
operators *Operators
wantScore int
wantTier string
}{
{
name: "single llm matcher",
operators: &Operators{Matchers: []*matchers.Matcher{llm()}},
wantScore: weightLLM,
wantTier: ConfidenceLow,
},
{
name: "llm matcher gets no multi-pattern bonus",
operators: &Operators{Matchers: []*matchers.Matcher{llmMulti()}},
wantScore: weightLLM,
wantTier: ConfidenceLow,
},
{
name: "two llm matchers are one evidence class",
operators: &Operators{
matchersCondition: matchers.ANDCondition,
Matchers: []*matchers.Matcher{llm(), llm()},
},
wantScore: weightLLM,
wantTier: ConfidenceLow,
},
{
name: "llm AND body word adds corroboration",
operators: &Operators{
matchersCondition: matchers.ANDCondition,
Matchers: []*matchers.Matcher{llm(), matcher(matchers.WordsMatcher, "body")},
},
wantScore: weightWordContent + corroborationPerClass,
wantTier: ConfidenceMedium,
},
{
name: "llm AND status adds corroboration",
operators: &Operators{
matchersCondition: matchers.ANDCondition,
Matchers: []*matchers.Matcher{llm(), matcher(matchers.StatusMatcher, "")},
},
wantScore: weightLLM + corroborationPerClass,
wantTier: ConfidenceMedium,
},
{
name: "llm AND body regex AND status caps corroboration",
operators: &Operators{
matchersCondition: matchers.ANDCondition,
Matchers: []*matchers.Matcher{llm(), matcher(matchers.RegexMatcher, "body"), matcher(matchers.StatusMatcher, "")},
},
wantScore: weightStrongContent + corroborationCap,
wantTier: ConfidenceHigh,
},
{
name: "llm OR body word takes the best class only",
operators: &Operators{
Matchers: []*matchers.Matcher{llm(), matcher(matchers.WordsMatcher, "body")},
},
wantScore: weightWordContent,
wantTier: ConfidenceMedium,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
score, tier := tc.operators.Confidence()
require.Equal(t, tc.wantScore, score)
require.Equal(t, tc.wantTier, tier)
})
}
}
+19
View File
@@ -9,6 +9,18 @@ import (
mapsutil "github.com/projectdiscovery/utils/maps"
)
// confidenceColor colorizes confidence text using the tier color.
func (w *StandardWriter) confidenceColor(text, tier string) string {
switch tier {
case "high":
return w.aurora.BrightGreen(text).String()
case "medium":
return w.aurora.BrightYellow(text).String()
default:
return w.aurora.BrightBlack(text).String()
}
}
// formatScreen formats the output for showing on screen.
func (w *StandardWriter) formatScreen(output *ResultEvent) []byte {
builder := &bytes.Buffer{}
@@ -51,6 +63,13 @@ func (w *StandardWriter) formatScreen(output *ResultEvent) []byte {
builder.WriteString("[")
builder.WriteString(w.severityColors(output.Info.SeverityHolder.Severity))
builder.WriteString("] ")
if output.Confidence != "" {
label := output.Confidence + " (" + strconv.Itoa(output.ConfidenceScore) + ")"
builder.WriteString("[")
builder.WriteString(w.confidenceColor(label, output.Confidence))
builder.WriteString("] ")
}
}
if output.Matched != "" {
builder.WriteString(output.Matched)
+21 -1
View File
@@ -213,6 +213,12 @@ type ResultEvent struct {
CURLCommand string `json:"curl-command,omitempty"`
// MatcherStatus is the status of the match
MatcherStatus bool `json:"matcher-status"`
// Confidence is a heuristic detection-reliability tier (low/medium/high),
// derived from matcher composition and out-of-band confirmation, and is
// independent of severity.
Confidence string `json:"confidence,omitempty"`
// ConfidenceScore is the 0-100 score (QoD-style) behind Confidence.
ConfidenceScore int `json:"confidence-score"`
// Lines is the line count for the specified match
Lines []int `json:"matched-line,omitempty"`
// GlobalMatchers identifies whether the matches was detected in the response
@@ -237,6 +243,11 @@ type ResultEvent struct {
FileToIndexPosition map[string]int `json:"-"`
TemplateVerifier string `json:"-"`
Error string `json:"error,omitempty"`
// TemplateInfo is the info as the template wrote it, kept only when
// placeholders changed the text. The changed fields can carry response data,
// so they are redacted like the raw request and response. It is bookkeeping,
// so no exporter writes it; mongo serializes with bson tags, not json ones.
TemplateInfo *model.Info `json:"-" bson:"-"`
}
type IssueTrackerMetadata struct {
@@ -364,6 +375,9 @@ func (w *StandardWriter) Write(event *ResultEvent) error {
event.Response = redactWithPatterns(event.Response, patterns)
event.CURLCommand = redactWithPatterns(event.CURLCommand, patterns)
event.Matched = redactWithPatterns(event.Matched, patterns)
for _, field := range event.Info.FilledFields(event.TemplateInfo) {
*field.Value = redactWithPatterns(*field.Value, patterns)
}
event.Timestamp = time.Now()
@@ -556,6 +570,10 @@ func (w *StandardWriter) Close() {
}
}
// ConfidenceNone is the confidence of a failure event: no matcher fired, so
// there is no detection and its confidence score is 0.
const ConfidenceNone = "none"
// WriteFailure writes the failure event for template to file and/or screen.
func (w *StandardWriter) WriteFailure(wrappedEvent *InternalWrappedEvent) error {
if !w.matcherStatus {
@@ -606,7 +624,9 @@ func (w *StandardWriter) WriteFailure(wrappedEvent *InternalWrappedEvent) error
Request: types.ToString(event["request"]),
Response: types.ToString(event["response"]),
MatcherStatus: false,
Timestamp: time.Now(),
// nothing matched, so there is no detection to be confident about
Confidence: ConfidenceNone,
Timestamp: time.Now(),
//FIXME: this is workaround to encode the template when no results were found
TemplateEncoded: w.encodeTemplate(types.ToString(event["template-path"])),
Error: types.ToString(event["error"]),
+76
View File
@@ -1,13 +1,17 @@
package output
import (
"encoding/json"
"fmt"
"strings"
"testing"
"github.com/pkg/errors"
"github.com/projectdiscovery/goflags"
"github.com/projectdiscovery/nuclei/v3/pkg/model"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/stretchr/testify/require"
"go.mongodb.org/mongo-driver/bson"
)
func TestStandardWriterRequest(t *testing.T) {
@@ -51,6 +55,78 @@ func TestStandardWriterRequest(t *testing.T) {
})
}
func TestFormatScreenIncludesConfidenceScore(t *testing.T) {
w, err := NewStandardWriter(&types.Options{NoColor: true})
require.NoError(t, err)
t.Cleanup(func() { w.Close() })
formatted := string(w.formatScreen(&ResultEvent{
TemplateID: "test-template",
Type: "http",
Host: "example.com",
Confidence: "low",
ConfidenceScore: 0,
}))
require.Contains(t, formatted, "[low (0)]")
}
func TestResultEventSerializesZeroConfidenceScore(t *testing.T) {
data, err := json.Marshal(&ResultEvent{Confidence: "low", ConfidenceScore: 0})
require.NoError(t, err)
require.Contains(t, string(data), `"confidence-score":0`)
}
func TestWriteFailureReportsNoConfidence(t *testing.T) {
w, err := NewStandardWriter(&types.Options{JSONL: true, MatcherStatus: true})
require.NoError(t, err)
out := &testWriteCloser{}
w.outputFile = out
w.DisableStdout = true
err = w.WriteFailure(&InternalWrappedEvent{InternalEvent: InternalEvent{
"template-id": "failed-template",
"host": "https://example.com",
"type": "http",
}})
require.NoError(t, err)
var event map[string]interface{}
require.NoError(t, json.Unmarshal([]byte(strings.TrimSpace(out.String())), &event))
require.Equal(t, false, event["matcher-status"])
require.Equal(t, ConfidenceNone, event["confidence"])
require.Equal(t, float64(0), event["confidence-score"])
}
// Only the fields filled from a response are redacted; text the template wrote
// stays as it is, even next to a filled field.
func TestStandardWriterRedactsFilledInfo(t *testing.T) {
w, err := NewStandardWriter(&types.Options{Redact: goflags.StringSlice{"password"}})
require.NoError(t, err)
w.DisableStdout = true
written := model.Info{Description: "login with password: {{password}}", Remediation: "Rotate the password: it is public"}
filled := &ResultEvent{TemplateInfo: &written, Info: model.Info{Description: "login with password: admin", Remediation: written.Remediation}}
static := &ResultEvent{Info: model.Info{Description: "send the password: header"}}
require.NoError(t, w.Write(filled))
require.NoError(t, w.Write(static))
require.Equal(t, "login with password: ***", filled.Info.Description)
require.Equal(t, "Rotate the password: it is public", filled.Info.Remediation)
require.Equal(t, "send the password: header", static.Info.Description)
}
// The mongo exporter serializes results with bson, which ignores json tags.
func TestResultEventKeepsTemplateInfoOutOfBSON(t *testing.T) {
written := model.Info{Description: "{{host}} is exposed"}
data, err := bson.Marshal(&ResultEvent{TemplateInfo: &written, Info: model.Info{Description: "example.com is exposed"}})
require.NoError(t, err)
var doc bson.M
require.NoError(t, bson.Unmarshal(data, &doc))
require.NotContains(t, doc, "templateinfo")
require.Contains(t, doc, "info")
}
type testWriteCloser struct {
strings.Builder
}
+75
View File
@@ -0,0 +1,75 @@
// Package baseline implements per-host catch-all detection used to lower the
// confidence of likely false-positive matches.
//
// Many hosts answer every request with the same "catch-all" page (custom 404s,
// SPA shells, WAF interstitials, login walls). Templates whose matchers fire on
// such a page produce false positives. The approach here follows the black-box
// scanner literature (x8, Metasploit Differential, ReScan): learn the host's
// response to a request that almost certainly does not exist, then check whether
// a template's matchers ALSO fire on that baseline. Unlike body-diffing schemes,
// replaying the operators is binary and robust: if a matcher matches a known
// non-existent path, the detection is not specific to the real target.
package baseline
import "sync"
// maxHosts bounds the number of cached baselines to keep memory predictable on
// very large scans; beyond it, baselines are still fetched but not retained.
const maxHosts = 10000
// Map is a learned baseline: the DSL field map produced from the control
// response, suitable for replaying a template's compiled operators against.
type Map = map[string]interface{}
// Cache learns and stores a per-host catch-all baseline so it can be replayed
// against every template without re-fetching. Each host is fetched at most once,
// even under concurrent access.
type Cache struct {
mu sync.Mutex
entries map[string]*entry
}
type entry struct {
once sync.Once
data Map
ok bool
}
// New returns an empty baseline cache.
func New() *Cache {
return &Cache{entries: make(map[string]*entry)}
}
// GetOrFetch returns the baseline for host, invoking fetch exactly once per host
// to learn it. fetch should return the DSL map of the control response; a nil
// map or error marks the host as having no usable baseline. The second return
// value reports whether a usable baseline is available.
func (c *Cache) GetOrFetch(host string, fetch func() (Map, error)) (Map, bool) {
if c == nil || host == "" {
return nil, false
}
c.mu.Lock()
e, found := c.entries[host]
if !found {
// Don't grow the cache without bound. Over the cap we fetch directly so
// detection still works, we just don't retain the baseline.
if len(c.entries) >= maxHosts {
c.mu.Unlock()
data, err := fetch()
return data, err == nil && data != nil
}
e = &entry{}
c.entries[host] = e
}
c.mu.Unlock()
e.once.Do(func() {
data, err := fetch()
if err == nil && data != nil {
e.data = data
e.ok = true
}
})
return e.data, e.ok
}
@@ -0,0 +1,67 @@
package baseline
import (
"errors"
"sync"
"sync/atomic"
"testing"
"github.com/stretchr/testify/require"
)
func TestGetOrFetchCachesPerHost(t *testing.T) {
cache := New()
var calls atomic.Int32
fetch := func() (Map, error) {
calls.Add(1)
return Map{"status_code": 200}, nil
}
for i := 0; i < 5; i++ {
data, ok := cache.GetOrFetch("https://example.com", fetch)
require.True(t, ok)
require.Equal(t, 200, data["status_code"])
}
require.Equal(t, int32(1), calls.Load(), "baseline should be fetched once per host")
cache.GetOrFetch("https://other.com", fetch)
require.Equal(t, int32(2), calls.Load(), "distinct host should trigger a new fetch")
}
func TestGetOrFetchFetchesOnceUnderConcurrency(t *testing.T) {
cache := New()
var calls atomic.Int32
fetch := func() (Map, error) {
calls.Add(1)
return Map{"ok": true}, nil
}
var wg sync.WaitGroup
for i := 0; i < 50; i++ {
wg.Add(1)
go func() {
defer wg.Done()
cache.GetOrFetch("https://example.com", fetch)
}()
}
wg.Wait()
require.Equal(t, int32(1), calls.Load(), "concurrent access must fetch the baseline exactly once")
}
func TestGetOrFetchErrorNotCached(t *testing.T) {
cache := New()
_, ok := cache.GetOrFetch("https://example.com", func() (Map, error) {
return nil, errors.New("boom")
})
require.False(t, ok, "failed fetch should report no usable baseline")
}
func TestGetOrFetchNilReceiverOrEmptyHost(t *testing.T) {
var cache *Cache
_, ok := cache.GetOrFetch("https://example.com", func() (Map, error) { return Map{}, nil })
require.False(t, ok)
cache = New()
_, ok = cache.GetOrFetch("", func() (Map, error) { return Map{}, nil })
require.False(t, ok)
}
+189
View File
@@ -0,0 +1,189 @@
package http
import (
"context"
"crypto/rand"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"io"
"net/http"
"sort"
"strings"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/baseline"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/http/httpclientpool"
"github.com/projectdiscovery/retryablehttp-go"
httpUtils "github.com/projectdiscovery/utils/http"
urlutil "github.com/projectdiscovery/utils/url"
)
// ErrNoClient is returned when no http client is available to fetch the baseline.
var ErrNoClient = errors.New("no http client available for baseline request")
// BaselineMatchedKey marks an event whose match also fired against the host's
// catch-all baseline, signalling the scorer to lower confidence.
const BaselineMatchedKey = "baseline_matched"
// baselineMatched reports whether this template's matchers also fire against the
// host's catch-all baseline: a control request to a path that almost certainly
// does not exist. When they do, the match is not specific to the target (the
// host answers everything the same way) and is very likely a false positive, so
// the caller lowers the reported confidence.
//
// Replaying the compiled operators against the baseline is deliberately binary
// and needs no body normalization: we are not diffing responses, only asking
// "would this template have matched a non-existent path too?".
func (request *Request) baselineMatched(input *contextargs.Context, generatedRequest *generatedRequest) bool {
if request.options.BaselineCache == nil || request.CompiledOperators == nil {
return false
}
target := ""
if generatedRequest != nil {
target = generatedRequest.URL()
}
if target == "" && input != nil {
target = input.MetaInput.Input
}
parsed, err := urlutil.Parse(target)
if err != nil || parsed.Host == "" {
return false
}
baseURL := parsed.Scheme + "://" + parsed.Host
baselineRequest, client, cacheKey, err := request.prepareBaseline(input, baseURL, parsed.Host)
if err != nil {
return false
}
ctx := context.Background()
if input != nil {
ctx = input.Context()
}
baselineMap, ok := request.options.BaselineCache.GetOrFetch(cacheKey, func() (baseline.Map, error) {
return request.fetchBaseline(ctx, client, baselineRequest, baseURL)
})
if !ok || baselineMap == nil {
return false
}
result, matched := request.CompiledOperators.Execute(baselineMap, request.Match, request.Extract, false)
return matched && result != nil && result.Matched
}
// prepareBaseline creates a control request with the same effective custom
// headers, authentication, and per-input cookie jar as a normal request.
func (request *Request) prepareBaseline(input *contextargs.Context, baseURL, host string) (*retryablehttp.Request, *retryablehttp.Client, string, error) {
target := strings.TrimRight(baseURL, "/") + "/" + randomBaselinePath()
req, err := retryablehttp.NewRequest(http.MethodGet, target, nil)
if err != nil {
return nil, nil, "", err
}
generated := &generatedRequest{original: request, request: req}
request.setCustomHeaders(generated)
if !request.SkipSecretFile {
generated.ApplyAuth(request.options.AuthProvider)
}
connConfig := request.connConfiguration
if input != nil && input.CookieJar != nil && !request.DisableCookie {
connConfig = connConfig.Clone()
connConfig.Connection.SetCookieJar(input.CookieJar)
}
// a pinned target gets its own client, like the normal request path, so a
// keep-alive connection to one ip never serves a baseline pinned to another
poolKey := host
if input != nil && input.MetaInput != nil && input.MetaInput.CustomIP != "" {
poolKey += "@" + input.MetaInput.CustomIP
}
client, err := httpclientpool.Get(request.options.Options, connConfig, poolKey)
if err != nil {
client, err = httpclientpool.Get(request.options.Options, connConfig, "")
}
if err != nil || client == nil {
return nil, nil, "", ErrNoClient
}
return req, client, baselineContextKey(baseURL, req, input), nil
}
// baselineContextKey prevents authenticated baselines from being shared across
// different header, query-auth, host-header, or cookie contexts.
func baselineContextKey(baseURL string, req *retryablehttp.Request, input *contextargs.Context) string {
hash := sha256.New()
_, _ = fmt.Fprintf(hash, "host=%q\nquery=%q\n", req.Host, req.RawQuery)
if input != nil && input.MetaInput != nil {
_, _ = fmt.Fprintf(hash, "customip=%q\n", input.MetaInput.CustomIP)
}
keys := make([]string, 0, len(req.Header))
for key := range req.Header {
keys = append(keys, key)
}
sort.Strings(keys)
for _, key := range keys {
values := append([]string(nil), req.Header.Values(key)...)
sort.Strings(values)
_, _ = fmt.Fprintf(hash, "header=%q:%q\n", key, values)
}
if input != nil && input.CookieJar != nil && req.Request != nil && req.Request.URL != nil {
cookies := input.CookieJar.Cookies(req.Request.URL)
values := make([]string, 0, len(cookies))
for _, cookie := range cookies {
values = append(values, cookie.String())
}
sort.Strings(values)
for _, cookie := range values {
_, _ = fmt.Fprintf(hash, "cookie=%q\n", cookie)
}
}
return baseURL + "#" + hex.EncodeToString(hash.Sum(nil))
}
// fetchBaseline sends the prepared control request and converts the response
// into a DSL map a template's operators can replay against.
func (request *Request) fetchBaseline(ctx context.Context, client *retryablehttp.Client, req *retryablehttp.Request, baseURL string) (baseline.Map, error) {
// respect user rate limiting and host backoff for the extra control request,
// keyed on the full URL like the normal requests to the same host
hostKey := req.String()
if err := request.rateLimitTake(ctx, hostKey); err != nil {
return nil, err
}
resp, err := client.Do(req.WithContext(ctx))
request.observeHostBackoff(hostKey, resp, err)
if err != nil {
return nil, err
}
defer func() {
_, _ = io.Copy(io.Discard, resp.Body)
_ = resp.Body.Close()
}()
maxBodyLimit := int64(MaxBodyRead)
if request.MaxSize > 0 {
maxBodyLimit = int64(request.MaxSize)
}
if request.options.Options.ResponseReadSize != 0 {
maxBodyLimit = int64(request.options.Options.ResponseReadSize)
}
respChain := httpUtils.NewResponseChain(resp, maxBodyLimit)
defer respChain.Close()
if err := respChain.Fill(); err != nil {
return nil, err
}
dslMap := request.responseToDSLMap(respChain.Response(), baseURL, req.String(), "", respChain.FullResponseString(), respChain.BodyString(), respChain.HeadersString(), 0, nil)
return baseline.Map(dslMap), nil
}
// randomBaselinePath returns a high-entropy path that is almost certainly absent
// on the target, so a 200/match against it indicates catch-all behavior.
func randomBaselinePath() string {
buf := make([]byte, 16)
if _, err := rand.Read(buf); err != nil {
return "nuclei-baseline-control-path"
}
return hex.EncodeToString(buf)
}
+38
View File
@@ -0,0 +1,38 @@
package http
import (
"net/http"
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/retryablehttp-go"
"github.com/stretchr/testify/require"
)
func TestBaselineContextKeyScopesAuthentication(t *testing.T) {
const baseURL = "https://example.com"
input := contextargs.NewWithInput(t.Context(), baseURL)
req, err := retryablehttp.NewRequest(http.MethodGet, baseURL+"/control", nil)
require.NoError(t, err)
plainKey := baselineContextKey(baseURL, req, input)
req.Header.Set("Authorization", "Bearer token")
authKey := baselineContextKey(baseURL, req, input)
require.NotEqual(t, plainKey, authKey)
input.CookieJar.SetCookies(req.Request.URL, []*http.Cookie{{Name: "session", Value: "one"}})
cookieKey := baselineContextKey(baseURL, req, input)
require.NotEqual(t, authKey, cookieKey)
req.RawQuery = "api_key=secret"
queryKey := baselineContextKey(baseURL, req, input)
require.NotEqual(t, cookieKey, queryKey)
input.MetaInput.CustomIP = "192.0.2.10"
pinnedKey := baselineContextKey(baseURL, req, input)
require.NotEqual(t, queryKey, pinnedKey)
input.MetaInput.CustomIP = "192.0.2.11"
require.NotEqual(t, pinnedKey, baselineContextKey(baseURL, req, input))
}
+67
View File
@@ -302,6 +302,73 @@ func TestHTTPMakeResult(t *testing.T) {
require.Equal(t, 1, len(finalEvent.Results), "could not get correct number of results")
require.Equal(t, "test", finalEvent.Results[0].MatcherName, "could not get correct matcher name of results")
require.Equal(t, "1.1.1.1", finalEvent.Results[0].ExtractedResults[0], "could not get correct extracted results")
require.Equal(t, operators.ConfidenceMedium, finalEvent.Results[0].Confidence, "could not get correct confidence tier")
require.Equal(t, 63, finalEvent.Results[0].ConfidenceScore, "could not get correct confidence score")
// interactsh_protocol alone does not pin confidence: the template has no
// interactsh matcher, so the field may come from a header the target sets.
event["interactsh_protocol"] = "dns"
oobEvent := &output.InternalWrappedEvent{InternalEvent: event, OperatorsResult: finalEvent.OperatorsResult}
oobResults := request.MakeResultEvent(oobEvent)
require.NotEmpty(t, oobResults, "expected a result event")
require.Equal(t, operators.ConfidenceMedium, oobResults[0].Confidence, "a spoofable interactsh field must not raise confidence")
require.Equal(t, 63, oobResults[0].ConfidenceScore, "a spoofable interactsh field must not change the score")
// a catch-all baseline match clamps confidence down for in-band matches.
delete(event, "interactsh_protocol")
event[BaselineMatchedKey] = true
baselineEvent := &output.InternalWrappedEvent{InternalEvent: event, OperatorsResult: finalEvent.OperatorsResult}
baselineResults := request.MakeResultEvent(baselineEvent)
require.NotEmpty(t, baselineResults, "expected a baseline result event")
require.Equal(t, operators.ConfidenceLow, baselineResults[0].Confidence, "catch-all baseline match should be low confidence")
require.Equal(t, operators.ConfidenceCatchAll, baselineResults[0].ConfidenceScore, "catch-all baseline match should use the clamped score")
}
func TestHTTPMakeResultRendersInfo(t *testing.T) {
options := testutils.DefaultOptions
testutils.Init(options)
templateID := "testing-http-info"
request := &Request{
ID: templateID,
Path: []string{"{{BaseURL}}"},
Method: HTTPMethodTypeHolder{MethodType: HTTPGet},
Operators: operators.Operators{
Matchers: []*matchers.Matcher{{
Part: "body",
Type: matchers.MatcherTypeHolder{MatcherType: matchers.WordsMatcher},
Words: []string{"version"},
}},
Extractors: []*extractors.Extractor{{
Name: "version",
Part: "body",
Type: extractors.ExtractorTypeHolder{ExtractorType: extractors.RegexExtractor},
Regex: []string{`version: ([0-9.]+)`},
RegexGroup: 1,
Internal: true,
}},
},
}
description := "Version {{version}} on {{host}}:{{port}} accepts password {{replace_regex(password, '^(.{2}).*$', '$1***')}}"
executerOpts := testutils.NewMockExecuterOptions(options, &testutils.TemplateInfo{
ID: templateID,
Info: model.Info{SeverityHolder: severity.Holder{Severity: severity.High}, Name: "test", Description: description},
})
require.NoError(t, request.Compile(executerOpts))
resp := &http.Response{Header: make(http.Header)}
// The payload values of the request that matched travel in the event.
payloads := map[string]interface{}{"password": "s3cr3t"}
event := request.responseToDSLMap(resp, "https://example.com:8443/", "https://example.com:8443/", "", "", "server version: 7.2.4", "", time.Second, payloads)
finalEvent := &output.InternalWrappedEvent{InternalEvent: event}
result, ok := request.CompiledOperators.Execute(event, request.Match, request.Extract, false)
require.True(t, ok)
finalEvent.OperatorsResult = result
finalEvent.Results = request.MakeResultEvent(finalEvent)
require.Len(t, finalEvent.Results, 1)
require.Equal(t, "Version 7.2.4 on example.com:8443 accepts password s3***", finalEvent.Results[0].Info.Description)
require.Equal(t, description, executerOpts.TemplateInfo.Description, "the template info must stay as written")
}
const exampleRawRequest = `GET / HTTP/1.1
+8
View File
@@ -1639,6 +1639,14 @@ func (request *Request) executeRequest(input *contextargs.Context, generatedRequ
if internalWrappedEvent.InternalEvent["response"] == "" || internalWrappedEvent.InternalEvent["response"] == nil {
internalWrappedEvent.InternalEvent["response"] = getFullResponse()
}
// only worth a control request when there is an actual match; if the
// same matchers also fire against the host's catch-all baseline the
// finding is likely a false positive, so flag it for the scorer.
if request.options.BaselineCache != nil && internalWrappedEvent.OperatorsResult.Matched {
if request.baselineMatched(input, generatedRequest) {
internalWrappedEvent.InternalEvent[BaselineMatchedKey] = true
}
}
})
if hasInteractMatchers {
+80
View File
@@ -0,0 +1,80 @@
package protocols
import (
"maps"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/render"
)
// renderInfo fills the {{placeholders}} in the free text of each result's info
// with the values of the event that produced it, so a finding can say what was
// found and where. Name, severity, tags and classification stay as written:
// they identify the template for filtering, grouping and issue trackers.
//
// Each result holds its own copy of the info, so the template is never changed.
// When filling changes the text, the result keeps the info as written in
// TemplateInfo.
func renderInfo(results []*output.ResultEvent, data map[string]interface{}) []*output.ResultEvent {
// A global matcher matches on another template's event, with that template's
// payloads, variables and extracted values, so only the result fields fill
// its info.
if isGlobal, _ := data["global-matchers"].(bool); isGlobal {
data = nil
}
for _, result := range results {
if !result.Info.HasPlaceholders() {
continue
}
written := result.Info
values := infoValues(result, data)
for _, field := range result.Info.PlaceholderFields() {
if field.HasPlaceholders() {
*field.Value = renderInfoText(result.TemplateID, *field.Value, values)
}
}
if len(result.Info.FilledFields(&written)) > 0 {
result.TemplateInfo = &written
}
}
return results
}
// infoValues returns the event data with the result's own fields on top, under
// their names in the JSON output. Those win over the event's keys: in an http
// event host is the raw input URL, while the output reports the hostname.
func infoValues(result *output.ResultEvent, data map[string]interface{}) map[string]interface{} {
values := make(map[string]interface{}, len(data)+9)
maps.Copy(values, data)
for key, value := range map[string]string{
"host": result.Host,
"port": result.Port,
"ip": result.IP,
"url": result.URL,
"scheme": result.Scheme,
"path": result.Path,
"matched-at": result.Matched,
"matcher-name": result.MatcherName,
"extractor-name": result.ExtractorName,
} {
if value != "" {
values[key] = value
}
}
return values
}
// renderInfoText renders the text through the shared renderer, as requests and
// matchers are, so nested placeholders and helpers behave the same way. Scan
// options are left out, so network helpers stay denied while results are built.
// If an expression fails, the text stays as written, as a word matcher's does:
// a typo in the template must not cost the finding.
func renderInfoText(templateID, text string, values map[string]interface{}) string {
rendered, err := render.Render(render.Input{Text: text, Values: values})
if err != nil {
gologger.Verbose().Msgf("[%s] Could not render template info: %s\n", templateID, err)
return text
}
return rendered.Text
}
+120
View File
@@ -0,0 +1,120 @@
package protocols
import (
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/model"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/stretchr/testify/require"
)
func TestRenderInfoLeavesStaticInfoAlone(t *testing.T) {
result := &output.ResultEvent{Host: "example.com", Info: model.Info{Description: "Redis is exposed"}}
renderInfo([]*output.ResultEvent{result}, map[string]interface{}{"host": "example.com"})
require.Equal(t, "Redis is exposed", result.Info.Description)
require.Nil(t, result.TemplateInfo)
}
// Braces that fill nothing, like an SSTI payload in the description, leave the
// text static, so it is neither redacted nor moved off the sarif rule.
func TestRenderInfoLeavesUnfilledBracesStatic(t *testing.T) {
result := &output.ResultEvent{Host: "example.com", Info: model.Info{Description: "Send {{7*7}} or {{config}} in the name parameter"}}
renderInfo([]*output.ResultEvent{result}, map[string]interface{}{})
require.Equal(t, "Send {{7*7}} or {{config}} in the name parameter", result.Info.Description)
require.Nil(t, result.TemplateInfo)
}
func TestRenderInfoFillsResultAndEventValues(t *testing.T) {
result := &output.ResultEvent{
Host: "example.com",
Port: "6379",
MatcherName: "auth",
Info: model.Info{
Name: "Redis {{version}}",
Description: "{{matcher-name}} on {{host}}:{{port}}, version {{version}}",
Impact: "{{to_upper(password)}}",
Remediation: "{{credentials}} stays",
},
}
// The event's host is the raw input, while the output reports the hostname.
data := map[string]interface{}{"host": "redis://example.com:6379", "version": "7.2.4", "password": "admin"}
renderInfo([]*output.ResultEvent{result}, data)
require.Equal(t, "auth on example.com:6379, version 7.2.4", result.Info.Description)
require.Equal(t, "ADMIN", result.Info.Impact)
require.Equal(t, "{{credentials}} stays", result.Info.Remediation)
require.Equal(t, "Redis {{version}}", result.Info.Name)
require.Equal(t, "redis://example.com:6379", data["host"])
require.NotNil(t, result.TemplateInfo)
require.Equal(t, "{{matcher-name}} on {{host}}:{{port}}, version {{version}}", result.TemplateInfo.Description)
require.Equal(t, "{{to_upper(password)}}", result.TemplateInfo.Impact)
}
// Info renders like requests do, so the usual nested placeholder idiom and an
// expression spread over lines work too.
func TestRenderInfoFillsNestedAndMultilinePlaceholders(t *testing.T) {
result := &output.ResultEvent{Info: model.Info{
Description: "Authorization: Basic {{base64('{{username}}:{{password}}')}}",
Remediation: "Upgrade from {{join(\", \",\n versions)}}",
}}
renderInfo([]*output.ResultEvent{result}, map[string]interface{}{
"username": "admin", "password": "secret", "versions": []string{"7.2.4", "6.0.9"},
})
require.Equal(t, "Authorization: Basic YWRtaW46c2VjcmV0", result.Info.Description)
require.Equal(t, "Upgrade from 7.2.4, 6.0.9", result.Info.Remediation)
}
// A hyphenated name reads as subtraction inside a helper unless it is bracketed,
// and the general §name§ marker fills like a {{name}} one.
func TestRenderInfoFillsBracketedNamesAndGeneralMarkers(t *testing.T) {
result := &output.ResultEvent{MatcherName: "auth", Info: model.Info{
Description: "{{to_upper([matcher-name])}} matched",
Impact: "default login §username§ accepted",
}}
renderInfo([]*output.ResultEvent{result}, map[string]interface{}{"username": "admin"})
require.Equal(t, "AUTH matched", result.Info.Description)
require.Equal(t, "default login admin accepted", result.Info.Impact)
}
// A failing expression leaves its field as written; the other fields render.
func TestRenderInfoKeepsAFieldWithAFailingExpression(t *testing.T) {
result := &output.ResultEvent{Host: "example.com", Info: model.Info{
Description: "{{host}} {{to_upper(missing)}}",
Impact: "{{host}} is exposed",
}}
renderInfo([]*output.ResultEvent{result}, map[string]interface{}{})
require.Equal(t, "{{host}} {{to_upper(missing)}}", result.Info.Description)
require.Equal(t, "example.com is exposed", result.Info.Impact)
require.Len(t, result.Info.FilledFields(result.TemplateInfo), 1)
}
// A global matcher's event belongs to the template whose response it matched,
// so only the result fields fill its info.
func TestRenderInfoFillsGlobalMatcherInfoFromResultFieldsOnly(t *testing.T) {
result := &output.ResultEvent{Host: "example.com", Info: model.Info{Description: "Leaked {{token}} on {{host}}"}}
renderInfo([]*output.ResultEvent{result}, map[string]interface{}{"global-matchers": true, "token": "origin-template-payload"})
require.Equal(t, "Leaked {{token}} on example.com", result.Info.Description)
}
// Results are built without scan options, so a network helper never runs.
func TestRenderInfoDeniesNetworkHelpers(t *testing.T) {
result := &output.ResultEvent{Host: "example.com", Info: model.Info{Description: "{{host}} resolves to {{resolve(host)}}"}}
renderInfo([]*output.ResultEvent{result}, map[string]interface{}{})
require.Equal(t, "{{host}} resolves to {{resolve(host)}}", result.Info.Description)
require.Nil(t, result.TemplateInfo)
}
+79 -38
View File
@@ -9,54 +9,95 @@ import (
"strings"
)
var (
noMinor = regexp.MustCompile(`^HTTP/([0-9]) `)
responseStatusRegex = regexp.MustCompile(`(?m)^HTTP/[0-9](?:\.[0-9])?\s+[0-9]{3}`)
)
// noMinor matches a leading status-line protocol with no minor version, such
// as "HTTP/2 ". The go parser only accepts HTTP/X.Y.
var noMinor = regexp.MustCompile(`^HTTP/[0-9] `)
// readResponseFromString reads a raw http response from a string.
// errNotRequest marks data that does not start with an HTTP request, so the
// caller parses it as a bare response instead.
var errNotRequest = errors.New("not an http request")
// readResponseFromString reads one raw HTTP response. Data may be the response
// itself or a request followed by that response. A status line is recognized
// only where a response starts, never by searching the body for "HTTP/".
func readResponseFromString(data string) (*http.Response, error) {
// Check if "data" contains RFC compatible Request followed by a response
br := bufio.NewReader(strings.NewReader(data))
if req, err := http.ReadRequest(br); err == nil {
if req.Body != nil {
_, _ = io.Copy(io.Discard, req.Body)
_ = req.Body.Close()
resp, err := readRequestResponse(data)
if err == nil || !errors.Is(err, errNotRequest) {
return resp, err
}
return readResponse(data, nil)
}
// readRequestResponse reads a raw HTTP request from the start of data and the
// response that follows it.
func readRequestResponse(data string) (*http.Response, error) {
sr := strings.NewReader(data)
br := bufio.NewReader(sr)
// unread bytes are either still in sr or buffered in br
unread := func() string { return data[len(data)-sr.Len()-br.Buffered():] }
req, err := http.ReadRequest(br)
if err != nil {
return nil, errNotRequest
}
afterHeaders := unread()
// the response follows the request body, so it has to be read first.
// a dump that keeps Content-Length but drops the body is tried at the headers.
if _, err := io.Copy(io.Discard, req.Body); err == nil {
if resp, err := readResponse(unread(), req); err == nil {
return resp, nil
}
remaining, _ := io.ReadAll(br)
if len(remaining) > 0 {
remainingStr := addMinorVersionToHTTP(string(remaining))
if resp, err := http.ReadResponse(bufio.NewReader(strings.NewReader(remainingStr)), req); err == nil {
return resp, nil
}
return readResponse(afterHeaders, req)
}
// maxInterimResponses bounds how many interim responses are skipped before the
// final one. Each skip copies the remainder of data.
const maxInterimResponses = 5
// readResponse reads a raw HTTP response whose status line starts data, after
// any leading blank lines. Interim responses such as 100 Continue are skipped,
// as the HTTP client does. The last interim response is returned when no final
// response follows it.
func readResponse(data string, req *http.Request) (*http.Response, error) {
var lastInterim *http.Response
for range maxInterimResponses + 1 {
data = strings.TrimLeft(data, "\r\n")
data = addMinorVersionToHTTP(data)
sr := strings.NewReader(data)
br := bufio.NewReader(sr)
resp, err := http.ReadResponse(br, req)
if err != nil {
if lastInterim != nil {
return lastInterim, nil
}
return nil, err
}
}
// otherwise tries to patch known cases such as http minor version
var final string
if strings.HasPrefix(data, "HTTP/") {
final = addMinorVersionToHTTP(data)
} else {
loc := responseStatusRegex.FindStringIndex(data)
if loc == nil {
return nil, errors.New("malformed raw http response")
if !isInterimResponse(resp) {
return resp, nil
}
final = addMinorVersionToHTTP(data[loc[0]:])
if lastInterim != nil && lastInterim.Body != nil {
_ = lastInterim.Body.Close()
}
lastInterim = resp
data = data[len(data)-sr.Len()-br.Buffered():]
}
return http.ReadResponse(bufio.NewReader(strings.NewReader(final)), nil)
return lastInterim, nil
}
// addMinorVersionToHTTP tries to add a minor version to http status header
// fixing the compatibility issue with go standard library.
// isInterimResponse reports whether resp is an interim response, such as
// 100 Continue, that comes before the final response. 101 Switching Protocols
// is the response.
func isInterimResponse(resp *http.Response) bool {
return resp != nil && resp.StatusCode >= 100 && resp.StatusCode <= 199 && resp.StatusCode != http.StatusSwitchingProtocols
}
// addMinorVersionToHTTP inserts ".0" into a leading protocol such as HTTP/2.
// Only those bytes change, so the reason phrase, headers, and body stay intact.
func addMinorVersionToHTTP(data string) string {
matches := noMinor.FindAllStringSubmatch(data, 1)
if len(matches) == 0 {
if !noMinor.MatchString(data) {
return data
}
if len(matches[0]) < 2 {
return data
}
replacedVersion := strings.Replace(matches[0][0], matches[0][1], matches[0][1]+".0", 1)
data = strings.Replace(data, matches[0][0], replacedVersion, 1)
return data
const protoLen = len("HTTP/X")
return data[:protoLen] + ".0" + data[protoLen:]
}
+183 -21
View File
@@ -6,6 +6,7 @@ import (
"net/http"
"net/http/httptest"
"net/http/httputil"
"strings"
"testing"
"time"
@@ -200,31 +201,101 @@ Server: Google Frontend
})
}
func TestReadResponseFromStringPreservesHTTPVersionText(t *testing.T) {
const body = "Supports HTTP/2 and HTTP/3."
func TestReadResponse_HTTPInBodyCases(t *testing.T) {
tests := []struct {
name string
data string
expectedBody string
statusCode int
name string
data string
wantProto string
wantStatus string
wantVia string
wantMethod string
}{
{
name: "http2 response without minor version and HTTP/ in body (#7924)",
data: "GET / HTTP/1.1\r\nHost: example.com\r\n\r\nHTTP/2 200 OK\r\nContent-Length: 27\r\n\r\nSupports HTTP/2 and HTTP/3.",
expectedBody: "Supports HTTP/2 and HTTP/3.",
statusCode: 200,
name: "body-with-content-length",
data: "HTTP/1.1 200 OK\r\nContent-Length: 27\r\n\r\n" + body,
wantProto: "HTTP/1.1",
wantStatus: "200 OK",
},
{
name: "request with body and HTTP/ in response body (#7924)",
data: "POST /search HTTP/1.1\r\nHost: example.com\r\nContent-Length: 3\r\n\r\nq=1HTTP/1.1 200 OK\r\nContent-Length: 27\r\n\r\nSupports HTTP/2 and HTTP/3.",
expectedBody: "Supports HTTP/2 and HTTP/3.",
statusCode: 200,
name: "body-without-content-length",
data: "HTTP/1.1 200 OK\r\n\r\n" + body,
wantProto: "HTTP/1.1",
wantStatus: "200 OK",
},
{
name: "control http 1.1 with HTTP/ in response body (#7924)",
data: "GET / HTTP/1.1\r\nHost: example.com\r\n\r\nHTTP/1.1 200 OK\r\nContent-Length: 27\r\n\r\nSupports HTTP/2 and HTTP/3.",
expectedBody: "Supports HTTP/2 and HTTP/3.",
statusCode: 200,
name: "body-with-chunked-encoding",
data: "HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n1b\r\n" + body + "\r\n0\r\n\r\n",
wantProto: "HTTP/1.1",
wantStatus: "200 OK",
},
{
name: "header-value",
data: "HTTP/1.1 200 OK\r\nVia: HTTP/2 edge\r\nContent-Length: 27\r\n\r\n" + body,
wantProto: "HTTP/1.1",
wantStatus: "200 OK",
wantVia: "HTTP/2 edge",
},
{
name: "reason-phrase",
data: "HTTP/1.1 200 Use HTTP/2 instead\r\nContent-Length: 27\r\n\r\n" + body,
wantProto: "HTTP/1.1",
wantStatus: "200 Use HTTP/2 instead",
},
{
name: "status-line-without-minor-version",
data: "HTTP/2 200 OK\r\nVia: HTTP/2 edge\r\nContent-Length: 27\r\n\r\n" + body,
wantProto: "HTTP/2.0",
wantStatus: "200 OK",
wantVia: "HTTP/2 edge",
},
{
name: "http3-status-line-without-minor-version",
data: "HTTP/3 200 OK\r\nContent-Length: 27\r\n\r\n" + body,
wantProto: "HTTP/3.0",
wantStatus: "200 OK",
},
{
name: "request-then-status-line-without-minor-version",
data: "GET / HTTP/1.1\r\nHost: example.com\r\n\r\nHTTP/2 200 OK\r\nContent-Length: 27\r\n\r\n" + body,
wantProto: "HTTP/2.0",
wantStatus: "200 OK",
wantMethod: "GET",
},
{
name: "blank-line-before-response",
data: "GET / HTTP/1.1\r\nHost: example.com\r\n\r\n\r\nHTTP/1.1 200 OK\r\nContent-Length: 27\r\n\r\n" + body,
wantProto: "HTTP/1.1",
wantStatus: "200 OK",
wantMethod: "GET",
},
{
name: "request-with-body-then-response",
data: "POST /search HTTP/1.1\r\nHost: example.com\r\nContent-Length: 3\r\n\r\nq=1HTTP/1.1 200 OK\r\nContent-Length: 27\r\n\r\n" + body,
wantProto: "HTTP/1.1",
wantStatus: "200 OK",
wantMethod: "POST",
},
{
name: "request-without-saved-body-then-response",
data: "POST /search HTTP/1.1\r\nHost: example.com\r\nContent-Length: 3\r\n\r\nHTTP/1.1 200 OK\r\nContent-Length: 27\r\n\r\n" + body,
wantProto: "HTTP/1.1",
wantStatus: "200 OK",
wantMethod: "POST",
},
{
name: "request-then-interim-response",
data: "POST /search HTTP/1.1\r\nHost: example.com\r\nExpect: 100-continue\r\nContent-Length: 3\r\n\r\nq=1HTTP/1.1 100 Continue\r\n\r\nHTTP/1.1 200 OK\r\nContent-Length: 27\r\n\r\n" + body,
wantProto: "HTTP/1.1",
wantStatus: "200 OK",
wantMethod: "POST",
},
{
name: "interim-response-without-minor-version",
data: "HTTP/2 103 Early Hints\r\nLink: </style.css>; rel=preload\r\n\r\nHTTP/2 200 OK\r\nContent-Length: 27\r\n\r\n" + body,
wantProto: "HTTP/2.0",
wantStatus: "200 OK",
},
}
@@ -232,11 +303,102 @@ func TestReadResponse_HTTPInBodyCases(t *testing.T) {
t.Run(tt.name, func(t *testing.T) {
resp, err := readResponseFromString(tt.data)
require.NoError(t, err, "could not read response from string")
require.Equal(t, tt.statusCode, resp.StatusCode)
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
require.Equal(t, tt.expectedBody, string(body))
respBody, err := io.ReadAll(resp.Body)
require.NoError(t, err, "could not read response body")
require.Equal(t, body, string(respBody), "could not get correct parsed body")
require.Equal(t, tt.wantProto, resp.Proto, "could not get correct protocol")
require.Equal(t, tt.wantStatus, resp.Status, "could not get correct status")
require.Equal(t, tt.wantVia, resp.Header.Get("Via"), "could not get correct headers")
var gotMethod string
if resp.Request != nil {
gotMethod = resp.Request.Method
}
require.Equal(t, tt.wantMethod, gotMethod, "could not get correct request")
})
}
}
func TestReadResponseFromStringInterimResponses(t *testing.T) {
tests := []struct {
name string
data string
wantStatus string
wantMethod string
}{
{
name: "interim-response-only",
data: "HTTP/1.1 100 Continue\r\n\r\n",
wantStatus: "100 Continue",
},
{
name: "request-then-interim-response-only",
data: "POST /search HTTP/1.1\r\nHost: example.com\r\nContent-Length: 3\r\n\r\nq=1HTTP/1.1 100 Continue\r\n\r\n",
wantStatus: "100 Continue",
wantMethod: "POST",
},
{
name: "request-then-interim-then-final-response",
data: "POST /search HTTP/1.1\r\nHost: example.com\r\nExpect: 100-continue\r\nContent-Length: 3\r\n\r\nq=1HTTP/1.1 100 Continue\r\n\r\nHTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok",
wantStatus: "200 OK",
wantMethod: "POST",
},
{
name: "request-then-interim-then-final-response-with-status-line-in-body",
data: "GET / HTTP/1.1\r\nHost: example.com\r\n\r\nHTTP/1.1 100 Continue\r\n\r\nHTTP/1.1 200 OK\r\n\r\nsee HTTP/1.1 404 fake\r\n\r\nx",
wantStatus: "200 OK",
wantMethod: "GET",
},
{
name: "request-then-interim-then-blank-line-then-final-response-with-status-line-in-body",
data: "GET / HTTP/1.1\r\nHost: example.com\r\n\r\nHTTP/1.1 100 Continue\r\n\r\n\r\nHTTP/1.1 200 OK\r\n\r\nsee HTTP/1.1 404 fake\r\n\r\nx",
wantStatus: "200 OK",
wantMethod: "GET",
},
{
name: "too-many-interim-responses",
data: strings.Repeat("HTTP/2 103 Early Hints\r\n\r\n", 6) + "HTTP/2 200 OK\r\n\r\n",
wantStatus: "103 Early Hints",
},
{
name: "request-then-interim-then-blank-line-then-final-response",
data: "POST /search HTTP/1.1\r\nHost: example.com\r\nExpect: 100-continue\r\nContent-Length: 3\r\n\r\nq=1HTTP/1.1 100 Continue\r\n\r\n\r\nHTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok",
wantStatus: "200 OK",
wantMethod: "POST",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
resp, err := readResponseFromString(tt.data)
require.NoError(t, err, "could not read response from string")
require.Equal(t, tt.wantStatus, resp.Status, "could not get correct status")
var gotMethod string
if resp.Request != nil {
gotMethod = resp.Request.Method
}
require.Equal(t, tt.wantMethod, gotMethod, "could not get correct request")
})
}
}
func TestReadResponseFromStringStopsAtFirstResponse(t *testing.T) {
data := "GET / HTTP/1.1\r\nHost: example.com\r\n\r\n\r\n" +
"HTTP/1.1 301 Moved Permanently\r\nLocation: /next\r\nContent-Length: 0\r\n\r\n" +
"GET /next HTTP/1.1\r\nHost: example.com\r\n\r\n" +
"HTTP/1.1 200 OK\r\nContent-Length: 4\r\n\r\nbody"
resp, err := readResponseFromString(data)
require.NoError(t, err)
require.Equal(t, "301 Moved Permanently", resp.Status)
require.Equal(t, "GET", resp.Request.Method)
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
require.Empty(t, body)
}
func TestReadResponseFromStringDoesNotSearchForHTTP(t *testing.T) {
_, err := readResponseFromString("prefix HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok")
require.Error(t, err)
}
+56 -4
View File
@@ -30,6 +30,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/progress"
"github.com/projectdiscovery/nuclei/v3/pkg/projectfile"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/baseline"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/globalmatchers"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/hostbackoff"
@@ -118,6 +119,10 @@ type ExecutorOptions struct {
InteractshScope *interactsh.RequestScope
// HostErrorsCache is an optional cache for handling host errors
HostErrorsCache hosterrorscache.CacheInterface
// BaselineCache is an optional per-host catch-all baseline cache used to
// lower confidence on likely false-positive matches. Enabled via the
// ConfidenceBaseline option.
BaselineCache *baseline.Cache
// HostBackoff paces a host that signals it is being overloaded, so a scan
// slows down for it instead of being blocked outright. Nil disables it.
HostBackoff *hostbackoff.Governor
@@ -397,6 +402,7 @@ func (e *ExecutorOptions) Copy() *ExecutorOptions {
Browser: e.Browser,
Interactsh: e.Interactsh,
HostErrorsCache: e.HostErrorsCache,
BaselineCache: e.BaselineCache,
InteractshScope: e.InteractshScope,
HostBackoff: e.HostBackoff,
StopAtFirstMatch: e.StopAtFirstMatch,
@@ -555,11 +561,27 @@ func MakeDefaultResultEvent(request Request, wrapped *output.InternalWrappedEven
results := make([]*output.ResultEvent, 0, len(wrapped.OperatorsResult.Matches)+1)
// confidence is a static, deterministic signal derived from the matcher
// composition, so it is identical for every result event of this match.
confidenceScore, confidenceTier := wrapped.OperatorsResult.Operators.Confidence()
// out-of-band confirmation (an interactsh interaction) is unambiguous proof
// of the issue, so it outranks any in-band static score. A catch-all baseline
// match (the same matchers fire on a non-existent path) only clamps in-band
// matches: real out-of-band proof is unaffected.
switch {
case isOutOfBandConfirmed(wrapped.OperatorsResult.Operators, wrapped.InternalEvent):
confidenceScore, confidenceTier = operators.ConfidenceCertain, operators.ConfidenceHigh
case isCatchAllBaselineMatch(wrapped.InternalEvent):
confidenceScore, confidenceTier = operators.ConfidenceCatchAll, operators.ConfidenceLow
}
// If we have multiple matchers with names, write each of them separately.
if len(wrapped.OperatorsResult.Matches) > 0 {
for matcherNames := range wrapped.OperatorsResult.Matches {
data := request.MakeResultEventItem(wrapped)
data.MatcherName = matcherNames
data.Confidence = confidenceTier
data.ConfidenceScore = confidenceScore
results = append(results, data)
}
} else if len(wrapped.OperatorsResult.Extracts) > 0 {
@@ -567,16 +589,46 @@ func MakeDefaultResultEvent(request Request, wrapped *output.InternalWrappedEven
data := request.MakeResultEventItem(wrapped)
data.ExtractorName = k
data.ExtractedResults = v
data.Confidence = confidenceTier
data.ConfidenceScore = confidenceScore
results = append(results, data)
}
} else {
data := request.MakeResultEventItem(wrapped)
data.Confidence = confidenceTier
data.ConfidenceScore = confidenceScore
results = append(results, data)
}
// Every protocol builds its results here, so attaching the llm audit at this
// one point keeps the verdict on findings from all of them rather than only
// the protocol that remembers to ask. It is a no-op when no llm operator ran.
return AttachLLMAudits(results, wrapped.InternalEvent)
// Every protocol builds its results here, so attaching the llm audit and
// rendering the info at this one point covers findings from all of them
// rather than only the protocol that remembers to ask. Each is a no-op when
// it has nothing to do: no llm operator ran, or the info has no placeholders.
results = AttachLLMAudits(results, wrapped.InternalEvent)
return renderInfo(results, wrapped.InternalEvent)
}
// isOutOfBandConfirmed reports whether the match was corroborated by an
// out-of-band interaction (interactsh), which the interactsh client records on
// the event before re-running the result builder. A response header named
// Interactsh-Protocol also lands on this key, so only templates that match on
// interactsh data, whose results come from the interactsh client, count.
func isOutOfBandConfirmed(ops *operators.Operators, event output.InternalEvent) bool {
if event == nil || !interactsh.HasMatchers(ops) {
return false
}
return types.ToString(event["interactsh_protocol"]) != ""
}
// isCatchAllBaselineMatch reports whether the match also fired against the
// host's catch-all baseline (set by the HTTP baseline detector), indicating a
// likely false positive. The key mirrors http.BaselineMatchedKey; it is not
// referenced directly to avoid an import cycle.
func isCatchAllBaselineMatch(event output.InternalEvent) bool {
if event == nil {
return false
}
matched, _ := event["baseline_matched"].(bool)
return matched
}
// MakeDefaultExtractFunc performs extracting operation for an extractor on model and returns true or false.
+12
View File
@@ -7,6 +7,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/operators"
"github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
"github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/interactsh"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/stretchr/testify/require"
@@ -145,3 +146,14 @@ func TestExecutorOptionsApplyNewEngineOptionsReplacesInteractshScope(t *testing.
require.Same(t, next, reused.InteractshScope)
}
func TestIsOutOfBandConfirmedNeedsInteractshMatchers(t *testing.T) {
event := output.InternalEvent{"interactsh_protocol": "dns"}
bodyOnly := &operators.Operators{Matchers: []*matchers.Matcher{{Part: "body", Words: []string{"hi"}}}}
oob := &operators.Operators{Matchers: []*matchers.Matcher{{Part: "interactsh_protocol", Words: []string{"dns"}}}}
// a response header named Interactsh-Protocol sets the same key
require.False(t, isOutOfBandConfirmed(bodyOnly, event))
require.True(t, isOutOfBandConfirmed(oob, event))
require.False(t, isOutOfBandConfirmed(oob, output.InternalEvent{}))
}
+5
View File
@@ -194,6 +194,11 @@ func renderFindings(doc *fpdf.Fpdf, results []output.ResultEvent) {
doc.CellFormat(0, 5, safeStr(r.Host), "0", 1, "", false, 0, "")
doc.CellFormat(30, 5, "Template:", "0", 0, "", false, 0, "")
doc.CellFormat(0, 5, safeStr(r.TemplateID), "0", 1, "", false, 0, "")
if r.Confidence != "" {
confidence := fmt.Sprintf("%s (%d)", capitalize(r.Confidence), r.ConfidenceScore)
doc.CellFormat(30, 5, "Confidence:", "0", 0, "", false, 0, "")
doc.CellFormat(0, 5, safeStr(confidence), "0", 1, "", false, 0, "")
}
if r.Info.Description != "" {
doc.SetFont("Helvetica", "I", 8)
doc.SetTextColor(60, 60, 60)
+37 -4
View File
@@ -20,6 +20,11 @@ const (
fallbackTemplateRuleID = "nuclei-template"
)
// maxInfoValueLength keeps filled info whole for every template in
// nuclei-templates (the longest field is under 3000 characters) while a
// placeholder such as {{body}} cannot pour a whole response into the report.
const maxInfoValueLength = 4096
// Exporter is an exporter for nuclei sarif output format.
type Exporter struct {
sarif *sarif.Report
@@ -130,6 +135,7 @@ func (exporter *Exporter) Export(event *output.ResultEvent) error {
}
// rule contain details of template
templateEvent := asWritten(event)
rule := sarif.ReportingDescriptor{
Id: ruleID,
Name: ruleName,
@@ -138,11 +144,11 @@ func (exporter *Exporter) Export(event *output.ResultEvent) error {
Text: ruleName,
},
FullDescription: &sarif.MultiformatMessageString{
Text: buildRuleDescription(event, ruleName),
Text: buildRuleDescription(templateEvent, ruleName),
},
Help: &sarif.MultiformatMessageString{
Text: buildRuleHelpText(event),
Markdown: buildRuleHelpMarkdown(event, ruleName),
Text: buildRuleHelpText(templateEvent),
Markdown: buildRuleHelpMarkdown(templateEvent, ruleName),
},
Properties: ghMeta,
}
@@ -156,6 +162,14 @@ func (exporter *Exporter) Export(event *output.ResultEvent) error {
}
location := buildLocation(event, resultTarget)
resultProperties := buildResultProperties(event, resultTarget)
if event.Confidence != "" {
if resultProperties == nil {
resultProperties = map[string]interface{}{}
}
resultProperties["confidence"] = event.Confidence
resultProperties["confidence-score"] = event.ConfidenceScore
}
// vulnerability report/result
result := &sarif.Result{
@@ -168,7 +182,7 @@ func (exporter *Exporter) Export(event *output.ResultEvent) error {
Rule: sarif.ReportingDescriptorReference{
Id: ruleID,
},
Properties: buildResultProperties(event, resultTarget),
Properties: resultProperties,
}
exporter.sarif.RegisterResult(*result)
@@ -203,6 +217,18 @@ func (exporter *Exporter) Close() error {
return nil
}
// asWritten returns the event with its info as the template wrote it. Every
// finding of a template shares one rule, so the rule describes the template;
// text filled for one finding goes on that finding's result message instead.
func asWritten(event *output.ResultEvent) *output.ResultEvent {
if event.TemplateInfo == nil {
return event
}
written := *event
written.Info = *event.TemplateInfo
return &written
}
func buildRuleDescription(event *output.ResultEvent, ruleName string) string {
description := normalizeValue(event.Info.Description)
if description == "" {
@@ -275,6 +301,13 @@ func buildRuleHelpMarkdown(event *output.ResultEvent, ruleName string) string {
func buildResultMessage(event *output.ResultEvent, resultHeader, ruleID string) *sarif.Message {
detailLines := []string{}
// Filled info is all the report has of this finding's text, so it gets a
// longer limit than the other details.
for _, field := range event.Info.FilledFields(event.TemplateInfo) {
if value := normalizeAndTruncate(*field.Value, maxInfoValueLength); value != "" {
detailLines = append(detailLines, field.Label+": "+value)
}
}
target := getResultTarget(event)
addDetail(&detailLines, "Target", target)
if matched := normalizeAndTruncate(event.Matched, maxDetailValueLength); matched != "" && matched != target {
+114 -15
View File
@@ -20,22 +20,22 @@ import (
// makeEvent returns a fully populated ResultEvent suitable for the integration tests.
func makeEvent() *output.ResultEvent {
return &output.ResultEvent{
TemplateID: "test-template",
TemplateURL: "https://templates.example.com/test-template.yaml",
Template: "/tmp/nuclei-templates/cves/test-template.yaml",
TemplatePath: "/tmp/nuclei-templates/cves/test-template.yaml",
Type: "http",
Host: "example.com",
URL: "https://example.com",
Path: "/admin",
Matched: "https://example.com/admin",
IP: "93.184.216.34",
MatcherName: "status-200",
ExtractorName: "version",
TemplateID: "test-template",
TemplateURL: "https://templates.example.com/test-template.yaml",
Template: "/tmp/nuclei-templates/cves/test-template.yaml",
TemplatePath: "/tmp/nuclei-templates/cves/test-template.yaml",
Type: "http",
Host: "example.com",
URL: "https://example.com",
Path: "/admin",
Matched: "https://example.com/admin",
IP: "93.184.216.34",
MatcherName: "status-200",
ExtractorName: "version",
ExtractedResults: []string{"v1.2.3", "v1.2.4"},
CURLCommand: "curl -X GET https://example.com/admin",
Lines: []int{0, 42},
Timestamp: time.Now(),
CURLCommand: "curl -X GET https://example.com/admin",
Lines: []int{0, 42},
Timestamp: time.Now(),
Info: model.Info{
Name: "Test Finding",
Authors: stringslice.StringSlice{Value: "tester"},
@@ -431,6 +431,78 @@ func TestExport_StableRuleIndices(t *testing.T) {
}
}
// The rule describes the template whatever finding arrives first, and each
// finding carries the text filled for it, while text the template wrote stays
// on the rule.
func TestExport_FilledInfoStaysOnTheFinding(t *testing.T) {
out := filepath.Join(t.TempDir(), "report.sarif")
exp, err := New(&Options{File: out})
require.NoError(t, err)
written := makeEvent().Info
written.Description = "Service {{version}} found on {{host}}:6379"
first := makeEvent()
first.TemplateInfo = &written
first.Info.Description = "Service 6.0.9 found on 10.0.0.2:6379"
second := makeEvent()
second.TemplateInfo = &written
second.Info.Description = "Service 7.2.4 found on 10.0.0.1:6379"
require.NoError(t, exp.Export(first))
require.NoError(t, exp.Export(second))
require.NoError(t, exp.Close())
bin, err := os.ReadFile(out)
require.NoError(t, err)
var log upstream.SarifLog
require.NoError(t, json.Unmarshal(bin, &log))
run := log.Runs[0]
require.Len(t, run.Tool.Driver.Rules, 1)
rule := run.Tool.Driver.Rules[0]
require.Contains(t, rule.FullDescription.Text, "Service {{version}} found on {{host}}:6379")
require.NotContains(t, rule.FullDescription.Text, "6.0.9")
require.Contains(t, rule.Help.Text, "Impact: Severe impact")
require.Len(t, run.Result, 2)
require.Contains(t, run.Result[0].Message.Text, "Description: Service 6.0.9 found on 10.0.0.2:6379")
require.Contains(t, run.Result[1].Message.Text, "Description: Service 7.2.4 found on 10.0.0.1:6379")
require.NotContains(t, run.Result[1].Message.Text, "Impact:")
}
func TestBuildResultMessageKeepsFilledInfoWhole(t *testing.T) {
event := makeEvent()
written := event.Info
event.TemplateInfo = &written
event.Info.Description = strings.Repeat("x", 2*maxDetailValueLength)
message := buildResultMessage(event, "header", "test-template")
require.Contains(t, message.Text, "Description: "+event.Info.Description)
}
// A placeholder such as {{body}} cannot pour a whole response into the report.
func TestBuildResultMessageCapsFilledInfo(t *testing.T) {
event := makeEvent()
written := event.Info
event.TemplateInfo = &written
event.Info.Description = strings.Repeat("x", 2*maxInfoValueLength)
message := buildResultMessage(event, "header", "test-template")
require.Contains(t, message.Text, "Description: "+strings.Repeat("x", maxInfoValueLength-3)+"...")
require.NotContains(t, message.Text, strings.Repeat("x", maxInfoValueLength))
}
// Static info describes the template, so it stays on the rule as before.
func TestExport_StaticInfoStaysOnTheRule(t *testing.T) {
exp, err := New(&Options{File: filepath.Join(t.TempDir(), "report.sarif")})
require.NoError(t, err)
require.NoError(t, exp.Export(makeEvent()))
require.Contains(t, exp.rules[0].FullDescription.Text, "A test vulnerability description.")
require.NotContains(t, buildResultMessage(makeEvent(), "header", "test-template").Text, "Description:")
}
func TestExport_CVSSScoreOverridesSeverity(t *testing.T) {
out := filepath.Join(t.TempDir(), "report.sarif")
exp, err := New(&Options{File: out})
@@ -477,3 +549,30 @@ func TestExport_Concurrency(t *testing.T) {
require.NoError(t, json.Unmarshal(bin, &log))
require.Len(t, log.Runs[0].Result, workers)
}
func TestSarifExportIncludesConfidence(t *testing.T) {
file := filepath.Join(t.TempDir(), "out.sarif")
exporter, err := New(&Options{File: file})
require.NoError(t, err)
event := &output.ResultEvent{
TemplateID: "test-template",
Host: "example.com",
Path: "/x",
Info: model.Info{
Name: "Test Template",
Description: "desc",
SeverityHolder: severity.Holder{Severity: severity.High},
},
Confidence: "high",
ConfidenceScore: 85,
}
require.NoError(t, exporter.Export(event))
require.NoError(t, exporter.Close())
data, err := os.ReadFile(file)
require.NoError(t, err)
require.Contains(t, string(data), "confidence")
require.Contains(t, string(data), "confidence-score")
require.Contains(t, string(data), `"target"`)
}
+15 -1
View File
@@ -47,10 +47,13 @@ func CreateReportDescription(event *output.ResultEvent, formatter ResultFormatte
builder := &bytes.Buffer{}
fmt.Fprintf(builder, "%s: %s matched at %s\n\n", formatter.MakeBold("Details"), formatter.MakeBold(template), event.Host)
attributes := utils.NewEmptyInsertionOrderedStringMap(3)
attributes := utils.NewEmptyInsertionOrderedStringMap(4)
attributes.Set("Protocol", strings.ToUpper(event.Type))
attributes.Set("Full URL", event.Matched)
attributes.Set("Timestamp", event.Timestamp.Format("Mon Jan 2 15:04:05 -0700 MST 2006"))
if event.Confidence != "" {
attributes.Set("Confidence", formatConfidence(event))
}
attributes.ForEach(func(key string, data interface{}) {
fmt.Fprintf(builder, "%s: %s\n\n", formatter.MakeBold(key), types.ToString(data))
})
@@ -250,3 +253,14 @@ func generateCVECWEIDLinksFromClassification(classification *model.Classificatio
func lineBreakToHTML(text string) string {
return strings.ReplaceAll(text, "\n", "<br>")
}
// formatConfidence renders the detection-confidence tier and 0-100 score, e.g.
// "High (85)".
func formatConfidence(event *output.ResultEvent) string {
tier := event.Confidence
if tier != "" {
tier = strings.ToUpper(tier[:1]) + tier[1:]
return fmt.Sprintf("%s (%d)", tier, event.ConfidenceScore)
}
return ""
}
+29
View File
@@ -49,6 +49,35 @@ func TestToMarkdownTableString(t *testing.T) {
require.ElementsMatch(t, expectedDynamicAttributes, actualAttributeSlice[dynamicAttributeIndex:]) // dynamic parameters are not ordered
}
func TestCreateReportDescription_Confidence(t *testing.T) {
base := &output.ResultEvent{
TemplateID: "test-template",
Host: "example.com",
Matched: "https://example.com/x",
Type: "http",
Info: model.Info{
Name: "Test Template",
SeverityHolder: severity.Holder{Severity: severity.High},
},
}
withConfidence := *base
withConfidence.Confidence = "high"
withConfidence.ConfidenceScore = 85
result := CreateReportDescription(&withConfidence, &util.MarkdownFormatter{}, true)
require.Contains(t, result, "Confidence")
require.Contains(t, result, "High (85)")
zeroConfidence := *base
zeroConfidence.Confidence = "low"
zeroResult := CreateReportDescription(&zeroConfidence, &util.MarkdownFormatter{}, true)
require.Contains(t, zeroResult, "Low (0)")
// confidence row is omitted entirely when no score was computed
without := CreateReportDescription(base, &util.MarkdownFormatter{}, true)
require.NotContains(t, without, "Confidence")
}
func TestCreateReportDescription_MarkdownInjection(t *testing.T) {
// Setup a mock result event with malicious payload in various fields
event := &output.ResultEvent{
+8 -5
View File
@@ -156,6 +156,7 @@ func (i *Integration) ShouldFilter(event *output.ResultEvent) bool {
return true
}
// findIssueByTitle searches every linked issue page for an exact title match.
func (i *Integration) findIssueByTitle(title string) (*gitea.Issue, error) {
// Fetch issues in pages to ensure older issues are also checked for duplicates.
pageSize := i.options.DuplicateIssuePageSize
@@ -177,7 +178,7 @@ func (i *Integration) findIssueByTitle(title string) (*gitea.Issue, error) {
return nil, nil
}
issueList, _, err := i.client.ListRepoIssues(i.options.ProjectOwner, i.options.ProjectName, opts)
issueList, response, err := i.client.ListRepoIssues(i.options.ProjectOwner, i.options.ProjectName, opts)
if err != nil {
return nil, err
}
@@ -188,12 +189,14 @@ func (i *Integration) findIssueByTitle(title string) (*gitea.Issue, error) {
}
}
if len(issueList) < opts.PageSize {
// Last page reached.
if response == nil || response.NextPage == 0 {
// Gitea's Link header indicates there is no next page.
return nil, nil
}
opts.Page++
if response.NextPage <= opts.Page {
return nil, fmt.Errorf("gitea returned a non-advancing issue page: current page %d, next page %d", opts.Page, response.NextPage)
}
opts.Page = response.NextPage
}
}
+172
View File
@@ -0,0 +1,172 @@
package gitea
import (
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/model"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/stretchr/testify/require"
)
const testIssueTitle = "Test finding (template-id) found on example.com"
type issueRequest struct {
Page string
Limit string
State string
Keyword string
}
type fakeGitea struct {
server *httptest.Server
requests []issueRequest
comments []int64
created int
issueHandler func(http.ResponseWriter, *http.Request, issueRequest)
}
// newFakeGitea starts a local Gitea API stub and returns its tracker integration.
func newFakeGitea(t *testing.T, issueHandler func(http.ResponseWriter, *http.Request, issueRequest)) (*Integration, *fakeGitea) {
t.Helper()
fake := &fakeGitea{issueHandler: issueHandler}
fake.server = httptest.NewServer(http.HandlerFunc(fake.serveHTTP))
t.Cleanup(fake.server.Close)
integration, err := New(&Options{
BaseURL: fake.server.URL,
Token: "test-token",
ProjectOwner: "owner",
ProjectName: "repo",
DuplicateIssueCheck: true,
})
require.NoError(t, err)
return integration, fake
}
// serveHTTP records tracker requests and serves the configured fake API responses.
func (f *fakeGitea) serveHTTP(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
switch {
case r.Method == http.MethodGet && strings.HasSuffix(r.URL.Path, "/version"):
_, _ = w.Write([]byte(`{"version":"1.21.0"}`))
case r.Method == http.MethodGet && strings.HasSuffix(r.URL.Path, "/labels"):
_, _ = w.Write([]byte("[]"))
case r.Method == http.MethodGet && strings.HasSuffix(r.URL.Path, "/issues"):
query := r.URL.Query()
request := issueRequest{
Page: query.Get("page"),
Limit: query.Get("limit"),
State: query.Get("state"),
Keyword: query.Get("q"),
}
f.requests = append(f.requests, request)
f.issueHandler(w, r, request)
case r.Method == http.MethodPost && strings.Contains(r.URL.Path, "/comments"):
parts := strings.Split(strings.Trim(r.URL.Path, "/"), "/")
issueNumber, _ := strconv.ParseInt(parts[len(parts)-2], 10, 64)
f.comments = append(f.comments, issueNumber)
_, _ = w.Write([]byte(`{"id":1}`))
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/issues"):
f.created++
_, _ = w.Write([]byte(`{"number":99,"title":"created","url":"https://gitea.test/issues/99"}`))
default:
http.NotFound(w, r)
}
}
// writeIssuePage writes issue objects as a JSON API response.
func writeIssuePage(w http.ResponseWriter, issues ...map[string]any) {
_ = json.NewEncoder(w).Encode(issues)
}
// TestCreateIssueFollowsLinkAfterCappedPageAndComments verifies duplicate detection continues after a capped page.
func TestCreateIssueFollowsLinkAfterCappedPageAndComments(t *testing.T) {
var server *httptest.Server
integration, fake := newFakeGitea(t, func(w http.ResponseWriter, r *http.Request, request issueRequest) {
if request.Page == "1" {
// Gitea may return only 50 items despite a requested limit of 100.
w.Header().Set("Link", fmt.Sprintf("<%s%s?limit=100&page=2&state=all>; rel=\"next\"", server.URL, r.URL.Path))
issues := make([]map[string]any, 50)
for index := range issues {
issues[index] = map[string]any{"number": index + 1, "title": "unrelated issue"}
}
writeIssuePage(w, issues...)
return
}
writeIssuePage(w, map[string]any{"number": 42, "title": testIssueTitle, "url": "https://gitea.test/issues/42"})
})
server = fake.server
response, err := integration.CreateIssue(testGiteaEvent())
require.NoError(t, err)
require.Equal(t, "42", response.IssueID)
require.Equal(t, "https://gitea.test/issues/42", response.IssueURL)
require.Equal(t, []int64{42}, fake.comments)
require.Zero(t, fake.created)
require.Equal(t, []string{"1", "2"}, []string{fake.requests[0].Page, fake.requests[1].Page})
for _, request := range fake.requests {
require.Equal(t, "100", request.Limit)
require.Equal(t, "all", request.State)
require.Empty(t, request.Keyword)
}
}
// TestFindIssueByTitleRespectsMaxPages verifies the configured page limit stops further requests.
func TestFindIssueByTitleRespectsMaxPages(t *testing.T) {
var server *httptest.Server
integration, fake := newFakeGitea(t, func(w http.ResponseWriter, r *http.Request, request issueRequest) {
w.Header().Set("Link", fmt.Sprintf("<%s%s?page=2>; rel=\"next\"", server.URL, r.URL.Path))
writeIssuePage(w, map[string]any{"number": 1, "title": "unrelated"})
})
server = fake.server
integration.options.DuplicateIssueMaxPages = 1
issue, err := integration.findIssueByTitle(testIssueTitle)
require.NoError(t, err)
require.Nil(t, issue)
require.Len(t, fake.requests, 1)
require.Equal(t, "1", fake.requests[0].Page)
}
// TestFindIssueByTitleStopsWhenLinkHeaderHasNoNextPage verifies pagination ends without a next link.
func TestFindIssueByTitleStopsWhenLinkHeaderHasNoNextPage(t *testing.T) {
integration, fake := newFakeGitea(t, func(w http.ResponseWriter, _ *http.Request, _ issueRequest) {
writeIssuePage(w, map[string]any{"number": 1, "title": "unrelated"})
})
issue, err := integration.findIssueByTitle(testIssueTitle)
require.NoError(t, err)
require.Nil(t, issue)
require.Len(t, fake.requests, 1)
}
// TestFindIssueByTitleRejectsNonAdvancingPage verifies malformed links return an error instead of looping.
func TestFindIssueByTitleRejectsNonAdvancingPage(t *testing.T) {
var server *httptest.Server
integration, fake := newFakeGitea(t, func(w http.ResponseWriter, r *http.Request, _ issueRequest) {
w.Header().Set("Link", fmt.Sprintf("<%s%s?page=1>; rel=\"next\"", server.URL, r.URL.Path))
writeIssuePage(w, map[string]any{"number": 1, "title": "unrelated"})
})
server = fake.server
issue, err := integration.findIssueByTitle(testIssueTitle)
require.ErrorContains(t, err, "non-advancing issue page")
require.Nil(t, issue)
require.Len(t, fake.requests, 1)
}
// testGiteaEvent builds a small result event for tracker tests.
func testGiteaEvent() *output.ResultEvent {
return &output.ResultEvent{
TemplateID: "template-id",
Host: "example.com",
Info: model.Info{Name: "Test finding"},
}
}
+11 -7
View File
@@ -41,7 +41,8 @@ type Options struct {
DenyList *filters.Filter `yaml:"deny-list"`
// DuplicateIssueCheck is a bool to enable duplicate tracking issue check and update the newest
DuplicateIssueCheck bool `yaml:"duplicate-issue-check" default:"false"`
// DuplicateIssuePageSize controls how many issues are fetched per page when searching for duplicates.
// DuplicateIssuePageSize is the number of issues to request per page when searching for duplicates.
// GitLab returns at most 100 issues per page.
// If unset or <=0, a default of 100 is used.
DuplicateIssuePageSize int `yaml:"duplicate-issue-page-size" default:"100"`
// DuplicateIssueMaxPages limits how many pages are fetched when searching for duplicates.
@@ -161,23 +162,23 @@ func (i *Integration) findIssueByTitle(title string) (*gitlab.Issue, error) {
if pageSize <= 0 {
pageSize = 100
}
maxPages := i.options.DuplicateIssueMaxPages
maxPages := int64(i.options.DuplicateIssueMaxPages)
searchIn := "title"
searchState := "all"
page := 1
page := int64(1)
for {
if maxPages > 0 && page > maxPages {
return nil, nil
}
issues, _, err := i.client.Issues.ListProjectIssues(i.options.ProjectName, &gitlab.ListProjectIssuesOptions{
issues, resp, err := i.client.Issues.ListProjectIssues(i.options.ProjectName, &gitlab.ListProjectIssuesOptions{
In: &searchIn,
State: &searchState,
Search: &title,
ListOptions: gitlab.ListOptions{
Page: int64(page),
Page: page,
PerPage: int64(pageSize),
},
})
@@ -191,11 +192,14 @@ func (i *Integration) findIssueByTitle(title string) (*gitlab.Issue, error) {
}
}
if len(issues) < pageSize {
// GitLab caps per_page at 100, so a short page is not necessarily
// the last one. Follow X-Next-Page instead, and stop when it is
// empty or does not move forward.
if resp.NextPage <= page {
return nil, nil
}
page++
page = resp.NextPage
}
}
@@ -0,0 +1,137 @@
package gitlab
import (
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"slices"
"strconv"
"sync"
"testing"
"github.com/stretchr/testify/require"
)
type fakeIssue struct {
ID int64 `json:"id"`
IID int64 `json:"iid"`
Title string `json:"title"`
}
// fakeGitLab serves the GitLab API endpoints that findIssueByTitle uses. Like
// GitLab, it caps per_page at maxPerPage and names the next page in the
// X-Next-Page header, which is empty on the last page. Every issue counts as
// a search hit, which leaves the exact title match to the tracker.
type fakeGitLab struct {
issues []fakeIssue
maxPerPage int
mu sync.Mutex
pages []int
}
func newFakeGitLab(issueCount, maxPerPage int) *fakeGitLab {
f := &fakeGitLab{maxPerPage: maxPerPage}
for n := 1; n <= issueCount; n++ {
f.issues = append(f.issues, fakeIssue{ID: int64(n), IID: int64(n), Title: fmt.Sprintf("issue %d", n)})
}
return f
}
func (f *fakeGitLab) ServeHTTP(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
switch r.URL.Path {
case "/api/v4/user":
_, _ = w.Write([]byte(`{"id":1}`))
case "/api/v4/projects/1234/issues":
f.listIssues(w, r)
default:
http.NotFound(w, r)
}
}
func (f *fakeGitLab) listIssues(w http.ResponseWriter, r *http.Request) {
page, _ := strconv.Atoi(r.URL.Query().Get("page"))
perPage, _ := strconv.Atoi(r.URL.Query().Get("per_page"))
if perPage <= 0 || perPage > f.maxPerPage {
perPage = f.maxPerPage
}
f.mu.Lock()
f.pages = append(f.pages, page)
f.mu.Unlock()
nextPage := ""
if page*perPage < len(f.issues) {
nextPage = strconv.Itoa(page + 1)
}
w.Header().Set("X-Next-Page", nextPage)
start := min((page-1)*perPage, len(f.issues))
end := min(start+perPage, len(f.issues))
_ = json.NewEncoder(w).Encode(f.issues[start:end])
}
func (f *fakeGitLab) requestedPages() []int {
f.mu.Lock()
defer f.mu.Unlock()
return slices.Clone(f.pages)
}
func TestFindIssueByTitle(t *testing.T) {
tests := []struct {
name string
title string
maxPages int
wantIID int64
wantPages []int
}{
{
name: "match after a server-capped first page",
title: "issue 110",
wantIID: 110,
wantPages: []int{1, 2},
},
{
name: "no match checks every page once",
title: "missing",
wantPages: []int{1, 2},
},
{
name: "max pages stops the search",
title: "issue 110",
maxPages: 1,
wantPages: []int{1},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// GitLab caps per_page at 100, so a configured page size of
// 150 still splits 120 issues into two pages.
fake := newFakeGitLab(120, 100)
server := httptest.NewServer(fake)
t.Cleanup(server.Close)
integration, err := New(&Options{
BaseURL: server.URL,
Token: "token",
ProjectName: "1234",
DuplicateIssuePageSize: 150,
DuplicateIssueMaxPages: tt.maxPages,
})
require.NoError(t, err)
issue, err := integration.findIssueByTitle(tt.title)
require.NoError(t, err)
if tt.wantIID == 0 {
require.Nil(t, issue)
} else {
require.NotNil(t, issue)
require.Equal(t, tt.wantIID, issue.IID)
}
require.Equal(t, tt.wantPages, fake.requestedPages())
})
}
}
+5 -2
View File
@@ -55,8 +55,11 @@ func Cluster(list []*Template) [][]*Template {
// Split up templates that might be clusterable
for _, template := range list {
// it is not possible to cluster flow and multiprotocol due to dependent execution
if template.Flow != "" || template.Options.IsMultiProtocol {
// it is not possible to cluster flow and multiprotocol due to dependent execution.
// Info placeholders are filled from the event a finding came from, and a
// cluster shares one event that already holds the first template's variables
// and extracted values, so a template with them runs on its own.
if template.Flow != "" || template.Options.IsMultiProtocol || template.Info.HasPlaceholders() {
final = append(final, []*Template{template})
continue
}
+28
View File
@@ -5,8 +5,10 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/model"
"github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/variables"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/dns"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/http"
"github.com/projectdiscovery/nuclei/v3/pkg/utils"
"github.com/projectdiscovery/nuclei/v3/internal/tests/testutils"
"github.com/stretchr/testify/require"
)
@@ -38,6 +40,32 @@ func TestClusterTemplates(t *testing.T) {
got := Cluster(tpls)
require.ElementsMatch(t, expected, got)
})
// A cluster shares one event that holds the first template's variables and
// extracted values, so a template with info placeholders runs on its own,
// even next to one with the same variables.
t.Run("no-cluster-with-info-placeholders", func(t *testing.T) {
withVariables := func(path, product string) *Template {
vars := variables.Variable{InsertionOrderedStringMap: *utils.NewEmptyInsertionOrderedStringMap(1)}
vars.Set("product", product)
tp := &Template{
Path: path,
RequestsHTTP: []*http.Request{{Path: []string{"{{BaseURL}}"}}},
Info: model.Info{Description: "{{product}} panel exposed"},
Variables: vars,
}
tp.Options = execOptions
return tp
}
grafana := withVariables("grafana.yaml", "Grafana")
alsoGrafana := withVariables("also-grafana.yaml", "Grafana")
static := &Template{Path: "static.yaml", RequestsHTTP: []*http.Request{{Path: []string{"{{BaseURL}}"}}}}
alsoStatic := &Template{Path: "also-static.yaml", RequestsHTTP: []*http.Request{{Path: []string{"{{BaseURL}}"}}}}
static.Options = execOptions
alsoStatic.Options = execOptions
got := Cluster([]*Template{grafana, alsoGrafana, static, alsoStatic})
require.ElementsMatch(t, [][]*Template{{grafana}, {alsoGrafana}, {static, alsoStatic}}, got)
})
t.Run("dns-cluster", func(t *testing.T) {
tp1 := &Template{Path: "first.yaml", RequestsDNS: []*dns.Request{{Name: "{{Hostname}}"}}}
tp2 := &Template{Path: "second.yaml", RequestsDNS: []*dns.Request{{Name: "{{Hostname}}"}}}
+5 -3
View File
@@ -203,16 +203,18 @@ func init() {
MODELInfoDoc.Fields[3].Name = "description"
MODELInfoDoc.Fields[3].Type = "string"
MODELInfoDoc.Fields[3].Note = ""
MODELInfoDoc.Fields[3].Description = "Description of the template.\n\nYou can go in-depth here on what the template actually does."
MODELInfoDoc.Fields[3].Description = "Description of the template.\n\nYou can go in-depth here on what the template actually does.\n\nDescription, impact and remediation may hold {{placeholders}} filled for\neach finding: named extractors, payloads, variables and DSL helpers\n(network helpers excluded), and the result fields host, port, ip, url,\nscheme, path, matched-at, matcher-name and extractor-name, which win over\nvalues of the same name. Use join(\", \", name) for an extractor with\nseveral values, and brackets for a hyphenated name inside a helper:\nto_upper([matcher-name]). Unknown placeholders stay as written, and so\ndoes a field whose expression fails. Global matcher templates match on\nother templates' responses, so only the result fields fill their text."
MODELInfoDoc.Fields[3].Comments[encoder.LineComment] = "Description of the template."
MODELInfoDoc.Fields[3].AddExample("", "Bower is a package manager which stores package information in the bower.json file")
MODELInfoDoc.Fields[3].AddExample("", "Subversion ALM for the enterprise before 8.8.2 allows reflected XSS at multiple locations")
MODELInfoDoc.Fields[3].AddExample("", "Redis {{version}} on {{host}}:{{port}} accepts the default password")
MODELInfoDoc.Fields[4].Name = "impact"
MODELInfoDoc.Fields[4].Type = "string"
MODELInfoDoc.Fields[4].Note = ""
MODELInfoDoc.Fields[4].Description = "Impact of the template.\n\nYou can go in-depth here on impact of the template."
MODELInfoDoc.Fields[4].Description = "Impact of the template.\n\nYou can go in-depth here on impact of the template.\n\nSupports the same placeholders as description."
MODELInfoDoc.Fields[4].Comments[encoder.LineComment] = "Impact of the template."
MODELInfoDoc.Fields[4].AddExample("", "Successful exploitation of this vulnerability could allow an attacker to execute arbitrary SQL queries, potentially leading to unauthorized access, data leakage, or data manipulation.")
@@ -245,7 +247,7 @@ func init() {
MODELInfoDoc.Fields[9].Name = "remediation"
MODELInfoDoc.Fields[9].Type = "string"
MODELInfoDoc.Fields[9].Note = ""
MODELInfoDoc.Fields[9].Description = "Remediation steps for the template.\n\nYou can go in-depth here on how to mitigate the problem found by this template."
MODELInfoDoc.Fields[9].Description = "Remediation steps for the template.\n\nYou can go in-depth here on how to mitigate the problem found by this template.\n\nSupports the same placeholders as description."
MODELInfoDoc.Fields[9].Comments[encoder.LineComment] = "Remediation steps for the template."
MODELInfoDoc.Fields[9].AddExample("", "Change the default administrative username and password of Apache ActiveMQ by editing the file jetty-realm.properties")
+3
View File
@@ -320,6 +320,9 @@ type Options struct {
EnvironmentVariables bool
// MatcherStatus displays optional status for the failed matches as well
MatcherStatus bool
// ConfidenceBaseline sends a per-host control request to detect catch-all
// responses and lower the confidence of likely false-positive matches.
ConfidenceBaseline bool
// ClientCertFile client certificate file (PEM-encoded) used for authenticating against scanned hosts
ClientCertFile string
// ClientKeyFile client key file (PEM-encoded) used for authenticating against scanned hosts