Commit Graph
108 Commits
Author SHA1 Message Date
Dogan Can Bakir defc07b1f4 add -store-events and -replay to rerun matchers over stored events of any protocol 2026-11-09 23:25:23 +03:00
Mzack9999 0505091a26 Merge pull request #7495 from DevamShah/feat/csv-result-exporter
feat(reporting): add CSV result exporter (-csv-export)
2026-11-09 14:55:38 +01:00
Dogan Can Bakir 4505f0dcf0 Merge pull request #8033 from projectdiscovery/feat/scan-plan
add -plan to write the work a scan would do with its cost
2026-11-09 13:15:27 +03:00
Mzack9999 2f71e68f9a Merge remote-tracking branch 'origin/dev' into feat/csv-result-exporter 2026-11-08 15:00:25 +01:00
Mzack9999 361f3b1383 merge dev 2026-11-07 18:49:32 +01:00
Dogan Can Bakir b7e7da9155 fix plan cluster and mixed-protocol costs, skip update checks and reject target-changing flags under -plan 2026-11-04 10:47:06 +03:00
Dogan Can Bakir b7b4fb97e0 add -plan to write the work a scan would do with its cost, without sending requests 2026-11-03 20:42:14 +03:00
Doğan Can Bakır 5c9f02a4fc drop http matches on a host's catch-all page with -auto-calibrate 2026-11-01 18:35:12 +03:00
Kamil Kamili 71c268d59e Make -max-request-body-inflight opt-in
Default it to 0 so the budget changes nothing unless it is set.
2026-10-29 19:24:41 +04:00
Arthur LoiselleandClaude Opus 5.5 e55509d881 fix(output): reject metadata keys that collide after string conversion
Keys of different types can format to the same string (`1` and `1.0`), and
keeping either value would depend on map iteration order. `ToJSONValue` and
`ToJSONMap` now return an error on such a collision, which both
`MarshalJSON` methods propagate.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 11:04:42 -04:00
Kamil Kamili 128ec95adb Trim comments in the request body budget change 2026-10-29 17:54:25 +04:00
Arthur LoiselleandClaude Opus 5.5 3ae6ca8e3d fix(output): encode nested template maps in JSON output
Template YAML decodes nested maps as `map[interface{}]interface{}`, which
the JSON encoder rejects, so a result whose `info.metadata` (or javascript
args carried in `meta`) held a map was dropped from `-j`/`-jsonl` output.
`model.Info` and `output.ResultEvent` now encode their metadata with
string-keyed maps.

Closes #7971

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 09:21:08 -04:00
Kamil Kamili a4422d8962 Bound request-body bytes held by in-flight HTTP requests
Add -max-request-body-inflight (default 256 MB, 0 disables it).
executeRequest acquires its body size from a shared budget before it
dumps and sends, and releases it when it returns, so concurrent
requests with multi-MB bodies queue instead of all holding their
copies at once. Bodies under 1 MB skip the budget, since the semaphore
is FIFO and a queued heavy request would stall them. Race requests
skip it too, because they must all reach the open gate together.

The HTTP timeout in ExecuteWithResults now starts once the budget
admits the request, so time spent queued does not time it out.
2026-10-29 14:53:14 +04:00
John Eric von Rosen 3f341f0222 Add -no-redact to disable the built-in credential redaction
Since #7469 the output writer always masks Authorization, Proxy-Authorization,
Cookie, Set-Cookie and X-Api-Key, which leaves exported findings without the
data needed to reproduce them by hand. -no-redact turns the built-in list off;
keys given with -redact are still masked.
2026-10-28 12:35:18 +01:00
Mzack9999 8e1e613a15 merge dev 2026-10-25 18:10:14 +01:00
Mzack9999 1086d5d391 keep matchers 2026-10-25 17:32:15 +01:00
Mzack9999 e17273911b Merge pull request #7914 from mazzz3r/feat/payload-random-order
feat(payloads): iterate payload combinations in a random order
2026-10-25 17:10:19 +01:00
Ice3manandClaude Opus 5.5 2954b81c85 http: stop copying large request bodies
Templates that send multi-MB bodies (e.g. CVE-2026-109618 sends 1, 8 and
20 MB) kept 8-10 copies of each body alive per request. With 25 hosts in
flight this reached ~10 GB RSS and OOM-killed scan VMs.

- add pkg/utils/httpbody: a zero-copy request body that rewinds on EOF;
  GetBody hands out new cursors over the same slice instead of cloning
- build raw, model and fuzzed request bodies with httpbody instead of
  retryablehttp's ReusableReadCloser, which tees every read into a second
  growing buffer and clones on each GetBody
- dump requests with http.Request.Write into a capped writer instead of
  retryablehttp Dump/httputil.DumpRequestOut; output is unchanged for
  bodies under the cap
- keep only the first 32 KiB of a body in request dumps (new
  -rqss/-request-size-save flag, 0 keeps all) and leave bodies over 4 KiB
  out of curl commands
- build the project cache key only when a project file is used, hashing
  the body when the dump is truncated
- replay sends the full body when the dump was truncated
- slice the raw request body from the request instead of reading it out
- pass replacer values to fasttemplate as []byte so writes don't copy them

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 18:48:04 +05:30
mazzz3r c99bc1b959 test(payloads): cover random order totals, reset, default-off and option copy 2026-10-22 09:02:02 +00:00
mazzz3r 05d328d936 feat(payloads): iterate payload combinations in a random order
Add -pro, -payload-random-order to run a template's payload combinations
in a random order instead of the sequential order of the payload sets.

The iterator shuffles the combination indices [0,total) and decodes each
index back to a combination, so it visits every combination exactly once
for batteringram, pitchfork and clusterbomb, keeping pitchfork's
index-aligned pairs and the full clusterbomb product. Total() is
unchanged, so request estimates and progress stay correct. When the flag
is off the sequential path is used as before.
2026-10-22 08:49:42 +00:00
Nitin 17101d0501 chore: add doc comments to functions touched by the templates channel 2026-10-15 16:54:00 +05:30
Nitin 866da46866 feat(installer): add templates update channel
Adds -utc / -update-templates-channel (stable, latest). stable keeps the
current behaviour. latest installs the head of the nuclei-templates main
branch, so templates merged after the last release are available without
waiting for the next one.

The version for the latest channel is the last release plus the short
commit as semver build metadata (e.g. v10.5.0+3de629d). Build metadata is
ignored when comparing versions, so switching back to stable doesn't
downgrade, and only a newer release makes it outdated there. Since the
version check only knows about releases, the latest channel compares the
installed version against the current head to decide when to update.

The channel can also be set in the config file like the other options.

Closes #5608
2026-10-15 16:26:17 +05:30
Dogan Can Bakir a151a1d54f Merge pull request #7793 from projectdiscovery/feat/7783-host-backoff
slow down a host that starts blocking
2026-10-14 16:06:57 +03:00
Doğan Can Bakır 830e012a10 Merge remote-tracking branch 'origin/dev' into feat/7783-host-backoff
# Conflicts:
#	pkg/protocols/common/hostratelimit/host_normalize.go
#	pkg/protocols/http/request.go
#	pkg/protocols/http/request_fuzz.go
#	pkg/types/types.go
2026-10-12 12:04:34 +03:00
Doğan Can Bakır d23849e996 feat: add -replay-proxy to re-send requests that produced a finding 2026-10-08 09:41:24 +03:00
Doğan Can Bakır 3a827b8c8b feat(input): deduplicate crawled urls before template execution 2026-09-29 16:10:55 +03:00
Doğan Can Bakır 97e13fae6e feat(scan): slow down a host that starts blocking 2026-09-28 15:40:19 +03:00
Devam Shah 6b1676736c feat(reporting): add CSV result exporter (-csv-export)
Adds a CSV exporter under pkg/reporting/exporters/csv, wired into the
reporting module and exposed via -csv-export / -ce. It writes one
flattened row per result with a stable, spreadsheet/SIEM-friendly schema:
template-id, severity, host, matched-at, cve, cvss, timestamp.

Mirrors the existing exporter contract (New/Export/Close) and the flag
wiring pattern used by jsonl/pdf. All fields are written through
encoding/csv so adversarial values (commas/quotes/newlines in
attacker-influenced host/matched-at) cannot break out of their cell or
forge columns. Unit tests cover header+row, missing classification,
structural-injection escaping, and multi-CVE joining.

Signed-off-by: Devam Shah <[email protected]>
2026-09-18 11:51:55 +05:30
Doğan Can Bakır 026a1890b8 Merge remote-tracking branch 'origin/dev' into feat/llm-matchers
# Conflicts:
#	go.mod
#	go.sum
2026-09-15 12:51:00 +03:00
Nakul Bharti 66068dbec2 perf: improve embedded scan runtime efficiency (#7717)
* feat(templates): add execution-local parser lifecycle

* fix(network): include ports and payloads in request count

* fix(headless): include payloads in request count

* feat(core): expose template execution lifecycle

* fix(core): report context-interrupted template executions

* perf(interactsh): allocate caches on first use

* perf(stats): reuse compiled waf detector

* perf(hosterrors): release cache storage on close

* fix(deps): use published tlsx release

* perf(templates): coalesce concurrent parses

* perf(stats): prefilter waf response regexes

* perf(stats): scan waf literals once per response

* fix(sdk): isolate shared interactsh executions

* fix(protocols): preserve interactsh request scope

* fix(interactsh): drain in-flight scoped callbacks

* perf(hosterrors): allocate storage on first failure

* fix(interactsh): recognize raw SNI callbacks

* fix(interactsh): guard non-fuzz callback accounting

* perf(jarm): integrate bounded shared fingerprints

* fix(jarm): honor configured scan proxy

* perf(core): support adaptive template concurrency

* perf(core): enforce shared template concurrency budget

* fix(http): smooth low per-host request limits

* perf(http): remove per-host limiter goroutines

* fix(http): retain host budgets for large embedded scans

* test: consolidate parser and headless coverage

* fix: address PR review comments

* fix(headless): honor scan cancellation
2026-09-15 02:35:21 +05:30
Doğan Can Bakır d4a40d2b04 Merge remote-tracking branch 'origin/dev' into feat/llm-matchers
# Conflicts:
#	go.mod
#	go.sum
2026-09-14 14:36:46 +03:00
Doğan Can Bakır a78ea81854 wire llm matcher client, flags, and capability gate 2026-09-11 17:51:19 +03:00
Doğan Can Bakır b69bf17ce1 add dast proxy mode to fuzz live traffic 2026-09-10 14:16:32 +03:00
f5d0ef81a5 feat(http): add RFC 9111 cache w/ per-request disable (#6746)
* feat(http): add RFC-9111 cache w/ per-request disable

* wrap HTTP transports with sandrolain/httpcache
  and expose global `--no-http-cache` flag.
* allow templates to set `disable-http-cache`.
* bump Go to 1.25.7.

Signed-off-by: Dwi Siswanto <[email protected]>

* ci: pin go version by go.mod

Signed-off-by: Dwi Siswanto <[email protected]>

* build: set GODEBUG=tlssha1=1

Signed-off-by: Dwi Siswanto <[email protected]>

* build(make): add more `GO*` envs for `go-build`

Signed-off-by: Dwi Siswanto <[email protected]>

* chore(cmd): fix typo

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* fix(types): add `DisableHTTPCache` to `Copy` method

Signed-off-by: Dwi Siswanto <[email protected]>

* test: add HTTP cache integration tests

Signed-off-by: Dwi Siswanto <[email protected]>

* refactor: inverse HTTP cache (RFC 9111) logic (opt-in)

Signed-off-by: Dwi Siswanto <[email protected]>

* chore(deps): go mod tidy

Signed-off-by: Dwi Siswanto <[email protected]>

* test(integration): add -http-cache to both HTTP cache test cases

Signed-off-by: Dwi Siswanto <[email protected]>

* filter cache

* bound cache

---------

Signed-off-by: Dwi Siswanto <[email protected]>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Mzack9999 <[email protected]>
2026-09-07 03:55:36 +07:00
dw1 b04fc59ad0 refactor: adopt XDG storage and layered configs (#7684)
* refactor: adopt XDG storage and layered configs

Load XDG system configuration (and /etc/nuclei/config.yaml) before the
active user configuration, selected config, profile, and command line.
Preserve explicit command-line values, replace collections between
layers, and defer stateful option handling until the final values are
known. Keep callback flags and authentication setup limited to the CLI.

Move configuration loading, profile resolution, inline targets, and
inline secrets into internal/configuration. Keep NUCLEI_CONFIG_DIR as
a compatibility override for the user configuration directory.

Use platform directories for configuration, template data, persistent
state, and cache data. Discover default template roots through
XDG_DATA_HOME and XDG_DATA_DIRS without assigning ownership semantics
to the selected path.

Store template state in XDG_STATE_HOME and migrate the legacy
.templates-config.json file when needed. Persist the active template
path, keep custom provider templates below that root, and apply the
same installation, update, health-check, and reset behavior to every
root.

Keep .nuclei-ignore with the active template root. Derive its local
hash from the file and pass the remote hash through the update check
without persisting either value.

Move generated resume and crash-recovery files into persistent state
and use XDG_CACHE_HOME for the catalog index. Write template state and
cache snapshots atomically with private permissions.

Validate every reset target before deleting any path. Reject broad or
unsafe targets, preserve explicit resume paths, and remove generated
inline-secret directories on normal, interrupted, and fatal exits.

Isolate configuration, state, cache, and template paths in functional
and integration tests, and add coverage for layered precedence, state
migration, active-root lifecycle, reset safety, and private file modes.

Closes #6755

Signed-off-by: Dwi Siswanto <[email protected]>

* chore(cmd): use memProfile+ext instead

Signed-off-by: Dwi Siswanto <[email protected]>

* fix(lib): gate SDK version check

Signed-off-by: Dwi Siswanto <[email protected]>

---------

Signed-off-by: Dwi Siswanto <[email protected]>
2026-09-03 03:21:50 +07:00
Mzack9999 1427b35cdb General maintenance and cleanup (#7469)
* general maintenance

* address review
2026-07-03 05:48:17 +05:30
Dwi SiswantoandMzack9999 4cabca06f6 chore(deps): strip dependencies (#7457)
* chore(deps): strip dependencies

Signed-off-by: Dwi Siswanto <[email protected]>

* chore: drop stripped dependencies

Signed-off-by: Dwi Siswanto <[email protected]>

* tests(integration): reduce external service flakiness

Signed-off-by: Dwi Siswanto <[email protected]>

* refactor(types): replace govalidator ASCII check

Signed-off-by: Dwi Siswanto <[email protected]>

* chore(trackers): update go-github import path

Signed-off-by: Dwi Siswanto <[email protected]>

* chore: use our PD wrapper

Signed-off-by: Dwi Siswanto <[email protected]>

* chore: use our PD cache packages

Signed-off-by: Dwi Siswanto <[email protected]>

* chore(utils): use x/net publicsuffix

Signed-off-by: Dwi Siswanto <[email protected]>

* chore(input): decode YAML without invopop/yaml

Signed-off-by: Dwi Siswanto <[email protected]>

* refactor(yaml): route callers through pkg/utils/yaml

Signed-off-by: Dwi Siswanto <[email protected]>

* refactor(json): route callers through pkg/utils/json

Signed-off-by: Dwi Siswanto <[email protected]>

* refactpr(scan): replace Echo handlers with net/http

Signed-off-by: Dwi Siswanto <[email protected]>

* refactor(server): replace Echo with net/http

Signed-off-by: Dwi Siswanto <[email protected]>

* refactor(fuzzplayground): replace Echo server with net/http

Signed-off-by: Dwi Siswanto <[email protected]>

* test: update subdomains testdata

Signed-off-by: Dwi Siswanto <[email protected]>

* chore(yaml): satisfy lints

Signed-off-by: Dwi Siswanto <[email protected]>

* fix codeql

---------

Signed-off-by: Dwi Siswanto <[email protected]>
Co-authored-by: Mzack9999 <[email protected]>
2026-06-21 05:41:12 +07:00
Mzack9999andIce3man c041bf2922 Fix connection reuse + port pre-flight (#6715)
* .

* .

* adding pre-flight

* add conn reuse

* making pre-flight optional

* adding per-host httpclient + ratelimit

* add conn reuse stats

* ignoring leak in lru

* adding global connection pooling with sharding

* lint

* addressing comments

* fix lint: disambiguate retryablehttp URL access

retryablehttp.Request embeds *urlutil.URL, so .URL.String() resolved to
the wrong type and tripped staticcheck QF1008. Use .Request.URL to read
the underlying http.Request URL for rate limiting and pooling keys.

* fix race

* regression harness

* ci tuning

* address review comments

- preflight: resolve via MetaInput.Target()/CustomIP, gate banner on Silent
- http: drop dead Priority 4 branch in AnalyzeConnectionReuse

* add tests

* host stats

* reuse timing

---------

Co-authored-by: Ice3man <[email protected]>
2026-06-19 15:22:16 +05:30
Dwi SiswantoandMzack9999 20cbf0bd73 refactor: use path-aware filesystem containment checks (#7420)
* feat(filepath): add IsPathWithinAnyDirectory func

Signed-off-by: Dwi Siswanto <[email protected]>

* refactor: use path-aware filesystem containment checks

Several filesystem trust boundaries relied on
lexical prefix checks to decide whether a path
fell under an allowed directory. That let sibling
paths be treated as if they were children of
trusted directories.

Replace those checks with canonical path
containment checks for helper payload loading,
template archive extraction, custom template
metadata, template path classification, and
headless screenshot output validation.

Signed-off-by: Dwi Siswanto <[email protected]>

* adding more tests

* fixing tests

* adding test

---------

Signed-off-by: Dwi Siswanto <[email protected]>
Co-authored-by: Mzack9999 <[email protected]>
2026-06-04 21:03:51 +05:30
Dwi Siswanto 6f2ade6a9b fix(js): respect allow-local-file-access in require (#7332)
* fix(js): respect `allow-local-file-access` in `require`

The goja `require() `function used the default
host filesystem loader which let JavaScript
templates import any local files even when
`allow-local-file-access` was disabled.

Pooled runtimes kept `require()` state around so
a module loaded during a privileged execution
could remain cached for a later restricted one.

Rebuild the require registry per execution after
setting the execution context, and route file-
backed module loads to preserve native modules
while enforcing the same sandbox rules (as
`nuclei/fs`).

Signed-off-by: Dwi Siswanto <[email protected]>

* fix: cross-platform sandbox path checks

Replace lexical prefix checks in the template file
sandbox with a shared path containment helper that
canonicalizes both paths before comparing them to
prevent false rejections when the configured
templates directory and the resolved file path
differ only due to symlink expansion on macOS or
path normalization on Windows.

Apply the helper in `protocolstate.NormalizePath()`
and `Options.GetValidAbsPath()` so JS `require()`-
based module loads and helper file resolution use
the same rules.

Signed-off-by: Dwi Siswanto <[email protected]>

---------

Signed-off-by: Dwi Siswanto <[email protected]>
2026-04-10 08:06:39 +07:00
Mzack9999 48e05ca32e Merge pull request #7277 from HarshadaGawas05/feat/honeypot-detection
feat: add honeypot detection to reduce scan noise (#6403)
2026-03-20 17:09:58 +01:00
Mzack9999 63d0df9506 fmt + docs 2026-03-20 16:16:17 +01:00
saurabhandMzack9999 5e8ef276a0 add inline targets and secrets to template profiless (#5567) (#6858)
* feat: template profile inline targets and secrets (#5567)

* fix inline secrets temp file edge cases

* fixing lint

---------

Co-authored-by: Mzack9999 <[email protected]>
2026-03-20 13:57:18 +01:00
Harshada Gawas 658df54f48 feat: add honeypot detection to reduce scan noise (#6403) 2026-03-20 14:05:21 +05:30
Gengyscan 42b7a11f2c feat(reporting): add PDF export option for scan results 2026-03-17 14:58:40 +01:00
Mzack9999 b5cb4ccb6c Merge pull request #7026 from n3integration/fix/executor-concurrency-improvement
fix: add Snapshot method to prevent InFlight map race condition
2026-03-08 00:21:48 +01:00
Mzack9999 29561766a8 fixing further race condition 2026-03-07 23:19:46 +01:00
Mzack9999 c40fec06ae use per field lock 2026-03-07 23:12:04 +01:00
Dwi Siswanto c159e62681 refactor(types)!: make resume state as cache data
Resume files are runtime artifacts and SHOULD NOT
be presented as persistent config.

Update reset messaging to separate config vs cache
and explicitly include resume state under cache
cleanup. Also delete the cache dir during `-reset`
to keep behavior aligned with the new semantics.

Close #6792

Signed-off-by: Dwi Siswanto <[email protected]>
2026-03-05 12:49:11 +07:00
n3integration 2d0076ec20 update Snapshot method signature 2026-02-27 09:18:06 -05:00