Compare commits

...
Author SHA1 Message Date
Sandeep Singh 8ff0e52c0b chore: add automatic security context updates 2026-09-05 06:45:37 +05:30
Sandeep Singh fdd7bd1d66 chore: add repository security context 2026-09-05 06:45:36 +05:30
Sandeep Singh 138f546080 chore: add repository security context 2026-09-03 06:12:44 +05:30
dw1 b04fc59ad0 refactor: adopt XDG storage and layered configs (#7684)
* refactor: adopt XDG storage and layered configs

Load XDG system configuration (and /etc/nuclei/config.yaml) before the
active user configuration, selected config, profile, and command line.
Preserve explicit command-line values, replace collections between
layers, and defer stateful option handling until the final values are
known. Keep callback flags and authentication setup limited to the CLI.

Move configuration loading, profile resolution, inline targets, and
inline secrets into internal/configuration. Keep NUCLEI_CONFIG_DIR as
a compatibility override for the user configuration directory.

Use platform directories for configuration, template data, persistent
state, and cache data. Discover default template roots through
XDG_DATA_HOME and XDG_DATA_DIRS without assigning ownership semantics
to the selected path.

Store template state in XDG_STATE_HOME and migrate the legacy
.templates-config.json file when needed. Persist the active template
path, keep custom provider templates below that root, and apply the
same installation, update, health-check, and reset behavior to every
root.

Keep .nuclei-ignore with the active template root. Derive its local
hash from the file and pass the remote hash through the update check
without persisting either value.

Move generated resume and crash-recovery files into persistent state
and use XDG_CACHE_HOME for the catalog index. Write template state and
cache snapshots atomically with private permissions.

Validate every reset target before deleting any path. Reject broad or
unsafe targets, preserve explicit resume paths, and remove generated
inline-secret directories on normal, interrupted, and fatal exits.

Isolate configuration, state, cache, and template paths in functional
and integration tests, and add coverage for layered precedence, state
migration, active-root lifecycle, reset safety, and private file modes.

Closes #6755

Signed-off-by: Dwi Siswanto <[email protected]>

* chore(cmd): use memProfile+ext instead

Signed-off-by: Dwi Siswanto <[email protected]>

* fix(lib): gate SDK version check

Signed-off-by: Dwi Siswanto <[email protected]>

---------

Signed-off-by: Dwi Siswanto <[email protected]>
2026-09-03 03:21:50 +07:00
Mzack9999 2a11278647 Merge pull request #7683 from threatmate/tm/restore-idle-conn-pool
Size the per-host idle connection pool to actual concurrency
2026-09-02 19:45:06 +04:00
Stan Ivanov 83dbc85a08 Let -c and payload threads raise the idle connection pool
Review asked for the pool to be sized from the configured concurrency instead of
a constant, as max(TemplateThreads, request Threads) with TemplateThreads
defaulting to 25. The first half of that is right and is what this does now:
TemplateThreads and a template's payload Threads both raise the pool, which fixes
-c 800 being served by a 500-deep pool.

They cannot also lower it, because -c is not an upper bound on how many
connections the transport has open. http.Transport starts a dial whenever a
request does not find an idle connection; if one frees up first the request
takes it and the racing dial still completes. Measured against a keep-alive
target at -c 25, 1,965 templates, 5,388 requests, counting at the target:

  requests in flight   peaked at 22-31
  open connections     peaked at 121-178

Sizing the pool at -c therefore discards most of what the transport opened:

  pool 500 (floor)   414 connections    13.0 req/conn
  pool 25  (= -c)  3,070 connections     1.8 req/conn
  pool 4   (v3.11.1) 4,352 connections   1.2 req/conn

So the floor stays at 500 and the configured concurrency raises it. A pool that
is never filled costs nothing: it is a ceiling on retained connections, not an
allocation, and the transport only ever holds connections it actually opened.

Tests: TestIdleConnPoolSize is the requested table (unset, -c 50, -c 800,
payload threads raising the pool, payload threads never shrinking it);
TestIdleConnPoolSizeIsWiredToTransport asserts the size reaches http.Transport
rather than only the helper; TestIdleConnPoolAllowsKeepAliveReuseWithPayloadThreads
covers the client that payload templates share -- Configuration.Threads is in the
client cache key, so every template with the same payload thread count piles onto
one pool. It holds each group of requests at a cyclic barrier so the overlap is
sustained for the whole run; with a single-shot barrier a pool that is too small
still passes, because once requests stop overlapping a shallow pool is enough.
2026-09-02 10:19:04 -04:00
github-actions[bot]andehsandeep 764f5a9744 chore: refresh security context (#7688)
Co-authored-by: ehsandeep <[email protected]>
2026-09-01 22:16:44 +05:30
Sandeep Singh 3e1e0e81f0 Add workflow to maintain security context
This workflow maintains the security context by analyzing the repository for vulnerabilities and updating the SECURITY_CONTEXT.md file accordingly.
2026-09-01 21:50:52 +05:30
Mzack9999 a2f9070f68 Merge pull request #7677 from dev-rehaann/fix/disable-cookie-shared-jar
fix(http): honor disable-cookie with shared jars
2026-09-01 17:13:21 +04:00
Stanimir Ivanov c07c84f1d1 Make the idle-pool test fail loudly instead of vacuously
The connection-count assertion held trivially in two cases the test did not
rule out: if every request errored (client.Do errors were swallowed, so zero
successful requests also means zero connections), or if the requests happened
to serialise, since the semaphore caps concurrency without forcing overlap.

Requests now block in the handler until "concurrency" of them have arrived, so
that many connections are genuinely required, and the test asserts every
request succeeded before looking at the connection count. A timer releases the
barrier so a shortfall fails on the assertions rather than hanging.

Verified both ways: reverting the pool to 4 fails on the connection count (400
requests over 176 connections), and pointing the client at a closed server
fails on "requests must succeed" rather than passing on zero connections.
2026-08-31 20:15:52 -04:00
Stanimir Ivanov d443462fda Add regression test for per-host idle connection pool sizing
Counts connections accepted by the server rather than requests served,
because the regression is invisible in request counts: the scan still issues
every request, it just opens a new connection for most of them.

Fails at MaxIdleConnsPerHost=4 (400 requests over 157 connections) and passes
with the pool sized to concurrency (400 requests over 50 connections).
2026-08-31 19:24:21 -04:00
Stanimir Ivanov 8bd6fabb04 Size the per-host idle connection pool to actual concurrency
The per-host client pool introduced in v3.10.0 sets MaxIdleConns and
MaxIdleConnsPerHost to 4. http.Transport closes a connection returned to a
host whose idle pool is already full, so an idle pool smaller than the number
of requests in flight against that host silently defeats keep-alive: every
request past the fourth pays a fresh TCP (and TLS) handshake instead of
reusing a pooled connection.

Clients are scoped to a single host and, in host-spray mode, every template
runs against that host, so concurrency is TemplateThreads multiplied by a
template's payload parallelism. With the defaults that is far above 4.

Measured against a local keep-alive target (6,674 templates, host-spray,
TemplateThreads=50), counting connections accepted by the target:

  before: 17,564 connections for 19,997 requests (1.14 requests/connection)
  after:      885 connections for 19,997 requests (22.6 requests/connection)

Identical request count and identical findings (7 and 7), so this is purely
connection reuse. The connection churn also defeats NAT: behind a Cloud NAT
gateway the pre-fix rate of ~147 new connections/sec per scanner exhausted
per-endpoint port mappings and the gateway began dropping outbound SYNs.

configuration.Threads may now only raise the pool, never lower it, since one
template's thread count does not bound total concurrency against the host.
2026-08-31 18:50:18 -04:00
Mzack9999 98d264c8a7 Merge pull request #7666 from buzhimingdeaikun/codex/fix-readme-grammar
docs: fix grammar in README
2026-08-31 17:53:39 +04:00
Mzack9999 d0d5100607 fix wording 2026-08-31 17:45:58 +04:00
dependabot[bot] 3f6e6931dd chore(deps): bump the modules group with 11 updates (#7682)
Bumps the modules group with 11 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/clistats](https://github.com/projectdiscovery/clistats) | `0.1.5` | `0.1.6` |
| [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.5.17` | `0.5.18` |
| [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.101` | `0.0.102` |
| [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns) | `1.0.115` | `1.0.116` |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.3.23` | `1.3.24` |
| [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.8.21` | `0.8.22` |
| [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.47` | `0.1.48` |
| [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.108` | `0.0.109` |
| [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.11.1` | `0.11.2` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.94` | `0.2.95` |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.49` | `1.2.50` |


Updates `github.com/projectdiscovery/clistats` from 0.1.5 to 0.1.6
- [Release notes](https://github.com/projectdiscovery/clistats/releases)
- [Commits](https://github.com/projectdiscovery/clistats/compare/v0.1.5...v0.1.6)

Updates `github.com/projectdiscovery/fastdialer` from 0.5.17 to 0.5.18
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases)
- [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.5.17...v0.5.18)

Updates `github.com/projectdiscovery/hmap` from 0.0.101 to 0.0.102
- [Release notes](https://github.com/projectdiscovery/hmap/releases)
- [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.101...v0.0.102)

Updates `github.com/projectdiscovery/retryabledns` from 1.0.115 to 1.0.116
- [Release notes](https://github.com/projectdiscovery/retryabledns/releases)
- [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.115...v1.0.116)

Updates `github.com/projectdiscovery/retryablehttp-go` from 1.3.23 to 1.3.24
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases)
- [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.3.23...v1.3.24)

Updates `github.com/projectdiscovery/dsl` from 0.8.21 to 0.8.22
- [Release notes](https://github.com/projectdiscovery/dsl/releases)
- [Commits](https://github.com/projectdiscovery/dsl/compare/v0.8.21...v0.8.22)

Updates `github.com/projectdiscovery/networkpolicy` from 0.1.47 to 0.1.48
- [Release notes](https://github.com/projectdiscovery/networkpolicy/releases)
- [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.47...v0.1.48)

Updates `github.com/projectdiscovery/useragent` from 0.0.108 to 0.0.109
- [Release notes](https://github.com/projectdiscovery/useragent/releases)
- [Commits](https://github.com/projectdiscovery/useragent/compare/v0.0.108...v0.0.109)

Updates `github.com/projectdiscovery/utils` from 0.11.1 to 0.11.2
- [Release notes](https://github.com/projectdiscovery/utils/releases)
- [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md)
- [Commits](https://github.com/projectdiscovery/utils/compare/v0.11.1...v0.11.2)

Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.94 to 0.2.95
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.94...v0.2.95)

Updates `github.com/projectdiscovery/cdncheck` from 1.2.49 to 1.2.50
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.49...v1.2.50)

---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/clistats
  dependency-version: 0.1.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/fastdialer
  dependency-version: 0.5.18
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/hmap
  dependency-version: 0.0.102
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryabledns
  dependency-version: 1.0.116
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryablehttp-go
  dependency-version: 1.3.24
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/dsl
  dependency-version: 0.8.22
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/networkpolicy
  dependency-version: 0.1.48
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/useragent
  dependency-version: 0.0.109
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/utils
  dependency-version: 0.11.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
  dependency-version: 0.2.95
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/cdncheck
  dependency-version: 1.2.50
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: modules
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-31 06:19:09 +00:00
Rehan Khan efd160c266 Merge branch 'dev' into fix/disable-cookie-shared-jar 2026-08-27 20:22:19 +05:00
Dogan Can Bakir da279d42a3 Merge pull request #7585 from projectdiscovery/5414-passive-exports
passive exports
2026-08-26 06:39:25 +03:00
Dogan Can Bakir f146fb3a06 Merge pull request #7674 from projectdiscovery/7673-template-id-error
fix(templates): invalid template id error message
2026-08-26 05:59:10 +03:00
Dogan Can Bakir 6a5c443f95 Merge pull request #7629 from projectdiscovery/7590-raw-http-input
feat(input): add raw http request input mode
2026-08-26 05:14:49 +03:00
Rehan Khan 71ccb1d568 chore: remove redundant test comment
Keep the regression test focused on behavior without restating the method name.

Signed-off-by: Muhammad Rehan Khan <[email protected]>
Signed-off-by: Rehan Khan <[email protected]>
2026-08-25 18:38:54 +05:00
Rehan Khan a976ca8d3f fix(http): honor disable-cookie with shared jars
Honor disable-cookie when a workflow supplies a shared cookie jar. Add an integration fixture proving that a cookie set by one request is not attached to a later request with cookie handling disabled.

Signed-off-by: Muhammad Rehan Khan <[email protected]>
Signed-off-by: Rehan Khan <[email protected]>
2026-08-25 16:06:26 +05:00
Mzack9999 6d6bfd7950 fix lint 2026-08-24 12:49:10 +04:00
dependabot[bot] 84b464e6bd chore(deps): bump the modules group with 4 updates (#7675)
Bumps the modules group with 4 updates: [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer), [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy), [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) and [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck).


Updates `github.com/projectdiscovery/fastdialer` from 0.5.16 to 0.5.17
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases)
- [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.5.16...v0.5.17)

Updates `github.com/projectdiscovery/networkpolicy` from 0.1.46 to 0.1.47
- [Release notes](https://github.com/projectdiscovery/networkpolicy/releases)
- [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.46...v0.1.47)

Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.93 to 0.2.94
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.93...v0.2.94)

Updates `github.com/projectdiscovery/cdncheck` from 1.2.48 to 1.2.49
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.48...v1.2.49)

---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/fastdialer
  dependency-version: 0.5.17
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/networkpolicy
  dependency-version: 0.1.47
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
  dependency-version: 0.2.94
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/cdncheck
  dependency-version: 1.2.49
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: modules
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-24 06:17:16 +00:00
Doğan Can Bakır 85fb938c39 fix(templates): render regex in invalid template id error 2026-08-24 07:52:04 +03:00
coriatwood bb79d8ed71 build: bump Go version in Dockerfile (#7669) 2026-08-20 15:52:23 +07:00
buzhimingdeaikun 7a8da3cd00 Fix grammatical errors in README.md 2026-08-19 02:23:52 +08:00
dependabot[bot] 34866efc22 chore(deps): bump the modules group with 5 updates (#7664)
Bumps the modules group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.5.15` | `0.5.16` |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.3.22` | `1.3.23` |
| [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.45` | `0.1.46` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.92` | `0.2.93` |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.47` | `1.2.48` |


Updates `github.com/projectdiscovery/fastdialer` from 0.5.15 to 0.5.16
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases)
- [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.5.15...v0.5.16)

Updates `github.com/projectdiscovery/retryablehttp-go` from 1.3.22 to 1.3.23
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases)
- [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.3.22...v1.3.23)

Updates `github.com/projectdiscovery/networkpolicy` from 0.1.45 to 0.1.46
- [Release notes](https://github.com/projectdiscovery/networkpolicy/releases)
- [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.45...v0.1.46)

Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.92 to 0.2.93
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.92...v0.2.93)

Updates `github.com/projectdiscovery/cdncheck` from 1.2.47 to 1.2.48
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.47...v1.2.48)

---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/fastdialer
  dependency-version: 0.5.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryablehttp-go
  dependency-version: 1.3.23
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/networkpolicy
  dependency-version: 0.1.46
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
  dependency-version: 0.2.93
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/cdncheck
  dependency-version: 1.2.48
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: modules
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-17 06:18:28 +00:00
Dwi Siswanto fd0aff232e perf(templates): reuse parsed templates across engines (#7661)
* test(templates): add parse across engine local caches bench test

Signed-off-by: Dwi Siswanto <[email protected]>

* perf(templates): reuse parsed templates across engines

Clone cached parsed templates before compiling
them with engine-local options to avoid reopening
and decoding the same source for every engine
without sharing mutable compiled state.

Signed-off-by: Dwi Siswanto <[email protected]>

* fix(templates): preserve aliases when cloning templates

Use one visited map for the full clone and
register maps and slices before walking their
contents to preserve shared values and handles
cycles without infinite recursion.

Signed-off-by: Dwi Siswanto <[email protected]>

---------

Signed-off-by: Dwi Siswanto <[email protected]>
2026-08-17 01:47:00 +07:00
Mzack9999 265b3a3dec Merge pull request #7659 from dwisiswant0/dwisiswant0/fix/lib/avoid-caching-engine-state-in-shared-parsers
fix(lib): avoid caching engine state in shared parsers
2026-08-15 11:19:38 +04:00
Dwi Siswanto dc0ebd19a5 fix(lib): avoid caching engine state in shared parsers
Compiled template cache entries retain mutable
`ExecutorOptions` and can keep closed engines
alive.

Move those entries into an engine-local cache.
Leave the caller-owned parsed cache intact and
purge only compiled entries on `Close`.

Fixes #7656

Signed-off-by: Dwi Siswanto <[email protected]>
2026-08-14 15:03:03 +07:00
Mzack9999 05e0e8387b Merge pull request #7657 from lucasonline0/fix/7654-proxy-keepalive
fix(http): avoid proxy stalls from keep-alive reuse
2026-08-13 20:27:12 +04:00
Lucas Madureira f1007e48d3 fix(http): avoid proxy stalls from keep-alive reuse 2026-08-13 15:02:26 +00:00
Dogan Can Bakir a3f1367fcd Merge pull request #7627 from projectdiscovery/7626-raw-request-parsing
fix(input): parse raw request line and headers by name, not position
2026-08-12 01:50:55 +03:00
Dwi Siswanto 1027207f5e fix(raw): preserve abs targets in unsafe requests (#7652)
* fix(raw): preserve abs targets in unsafe requests

Keep absolute request targets unchanged on the
wire. Derive the effective URL from the scan
target authority and the raw target path and query,
without merging the input path or query. Also
handle mixed-case schemes and single-label hosts.

Fixes #7382
Fixes #7649

Signed-off-by: Dwi Siswanto <[email protected]>

* test(raw): add unsafe w/ full URL w/o path test

Signed-off-by: Dwi Siswanto <[email protected]>

---------

Signed-off-by: Dwi Siswanto <[email protected]>
2026-08-11 17:08:00 +07:00
dependabot[bot] 73be40cae7 chore(deps): bump crate-ci/typos in the workflows group (#7648)
Bumps the workflows group with 1 update: [crate-ci/typos](https://github.com/crate-ci/typos).


Updates `crate-ci/typos` from 1.48.0 to 1.49.0
- [Release notes](https://github.com/crate-ci/typos/releases)
- [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md)
- [Commits](https://github.com/crate-ci/typos/compare/v1.48.0...v1.49.0)

---
updated-dependencies:
- dependency-name: crate-ci/typos
  dependency-version: 1.49.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: workflows
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-11 17:07:18 +07:00
Dwi Siswanto 0df0976899 chore(deps): bump [email protected] (#7653)
Signed-off-by: Dwi Siswanto <[email protected]>
2026-08-11 17:06:40 +07:00
dependabot[bot] dd7153518e chore(deps): bump the modules group with 10 updates (#7647)
Bumps the modules group with 10 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/clistats](https://github.com/projectdiscovery/clistats) | `0.1.4` | `0.1.5` |
| [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.5.14` | `0.5.15` |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.3.21` | `1.3.22` |
| [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.8.20` | `0.8.21` |
| [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.71` | `1.1.72` |
| [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.44` | `0.1.45` |
| [github.com/projectdiscovery/tlsx](https://github.com/projectdiscovery/tlsx) | `1.2.2` | `1.3.2` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.91` | `0.2.92` |
| [github.com/projectdiscovery/blackrock](https://github.com/projectdiscovery/blackrock) | `0.0.1` | `0.0.2` |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.46` | `1.2.47` |


Updates `github.com/projectdiscovery/clistats` from 0.1.4 to 0.1.5
- [Release notes](https://github.com/projectdiscovery/clistats/releases)
- [Commits](https://github.com/projectdiscovery/clistats/compare/v0.1.4...v0.1.5)

Updates `github.com/projectdiscovery/fastdialer` from 0.5.14 to 0.5.15
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases)
- [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.5.14...v0.5.15)

Updates `github.com/projectdiscovery/retryablehttp-go` from 1.3.21 to 1.3.22
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases)
- [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.3.21...v1.3.22)

Updates `github.com/projectdiscovery/dsl` from 0.8.20 to 0.8.21
- [Release notes](https://github.com/projectdiscovery/dsl/releases)
- [Commits](https://github.com/projectdiscovery/dsl/compare/v0.8.20...v0.8.21)

Updates `github.com/projectdiscovery/gologger` from 1.1.71 to 1.1.72
- [Release notes](https://github.com/projectdiscovery/gologger/releases)
- [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.71...v1.1.72)

Updates `github.com/projectdiscovery/networkpolicy` from 0.1.44 to 0.1.45
- [Release notes](https://github.com/projectdiscovery/networkpolicy/releases)
- [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.44...v0.1.45)

Updates `github.com/projectdiscovery/tlsx` from 1.2.2 to 1.3.2
- [Release notes](https://github.com/projectdiscovery/tlsx/releases)
- [Commits](https://github.com/projectdiscovery/tlsx/commits)

Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.91 to 0.2.92
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.91...v0.2.92)

Updates `github.com/projectdiscovery/blackrock` from 0.0.1 to 0.0.2
- [Release notes](https://github.com/projectdiscovery/blackrock/releases)
- [Commits](https://github.com/projectdiscovery/blackrock/compare/v0.0.1...v0.0.2)

Updates `github.com/projectdiscovery/cdncheck` from 1.2.46 to 1.2.47
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.46...v1.2.47)

---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/clistats
  dependency-version: 0.1.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/fastdialer
  dependency-version: 0.5.15
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryablehttp-go
  dependency-version: 1.3.22
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/dsl
  dependency-version: 0.8.21
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/gologger
  dependency-version: 1.1.72
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/networkpolicy
  dependency-version: 0.1.45
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/tlsx
  dependency-version: 1.3.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
  dependency-version: 0.2.92
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/blackrock
  dependency-version: 0.0.2
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/cdncheck
  dependency-version: 1.2.47
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: modules
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-10 06:19:34 +00:00
dependabot[bot] dfdfa5fd65 chore(deps): bump github.com/go-git/go-git/v5 (#7644)
Bumps the go_modules group with 1 update in the / directory: [github.com/go-git/go-git/v5](https://github.com/go-git/go-git).


Updates `github.com/go-git/go-git/v5` from 5.19.1 to 5.19.2
- [Release notes](https://github.com/go-git/go-git/releases)
- [Changelog](https://github.com/go-git/go-git/blob/main/HISTORY.md)
- [Commits](https://github.com/go-git/go-git/compare/v5.19.1...v5.19.2)

---
updated-dependencies:
- dependency-name: github.com/go-git/go-git/v5
  dependency-version: 5.19.2
  dependency-type: direct:production
  dependency-group: go_modules
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-09 05:55:28 +00:00
Mzack9999 379687596f Normalize schemes 2026-08-03 10:14:40 +04:00
Doğan Can Bakır 780e20b33a feat(input): add raw http request input mode
Fuzzing takes the request shape from the input, so a bare URL is always fuzzed
as a GET and anything else needed a spec, captured traffic or a hand-written
proxify document. -im http reads requests as they appear on the wire, which is
what Burp and browser devtools already put on the clipboard.
2026-07-31 15:28:36 +03:00
Doğan Can Bakır 49f1d49a1d fix(input): parse raw request line and headers by name, not position
The second line was always consumed as the Host line, so any request-shaped
input whose Host header sat elsewhere lost that header silently and skipped
fuzzing. Reading values by skipping one byte past the colon also truncated
unspaced values and panicked on valueless headers, crashing the scan. Absolute
request targets, as used by proxy captures, were appended to the authority
instead of replacing it.
2026-07-30 15:09:00 +03:00
Mzack9999 0ff225c1c8 review fixes 2026-07-26 14:24:08 +04:00
Mzack9999 5ba1d41108 merge dev 2026-07-26 14:22:29 +04:00
Mzack9999 c2ac1d859f passive exports 2026-07-24 16:33:16 +04:00
98 changed files with 5418 additions and 1026 deletions

No files matched your search

+127
View File
@@ -0,0 +1,127 @@
name: Maintain security context
on:
workflow_dispatch:
schedule:
- cron: "17 4 * * 1"
permissions:
contents: write
pull-requests: write
concurrency:
group: security-context
cancel-in-progress: true
jobs:
refresh:
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
# The API analyzes the repository default branch, so use that branch for
# both scheduled and manually dispatched updates.
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
ref: ${{ github.event.repository.default_branch }}
# Analyze this commit on securitycontext.dev, wait for the result, and
# download it. The run's own token authorizes private repositories —
# no secrets to configure.
- name: Generate SECURITY_CONTEXT.md
id: context
env:
GITHUB_TOKEN: ${{ github.token }}
run: |
sc() { curl --fail-with-body -sS -H "X-GitHub-Token: $GITHUB_TOKEN" "$@"; }
base="https://securitycontext.dev"
head="$(git rev-parse HEAD)"
sc -X POST "$base/api/v1/context?wait=60" -H "Content-Type: application/json" \
-H "X-Secc-Client: github-action" --data "{\"repo\":\"$GITHUB_REPOSITORY\",\"head\":\"$head\"}" >/dev/null
for _ in {1..120}; do
state="$(sc "$base/api/v1/context/$GITHUB_REPOSITORY")"
status="$(jq -r .status <<<"$state")"
if [ "$status" = "error" ]; then
jq -r '.error // "analysis failed"' <<<"$state" >&2
exit 1
fi
if [ "$status" = "ready" ] && [ "$(jq -r '.summary.head_sha // empty' <<<"$state")" = "$head" ]; then
sc -L "$base/r/$GITHUB_REPOSITORY.md" -o SECURITY_CONTEXT.md
# A private report is access-gated on the web, so the pull request
# links the rendered file on this branch instead.
jq -r '"visibility=" + (.visibility // "public")' <<<"$state" >> "$GITHUB_OUTPUT"
exit 0
fi
sleep 15
done
echo "Timed out waiting for analysis of $head" >&2
exit 1
# A pull request should mean the analysis changed. When only the volatile
# header moved (timestamps, analyzed commit, commit counts), restore the
# committed file so no PR is opened and any header-only PR closes.
- name: Skip the update unless the analysis changed
id: report
run: |
changed=true
if [ -n "$(git ls-files SECURITY_CONTEXT.md)" ] && git diff --quiet -I '^<!-- Generated' -I '^_Generated' -I '^_Analyzed commit' -I '^- Commits analyzed' -I '^- Commits flagged' -- SECURITY_CONTEXT.md; then
git checkout -- SECURITY_CONTEXT.md
changed=false
fi
echo "changed=$changed" >> "$GITHUB_OUTPUT"
# Point every supported root agent-instructions file at the context.
# Create AGENTS.md as the cross-agent default when the repo has none.
- name: Reference the context from agent instructions
id: agents
env:
REPORT_CHANGED: ${{ steps.report.outputs.changed }}
run: |
line="Before writing or reviewing code, read SECURITY_CONTEXT.md. Check new and modified code against its historical fixes and recurring vulnerability patterns. Do not reintroduce a previously fixed issue or a similar variant."
touched=""
paths="SECURITY_CONTEXT.md"
instruction_files=""
for f in AGENTS.md CLAUDE.md; do
if [ -f "$f" ]; then instruction_files="${instruction_files} $f"; fi
done
if [ -z "$instruction_files" ]; then
printf '%s\n' '# Agent instructions' > AGENTS.md
instruction_files="AGENTS.md"
fi
for f in $instruction_files; do
paths="$(printf '%s\n%s' "$paths" "$f")"
if grep -q "SECURITY_CONTEXT.md" "$f"; then continue; fi
printf '\n%s\n' "$line" >> "$f"
if [ -n "$touched" ]; then touched="$touched and $f"; else touched="$f"; fi
done
echo "touched=$touched" >> "$GITHUB_OUTPUT"
# Title the pull request for what it actually carries: when the
# analysis is unchanged, the only edit is the agent-instructions line.
if [ "$REPORT_CHANGED" = "false" ]; then
echo "subject=chore: reference security context from agent instructions" >> "$GITHUB_OUTPUT"
else
echo "subject=chore: refresh security context" >> "$GITHUB_OUTPUT"
fi
{
echo "paths<<SC_EOF"
echo "$paths"
echo "SC_EOF"
} >> "$GITHUB_OUTPUT"
- name: Open an update pull request
uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1
with:
branch: security-context/refresh
commit-message: ${{ steps.agents.outputs.subject }}
title: ${{ steps.agents.outputs.subject }}
body: |
Refreshes `SECURITY_CONTEXT.md`, this repository's security-fix history as working context, so agents and reviewers don't reintroduce fixes that already shipped.
${{ steps.context.outputs.visibility == 'private' && format('[Read the updated file]({0}/{1}/blob/security-context/refresh/SECURITY_CONTEXT.md)', github.server_url, github.repository) || format('[Full report for this repository](https://securitycontext.dev/r/{0})', github.repository) }}
${{ steps.agents.outputs.touched != '' && format('Also adds a one-line reference to {0} so your coding agent picks this up automatically.', steps.agents.outputs.touched) || 'The repository agent instructions already reference `SECURITY_CONTEXT.md`.' }}
---
_Your agent doesn't know this repo's security history. [Security Context](https://securitycontext.dev) keeps it where your agent can see it, so old vulnerabilities and their variants don't come back in new code._
add-paths: ${{ steps.agents.outputs.paths }}
+1 -1
View File
@@ -17,4 +17,4 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: crate-ci/[email protected]8.0
- uses: crate-ci/[email protected]9.0
+2 -1
View File
@@ -81,4 +81,5 @@ Each protocol (HTTP, DNS, Network, etc.) implements:
- **internal/tests/functional/** - CI-only native functional comparison suite and testcase assets
- **pkg/fuzz/** - Fuzzing engine and DAST capabilities
- **pkg/input/** - Input processing for various formats (Burp, OpenAPI, etc.)
- **pkg/reporting/** - Result export and issue tracking integrations
- **pkg/reporting/** - Result export and issue tracking integrations
See SECURITY_CONTEXT.md for this repo's known vulnerabilities and recurring weak spots. Check it before writing or reviewing security-sensitive code.
+1 -1
View File
@@ -1,5 +1,5 @@
# Build
FROM golang:1.24-alpine AS builder
FROM golang:1.27-alpine AS builder
RUN apk add build-base
WORKDIR /app
+5 -5
View File
@@ -64,11 +64,11 @@
### **1. Nuclei CLI**
_Install Nuclei on your machine. Get started by following the installation guide [**`here`**](https://docs.projectdiscovery.io/tools/nuclei/install?utm_source=github&utm_medium=web&utm_campaign=nuclei_readme). Additionally, We provide [**`a free cloud tier`**](https://cloud.projectdiscovery.io/sign-up) and comes with a generous monthly free limits:_
_Install Nuclei on your machine. Get started by following the installation guide [**`here`**](https://docs.projectdiscovery.io/tools/nuclei/install?utm_source=github&utm_medium=web&utm_campaign=nuclei_readme). Additionally, we provide [**`a free cloud tier`**](https://cloud.projectdiscovery.io/sign-up) that comes with generous monthly free limits:_
- Store and visualize your vulnerability findings
- Write and manage your nuclei templates
- Access latest nuclei templates
- Write and manage your Nuclei templates
- Access the latest Nuclei templates
- Discover and store your targets
> [!Important]
@@ -91,7 +91,7 @@ _For security teams and enterprises, we provide a cloud-hosted service built on
- We're constantly [**`adding new features`**](https://feedback.projectdiscovery.io/changelog)!
- **Ideal for:** Pentesters, security teams, and enterprises
[**`Sign up to Pro`**](https://projectdiscovery.io/pricing?utm_source=github&utm_medium=web&utm_campaign=nuclei_readme) or [**`Talk to our team`**](https://projectdiscovery.io/request-demo?utm_source=github&utm_medium=web&utm_campaign=nuclei_readme) if you have large organization and complex requirements.
[**`Sign up to Pro`**](https://projectdiscovery.io/pricing?utm_source=github&utm_medium=web&utm_campaign=nuclei_readme) or [**`Talk to our team`**](https://projectdiscovery.io/request-demo?utm_source=github&utm_medium=web&utm_campaign=nuclei_readme) if you have a large organization and complex requirements.
<br>
<br>
@@ -147,7 +147,7 @@ TARGET:
-iv, -ip-version string[] IP version to scan of hostname (4,6) - (default 4)
TARGET-FORMAT:
-im, -input-mode string mode of input file (list, burp, jsonl, yaml, openapi, swagger) (default "list")
-im, -input-mode string mode of input file (list, burp, jsonl, yaml, openapi, swagger, http) (default "list")
-ro, -required-only use only required fields in input format when generating requests
-sfv, -skip-format-validation skip format validation (like missing vars) when parsing input file
-vtt, -vars-text-templating enable text templating for vars in input file (only for yaml input mode)
+1 -1
View File
@@ -147,7 +147,7 @@ TARGET:
-iv, -ip-version string[] IP version to scan of hostname (4,6) - (default 4)
TARGET-FORMAT:
-im, -input-mode string mode of input file (list, burp, jsonl, yaml, openapi, swagger) (default "list")
-im, -input-mode string mode of input file (list, burp, jsonl, yaml, openapi, swagger, http) (default "list")
-ro, -required-only use only required fields in input format when generating requests
-sfv, -skip-format-validation skip format validation (like missing vars) when parsing input file
-vtt, -vars-text-templating enable text templating for vars in input file (only for yaml input mode)
+1 -1
View File
@@ -147,7 +147,7 @@ TARGET:
-iv, -ip-version string[] IP version to scan of hostname (4,6) - (default 4)
TARGET-FORMAT:
-im, -input-mode string mode of input file (list, burp, jsonl, yaml, openapi, swagger) (default "list")
-im, -input-mode string mode of input file (list, burp, jsonl, yaml, openapi, swagger, http) (default "list")
-ro, -required-only use only required fields in input format when generating requests
-sfv, -skip-format-validation skip format validation (like missing vars) when parsing input file
-vtt, -vars-text-templating enable text templating for vars in input file (only for yaml input mode)
+1 -1
View File
@@ -147,7 +147,7 @@ TARGET:
-iv, -ip-version string[] IP version to scan of hostname (4,6) - (default 4)
TARGET-FORMAT:
-im, -input-mode string mode of input file (list, burp, jsonl, yaml, openapi, swagger) (default "list")
-im, -input-mode string mode of input file (list, burp, jsonl, yaml, openapi, swagger, http) (default "list")
-ro, -required-only use only required fields in input format when generating requests
-sfv, -skip-format-validation skip format validation (like missing vars) when parsing input file
-vtt, -vars-text-templating enable text templating for vars in input file (only for yaml input mode)
+1 -1
View File
@@ -147,7 +147,7 @@ TARGET:
-iv, -ip-version string[] IP version to scan of hostname (4,6) - (default 4)
TARGET-FORMAT:
-im, -input-mode string mode of input file (list, burp, jsonl, yaml, openapi, swagger) (default "list")
-im, -input-mode string mode of input file (list, burp, jsonl, yaml, openapi, swagger, http) (default "list")
-ro, -required-only use only required fields in input format when generating requests
-sfv, -skip-format-validation skip format validation (like missing vars) when parsing input file
-vtt, -vars-text-templating enable text templating for vars in input file (only for yaml input mode)
+1 -1
View File
@@ -147,7 +147,7 @@ TARGET:
-iv, -ip-version string[] IP version to scan of hostname (4,6) - (default 4)
TARGET-FORMAT:
-im, -input-mode string mode of input file (list, burp, jsonl, yaml, openapi, swagger) (default "list")
-im, -input-mode string mode of input file (list, burp, jsonl, yaml, openapi, swagger, http) (default "list")
-ro, -required-only use only required fields in input format when generating requests
-sfv, -skip-format-validation skip format validation (like missing vars) when parsing input file
-vtt, -vars-text-templating enable text templating for vars in input file (only for yaml input mode)
+1 -1
View File
@@ -147,7 +147,7 @@ TARGET:
-iv, -ip-version string[] IP version to scan of hostname (4,6) - (default 4)
TARGET-FORMAT:
-im, -input-mode string mode of input file (list, burp, jsonl, yaml, openapi, swagger) (default "list")
-im, -input-mode string mode of input file (list, burp, jsonl, yaml, openapi, swagger, http) (default "list")
-ro, -required-only use only required fields in input format when generating requests
-sfv, -skip-format-validation skip format validation (like missing vars) when parsing input file
-vtt, -vars-text-templating enable text templating for vars in input file (only for yaml input mode)
+1 -1
View File
@@ -147,7 +147,7 @@ TARGET:
-iv, -ip-version string[] IP version to scan of hostname (4,6) - (default 4)
TARGET-FORMAT:
-im, -input-mode string mode of input file (list, burp, jsonl, yaml, openapi, swagger) (default "list")
-im, -input-mode string mode of input file (list, burp, jsonl, yaml, openapi, swagger, http) (default "list")
-ro, -required-only use only required fields in input format when generating requests
-sfv, -skip-format-validation skip format validation (like missing vars) when parsing input file
-vtt, -vars-text-templating enable text templating for vars in input file (only for yaml input mode)
+302
View File
@@ -0,0 +1,302 @@
<!-- Generated by securitycontext.dev. Source HEAD: b04fc59ad056c3d818aa103a85164c633e856277. -->
# Security Context: projectdiscovery/nuclei @ dev
> Reference context for editing projectdiscovery/nuclei. Before changing any component below, **preserve the listed guards and do not reintroduce these fixed issues**. Each fingerprint is a regression check, and each shared attack surface is a guard that must keep holding on every path.
_Generated 2026-09-03 00:44 UTC by secc_
_Analyzed commit: b04fc59ad056c3d818aa103a85164c633e856277_
**Remediation coverage: 26.5%**
_9 of 34 historical fixes have no remaining unguarded variant; 522 open remediation leads to verify. Measures fix consistency, not a vulnerability verdict._
- Commits analyzed: **4820**
- Commits flagged for review: **415**
- Confirmed historical security fixes: **38**
- Known published CVEs: **6**
- Project type: **client**, excluding 34 availability-only (DoS/crash) finding(s) as out-of-scope for a non-server project
## Summary
Nuclei's historical security posture is dominated by sandbox and trust-boundary enforcement failures: template-driven code execution, local file access, and network egress are repeatedly gated inconsistently across loaders, protocol runtimes, and JS helper libraries. Signature verification bypasses (CVE-2024-43405, CVE-2024-27920) and expression/DSL injection (CVE-2026-41645, CVE-2026-41282) show that untrusted template content and response data are persistently re-evaluated or executed. Path traversal and arbitrary file write/read issues cluster in file-loading and output-writing paths, often due to weak prefix checks or missing sandbox flags. The most critical theme is asymmetric enforcement: a guard applied on one entry point (e.g., direct template load) is frequently missing on sibling paths (workflows, JS runtime, protocol sinks, helper libraries).
## Recurring Security Patterns
**18.4% of classified fixes repeat a previously fixed pattern** (7 recurring, 31 first-seen). A recurring pattern means this repository fixed a similar issue before — not that the code is currently vulnerable.
- **Code Execution / Signature Bypass:** [fac0b7196d](https://github.com/projectdiscovery/nuclei/commit/fac0b7196d2fbca56c8af8ff8e1899c111041a37) → [9de96e4dda](https://github.com/projectdiscovery/nuclei/commit/9de96e4dda5a03da963b9ae6582f03ea55791a76) — matched on `CWE-347`, `component:pkg/catalog/loader/loader.go`
- **Auth Bypass:** [1c440e755a](https://github.com/projectdiscovery/nuclei/commit/1c440e755a97471c56fb0276ee8a8c4132974645) → [fac0b7196d](https://github.com/projectdiscovery/nuclei/commit/fac0b7196d2fbca56c8af8ff8e1899c111041a37) — matched on `CWE-347`, `component:pkg/catalog/loader/loader.go`
- **SSRF:** [a167e6c57b](https://github.com/projectdiscovery/nuclei/commit/a167e6c57b80f5dc7be231231b62d7c594ee2770) → [9def9af7e3](https://github.com/projectdiscovery/nuclei/commit/9def9af7e3f7ebcf5cd561c37b623b9999ee7572) — matched on `class:ssrf`, `CWE-918`, `component:pkg/js/libs/ldap/ldap.go`
- **Path Traversal:** [e510bc207f](https://github.com/projectdiscovery/nuclei/commit/e510bc207f4336058f7b1536ddbb3942b394b05c) → [f663d1c9cf](https://github.com/projectdiscovery/nuclei/commit/f663d1c9cf77da53533bf328750621b4a9fa996d) — matched on `class:path traversal`, `CWE-22`, `sink:os openfile`
- **Path Traversal:** [e7fb40a413](https://github.com/projectdiscovery/nuclei/commit/e7fb40a41308f6c41267e858d1781eb67f7e0572) → [e5154d362a](https://github.com/projectdiscovery/nuclei/commit/e5154d362af900e83b30fb79028bfab1a7f0c648) — matched on `class:path traversal`, `CWE-22`, `component:v2/pkg/protocols/common/generators`
- **Insecure File Permissions:** [4bccb6cf8a](https://github.com/projectdiscovery/nuclei/commit/4bccb6cf8a830724b3bda624cb3bfe43eff31683) → [af4854f90d](https://github.com/projectdiscovery/nuclei/commit/af4854f90de728bb08689d5cae3dec4acfdb6b9a) — matched on `CWE-732`, `sink:os openfile`
- **Path Traversal:** [f9c214a66f](https://github.com/projectdiscovery/nuclei/commit/f9c214a66f37a00fc29fe64232b690e6285ec59c) → [e510bc207f](https://github.com/projectdiscovery/nuclei/commit/e510bc207f4336058f7b1536ddbb3942b394b05c) — matched on `class:path traversal`, `CWE-22`, `sink:os openfile`
## Top Risks
### 1. Code Execution / Signature Bypass: `pkg/catalog/loader/loader.go, pkg/templates/compile.go, pkg/protocols/javascript/js.go` (critical)
Unsigned or unverified templates with JavaScript init blocks were compiled and executed; verification results were not bound to content digests, allowing cache reuse of stale verification. This is the root of multiple RCE and signature bypass CVEs.
Evidence: 9de96e4dda5a03da963b9ae6582f03ea55791a76, CVE-2024-43405, CVE-2024-27920
### 2. Remote Code Execution: `pkg/protocols/code, pkg/protocols, pkg/templates/workflows.go` (high)
Code protocol execution sink lacked gating for signature verification and the -code flag; workflows could load unsigned code templates; dynamic runtime values were recursively evaluated as DSL expressions. Multiple independent paths to local code execution.
Evidence: 1427b35cdbcb11a63c338700fe8e228ada8ec59d, ccbfb12bd01447ba25f6e755dfb2bee677736da6, e86f38299765b82ad724fdb701557e0eaff3884d
### 3. Path Traversal: `pkg/catalog/config, pkg/js/compiler, v2/pkg/protocols/common/generators, v2/internal/runner/update.go` (high)
Weak lexical prefix checks allowed sibling paths to bypass containment; JS runtime allowed local file reads via require(); payload loading and template updater lacked path canonicalization, enabling arbitrary file read/write and Zip Slip.
Evidence: 20cbf0bd73893c1270582c9778b6b26a5cb11206, 6f2ade6a9b427c284c15a43445f9c7f055e60e5d, e5154d362af900e83b30fb79028bfab1a7f0c648, f9c214a66f37a00fc29fe64232b690e6285ec59c
### 4. Expression Language Injection: `pkg/protocols/common/expressions` (high)
Template-authored expressions were parsed after placeholder substitution, allowing untrusted response-derived data to be re-evaluated as DSL expressions. Related CVEs confirm multi-step template injection with -env-vars.
Evidence: d2217320162d5782ca7cb95bef9dda17063818f3, CVE-2026-41645, CVE-2026-41282
### 5. SSRF: `pkg/js/libs/ldap/ldap.go, pkg/js/libs/smb/smb.go` (high)
LDAP host validation checked the domain instead of the actual target address; SMB dialing lacked host allowlist checks. JS helper libraries bypassed the global network policy.
Evidence: a167e6c57b80f5dc7be231231b62d7c594ee2770, a677fca192ee732a7b0a278f725517fa998b7897
### 6. Sandbox Escape: `v2/pkg/protocols/common/generators/validate.go, v2/internal/runner/processor.go` (high)
Go-template payload generators and Tengo scripting in workflows allowed unsafe functions and arbitrary imports, escaping the intended sandbox and enabling system command execution.
Evidence: aefa2717f7d6d1702e3008ac18ebdcc3660dc256, 96049a32fa70aef0483d4180d386a57b3c2d973a
## Shared Attack Surfaces
Capabilities guarded by more than one past fix on different code paths. A guard enforced on one entry point is often missing on a sibling; verify each alternate path before assuming it's covered.
### 1. Privileged-type opt-in for template execution (code, JavaScript, self-contained, file-protocol, DAST)
- Guard that should hold everywhere: signature verification (Verify, parsed.IsUnsignedJavascriptTemplate), -code flag, parsed.SelfContained flag, parsed.HasFileProtocol() flag, DASTServerToken length >= 16
- Entry points: `pkg/catalog/loader/loader.go`, `pkg/templates/compile.go`, `pkg/protocols/code`, `pkg/protocols/javascript/js.go`, `pkg/templates/workflows.go`
- Check: Verify the workflow sub-template loader enforces signature verification and the -code flag for code templates, AND the self-contained and file-protocol opt-ins, not just the code-template one. Also check the DAST-mode loader path for the same full set.
- Evidence: 9de96e4dda5a03da963b9ae6582f03ea55791a76, fac0b7196d2fbca56c8af8ff8e1899c111041a37, 1427b35cdbcb11a63c338700fe8e228ada8ec59d, 1c440e755a97471c56fb0276ee8a8c4132974645, 63687c2ce07e464f6b38f797d81a8b92597f6ef9, e86f38299765b82ad724fdb701557e0eaff3884d
### 2. Local file access sandbox (LFA) across JS runtime, payloads, extractors, headless
- Guard that should hold everywhere: path canonicalization + prefix check, LfaAllowed flag, template-directory confinement, file:// opt-in, FilesInput opt-in
- Entry points: `pkg/js/compiler`, `pkg/js/global/scripts.go`, `v2/pkg/protocols/common/generators`, `pkg/operators/extractors`, `v2/pkg/protocols/headless/engine/page_actions.go`, `pkg/protocols/common/protocolstate/file.go`
- Check: Inspect the JS runtime's require() source loader and the headless page-actions path: they must enforce the same canonical path containment and LFA flag as the payload generator and extractor paths.
- Evidence: 6f2ade6a9b427c284c15a43445f9c7f055e60e5d, e5154d362af900e83b30fb79028bfab1a7f0c648, e7fb40a41308f6c41267e858d1781eb67f7e0572, f663d1c9cf77da53533bf328750621b4a9fa996d, d3928e080d819f25306bfbaecc399f03516db5aa, 5b89811b906f086e47fdf5357a5ee367f8f1daef
### 3. Network egress policy (host allowlist) for JS helper protocols and remote template loading
- Guard that should hold everywhere: protocolstate.IsHostAllowed, dialers.Fastdialer.Dial/DialTLSWithConfig, remote-template-domain allowlist
- Entry points: `pkg/js/libs/ldap/ldap.go`, `pkg/js/libs/smb/smb.go`, `pkg/js/libs/mysql/mysql.go`, `v2/pkg/catalog/loader`
- Check: Verify the MySQL DSN and LDAP dial paths check the actual target host (not just a derived domain) and that the remote-template loader applies the domain allowlist before http.Get, matching the SMB dial check.
- Evidence: 9def9af7e3f7ebcf5cd561c37b623b9999ee7572, a167e6c57b80f5dc7be231231b62d7c594ee2770, a677fca192ee732a7b0a278f725517fa998b7897, 9611926a9a1ca99e8ed554d1b26fbeff60e06368, d1303797c0e5f88343a1f2ed781955dd086df03d
### 4. Template expression evaluation boundary (response data vs. template DSL)
- Guard that should hold everywhere: collect expressions before placeholder substitution (FindExpressions), treat runtime values as literal data (expressions.Evaluate)
- Entry points: `pkg/protocols/common/expressions`, `pkg/protocols`
- Check: Inspect multi-step template processing where HTTP response data flows into expression evaluation: ensure expressions are collected before substitution on every path, including headless and JavaScript protocol steps.
- Evidence: d2217320162d5782ca7cb95bef9dda17063818f3, ccbfb12bd01447ba25f6e755dfb2bee677736da6
## Historical Vulnerability Classes
| Class | Count |
|---|---|
| Path Traversal | 10 |
| Auth Bypass | 6 |
| Remote Code Execution | 3 |
| Sandbox Escape | 3 |
| SSRF | 2 |
| Arbitrary Code Execution | 1 |
| Arbitrary File Read | 1 |
| Code Execution | 1 |
| Code Execution / Signature Bypass | 1 |
| Cryptographically Weak PRNG | 1 |
| Expression Language Injection | 1 |
| Information Disclosure | 1 |
| Insecure File Permissions | 1 |
| Insecure Permissions | 1 |
| Race Condition | 1 |
| SQL Injection | 1 |
| Sandbox Bypass | 1 |
| Server-Side Request Forgery | 1 |
| Signature Bypass | 1 |
## Fixed-Issue Fingerprints
These are real security fixes mined from git history. Use them as regression checks.
- **Code Execution / Signature Bypass** (critical) [CWE-347]: This commit fixes a signature verification bypass that allowed unsigned templates with JavaScript init blocks to be compiled and executed, by binding verification results to content digests and enforcing verification before compilation.
- sink: `Template compilation and JavaScript execution of unsigned or unverified templates`
- fix: Signature verification enforced before protocol compilation and at request execution; verification result bound to content digest and verifier fingerprint to prevent cache reuse of stale verification.
- commit: https://github.com/projectdiscovery/nuclei/commit/9de96e4dda5a03da963b9ae6582f03ea55791a76 (2026-08-04)
- **Auth Bypass** (high) [CWE-347]: Enforce template signature verification for javascript-protocol templates before loading and executing them.
- sink: `execution path`
- fix: signature check for javascript templates
- commit: https://github.com/projectdiscovery/nuclei/commit/fac0b7196d2fbca56c8af8ff8e1899c111041a37 (2026-07-03)
- **Remote Code Execution** (high) [CWE-94]: Enforce signature verification and the code execution flag (-code) directly within the code protocol execution sink to prevent execution of untrusted templates via indirect SDK/API paths.
- sink: `ExecuteWithResults`
- fix: gated check for enabled and signature-verified templates
- commit: https://github.com/projectdiscovery/nuclei/commit/1427b35cdbcb11a63c338700fe8e228ada8ec59d (2026-07-03)
- **Remote Code Execution** (high) [CWE-94]: Fixes a template-injection risk where dynamic runtime values/extracted payloads were recursively evaluated as DSL expressions instead of being treated as literal data.
- sink: `expressions.Evaluate`
- fix: template rendering boundary
- commit: https://github.com/projectdiscovery/nuclei/commit/ccbfb12bd01447ba25f6e755dfb2bee677736da6 (2026-06-30)
- **Path Traversal** (medium) [CWE-22]: Enforce Nuclei's local file sandbox restrictions on the Kerberos credential cache (ccache) output paths in krbforge helper to prevent arbitrary file writes.
- sink: `github.com/Mzack9999/goimpacket/pkg/kerberos.CreateTicket`
- fix: path canonicalization + prefix check
- commit: https://github.com/projectdiscovery/nuclei/commit/3a8b65cb45f121fcdf418f60e5177ed9f250c122 (2026-06-24)
- **SSRF** (medium) [CWE-918]: Enforce the configured network policy and local network restrictions on LDAP/LDAPS/LDAPI protocols prior to dialing hosts.
- sink: `dialers.Fastdialer.Dial`
- fix: host allowlist
- commit: https://github.com/projectdiscovery/nuclei/commit/9def9af7e3f7ebcf5cd561c37b623b9999ee7572 (2026-06-24)
- **Arbitrary File Read** (high) [CWE-200]: Enforces local file access sandbox restrictions on MySQL connections by disabling the allowAllFiles option in the DSN unless the local file access (-lfa) flag is set.
- sink: `sql.Open`
- fix: host allowlist
- commit: https://github.com/projectdiscovery/nuclei/commit/9611926a9a1ca99e8ed554d1b26fbeff60e06368 (2026-06-19)
- **Code Execution** (high) [CWE-347]: Fixes a signature validation bypass that allowed unsigned Code templates to execute arbitrary local commands when run under DAST mode.
- sink: `LoadTemplatesWithTags`
- fix: signature verification before execution
- commit: https://github.com/projectdiscovery/nuclei/commit/1c440e755a97471c56fb0276ee8a8c4132974645 (2026-06-19)
- **Path Traversal** (medium) [CWE-22]: Fixes path containment validation vulnerabilities across several filesystem boundaries by replacing weak lexical prefix checks (which allowed sibling paths of trusted directories to bypass security restrictions) with path-aware canonical containment checks.
- sink: `filepath.Rel`
- fix: path canonicalization + prefix check
- commit: https://github.com/projectdiscovery/nuclei/commit/20cbf0bd73893c1270582c9778b6b26a5cb11206 (2026-06-04)
- **Path Traversal** (high) [CWE-22]: Fixes a sandbox bypass in the JavaScript runtime compiler and template runner that allowed local file access and module importing regardless of restrictions by canonicalizing symlinks and clearing cached modules between runtime executions.
- sink: `require.DefaultSourceLoader`
- fix: path canonicalization + prefix check
- commit: https://github.com/projectdiscovery/nuclei/commit/6f2ade6a9b427c284c15a43445f9c7f055e60e5d (2026-04-10)
- **Expression Language Injection** (high) [CWE-917]: Fixes an expression injection vulnerability by parsing and collecting template-authored expressions before placeholder substitution, preventing untrusted response-derived data from being re-evaluated as DSL expressions.
- sink: `pkg/protocols/common/expressions/expressions.go`
- fix: collect expressions before placeholder substitution
- commit: https://github.com/projectdiscovery/nuclei/commit/d2217320162d5782ca7cb95bef9dda17063818f3 (2026-03-16)
- **Cryptographically Weak PRNG** (medium) [CWE-338]: Replaces the cryptographically insecure math/rand PRNG with crypto/rand in JavaScript helper functions to prevent predictable token generation.
- sink: `math/rand.Intn`
- fix: crypto/rand replacement
- commit: https://github.com/projectdiscovery/nuclei/commit/44ef9ce34ffcfdfca3a96626abc73d99dd0e83ab (2026-03-15)
- **SQL Injection** (medium) [CWE-89]: Fixes a path-based fuzzing component encoding logic issue that prevented successful detection and isolated fuzz testing of SQL Injection paths.
- sink: `pkg/testutils/fuzzplayground/sqli_test.go`
- fix: path component fuzzing fix
- commit: https://github.com/projectdiscovery/nuclei/commit/19247ae74b99c5871b62d7c465a63edb446ceba1 (2025-08-25)
- **Auth Bypass** (high) [CWE-341]: Fix a race condition in concurrent Nuclei engine executions where the global `LfaAllowed` state could be shared across different execution contexts, allowing unauthorized local file access.
- sink: `protocolstate.NormalizePath`
- fix: path canonicalization + prefix check
- commit: https://github.com/projectdiscovery/nuclei/commit/5b89811b906f086e47fdf5357a5ee367f8f1daef (2025-07-18)
- **Auth Bypass** (medium) [CWE-307]: Enforce a minimum length restriction of 16 characters on the DAST server token option to prevent weak or easily guessable authentication credentials.
- sink: `options.DASTServerToken`
- fix: minimum token length constraint
- commit: https://github.com/projectdiscovery/nuclei/commit/5f0b7eb19b6cf160c8b277c9774f8e624d2069cf (2025-02-13)
- **Auth Bypass** (medium) [CWE-862]: Disables self-contained and file-protocol templates by default to enforce safer execution boundaries and prevent unauthorized local file access or unexpected outbound connections unless explicitly enabled.
- sink: `store.config.ExecutorOptions.Options`
- fix: host allowlist
- commit: https://github.com/projectdiscovery/nuclei/commit/63687c2ce07e464f6b38f797d81a8b92597f6ef9 (2024-11-19)
- **Race Condition** (medium) [CWE-362]: Fixes a race condition in the HTTP request execution where shared connection configurations (including cookie jars and timeouts) were concurrently modified, which could lead to cookie leakage across concurrent sessions.
- sink: `connConfiguration.Connection.SetCookieJar`
- fix: struct cloning
- commit: https://github.com/projectdiscovery/nuclei/commit/3064788d35a8ec8623eb3a54eca920ea289f1259 (2024-08-19)
- **Signature Bypass** (high) [CWE-347]: Fixes a template signature verification bypass by ensuring the digest/signature is strictly parsed from the end of the template rather than extracted via an arbitrary regex match anywhere in the file.
- sink: `Verify`
- fix: signature extraction logic refinement
- commit: https://github.com/projectdiscovery/nuclei/commit/0da993afe6d41b4b1b814e8fad23a2acba13c60a (2024-08-19)
- **Auth Bypass** (high) [CWE-566]: Fixes a Go loop variable reference bug in FileAuthProvider that causes all registered domains to use the credentials of the last secret entry in the configuration list.
- sink: `f.compiled[domain]`
- fix: loop variable copy
- commit: https://github.com/projectdiscovery/nuclei/commit/350fa4c10c43c0888561dfe1ee5fa473d5023f73 (2024-08-05)
- **Arbitrary Code Execution** (high) [CWE-94]: Enforces validation checks for code-execution templates referenced within workflows, preventing unauthorized local code execution via untrusted workflows.
- sink: `workflowTemplates = append(workflowTemplates, template)`
- fix: verification and flag checks before loading code templates in workflows
- commit: https://github.com/projectdiscovery/nuclei/commit/e86f38299765b82ad724fdb701557e0eaff3884d (2024-03-01)
- **SSRF** (high) [CWE-918]: Fixes a security bypass in the LDAP client where host validation checked the domain instead of the actual target domain controller address, allowing SSRF to restricted networks.
- sink: `c.newLdapSession`
- fix: validate target host instead of domain name
- commit: https://github.com/projectdiscovery/nuclei/commit/a167e6c57b80f5dc7be231231b62d7c594ee2770 (2024-01-20)
- **Server-Side Request Forgery** (high) [CWE-918]: Fixes an SSRF/network policy bypass in SMB helper functions by introducing `protocolstate.IsHostAllowed` checks prior to dialing.
- sink: `protocolstate.Dialer.Dial`
- fix: host allowlist
- commit: https://github.com/projectdiscovery/nuclei/commit/a677fca192ee732a7b0a278f725517fa998b7897 (2024-01-18)
- **Path Traversal** (high) [CWE-22]: Removes the insecure file extraction `to` feature from extractors and enforces local file access (LFA) boundary checks on headless screenshot file writes to prevent arbitrary file write vulnerabilities via malicious templates.
- sink: `os.OpenFile`
- fix: path canonicalization + prefix check
- commit: https://github.com/projectdiscovery/nuclei/commit/f663d1c9cf77da53533bf328750621b4a9fa996d (2024-01-05)
- **Sandbox Bypass** (high) [CWE-693]: Disables the 'eval' function and strengthens file validation within the JavaScript sandbox environment to prevent sandbox escape and unauthorized code execution.
- sink: `eval`
- fix: disable eval function
- commit: https://github.com/projectdiscovery/nuclei/commit/c35162c8ef061a96dd74b54c8f022c1a41cf5833 (2023-10-13)
- **Path Traversal** (high) [CWE-22]: Restricts local file input actions and file:// protocol access in headless browser protocols unless explicitly enabled via command-line flags.
- sink: `p.FilesInput`
- fix: host allowlist
- commit: https://github.com/projectdiscovery/nuclei/commit/d3928e080d819f25306bfbaecc399f03516db5aa (2023-08-25)
- **Path Traversal** (medium) [CWE-22]: Implements a file-access sandbox to prevent malicious templates from reading arbitrary local files via payloads by restricting access to the template directory unless explicitly allowed.
- sink: `os.Open`
- fix: path canonicalization + prefix check
- commit: https://github.com/projectdiscovery/nuclei/commit/e5154d362af900e83b30fb79028bfab1a7f0c648 (2023-07-14)
- **Auth Bypass** (high) [CWE-347]: Introduces a code protocol for template execution alongside a cryptographic signature verification mechanism (RSA/ECDSA) to ensure that only signed, trusted templates can execute local commands or scripts.
- sink: `v2/pkg/protocols/code/code.go`
- fix: introducing command + trivial trust store mechanism
- commit: https://github.com/projectdiscovery/nuclei/commit/a7fb15d0bd548257fe70af3cdddd469f21a75e6a (2023-06-09)
- **Path Traversal** (medium) [CWE-22]: Introduces a sandbox restriction that validates and confines template payload file paths to the template directory, preventing arbitrary local file reads when safe template execution is enabled.
- sink: `generator.loadPayloadsFromFile(pt)`
- fix: path canonicalization + prefix check
- commit: https://github.com/projectdiscovery/nuclei/commit/e7fb40a41308f6c41267e858d1781eb67f7e0572 (2022-11-16)
- **Insecure File Permissions** (medium) [CWE-732]: Restricts the file permissions of written debug data files from world-writable/readable (os.ModePerm) to owner-readable/writable (0644).
- sink: `os.OpenFile`
- fix: restrict file permissions to 0644
- commit: https://github.com/projectdiscovery/nuclei/commit/af4854f90de728bb08689d5cae3dec4acfdb6b9a (2022-06-20)
- **Path Traversal** (medium) [CWE-22]: Sanitizes special and directory traversal characters from generated markdown filenames to prevent arbitrary file write/path traversal.
- sink: `os.OpenFile`
- fix: character replacement/sanitization
- commit: https://github.com/projectdiscovery/nuclei/commit/e510bc207f4336058f7b1536ddbb3942b394b05c (2022-04-06)
- **Remote Code Execution** (high) [CWE-829]: Implements a domain restriction mechanism (remote-template-domain allowlist) to prevent loading and executing arbitrary untrusted remote templates from arbitrary hostnames.
- sink: `http.Get`
- fix: host allowlist
- commit: https://github.com/projectdiscovery/nuclei/commit/d1303797c0e5f88343a1f2ed781955dd086df03d (2022-01-12)
- **Information Disclosure** (medium) [CWE-532]: Prevents disclosure of sensitive AWS credentials by pruning internal-only signing variables (aws-id and aws-secret) from the event metadata payload.
- sink: `eventcreator.CreateEventWithAdditionalOptions`
- fix: output pruning
- commit: https://github.com/projectdiscovery/nuclei/commit/c26a1ac21c32f716635738387d5e7d369d7538d1 (2022-01-09)
- **Path Traversal** (medium) [CWE-22]: Replaces filepath.Join with path.Join when processing HTTP request paths to prevent platform-specific path manipulation issues on Windows.
- sink: `filepath.Join`
- fix: replace filepath with path
- commit: https://github.com/projectdiscovery/nuclei/commit/d15298648527f4edd1015e077201551b3a6eab06 (2021-12-16)
- **Sandbox Escape** (low) [CWE-250]: Enables Chromium's security sandbox by default in headless mode, disabling it only when running as root on Linux where it is strictly required.
- sink: `chromeLauncher`
- fix: conditional sandbox disablement based on OS and user privileges
- commit: https://github.com/projectdiscovery/nuclei/commit/df55f7a2eb591a7949b5c66a9c54db54aa28f450 (2021-12-05)
- **Path Traversal** (high) [CWE-22]: Fixes a Zip Slip (Path Traversal) vulnerability in the template updater by validating and sanitizing paths before writing unzipped files.
- sink: `os.OpenFile`
- fix: path validation in calculateTemplateAbsolutePath
- commit: https://github.com/projectdiscovery/nuclei/commit/f9c214a66f37a00fc29fe64232b690e6285ec59c (2021-11-25)
- **Insecure Permissions** (medium) [CWE-732]: Fixes an insecure file and directory creation permissions vulnerability by replacing overly permissive permissions (0777/ModePerm) with restrictive defaults (0644 for files, 0755 for directories).
- sink: `os.OpenFile,ioutil.WriteFile,os.MkdirAll`
- fix: restrictive file/directory permissions (0644/0755)
- commit: https://github.com/projectdiscovery/nuclei/commit/4bccb6cf8a830724b3bda624cb3bfe43eff31683 (2021-11-24)
- **Sandbox Escape** (high) [CWE-116]: Introduces payload validation for template generators to restrict execution of unsafe functions within Nuclei templates.
- sink: `Go-template compiler/executor`
- fix: input validation / payload sandbox checks
- commit: https://github.com/projectdiscovery/nuclei/commit/aefa2717f7d6d1702e3008ac18ebdcc3660dc256 (2020-12-29)
- **Sandbox Escape** (high) [CWE-265]: Restricts the Tengo scripting engine imports in workflows to a safe subset of standard library modules, preventing arbitrary system command execution.
- sink: `tengo.NewScript`
- fix: import allowlist restriction
- commit: https://github.com/projectdiscovery/nuclei/commit/96049a32fa70aef0483d4180d386a57b3c2d973a (2020-12-13)
## Known CVEs
- **CVE-2026-41645** (medium 5.3) CWE-94
- Nuclei 3.0.0 to < 3.8.0 contains an expression injection vulnerability caused by reuse of HTTP response data with helper/function syntax in multi-step templates, letting malicious target servers execu...
- **CVE-2026-41646** (medium 5.5) CWE-284
- Nuclei 3.0.0 to < 3.8.0 contains a local file read vulnerability caused by JavaScript protocol runtime allowing require() to access local .js and .json files, letting attackers read restricted files, ...
- **CVE-2026-41282** (medium 4.0) CWE-94
- ProjectDiscovery Nuclei < 3.8.0 contains a template injection caused by unsafe use of -env-vars in multi-step templates against untrusted targets, letting attackers inject DSL expressions, exploit req...
- **CVE-2024-43405** (high 7.4) CWE-78
- Nuclei versions prior to 3.3.2 contain a signature verification bypass caused by discrepancies in newline handling between the verification process and YAML parser, letting attackers inject malicious ...
- **CVE-2024-27920** (high 7.4) CWE-78
- projectdiscovery/nuclei v3 contains a vulnerability caused by execution of unsigned code templates through workflows, letting malicious code run on the user's system, exploit requires using custom wor...
- **CVE-2023-37896** (high 7.5) CWE-22
- Nuclei prior to version 2.9.9 contains a sandbox sanitization issue caused by relative path handling in payload loading, letting arbitrary files be read on the filesystem when used as Go SDK, exploit ...
---
_Fixed-issue fingerprints and CVE matches are derived from real data; LLM-written prose is advisory and should be verified against the cited commits._
+72
View File
@@ -0,0 +1,72 @@
package main
import (
"os"
"path/filepath"
"testing"
"github.com/adrg/xdg"
"github.com/projectdiscovery/goflags"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/internal/configuration"
catalogconfig "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
)
func TestAuthValueIsCLIOnly(t *testing.T) {
oldConfigHome := xdg.ConfigHome
oldConfigDirs := xdg.ConfigDirs
t.Cleanup(func() {
xdg.ConfigHome = oldConfigHome
xdg.ConfigDirs = oldConfigDirs
})
const apiKey = "12345678-1234-1234-1234-123456789012"
tests := []struct {
name string
args func(string) []string
file string
want string
}{
{name: "explicit config", args: func(path string) []string { return []string{"-config=" + path} }, file: "auth: true\n"},
{name: "profile", args: func(path string) []string { return []string{"-profile=" + path} }, file: "auth: " + apiKey + "\n"},
{name: "CLI", args: func(string) []string { return []string{"-auth=" + apiKey} }, want: apiKey},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
dir := t.TempDir()
xdg.ConfigHome = filepath.Join(dir, "config-home")
xdg.ConfigDirs = nil
selectedPath := filepath.Join(dir, "selected.yaml")
if test.file != "" {
if err := os.WriteFile(selectedPath, []byte(test.file), 0o600); err != nil {
t.Fatalf("write selected configuration: %v", err)
}
}
var auth string
var selected configuration.Selection
flagSet := goflags.NewFlagSet()
flagSet.DynamicVar(&auth, "auth", "true", "")
flagSet.StringVar(&selected.Config, "config", "", "")
flagSet.StringVar(&selected.Profile, "profile", "", "")
cfg := &catalogconfig.Config{Logger: gologger.DefaultLogger}
cfg.SetConfigDir(filepath.Join(xdg.ConfigHome, catalogconfig.BinaryName))
warnings, err := configuration.Load(cfg, flagSet, &selected, func(path string) (string, error) {
return path, nil
}, test.args(selectedPath)...)
if err != nil {
t.Fatalf("load configuration: %v", err)
}
if len(warnings) != 0 {
t.Fatalf("configuration warnings: %v", warnings)
}
if got := cliOnlyDynamicValue(flagSet, "auth", auth); got != test.want {
t.Fatalf("auth value = %q, want %q", got, test.want)
}
})
}
}
+186 -275
View File
@@ -2,6 +2,7 @@ package main
import (
"bufio"
"flag"
"fmt"
"io/fs"
"os"
@@ -11,11 +12,11 @@ import (
"runtime/pprof"
"runtime/trace"
"strings"
"sync"
"time"
"github.com/projectdiscovery/gologger"
_pdcp "github.com/projectdiscovery/nuclei/v3/internal/pdcp"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/yaml"
"github.com/projectdiscovery/utils/auth/pdcp"
"github.com/projectdiscovery/utils/env"
_ "github.com/projectdiscovery/utils/pprof"
@@ -25,6 +26,7 @@ import (
"github.com/projectdiscovery/goflags"
"github.com/projectdiscovery/gologger/levels"
"github.com/projectdiscovery/interactsh/pkg/client"
"github.com/projectdiscovery/nuclei/v3/internal/configuration"
"github.com/projectdiscovery/nuclei/v3/internal/runner"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/input/provider"
@@ -47,27 +49,22 @@ import (
)
var (
cfgFile string
templateProfile string
memProfile string // optional profile file path
options = &types.Options{}
inlineSecretsTempFiles []string
memProfile string // optional profile file path
options = &types.Options{}
inlineSecretsTempDir string
inlineSecretsTempMu sync.Mutex
)
func main() {
options.Logger = gologger.DefaultLogger
defer func() {
for _, f := range inlineSecretsTempFiles {
_ = os.Remove(f)
}
}()
defer cleanupInlineSecretsDirs()
// enables CLI specific configs mostly interactive behavior
config.CurrentAppMode = config.AppModeCLI
if err := runner.ConfigureOptions(); err != nil {
options.Logger.Fatal().Msgf("Could not initialize options: %s\n", err)
fatalf("Could not initialize options: %s\n", err)
}
_ = readConfig()
@@ -83,7 +80,7 @@ func main() {
templates.UseOptionsForSigner(options)
tsigner, err := signer.NewTemplateSigner(nil, nil) // will read from env , config or generate new keys
if err != nil {
options.Logger.Fatal().Msgf("couldn't initialize signer crypto engine: %s\n", err)
fatalf("Could not initialize signer crypto engine: %s\n", err)
}
successCounter := 0
@@ -122,7 +119,7 @@ func main() {
createProfileFile := func(ext, profileType string) *os.File {
f, err := os.Create(memProfile + ext)
if err != nil {
options.Logger.Fatal().Msgf("profile: could not create %s profile %q file: %v", profileType, f.Name(), err)
fatalf("profile: could not create %s profile %q file: %v", profileType, memProfile+ext, err)
}
return f
}
@@ -136,18 +133,18 @@ func main() {
// Start tracing
if err := trace.Start(traceFile); err != nil {
options.Logger.Fatal().Msgf("profile: could not start trace: %v", err)
fatalf("profile: could not start trace: %v", err)
}
// Start CPU profiling
if err := pprof.StartCPUProfile(cpuProfileFile); err != nil {
options.Logger.Fatal().Msgf("profile: could not start CPU profile: %v", err)
fatalf("profile: could not start CPU profile: %v", err)
}
defer func() {
// Start heap memory snapshot
if err := pprof.WriteHeapProfile(memProfileFile); err != nil {
options.Logger.Fatal().Msgf("profile: could not write memory profile: %v", err)
fatalf("profile: could not write memory profile: %v", err)
}
pprof.StopCPUProfile()
@@ -169,14 +166,14 @@ func main() {
if options.ScanUploadFile != "" {
if err := runner.UploadResultsToCloud(options); err != nil {
options.Logger.Fatal().Msgf("could not upload scan results to cloud dashboard: %s\n", err)
fatalf("Could not upload scan results to cloud dashboard: %s\n", err)
}
return
}
nucleiRunner, err := runner.New(options)
if err != nil {
options.Logger.Fatal().Msgf("Could not create runner: %s\n", err)
fatalf("Could not create runner: %s\n", err)
}
if nucleiRunner == nil {
return
@@ -187,7 +184,7 @@ func main() {
cancel := stackMonitor.Start(10 * time.Second)
defer cancel()
stackMonitor.RegisterCallback(func(dumpID string) error {
resumeFileName := fmt.Sprintf("crash-resume-file-%s.dump", dumpID)
resumeFileName := types.DefaultCrashResumeFilePath(dumpID)
if options.EnableCloudUpload {
options.Logger.Info().Msgf("Uploading scan results to cloud...")
}
@@ -213,6 +210,7 @@ func main() {
options.Logger.Info().Msgf("CTRL+C pressed: Exiting\n")
if options.DASTServer {
nucleiRunner.Close()
cleanupInlineSecretsDirs()
os.Exit(1)
}
@@ -228,17 +226,15 @@ func main() {
options.Logger.Error().Msgf("Couldn't create resume file: %s\n", err)
}
}
for _, f := range inlineSecretsTempFiles {
_ = os.Remove(f)
}
cleanupInlineSecretsDirs()
os.Exit(1)
}()
if err := nucleiRunner.RunEnumeration(); err != nil {
if options.Validate {
options.Logger.Fatal().Msgf("Could not validate templates: %s\n", err)
fatalf("Could not validate templates: %s\n", err)
} else {
options.Logger.Fatal().Msgf("Could not run nuclei: %s\n", err)
fatalf("Could not run nuclei: %s\n", err)
}
}
nucleiRunner.Close()
@@ -249,10 +245,12 @@ func main() {
}
func readConfig() *goflags.FlagSet {
var selectedFiles configuration.Selection
// when true updates nuclei binary to latest version
var updateNucleiBinary bool
var pdcpauth string
var disableUpdateCheck bool
var resetRequested bool
var pdcpAuth string
var fuzzFlag bool
flagSet := goflags.NewFlagSet()
@@ -348,8 +346,8 @@ on extensive configurability, massive extensibility and ease of use.`)
)
flagSet.CreateGroup("configs", "Configurations",
flagSet.StringVar(&cfgFile, "config", "", "path to the nuclei configuration file"),
flagSet.StringVarP(&templateProfile, "profile", "tp", "", "template profile config file to run"),
flagSet.StringVar(&selectedFiles.Config, "config", "", "path to the nuclei configuration file"),
flagSet.StringVarP(&selectedFiles.Profile, "profile", "tp", "", "template profile config file to run"),
flagSet.BoolVarP(&options.ListTemplateProfiles, "profile-list", "tpl", false, "list community template profiles"),
flagSet.BoolVarP(&options.FollowRedirects, "follow-redirects", "fr", false, "enable following redirects for http templates"),
flagSet.BoolVarP(&options.FollowHostRedirects, "follow-host-redirects", "fhr", false, "follow redirects on the same host"),
@@ -378,7 +376,7 @@ on extensive configurability, massive extensibility and ease of use.`)
flagSet.StringVarP(&options.SourceIP, "source-ip", "sip", "", "source ip address to use for network scan"),
flagSet.IntVarP(&options.ResponseReadSize, "response-size-read", "rsr", 0, "max response size to read in bytes"),
flagSet.IntVarP(&options.ResponseSaveSize, "response-size-save", "rss", unitutils.Mega, "max response size to read in bytes"),
flagSet.CallbackVar(resetCallback, "reset", "reset removes all nuclei configuration and data files (including nuclei-templates)"),
flagSet.CallbackVar(func() { resetRequested = true }, "reset", "reset removes all nuclei configuration and data files (including nuclei-templates)"),
flagSet.BoolVarP(&options.TlsImpersonate, "tls-impersonate", "tlsi", false, "enable experimental client hello (ja3) tls randomization"),
flagSet.StringVarP(&options.HttpApiEndpoint, "http-api-endpoint", "hae", "", "experimental http api endpoint"),
)
@@ -489,7 +487,7 @@ on extensive configurability, massive extensibility and ease of use.`)
flagSet.BoolVarP(&updateNucleiBinary, "update", "up", false, "update nuclei engine to the latest released version"),
flagSet.BoolVarP(&options.UpdateTemplates, "update-templates", "ut", false, "update nuclei-templates to latest released version"),
flagSet.StringVarP(&options.NewTemplatesDirectory, "update-template-dir", "ud", "", "custom directory to install / update nuclei-templates"),
flagSet.CallbackVarP(disableUpdatesCallback, "disable-update-check", "duc", "disable automatic nuclei/templates update check"),
flagSet.BoolVarP(&disableUpdateCheck, "disable-update-check", "duc", false, "disable automatic nuclei/templates update check"),
)
flagSet.CreateGroup("Honeypot", "Honeypot",
@@ -507,7 +505,7 @@ on extensive configurability, massive extensibility and ease of use.`)
)
flagSet.CreateGroup("cloud", "Cloud",
flagSet.DynamicVar(&pdcpauth, "auth", "true", "configure projectdiscovery cloud (pdcp) api key"),
flagSet.DynamicVar(&pdcpAuth, "auth", "true", "configure projectdiscovery cloud (pdcp) api key"),
flagSet.StringVarP(&options.TeamID, "team-id", "tid", _pdcp.TeamIDEnv, "upload scan results to given team id (optional)"),
flagSet.BoolVarP(&options.EnableCloudUpload, "cloud-upload", "cup", false, "upload scan results to pdcp dashboard [DEPRECATED use -dashboard]"),
flagSet.StringVarP(&options.ScanID, "scan-id", "sid", "", "upload scan results to existing scan id (optional)"),
@@ -540,12 +538,32 @@ Run nuclei with sorted Markdown outputs (with environment variables):
Additional documentation is available at: https://docs.nuclei.sh/getting-started/running
`)
// nuclei has multiple migrations
// ex: resume.cfg moved to platform standard cache dir from config dir
// ex: config.yaml moved to platform standard config dir from linux specific config dir
// and hence it will be attempted in config package during init
// Nuclei handles config-path compatibility before parsing its layered files.
goflags.DisableAutoConfigMigration = true
_ = flagSet.Parse()
warnings, err := configuration.Load(config.DefaultConfig, flagSet, &selectedFiles, func(path string) (string, error) {
return configuration.ResolveProfilePath(path, configuration.TemplatesDirectory(options, config.DefaultConfig))
})
for _, warning := range warnings {
options.Logger.Warning().Msgf("Could not load configuration: %s\n", warning)
}
if err != nil {
fatalf("Could not load configuration: %s\n", err)
}
pdcpAuth = cliOnlyDynamicValue(flagSet, "auth", pdcpAuth)
if disableUpdateCheck {
config.DefaultConfig.DisableUpdateCheck()
}
if templatesDir, overridden := configuration.TemplatesDirectoryOverride(options); overridden {
config.DefaultConfig.SetTemplatesDir(templatesDir)
}
if resetRequested {
resetCallback()
}
// when fuzz flag is enabled, set the dast flag to true
if fuzzFlag {
@@ -559,13 +577,13 @@ Additional documentation is available at: https://docs.nuclei.sh/getting-started
}
// api key hierarchy: cli flag > env var > .pdcp/credential file
if pdcpauth == "true" {
if pdcpAuth == "true" {
runner.AuthWithPDCP()
} else if len(pdcpauth) == 36 {
} else if len(pdcpAuth) == 36 {
ph := pdcp.PDCPCredHandler{}
if _, err := ph.GetCreds(); err == pdcp.ErrNoCreds {
apiServer := env.GetEnvOrDefault("PDCP_API_SERVER", pdcp.DefaultApiServer)
if validatedCreds, err := ph.ValidateAPIKey(pdcpauth, apiServer, config.BinaryName); err == nil {
if validatedCreds, err := ph.ValidateAPIKey(pdcpAuth, apiServer, config.BinaryName); err == nil {
_ = ph.SaveCreds(validatedCreds)
}
}
@@ -576,7 +594,7 @@ Additional documentation is available at: https://docs.nuclei.sh/getting-started
h := &pdcp.PDCPCredHandler{}
_, err := h.GetCreds()
if err != nil {
options.Logger.Fatal().Msg("To utilize the `-ai` flag, please configure your API key with the `-auth` flag or set the `PDCP_API_KEY` environment variable")
fatalf("To utilize the `-ai` flag, please configure your API key with the `-auth` flag or set the `PDCP_API_KEY` environment variable")
}
}
@@ -595,143 +613,34 @@ Additional documentation is available at: https://docs.nuclei.sh/getting-started
runner.NucleiToolUpdateCallback()
}
tempSecretsDir, err := configuration.ApplyProfile(selectedFiles.Profile, options)
if err != nil {
fatalf("Could not process template profile: %s\n", err)
}
if tempSecretsDir != "" {
setInlineSecretsTempDir(tempSecretsDir)
}
if options.LeaveDefaultPorts {
http.LeaveDefaultPorts = true
}
if customConfigDir := os.Getenv(config.NucleiConfigDirEnv); customConfigDir != "" {
config.DefaultConfig.SetConfigDir(customConfigDir)
readFlagsConfig(flagSet)
}
if cfgFile != "" {
if !fileutil.FileExists(cfgFile) {
options.Logger.Fatal().Msgf("given config file '%s' does not exist", cfgFile)
}
// merge config file with flags
if err := flagSet.MergeConfigFile(cfgFile); err != nil {
options.Logger.Fatal().Msgf("Could not read config: %s\n", err)
}
if !options.Vars.IsEmpty() {
// Maybe we should add vars to the config file as well even if they are set via flags?
file, err := os.Open(cfgFile)
if err != nil {
gologger.Fatal().Msgf("Could not open config file: %s\n", err)
}
defer func() {
_ = file.Close()
}()
data := make(map[string]interface{})
err = yaml.NewDecoder(file).Decode(&data)
if err != nil {
gologger.Fatal().Msgf("Could not decode config file: %s\n", err)
}
variables := data["var"]
if variables != nil {
if varSlice, ok := variables.([]interface{}); ok {
for _, value := range varSlice {
if strVal, ok := value.(string); ok {
err = options.Vars.Set(strVal)
if err != nil {
gologger.Warning().Msgf("Could not set variable from config file: %s\n", err)
}
} else {
gologger.Warning().Msgf("Skipping non-string variable in config: %#v", value)
}
}
} else {
gologger.Warning().Msgf("No 'var' section found in config file: %s", cfgFile)
}
}
}
}
templatesDir := options.NewTemplatesDirectory
if templatesDir == "" {
templatesDir = os.Getenv(config.NucleiTemplatesDirEnv)
}
if templatesDir != "" {
config.DefaultConfig.SetTemplatesDir(templatesDir)
}
defaultProfilesPath := filepath.Join(config.DefaultConfig.GetTemplateDir(), "profiles")
if templateProfile != "" {
if filepath.Ext(templateProfile) == "" {
if tp := findProfilePathById(templateProfile, defaultProfilesPath); tp != "" {
templateProfile = tp
} else {
options.Logger.Fatal().Msgf("'%s' is not a profile-id or profile path", templateProfile)
}
}
if !filepath.IsAbs(templateProfile) {
if filepath.Dir(templateProfile) == "profiles" {
defaultProfilesPath = filepath.Join(config.DefaultConfig.GetTemplateDir())
}
currentDir, err := os.Getwd()
if err == nil && fileutil.FileExists(filepath.Join(currentDir, templateProfile)) {
templateProfile = filepath.Join(currentDir, templateProfile)
} else {
templateProfile = filepath.Join(defaultProfilesPath, templateProfile)
}
}
if !fileutil.FileExists(templateProfile) {
options.Logger.Fatal().Msgf("given template profile file '%s' does not exist", templateProfile)
}
if err := flagSet.MergeConfigFile(templateProfile); err != nil {
options.Logger.Fatal().Msgf("Could not read template profile: %s\n", err)
}
// Process inline target list from profile.
// Supports both the dedicated targets-inline key and multiline
// content in the list key (which normally holds a file path).
if options.InlineTargetsList != "" {
inlineTargets := strings.Split(strings.TrimSpace(options.InlineTargetsList), "\n")
for _, target := range inlineTargets {
target = strings.TrimSpace(target)
if target != "" && !strings.HasPrefix(target, "#") {
options.Targets = append(options.Targets, target)
}
}
}
if strings.Contains(options.TargetsFilePath, "\n") {
// list key has multiline content, treat as inline targets
inlineTargets := strings.Split(strings.TrimSpace(options.TargetsFilePath), "\n")
for _, target := range inlineTargets {
target = strings.TrimSpace(target)
if target != "" && !strings.HasPrefix(target, "#") {
options.Targets = append(options.Targets, target)
}
}
options.TargetsFilePath = ""
}
// Process inline secrets from profile YAML
tempSecretsFile, err := processInlineSecretsFromProfile(templateProfile, options)
if err != nil {
options.Logger.Fatal().Msgf("Could not process inline secrets: %s\n", err)
}
if tempSecretsFile != "" {
inlineSecretsTempFiles = append(inlineSecretsTempFiles, tempSecretsFile)
}
}
if len(options.SecretsFile) > 0 {
for _, secretFile := range options.SecretsFile {
if !fileutil.FileExists(secretFile) {
options.Logger.Fatal().Msgf("given secrets file '%s' does not exist", secretFile)
fatalf("Secrets file '%s' does not exist", secretFile)
}
}
}
cleanupOldResumeFiles()
return flagSet
}
// cleanupOldResumeFiles cleans up resume files older than 10 days.
func cleanupOldResumeFiles() {
root := config.DefaultConfig.GetCacheDir()
root := config.DefaultConfig.GetStateDir()
filter := fileutil.FileFilters{
OlderThan: 24 * time.Hour * 10, // cleanup on the 10th day
Prefix: "resume-",
@@ -739,40 +648,6 @@ func cleanupOldResumeFiles() {
_ = fileutil.DeleteFilesOlderThan(root, filter)
}
// readFlagsConfig reads the config file from the default config dir and copies it to the current config dir.
func readFlagsConfig(flagset *goflags.FlagSet) {
// check if config.yaml file exists
defaultCfgFile, err := flagset.GetConfigFilePath()
if err != nil {
// something went wrong either dir is not readable or something else went wrong upstream in `goflags`
// warn and exit in this case
options.Logger.Warning().Msgf("Could not read config file: %s\n", err)
return
}
cfgFile := config.DefaultConfig.GetFlagsConfigFilePath()
if !fileutil.FileExists(cfgFile) {
if !fileutil.FileExists(defaultCfgFile) {
// if default config does not exist, warn and exit
options.Logger.Warning().Msgf("missing default config file : %s", defaultCfgFile)
return
}
// if does not exist copy it from the default config
if err = fileutil.CopyFile(defaultCfgFile, cfgFile); err != nil {
options.Logger.Warning().Msgf("Could not copy config file: %s\n", err)
}
return
}
// if config file exists, merge it with the default config
if err = flagset.MergeConfigFile(cfgFile); err != nil {
options.Logger.Warning().Msgf("failed to merge configfile with flags got: %s\n", err)
}
}
// disableUpdatesCallback disables the update check.
func disableUpdatesCallback() {
config.DefaultConfig.DisableUpdateCheck()
}
// printVersion prints the nuclei version and exits.
func printVersion() {
options.Logger.Info().Msgf("Nuclei Engine Version: %s", config.Version)
@@ -800,119 +675,155 @@ func printTemplateVersion() {
os.Exit(0)
}
type resetPath struct {
name string
path string
}
func resetCallback() {
cfg := config.DefaultConfig
templateRoot := cfg.TemplatesDirectory
templateAction := fmt.Sprintf("4. Active nuclei-templates at %v", templateRoot)
warning := fmt.Sprintf(`
Using '-reset' will delete all nuclei configurations files and all nuclei-templates
Using '-reset' will delete Nuclei configuration, state, cache, and template files
Following files will be deleted:
1. All config files at %v
2. All cache files (including resume state) at %v
3. All nuclei-templates at %v
2. All state files (including generated resume state) at %v
3. All cache files at %v
%s
Note: Make sure you have backup of your custom nuclei-templates before proceeding
`, config.DefaultConfig.GetConfigDir(), config.DefaultConfig.GetCacheDir(), config.DefaultConfig.TemplatesDirectory)
`, cfg.GetConfigDir(), cfg.GetStateDir(), cfg.GetCacheDir(), templateAction)
options.Logger.Print().Msg(warning)
reader := bufio.NewReader(os.Stdin)
for {
fmt.Print("Are you sure you want to continue? [y/n]: ")
resp, err := reader.ReadString('\n')
if err != nil {
options.Logger.Fatal().Msgf("could not read response: %s", err)
fatalf("Could not read response: %s", err)
}
resp = strings.TrimSpace(resp)
if stringsutil.EqualFoldAny(resp, "y", "yes") {
break
}
if stringsutil.EqualFoldAny(resp, "n", "no", "") {
fmt.Println("Exiting...")
os.Exit(0)
}
}
err := os.RemoveAll(config.DefaultConfig.GetConfigDir())
if err != nil {
options.Logger.Fatal().Msgf("could not delete config dir: %s", err)
resetPaths := []resetPath{
{name: "config", path: cfg.GetConfigDir()},
{name: "state", path: cfg.GetStateDir()},
{name: "cache", path: cfg.GetCacheDir()},
{name: "templates", path: templateRoot},
}
err = os.RemoveAll(config.DefaultConfig.GetCacheDir())
if err != nil {
options.Logger.Fatal().Msgf("could not delete cache dir: %s", err)
if err := removeResetPaths(resetPaths); err != nil {
fatalf("Could not reset paths: %s", err)
}
err = os.RemoveAll(config.DefaultConfig.TemplatesDirectory)
if err != nil {
options.Logger.Fatal().Msgf("could not delete templates dir: %s", err)
}
options.Logger.Info().Msgf("Successfully deleted all nuclei configurations files and nuclei-templates")
options.Logger.Info().Msgf("Successfully reset Nuclei files")
os.Exit(0)
}
func findProfilePathById(profileId, templatesDir string) string {
var profilePath string
err := filepath.WalkDir(templatesDir, func(iterItem string, d fs.DirEntry, err error) error {
ext := filepath.Ext(iterItem)
isYaml := ext == extensions.YAML || ext == extensions.YML
if err != nil || d.IsDir() || !isYaml {
// skip non yaml files
return nil
func removeResetPaths(paths []resetPath) error {
for _, path := range paths {
if err := validateResetPath(path.path); err != nil {
return fmt.Errorf("error validating %s path: %w", path.name, err)
}
if strings.TrimSuffix(filepath.Base(iterItem), ext) == profileId {
profilePath = iterItem
return fmt.Errorf("FOUND")
}
for _, path := range paths {
if err := os.RemoveAll(path.path); err != nil {
return fmt.Errorf("error deleting %s path: %w", path.name, err)
}
}
return nil
}
func validateResetPath(path string) error {
if path == "" {
return fmt.Errorf("path is empty")
}
absPath, err := filepath.Abs(path)
if err != nil {
return fmt.Errorf("resolve %q: %w", path, err)
}
absPath = filepath.Clean(absPath)
volume := filepath.VolumeName(absPath)
if absPath == filepath.Clean(volume+string(filepath.Separator)) {
return fmt.Errorf("path %q is a filesystem root", absPath)
}
relativePath := strings.Trim(filepath.ToSlash(strings.TrimPrefix(absPath, volume)), "/")
if relativePath == "" || len(strings.Split(relativePath, "/")) < 2 {
return fmt.Errorf("path %q is too broad", absPath)
}
if homeDir, err := os.UserHomeDir(); err == nil && absPath == filepath.Clean(homeDir) {
return fmt.Errorf("path %q is the user home directory", absPath)
}
if absPath == filepath.Clean(os.TempDir()) {
return fmt.Errorf("path %q is the system temporary directory", absPath)
}
if cwd, err := os.Getwd(); err == nil && absPath == filepath.Clean(cwd) {
return fmt.Errorf("path %q is the current working directory", absPath)
}
if resolvedPath, err := filepath.EvalSymlinks(absPath); err == nil && resolvedPath != absPath {
if err := validateResetPath(resolvedPath); err != nil {
return fmt.Errorf("resolved path %q is unsafe: %w", resolvedPath, err)
}
}
return nil
}
func cleanupInlineSecretsDirs() {
inlineSecretsTempMu.Lock()
dir := inlineSecretsTempDir
inlineSecretsTempDir = ""
inlineSecretsTempMu.Unlock()
if dir != "" {
_ = os.RemoveAll(dir)
}
}
func setInlineSecretsTempDir(dir string) {
inlineSecretsTempMu.Lock()
inlineSecretsTempDir = dir
inlineSecretsTempMu.Unlock()
}
func cliOnlyDynamicValue(flagSet *goflags.FlagSet, name, value string) string {
found := false
flagSet.CommandLine.Visit(func(current *flag.Flag) {
if current.Name == name {
found = true
}
return nil
})
if err != nil && err.Error() != "FOUND" {
options.Logger.Error().Msgf("%s\n", err)
if !found {
return ""
}
return profilePath
return value
}
// profileSecrets is a helper struct to extract secrets section from a template profile YAML
type profileSecrets struct {
Secrets interface{} `yaml:"secrets"`
}
// processInlineSecretsFromProfile parses the profile YAML file for inline secrets
// and creates a temporary secrets file compatible with nuclei's auth provider.
// Returns the path to the temp file or empty string if no secrets found.
func processInlineSecretsFromProfile(profilePath string, options *types.Options) (string, error) {
data, err := os.ReadFile(profilePath)
if err != nil {
return "", fmt.Errorf("could not read profile file: %w", err)
}
var profile profileSecrets
if err := yaml.Unmarshal(data, &profile); err != nil {
return "", fmt.Errorf("could not parse profile YAML: %w", err)
}
if profile.Secrets == nil {
return "", nil
}
secretsData, err := yaml.Marshal(profile.Secrets)
if err != nil {
return "", fmt.Errorf("could not marshal inline secrets: %w", err)
}
tempDir := filepath.Join(os.TempDir(), "nuclei-secrets")
if err := os.MkdirAll(tempDir, 0700); err != nil {
return "", fmt.Errorf("could not create temp directory: %w", err)
}
tempFile, err := os.CreateTemp(tempDir, "inline-secrets-*.yaml")
if err != nil {
return "", fmt.Errorf("could not create temp secrets file: %w", err)
}
defer func() {
_ = tempFile.Close()
}()
if _, err := tempFile.Write(secretsData); err != nil {
_ = tempFile.Close()
_ = os.Remove(tempFile.Name())
return "", fmt.Errorf("could not write to temp secrets file: %w", err)
}
options.SecretsFile = append(options.SecretsFile, tempFile.Name())
return tempFile.Name(), nil
func fatalf(format string, args ...any) {
cleanupInlineSecretsDirs()
options.Logger.Fatal().Msgf(format, args...)
}
+184
View File
@@ -0,0 +1,184 @@
package main
import (
"bytes"
"os"
"os/exec"
"path/filepath"
"sync"
"testing"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
)
const inlineSecretsFatalChildEnv = "NUCLEI_INLINE_SECRETS_FATAL_CHILD"
func TestValidateResetPathRejectsBroadTargets(t *testing.T) {
if err := validateResetPath(""); err == nil {
t.Fatal("empty reset path was accepted")
}
if err := validateResetPath(string(filepath.Separator)); err == nil {
t.Fatal("filesystem root reset path was accepted")
}
if err := validateResetPath(filepath.Join(string(filepath.Separator), "var")); err == nil {
t.Fatal("broad top-level reset path was accepted")
}
if err := validateResetPath(os.TempDir()); err == nil {
t.Fatal("system temporary directory reset path was accepted")
}
if cwd, err := os.Getwd(); err == nil {
if err := validateResetPath(cwd); err == nil {
t.Fatal("current working directory reset path was accepted")
}
}
if homeDir, err := os.UserHomeDir(); err == nil {
if err := validateResetPath(homeDir); err == nil {
t.Fatal("user home reset path was accepted")
}
}
if err := validateResetPath(t.TempDir()); err != nil {
t.Fatalf("scoped reset path was rejected: %v", err)
}
}
func TestRemoveResetPathsValidatesAllTargetsBeforeDeletion(t *testing.T) {
root := t.TempDir()
configDir := filepath.Join(root, "config")
if err := os.Mkdir(configDir, 0o700); err != nil {
t.Fatalf("create config directory: %v", err)
}
paths := []resetPath{
{name: "config", path: configDir},
{name: "unsafe", path: string(filepath.Separator)},
}
if err := removeResetPaths(paths); err == nil {
t.Fatal("unsafe reset target was accepted")
}
if _, err := os.Stat(configDir); err != nil {
t.Fatalf("config directory was removed before validation completed: %v", err)
}
}
func TestCleanupInlineSecretsDirsRemovesPrivateDirectory(t *testing.T) {
secretsDir := filepath.Join(t.TempDir(), "nuclei-secrets-run")
if err := os.Mkdir(secretsDir, 0o700); err != nil {
t.Fatalf("create inline secrets directory: %v", err)
}
secretsPath := filepath.Join(secretsDir, "inline-secrets.yaml")
if err := os.WriteFile(secretsPath, []byte("static: []\n"), 0o600); err != nil {
t.Fatalf("write inline secrets file: %v", err)
}
inlineSecretsTempMu.Lock()
previous := inlineSecretsTempDir
inlineSecretsTempDir = secretsDir
inlineSecretsTempMu.Unlock()
t.Cleanup(func() {
inlineSecretsTempMu.Lock()
inlineSecretsTempDir = previous
inlineSecretsTempMu.Unlock()
})
cleanupInlineSecretsDirs()
if _, err := os.Stat(secretsDir); !os.IsNotExist(err) {
t.Fatalf("inline secrets directory still exists: %v", err)
}
}
func TestCleanupInlineSecretsDirsIsConcurrentSafe(t *testing.T) {
root := t.TempDir()
dir := filepath.Join(root, "nuclei-secrets")
if err := os.Mkdir(dir, 0o700); err != nil {
t.Fatalf("create inline secrets directory: %v", err)
}
inlineSecretsTempMu.Lock()
previous := inlineSecretsTempDir
inlineSecretsTempDir = dir
inlineSecretsTempMu.Unlock()
t.Cleanup(func() {
inlineSecretsTempMu.Lock()
inlineSecretsTempDir = previous
inlineSecretsTempMu.Unlock()
})
start := make(chan struct{})
var wg sync.WaitGroup
for range 8 {
wg.Add(1)
go func() {
defer wg.Done()
<-start
cleanupInlineSecretsDirs()
}()
}
close(start)
wg.Wait()
if _, err := os.Stat(dir); !os.IsNotExist(err) {
t.Fatalf("inline secrets directory still exists: %s: %v", dir, err)
}
}
func TestReadConfigFatalCleansInlineSecretsDirectory(t *testing.T) {
if os.Getenv(inlineSecretsFatalChildEnv) == "1" {
options = &types.Options{Logger: gologger.DefaultLogger}
os.Args = []string{
os.Args[0],
"-profile", os.Getenv("NUCLEI_TEST_PROFILE"),
"-secret-file", os.Getenv("NUCLEI_TEST_MISSING_SECRET"),
"-disable-update-check",
}
readConfig()
os.Exit(0)
}
root := t.TempDir()
profilePath := filepath.Join(root, "profile.yaml")
if err := os.WriteFile(profilePath, []byte("secrets:\n static: []\n"), 0o600); err != nil {
t.Fatalf("write profile: %v", err)
}
missingSecret := filepath.Join(root, "missing-secrets.yaml")
tempRoot := filepath.Join(root, "tmp")
if err := os.Mkdir(tempRoot, 0o700); err != nil {
t.Fatalf("create temporary root: %v", err)
}
cmd := exec.Command(os.Args[0], "-test.run=^TestReadConfigFatalCleansInlineSecretsDirectory$", "-test.count=1")
cmd.Env = isolatedNucleiStorageEnv(root, tempRoot,
inlineSecretsFatalChildEnv+"=1",
"NUCLEI_TEST_PROFILE="+profilePath,
"NUCLEI_TEST_MISSING_SECRET="+missingSecret,
)
output, err := cmd.CombinedOutput()
if err == nil {
t.Fatal("child process did not exit on the missing secrets file")
}
if !bytes.Contains(output, []byte("Secrets file")) || !bytes.Contains(output, []byte("does not exist")) {
t.Fatalf("child did not reach the missing secrets file failure:\n%s", output)
}
matches, err := filepath.Glob(filepath.Join(tempRoot, "nuclei-secrets-*"))
if err != nil {
t.Fatalf("list inline secrets directories: %v", err)
}
if len(matches) != 0 {
t.Fatalf("inline secrets directories remain after fatal exit: %v", matches)
}
}
func isolatedNucleiStorageEnv(root, tempRoot string, extra ...string) []string {
env := append(os.Environ(),
"TMPDIR="+tempRoot,
"TMP="+tempRoot,
"TEMP="+tempRoot,
"XDG_CONFIG_HOME="+filepath.Join(root, "config"),
"XDG_STATE_HOME="+filepath.Join(root, "state"),
"XDG_CACHE_HOME="+filepath.Join(root, "cache"),
"XDG_DATA_HOME="+filepath.Join(root, "data"),
"NUCLEI_CONFIG_DIR="+filepath.Join(root, "config", "nuclei"),
)
return append(env, extra...)
}
+25 -27
View File
@@ -16,13 +16,13 @@ require (
github.com/miekg/dns v1.1.72
github.com/olekukonko/tablewriter v1.0.8
github.com/pkg/errors v0.9.1
github.com/projectdiscovery/clistats v0.1.4
github.com/projectdiscovery/fastdialer v0.5.14
github.com/projectdiscovery/hmap v0.0.101
github.com/projectdiscovery/clistats v0.1.6
github.com/projectdiscovery/fastdialer v0.5.18
github.com/projectdiscovery/hmap v0.0.102
github.com/projectdiscovery/interactsh v1.3.1
github.com/projectdiscovery/rawhttp v0.1.91
github.com/projectdiscovery/retryabledns v1.0.115
github.com/projectdiscovery/retryablehttp-go v1.3.21
github.com/projectdiscovery/retryabledns v1.0.116
github.com/projectdiscovery/retryablehttp-go v1.3.24
github.com/projectdiscovery/yamldoc-go v1.0.6
github.com/remeh/sizedwaitgroup v1.0.0
github.com/rs/xid v1.6.0
@@ -32,9 +32,9 @@ require (
github.com/syndtr/goleveldb v1.0.0
github.com/weppos/publicsuffix-go v0.50.3 // indirect
go.uber.org/multierr v1.11.0
golang.org/x/net v0.56.0
golang.org/x/net v0.57.0
golang.org/x/oauth2 v0.36.0
golang.org/x/text v0.39.0
golang.org/x/text v0.40.0
)
require (
@@ -49,6 +49,7 @@ require (
github.com/Mzack9999/go-rsync v0.0.0-20250821180103-81ffa574ef4d
github.com/Mzack9999/goimpacket v0.0.0-20260422121140-7085336a0415
github.com/RedTeamPentesting/adauth v0.5.4-0.20260511073005-3d18e8a5a687
github.com/adrg/xdg v0.5.3
github.com/alexsnet/go-vnc v0.1.0
github.com/alitto/pond v1.9.2
github.com/antchfx/xmlquery v1.4.4
@@ -67,7 +68,7 @@ require (
github.com/fatih/structs v1.1.0
github.com/fullstorydev/grpcurl v1.9.3
github.com/getkin/kin-openapi v0.144.0
github.com/go-git/go-git/v5 v5.19.1
github.com/go-git/go-git/v5 v5.19.2
github.com/go-ldap/ldap/v3 v3.4.12
github.com/go-pdf/fpdf v0.9.0
github.com/go-pg/pg/v10 v10.15.0
@@ -91,40 +92,40 @@ require (
github.com/oiweiwei/go-msrpc v1.2.12
github.com/ory/dockertest/v3 v3.12.0
github.com/praetorian-inc/fingerprintx v1.1.15
github.com/projectdiscovery/dsl v0.8.20
github.com/projectdiscovery/dsl v0.8.22
github.com/projectdiscovery/fasttemplate v0.0.2
github.com/projectdiscovery/gcache v0.0.0-20241015120333-12546c6e3f4c
github.com/projectdiscovery/goflags v0.1.75
github.com/projectdiscovery/goflags v0.2.0
github.com/projectdiscovery/goja v0.0.0-20260618133720-acb73e419534
github.com/projectdiscovery/goja_nodejs v0.0.0-20260618132410-8519f75f703d
github.com/projectdiscovery/gologger v1.1.71
github.com/projectdiscovery/gologger v1.1.72
github.com/projectdiscovery/gostruct v0.0.2
github.com/projectdiscovery/govaluate v0.0.0-20260615100919-5ee2581bbf7e
github.com/projectdiscovery/gozero v0.1.1-0.20260530071156-fa1dad563d76
github.com/projectdiscovery/httpx v1.10.0
github.com/projectdiscovery/mapcidr v1.1.97
github.com/projectdiscovery/n3iwf v0.0.0-20230523120440-b8cd232ff1f5
github.com/projectdiscovery/networkpolicy v0.1.44
github.com/projectdiscovery/networkpolicy v0.1.48
github.com/projectdiscovery/ratelimit v0.0.88
github.com/projectdiscovery/rdap v0.9.0
github.com/projectdiscovery/sarif v0.1.0
github.com/projectdiscovery/tlsx v1.2.2
github.com/projectdiscovery/tlsx v1.3.2
github.com/projectdiscovery/uncover v1.2.1
github.com/projectdiscovery/useragent v0.0.108
github.com/projectdiscovery/utils v0.11.1
github.com/projectdiscovery/wappalyzergo v0.2.91
github.com/projectdiscovery/useragent v0.0.109
github.com/projectdiscovery/utils v0.11.2
github.com/projectdiscovery/wappalyzergo v0.2.95
github.com/redis/go-redis/v9 v9.11.0
github.com/rs/zerolog v1.34.0
github.com/shurcooL/graphql v0.0.0-20230722043721-ed46e5a46466
github.com/sijms/go-ora/v2 v2.9.0
github.com/stretchr/testify v1.11.1
github.com/stretchr/testify v1.12.1
github.com/tarunKoyalwar/goleak v0.0.0-20240429141123-0efa90dbdcf9
github.com/yassinebenaid/godump v0.11.1
github.com/zmap/zgrab2 v0.1.8
gitlab.com/gitlab-org/api/client-go v1.9.1
go.mongodb.org/mongo-driver v1.17.9
golang.org/x/sync v0.21.0
golang.org/x/term v0.44.0
golang.org/x/sync v0.22.0
golang.org/x/term v0.45.0
google.golang.org/grpc v1.82.1
gopkg.in/yaml.v3 v3.0.1
moul.io/http2curl v1.0.0
@@ -204,7 +205,6 @@ require (
github.com/containerd/errdefs v1.0.0 // indirect
github.com/containerd/errdefs/pkg v0.3.0 // indirect
github.com/cyphar/filepath-securejoin v0.6.1 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/davidmz/go-pageant v1.0.2 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/distribution/reference v0.6.0 // indirect
@@ -322,11 +322,10 @@ require (
github.com/pjbgf/sha1cd v0.6.0 // indirect
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect
github.com/projectdiscovery/asnmap v1.1.1 // indirect
github.com/projectdiscovery/blackrock v0.0.1 // indirect
github.com/projectdiscovery/cdncheck v1.2.46 // indirect
github.com/projectdiscovery/blackrock v0.0.2 // indirect
github.com/projectdiscovery/cdncheck v1.2.50 // indirect
github.com/projectdiscovery/freeport v0.0.7 // indirect
github.com/projectdiscovery/ldapserver v1.0.2-0.20240219154113-dcc758ebc0cb // indirect
github.com/projectdiscovery/machineid v0.0.0-20250715113114-c77eb3567582 // indirect
@@ -342,7 +341,6 @@ require (
github.com/spf13/afero v1.15.0 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/spiffe/go-spiffe/v2 v2.6.0 // indirect
github.com/stretchr/objx v0.5.3 // indirect
github.com/tidwall/btree v1.8.1 // indirect
github.com/tidwall/buntdb v1.3.2 // indirect
github.com/tidwall/gjson v1.18.0 // indirect
@@ -386,7 +384,7 @@ require (
go.opentelemetry.io/otel/metric v1.44.0 // indirect
go.opentelemetry.io/otel/trace v1.44.0 // indirect
go.uber.org/zap/exp v0.3.0 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect
go4.org v0.0.0-20230225012048-214862532bf5 // indirect
golang.org/x/arch v0.3.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260414002931-afd174a4e478 // indirect
@@ -413,10 +411,10 @@ require (
go.etcd.io/bbolt v1.4.3 // indirect
go.uber.org/zap v1.27.0 // indirect
goftp.io/server/v2 v2.0.1 // indirect
golang.org/x/crypto v0.53.0 // indirect
golang.org/x/crypto v0.54.0 // indirect
golang.org/x/exp v0.0.0-20260527015227-08cc5374adb3
golang.org/x/mod v0.37.0 // indirect
golang.org/x/sys v0.46.0 // indirect
golang.org/x/sys v0.47.0 // indirect
golang.org/x/time v0.14.0 // indirect
golang.org/x/tools v0.47.0
google.golang.org/protobuf v1.36.11
+50 -50
View File
@@ -106,6 +106,8 @@ github.com/STARRY-S/zip v0.2.3 h1:luE4dMvRPDOWQdeDdUxUoZkzUIpTccdKdhHHsQJ1fm4=
github.com/STARRY-S/zip v0.2.3/go.mod h1:lqJ9JdeRipyOQJrYSOtpNAiaesFO6zVDsE8GIGFaoSk=
github.com/VividCortex/ewma v1.2.0 h1:f58SaIzcDXrSy3kWaHNvuJgJ3Nmz59Zji6XoJR/q1ow=
github.com/VividCortex/ewma v1.2.0/go.mod h1:nz4BbCtbLyFDeC9SUHbtcT5644juEuWfUAUnGx7j5l4=
github.com/adrg/xdg v0.5.3 h1:xRnxJXne7+oWDatRhR1JLnvuccuIeCoBu2rtuLqQB78=
github.com/adrg/xdg v0.5.3/go.mod h1:nlTsY+NNiCBGCK2tpm09vRqfVzrc2fLmXGpBLF0zlTQ=
github.com/akrylysov/pogreb v0.10.2 h1:e6PxmeyEhWyi2AKOBIJzAEi4HkiC+lKyCocRGlnDi78=
github.com/akrylysov/pogreb v0.10.2/go.mod h1:pNs6QmpQ1UlTJKDezuRWmaqkgUE2TuU0YTWyqJZ7+lI=
github.com/alecthomas/assert/v2 v2.11.0 h1:2Q9r3ki8+JYXvGsDyBXwH3LcJ+WK5D0gc5E8vS6K3D0=
@@ -393,8 +395,8 @@ github.com/go-git/go-billy/v5 v5.9.0 h1:jItGXszUDRtR/AlferWPTMN4j38BQ88XnXKbilmm
github.com/go-git/go-billy/v5 v5.9.0/go.mod h1:jCnQMLj9eUgGU7+ludSTYoZL/GGmii14RxKFj7ROgHw=
github.com/go-git/go-git-fixtures/v4 v4.3.2-0.20231010084843-55a94097c399 h1:eMje31YglSBqCdIqdhKBW8lokaMrL3uTkpGYlE2OOT4=
github.com/go-git/go-git-fixtures/v4 v4.3.2-0.20231010084843-55a94097c399/go.mod h1:1OCfN199q1Jm3HZlxleg+Dw/mwps2Wbk9frAWm+4FII=
github.com/go-git/go-git/v5 v5.19.1 h1:nX27AnaU43/K5bKktKwgBmR9lawoYVe1Ckg0rgzzN00=
github.com/go-git/go-git/v5 v5.19.1/go.mod h1:Pb1v0c7/g8aGQJwx9Us09W85yGoyvSwuhEGMH7zjDKQ=
github.com/go-git/go-git/v5 v5.19.2 h1:wkfn7vOlUBu8ivAWKBWisTiwJK4jYHzTF8Ndv1LyGqY=
github.com/go-git/go-git/v5 v5.19.2/go.mod h1:QqCBE1EFN5ddFmrliLQ3/ntRCUjZU3EJuwuB/jWEHjk=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
@@ -825,46 +827,44 @@ github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINE
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 h1:GFCKgmp0tecUJ0sJuv4pzYCqS9+RGSn52M3FUwPs+uo=
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10/go.mod h1:t/avpk3KcrXxUnYOhZhMXJlSEyie6gQbtLq5NM3loB8=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 h1:o4JXh1EVt9k/+g42oCprj/FisM4qX9L3sZB3upGN2ZU=
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE=
github.com/praetorian-inc/fingerprintx v1.1.15 h1:CVIxrIQARbmdk5h8E9tIJZbvFoY2sGLLG9rpFVfQqpA=
github.com/praetorian-inc/fingerprintx v1.1.15/go.mod h1:hqRroITBwKpP8BOGF+n/A+qv9wSF7OSVinmu5NCyOUI=
github.com/projectdiscovery/asnmap v1.1.1 h1:ImJiKIaACOT7HPx4Pabb5dksolzaFYsD1kID2iwsDqI=
github.com/projectdiscovery/asnmap v1.1.1/go.mod h1:QT7jt9nQanj+Ucjr9BqGr1Q2veCCKSAVyUzLXfEcQ60=
github.com/projectdiscovery/blackrock v0.0.1 h1:lHQqhaaEFjgf5WkuItbpeCZv2DUIE45k0VbGJyft6LQ=
github.com/projectdiscovery/blackrock v0.0.1/go.mod h1:ANUtjDfaVrqB453bzToU+YB4cUbvBRpLvEwoWIwlTss=
github.com/projectdiscovery/cdncheck v1.2.46 h1:ELtTCaT5dLigd0FGmw6z+JuIpbFAr422hbgAFXMmKvY=
github.com/projectdiscovery/cdncheck v1.2.46/go.mod h1:9oE9KKxCSHNvUf0UaMeqqUwWpC38FkNaTll0ScIBT3w=
github.com/projectdiscovery/clistats v0.1.4 h1:kDnXoNxIdOvQElOF7k2Mt6XosGa5GbMKPtRXdPHMVzU=
github.com/projectdiscovery/clistats v0.1.4/go.mod h1:hjJYNcUubk9T3cuFvA+JkLhZGjzYW50fkC48dqUAtbU=
github.com/projectdiscovery/dsl v0.8.20 h1:CxWcKuoHFpOSS1kzqnbJuK5No/6qoRG8IzNDMnZ6c/M=
github.com/projectdiscovery/dsl v0.8.20/go.mod h1:e1oHi7mxAxF+UhBhD5gOk90Ga6LQqvFea2voMO1E5D0=
github.com/projectdiscovery/fastdialer v0.5.14 h1:83hkWNnbWq8IxDM65h1CCmHf/GoVj1fpwbR+wACiBhA=
github.com/projectdiscovery/fastdialer v0.5.14/go.mod h1:+QaDuVAsvPk4VPSk9aVQCdZzHVbtI9+2BeeFhQUyebo=
github.com/projectdiscovery/blackrock v0.0.2 h1:mxXdu0uM8P2L2Qi210COlU8QiICPFW/Rxk5QUhlPO2k=
github.com/projectdiscovery/blackrock v0.0.2/go.mod h1:ANUtjDfaVrqB453bzToU+YB4cUbvBRpLvEwoWIwlTss=
github.com/projectdiscovery/cdncheck v1.2.50 h1:KZBYT4jKNvlmlDKJqGMJ7r6XtwAim82oEZLaI+NAWWk=
github.com/projectdiscovery/cdncheck v1.2.50/go.mod h1:9oE9KKxCSHNvUf0UaMeqqUwWpC38FkNaTll0ScIBT3w=
github.com/projectdiscovery/clistats v0.1.6 h1:e2Hc7cVWKJ4sCbyQ7th5zuBvgqsZcLwRq1qY0vmQXY4=
github.com/projectdiscovery/clistats v0.1.6/go.mod h1:RxRGxhMzkT344VTTvZWIrojurUXdQQRN1yfwgg/88I8=
github.com/projectdiscovery/dsl v0.8.22 h1:ZfPunLaaYUJTBffE1ShuAAV9hkSH2thvlszK48FlZS0=
github.com/projectdiscovery/dsl v0.8.22/go.mod h1:fr8WMU381As89e4Z+SQ8tYQ1NudniWAS6NLPHrhe+Vs=
github.com/projectdiscovery/fastdialer v0.5.18 h1:bBGJdni/xruoUISBug86KwRpuewLg1lEUfQLTg3p0G0=
github.com/projectdiscovery/fastdialer v0.5.18/go.mod h1:fH4CDmC7Dk7Shm4v3Rsql7ql3OhlKeZfOdAlc/muP1I=
github.com/projectdiscovery/fasttemplate v0.0.2 h1:h2cISk5xDhlJEinlBQS6RRx0vOlOirB2y3Yu4PJzpiA=
github.com/projectdiscovery/fasttemplate v0.0.2/go.mod h1:XYWWVMxnItd+r0GbjA1GCsUopMw1/XusuQxdyAIHMCw=
github.com/projectdiscovery/freeport v0.0.7 h1:Q6uXo/j8SaV/GlAHkEYQi8WQoPXyJWxyspx+aFmz9Qk=
github.com/projectdiscovery/freeport v0.0.7/go.mod h1:cOhWKvNBe9xM6dFJ3RrrLvJ5vXx2NQ36SecuwjenV2k=
github.com/projectdiscovery/gcache v0.0.0-20241015120333-12546c6e3f4c h1:s+lLAlrOrgwlPZQ9DFqNw+kia2nteKnJZ2Ek313yoUc=
github.com/projectdiscovery/gcache v0.0.0-20241015120333-12546c6e3f4c/go.mod h1:rN35/D3lVx2YDeENFFz06uj8j3XIqK1Ym9XcISF5fzg=
github.com/projectdiscovery/goflags v0.1.75 h1:njEBnyueQaFa2ptWxbyl9zX0OClNdlN2AzZveNHiBOs=
github.com/projectdiscovery/goflags v0.1.75/go.mod h1:7nAP1r2Dqgn/rwmOE3EWbZWUCEJKNIhVSBGpuzJAIns=
github.com/projectdiscovery/goflags v0.2.0 h1:zj16hnTmLmlnVYm02hfOrCY6oYTu/AQDBE99yZkS+s8=
github.com/projectdiscovery/goflags v0.2.0/go.mod h1:7nAP1r2Dqgn/rwmOE3EWbZWUCEJKNIhVSBGpuzJAIns=
github.com/projectdiscovery/goja v0.0.0-20260618133720-acb73e419534 h1:hYd1zQA/dxO2ASyQ6Re73TcJkW1LjLQvt4+86Hxefz8=
github.com/projectdiscovery/goja v0.0.0-20260618133720-acb73e419534/go.mod h1:SO0AP+uKfYeeoR6jyVH/PKRYJE/f5gJrPMAM00iGEMc=
github.com/projectdiscovery/goja_nodejs v0.0.0-20260618132410-8519f75f703d h1:fqqH9LHpN2WDz9QuxFrhKNxXSRtzk+Sa6jAhbB7tXcQ=
github.com/projectdiscovery/goja_nodejs v0.0.0-20260618132410-8519f75f703d/go.mod h1:Ezmbgdaw4EunGGBU4MQViLoGMJc37LA3ip55YV3KeRI=
github.com/projectdiscovery/gologger v1.1.71 h1:IYU4mw9viKdSzMTIGVpYuw1Gtg7QIHIStqAQgeNXcBQ=
github.com/projectdiscovery/gologger v1.1.71/go.mod h1:mJwODZcFDg70ihINpOvZevmBtgvpP8H9/l8Y+OPhZPY=
github.com/projectdiscovery/gologger v1.1.72 h1:PKk+aSx3jYOCBPi+FD+nTnhe+Vxx9u3K7Ega0GVlMCI=
github.com/projectdiscovery/gologger v1.1.72/go.mod h1:mJwODZcFDg70ihINpOvZevmBtgvpP8H9/l8Y+OPhZPY=
github.com/projectdiscovery/gostruct v0.0.2 h1:s8gP8ApugGM4go1pA+sVlPDXaWqNP5BBDDSv7VEdG1M=
github.com/projectdiscovery/gostruct v0.0.2/go.mod h1:H86peL4HKwMXcQQtEa6lmC8FuD9XFt6gkNR0B/Mu5PE=
github.com/projectdiscovery/govaluate v0.0.0-20260615100919-5ee2581bbf7e h1:vxzgQlz2Cy/YvizYDQx9OhucBcmBotfDhbQ4yCY2vfA=
github.com/projectdiscovery/govaluate v0.0.0-20260615100919-5ee2581bbf7e/go.mod h1:xH7bPwHxUlz1yx9UlVeTF+UVCUaKhTnZgaxHb5z362E=
github.com/projectdiscovery/gozero v0.1.1-0.20260530071156-fa1dad563d76 h1:AN70bbi6BBs7KpIM9w0LxygUN7uzT/oH+owDIQ+Fz/k=
github.com/projectdiscovery/gozero v0.1.1-0.20260530071156-fa1dad563d76/go.mod h1:cWHYnRXoYWHtTpOYyAp5laGYX8GH8ITUhgQaP8G/8FA=
github.com/projectdiscovery/hmap v0.0.101 h1:zXM6YtLmsn8Q0CUUw8QavhqWmiQYwaw+/U679Rr00pc=
github.com/projectdiscovery/hmap v0.0.101/go.mod h1:w6N9/a5H8kvyx53AhtPDUWe5Qq3D6NBDPA23glHpa/Q=
github.com/projectdiscovery/hmap v0.0.102 h1:ybPuf7UQvuOLiHLOUbVMvd+PSlItQU5iQpP1Erl8kaU=
github.com/projectdiscovery/hmap v0.0.102/go.mod h1:IZ92wXFmxYpdn0euZ7VFmypzOBNDwq8c3yRyIIol6XE=
github.com/projectdiscovery/httpx v1.10.0 h1:sZvfeJDeX4eJfB5Oh29My4gOHmLfvpCt2wCqhargJrE=
github.com/projectdiscovery/httpx v1.10.0/go.mod h1:gfdNnFznLLlAgAgSQlYjUeEVvYzKrNV0jO5Qyo5w+FI=
github.com/projectdiscovery/interactsh v1.3.1 h1:5HzeVGVCAX/cjTguJ+7ClOmML5r97Ty7op9s+/F7BiM=
@@ -877,32 +877,32 @@ github.com/projectdiscovery/mapcidr v1.1.97 h1:7FkxNNVXp+m1rIu5Nv/2SrF9k4+LwP8Qu
github.com/projectdiscovery/mapcidr v1.1.97/go.mod h1:9dgTJh1SP02gYZdpzMjm6vtYFkEHQHoTyaVNvaeJ7lA=
github.com/projectdiscovery/n3iwf v0.0.0-20230523120440-b8cd232ff1f5 h1:L/e8z8yw1pfT6bg35NiN7yd1XKtJap5Nk6lMwQ0RNi8=
github.com/projectdiscovery/n3iwf v0.0.0-20230523120440-b8cd232ff1f5/go.mod h1:pGW2ncnTxTxHtP9wzcIJAB+3/NMp6IiuQWd2NK7K+oc=
github.com/projectdiscovery/networkpolicy v0.1.44 h1:+fnpYNQVH9mYZY7VrsTqdL4YJGVZSA8XLAlRHBMl5QE=
github.com/projectdiscovery/networkpolicy v0.1.44/go.mod h1:q1KeQiHchXdElScEMWc5mShWNRNoJXI5koRnVaX6Qh8=
github.com/projectdiscovery/networkpolicy v0.1.48 h1:s/d0nEyfkHYVBeZI/bgRiOCS66KDxv3+s7Hb+AvepHY=
github.com/projectdiscovery/networkpolicy v0.1.48/go.mod h1:bb2ZkDnOeVuR9Ok8BdCdoBEz8S9Y9EFvFFcn5FTUV2U=
github.com/projectdiscovery/ratelimit v0.0.88 h1:AcurW9aLRzlEyPe9kSjnOpr3XzLMWTpiWAlW/w73ALU=
github.com/projectdiscovery/ratelimit v0.0.88/go.mod h1:CU1s+68UUG2mctSl2wi32/DHLJA6TMg+4rxgP59LfVk=
github.com/projectdiscovery/rawhttp v0.1.91 h1:6EIhZxCkBn27kJAl47FieCFCe4H2ZdbhoBVj0J6ZCVY=
github.com/projectdiscovery/rawhttp v0.1.91/go.mod h1:VZYAM25UI/wVB3URZ95ZaftgOnsbphxyAw/XnQRRz4Y=
github.com/projectdiscovery/rdap v0.9.0 h1:wPhHx5pQ2QI+WGhyNb2PjhTl0NtB39Nk7YFZ9cp8ZGA=
github.com/projectdiscovery/rdap v0.9.0/go.mod h1:zk4yrJFQ2Hy36Aqk+DvotYQxYAeALaCJ5ORySkff36Q=
github.com/projectdiscovery/retryabledns v1.0.115 h1:RKV63FNIznFHUoawg/1hs53pVH3wqPtFhwstCuxVSoA=
github.com/projectdiscovery/retryabledns v1.0.115/go.mod h1:+fEMWoPigw+M0lGNKY7AZ+g8FIgj+4sONjsinMmeL3k=
github.com/projectdiscovery/retryablehttp-go v1.3.21 h1:HytR9e8AfhIF/4xUo6szqg213a7pOq5tE3kpNZCyYwI=
github.com/projectdiscovery/retryablehttp-go v1.3.21/go.mod h1:0SCELxSKpkqHTy9pcZDZ2ry+5djp4geKpEURJ1uSB24=
github.com/projectdiscovery/retryabledns v1.0.116 h1:tK28PJ4tLO7bgM89RpIFjkfTPMSbRegwvg9mN/q6z78=
github.com/projectdiscovery/retryabledns v1.0.116/go.mod h1:r3vH0AKYtJ9fculcqRFrdVYe2ABcHwZeR4C6j3rqCbY=
github.com/projectdiscovery/retryablehttp-go v1.3.24 h1:bODdeUEka/U5SS9wfnhoAjBRg1wq/AJdwb9CUNIOR88=
github.com/projectdiscovery/retryablehttp-go v1.3.24/go.mod h1:OL5zZEBd59GXEMaDjEHW/bfGlFsV441ADz64hWajgxc=
github.com/projectdiscovery/sarif v0.1.0 h1:O541T+a448nSJsmIMnXXSOeDQEzpnCAYvRfe0eG5h74=
github.com/projectdiscovery/sarif v0.1.0/go.mod h1:LBC+reM3bkI3qIIhE0rZaINaYX6VG+En6u2hHa5mA7E=
github.com/projectdiscovery/stringsutil v0.0.2 h1:uzmw3IVLJSMW1kEg8eCStG/cGbYYZAja8BH3LqqJXMA=
github.com/projectdiscovery/stringsutil v0.0.2/go.mod h1:EJ3w6bC5fBYjVou6ryzodQq37D5c6qbAYQpGmAy+DC0=
github.com/projectdiscovery/tlsx v1.2.2 h1:Y96QBqeD2anpzEtBl4kqNbwzXh2TrzJuXfgiBLvK+SE=
github.com/projectdiscovery/tlsx v1.2.2/go.mod h1:ZJl9F1sSl0sdwE+lR0yuNHVX4Zx6tCSTqnNxnHCFZB4=
github.com/projectdiscovery/tlsx v1.3.2 h1:1Lh2ith79o8R+rOOSBZiMq8EiZ8j4D3T/8ENO+gAqXA=
github.com/projectdiscovery/tlsx v1.3.2/go.mod h1:2wgTGC/sourHvoR+RX8cAok0Sa8YK6QbEP3xo5SDzPI=
github.com/projectdiscovery/uncover v1.2.1 h1:8U46T/96CLT7BPoXBgkTvWqB06lOyeTSLvh5+UjzATE=
github.com/projectdiscovery/uncover v1.2.1/go.mod h1:0p8onrWxfpXQEYs90ZDzTSpu1107gWmodX1NWqu/+z4=
github.com/projectdiscovery/useragent v0.0.108 h1:fb+uLuFJvC+MHZjCtxQJxtvp1X6A8n98CUGPyFcg3NE=
github.com/projectdiscovery/useragent v0.0.108/go.mod h1:XdNRrlvtDmYfVL1Oybat4uMe+W6cLwsK9S18ond17CI=
github.com/projectdiscovery/utils v0.11.1 h1:PWj1KjIASxt8icxommH72C0TQqNOvGkcSODRkiq0SQw=
github.com/projectdiscovery/utils v0.11.1/go.mod h1:yktGrHGk2CTjNiccXovnvGrLHX9sV2bqz9nSnbA3V8M=
github.com/projectdiscovery/wappalyzergo v0.2.91 h1:pjbEOCJKmxfEG5xK3WnUNY4SuYjPoYUuAUxa3F1QN1Y=
github.com/projectdiscovery/wappalyzergo v0.2.91/go.mod h1:gMH0o5lBp65sKMwHx/tuUdOtW2RjodC6Ti+9QDsYMkY=
github.com/projectdiscovery/useragent v0.0.109 h1:b86BPdZvmgbJousa04jAugDRY4Y+rAvEamE0TFCPmeE=
github.com/projectdiscovery/useragent v0.0.109/go.mod h1:sRyBqDAcD31RKYrO8fyXGowLd36fNwY1g0MNW9mp7zk=
github.com/projectdiscovery/utils v0.11.2 h1:gfvXFBEHN4DgX8HBRzsoO1s2wH/2R+cH17cLpC865x4=
github.com/projectdiscovery/utils v0.11.2/go.mod h1:HMxhxLigsAr+M9Oa8n9Z0ROZcs8wAJHGsgR7UGb+oUE=
github.com/projectdiscovery/wappalyzergo v0.2.95 h1:XHjbQVrrxusezFq7ciWmtznPKWhpr/srWp7Yk2noxqg=
github.com/projectdiscovery/wappalyzergo v0.2.95/go.mod h1:E2p8L90ysTTUkU5FUOgGoCQ7ucEUgqJ/tvYgiGGAlEM=
github.com/projectdiscovery/yamldoc-go v1.0.6 h1:GCEdIRlQjDux28xTXKszM7n3jlMf152d5nqVpVoetas=
github.com/projectdiscovery/yamldoc-go v1.0.6/go.mod h1:R5lWrNzP+7Oyn77NDVPnBsxx2/FyQZBBkIAaSaCQFxw=
github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
@@ -1013,8 +1013,8 @@ github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o
github.com/stretchr/testify v1.8.3/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
github.com/syndtr/goleveldb v1.0.0 h1:fBdIW9lB4Iz0n9khmH8w27SJ3QEJ7+IgjPEwGSZiFdE=
github.com/syndtr/goleveldb v1.0.0/go.mod h1:ZVVdQEZoIme9iO1Ch2Jdy24qqXrMMOU6lpPAyBWyWuQ=
github.com/tarunKoyalwar/goleak v0.0.0-20240429141123-0efa90dbdcf9 h1:GXIyLuIJ5Qk46lI8WJ83qHBZKUI3zhmMmuoY9HICUIQ=
@@ -1190,8 +1190,8 @@ go.uber.org/zap v1.27.0 h1:aJMhYGrd5QSmlpLMr2MftRKl7t8J8PTZPA732ud/XR8=
go.uber.org/zap v1.27.0/go.mod h1:GB2qFLM7cTU87MWRP2mPIjqfIDnGu+VIO4V/SdhGo2E=
go.uber.org/zap/exp v0.3.0 h1:6JYzdifzYkGmTdRR59oYH+Ng7k49H9qVpWwNSsGJj3U=
go.uber.org/zap/exp v0.3.0/go.mod h1:5I384qq7XGxYyByIhHm6jg5CHkGY0nsTfbDLgDDlgJQ=
go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
go4.org v0.0.0-20230225012048-214862532bf5 h1:nifaUDeh+rPaBCMPMQHZmvJf+QdpLFnuQPwx+LxVmtc=
go4.org v0.0.0-20230225012048-214862532bf5/go.mod h1:F57wTi5Lrj6WLyswp5EYV1ncrEbFGHD4hhz6S1ZYeaU=
goftp.io/server/v2 v2.0.1 h1:H+9UbCX2N206ePDSVNCjBftOKOgil6kQ5RAQNx5hJwE=
@@ -1222,8 +1222,8 @@ golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDf
golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8=
golang.org/x/crypto v0.25.0/go.mod h1:T+wALwcMOSE0kXgUAnPAHqTLW+XHgcELELW8VaDgm/M=
golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto=
golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio=
golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw=
golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@@ -1312,8 +1312,8 @@ golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
golang.org/x/net v0.27.0/go.mod h1:dDi0PyhWNoiUOrAS8uXv/vnScO4wnHQO4mj9fn/RytE=
golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4=
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -1341,8 +1341,8 @@ golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y=
golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -1409,8 +1409,8 @@ golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.22.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw=
golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/telemetry v0.0.0-20240228155512-f48c80bd79b2/go.mod h1:TeRTkGYfJXctD9OcfyVLyj2J3IxLnKwHJR8f4D8a3YE=
golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
@@ -1424,8 +1424,8 @@ golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY=
golang.org/x/term v0.22.0/go.mod h1:F3qCibpT5AMpCRfhfT53vVJwhLtIVHhB9XDjfFvnMI4=
golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM=
golang.org/x/term v0.44.0 h1:0rLvDRCtNj0gZkyIXhCyOb2OAzEhLVqc4B+hrsBhrmc=
golang.org/x/term v0.44.0/go.mod h1:7ze4MdzUzLXpSAoFP1H0bOI9aXDqveSvatT5vKcFh2Y=
golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=
golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -1443,8 +1443,8 @@ golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.16.0/go.mod h1:GhwF1Be+LQoKShO3cGOHzqOgRrGaYc9AvblQOmPVHnI=
golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
golang.org/x/text v0.39.0 h1:UbZz4pLOvn600D6Oh6GGEI6VAmndrEBLv8/6BEXzyus=
golang.org/x/text v0.39.0/go.mod h1:3UwRclnC2g0TU9x8PZiyfOajCd1zaUNHF9cvqcQZ+ZM=
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
+198
View File
@@ -0,0 +1,198 @@
package configuration
import (
"errors"
"fmt"
"os"
"path"
"path/filepath"
"runtime"
"slices"
"github.com/adrg/xdg"
"github.com/projectdiscovery/goflags"
catalogconfig "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
)
// Selection contains the config and profile selected for one invocation.
type Selection struct {
Config string
Profile string
}
// Load applies automatic config files, one selected config file, and one
// selected profile in priority order. Explicit CLI flags retain precedence.
func Load(
cfg *catalogconfig.Config,
flagSet *goflags.FlagSet,
selected *Selection,
resolveProfile func(string) (string, error),
args ...string,
) ([]error, error) {
userConfigFile := cfg.GetFlagsConfigFilePath()
defaultUserConfigFile := filepath.Join(xdg.ConfigHome, catalogconfig.BinaryName, catalogconfig.CLIConfigFileName)
automaticFiles := automaticConfigFiles(xdg.ConfigDirs, runtime.GOOS, userConfigFile)
return loadConfigFiles(userConfigFile, defaultUserConfigFile, automaticFiles, flagSet, selected, resolveProfile, args...)
}
func loadConfigFiles(
userConfigFile string,
defaultUserConfigFile string,
automaticFiles []string,
flagSet *goflags.FlagSet,
selected *Selection,
resolveProfile func(string) (string, error),
args ...string,
) ([]error, error) {
var warnings []error
if err := prepareUserConfigFile(userConfigFile, defaultUserConfigFile); err != nil {
warnings = append(warnings, err)
}
flagSet.SetConfigFilePaths(automaticFiles...)
if err := flagSet.Parse(args...); err != nil {
warnings = append(warnings, fmt.Errorf("automatic configuration was ignored: %w", err))
}
if err := mergeSelectedFiles(flagSet, selected, resolveProfile); err != nil {
return warnings, err
}
return warnings, nil
}
func mergeSelectedFiles(
flagSet *goflags.FlagSet,
selected *Selection,
resolveProfile func(string) (string, error),
) error {
configFile := selected.Config
if configFile != "" {
if err := flagSet.MergeConfigFile(configFile); err != nil {
return fmt.Errorf("read config file %q: %w", configFile, err)
}
// A selected config may choose a profile, but it cannot redirect its
// completed config stage.
selected.Config = configFile
}
if selected.Profile == "" {
return nil
}
profileFile, err := resolveProfile(selected.Profile)
if err != nil {
return err
}
if err := flagSet.MergeConfigFile(profileFile); err != nil {
return fmt.Errorf("read template profile %q: %w", profileFile, err)
}
// A profile cannot redirect either completed file-selection stage.
selected.Config = configFile
selected.Profile = profileFile
return nil
}
func automaticConfigFiles(configDirs []string, goos, userConfigFile string) []string {
highestFirst := make([]string, 0, len(configDirs)+2)
highestFirst = append(highestFirst, userConfigFile)
if unixSystemFile := unixSystemConfigFilePath(goos); unixSystemFile != "" {
highestFirst = append(highestFirst, unixSystemFile)
}
for _, configDir := range configDirs {
if filepath.IsAbs(configDir) {
highestFirst = append(highestFirst, filepath.Join(configDir, catalogconfig.BinaryName, catalogconfig.CLIConfigFileName))
}
}
files := make([]string, 0, len(highestFirst))
seen := make(map[string]struct{}, len(highestFirst))
for _, file := range highestFirst {
if file == "" {
continue
}
cleanFile := filepath.Clean(file)
if _, exists := seen[cleanFile]; exists {
continue
}
seen[cleanFile] = struct{}{}
files = append(files, file)
}
slices.Reverse(files)
return files
}
func unixSystemConfigFilePath(goos string) string {
switch goos {
case "aix", "darwin", "dragonfly", "freebsd", "linux", "netbsd", "openbsd", "solaris":
return path.Join("/etc", catalogconfig.BinaryName, catalogconfig.CLIConfigFileName)
default:
return ""
}
}
func prepareUserConfigFile(configFile, defaultConfigFile string) error {
if filepath.Clean(configFile) == filepath.Clean(defaultConfigFile) {
return nil
}
if _, err := os.Stat(configFile); err == nil {
return nil
} else if !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("inspect custom config file %q: %w", configFile, err)
}
data, err := os.ReadFile(defaultConfigFile)
if errors.Is(err, os.ErrNotExist) {
return nil
}
if err != nil {
return fmt.Errorf("read default config file %q: %w", defaultConfigFile, err)
}
if err := os.MkdirAll(filepath.Dir(configFile), 0o700); err != nil {
return fmt.Errorf("create custom config directory %q: %w", filepath.Dir(configFile), err)
}
file, err := os.OpenFile(configFile, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o600)
if errors.Is(err, os.ErrExist) {
return nil
}
if err != nil {
return fmt.Errorf("create custom config file %q: %w", configFile, err)
}
written := false
defer func() {
_ = file.Close()
if !written {
_ = os.Remove(configFile)
}
}()
if _, err := file.Write(data); err != nil {
return fmt.Errorf("copy default config from %q to %q: %w", defaultConfigFile, configFile, err)
}
if err := file.Close(); err != nil {
return fmt.Errorf("close custom config file %q: %w", configFile, err)
}
written = true
return nil
}
+443
View File
@@ -0,0 +1,443 @@
package configuration
import (
"errors"
"fmt"
"os"
"path/filepath"
"slices"
"strings"
"testing"
"github.com/projectdiscovery/goflags"
catalogconfig "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
)
func TestConfigurationPrecedence(t *testing.T) {
dir := t.TempDir()
preferredSystemDir := filepath.Join(dir, "preferred-system")
fallbackSystemDir := filepath.Join(dir, "fallback-system")
preferredSystemPath := filepath.Join(preferredSystemDir, catalogconfig.BinaryName, catalogconfig.CLIConfigFileName)
fallbackSystemPath := filepath.Join(fallbackSystemDir, catalogconfig.BinaryName, catalogconfig.CLIConfigFileName)
unixSystemPath := filepath.Join(dir, "etc", catalogconfig.BinaryName, catalogconfig.CLIConfigFileName)
userPath := filepath.Join(dir, "user.yaml")
explicitPath := filepath.Join(dir, "explicit.yaml")
profilePath := filepath.Join(dir, "profile.yaml")
writeConfigFile(t, fallbackSystemPath, `
fallback-only: fallback
preferred-wins: fallback
local-wins: fallback
user-wins: fallback
explicit-wins: fallback
profile-wins: fallback
cli-wins: fallback
default-wins: fallback
config: ignored.yaml
profile: ignored.yaml
`)
writeConfigFile(t, preferredSystemPath, `
preferred-wins: preferred
local-wins: preferred
user-wins: preferred
explicit-wins: preferred
profile-wins: preferred
cli-wins: preferred
default-wins: preferred
`)
writeConfigFile(t, unixSystemPath, `
local-wins: local
user-wins: local
explicit-wins: local
profile-wins: local
cli-wins: local
default-wins: local
`)
writeConfigFile(t, userPath, fmt.Sprintf(`
user-wins: user
explicit-wins: user
profile-wins: user
cli-wins: user
default-wins: user
config: %q
profile: ignored-user.yaml
`, explicitPath))
writeConfigFile(t, explicitPath, fmt.Sprintf(`
explicit-wins: explicit
profile-wins: explicit
cli-wins: explicit
default-wins: explicit
profile: %q
`, profilePath))
writeConfigFile(t, profilePath, `
profile-wins: profile
cli-wins: profile
default-wins: profile
`)
var fallbackOnly, preferredWins, localWins, userWins, explicitWins, profileWins, cliWins, defaultWins string
var selected Selection
flagSet := goflags.NewFlagSet()
flagSet.StringVar(&fallbackOnly, "fallback-only", "built-in", "")
flagSet.StringVar(&preferredWins, "preferred-wins", "built-in", "")
flagSet.StringVar(&localWins, "local-wins", "built-in", "")
flagSet.StringVar(&userWins, "user-wins", "built-in", "")
flagSet.StringVar(&explicitWins, "explicit-wins", "built-in", "")
flagSet.StringVar(&profileWins, "profile-wins", "built-in", "")
flagSet.StringVar(&cliWins, "cli-wins", "built-in", "")
flagSet.StringVar(&defaultWins, "default-wins", "built-in", "")
flagSet.StringVar(&selected.Config, "config", "", "")
flagSet.StringVar(&selected.Profile, "profile", "", "")
automaticFiles := []string{fallbackSystemPath, preferredSystemPath, unixSystemPath, userPath}
automaticErr, err := loadTestFlags(flagSet, automaticFiles, &selected, existingProfilePath, "-cli-wins=cli", "-default-wins=built-in")
if err != nil {
t.Fatalf("load configuration failed: %v", err)
}
if automaticErr != nil {
t.Fatalf("automatic configuration failed: %v", automaticErr)
}
assertEqual(t, "fallback-only", fallbackOnly, "fallback")
assertEqual(t, "preferred-wins", preferredWins, "preferred")
assertEqual(t, "local-wins", localWins, "local")
assertEqual(t, "user-wins", userWins, "user")
assertEqual(t, "explicit-wins", explicitWins, "explicit")
assertEqual(t, "profile-wins", profileWins, "profile")
assertEqual(t, "cli-wins", cliWins, "cli")
assertEqual(t, "default-wins", defaultWins, "built-in")
assertEqual(t, "selected config", selected.Config, explicitPath)
assertEqual(t, "selected profile", selected.Profile, profilePath)
}
func TestAutomaticConfigFiles(t *testing.T) {
dir := t.TempDir()
preferredSystemDir := filepath.Join(dir, "preferred")
fallbackSystemDir := filepath.Join(dir, "fallback")
userConfigDir := filepath.Join(dir, "user")
preferredSystemFile := filepath.Join(preferredSystemDir, catalogconfig.BinaryName, catalogconfig.CLIConfigFileName)
fallbackSystemFile := filepath.Join(fallbackSystemDir, catalogconfig.BinaryName, catalogconfig.CLIConfigFileName)
userConfigFile := filepath.Join(userConfigDir, catalogconfig.BinaryName, catalogconfig.CLIConfigFileName)
unixSystemFile := unixSystemConfigFilePath("linux")
got := automaticConfigFiles([]string{preferredSystemDir, "relative-system", fallbackSystemDir}, "linux", userConfigFile)
want := []string{
fallbackSystemFile,
preferredSystemFile,
unixSystemFile,
userConfigFile,
}
if !slices.Equal(got, want) {
t.Fatalf("automatic config files = %v, want %v", got, want)
}
got = automaticConfigFiles([]string{preferredSystemDir, "/etc", userConfigDir}, "linux", userConfigFile)
want = []string{
preferredSystemFile,
unixSystemFile,
userConfigFile,
}
if !slices.Equal(got, want) {
t.Fatalf("automatic config files with duplicate system paths = %v, want %v", got, want)
}
got = automaticConfigFiles([]string{preferredSystemDir, "/etc"}, "linux", unixSystemFile)
want = []string{
preferredSystemFile,
unixSystemFile,
}
if !slices.Equal(got, want) {
t.Fatalf("automatic config files with Unix path used as user path = %v, want %v", got, want)
}
}
func TestHigherPriorityFilesReplaceCollections(t *testing.T) {
dir := t.TempDir()
systemPath := filepath.Join(dir, "system.yaml")
userPath := filepath.Join(dir, "user.yaml")
explicitPath := filepath.Join(dir, "explicit.yaml")
profilePath := filepath.Join(dir, "profile.yaml")
writeConfigFile(t, systemPath, "replace: [system-a, system-b]\nclear: [system]\nvar: [system=one]\n")
writeConfigFile(t, userPath, fmt.Sprintf("replace: [user]\nclear: [user]\nvar: [user=two]\nconfig: %q\n", explicitPath))
writeConfigFile(t, explicitPath, fmt.Sprintf("replace: [explicit-a, explicit-b]\nclear: [explicit]\nvar: [explicit=three]\nprofile: %q\n", profilePath))
writeConfigFile(t, profilePath, "replace: [profile]\nclear: []\nvar: [profile=four]\n")
var replace, clear goflags.StringSlice
var variables goflags.RuntimeMap
var selected Selection
flagSet := goflags.NewFlagSet()
flagSet.StringSliceVar(&replace, "replace", nil, "", goflags.StringSliceOptions)
flagSet.StringSliceVar(&clear, "clear", nil, "", goflags.StringSliceOptions)
flagSet.RuntimeMapVar(&variables, "var", nil, "")
flagSet.StringVar(&selected.Config, "config", "", "")
flagSet.StringVar(&selected.Profile, "profile", "", "")
automaticErr, err := loadTestFlags(flagSet, []string{systemPath, userPath}, &selected, existingProfilePath, "-config="+explicitPath)
if err != nil {
t.Fatalf("load configuration failed: %v", err)
}
if automaticErr != nil {
t.Fatalf("automatic configuration failed: %v", automaticErr)
}
if !slices.Equal(replace, []string{"profile"}) {
t.Fatalf("replace = %v, want [profile]", replace)
}
if len(clear) != 0 {
t.Fatalf("clear = %v, want an empty list", clear)
}
if got := variables.AsMap(); len(got) != 1 || got["profile"] != "four" {
t.Fatalf("var = %v, want only profile=four", got)
}
}
func TestParsingCreatesPrimaryUserConfig(t *testing.T) {
tests := []struct {
name string
createSystem bool
wantValue string
forbidInUser string
}{
{name: "missing system", wantValue: "built-in"},
{name: "system values are not materialized", createSystem: true, wantValue: "system", forbidInUser: "system"},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
dir := t.TempDir()
systemPath := filepath.Join(dir, "system.yaml")
userPath := filepath.Join(dir, "user", "yaml")
if test.createSystem {
writeConfigFile(t, systemPath, "value: system\n")
}
value := "built-in"
var selected Selection
var noop bool
flagSet := goflags.NewFlagSet()
flagSet.StringVar(&value, "value", "built-in", "value description")
flagSet.StringVar(&selected.Config, "config", "", "")
flagSet.StringVar(&selected.Profile, "profile", "", "")
flagSet.BoolVar(&noop, "noop", false, "")
automaticErr, err := loadTestFlags(flagSet, []string{systemPath, userPath}, &selected, existingProfilePath, "-noop=false")
if err != nil {
t.Fatalf("load configuration failed: %v", err)
}
if automaticErr != nil {
t.Fatalf("automatic configuration failed: %v", automaticErr)
}
assertEqual(t, "value", value, test.wantValue)
data, err := os.ReadFile(userPath)
if err != nil {
t.Fatalf("read generated user config: %v", err)
}
content := string(data)
if !strings.Contains(content, "#value: built-in") {
t.Fatalf("generated user config is not a commented built-in config:\n%s", content)
}
if test.forbidInUser != "" && strings.Contains(content, test.forbidInUser) {
t.Fatalf("generated user config materialized %q:\n%s", test.forbidInUser, content)
}
})
}
}
func TestSelectedConfigLoadsAfterMalformedAutomaticConfig(t *testing.T) {
dir := t.TempDir()
systemPath := filepath.Join(dir, "system.yaml")
userPath := filepath.Join(dir, "user.yaml")
explicitPath := filepath.Join(dir, "explicit.yaml")
writeConfigFile(t, systemPath, "value: system\nautomatic-only: system\n")
writeConfigFile(t, userPath, "value: [\n")
writeConfigFile(t, explicitPath, "value: explicit\n")
value := "built-in"
automaticOnly := "built-in"
var selected Selection
flagSet := goflags.NewFlagSet()
flagSet.StringVar(&value, "value", "built-in", "")
flagSet.StringVar(&automaticOnly, "automatic-only", "built-in", "")
flagSet.StringVar(&selected.Config, "config", "", "")
flagSet.StringVar(&selected.Profile, "profile", "", "")
automaticErr, err := loadTestFlags(flagSet, []string{systemPath, userPath}, &selected, existingProfilePath, "-config="+explicitPath)
if err != nil {
t.Fatalf("valid explicit config failed after malformed automatic config: %v", err)
}
if automaticErr == nil {
t.Fatal("expected malformed automatic configuration error")
}
assertEqual(t, "value", value, "explicit")
assertEqual(t, "automatic-only", automaticOnly, "built-in")
}
func TestDeferredUpdateConfiguration(t *testing.T) {
dir := t.TempDir()
userPath := filepath.Join(dir, "user.yaml")
updateDir := filepath.Join(dir, "templates")
writeConfigFile(t, userPath, fmt.Sprintf(`
disable-update-check: true
update: true
update-templates: true
update-template-dir: %q
`, updateDir))
var disableUpdateCheck, updateBinary, updateTemplates, noop bool
var selected Selection
var selectedUpdateDir string
flagSet := goflags.NewFlagSet()
flagSet.BoolVar(&disableUpdateCheck, "disable-update-check", false, "")
flagSet.BoolVar(&updateBinary, "update", false, "")
flagSet.BoolVar(&updateTemplates, "update-templates", false, "")
flagSet.StringVar(&selectedUpdateDir, "update-template-dir", "", "")
flagSet.StringVar(&selected.Config, "config", "", "")
flagSet.StringVar(&selected.Profile, "profile", "", "")
flagSet.BoolVar(&noop, "noop", false, "")
automaticErr, err := loadTestFlags(flagSet, []string{filepath.Join(dir, "missing-system.yaml"), userPath}, &selected, existingProfilePath, "-noop=false")
if err != nil {
t.Fatalf("load configuration failed: %v", err)
}
if automaticErr != nil {
t.Fatalf("automatic configuration failed: %v", automaticErr)
}
if !disableUpdateCheck || !updateBinary || !updateTemplates {
t.Fatalf("update values were not loaded: disable=%t update=%t templates=%t", disableUpdateCheck, updateBinary, updateTemplates)
}
assertEqual(t, "update template directory", selectedUpdateDir, updateDir)
testConfig := &catalogconfig.Config{}
if !testConfig.CanCheckForUpdates() {
t.Fatal("configuration loading applied disable-update-check too early")
}
if disableUpdateCheck {
testConfig.DisableUpdateCheck()
}
if testConfig.CanCheckForUpdates() {
t.Fatal("deferred disable-update-check was not applied")
}
}
func TestConfigFilesDoNotExecuteCallbacks(t *testing.T) {
dir := t.TempDir()
systemPath := filepath.Join(dir, "system.yaml")
userPath := filepath.Join(dir, "user.yaml")
explicitPath := filepath.Join(dir, "explicit.yaml")
profilePath := filepath.Join(dir, "profile.yaml")
callbackYAML := "reset: true\nversion: true\ntemplates-version: true\n"
writeConfigFile(t, systemPath, callbackYAML)
writeConfigFile(t, userPath, fmt.Sprintf("%sconfig: %q\n", callbackYAML, explicitPath))
writeConfigFile(t, explicitPath, fmt.Sprintf("%sprofile: %q\n", callbackYAML, profilePath))
writeConfigFile(t, profilePath, callbackYAML)
callbackCount := 0
var selected Selection
var noop bool
flagSet := goflags.NewFlagSet()
flagSet.CallbackVar(func() { callbackCount++ }, "reset", "")
flagSet.CallbackVar(func() { callbackCount++ }, "version", "")
flagSet.CallbackVar(func() { callbackCount++ }, "templates-version", "")
flagSet.StringVar(&selected.Config, "config", "", "")
flagSet.StringVar(&selected.Profile, "profile", "", "")
flagSet.BoolVar(&noop, "noop", false, "")
automaticErr, err := loadTestFlags(flagSet, []string{systemPath, userPath}, &selected, existingProfilePath, "-noop=false")
if err != nil {
t.Fatalf("load configuration failed: %v", err)
}
if automaticErr != nil {
t.Fatalf("automatic configuration failed: %v", automaticErr)
}
if callbackCount != 0 {
t.Fatalf("configuration executed %d callbacks", callbackCount)
}
flagSet = goflags.NewFlagSet()
flagSet.CallbackVar(func() { callbackCount++ }, "reset", "")
flagSet.StringVar(&selected.Config, "config", "", "")
flagSet.StringVar(&selected.Profile, "profile", "", "")
automaticErr, err = loadTestFlags(flagSet, []string{systemPath, userPath}, &selected, existingProfilePath, "-reset")
if err != nil {
t.Fatalf("load configuration with CLI callback failed: %v", err)
}
if automaticErr != nil {
t.Fatalf("automatic configuration with CLI callback failed: %v", automaticErr)
}
if callbackCount != 1 {
t.Fatalf("CLI callback count = %d, want 1", callbackCount)
}
}
func TestSelectedFilesCannotRedirectTheirOwnStage(t *testing.T) {
dir := t.TempDir()
userPath := filepath.Join(dir, "user.yaml")
configPath := filepath.Join(dir, "config.yaml")
otherConfigPath := filepath.Join(dir, "other-config.yaml")
profilePath := filepath.Join(dir, "profile.yaml")
otherProfilePath := filepath.Join(dir, "other-profile.yaml")
writeConfigFile(t, userPath, fmt.Sprintf("config: %q\n", configPath))
writeConfigFile(t, configPath, fmt.Sprintf("config: %q\nprofile: %q\n", otherConfigPath, profilePath))
writeConfigFile(t, otherConfigPath, "other-config-loaded: true\n")
writeConfigFile(t, profilePath, fmt.Sprintf("config: %q\nprofile: %q\n", otherConfigPath, otherProfilePath))
writeConfigFile(t, otherProfilePath, "other-profile-loaded: true\n")
var selected Selection
var otherConfigLoaded, otherProfileLoaded bool
flagSet := goflags.NewFlagSet()
flagSet.StringVar(&selected.Config, "config", "", "")
flagSet.StringVar(&selected.Profile, "profile", "", "")
flagSet.BoolVar(&otherConfigLoaded, "other-config-loaded", false, "")
flagSet.BoolVar(&otherProfileLoaded, "other-profile-loaded", false, "")
automaticErr, err := loadTestFlags(flagSet, []string{userPath}, &selected, existingProfilePath, "--")
if err != nil {
t.Fatalf("load configuration failed: %v", err)
}
if automaticErr != nil {
t.Fatalf("automatic configuration failed: %v", automaticErr)
}
assertEqual(t, "selected config", selected.Config, configPath)
assertEqual(t, "selected profile", selected.Profile, profilePath)
if otherConfigLoaded || otherProfileLoaded {
t.Fatalf("loading followed a redirected stage: config=%t profile=%t", otherConfigLoaded, otherProfileLoaded)
}
}
func loadTestFlags(
flagSet *goflags.FlagSet,
automaticFiles []string,
selected *Selection,
resolveProfile func(string) (string, error),
args ...string,
) (error, error) {
userConfigFile := automaticFiles[len(automaticFiles)-1]
warnings, err := loadConfigFiles(userConfigFile, userConfigFile, automaticFiles, flagSet, selected, resolveProfile, args...)
return errors.Join(warnings...), err
}
func existingProfilePath(path string) (string, error) {
if _, err := os.Stat(path); err != nil {
return "", err
}
return path, nil
}
func writeConfigFile(t *testing.T, path, content string) {
t.Helper()
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
t.Fatalf("create config directory: %v", err)
}
if err := os.WriteFile(path, []byte(strings.TrimPrefix(content, "\n")), 0o600); err != nil {
t.Fatalf("write config file %q: %v", path, err)
}
}
func assertEqual[T comparable](t *testing.T, name string, got, want T) {
t.Helper()
if got != want {
t.Fatalf("%s = %v, want %v", name, got, want)
}
}
+67
View File
@@ -0,0 +1,67 @@
package configuration
import (
"os"
"path/filepath"
"testing"
catalogconfig "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
)
func TestUnixSystemConfigFilePath(t *testing.T) {
for _, goos := range []string{"aix", "darwin", "dragonfly", "freebsd", "linux", "netbsd", "openbsd", "solaris"} {
if got, want := unixSystemConfigFilePath(goos), "/etc/nuclei/config.yaml"; got != want {
t.Fatalf("unix system config for %s = %q, want %q", goos, got, want)
}
}
for _, goos := range []string{"windows", "plan9", "js"} {
if got := unixSystemConfigFilePath(goos); got != "" {
t.Fatalf("system config for %s = %q, want empty", goos, got)
}
}
}
func TestPrepareUserConfigFile(t *testing.T) {
dir := t.TempDir()
defaultConfigFile := filepath.Join(dir, "default", catalogconfig.CLIConfigFileName)
customConfigFile := filepath.Join(dir, "custom", catalogconfig.CLIConfigFileName)
if err := os.MkdirAll(filepath.Dir(defaultConfigFile), 0o700); err != nil {
t.Fatalf("create default config directory: %v", err)
}
if err := os.WriteFile(defaultConfigFile, []byte("value: inherited\n"), 0o600); err != nil {
t.Fatalf("write default config: %v", err)
}
if err := prepareUserConfigFile(customConfigFile, defaultConfigFile); err != nil {
t.Fatalf("prepare user config file: %v", err)
}
data, err := os.ReadFile(customConfigFile)
if err != nil {
t.Fatalf("read inherited config: %v", err)
}
if got, want := string(data), "value: inherited\n"; got != want {
t.Fatalf("inherited config = %q, want %q", got, want)
}
if err := os.WriteFile(defaultConfigFile, []byte("value: changed\n"), 0o600); err != nil {
t.Fatalf("update default config: %v", err)
}
if err := prepareUserConfigFile(customConfigFile, defaultConfigFile); err != nil {
t.Fatalf("prepare existing user config: %v", err)
}
data, err = os.ReadFile(customConfigFile)
if err != nil {
t.Fatalf("read existing user config: %v", err)
}
if got, want := string(data), "value: inherited\n"; got != want {
t.Fatalf("existing user config = %q, want %q", got, want)
}
missingConfigFile := filepath.Join(dir, "missing-custom", catalogconfig.CLIConfigFileName)
if err := prepareUserConfigFile(missingConfigFile, filepath.Join(dir, "missing-default.yaml")); err != nil {
t.Fatalf("prepare config without default source: %v", err)
}
if _, err := os.Stat(missingConfigFile); !os.IsNotExist(err) {
t.Fatalf("missing default source created user config: %v", err)
}
}
+187
View File
@@ -0,0 +1,187 @@
package configuration
import (
"fmt"
"io/fs"
"os"
"path/filepath"
"strings"
catalogconfig "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/templates/extensions"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/yaml"
fileutil "github.com/projectdiscovery/utils/file"
)
// TemplatesDirectory returns the final template directory used while loading
// configuration and resolving profiles.
func TemplatesDirectory(options *types.Options, config *catalogconfig.Config) string {
if templatesDir, overridden := TemplatesDirectoryOverride(options); overridden {
return templatesDir
}
return config.GetTemplateDir()
}
// TemplatesDirectoryOverride returns an explicit flag/config or environment
// template directory.
func TemplatesDirectoryOverride(options *types.Options) (string, bool) {
templatesDir := options.NewTemplatesDirectory
if templatesDir == "" {
templatesDir = os.Getenv(catalogconfig.NucleiTemplatesDirEnv)
}
if templatesDir == "" {
return "", false
}
if absPath, err := filepath.Abs(templatesDir); err == nil {
return absPath, true
}
return templatesDir, true
}
// ResolveProfilePath resolves a profile path or profile ID against the active
// template root.
func ResolveProfilePath(profilePath, templatesDir string) (string, error) {
defaultProfilesPath := filepath.Join(templatesDir, "profiles")
if filepath.Ext(profilePath) == "" {
path, err := findProfilePathByID(profilePath, defaultProfilesPath)
if err != nil {
return "", err
}
if path == "" {
return "", fmt.Errorf("%q is not a profile ID or profile path", profilePath)
}
profilePath = path
}
if !filepath.IsAbs(profilePath) {
if filepath.Dir(profilePath) == "profiles" {
defaultProfilesPath = templatesDir
}
currentDir, err := os.Getwd()
if err == nil && fileutil.FileExists(filepath.Join(currentDir, profilePath)) {
profilePath = filepath.Join(currentDir, profilePath)
} else {
profilePath = filepath.Join(defaultProfilesPath, profilePath)
}
}
if !fileutil.FileExists(profilePath) {
return "", fmt.Errorf("given template profile file %q does not exist", profilePath)
}
return profilePath, nil
}
func findProfilePathByID(profileID, profilesDir string) (string, error) {
var profilePath string
err := filepath.WalkDir(profilesDir, func(path string, entry fs.DirEntry, err error) error {
if err != nil {
return err
}
if entry.IsDir() {
return nil
}
ext := filepath.Ext(path)
if ext != extensions.YAML && ext != extensions.YML {
return nil
}
if strings.TrimSuffix(filepath.Base(path), ext) == profileID {
profilePath = path
return fs.SkipAll
}
return nil
})
if err != nil {
return "", fmt.Errorf("search profile ID %q in %q: %w", profileID, profilesDir, err)
}
return profilePath, nil
}
// ApplyProfile processes profile-only inline targets and secrets after its
// flag values have been merged. It returns the private directory that owns a
// generated inline-secrets file.
func ApplyProfile(profilePath string, options *types.Options) (string, error) {
if profilePath == "" {
return "", nil
}
options.Targets = appendInlineTargets(options.Targets, options.InlineTargetsList)
if strings.Contains(options.TargetsFilePath, "\n") {
options.Targets = appendInlineTargets(options.Targets, options.TargetsFilePath)
options.TargetsFilePath = ""
}
return processInlineSecretsFromProfile(profilePath, options)
}
func appendInlineTargets(targets []string, input string) []string {
for target := range strings.SplitSeq(strings.TrimSpace(input), "\n") {
target = strings.TrimSpace(target)
if target != "" && !strings.HasPrefix(target, "#") {
targets = append(targets, target)
}
}
return targets
}
type profileSecrets struct {
Secrets any `yaml:"secrets"`
}
func processInlineSecretsFromProfile(profilePath string, options *types.Options) (string, error) {
data, err := os.ReadFile(profilePath)
if err != nil {
return "", fmt.Errorf("read profile file %q: %w", profilePath, err)
}
var profile profileSecrets
if err := yaml.Unmarshal(data, &profile); err != nil {
return "", fmt.Errorf("parse profile YAML %q: %w", profilePath, err)
}
if profile.Secrets == nil {
return "", nil
}
secretsData, err := yaml.Marshal(profile.Secrets)
if err != nil {
return "", fmt.Errorf("marshal inline profile secrets: %w", err)
}
tempDir, err := os.MkdirTemp("", "nuclei-secrets-*")
if err != nil {
return "", fmt.Errorf("create inline secrets directory: %w", err)
}
removeDir := true
defer func() {
if removeDir {
_ = os.RemoveAll(tempDir)
}
}()
tempPath := filepath.Join(tempDir, "inline-secrets.yaml")
if err := os.WriteFile(tempPath, secretsData, 0o600); err != nil {
return "", fmt.Errorf("write inline secrets file %q: %w", tempPath, err)
}
removeDir = false
options.SecretsFile = append(options.SecretsFile, tempPath)
return tempDir, nil
}
@@ -1,254 +1,356 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
)
func TestProcessInlineSecretsFromProfile(t *testing.T) {
t.Run("profile with inline secrets", func(t *testing.T) {
profileContent := `
secrets:
static:
- type: header
domains:
- api.example.com
headers:
- key: x-api-key
value: test-key-123
`
tmpFile, err := os.CreateTemp(t.TempDir(), "test-profile-*.yaml")
if err != nil {
t.Fatalf("could not create temp file: %v", err)
}
if _, err := tmpFile.WriteString(profileContent); err != nil {
t.Fatalf("could not write profile: %v", err)
}
_ = tmpFile.Close()
opts := &types.Options{}
tempPath, err := processInlineSecretsFromProfile(tmpFile.Name(), opts)
if err != nil {
t.Fatalf("processInlineSecretsFromProfile failed: %v", err)
}
defer func() {
_ = os.Remove(tempPath)
}()
if len(opts.SecretsFile) != 1 {
t.Fatalf("expected 1 secrets file, got %d", len(opts.SecretsFile))
}
secretsPath := opts.SecretsFile[0]
if !strings.Contains(secretsPath, "inline-secrets-") {
t.Errorf("secrets file path should contain 'inline-secrets-', got %s", secretsPath)
}
data, err := os.ReadFile(secretsPath)
if err != nil {
t.Fatalf("could not read generated secrets file: %v", err)
}
content := string(data)
if !strings.Contains(content, "x-api-key") {
t.Errorf("secrets file should contain header key, got:\n%s", content)
}
if !strings.Contains(content, "test-key-123") {
t.Errorf("secrets file should contain header value, got:\n%s", content)
}
if !strings.Contains(content, "api.example.com") {
t.Errorf("secrets file should contain domain, got:\n%s", content)
}
})
t.Run("profile without secrets", func(t *testing.T) {
profileContent := `
severity:
- critical
- high
`
tmpFile, err := os.CreateTemp(t.TempDir(), "test-profile-*.yaml")
if err != nil {
t.Fatalf("could not create temp file: %v", err)
}
if _, err := tmpFile.WriteString(profileContent); err != nil {
t.Fatalf("could not write profile: %v", err)
}
if err := tmpFile.Close(); err != nil {
t.Fatalf("could not close temp file: %v", err)
}
opts := &types.Options{}
tempPath, err := processInlineSecretsFromProfile(tmpFile.Name(), opts)
if err != nil {
t.Fatalf("processInlineSecretsFromProfile should not fail: %v", err)
}
if tempPath != "" {
t.Errorf("expected empty temp path for profile without secrets, got %s", tempPath)
}
if len(opts.SecretsFile) != 0 {
t.Errorf("expected 0 secrets files for profile without secrets, got %d", len(opts.SecretsFile))
}
})
t.Run("nonexistent profile", func(t *testing.T) {
opts := &types.Options{}
_, err := processInlineSecretsFromProfile(filepath.Join(t.TempDir(), "nonexistent.yaml"), opts)
if err == nil {
t.Error("expected error for nonexistent file")
}
})
}
func TestInlineTargetsParsing(t *testing.T) {
t.Run("multiline list key treated as inline targets", func(t *testing.T) {
opts := &types.Options{
TargetsFilePath: "example.com\ntest.com\nscanme.sh\n",
}
if strings.Contains(opts.TargetsFilePath, "\n") {
inlineTargets := strings.Split(strings.TrimSpace(opts.TargetsFilePath), "\n")
for _, target := range inlineTargets {
target = strings.TrimSpace(target)
if target != "" && !strings.HasPrefix(target, "#") {
opts.Targets = append(opts.Targets, target)
}
}
opts.TargetsFilePath = ""
}
if len(opts.Targets) != 3 {
t.Fatalf("expected 3 targets, got %d: %v", len(opts.Targets), opts.Targets)
}
if opts.Targets[0] != "example.com" {
t.Errorf("expected first target 'example.com', got '%s'", opts.Targets[0])
}
if opts.Targets[1] != "test.com" {
t.Errorf("expected second target 'test.com', got '%s'", opts.Targets[1])
}
if opts.Targets[2] != "scanme.sh" {
t.Errorf("expected third target 'scanme.sh', got '%s'", opts.Targets[2])
}
if opts.TargetsFilePath != "" {
t.Errorf("TargetsFilePath should be cleared, got '%s'", opts.TargetsFilePath)
}
})
t.Run("single line list key remains as file path", func(t *testing.T) {
opts := &types.Options{
TargetsFilePath: "/path/to/targets.txt",
}
if strings.Contains(opts.TargetsFilePath, "\n") {
t.Error("single-line path should not be treated as inline targets")
}
if opts.TargetsFilePath != "/path/to/targets.txt" {
t.Errorf("file path should remain unchanged, got '%s'", opts.TargetsFilePath)
}
})
t.Run("inline targets with comments and blank lines", func(t *testing.T) {
opts := &types.Options{
TargetsFilePath: "example.com\n# this is a comment\n\ntest.com\n",
}
if strings.Contains(opts.TargetsFilePath, "\n") {
inlineTargets := strings.Split(strings.TrimSpace(opts.TargetsFilePath), "\n")
for _, target := range inlineTargets {
target = strings.TrimSpace(target)
if target != "" && !strings.HasPrefix(target, "#") {
opts.Targets = append(opts.Targets, target)
}
}
opts.TargetsFilePath = ""
}
if len(opts.Targets) != 2 {
t.Fatalf("expected 2 targets (comments/blanks filtered), got %d: %v", len(opts.Targets), opts.Targets)
}
})
t.Run("targets-inline key", func(t *testing.T) {
opts := &types.Options{
InlineTargetsList: "host1.com\nhost2.com\nhost3.com",
}
if opts.InlineTargetsList != "" {
inlineTargets := strings.Split(strings.TrimSpace(opts.InlineTargetsList), "\n")
for _, target := range inlineTargets {
target = strings.TrimSpace(target)
if target != "" && !strings.HasPrefix(target, "#") {
opts.Targets = append(opts.Targets, target)
}
}
}
if len(opts.Targets) != 3 {
t.Fatalf("expected 3 targets from targets-inline, got %d", len(opts.Targets))
}
})
}
func TestInlineSecretsFileFormat(t *testing.T) {
// Verify the generated secrets file has the correct YAML structure
// that matches what authx.GetAuthDataFromYAML expects
profileContent := `
secrets:
static:
- type: header
domains:
- api.example.com
headers:
- key: Authorization
value: Bearer test-token
dynamic:
- template: oauth-flow.yaml
variables:
- key: username
value: testuser
type: cookie
domains:
- auth.example.com
`
tmpFile, err := os.CreateTemp(t.TempDir(), "test-secrets-*.yaml")
if err != nil {
t.Fatalf("could not create temp file: %v", err)
}
if _, err := tmpFile.WriteString(profileContent); err != nil {
t.Fatalf("could not write profile: %v", err)
}
if err := tmpFile.Close(); err != nil {
t.Fatalf("could not close temp file: %v", err)
}
opts := &types.Options{}
tempPath, err := processInlineSecretsFromProfile(tmpFile.Name(), opts)
if err != nil {
t.Fatalf("processInlineSecretsFromProfile failed: %v", err)
}
defer func() {
_ = os.Remove(tempPath)
}()
data, err := os.ReadFile(opts.SecretsFile[0])
if err != nil {
t.Fatalf("could not read secrets file: %v", err)
}
content := string(data)
// The secrets section should contain static and dynamic at root level,
// matching the Authx struct yaml tags
if !strings.Contains(content, "static:") {
t.Errorf("secrets file should have 'static:' key for Authx compatibility, got:\n%s", content)
}
if !strings.Contains(content, "dynamic:") {
t.Errorf("secrets file should have 'dynamic:' key for Authx compatibility, got:\n%s", content)
}
}
package configuration
import (
"os"
"path/filepath"
"strings"
"testing"
catalogconfig "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
)
func TestProcessInlineSecretsFromProfile(t *testing.T) {
t.Run("profile with inline secrets", func(t *testing.T) {
profileContent := `
secrets:
static:
- type: header
domains:
- api.example.com
headers:
- key: x-api-key
value: test-key-123
`
tmpFile, err := os.CreateTemp(t.TempDir(), "test-profile-*.yaml")
if err != nil {
t.Fatalf("could not create temp file: %v", err)
}
if _, err := tmpFile.WriteString(profileContent); err != nil {
t.Fatalf("could not write profile: %v", err)
}
_ = tmpFile.Close()
opts := &types.Options{}
tempDir, err := processInlineSecretsFromProfile(tmpFile.Name(), opts)
if err != nil {
t.Fatalf("processInlineSecretsFromProfile failed: %v", err)
}
defer func() {
_ = os.RemoveAll(tempDir)
}()
if len(opts.SecretsFile) != 1 {
t.Fatalf("expected 1 secrets file, got %d", len(opts.SecretsFile))
}
secretsPath := opts.SecretsFile[0]
if filepath.Base(secretsPath) != "inline-secrets.yaml" {
t.Errorf("secrets file name = %q, want inline-secrets.yaml", filepath.Base(secretsPath))
}
data, err := os.ReadFile(secretsPath)
if err != nil {
t.Fatalf("could not read generated secrets file: %v", err)
}
content := string(data)
if !strings.Contains(content, "x-api-key") {
t.Errorf("secrets file should contain header key, got:\n%s", content)
}
if !strings.Contains(content, "test-key-123") {
t.Errorf("secrets file should contain header value, got:\n%s", content)
}
if !strings.Contains(content, "api.example.com") {
t.Errorf("secrets file should contain domain, got:\n%s", content)
}
})
t.Run("profile without secrets", func(t *testing.T) {
profileContent := `
severity:
- critical
- high
`
tmpFile, err := os.CreateTemp(t.TempDir(), "test-profile-*.yaml")
if err != nil {
t.Fatalf("could not create temp file: %v", err)
}
if _, err := tmpFile.WriteString(profileContent); err != nil {
t.Fatalf("could not write profile: %v", err)
}
if err := tmpFile.Close(); err != nil {
t.Fatalf("could not close temp file: %v", err)
}
opts := &types.Options{}
tempDir, err := processInlineSecretsFromProfile(tmpFile.Name(), opts)
if err != nil {
t.Fatalf("processInlineSecretsFromProfile should not fail: %v", err)
}
if tempDir != "" {
t.Errorf("expected no temporary directory for profile without secrets, got %s", tempDir)
}
if len(opts.SecretsFile) != 0 {
t.Errorf("expected 0 secrets files for profile without secrets, got %d", len(opts.SecretsFile))
}
})
t.Run("nonexistent profile", func(t *testing.T) {
opts := &types.Options{}
_, err := processInlineSecretsFromProfile(filepath.Join(t.TempDir(), "nonexistent.yaml"), opts)
if err == nil {
t.Error("expected error for nonexistent file")
}
})
t.Run("does not follow a shared temporary directory symlink", func(t *testing.T) {
tempRoot := t.TempDir()
t.Setenv("TMPDIR", tempRoot)
t.Setenv("TMP", tempRoot)
t.Setenv("TEMP", tempRoot)
victimDir := filepath.Join(tempRoot, "victim")
if err := os.Mkdir(victimDir, 0o755); err != nil {
t.Fatalf("create victim directory: %v", err)
}
before, err := os.Stat(victimDir)
if err != nil {
t.Fatalf("stat victim directory before processing: %v", err)
}
sharedPath := filepath.Join(tempRoot, "nuclei-secrets")
if err := os.Symlink(victimDir, sharedPath); err != nil {
t.Skipf("create directory symlink: %v", err)
}
profilePath := filepath.Join(t.TempDir(), "profile.yaml")
writeConfigFile(t, profilePath, "secrets:\n static: []\n")
opts := &types.Options{}
tempDir, err := processInlineSecretsFromProfile(profilePath, opts)
if err != nil {
t.Fatalf("process inline secrets: %v", err)
}
t.Cleanup(func() { _ = os.RemoveAll(tempDir) })
if tempDir == sharedPath {
t.Fatalf("inline secrets used shared path %q", sharedPath)
}
if got := filepath.Dir(opts.SecretsFile[0]); got != tempDir {
t.Fatalf("inline secrets file directory = %q, want %q", got, tempDir)
}
after, err := os.Stat(victimDir)
if err != nil {
t.Fatalf("stat victim directory after processing: %v", err)
}
if before.Mode().Perm() != after.Mode().Perm() {
t.Fatalf("victim directory mode changed from %o to %o", before.Mode().Perm(), after.Mode().Perm())
}
})
}
func TestInlineTargetsParsing(t *testing.T) {
t.Run("multiline list key treated as inline targets", func(t *testing.T) {
opts := &types.Options{
TargetsFilePath: "example.com\ntest.com\nscanme.sh\n",
}
if strings.Contains(opts.TargetsFilePath, "\n") {
inlineTargets := strings.Split(strings.TrimSpace(opts.TargetsFilePath), "\n")
for _, target := range inlineTargets {
target = strings.TrimSpace(target)
if target != "" && !strings.HasPrefix(target, "#") {
opts.Targets = append(opts.Targets, target)
}
}
opts.TargetsFilePath = ""
}
if len(opts.Targets) != 3 {
t.Fatalf("expected 3 targets, got %d: %v", len(opts.Targets), opts.Targets)
}
if opts.Targets[0] != "example.com" {
t.Errorf("expected first target 'example.com', got '%s'", opts.Targets[0])
}
if opts.Targets[1] != "test.com" {
t.Errorf("expected second target 'test.com', got '%s'", opts.Targets[1])
}
if opts.Targets[2] != "scanme.sh" {
t.Errorf("expected third target 'scanme.sh', got '%s'", opts.Targets[2])
}
if opts.TargetsFilePath != "" {
t.Errorf("TargetsFilePath should be cleared, got '%s'", opts.TargetsFilePath)
}
})
t.Run("single line list key remains as file path", func(t *testing.T) {
opts := &types.Options{
TargetsFilePath: "/path/to/targets.txt",
}
if strings.Contains(opts.TargetsFilePath, "\n") {
t.Error("single-line path should not be treated as inline targets")
}
if opts.TargetsFilePath != "/path/to/targets.txt" {
t.Errorf("file path should remain unchanged, got '%s'", opts.TargetsFilePath)
}
})
t.Run("inline targets with comments and blank lines", func(t *testing.T) {
opts := &types.Options{
TargetsFilePath: "example.com\n# this is a comment\n\ntest.com\n",
}
if strings.Contains(opts.TargetsFilePath, "\n") {
inlineTargets := strings.Split(strings.TrimSpace(opts.TargetsFilePath), "\n")
for _, target := range inlineTargets {
target = strings.TrimSpace(target)
if target != "" && !strings.HasPrefix(target, "#") {
opts.Targets = append(opts.Targets, target)
}
}
opts.TargetsFilePath = ""
}
if len(opts.Targets) != 2 {
t.Fatalf("expected 2 targets (comments/blanks filtered), got %d: %v", len(opts.Targets), opts.Targets)
}
})
t.Run("targets-inline key", func(t *testing.T) {
opts := &types.Options{
InlineTargetsList: "host1.com\nhost2.com\nhost3.com",
}
if opts.InlineTargetsList != "" {
inlineTargets := strings.Split(strings.TrimSpace(opts.InlineTargetsList), "\n")
for _, target := range inlineTargets {
target = strings.TrimSpace(target)
if target != "" && !strings.HasPrefix(target, "#") {
opts.Targets = append(opts.Targets, target)
}
}
}
if len(opts.Targets) != 3 {
t.Fatalf("expected 3 targets from targets-inline, got %d", len(opts.Targets))
}
})
}
func TestInlineSecretsFileFormat(t *testing.T) {
// Verify the generated secrets file has the correct YAML structure
// that matches what authx.GetAuthDataFromYAML expects
profileContent := `
secrets:
static:
- type: header
domains:
- api.example.com
headers:
- key: Authorization
value: Bearer test-token
dynamic:
- template: oauth-flow.yaml
variables:
- key: username
value: testuser
type: cookie
domains:
- auth.example.com
`
tmpFile, err := os.CreateTemp(t.TempDir(), "test-secrets-*.yaml")
if err != nil {
t.Fatalf("could not create temp file: %v", err)
}
if _, err := tmpFile.WriteString(profileContent); err != nil {
t.Fatalf("could not write profile: %v", err)
}
if err := tmpFile.Close(); err != nil {
t.Fatalf("could not close temp file: %v", err)
}
opts := &types.Options{}
tempDir, err := processInlineSecretsFromProfile(tmpFile.Name(), opts)
if err != nil {
t.Fatalf("processInlineSecretsFromProfile failed: %v", err)
}
defer func() {
_ = os.RemoveAll(tempDir)
}()
data, err := os.ReadFile(opts.SecretsFile[0])
if err != nil {
t.Fatalf("could not read secrets file: %v", err)
}
content := string(data)
// The secrets section should contain static and dynamic at root level,
// matching the Authx struct yaml tags
if !strings.Contains(content, "static:") {
t.Errorf("secrets file should have 'static:' key for Authx compatibility, got:\n%s", content)
}
if !strings.Contains(content, "dynamic:") {
t.Errorf("secrets file should have 'dynamic:' key for Authx compatibility, got:\n%s", content)
}
}
func TestApplyProfileParsesInlineTargets(t *testing.T) {
profilePath := filepath.Join(t.TempDir(), "profile.yaml")
writeConfigFile(t, profilePath, "severity: [high]\n")
options := &types.Options{
InlineTargetsList: "inline.example\n# comment\n",
TargetsFilePath: "list.example\n\nsecond.example\n",
}
tempSecretsDir, err := ApplyProfile(profilePath, options)
if err != nil {
t.Fatalf("apply profile: %v", err)
}
if tempSecretsDir != "" {
t.Fatalf("temporary secrets directory = %q, want empty", tempSecretsDir)
}
want := []string{"inline.example", "list.example", "second.example"}
if strings.Join(options.Targets, ",") != strings.Join(want, ",") {
t.Fatalf("targets = %v, want %v", options.Targets, want)
}
if options.TargetsFilePath != "" {
t.Fatalf("targets file path = %q, want empty", options.TargetsFilePath)
}
}
func TestResolveProfilePathByID(t *testing.T) {
templatesDir := t.TempDir()
profilePath := filepath.Join(templatesDir, "profiles", "nested", "recommended.yaml")
writeConfigFile(t, profilePath, "severity: [high]\n")
got, err := ResolveProfilePath("recommended", templatesDir)
if err != nil {
t.Fatalf("resolve profile ID: %v", err)
}
if got != profilePath {
t.Fatalf("profile path = %q, want %q", got, profilePath)
}
}
func TestTemplatesDirectoryOverridePrecedence(t *testing.T) {
dir := t.TempDir()
envDir := filepath.Join(dir, "environment")
flagDir := filepath.Join(dir, "flag")
t.Setenv(catalogconfig.NucleiTemplatesDirEnv, envDir)
got, overridden := TemplatesDirectoryOverride(&types.Options{NewTemplatesDirectory: flagDir})
if !overridden {
t.Fatal("flag template directory was not reported as an override")
}
want, err := filepath.Abs(flagDir)
if err != nil {
t.Fatalf("resolve expected path: %v", err)
}
if got != want {
t.Fatalf("template directory = %q, want flag path %q", got, want)
}
}
+27 -10
View File
@@ -10,26 +10,43 @@ import (
pdcpauth "github.com/projectdiscovery/utils/auth/pdcp"
)
// AppendDirectoryInfo writes nuclei config/cache/PDCP directory paths.
func AppendDirectoryInfo(w io.Writer) {
_, _ = fmt.Fprintf(w, "Nuclei Config Directory: %s\n", config.DefaultConfig.GetConfigDir())
_, _ = fmt.Fprintf(w, "Nuclei Cache Directory: %s\n", config.DefaultConfig.GetCacheDir())
_, _ = fmt.Fprintf(w, "PDCP Directory: %s\n", pdcpauth.PDCPDir)
type directoryRecord struct {
label string
path string
}
// DirectoryInfo returns nuclei config/cache/PDCP directory paths as a string.
func directoryRecords() []directoryRecord {
return []directoryRecord{
{label: "Nuclei Config Directory", path: config.DefaultConfig.GetConfigDir()},
{label: "Nuclei State Directory", path: config.DefaultConfig.GetStateDir()},
{label: "Nuclei Cache Directory", path: config.DefaultConfig.GetCacheDir()},
{label: "PDCP Directory", path: pdcpauth.PDCPDir},
}
}
// AppendDirectoryInfo writes nuclei config/state/cache/PDCP directory paths.
func AppendDirectoryInfo(w io.Writer) {
for _, record := range directoryRecords() {
_, _ = fmt.Fprintf(w, "%s: %s\n", record.label, record.path)
}
}
// DirectoryInfo returns nuclei config/state/cache/PDCP directory paths as a string.
func DirectoryInfo() string {
var b strings.Builder
AppendDirectoryInfo(&b)
return b.String()
}
// LogDirectoryInfo logs nuclei config/cache/PDCP directory paths at info level.
// LogDirectoryInfo logs nuclei config/state/cache/PDCP directory paths at info level.
func LogDirectoryInfo(logger *gologger.Logger) {
if logger == nil {
return
}
logger.Info().Msgf("Nuclei Config Directory: %s", config.DefaultConfig.GetConfigDir())
logger.Info().Msgf("Nuclei Cache Directory: %s", config.DefaultConfig.GetCacheDir())
logger.Info().Msgf("PDCP Directory: %s", pdcpauth.PDCPDir)
for _, record := range directoryRecords() {
logger.Info().Msgf("%s: %s", record.label, record.path)
}
}
+30 -1
View File
@@ -5,21 +5,50 @@ import (
"strings"
"testing"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/gologger/formatter"
"github.com/projectdiscovery/gologger/levels"
"github.com/stretchr/testify/require"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
pdcpauth "github.com/projectdiscovery/utils/auth/pdcp"
)
type directoryLogWriter struct {
records []string
}
func (w *directoryLogWriter) Write(data []byte, _ levels.Level) {
w.records = append(w.records, string(data))
}
func TestDirectoryInfoContainsKnownLabels(t *testing.T) {
info := DirectoryInfo()
cfg := config.DefaultConfig
require.Equal(t, strings.Join([]string{
fmt.Sprintf("Nuclei Config Directory: %s", cfg.GetConfigDir()),
fmt.Sprintf("Nuclei State Directory: %s", cfg.GetStateDir()),
fmt.Sprintf("Nuclei Cache Directory: %s", cfg.GetCacheDir()),
fmt.Sprintf("PDCP Directory: %s", pdcpauth.PDCPDir),
"",
}, "\n"), info)
require.Equal(t, 3, strings.Count(info, "\n"))
require.Equal(t, 4, strings.Count(info, "\n"))
}
func TestLogDirectoryInfoContainsKnownLabels(t *testing.T) {
writer := &directoryLogWriter{}
logger := &gologger.Logger{}
logger.SetFormatter(formatter.NewCLI(true))
logger.SetWriter(writer)
logger.SetMaxLevel(levels.LevelInfo)
LogDirectoryInfo(logger)
require.Equal(t, []string{
fmt.Sprintf("[INF] Nuclei Config Directory: %s", config.DefaultConfig.GetConfigDir()),
fmt.Sprintf("[INF] Nuclei State Directory: %s", config.DefaultConfig.GetStateDir()),
fmt.Sprintf("[INF] Nuclei Cache Directory: %s", config.DefaultConfig.GetCacheDir()),
fmt.Sprintf("[INF] PDCP Directory: %s", pdcpauth.PDCPDir),
}, writer.records)
}
+26 -1
View File
@@ -23,17 +23,31 @@ func DoHealthCheck(options *types.Options) string {
AppendDirectoryInfo(&test)
var testResult string
cfg := config.DefaultConfig
for _, filename := range []string{cfg.GetFlagsConfigFilePath(), cfg.GetIgnoreFilePath(), cfg.GetChecksumFilePath()} {
if err := cfg.InitializationError(); err != nil {
fmt.Fprintf(&test, "Configuration initialization => Ko (%s)\n", err)
} else {
fmt.Fprintln(&test, "Configuration initialization => Ok")
}
for _, filename := range []string{
cfg.GetFlagsConfigFilePath(),
cfg.GetTemplatesStateFilePath(),
cfg.GetActiveIgnoreFilePath(),
cfg.GetChecksumFilePath(),
} {
ok, err := fileutil.IsReadable(filename)
if ok {
testResult = "Ok"
} else {
testResult = "Ko"
}
if err != nil {
testResult += fmt.Sprintf(" (%s)", err)
}
fmt.Fprintf(&test, "File \"%s\" Read => %s\n", filename, testResult)
ok, err = fileutil.IsWriteable(filename)
if ok {
@@ -41,37 +55,48 @@ func DoHealthCheck(options *types.Options) string {
} else {
testResult = "Ko"
}
if err != nil {
testResult += fmt.Sprintf(" (%s)", err)
}
fmt.Fprintf(&test, "File \"%s\" Write => %s\n", filename, testResult)
}
c4, err := net.Dial("tcp4", "scanme.sh:80")
if err == nil && c4 != nil {
_ = c4.Close()
}
testResult = "Ok"
if err != nil {
testResult = fmt.Sprintf("Ko (%s)", err)
}
fmt.Fprintf(&test, "IPv4 connectivity to scanme.sh:80 => %s\n", testResult)
c6, err := net.Dial("tcp6", "scanme.sh:80")
if err == nil && c6 != nil {
_ = c6.Close()
}
testResult = "Ok"
if err != nil {
testResult = fmt.Sprintf("Ko (%s)", err)
}
fmt.Fprintf(&test, "IPv6 connectivity to scanme.sh:80 => %s\n", testResult)
u4, err := net.Dial("udp4", "scanme.sh:53")
if err == nil && u4 != nil {
_ = u4.Close()
}
testResult = "Ok"
if err != nil {
testResult = fmt.Sprintf("Ko (%s)", err)
}
fmt.Fprintf(&test, "IPv4 UDP connectivity to scanme.sh:53 => %s\n", testResult)
return test.String()
+63 -21
View File
@@ -117,6 +117,10 @@ func New(options *types.Options) (*Runner, error) {
Logger: options.Logger,
}
if err := config.DefaultConfig.InitializationError(); err != nil && !options.HealthCheck {
return nil, fmt.Errorf("initialize nuclei configuration: %w", err)
}
if options.HealthCheck {
runner.Logger.Print().Msgf("%s\n", DoHealthCheck(options))
os.Exit(0)
@@ -128,9 +132,10 @@ func New(options *types.Options) (*Runner, error) {
// Version check by default
if config.DefaultConfig.CanCheckForUpdates() {
if err := installer.NucleiVersionCheck(); err != nil {
latestIgnoreHash, versionCheckErr := installer.NucleiVersionCheck()
if versionCheckErr != nil {
if options.Verbose || options.Debug {
runner.Logger.Error().Msgf("nuclei version check failed got: %s\n", err)
runner.Logger.Error().Msgf("nuclei version check failed got: %s\n", versionCheckErr)
}
}
@@ -147,15 +152,15 @@ func New(options *types.Options) (*Runner, error) {
DisablePublicTemplates: options.PublicTemplateDisableDownload,
}
if err := tm.FreshInstallIfNotExists(); err != nil {
runner.Logger.Warning().Msgf("failed to install nuclei templates: %s\n", err)
runner.Logger.Warning().Msgf("Failed to install nuclei templates: %s\n", err)
}
if err := tm.UpdateIfOutdated(); err != nil {
runner.Logger.Warning().Msgf("failed to update nuclei templates: %s\n", err)
runner.Logger.Warning().Msgf("Failed to update nuclei templates: %s\n", err)
}
if config.DefaultConfig.NeedsIgnoreFileUpdate() {
if config.DefaultConfig.IgnoreFileNeedsUpdate(latestIgnoreHash) {
if err := installer.UpdateIgnoreFile(); err != nil {
runner.Logger.Warning().Msgf("failed to update nuclei ignore file: %s\n", err)
runner.Logger.Warning().Msgf("Failed to update .nuclei-ignore file: %s\n", err)
}
}
@@ -165,6 +170,7 @@ func New(options *types.Options) (*Runner, error) {
if !config.DefaultConfig.NeedsTemplateUpdate() {
runner.Logger.Info().Msgf("No new updates found for nuclei templates")
}
// manually trigger update of custom templates
if ctm != nil {
ctm.Update(context.TODO())
@@ -181,6 +187,7 @@ func New(options *types.Options) (*Runner, error) {
if options.Validate {
parser.ShouldValidate = true
}
// TODO: refactor to pass options reference globally without cycles
parser.NoStrictSyntax = options.NoStrictSyntax
runner.parser = parser
@@ -193,10 +200,12 @@ func New(options *types.Options) (*Runner, error) {
if engine.MustDisableSandbox() {
runner.Logger.Warning().Msgf("The current platform and privileged user will run the browser without sandbox\n")
}
browser, err := engine.New(options)
if err != nil {
return nil, err
}
runner.browser = browser
}
@@ -205,6 +214,7 @@ func New(options *types.Options) (*Runner, error) {
var httpclient *retryablehttp.Client
if options.ProxyInternal && options.AliveHttpProxy != "" || options.AliveSocksProxy != "" {
var err error
httpclient, err = httpclientpool.Get(options, &httpclientpool.Configuration{}, "")
if err != nil {
return nil, err
@@ -214,10 +224,12 @@ func New(options *types.Options) (*Runner, error) {
if err := reporting.CreateConfigIfNotExists(); err != nil {
return nil, err
}
reportingOptions, err := createReportingOptions(options)
if err != nil {
return nil, err
}
if reportingOptions != nil && httpclient != nil {
reportingOptions.HttpClient = httpclient
}
@@ -227,6 +239,7 @@ func New(options *types.Options) (*Runner, error) {
if err != nil {
return nil, errors.Wrap(err, "could not create issue reporting client")
}
runner.issuesClient = client
}
@@ -289,10 +302,12 @@ func New(options *types.Options) (*Runner, error) {
if err != nil {
return nil, errors.Wrap(err, "could not create output file")
}
if hpDetector != nil {
outputWriter.SetHoneypotDetector(hpDetector)
runner.honeypotDetector = hpDetector
}
// setup a proxy writer to automatically upload results to PDCP
runner.output = runner.setupPDCPUpload(outputWriter)
if options.HTTPStats {
@@ -303,9 +318,11 @@ func New(options *types.Options) (*Runner, error) {
if options.JSONL && options.EnableProgressBar {
options.StatsJSON = true
}
if options.StatsJSON {
options.EnableProgressBar = true
}
// Creates the progress tracking object
var progressErr error
statsInterval := options.StatsInterval
@@ -331,12 +348,15 @@ func New(options *types.Options) (*Runner, error) {
if err != nil {
return nil, err
}
err = json.Unmarshal(file, &resumeCfg)
if err != nil {
return nil, err
}
resumeCfg.Compile()
}
runner.resumeCfg = resumeCfg
if options.DASTReport || options.DASTServer {
@@ -345,11 +365,13 @@ func New(options *types.Options) (*Runner, error) {
if err != nil {
return nil, errors.Wrap(err, "could not create fuzz stats db")
}
if !options.DASTServer {
dastServer, err := server.NewStatsServer(runner.fuzzStats)
if err != nil {
return nil, errors.Wrap(err, "could not create dast server")
}
runner.dastServer = dastServer
}
}
@@ -359,6 +381,7 @@ func New(options *types.Options) (*Runner, error) {
if request.Error == "none" || request.Error == "" {
return
}
runner.fuzzStats.RecordErrorEvent(fuzzStats.ErrorEvent{
TemplateID: request.Template,
URL: request.Input,
@@ -396,6 +419,7 @@ func New(options *types.Options) (*Runner, error) {
// in testing it was found most of times when interactsh failed, it was due to failure in registering /polling requests
opts.HTTPClient = retryablehttp.NewClient(retryablehttp.DefaultOptionsSingle)
}
interactshClient, err := interactsh.New(opts)
if err != nil {
runner.Logger.Error().Msgf("Could not create interactsh client: %s", err)
@@ -408,9 +432,11 @@ func New(options *types.Options) (*Runner, error) {
options.RateLimit = options.RateLimitMinute
options.RateLimitDuration = time.Minute
}
if options.RateLimit > 0 && options.RateLimitDuration == 0 {
options.RateLimitDuration = time.Second
}
// If per-host rate limiting is enabled, make global rate limiter unlimited
if options.PerHostRateLimit {
runner.rateLimiter = utils.GetRateLimiter(context.Background(), 0, 0)
@@ -420,6 +446,7 @@ func New(options *types.Options) (*Runner, error) {
// Initialization successful, disable cleanup on error
cleanupOnError = false
return runner, nil
}
@@ -428,6 +455,7 @@ func (r *Runner) runStandardEnumeration(executerOpts *protocols.ExecutorOptions,
if r.options.AutomaticScan {
return r.executeSmartWorkflowInput(executerOpts, store, engine)
}
return r.executeTemplatesInput(store, engine)
}
@@ -680,8 +708,8 @@ func (r *Runner) RunEnumeration() error {
// If using input-file flags, only load http fuzzing based templates.
loaderConfig := loader.NewConfig(r.options, r.catalog, executorOpts)
if !strings.EqualFold(r.options.InputFileMode, "list") || r.options.DAST {
// if input type is not list (implicitly enable fuzzing)
if (!strings.EqualFold(r.options.InputFileMode, "list") || r.options.DAST) && !r.options.OfflineHTTP {
// if input type is not list (implicitly enable fuzzing), unless passive/offlinehttp
r.options.DAST = true
}
store, err := loader.New(loaderConfig)
@@ -971,15 +999,8 @@ func (r *Runner) displayExecutionInfo(store *loader.Store) {
cfg := config.DefaultConfig
updateutils.Aurora = r.colorizer
versionInfo := func(version, latestVersion, versionType string) string {
if !cfg.CanCheckForUpdates() {
return fmt.Sprintf("Current %s version: %v (%s) - remove '-duc' flag to enable update checks", versionType, version, r.colorizer.BrightYellow("unknown"))
}
return fmt.Sprintf("Current %s version: %v %v", versionType, version, updateutils.GetVersionDescription(version, latestVersion))
}
gologger.Info().Msg(versionInfo(config.Version, cfg.LatestNucleiVersion, "nuclei"))
gologger.Info().Msg(versionInfo(cfg.TemplateVersion, cfg.LatestNucleiTemplatesVersion, "nuclei-templates"))
gologger.Info().Msg(versionInfo(cfg, r.colorizer, config.Version, cfg.LatestNucleiVersion, "nuclei"))
gologger.Info().Msg(templateVersionInfo(cfg, r.colorizer))
if !HideAutoSaveMsg {
if r.pdcpUploadErrMsg != "" {
r.Logger.Warning().Msgf("%s", r.pdcpUploadErrMsg)
@@ -1013,19 +1034,40 @@ func (r *Runner) displayExecutionInfo(store *loader.Store) {
}
}
func versionInfo(cfg *config.Config, colorizer *aurora.Aurora, version, latestVersion, versionType string) string {
if !cfg.CanCheckForUpdates() {
return fmt.Sprintf("Current %s version: %v (%s) - remove '-duc' flag to enable update checks", versionType, version, colorizer.BrightYellow("unknown"))
}
return fmt.Sprintf("Current %s version: %v %v", versionType, version, updateutils.GetVersionDescription(version, latestVersion))
}
func templateVersionInfo(cfg *config.Config, colorizer *aurora.Aurora) string {
return versionInfo(cfg, colorizer, cfg.TemplateVersion, cfg.LatestNucleiTemplatesVersion, "nuclei-templates")
}
// SaveResumeConfig to file
func (r *Runner) SaveResumeConfig(path string) error {
dir := filepath.Dir(path)
if !fileutil.FolderExists(dir) {
if err := os.MkdirAll(dir, os.ModePerm); err != nil {
return err
if err := os.MkdirAll(dir, 0o700); err != nil {
return fmt.Errorf("create resume directory %q: %w", dir, err)
}
}
resumeCfgClone := r.resumeCfg.Clone()
resumeCfgClone.ResumeFrom = resumeCfgClone.Current
data, _ := json.MarshalIndent(resumeCfgClone, "", "\t")
return os.WriteFile(path, data, permissionutil.ConfigFilePermission)
data, err := json.MarshalIndent(resumeCfgClone, "", "\t")
if err != nil {
return fmt.Errorf("encode resume configuration: %w", err)
}
if err := os.WriteFile(path, data, permissionutil.ConfigFilePermission); err != nil {
return fmt.Errorf("write resume configuration %q: %w", path, err)
}
return nil
}
// upload existing scan results to cloud with progress
+45
View File
@@ -0,0 +1,45 @@
package runner
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
)
func TestTemplateVersionInfoIgnoresLegacyDiscoverySource(t *testing.T) {
dir := t.TempDir()
root := filepath.Join(dir, "system", config.BinaryName, config.NucleiTemplatesDirName)
cfg := &config.Config{}
cfg.SetStateDir(filepath.Join(dir, "state"))
state, err := json.Marshal(map[string]string{
"nuclei-templates-directory": root,
"nuclei-templates-directory-source": "xdg-system",
"nuclei-templates-version": "v1.2.3",
"nuclei-templates-latest-version": "v99.0.0",
})
if err != nil {
t.Fatalf("marshal templates state: %v", err)
}
if err := os.MkdirAll(filepath.Dir(cfg.GetTemplatesStateFilePath()), 0o700); err != nil {
t.Fatalf("create state directory: %v", err)
}
if err := os.WriteFile(cfg.GetTemplatesStateFilePath(), state, 0o600); err != nil {
t.Fatalf("write templates state: %v", err)
}
if err := cfg.ReadTemplatesConfig(); err != nil {
t.Fatalf("read templates state: %v", err)
}
info := templateVersionInfo(cfg, nil)
if !strings.Contains(info, "v1.2.3") || !strings.Contains(info, "outdated") {
t.Fatalf("template version info = %q, want normal version comparison", info)
}
if strings.Contains(info, "package-managed") {
t.Fatalf("template version info retained discovery ownership: %q", info)
}
}
+2 -2
View File
@@ -117,8 +117,8 @@ func newNucleiExecutor(opts *NucleiExecutorOptions) (*nucleiExecutor, error) {
// If using input-file flags, only load http fuzzing based templates.
loaderConfig := loader.NewConfig(opts.Options, opts.Catalog, executorOpts)
if !strings.EqualFold(opts.Options.InputFileMode, "list") || opts.Options.DAST || opts.Options.DASTServer {
// if input type is not list (implicitly enable fuzzing)
if (!strings.EqualFold(opts.Options.InputFileMode, "list") || opts.Options.DAST || opts.Options.DASTServer) && !opts.Options.OfflineHTTP {
// if input type is not list (implicitly enable fuzzing), unless passive/offlinehttp
opts.Options.DAST = true
}
store, err := loader.New(loaderConfig)
+23 -10
View File
@@ -59,7 +59,12 @@ func runTestMain(m *testing.M) int {
ci: isCI(),
runner: testutils.NewRunner(
testutils.WithWorkingDir(repoRoot),
testutils.WithBaseEnv("NUCLEI_CONFIG_DIR="+configDir),
testutils.WithBaseEnv(
"NUCLEI_CONFIG_DIR="+configDir,
"NUCLEI_TEMPLATES_DIR="+filepath.Join(configDir, "templates"),
"XDG_STATE_HOME="+filepath.Join(configDir, "state"),
"XDG_CACHE_HOME="+filepath.Join(configDir, "cache"),
),
),
}
defer func() {
@@ -447,23 +452,31 @@ func candidateExecutableNames(name string) []string {
}
func prepareFunctionalEnvironment(binaryPath, repoRoot, configDir string) error {
configRoot := filepath.Join(configDir, "nuclei")
if err := os.MkdirAll(configRoot, 0755); err != nil {
return fmt.Errorf("failed to create functional nuclei config root: %w", err)
}
ignoreFile := filepath.Join(configRoot, ".nuclei-ignore")
if err := os.WriteFile(ignoreFile, nil, 0644); err != nil {
return fmt.Errorf("failed to create functional ignore file: %w", err)
}
templatesDir := filepath.Join(configDir, "templates")
for _, args := range [][]string{{"-update-templates"}, {"-validate"}} {
cmd := exec.Command(binaryPath, args...)
cmd.Dir = repoRoot
cmd.Env = append(os.Environ(), "NUCLEI_CONFIG_DIR="+configDir)
cmd.Env = append(os.Environ(),
"NUCLEI_CONFIG_DIR="+configDir,
"NUCLEI_TEMPLATES_DIR="+templatesDir,
"XDG_STATE_HOME="+filepath.Join(configDir, "state"),
"XDG_CACHE_HOME="+filepath.Join(configDir, "cache"),
)
output, err := cmd.CombinedOutput()
if err != nil {
return fmt.Errorf("functional setup command %v failed: %w\n%s", args, err, strings.TrimSpace(string(output)))
}
}
ignoreFile, err := os.ReadFile(filepath.Join(templatesDir, ".nuclei-ignore"))
if err != nil {
return fmt.Errorf("failed to read functional ignore file: %w", err)
}
if err := os.WriteFile(filepath.Join(configDir, ".nuclei-ignore"), ignoreFile, 0o600); err != nil {
return fmt.Errorf("failed to write legacy functional ignore file: %w", err)
}
return nil
}
@@ -119,7 +119,9 @@ func TestTemplatesDirEnv(t *testing.T) {
func TestCustomConfigDir(t *testing.T) {
t.Run("ConfigDirIsolated", func(t *testing.T) {
customTempDir := t.TempDir()
results, err := testutils.RunNucleiBareArgsAndGetResults(suite.debug, []string{"NUCLEI_CONFIG_DIR=" + customTempDir}, "-t", "protocols/http/get.yaml", "-u", newTemplateDirTarget(t))
stateHome := t.TempDir()
templatesDir := t.TempDir()
results, err := testutils.RunNucleiBareArgsAndGetResults(suite.debug, []string{"NUCLEI_CONFIG_DIR=" + customTempDir, "XDG_STATE_HOME=" + stateHome, "NUCLEI_TEMPLATES_DIR=" + templatesDir}, "-t", "protocols/http/get.yaml", "-u", newTemplateDirTarget(t))
if err != nil {
t.Fatalf("custom config dir request failed: %v", err)
}
@@ -132,11 +134,14 @@ func TestCustomConfigDir(t *testing.T) {
for _, file := range files {
fileNames = append(fileNames, file.Name())
}
for _, requiredFile := range []string{".templates-config.json", "config.yaml", "reporting-config.yaml"} {
for _, requiredFile := range []string{"config.yaml", "reporting-config.yaml"} {
if !slices.Contains(fileNames, requiredFile) {
t.Fatalf("missing required config file %q in custom config dir: %v", requiredFile, fileNames)
}
}
if _, err := os.Stat(filepath.Join(stateHome, "nuclei", "templates.json")); err != nil {
t.Fatalf("missing templates state file: %v", err)
}
}
})
}
+53
View File
@@ -5,9 +5,13 @@ package integration_test
import (
"fmt"
"io"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"strings"
"github.com/julienschmidt/httprouter"
"github.com/projectdiscovery/nuclei/v3/internal/tests/testutils"
@@ -36,6 +40,55 @@ var fuzzingTestCases = []integrationCase{
{Path: "fuzz/fuzz-body-params-sqli.yaml", TestCase: &genericFuzzTestCase{expectedResults: 1}},
{Path: "fuzz/fuzz-body-xml-sqli.yaml", TestCase: &genericFuzzTestCase{expectedResults: 1}},
{Path: "fuzz/fuzz-body-generic-sqli.yaml", TestCase: &genericFuzzTestCase{expectedResults: 4}},
{Path: "fuzz/fuzz-raw-http-input.yaml", TestCase: &rawHttpInputFuzz{}},
}
// rawHttpInputFuzz seeds the base request from a raw HTTP request file, the
// shape users have on hand when there is no spec or captured traffic to feed in.
type rawHttpInputFuzz struct{}
func (h *rawHttpInputFuzz) Execute(filePath string) error {
router := httprouter.New()
router.POST("/api/login", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
body, err := io.ReadAll(r.Body)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
w.Header().Set("Content-Type", "text/html")
var payload map[string]string
if err := json.Unmarshal(body, &payload); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
// stand in for a backend that interpolates the value into a query
if strings.Contains(payload["user"], "'") {
_, _ = fmt.Fprint(w, "unrecognized token:")
return
}
_, _ = fmt.Fprintf(w, "welcome %s", payload["user"])
})
ts := httptest.NewServer(router)
defer ts.Close()
target, err := url.Parse(ts.URL)
if err != nil {
return err
}
rawRequest := fmt.Sprintf("POST /api/login HTTP/1.1\r\nHost: %s\r\nContent-Type: application/json\r\n\r\n{\"user\":\"admin\"}", target.Host)
targetPath := filepath.Join(os.TempDir(), "nuclei-raw-http-input.http")
if err := os.WriteFile(targetPath, []byte(rawRequest), 0644); err != nil {
return err
}
defer func() {
_ = os.Remove(targetPath)
}()
results, err := testutils.RunNucleiWithArgsAndGetResults(debug, "-t", filePath, "-l", targetPath, "-im", "http", "-dast")
if err != nil {
return err
}
return expectResultsCount(results, 1)
}
type genericFuzzTestCase struct {
+24
View File
@@ -43,6 +43,7 @@ var httpTestcases = []integrationCase{
{Path: "protocols/http/post-json-body.yaml", TestCase: &httpPostJSONBody{}},
{Path: "protocols/http/post-multipart-body.yaml", TestCase: &httpPostMultipartBody{}},
{Path: "protocols/http/raw-cookie-reuse.yaml", TestCase: &httpRawCookieReuse{}},
{Path: "protocols/http/disable-cookie-reuse.yaml", TestCase: &httpDisableCookieReuse{}},
{Path: "protocols/http/raw-dynamic-extractor.yaml", TestCase: &httpRawDynamicExtractor{}},
{Path: "protocols/http/raw-get-query.yaml", TestCase: &httpRawGetQuery{}},
{Path: "protocols/http/raw-get.yaml", TestCase: &httpRawGet{}},
@@ -902,6 +903,29 @@ func (h *httpPaths) Execute(filepath string) error {
type httpRawCookieReuse struct{}
type httpDisableCookieReuse struct{}
func (h *httpDisableCookieReuse) Execute(filePath string) error {
router := httprouter.New()
router.GET("/login", func(w http.ResponseWriter, _ *http.Request, _ httprouter.Params) {
http.SetCookie(w, &http.Cookie{Name: "nuclei", Value: "test"})
_, _ = fmt.Fprint(w, "logged in")
})
router.GET("/anonymous", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
if _, err := r.Cookie("nuclei"); errors.Is(err, http.ErrNoCookie) {
_, _ = fmt.Fprint(w, "anonymous")
}
})
ts := httptest.NewServer(router)
defer ts.Close()
results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
if err != nil {
return err
}
return expectResultsCount(results, 1)
}
// Execute executes a test case and returns an error if occurred
func (h *httpRawCookieReuse) Execute(filePath string) error {
router := httprouter.New()
@@ -18,6 +18,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/internal/fuzzplayground"
"github.com/projectdiscovery/nuclei/v3/internal/tests/testutils"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
)
type integrationHarness struct {
@@ -63,12 +64,26 @@ func TestMain(m *testing.M) {
_ = os.RemoveAll(tempDir)
os.Exit(1)
}
homeDir, err := os.UserHomeDir()
if err != nil {
fmt.Fprintf(os.Stderr, "failed to determine user home directory: %v\n", err)
_ = os.RemoveAll(tempDir)
os.Exit(1)
}
templatesDir := filepath.Join(homeDir, config.NucleiTemplatesDirName)
config.DefaultConfig.SetTemplatesDir(templatesDir)
previousRunner := testutils.DefaultRunner()
runner := testutils.NewRunner(
testutils.WithBinaryPath(binaryPath),
testutils.WithWorkingDir(workingFixturesDir),
testutils.WithExtraArgs(integrationExtraArgs...),
testutils.WithBaseEnv(
"NUCLEI_TEMPLATES_DIR="+templatesDir,
"XDG_CONFIG_HOME="+filepath.Join(tempDir, "config"),
"XDG_STATE_HOME="+filepath.Join(tempDir, "state"),
"XDG_CACHE_HOME="+filepath.Join(tempDir, "cache"),
),
)
testutils.SetDefaultRunner(runner)
+1 -1
View File
@@ -106,7 +106,7 @@ func integrationFamilies() []integrationFamily {
{Name: "headless", Cases: headlessTestcases},
{Name: "whois", Cases: whoisTestCases},
{Name: "library", Cases: libraryTestcases},
{Name: "template-path", Cases: templatesPathTestCases},
{Name: "template-path", Cases: templatesPathTestCases()},
{Name: "offline-http", Cases: offlineHttpTestcases},
{Name: "fuzz", Cases: fuzzingTestCases},
{Name: "generic", Cases: genericTestcases},
@@ -5,25 +5,24 @@ package integration_test
import (
"fmt"
"path/filepath"
"strings"
"github.com/projectdiscovery/nuclei/v3/internal/tests/testutils"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
)
func getTemplatePath() string {
return config.DefaultConfig.TemplatesDirectory
}
var templatesPathTestCases = []integrationCase{
//template folder path issue
{Path: "protocols/http/get.yaml", TestCase: &folderPathTemplateTest{}},
//cwd
{Path: "./dns/detect-dangling-cname.yaml", TestCase: &cwdTemplateTest{}},
//relative path
{Path: "dns/dns-saas-service-detection.yaml", TestCase: &relativePathTemplateTest{}},
//absolute path
{Path: fmt.Sprintf("%v/dns/dns-saas-service-detection.yaml", getTemplatePath()), TestCase: &absolutePathTemplateTest{}},
func templatesPathTestCases() []integrationCase {
return []integrationCase{
//template folder path issue
{Path: "protocols/http/get.yaml", TestCase: &folderPathTemplateTest{}},
//cwd
{Path: "./dns/detect-dangling-cname.yaml", TestCase: &cwdTemplateTest{}},
//relative path
{Path: "dns/dns-saas-service-detection.yaml", TestCase: &relativePathTemplateTest{}},
//absolute path
{Path: filepath.Join(config.DefaultConfig.TemplatesDirectory, "dns", "dns-saas-service-detection.yaml"), TestCase: &absolutePathTemplateTest{}},
}
}
type cwdTemplateTest struct{}
@@ -0,0 +1,34 @@
id: raw-http-input-body-fuzz
info:
name: fuzzing json body seeded from a raw http request file
author: pdteam
severity: info
description: |
This template fuzzes the json body of a base request seeded from a raw HTTP
request file (-im http), covering the shortest path from a single request to
a DAST scan.
http:
- pre-condition:
- type: dsl
dsl:
- method == "POST"
- contains(content_type, "application/json")
condition: and
payloads:
injection:
- "'"
fuzzing:
- part: body
type: postfix
mode: single
fuzz:
- '{{injection}}'
matchers:
- type: word
words:
- "unrecognized token:"
@@ -0,0 +1,27 @@
id: disable-cookie-reuse
info:
name: Disable Cookie Reuse
author: pdteam
severity: info
http:
- method: GET
path:
- "{{BaseURL}}/login"
matchers:
- type: word
words:
- "logged in"
internal: true
- method: GET
path:
- "{{BaseURL}}/anonymous"
disable-cookie: true
matchers:
- type: word
words:
- "anonymous"
-1
View File
@@ -40,7 +40,6 @@ func Cleanup(options *types.Options) {
// DefaultOptions is the default options structure for nuclei during mocking.
var DefaultOptions = &types.Options{
Metrics: false,
Debug: false,
DebugRequests: false,
DebugResponse: false,
+70
View File
@@ -0,0 +1,70 @@
package nuclei
import (
"context"
"os"
"path/filepath"
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/templates"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/stretchr/testify/require"
)
func TestCallerParserUsesEngineLocalCompiledCache(t *testing.T) {
templatePath := filepath.Join(t.TempDir(), "template.yaml")
require.NoError(t, os.WriteFile(templatePath, []byte(`id: parser-lifecycle
info:
name: Parser lifecycle
author: pdteam
severity: info
http:
- method: GET
path:
- "{{BaseURL}}"
`), 0o600))
callerParser := templates.NewParser()
callerParser.Cache().StoreWithoutRaw("sentinel", &templates.Template{}, nil)
for range 3 {
compiledParser := func() *templates.Parser {
opts := types.DefaultOptions()
opts.Parser = callerParser
opts.Templates = []string{templatePath}
engine, err := NewNucleiEngineCtx(context.Background(), DisableUpdateCheck(), WithOptions(opts))
require.NoError(t, err)
engineParser := engine.GetParser()
require.Same(t, callerParser, engineParser)
require.False(t, engine.GetExecuterOptions().DoNotCache)
compiledParser, ok := engine.GetExecuterOptions().Parser.(*templates.Parser)
require.True(t, ok)
require.NotSame(t, callerParser, compiledParser)
require.Same(t, callerParser.Cache(), compiledParser.Cache())
require.NoError(t, engine.LoadAllTemplates())
require.Len(t, engine.GetTemplates(), 1)
require.Equal(t, 0, callerParser.CompiledCount())
require.Equal(t, 1, compiledParser.CompiledCount())
require.Equal(t, 2, callerParser.ParsedCount())
require.NoError(t, engine.LoadAllTemplates())
require.Len(t, engine.GetTemplates(), 1)
require.Equal(t, 1, compiledParser.CompiledCount())
engine.Close()
return compiledParser
}()
require.Equal(t, 0, callerParser.CompiledCount())
require.Equal(t, 0, compiledParser.CompiledCount())
require.Equal(t, 2, callerParser.ParsedCount())
cached, err := callerParser.Cache().Get("sentinel")
require.NoError(t, err)
require.NotNil(t, cached)
}
}
+6 -4
View File
@@ -81,6 +81,7 @@ type NucleiEngine struct {
browserInstance *engine.Browser
httpClient *retryablehttp.Client
parser *templates.Parser
compiledParser *templates.Parser
ownsParser bool // true when the engine created the parser and may purge it on Close
authprovider authprovider.AuthProvider
@@ -276,10 +277,11 @@ func (e *NucleiEngine) closeInternal() {
if e.opts != nil {
generators.ClearOptionsPayloadMap(e.opts)
}
// Purge the template caches (compiled templates are heap-heavy) so a
// long-running embedder does not retain them for the process lifetime.
// Only do this when the engine created the parser; a caller-supplied
// parser is an opt-in shared cache the caller owns.
// Purge engine-owned template caches. A caller-supplied parser remains
// untouched, while its engine-local compiled cache is always released.
if e.compiledParser != nil && e.compiledParser != e.parser {
e.compiledParser.CompiledCache().Purge()
}
if e.ownsParser && e.parser != nil {
e.parser.Purge()
}
+28 -9
View File
@@ -144,6 +144,9 @@ func (e *NucleiEngine) applyRequiredDefaults(ctx context.Context) {
// init
func (e *NucleiEngine) init(ctx context.Context) error {
if err := config.DefaultConfig.InitializationError(); err != nil {
return errors.Wrap(err, "initialize nuclei configuration")
}
// Update logger ref (if it was changed by [WithLogger])
// (Logger is already initialized)
if e.opts.Logger != e.Logger {
@@ -172,6 +175,14 @@ func (e *NucleiEngine) init(ctx context.Context) error {
e.parser = templates.NewParser()
e.ownsParser = true
}
e.compiledParser = e.parser
if !e.ownsParser {
// Compiled templates retain engine-local ExecutorOptions. Keep their
// cache private while the caller-owned parser provides the parsed cache.
e.compiledParser = templates.NewParserWithParsedCache(e.parser.Cache())
e.compiledParser.ShouldValidate = e.parser.ShouldValidate
e.compiledParser.NoStrictSyntax = e.parser.NoStrictSyntax
}
if protocolstate.ShouldInit(e.opts.ExecutionId) {
_ = protocolinit.Init(e.opts)
@@ -250,7 +261,8 @@ func (e *NucleiEngine) init(ctx context.Context) error {
Colorizer: aurora.New(aurora.WithColors(true)),
ResumeCfg: types.NewResumeCfg(),
Browser: e.browserInstance,
Parser: e.parser,
Parser: e.compiledParser,
DoNotCache: e.opts.DoNotCacheTemplates,
InputHelper: input.NewHelper(),
TemporaryDirectory: e.tmpDir,
Logger: e.opts.Logger,
@@ -306,8 +318,6 @@ func (e *NucleiEngine) init(ctx context.Context) error {
if cachedParser, ok := e.opts.Parser.(*templates.Parser); ok {
e.parser = cachedParser
e.opts.Parser = cachedParser
e.executerOpts.Parser = cachedParser
e.executerOpts.Options.Parser = cachedParser
}
}
@@ -332,14 +342,23 @@ func (e *NucleiEngine) init(ctx context.Context) error {
e.httpxClient = nucleiUtils.GetInputLivenessChecker(client)
}
// Only Happens once regardless how many times this function is called
// This will update ignore file to filter out templates with weak matchers to avoid false positives
// and also upgrade templates to latest version if available
installer.NucleiSDKVersionCheck()
if DefaultConfig.CanCheckForUpdates() {
return e.processUpdateCheckResults()
// Only Happens once regardless how many times this function is called
// This will update ignore file to filter out templates with weak matchers to avoid false positives
// and also upgrade templates to latest version if available
latestIgnoreHash, _ := installer.NucleiSDKVersionCheck()
if err := e.processUpdateCheckResults(); err != nil {
return err
}
if DefaultConfig.IgnoreFileNeedsUpdate(latestIgnoreHash) {
if err := installer.UpdateIgnoreFile(); err != nil {
e.opts.Logger.Warning().Msgf("failed to update nuclei ignore file: %s\n", err)
}
}
}
return nil
}
+8 -2
View File
@@ -21,9 +21,15 @@ var (
)
const (
TemplateConfigFileName = ".templates-config.json"
// TODO(dwisiswant0): do not use this in the future.
legacyTemplatesConfigFileName = ".templates-config.json"
// TemplateConfigFileName is the legacy templates-state filename.
//
// Deprecated: use TemplatesStateFileName for current templates state.
TemplateConfigFileName = legacyTemplatesConfigFileName
TemplatesStateFileName = "templates.json"
NucleiTemplatesDirName = "nuclei-templates"
OfficialNucleiTemplatesRepoName = "nuclei-templates"
OfficialNucleiTemplatesRepoName = NucleiTemplatesDirName
NucleiIgnoreFileName = ".nuclei-ignore"
NucleiTemplatesIndexFileName = ".templates-index" // contains index of official nuclei templates
NucleiTemplatesCheckSumFileName = ".checksum"
+64 -4
View File
@@ -1,6 +1,8 @@
package config
import (
"crypto/md5"
"fmt"
"os"
"github.com/projectdiscovery/gologger"
@@ -13,11 +15,12 @@ type IgnoreFile struct {
Files []string `yaml:"files"`
}
// ReadIgnoreFile reads the nuclei ignore file returning blocked tags and paths
// ReadIgnoreFile reads the .nuclei-ignore file returning blocked tags and paths.
func ReadIgnoreFile() IgnoreFile {
file, err := os.Open(DefaultConfig.GetIgnoreFilePath())
path := DefaultConfig.GetActiveIgnoreFilePath()
file, err := os.Open(path)
if err != nil {
gologger.Error().Msgf("Could not read nuclei-ignore file: %s\n", err)
gologger.Error().Msgf("Could not read .nuclei-ignore file %q: %s\n", path, err)
return IgnoreFile{}
}
defer func() {
@@ -26,8 +29,65 @@ func ReadIgnoreFile() IgnoreFile {
ignore := IgnoreFile{}
if err := yaml.NewDecoder(file).Decode(&ignore); err != nil {
gologger.Error().Msgf("Could not parse nuclei-ignore file: %s\n", err)
gologger.Error().Msgf("Could not parse .nuclei-ignore file %q: %s\n", path, err)
return IgnoreFile{}
}
return ignore
}
// WriteActiveIgnoreFile atomically replaces the active root ignore file.
func (c *Config) WriteActiveIgnoreFile(data []byte) error {
if err := atomicWriteFile(c.GetActiveIgnoreFilePath(), data, 0o600); err != nil {
return fmt.Errorf("write active .nuclei-ignore file: %w", err)
}
return nil
}
// IgnoreFileNeedsUpdate reports whether the active ignore file is
// missing or differs from the latest known hash.
func (c *Config) IgnoreFileNeedsUpdate(latestHash string) bool {
ignoreFilePath := c.GetActiveIgnoreFilePath()
contents, err := os.ReadFile(ignoreFilePath)
if err != nil {
return true
}
if latestHash == "" {
return false
}
return ignoreFileHash(contents) != latestHash
}
// NeedsIgnoreFileUpdate reports whether the active ignore file differs
// from the deprecated in-memory remote hash.
//
// Deprecated: pass the transient remote hash to IgnoreFileNeedsUpdate.
func (c *Config) NeedsIgnoreFileUpdate() bool {
c.m.Lock()
latestHash := c.LatestNucleiIgnoreHash
c.m.Unlock()
return c.IgnoreFileNeedsUpdate(latestHash)
}
// UpdateNucleiIgnoreHash refreshes the deprecated in-memory compatibility
// field from the active ignore file.
//
// Deprecated: local ignore-file hashes are derived from the active file.
func (c *Config) UpdateNucleiIgnoreHash() error {
ignoreFilePath := c.GetActiveIgnoreFilePath()
contents, err := os.ReadFile(ignoreFilePath)
if err != nil {
return fmt.Errorf("read active .nuclei-ignore file %q: %w", ignoreFilePath, err)
}
c.NucleiIgnoreHash = ignoreFileHash(contents)
return nil
}
func ignoreFileHash(contents []byte) string {
return fmt.Sprintf("%x", md5.Sum(contents))
}
+54
View File
@@ -0,0 +1,54 @@
package config
import (
"os"
"path/filepath"
"testing"
)
func TestReadIgnoreFileAllowsMissingOrInvalidActiveFile(t *testing.T) {
root := t.TempDir()
cfg := DefaultConfig
oldRoot := cfg.TemplatesDirectory
t.Cleanup(func() { cfg.SetTemplatesDir(oldRoot) })
cfg.SetTemplatesDir(root)
if ignore := ReadIgnoreFile(); len(ignore.Tags) != 0 || len(ignore.Files) != 0 {
t.Fatalf("missing ignore file = %+v, want empty", ignore)
}
if err := os.WriteFile(filepath.Join(root, NucleiIgnoreFileName), []byte("tags: ["), 0o600); err != nil {
t.Fatalf("write invalid ignore file: %v", err)
}
if ignore := ReadIgnoreFile(); len(ignore.Tags) != 0 || len(ignore.Files) != 0 {
t.Fatalf("invalid ignore file = %+v, want empty", ignore)
}
if err := os.WriteFile(filepath.Join(root, NucleiIgnoreFileName), []byte("tags: [weak]\nfiles: [blocked.yaml]\n"), 0o600); err != nil {
t.Fatalf("write ignore file: %v", err)
}
ignore := ReadIgnoreFile()
if len(ignore.Tags) != 1 || ignore.Tags[0] != "weak" {
t.Fatalf("ignore tags = %v, want [weak]", ignore.Tags)
}
if len(ignore.Files) != 1 || ignore.Files[0] != "blocked.yaml" {
t.Fatalf("ignore files = %v, want [blocked.yaml]", ignore.Files)
}
}
func TestSetTemplatesDirDoesNotCopyLegacyIgnoreFile(t *testing.T) {
dir := t.TempDir()
configDir := filepath.Join(dir, "config")
legacyPath := filepath.Join(configDir, NucleiIgnoreFileName)
writeTestFile(t, legacyPath, []byte("tags: [legacy]\n"), 0o600)
root := filepath.Join(dir, "active")
if err := os.MkdirAll(root, 0o700); err != nil {
t.Fatalf("create active root: %v", err)
}
cfg := &Config{configDir: configDir}
cfg.SetTemplatesDir(root)
if _, err := os.Stat(cfg.GetActiveIgnoreFilePath()); !os.IsNotExist(err) {
t.Fatalf("legacy ignore file was copied into active root: %v", err)
}
}
+155 -132
View File
@@ -2,22 +2,20 @@ package config
import (
"bytes"
"crypto/md5"
"fmt"
"errors"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"github.com/adrg/xdg"
"github.com/projectdiscovery/gologger"
filepathutil "github.com/projectdiscovery/nuclei/v3/pkg/utils/filepath"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/json"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/yaml"
"github.com/projectdiscovery/utils/env"
"github.com/projectdiscovery/utils/errkit"
fileutil "github.com/projectdiscovery/utils/file"
folderutil "github.com/projectdiscovery/utils/folder"
)
// DefaultConfig is the default nuclei configuration
@@ -27,32 +25,44 @@ var DefaultConfig *Config
type Config struct {
TemplatesDirectory string `json:"nuclei-templates-directory,omitempty"`
// customtemplates exists in templates directory with the name of custom-templates provider
// below custom paths are absolute paths to respective custom-templates directories
// Custom template paths are absolute provider-specific directories under the
// active template root.
CustomS3TemplatesDirectory string `json:"custom-s3-templates-directory"`
CustomGitHubTemplatesDirectory string `json:"custom-github-templates-directory"`
CustomGitLabTemplatesDirectory string `json:"custom-gitlab-templates-directory"`
CustomAzureTemplatesDirectory string `json:"custom-azure-templates-directory"`
TemplateVersion string `json:"nuclei-templates-version,omitempty"`
NucleiIgnoreHash string `json:"nuclei-ignore-hash,omitempty"`
TemplateVersion string `json:"nuclei-templates-version,omitempty"`
// NucleiIgnoreHash is retained for SDK compatibility. Nuclei derives the
// current hash from the active ignore file and does not persist this field.
//
// Deprecated: local ignore-file hashes are derived from the active file.
NucleiIgnoreHash string `json:"-"`
LogAllEvents bool `json:"-"` // when enabled logs all events (more than verbose)
HideTemplateSigWarning bool `json:"-"` // when enabled disables template signature warning
// LatestXXX are not meant to be used directly and is used as
// local cache of nuclei version check endpoint
// these fields are only update during nuclei version check
// TODO: move these fields to a separate unexported struct as they are not meant to be used directly
LatestNucleiVersion string `json:"nuclei-latest-version"`
LatestNucleiTemplatesVersion string `json:"nuclei-templates-latest-version"`
LatestNucleiIgnoreHash string `json:"nuclei-latest-ignore-hash,omitempty"`
Logger *gologger.Logger `json:"-"` // logger
// These fields cache version-check results.
LatestNucleiVersion string `json:"nuclei-latest-version"`
LatestNucleiTemplatesVersion string `json:"nuclei-templates-latest-version"`
// LatestNucleiIgnoreHash is retained for SDK compatibility. Nuclei passes
// remote ignore hashes directly from version checks to the active-file
// comparison and does not persist this field.
//
// Deprecated: remote ignore hashes are transient version-check results.
LatestNucleiIgnoreHash string `json:"-"`
Logger *gologger.Logger `json:"-"` // logger
// internal / unexported fields
disableUpdates bool `json:"-"` // disable updates both version check and template updates
homeDir string `json:"-"` // User Home Directory
configDir string `json:"-"` // Nuclei Global Config Directory
debugArgs []string `json:"-"` // debug args
disableUpdates bool `json:"-"` // disable updates both version check and template updates
configDir string `json:"-"` // Nuclei Global Config Directory
stateDir string `json:"-"` // Restart-persistent operational state
cacheDir string `json:"-"` // Regenerable cached data
dataHome string `json:"-"` // User template data
dataDirs []string `json:"-"` // Template data, in preference order
debugArgs []string `json:"-"` // debug args
initializationErr error `json:"-"`
m sync.Mutex
}
@@ -76,37 +86,46 @@ func (c *Config) IsCustomTemplate(templatePath string) bool {
return true
}
}
if c.TemplatesDirectory == "" {
return false
}
return !filepathutil.IsPathWithinDirectory(templatePath, c.TemplatesDirectory)
}
// WriteVersionCheckData writes version check data to config file
func (c *Config) WriteVersionCheckData(ignorehash, nucleiVersion, templatesVersion string) error {
func (c *Config) WriteVersionCheckData(ignoreHash, nucleiVersion, templatesVersion string) error {
updated := false
if ignorehash != "" && c.LatestNucleiIgnoreHash != ignorehash {
c.LatestNucleiIgnoreHash = ignorehash
updated = true
if ignoreHash != "" && c.LatestNucleiIgnoreHash != ignoreHash {
// Retain the historical in-memory SDK behavior without persisting remote
// ignore metadata.
c.LatestNucleiIgnoreHash = ignoreHash
}
if nucleiVersion != "" && c.LatestNucleiVersion != nucleiVersion {
c.LatestNucleiVersion = nucleiVersion
updated = true
}
if templatesVersion != "" && c.LatestNucleiTemplatesVersion != templatesVersion {
c.LatestNucleiTemplatesVersion = templatesVersion
updated = true
}
// write config to disk if any of the fields are updated
if updated {
return c.WriteTemplatesConfig()
}
return nil
}
// GetTemplateDir returns the nuclei templates directory absolute path
func (c *Config) GetTemplateDir() string {
val, _ := filepath.Abs(c.TemplatesDirectory)
return val
}
@@ -114,6 +133,7 @@ func (c *Config) GetTemplateDir() string {
func (c *Config) DisableUpdateCheck() {
c.m.Lock()
defer c.m.Unlock()
c.disableUpdates = true
}
@@ -121,6 +141,7 @@ func (c *Config) DisableUpdateCheck() {
func (c *Config) CanCheckForUpdates() bool {
c.m.Lock()
defer c.m.Unlock()
return !c.disableUpdates
}
@@ -128,30 +149,9 @@ func (c *Config) CanCheckForUpdates() bool {
func (c *Config) NeedsTemplateUpdate() bool {
c.m.Lock()
defer c.m.Unlock()
return !c.disableUpdates && (c.TemplateVersion == "" || IsOutdatedVersion(c.TemplateVersion, c.LatestNucleiTemplatesVersion) || !fileutil.FolderExists(c.TemplatesDirectory))
}
// NeedsIgnoreFileUpdate returns true if Ignore file hash is different (aka ignore file is outdated)
func (c *Config) NeedsIgnoreFileUpdate() bool {
c.m.Lock()
defer c.m.Unlock()
return c.NucleiIgnoreHash == "" || c.NucleiIgnoreHash != c.LatestNucleiIgnoreHash
}
// UpdateNucleiIgnoreHash updates the nuclei ignore hash in config
func (c *Config) UpdateNucleiIgnoreHash() error {
// calculate hash of ignore file and update config
ignoreFilePath := c.GetIgnoreFilePath()
if fileutil.FileExists(ignoreFilePath) {
bin, err := os.ReadFile(ignoreFilePath)
if err != nil {
return errkit.Newf("could not read nuclei ignore file: %v", err)
}
c.NucleiIgnoreHash = fmt.Sprintf("%x", md5.Sum(bin))
// write config to disk
return c.WriteTemplatesConfig()
}
return errkit.New("ignore file not found: could not update nuclei ignore hash")
return !c.disableUpdates &&
(c.TemplateVersion == "" || IsOutdatedVersion(c.TemplateVersion, c.LatestNucleiTemplatesVersion) || !fileutil.FolderExists(c.TemplatesDirectory))
}
// GetConfigDir returns the nuclei configuration directory
@@ -159,6 +159,21 @@ func (c *Config) GetConfigDir() string {
return c.configDir
}
// GetStateDir returns the directory for restart-persistent operational state.
func (c *Config) GetStateDir() string {
if c.stateDir != "" {
return c.stateDir
}
return defaultStateDir()
}
// SetStateDir changes the templates state directory. It is primarily useful to
// SDK callers that isolate all Nuclei storage.
func (c *Config) SetStateDir(dir string) {
c.stateDir = dir
}
// GetKeysDir returns the nuclei signer keys directory
func (c *Config) GetKeysDir() string {
return filepath.Join(c.configDir, "keys")
@@ -175,10 +190,19 @@ func (c *Config) GetReportingConfigFilePath() string {
}
// GetIgnoreFilePath returns the nuclei ignore file path
//
// Deprecated: use GetActiveIgnoreFilePath. The config-directory ignore file is
// no longer active.
func (c *Config) GetIgnoreFilePath() string {
return filepath.Join(c.configDir, NucleiIgnoreFileName)
}
// GetActiveIgnoreFilePath returns the ignore file in the active public template
// root.
func (c *Config) GetActiveIgnoreFilePath() string {
return filepath.Join(c.TemplatesDirectory, NucleiIgnoreFileName)
}
func (c *Config) GetTemplateIndexFilePath() string {
return filepath.Join(c.TemplatesDirectory, NucleiTemplatesIndexFileName)
}
@@ -197,58 +221,75 @@ func (c *Config) GetFlagsConfigFilePath() string {
// if .new-additions file is not present empty slice is returned
func (c *Config) GetNewAdditions() []string {
arr := []string{}
newAdditionsPath := filepath.Join(c.TemplatesDirectory, NewTemplateAdditionsFileName)
if !fileutil.FileExists(newAdditionsPath) {
return arr
}
bin, err := os.ReadFile(newAdditionsPath)
if err != nil {
return arr
}
for v := range strings.FieldsSeq(string(bin)) {
if IsTemplateWithRoot(v, c.TemplatesDirectory) {
arr = append(arr, v)
}
}
return arr
}
// GetCacheDir returns the nuclei cache directory
// with new version of nuclei cache directory is changed
// instead of saving resume files in nuclei config directory
// they are saved in nuclei cache directory
// GetCacheDir returns the directory for regenerable cached data.
func (c *Config) GetCacheDir() string {
return folderutil.AppCacheDirOrDefault(".nuclei-cache", BinaryName)
if c.cacheDir != "" {
return c.cacheDir
}
return defaultCacheDir()
}
// SetConfigDir sets the nuclei configuration directory
// and appropriate changes are made to the config
func (c *Config) SetConfigDir(dir string) {
c.configDir = dir
// Preserve the historical SDK behavior that SetConfigDir also isolates
// templates state. NUCLEI_CONFIG_DIR is resolved separately and does not move
// state.
c.stateDir = dir
if err := c.createConfigDirIfNotExists(); err != nil {
c.Logger.Fatal().Msgf("Could not create nuclei config directory at %s: %s", c.configDir, err)
}
// if folder already exists read config or create new
if err := c.ReadTemplatesConfig(); err != nil {
if !errors.Is(err, os.ErrNotExist) {
c.Logger.Fatal().Msgf("Could not read templates state: %s", err)
}
// create new config
applyDefaultConfig()
if defaultsErr := c.applyDefaultConfig(); defaultsErr != nil {
c.Logger.Fatal().Msgf("Could not select default templates directory: %s", defaultsErr)
}
if err2 := c.WriteTemplatesConfig(); err2 != nil {
c.Logger.Fatal().Msgf("Could not create nuclei config file at %s: %s", c.getTemplatesConfigFilePath(), err2)
}
}
// while other config files are optional, ignore file is mandatory
// since it is used to ignore templates with weak matchers
c.copyIgnoreFile()
}
// SetTemplatesDir sets the new nuclei templates directory
func (c *Config) SetTemplatesDir(dirPath string) {
c.setTemplatesDir(dirPath)
}
func (c *Config) setTemplatesDir(dirPath string) {
if dirPath != "" && !filepath.IsAbs(dirPath) {
cwd, _ := os.Getwd()
dirPath = filepath.Join(cwd, dirPath)
}
c.TemplatesDirectory = dirPath
// Update the custom templates directory
c.CustomGitHubTemplatesDirectory = filepath.Join(dirPath, CustomGitHubTemplatesDirName)
@@ -260,64 +301,25 @@ func (c *Config) SetTemplatesDir(dirPath string) {
// SetTemplatesVersion sets the new nuclei templates version
func (c *Config) SetTemplatesVersion(version string) error {
c.TemplateVersion = version
// write config to disk
if err := c.WriteTemplatesConfig(); err != nil {
return errkit.Newf("could not write nuclei config file at %s: %v", c.getTemplatesConfigFilePath(), err)
}
return nil
}
// ReadTemplatesConfig reads the nuclei templates config file
func (c *Config) ReadTemplatesConfig() error {
if !fileutil.FileExists(c.getTemplatesConfigFilePath()) {
return errkit.Newf("nuclei config file at %s does not exist", c.getTemplatesConfigFilePath())
}
var cfg *Config
bin, err := os.ReadFile(c.getTemplatesConfigFilePath())
if err != nil {
return errkit.Newf("could not read nuclei config file at %s: %v", c.getTemplatesConfigFilePath(), err)
}
if err := json.Unmarshal(bin, &cfg); err != nil {
return errkit.Newf("could not unmarshal nuclei config file at %s: %v", c.getTemplatesConfigFilePath(), err)
}
// apply config
c.TemplatesDirectory = cfg.TemplatesDirectory
c.TemplateVersion = cfg.TemplateVersion
c.NucleiIgnoreHash = cfg.NucleiIgnoreHash
c.LatestNucleiIgnoreHash = cfg.LatestNucleiIgnoreHash
c.LatestNucleiTemplatesVersion = cfg.LatestNucleiTemplatesVersion
return nil
}
// WriteTemplatesConfig writes the nuclei templates config file
func (c *Config) WriteTemplatesConfig() error {
// check if config folder exists if not create one
if err := c.createConfigDirIfNotExists(); err != nil {
return err
}
bin, err := json.Marshal(c)
if err != nil {
return errkit.Newf("failed to marshal nuclei config: %v", err)
}
if err = os.WriteFile(c.getTemplatesConfigFilePath(), bin, 0600); err != nil {
return errkit.Newf("failed to write nuclei config file at %s: %v", c.getTemplatesConfigFilePath(), err)
}
return nil
}
// WriteTemplatesIndex writes the nuclei templates index file
func (c *Config) WriteTemplatesIndex(index map[string]string) error {
indexFile := c.GetTemplateIndexFilePath()
var buff bytes.Buffer
for k, v := range index {
_, _ = buff.WriteString(k + "," + v + "\n")
}
return os.WriteFile(indexFile, buff.Bytes(), 0600)
}
// getTemplatesConfigFilePath returns configDir/.templates-config.json file path
func (c *Config) getTemplatesConfigFilePath() string {
return filepath.Join(c.configDir, TemplateConfigFileName)
return os.WriteFile(indexFile, buff.Bytes(), 0600)
}
// createConfigDirIfNotExists creates the nuclei config directory if not exists
@@ -327,23 +329,8 @@ func (c *Config) createConfigDirIfNotExists() error {
return errkit.Newf("could not create nuclei config directory at %s: %v", c.configDir, err)
}
}
return nil
}
// copyIgnoreFile copies the nuclei ignore file default config directory
// to the current config directory
func (c *Config) copyIgnoreFile() {
if err := c.createConfigDirIfNotExists(); err != nil {
c.Logger.Error().Msgf("Could not create nuclei config directory at %s: %s", c.configDir, err)
return
}
ignoreFilePath := c.GetIgnoreFilePath()
if !fileutil.FileExists(ignoreFilePath) {
// copy ignore file from default config directory
if err := fileutil.CopyFile(filepath.Join(folderutil.AppConfigDirOrDefault(FallbackConfigFolderName, BinaryName), NucleiIgnoreFileName), ignoreFilePath); err != nil {
c.Logger.Error().Msgf("Could not copy nuclei ignore file at %s: %s", ignoreFilePath, err)
}
}
return nil
}
// IsDebugArgEnabled checks if debug arg is enabled
@@ -359,14 +346,17 @@ func (c *Config) parseDebugArgs(data string) {
for v := range strings.FieldsSeq(data) {
key := v
value := ""
// if it is key value pair then split it
if k, val, ok := strings.Cut(v, "="); ok {
key, value = strings.TrimSpace(k), strings.TrimSpace(val)
}
if value == "false" || value == "0" {
// if false or disabled then skip
continue
}
switch key {
case DebugArgHostErrorStats:
c.debugArgs = append(c.debugArgs, DebugArgHostErrorStats)
@@ -377,7 +367,7 @@ func (c *Config) parseDebugArgs(data string) {
}
func init() {
ConfigDir := folderutil.AppConfigDirOrDefault(FallbackConfigFolderName, BinaryName)
ConfigDir := defaultConfigDir()
if cfgDir := os.Getenv(NucleiConfigDirEnv); cfgDir != "" {
ConfigDir = cfgDir
@@ -389,9 +379,13 @@ func init() {
gologger.Error().Msgf("failed to create config directory at %v got: %s", ConfigDir, err)
}
}
DefaultConfig = &Config{
homeDir: folderutil.HomeDirOrDefault(""),
configDir: ConfigDir,
stateDir: defaultStateDir(),
cacheDir: defaultCacheDir(),
dataHome: xdg.DataHome,
dataDirs: append([]string(nil), xdg.DataDirs...),
Logger: gologger.DefaultLogger,
}
@@ -407,19 +401,24 @@ func init() {
// try to read config from file
if err := DefaultConfig.ReadTemplatesConfig(); err != nil {
gologger.Verbose().Msgf("config file not found, creating new config file at %s", DefaultConfig.getTemplatesConfigFilePath())
applyDefaultConfig()
// write config to file
if err := DefaultConfig.WriteTemplatesConfig(); err != nil {
gologger.Error().Msgf("failed to write config file at %s got: %s", DefaultConfig.getTemplatesConfigFilePath(), err)
if !errors.Is(err, os.ErrNotExist) {
DefaultConfig.initializationErr = err
gologger.Error().Msgf("Failed to read templates state: %s", err)
} else {
gologger.Verbose().Msgf("Templates state not found, creating %s", DefaultConfig.getTemplatesConfigFilePath())
}
if defaultsErr := DefaultConfig.applyDefaultConfig(); defaultsErr != nil {
DefaultConfig.initializationErr = errors.Join(DefaultConfig.initializationErr, defaultsErr)
gologger.Error().Msgf("Failed to select default templates directory: %s", defaultsErr)
} else if DefaultConfig.initializationErr == nil {
if writeErr := DefaultConfig.WriteTemplatesConfig(); writeErr != nil {
DefaultConfig.initializationErr = writeErr
gologger.Error().Msgf("Failed to write templates state at %s: %s", DefaultConfig.getTemplatesConfigFilePath(), writeErr)
}
}
}
// Loads/updates paths of custom templates
// Note: custom templates paths should not be updated in config file
// and even if it is changed we don't follow it since it is not expected behavior
// If custom templates are in default locations only then they are loaded while running nuclei
DefaultConfig.SetTemplatesDir(DefaultConfig.TemplatesDirectory)
DefaultConfig.parseDebugArgs(env.GetEnvOrDefault("NUCLEI_ARGS", ""))
// expose the templates directory to the yaml preprocessor so include
@@ -430,9 +429,33 @@ func init() {
}
}
// Add Default Config adds default when .templates-config.json file is not present
func applyDefaultConfig() {
DefaultConfig.TemplatesDirectory = filepath.Join(DefaultConfig.homeDir, NucleiTemplatesDirName)
// updates all necessary paths
DefaultConfig.SetTemplatesDir(DefaultConfig.TemplatesDirectory)
// InitializationError returns a templates state or path error detected during
// package initialization.
func (c *Config) InitializationError() error {
return c.initializationErr
}
func (c *Config) applyDefaultConfig() error {
dataHome := c.dataHome
if dataHome == "" {
dataHome = xdg.DataHome
}
dataDirs := c.dataDirs
if dataDirs == nil {
dataDirs = xdg.DataDirs
}
root, err := selectXDGTemplateRoot(dataHome, dataDirs)
if err != nil {
return err
}
rootChanged := filepath.Clean(c.TemplatesDirectory) != filepath.Clean(root)
c.setTemplatesDir(root)
if rootChanged {
c.TemplateVersion = ""
}
return nil
}
+83
View File
@@ -1,6 +1,10 @@
package config
import (
"crypto/md5"
"encoding/json"
"fmt"
"os"
"path/filepath"
"testing"
)
@@ -46,3 +50,82 @@ func TestIsCustomTemplateUsesPathBoundaries(t *testing.T) {
})
}
}
func TestTemplateRootLifecycleIgnoresLegacyDiscoverySource(t *testing.T) {
dir := t.TempDir()
root := filepath.Join(dir, "system", BinaryName, NucleiTemplatesDirName)
stateDir := filepath.Join(dir, "state")
state, err := json.Marshal(map[string]string{
"nuclei-templates-directory": root,
"nuclei-templates-directory-source": "xdg-system",
"nuclei-templates-version": "v1.0.0",
})
if err != nil {
t.Fatalf("marshal templates state: %v", err)
}
writeTestFile(t, filepath.Join(stateDir, TemplatesStateFileName), state, 0o600)
cfg := &Config{stateDir: stateDir, configDir: filepath.Join(dir, "config")}
if err := cfg.ReadTemplatesConfig(); err != nil {
t.Fatalf("read templates state: %v", err)
}
cfg.LatestNucleiTemplatesVersion = "v99.0.0"
if !cfg.NeedsTemplateUpdate() {
t.Fatal("discovered root did not request an available template update")
}
if !cfg.IgnoreFileNeedsUpdate("changed") {
t.Fatal("discovered root did not request a missing ignore file")
}
if err := cfg.WriteActiveIgnoreFile([]byte("tags: []\n")); err != nil {
t.Fatalf("write discovered-root ignore file: %v", err)
}
if err := cfg.WriteTemplatesIndex(map[string]string{"id": "path"}); err != nil {
t.Fatalf("write discovered-root template index: %v", err)
}
if err := cfg.SetTemplatesVersion("v2.0.0"); err != nil {
t.Fatalf("set discovered-root template version: %v", err)
}
for _, customDir := range cfg.GetAllCustomTemplateDirs() {
if filepath.Dir(customDir) != root {
t.Fatalf("custom directory %q is not under active root %q", customDir, root)
}
}
}
func TestNeedsIgnoreFileUpdateUsesActiveFile(t *testing.T) {
root := t.TempDir()
cfg := &Config{}
cfg.SetTemplatesDir(root)
contents := []byte("tags: [weak]\n")
latestHash := fmt.Sprintf("%x", md5.Sum(contents))
// The compatibility field must not hide a missing active file.
cfg.NucleiIgnoreHash = latestHash
if !cfg.IgnoreFileNeedsUpdate(latestHash) {
t.Fatal("missing active ignore file did not require an update")
}
if err := os.WriteFile(cfg.GetActiveIgnoreFilePath(), contents, 0o600); err != nil {
t.Fatalf("write active ignore file: %v", err)
}
// The compatibility field must not override the active file contents.
cfg.NucleiIgnoreHash = "stale"
if cfg.IgnoreFileNeedsUpdate(latestHash) {
t.Fatal("matching active ignore file required an update")
}
if err := os.WriteFile(cfg.GetActiveIgnoreFilePath(), []byte("tags: []\n"), 0o600); err != nil {
t.Fatalf("replace active ignore file: %v", err)
}
cfg.NucleiIgnoreHash = latestHash
if !cfg.IgnoreFileNeedsUpdate(latestHash) {
t.Fatal("modified active ignore file did not require an update")
}
if cfg.IgnoreFileNeedsUpdate("") {
t.Fatal("valid active ignore file required an update without a remote hash")
}
}
+73
View File
@@ -0,0 +1,73 @@
package config
import (
"errors"
"fmt"
"os"
"path/filepath"
"github.com/adrg/xdg"
)
func defaultConfigDir() string {
return filepath.Join(xdg.ConfigHome, BinaryName)
}
func defaultStateDir() string {
return filepath.Join(xdg.StateHome, BinaryName)
}
func defaultCacheDir() string {
return filepath.Join(xdg.CacheHome, BinaryName)
}
func selectXDGTemplateRoot(dataHome string, dataDirs []string) (string, error) {
if !filepath.IsAbs(dataHome) {
return "", fmt.Errorf("XDG data home %q is not absolute", dataHome)
}
userRoot := filepath.Join(dataHome, BinaryName, NucleiTemplatesDirName)
exists, err := directoryExists(userRoot)
if err != nil {
return "", err
}
if exists {
return userRoot, nil
}
for _, dataDir := range dataDirs {
if !filepath.IsAbs(dataDir) {
continue
}
root := filepath.Join(dataDir, BinaryName, NucleiTemplatesDirName)
exists, err := directoryExists(root)
if err != nil {
return "", err
}
if exists {
return root, nil
}
}
return userRoot, nil
}
func directoryExists(path string) (bool, error) {
info, err := os.Stat(path)
if errors.Is(err, os.ErrNotExist) {
return false, nil
}
if err != nil {
return false, fmt.Errorf("inspect directory %q: %w", path, err)
}
if !info.IsDir() {
return false, fmt.Errorf("path %q is not a directory", path)
}
return true, nil
}
+84
View File
@@ -0,0 +1,84 @@
package config
import (
"os"
"path/filepath"
"testing"
"github.com/adrg/xdg"
)
func TestXDGConfigPaths(t *testing.T) {
if got, want := defaultConfigDir(), filepath.Join(xdg.ConfigHome, BinaryName); got != want {
t.Fatalf("default config directory = %q, want %q", got, want)
}
if got, want := (&Config{}).GetCacheDir(), filepath.Join(xdg.CacheHome, BinaryName); got != want {
t.Fatalf("cache directory = %q, want %q", got, want)
}
if got, want := (&Config{}).GetStateDir(), filepath.Join(xdg.StateHome, BinaryName); got != want {
t.Fatalf("state directory = %q, want %q", got, want)
}
}
func TestSelectXDGTemplateRoot(t *testing.T) {
t.Run("user data wins", func(t *testing.T) {
dir := t.TempDir()
dataHome := filepath.Join(dir, "user")
userRoot := filepath.Join(dataHome, BinaryName, NucleiTemplatesDirName)
if err := os.MkdirAll(userRoot, 0o700); err != nil {
t.Fatalf("create user root: %v", err)
}
systemRoot := filepath.Join(dir, "system", BinaryName, NucleiTemplatesDirName)
if err := os.MkdirAll(systemRoot, 0o755); err != nil {
t.Fatalf("create system root: %v", err)
}
root, err := selectXDGTemplateRoot(dataHome, []string{filepath.Join(dir, "system")})
if err != nil {
t.Fatalf("select root: %v", err)
}
if root != userRoot {
t.Fatalf("root = %q, want %q", root, userRoot)
}
})
t.Run("first system directory wins", func(t *testing.T) {
dir := t.TempDir()
firstDir := filepath.Join(dir, "first")
secondDir := filepath.Join(dir, "second")
for _, base := range []string{firstDir, secondDir} {
if err := os.MkdirAll(filepath.Join(base, BinaryName, NucleiTemplatesDirName), 0o755); err != nil {
t.Fatalf("create system root: %v", err)
}
}
root, err := selectXDGTemplateRoot(filepath.Join(dir, "user"), []string{firstDir, secondDir})
if err != nil {
t.Fatalf("select root: %v", err)
}
want := filepath.Join(firstDir, BinaryName, NucleiTemplatesDirName)
if root != want {
t.Fatalf("root = %q, want %q", root, want)
}
})
t.Run("missing roots select user installation target", func(t *testing.T) {
dataHome := filepath.Join(t.TempDir(), "user")
root, err := selectXDGTemplateRoot(dataHome, nil)
if err != nil {
t.Fatalf("select root: %v", err)
}
want := filepath.Join(dataHome, BinaryName, NucleiTemplatesDirName)
if root != want {
t.Fatalf("root = %q, want %q", root, want)
}
})
t.Run("relative user data home is rejected", func(t *testing.T) {
if _, err := selectXDGTemplateRoot("relative", nil); err == nil {
t.Fatal("relative XDG data home was accepted")
}
})
}
+142
View File
@@ -0,0 +1,142 @@
package config
import (
"errors"
"fmt"
"os"
"path/filepath"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/json"
)
type templatesState struct {
TemplatesDirectory string `json:"nuclei-templates-directory,omitempty"`
TemplateVersion string `json:"nuclei-templates-version,omitempty"`
LatestNucleiVersion string `json:"nuclei-latest-version"`
LatestNucleiTemplatesVersion string `json:"nuclei-templates-latest-version"`
}
// ReadTemplatesConfig reads templates state, migrating the legacy config file
// only when the new state file does not exist.
func (c *Config) ReadTemplatesConfig() error {
statePath := c.getTemplatesConfigFilePath()
data, err := os.ReadFile(statePath)
migrate := false
if errors.Is(err, os.ErrNotExist) {
legacyPath := c.getLegacyTemplatesConfigFilePath()
data, err = os.ReadFile(legacyPath)
if err != nil {
return fmt.Errorf("read templates state %q or legacy state %q: %w", statePath, legacyPath, err)
}
migrate = true
} else if err != nil {
return fmt.Errorf("read templates state %q: %w", statePath, err)
}
var state templatesState
if err := json.Unmarshal(data, &state); err != nil {
return fmt.Errorf("decode templates state %q: %w", statePath, err)
}
if state.TemplatesDirectory == "" {
return fmt.Errorf("decode templates state %q: templates directory is empty", statePath)
}
c.setTemplatesDir(state.TemplatesDirectory)
c.TemplateVersion = state.TemplateVersion
c.LatestNucleiVersion = state.LatestNucleiVersion
c.LatestNucleiTemplatesVersion = state.LatestNucleiTemplatesVersion
if migrate {
if err := c.WriteTemplatesConfig(); err != nil {
return fmt.Errorf("migrate templates state: %w", err)
}
}
return nil
}
// WriteTemplatesConfig atomically writes restart-persistent templates state.
func (c *Config) WriteTemplatesConfig() error {
state := templatesState{
TemplatesDirectory: c.TemplatesDirectory,
TemplateVersion: c.TemplateVersion,
LatestNucleiVersion: c.LatestNucleiVersion,
LatestNucleiTemplatesVersion: c.LatestNucleiTemplatesVersion,
}
data, err := json.Marshal(&state)
if err != nil {
return fmt.Errorf("encode templates state: %w", err)
}
if err := atomicWriteFile(c.getTemplatesConfigFilePath(), data, 0o600); err != nil {
return fmt.Errorf("write templates state: %w", err)
}
return nil
}
func (c *Config) getTemplatesConfigFilePath() string {
return filepath.Join(c.GetStateDir(), TemplatesStateFileName)
}
// GetTemplatesStateFilePath returns the restart-persistent templates state
// file path.
func (c *Config) GetTemplatesStateFilePath() string {
return c.getTemplatesConfigFilePath()
}
// TODO(dwisiswant0): remove this in the future, only used for legacy migration.
func (c *Config) getLegacyTemplatesConfigFilePath() string {
return filepath.Join(c.configDir, legacyTemplatesConfigFileName)
}
func atomicWriteFile(path string, data []byte, mode os.FileMode) error {
dir := filepath.Dir(path)
if err := os.MkdirAll(dir, 0o700); err != nil {
return fmt.Errorf("create directory %q: %w", dir, err)
}
temp, err := os.CreateTemp(dir, "."+filepath.Base(path)+"-*")
if err != nil {
return fmt.Errorf("create temporary file for %q: %w", path, err)
}
tempPath := temp.Name()
defer func() {
if temp != nil {
_ = temp.Close()
}
_ = os.Remove(tempPath)
}()
if err := temp.Chmod(mode); err != nil {
return fmt.Errorf("set temporary file mode for %q: %w", path, err)
}
if _, err := temp.Write(data); err != nil {
return fmt.Errorf("write temporary file for %q: %w", path, err)
}
if err := temp.Sync(); err != nil {
return fmt.Errorf("sync temporary file for %q: %w", path, err)
}
if err := temp.Close(); err != nil {
return fmt.Errorf("close temporary file for %q: %w", path, err)
}
temp = nil
if err := os.Rename(tempPath, path); err != nil {
return fmt.Errorf("replace %q: %w", path, err)
}
return nil
}
+141
View File
@@ -0,0 +1,141 @@
package config
import (
"encoding/json"
"errors"
"os"
"path/filepath"
"runtime"
"strings"
"testing"
)
func TestReadTemplatesConfigMigratesLegacyState(t *testing.T) {
dir := t.TempDir()
configDir := filepath.Join(dir, "config")
stateDir := filepath.Join(dir, "state")
// Keep a backslash in the path so JSON escaping is exercised on all platforms.
legacyRoot := filepath.Join(dir, `legacy\templates`)
legacyState, err := json.Marshal(map[string]string{
"nuclei-templates-directory": legacyRoot,
"nuclei-templates-version": "v1.2.3",
"nuclei-ignore-hash": "legacy-local-hash",
"nuclei-latest-ignore-hash": "legacy-remote-hash",
})
if err != nil {
t.Fatalf("marshal legacy state: %v", err)
}
writeTestFile(t, filepath.Join(configDir, legacyTemplatesConfigFileName), legacyState, 0o600)
cfg := &Config{configDir: configDir, stateDir: stateDir}
if err := cfg.ReadTemplatesConfig(); err != nil {
t.Fatalf("read templates state: %v", err)
}
if cfg.TemplatesDirectory != legacyRoot {
t.Fatalf("root = %q, want %q", cfg.TemplatesDirectory, legacyRoot)
}
if cfg.TemplateVersion != "v1.2.3" {
t.Fatalf("template version = %q, want v1.2.3", cfg.TemplateVersion)
}
if cfg.NucleiIgnoreHash != "" {
t.Fatalf("local ignore hash = %q, want empty", cfg.NucleiIgnoreHash)
}
if cfg.LatestNucleiIgnoreHash != "" {
t.Fatalf("remote ignore hash = %q, want empty", cfg.LatestNucleiIgnoreHash)
}
cfg.NucleiIgnoreHash = "current-local-hash"
cfg.LatestNucleiIgnoreHash = "current-remote-hash"
if err := cfg.WriteTemplatesConfig(); err != nil {
t.Fatalf("rewrite templates state: %v", err)
}
statePath := filepath.Join(stateDir, TemplatesStateFileName)
info, err := os.Stat(statePath)
if err != nil {
t.Fatalf("stat migrated state: %v", err)
}
if got := info.Mode().Perm(); runtime.GOOS != "windows" && got != 0o600 {
t.Fatalf("state mode = %o, want 600", got)
}
state, err := os.ReadFile(statePath)
if err != nil {
t.Fatalf("read migrated state: %v", err)
}
if strings.Contains(string(state), "nuclei-ignore-hash") {
t.Fatalf("migrated state persisted local ignore hash: %s", state)
}
if strings.Contains(string(state), "nuclei-latest-ignore-hash") {
t.Fatalf("migrated state persisted remote ignore hash: %s", state)
}
if _, err := os.Stat(filepath.Join(configDir, legacyTemplatesConfigFileName)); err != nil {
t.Fatalf("legacy state was removed: %v", err)
}
}
func TestReadTemplatesConfigDoesNotHideCorruptState(t *testing.T) {
dir := t.TempDir()
configDir := filepath.Join(dir, "config")
stateDir := filepath.Join(dir, "state")
writeTestFile(t, filepath.Join(configDir, legacyTemplatesConfigFileName), []byte(`{"nuclei-templates-directory":"legacy"}`), 0o600)
writeTestFile(t, filepath.Join(stateDir, TemplatesStateFileName), []byte(`{"nuclei-templates-directory":`), 0o600)
cfg := &Config{configDir: configDir, stateDir: stateDir}
err := cfg.ReadTemplatesConfig()
if err == nil {
t.Fatal("expected corrupt new state to fail")
}
if errors.Is(err, os.ErrNotExist) {
t.Fatalf("corrupt state reported as missing: %v", err)
}
if cfg.TemplatesDirectory == "legacy" {
t.Fatal("corrupt new state fell back to legacy state")
}
}
func TestReadTemplatesConfigIgnoresLegacyRootSource(t *testing.T) {
dir := t.TempDir()
stateDir := filepath.Join(dir, "state")
root := filepath.Join(dir, "system", BinaryName, NucleiTemplatesDirName)
state, err := json.Marshal(map[string]string{
"nuclei-templates-directory": root,
"nuclei-templates-directory-source": "xdg-system",
"nuclei-templates-version": "v1.2.3",
})
if err != nil {
t.Fatalf("marshal templates state: %v", err)
}
statePath := filepath.Join(stateDir, TemplatesStateFileName)
writeTestFile(t, statePath, state, 0o600)
cfg := &Config{stateDir: stateDir, configDir: filepath.Join(dir, "config")}
if err := cfg.ReadTemplatesConfig(); err != nil {
t.Fatalf("read templates state: %v", err)
}
if cfg.TemplatesDirectory != root {
t.Fatalf("root = %q, want %q", cfg.TemplatesDirectory, root)
}
if cfg.TemplateVersion != "v1.2.3" {
t.Fatalf("template version = %q, want v1.2.3", cfg.TemplateVersion)
}
if err := cfg.WriteTemplatesConfig(); err != nil {
t.Fatalf("rewrite templates state: %v", err)
}
rewritten, err := os.ReadFile(statePath)
if err != nil {
t.Fatalf("read rewritten templates state: %v", err)
}
if strings.Contains(string(rewritten), "nuclei-templates-directory-source") {
t.Fatalf("rewritten state retained legacy root source: %s", rewritten)
}
}
func writeTestFile(t *testing.T, path string, data []byte, mode os.FileMode) {
t.Helper()
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
t.Fatalf("create test directory: %v", err)
}
if err := os.WriteFile(path, data, mode); err != nil {
t.Fatalf("write test file: %v", err)
}
}
+9 -4
View File
@@ -11,7 +11,6 @@ import (
"github.com/maypok86/otter/v2"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/templates"
folderutil "github.com/projectdiscovery/utils/folder"
)
const (
@@ -47,10 +46,10 @@ type cacheSnapshot struct {
// NewIndex creates a new template metadata cache with the given options.
func NewIndex(cacheDir string) (*Index, error) {
if cacheDir == "" {
cacheDir = folderutil.AppCacheDirOrDefault(".nuclei-cache", config.BinaryName)
cacheDir = config.DefaultConfig.GetCacheDir()
}
if err := os.MkdirAll(cacheDir, 0755); err != nil {
if err := os.MkdirAll(cacheDir, 0o700); err != nil {
return nil, err
}
@@ -226,10 +225,16 @@ func (i *Index) Save() error {
// NOTE(dwisiswant0): write to temp for atomic op.
tmpFile := i.cacheFile + ".tmp"
file, err := os.Create(tmpFile)
file, err := os.OpenFile(tmpFile, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
if err != nil {
return err
}
if err := file.Chmod(0o600); err != nil {
_ = file.Close()
_ = os.Remove(tmpFile)
return err
}
encoder := gob.NewEncoder(file)
if err := encoder.Encode(snapshot); err != nil {
+28
View File
@@ -4,9 +4,12 @@ import (
"fmt"
"os"
"path/filepath"
"runtime"
"testing"
"time"
"github.com/adrg/xdg"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/model"
"github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
"github.com/projectdiscovery/nuclei/v3/pkg/model/types/stringslice"
@@ -29,9 +32,31 @@ func TestNewIndex(t *testing.T) {
})
t.Run("with default directory", func(t *testing.T) {
oldConfig := config.DefaultConfig
oldCacheHome := xdg.CacheHome
cacheHome := t.TempDir()
xdg.CacheHome = cacheHome
config.DefaultConfig = &config.Config{}
t.Cleanup(func() {
config.DefaultConfig = oldConfig
xdg.CacheHome = oldCacheHome
})
cache, err := NewDefaultIndex()
require.NoError(t, err, "Failed to create cache with default directory")
require.NotNil(t, cache, "Cache should not be nil")
require.Equal(t, filepath.Join(cacheHome, config.BinaryName, IndexFileName), cache.cacheFile)
})
t.Run("creates private directory", func(t *testing.T) {
cacheDir := filepath.Join(t.TempDir(), "cache")
_, err := NewIndex(cacheDir)
require.NoError(t, err)
info, err := os.Stat(cacheDir)
require.NoError(t, err)
if runtime.GOOS != "windows" {
require.Equal(t, os.FileMode(0o700), info.Mode().Perm())
}
})
}
@@ -125,6 +150,9 @@ func TestCachePersistence(t *testing.T) {
stat, err := os.Stat(cacheFile)
require.NoError(t, err, "Cache file should exist")
require.Greater(t, stat.Size(), int64(0), "Cache file should not be empty")
if runtime.GOOS != "windows" {
require.Equal(t, os.FileMode(0o600), stat.Mode().Perm())
}
// Create new cache and load
cache2, err := NewIndex(tmpDir)
+37
View File
@@ -27,3 +27,40 @@ func NewInputProvider(opts InputOptions) (InputProvider, error)
This function returns a InputProvider based by appropriately selecting input provider based on the input format (i.e. either list or http) and returns the provider that can handle that input format.
## Formats
Fuzzing and DAST take the request shape (method, body, params) from the input, not from CLI flags, so a bare URL is always fuzzed as a `GET`. To fuzz anything else, feed a request-shaped input with `-im`:
| `-im` | Input |
| --- | --- |
| `list` | urls, domains, ips, cidrs (default) |
| `http` | raw HTTP requests |
| `burp` | Burp Suite xml export |
| `jsonl` | proxify jsonl output |
| `yaml` | proxify yaml multidoc output |
| `openapi` | OpenAPI 3 spec |
| `swagger` | Swagger 2 spec |
### Raw HTTP requests (`-im http`)
The shortest path from a single endpoint to a DAST scan, for targets with no spec or captured traffic. Save the request as it appears on the wire, which is what "copy as raw request" in Burp and browser devtools produce:
```
POST /api/login HTTP/1.1
Host: example.com
Content-Type: application/json
{"user":"admin","pass":"secret"}
```
```console
nuclei -l login.http -im http -dast -t fuzzing-templates/
```
Notes:
- Separate multiple requests with a line starting with `###`, the `.http` file convention.
- The scheme is not part of the request, so it is inferred from the authority. Use an absolute request target (`POST https://example.com/api/login HTTP/1.1`) to state it explicitly.
- The target comes from the `Host` header or from an absolute request target; a request carrying neither is skipped.
+16 -7
View File
@@ -1,14 +1,12 @@
package burp
import (
"encoding/base64"
"io"
"strings"
"github.com/pkg/errors"
"github.com/projectdiscovery/nuclei/v3/pkg/input/formats"
"github.com/projectdiscovery/nuclei/v3/pkg/input/types"
"github.com/projectdiscovery/utils/conversion"
burpxml "github.com/projectdiscovery/utils/parsers/burp/xml"
)
@@ -42,17 +40,28 @@ func (j *BurpFormat) Parse(input io.Reader, resultsCb formats.ParseReqRespCallba
}
for _, item := range items.Items {
binx, err := base64.StdEncoding.DecodeString(item.Request.Raw)
if err != nil {
return errors.Wrap(err, "could not decode base64")
// Prefer decoded Body from burpxml; fall back to Raw. Do not TrimSpace the
// payload — trailing blank lines mark end-of-headers for HTTP parsers.
reqRaw := item.Request.Body
if strings.TrimSpace(reqRaw) == "" {
reqRaw = item.Request.Raw
}
if strings.TrimSpace(conversion.String(binx)) == "" {
if strings.TrimSpace(reqRaw) == "" {
continue
}
rawRequest, err := types.ParseRawRequestWithURL(conversion.String(binx), item.URL)
rawRequest, err := types.ParseRawRequestWithURL(reqRaw, item.URL)
if err != nil {
return errors.Wrap(err, "could not parse raw request")
}
respRaw := item.Response.Body
if strings.TrimSpace(respRaw) == "" {
respRaw = item.Response.Raw
}
if strings.TrimSpace(respRaw) != "" {
rawRequest.Response = &types.HttpResponse{Raw: respRaw}
}
resultsCb(rawRequest)
}
return nil
+26
View File
@@ -14,6 +14,7 @@ func TestBurpParse(t *testing.T) {
proxifyInputFile := "../testdata/burp.xml"
var gotMethodsToURLs []string
var gotResponses int
file, err := os.Open(proxifyInputFile)
require.Nilf(t, err, "error opening proxify input file: %v", err)
@@ -23,6 +24,10 @@ func TestBurpParse(t *testing.T) {
err = format.Parse(file, func(request *types.RequestResponse) bool {
gotMethodsToURLs = append(gotMethodsToURLs, request.URL.String())
if request.Response != nil && request.Response.Raw != "" {
gotResponses++
require.Contains(t, request.Response.Raw, "HTTP/")
}
return false
}, proxifyInputFile)
if err != nil {
@@ -37,4 +42,25 @@ func TestBurpParse(t *testing.T) {
"http://google.com/",
}
require.ElementsMatch(t, expectedURLs, gotMethodsToURLs, "could not get burp urls")
require.Equal(t, 2, gotResponses, "burp items should include response bodies for passive mode")
}
func TestBurpParseResponseBodyContent(t *testing.T) {
format := New()
file, err := os.Open("../testdata/burp.xml")
require.NoError(t, err)
defer func() { _ = file.Close() }()
var first *types.RequestResponse
err = format.Parse(file, func(request *types.RequestResponse) bool {
if first == nil {
first = request
}
return false
}, "../testdata/burp.xml")
require.NoError(t, err)
require.NotNil(t, first)
require.NotNil(t, first.Response)
require.Contains(t, first.Response.Raw, `"id":"1"`)
require.Contains(t, first.Request.Raw, "POST /scans")
}
+29
View File
@@ -2,6 +2,7 @@ package json
import (
"io"
"strings"
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
@@ -32,6 +33,13 @@ type proxifyRequest struct {
Raw string `json:"raw"`
Endpoint string `json:"endpoint"`
} `json:"request"`
Response *proxifyResponse `json:"response,omitempty"`
}
type proxifyResponse struct {
Header map[string]string `json:"header"`
Body string `json:"body"`
Raw string `json:"raw"`
}
// Name returns the name of the format
@@ -65,7 +73,28 @@ func (j *JSONFormat) Parse(input io.Reader, resultsCb formats.ParseReqRespCallba
gologger.Warning().Msgf("jsonl: Could not parse raw request %s: %s\n", request.URL, err)
continue
}
if resp := buildProxifyResponse(request.Response); resp != nil {
rawRequest.Response = resp
}
resultsCb(rawRequest)
}
return nil
}
func buildProxifyResponse(resp *proxifyResponse) *types.HttpResponse {
if resp == nil {
return nil
}
raw := resp.Raw
if raw == "" {
return nil
}
// Proxify often stores headers in raw and the body separately (e.g. gzip).
if resp.Body != "" && (strings.HasSuffix(raw, "\r\n\r\n") || strings.HasSuffix(raw, "\n\n")) {
raw += resp.Body
}
return &types.HttpResponse{
Raw: raw,
Body: resp.Body,
}
}
+16
View File
@@ -49,8 +49,13 @@ func TestJSONFormatterParse(t *testing.T) {
}()
var urls []string
var withResponse int
err = format.Parse(file, func(request *types.RequestResponse) bool {
urls = append(urls, request.URL.String())
if request.Response != nil && request.Response.Raw != "" {
withResponse++
require.Contains(t, request.Response.Raw, "HTTP/")
}
return false
}, proxifyInputFile)
if err != nil {
@@ -61,4 +66,15 @@ func TestJSONFormatterParse(t *testing.T) {
t.Fatalf("invalid number of urls: %d", len(urls))
}
require.ElementsMatch(t, urls, expectedURLs)
require.Equal(t, len(expectedURLs), withResponse, "jsonl items should include response bodies for passive mode")
}
func TestBuildProxifyResponseAppendsBody(t *testing.T) {
resp := buildProxifyResponse(&proxifyResponse{
Raw: "HTTP/1.1 200 OK\r\nContent-Length: 5\r\n\r\n",
Body: "hello",
})
require.NotNil(t, resp)
require.Equal(t, "HTTP/1.1 200 OK\r\nContent-Length: 5\r\n\r\nhello", resp.Raw)
require.Equal(t, "hello", resp.Body)
}
+99
View File
@@ -0,0 +1,99 @@
// Package raw parses input files holding raw HTTP requests, the shape produced
// by "copy as raw request" in Burp and browser devtools and stored in .http
// files. It is the shortest path from a single request to a DAST scan, for
// targets that have no OpenAPI spec or captured traffic to feed in.
package raw
import (
"io"
"strings"
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/input/formats"
"github.com/projectdiscovery/nuclei/v3/pkg/input/types"
"github.com/projectdiscovery/nuclei/v3/pkg/utils"
)
// requestSeparator delimits requests when a file carries more than one,
// following the .http file convention
const requestSeparator = "###"
// RawFormat is a parser for files containing one or more raw HTTP requests
type RawFormat struct {
opts formats.InputFormatOptions
}
// New creates a new raw HTTP request parser
func New() *RawFormat {
return &RawFormat{}
}
var _ formats.Format = &RawFormat{}
// Name returns the name of the format
func (r *RawFormat) Name() string {
return "http"
}
func (r *RawFormat) SetOptions(options formats.InputFormatOptions) {
r.opts = options
}
// Parse parses the input and calls the provided callback
// function for each RawRequest it discovers.
func (r *RawFormat) Parse(input io.Reader, resultsCb formats.ParseReqRespCallback, filePath string) error {
data, err := io.ReadAll(input)
if err != nil {
return errors.Wrap(err, "could not read raw request file")
}
for _, request := range splitRequests(string(data)) {
reqResp, err := types.ParseRawRequest(terminateHeaders(request))
if err != nil {
gologger.Warning().Msgf("http: could not parse raw request in %s: %s\n", filePath, err)
continue
}
if reqResp.URL.Host == "" {
gologger.Warning().Msgf("http: skipped raw request in %s: no target, add a Host header or an absolute request target\n", filePath)
continue
}
if reqResp.URL.Scheme == "" {
// nothing in the request states the scheme, so settle it the same
// way nuclei settles it for scheme-less list inputs
reqResp.URL.Scheme = utils.DetermineSchemeOrder(reqResp.URL.Host)[0]
}
resultsCb(reqResp)
}
return nil
}
// splitRequests splits the file contents into individual raw requests
func splitRequests(data string) []string {
var requests []string
var current []string
flush := func() {
if request := strings.Join(current, "\n"); strings.TrimSpace(request) != "" {
requests = append(requests, strings.TrimLeft(request, "\r\n"))
}
current = nil
}
for _, line := range strings.Split(data, "\n") {
if strings.HasPrefix(strings.TrimSpace(line), requestSeparator) {
flush()
continue
}
current = append(current, line)
}
flush()
return requests
}
// terminateHeaders appends the blank line that separates headers from the body.
// Requests copied out of a browser or Burp routinely end at the last header,
// and the parser needs that terminator to know where the headers stop.
func terminateHeaders(request string) string {
if strings.Contains(request, "\n\r\n") || strings.Contains(request, "\n\n") {
return request
}
return request + "\r\n\r\n"
}
+119
View File
@@ -0,0 +1,119 @@
package raw
import (
"strings"
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/input/formats"
"github.com/projectdiscovery/nuclei/v3/pkg/input/types"
"github.com/stretchr/testify/require"
)
func parseAll(t *testing.T, data string) []*types.RequestResponse {
t.Helper()
var got []*types.RequestResponse
err := New().Parse(strings.NewReader(data), func(rr *types.RequestResponse) bool {
got = append(got, rr)
return false
}, "test.http")
require.NoError(t, err)
return got
}
func TestParseSingleRequest(t *testing.T) {
got := parseAll(t, "POST /api/login HTTP/1.1\r\nHost: example.com\r\nContent-Type: application/json\r\n\r\n{\"user\":\"admin\"}")
require.Len(t, got, 1)
require.Equal(t, "POST", got[0].Request.Method)
require.Equal(t, "https://example.com/api/login", got[0].URL.String())
require.Equal(t, `{"user":"admin"}`, got[0].Request.Body)
contentType, _ := got[0].Request.Headers.Get("Content-Type")
require.Equal(t, "application/json", contentType)
}
func TestParseMultipleRequests(t *testing.T) {
got := parseAll(t, strings.Join([]string{
"GET /one HTTP/1.1",
"Host: example.com",
"",
"### second request",
"POST /two HTTP/1.1",
"Host: example.com",
"",
"body=1",
}, "\n"))
require.Len(t, got, 2)
require.Equal(t, "https://example.com/one", got[0].URL.String())
require.Equal(t, "https://example.com/two", got[1].URL.String())
require.Equal(t, "body=1", got[1].Request.Body)
}
// A request pasted out of a browser or Burp commonly ends at the last header.
func TestParseWithoutTrailingBlankLine(t *testing.T) {
got := parseAll(t, "GET /api/users?id=1 HTTP/1.1\r\nHost: example.com")
require.Len(t, got, 1)
require.Equal(t, "https://example.com/api/users?id=1", got[0].URL.String())
require.Empty(t, got[0].Request.Body)
}
// The scheme is not part of an origin-form request, so it is inferred from the
// authority unless the request target states it.
func TestParseResolvesScheme(t *testing.T) {
tests := []struct {
name string
raw string
url string
}{
{"no port", "GET / HTTP/1.1\r\nHost: example.com\r\n\r\n", "https://example.com/"},
{"port 80", "GET / HTTP/1.1\r\nHost: example.com:80\r\n\r\n", "http://example.com:80/"},
{"high port", "GET / HTTP/1.1\r\nHost: 127.0.0.1:18080\r\n\r\n", "http://127.0.0.1:18080/"},
{"absolute target", "GET https://example.com/p HTTP/1.1\r\nHost: example.com\r\n\r\n", "https://example.com/p"},
{"absolute plain target", "GET http://example.com/p HTTP/1.1\r\nHost: example.com\r\n\r\n", "http://example.com/p"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := parseAll(t, tt.raw)
require.Len(t, got, 1)
require.Equal(t, tt.url, got[0].URL.String())
})
}
}
// A request with no way to resolve a target is dropped rather than scanned
// against nothing, and it must not take the rest of the file down with it.
func TestParseSkipsRequestsWithoutTarget(t *testing.T) {
got := parseAll(t, strings.Join([]string{
"GET /no-host HTTP/1.1",
"X-Test: 1",
"",
"###",
"GET /valid HTTP/1.1",
"Host: example.com",
"",
}, "\n"))
require.Len(t, got, 1)
require.Equal(t, "https://example.com/valid", got[0].URL.String())
}
// The callback's return value is unused across every input format, and its two
// call sites in the provider give it opposite meanings, so a request must be
// reported regardless of what the previous one returned.
func TestParseIgnoresCallbackReturn(t *testing.T) {
var count int
err := New().Parse(strings.NewReader("GET /one HTTP/1.1\nHost: example.com\n\n###\nGET /two HTTP/1.1\nHost: example.com\n\n"), func(rr *types.RequestResponse) bool {
count++
return true
}, "test.http")
require.NoError(t, err)
require.Equal(t, 2, count)
}
func TestFormatInterface(t *testing.T) {
var format formats.Format = New()
require.Equal(t, "http", format.Name())
}
+33 -5
View File
@@ -3,6 +3,7 @@ package yaml
import (
"bytes"
"io"
"strings"
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
@@ -26,12 +27,19 @@ var _ formats.Format = &YamlMultiDocFormat{}
// proxifyRequest is a request for proxify
type proxifyRequest struct {
URL string `json:"url"`
URL string `json:"url" yaml:"url"`
Request struct {
Header map[string]string `json:"header"`
Body string `json:"body"`
Raw string `json:"raw"`
} `json:"request"`
Header map[string]string `json:"header" yaml:"header"`
Body string `json:"body" yaml:"body"`
Raw string `json:"raw" yaml:"raw"`
} `json:"request" yaml:"request"`
Response *proxifyResponse `json:"response,omitempty" yaml:"response,omitempty"`
}
type proxifyResponse struct {
Header map[string]string `json:"header" yaml:"header"`
Body string `json:"body" yaml:"body"`
Raw string `json:"raw" yaml:"raw"`
}
// Name returns the name of the format
@@ -83,7 +91,27 @@ func (j *YamlMultiDocFormat) Parse(input io.Reader, resultsCb formats.ParseReqRe
gologger.Warning().Msgf("multidoc-yaml: Could not parse raw request %s: %s", request.URL, err)
continue
}
if resp := buildProxifyResponse(request.Response); resp != nil {
rawRequest.Response = resp
}
resultsCb(rawRequest)
}
return nil
}
func buildProxifyResponse(resp *proxifyResponse) *types.HttpResponse {
if resp == nil {
return nil
}
raw := resp.Raw
if raw == "" {
return nil
}
if resp.Body != "" && (strings.HasSuffix(raw, "\r\n\r\n") || strings.HasSuffix(raw, "\n\n")) {
raw += resp.Body
}
return &types.HttpResponse{
Raw: raw,
Body: resp.Body,
}
}
+16
View File
@@ -27,13 +27,29 @@ func TestYamlFormatterParse(t *testing.T) {
}()
var urls []string
var withResponse int
err = format.Parse(file, func(request *types.RequestResponse) bool {
urls = append(urls, request.URL.String())
if request.Response != nil && request.Response.Raw != "" {
withResponse++
require.Contains(t, request.Response.Raw, "HTTP/")
}
return false
}, proxifyInputFile)
require.Nilf(t, err, "error parsing yaml file: %v", err)
require.Len(t, urls, len(expectedUrls), "invalid number of urls")
require.ElementsMatch(t, urls, expectedUrls, "invalid urls")
require.Equal(t, len(expectedUrls), withResponse, "yaml items should include response bodies for passive mode")
}
func TestBuildProxifyResponseAppendsBody(t *testing.T) {
resp := buildProxifyResponse(&proxifyResponse{
Raw: "HTTP/1.1 200 OK\r\nContent-Length: 5\r\n\r\n",
Body: "hello",
})
require.NotNil(t, resp)
require.Equal(t, "HTTP/1.1 200 OK\r\nContent-Length: 5\r\n\r\nhello", resp.Raw)
require.Equal(t, "hello", resp.Body)
}
func TestYamlFormatterParseWithVariables(t *testing.T) {
+2
View File
@@ -12,6 +12,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/input/formats/burp"
"github.com/projectdiscovery/nuclei/v3/pkg/input/formats/json"
"github.com/projectdiscovery/nuclei/v3/pkg/input/formats/openapi"
"github.com/projectdiscovery/nuclei/v3/pkg/input/formats/raw"
"github.com/projectdiscovery/nuclei/v3/pkg/input/formats/swagger"
"github.com/projectdiscovery/nuclei/v3/pkg/input/formats/yaml"
"github.com/projectdiscovery/nuclei/v3/pkg/input/types"
@@ -145,6 +146,7 @@ var providersList = []formats.Format{
yaml.New(),
openapi.New(),
swagger.New(),
raw.New(),
}
// SupportedFormats returns the list of supported formats in comma-separated
+30 -17
View File
@@ -15,6 +15,7 @@ import (
"github.com/projectdiscovery/useragent"
"github.com/projectdiscovery/utils/conversion"
mapsutil "github.com/projectdiscovery/utils/maps"
stringsutil "github.com/projectdiscovery/utils/strings"
urlutil "github.com/projectdiscovery/utils/url"
)
@@ -239,25 +240,25 @@ func ParseRawRequest(raw string) (rr *RequestResponse, err error) {
method := parts[0]
rr.Request.Method = method
// parse relative url
urlx, err := urlutil.ParseRawRelativePath(parts[1], true)
// the request target is normally an origin-form path, but proxy captures and
// .http files use the absolute form, which already carries the authority
var urlx *urlutil.URL
target := parts[1]
if stringsutil.HasPrefixAnyI(target, urlutil.HTTP+urlutil.SchemeSeparator, urlutil.HTTPS+urlutil.SchemeSeparator) {
// urlutil.ParseAbsoluteURL only accepts lowercase schemes; preserve the
// remainder of the request target unchanged.
if scheme, rest, ok := strings.Cut(target, urlutil.SchemeSeparator); ok {
target = strings.ToLower(scheme) + urlutil.SchemeSeparator + rest
}
urlx, err = urlutil.ParseAbsoluteURL(target, true)
} else {
urlx, err = urlutil.ParseRawRelativePath(target, true)
}
if err != nil {
return nil, fmt.Errorf("failed to parse url: %s", err)
}
rr.URL = *urlx
// parse host line
hostLine, err := protoReader.ReadLine()
if err != nil {
return nil, fmt.Errorf("failed to read host line: %s", err)
}
sep := strings.Index(hostLine, ":")
if sep <= 0 || sep >= len(hostLine)-1 {
return nil, fmt.Errorf("invalid host line: %s", hostLine)
}
hostLine = hostLine[sep+2:]
rr.URL.Host = hostLine
// parse headers
rr.Request.Headers = mapsutil.NewOrderedMap[string, string]()
for {
@@ -269,11 +270,23 @@ func ParseRawRequest(raw string) (rr *RequestResponse, err error) {
// end of headers next is body
break
}
parts := strings.SplitN(line, ":", 2)
if len(parts) != 2 {
key, value, found := strings.Cut(line, ":")
if !found || key == "" {
return nil, fmt.Errorf("invalid header line: %s", line)
}
rr.Request.Headers.Set(parts[0], parts[1][1:])
value = strings.TrimSpace(value)
// Host carries the authority rather than request metadata, and callers
// read it off the URL: retryablehttp derives the wire Host from there,
// and keeping it in the header map would expose it to header fuzzing as
// if it were an ordinary header.
if strings.EqualFold(key, "Host") {
// an absolute request target takes precedence over the Host header
if rr.URL.Host == "" {
rr.URL.Host = value
}
continue
}
rr.Request.Headers.Set(key, value)
}
// parse body
+133
View File
@@ -121,6 +121,139 @@ func TestParseRawRequestBodyTrailingNewlines(t *testing.T) {
}
}
// Headers used to be parsed positionally: the second line was always taken as
// the Host line and dropped from the header map, and values were read by
// skipping a single byte after the colon. Any request-shaped input (burp, jsonl,
// yaml, openapi) hitting one of these shapes either lost a header silently,
// truncated a value, or crashed the whole scan.
func TestParseRawRequestHeaderParsing(t *testing.T) {
tests := []struct {
name string
raw string
host string
headers map[string]string
}{
{
name: "host first",
raw: "POST /login HTTP/1.1\r\nHost: example.com\r\nContent-Type: application/json\r\n\r\n{}",
host: "example.com",
headers: map[string]string{"Content-Type": "application/json"},
},
{
name: "host after other headers",
raw: "POST /login HTTP/1.1\r\nContent-Type: application/json\r\nHost: example.com\r\n\r\n{}",
host: "example.com",
headers: map[string]string{"Content-Type": "application/json"},
},
{
name: "host absent",
raw: "POST /login HTTP/1.1\r\nContent-Type: application/json\r\n\r\n{}",
host: "",
headers: map[string]string{"Content-Type": "application/json"},
},
{
name: "no space after colon",
raw: "GET / HTTP/1.1\r\nHost:example.com\r\nX-Token:abc\r\n\r\n",
host: "example.com",
headers: map[string]string{"X-Token": "abc"},
},
{
name: "valueless header",
raw: "GET / HTTP/1.1\r\nHost: example.com\r\nX-Empty:\r\n\r\n",
host: "example.com",
headers: map[string]string{"X-Empty": ""},
},
{
name: "lowercase host key",
raw: "GET / HTTP/1.1\r\ncontent-type: text/plain\r\nhost: example.com\r\n\r\n",
host: "example.com",
headers: map[string]string{"content-type": "text/plain"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var rr *RequestResponse
var err error
require.NotPanics(t, func() {
rr, err = ParseRawRequest(tt.raw)
})
require.NoError(t, err)
require.Equal(t, tt.host, rr.URL.Host)
got := map[string]string{}
rr.Request.Headers.Iterate(func(k, v string) bool {
got[k] = v
return true
})
require.Equal(t, tt.headers, got)
})
}
}
// Requests captured through a proxy carry an absolute request target, which the
// parser used to append to the authority as if it were a path.
func TestParseRawRequestAbsoluteTarget(t *testing.T) {
tests := []struct {
name string
raw string
url string
host string
path string
}{
{
name: "absolute target",
raw: "GET http://example.com/p?q=1 HTTP/1.1\r\nHost: example.com\r\n\r\n",
url: "http://example.com/p?q=1",
},
{
name: "absolute target wins over host header",
raw: "GET https://example.com/p HTTP/1.1\r\nHost: proxy.internal\r\n\r\n",
url: "https://example.com/p",
},
{
name: "uppercase scheme absolute target",
raw: "GET HTTP://example.com/p?q=1 HTTP/1.1\r\nHost: example.com\r\n\r\n",
url: "http://example.com/p?q=1",
host: "example.com",
path: "/p",
},
{
name: "mixed-case https absolute target wins over host",
raw: "GET Https://Example.COM/p HTTP/1.1\r\nHost: proxy.internal\r\n\r\n",
url: "https://Example.COM/p",
host: "Example.COM",
path: "/p",
},
{
name: "origin form keeps host header",
raw: "GET /p HTTP/1.1\r\nHost: example.com\r\n\r\n",
url: "example.com/p",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
rr, err := ParseRawRequest(tt.raw)
require.NoError(t, err)
require.Equal(t, tt.url, rr.URL.String())
if tt.host != "" {
require.Equal(t, tt.host, rr.URL.Host)
}
if tt.path != "" {
require.Equal(t, tt.path, rr.URL.Path)
}
})
}
}
// A header line without a colon is not a header, and must not be mistaken for
// the start of the body.
func TestParseRawRequestRejectsMalformedHeader(t *testing.T) {
_, err := ParseRawRequest("GET / HTTP/1.1\r\nHost: example.com\r\nnot-a-header\r\n\r\n")
require.Error(t, err)
}
func TestUnmarshalJSON(t *testing.T) {
tests := []struct {
name string
+56
View File
@@ -0,0 +1,56 @@
package installer
import (
"encoding/json"
"os"
"path/filepath"
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
)
func TestTemplateRootLifecycleIgnoresLegacyDiscoverySource(t *testing.T) {
dir := t.TempDir()
// Keep a backslash in the path so JSON escaping is exercised on all platforms.
root := filepath.Join(dir, `system\nuclei\nuclei-templates`)
if err := os.MkdirAll(root, 0o755); err != nil {
t.Fatalf("create template root: %v", err)
}
stateDir := filepath.Join(dir, "state")
if err := os.MkdirAll(stateDir, 0o700); err != nil {
t.Fatalf("create state directory: %v", err)
}
state, err := json.Marshal(map[string]string{
"nuclei-templates-directory": root,
"nuclei-templates-directory-source": "xdg-system",
})
if err != nil {
t.Fatalf("marshal templates state: %v", err)
}
if err := os.WriteFile(filepath.Join(stateDir, config.TemplatesStateFileName), state, 0o600); err != nil {
t.Fatalf("write templates state: %v", err)
}
oldConfig := config.DefaultConfig
cfg := &config.Config{}
cfg.SetStateDir(stateDir)
if err := cfg.ReadTemplatesConfig(); err != nil {
t.Fatalf("read templates state: %v", err)
}
config.DefaultConfig = cfg
t.Cleanup(func() { config.DefaultConfig = oldConfig })
manager := &TemplateManager{DisablePublicTemplates: true}
if err := manager.installTemplatesAt(root); err != nil {
t.Fatalf("install into discovered root: %v", err)
}
if err := manager.updateTemplatesAt(root); err != nil {
t.Fatalf("update discovered root: %v", err)
}
if err := manager.writeChecksumFileInDir(root); err != nil {
t.Fatalf("write discovered-root checksum: %v", err)
}
if _, err := os.Stat(config.DefaultConfig.GetChecksumFilePath()); err != nil {
t.Fatalf("stat discovered-root checksum: %v", err)
}
}
+11 -6
View File
@@ -80,13 +80,16 @@ func (t *TemplateManager) FreshInstallIfNotExists() error {
if fileutil.FolderExists(config.DefaultConfig.TemplatesDirectory) {
return nil
}
gologger.Info().Msgf("nuclei-templates are not installed, installing...")
if err := t.installTemplatesAt(config.DefaultConfig.TemplatesDirectory); err != nil {
return errkit.Wrapf(err, "failed to install templates at %s", config.DefaultConfig.TemplatesDirectory)
}
if t.CustomTemplates != nil {
t.CustomTemplates.Download(context.TODO())
}
return nil
}
@@ -130,10 +133,12 @@ func (t *TemplateManager) installTemplatesAt(dir string) error {
return errkit.Wrapf(err, "failed to create directory at %s", dir)
}
}
if t.DisablePublicTemplates {
gologger.Info().Msgf("Skipping installation of public nuclei-templates")
return nil
}
ghrd, err := updateutils.NewghReleaseDownloader(config.OfficialNucleiTemplatesRepoName)
if err != nil {
return errkit.Wrapf(err, "failed to install templates at %s", dir)
@@ -144,7 +149,9 @@ func (t *TemplateManager) installTemplatesAt(dir string) error {
if err != nil {
return errkit.Wrapf(err, "failed to write templates to disk at %s", dir)
}
gologger.Info().Msgf("Successfully installed nuclei-templates at %s", dir)
return nil
}
@@ -154,6 +161,7 @@ func (t *TemplateManager) updateTemplatesAt(dir string) error {
gologger.Info().Msgf("Skipping update of public nuclei-templates")
return nil
}
// firstly, read checksums from .checksum file these are used to generate stats
oldchecksums, err := t.getChecksumFromDir(dir)
if err != nil {
@@ -259,7 +267,7 @@ func (t *TemplateManager) summarizeChanges(old, new map[string]string) *template
func (t *TemplateManager) getAbsoluteFilePath(templateDir, uri string, f fs.FileInfo) string {
// overwrite .nuclei-ignore every time nuclei-templates are downloaded
if f.Name() == config.NucleiIgnoreFileName {
return config.DefaultConfig.GetIgnoreFilePath()
return config.DefaultConfig.GetActiveIgnoreFilePath()
}
// skip all meta files
if !strings.EqualFold(f.Name(), config.NewTemplateAdditionsFileName) {
@@ -384,11 +392,6 @@ func (t *TemplateManager) writeTemplatesToDisk(ghrd *updateutils.GHReleaseDownlo
if err := config.DefaultConfig.WriteTemplatesConfig(); err != nil {
return nil, errkit.Wrap(err, "failed to write templates config")
}
// update ignore hash after writing new templates
if err := config.DefaultConfig.UpdateNucleiIgnoreHash(); err != nil {
return nil, errkit.Wrap(err, "failed to update nuclei ignore hash")
}
// update templates version in config file
if err := config.DefaultConfig.SetTemplatesVersion(ghrd.Latest.GetTagName()); err != nil {
return nil, errkit.Wrap(err, "failed to update templates version")
@@ -610,6 +613,7 @@ func (t *TemplateManager) writeChecksumFileInDir(dir string) error {
if err != nil {
return err
}
var buff bytes.Buffer
for k, v := range checksumMap {
buff.WriteString(k)
@@ -617,6 +621,7 @@ func (t *TemplateManager) writeChecksumFileInDir(dir string) error {
buff.WriteString(v)
buff.WriteString(";")
}
return os.WriteFile(config.DefaultConfig.GetChecksumFilePath(), buff.Bytes(), checkSumFilePerm)
}
+1 -1
View File
@@ -58,7 +58,7 @@ func TestTemplateInstallation(t *testing.T) {
// we should have at least 1000 templates
require.Greater(t, counter, 1000)
// every time we install templates, it should override the ignore file with latest one
require.FileExists(t, config.DefaultConfig.GetIgnoreFilePath())
require.FileExists(t, config.DefaultConfig.GetActiveIgnoreFilePath())
t.Logf("Installed %d templates", counter)
}
+52 -43
View File
@@ -1,9 +1,10 @@
package installer
import (
"fmt"
"io"
"net/http"
"net/url"
"os"
"sync"
"github.com/projectdiscovery/gologger"
@@ -31,30 +32,24 @@ type PdtmAPIResponse struct {
} `json:"tools"`
}
// NucleiVersionCheck checks for the latest version of nuclei and nuclei templates
// and returns an error if it fails to check on success it returns nil and changes are
// made to the default config in config.DefaultConfig
func NucleiVersionCheck() error {
// NucleiVersionCheck checks for the latest Nuclei and template versions. It
// returns the current remote ignore-file hash without persisting it.
func NucleiVersionCheck() (string, error) {
return doVersionCheck(false)
}
// this will be updated by features of 1.21 release (which directly provides sync.Once(func()))
type sdkUpdateCheck struct {
sync.Once
var sdkVersionCheck = sync.OnceValues(func() (string, error) {
return doVersionCheck(true)
})
// NucleiSDKVersionCheck checks for the latest versions in SDK mode. It returns
// the same result for every call in the process.
func NucleiSDKVersionCheck() (string, error) {
return sdkVersionCheck()
}
var sdkUpdateCheckInstance = &sdkUpdateCheck{}
// NucleiSDKVersionCheck checks for latest version of nuclei which running in sdk mode
// this only happens once per process regardless of how many times this function is called
func NucleiSDKVersionCheck() {
sdkUpdateCheckInstance.Do(func() {
_ = doVersionCheck(true)
})
}
// getpdtmParams returns encoded query parameters sent to update check endpoint
func getpdtmParams(isSDK bool) string {
// getPDTMParams returns encoded query parameters sent to update check endpoint
func getPDTMParams(isSDK bool) string {
values, err := url.ParseQuery(updateutils.GetpdtmParams(config.Version))
if err != nil {
gologger.Verbose().Msgf("error parsing update check params: %v", err)
@@ -68,40 +63,50 @@ func getpdtmParams(isSDK bool) string {
// UpdateIgnoreFile updates default ignore file by downloading latest ignore file
func UpdateIgnoreFile() error {
resp, err := retryableHttpClient.Get(pdtmNucleiIgnoreFileEndpoint + "?" + getpdtmParams(false))
if err != nil {
return err
}
bin, err := io.ReadAll(resp.Body)
if err != nil {
return err
}
if err := os.WriteFile(config.DefaultConfig.GetIgnoreFilePath(), bin, 0644); err != nil {
return err
}
return config.DefaultConfig.UpdateNucleiIgnoreHash()
}
func doVersionCheck(isSDK bool) error {
// we use global retryablehttp client so its not immediately gc'd if any references are held
// and according our config we have idle connections which are shown as leaked by goleak in tests
// i.e we close all idle connections after our use and it doesn't affect any other part of the code
defer retryableHttpClient.HTTPClient.CloseIdleConnections()
resp, err := retryableHttpClient.Get(pdtmNucleiVersionEndpoint + "?" + getpdtmParams(isSDK))
resp, err := retryableHttpClient.Get(pdtmNucleiIgnoreFileEndpoint + "?" + getPDTMParams(false))
if err != nil {
return err
}
defer func() {
_ = resp.Body.Close()
}()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("download .nuclei-ignore file: unexpected HTTP status %s", resp.Status)
}
bin, err := io.ReadAll(resp.Body)
if err != nil {
return err
}
if err := config.DefaultConfig.WriteActiveIgnoreFile(bin); err != nil {
return err
}
return nil
}
func doVersionCheck(isSDK bool) (string, error) {
// we use global retryablehttp client so its not immediately gc'd if any references are held
// and according our config we have idle connections which are shown as leaked by goleak in tests
// i.e we close all idle connections after our use and it doesn't affect any other part of the code
defer retryableHttpClient.HTTPClient.CloseIdleConnections()
resp, err := retryableHttpClient.Get(pdtmNucleiVersionEndpoint + "?" + getPDTMParams(isSDK))
if err != nil {
return "", err
}
defer func() {
_ = resp.Body.Close()
}()
bin, err := io.ReadAll(resp.Body)
if err != nil {
return "", err
}
var pdtmResp PdtmAPIResponse
if err := json.Unmarshal(bin, &pdtmResp); err != nil {
return err
return "", err
}
var nucleiversion, templateversion string
for _, tool := range pdtmResp.Tools {
@@ -117,5 +122,9 @@ func doVersionCheck(isSDK bool) error {
}
}
}
return config.DefaultConfig.WriteVersionCheckData(pdtmResp.IgnoreHash, nucleiversion, templateversion)
if err := config.DefaultConfig.WriteVersionCheckData("", nucleiversion, templateversion); err != nil {
return "", err
}
return pdtmResp.IgnoreHash, nil
}
+3 -3
View File
@@ -9,11 +9,11 @@ import (
)
func TestVersionCheck(t *testing.T) {
err := NucleiVersionCheck()
ignoreHash, err := NucleiVersionCheck()
require.Nil(t, err)
cfg := config.DefaultConfig
if generic.EqualsAny("", cfg.LatestNucleiIgnoreHash, cfg.LatestNucleiVersion, cfg.LatestNucleiTemplatesVersion) {
if generic.EqualsAny("", ignoreHash, cfg.LatestNucleiVersion, cfg.LatestNucleiTemplatesVersion) {
// all above values cannot be empty
t.Errorf("something went wrong got empty response nuclei-version=%v templates-version=%v ignore-hash=%v", cfg.LatestNucleiVersion, cfg.LatestNucleiTemplatesVersion, cfg.LatestNucleiIgnoreHash)
t.Errorf("something went wrong got empty response nuclei-version=%v templates-version=%v ignore-hash=%v", cfg.LatestNucleiVersion, cfg.LatestNucleiTemplatesVersion, ignoreHash)
}
}
@@ -165,7 +165,6 @@ func initDialers(options *types.Options) error {
forward = &net.Dialer{
Timeout: opts.DialerTimeout,
KeepAlive: opts.DialerKeepAlive,
DualStack: true,
}
}
dialer, err := proxy.FromURL(proxyURL, forward)
+36
View File
@@ -4,6 +4,7 @@ import (
"context"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
@@ -200,6 +201,41 @@ Host: {{Hostname}}`},
require.Equal(t, "http", parsedURL.Scheme, "scheme should inherit from input when annotation host has no scheme")
}
func TestUnsafeRawRequestPreservesAbsoluteRequestTarget(t *testing.T) {
options := testutils.DefaultOptions
testutils.Init(options)
const requestLine = "GET http://intranet/foo?request=1 HTTP/1.1"
request := &Request{
ID: "testing-http-unsafe-absolute-request-target",
Name: "testing",
Unsafe: true,
Raw: []string{requestLine + `
Host: {{Hostname}}`},
}
executerOpts := testutils.NewMockExecuterOptions(options, &testutils.TemplateInfo{
ID: request.ID,
Info: model.Info{SeverityHolder: severity.Holder{Severity: severity.Low}, Name: "test"},
})
require.NoError(t, request.Compile(executerOpts), "could not compile unsafe raw request")
generator := request.newGenerator(false)
inputData, payloads, _ := generator.nextValue()
generated, err := generator.Make(context.Background(), contextargs.NewWithInput(context.Background(), "http://example.com/bar?input=1"), inputData, payloads, map[string]interface{}{})
require.NoError(t, err, "could not make unsafe raw request")
generatedURL, err := urlutil.ParseAbsoluteURL(generated.URL(), true)
require.NoError(t, err, "could not parse generated request URL")
require.Equal(t, "example.com", generatedURL.Host, "absolute request target must not override the scan target")
require.Equal(t, "/foo", generatedURL.Path, "absolute request target path must not merge with the scan target path")
require.Equal(t, "1", generatedURL.Params.Get("request"), "generated URL must include the absolute request target query")
require.Empty(t, generatedURL.Params.Get("input"), "generated URL must not include the scan target query")
requestBytes, err := dump(generated, generated.URL())
require.NoError(t, err, "could not dump unsafe raw request")
actualRequestLine := strings.TrimSuffix(strings.SplitN(string(requestBytes), "\n", 2)[0], "\r")
require.Equal(t, requestLine, actualRequestLine, "unsafe raw request line must remain unchanged")
}
func TestMakeRequestFromModelUniqueInteractsh(t *testing.T) {
options := testutils.DefaultOptions
+15
View File
@@ -23,6 +23,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/http/httpclientpool"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/network/networkclientpool"
"github.com/projectdiscovery/nuclei/v3/pkg/types/scanstrategy"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/json"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/stats"
"github.com/projectdiscovery/rawhttp"
@@ -558,6 +559,11 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
request.Threads = options.GetThreadsForNPayloadRequests(request.Requests(), request.Threads)
}
}
// Avoid reusing client-side HTTP proxy connections for the legacy non-threaded spray path.
if shouldDisableKeepAliveForHTTPProxy(request, options) {
request.connConfiguration.Connection.DisableKeepAlive = true
}
return nil
}
@@ -607,6 +613,15 @@ func (request *Request) HasFuzzing() bool {
return len(request.Fuzzing) > 0
}
// shouldDisableKeepAliveForHTTPProxy preserves the pre-pooling behavior only for
// standard HTTP proxies in non-threaded template/auto spray scans.
func shouldDisableKeepAliveForHTTPProxy(request *Request, options *protocols.ExecutorOptions) bool {
if request == nil || options == nil || options.Options == nil || options.Options.AliveHttpProxy == "" {
return false
}
return request.Threads <= 0 && options.Options.ScanStrategy != scanstrategy.HostSpray.String()
}
// AnalyzeConnectionReuse determines if a request can safely reuse connections.
// Returns ReuseUnsafe if connection closure is required, ReuseSafe otherwise.
// This analysis ensures backward compatibility by preserving connection-close behavior
@@ -278,6 +278,45 @@ func GetRawHTTP(options *protocols.ExecutorOptions) *rawhttp.Client {
return dialers.RawHTTPClient
}
// defaultIdleConnPoolSize is the floor for the per-host idle connection pool.
//
// The pool has to cover the connections the transport has OPEN against a host
// at once, and that is not the same as the number of requests in flight.
// http.Transport starts a dial whenever a request does not find an idle
// connection immediately; if one frees up first the request takes it and the
// racing dial still completes, so a scan holds many more connections than it
// has simultaneous requests. Measured against a keep-alive target at -c 25:
// requests in flight peaked at 22-31, while open connections peaked at 121-178.
//
// http.Transport closes a connection returned to a host whose idle pool is
// already full, so a pool sized to the nominal concurrency discards most of
// those and the next request dials again: 3,070 connections for 5,388 requests
// at a pool of 25, against 414 at 500.
//
// A pool that is never filled costs nothing -- it is a ceiling on retained
// connections, not an allocation; the transport only ever holds connections it
// actually opened and reaps them after IdleConnTimeout. 500 is the value this
// used before the per-host client pool was introduced.
const defaultIdleConnPoolSize = 500
// idleConnPoolSize returns the per-host idle connection pool size for a client.
//
// templateThreads (-c) and requestThreads (a template's own payload
// parallelism) only ever raise the floor. Neither may lower it: -c is not an
// upper bound on open connections (see above), and a template that declares two
// payload threads must not shrink a pool shared with everything else running
// against that host.
func idleConnPoolSize(templateThreads, requestThreads int) int {
size := defaultIdleConnPoolSize
if templateThreads > size {
size = templateThreads
}
if requestThreads > size {
size = requestThreads
}
return size
}
// Get creates or gets a client for the protocol based on custom configuration.
// The host parameter scopes the client to a specific target, enabling per-host
// connection reuse with keep-alive. Pass an empty string for non-scanning uses.
@@ -322,13 +361,9 @@ func wrappedGet(options *types.Options, configuration *Configuration, host strin
retryableHttpOptions.Timeout = configuration.ResponseHeaderTimeout
}
maxIdleConns := 4
maxIdleConnsPerHost := 4
maxIdleConnsPerHost := idleConnPoolSize(options.TemplateThreads, configuration.Threads)
maxIdleConns := maxIdleConnsPerHost
maxConnsPerHost := 0 // unlimited by default; the SPM handler controls concurrency
if configuration.Threads > 0 {
maxIdleConnsPerHost = configuration.Threads
maxIdleConns = configuration.Threads
}
disableKeepAlives := configuration.Connection != nil && configuration.Connection.DisableKeepAlive
@@ -0,0 +1,345 @@
package httpclientpool
import (
"fmt"
"io"
"net"
"net/http"
"net/http/httptest"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/stretchr/testify/require"
"github.com/projectdiscovery/retryablehttp-go"
)
// TestIdleConnPoolAllowsKeepAliveReuse is a regression test for the per-host
// idle connection pool being smaller than the concurrency that runs against a
// host.
//
// http.Transport closes a connection returned to a host whose idle pool is
// already full. When MaxIdleConnsPerHost was 4, a scan running templates
// concurrently against one host reused almost nothing: connections past the
// fourth were closed on return and the next request dialled again, so the
// target saw close to one TCP connection per HTTP request.
//
// The server counts accepted connections, which is what actually matters:
// connection churn is invisible in request counts but is what pays a TCP (and
// TLS) handshake per request and exhausts NAT port mappings.
func TestIdleConnPoolAllowsKeepAliveReuse(t *testing.T) {
const (
requests = 400
concurrency = 50
)
var accepted, arrived atomic.Int64
// Hold the first `concurrency` requests inside the handler until all of
// them have arrived, so the transport is genuinely forced to keep that
// many connections open at once. Without this the requests could
// serialise and the connection count would prove nothing.
barrier := make(chan struct{})
var releaseOnce sync.Once
release := func() { releaseOnce.Do(func() { close(barrier) }) }
// Safety valve: never hang the suite if fewer than `concurrency` requests
// ever reach the handler.
timer := time.AfterFunc(30*time.Second, release)
defer timer.Stop()
server := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if arrived.Add(1) >= concurrency {
release()
}
<-barrier
_, _ = w.Write([]byte("ok"))
}))
server.Config.ConnState = func(_ net.Conn, state http.ConnState) {
if state == http.StateNew {
accepted.Add(1)
}
}
server.Start()
defer server.Close()
opts := newTestOptions(t, "test-idle-conn-pool-reuse")
opts.TemplateThreads = concurrency
client, err := Get(opts, &Configuration{}, server.Listener.Addr().String())
require.NoError(t, err)
var succeeded atomic.Int64
var errMu sync.Mutex
var firstErr error
sem := make(chan struct{}, concurrency)
var wg sync.WaitGroup
for i := 0; i < requests; i++ {
wg.Add(1)
sem <- struct{}{}
go func(i int) {
defer wg.Done()
defer func() { <-sem }()
recordErr := func(err error) {
errMu.Lock()
defer errMu.Unlock()
if firstErr == nil {
firstErr = err
}
}
req, err := retryablehttp.NewRequest(http.MethodGet, fmt.Sprintf("%s/path-%d", server.URL, i), nil)
if err != nil {
recordErr(err)
return
}
resp, err := client.Do(req)
if err != nil {
recordErr(err)
return
}
// Drain and close so the connection is eligible for reuse.
if _, err := io.Copy(io.Discard, resp.Body); err != nil {
_ = resp.Body.Close()
recordErr(err)
return
}
if err := resp.Body.Close(); err != nil {
recordErr(err)
return
}
succeeded.Add(1)
}(i)
}
wg.Wait()
// Without these the connection assertion is vacuous: zero successful
// requests also means zero connections.
require.NoError(t, firstErr, "requests must succeed for the connection count to mean anything")
require.Equal(t, int64(requests), succeeded.Load(), "all requests must succeed")
require.GreaterOrEqual(t, arrived.Load(), int64(concurrency),
"handler must have seen at least `concurrency` requests for the overlap barrier to have engaged")
conns := accepted.Load()
t.Logf("%d requests over %d connections (%.1f requests/connection)",
requests, conns, float64(requests)/float64(conns))
// `concurrency` connections are genuinely required by the barrier above.
// Anything much beyond that is the pool discarding connections instead of
// reusing them; the regression lands near one connection per request.
require.LessOrEqual(t, conns, int64(concurrency*2),
"connections should be bounded by concurrency, not by request count")
}
// TestIdleConnPoolSize covers the sizing rule itself: a floor that -c raises
// when it is larger, payload threads raising it too, and nothing either of them
// declares shrinking it.
func TestIdleConnPoolSize(t *testing.T) {
for _, testCase := range []struct {
name string
templateThreads int
requestThreads int
want int
}{
{name: "unset", templateThreads: 0, requestThreads: 0, want: defaultIdleConnPoolSize},
{name: "default -c", templateThreads: 25, requestThreads: 0, want: defaultIdleConnPoolSize},
{name: "-c 50", templateThreads: 50, requestThreads: 0, want: defaultIdleConnPoolSize},
{name: "-c 800", templateThreads: 800, requestThreads: 0, want: 800},
{name: "payload threads raise the pool", templateThreads: 25, requestThreads: 900, want: 900},
{name: "payload threads never shrink the pool", templateThreads: 25, requestThreads: 2, want: defaultIdleConnPoolSize},
{name: "payload threads never shrink -c 800", templateThreads: 800, requestThreads: 2, want: 800},
{name: "negative payload threads never shrink the pool", templateThreads: 50, requestThreads: -1, want: defaultIdleConnPoolSize},
} {
t.Run(testCase.name, func(t *testing.T) {
got := idleConnPoolSize(testCase.templateThreads, testCase.requestThreads)
require.Equal(t, testCase.want, got)
require.GreaterOrEqual(t, got, defaultIdleConnPoolSize,
"the floor exists because -c does not bound how many connections the transport opens")
require.GreaterOrEqual(t, got, testCase.templateThreads,
"the pool must never be smaller than the template concurrency it serves")
})
}
}
// TestIdleConnPoolSizeIsWiredToTransport checks that the size actually reaches
// the transport, not just the helper: a sizing rule that never lands on
// http.Transport would leave the reuse bug in place while the unit test above
// still passed.
func TestIdleConnPoolSizeIsWiredToTransport(t *testing.T) {
opts := newTestOptions(t, "test-idle-conn-pool-wiring")
opts.TemplateThreads = 800
client, err := Get(opts, &Configuration{}, "example.com")
require.NoError(t, err)
tracking, ok := client.HTTPClient.Transport.(*connTrackingTransport)
require.True(t, ok, "expected the connection-tracking transport wrapper")
transport, ok := tracking.base.(*http.Transport)
require.True(t, ok, "expected an *http.Transport underneath")
require.Equal(t, 800, transport.MaxIdleConnsPerHost)
require.Equal(t, 800, transport.MaxIdleConns)
}
// cyclicBarrier releases waiters in groups of n. A single-shot barrier only
// forces the FIRST group of requests to overlap, and a pool that is too small
// still looks fine afterwards, because once the requests stop overlapping a
// shallow pool is enough. Holding every group keeps the concurrency up for the
// whole run, which is what a scan does to a host.
type cyclicBarrier struct {
mu sync.Mutex
n int
count int
gate chan struct{}
abandoned bool
}
func newCyclicBarrier(n int) *cyclicBarrier {
return &cyclicBarrier{n: n, gate: make(chan struct{})}
}
func (b *cyclicBarrier) wait() {
b.mu.Lock()
if b.abandoned {
b.mu.Unlock()
return
}
b.count++
gate := b.gate
if b.count == b.n {
b.count = 0
b.gate = make(chan struct{})
close(gate)
b.mu.Unlock()
return
}
b.mu.Unlock()
<-gate
}
// abandon is the safety valve: it stops blocking so a shortfall fails on the
// assertions instead of hanging the suite.
func (b *cyclicBarrier) abandon() {
b.mu.Lock()
defer b.mu.Unlock()
if b.abandoned {
return
}
b.abandoned = true
close(b.gate)
}
func (b *cyclicBarrier) wasAbandoned() bool {
b.mu.Lock()
defer b.mu.Unlock()
return b.abandoned
}
// TestIdleConnPoolAllowsKeepAliveReuseWithPayloadThreads covers the client that
// payload templates share. Configuration.Threads is part of the client cache
// key, so every template declaring the same payload thread count hashes to one
// entry and its workers pile onto a single pool -- templateThreads templates
// times payloadThreads workers each. That client has to reuse connections just
// like the default one, and a rule that sized the pool from a single template's
// threads would leave it far below the concurrency it actually serves.
func TestIdleConnPoolAllowsKeepAliveReuseWithPayloadThreads(t *testing.T) {
const (
templateThreads = 10
payloadThreads = 5
concurrency = templateThreads * payloadThreads
requestsPerWorker = 8
requests = concurrency * requestsPerWorker
)
var accepted atomic.Int64
barrier := newCyclicBarrier(concurrency)
timer := time.AfterFunc(30*time.Second, barrier.abandon)
defer timer.Stop()
server := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
barrier.wait()
_, _ = w.Write([]byte("ok"))
}))
server.Config.ConnState = func(_ net.Conn, state http.ConnState) {
if state == http.StateNew {
accepted.Add(1)
}
}
server.Start()
defer server.Close()
opts := newTestOptions(t, "test-idle-conn-pool-payload-threads")
opts.TemplateThreads = templateThreads
// One client, as the engine would hand out: every payload template with
// this thread count hashes to the same cache entry.
client, err := Get(opts, &Configuration{Threads: payloadThreads}, server.Listener.Addr().String())
require.NoError(t, err)
var succeeded atomic.Int64
var errMu sync.Mutex
var firstErr error
recordErr := func(err error) {
errMu.Lock()
defer errMu.Unlock()
if firstErr == nil {
firstErr = err
}
}
// templateThreads templates in flight, each running payloadThreads workers.
var wg sync.WaitGroup
for template := 0; template < templateThreads; template++ {
for worker := 0; worker < payloadThreads; worker++ {
wg.Add(1)
go func(template, worker int) {
defer wg.Done()
for i := 0; i < requestsPerWorker; i++ {
req, err := retryablehttp.NewRequest(http.MethodGet,
fmt.Sprintf("%s/t-%d/w-%d/%d", server.URL, template, worker, i), nil)
if err != nil {
recordErr(err)
barrier.abandon()
return
}
resp, err := client.Do(req)
if err != nil {
recordErr(err)
barrier.abandon()
return
}
if _, err := io.Copy(io.Discard, resp.Body); err != nil {
_ = resp.Body.Close()
recordErr(err)
return
}
if err := resp.Body.Close(); err != nil {
recordErr(err)
return
}
succeeded.Add(1)
}
}(template, worker)
}
}
wg.Wait()
// Without these the connection assertion is vacuous: zero successful
// requests also means zero connections.
require.NoError(t, firstErr, "requests must succeed for the connection count to mean anything")
require.Equal(t, int64(requests), succeeded.Load(), "all requests must succeed")
require.False(t, barrier.wasAbandoned(),
"the barrier must have held every group; the safety valve firing means the overlap was not sustained")
conns := accepted.Load()
t.Logf("%d requests over %d connections (%.1f requests/connection), pool=%d",
requests, conns, float64(requests)/float64(conns),
idleConnPoolSize(templateThreads, payloadThreads))
require.LessOrEqual(t, conns, int64(concurrency*2),
"connections should be bounded by the concurrency, not by the request count")
}
@@ -0,0 +1,20 @@
package http
import (
"github.com/projectdiscovery/nuclei/v3/pkg/protocols"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/nuclei/v3/pkg/types/scanstrategy"
"github.com/stretchr/testify/require"
"testing"
)
func TestShouldDisableKeepAliveForHTTPProxy(t *testing.T) {
opts := types.DefaultOptions()
opts.AliveHttpProxy = "http://127.0.0.1:8080"
opts.ScanStrategy = scanstrategy.TemplateSpray.String()
require.True(t, shouldDisableKeepAliveForHTTPProxy(&Request{}, &protocols.ExecutorOptions{Options: opts}))
opts.ScanStrategy = scanstrategy.HostSpray.String()
require.False(t, shouldDisableKeepAliveForHTTPProxy(&Request{}, &protocols.ExecutorOptions{Options: opts}))
require.False(t, shouldDisableKeepAliveForHTTPProxy(&Request{Threads: 10}, &protocols.ExecutorOptions{Options: opts}))
}
+26 -9
View File
@@ -39,22 +39,39 @@ func Parse(request string, inputURL *urlutil.URL, unsafe, disablePathAutomerge b
}
// handle full URLs first (before checking unsafe flag) to extract relative path
if strings.HasPrefix(rawrequest.Path, "http://") || strings.HasPrefix(rawrequest.Path, "https://") {
urlx, err := urlutil.ParseURL(rawrequest.Path, true)
lowerPath := strings.ToLower(rawrequest.Path)
hasAbsoluteRequestTarget := strings.HasPrefix(lowerPath, "http://") || strings.HasPrefix(lowerPath, "https://")
if hasAbsoluteRequestTarget {
var requestTarget string
if strings.HasPrefix(lowerPath, "http://") {
requestTarget = "http://" + rawrequest.Path[len("http://"):]
} else {
requestTarget = "https://" + rawrequest.Path[len("https://"):]
}
urlx, err := urlutil.ParseAbsoluteURL(requestTarget, true)
if err != nil {
return nil, errkit.Wrapf(err, "failed to parse url %v from template", rawrequest.Path)
}
prevPath := rawrequest.Path
relPath := urlx.GetRelativePath()
// NOTE(dwisiswant0): Use rel path instead if unsafe.
// See https://github.com/projectdiscovery/nuclei/issues/6558.
if unsafe {
rawrequest.UnsafeRawBytes = bytes.Replace(rawrequest.UnsafeRawBytes, []byte(prevPath), []byte(relPath), 1)
}
// rotate full URL with rel path
rawrequest.Path = relPath
if unsafe {
cloned := inputURL.Clone()
cloned.Path = ""
cloned.Fragment = ""
cloned.Params = urlutil.NewOrderedParams()
cloned.Params.IncludeEquals = true
if err := cloned.MergePath(relPath, true); err != nil {
return nil, errkit.Wrapf(err, "failed to build request url for absolute target %v", requestTarget)
}
rawrequest.FullURL = cloned.String()
return rawrequest, nil
}
}
switch {
+43 -22
View File
@@ -287,69 +287,81 @@ func TestDisableMergePath(t *testing.T) {
}
func TestUnsafeWithFullURL(t *testing.T) {
// Test unsafe mode with full URL - should extract relative path
// The relative path is used internally, but unsafe raw bytes remain unchanged.
request, err := Parse(`GET http://127.0.0.1/foo HTTP/1.1
Host: {{Hostname}}
User-Agent: Mozilla/5.0
Connection: close`, parseURL(t, "http://httpbin.org/bar"), true, true)
require.Nil(t, err, "could not parse unsafe request with full URL")
require.Equal(t, "/foo", request.Path, "Could not extract relative path from full URL in unsafe mode")
require.Contains(t, string(request.UnsafeRawBytes), "GET /foo HTTP/1.1", "UnsafeRawBytes should contain relative path, not full URL")
require.NotContains(t, string(request.UnsafeRawBytes), "http://127.0.0.1", "UnsafeRawBytes should not contain full URL")
require.Contains(t, string(request.UnsafeRawBytes), "GET http://127.0.0.1/foo HTTP/1.1", "unsafe request line must preserve the full URL")
}
func TestUnsafeWithFullURLAndPath(t *testing.T) {
// Test unsafe mode with full URL and target URL that has a path
func TestUnsafeWithFullURLWithoutPath(t *testing.T) {
const requestLine = "GET http://127.0.0.1:22 HTTP/1.1"
request, err := Parse(requestLine+"\nHost: {{Hostname}}", parseURL(t, "http://httpbin.org"), true, false)
require.NoError(t, err, "could not parse unsafe request with a pathless full URL")
require.Contains(t, string(request.UnsafeRawBytes), requestLine, "unsafe request line must preserve the pathless full URL")
}
func TestUnsafeWithFullURLIgnoresInputPath(t *testing.T) {
// The absolute request target supplies the path in unsafe mode.
request, err := Parse(`GET http://127.0.0.1/foo HTTP/1.1
Host: {{Hostname}}
User-Agent: Mozilla/5.0
Connection: close`, parseURL(t, "http://httpbin.org/bar"), true, false)
require.Nil(t, err, "could not parse unsafe request with full URL and path merge")
require.Equal(t, "/bar/foo", request.Path, "Could not merge path correctly from full URL in unsafe mode")
require.Contains(t, string(request.UnsafeRawBytes), "GET /bar/foo HTTP/1.1", "UnsafeRawBytes should contain merged relative path")
require.NotContains(t, string(request.UnsafeRawBytes), "http://127.0.0.1", "UnsafeRawBytes should not contain full URL")
require.Nil(t, err, "could not parse unsafe request with a full URL and input path")
require.Equal(t, "/foo", request.Path, "input path must not merge with an absolute request target")
require.Contains(t, string(request.UnsafeRawBytes), "GET http://127.0.0.1/foo HTTP/1.1", "unsafe request line must preserve the full URL")
}
func TestUnsafeWithMixedCaseFullURL(t *testing.T) {
request, err := Parse(`GET hTtP://127.0.0.1/foo HTTP/1.1
Host: {{Hostname}}
Connection: close`, parseURL(t, "http://httpbin.org/bar"), true, false)
require.NoError(t, err, "could not parse unsafe request with a mixed-case full URL")
require.Equal(t, "/foo", request.Path, "could not extract the relative path from a mixed-case full URL")
require.Contains(t, string(request.UnsafeRawBytes), "GET hTtP://127.0.0.1/foo HTTP/1.1", "unsafe request line must preserve the mixed-case full URL")
}
func TestUnsafeWithFullURLAndQueryParams(t *testing.T) {
// Test unsafe mode with full URL containing query parameters
// Query parameters are available internally without changing unsafe raw bytes.
request, err := Parse(`GET http://127.0.0.1/foo?id=123&name=test HTTP/1.1
Host: {{Hostname}}
User-Agent: Mozilla/5.0
Connection: close`, parseURL(t, "http://httpbin.org/bar"), true, true)
require.Nil(t, err, "could not parse unsafe request with full URL and query params")
require.Equal(t, "/foo?id=123&name=test", request.Path, "Could not extract relative path with query params from full URL in unsafe mode")
require.Contains(t, string(request.UnsafeRawBytes), "GET /foo?id=123&name=test HTTP/1.1", "UnsafeRawBytes should contain relative path with query params")
require.NotContains(t, string(request.UnsafeRawBytes), "http://127.0.0.1", "UnsafeRawBytes should not contain full URL")
require.Contains(t, string(request.UnsafeRawBytes), "GET http://127.0.0.1/foo?id=123&name=test HTTP/1.1", "unsafe request line must preserve the full URL")
}
func TestUnsafeWithHTTPSFullURL(t *testing.T) {
// Test unsafe mode with HTTPS full URL
// HTTPS absolute request targets follow the same raw-byte rule.
request, err := Parse(`GET https://secure.example.com/api/v1/users HTTP/1.1
Host: {{Hostname}}
Authorization: Bearer token123
Connection: close`, parseURL(t, "https://target.com/test"), true, true)
require.Nil(t, err, "could not parse unsafe request with HTTPS full URL")
require.Equal(t, "/api/v1/users", request.Path, "Could not extract relative path from HTTPS full URL in unsafe mode")
require.Contains(t, string(request.UnsafeRawBytes), "GET /api/v1/users HTTP/1.1", "UnsafeRawBytes should contain relative path")
require.NotContains(t, string(request.UnsafeRawBytes), "https://secure.example.com", "UnsafeRawBytes should not contain full URL")
require.Contains(t, string(request.UnsafeRawBytes), "GET https://secure.example.com/api/v1/users HTTP/1.1", "unsafe request line must preserve the full URL")
}
func TestUnsafeWithFullURLRootPath(t *testing.T) {
// Test unsafe mode with full URL pointing to root path
// When disable-path-automerge is true and path is /, it becomes empty string (expected behavior)
// Test unsafe mode with full URL pointing to root path.
request, err := Parse(`GET http://example.com/ HTTP/1.1
Host: {{Hostname}}
Connection: close`, parseURL(t, "http://target.com/api"), true, true)
require.Nil(t, err, "could not parse unsafe request with full URL root path")
// With disable-path-automerge=true and root path, it becomes empty per existing logic
require.Equal(t, "", request.Path, "Root path with disable-path-automerge should be empty")
require.Equal(t, "/", request.Path, "absolute root path must remain unchanged when path automerge is disabled")
require.Contains(t, string(request.UnsafeRawBytes), "GET http://example.com/ HTTP/1.1", "unsafe request line must preserve the full URL")
// Test with disable-path-automerge=false
// Test with disable-path-automerge=false.
request, err = Parse(`GET http://example.com/ HTTP/1.1
Host: {{Hostname}}
Connection: close`, parseURL(t, "http://target.com/api"), true, false)
require.Nil(t, err, "could not parse unsafe request with full URL root path and merge")
require.Equal(t, "/api", request.Path, "Should merge with target path when automerge enabled")
require.Nil(t, err, "could not parse unsafe request with a full URL root path and path automerge enabled")
require.Equal(t, "/", request.Path, "absolute root path must remain unchanged when path automerge is enabled")
require.Contains(t, string(request.UnsafeRawBytes), "GET http://example.com/ HTTP/1.1", "unsafe request line must preserve the full URL")
}
func TestSafeWithFullURL(t *testing.T) {
@@ -362,6 +374,15 @@ Connection: close`, parseURL(t, "http://target.com/v1"), false, true)
require.Equal(t, "http://target.com/api/users", request.FullURL, "Could not build correct FullURL in safe mode")
}
func TestSafeWithSingleLabelFullURL(t *testing.T) {
request, err := Parse(`GET http://intranet/foo HTTP/1.1
Host: {{Hostname}}
Connection: close`, parseURL(t, "http://target.com/bar"), false, true)
require.NoError(t, err, "could not parse safe request with a single-label full URL")
require.Equal(t, "/foo", request.Path, "could not extract the path from a single-label full URL")
require.Equal(t, "http://target.com/foo", request.FullURL, "single-label request target host must not replace the scan target")
}
func parseURL(t *testing.T, inputurl string) *urlutil.URL {
urlx, err := urlutil.Parse(inputurl)
if err != nil {
+1 -1
View File
@@ -846,7 +846,7 @@ func (request *Request) executeRequest(input *contextargs.Context, generatedRequ
}
connConfig := request.connConfiguration
if input.CookieJar != nil {
if input.CookieJar != nil && !request.DisableCookie {
connConfig = connConfig.Clone()
connConfig.Connection.SetCookieJar(input.CookieJar)
}
+21 -1
View File
@@ -6,6 +6,7 @@ import (
"net/http"
"net/http/httputil"
"os"
"strings"
"github.com/pkg/errors"
@@ -37,8 +38,27 @@ var RawInputMode = false
// ExecuteWithResults executes the protocol requests and returns results instead of writing them.
func (request *Request) ExecuteWithResults(input *contextargs.Context, metadata, previous output.InternalEvent, callback protocols.OutputEventCallback) error {
// Prefer response bodies carried on multi-format inputs (burp/jsonl/yaml).
if rr := input.MetaInput.ReqResp; rr != nil && rr.Response != nil && strings.TrimSpace(rr.Response.Raw) != "" {
err := request.executeRawInput(rr.Response.Raw, rr.URL.String(), input, callback)
if err == nil {
request.options.Progress.IncrementRequests()
}
return err
}
if RawInputMode {
return request.executeRawInput(input.MetaInput.Input, "", input, callback)
err := request.executeRawInput(input.MetaInput.Input, "", input, callback)
if err == nil {
request.options.Progress.IncrementRequests()
}
return err
}
// Export inputs attach ReqResp with a URL-shaped MetaInput.Input. Without a
// response body there is nothing to match offline; do not treat the URL as a filepath.
if input.MetaInput.ReqResp != nil {
return nil
}
wg, err := syncutil.New(syncutil.WithSize(request.options.Options.BulkSize))
@@ -0,0 +1,104 @@
package offlinehttp
import (
"testing"
"github.com/stretchr/testify/require"
"github.com/projectdiscovery/nuclei/v3/pkg/model"
"github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
"github.com/projectdiscovery/nuclei/v3/pkg/operators"
"github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/input/types"
"github.com/projectdiscovery/nuclei/v3/pkg/progress"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v3/internal/tests/testutils"
urlutil "github.com/projectdiscovery/utils/url"
)
func TestExecuteWithResultsUsesReqRespResponse(t *testing.T) {
options := testutils.DefaultOptions
testutils.Init(options)
request := &Request{}
executerOpts := testutils.NewMockExecuterOptions(options, &testutils.TemplateInfo{
ID: "offline-export",
Info: model.Info{SeverityHolder: severity.Holder{Severity: severity.Info}, Name: "test"},
})
counter := &countingProgress{Progress: executerOpts.Progress}
executerOpts.Progress = counter
ops := &operators.Operators{
Matchers: []*matchers.Matcher{{
Part: "body",
Type: matchers.MatcherTypeHolder{MatcherType: matchers.WordsMatcher},
Words: []string{`"id":"1"`},
}},
}
require.NoError(t, ops.Compile())
executerOpts.Operators = []*operators.Operators{ops}
require.NoError(t, request.Compile(executerOpts))
parsedURL, err := urlutil.ParseAbsoluteURL("http://localhost:8087/scans", false)
require.NoError(t, err)
rawResponse := "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: 11\r\nConnection: close\r\n\r\n{\"id\":\"1\"}\n"
input := contextargs.New(t.Context())
input.MetaInput.Input = parsedURL.String()
input.MetaInput.ReqResp = &types.RequestResponse{
URL: *parsedURL,
Response: &types.HttpResponse{
Raw: rawResponse,
},
}
var gotEvent *output.InternalWrappedEvent
err = request.ExecuteWithResults(input, nil, nil, func(event *output.InternalWrappedEvent) {
gotEvent = event
})
require.NoError(t, err)
require.NotNil(t, gotEvent)
require.True(t, gotEvent.HasOperatorResult())
require.True(t, gotEvent.OperatorsResult.Matched)
require.Equal(t, 1, counter.requests)
}
type countingProgress struct {
progress.Progress
requests int
}
func (c *countingProgress) IncrementRequests() {
c.requests++
if c.Progress != nil {
c.Progress.IncrementRequests()
}
}
func TestExecuteWithResultsSkipsWhenNoResponseOnReqResp(t *testing.T) {
options := testutils.DefaultOptions
options.BulkSize = 1
testutils.Init(options)
request := &Request{}
executerOpts := testutils.NewMockExecuterOptions(options, &testutils.TemplateInfo{
ID: "offline-export-empty",
Info: model.Info{SeverityHolder: severity.Holder{Severity: severity.Info}, Name: "test"},
})
executerOpts.Operators = []*operators.Operators{{}}
require.NoError(t, request.Compile(executerOpts))
parsedURL, err := urlutil.ParseAbsoluteURL("http://example.com/", false)
require.NoError(t, err)
input := contextargs.New(t.Context())
// URL-shaped input would fail filepath walk; without Response we fall through.
input.MetaInput.Input = parsedURL.String()
input.MetaInput.ReqResp = &types.RequestResponse{URL: *parsedURL}
err = request.ExecuteWithResults(input, nil, nil, func(event *output.InternalWrappedEvent) {
t.Fatalf("unexpected event: %#v", event)
})
// getInputPaths on a URL yields no files; should not panic and should return nil.
require.NoError(t, err)
}
-1
View File
@@ -146,7 +146,6 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
}
tlsxOptions := &clients.Options{
AllCiphers: true,
ScanMode: request.ScanMode,
Expired: true,
SelfSigned: true,
+116
View File
@@ -0,0 +1,116 @@
package templates
import (
"reflect"
"github.com/projectdiscovery/nuclei/v3/pkg/utils"
)
type cloneVisit struct {
typeOf reflect.Type
pointer uintptr
length int
capacity int
}
// cloneTemplate copies a clean parsed template before protocol compilation
// adds engine-local state. Unexported fields hold immutable parse state at this
// point and are copied by value; exported maps, slices, pointers, and
// interfaces are copied recursively.
func cloneTemplate(template *Template) *Template {
visited := make(map[cloneVisit]reflect.Value)
cloned := cloneTemplateValue(reflect.ValueOf(template), visited)
clonedTemplate := cloned.Interface().(*Template)
clonedTemplate.Variables.InsertionOrderedStringMap = *utils.NewEmptyInsertionOrderedStringMap(template.Variables.Len())
template.Variables.ForEach(func(key string, value interface{}) {
clonedValue := cloneTemplateValue(reflect.ValueOf(value), visited)
if clonedValue.IsValid() {
clonedTemplate.Variables.Set(key, clonedValue.Interface())
} else {
clonedTemplate.Variables.Set(key, nil)
}
})
return clonedTemplate
}
func cloneTemplateValue(value reflect.Value, visited map[cloneVisit]reflect.Value) reflect.Value {
if !value.IsValid() {
return value
}
switch value.Kind() {
case reflect.Interface:
if value.IsNil() {
return reflect.Zero(value.Type())
}
cloned := cloneTemplateValue(value.Elem(), visited)
result := reflect.New(value.Type()).Elem()
result.Set(cloned)
return result
case reflect.Pointer:
if value.IsNil() {
return reflect.Zero(value.Type())
}
visit := cloneVisit{typeOf: value.Type(), pointer: value.Pointer()}
if cloned, ok := visited[visit]; ok {
return cloned
}
cloned := reflect.New(value.Type().Elem())
visited[visit] = cloned
cloned.Elem().Set(cloneTemplateValue(value.Elem(), visited))
return cloned
case reflect.Struct:
cloned := reflect.New(value.Type()).Elem()
cloned.Set(value)
for i := 0; i < value.NumField(); i++ {
if value.Type().Field(i).IsExported() {
cloned.Field(i).Set(cloneTemplateValue(value.Field(i), visited))
}
}
return cloned
case reflect.Slice:
if value.IsNil() {
return reflect.Zero(value.Type())
}
visit := cloneVisit{
typeOf: value.Type(),
pointer: value.Pointer(),
length: value.Len(),
capacity: value.Cap(),
}
if cloned, ok := visited[visit]; ok {
return cloned
}
cloned := reflect.MakeSlice(value.Type(), value.Len(), value.Len())
visited[visit] = cloned
for i := 0; i < value.Len(); i++ {
cloned.Index(i).Set(cloneTemplateValue(value.Index(i), visited))
}
return cloned
case reflect.Map:
if value.IsNil() {
return reflect.Zero(value.Type())
}
visit := cloneVisit{typeOf: value.Type(), pointer: value.Pointer()}
if cloned, ok := visited[visit]; ok {
return cloned
}
cloned := reflect.MakeMapWithSize(value.Type(), value.Len())
visited[visit] = cloned
iterator := value.MapRange()
for iterator.Next() {
key := cloneTemplateValue(iterator.Key(), visited)
item := cloneTemplateValue(iterator.Value(), visited)
cloned.SetMapIndex(key, item)
}
return cloned
case reflect.Array:
cloned := reflect.New(value.Type()).Elem()
for i := 0; i < value.Len(); i++ {
cloned.Index(i).Set(cloneTemplateValue(value.Index(i), visited))
}
return cloned
default:
return value
}
}
+104
View File
@@ -0,0 +1,104 @@
package templates
import (
"os"
"path/filepath"
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/disk"
"github.com/projectdiscovery/nuclei/v3/pkg/utils"
"github.com/stretchr/testify/require"
)
func TestCloneTemplatePreservesCleanTemplateIsolation(t *testing.T) {
templatePath := filepath.Join(t.TempDir(), "multi-protocol.yaml")
require.NoError(t, os.WriteFile(templatePath, []byte(`id: clone-isolation
info:
name: Clone isolation
author: pdteam
severity: info
variables:
shared: original
dns:
- name: "{{FQDN}}"
type: A
http:
- method: GET
path:
- "{{BaseURL}}"
matchers:
- type: word
words:
- original
`), 0o600))
parsed, err := NewParser().ParseTemplate(templatePath, disk.NewCatalog(""))
require.NoError(t, err)
original := parsed.(*Template)
cloned := cloneTemplate(original)
require.NotSame(t, original, cloned)
require.NotSame(t, original.RequestsDNS[0], cloned.RequestsDNS[0])
require.NotSame(t, original.RequestsHTTP[0], cloned.RequestsHTTP[0])
require.Same(t, cloned.RequestsDNS[0], cloned.RequestsQueue[0])
require.Same(t, cloned.RequestsHTTP[0], cloned.RequestsQueue[1])
cloned.RequestsHTTP[0].Matchers[0].Words[0] = "changed"
cloned.Variables.Set("shared", "changed")
require.Equal(t, "original", original.RequestsHTTP[0].Matchers[0].Words[0])
require.Equal(t, map[string]interface{}{"shared": "original"}, original.Variables.GetAll())
}
func TestCloneTemplatePreservesMutableAliases(t *testing.T) {
sharedMap := map[string]interface{}{"value": "original"}
sharedSlice := []interface{}{"original"}
original := &Template{
Constants: map[string]interface{}{
"map": sharedMap,
"first-slice": sharedSlice,
"next-slice": sharedSlice,
},
}
original.Variables.InsertionOrderedStringMap = *utils.NewEmptyInsertionOrderedStringMap(1)
original.Variables.Set("map", sharedMap)
cloned := cloneTemplate(original)
clonedMap := cloned.Constants["map"].(map[string]interface{})
clonedVariableMap := cloned.Variables.GetAll()["map"].(map[string]interface{})
clonedFirstSlice := cloned.Constants["first-slice"].([]interface{})
clonedNextSlice := cloned.Constants["next-slice"].([]interface{})
clonedMap["value"] = "changed"
clonedFirstSlice[0] = "changed"
require.Equal(t, "changed", clonedVariableMap["value"])
require.Equal(t, "changed", clonedNextSlice[0])
require.Equal(t, "original", sharedMap["value"])
require.Equal(t, "original", sharedSlice[0])
}
func TestCloneTemplateHandlesMutableCycles(t *testing.T) {
cyclicMap := make(map[string]interface{})
cyclicMap["self"] = cyclicMap
cyclicSlice := make([]interface{}, 1)
cyclicSlice[0] = cyclicSlice
original := &Template{Constants: map[string]interface{}{
"map": cyclicMap,
"slice": cyclicSlice,
}}
cloned := cloneTemplate(original)
clonedMap := cloned.Constants["map"].(map[string]interface{})
clonedMap["self"].(map[string]interface{})["changed"] = true
clonedSlice := cloned.Constants["slice"].([]interface{})
clonedSlice[0].([]interface{})[0] = "changed"
require.True(t, clonedMap["changed"].(bool))
require.Equal(t, "changed", clonedSlice[0])
require.NotContains(t, cyclicMap, "changed")
require.IsType(t, []interface{}{}, cyclicSlice[0])
}
+65 -20
View File
@@ -1,6 +1,7 @@
package templates
import (
"bytes"
"crypto/sha256"
"fmt"
"io"
@@ -104,19 +105,36 @@ func updateRequestOptions(template *Template) {
// parseFromSource parses a template from source with caching support
func parseFromSource(filePath string, preprocessor Preprocessor, options *protocols.ExecutorOptions, parser *Parser) (*Template, error) {
reader, err := utils.ReaderFromPathOrURL(filePath, options.Catalog)
if err != nil {
return nil, err
}
defer func() {
_ = reader.Close()
}()
options = options.Copy()
options.TemplatePath = filePath
template, err := ParseTemplateFromReader(reader, preprocessor, options)
var template *Template
var err error
if !options.DoNotCache {
parsed, raw, cachedErr := parser.parsedTemplatesCache.Has(filePath)
if cachedErr != nil {
return nil, cachedErr
}
if parsed != nil && len(raw) > 0 {
template, err = parseCachedTemplate(parsed, raw, preprocessor, options)
}
}
if template == nil && err == nil {
reader, openErr := utils.ReaderFromPathOrURL(filePath, options.Catalog)
if openErr != nil {
return nil, openErr
}
defer func() {
_ = reader.Close()
}()
template, err = ParseTemplateFromReader(reader, preprocessor, options)
}
if err != nil {
return nil, err
}
@@ -154,6 +172,25 @@ func parseFromSource(filePath string, preprocessor Preprocessor, options *protoc
return template, nil
}
func parseCachedTemplate(cached *Template, data []byte, preprocessor Preprocessor, options *protocols.ExecutorOptions) (*Template, error) {
if hasTemplatePreprocessor(data, preprocessor) {
return ParseTemplateFromReader(bytes.NewReader(data), preprocessor, options)
}
template, err := prepareTemplate(cloneTemplate(cached), options)
if err != nil {
return nil, err
}
if err := verifyAndCompileTemplate(template, data); err != nil {
return nil, err
}
if !template.Verified && len(template.Workflows) == 0 && config.DefaultConfig.LogAllEvents {
gologger.DefaultLogger.Print().Msgf("[%v] Template %s is not signed or tampered\n", aurora.Yellow("WRN").String(), template.ID)
}
return template, nil
}
// getParser returns a cached parser instance
func getParser(options *protocols.ExecutorOptions) *Parser {
parser, ok := options.Parser.(*Parser)
@@ -165,7 +202,6 @@ func getParser(options *protocols.ExecutorOptions) *Parser {
}
// Parse parses a yaml request template file
// TODO make sure reading from the disk the template parsing happens once: see parsers.ParseTemplate vs templates.Parse
func Parse(filePath string, preprocessor Preprocessor, options *protocols.ExecutorOptions) (*Template, error) {
parser := getParser(options)
@@ -439,14 +475,8 @@ func ParseTemplateFromReader(reader io.Reader, preprocessor Preprocessor, option
// a preprocessor is a variable like
// {{randstr}} which is replaced before unmarshalling
// as it is known to be a random static value per template
hasPreprocessor := false
hasPreprocessor := hasTemplatePreprocessor(data, preprocessor)
allPreprocessors := getPreprocessors(preprocessor)
for _, preprocessor := range allPreprocessors {
if preprocessor.Exists(data) {
hasPreprocessor = true
break
}
}
if !hasPreprocessor {
// if no preprocessors exists parse template and exit
@@ -498,6 +528,16 @@ func ParseTemplateFromReader(reader io.Reader, preprocessor Preprocessor, option
return template, nil
}
func hasTemplatePreprocessor(data []byte, preprocessor Preprocessor) bool {
for _, candidate := range getPreprocessors(preprocessor) {
if candidate.Exists(data) {
return true
}
}
return false
}
// parseTemplate parses the template and applies verification.
func parseTemplate(data []byte, srcOptions *protocols.ExecutorOptions) (*Template, error) {
template, err := parseTemplateNoVerify(data, srcOptions)
@@ -522,8 +562,6 @@ func verifyAndCompileTemplate(template *Template, data []byte) error {
// parseTemplateNoVerify parses the template without applying any verification.
func parseTemplateNoVerify(data []byte, srcOptions *protocols.ExecutorOptions) (*Template, error) {
// Create a copy of the options specifically for this template
options := srcOptions.Copy()
template := &Template{}
var err error
@@ -544,6 +582,13 @@ func parseTemplateNoVerify(data []byte, srcOptions *protocols.ExecutorOptions) (
return nil, errkit.Wrapf(err, "failed to parse %s", template.Path)
}
return prepareTemplate(template, srcOptions)
}
func prepareTemplate(template *Template, srcOptions *protocols.ExecutorOptions) (*Template, error) {
// Create a copy of the options specifically for this template.
options := srcOptions.Copy()
if utils.IsBlank(template.Info.Name) {
return nil, errors.New("no template name field provided")
}
+38
View File
@@ -6,6 +6,7 @@ import (
"os"
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols"
"github.com/projectdiscovery/nuclei/v3/pkg/templates"
)
@@ -24,6 +25,43 @@ func BenchmarkParse(b *testing.B) {
}
}
func BenchmarkParseAcrossEngineLocalCaches(b *testing.B) {
const engineCount = 5
filePath := "tests/match-1.yaml"
setup()
sharedParser := templates.NewParser()
_, err := sharedParser.ParseTemplate(filePath, executerOpts.Catalog)
if err != nil {
b.Fatalf("could not warm parsed template cache: %s", err)
}
engineOptions := make([]*protocols.ExecutorOptions, engineCount)
engineParsers := make([]*templates.Parser, engineCount)
for i := range engineCount {
engineParsers[i] = templates.NewParserWithParsedCache(sharedParser.Cache())
engineOptions[i] = executerOpts.Copy()
engineOptions[i].Parser = engineParsers[i]
}
b.ResetTimer()
b.ReportAllocs()
for b.Loop() {
for _, options := range engineOptions {
_, err := templates.Parse(filePath, nil, options)
if err != nil {
b.Fatalf("could not parse template: %s", err)
}
}
for _, parser := range engineParsers {
parser.CompiledCache().Purge()
}
}
}
func BenchmarkParseTemplateFromReader(b *testing.B) {
filePath := "tests/match-1.yaml"
+106
View File
@@ -11,6 +11,8 @@ import (
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
@@ -597,6 +599,110 @@ javascript:
require.True(t, cached.Options.Verified)
}
func TestParseReusesSharedParsedTemplateAcrossEngineCaches(t *testing.T) {
var sourceReads atomic.Int32
templateSource := `id: shared-parsed-template
info:
name: Shared parsed template
author: pdteam
severity: info
http:
- method: GET
path:
- "{{BaseURL}}"
body: |
first line
second line
`
server := httptest.NewServer(netHttp.HandlerFunc(func(w netHttp.ResponseWriter, _ *netHttp.Request) {
sourceReads.Add(1)
_, _ = w.Write([]byte(templateSource))
}))
t.Cleanup(server.Close)
setup()
templatePath := server.URL + "/template.yaml"
sharedParser := templates.NewParser()
_, err := sharedParser.ParseTemplate(templatePath, executerOpts.Catalog)
require.NoError(t, err)
compiledTemplates := make([]*templates.Template, 5)
parseErrors := make([]error, len(compiledTemplates))
var waitGroup sync.WaitGroup
for i := range compiledTemplates {
engineOptions := executerOpts.Copy()
engineOptions.Parser = templates.NewParserWithParsedCache(sharedParser.Cache())
engineOptions.TemplateVerificationCallback = func(string) *protocols.TemplateVerification {
return trustedVerificationForTest(templateSource)
}
waitGroup.Add(1)
go func() {
defer waitGroup.Done()
compiledTemplates[i], parseErrors[i] = templates.Parse(templatePath, nil, engineOptions)
}()
}
waitGroup.Wait()
for i := range compiledTemplates {
require.NoError(t, parseErrors[i])
require.True(t, compiledTemplates[i].Verified)
}
require.Equal(t, int32(1), sourceReads.Load())
cachedTemplate, err := sharedParser.Cache().Get(templatePath)
require.NoError(t, err)
require.NotNil(t, cachedTemplate)
require.NotContains(t, cachedTemplate.RequestsHTTP[0].Body, "\r\n")
for i := 1; i < len(compiledTemplates); i++ {
require.NotSame(t, compiledTemplates[0], compiledTemplates[i])
require.NotSame(t, compiledTemplates[0].RequestsHTTP[0], compiledTemplates[i].RequestsHTTP[0])
require.NotSame(t, compiledTemplates[0].RequestsHTTP[0].Options(), compiledTemplates[i].RequestsHTTP[0].Options())
}
require.NotSame(t, cachedTemplate.RequestsHTTP[0], compiledTemplates[0].RequestsHTTP[0])
}
func TestParseSharedTemplatePreservesRuntimePreprocessing(t *testing.T) {
var sourceReads int
templateSource := `id: shared-preprocessed-template
info:
name: Shared preprocessed template
author: pdteam
severity: info
http:
- method: GET
path:
- "{{BaseURL}}/{{randstr}}"
`
server := httptest.NewServer(netHttp.HandlerFunc(func(w netHttp.ResponseWriter, _ *netHttp.Request) {
sourceReads++
_, _ = w.Write([]byte(templateSource))
}))
t.Cleanup(server.Close)
setup()
templatePath := server.URL + "/template.yaml"
sharedParser := templates.NewParser()
_, err := sharedParser.ParseTemplate(templatePath, executerOpts.Catalog)
require.NoError(t, err)
compiledPaths := make([]string, 2)
for i := range compiledPaths {
engineOptions := executerOpts.Copy()
engineOptions.Parser = templates.NewParserWithParsedCache(sharedParser.Cache())
compiled, parseErr := templates.Parse(templatePath, nil, engineOptions)
require.NoError(t, parseErr)
compiledPaths[i] = compiled.RequestsHTTP[0].Path[0]
require.NotContains(t, compiledPaths[i], "{{randstr}}")
}
require.Equal(t, 1, sourceReads)
require.NotEqual(t, compiledPaths[0], compiledPaths[1])
}
func Test_WrongTemplate(t *testing.T) {
setup()
+23 -30
View File
@@ -1,6 +1,7 @@
package templates
import (
"bytes"
"fmt"
"io"
"strings"
@@ -20,10 +21,9 @@ type Parser struct {
ShouldValidate bool
NoStrictSyntax bool
// parsedTemplatesCache stores lightweight parsed template structures
// (without raw bytes).
// Used for validation and filtering. This cache can be copied safely
// between ephemeral instances.
// parsedTemplatesCache stores clean parsed template structures used for
// validation, filtering, and engine-local compilation. Entries also retain
// source bytes when they exactly match the parsed structure.
parsedTemplatesCache *Cache
// compiledTemplatesCache stores fully compiled templates with all protocol
@@ -149,49 +149,38 @@ func (p *Parser) ParseTemplate(templatePath string, catalog catalog.Catalog) (an
_ = reader.Close()
}()
// For local YAML files, check if preprocessing is needed
var data []byte
sourceData, err := io.ReadAll(reader)
if err != nil {
return nil, err
}
data := sourceData
cacheSource := true
// For local YAML files, check if preprocessing is needed.
if fileutil.FileExists(templatePath) && config.GetTemplateFormatFromExt(templatePath) == config.YAML {
data, err = io.ReadAll(reader)
if err != nil {
return nil, err
}
data, err = yamlutil.PreProcess(data, templatePath)
if err != nil {
return nil, err
}
cacheSource = bytes.Equal(data, sourceData)
}
template := &Template{}
switch config.GetTemplateFormatFromExt(templatePath) {
case config.JSON:
if data == nil {
data, err = io.ReadAll(reader)
if err != nil {
return nil, err
}
}
if p.NoStrictSyntax {
err = json.Unmarshal(data, template)
} else {
err = template.unmarshalJSONStrict(data)
}
case config.YAML:
if data != nil {
// Already read and preprocessed
if p.NoStrictSyntax {
err = yamlutil.Unmarshal(data, template)
} else {
err = yamlutil.UnmarshalStrict(data, template)
}
if p.NoStrictSyntax {
err = yamlutil.Unmarshal(data, template)
} else {
// Stream directly from reader
decoder := yamlutil.NewDecoder(reader)
if !p.NoStrictSyntax {
decoder.SetStrict(true)
}
err = decoder.Decode(template)
err = yamlutil.UnmarshalStrict(data, template)
}
default:
err = fmt.Errorf("failed to identify template format expected JSON or YAML but got %v", templatePath)
@@ -200,7 +189,11 @@ func (p *Parser) ParseTemplate(templatePath string, catalog catalog.Catalog) (an
return nil, err
}
p.parsedTemplatesCache.StoreWithoutRaw(templatePath, template, nil)
if cacheSource {
p.parsedTemplatesCache.Store(templatePath, template, sourceData, nil)
} else {
p.parsedTemplatesCache.StoreWithoutRaw(templatePath, template, nil)
}
return template, nil
}
+1 -1
View File
@@ -351,7 +351,7 @@ func (template *Template) UnmarshalYAML(unmarshal func(interface{}) error) error
*template = Template(*alias)
if !ReTemplateID.MatchString(template.ID) {
return errkit.New("template id must match expression %v", ReTemplateID, "tag", "invalid_template")
return errkit.New("template id must match expression "+ReTemplateID.String(), "tag", "invalid_template")
}
info := template.Info
+3 -2
View File
@@ -7,6 +7,7 @@ import (
"github.com/projectdiscovery/goja"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols"
"github.com/projectdiscovery/nuclei/v3/pkg/tmplexec/utils"
"github.com/projectdiscovery/utils/errkit"
mapsutil "github.com/projectdiscovery/utils/maps"
)
@@ -35,7 +36,7 @@ func (f *FlowExecutor) requestExecutor(runtime *goja.Runtime, reqMap mapsutil.Ma
req := f.allProtocols[opts.protoName][index]
// transform input if required
inputItem := f.ctx.Input.Clone()
if f.options.InputHelper != nil && f.ctx.Input.MetaInput.Input != "" {
if f.options.InputHelper != nil && f.ctx.Input.MetaInput.Input != "" && !utils.HasOfflineHTTPResponse(f.ctx.Input.MetaInput, req.Type()) {
if inputItem.MetaInput.Input = f.options.InputHelper.Transform(inputItem.MetaInput.Input, req.Type()); inputItem.MetaInput.Input == "" {
f.ctx.LogError(fmt.Errorf("failed to transform input for protocol %s", req.Type()))
return false
@@ -72,7 +73,7 @@ func (f *FlowExecutor) requestExecutor(runtime *goja.Runtime, reqMap mapsutil.Ma
}
// transform input if required
inputItem := f.ctx.Input.Clone()
if f.options.InputHelper != nil && f.ctx.Input.MetaInput.Input != "" {
if f.options.InputHelper != nil && f.ctx.Input.MetaInput.Input != "" && !utils.HasOfflineHTTPResponse(f.ctx.Input.MetaInput, req.Type()) {
if inputItem.MetaInput.Input = f.options.InputHelper.Transform(inputItem.MetaInput.Input, req.Type()); inputItem.MetaInput.Input == "" {
f.ctx.LogError(fmt.Errorf("failed to transform input for protocol %s", req.Type()))
return false
+1 -1
View File
@@ -52,7 +52,7 @@ func (g *Generic) ExecuteWithResults(ctx *scan.ScanContext) error {
}
inputItem := ctx.Input.Clone()
if g.options.InputHelper != nil && ctx.Input.MetaInput.Input != "" {
if g.options.InputHelper != nil && ctx.Input.MetaInput.Input != "" && !utils.HasOfflineHTTPResponse(ctx.Input.MetaInput, req.Type()) {
if inputItem.MetaInput.Input = g.options.InputHelper.Transform(inputItem.MetaInput.Input, req.Type()); inputItem.MetaInput.Input == "" {
return nil
}
+1 -1
View File
@@ -95,7 +95,7 @@ func (m *MultiProtocol) ExecuteWithResults(ctx *scan.ScanContext) error {
default:
}
inputItem := ctx.Input.Clone()
if m.options.InputHelper != nil && ctx.Input.MetaInput.Input != "" {
if m.options.InputHelper != nil && ctx.Input.MetaInput.Input != "" && !utils.HasOfflineHTTPResponse(ctx.Input.MetaInput, req.Type()) {
if inputItem.MetaInput.Input = m.options.InputHelper.Transform(inputItem.MetaInput.Input, req.Type()); inputItem.MetaInput.Input == "" {
return nil
}
+14
View File
@@ -4,6 +4,8 @@ import (
"strings"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
templateTypes "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
mapsutil "github.com/projectdiscovery/utils/maps"
)
@@ -32,3 +34,15 @@ func FillPreviousEvent(reqID string, event *output.InternalWrappedEvent, previou
_ = previous.Set(builder.String(), v)
}
}
// HasOfflineHTTPResponse reports whether this input already carries a response
// body for offline/passive HTTP matching (e.g. burp/jsonl/yaml exports).
// When true, InputHelper filepath transforms must be skipped so URL-shaped
// MetaInput.Input values are not discarded.
func HasOfflineHTTPResponse(meta *contextargs.MetaInput, proto templateTypes.ProtocolType) bool {
if meta == nil || proto != templateTypes.OfflineHTTPProtocol {
return false
}
rr := meta.ReqResp
return rr != nil && rr.Response != nil && strings.TrimSpace(rr.Response.Raw) != ""
}
+24
View File
@@ -0,0 +1,24 @@
package utils
import (
"testing"
"github.com/stretchr/testify/require"
"github.com/projectdiscovery/nuclei/v3/pkg/input/types"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
templateTypes "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
)
func TestHasOfflineHTTPResponse(t *testing.T) {
meta := contextargs.NewMetaInput()
require.False(t, HasOfflineHTTPResponse(meta, templateTypes.OfflineHTTPProtocol))
meta.ReqResp = &types.RequestResponse{}
require.False(t, HasOfflineHTTPResponse(meta, templateTypes.OfflineHTTPProtocol))
meta.ReqResp.Response = &types.HttpResponse{Raw: "HTTP/1.1 200 OK\r\n\r\n"}
require.True(t, HasOfflineHTTPResponse(meta, templateTypes.OfflineHTTPProtocol))
require.False(t, HasOfflineHTTPResponse(meta, templateTypes.HTTPProtocol))
require.False(t, HasOfflineHTTPResponse(nil, templateTypes.OfflineHTTPProtocol))
}
+13 -4
View File
@@ -10,15 +10,24 @@ import (
"github.com/rs/xid"
)
// Default resume file
const DefaultResumeFileName = "resume-%s.cfg"
// DefaultResumeFileName is the default filename for the resume file.
const (
DefaultResumeFileName = "resume-%s.cfg"
defaultCrashResumeFileName = "crash-resume-file-%s.dump"
)
func DefaultResumeFilePath() string {
cacheDir := config.DefaultConfig.GetCacheDir()
resumeFile := filepath.Join(cacheDir, fmt.Sprintf(DefaultResumeFileName, xid.New().String()))
stateDir := config.DefaultConfig.GetStateDir()
resumeFile := filepath.Join(stateDir, fmt.Sprintf(DefaultResumeFileName, xid.New().String()))
return resumeFile
}
// DefaultCrashResumeFilePath returns the generated crash checkpoint path.
func DefaultCrashResumeFilePath(id string) string {
return filepath.Join(config.DefaultConfig.GetStateDir(), fmt.Sprintf(defaultCrashResumeFileName, id))
}
// ResumeCfg contains the scan progression
type ResumeCfg struct {
sync.RWMutex
+24
View File
@@ -0,0 +1,24 @@
package types
import (
"path/filepath"
"strings"
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
)
func TestGeneratedResumePathsUseStateDirectory(t *testing.T) {
stateDir := t.TempDir()
oldStateDir := config.DefaultConfig.GetStateDir()
config.DefaultConfig.SetStateDir(stateDir)
t.Cleanup(func() { config.DefaultConfig.SetStateDir(oldStateDir) })
resumePath := DefaultResumeFilePath()
if filepath.Dir(resumePath) != stateDir || !strings.HasPrefix(filepath.Base(resumePath), "resume-") {
t.Fatalf("resume path = %q, want generated file under %q", resumePath, stateDir)
}
if got, want := DefaultCrashResumeFilePath("dump"), filepath.Join(stateDir, "crash-resume-file-dump.dump"); got != want {
t.Fatalf("crash resume path = %q, want %q", got, want)
}
}
+4 -3
View File
@@ -27,8 +27,9 @@ var httpFirstSchemes = []string{
"https",
}
// determineSchemeOrder for the input
func determineSchemeOrder(input string) []string {
// DetermineSchemeOrder returns the schemes to try for a scheme-less input,
// most likely first
func DetermineSchemeOrder(input string) []string {
if _, port, err := net.SplitHostPort(input); err == nil {
// if input has port that is commonly used for HTTP, return http then https
if sliceutil.Contains(commonHttpPorts, port) {
@@ -51,7 +52,7 @@ func determineSchemeOrder(input string) []string {
// If none succeeds, probing is abandoned for such URLs.
func ProbeURL(input string, httpxclient *httpx.HTTPX) string {
normalizedInput := normalizeProbeInput(input)
schemes := determineSchemeOrder(normalizedInput)
schemes := DetermineSchemeOrder(normalizedInput)
for _, scheme := range schemes {
formedURL := fmt.Sprintf("%s://%s", scheme, normalizedInput)
req, err := httpxclient.NewRequest(http.MethodHead, formedURL)
+3 -3
View File
@@ -30,7 +30,7 @@ func TestDetermineSchemeOrder(t *testing.T) {
for _, tc := range tests {
t.Run(tc.input, func(t *testing.T) {
actual := determineSchemeOrder(tc.input)
actual := DetermineSchemeOrder(tc.input)
require.Equal(t, tc.expected, actual)
})
}
@@ -52,7 +52,7 @@ func TestDetermineSchemeOrderWithHighPorts(t *testing.T) {
for _, tc := range tests {
t.Run(tc.input, func(t *testing.T) {
actual := determineSchemeOrder(tc.input)
actual := DetermineSchemeOrder(tc.input)
require.Equal(t, tc.expected, actual)
})
}
@@ -69,7 +69,7 @@ func TestDetermineSchemeOrderAmbiguousIPv6Literal(t *testing.T) {
for _, tc := range tests {
t.Run(tc.input, func(t *testing.T) {
actual := determineSchemeOrder(normalizeProbeInput(tc.input))
actual := DetermineSchemeOrder(normalizeProbeInput(tc.input))
require.Equal(t, tc.expected, actual)
})
}