Compare commits

...
Author SHA1 Message Date
Mzack9999 40321dd68a Merge dev 2026-10-27 16:09:42 +01:00
Mzack9999 abcdcf19a4 Merge pull request #7940 from mazzz3r/feat/info-placeholders
feat(templates): fill placeholders in info description, impact and remediation
2026-10-27 15:21:09 +01:00
Michael Rodger 9dc004926d fix(trackers): gitea reporter pagination (#7945) 2026-10-27 14:11:01 +07:00
Mzack9999 3d2f411e1d Merge pull request #7925 from projectdiscovery/dwisiswant0/fix/offlinehttp/read-status-lines-by-position-not-text-search
fix(offlinehttp): read status lines by position, not text search
2026-10-26 21:38:57 +01:00
Mzack9999 763e490f42 merge dev 2026-10-26 21:10:10 +01:00
Mzack9999 421ec70686 Merge pull request #7947 from projectdiscovery/dwisiswant0/fix/trackers/follow-x-next-page-in-gitlab-duplicate-search
fix(trackers): follow X-Next-Page in gitlab duplicate search
2026-10-26 20:47:55 +01:00
Mzack9999 55d17f5a29 Pin baseline 2026-10-26 16:29:39 +01:00
Mzack9999 b7e66ae57c Fix test 2026-10-26 15:57:03 +01:00
Dwi Siswanto aac1427cdc fix(trackers): follow X-Next-Page in gitlab duplicate search
The GitLab tracker ended the duplicate search at the first page holding
fewer issues than duplicate-issue-page-size. GitLab caps per_page at
100, so with a page size above 100 every page looked like the last one.
The tracker read only the first 100 search results and opened a new
issue when the match came later.

The client library already parses X-Next-Page into Response.NextPage.
Follow it instead of guessing from the page length. This also drops the
empty-page request the old check sent when the last page was full.

Fixes: #7946
Refs: #7944
Signed-off-by: Dwi Siswanto <[email protected]>
2026-10-26 21:10:51 +07:00
Mzack9999 9aef358094 Score llm 2026-10-26 12:26:43 +01:00
Mzack9999 b62180c641 Merge dev 2026-10-26 12:04:12 +01:00
Mzack9999 1d3f1f5bc2 Gate interactsh 2026-10-26 11:32:06 +01:00
Mzack9999 263bc13412 Merge pull request #7941 from TassioSales/fix/passive-raw-response-http-body
fix(offlinehttp): handle request body and unversioned HTTP/2 in passive mode (#7924)
2026-10-26 07:35:25 +01:00
dependabot[bot] 660b7438d8 chore(deps): bump crate-ci/typos (#7943)
Bumps the workflows group with 1 update: [crate-ci/typos](https://github.com/crate-ci/typos).


Updates `crate-ci/typos` from 1.49.0 to 1.51.1
- [Release notes](https://github.com/crate-ci/typos/releases)
- [Changelog](https://github.com/crate-ci/typos/blob/main/CHANGELOG.md)
- [Commits](https://github.com/crate-ci/typos/compare/v1.49.0...v1.51.1)

---
updated-dependencies:
- dependency-name: crate-ci/typos
  dependency-version: 1.51.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: workflows
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-26 06:27:48 +00:00
Mzack9999 4b3c1633ee Merge dev 2026-10-26 01:00:32 +01:00
Mzack9999 d2f862edf5 Merge origin 2026-10-26 00:33:21 +01:00
TassioSales aa761b9c33 fix(offlinehttp): handle request body and unversioned HTTP/2 in passive mode (#7924)
Signed-off-by: TassioSales <[email protected]>
2026-10-25 18:20:30 -03:00
Mzack9999 731d438319 Merge pull request #7690 from IgorDaniel45/fix-threadsafe-compiled-template-cache
fix: reuse compiled templates in thread-safe engine
2026-10-25 22:09:59 +01:00
Mzack9999 a85a545323 Merge pull request #7655 from hu-matin/translation-fa
docs: add Persian(FA) translation for README
2026-10-25 22:02:43 +01:00
Mzack9999 5ea11e5f43 Merge pull request #7784 from root-Manas/feat/openapi-oauth-security
feat(openapi): support OAuth2 and OpenID Connect access tokens
2026-10-25 22:02:38 +01:00
Mzack9999 230a17d4fa Merge pull request #7680 from chauhand2463/feat/http-tls-key-exchange
feat (http) : expose TLS key exchange metadata
2026-10-25 22:02:33 +01:00
Mzack9999 d2499b74da trim cache 2026-10-25 21:46:31 +01:00
Mzack9999 93483eaff9 merge dev 2026-10-25 21:46:30 +01:00
Mzack9999 a81624e595 fix readme 2026-10-25 21:20:18 +01:00
Mzack9999 a9df3eff9e merge dev 2026-10-25 21:19:34 +01:00
Mzack9999 08820abff6 merge dev 2026-10-25 21:18:56 +01:00
Mzack9999 fa3fa92092 merge dev 2026-10-25 21:15:30 +01:00
Mzack9999 be7b2648ef document ssl 2026-10-25 21:15:30 +01:00
mazzz3r c4552c6011 feat(templates): fill placeholders in info description, impact and remediation
Description, impact and remediation can reference values of the finding
with the usual {{...}} syntax: named extractors, payload values of the
matching request, variables, DSL helpers, and the result fields host,
port, ip, url, scheme, path, matched-at, matcher-name and
extractor-name, which win over values of the same name. Name, severity,
tags and classification stay static, since filtering, grouping and
issue trackers key on them.

MakeDefaultResultEvent renders each result's copy of the info, so every
protocol gets it and the template never changes. Each field goes
through render.Render without scan options, like request and matcher
text, so nested placeholders and helpers behave the same and network
helpers stay denied. A field whose expression fails keeps the
template's text.

When placeholders change the text, the result keeps the written info in
TemplateInfo (json:"-" bson:"-"):
- the writer runs the -redact patterns over the changed fields, as it
  does over the raw request and response;
- SARIF builds the shared rule from the written info and puts each
  finding's filled text, capped at 4096 characters, in its result
  message.

Info is filled only from a template's own event. Templates with info
placeholders are not clustered, since a cluster's shared event holds the
first template's variables and extracted values, and a global matcher
fills its info from the result fields alone.
2026-10-25 19:37:42 +00:00
Mzack9999 47398198a0 Merge pull request #7770 from hodeswildsmith455-boop/orcarouter/task-7500
Select OrcaRouter as an LLM provider
2026-10-25 19:29:59 +01:00
Mzack9999 f846a5553d Merge pull request #7863 from mazzz3r/feat/target-line-ip
feat(input): pin a target to an ip with the ip key of target lines
2026-10-25 18:56:19 +01:00
Mzack9999 39c5d8764e Merge pull request #7928 from projectdiscovery/fix/http-large-request-body-memory
http: stop copying large request bodies (fixes OOM on multi-MB body templates)
2026-10-25 18:54:34 +01:00
Mzack9999 93b64043b1 merge dev 2026-10-25 18:27:19 +01:00
Mzack9999 7d6b2fef6f bump utils 2026-10-25 18:27:03 +01:00
Mzack9999 445e6ab972 Merge pull request #7753 from gnuletik/perf/lazy-full-response
perf(http): build the full response only when an operator reads it
2026-10-25 18:24:42 +01:00
Mzack9999 7f8abe2c0f Merge pull request #7678 from gnuletik/fix/cancellable-template-and-js-waits
fix(core): honour the scan context when acquiring worker slots and JS runtimes
2026-10-25 18:20:28 +01:00
Mzack9999 bfd8d4585c slim provider 2026-10-25 18:18:28 +01:00
Mzack9999 1c3cc97748 merge dev 2026-10-25 18:17:48 +01:00
Georgii Butakov 947a2a9a2a fix(http): dial pipelined requests through the pinned ip
executeTurboHTTP dialed pipelined connections with context.Background(),
so a pipelined request of a pinned target resolved the host instead of
connecting to its ip. It now dials with the input context, which carries
the pin. protocols/http/target-line-ip-pipeline.yaml runs the pinned
target test with pipeline: true, and gets no result without the change.

ProbeKey joins the input and the ip with a NUL byte, which no input
holds, instead of "@", which a scheme-less input may hold itself.
2026-10-25 17:12:13 +00:00
Mzack9999 8e1e613a15 merge dev 2026-10-25 18:10:14 +01:00
Mzack9999 5a339a8da1 fix lint 2026-10-25 17:56:27 +01:00
Georgii Butakov 779868985d fix(input): do not give a pinned input the probe result of its host
A pinned input whose ip had no probe result fell back to Transform,
which takes the result probed for the input without an ip, so the
pinned target could get the scheme of another server. It now gets the
conversion of an input that was not probed.
2026-10-25 16:50:30 +00:00
Mzack9999 c4d933d165 merge dev 2026-10-25 17:50:20 +01:00
Mzack9999 f58b21e6c3 merge dev 2026-10-25 17:45:44 +01:00
Mzack9999 71805f5093 Merge pull request #7857 from AdamMagued/fix-issue-7829
perf(protocols/http): defer building FullResponseString when unused by operators
2026-10-25 17:33:54 +01:00
Mzack9999 1086d5d391 keep matchers 2026-10-25 17:32:15 +01:00
Mzack9999 75434a01e9 Merge pull request #7747 from MGreif/mgr-interact-client-interface
Improvement: implementation of interface for interactsh
2026-10-25 17:24:11 +01:00
Mzack9999 e9f062f600 Merge pull request #7861 from mazzz3r/feat/task-based-progress
feat(progress): report completion by executed templates
2026-10-25 17:19:28 +01:00
root-Manas 5bd7c79be4 test(openapi): check request body cleanup errors 2026-10-25 21:45:04 +05:30
Mzack9999 8f31e9d24d Merge pull request #7612 from DPS0340/fix/regex-group-out-of-range
fix(extractors): reject a regex group the pattern cannot produce
2026-10-25 17:10:25 +01:00
Mzack9999 e17273911b Merge pull request #7914 from mazzz3r/feat/payload-random-order
feat(payloads): iterate payload combinations in a random order
2026-10-25 17:10:19 +01:00
Mzack9999 e2840f3644 fix lint 2026-10-25 17:09:07 +01:00
Georgii Butakov be6e32d5ae fix(runner): keep the http probe result of each pinned ip
A target without a scheme is probed through its pinned ip, but the
result was stored under the target alone, so the last probe of a host
pinned to several ips set the scheme for all of them, and a server that
answered only on the other scheme was not scanned. The result is now
stored under the target and its ip, and TransformPinned looks it up for
a pinned input.
2026-10-25 16:05:09 +00:00
Mzack9999 86d9a9bd54 Merge pull request #7882 from NitinReddy-A/feat/5608-templates-update-channel
feat(installer): add templates update channel
2026-10-25 16:57:00 +01:00
Mzack9999 7c2b74424e Merge pull request #7618 from aviadavi/feat/dsl-in-custom-headers
feat: evaluate DSL helpers in -H values per request
2026-10-25 16:56:54 +01:00
Mzack9999 f1e79cd82b fix response 2026-10-25 16:54:03 +01:00
Mzack9999 4ae1faa64f merge dev 2026-10-25 16:53:53 +01:00
Mzack9999 834cff9fa2 Merge pull request #7786 from omlahore/fix/openapi-schemaless-bodies
fix(openapi): build request bodies from an example of the empty schema
2026-10-25 16:18:34 +01:00
Georgii Butakov 23f766eb61 feat(runner): refuse -replay-proxy with targets pinned to an ip
-replay-proxy re-sends the requests behind a finding through a proxy,
which resolves the host itself, so the replay of a pinned target would
not reach the pinned ip. The pinned-target checks move into
pinnedTargetConflict with a table test, and the input README lists
-replay-proxy next to -proxy and -automatic-scan.
2026-10-25 15:14:50 +00:00
Georgii Butakov 43f99eadde Merge branch 'dev' into feat/target-line-ip
dev requires fastdialer v0.6.0, which has WithResolveOverride and the
dial cache fix, so go.mod and go.sum take dev's versions and the
fastdialer replace is dropped.
2026-10-25 15:14:50 +00:00
Mzack9999 baaeb15022 merge dev 2026-10-25 16:05:21 +01:00
Mzack9999 d4b8a07e80 add scope 2026-10-25 15:57:30 +01:00
Mzack9999 319baa3152 merge dev 2026-10-25 15:51:39 +01:00
Mzack9999 e24855067e merge dev 2026-10-25 15:50:36 +01:00
Georgii Butakov a4c597fd31 Merge branch 'dev' into feat/task-based-progress
Resolved pkg/core/executors.go: kept the task accounting and took dev's
per-template host-errors cache key from #7859.
2026-10-25 14:50:25 +00:00
Mzack9999 87afa2fd4e Merge pull request #7926 from projectdiscovery/feat/7894-js-backoff
apply host backoff to javascript template dials
2026-10-25 15:46:57 +01:00
Mzack9999 b4c9cc10c9 pass options 2026-10-25 15:46:14 +01:00
Mzack9999 7f6cfd42f3 fix callback 2026-10-25 15:35:50 +01:00
Mzack9999 a03f13e929 merge dev 2026-10-25 15:35:50 +01:00
Mzack9999 02f5d9a6c3 merge dev 2026-10-25 15:32:51 +01:00
Mzack9999 f46ee55118 Merge pull request #7930 from projectdiscovery/compiled-cache-options
Keep template options on compiled-cache hits
2026-10-25 15:26:37 +01:00
Mzack9999 a9d358a56a Merge pull request #7929 from projectdiscovery/sdk-ignore-after-init
Read the nuclei ignore file after init updates it
2026-10-25 15:24:17 +01:00
Mzack9999 9545a9246c copy operators 2026-10-25 14:51:11 +01:00
Mzack9999 48e04c7317 fix ignore 2026-10-25 14:51:10 +01:00
Ice3manandClaude Opus 5.5 2954b81c85 http: stop copying large request bodies
Templates that send multi-MB bodies (e.g. CVE-2026-109618 sends 1, 8 and
20 MB) kept 8-10 copies of each body alive per request. With 25 hosts in
flight this reached ~10 GB RSS and OOM-killed scan VMs.

- add pkg/utils/httpbody: a zero-copy request body that rewinds on EOF;
  GetBody hands out new cursors over the same slice instead of cloning
- build raw, model and fuzzed request bodies with httpbody instead of
  retryablehttp's ReusableReadCloser, which tees every read into a second
  growing buffer and clones on each GetBody
- dump requests with http.Request.Write into a capped writer instead of
  retryablehttp Dump/httputil.DumpRequestOut; output is unchanged for
  bodies under the cap
- keep only the first 32 KiB of a body in request dumps (new
  -rqss/-request-size-save flag, 0 keeps all) and leave bodies over 4 KiB
  out of curl commands
- build the project cache key only when a project file is used, hashing
  the body when the dump is truncated
- replay sends the full body when the dump was truncated
- slice the raw request body from the request instead of reading it out
- pass replacer values to fasttemplate as []byte so writes don't copy them

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 18:48:04 +05:30
Mzack9999 052855717d Merge pull request #7679 from projectdiscovery/idle-conn-quarantine
Stop reusing hosts after idle leftovers
2026-10-25 14:09:42 +01:00
Mzack9999 e1f656033d fix idle 2026-10-25 13:39:50 +01:00
Mzack9999 9301048bc4 merge dev 2026-10-25 13:37:52 +01:00
Dwi Siswanto 4331cd0f7a fix(offlinehttp): read status lines by position, not text search
Passive mode added a minor version to status lines such as "HTTP/2 200
OK" by rewriting the first "HTTP/<digit> " anywhere in the raw data. In
a valid HTTP/1.1 response that text could sit in a header, the reason
phrase, or the body, which was then altered: a Content-Length body came
out truncated and a chunked one failed to decode. Only the protocol
token at the start of the response is normalized now.

When the data starts with a request, the response is now read from
where the request body ends, with the same normalization. That path
used to reject HTTP/2 status lines and read a request body as the
status line, then fall back to the last "HTTP/" in the data, which
fails whenever the response body contains one. A dump that keeps a
request's Content-Length but leaves out its body is retried right after
the headers. Two exchanges back to back now yield the first response
for HTTP/2, as they already did for HTTP/1.1.

Up to five interim 1xx responses are skipped to reach the final one,
and an interim response is returned only when no final response is
found. A response read after a request carries that request. Data that
cannot be read this way still uses the last "HTTP/" search; httpx
-store-chain output, which puts a newline after the first request,
relies on it to return the chain's final response.

This supersedes #7922, which anchors the version pattern and so fixes

Fixes: #7921
Fixes: #7924
Reported-by: @MasterchiefGalaxy
Signed-off-by: Dwi Siswanto <[email protected]>
2026-10-25 13:16:36 +01:00
Mzack9999 e21e636530 Merge pull request #7922 from MasterchiefGalaxy/fix/offline-http-preserve-response
fix(offlinehttp): preserve response content during HTTP version normalization
2026-10-25 00:59:31 +02:00
Mzack9999 6d433e6a6f Merge pull request #7855 from projectdiscovery/dependabot/go_modules/go_modules-e1bf5cf652
chore(deps): bump the go_modules group across 1 directory with 2 updates
2026-10-24 23:48:14 +02:00
ghost e8f1a788ac chore(js): update memoized functions 🤖 2026-10-24 21:45:37 +00:00
Mzack9999 f1702dc401 Merge pull request #7908 from gnuletik/fix/js-redis-reply-cap
fix(js): cap what a redis connection may read
2026-10-24 23:44:46 +02:00
Doğan Can Bakır 3ab0a66962 bump fastdialer to v0.6.0 2026-10-24 23:22:28 +03:00
Doğan Can Bakır bf53db167d apply host backoff to javascript template dials 2026-10-24 23:22:24 +03:00
Mzack9999 d13b58d987 fix lint 2026-10-24 18:19:06 +02:00
MasterchiefGalaxy 206446f567 fix(offlinehttp): preserve response content during version normalization 2026-10-23 21:46:03 -05:00
Mzack9999 b901fb8aea Merge pull request #7868 from TassioSales/fix/headless-ws-exclude-hosts
fix(headless): enforce network policy on ws and wss schemes (#7867)
2026-10-24 02:28:10 +02:00
Mzack9999 7e1a8f7da9 Merge pull request #7633 from prithvee07/dev
Keep unique markdown report names
2026-10-24 01:51:15 +02:00
Mzack9999 943ec350bb Merge pull request #7876 from TassioSales/fix/proxy-internal-env-vars
fix(runner): set HTTPS_PROXY and all proxy env vars when proxy-internal is enabled
2026-10-24 01:47:18 +02:00
Mzack9999 a54a6252a6 fix test 2026-10-24 01:45:56 +02:00
Mzack9999 131b7b5e2c fix filenames 2026-10-24 01:19:20 +02:00
Mzack9999 773ddf1411 merge dev 2026-10-24 01:17:29 +02:00
Mzack9999 665914747c fix lint 2026-10-24 01:11:55 +02:00
Mzack9999 bb06c6a470 Merge pull request #7812 from CalvinTjoaquinn/fix/raw-versionless-request-line
fix(raw): keep a path that contains "HTTP" in a versionless request line
2026-10-24 01:08:00 +02:00
Mzack9999 bdbdc1e8de fix lint 2026-10-24 01:07:20 +02:00
Mzack9999 68cc4b17cd guard proxy 2026-10-24 01:04:17 +02:00
Mzack9999 308044e234 merge dev 2026-10-24 01:01:45 +02:00
Mzack9999 64ad459151 Merge pull request #7756 from baturdenada/fix/dialer-panic
Fix dialer panics in JS protocol and input providers
2026-10-24 00:54:34 +02:00
Mzack9999 4e6f69cb97 merge dev 2026-10-24 00:53:47 +02:00
Mzack9999 cdf3798bb7 Merge pull request #7796 from comradezephyr/fix/requestresponse-json-roundtrip
fix(input): emit JSON objects in RequestResponse.MarshalJSON
2026-10-24 00:48:47 +02:00
Mzack9999 d2699653fd Merge pull request #7810 from TassioSales/fix/extractor-field-type-validation
fix(extractors): reject misapplied group and attribute fields on wrong extractor types (#7808)
2026-10-24 00:46:56 +02:00
Mzack9999 663c3528c7 Merge pull request #7802 from omlahore/fix/openapi-binary-content-type
fix(openapi): send format: binary octet-stream bodies as octet-stream
2026-10-24 00:45:31 +02:00
Mzack9999 9dbbe4c225 Merge pull request #7831 from m10x/dev
fix(installer): fail fast on unremovable templates update lock
2026-10-24 00:39:05 +02:00
Mzack9999 40e165a065 merge dev 2026-10-24 00:37:21 +02:00
Mzack9999 25c4b0d623 Merge pull request #7799 from CalvinTjoaquinn/fix/raw-header-without-colon
fix(raw): two defects in the raw header loop
2026-10-24 00:35:12 +02:00
Mzack9999 cb8434c81c Merge pull request #7828 from ruzfi/fix/http/ip-var-empty-behind-proxy
fix(http): set ip var from IP-literal target when dial history is empty
2026-10-24 00:28:52 +02:00
Mzack9999 e7a383124b merge dev 2026-10-24 00:27:32 +02:00
Mzack9999 9cd0fbb983 fix dialers 2026-10-24 00:25:03 +02:00
Mzack9999 1e8b85a133 drop duplicate 2026-10-24 00:18:52 +02:00
Mzack9999 c1b2553022 merge dev 2026-10-24 00:17:51 +02:00
Mzack9999 99c22b36f0 Merge pull request #7853 from NitinReddy-A/fix/7610-exact-config-basename
fix(catalog): match known config files by exact basename
2026-10-24 00:17:22 +02:00
Mzack9999 213b74b683 Merge pull request #7809 from sandiyochristan/fix/extractor-field-type-ownership
fix(extractors): reject fields the extractor type never reads
2026-10-24 00:13:55 +02:00
Mzack9999 ee9a2ed36d quote paths 2026-10-24 00:08:43 +02:00
Mzack9999 9fd430d803 merge dev 2026-10-24 00:05:23 +02:00
Mzack9999 20c1510069 merge dev 2026-10-24 00:03:13 +02:00
Mzack9999 6ab068a356 merge dev 2026-10-24 00:01:57 +02:00
Mzack9999 a4f855aeee Merge pull request #7850 from SamIntruder/fix/http-to-https-self-heal-on-detect
fix(http): self-heal HTTP-to-HTTPS requests immediately on detection
2026-10-24 00:01:17 +02:00
Mzack9999 30c6e1f0ca merge dev 2026-10-23 23:53:03 +02:00
Mzack9999 7448d2ab23 keep group 2026-10-23 23:49:32 +02:00
Mzack9999 833bd0102c merge dev 2026-10-23 23:48:57 +02:00
Mzack9999 cefccf97f3 merge dev 2026-10-23 23:43:57 +02:00
Mzack9999 b229dadf45 Merge pull request #7854 from TassioSales/fix/xpath-extractor-validation
fix(extractors): reject invalid xpath expressions during extractor compilation (#7750)
2026-10-23 23:43:06 +02:00
Mzack9999 332cfd5372 merge dev 2026-10-23 23:37:09 +02:00
Mzack9999 bad0e62459 fix lint 2026-10-23 23:34:38 +02:00
Mzack9999 cebb1a45b0 merge dev 2026-10-23 23:32:09 +02:00
Mzack9999 39ab5141a8 retry lock 2026-10-23 23:24:19 +02:00
Mzack9999 347b996d2c merge dev 2026-10-23 23:23:10 +02:00
Mzack9999 5dde4c2b30 retry https 2026-10-23 23:16:53 +02:00
Mzack9999 3a686c1ba5 Merge pull request #7877 from TassioSales/fix/honour-template-max-size
fix(http): honour template max-size under global response-size-read limit
2026-10-23 22:59:50 +02:00
Mzack9999 8c214de616 merge dev 2026-10-23 22:58:20 +02:00
Mzack9999 8081bcbee8 guard xpath 2026-10-23 22:52:44 +02:00
Mzack9999 913d4161d1 merge dev 2026-10-23 22:52:11 +02:00
Mzack9999 5c3c00f2fa guard limit 2026-10-23 22:31:36 +02:00
Mzack9999 ab5c51640e merge dev 2026-10-23 22:30:05 +02:00
Mzack9999 beccd36038 Merge pull request #7910 from gnuletik/fix/js-smtp-reply-cap
fix(js): cap what an smtp connection may read
2026-10-23 22:11:41 +02:00
Mzack9999 2b39243c73 merge dev 2026-10-23 19:32:40 +02:00
Mzack9999 bc53323223 Merge pull request #7906 from gnuletik/fix/js-ldap-smtp-close-on-cancel
fix(js): close ldap and smtp connections when the template ends
2026-10-23 19:31:25 +02:00
Mzack9999 0cfd312330 unstick timeout 2026-10-23 19:04:09 +02:00
Mzack9999 8be6ff6389 allow helo 2026-10-23 18:40:53 +02:00
Mzack9999 9ece5bc5f6 Merge pull request #7886 from TassioSales/fix/sdk-intermittent-zero-templates-7859
fix(sdk): prevent custom-port HTTP failures from poisoning base host cache and filter skip events
2026-10-23 18:36:14 +02:00
Mzack9999 d117fcada4 merge dev 2026-10-23 18:29:52 +02:00
Mzack9999 316de3dc43 cap lengths 2026-10-23 18:28:58 +02:00
Mzack9999 86b0f3f14f merge dev 2026-10-23 18:12:05 +02:00
Mzack9999 e284b6bc24 Merge pull request #7897 from projectdiscovery/feat/7893-headless-backoff
apply host backoff to headless navigations
2026-10-23 18:05:53 +02:00
Mzack9999 1248a14a62 merge dev 2026-10-23 18:03:30 +02:00
Mzack9999 4a2eb3b29d Merge pull request #7904 from TassioSales/fix/interactsh-client-fastdialer
fix(runner): use protocolstate fastdialer for default interactsh client (#7903)
2026-10-23 17:50:29 +02:00
Mzack9999 413a418714 reuse transport 2026-10-23 17:18:43 +02:00
Mzack9999 4cf0bbc08a merge dev 2026-10-23 17:16:07 +02:00
Mzack9999 1628b4c437 follow redirects 2026-10-23 17:16:07 +02:00
Mzack9999 e554a7e514 merge dev 2026-10-23 17:04:02 +02:00
Mzack9999 1fb6a9a287 keep proxy 2026-10-23 17:04:00 +02:00
Mzack9999 214d5e1310 Merge pull request #7889 from IBazsecI/fix/js-net-proxy-network-policy
fix(js/net): enforce network policy on proxied target in JS net dial
2026-10-23 16:51:43 +02:00
Mzack9999 54134f0f5b reorder check 2026-10-23 16:12:43 +02:00
Mzack9999 3e68f2b99f merge dev 2026-10-23 15:43:27 +02:00
Mzack9999 f8451703a2 check host 2026-10-23 15:43:27 +02:00
Mzack9999 5e0c3e3d4f Merge pull request #7902 from projectdiscovery/fix/go-toolchain-cve
bump Go toolchain for CVE-2026-111943
2026-10-23 14:27:52 +02:00
Georgii Butakov 9f4e53b933 chore(deps): point the fastdialer replace at the upstream commit
fastdialer#580 is merged into main, so the fork branch it was on no longer
exists. The replace resolves the same commit through the upstream module
path github.com/projectdiscovery/fastdialer.
2026-10-22 12:00:39 +00:00
mazzz3r aba8fda759 fix(payloads): widen the Feistel domain to cover the full int range
The domain search stopped at 62 bits, so a total above 2^62 was handed a
domain smaller than the range and encrypt dropped the top bit, breaking the
one-to-one mapping. Stop at 63 bits (rounded up to an even 64), which covers
every representable combination total.
2026-10-22 11:10:55 +00:00
mazzz3r c2767d6ef6 fix(payloads): generate the random order on demand with bounded memory
The random-order iterator built the shuffled order with rand.Perm(total),
allocating an int slice for the whole combination space before the first
request. For a clusterbomb product that is O(total): four 100-value lists
need ~800 MB up front, which can exhaust memory on the large, partially
sent scans the flag is meant for.

Replace it with an on-demand pseudo-random permutation of [0,total): a
4-round Feistel network over the enclosing power-of-two domain, restricted
to the range by cycle-walking. It visits every combination exactly once in
a random order, stays deterministic per iterator so Reset replays the same
order, and uses O(1) memory. The combination decode and Total() are
unchanged.

Add coverage for larger, non-power-of-two totals to exercise the
cycle-walking, and point the default-off guard at the new field.
2026-10-22 10:24:42 +00:00
mazzz3r c99bc1b959 test(payloads): cover random order totals, reset, default-off and option copy 2026-10-22 09:02:02 +00:00
mazzz3r 05d328d936 feat(payloads): iterate payload combinations in a random order
Add -pro, -payload-random-order to run a template's payload combinations
in a random order instead of the sequential order of the payload sets.

The iterator shuffles the combination indices [0,total) and decodes each
index back to a combination, so it visits every combination exactly once
for batteringram, pitchfork and clusterbomb, keeping pitchfork's
index-aligned pairs and the full clusterbomb product. Total() is
unchanged, so request estimates and progress stay correct. When the flag
is off the sequential path is used as before.
2026-10-22 08:49:42 +00:00
Georgii Butakov c1fcd4d8d2 Merge branch 'dev' into feat/target-line-ip
go.mod takes dev's versions, including fastdialer v0.5.26, and keeps the
fastdialer replace.
2026-10-21 20:25:04 +00:00
gnuletikandClaude Opus 5.5 d1bc79be95 fix(js): cap what an smtp connection may read
net/smtp reads replies through net/textproto, which buffers a line until
"\n" with no length limit, so IsOpenRelay and SendMail buffered whatever a
port streaming without newlines sent. Their reads now fail after 1 MiB.

SendMail dials through the execution's dialer like IsOpenRelay so its
connection can be wrapped; smtp.SendMail dials on its own.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-21 19:10:57 +02:00
gnuletikandClaude Opus 5.5 c505b5f3c4 fix(js): cap what a redis connection may read
go-redis reads a reply line until "\n" with no length limit, so a port that
streams bytes without one was buffered for as long as it kept sending: one
GetServerInfoAuth call against such a port grew the heap by about 1 GiB.

The redis clients now dial through go-redis's default dialer and wrap the
connection so reads fail past 16 MiB, far above any reply these calls expect.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-21 18:26:16 +02:00
gnuletikandClaude Opus 5.5 6a4e6fab6f fix(js): close ldap and smtp connections when the template ends
go-ldap waits for a reply with no timeout and net/smtp reads with no
deadline. Neither takes a context, so on a server that accepts and never
answers the call stays blocked after the session is abandoned, and the
goroutine keeps the runtime and everything it references alive for the life
of the process.

Close the connection when the template's context is done. SendMail now
dials through the execution's dialer like IsSMTP and IsOpenRelay, instead of
net/smtp.SendMail dialing on its own past the network policy.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-21 13:45:11 +02:00
Doğan Can Bakır 9bf950c861 require patched go version 2026-10-21 11:43:54 +03:00
TassioSales 4edd5c2c68 fix(runner): use protocolstate fastdialer for default interactsh client (fixes #7903) 2026-10-20 22:36:36 -03:00
TassioSales 5cf7f3ba47 fix(http): scope projectCacheKey with effective body read limit
Include the effective read limit in projectCacheKey so requests with different
limits (such as small template max-size previews) do not share truncated cached
bodies with larger reads.
2026-10-20 21:07:35 -03:00
TassioSales c918d010c5 fix(http): honour template max-size under global response-size-read limit
When -response-size-read (-rsr) is configured, use it as a ceiling while
preserving a template's own MaxSize when smaller, rather than unconditionally
overwriting it. Also account for ResponseReadSize in response truncation detection.

Fixes #7873
2026-10-20 21:07:35 -03:00
Doğan Can Bakır ed46eb76e3 bump go toolchain for CVE-2026-111943 2026-10-20 21:39:34 +03:00
Doğan Can Bakır d7730705af apply host backoff to headless navigations 2026-10-19 22:04:52 +03:00
Mzack9999 b4761d4fc4 merge dev 2026-10-19 15:05:46 +02:00
Mzack9999 e4d1535691 merge dev 2026-10-19 14:26:55 +02:00
Kemal BazandClaude Opus 4.8 1c0f77d28e fix(js/net): enforce network policy on proxied target in JS net dial
When an HTTP(S) proxy is configured, the nuclei/net JS library dialed the
proxy through Fastdialer (which validates the proxy host) but tunneled the
connection target inside an HTTP CONNECT without validating it. This let a
template reach a deny-listed destination (-exclude-hosts /
-restrict-local-network-access) that a direct dial would have blocked,
bypassing the network policy.

Validate the target host against the network policy in the proxy branch of
Dial() before contacting the proxy, matching the guard the direct path gets
from Fastdialer. Adds a regression test asserting the proxy is never
contacted for a deny-listed target.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-18 14:32:26 +03:00
TassioSales 3827928531 fix(sdk): separate template auto-upgrade sync.Once and avoid FailNow in callback goroutine 2026-10-17 13:32:20 -03:00
TassioSales b296cc0d46 fix(sdk): prevent custom-port HTTP failures from poisoning base host cache and filter skip events (#7859) 2026-10-17 13:06:21 -03:00
IgorDaniel45 9a20d1cc69 merge dev and preserve both protocol test suites 2026-10-16 17:34:39 -03:00
IgorDaniel45 3bb429522f fix template flow options on compiled cache hits 2026-10-16 17:27:25 -03:00
AdamMagued 535e04d555 perf(protocols/http): defer building FullResponseString when unused by operators
Signed-off-by: AdamMagued <[email protected]>
2026-10-16 01:35:26 +03:00
TassioSales 4ce6dea3e9 fix(runner): set HTTPS_PROXY and all proxy env vars when proxy-internal is enabled
Fixes #7875

When options.ProxyInternal was enabled, only HTTP_PROXY was set in the environment.
Because Go standard library http.ProxyFromEnvironment inspects HTTPS_PROXY for https URLs,
internal HTTPS requests (such as template updates via GitHub API / releases, pdtm, etc.)
failed to route through the specified proxy in restricted or proxy-required environments.

This sets HTTP_PROXY, HTTPS_PROXY, ALL_PROXY (and lowercase counterparts) when ProxyInternal
is enabled, and adds unit test coverage. Also fixes operator precedence for ProxyInternal
checks in runner.go and sdk_private.go.
2026-10-15 09:15:21 -03:00
Nitin b31f787d79 chore: fix ValidateOptions doc comment name 2026-10-15 16:59:22 +05:30
Nitin 17101d0501 chore: add doc comments to functions touched by the templates channel 2026-10-15 16:54:00 +05:30
Nitin a4cc18f30e fix(installer): download templates archive without the github token
The archive link points to a different host than the API, and the github
client sends the token to every request it makes. The archive is public,
so only the API call needs the token now, and the archive url has to be
https.

Also adds doc comments to the new functions and tests.
2026-10-15 16:43:31 +05:30
Nitin 866da46866 feat(installer): add templates update channel
Adds -utc / -update-templates-channel (stable, latest). stable keeps the
current behaviour. latest installs the head of the nuclei-templates main
branch, so templates merged after the last release are available without
waiting for the next one.

The version for the latest channel is the last release plus the short
commit as semver build metadata (e.g. v10.5.0+3de629d). Build metadata is
ignored when comparing versions, so switching back to stable doesn't
downgrade, and only a newer release makes it outdated there. Since the
version check only knows about releases, the latest channel compares the
installed version against the current head to decide when to update.

The channel can also be set in the config file like the other options.

Closes #5608
2026-10-15 16:26:17 +05:30
TassioSales 8b440a5a77 fix(headless): address errcheck linter warnings for unchecked returns in proxy and tests 2026-10-14 00:34:33 -03:00
Georgii Butakov 4716246849 Merge branch 'dev' into feat/target-line-ip
go.mod takes dev's versions, including fastdialer v0.5.25, and keeps
fastdialer replaced by the branch of projectdiscovery/fastdialer#580,
which now includes v0.5.25.
2026-10-13 19:55:18 +00:00
TassioSales d43c1f547f fix(headless): canonicalize IPv4-mapped IPv6 literals and re-validate dial targets 2026-10-13 15:01:50 -03:00
TassioSales 506e4fb0fc fix(headless): enforce network policy on WebSocket handshakes via loopback proxy 2026-10-13 14:16:53 -03:00
TassioSales 52960db710 fix(headless): harden blocked URL staging, URL target parsing, and wsMap cleanup 2026-10-13 02:37:33 -03:00
TassioSales ca3e27c5f5 fix(headless): accumulate dynamic blocked URL patterns across denied WebSockets 2026-10-13 02:22:21 -03:00
TassioSales a94aa7f00d fix(headless): propagate NetworkEnable errors during policy initialization 2026-10-13 02:12:01 -03:00
TassioSales c14335d351 fix(headless): address code review feedback on page lifecycle and CIDR handling 2026-10-13 02:02:26 -03:00
TassioSales c5a528cc60 fix(headless): enforce network policy and blocked URLs for WebSocket connections via CDP 2026-10-13 01:45:12 -03:00
TassioSales d9dabbb4a1 fix(headless): enforce network policy on ws and wss schemes (#7867)
Previously, isValidHost only checked targets with http: and https:
prefixes, allowing ws: and wss: connections in headless mode to bypass
configured network policies (e.g. -exclude-hosts or local network restrictions).

This commit normalizes the scheme check and enforces host validation
against the network policy for ws: and wss: protocols as well.
2026-10-12 23:50:25 -03:00
Georgii Butakov d21749be3a feat(input): pin a target to an ip with the ip key of target lines
Scanning a vhost on a given server, or an origin behind a CDN, needs the
hostname in the Host header, SNI and template variables while the
connection goes to another ip, and one host often has to be scanned on
several ips. -sa scans the ips DNS returns, and a target of the ip with
a Host header sends no SNI and sets {{Hostname}} to the ip.

A target line can now pin its target to an ip, like curl --resolve:

  {"target": "https://app.example", "ip": "192.0.2.10"}

Each target and ip pair is a separate target with its own template
selection. The ip is validated and canonicalized, the target must have
a hostname, -sa and -iv don't resolve a pinned target, and an ip the
network policy denies is skipped. Pinned targets are refused with
-proxy and -automatic-scan.

The engine pins the host of every target with a CustomIP, from a target
line or -sa, with fastdialer.WithResolveOverride on the context each
template runs with. HTTP, network, websocket and javascript templates
then connect to the ip, while redirects to other hosts and the proxy
dial are no longer pinned. HTTP keeps a client per pinned ip, network
and websocket results report the ip, and scheme-less targets are probed
through it.

The host errors cache keys a failed dial of a pinned target by its ip,
also for IDN hosts, and -mup counts the urls of each ip apart, so a url
listed with several ips, or resolved to several with -sa, is kept on
each of them.

go.mod points fastdialer at a fork commit with WithResolveOverride until
a fastdialer release includes it.
2026-10-12 17:02:39 +00:00
Georgii Butakov 1568f1e68e feat(progress): report completion by executed templates
The -stats percentage was requests sent against a total estimated
before the scan: TotalRequests of each template after clustering, times
the number of targets. The estimate leaves out workflows, whose requests
are added while they run, counts a self-contained template once per
target although it runs once, and doesn't follow stop-at-first-match or
a template that runs once per origin. A scan could end at 108% or stop
at 25%.

The engine now counts tasks: one per template, cluster or workflow run
on a target, and one per self-contained template. The total is known
when the scan starts. A task is done when it runs, when -resume skips
it, when its host is unresponsive, or when it repeats its origin.

progress.TaskTracker is an optional interface that StatsTicker
implements, so Progress and lib.StatsWriter don't change. The stats
line shows Tasks: done/total and takes the percentage from it, and -sj
and the metrics endpoint report tasks, totalTasks and the same percent.
2026-10-10 12:00:33 +00:00
dependabot[bot] 007ee7d4ba chore(deps): bump the go_modules group across 1 directory with 2 updates
Bumps the go_modules group with 2 updates in the / directory: [github.com/moby/sys/user](https://github.com/moby/sys) and [go.opentelemetry.io/otel/sdk](https://github.com/open-telemetry/opentelemetry-go).


Updates `github.com/moby/sys/user` from 0.4.0 to 0.4.1
- [Release notes](https://github.com/moby/sys/releases)
- [Commits](https://github.com/moby/sys/compare/user/v0.4.0...user/v0.4.1)

Updates `go.opentelemetry.io/otel/sdk` from 1.44.0 to 1.45.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.44.0...v1.45.0)

---
updated-dependencies:
- dependency-name: github.com/moby/sys/user
  dependency-version: 0.4.1
  dependency-type: indirect
  dependency-group: go_modules
- dependency-name: go.opentelemetry.io/otel/sdk
  dependency-version: 1.45.0
  dependency-type: indirect
  dependency-group: go_modules
...

Signed-off-by: dependabot[bot] <[email protected]>
2026-10-09 18:51:19 +00:00
TassioSales 50a6526721 fix(extractors): reject invalid xpath expressions during extractor compilation (fixes #7750) 2026-10-08 17:33:24 -03:00
Nitin 7010ce6a91 test(catalog): add doc comments to catalog tests 2026-10-08 15:46:15 +05:30
Nitin 5d9da1d199 fix(catalog): match known config files by exact basename
The catalog skipped any template whose name contained a known config file
name (cves.json, contributors.json, TEMPLATES-STATS.json) since the check
used substring matching. A template like http/cves.json.yaml was silently
dropped both when passed directly and during directory traversal.

Add config.IsKnownConfigFile which compares the base name exactly, use it
in IsTemplateWithRoot and GetTemplatesPath, and drop the duplicate list in
the disk package.

Closes #7610
2026-10-08 15:23:23 +05:30
Sam PizzeyandClaude Sonnet 5 8d33e4a4c7 fix(http): self-heal HTTP-to-HTTPS requests immediately on detection
The http-to-https-tracker only recorded a host:port as https-only
after a plain-HTTP request received the "plain HTTP request was sent
to HTTPS port" 400 response, and used that record solely to
pre-emptively rewrite the scheme of later requests. The request that
first discovered the mismatch was never retried, so it matched
against the wrong-scheme 400 instead of the real page.

This produced a false negative on whichever request happened to run
first against a given host:port, including every request racing out
before the tracker learned about it under default concurrency - and
template loading itself races, so results could flip between runs of
the same scan even at -c 1.

Peek the response immediately after the first send and, if it matches
the signature, retry the same request over https before handing it to
the matcher pipeline, instead of only caching the fact for next time.

Fixes #7849

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-10-07 11:46:38 +01:00
Mzack9999 113ea8afa9 fix lint 2026-10-06 15:19:34 +02:00
Mzack9999 a5d0eb9fda fix cluster 2026-10-06 15:13:19 +02:00
Mzack9999 65732c382f add tests 2026-10-06 14:56:50 +02:00
Mzack9999 ab8c7feeba fix exports 2026-10-06 14:30:41 +02:00
Maximilian Hildebrand 2815cb2aef fix(installer): fail fast on unremovable templates update lock 2026-10-06 10:12:11 +02:00
Mzack9999 cb824a4718 cover consumers 2026-10-05 23:43:43 +02:00
Mzack9999 78a5cadc67 fix response 2026-10-05 23:28:43 +02:00
Riyan Firmansyah f5aa294aa6 Merge pull request #1 from ruzfi/fix/http-ip-var-empty-behind-proxy 2026-10-06 02:19:05 +07:00
ruzfi 77b34db0cc fix(http): set hostname for pipelined non-raw requests
The pipeline branch that sends a built request via pipelinedClient.Dor
never set hostname, unlike the raw pipeline branch. With an empty
hostname, the dial-history lookup, the IP-literal fallback in outputIP
and the cname lookup had nothing to work with, so `ip` was always empty
for these requests.

Set hostname from the request URL's host when the URL is non-nil,
matching the normal request path.
2026-10-06 02:14:36 +07:00
ruzfi 87e0a1d2a4 fix(http): set ip var from IP-literal target when dial history is empty
Resolve the output ip in a single helper, outputIP, used by both the
request-error and response paths: CustomIP first, then dial history,
then the target host if it is an IP literal. The helper splits the port
from the unstripped host with net.SplitHostPort, because the existing
LastIndex(":") strip truncates bracketed IPv6 without a port and, in
the redirect loop, truncates bracketed IPv6 further on each iteration.
2026-10-06 01:14:35 +07:00
Sandiyo Christan 352f46cc35 fix(extractors): reject attribute on non-xpath extractors
`attribute` is read only by the xpath extractors, but it is accepted on
every extractor type and then dropped. A template that sets it on a json
extractor compiles, runs, and returns whatever the extractor collects.
Nothing reports the drop, so the result is indistinguishable from "the
target did not match" — a silent false negative for a scanner.

CompileExtractors already rejects a misapplied field, `case-insensitive`
outside kval extractors. Extend that pattern so a misapplied `attribute`
fails at template load through Operators.Compile, which reports it as
"could not compile extractor". The check runs after the per-type
required-value validation so the more useful "requires at least one
<field> value" error still wins.

`group` is a no-op outside regex extractors in the same way but is
deliberately left alone. A survey of all 14,029 templates in
nuclei-templates found 17 extractors that set `group` on a json or kval
extractor, all published CVE detections such as CVE-2024-3848, so
enforcing it breaks real templates. Two tests pin that behaviour.

Verified with nuclei -validate over all 14,029 templates: "All templates
validated successfully".
2026-10-02 12:09:18 +05:30
Calvin Tjoaquinn 98df72854b fix(raw): keep a path that contains "HTTP" in a versionless request line
A two-field request line omits the path, so the second field is the
version, and the branch that handles it tested the field with
strings.Contains(parts[1], "HTTP"). Any path containing that substring
matched, so the field was treated as the version and the path was
replaced with "".

    GET /HTTPProxy          ->  Method=GET  Path=""
    GET /api/HTTPHandler    ->  Method=GET  Path=""
    GET /admin              ->  Method=GET  Path="/admin"

For a normal raw template the request then goes to the path of the input
URL instead of the one the template asked for, with no warning. For a
self-contained template the symptom is worse: ParseRawRequest refuses an
empty path, so the template fails with "path cannot be empty in self
contained request" while the path is plainly there.

Testing the field with HasPrefix(parts[1], "HTTP/") keeps the case the
branch exists for, "GET HTTP/1.1", and stops a path from being mistaken
for a version. The sibling raw parser in pkg/input/types already works
this way: it requires three fields and uses a prefix test for the scheme.
2026-10-01 18:04:42 +07:00
Calvin Tjoaquinn 7fa79ba6a6 fix(raw): stop an empty line at EOF from becoming a header
The header loop treats an empty line as the end of the block only when
the read succeeded. When the request ends with a newline and the next
read returns io.EOF, the guard falls through and the empty line is
processed as a header, so Headers gains an entry named "".

That entry reaches the wire. rawhttp's serializer writes a header with
an empty value as just the key followed by CRLF, so a key of "" is
written as a bare CRLF, which ends the header block. Map iteration order
decides where it lands, so a random subset of the remaining headers is
sent as body instead:

  GET /admin HTTP/1.1
  Host: example.com
                                  <- header block ends here
  Authorization: Bearer s3cr3t

Before the previous commit the entry held the preceding header's value
and serialized as a malformed ": value" line. With values no longer
reused it holds "", which is why the fix belongs with that change rather
than after it.

An empty line now ends the block regardless of the read error, and a
real read error still stops the loop without storing anything.
2026-10-01 14:23:28 +07:00
TassioSales 52245202ff fix(extractors): reject misapplied group and attribute fields on wrong extractor types (#7808)
- Reject non-zero 'group' on non-regex extractor types with descriptive compilation error

- Reject non-empty 'attribute' on non-xpath extractor types with descriptive compilation error

- Implement String() on ExtractorTypeHolder so formatted error strings output clean type names

- Add unit tests covering rejection of misapplied fields across all extractor types
2026-09-30 22:13:48 -03:00
Om b104625865 fix(openapi): send format: binary octet-stream bodies as octet-stream
The octet-stream branch compared the schema format against "bindary",
so a requestBody with format: binary went out with Content-Type:
text/plain and only format: byte kept application/octet-stream.
2026-09-30 03:51:50 +05:30
Calvin Tjoaquinn 73d976be6c fix(raw): stop a colon-less header line from taking the previous value
readRawRequest reused key and value across loop iterations and only
reassigned value when the line contained a colon. A line carrying no colon
therefore stored the value of the line before it under its own name:

  GET /admin HTTP/1.1
  Host: example.com
  Authorization: Bearer s3cr3t
  X-Forwarded-For

parsed to X-Forwarded-For: Bearer s3cr3t, so the credential went out under
the wrong header name.

Declaring both inside the loop fixes it by construction rather than by
remembering to reset. The parser stays as lenient as before: the malformed
line is still accepted as a header, now with an empty value.
2026-09-29 14:02:24 +07:00
comradezephyr 06dd497f85 fix(input): emit JSON objects in RequestResponse.MarshalJSON
RequestResponse.MarshalJSON stored the pre-marshaled request/response bytes as []byte inside map[string]interface{}, which encoding/json (and sonic) encode as base64 strings. The type's own UnmarshalJSON expects JSON objects under those keys, so the marshal -> unmarshal round-trip always failed when Request != nil, breaking the json.JSONCodec contract asserted at pkg/input/types/http.go.

MetaInput.ReqResp serializes under the raw-request key with this codec (MetaInput.MarshalString -> MetaInput.Unmarshal), used by host-map scans and exclusion processing in pkg/input/provider/list/hmap.go and internal/runner/preflight_portscan.go: a captured request attached to a MetaInput produced an encoded key that could not be decoded.

Wrap the pre-marshaled bytes as json.Message so they are emitted verbatim, and omit the keys for nil Request/Response instead of encoding base64 null, mirroring UnmarshalJSON's optional-key behavior.

Fixes #7795
2026-09-28 16:02:40 -04:00
Om 349299c9ab fix(openapi): skip a request whose xml body cannot be encoded
An XML array example has no single root to encode, and the request
still went out with no body or Content-Type. Skip it after the warning,
like the default branch does for content types it cannot build.
2026-09-28 02:30:09 +05:30
Om f76bd78d15 fix(openapi): build request bodies from an example of the empty schema
When a request body had no schema, the generator used the schema that
generateEmptySchemaValue returned as the body value itself. An
application/xml body then hit the unchecked map assertion and nuclei
panicked while loading the spec, a JSON body was sent as
{"type":"object"}, and a text/plain body as a dump of the schema struct.

Generate an example from the empty schema instead, and let the XML
branch take a string example as the body and warn on types it cannot
encode instead of panicking. The empty schema for
application/octet-stream has no example, so that body now gets the
placeholder the octet-stream branch already has for an empty value.
2026-09-28 01:52:46 +05:30
root-Manas cbbf15bc1e fix(openapi): select complete authentication requirements 2026-09-27 18:31:48 +05:30
root-Manas 591a019e83 fix(openapi): report missing credentials and reject empty values 2026-09-27 11:29:42 +05:30
root-Manas 5b4bc8efc8 fix(openapi): validate credentials per operation 2026-09-27 11:06:44 +05:30
root-Manas 8496c48c6f feat(openapi): support OAuth2 and OpenID Connect access tokens 2026-09-27 10:51:06 +05:30
IgorDaniel45 ccd0aaca40 fix compiled cache after metadata load 2026-09-25 10:56:02 -03:00
Mzack9999 3fb407b345 restore tls 2026-09-23 23:47:09 +02:00
Mzack9999 c987c2df6a fix quarantine 2026-09-23 21:58:13 +02:00
Mzack9999 717062fd0e fix confidence 2026-09-23 21:58:12 +02:00
Mzack9999 213d3badfd merge dev 2026-09-23 20:23:30 +02:00
Mzack9999 6c2ea4edcb merge dev 2026-09-23 20:23:17 +02:00
hodeswildsmith455-boop 268ea525bd feat: add OrcaRouter as a first-class LLM provider with API key and PKCE login
Signed-off-by: hodeswildsmith455-boop <[email protected]>
2026-09-22 15:17:55 +00:00
Bounty Hunter 0270f5f1da Add test case for missing dialers in JS protocol 2026-09-18 21:57:16 +03:00
Bounty Hunter 9cf5459dd9 Fix dialer panics in JS protocol and input providers 2026-09-18 21:45:44 +03:00
gnuletikandClaude Opus 5 ff7b951bf0 style: drop the redundant embedded Operators selector
staticcheck QF1008: Operators is embedded in Request, so Matchers and
Extractors are reachable directly.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-17 22:18:59 +02:00
gnuletikandClaude Opus 5 4a6aed5eab perf(http): build the full response only when an operator reads it
executeRequest calls respChain.FullResponseString() for every response.
That allocates a copy of headers+body (a make plus two copies) and keeps
it reachable for as long as the event is -- including inside the
interactsh request cache, which holds events until their OAST callback
arrives or the eviction window closes.

Almost nothing reads it. Across the public template corpus only 81 of
11,634 http templates (0.7%) reference "response" or part "all". The rest
match on body or headers, which are already materialised separately.

Gate the build on a scan of the request's own matchers and extractors,
plus the debug and fuzz-stats paths that consume it independently of any
template.

Measured against 4 hosts serving 2MB responses, 4 concurrent runs, with
GOMEMLIMIT at 1.73GiB:

  full profile (73,856 requests)   9,559MB -> 4,760MB peak
  http/vulnerabilities/ subset     7,445MB -> 3,796MB peak

Behaviour is unchanged for part: response, part: body, part: all and a
dsl contains(response, ...) expression, checked against a target whose
token appears only in a response header, so the concatenation is the
only thing that can match it.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-17 19:59:43 +02:00
MGreif 228cf4b9a2 Change test 2026-09-16 09:57:18 +02:00
MGreif f5dcb9bf31 Add nil receiver guard and regression test 2026-09-16 09:48:02 +02:00
MGreif b9f971c8fa Add: cb syncing 2026-09-16 09:34:44 +02:00
MGreif 65964f2819 Add back rebase corruptions 2026-09-16 09:14:11 +02:00
MGreif 40715999a5 Change: Rename interface 2026-09-16 09:08:28 +02:00
MGreif 223daf2615 Fix: tests 2026-09-15 11:11:12 +02:00
MGreif 63c656f7ee Add: interactsh-client interface for modular design
# Conflicts:
#	pkg/protocols/common/interactsh/interactsh.go
#	pkg/protocols/protocols.go
2026-09-15 11:11:10 +02:00
Mzack9999 1898b4aec4 merge dev 2026-09-14 22:34:30 +02:00
Mzack9999 927f30464e fix race 2026-09-05 17:48:50 +04:00
Mzack9999 bf11a866e1 fix lint 2026-09-05 16:09:50 +04:00
Mzack9999 ca0688f086 clone variables 2026-09-05 15:55:38 +04:00
Mzack9999 8f87aa7440 normalize context 2026-09-05 14:27:44 +04:00
Mzack9999 b00db1d984 cancel listing 2026-09-05 14:17:05 +04:00
Mzack9999 db2b48a3a0 stop enumeration 2026-09-05 13:07:11 +04:00
Mzack9999 e505fb840e restore signature 2026-09-05 12:55:05 +04:00
Mzack9999 7496794e58 honour cancellation 2026-09-05 12:42:01 +04:00
Mzack9999 b996c5d4ce merge dev 2026-09-05 11:59:54 +04:00
IgorDaniel45 eded59972e fix: address compiled cache review feedback 2026-09-02 19:13:00 -03:00
Igor Daniel 4fb773d8b2 fix: isolate cached template execution state 2026-09-02 17:58:59 -03:00
Igor Daniel 9f47943e11 fix: reuse compiled templates in thread-safe engine 2026-09-02 16:49:30 -03:00
chauhand2463 a1c26f9d98 feat(http): expose TLS key exchange metadata 2026-08-28 09:41:07 +05:30
Mzack9999 034cf04d47 quarantine hosts 2026-08-26 10:37:40 +04:00
gnuletik 841cd851ca fix(core): honour the scan context when acquiring worker slots and JS runtimes
Four acquires in the scan path ignored the context they were given, so a
cancelled scan could not leave until unrelated work finished:

- executeTemplateSpray and executeTemplatesOnTarget acquire a worker slot via
  AdaptiveWaitGroup.Add, which passes context.Background().
- executeHostSpray does the same.
- flow.GetJSRuntime acquires from a process-global goja pool with
  sizedpool.Get(context.TODO()).

Each of these can wait a long time. A worker slot is held for as long as a
template runs, and a JS runtime stays checked out for the whole flow including
any wait on the rate limiter, so the pool saturates under concurrent scans.
A scan whose context is already cancelled then queues for a runtime it only
needs in order to unwind, and its own cancellation cannot break the wait.

Both spray loops already check ctx.Done() at the top of each iteration, so the
acquire was the only place cancellation was invisible.

GetJSRuntime now takes a context and returns an error. This changes an
exported signature; it has one caller in-tree.
2026-08-26 03:15:52 +02:00
prithvee07 59f8657566 Merge pull request #6 from prithvee07/codex/review-the-codebase-r6nml5
Harden markdown report filenames, improve HTTP input parsing/progress, and upgrade dependencies
2026-08-18 22:14:01 +05:30
prithvee07 7a1d539d7e Merge branch 'dev' into codex/review-the-codebase-r6nml5 2026-08-18 21:32:29 +05:30
Claude 03adf593d4 Merge remote-tracking branch 'upstream/dev' into dev
# Conflicts:
#	pkg/input/types/http.go
2026-08-18 15:54:52 +00:00
prithvee07 fd91245001 Merge pull request #7 from prithvee07/claude/project-contribution-work-gwxmho
Fix panic when parsing unsafe HTTP requests with only method
2026-08-18 21:13:06 +05:30
Claude e6f6274e99 Fix index-out-of-range panic parsing unsafe raw requests with no path
readRawRequest() skips the malformed-field guard when unsafe is true,
so a raw request line with a single token (e.g. "GET\n", no path or
HTTP version) falls through to rawRequest.Path = parts[1] and panics
with "index out of range [1] with length 1". Since unsafe mode exists
specifically to tolerate malformed request lines, this is directly
reachable via a template's raw HTTP request block.

Guard the assignment so a one-token request line degrades to an empty
path instead of panicking; the existing case rawrequest.Path == "" in
Parse() already handles an empty path correctly by automerging with
the input URL.
2026-08-18 15:41:34 +00:00
Matin 0316732363 docs: update heading level to preserve outline 2026-08-13 19:59:04 +04:30
hu-matin 960ba0bb7f docs: add image alt tags and fix table of contents links 2026-08-13 19:51:25 +04:30
hu-matin 04c6db7de0 docs: link Persian README in main README 2026-08-13 13:12:29 +04:30
hu-matin caa5dfafea docs: add Persian(FA) translation for README 2026-08-13 12:50:42 +04:30
prithvee07 8a3a0f33de fix(input): parse HTTP authority case-insensitively 2026-08-04 11:16:50 +05:30
aviadavi 0f5934cfaa test: assert unsafe raw -H evaluation on the generated wire bytes
The previous test called the resolver directly, so it passed even with the
TryFillCustomHeaders wiring reverted and therefore guarded nothing.

Drive real request generation instead and read UnsafeRawBytes, asserting
the marker is absent, the tag survives, and the user agent still varies
across requests. Confirmed by reverting the fix: the test now fails with
"expression reached the wire unevaluated".
2026-08-03 17:18:04 +03:00
aviadavi bfd48c7740 fix: evaluate -H expressions on the unsafe raw request path too
Unsafe raw requests do not go through setCustomHeaders for the value that
reaches the wire: TryFillCustomHeaders splices the -H line straight into
UnsafeRawBytes, and rawhttp sends those bytes. So a header carrying an
expression was sent with the literal {{...}} marker intact.

Evaluate the -H strings before they are spliced in, and share one
evaluation helper between both paths so they cannot drift.

Verified against a local server with an unsafe: true template:

  before: {{rand_user_agent()}} myprop/value
  after:  Mozilla/5.0 (Macintosh ...) Safari/605.1.15 myprop/value
2026-08-03 17:18:04 +03:00
aviadavi 92f3a55ee7 feat: evaluate DSL helpers in -H values per request
Adds a rand_user_agent() DSL helper and evaluates helper expressions in
global custom header values once per generated request.

This implements the approach proposed in #7446 instead of a dedicated
CLI flag, so attribution tagging needs no new global option:

  -H "User-Agent: {{rand_user_agent()}} myprop/value"

The helper is registered with cacheable=false, otherwise a cached result
would freeze the user agent for the whole run. It is backed by the
existing projectdiscovery/useragent pool that nuclei already uses for UA
randomization, so tagged scans send the same class of user agent as
untagged ones and no new dependency is introduced. The name matches the
slug dsl.FakerFunctions() would expose, so wiring faker later is a
drop in.

Values come from the operator's own -H flag or config file, the same
trust boundary as a template, and response data is never evaluated here.
Values with no {{ }} marker skip evaluation entirely, and compile or
evaluation errors keep the original value so a typo degrades the header
rather than failing the scan. The unsafe raw request path is unchanged.
2026-08-03 17:18:04 +03:00
prithvee07 b19e739961 Merge branch 'projectdiscovery:dev' into dev 2026-08-03 17:03:26 +05:30
prithvee07 b24ade541a Merge pull request #5 from prithvee07/claude/codebase-review-3q1x4d
Add ReadHeaderTimeout to DASTServer's http.Server
2026-08-03 12:01:27 +05:30
Claude 58d3ad6063 Add ReadHeaderTimeout to DASTServer's http.Server
The DAST fuzzing/stats server (-dast-server) constructed its http.Server
with no timeouts, leaving it open to a Slowloris-style slow-header DoS.
Only ReadHeaderTimeout is set (not ReadTimeout/WriteTimeout) since /fuzz
requests can legitimately take a while to process. Same class of fix
already applied to internal/httpapi/apiendpoint.go.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01QSCzGpK3LLQDKXtHCew5CG
2026-08-03 06:21:20 +00:00
prithvee07 1ff643d72a Merge branch 'projectdiscovery:dev' into dev 2026-08-03 11:44:22 +05:30
prithvee07 7f2254ae57 Merge pull request #4 from prithvee07/claude/codebase-review-3q1x4d
Fix ParseRawRequest header/host parsing panics and data loss
2026-08-03 11:42:56 +05:30
Claude 5371ac689a Fix ParseRawRequest header/host parsing panics and data loss
ParseRawRequest read the request line and headers positionally: the
second line was always treated as the Host line regardless of content,
and header values were read by skipping exactly one byte past the
colon. This meant:
- a valueless header (e.g. "X-Empty:") panicked with a slice-bounds
  error, taking down the whole scan
- a Host header anywhere but the second line was silently dropped from
  the header map, so header-based preconditions never matched
- "Host:target" (no space after colon) parsed the host as "arget"
- an absolute-form request target, as used by proxy captures
  (GET http://target/p HTTP/1.1), got appended to the Host rather than
  replacing it

Every request-shaped input mode (burp, jsonl, yaml, openapi) goes
through this parser. Parse headers uniformly first, look up Host by
name, trim header values instead of assuming one space, and treat an
absolute-form request target as replacing the authority outright -
matching how pkg/protocols/http/raw.readRawRequest already handles
raw template requests.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01QSCzGpK3LLQDKXtHCew5CG
2026-08-03 06:08:52 +00:00
prithvee07 675fcc4ea8 Merge pull request #3 from prithvee07/claude/codebase-review-3q1x4d
Security hardening: Add HTTP API token auth and improve file permissions
2026-08-03 11:28:32 +05:30
Claude a20aa26024 Fix pprof/auth exposure on http-api-endpoint and harden file permissions
The experimental -http-api-endpoint server used http.ListenAndServe(addr,
nil), which serves Go's global DefaultServeMux. Since net/http/pprof is
blank-imported elsewhere in the binary for the separate, opt-in
-enable-pprof feature, enabling -http-api-endpoint silently exposed
/debug/pprof/* (process cmdline, heap/goroutine dumps, CPU profiling) on
the same address. The endpoint's PUT /api/concurrency also had no
authentication. Give the server its own ServeMux, add an optional
-http-api-token flag (mirroring -dast-server-token) with a warning when
unset, and add request timeouts.

Also tighten file/directory permissions (0644->0600, 0755/0775->0750)
across report exporters, the template installer/signer, custom-template
downloaders, and output writers, so scan results and downloaded
templates aren't left world-readable on shared systems.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01QSCzGpK3LLQDKXtHCew5CG
2026-08-03 05:54:23 +00:00
prithvee07 06771d7197 Merge pull request #2 from prithvee07/codex/review-the-codebase
Harden Markdown report filenames and update Go dependencies (go1.26.5, go-github v81)

# Markdown and security change report

This file is a standalone, downloadable summary of the Markdown report filename
hardening and Go vulnerability remediation work.

## Executive summary

The change set:

- prevents Markdown report paths from being influenced by directory traversal
  sequences;
- prevents long template IDs and hosts from removing the per-finding UUID;
- includes the selected matcher or extractor name in result filenames;
- truncates filenames without producing invalid UTF-8;
- updates the Go toolchain and vulnerable dependencies; and
- reduces reachable `govulncheck` findings from 23 to one upstream advisory
  with no fixed dependency version.

## Markdown filename hardening

Markdown finding files use this structure:

```text
<template-id>-<host>-<uuid>-<matcher-or-extractor>.md
```

The matcher name is selected when both a matcher and extractor name are
available. Extractor-only findings use the extractor name.

Before a file is written, unsafe filesystem characters, path separators, and
parent-directory references are replaced. The resulting filename is limited to
255 bytes and truncated only at a valid UTF-8 boundary. Prefix truncation does
not remove the UUID or `.md` extension, so findings cannot collide merely
because their long template and host prefixes are identical.

## Dependency remediation

| Component | Previous version | Updated version |
| --- | --- | --- |
| Go toolchain | 1.26.0 | 1.26.5 |
| `golang.org/x/text` | v0.38.0 | v0.39.0 |
| `github.com/yuin/goldmark` | v1.7.13 | v1.7.17 |
| Direct `github.com/google/go-github` use | v30.1.0 | v81.0.0 |

The direct GitHub client migration covers the custom-template downloader and
GitHub reporting tracker. Module metadata was refreshed after the upgrades.

## Vulnerability scan result

The initial source scan reported 23 reachable vulnerabilities. After the
updates, the scan reports only GO-2026-5932.

GO-2026-5932 remains reachable through the latest available
`github.com/projectdiscovery/utils/update` dependency, which transitively uses
`github.com/google/go-github/v30` and `golang.org/x/crypto/openpgp`. The advisory
does not identify a fixed `golang.org/x/crypto` release. Eliminating this final
result requires the upstream update package to migrate away from the older
GitHub client or a replacement of that update subsystem.

Run the vulnerability scan with:

```console
go run golang.org/x/vuln/cmd/govulncheck@latest \
  -db=https://storage.googleapis.com/go-vulndb ./...
```

An exit status indicating GO-2026-5932 is expected until the upstream
dependency is migrated.

## Validation commands

```console
go mod verify
go test ./pkg/reporting/exporters/markdown
go test ./pkg/external/customtemplates -run '^$'
go test ./pkg/reporting/trackers/github
go vet ./pkg/external/customtemplates \
  ./pkg/reporting/trackers/github \
  ./pkg/reporting/exporters/markdown
```

The `-run '^$'` custom-template command checks compilation without running the
network-dependent GitHub download tests.

## Detailed documentation

- [Markdown report filenames](markdown-report-filenames.md)
- [Go vulnerability remediation](security-remediation.md)
2026-08-02 22:56:16 +05:30
prithvee07 efbb48c167 docs: add downloadable change report 2026-08-02 22:55:13 +05:30
prithvee07 c03f5c8df1 Merge pull request #1 from prithvee07/codex/review-the-codebase
Sanitize/truncate markdown report filenames; update GitHub client and dependency versions
2026-08-02 22:41:13 +05:30
prithvee07 865c90e627 fix(deps): remediate reachable Go vulnerabilities 2026-08-02 22:40:26 +05:30
Jiho Lee 2766609119 fix(extractors): reject a regex group the pattern cannot produce
ExtractRegex skips any submatch where len(match) < RegexGroup+1, and
FindAllStringSubmatch returns exactly NumSubexp()+1 entries. So a group
index beyond the pattern's capture count can never match: the template
compiles, runs, and silently extracts nothing. An empty result is
indistinguishable from 'the pattern did not match the target', which is
the same silent-failure class as the negative-group case already
rejected two lines above.

Scanning nuclei-templates (13451 files, 2100 regex extractors with
group > 0) finds 4 real instances, and they look like genuine bugs
rather than false positives. The clearest is
smtp-credentials-exposure.yaml, where two patterns share one group: 1:

    regex:
      - smtp_username":".*"                        <- 0 groups
      - <smtp_username>(.*)</smtp_username>       <- 1 group

Against a body containing both forms it extracts only the XML one; the
JSON credential is missed with no indication. That template has been
half-working since it was written.

Validation runs on the cached-regex branch too. The compile path
short-circuits on a cache hit, so checking only the freshly-compiled
branch would let the same pattern through unvalidated as soon as any
other template had already compiled it.

The fuzz harness picks a group index independently of its patterns (its
own defaults include the group-less https?://[^\s"']+), so it now
clamps the group to what the narrowest pattern captures -- otherwise it
would fail at compile time on inputs it deliberately generates, instead
of exercising extraction.

Tests fail without the fix: removing both call sites gives 2 failures,
and removing only the cached-branch check still fails the cache test.
2026-07-26 23:45:05 +09:00
Mzack9999 e3864b60f2 Resolve conflicts 2026-07-15 19:02:06 +04:00
Mzack9999 187d855145 allowlist openvas 2026-06-26 01:48:22 +02:00
Mzack9999 1f276227ff report confidence 2026-06-26 01:31:23 +02:00
Mzack9999 e0fd39a862 protocol parts 2026-06-26 01:24:31 +02:00
Mzack9999 e0ed203765 negative evidence 2026-06-26 01:14:08 +02:00
Mzack9999 3ae786749d tune scoring 2026-06-26 01:08:50 +02:00
Mzack9999 b3881248ca refine scoring 2026-06-26 01:00:51 +02:00
Mzack9999 1ddd00efef baseline detection 2026-06-26 00:57:58 +02:00
Mzack9999 bc776d29a8 score confidence 2026-06-26 00:57:58 +02:00
233 changed files with 15845 additions and 911 deletions

No files matched your search

+1 -1
View File
@@ -17,4 +17,4 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: crate-ci/typos@v1.49.0
- uses: crate-ci/typos@v1.51.1
+14 -10
View File
@@ -1,7 +1,7 @@
![nuclei](/static/nuclei-cover-image.png)
<div align="center">
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README.md">`English`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_CN.md">`中文`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_KR.md">`Korean`</a> •
@@ -9,8 +9,9 @@
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_ES.md">`Spanish`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_JP.md">`日本語`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_PT-BR.md">`Portuguese`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_TR.md">`Türkçe`</a>
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_TR.md">`Türkçe`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_FA.md">`فارسی`</a>
</div>
<p align="center">
@@ -110,7 +111,7 @@ Browse the full Nuclei [**`documentation here`**](https://docs.projectdiscovery.
### Installation
`nuclei` requires **go >= 1.24.2** to install successfully. Run the following command to get the repo:
`nuclei` requires **go >= 1.27.3** to install successfully. Run the following command to get the repo:
```sh
go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest
@@ -176,7 +177,7 @@ TEMPLATES:
-egm, -enable-global-matchers enable loading global matchers templates
-file enable loading file templates
-llm enable llm matchers and extractors (semantic matching)
-llm-provider string llm provider for semantic matching (openai, ollama, llamacpp, vllm, lmstudio, groq, openrouter, together) (default "openai")
-llm-provider string llm provider for semantic matching (openai, ollama, llamacpp, vllm, lmstudio, groq, openrouter, together, orcarouter) (default "openai")
-llm-base-url string openai-compatible endpoint for semantic matching (overrides -llm-provider, e.g. a local model)
-llm-model string model used for semantic matching
-llm-timeout int time in seconds to wait for a single llm call (default 30)
@@ -249,9 +250,11 @@ CONFIGURATIONS:
-lna, -restrict-local-network-access blocks connections to the local / private network
-i, -interface string network interface to use for network scan
-at, -attack-type string type of payload combinations to perform (batteringram,pitchfork,clusterbomb)
-pro, -payload-random-order iterate payload combinations in a random order instead of sequentially
-sip, -source-ip string source ip address to use for network scan
-rsr, -response-size-read int max response size to read in bytes
-rss, -response-size-save int max response size to read in bytes (default 1048576)
-rqss, -request-size-save int max request body size to save in bytes (0 to save the whole body) (default 32768)
-reset reset removes all nuclei configuration and data files (including nuclei-templates)
-tlsi, -tls-impersonate enable experimental client hello (ja3) tls randomization
-hae, -http-api-endpoint string experimental http api endpoint
@@ -355,10 +358,11 @@ DEBUG:
-hc, -health-check run diagnostic check up
UPDATE:
-up, -update update nuclei engine to the latest released version
-ut, -update-templates update nuclei-templates to latest released version
-ud, -update-template-dir string custom directory to install / update nuclei-templates
-duc, -disable-update-check disable automatic nuclei/templates update check
-up, -update update nuclei engine to the latest released version
-ut, -update-templates update nuclei-templates to latest released version
-utc, -update-templates-channel string nuclei-templates update channel (stable, latest) (default "stable")
-ud, -update-template-dir string custom directory to install / update nuclei-templates
-duc, -disable-update-check disable automatic nuclei/templates update check
HONEYPOT:
-hpd, -honeypot-detect detect potential honeypot hosts based on match concentration
@@ -680,7 +684,7 @@ Thanks to all the amazing [**`community contributors for sending PRs`**](https:/
<br>
<div align="center">
<sub>**`nuclei`** is distributed under [**MIT License**](https://github.com/projectdiscovery/nuclei/blob/main/LICENSE.md)</sub>
</div>
+10 -7
View File
@@ -9,7 +9,8 @@
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_ES.md">`Spanish`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_JP.md">`日本語`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_PT-BR.md">`Portuguese`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_TR.md">`Türkçe`</a>
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_TR.md">`Türkçe`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_FA.md">`فارسی`</a>
</div>
@@ -110,7 +111,7 @@ _针对安全团队和企业,我们提供了构建在 Nuclei OSS 之上的云
### 安装
`nuclei` 需要 **go >= 1.24.2** 才能成功安装。运行以下命令获取仓库:
`nuclei` 需要 **go >= 1.27.3** 才能成功安装。运行以下命令获取仓库:
```sh
go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest
@@ -174,7 +175,7 @@ TEMPLATES:
-egm, -enable-global-matchers enable loading global matchers templates
-file enable loading file templates
-llm enable llm matchers and extractors (semantic matching)
-llm-provider string llm provider for semantic matching (openai, ollama, llamacpp, vllm, lmstudio, groq, openrouter, together) (default "openai")
-llm-provider string llm provider for semantic matching (openai, ollama, llamacpp, vllm, lmstudio, groq, openrouter, together, orcarouter) (default "openai")
-llm-base-url string openai-compatible endpoint for semantic matching (overrides -llm-provider, e.g. a local model)
-llm-model string model used for semantic matching
-llm-timeout int time in seconds to wait for a single llm call (default 30)
@@ -250,6 +251,7 @@ CONFIGURATIONS:
-sip, -source-ip string source ip address to use for network scan
-rsr, -response-size-read int max response size to read in bytes
-rss, -response-size-save int max response size to read in bytes (default 1048576)
-rqss, -request-size-save int max request body size to save in bytes (0 to save the whole body) (default 32768)
-reset reset removes all nuclei configuration and data files (including nuclei-templates)
-tlsi, -tls-impersonate enable experimental client hello (ja3) tls randomization
-hae, -http-api-endpoint string experimental http api endpoint
@@ -350,10 +352,11 @@ DEBUG:
-hc, -health-check run diagnostic check up
UPDATE:
-up, -update update nuclei engine to the latest released version
-ut, -update-templates update nuclei-templates to latest released version
-ud, -update-template-dir string custom directory to install / update nuclei-templates
-duc, -disable-update-check disable automatic nuclei/templates update check
-up, -update update nuclei engine to the latest released version
-ut, -update-templates update nuclei-templates to latest released version
-utc, -update-templates-channel string nuclei-templates update channel (stable, latest) (default "stable")
-ud, -update-template-dir string custom directory to install / update nuclei-templates
-duc, -disable-update-check disable automatic nuclei/templates update check
HONEYPOT:
-hpd, -honeypot-detect detect potential honeypot hosts based on match concentration
+10 -7
View File
@@ -9,7 +9,8 @@
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_ES.md">`Spanish`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_JP.md">`日本語`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_PT-BR.md">`Portuguese`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_TR.md">`Türkçe`</a>
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_TR.md">`Türkçe`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_FA.md">`فارسی`</a>
</div>
@@ -110,7 +111,7 @@ Consulta la [**`documentación completa de Nuclei aquí`**](https://docs.project
### Instalación
`nuclei` requiere **go >= 1.24.2** para instalarse correctamente. Ejecuta el siguiente comando para obtener el repositorio:
`nuclei` requiere **go >= 1.27.3** para instalarse correctamente. Ejecuta el siguiente comando para obtener el repositorio:
```sh
go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest
@@ -174,7 +175,7 @@ TEMPLATES:
-egm, -enable-global-matchers enable loading global matchers templates
-file enable loading file templates
-llm enable llm matchers and extractors (semantic matching)
-llm-provider string llm provider for semantic matching (openai, ollama, llamacpp, vllm, lmstudio, groq, openrouter, together) (default "openai")
-llm-provider string llm provider for semantic matching (openai, ollama, llamacpp, vllm, lmstudio, groq, openrouter, together, orcarouter) (default "openai")
-llm-base-url string openai-compatible endpoint for semantic matching (overrides -llm-provider, e.g. a local model)
-llm-model string model used for semantic matching
-llm-timeout int time in seconds to wait for a single llm call (default 30)
@@ -250,6 +251,7 @@ CONFIGURATIONS:
-sip, -source-ip string source ip address to use for network scan
-rsr, -response-size-read int max response size to read in bytes
-rss, -response-size-save int max response size to read in bytes (default 1048576)
-rqss, -request-size-save int max request body size to save in bytes (0 to save the whole body) (default 32768)
-reset reset removes all nuclei configuration and data files (including nuclei-templates)
-tlsi, -tls-impersonate enable experimental client hello (ja3) tls randomization
-hae, -http-api-endpoint string experimental http api endpoint
@@ -350,10 +352,11 @@ DEBUG:
-hc, -health-check run diagnostic check up
UPDATE:
-up, -update update nuclei engine to the latest released version
-ut, -update-templates update nuclei-templates to latest released version
-ud, -update-template-dir string custom directory to install / update nuclei-templates
-duc, -disable-update-check disable automatic nuclei/templates update check
-up, -update update nuclei engine to the latest released version
-ut, -update-templates update nuclei-templates to latest released version
-utc, -update-templates-channel string nuclei-templates update channel (stable, latest) (default "stable")
-ud, -update-template-dir string custom directory to install / update nuclei-templates
-duc, -disable-update-check disable automatic nuclei/templates update check
HONEYPOT:
-hpd, -honeypot-detect detect potential honeypot hosts based on match concentration
+725
View File
@@ -0,0 +1,725 @@
![nuclei](/static/nuclei-cover-image.png)
<div align="center">
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README.md">`English`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_CN.md">`中文`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_KR.md">`Korean`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_ID.md">`Indonesia`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_ES.md">`Spanish`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_JP.md">`日本語`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_PT-BR.md">`Portuguese`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_TR.md">`Türkçe`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_FA.md">`فارسی`</a>
</div>
<p align="center">
<a href="https://docs.projectdiscovery.io/tools/nuclei/overview?utm_source=github&utm_medium=web&utm_campaign=nuclei_readme"><img src="https://img.shields.io/badge/Documentation-%23000000.svg?style=for-the-badge&logo=data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIyNCIgaGVpZ2h0PSIyNCIgdmlld0JveD0iMCAwIDI0IDI0IiBmaWxsPSJub25lIiBzdHJva2U9IiNmZmZmZmYiIHN0cm9rZS13aWR0aD0iMiIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIiBjbGFzcz0ibHVjaWRlIGx1Y2lkZS1ib29rLW9wZW4iPjxwYXRoIGQ9Ik0xMiA3djE0Ii8+PHBhdGggZD0iTTMgMThhMSAxIDAgMCAxLTEtMVY0YTEgMSAwIDAgMSAxLTFoNWE0IDQgMCAwIDEgNCA0IDQgNCAwIDAgMSA0LTRoNWExIDEgMCAwIDEgMSAxdjEzYTEgMSAwIDAgMS0xIDFoLTZhMyAzIDAgMCAwLTMgMyAzIDMgMCAwIDAtMy0zeiIvPjwvc3ZnPg==&logoColor=white" alt="Documentation Badge"></a>
&nbsp;&nbsp;
<a href="https://github.com/projectdiscovery/nuclei-templates"><img src="https://img.shields.io/badge/Templates Library-%23000000.svg?style=for-the-badge&logo=data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIxNiIgaGVpZ2h0PSIxNiIgdmlld0JveD0iMCAwIDI0IDI0IiBmaWxsPSJub25lIiBzdHJva2U9IiNmZmZmZmYiIHN0cm9rZS13aWR0aD0iMS41IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIGNsYXNzPSJsdWNpZGUgbHVjaWRlLXNoaWVsZCI+PHBhdGggZD0iTTIwIDEzYzAgNS0zLjUgNy41LTcuNjYgOC45NWExIDEgMCAwIDEtLjY3LS4wMUM3LjUgMjAuNSA0IDE4IDQgMTNWNmExIDEgMCAwIDEgMS0xYzIgMCA0LjUtMS4yIDYuMjQtMi43MmExLjE3IDEuMTcgMCAwIDEgMS41MiAwQzE0LjUxIDMuODEgMTcgNSAxOSA1YTEgMSAwIDAgMSAxIDF6Ii8+PC9zdmc+&logoColor=white"></a>
&nbsp;&nbsp;
<a href="https://discord.gg/projectdiscovery?utm_source=github&utm_medium=web&utm_campaign=nuclei_readme"><img src="https://img.shields.io/badge/Discord-%235865F2.svg?style=for-the-badge&logo=discord&logoColor=white" alt="Discord Badge"></a>
<hr>
</p>
<div dir="rtl">
<br>
<br>
**Nuclei یک اسکنر آسیب‌پذیری مدرن و با کارایی بالاست که از قالب‌های ساده بر پایه YAML استفاده می‌کند. این ابزار به شما امکان می‌دهد سناریوهای سفارشی برای شناسایی آسیب‌پذیری‌ها طراحی کنید که شرایط واقعی را شبیه‌سازی می‌کنند و منجر به صفر شدن خطاهای مثبت کاذب (گزارش‌های اشتباه) می‌شوند.**
- فرمت ساده YAML برای ساخت و سفارشی‌سازی قالب‌های آسیب‌پذیری.
- مشارکت توسط هزاران متخصص امنیت جهت مقابله با آسیب‌پذیری‌های نوظهور و رایج.
- کاهش خطاهای مثبت کاذب با شبیه‌سازی مراحل واقعی جهت تایید یک آسیب‌پذیری.
- پردازش موازی فوق‌سریع اسکن‌ها و خوشه‌بندی درخواست‌ها.
- قابلیت ادغام در خطوط لوله CI/CD برای شناسایی آسیب‌پذیری‌ها و تست‌های رگرسیون.
- پشتیبانی از پروتکل‌های متعدد مانند TCP، DNS، HTTP، SSL، WHOIS، جاوااسکریپت، کد و موارد دیگر.
- قابلیت یکپارچه‌سازی با Jira، Splunk، GitHub، Elastic و GitLab.
<br>
<br>
## فهرست مطالب
- [**`شروع کار`**](#شروع-کار)
- [_`۱. ابزار خط فرمان Nuclei`_](#۱-ابزار-خط-فرمان-nuclei)
- [_`۲. نسخه‌های حرفه‌ای و سازمانی`_](#۲-نسخه‌های-حرفه‌ای-و-سازمانی)
- [**`مستندات`**](#مستندات)
- [_`دستورات خط فرمان`_](#دستورات-خط-فرمان)
- [_`اسکن یک هدف واحد`_](#اسکن-یک-هدف)
- [_`اسکن اهداف متعدد`_](#اسکن-چندین-هدف)
- [_`اسکن شبکه`_](#اسکن-شبکه)
- [_`اسکن با قالب سفارشی شما`_](#اسکن-با-استفاده-از-قالب-سفارشی-خودتان)
- [_`اتصال Nuclei به ProjectDiscovery`_](#اتصال-nuclei-به-projectdiscovery)
- [**`قالب‌های Nuclei، جامعه کاربری و پاداش‌ها`**](#قالب‌های-nuclei-جامعه-و-پاداش‌ها-) 💎
- [**`ماموریت ما`**](#ماموریت-ما)
- [**`مشارکت‌کنندگان`**](#مشارکت‌کنندگان-) ❤
- [**`مجوز استفاده`**](#license)
<br>
<br>
## شروع کار
### ۱. ابزار خط فرمان Nuclei
_ابزار Nuclei را روی سیستم خود نصب کنید. کار خود را با دنبال کردن راهنمای نصب در [**`اینجا`**](https://docs.projectdiscovery.io/tools/nuclei/install?utm_source=github&utm_medium=web&utm_campaign=nuclei_readme) آغاز کنید. علاوه بر این، ما [**`یک سطح ابری رایگان`**](https://cloud.projectdiscovery.io/sign-up) ارائه می‌دهیم که همراه با محدودیت‌های رایگان ماهانه سخاوتمندانه‌ای عرضه می‌شود:_
- ذخیره‌سازی و بصری‌سازی یافته‌های آسیب‌پذیری شما
- نوشتن و مدیریت قالب‌های Nuclei شما
- دسترسی به آخرین قالب‌های Nuclei
- شناسایی و ذخیره‌سازی اهداف شما
> [!Important]
> |**این پروژه در حال توسعه فعال است**. انتظار تغییرات اساسی در نسخه‌های جدید را داشته باشید. لطفاً قبل از به‌روزرسانی، لیست تغییرات نسخه (Changelog) را بررسی کنید.|
> |:--------------------------------|
> | این پروژه در درجه اول برای استفاده به عنوان یک ابزار خط فرمان (CLI) مستقل ساخته شده است. **اجرای Nuclei به عنوان یک سرویس ممکن است خطرات امنیتی به همراه داشته باشد.** توصیه می‌شود با احتیاط و با در نظر گرفتن تدابیر امنیتی اضافی از آن استفاده کنید. |
<br>
### ۲. نسخه‌های حرفه‌ای و سازمانی
_برای تیم‌های امنیتی و سازمان‌ها، ما یک سرویس میزبانی‌شده در ابر بر پایه نسخه متن‌باز (OSS) ابزار Nuclei ارائه می‌دهیم؛ این سرویس به‌طور دقیق بهینه‌سازی شده است تا به شما کمک کند اسکن‌های آسیب‌پذیری را به‌صورت مداوم، در مقیاس بزرگ و همراه با تیم و گردش‌های کاری فعلی خود اجرا کنید:_
- اسکن‌های ۵۰ برابر سریع‌تر
- اسکن در مقیاس بزرگ با دقت بسیار بالا
- یکپارچه‌سازی با سرویس‌های ابری (AWS، GCP، Azure، Cloudflare، Fastly، Terraform، Kubernetes)
- اتصال به Jira، Slack، Linear، رابط‌های برنامه‌نویسی (APIs) و وب‌هوک‌ها (Webhooks)
- گزارش‌دهی‌های مدیریتی و سازگاری با استانداردهای امنیتی (Compliance)
- به‌علاوه: اسکن بی‌درنگ (Real-time)، پشتیبانی از SAML SSO، پلتفرم منطبق با استاندارد SOC 2 (همراه با گزینه‌های میزبانی در اتحادیه اروپا و آمریکا)، فضاهای کاری مشترک تیمی و موارد دیگر
- ما به‌طور مداوم در حال [**`افزودن قابلیت‌های جدید`**](https://feedback.projectdiscovery.io/changelog) هستیم!
- **ایده‌آل برای:** نفوذگران قانون‌مند (Pentesters)، تیم‌های امنیتی و سازمان‌ها
در صورتی که سازمان بزرگی دارید و نیازمندی‌های شما پیچیده است، می‌توانید در [**`نسخه پرو ثبت‌نام کنید`**](https://projectdiscovery.io/pricing?utm_source=github&utm_medium=web&utm_campaign=nuclei_readme) یا [**`با تیم ما گفتگو کنید`**](https://projectdiscovery.io/request-demo?utm_source=github&utm_medium=web&utm_campaign=nuclei_readme).
<br>
<br>
## مستندات
[**`مستندات کامل`**](https://docs.projectdiscovery.io/tools/nuclei/running) ابزار Nuclei را در اینجا مرور کنید. اگر به‌تازگی کار با Nuclei را شروع کرده‌اید، [**`مجموعه ویدیوهای آموزشی پایه‌ای ما در یوتیوب`**](https://www.youtube.com/playlist?list=PLZRbR9aMzTTpItEdeNSulo8bYsvil80Rl) را بررسی کنید.
<div align="center">
<a href="https://www.youtube.com/watch?v=b5qMyQvL1ZA&list=PLZRbR9aMzTTpItEdeNSulo8bYsvil80Rl&utm_source=github&utm_medium=web&utm_campaign=nuclei_readme" target="_blank"><img src="/static/nuclei-getting-started.png" width="350px" alt="Nuclei Getting Started Video Tutorial"></a> <a href="https://www.youtube.com/watch?v=nFXygQdtjyw&utm_source=github&utm_medium=web&utm_campaign=nuclei_readme" target="_blank"><img src="/static/nuclei-write-your-first-template.png" width="350px" alt="Writing Your First Nuclei Template Video Tutorial"></a>
</div>
<br>
### نصب
برای نصب موفقیت‌آمیز `nuclei` به نسخه **go >= 1.24.2** نیاز است. دستور زیر را برای دریافت مخزن اجرا کنید:
<div dir="ltr">
```sh
go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest
```
</div>
برای اطلاعات بیشتر درباره نحوه نصب nuclei، به آدرس `https://docs.projectdiscovery.io/tools/nuclei/install` مراجعه کنید.
### دستورات خط فرمان
برای نمایش تمامی پارامترها و سوییچ‌های این ابزار:
<div dir="ltr">
```sh
nuclei -h
```
</div>
<details>
  <summary>گسترش تمام فلگ‌های راهنما</summary>
<div dir="ltr">
```yaml
Nuclei is a fast, template based vulnerability scanner focusing
on extensive configurability, massive extensibility and ease of use.
Usage:
./nuclei [flags]
Flags:
TARGET:
-u, -target string[] target URLs/hosts to scan
-l, -list string path to file containing a list of target URLs/hosts to scan (one per line)
-targets-inline string inline multiline target list (for use in template profiles)
-eh, -exclude-hosts string[] hosts to exclude to scan from the input list (ip, cidr, hostname)
-resume string resume scan from and save to specified file (clustering will be disabled)
-sa, -scan-all-ips scan all the IP's associated with dns record
-iv, -ip-version string[] IP version to scan of hostname (4,6) - (default 4)
-dun, -disable-url-normalization keep targets exactly as supplied instead of collapsing urls that address the same resource
-mup, -max-urls-per-pattern int scan at most this many urls sharing one structural shape, e.g. /user/1 and /user/2 (0 to scan all)
TARGET-FORMAT:
-im, -input-mode string mode of input file (list, burp, jsonl, yaml, openapi, swagger, http) (default "list")
-ro, -required-only use only required fields in input format when generating requests
-sfv, -skip-format-validation skip format validation (like missing vars) when parsing input file
-vtt, -vars-text-templating enable text templating for vars in input file (only for yaml input mode)
-vfp, -var-file-paths string[] list of yaml file contained vars to inject into yaml input
TEMPLATES:
-nt, -new-templates run only new templates added in latest nuclei-templates release
-ntv, -new-templates-version string[] run new templates added in specific version
-as, -automatic-scan automatic web scan using wappalyzer technology detection to tags mapping
-t, -templates string[] list of template or template directory to run (comma-separated, file)
-turl, -template-url string[] template url or list containing template urls to run (comma-separated, file)
-ai, -prompt string generate and run template using ai prompt
-w, -workflows string[] list of workflow or workflow directory to run (comma-separated, file)
-wurl, -workflow-url string[] workflow url or list containing workflow urls to run (comma-separated, file)
-validate validate the passed templates to nuclei
-nss, -no-strict-syntax disable strict syntax check on templates
-td, -template-display displays the templates content
-tl list all templates matching current filters
-tgl list all available tags
-sign signs the templates with the private key defined in NUCLEI_SIGNATURE_PRIVATE_KEY env variable
-code enable loading code protocol-based templates
-dut, -disable-unsigned-templates disable running unsigned templates or templates with mismatched signature
-esc, -enable-self-contained enable loading self-contained templates
-egm, -enable-global-matchers enable loading global matchers templates
-file enable loading file templates
-llm enable llm matchers and extractors (semantic matching)
-llm-provider string llm provider for semantic matching (openai, ollama, llamacpp, vllm, lmstudio, groq, openrouter, together, orcarouter) (default "openai")
-llm-base-url string openai-compatible endpoint for semantic matching (overrides -llm-provider, e.g. a local model)
-llm-model string model used for semantic matching
-llm-timeout int time in seconds to wait for a single llm call (default 30)
-llm-max-calls int maximum llm calls per scan (0 for unlimited) (default 5000)
-llm-concurrency int maximum concurrent llm calls (default 4)
-llm-cache cache llm responses within a scan (default true)
FILTERING:
-a, -author string[] templates to run based on authors (comma-separated, file)
-tags string[] templates to run based on tags (comma-separated, file)
-etags, -exclude-tags string[] templates to exclude based on tags (comma-separated, file)
-itags, -include-tags string[] tags to be executed even if they are excluded either by default or configuration
-id, -template-id string[] templates to run based on template ids (comma-separated, file, allow-wildcard)
-eid, -exclude-id string[] templates to exclude based on template ids (comma-separated, file)
-it, -include-templates string[] path to template file or directory to be executed even if they are excluded either by default or configuration
-et, -exclude-templates string[] path to template file or directory to exclude (comma-separated, file)
-em, -exclude-matchers string[] template matchers to exclude in result
-s, -severity value[] templates to run based on severity. Possible values: info, low, medium, high, critical, unknown
-es, -exclude-severity value[] templates to exclude based on severity. Possible values: info, low, medium, high, critical, unknown
-pt, -type value[] templates to run based on protocol type. Possible values: dns, file, http, headless, tcp, workflow, ssl, websocket, whois, code, javascript
-ept, -exclude-type value[] templates to exclude based on protocol type. Possible values: dns, file, http, headless, tcp, workflow, ssl, websocket, whois, code, javascript
-tc, -template-condition string[] templates to run based on expression condition
OUTPUT:
-o, -output string output file to write found issues/vulnerabilities
-sresp, -store-resp store all request/response passed through nuclei to output directory
-srd, -store-resp-dir string store all request/response passed through nuclei to custom directory (default "output")
-silent display findings only
-nc, -no-color disable output content coloring (ANSI escape codes)
-j, -jsonl write output in JSONL(ines) format
-irr, -include-rr -omit-raw include request/response pairs in the JSON, JSONL, and Markdown outputs (for findings only) [DEPRECATED use -omit-raw] (default true)
-or, -omit-raw omit request/response pairs in the JSON, JSONL, and Markdown outputs (for findings only)
-ot, -omit-template omit encoded template in the JSON, JSONL output
-nm, -no-meta disable printing result metadata in cli output
-ts, -timestamp enables printing timestamp in cli output
-rdb, -report-db string nuclei reporting database (always use this to persist report data)
-ms, -matcher-status display match failure status
-me, -markdown-export string directory to export results in markdown format
-se, -sarif-export string file to export results in SARIF format
-je, -json-export string file to export results in JSON format
-jle, -jsonl-export string file to export results in JSONL(ine) format
-pe, -pdf-export string file to export results in PDF format
-rd, -redact string[] redact given list of keys from query parameter, request header and body
CONFIGURATIONS:
-config string path to the nuclei configuration file
-tp, -profile string template profile config file to run
-tpl, -profile-list list community template profiles
-fr, -follow-redirects enable following redirects for http templates
-fhr, -follow-host-redirects follow redirects on the same host
-mr, -max-redirects int max number of redirects to follow for http templates (default 10)
-dr, -disable-redirects disable redirects for http templates
-rc, -report-config string nuclei reporting module configuration file
-H, -header string[] custom header/cookie to include in all http request in header:value format (cli, file)
-V, -var value custom vars in key=value format
-r, -resolvers string file containing resolver list for nuclei
-sr, -system-resolvers use system DNS resolving as error fallback
-dc, -disable-clustering disable clustering of requests
-passive enable passive HTTP response processing mode
-fh2, -force-http2 force http2 connection on requests
-ev, -env-vars enable environment variables to be used in template
-cc, -client-cert string client certificate file (PEM-encoded) used for authenticating against scanned hosts
-ck, -client-key string client key file (PEM-encoded) used for authenticating against scanned hosts
-ca, -client-ca string client certificate authority file (PEM-encoded) used for authenticating against scanned hosts
-sml, -show-match-line show match lines for file templates, works with extractors only
-ztls use ztls library with autofallback to standard one for tls13 [Deprecated] autofallback to ztls is enabled by default
-sni string tls sni hostname to use (default: input domain name)
-dka, -dialer-keep-alive value keep-alive duration for network requests.
-lfa, -allow-local-file-access allow all templates in the run to read any file on the system (not just payloads)
-lna, -restrict-local-network-access blocks connections to the local / private network
-i, -interface string network interface to use for network scan
-at, -attack-type string type of payload combinations to perform (batteringram,pitchfork,clusterbomb)
-pro, -payload-random-order iterate payload combinations in a random order instead of sequentially
-sip, -source-ip string source ip address to use for network scan
-rsr, -response-size-read int max response size to read in bytes
-rss, -response-size-save int max response size to read in bytes (default 1048576)
-rqss, -request-size-save int max request body size to save in bytes (0 to save the whole body) (default 32768)
-reset reset removes all nuclei configuration and data files (including nuclei-templates)
-tlsi, -tls-impersonate enable experimental client hello (ja3) tls randomization
-hae, -http-api-endpoint string experimental http api endpoint
INTERACTSH:
-iserver, -interactsh-server string interactsh server url for self-hosted instance (default: oast.pro,oast.live,oast.site,oast.online,oast.fun,oast.me)
-itoken, -interactsh-token string authentication token for self-hosted interactsh server
-interactions-cache-size int number of requests to keep in the interactions cache (default 5000)
-interactions-eviction int number of seconds to wait before evicting requests from cache (default 60)
-interactions-poll-duration int number of seconds to wait before each interaction poll request (default 5)
-interactions-cooldown-period int extra time for interaction polling before exiting (default 5)
-ni, -no-interactsh disable interactsh server for OAST testing, exclude OAST based templates
FUZZING:
-ft, -fuzzing-type string overrides fuzzing type set in template (replace, prefix, postfix, infix)
-fm, -fuzzing-mode string overrides fuzzing mode set in template (multiple, single)
-fuzz enable loading fuzzing templates (Deprecated: use -dast instead)
-dast enable / run dast (fuzz) nuclei templates
-dts, -dast-server enable dast server mode (live fuzzing)
-dtr, -dast-report write dast scan report to file
-dtst, -dast-server-token string dast server token (optional)
-dtsa, -dast-server-address string dast server address (default "localhost:9055")
-dtp, -dast-proxy enable dast proxy mode (fuzz live traffic proxied through nuclei)
-dtpa, -dast-proxy-address string dast proxy listen address (default "127.0.0.1:9056")
-dtpau, -dast-proxy-auth string dast proxy basic auth in user:pass format (required for non-loopback address)
-dfp, -display-fuzz-points display fuzz points in the output for debugging
-fuzz-param-frequency int frequency of uninteresting parameters for fuzzing before skipping (default 10)
-fa, -fuzz-aggression string fuzzing aggression level controls payload count for fuzz (low, medium, high) (default "low")
-cs, -fuzz-scope string[] in scope url regex to be followed by fuzzer
-cos, -fuzz-out-scope string[] out of scope url regex to be excluded by fuzzer
UNCOVER:
-uc, -uncover enable uncover engine
-uq, -uncover-query string[] uncover search query
-ue, -uncover-engine string[] uncover search engine (shodan,censys,fofa,shodan-idb,quake,hunter,zoomeye,netlas,criminalip,publicwww,hunterhow,google) (default shodan)
-uf, -uncover-field string uncover fields to return (ip,port,host) (default "ip:port")
-ul, -uncover-limit int uncover results to return (default 100)
-ur, -uncover-ratelimit int override ratelimit of engines with unknown ratelimit (default 60 req/min) (default 60)
RATE-LIMIT:
-rl, -rate-limit int maximum number of requests to send per second (default 150)
-rld, -rate-limit-duration value maximum number of requests to send per second (default 1s)
-host-backoff slow down a host that starts blocking (429, repeated 403s, connection failures)
-host-backoff-max value maximum delay applied to a host that is blocking (default 30s)
-per-host-rate-limit enable per-host rate limiting (global rate limit becomes unlimited when enabled)
-rlm, -rate-limit-minute int maximum number of requests to send per minute (DEPRECATED)
-bs, -bulk-size int maximum number of hosts to be analyzed in parallel per template (default 25)
-c, -concurrency int maximum number of templates to be executed in parallel (default 25)
-hbs, -headless-bulk-size int maximum number of headless hosts to be analyzed in parallel per template (default 10)
-headc, -headless-concurrency int maximum number of headless templates to be executed in parallel (default 10)
-jsc, -js-concurrency int maximum number of javascript runtimes to be executed in parallel (default 120)
-pc, -payload-concurrency int max payload concurrency for each template (default 25)
-prc, -probe-concurrency int http probe concurrency with httpx (default 50)
-tlc, -template-loading-concurrency int maximum number of concurrent template loading operations (default 50)
OPTIMIZATIONS:
-timeout int time to wait in seconds before timeout (default 10)
-retries int number of times to retry a failed request (default 1)
-ldp, -leave-default-ports leave default HTTP/HTTPS ports (eg. host:80,host:443)
-mhe, -max-host-error int max errors for a host before skipping from scan (default 30)
-te, -track-error string[] adds given error to max-host-error watchlist (standard, file)
-nmhe, -no-mhe disable skipping host from scan based on errors
-project use a project folder to avoid sending same request multiple times
-project-path string set a specific project path (default "/tmp")
-spm, -stop-at-first-match stop processing HTTP requests after the first match (may break template/workflow logic)
-stream stream mode - start elaborating without sorting the input
-ss, -scan-strategy value strategy to use while scanning(auto/host-spray/template-spray) (default auto)
-irt, -input-read-timeout value timeout on input read (default 3m0s)
-nh, -no-httpx disable httpx probing for non-url input
-preflight-portscan run preflight resolve + TCP portscan and filter targets before scanning (disabled by default)
-no-stdin disable stdin processing
HEADLESS:
-headless enable templates that require headless browser support (root user on Linux will disable sandbox)
-page-timeout int seconds to wait for each page in headless mode (default 20)
-sb, -show-browser show the browser on the screen when running templates with headless mode
-ho, -headless-options string[] start headless chrome with additional options
-sc, -system-chrome use local installed Chrome browser instead of nuclei installed
-cdpe, -cdp-endpoint string use remote browser via Chrome DevTools Protocol (CDP) endpoint
-lha, -list-headless-action list available headless actions
DEBUG:
-debug show all requests and responses
-dreq, -debug-req show all sent requests
-dresp, -debug-resp show all received responses
-p, -proxy string[] list of http/socks5 proxy to use (comma separated or file input)
-pi, -proxy-internal proxy all internal requests
-rp, -replay-proxy string http/socks5 proxy to re-send requests that produced a finding through (e.g. burp)
-ldf, -list-dsl-function list all supported DSL function signatures
-tlog, -trace-log string file to write sent requests trace log
-elog, -error-log string file to write sent requests error log
-version show nuclei version
-hm, -hang-monitor enable nuclei hang monitoring
-v, -verbose show verbose output
-profile-mem string generate memory (heap) profile & trace files
-vv display templates loaded for scan
-svd, -show-var-dump show variables dump for debugging
-vdl, -var-dump-limit int limit the number of characters displayed in var dump (default 255)
-ep, -enable-pprof enable pprof debugging server
-tv, -templates-version shows the version of the installed nuclei-templates
-hc, -health-check run diagnostic check up
UPDATE:
-up, -update update nuclei engine to the latest released version
-ut, -update-templates update nuclei-templates to latest released version
-utc, -update-templates-channel string nuclei-templates update channel (stable, latest) (default "stable")
-ud, -update-template-dir string custom directory to install / update nuclei-templates
-duc, -disable-update-check disable automatic nuclei/templates update check
HONEYPOT:
-hpd, -honeypot-detect detect potential honeypot hosts based on match concentration
-hpt, -honeypot-threshold int number of distinct template IDs required to flag a honeypot host (default 15)
-shp, -suppress-honeypot suppress output for flagged honeypot hosts
STATISTICS:
-stats display statistics about the running scan
-sj, -stats-json display statistics in JSONL(ines) format
-si, -stats-interval int number of seconds to wait between showing a statistics update (default 5)
-mp, -metrics-port int port to expose nuclei metrics on (default 9092)
-hps, -http-stats enable http status capturing (experimental)
CLOUD:
-auth configure projectdiscovery cloud (pdcp) api key (default true)
-tid, -team-id string upload scan results to given team id (optional) (default "none")
-cup, -cloud-upload upload scan results to pdcp dashboard [DEPRECATED use -dashboard]
-sid, -scan-id string upload scan results to existing scan id (optional)
-sname, -scan-name string scan name to set (optional)
-pd, -dashboard upload / view nuclei results in projectdiscovery cloud (pdcp) UI dashboard
-pdu, -dashboard-upload string upload / view nuclei results file (jsonl) in projectdiscovery cloud (pdcp) UI dashboard
AUTHENTICATION:
-sf, -secret-file string[] path to config file containing secrets for nuclei authenticated scan
-ps, -prefetch-secrets prefetch secrets from the secrets file
# NOTE: Headers in secrets files preserve exact casing (useful for case-sensitive APIs)
EXAMPLES:
Run nuclei on single host:
$ nuclei -target example.com
Run nuclei with specific template directories:
$ nuclei -target example.com -t http/cves/ -t ssl
Run nuclei against a list of hosts:
$ nuclei -list hosts.txt
Run nuclei with a JSON output:
$ nuclei -target example.com -json-export output.json
Run nuclei with sorted Markdown outputs (with environment variables):
$ MARKDOWN_EXPORT_SORT_MODE=template nuclei -target example.com -markdown-export nuclei_report/
Additional documentation is available at: https://docs.projectdiscovery.io/getting-started/running
```
</div>
مستندات تکمیلی در این آدرس در دسترس است: [**`docs.projectdiscovery.io/getting-started/running`**](https://docs.projectdiscovery.io/getting-started/running?utm_source=github&utm_medium=web&utm_campaign=nuclei_readme)
</details>
### اسکن یک هدف
برای انجام یک اسکن سریع روی یک وب‌اپلیکیشن:
<div dir="ltr">
```sh
nuclei -target https://example.com
```
</div>
### اسکن چندین هدف
Nuclei می‌تواند با ارائهٔ فهرستی از اهداف، اسکن گروهی را انجام دهد. می‌توانید از فایلی استفاده کنید که شامل چندین URL است.
<div dir="ltr">
```sh
nuclei -list urls.txt
```
</div>
### اسکن شبکه
این کار کل زیرشبکه را برای مشکلات مرتبط با شبکه، مانند پورت‌های باز یا سرویس‌های پیکربندی نادرست، اسکن می‌کند.
<div dir="ltr">
```sh
nuclei -target 192.168.1.0/24
```
</div>
### اسکن با استفاده از قالب سفارشی خودتان
برای نوشتن و استفاده از قالب خودتان، یک فایل `.yaml` با قوانین مشخص ایجاد کنید، سپس آن را به‌صورت زیر استفاده کنید.
<div dir="ltr">
```sh
nuclei -u https://example.com -t /path/to/your-template.yaml
```
</div>
### اتصال Nuclei به ProjectDiscovery
می‌توانید اسکن‌ها را روی دستگاه خود اجرا کرده و نتایج را برای تحلیل و رفع مشکلات بیشتر، در پلتفرم ابری بارگذاری کنید.
<div dir="ltr">
```sh
nuclei -target https://example.com -dashboard
```
</div>
> [!NOTE]
> این قابلیت کاملاً رایگان است و به هیچ‌گونه اشتراکی نیاز ندارد. برای راهنمایی دقیق‌تر، به [**`مستندات`**](https://docs.projectdiscovery.io/cloud/scanning/overview?utm_source=github&utm_medium=web&utm_campaign=nuclei_readme) مراجعه کنید.
<br>
<br>
## قالب‌های Nuclei، جامعه و پاداش‌ها 💎
[**قالب‌های Nuclei**](https://github.com/projectdiscovery/nuclei-templates) بر اساس مفهوم فایل‌های قالب مبتنی بر YAML طراحی شده‌اند که نحوه ارسال و پردازش درخواست‌ها را تعریف می‌کنند. این موضوع امکان توسعه‌پذیری آسان Nuclei را فراهم می‌کند. قالب‌ها با استفاده از YAML نوشته می‌شوند که قالبی ساده و قابل‌خواندن برای انسان ارائه می‌دهد تا بتوان فرآیند اجرا را به‌سرعت تعریف کرد.
**همین حالا به‌صورت آنلاین با ویرایشگر رایگان قالب‌های Nuclei مجهز به هوش مصنوعی ما امتحان کنید؛** [**`اینجا کلیک کنید`**](https://cloud.projectdiscovery.io/templates).
قالب‌های Nuclei روشی ساده و یکپارچه برای شناسایی و گزارش آسیب‌پذیری‌ها ارائه می‌دهند و جزئیات ضروری مانند سطح شدت و روش‌های شناسایی را در بر می‌گیرند. این ابزار متن‌باز که توسط جامعه توسعه داده می‌شود، واکنش به تهدیدات را سرعت می‌بخشد و در دنیای امنیت سایبری به‌طور گسترده شناخته شده است. قالب‌های Nuclei به‌طور فعال توسط هزاران پژوهشگر امنیتی در سراسر جهان توسعه داده می‌شوند. ما دو برنامه برای مشارکت‌کنندگان خود داریم: [**`Pioneers`**](https://projectdiscovery.io/pioneers) و [**`💎 پاداش‌ها`**](https://github.com/projectdiscovery/nuclei-templates/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22%F0%9F%92%8E%20Bounty%22).
<p align="right">
<a href="/static/nuclei-templates-teamcity.png" target="_blank"><img src="/static/nuclei-templates-teamcity.png" width="1200px" alt="نمونه قالب Nuclei برای شناسایی پیکربندی نادرست TeamCity" /></a>
</p>
### مثال‌ها
برای مشاهده موارد استفاده و ایده‌ها، به [**مستندات ما**](https://docs.projectdiscovery.io/templates/introduction) مراجعه کنید.
| قالب Nuclei | مورد استفاده |
| ---: | ---: |
| **[CVE-2021-44228 (Log4Shell)](https://cloud.projectdiscovery.io/public/CVE-2021-45046)** | شناسایی CVEهای شناخته‌شده |
| **[Blind SQL Injection via OOB](https://cloud.projectdiscovery.io/public/CVE-2024-22120)** | شناسایی آسیب‌پذیری‌های Out-of-Band |
| **[Generic SQL Injection](https://cloud.projectdiscovery.io/public/CVE-2022-34265)** | شناسایی SQL Injection |
| **[Reflected XSS Detection](https://cloud.projectdiscovery.io/public/CVE-2023-4173)** | Cross-Site Scripting (XSS) |
| **[Default Credentials Check](https://cloud.projectdiscovery.io/public/airflow-default-login)** | رمزهای عبور پیش‌فرض یا ضعیف |
| **[Sensitive File Disclosure](https://cloud.projectdiscovery.io/public/airflow-configuration-exposure)** | فایل‌های محرمانه یا افشای داده‌ها |
| **[Open Redirect Detection](https://cloud.projectdiscovery.io/public/open-redirect)** | شناسایی Open Redirect |
| **[Subdomain Takeover Templates](https://cloud.projectdiscovery.io/public/azure-takeover-detection)** | شناسایی Subdomain Takeover |
| **[Unprotected Jenkins Console](https://cloud.projectdiscovery.io/public/unauthenticated-jenkins)** | پیکربندی‌های نادرست امنیتی |
| **[SSL Certificate Expiry](https://cloud.projectdiscovery.io/public/expired-ssl)** | پیکربندی‌های ضعیف SSL/TLS |
| **[Open S3 Bucket Detection](https://cloud.projectdiscovery.io/public/s3-public-read-acp)** | سرویس‌های ابری با پیکربندی نادرست |
| **[RCE Detection Templates](https://cloud.projectdiscovery.io/public/CVE-2024-29824)** | آسیب‌پذیری‌های اجرای کد از راه دور |
| **[Path Traversal Detection](https://cloud.projectdiscovery.io/public/oracle-fatwire-lfi)** | حملات Directory Traversal |
| **[Local/Remote File Inclusion](https://cloud.projectdiscovery.io/public/CVE-2023-6977)** | آسیب‌پذیری‌های File Inclusion |
<br>
<br>
## ماموریت ما
اسکنرهای سنتی آسیب‌پذیری دهه‌ها پیش ساخته شده‌اند. آن‌ها متن‌بسته، فوق‌العاده کند و وابسته به فروشندگان هستند. مهاجمان امروزی، برخلاف گذشته که بهره‌برداری از آسیب‌پذیری‌ها سال‌ها طول می‌کشید، ظرف چند روز پس از انتشار CVEهای جدید، آن‌ها را در سراسر اینترنت به‌صورت گسترده مورد سوءاستفاده قرار می‌دهند. این تغییر، نیازمند رویکردی کاملاً متفاوت برای مقابله با اکسپلویت‌های پرطرفدار در اینترنت است.
ما Nuclei را برای حل این چالش ساختیم. ما کل چارچوب موتور اسکن را متن‌باز و قابل سفارشی‌سازی کردیم تا جامعه امنیتی جهانی بتواند با یکدیگر همکاری کرده و با بردارهای حمله و آسیب‌پذیری‌های پرطرفدار در اینترنت مقابله کند. امروزه Nuclei توسط شرکت‌های Fortune 500، سازمان‌های دولتی و دانشگاه‌ها استفاده و توسعه داده می‌شود.
شما نیز می‌توانید با مشارکت در کد ما، [**`کتابخانه قالب‌ها`**](https://github.com/projectdiscovery/nuclei-templates)، یا [**`پیوستن به تیم ما`**](https://projectdiscovery.io/) مشارکت کنید.
<br>
<br>
## مشارکت‌کنندگان :heart:
از تمام مشارکت‌کنندگان فوق‌العاده [**`جامعه که با ارسال PRها`**](https://github.com/projectdiscovery/nuclei/graphs/contributors) و به‌روز نگه‌داشتن این پروژه از آن حمایت می‌کنند، سپاسگزاریم. :heart:
<p align="left">
<a href="https://github.com/Ice3man543"><img src="https://avatars.githubusercontent.com/u/22318055?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/apps/dependabot"><img src="https://avatars.githubusercontent.com/in/29110?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/ehsandeep"><img src="https://avatars.githubusercontent.com/u/8293321?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/Mzack9999"><img src="https://avatars.githubusercontent.com/u/13421144?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/forgedhallpass"><img src="https://avatars.githubusercontent.com/u/13679401?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/tarunKoyalwar"><img src="https://avatars.githubusercontent.com/u/45962551?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/manuelbua"><img src="https://avatars.githubusercontent.com/u/819314?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/actions-user"><img src="https://avatars.githubusercontent.com/u/65916846?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/dogancanbakir"><img src="https://avatars.githubusercontent.com/u/65292895?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/parrasajad"><img src="https://avatars.githubusercontent.com/u/16835787?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/vzamanillo"><img src="https://avatars.githubusercontent.com/u/10209695?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/ShubhamRasal"><img src="https://avatars.githubusercontent.com/u/45902122?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/RamanaReddy0M"><img src="https://avatars.githubusercontent.com/u/90540245?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/LuitelSamikshya"><img src="https://avatars.githubusercontent.com/u/85764322?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/kchason"><img src="https://avatars.githubusercontent.com/u/1111099?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/pmareke"><img src="https://avatars.githubusercontent.com/u/3502075?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/dwisiswant0"><img src="https://avatars.githubusercontent.com/u/25837540?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/xm1k3"><img src="https://avatars.githubusercontent.com/u/73166077?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/5amu"><img src="https://avatars.githubusercontent.com/u/39925709?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/ehrishirajsharma"><img src="https://avatars.githubusercontent.com/u/35542790?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/zerodivisi0n"><img src="https://avatars.githubusercontent.com/u/687694?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/geeknik"><img src="https://avatars.githubusercontent.com/u/466878?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/TerminalFi"><img src="https://avatars.githubusercontent.com/u/32599364?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/KaulSe"><img src="https://avatars.githubusercontent.com/u/45340011?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/sullo"><img src="https://avatars.githubusercontent.com/u/1474884?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/wdahlenburg"><img src="https://avatars.githubusercontent.com/u/4451504?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/ghost"><img src="https://avatars.githubusercontent.com/u/10137?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/Nishan8583"><img src="https://avatars.githubusercontent.com/u/20457968?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/jdk2588"><img src="https://avatars.githubusercontent.com/u/985054?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/nothinux"><img src="https://avatars.githubusercontent.com/u/17433202?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/CodFrm"><img src="https://avatars.githubusercontent.com/u/22783163?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/CasperGN"><img src="https://avatars.githubusercontent.com/u/5549643?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/ankh2054"><img src="https://avatars.githubusercontent.com/u/6784287?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/revblock"><img src="https://avatars.githubusercontent.com/u/72813848?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/cn-kali-team"><img src="https://avatars.githubusercontent.com/u/30642514?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/EndPositive"><img src="https://avatars.githubusercontent.com/u/25148195?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/jimen0"><img src="https://avatars.githubusercontent.com/u/6826244?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/xstevens"><img src="https://avatars.githubusercontent.com/u/69216?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/mjkim610"><img src="https://avatars.githubusercontent.com/u/17107206?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/organiccrap"><img src="https://avatars.githubusercontent.com/u/376317?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/lu4nx"><img src="https://avatars.githubusercontent.com/u/3006875?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/souvikhazra1"><img src="https://avatars.githubusercontent.com/u/13842393?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/tovask"><img src="https://avatars.githubusercontent.com/u/22732484?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/Marmelatze"><img src="https://avatars.githubusercontent.com/u/199681?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/doug-threatmate"><img src="https://avatars.githubusercontent.com/u/127235272?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/yabeow"><img src="https://avatars.githubusercontent.com/u/21117771?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/olearycrew"><img src="https://avatars.githubusercontent.com/u/6044920?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/gano3s"><img src="https://avatars.githubusercontent.com/u/2551605?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/alizmhdi"><img src="https://avatars.githubusercontent.com/u/79321261?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/hackerpain"><img src="https://avatars.githubusercontent.com/u/61242234?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/lc"><img src="https://avatars.githubusercontent.com/u/19563282?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/savushkin-yauheni"><img src="https://avatars.githubusercontent.com/u/5173352?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/74616e696d"><img src="https://avatars.githubusercontent.com/u/97121933?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/edoardottt"><img src="https://avatars.githubusercontent.com/u/35783570?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/zt2"><img src="https://avatars.githubusercontent.com/u/7644862?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/moonD4rk"><img src="https://avatars.githubusercontent.com/u/24284231?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/wk8"><img src="https://avatars.githubusercontent.com/u/2536231?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/mikerott"><img src="https://avatars.githubusercontent.com/u/857712?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/toufik-airane"><img src="https://avatars.githubusercontent.com/u/5610269?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/hktalent"><img src="https://avatars.githubusercontent.com/u/18223385?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/jturner"><img src="https://avatars.githubusercontent.com/u/1825202?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/gaby"><img src="https://avatars.githubusercontent.com/u/835733?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/vavkamil"><img src="https://avatars.githubusercontent.com/u/47953210?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/leonjza"><img src="https://avatars.githubusercontent.com/u/1148127?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/mionskowski-form3"><img src="https://avatars.githubusercontent.com/u/91873652?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/chenrui333"><img src="https://avatars.githubusercontent.com/u/1580956?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/iamargus95"><img src="https://avatars.githubusercontent.com/u/77744293?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/shashikarsiddharth"><img src="https://avatars.githubusercontent.com/u/60960197?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/trypa11"><img src="https://avatars.githubusercontent.com/u/67585616?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/Zeokat"><img src="https://avatars.githubusercontent.com/u/1313154?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/alban-stourbe-wmx"><img src="https://avatars.githubusercontent.com/u/159776828?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/anykno"><img src="https://avatars.githubusercontent.com/u/2528207?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/ronaudinho"><img src="https://avatars.githubusercontent.com/u/10264710?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/boy-hack"><img src="https://avatars.githubusercontent.com/u/18695984?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/iuliu8899"><img src="https://avatars.githubusercontent.com/u/31680027?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/debasishbsws"><img src="https://avatars.githubusercontent.com/u/65381620?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/denysvitali-niantic"><img src="https://avatars.githubusercontent.com/u/157139422?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/fail-open"><img src="https://avatars.githubusercontent.com/u/72417455?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/Xc1Ym"><img src="https://avatars.githubusercontent.com/u/29765332?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/XTeam-Wing"><img src="https://avatars.githubusercontent.com/u/25416365?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/Weltolk"><img src="https://avatars.githubusercontent.com/u/40228052?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/tonghuaroot"><img src="https://avatars.githubusercontent.com/u/23011166?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/praetorian-thendrickson"><img src="https://avatars.githubusercontent.com/u/69640071?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/S0obi"><img src="https://avatars.githubusercontent.com/u/4180104?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/skahn007gl"><img src="https://avatars.githubusercontent.com/u/144735608?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/shouichi"><img src="https://avatars.githubusercontent.com/u/99586?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/seb-elttam"><img src="https://avatars.githubusercontent.com/u/111818823?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/AdallomRoy"><img src="https://avatars.githubusercontent.com/u/4046118?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/rotemreiss"><img src="https://avatars.githubusercontent.com/u/9288082?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/oscarintherocks"><img src="https://avatars.githubusercontent.com/u/1785821?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/xxcdd"><img src="https://avatars.githubusercontent.com/u/42600601?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/chen2aaron"><img src="https://avatars.githubusercontent.com/u/9978183?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/voidz0r"><img src="https://avatars.githubusercontent.com/u/1032286?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/vince-isec"><img src="https://avatars.githubusercontent.com/u/149686094?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/true13"><img src="https://avatars.githubusercontent.com/u/18207552?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/skhalsa-sigsci"><img src="https://avatars.githubusercontent.com/u/68570441?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/ShuBo6"><img src="https://avatars.githubusercontent.com/u/41125338?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/seeyarh"><img src="https://avatars.githubusercontent.com/u/16869800?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/securibee"><img src="https://avatars.githubusercontent.com/u/51520913?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/sduc"><img src="https://avatars.githubusercontent.com/u/2879617?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/scottdharvey"><img src="https://avatars.githubusercontent.com/u/25498254?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/rykkard"><img src="https://avatars.githubusercontent.com/u/51889048?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/monitor403"><img src="https://avatars.githubusercontent.com/u/45124775?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/mlec1"><img src="https://avatars.githubusercontent.com/u/42201667?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/meme-lord"><img src="https://avatars.githubusercontent.com/u/17912559?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/LazyMaple"><img src="https://avatars.githubusercontent.com/u/12314941?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/lvyaoting"><img src="https://avatars.githubusercontent.com/u/166296299?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/llussy"><img src="https://avatars.githubusercontent.com/u/18432966?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/linchizhen"><img src="https://avatars.githubusercontent.com/u/170242051?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/kiokuless"><img src="https://avatars.githubusercontent.com/u/110003596?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/Jarnpher553"><img src="https://avatars.githubusercontent.com/u/10233873?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/c-f"><img src="https://avatars.githubusercontent.com/u/35263248?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/hanghuge"><img src="https://avatars.githubusercontent.com/u/166206050?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/testwill"><img src="https://avatars.githubusercontent.com/u/8717479?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/galoget"><img src="https://avatars.githubusercontent.com/u/8353133?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/fudancoder"><img src="https://avatars.githubusercontent.com/u/171416994?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/revolunet"><img src="https://avatars.githubusercontent.com/u/124937?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/jsoref"><img src="https://avatars.githubusercontent.com/u/2119212?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/MachadoOtto"><img src="https://avatars.githubusercontent.com/u/93268441?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/jonathanwalker"><img src="https://avatars.githubusercontent.com/u/14978093?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/nHurD"><img src="https://avatars.githubusercontent.com/u/233374?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/jessekelly881"><img src="https://avatars.githubusercontent.com/u/22938931?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/JaneX8"><img src="https://avatars.githubusercontent.com/u/5116641?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/eltociear"><img src="https://avatars.githubusercontent.com/u/22633385?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/atomiczsec"><img src="https://avatars.githubusercontent.com/u/75549184?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/M-Faheem-Khan"><img src="https://avatars.githubusercontent.com/u/17150767?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/denandz"><img src="https://avatars.githubusercontent.com/u/5291556?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/tibbon"><img src="https://avatars.githubusercontent.com/u/82880?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/dany74q"><img src="https://avatars.githubusercontent.com/u/2129762?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/0x123456789"><img src="https://avatars.githubusercontent.com/u/36066426?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/danigoland"><img src="https://avatars.githubusercontent.com/u/15079567?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/ChrisMandich"><img src="https://avatars.githubusercontent.com/u/14286797?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/austintraver"><img src="https://avatars.githubusercontent.com/u/25112463?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/socialsister"><img src="https://avatars.githubusercontent.com/u/155628741?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/Anemys"><img src="https://avatars.githubusercontent.com/u/51196227?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/andreangelucci"><img src="https://avatars.githubusercontent.com/u/18552197?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/AlexS778"><img src="https://avatars.githubusercontent.com/u/98418121?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/noraj"><img src="https://avatars.githubusercontent.com/u/16578570?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/akkuman"><img src="https://avatars.githubusercontent.com/u/7813511?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/zrquan"><img src="https://avatars.githubusercontent.com/u/33086594?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/1efty"><img src="https://avatars.githubusercontent.com/u/18194777?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/rsrdesarrollo"><img src="https://avatars.githubusercontent.com/u/5142014?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/razin99"><img src="https://avatars.githubusercontent.com/u/44442082?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/MetzinAround"><img src="https://avatars.githubusercontent.com/u/65838556?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/vil02"><img src="https://avatars.githubusercontent.com/u/65706193?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/mrschyte"><img src="https://avatars.githubusercontent.com/u/8571?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/PeterDaveHello"><img src="https://avatars.githubusercontent.com/u/3691490?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/parthmalhotra"><img src="https://avatars.githubusercontent.com/u/28601533?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/owenrumney"><img src="https://avatars.githubusercontent.com/u/3049157?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/Ovi3"><img src="https://avatars.githubusercontent.com/u/29408109?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/Bisstocuz"><img src="https://avatars.githubusercontent.com/u/42398278?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/daffainfo"><img src="https://avatars.githubusercontent.com/u/36522826?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/mhmdiaa"><img src="https://avatars.githubusercontent.com/u/19687798?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/MiryangJung"><img src="https://avatars.githubusercontent.com/u/48237511?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/0xmin"><img src="https://avatars.githubusercontent.com/u/44919834?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/maikthulhu"><img src="https://avatars.githubusercontent.com/u/680830?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/sttlr"><img src="https://avatars.githubusercontent.com/u/40246850?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/iamRjarpan"><img src="https://avatars.githubusercontent.com/u/45498226?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/leoloobeek"><img src="https://avatars.githubusercontent.com/u/8801754?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/KristinnVikar"><img src="https://avatars.githubusercontent.com/u/93918469?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/kant01ne"><img src="https://avatars.githubusercontent.com/u/5072452?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/KeisukeYamashita"><img src="https://avatars.githubusercontent.com/u/23056537?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
<a href="https://github.com/1hehaq"><img src="https://avatars.githubusercontent.com/u/162917546?v=4" width="50" height="50" alt="" style="max-width: 100%;"></a>
</p>
<br>
<br>
<br>
<div align="center">
<sub>**`nuclei`** تحت [**مجوز MIT**](https://github.com/projectdiscovery/nuclei/blob/main/LICENSE.md) منتشر شده است.</sub>
</div>
</div>
+10 -7
View File
@@ -9,7 +9,8 @@
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_ES.md">`Spanish`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_JP.md">`日本語`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_PT-BR.md">`Portuguese`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_TR.md">`Türkçe`</a>
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_TR.md">`Türkçe`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_FA.md">`فارسی`</a>
</div>
@@ -110,7 +111,7 @@ Telusuri [**`dokumentasi Nuclei lengkap di sini`**](https://docs.projectdiscover
### Instalasi
`nuclei` memerlukan **go >= 1.24.2** agar dapat diinstal dengan sukses. Jalankan perintah berikut untuk mengambil repo:
`nuclei` memerlukan **go >= 1.27.3** agar dapat diinstal dengan sukses. Jalankan perintah berikut untuk mengambil repo:
```sh
go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest
@@ -174,7 +175,7 @@ TEMPLATES:
-egm, -enable-global-matchers enable loading global matchers templates
-file enable loading file templates
-llm enable llm matchers and extractors (semantic matching)
-llm-provider string llm provider for semantic matching (openai, ollama, llamacpp, vllm, lmstudio, groq, openrouter, together) (default "openai")
-llm-provider string llm provider for semantic matching (openai, ollama, llamacpp, vllm, lmstudio, groq, openrouter, together, orcarouter) (default "openai")
-llm-base-url string openai-compatible endpoint for semantic matching (overrides -llm-provider, e.g. a local model)
-llm-model string model used for semantic matching
-llm-timeout int time in seconds to wait for a single llm call (default 30)
@@ -250,6 +251,7 @@ CONFIGURATIONS:
-sip, -source-ip string source ip address to use for network scan
-rsr, -response-size-read int max response size to read in bytes
-rss, -response-size-save int max response size to read in bytes (default 1048576)
-rqss, -request-size-save int max request body size to save in bytes (0 to save the whole body) (default 32768)
-reset reset removes all nuclei configuration and data files (including nuclei-templates)
-tlsi, -tls-impersonate enable experimental client hello (ja3) tls randomization
-hae, -http-api-endpoint string experimental http api endpoint
@@ -350,10 +352,11 @@ DEBUG:
-hc, -health-check run diagnostic check up
UPDATE:
-up, -update update nuclei engine to the latest released version
-ut, -update-templates update nuclei-templates to latest released version
-ud, -update-template-dir string custom directory to install / update nuclei-templates
-duc, -disable-update-check disable automatic nuclei/templates update check
-up, -update update nuclei engine to the latest released version
-ut, -update-templates update nuclei-templates to latest released version
-utc, -update-templates-channel string nuclei-templates update channel (stable, latest) (default "stable")
-ud, -update-template-dir string custom directory to install / update nuclei-templates
-duc, -disable-update-check disable automatic nuclei/templates update check
HONEYPOT:
-hpd, -honeypot-detect detect potential honeypot hosts based on match concentration
+10 -7
View File
@@ -9,7 +9,8 @@
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_ES.md">`Spanish`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_JP.md">`日本語`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_PT-BR.md">`Portuguese`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_TR.md">`Türkçe`</a>
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_TR.md">`Türkçe`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_FA.md">`فارسی`</a>
</div>
@@ -110,7 +111,7 @@ Nuclei の完全な[**`ドキュメントはこちら`**](https://docs.projectdi
### インストール
`nuclei` のインストールには **go >= 1.24.2** が必要です。以下のコマンドを実行してリポジトリを取得してください:
`nuclei` のインストールには **go >= 1.27.3** が必要です。以下のコマンドを実行してリポジトリを取得してください:
```sh
go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest
@@ -174,7 +175,7 @@ TEMPLATES:
-egm, -enable-global-matchers enable loading global matchers templates
-file enable loading file templates
-llm enable llm matchers and extractors (semantic matching)
-llm-provider string llm provider for semantic matching (openai, ollama, llamacpp, vllm, lmstudio, groq, openrouter, together) (default "openai")
-llm-provider string llm provider for semantic matching (openai, ollama, llamacpp, vllm, lmstudio, groq, openrouter, together, orcarouter) (default "openai")
-llm-base-url string openai-compatible endpoint for semantic matching (overrides -llm-provider, e.g. a local model)
-llm-model string model used for semantic matching
-llm-timeout int time in seconds to wait for a single llm call (default 30)
@@ -250,6 +251,7 @@ CONFIGURATIONS:
-sip, -source-ip string source ip address to use for network scan
-rsr, -response-size-read int max response size to read in bytes
-rss, -response-size-save int max response size to read in bytes (default 1048576)
-rqss, -request-size-save int max request body size to save in bytes (0 to save the whole body) (default 32768)
-reset reset removes all nuclei configuration and data files (including nuclei-templates)
-tlsi, -tls-impersonate enable experimental client hello (ja3) tls randomization
-hae, -http-api-endpoint string experimental http api endpoint
@@ -350,10 +352,11 @@ DEBUG:
-hc, -health-check run diagnostic check up
UPDATE:
-up, -update update nuclei engine to the latest released version
-ut, -update-templates update nuclei-templates to latest released version
-ud, -update-template-dir string custom directory to install / update nuclei-templates
-duc, -disable-update-check disable automatic nuclei/templates update check
-up, -update update nuclei engine to the latest released version
-ut, -update-templates update nuclei-templates to latest released version
-utc, -update-templates-channel string nuclei-templates update channel (stable, latest) (default "stable")
-ud, -update-template-dir string custom directory to install / update nuclei-templates
-duc, -disable-update-check disable automatic nuclei/templates update check
HONEYPOT:
-hpd, -honeypot-detect detect potential honeypot hosts based on match concentration
+10 -7
View File
@@ -9,7 +9,8 @@
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_ES.md">`Spanish`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_JP.md">`日本語`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_PT-BR.md">`Portuguese`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_TR.md">`Türkçe`</a>
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_TR.md">`Türkçe`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_FA.md">`فارسی`</a>
</div>
@@ -110,7 +111,7 @@ Nuclei 전체 [**`문서를 여기에서`**](https://docs.projectdiscovery.io/to
### 설치
`nuclei` 설치에는 **go >= 1.24.2** 가 필요합니다. 다음 명령어를 실행하여 저장소를 가져옵니다:
`nuclei` 설치에는 **go >= 1.27.3** 가 필요합니다. 다음 명령어를 실행하여 저장소를 가져옵니다:
```sh
go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest
@@ -174,7 +175,7 @@ TEMPLATES:
-egm, -enable-global-matchers enable loading global matchers templates
-file enable loading file templates
-llm enable llm matchers and extractors (semantic matching)
-llm-provider string llm provider for semantic matching (openai, ollama, llamacpp, vllm, lmstudio, groq, openrouter, together) (default "openai")
-llm-provider string llm provider for semantic matching (openai, ollama, llamacpp, vllm, lmstudio, groq, openrouter, together, orcarouter) (default "openai")
-llm-base-url string openai-compatible endpoint for semantic matching (overrides -llm-provider, e.g. a local model)
-llm-model string model used for semantic matching
-llm-timeout int time in seconds to wait for a single llm call (default 30)
@@ -250,6 +251,7 @@ CONFIGURATIONS:
-sip, -source-ip string source ip address to use for network scan
-rsr, -response-size-read int max response size to read in bytes
-rss, -response-size-save int max response size to read in bytes (default 1048576)
-rqss, -request-size-save int max request body size to save in bytes (0 to save the whole body) (default 32768)
-reset reset removes all nuclei configuration and data files (including nuclei-templates)
-tlsi, -tls-impersonate enable experimental client hello (ja3) tls randomization
-hae, -http-api-endpoint string experimental http api endpoint
@@ -350,10 +352,11 @@ DEBUG:
-hc, -health-check run diagnostic check up
UPDATE:
-up, -update update nuclei engine to the latest released version
-ut, -update-templates update nuclei-templates to latest released version
-ud, -update-template-dir string custom directory to install / update nuclei-templates
-duc, -disable-update-check disable automatic nuclei/templates update check
-up, -update update nuclei engine to the latest released version
-ut, -update-templates update nuclei-templates to latest released version
-utc, -update-templates-channel string nuclei-templates update channel (stable, latest) (default "stable")
-ud, -update-template-dir string custom directory to install / update nuclei-templates
-duc, -disable-update-check disable automatic nuclei/templates update check
HONEYPOT:
-hpd, -honeypot-detect detect potential honeypot hosts based on match concentration
+10 -7
View File
@@ -9,7 +9,8 @@
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_ES.md">`Spanish`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_JP.md">`日本語`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_PT-BR.md">`Portuguese`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_TR.md">`Türkçe`</a>
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_TR.md">`Türkçe`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_FA.md">`فارسی`</a>
</div>
@@ -110,7 +111,7 @@ Acesse a [**`documentação completa do Nuclei aqui`**](https://docs.projectdisc
### Instalação
O `nuclei` requer **go >= 1.24.2** para instalar com sucesso. Execute o seguinte comando para obter o repositório:
O `nuclei` requer **go >= 1.27.3** para instalar com sucesso. Execute o seguinte comando para obter o repositório:
```sh
go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest
@@ -174,7 +175,7 @@ TEMPLATES:
-egm, -enable-global-matchers enable loading global matchers templates
-file enable loading file templates
-llm enable llm matchers and extractors (semantic matching)
-llm-provider string llm provider for semantic matching (openai, ollama, llamacpp, vllm, lmstudio, groq, openrouter, together) (default "openai")
-llm-provider string llm provider for semantic matching (openai, ollama, llamacpp, vllm, lmstudio, groq, openrouter, together, orcarouter) (default "openai")
-llm-base-url string openai-compatible endpoint for semantic matching (overrides -llm-provider, e.g. a local model)
-llm-model string model used for semantic matching
-llm-timeout int time in seconds to wait for a single llm call (default 30)
@@ -250,6 +251,7 @@ CONFIGURATIONS:
-sip, -source-ip string source ip address to use for network scan
-rsr, -response-size-read int max response size to read in bytes
-rss, -response-size-save int max response size to read in bytes (default 1048576)
-rqss, -request-size-save int max request body size to save in bytes (0 to save the whole body) (default 32768)
-reset reset removes all nuclei configuration and data files (including nuclei-templates)
-tlsi, -tls-impersonate enable experimental client hello (ja3) tls randomization
-hae, -http-api-endpoint string experimental http api endpoint
@@ -350,10 +352,11 @@ DEBUG:
-hc, -health-check run diagnostic check up
UPDATE:
-up, -update update nuclei engine to the latest released version
-ut, -update-templates update nuclei-templates to latest released version
-ud, -update-template-dir string custom directory to install / update nuclei-templates
-duc, -disable-update-check disable automatic nuclei/templates update check
-up, -update update nuclei engine to the latest released version
-ut, -update-templates update nuclei-templates to latest released version
-utc, -update-templates-channel string nuclei-templates update channel (stable, latest) (default "stable")
-ud, -update-template-dir string custom directory to install / update nuclei-templates
-duc, -disable-update-check disable automatic nuclei/templates update check
HONEYPOT:
-hpd, -honeypot-detect detect potential honeypot hosts based on match concentration
+10 -7
View File
@@ -9,7 +9,8 @@
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_ES.md">`Spanish`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_JP.md">`日本語`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_PT-BR.md">`Portuguese`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_TR.md">`Türkçe`</a>
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_TR.md">`Türkçe`</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/main/README_FA.md">`فارسی`</a>
</div>
@@ -110,7 +111,7 @@ Nuclei'nin tam [**`dokümantasyonuna buradan`**](https://docs.projectdiscovery.i
### Kurulum
`nuclei` yüklemek için **go >= 1.24.2** gerektirir. Repoyu almak için aşağıdaki komutu çalıştırın:
`nuclei` yüklemek için **go >= 1.27.3** gerektirir. Repoyu almak için aşağıdaki komutu çalıştırın:
```sh
go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest
@@ -174,7 +175,7 @@ TEMPLATES:
-egm, -enable-global-matchers enable loading global matchers templates
-file enable loading file templates
-llm enable llm matchers and extractors (semantic matching)
-llm-provider string llm provider for semantic matching (openai, ollama, llamacpp, vllm, lmstudio, groq, openrouter, together) (default "openai")
-llm-provider string llm provider for semantic matching (openai, ollama, llamacpp, vllm, lmstudio, groq, openrouter, together, orcarouter) (default "openai")
-llm-base-url string openai-compatible endpoint for semantic matching (overrides -llm-provider, e.g. a local model)
-llm-model string model used for semantic matching
-llm-timeout int time in seconds to wait for a single llm call (default 30)
@@ -250,6 +251,7 @@ CONFIGURATIONS:
-sip, -source-ip string source ip address to use for network scan
-rsr, -response-size-read int max response size to read in bytes
-rss, -response-size-save int max response size to read in bytes (default 1048576)
-rqss, -request-size-save int max request body size to save in bytes (0 to save the whole body) (default 32768)
-reset reset removes all nuclei configuration and data files (including nuclei-templates)
-tlsi, -tls-impersonate enable experimental client hello (ja3) tls randomization
-hae, -http-api-endpoint string experimental http api endpoint
@@ -350,10 +352,11 @@ DEBUG:
-hc, -health-check run diagnostic check up
UPDATE:
-up, -update update nuclei engine to the latest released version
-ut, -update-templates update nuclei-templates to latest released version
-ud, -update-template-dir string custom directory to install / update nuclei-templates
-duc, -disable-update-check disable automatic nuclei/templates update check
-up, -update update nuclei engine to the latest released version
-ut, -update-templates update nuclei-templates to latest released version
-utc, -update-templates-channel string nuclei-templates update channel (stable, latest) (default "stable")
-ud, -update-template-dir string custom directory to install / update nuclei-templates
-duc, -disable-update-check disable automatic nuclei/templates update check
HONEYPOT:
-hpd, -honeypot-detect detect potential honeypot hosts based on match concentration
+20
View File
@@ -530,6 +530,16 @@ Description of the template.
You can go in-depth here on what the template actually does.
Description, impact and remediation may hold {{placeholders}} filled for
each finding: named extractors, payloads, variables and DSL helpers
(network helpers excluded), and the result fields host, port, ip, url,
scheme, path, matched-at, matcher-name and extractor-name, which win over
values of the same name. Use join(", ", name) for an extractor with
several values, and brackets for a hyphenated name inside a helper:
to_upper([matcher-name]). Unknown placeholders stay as written, and so
does a field whose expression fails. Global matcher templates match on
other templates' responses, so only the result fields fill their text.
Examples:
@@ -543,6 +553,10 @@ description: Bower is a package manager which stores package information in the
description: Subversion ALM for the enterprise before 8.8.2 allows reflected XSS at multiple locations
```
```yaml
description: Redis {{version}} on {{host}}:{{port}} accepts the default password
```
</div>
@@ -559,6 +573,8 @@ Impact of the template.
You can go in-depth here on impact of the template.
Supports the same placeholders as description.
Examples:
@@ -665,6 +681,8 @@ Remediation steps for the template.
You can go in-depth here on how to mitigate the problem found by this template.
Supports the same placeholders as description.
Examples:
@@ -1011,6 +1029,7 @@ Part Definitions:
- <code>headers_from_response</code> - HTTP response headers in name:value format
- <code>tls_version</code> - TLS version negotiated for the HTTP connection
- <code>cipher</code> - TLS cipher suite negotiated for the HTTP connection
- <code>key_exchange</code> - TLS key exchange group negotiated for the HTTP connection
- <code>sni</code> - SNI value used in the TLS handshake
- <code>subject_cn</code> - Leaf certificate subject common name
- <code>subject_dn</code> - Leaf certificate subject distinguished name
@@ -4144,6 +4163,7 @@ Part Definitions:
- <code>issuer_cn</code> - Issuer CN is the common name of the certificate issuer
- <code>issuer_dn</code> - Issuer DN is the distinguished name of the certificate issuer
- <code>issuer_org</code> - Issuer organization is the organization of the certificate issuer
- <code>key_exchange</code> - Key exchange is the TLS key exchange group negotiated in the handshake
- <code>not_after</code> - Timestamp after which the remote cert expires
- <code>not_before</code> - Timestamp before which the certificate is not valid
- <code>probe_status</code> - Probe status indicates if the probe was successful
+4
View File
@@ -23,6 +23,8 @@ extend-exclude = [
]
[default.extend-identifiers]
# Proper noun (OpenVAS scanner, source of the QoD confidence model)
OpenVAS = "OpenVAS"
# Go identifiers that cannot be renamed without breaking API
MisMatched = "MisMatched"
IsMisMatchedCert = "IsMisMatchedCert" # tlsx clients API
@@ -32,6 +34,8 @@ AllowdTypes = "AllowdTypes"
ExludedDastTmplStats = "ExludedDastTmplStats"
[default.extend-words]
# SMTP greeting command
HELO = "HELO"
# External package API (fileutil.IsWriteable)
Writeable = "Writeable" # TODO(dwisiswant0): Fix in upstream.
# Variable name used for flow annotations
+2 -1
View File
@@ -57,7 +57,8 @@
# severity: low
# # duplicate-issue-check (optional) flag to enable duplicate tracking issue check
# duplicate-issue-check: false
# # duplicate-issue-page-size (optional) controls how many issues to fetch per page when searching for duplicates
# # duplicate-issue-page-size (optional) sets the number of issues to request per page when searching for duplicates
# # (GitLab returns at most 100 issues per page)
# duplicate-issue-page-size: 100
# # duplicate-issue-max-pages (optional) limits how many pages to fetch when searching for duplicates (0 = no limit)
# duplicate-issue-max-pages: 0
+7 -1
View File
@@ -244,6 +244,8 @@ func main() {
}
}
// readConfig defines the cli flags and loads them from the command line and
// config files.
func readConfig() *goflags.FlagSet {
var selectedFiles configuration.Selection
// when true updates nuclei binary to latest version
@@ -307,7 +309,7 @@ on extensive configurability, massive extensibility and ease of use.`)
flagSet.BoolVarP(&options.EnableGlobalMatchersTemplates, "enable-global-matchers", "egm", false, "enable loading global matchers templates"),
flagSet.BoolVar(&options.EnableFileTemplates, "file", false, "enable loading file templates"),
flagSet.BoolVar(&options.EnableLLM, "llm", false, "enable llm matchers and extractors (semantic matching)"),
flagSet.StringVar(&options.LLMProvider, "llm-provider", "openai", "llm provider for semantic matching (openai, ollama, llamacpp, vllm, lmstudio, groq, openrouter, together)"),
flagSet.StringVar(&options.LLMProvider, "llm-provider", "openai", "llm provider for semantic matching (openai, ollama, llamacpp, vllm, lmstudio, groq, openrouter, together, orcarouter)"),
flagSet.StringVar(&options.LLMBaseURL, "llm-base-url", "", "openai-compatible endpoint for semantic matching (overrides -llm-provider, e.g. a local model)"),
flagSet.StringVar(&options.LLMModel, "llm-model", "", "model used for semantic matching"),
flagSet.IntVar(&options.LLMTimeout, "llm-timeout", 30, "time in seconds to wait for a single llm call"),
@@ -383,9 +385,11 @@ on extensive configurability, massive extensibility and ease of use.`)
flagSet.BoolVarP(&options.RestrictLocalNetworkAccess, "restrict-local-network-access", "lna", false, "blocks connections to the local / private network"),
flagSet.StringVarP(&options.Interface, "interface", "i", "", "network interface to use for network scan"),
flagSet.StringVarP(&options.AttackType, "attack-type", "at", "", "type of payload combinations to perform (batteringram,pitchfork,clusterbomb)"),
flagSet.BoolVarP(&options.PayloadRandomOrder, "payload-random-order", "pro", false, "iterate payload combinations in a random order instead of sequentially"),
flagSet.StringVarP(&options.SourceIP, "source-ip", "sip", "", "source ip address to use for network scan"),
flagSet.IntVarP(&options.ResponseReadSize, "response-size-read", "rsr", 0, "max response size to read in bytes"),
flagSet.IntVarP(&options.ResponseSaveSize, "response-size-save", "rss", unitutils.Mega, "max response size to read in bytes"),
flagSet.IntVarP(&options.RequestSaveSize, "request-size-save", "rqss", 32*unitutils.Kilo, "max request body size to save in bytes (0 to save the whole body)"),
flagSet.CallbackVar(func() { resetRequested = true }, "reset", "reset removes all nuclei configuration and data files (including nuclei-templates)"),
flagSet.BoolVarP(&options.TlsImpersonate, "tls-impersonate", "tlsi", false, "enable experimental client hello (ja3) tls randomization"),
flagSet.StringVarP(&options.HttpApiEndpoint, "http-api-endpoint", "hae", "", "experimental http api endpoint"),
@@ -465,6 +469,7 @@ on extensive configurability, massive extensibility and ease of use.`)
flagSet.BoolVarP(&options.DisableHTTPProbe, "no-httpx", "nh", false, "disable httpx probing for non-url input"),
flagSet.BoolVar(&options.PreflightPortScan, "preflight-portscan", false, "run preflight resolve + TCP portscan and filter targets before scanning (disabled by default)"),
flagSet.BoolVar(&options.DisableStdin, "no-stdin", false, "disable stdin processing"),
flagSet.BoolVarP(&options.ConfidenceBaseline, "confidence-baseline", "cob", false, "send a per-host control request to lower confidence on catch-all (false-positive) matches"),
flagSet.BoolVar(&options.EnableHTTPCache, "http-cache", false, "enable HTTP cache (RFC 9111) for HTTP requests"),
)
@@ -503,6 +508,7 @@ on extensive configurability, massive extensibility and ease of use.`)
flagSet.CreateGroup("update", "Update",
flagSet.BoolVarP(&updateNucleiBinary, "update", "up", false, "update nuclei engine to the latest released version"),
flagSet.BoolVarP(&options.UpdateTemplates, "update-templates", "ut", false, "update nuclei-templates to latest released version"),
flagSet.StringVarP(&options.UpdateTemplatesChannel, "update-templates-channel", "utc", installer.TemplatesChannelStable, "nuclei-templates update channel (stable, latest)"),
flagSet.StringVarP(&options.NewTemplatesDirectory, "update-template-dir", "ud", "", "custom directory to install / update nuclei-templates"),
flagSet.BoolVarP(&disableUpdateCheck, "disable-update-check", "duc", false, "disable automatic nuclei/templates update check"),
)
+11 -11
View File
@@ -1,6 +1,6 @@
module github.com/projectdiscovery/nuclei/v3
go 1.27.1
go 1.27.3
require (
carvel.dev/ytt v0.52.0
@@ -76,7 +76,7 @@ require (
github.com/praetorian-inc/fingerprintx v1.1.15
github.com/projectdiscovery/clistats v0.1.7
github.com/projectdiscovery/dsl v0.8.24
github.com/projectdiscovery/fastdialer v0.5.26
github.com/projectdiscovery/fastdialer v0.6.0
github.com/projectdiscovery/fasttemplate v0.0.2
github.com/projectdiscovery/gcache v0.0.0-20241015120333-12546c6e3f4c
github.com/projectdiscovery/goflags v0.2.1
@@ -92,7 +92,7 @@ require (
github.com/projectdiscovery/interactsh v1.4.1
github.com/projectdiscovery/mapcidr v1.1.97
github.com/projectdiscovery/n3iwf v0.0.0-20230523120440-b8cd232ff1f5
github.com/projectdiscovery/networkpolicy v0.1.55
github.com/projectdiscovery/networkpolicy v0.1.56
github.com/projectdiscovery/ratelimit v0.0.94
github.com/projectdiscovery/rawhttp v0.2.0
github.com/projectdiscovery/rdap v0.9.0
@@ -102,7 +102,7 @@ require (
github.com/projectdiscovery/tlsx v1.4.3-0.20261020052910-acb80c2c7c75
github.com/projectdiscovery/uncover v1.2.1
github.com/projectdiscovery/useragent v0.0.109
github.com/projectdiscovery/utils v0.11.10
github.com/projectdiscovery/utils v0.11.12-0.20261025172519-64b3c221cfe5
github.com/projectdiscovery/wappalyzergo v0.3.5
github.com/projectdiscovery/yamldoc-go v1.0.7
github.com/redis/go-redis/v9 v9.19.0
@@ -263,7 +263,7 @@ require (
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
github.com/go-git/go-billy/v5 v5.9.0 // indirect
github.com/go-jose/go-jose/v4 v4.1.4 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/logr v1.4.4 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.3.0 // indirect
github.com/go-openapi/jsonpointer v0.22.5 // indirect
@@ -349,7 +349,7 @@ require (
github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/moby/moby/api v1.54.2 // indirect
github.com/moby/moby/client v0.4.1 // indirect
github.com/moby/sys/user v0.4.0 // indirect
github.com/moby/sys/user v0.4.1 // indirect
github.com/moby/term v0.5.2 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
@@ -445,11 +445,11 @@ require (
go.opentelemetry.io/contrib/detectors/gcp v1.44.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.61.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.68.0 // indirect
go.opentelemetry.io/otel v1.44.0 // indirect
go.opentelemetry.io/otel/metric v1.44.0 // indirect
go.opentelemetry.io/otel/sdk v1.44.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.44.0 // indirect
go.opentelemetry.io/otel/trace v1.44.0 // indirect
go.opentelemetry.io/otel v1.45.0 // indirect
go.opentelemetry.io/otel/metric v1.45.0 // indirect
go.opentelemetry.io/otel/sdk v1.45.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.45.0 // indirect
go.opentelemetry.io/otel/trace v1.45.0 // indirect
go.uber.org/atomic v1.11.0 // indirect
go.uber.org/zap v1.27.1 // indirect
go.uber.org/zap/exp v0.3.0 // indirect
+22 -22
View File
@@ -470,8 +470,8 @@ github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V
github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A=
github.com/go-logfmt/logfmt v0.5.1/go.mod h1:WYhtIu8zTZfxdn5+rREduYbwxfcBr/Vr6KEVveWlfTs=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8=
github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
@@ -848,8 +848,8 @@ github.com/moby/moby/api v1.54.2 h1:wiat9QAhnDQjA7wk1kh/TqHz2I1uUA7M7t9SAl/JNXg=
github.com/moby/moby/api v1.54.2/go.mod h1:+RQ6wluLwtYaTd1WnPLykIDPekkuyD/ROWQClE83pzs=
github.com/moby/moby/client v0.4.1 h1:DMQgisVoMkmMs7fp3ROSdiBnoAu8+vo3GggFl06M/wY=
github.com/moby/moby/client v0.4.1/go.mod h1:z52C9O2POPOsnxZAy//WtKcQ32P+jT/NGeXu/7nfjGQ=
github.com/moby/sys/user v0.4.0 h1:jhcMKit7SA80hivmFJcbB1vqmw//wU61Zdui2eQXuMs=
github.com/moby/sys/user v0.4.0/go.mod h1:bG+tYYYJgaMtRKgEmuueC0hJEAZWwtIbZTB+85uoHjs=
github.com/moby/sys/user v0.4.1 h1:RgjRlaDKi/Xmyrz4t8lyzXT6v2ooFeO/7xtchmhVWE0=
github.com/moby/sys/user v0.4.1/go.mod h1:E9QsW5WRe1kUAf7kW8hXKwu1uhsZEAdPLYHYSDudF4Y=
github.com/moby/term v0.5.2 h1:6qk3FJAFDs6i/q3W/pQ97SX192qKfZgGjCQqfCJkgzQ=
github.com/moby/term v0.5.2/go.mod h1:d3djjFCrjnB+fl8NJux+EJzu0msscUP+f8it8hPkFLc=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
@@ -937,8 +937,8 @@ github.com/projectdiscovery/clistats v0.1.7 h1:NXf4lUzx1QcbJd6QycNwwYHTv2FsW9Mch
github.com/projectdiscovery/clistats v0.1.7/go.mod h1:RxRGxhMzkT344VTTvZWIrojurUXdQQRN1yfwgg/88I8=
github.com/projectdiscovery/dsl v0.8.24 h1:rN2w6LSIjA16qKGRjyj70oxdwXGaspe6sAjEkP7F2fk=
github.com/projectdiscovery/dsl v0.8.24/go.mod h1:V+EludvS5PS19UNMlM3CuBwZ+sFnMaQhCFGxysDUanQ=
github.com/projectdiscovery/fastdialer v0.5.26 h1:wCCsopLsO1EF1V1+9Wqkxj1bMm6sRo8NdlzwE2Ib2Ss=
github.com/projectdiscovery/fastdialer v0.5.26/go.mod h1:dZzEPnpsxqCiD3IkM5CN3RglW3l1gXJNyTkxqjPZIbs=
github.com/projectdiscovery/fastdialer v0.6.0 h1:4ZeFQ6LyRH4D8e3qO/yzqIP4Cf9Fr0DduaVpJD6ENnE=
github.com/projectdiscovery/fastdialer v0.6.0/go.mod h1:FpHwa9S5PUgfPbh+irOm+cdU6AO631XidxkhPwPv7yY=
github.com/projectdiscovery/fasttemplate v0.0.2 h1:h2cISk5xDhlJEinlBQS6RRx0vOlOirB2y3Yu4PJzpiA=
github.com/projectdiscovery/fasttemplate v0.0.2/go.mod h1:XYWWVMxnItd+r0GbjA1GCsUopMw1/XusuQxdyAIHMCw=
github.com/projectdiscovery/freeport v0.0.7 h1:Q6uXo/j8SaV/GlAHkEYQi8WQoPXyJWxyspx+aFmz9Qk=
@@ -977,8 +977,8 @@ github.com/projectdiscovery/mapcidr v1.1.97 h1:7FkxNNVXp+m1rIu5Nv/2SrF9k4+LwP8Qu
github.com/projectdiscovery/mapcidr v1.1.97/go.mod h1:9dgTJh1SP02gYZdpzMjm6vtYFkEHQHoTyaVNvaeJ7lA=
github.com/projectdiscovery/n3iwf v0.0.0-20230523120440-b8cd232ff1f5 h1:L/e8z8yw1pfT6bg35NiN7yd1XKtJap5Nk6lMwQ0RNi8=
github.com/projectdiscovery/n3iwf v0.0.0-20230523120440-b8cd232ff1f5/go.mod h1:pGW2ncnTxTxHtP9wzcIJAB+3/NMp6IiuQWd2NK7K+oc=
github.com/projectdiscovery/networkpolicy v0.1.55 h1:D30pipxOdWnSq3r7sX0kQvGyb18sUgopuda1j9NAmnQ=
github.com/projectdiscovery/networkpolicy v0.1.55/go.mod h1:N5bXm8L3UsDw51CeA1nhTsGEbQMbgL5lXTiIU013xd0=
github.com/projectdiscovery/networkpolicy v0.1.56 h1:M/L421HFuN707oX0tLTzKZY46uRyaKwxAeyi0q8PlxI=
github.com/projectdiscovery/networkpolicy v0.1.56/go.mod h1:N5bXm8L3UsDw51CeA1nhTsGEbQMbgL5lXTiIU013xd0=
github.com/projectdiscovery/ratelimit v0.0.94 h1:2qciSuUxarn73YGICZeprZtjrwyAH+0iVpdKzKCXoXE=
github.com/projectdiscovery/ratelimit v0.0.94/go.mod h1:GAerX6s+eWRuACedHRv2/xmMtlW+VahZmbRstQh0sS0=
github.com/projectdiscovery/rawhttp v0.2.0 h1:TGiHC5hydL97/zoi0WjEuNrfO6Gjtg9LeXndUf1Cq3M=
@@ -999,8 +999,8 @@ github.com/projectdiscovery/uncover v1.2.1 h1:8U46T/96CLT7BPoXBgkTvWqB06lOyeTSLv
github.com/projectdiscovery/uncover v1.2.1/go.mod h1:0p8onrWxfpXQEYs90ZDzTSpu1107gWmodX1NWqu/+z4=
github.com/projectdiscovery/useragent v0.0.109 h1:b86BPdZvmgbJousa04jAugDRY4Y+rAvEamE0TFCPmeE=
github.com/projectdiscovery/useragent v0.0.109/go.mod h1:sRyBqDAcD31RKYrO8fyXGowLd36fNwY1g0MNW9mp7zk=
github.com/projectdiscovery/utils v0.11.10 h1:6mtjWZifPT+UFXCeWSSzwb10KadrlkhYTm03jfhJ2EY=
github.com/projectdiscovery/utils v0.11.10/go.mod h1:T7KzyElivWJiPP+B4sXqkp1CcFe6O/P0qBn/MjDNa74=
github.com/projectdiscovery/utils v0.11.12-0.20261025172519-64b3c221cfe5 h1:AOspf7ScbwZr8A2YnjCFd4ZJAMSFqzfEaIJIbJPuu10=
github.com/projectdiscovery/utils v0.11.12-0.20261025172519-64b3c221cfe5/go.mod h1:TJgsyVjB02CavwZVFzUzPg6YOSJXQxLhL808JCiQKlU=
github.com/projectdiscovery/wappalyzergo v0.3.5 h1:UZradYF1GIcruSIV5wwo/Coph9/VnM17oajLD9eXR2U=
github.com/projectdiscovery/wappalyzergo v0.3.5/go.mod h1:4E6+fh/ckaY3wo5tlM7lNYSkhuTYRHzu2YmGIRraXP0=
github.com/projectdiscovery/yamldoc-go v1.0.7 h1:Odmjd1o4EkMff2F6QHi1dIYPIQPZUFy24YqmQCprmTk=
@@ -1284,20 +1284,20 @@ go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.6
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.61.0/go.mod h1:snMWehoOh2wsEwnvvwtDyFCxVeDAODenXHtn5vzrKjo=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.68.0 h1:CqXxU8VOmDefoh0+ztfGaymYbhdB/tT3zs79QaZTNGY=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.68.0/go.mod h1:BuhAPThV8PBHBvg8ZzZ/Ok3idOdhWIodywz2xEcRbJo=
go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU=
go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc=
go.opentelemetry.io/otel v1.45.0 h1:pdrWmLHofpubmArBv1LgFSv1Z0Ie/ppdZzu+kUN5EeU=
go.opentelemetry.io/otel v1.45.0/go.mod h1:XZxIqPapzEYnhNSScF5DIqXhm/rYi0FzCe2XddAwZfQ=
go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.36.0 h1:rixTyDGXFxRy1xzhKrotaHy3/KXdPhlWARrCgK+eqUY=
go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.36.0/go.mod h1:dowW6UsM9MKbJq5JTz2AMVp3/5iW5I/TStsk8S+CfHw=
go.opentelemetry.io/otel/metric v1.44.0 h1:1w0gILTcHdr3YI+ixLyjemwrVnsMURbTZFrSYCdDdmc=
go.opentelemetry.io/otel/metric v1.44.0/go.mod h1:8O7hanEPBNgEMmybD3s2VBKcgWOCsA6tzHBPODAiquo=
go.opentelemetry.io/otel/metric/x v0.66.0 h1:YkCrx1zLOChi9ZcZ6euupOcsgzbVlec7D/xoEU1+cTA=
go.opentelemetry.io/otel/metric/x v0.66.0/go.mod h1:d1+BDj9t96do0/1LoU1ayfCv79ZgNE41qbhBvnMOBZk=
go.opentelemetry.io/otel/sdk v1.44.0 h1:nHYwb9lK+fJPU/dnT6s7W7Z8itMWyqrnVfbheVYrZ58=
go.opentelemetry.io/otel/sdk v1.44.0/go.mod h1:Osuydd3Se74nqjAKxid74N5eC+jfEqfTegHRnq58oK0=
go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRks6si09iEfI=
go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA=
go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk=
go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE=
go.opentelemetry.io/otel/metric v1.45.0 h1:7Eg1uH7CJ5cXv9is6tnBe1FI6rj1nwUdbFypRm3br/M=
go.opentelemetry.io/otel/metric v1.45.0/go.mod h1:HAPbm1nd3p1PmFH7v2dR+6BjXxw+Lq4a2+pndMAm08s=
go.opentelemetry.io/otel/metric/x v0.67.0 h1:PcicCNZFkZ4bXfSooXdo3WN7RBOVOtjVdo1wD358Uns=
go.opentelemetry.io/otel/metric/x v0.67.0/go.mod h1:FBjCWZe6wgcqxcMtjdGiClDKXb2YxxXii0CXftE4QtI=
go.opentelemetry.io/otel/sdk v1.45.0 h1:4VVSMgQ83dUgW2aoX5f6JgLvHwIvzcuLnF9lUdCSpCw=
go.opentelemetry.io/otel/sdk v1.45.0/go.mod h1:Sr40LgXV7DsKMMJMKOhUWOgMWTfAaqvm2kF0g7ilwuA=
go.opentelemetry.io/otel/sdk/metric v1.45.0 h1:oVFszMfyj1Am6s24Vtc7wBb8BKLcwepJjNEYILuiE3o=
go.opentelemetry.io/otel/sdk/metric v1.45.0/go.mod h1:vUWUxDZvu1WVRj8JA8S0AdhsPrZoDpA2DdZauIh4mDA=
go.opentelemetry.io/otel/trace v1.45.0 h1:l/mP6Uv7oNO7/TblbhpbgMidxhq1uO/rPsikOyVhxag=
go.opentelemetry.io/otel/trace v1.45.0/go.mod h1:qoJJA2xNMnxRrdISU/kLtfUH2wNeQbiv+jhs/CxI8bc=
go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE=
go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
+7 -3
View File
@@ -9,6 +9,7 @@ import (
"github.com/pkg/errors"
"github.com/projectdiscovery/hmap/store/hybrid"
"github.com/projectdiscovery/httpx/common/httpx"
"github.com/projectdiscovery/nuclei/v3/pkg/input"
"github.com/projectdiscovery/nuclei/v3/pkg/input/provider"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
@@ -68,12 +69,15 @@ func (r *Runner) initializeTemplatesHTTPInput() (*hybrid.HybridMap, error) {
}
swg.Add()
go func(input *contextargs.MetaInput) {
go func(target *contextargs.MetaInput) {
defer swg.Done()
if result := utils.ProbeURL(input.Input, httpxClient); result != "" {
// a pinned target is probed through its ip, and each ip keeps its
// own result, as servers of one host may serve different schemes
ctx := contextargs.WithCustomIP(context.Background(), target)
if result := utils.ProbeURLWithContext(ctx, target.Input, httpxClient); result != "" {
count.Add(1)
_ = hm.Set(input.Input, []byte(result))
_ = hm.Set(input.ProbeKey(target.Input, target.CustomIP), []byte(result))
}
}(value)
return true
+24 -19
View File
@@ -39,25 +39,30 @@ func GetAuthTmplStore(opts *types.Options, catalog catalog.Catalog, execOpts *pr
}
tmpls = append(tmpls, data...)
}
opts.Templates = tmpls
opts.Workflows = nil
opts.RemoteTemplateDomainList = nil
opts.TemplateURLs = nil
opts.WorkflowURLs = nil
opts.ExcludedTemplates = nil
opts.Tags = nil
opts.ExcludeTags = nil
opts.IncludeTemplates = nil
opts.Authors = nil
opts.Severities = nil
opts.ExcludeSeverities = nil
opts.IncludeTags = nil
opts.IncludeIds = nil
opts.ExcludeIds = nil
opts.Protocols = nil
opts.ExcludeProtocols = nil
opts.IncludeConditions = nil
cfg := loader.NewConfig(opts, catalog, execOpts)
// The auth store only wants the secret-file templates. Copy first so
// this scoping cannot clear Templates, ExcludeTags, or the other filters
// on the caller's options. The CLI already copies; the SDK passes the
// engine options directly.
scoped := opts.Copy()
scoped.Templates = tmpls
scoped.Workflows = nil
scoped.RemoteTemplateDomainList = nil
scoped.TemplateURLs = nil
scoped.WorkflowURLs = nil
scoped.ExcludedTemplates = nil
scoped.Tags = nil
scoped.ExcludeTags = nil
scoped.IncludeTemplates = nil
scoped.Authors = nil
scoped.Severities = nil
scoped.ExcludeSeverities = nil
scoped.IncludeTags = nil
scoped.IncludeIds = nil
scoped.ExcludeIds = nil
scoped.Protocols = nil
scoped.ExcludeProtocols = nil
scoped.IncludeConditions = nil
cfg := loader.NewConfig(scoped, catalog, execOpts)
cfg.StoreId = loader.AuthStoreId
store, err := loader.New(cfg)
if err != nil {
+7 -1
View File
@@ -20,6 +20,7 @@ import (
"github.com/projectdiscovery/gologger/levels"
"github.com/projectdiscovery/nuclei/v3/internal/server/proxy"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/installer"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolinit"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/utils/vardump"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/headless/engine"
@@ -149,7 +150,7 @@ func ParseOptions(options *types.Options) {
}
}
// validateOptions validates the configuration options passed
// ValidateOptions validates the configuration options passed
func ValidateOptions(options *types.Options) error {
if err := validateOptions.Struct(options); err != nil {
if _, ok := err.(*validator.InvalidValidationError); ok {
@@ -175,6 +176,11 @@ func ValidateOptions(options *types.Options) error {
if options.ShouldFollowHTTPRedirects() && options.DisableRedirects {
return errors.New("both follow redirects and disable redirects specified")
}
switch options.UpdateTemplatesChannel {
case "", installer.TemplatesChannelStable, installer.TemplatesChannelLatest:
default:
return fmt.Errorf("invalid templates update channel %q, must be %q or %q", options.UpdateTemplatesChannel, installer.TemplatesChannelStable, installer.TemplatesChannelLatest)
}
// loading the proxy server list from file or cli and test the connectivity
if err := loadProxyServers(options); err != nil {
return err
+10
View File
@@ -5,6 +5,7 @@ import (
"testing"
"github.com/projectdiscovery/goflags"
"github.com/projectdiscovery/nuclei/v3/pkg/installer"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/stretchr/testify/require"
)
@@ -60,6 +61,15 @@ func TestParseHeadlessOptionalArguments(t *testing.T) {
}
}
// TestValidateOptionsUpdateTemplatesChannel checks that only known templates
// update channels are accepted.
func TestValidateOptionsUpdateTemplatesChannel(t *testing.T) {
for _, channel := range []string{"", installer.TemplatesChannelStable, installer.TemplatesChannelLatest} {
require.NoError(t, ValidateOptions(&types.Options{UpdateTemplatesChannel: channel}), "channel %q", channel)
}
require.ErrorContains(t, ValidateOptions(&types.Options{UpdateTemplatesChannel: "nightly"}), `invalid templates update channel "nightly"`)
}
func TestReadsTargetsFromStdin(t *testing.T) {
for _, tc := range []struct {
name string
+24 -4
View File
@@ -14,9 +14,25 @@ import (
)
const (
HTTP_PROXY_ENV = "HTTP_PROXY"
HTTP_PROXY_ENV = "HTTP_PROXY"
HTTPS_PROXY_ENV = "HTTPS_PROXY"
)
var proxyEnvVars = []string{
"HTTP_PROXY",
"HTTPS_PROXY",
"http_proxy",
"https_proxy",
"ALL_PROXY",
"all_proxy",
}
func setProxyEnv(proxyURL string) {
for _, env := range proxyEnvVars {
_ = os.Setenv(env, proxyURL)
}
}
// loadProxyServers load list of proxy servers from file or comma separated
func loadProxyServers(options *types.Options) error {
if len(options.Proxy) == 0 {
@@ -53,15 +69,19 @@ func loadProxyServers(options *types.Options) error {
return errkit.Wrapf(err, "failed to parse proxy got %v", err)
}
if options.ProxyInternal {
_ = os.Setenv(HTTP_PROXY_ENV, proxyURL.String())
setProxyEnv(proxyURL.String())
}
switch proxyURL.Scheme {
case proxyutils.HTTP, proxyutils.HTTPS:
options.Logger.Verbose().Msgf("Using %s as proxy server", proxyURL.String())
if options.Logger != nil {
options.Logger.Verbose().Msgf("Using %s as proxy server", proxyURL.String())
}
options.AliveHttpProxy = proxyURL.String()
case proxyutils.SOCKS5:
options.AliveSocksProxy = proxyURL.String()
options.Logger.Verbose().Msgf("Using %s as socket proxy server", proxyURL.String())
if options.Logger != nil {
options.Logger.Verbose().Msgf("Using %s as socket proxy server", proxyURL.String())
}
}
return nil
}
+81
View File
@@ -1,8 +1,14 @@
package runner
import (
"net/http"
"net/http/httptest"
"os"
"testing"
"github.com/projectdiscovery/goflags"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/stretchr/testify/require"
)
@@ -14,3 +20,78 @@ func TestValidateReplayProxy(t *testing.T) {
require.Error(t, validateReplayProxy(invalid), invalid)
}
}
type envState struct {
val string
set bool
}
func saveAndRestoreProxyEnv(t *testing.T) {
t.Helper()
original := make(map[string]envState, len(proxyEnvVars))
for _, env := range proxyEnvVars {
val, set := os.LookupEnv(env)
original[env] = envState{val: val, set: set}
_ = os.Unsetenv(env)
}
t.Cleanup(func() {
for env, state := range original {
if state.set {
_ = os.Setenv(env, state.val)
} else {
_ = os.Unsetenv(env)
}
}
})
}
func TestSetProxyEnv(t *testing.T) {
saveAndRestoreProxyEnv(t)
testProxy := "http://127.0.0.1:8080"
setProxyEnv(testProxy)
for _, env := range proxyEnvVars {
require.Equal(t, testProxy, os.Getenv(env), "expected %s to be set", env)
}
}
func TestLoadProxyServers_ProxyInternal(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
}))
defer server.Close()
t.Run("proxy internal enabled sets all env vars", func(t *testing.T) {
saveAndRestoreProxyEnv(t)
options := &types.Options{
Proxy: goflags.StringSlice{server.URL},
ProxyInternal: true,
Timeout: 5,
Logger: &gologger.Logger{},
}
err := loadProxyServers(options)
require.NoError(t, err)
for _, env := range proxyEnvVars {
require.Equal(t, server.URL, os.Getenv(env), "expected %s to be set", env)
}
})
t.Run("proxy internal disabled does not set env vars", func(t *testing.T) {
saveAndRestoreProxyEnv(t)
options := &types.Options{
Proxy: goflags.StringSlice{server.URL},
ProxyInternal: false,
Timeout: 5,
Logger: &gologger.Logger{},
}
err := loadProxyServers(options)
require.NoError(t, err)
for _, env := range proxyEnvVars {
require.Empty(t, os.Getenv(env), "expected %s not to be set", env)
}
})
}
+54 -11
View File
@@ -3,6 +3,8 @@ package runner
import (
"context"
"fmt"
"net"
"net/http"
"os"
"path/filepath"
"reflect"
@@ -53,6 +55,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/projectfile"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/automaticscan"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/baseline"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/globalmatchers"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/honeypotdetector"
@@ -86,7 +89,7 @@ var (
// Runner is a client for running the enumeration process.
type Runner struct {
output output.Writer
interactsh *interactsh.Client
interactsh interactsh.IClient
options *types.Options
projectFile *projectfile.ProjectFile
catalog catalog.Catalog
@@ -161,7 +164,9 @@ func New(options *types.Options) (*Runner, error) {
tm := &installer.TemplateManager{
CustomTemplates: ctm,
DisablePublicTemplates: options.PublicTemplateDisableDownload,
Channel: options.UpdateTemplatesChannel,
}
templatesVersion := config.DefaultConfig.TemplateVersion
if err := tm.FreshInstallIfNotExists(); err != nil {
runner.Logger.Warning().Msgf("Failed to install nuclei templates: %s\n", err)
}
@@ -178,7 +183,12 @@ func New(options *types.Options) (*Runner, error) {
if options.UpdateTemplates {
// we automatically check for updates unless explicitly disabled
// this print statement is only to inform the user that there are no updates
if !config.DefaultConfig.NeedsTemplateUpdate() {
noUpdates := !config.DefaultConfig.NeedsTemplateUpdate()
if tm.Channel == installer.TemplatesChannelLatest {
// the latest channel updates without a new release
noUpdates = config.DefaultConfig.TemplateVersion == templatesVersion
}
if noUpdates {
runner.Logger.Info().Msgf("No new updates found for nuclei templates")
}
@@ -223,7 +233,7 @@ func New(options *types.Options) (*Runner, error) {
runner.catalog = disk.NewCatalog(config.DefaultConfig.TemplatesDirectory)
var httpclient *retryablehttp.Client
if options.ProxyInternal && options.AliveHttpProxy != "" || options.AliveSocksProxy != "" {
if options.ProxyInternal && (options.AliveHttpProxy != "" || options.AliveSocksProxy != "") {
var err error
httpclient, err = httpclientpool.Get(options, &httpclientpool.Configuration{}, "")
@@ -422,13 +432,7 @@ func New(options *types.Options) (*Runner, error) {
opts.HTTPClient = httpclient
}
if opts.HTTPClient == nil {
httpOpts := retryablehttp.DefaultOptionsSingle
httpOpts.Timeout = 20 * time.Second // for stability reasons
if options.Timeout > 20 {
httpOpts.Timeout = time.Duration(options.Timeout) * time.Second
}
// in testing it was found most of times when interactsh failed, it was due to failure in registering /polling requests
opts.HTTPClient = retryablehttp.NewClient(retryablehttp.DefaultOptionsSingle)
opts.HTTPClient = newInteractshHTTPClient(options)
}
interactshClient, err := interactsh.New(opts)
@@ -692,6 +696,10 @@ func (r *Runner) RunEnumeration() error {
executorOpts.ExportReqURLPattern = true
}
if r.options.ConfidenceBaseline {
executorOpts.BaselineCache = baseline.New()
}
if len(r.options.SecretsFile) > 0 && !r.options.Validate {
// Clone options so GetAuthTmplStore can modify them without affecting the original
authOptions := r.options.Copy()
@@ -750,6 +758,11 @@ func (r *Runner) RunEnumeration() error {
}
loaderConfig.TargetFilter = profiles.LoadFilter()
}
if r.hasPinnedTargets() {
if err := pinnedTargetConflict(r.options); err != nil {
return err
}
}
if (!strings.EqualFold(r.options.InputFileMode, "list") || r.options.DAST) && !r.options.OfflineHTTP {
// if input type is not list (implicitly enable fuzzing), unless passive/offlinehttp
r.options.DAST = true
@@ -794,6 +807,11 @@ func (r *Runner) RunEnumeration() error {
if err := store.Load(); err != nil {
return err
}
// Each template is parsed once per scan, so the compiled cache only holds
// pre-compilation snapshots that nothing reads after loading.
if parser, ok := r.parser.(*templates.Parser); ok {
parser.PurgeCompiled()
}
if profiles != nil {
profiles.Prepare(slices.Concat(store.Templates(), store.Workflows()))
executorOpts.TargetScope = profiles
@@ -992,7 +1010,7 @@ func (r *Runner) executeSmartWorkflowInput(executorOpts *protocols.ExecutorOptio
if err != nil {
return nil, errors.Wrap(err, "could not create automatic scan service")
}
if err := service.Execute(); err != nil {
if err := service.Execute(context.Background()); err != nil {
return nil, errors.Wrap(err, "could not execute automatic scan")
}
result := &atomic.Bool{}
@@ -1235,3 +1253,28 @@ func init() {
HideAutoSaveMsg = env.GetEnvOrDefault("DISABLE_CLOUD_UPLOAD_WRN", false)
EnableCloudUpload = env.GetEnvOrDefault("ENABLE_CLOUD_UPLOAD", false)
}
func newInteractshHTTPClient(options *types.Options) *retryablehttp.Client {
httpOpts := retryablehttp.DefaultOptionsSingle
httpOpts.Timeout = 20 * time.Second // for stability reasons
if options.Timeout > 20 {
httpOpts.Timeout = time.Duration(options.Timeout) * time.Second
}
if dialers := protocolstate.GetDialersWithId(options.ExecutionId); dialers != nil && dialers.Fastdialer != nil {
// The pooled default keeps ProxyFromEnvironment and the TLS config that
// INTERACTSH_TLS_VERIFY edits. Nuclei only exports HTTP_PROXY when -pi
// is set, and that path uses the pooled client, so -proxy without -pi
// still dials the interactsh server directly.
transport := retryablehttp.DefaultReusePooledTransport()
transport.DialContext = dialers.Fastdialer.Dial
transport.DialTLSContext = func(ctx context.Context, network, addr string) (net.Conn, error) {
return dialers.Fastdialer.DialTLS(ctx, network, addr)
}
httpOpts.HttpClient = &http.Client{
Transport: transport,
Timeout: httpOpts.Timeout,
}
}
// in testing it was found most of times when interactsh failed, it was due to failure in registering /polling requests
return retryablehttp.NewClient(httpOpts)
}
+54
View File
@@ -1,13 +1,18 @@
package runner
import (
"net"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/require"
"github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
)
@@ -143,3 +148,52 @@ func TestWalkReflectStructHandlesNestedStructs(t *testing.T) {
require.Equal(t, "2", testStruct.Struct.B)
require.Equal(t, "true", testStruct.Struct.C)
}
func TestNewInteractshHTTPClient(t *testing.T) {
// ProxyFromEnvironment reads the environment once per process. Set it
// before the first request. NO_PROXY keeps the local dial direct.
for _, key := range []string{"HTTP_PROXY", "http_proxy", "ALL_PROXY", "all_proxy", "https_proxy", "no_proxy"} {
t.Setenv(key, "")
}
t.Setenv("HTTPS_PROXY", "http://127.0.0.1:9")
t.Setenv("NO_PROXY", "127.0.0.1,localhost")
options := &types.Options{
ExecutionId: "test-interactsh-client-dialer",
Timeout: 35,
}
require.NoError(t, protocolstate.Init(options))
t.Cleanup(func() { protocolstate.Close(options.ExecutionId) })
client := newInteractshHTTPClient(options)
require.NotNil(t, client)
require.Equal(t, 35*time.Second, client.HTTPClient.Timeout)
transport, ok := client.HTTPClient.Transport.(*http.Transport)
require.True(t, ok)
require.NotNil(t, transport.Proxy)
require.NotNil(t, transport.DialContext)
require.NotNil(t, transport.DialTLSContext)
require.NotNil(t, transport.TLSClientConfig, "INTERACTSH_TLS_VERIFY edits this config")
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusNoContent)
}))
t.Cleanup(srv.Close)
resp, err := client.HTTPClient.Get(srv.URL)
require.NoError(t, err)
require.Equal(t, http.StatusNoContent, resp.StatusCode)
require.NoError(t, resp.Body.Close())
host := srv.Listener.Addr().(*net.TCPAddr).IP.String()
dialers := protocolstate.GetDialersWithId(options.ExecutionId)
require.Equal(t, host, dialers.Fastdialer.GetDialedIP(host))
req, err := http.NewRequest(http.MethodGet, "https://example.com", nil)
require.NoError(t, err)
proxyURL, err := transport.Proxy(req)
require.NoError(t, err)
require.NotNil(t, proxyURL)
require.Equal(t, "http://127.0.0.1:9", proxyURL.String())
}
+24
View File
@@ -1,7 +1,9 @@
package runner
import (
"github.com/pkg/errors"
"github.com/projectdiscovery/nuclei/v3/pkg/input/targetprofile"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
)
// targetProfiles returns the per-target profiles read from input, or nil when
@@ -15,3 +17,25 @@ func (r *Runner) targetProfiles() *targetprofile.Registry {
}
return provider.TargetProfiles()
}
// hasPinnedTargets reports whether a target line pinned a target to an ip.
func (r *Runner) hasPinnedTargets() bool {
provider, ok := r.inputProvider.(interface {
PinnedTargets() bool
})
return ok && provider.PinnedTargets()
}
// pinnedTargetConflict returns an error for an option that cannot run with
// targets pinned to an ip.
func pinnedTargetConflict(options *types.Options) error {
switch {
case options.AutomaticScan:
return errors.New("targets pinned to an ip cannot be combined with automatic scan")
case options.AliveHttpProxy != "" || options.AliveSocksProxy != "":
return errors.New("targets pinned to an ip cannot be combined with a proxy, as the proxy resolves the host itself")
case options.ReplayProxy != "":
return errors.New("targets pinned to an ip cannot be combined with -replay-proxy, as the replay proxy resolves the host itself")
}
return nil
}
+27
View File
@@ -0,0 +1,27 @@
package runner
import (
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/stretchr/testify/require"
)
func TestPinnedTargetConflict(t *testing.T) {
tests := []struct {
name string
options *types.Options
wantErr bool
}{
{name: "no conflicting option", options: &types.Options{}},
{name: "automatic scan", options: &types.Options{AutomaticScan: true}, wantErr: true},
{name: "http proxy", options: &types.Options{AliveHttpProxy: "http://127.0.0.1:8080"}, wantErr: true},
{name: "socks proxy", options: &types.Options{AliveSocksProxy: "socks5://127.0.0.1:1080"}, wantErr: true},
{name: "replay proxy", options: &types.Options{ReplayProxy: "http://127.0.0.1:8080"}, wantErr: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
require.Equal(t, tt.wantErr, pinnedTargetConflict(tt.options) != nil)
})
}
}
+1 -1
View File
@@ -52,7 +52,7 @@ type NucleiExecutorOptions struct {
Catalog catalog.Catalog
IssuesClient reporting.Client
RateLimiter *ratelimit.Limiter
Interactsh *interactsh.Client
Interactsh interactsh.IClient
ProjectFile *projectfile.ProjectFile
Browser *browserEngine.Browser
FuzzStatsDB *stats.Tracker
+67
View File
@@ -6,6 +6,7 @@ package integration_test
import (
"errors"
"fmt"
"net"
"net/http"
"net/http/httptest"
"net/http/httputil"
@@ -83,6 +84,8 @@ var httpTestcases = []integrationCase{
{Path: "protocols/http/annotation-timeout.yaml", TestCase: &annotationTimeout{}},
{Path: "protocols/http/custom-attack-type.yaml", TestCase: &customAttackType{}},
{Path: "protocols/http/get-all-ips.yaml", TestCase: &scanAllIPS{}},
{Path: "protocols/http/target-line-ip.yaml", TestCase: &httpTargetLineIP{}},
{Path: "protocols/http/target-line-ip-pipeline.yaml", TestCase: &httpTargetLineIP{}},
{Path: "protocols/http/get-without-scheme.yaml", TestCase: &httpGetWithoutScheme{}},
{Path: "protocols/http/cl-body-without-header.yaml", TestCase: &httpCLBodyWithoutHeader{}},
{Path: "protocols/http/cl-body-with-header.yaml", TestCase: &httpCLBodyWithHeader{}},
@@ -1606,6 +1609,70 @@ func (h *scanAllIPS) Execute(filePath string) error {
return expectResultsCount(got, 1)
}
// target lines pin a hostname that does not resolve to the test servers
type httpTargetLineIP struct{}
// Execute executes a test case and returns an error if occurred
func (h *httpTargetLineIP) Execute(filePath string) error {
serve := func(addr, name string) (*httptest.Server, error) {
listener, err := net.Listen("tcp", addr)
if err != nil {
return nil, err
}
ts := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = fmt.Fprintf(w, "host=%s server=%s", r.Host, name)
}))
_ = ts.Listener.Close()
ts.Listener = listener
ts.Start()
return ts, nil
}
one, err := serve("127.0.0.1:0", "one")
if err != nil {
return err
}
defer one.Close()
_, port, err := net.SplitHostPort(one.Listener.Addr().String())
if err != nil {
return err
}
lines := []string{fmt.Sprintf(`{"target": "http://pinned.test:%s", "ip": "127.0.0.1"}`, port)}
want := []string{"one"}
// the same host pinned to a second ip, where 127.0.0.2 is available
if two, err := serve(net.JoinHostPort("127.0.0.2", port), "two"); err == nil {
defer two.Close()
lines = append(lines, fmt.Sprintf(`{"target": "http://pinned.test:%s", "ip": "127.0.0.2"}`, port))
want = append(want, "two")
}
targets, err := os.CreateTemp("", "target-lines-*.txt")
if err != nil {
return err
}
defer func() { _ = os.Remove(targets.Name()) }()
if _, err := targets.WriteString(strings.Join(lines, "\n")); err != nil {
return err
}
_ = targets.Close()
got, err := testutils.RunNucleiBareArgsAndGetResults(debug, nil, "-t", filePath, "-l", targets.Name())
if err != nil {
return err
}
if err := expectResultsCount(got, len(want)); err != nil {
return err
}
joined := strings.Join(got, "\n")
for _, name := range want {
if !strings.Contains(joined, name) {
return fmt.Errorf("no result from server %q: %v", name, got)
}
}
return nil
}
// ensure that ip|host are handled without http|https scheme
type httpGetWithoutScheme struct{}
@@ -5,10 +5,12 @@ package integration_test
import (
"database/sql"
"encoding/json"
"fmt"
"io"
"net"
"os"
"strings"
"time"
"github.com/go-pg/pg/v10"
@@ -16,6 +18,7 @@ import (
"github.com/ory/dockertest/v3"
dc "github.com/ory/dockertest/v3/docker"
"github.com/projectdiscovery/nuclei/v3/internal/tests/testutils"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
osutils "github.com/projectdiscovery/utils/os"
"github.com/projectdiscovery/utils/reader"
"go.uber.org/multierr"
@@ -34,6 +37,7 @@ var jsTestcases = []integrationCase{
{Path: "protocols/javascript/redis-lua-script.yaml", TestCase: &javascriptRedisLuaScript{}, DisableOn: javascriptDockerDisabled, Serial: true},
{Path: "protocols/javascript/ssh-server-fingerprint.yaml", TestCase: &javascriptSSHServerFingerprint{}, DisableOn: javascriptDockerDisabled, Serial: true},
{Path: "protocols/javascript/net-multi-step.yaml", TestCase: &networkMultiStep{}},
{Path: "protocols/javascript/target-line-ip.yaml", TestCase: &javascriptTargetLineIP{}},
{Path: "protocols/javascript/net-https.yaml", TestCase: &javascriptNetHttps{}},
{Path: "protocols/javascript/grpc-health.yaml", TestCase: &javascriptGRPCHealth{}},
{Path: "protocols/javascript/grpc-denied.yaml", TestCase: &javascriptGRPCDenied{}},
@@ -467,6 +471,87 @@ func (j *networkMultiStep) Execute(filePath string) error {
return expectResultsCount(results, 1)
}
// target lines pin a hostname that does not resolve, so the javascript
// connection reaches a server only through the pinned ip
type javascriptTargetLineIP struct{}
func (j *javascriptTargetLineIP) Execute(filePath string) error {
serve := func(addr, banner string) (net.Listener, error) {
listener, err := net.Listen("tcp", addr)
if err != nil {
return nil, err
}
go func() {
for {
conn, err := listener.Accept()
if err != nil {
return
}
_, _ = conn.Write([]byte(banner))
_ = conn.Close()
}
}()
return listener, nil
}
one, err := serve("127.0.0.1:0", "one")
if err != nil {
return err
}
defer func() { _ = one.Close() }()
_, port, err := net.SplitHostPort(one.Addr().String())
if err != nil {
return err
}
want := map[string]string{"127.0.0.1": "one"}
// the same host pinned to a second ip, where 127.0.0.2 is available
if two, err := serve(net.JoinHostPort("127.0.0.2", port), "two"); err == nil {
defer func() { _ = two.Close() }()
want["127.0.0.2"] = "two"
}
lines := make([]string, 0, len(want))
for ip := range want {
lines = append(lines, fmt.Sprintf(`{"target": "pinned.test:%s", "ip": %q}`, port, ip))
}
targets, err := os.CreateTemp("", "target-lines-*.txt")
if err != nil {
return err
}
defer func() { _ = os.Remove(targets.Name()) }()
if _, err := targets.WriteString(strings.Join(lines, "\n")); err != nil {
return err
}
_ = targets.Close()
if err := ensureSignedIntegrationTemplates(); err != nil {
return err
}
results, err := testutils.RunNucleiBareArgsAndGetResults(debug, getEnvValues(), "-t", filePath, "-l", targets.Name(), "-jsonl")
if err != nil {
return err
}
got := make(map[string]string)
for _, result := range results {
var event output.ResultEvent
if !strings.HasPrefix(result, "{") || json.Unmarshal([]byte(result), &event) != nil {
continue
}
got[event.IP] = strings.Join(event.ExtractedResults, ",")
}
if len(got) != len(want) {
return fmt.Errorf("got results for %d ips, want %d: %v", len(got), len(want), got)
}
for ip, banner := range want {
if got[ip] != banner {
return fmt.Errorf("ip %s answered %q, want %q", ip, got[ip], banner)
}
}
return nil
}
func runJavascriptDockerCase(filePath string, spec javascriptDockerSpec, expectedNumbers ...int) error {
if len(expectedNumbers) == 0 {
expectedNumbers = []int{1}
@@ -6,6 +6,8 @@ package integration_test
import (
"fmt"
"net"
"os"
"path/filepath"
"testing"
"time"
@@ -44,6 +46,38 @@ func TestNetwork(t *testing.T) {
}
})
t.Run("PinnedTargetLine", func(t *testing.T) {
server := testutils.NewTCPServer(nil, 0, func(conn net.Conn) {
defer func() { _ = conn.Close() }()
data, err := reader.ConnReadNWithTimeout(conn, 4, 5*time.Second)
if err == nil && string(data) == "PING" {
_, _ = conn.Write([]byte("PONG"))
}
})
defer server.Close()
_, port, err := net.SplitHostPort(server.URL)
if err != nil {
t.Fatal(err)
}
targets := filepath.Join(t.TempDir(), "targets.txt")
line := fmt.Sprintf(`{"target": "pinned.test:%s", "ip": "127.0.0.1"}`, port)
if err := os.WriteFile(targets, []byte(line), 0o600); err != nil {
t.Fatal(err)
}
// pinned.test does not resolve: the network request reaches the server
// only through the pinned ip
results, err := testutils.RunNucleiBareArgsAndGetResults(suite.debug, nil, "-t", "protocols/network/basic.yaml", "-l", targets)
if err != nil {
t.Fatalf("pinned network request failed: %v", err)
}
if err := expectResultsCount(results, 1); err != nil {
t.Fatal(err)
}
})
t.Run("Hex", func(t *testing.T) {
errState := &capturedError{}
server := testutils.NewTCPServer(nil, 0, func(conn net.Conn) {
@@ -0,0 +1,25 @@
id: target-line-ip-pipeline
info:
name: Target Line IP Pipeline
author: pdteam
severity: info
http:
- raw:
- |
GET / HTTP/1.1
Host: {{Hostname}}
pipeline: true
pipeline-concurrent-connections: 1
pipeline-requests-per-connection: 1
matchers:
- type: word
words:
- "host=pinned.test:"
extractors:
- type: regex
group: 1
regex:
- "server=([a-z]+)"
@@ -0,0 +1,20 @@
id: target-line-ip
info:
name: Target Line IP
author: pdteam
severity: info
http:
- method: GET
path:
- "{{BaseURL}}"
matchers:
- type: word
words:
- "host=pinned.test:"
extractors:
- type: regex
group: 1
regex:
- "server=([a-z]+)"
@@ -0,0 +1,25 @@
id: javascript-target-line-ip
info:
name: JavaScript Target Line IP
author: pdteam
severity: info
javascript:
- code: |
var m = require("nuclei/net");
var conn = m.Open("tcp", address);
conn.SetTimeout(timeout);
conn.RecvString(3)
args:
address: "{{Host}}:{{Port}}"
Host: "{{Host}}"
Port: 5431
timeout: 3
extractors:
- type: regex
part: response
regex:
- "[a-z]+"
+71
View File
@@ -0,0 +1,71 @@
package nuclei_test
import (
"context"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"sync/atomic"
"testing"
nuclei "github.com/projectdiscovery/nuclei/v3/lib"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/stretchr/testify/require"
)
func TestThreadSafeCachedFlowPreservesPrecondition(t *testing.T) {
for _, positive := range []bool{false, true} {
name := "negative"
if positive {
name = "positive"
}
t.Run(name, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "flow.yaml")
require.NoError(t, os.WriteFile(path, []byte(`id: cached-flow
info:
name: Cached flow
author: pdteam
severity: info
flow: http(1) && http(2)
http:
- method: GET
path: ["{{BaseURL}}/precondition"]
matchers:
- type: status
status: [200]
internal: true
- method: GET
path: ["{{BaseURL}}/check"]
matchers:
- type: status
status: [404]
`), 0o600))
var checks, findings atomic.Int64
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/check" {
checks.Add(1)
}
if positive && r.URL.Path == "/precondition" {
w.WriteHeader(200)
return
}
http.NotFound(w, r)
}))
defer server.Close()
engine, err := nuclei.NewThreadSafeNucleiEngineCtx(context.Background(), nuclei.DisableUpdateCheck())
require.NoError(t, err)
defer engine.Close()
engine.GlobalResultCallback(func(*output.ResultEvent) { findings.Add(1) })
for call := int64(1); call <= 3; call++ {
require.NoError(t, engine.ExecuteNucleiWithOptsCtx(context.Background(), []string{server.URL}, nuclei.WithTemplatesOrWorkflows(nuclei.TemplateSources{Templates: []string{path}})))
expected := int64(0)
if positive {
expected = call
}
require.Equal(t, expected, checks.Load(), "check requests after call %d", call)
require.Equal(t, expected, findings.Load(), "findings after call %d", call)
}
})
}
}
+3 -1
View File
@@ -367,7 +367,9 @@ func UseOutputWriter(writer OutputWriter) NucleiSDKOptions {
}
}
// StatsWriter
// StatsWriter receives scan statistics. A writer that also implements
// progress.TaskTracker is told how many template executions the scan runs
// and how many are done.
type StatsWriter progress.Progress
// UseStatsWriter allows setting a custom stats writer
+61 -1
View File
@@ -8,9 +8,15 @@ import (
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/stretchr/testify/require"
)
// Update checks are disabled explicitly so this pins corrupt-file rejection on
// its own. The ignore file is read after init's update block, which may replace
// a corrupt file via UpdateIgnoreFile — with update checks left on, this test
// would assert rejection or repair depending on whether the process could reach
// the network.
func TestNewNucleiEngineRejectsCorruptActiveIgnoreFile(t *testing.T) {
root := t.TempDir()
path := filepath.Join(root, config.NucleiIgnoreFileName)
@@ -21,8 +27,62 @@ func TestNewNucleiEngineRejectsCorruptActiveIgnoreFile(t *testing.T) {
t.Cleanup(func() { cfg.SetTemplatesDir(oldRoot) })
cfg.SetTemplatesDir(root)
engine, err := NewNucleiEngineCtx(context.Background())
engine, err := NewNucleiEngineCtx(context.Background(), DisableUpdateCheck())
require.Nil(t, engine)
require.ErrorContains(t, err, "error parsing")
require.ErrorContains(t, err, strconv.Quote(path))
}
// TestIgnoreFileSurvivesAuthTemplateStore pins two init orderings.
//
// The ignore file is read after the update block, so a file this same call
// creates is the one that becomes the deny-list. GetAuthTmplStore also runs
// during init whenever SecretsFile is set. It must scope a copy of the
// options: the caller's Templates and ExcludeTags stay put, and the ignore
// file is still applied on top.
//
// A static-only secrets file is enough to build that store.
func TestIgnoreFileSurvivesAuthTemplateStore(t *testing.T) {
root := t.TempDir()
require.NoError(t, os.WriteFile(
filepath.Join(root, config.NucleiIgnoreFileName),
[]byte("tags:\n - dos\nfiles:\n - dns/soa-detect.yaml\n"),
0o600,
))
cfg := config.DefaultConfig
oldRoot := cfg.TemplatesDirectory
t.Cleanup(func() { cfg.SetTemplatesDir(oldRoot) })
cfg.SetTemplatesDir(root)
secrets := filepath.Join(root, "secrets.yaml")
require.NoError(t, os.WriteFile(secrets, []byte(`static:
- type: header
domains:
- example.com
headers:
- key: x-api-key
value: not-a-real-secret
`), 0o600))
opts := types.DefaultOptions()
opts.SecretsFile = []string{secrets}
opts.Templates = []string{"http/kept.yaml"}
opts.ExcludeTags = []string{"custom"}
engine, err := NewNucleiEngineCtx(context.Background(),
WithOptions(opts),
DisableUpdateCheck(),
)
require.NoError(t, err)
t.Cleanup(engine.Close)
require.Equal(t, []string{"http/kept.yaml"}, []string(engine.Options().Templates),
"the auth store must not replace the caller's template list")
require.Contains(t, []string(engine.Options().ExcludeTags), "custom",
"the caller's own excluded tags must survive auth-store setup")
require.Contains(t, []string(engine.Options().ExcludeTags), "dos",
"the .nuclei-ignore deny-list must be applied after auth-store setup")
require.Contains(t, []string(engine.Options().ExcludedTemplates), "dns/soa-detect.yaml",
"the files section must be applied too")
}
+1 -5
View File
@@ -44,11 +44,7 @@ func createEphemeralObjects(ctx context.Context, base *NucleiEngine, opts *types
ResumeCfg: types.NewResumeCfg(),
Parser: base.parser,
Browser: base.browserInstance,
// Thread-safe executes can run concurrently with different Output writers.
// The compiled-template cache shallow-copies requests and mutates shared
// ExecutorOptions.Output via UpdateOptions/ApplyNewEngineOptions, which
// would otherwise route all findings to whichever call last won the race.
DoNotCache: true,
DoNotCache: opts.DoNotCacheTemplates,
}
if base.interactshClient != nil {
u.executerOpts.InteractshScope = base.interactshClient.NewRequestScope()
+88
View File
@@ -2,10 +2,14 @@ package nuclei
import (
"context"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"sync"
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/templates"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/stretchr/testify/require"
@@ -68,3 +72,87 @@ http:
require.NotNil(t, cached)
}
}
func TestThreadSafeExecuteUsesSharedCompiledCache(t *testing.T) {
templatePath := filepath.Join(t.TempDir(), "template.yaml")
require.NoError(t, os.WriteFile(templatePath, []byte(`id: thread-safe-shared-cache
info:
name: Thread safe shared cache
author: pdteam
severity: info
tags: thread-safe-shared-cache
http:
- method: GET
path:
- "{{BaseURL}}"
matchers:
- type: word
words:
- "thread-safe-shared-cache"
`), 0o600))
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte("thread-safe-shared-cache"))
}))
t.Cleanup(server.Close)
engine, err := NewThreadSafeNucleiEngineCtx(context.Background())
require.NoError(t, err)
t.Cleanup(engine.Close)
const executions = 4
var waitGroup sync.WaitGroup
errors := make([]error, executions)
for i := range executions {
waitGroup.Add(1)
go func() {
defer waitGroup.Done()
errors[i] = engine.ExecuteNucleiWithOptsCtx(context.Background(), []string{server.URL},
WithTemplatesOrWorkflows(TemplateSources{Templates: []string{templatePath}}),
WithTemplateFilters(TemplateFilters{Tags: []string{"thread-safe-shared-cache"}}),
WithResultCallback(func(*output.ResultEvent) {}),
)
}()
}
waitGroup.Wait()
for _, err := range errors {
require.NoError(t, err)
}
require.Equal(t, 1, engine.eng.GetParser().CompiledCount())
}
func TestThreadSafeExecuteHonorsDisableTemplateCache(t *testing.T) {
templatePath := filepath.Join(t.TempDir(), "template.yaml")
require.NoError(t, os.WriteFile(templatePath, []byte(`id: thread-safe-disable-cache
info:
name: Thread safe disable cache
author: pdteam
severity: info
tags: thread-safe-disable-cache
http:
- method: GET
path:
- "{{BaseURL}}"
`), 0o600))
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte("ok"))
}))
t.Cleanup(server.Close)
engine, err := NewThreadSafeNucleiEngineCtx(context.Background(), DisableTemplateCache())
require.NoError(t, err)
t.Cleanup(engine.Close)
err = engine.ExecuteNucleiWithOptsCtx(context.Background(), []string{server.URL},
WithTemplatesOrWorkflows(TemplateSources{Templates: []string{templatePath}}),
WithTemplateFilters(TemplateFilters{Tags: []string{"thread-safe-disable-cache"}}),
)
require.NoError(t, err)
require.Equal(t, 0, engine.eng.GetParser().CompiledCount())
}
+16 -4
View File
@@ -70,7 +70,7 @@ type NucleiEngine struct {
// unexported core fields
ctx context.Context
interactshClient *interactsh.Client
interactshClient interactsh.IClient
catalog catalog.Catalog
rateLimiter *ratelimit.Limiter
store *loader.Store
@@ -151,6 +151,9 @@ func (e *NucleiEngine) GetTemplates() []*templates.Template {
if !e.templatesLoaded {
_ = e.LoadAllTemplates()
}
if e.store == nil {
return nil
}
return e.store.Templates()
}
@@ -159,6 +162,9 @@ func (e *NucleiEngine) GetWorkflows() []*templates.Template {
if !e.templatesLoaded {
_ = e.LoadAllTemplates()
}
if e.store == nil {
return nil
}
return e.store.Workflows()
}
@@ -251,7 +257,7 @@ func (e *NucleiEngine) closeInternal() {
if e.hostErrCache != nil {
e.hostErrCache.Close()
}
if e.executerOpts.RateLimiter != nil {
if e.executerOpts != nil && e.executerOpts.RateLimiter != nil {
e.executerOpts.RateLimiter.Stop()
}
if e.rateLimiter != nil {
@@ -297,9 +303,11 @@ func (e *NucleiEngine) Close() {
// enable matcher-status option if you expect this callback to be called for all results regardless if it matched or not
func (e *NucleiEngine) ExecuteCallbackWithCtx(ctx context.Context, callback ...func(event *output.ResultEvent)) error {
if !e.templatesLoaded {
_ = e.LoadAllTemplates()
if err := e.LoadAllTemplates(); err != nil {
return err
}
}
if len(e.store.Templates()) == 0 && len(e.store.Workflows()) == 0 {
if e.store == nil || (len(e.store.Templates()) == 0 && len(e.store.Workflows()) == 0) {
return ErrNoTemplatesAvailable
}
if e.inputProvider.Count() == 0 {
@@ -378,6 +386,10 @@ func NewNucleiEngineCtx(ctx context.Context, options ...NucleiSDKOptions) (*Nucl
}
}
if err := e.init(ctx); err != nil {
// init may already own interactsh, a browser, a temp dir, and
// reporting. Close is safe on a partial engine: executerOpts is
// checked, and protocolinit.Close is a no-op before Init.
e.Close()
return nil, err
}
return e, nil
+34 -8
View File
@@ -44,6 +44,9 @@ import (
func (e *NucleiEngine) applyRequiredDefaults(ctx context.Context) {
mockoutput := testutils.NewMockOutputWriter(e.opts.OmitTemplate)
mockoutput.WriteCallback = func(event *output.ResultEvent) {
if (event.Error != "" || !event.MatcherStatus) && !e.opts.MatcherStatus {
return
}
if len(e.resultCallbacks) > 0 {
for _, callback := range e.resultCallbacks {
if callback != nil {
@@ -184,10 +187,6 @@ func (e *NucleiEngine) init(ctx context.Context) error {
return err
}
if err := e.loadIgnoreFile(); err != nil {
return err
}
if e.opts.Parser != nil {
if op, ok := e.opts.Parser.(*templates.Parser); ok {
e.parser = op
@@ -211,7 +210,7 @@ func (e *NucleiEngine) init(ctx context.Context) error {
_ = protocolinit.Init(e.opts)
}
if e.opts.ProxyInternal && e.opts.AliveHttpProxy != "" || e.opts.AliveSocksProxy != "" {
if e.opts.ProxyInternal && (e.opts.AliveHttpProxy != "" || e.opts.AliveSocksProxy != "") {
httpclient, err := httpclientpool.Get(e.opts, &httpclientpool.Configuration{}, "")
if err != nil {
return err
@@ -382,6 +381,26 @@ func (e *NucleiEngine) init(ctx context.Context) error {
}
}
// Read .nuclei-ignore LAST, once init has finished managing that file and
// the options it writes into.
//
// It must come after the update block above: init creates or replaces the
// ignore file there (UpdateIgnoreFile), so reading earlier reads a file this
// same call is about to write. On a host without a pre-existing ignore file
// — a fresh install, or any container whose filesystem was reset — the read
// found nothing, warned, and left the engine with an empty deny-list for its
// whole lifetime, because ExcludeTags is never re-read after init. Reading
// afterwards also makes the corrupt-file check recoverable rather than fatal:
// a malformed local file that IgnoreFileNeedsUpdate replaces no longer fails
// the scan.
//
// GetAuthTmplStore scopes a copy of the options, so it no longer clears
// the engine filters. This read is still what installs the deny-list, and
// it has to follow the update block above.
if err := e.loadIgnoreFile(); err != nil {
return err
}
return nil
}
@@ -389,7 +408,10 @@ type syncOnce struct {
sync.Once
}
var updateCheckInstance = &syncOnce{}
var (
updateCheckInstance = &syncOnce{}
templateUpdateInstance = &syncOnce{}
)
// processUpdateCheckResults processes update check results
func (e *NucleiEngine) processUpdateCheckResults() error {
@@ -398,8 +420,12 @@ func (e *NucleiEngine) processUpdateCheckResults() error {
if e.onUpdateAvailableCallback != nil {
e.onUpdateAvailableCallback(config.DefaultConfig.LatestNucleiTemplatesVersion)
}
tm := installer.TemplateManager{}
err = tm.UpdateIfOutdated()
})
if !e.disableTemplatesAutoUpgrade {
templateUpdateInstance.Do(func() {
tm := installer.TemplateManager{}
err = tm.UpdateIfOutdated()
})
}
return err
}
+172
View File
@@ -2,13 +2,20 @@ package nuclei_test
import (
"context"
"fmt"
"log"
"net"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"sync"
"testing"
"time"
nuclei "github.com/projectdiscovery/nuclei/v3/lib"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/stretchr/testify/require"
)
@@ -155,3 +162,168 @@ func TestIgnoreFileFilesSectionIsApplied(t *testing.T) {
require.Contains(t, ne.Options().ExcludedTemplates, "http/fuzzing/wordpress-themes-detect.yaml",
"files section should reach ExcludedTemplates")
}
func TestSDKCustomPortTemplateDoesNotPoisonBaseHost(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = fmt.Fprint(w, "server-ok")
}))
defer ts.Close()
// Allocate and immediately close a TCP listener to obtain a guaranteed closed local port.
ln, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
closedPort := ln.Addr().(*net.TCPAddr).Port
require.NoError(t, ln.Close())
tplDir := t.TempDir()
closedPortTpl := fmt.Sprintf(`id: closed-port-parallel-probe
info:
name: Closed Port Parallel Probe
author: test
severity: info
http:
- method: GET
path:
- "http://{{Hostname}}:%d/probe?p={{val}}"
threads: 2
payloads:
val:
- "1"
- "2"
matchers:
- type: word
words:
- "never-matches"
`, closedPort)
healthyTpl := `id: healthy-base-target-match
info:
name: Healthy Base Target Match
author: test
severity: high
http:
- method: GET
path:
- "{{BaseURL}}/"
matchers:
- type: word
words:
- "server-ok"
`
closedPortPath := filepath.Join(tplDir, "01-closed-port.yaml")
healthyPath := filepath.Join(tplDir, "02-healthy.yaml")
require.NoError(t, os.WriteFile(closedPortPath, []byte(closedPortTpl), 0o600))
require.NoError(t, os.WriteFile(healthyPath, []byte(healthyTpl), 0o600))
for run := 1; run <= 3; run++ {
ne, err := nuclei.NewNucleiEngineCtx(
context.Background(),
nuclei.DisableUpdateCheck(),
nuclei.WithTemplatesOrWorkflows(nuclei.TemplateSources{
Templates: []string{closedPortPath, healthyPath},
}),
nuclei.WithConcurrency(nuclei.Concurrency{
TemplateConcurrency: 1,
HostConcurrency: 1,
HeadlessHostConcurrency: 1,
HeadlessTemplateConcurrency: 1,
JavascriptTemplateConcurrency: 1,
TemplatePayloadConcurrency: 2,
ProbeConcurrency: 1,
}),
)
require.NoError(t, err)
ne.Options().RestrictLocalNetworkAccess = false
ne.Options().Timeout = 2
ne.Options().Retries = 0
ne.LoadTargets([]string{ts.URL}, false)
var (
mu sync.Mutex
matched []string
violations []*output.ResultEvent
)
err = ne.ExecuteCallbackWithCtx(context.Background(), func(event *output.ResultEvent) {
mu.Lock()
defer mu.Unlock()
if !event.MatcherStatus || event.Error != "" {
violations = append(violations, event)
}
matched = append(matched, event.TemplateID)
})
ne.Close()
require.NoError(t, err, "run %d failed", run)
require.Empty(t, violations, "ExecuteCallbackWithCtx must not emit non-matching or skip events when MatcherStatus is false")
require.Equal(t, []string{"healthy-base-target-match"}, matched, "run %d should consistently match healthy base target after custom-port failure", run)
}
}
func TestSDKSkipEventsRespectMatcherStatus(t *testing.T) {
ln, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
closedAddr := ln.Addr().String()
require.NoError(t, ln.Close())
tplDir := t.TempDir()
for i := 1; i <= 3; i++ {
tpl := fmt.Sprintf(`id: unreachable-check-%d
info:
name: Unreachable Check %d
author: test
severity: info
http:
- method: GET
path:
- "{{BaseURL}}/path-%d"
matchers:
- type: word
words:
- "ok"
`, i, i, i)
require.NoError(t, os.WriteFile(filepath.Join(tplDir, fmt.Sprintf("0%d-tpl.yaml", i)), []byte(tpl), 0o600))
}
runScan := func(matcherStatus bool) []*output.ResultEvent {
ne, err := nuclei.NewNucleiEngineCtx(
context.Background(),
nuclei.DisableUpdateCheck(),
nuclei.WithTemplatesOrWorkflows(nuclei.TemplateSources{
Templates: []string{tplDir},
}),
nuclei.WithConcurrency(nuclei.Concurrency{
TemplateConcurrency: 1,
HostConcurrency: 1,
HeadlessHostConcurrency: 1,
HeadlessTemplateConcurrency: 1,
JavascriptTemplateConcurrency: 1,
TemplatePayloadConcurrency: 1,
ProbeConcurrency: 1,
}),
)
require.NoError(t, err)
ne.Options().RestrictLocalNetworkAccess = false
ne.Options().Timeout = 1
ne.Options().Retries = 0
ne.Options().MatcherStatus = matcherStatus
ne.LoadTargets([]string{"http://" + closedAddr}, false)
var (
mu sync.Mutex
events []*output.ResultEvent
)
err = ne.ExecuteCallbackWithCtx(context.Background(), func(event *output.ResultEvent) {
mu.Lock()
defer mu.Unlock()
events = append(events, event)
})
ne.Close()
require.NoError(t, err)
return events
}
require.Empty(t, runScan(false), "skip/error events must not be emitted when MatcherStatus is false")
require.NotEmpty(t, runScan(true), "skip/error events should be emitted when MatcherStatus is true")
}
+9 -1
View File
@@ -5,6 +5,7 @@ import (
"io"
"os"
"path/filepath"
"slices"
"strings"
"github.com/projectdiscovery/nuclei/v3/pkg/templates/extensions"
@@ -32,6 +33,13 @@ func GetKnownConfigFiles() []string {
return knownConfigFiles
}
// IsKnownConfigFile returns true if the base name of fpath is exactly one of
// the known config files. Names that only contain one of them, such as
// "cves.json.yaml", are not considered config files.
func IsKnownConfigFile(fpath string) bool {
return slices.Contains(knownConfigFiles, filepath.Base(filepath.FromSlash(fpath)))
}
// GetKnownMiscDirectories returns known misc directories with trailing slashes.
//
// The trailing slash ensures that directory matching is explicit and avoids
@@ -78,7 +86,7 @@ func IsTemplateWithRoot(fpath, rootDir string) bool {
fname := filepath.Base(fpath)
fext := strings.ToLower(filepath.Ext(fpath))
if stringsutil.ContainsAny(fname, GetKnownConfigFiles()...) {
if IsKnownConfigFile(fname) {
return false
}
+45
View File
@@ -9,6 +9,8 @@ import (
"github.com/stretchr/testify/require"
)
// toAbs converts a slash separated absolute path into an OS specific absolute
// path, prefixing it with the system volume on Windows.
func toAbs(p string) string {
if osutils.IsWindows() {
// Infer the drive letter from the Windows system path
@@ -28,6 +30,8 @@ func toAbs(p string) string {
return filepath.FromSlash(p)
}
// TestIsTemplate checks template detection for supported extensions, excluded
// directories and known config files.
func TestIsTemplate(t *testing.T) {
tests := []struct {
name string
@@ -89,6 +93,24 @@ func TestIsTemplate(t *testing.T) {
rootDir: "",
want: false,
},
{
name: "excluded config file in subdirectory",
fpath: "nuclei-templates/TEMPLATES-STATS.json",
rootDir: "",
want: false,
},
{
name: "template name containing config file name",
fpath: "http/cves.json.yaml",
rootDir: "",
want: true,
},
{
name: "json template name containing config file name",
fpath: "http/contributors.json.json",
rootDir: "",
want: true,
},
}
for _, tt := range tests {
@@ -99,6 +121,29 @@ func TestIsTemplate(t *testing.T) {
}
}
// TestIsKnownConfigFile ensures only exact config file base names match, and
// names that merely contain one (e.g. cves.json.yaml) do not.
func TestIsKnownConfigFile(t *testing.T) {
for _, fpath := range []string{
"cves.json",
"contributors.json",
"TEMPLATES-STATS.json",
toAbs("/home/user/nuclei-templates/cves.json"),
"https://example.com/nuclei-templates/cves.json",
} {
require.True(t, IsKnownConfigFile(fpath), "IsKnownConfigFile(%q)", fpath)
}
for _, fpath := range []string{
"cves.json.yaml",
"http/cves.json.yaml",
"my-contributors.json",
toAbs("/home/user/cves.json/test.yaml"),
} {
require.False(t, IsKnownConfigFile(fpath), "IsKnownConfigFile(%q)", fpath)
}
}
func TestGetNucleiTemplatesIndexIncludesCustomDirSiblingPrefix(t *testing.T) {
tmpDir := t.TempDir()
cfgDir := t.TempDir()
+2 -6
View File
@@ -26,9 +26,7 @@ func (c *DiskCatalog) GetTemplatesPath(definitions []string) ([]string, map[stri
erred := make(map[string]error)
for _, t := range definitions {
if stringsutil.ContainsAny(t, knownConfigFiles...) {
// TODO: this is a temporary fix to avoid treating these files as templates
// this should be replaced with more appropriate and robust logic
if config.IsKnownConfigFile(t) {
continue
}
if strings.Contains(t, urlutil.SchemeSeparator) && stringsutil.ContainsAny(t, config.GetSupportTemplateFileExtensions()...) {
@@ -52,9 +50,7 @@ func (c *DiskCatalog) GetTemplatesPath(definitions []string) ([]string, map[stri
// purge all false positives
filteredTemplates := []string{}
for _, v := range allTemplates {
// TODO: this is a temporary fix to avoid treating these files as templates
// this should be replaced with more appropriate and robust logic
if !stringsutil.ContainsAny(v, knownConfigFiles...) {
if !config.IsKnownConfigFile(v) {
filteredTemplates = append(filteredTemplates, v)
}
}
+43 -1
View File
@@ -1,9 +1,10 @@
package disk
import (
"testing/fstest"
"os"
"path/filepath"
"testing"
"testing/fstest"
"github.com/stretchr/testify/require"
)
@@ -42,6 +43,8 @@ func TestFindGlobPathMatchesUsesCanonicalContainment(t *testing.T) {
}
}
// TestFindGlobPathMatchesResolvesContainedPath ensures a glob inside the
// templates directory resolves against the embedded FS.
func TestFindGlobPathMatchesResolvesContainedPath(t *testing.T) {
memFS := fstest.MapFS{
"http/test.yaml": {Data: []byte("legit")},
@@ -56,3 +59,42 @@ func TestFindGlobPathMatchesResolvesContainedPath(t *testing.T) {
require.NoError(t, err)
require.Equal(t, []string{"http/test.yaml"}, matches)
}
// TestGetTemplatesPathKnownConfigFiles ensures that only exact known config
// file names are skipped, and templates whose names merely contain one (e.g.
// cves.json.yaml) are still loaded.
func TestGetTemplatesPathKnownConfigFiles(t *testing.T) {
templatesDir := t.TempDir()
httpDir := filepath.Join(templatesDir, "http")
require.NoError(t, os.MkdirAll(httpDir, 0755))
template := filepath.Join(httpDir, "cves.json.yaml")
for _, f := range []string{
template,
filepath.Join(templatesDir, "cves.json"),
filepath.Join(templatesDir, "contributors.json"),
filepath.Join(templatesDir, "TEMPLATES-STATS.json"),
} {
require.NoError(t, os.WriteFile(f, []byte("id: test"), 0644))
}
cat := NewCatalog(templatesDir)
t.Run("direct path", func(t *testing.T) {
paths, errs := cat.GetTemplatesPath([]string{template})
require.Empty(t, errs)
require.Equal(t, []string{template}, paths)
})
t.Run("directory traversal", func(t *testing.T) {
paths, errs := cat.GetTemplatesPath([]string{templatesDir})
require.Empty(t, errs)
require.Equal(t, []string{template}, paths)
})
t.Run("config file is skipped", func(t *testing.T) {
paths, errs := cat.GetTemplatesPath([]string{filepath.Join(templatesDir, "cves.json")})
require.Empty(t, errs)
require.Empty(t, paths)
})
}
-3
View File
@@ -1,3 +0,0 @@
package disk
var knownConfigFiles = []string{"cves.json", "contributors.json", "TEMPLATES-STATS.json"}
+46 -23
View File
@@ -63,20 +63,6 @@ func (e *Engine) ExecuteScanWithOpts(ctx context.Context, templatesList []*templ
return &atomic.Bool{}
}
if e.executerOpts.Progress != nil {
// Notes:
// workflow requests are not counted as they can be conditional
// templateList count is user requested templates count (before clustering)
// totalReqAfterClustering is total requests count after clustering
e.executerOpts.Progress.Init(target.Count(), len(templatesList), e.expectedRequests(finalTemplates, target, totalReqAfterClustering))
}
if stringsutil.EqualFoldAny(e.options.ScanStrategy, scanstrategy.Auto.String(), "") {
// TODO: this is only a placeholder, auto scan strategy should choose scan strategy
// based on no of hosts , templates , stream and other optimization parameters
e.options.ScanStrategy = scanstrategy.TemplateSpray.String()
}
filtered := []*templates.Template{}
selfContained := []*templates.Template{}
// Filter Self Contained templates since they are not bound to target
@@ -88,6 +74,23 @@ func (e *Engine) ExecuteScanWithOpts(ctx context.Context, templatesList []*templ
}
}
if e.executerOpts.Progress != nil {
// Notes:
// workflow requests are not counted as they can be conditional
// templateList count is user requested templates count (before clustering)
// totalReqAfterClustering is total requests count after clustering
requests, tasks := e.expectedWork(finalTemplates, filtered, target, totalReqAfterClustering)
e.executerOpts.Progress.Init(target.Count(), len(templatesList), requests)
// a self-contained template runs once
e.addTaskTotal(tasks + int64(len(selfContained)))
}
if stringsutil.EqualFoldAny(e.options.ScanStrategy, scanstrategy.Auto.String(), "") {
// TODO: this is only a placeholder, auto scan strategy should choose scan strategy
// based on no of hosts , templates , stream and other optimization parameters
e.options.ScanStrategy = scanstrategy.TemplateSpray.String()
}
// Execute All SelfContained in parallel
e.executeAllSelfContained(ctx, selfContained, results, selfcontainedWg)
@@ -123,7 +126,7 @@ func (e *Engine) executeTemplateSpray(ctx context.Context, templatesList []*temp
}
// resize check point - nop if there are no changes
wp.RefreshWithConfig(e.GetWorkPoolConfig())
wp.RefreshWithConfigContext(ctx, e.GetWorkPoolConfig())
templateType := template.Type()
var wg *syncutil.AdaptiveWaitGroup
@@ -170,7 +173,10 @@ func (e *Engine) executeHostSpray(ctx context.Context, templatesList []*template
default:
}
wp.Add()
if err := wp.AddWithContext(ctx); err != nil {
return false
}
go func(targetval *contextargs.MetaInput) {
defer wp.Done()
e.executeTemplatesOnTarget(ctx, templatesList, targetval, results)
@@ -180,23 +186,40 @@ func (e *Engine) executeHostSpray(ctx context.Context, templatesList []*template
return results
}
// expectedRequests counts only the template and target pairs that per-target
// profiles select, so progress reflects the requests that will be sent.
func (e *Engine) expectedRequests(templatesList []*templates.Template, target provider.InputProvider, unscoped int) int64 {
// expectedWork returns the requests and the tasks the target-bound templates
// will run. It counts only the template and target pairs that per-target
// profiles select, so progress reflects the work that will be done. A task is
// one template, cluster or workflow run on one target.
func (e *Engine) expectedWork(templatesList, targetBound []*templates.Template, target provider.InputProvider, unscopedRequests int) (requests, tasks int64) {
scope := e.executerOpts.TargetScope
if scope == nil {
return int64(unscoped)
return int64(unscopedRequests), target.Count() * int64(len(targetBound))
}
targets := make(map[protocols.TemplateSelection]int64)
target.Iterate(func(value *contextargs.MetaInput) bool {
targets[scope.For(value)]++
return true
})
var total int64
for selection, count := range targets {
total += count * int64(getSelectedRequestCount(templatesList, selection))
requests += count * int64(getSelectedRequestCount(templatesList, selection))
tasks += count * int64(getSelectedCount(targetBound, selection))
}
return total
return requests, tasks
}
// getSelectedCount returns the number of templates selection selects; a nil
// selection selects every template.
func getSelectedCount(templates []*templates.Template, selection protocols.TemplateSelection) int {
if selection == nil {
return len(templates)
}
count := 0
for _, template := range templates {
if template.SelectedBy(selection) {
count++
}
}
return count
}
// returns total requests count
+56 -12
View File
@@ -9,6 +9,8 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/input/normalize"
"github.com/projectdiscovery/nuclei/v3/pkg/input/provider"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/progress"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v3/pkg/scan"
"github.com/projectdiscovery/nuclei/v3/pkg/templates"
@@ -36,6 +38,7 @@ func (e *Engine) executeAllSelfContained(ctx context.Context, alltemplates []*te
}
finished := e.templateExecutionStarted(template, "")
defer func() { finished(ctx.Err()) }()
defer e.tasksDone(1)
var err error
var match bool
ctx := scan.NewScanContext(ctx, contextargs.New(ctx))
@@ -166,12 +169,18 @@ func (e *Engine) executeTemplateWithTargets(ctx context.Context, template *templ
currentInfo.Unlock()
// Skip if the host has had errors
if inScope && e.executerOpts.HostErrorsCache != nil && e.executerOpts.HostErrorsCache.Check(e.executerOpts.ProtocolType.String(), contextargs.NewWithMetaInput(ctx, scannedValue)) {
hostDown := inScope && e.executerOpts.HostErrorsCache != nil && e.executerOpts.HostErrorsCache.Check(template.Type().String(), contextargs.NewWithMetaInput(ctx, scannedValue))
// in-scope pairs that never reach executeTemplateOnInput are done here:
// those done by the resumed scan and those of an unresponsive host
if inScope && (skip || hostDown) {
e.tasksDone(1)
}
if hostDown {
skipEvent := &output.ResultEvent{
TemplateID: template.ID,
TemplatePath: template.Path,
Info: template.Info,
Type: e.executerOpts.ProtocolType.String(),
Type: template.Type().String(),
Host: scannedValue.Input,
MatcherStatus: false,
Error: "host was skipped as it was found unresponsive",
@@ -210,7 +219,7 @@ func (e *Engine) executeTemplatesOnTarget(ctx context.Context, alltemplates []*t
wp := e.GetWorkPool()
defer wp.Wait()
for _, tpl := range alltemplates {
for i, tpl := range alltemplates {
select {
case <-ctx.Done():
return
@@ -224,12 +233,12 @@ func (e *Engine) executeTemplatesOnTarget(ctx context.Context, alltemplates []*t
// Check whether the target has already been marked as permanently
// unresponsive by HostErrorsCache before spawning another goroutine.
if e.executerOpts.HostErrorsCache != nil &&
e.executerOpts.HostErrorsCache.Check(e.executerOpts.ProtocolType.String(), contextargs.NewWithMetaInput(ctx, target)) {
e.executerOpts.HostErrorsCache.Check(tpl.Type().String(), contextargs.NewWithMetaInput(ctx, target)) {
skipEvent := &output.ResultEvent{
TemplateID: tpl.ID,
TemplatePath: tpl.Path,
Info: tpl.Info,
Type: e.executerOpts.ProtocolType.String(),
Type: tpl.Type().String(),
Host: target.Input,
MatcherStatus: false,
Error: "host was skipped as it was found unresponsive",
@@ -240,11 +249,13 @@ func (e *Engine) executeTemplatesOnTarget(ctx context.Context, alltemplates []*t
} else if e.executerOpts.Output != nil {
_ = e.executerOpts.Output.Write(skipEvent)
}
// the remaining templates are skipped for this target as well
e.tasksDone(int64(getSelectedCount(alltemplates[i:], e.selectionFor(target))))
break
}
// resize check point - nop if there are no changes
wp.RefreshWithConfig(e.GetWorkPoolConfig())
wp.RefreshWithConfigContext(ctx, e.GetWorkPoolConfig())
var sg *syncutil.AdaptiveWaitGroup
if tpl.Type() == types.HeadlessProtocol {
@@ -279,15 +290,46 @@ func (e *Engine) executeTemplatesOnTarget(ctx context.Context, alltemplates []*t
// inScope reports whether per-target profiles select template for input.
func (e *Engine) inScope(template *templates.Template, input *contextargs.MetaInput) bool {
if e.executerOpts.TargetScope == nil {
return true
}
selection := e.executerOpts.TargetScope.For(input)
selection := e.selectionFor(input)
return selection == nil || template.SelectedBy(selection)
}
// selectionFor returns the templates per-target profiles select for input; nil
// selects every template.
func (e *Engine) selectionFor(input *contextargs.MetaInput) protocols.TemplateSelection {
if e.executerOpts.TargetScope == nil {
return nil
}
return e.executerOpts.TargetScope.For(input)
}
// taskTracker returns the progress when it counts tasks, nil otherwise.
func (e *Engine) taskTracker() progress.TaskTracker {
if e.executerOpts == nil {
return nil
}
tracker, _ := e.executerOpts.Progress.(progress.TaskTracker)
return tracker
}
// addTaskTotal adds count tasks to the total when the progress counts tasks.
func (e *Engine) addTaskTotal(count int64) {
if tracker := e.taskTracker(); tracker != nil {
tracker.AddToTaskTotal(count)
}
}
// tasksDone marks count tasks as done when the progress counts tasks.
func (e *Engine) tasksDone(count int64) {
if tracker := e.taskTracker(); tracker != nil {
tracker.IncrementTasksBy(count)
}
}
// executeTemplateOnInput performs template execution for a single input and returns match status and error
func (e *Engine) executeTemplateOnInput(ctx context.Context, template *templates.Template, value *contextargs.MetaInput) (matched bool, err error) {
// a pair skipped as a repeat of its origin is done as well
defer e.tasksDone(1)
// A template fixed to the target's origin makes the same requests for every
// target that shares one, which a crawled list produces by the hundred.
// Running it once per origin sends exactly the same traffic, minus the
@@ -300,9 +342,11 @@ func (e *Engine) executeTemplateOnInput(ctx context.Context, template *templates
defer func() { finish(err == nil) }()
finished := e.templateExecutionStarted(template, value.Input)
defer func() { finished(ctx.Err()) }()
ctxArgs := contextargs.New(ctx)
// every protocol dials the target's host through its pinned ip, if any
pinnedCtx := contextargs.WithCustomIP(ctx, value)
ctxArgs := contextargs.New(pinnedCtx)
ctxArgs.MetaInput = value
scanCtx := scan.NewScanContext(ctx, ctxArgs)
scanCtx := scan.NewScanContext(pinnedCtx, ctxArgs)
switch template.Type() {
case types.WorkflowProtocol:
+111
View File
@@ -10,6 +10,7 @@ import (
"testing"
"time"
"github.com/projectdiscovery/gologger"
inputtypes "github.com/projectdiscovery/nuclei/v3/pkg/input/types"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols"
@@ -655,3 +656,113 @@ func Test_executeTemplateWithTargets_RespectsCancellation(t *testing.T) {
var matched atomic.Bool
e.executeTemplateWithTargets(ctx, tpl, targets, &matched)
}
func newCancelTestEngine() *Engine {
opts := &types.Options{
TemplateThreads: 1,
HeadlessTemplateThreads: 1,
BulkSize: 1,
HeadlessBulkSize: 1,
Logger: gologger.DefaultLogger,
}
e := New(opts)
e.SetExecuterOptions(&protocols.ExecutorOptions{Logger: e.Logger, ResumeCfg: types.NewResumeCfg(), ProtocolType: tmpltypes.HTTPProtocol})
return e
}
type gateExecuter struct {
started chan struct{}
once sync.Once
}
func (g *gateExecuter) Compile() error { return nil }
func (g *gateExecuter) Requests() int { return 1 }
func (g *gateExecuter) Execute(ctx *scan.ScanContext) (bool, error) {
g.once.Do(func() { close(g.started) })
<-ctx.Context().Done()
return false, ctx.Context().Err()
}
func (g *gateExecuter) ExecuteWithResults(ctx *scan.ScanContext) ([]*output.ResultEvent, error) {
_, err := g.Execute(ctx)
return nil, err
}
func TestExecuteTemplateSprayUnblocksCancelledAcquire(t *testing.T) {
e := newCancelTestEngine()
gate := &gateExecuter{started: make(chan struct{})}
templatesList := []*templates.Template{{Executer: gate}, {Executer: &gateExecuter{started: make(chan struct{})}}}
targets := &fakeTargetProvider{values: []*contextargs.MetaInput{{Input: "a"}}}
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() {
defer close(done)
e.executeTemplateSpray(ctx, templatesList, targets)
}()
select {
case <-gate.started:
case <-time.After(2 * time.Second):
t.Fatal("first template did not start")
}
cancel()
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("executeTemplateSpray stayed blocked after cancel")
}
}
func TestExecuteHostSprayUnblocksCancelledAcquire(t *testing.T) {
e := newCancelTestEngine()
gate := &gateExecuter{started: make(chan struct{})}
templatesList := []*templates.Template{{Executer: gate}}
targets := &fakeTargetProvider{values: []*contextargs.MetaInput{{Input: "a"}, {Input: "b"}}}
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() {
defer close(done)
e.executeHostSpray(ctx, templatesList, targets)
}()
select {
case <-gate.started:
case <-time.After(2 * time.Second):
t.Fatal("first host did not start")
}
cancel()
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("executeHostSpray stayed blocked after cancel")
}
}
func TestExecuteTemplatesOnTargetUnblocksCancelledAcquire(t *testing.T) {
e := newCancelTestEngine()
gate := &gateExecuter{started: make(chan struct{})}
var matched atomic.Bool
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() {
defer close(done)
e.executeTemplatesOnTarget(ctx, []*templates.Template{{Executer: gate}, {Executer: &gateExecuter{started: make(chan struct{})}}}, &contextargs.MetaInput{Input: "a"}, &matched)
}()
select {
case <-gate.started:
case <-time.After(2 * time.Second):
t.Fatal("first template did not start")
}
cancel()
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("executeTemplatesOnTarget stayed blocked after cancel")
}
}
+242
View File
@@ -0,0 +1,242 @@
package core
import (
"context"
"sync"
"sync/atomic"
"testing"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/progress"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/hosterrorscache"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/http"
"github.com/projectdiscovery/nuclei/v3/pkg/templates"
tmpltypes "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/nuclei/v3/pkg/types/scanstrategy"
"github.com/stretchr/testify/require"
)
// taskRecorder counts the tasks the engine schedules and completes.
type taskRecorder struct {
progress.Progress
total atomic.Int64
done atomic.Int64
}
func (r *taskRecorder) Init(int64, int, int64) {}
func (r *taskRecorder) AddToTaskTotal(delta int64) { r.total.Add(delta) }
func (r *taskRecorder) IncrementTasksBy(count int64) { r.done.Add(count) }
// downHosts reports the listed inputs as unresponsive.
type downHosts struct {
hosterrorscache.CacheInterface
inputs map[string]bool
}
func (d *downHosts) Check(_ string, ctx *contextargs.Context) bool {
return d.inputs[ctx.MetaInput.Input]
}
// newTaskEngine returns an engine for strategy whose progress records tasks.
// down lists the unresponsive inputs, and a nil resume starts a fresh scan.
func newTaskEngine(strategy scanstrategy.ScanStrategy, scope protocols.TargetScope, down map[string]bool, resume *types.ResumeCfg) (*Engine, *taskRecorder) {
options := &types.Options{
BulkSize: 2,
TemplateThreads: 2,
HeadlessBulkSize: 1,
HeadlessTemplateThreads: 1,
ScanStrategy: strategy.String(),
Logger: gologger.DefaultLogger,
}
if resume == nil {
resume = types.NewResumeCfg()
}
recorder := &taskRecorder{}
executerOpts := &protocols.ExecutorOptions{
Options: options,
ResumeCfg: resume,
ProtocolType: tmpltypes.HTTPProtocol,
Progress: recorder,
TargetScope: scope,
}
if down != nil {
executerOpts.HostErrorsCache = &downHosts{inputs: down}
}
engine := New(options)
executerOpts.Logger = engine.Logger
engine.SetExecuterOptions(executerOpts)
return engine, recorder
}
// targetsOf returns a target provider of inputs.
func targetsOf(inputs ...string) *fakeTargetProvider {
targets := &fakeTargetProvider{}
for _, input := range inputs {
targets.values = append(targets.values, &contextargs.MetaInput{Input: input})
}
return targets
}
func TestExecuteScanCountsTasks(t *testing.T) {
scope := scopeByInput{
"php": {paths: []string{"thinkphp.yaml"}},
"tomcat": {paths: []string{"tomcat.yaml", "generic.yaml"}},
"none": {},
}
inputs := []string{"php", "tomcat", "none", "plain"}
tests := []struct {
name string
scope protocols.TargetScope
down map[string]bool
wantRuns int
wantTotal int64
wantDone int64
}{
{
name: "every pair",
// 3 templates on 4 targets, and the self-contained template once
wantRuns: 13,
wantTotal: 13,
wantDone: 13,
},
{
name: "per-target profiles",
scope: scope,
// php 1, tomcat 2, none 0, plain 3, self-contained 1
wantRuns: 7,
wantTotal: 7,
wantDone: 7,
},
{
name: "unresponsive host",
down: map[string]bool{"tomcat": true},
// tomcat is skipped for all 3 templates
wantRuns: 10,
wantTotal: 13,
wantDone: 13,
},
{
name: "unresponsive host with per-target profiles",
scope: scope,
down: map[string]bool{"tomcat": true},
// tomcat is skipped for its 2 templates
wantRuns: 5,
wantTotal: 7,
wantDone: 7,
},
}
for _, strategy := range []scanstrategy.ScanStrategy{scanstrategy.TemplateSpray, scanstrategy.HostSpray} {
for _, test := range tests {
t.Run(strategy.String()+"/"+test.name, func(t *testing.T) {
engine, recorder := newTaskEngine(strategy, test.scope, test.down, nil)
var (
mu sync.Mutex
runs []string
)
var templatesList []*templates.Template
for _, path := range []string{"thinkphp.yaml", "tomcat.yaml", "generic.yaml"} {
templatesList = append(templatesList, &templates.Template{
ID: path,
Path: path,
TotalRequests: 1,
Executer: &recordingExecuter{path: path, mu: &mu, runs: &runs},
})
}
templatesList = append(templatesList, &templates.Template{
ID: "self.yaml",
Path: "self.yaml",
TotalRequests: 1,
SelfContained: true,
Executer: &recordingExecuter{path: "self.yaml", mu: &mu, runs: &runs},
})
engine.ExecuteScanWithOpts(context.Background(), templatesList, targetsOf(inputs...), true)
require.Len(t, runs, test.wantRuns)
require.Equal(t, test.wantTotal, recorder.total.Load(), "task total")
require.Equal(t, test.wantDone, recorder.done.Load(), "done tasks")
})
}
}
}
func TestExecuteScanCountsOriginScopedTasks(t *testing.T) {
// a {{RootURL}} template runs once per origin, and the skipped repeats
// are done tasks too
inputs := []string{"http://a.test/x", "http://a.test/y", "http://a.test/z", "http://b.test/x"}
for _, strategy := range []scanstrategy.ScanStrategy{scanstrategy.TemplateSpray, scanstrategy.HostSpray} {
t.Run(strategy.String(), func(t *testing.T) {
engine, recorder := newTaskEngine(strategy, nil, nil, nil)
var (
mu sync.Mutex
runs []string
)
templatesList := []*templates.Template{{
ID: "root.yaml",
Path: "root.yaml",
TotalRequests: 1,
RequestsHTTP: []*http.Request{{Path: []string{"{{RootURL}}/"}}},
Executer: &recordingExecuter{path: "root.yaml", mu: &mu, runs: &runs},
}}
require.True(t, templatesList[0].IsOriginScoped())
engine.ExecuteScanWithOpts(context.Background(), templatesList, targetsOf(inputs...), true)
require.Len(t, runs, 2, "one run per origin")
require.EqualValues(t, 4, recorder.total.Load(), "task total")
require.EqualValues(t, 4, recorder.done.Load(), "done tasks")
})
}
}
func TestExecuteScanCountsResumedTasks(t *testing.T) {
tests := []struct {
name string
down map[string]bool
wantRuns []string
}{
{
name: "responsive hosts",
wantRuns: []string{"todo.yaml@a", "todo.yaml@b"},
},
{
// a resumed pair on an unresponsive host is counted once
name: "unresponsive host",
down: map[string]bool{"a": true},
wantRuns: []string{"todo.yaml@b"},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
resumeCfg := types.NewResumeCfg()
resumeCfg.ResumeFrom["done.yaml"] = &types.ResumeInfo{Completed: true}
engine, recorder := newTaskEngine(scanstrategy.TemplateSpray, nil, test.down, resumeCfg)
var (
mu sync.Mutex
runs []string
)
var templatesList []*templates.Template
for _, path := range []string{"done.yaml", "todo.yaml"} {
templatesList = append(templatesList, &templates.Template{
ID: path,
Path: path,
TotalRequests: 1,
Executer: &recordingExecuter{path: path, mu: &mu, runs: &runs},
})
}
engine.ExecuteScanWithOpts(context.Background(), templatesList, targetsOf("a", "b"), true)
require.ElementsMatch(t, test.wantRuns, runs)
require.EqualValues(t, 4, recorder.total.Load(), "task total")
require.EqualValues(t, 4, recorder.done.Load(), "done tasks")
})
}
}
+6 -2
View File
@@ -132,7 +132,9 @@ func (e *Engine) runWorkflowStep(template *workflows.WorkflowTemplate, ctx *scan
}
for _, subtemplate := range matcher.Subtemplates {
swg.Add()
if err := swg.AddWithContext(ctx.Context()); err != nil {
return
}
go func(subtemplate *workflows.WorkflowTemplate) {
defer swg.Done()
@@ -161,7 +163,9 @@ func (e *Engine) runWorkflowStep(template *workflows.WorkflowTemplate, ctx *scan
}
if len(template.Subtemplates) > 0 && firstMatched {
for _, subtemplate := range template.Subtemplates {
swg.Add()
if err := swg.AddWithContext(ctx.Context()); err != nil {
return err
}
go func(template *workflows.WorkflowTemplate) {
// create a new context with the same input but with unset callbacks
+57
View File
@@ -2,7 +2,9 @@ package core
import (
"context"
"sync/atomic"
"testing"
"time"
"github.com/projectdiscovery/nuclei/v3/pkg/model/types/stringslice"
"github.com/projectdiscovery/nuclei/v3/pkg/operators"
@@ -301,6 +303,61 @@ func TestWorkflowsSameStepExecutersRefreshTemplateContext(t *testing.T) {
require.Equal(t, "foo", secondWhoami, "could not refresh workflow context into later executers in the same step")
}
func TestWorkflowSubtemplatesUnblockCancelledAcquire(t *testing.T) {
progressBar, _ := progress.NewStatsTicker(0, false, false, false, 0)
var started atomic.Int32
block := &mockExecuter{result: true, executeScanHook: func(ctx *scan.ScanContext) {
started.Add(1)
<-ctx.Context().Done()
}}
var waitingStarted atomic.Bool
waiting := &mockExecuter{result: true, executeScanHook: func(ctx *scan.ScanContext) {
waitingStarted.Store(true)
<-ctx.Context().Done()
}}
// Two slots: the two blockers occupy them so the third AddWithContext waits.
workflow := &workflows.Workflow{Options: &protocols.ExecutorOptions{Options: &types.Options{TemplateThreads: 2}}, Workflows: []*workflows.WorkflowTemplate{
{Executers: []*workflows.ProtocolExecuterPair{{
Executer: &mockExecuter{result: true, outputs: []*output.InternalWrappedEvent{
{OperatorsResult: &operators.Result{}, Results: []*output.ResultEvent{{}}},
}}, Options: &protocols.ExecutorOptions{Progress: progressBar}},
}, Subtemplates: []*workflows.WorkflowTemplate{
{Executers: []*workflows.ProtocolExecuterPair{{Executer: block, Options: &protocols.ExecutorOptions{Progress: progressBar}}}},
{Executers: []*workflows.ProtocolExecuterPair{{Executer: block, Options: &protocols.ExecutorOptions{Progress: progressBar}}}},
{Executers: []*workflows.ProtocolExecuterPair{{Executer: waiting, Options: &protocols.ExecutorOptions{Progress: progressBar}}}},
}},
}}
engine := &Engine{}
parent, cancel := context.WithCancel(context.Background())
input := contextargs.NewWithInput(parent, "https://test.com")
ctx := scan.NewScanContext(parent, input)
done := make(chan struct{})
go func() {
defer close(done)
engine.executeWorkflow(ctx, workflow)
}()
deadline := time.After(2 * time.Second)
for started.Load() < 2 {
select {
case <-deadline:
t.Fatal("blocking subtemplates did not occupy both slots")
case <-time.After(10 * time.Millisecond):
}
}
cancel()
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("workflow stayed blocked after cancel")
}
require.False(t, waitingStarted.Load(), "third subtemplate must not start while slots are held")
}
type mockExecuter struct {
result bool
executeHook func(input *contextargs.MetaInput)
+5 -1
View File
@@ -62,6 +62,10 @@ func (w *WorkPool) InputPool(templateType types.ProtocolType) *syncutil.Adaptive
}
func (w *WorkPool) RefreshWithConfig(config WorkPoolConfig) {
w.RefreshWithConfigContext(context.Background(), config)
}
func (w *WorkPool) RefreshWithConfigContext(ctx context.Context, config WorkPoolConfig) {
if w.config.TypeConcurrency != config.TypeConcurrency {
w.config.TypeConcurrency = config.TypeConcurrency
}
@@ -74,7 +78,7 @@ func (w *WorkPool) RefreshWithConfig(config WorkPoolConfig) {
if w.config.HeadlessInputConcurrency != config.HeadlessInputConcurrency {
w.config.HeadlessInputConcurrency = config.HeadlessInputConcurrency
}
w.Refresh(context.Background())
w.Refresh(ctx)
}
func (w *WorkPool) Refresh(ctx context.Context) {
+2 -4
View File
@@ -10,6 +10,7 @@ import (
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/fuzz/dataformat"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/httpbody"
"github.com/projectdiscovery/retryablehttp-go"
)
@@ -147,10 +148,7 @@ func (b *Body) Rebuild() (*retryablehttp.Request, error) {
return nil, errors.Wrap(err, "could not encode body")
}
cloned := b.req.Clone(context.Background())
err = cloned.SetBodyString(encoded)
if err != nil {
return nil, errors.Wrap(err, "could not set body")
}
httpbody.SetString(cloned, encoded)
cloned.Header.Set("Content-Length", strconv.Itoa(len(encoded)))
return cloned, nil
}
+2
View File
@@ -9,6 +9,7 @@ import (
"sync"
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/httpbody"
"github.com/projectdiscovery/retryablehttp-go"
urlutil "github.com/projectdiscovery/utils/url"
"github.com/stretchr/testify/require"
@@ -45,6 +46,7 @@ func TestBodyComponent(t *testing.T) {
t.Fatal(err)
}
require.IsType(t, &httpbody.Body{}, rebuilt.Body, "rebuilt body must not be copied per read")
newBody, err := io.ReadAll(rebuilt.Body)
if err != nil {
t.Fatal(err)
+1 -1
View File
@@ -13,7 +13,7 @@ import (
"github.com/stretchr/testify/require"
)
func newFuzzTestInteractshClient(t *testing.T) *interactsh.Client {
func newFuzzTestInteractshClient(t *testing.T) interactsh.IClient {
t.Helper()
client, err := interactsh.New(&interactsh.Options{
+18
View File
@@ -43,6 +43,7 @@ https://d.example
| Key | Selects |
| --- | --- |
| `target` | the target (URL, host, IP, CIDR or ASN), required |
| `ip` | the address to connect to instead of resolving the target's host, see below |
| `profile` | a template profile ID from the templates `profiles` directory (paths are rejected); the line's other keys override it |
| `templates` | template paths, relative to the templates directory |
| `tags`, `exclude-tags`, `include-tags`, `author`, `template-id`, `exclude-id`, `severity`, `exclude-severity`, `type`, `exclude-type` | same as the CLI flags |
@@ -54,6 +55,23 @@ https://d.example
- Unknown keys fail the run rather than widening the scan. Profile keys that configure the whole scan, such as `code` or `var`, are ignored per target.
- Not supported with `-automatic-scan`. Global matchers still apply to every target.
### Pinning a target to an IP
`ip` sends the target's traffic to one address, like `curl --resolve`: the hostname is still used for the Host header, SNI and template variables such as `{{Hostname}}`, while connections to that hostname go to the ip. This scans a vhost on a given server, or an origin behind a CDN.
```jsonl
{"target": "https://app.example", "ip": "192.0.2.10"}
{"target": "https://app.example", "ip": "192.0.2.11", "tags": ["cve"]}
```
- Each `target` and `ip` pair is a separate target: the same host can be listed with several ips, and each pair has its own selection. `{{ip}}` and the `ip` field of results hold the pinned ip.
- The target must have a hostname; IP, CIDR and ASN targets cannot be pinned. `-sa` and `-iv` do not apply to a pinned target.
- A pinned ip excluded with `-eh`, or a local one with `-lna`, is skipped.
- `-mup` counts the urls of each ip apart, so a url listed with several ips is kept on each of them.
- HTTP, network, websocket, javascript and ssl templates connect to the ip. Headless templates and `unsafe: true` raw requests do not.
- Pinned targets cannot be combined with `-proxy`, `-replay-proxy` or `-automatic-scan`.
- `-u` splits its value on commas unless the value is quoted, so pass target lines in a file (`-l`) or on stdin.
## Formats
Fuzzing and DAST take the request shape (method, body, params) from the input, not from CLI flags, so a bare URL is always fuzzed as a `GET`. To fuzz anything else, feed a request-shaped input with `-im`:
+19 -1
View File
@@ -53,7 +53,25 @@ For unsupported media types, no appropriate content type is found for the body.
This module converts each operation into one or more example HTTP requests. Each request is dumped into a string format, accompanied by its method, URL, headers, and body. These are send as a callback for further processing.
_Please note: This document does not cover other features of OpenAPI specification like responses, security schemes, links, callbacks, etc. as these are not currently handled by the module._
### Security Schemes
The OpenAPI input supports HTTP basic and bearer authentication, API keys in headers, query parameters or cookies, and OAuth2/OpenID Connect access tokens. Security requirements can be declared globally or per operation.
For OAuth2 (`type: oauth2`) and OpenID Connect (`type: openIdConnect`), supply an existing access token as the full `Authorization` header value:
```sh
nuclei -l openapi.yaml -im openapi -var 'Authorization=Bearer YOUR_ACCESS_TOKEN'
```
The same variable is used for HTTP bearer authentication. For basic authentication, provide the full `Basic ...` value; for API keys, use the parameter name specified by the security scheme.
Nuclei does not perform OAuth authorization flows, discover OpenID Connect providers, acquire tokens, or refresh them. Obtain an access token with the scopes required by the API before running the scan. The provided header value is used unchanged.
Security requirement objects are alternatives. The generator selects the first supported alternative, in document order, for which all required credentials are supplied, and includes every scheme in that object. An empty object (`{}`) permits anonymous access. If no supported alternative has all its credentials, the first supported alternative is used to report missing values; credentials from different alternatives are not combined.
Operations declaring `security: []` can be generated without the global token. Absent, nil or empty credential values cause affected requests to be skipped. Normal parameter validation reports missing credentials even when another required parameter is also missing. With `-skip-format-validation`, requests with missing credentials are silently skipped and the remaining operations are generated without inventing placeholder credentials.
_Please note: Other OpenAPI features such as responses, links and callbacks are not currently handled by the module._
## Postman Collection file
+113 -67
View File
@@ -36,33 +36,7 @@ func GenerateRequestsFromSchema(schema *openapi3.T, opts formats.InputFormatOpti
if len(schema.Servers) == 0 {
return errors.New("no servers found in openapi schema")
}
// new set of globalParams obtained from security schemes
globalParams := openapi3.NewParameters()
if len(schema.Security) > 0 {
params, err := GetGlobalParamsForSecurityRequirement(schema, &schema.Security)
if err != nil {
return err
}
globalParams = append(globalParams, params...)
}
// validate global param requirements
for _, param := range globalParams {
if val, ok := opts.Variables[param.Value.Name]; ok {
param.Value.Example = val
} else {
// if missing check for validation
if opts.SkipFormatValidation {
gologger.Verbose().Msgf("openapi: skipping all requests due to missing global auth parameter: %s\n", param.Value.Name)
return nil
} else {
// fatal error
gologger.Fatal().Msgf("openapi: missing global auth parameter: %s\n", param.Value.Name)
}
}
}
globalParams, globalSecurityErr := selectSecurityParameters(schema, &schema.Security, opts.Variables)
missingVarMap := make(map[string]struct{})
optionalVarMap := make(map[string]struct{})
@@ -92,6 +66,10 @@ func GenerateRequestsFromSchema(schema *openapi3.T, opts formats.InputFormatOpti
requestPath = serverPath + path
}
for method, ov := range ops {
// An override need not use an unsupported global security scheme.
if ov.Security == nil && globalSecurityErr != nil {
return globalSecurityErr
}
if err := generateRequestsFromOp(&generateReqOptions{
requiredOnly: opts.RequiredOnly,
method: method,
@@ -148,7 +126,7 @@ type generateReqOptions struct {
// post request generation callback
callback formats.ParseReqRespCallback
// global parameters
// Selected global authentication parameters, used only without an override.
globalParams openapi3.Parameters
// requestparams map
reqParams openapi3.Parameters
@@ -175,21 +153,42 @@ func generateRequestsFromOp(opts *generateReqOptions) error {
}
// add existing req params
reqParams = append(reqParams, opts.op.Parameters...)
// check for endpoint specific auth
// Select one complete requirement for this operation before adding its
// authentication parameters. Operation security overrides global security.
authParams := opts.globalParams
if opts.op.Security != nil {
params, err := GetGlobalParamsForSecurityRequirement(opts.schema, opts.op.Security)
authParams, err = selectSecurityParameters(opts.schema, opts.op.Security, opts.opts.Variables)
if err != nil {
return err
}
reqParams = append(reqParams, params...)
} else {
reqParams = append(reqParams, opts.globalParams...)
}
reqParams = append(reqParams, authParams...)
authParameters := make(map[*openapi3.Parameter]struct{}, len(authParams))
for _, parameter := range authParams {
authParameters[parameter.Value] = struct{}{}
}
// Collect missing credentials before ordinary parameter validation can return
// early. Still validate ordinary parameters so their diagnostics are retained.
missingAuth := false
for _, parameter := range authParams {
value := parameter.Value
if types.ToString(opts.opts.Variables[value.Name]) != "" {
continue
}
missingAuth = true
if opts.missingParamValueCallback != nil {
opts.missingParamValueCallback(value, opts)
}
}
query := url.Values{}
for _, parameter := range reqParams {
value := parameter.Value
if _, isAuth := authParameters[value]; missingAuth && isAuth {
continue
}
if value.Schema == nil || value.Schema.Value == nil {
continue
}
@@ -261,6 +260,9 @@ func generateRequestsFromOp(opts *generateReqOptions) error {
req.AddCookie(&http.Cookie{Name: value.Name, Value: types.ToString(paramValue)})
}
}
if missingAuth {
return nil
}
req.URL.RawQuery = query.Encode()
req.URL.Path = opts.requestPath
@@ -271,7 +273,7 @@ func generateRequestsFromOp(opts *generateReqOptions) error {
var val interface{}
if value.Schema == nil || value.Schema.Value == nil {
val = generateEmptySchemaValue(content)
val, _ = generateExampleFromSchema(generateEmptySchemaValue(content))
} else {
var err error
@@ -291,15 +293,25 @@ func generateRequestsFromOp(opts *generateReqOptions) error {
cloned.Header.Set("Content-Type", "application/json")
}
case "application/xml":
values := mxj.Map(val.(map[string]interface{}))
var marshalled []byte
var err error
switch v := val.(type) {
case map[string]interface{}:
marshalled, err = mxj.Map(v).Xml()
case string:
marshalled = []byte(v)
default:
err = errors.Errorf("unsupported xml example type %T", val)
}
if marshalled, err := values.Xml(); err == nil {
if err == nil {
// body = string(marshalled)
cloned.Body = io.NopCloser(bytes.NewReader(marshalled))
cloned.ContentLength = int64(len(marshalled))
cloned.Header.Set("Content-Type", "application/xml")
} else {
gologger.Warning().Msgf("openapi: could not encode xml")
continue
}
case "application/x-www-form-urlencoded":
if values, ok := val.(map[string]interface{}); ok {
@@ -345,7 +357,7 @@ func generateRequestsFromOp(opts *generateReqOptions) error {
// use two strings
str = "string1\nstring2"
}
if value.Schema != nil && generic.EqualsAny(value.Schema.Value.Format, "bindary", "byte") {
if value.Schema != nil && generic.EqualsAny(value.Schema.Value.Format, "binary", "byte") {
cloned.Body = io.NopCloser(bytes.NewReader([]byte(str)))
cloned.ContentLength = int64(len(str))
cloned.Header.Set("Content-Type", "application/octet-stream")
@@ -391,45 +403,79 @@ func generateRequestsFromOp(opts *generateReqOptions) error {
return nil
}
// GetGlobalParamsForSecurityRequirement returns the global parameters for a security requirement
// GetGlobalParamsForSecurityRequirement returns parameters for the first supported
// security alternative, or no parameters when anonymous access is permitted.
func GetGlobalParamsForSecurityRequirement(schema *openapi3.T, requirement *openapi3.SecurityRequirements) ([]*openapi3.ParameterRef, error) {
globalParams := openapi3.NewParameters()
if len(schema.Components.SecuritySchemes) == 0 {
return nil, errkit.Newf("security requirements (%+v) without any security schemes found in openapi file", schema.Security)
return selectSecurityParameters(schema, requirement, nil)
}
// selectSecurityParameters chooses the first fully supplied alternative. If none
// is supplied, return the first supported alternative for missing-value reporting.
func selectSecurityParameters(schema *openapi3.T, requirement *openapi3.SecurityRequirements, variables map[string]interface{}) ([]*openapi3.ParameterRef, error) {
if requirement == nil || len(*requirement) == 0 {
return nil, nil
}
found := false
// this api is protected for each security scheme pull its corresponding scheme
schemaLabel:
var fallback []*openapi3.ParameterRef
var firstError error
for _, security := range *requirement {
for name := range security {
if scheme, ok := schema.Components.SecuritySchemes[name]; ok {
found = true
param, err := GenerateParameterFromSecurityScheme(scheme)
if err != nil {
return nil, err
}
globalParams = append(globalParams, &openapi3.ParameterRef{Value: param})
continue schemaLabel
if len(security) == 0 {
return nil, nil
}
params := openapi3.NewParameters()
complete, supported := true, true
// Map order must not change which schemes are emitted or reported missing.
for _, name := range mapsutil.GetSortedKeys(security) {
var scheme *openapi3.SecuritySchemeRef
if schema.Components != nil {
scheme = schema.Components.SecuritySchemes[name]
}
if scheme == nil || scheme.Value == nil {
if firstError == nil {
firstError = errkit.Newf("security scheme (%s) not found in openapi file", name)
}
supported = false
break
}
param, err := GenerateParameterFromSecurityScheme(scheme)
if err != nil {
if firstError == nil {
firstError = err
}
supported = false
break
}
params = append(params, &openapi3.ParameterRef{Value: param})
complete = complete && types.ToString(variables[param.Name]) != ""
}
if !found && len(security) > 1 {
// if this is case then both security schemes are required
return nil, errkit.Newf("security requirement (%+v) not found in openapi file", security)
if !supported {
continue
}
if complete {
return params, nil
}
if fallback == nil {
fallback = params
}
}
if !found {
return nil, errkit.Newf("security requirement (%+v) not found in openapi file", requirement)
if fallback != nil {
return fallback, nil
}
return globalParams, nil
return nil, firstError
}
// GenerateParameterFromSecurityScheme generates an example from a schema object
func GenerateParameterFromSecurityScheme(scheme *openapi3.SecuritySchemeRef) (*openapi3.Parameter, error) {
if !generic.EqualsAny(scheme.Value.Type, "http", "apiKey") {
if !generic.EqualsAny(scheme.Value.Type, "http", "apiKey", "oauth2", "openIdConnect") {
return nil, errkit.Newf("unsupported security scheme type (%s) found in openapi file", scheme.Value.Type)
}
if generic.EqualsAny(scheme.Value.Type, "oauth2", "openIdConnect") {
// Access tokens are supplied by the caller through the Authorization variable.
// Request generation does not acquire or refresh tokens from the provider.
h := openapi3.NewHeaderParameter(DEFAULT_HTTP_SCHEME_HEADER).WithSchema(openapi3.NewStringSchema())
h.Required = true
h.Description = globalAuth
return h, nil
}
if scheme.Value.Type == "http" {
// check scheme
if !generic.EqualsAny(scheme.Value.Scheme, "basic", "bearer") {
@@ -443,12 +489,12 @@ func GenerateParameterFromSecurityScheme(scheme *openapi3.SecuritySchemeRef) (*o
// create parameters using the scheme
switch scheme.Value.Scheme {
case "basic":
h := openapi3.NewHeaderParameter(headerName)
h := openapi3.NewHeaderParameter(headerName).WithSchema(openapi3.NewStringSchema())
h.Required = true
h.Description = globalAuth // differentiator for normal variables and global auth
return h, nil
case "bearer":
h := openapi3.NewHeaderParameter(headerName)
h := openapi3.NewHeaderParameter(headerName).WithSchema(openapi3.NewStringSchema())
h.Required = true
h.Description = globalAuth // differentiator for normal variables and global auth
return h, nil
@@ -466,17 +512,17 @@ func GenerateParameterFromSecurityScheme(scheme *openapi3.SecuritySchemeRef) (*o
// create parameters using the scheme
switch scheme.Value.In {
case "query":
q := openapi3.NewQueryParameter(scheme.Value.Name)
q := openapi3.NewQueryParameter(scheme.Value.Name).WithSchema(openapi3.NewStringSchema())
q.Required = true
q.Description = globalAuth // differentiator for normal variables and global auth
return q, nil
case "header":
h := openapi3.NewHeaderParameter(scheme.Value.Name)
h := openapi3.NewHeaderParameter(scheme.Value.Name).WithSchema(openapi3.NewStringSchema())
h.Required = true
h.Description = globalAuth // differentiator for normal variables and global auth
return h, nil
case "cookie":
c := openapi3.NewCookieParameter(scheme.Value.Name)
c := openapi3.NewCookieParameter(scheme.Value.Name).WithSchema(openapi3.NewStringSchema())
c.Required = true
c.Description = globalAuth // differentiator for normal variables and global auth
return c, nil
@@ -0,0 +1,46 @@
package openapi
import (
"strings"
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/input/types"
"github.com/stretchr/testify/require"
)
func TestOpenAPIOctetStreamContentType(t *testing.T) {
tests := []struct {
name string
schema string
contentType string
}{
{name: "binary format", schema: "{type: string, format: binary}", contentType: "application/octet-stream"},
{name: "byte format", schema: "{type: string, format: byte}", contentType: "application/octet-stream"},
{name: "plain string", schema: "{type: string}", contentType: "text/plain"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
spec := `openapi: 3.0.0
info: {title: t, version: "1"}
servers: [{url: "http://example.com"}]
paths:
/upload:
post:
requestBody:
content:
application/octet-stream: {schema: ` + tt.schema + `}
responses:
"200": {description: ok}
`
var contentTypes []string
err := New().Parse(strings.NewReader(spec), func(rr *types.RequestResponse) bool {
value, _ := rr.Request.Headers.Get("Content-Type")
contentTypes = append(contentTypes, value)
return false
}, "spec.yaml")
require.NoError(t, err)
require.Equal(t, []string{tt.contentType}, contentTypes)
})
}
}
+350
View File
@@ -0,0 +1,350 @@
package openapi
import (
"bufio"
"context"
"fmt"
"net/http"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"
"github.com/getkin/kin-openapi/openapi3"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/input/formats"
httpTypes "github.com/projectdiscovery/nuclei/v3/pkg/input/types"
"github.com/stretchr/testify/require"
)
func TestGenerateRequestsWithSecuritySchemes(t *testing.T) {
for _, test := range []struct {
name string
scheme string
variable string
value string
in string
}{
{
name: "oauth2", scheme: `{"type":"oauth2","flows":{"clientCredentials":{"tokenUrl":"https://example.com/token","scopes":{"read":"Read access"}}}}`,
variable: "Authorization", value: "Bearer supplied-token", in: "header",
},
{
name: "openIdConnect", scheme: `{"type":"openIdConnect","openIdConnectUrl":"https://example.com/.well-known/openid-configuration"}`,
variable: "Authorization", value: "Bearer supplied-token", in: "header",
},
{
name: "bearer", scheme: `{"type":"http","scheme":"bearer"}`,
variable: "Authorization", value: "Bearer supplied-token", in: "header",
},
{
name: "basic", scheme: `{"type":"http","scheme":"basic"}`,
variable: "Authorization", value: "Basic dXNlcjpwYXNz", in: "header",
},
{
name: "apiKey-header", scheme: `{"type":"apiKey","name":"X-API-Key","in":"header"}`,
variable: "X-API-Key", value: "supplied-key", in: "header",
},
{
name: "apiKey-query", scheme: `{"type":"apiKey","name":"api_key","in":"query"}`,
variable: "api_key", value: "supplied-key", in: "query",
},
{
name: "apiKey-cookie", scheme: `{"type":"apiKey","name":"session","in":"cookie"}`,
variable: "session", value: "supplied-key", in: "cookie",
},
} {
for _, scope := range []string{"global", "operation"} {
t.Run(test.name+"/"+scope, func(t *testing.T) {
security := `"security":[{"auth":[]}],`
if test.name == "oauth2" || test.name == "openIdConnect" {
security = `"security":[{"auth":["read"]}],`
}
globalSecurity, operationSecurity := security, ""
if scope == "operation" {
globalSecurity, operationSecurity = "", security
}
document := fmt.Sprintf(`{
"openapi":"3.0.3",
"info":{"title":"Authentication test","version":"1.0"},
"servers":[{"url":"https://example.com"}],
"components":{"securitySchemes":{"auth":%s}},
%s
"paths":{"/protected":{"get":{%s"responses":{"200":{"description":"OK"}}}}}
}`, test.scheme, globalSecurity, operationSecurity)
parser := New()
parser.SetOptions(formats.InputFormatOptions{
RequiredOnly: true,
Variables: map[string]interface{}{test.variable: test.value},
})
var requests []*http.Request
err := parser.Parse(strings.NewReader(document), func(rr *httpTypes.RequestResponse) bool {
req, err := http.ReadRequest(bufio.NewReader(strings.NewReader(rr.Request.Raw)))
require.NoError(t, err)
requests = append(requests, req)
return false
}, "")
require.NoError(t, err)
require.Len(t, requests, 1)
req := requests[0]
t.Cleanup(func() { require.NoError(t, req.Body.Close()) })
require.Equal(t, "/protected", req.URL.Path)
switch test.in {
case "header":
require.Equal(t, test.value, req.Header.Get(test.variable))
case "query":
require.Equal(t, test.value, req.URL.Query().Get(test.variable))
case "cookie":
cookie, err := req.Cookie(test.variable)
require.NoError(t, err)
require.Equal(t, test.value, cookie.Value)
}
})
}
}
}
func TestGenerateRequestsOAuthOperationOverride(t *testing.T) {
parser := New()
parser.SetOptions(formats.InputFormatOptions{
Variables: map[string]interface{}{
"Authorization": "Bearer supplied-token",
"X-API-Key": "supplied-key",
},
})
document := `{
"openapi":"3.0.3",
"info":{"title":"Authentication override test","version":"1.0"},
"servers":[{"url":"https://example.com"}],
"components":{"securitySchemes":{
"oauth":{"type":"oauth2","flows":{"clientCredentials":{"tokenUrl":"https://example.com/token","scopes":{}}}},
"key":{"type":"apiKey","name":"X-API-Key","in":"header"}
}},
"security":[{"oauth":[]}],
"paths":{
"/protected":{"get":{"security":[{"key":[]}],"responses":{"200":{"description":"OK"}}}},
"/public":{"get":{"security":[],"responses":{"200":{"description":"OK"}}}}
}
}`
paths := make(map[string]bool)
err := parser.Parse(strings.NewReader(document), func(rr *httpTypes.RequestResponse) bool {
paths[rr.URL.Path] = true
key, _ := rr.Request.Headers.Get("X-Api-Key")
if rr.URL.Path == "/protected" {
require.Equal(t, "supplied-key", key)
} else {
require.Empty(t, key)
}
auth, _ := rr.Request.Headers.Get("Authorization")
require.Empty(t, auth)
return false
}, "")
require.NoError(t, err)
require.Equal(t, map[string]bool{"/protected": true, "/public": true}, paths)
}
func TestGenerateRequestsOAuthRequiresToken(t *testing.T) {
for _, skipValidation := range []bool{false, true} {
for _, schemeType := range []string{"oauth2", "openIdConnect"} {
t.Run(fmt.Sprintf("%s/skip-validation=%t", schemeType, skipValidation), func(t *testing.T) {
param, err := GenerateParameterFromSecurityScheme(&openapi3.SecuritySchemeRef{
Value: &openapi3.SecurityScheme{Type: schemeType},
})
require.NoError(t, err)
var missing []string
err = generateRequestsFromOp(&generateReqOptions{
method: http.MethodGet,
pathURL: "https://example.com",
requestPath: "/protected",
op: &openapi3.Operation{},
opts: formats.InputFormatOptions{SkipFormatValidation: skipValidation},
globalParams: openapi3.Parameters{&openapi3.ParameterRef{Value: param}},
missingParamValueCallback: func(param *openapi3.Parameter, _ *generateReqOptions) {
missing = append(missing, param.Name)
},
callback: func(_ *httpTypes.RequestResponse) bool {
t.Fatal("must not generate an authenticated request without a token")
return false
},
})
require.NoError(t, err)
require.Equal(t, []string{"Authorization"}, missing)
})
}
}
}
func oauthSecurityDocument(paths string) string {
return fmt.Sprintf(`{
"openapi":"3.0.3",
"info":{"title":"Authentication scope test","version":"1.0"},
"servers":[{"url":"https://example.com"}],
"components":{"securitySchemes":{
"oauth":{"type":"oauth2","flows":{"clientCredentials":{"tokenUrl":"https://example.com/token","scopes":{}}}},
"key":{"type":"apiKey","name":"X-API-Key","in":"header"}
}},
"security":[{"oauth":[]}],
"paths":{%s}
}`, paths)
}
func TestGenerateRequestsOAuthWithoutGlobalToken(t *testing.T) {
for _, skipValidation := range []bool{true, false} {
for _, test := range []struct {
name string
paths string
want []string
}{
{
name: "public-only",
paths: `"/public":{"get":{"security":[],"responses":{"200":{"description":"OK"}}}}`,
want: []string{"/public"},
},
{
name: "mixed-security",
paths: `
"/protected":{"get":{"responses":{"200":{"description":"OK"}}}},
"/public":{"get":{"security":[],"responses":{"200":{"description":"OK"}}}},
"/key":{"get":{"security":[{"key":[]}],"responses":{"200":{"description":"OK"}}}}
`,
want: []string{"/public", "/key"},
},
} {
t.Run(fmt.Sprintf("%s/skip-validation=%t", test.name, skipValidation), func(t *testing.T) {
parser := New()
parser.SetOptions(formats.InputFormatOptions{
SkipFormatValidation: skipValidation,
Variables: map[string]interface{}{"X-API-Key": "supplied-key"},
})
var paths []string
err := parser.Parse(strings.NewReader(oauthSecurityDocument(test.paths)), func(rr *httpTypes.RequestResponse) bool {
paths = append(paths, rr.URL.Path)
auth, _ := rr.Request.Headers.Get("Authorization")
require.Empty(t, auth)
key, _ := rr.Request.Headers.Get("X-Api-Key")
if rr.URL.Path == "/key" {
require.Equal(t, "supplied-key", key)
} else {
require.Empty(t, key)
}
return false
}, "")
require.NoError(t, err)
require.ElementsMatch(t, test.want, paths)
})
}
}
}
func TestGenerateRequestsOAuthMissingTokenCLI(t *testing.T) {
const helperEnv = "NUCLEI_TEST_OPENAPI_MISSING_TOKEN"
if os.Getenv(helperEnv) == "1" {
config.CurrentAppMode = config.AppModeCLI
document := oauthSecurityDocument(`
"/protected/{id}":{"get":{
"parameters":[{"name":"id","in":"path","required":true,"schema":{"type":"string"}}],
"responses":{"200":{"description":"OK"}}
}},
"/public":{"get":{"security":[],"responses":{"200":{"description":"OK"}}}}
`)
err := New().Parse(strings.NewReader(document), func(rr *httpTypes.RequestResponse) bool {
fmt.Println("generated:", rr.URL.Path)
return false
}, "")
require.NoError(t, err)
return
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, os.Args[0], "-test.run=^TestGenerateRequestsOAuthMissingTokenCLI$")
cmd.Env = append(os.Environ(), helperEnv+"=1")
cmd.Dir = t.TempDir()
output, err := cmd.CombinedOutput()
var exitErr *exec.ExitError
require.ErrorAs(t, err, &exitErr)
require.Equal(t, 1, exitErr.ExitCode())
require.Contains(t, string(output), "generated: /public")
require.NotContains(t, string(output), "generated: /protected")
parameters, err := os.ReadFile(filepath.Join(cmd.Dir, formats.DefaultVarDumpFileName))
require.NoError(t, err)
require.Contains(t, string(parameters), "Authorization=")
require.Contains(t, string(parameters), "id=")
}
func TestGenerateRequestsCredentials(t *testing.T) {
for _, auth := range []struct {
scheme string
variable string
}{
{scheme: "oauth", variable: "Authorization"},
{scheme: "key", variable: "X-API-Key"},
} {
for _, scope := range []string{"global", "operation"} {
for _, skipValidation := range []bool{false, true} {
for _, credential := range []struct {
name string
present bool
value interface{}
wantProtected bool
}{
{name: "absent"},
{name: "nil", present: true},
{name: "empty", present: true, value: ""},
{name: "supplied", present: true, value: "opaque-token", wantProtected: true},
} {
t.Run(fmt.Sprintf("%s/%s/%s/skip-validation=%t", auth.scheme, scope, credential.name, skipValidation), func(t *testing.T) {
document := oauthSecurityDocument(`
"/protected":{"get":{"responses":{"200":{"description":"OK"}}}},
"/public":{"get":{"security":[],"responses":{"200":{"description":"OK"}}}}
`)
schema, err := openapi3.NewLoader().LoadFromData([]byte(document))
require.NoError(t, err)
security := openapi3.SecurityRequirements{{auth.scheme: []string{}}}
if scope == "global" {
schema.Security = security
} else {
schema.Security = nil
schema.Paths.Map()["/protected"].Get.Security = &security
}
variables := map[string]interface{}{}
if credential.present {
variables[auth.variable] = credential.value
}
var paths []string
err = GenerateRequestsFromSchema(schema, formats.InputFormatOptions{
Variables: variables,
SkipFormatValidation: skipValidation,
}, func(rr *httpTypes.RequestResponse) bool {
paths = append(paths, rr.URL.Path)
for _, header := range []string{"Authorization", "X-API-Key"} {
value, _ := rr.Request.Headers.Get(http.CanonicalHeaderKey(header))
if rr.URL.Path == "/protected" && header == auth.variable && credential.wantProtected {
require.Equal(t, credential.value, value)
} else {
require.Empty(t, value)
}
}
return false
})
require.NoError(t, err)
wantPaths := []string{"/public"}
if credential.wantProtected {
wantPaths = append(wantPaths, "/protected")
}
require.ElementsMatch(t, wantPaths, paths)
})
}
}
}
}
}
func TestGenerateParameterUnsupportedSecurityScheme(t *testing.T) {
_, err := GenerateParameterFromSecurityScheme(&openapi3.SecuritySchemeRef{
Value: &openapi3.SecurityScheme{Type: "mutualTLS"},
})
require.ErrorContains(t, err, "unsupported security scheme type (mutualTLS)")
}
+39
View File
@@ -68,3 +68,42 @@ func TestOpenAPIParser(t *testing.T) {
require.ElementsMatch(t, urls, methodToURLs[method], "invalid urls for method %s", method)
}
}
func TestOpenAPIRequestBodies(t *testing.T) {
tests := []struct {
name string
content string
bodies []string
}{
{name: "xml without schema", content: "application/xml: {}", bodies: []string{`<?xml version="1.0"?><root/>`}},
{name: "xml string schema", content: "application/xml: {schema: {type: string}}", bodies: []string{"string"}},
{name: "xml array schema", content: "application/xml: {schema: {type: array, items: {type: string}}}", bodies: nil},
{name: "json without schema", content: "application/json: {}", bodies: []string{"{}"}},
{name: "text without schema", content: "text/plain: {}", bodies: []string{"string"}},
{name: "octet-stream without schema", content: "application/octet-stream: {}", bodies: []string{"string1\nstring2"}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
spec := `openapi: 3.0.0
info: {title: t, version: "1"}
servers: [{url: "http://example.com"}]
paths:
/items:
post:
requestBody:
content:
` + tt.content + `
responses:
"200": {description: ok}
`
var bodies []string
err := New().Parse(strings.NewReader(spec), func(rr *types.RequestResponse) bool {
bodies = append(bodies, rr.Request.Body)
return false
}, "spec.yaml")
require.NoError(t, err)
require.Equal(t, tt.bodies, bodies)
})
}
}
+152
View File
@@ -0,0 +1,152 @@
package openapi
import (
"bufio"
"fmt"
"net/http"
"strings"
"testing"
"github.com/getkin/kin-openapi/openapi3"
"github.com/projectdiscovery/nuclei/v3/pkg/input/formats"
httpTypes "github.com/projectdiscovery/nuclei/v3/pkg/input/types"
"github.com/stretchr/testify/require"
)
func TestGenerateRequestsSecurityAlternatives(t *testing.T) {
for _, tc := range []struct {
name, security string
variables map[string]interface{}
wantCount int
wantAuth string
wantKey string
}{
{"first alternative", `[{"oauth":[]},{"key":[]}]`, map[string]interface{}{"Authorization": "Bearer token"}, 1, "Bearer token", ""},
{"second alternative", `[{"oauth":[]},{"key":[]}]`, map[string]interface{}{"X-API-Key": "key"}, 1, "", "key"},
{"both alternatives available", `[{"oauth":[]},{"key":[]}]`, map[string]interface{}{"Authorization": "Bearer token", "X-API-Key": "key"}, 1, "Bearer token", ""},
{"neither alternative available", `[{"oauth":[]},{"key":[]}]`, nil, 0, "", ""},
{"combined complete", `[{"oauth":[],"key":[]}]`, map[string]interface{}{"Authorization": "Bearer token", "X-API-Key": "key"}, 1, "Bearer token", "key"},
{"combined missing key", `[{"oauth":[],"key":[]}]`, map[string]interface{}{"Authorization": "Bearer token"}, 0, "", ""},
{"combined missing token", `[{"oauth":[],"key":[]}]`, map[string]interface{}{"X-API-Key": "key"}, 0, "", ""},
{"complete alternative after incomplete combined", `[{"oauth":[],"key":[]},{"key":[]}]`, map[string]interface{}{"X-API-Key": "key"}, 1, "", "key"},
{"anonymous alternative first", `[{}, {"oauth":[]}]`, map[string]interface{}{"Authorization": "Bearer token"}, 1, "", ""},
{"anonymous alternative last", `[{"oauth":[]},{}]`, nil, 1, "", ""},
{"empty security list", `[]`, nil, 1, "", ""},
{"unsupported alternative first", `[{"unsupported":[]},{"key":[]}]`, map[string]interface{}{"X-API-Key": "key"}, 1, "", "key"},
} {
for _, scope := range []string{"global", "operation"} {
for _, skip := range []bool{false, true} {
t.Run(fmt.Sprintf("%s/%s/skip=%t", tc.name, scope, skip), func(t *testing.T) {
global, operation := `"security":`+tc.security+`,`, ""
if scope == "operation" {
global, operation = "", global
}
document := fmt.Sprintf(`{
"openapi":"3.0.3","info":{"title":"Security alternatives","version":"1"},
"servers":[{"url":"https://example.com"}],
"components":{"securitySchemes":{
"oauth":{"type":"oauth2","flows":{"clientCredentials":{"tokenUrl":"https://example.com/token","scopes":{}}}},
"key":{"type":"apiKey","in":"header","name":"X-API-Key"},
"unsupported":{"type":"http","scheme":"digest"}
}},%s"paths":{"/protected":{"get":{%s"responses":{"200":{"description":"OK"}}}}}
}`, global, operation)
parser := New()
parser.SetOptions(formats.InputFormatOptions{SkipFormatValidation: skip, Variables: tc.variables})
var requests []*http.Request
err := parser.Parse(strings.NewReader(document), func(rr *httpTypes.RequestResponse) bool {
req, err := http.ReadRequest(bufio.NewReader(strings.NewReader(rr.Request.Raw)))
require.NoError(t, err)
requests = append(requests, req)
return false
}, "")
require.NoError(t, err)
require.Len(t, requests, tc.wantCount)
for _, req := range requests {
t.Cleanup(func() { require.NoError(t, req.Body.Close()) })
require.Equal(t, tc.wantAuth, req.Header.Get("Authorization"))
require.Equal(t, tc.wantKey, req.Header.Get("X-API-Key"))
}
})
}
}
}
}
func TestGenerateRequestsOrdinaryParameterAuthDescription(t *testing.T) {
for _, skip := range []bool{false, true} {
t.Run(fmt.Sprintf("skip=%t", skip), func(t *testing.T) {
parser := New()
parser.SetOptions(formats.InputFormatOptions{SkipFormatValidation: skip})
const document = `{
"openapi":"3.0.3","info":{"title":"Public operation","version":"1"},
"servers":[{"url":"https://example.com"}],
"paths":{"/public":{"get":{
"parameters":[{"name":"display","in":"query","required":false,"description":"globalAuth","schema":{"type":"string","default":"compact"}}],
"responses":{"200":{"description":"OK"}}
}}}
}`
count := 0
err := parser.Parse(strings.NewReader(document), func(rr *httpTypes.RequestResponse) bool {
count++
require.Equal(t, "display=compact", rr.URL.RawQuery)
return false
}, "")
require.NoError(t, err)
require.Equal(t, 1, count)
})
}
}
func TestGenerateRequestsPublicOverrideUnsupportedGlobalSecurity(t *testing.T) {
const document = `{
"openapi":"3.0.3","info":{"title":"Public override","version":"1"},
"servers":[{"url":"https://example.com"}],
"components":{"securitySchemes":{"auth":{"type":"http","scheme":"digest"}}},
"security":[{"auth":[]}],
"paths":{"/public":{"get":{"security":[],"responses":{"200":{"description":"OK"}}}}}
}`
count := 0
err := New().Parse(strings.NewReader(document), func(rr *httpTypes.RequestResponse) bool {
count++
return false
}, "")
require.NoError(t, err)
require.Equal(t, 1, count)
}
func TestSecuritySelectionMissingAndUnsupportedSchemes(t *testing.T) {
schema := &openapi3.T{Components: &openapi3.Components{SecuritySchemes: openapi3.SecuritySchemes{
"key": {Value: &openapi3.SecurityScheme{Type: "apiKey", In: "header", Name: "X-API-Key"}},
"oauth": {Value: &openapi3.SecurityScheme{Type: "oauth2"}},
}}}
for _, tc := range []struct {
name string
requirements openapi3.SecurityRequirements
wantNames []string
wantError bool
}{
{"missing combined credentials", openapi3.SecurityRequirements{{"oauth": nil, "key": nil}}, []string{"X-API-Key", "Authorization"}, false},
{"first alternative diagnostics", openapi3.SecurityRequirements{{"oauth": nil}, {"key": nil}}, []string{"Authorization"}, false},
{"unknown scheme", openapi3.SecurityRequirements{{"unknown": nil}}, nil, true},
{"known and unknown in same object", openapi3.SecurityRequirements{{"key": nil, "unknown": nil}}, nil, true},
{"known alternative after unknown", openapi3.SecurityRequirements{{"unknown": nil}, {"key": nil}}, []string{"X-API-Key"}, false},
} {
t.Run(tc.name, func(t *testing.T) {
params, err := selectSecurityParameters(schema, &tc.requirements, nil)
if tc.wantError {
require.Error(t, err)
return
}
require.NoError(t, err)
var names []string
for _, param := range params {
names = append(names, param.Value.Name)
}
require.Equal(t, tc.wantNames, names)
})
}
// Anonymous requirements do not need a components/securitySchemes section.
params, err := GetGlobalParamsForSecurityRequirement(&openapi3.T{}, &openapi3.SecurityRequirements{{}})
require.NoError(t, err)
require.Empty(t, params)
}
+9
View File
@@ -89,6 +89,12 @@ func (f *Fingerprinter) Enabled() bool {
// pattern. A value that is not a URL is always accepted: it has no path to
// group by, and guessing would lose targets.
func (f *Fingerprinter) Accept(rawURL string) bool {
return f.AcceptOn(rawURL, "")
}
// AcceptOn is Accept for a target connected to through ip. Each ip of a host
// is a server of its own, so it keeps its own count for every pattern.
func (f *Fingerprinter) AcceptOn(rawURL, ip string) bool {
if !f.Enabled() {
return true
}
@@ -96,6 +102,9 @@ func (f *Fingerprinter) Accept(rawURL string) bool {
if !ok {
return true
}
if ip != "" {
pattern += "@" + ip
}
f.mu.Lock()
defer f.mu.Unlock()
+8
View File
@@ -80,6 +80,14 @@ func TestAcceptKeepsUpToTheCap(t *testing.T) {
require.True(t, f.Accept("http://acme.test/admin"), "a different pattern has its own budget")
}
func TestAcceptOnCountsEachIPApart(t *testing.T) {
f := NewFingerprinter(1)
require.True(t, f.AcceptOn("http://acme.test/user/1", "192.0.2.10"))
require.True(t, f.AcceptOn("http://acme.test/user/1", "192.0.2.11"), "another ip is another server")
require.False(t, f.AcceptOn("http://acme.test/user/2", "192.0.2.10"), "past the cap of its ip")
require.True(t, f.Accept("http://acme.test/user/2"), "the host without an ip has its own budget")
}
// Off by default: dropping a target means never scanning it, so it has to be
// asked for.
func TestDisabledFingerprinterKeepsEverything(t *testing.T) {
+41 -9
View File
@@ -54,6 +54,7 @@ type ListInputProvider struct {
hostMapStream *filekv.FileDB
hostMapStreamOnce sync.Once
profiles *targetprofile.Registry
pinned bool
// normalizeURLs collapses targets that address the same resource onto one
// form before they are deduplicated.
normalizeURLs bool
@@ -205,7 +206,8 @@ func (i *ListInputProvider) set(executionId string, value string, selection *tar
// scan all ips
dialers := protocolstate.GetDialersWithId(executionId)
if dialers == nil {
panic("dialers with executionId " + executionId + " not found")
gologger.Error().Msgf("dialers with executionId %s not found", executionId)
return
}
dnsData, err := dialers.Fastdialer.GetDNSData(urlx.Hostname())
@@ -242,7 +244,8 @@ func (i *ListInputProvider) set(executionId string, value string, selection *tar
if i.ipOptions.IPV6 {
dialers := protocolstate.GetDialersWithId(executionId)
if dialers == nil {
panic("dialers with executionId " + executionId + " not found")
gologger.Error().Msgf("dialers with executionId %s not found", executionId)
return
}
dnsData, err := dialers.Fastdialer.GetDNSData(urlx.Hostname())
@@ -403,11 +406,15 @@ func (i *ListInputProvider) scanInputFromReader(executionId string, reader io.Re
func (i *ListInputProvider) addInput(executionId string, item string) error {
var selection *targetprofile.Selection
if targetprofile.IsLine(item) {
target, parsed, err := i.profiles.ParseLine(item)
line, err := i.profiles.ParseLine(item)
if err != nil {
return errors.Wrap(err, "could not parse target line")
}
item, selection = target, parsed
if line.IP != "" {
i.setPinned(executionId, line)
return nil
}
item, selection = line.Target, line.Selection
}
switch {
@@ -421,6 +428,29 @@ func (i *ListInputProvider) addInput(executionId string, item string) error {
return nil
}
// setPinned stores a target pinned to an ip. Its hostname is kept for the Host
// header, SNI and template variables, and connections go to the ip, so the ip
// is checked against the network policy (-eh, -lna) here.
func (i *ListInputProvider) setPinned(executionId string, line targetprofile.Line) {
if dialers := protocolstate.GetDialersWithId(executionId); dialers != nil && dialers.NetworkPolicy != nil {
if !dialers.NetworkPolicy.Validate(line.IP) {
gologger.Warning().Msgf("Skipping %s: ip %s is not allowed by the network policy", line.Target, line.IP)
i.skippedCount++
return
}
}
metaInput := contextargs.NewMetaInput()
metaInput.Input = line.Target
metaInput.CustomIP = line.IP
i.setItem(metaInput, line.Selection)
i.pinned = true
}
// PinnedTargets reports whether any target was pinned to an ip.
func (i *ListInputProvider) PinnedTargets() bool {
return i.pinned
}
// isExcluded checks if a URL is in the exclusion list
func (i *ListInputProvider) isExcluded(URL string) bool {
metaInput := contextargs.NewMetaInput()
@@ -467,7 +497,8 @@ func (i *ListInputProvider) Del(executionId string, value string) {
// scan all ips
dialers := protocolstate.GetDialersWithId(executionId)
if dialers == nil {
panic("dialers with executionId " + executionId + " not found")
gologger.Error().Msgf("dialers with executionId %s not found", executionId)
return
}
dnsData, err := dialers.Fastdialer.GetDNSData(urlx.Hostname())
@@ -504,7 +535,8 @@ func (i *ListInputProvider) Del(executionId string, value string) {
if i.ipOptions.IPV6 {
dialers := protocolstate.GetDialersWithId(executionId)
if dialers == nil {
panic("dialers with executionId " + executionId + " not found")
gologger.Error().Msgf("dialers with executionId %s not found", executionId)
return
}
dnsData, err := dialers.Fastdialer.GetDNSData(urlx.Hostname())
@@ -556,19 +588,19 @@ func (i *ListInputProvider) setItem(metaInput *contextargs.MetaInput, selection
if _, ok := i.hostMap.Get(dedupKey); ok {
i.dupeCount++
if i.profiles != nil {
i.profiles.Merge(metaInput.Input, selection)
i.profiles.Merge(metaInput, selection)
}
return
}
// Past the per-pattern cap the shape is already covered by the targets kept
// for it, so scanning another of the same shape buys nothing.
if i.patterns != nil && !i.patterns.Accept(metaInput.Input) {
if i.patterns != nil && !i.patterns.AcceptOn(metaInput.Input, metaInput.CustomIP) {
i.patternSkippedCount++
return
}
if i.profiles != nil {
i.profiles.Bind(metaInput.Input, selection)
i.profiles.Bind(metaInput, selection)
}
i.inputCount++ // tracks target count
// the entry is keyed by the collapsed form and carries the original, which
+18
View File
@@ -242,6 +242,24 @@ func Test_scanallips_normalizeStoreInputValue(t *testing.T) {
require.ElementsMatchf(t, tt.expected, got, "could not get correct ips for hostname %v", tt.hostname)
input.Close()
}
// -mup 1 keeps the url on each ip, as every ip is a server of its own
hm, err := hybrid.New(hybrid.DefaultDiskOptions)
require.NoError(t, err)
input := &ListInputProvider{
hostMap: hm,
ipOptions: &ipOptions{ScanAllIPs: true, IPV4: true, IPV6: true},
patterns: normalize.NewFingerprinter(1),
}
defer input.Close()
input.Set(defaultOpts.ExecutionId, "https://scanme.sh/user/1")
input.Set(defaultOpts.ExecutionId, "https://scanme.sh/user/2")
var got []string
input.Iterate(func(value *contextargs.MetaInput) bool {
got = append(got, value.Input+" "+value.CustomIP)
return true
})
require.ElementsMatch(t, []string{"https://scanme.sh/user/1 128.199.158.128", "https://scanme.sh/user/1 2400:6180:0:d0::91:1001"}, got)
}
func Test_expandASNInputValue(t *testing.T) {
@@ -5,8 +5,11 @@ import (
"testing"
"github.com/projectdiscovery/hmap/store/hybrid"
"github.com/projectdiscovery/nuclei/v3/pkg/input/normalize"
"github.com/projectdiscovery/nuclei/v3/pkg/input/targetprofile"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/stretchr/testify/require"
)
@@ -34,3 +37,74 @@ func TestTargetLines(t *testing.T) {
err = input.scanInputFromReader("", strings.NewReader(`{"target": "192.0.2.30", "tagz": ["x"]}`))
require.ErrorContains(t, err, "unknown keys tagz")
}
func TestPinnedTargetLines(t *testing.T) {
hm, err := hybrid.New(hybrid.DefaultDiskOptions)
require.NoError(t, err)
// pinned lines never resolve the target, even with -sa
input := &ListInputProvider{hostMap: hm, ipOptions: &ipOptions{ScanAllIPs: true, IPV4: true}, profiles: targetprofile.NewRegistry(t.TempDir())}
defer input.Close()
lines := strings.Join([]string{
`{"target": "https://a.example", "ip": "192.0.2.10", "tags": ["thinkphp"]}`,
`{"target": "https://a.example", "ip": "192.0.2.11"}`,
`{"target": "https://a.example", "ip": "::ffff:192.0.2.11"}`,
}, "\n")
require.NoError(t, input.scanInputFromReader("", strings.NewReader(lines)))
require.True(t, input.PinnedTargets())
require.EqualValues(t, 2, input.Count(), "the IPv4-mapped ip is a duplicate")
var got []string
input.Iterate(func(value *contextargs.MetaInput) bool {
got = append(got, value.Input+" "+value.CustomIP)
return true
})
require.ElementsMatch(t, []string{"https://a.example 192.0.2.10", "https://a.example 192.0.2.11"}, got)
require.NotNil(t, input.profiles.For(&contextargs.MetaInput{Input: "https://a.example", CustomIP: "192.0.2.10"}))
require.Nil(t, input.profiles.For(&contextargs.MetaInput{Input: "https://a.example", CustomIP: "192.0.2.11"}))
}
func TestPinnedTargetLinesKeepEveryIPUnderPatternCap(t *testing.T) {
hm, err := hybrid.New(hybrid.DefaultDiskOptions)
require.NoError(t, err)
// -mup 1
input := &ListInputProvider{hostMap: hm, ipOptions: &ipOptions{IPV4: true}, profiles: targetprofile.NewRegistry(t.TempDir()), patterns: normalize.NewFingerprinter(1)}
defer input.Close()
lines := strings.Join([]string{
`{"target": "https://a.example/user/1", "ip": "192.0.2.10"}`,
`{"target": "https://a.example/user/1", "ip": "192.0.2.11"}`,
`{"target": "https://a.example/user/2", "ip": "192.0.2.10"}`,
}, "\n")
require.NoError(t, input.scanInputFromReader("", strings.NewReader(lines)))
var got []string
input.Iterate(func(value *contextargs.MetaInput) bool {
got = append(got, value.Input+" "+value.CustomIP)
return true
})
require.ElementsMatch(t, []string{"https://a.example/user/1 192.0.2.10", "https://a.example/user/1 192.0.2.11"}, got)
require.EqualValues(t, 1, input.patternSkippedCount, "user/2 has the shape of user/1 on the same ip")
}
func TestPinnedTargetLinesFollowNetworkPolicy(t *testing.T) {
options := types.DefaultOptions()
options.ExecutionId = "pinned-target-lines"
options.ExcludeTargets = []string{"192.0.2.0/28"}
require.NoError(t, protocolstate.Init(options))
t.Cleanup(func() { protocolstate.Close(options.ExecutionId) })
hm, err := hybrid.New(hybrid.DefaultDiskOptions)
require.NoError(t, err)
input := &ListInputProvider{hostMap: hm, ipOptions: &ipOptions{IPV4: true}, profiles: targetprofile.NewRegistry(t.TempDir())}
defer input.Close()
lines := strings.Join([]string{
`{"target": "https://a.example", "ip": "192.0.2.10"}`,
`{"target": "https://a.example", "ip": "192.0.2.20"}`,
}, "\n")
require.NoError(t, input.scanInputFromReader(options.ExecutionId, strings.NewReader(lines)))
require.EqualValues(t, 1, input.Count())
require.EqualValues(t, 1, input.skippedCount, "the excluded ip is skipped")
}
+46 -20
View File
@@ -22,7 +22,7 @@ type Registry struct {
templatesDir string
mu sync.RWMutex
targets map[string]*Selection
targets map[targetKey]*Selection
selections map[string]*Selection
rules map[string]*rule
profiles map[string]selector
@@ -38,23 +38,47 @@ var _ protocols.TargetScope = (*Registry)(nil)
func NewRegistry(templatesDir string) *Registry {
return &Registry{
templatesDir: templatesDir,
targets: make(map[string]*Selection),
targets: make(map[targetKey]*Selection),
selections: make(map[string]*Selection),
rules: make(map[string]*rule),
profiles: make(map[string]selector),
}
}
// ParseLine parses a target line into its target and selection.
func (r *Registry) ParseLine(line string) (string, *Selection, error) {
target, profile, sel, err := parseLine(line)
// Line is a parsed target line.
type Line struct {
Target string
// IP is the address the target is pinned to, empty when it is not.
IP string
// Selection is nil when the line selects every template.
Selection *Selection
}
// targetKey identifies a stored input: the same target pinned to different
// ips is scanned, and selected, separately.
type targetKey struct {
input string
ip string
}
func keyOf(input *contextargs.MetaInput) targetKey {
return targetKey{input: input.Input, ip: input.CustomIP}
}
// ParseLine parses a target line into its target, ip and selection.
func (r *Registry) ParseLine(line string) (Line, error) {
target, profile, ip, sel, err := parseLine(line)
if err != nil {
return "", nil, err
return Line{}, err
}
if profile == "" && len(sel) == 0 {
// nothing narrows the templates, e.g. a line that only sets an ip
return Line{Target: target, IP: ip}, nil
}
if profile != "" {
base, err := r.profile(profile)
if err != nil {
return "", nil, fmt.Errorf("target %q: %w", target, err)
return Line{}, fmt.Errorf("target %q: %w", target, err)
}
sel = base.merge(sel)
}
@@ -64,48 +88,50 @@ func (r *Registry) ParseLine(line string) (string, *Selection, error) {
compiled, err := r.internRule(sel)
if err != nil {
return "", nil, fmt.Errorf("target %q: %w", target, err)
return Line{}, fmt.Errorf("target %q: %w", target, err)
}
return target, r.internSelection([]*rule{compiled}), nil
return Line{Target: target, IP: ip, Selection: r.internSelection([]*rule{compiled})}, nil
}
// Bind records the selection of a newly stored input; nil means unrestricted.
func (r *Registry) Bind(input string, selection *Selection) {
func (r *Registry) Bind(input *contextargs.MetaInput, selection *Selection) {
r.mu.Lock()
defer r.mu.Unlock()
key := keyOf(input)
if selection == nil {
r.unrestricted = true
if _, ok := r.targets[input]; ok {
r.targets[input] = nil
if _, ok := r.targets[key]; ok {
r.targets[key] = nil
}
return
}
current, ok := r.targets[input]
current, ok := r.targets[key]
switch {
case !ok:
r.targets[input] = selection
r.targets[key] = selection
case current != nil:
r.targets[input] = r.union(current, selection)
r.targets[key] = r.union(current, selection)
}
}
// Merge records the selection of an input that was already stored, so a
// target listed more than once runs the union of its selections and is
// scanned once.
func (r *Registry) Merge(input string, selection *Selection) {
func (r *Registry) Merge(input *contextargs.MetaInput, selection *Selection) {
r.mu.Lock()
defer r.mu.Unlock()
current, ok := r.targets[input]
key := keyOf(input)
current, ok := r.targets[key]
switch {
case !ok || current == nil:
// already unrestricted
case selection == nil:
r.unrestricted = true
r.targets[input] = nil
r.targets[key] = nil
default:
r.targets[input] = r.union(current, selection)
r.targets[key] = r.union(current, selection)
}
}
@@ -153,7 +179,7 @@ func (r *Registry) Prepare(tpls []*templates.Template) {
// For returns the selection of input, or nil when every template applies.
func (r *Registry) For(input *contextargs.MetaInput) protocols.TemplateSelection {
r.mu.RLock()
selection := r.targets[input.Input]
selection := r.targets[keyOf(input)]
r.mu.RUnlock()
// a nil *Selection must not become a non-nil interface
+78 -25
View File
@@ -49,10 +49,14 @@ func selected(t *testing.T, r *Registry, input string) []string {
func bindLine(t *testing.T, r *Registry, line string) string {
t.Helper()
target, selection, err := r.ParseLine(line)
parsed, err := r.ParseLine(line)
require.NoError(t, err)
r.Bind(target, selection)
return target
r.Bind(&contextargs.MetaInput{Input: parsed.Target, CustomIP: parsed.IP}, parsed.Selection)
return parsed.Target
}
func input(target string) *contextargs.MetaInput {
return &contextargs.MetaInput{Input: target}
}
func TestParseLineRejectsInvalidLines(t *testing.T) {
@@ -65,14 +69,62 @@ func TestParseLineRejectsInvalidLines(t *testing.T) {
"invalid type": `{"target": "a.com", "type": ["smtp"]}`,
"non string value": `{"target": "a.com", "tags": [1]}`,
"missing profile": `{"target": "a.com", "profile": "does-not-exist"}`,
"non string ip": `{"target": "a.com", "ip": 10}`,
"invalid ip": `{"target": "a.com", "ip": "10.0.0"}`,
"ip with zone": `{"target": "a.com", "ip": "fe80::1%eth0"}`,
"unspecified ip": `{"target": "a.com", "ip": "0.0.0.0"}`,
"ip target": `{"target": "https://192.0.2.1", "ip": "192.0.2.10"}`,
"cidr target": `{"target": "192.0.2.0/24", "ip": "192.0.2.10"}`,
"asn target": `{"target": "AS13335", "ip": "192.0.2.10"}`,
} {
t.Run(name, func(t *testing.T) {
_, _, err := r.ParseLine(line)
_, err := r.ParseLine(line)
require.Error(t, err)
})
}
}
func TestParseLineIP(t *testing.T) {
r := NewRegistry(templatesDir)
for line, want := range map[string]Line{
`{"target": "https://a.com", "ip": "192.0.2.10"}`: {Target: "https://a.com", IP: "192.0.2.10"},
`{"target": "a.com:8443", "ip": " 2001:DB8::1 "}`: {Target: "a.com:8443", IP: "2001:db8::1"},
`{"target": "a.com", "ip": "::ffff:192.0.2.10"}`: {Target: "a.com", IP: "192.0.2.10"},
`{"target": "a.com"}`: {Target: "a.com"},
`{"target": "https://a.com", "ip": "192.0.2.10", "tags": []}`: {Target: "https://a.com", IP: "192.0.2.10"},
} {
t.Run(line, func(t *testing.T) {
parsed, err := r.ParseLine(line)
require.NoError(t, err)
require.Equal(t, want, parsed)
})
}
require.False(t, r.Scoped(), "lines that only set an ip do not scope the scan")
require.Nil(t, r.LoadFilter())
}
func TestPinnedTargetsKeepTheirSelection(t *testing.T) {
r := NewRegistry(templatesDir)
bindLine(t, r, `{"target": "https://a.com", "ip": "192.0.2.10", "tags": ["thinkphp"]}`)
bindLine(t, r, `{"target": "https://a.com", "ip": "192.0.2.11", "tags": ["tomcat"]}`)
bindLine(t, r, `{"target": "https://a.com", "ip": "192.0.2.12"}`)
r.Prepare(allTemplates)
selectedFor := func(ip string) []string {
selection := r.For(&contextargs.MetaInput{Input: "https://a.com", CustomIP: ip})
var ids []string
for _, tpl := range allTemplates {
if selection == nil || selection.Allows(tpl.Path) {
ids = append(ids, tpl.ID)
}
}
return ids
}
require.Equal(t, []string{"thinkphp-rce"}, selectedFor("192.0.2.10"))
require.Equal(t, []string{"tomcat-panel", "CVE-2020-1938"}, selectedFor("192.0.2.11"))
require.Equal(t, []string{"thinkphp-rce", "tomcat-panel", "CVE-2020-1938", "git-config"}, selectedFor("192.0.2.12"))
}
func TestProfileMustBeAnID(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "profiles"), 0o755))
@@ -85,14 +137,14 @@ func TestProfileMustBeAnID(t *testing.T) {
t.Run(profile, func(t *testing.T) {
line, err := json.Marshal(map[string]string{"target": "a.com", "profile": profile})
require.NoError(t, err)
_, _, err = r.ParseLine(string(line))
_, err = r.ParseLine(string(line))
require.ErrorContains(t, err, "profile ID")
})
}
_, selection, err := r.ParseLine(`{"target": "a.com", "profile": "php"}`)
parsed, err := r.ParseLine(`{"target": "a.com", "profile": "php"}`)
require.NoError(t, err)
require.Equal(t, []string{"thinkphp"}, selection.rules[0].filter.Tags)
require.Equal(t, []string{"thinkphp"}, parsed.Selection.rules[0].filter.Tags)
}
func TestSelectionFilters(t *testing.T) {
@@ -129,37 +181,38 @@ func TestProfileWithInlineOverride(t *testing.T) {
require.NoError(t, os.WriteFile(filepath.Join(dir, "profiles", "php.yml"), []byte(profile), 0o600))
r := NewRegistry(dir)
target, selection, err := r.ParseLine(`{"target": "https://a.com", "profile": "php", "severity": "critical"}`)
parsed, err := r.ParseLine(`{"target": "https://a.com", "profile": "php", "severity": "critical"}`)
require.NoError(t, err)
require.Len(t, selection.rules, 1)
require.Equal(t, []string{"thinkphp"}, selection.rules[0].filter.Tags)
require.Equal(t, []severity.Severity{severity.Critical}, selection.rules[0].filter.Severities)
require.Equal(t, "https://a.com", target)
require.Len(t, parsed.Selection.rules, 1)
require.Equal(t, []string{"thinkphp"}, parsed.Selection.rules[0].filter.Tags)
require.Equal(t, []severity.Severity{severity.Critical}, parsed.Selection.rules[0].filter.Severities)
require.Equal(t, "https://a.com", parsed.Target)
}
func TestDuplicateTargetsMerge(t *testing.T) {
r := NewRegistry(templatesDir)
_, php, err := r.ParseLine(`{"target": "https://a.com", "tags": ["thinkphp"]}`)
phpLine, err := r.ParseLine(`{"target": "https://a.com", "tags": ["thinkphp"]}`)
require.NoError(t, err)
_, tomcat, err := r.ParseLine(`{"target": "https://a.com", "tags": ["tomcat"]}`)
tomcatLine, err := r.ParseLine(`{"target": "https://a.com", "tags": ["tomcat"]}`)
require.NoError(t, err)
php, tomcat := phpLine.Selection, tomcatLine.Selection
r.Bind("https://a.com", php)
r.Merge("https://a.com", tomcat)
r.Merge("https://a.com", php)
r.Bind("https://b.com", php)
r.Merge("https://b.com", nil)
r.Bind("https://c.com", nil)
r.Merge("https://c.com", php)
r.Bind(input("https://a.com"), php)
r.Merge(input("https://a.com"), tomcat)
r.Merge(input("https://a.com"), php)
r.Bind(input("https://b.com"), php)
r.Merge(input("https://b.com"), nil)
r.Bind(input("https://c.com"), nil)
r.Merge(input("https://c.com"), php)
r.Prepare(allTemplates)
require.Equal(t, []string{"thinkphp-rce", "tomcat-panel", "CVE-2020-1938"}, selected(t, r, "https://a.com"))
require.Nil(t, r.For(&contextargs.MetaInput{Input: "https://b.com"}), "a plain duplicate lifts the restriction")
require.Nil(t, r.For(&contextargs.MetaInput{Input: "https://c.com"}), "a plain target stays unrestricted")
_, again, err := r.ParseLine(`{"target": "https://z.com", "tags": ["thinkphp"]}`)
again, err := r.ParseLine(`{"target": "https://z.com", "tags": ["thinkphp"]}`)
require.NoError(t, err)
require.Same(t, php, again, "identical selectors are interned")
require.Same(t, php, again.Selection, "identical selectors are interned")
}
func TestLoadFilter(t *testing.T) {
@@ -183,14 +236,14 @@ func TestLoadFilter(t *testing.T) {
t.Run("unrestricted target loads everything", func(t *testing.T) {
r := NewRegistry(templatesDir)
bindLine(t, r, `{"target": "https://a.com", "tags": ["thinkphp"]}`)
r.Bind("https://b.com", nil)
r.Bind(input("https://b.com"), nil)
require.Nil(t, r.LoadFilter())
})
t.Run("plain duplicate of a target line loads everything", func(t *testing.T) {
r := NewRegistry(templatesDir)
target := bindLine(t, r, `{"target": "https://a.com", "tags": ["thinkphp"]}`)
r.Merge(target, nil)
r.Merge(input(target), nil)
require.Nil(t, r.LoadFilter())
})
}
+44 -9
View File
@@ -2,27 +2,34 @@
// single scan can run a different profile on each target.
//
// A target line is a JSON object whose keys mirror template profiles and CLI
// flags:
// flags, and may pin the target to an ip:
//
// {"target": "https://a.example", "tags": ["thinkphp"], "tech": ["php"]}
// {"target": "https://b.example", "profile": "wordpress", "severity": ["high"]}
// {"target": "https://c.example", "ip": "192.0.2.10"}
package targetprofile
import (
"errors"
"fmt"
"net/netip"
"os"
"path/filepath"
"slices"
"strings"
"github.com/projectdiscovery/mapcidr/asn"
"github.com/projectdiscovery/nuclei/v3/internal/configuration"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/json"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/yaml"
iputil "github.com/projectdiscovery/utils/ip"
urlutil "github.com/projectdiscovery/utils/url"
)
const (
keyTarget = "target"
keyProfile = "profile"
keyIP = "ip"
keyTemplates = "templates"
keyTags = "tags"
@@ -54,36 +61,64 @@ func IsLine(line string) bool {
// parseLine decodes a target line. Unknown keys are rejected so that a typo
// cannot silently widen the scan to every template.
func parseLine(line string) (target, profile string, sel selector, err error) {
func parseLine(line string) (target, profile, ip string, sel selector, err error) {
var raw map[string]any
if err := json.Unmarshal([]byte(line), &raw); err != nil {
return "", "", nil, fmt.Errorf("invalid target line %q: %w", line, err)
return "", "", "", nil, fmt.Errorf("invalid target line %q: %w", line, err)
}
target, err = stringValue(raw, keyTarget)
if err != nil {
return "", "", nil, err
return "", "", "", nil, err
}
target = strings.TrimSpace(target)
if target == "" {
return "", "", nil, fmt.Errorf("target line %q has no %q", line, keyTarget)
return "", "", "", nil, fmt.Errorf("target line %q has no %q", line, keyTarget)
}
profile, err = stringValue(raw, keyProfile)
if err != nil {
return "", "", nil, err
return "", "", "", nil, err
}
ip, err = stringValue(raw, keyIP)
if err != nil {
return "", "", "", nil, err
}
if ip != "" {
if ip, err = pinnedIP(target, ip); err != nil {
return "", "", "", nil, fmt.Errorf("target %q: %w", target, err)
}
}
delete(raw, keyTarget)
delete(raw, keyProfile)
delete(raw, keyIP)
sel, ignored, err := toSelector(raw)
if err != nil {
return "", "", nil, fmt.Errorf("target %q: %w", target, err)
return "", "", "", nil, fmt.Errorf("target %q: %w", target, err)
}
if len(ignored) > 0 {
return "", "", nil, fmt.Errorf("target %q: unknown keys %s (supported: target, profile, %s)", target, strings.Join(ignored, ", "), strings.Join(selectorKeys, ", "))
return "", "", "", nil, fmt.Errorf("target %q: unknown keys %s (supported: target, ip, profile, %s)", target, strings.Join(ignored, ", "), strings.Join(selectorKeys, ", "))
}
return target, profile, sel, nil
return target, profile, ip, sel, nil
}
// pinnedIP validates the ip a target is pinned to and returns it in canonical
// form. Only a target with a hostname can be pinned: the hostname is kept for
// the Host header, SNI and template variables while connections go to the ip.
func pinnedIP(target, value string) (string, error) {
addr, err := netip.ParseAddr(strings.TrimSpace(value))
if err != nil || addr.Zone() != "" || addr.IsUnspecified() {
return "", fmt.Errorf("invalid ip %q", value)
}
if iputil.IsCIDR(target) || asn.IsASN(target) {
return "", errors.New("a CIDR or ASN target cannot have an ip")
}
parsed, err := urlutil.Parse(target)
if err != nil || parsed.Hostname() == "" || iputil.IsIP(parsed.Hostname()) {
return "", errors.New("a target with an ip must have a hostname")
}
return addr.Unmap().String(), nil
}
// loadProfile reads the selection keys of a template profile. Keys that
+26
View File
@@ -53,6 +53,32 @@ func (h *Helper) Transform(input string, protocol templateTypes.ProtocolType) st
return input
}
// TransformPinned is Transform for an input pinned to ip, which takes the
// HTTP probe result made through that ip and never the result of the input
// without an ip, as that one may come from another server.
func (h *Helper) TransformPinned(input, ip string, protocol templateTypes.ProtocolType) string {
if ip == "" || (protocol != templateTypes.HTTPProtocol && protocol != templateTypes.HeadlessProtocol) {
return h.Transform(input, protocol)
}
if h.InputsHTTP != nil {
if probed, ok := h.InputsHTTP.Get(ProbeKey(input, ip)); ok {
return string(probed)
}
}
// no probe through this ip: convert it as an input that was not probed
return (&Helper{}).Transform(input, protocol)
}
// ProbeKey returns the key of the HTTP probe result of input in InputsHTTP.
// A target pinned to an ip is probed through that ip, so each ip has its own.
// The separator cannot occur in an input, which may hold an "@" of its own.
func ProbeKey(input, ip string) string {
if ip == "" {
return input
}
return input + "\x00" + ip
}
type inputType int
const (
+19
View File
@@ -4,9 +4,28 @@ import (
"testing"
"github.com/projectdiscovery/hmap/store/hybrid"
templateTypes "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
"github.com/stretchr/testify/require"
)
func TestTransformPinned(t *testing.T) {
hm, err := hybrid.New(hybrid.DefaultDiskOptions)
require.NoError(t, err, "could not create hybrid map")
defer func() {
_ = hm.Close()
}()
helper := &Helper{InputsHTTP: hm}
_ = hm.Set(ProbeKey("app.test", ""), []byte("http://app.test"))
_ = hm.Set(ProbeKey("app.test", "127.0.0.2"), []byte("https://app.test"))
require.Equal(t, "https://app.test", helper.TransformPinned("app.test", "127.0.0.2", templateTypes.HTTPProtocol), "the probe made through the pinned ip")
require.Equal(t, "http://app.test", helper.TransformPinned("app.test", "", templateTypes.HTTPProtocol), "an unpinned input")
require.Equal(t, "app.test", helper.TransformPinned("app.test", "127.0.0.3", templateTypes.HTTPProtocol), "an ip without a probe of its own does not take the unpinned result")
require.Equal(t, "app.test", helper.TransformPinned("app.test", "127.0.0.2", templateTypes.DNSProtocol), "a protocol that is not probed")
require.NotEqual(t, ProbeKey("[email protected]", ""), ProbeKey("u.example", "192.0.2.10"), "an input holding an @ does not collide with a pinned one")
}
func TestConvertInputToType(t *testing.T) {
helper := &Helper{}
+19 -8
View File
@@ -112,16 +112,27 @@ func (rr *RequestResponse) ID() string {
func (rr *RequestResponse) MarshalJSON() ([]byte, error) {
m := make(map[string]interface{})
m["url"] = rr.URL.String()
reqBin, err := json.Marshal(rr.Request)
if err != nil {
return nil, err
// Request and Response are optional (UnmarshalJSON only reads the keys when
// present), so nil pointers omit their keys instead of emitting "null".
//
// The pre-marshaled bytes must be wrapped as json.Message: encoding/json
// (and sonic) encode a bare []byte as a base64 *string*, which this type's
// own UnmarshalJSON — expecting a JSON object — cannot decode, breaking the
// json.JSONCodec round-trip contract asserted above.
if rr.Request != nil {
reqBin, err := json.Marshal(rr.Request)
if err != nil {
return nil, err
}
m["request"] = json.Message(reqBin)
}
m["request"] = reqBin
respBin, err := json.Marshal(rr.Response)
if err != nil {
return nil, err
if rr.Response != nil {
respBin, err := json.Marshal(rr.Response)
if err != nil {
return nil, err
}
m["response"] = json.Message(respBin)
}
m["response"] = respBin
return json.Marshal(m)
}
+64
View File
@@ -0,0 +1,64 @@
package types
import (
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/json"
"github.com/stretchr/testify/require"
)
// TestRequestResponseJSONRoundTrip pins the json.JSONCodec contract asserted
// on RequestResponse: MarshalJSON output must be decodable by the type's own
// UnmarshalJSON.
//
// It previously failed because MarshalJSON put the pre-marshaled request and
// response into a map[string]interface{} as []byte, which encoding/json (and
// sonic) encode as base64 *strings* - while UnmarshalJSON expects JSON
// objects:
//
// {"request":"eyJtZXRob2QiOiJHRVQifQ==", ...} -> mismatched type string
//
// MetaInput serializes a captured request as the "raw-request" key with this
// codec, so the broken round-trip surfaced through MetaInput.MarshalString ->
// MetaInput.Unmarshal (pkg/input/provider/list/hmap.go) as a decode error.
func TestRequestResponseJSONRoundTrip(t *testing.T) {
rr, err := ParseRawRequest("GET /path?q=1 HTTP/1.1\r\nHost: example.com\r\nX-Test: hello\r\n\r\nbody")
require.NoError(t, err)
rr.Response = &HttpResponse{
StatusCode: 200,
Body: "ok",
Raw: "HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok",
}
encoded, err := json.Marshal(rr)
require.NoError(t, err)
var decoded RequestResponse
require.NoError(t, json.Unmarshal(encoded, &decoded), "marshal/unmarshal round-trip must succeed, got %s", encoded)
require.Equal(t, rr.URL.String(), decoded.URL.String())
require.NotNil(t, decoded.Request)
require.Equal(t, rr.Request.Method, decoded.Request.Method)
require.Equal(t, rr.Request.Raw, decoded.Request.Raw)
require.Equal(t, rr.Request.Body, decoded.Request.Body)
require.NotNil(t, decoded.Response)
require.Equal(t, rr.Response.StatusCode, decoded.Response.StatusCode)
require.Equal(t, rr.Response.Raw, decoded.Response.Raw)
}
// TestRequestResponseMarshalOmitsNilParts pins that nil Request/Response stay
// nil across the round-trip instead of turning into zero-valued structs (or a
// decode error, which is what the previous base64 "null" encoding produced).
func TestRequestResponseMarshalOmitsNilParts(t *testing.T) {
rr, err := ParseRawRequest("GET / HTTP/1.1\r\nHost: example.com\r\n\r\n")
require.NoError(t, err)
require.Nil(t, rr.Response)
encoded, err := json.Marshal(rr)
require.NoError(t, err)
var decoded RequestResponse
require.NoError(t, json.Unmarshal(encoded, &decoded), "got %s", encoded)
require.NotNil(t, decoded.Request)
require.Nil(t, decoded.Response)
}
+91
View File
@@ -1,10 +1,28 @@
package installer
import (
"context"
"fmt"
"net/http"
"os"
"strings"
"github.com/google/go-github/v92/github"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
updateutils "github.com/projectdiscovery/utils/update"
)
const (
// TemplatesChannelStable updates nuclei-templates to the latest release.
TemplatesChannelStable = "stable"
// TemplatesChannelLatest updates nuclei-templates to the head of the
// default branch, including templates merged since the latest release.
TemplatesChannelLatest = "latest"
templatesDefaultBranch = "main"
shortCommitLength = 7
)
// templateRelease is a published release of the official nuclei-templates.
type templateRelease interface {
Version() string
@@ -30,3 +48,76 @@ func latestGitHubTemplateRelease() (templateRelease, error) {
}
return githubTemplateRelease{downloader}, nil
}
// githubTemplateHead is the head commit of the official nuclei-templates
// default branch.
type githubTemplateHead struct {
release templateRelease // latest release, the commit is ahead of it
commit string
fetchArchive templateArchiveFetcher
}
// Version returns the latest release with the short commit as semver build
// metadata (e.g. v10.3.1+3de629d). Build metadata is ignored when comparing
// versions, so it is only outdated once a newer release is published.
func (h githubTemplateHead) Version() string {
return h.release.Version() + "+" + h.commit[:min(len(h.commit), shortCommitLength)]
}
// Changelog is empty since unreleased commits have no release notes.
func (h githubTemplateHead) Changelog() string { return "" }
// DownloadSource downloads the archive of the head commit and calls callback
// for every file in it.
func (h githubTemplateHead) DownloadSource(_ bool, callback updateutils.AssetFileCallback) error {
archive, err := h.fetchArchive(h.commit)
if err != nil {
return err
}
return updateutils.UnpackAssetWithCallback(updateutils.Zip, archive, callback)
}
// latestGitHubTemplateHead returns the current head of the nuclei-templates
// default branch, based on the latest release.
func latestGitHubTemplateHead() (templateRelease, error) {
release, err := latestGitHubTemplateRelease()
if err != nil {
return nil, err
}
client, err := newGitHubClient()
if err != nil {
return nil, err
}
commit, _, err := client.Repositories.GetCommitSHA1(context.Background(), updateutils.Organization, config.OfficialNucleiTemplatesRepoName, templatesDefaultBranch, "")
if err != nil {
return nil, fmt.Errorf("get nuclei-templates %s head: %w", templatesDefaultBranch, err)
}
return githubTemplateHead{release: release, commit: commit, fetchArchive: fetchTemplateArchive}, nil
}
// templateVersionRef returns the git ref of an installed templates version.
// Versions installed from the latest channel carry their commit as build
// metadata.
func templateVersionRef(version string) string {
if _, commit, ok := strings.Cut(version, "+"); ok && commit != "" {
return commit
}
return version
}
// newGitHubClient returns a GitHub client that uses GITHUB_TOKEN when set,
// like the release downloader does.
func newGitHubClient() (*github.Client, error) {
opts := []github.ClientOptionsFunc{github.WithHTTPClient(&http.Client{Timeout: updateutils.DownloadUpdateTimeout})}
if token := os.Getenv("GITHUB_TOKEN"); token != "" {
opts = append(opts, github.WithAuthToken(token))
}
client, err := github.NewClient(opts...)
if err != nil {
return nil, fmt.Errorf("create github client: %w", err)
}
return client, nil
}
+55
View File
@@ -1,14 +1,19 @@
package installer
import (
"bytes"
"io"
"io/fs"
"maps"
"path/filepath"
"slices"
"strings"
"testing"
"time"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
updateutils "github.com/projectdiscovery/utils/update"
"github.com/stretchr/testify/require"
)
// fakeTemplateRelease is an in-memory nuclei-templates release, mapping paths
@@ -35,6 +40,19 @@ func (r fakeTemplateRelease) DownloadSource(_ bool, callback updateutils.AssetFi
return nil
}
// installedTemplateRelease is a release that is already installed, so
// downloading it fails the test.
type installedTemplateRelease struct {
fakeTemplateRelease
t *testing.T
}
// DownloadSource fails the test since the release is already installed.
func (r installedTemplateRelease) DownloadSource(bool, updateutils.AssetFileCallback) error {
r.t.Fatalf("downloaded already installed templates %s", r.version)
return nil
}
type fakeFileInfo struct {
name string
size int64
@@ -46,3 +64,40 @@ func (f fakeFileInfo) Mode() fs.FileMode { return 0o644 }
func (f fakeFileInfo) ModTime() time.Time { return time.Time{} }
func (f fakeFileInfo) IsDir() bool { return false }
func (f fakeFileInfo) Sys() any { return nil }
// TestGithubTemplateHead checks the version of the head and that it downloads
// the exact head commit.
func TestGithubTemplateHead(t *testing.T) {
commit := "3de629d823bb6eb00aa3769edc365b6525134fb1"
entry := "projectdiscovery-nuclei-templates-3de629d/http/head.yaml"
var fetchedRef string
head := githubTemplateHead{
release: fakeTemplateRelease{version: "v10.3.1"},
commit: commit,
fetchArchive: func(ref string) (*bytes.Reader, error) {
fetchedRef = ref
return newTemplateReleaseArchive(t, entry, []byte("id: head")), nil
},
}
require.Equal(t, "v10.3.1+3de629d", head.Version())
// build metadata is ignored, so only a newer release outdates the head
require.False(t, config.IsOutdatedVersion(head.Version(), "v10.3.1"))
require.True(t, config.IsOutdatedVersion(head.Version(), "v10.3.2"))
var entries []string
require.NoError(t, head.DownloadSource(false, func(uri string, _ fs.FileInfo, _ io.Reader) error {
entries = append(entries, uri)
return nil
}))
require.Equal(t, commit, fetchedRef, "should download the exact head commit")
require.Equal(t, []string{entry}, entries)
}
// TestTemplateVersionRef checks the git ref of release and latest channel
// versions.
func TestTemplateVersionRef(t *testing.T) {
require.Equal(t, "v10.3.1", templateVersionRef("v10.3.1"))
require.Equal(t, "3de629d", templateVersionRef("v10.3.1+3de629d"))
require.Equal(t, "v10.3.1+", templateVersionRef("v10.3.1+"))
}
+72 -15
View File
@@ -77,15 +77,24 @@ type TemplateManager struct {
DisablePublicTemplates bool // if true,
// public templates are not downloaded from the GitHub nuclei-templates repository
// Channel is the nuclei-templates update channel, TemplatesChannelStable
// (default) or TemplatesChannelLatest
Channel string
// fetchLatestRelease overrides where the official release comes from; nil
// means GitHub
fetchLatestRelease func() (templateRelease, error)
}
// latestRelease returns the official templates to install for the configured
// channel.
func (t *TemplateManager) latestRelease() (templateRelease, error) {
if t.fetchLatestRelease != nil {
return t.fetchLatestRelease()
}
if t.Channel == TemplatesChannelLatest {
return latestGitHubTemplateHead()
}
return latestGitHubTemplateRelease()
}
@@ -113,6 +122,8 @@ func (t *TemplateManager) UpdateIfOutdated() error {
return withTemplatesUpdateLock(config.DefaultConfig.TemplatesDirectory, t.updateIfOutdatedLocked)
}
// updateIfOutdatedLocked installs or updates the templates when they are
// outdated. It must be called while holding the templates update lock.
func (t *TemplateManager) updateIfOutdatedLocked() error {
// if the templates folder does not exist, it's a fresh installation and do not update
if !fileutil.FolderExists(config.DefaultConfig.TemplatesDirectory) {
@@ -129,6 +140,17 @@ func (t *TemplateManager) updateIfOutdatedLocked() error {
gologger.Debug().Msgf("Could not reload templates version, using the one loaded at startup: %s", err)
}
if t.Channel == TemplatesChannelLatest {
needsUpdate, err := t.latestCommitNeedsUpdate()
if err != nil {
return err
}
if needsUpdate {
return t.updateTemplatesAt(config.DefaultConfig.TemplatesDirectory)
}
return nil
}
needsUpdate := config.DefaultConfig.NeedsTemplateUpdate()
// NOTE(dwisiswant0): if PDTM API data is not available
@@ -151,6 +173,22 @@ func (t *TemplateManager) updateIfOutdatedLocked() error {
return nil
}
// latestCommitNeedsUpdate returns true if the head of the nuclei-templates
// default branch differs from the installed templates. The version check only
// knows about releases, so it can't tell when the head moves.
func (t *TemplateManager) latestCommitNeedsUpdate() (bool, error) {
if t.DisablePublicTemplates || !config.DefaultConfig.CanCheckForUpdates() {
return false, nil
}
release, err := t.latestRelease()
if err != nil {
return false, errkit.Wrap(err, "failed to get latest nuclei-templates commit")
}
return release.Version() != config.DefaultConfig.TemplateVersion, nil
}
// installTemplatesAt installs templates at given directory
func (t *TemplateManager) installTemplatesAt(dir string) error {
if !fileutil.FolderExists(dir) {
@@ -234,6 +272,9 @@ func (t *TemplateManager) finalizeTemplateWrite(cfg *config.Config, writtenOutpu
return t.finalizeTemplateRelease(cfg, writtenOutputs, version)
}
// bootstrapTemplateOwnership builds the ownership manifest from the archive of
// the installed version when it is missing, so an update can tell official
// templates apart from local ones.
func (t *TemplateManager) bootstrapTemplateOwnership(dir, version string, fetchArchive templateArchiveFetcher) error {
if _, err := loadTemplateOwnership(dir); err == nil {
return nil
@@ -264,7 +305,7 @@ func (t *TemplateManager) bootstrapTemplateOwnership(dir, version string, fetchA
return nil
}
archive, err := fetchArchive(version)
archive, err := fetchArchive(templateVersionRef(version))
if err != nil {
gologger.Verbose().Msgf("Continuing template update without prior ownership for %q: %s", version, err)
return nil
@@ -282,40 +323,54 @@ func (t *TemplateManager) bootstrapTemplateOwnership(dir, version string, fetchA
return nil
}
func fetchTemplateReleaseArchive(version string) (*bytes.Reader, error) {
// fetchTemplateArchive downloads the nuclei-templates zipball at ref, a release
// tag or a commit.
func fetchTemplateArchive(ref string) (*bytes.Reader, error) {
ctx := context.Background()
httpClient := &http.Client{Timeout: updateutils.DownloadUpdateTimeout}
client, err := github.NewClient(github.WithHTTPClient(httpClient))
client, err := newGitHubClient()
if err != nil {
return nil, fmt.Errorf("create github client for prior template release %q: %w", version, err)
return nil, err
}
archiveURL, _, err := client.Repositories.GetArchiveLink(
ctx,
updateutils.Organization,
config.OfficialNucleiTemplatesRepoName,
github.Zipball,
&github.RepositoryContentGetOptions{Ref: version},
&github.RepositoryContentGetOptions{Ref: ref},
1, // Follow one repository rename, as the previous client did.
)
if err != nil {
return nil, fmt.Errorf("resolve prior template release %q archive: %w", version, err)
return nil, fmt.Errorf("resolve nuclei-templates %q archive: %w", ref, err)
}
if archiveURL.Scheme != "https" {
return nil, fmt.Errorf("resolve nuclei-templates %q archive: unexpected archive url %q", ref, archiveURL)
}
request, err := http.NewRequestWithContext(ctx, http.MethodGet, archiveURL.String(), nil)
if err != nil {
return nil, fmt.Errorf("create prior template release %q archive request: %w", version, err)
return nil, fmt.Errorf("create nuclei-templates %q archive request: %w", ref, err)
}
var contents bytes.Buffer
response, err := client.Do(request, &contents)
// the archive is public, so it is downloaded without the token the github
// client sends to every host
httpClient := &http.Client{Timeout: updateutils.DownloadUpdateTimeout}
response, err := httpClient.Do(request)
if err != nil {
return nil, fmt.Errorf("download prior template release %q: %w", version, err)
return nil, fmt.Errorf("download nuclei-templates %q: %w", ref, err)
}
defer func() {
_ = response.Body.Close()
}()
if response.StatusCode != http.StatusOK {
return nil, fmt.Errorf("download prior template release %q: unexpected HTTP status %s", version, response.Status)
return nil, fmt.Errorf("download nuclei-templates %q: unexpected HTTP status %s", ref, response.Status)
}
return bytes.NewReader(contents.Bytes()), nil
contents, err := io.ReadAll(response.Body)
if err != nil {
return nil, fmt.Errorf("read nuclei-templates %q archive: %w", ref, err)
}
return bytes.NewReader(contents), nil
}
func (t *TemplateManager) bootstrapTemplateOwnershipFromArchive(dir string, archive *bytes.Reader) error {
@@ -381,7 +436,7 @@ func (t *TemplateManager) updateTemplatesAt(dir string) error {
return nil
}
if err := t.bootstrapTemplateOwnership(dir, config.DefaultConfig.TemplateVersion, fetchTemplateReleaseArchive); err != nil {
if err := t.bootstrapTemplateOwnership(dir, config.DefaultConfig.TemplateVersion, fetchTemplateArchive); err != nil {
return errkit.Wrapf(err, "failed to migrate template ownership at %s", dir)
}
@@ -400,7 +455,9 @@ func (t *TemplateManager) updateTemplatesAt(dir string) error {
latestVersion := release.Version()
currentVersion := config.DefaultConfig.TemplateVersion
if config.IsOutdatedVersion(currentVersion, latestVersion) {
// versions on the latest channel only differ in build metadata, which
// version comparison ignores
if t.Channel == TemplatesChannelLatest || config.IsOutdatedVersion(currentVersion, latestVersion) {
gologger.Info().Msgf("Your current nuclei-templates %s are outdated. Latest is %s\n", currentVersion, latestVersion)
} else {
gologger.Debug().Msgf("Updating nuclei-templates from %s to %s (forced update)\n", currentVersion, latestVersion)
+53
View File
@@ -83,6 +83,59 @@ func TestUpdateIfOutdatedSkipsReleaseInstalledWhileWaiting(t *testing.T) {
require.Equal(t, "v2.0.0", cfg.TemplateVersion)
}
// TestUpdateIfOutdatedLatestChannel checks that the latest channel updates when
// the head moves, even though the release stays the same.
func TestUpdateIfOutdatedLatestChannel(t *testing.T) {
HideProgressBar = true
cfg := &config.Config{Logger: gologger.DefaultLogger}
cfg.SetConfigDir(t.TempDir())
cfg.SetTemplatesDir(filepath.Join(t.TempDir(), "templates"))
previousConfig := config.DefaultConfig
config.DefaultConfig = cfg
t.Cleanup(func() { config.DefaultConfig = previousConfig })
installed := fakeTemplateRelease{version: "v1.0.0+aaaaaaa", files: map[string]string{
"http/old.yaml": "id: old",
}}
tm := &TemplateManager{Channel: TemplatesChannelLatest, fetchLatestRelease: installed.fetch}
require.NoError(t, tm.FreshInstallIfNotExists())
require.Equal(t, "v1.0.0+aaaaaaa", cfg.TemplateVersion)
// the version check reports the release the installed commit is based on,
// so the stable check alone would never update
cfg.LatestNucleiTemplatesVersion = "v1.0.0"
require.False(t, cfg.NeedsTemplateUpdate())
t.Run("updates when the head moves", func(t *testing.T) {
head := fakeTemplateRelease{version: "v1.0.0+bbbbbbb", files: map[string]string{
"http/old.yaml": "id: old",
"http/new.yaml": "id: new",
}}
tm := &TemplateManager{Channel: TemplatesChannelLatest, fetchLatestRelease: head.fetch}
require.NoError(t, tm.UpdateIfOutdated())
require.Equal(t, "v1.0.0+bbbbbbb", cfg.TemplateVersion)
require.FileExists(t, filepath.Join(cfg.TemplatesDirectory, "http", "new.yaml"))
})
t.Run("skips the download when already at the head", func(t *testing.T) {
head := installedTemplateRelease{fakeTemplateRelease: fakeTemplateRelease{version: "v1.0.0+bbbbbbb"}, t: t}
tm := &TemplateManager{Channel: TemplatesChannelLatest, fetchLatestRelease: func() (templateRelease, error) {
return head, nil
}}
require.NoError(t, tm.UpdateIfOutdated())
require.Equal(t, "v1.0.0+bbbbbbb", cfg.TemplateVersion)
})
t.Run("returns the error when the head can't be fetched", func(t *testing.T) {
tm := &TemplateManager{Channel: TemplatesChannelLatest, fetchLatestRelease: func() (templateRelease, error) {
return nil, errors.New("rate limited")
}}
require.ErrorContains(t, tm.UpdateIfOutdated(), "rate limited")
require.Equal(t, "v1.0.0+bbbbbbb", cfg.TemplateVersion)
})
}
func TestIsOutdatedVersion(t *testing.T) {
testCases := []struct {
current string
+112 -5
View File
@@ -2,22 +2,36 @@ package installer
import (
"crypto/sha256"
"errors"
"fmt"
"os"
"path/filepath"
"reflect"
"strings"
"time"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/utils/errkit"
)
// Tunables for withTemplatesUpdateLock. Package vars (instead of constants) so
// tests can shrink the 10m/15m windows without waiting them out.
var (
templatesUpdateLockTimeout = 10 * time.Minute
templatesUpdateLockStaleAfter = 15 * time.Minute
templatesUpdateLockPollInterval = 250 * time.Millisecond
// removeTemplatesUpdateLock is os.Remove except in tests that simulate a
// lock disappearing, or refusing to disappear, between Stat and Remove.
removeTemplatesUpdateLock = os.Remove
)
// withTemplatesUpdateLock serializes template install/update of templatesDir
// across processes. Parallel `go test` packages each have their own sync.Once,
// so without a cross-process lock they race on a shared templates directory
// (ENOENT, partial trees, checksum mismatches).
func withTemplatesUpdateLock(templatesDir string, fn func() error) error {
lockPath := templatesUpdateLockPath(templatesDir)
deadline := time.Now().Add(10 * time.Minute)
deadline := time.Now().Add(templatesUpdateLockTimeout)
for {
f, err := os.OpenFile(lockPath, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600)
@@ -30,17 +44,110 @@ func withTemplatesUpdateLock(templatesDir string, fn func() error) error {
return fn()
}
if info, statErr := os.Stat(lockPath); statErr == nil && time.Since(info.ModTime()) > 15*time.Minute {
_ = os.Remove(lockPath)
// Anything other than "already exists" (e.g. /tmp not writable,
// read-only filesystem) will never succeed by retrying: fail fast
// instead of spinning until the deadline.
if !errors.Is(err, os.ErrExist) {
return errkit.Wrapf(err, "could not create nuclei templates update lock at %s", lockPath)
}
info, statErr := os.Stat(lockPath)
if statErr != nil {
if os.IsNotExist(statErr) {
// Lost a race with the holder releasing the lock; retry
// immediately instead of sleeping through the window.
continue
}
if time.Now().After(deadline) {
return errkit.Wrapf(statErr, "timed out waiting for nuclei templates update lock at %s (could not stat lock); remove it manually if it is stale, or rerun with -duc/-disable-update-check to skip the automatic template update", lockPath)
}
time.Sleep(templatesUpdateLockPollInterval)
continue
}
if age := time.Since(info.ModTime()); age > templatesUpdateLockStaleAfter {
if rmErr := removeTemplatesUpdateLock(lockPath); rmErr != nil && !os.IsNotExist(rmErr) {
// The classic multi-user hang: the lock lives in the shared
// os.TempDir(), which has the sticky bit, so a lock file
// created by another user cannot be unlinked (EPERM) by this
// process. Retrying is pointless; surface who owns it and
// how to recover instead of looping until the deadline.
return errkit.Wrapf(rmErr, "stale nuclei templates update lock at %s (%s) could not be removed; it may belong to another user and cannot be unlinked from the shared temp dir, remove it manually (e.g. sudo rm %s) or rerun with -duc/-disable-update-check to skip the automatic template update", lockPath, describeUpdateLock(info, lockPath), lockPath)
}
continue
}
if time.Now().After(deadline) {
return errkit.Wrap(err, "timed out waiting for nuclei templates update lock")
return errkit.Wrapf(err, "timed out waiting for nuclei templates update lock at %s (%s); another process may be updating templates, if no update is running remove the stale lock manually (e.g. sudo rm %s) or rerun with -duc/-disable-update-check to skip the automatic template update", lockPath, describeUpdateLock(info, lockPath), lockPath)
}
time.Sleep(250 * time.Millisecond)
time.Sleep(templatesUpdateLockPollInterval)
}
}
// describeUpdateLock renders age, ownership and recorded pid of an existing
// lock for error messages. Everything is best-effort: the file is usually
// 0600, so a lock owned by another user may be neither readable nor statable
// beyond what os.Stat already returned.
func describeUpdateLock(info os.FileInfo, lockPath string) string {
var parts []string
parts = append(parts, fmt.Sprintf("age %s", time.Since(info.ModTime()).Round(time.Second)))
parts = append(parts, fmt.Sprintf("mode %s", info.Mode()))
if uid, gid, ok := updateLockOwner(info); ok {
extra := fmt.Sprintf("uid=%d gid=%d", uid, gid)
if current := os.Getuid(); current >= 0 && uid != uint64(current) {
extra += " (another user)"
}
parts = append(parts, extra)
}
if pid, err := os.ReadFile(lockPath); err == nil {
if trimmed := strings.TrimSpace(string(pid)); trimmed != "" {
parts = append(parts, fmt.Sprintf("pid %s", trimmed))
}
} else if os.IsPermission(err) {
parts = append(parts, "pid unreadable (permission denied, likely another user)")
}
return strings.Join(parts, ", ")
}
// updateLockOwner extracts uid/gid from os.Stat without importing syscall
// directly (syscall.Stat_t does not exist on windows). It uses reflection on
// info.Sys(): on unix this is *syscall.Stat_t with Uid/Gid fields, elsewhere
// the assertion fails and ok=false is reported.
func updateLockOwner(info os.FileInfo) (uid, gid uint64, ok bool) {
sys := info.Sys()
if sys == nil {
return 0, 0, false
}
v := reflect.ValueOf(sys)
if v.Kind() == reflect.Pointer && !v.IsNil() {
v = v.Elem()
}
if v.Kind() != reflect.Struct {
return 0, 0, false
}
uidField := v.FieldByName("Uid")
gidField := v.FieldByName("Gid")
if !uidField.IsValid() || !gidField.IsValid() {
return 0, 0, false
}
toUint := func(rv reflect.Value) (uint64, bool) {
switch rv.Kind() {
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
return rv.Uint(), true
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
if n := rv.Int(); n >= 0 {
return uint64(n), true
}
}
return 0, false
}
u, okU := toUint(uidField)
g, okG := toUint(gidField)
if !okU || !okG {
return 0, 0, false
}
return u, g, true
}
// templatesUpdateLockPath derives one lock per templates directory, so updates
// of different directories do not wait on each other. Every alias of a
// directory maps to the same lock, including a symlink whose target does not
+105
View File
@@ -3,11 +3,24 @@ package installer
import (
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/stretchr/testify/require"
)
// errkit quotes the error chain, so a Windows path's backslashes are escaped
// in Error() even though the message itself contains the real path.
func requireErrorContainsPath(t *testing.T, err error, path string) {
t.Helper()
rendered := err.Error()
if strings.Contains(rendered, path) || strings.Contains(rendered, strings.ReplaceAll(path, `\`, `\\`)) {
return
}
require.Contains(t, rendered, path)
}
func TestTemplatesUpdateLockPath(t *testing.T) {
t.Run("same directory, same lock", func(t *testing.T) {
dir := t.TempDir()
@@ -58,3 +71,95 @@ func TestTemplatesUpdateLockPath(t *testing.T) {
require.Equal(t, templatesUpdateLockPath(target), templatesUpdateLockPath(link))
})
}
func TestWithTemplatesUpdateLockStaleLockIsCleared(t *testing.T) {
dir := t.TempDir()
lockPath := templatesUpdateLockPath(dir)
require.NoError(t, os.WriteFile(lockPath, []byte("1234\n"), 0o600))
stale := time.Now().Add(-1 * time.Hour)
require.NoError(t, os.Chtimes(lockPath, stale, stale))
t.Cleanup(func() { _ = os.Remove(lockPath) })
ran := false
require.NoError(t, withTemplatesUpdateLock(dir, func() error {
ran = true
return nil
}))
require.True(t, ran, "fn should run after the stale lock is cleared")
}
func TestWithTemplatesUpdateLockFreshLockTimesOutWithHelpfulMessage(t *testing.T) {
oldTimeout, oldStale, oldPoll := templatesUpdateLockTimeout, templatesUpdateLockStaleAfter, templatesUpdateLockPollInterval
templatesUpdateLockTimeout = 200 * time.Millisecond
templatesUpdateLockStaleAfter = time.Hour
templatesUpdateLockPollInterval = 10 * time.Millisecond
defer func() {
templatesUpdateLockTimeout, templatesUpdateLockStaleAfter, templatesUpdateLockPollInterval = oldTimeout, oldStale, oldPoll
}()
dir := t.TempDir()
lockPath := templatesUpdateLockPath(dir)
require.NoError(t, os.WriteFile(lockPath, []byte("99999\n"), 0o600))
t.Cleanup(func() { _ = os.Remove(lockPath) })
start := time.Now()
err := withTemplatesUpdateLock(dir, func() error { return nil })
elapsed := time.Since(start)
require.Error(t, err)
require.Less(t, elapsed, 10*time.Second, "must fail fast instead of hanging")
require.Contains(t, err.Error(), "timed out waiting for nuclei templates update lock")
requireErrorContainsPath(t, err, lockPath)
require.Contains(t, err.Error(), "-duc")
}
func TestWithTemplatesUpdateLockStaleLockAlreadyRemoved(t *testing.T) {
previous := removeTemplatesUpdateLock
t.Cleanup(func() { removeTemplatesUpdateLock = previous })
removeTemplatesUpdateLock = func(path string) error {
if err := os.Remove(path); err != nil && !os.IsNotExist(err) {
return err
}
// Another process removed the stale lock after we stat-ed it.
return os.ErrNotExist
}
dir := t.TempDir()
lockPath := templatesUpdateLockPath(dir)
require.NoError(t, os.WriteFile(lockPath, []byte("1234\n"), 0o600))
stale := time.Now().Add(-1 * time.Hour)
require.NoError(t, os.Chtimes(lockPath, stale, stale))
t.Cleanup(func() { _ = os.Remove(lockPath) })
ran := false
require.NoError(t, withTemplatesUpdateLock(dir, func() error {
ran = true
return nil
}))
require.True(t, ran, "a lock another process already removed must be taken, not treated as a failure")
}
func TestWithTemplatesUpdateLockUnremovableStaleLockFailsFast(t *testing.T) {
previous := removeTemplatesUpdateLock
t.Cleanup(func() { removeTemplatesUpdateLock = previous })
removeTemplatesUpdateLock = func(string) error { return os.ErrPermission }
dir := t.TempDir()
lockPath := templatesUpdateLockPath(dir)
require.NoError(t, os.WriteFile(lockPath, []byte("1234\n"), 0o600))
stale := time.Now().Add(-1 * time.Hour)
require.NoError(t, os.Chtimes(lockPath, stale, stale))
t.Cleanup(func() { _ = os.Remove(lockPath) })
ran := false
start := time.Now()
err := withTemplatesUpdateLock(dir, func() error {
ran = true
return nil
})
require.Error(t, err)
require.False(t, ran)
require.Less(t, time.Since(start), 2*time.Second)
require.ErrorIs(t, err, os.ErrPermission)
require.Contains(t, err.Error(), "could not be removed")
requireErrorContainsPath(t, err, lockPath)
}
+2 -2
View File
@@ -10,8 +10,8 @@ import (
"reflect"
"time"
"github.com/projectdiscovery/goja"
"github.com/logrusorgru/aurora/v4"
"github.com/projectdiscovery/goja"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/js/gojs"
jsnet "github.com/projectdiscovery/nuclei/v3/pkg/js/libs/net"
@@ -156,7 +156,7 @@ func initBuiltInFunc(runtime *goja.Runtime) {
executionId := ctx.Value("executionId").(string)
dialer := protocolstate.GetDialersWithId(executionId)
if dialer == nil {
panic("dialers with executionId " + executionId + " not found")
return false, errkit.New("dialers with executionId %s not found", executionId)
}
conn, err := dialer.Fastdialer.Dial(ctx, "udp", net.JoinHostPort(host, port))
+8 -1
View File
@@ -1,6 +1,7 @@
package ldap
import (
"context"
"crypto/tls"
"fmt"
"net"
@@ -9,6 +10,7 @@ import (
"github.com/go-ldap/ldap/v3"
"github.com/projectdiscovery/goja"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/js/utils"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
)
@@ -92,7 +94,8 @@ func NewClient(call goja.ConstructorCall, runtime *goja.Runtime) *goja.Object {
executionId := c.nj.ExecutionId()
dialers := protocolstate.GetDialersWithId(executionId)
if dialers == nil {
panic("dialers with executionId " + executionId + " not found")
gologger.Error().Msgf("dialers with executionId %s not found", executionId)
return nil
}
dialCtx := c.nj.Context()
@@ -143,6 +146,10 @@ func NewClient(call goja.ConstructorCall, runtime *goja.Runtime) *goja.Object {
}
c.nj.HandleError(err, "failed to connect to ldap server")
}
// go-ldap waits for a reply with no timeout, and an interrupt cannot stop the runtime
// inside that read: a server that accepts and never answers would keep the goroutine,
// and everything the abandoned runtime references, alive for the life of the process.
context.AfterFunc(dialCtx, func() { _ = conn.Close() })
c.conn = ldap.NewConn(conn, u.Scheme == "ldaps")
if u.Scheme != "ldaps" && c.cfg.Upgrade {
serverName := u.Hostname()
+60 -1
View File
@@ -76,8 +76,67 @@ func TestNewClientAllowsTCPWhenNetworkPolicyAllows(t *testing.T) {
}
}
// A server that accepts and never answers used to keep Authenticate, and the runtime
// abandoned around it, blocked for the life of the process.
func TestAuthenticateReturnsWhenTheContextIsDone(t *testing.T) {
listener, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
_ = listener.Close()
})
go func() {
var held []net.Conn
for {
conn, err := listener.Accept()
if err != nil {
for _, c := range held {
_ = c.Close()
}
return
}
held = append(held, conn)
}
}()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
client, err := newLDAPClientWithContextForTest(t, ctx, &types.Options{}, "ldap://"+listener.Addr().String())
if err != nil {
t.Fatal(err)
}
done := make(chan struct{})
go func() {
defer close(done)
defer func() {
_ = recover() // the failed bind is thrown to the runtime
}()
client.Authenticate("user", "")
}()
select {
case <-done:
t.Fatal("Authenticate returned before the server answered or the context was done")
case <-time.After(200 * time.Millisecond):
}
cancel()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("Authenticate still blocked after the context was done")
}
}
func newLDAPClientForTest(t *testing.T, options *types.Options, ldapURL string) (*Client, error) {
t.Helper()
return newLDAPClientWithContextForTest(t, context.Background(), options, ldapURL)
}
func newLDAPClientWithContextForTest(t *testing.T, ctx context.Context, options *types.Options, ldapURL string) (*Client, error) {
t.Helper()
executionID := "ldap-" + strings.NewReplacer("/", "-", " ", "-").Replace(t.Name())
options.ExecutionId = executionID
@@ -90,7 +149,7 @@ func newLDAPClientForTest(t *testing.T, options *types.Options, ldapURL string)
runtime := goja.New()
runtime.SetContextValue("executionId", executionID)
runtime.SetContextValue("ctx", context.Background())
runtime.SetContextValue("ctx", ctx)
obj, err := runtime.New(runtime.ToValue(NewClient), runtime.ToValue(ldapURL), runtime.ToValue("corp.internal"))
if err != nil {
+23
View File
@@ -146,6 +146,16 @@ func (c *bufferedConn) Read(b []byte) (int, error) {
return c.reader.Read(b)
}
// dialTargetHost returns the host portion of a dial address. Addresses that
// are already a bare host are returned unchanged.
func dialTargetHost(address string) string {
host, _, err := net.SplitHostPort(address)
if err != nil {
return address
}
return host
}
// Dial opens a connection using the execution dialer.
// When an HTTP(S) proxy is present in context, TCP dials use HTTP CONNECT.
// Misconfigured proxies return an error instead of silently dialing direct.
@@ -169,6 +179,19 @@ func Dial(ctx context.Context, protocol, address string) (net.Conn, error) {
}
switch parsed.Scheme {
case "http", "https":
// Enforce the network policy on the tunneled target. Direct dials go
// through Fastdialer, which validates the destination itself, but the
// proxy path only dials the proxy through Fastdialer and tunnels the
// target inside an HTTP CONNECT, so the target would otherwise skip the
// deny-list (-exclude-hosts / -restrict-local-network-access) checks.
// Split the port off first. IsHostAllowed only does that for a single
// colon, so an address like [::1]:80 would skip the IP deny list.
// Do this only for schemes that actually tunnel. IsHostAllowed resolves
// hostnames, and an earlier check would hide "unsupported proxy scheme"
// behind a DNS failure.
if !protocolstate.IsHostAllowed(executionId, dialTargetHost(address)) {
return nil, fmt.Errorf("host %s is not allowed by network policy", address)
}
return dialHTTPProxy(ctx, dialer.Fastdialer.Dial, proxyURL, address, getTimeoutFromContext(ctx))
default:
return nil, fmt.Errorf("unsupported proxy scheme %q in %s (expected http or https)", parsed.Scheme, redactProxyURL(proxyURL))
+80 -5
View File
@@ -221,7 +221,7 @@ func TestDial(t *testing.T) {
t.Cleanup(func() { protocolstate.Close(opts.ExecutionId) })
ctx := context.WithValue(context.Background(), "executionId", opts.ExecutionId) // nolint: staticcheck
ctx = context.WithValue(ctx, "proxyURL", "socks5://127.0.0.1:1080") // nolint: staticcheck
ctx = context.WithValue(ctx, "proxyURL", "socks5://127.0.0.1:1080") // nolint: staticcheck
_, err := Dial(ctx, "tcp", "example.com:80")
require.Error(t, err)
require.Contains(t, err.Error(), "unsupported proxy scheme")
@@ -233,7 +233,7 @@ func TestDial(t *testing.T) {
t.Cleanup(func() { protocolstate.Close(opts.ExecutionId) })
ctx := context.WithValue(context.Background(), "executionId", opts.ExecutionId) // nolint: staticcheck
ctx = context.WithValue(ctx, "proxyURL", "://bad") // nolint: staticcheck
ctx = context.WithValue(ctx, "proxyURL", "://bad") // nolint: staticcheck
_, err := Dial(ctx, "tcp", "example.com:80")
require.Error(t, err)
require.Contains(t, err.Error(), "invalid proxy URL")
@@ -245,12 +245,87 @@ func TestDial(t *testing.T) {
t.Cleanup(func() { protocolstate.Close(opts.ExecutionId) })
ctx := context.WithValue(context.Background(), "executionId", opts.ExecutionId) // nolint: staticcheck
ctx = context.WithValue(ctx, "proxyURL", "http://127.0.0.1:8080") // nolint: staticcheck
ctx = context.WithValue(ctx, "proxyURL", "http://127.0.0.1:8080") // nolint: staticcheck
_, err := Dial(ctx, "udp", "example.com:53")
require.Error(t, err)
require.Contains(t, err.Error(), "does not support protocol")
})
t.Run("proxy path enforces network policy on the target", func(t *testing.T) {
// A proxy dial must not let a template reach a deny-listed target by
// tunnelling it inside an HTTP CONNECT. The policy check has to run on
// the target before the proxy is contacted.
proxyLn, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
defer func() { _ = proxyLn.Close() }()
proxyHit := make(chan struct{}, 1)
go func() {
conn, err := proxyLn.Accept()
if err != nil {
return
}
defer func() { _ = conn.Close() }()
proxyHit <- struct{}{}
}()
opts := &types.Options{
ExecutionId: "net-dial-policy-" + t.Name(),
ExcludeTargets: []string{"127.0.0.2"},
}
require.NoError(t, protocolstate.Init(opts))
t.Cleanup(func() { protocolstate.Close(opts.ExecutionId) })
ctx := context.WithValue(context.Background(), "executionId", opts.ExecutionId) // nolint: staticcheck
ctx = context.WithValue(ctx, "proxyURL", "http://"+proxyLn.Addr().String()) // nolint: staticcheck
_, err = Dial(ctx, "tcp", "127.0.0.2:9999")
require.Error(t, err)
require.Contains(t, err.Error(), "not allowed by network policy")
select {
case <-proxyHit:
t.Fatal("proxy was contacted for a deny-listed target")
default:
}
})
t.Run("proxy path enforces network policy on an ipv6 target", func(t *testing.T) {
proxyLn, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
defer func() { _ = proxyLn.Close() }()
proxyHit := make(chan struct{}, 1)
go func() {
conn, err := proxyLn.Accept()
if err != nil {
return
}
defer func() { _ = conn.Close() }()
proxyHit <- struct{}{}
}()
opts := &types.Options{
ExecutionId: "net-dial-policy-v6-" + t.Name(),
ExcludeTargets: []string{"::1"},
}
require.NoError(t, protocolstate.Init(opts))
t.Cleanup(func() { protocolstate.Close(opts.ExecutionId) })
ctx := context.WithValue(context.Background(), "executionId", opts.ExecutionId) // nolint: staticcheck
ctx = context.WithValue(ctx, "proxyURL", "http://"+proxyLn.Addr().String()) // nolint: staticcheck
_, err = Dial(ctx, "tcp", "[::1]:9999")
require.Error(t, err)
require.Contains(t, err.Error(), "not allowed by network policy")
select {
case <-proxyHit:
t.Fatal("proxy was contacted for a deny-listed ipv6 target")
default:
}
})
t.Run("open through local http proxy", func(t *testing.T) {
target, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
@@ -299,8 +374,8 @@ func TestDial(t *testing.T) {
require.NoError(t, protocolstate.Init(opts))
t.Cleanup(func() { protocolstate.Close(opts.ExecutionId) })
ctx := context.WithValue(context.Background(), "executionId", opts.ExecutionId) // nolint: staticcheck
ctx = context.WithValue(ctx, "proxyURL", "http://"+proxyLn.Addr().String()) // nolint: staticcheck
ctx := context.WithValue(context.Background(), "executionId", opts.ExecutionId) // nolint: staticcheck
ctx = context.WithValue(ctx, "proxyURL", "http://"+proxyLn.Addr().String()) // nolint: staticcheck
ctx = context.WithValue(ctx, "timeoutVariants", &types.Timeouts{TcpReadTimeout: 5 * time.Second}) // nolint: staticcheck
nconn, err := Open(ctx, "tcp", target.Addr().String())
+1
View File
@@ -3,6 +3,7 @@ package redis
import (
"context"
"errors"
"fmt"
+179 -20
View File
@@ -1,8 +1,13 @@
package redis
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"net"
"strconv"
"time"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
@@ -30,11 +35,7 @@ func getServerInfo(ctx context.Context, executionId string, host string, port in
return "", protocolstate.ErrHostDenied.Msgf(host)
}
// create a new client
client := redis.NewClient(&redis.Options{
Addr: fmt.Sprintf("%s:%d", host, port),
Password: "", // no password set
DB: 0, // use default DB
})
client := newClient(fmt.Sprintf("%s:%d", host, port), "")
defer func() {
_ = client.Close()
}()
@@ -72,11 +73,7 @@ func connect(ctx context.Context, executionId string, host string, port int, pas
return false, protocolstate.ErrHostDenied.Msgf(host)
}
// create a new client
client := redis.NewClient(&redis.Options{
Addr: fmt.Sprintf("%s:%d", host, port),
Password: password, // no password set
DB: 0, // use default DB
})
client := newClient(fmt.Sprintf("%s:%d", host, port), password)
defer func() {
_ = client.Close()
}()
@@ -112,11 +109,7 @@ func getServerInfoAuth(ctx context.Context, executionId string, host string, por
return "", protocolstate.ErrHostDenied.Msgf(host)
}
// create a new client
client := redis.NewClient(&redis.Options{
Addr: fmt.Sprintf("%s:%d", host, port),
Password: password, // no password set
DB: 0, // use default DB
})
client := newClient(fmt.Sprintf("%s:%d", host, port), password)
defer func() {
_ = client.Close()
}()
@@ -187,11 +180,7 @@ func RunLuaScript(ctx context.Context, host string, port int, password string, s
return false, protocolstate.ErrHostDenied.Msgf(host)
}
// create a new client
client := redis.NewClient(&redis.Options{
Addr: fmt.Sprintf("%s:%d", host, port),
Password: password,
DB: 0, // use default DB
})
client := newClient(fmt.Sprintf("%s:%d", host, port), password)
defer func() {
_ = client.Close()
}()
@@ -261,3 +250,173 @@ func RunLuaScript(ctx context.Context, host string, port int, password string, s
return infoCmd.Val(), nil
}
// maxReadBytes bounds what one connection may read. go-redis reads a reply line until "\n" with
// no length limit, so a port that streams bytes without one was buffered for as long as it kept
// sending. The replies these calls expect, INFO included, are a few kilobytes.
//
// A RESP length header is rejected before it is returned. go-redis allocates the declared
// bulk size, or one interface per array element, before reading the payload.
const maxReadBytes = 16 << 20
// maxAggregate is how many array, set, map, attribute, or push elements fit in maxReadBytes,
// since each element is an interface value.
const maxAggregate = maxReadBytes / 16
var errReadLimit = fmt.Errorf("redis: reply exceeds %d bytes", maxReadBytes)
// newClient is redis.NewClient on the default dialer, with each connection's reads capped at
// maxReadBytes.
func newClient(addr, password string) *redis.Client {
opts := &redis.Options{
Addr: addr,
Password: password,
DB: 0,
}
dial := redis.NewDialer(opts)
opts.Dialer = func(ctx context.Context, network, addr string) (net.Conn, error) {
conn, err := dial(ctx, network, addr)
if err != nil {
return nil, err
}
return &limitedConn{Conn: conn, remaining: maxReadBytes}, nil
}
return redis.NewClient(opts)
}
// limitedConn fails a read once remaining is spent, and withholds a RESP length line whose
// declared size is over the cap so go-redis never allocates it.
type limitedConn struct {
net.Conn
remaining int64
pending []byte
partial []byte
err error
scratch [32 << 10]byte
}
func (c *limitedConn) Read(b []byte) (int, error) {
if len(b) == 0 {
return 0, nil
}
if len(c.pending) == 0 {
if err := c.fill(); err != nil {
return 0, err
}
}
n := copy(b, c.pending)
c.pending = c.pending[n:]
return n, nil
}
func (c *limitedConn) fill() error {
if len(c.pending) > 0 {
return nil
}
if c.err != nil {
return c.err
}
for len(c.pending) == 0 {
if c.remaining <= 0 {
c.err = errReadLimit
return c.err
}
want := int64(len(c.scratch))
if want > c.remaining {
want = c.remaining
}
n, err := c.Conn.Read(c.scratch[:want])
if n > 0 {
c.remaining -= int64(n)
if aerr := c.accept(c.scratch[:n]); aerr != nil {
c.err = aerr
if len(c.pending) > 0 {
return nil
}
return aerr
}
}
if err != nil {
// A read deadline belongs to one call. go-redis peeks for push notifications
// with a few microseconds of deadline, and on Windows that peek always runs.
// Keeping the timeout fails the command that follows.
if timeoutErr(err) {
if len(c.pending) > 0 {
return nil
}
return err
}
c.err = err
if len(c.pending) > 0 {
return nil
}
return err
}
if n == 0 {
c.err = io.ErrUnexpectedEOF
return c.err
}
}
return nil
}
func (c *limitedConn) accept(p []byte) error {
c.partial = append(c.partial, p...)
for {
i := bytes.IndexByte(c.partial, '\n')
if i < 0 {
if int64(len(c.partial)) > maxReadBytes {
c.partial = nil
return errReadLimit
}
return nil
}
line := c.partial[:i+1]
if lengthTooLarge(line) {
c.partial = nil
return errReadLimit
}
copied := make([]byte, len(line))
copy(copied, line)
c.pending = append(c.pending, copied...)
rest := make([]byte, len(c.partial)-(i+1))
copy(rest, c.partial[i+1:])
c.partial = rest
}
}
// lengthTooLarge reports whether line is a RESP length header go-redis would allocate from.
// Bulk strings allocate the declared byte length. Arrays, sets, maps, attributes, and pushes
// allocate one interface per declared element.
func lengthTooLarge(line []byte) bool {
if len(line) < 3 {
return false
}
var limit int64
switch line[0] {
case '$', '=', '!':
limit = maxReadBytes
case '*', '~', '>', '%', '|':
limit = maxAggregate
default:
return false
}
body := line[1 : len(line)-1]
if len(body) > 0 && body[len(body)-1] == '\r' {
body = body[:len(body)-1]
}
if len(body) == 0 || body[0] == '-' {
return false
}
n, err := strconv.ParseInt(string(body), 10, 64)
if err != nil {
return true
}
return n > limit
}
func timeoutErr(err error) bool {
var netErr net.Error
return errors.As(err, &netErr) && netErr.Timeout()
}
+309
View File
@@ -0,0 +1,309 @@
package redis
import (
"bufio"
"bytes"
"context"
"fmt"
"net"
"runtime"
"strconv"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/stretchr/testify/require"
)
// A port that streams bytes without a newline used to be buffered whole: go-redis reads a reply
// line until "\n" with no length limit.
func TestGetServerInfoAuthBoundsAReplyWithoutNewline(t *testing.T) {
listener, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
t.Cleanup(func() {
_ = listener.Close()
})
// Bounded so the test cannot exhaust the machine when the limit is missing.
const streamBytes = 256 << 20
chunk := bytes.Repeat([]byte("A"), 32<<10)
go func() {
for {
conn, err := listener.Accept()
if err != nil {
return
}
go func() {
defer func() {
_ = conn.Close()
}()
for sent := 0; sent < streamBytes; sent += len(chunk) {
if _, err := conn.Write(chunk); err != nil {
return
}
}
}()
}
}()
executionID := "redis-" + strings.ReplaceAll(t.Name(), "/", "-")
require.NoError(t, protocolstate.Init(&types.Options{ExecutionId: executionID}))
t.Cleanup(func() {
protocolstate.Close(executionID)
})
addr, ok := listener.Addr().(*net.TCPAddr)
require.True(t, ok)
runtime.GC()
var before runtime.MemStats
runtime.ReadMemStats(&before)
var peak atomic.Uint64
done := make(chan struct{})
go func() {
for {
select {
case <-done:
return
case <-time.After(10 * time.Millisecond):
}
var m runtime.MemStats
runtime.ReadMemStats(&m)
if m.HeapInuse > peak.Load() {
peak.Store(m.HeapInuse)
}
}
}()
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
_, err = getServerInfoAuth(ctx, executionID, "127.0.0.1", addr.Port, "")
close(done)
require.Error(t, err)
growth := int64(peak.Load()) - int64(before.HeapInuse)
require.Less(t, growth, int64(64<<20), "heap grew %d MiB reading one reply", growth>>20)
}
func TestGetServerInfoAuthReadsAnInfoReply(t *testing.T) {
listener, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
t.Cleanup(func() {
_ = listener.Close()
})
const info = "# Server\r\nredis_version:7.2.4\r\n"
go func() {
for {
conn, err := listener.Accept()
if err != nil {
return
}
go serveRESP(conn, info)
}
}()
executionID := "redis-" + strings.ReplaceAll(t.Name(), "/", "-")
require.NoError(t, protocolstate.Init(&types.Options{ExecutionId: executionID}))
t.Cleanup(func() {
protocolstate.Close(executionID)
})
addr, ok := listener.Addr().(*net.TCPAddr)
require.True(t, ok)
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
got, err := getServerInfoAuth(ctx, executionID, "127.0.0.1", addr.Port, "")
require.NoError(t, err)
require.Equal(t, info, got)
}
// serveRESP answers HELLO as a Redis without RESP3 does, so the client falls back to RESP2, and
// CLIENT, PING and INFO as a Redis would.
func serveRESP(conn net.Conn, info string) {
defer func() {
_ = conn.Close()
}()
rd := bufio.NewReader(conn)
for {
command, err := readRESPCommand(rd)
if err != nil {
return
}
var reply string
switch strings.ToUpper(command) {
case "HELLO":
reply = "-ERR unknown command 'HELLO'\r\n"
case "PING":
reply = "+PONG\r\n"
case "INFO":
reply = fmt.Sprintf("$%d\r\n%s\r\n", len(info), info)
default:
reply = "+OK\r\n"
}
if _, err := conn.Write([]byte(reply)); err != nil {
return
}
}
}
// readRESPCommand reads one request array and returns its command name.
func readRESPCommand(rd *bufio.Reader) (string, error) {
header, err := rd.ReadString('\n')
if err != nil {
return "", err
}
count, err := strconv.Atoi(strings.TrimSpace(strings.TrimPrefix(header, "*")))
if err != nil {
return "", err
}
var command string
for i := range count {
if _, err := rd.ReadString('\n'); err != nil { // $<len>
return "", err
}
arg, err := rd.ReadString('\n')
if err != nil {
return "", err
}
if i == 0 {
command = strings.TrimSpace(arg)
}
}
return command, nil
}
func TestLengthTooLarge(t *testing.T) {
tests := []struct {
line string
want bool
}{
{line: fmt.Sprintf("$%d\r\n", maxReadBytes), want: false},
{line: fmt.Sprintf("$%d\r\n", maxReadBytes+1), want: true},
{line: "$-1\r\n", want: false},
{line: "+OK\r\n", want: false},
{line: fmt.Sprintf("*%d\r\n", maxAggregate), want: false},
{line: fmt.Sprintf("*%d\r\n", maxAggregate+1), want: true},
{line: "$999999999999999999999\r\n", want: true},
}
for _, tt := range tests {
t.Run(tt.line, func(t *testing.T) {
require.Equal(t, tt.want, lengthTooLarge([]byte(tt.line)))
})
}
}
// A short RESP length header used to make go-redis allocate the declared size before reading
// any of the payload, so the byte cap never applied.
func TestGetServerInfoAuthRejectsAHugeDeclaredLength(t *testing.T) {
// The declared size is large enough to show up in the heap and small enough to survive a
// missing check. The allowed growth stays under that allocation.
tests := []struct {
name string
header string
limit int64
}{
// The handshake reads a map. A bulk length has to be one of its values, or go-redis
// rejects the header as the wrong type before allocating it.
{name: "bulk string", header: fmt.Sprintf("%%1\r\n$1\r\nk\r\n$%d\r\n", 32<<20), limit: 16 << 20},
{name: "array", header: fmt.Sprintf("*%d\r\n", 8<<20), limit: 64 << 20},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
listener, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
t.Cleanup(func() {
_ = listener.Close()
})
go func() {
conn, err := listener.Accept()
if err != nil {
return
}
defer func() {
_ = conn.Close()
}()
_, _ = conn.Write([]byte(tt.header))
_, _ = conn.Read(make([]byte, 1))
}()
executionID := "redis-" + strings.ReplaceAll(t.Name(), "/", "-")
require.NoError(t, protocolstate.Init(&types.Options{ExecutionId: executionID}))
t.Cleanup(func() {
protocolstate.Close(executionID)
})
addr, ok := listener.Addr().(*net.TCPAddr)
require.True(t, ok)
var before runtime.MemStats
runtime.ReadMemStats(&before)
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
_, err = getServerInfoAuth(ctx, executionID, "127.0.0.1", addr.Port, "")
var after runtime.MemStats
runtime.ReadMemStats(&after)
require.Error(t, err)
// TotalAlloc counts the slice go-redis makes from the length header even when that
// slice is freed before the call returns.
growth := int64(after.TotalAlloc - before.TotalAlloc)
require.Less(t, growth, tt.limit, "allocated %d MiB reading one reply", growth>>20)
})
}
}
// go-redis peeks for push notifications with a microseconds-long deadline. On Windows that
// peek always runs, and the timeout has to stay on that one read.
func TestLimitedConnDeadlineIsNotSticky(t *testing.T) {
listener, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
t.Cleanup(func() {
_ = listener.Close()
})
accepted := make(chan net.Conn, 1)
go func() {
conn, err := listener.Accept()
if err != nil {
return
}
accepted <- conn
}()
client, err := net.Dial("tcp", listener.Addr().String())
require.NoError(t, err)
t.Cleanup(func() {
_ = client.Close()
})
server := <-accepted
t.Cleanup(func() {
_ = server.Close()
})
lc := &limitedConn{Conn: client, remaining: maxReadBytes}
require.NoError(t, lc.SetReadDeadline(time.Now().Add(-time.Second)))
_, err = lc.Read(make([]byte, 8))
var netErr net.Error
require.ErrorAs(t, err, &netErr)
require.True(t, netErr.Timeout())
_, err = server.Write([]byte("+PONG\r\n"))
require.NoError(t, err)
require.NoError(t, lc.SetReadDeadline(time.Time{}))
buf := make([]byte, 16)
n, err := lc.Read(buf)
require.NoError(t, err)
require.Equal(t, "+PONG\r\n", string(buf[:n]))
}
+2 -2
View File
@@ -29,7 +29,7 @@ func listDir(ctx context.Context, executionId string, host string, port int, use
return nil, err
}
defer sess.Close()
return sess.ListDir(share, dir)
return sess.ListDirContext(ctx, share, dir)
}
// @memo
@@ -55,5 +55,5 @@ func listTree(ctx context.Context, executionId string, host string, port int, us
return nil, err
}
defer sess.Close()
return sess.ListTree(share, dir, smbsession.DefaultMaxTreeDepth, smbsession.DefaultMaxTreeEntries)
return sess.ListTreeContext(ctx, share, dir, smbsession.DefaultMaxTreeDepth, smbsession.DefaultMaxTreeEntries)
}
+71 -10
View File
@@ -7,6 +7,7 @@ import (
"os"
"path"
"strings"
"sync"
"time"
gpsession "github.com/projectdiscovery/goimpacket/pkg/session"
@@ -57,8 +58,10 @@ type shareOpener interface {
// Session wraps an authenticated goimpacket SMB client.
type Session struct {
client *gpsmb.Client
backend shareBackend // when set (tests), used instead of client
client *gpsmb.Client
backend shareBackend // when set (tests), used instead of client
closed chan struct{}
closeOnce sync.Once
}
// Dial connects and authenticates to host:port using the execution's dialer.
@@ -103,15 +106,22 @@ func Dial(ctx context.Context, executionID, host string, port int, creds Creds)
// FromClient wraps an already-connected goimpacket SMB client (e.g. dcerpc's).
func FromClient(client *gpsmb.Client) *Session {
return &Session{client: client}
return &Session{client: client, closed: make(chan struct{})}
}
// Close tears down the SMB session. Safe on nil.
// Close tears down the SMB session. Safe on nil and idempotent.
func (s *Session) Close() {
if s == nil || s.client == nil {
if s == nil {
return
}
s.client.Close()
s.closeOnce.Do(func() {
if s.closed != nil {
close(s.closed)
}
if s.client != nil {
s.client.Close()
}
})
}
// Native returns the underlying goimpacket client for advanced callers.
@@ -146,11 +156,19 @@ func (s *Session) ListShares() ([]string, error) {
// ListDir lists one directory on share (share-relative path).
func (s *Session) ListDir(share, dir string) ([]Entry, error) {
return s.ListDirContext(context.Background(), share, dir)
}
// ListDirContext lists one directory and stops waiting when ctx is cancelled
// by closing the session, matching Dial.
func (s *Session) ListDirContext(ctx context.Context, share, dir string) ([]Entry, error) {
ops := s.ops()
if ops == nil {
return nil, fmt.Errorf("smb session not connected")
}
return listDir(ops, share, dir)
return s.awaitEntries(ctx, func(ctx context.Context) ([]Entry, error) {
return listDir(ctx, ops, share, dir)
})
}
// ReadFile reads a file from share, capped at maxBytes (default DefaultMaxReadBytes).
@@ -164,14 +182,51 @@ func (s *Session) ReadFile(share, filePath string, maxBytes int64) (string, erro
// ListTree walks share directories up to maxDepth / maxEntries.
func (s *Session) ListTree(share, root string, maxDepth, maxEntries int) ([]Entry, error) {
return s.ListTreeContext(context.Background(), share, root, maxDepth, maxEntries)
}
// ListTreeContext walks share directories and stops waiting when ctx is cancelled
// by closing the session, matching Dial.
func (s *Session) ListTreeContext(ctx context.Context, share, root string, maxDepth, maxEntries int) ([]Entry, error) {
ops := s.ops()
if ops == nil {
return nil, fmt.Errorf("smb session not connected")
}
return listTree(ops, share, root, maxDepth, maxEntries)
return s.awaitEntries(ctx, func(ctx context.Context) ([]Entry, error) {
return listTree(ctx, ops, share, root, maxDepth, maxEntries)
})
}
func listDir(ops shareBackend, share, dir string) ([]Entry, error) {
func (s *Session) awaitEntries(ctx context.Context, fn func(context.Context) ([]Entry, error)) ([]Entry, error) {
if ctx == nil {
ctx = context.Background()
}
if err := ctx.Err(); err != nil {
return nil, err
}
type rec struct {
entries []Entry
err error
}
ch := make(chan rec, 1)
go func() {
entries, err := fn(ctx)
ch <- rec{entries, err}
}()
select {
case <-ctx.Done():
s.Close()
<-ch
return nil, ctx.Err()
case r := <-ch:
return r.entries, r.err
}
}
func listDir(ctx context.Context, ops shareBackend, share, dir string) ([]Entry, error) {
if err := ctx.Err(); err != nil {
return nil, err
}
if err := RequireShareName(share); err != nil {
return nil, err
}
@@ -242,7 +297,10 @@ func readFile(ops shareBackend, share, filePath string, maxBytes int64) (string,
return body, nil
}
func listTree(ops shareBackend, share, root string, maxDepth, maxEntries int) ([]Entry, error) {
func listTree(ctx context.Context, ops shareBackend, share, root string, maxDepth, maxEntries int) ([]Entry, error) {
if err := ctx.Err(); err != nil {
return nil, err
}
if err := RequireShareName(share); err != nil {
return nil, err
}
@@ -263,6 +321,9 @@ func listTree(ops shareBackend, share, root string, maxDepth, maxEntries int) ([
var out []Entry
var walk func(rel string, depth int) error
walk = func(rel string, depth int) error {
if err := ctx.Err(); err != nil {
return err
}
if len(out) >= maxEntries {
return fmt.Errorf("tree listing exceeded max entries (%d)", maxEntries)
}
+115
View File
@@ -7,6 +7,7 @@ import (
"os"
"path"
"strings"
"sync"
"testing"
"time"
@@ -59,6 +60,107 @@ func TestListDirOnFakeBackend(t *testing.T) {
require.Equal(t, "backup", fake.mounted)
}
func TestListDirContextAlreadyCancelled(t *testing.T) {
started := make(chan struct{})
hang := &hangBackend{fakeBackend: *newFakeBackend(nil), started: started, wait: make(chan struct{})}
sess := &Session{backend: hang, closed: make(chan struct{})}
ctx, cancel := context.WithCancel(context.Background())
cancel()
_, err := sess.ListDirContext(ctx, "backup", ".")
require.ErrorIs(t, err, context.Canceled)
select {
case <-started:
t.Fatal("Ls must not run when the context is already cancelled")
default:
}
}
func TestListDirContextUnblocksCancelledLs(t *testing.T) {
started := make(chan struct{})
closed := make(chan struct{})
hang := &hangBackend{fakeBackend: *newFakeBackend(nil), started: started, wait: closed}
sess := &Session{backend: hang, closed: closed}
ctx, cancel := context.WithCancel(context.Background())
errCh := make(chan error, 1)
go func() {
_, err := sess.ListDirContext(ctx, "backup", ".")
errCh <- err
}()
select {
case <-started:
case <-time.After(2 * time.Second):
t.Fatal("Ls did not start")
}
cancel()
select {
case err := <-errCh:
require.ErrorIs(t, err, context.Canceled)
case <-time.After(2 * time.Second):
t.Fatal("listing stayed blocked after cancel")
}
}
func TestListTreeContextUnblocksCancelledLs(t *testing.T) {
started := make(chan struct{})
closed := make(chan struct{})
hang := &hangBackend{fakeBackend: *newFakeBackend(nil), started: started, wait: closed}
sess := &Session{backend: hang, closed: closed}
ctx, cancel := context.WithCancel(context.Background())
errCh := make(chan error, 1)
go func() {
_, err := sess.ListTreeContext(ctx, "backup", ".", 2, 100)
errCh <- err
}()
select {
case <-started:
case <-time.After(2 * time.Second):
t.Fatal("Ls did not start")
}
cancel()
select {
case err := <-errCh:
require.ErrorIs(t, err, context.Canceled)
case <-time.After(2 * time.Second):
t.Fatal("tree listing stayed blocked after cancel")
}
}
func TestListDirContextNilContext(t *testing.T) {
fake := newFakeBackend(map[string][]fakeNode{
".": {{name: "a.txt", size: 1, content: "x"}},
})
sess := &Session{backend: fake, closed: make(chan struct{})}
var ctx context.Context
entries, err := sess.ListDirContext(ctx, "backup", ".")
require.NoError(t, err)
require.Len(t, entries, 1)
require.Equal(t, "a.txt", entries[0].Name)
}
func TestListTreeContextNilContext(t *testing.T) {
fake := newFakeBackend(map[string][]fakeNode{
".": {{name: "a.txt", size: 1, content: "x"}},
})
sess := &Session{backend: fake, closed: make(chan struct{})}
var ctx context.Context
entries, err := sess.ListTreeContext(ctx, "backup", ".", 1, 100)
require.NoError(t, err)
require.Len(t, entries, 1)
require.Equal(t, "a.txt", entries[0].Name)
}
func TestCloseIdempotent(t *testing.T) {
sess := &Session{closed: make(chan struct{})}
sess.Close()
sess.Close()
}
func TestListDirRejectsEscapeAndBadShare(t *testing.T) {
sess := &Session{backend: newFakeBackend(nil)}
_, err := sess.ListDir("backup", "../etc")
@@ -173,6 +275,19 @@ type fakeBackend struct {
mounted string
}
type hangBackend struct {
fakeBackend
started chan struct{}
startOnce sync.Once
wait <-chan struct{}
}
func (h *hangBackend) Ls(string) ([]os.FileInfo, error) {
h.startOnce.Do(func() { close(h.started) })
<-h.wait
return nil, context.Canceled
}
func newFakeBackend(dirs map[string][]fakeNode) *fakeBackend {
if dirs == nil {
dirs = map[string][]fakeNode{}
+90 -5
View File
@@ -1,14 +1,17 @@
package smtp
import (
"context"
"crypto/tls"
"errors"
"fmt"
"net"
"net/smtp"
"strconv"
"time"
"github.com/projectdiscovery/goja"
"github.com/praetorian-inc/fingerprintx/pkg/plugins"
"github.com/projectdiscovery/goja"
"github.com/projectdiscovery/nuclei/v3/pkg/js/utils"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
@@ -145,7 +148,10 @@ func (c *Client) IsOpenRelay(msg *SMTPMessage) (bool, error) {
defer func() {
_ = conn.Close()
}()
client, err := smtp.NewClient(conn, c.host)
// net/smtp reads with no deadline, and an interrupt cannot stop the runtime inside that
// read: closing on cancel is what keeps a server that never answers from pinning it.
defer context.AfterFunc(c.nj.Context(), func() { _ = conn.Close() })()
client, err := smtp.NewClient(&limitedConn{Conn: conn, remaining: maxReadBytes}, c.host)
if err != nil {
return false, err
}
@@ -203,10 +209,89 @@ func (c *Client) SendMail(msg *SMTPMessage) (bool, error) {
auth = smtp.PlainAuth("", msg.user, msg.pass, c.host)
}
// send mail
addr := net.JoinHostPort(c.host, c.port)
if err := smtp.SendMail(addr, auth, msg.from, msg.to, []byte(msg.String())); err != nil {
if err := c.sendMail(auth, msg); err != nil {
c.nj.Throw("failed to send mail with message(%s) got %v", msg.String(), err)
}
return true, nil
}
// sendMail is smtp.SendMail over the execution's dialer. smtp.SendMail dials on its own,
// past the network policy, so nothing could bound what it reads or close it on cancel.
func (c *Client) sendMail(auth smtp.Auth, msg *SMTPMessage) error {
executionId := c.nj.ExecutionId()
dialer := protocolstate.GetDialersWithId(executionId)
if dialer == nil {
return fmt.Errorf("dialers not initialized for %s", executionId)
}
conn, err := dialer.Fastdialer.Dial(c.nj.Context(), "tcp", net.JoinHostPort(c.host, c.port))
if err != nil {
return err
}
defer func() {
_ = conn.Close()
}()
defer context.AfterFunc(c.nj.Context(), func() { _ = conn.Close() })()
client, err := smtp.NewClient(&limitedConn{Conn: conn, remaining: maxReadBytes}, c.host)
if err != nil {
return err
}
if ok, _ := client.Extension("STARTTLS"); ok {
if err := client.StartTLS(&tls.Config{ServerName: c.host}); err != nil {
return err
}
}
if auth != nil {
if ok, _ := client.Extension("AUTH"); !ok {
return errors.New("smtp: server doesn't support AUTH")
}
if err := client.Auth(auth); err != nil {
return err
}
}
if err := client.Mail(msg.from); err != nil {
return err
}
for _, to := range msg.to {
if err := client.Rcpt(to); err != nil {
return err
}
}
wc, err := client.Data()
if err != nil {
return err
}
if _, err := wc.Write([]byte(msg.String())); err != nil {
return err
}
if err := wc.Close(); err != nil {
return err
}
return client.Quit()
}
// maxReadBytes bounds what one connection may read. net/smtp reads replies through
// net/textproto, which buffers a line until "\n" with no limit; a whole dialogue, TLS
// handshake included, fits in tens of kilobytes.
const maxReadBytes = 1 << 20
var errReadLimit = fmt.Errorf("smtp: server sent more than %d bytes", maxReadBytes)
// limitedConn fails reads once remaining is spent.
type limitedConn struct {
net.Conn
remaining int64
}
func (c *limitedConn) Read(b []byte) (int, error) {
if c.remaining <= 0 {
return 0, errReadLimit
}
if int64(len(b)) > c.remaining {
b = b[:c.remaining]
}
n, err := c.Conn.Read(b)
c.remaining -= int64(n)
return n, err
}
+180
View File
@@ -0,0 +1,180 @@
package smtp
import (
"bytes"
"context"
"net"
"net/textproto"
"runtime"
"strings"
"testing"
"time"
"github.com/projectdiscovery/goja"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/stretchr/testify/require"
)
// A port that streams bytes without a newline used to be buffered whole: net/smtp reads replies
// through net/textproto, which reads a line until "\n" with no length limit.
func TestCallsBoundAReplyWithoutNewline(t *testing.T) {
tests := []struct {
name string
call func(*Client, *SMTPMessage)
}{
{"SendMail", func(c *Client, m *SMTPMessage) { _, _ = c.SendMail(m) }},
{"IsOpenRelay", func(c *Client, m *SMTPMessage) { _, _ = c.IsOpenRelay(m) }},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
listener := listenForLimitTest(t)
go streamWithoutNewline(listener)
client := newSMTPClientForLimitTest(t, listener.Addr().String())
msg := (&SMTPMessage{}).From("[email protected]").To("[email protected]")
allocated := allocatedBy(func() {
defer func() {
_ = recover() // SendMail throws its error to the runtime
}()
tt.call(client, msg)
})
require.Less(t, allocated, uint64(64<<20), "allocated %d MiB reading one reply", allocated>>20)
})
}
}
// The cap must leave room for a normal dialogue.
func TestSendMailStillDeliversUnderTheReadLimit(t *testing.T) {
listener := listenForLimitTest(t)
received := serveOneSMTPSessionForLimitTest(listener)
client := newSMTPClientForLimitTest(t, listener.Addr().String())
msg := (&SMTPMessage{}).From("[email protected]").To("[email protected]").Subject("hello").Body([]byte("world"))
sent, err := client.SendMail(msg)
require.NoError(t, err)
require.True(t, sent)
select {
case data := <-received:
require.Contains(t, data, "Subject: hello")
require.Contains(t, data, "world")
case <-time.After(5 * time.Second):
t.Fatal("the server never saw the message")
}
}
func newSMTPClientForLimitTest(t *testing.T, addr string) *Client {
t.Helper()
executionID := "smtp-" + strings.NewReplacer("/", "-", " ", "-").Replace(t.Name())
require.NoError(t, protocolstate.Init(&types.Options{ExecutionId: executionID}))
t.Cleanup(func() {
protocolstate.Close(executionID)
})
vm := goja.New()
vm.SetContextValue("executionId", executionID)
vm.SetContextValue("ctx", context.Background())
host, port, err := net.SplitHostPort(addr)
require.NoError(t, err)
obj, err := vm.New(vm.ToValue(NewSMTPClient), vm.ToValue(host), vm.ToValue(port))
require.NoError(t, err)
client, ok := obj.Export().(*Client)
require.True(t, ok, "expected *Client export, got %T", obj.Export())
return client
}
func listenForLimitTest(t *testing.T) net.Listener {
t.Helper()
listener, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
t.Cleanup(func() {
_ = listener.Close()
})
return listener
}
// streamWithoutNewline sends 256 MiB without a newline to each connection: bounded so the test
// cannot exhaust the machine when the limit is missing.
func streamWithoutNewline(listener net.Listener) {
const streamBytes = 256 << 20
chunk := bytes.Repeat([]byte("A"), 32<<10)
for {
conn, err := listener.Accept()
if err != nil {
return
}
go func() {
defer func() {
_ = conn.Close()
}()
for sent := 0; sent < streamBytes; sent += len(chunk) {
if _, err := conn.Write(chunk); err != nil {
return
}
}
}()
}
}
// allocatedBy reports how many bytes call allocated, freed or not.
func allocatedBy(call func()) uint64 {
var before, after runtime.MemStats
runtime.ReadMemStats(&before)
call()
runtime.ReadMemStats(&after)
return after.TotalAlloc - before.TotalAlloc
}
// serveOneSMTPSessionForLimitTest answers a single session like a minimal MTA with no
// extensions, and reports the DATA it was sent once the client quits.
func serveOneSMTPSessionForLimitTest(listener net.Listener) <-chan string {
received := make(chan string, 1)
go func() {
conn, err := listener.Accept()
if err != nil {
return
}
defer func() {
_ = conn.Close()
}()
tp := textproto.NewConn(conn)
var data string
_ = tp.PrintfLine("220 test ESMTP")
for {
line, err := tp.ReadLine()
if err != nil {
return
}
switch verb, _, _ := strings.Cut(line, " "); strings.ToUpper(verb) {
case "EHLO", "HELO":
_ = tp.PrintfLine("250 test")
case "MAIL", "RCPT":
_ = tp.PrintfLine("250 ok")
case "DATA":
_ = tp.PrintfLine("354 go ahead")
body, err := tp.ReadDotBytes()
if err != nil {
return
}
data = string(body)
_ = tp.PrintfLine("250 queued")
case "QUIT":
_ = tp.PrintfLine("221 bye")
received <- data
return
default:
_ = tp.PrintfLine("502 not implemented")
}
}
}()
return received
}
+201
View File
@@ -0,0 +1,201 @@
package smtp
import (
"context"
"net"
"net/textproto"
"strings"
"testing"
"time"
"github.com/projectdiscovery/goja"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
)
type receivedMail struct {
from string
to []string
data string
}
func TestSendMailDeliversTheMessage(t *testing.T) {
listener := listenForTest(t)
received := serveOneSMTPSession(listener)
client := newSMTPClientForTest(t, context.Background(), listener.Addr().String())
msg := (&SMTPMessage{}).From("[email protected]").To("[email protected]").Subject("hello").Body([]byte("world"))
sent, err := client.SendMail(msg)
if err != nil || !sent {
t.Fatalf("SendMail() = %v, %v; want true, nil", sent, err)
}
select {
case mail := <-received:
if !strings.Contains(mail.from, "<[email protected]>") {
t.Errorf("MAIL command = %q, want the sender", mail.from)
}
if len(mail.to) != 1 || !strings.Contains(mail.to[0], "<[email protected]>") {
t.Errorf("RCPT commands = %q, want the one recipient", mail.to)
}
if !strings.Contains(mail.data, "Subject: hello") || !strings.Contains(mail.data, "world") {
t.Errorf("DATA = %q, want the subject and body", mail.data)
}
case <-time.After(5 * time.Second):
t.Fatal("the server never saw the session end")
}
}
// A server that accepts and never answers used to keep these calls, and the runtime
// abandoned around them, blocked for the life of the process.
func TestCallsReturnWhenTheContextIsDone(t *testing.T) {
tests := []struct {
name string
call func(*Client, *SMTPMessage)
}{
{"SendMail", func(c *Client, m *SMTPMessage) { _, _ = c.SendMail(m) }},
{"IsOpenRelay", func(c *Client, m *SMTPMessage) { _, _ = c.IsOpenRelay(m) }},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
listener := listenForTest(t)
go holdConnections(listener)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
client := newSMTPClientForTest(t, ctx, listener.Addr().String())
msg := (&SMTPMessage{}).From("[email protected]").To("[email protected]")
done := make(chan struct{})
go func() {
defer close(done)
defer func() {
_ = recover() // SendMail throws its error to the runtime
}()
tt.call(client, msg)
}()
select {
case <-done:
t.Fatal("returned before the server answered or the context was done")
case <-time.After(200 * time.Millisecond):
}
cancel()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("still blocked after the context was done")
}
})
}
}
func newSMTPClientForTest(t *testing.T, ctx context.Context, addr string) *Client {
t.Helper()
executionID := "smtp-" + strings.NewReplacer("/", "-", " ", "-").Replace(t.Name())
if err := protocolstate.Init(&types.Options{ExecutionId: executionID}); err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
protocolstate.Close(executionID)
})
runtime := goja.New()
runtime.SetContextValue("executionId", executionID)
runtime.SetContextValue("ctx", ctx)
host, port, err := net.SplitHostPort(addr)
if err != nil {
t.Fatal(err)
}
obj, err := runtime.New(runtime.ToValue(NewSMTPClient), runtime.ToValue(host), runtime.ToValue(port))
if err != nil {
t.Fatal(err)
}
client, ok := obj.Export().(*Client)
if !ok {
t.Fatalf("expected *Client export, got %T", obj.Export())
}
return client
}
func listenForTest(t *testing.T) net.Listener {
t.Helper()
listener, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
_ = listener.Close()
})
return listener
}
// holdConnections accepts and never answers, like a tarpit.
func holdConnections(listener net.Listener) {
var held []net.Conn
for {
conn, err := listener.Accept()
if err != nil {
for _, c := range held {
_ = c.Close()
}
return
}
held = append(held, conn)
}
}
// serveOneSMTPSession answers a single session like a minimal MTA with no extensions,
// and reports what it was sent once the client quits.
func serveOneSMTPSession(listener net.Listener) <-chan receivedMail {
received := make(chan receivedMail, 1)
go func() {
conn, err := listener.Accept()
if err != nil {
return
}
defer func() {
_ = conn.Close()
}()
tp := textproto.NewConn(conn)
var mail receivedMail
_ = tp.PrintfLine("220 test ESMTP")
for {
line, err := tp.ReadLine()
if err != nil {
return
}
switch verb, _, _ := strings.Cut(line, " "); strings.ToUpper(verb) {
case "EHLO", "HELO":
_ = tp.PrintfLine("250 test")
case "MAIL":
mail.from = line
_ = tp.PrintfLine("250 ok")
case "RCPT":
mail.to = append(mail.to, line)
_ = tp.PrintfLine("250 ok")
case "DATA":
_ = tp.PrintfLine("354 go ahead")
body, err := tp.ReadDotBytes()
if err != nil {
return
}
mail.data = string(body)
_ = tp.PrintfLine("250 queued")
case "QUIT":
_ = tp.PrintfLine("221 bye")
received <- mail
return
default:
_ = tp.PrintfLine("502 not implemented")
}
}
}()
return received
}
+63
View File
@@ -1,9 +1,13 @@
package model
import (
"slices"
"strings"
"github.com/invopop/jsonschema"
"github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
"github.com/projectdiscovery/nuclei/v3/pkg/model/types/stringslice"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/marker"
)
type schemaMetadata struct {
@@ -47,15 +51,28 @@ type Info struct {
//
// You can go in-depth here on what the template actually does.
//
// Description, impact and remediation may hold {{placeholders}} filled for
// each finding: named extractors, payloads, variables and DSL helpers
// (network helpers excluded), and the result fields host, port, ip, url,
// scheme, path, matched-at, matcher-name and extractor-name, which win over
// values of the same name. Use join(", ", name) for an extractor with
// several values, and brackets for a hyphenated name inside a helper:
// to_upper([matcher-name]). Unknown placeholders stay as written, and so
// does a field whose expression fails. Global matcher templates match on
// other templates' responses, so only the result fields fill their text.
//
// examples:
// - value: "\"Bower is a package manager which stores package information in the bower.json file\""
// - value: "\"Subversion ALM for the enterprise before 8.8.2 allows reflected XSS at multiple locations\""
// - value: "\"Redis {{version}} on {{host}}:{{port}} accepts the default password\""
Description string `json:"description,omitempty" yaml:"description,omitempty" jsonschema:"title=description of the template,description=In-depth explanation on what the template does,type=string,example=Bower is a package manager which stores package information in the bower.json file"`
// description: |
// Impact of the template.
//
// You can go in-depth here on impact of the template.
//
// Supports the same placeholders as description.
//
// examples:
// - value: "\"Successful exploitation of this vulnerability could allow an attacker to execute arbitrary SQL queries, potentially leading to unauthorized access, data leakage, or data manipulation.\""
// - value: "\"Successful exploitation of this vulnerability could allow an attacker to execute arbitrary script code in the context of the victim's browser, potentially leading to session hijacking, defacement, or theft of sensitive information.\""
@@ -89,11 +106,57 @@ type Info struct {
//
// You can go in-depth here on how to mitigate the problem found by this template.
//
// Supports the same placeholders as description.
//
// examples:
// - value: "\"Change the default administrative username and password of Apache ActiveMQ by editing the file jetty-realm.properties\""
Remediation string `json:"remediation,omitempty" yaml:"remediation,omitempty" jsonschema:"title=remediation steps for the template,description=In-depth explanation on how to fix the issues found by the template,example=Change the default administrative username and password of Apache ActiveMQ by editing the file jetty-realm.properties,type=string"`
}
// TextField is a free-text field of the info, labeled as reports show it.
type TextField struct {
Label string
Value *string
}
// PlaceholderFields returns the free-text fields that may hold {{placeholders}}
// filled for each finding. Rendering, redaction and the sarif export all walk
// this one list, so a field added here is handled by every one of them.
func (i *Info) PlaceholderFields() []TextField {
return []TextField{
{Label: "Description", Value: &i.Description},
{Label: "Impact", Value: &i.Impact},
{Label: "Remediation", Value: &i.Remediation},
}
}
// HasPlaceholders reports whether the field holds a {{...}} or §...§ marker
// the renderer could fill.
func (f TextField) HasPlaceholders() bool {
return strings.Contains(*f.Value, marker.ParenthesisOpen) || strings.Contains(*f.Value, marker.General)
}
// HasPlaceholders reports whether any placeholder field holds a marker.
func (i *Info) HasPlaceholders() bool {
return slices.ContainsFunc(i.PlaceholderFields(), TextField.HasPlaceholders)
}
// FilledFields returns the placeholder fields whose text differs from written,
// the info as the template wrote it. A nil written means nothing was filled.
func (i *Info) FilledFields(written *Info) []TextField {
if written == nil {
return nil
}
writtenFields := written.PlaceholderFields()
var filled []TextField
for n, field := range i.PlaceholderFields() {
if *field.Value != *writtenFields[n].Value {
filled = append(filled, field)
}
}
return filled
}
// JSONSchemaProperty returns the JSON schema property for the Info object.
func (i Info) JSONSchemaExtend(base *jsonschema.Schema) {
// since we are re-using a stringslice and rawStringSlice everywhere, we can extend/edit the schema here
+18
View File
@@ -135,3 +135,21 @@ func TestUnmarshal(t *testing.T) {
info2 := assertUnmarshalledTemplateInfo(t, yamlPayload2)
require.Equal(t, info1, info2)
}
func TestInfoFilledFields(t *testing.T) {
written := Info{Description: "{{host}} is exposed", Impact: "Data loss"}
filled := written
filled.Description = "example.com is exposed"
require.Nil(t, filled.FilledFields(nil))
fields := filled.FilledFields(&written)
require.Len(t, fields, 1)
require.Equal(t, "Description", fields[0].Label)
require.Equal(t, "example.com is exposed", *fields[0].Value)
}
func TestInfoHasPlaceholders(t *testing.T) {
require.False(t, (&Info{Description: "Redis is exposed"}).HasPlaceholders())
require.True(t, (&Info{Impact: "{{host}} is exposed"}).HasPlaceholders())
require.True(t, (&Info{Remediation: "rotate §password§"}).HasPlaceholders())
}
+9
View File
@@ -8,6 +8,7 @@ import (
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/govaluate"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/useragent"
stringsutil "github.com/projectdiscovery/utils/strings"
)
@@ -41,6 +42,14 @@ func init() {
}
return port, nil
}))
// cacheable is false on purpose: a cached result would return the same
// user agent for the rest of the run, defeating the randomization.
_ = dsl.AddFunction(dsl.NewWithSingleSignature("rand_user_agent", "() string", false, func(args ...interface{}) (interface{}, error) {
if len(args) != 0 {
return nil, dsl.ErrInvalidDslFunction
}
return useragent.PickRandom().Raw, nil
}))
dsl.PrintDebugCallback = func(args ...interface{}) error {
gologger.Debug().Msgf("print_debug value: %s", fmt.Sprint(args...))
@@ -0,0 +1,33 @@
package dsl
import (
"testing"
"github.com/projectdiscovery/govaluate"
"github.com/stretchr/testify/require"
)
// TestRandUserAgentIsNotCached guards the cacheable=false registration: a
// cached helper would hand back one frozen user agent for the whole run.
func TestRandUserAgentIsNotCached(t *testing.T) {
compiled, err := govaluate.NewEvaluableExpressionWithFunctions("rand_user_agent()", HelperFunctions)
require.NoError(t, err, "could not compile rand_user_agent()")
seen := make(map[string]struct{})
for i := 0; i < 50; i++ {
result, err := compiled.Evaluate(make(map[string]interface{}))
require.NoError(t, err, "could not evaluate rand_user_agent()")
require.NotEmpty(t, result, "rand_user_agent() returned an empty user agent")
seen[result.(string)] = struct{}{}
}
require.Greater(t, len(seen), 1, "rand_user_agent() returned a cached value")
}
func TestRandUserAgentRejectsArguments(t *testing.T) {
compiled, err := govaluate.NewEvaluableExpressionWithFunctions(`rand_user_agent("chrome")`, HelperFunctions)
require.NoError(t, err, "could not compile rand_user_agent() with an argument")
_, err = compiled.Evaluate(make(map[string]interface{}))
require.Error(t, err, "rand_user_agent() should reject arguments")
}
+300
View File
@@ -0,0 +1,300 @@
package operators
import (
"strings"
"github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
)
// Confidence tiers reported alongside a result. They express how reliable the
// detection is, independent of severity (how bad the issue is). The split
// follows the Burp "tentative/firm/certain" and OpenVAS QoD models.
const (
ConfidenceLow = "low"
ConfidenceMedium = "medium"
ConfidenceHigh = "high"
)
// confidence is reported on a 0-100 scale modelled on OpenVAS "Quality of
// Detection" (QoD). Tier thresholds align with the scale; OpenVAS uses 70 as its
// default trustworthy cutoff, which sits inside the medium band here.
const (
confidenceMediumThreshold = 50
confidenceHighThreshold = 80
// ConfidenceCertain is the score for out-of-band confirmed detections
// (e.g. an interactsh interaction): unambiguous proof, so the maximum.
ConfidenceCertain = 100
// ConfidenceCatchAll is the clamped score for matches that also fire against
// the host's catch-all baseline: a strong false-positive signal.
ConfidenceCatchAll = 10
)
// Base reliability of each evidence class. The ordering follows the black-box
// scanner literature and QoD detection methods: matching on a status code or
// response size is the most false-positive prone form of inference (an HTTP 500
// can be any error, a banner can be backported), matching content with a
// word is moderate, and matching content with a regex/dsl/binary/xpath is the
// strongest in-band signal. Out-of-band confirmation (handled by the caller)
// outranks all of these.
const (
weightStatusOrSize = 25
// weightLLM scores an llm matcher. A model verdict is a judgement rather
// than a deterministic check of the response, so it sits below word
// matching and needs deterministic matchers to reach a higher tier.
weightLLM = 40
weightWordHeader = 45
weightStrongHeader = 55
weightWordContent = 60
weightStrongContent = 75
// corroboration adds confidence only across DISTINCT, AND-combined evidence
// classes (independent confirmation), with diminishing returns.
corroborationPerClass = 10
corroborationCap = 20
negativeMatcherBonus = 5
extractorBonus = 3
// multiPatternBonus rewards a single matcher that requires several distinct
// patterns to all be present (condition: and / match-all). Needing many
// independent strings is far harder to trigger by accident than one.
multiPatternBonus = 8
)
// dslWeakIndicators are DSL variables that only inspect response metadata.
// dslContentIndicators signal that a DSL expression actually inspects the
// response body/headers, which is a far stronger detection method.
var (
dslWeakIndicators = []string{"status_code", "content_length", "duration"}
dslContentIndicators = []string{"body", "header", "all_headers", "contains", "regex", "len(", "tolower", "toupper", "title", "hash", "md5", "sha", "base64", "html", "response", "data"}
)
// dslWeakOnly reports whether a DSL matcher only inspects status/size metadata.
// Such a matcher is a "strong" matcher type but carries status-level reliability
// (an HTTP 500 can be any error), so it must not be scored as body evidence.
func dslWeakOnly(m *matchers.Matcher) bool {
if len(m.DSL) == 0 {
return false
}
weak := false
for _, expr := range m.DSL {
e := strings.ToLower(expr)
for _, c := range dslContentIndicators {
if strings.Contains(e, c) {
return false
}
}
for _, w := range dslWeakIndicators {
if strings.Contains(e, w) {
weak = true
}
}
}
return weak
}
// requiresAllPatterns reports whether every pattern in the matcher must be
// present (condition: and or match-all), rather than any single one.
func requiresAllPatterns(m *matchers.Matcher) bool {
return m.MatchAll || strings.EqualFold(m.Condition, "and")
}
// patternCount counts the independent content patterns a matcher carries.
// Status/size codes are intentionally excluded: matching any of several status
// codes broadens rather than narrows a match.
func patternCount(m *matchers.Matcher) int {
return len(m.Words) + len(m.Regex) + len(m.Binary) + len(m.DSL) + len(m.XPath)
}
// metadataParts are response parts that carry weak, false-positive-prone signal
// across every protocol: protocol metadata (status/size/rcode/duration), HTTP
// response headers (the generic part and the common header fields used directly)
// and request-side echoes. Matching these is more backportable / accidental than
// matching real response content. Everything not listed here (response bodies,
// DNS answer/ns/extra sections, TLS certificate fields, command stdout/stderr,
// websocket/whois/javascript payloads, ...) is treated as content, so non-HTTP
// protocols are scored on the strength of what they actually inspect.
var metadataParts = map[string]struct{}{
"header": {}, "all_headers": {}, "headers": {},
"status": {}, "status_code": {}, "size": {}, "content_length": {}, "duration": {},
"rcode": {},
"request": {}, "host": {}, "ip": {}, "port": {}, "matched": {}, "type": {},
// common HTTP response header fields that templates match on directly
"content_type": {}, "content-type": {}, "server": {}, "location": {},
"set_cookie": {}, "set-cookie": {}, "www_authenticate": {}, "www-authenticate": {},
"x_powered_by": {}, "x-powered-by": {}, "cache_control": {}, "cache-control": {},
}
// stripIndexSuffix removes the _<n> suffix that flow / multi-request templates
// append to response parts (body_2, header_3, content_type_2) so they classify
// the same as their base part.
func stripIndexSuffix(p string) string {
i := strings.LastIndexByte(p, '_')
if i <= 0 || i == len(p)-1 {
return p
}
for j := i + 1; j < len(p); j++ {
if p[j] < '0' || p[j] > '9' {
return p
}
}
return p[:i]
}
// isContentPart reports whether a matcher part inspects response content rather
// than protocol metadata or headers. Content is the default: only the known
// weak metadata parts are excluded. Content matches are harder to trigger by
// accident, so they carry higher confidence weight.
func isContentPart(part string) bool {
p := stripIndexSuffix(strings.ToLower(strings.TrimSpace(part)))
_, meta := metadataParts[p]
return !meta
}
// evidenceClass identifies an independent line of evidence. Corroboration is
// rewarded across distinct classes, not redundant repeats of the same weak
// matcher, so three status matchers do not masquerade as strong evidence.
type evidenceClass struct {
kind string
content bool
}
// classify returns the evidence class and base reliability weight for a matcher.
func classify(m *matchers.Matcher) (evidenceClass, int) {
content := isContentPart(m.Part)
switch m.Type.MatcherType {
case matchers.StatusMatcher:
return evidenceClass{kind: "status"}, weightStatusOrSize
case matchers.SizeMatcher:
return evidenceClass{kind: "size"}, weightStatusOrSize
case matchers.WordsMatcher:
if content {
return evidenceClass{kind: "word", content: true}, weightWordContent
}
return evidenceClass{kind: "word"}, weightWordHeader
case matchers.LLMMatcher:
// one class whatever part it reads: several llm matchers are not
// independent evidence, but an llm matcher AND-combined with a
// deterministic matcher still counts as corroboration.
return evidenceClass{kind: "llm"}, weightLLM
case matchers.DSLMatcher:
if dslWeakOnly(m) {
// a dsl that only checks status_code/content_length is metadata
// inference, not body evidence, despite being a "strong" type.
return evidenceClass{kind: "status"}, weightStatusOrSize
}
if content {
return evidenceClass{kind: "strong", content: true}, weightStrongContent
}
return evidenceClass{kind: "strong"}, weightStrongHeader
default: // regex, binary, xpath
if content {
return evidenceClass{kind: "strong", content: true}, weightStrongContent
}
return evidenceClass{kind: "strong"}, weightStrongHeader
}
}
// Confidence computes a static, deterministic detection-confidence score (0-100)
// and tier from the template's matcher composition. It rewards strong matcher
// techniques, content-level matching, independent corroboration across distinct
// AND-combined evidence classes and negative matchers, while capping redundant
// weak matchers. Hidden (internal) helper matchers are ignored. The score is
// derived from the template definition alone and sends no additional traffic;
// out-of-band confirmation is folded in by the caller.
func (operators *Operators) Confidence() (int, string) {
if operators == nil || len(operators.Matchers) == 0 {
// extraction-only or matcher-less detections carry the weakest signal
return 0, ConfidenceLow
}
// positive (direct) evidence and negative (absence) evidence are tracked
// separately: a negative matcher behaves as a false-positive guard when it
// accompanies positive evidence, but is the detection itself when it stands
// alone (e.g. token-spray validity checks, "setting reverted" CVE steps).
classes := make(map[evidenceClass]struct{})
best := 0
hasMultiPattern := false
negClasses := make(map[evidenceClass]struct{})
negBest := 0
negMultiPattern := false
negCount := 0
for _, m := range operators.Matchers {
if m == nil || m.Internal {
// internal matchers feed dynamic extraction chains and are hidden
// from output, so they are not evidence for the user-facing match.
continue
}
class, weight := classify(m)
// a single matcher that requires several distinct content patterns to
// all hold is highly specific. Status/size and weak-dsl matchers are
// excluded (weight gate) since matching any of several codes broadens.
// An llm matcher gets no bonus: its inputs are not independent patterns.
multi := weight > weightStatusOrSize && class.kind != "llm" && requiresAllPatterns(m) && patternCount(m) > 1
if m.Negative {
negCount++
negClasses[class] = struct{}{}
if weight > negBest {
negBest = weight
}
if multi {
negMultiPattern = true
}
continue
}
classes[class] = struct{}{}
if weight > best {
best = weight
}
if multi {
hasMultiPattern = true
}
}
// a negative matcher is a guard only when it backs up positive evidence.
negativeAsGuard := best > 0 && negCount > 0
if best == 0 {
// no positive evidence: the absence assertions are the detection. Score
// them by their own weight rather than discarding the match entirely.
best, classes, hasMultiPattern = negBest, negClasses, negMultiPattern
}
if best == 0 {
return 0, ConfidenceLow
}
score := best
if hasMultiPattern {
score += multiPatternBonus
}
// Independent corroboration raises confidence, but only when every matcher
// must hold (AND) and only across distinct evidence classes.
if operators.matchersCondition == matchers.ANDCondition && len(classes) > 1 {
bonus := (len(classes) - 1) * corroborationPerClass
if bonus > corroborationCap {
bonus = corroborationCap
}
score += bonus
}
if negativeAsGuard {
score += negativeMatcherBonus
}
if len(operators.Extractors) > 0 {
score += extractorBonus
}
if score > ConfidenceCertain {
score = ConfidenceCertain
}
return score, ScoreToTier(score)
}
// ScoreToTier maps a 0-100 confidence score to its tier.
func ScoreToTier(score int) string {
switch {
case score >= confidenceHighThreshold:
return ConfidenceHigh
case score >= confidenceMediumThreshold:
return ConfidenceMedium
default:
return ConfidenceLow
}
}
Loaded 100 of 233 files, more files were not shown because too many files have changed in this diff. Show more