Commit Graph
277 Commits
Author SHA1 Message Date
Mzack9999 fe24f8638b review fixes 2026-11-04 05:09:15 +01:00
Mzack9999 59ae4cac04 tls dialer 2026-11-04 03:00:58 +01:00
Doğan Can Bakır 613e581542 calibrate against the host root and cover the http hook 2026-11-01 18:53:48 +03:00
Doğan Can Bakır 5c9f02a4fc drop http matches on a host's catch-all page with -auto-calibrate 2026-11-01 18:35:12 +03:00
Mzack9999 abcdcf19a4 Merge pull request #7940 from mazzz3r/feat/info-placeholders
feat(templates): fill placeholders in info description, impact and remediation
2026-10-27 15:21:09 +01:00
Mzack9999 731d438319 Merge pull request #7690 from IgorDaniel45/fix-threadsafe-compiled-template-cache
fix: reuse compiled templates in thread-safe engine
2026-10-25 22:09:59 +01:00
Mzack9999 93483eaff9 merge dev 2026-10-25 21:46:30 +01:00
Mzack9999 fa3fa92092 merge dev 2026-10-25 21:15:30 +01:00
mazzz3r c4552c6011 feat(templates): fill placeholders in info description, impact and remediation
Description, impact and remediation can reference values of the finding
with the usual {{...}} syntax: named extractors, payload values of the
matching request, variables, DSL helpers, and the result fields host,
port, ip, url, scheme, path, matched-at, matcher-name and
extractor-name, which win over values of the same name. Name, severity,
tags and classification stay static, since filtering, grouping and
issue trackers key on them.

MakeDefaultResultEvent renders each result's copy of the info, so every
protocol gets it and the template never changes. Each field goes
through render.Render without scan options, like request and matcher
text, so nested placeholders and helpers behave the same and network
helpers stay denied. A field whose expression fails keeps the
template's text.

When placeholders change the text, the result keeps the written info in
TemplateInfo (json:"-" bson:"-"):
- the writer runs the -redact patterns over the changed fields, as it
  does over the raw request and response;
- SARIF builds the shared rule from the written info and puts each
  finding's filled text, capped at 4096 characters, in its result
  message.

Info is filled only from a template's own event. Templates with info
placeholders are not clustered, since a cluster's shared event holds the
first template's variables and extracted values, and a global matcher
fills its info from the result fields alone.
2026-10-25 19:37:42 +00:00
Mzack9999 f846a5553d Merge pull request #7863 from mazzz3r/feat/target-line-ip
feat(input): pin a target to an ip with the ip key of target lines
2026-10-25 18:56:19 +01:00
Mzack9999 39c5d8764e Merge pull request #7928 from projectdiscovery/fix/http-large-request-body-memory
http: stop copying large request bodies (fixes OOM on multi-MB body templates)
2026-10-25 18:54:34 +01:00
Mzack9999 445e6ab972 Merge pull request #7753 from gnuletik/perf/lazy-full-response
perf(http): build the full response only when an operator reads it
2026-10-25 18:24:42 +01:00
Georgii Butakov 947a2a9a2a fix(http): dial pipelined requests through the pinned ip
executeTurboHTTP dialed pipelined connections with context.Background(),
so a pipelined request of a pinned target resolved the host instead of
connecting to its ip. It now dials with the input context, which carries
the pin. protocols/http/target-line-ip-pipeline.yaml runs the pinned
target test with pipeline: true, and gets no result without the change.

ProbeKey joins the input and the ip with a NUL byte, which no input
holds, instead of "@", which a scheme-less input may hold itself.
2026-10-25 17:12:13 +00:00
Mzack9999 8e1e613a15 merge dev 2026-10-25 18:10:14 +01:00
Mzack9999 5a339a8da1 fix lint 2026-10-25 17:56:27 +01:00
Mzack9999 c4d933d165 merge dev 2026-10-25 17:50:20 +01:00
Mzack9999 f58b21e6c3 merge dev 2026-10-25 17:45:44 +01:00
Mzack9999 71805f5093 Merge pull request #7857 from AdamMagued/fix-issue-7829
perf(protocols/http): defer building FullResponseString when unused by operators
2026-10-25 17:33:54 +01:00
Mzack9999 1086d5d391 keep matchers 2026-10-25 17:32:15 +01:00
Mzack9999 e2840f3644 fix lint 2026-10-25 17:09:07 +01:00
Mzack9999 f1e79cd82b fix response 2026-10-25 16:54:03 +01:00
Mzack9999 4ae1faa64f merge dev 2026-10-25 16:53:53 +01:00
Georgii Butakov 43f99eadde Merge branch 'dev' into feat/target-line-ip
dev requires fastdialer v0.6.0, which has WithResolveOverride and the
dial cache fix, so go.mod and go.sum take dev's versions and the
fastdialer replace is dropped.
2026-10-25 15:14:50 +00:00
Mzack9999 319baa3152 merge dev 2026-10-25 15:51:39 +01:00
Ice3manandClaude Opus 5.5 2954b81c85 http: stop copying large request bodies
Templates that send multi-MB bodies (e.g. CVE-2026-109618 sends 1, 8 and
20 MB) kept 8-10 copies of each body alive per request. With 25 hosts in
flight this reached ~10 GB RSS and OOM-killed scan VMs.

- add pkg/utils/httpbody: a zero-copy request body that rewinds on EOF;
  GetBody hands out new cursors over the same slice instead of cloning
- build raw, model and fuzzed request bodies with httpbody instead of
  retryablehttp's ReusableReadCloser, which tees every read into a second
  growing buffer and clones on each GetBody
- dump requests with http.Request.Write into a capped writer instead of
  retryablehttp Dump/httputil.DumpRequestOut; output is unchanged for
  bodies under the cap
- keep only the first 32 KiB of a body in request dumps (new
  -rqss/-request-size-save flag, 0 keeps all) and leave bodies over 4 KiB
  out of curl commands
- build the project cache key only when a project file is used, hashing
  the body when the dump is truncated
- replay sends the full body when the dump was truncated
- slice the raw request body from the request instead of reading it out
- pass replacer values to fasttemplate as []byte so writes don't copy them

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 18:48:04 +05:30
Mzack9999 e1f656033d fix idle 2026-10-25 13:39:50 +01:00
Mzack9999 9301048bc4 merge dev 2026-10-25 13:37:52 +01:00
Mzack9999 40e165a065 merge dev 2026-10-24 00:37:21 +02:00
Mzack9999 25c4b0d623 Merge pull request #7799 from CalvinTjoaquinn/fix/raw-header-without-colon
fix(raw): two defects in the raw header loop
2026-10-24 00:35:12 +02:00
Mzack9999 20c1510069 merge dev 2026-10-24 00:03:13 +02:00
Mzack9999 6ab068a356 merge dev 2026-10-24 00:01:57 +02:00
Mzack9999 332cfd5372 merge dev 2026-10-23 23:37:09 +02:00
Mzack9999 cebb1a45b0 merge dev 2026-10-23 23:32:09 +02:00
Mzack9999 5dde4c2b30 retry https 2026-10-23 23:16:53 +02:00
Mzack9999 5c3c00f2fa guard limit 2026-10-23 22:31:36 +02:00
Mzack9999 ab5c51640e merge dev 2026-10-23 22:30:05 +02:00
Mzack9999 1248a14a62 merge dev 2026-10-23 18:03:30 +02:00
Georgii Butakov c1fcd4d8d2 Merge branch 'dev' into feat/target-line-ip
go.mod takes dev's versions, including fastdialer v0.5.26, and keeps the
fastdialer replace.
2026-10-21 20:25:04 +00:00
TassioSales 5cf7f3ba47 fix(http): scope projectCacheKey with effective body read limit
Include the effective read limit in projectCacheKey so requests with different
limits (such as small template max-size previews) do not share truncated cached
bodies with larger reads.
2026-10-20 21:07:35 -03:00
TassioSales c918d010c5 fix(http): honour template max-size under global response-size-read limit
When -response-size-read (-rsr) is configured, use it as a ceiling while
preserving a template's own MaxSize when smaller, rather than unconditionally
overwriting it. Also account for ResponseReadSize in response truncation detection.

Fixes #7873
2026-10-20 21:07:35 -03:00
Mzack9999 b4761d4fc4 merge dev 2026-10-19 15:05:46 +02:00
Mzack9999 e4d1535691 merge dev 2026-10-19 14:26:55 +02:00
Doğan Can Bakır c8a0d45d38 apply host backoff to network, ssl and websocket dials 2026-10-19 09:10:52 +03:00
Mzack9999 767655e306 copy bodies 2026-10-18 16:29:21 +02:00
TassioSales b296cc0d46 fix(sdk): prevent custom-port HTTP failures from poisoning base host cache and filter skip events (#7859) 2026-10-17 13:06:21 -03:00
IgorDaniel45 9a20d1cc69 merge dev and preserve both protocol test suites 2026-10-16 17:34:39 -03:00
AdamMagued 535e04d555 perf(protocols/http): defer building FullResponseString when unused by operators
Signed-off-by: AdamMagued <[email protected]>
2026-10-16 01:35:26 +03:00
Doğan Can Bakır 7989011cdf fix: keep the request body in the dump taken after sending 2026-10-15 11:52:15 +03:00
Dogan Can Bakir a151a1d54f Merge pull request #7793 from projectdiscovery/feat/7783-host-backoff
slow down a host that starts blocking
2026-10-14 16:06:57 +03:00
Dogan Can Bakir 1db63fdb42 Merge pull request #7852 from projectdiscovery/feat/replay-proxy
add -replay-proxy to re-send requests that produced a finding
2026-10-14 16:04:55 +03:00