Raw requests applied @Host, @tls-sni and @timeout from any matching
line of the rendered request, headers and body included. A response
value reused in a later request could carry an "@Host:" line and send
that request, with its body, to a host the template does not name.
The raw parser and the docs already define annotations as the "@"
lines before the request line. raw.SplitAnnotations now holds that
rule once, and the parser, the request annotations, @once and the
compile-time @timeout check all read from it.
Fixes: #8061
Signed-off-by: Dwi Siswanto <[email protected]>
The "(Truncated)" marker on dumped responses rebuilt the read-limit
rule from the template max-size and -rsr instead of using the limit
the body was read with. With neither set, the body is cut at the
10 MB default, but the dump was not marked truncated.
Comparing the body length with the effective limit keeps one rule for
both the read and the marker. c918d010c already made -rsr a ceiling
over a smaller template max-size; no release contains it yet, which
is why v3.8.0 still reads the full -rsr limit per response.
Fixes: #8045
Reported-by: @Lidar-Baruch
Signed-off-by: Dwi Siswanto <[email protected]>
A self-contained raw request was rendered twice: once to read the URL
from its request line, and again with the variables that URL provides,
such as {{Hostname}}. The second pass evaluated values that the first
pass had inserted. A response body of {{secret}} reused in a later
request was sent as the template variable's value, and with -ev a
response body naming an environment variable was sent as its value.
Render the request line first and check its host. Then evaluate again
only the template variables and payloads whose template text names a
URL variable, and render the annotations and the rest of the request
once. Response data is now sent unchanged, as in templates that are
not self-contained. A value that changes which line is the request
line is rejected, so the host checked against the network policy is
the host that is sent.
Fixes: #8058
Signed-off-by: Dwi Siswanto <[email protected]>
generator.Make builds the whole body from the template's DSL before
executeRequest takes it from the budget, and building costs about twice
the body. With 25 concurrent 23 MB builds outside the budget, it did
not lower peak memory.
ExecuteWithResults now reserves the template's largest body size seen
so far before calling Make. The size is only known after the DSL has
run, so a template's first build runs alone and records it. A body
larger than the reservation releases it and acquires its own size
again, instead of waiting while holding part of the budget, which could
deadlock two growing requests.
Add -max-request-body-inflight (default 256 MB, 0 disables it).
executeRequest acquires its body size from a shared budget before it
dumps and sends, and releases it when it returns, so concurrent
requests with multi-MB bodies queue instead of all holding their
copies at once. Bodies under 1 MB skip the budget, since the semaphore
is FIFO and a queued heavy request would stall them. Race requests
skip it too, because they must all reach the open gate together.
The HTTP timeout in ExecuteWithResults now starts once the budget
admits the request, so time spent queued does not time it out.
executeRequest dumps every request in full after sending it, so
matchers can read the whole body. For a 23 MB body that dump allocated
about 5x the body per request: cappedWriter grew its buffer by
doubling, the Accept-Encoding insertion copied the whole dump again,
and savedRequest copied it once more before capping it.
Size the buffer for the body up front, insert the header in place, and
let savedRequest cap the dump without converting it to bytes first. The
dump bytes are unchanged.
Description, impact and remediation can reference values of the finding
with the usual {{...}} syntax: named extractors, payload values of the
matching request, variables, DSL helpers, and the result fields host,
port, ip, url, scheme, path, matched-at, matcher-name and
extractor-name, which win over values of the same name. Name, severity,
tags and classification stay static, since filtering, grouping and
issue trackers key on them.
MakeDefaultResultEvent renders each result's copy of the info, so every
protocol gets it and the template never changes. Each field goes
through render.Render without scan options, like request and matcher
text, so nested placeholders and helpers behave the same and network
helpers stay denied. A field whose expression fails keeps the
template's text.
When placeholders change the text, the result keeps the written info in
TemplateInfo (json:"-" bson:"-"):
- the writer runs the -redact patterns over the changed fields, as it
does over the raw request and response;
- SARIF builds the shared rule from the written info and puts each
finding's filled text, capped at 4096 characters, in its result
message.
Info is filled only from a template's own event. Templates with info
placeholders are not clustered, since a cluster's shared event holds the
first template's variables and extracted values, and a global matcher
fills its info from the result fields alone.
executeTurboHTTP dialed pipelined connections with context.Background(),
so a pipelined request of a pinned target resolved the host instead of
connecting to its ip. It now dials with the input context, which carries
the pin. protocols/http/target-line-ip-pipeline.yaml runs the pinned
target test with pipeline: true, and gets no result without the change.
ProbeKey joins the input and the ip with a NUL byte, which no input
holds, instead of "@", which a scheme-less input may hold itself.
dev requires fastdialer v0.6.0, which has WithResolveOverride and the
dial cache fix, so go.mod and go.sum take dev's versions and the
fastdialer replace is dropped.
Templates that send multi-MB bodies (e.g. CVE-2026-109618 sends 1, 8 and
20 MB) kept 8-10 copies of each body alive per request. With 25 hosts in
flight this reached ~10 GB RSS and OOM-killed scan VMs.
- add pkg/utils/httpbody: a zero-copy request body that rewinds on EOF;
GetBody hands out new cursors over the same slice instead of cloning
- build raw, model and fuzzed request bodies with httpbody instead of
retryablehttp's ReusableReadCloser, which tees every read into a second
growing buffer and clones on each GetBody
- dump requests with http.Request.Write into a capped writer instead of
retryablehttp Dump/httputil.DumpRequestOut; output is unchanged for
bodies under the cap
- keep only the first 32 KiB of a body in request dumps (new
-rqss/-request-size-save flag, 0 keeps all) and leave bodies over 4 KiB
out of curl commands
- build the project cache key only when a project file is used, hashing
the body when the dump is truncated
- replay sends the full body when the dump was truncated
- slice the raw request body from the request instead of reading it out
- pass replacer values to fasttemplate as []byte so writes don't copy them
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Include the effective read limit in projectCacheKey so requests with different
limits (such as small template max-size previews) do not share truncated cached
bodies with larger reads.
When -response-size-read (-rsr) is configured, use it as a ceiling while
preserving a template's own MaxSize when smaller, rather than unconditionally
overwriting it. Also account for ResponseReadSize in response truncation detection.
Fixes#7873
Mzack9999