Scanning a vhost on a given server, or an origin behind a CDN, needs the
hostname in the Host header, SNI and template variables while the
connection goes to another ip, and one host often has to be scanned on
several ips. -sa scans the ips DNS returns, and a target of the ip with
a Host header sends no SNI and sets {{Hostname}} to the ip.
A target line can now pin its target to an ip, like curl --resolve:
{"target": "https://app.example", "ip": "192.0.2.10"}
Each target and ip pair is a separate target with its own template
selection. The ip is validated and canonicalized, the target must have
a hostname, -sa and -iv don't resolve a pinned target, and an ip the
network policy denies is skipped. Pinned targets are refused with
-proxy and -automatic-scan.
The engine pins the host of every target with a CustomIP, from a target
line or -sa, with fastdialer.WithResolveOverride on the context each
template runs with. HTTP, network, websocket and javascript templates
then connect to the ip, while redirects to other hosts and the proxy
dial are no longer pinned. HTTP keeps a client per pinned ip, network
and websocket results report the ip, and scheme-less targets are probed
through it.
The host errors cache keys a failed dial of a pinned target by its ip,
also for IDN hosts, and -mup counts the urls of each ip apart, so a url
listed with several ips, or resolved to several with -sa, is kept on
each of them.
go.mod points fastdialer at a fork commit with WithResolveOverride until
a fastdialer release includes it.
* fix(hosterrorscache): skip hosts that consistently time out
Count ErrKindNetworkTemporary (request timeouts) toward MaxHostError so a
host that times out on every request is skipped, instead of being probed by
every template. Reset-on-success still protects slow-but-alive hosts, so only
consecutive timeouts with no successful response reach the threshold.
Closes#7454
* fix(hosterrorscache): count rawhttp i/o timeouts toward host skip
rawhttp/unsafe templates surface read timeouts as a plain-string error
("ReadStatusLine: ... i/o timeout") that errkit cannot classify, so it
reaches the regex fallback. The regex did not list i/o timeout, so a host
that timed out on every rawhttp request was never skipped. Add i/o timeout
to the pattern. Verified: 30 unsafe templates vs one unresponsive host went
from 271s with no skip to skipping after the threshold.
* fix(hosterrorscache): reset on success and ignore parent-context cancellation
Addresses review feedback on the timeout-skip change:
- The non-clustered HTTP path never reset the host-errors cache on a
successful response (markHostError ignores nil, success branch only
incremented progress), so timeout counts were not consecutive and a live
host with intermittent timeouts could be skipped. Add markHostSuccess and
call it on a successful request, mirroring the clustered path.
- context.DeadlineExceeded classifies as ErrKindNetworkTemporary, so a
failure from the parent scan context being cancelled/deadlined would count
as a host failure. Ignore failures in MarkFailedOrRemove when the caller's
context is already done; success still resets.
Tests: parent-context cancellation not counted, non-consecutive timeouts do
not skip, and the HTTP path resets the cache on success.
* fix(hosterrorscache): reset host cache on success across all HTTP paths
The timeout-counting change is global (in the cache), but reset-on-success was
only wired into the sequential HTTP path. The parallel, race and pipeline paths
counted failures without resetting on success, so a live host with intermittent
timeouts could be falsely skipped in those modes.
Consolidate mark-and-reset into recordHostResult and call it from every HTTP
execution path (sequential, parallel, race, pipeline). The SPM handler invokes
the result callback for every request outcome including success, so resetting
there is sufficient.
* test(http): add test for the parallel generated-host case
Signed-off-by: Dwi Siswanto <[email protected]>
* fix(http): use generated input when record host errors
The async HTTP paths checked generated requests
with the updated input, but recorded host-error
results with the original input. If a generated
URL changed the effective host/port, `Check()` and
`MarkFailedOrRemove()` operated on different cache
keys.
So record race, parallel, and pipeline results with
the same updated input used for the request to
keep timeout counts and success resets attached to
the host that was actually checked.
Signed-off-by: Dwi Siswanto <[email protected]>
---------
Signed-off-by: Dwi Siswanto <[email protected]>
Co-authored-by: Dwi Siswanto <[email protected]>
The "Skipped X from target list as found
unresponsive permanently" message was logged on
every `(*Cache).Check()` call for hosts with
permanent errors, resulting in thousands of
duplicate log entries in verbose mode.
Wrap the log statement in `sync.Once` to match the
behavior already used for non-permanent error
logging.
Signed-off-by: Dwi Siswanto <[email protected]>
* feat: fixed max-host-error blocking wrong port for template with error
* feat: log total results with time taken at end of execution
* bugfix: skip non-executed requests with progress in flow protocol
* feat: fixed request calculation in http protocol for progress
* misc adjustments
---------
Co-authored-by: Ice3man <[email protected]>
Georgii Butakov