Two problems the review found in the previous commit.
`rateLimitRetries` lived on the run, so a 403 on a page that then
succeeded still spent budget. With one exhausted token and one healthy
one in the pool, every page answers 403 once and 200 once, and the
third such page ran the counter past the pool size: enumeration ended
early and reported that every token was rate limited while a working
token was still in the pool. The budget is now a `retries` argument
that a new URL starts over at zero, so a page that succeeds cannot eat
into the budget of the pages after it.
`DiscardHTTPResponse` returned before `Close()` when draining failed,
leaking the connection in exactly the case the helper exists to handle,
and every source now reaches it through a defer. It closes in both
paths.
Both tests fail without their fix: the retry one reports page 3 being
requested once rather than twice, the discard one reports the body
never being closed.
httpRequestWrapper returns the response *and* an error for any non-200
status, so a source that returns on `err != nil` before registering its
defer leaves that body undrained. Seven sources did:
certspotter, digitalyama, driftnet, leakix, merklemap, reconeer,
securitytrails
For a key-based source this is the common path, not a rare one: 429 once a
free-tier key is exhausted, 403 once it is rejected. Moving the deferred
DiscardHTTPResponse above the error check fixes it, and the helper already
ignores a nil response. securitytrails is different in shape: its 403 retry
overwrote resp without draining the first reply. digitalyama closed its body
without draining, so the connection could not be reused either.
The new sweep also uncovered a crash. github recursed into enumerate on every
403 with X-Ratelimit-Remaining 0, and Tokens.Get hands the pool out round
robin without skipping the tokens already marked exceeded. Once every token is
limited there was nothing to stop the recursion: one stack frame per 403 until
the process died with a stack overflow, which is what the test hit before this
change. Each token now gets one attempt, then the run reports the rate limit
and ends.
TestSourcesCloseResponseBodyOnErrorStatus runs every source against a 429 and
a 403 and requires every body handed out to be closed, so a source added later
cannot quietly reintroduce this.
* chore: go mod tidy
Signed-off-by: Dwi Siswanto <[email protected]>
* refactor(sources): isolate state for concurrent source runs
Keep counters and skipped state local to each Run call so overlapping
runs cannot reset each other's accounting or consume each other's result
limits. Protect statistics publication and reads with a mutex, returning
a snapshot of the most recently completed run.
Snapshot API keys at the start of each run and copy supplied key slices
in AddApiKeys. Keep atomic counters scoped to one run for the GitHub,
GitLab, and Driftnet workers.
Move GitLab's deferred wg.Done call before the worker's error paths so
failed file requests do not leave enumeration blocked.
Fixes#1952Closes#1947
Signed-off-by: Dwi Siswanto <[email protected]>
---------
Signed-off-by: Dwi Siswanto <[email protected]>
* fix(scanmalware): statistics races between concurrent runs
Concurrent Run calls reset and update counters on the same Source,
causing data races and mixing statistics from separate enumerations.
Statistics also reads those fields without synchronization.
Keep counters local to each run and publish the completed statistics
under a mutex before closing the results channel. Statistics returns
a snapshot of the most recently completed run. Requests remain
concurrent.
Fixes#1948
Signed-off-by: Dwi Siswanto <[email protected]>
* skip submd
* relax timing
---------
Signed-off-by: Dwi Siswanto <[email protected]>
Co-authored-by: Mzack9999 <[email protected]>
* fix(zoomeyeapi): switch to the v2 search API
Replace GET /domain/search with POST /v2/search, using a base64-encoded
domain query and reading data[].domain. Select web assets explicitly
and preserve the existing host:key configuration.
Stop pagination at the reported total or an empty page. Report HTTP
403 and API errors, close responses before sending errors, and make
error delivery cancelable.
Fixes#1943
Signed-off-by: Dwi Siswanto <[email protected]>
* add tests
---------
Signed-off-by: Dwi Siswanto <[email protected]>
Co-authored-by: Mzack9999 <[email protected]>
Drop the default Connection: close header so follow-up requests to the
same host can reuse a session's connection pool. That skips repeating
TCP setup and TLS handshakes on paginated sources. Explicit Connection
headers from sources still win.
The header landed in b03f0933d1 to stop descriptor exhaustion from
bound how long idle conns stick around. Existing Session.Close cleanup
still drains the pool when enumeration finishes.
Add HTTP and HTTPS regression tests for reuse, explicit close overrides,
and session cleanup. Add a Linux subprocess test that enumerates 128
domains under a 64-descriptor cap, makes two requests per domain, and
checks that each pair shares one connection. Disable enumeration's
session cleanup via a test overlay and the same test dies on the 30th
domain with "too many open files".
Add a benchmark for successive session requests, including connection
counts and allocations. Median of five runs of 500 requests against a
local server:
before after
HTTP ns/op 197234 82625
HTTPS ns/op 2871946 88933
HTTP allocs/op 145 80
HTTPS allocs/op 929 85
connections/run 500 1
These numbers are local repeated requests, not full enumeration
throughput.
Signed-off-by: Dwi Siswanto <[email protected]>
Process domain lists with a worker pool bounded by Options.Threads.
Sources already run concurrently within each domain, but the runner
used to wait for every source to finish before starting the next one.
Reuse -t so those waits can overlap instead of adding another flag.
Default stays at 10; -t 1 keeps the old sequential domain behavior.
Create a fresh built-in source instance per domain. Snapshot source
selection plus provider-file and environment keys when the runner is
constructed, and keep environment-key precedence. That way mutable
source state and stats stay isolated across concurrent enumerations.
Preserve registered custom sources and overrides. Serialize their runs
by source name, drain results on cancel, and capture stats before
releasing the source. Custom sources without explicit keys keep the
registered instance's shared config. Leave the legacy passive
constructor and provider-loading API in place.
Share provider budgets across the whole batch instead of handing every
domain a fresh request burst. Built-in sources get cancellable
fixed-window waits, with no background timers for unlimited entries.
Keep per-source overrides of the global rate setting. Custom sources
share one legacy limiter so direct Session.MultiRateLimiter users still
work, including the existing blocking wait. Sessions borrow shared
budgets and do not stop them when a domain finishes.
Apply -t to the aggregate active DNS workload, including wildcard
probes. Reserve shared capacity before spinning up resolution task
goroutines so we don't create Threads workers for every concurrent
domain. Watch cancellation while waiting for capacity, receiving tasks,
and sending results. Keep wildcard state local to each domain and wait
for passive result processing before merging wildcard metadata.
Serialize callbacks and complete-domain output writes. Validate output
paths before any provider work starts, open shared output once on the
first accepted domain, and serialize file creation for repeated domain
paths. Leave output alone when input is empty. Return scanner errors,
preserve write and close errors, and cancel pending work on the first
failure. Domain output order is unspecified.
Reject nonpositive CLI thread counts, but keep a one-worker fallback
for SDK callers that leave Threads unset. Document the expanded -t
behavior and the built-in vs custom-source config boundaries.
Add a benchmark before touching the scheduler, plus regression tests
for concurrency bounds, cancellation, shared budgets, source and
credential isolation, custom-source compatibility, DNS worker growth,
callbacks, output files, and input errors.
For 40 domains with a simulated 5 ms source wait, median batch times
over five runs on Go 1.26.5 linux/amd64 were:
Threads Before After
1 225.35 ms 219.20 ms
4 225.22 ms 55.43 ms
10 227.32 ms 22.63 ms
All domains completed and peak concurrency matched the requested bound.
At Threads=10, allocated bytes went from about 643 KB to 720 KB per
batch. This benchmark only measures scheduling with a mock source; it
does not measure production source construction or live provider
throughput.
Signed-off-by: Dwi Siswanto <[email protected]>
Query crt.name /v1/search for CT-indexed subdomains. Free tier needs no
key; a Bearer token is optional. Not recursive because the API only
accepts eTLD+1.
Closesprojectdiscovery/subfinder#1831
* feat: add scanmalware passive source
ScanMalware is a free sandboxed URL scanner with a public archive. The source
needs no API key: the API is anonymous, and a key only raises the rate limit.
Two endpoints are queried and merged. /api/v1/ct/dns/{domain} returns subdomains
observed in Certificate Transparency DNS records and carries most of the answer.
/api/v1/search/smql?q=domain:*.{domain} returns hosts present in the public scan
archive, which is not redundant: a host that was scanned but never held a
certificate appears only there.
The API caps its subdomain list and reports subdomains_truncated when it does;
that is surfaced as an error result rather than silently returning a partial list
as if it were complete.
Requests carry an explicit User-Agent so the traffic is attributable. The session
sets a random browser User-Agent by default, which headers passed to session.Get
override.
* feat(scanmalware): also read browser-observed hosts
Adds a third query, to /api/v1/hosts/{domain}, returning the hosts a browser
resolved and requested while rendering pages on the domain. That is a third
population: dev and staging hosts holding no certificate and never scanned
directly appear only there.
The default source set is used deliberately. The endpoint can also return hosts
named in a page's CSP or extracted statically from script source, but only about
a third of the statically extracted ones were ever contacted, and subfinder has
no way to carry that distinction through to the user.
Truncation is reported per source and surfaced rather than silently returning a
partial list.
* fix(scanmalware): check the error from closing response bodies
golangci-lint errcheck flagged three unchecked resp.Body.Close() calls. Close
errors are now reported as an error result, matching how the other sources
handle them.
* fix(scanmalware): do not skip the other queries when one endpoint fails
The three queries return independent populations, so a failure in the first was
silently skipping the archive and observed-host queries for the whole run. Only
a cancelled context stops the run now; an endpoint error is reported and the
remaining queries still execute.
Reported in review.
* fix(scanmalware): report archive pagination truncation
The archive query stops after a fixed number of pages. When the last allowed
page came back full there were more results than were read, and the source
returned the shortened set without saying so.
The other two queries already report truncation, so this was the one path where
a partial answer could be mistaken for a complete one.
Reported in review.
---------
Co-authored-by: Sandeep Singh <[email protected]>
golangci-lint reports SA1019 for net/http.Transport.Dial, deprecated since
Go 1.7. Because Test Builds declares needs: [lint], that single finding fails
the lint job and skips the build, test, race and example steps on every pull
request.
DialContext is the documented replacement and takes priority when both are set,
so behaviour is unchanged. golangci-lint goes from 1 issue to 0 across the tree.
golangci-lint v2.12.2 enables govet's inline analyzer, which reports
"cannot inline: type parameter inference is not yet supported" on calls
into golang.org/x/exp/maps and golang.org/x/exp/slices, since those are
marked //go:fix inline upstream.
Switch to the stdlib maps/slices packages:
- maps.Copy, slices.Contains are drop-in
- maps.Keys/maps.Values now return iterators, so wrap in
slices.Collect (or slices.Sorted where the result was already sorted)
golang.org/x/exp drops to an indirect dependency.
Fold the source metadata emitter back into listSources instead of a
separate file: the exported SourceInfo type had no consumer, and sorting
AllSources (a compile-time array, already alphabetical) only made the
JSON order differ from the human-readable listing.
Drops the accompanying tests, which re-derived the struct from the same
interface calls and asserted that json.Encoder emits JSON.
The -ls listing only prints source names with "*"/"~" markers, which is
awkward for automation that wants to pick sources by property (key-less,
recursive, default). When -ls is combined with -oJ, emit one JSON object
per source with its name, default, recursive and keyRequirement fields.
The default human-readable listing is unchanged.
Add tests covering the metadata builder and the JSONL writer.
NewSubdomainExtractor interpolated the raw domain into the regex, so the
dots in the domain were treated as "any character" (e.g. "example.com"
also matched "exampleXcom"). The pattern was also unbounded on the right,
so text such as "foo.example.com.evil.org" yielded the bogus subdomain
"foo.example.com", which passes the runner's HasSuffix check and reaches
the output. This affects every source that extracts hosts from free text
(crtsh, waybackarchive, commoncrawl, hackertarget, digitorus, rapiddns,
urlscan, sitedossier, ...).
Escape the domain with regexp.QuoteMeta and drop matches that are only a
prefix of a longer hostname, while still accepting the trailing-dot FQDN
form. Add table-driven tests for the extractor.
The -max-results flag was only honored by the virustotal source; every
other paginating source kept walking its cursor/pages and burning API
quota even when a cap was requested.
Apply the same early-stop pattern to the remaining paginating sources
(censys, certspotter, shodan, securitytrails, quake, dnsdb, onyphe,
rsecloud, redhuntlabs, urlscan, merklemap, commoncrawl): once
session.MaxResults results have been emitted the source returns instead
of requesting the next page. Single-request sources are unaffected.
Add httptest-based coverage for censys and certspotter proving the cap
stops pagination and bounds the request count, and update the flag help
text and docs to reflect the broader support.
Passive sources read third-party (and, since TLS verification is skipped,
potentially MITM'd) HTTP response bodies with unbounded io.ReadAll /
json.Decoder / bufio.Scanner. The client Timeout bounds time, not size, so a
malicious or compromised upstream could stream an effectively unbounded body
and OOM the process.
Cap the body centrally at the session layer (httpRequestWrapper) so every
source inherits the limit regardless of how it consumes the body. The
threatcrowd source issues a raw session.Client.Do, bypassing the wrapper, so it
opts in explicitly via the exported LimitResponseBody helper. The wrapper
preserves the original body's Close, so no body/connection leak.
50MB cap is generous enough that legitimate large subdomain lists are
unaffected.
Co-Authored-By: Claude Fable 5 <[email protected]>
Signed-off-by: Jack Adams-Lovell <[email protected]>
Doğan Can Bakır