Build and push the Docker image in the release workflow, and remove the
separate Docker Hub workflow that rebuilt the image after
"🎉 Release Binary" finished.
GoReleaser now builds the image with dockers_v2 from the binaries it
compiles in the same run. The Dockerfile no longer builds subfinder. It
copies $TARGETPLATFORM/subfinder from the GoReleaser build context and
sets OCI labels. dockers_v2 replaces the deprecated dockers and
docker_manifests.
Rename release-binary.yml to release.yml and the workflow to
"🎉 Release", since it now publishes images too. The workflow now logs in
to Docker Hub before projectdiscovery/actions/goreleaser runs. With
release: true, that action already sets up QEMU and Buildx.
Tag images with the release tag, vX.Y, vX and latest. The old workflow
pushed only the release tag and latest.
Disable image builds for snapshot runs in CI, where runners may lack
QEMU. Local snapshot runs still build images.
Removing the old workflow also removes its workflow_dispatch trigger, so
images can no longer be re-pushed without a release run.
Build the 32-bit ARM image for linux/arm/v6 instead of linux/arm.
GoReleaser stores the default GOARM=6 binary under linux/arm/v6, but
buildx resolves linux/arm to arm/v7, where there is no binary. armv7
hosts can still run the arm/v6 image.
Closes#1964
Signed-off-by: Dwi Siswanto <[email protected]>
Two problems the review found in the previous commit.
`rateLimitRetries` lived on the run, so a 403 on a page that then
succeeded still spent budget. With one exhausted token and one healthy
one in the pool, every page answers 403 once and 200 once, and the
third such page ran the counter past the pool size: enumeration ended
early and reported that every token was rate limited while a working
token was still in the pool. The budget is now a `retries` argument
that a new URL starts over at zero, so a page that succeeds cannot eat
into the budget of the pages after it.
`DiscardHTTPResponse` returned before `Close()` when draining failed,
leaking the connection in exactly the case the helper exists to handle,
and every source now reaches it through a defer. It closes in both
paths.
Both tests fail without their fix: the retry one reports page 3 being
requested once rather than twice, the discard one reports the body
never being closed.
httpRequestWrapper returns the response *and* an error for any non-200
status, so a source that returns on `err != nil` before registering its
defer leaves that body undrained. Seven sources did:
certspotter, digitalyama, driftnet, leakix, merklemap, reconeer,
securitytrails
For a key-based source this is the common path, not a rare one: 429 once a
free-tier key is exhausted, 403 once it is rejected. Moving the deferred
DiscardHTTPResponse above the error check fixes it, and the helper already
ignores a nil response. securitytrails is different in shape: its 403 retry
overwrote resp without draining the first reply. digitalyama closed its body
without draining, so the connection could not be reused either.
The new sweep also uncovered a crash. github recursed into enumerate on every
403 with X-Ratelimit-Remaining 0, and Tokens.Get hands the pool out round
robin without skipping the tokens already marked exceeded. Once every token is
limited there was nothing to stop the recursion: one stack frame per 403 until
the process died with a stack overflow, which is what the test hit before this
change. Each token now gets one attempt, then the run reports the rate limit
and ends.
TestSourcesCloseResponseBodyOnErrorStatus runs every source against a 429 and
a 403 and requires every body handed out to be closed, so a source added later
cannot quietly reintroduce this.
* chore: go mod tidy
Signed-off-by: Dwi Siswanto <[email protected]>
* refactor(sources): isolate state for concurrent source runs
Keep counters and skipped state local to each Run call so overlapping
runs cannot reset each other's accounting or consume each other's result
limits. Protect statistics publication and reads with a mutex, returning
a snapshot of the most recently completed run.
Snapshot API keys at the start of each run and copy supplied key slices
in AddApiKeys. Keep atomic counters scoped to one run for the GitHub,
GitLab, and Driftnet workers.
Move GitLab's deferred wg.Done call before the worker's error paths so
failed file requests do not leave enumeration blocked.
Fixes#1952Closes#1947
Signed-off-by: Dwi Siswanto <[email protected]>
---------
Signed-off-by: Dwi Siswanto <[email protected]>
* fix(scanmalware): statistics races between concurrent runs
Concurrent Run calls reset and update counters on the same Source,
causing data races and mixing statistics from separate enumerations.
Statistics also reads those fields without synchronization.
Keep counters local to each run and publish the completed statistics
under a mutex before closing the results channel. Statistics returns
a snapshot of the most recently completed run. Requests remain
concurrent.
Fixes#1948
Signed-off-by: Dwi Siswanto <[email protected]>
* skip submd
* relax timing
---------
Signed-off-by: Dwi Siswanto <[email protected]>
Co-authored-by: Mzack9999 <[email protected]>
* fix(zoomeyeapi): switch to the v2 search API
Replace GET /domain/search with POST /v2/search, using a base64-encoded
domain query and reading data[].domain. Select web assets explicitly
and preserve the existing host:key configuration.
Stop pagination at the reported total or an empty page. Report HTTP
403 and API errors, close responses before sending errors, and make
error delivery cancelable.
Fixes#1943
Signed-off-by: Dwi Siswanto <[email protected]>
* add tests
---------
Signed-off-by: Dwi Siswanto <[email protected]>
Co-authored-by: Mzack9999 <[email protected]>
Drop the default Connection: close header so follow-up requests to the
same host can reuse a session's connection pool. That skips repeating
TCP setup and TLS handshakes on paginated sources. Explicit Connection
headers from sources still win.
The header landed in b03f0933d1 to stop descriptor exhaustion from
bound how long idle conns stick around. Existing Session.Close cleanup
still drains the pool when enumeration finishes.
Add HTTP and HTTPS regression tests for reuse, explicit close overrides,
and session cleanup. Add a Linux subprocess test that enumerates 128
domains under a 64-descriptor cap, makes two requests per domain, and
checks that each pair shares one connection. Disable enumeration's
session cleanup via a test overlay and the same test dies on the 30th
domain with "too many open files".
Add a benchmark for successive session requests, including connection
counts and allocations. Median of five runs of 500 requests against a
local server:
before after
HTTP ns/op 197234 82625
HTTPS ns/op 2871946 88933
HTTP allocs/op 145 80
HTTPS allocs/op 929 85
connections/run 500 1
These numbers are local repeated requests, not full enumeration
throughput.
Signed-off-by: Dwi Siswanto <[email protected]>
Process domain lists with a worker pool bounded by Options.Threads.
Sources already run concurrently within each domain, but the runner
used to wait for every source to finish before starting the next one.
Reuse -t so those waits can overlap instead of adding another flag.
Default stays at 10; -t 1 keeps the old sequential domain behavior.
Create a fresh built-in source instance per domain. Snapshot source
selection plus provider-file and environment keys when the runner is
constructed, and keep environment-key precedence. That way mutable
source state and stats stay isolated across concurrent enumerations.
Preserve registered custom sources and overrides. Serialize their runs
by source name, drain results on cancel, and capture stats before
releasing the source. Custom sources without explicit keys keep the
registered instance's shared config. Leave the legacy passive
constructor and provider-loading API in place.
Share provider budgets across the whole batch instead of handing every
domain a fresh request burst. Built-in sources get cancellable
fixed-window waits, with no background timers for unlimited entries.
Keep per-source overrides of the global rate setting. Custom sources
share one legacy limiter so direct Session.MultiRateLimiter users still
work, including the existing blocking wait. Sessions borrow shared
budgets and do not stop them when a domain finishes.
Apply -t to the aggregate active DNS workload, including wildcard
probes. Reserve shared capacity before spinning up resolution task
goroutines so we don't create Threads workers for every concurrent
domain. Watch cancellation while waiting for capacity, receiving tasks,
and sending results. Keep wildcard state local to each domain and wait
for passive result processing before merging wildcard metadata.
Serialize callbacks and complete-domain output writes. Validate output
paths before any provider work starts, open shared output once on the
first accepted domain, and serialize file creation for repeated domain
paths. Leave output alone when input is empty. Return scanner errors,
preserve write and close errors, and cancel pending work on the first
failure. Domain output order is unspecified.
Reject nonpositive CLI thread counts, but keep a one-worker fallback
for SDK callers that leave Threads unset. Document the expanded -t
behavior and the built-in vs custom-source config boundaries.
Add a benchmark before touching the scheduler, plus regression tests
for concurrency bounds, cancellation, shared budgets, source and
credential isolation, custom-source compatibility, DNS worker growth,
callbacks, output files, and input errors.
For 40 domains with a simulated 5 ms source wait, median batch times
over five runs on Go 1.26.5 linux/amd64 were:
Threads Before After
1 225.35 ms 219.20 ms
4 225.22 ms 55.43 ms
10 227.32 ms 22.63 ms
All domains completed and peak concurrency matched the requested bound.
At Threads=10, allocated bytes went from about 643 KB to 720 KB per
batch. This benchmark only measures scheduling with a mock source; it
does not measure production source construction or live provider
throughput.
Signed-off-by: Dwi Siswanto <[email protected]>
Query crt.name /v1/search for CT-indexed subdomains. Free tier needs no
key; a Bearer token is optional. Not recursive because the API only
accepts eTLD+1.
Closesprojectdiscovery/subfinder#1831
* feat: add scanmalware passive source
ScanMalware is a free sandboxed URL scanner with a public archive. The source
needs no API key: the API is anonymous, and a key only raises the rate limit.
Two endpoints are queried and merged. /api/v1/ct/dns/{domain} returns subdomains
observed in Certificate Transparency DNS records and carries most of the answer.
/api/v1/search/smql?q=domain:*.{domain} returns hosts present in the public scan
archive, which is not redundant: a host that was scanned but never held a
certificate appears only there.
The API caps its subdomain list and reports subdomains_truncated when it does;
that is surfaced as an error result rather than silently returning a partial list
as if it were complete.
Requests carry an explicit User-Agent so the traffic is attributable. The session
sets a random browser User-Agent by default, which headers passed to session.Get
override.
* feat(scanmalware): also read browser-observed hosts
Adds a third query, to /api/v1/hosts/{domain}, returning the hosts a browser
resolved and requested while rendering pages on the domain. That is a third
population: dev and staging hosts holding no certificate and never scanned
directly appear only there.
The default source set is used deliberately. The endpoint can also return hosts
named in a page's CSP or extracted statically from script source, but only about
a third of the statically extracted ones were ever contacted, and subfinder has
no way to carry that distinction through to the user.
Truncation is reported per source and surfaced rather than silently returning a
partial list.
* fix(scanmalware): check the error from closing response bodies
golangci-lint errcheck flagged three unchecked resp.Body.Close() calls. Close
errors are now reported as an error result, matching how the other sources
handle them.
* fix(scanmalware): do not skip the other queries when one endpoint fails
The three queries return independent populations, so a failure in the first was
silently skipping the archive and observed-host queries for the whole run. Only
a cancelled context stops the run now; an endpoint error is reported and the
remaining queries still execute.
Reported in review.
* fix(scanmalware): report archive pagination truncation
The archive query stops after a fixed number of pages. When the last allowed
page came back full there were more results than were read, and the source
returned the shortened set without saying so.
The other two queries already report truncation, so this was the one path where
a partial answer could be mistaken for a complete one.
Reported in review.
---------
Co-authored-by: Sandeep Singh <[email protected]>
golangci-lint reports SA1019 for net/http.Transport.Dial, deprecated since
Go 1.7. Because Test Builds declares needs: [lint], that single finding fails
the lint job and skips the build, test, race and example steps on every pull
request.
DialContext is the documented replacement and takes priority when both are set,
so behaviour is unchanged. golangci-lint goes from 1 issue to 0 across the tree.
Dogan Can Bakir