mirror of
https://github.com/projectdiscovery/subfinder.git
synced 2026-10-12 02:21:20 +00:00
Dwi Siswanto
6d1b124dbd
Process domain lists with a worker pool bounded by Options.Threads.
Sources already run concurrently within each domain, but the runner
used to wait for every source to finish before starting the next one.
Reuse -t so those waits can overlap instead of adding another flag.
Default stays at 10; -t 1 keeps the old sequential domain behavior.
Create a fresh built-in source instance per domain. Snapshot source
selection plus provider-file and environment keys when the runner is
constructed, and keep environment-key precedence. That way mutable
source state and stats stay isolated across concurrent enumerations.
Preserve registered custom sources and overrides. Serialize their runs
by source name, drain results on cancel, and capture stats before
releasing the source. Custom sources without explicit keys keep the
registered instance's shared config. Leave the legacy passive
constructor and provider-loading API in place.
Share provider budgets across the whole batch instead of handing every
domain a fresh request burst. Built-in sources get cancellable
fixed-window waits, with no background timers for unlimited entries.
Keep per-source overrides of the global rate setting. Custom sources
share one legacy limiter so direct Session.MultiRateLimiter users still
work, including the existing blocking wait. Sessions borrow shared
budgets and do not stop them when a domain finishes.
Apply -t to the aggregate active DNS workload, including wildcard
probes. Reserve shared capacity before spinning up resolution task
goroutines so we don't create Threads workers for every concurrent
domain. Watch cancellation while waiting for capacity, receiving tasks,
and sending results. Keep wildcard state local to each domain and wait
for passive result processing before merging wildcard metadata.
Serialize callbacks and complete-domain output writes. Validate output
paths before any provider work starts, open shared output once on the
first accepted domain, and serialize file creation for repeated domain
paths. Leave output alone when input is empty. Return scanner errors,
preserve write and close errors, and cancel pending work on the first
failure. Domain output order is unspecified.
Reject nonpositive CLI thread counts, but keep a one-worker fallback
for SDK callers that leave Threads unset. Document the expanded -t
behavior and the built-in vs custom-source config boundaries.
Add a benchmark before touching the scheduler, plus regression tests
for concurrency bounds, cancellation, shared budgets, source and
credential isolation, custom-source compatibility, DNS worker growth,
callbacks, output files, and input errors.
For 40 domains with a simulated 5 ms source wait, median batch times
over five runs on Go 1.26.5 linux/amd64 were:
Threads Before After
1 225.35 ms 219.20 ms
4 225.22 ms 55.43 ms
10 227.32 ms 22.63 ms
All domains completed and peak concurrency matched the requested bound.
At Threads=10, allocated bytes went from about 643 KB to 720 KB per
batch. This benchmark only measures scheduling with a mock source; it
does not measure production source construction or live provider
throughput.
Signed-off-by: Dwi Siswanto <[email protected]>