The httpx client that probes non-URL inputs for http or https was built
from httpx.DefaultOptions without the configured resolvers, so the probe
resolved inputs through fastdialer's default resolvers (resolv.conf plus
1.1.1.1, 1.0.0.1, 8.8.8.8 and 8.8.4.4) even when -resolvers was set.
Pass InternalResolversList to httpxOptions.Resolvers in the runner and in
the SDK. httpx then uses only those resolvers. Without -resolvers the
probe behaves as before.
Closes#7969
Fixes#7875
When options.ProxyInternal was enabled, only HTTP_PROXY was set in the environment.
Because Go standard library http.ProxyFromEnvironment inspects HTTPS_PROXY for https URLs,
internal HTTPS requests (such as template updates via GitHub API / releases, pdtm, etc.)
failed to route through the specified proxy in restricted or proxy-required environments.
This sets HTTP_PROXY, HTTPS_PROXY, ALL_PROXY (and lowercase counterparts) when ProxyInternal
is enabled, and adds unit test coverage. Also fixes operator precedence for ProxyInternal
checks in runner.go and sdk_private.go.
The -stats percentage was requests sent against a total estimated
before the scan: TotalRequests of each template after clustering, times
the number of targets. The estimate leaves out workflows, whose requests
are added while they run, counts a self-contained template once per
target although it runs once, and doesn't follow stop-at-first-match or
a template that runs once per origin. A scan could end at 108% or stop
at 25%.
The engine now counts tasks: one per template, cluster or workflow run
on a target, and one per self-contained template. The total is known
when the scan starts. A task is done when it runs, when -resume skips
it, when its host is unresponsive, or when it repeats its origin.
progress.TaskTracker is an optional interface that StatsTicker
implements, so Progress and lib.StatsWriter don't change. The stats
line shows Tasks: done/total and takes the percentage from it, and -sj
and the metrics endpoint report tasks, totalTasks and the same percent.
applyRequiredDefaults installs the .nuclei-ignore deny-list into
Options.ExcludeTags once, while the engine is built.
ExecuteNucleiWithOptsCtx applies per-execution options to a COPY of
those options afterwards, and WithTemplateFilters assigns the whole
filter set — so a literal naming only Tags cleared ExcludeTags.
Nothing re-reads the ignore file after construction, so every execution
that varied tags per call ran with an empty exclude list and executed
templates tagged dos, local, fuzz, bruteforce and txt-service. The
failure was silent. Varying filters across concurrent executions is the
documented purpose of the thread-safe engine, so this was reachable
through intended use.
Restore the engine's baseline exclusions after per-execution options are
applied: a per-execution filter can now add exclusions but not silently
discard them. IncludeTags is untouched and remains the explicit per-tag
override for callers who do want an ignored template to run.
No extra I/O — the deny-list is already on the base engine's options —
and only missing entries are appended, so the helper is idempotent
across executions against a long-lived engine.
Fixes#7695
ReadIgnoreFile returns both `tags` and `files`, but applyRequiredDefaults
applied only `tags`, leaving ExcludedTemplates empty. The CLI runner
applies both, so the `files` section was inert for every SDK consumer.
That section exists to suppress templates known to have weak matchers,
so library users received false positives the CLI would never report.
Populate ExcludedTemplates from ignoreFile.Files, matching
internal/runner. ExcludedTemplates already flows to the loader as
Config.ExcludeTemplates, so no new plumbing is needed.
Fixes#7696
Return .nuclei-ignore read and parse failures to
callers instead of continuing with an empty
exclusion set.
Treat only a missing active ignore file as
recoverable and log it as a warning. Apply this
behavior to the CLI, SDK, and loader paths.
Fixes#7691
Signed-off-by: Dwi Siswanto <[email protected]>
* refactor: adopt XDG storage and layered configs
Load XDG system configuration (and /etc/nuclei/config.yaml) before the
active user configuration, selected config, profile, and command line.
Preserve explicit command-line values, replace collections between
layers, and defer stateful option handling until the final values are
known. Keep callback flags and authentication setup limited to the CLI.
Move configuration loading, profile resolution, inline targets, and
inline secrets into internal/configuration. Keep NUCLEI_CONFIG_DIR as
a compatibility override for the user configuration directory.
Use platform directories for configuration, template data, persistent
state, and cache data. Discover default template roots through
XDG_DATA_HOME and XDG_DATA_DIRS without assigning ownership semantics
to the selected path.
Store template state in XDG_STATE_HOME and migrate the legacy
.templates-config.json file when needed. Persist the active template
path, keep custom provider templates below that root, and apply the
same installation, update, health-check, and reset behavior to every
root.
Keep .nuclei-ignore with the active template root. Derive its local
hash from the file and pass the remote hash through the update check
without persisting either value.
Move generated resume and crash-recovery files into persistent state
and use XDG_CACHE_HOME for the catalog index. Write template state and
cache snapshots atomically with private permissions.
Validate every reset target before deleting any path. Reject broad or
unsafe targets, preserve explicit resume paths, and remove generated
inline-secret directories on normal, interrupted, and fatal exits.
Isolate configuration, state, cache, and template paths in functional
and integration tests, and add coverage for layered precedence, state
migration, active-root lifecycle, reset safety, and private file modes.
Closes#6755
Signed-off-by: Dwi Siswanto <[email protected]>
* chore(cmd): use memProfile+ext instead
Signed-off-by: Dwi Siswanto <[email protected]>
* fix(lib): gate SDK version check
Signed-off-by: Dwi Siswanto <[email protected]>
---------
Signed-off-by: Dwi Siswanto <[email protected]>
Compiled template cache entries retain mutable
`ExecutorOptions` and can keep closed engines
alive.
Move those entries into an engine-local cache.
Leave the caller-owned parsed cache intact and
purge only compiled entries on `Close`.
Fixes#7656
Signed-off-by: Dwi Siswanto <[email protected]>
Keep one engine-owned metadata index across SDK
calls and persist it at shutdown instead of
loading and saving a new index for each execution.
Track dirty state and parser validation so
compatible metadata can skip redundant filtering
parses w/o trusting lax entries in strict mode.
Defensively copy cached values, invalidate stale
entries synchronously, and remove the unused
ephemeral engine allocation.
Signed-off-by: Dwi Siswanto <[email protected]>
* refactor: replace Split in loops with more efficient SplitSeq
Signed-off-by: stringsbuilder <[email protected]>
* minor refactor
* strict checksum
* fix typo
* harden sdk tests
Avoid shared templates-dir update races across parallel go test packages and ignore known leveldb/ratelimit/memguardian goroutines in goleak.
* bind splitseq temps
Keep SplitSeq inputs in locals so iterator substrings cannot outlive a temporary string expression.
* fix sdk templates
Keep goleak ignores, drop DisableUpdateCheck so CI can install templates, and serialize UpdateIfOutdated across processes.
---------
Signed-off-by: stringsbuilder <[email protected]>
Co-authored-by: Mzack9999 <[email protected]>
Bumps clistats, fastdialer, retryablehttp-go, dsl, networkpolicy,
ratelimit, sarif, utils, wappalyzergo, cdncheck. dsl v0.8.18 moves to
the projectdiscovery/govaluate fork, and utils v0.11.0 pulls aurora v4,
so the corresponding imports and type references are migrated.
Refs #7380.
WithConfigFile/Bytes now decode YAML directly into a typed RuntimeConfig
struct and apply it via MergeOptions(opts). The goflags-backed reflection
overlay had too much surface area (266-line flag inventory duplicate plus
a flag-default vs DefaultOptions diff) for the actual SDK use case, which
is accepting a known set of scan knobs from cloud-shipped YAML.
RuntimeConfig schema covers tags/severity filtering, headers/vars,
interactsh, socks5-proxy, plus rate-limit/bulk-size/concurrency/timeout/
retries/rate-limit-host. Scalar knobs use *int so omitted keys preserve
the engine's existing value.
Drops internal/runner/flags.go (266 lines) and lib/config_load.go (86
lines). Reverts internal/runner/runner.go to dev (no SetupPDCPUpload
export; SDK inlines its own PDCP wrap). Keeps the LoadReportingOptionsFromBytes
extract in internal/runner/options.go for WithReportingConfigFile/Bytes.
DisableUpdateCheck() mutates DefaultConfig process-globally. config_test.go
files run alphabetically before sdk_test.go, so my tests were disabling the
update-check globally and TestContextCancelNucleiEngine (which depends on
template auto-install) was hitting an empty template store on first run.
Drop the option from the seven test sites. First test in the binary now
triggers template install (sync.Once gated; subsequent tests free).
- L1: sdk_private.go references SetupPDCPUpload (was lowercase).
- L2: config_test.go test comment describes how tmpEngine is constructed
rather than incorrectly claiming "inherits from parent".
- Trim godocs and inline comments across the PR — drop restated WHAT,
keep WHY and limitations.
- Inline (*Runner).setupPDCPUpload — wrapper just delegated to the
package-level SetupPDCPUpload. Call site inlines the three lines.
- Dedupe reporting-config file load: WithReportingConfigFile and
loadImplicitReportingConfig share loadReportingConfigFromPath.
- Move config-overlay/load helpers (newConfigFlagSet, overlayConfigFromFile,
applyOverlay, loadReportingConfigFromPath, loadImplicitReportingConfig)
out of lib/config.go into lib/config_load.go.
- Drop threadSafePerScan engine mode and isThreadSafe helper. The four
new With* options no longer gate per-scan use. pd-agent doesn't enter
thread-safe mode; reintroducing the gate ships as a follow-up PR with
its own scope.
SDK consumers had to rebuild types.Options by hand or shell out to the
CLI to get config-file parity. This shares BindOptionFlags,
LoadReportingOptionsFromBytes, ApplyExporterOptionsFromTypes, and
SetupPDCPUpload between cmd/nuclei and lib/ so both paths produce
identical engine state.
WithConfigFile/Bytes apply only YAML-set fields (reflection diff against
goflags defaults) so prior With* and DefaultOptions() values survive
YAML that omits them. Per-scan ExecuteNucleiWithOpts rejects these
options via a new threadSafePerScan mode, matching the pattern for
options that only make sense at construction.
when creating default limiter.
The engine could be set up with custom rate limits
through `WithOptions()`, but internal was still
hardcoding `e.rateLimiter` to 150 RPS when no
global limiter was explicitly set.
Now it builds the default limiter from the
resolved options:
1. Applies `RateLimitMinute` translation,
2. Defaults `RateLimitDuration` to 1s when
`RateLimit > 0` and no duration is given, then
3. Creates the limiter via `GetRateLimiter()`.
Fixes#7341
Signed-off-by: Dwi Siswanto <[email protected]>
Eren-Akdag