Compare commits

..
Author SHA1 Message Date
Ice3man d375b66d78 misc 2025-08-06 11:34:36 +05:30
Ice3man 3210091c69 misc 2025-08-06 11:30:12 +05:30
Ice3man 4b142c2e89 misc updates to go.mod 2025-08-06 11:21:45 +05:30
Ice3man 1491c4d470 misc 2025-08-06 04:27:34 +05:30
99 changed files with 636 additions and 3476 deletions

No files matched your search

-35
View File
@@ -1,35 +0,0 @@
{
"permissions": {
"allow": [
"Bash(find:*)",
"Bash(mkdir:*)",
"Bash(cp:*)",
"Bash(ls:*)",
"Bash(make:*)",
"Bash(go:*)",
"Bash(golangci-lint:*)",
"Bash(git merge:*)",
"Bash(git add:*)",
"Bash(git commit:*)",
"Bash(git push:*)",
"Bash(git pull:*)",
"Bash(git fetch:*)",
"Bash(git checkout:*)",
"WebFetch(*)",
"Write(*)",
"WebSearch(*)",
"MultiEdit(*)",
"Edit(*)",
"Bash(gh:*)",
"Bash(grep:*)",
"Bash(tree:*)",
"Bash(./nuclei:*)",
"WebFetch(domain:github.com)"
],
"deny": [
"Bash(make run:*)",
"Bash(./bin/nuclei:*)"
],
"defaultMode": "acceptEdits"
}
}
-76
View File
@@ -1,76 +0,0 @@
# Nuclei Discussion Guidelines
## Before Creating a Discussion
1. **Search existing discussions and issues** to avoid duplicates
2. **Check the documentation** and README first
3. **Browse the FAQ** and common questions
## Bug Reports in Discussions
When reporting a bug in [Q&A Discussions](https://github.com/projectdiscovery/nuclei/discussions/categories/q-a), please include:
### Required Information:
- **Clear title** with `[BUG]` prefix (e.g., "[BUG] Nuclei crashes when...")
- **Current behavior** - What's happening now?
- **Expected behavior** - What should happen instead?
- **Steps to reproduce** - Commands or actions that trigger the issue
- **Environment details**:
- OS and version
- Nuclei version (`nuclei -version`)
- Go version (if installed via `go install`)
- **Log output** - Run with `-verbose` or `-debug` for detailed logs
- **Redact sensitive information** - Remove target URLs, credentials, etc.
### After Discussion:
- Maintainers will review and validate the bug report
- Valid bugs will be converted to issues with proper labels and tracking
- Questions and misconfigurations will be resolved in the discussion
## Feature Requests in Discussions
When requesting a feature in [Ideas Discussions](https://github.com/projectdiscovery/nuclei/discussions/categories/ideas), please include:
### Required Information:
- **Clear title** with `[FEATURE]` prefix (e.g., "[FEATURE] Add support for...")
- **Feature description** - What do you want to be added?
- **Use case** - Why is this feature needed? What problem does it solve?
- **Implementation ideas** - If you have suggestions on how it could work
- **Alternatives considered** - What other solutions have you thought about?
### After Discussion:
- Community and maintainers will discuss the feasibility
- Popular and viable features will be converted to issues
- Similar features may be grouped together
- Rejected features will be explained in the discussion
## Getting Help
For general questions, troubleshooting, and "how-to" topics:
- Use [Q&A Discussions](https://github.com/projectdiscovery/nuclei/discussions/categories/q-a)
- Join the [Discord server](https://discord.gg/projectdiscovery) #nuclei channel
- Check existing discussions for similar questions
## Discussion to Issue Conversion Process
Only maintainers can convert discussions to issues. The process:
1. **Validation** - Maintainers review the discussion for completeness and validity
2. **Classification** - Determine if it's a bug, feature, enhancement, etc.
3. **Issue creation** - Create a properly formatted issue with appropriate labels
4. **Linking** - Link the issue back to the original discussion
5. **Resolution** - Mark the discussion as resolved or close it
This process ensures:
- High-quality issues that are actionable
- Proper triage and labeling
- Reduced noise in the issue tracker
- Community involvement in the validation process
## Why This Process?
- **Better organization** - Issues contain only validated, actionable items
- **Community input** - Discussions allow for community feedback before escalation
- **Quality control** - Maintainers ensure proper formatting and information
- **Reduced maintenance** - Fewer invalid or duplicate issues to manage
- **Clear separation** - Questions vs. actual bugs/features are clearly distinguished
+8 -16
View File
@@ -2,22 +2,14 @@ blank_issues_enabled: false
contact_links:
- name: 🐛 Report a Bug (Start with Discussion)
url: https://github.com/orgs/projectdiscovery/discussions/new?category=q-a
about: Start by reporting your issue in discussions for proper triage. Issues will be created after review to avoid duplicate/invalid reports.
- name: Ask an question / advise on using nuclei
url: https://github.com/projectdiscovery/nuclei/discussions/categories/q-a
about: Ask a question or request support for using nuclei
- name: 💡 Request a Feature (Start with Discussion)
url: https://github.com/orgs/projectdiscovery/discussions/new?category=ideas
about: Share your feature idea in discussions first. This helps validate and refine the request before creating an issue.
- name: Share idea / feature to discuss for nuclei
url: https://github.com/projectdiscovery/nuclei/discussions/categories/ideas
about: Share idea / feature to discuss for nuclei
- name: ❓ Ask Questions / Get Help
url: https://github.com/orgs/projectdiscovery/discussions
about: Get help and ask questions about using Nuclei. Many questions don't require issues.
- name: 🔍 Browse Existing Issues
url: https://github.com/projectdiscovery/nuclei/issues
about: Check existing issues to see if your problem has already been reported or is being worked on.
- name: 💬 Connect with PD Team (Discord)
- name: Connect with PD Team (Discord)
url: https://discord.gg/projectdiscovery
about: Join our Discord for real-time discussions and community support on the #nuclei channel.
about: Connect with PD Team for direct communication
@@ -1,45 +0,0 @@
# Issue Template References
## Overview
This folder contains the preserved issue templates that are **not** directly accessible to users. These templates serve as references for maintainers when converting discussions to issues.
## New Workflow
### For Users:
1. **All reports start in Discussions** - Users cannot create issues directly
2. Bug reports go to [Q&A Discussions](https://github.com/projectdiscovery/nuclei/discussions/categories/q-a)
3. Feature requests go to [Ideas Discussions](https://github.com/projectdiscovery/nuclei/discussions/categories/ideas)
4. This helps filter out duplicate questions, invalid reports, and ensures proper triage
### For Maintainers:
1. **Review discussions** in both Q&A and Ideas categories
2. **Validate the reports** - ensure they're actual bugs/valid feature requests
3. **Use reference templates** when converting discussions to issues:
- Copy content from `bug-report-reference.yml` or `feature-request-reference.yml`
- Create a new issue manually with the appropriate template structure
- Link back to the original discussion
- Close the discussion or mark it as resolved
## Benefits
- **Better triage**: Avoid cluttering issues with questions and invalid reports
- **Community involvement**: Discussions allow for community input before creating issues
- **Quality control**: Maintainers can ensure issues follow proper format and contain necessary information
- **Reduced noise**: Only validated, actionable items become issues
## Reference Templates
- `bug-report-reference.yml` - Use when converting bug reports from discussions to issues
- `feature-request-reference.yml` - Use when converting feature requests from discussions to issues
## Converting a Discussion to Issue
1. Identify a valid discussion that needs to become an issue
2. Go to the main repository's Issues tab
3. Click "New Issue"
4. Manually create the issue using the reference template structure
5. Include all relevant information from the discussion
6. Add a comment linking back to the original discussion
7. Apply appropriate labels
8. Close or mark the discussion as resolved with a link to the created issue
+1 -1
View File
@@ -24,7 +24,7 @@ jobs:
steps:
- uses: actions/checkout@v4
- uses: projectdiscovery/actions/setup/go@v1
- uses: projectdiscovery/actions/golangci-lint/v2@v1
- uses: projectdiscovery/actions/golangci-lint@v1
tests:
name: "Tests"
-2
View File
@@ -28,8 +28,6 @@
/scrapefunc
/scrapefuncs
/tsgen
/integration_tests/integration-test
/integration_tests/nuclei
# Templates
/*.yaml
-83
View File
@@ -1,83 +0,0 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
Nuclei is a modern, high-performance vulnerability scanner built in Go that leverages YAML-based templates for customizable vulnerability detection. It supports multiple protocols (HTTP, DNS, TCP, SSL, WebSocket, WHOIS, JavaScript, Code) and is designed for zero false positives through real-world condition simulation.
## Development Commands
### Building and Testing
- `make build` - Build the main nuclei binary to ./bin/nuclei
- `make test` - Run unit tests with race detection
- `make integration` - Run integration tests (builds and runs test suite)
- `make functional` - Run functional tests
- `make vet` - Run go vet for code analysis
- `make tidy` - Clean up go modules
### Validation and Linting
- `make template-validate` - Validate nuclei templates using the built binary
- `go fmt ./...` - Format Go code
- `go vet ./...` - Static analysis
### Development Tools
- `make devtools-all` - Build all development tools (bindgen, tsgen, scrapefuncs)
- `make jsupdate-all` - Update JavaScript bindings and TypeScript definitions
- `make docs` - Generate documentation
- `make memogen` - Generate memoization code for JavaScript libraries
### Testing Specific Components
- Run single test: `go test -v ./pkg/path/to/package -run TestName`
- Integration tests are in `integration_tests/` and can be run via `make integration`
## Architecture Overview
### Core Components
- **cmd/nuclei** - Main CLI entry point with flag parsing and configuration
- **internal/runner** - Core runner that orchestrates the entire scanning process
- **pkg/core** - Execution engine with work pools and template clustering
- **pkg/templates** - Template parsing, compilation, and management
- **pkg/protocols** - Protocol implementations (HTTP, DNS, Network, etc.)
- **pkg/operators** - Matching and extraction logic (matchers/extractors)
- **pkg/catalog** - Template discovery and loading from disk/remote sources
### Protocol Architecture
Each protocol (HTTP, DNS, Network, etc.) implements:
- Request interface with Compile(), ExecuteWithResults(), Match(), Extract() methods
- Operators embedding for matching/extraction functionality
- Protocol-specific request building and execution logic
### Template System
- Templates are YAML files defining vulnerability detection logic
- Compiled into executable requests with operators (matchers/extractors)
- Support for workflows (multi-step template execution)
- Template clustering optimizes identical requests across multiple templates
### Key Execution Flow
1. Template loading and compilation via pkg/catalog/loader
2. Input provider setup for targets
3. Engine creation with work pools for concurrency
4. Template execution with result collection via operators
5. Output writing and reporting integration
### JavaScript Integration
- Custom JavaScript runtime for code protocol templates
- Auto-generated bindings in pkg/js/generated/
- Library implementations in pkg/js/libs/
- Development tools for binding generation in pkg/js/devtools/
## Template Development
- Templates located in separate nuclei-templates repository
- YAML format with info, requests, and operators sections
- Support for multiple protocol types in single template
- Built-in DSL functions for dynamic content generation
- Template validation available via `make template-validate`
## Key Directories
- **lib/** - SDK for embedding nuclei as a library
- **examples/** - Usage examples for different scenarios
- **integration_tests/** - Integration test suite with protocol-specific tests
- **pkg/fuzz/** - Fuzzing engine and DAST capabilities
- **pkg/input/** - Input processing for various formats (Burp, OpenAPI, etc.)
- **pkg/reporting/** - Result export and issue tracking integrations
+4 -17
View File
@@ -15,8 +15,8 @@ ifneq ($(shell go env GOOS),darwin)
endif
.PHONY: all build build-stats clean devtools-all devtools-bindgen devtools-scrapefuncs
.PHONY: devtools-tsgen docs docgen dsl-docs functional fuzzplayground go-build lint lint-strict syntax-docs
.PHONY: integration jsupdate-all jsupdate-bindgen jsupdate-tsgen memogen scan-charts test test-with-lint
.PHONY: devtools-tsgen docs docgen dsl-docs functional fuzzplayground go-build syntax-docs
.PHONY: integration jsupdate-all jsupdate-bindgen jsupdate-tsgen memogen scan-charts test
.PHONY: tidy ts verify download vet template-validate
all: build
@@ -146,18 +146,5 @@ dsl-docs:
template-validate: build
template-validate:
./bin/nuclei -ut
./bin/nuclei -validate \
-et .github/ \
-et helpers/payloads/ \
-et http/technologies \
-t dns \
-t ssl \
-t network \
-t http/exposures \
-ept code
./bin/nuclei -validate \
-w workflows \
-et .github/ \
-et helpers/payloads/ \
-et http/technologies \
-ept code
./bin/nuclei -validate -et http/technologies
./bin/nuclei -validate -w workflows -et http/technologies
-104
View File
@@ -1,104 +0,0 @@
package main
import (
"context"
"fmt"
"log"
"time"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/mongo"
"github.com/testcontainers/testcontainers-go"
mongocontainer "github.com/testcontainers/testcontainers-go/modules/mongodb"
osutil "github.com/projectdiscovery/utils/os"
mongoclient "go.mongodb.org/mongo-driver/mongo"
mongooptions "go.mongodb.org/mongo-driver/mongo/options"
)
const (
dbName = "test"
dbImage = "mongo:8"
)
var exportersTestCases = []TestCaseInfo{
{Path: "exporters/mongo", TestCase: &mongoExporter{}, DisableOn: func() bool {
return osutil.IsWindows() || osutil.IsOSX()
}},
}
type mongoExporter struct{}
func (m *mongoExporter) Execute(filepath string) error {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
defer cancel()
// Start a MongoDB container
mongodbContainer, err := mongocontainer.Run(ctx, dbImage)
defer func() {
if err := testcontainers.TerminateContainer(mongodbContainer); err != nil {
log.Printf("failed to terminate container: %s", err)
}
}()
if err != nil {
return fmt.Errorf("failed to start container: %w", err)
}
connString, err := mongodbContainer.ConnectionString(ctx)
if err != nil {
return fmt.Errorf("failed to get connection string for MongoDB container: %s", err)
}
connString = connString + dbName
// Create a MongoDB exporter and write a test result to the database
opts := mongo.Options{
ConnectionString: connString,
CollectionName: "test",
BatchSize: 1, // Ensure we write the result immediately
}
exporter, err := mongo.New(&opts)
if err != nil {
return fmt.Errorf("failed to create MongoDB exporter: %s", err)
}
defer func() {
if err := exporter.Close(); err != nil {
fmt.Printf("failed to close exporter: %s\n", err)
}
}()
res := &output.ResultEvent{
Request: "test request",
Response: "test response",
}
err = exporter.Export(res)
if err != nil {
return fmt.Errorf("failed to export result event to MongoDB: %s", err)
}
// Verify that the result was written to the database
clientOptions := mongooptions.Client().ApplyURI(connString)
client, err := mongoclient.Connect(ctx, clientOptions)
if err != nil {
return fmt.Errorf("error creating MongoDB client: %s", err)
}
defer func() {
if err := client.Disconnect(ctx); err != nil {
fmt.Printf("failed to disconnect from MongoDB: %s\n", err)
}
}()
collection := client.Database(dbName).Collection(opts.CollectionName)
var actualRes output.ResultEvent
err = collection.FindOne(ctx, map[string]interface{}{"request": res.Request}).Decode(&actualRes)
if err != nil {
return fmt.Errorf("failed to find document in MongoDB: %s", err)
}
if actualRes.Request != res.Request || actualRes.Response != res.Response {
return fmt.Errorf("exported result does not match expected result: got %v, want %v", actualRes, res)
}
return nil
}
+9 -9
View File
@@ -19,7 +19,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/testutils"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/json"
"github.com/projectdiscovery/retryablehttp-go"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
logutil "github.com/projectdiscovery/utils/log"
sliceutil "github.com/projectdiscovery/utils/slice"
stringsutil "github.com/projectdiscovery/utils/strings"
@@ -196,7 +196,7 @@ func (d *httpDefaultMatcherCondition) Execute(filePath string) error {
return err
}
if routerErr != nil {
return errkit.Wrap(routerErr, "failed to send http request to interactsh server")
return errorutil.NewWithErr(routerErr).Msgf("failed to send http request to interactsh server")
}
if err := expectResultsCount(results, 1); err != nil {
return err
@@ -628,10 +628,10 @@ func (h *httpRawWithParams) Execute(filePath string) error {
// we intentionally use params["test"] instead of params.Get("test") to test the case where
// there are multiple parameters with the same name
if !reflect.DeepEqual(params["key1"], []string{"value1"}) {
errx = errkit.Append(errx, errkit.New("key1 not found in params", "expected", []string{"value1"}, "got", params["key1"]))
errx = errorutil.WrapfWithNil(errx, "expected %v, got %v", []string{"value1"}, params["key1"])
}
if !reflect.DeepEqual(params["key2"], []string{"value2"}) {
errx = errkit.Append(errx, errkit.New("key2 not found in params", "expected", []string{"value2"}, "got", params["key2"]))
errx = errorutil.WrapfWithNil(errx, "expected %v, got %v", []string{"value2"}, params["key2"])
}
_, _ = fmt.Fprintf(w, "Test is test raw-params-matcher text")
})
@@ -971,10 +971,10 @@ func (h *httpRequestSelfContainedWithParams) Execute(filePath string) error {
// we intentionally use params["test"] instead of params.Get("test") to test the case where
// there are multiple parameters with the same name
if !reflect.DeepEqual(params["something"], []string{"here"}) {
errx = errkit.Append(errx, errkit.New("something not found in params", "expected", []string{"here"}, "got", params["something"]))
errx = errorutil.WrapfWithNil(errx, "expected %v, got %v", []string{"here"}, params["something"])
}
if !reflect.DeepEqual(params["key"], []string{"value"}) {
errx = errkit.Append(errx, errkit.New("key not found in params", "expected", []string{"value"}, "got", params["key"]))
errx = errorutil.WrapfWithNil(errx, "expected %v, got %v", []string{"value"}, params["key"])
}
_, _ = w.Write([]byte("This is self-contained response"))
})
@@ -1027,10 +1027,10 @@ func (h *httpRequestSelfContainedFileInput) Execute(filePath string) error {
// create temp file
FileLoc, err := os.CreateTemp("", "self-contained-payload-*.txt")
if err != nil {
return errkit.Wrap(err, "failed to create temp file")
return errorutil.NewWithErr(err).Msgf("failed to create temp file")
}
if _, err := FileLoc.Write([]byte("one\ntwo\n")); err != nil {
return errkit.Wrap(err, "failed to write payload to temp file")
return errorutil.NewWithErr(err).Msgf("failed to write payload to temp file")
}
defer func() {
_ = FileLoc.Close()
@@ -1046,7 +1046,7 @@ func (h *httpRequestSelfContainedFileInput) Execute(filePath string) error {
}
if !sliceutil.ElementsMatch(gotReqToEndpoints, []string{"/one", "/two", "/one", "/two"}) {
return errkit.New("expected requests to be sent to `/one` and `/two` endpoints but were sent to `%v`", gotReqToEndpoints, "filePath", filePath)
return errorutil.NewWithTag(filePath, "expected requests to be sent to `/one` and `/two` endpoints but were sent to `%v`", gotReqToEndpoints)
}
return nil
}
-1
View File
@@ -57,7 +57,6 @@ var (
"flow": flowTestcases,
"javascript": jsTestcases,
"matcher-status": matcherStatusTestcases,
"exporters": exportersTestCases,
}
// flakyTests are run with a retry count of 3
flakyTests = map[string]bool{
+2 -2
View File
@@ -10,7 +10,7 @@ import (
"github.com/julienschmidt/httprouter"
"github.com/projectdiscovery/nuclei/v3/pkg/testutils"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
permissionutil "github.com/projectdiscovery/utils/permission"
)
@@ -223,7 +223,7 @@ type loadTemplateWithID struct{}
func (h *loadTemplateWithID) Execute(nooop string) error {
results, err := testutils.RunNucleiBareArgsAndGetResults(debug, nil, "-target", "scanme.sh", "-id", "self-signed-ssl")
if err != nil {
return errkit.Wrap(err, "failed to load template with id")
return errorutil.NewWithErr(err).Msgf("failed to load template with id")
}
return expectResultsCount(results, 1)
}
+4 -4
View File
@@ -4,7 +4,7 @@ import (
"fmt"
"github.com/projectdiscovery/nuclei/v3/pkg/testutils"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
)
var profileLoaderTestcases = []TestCaseInfo{
@@ -18,7 +18,7 @@ type profileLoaderByRelFile struct{}
func (h *profileLoaderByRelFile) Execute(testName string) error {
results, err := testutils.RunNucleiWithArgsAndGetResults(debug, "-tl", "-tp", "cloud.yml")
if err != nil {
return errkit.Wrap(err, "failed to load template with id")
return errorutil.NewWithErr(err).Msgf("failed to load template with id")
}
if len(results) <= 10 {
return fmt.Errorf("incorrect result: expected more results than %d, got %v", 10, len(results))
@@ -31,7 +31,7 @@ type profileLoaderById struct{}
func (h *profileLoaderById) Execute(testName string) error {
results, err := testutils.RunNucleiWithArgsAndGetResults(debug, "-tl", "-tp", "cloud")
if err != nil {
return errkit.Wrap(err, "failed to load template with id")
return errorutil.NewWithErr(err).Msgf("failed to load template with id")
}
if len(results) <= 10 {
return fmt.Errorf("incorrect result: expected more results than %d, got %v", 10, len(results))
@@ -45,7 +45,7 @@ type customProfileLoader struct{}
func (h *customProfileLoader) Execute(filepath string) error {
results, err := testutils.RunNucleiWithArgsAndGetResults(debug, "-tl", "-tp", filepath)
if err != nil {
return errkit.Wrap(err, "failed to load template with id")
return errorutil.NewWithErr(err).Msgf("failed to load template with id")
}
if len(results) < 1 {
return fmt.Errorf("incorrect result: expected more results than %d, got %v", 1, len(results))
+2 -2
View File
@@ -4,7 +4,7 @@ import (
"os"
"github.com/projectdiscovery/nuclei/v3/pkg/testutils"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
)
var templatesDirTestCases = []TestCaseInfo{
@@ -17,7 +17,7 @@ type templateDirWithTargetTest struct{}
func (h *templateDirWithTargetTest) Execute(filePath string) error {
tempdir, err := os.MkdirTemp("", "nuclei-update-dir-*")
if err != nil {
return errkit.Wrap(err, "failed to create temp dir")
return errorutil.NewWithErr(err).Msgf("failed to create temp dir")
}
defer func() {
_ = os.RemoveAll(tempdir)
+9 -2
View File
@@ -39,7 +39,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/nuclei/v3/pkg/types/scanstrategy"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/monitor"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
fileutil "github.com/projectdiscovery/utils/file"
unitutils "github.com/projectdiscovery/utils/unit"
updateutils "github.com/projectdiscovery/utils/update"
@@ -187,7 +187,7 @@ func main() {
options.Logger.Info().Msgf("Creating resume file: %s\n", resumeFileName)
err := nucleiRunner.SaveResumeConfig(resumeFileName)
if err != nil {
return errkit.Wrap(err, "couldn't create crash resume file")
return errorutil.NewWithErr(err).Msgf("couldn't create crash resume file")
}
return nil
})
@@ -757,3 +757,10 @@ func findProfilePathById(profileId, templatesDir string) string {
}
return profilePath
}
func init() {
// print stacktrace of errors in debug mode
if strings.EqualFold(os.Getenv("DEBUG"), "true") {
errorutil.ShowStackTrace = true
}
}
+23 -23
View File
@@ -23,7 +23,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/json"
"github.com/projectdiscovery/retryablehttp-go"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
"gopkg.in/yaml.v3"
)
@@ -243,7 +243,7 @@ func enhanceTemplate(data string) (string, bool, error) {
return data, false, err
}
if resp.StatusCode != 200 {
return data, false, errkit.New("unexpected status code: %v", resp.Status)
return data, false, errorutil.New("unexpected status code: %v", resp.Status)
}
var templateResp TemplateResp
if err := json.NewDecoder(resp.Body).Decode(&templateResp); err != nil {
@@ -254,20 +254,20 @@ func enhanceTemplate(data string) (string, bool, error) {
}
if templateResp.ValidateErrorCount > 0 {
if len(templateResp.ValidateError) > 0 {
return data, false, errkit.New(templateResp.ValidateError[0].Message+": at line %v", templateResp.ValidateError[0].Mark.Line, "tag", "validate")
return data, false, errorutil.NewWithTag("validate", templateResp.ValidateError[0].Message+": at line %v", templateResp.ValidateError[0].Mark.Line)
}
return data, false, errkit.New("validation failed", "tag", "validate")
return data, false, errorutil.New("validation failed").WithTag("validate")
}
if templateResp.Error.Name != "" {
return data, false, errkit.New("%s", templateResp.Error.Name)
return data, false, errorutil.New("%s", templateResp.Error.Name)
}
if strings.TrimSpace(templateResp.Enhanced) == "" && !templateResp.Lint {
if templateResp.LintError.Reason != "" {
return data, false, errkit.New(templateResp.LintError.Reason+" : at line %v", templateResp.LintError.Mark.Line, "tag", "lint")
return data, false, errorutil.NewWithTag("lint", templateResp.LintError.Reason+" : at line %v", templateResp.LintError.Mark.Line)
}
return data, false, errkit.New("at line: %v", templateResp.LintError.Mark.Line, "tag", "lint")
return data, false, errorutil.NewWithTag("lint", "at line: %v", templateResp.LintError.Mark.Line)
}
return data, false, errkit.New("template enhance failed")
return data, false, errorutil.New("template enhance failed")
}
// formatTemplate formats template data using templateman format api
@@ -277,7 +277,7 @@ func formatTemplate(data string) (string, bool, error) {
return data, false, err
}
if resp.StatusCode != 200 {
return data, false, errkit.New("unexpected status code: %v", resp.Status)
return data, false, errorutil.New("unexpected status code: %v", resp.Status)
}
var templateResp TemplateResp
if err := json.NewDecoder(resp.Body).Decode(&templateResp); err != nil {
@@ -288,20 +288,20 @@ func formatTemplate(data string) (string, bool, error) {
}
if templateResp.ValidateErrorCount > 0 {
if len(templateResp.ValidateError) > 0 {
return data, false, errkit.New(templateResp.ValidateError[0].Message+": at line %v", templateResp.ValidateError[0].Mark.Line, "tag", "validate")
return data, false, errorutil.NewWithTag("validate", templateResp.ValidateError[0].Message+": at line %v", templateResp.ValidateError[0].Mark.Line)
}
return data, false, errkit.New("validation failed", "tag", "validate")
return data, false, errorutil.New("validation failed").WithTag("validate")
}
if templateResp.Error.Name != "" {
return data, false, errkit.New("%s", templateResp.Error.Name)
return data, false, errorutil.New("%s", templateResp.Error.Name)
}
if strings.TrimSpace(templateResp.Updated) == "" && !templateResp.Lint {
if templateResp.LintError.Reason != "" {
return data, false, errkit.New(templateResp.LintError.Reason+" : at line %v", templateResp.LintError.Mark.Line, "tag", "lint")
return data, false, errorutil.NewWithTag("lint", templateResp.LintError.Reason+" : at line %v", templateResp.LintError.Mark.Line)
}
return data, false, errkit.New("at line: %v", templateResp.LintError.Mark.Line, "tag", "lint")
return data, false, errorutil.NewWithTag("lint", "at line: %v", templateResp.LintError.Mark.Line)
}
return data, false, errkit.New("template format failed")
return data, false, errorutil.New("template format failed")
}
// lintTemplate lints template data using templateman lint api
@@ -311,7 +311,7 @@ func lintTemplate(data string) (bool, error) {
return false, err
}
if resp.StatusCode != 200 {
return false, errkit.New("unexpected status code: %v", resp.Status)
return false, errorutil.New("unexpected status code: %v", resp.Status)
}
var lintResp TemplateLintResp
if err := json.NewDecoder(resp.Body).Decode(&lintResp); err != nil {
@@ -321,9 +321,9 @@ func lintTemplate(data string) (bool, error) {
return true, nil
}
if lintResp.LintError.Reason != "" {
return false, errkit.New(lintResp.LintError.Reason+" : at line %v", lintResp.LintError.Mark.Line, "tag", "lint")
return false, errorutil.NewWithTag("lint", lintResp.LintError.Reason+" : at line %v", lintResp.LintError.Mark.Line)
}
return false, errkit.New("at line: %v", lintResp.LintError.Mark.Line, "tag", "lint")
return false, errorutil.NewWithTag("lint", "at line: %v", lintResp.LintError.Mark.Line)
}
// validateTemplate validates template data using templateman validate api
@@ -333,7 +333,7 @@ func validateTemplate(data string) (bool, error) {
return false, err
}
if resp.StatusCode != 200 {
return false, errkit.New("unexpected status code: %v", resp.Status)
return false, errorutil.New("unexpected status code: %v", resp.Status)
}
var validateResp TemplateResp
if err := json.NewDecoder(resp.Body).Decode(&validateResp); err != nil {
@@ -344,14 +344,14 @@ func validateTemplate(data string) (bool, error) {
}
if validateResp.ValidateErrorCount > 0 {
if len(validateResp.ValidateError) > 0 {
return false, errkit.New(validateResp.ValidateError[0].Message+": at line %v", validateResp.ValidateError[0].Mark.Line, "tag", "validate")
return false, errorutil.NewWithTag("validate", validateResp.ValidateError[0].Message+": at line %v", validateResp.ValidateError[0].Mark.Line)
}
return false, errkit.New("validation failed", "tag", "validate")
return false, errorutil.New("validation failed").WithTag("validate")
}
if validateResp.Error.Name != "" {
return false, errkit.New("%s", validateResp.Error.Name)
return false, errorutil.New("%s", validateResp.Error.Name)
}
return false, errkit.New("template validation failed")
return false, errorutil.New("template validation failed")
}
// parseAndAddMaxRequests parses and adds max requests to templates
+38 -67
View File
@@ -20,12 +20,12 @@ require (
github.com/olekukonko/tablewriter v1.0.8
github.com/pkg/errors v0.9.1
github.com/projectdiscovery/clistats v0.1.1
github.com/projectdiscovery/fastdialer v0.4.7
github.com/projectdiscovery/hmap v0.0.93
github.com/projectdiscovery/fastdialer v0.4.2
github.com/projectdiscovery/hmap v0.0.91
github.com/projectdiscovery/interactsh v1.2.4
github.com/projectdiscovery/rawhttp v0.1.90
github.com/projectdiscovery/retryabledns v1.0.106
github.com/projectdiscovery/retryablehttp-go v1.0.121
github.com/projectdiscovery/retryabledns v1.0.103
github.com/projectdiscovery/retryablehttp-go v1.0.118
github.com/projectdiscovery/yamldoc-go v1.0.6
github.com/remeh/sizedwaitgroup v1.0.0
github.com/rs/xid v1.6.0
@@ -35,11 +35,11 @@ require (
github.com/spf13/cast v1.9.2
github.com/syndtr/goleveldb v1.0.0
github.com/valyala/fasttemplate v1.2.2
github.com/weppos/publicsuffix-go v0.50.0
github.com/weppos/publicsuffix-go v0.40.3-0.20250408071509-6074bbe7fd39
go.uber.org/multierr v1.11.0
golang.org/x/net v0.43.0
golang.org/x/net v0.42.0
golang.org/x/oauth2 v0.30.0
golang.org/x/text v0.28.0
golang.org/x/text v0.27.0
gopkg.in/yaml.v2 v2.4.0
)
@@ -67,6 +67,8 @@ require (
github.com/clbanning/mxj/v2 v2.7.0
github.com/ditashi/jsbeautifier-go v0.0.0-20141206144643-2520a8026a9c
github.com/docker/go-units v0.5.0
github.com/dop251/goja v0.0.0-20250630131328-58d95d85e994
github.com/dop251/goja_nodejs v0.0.0-20250409162600-f7acab6894b0
github.com/fatih/structs v1.1.0
github.com/getkin/kin-openapi v0.132.0
github.com/go-git/go-git/v5 v5.16.2
@@ -87,51 +89,49 @@ require (
github.com/microsoft/go-mssqldb v1.9.2
github.com/ory/dockertest/v3 v3.12.0
github.com/praetorian-inc/fingerprintx v1.1.15
github.com/projectdiscovery/dsl v0.5.1
github.com/projectdiscovery/dsl v0.5.0
github.com/projectdiscovery/fasttemplate v0.0.2
github.com/projectdiscovery/gcache v0.0.0-20241015120333-12546c6e3f4c
github.com/projectdiscovery/go-smb2 v0.0.0-20240129202741-052cc450c6cb
github.com/projectdiscovery/goflags v0.1.74
github.com/projectdiscovery/gologger v1.1.54
github.com/projectdiscovery/gostruct v0.0.2
github.com/projectdiscovery/gozero v0.1.0
github.com/projectdiscovery/httpx v1.7.1-0.20250901093029-3909667cbaa6
github.com/projectdiscovery/gozero v0.0.3
github.com/projectdiscovery/httpx v1.7.1-0.20250806055928-39d76476fddf
github.com/projectdiscovery/mapcidr v1.1.34
github.com/projectdiscovery/n3iwf v0.0.0-20230523120440-b8cd232ff1f5
github.com/projectdiscovery/networkpolicy v0.1.21
github.com/projectdiscovery/ratelimit v0.0.82
github.com/projectdiscovery/networkpolicy v0.1.18
github.com/projectdiscovery/ratelimit v0.0.81
github.com/projectdiscovery/rdap v0.9.0
github.com/projectdiscovery/sarif v0.0.1
github.com/projectdiscovery/tlsx v1.2.1
github.com/projectdiscovery/tlsx v1.2.0
github.com/projectdiscovery/uncover v1.1.0
github.com/projectdiscovery/useragent v0.0.101
github.com/projectdiscovery/utils v0.4.24-0.20250823123502-bd7f2849ddb4
github.com/projectdiscovery/wappalyzergo v0.2.43
github.com/projectdiscovery/utils v0.4.22
github.com/projectdiscovery/wappalyzergo v0.2.38
github.com/redis/go-redis/v9 v9.11.0
github.com/seh-msft/burpxml v1.0.1
github.com/shurcooL/graphql v0.0.0-20230722043721-ed46e5a46466
github.com/stretchr/testify v1.11.0
github.com/stretchr/testify v1.10.0
github.com/tarunKoyalwar/goleak v0.0.0-20240429141123-0efa90dbdcf9
github.com/testcontainers/testcontainers-go v0.38.0
github.com/testcontainers/testcontainers-go/modules/mongodb v0.37.0
github.com/yassinebenaid/godump v0.11.1
github.com/zmap/zgrab2 v0.1.8
gitlab.com/gitlab-org/api/client-go v0.130.1
go.mongodb.org/mongo-driver v1.17.4
golang.org/x/term v0.34.0
golang.org/x/term v0.33.0
gopkg.in/yaml.v3 v3.0.1
moul.io/http2curl v1.0.0
)
require (
aead.dev/minisign v0.2.0 // indirect
dario.cat/mergo v1.0.2 // indirect
dario.cat/mergo v1.0.0 // indirect
filippo.io/edwards25519 v1.1.0 // indirect
git.mills.io/prologic/smtpd v0.0.0-20210710122116-a525b76c287a // indirect
github.com/42wim/httpsig v1.2.2 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.18.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.1 // indirect
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect
github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 // indirect
github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.4.2 // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
@@ -187,29 +187,21 @@ require (
github.com/cloudwego/base64x v0.1.5 // indirect
github.com/cnf/structhash v0.0.0-20201127153200-e1b16c1ebc08 // indirect
github.com/containerd/continuity v0.4.5 // indirect
github.com/containerd/errdefs v1.0.0 // indirect
github.com/containerd/errdefs/pkg v0.3.0 // indirect
github.com/containerd/log v0.1.0 // indirect
github.com/containerd/platforms v0.2.1 // indirect
github.com/cpuguy83/dockercfg v0.3.2 // indirect
github.com/cyphar/filepath-securejoin v0.4.1 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/davidmz/go-pageant v1.0.2 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/distribution/reference v0.6.0 // indirect
github.com/dlclark/regexp2 v1.11.5 // indirect
github.com/docker/cli v27.4.1+incompatible // indirect
github.com/docker/docker v28.3.3+incompatible // indirect
github.com/docker/go-connections v0.6.0 // indirect
github.com/docker/docker v27.1.1+incompatible // indirect
github.com/docker/go-connections v0.5.0 // indirect
github.com/dsnet/compress v0.0.2-0.20230904184137-39efe44ab707 // indirect
github.com/ebitengine/purego v0.8.4 // indirect
github.com/emirpasic/gods v1.18.1 // indirect
github.com/fatih/color v1.18.0 // indirect
github.com/felixge/fgprof v0.9.5 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/free5gc/util v1.0.5-0.20230511064842-2e120956883b // indirect
github.com/gabriel-vasile/mimetype v1.4.8 // indirect
github.com/gaissmai/bart v0.24.0 // indirect
github.com/gaissmai/bart v0.20.5 // indirect
github.com/geoffgarside/ber v1.1.0 // indirect
github.com/gin-contrib/sse v0.1.0 // indirect
github.com/gin-gonic/gin v1.9.1 // indirect
@@ -217,9 +209,7 @@ require (
github.com/go-fed/httpsig v1.1.0 // indirect
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
github.com/go-git/go-billy/v5 v5.6.2 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.3.0 // indirect
github.com/go-ole/go-ole v1.2.6 // indirect
github.com/go-openapi/jsonpointer v0.21.0 // indirect
github.com/go-openapi/swag v0.23.0 // indirect
github.com/go-playground/locales v0.14.1 // indirect
@@ -267,9 +257,8 @@ require (
github.com/logrusorgru/aurora/v4 v4.0.0 // indirect
github.com/lor00x/goldap v0.0.0-20180618054307-a546dffdd1a3 // indirect
github.com/lucasb-eyer/go-colorful v1.2.0 // indirect
github.com/lufia/plan9stats v0.0.0-20250821153705-5981dea3221d // indirect
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
github.com/mackerelio/go-osstat v0.2.4 // indirect
github.com/magiconair/properties v1.8.10 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
@@ -281,17 +270,12 @@ require (
github.com/minio/selfupdate v0.6.1-0.20230907112617-f11e74f84ca7 // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/moby/go-archive v0.1.0 // indirect
github.com/moby/patternmatcher v0.6.0 // indirect
github.com/moby/sys/sequential v0.6.0 // indirect
github.com/moby/sys/user v0.4.0 // indirect
github.com/moby/sys/userns v0.1.0 // indirect
github.com/moby/term v0.5.2 // indirect
github.com/moby/sys/user v0.3.0 // indirect
github.com/moby/term v0.5.0 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 // indirect
github.com/montanaflynn/stats v0.7.1 // indirect
github.com/morikuni/aec v1.0.0 // indirect
github.com/muesli/reflow v0.3.0 // indirect
github.com/muesli/termenv v0.16.0 // indirect
github.com/nwaples/rardecode/v2 v2.1.0 // indirect
@@ -300,7 +284,7 @@ require (
github.com/olekukonko/errors v1.1.0 // indirect
github.com/olekukonko/ll v0.0.9 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/opencontainers/image-spec v1.1.0 // indirect
github.com/opencontainers/runc v1.2.3 // indirect
github.com/openrdap/rdap v0.9.1 // indirect
github.com/pelletier/go-toml/v2 v2.0.8 // indirect
@@ -309,10 +293,10 @@ require (
github.com/pjbgf/sha1cd v0.3.2 // indirect
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
github.com/projectdiscovery/asnmap v1.1.1 // indirect
github.com/projectdiscovery/blackrock v0.0.1 // indirect
github.com/projectdiscovery/cdncheck v1.1.33 // indirect
github.com/projectdiscovery/cdncheck v1.1.28 // indirect
github.com/projectdiscovery/freeport v0.0.7 // indirect
github.com/projectdiscovery/ldapserver v1.0.2-0.20240219154113-dcc758ebc0cb // indirect
github.com/projectdiscovery/machineid v0.0.0-20240226150047-2e2c51e35983 // indirect
@@ -320,7 +304,6 @@ require (
github.com/sashabaranov/go-openai v1.37.0 // indirect
github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 // indirect
github.com/shirou/gopsutil v3.21.11+incompatible // indirect
github.com/shirou/gopsutil/v4 v4.25.7 // indirect
github.com/shoenig/go-m1cpu v0.1.6 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/skeema/knownhosts v1.3.1 // indirect
@@ -334,8 +317,8 @@ require (
github.com/tidwall/rtred v0.1.2 // indirect
github.com/tidwall/tinyqueue v0.1.1 // indirect
github.com/tim-ywliu/nested-logrus-formatter v1.3.2 // indirect
github.com/tklauser/go-sysconf v0.3.15 // indirect
github.com/tklauser/numcpus v0.10.0 // indirect
github.com/tklauser/go-sysconf v0.3.12 // indirect
github.com/tklauser/numcpus v0.6.1 // indirect
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
github.com/ugorji/go/codec v1.2.11 // indirect
github.com/ulikunitz/xz v0.5.12 // indirect
@@ -352,15 +335,10 @@ require (
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 // indirect
github.com/ysmood/fetchup v0.2.3 // indirect
github.com/ysmood/got v0.40.0 // indirect
github.com/yuin/goldmark v1.7.8 // indirect
github.com/yuin/goldmark v1.7.11 // indirect
github.com/yuin/goldmark-emoji v1.0.5 // indirect
github.com/zcalusic/sysinfo v1.0.2 // indirect
github.com/zeebo/blake3 v0.2.3 // indirect
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.62.0 // indirect
go.opentelemetry.io/otel v1.37.0 // indirect
go.opentelemetry.io/otel/metric v1.37.0 // indirect
go.opentelemetry.io/otel/trace v1.37.0 // indirect
go4.org v0.0.0-20230225012048-214862532bf5 // indirect
golang.org/x/arch v0.3.0 // indirect
golang.org/x/sync v0.16.0 // indirect
@@ -387,12 +365,12 @@ require (
go.etcd.io/bbolt v1.4.0 // indirect
go.uber.org/zap v1.27.0 // indirect
goftp.io/server/v2 v2.0.1 // indirect
golang.org/x/crypto v0.41.0 // indirect
golang.org/x/crypto v0.40.0 // indirect
golang.org/x/exp v0.0.0-20250620022241-b7579e27df2b
golang.org/x/mod v0.26.0 // indirect
golang.org/x/sys v0.35.0 // indirect
golang.org/x/mod v0.25.0 // indirect
golang.org/x/sys v0.34.0 // indirect
golang.org/x/time v0.11.0 // indirect
golang.org/x/tools v0.35.0
golang.org/x/tools v0.34.0
google.golang.org/protobuf v1.36.6 // indirect
gopkg.in/alecthomas/kingpin.v2 v2.2.6 // indirect
gopkg.in/corvus-ch/zbase32.v1 v1.0.0 // indirect
@@ -406,10 +384,3 @@ require (
// https://go.dev/ref/mod#go-mod-file-retract
retract v3.2.0 // retract due to broken js protocol issue
// Fix genproto version conflicts
replace (
google.golang.org/genproto => google.golang.org/genproto v0.0.0-20240814211410-ddb44dafa142
google.golang.org/genproto/googleapis/api => google.golang.org/genproto/googleapis/api v0.0.0-20240814211410-ddb44dafa142
google.golang.org/genproto/googleapis/rpc => google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1
)
+111 -1050
View File
File diff suppressed because it is too large. Load diff
+9 -9
View File
@@ -19,7 +19,7 @@ import (
"github.com/projectdiscovery/retryablehttp-go"
pdcpauth "github.com/projectdiscovery/utils/auth/pdcp"
"github.com/projectdiscovery/utils/env"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
unitutils "github.com/projectdiscovery/utils/unit"
updateutils "github.com/projectdiscovery/utils/update"
urlutil "github.com/projectdiscovery/utils/url"
@@ -77,11 +77,11 @@ func NewUploadWriter(ctx context.Context, logger *gologger.Logger, creds *pdcpau
output.WithJson(true, true),
)
if err != nil {
return nil, errkit.Wrap(err, "could not create output writer")
return nil, errorutil.NewWithErr(err).Msgf("could not create output writer")
}
tmp, err := urlutil.Parse(creds.Server)
if err != nil {
return nil, errkit.Wrap(err, "could not parse server url")
return nil, errorutil.NewWithErr(err).Msgf("could not parse server url")
}
tmp.Path = uploadEndpoint
tmp.Update()
@@ -199,7 +199,7 @@ func (u *UploadWriter) autoCommit(ctx context.Context, r *io.PipeReader) {
// uploadChunk uploads a chunk of data to the server
func (u *UploadWriter) uploadChunk(buff *bytes.Buffer) error {
if err := u.upload(buff.Bytes()); err != nil {
return errkit.Wrap(err, "could not upload chunk")
return errorutil.NewWithErr(err).Msgf("could not upload chunk")
}
// if successful, reset the buffer
buff.Reset()
@@ -211,25 +211,25 @@ func (u *UploadWriter) uploadChunk(buff *bytes.Buffer) error {
func (u *UploadWriter) upload(data []byte) error {
req, err := u.getRequest(data)
if err != nil {
return errkit.Wrap(err, "could not create upload request")
return errorutil.NewWithErr(err).Msgf("could not create upload request")
}
resp, err := u.client.Do(req)
if err != nil {
return errkit.Wrap(err, "could not upload results")
return errorutil.NewWithErr(err).Msgf("could not upload results")
}
defer func() {
_ = resp.Body.Close()
}()
bin, err := io.ReadAll(resp.Body)
if err != nil {
return errkit.Wrap(err, "could not get id from response")
return errorutil.NewWithErr(err).Msgf("could not get id from response")
}
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("could not upload results got status code %v on %v", resp.StatusCode, resp.Request.URL.String())
}
var uploadResp uploadResponse
if err := json.Unmarshal(bin, &uploadResp); err != nil {
return errkit.Wrap(err, fmt.Sprintf("could not unmarshal response got %v", string(bin)))
return errorutil.NewWithErr(err).Msgf("could not unmarshal response got %v", string(bin))
}
if uploadResp.ID != "" && u.scanID == "" {
u.scanID = uploadResp.ID
@@ -254,7 +254,7 @@ func (u *UploadWriter) getRequest(bin []byte) (*retryablehttp.Request, error) {
}
req, err := retryablehttp.NewRequest(method, url, bytes.NewReader(bin))
if err != nil {
return nil, errkit.Wrap(err, "could not create cloud upload request")
return nil, errorutil.NewWithErr(err).Msgf("could not create cloud upload request")
}
// add pdtm meta params
req.Params.Merge(updateutils.GetpdtmParams(config.Version))
+3 -3
View File
@@ -17,7 +17,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/scan"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/utils/env"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
)
type AuthLazyFetchOptions struct {
@@ -32,7 +32,7 @@ func GetAuthTmplStore(opts *types.Options, catalog catalog.Catalog, execOpts *pr
for _, file := range opts.SecretsFile {
data, err := authx.GetTemplatePathsFromSecretFile(file)
if err != nil {
return nil, errkit.Wrap(err, "failed to get template paths from secrets file")
return nil, errorutil.NewWithErr(err).Msgf("failed to get template paths from secrets file")
}
tmpls = append(tmpls, data...)
}
@@ -58,7 +58,7 @@ func GetAuthTmplStore(opts *types.Options, catalog catalog.Catalog, execOpts *pr
cfg.StoreId = loader.AuthStoreId
store, err := loader.New(cfg)
if err != nil {
return nil, errkit.Wrap(err, "failed to initialize dynamic auth templates store")
return nil, errorutil.NewWithErr(err).Msgf("failed to initialize dynamic auth templates store")
}
return store, nil
}
+2 -2
View File
@@ -8,7 +8,7 @@ import (
"strings"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
fileutil "github.com/projectdiscovery/utils/file"
proxyutils "github.com/projectdiscovery/utils/proxy"
)
@@ -50,7 +50,7 @@ func loadProxyServers(options *types.Options) error {
}
proxyURL, err := url.Parse(aliveProxy)
if err != nil {
return errkit.Wrapf(err, "failed to parse proxy got %v", err)
return errorutil.WrapfWithNil(err, "failed to parse proxy got %v", err)
}
if options.ProxyInternal {
_ = os.Setenv(HTTP_PROXY_ENV, proxyURL.String())
-8
View File
@@ -125,14 +125,6 @@ func New(options *types.Options) (*Runner, error) {
}
}
// if template list or template display is enabled, enable all templates
if options.TemplateList || options.TemplateDisplay {
options.EnableCodeTemplates = true
options.EnableFileTemplates = true
options.EnableSelfContainedTemplates = true
options.EnableGlobalMatchersTemplates = true
}
// check for custom template updates and update if available
ctm, err := customtemplates.NewCustomTemplatesManager(options)
if err != nil {
+9 -10
View File
@@ -8,7 +8,6 @@ import (
"github.com/projectdiscovery/goflags"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/ratelimit"
"github.com/projectdiscovery/utils/errkit"
"github.com/projectdiscovery/nuclei/v3/pkg/authprovider"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog"
@@ -103,7 +102,7 @@ type InteractshOpts interactsh.Options
func WithInteractshOptions(opts InteractshOpts) NucleiSDKOptions {
return func(e *NucleiEngine) error {
if e.mode == threadSafe {
return errkit.Wrap(ErrOptionsNotSupported, "WithInteractshOptions")
return ErrOptionsNotSupported.Msgf("WithInteractshOptions")
}
optsPtr := &opts
e.interactshOpts = (*interactsh.Options)(optsPtr)
@@ -230,7 +229,7 @@ type StatsOptions struct {
func EnableStatsWithOpts(opts StatsOptions) NucleiSDKOptions {
return func(e *NucleiEngine) error {
if e.mode == threadSafe {
return errkit.Wrap(ErrOptionsNotSupported, "EnableStatsWithOpts")
return ErrOptionsNotSupported.Msgf("EnableStatsWithOpts")
}
if opts.Interval == 0 {
opts.Interval = 5 //sec
@@ -258,7 +257,7 @@ type VerbosityOptions struct {
func WithVerbosity(opts VerbosityOptions) NucleiSDKOptions {
return func(e *NucleiEngine) error {
if e.mode == threadSafe {
return errkit.Wrap(ErrOptionsNotSupported, "WithVerbosity")
return ErrOptionsNotSupported.Msgf("WithVerbosity")
}
e.opts.Verbose = opts.Verbose
e.opts.Silent = opts.Silent
@@ -291,7 +290,7 @@ type NetworkConfig struct {
func WithNetworkConfig(opts NetworkConfig) NucleiSDKOptions {
return func(e *NucleiEngine) error {
if e.mode == threadSafe {
return errkit.Wrap(ErrOptionsNotSupported, "WithNetworkConfig")
return ErrOptionsNotSupported.Msgf("WithNetworkConfig")
}
e.opts.NoHostErrors = opts.DisableMaxHostErr
e.opts.MaxHostError = opts.MaxHostError
@@ -322,7 +321,7 @@ func WithNetworkConfig(opts NetworkConfig) NucleiSDKOptions {
func WithProxy(proxy []string, proxyInternalRequests bool) NucleiSDKOptions {
return func(e *NucleiEngine) error {
if e.mode == threadSafe {
return errkit.Wrap(ErrOptionsNotSupported, "WithProxy")
return ErrOptionsNotSupported.Msgf("WithProxy")
}
e.opts.Proxy = proxy
e.opts.ProxyInternal = proxyInternalRequests
@@ -347,7 +346,7 @@ type OutputWriter output.Writer
func UseOutputWriter(writer OutputWriter) NucleiSDKOptions {
return func(e *NucleiEngine) error {
if e.mode == threadSafe {
return errkit.Wrap(ErrOptionsNotSupported, "UseOutputWriter")
return ErrOptionsNotSupported.Msgf("UseOutputWriter")
}
e.customWriter = writer
return nil
@@ -362,7 +361,7 @@ type StatsWriter progress.Progress
func UseStatsWriter(writer StatsWriter) NucleiSDKOptions {
return func(e *NucleiEngine) error {
if e.mode == threadSafe {
return errkit.Wrap(ErrOptionsNotSupported, "UseStatsWriter")
return ErrOptionsNotSupported.Msgf("UseStatsWriter")
}
e.customProgress = writer
return nil
@@ -376,7 +375,7 @@ func UseStatsWriter(writer StatsWriter) NucleiSDKOptions {
func WithTemplateUpdateCallback(disableTemplatesAutoUpgrade bool, callback func(newVersion string)) NucleiSDKOptions {
return func(e *NucleiEngine) error {
if e.mode == threadSafe {
return errkit.Wrap(ErrOptionsNotSupported, "WithTemplateUpdateCallback")
return ErrOptionsNotSupported.Msgf("WithTemplateUpdateCallback")
}
e.disableTemplatesAutoUpgrade = disableTemplatesAutoUpgrade
e.onUpdateAvailableCallback = callback
@@ -388,7 +387,7 @@ func WithTemplateUpdateCallback(disableTemplatesAutoUpgrade bool, callback func(
func WithSandboxOptions(allowLocalFileAccess bool, restrictLocalNetworkAccess bool) NucleiSDKOptions {
return func(e *NucleiEngine) error {
if e.mode == threadSafe {
return errkit.Wrap(ErrOptionsNotSupported, "WithSandboxOptions")
return ErrOptionsNotSupported.Msgf("WithSandboxOptions")
}
e.opts.AllowLocalFileAccess = allowLocalFileAccess
e.opts.RestrictLocalNetworkAccess = restrictLocalNetworkAccess
+3 -3
View File
@@ -13,7 +13,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/protocols"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/ratelimit"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
"github.com/rs/xid"
)
@@ -147,13 +147,13 @@ func (e *ThreadSafeNucleiEngine) ExecuteNucleiWithOptsCtx(ctx context.Context, t
// load templates
workflowLoader, err := workflow.NewLoader(unsafeOpts.executerOpts)
if err != nil {
return errkit.Wrapf(err, "Could not create workflow loader: %s", err)
return errorutil.New("Could not create workflow loader: %s\n", err)
}
unsafeOpts.executerOpts.WorkflowLoader = workflowLoader
store, err := loader.New(loader.NewConfig(tmpEngine.opts, e.eng.catalog, unsafeOpts.executerOpts))
if err != nil {
return errkit.Wrapf(err, "Could not create loader client: %s", err)
return errorutil.New("Could not create loader client: %s\n", err)
}
store.Load()
+7 -7
View File
@@ -28,7 +28,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/ratelimit"
"github.com/projectdiscovery/retryablehttp-go"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
"github.com/rs/xid"
)
@@ -37,13 +37,13 @@ type NucleiSDKOptions func(e *NucleiEngine) error
var (
// ErrNotImplemented is returned when a feature is not implemented
ErrNotImplemented = errkit.New("Not implemented")
ErrNotImplemented = errorutil.New("Not implemented")
// ErrNoTemplatesAvailable is returned when no templates are available to execute
ErrNoTemplatesAvailable = errkit.New("No templates available")
ErrNoTemplatesAvailable = errorutil.New("No templates available")
// ErrNoTargetsAvailable is returned when no targets are available to scan
ErrNoTargetsAvailable = errkit.New("No targets available")
ErrNoTargetsAvailable = errorutil.New("No targets available")
// ErrOptionsNotSupported is returned when an option is not supported in thread safe mode
ErrOptionsNotSupported = errkit.New("Option not supported in thread safe mode")
ErrOptionsNotSupported = errorutil.NewWithFmt("Option %v not supported in thread safe mode")
)
type engineMode uint
@@ -98,13 +98,13 @@ type NucleiEngine struct {
func (e *NucleiEngine) LoadAllTemplates() error {
workflowLoader, err := workflow.NewLoader(e.executerOpts)
if err != nil {
return errkit.Wrapf(err, "Could not create workflow loader: %s", err)
return errorutil.New("Could not create workflow loader: %s\n", err)
}
e.executerOpts.WorkflowLoader = workflowLoader
e.store, err = loader.New(loader.NewConfig(e.opts, e.catalog, e.executerOpts))
if err != nil {
return errkit.Wrapf(err, "Could not create loader client: %s", err)
return errorutil.New("Could not create loader client: %s\n", err)
}
e.store.Load()
e.templatesLoaded = true
+1 -22
View File
@@ -3,7 +3,6 @@ package nuclei
import (
"context"
"fmt"
"os"
"strings"
"sync"
"time"
@@ -38,21 +37,6 @@ import (
"github.com/projectdiscovery/ratelimit"
)
// enginePool provides shared parsed-cache across engines (opt-in)
var (
sharedParsedOnce sync.Once
sharedParsed *templates.Cache
)
func getSharedParser() *templates.Parser {
// Initialize the shared parsed cache once
sharedParsedOnce.Do(func() {
sharedParsed = templates.NewCache()
})
// Return a fresh Parser each call that reuses only the shared parsed cache,
return templates.NewParserWithParsedCache(sharedParsed)
}
// applyRequiredDefaults to options
func (e *NucleiEngine) applyRequiredDefaults(ctx context.Context) {
mockoutput := testutils.NewMockOutputWriter(e.opts.OmitTemplate)
@@ -139,12 +123,7 @@ func (e *NucleiEngine) init(ctx context.Context) error {
}
if e.parser == nil {
//TODO: remove this feature flag after testing
if os.Getenv("NUCLEI_USE_SHARED_COMPILED") == "1" {
e.parser = templates.NewSharedParserWithCompiledCache()
} else {
e.parser = templates.NewParser()
}
e.parser = templates.NewParser()
}
if protocolstate.ShouldInit(e.opts.ExecutionId) {
-26
View File
@@ -3,7 +3,6 @@ package nuclei_test
import (
"context"
"log"
"os"
"testing"
"time"
@@ -36,28 +35,3 @@ func TestContextCancelNucleiEngine(t *testing.T) {
}
defer ne.Close()
}
func TestSharedParserOptIn(t *testing.T) {
os.Setenv("NUCLEI_USE_SHARED_PARSER", "1")
t.Cleanup(func() { os.Unsetenv("NUCLEI_USE_SHARED_PARSER") })
ne, err := nuclei.NewNucleiEngineCtx(context.Background())
if err != nil {
t.Fatalf("engine error: %v", err)
}
p := ne.GetParser()
if p == nil {
t.Fatalf("expected templates.Parser")
}
ne2, err := nuclei.NewNucleiEngineCtx(context.Background())
if err != nil {
t.Fatalf("engine2 error: %v", err)
}
p2 := ne2.GetParser()
if p2 == nil {
t.Fatalf("expected templates.Parser2")
}
if p.Cache() != p2.Cache() {
t.Fatalf("expected shared parsed cache across engines when opt-in is set")
}
}
+4 -4
View File
@@ -8,7 +8,7 @@ import (
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/replacer"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/json"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
sliceutil "github.com/projectdiscovery/utils/slice"
)
@@ -53,7 +53,7 @@ func (d *Dynamic) GetDomainAndDomainRegex() ([]string, []string) {
func (d *Dynamic) UnmarshalJSON(data []byte) error {
if d == nil {
return errkit.New("cannot unmarshal into nil Dynamic struct")
return errorutil.New("cannot unmarshal into nil Dynamic struct")
}
// Use an alias type (auxiliary) to avoid a recursive call in this method.
@@ -72,10 +72,10 @@ func (d *Dynamic) UnmarshalJSON(data []byte) error {
func (d *Dynamic) Validate() error {
d.m = &sync.Mutex{}
if d.TemplatePath == "" {
return errkit.New(" template-path is required for dynamic secret")
return errorutil.New(" template-path is required for dynamic secret")
}
if len(d.Variables) == 0 {
return errkit.New("variables are required for dynamic secret")
return errorutil.New("variables are required for dynamic secret")
}
if d.Secret != nil {
+3 -7
View File
@@ -8,7 +8,7 @@ import (
"strings"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/json"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
"github.com/projectdiscovery/utils/generic"
stringsutil "github.com/projectdiscovery/utils/strings"
"gopkg.in/yaml.v3"
@@ -237,9 +237,7 @@ func GetAuthDataFromYAML(data []byte) (*Authx, error) {
var auth Authx
err := yaml.Unmarshal(data, &auth)
if err != nil {
errorErr := errkit.FromError(err)
errorErr.Msgf("could not unmarshal yaml")
return nil, errorErr
return nil, errorutil.NewWithErr(err).Msgf("could not unmarshal yaml")
}
return &auth, nil
}
@@ -249,9 +247,7 @@ func GetAuthDataFromJSON(data []byte) (*Authx, error) {
var auth Authx
err := json.Unmarshal(data, &auth)
if err != nil {
errorErr := errkit.FromError(err)
errorErr.Msgf("could not unmarshal json")
return nil, errorErr
return nil, errorutil.NewWithErr(err).Msgf("could not unmarshal json")
}
return &auth, nil
}
+4 -8
View File
@@ -7,7 +7,7 @@ import (
"strings"
"github.com/projectdiscovery/nuclei/v3/pkg/authprovider/authx"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
urlutil "github.com/projectdiscovery/utils/url"
)
@@ -30,20 +30,16 @@ func NewFileAuthProvider(path string, callback authx.LazyFetchSecret) (AuthProvi
return nil, ErrNoSecrets
}
if len(store.Dynamic) > 0 && callback == nil {
return nil, errkit.New("lazy fetch callback is required for dynamic secrets")
return nil, errorutil.New("lazy fetch callback is required for dynamic secrets")
}
for _, secret := range store.Secrets {
if err := secret.Validate(); err != nil {
errorErr := errkit.FromError(err)
errorErr.Msgf("invalid secret in file: %s", path)
return nil, errorErr
return nil, errorutil.NewWithErr(err).Msgf("invalid secret in file: %s", path)
}
}
for i, dynamic := range store.Dynamic {
if err := dynamic.Validate(); err != nil {
errorErr := errkit.FromError(err)
errorErr.Msgf("invalid dynamic in file: %s", path)
return nil, errorErr
return nil, errorutil.NewWithErr(err).Msgf("invalid dynamic in file: %s", path)
}
dynamic.SetLazyFetchCallback(callback)
store.Dynamic[i] = dynamic
+7 -10
View File
@@ -31,7 +31,7 @@ const (
CLIConfigFileName = "config.yaml"
ReportingConfigFilename = "reporting-config.yaml"
// Version is the current version of nuclei
Version = `v3.4.10`
Version = `v3.4.7`
// Directory Names of custom templates
CustomS3TemplatesDirName = "s3"
CustomGitHubTemplatesDirName = "github"
@@ -46,21 +46,18 @@ const (
// if the current version is outdated
func IsOutdatedVersion(current, latest string) bool {
if latest == "" {
// NOTE(dwisiswant0): if PDTM API call failed or returned empty, we
// cannot determine if templates are outdated w/o additional checks
// return false to avoid unnecessary updates.
return false
// if pdtm api call failed it's assumed that the current version is outdated
// and it will be confirmed while updating from GitHub
// this fixes `version string empty` errors
return true
}
current = trimDevIfExists(current)
currentVer, _ := semver.NewVersion(current)
newVer, _ := semver.NewVersion(latest)
if currentVer == nil || newVer == nil {
// fallback to naive comparison - return true only if they are different
return current != latest
// fallback to naive comparison
return current == latest
}
return newVer.GreaterThan(currentVer)
}
+10 -10
View File
@@ -13,7 +13,7 @@ import (
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/json"
"github.com/projectdiscovery/utils/env"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
fileutil "github.com/projectdiscovery/utils/file"
folderutil "github.com/projectdiscovery/utils/folder"
)
@@ -140,13 +140,13 @@ func (c *Config) UpdateNucleiIgnoreHash() error {
if fileutil.FileExists(ignoreFilePath) {
bin, err := os.ReadFile(ignoreFilePath)
if err != nil {
return errkit.Newf("could not read nuclei ignore file: %v", err)
return errorutil.NewWithErr(err).Msgf("could not read nuclei ignore file")
}
c.NucleiIgnoreHash = fmt.Sprintf("%x", md5.Sum(bin))
// write config to disk
return c.WriteTemplatesConfig()
}
return errkit.New("ignore file not found: could not update nuclei ignore hash")
return errorutil.NewWithTag("config", "ignore file not found: could not update nuclei ignore hash")
}
// GetConfigDir returns the nuclei configuration directory
@@ -257,7 +257,7 @@ func (c *Config) SetTemplatesVersion(version string) error {
c.TemplateVersion = version
// write config to disk
if err := c.WriteTemplatesConfig(); err != nil {
return errkit.Newf("could not write nuclei config file at %s: %v", c.getTemplatesConfigFilePath(), err)
return errorutil.NewWithErr(err).Msgf("could not write nuclei config file at %s", c.getTemplatesConfigFilePath())
}
return nil
}
@@ -265,15 +265,15 @@ func (c *Config) SetTemplatesVersion(version string) error {
// ReadTemplatesConfig reads the nuclei templates config file
func (c *Config) ReadTemplatesConfig() error {
if !fileutil.FileExists(c.getTemplatesConfigFilePath()) {
return errkit.Newf("nuclei config file at %s does not exist", c.getTemplatesConfigFilePath())
return errorutil.NewWithTag("config", "nuclei config file at %s does not exist", c.getTemplatesConfigFilePath())
}
var cfg *Config
bin, err := os.ReadFile(c.getTemplatesConfigFilePath())
if err != nil {
return errkit.Newf("could not read nuclei config file at %s: %v", c.getTemplatesConfigFilePath(), err)
return errorutil.NewWithErr(err).Msgf("could not read nuclei config file at %s", c.getTemplatesConfigFilePath())
}
if err := json.Unmarshal(bin, &cfg); err != nil {
return errkit.Newf("could not unmarshal nuclei config file at %s: %v", c.getTemplatesConfigFilePath(), err)
return errorutil.NewWithErr(err).Msgf("could not unmarshal nuclei config file at %s", c.getTemplatesConfigFilePath())
}
// apply config
c.TemplatesDirectory = cfg.TemplatesDirectory
@@ -292,10 +292,10 @@ func (c *Config) WriteTemplatesConfig() error {
}
bin, err := json.Marshal(c)
if err != nil {
return errkit.Newf("failed to marshal nuclei config: %v", err)
return errorutil.NewWithErr(err).Msgf("failed to marshal nuclei config")
}
if err = os.WriteFile(c.getTemplatesConfigFilePath(), bin, 0600); err != nil {
return errkit.Newf("failed to write nuclei config file at %s: %v", c.getTemplatesConfigFilePath(), err)
return errorutil.NewWithErr(err).Msgf("failed to write nuclei config file at %s", c.getTemplatesConfigFilePath())
}
return nil
}
@@ -319,7 +319,7 @@ func (c *Config) getTemplatesConfigFilePath() string {
func (c *Config) createConfigDirIfNotExists() error {
if !fileutil.FolderExists(c.configDir) {
if err := fileutil.CreateFolder(c.configDir); err != nil {
return errkit.Newf("could not create nuclei config directory at %s: %v", c.configDir, err)
return errorutil.NewWithErr(err).Msgf("could not create nuclei config directory at %s", c.configDir)
}
}
return nil
+15 -15
View File
@@ -14,7 +14,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/retryablehttp-go"
pdcpauth "github.com/projectdiscovery/utils/auth/pdcp"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
)
const (
@@ -33,27 +33,27 @@ type AITemplateResponse struct {
func getAIGeneratedTemplates(prompt string, options *types.Options) ([]string, error) {
prompt = strings.TrimSpace(prompt)
if len(prompt) < 5 {
return nil, errkit.Newf("Prompt is too short. Please provide a more descriptive prompt")
return nil, errorutil.New("Prompt is too short. Please provide a more descriptive prompt")
}
if len(prompt) > 3000 {
return nil, errkit.Newf("Prompt is too long. Please limit to 3000 characters")
return nil, errorutil.New("Prompt is too long. Please limit to 3000 characters")
}
template, templateID, err := generateAITemplate(prompt)
if err != nil {
return nil, errkit.Newf("Failed to generate template: %v", err)
return nil, errorutil.New("Failed to generate template: %v", err)
}
pdcpTemplateDir := filepath.Join(config.DefaultConfig.GetTemplateDir(), "pdcp")
if err := os.MkdirAll(pdcpTemplateDir, 0755); err != nil {
return nil, errkit.Newf("Failed to create pdcp template directory: %v", err)
return nil, errorutil.New("Failed to create pdcp template directory: %v", err)
}
templateFile := filepath.Join(pdcpTemplateDir, templateID+".yaml")
err = os.WriteFile(templateFile, []byte(template), 0644)
if err != nil {
return nil, errkit.Newf("Failed to generate template: %v", err)
return nil, errorutil.New("Failed to generate template: %v", err)
}
options.Logger.Info().Msgf("Generated template available at: https://cloud.projectdiscovery.io/templates/%s", templateID)
@@ -91,22 +91,22 @@ func generateAITemplate(prompt string) (string, string, error) {
}
jsonBody, err := json.Marshal(reqBody)
if err != nil {
return "", "", errkit.Newf("Failed to marshal request body: %v", err)
return "", "", errorutil.New("Failed to marshal request body: %v", err)
}
req, err := http.NewRequest(http.MethodPost, aiTemplateGeneratorAPIEndpoint, bytes.NewBuffer(jsonBody))
if err != nil {
return "", "", errkit.Newf("Failed to create HTTP request: %v", err)
return "", "", errorutil.New("Failed to create HTTP request: %v", err)
}
ph := pdcpauth.PDCPCredHandler{}
creds, err := ph.GetCreds()
if err != nil {
return "", "", errkit.Newf("Failed to get PDCP credentials: %v", err)
return "", "", errorutil.New("Failed to get PDCP credentials: %v", err)
}
if creds == nil {
return "", "", errkit.Newf("PDCP API Key not configured, Create one for free at https://cloud.projectdiscovery.io/")
return "", "", errorutil.New("PDCP API Key not configured, Create one for free at https://cloud.projectdiscovery.io/")
}
req.Header.Set("Content-Type", "application/json")
@@ -114,28 +114,28 @@ func generateAITemplate(prompt string) (string, string, error) {
resp, err := retryablehttp.DefaultClient().Do(req)
if err != nil {
return "", "", errkit.Newf("Failed to send HTTP request: %v", err)
return "", "", errorutil.New("Failed to send HTTP request: %v", err)
}
defer func() {
_ = resp.Body.Close()
}()
if resp.StatusCode == http.StatusUnauthorized {
return "", "", errkit.Newf("Invalid API Key or API Key not configured, Create one for free at https://cloud.projectdiscovery.io/")
return "", "", errorutil.New("Invalid API Key or API Key not configured, Create one for free at https://cloud.projectdiscovery.io/")
}
if resp.StatusCode != http.StatusOK {
body, _ := io.ReadAll(resp.Body)
return "", "", errkit.Newf("API returned status code %d: %s", resp.StatusCode, string(body))
return "", "", errorutil.New("API returned status code %d: %s", resp.StatusCode, string(body))
}
var result AITemplateResponse
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
return "", "", errkit.Newf("Failed to decode API response: %v", err)
return "", "", errorutil.New("Failed to decode API response: %v", err)
}
if result.TemplateID == "" || result.Completion == "" {
return "", "", errkit.Newf("Failed to generate template")
return "", "", errorutil.New("Failed to generate template")
}
return result.Completion, result.TemplateID, nil
+2 -2
View File
@@ -24,7 +24,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/utils/stats"
"github.com/projectdiscovery/nuclei/v3/pkg/workflows"
"github.com/projectdiscovery/retryablehttp-go"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
sliceutil "github.com/projectdiscovery/utils/slice"
stringsutil "github.com/projectdiscovery/utils/strings"
syncutil "github.com/projectdiscovery/utils/sync"
@@ -238,7 +238,7 @@ func (store *Store) ReadTemplateFromURI(uri string, remote bool) ([]byte, error)
uri = handleTemplatesEditorURLs(uri)
remoteTemplates, _, err := getRemoteTemplatesAndWorkflows([]string{uri}, nil, store.config.RemoteTemplateDomainList)
if err != nil || len(remoteTemplates) == 0 {
return nil, errkit.Wrapf(err, "Could not load template %s: got %v", uri, remoteTemplates)
return nil, errorutil.NewWithErr(err).Msgf("Could not load template %s: got %v", uri, remoteTemplates)
}
resp, err := retryablehttp.Get(remoteTemplates[0])
if err != nil {
+2 -4
View File
@@ -12,7 +12,7 @@ import (
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
)
var _ Provider = &customTemplateAzureBlob{}
@@ -29,9 +29,7 @@ func NewAzureProviders(options *types.Options) ([]*customTemplateAzureBlob, erro
// Establish a connection to Azure and build a client object with which to download templates from Azure Blob Storage
azClient, err := getAzureBlobClient(options.AzureTenantID, options.AzureClientID, options.AzureClientSecret, options.AzureServiceURL)
if err != nil {
errx := errkit.FromError(err)
errx.Msgf("Error establishing Azure Blob client for %s", options.AzureContainerName)
return nil, errx
return nil, errorutil.NewWithErr(err).Msgf("Error establishing Azure Blob client for %s", options.AzureContainerName)
}
// Create a new Azure Blob Storage container object
+2 -4
View File
@@ -9,7 +9,7 @@ import (
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
gitlab "gitlab.com/gitlab-org/api/client-go"
)
@@ -28,9 +28,7 @@ func NewGitLabProviders(options *types.Options) ([]*customTemplateGitLabRepo, er
// Establish a connection to GitLab and build a client object with which to download templates from GitLab
gitLabClient, err := getGitLabClient(options.GitLabServerURL, options.GitLabToken)
if err != nil {
errx := errkit.FromError(err)
errx.Msgf("Error establishing GitLab client for %s %s", options.GitLabServerURL, err)
return nil, errx
return nil, errorutil.NewWithErr(err).Msgf("Error establishing GitLab client for %s %s", options.GitLabServerURL, err)
}
// Create a new GitLab service client
+2 -4
View File
@@ -14,7 +14,7 @@ import (
"github.com/projectdiscovery/gologger"
nucleiConfig "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
stringsutil "github.com/projectdiscovery/utils/strings"
)
@@ -64,9 +64,7 @@ func NewS3Providers(options *types.Options) ([]*customTemplateS3Bucket, error) {
if options.AwsBucketName != "" && !options.AwsTemplateDisableDownload {
s3c, err := getS3Client(context.TODO(), options.AwsAccessKey, options.AwsSecretKey, options.AwsRegion, options.AwsProfile)
if err != nil {
errx := errkit.FromError(err)
errx.Msgf("error downloading s3 bucket %s", options.AwsBucketName)
return nil, errx
return nil, errorutil.NewWithErr(err).Msgf("error downloading s3 bucket %s", options.AwsBucketName)
}
ctBucket := &customTemplateS3Bucket{
bucketName: options.AwsBucketName,
+5 -13
View File
@@ -4,7 +4,7 @@ import (
"context"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
)
type Provider interface {
@@ -38,9 +38,7 @@ func NewCustomTemplatesManager(options *types.Options) (*CustomTemplatesManager,
// Add GitHub providers
githubProviders, err := NewGitHubProviders(options)
if err != nil {
errx := errkit.FromError(err)
errx.Msgf("could not create github providers for custom templates")
return nil, errx
return nil, errorutil.NewWithErr(err).Msgf("could not create github providers for custom templates")
}
for _, v := range githubProviders {
ctm.providers = append(ctm.providers, v)
@@ -49,9 +47,7 @@ func NewCustomTemplatesManager(options *types.Options) (*CustomTemplatesManager,
// Add AWS S3 providers
s3Providers, err := NewS3Providers(options)
if err != nil {
errx := errkit.FromError(err)
errx.Msgf("could not create s3 providers for custom templates")
return nil, errx
return nil, errorutil.NewWithErr(err).Msgf("could not create s3 providers for custom templates")
}
for _, v := range s3Providers {
ctm.providers = append(ctm.providers, v)
@@ -60,9 +56,7 @@ func NewCustomTemplatesManager(options *types.Options) (*CustomTemplatesManager,
// Add Azure providers
azureProviders, err := NewAzureProviders(options)
if err != nil {
errx := errkit.FromError(err)
errx.Msgf("could not create azure providers for custom templates")
return nil, errx
return nil, errorutil.NewWithErr(err).Msgf("could not create azure providers for custom templates")
}
for _, v := range azureProviders {
ctm.providers = append(ctm.providers, v)
@@ -71,9 +65,7 @@ func NewCustomTemplatesManager(options *types.Options) (*CustomTemplatesManager,
// Add GitLab providers
gitlabProviders, err := NewGitLabProviders(options)
if err != nil {
errx := errkit.FromError(err)
errx.Msgf("could not create gitlab providers for custom templates")
return nil, errx
return nil, errorutil.NewWithErr(err).Msgf("could not create gitlab providers for custom templates")
}
for _, v := range gitlabProviders {
ctm.providers = append(ctm.providers, v)
+36 -49
View File
@@ -7,6 +7,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/fuzz/dataformat"
"github.com/projectdiscovery/retryablehttp-go"
mapsutil "github.com/projectdiscovery/utils/maps"
urlutil "github.com/projectdiscovery/utils/url"
)
@@ -37,18 +38,12 @@ func (q *Path) Parse(req *retryablehttp.Request) (bool, error) {
splitted := strings.Split(req.Path, "/")
values := make(map[string]interface{})
for i, segment := range splitted {
if segment == "" && i == 0 {
// Skip the first empty segment from leading "/"
for i := range splitted {
pathTillNow := strings.Join(splitted[:i+1], "/")
if pathTillNow == "" {
continue
}
if segment == "" {
// Skip any other empty segments
continue
}
// Use 1-based indexing and store individual segments
key := strconv.Itoa(len(values) + 1)
values[key] = segment
values[strconv.Itoa(i)] = pathTillNow
}
q.value.SetParsed(dataformat.KVMap(values), "")
return true, nil
@@ -69,7 +64,7 @@ func (q *Path) Iterate(callback func(key string, value interface{}) error) (err
// SetValue sets a value in the component
// for a key
func (q *Path) SetValue(key string, value string) error {
escaped := urlutil.PathEncode(value)
escaped := urlutil.ParamEncode(value)
if !q.value.SetParsedValue(key, escaped) {
return ErrSetValue
}
@@ -87,49 +82,41 @@ func (q *Path) Delete(key string) error {
// Rebuild returns a new request with the
// component rebuilt
func (q *Path) Rebuild() (*retryablehttp.Request, error) {
// Get the original path segments
originalSplitted := strings.Split(q.req.Path, "/")
// Create a new slice to hold the rebuilt segments
rebuiltSegments := make([]string, 0, len(originalSplitted))
// Add the first empty segment (from leading "/")
if len(originalSplitted) > 0 && originalSplitted[0] == "" {
rebuiltSegments = append(rebuiltSegments, "")
}
// Process each segment
segmentIndex := 1 // 1-based indexing for our stored values
for i := 1; i < len(originalSplitted); i++ {
originalSegment := originalSplitted[i]
if originalSegment == "" {
// Skip empty segments
originalValues := mapsutil.Map[string, any]{}
splitted := strings.Split(q.req.Path, "/")
for i := range splitted {
pathTillNow := strings.Join(splitted[:i+1], "/")
if pathTillNow == "" {
continue
}
// Check if we have a replacement for this segment
key := strconv.Itoa(segmentIndex)
if newValue, exists := q.value.parsed.Map.GetOrDefault(key, "").(string); exists && newValue != "" {
rebuiltSegments = append(rebuiltSegments, newValue)
} else {
rebuiltSegments = append(rebuiltSegments, originalSegment)
originalValues[strconv.Itoa(i)] = pathTillNow
}
originalPath := q.req.Path
lengthSplitted := len(q.value.parsed.Map)
for i := lengthSplitted; i > 0; i-- {
key := strconv.Itoa(i)
original, ok := originalValues.GetOrDefault(key, "").(string)
if !ok {
continue
}
segmentIndex++
}
// Join the segments back into a path
rebuiltPath := strings.Join(rebuiltSegments, "/")
if unescaped, err := urlutil.PathDecode(rebuiltPath); err == nil {
// this is handle the case where anyportion of path has url encoded data
// by default the http/request official library will escape/encode special characters in path
// to avoid double encoding we unescape/decode already encoded value
//
// if there is a invalid url encoded value like %99 then it will still be encoded as %2599 and not %99
// the only way to make sure it stays as %99 is to implement raw request and unsafe for fuzzing as well
rebuiltPath = unescaped
new, ok := q.value.parsed.Map.GetOrDefault(key, "").(string)
if !ok {
continue
}
if new == original {
// no need to replace
continue
}
originalPath = strings.Replace(originalPath, original, new, 1)
}
rebuiltPath := originalPath
// Clone the request and update the path
cloned := q.req.Clone(context.Background())
if err := cloned.UpdateRelPath(rebuiltPath, true); err != nil {
+5 -54
View File
@@ -29,9 +29,9 @@ func TestURLComponent(t *testing.T) {
})
require.Equal(t, []string{"1"}, keys, "unexpected keys")
require.Equal(t, []string{"testpath"}, values, "unexpected values")
require.Equal(t, []string{"/testpath"}, values, "unexpected values")
err = urlComponent.SetValue("1", "newpath")
err = urlComponent.SetValue("1", "/newpath")
if err != nil {
t.Fatal(err)
}
@@ -61,10 +61,9 @@ func TestURLComponent_NestedPaths(t *testing.T) {
isSet := false
_ = path.Iterate(func(key string, value interface{}) error {
t.Logf("Key: %s, Value: %s", key, value.(string))
if !isSet && value.(string) == "753" {
if !isSet && value.(string) == "/user/753" {
isSet = true
if setErr := path.SetValue(key, "753'"); setErr != nil {
if setErr := path.SetValue(key, "/user/753'"); setErr != nil {
t.Fatal(setErr)
}
}
@@ -76,54 +75,6 @@ func TestURLComponent_NestedPaths(t *testing.T) {
t.Fatal(err)
}
if newReq.Path != "/user/753'/profile" {
t.Fatalf("expected path to be '/user/753'/profile', got '%s'", newReq.Path)
t.Fatal("expected path to be modified")
}
}
func TestPathComponent_SQLInjection(t *testing.T) {
path := NewPath()
req, err := retryablehttp.NewRequest(http.MethodGet, "https://example.com/user/55/profile", nil)
if err != nil {
t.Fatal(err)
}
found, err := path.Parse(req)
if err != nil {
t.Fatal(err)
}
if !found {
t.Fatal("expected path to be found")
}
t.Logf("Original path: %s", req.Path)
// Let's see what path segments are available for fuzzing
err = path.Iterate(func(key string, value interface{}) error {
t.Logf("Key: %s, Value: %s", key, value.(string))
// Try fuzzing the "55" segment specifically (which should be key "2")
if value.(string) == "55" {
if setErr := path.SetValue(key, "55 OR True"); setErr != nil {
t.Fatal(setErr)
}
}
return nil
})
if err != nil {
t.Fatal(err)
}
newReq, err := path.Rebuild()
if err != nil {
t.Fatal(err)
}
t.Logf("Modified path: %s", newReq.Path)
// Now with PathEncode, spaces are preserved correctly for SQL injection
if newReq.Path != "/user/55 OR True/profile" {
t.Fatalf("expected path to be '/user/55 OR True/profile', got '%s'", newReq.Path)
}
// Let's also test what the actual URL looks like
t.Logf("Full URL: %s", newReq.String())
}
+26 -45
View File
@@ -49,61 +49,42 @@ func (m *MultiPartForm) Encode(data KV) (string, error) {
var fw io.Writer
var err error
if fileMetadata, ok := m.filesMetadata[key]; ok {
if filesArray, isArray := value.([]any); isArray {
for _, file := range filesArray {
h := make(textproto.MIMEHeader)
h.Set("Content-Disposition",
fmt.Sprintf(`form-data; name=%q; filename=%q`,
key, fileMetadata.Filename))
h.Set("Content-Type", fileMetadata.ContentType)
if filesArray, ok := value.([]interface{}); ok {
fileMetadata, ok := m.filesMetadata[key]
if !ok {
Itererr = fmt.Errorf("file metadata not found for key %s", key)
return false
}
if fw, err = w.CreatePart(h); err != nil {
Itererr = err
return false
}
for _, file := range filesArray {
h := make(textproto.MIMEHeader)
h.Set("Content-Disposition",
fmt.Sprintf(`form-data; name=%q; filename=%q`,
key, fileMetadata.Filename))
h.Set("Content-Type", fileMetadata.ContentType)
if _, err = fw.Write([]byte(file.(string))); err != nil {
Itererr = err
return false
}
if fw, err = w.CreatePart(h); err != nil {
Itererr = err
return false
}
return true
if _, err = fw.Write([]byte(file.(string))); err != nil {
Itererr = err
return false
}
}
return true
}
// Add field
var values []string
switch v := value.(type) {
case nil:
values = []string{""}
case string:
values = []string{v}
case []string:
values = v
case []any:
values = make([]string, len(v))
for i, item := range v {
if item == nil {
values[i] = ""
} else {
values[i] = fmt.Sprint(item)
}
}
default:
values = []string{fmt.Sprintf("%v", v)}
if fw, err = w.CreateFormField(key); err != nil {
Itererr = err
return false
}
for _, val := range values {
if fw, err = w.CreateFormField(key); err != nil {
Itererr = err
return false
}
if _, err = fw.Write([]byte(val)); err != nil {
Itererr = err
return false
}
if _, err = fw.Write([]byte(value.(string))); err != nil {
Itererr = err
return false
}
return true
})
-242
View File
@@ -1,242 +0,0 @@
package dataformat
import (
"testing"
mapsutil "github.com/projectdiscovery/utils/maps"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestMultiPartFormEncode(t *testing.T) {
tests := []struct {
name string
fields map[string]any
wantErr bool
expected map[string]any
}{
{
name: "duplicate fields ([]string) - checkbox scenario",
fields: map[string]any{
"interests": []string{"sports", "music", "reading"},
"colors": []string{"red", "blue"},
},
expected: map[string]any{
"interests": []string{"sports", "music", "reading"},
"colors": []string{"red", "blue"},
},
},
{
name: "single string fields - backward compatibility",
fields: map[string]any{
"username": "john",
"email": "[email protected]",
},
expected: map[string]any{
"username": "john",
"email": "[email protected]",
},
},
{
name: "mixed types",
fields: map[string]any{
"string": "text",
"array": []string{"item1", "item2"},
"number": 42, // tests fmt.Sprint fallback
"float": 3.14, // tests float conversion
"boolean": true, // tests boolean conversion
"zero": 0, // tests zero value
"emptyStr": "", // tests empty string
"negative": -123, // tests negative number
"nil": nil, // tests nil value
"mixedArray": []any{"str", 123, false, nil}, // tests mixed type array
},
expected: map[string]any{
"string": "text",
"array": []string{"item1", "item2"},
"number": "42", // numbers are converted to strings in multipart
"float": "3.14", // floats are converted to strings
"boolean": "true", // booleans are converted to strings
"zero": "0", // zero value converted to string
"emptyStr": "", // empty string remains empty
"negative": "-123", // negative numbers converted to strings
"nil": "", // nil values converted to "" string
"mixedArray": []string{"str", "123", "false", ""}, // mixed array converted to string array
},
},
{
name: "empty array - should not appear in output",
fields: map[string]any{
"emptyArray": []string{},
"normalField": "value",
},
expected: map[string]any{
"normalField": "value",
// emptyArray should not appear in decoded output
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
defer func() {
if r := recover(); r != nil {
t.Errorf("Test panicked: %v", r)
}
}()
form := NewMultiPartForm()
form.boundary = "----WebKitFormBoundary7MA4YWxkTrZu0gW"
kv := mapsutil.NewOrderedMap[string, any]()
for k, v := range tt.fields {
kv.Set(k, v)
}
encoded, err := form.Encode(KVOrderedMap(&kv))
if tt.wantErr {
require.Error(t, err)
return
}
require.NoError(t, err)
// Decode the encoded multipart data
decoded, err := form.Decode(encoded)
require.NoError(t, err)
// Compare decoded values with expected values
for expectedKey, expectedValue := range tt.expected {
actualValue := decoded.Get(expectedKey)
switch expected := expectedValue.(type) {
case []string:
actual, ok := actualValue.([]string)
require.True(t, ok, "Expected []string for key %s, got %T", expectedKey, actualValue)
assert.ElementsMatch(t, expected, actual, "Values mismatch for key %s", expectedKey)
case []any:
actual, ok := actualValue.([]any)
require.True(t, ok, "Expected []any for key %s, got %T", expectedKey, actualValue)
assert.ElementsMatch(t, expected, actual, "Values mismatch for key %s", expectedKey)
case string:
actual, ok := actualValue.(string)
require.True(t, ok, "Expected string for key %s, got %T", expectedKey, actualValue)
assert.Equal(t, expected, actual, "Values mismatch for key %s", expectedKey)
default:
assert.Equal(t, expected, actualValue, "Values mismatch for key %s", expectedKey)
}
}
// Ensure no unexpected keys are present in decoded output
decoded.Iterate(func(key string, value any) bool {
_, exists := tt.expected[key]
assert.True(t, exists, "Unexpected key %s found in decoded output", key)
return true
})
t.Logf("Encoded output:\n%s", encoded)
})
}
}
func TestMultiPartFormRoundTrip(t *testing.T) {
defer func() {
if r := recover(); r != nil {
t.Errorf("Test panicked: %v", r)
}
}()
form := NewMultiPartForm()
form.boundary = "----WebKitFormBoundary7MA4YWxkTrZu0gW"
original := mapsutil.NewOrderedMap[string, any]()
original.Set("username", "john")
original.Set("interests", []string{"sports", "music", "reading"})
encoded, err := form.Encode(KVOrderedMap(&original))
require.NoError(t, err)
decoded, err := form.Decode(encoded)
require.NoError(t, err)
assert.Equal(t, "john", decoded.Get("username"))
assert.ElementsMatch(t, []string{"sports", "music", "reading"}, decoded.Get("interests"))
t.Logf("Encoded output:\n%s", encoded)
}
func TestMultiPartFormFileUpload(t *testing.T) {
defer func() {
if r := recover(); r != nil {
t.Errorf("Test panicked: %v", r)
}
}()
// Test decoding of a manually crafted multipart form with files
form := NewMultiPartForm()
form.boundary = "----WebKitFormBoundaryFileUploadTest"
// Manually craft a multipart form with file uploads
multipartData := `------WebKitFormBoundaryFileUploadTest
Content-Disposition: form-data; name="name"
John Doe
------WebKitFormBoundaryFileUploadTest
Content-Disposition: form-data; name="email"
[email protected]
------WebKitFormBoundaryFileUploadTest
Content-Disposition: form-data; name="profile_picture"; filename="profile.jpg"
Content-Type: image/jpeg
fake_jpeg_binary_data_here
------WebKitFormBoundaryFileUploadTest
Content-Disposition: form-data; name="documents"; filename="resume.pdf"
Content-Type: application/pdf
fake_pdf_content_1
------WebKitFormBoundaryFileUploadTest
Content-Disposition: form-data; name="documents"; filename="cover_letter.pdf"
Content-Type: application/pdf
fake_pdf_content_2
------WebKitFormBoundaryFileUploadTest
Content-Disposition: form-data; name="skills"
Go
------WebKitFormBoundaryFileUploadTest
Content-Disposition: form-data; name="skills"
JavaScript
------WebKitFormBoundaryFileUploadTest
Content-Disposition: form-data; name="skills"
Python
------WebKitFormBoundaryFileUploadTest--
`
// Test decoding
decoded, err := form.Decode(multipartData)
require.NoError(t, err)
// Verify regular fields
assert.Equal(t, "John Doe", decoded.Get("name"))
assert.Equal(t, "[email protected]", decoded.Get("email"))
assert.Equal(t, []string{"Go", "JavaScript", "Python"}, decoded.Get("skills"))
// Verify file fields
profilePicture := decoded.Get("profile_picture")
require.NotNil(t, profilePicture)
profileArray, ok := profilePicture.([]interface{})
require.True(t, ok, "Expected []interface{} for profile_picture")
require.Len(t, profileArray, 1)
assert.Equal(t, "fake_jpeg_binary_data_here", profileArray[0])
documents := decoded.Get("documents")
require.NotNil(t, documents)
documentsArray, ok := documents.([]interface{})
require.True(t, ok, "Expected []interface{} for documents")
require.Len(t, documentsArray, 2)
assert.Contains(t, documentsArray, "fake_pdf_content_1")
assert.Contains(t, documentsArray, "fake_pdf_content_2")
}
+9 -69
View File
@@ -14,17 +14,16 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/expressions"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/marker"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/json"
"github.com/projectdiscovery/retryablehttp-go"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
mapsutil "github.com/projectdiscovery/utils/maps"
sliceutil "github.com/projectdiscovery/utils/slice"
urlutil "github.com/projectdiscovery/utils/url"
)
var (
ErrRuleNotApplicable = errkit.New("rule not applicable")
ErrRuleNotApplicable = errorutil.NewWithFmt("rule not applicable : %v")
)
// IsErrRuleNotApplicable checks if an error is due to rule not applicable
@@ -89,10 +88,10 @@ type GeneratedRequest struct {
// goroutines.
func (rule *Rule) Execute(input *ExecuteRuleInput) (err error) {
if !rule.isInputURLValid(input.Input) {
return errkit.Newf("rule not applicable: invalid input url: %v", input.Input.MetaInput.Input)
return ErrRuleNotApplicable.Msgf("invalid input url: %v", input.Input.MetaInput.Input)
}
if input.BaseRequest == nil && input.Input.MetaInput.ReqResp == nil {
return errkit.Newf("rule not applicable: both base request and reqresp are nil for %v", input.Input.MetaInput.Input)
return ErrRuleNotApplicable.Msgf("both base request and reqresp are nil for %v", input.Input.MetaInput.Input)
}
var finalComponentList []component.Component
@@ -144,7 +143,7 @@ func (rule *Rule) Execute(input *ExecuteRuleInput) (err error) {
}
if len(finalComponentList) == 0 {
return errkit.Newf("rule not applicable: no component matched on this rule")
return ErrRuleNotApplicable.Msgf("no component matched on this rule")
}
baseValues := input.Values
@@ -190,33 +189,6 @@ mainLoop:
return nil
}
// evaluateVars evaluates variables in a string using available executor options
func (rule *Rule) evaluateVars(input string) (string, error) {
if rule.options == nil {
return input, nil
}
data := generators.MergeMaps(
rule.options.Variables.GetAll(),
rule.options.Constants,
rule.options.Options.Vars.AsMap(),
)
exprs := expressions.FindExpressions(input, marker.ParenthesisOpen, marker.ParenthesisClose, data)
err := expressions.ContainsUnresolvedVariables(exprs...)
if err != nil {
return input, err
}
eval, err := expressions.Evaluate(input, data)
if err != nil {
return input, err
}
return eval, nil
}
// evaluateVarsWithInteractsh evaluates the variables with Interactsh URLs and updates them accordingly.
func (rule *Rule) evaluateVarsWithInteractsh(data map[string]interface{}, interactshUrls []string) (map[string]interface{}, []string) {
// Check if Interactsh options are configured
@@ -369,47 +341,23 @@ func (rule *Rule) Compile(generator *generators.PayloadGenerator, options *proto
if len(rule.Keys) > 0 {
rule.keysMap = make(map[string]struct{})
}
// eval vars in "keys"
for _, key := range rule.Keys {
evaluatedKey, err := rule.evaluateVars(key)
if err != nil {
return errors.Wrap(err, "could not evaluate key")
}
rule.keysMap[strings.ToLower(evaluatedKey)] = struct{}{}
rule.keysMap[strings.ToLower(key)] = struct{}{}
}
// eval vars in "values"
for _, value := range rule.ValuesRegex {
evaluatedValue, err := rule.evaluateVars(value)
if err != nil {
return errors.Wrap(err, "could not evaluate value regex")
}
compiled, err := regexp.Compile(evaluatedValue)
compiled, err := regexp.Compile(value)
if err != nil {
return errors.Wrap(err, "could not compile value regex")
}
rule.valuesRegex = append(rule.valuesRegex, compiled)
}
// eval vars in "keys-regex"
for _, value := range rule.KeysRegex {
evaluatedValue, err := rule.evaluateVars(value)
if err != nil {
return errors.Wrap(err, "could not evaluate key regex")
}
compiled, err := regexp.Compile(evaluatedValue)
compiled, err := regexp.Compile(value)
if err != nil {
return errors.Wrap(err, "could not compile key regex")
}
rule.keysRegex = append(rule.keysRegex, compiled)
}
if rule.ruleType != replaceRegexRuleType {
if rule.ReplaceRegex != "" {
return errors.Errorf("replace-regex is only applicable for replace and replace-regex rule types")
@@ -418,19 +366,11 @@ func (rule *Rule) Compile(generator *generators.PayloadGenerator, options *proto
if rule.ReplaceRegex == "" {
return errors.Errorf("replace-regex is required for replace-regex rule type")
}
evalReplaceRegex, err := rule.evaluateVars(rule.ReplaceRegex)
if err != nil {
return errors.Wrap(err, "could not evaluate replace regex")
}
compiled, err := regexp.Compile(evalReplaceRegex)
compiled, err := regexp.Compile(rule.ReplaceRegex)
if err != nil {
return errors.Wrap(err, "could not compile replace regex")
}
rule.replaceRegex = compiled
}
return nil
}
-221
View File
@@ -3,11 +3,6 @@ package fuzz
import (
"testing"
"github.com/projectdiscovery/goflags"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/variables"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/nuclei/v3/pkg/utils"
"github.com/stretchr/testify/require"
)
@@ -42,219 +37,3 @@ func TestRuleMatchKeyOrValue(t *testing.T) {
require.False(t, result, "could not get correct result")
})
}
func TestEvaluateVariables(t *testing.T) {
t.Run("keys", func(t *testing.T) {
rule := &Rule{
Keys: []string{"{{foo_var}}"},
Part: "query",
}
// mock
templateVars := variables.Variable{
InsertionOrderedStringMap: *utils.NewEmptyInsertionOrderedStringMap(1),
}
templateVars.Set("foo_var", "foo_var_value")
constants := map[string]interface{}{
"const_key": "const_value",
}
options := &types.Options{}
// runtime vars (to simulate CLI)
runtimeVars := goflags.RuntimeMap{}
_ = runtimeVars.Set("runtime_key=runtime_value")
options.Vars = runtimeVars
executorOpts := &protocols.ExecutorOptions{
Variables: templateVars,
Constants: constants,
Options: options,
}
err := rule.Compile(nil, executorOpts)
require.NoError(t, err, "could not compile rule")
result := rule.matchKeyOrValue("foo_var_value", "test_value")
require.True(t, result, "should match evaluated variable key")
result = rule.matchKeyOrValue("{{foo_var}}", "test_value")
require.False(t, result, "should not match unevaluated variable key")
})
t.Run("keys-regex", func(t *testing.T) {
rule := &Rule{
KeysRegex: []string{"^{{foo_var}}"},
Part: "query",
}
templateVars := variables.Variable{
InsertionOrderedStringMap: *utils.NewEmptyInsertionOrderedStringMap(1),
}
templateVars.Set("foo_var", "foo_var_value")
executorOpts := &protocols.ExecutorOptions{
Variables: templateVars,
Constants: map[string]interface{}{},
Options: &types.Options{},
}
err := rule.Compile(nil, executorOpts)
require.NoError(t, err, "could not compile rule")
result := rule.matchKeyOrValue("foo_var_value", "test_value")
require.True(t, result, "should match evaluated variable in regex")
result = rule.matchKeyOrValue("other_key", "test_value")
require.False(t, result, "should not match non-matching key")
})
t.Run("values-regex", func(t *testing.T) {
rule := &Rule{
ValuesRegex: []string{"{{foo_var}}"},
Part: "query",
}
templateVars := variables.Variable{
InsertionOrderedStringMap: *utils.NewEmptyInsertionOrderedStringMap(1),
}
templateVars.Set("foo_var", "test_pattern")
executorOpts := &protocols.ExecutorOptions{
Variables: templateVars,
Constants: map[string]interface{}{},
Options: &types.Options{},
}
err := rule.Compile(nil, executorOpts)
require.NoError(t, err, "could not compile rule")
result := rule.matchKeyOrValue("test_key", "test_pattern")
require.True(t, result, "should match evaluated variable in values regex")
result = rule.matchKeyOrValue("test_key", "other_value")
require.False(t, result, "should not match non-matching value")
})
// complex vars w/ consts and runtime vars
t.Run("complex-variables", func(t *testing.T) {
rule := &Rule{
Keys: []string{"{{template_var}}", "{{const_key}}", "{{runtime_key}}"},
Part: "query",
}
templateVars := variables.Variable{
InsertionOrderedStringMap: *utils.NewEmptyInsertionOrderedStringMap(1),
}
templateVars.Set("template_var", "template_value")
constants := map[string]interface{}{
"const_key": "const_value",
}
options := &types.Options{}
runtimeVars := goflags.RuntimeMap{}
_ = runtimeVars.Set("runtime_key=runtime_value")
options.Vars = runtimeVars
executorOpts := &protocols.ExecutorOptions{
Variables: templateVars,
Constants: constants,
Options: options,
}
err := rule.Compile(nil, executorOpts)
require.NoError(t, err, "could not compile rule")
result := rule.matchKeyOrValue("template_value", "test")
require.True(t, result, "should match template variable")
result = rule.matchKeyOrValue("const_value", "test")
require.True(t, result, "should match constant")
result = rule.matchKeyOrValue("runtime_value", "test")
require.True(t, result, "should match runtime variable")
result = rule.matchKeyOrValue("{{template_var}}", "test")
require.False(t, result, "should not match unevaluated template variable")
})
t.Run("invalid-variables", func(t *testing.T) {
rule := &Rule{
Keys: []string{"{{nonexistent_var}}"},
Part: "query",
}
executorOpts := &protocols.ExecutorOptions{
Variables: variables.Variable{
InsertionOrderedStringMap: *utils.NewEmptyInsertionOrderedStringMap(0),
},
Constants: map[string]interface{}{},
Options: &types.Options{},
}
err := rule.Compile(nil, executorOpts)
if err != nil {
require.Contains(t, err.Error(), "unresolved", "error should mention unresolved variables")
} else {
result := rule.matchKeyOrValue("some_key", "some_value")
require.False(t, result, "should not match when variables are unresolved")
}
})
t.Run("evaluateVars-function", func(t *testing.T) {
rule := &Rule{}
templateVars := variables.Variable{
InsertionOrderedStringMap: *utils.NewEmptyInsertionOrderedStringMap(1),
}
templateVars.Set("test_var", "test_value")
constants := map[string]interface{}{
"const_var": "const_value",
}
options := &types.Options{}
runtimeVars := goflags.RuntimeMap{}
_ = runtimeVars.Set("runtime_var=runtime_value")
options.Vars = runtimeVars
executorOpts := &protocols.ExecutorOptions{
Variables: templateVars,
Constants: constants,
Options: options,
}
rule.options = executorOpts
// Test simple var substitution
result, err := rule.evaluateVars("{{test_var}}")
require.NoError(t, err, "should evaluate template variable")
require.Equal(t, "test_value", result, "should return evaluated value")
// Test constant substitution
result, err = rule.evaluateVars("{{const_var}}")
require.NoError(t, err, "should evaluate constant")
require.Equal(t, "const_value", result, "should return constant value")
// Test runtime var substitution
result, err = rule.evaluateVars("{{runtime_var}}")
require.NoError(t, err, "should evaluate runtime variable")
require.Equal(t, "runtime_value", result, "should return runtime value")
// Test mixed content
result, err = rule.evaluateVars("prefix-{{test_var}}-suffix")
require.NoError(t, err, "should evaluate mixed content")
require.Equal(t, "prefix-test_value-suffix", result, "should return mixed evaluated content")
// Test unresolved var - should either fail during evaluation or return original string
result2, err := rule.evaluateVars("{{nonexistent}}")
if err != nil {
require.Contains(t, err.Error(), "unresolved", "should fail for unresolved variable")
} else {
// If no error, it should return the original unresolved variable
require.Equal(t, "{{nonexistent}}", result2, "should return original string for unresolved variable")
}
})
}
+9 -9
View File
@@ -20,7 +20,7 @@ import (
httpTypes "github.com/projectdiscovery/nuclei/v3/pkg/input/types"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/json"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
"github.com/projectdiscovery/utils/generic"
mapsutil "github.com/projectdiscovery/utils/maps"
"github.com/valyala/fasttemplate"
@@ -395,7 +395,7 @@ func generateRequestsFromOp(opts *generateReqOptions) error {
func GetGlobalParamsForSecurityRequirement(schema *openapi3.T, requirement *openapi3.SecurityRequirements) ([]*openapi3.ParameterRef, error) {
globalParams := openapi3.NewParameters()
if len(schema.Components.SecuritySchemes) == 0 {
return nil, errkit.Newf("security requirements (%+v) without any security schemes found in openapi file", schema.Security)
return nil, errorutil.NewWithTag("openapi", "security requirements (%+v) without any security schemes found in openapi file", schema.Security)
}
found := false
// this api is protected for each security scheme pull its corresponding scheme
@@ -415,11 +415,11 @@ schemaLabel:
}
if !found && len(security) > 1 {
// if this is case then both security schemes are required
return nil, errkit.Newf("security requirement (%+v) not found in openapi file", security)
return nil, errorutil.NewWithTag("openapi", "security requirement (%+v) not found in openapi file", security)
}
}
if !found {
return nil, errkit.Newf("security requirement (%+v) not found in openapi file", requirement)
return nil, errorutil.NewWithTag("openapi", "security requirement (%+v) not found in openapi file", requirement)
}
return globalParams, nil
@@ -428,12 +428,12 @@ schemaLabel:
// GenerateParameterFromSecurityScheme generates an example from a schema object
func GenerateParameterFromSecurityScheme(scheme *openapi3.SecuritySchemeRef) (*openapi3.Parameter, error) {
if !generic.EqualsAny(scheme.Value.Type, "http", "apiKey") {
return nil, errkit.Newf("unsupported security scheme type (%s) found in openapi file", scheme.Value.Type)
return nil, errorutil.NewWithTag("openapi", "unsupported security scheme type (%s) found in openapi file", scheme.Value.Type)
}
if scheme.Value.Type == "http" {
// check scheme
if !generic.EqualsAny(scheme.Value.Scheme, "basic", "bearer") {
return nil, errkit.Newf("unsupported security scheme (%s) found in openapi file", scheme.Value.Scheme)
return nil, errorutil.NewWithTag("openapi", "unsupported security scheme (%s) found in openapi file", scheme.Value.Scheme)
}
// HTTP authentication schemes basic or bearer use the Authorization header
headerName := scheme.Value.Name
@@ -458,10 +458,10 @@ func GenerateParameterFromSecurityScheme(scheme *openapi3.SecuritySchemeRef) (*o
if scheme.Value.Type == "apiKey" {
// validate name and in
if scheme.Value.Name == "" {
return nil, errkit.Newf("security scheme (%s) name is empty", scheme.Value.Type)
return nil, errorutil.NewWithTag("openapi", "security scheme (%s) name is empty", scheme.Value.Type)
}
if !generic.EqualsAny(scheme.Value.In, "query", "header", "cookie") {
return nil, errkit.Newf("unsupported security scheme (%s) in (%s) found in openapi file", scheme.Value.Type, scheme.Value.In)
return nil, errorutil.NewWithTag("openapi", "unsupported security scheme (%s) in (%s) found in openapi file", scheme.Value.Type, scheme.Value.In)
}
// create parameters using the scheme
switch scheme.Value.In {
@@ -482,5 +482,5 @@ func GenerateParameterFromSecurityScheme(scheme *openapi3.SecuritySchemeRef) (*o
return c, nil
}
}
return nil, errkit.Newf("unsupported security scheme type (%s) found in openapi file", scheme.Value.Type)
return nil, errorutil.NewWithTag("openapi", "unsupported security scheme type (%s) found in openapi file", scheme.Value.Type)
}
+2 -2
View File
@@ -13,12 +13,12 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
configTypes "github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
stringsutil "github.com/projectdiscovery/utils/strings"
)
var (
ErrNotImplemented = errkit.New("provider does not implement method")
ErrNotImplemented = errorutil.NewWithFmt("provider %s does not implement %s")
ErrInactiveInput = fmt.Errorf("input is inactive")
)
+18 -42
View File
@@ -17,7 +17,7 @@ import (
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/external/customtemplates"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
fileutil "github.com/projectdiscovery/utils/file"
stringsutil "github.com/projectdiscovery/utils/strings"
updateutils "github.com/projectdiscovery/utils/update"
@@ -80,7 +80,7 @@ func (t *TemplateManager) FreshInstallIfNotExists() error {
}
gologger.Info().Msgf("nuclei-templates are not installed, installing...")
if err := t.installTemplatesAt(config.DefaultConfig.TemplatesDirectory); err != nil {
return errkit.Wrapf(err, "failed to install templates at %s", config.DefaultConfig.TemplatesDirectory)
return errorutil.NewWithErr(err).Msgf("failed to install templates at %s", config.DefaultConfig.TemplatesDirectory)
}
if t.CustomTemplates != nil {
t.CustomTemplates.Download(context.TODO())
@@ -94,24 +94,7 @@ func (t *TemplateManager) UpdateIfOutdated() error {
if !fileutil.FolderExists(config.DefaultConfig.TemplatesDirectory) {
return t.FreshInstallIfNotExists()
}
needsUpdate := config.DefaultConfig.NeedsTemplateUpdate()
// NOTE(dwisiswant0): if PDTM API data is not available
// (LatestNucleiTemplatesVersion is empty) but we have a current template
// version, so we MUST verify against GitHub directly.
if !needsUpdate && config.DefaultConfig.LatestNucleiTemplatesVersion == "" && config.DefaultConfig.TemplateVersion != "" {
ghrd, err := updateutils.NewghReleaseDownloader(config.OfficialNucleiTemplatesRepoName)
if err == nil {
latestVersion := ghrd.Latest.GetTagName()
if config.IsOutdatedVersion(config.DefaultConfig.TemplateVersion, latestVersion) {
needsUpdate = true
gologger.Debug().Msgf("PDTM API unavailable, verified update needed via GitHub API: %s -> %s", config.DefaultConfig.TemplateVersion, latestVersion)
}
}
}
if needsUpdate {
if config.DefaultConfig.NeedsTemplateUpdate() {
return t.updateTemplatesAt(config.DefaultConfig.TemplatesDirectory)
}
return nil
@@ -121,7 +104,7 @@ func (t *TemplateManager) UpdateIfOutdated() error {
func (t *TemplateManager) installTemplatesAt(dir string) error {
if !fileutil.FolderExists(dir) {
if err := fileutil.CreateFolder(dir); err != nil {
return errkit.Wrapf(err, "failed to create directory at %s", dir)
return errorutil.NewWithErr(err).Msgf("failed to create directory at %s", dir)
}
}
if t.DisablePublicTemplates {
@@ -130,12 +113,12 @@ func (t *TemplateManager) installTemplatesAt(dir string) error {
}
ghrd, err := updateutils.NewghReleaseDownloader(config.OfficialNucleiTemplatesRepoName)
if err != nil {
return errkit.Wrapf(err, "failed to install templates at %s", dir)
return errorutil.NewWithErr(err).Msgf("failed to install templates at %s", dir)
}
// write templates to disk
if err := t.writeTemplatesToDisk(ghrd, dir); err != nil {
return errkit.Wrapf(err, "failed to write templates to disk at %s", dir)
return errorutil.NewWithErr(err).Msgf("failed to write templates to disk at %s", dir)
}
gologger.Info().Msgf("Successfully installed nuclei-templates at %s", dir)
return nil
@@ -156,17 +139,10 @@ func (t *TemplateManager) updateTemplatesAt(dir string) error {
ghrd, err := updateutils.NewghReleaseDownloader(config.OfficialNucleiTemplatesRepoName)
if err != nil {
return errkit.Wrapf(err, "failed to install templates at %s", dir)
return errorutil.NewWithErr(err).Msgf("failed to install templates at %s", dir)
}
latestVersion := ghrd.Latest.GetTagName()
currentVersion := config.DefaultConfig.TemplateVersion
if config.IsOutdatedVersion(currentVersion, latestVersion) {
gologger.Info().Msgf("Your current nuclei-templates %s are outdated. Latest is %s\n", currentVersion, latestVersion)
} else {
gologger.Debug().Msgf("Updating nuclei-templates from %s to %s (forced update)\n", currentVersion, latestVersion)
}
gologger.Info().Msgf("Your current nuclei-templates %s are outdated. Latest is %s\n", config.DefaultConfig.TemplateVersion, ghrd.Latest.GetTagName())
// write templates to disk
if err := t.writeTemplatesToDisk(ghrd, dir); err != nil {
@@ -177,7 +153,7 @@ func (t *TemplateManager) updateTemplatesAt(dir string) error {
newchecksums, err := t.getChecksumFromDir(dir)
if err != nil {
// unlikely this case will happen
return errkit.Wrapf(err, "failed to get checksums from %s after update", dir)
return errorutil.NewWithErr(err).Msgf("failed to get checksums from %s after update", dir)
}
// summarize all changes
@@ -299,7 +275,7 @@ func (t *TemplateManager) writeTemplatesToDisk(ghrd *updateutils.GHReleaseDownlo
bin, err := io.ReadAll(r)
if err != nil {
// if error occurs, iteration also stops
return errkit.Wrapf(err, "failed to read file %s", uri)
return errorutil.NewWithErr(err).Msgf("failed to read file %s", uri)
}
// TODO: It might be better to just download index file from nuclei templates repo
// instead of creating it from scratch
@@ -310,7 +286,7 @@ func (t *TemplateManager) writeTemplatesToDisk(ghrd *updateutils.GHReleaseDownlo
if oldPath != writePath {
// write new template at a new path and delete old template
if err := os.WriteFile(writePath, bin, f.Mode()); err != nil {
return errkit.Wrapf(err, "failed to write file %s", uri)
return errorutil.NewWithErr(err).Msgf("failed to write file %s", uri)
}
// after successful write, remove old template
if err := os.Remove(oldPath); err != nil {
@@ -325,20 +301,20 @@ func (t *TemplateManager) writeTemplatesToDisk(ghrd *updateutils.GHReleaseDownlo
}
err = ghrd.DownloadSourceWithCallback(!HideProgressBar, callbackFunc)
if err != nil {
return errkit.Wrap(err, "failed to download templates")
return errorutil.NewWithErr(err).Msgf("failed to download templates")
}
if err := config.DefaultConfig.WriteTemplatesConfig(); err != nil {
return errkit.Wrap(err, "failed to write templates config")
return errorutil.NewWithErr(err).Msgf("failed to write templates config")
}
// update ignore hash after writing new templates
if err := config.DefaultConfig.UpdateNucleiIgnoreHash(); err != nil {
return errkit.Wrap(err, "failed to update nuclei ignore hash")
return errorutil.NewWithErr(err).Msgf("failed to update nuclei ignore hash")
}
// update templates version in config file
if err := config.DefaultConfig.SetTemplatesVersion(ghrd.Latest.GetTagName()); err != nil {
return errkit.Wrap(err, "failed to update templates version")
return errorutil.NewWithErr(err).Msgf("failed to update templates version")
}
PurgeEmptyDirectories(dir)
@@ -348,11 +324,11 @@ func (t *TemplateManager) writeTemplatesToDisk(ghrd *updateutils.GHReleaseDownlo
index, err := config.GetNucleiTemplatesIndex()
if err != nil {
return errkit.Wrap(err, "failed to get nuclei templates index")
return errorutil.NewWithErr(err).Msgf("failed to get nuclei templates index")
}
if err = config.DefaultConfig.WriteTemplatesIndex(index); err != nil {
return errkit.Wrap(err, "failed to write nuclei templates index")
return errorutil.NewWithErr(err).Msgf("failed to write nuclei templates index")
}
if !HideReleaseNotes {
@@ -448,5 +424,5 @@ func (t *TemplateManager) calculateChecksumMap(dir string) (map[string]string, e
}
return nil
})
return checksumMap, errkit.Wrap(err, "failed to calculate checksums of templates")
return checksumMap, errorutil.WrapfWithNil(err, "failed to calculate checksums of templates")
}
-39
View File
@@ -59,42 +59,3 @@ func TestTemplateInstallation(t *testing.T) {
require.FileExists(t, config.DefaultConfig.GetIgnoreFilePath())
t.Logf("Installed %d templates", counter)
}
func TestIsOutdatedVersion(t *testing.T) {
testCases := []struct {
current string
latest string
expected bool
desc string
}{
// Test the empty latest version case (main bug fix)
{"v10.2.7", "", false, "Empty latest version should not trigger update"},
// Test same versions
{"v10.2.7", "v10.2.7", false, "Same versions should not trigger update"},
// Test outdated version
{"v10.2.6", "v10.2.7", true, "Older version should trigger update"},
// Test newer current version (edge case)
{"v10.2.8", "v10.2.7", false, "Newer current version should not trigger update"},
// Test dev versions
{"v10.2.7-dev", "v10.2.7", false, "Dev version matching release should not trigger update"},
{"v10.2.6-dev", "v10.2.7", true, "Outdated dev version should trigger update"},
// Test invalid semver fallback
{"invalid-version", "v10.2.7", true, "Invalid current version should trigger update (fallback)"},
{"v10.2.7", "invalid-version", true, "Invalid latest version should trigger update (fallback)"},
{"same-invalid", "same-invalid", false, "Same invalid versions should not trigger update (fallback)"},
}
for _, tc := range testCases {
t.Run(tc.desc, func(t *testing.T) {
result := config.IsOutdatedVersion(tc.current, tc.latest)
require.Equal(t, tc.expected, result,
"IsOutdatedVersion(%q, %q) = %t, expected %t",
tc.current, tc.latest, result, tc.expected)
})
}
}
+2 -2
View File
@@ -14,7 +14,7 @@ import (
"github.com/Masterminds/semver/v3"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
)
// GetNewTemplatesInVersions returns templates path of all newly added templates
@@ -52,7 +52,7 @@ func getNewAdditionsFileFromGitHub(version string) ([]string, error) {
return nil, err
}
if resp.StatusCode != http.StatusOK {
return nil, errkit.New("version not found")
return nil, errorutil.New("version not found")
}
data, err := io.ReadAll(resp.Body)
if err != nil {
+4 -4
View File
@@ -6,7 +6,7 @@ import (
"regexp"
"strings"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
)
// scrape.go scrapes all information of exported type from different package
@@ -21,17 +21,17 @@ func (p *EntityParser) scrapeAndCreate(typeName string) error {
// get package
pkg, ok := p.imports[pkgName]
if !ok {
return errkit.Newf("package %v for type %v not found", pkgName, typeName)
return errorutil.New("package %v for type %v not found", pkgName, typeName)
}
// get type
obj := pkg.Types.Scope().Lookup(baseTypeName)
if obj == nil {
return errkit.Newf("type %v not found in package %+v", typeName, pkg)
return errorutil.New("type %v not found in package %+v", typeName, pkg)
}
// Ensure the object is a type name
typeNameObj, ok := obj.(*types.TypeName)
if !ok {
return errkit.Newf("%v is not a type name", typeName)
return errorutil.New("%v is not a type name", typeName)
}
// Ensure the type is a named struct type
namedStruct, ok := typeNameObj.Type().Underlying().(*types.Struct)
+2 -1
View File
@@ -17,6 +17,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/utils/vardump"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
stringsutil "github.com/projectdiscovery/utils/strings"
)
@@ -256,7 +257,7 @@ func RegisterNativeScripts(runtime *goja.Runtime) error {
// import default modules
_, err = runtime.RunString(defaultImports)
if err != nil {
return errkit.Wrapf(err, "could not import default modules %v", defaultImports)
return errorutil.NewWithErr(err).Msgf("could not import default modules %v", defaultImports)
}
return nil
+1 -1
View File
@@ -59,7 +59,7 @@ func wrapModuleFunc(runtime *goja.Runtime, fn interface{}) interface{} {
}
// Only wrap if first parameter is context.Context
if fnType.NumIn() == 0 || fnType.In(0) != reflect.TypeFor[context.Context]() {
if fnType.NumIn() == 0 || fnType.In(0) != reflect.TypeOf((*context.Context)(nil)).Elem() {
return fn // Return original function unchanged if it doesn't have context.Context as first arg
}
+5 -5
View File
@@ -5,12 +5,12 @@ import (
"reflect"
"github.com/Mzack9999/goja"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
)
var (
ErrInvalidFuncOpts = errkit.New("invalid function options")
ErrNilRuntime = errkit.New("runtime is nil")
ErrInvalidFuncOpts = errorutil.NewWithFmt("invalid function options: %v")
ErrNilRuntime = errorutil.New("runtime is nil")
)
type FuncOpts struct {
@@ -34,7 +34,7 @@ func wrapWithContext(runtime *goja.Runtime, fn interface{}) interface{} {
}
// Only wrap if first parameter is context.Context
if fnType.NumIn() == 0 || fnType.In(0) != reflect.TypeFor[context.Context]() {
if fnType.NumIn() == 0 || fnType.In(0) != reflect.TypeOf((*context.Context)(nil)).Elem() {
return fn // Return original function unchanged if it doesn't have context.Context as first arg
}
@@ -83,7 +83,7 @@ func RegisterFuncWithSignature(runtime *goja.Runtime, opts FuncOpts) error {
return ErrNilRuntime
}
if !opts.valid() {
return errkit.Newf("invalid function options: name: %s, signatures: %v, description: %s", opts.Name, opts.Signatures, opts.Description)
return ErrInvalidFuncOpts.Msgf("name: %s, signatures: %v, description: %s", opts.Name, opts.Signatures, opts.Description)
}
// Wrap the function with context injection
+3 -3
View File
@@ -10,7 +10,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
"github.com/projectdiscovery/utils/reader"
)
@@ -201,7 +201,7 @@ func (c *NetConn) RecvFull(N int) ([]byte, error) {
}
bin, err := reader.ConnReadNWithTimeout(c.conn, int64(N), c.timeout)
if err != nil {
return []byte{}, errkit.Wrapf(err, "failed to read %d bytes", N)
return []byte{}, errorutil.NewWithErr(err).Msgf("failed to read %d bytes", N)
}
return bin, nil
}
@@ -226,7 +226,7 @@ func (c *NetConn) Recv(N int) ([]byte, error) {
b := make([]byte, N)
n, err := c.conn.Read(b)
if err != nil {
return []byte{}, errkit.Wrapf(err, "failed to read %d bytes", N)
return []byte{}, errorutil.NewWithErr(err).Msgf("failed to read %d bytes", N)
}
return b[:n], nil
}
+4 -4
View File
@@ -7,7 +7,7 @@ import (
"time"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
"github.com/zmap/zgrab2/lib/ssh"
)
@@ -129,7 +129,7 @@ func (c *SSHClient) ConnectSSHInfoMode(ctx context.Context, host string, port in
// ```
func (c *SSHClient) Run(cmd string) (string, error) {
if c.connection == nil {
return "", errkit.New("no connection")
return "", errorutil.New("no connection")
}
session, err := c.connection.NewSession()
if err != nil {
@@ -177,10 +177,10 @@ type connectOptions struct {
func (c *connectOptions) validate() error {
if c.Host == "" {
return errkit.New("host is required")
return errorutil.New("host is required")
}
if c.Port <= 0 {
return errkit.New("port is required")
return errorutil.New("port is required")
}
if !protocolstate.IsHostAllowed(c.ExecutionId, c.Host) {
// host is not valid according to network policy
+2 -3
View File
@@ -61,12 +61,11 @@ func init() {
return nil, fmt.Errorf("invalid dns type")
}
options := &types.Options{}
err := dnsclientpool.Init(options)
err := dnsclientpool.Init(&types.Options{})
if err != nil {
return nil, err
}
dnsClient, err := dnsclientpool.Get(options, &dnsclientpool.Configuration{})
dnsClient, err := dnsclientpool.Get(nil, &dnsclientpool.Configuration{})
if err != nil {
return nil, err
}
+2 -13
View File
@@ -5,15 +5,10 @@ import (
"testing"
"github.com/Knetic/govaluate"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/dns/dnsclientpool"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/stretchr/testify/require"
)
func TestDslExpressions(t *testing.T) {
// Use Google DNS for more reliable testing
googleDNS := []string{"8.8.8.8:53", "8.8.4.4:53"}
dslExpressions := map[string]interface{}{
`resolve("scanme.sh")`: "128.199.158.128",
`resolve("scanme.sh","a")`: "128.199.158.128",
@@ -22,7 +17,7 @@ func TestDslExpressions(t *testing.T) {
`resolve("scanme.sh","soa")`: "ns69.domaincontrol.com",
}
testDslExpressionScenariosWithDNS(t, dslExpressions, googleDNS)
testDslExpressionScenarios(t, dslExpressions)
}
func evaluateExpression(t *testing.T, dslExpression string) interface{} {
@@ -39,13 +34,7 @@ func evaluateExpression(t *testing.T, dslExpression string) interface{} {
return actualResult
}
func testDslExpressionScenariosWithDNS(t *testing.T, dslExpressions map[string]interface{}, resolvers []string) {
// Initialize DNS client pool with custom resolvers for testing
err := dnsclientpool.Init(&types.Options{
InternalResolversList: resolvers,
})
require.NoError(t, err, "Failed to initialize DNS client pool with custom resolvers")
func testDslExpressionScenarios(t *testing.T, dslExpressions map[string]interface{}) {
for dslExpression, expectedResult := range dslExpressions {
t.Run(dslExpression, func(t *testing.T) {
actualResult := evaluateExpression(t, dslExpression)
+2 -1
View File
@@ -8,6 +8,7 @@ import (
"github.com/antchfx/htmlquery"
"github.com/antchfx/xmlquery"
dslRepo "github.com/projectdiscovery/dsl"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/operators/common/dsl"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/expressions"
@@ -315,7 +316,7 @@ func (m *Matcher) ignoreErr(err error) bool {
if showDSLErr {
return false
}
if stringsutil.ContainsAny(err.Error(), "No parameter", "error parsing argument value") {
if stringsutil.ContainsAny(err.Error(), "No parameter", dslRepo.ErrParsingArg.Error()) {
return true
}
return false
-4
View File
@@ -295,10 +295,6 @@ func (w *StandardWriter) ResultCount() int {
// Write writes the event to file and/or screen.
func (w *StandardWriter) Write(event *ResultEvent) error {
if event.Error != "" && !w.matcherStatus {
return nil
}
// Enrich the result event with extra metadata on the template-path and url.
if event.TemplatePath != "" {
event.Template, event.TemplateURL = utils.TemplatePathURL(types.ToString(event.TemplatePath), types.ToString(event.TemplateID), event.TemplateVerifier)
+6 -4
View File
@@ -11,6 +11,7 @@ import (
"github.com/Mzack9999/goja"
"github.com/alecthomas/chroma/quick"
"github.com/ditashi/jsbeautifier-go/jsbeautifier"
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/gozero"
@@ -32,6 +33,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/types"
contextutil "github.com/projectdiscovery/utils/context"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
)
const (
@@ -112,7 +114,7 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
if options.Options.Validate {
options.Logger.Error().Msgf("%s <- %s", errMsg, err)
} else {
return errkit.Wrap(err, errMsg)
return errorutil.NewWithErr(err).Msgf(errMsg)
}
} else {
request.gozero = engine
@@ -131,7 +133,7 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
compiled.ExcludeMatchers = options.ExcludeMatchers
compiled.TemplateID = options.TemplateID
if err := compiled.Compile(); err != nil {
return errkit.Wrap(err, "could not compile operators")
return errors.Wrap(err, "could not compile operators")
}
for _, matcher := range compiled.Matchers {
// default matcher part for code protocol is response
@@ -152,7 +154,7 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
if request.PreCondition != "" {
preConditionCompiled, err := compiler.SourceAutoMode(request.PreCondition, false)
if err != nil {
return errkit.Newf("could not compile pre-condition: %s", err)
return errorutil.NewWithTag(request.TemplateID, "could not compile pre-condition: %s", err)
}
request.preConditionCompiled = preConditionCompiled
}
@@ -229,7 +231,7 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicVa
Context: input.Context(),
})
if err != nil {
return errkit.Newf("could not execute pre-condition: %s", err)
return errorutil.NewWithTag(request.TemplateID, "could not execute pre-condition: %s", err)
}
if !result.GetSuccess() || types.ToString(result["error"]) != "" {
gologger.Warning().Msgf("[%s] Precondition for request %s was not satisfied\n", request.TemplateID, request.PreCondition)
+1 -1
View File
@@ -8,7 +8,7 @@ import (
var (
defaultInteractionDuration = 60 * time.Second
interactshURLMarkerRegex = regexp.MustCompile(`(%7[B|b]|\{){2}(interactsh-url(?:_[0-9]+){0,3})(%7[D|d]|\}){2}`)
interactshURLMarkerRegex = regexp.MustCompile(`(%7[B|b]|\{){2}(interactsh-url(?:_[0-9]+){0,3})(%7[D|d]|\}){2}`)
ErrInteractshClientNotInitialized = errors.New("interactsh client not initialized")
)
@@ -22,7 +22,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/responsehighlighter"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/writer"
"github.com/projectdiscovery/retryablehttp-go"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
stringsutil "github.com/projectdiscovery/utils/strings"
)
@@ -88,7 +88,7 @@ func (c *Client) poll() error {
KeepAliveInterval: time.Minute,
})
if err != nil {
return errkit.Wrap(err, "could not create client")
return errorutil.NewWithErr(err).Msgf("could not create client")
}
c.interactsh = interactsh
@@ -109,7 +109,7 @@ func (c *Client) poll() error {
// If we don't have any request for this ID, add it to temporary
// lru cache, so we can correlate when we get an add request.
items, err := c.interactions.Get(interaction.UniqueID)
if errkit.Is(err, gcache.KeyNotFoundError) || items == nil {
if errorutil.IsAny(err, gcache.KeyNotFoundError) || items == nil {
_ = c.interactions.SetWithExpire(interaction.UniqueID, []*server.Interaction{interaction}, defaultInteractionDuration)
} else {
items = append(items, interaction)
@@ -128,7 +128,7 @@ func (c *Client) poll() error {
})
if err != nil {
return errkit.Wrap(err, "could not perform interactsh polling")
return errorutil.NewWithErr(err).Msgf("could not perform interactsh polling")
}
return nil
}
@@ -239,7 +239,7 @@ func (c *Client) URL() (string, error) {
err = c.poll()
})
if err != nil {
return "", errkit.Wrap(ErrInteractshClientNotInitialized, err.Error())
return "", errorutil.NewWithErr(err).Wrap(ErrInteractshClientNotInitialized)
}
if c.interactsh == nil {
+3 -3
View File
@@ -5,7 +5,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
fileutil "github.com/projectdiscovery/utils/file"
mapsutil "github.com/projectdiscovery/utils/maps"
)
@@ -67,12 +67,12 @@ func NormalizePath(options *types.Options, filePath string) (string, error) {
}
cleaned, err := fileutil.ResolveNClean(filePath, config.DefaultConfig.GetTemplateDir())
if err != nil {
return "", errkit.Wrapf(err, "could not resolve and clean path %v", filePath)
return "", errorutil.NewWithErr(err).Msgf("could not resolve and clean path %v", filePath)
}
// only allow files inside nuclei-templates directory
// even current working directory is not allowed
if strings.HasPrefix(cleaned, config.DefaultConfig.GetTemplateDir()) {
return cleaned, nil
}
return "", errkit.Newf("path %v is outside nuclei-template directory and -lfa is not enabled", filePath)
return "", errorutil.New("path %v is outside nuclei-template directory and -lfa is not enabled", filePath)
}
+6 -15
View File
@@ -9,7 +9,7 @@ import (
"github.com/go-rod/rod/lib/proto"
"github.com/projectdiscovery/networkpolicy"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
stringsutil "github.com/projectdiscovery/utils/strings"
urlutil "github.com/projectdiscovery/utils/url"
"go.uber.org/multierr"
@@ -18,19 +18,10 @@ import (
// initialize state of headless protocol
var (
ErrURLDenied = errkit.New("headless: url dropped by rule")
ErrHostDenied = errorTemplate{format: "host %v dropped by network policy"}
ErrURLDenied = errorutil.NewWithFmt("headless: url %v dropped by rule: %v")
ErrHostDenied = errorutil.NewWithFmt("host %v dropped by network policy")
)
// errorTemplate provides a way to create formatted errors like the old errorutil.NewWithFmt
type errorTemplate struct {
format string
}
func (e errorTemplate) Msgf(args ...interface{}) error {
return errkit.Newf(e.format, args...)
}
func GetNetworkPolicy(ctx context.Context) *networkpolicy.NetworkPolicy {
execCtx := GetExecutionContext(ctx)
if execCtx == nil {
@@ -50,15 +41,15 @@ func ValidateNFailRequest(options *types.Options, page *rod.Page, e *proto.Fetch
normalized := strings.ToLower(reqURL) // normalize url to lowercase
normalized = strings.TrimSpace(normalized) // trim leading & trailing whitespaces
if !IsLfaAllowed(options) && stringsutil.HasPrefixI(normalized, "file:") {
return multierr.Combine(FailWithReason(page, e), errkit.Newf("headless: url %v dropped by rule: %v", reqURL, "use of file:// protocol disabled use '-lfa' to enable"))
return multierr.Combine(FailWithReason(page, e), ErrURLDenied.Msgf(reqURL, "use of file:// protocol disabled use '-lfa' to enable"))
}
// validate potential invalid schemes
// javascript protocol is allowed for xss fuzzing
if stringsutil.HasPrefixAnyI(normalized, "ftp:", "externalfile:", "chrome:", "chrome-extension:") {
return multierr.Combine(FailWithReason(page, e), errkit.Newf("headless: url %v dropped by rule: %v", reqURL, "protocol blocked by network policy"))
return multierr.Combine(FailWithReason(page, e), ErrURLDenied.Msgf(reqURL, "protocol blocked by network policy"))
}
if !isValidHost(options, reqURL) {
return multierr.Combine(FailWithReason(page, e), errkit.Newf("headless: url %v dropped by rule: %v", reqURL, "address blocked by network policy"))
return multierr.Combine(FailWithReason(page, e), ErrURLDenied.Msgf(reqURL, "address blocked by network policy"))
}
return nil
}
+7 -41
View File
@@ -20,13 +20,12 @@ import (
// Browser is a browser structure for nuclei headless module
type Browser struct {
customAgent string
defaultHeaders map[string]string
tempDir string
previousPIDs map[int32]struct{} // track already running PIDs
engine *rod.Browser
options *types.Options
launcher *launcher.Launcher
customAgent string
tempDir string
previousPIDs map[int32]struct{} // track already running PIDs
engine *rod.Browser
options *types.Options
launcher *launcher.Launcher
// use getHTTPClient to get the http client
httpClient *http.Client
@@ -96,7 +95,6 @@ func New(options *types.Options) (*Browser, error) {
if browserErr := browser.Connect(); browserErr != nil {
return nil, browserErr
}
defaultHeaders := make(map[string]string)
customAgent := ""
for _, option := range options.CustomHeaders {
parts := strings.SplitN(option, ":", 2)
@@ -105,20 +103,12 @@ func New(options *types.Options) (*Browser, error) {
}
if strings.EqualFold(parts[0], "User-Agent") {
customAgent = parts[1]
} else {
k := strings.TrimSpace(parts[0])
v := strings.TrimSpace(parts[1])
if k == "" || v == "" {
continue
}
defaultHeaders[k] = v
}
}
engine := &Browser{
tempDir: dataStore,
customAgent: customAgent,
defaultHeaders: defaultHeaders,
engine: browser,
options: options,
httpClientOnce: &sync.Once{},
@@ -145,30 +135,6 @@ func (b *Browser) UserAgent() string {
return b.customAgent
}
// applyDefaultHeaders setsheaders passed via cli -H flag
func (b *Browser) applyDefaultHeaders(p *rod.Page) error {
pairs := make([]string, 0, len(b.defaultHeaders)*2+2)
hasAcceptLanguage := false
for k := range b.defaultHeaders {
if strings.EqualFold(k, "Accept-Language") {
hasAcceptLanguage = true
break
}
}
if !hasAcceptLanguage {
pairs = append(pairs, "Accept-Language", "en, en-GB, en-us;")
}
for k, v := range b.defaultHeaders {
pairs = append(pairs, k, v)
}
if len(pairs) == 0 {
return nil
}
_, err := p.SetExtraHeaders(pairs)
return err
}
func (b *Browser) getHTTPClient() (*http.Client, error) {
var err error
b.httpClientOnce.Do(func() {
@@ -181,6 +147,6 @@ func (b *Browser) getHTTPClient() (*http.Client, error) {
func (b *Browser) Close() {
_ = b.engine.Close()
b.launcher.Kill()
_ = os.RemoveAll(b.tempDir)
os.RemoveAll(b.tempDir)
processutil.CloseProcesses(processutil.IsChromeProcess, b.previousPIDs)
}
+6 -6
View File
@@ -18,7 +18,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
httputil "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils/http"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
urlutil "github.com/projectdiscovery/utils/url"
)
@@ -61,10 +61,6 @@ func (i *Instance) Run(ctx *contextargs.Context, actions []*Action, payloads map
}
page = page.Timeout(options.Timeout)
if err = i.browser.applyDefaultHeaders(page); err != nil {
return nil, nil, err
}
if i.browser.customAgent != "" {
if userAgentErr := page.SetUserAgent(&proto.NetworkSetUserAgentOverride{UserAgent: i.browser.customAgent}); userAgentErr != nil {
return nil, nil, userAgentErr
@@ -78,7 +74,7 @@ func (i *Instance) Run(ctx *contextargs.Context, actions []*Action, payloads map
target := ctx.MetaInput.Input
input, err := urlutil.Parse(target)
if err != nil {
return nil, nil, errkit.Wrapf(err, "could not parse URL %s", target)
return nil, nil, errorutil.NewWithErr(err).Msgf("could not parse URL %s", target)
}
hasTrailingSlash := httputil.HasTrailingSlash(target)
@@ -134,6 +130,10 @@ func (i *Instance) Run(ctx *contextargs.Context, actions []*Action, payloads map
return nil, nil, err
}
if _, err := page.SetExtraHeaders([]string{"Accept-Language", "en, en-GB, en-us;"}); err != nil {
return nil, nil, err
}
// inject cookies
// each http request is performed via the native go http client
// we first inject the shared cookies
+14 -13
View File
@@ -23,6 +23,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
contextutil "github.com/projectdiscovery/utils/context"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
fileutil "github.com/projectdiscovery/utils/file"
folderutil "github.com/projectdiscovery/utils/folder"
stringsutil "github.com/projectdiscovery/utils/strings"
@@ -31,8 +32,8 @@ import (
)
var (
errinvalidArguments = errkit.New("invalid arguments provided")
ErrLFAccessDenied = errkit.New("Use -allow-local-file-access flag to enable local file access")
errinvalidArguments = errorutil.New("invalid arguments provided")
ErrLFAccessDenied = errorutil.New("Use -allow-local-file-access flag to enable local file access")
// ErrActionExecDealine is the error returned when alloted time for action execution exceeds
ErrActionExecDealine = errkit.New("headless action execution deadline exceeded").SetKind(errkit.ErrKindDeadline).Build()
)
@@ -59,7 +60,7 @@ func (p *Page) ExecuteActions(input *contextargs.Context, actions []*Action) (ou
}
if r := recover(); r != nil {
err = errkit.Newf("panic on headless action: %v", r)
err = errorutil.New("panic on headless action: %v", r)
}
}()
@@ -72,7 +73,7 @@ func (p *Page) ExecuteActions(input *contextargs.Context, actions []*Action) (ou
for _, waitFunc := range waitFuncs {
if waitFunc != nil {
if err := waitFunc(); err != nil {
return nil, errkit.Wrap(err, "error occurred while executing waitFunc")
return nil, errorutil.NewWithErr(err).Msgf("error occurred while executing waitFunc")
}
}
}
@@ -400,7 +401,7 @@ func (p *Page) NavigateURL(action *Action, out ActionData) error {
parsedURL, err := urlutil.ParseURL(url, true)
if err != nil {
return errkit.Newf("failed to parse url %v while creating http request", url)
return errorutil.NewWithTag("headless", "failed to parse url %v while creating http request", url)
}
// ===== parameter automerge =====
@@ -410,7 +411,7 @@ func (p *Page) NavigateURL(action *Action, out ActionData) error {
parsedURL.Params = finalparams
if err := p.page.Navigate(parsedURL.String()); err != nil {
return errkit.Wrapf(err, "could not navigate to url %s", parsedURL.String())
return errorutil.NewWithErr(err).Msgf("could not navigate to url %s", parsedURL.String())
}
p.updateLastNavigatedURL()
@@ -524,14 +525,14 @@ func (p *Page) Screenshot(act *Action, out ActionData) error {
to, err = fileutil.CleanPath(to)
if err != nil {
return errkit.Newf("could not clean output screenshot path %s", to)
return errorutil.New("could not clean output screenshot path %s", to)
}
// allow if targetPath is child of current working directory
if !protocolstate.IsLfaAllowed(p.options.Options) {
cwd, err := os.Getwd()
if err != nil {
return errkit.Wrap(err, "could not get current working directory")
return errorutil.NewWithErr(err).Msgf("could not get current working directory")
}
if !strings.HasPrefix(to, cwd) {
@@ -550,7 +551,7 @@ func (p *Page) Screenshot(act *Action, out ActionData) error {
// creates new directory if needed based on path `to`
// TODO: replace all permission bits with fileutil constants (https://github.com/projectdiscovery/utils/issues/113)
if err := os.MkdirAll(filepath.Dir(to), 0700); err != nil {
return errkit.Wrap(err, "failed to create directory while writing screenshot")
return errorutil.NewWithErr(err).Msgf("failed to create directory while writing screenshot")
}
}
@@ -562,7 +563,7 @@ func (p *Page) Screenshot(act *Action, out ActionData) error {
if fileutil.FileExists(filePath) {
// return custom error as overwriting files is not supported
return errkit.Newf("failed to write screenshot, file %v already exists", filePath)
return errorutil.NewWithTag("screenshot", "failed to write screenshot, file %v already exists", filePath)
}
err = os.WriteFile(filePath, data, 0540)
if err != nil {
@@ -805,12 +806,12 @@ func (p *Page) WaitEvent(act *Action, out ActionData) (func() error, error) {
gotType := proto.GetType(event)
if gotType == nil {
return nil, errkit.Newf("event %q does not exist", event)
return nil, errorutil.New("event %q does not exist", event)
}
tmp, ok := reflect.New(gotType).Interface().(proto.Event)
if !ok {
return nil, errkit.Newf("event %q is not a page event", event)
return nil, errorutil.New("event %q is not a page event", event)
}
waitEvent = tmp
@@ -947,7 +948,7 @@ func (p *Page) getActionArg(action *Action, arg string) (string, error) {
err = expressions.ContainsUnresolvedVariables(exprs...)
if err != nil {
return "", errkit.Wrapf(err, "argument %q, value: %q", arg, argValue)
return "", errorutil.NewWithErr(err).Msgf("argument %q, value: %q", arg, argValue)
}
argValue, err = expressions.Evaluate(argValue, p.variables)
+16 -46
View File
@@ -27,7 +27,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/types/scanstrategy"
"github.com/projectdiscovery/rawhttp"
"github.com/projectdiscovery/retryablehttp-go"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
readerutil "github.com/projectdiscovery/utils/reader"
stringsutil "github.com/projectdiscovery/utils/strings"
urlutil "github.com/projectdiscovery/utils/url"
@@ -37,40 +37,10 @@ const (
ReqURLPatternKey = "req_url_pattern"
)
// ErrEvalExpression
type errorTemplate struct {
format string
}
func (e errorTemplate) Wrap(err error) wrapperError {
return wrapperError{template: e, err: err}
}
func (e errorTemplate) Msgf(args ...interface{}) error {
return errkit.Newf(e.format, args...)
}
type wrapperError struct {
template errorTemplate
err error
}
func (w wrapperError) WithTag(tag string) error {
return errkit.Wrap(w.err, w.template.format)
}
func (w wrapperError) Msgf(format string, args ...interface{}) error {
return errkit.Wrapf(w.err, format, args...)
}
func (w wrapperError) Error() string {
return errkit.Wrap(w.err, w.template.format).Error()
}
// ErrEvalExpression
var (
ErrEvalExpression = errorTemplate{"could not evaluate helper expressions"}
ErrUnresolvedVars = errorTemplate{"unresolved variables `%v` found in request"}
ErrEvalExpression = errorutil.NewWithTag("expr", "could not evaluate helper expressions")
ErrUnresolvedVars = errorutil.NewWithFmt("unresolved variables `%v` found in request")
)
// generatedRequest is a single generated request wrapped for a template request
@@ -223,7 +193,7 @@ func (r *requestGenerator) Make(ctx context.Context, input *contextargs.Context,
for payloadName, payloadValue := range payloads {
payloads[payloadName], err = expressions.Evaluate(types.ToString(payloadValue), allVars)
if err != nil {
return nil, errkit.Wrap(err, "could not evaluate helper expressions")
return nil, ErrEvalExpression.Wrap(err).WithTag("http")
}
}
// finalVars contains allVars and any generator/fuzzing specific payloads
@@ -240,7 +210,7 @@ func (r *requestGenerator) Make(ctx context.Context, input *contextargs.Context,
// Evaluate (replace) variable with final values
reqData, err = expressions.Evaluate(reqData, finalVars)
if err != nil {
return nil, errkit.Wrap(err, "could not evaluate helper expressions")
return nil, ErrEvalExpression.Wrap(err).WithTag("http")
}
if isRawRequest {
@@ -249,7 +219,7 @@ func (r *requestGenerator) Make(ctx context.Context, input *contextargs.Context,
reqURL, err := urlutil.ParseAbsoluteURL(reqData, true)
if err != nil {
return nil, errkit.Newf("failed to parse url %v while creating http request", reqData)
return nil, errorutil.NewWithTag("http", "failed to parse url %v while creating http request", reqData)
}
// while merging parameters first preference is given to target params
finalparams := parsed.Params
@@ -282,7 +252,7 @@ func (r *requestGenerator) makeSelfContainedRequest(ctx context.Context, data st
// evaluate request
data, err := expressions.Evaluate(data, values)
if err != nil {
return nil, errkit.Wrap(err, "could not evaluate helper expressions")
return nil, ErrEvalExpression.Wrap(err).WithTag("self-contained")
}
// If the request is a raw request, get the URL from the request
// header and use it to make the request.
@@ -305,7 +275,7 @@ func (r *requestGenerator) makeSelfContainedRequest(ctx context.Context, data st
}
if err := expressions.ContainsUnresolvedVariables(parts[1]); err != nil && !r.request.SkipVariablesCheck {
return nil, errkit.Newf("unresolved variables `%v` found in request", parts[1])
return nil, ErrUnresolvedVars.Msgf(parts[1])
}
parsed, err := urlutil.ParseURL(parts[1], true)
@@ -319,19 +289,19 @@ func (r *requestGenerator) makeSelfContainedRequest(ctx context.Context, data st
// Evaluate (replace) variable with final values
data, err = expressions.Evaluate(data, values)
if err != nil {
return nil, errkit.Wrap(err, "could not evaluate helper expressions")
return nil, ErrEvalExpression.Wrap(err).WithTag("self-contained", "raw")
}
return r.generateRawRequest(ctx, data, parsed, values, payloads)
}
if err := expressions.ContainsUnresolvedVariables(data); err != nil && !r.request.SkipVariablesCheck {
// early exit: if there are any unresolved variables in `path` after evaluation
// then return early since this will definitely fail
return nil, errkit.Newf("unresolved variables `%v` found in request", data)
return nil, ErrUnresolvedVars.Msgf(data)
}
urlx, err := urlutil.ParseURL(data, true)
if err != nil {
return nil, errkit.Wrapf(err, "failed to parse %v in self contained request", data)
return nil, errorutil.NewWithErr(err).Msgf("failed to parse %v in self contained request", data).WithTag("self-contained")
}
return r.generateHttpRequest(ctx, urlx, values, payloads)
}
@@ -342,7 +312,7 @@ func (r *requestGenerator) makeSelfContainedRequest(ctx context.Context, data st
func (r *requestGenerator) generateHttpRequest(ctx context.Context, urlx *urlutil.URL, finalVars, generatorValues map[string]interface{}) (*generatedRequest, error) {
method, err := expressions.Evaluate(r.request.Method.String(), finalVars)
if err != nil {
return nil, errkit.Wrap(err, "failed to evaluate while generating http request")
return nil, ErrEvalExpression.Wrap(err).Msgf("failed to evaluate while generating http request")
}
// Build a request on the specified URL
req, err := retryablehttp.NewRequestFromURLWithContext(ctx, method, urlx, nil)
@@ -371,7 +341,7 @@ func (r *requestGenerator) generateRawRequest(ctx context.Context, rawRequest st
rawRequestData, err = raw.Parse(rawRequest, baseURL, r.request.Unsafe, r.request.DisablePathAutomerge)
}
if err != nil {
return nil, errkit.Wrap(err, "failed to parse raw request")
return nil, errorutil.NewWithErr(err).Msgf("failed to parse raw request")
}
// Unsafe option uses rawhttp library
@@ -387,7 +357,7 @@ func (r *requestGenerator) generateRawRequest(ctx context.Context, rawRequest st
}
urlx, err := urlutil.ParseAbsoluteURL(rawRequestData.FullURL, true)
if err != nil {
return nil, errkit.Wrapf(err, "failed to create request with url %v got %v", rawRequestData.FullURL, err)
return nil, errorutil.NewWithErr(err).Msgf("failed to create request with url %v got %v", rawRequestData.FullURL, err).WithTag("raw")
}
req, err := retryablehttp.NewRequestFromURLWithContext(ctx, rawRequestData.Method, urlx, rawRequestData.Data)
if err != nil {
@@ -444,7 +414,7 @@ func (r *requestGenerator) fillRequest(req *retryablehttp.Request, values map[st
}
value, err := expressions.Evaluate(value, values)
if err != nil {
return nil, errkit.Wrap(err, "failed to evaluate while adding headers to request")
return nil, ErrEvalExpression.Wrap(err).Msgf("failed to evaluate while adding headers to request")
}
req.Header[header] = []string{value}
if header == "Host" {
@@ -465,7 +435,7 @@ func (r *requestGenerator) fillRequest(req *retryablehttp.Request, values map[st
}
body, err := expressions.Evaluate(body, values)
if err != nil {
return nil, errkit.Wrap(err, "could not evaluate helper expressions")
return nil, ErrEvalExpression.Wrap(err)
}
bodyReader, err := readerutil.NewReusableReadCloser([]byte(body))
if err != nil {
+8 -8
View File
@@ -12,7 +12,7 @@ import (
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/authprovider/authx"
"github.com/projectdiscovery/rawhttp/client"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
stringsutil "github.com/projectdiscovery/utils/strings"
urlutil "github.com/projectdiscovery/utils/url"
)
@@ -48,7 +48,7 @@ func Parse(request string, inputURL *urlutil.URL, unsafe, disablePathAutomerge b
case strings.HasPrefix(rawrequest.Path, "http") && !unsafe:
urlx, err := urlutil.ParseURL(rawrequest.Path, true)
if err != nil {
return nil, errkit.Wrapf(err, "failed to parse url %v from template", rawrequest.Path)
return nil, errorutil.NewWithErr(err).WithTag("raw").Msgf("failed to parse url %v from template", rawrequest.Path)
}
cloned := inputURL.Clone()
cloned.Params.IncludeEquals = true
@@ -57,7 +57,7 @@ func Parse(request string, inputURL *urlutil.URL, unsafe, disablePathAutomerge b
}
parseErr := cloned.MergePath(urlx.GetRelativePath(), true)
if parseErr != nil {
return nil, errkit.Wrapf(parseErr, "could not automergepath for template path %v", urlx.GetRelativePath())
return nil, errorutil.NewWithTag("raw", "could not automergepath for template path %v", urlx.GetRelativePath()).Wrap(parseErr)
}
rawrequest.Path = cloned.GetRelativePath()
// If unsafe changes must be made in raw request string itself
@@ -94,7 +94,7 @@ func Parse(request string, inputURL *urlutil.URL, unsafe, disablePathAutomerge b
}
err = cloned.MergePath(rawrequest.Path, true)
if err != nil {
return nil, errkit.Wrapf(err, "failed to automerge %v from unsafe template", rawrequest.Path)
return nil, errorutil.NewWithErr(err).WithTag("raw").Msgf("failed to automerge %v from unsafe template", rawrequest.Path)
}
unsafeRelativePath = cloned.GetRelativePath()
}
@@ -116,7 +116,7 @@ func Parse(request string, inputURL *urlutil.URL, unsafe, disablePathAutomerge b
}
parseErr := cloned.MergePath(rawrequest.Path, true)
if parseErr != nil {
return nil, errkit.Wrapf(parseErr, "could not automergepath for template path %v", rawrequest.Path)
return nil, errorutil.NewWithTag("raw", "could not automergepath for template path %v", rawrequest.Path).Wrap(parseErr)
}
rawrequest.Path = cloned.GetRelativePath()
}
@@ -145,18 +145,18 @@ func ParseRawRequest(request string, unsafe bool) (*Request, error) {
if strings.HasPrefix(req.Path, "http") {
urlx, err := urlutil.Parse(req.Path)
if err != nil {
return nil, errkit.Wrapf(err, "failed to parse url %v", req.Path)
return nil, errorutil.NewWithErr(err).Msgf("failed to parse url %v", req.Path)
}
req.Path = urlx.GetRelativePath()
req.FullURL = urlx.String()
} else {
if req.Path == "" {
return nil, errkit.New("path cannot be empty in self contained request")
return nil, errorutil.NewWithTag("self-contained-raw", "path cannot be empty in self contained request")
}
// given url is relative construct one using Host Header
if _, ok := req.Headers["Host"]; !ok {
return nil, errkit.New("host header is required for relative path")
return nil, errorutil.NewWithTag("self-contained-raw", "host header is required for relative path")
}
// Review: Current default scheme in self contained templates if relative path is provided is http
req.FullURL = fmt.Sprintf("%s://%s%s", urlutil.HTTP, strings.TrimSpace(req.Headers["Host"]), req.Path)
+5 -6
View File
@@ -76,7 +76,7 @@ func (request *Request) executeFuzzingRule(input *contextargs.Context, previous
if errors.Is(err, ErrMissingVars) {
return err
}
gologger.Verbose().Msgf("[%s] fuzz: payload request execution failed: %s\n", request.options.TemplateID, err)
gologger.Verbose().Msgf("[%s] fuzz: payload request execution failed : %s\n", request.options.TemplateID, err)
}
return nil
}
@@ -103,13 +103,13 @@ func (request *Request) executeFuzzingRule(input *contextargs.Context, previous
// in case of any error, return it
if fuzz.IsErrRuleNotApplicable(err) {
// log and fail silently
gologger.Verbose().Msgf("[%s] fuzz: %s\n", request.options.TemplateID, err)
gologger.Verbose().Msgf("[%s] fuzz: rule not applicable : %s\n", request.options.TemplateID, err)
return nil
}
if errors.Is(err, ErrMissingVars) {
return err
}
gologger.Verbose().Msgf("[%s] fuzz: payload request execution failed: %s\n", request.options.TemplateID, err)
gologger.Verbose().Msgf("[%s] fuzz: payload request execution failed : %s\n", request.options.TemplateID, err)
}
return nil
}
@@ -158,7 +158,7 @@ func (request *Request) executeAllFuzzingRules(input *contextargs.Context, value
continue
}
if fuzz.IsErrRuleNotApplicable(err) {
gologger.Verbose().Msgf("[%s] fuzz: %s\n", request.options.TemplateID, err)
gologger.Verbose().Msgf("[%s] fuzz: rule not applicable : %s\n", request.options.TemplateID, err)
continue
}
if err == types.ErrNoMoreRequests {
@@ -168,9 +168,8 @@ func (request *Request) executeAllFuzzingRules(input *contextargs.Context, value
}
if !applicable {
return fmt.Errorf("no rule was applicable for this request: %v", input.MetaInput.Input)
return fuzz.ErrRuleNotApplicable.Msgf(fmt.Sprintf("no rule was applicable for this request: %v", input.MetaInput.Input))
}
return nil
}
+2 -2
View File
@@ -14,7 +14,7 @@ import (
awsconfig "github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
)
const defaultEmptyPayloadHash = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
@@ -60,7 +60,7 @@ func (a *AWSSigner) SignHTTP(ctx context.Context, request *http.Request) error {
// contentHash is sha256 hash of response body
contentHash := a.getPayloadHash(request)
if err := a.signer.SignHTTP(ctx, *a.creds, request, contentHash, a.options.Service, a.options.Region, time.Now()); err != nil {
return errkit.Wrap(err, "failed to sign http request using aws v4 signer")
return errorutil.NewWithErr(err).Msgf("failed to sign http request using aws v4 signer")
}
// add x-amz-content-sha256 header to request
request.Header.Set("x-amz-content-sha256", contentHash)
+3 -3
View File
@@ -6,7 +6,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
"github.com/projectdiscovery/rawhttp"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
)
// dump creates a dump of the http request in form of a byte slice
@@ -15,14 +15,14 @@ func dump(req *generatedRequest, reqURL string) ([]byte, error) {
// Use a clone to avoid a race condition with the http transport
bin, err := req.request.Clone(req.request.Context()).Dump()
if err != nil {
return nil, errkit.Wrapf(err, "could not dump request: %v", req.request.String())
return nil, errorutil.NewWithErr(err).WithTag("http").Msgf("could not dump request: %v", req.request.String())
}
return bin, nil
}
rawHttpOptions := &rawhttp.Options{CustomHeaders: req.rawRequest.UnsafeHeaders, CustomRawBytes: req.rawRequest.UnsafeRawBytes}
bin, err := rawhttp.DumpRequestRaw(req.rawRequest.Method, reqURL, req.rawRequest.Path, generators.ExpandMapValues(req.rawRequest.Headers), io.NopCloser(strings.NewReader(req.rawRequest.Data)), rawHttpOptions)
if err != nil {
return nil, errkit.Wrapf(err, "could not dump request: %v", reqURL)
return nil, errorutil.NewWithErr(err).WithTag("http").Msgf("could not dump request: %v", reqURL)
}
return bin, nil
}
+7 -6
View File
@@ -34,6 +34,7 @@ import (
templateTypes "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
iputil "github.com/projectdiscovery/utils/ip"
mapsutil "github.com/projectdiscovery/utils/maps"
syncutil "github.com/projectdiscovery/utils/sync"
@@ -127,14 +128,14 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
}
}
if err := compiled.Compile(); err != nil {
return errkit.Newf("could not compile operators got %v", err)
return errorutil.NewWithTag(request.TemplateID, "could not compile operators got %v", err)
}
request.CompiledOperators = compiled
}
// "Port" is a special variable and it should not contains any dsl expressions
if strings.Contains(request.getPort(), "{{") {
return errkit.New("'Port' variable cannot contain any dsl expressions")
return errorutil.NewWithTag(request.TemplateID, "'Port' variable cannot contain any dsl expressions")
}
if request.Init != "" {
@@ -218,11 +219,11 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
initCompiled, err := compiler.SourceAutoMode(request.Init, false)
if err != nil {
return errkit.Newf("could not compile init code: %s", err)
return errorutil.NewWithTag(request.TemplateID, "could not compile init code: %s", err)
}
result, err := request.options.JsCompiler.ExecuteWithOptions(initCompiled, args, opts)
if err != nil {
return errkit.Newf("could not execute pre-condition: %s", err)
return errorutil.NewWithTag(request.TemplateID, "could not execute pre-condition: %s", err)
}
if types.ToString(result["error"]) != "" {
gologger.Warning().Msgf("[%s] Init failed with error %v\n", request.TemplateID, result["error"])
@@ -239,7 +240,7 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
if request.PreCondition != "" {
preConditionCompiled, err := compiler.SourceAutoMode(request.PreCondition, false)
if err != nil {
return errkit.Newf("could not compile pre-condition: %s", err)
return errorutil.NewWithTag(request.TemplateID, "could not compile pre-condition: %s", err)
}
request.preConditionCompiled = preConditionCompiled
}
@@ -248,7 +249,7 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
if request.Code != "" {
scriptCompiled, err := compiler.SourceAutoMode(request.Code, false)
if err != nil {
return errkit.Newf("could not compile javascript code: %s", err)
return errorutil.NewWithTag(request.TemplateID, "could not compile javascript code: %s", err)
}
request.scriptCompiled = scriptCompiled
}
+3 -3
View File
@@ -12,7 +12,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/expressions"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/network/networkclientpool"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
fileutil "github.com/projectdiscovery/utils/file"
)
@@ -196,10 +196,10 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
}
portInt, err := strconv.Atoi(port)
if err != nil {
return errkit.Wrapf(err, "could not parse port %v from '%s'", port, request.Port)
return errorutil.NewWithErr(err).Msgf("could not parse port %v from '%s'", port, request.Port)
}
if portInt < 1 || portInt > 65535 {
return errkit.Newf("port %v is not in valid range", portInt)
return errorutil.NewWithTag(request.TemplateID, "port %v is not in valid range", portInt)
}
request.ports = append(request.ports, port)
}
+2 -2
View File
@@ -31,7 +31,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/network/networkclientpool"
protocolutils "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
templateTypes "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
mapsutil "github.com/projectdiscovery/utils/maps"
"github.com/projectdiscovery/utils/reader"
syncutil "github.com/projectdiscovery/utils/sync"
@@ -362,7 +362,7 @@ func (request *Request) executeRequestWithPayloads(variables map[string]interfac
if input.Read > 0 {
buffer, err := ConnReadNWithTimeout(conn, int64(input.Read), request.options.Options.GetTimeouts().TcpReadTimeout)
if err != nil {
return errkit.Wrap(err, "could not read response from connection")
return errorutil.NewWithErr(err).Msgf("could not read response from connection")
}
responseBuilder.Write(buffer)
+19
View File
@@ -447,8 +447,21 @@ func (e *ExecutorOptions) ApplyNewEngineOptions(n *ExecutorOptions) {
return
}
// The types.Options include the ExecutionID among other things
e.Options = n.Options.Copy()
// Keep the template-specific fields, but replace the rest
/*
e.TemplateID = n.TemplateID
e.TemplatePath = n.TemplatePath
e.TemplateInfo = n.TemplateInfo
e.TemplateVerifier = n.TemplateVerifier
e.RawTemplate = n.RawTemplate
e.Variables = n.Variables
e.Constants = n.Constants
*/
e.Output = n.Output
e.Options = n.Options
e.IssuesClient = n.IssuesClient
e.Progress = n.Progress
e.RateLimiter = n.RateLimiter
@@ -457,6 +470,8 @@ func (e *ExecutorOptions) ApplyNewEngineOptions(n *ExecutorOptions) {
e.Browser = n.Browser
e.Interactsh = n.Interactsh
e.HostErrorsCache = n.HostErrorsCache
e.StopAtFirstMatch = n.StopAtFirstMatch
e.ExcludeMatchers = n.ExcludeMatchers
e.InputHelper = n.InputHelper
e.FuzzParamsFrequency = n.FuzzParamsFrequency
e.FuzzStatsDB = n.FuzzStatsDB
@@ -464,6 +479,10 @@ func (e *ExecutorOptions) ApplyNewEngineOptions(n *ExecutorOptions) {
e.Colorizer = n.Colorizer
e.WorkflowLoader = n.WorkflowLoader
e.ResumeCfg = n.ResumeCfg
e.ProtocolType = n.ProtocolType
e.Flow = n.Flow
e.IsMultiProtocol = n.IsMultiProtocol
e.templateCtxStore = n.templateCtxStore
e.JsCompiler = n.JsCompiler
e.AuthProvider = n.AuthProvider
e.TemporaryDirectory = n.TemporaryDirectory
+9 -9
View File
@@ -33,7 +33,7 @@ import (
"github.com/projectdiscovery/tlsx/pkg/tlsx"
"github.com/projectdiscovery/tlsx/pkg/tlsx/clients"
"github.com/projectdiscovery/tlsx/pkg/tlsx/openssl"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
stringsutil "github.com/projectdiscovery/utils/strings"
)
@@ -121,7 +121,7 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
CustomDialer: options.CustomFastdialer,
})
if err != nil {
return errkit.Wrap(err, "could not get network client")
return errorutil.NewWithTag("ssl", "could not get network client").Wrap(err)
}
request.dialer = client
switch {
@@ -130,7 +130,7 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
request.ScanMode = "auto"
case !stringsutil.EqualFoldAny(request.ScanMode, "auto", "openssl", "ztls", "ctls"):
return errkit.Newf("template %v does not contain valid scan-mode", request.TemplateID)
return errorutil.NewWithTag(request.TemplateID, "template %v does not contain valid scan-mode", request.TemplateID)
case request.ScanMode == "openssl" && !openssl.IsAvailable():
// if openssl is not installed instead of failing "auto" scanmode is used
@@ -169,7 +169,7 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
tlsxService, err := tlsx.New(tlsxOptions)
if err != nil {
return errkit.New("could not create tlsx service")
return errorutil.NewWithTag(request.TemplateID, "could not create tlsx service")
}
request.tlsx = tlsxService
@@ -178,7 +178,7 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
compiled.ExcludeMatchers = options.ExcludeMatchers
compiled.TemplateID = options.TemplateID
if err := compiled.Compile(); err != nil {
return errkit.Newf("could not compile operators got %v", err)
return errorutil.NewWithTag(request.TemplateID, "could not compile operators got %v", err)
}
request.CompiledOperators = compiled
}
@@ -236,7 +236,7 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicVa
addressToDial := string(finalAddress)
host, port, err := net.SplitHostPort(addressToDial)
if err != nil {
return errkit.Wrap(err, "could not split input host port")
return errorutil.NewWithErr(err).Msgf("could not split input host port")
}
var hostIp string
@@ -250,7 +250,7 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicVa
if err != nil {
requestOptions.Output.Request(requestOptions.TemplateID, input.MetaInput.Input, request.Type().String(), err)
requestOptions.Progress.IncrementFailedRequestsBy(1)
return errkit.Wrap(err, "could not connect to server")
return errorutil.NewWithTag(request.TemplateID, "could not connect to server").Wrap(err)
}
requestOptions.Output.Request(requestOptions.TemplateID, hostPort, request.Type().String(), err)
@@ -287,7 +287,7 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicVa
// if response is not struct compatible, error out
if !structs.IsStruct(response) {
return errkit.Newf("response cannot be parsed into a struct: %v", response)
return errorutil.NewWithTag("ssl", "response cannot be parsed into a struct: %v", response)
}
// Convert response to key value pairs and first cert chain item as well
@@ -307,7 +307,7 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicVa
// if certificate response is not struct compatible, error out
if !structs.IsStruct(response.CertificateResponse) {
return errkit.Newf("certificate response cannot be parsed into a struct: %v", response.CertificateResponse)
return errorutil.NewWithTag("ssl", "certificate response cannot be parsed into a struct: %v", response.CertificateResponse)
}
responseParsed = structs.New(response.CertificateResponse)
@@ -5,7 +5,7 @@ import (
"fmt"
"strings"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
)
func CreateLink(title string, url string) string {
@@ -20,7 +20,7 @@ func CreateTable(headers []string, rows [][]string) (string, error) {
builder := &bytes.Buffer{}
headerSize := len(headers)
if headers == nil || headerSize == 0 {
return "", errkit.New("No headers provided")
return "", errorutil.New("No headers provided")
}
builder.WriteString(CreateTableHeader(headers...))
@@ -34,7 +34,7 @@ func CreateTable(headers []string, rows [][]string) (string, error) {
copy(extendedRows, row)
builder.WriteString(CreateTableRow(extendedRows...))
} else {
return "", errkit.New("Too many columns for the given headers")
return "", errorutil.New("Too many columns for the given headers")
}
}
+14 -14
View File
@@ -31,7 +31,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/reporting/trackers/gitlab"
"github.com/projectdiscovery/nuclei/v3/pkg/reporting/trackers/jira"
"github.com/projectdiscovery/nuclei/v3/pkg/reporting/trackers/linear"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
fileutil "github.com/projectdiscovery/utils/file"
)
@@ -84,7 +84,7 @@ func New(options *Options, db string, doNotDedupe bool) (Client, error) {
options.GitHub.OmitRaw = options.OmitRaw
tracker, err := github.New(options.GitHub)
if err != nil {
return nil, errkit.Wrapf(err, "could not create reporting client: %v", ErrReportingClientCreation)
return nil, errorutil.NewWithErr(err).Wrap(ErrReportingClientCreation)
}
client.trackers = append(client.trackers, tracker)
}
@@ -93,7 +93,7 @@ func New(options *Options, db string, doNotDedupe bool) (Client, error) {
options.GitLab.OmitRaw = options.OmitRaw
tracker, err := gitlab.New(options.GitLab)
if err != nil {
return nil, errkit.Wrapf(err, "could not create reporting client: %v", ErrReportingClientCreation)
return nil, errorutil.NewWithErr(err).Wrap(ErrReportingClientCreation)
}
client.trackers = append(client.trackers, tracker)
}
@@ -102,7 +102,7 @@ func New(options *Options, db string, doNotDedupe bool) (Client, error) {
options.Gitea.OmitRaw = options.OmitRaw
tracker, err := gitea.New(options.Gitea)
if err != nil {
return nil, errkit.Wrapf(err, "could not create reporting client: %v", ErrReportingClientCreation)
return nil, errorutil.NewWithErr(err).Wrap(ErrReportingClientCreation)
}
client.trackers = append(client.trackers, tracker)
}
@@ -111,7 +111,7 @@ func New(options *Options, db string, doNotDedupe bool) (Client, error) {
options.Jira.OmitRaw = options.OmitRaw
tracker, err := jira.New(options.Jira)
if err != nil {
return nil, errkit.Wrapf(err, "could not create reporting client: %v", ErrReportingClientCreation)
return nil, errorutil.NewWithErr(err).Wrap(ErrReportingClientCreation)
}
client.trackers = append(client.trackers, tracker)
}
@@ -120,35 +120,35 @@ func New(options *Options, db string, doNotDedupe bool) (Client, error) {
options.Linear.OmitRaw = options.OmitRaw
tracker, err := linear.New(options.Linear)
if err != nil {
return nil, errkit.Wrapf(err, "could not create reporting client: %v", ErrReportingClientCreation)
return nil, errorutil.NewWithErr(err).Wrap(ErrReportingClientCreation)
}
client.trackers = append(client.trackers, tracker)
}
if options.MarkdownExporter != nil {
exporter, err := markdown.New(options.MarkdownExporter)
if err != nil {
return nil, errkit.Wrapf(err, "could not create export client: %v", ErrExportClientCreation)
return nil, errorutil.NewWithErr(err).Wrap(ErrExportClientCreation)
}
client.exporters = append(client.exporters, exporter)
}
if options.SarifExporter != nil {
exporter, err := sarif.New(options.SarifExporter)
if err != nil {
return nil, errkit.Wrapf(err, "could not create export client: %v", ErrExportClientCreation)
return nil, errorutil.NewWithErr(err).Wrap(ErrExportClientCreation)
}
client.exporters = append(client.exporters, exporter)
}
if options.JSONExporter != nil {
exporter, err := json_exporter.New(options.JSONExporter)
if err != nil {
return nil, errkit.Wrapf(err, "could not create export client: %v", ErrExportClientCreation)
return nil, errorutil.NewWithErr(err).Wrap(ErrExportClientCreation)
}
client.exporters = append(client.exporters, exporter)
}
if options.JSONLExporter != nil {
exporter, err := jsonl.New(options.JSONLExporter)
if err != nil {
return nil, errkit.Wrapf(err, "could not create export client: %v", ErrExportClientCreation)
return nil, errorutil.NewWithErr(err).Wrap(ErrExportClientCreation)
}
client.exporters = append(client.exporters, exporter)
}
@@ -157,7 +157,7 @@ func New(options *Options, db string, doNotDedupe bool) (Client, error) {
options.ElasticsearchExporter.ExecutionId = options.ExecutionId
exporter, err := es.New(options.ElasticsearchExporter)
if err != nil {
return nil, errkit.Wrapf(err, "could not create export client: %v", ErrExportClientCreation)
return nil, errorutil.NewWithErr(err).Wrap(ErrExportClientCreation)
}
client.exporters = append(client.exporters, exporter)
}
@@ -166,14 +166,14 @@ func New(options *Options, db string, doNotDedupe bool) (Client, error) {
options.SplunkExporter.ExecutionId = options.ExecutionId
exporter, err := splunk.New(options.SplunkExporter)
if err != nil {
return nil, errkit.Wrapf(err, "could not create export client: %v", ErrExportClientCreation)
return nil, errorutil.NewWithErr(err).Wrap(ErrExportClientCreation)
}
client.exporters = append(client.exporters, exporter)
}
if options.MongoDBExporter != nil {
exporter, err := mongo.New(options.MongoDBExporter)
if err != nil {
return nil, errkit.Wrapf(err, "could not create export client: %v", ErrExportClientCreation)
return nil, errorutil.NewWithErr(err).Wrap(ErrExportClientCreation)
}
client.exporters = append(client.exporters, exporter)
}
@@ -227,7 +227,7 @@ func CreateConfigIfNotExists() error {
}
reportingFile, err := os.Create(reportingConfig)
if err != nil {
return errkit.Wrap(err, "could not create config file")
return errorutil.NewWithErr(err).Msgf("could not create config file")
}
defer func() {
_ = reportingFile.Close()
+1 -1
View File
@@ -54,7 +54,7 @@ type Options struct {
// New creates a new issue tracker integration client based on options.
func New(options *Options) (*Integration, error) {
var transport = http.DefaultTransport
var transport http.RoundTripper = http.DefaultTransport
if options.HttpClient != nil && options.HttpClient.HTTPClient.Transport != nil {
transport = options.HttpClient.HTTPClient.Transport
}
+8 -83
View File
@@ -25,7 +25,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/tmplexec"
"github.com/projectdiscovery/nuclei/v3/pkg/utils"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/json"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
stringsutil "github.com/projectdiscovery/utils/strings"
)
@@ -64,13 +64,6 @@ func Parse(filePath string, preprocessor Preprocessor, options *protocols.Execut
newBase.TemplateInfo = tplCopy.Options.TemplateInfo
newBase.TemplateVerifier = tplCopy.Options.TemplateVerifier
newBase.RawTemplate = tplCopy.Options.RawTemplate
if tplCopy.Options.Variables.Len() > 0 {
newBase.Variables = tplCopy.Options.Variables
}
if len(tplCopy.Options.Constants) > 0 {
newBase.Constants = tplCopy.Options.Constants
}
tplCopy.Options = newBase
tplCopy.Options.ApplyNewEngineOptions(options)
@@ -163,16 +156,12 @@ func Parse(filePath string, preprocessor Preprocessor, options *protocols.Execut
// Compile the workflow request
if len(template.Workflows) > 0 {
compiled := &template.Workflow
compileWorkflow(filePath, preprocessor, tplCopy.Options, compiled, tplCopy.Options.WorkflowLoader)
compileWorkflow(filePath, preprocessor, options, compiled, options.WorkflowLoader)
template.CompiledWorkflow = compiled
template.CompiledWorkflow.Options = tplCopy.Options
template.CompiledWorkflow.Options = options
}
if isCachedTemplateValid(template) {
// options.Logger.Error().Msgf("returning cached template %s after recompiling %d requests", tplCopy.Options.TemplateID, tplCopy.Requests())
return template, nil
}
// else: fallthrough to re-parse template from scratch
// options.Logger.Error().Msgf("returning cached template %s after recompiling %d requests", tplCopy.Options.TemplateID, tplCopy.Requests())
return template, nil
}
}
@@ -224,31 +213,11 @@ func Parse(filePath string, preprocessor Preprocessor, options *protocols.Execut
}
template.Path = filePath
if !options.DoNotCache {
// Store a sanitized template in compiled cache to avoid retaining engine-scoped state.
cacheTpl := *template
cacheTpl.Options = sanitizeOptionsForCache(template.Options)
// Raw template bytes are not needed in compiled cache; keep per-engine
cacheTpl.Options.RawTemplate = nil
parser.compiledTemplatesCache.Store(filePath, &cacheTpl, nil, err)
parser.compiledTemplatesCache.Store(filePath, template, nil, err)
}
return template, nil
}
// sanitizeOptionsForCache strips engine-scoped fields from ExecutorOptions to avoid
// retaining per-engine references in the shared compiled cache.
func sanitizeOptionsForCache(src *protocols.ExecutorOptions) *protocols.ExecutorOptions {
if src == nil {
return nil
}
return &protocols.ExecutorOptions{
// Intentionally exclude TemplateID/Path/Verifier and RawTemplate to avoid engine leakage
StopAtFirstMatch: src.StopAtFirstMatch,
ProtocolType: src.ProtocolType,
Flow: src.Flow,
IsMultiProtocol: src.IsMultiProtocol,
}
}
// isGlobalMatchersEnabled checks if any of requests in the template
// have global matchers enabled. It iterates through all requests and
// returns true if at least one request has global matchers enabled;
@@ -511,7 +480,7 @@ func parseTemplate(data []byte, srcOptions *protocols.ExecutorOptions) (*Templat
}
}
if err != nil {
return nil, errkit.Wrapf(err, "failed to parse %s", template.Path)
return nil, errorutil.NewWithErr(err).Msgf("failed to parse %s", template.Path)
}
if utils.IsBlank(template.Info.Name) {
@@ -571,7 +540,7 @@ func parseTemplate(data []byte, srcOptions *protocols.ExecutorOptions) (*Templat
// load `flow` and `source` in code protocol from file
// if file is referenced instead of actual source code
if err := template.ImportFileRefs(template.Options); err != nil {
return nil, errkit.Wrapf(err, "failed to load file refs for %s", template.ID)
return nil, errorutil.NewWithErr(err).Msgf("failed to load file refs for %s", template.ID)
}
if err := template.compileProtocolRequests(template.Options); err != nil {
@@ -610,50 +579,6 @@ func parseTemplate(data []byte, srcOptions *protocols.ExecutorOptions) (*Templat
return template, nil
}
// isCachedTemplateValid validates that a cached template is still usable after
// option updates
func isCachedTemplateValid(template *Template) bool {
// no requests or workflows
if template.Requests() == 0 && len(template.Workflows) == 0 {
return false
}
// options not initialized
if template.Options == nil {
return false
}
// executer not available for non-workflow template
if len(template.Workflows) == 0 && template.Executer == nil {
return false
}
// compiled workflow not available
if len(template.Workflows) > 0 && template.CompiledWorkflow == nil {
return false
}
// template ID mismatch
if template.Options.TemplateID != template.ID {
return false
}
// executer exists but no requests or flow available
if template.Executer != nil {
// NOTE(dwisiswant0): This is a basic sanity check since we can't access
// private fields, but we can check requests tho
if template.Requests() == 0 && template.Options.Flow == "" {
return false
}
}
if template.Options.Options == nil {
return false
}
return true
}
var (
jsCompiler *compiler.Compiler
jsCompilerOnce = sync.OnceFunc(func() {
-158
View File
@@ -206,161 +206,3 @@ func TestWrongWorkflow(t *testing.T) {
require.Nil(t, got, "could not parse template")
require.ErrorContains(t, err, "workflows cannot have other protocols")
}
func Test_SharedCompiledCache_SharedAcrossParsers(t *testing.T) {
setup()
p1 := templates.NewSharedParserWithCompiledCache()
p2 := templates.NewSharedParserWithCompiledCache()
exec1 := &protocols.ExecutorOptions{
Output: testutils.NewMockOutputWriter(testutils.DefaultOptions.OmitTemplate),
Options: testutils.DefaultOptions,
Progress: executerOpts.Progress,
Catalog: executerOpts.Catalog,
RateLimiter: executerOpts.RateLimiter,
Parser: p1,
}
// reinit options fully for isolation
opts2 := testutils.DefaultOptions
testutils.Init(opts2)
progressImpl, _ := progress.NewStatsTicker(0, false, false, false, 0)
exec2 := &protocols.ExecutorOptions{
Output: testutils.NewMockOutputWriter(opts2.OmitTemplate),
Options: opts2,
Progress: progressImpl,
Catalog: executerOpts.Catalog,
RateLimiter: executerOpts.RateLimiter,
Parser: p2,
}
filePath := "tests/match-1.yaml"
got1, err := templates.Parse(filePath, nil, exec1)
require.NoError(t, err)
require.NotNil(t, got1)
got2, err := templates.Parse(filePath, nil, exec2)
require.NoError(t, err)
require.NotNil(t, got2)
require.Equal(t, p1.CompiledCache(), p2.CompiledCache())
require.Greater(t, p1.CompiledCount(), 0)
require.Equal(t, p1.CompiledCount(), p2.CompiledCount())
}
func Test_SharedCompiledCache_OptionsIsolation(t *testing.T) {
setup()
p1 := templates.NewSharedParserWithCompiledCache()
p2 := templates.NewSharedParserWithCompiledCache()
exec1 := &protocols.ExecutorOptions{
Output: testutils.NewMockOutputWriter(testutils.DefaultOptions.OmitTemplate),
Options: testutils.DefaultOptions,
Progress: executerOpts.Progress,
Catalog: executerOpts.Catalog,
RateLimiter: executerOpts.RateLimiter,
Parser: p1,
}
// reinit options fully for isolation
opts2 := testutils.DefaultOptions
testutils.Init(opts2)
progressImpl, _ := progress.NewStatsTicker(0, false, false, false, 0)
exec2 := &protocols.ExecutorOptions{
Output: testutils.NewMockOutputWriter(opts2.OmitTemplate),
Options: opts2,
Progress: progressImpl,
Catalog: executerOpts.Catalog,
RateLimiter: executerOpts.RateLimiter,
Parser: p2,
}
filePath := "tests/match-1.yaml"
got1, err := templates.Parse(filePath, nil, exec1)
require.NoError(t, err)
require.NotNil(t, got1)
got2, err := templates.Parse(filePath, nil, exec2)
require.NoError(t, err)
require.NotNil(t, got2)
require.NotEqual(t, got1.Options, got2.Options)
}
// compiled cache does not retain engine-scoped fields
func Test_CompiledCache_SanitizesOptions(t *testing.T) {
setup()
p := templates.NewSharedParserWithCompiledCache()
exec := executerOpts
exec.Parser = p
filePath := "tests/match-1.yaml"
got, err := templates.Parse(filePath, nil, exec)
require.NoError(t, err)
require.NotNil(t, got)
cached, raw, err := p.CompiledCache().Has(filePath)
require.NoError(t, err)
require.NotNil(t, cached)
require.Nil(t, raw)
// cached template must not hold engine-scoped references
require.Nil(t, cached.Options.Options)
require.Empty(t, cached.Options.TemplateVerifier)
require.Empty(t, cached.Options.TemplateID)
require.Empty(t, cached.Options.TemplatePath)
require.False(t, cached.Options.StopAtFirstMatch)
}
// different engines see different Options pointers
func Test_EngineIsolation_NoCrossLeaks(t *testing.T) {
setup()
p1 := templates.NewSharedParserWithCompiledCache()
p2 := templates.NewSharedParserWithCompiledCache()
// engine 1
exec1 := &protocols.ExecutorOptions{
Output: executerOpts.Output,
Options: executerOpts.Options,
Progress: executerOpts.Progress,
Catalog: executerOpts.Catalog,
RateLimiter: executerOpts.RateLimiter,
Parser: p1,
}
// engine 2 with a fresh options instance
opts2 := testutils.DefaultOptions
testutils.Init(opts2)
progress2, _ := progress.NewStatsTicker(0, false, false, false, 0)
exec2 := &protocols.ExecutorOptions{
Output: testutils.NewMockOutputWriter(opts2.OmitTemplate),
Options: opts2,
Progress: progress2,
Catalog: executerOpts.Catalog,
RateLimiter: executerOpts.RateLimiter,
Parser: p2,
}
filePath := "tests/match-1.yaml"
got1, err := templates.Parse(filePath, nil, exec1)
require.NoError(t, err)
got2, err := templates.Parse(filePath, nil, exec2)
require.NoError(t, err)
// template options must be distinct per engine
require.NotEqual(t, got1.Options, got2.Options)
// http request options must bind to engine-specific ExecutorOptions copies (not shared)
require.NotEmpty(t, got1.RequestsHTTP)
require.NotEmpty(t, got2.RequestsHTTP)
r1 := got1.RequestsHTTP[0]
r2 := got2.RequestsHTTP[0]
// ensure options structs are not the same pointer
require.NotSame(t, r1.Options().Options, r2.Options().Options)
// mutate engine2 options and ensure it doesn't affect engine1
r2.Options().Options.RateLimit = 999
require.NotEqual(t, r1.Options().Options.RateLimit, r2.Options().Options.RateLimit)
// compiled cache instance shared, but without engine leakage
require.Equal(t, p1.CompiledCache(), p2.CompiledCache())
}
+14 -37
View File
@@ -12,7 +12,6 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/utils/json"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/stats"
yamlutil "github.com/projectdiscovery/nuclei/v3/pkg/utils/yaml"
"github.com/projectdiscovery/utils/errkit"
fileutil "github.com/projectdiscovery/utils/file"
"gopkg.in/yaml.v2"
@@ -29,42 +28,20 @@ type Parser struct {
sync.Mutex
}
var (
sharedParsedCacheOnce sync.Once
sharedParsedCache *Cache
)
var (
sharedCompiledCacheOnce sync.Once
sharedCompiledCache *Cache
)
// NewParser returns a new parser with a fresh cache
func NewParser() *Parser {
return &Parser{parsedTemplatesCache: NewCache(), compiledTemplatesCache: NewCache()}
p := &Parser{
parsedTemplatesCache: NewCache(),
compiledTemplatesCache: NewCache(),
}
return p
}
// NewParserWithParsedCache returns a parser using provided cache
func NewParserWithParsedCache(cache *Cache) *Parser {
return &Parser{parsedTemplatesCache: cache, compiledTemplatesCache: NewCache()}
}
// NewSharedParser returns a parser backed by a process-wide shared parsed cache.
// Safe for concurrent use since Cache is concurrency-safe.
func NewSharedParser() *Parser {
sharedParsedCacheOnce.Do(func() {
sharedParsedCache = NewCache()
})
return &Parser{parsedTemplatesCache: sharedParsedCache, compiledTemplatesCache: NewCache()}
}
// NewSharedParserWithCompiledCache returns a parser backed by process-wide shared
// parsed and compiled caches. Intended for scenarios where compiled executers
// can be safely reused across engines by copying option-bearing fields.
func NewSharedParserWithCompiledCache() *Parser {
sharedParsedCacheOnce.Do(func() { sharedParsedCache = NewCache() })
sharedCompiledCacheOnce.Do(func() { sharedCompiledCache = NewCache() })
return &Parser{parsedTemplatesCache: sharedParsedCache, compiledTemplatesCache: sharedCompiledCache}
return &Parser{
parsedTemplatesCache: cache,
compiledTemplatesCache: NewCache(),
}
}
// Cache returns the parsed templates cache
@@ -105,7 +82,7 @@ func (p *Parser) LoadTemplate(templatePath string, t any, extraTags []string, ca
t, templateParseError := p.ParseTemplate(templatePath, catalog)
if templateParseError != nil {
checkOpenFileError(templateParseError)
return false, errkit.Newf("Could not load template %s: %s", templatePath, templateParseError)
return false, ErrCouldNotLoadTemplate.Msgf(templatePath, templateParseError)
}
template, ok := t.(*Template)
if !ok {
@@ -119,13 +96,13 @@ func (p *Parser) LoadTemplate(templatePath string, t any, extraTags []string, ca
validationError := validateTemplateMandatoryFields(template)
if validationError != nil {
stats.Increment(SyntaxErrorStats)
return false, errkit.Newf("Could not load template %s: %s", templatePath, validationError)
return false, ErrCouldNotLoadTemplate.Msgf(templatePath, validationError)
}
ret, err := isTemplateInfoMetadataMatch(tagFilter, template, extraTags)
if err != nil {
checkOpenFileError(err)
return ret, errkit.Newf("Could not load template %s: %s", templatePath, err)
return ret, ErrCouldNotLoadTemplate.Msgf(templatePath, err)
}
// if template loaded then check the template for optional fields to add warnings
if ret {
@@ -133,7 +110,7 @@ func (p *Parser) LoadTemplate(templatePath string, t any, extraTags []string, ca
if validationWarning != nil {
stats.Increment(SyntaxWarningStats)
checkOpenFileError(validationWarning)
return ret, errkit.Newf("Could not load template %s: %s", templatePath, validationWarning)
return ret, ErrCouldNotLoadTemplate.Msgf(templatePath, validationWarning)
}
}
return ret, nil
+6 -6
View File
@@ -1,13 +1,13 @@
package templates
import (
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
)
var (
ErrMandatoryFieldMissingFmt = errkit.New("mandatory field is missing")
ErrInvalidField = errkit.New("invalid field format")
ErrWarningFieldMissing = errkit.New("field is missing")
ErrCouldNotLoadTemplate = errkit.New("could not load template")
ErrLoadedWithWarnings = errkit.New("loaded template with syntax warning")
ErrMandatoryFieldMissingFmt = errorutil.NewWithFmt("mandatory '%s' field is missing")
ErrInvalidField = errorutil.NewWithFmt("invalid field format for '%s' (allowed format is %s)")
ErrWarningFieldMissing = errorutil.NewWithFmt("field '%s' is missing")
ErrCouldNotLoadTemplate = errorutil.NewWithFmt("Could not load template %s: %s")
ErrLoadedWithWarnings = errorutil.NewWithFmt("Loaded template %s: with syntax warning : %s")
)
+2 -31
View File
@@ -3,7 +3,6 @@ package templates
import (
"errors"
"fmt"
"sync"
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/disk"
@@ -42,7 +41,7 @@ func TestLoadTemplate(t *testing.T) {
name: "emptyTemplate",
template: &Template{},
isValid: false,
expectedErr: errors.New("cause=\"Could not load template emptyTemplate: cause=\\\"mandatory 'name' field is missing\\\"\\ncause=\\\"mandatory 'author' field is missing\\\"\\ncause=\\\"mandatory 'id' field is missing\\\"\""),
expectedErr: errors.New("mandatory 'name' field is missing\nmandatory 'author' field is missing\nmandatory 'id' field is missing"),
},
{
name: "emptyNameWithInvalidID",
@@ -53,7 +52,7 @@ func TestLoadTemplate(t *testing.T) {
SeverityHolder: severity.Holder{Severity: severity.Medium},
},
},
expectedErr: errors.New("cause=\"Could not load template emptyNameWithInvalidID: cause=\\\"mandatory 'name' field is missing\\\"\\ncause=\\\"invalid field format for 'id' (allowed format is ^([a-zA-Z0-9]+[-_])*[a-zA-Z0-9]+$)\\\"\""),
expectedErr: errors.New("mandatory 'name' field is missing\ninvalid field format for 'id' (allowed format is ^([a-zA-Z0-9]+[-_])*[a-zA-Z0-9]+$)"),
},
{
name: "emptySeverity",
@@ -157,31 +156,3 @@ func TestLoadTemplate(t *testing.T) {
}
})
}
func TestNewSharedParserSharesCache(t *testing.T) {
p1 := NewSharedParser()
p2 := NewSharedParser()
if p1.Cache() != p2.Cache() {
t.Fatalf("expected shared cache instance")
}
}
func TestNewSharedParserConcurrency(t *testing.T) {
var wg sync.WaitGroup
const goroutines = 50
parsers := make([]*Parser, goroutines)
wg.Add(goroutines)
for i := 0; i < goroutines; i++ {
go func(i int) {
defer wg.Done()
parsers[i] = NewSharedParser()
}(i)
}
wg.Wait()
base := parsers[0].Cache()
for i := 1; i < goroutines; i++ {
if parsers[i].Cache() != base {
t.Fatalf("expected all parsers to share the same cache")
}
}
}
+5 -6
View File
@@ -5,7 +5,6 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
"github.com/projectdiscovery/nuclei/v3/pkg/utils"
"github.com/projectdiscovery/utils/errkit"
)
// validateTemplateMandatoryFields validates the mandatory fields of a template
@@ -16,17 +15,17 @@ func validateTemplateMandatoryFields(template *Template) error {
var validateErrors []error
if utils.IsBlank(info.Name) {
validateErrors = append(validateErrors, errkit.Newf("mandatory '%s' field is missing", "name"))
validateErrors = append(validateErrors, ErrMandatoryFieldMissingFmt.Msgf("name"))
}
if info.Authors.IsEmpty() {
validateErrors = append(validateErrors, errkit.Newf("mandatory '%s' field is missing", "author"))
validateErrors = append(validateErrors, ErrMandatoryFieldMissingFmt.Msgf("author"))
}
if template.ID == "" {
validateErrors = append(validateErrors, errkit.Newf("mandatory '%s' field is missing", "id"))
validateErrors = append(validateErrors, ErrMandatoryFieldMissingFmt.Msgf("id"))
} else if !ReTemplateID.MatchString(template.ID) {
validateErrors = append(validateErrors, errkit.Newf("invalid field format for '%s' (allowed format is %s)", "id", ReTemplateID.String()))
validateErrors = append(validateErrors, ErrInvalidField.Msgf("id", ReTemplateID.String()))
}
if len(validateErrors) > 0 {
@@ -54,7 +53,7 @@ func validateTemplateOptionalFields(template *Template) error {
var warnings []error
if template.Type() != types.WorkflowProtocol && utils.IsBlank(info.SeverityHolder.Severity.String()) {
warnings = append(warnings, errkit.Newf("field '%s' is missing", "severity"))
warnings = append(warnings, ErrWarningFieldMissing.Msgf("severity"))
}
if len(warnings) > 0 {
+2 -2
View File
@@ -4,7 +4,7 @@ import (
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/keys"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
)
// DefaultTemplateVerifiers contains the default template verifiers
@@ -34,7 +34,7 @@ func init() {
// AddSignerToDefault adds a signer to the default list of signers
func AddSignerToDefault(s *TemplateSigner) error {
if s == nil {
return errkit.New("signer is nil")
return errorutil.New("signer is nil")
}
DefaultTemplateVerifiers = append(DefaultTemplateVerifiers, s)
return nil
+3 -3
View File
@@ -16,7 +16,7 @@ import (
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
)
var (
@@ -82,13 +82,13 @@ func (t *TemplateSigner) Sign(data []byte, tmpl SignableTemplate) (string, error
arr := strings.SplitN(string(existingSignature), ":", 3)
if len(arr) == 2 {
// signature has no fragment
return "", errkit.New("re-signing code templates are not allowed for security reasons.")
return "", errorutil.NewWithTag("signer", "re-signing code templates are not allowed for security reasons.")
}
if len(arr) == 3 {
// signature has fragment verify if it is equal to current fragment
fragment := t.GetUserFragment()
if fragment != arr[2] {
return "", errkit.New("re-signing code templates are not allowed for security reasons.")
return "", errorutil.NewWithTag("signer", "re-signing code templates are not allowed for security reasons.")
}
}
}
+4 -4
View File
@@ -14,7 +14,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/templates/extensions"
"github.com/projectdiscovery/nuclei/v3/pkg/templates/signer"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
)
// Due to file references in sensitive fields of template
@@ -28,7 +28,7 @@ var (
_ = protocolstate.Init(defaultOpts)
_ = protocolinit.Init(defaultOpts)
})
ErrNotATemplate = errkit.New("given filePath is not a template", "tag", "signer")
ErrNotATemplate = errorutil.NewWithTag("signer", "given filePath is not a template")
)
// UseOptionsForSigner sets the options to use for signing templates
@@ -68,7 +68,7 @@ func SignTemplate(templateSigner *signer.TemplateSigner, templatePath string) er
template, bin, err := getTemplate(templatePath)
if err != nil {
return errkit.Wrap(err, "failed to get template from disk")
return errorutil.NewWithErr(err).Msgf("failed to get template from disk")
}
if len(template.Workflows) > 0 {
// signing workflows is not supported at least yet
@@ -100,7 +100,7 @@ func getTemplate(templatePath string) (*Template, []byte, error) {
}
template, err := ParseTemplateFromReader(bytes.NewReader(bin), nil, executerOpts)
if err != nil {
return nil, bin, errkit.Wrap(err, "failed to parse template")
return nil, bin, errorutil.NewWithErr(err).Msgf("failed to parse template")
}
return template, bin, nil
}
+8 -8
View File
@@ -24,7 +24,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/utils"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/json"
"github.com/projectdiscovery/nuclei/v3/pkg/workflows"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
fileutil "github.com/projectdiscovery/utils/file"
"go.uber.org/multierr"
"gopkg.in/yaml.v2"
@@ -325,14 +325,14 @@ func (template *Template) UnmarshalYAML(unmarshal func(interface{}) error) error
*template = Template(*alias)
if !ReTemplateID.MatchString(template.ID) {
return errkit.New("template id must match expression %v", ReTemplateID, "tag", "invalid_template")
return errorutil.New("template id must match expression %v", ReTemplateID).WithTag("invalid template")
}
info := template.Info
if utils.IsBlank(info.Name) {
return errkit.New("no template name field provided", "tag", "invalid_template")
return errorutil.New("no template name field provided").WithTag("invalid template")
}
if info.Authors.IsEmpty() {
return errkit.New("no template author field provided", "tag", "invalid_template")
return errorutil.New("no template author field provided").WithTag("invalid template")
}
if len(template.RequestsHTTP) > 0 || len(template.RequestsNetwork) > 0 {
@@ -340,10 +340,10 @@ func (template *Template) UnmarshalYAML(unmarshal func(interface{}) error) error
}
if len(alias.RequestsHTTP) > 0 && len(alias.RequestsWithHTTP) > 0 {
return errkit.New("use http or requests, both are not supported", "tag", "invalid_template")
return errorutil.New("use http or requests, both are not supported").WithTag("invalid template")
}
if len(alias.RequestsNetwork) > 0 && len(alias.RequestsWithTCP) > 0 {
return errkit.New("use tcp or network, both are not supported", "tag", "invalid_template")
return errorutil.New("use tcp or network, both are not supported").WithTag("invalid template")
}
if len(alias.RequestsWithHTTP) > 0 {
template.RequestsHTTP = alias.RequestsWithHTTP
@@ -361,7 +361,7 @@ func (template *Template) UnmarshalYAML(unmarshal func(interface{}) error) error
var tempmap yaml.MapSlice
err = unmarshal(&tempmap)
if err != nil {
return errkit.Wrapf(err, "failed to unmarshal multi protocol template %s", template.ID)
return errorutil.NewWithErr(err).Msgf("failed to unmarshal multi protocol template %s", template.ID)
}
arr := []string{}
for _, v := range tempmap {
@@ -545,7 +545,7 @@ func (template *Template) UnmarshalJSON(data []byte) error {
var tempMap map[string]interface{}
err = json.Unmarshal(data, &tempMap)
if err != nil {
return errkit.Wrapf(err, "failed to unmarshal multi protocol template %s", template.ID)
return errorutil.NewWithErr(err).Msgf("failed to unmarshal multi protocol template %s", template.ID)
}
arr := []string{}
for k := range tempMap {
-92
View File
@@ -1,92 +0,0 @@
package fuzzplayground
import (
"fmt"
"net/http"
"net/http/httptest"
"testing"
"github.com/stretchr/testify/require"
)
func TestSQLInjectionBehavior(t *testing.T) {
server := GetPlaygroundServer()
ts := httptest.NewServer(server)
defer ts.Close()
tests := []struct {
name string
path string
expectedStatus int
shouldContainAdmin bool
}{
{
name: "Normal request",
path: "/user/75/profile", // User 75 exists and has role 'user'
expectedStatus: 200,
shouldContainAdmin: false,
},
{
name: "SQL injection with OR 1=1",
path: "/user/75 OR 1=1/profile",
expectedStatus: 200, // Should work but might return first user (admin)
shouldContainAdmin: true, // Should return admin user data
},
{
name: "SQL injection with UNION",
path: "/user/1 UNION SELECT 1,'admin',30,'admin'/profile",
expectedStatus: 200,
shouldContainAdmin: true,
},
{
name: "Template payload test - OR True with 75",
path: "/user/75 OR True/profile", // What the template actually sends
expectedStatus: 200, // Actually works!
shouldContainAdmin: true, // Let's see if it returns admin
},
{
name: "Template payload test - OR True with 55 (non-existent)",
path: "/user/55 OR True/profile", // What the template should actually send
expectedStatus: 200, // Should work due to SQL injection
shouldContainAdmin: true, // Should return admin due to OR True
},
{
name: "Test original user 55 issue",
path: "/user/55/profile", // This should fail because user 55 doesn't exist
expectedStatus: 500,
shouldContainAdmin: false,
},
{
name: "Invalid ID - non-existent",
path: "/user/999/profile",
expectedStatus: 500, // Should error due to no such user
shouldContainAdmin: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
resp, err := http.Get(ts.URL + tt.path)
require.NoError(t, err)
defer func() {
if err := resp.Body.Close(); err != nil {
t.Logf("Failed to close response body: %v", err)
}
}()
require.Equal(t, tt.expectedStatus, resp.StatusCode)
body := make([]byte, 1024)
n, _ := resp.Body.Read(body)
bodyStr := string(body[:n])
fmt.Printf("Request: %s\n", tt.path)
fmt.Printf("Status: %d\n", resp.StatusCode)
fmt.Printf("Response: %s\n\n", bodyStr)
if tt.shouldContainAdmin {
require.Contains(t, bodyStr, "admin")
}
})
}
}
+5 -5
View File
@@ -16,7 +16,7 @@ import (
"github.com/kitabisa/go-ci"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
fileutil "github.com/projectdiscovery/utils/file"
mapsutil "github.com/projectdiscovery/utils/maps"
"go.uber.org/multierr"
@@ -24,7 +24,7 @@ import (
var (
// ErrInvalidRequestID is a request id error
ErrInvalidRequestID = errkit.New("invalid request id provided")
ErrInvalidRequestID = errorutil.NewWithFmt("[%s] invalid request id '%s' provided")
)
// ProtoOptions are options that can be passed to flow protocol callback
@@ -256,12 +256,12 @@ func (f *FlowExecutor) ExecuteWithResults(ctx *scan.ScanContext) error {
f.reconcileProgress()
if err != nil {
ctx.LogError(err)
return errkit.Wrapf(err, "failed to execute flow\n%v\n", f.options.Flow)
return errorutil.NewWithErr(err).Msgf("failed to execute flow\n%v\n", f.options.Flow)
}
runtimeErr := f.GetRuntimeErrors()
if runtimeErr != nil {
ctx.LogError(runtimeErr)
return errkit.Wrap(runtimeErr, "got following errors while executing flow")
return errorutil.NewWithErr(runtimeErr).Msgf("got following errors while executing flow")
}
return nil
@@ -283,7 +283,7 @@ func (f *FlowExecutor) reconcileProgress() {
func (f *FlowExecutor) GetRuntimeErrors() error {
errs := []error{}
for proto, err := range f.allErrs.GetAll() {
errs = append(errs, errkit.Wrapf(err, "failed to execute %v protocol", proto))
errs = append(errs, errorutil.NewWithErr(err).Msgf("failed to execute %v protocol", proto))
}
return multierr.Combine(errs...)
}
+1 -2
View File
@@ -7,7 +7,6 @@ import (
"github.com/Mzack9999/goja"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols"
"github.com/projectdiscovery/utils/errkit"
mapsutil "github.com/projectdiscovery/utils/maps"
)
@@ -62,7 +61,7 @@ func (f *FlowExecutor) requestExecutor(runtime *goja.Runtime, reqMap mapsutil.Ma
if !ok {
f.ctx.LogError(fmt.Errorf("[%v] invalid request id '%s' provided", f.options.TemplateID, id))
// compile error
if err := f.allErrs.Set(opts.protoName+":"+id, errkit.Newf("[%s] invalid request id '%s' provided", f.options.TemplateID, id)); err != nil {
if err := f.allErrs.Set(opts.protoName+":"+id, ErrInvalidRequestID.Msgf(f.options.TemplateID, id)); err != nil {
f.ctx.LogError(fmt.Errorf("failed to store flow runtime errors got %v", err))
}
return matcherStatus.Load()
+5 -5
View File
@@ -14,7 +14,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
"github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
"github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
fileutil "github.com/projectdiscovery/utils/file"
folderutil "github.com/projectdiscovery/utils/folder"
unitutils "github.com/projectdiscovery/utils/unit"
@@ -830,7 +830,7 @@ func (options *Options) defaultLoadHelperFile(helperFile, templatePath string, c
}
f, err := os.Open(helperFile)
if err != nil {
return nil, errkit.Wrapf(err, "could not open file %v", helperFile)
return nil, errorutil.NewWithErr(err).Msgf("could not open file %v", helperFile)
}
return f, nil
}
@@ -855,12 +855,12 @@ func (o *Options) GetValidAbsPath(helperFilePath, templatePath string) (string,
// CleanPath resolves using CWD and cleans the path
helperFilePath, err = fileutil.CleanPath(helperFilePath)
if err != nil {
return "", errkit.Wrapf(err, "could not clean helper file path %v", helperFilePath)
return "", errorutil.NewWithErr(err).Msgf("could not clean helper file path %v", helperFilePath)
}
templatePath, err = fileutil.CleanPath(templatePath)
if err != nil {
return "", errkit.Wrapf(err, "could not clean template path %v", templatePath)
return "", errorutil.NewWithErr(err).Msgf("could not clean template path %v", templatePath)
}
// As per rule 2, if template and helper file exist in same directory or helper file existed in any child dir of template dir
@@ -871,7 +871,7 @@ func (o *Options) GetValidAbsPath(helperFilePath, templatePath string) (string,
}
// all other cases are denied
return "", errkit.Newf("access to helper file %v denied", helperFilePath)
return "", errorutil.New("access to helper file %v denied", helperFilePath)
}
// SetExecutionID sets the execution ID for the options
-74
View File
@@ -1,74 +0,0 @@
package testing
import (
"context"
"os"
"path/filepath"
"testing"
nuclei "github.com/projectdiscovery/nuclei/v3/lib"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
)
// BenchmarkChunkEngines simulates the aurora agent pattern where multiple chunks
// create engines against the same template set. It compares shared vs non-shared parser.
// We only load templates (no scan execution) to stress parsing/compile memory like the loader path.
func BenchmarkChunkEngines(b *testing.B) {
templatesDir := config.DefaultConfig.TemplatesDirectory
if fi, err := os.Stat(filepath.Clean(templatesDir)); err != nil || !fi.IsDir() {
b.Skipf("templates directory not available: %s", templatesDir)
return
}
// number of synthetic "chunks" (engines) to simulate
const chunks = 20
b.Run("no_shared", func(b *testing.B) {
_ = os.Unsetenv("NUCLEI_USE_SHARED_PARSER")
b.ReportAllocs()
for i := 0; i < b.N; i++ {
// create engines like separate chunks would
engines := make([]*nuclei.NucleiEngine, 0, chunks)
for c := 0; c < chunks; c++ {
ne, err := nuclei.NewNucleiEngineCtx(context.Background())
if err != nil {
b.Fatalf("engine error: %v", err)
}
engines = append(engines, ne)
}
// load templates on each engine (same set)
for _, ne := range engines {
if err := ne.LoadAllTemplates(); err != nil {
b.Fatalf("load templates error: %v", err)
}
}
for _, ne := range engines {
ne.Close()
}
}
})
b.Run("shared", func(b *testing.B) {
_ = os.Setenv("NUCLEI_USE_SHARED_PARSER", "1")
b.Cleanup(func() { _ = os.Unsetenv("NUCLEI_USE_SHARED_PARSER") })
b.ReportAllocs()
for i := 0; i < b.N; i++ {
engines := make([]*nuclei.NucleiEngine, 0, chunks)
for c := 0; c < chunks; c++ {
ne, err := nuclei.NewNucleiEngineCtx(context.Background())
if err != nil {
b.Fatalf("engine error: %v", err)
}
engines = append(engines, ne)
}
for _, ne := range engines {
if err := ne.LoadAllTemplates(); err != nil {
b.Fatalf("load templates error: %v", err)
}
}
for _, ne := range engines {
ne.Close()
}
}
})
}
-62
View File
@@ -1,62 +0,0 @@
package testing
import (
"context"
"os"
"path/filepath"
"testing"
nuclei "github.com/projectdiscovery/nuclei/v3/lib"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
)
// BenchmarkSharedParser benchmarks LoadAllTemplates with and without shared parsed cache.
// It skips if nuclei-templates directory is not present to avoid fetching during benchmarks.
func BenchmarkSharedParser(b *testing.B) {
templatesDir := config.DefaultConfig.TemplatesDirectory
if fi, err := os.Stat(filepath.Clean(templatesDir)); err != nil || !fi.IsDir() {
b.Skipf("templates directory not available: %s", templatesDir)
return
}
b.Run("no_shared", func(b *testing.B) {
_ = os.Unsetenv("NUCLEI_USE_SHARED_PARSER")
b.ReportAllocs()
for i := 0; i < b.N; i++ {
ne, err := nuclei.NewNucleiEngineCtx(context.Background())
if err != nil {
b.Fatalf("engine error: %v", err)
}
if err := ne.LoadAllTemplates(); err != nil {
b.Fatalf("load templates error: %v", err)
}
ne.Close()
}
})
b.Run("shared", func(b *testing.B) {
_ = os.Setenv("NUCLEI_USE_SHARED_PARSER", "1")
b.Cleanup(func() { _ = os.Unsetenv("NUCLEI_USE_SHARED_PARSER") })
// warm up shared cache once
warm, err := nuclei.NewNucleiEngineCtx(context.Background())
if err != nil {
b.Fatalf("warm engine error: %v", err)
}
if err := warm.LoadAllTemplates(); err != nil {
b.Fatalf("warm load error: %v", err)
}
warm.Close()
b.ReportAllocs()
for i := 0; i < b.N; i++ {
ne, err := nuclei.NewNucleiEngineCtx(context.Background())
if err != nil {
b.Fatalf("engine error: %v", err)
}
if err := ne.LoadAllTemplates(); err != nil {
b.Fatalf("load templates error: %v", err)
}
ne.Close()
}
})
}
-184
View File
@@ -1,184 +0,0 @@
package main
import (
"context"
"flag"
"fmt"
"os"
"runtime"
"strings"
"sync"
"time"
nuclei "github.com/projectdiscovery/nuclei/v3/lib"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/templates"
)
func memMB() uint64 {
var m runtime.MemStats
runtime.ReadMemStats(&m)
return m.Alloc / (1024 * 1024)
}
func snapshotCaches(label string, engines []*nuclei.NucleiEngine) {
fmt.Println(label)
max := 3
if len(engines) < max {
max = len(engines)
}
var base *templates.Parser
for i := 0; i < max; i++ {
p := engines[i].GetParser()
if base == nil {
base = p
}
fmt.Printf(" engine[%d]: parsed_cache_ptr=%p compiled_cache_ptr=%p parsed_count=%d compiled_count=%d\n", i, p.Cache(), p.CompiledCache(), p.ParsedCount(), p.CompiledCount())
}
equalParsed := true
equalCompiled := true
for i := 1; i < len(engines); i++ {
if engines[i].GetParser().Cache() != base.Cache() {
equalParsed = false
}
if engines[i].GetParser().CompiledCache() != base.CompiledCache() {
equalCompiled = false
}
}
fmt.Println(" parsed cache shared across engines:", equalParsed)
fmt.Println(" compiled cache shared across engines:", equalCompiled)
}
func runEngines(chunks int, targets []string, execute bool) (time.Duration, error) {
start := time.Now()
engines := make([]*nuclei.NucleiEngine, 0, chunks)
for i := 0; i < chunks; i++ {
ne, err := nuclei.NewNucleiEngineCtx(context.Background())
if err != nil {
return 0, fmt.Errorf("engine create: %w", err)
}
engines = append(engines, ne)
}
// Load templates
for _, ne := range engines {
if err := ne.LoadAllTemplates(); err != nil {
return 0, fmt.Errorf("load templates: %w", err)
}
}
if execute {
// Execute scans concurrently with a global 60s timeout to prevent long runs
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
var wg sync.WaitGroup
for _, ne := range engines {
ne := ne
ne.LoadTargets(targets, false)
wg.Add(1)
go func() {
defer wg.Done()
// ignore callback output
_ = ne.ExecuteCallbackWithCtx(ctx)
}()
}
wg.Wait()
}
for _, ne := range engines {
ne.Close()
}
return time.Since(start), nil
}
func main() {
var chunks int
var targetsCSV string
var execute bool
flag.IntVar(&chunks, "chunks", 10, "number of simulated chunk engines")
flag.StringVar(&targetsCSV, "targets", "https://scanme.sh,https://honey.scanme.sh", "comma-separated targets")
flag.BoolVar(&execute, "execute", false, "execute scans after loading templates (default: false)")
flag.Parse()
// Ensure templates directory exists
templatesDir := config.DefaultConfig.TemplatesDirectory
if fi, err := os.Stat(templatesDir); err != nil || !fi.IsDir() {
fmt.Printf("templates directory not found: %s\n", templatesDir)
os.Exit(1)
}
targets := []string{}
for _, t := range strings.Split(targetsCSV, ",") {
t = strings.TrimSpace(t)
if t != "" {
targets = append(targets, t)
}
}
fmt.Println("=== Agent-sim: without shared caches ===")
_ = os.Unsetenv("NUCLEI_USE_SHARED_COMPILED")
runtime.GC()
before := memMB()
// create engines to snapshot cache sharing behavior
enginesNS := make([]*nuclei.NucleiEngine, 0, chunks)
for i := 0; i < chunks; i++ {
ne, err := nuclei.NewNucleiEngineCtx(context.Background())
if err != nil {
fmt.Println("error:", err)
os.Exit(1)
}
enginesNS = append(enginesNS, ne)
}
for _, ne := range enginesNS {
if err := ne.LoadAllTemplates(); err != nil {
fmt.Println("error:", err)
os.Exit(1)
}
}
snapshotCaches("cache state (no_shared):", enginesNS)
for _, ne := range enginesNS {
ne.Close()
}
durNoShared, err := runEngines(chunks, targets, execute)
if err != nil {
fmt.Println("error:", err)
os.Exit(1)
}
runtime.GC()
after := memMB()
fmt.Printf("no_shared: duration=%s heap_before=%dMB heap_after=%dMB\n", durNoShared, before, after)
fmt.Println("\n=== Agent-sim: with shared compiled cache ===")
_ = os.Setenv("NUCLEI_USE_SHARED_COMPILED", "1")
runtime.GC()
before = memMB()
enginesSC := make([]*nuclei.NucleiEngine, 0, chunks)
for i := 0; i < chunks; i++ {
ne, err := nuclei.NewNucleiEngineCtx(context.Background())
if err != nil {
fmt.Println("error:", err)
os.Exit(1)
}
enginesSC = append(enginesSC, ne)
}
for _, ne := range enginesSC {
if err := ne.LoadAllTemplates(); err != nil {
fmt.Println("error:", err)
os.Exit(1)
}
}
snapshotCaches("cache state (shared-compiled):", enginesSC)
for _, ne := range enginesSC {
ne.Close()
}
durSharedCompiled, err := runEngines(chunks, targets, execute)
if err != nil {
fmt.Println("error:", err)
os.Exit(1)
}
runtime.GC()
after = memMB()
fmt.Printf("shared_compiled: duration=%s heap_before=%dMB heap_after=%dMB\n", durSharedCompiled, before, after)
fmt.Println("\nDone.")
}