Compare commits

...
Author SHA1 Message Date
knakul853 eeedf6cce6 Implement shared parser functionality and related tests
- Added shared parsed and compiled caches to the parser for improved performance across engines.
- Introduced `getSharedParser` function to initialize a shared parser instance.
- Updated `NucleiEngine` initialization to optionally use shared caches based on environment variable.
- Added tests to validate shared parser behavior and benchmark performance with and without shared caches.
- Implemented sanitization of options in the compiled cache to prevent engine-scoped state retention.
2025-09-04 00:22:57 +05:30
dependabot[bot]andMzack9999 32dfeacd9d chore(deps): bump the modules group across 1 directory with 11 updates (#6438)
* chore(deps): bump the modules group across 1 directory with 11 updates

Bumps the modules group with 10 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.6` | `0.4.7` |
| [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.92` | `0.0.93` |
| [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns) | `1.0.105` | `1.0.106` |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.120` | `1.0.121` |
| [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.5.0` | `0.5.1` |
| [github.com/projectdiscovery/gozero](https://github.com/projectdiscovery/gozero) | `0.0.3` | `0.1.0` |
| [github.com/projectdiscovery/ratelimit](https://github.com/projectdiscovery/ratelimit) | `0.0.81` | `0.0.82` |
| [github.com/projectdiscovery/tlsx](https://github.com/projectdiscovery/tlsx) | `1.1.9` | `1.2.0` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.37` | `0.2.43` |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.1.27` | `1.1.33` |



Updates `github.com/projectdiscovery/fastdialer` from 0.4.6 to 0.4.7
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases)
- [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.6...v0.4.7)

Updates `github.com/projectdiscovery/hmap` from 0.0.92 to 0.0.93
- [Release notes](https://github.com/projectdiscovery/hmap/releases)
- [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.92...v0.0.93)

Updates `github.com/projectdiscovery/retryabledns` from 1.0.105 to 1.0.106
- [Release notes](https://github.com/projectdiscovery/retryabledns/releases)
- [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.105...v1.0.106)

Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.120 to 1.0.121
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases)
- [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.120...v1.0.121)

Updates `github.com/projectdiscovery/dsl` from 0.5.0 to 0.5.1
- [Release notes](https://github.com/projectdiscovery/dsl/releases)
- [Commits](https://github.com/projectdiscovery/dsl/compare/v0.5.0...v0.5.1)

Updates `github.com/projectdiscovery/gozero` from 0.0.3 to 0.1.0
- [Release notes](https://github.com/projectdiscovery/gozero/releases)
- [Commits](https://github.com/projectdiscovery/gozero/compare/v0.0.3...v0.1.0)

Updates `github.com/projectdiscovery/networkpolicy` from 0.1.20 to 0.1.21
- [Release notes](https://github.com/projectdiscovery/networkpolicy/releases)
- [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.20...v0.1.21)

Updates `github.com/projectdiscovery/ratelimit` from 0.0.81 to 0.0.82
- [Release notes](https://github.com/projectdiscovery/ratelimit/releases)
- [Commits](https://github.com/projectdiscovery/ratelimit/compare/v0.0.81...v0.0.82)

Updates `github.com/projectdiscovery/tlsx` from 1.1.9 to 1.2.0
- [Release notes](https://github.com/projectdiscovery/tlsx/releases)
- [Changelog](https://github.com/projectdiscovery/tlsx/blob/main/.goreleaser.yml)
- [Commits](https://github.com/projectdiscovery/tlsx/compare/v1.1.9...v1.2.0)

Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.37 to 0.2.43
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.37...v0.2.43)

Updates `github.com/projectdiscovery/cdncheck` from 1.1.27 to 1.1.33
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml)
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.1.27...v1.1.33)

---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/fastdialer
  dependency-version: 0.4.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/hmap
  dependency-version: 0.0.93
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryabledns
  dependency-version: 1.0.106
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryablehttp-go
  dependency-version: 1.0.121
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/dsl
  dependency-version: 0.5.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/gozero
  dependency-version: 0.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/networkpolicy
  dependency-version: 0.1.21
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/ratelimit
  dependency-version: 0.0.82
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/tlsx
  dependency-version: 1.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
  dependency-version: 0.2.43
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/cdncheck
  dependency-version: 1.1.33
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: modules
...

Signed-off-by: dependabot[bot] <[email protected]>

* bump

* httpx dev

* mod tidy

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Mzack9999 <[email protected]>
2025-09-01 17:52:46 +07:00
PDTeamX 48c389b063 link update 2025-08-31 03:16:13 +05:30
PDTeamX 36b4f68eec misc hyperlink update 2025-08-31 03:13:14 +05:30
PDTeamX af7b2f166e issue / discussion template update 2025-08-31 03:10:51 +05:30
Dogan Can Bakir b25937b310 Merge pull request #6425 from projectdiscovery/bump_httpx_version
bump httpx version
2025-08-28 10:04:35 +03:00
Dogan Can Bakir 100d6528f5 Merge branch 'dev' into bump_httpx_version 2025-08-28 08:55:23 +03:00
cui d76187f99a Refactor to use reflect.TypeFor (#6428) 2025-08-27 22:31:04 +05:30
Lorenzo SusiniandMzack9999 8194fabcf8 test(reporting/exporters/mongo): add mongo integration test with test… (#6237)
* test(reporting/exporters/mongo): add mongo integration test with testcontainer-go module

Signed-off-by: Lorenzo Susini <[email protected]>

* execute exportes only on linux

---------

Signed-off-by: Lorenzo Susini <[email protected]>
Co-authored-by: Mzack9999 <[email protected]>
2025-08-27 04:25:31 +05:30
dependabot[bot] 5063af46b1 chore(deps): bump github.com/go-viper/mapstructure/v2
Bumps the go_modules group with 1 update in the / directory: [github.com/go-viper/mapstructure/v2](https://github.com/go-viper/mapstructure).


Updates `github.com/go-viper/mapstructure/v2` from 2.3.0 to 2.4.0
- [Release notes](https://github.com/go-viper/mapstructure/releases)
- [Changelog](https://github.com/go-viper/mapstructure/blob/main/CHANGELOG.md)
- [Commits](https://github.com/go-viper/mapstructure/compare/v2.3.0...v2.4.0)

---
updated-dependencies:
- dependency-name: github.com/go-viper/mapstructure/v2
  dependency-version: 2.4.0
  dependency-type: indirect
  dependency-group: go_modules
...

Signed-off-by: dependabot[bot] <[email protected]>
2025-08-26 07:08:27 +00:00
Doğan Can Bakır 776cb4fcf2 bump httpx version 2025-08-26 10:04:36 +03:00
PDTeamX 0f7b33cebf limited test, instead of all 2025-08-25 13:56:03 +05:30
Dwi Siswantoandyusei-wy a1b5a0ed99 fix(fuzz): handles duplicate multipart form field names (#6404)
* fix: handle duplicate field names in multipart form encoding

* fix(fuzz): handles `[]any` type in `*MultiPartForm.Encode`

Signed-off-by: Dwi Siswanto <[email protected]>

* test(fuzz): adds panic recovery & display encoded out

Signed-off-by: Dwi Siswanto <[email protected]>

* fix(fuzz): incorrectly treated mixed type field

in `*MultiPartForm.Encode`

Signed-off-by: Dwi Siswanto <[email protected]>

* test(fuzz): refactor compare w decoded instead

Signed-off-by: Dwi Siswanto <[email protected]>

* chore(fuzz): prealloc for `[]any` type

Signed-off-by: Dwi Siswanto <[email protected]>

* fix(fuzz): treats nil value as empty string

Signed-off-by: Dwi Siswanto <[email protected]>

* chore(fuzz): rm early error return for non-array file

Signed-off-by: Dwi Siswanto <[email protected]>

* test(fuzz): adds `TestMultiPartFormFileUpload` test

Signed-off-by: Dwi Siswanto <[email protected]>

---------

Signed-off-by: Dwi Siswanto <[email protected]>
Co-authored-by: yusei-wy <[email protected]>
2025-08-25 13:42:51 +05:30
Tarun Koyalwar 19247ae74b Path-Based Fuzzing SQL fix (#6400)
* setup claude

* migrate to using errkit

* fix unused imports + lint errors

* update settings.json

* fix url encoding issue

* fix lint error

* fix the path fuzzing component

* fix lint error
2025-08-25 13:36:58 +05:30
93 changed files with 2953 additions and 599 deletions

No files matched your search

+35
View File
@@ -0,0 +1,35 @@
{
"permissions": {
"allow": [
"Bash(find:*)",
"Bash(mkdir:*)",
"Bash(cp:*)",
"Bash(ls:*)",
"Bash(make:*)",
"Bash(go:*)",
"Bash(golangci-lint:*)",
"Bash(git merge:*)",
"Bash(git add:*)",
"Bash(git commit:*)",
"Bash(git push:*)",
"Bash(git pull:*)",
"Bash(git fetch:*)",
"Bash(git checkout:*)",
"WebFetch(*)",
"Write(*)",
"WebSearch(*)",
"MultiEdit(*)",
"Edit(*)",
"Bash(gh:*)",
"Bash(grep:*)",
"Bash(tree:*)",
"Bash(./nuclei:*)",
"WebFetch(domain:github.com)"
],
"deny": [
"Bash(make run:*)",
"Bash(./bin/nuclei:*)"
],
"defaultMode": "acceptEdits"
}
}
+76
View File
@@ -0,0 +1,76 @@
# Nuclei Discussion Guidelines
## Before Creating a Discussion
1. **Search existing discussions and issues** to avoid duplicates
2. **Check the documentation** and README first
3. **Browse the FAQ** and common questions
## Bug Reports in Discussions
When reporting a bug in [Q&A Discussions](https://github.com/projectdiscovery/nuclei/discussions/categories/q-a), please include:
### Required Information:
- **Clear title** with `[BUG]` prefix (e.g., "[BUG] Nuclei crashes when...")
- **Current behavior** - What's happening now?
- **Expected behavior** - What should happen instead?
- **Steps to reproduce** - Commands or actions that trigger the issue
- **Environment details**:
- OS and version
- Nuclei version (`nuclei -version`)
- Go version (if installed via `go install`)
- **Log output** - Run with `-verbose` or `-debug` for detailed logs
- **Redact sensitive information** - Remove target URLs, credentials, etc.
### After Discussion:
- Maintainers will review and validate the bug report
- Valid bugs will be converted to issues with proper labels and tracking
- Questions and misconfigurations will be resolved in the discussion
## Feature Requests in Discussions
When requesting a feature in [Ideas Discussions](https://github.com/projectdiscovery/nuclei/discussions/categories/ideas), please include:
### Required Information:
- **Clear title** with `[FEATURE]` prefix (e.g., "[FEATURE] Add support for...")
- **Feature description** - What do you want to be added?
- **Use case** - Why is this feature needed? What problem does it solve?
- **Implementation ideas** - If you have suggestions on how it could work
- **Alternatives considered** - What other solutions have you thought about?
### After Discussion:
- Community and maintainers will discuss the feasibility
- Popular and viable features will be converted to issues
- Similar features may be grouped together
- Rejected features will be explained in the discussion
## Getting Help
For general questions, troubleshooting, and "how-to" topics:
- Use [Q&A Discussions](https://github.com/projectdiscovery/nuclei/discussions/categories/q-a)
- Join the [Discord server](https://discord.gg/projectdiscovery) #nuclei channel
- Check existing discussions for similar questions
## Discussion to Issue Conversion Process
Only maintainers can convert discussions to issues. The process:
1. **Validation** - Maintainers review the discussion for completeness and validity
2. **Classification** - Determine if it's a bug, feature, enhancement, etc.
3. **Issue creation** - Create a properly formatted issue with appropriate labels
4. **Linking** - Link the issue back to the original discussion
5. **Resolution** - Mark the discussion as resolved or close it
This process ensures:
- High-quality issues that are actionable
- Proper triage and labeling
- Reduced noise in the issue tracker
- Community involvement in the validation process
## Why This Process?
- **Better organization** - Issues contain only validated, actionable items
- **Community input** - Discussions allow for community feedback before escalation
- **Quality control** - Maintainers ensure proper formatting and information
- **Reduced maintenance** - Fewer invalid or duplicate issues to manage
- **Clear separation** - Questions vs. actual bugs/features are clearly distinguished
+16 -8
View File
@@ -2,14 +2,22 @@ blank_issues_enabled: false
contact_links:
- name: Ask an question / advise on using nuclei
url: https://github.com/projectdiscovery/nuclei/discussions/categories/q-a
about: Ask a question or request support for using nuclei
- name: 🐛 Report a Bug (Start with Discussion)
url: https://github.com/orgs/projectdiscovery/discussions/new?category=q-a
about: Start by reporting your issue in discussions for proper triage. Issues will be created after review to avoid duplicate/invalid reports.
- name: Share idea / feature to discuss for nuclei
url: https://github.com/projectdiscovery/nuclei/discussions/categories/ideas
about: Share idea / feature to discuss for nuclei
- name: 💡 Request a Feature (Start with Discussion)
url: https://github.com/orgs/projectdiscovery/discussions/new?category=ideas
about: Share your feature idea in discussions first. This helps validate and refine the request before creating an issue.
- name: Connect with PD Team (Discord)
- name: ❓ Ask Questions / Get Help
url: https://github.com/orgs/projectdiscovery/discussions
about: Get help and ask questions about using Nuclei. Many questions don't require issues.
- name: 🔍 Browse Existing Issues
url: https://github.com/projectdiscovery/nuclei/issues
about: Check existing issues to see if your problem has already been reported or is being worked on.
- name: 💬 Connect with PD Team (Discord)
url: https://discord.gg/projectdiscovery
about: Connect with PD Team for direct communication
about: Join our Discord for real-time discussions and community support on the #nuclei channel.
@@ -0,0 +1,45 @@
# Issue Template References
## Overview
This folder contains the preserved issue templates that are **not** directly accessible to users. These templates serve as references for maintainers when converting discussions to issues.
## New Workflow
### For Users:
1. **All reports start in Discussions** - Users cannot create issues directly
2. Bug reports go to [Q&A Discussions](https://github.com/projectdiscovery/nuclei/discussions/categories/q-a)
3. Feature requests go to [Ideas Discussions](https://github.com/projectdiscovery/nuclei/discussions/categories/ideas)
4. This helps filter out duplicate questions, invalid reports, and ensures proper triage
### For Maintainers:
1. **Review discussions** in both Q&A and Ideas categories
2. **Validate the reports** - ensure they're actual bugs/valid feature requests
3. **Use reference templates** when converting discussions to issues:
- Copy content from `bug-report-reference.yml` or `feature-request-reference.yml`
- Create a new issue manually with the appropriate template structure
- Link back to the original discussion
- Close the discussion or mark it as resolved
## Benefits
- **Better triage**: Avoid cluttering issues with questions and invalid reports
- **Community involvement**: Discussions allow for community input before creating issues
- **Quality control**: Maintainers can ensure issues follow proper format and contain necessary information
- **Reduced noise**: Only validated, actionable items become issues
## Reference Templates
- `bug-report-reference.yml` - Use when converting bug reports from discussions to issues
- `feature-request-reference.yml` - Use when converting feature requests from discussions to issues
## Converting a Discussion to Issue
1. Identify a valid discussion that needs to become an issue
2. Go to the main repository's Issues tab
3. Click "New Issue"
4. Manually create the issue using the reference template structure
5. Include all relevant information from the discussion
6. Add a comment linking back to the original discussion
7. Apply appropriate labels
8. Close or mark the discussion as resolved with a link to the created issue
+2
View File
@@ -28,6 +28,8 @@
/scrapefunc
/scrapefuncs
/tsgen
/integration_tests/integration-test
/integration_tests/nuclei
# Templates
/*.yaml
+83
View File
@@ -0,0 +1,83 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
Nuclei is a modern, high-performance vulnerability scanner built in Go that leverages YAML-based templates for customizable vulnerability detection. It supports multiple protocols (HTTP, DNS, TCP, SSL, WebSocket, WHOIS, JavaScript, Code) and is designed for zero false positives through real-world condition simulation.
## Development Commands
### Building and Testing
- `make build` - Build the main nuclei binary to ./bin/nuclei
- `make test` - Run unit tests with race detection
- `make integration` - Run integration tests (builds and runs test suite)
- `make functional` - Run functional tests
- `make vet` - Run go vet for code analysis
- `make tidy` - Clean up go modules
### Validation and Linting
- `make template-validate` - Validate nuclei templates using the built binary
- `go fmt ./...` - Format Go code
- `go vet ./...` - Static analysis
### Development Tools
- `make devtools-all` - Build all development tools (bindgen, tsgen, scrapefuncs)
- `make jsupdate-all` - Update JavaScript bindings and TypeScript definitions
- `make docs` - Generate documentation
- `make memogen` - Generate memoization code for JavaScript libraries
### Testing Specific Components
- Run single test: `go test -v ./pkg/path/to/package -run TestName`
- Integration tests are in `integration_tests/` and can be run via `make integration`
## Architecture Overview
### Core Components
- **cmd/nuclei** - Main CLI entry point with flag parsing and configuration
- **internal/runner** - Core runner that orchestrates the entire scanning process
- **pkg/core** - Execution engine with work pools and template clustering
- **pkg/templates** - Template parsing, compilation, and management
- **pkg/protocols** - Protocol implementations (HTTP, DNS, Network, etc.)
- **pkg/operators** - Matching and extraction logic (matchers/extractors)
- **pkg/catalog** - Template discovery and loading from disk/remote sources
### Protocol Architecture
Each protocol (HTTP, DNS, Network, etc.) implements:
- Request interface with Compile(), ExecuteWithResults(), Match(), Extract() methods
- Operators embedding for matching/extraction functionality
- Protocol-specific request building and execution logic
### Template System
- Templates are YAML files defining vulnerability detection logic
- Compiled into executable requests with operators (matchers/extractors)
- Support for workflows (multi-step template execution)
- Template clustering optimizes identical requests across multiple templates
### Key Execution Flow
1. Template loading and compilation via pkg/catalog/loader
2. Input provider setup for targets
3. Engine creation with work pools for concurrency
4. Template execution with result collection via operators
5. Output writing and reporting integration
### JavaScript Integration
- Custom JavaScript runtime for code protocol templates
- Auto-generated bindings in pkg/js/generated/
- Library implementations in pkg/js/libs/
- Development tools for binding generation in pkg/js/devtools/
## Template Development
- Templates located in separate nuclei-templates repository
- YAML format with info, requests, and operators sections
- Support for multiple protocol types in single template
- Built-in DSL functions for dynamic content generation
- Template validation available via `make template-validate`
## Key Directories
- **lib/** - SDK for embedding nuclei as a library
- **examples/** - Usage examples for different scenarios
- **integration_tests/** - Integration test suite with protocol-specific tests
- **pkg/fuzz/** - Fuzzing engine and DAST capabilities
- **pkg/input/** - Input processing for various formats (Burp, OpenAPI, etc.)
- **pkg/reporting/** - Result export and issue tracking integrations
+4
View File
@@ -150,6 +150,10 @@ template-validate:
-et .github/ \
-et helpers/payloads/ \
-et http/technologies \
-t dns \
-t ssl \
-t network \
-t http/exposures \
-ept code
./bin/nuclei -validate \
-w workflows \
+104
View File
@@ -0,0 +1,104 @@
package main
import (
"context"
"fmt"
"log"
"time"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/mongo"
"github.com/testcontainers/testcontainers-go"
mongocontainer "github.com/testcontainers/testcontainers-go/modules/mongodb"
osutil "github.com/projectdiscovery/utils/os"
mongoclient "go.mongodb.org/mongo-driver/mongo"
mongooptions "go.mongodb.org/mongo-driver/mongo/options"
)
const (
dbName = "test"
dbImage = "mongo:8"
)
var exportersTestCases = []TestCaseInfo{
{Path: "exporters/mongo", TestCase: &mongoExporter{}, DisableOn: func() bool {
return osutil.IsWindows() || osutil.IsOSX()
}},
}
type mongoExporter struct{}
func (m *mongoExporter) Execute(filepath string) error {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
defer cancel()
// Start a MongoDB container
mongodbContainer, err := mongocontainer.Run(ctx, dbImage)
defer func() {
if err := testcontainers.TerminateContainer(mongodbContainer); err != nil {
log.Printf("failed to terminate container: %s", err)
}
}()
if err != nil {
return fmt.Errorf("failed to start container: %w", err)
}
connString, err := mongodbContainer.ConnectionString(ctx)
if err != nil {
return fmt.Errorf("failed to get connection string for MongoDB container: %s", err)
}
connString = connString + dbName
// Create a MongoDB exporter and write a test result to the database
opts := mongo.Options{
ConnectionString: connString,
CollectionName: "test",
BatchSize: 1, // Ensure we write the result immediately
}
exporter, err := mongo.New(&opts)
if err != nil {
return fmt.Errorf("failed to create MongoDB exporter: %s", err)
}
defer func() {
if err := exporter.Close(); err != nil {
fmt.Printf("failed to close exporter: %s\n", err)
}
}()
res := &output.ResultEvent{
Request: "test request",
Response: "test response",
}
err = exporter.Export(res)
if err != nil {
return fmt.Errorf("failed to export result event to MongoDB: %s", err)
}
// Verify that the result was written to the database
clientOptions := mongooptions.Client().ApplyURI(connString)
client, err := mongoclient.Connect(ctx, clientOptions)
if err != nil {
return fmt.Errorf("error creating MongoDB client: %s", err)
}
defer func() {
if err := client.Disconnect(ctx); err != nil {
fmt.Printf("failed to disconnect from MongoDB: %s\n", err)
}
}()
collection := client.Database(dbName).Collection(opts.CollectionName)
var actualRes output.ResultEvent
err = collection.FindOne(ctx, map[string]interface{}{"request": res.Request}).Decode(&actualRes)
if err != nil {
return fmt.Errorf("failed to find document in MongoDB: %s", err)
}
if actualRes.Request != res.Request || actualRes.Response != res.Response {
return fmt.Errorf("exported result does not match expected result: got %v, want %v", actualRes, res)
}
return nil
}
+8 -8
View File
@@ -196,7 +196,7 @@ func (d *httpDefaultMatcherCondition) Execute(filePath string) error {
return err
}
if routerErr != nil {
return errkit.Append(errkit.New("failed to send http request to interactsh server"), routerErr)
return errkit.Wrap(routerErr, "failed to send http request to interactsh server")
}
if err := expectResultsCount(results, 1); err != nil {
return err
@@ -628,10 +628,10 @@ func (h *httpRawWithParams) Execute(filePath string) error {
// we intentionally use params["test"] instead of params.Get("test") to test the case where
// there are multiple parameters with the same name
if !reflect.DeepEqual(params["key1"], []string{"value1"}) {
errx = errkit.Append(errkit.New(fmt.Sprintf("expected %v, got %v", []string{"value1"}, params["key1"])), errx)
errx = errkit.Append(errx, errkit.New("key1 not found in params", "expected", []string{"value1"}, "got", params["key1"]))
}
if !reflect.DeepEqual(params["key2"], []string{"value2"}) {
errx = errkit.Append(errkit.New(fmt.Sprintf("expected %v, got %v", []string{"value2"}, params["key2"])), errx)
errx = errkit.Append(errx, errkit.New("key2 not found in params", "expected", []string{"value2"}, "got", params["key2"]))
}
_, _ = fmt.Fprintf(w, "Test is test raw-params-matcher text")
})
@@ -971,10 +971,10 @@ func (h *httpRequestSelfContainedWithParams) Execute(filePath string) error {
// we intentionally use params["test"] instead of params.Get("test") to test the case where
// there are multiple parameters with the same name
if !reflect.DeepEqual(params["something"], []string{"here"}) {
errx = errkit.Append(errkit.New(fmt.Sprintf("expected %v, got %v", []string{"here"}, params["something"])), errx)
errx = errkit.Append(errx, errkit.New("something not found in params", "expected", []string{"here"}, "got", params["something"]))
}
if !reflect.DeepEqual(params["key"], []string{"value"}) {
errx = errkit.Append(errkit.New(fmt.Sprintf("expected %v, got %v", []string{"value"}, params["key"])), errx)
errx = errkit.Append(errx, errkit.New("key not found in params", "expected", []string{"value"}, "got", params["key"]))
}
_, _ = w.Write([]byte("This is self-contained response"))
})
@@ -1027,10 +1027,10 @@ func (h *httpRequestSelfContainedFileInput) Execute(filePath string) error {
// create temp file
FileLoc, err := os.CreateTemp("", "self-contained-payload-*.txt")
if err != nil {
return errkit.Append(errkit.New("failed to create temp file"), err)
return errkit.Wrap(err, "failed to create temp file")
}
if _, err := FileLoc.Write([]byte("one\ntwo\n")); err != nil {
return errkit.Append(errkit.New("failed to write payload to temp file"), err)
return errkit.Wrap(err, "failed to write payload to temp file")
}
defer func() {
_ = FileLoc.Close()
@@ -1046,7 +1046,7 @@ func (h *httpRequestSelfContainedFileInput) Execute(filePath string) error {
}
if !sliceutil.ElementsMatch(gotReqToEndpoints, []string{"/one", "/two", "/one", "/two"}) {
return errkit.New(fmt.Sprintf("%s: expected requests to be sent to `/one` and `/two` endpoints but were sent to `%v`", filePath, gotReqToEndpoints)).Build()
return errkit.New("expected requests to be sent to `/one` and `/two` endpoints but were sent to `%v`", gotReqToEndpoints, "filePath", filePath)
}
return nil
}
+1
View File
@@ -57,6 +57,7 @@ var (
"flow": flowTestcases,
"javascript": jsTestcases,
"matcher-status": matcherStatusTestcases,
"exporters": exportersTestCases,
}
// flakyTests are run with a retry count of 3
flakyTests = map[string]bool{
+1 -1
View File
@@ -223,7 +223,7 @@ type loadTemplateWithID struct{}
func (h *loadTemplateWithID) Execute(nooop string) error {
results, err := testutils.RunNucleiBareArgsAndGetResults(debug, nil, "-target", "scanme.sh", "-id", "self-signed-ssl")
if err != nil {
return errkit.Append(errkit.New("failed to load template with id"), err)
return errkit.Wrap(err, "failed to load template with id")
}
return expectResultsCount(results, 1)
}
+3 -3
View File
@@ -18,7 +18,7 @@ type profileLoaderByRelFile struct{}
func (h *profileLoaderByRelFile) Execute(testName string) error {
results, err := testutils.RunNucleiWithArgsAndGetResults(debug, "-tl", "-tp", "cloud.yml")
if err != nil {
return errkit.Append(errkit.New("failed to load template with id"), err)
return errkit.Wrap(err, "failed to load template with id")
}
if len(results) <= 10 {
return fmt.Errorf("incorrect result: expected more results than %d, got %v", 10, len(results))
@@ -31,7 +31,7 @@ type profileLoaderById struct{}
func (h *profileLoaderById) Execute(testName string) error {
results, err := testutils.RunNucleiWithArgsAndGetResults(debug, "-tl", "-tp", "cloud")
if err != nil {
return errkit.Append(errkit.New("failed to load template with id"), err)
return errkit.Wrap(err, "failed to load template with id")
}
if len(results) <= 10 {
return fmt.Errorf("incorrect result: expected more results than %d, got %v", 10, len(results))
@@ -45,7 +45,7 @@ type customProfileLoader struct{}
func (h *customProfileLoader) Execute(filepath string) error {
results, err := testutils.RunNucleiWithArgsAndGetResults(debug, "-tl", "-tp", filepath)
if err != nil {
return errkit.Append(errkit.New("failed to load template with id"), err)
return errkit.Wrap(err, "failed to load template with id")
}
if len(results) < 1 {
return fmt.Errorf("incorrect result: expected more results than %d, got %v", 1, len(results))
+1 -1
View File
@@ -17,7 +17,7 @@ type templateDirWithTargetTest struct{}
func (h *templateDirWithTargetTest) Execute(filePath string) error {
tempdir, err := os.MkdirTemp("", "nuclei-update-dir-*")
if err != nil {
return errkit.Append(errkit.New("failed to create temp dir"), err)
return errkit.Wrap(err, "failed to create temp dir")
}
defer func() {
_ = os.RemoveAll(tempdir)
+1 -1
View File
@@ -187,7 +187,7 @@ func main() {
options.Logger.Info().Msgf("Creating resume file: %s\n", resumeFileName)
err := nucleiRunner.SaveResumeConfig(resumeFileName)
if err != nil {
return errkit.Append(errkit.New("couldn't create crash resume file"), err)
return errkit.Wrap(err, "couldn't create crash resume file")
}
return nil
})
+22 -22
View File
@@ -243,7 +243,7 @@ func enhanceTemplate(data string) (string, bool, error) {
return data, false, err
}
if resp.StatusCode != 200 {
return data, false, errkit.New(fmt.Sprintf("unexpected status code: %v", resp.Status)).Build()
return data, false, errkit.New("unexpected status code: %v", resp.Status)
}
var templateResp TemplateResp
if err := json.NewDecoder(resp.Body).Decode(&templateResp); err != nil {
@@ -254,20 +254,20 @@ func enhanceTemplate(data string) (string, bool, error) {
}
if templateResp.ValidateErrorCount > 0 {
if len(templateResp.ValidateError) > 0 {
return data, false, errkit.New(fmt.Sprintf("validate: %s: at line %v", templateResp.ValidateError[0].Message, templateResp.ValidateError[0].Mark.Line)).Build()
return data, false, errkit.New(templateResp.ValidateError[0].Message+": at line %v", templateResp.ValidateError[0].Mark.Line, "tag", "validate")
}
return data, false, errkit.New("validate: validation failed").Build()
return data, false, errkit.New("validation failed", "tag", "validate")
}
if templateResp.Error.Name != "" {
return data, false, errkit.New(templateResp.Error.Name).Build()
return data, false, errkit.New("%s", templateResp.Error.Name)
}
if strings.TrimSpace(templateResp.Enhanced) == "" && !templateResp.Lint {
if templateResp.LintError.Reason != "" {
return data, false, errkit.New(fmt.Sprintf("lint: %s : at line %v", templateResp.LintError.Reason, templateResp.LintError.Mark.Line)).Build()
return data, false, errkit.New(templateResp.LintError.Reason+" : at line %v", templateResp.LintError.Mark.Line, "tag", "lint")
}
return data, false, errkit.New(fmt.Sprintf("lint: at line: %v", templateResp.LintError.Mark.Line)).Build()
return data, false, errkit.New("at line: %v", templateResp.LintError.Mark.Line, "tag", "lint")
}
return data, false, errkit.New("template enhance failed").Build()
return data, false, errkit.New("template enhance failed")
}
// formatTemplate formats template data using templateman format api
@@ -277,7 +277,7 @@ func formatTemplate(data string) (string, bool, error) {
return data, false, err
}
if resp.StatusCode != 200 {
return data, false, errkit.New(fmt.Sprintf("unexpected status code: %v", resp.Status)).Build()
return data, false, errkit.New("unexpected status code: %v", resp.Status)
}
var templateResp TemplateResp
if err := json.NewDecoder(resp.Body).Decode(&templateResp); err != nil {
@@ -288,20 +288,20 @@ func formatTemplate(data string) (string, bool, error) {
}
if templateResp.ValidateErrorCount > 0 {
if len(templateResp.ValidateError) > 0 {
return data, false, errkit.New(fmt.Sprintf("validate: %s: at line %v", templateResp.ValidateError[0].Message, templateResp.ValidateError[0].Mark.Line)).Build()
return data, false, errkit.New(templateResp.ValidateError[0].Message+": at line %v", templateResp.ValidateError[0].Mark.Line, "tag", "validate")
}
return data, false, errkit.New("validate: validation failed").Build()
return data, false, errkit.New("validation failed", "tag", "validate")
}
if templateResp.Error.Name != "" {
return data, false, errkit.New(templateResp.Error.Name).Build()
return data, false, errkit.New("%s", templateResp.Error.Name)
}
if strings.TrimSpace(templateResp.Updated) == "" && !templateResp.Lint {
if templateResp.LintError.Reason != "" {
return data, false, errkit.New(fmt.Sprintf("lint: %s : at line %v", templateResp.LintError.Reason, templateResp.LintError.Mark.Line)).Build()
return data, false, errkit.New(templateResp.LintError.Reason+" : at line %v", templateResp.LintError.Mark.Line, "tag", "lint")
}
return data, false, errkit.New(fmt.Sprintf("lint: at line: %v", templateResp.LintError.Mark.Line)).Build()
return data, false, errkit.New("at line: %v", templateResp.LintError.Mark.Line, "tag", "lint")
}
return data, false, errkit.New("template format failed").Build()
return data, false, errkit.New("template format failed")
}
// lintTemplate lints template data using templateman lint api
@@ -311,7 +311,7 @@ func lintTemplate(data string) (bool, error) {
return false, err
}
if resp.StatusCode != 200 {
return false, errkit.New(fmt.Sprintf("unexpected status code: %v", resp.Status)).Build()
return false, errkit.New("unexpected status code: %v", resp.Status)
}
var lintResp TemplateLintResp
if err := json.NewDecoder(resp.Body).Decode(&lintResp); err != nil {
@@ -321,9 +321,9 @@ func lintTemplate(data string) (bool, error) {
return true, nil
}
if lintResp.LintError.Reason != "" {
return false, errkit.New(fmt.Sprintf("lint: %s : at line %v", lintResp.LintError.Reason, lintResp.LintError.Mark.Line)).Build()
return false, errkit.New(lintResp.LintError.Reason+" : at line %v", lintResp.LintError.Mark.Line, "tag", "lint")
}
return false, errkit.New(fmt.Sprintf("lint: at line: %v", lintResp.LintError.Mark.Line)).Build()
return false, errkit.New("at line: %v", lintResp.LintError.Mark.Line, "tag", "lint")
}
// validateTemplate validates template data using templateman validate api
@@ -333,7 +333,7 @@ func validateTemplate(data string) (bool, error) {
return false, err
}
if resp.StatusCode != 200 {
return false, errkit.New(fmt.Sprintf("unexpected status code: %v", resp.Status)).Build()
return false, errkit.New("unexpected status code: %v", resp.Status)
}
var validateResp TemplateResp
if err := json.NewDecoder(resp.Body).Decode(&validateResp); err != nil {
@@ -344,14 +344,14 @@ func validateTemplate(data string) (bool, error) {
}
if validateResp.ValidateErrorCount > 0 {
if len(validateResp.ValidateError) > 0 {
return false, errkit.New(fmt.Sprintf("validate: %s: at line %v", validateResp.ValidateError[0].Message, validateResp.ValidateError[0].Mark.Line)).Build()
return false, errkit.New(validateResp.ValidateError[0].Message+": at line %v", validateResp.ValidateError[0].Mark.Line, "tag", "validate")
}
return false, errkit.New("validate: validation failed").Build()
return false, errkit.New("validation failed", "tag", "validate")
}
if validateResp.Error.Name != "" {
return false, errkit.New(validateResp.Error.Name).Build()
return false, errkit.New("%s", validateResp.Error.Name)
}
return false, errkit.New("template validation failed").Build()
return false, errkit.New("template validation failed")
}
// parseAndAddMaxRequests parses and adds max requests to templates
+73 -42
View File
@@ -20,12 +20,12 @@ require (
github.com/olekukonko/tablewriter v1.0.8
github.com/pkg/errors v0.9.1
github.com/projectdiscovery/clistats v0.1.1
github.com/projectdiscovery/fastdialer v0.4.4
github.com/projectdiscovery/hmap v0.0.92
github.com/projectdiscovery/fastdialer v0.4.7
github.com/projectdiscovery/hmap v0.0.93
github.com/projectdiscovery/interactsh v1.2.4
github.com/projectdiscovery/rawhttp v0.1.90
github.com/projectdiscovery/retryabledns v1.0.105
github.com/projectdiscovery/retryablehttp-go v1.0.119
github.com/projectdiscovery/retryabledns v1.0.106
github.com/projectdiscovery/retryablehttp-go v1.0.121
github.com/projectdiscovery/yamldoc-go v1.0.6
github.com/remeh/sizedwaitgroup v1.0.0
github.com/rs/xid v1.6.0
@@ -35,11 +35,11 @@ require (
github.com/spf13/cast v1.9.2
github.com/syndtr/goleveldb v1.0.0
github.com/valyala/fasttemplate v1.2.2
github.com/weppos/publicsuffix-go v0.40.3-0.20250408071509-6074bbe7fd39
github.com/weppos/publicsuffix-go v0.50.0
go.uber.org/multierr v1.11.0
golang.org/x/net v0.41.0
golang.org/x/net v0.43.0
golang.org/x/oauth2 v0.30.0
golang.org/x/text v0.26.0
golang.org/x/text v0.28.0
gopkg.in/yaml.v2 v2.4.0
)
@@ -87,49 +87,51 @@ require (
github.com/microsoft/go-mssqldb v1.9.2
github.com/ory/dockertest/v3 v3.12.0
github.com/praetorian-inc/fingerprintx v1.1.15
github.com/projectdiscovery/dsl v0.5.0
github.com/projectdiscovery/dsl v0.5.1
github.com/projectdiscovery/fasttemplate v0.0.2
github.com/projectdiscovery/gcache v0.0.0-20241015120333-12546c6e3f4c
github.com/projectdiscovery/go-smb2 v0.0.0-20240129202741-052cc450c6cb
github.com/projectdiscovery/goflags v0.1.74
github.com/projectdiscovery/gologger v1.1.54
github.com/projectdiscovery/gostruct v0.0.2
github.com/projectdiscovery/gozero v0.0.3
github.com/projectdiscovery/httpx v1.7.0
github.com/projectdiscovery/gozero v0.1.0
github.com/projectdiscovery/httpx v1.7.1-0.20250901093029-3909667cbaa6
github.com/projectdiscovery/mapcidr v1.1.34
github.com/projectdiscovery/n3iwf v0.0.0-20230523120440-b8cd232ff1f5
github.com/projectdiscovery/networkpolicy v0.1.18
github.com/projectdiscovery/ratelimit v0.0.81
github.com/projectdiscovery/networkpolicy v0.1.21
github.com/projectdiscovery/ratelimit v0.0.82
github.com/projectdiscovery/rdap v0.9.0
github.com/projectdiscovery/sarif v0.0.1
github.com/projectdiscovery/tlsx v1.1.9
github.com/projectdiscovery/tlsx v1.2.1
github.com/projectdiscovery/uncover v1.1.0
github.com/projectdiscovery/useragent v0.0.101
github.com/projectdiscovery/utils v0.4.23
github.com/projectdiscovery/wappalyzergo v0.2.36
github.com/projectdiscovery/utils v0.4.24-0.20250823123502-bd7f2849ddb4
github.com/projectdiscovery/wappalyzergo v0.2.43
github.com/redis/go-redis/v9 v9.11.0
github.com/seh-msft/burpxml v1.0.1
github.com/shurcooL/graphql v0.0.0-20230722043721-ed46e5a46466
github.com/stretchr/testify v1.10.0
github.com/stretchr/testify v1.11.0
github.com/tarunKoyalwar/goleak v0.0.0-20240429141123-0efa90dbdcf9
github.com/testcontainers/testcontainers-go v0.38.0
github.com/testcontainers/testcontainers-go/modules/mongodb v0.37.0
github.com/yassinebenaid/godump v0.11.1
github.com/zmap/zgrab2 v0.1.8
gitlab.com/gitlab-org/api/client-go v0.130.1
go.mongodb.org/mongo-driver v1.17.4
golang.org/x/term v0.32.0
golang.org/x/term v0.34.0
gopkg.in/yaml.v3 v3.0.1
moul.io/http2curl v1.0.0
)
require (
aead.dev/minisign v0.2.0 // indirect
dario.cat/mergo v1.0.0 // indirect
dario.cat/mergo v1.0.2 // indirect
filippo.io/edwards25519 v1.1.0 // indirect
git.mills.io/prologic/smtpd v0.0.0-20210710122116-a525b76c287a // indirect
github.com/42wim/httpsig v1.2.2 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.18.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.1 // indirect
github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 // indirect
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect
github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.4.2 // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
@@ -179,27 +181,35 @@ require (
github.com/charmbracelet/x/cellbuf v0.0.13 // indirect
github.com/charmbracelet/x/exp/slice v0.0.0-20250327172914-2fdc97757edf // indirect
github.com/charmbracelet/x/term v0.2.1 // indirect
github.com/cheggaaa/pb/v3 v3.1.4 // indirect
github.com/cheggaaa/pb/v3 v3.1.6 // indirect
github.com/cloudflare/cfssl v1.6.4 // indirect
github.com/cloudflare/circl v1.6.1 // indirect
github.com/cloudwego/base64x v0.1.5 // indirect
github.com/cnf/structhash v0.0.0-20201127153200-e1b16c1ebc08 // indirect
github.com/containerd/continuity v0.4.5 // indirect
github.com/containerd/errdefs v1.0.0 // indirect
github.com/containerd/errdefs/pkg v0.3.0 // indirect
github.com/containerd/log v0.1.0 // indirect
github.com/containerd/platforms v0.2.1 // indirect
github.com/cpuguy83/dockercfg v0.3.2 // indirect
github.com/cyphar/filepath-securejoin v0.4.1 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/davidmz/go-pageant v1.0.2 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/distribution/reference v0.6.0 // indirect
github.com/dlclark/regexp2 v1.11.5 // indirect
github.com/docker/cli v27.4.1+incompatible // indirect
github.com/docker/docker v28.0.0+incompatible // indirect
github.com/docker/go-connections v0.5.0 // indirect
github.com/docker/docker v28.3.3+incompatible // indirect
github.com/docker/go-connections v0.6.0 // indirect
github.com/dsnet/compress v0.0.2-0.20230904184137-39efe44ab707 // indirect
github.com/ebitengine/purego v0.8.4 // indirect
github.com/emirpasic/gods v1.18.1 // indirect
github.com/fatih/color v1.18.0 // indirect
github.com/felixge/fgprof v0.9.5 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/free5gc/util v1.0.5-0.20230511064842-2e120956883b // indirect
github.com/gabriel-vasile/mimetype v1.4.8 // indirect
github.com/gaissmai/bart v0.20.5 // indirect
github.com/gaissmai/bart v0.24.0 // indirect
github.com/geoffgarside/ber v1.1.0 // indirect
github.com/gin-contrib/sse v0.1.0 // indirect
github.com/gin-gonic/gin v1.9.1 // indirect
@@ -207,20 +217,22 @@ require (
github.com/go-fed/httpsig v1.1.0 // indirect
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
github.com/go-git/go-billy/v5 v5.6.2 // indirect
github.com/go-ole/go-ole v1.2.6 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.3.0 // indirect
github.com/go-openapi/jsonpointer v0.21.0 // indirect
github.com/go-openapi/swag v0.23.0 // indirect
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-sourcemap/sourcemap v2.1.4+incompatible // indirect
github.com/go-viper/mapstructure/v2 v2.3.0 // indirect
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang-jwt/jwt/v5 v5.2.2 // indirect
github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9 // indirect
github.com/golang-sql/sqlexp v0.1.0 // indirect
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/certificate-transparency-go v1.1.4 // indirect
github.com/google/certificate-transparency-go v1.3.2 // indirect
github.com/google/go-github/v30 v30.1.0 // indirect
github.com/google/pprof v0.0.0-20240727154555-813a5fbdbec8 // indirect
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
@@ -255,8 +267,9 @@ require (
github.com/logrusorgru/aurora/v4 v4.0.0 // indirect
github.com/lor00x/goldap v0.0.0-20180618054307-a546dffdd1a3 // indirect
github.com/lucasb-eyer/go-colorful v1.2.0 // indirect
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
github.com/lufia/plan9stats v0.0.0-20250821153705-5981dea3221d // indirect
github.com/mackerelio/go-osstat v0.2.4 // indirect
github.com/magiconair/properties v1.8.10 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
@@ -268,12 +281,17 @@ require (
github.com/minio/selfupdate v0.6.1-0.20230907112617-f11e74f84ca7 // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/moby/sys/user v0.3.0 // indirect
github.com/moby/term v0.5.0 // indirect
github.com/moby/go-archive v0.1.0 // indirect
github.com/moby/patternmatcher v0.6.0 // indirect
github.com/moby/sys/sequential v0.6.0 // indirect
github.com/moby/sys/user v0.4.0 // indirect
github.com/moby/sys/userns v0.1.0 // indirect
github.com/moby/term v0.5.2 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 // indirect
github.com/montanaflynn/stats v0.7.1 // indirect
github.com/morikuni/aec v1.0.0 // indirect
github.com/muesli/reflow v0.3.0 // indirect
github.com/muesli/termenv v0.16.0 // indirect
github.com/nwaples/rardecode/v2 v2.1.0 // indirect
@@ -282,7 +300,7 @@ require (
github.com/olekukonko/errors v1.1.0 // indirect
github.com/olekukonko/ll v0.0.9 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/opencontainers/runc v1.2.3 // indirect
github.com/openrdap/rdap v0.9.1 // indirect
github.com/pelletier/go-toml/v2 v2.0.8 // indirect
@@ -291,10 +309,10 @@ require (
github.com/pjbgf/sha1cd v0.3.2 // indirect
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect
github.com/projectdiscovery/asnmap v1.1.1 // indirect
github.com/projectdiscovery/blackrock v0.0.1 // indirect
github.com/projectdiscovery/cdncheck v1.1.26 // indirect
github.com/projectdiscovery/cdncheck v1.1.33 // indirect
github.com/projectdiscovery/freeport v0.0.7 // indirect
github.com/projectdiscovery/ldapserver v1.0.2-0.20240219154113-dcc758ebc0cb // indirect
github.com/projectdiscovery/machineid v0.0.0-20240226150047-2e2c51e35983 // indirect
@@ -302,6 +320,7 @@ require (
github.com/sashabaranov/go-openai v1.37.0 // indirect
github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 // indirect
github.com/shirou/gopsutil v3.21.11+incompatible // indirect
github.com/shirou/gopsutil/v4 v4.25.7 // indirect
github.com/shoenig/go-m1cpu v0.1.6 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/skeema/knownhosts v1.3.1 // indirect
@@ -315,8 +334,8 @@ require (
github.com/tidwall/rtred v0.1.2 // indirect
github.com/tidwall/tinyqueue v0.1.1 // indirect
github.com/tim-ywliu/nested-logrus-formatter v1.3.2 // indirect
github.com/tklauser/go-sysconf v0.3.12 // indirect
github.com/tklauser/numcpus v0.6.1 // indirect
github.com/tklauser/go-sysconf v0.3.15 // indirect
github.com/tklauser/numcpus v0.10.0 // indirect
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
github.com/ugorji/go/codec v1.2.11 // indirect
github.com/ulikunitz/xz v0.5.12 // indirect
@@ -337,9 +356,14 @@ require (
github.com/yuin/goldmark-emoji v1.0.5 // indirect
github.com/zcalusic/sysinfo v1.0.2 // indirect
github.com/zeebo/blake3 v0.2.3 // indirect
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.62.0 // indirect
go.opentelemetry.io/otel v1.37.0 // indirect
go.opentelemetry.io/otel/metric v1.37.0 // indirect
go.opentelemetry.io/otel/trace v1.37.0 // indirect
go4.org v0.0.0-20230225012048-214862532bf5 // indirect
golang.org/x/arch v0.3.0 // indirect
golang.org/x/sync v0.15.0 // indirect
golang.org/x/sync v0.16.0 // indirect
gopkg.in/djherbis/times.v1 v1.3.0 // indirect
mellium.im/sasl v0.3.2 // indirect
)
@@ -360,16 +384,16 @@ require (
github.com/yusufpapurcu/wmi v1.2.4 // indirect
github.com/zmap/rc2 v0.0.0-20190804163417-abaa70531248 // indirect
github.com/zmap/zcrypto v0.0.0-20240512203510-0fef58d9a9db // indirect
go.etcd.io/bbolt v1.3.10 // indirect
go.uber.org/zap v1.25.0 // indirect
go.etcd.io/bbolt v1.4.0 // indirect
go.uber.org/zap v1.27.0 // indirect
goftp.io/server/v2 v2.0.1 // indirect
golang.org/x/crypto v0.39.0 // indirect
golang.org/x/crypto v0.41.0 // indirect
golang.org/x/exp v0.0.0-20250620022241-b7579e27df2b
golang.org/x/mod v0.25.0 // indirect
golang.org/x/sys v0.33.0 // indirect
golang.org/x/mod v0.26.0 // indirect
golang.org/x/sys v0.35.0 // indirect
golang.org/x/time v0.11.0 // indirect
golang.org/x/tools v0.34.0
google.golang.org/protobuf v1.35.1 // indirect
golang.org/x/tools v0.35.0
google.golang.org/protobuf v1.36.6 // indirect
gopkg.in/alecthomas/kingpin.v2 v2.2.6 // indirect
gopkg.in/corvus-ch/zbase32.v1 v1.0.0 // indirect
)
@@ -382,3 +406,10 @@ require (
// https://go.dev/ref/mod#go-mod-file-retract
retract v3.2.0 // retract due to broken js protocol issue
// Fix genproto version conflicts
replace (
google.golang.org/genproto => google.golang.org/genproto v0.0.0-20240814211410-ddb44dafa142
google.golang.org/genproto/googleapis/api => google.golang.org/genproto/googleapis/api v0.0.0-20240814211410-ddb44dafa142
google.golang.org/genproto/googleapis/rpc => google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1
)
+1062 -121
View File
File diff suppressed because it is too large. Load diff
+8 -8
View File
@@ -77,11 +77,11 @@ func NewUploadWriter(ctx context.Context, logger *gologger.Logger, creds *pdcpau
output.WithJson(true, true),
)
if err != nil {
return nil, errkit.Append(errkit.New("could not create output writer"), err)
return nil, errkit.Wrap(err, "could not create output writer")
}
tmp, err := urlutil.Parse(creds.Server)
if err != nil {
return nil, errkit.Append(errkit.New("could not parse server url"), err)
return nil, errkit.Wrap(err, "could not parse server url")
}
tmp.Path = uploadEndpoint
tmp.Update()
@@ -199,7 +199,7 @@ func (u *UploadWriter) autoCommit(ctx context.Context, r *io.PipeReader) {
// uploadChunk uploads a chunk of data to the server
func (u *UploadWriter) uploadChunk(buff *bytes.Buffer) error {
if err := u.upload(buff.Bytes()); err != nil {
return errkit.Append(errkit.New("could not upload chunk"), err)
return errkit.Wrap(err, "could not upload chunk")
}
// if successful, reset the buffer
buff.Reset()
@@ -211,25 +211,25 @@ func (u *UploadWriter) uploadChunk(buff *bytes.Buffer) error {
func (u *UploadWriter) upload(data []byte) error {
req, err := u.getRequest(data)
if err != nil {
return errkit.Append(errkit.New("could not create upload request"), err)
return errkit.Wrap(err, "could not create upload request")
}
resp, err := u.client.Do(req)
if err != nil {
return errkit.Append(errkit.New("could not upload results"), err)
return errkit.Wrap(err, "could not upload results")
}
defer func() {
_ = resp.Body.Close()
}()
bin, err := io.ReadAll(resp.Body)
if err != nil {
return errkit.Append(errkit.New("could not get id from response"), err)
return errkit.Wrap(err, "could not get id from response")
}
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("could not upload results got status code %v on %v", resp.StatusCode, resp.Request.URL.String())
}
var uploadResp uploadResponse
if err := json.Unmarshal(bin, &uploadResp); err != nil {
return errkit.Append(errkit.New(fmt.Sprintf("could not unmarshal response got %v", string(bin))), err)
return errkit.Wrap(err, fmt.Sprintf("could not unmarshal response got %v", string(bin)))
}
if uploadResp.ID != "" && u.scanID == "" {
u.scanID = uploadResp.ID
@@ -254,7 +254,7 @@ func (u *UploadWriter) getRequest(bin []byte) (*retryablehttp.Request, error) {
}
req, err := retryablehttp.NewRequest(method, url, bytes.NewReader(bin))
if err != nil {
return nil, errkit.Append(errkit.New("could not create cloud upload request"), err)
return nil, errkit.Wrap(err, "could not create cloud upload request")
}
// add pdtm meta params
req.Params.Merge(updateutils.GetpdtmParams(config.Version))
+2 -2
View File
@@ -32,7 +32,7 @@ func GetAuthTmplStore(opts *types.Options, catalog catalog.Catalog, execOpts *pr
for _, file := range opts.SecretsFile {
data, err := authx.GetTemplatePathsFromSecretFile(file)
if err != nil {
return nil, errkit.Append(errkit.New("failed to get template paths from secrets file"), err)
return nil, errkit.Wrap(err, "failed to get template paths from secrets file")
}
tmpls = append(tmpls, data...)
}
@@ -58,7 +58,7 @@ func GetAuthTmplStore(opts *types.Options, catalog catalog.Catalog, execOpts *pr
cfg.StoreId = loader.AuthStoreId
store, err := loader.New(cfg)
if err != nil {
return nil, errkit.Append(errkit.New("failed to initialize dynamic auth templates store"), err)
return nil, errkit.Wrap(err, "failed to initialize dynamic auth templates store")
}
return store, nil
}
+1 -1
View File
@@ -50,7 +50,7 @@ func loadProxyServers(options *types.Options) error {
}
proxyURL, err := url.Parse(aliveProxy)
if err != nil {
return errkit.Append(errkit.New(fmt.Sprintf("failed to parse proxy got %v", err)), err)
return errkit.Wrapf(err, "failed to parse proxy got %v", err)
}
if options.ProxyInternal {
_ = os.Setenv(HTTP_PROXY_ENV, proxyURL.String())
+10 -9
View File
@@ -8,6 +8,7 @@ import (
"github.com/projectdiscovery/goflags"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/ratelimit"
"github.com/projectdiscovery/utils/errkit"
"github.com/projectdiscovery/nuclei/v3/pkg/authprovider"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog"
@@ -102,7 +103,7 @@ type InteractshOpts interactsh.Options
func WithInteractshOptions(opts InteractshOpts) NucleiSDKOptions {
return func(e *NucleiEngine) error {
if e.mode == threadSafe {
return ErrOptionsNotSupported("WithInteractshOptions")
return errkit.Wrap(ErrOptionsNotSupported, "WithInteractshOptions")
}
optsPtr := &opts
e.interactshOpts = (*interactsh.Options)(optsPtr)
@@ -229,7 +230,7 @@ type StatsOptions struct {
func EnableStatsWithOpts(opts StatsOptions) NucleiSDKOptions {
return func(e *NucleiEngine) error {
if e.mode == threadSafe {
return ErrOptionsNotSupported("EnableStatsWithOpts")
return errkit.Wrap(ErrOptionsNotSupported, "EnableStatsWithOpts")
}
if opts.Interval == 0 {
opts.Interval = 5 //sec
@@ -257,7 +258,7 @@ type VerbosityOptions struct {
func WithVerbosity(opts VerbosityOptions) NucleiSDKOptions {
return func(e *NucleiEngine) error {
if e.mode == threadSafe {
return ErrOptionsNotSupported("WithVerbosity")
return errkit.Wrap(ErrOptionsNotSupported, "WithVerbosity")
}
e.opts.Verbose = opts.Verbose
e.opts.Silent = opts.Silent
@@ -290,7 +291,7 @@ type NetworkConfig struct {
func WithNetworkConfig(opts NetworkConfig) NucleiSDKOptions {
return func(e *NucleiEngine) error {
if e.mode == threadSafe {
return ErrOptionsNotSupported("WithNetworkConfig")
return errkit.Wrap(ErrOptionsNotSupported, "WithNetworkConfig")
}
e.opts.NoHostErrors = opts.DisableMaxHostErr
e.opts.MaxHostError = opts.MaxHostError
@@ -321,7 +322,7 @@ func WithNetworkConfig(opts NetworkConfig) NucleiSDKOptions {
func WithProxy(proxy []string, proxyInternalRequests bool) NucleiSDKOptions {
return func(e *NucleiEngine) error {
if e.mode == threadSafe {
return ErrOptionsNotSupported("WithProxy")
return errkit.Wrap(ErrOptionsNotSupported, "WithProxy")
}
e.opts.Proxy = proxy
e.opts.ProxyInternal = proxyInternalRequests
@@ -346,7 +347,7 @@ type OutputWriter output.Writer
func UseOutputWriter(writer OutputWriter) NucleiSDKOptions {
return func(e *NucleiEngine) error {
if e.mode == threadSafe {
return ErrOptionsNotSupported("UseOutputWriter")
return errkit.Wrap(ErrOptionsNotSupported, "UseOutputWriter")
}
e.customWriter = writer
return nil
@@ -361,7 +362,7 @@ type StatsWriter progress.Progress
func UseStatsWriter(writer StatsWriter) NucleiSDKOptions {
return func(e *NucleiEngine) error {
if e.mode == threadSafe {
return ErrOptionsNotSupported("UseStatsWriter")
return errkit.Wrap(ErrOptionsNotSupported, "UseStatsWriter")
}
e.customProgress = writer
return nil
@@ -375,7 +376,7 @@ func UseStatsWriter(writer StatsWriter) NucleiSDKOptions {
func WithTemplateUpdateCallback(disableTemplatesAutoUpgrade bool, callback func(newVersion string)) NucleiSDKOptions {
return func(e *NucleiEngine) error {
if e.mode == threadSafe {
return ErrOptionsNotSupported("WithTemplateUpdateCallback")
return errkit.Wrap(ErrOptionsNotSupported, "WithTemplateUpdateCallback")
}
e.disableTemplatesAutoUpgrade = disableTemplatesAutoUpgrade
e.onUpdateAvailableCallback = callback
@@ -387,7 +388,7 @@ func WithTemplateUpdateCallback(disableTemplatesAutoUpgrade bool, callback func(
func WithSandboxOptions(allowLocalFileAccess bool, restrictLocalNetworkAccess bool) NucleiSDKOptions {
return func(e *NucleiEngine) error {
if e.mode == threadSafe {
return ErrOptionsNotSupported("WithSandboxOptions")
return errkit.Wrap(ErrOptionsNotSupported, "WithSandboxOptions")
}
e.opts.AllowLocalFileAccess = allowLocalFileAccess
e.opts.RestrictLocalNetworkAccess = restrictLocalNetworkAccess
+2 -2
View File
@@ -147,13 +147,13 @@ func (e *ThreadSafeNucleiEngine) ExecuteNucleiWithOptsCtx(ctx context.Context, t
// load templates
workflowLoader, err := workflow.NewLoader(unsafeOpts.executerOpts)
if err != nil {
return errkit.Append(errkit.New("Could not create workflow loader"), err)
return errkit.Wrapf(err, "Could not create workflow loader: %s", err)
}
unsafeOpts.executerOpts.WorkflowLoader = workflowLoader
store, err := loader.New(loader.NewConfig(tmpEngine.opts, e.eng.catalog, unsafeOpts.executerOpts))
if err != nil {
return errkit.Append(errkit.New("Could not create loader client"), err)
return errkit.Wrapf(err, "Could not create loader client: %s", err)
}
store.Load()
+7 -11
View File
@@ -4,7 +4,6 @@ import (
"bufio"
"bytes"
"context"
"fmt"
"io"
"sync"
@@ -38,18 +37,15 @@ type NucleiSDKOptions func(e *NucleiEngine) error
var (
// ErrNotImplemented is returned when a feature is not implemented
ErrNotImplemented = errkit.New("Not implemented").Build()
ErrNotImplemented = errkit.New("Not implemented")
// ErrNoTemplatesAvailable is returned when no templates are available to execute
ErrNoTemplatesAvailable = errkit.New("No templates available").Build()
ErrNoTemplatesAvailable = errkit.New("No templates available")
// ErrNoTargetsAvailable is returned when no targets are available to scan
ErrNoTargetsAvailable = errkit.New("No targets available").Build()
ErrNoTargetsAvailable = errkit.New("No targets available")
// ErrOptionsNotSupported is returned when an option is not supported in thread safe mode
ErrOptionsNotSupported = errkit.New("Option not supported in thread safe mode")
)
// ErrOptionsNotSupported returns an error when an option is not supported in thread safe mode
func ErrOptionsNotSupported(option string) error {
return errkit.New(fmt.Sprintf("Option %v not supported in thread safe mode", option)).Build()
}
type engineMode uint
const (
@@ -102,13 +98,13 @@ type NucleiEngine struct {
func (e *NucleiEngine) LoadAllTemplates() error {
workflowLoader, err := workflow.NewLoader(e.executerOpts)
if err != nil {
return errkit.Append(errkit.New("Could not create workflow loader"), err)
return errkit.Wrapf(err, "Could not create workflow loader: %s", err)
}
e.executerOpts.WorkflowLoader = workflowLoader
e.store, err = loader.New(loader.NewConfig(e.opts, e.catalog, e.executerOpts))
if err != nil {
return errkit.Append(errkit.New("Could not create loader client"), err)
return errkit.Wrapf(err, "Could not create loader client: %s", err)
}
e.store.Load()
e.templatesLoaded = true
+22 -1
View File
@@ -3,6 +3,7 @@ package nuclei
import (
"context"
"fmt"
"os"
"strings"
"sync"
"time"
@@ -37,6 +38,21 @@ import (
"github.com/projectdiscovery/ratelimit"
)
// enginePool provides shared parsed-cache across engines (opt-in)
var (
sharedParsedOnce sync.Once
sharedParsed *templates.Cache
)
func getSharedParser() *templates.Parser {
// Initialize the shared parsed cache once
sharedParsedOnce.Do(func() {
sharedParsed = templates.NewCache()
})
// Return a fresh Parser each call that reuses only the shared parsed cache,
return templates.NewParserWithParsedCache(sharedParsed)
}
// applyRequiredDefaults to options
func (e *NucleiEngine) applyRequiredDefaults(ctx context.Context) {
mockoutput := testutils.NewMockOutputWriter(e.opts.OmitTemplate)
@@ -123,7 +139,12 @@ func (e *NucleiEngine) init(ctx context.Context) error {
}
if e.parser == nil {
e.parser = templates.NewParser()
//TODO: remove this feature flag after testing
if os.Getenv("NUCLEI_USE_SHARED_COMPILED") == "1" {
e.parser = templates.NewSharedParserWithCompiledCache()
} else {
e.parser = templates.NewParser()
}
}
if protocolstate.ShouldInit(e.opts.ExecutionId) {
+26
View File
@@ -3,6 +3,7 @@ package nuclei_test
import (
"context"
"log"
"os"
"testing"
"time"
@@ -35,3 +36,28 @@ func TestContextCancelNucleiEngine(t *testing.T) {
}
defer ne.Close()
}
func TestSharedParserOptIn(t *testing.T) {
os.Setenv("NUCLEI_USE_SHARED_PARSER", "1")
t.Cleanup(func() { os.Unsetenv("NUCLEI_USE_SHARED_PARSER") })
ne, err := nuclei.NewNucleiEngineCtx(context.Background())
if err != nil {
t.Fatalf("engine error: %v", err)
}
p := ne.GetParser()
if p == nil {
t.Fatalf("expected templates.Parser")
}
ne2, err := nuclei.NewNucleiEngineCtx(context.Background())
if err != nil {
t.Fatalf("engine2 error: %v", err)
}
p2 := ne2.GetParser()
if p2 == nil {
t.Fatalf("expected templates.Parser2")
}
if p.Cache() != p2.Cache() {
t.Fatalf("expected shared parsed cache across engines when opt-in is set")
}
}
+3 -3
View File
@@ -53,7 +53,7 @@ func (d *Dynamic) GetDomainAndDomainRegex() ([]string, []string) {
func (d *Dynamic) UnmarshalJSON(data []byte) error {
if d == nil {
return errkit.New("cannot unmarshal into nil Dynamic struct").Build()
return errkit.New("cannot unmarshal into nil Dynamic struct")
}
// Use an alias type (auxiliary) to avoid a recursive call in this method.
@@ -72,10 +72,10 @@ func (d *Dynamic) UnmarshalJSON(data []byte) error {
func (d *Dynamic) Validate() error {
d.m = &sync.Mutex{}
if d.TemplatePath == "" {
return errkit.New(" template-path is required for dynamic secret").Build()
return errkit.New(" template-path is required for dynamic secret")
}
if len(d.Variables) == 0 {
return errkit.New("variables are required for dynamic secret").Build()
return errkit.New("variables are required for dynamic secret")
}
if d.Secret != nil {
+6 -2
View File
@@ -237,7 +237,9 @@ func GetAuthDataFromYAML(data []byte) (*Authx, error) {
var auth Authx
err := yaml.Unmarshal(data, &auth)
if err != nil {
return nil, errkit.Append(errkit.New("could not unmarshal yaml"), err)
errorErr := errkit.FromError(err)
errorErr.Msgf("could not unmarshal yaml")
return nil, errorErr
}
return &auth, nil
}
@@ -247,7 +249,9 @@ func GetAuthDataFromJSON(data []byte) (*Authx, error) {
var auth Authx
err := json.Unmarshal(data, &auth)
if err != nil {
return nil, errkit.Append(errkit.New("could not unmarshal json"), err)
errorErr := errkit.FromError(err)
errorErr.Msgf("could not unmarshal json")
return nil, errorErr
}
return &auth, nil
}
+7 -4
View File
@@ -1,7 +1,6 @@
package authprovider
import (
"fmt"
"net"
"net/url"
"regexp"
@@ -31,16 +30,20 @@ func NewFileAuthProvider(path string, callback authx.LazyFetchSecret) (AuthProvi
return nil, ErrNoSecrets
}
if len(store.Dynamic) > 0 && callback == nil {
return nil, errkit.New("lazy fetch callback is required for dynamic secrets").Build()
return nil, errkit.New("lazy fetch callback is required for dynamic secrets")
}
for _, secret := range store.Secrets {
if err := secret.Validate(); err != nil {
return nil, errkit.Append(errkit.New(fmt.Sprintf("invalid secret in file: %s", path)), err)
errorErr := errkit.FromError(err)
errorErr.Msgf("invalid secret in file: %s", path)
return nil, errorErr
}
}
for i, dynamic := range store.Dynamic {
if err := dynamic.Validate(); err != nil {
return nil, errkit.Append(errkit.New(fmt.Sprintf("invalid dynamic in file: %s", path)), err)
errorErr := errkit.FromError(err)
errorErr.Msgf("invalid dynamic in file: %s", path)
return nil, errorErr
}
dynamic.SetLazyFetchCallback(callback)
store.Dynamic[i] = dynamic
+9 -9
View File
@@ -140,13 +140,13 @@ func (c *Config) UpdateNucleiIgnoreHash() error {
if fileutil.FileExists(ignoreFilePath) {
bin, err := os.ReadFile(ignoreFilePath)
if err != nil {
return errkit.Append(errkit.New("could not read nuclei ignore file"), err)
return errkit.Newf("could not read nuclei ignore file: %v", err)
}
c.NucleiIgnoreHash = fmt.Sprintf("%x", md5.Sum(bin))
// write config to disk
return c.WriteTemplatesConfig()
}
return errkit.New("config: ignore file not found: could not update nuclei ignore hash").Build()
return errkit.New("ignore file not found: could not update nuclei ignore hash")
}
// GetConfigDir returns the nuclei configuration directory
@@ -257,7 +257,7 @@ func (c *Config) SetTemplatesVersion(version string) error {
c.TemplateVersion = version
// write config to disk
if err := c.WriteTemplatesConfig(); err != nil {
return errkit.Append(errkit.New(fmt.Sprintf("could not write nuclei config file at %s", c.getTemplatesConfigFilePath())), err)
return errkit.Newf("could not write nuclei config file at %s: %v", c.getTemplatesConfigFilePath(), err)
}
return nil
}
@@ -265,15 +265,15 @@ func (c *Config) SetTemplatesVersion(version string) error {
// ReadTemplatesConfig reads the nuclei templates config file
func (c *Config) ReadTemplatesConfig() error {
if !fileutil.FileExists(c.getTemplatesConfigFilePath()) {
return errkit.New(fmt.Sprintf("config: nuclei config file at %s does not exist", c.getTemplatesConfigFilePath())).Build()
return errkit.Newf("nuclei config file at %s does not exist", c.getTemplatesConfigFilePath())
}
var cfg *Config
bin, err := os.ReadFile(c.getTemplatesConfigFilePath())
if err != nil {
return errkit.Append(errkit.New(fmt.Sprintf("could not read nuclei config file at %s", c.getTemplatesConfigFilePath())), err)
return errkit.Newf("could not read nuclei config file at %s: %v", c.getTemplatesConfigFilePath(), err)
}
if err := json.Unmarshal(bin, &cfg); err != nil {
return errkit.Append(errkit.New(fmt.Sprintf("could not unmarshal nuclei config file at %s", c.getTemplatesConfigFilePath())), err)
return errkit.Newf("could not unmarshal nuclei config file at %s: %v", c.getTemplatesConfigFilePath(), err)
}
// apply config
c.TemplatesDirectory = cfg.TemplatesDirectory
@@ -292,10 +292,10 @@ func (c *Config) WriteTemplatesConfig() error {
}
bin, err := json.Marshal(c)
if err != nil {
return errkit.Append(errkit.New("failed to marshal nuclei config"), err)
return errkit.Newf("failed to marshal nuclei config: %v", err)
}
if err = os.WriteFile(c.getTemplatesConfigFilePath(), bin, 0600); err != nil {
return errkit.Append(errkit.New(fmt.Sprintf("failed to write nuclei config file at %s", c.getTemplatesConfigFilePath())), err)
return errkit.Newf("failed to write nuclei config file at %s: %v", c.getTemplatesConfigFilePath(), err)
}
return nil
}
@@ -319,7 +319,7 @@ func (c *Config) getTemplatesConfigFilePath() string {
func (c *Config) createConfigDirIfNotExists() error {
if !fileutil.FolderExists(c.configDir) {
if err := fileutil.CreateFolder(c.configDir); err != nil {
return errkit.Append(errkit.New(fmt.Sprintf("could not create nuclei config directory at %s", c.configDir)), err)
return errkit.Newf("could not create nuclei config directory at %s: %v", c.configDir, err)
}
}
return nil
+14 -15
View File
@@ -3,7 +3,6 @@ package loader
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
@@ -34,27 +33,27 @@ type AITemplateResponse struct {
func getAIGeneratedTemplates(prompt string, options *types.Options) ([]string, error) {
prompt = strings.TrimSpace(prompt)
if len(prompt) < 5 {
return nil, errkit.New("Prompt is too short. Please provide a more descriptive prompt").Build()
return nil, errkit.Newf("Prompt is too short. Please provide a more descriptive prompt")
}
if len(prompt) > 3000 {
return nil, errkit.New("Prompt is too long. Please limit to 3000 characters").Build()
return nil, errkit.Newf("Prompt is too long. Please limit to 3000 characters")
}
template, templateID, err := generateAITemplate(prompt)
if err != nil {
return nil, errkit.New(fmt.Sprintf("Failed to generate template: %v", err)).Build()
return nil, errkit.Newf("Failed to generate template: %v", err)
}
pdcpTemplateDir := filepath.Join(config.DefaultConfig.GetTemplateDir(), "pdcp")
if err := os.MkdirAll(pdcpTemplateDir, 0755); err != nil {
return nil, errkit.New(fmt.Sprintf("Failed to create pdcp template directory: %v", err)).Build()
return nil, errkit.Newf("Failed to create pdcp template directory: %v", err)
}
templateFile := filepath.Join(pdcpTemplateDir, templateID+".yaml")
err = os.WriteFile(templateFile, []byte(template), 0644)
if err != nil {
return nil, errkit.New(fmt.Sprintf("Failed to generate template: %v", err)).Build()
return nil, errkit.Newf("Failed to generate template: %v", err)
}
options.Logger.Info().Msgf("Generated template available at: https://cloud.projectdiscovery.io/templates/%s", templateID)
@@ -92,22 +91,22 @@ func generateAITemplate(prompt string) (string, string, error) {
}
jsonBody, err := json.Marshal(reqBody)
if err != nil {
return "", "", errkit.New(fmt.Sprintf("Failed to marshal request body: %v", err)).Build()
return "", "", errkit.Newf("Failed to marshal request body: %v", err)
}
req, err := http.NewRequest(http.MethodPost, aiTemplateGeneratorAPIEndpoint, bytes.NewBuffer(jsonBody))
if err != nil {
return "", "", errkit.New(fmt.Sprintf("Failed to create HTTP request: %v", err)).Build()
return "", "", errkit.Newf("Failed to create HTTP request: %v", err)
}
ph := pdcpauth.PDCPCredHandler{}
creds, err := ph.GetCreds()
if err != nil {
return "", "", errkit.New(fmt.Sprintf("Failed to get PDCP credentials: %v", err)).Build()
return "", "", errkit.Newf("Failed to get PDCP credentials: %v", err)
}
if creds == nil {
return "", "", errkit.New("PDCP API Key not configured, Create one for free at https://cloud.projectdiscovery.io/").Build()
return "", "", errkit.Newf("PDCP API Key not configured, Create one for free at https://cloud.projectdiscovery.io/")
}
req.Header.Set("Content-Type", "application/json")
@@ -115,28 +114,28 @@ func generateAITemplate(prompt string) (string, string, error) {
resp, err := retryablehttp.DefaultClient().Do(req)
if err != nil {
return "", "", errkit.New(fmt.Sprintf("Failed to send HTTP request: %v", err)).Build()
return "", "", errkit.Newf("Failed to send HTTP request: %v", err)
}
defer func() {
_ = resp.Body.Close()
}()
if resp.StatusCode == http.StatusUnauthorized {
return "", "", errkit.New("Invalid API Key or API Key not configured, Create one for free at https://cloud.projectdiscovery.io/").Build()
return "", "", errkit.Newf("Invalid API Key or API Key not configured, Create one for free at https://cloud.projectdiscovery.io/")
}
if resp.StatusCode != http.StatusOK {
body, _ := io.ReadAll(resp.Body)
return "", "", errkit.New(fmt.Sprintf("API returned status code %d: %s", resp.StatusCode, string(body))).Build()
return "", "", errkit.Newf("API returned status code %d: %s", resp.StatusCode, string(body))
}
var result AITemplateResponse
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
return "", "", errkit.New(fmt.Sprintf("Failed to decode API response: %v", err)).Build()
return "", "", errkit.Newf("Failed to decode API response: %v", err)
}
if result.TemplateID == "" || result.Completion == "" {
return "", "", errkit.New("Failed to generate template").Build()
return "", "", errkit.Newf("Failed to generate template")
}
return result.Completion, result.TemplateID, nil
+1 -1
View File
@@ -238,7 +238,7 @@ func (store *Store) ReadTemplateFromURI(uri string, remote bool) ([]byte, error)
uri = handleTemplatesEditorURLs(uri)
remoteTemplates, _, err := getRemoteTemplatesAndWorkflows([]string{uri}, nil, store.config.RemoteTemplateDomainList)
if err != nil || len(remoteTemplates) == 0 {
return nil, errkit.Append(errkit.New(fmt.Sprintf("Could not load template %s: got %v", uri, remoteTemplates)), err)
return nil, errkit.Wrapf(err, "Could not load template %s: got %v", uri, remoteTemplates)
}
resp, err := retryablehttp.Get(remoteTemplates[0])
if err != nil {
+3 -2
View File
@@ -3,7 +3,6 @@ package customtemplates
import (
"bytes"
"context"
"fmt"
"os"
"path/filepath"
"strings"
@@ -30,7 +29,9 @@ func NewAzureProviders(options *types.Options) ([]*customTemplateAzureBlob, erro
// Establish a connection to Azure and build a client object with which to download templates from Azure Blob Storage
azClient, err := getAzureBlobClient(options.AzureTenantID, options.AzureClientID, options.AzureClientSecret, options.AzureServiceURL)
if err != nil {
return nil, errkit.Append(errkit.New(fmt.Sprintf("Error establishing Azure Blob client for %s", options.AzureContainerName)), err)
errx := errkit.FromError(err)
errx.Msgf("Error establishing Azure Blob client for %s", options.AzureContainerName)
return nil, errx
}
// Create a new Azure Blob Storage container object
+3 -2
View File
@@ -3,7 +3,6 @@ package customtemplates
import (
"context"
"encoding/base64"
"fmt"
"os"
"path/filepath"
@@ -29,7 +28,9 @@ func NewGitLabProviders(options *types.Options) ([]*customTemplateGitLabRepo, er
// Establish a connection to GitLab and build a client object with which to download templates from GitLab
gitLabClient, err := getGitLabClient(options.GitLabServerURL, options.GitLabToken)
if err != nil {
return nil, errkit.Append(errkit.New(fmt.Sprintf("Error establishing GitLab client for %s %s", options.GitLabServerURL, err)), err)
errx := errkit.FromError(err)
errx.Msgf("Error establishing GitLab client for %s %s", options.GitLabServerURL, err)
return nil, errx
}
// Create a new GitLab service client
+3 -2
View File
@@ -2,7 +2,6 @@ package customtemplates
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
@@ -65,7 +64,9 @@ func NewS3Providers(options *types.Options) ([]*customTemplateS3Bucket, error) {
if options.AwsBucketName != "" && !options.AwsTemplateDisableDownload {
s3c, err := getS3Client(context.TODO(), options.AwsAccessKey, options.AwsSecretKey, options.AwsRegion, options.AwsProfile)
if err != nil {
return nil, errkit.Append(errkit.New(fmt.Sprintf("error downloading s3 bucket %s", options.AwsBucketName)), err)
errx := errkit.FromError(err)
errx.Msgf("error downloading s3 bucket %s", options.AwsBucketName)
return nil, errx
}
ctBucket := &customTemplateS3Bucket{
bucketName: options.AwsBucketName,
+12 -4
View File
@@ -38,7 +38,9 @@ func NewCustomTemplatesManager(options *types.Options) (*CustomTemplatesManager,
// Add GitHub providers
githubProviders, err := NewGitHubProviders(options)
if err != nil {
return nil, errkit.Append(errkit.New("could not create github providers for custom templates"), err)
errx := errkit.FromError(err)
errx.Msgf("could not create github providers for custom templates")
return nil, errx
}
for _, v := range githubProviders {
ctm.providers = append(ctm.providers, v)
@@ -47,7 +49,9 @@ func NewCustomTemplatesManager(options *types.Options) (*CustomTemplatesManager,
// Add AWS S3 providers
s3Providers, err := NewS3Providers(options)
if err != nil {
return nil, errkit.Append(errkit.New("could not create s3 providers for custom templates"), err)
errx := errkit.FromError(err)
errx.Msgf("could not create s3 providers for custom templates")
return nil, errx
}
for _, v := range s3Providers {
ctm.providers = append(ctm.providers, v)
@@ -56,7 +60,9 @@ func NewCustomTemplatesManager(options *types.Options) (*CustomTemplatesManager,
// Add Azure providers
azureProviders, err := NewAzureProviders(options)
if err != nil {
return nil, errkit.Append(errkit.New("could not create azure providers for custom templates"), err)
errx := errkit.FromError(err)
errx.Msgf("could not create azure providers for custom templates")
return nil, errx
}
for _, v := range azureProviders {
ctm.providers = append(ctm.providers, v)
@@ -65,7 +71,9 @@ func NewCustomTemplatesManager(options *types.Options) (*CustomTemplatesManager,
// Add GitLab providers
gitlabProviders, err := NewGitLabProviders(options)
if err != nil {
return nil, errkit.Append(errkit.New("could not create gitlab providers for custom templates"), err)
errx := errkit.FromError(err)
errx.Msgf("could not create gitlab providers for custom templates")
return nil, errx
}
for _, v := range gitlabProviders {
ctm.providers = append(ctm.providers, v)
+48 -35
View File
@@ -7,7 +7,6 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/fuzz/dataformat"
"github.com/projectdiscovery/retryablehttp-go"
mapsutil "github.com/projectdiscovery/utils/maps"
urlutil "github.com/projectdiscovery/utils/url"
)
@@ -38,12 +37,18 @@ func (q *Path) Parse(req *retryablehttp.Request) (bool, error) {
splitted := strings.Split(req.Path, "/")
values := make(map[string]interface{})
for i := range splitted {
pathTillNow := strings.Join(splitted[:i+1], "/")
if pathTillNow == "" {
for i, segment := range splitted {
if segment == "" && i == 0 {
// Skip the first empty segment from leading "/"
continue
}
values[strconv.Itoa(i)] = pathTillNow
if segment == "" {
// Skip any other empty segments
continue
}
// Use 1-based indexing and store individual segments
key := strconv.Itoa(len(values) + 1)
values[key] = segment
}
q.value.SetParsed(dataformat.KVMap(values), "")
return true, nil
@@ -64,7 +69,7 @@ func (q *Path) Iterate(callback func(key string, value interface{}) error) (err
// SetValue sets a value in the component
// for a key
func (q *Path) SetValue(key string, value string) error {
escaped := urlutil.ParamEncode(value)
escaped := urlutil.PathEncode(value)
if !q.value.SetParsedValue(key, escaped) {
return ErrSetValue
}
@@ -82,40 +87,48 @@ func (q *Path) Delete(key string) error {
// Rebuild returns a new request with the
// component rebuilt
func (q *Path) Rebuild() (*retryablehttp.Request, error) {
originalValues := mapsutil.Map[string, any]{}
splitted := strings.Split(q.req.Path, "/")
for i := range splitted {
pathTillNow := strings.Join(splitted[:i+1], "/")
if pathTillNow == "" {
continue
}
originalValues[strconv.Itoa(i)] = pathTillNow
// Get the original path segments
originalSplitted := strings.Split(q.req.Path, "/")
// Create a new slice to hold the rebuilt segments
rebuiltSegments := make([]string, 0, len(originalSplitted))
// Add the first empty segment (from leading "/")
if len(originalSplitted) > 0 && originalSplitted[0] == "" {
rebuiltSegments = append(rebuiltSegments, "")
}
originalPath := q.req.Path
lengthSplitted := len(q.value.parsed.Map)
for i := lengthSplitted; i > 0; i-- {
key := strconv.Itoa(i)
original, ok := originalValues.GetOrDefault(key, "").(string)
if !ok {
// Process each segment
segmentIndex := 1 // 1-based indexing for our stored values
for i := 1; i < len(originalSplitted); i++ {
originalSegment := originalSplitted[i]
if originalSegment == "" {
// Skip empty segments
continue
}
new, ok := q.value.parsed.Map.GetOrDefault(key, "").(string)
if !ok {
continue
// Check if we have a replacement for this segment
key := strconv.Itoa(segmentIndex)
if newValue, exists := q.value.parsed.Map.GetOrDefault(key, "").(string); exists && newValue != "" {
rebuiltSegments = append(rebuiltSegments, newValue)
} else {
rebuiltSegments = append(rebuiltSegments, originalSegment)
}
if new == original {
// no need to replace
continue
}
originalPath = strings.Replace(originalPath, original, new, 1)
segmentIndex++
}
// Join the segments back into a path
rebuiltPath := strings.Join(rebuiltSegments, "/")
if unescaped, err := urlutil.PathDecode(rebuiltPath); err == nil {
// this is handle the case where anyportion of path has url encoded data
// by default the http/request official library will escape/encode special characters in path
// to avoid double encoding we unescape/decode already encoded value
//
// if there is a invalid url encoded value like %99 then it will still be encoded as %2599 and not %99
// the only way to make sure it stays as %99 is to implement raw request and unsafe for fuzzing as well
rebuiltPath = unescaped
}
rebuiltPath := originalPath
// Clone the request and update the path
cloned := q.req.Clone(context.Background())
+54 -5
View File
@@ -29,9 +29,9 @@ func TestURLComponent(t *testing.T) {
})
require.Equal(t, []string{"1"}, keys, "unexpected keys")
require.Equal(t, []string{"/testpath"}, values, "unexpected values")
require.Equal(t, []string{"testpath"}, values, "unexpected values")
err = urlComponent.SetValue("1", "/newpath")
err = urlComponent.SetValue("1", "newpath")
if err != nil {
t.Fatal(err)
}
@@ -61,9 +61,10 @@ func TestURLComponent_NestedPaths(t *testing.T) {
isSet := false
_ = path.Iterate(func(key string, value interface{}) error {
if !isSet && value.(string) == "/user/753" {
t.Logf("Key: %s, Value: %s", key, value.(string))
if !isSet && value.(string) == "753" {
isSet = true
if setErr := path.SetValue(key, "/user/753'"); setErr != nil {
if setErr := path.SetValue(key, "753'"); setErr != nil {
t.Fatal(setErr)
}
}
@@ -75,6 +76,54 @@ func TestURLComponent_NestedPaths(t *testing.T) {
t.Fatal(err)
}
if newReq.Path != "/user/753'/profile" {
t.Fatal("expected path to be modified")
t.Fatalf("expected path to be '/user/753'/profile', got '%s'", newReq.Path)
}
}
func TestPathComponent_SQLInjection(t *testing.T) {
path := NewPath()
req, err := retryablehttp.NewRequest(http.MethodGet, "https://example.com/user/55/profile", nil)
if err != nil {
t.Fatal(err)
}
found, err := path.Parse(req)
if err != nil {
t.Fatal(err)
}
if !found {
t.Fatal("expected path to be found")
}
t.Logf("Original path: %s", req.Path)
// Let's see what path segments are available for fuzzing
err = path.Iterate(func(key string, value interface{}) error {
t.Logf("Key: %s, Value: %s", key, value.(string))
// Try fuzzing the "55" segment specifically (which should be key "2")
if value.(string) == "55" {
if setErr := path.SetValue(key, "55 OR True"); setErr != nil {
t.Fatal(setErr)
}
}
return nil
})
if err != nil {
t.Fatal(err)
}
newReq, err := path.Rebuild()
if err != nil {
t.Fatal(err)
}
t.Logf("Modified path: %s", newReq.Path)
// Now with PathEncode, spaces are preserved correctly for SQL injection
if newReq.Path != "/user/55 OR True/profile" {
t.Fatalf("expected path to be '/user/55 OR True/profile', got '%s'", newReq.Path)
}
// Let's also test what the actual URL looks like
t.Logf("Full URL: %s", newReq.String())
}
+45 -26
View File
@@ -49,42 +49,61 @@ func (m *MultiPartForm) Encode(data KV) (string, error) {
var fw io.Writer
var err error
if filesArray, ok := value.([]interface{}); ok {
fileMetadata, ok := m.filesMetadata[key]
if !ok {
Itererr = fmt.Errorf("file metadata not found for key %s", key)
return false
}
if fileMetadata, ok := m.filesMetadata[key]; ok {
if filesArray, isArray := value.([]any); isArray {
for _, file := range filesArray {
h := make(textproto.MIMEHeader)
h.Set("Content-Disposition",
fmt.Sprintf(`form-data; name=%q; filename=%q`,
key, fileMetadata.Filename))
h.Set("Content-Type", fileMetadata.ContentType)
for _, file := range filesArray {
h := make(textproto.MIMEHeader)
h.Set("Content-Disposition",
fmt.Sprintf(`form-data; name=%q; filename=%q`,
key, fileMetadata.Filename))
h.Set("Content-Type", fileMetadata.ContentType)
if fw, err = w.CreatePart(h); err != nil {
Itererr = err
return false
}
if fw, err = w.CreatePart(h); err != nil {
Itererr = err
return false
if _, err = fw.Write([]byte(file.(string))); err != nil {
Itererr = err
return false
}
}
if _, err = fw.Write([]byte(file.(string))); err != nil {
Itererr = err
return false
}
return true
}
return true
}
// Add field
if fw, err = w.CreateFormField(key); err != nil {
Itererr = err
return false
var values []string
switch v := value.(type) {
case nil:
values = []string{""}
case string:
values = []string{v}
case []string:
values = v
case []any:
values = make([]string, len(v))
for i, item := range v {
if item == nil {
values[i] = ""
} else {
values[i] = fmt.Sprint(item)
}
}
default:
values = []string{fmt.Sprintf("%v", v)}
}
if _, err = fw.Write([]byte(value.(string))); err != nil {
Itererr = err
return false
for _, val := range values {
if fw, err = w.CreateFormField(key); err != nil {
Itererr = err
return false
}
if _, err = fw.Write([]byte(val)); err != nil {
Itererr = err
return false
}
}
return true
})
+242
View File
@@ -0,0 +1,242 @@
package dataformat
import (
"testing"
mapsutil "github.com/projectdiscovery/utils/maps"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestMultiPartFormEncode(t *testing.T) {
tests := []struct {
name string
fields map[string]any
wantErr bool
expected map[string]any
}{
{
name: "duplicate fields ([]string) - checkbox scenario",
fields: map[string]any{
"interests": []string{"sports", "music", "reading"},
"colors": []string{"red", "blue"},
},
expected: map[string]any{
"interests": []string{"sports", "music", "reading"},
"colors": []string{"red", "blue"},
},
},
{
name: "single string fields - backward compatibility",
fields: map[string]any{
"username": "john",
"email": "[email protected]",
},
expected: map[string]any{
"username": "john",
"email": "[email protected]",
},
},
{
name: "mixed types",
fields: map[string]any{
"string": "text",
"array": []string{"item1", "item2"},
"number": 42, // tests fmt.Sprint fallback
"float": 3.14, // tests float conversion
"boolean": true, // tests boolean conversion
"zero": 0, // tests zero value
"emptyStr": "", // tests empty string
"negative": -123, // tests negative number
"nil": nil, // tests nil value
"mixedArray": []any{"str", 123, false, nil}, // tests mixed type array
},
expected: map[string]any{
"string": "text",
"array": []string{"item1", "item2"},
"number": "42", // numbers are converted to strings in multipart
"float": "3.14", // floats are converted to strings
"boolean": "true", // booleans are converted to strings
"zero": "0", // zero value converted to string
"emptyStr": "", // empty string remains empty
"negative": "-123", // negative numbers converted to strings
"nil": "", // nil values converted to "" string
"mixedArray": []string{"str", "123", "false", ""}, // mixed array converted to string array
},
},
{
name: "empty array - should not appear in output",
fields: map[string]any{
"emptyArray": []string{},
"normalField": "value",
},
expected: map[string]any{
"normalField": "value",
// emptyArray should not appear in decoded output
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
defer func() {
if r := recover(); r != nil {
t.Errorf("Test panicked: %v", r)
}
}()
form := NewMultiPartForm()
form.boundary = "----WebKitFormBoundary7MA4YWxkTrZu0gW"
kv := mapsutil.NewOrderedMap[string, any]()
for k, v := range tt.fields {
kv.Set(k, v)
}
encoded, err := form.Encode(KVOrderedMap(&kv))
if tt.wantErr {
require.Error(t, err)
return
}
require.NoError(t, err)
// Decode the encoded multipart data
decoded, err := form.Decode(encoded)
require.NoError(t, err)
// Compare decoded values with expected values
for expectedKey, expectedValue := range tt.expected {
actualValue := decoded.Get(expectedKey)
switch expected := expectedValue.(type) {
case []string:
actual, ok := actualValue.([]string)
require.True(t, ok, "Expected []string for key %s, got %T", expectedKey, actualValue)
assert.ElementsMatch(t, expected, actual, "Values mismatch for key %s", expectedKey)
case []any:
actual, ok := actualValue.([]any)
require.True(t, ok, "Expected []any for key %s, got %T", expectedKey, actualValue)
assert.ElementsMatch(t, expected, actual, "Values mismatch for key %s", expectedKey)
case string:
actual, ok := actualValue.(string)
require.True(t, ok, "Expected string for key %s, got %T", expectedKey, actualValue)
assert.Equal(t, expected, actual, "Values mismatch for key %s", expectedKey)
default:
assert.Equal(t, expected, actualValue, "Values mismatch for key %s", expectedKey)
}
}
// Ensure no unexpected keys are present in decoded output
decoded.Iterate(func(key string, value any) bool {
_, exists := tt.expected[key]
assert.True(t, exists, "Unexpected key %s found in decoded output", key)
return true
})
t.Logf("Encoded output:\n%s", encoded)
})
}
}
func TestMultiPartFormRoundTrip(t *testing.T) {
defer func() {
if r := recover(); r != nil {
t.Errorf("Test panicked: %v", r)
}
}()
form := NewMultiPartForm()
form.boundary = "----WebKitFormBoundary7MA4YWxkTrZu0gW"
original := mapsutil.NewOrderedMap[string, any]()
original.Set("username", "john")
original.Set("interests", []string{"sports", "music", "reading"})
encoded, err := form.Encode(KVOrderedMap(&original))
require.NoError(t, err)
decoded, err := form.Decode(encoded)
require.NoError(t, err)
assert.Equal(t, "john", decoded.Get("username"))
assert.ElementsMatch(t, []string{"sports", "music", "reading"}, decoded.Get("interests"))
t.Logf("Encoded output:\n%s", encoded)
}
func TestMultiPartFormFileUpload(t *testing.T) {
defer func() {
if r := recover(); r != nil {
t.Errorf("Test panicked: %v", r)
}
}()
// Test decoding of a manually crafted multipart form with files
form := NewMultiPartForm()
form.boundary = "----WebKitFormBoundaryFileUploadTest"
// Manually craft a multipart form with file uploads
multipartData := `------WebKitFormBoundaryFileUploadTest
Content-Disposition: form-data; name="name"
John Doe
------WebKitFormBoundaryFileUploadTest
Content-Disposition: form-data; name="email"
[email protected]
------WebKitFormBoundaryFileUploadTest
Content-Disposition: form-data; name="profile_picture"; filename="profile.jpg"
Content-Type: image/jpeg
fake_jpeg_binary_data_here
------WebKitFormBoundaryFileUploadTest
Content-Disposition: form-data; name="documents"; filename="resume.pdf"
Content-Type: application/pdf
fake_pdf_content_1
------WebKitFormBoundaryFileUploadTest
Content-Disposition: form-data; name="documents"; filename="cover_letter.pdf"
Content-Type: application/pdf
fake_pdf_content_2
------WebKitFormBoundaryFileUploadTest
Content-Disposition: form-data; name="skills"
Go
------WebKitFormBoundaryFileUploadTest
Content-Disposition: form-data; name="skills"
JavaScript
------WebKitFormBoundaryFileUploadTest
Content-Disposition: form-data; name="skills"
Python
------WebKitFormBoundaryFileUploadTest--
`
// Test decoding
decoded, err := form.Decode(multipartData)
require.NoError(t, err)
// Verify regular fields
assert.Equal(t, "John Doe", decoded.Get("name"))
assert.Equal(t, "[email protected]", decoded.Get("email"))
assert.Equal(t, []string{"Go", "JavaScript", "Python"}, decoded.Get("skills"))
// Verify file fields
profilePicture := decoded.Get("profile_picture")
require.NotNil(t, profilePicture)
profileArray, ok := profilePicture.([]interface{})
require.True(t, ok, "Expected []interface{} for profile_picture")
require.Len(t, profileArray, 1)
assert.Equal(t, "fake_jpeg_binary_data_here", profileArray[0])
documents := decoded.Get("documents")
require.NotNil(t, documents)
documentsArray, ok := documents.([]interface{})
require.True(t, ok, "Expected []interface{} for documents")
require.Len(t, documentsArray, 2)
assert.Contains(t, documentsArray, "fake_pdf_content_1")
assert.Contains(t, documentsArray, "fake_pdf_content_2")
}
+6 -7
View File
@@ -23,10 +23,9 @@ import (
urlutil "github.com/projectdiscovery/utils/url"
)
// ErrRuleNotApplicable returns a rule not applicable error
func ErrRuleNotApplicable(reason interface{}) error {
return errkit.New(fmt.Sprintf("rule not applicable: %v", reason)).Build()
}
var (
ErrRuleNotApplicable = errkit.New("rule not applicable")
)
// IsErrRuleNotApplicable checks if an error is due to rule not applicable
func IsErrRuleNotApplicable(err error) bool {
@@ -90,10 +89,10 @@ type GeneratedRequest struct {
// goroutines.
func (rule *Rule) Execute(input *ExecuteRuleInput) (err error) {
if !rule.isInputURLValid(input.Input) {
return ErrRuleNotApplicable(fmt.Sprintf("invalid input url: %v", input.Input.MetaInput.Input))
return errkit.Newf("rule not applicable: invalid input url: %v", input.Input.MetaInput.Input)
}
if input.BaseRequest == nil && input.Input.MetaInput.ReqResp == nil {
return ErrRuleNotApplicable(fmt.Sprintf("both base request and reqresp are nil for %v", input.Input.MetaInput.Input))
return errkit.Newf("rule not applicable: both base request and reqresp are nil for %v", input.Input.MetaInput.Input)
}
var finalComponentList []component.Component
@@ -145,7 +144,7 @@ func (rule *Rule) Execute(input *ExecuteRuleInput) (err error) {
}
if len(finalComponentList) == 0 {
return ErrRuleNotApplicable("no component matched on this rule")
return errkit.Newf("rule not applicable: no component matched on this rule")
}
baseValues := input.Values
+8 -8
View File
@@ -395,7 +395,7 @@ func generateRequestsFromOp(opts *generateReqOptions) error {
func GetGlobalParamsForSecurityRequirement(schema *openapi3.T, requirement *openapi3.SecurityRequirements) ([]*openapi3.ParameterRef, error) {
globalParams := openapi3.NewParameters()
if len(schema.Components.SecuritySchemes) == 0 {
return nil, errkit.New(fmt.Sprintf("openapi: security requirements (%+v) without any security schemes found in openapi file", schema.Security)).Build()
return nil, errkit.Newf("security requirements (%+v) without any security schemes found in openapi file", schema.Security)
}
found := false
// this api is protected for each security scheme pull its corresponding scheme
@@ -415,11 +415,11 @@ schemaLabel:
}
if !found && len(security) > 1 {
// if this is case then both security schemes are required
return nil, errkit.New(fmt.Sprintf("openapi: security requirement (%+v) not found in openapi file", security)).Build()
return nil, errkit.Newf("security requirement (%+v) not found in openapi file", security)
}
}
if !found {
return nil, errkit.New(fmt.Sprintf("openapi: security requirement (%+v) not found in openapi file", requirement)).Build()
return nil, errkit.Newf("security requirement (%+v) not found in openapi file", requirement)
}
return globalParams, nil
@@ -428,12 +428,12 @@ schemaLabel:
// GenerateParameterFromSecurityScheme generates an example from a schema object
func GenerateParameterFromSecurityScheme(scheme *openapi3.SecuritySchemeRef) (*openapi3.Parameter, error) {
if !generic.EqualsAny(scheme.Value.Type, "http", "apiKey") {
return nil, errkit.New(fmt.Sprintf("openapi: unsupported security scheme type (%s) found in openapi file", scheme.Value.Type)).Build()
return nil, errkit.Newf("unsupported security scheme type (%s) found in openapi file", scheme.Value.Type)
}
if scheme.Value.Type == "http" {
// check scheme
if !generic.EqualsAny(scheme.Value.Scheme, "basic", "bearer") {
return nil, errkit.New(fmt.Sprintf("openapi: unsupported security scheme (%s) found in openapi file", scheme.Value.Scheme)).Build()
return nil, errkit.Newf("unsupported security scheme (%s) found in openapi file", scheme.Value.Scheme)
}
// HTTP authentication schemes basic or bearer use the Authorization header
headerName := scheme.Value.Name
@@ -458,10 +458,10 @@ func GenerateParameterFromSecurityScheme(scheme *openapi3.SecuritySchemeRef) (*o
if scheme.Value.Type == "apiKey" {
// validate name and in
if scheme.Value.Name == "" {
return nil, errkit.New(fmt.Sprintf("openapi: security scheme (%s) name is empty", scheme.Value.Type)).Build()
return nil, errkit.Newf("security scheme (%s) name is empty", scheme.Value.Type)
}
if !generic.EqualsAny(scheme.Value.In, "query", "header", "cookie") {
return nil, errkit.New(fmt.Sprintf("openapi: unsupported security scheme (%s) in (%s) found in openapi file", scheme.Value.Type, scheme.Value.In)).Build()
return nil, errkit.Newf("unsupported security scheme (%s) in (%s) found in openapi file", scheme.Value.Type, scheme.Value.In)
}
// create parameters using the scheme
switch scheme.Value.In {
@@ -482,5 +482,5 @@ func GenerateParameterFromSecurityScheme(scheme *openapi3.SecuritySchemeRef) (*o
return c, nil
}
}
return nil, errkit.New(fmt.Sprintf("openapi: unsupported security scheme type (%s) found in openapi file", scheme.Value.Type)).Build()
return nil, errkit.Newf("unsupported security scheme type (%s) found in openapi file", scheme.Value.Type)
}
+2 -6
View File
@@ -18,14 +18,10 @@ import (
)
var (
ErrInactiveInput = fmt.Errorf("input is inactive")
ErrNotImplemented = errkit.New("provider does not implement method")
ErrInactiveInput = fmt.Errorf("input is inactive")
)
// ErrNotImplemented returns an error when a provider does not implement a method
func ErrNotImplemented(provider, method string) error {
return errkit.New(fmt.Sprintf("provider %s does not implement %s", provider, method)).Build()
}
const (
MultiFormatInputProvider = "MultiFormatInputProvider"
ListInputProvider = "ListInputProvider"
+15 -18
View File
@@ -80,7 +80,7 @@ func (t *TemplateManager) FreshInstallIfNotExists() error {
}
gologger.Info().Msgf("nuclei-templates are not installed, installing...")
if err := t.installTemplatesAt(config.DefaultConfig.TemplatesDirectory); err != nil {
return errkit.Append(errkit.New(fmt.Sprintf("failed to install templates at %s", config.DefaultConfig.TemplatesDirectory)), err)
return errkit.Wrapf(err, "failed to install templates at %s", config.DefaultConfig.TemplatesDirectory)
}
if t.CustomTemplates != nil {
t.CustomTemplates.Download(context.TODO())
@@ -121,7 +121,7 @@ func (t *TemplateManager) UpdateIfOutdated() error {
func (t *TemplateManager) installTemplatesAt(dir string) error {
if !fileutil.FolderExists(dir) {
if err := fileutil.CreateFolder(dir); err != nil {
return errkit.Append(errkit.New(fmt.Sprintf("failed to create directory at %s", dir)), err)
return errkit.Wrapf(err, "failed to create directory at %s", dir)
}
}
if t.DisablePublicTemplates {
@@ -130,12 +130,12 @@ func (t *TemplateManager) installTemplatesAt(dir string) error {
}
ghrd, err := updateutils.NewghReleaseDownloader(config.OfficialNucleiTemplatesRepoName)
if err != nil {
return errkit.Append(errkit.New(fmt.Sprintf("failed to install templates at %s", dir)), err)
return errkit.Wrapf(err, "failed to install templates at %s", dir)
}
// write templates to disk
if err := t.writeTemplatesToDisk(ghrd, dir); err != nil {
return errkit.Append(errkit.New(fmt.Sprintf("failed to write templates to disk at %s", dir)), err)
return errkit.Wrapf(err, "failed to write templates to disk at %s", dir)
}
gologger.Info().Msgf("Successfully installed nuclei-templates at %s", dir)
return nil
@@ -156,7 +156,7 @@ func (t *TemplateManager) updateTemplatesAt(dir string) error {
ghrd, err := updateutils.NewghReleaseDownloader(config.OfficialNucleiTemplatesRepoName)
if err != nil {
return errkit.Append(errkit.New(fmt.Sprintf("failed to install templates at %s", dir)), err)
return errkit.Wrapf(err, "failed to install templates at %s", dir)
}
latestVersion := ghrd.Latest.GetTagName()
@@ -177,7 +177,7 @@ func (t *TemplateManager) updateTemplatesAt(dir string) error {
newchecksums, err := t.getChecksumFromDir(dir)
if err != nil {
// unlikely this case will happen
return errkit.Append(errkit.New(fmt.Sprintf("failed to get checksums from %s after update", dir)), err)
return errkit.Wrapf(err, "failed to get checksums from %s after update", dir)
}
// summarize all changes
@@ -299,7 +299,7 @@ func (t *TemplateManager) writeTemplatesToDisk(ghrd *updateutils.GHReleaseDownlo
bin, err := io.ReadAll(r)
if err != nil {
// if error occurs, iteration also stops
return errkit.Append(errkit.New(fmt.Sprintf("failed to read file %s", uri)), err)
return errkit.Wrapf(err, "failed to read file %s", uri)
}
// TODO: It might be better to just download index file from nuclei templates repo
// instead of creating it from scratch
@@ -310,7 +310,7 @@ func (t *TemplateManager) writeTemplatesToDisk(ghrd *updateutils.GHReleaseDownlo
if oldPath != writePath {
// write new template at a new path and delete old template
if err := os.WriteFile(writePath, bin, f.Mode()); err != nil {
return errkit.Append(errkit.New(fmt.Sprintf("failed to write file %s", uri)), err)
return errkit.Wrapf(err, "failed to write file %s", uri)
}
// after successful write, remove old template
if err := os.Remove(oldPath); err != nil {
@@ -325,20 +325,20 @@ func (t *TemplateManager) writeTemplatesToDisk(ghrd *updateutils.GHReleaseDownlo
}
err = ghrd.DownloadSourceWithCallback(!HideProgressBar, callbackFunc)
if err != nil {
return errkit.Append(errkit.New("failed to download templates"), err)
return errkit.Wrap(err, "failed to download templates")
}
if err := config.DefaultConfig.WriteTemplatesConfig(); err != nil {
return errkit.Append(errkit.New("failed to write templates config"), err)
return errkit.Wrap(err, "failed to write templates config")
}
// update ignore hash after writing new templates
if err := config.DefaultConfig.UpdateNucleiIgnoreHash(); err != nil {
return errkit.Append(errkit.New("failed to update nuclei ignore hash"), err)
return errkit.Wrap(err, "failed to update nuclei ignore hash")
}
// update templates version in config file
if err := config.DefaultConfig.SetTemplatesVersion(ghrd.Latest.GetTagName()); err != nil {
return errkit.Append(errkit.New("failed to update templates version"), err)
return errkit.Wrap(err, "failed to update templates version")
}
PurgeEmptyDirectories(dir)
@@ -348,11 +348,11 @@ func (t *TemplateManager) writeTemplatesToDisk(ghrd *updateutils.GHReleaseDownlo
index, err := config.GetNucleiTemplatesIndex()
if err != nil {
return errkit.Append(errkit.New("failed to get nuclei templates index"), err)
return errkit.Wrap(err, "failed to get nuclei templates index")
}
if err = config.DefaultConfig.WriteTemplatesIndex(index); err != nil {
return errkit.Append(errkit.New("failed to write nuclei templates index"), err)
return errkit.Wrap(err, "failed to write nuclei templates index")
}
if !HideReleaseNotes {
@@ -448,8 +448,5 @@ func (t *TemplateManager) calculateChecksumMap(dir string) (map[string]string, e
}
return nil
})
if err != nil {
return nil, errkit.Append(errkit.New("failed to calculate checksums of templates"), err)
}
return checksumMap, nil
return checksumMap, errkit.Wrap(err, "failed to calculate checksums of templates")
}
+1 -1
View File
@@ -52,7 +52,7 @@ func getNewAdditionsFileFromGitHub(version string) ([]string, error) {
return nil, err
}
if resp.StatusCode != http.StatusOK {
return nil, errkit.New("version not found").Build()
return nil, errkit.New("version not found")
}
data, err := io.ReadAll(resp.Body)
if err != nil {
+3 -3
View File
@@ -21,17 +21,17 @@ func (p *EntityParser) scrapeAndCreate(typeName string) error {
// get package
pkg, ok := p.imports[pkgName]
if !ok {
return errkit.New(fmt.Sprintf("package %v for type %v not found", pkgName, typeName)).Build()
return errkit.Newf("package %v for type %v not found", pkgName, typeName)
}
// get type
obj := pkg.Types.Scope().Lookup(baseTypeName)
if obj == nil {
return errkit.New(fmt.Sprintf("type %v not found in package %+v", typeName, pkg)).Build()
return errkit.Newf("type %v not found in package %+v", typeName, pkg)
}
// Ensure the object is a type name
typeNameObj, ok := obj.(*types.TypeName)
if !ok {
return errkit.New(fmt.Sprintf("%v is not a type name", typeName)).Build()
return errkit.Newf("%v is not a type name", typeName)
}
// Ensure the type is a named struct type
namedStruct, ok := typeNameObj.Type().Underlying().(*types.Struct)
+1 -2
View File
@@ -4,7 +4,6 @@ import (
"bytes"
"context"
"embed"
"fmt"
"math/rand"
"net"
"reflect"
@@ -257,7 +256,7 @@ func RegisterNativeScripts(runtime *goja.Runtime) error {
// import default modules
_, err = runtime.RunString(defaultImports)
if err != nil {
return errkit.Append(errkit.New(fmt.Sprintf("could not import default modules %v", defaultImports)), err)
return errkit.Wrapf(err, "could not import default modules %v", defaultImports)
}
return nil
+1 -1
View File
@@ -59,7 +59,7 @@ func wrapModuleFunc(runtime *goja.Runtime, fn interface{}) interface{} {
}
// Only wrap if first parameter is context.Context
if fnType.NumIn() == 0 || fnType.In(0) != reflect.TypeOf((*context.Context)(nil)).Elem() {
if fnType.NumIn() == 0 || fnType.In(0) != reflect.TypeFor[context.Context]() {
return fn // Return original function unchanged if it doesn't have context.Context as first arg
}
+4 -5
View File
@@ -2,7 +2,6 @@ package gojs
import (
"context"
"fmt"
"reflect"
"github.com/Mzack9999/goja"
@@ -10,8 +9,8 @@ import (
)
var (
ErrInvalidFuncOpts = errkit.New("invalid function options: %v").Build()
ErrNilRuntime = errkit.New("runtime is nil").Build()
ErrInvalidFuncOpts = errkit.New("invalid function options")
ErrNilRuntime = errkit.New("runtime is nil")
)
type FuncOpts struct {
@@ -35,7 +34,7 @@ func wrapWithContext(runtime *goja.Runtime, fn interface{}) interface{} {
}
// Only wrap if first parameter is context.Context
if fnType.NumIn() == 0 || fnType.In(0) != reflect.TypeOf((*context.Context)(nil)).Elem() {
if fnType.NumIn() == 0 || fnType.In(0) != reflect.TypeFor[context.Context]() {
return fn // Return original function unchanged if it doesn't have context.Context as first arg
}
@@ -84,7 +83,7 @@ func RegisterFuncWithSignature(runtime *goja.Runtime, opts FuncOpts) error {
return ErrNilRuntime
}
if !opts.valid() {
return errkit.New(fmt.Sprintf("invalid function options: name: %s, signatures: %v, description: %s", opts.Name, opts.Signatures, opts.Description)).Build()
return errkit.Newf("invalid function options: name: %s, signatures: %v, description: %s", opts.Name, opts.Signatures, opts.Description)
}
// Wrap the function with context injection
+3 -3
View File
@@ -63,7 +63,7 @@ func connect(executionId string, host string, port int, username string, passwor
}
if !protocolstate.IsHostAllowed(executionId, host) {
// host is not valid according to network policy
return false, protocolstate.ErrHostDenied(host)
return false, protocolstate.ErrHostDenied.Msgf(host)
}
target := net.JoinHostPort(host, fmt.Sprintf("%d", port))
@@ -118,7 +118,7 @@ func (c *MSSQLClient) IsMssql(ctx context.Context, host string, port int) (bool,
func isMssql(executionId string, host string, port int) (bool, error) {
if !protocolstate.IsHostAllowed(executionId, host) {
// host is not valid according to network policy
return false, protocolstate.ErrHostDenied(host)
return false, protocolstate.ErrHostDenied.Msgf(host)
}
dialer := protocolstate.GetDialersWithId(executionId)
@@ -162,7 +162,7 @@ func (c *MSSQLClient) ExecuteQuery(ctx context.Context, host string, port int, u
}
if !protocolstate.IsHostAllowed(executionId, host) {
// host is not valid according to network policy
return nil, protocolstate.ErrHostDenied(host)
return nil, protocolstate.ErrHostDenied.Msgf(host)
}
target := net.JoinHostPort(host, fmt.Sprintf("%d", port))
+4 -4
View File
@@ -45,7 +45,7 @@ func (c *MySQLClient) IsMySQL(ctx context.Context, host string, port int) (bool,
func isMySQL(executionId string, host string, port int) (bool, error) {
if !protocolstate.IsHostAllowed(executionId, host) {
// host is not valid according to network policy
return false, protocolstate.ErrHostDenied(host)
return false, protocolstate.ErrHostDenied.Msgf(host)
}
dialer := protocolstate.GetDialersWithId(executionId)
if dialer == nil {
@@ -85,7 +85,7 @@ func (c *MySQLClient) Connect(ctx context.Context, host string, port int, userna
executionId := ctx.Value("executionId").(string)
if !protocolstate.IsHostAllowed(executionId, host) {
// host is not valid according to network policy
return false, protocolstate.ErrHostDenied(host)
return false, protocolstate.ErrHostDenied.Msgf(host)
}
// executing queries implies the remote mysql service
@@ -144,7 +144,7 @@ func fingerprintMySQL(executionId string, host string, port int) (MySQLInfo, err
info := MySQLInfo{}
if !protocolstate.IsHostAllowed(executionId, host) {
// host is not valid according to network policy
return info, protocolstate.ErrHostDenied(host)
return info, protocolstate.ErrHostDenied.Msgf(host)
}
dialer := protocolstate.GetDialersWithId(executionId)
if dialer == nil {
@@ -209,7 +209,7 @@ func (c *MySQLClient) ExecuteQueryWithOpts(ctx context.Context, opts MySQLOption
executionId := ctx.Value("executionId").(string)
if !protocolstate.IsHostAllowed(executionId, opts.Host) {
// host is not valid according to network policy
return nil, protocolstate.ErrHostDenied(opts.Host)
return nil, protocolstate.ErrHostDenied.Msgf(opts.Host)
}
// executing queries implies the remote mysql service
+2 -2
View File
@@ -201,7 +201,7 @@ func (c *NetConn) RecvFull(N int) ([]byte, error) {
}
bin, err := reader.ConnReadNWithTimeout(c.conn, int64(N), c.timeout)
if err != nil {
return []byte{}, errkit.Append(errkit.New(fmt.Sprintf("failed to read %d bytes", N)), err)
return []byte{}, errkit.Wrapf(err, "failed to read %d bytes", N)
}
return bin, nil
}
@@ -226,7 +226,7 @@ func (c *NetConn) Recv(N int) ([]byte, error) {
b := make([]byte, N)
n, err := c.conn.Read(b)
if err != nil {
return []byte{}, errkit.Append(errkit.New(fmt.Sprintf("failed to read %d bytes", N)), err)
return []byte{}, errkit.Wrapf(err, "failed to read %d bytes", N)
}
return b[:n], nil
}
+2 -2
View File
@@ -122,7 +122,7 @@ func (c *PGClient) ExecuteQuery(ctx context.Context, host string, port int, user
func executeQuery(executionId string, host string, port int, username string, password string, dbName string, query string) (*utils.SQLResult, error) {
if !protocolstate.IsHostAllowed(executionId, host) {
// host is not valid according to network policy
return nil, protocolstate.ErrHostDenied(host)
return nil, protocolstate.ErrHostDenied.Msgf(host)
}
target := net.JoinHostPort(host, fmt.Sprintf("%d", port))
@@ -179,7 +179,7 @@ func connect(executionId string, host string, port int, username string, passwor
if !protocolstate.IsHostAllowed(executionId, host) {
// host is not valid according to network policy
return false, protocolstate.ErrHostDenied(host)
return false, protocolstate.ErrHostDenied.Msgf(host)
}
target := net.JoinHostPort(host, fmt.Sprintf("%d", port))
+4 -4
View File
@@ -27,7 +27,7 @@ func GetServerInfo(ctx context.Context, host string, port int) (string, error) {
func getServerInfo(executionId string, host string, port int) (string, error) {
if !protocolstate.IsHostAllowed(executionId, host) {
// host is not valid according to network policy
return "", protocolstate.ErrHostDenied(host)
return "", protocolstate.ErrHostDenied.Msgf(host)
}
// create a new client
client := redis.NewClient(&redis.Options{
@@ -69,7 +69,7 @@ func Connect(ctx context.Context, host string, port int, password string) (bool,
func connect(executionId string, host string, port int, password string) (bool, error) {
if !protocolstate.IsHostAllowed(executionId, host) {
// host is not valid according to network policy
return false, protocolstate.ErrHostDenied(host)
return false, protocolstate.ErrHostDenied.Msgf(host)
}
// create a new client
client := redis.NewClient(&redis.Options{
@@ -109,7 +109,7 @@ func GetServerInfoAuth(ctx context.Context, host string, port int, password stri
func getServerInfoAuth(executionId string, host string, port int, password string) (string, error) {
if !protocolstate.IsHostAllowed(executionId, host) {
// host is not valid according to network policy
return "", protocolstate.ErrHostDenied(host)
return "", protocolstate.ErrHostDenied.Msgf(host)
}
// create a new client
client := redis.NewClient(&redis.Options{
@@ -181,7 +181,7 @@ func RunLuaScript(ctx context.Context, host string, port int, password string, s
executionId := ctx.Value("executionId").(string)
if !protocolstate.IsHostAllowed(executionId, host) {
// host is not valid according to network policy
return false, protocolstate.ErrHostDenied(host)
return false, protocolstate.ErrHostDenied.Msgf(host)
}
// create a new client
client := redis.NewClient(&redis.Options{
+3 -3
View File
@@ -43,7 +43,7 @@ func (c *SMBClient) ConnectSMBInfoMode(ctx context.Context, host string, port in
func connectSMBInfoMode(executionId string, host string, port int) (*smb.SMBLog, error) {
if !protocolstate.IsHostAllowed(executionId, host) {
// host is not valid according to network policy
return nil, protocolstate.ErrHostDenied(host)
return nil, protocolstate.ErrHostDenied.Msgf(host)
}
dialer := protocolstate.GetDialersWithId(executionId)
if dialer == nil {
@@ -90,7 +90,7 @@ func (c *SMBClient) ListSMBv2Metadata(ctx context.Context, host string, port int
executionId := ctx.Value("executionId").(string)
if !protocolstate.IsHostAllowed(executionId, host) {
// host is not valid according to network policy
return nil, protocolstate.ErrHostDenied(host)
return nil, protocolstate.ErrHostDenied.Msgf(host)
}
return memoizedcollectSMBv2Metadata(executionId, host, port, 5*time.Second)
}
@@ -119,7 +119,7 @@ func (c *SMBClient) ListShares(ctx context.Context, host string, port int, user,
func listShares(executionId string, host string, port int, user string, password string) ([]string, error) {
if !protocolstate.IsHostAllowed(executionId, host) {
// host is not valid according to network policy
return nil, protocolstate.ErrHostDenied(host)
return nil, protocolstate.ErrHostDenied.Msgf(host)
}
dialer := protocolstate.GetDialersWithId(executionId)
if dialer == nil {
+1 -1
View File
@@ -35,7 +35,7 @@ func (c *SMBClient) DetectSMBGhost(ctx context.Context, host string, port int) (
func detectSMBGhost(executionId string, host string, port int) (bool, error) {
if !protocolstate.IsHostAllowed(executionId, host) {
// host is not valid according to network policy
return false, protocolstate.ErrHostDenied(host)
return false, protocolstate.ErrHostDenied.Msgf(host)
}
addr := net.JoinHostPort(host, strconv.Itoa(port))
dialer := protocolstate.GetDialersWithId(executionId)
+1 -1
View File
@@ -68,7 +68,7 @@ func NewSMTPClient(call goja.ConstructorCall, runtime *goja.Runtime) *goja.Objec
executionId := c.nj.ExecutionId()
// check if this is allowed address
c.nj.Require(protocolstate.IsHostAllowed(executionId, host+":"+port), protocolstate.ErrHostDenied(host+":"+port).Error())
c.nj.Require(protocolstate.IsHostAllowed(executionId, host+":"+port), protocolstate.ErrHostDenied.Msgf(host+":"+port).Error())
// Link Constructor to Client and return
return utils.LinkConstructor(call, runtime, c)
+4 -4
View File
@@ -129,7 +129,7 @@ func (c *SSHClient) ConnectSSHInfoMode(ctx context.Context, host string, port in
// ```
func (c *SSHClient) Run(cmd string) (string, error) {
if c.connection == nil {
return "", errkit.New("no connection").Build()
return "", errkit.New("no connection")
}
session, err := c.connection.NewSession()
if err != nil {
@@ -177,14 +177,14 @@ type connectOptions struct {
func (c *connectOptions) validate() error {
if c.Host == "" {
return errkit.New("host is required").Build()
return errkit.New("host is required")
}
if c.Port <= 0 {
return errkit.New("port is required").Build()
return errkit.New("port is required")
}
if !protocolstate.IsHostAllowed(c.ExecutionId, c.Host) {
// host is not valid according to network policy
return protocolstate.ErrHostDenied(c.Host)
return protocolstate.ErrHostDenied.Msgf(c.Host)
}
if c.Timeout == 0 {
c.Timeout = 10 * time.Second
+4 -5
View File
@@ -11,7 +11,6 @@ import (
"github.com/Mzack9999/goja"
"github.com/alecthomas/chroma/quick"
"github.com/ditashi/jsbeautifier-go/jsbeautifier"
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/gozero"
@@ -113,7 +112,7 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
if options.Options.Validate {
options.Logger.Error().Msgf("%s <- %s", errMsg, err)
} else {
return errkit.Append(errkit.New(errMsg), err)
return errkit.Wrap(err, errMsg)
}
} else {
request.gozero = engine
@@ -132,7 +131,7 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
compiled.ExcludeMatchers = options.ExcludeMatchers
compiled.TemplateID = options.TemplateID
if err := compiled.Compile(); err != nil {
return errors.Wrap(err, "could not compile operators")
return errkit.Wrap(err, "could not compile operators")
}
for _, matcher := range compiled.Matchers {
// default matcher part for code protocol is response
@@ -153,7 +152,7 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
if request.PreCondition != "" {
preConditionCompiled, err := compiler.SourceAutoMode(request.PreCondition, false)
if err != nil {
return errkit.New(fmt.Sprintf("%s: could not compile pre-condition: %s", request.TemplateID, err)).Build()
return errkit.Newf("could not compile pre-condition: %s", err)
}
request.preConditionCompiled = preConditionCompiled
}
@@ -230,7 +229,7 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicVa
Context: input.Context(),
})
if err != nil {
return errkit.New(fmt.Sprintf("%s: could not execute pre-condition: %s", request.TemplateID, err)).Build()
return errkit.Newf("could not execute pre-condition: %s", err)
}
if !result.GetSuccess() || types.ToString(result["error"]) != "" {
gologger.Warning().Msgf("[%s] Precondition for request %s was not satisfied\n", request.TemplateID, request.PreCondition)
+1 -1
View File
@@ -8,7 +8,7 @@ import (
var (
defaultInteractionDuration = 60 * time.Second
interactshURLMarkerRegex = regexp.MustCompile(`(%7[B|b]|\{){2}(interactsh-url(?:_[0-9]+){0,3})(%7[D|d]|\}){2}`)
interactshURLMarkerRegex = regexp.MustCompile(`(%7[B|b]|\{){2}(interactsh-url(?:_[0-9]+){0,3})(%7[D|d]|\}){2}`)
ErrInteractshClientNotInitialized = errors.New("interactsh client not initialized")
)
@@ -88,7 +88,7 @@ func (c *Client) poll() error {
KeepAliveInterval: time.Minute,
})
if err != nil {
return errkit.Append(errkit.New("could not create client"), err)
return errkit.Wrap(err, "could not create client")
}
c.interactsh = interactsh
@@ -109,7 +109,7 @@ func (c *Client) poll() error {
// If we don't have any request for this ID, add it to temporary
// lru cache, so we can correlate when we get an add request.
items, err := c.interactions.Get(interaction.UniqueID)
if errors.Is(err, gcache.KeyNotFoundError) || items == nil {
if errkit.Is(err, gcache.KeyNotFoundError) || items == nil {
_ = c.interactions.SetWithExpire(interaction.UniqueID, []*server.Interaction{interaction}, defaultInteractionDuration)
} else {
items = append(items, interaction)
@@ -128,7 +128,7 @@ func (c *Client) poll() error {
})
if err != nil {
return errkit.Append(errkit.New("could not perform interactsh polling"), err)
return errkit.Wrap(err, "could not perform interactsh polling")
}
return nil
}
@@ -239,7 +239,7 @@ func (c *Client) URL() (string, error) {
err = c.poll()
})
if err != nil {
return "", errkit.Append(ErrInteractshClientNotInitialized, err)
return "", errkit.Wrap(ErrInteractshClientNotInitialized, err.Error())
}
if c.interactsh == nil {
+2 -3
View File
@@ -1,7 +1,6 @@
package protocolstate
import (
"fmt"
"strings"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
@@ -68,12 +67,12 @@ func NormalizePath(options *types.Options, filePath string) (string, error) {
}
cleaned, err := fileutil.ResolveNClean(filePath, config.DefaultConfig.GetTemplateDir())
if err != nil {
return "", errkit.Append(errkit.New(fmt.Sprintf("could not resolve and clean path %v", filePath)), err)
return "", errkit.Wrapf(err, "could not resolve and clean path %v", filePath)
}
// only allow files inside nuclei-templates directory
// even current working directory is not allowed
if strings.HasPrefix(cleaned, config.DefaultConfig.GetTemplateDir()) {
return cleaned, nil
}
return "", errkit.New(fmt.Sprintf("path %v is outside nuclei-template directory and -lfa is not enabled", filePath)).Build()
return "", errkit.Newf("path %v is outside nuclei-template directory and -lfa is not enabled", filePath)
}
+13 -10
View File
@@ -2,7 +2,6 @@ package protocolstate
import (
"context"
"fmt"
"net"
"strings"
@@ -18,14 +17,18 @@ import (
// initialize state of headless protocol
// ErrURLDenied returns an error when a URL is denied by network policy
func ErrURLDenied(url, rule string) error {
return errkit.New(fmt.Sprintf("headless: url %v dropped by rule: %v", url, rule)).Build()
var (
ErrURLDenied = errkit.New("headless: url dropped by rule")
ErrHostDenied = errorTemplate{format: "host %v dropped by network policy"}
)
// errorTemplate provides a way to create formatted errors like the old errorutil.NewWithFmt
type errorTemplate struct {
format string
}
// ErrHostDenied returns an error when a host is denied by network policy
func ErrHostDenied(host string) error {
return errkit.New(fmt.Sprintf("host %v dropped by network policy", host)).Build()
func (e errorTemplate) Msgf(args ...interface{}) error {
return errkit.Newf(e.format, args...)
}
func GetNetworkPolicy(ctx context.Context) *networkpolicy.NetworkPolicy {
@@ -47,15 +50,15 @@ func ValidateNFailRequest(options *types.Options, page *rod.Page, e *proto.Fetch
normalized := strings.ToLower(reqURL) // normalize url to lowercase
normalized = strings.TrimSpace(normalized) // trim leading & trailing whitespaces
if !IsLfaAllowed(options) && stringsutil.HasPrefixI(normalized, "file:") {
return multierr.Combine(FailWithReason(page, e), ErrURLDenied(reqURL, "use of file:// protocol disabled use '-lfa' to enable"))
return multierr.Combine(FailWithReason(page, e), errkit.Newf("headless: url %v dropped by rule: %v", reqURL, "use of file:// protocol disabled use '-lfa' to enable"))
}
// validate potential invalid schemes
// javascript protocol is allowed for xss fuzzing
if stringsutil.HasPrefixAnyI(normalized, "ftp:", "externalfile:", "chrome:", "chrome-extension:") {
return multierr.Combine(FailWithReason(page, e), ErrURLDenied(reqURL, "protocol blocked by network policy"))
return multierr.Combine(FailWithReason(page, e), errkit.Newf("headless: url %v dropped by rule: %v", reqURL, "protocol blocked by network policy"))
}
if !isValidHost(options, reqURL) {
return multierr.Combine(FailWithReason(page, e), ErrURLDenied(reqURL, "address blocked by network policy"))
return multierr.Combine(FailWithReason(page, e), errkit.Newf("headless: url %v dropped by rule: %v", reqURL, "address blocked by network policy"))
}
return nil
}
+1 -1
View File
@@ -78,7 +78,7 @@ func (i *Instance) Run(ctx *contextargs.Context, actions []*Action, payloads map
target := ctx.MetaInput.Input
input, err := urlutil.Parse(target)
if err != nil {
return nil, nil, errkit.Append(errkit.New(fmt.Sprintf("could not parse URL %s", target)), err)
return nil, nil, errkit.Wrapf(err, "could not parse URL %s", target)
}
hasTrailingSlash := httputil.HasTrailingSlash(target)
+13 -13
View File
@@ -31,8 +31,8 @@ import (
)
var (
errinvalidArguments = errkit.New("invalid arguments provided").Build()
ErrLFAccessDenied = errkit.New("Use -allow-local-file-access flag to enable local file access").Build()
errinvalidArguments = errkit.New("invalid arguments provided")
ErrLFAccessDenied = errkit.New("Use -allow-local-file-access flag to enable local file access")
// ErrActionExecDealine is the error returned when alloted time for action execution exceeds
ErrActionExecDealine = errkit.New("headless action execution deadline exceeded").SetKind(errkit.ErrKindDeadline).Build()
)
@@ -59,7 +59,7 @@ func (p *Page) ExecuteActions(input *contextargs.Context, actions []*Action) (ou
}
if r := recover(); r != nil {
err = errkit.New(fmt.Sprintf("panic on headless action: %v", r)).Build()
err = errkit.Newf("panic on headless action: %v", r)
}
}()
@@ -72,7 +72,7 @@ func (p *Page) ExecuteActions(input *contextargs.Context, actions []*Action) (ou
for _, waitFunc := range waitFuncs {
if waitFunc != nil {
if err := waitFunc(); err != nil {
return nil, errkit.Append(errkit.New("error occurred while executing waitFunc"), err)
return nil, errkit.Wrap(err, "error occurred while executing waitFunc")
}
}
}
@@ -400,7 +400,7 @@ func (p *Page) NavigateURL(action *Action, out ActionData) error {
parsedURL, err := urlutil.ParseURL(url, true)
if err != nil {
return errkit.New(fmt.Sprintf("headless: failed to parse url %v while creating http request", url)).Build()
return errkit.Newf("failed to parse url %v while creating http request", url)
}
// ===== parameter automerge =====
@@ -410,7 +410,7 @@ func (p *Page) NavigateURL(action *Action, out ActionData) error {
parsedURL.Params = finalparams
if err := p.page.Navigate(parsedURL.String()); err != nil {
return errkit.Append(errkit.New(fmt.Sprintf("could not navigate to url %s", parsedURL.String())), err)
return errkit.Wrapf(err, "could not navigate to url %s", parsedURL.String())
}
p.updateLastNavigatedURL()
@@ -524,14 +524,14 @@ func (p *Page) Screenshot(act *Action, out ActionData) error {
to, err = fileutil.CleanPath(to)
if err != nil {
return errkit.New(fmt.Sprintf("could not clean output screenshot path %s", to)).Build()
return errkit.Newf("could not clean output screenshot path %s", to)
}
// allow if targetPath is child of current working directory
if !protocolstate.IsLfaAllowed(p.options.Options) {
cwd, err := os.Getwd()
if err != nil {
return errkit.Append(errkit.New("could not get current working directory"), err)
return errkit.Wrap(err, "could not get current working directory")
}
if !strings.HasPrefix(to, cwd) {
@@ -550,7 +550,7 @@ func (p *Page) Screenshot(act *Action, out ActionData) error {
// creates new directory if needed based on path `to`
// TODO: replace all permission bits with fileutil constants (https://github.com/projectdiscovery/utils/issues/113)
if err := os.MkdirAll(filepath.Dir(to), 0700); err != nil {
return errkit.Append(errkit.New("failed to create directory while writing screenshot"), err)
return errkit.Wrap(err, "failed to create directory while writing screenshot")
}
}
@@ -562,7 +562,7 @@ func (p *Page) Screenshot(act *Action, out ActionData) error {
if fileutil.FileExists(filePath) {
// return custom error as overwriting files is not supported
return errkit.New(fmt.Sprintf("screenshot: failed to write screenshot, file %v already exists", filePath)).Build()
return errkit.Newf("failed to write screenshot, file %v already exists", filePath)
}
err = os.WriteFile(filePath, data, 0540)
if err != nil {
@@ -805,12 +805,12 @@ func (p *Page) WaitEvent(act *Action, out ActionData) (func() error, error) {
gotType := proto.GetType(event)
if gotType == nil {
return nil, errkit.New(fmt.Sprintf("event %q does not exist", event)).Build()
return nil, errkit.Newf("event %q does not exist", event)
}
tmp, ok := reflect.New(gotType).Interface().(proto.Event)
if !ok {
return nil, errkit.New(fmt.Sprintf("event %q is not a page event", event)).Build()
return nil, errkit.Newf("event %q is not a page event", event)
}
waitEvent = tmp
@@ -947,7 +947,7 @@ func (p *Page) getActionArg(action *Action, arg string) (string, error) {
err = expressions.ContainsUnresolvedVariables(exprs...)
if err != nil {
return "", errkit.Append(errkit.New(fmt.Sprintf("argument %q, value: %q", arg, argValue)), err)
return "", errkit.Wrapf(err, "argument %q, value: %q", arg, argValue)
}
argValue, err = expressions.Evaluate(argValue, p.variables)
+45 -21
View File
@@ -37,18 +37,42 @@ const (
ReqURLPatternKey = "req_url_pattern"
)
// ErrEvalExpression returns an error when helper expressions cannot be evaluated
func ErrEvalExpression(tag string) func(error) error {
return func(err error) error {
return errkit.Append(errkit.New(fmt.Sprintf("%s: could not evaluate helper expressions", tag)), err)
}
// ErrEvalExpression
type errorTemplate struct {
format string
}
// ErrUnresolvedVars returns an error when unresolved variables are found in request
func ErrUnresolvedVars(vars string) error {
return errkit.New(fmt.Sprintf("unresolved variables `%v` found in request", vars)).Build()
func (e errorTemplate) Wrap(err error) wrapperError {
return wrapperError{template: e, err: err}
}
func (e errorTemplate) Msgf(args ...interface{}) error {
return errkit.Newf(e.format, args...)
}
type wrapperError struct {
template errorTemplate
err error
}
func (w wrapperError) WithTag(tag string) error {
return errkit.Wrap(w.err, w.template.format)
}
func (w wrapperError) Msgf(format string, args ...interface{}) error {
return errkit.Wrapf(w.err, format, args...)
}
func (w wrapperError) Error() string {
return errkit.Wrap(w.err, w.template.format).Error()
}
// ErrEvalExpression
var (
ErrEvalExpression = errorTemplate{"could not evaluate helper expressions"}
ErrUnresolvedVars = errorTemplate{"unresolved variables `%v` found in request"}
)
// generatedRequest is a single generated request wrapped for a template request
type generatedRequest struct {
original *Request
@@ -199,7 +223,7 @@ func (r *requestGenerator) Make(ctx context.Context, input *contextargs.Context,
for payloadName, payloadValue := range payloads {
payloads[payloadName], err = expressions.Evaluate(types.ToString(payloadValue), allVars)
if err != nil {
return nil, ErrEvalExpression("http")(err)
return nil, errkit.Wrap(err, "could not evaluate helper expressions")
}
}
// finalVars contains allVars and any generator/fuzzing specific payloads
@@ -216,7 +240,7 @@ func (r *requestGenerator) Make(ctx context.Context, input *contextargs.Context,
// Evaluate (replace) variable with final values
reqData, err = expressions.Evaluate(reqData, finalVars)
if err != nil {
return nil, ErrEvalExpression("http")(err)
return nil, errkit.Wrap(err, "could not evaluate helper expressions")
}
if isRawRequest {
@@ -225,7 +249,7 @@ func (r *requestGenerator) Make(ctx context.Context, input *contextargs.Context,
reqURL, err := urlutil.ParseAbsoluteURL(reqData, true)
if err != nil {
return nil, errkit.New(fmt.Sprintf("http: failed to parse url %v while creating http request", reqData)).Build()
return nil, errkit.Newf("failed to parse url %v while creating http request", reqData)
}
// while merging parameters first preference is given to target params
finalparams := parsed.Params
@@ -258,7 +282,7 @@ func (r *requestGenerator) makeSelfContainedRequest(ctx context.Context, data st
// evaluate request
data, err := expressions.Evaluate(data, values)
if err != nil {
return nil, ErrEvalExpression("self-contained")(err)
return nil, errkit.Wrap(err, "could not evaluate helper expressions")
}
// If the request is a raw request, get the URL from the request
// header and use it to make the request.
@@ -281,7 +305,7 @@ func (r *requestGenerator) makeSelfContainedRequest(ctx context.Context, data st
}
if err := expressions.ContainsUnresolvedVariables(parts[1]); err != nil && !r.request.SkipVariablesCheck {
return nil, ErrUnresolvedVars(parts[1])
return nil, errkit.Newf("unresolved variables `%v` found in request", parts[1])
}
parsed, err := urlutil.ParseURL(parts[1], true)
@@ -295,19 +319,19 @@ func (r *requestGenerator) makeSelfContainedRequest(ctx context.Context, data st
// Evaluate (replace) variable with final values
data, err = expressions.Evaluate(data, values)
if err != nil {
return nil, ErrEvalExpression("self-contained")(err)
return nil, errkit.Wrap(err, "could not evaluate helper expressions")
}
return r.generateRawRequest(ctx, data, parsed, values, payloads)
}
if err := expressions.ContainsUnresolvedVariables(data); err != nil && !r.request.SkipVariablesCheck {
// early exit: if there are any unresolved variables in `path` after evaluation
// then return early since this will definitely fail
return nil, ErrUnresolvedVars(data)
return nil, errkit.Newf("unresolved variables `%v` found in request", data)
}
urlx, err := urlutil.ParseURL(data, true)
if err != nil {
return nil, errkit.New(fmt.Sprintf("self-contained: failed to parse %v in self contained request: %s", data, err)).Build()
return nil, errkit.Wrapf(err, "failed to parse %v in self contained request", data)
}
return r.generateHttpRequest(ctx, urlx, values, payloads)
}
@@ -318,7 +342,7 @@ func (r *requestGenerator) makeSelfContainedRequest(ctx context.Context, data st
func (r *requestGenerator) generateHttpRequest(ctx context.Context, urlx *urlutil.URL, finalVars, generatorValues map[string]interface{}) (*generatedRequest, error) {
method, err := expressions.Evaluate(r.request.Method.String(), finalVars)
if err != nil {
return nil, ErrEvalExpression("http")(err)
return nil, errkit.Wrap(err, "failed to evaluate while generating http request")
}
// Build a request on the specified URL
req, err := retryablehttp.NewRequestFromURLWithContext(ctx, method, urlx, nil)
@@ -347,7 +371,7 @@ func (r *requestGenerator) generateRawRequest(ctx context.Context, rawRequest st
rawRequestData, err = raw.Parse(rawRequest, baseURL, r.request.Unsafe, r.request.DisablePathAutomerge)
}
if err != nil {
return nil, errkit.Append(errkit.New("failed to parse raw request"), err)
return nil, errkit.Wrap(err, "failed to parse raw request")
}
// Unsafe option uses rawhttp library
@@ -363,7 +387,7 @@ func (r *requestGenerator) generateRawRequest(ctx context.Context, rawRequest st
}
urlx, err := urlutil.ParseAbsoluteURL(rawRequestData.FullURL, true)
if err != nil {
return nil, errkit.New(fmt.Sprintf("raw: failed to create request with url %v got %v", rawRequestData.FullURL, err)).Build()
return nil, errkit.Wrapf(err, "failed to create request with url %v got %v", rawRequestData.FullURL, err)
}
req, err := retryablehttp.NewRequestFromURLWithContext(ctx, rawRequestData.Method, urlx, rawRequestData.Data)
if err != nil {
@@ -420,7 +444,7 @@ func (r *requestGenerator) fillRequest(req *retryablehttp.Request, values map[st
}
value, err := expressions.Evaluate(value, values)
if err != nil {
return nil, ErrEvalExpression("http")(err)
return nil, errkit.Wrap(err, "failed to evaluate while adding headers to request")
}
req.Header[header] = []string{value}
if header == "Host" {
@@ -441,7 +465,7 @@ func (r *requestGenerator) fillRequest(req *retryablehttp.Request, values map[st
}
body, err := expressions.Evaluate(body, values)
if err != nil {
return nil, ErrEvalExpression("http")(err)
return nil, errkit.Wrap(err, "could not evaluate helper expressions")
}
bodyReader, err := readerutil.NewReusableReadCloser([]byte(body))
if err != nil {
+7 -7
View File
@@ -48,7 +48,7 @@ func Parse(request string, inputURL *urlutil.URL, unsafe, disablePathAutomerge b
case strings.HasPrefix(rawrequest.Path, "http") && !unsafe:
urlx, err := urlutil.ParseURL(rawrequest.Path, true)
if err != nil {
return nil, errkit.New(fmt.Sprintf("raw: failed to parse url %v from template: %s", rawrequest.Path, err)).Build()
return nil, errkit.Wrapf(err, "failed to parse url %v from template", rawrequest.Path)
}
cloned := inputURL.Clone()
cloned.Params.IncludeEquals = true
@@ -57,7 +57,7 @@ func Parse(request string, inputURL *urlutil.URL, unsafe, disablePathAutomerge b
}
parseErr := cloned.MergePath(urlx.GetRelativePath(), true)
if parseErr != nil {
return nil, errkit.Append(errkit.New(fmt.Sprintf("raw: could not automergepath for template path %v", urlx.GetRelativePath())), parseErr)
return nil, errkit.Wrapf(parseErr, "could not automergepath for template path %v", urlx.GetRelativePath())
}
rawrequest.Path = cloned.GetRelativePath()
// If unsafe changes must be made in raw request string itself
@@ -94,7 +94,7 @@ func Parse(request string, inputURL *urlutil.URL, unsafe, disablePathAutomerge b
}
err = cloned.MergePath(rawrequest.Path, true)
if err != nil {
return nil, errkit.New(fmt.Sprintf("raw: failed to automerge %v from unsafe template: %s", rawrequest.Path, err)).Build()
return nil, errkit.Wrapf(err, "failed to automerge %v from unsafe template", rawrequest.Path)
}
unsafeRelativePath = cloned.GetRelativePath()
}
@@ -116,7 +116,7 @@ func Parse(request string, inputURL *urlutil.URL, unsafe, disablePathAutomerge b
}
parseErr := cloned.MergePath(rawrequest.Path, true)
if parseErr != nil {
return nil, errkit.Append(errkit.New(fmt.Sprintf("raw: could not automergepath for template path %v", rawrequest.Path)), parseErr)
return nil, errkit.Wrapf(parseErr, "could not automergepath for template path %v", rawrequest.Path)
}
rawrequest.Path = cloned.GetRelativePath()
}
@@ -145,18 +145,18 @@ func ParseRawRequest(request string, unsafe bool) (*Request, error) {
if strings.HasPrefix(req.Path, "http") {
urlx, err := urlutil.Parse(req.Path)
if err != nil {
return nil, errkit.Append(errkit.New(fmt.Sprintf("failed to parse url %v", req.Path)), err)
return nil, errkit.Wrapf(err, "failed to parse url %v", req.Path)
}
req.Path = urlx.GetRelativePath()
req.FullURL = urlx.String()
} else {
if req.Path == "" {
return nil, errkit.New("self-contained-raw: path cannot be empty in self contained request").Build()
return nil, errkit.New("path cannot be empty in self contained request")
}
// given url is relative construct one using Host Header
if _, ok := req.Headers["Host"]; !ok {
return nil, errkit.New("self-contained-raw: host header is required for relative path").Build()
return nil, errkit.New("host header is required for relative path")
}
// Review: Current default scheme in self contained templates if relative path is provided is http
req.FullURL = fmt.Sprintf("%s://%s%s", urlutil.HTTP, strings.TrimSpace(req.Headers["Host"]), req.Path)
+1 -1
View File
@@ -60,7 +60,7 @@ func (a *AWSSigner) SignHTTP(ctx context.Context, request *http.Request) error {
// contentHash is sha256 hash of response body
contentHash := a.getPayloadHash(request)
if err := a.signer.SignHTTP(ctx, *a.creds, request, contentHash, a.options.Service, a.options.Region, time.Now()); err != nil {
return errkit.Append(errkit.New("failed to sign http request using aws v4 signer"), err)
return errkit.Wrap(err, "failed to sign http request using aws v4 signer")
}
// add x-amz-content-sha256 header to request
request.Header.Set("x-amz-content-sha256", contentHash)
+2 -3
View File
@@ -1,7 +1,6 @@
package http
import (
"fmt"
"io"
"strings"
@@ -16,14 +15,14 @@ func dump(req *generatedRequest, reqURL string) ([]byte, error) {
// Use a clone to avoid a race condition with the http transport
bin, err := req.request.Clone(req.request.Context()).Dump()
if err != nil {
return nil, errkit.New(fmt.Sprintf("http: could not dump request: %v: %s", req.request.String(), err)).Build()
return nil, errkit.Wrapf(err, "could not dump request: %v", req.request.String())
}
return bin, nil
}
rawHttpOptions := &rawhttp.Options{CustomHeaders: req.rawRequest.UnsafeHeaders, CustomRawBytes: req.rawRequest.UnsafeRawBytes}
bin, err := rawhttp.DumpRequestRaw(req.rawRequest.Method, reqURL, req.rawRequest.Path, generators.ExpandMapValues(req.rawRequest.Headers), io.NopCloser(strings.NewReader(req.rawRequest.Data)), rawHttpOptions)
if err != nil {
return nil, errkit.New(fmt.Sprintf("http: could not dump request: %v: %s", reqURL, err)).Build()
return nil, errkit.Wrapf(err, "could not dump request: %v", reqURL)
}
return bin, nil
}
+6 -6
View File
@@ -127,14 +127,14 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
}
}
if err := compiled.Compile(); err != nil {
return errkit.New(fmt.Sprintf("%s: could not compile operators got %v", request.TemplateID, err)).Build()
return errkit.Newf("could not compile operators got %v", err)
}
request.CompiledOperators = compiled
}
// "Port" is a special variable and it should not contains any dsl expressions
if strings.Contains(request.getPort(), "{{") {
return errkit.New(fmt.Sprintf("%s: 'Port' variable cannot contain any dsl expressions", request.TemplateID)).Build()
return errkit.New("'Port' variable cannot contain any dsl expressions")
}
if request.Init != "" {
@@ -218,11 +218,11 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
initCompiled, err := compiler.SourceAutoMode(request.Init, false)
if err != nil {
return errkit.New(fmt.Sprintf("%s: could not compile init code: %s", request.TemplateID, err)).Build()
return errkit.Newf("could not compile init code: %s", err)
}
result, err := request.options.JsCompiler.ExecuteWithOptions(initCompiled, args, opts)
if err != nil {
return errkit.New(fmt.Sprintf("%s: could not execute pre-condition: %s", request.TemplateID, err)).Build()
return errkit.Newf("could not execute pre-condition: %s", err)
}
if types.ToString(result["error"]) != "" {
gologger.Warning().Msgf("[%s] Init failed with error %v\n", request.TemplateID, result["error"])
@@ -239,7 +239,7 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
if request.PreCondition != "" {
preConditionCompiled, err := compiler.SourceAutoMode(request.PreCondition, false)
if err != nil {
return errkit.New(fmt.Sprintf("%s: could not compile pre-condition: %s", request.TemplateID, err)).Build()
return errkit.Newf("could not compile pre-condition: %s", err)
}
request.preConditionCompiled = preConditionCompiled
}
@@ -248,7 +248,7 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
if request.Code != "" {
scriptCompiled, err := compiler.SourceAutoMode(request.Code, false)
if err != nil {
return errkit.New(fmt.Sprintf("%s: could not compile javascript code: %s", request.TemplateID, err)).Build()
return errkit.Newf("could not compile javascript code: %s", err)
}
request.scriptCompiled = scriptCompiled
}
+2 -3
View File
@@ -1,7 +1,6 @@
package network
import (
"fmt"
"strconv"
"strings"
@@ -197,10 +196,10 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
}
portInt, err := strconv.Atoi(port)
if err != nil {
return errkit.Append(errkit.New(fmt.Sprintf("could not parse port %v from '%s'", port, request.Port)), err)
return errkit.Wrapf(err, "could not parse port %v from '%s'", port, request.Port)
}
if portInt < 1 || portInt > 65535 {
return errkit.New(fmt.Sprintf("%s: port %v is not in valid range", request.TemplateID, portInt)).Build()
return errkit.Newf("port %v is not in valid range", portInt)
}
request.ports = append(request.ports, port)
}
+1 -1
View File
@@ -362,7 +362,7 @@ func (request *Request) executeRequestWithPayloads(variables map[string]interfac
if input.Read > 0 {
buffer, err := ConnReadNWithTimeout(conn, int64(input.Read), request.options.Options.GetTimeouts().TcpReadTimeout)
if err != nil {
return errkit.Append(errkit.New("could not read response from connection"), err)
return errkit.Wrap(err, "could not read response from connection")
}
responseBuilder.Write(buffer)
+8 -8
View File
@@ -121,7 +121,7 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
CustomDialer: options.CustomFastdialer,
})
if err != nil {
return errkit.Append(errkit.New("ssl: could not get network client"), err)
return errkit.Wrap(err, "could not get network client")
}
request.dialer = client
switch {
@@ -130,7 +130,7 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
request.ScanMode = "auto"
case !stringsutil.EqualFoldAny(request.ScanMode, "auto", "openssl", "ztls", "ctls"):
return errkit.New(fmt.Sprintf("%s: template %v does not contain valid scan-mode", request.TemplateID, request.TemplateID)).Build()
return errkit.Newf("template %v does not contain valid scan-mode", request.TemplateID)
case request.ScanMode == "openssl" && !openssl.IsAvailable():
// if openssl is not installed instead of failing "auto" scanmode is used
@@ -169,7 +169,7 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
tlsxService, err := tlsx.New(tlsxOptions)
if err != nil {
return errkit.New(fmt.Sprintf("%s: could not create tlsx service", request.TemplateID)).Build()
return errkit.New("could not create tlsx service")
}
request.tlsx = tlsxService
@@ -178,7 +178,7 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
compiled.ExcludeMatchers = options.ExcludeMatchers
compiled.TemplateID = options.TemplateID
if err := compiled.Compile(); err != nil {
return errkit.New(fmt.Sprintf("%s: could not compile operators got %v", request.TemplateID, err)).Build()
return errkit.Newf("could not compile operators got %v", err)
}
request.CompiledOperators = compiled
}
@@ -236,7 +236,7 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicVa
addressToDial := string(finalAddress)
host, port, err := net.SplitHostPort(addressToDial)
if err != nil {
return errkit.Append(errkit.New("could not split input host port"), err)
return errkit.Wrap(err, "could not split input host port")
}
var hostIp string
@@ -250,7 +250,7 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicVa
if err != nil {
requestOptions.Output.Request(requestOptions.TemplateID, input.MetaInput.Input, request.Type().String(), err)
requestOptions.Progress.IncrementFailedRequestsBy(1)
return errkit.Append(errkit.New(fmt.Sprintf("%s: could not connect to server", request.TemplateID)), err)
return errkit.Wrap(err, "could not connect to server")
}
requestOptions.Output.Request(requestOptions.TemplateID, hostPort, request.Type().String(), err)
@@ -287,7 +287,7 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicVa
// if response is not struct compatible, error out
if !structs.IsStruct(response) {
return errkit.New(fmt.Sprintf("ssl: response cannot be parsed into a struct: %v", response)).Build()
return errkit.Newf("response cannot be parsed into a struct: %v", response)
}
// Convert response to key value pairs and first cert chain item as well
@@ -307,7 +307,7 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicVa
// if certificate response is not struct compatible, error out
if !structs.IsStruct(response.CertificateResponse) {
return errkit.New(fmt.Sprintf("ssl: certificate response cannot be parsed into a struct: %v", response.CertificateResponse)).Build()
return errkit.Newf("certificate response cannot be parsed into a struct: %v", response.CertificateResponse)
}
responseParsed = structs.New(response.CertificateResponse)
@@ -20,7 +20,7 @@ func CreateTable(headers []string, rows [][]string) (string, error) {
builder := &bytes.Buffer{}
headerSize := len(headers)
if headers == nil || headerSize == 0 {
return "", errkit.New("No headers provided").Build()
return "", errkit.New("No headers provided")
}
builder.WriteString(CreateTableHeader(headers...))
@@ -34,7 +34,7 @@ func CreateTable(headers []string, rows [][]string) (string, error) {
copy(extendedRows, row)
builder.WriteString(CreateTableRow(extendedRows...))
} else {
return "", errkit.New("Too many columns for the given headers").Build()
return "", errkit.New("Too many columns for the given headers")
}
}
+13 -13
View File
@@ -84,7 +84,7 @@ func New(options *Options, db string, doNotDedupe bool) (Client, error) {
options.GitHub.OmitRaw = options.OmitRaw
tracker, err := github.New(options.GitHub)
if err != nil {
return nil, errkit.Append(ErrReportingClientCreation, err)
return nil, errkit.Wrapf(err, "could not create reporting client: %v", ErrReportingClientCreation)
}
client.trackers = append(client.trackers, tracker)
}
@@ -93,7 +93,7 @@ func New(options *Options, db string, doNotDedupe bool) (Client, error) {
options.GitLab.OmitRaw = options.OmitRaw
tracker, err := gitlab.New(options.GitLab)
if err != nil {
return nil, errkit.Append(ErrReportingClientCreation, err)
return nil, errkit.Wrapf(err, "could not create reporting client: %v", ErrReportingClientCreation)
}
client.trackers = append(client.trackers, tracker)
}
@@ -102,7 +102,7 @@ func New(options *Options, db string, doNotDedupe bool) (Client, error) {
options.Gitea.OmitRaw = options.OmitRaw
tracker, err := gitea.New(options.Gitea)
if err != nil {
return nil, errkit.Append(ErrReportingClientCreation, err)
return nil, errkit.Wrapf(err, "could not create reporting client: %v", ErrReportingClientCreation)
}
client.trackers = append(client.trackers, tracker)
}
@@ -111,7 +111,7 @@ func New(options *Options, db string, doNotDedupe bool) (Client, error) {
options.Jira.OmitRaw = options.OmitRaw
tracker, err := jira.New(options.Jira)
if err != nil {
return nil, errkit.Append(ErrReportingClientCreation, err)
return nil, errkit.Wrapf(err, "could not create reporting client: %v", ErrReportingClientCreation)
}
client.trackers = append(client.trackers, tracker)
}
@@ -120,35 +120,35 @@ func New(options *Options, db string, doNotDedupe bool) (Client, error) {
options.Linear.OmitRaw = options.OmitRaw
tracker, err := linear.New(options.Linear)
if err != nil {
return nil, errkit.Append(ErrReportingClientCreation, err)
return nil, errkit.Wrapf(err, "could not create reporting client: %v", ErrReportingClientCreation)
}
client.trackers = append(client.trackers, tracker)
}
if options.MarkdownExporter != nil {
exporter, err := markdown.New(options.MarkdownExporter)
if err != nil {
return nil, errkit.Append(ErrExportClientCreation, err)
return nil, errkit.Wrapf(err, "could not create export client: %v", ErrExportClientCreation)
}
client.exporters = append(client.exporters, exporter)
}
if options.SarifExporter != nil {
exporter, err := sarif.New(options.SarifExporter)
if err != nil {
return nil, errkit.Append(ErrExportClientCreation, err)
return nil, errkit.Wrapf(err, "could not create export client: %v", ErrExportClientCreation)
}
client.exporters = append(client.exporters, exporter)
}
if options.JSONExporter != nil {
exporter, err := json_exporter.New(options.JSONExporter)
if err != nil {
return nil, errkit.Append(ErrExportClientCreation, err)
return nil, errkit.Wrapf(err, "could not create export client: %v", ErrExportClientCreation)
}
client.exporters = append(client.exporters, exporter)
}
if options.JSONLExporter != nil {
exporter, err := jsonl.New(options.JSONLExporter)
if err != nil {
return nil, errkit.Append(ErrExportClientCreation, err)
return nil, errkit.Wrapf(err, "could not create export client: %v", ErrExportClientCreation)
}
client.exporters = append(client.exporters, exporter)
}
@@ -157,7 +157,7 @@ func New(options *Options, db string, doNotDedupe bool) (Client, error) {
options.ElasticsearchExporter.ExecutionId = options.ExecutionId
exporter, err := es.New(options.ElasticsearchExporter)
if err != nil {
return nil, errkit.Append(ErrExportClientCreation, err)
return nil, errkit.Wrapf(err, "could not create export client: %v", ErrExportClientCreation)
}
client.exporters = append(client.exporters, exporter)
}
@@ -166,14 +166,14 @@ func New(options *Options, db string, doNotDedupe bool) (Client, error) {
options.SplunkExporter.ExecutionId = options.ExecutionId
exporter, err := splunk.New(options.SplunkExporter)
if err != nil {
return nil, errkit.Append(ErrExportClientCreation, err)
return nil, errkit.Wrapf(err, "could not create export client: %v", ErrExportClientCreation)
}
client.exporters = append(client.exporters, exporter)
}
if options.MongoDBExporter != nil {
exporter, err := mongo.New(options.MongoDBExporter)
if err != nil {
return nil, errkit.Append(ErrExportClientCreation, err)
return nil, errkit.Wrapf(err, "could not create export client: %v", ErrExportClientCreation)
}
client.exporters = append(client.exporters, exporter)
}
@@ -227,7 +227,7 @@ func CreateConfigIfNotExists() error {
}
reportingFile, err := os.Create(reportingConfig)
if err != nil {
return errkit.Append(errkit.New("could not create config file"), err)
return errkit.Wrap(err, "could not create config file")
}
defer func() {
_ = reportingFile.Close()
+23 -3
View File
@@ -224,11 +224,31 @@ func Parse(filePath string, preprocessor Preprocessor, options *protocols.Execut
}
template.Path = filePath
if !options.DoNotCache {
parser.compiledTemplatesCache.Store(filePath, template, nil, err)
// Store a sanitized template in compiled cache to avoid retaining engine-scoped state.
cacheTpl := *template
cacheTpl.Options = sanitizeOptionsForCache(template.Options)
// Raw template bytes are not needed in compiled cache; keep per-engine
cacheTpl.Options.RawTemplate = nil
parser.compiledTemplatesCache.Store(filePath, &cacheTpl, nil, err)
}
return template, nil
}
// sanitizeOptionsForCache strips engine-scoped fields from ExecutorOptions to avoid
// retaining per-engine references in the shared compiled cache.
func sanitizeOptionsForCache(src *protocols.ExecutorOptions) *protocols.ExecutorOptions {
if src == nil {
return nil
}
return &protocols.ExecutorOptions{
// Intentionally exclude TemplateID/Path/Verifier and RawTemplate to avoid engine leakage
StopAtFirstMatch: src.StopAtFirstMatch,
ProtocolType: src.ProtocolType,
Flow: src.Flow,
IsMultiProtocol: src.IsMultiProtocol,
}
}
// isGlobalMatchersEnabled checks if any of requests in the template
// have global matchers enabled. It iterates through all requests and
// returns true if at least one request has global matchers enabled;
@@ -491,7 +511,7 @@ func parseTemplate(data []byte, srcOptions *protocols.ExecutorOptions) (*Templat
}
}
if err != nil {
return nil, errkit.Append(errkit.New(fmt.Sprintf("failed to parse %s", template.Path)), err)
return nil, errkit.Wrapf(err, "failed to parse %s", template.Path)
}
if utils.IsBlank(template.Info.Name) {
@@ -551,7 +571,7 @@ func parseTemplate(data []byte, srcOptions *protocols.ExecutorOptions) (*Templat
// load `flow` and `source` in code protocol from file
// if file is referenced instead of actual source code
if err := template.ImportFileRefs(template.Options); err != nil {
return nil, errkit.Append(errkit.New(fmt.Sprintf("failed to load file refs for %s", template.ID)), err)
return nil, errkit.Wrapf(err, "failed to load file refs for %s", template.ID)
}
if err := template.compileProtocolRequests(template.Options); err != nil {
+158
View File
@@ -206,3 +206,161 @@ func TestWrongWorkflow(t *testing.T) {
require.Nil(t, got, "could not parse template")
require.ErrorContains(t, err, "workflows cannot have other protocols")
}
func Test_SharedCompiledCache_SharedAcrossParsers(t *testing.T) {
setup()
p1 := templates.NewSharedParserWithCompiledCache()
p2 := templates.NewSharedParserWithCompiledCache()
exec1 := &protocols.ExecutorOptions{
Output: testutils.NewMockOutputWriter(testutils.DefaultOptions.OmitTemplate),
Options: testutils.DefaultOptions,
Progress: executerOpts.Progress,
Catalog: executerOpts.Catalog,
RateLimiter: executerOpts.RateLimiter,
Parser: p1,
}
// reinit options fully for isolation
opts2 := testutils.DefaultOptions
testutils.Init(opts2)
progressImpl, _ := progress.NewStatsTicker(0, false, false, false, 0)
exec2 := &protocols.ExecutorOptions{
Output: testutils.NewMockOutputWriter(opts2.OmitTemplate),
Options: opts2,
Progress: progressImpl,
Catalog: executerOpts.Catalog,
RateLimiter: executerOpts.RateLimiter,
Parser: p2,
}
filePath := "tests/match-1.yaml"
got1, err := templates.Parse(filePath, nil, exec1)
require.NoError(t, err)
require.NotNil(t, got1)
got2, err := templates.Parse(filePath, nil, exec2)
require.NoError(t, err)
require.NotNil(t, got2)
require.Equal(t, p1.CompiledCache(), p2.CompiledCache())
require.Greater(t, p1.CompiledCount(), 0)
require.Equal(t, p1.CompiledCount(), p2.CompiledCount())
}
func Test_SharedCompiledCache_OptionsIsolation(t *testing.T) {
setup()
p1 := templates.NewSharedParserWithCompiledCache()
p2 := templates.NewSharedParserWithCompiledCache()
exec1 := &protocols.ExecutorOptions{
Output: testutils.NewMockOutputWriter(testutils.DefaultOptions.OmitTemplate),
Options: testutils.DefaultOptions,
Progress: executerOpts.Progress,
Catalog: executerOpts.Catalog,
RateLimiter: executerOpts.RateLimiter,
Parser: p1,
}
// reinit options fully for isolation
opts2 := testutils.DefaultOptions
testutils.Init(opts2)
progressImpl, _ := progress.NewStatsTicker(0, false, false, false, 0)
exec2 := &protocols.ExecutorOptions{
Output: testutils.NewMockOutputWriter(opts2.OmitTemplate),
Options: opts2,
Progress: progressImpl,
Catalog: executerOpts.Catalog,
RateLimiter: executerOpts.RateLimiter,
Parser: p2,
}
filePath := "tests/match-1.yaml"
got1, err := templates.Parse(filePath, nil, exec1)
require.NoError(t, err)
require.NotNil(t, got1)
got2, err := templates.Parse(filePath, nil, exec2)
require.NoError(t, err)
require.NotNil(t, got2)
require.NotEqual(t, got1.Options, got2.Options)
}
// compiled cache does not retain engine-scoped fields
func Test_CompiledCache_SanitizesOptions(t *testing.T) {
setup()
p := templates.NewSharedParserWithCompiledCache()
exec := executerOpts
exec.Parser = p
filePath := "tests/match-1.yaml"
got, err := templates.Parse(filePath, nil, exec)
require.NoError(t, err)
require.NotNil(t, got)
cached, raw, err := p.CompiledCache().Has(filePath)
require.NoError(t, err)
require.NotNil(t, cached)
require.Nil(t, raw)
// cached template must not hold engine-scoped references
require.Nil(t, cached.Options.Options)
require.Empty(t, cached.Options.TemplateVerifier)
require.Empty(t, cached.Options.TemplateID)
require.Empty(t, cached.Options.TemplatePath)
require.False(t, cached.Options.StopAtFirstMatch)
}
// different engines see different Options pointers
func Test_EngineIsolation_NoCrossLeaks(t *testing.T) {
setup()
p1 := templates.NewSharedParserWithCompiledCache()
p2 := templates.NewSharedParserWithCompiledCache()
// engine 1
exec1 := &protocols.ExecutorOptions{
Output: executerOpts.Output,
Options: executerOpts.Options,
Progress: executerOpts.Progress,
Catalog: executerOpts.Catalog,
RateLimiter: executerOpts.RateLimiter,
Parser: p1,
}
// engine 2 with a fresh options instance
opts2 := testutils.DefaultOptions
testutils.Init(opts2)
progress2, _ := progress.NewStatsTicker(0, false, false, false, 0)
exec2 := &protocols.ExecutorOptions{
Output: testutils.NewMockOutputWriter(opts2.OmitTemplate),
Options: opts2,
Progress: progress2,
Catalog: executerOpts.Catalog,
RateLimiter: executerOpts.RateLimiter,
Parser: p2,
}
filePath := "tests/match-1.yaml"
got1, err := templates.Parse(filePath, nil, exec1)
require.NoError(t, err)
got2, err := templates.Parse(filePath, nil, exec2)
require.NoError(t, err)
// template options must be distinct per engine
require.NotEqual(t, got1.Options, got2.Options)
// http request options must bind to engine-specific ExecutorOptions copies (not shared)
require.NotEmpty(t, got1.RequestsHTTP)
require.NotEmpty(t, got2.RequestsHTTP)
r1 := got1.RequestsHTTP[0]
r2 := got2.RequestsHTTP[0]
// ensure options structs are not the same pointer
require.NotSame(t, r1.Options().Options, r2.Options().Options)
// mutate engine2 options and ensure it doesn't affect engine1
r2.Options().Options.RateLimit = 999
require.NotEqual(t, r1.Options().Options.RateLimit, r2.Options().Options.RateLimit)
// compiled cache instance shared, but without engine leakage
require.Equal(t, p1.CompiledCache(), p2.CompiledCache())
}
+37 -14
View File
@@ -12,6 +12,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/utils/json"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/stats"
yamlutil "github.com/projectdiscovery/nuclei/v3/pkg/utils/yaml"
"github.com/projectdiscovery/utils/errkit"
fileutil "github.com/projectdiscovery/utils/file"
"gopkg.in/yaml.v2"
@@ -28,20 +29,42 @@ type Parser struct {
sync.Mutex
}
func NewParser() *Parser {
p := &Parser{
parsedTemplatesCache: NewCache(),
compiledTemplatesCache: NewCache(),
}
var (
sharedParsedCacheOnce sync.Once
sharedParsedCache *Cache
)
return p
var (
sharedCompiledCacheOnce sync.Once
sharedCompiledCache *Cache
)
// NewParser returns a new parser with a fresh cache
func NewParser() *Parser {
return &Parser{parsedTemplatesCache: NewCache(), compiledTemplatesCache: NewCache()}
}
// NewParserWithParsedCache returns a parser using provided cache
func NewParserWithParsedCache(cache *Cache) *Parser {
return &Parser{
parsedTemplatesCache: cache,
compiledTemplatesCache: NewCache(),
}
return &Parser{parsedTemplatesCache: cache, compiledTemplatesCache: NewCache()}
}
// NewSharedParser returns a parser backed by a process-wide shared parsed cache.
// Safe for concurrent use since Cache is concurrency-safe.
func NewSharedParser() *Parser {
sharedParsedCacheOnce.Do(func() {
sharedParsedCache = NewCache()
})
return &Parser{parsedTemplatesCache: sharedParsedCache, compiledTemplatesCache: NewCache()}
}
// NewSharedParserWithCompiledCache returns a parser backed by process-wide shared
// parsed and compiled caches. Intended for scenarios where compiled executers
// can be safely reused across engines by copying option-bearing fields.
func NewSharedParserWithCompiledCache() *Parser {
sharedParsedCacheOnce.Do(func() { sharedParsedCache = NewCache() })
sharedCompiledCacheOnce.Do(func() { sharedCompiledCache = NewCache() })
return &Parser{parsedTemplatesCache: sharedParsedCache, compiledTemplatesCache: sharedCompiledCache}
}
// Cache returns the parsed templates cache
@@ -82,7 +105,7 @@ func (p *Parser) LoadTemplate(templatePath string, t any, extraTags []string, ca
t, templateParseError := p.ParseTemplate(templatePath, catalog)
if templateParseError != nil {
checkOpenFileError(templateParseError)
return false, ErrCouldNotLoadTemplate(templatePath, templateParseError.Error())
return false, errkit.Newf("Could not load template %s: %s", templatePath, templateParseError)
}
template, ok := t.(*Template)
if !ok {
@@ -96,13 +119,13 @@ func (p *Parser) LoadTemplate(templatePath string, t any, extraTags []string, ca
validationError := validateTemplateMandatoryFields(template)
if validationError != nil {
stats.Increment(SyntaxErrorStats)
return false, ErrCouldNotLoadTemplate(templatePath, validationError.Error())
return false, errkit.Newf("Could not load template %s: %s", templatePath, validationError)
}
ret, err := isTemplateInfoMetadataMatch(tagFilter, template, extraTags)
if err != nil {
checkOpenFileError(err)
return ret, ErrCouldNotLoadTemplate(templatePath, err.Error())
return ret, errkit.Newf("Could not load template %s: %s", templatePath, err)
}
// if template loaded then check the template for optional fields to add warnings
if ret {
@@ -110,7 +133,7 @@ func (p *Parser) LoadTemplate(templatePath string, t any, extraTags []string, ca
if validationWarning != nil {
stats.Increment(SyntaxWarningStats)
checkOpenFileError(validationWarning)
return ret, ErrCouldNotLoadTemplate(templatePath, validationWarning.Error())
return ret, errkit.Newf("Could not load template %s: %s", templatePath, validationWarning)
}
}
return ret, nil
+7 -22
View File
@@ -1,28 +1,13 @@
package templates
import (
"fmt"
"github.com/projectdiscovery/utils/errkit"
)
// Helper functions for template errors with formatting
func ErrMandatoryFieldMissingFmt(field string) error {
return errkit.New(fmt.Sprintf("mandatory '%s' field is missing", field)).Build()
}
func ErrInvalidField(field, format string) error {
return errkit.New(fmt.Sprintf("invalid field format for '%s' (allowed format is %s)", field, format)).Build()
}
func ErrWarningFieldMissing(field string) error {
return errkit.New(fmt.Sprintf("field '%s' is missing", field)).Build()
}
func ErrCouldNotLoadTemplate(path, reason string) error {
return errkit.New(fmt.Sprintf("Could not load template %s: %s", path, reason)).Build()
}
func ErrLoadedWithWarnings(path, warning string) error {
return errkit.New(fmt.Sprintf("Loaded template %s: with syntax warning : %s", path, warning)).Build()
}
var (
ErrMandatoryFieldMissingFmt = errkit.New("mandatory field is missing")
ErrInvalidField = errkit.New("invalid field format")
ErrWarningFieldMissing = errkit.New("field is missing")
ErrCouldNotLoadTemplate = errkit.New("could not load template")
ErrLoadedWithWarnings = errkit.New("loaded template with syntax warning")
)
+29
View File
@@ -3,6 +3,7 @@ package templates
import (
"errors"
"fmt"
"sync"
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/disk"
@@ -156,3 +157,31 @@ func TestLoadTemplate(t *testing.T) {
}
})
}
func TestNewSharedParserSharesCache(t *testing.T) {
p1 := NewSharedParser()
p2 := NewSharedParser()
if p1.Cache() != p2.Cache() {
t.Fatalf("expected shared cache instance")
}
}
func TestNewSharedParserConcurrency(t *testing.T) {
var wg sync.WaitGroup
const goroutines = 50
parsers := make([]*Parser, goroutines)
wg.Add(goroutines)
for i := 0; i < goroutines; i++ {
go func(i int) {
defer wg.Done()
parsers[i] = NewSharedParser()
}(i)
}
wg.Wait()
base := parsers[0].Cache()
for i := 1; i < goroutines; i++ {
if parsers[i].Cache() != base {
t.Fatalf("expected all parsers to share the same cache")
}
}
}
+6 -5
View File
@@ -5,6 +5,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
"github.com/projectdiscovery/nuclei/v3/pkg/utils"
"github.com/projectdiscovery/utils/errkit"
)
// validateTemplateMandatoryFields validates the mandatory fields of a template
@@ -15,17 +16,17 @@ func validateTemplateMandatoryFields(template *Template) error {
var validateErrors []error
if utils.IsBlank(info.Name) {
validateErrors = append(validateErrors, ErrMandatoryFieldMissingFmt("name"))
validateErrors = append(validateErrors, errkit.Newf("mandatory '%s' field is missing", "name"))
}
if info.Authors.IsEmpty() {
validateErrors = append(validateErrors, ErrMandatoryFieldMissingFmt("author"))
validateErrors = append(validateErrors, errkit.Newf("mandatory '%s' field is missing", "author"))
}
if template.ID == "" {
validateErrors = append(validateErrors, ErrMandatoryFieldMissingFmt("id"))
validateErrors = append(validateErrors, errkit.Newf("mandatory '%s' field is missing", "id"))
} else if !ReTemplateID.MatchString(template.ID) {
validateErrors = append(validateErrors, ErrInvalidField("id", ReTemplateID.String()))
validateErrors = append(validateErrors, errkit.Newf("invalid field format for '%s' (allowed format is %s)", "id", ReTemplateID.String()))
}
if len(validateErrors) > 0 {
@@ -53,7 +54,7 @@ func validateTemplateOptionalFields(template *Template) error {
var warnings []error
if template.Type() != types.WorkflowProtocol && utils.IsBlank(info.SeverityHolder.Severity.String()) {
warnings = append(warnings, ErrWarningFieldMissing("severity"))
warnings = append(warnings, errkit.Newf("field '%s' is missing", "severity"))
}
if len(warnings) > 0 {
+1 -1
View File
@@ -34,7 +34,7 @@ func init() {
// AddSignerToDefault adds a signer to the default list of signers
func AddSignerToDefault(s *TemplateSigner) error {
if s == nil {
return errkit.New("signer is nil").Build()
return errkit.New("signer is nil")
}
DefaultTemplateVerifiers = append(DefaultTemplateVerifiers, s)
return nil
+2 -2
View File
@@ -82,13 +82,13 @@ func (t *TemplateSigner) Sign(data []byte, tmpl SignableTemplate) (string, error
arr := strings.SplitN(string(existingSignature), ":", 3)
if len(arr) == 2 {
// signature has no fragment
return "", errkit.New("signer: re-signing code templates are not allowed for security reasons.").Build()
return "", errkit.New("re-signing code templates are not allowed for security reasons.")
}
if len(arr) == 3 {
// signature has fragment verify if it is equal to current fragment
fragment := t.GetUserFragment()
if fragment != arr[2] {
return "", errkit.New("signer: re-signing code templates are not allowed for security reasons.").Build()
return "", errkit.New("re-signing code templates are not allowed for security reasons.")
}
}
}
+3 -3
View File
@@ -28,7 +28,7 @@ var (
_ = protocolstate.Init(defaultOpts)
_ = protocolinit.Init(defaultOpts)
})
ErrNotATemplate = errkit.New("signer: given filePath is not a template").Build()
ErrNotATemplate = errkit.New("given filePath is not a template", "tag", "signer")
)
// UseOptionsForSigner sets the options to use for signing templates
@@ -68,7 +68,7 @@ func SignTemplate(templateSigner *signer.TemplateSigner, templatePath string) er
template, bin, err := getTemplate(templatePath)
if err != nil {
return errkit.Append(errkit.New("failed to get template from disk"), err)
return errkit.Wrap(err, "failed to get template from disk")
}
if len(template.Workflows) > 0 {
// signing workflows is not supported at least yet
@@ -100,7 +100,7 @@ func getTemplate(templatePath string) (*Template, []byte, error) {
}
template, err := ParseTemplateFromReader(bytes.NewReader(bin), nil, executerOpts)
if err != nil {
return nil, bin, errkit.Append(errkit.New("failed to parse template"), err)
return nil, bin, errkit.Wrap(err, "failed to parse template")
}
return template, bin, nil
}
+7 -8
View File
@@ -2,7 +2,6 @@
package templates
import (
"fmt"
"io"
"path/filepath"
"strconv"
@@ -326,14 +325,14 @@ func (template *Template) UnmarshalYAML(unmarshal func(interface{}) error) error
*template = Template(*alias)
if !ReTemplateID.MatchString(template.ID) {
return errkit.New(fmt.Sprintf("invalid template: template id must match expression %v", ReTemplateID)).Build()
return errkit.New("template id must match expression %v", ReTemplateID, "tag", "invalid_template")
}
info := template.Info
if utils.IsBlank(info.Name) {
return errkit.New("invalid template: no template name field provided").Build()
return errkit.New("no template name field provided", "tag", "invalid_template")
}
if info.Authors.IsEmpty() {
return errkit.New("invalid template: no template author field provided").Build()
return errkit.New("no template author field provided", "tag", "invalid_template")
}
if len(template.RequestsHTTP) > 0 || len(template.RequestsNetwork) > 0 {
@@ -341,10 +340,10 @@ func (template *Template) UnmarshalYAML(unmarshal func(interface{}) error) error
}
if len(alias.RequestsHTTP) > 0 && len(alias.RequestsWithHTTP) > 0 {
return errkit.New("invalid template: use http or requests, both are not supported").Build()
return errkit.New("use http or requests, both are not supported", "tag", "invalid_template")
}
if len(alias.RequestsNetwork) > 0 && len(alias.RequestsWithTCP) > 0 {
return errkit.New("invalid template: use tcp or network, both are not supported").Build()
return errkit.New("use tcp or network, both are not supported", "tag", "invalid_template")
}
if len(alias.RequestsWithHTTP) > 0 {
template.RequestsHTTP = alias.RequestsWithHTTP
@@ -362,7 +361,7 @@ func (template *Template) UnmarshalYAML(unmarshal func(interface{}) error) error
var tempmap yaml.MapSlice
err = unmarshal(&tempmap)
if err != nil {
return errkit.Append(errkit.New(fmt.Sprintf("failed to unmarshal multi protocol template %s", template.ID)), err)
return errkit.Wrapf(err, "failed to unmarshal multi protocol template %s", template.ID)
}
arr := []string{}
for _, v := range tempmap {
@@ -546,7 +545,7 @@ func (template *Template) UnmarshalJSON(data []byte) error {
var tempMap map[string]interface{}
err = json.Unmarshal(data, &tempMap)
if err != nil {
return errkit.Append(errkit.New(fmt.Sprintf("failed to unmarshal multi protocol template %s", template.ID)), err)
return errkit.Wrapf(err, "failed to unmarshal multi protocol template %s", template.ID)
}
arr := []string{}
for k := range tempMap {
+92
View File
@@ -0,0 +1,92 @@
package fuzzplayground
import (
"fmt"
"net/http"
"net/http/httptest"
"testing"
"github.com/stretchr/testify/require"
)
func TestSQLInjectionBehavior(t *testing.T) {
server := GetPlaygroundServer()
ts := httptest.NewServer(server)
defer ts.Close()
tests := []struct {
name string
path string
expectedStatus int
shouldContainAdmin bool
}{
{
name: "Normal request",
path: "/user/75/profile", // User 75 exists and has role 'user'
expectedStatus: 200,
shouldContainAdmin: false,
},
{
name: "SQL injection with OR 1=1",
path: "/user/75 OR 1=1/profile",
expectedStatus: 200, // Should work but might return first user (admin)
shouldContainAdmin: true, // Should return admin user data
},
{
name: "SQL injection with UNION",
path: "/user/1 UNION SELECT 1,'admin',30,'admin'/profile",
expectedStatus: 200,
shouldContainAdmin: true,
},
{
name: "Template payload test - OR True with 75",
path: "/user/75 OR True/profile", // What the template actually sends
expectedStatus: 200, // Actually works!
shouldContainAdmin: true, // Let's see if it returns admin
},
{
name: "Template payload test - OR True with 55 (non-existent)",
path: "/user/55 OR True/profile", // What the template should actually send
expectedStatus: 200, // Should work due to SQL injection
shouldContainAdmin: true, // Should return admin due to OR True
},
{
name: "Test original user 55 issue",
path: "/user/55/profile", // This should fail because user 55 doesn't exist
expectedStatus: 500,
shouldContainAdmin: false,
},
{
name: "Invalid ID - non-existent",
path: "/user/999/profile",
expectedStatus: 500, // Should error due to no such user
shouldContainAdmin: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
resp, err := http.Get(ts.URL + tt.path)
require.NoError(t, err)
defer func() {
if err := resp.Body.Close(); err != nil {
t.Logf("Failed to close response body: %v", err)
}
}()
require.Equal(t, tt.expectedStatus, resp.StatusCode)
body := make([]byte, 1024)
n, _ := resp.Body.Read(body)
bodyStr := string(body[:n])
fmt.Printf("Request: %s\n", tt.path)
fmt.Printf("Status: %d\n", resp.StatusCode)
fmt.Printf("Response: %s\n\n", bodyStr)
if tt.shouldContainAdmin {
require.Contains(t, bodyStr, "admin")
}
})
}
}
+7 -7
View File
@@ -22,10 +22,10 @@ import (
"go.uber.org/multierr"
)
// ErrInvalidRequestID returns an error for invalid request IDs
func ErrInvalidRequestID(templateID, requestID string) error {
return errkit.New(fmt.Sprintf("[%s] invalid request id '%s' provided", templateID, requestID)).Build()
}
var (
// ErrInvalidRequestID is a request id error
ErrInvalidRequestID = errkit.New("invalid request id provided")
)
// ProtoOptions are options that can be passed to flow protocol callback
// ex: dns(protoOptions) <- protoOptions are optional and can be anything
@@ -256,12 +256,12 @@ func (f *FlowExecutor) ExecuteWithResults(ctx *scan.ScanContext) error {
f.reconcileProgress()
if err != nil {
ctx.LogError(err)
return errkit.Append(errkit.New(fmt.Sprintf("failed to execute flow\n%v\n", f.options.Flow)), err)
return errkit.Wrapf(err, "failed to execute flow\n%v\n", f.options.Flow)
}
runtimeErr := f.GetRuntimeErrors()
if runtimeErr != nil {
ctx.LogError(runtimeErr)
return errkit.Append(errkit.New("got following errors while executing flow"), runtimeErr)
return errkit.Wrap(runtimeErr, "got following errors while executing flow")
}
return nil
@@ -283,7 +283,7 @@ func (f *FlowExecutor) reconcileProgress() {
func (f *FlowExecutor) GetRuntimeErrors() error {
errs := []error{}
for proto, err := range f.allErrs.GetAll() {
errs = append(errs, errkit.Append(errkit.New(fmt.Sprintf("failed to execute %v protocol", proto)), err))
errs = append(errs, errkit.Wrapf(err, "failed to execute %v protocol", proto))
}
return multierr.Combine(errs...)
}
+2 -1
View File
@@ -7,6 +7,7 @@ import (
"github.com/Mzack9999/goja"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols"
"github.com/projectdiscovery/utils/errkit"
mapsutil "github.com/projectdiscovery/utils/maps"
)
@@ -61,7 +62,7 @@ func (f *FlowExecutor) requestExecutor(runtime *goja.Runtime, reqMap mapsutil.Ma
if !ok {
f.ctx.LogError(fmt.Errorf("[%v] invalid request id '%s' provided", f.options.TemplateID, id))
// compile error
if err := f.allErrs.Set(opts.protoName+":"+id, ErrInvalidRequestID(f.options.TemplateID, id)); err != nil {
if err := f.allErrs.Set(opts.protoName+":"+id, errkit.Newf("[%s] invalid request id '%s' provided", f.options.TemplateID, id)); err != nil {
f.ctx.LogError(fmt.Errorf("failed to store flow runtime errors got %v", err))
}
return matcherStatus.Load()
+4 -5
View File
@@ -1,7 +1,6 @@
package types
import (
"fmt"
"io"
"os"
"path/filepath"
@@ -831,7 +830,7 @@ func (options *Options) defaultLoadHelperFile(helperFile, templatePath string, c
}
f, err := os.Open(helperFile)
if err != nil {
return nil, errkit.Append(errkit.New(fmt.Sprintf("could not open file %v", helperFile)), err)
return nil, errkit.Wrapf(err, "could not open file %v", helperFile)
}
return f, nil
}
@@ -856,12 +855,12 @@ func (o *Options) GetValidAbsPath(helperFilePath, templatePath string) (string,
// CleanPath resolves using CWD and cleans the path
helperFilePath, err = fileutil.CleanPath(helperFilePath)
if err != nil {
return "", errkit.Append(errkit.New(fmt.Sprintf("could not clean helper file path %v", helperFilePath)), err)
return "", errkit.Wrapf(err, "could not clean helper file path %v", helperFilePath)
}
templatePath, err = fileutil.CleanPath(templatePath)
if err != nil {
return "", errkit.Append(errkit.New(fmt.Sprintf("could not clean template path %v", templatePath)), err)
return "", errkit.Wrapf(err, "could not clean template path %v", templatePath)
}
// As per rule 2, if template and helper file exist in same directory or helper file existed in any child dir of template dir
@@ -872,7 +871,7 @@ func (o *Options) GetValidAbsPath(helperFilePath, templatePath string) (string,
}
// all other cases are denied
return "", errkit.New(fmt.Sprintf("access to helper file %v denied", helperFilePath)).Build()
return "", errkit.Newf("access to helper file %v denied", helperFilePath)
}
// SetExecutionID sets the execution ID for the options
+74
View File
@@ -0,0 +1,74 @@
package testing
import (
"context"
"os"
"path/filepath"
"testing"
nuclei "github.com/projectdiscovery/nuclei/v3/lib"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
)
// BenchmarkChunkEngines simulates the aurora agent pattern where multiple chunks
// create engines against the same template set. It compares shared vs non-shared parser.
// We only load templates (no scan execution) to stress parsing/compile memory like the loader path.
func BenchmarkChunkEngines(b *testing.B) {
templatesDir := config.DefaultConfig.TemplatesDirectory
if fi, err := os.Stat(filepath.Clean(templatesDir)); err != nil || !fi.IsDir() {
b.Skipf("templates directory not available: %s", templatesDir)
return
}
// number of synthetic "chunks" (engines) to simulate
const chunks = 20
b.Run("no_shared", func(b *testing.B) {
_ = os.Unsetenv("NUCLEI_USE_SHARED_PARSER")
b.ReportAllocs()
for i := 0; i < b.N; i++ {
// create engines like separate chunks would
engines := make([]*nuclei.NucleiEngine, 0, chunks)
for c := 0; c < chunks; c++ {
ne, err := nuclei.NewNucleiEngineCtx(context.Background())
if err != nil {
b.Fatalf("engine error: %v", err)
}
engines = append(engines, ne)
}
// load templates on each engine (same set)
for _, ne := range engines {
if err := ne.LoadAllTemplates(); err != nil {
b.Fatalf("load templates error: %v", err)
}
}
for _, ne := range engines {
ne.Close()
}
}
})
b.Run("shared", func(b *testing.B) {
_ = os.Setenv("NUCLEI_USE_SHARED_PARSER", "1")
b.Cleanup(func() { _ = os.Unsetenv("NUCLEI_USE_SHARED_PARSER") })
b.ReportAllocs()
for i := 0; i < b.N; i++ {
engines := make([]*nuclei.NucleiEngine, 0, chunks)
for c := 0; c < chunks; c++ {
ne, err := nuclei.NewNucleiEngineCtx(context.Background())
if err != nil {
b.Fatalf("engine error: %v", err)
}
engines = append(engines, ne)
}
for _, ne := range engines {
if err := ne.LoadAllTemplates(); err != nil {
b.Fatalf("load templates error: %v", err)
}
}
for _, ne := range engines {
ne.Close()
}
}
})
}
+62
View File
@@ -0,0 +1,62 @@
package testing
import (
"context"
"os"
"path/filepath"
"testing"
nuclei "github.com/projectdiscovery/nuclei/v3/lib"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
)
// BenchmarkSharedParser benchmarks LoadAllTemplates with and without shared parsed cache.
// It skips if nuclei-templates directory is not present to avoid fetching during benchmarks.
func BenchmarkSharedParser(b *testing.B) {
templatesDir := config.DefaultConfig.TemplatesDirectory
if fi, err := os.Stat(filepath.Clean(templatesDir)); err != nil || !fi.IsDir() {
b.Skipf("templates directory not available: %s", templatesDir)
return
}
b.Run("no_shared", func(b *testing.B) {
_ = os.Unsetenv("NUCLEI_USE_SHARED_PARSER")
b.ReportAllocs()
for i := 0; i < b.N; i++ {
ne, err := nuclei.NewNucleiEngineCtx(context.Background())
if err != nil {
b.Fatalf("engine error: %v", err)
}
if err := ne.LoadAllTemplates(); err != nil {
b.Fatalf("load templates error: %v", err)
}
ne.Close()
}
})
b.Run("shared", func(b *testing.B) {
_ = os.Setenv("NUCLEI_USE_SHARED_PARSER", "1")
b.Cleanup(func() { _ = os.Unsetenv("NUCLEI_USE_SHARED_PARSER") })
// warm up shared cache once
warm, err := nuclei.NewNucleiEngineCtx(context.Background())
if err != nil {
b.Fatalf("warm engine error: %v", err)
}
if err := warm.LoadAllTemplates(); err != nil {
b.Fatalf("warm load error: %v", err)
}
warm.Close()
b.ReportAllocs()
for i := 0; i < b.N; i++ {
ne, err := nuclei.NewNucleiEngineCtx(context.Background())
if err != nil {
b.Fatalf("engine error: %v", err)
}
if err := ne.LoadAllTemplates(); err != nil {
b.Fatalf("load templates error: %v", err)
}
ne.Close()
}
})
}
+184
View File
@@ -0,0 +1,184 @@
package main
import (
"context"
"flag"
"fmt"
"os"
"runtime"
"strings"
"sync"
"time"
nuclei "github.com/projectdiscovery/nuclei/v3/lib"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/templates"
)
func memMB() uint64 {
var m runtime.MemStats
runtime.ReadMemStats(&m)
return m.Alloc / (1024 * 1024)
}
func snapshotCaches(label string, engines []*nuclei.NucleiEngine) {
fmt.Println(label)
max := 3
if len(engines) < max {
max = len(engines)
}
var base *templates.Parser
for i := 0; i < max; i++ {
p := engines[i].GetParser()
if base == nil {
base = p
}
fmt.Printf(" engine[%d]: parsed_cache_ptr=%p compiled_cache_ptr=%p parsed_count=%d compiled_count=%d\n", i, p.Cache(), p.CompiledCache(), p.ParsedCount(), p.CompiledCount())
}
equalParsed := true
equalCompiled := true
for i := 1; i < len(engines); i++ {
if engines[i].GetParser().Cache() != base.Cache() {
equalParsed = false
}
if engines[i].GetParser().CompiledCache() != base.CompiledCache() {
equalCompiled = false
}
}
fmt.Println(" parsed cache shared across engines:", equalParsed)
fmt.Println(" compiled cache shared across engines:", equalCompiled)
}
func runEngines(chunks int, targets []string, execute bool) (time.Duration, error) {
start := time.Now()
engines := make([]*nuclei.NucleiEngine, 0, chunks)
for i := 0; i < chunks; i++ {
ne, err := nuclei.NewNucleiEngineCtx(context.Background())
if err != nil {
return 0, fmt.Errorf("engine create: %w", err)
}
engines = append(engines, ne)
}
// Load templates
for _, ne := range engines {
if err := ne.LoadAllTemplates(); err != nil {
return 0, fmt.Errorf("load templates: %w", err)
}
}
if execute {
// Execute scans concurrently with a global 60s timeout to prevent long runs
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
var wg sync.WaitGroup
for _, ne := range engines {
ne := ne
ne.LoadTargets(targets, false)
wg.Add(1)
go func() {
defer wg.Done()
// ignore callback output
_ = ne.ExecuteCallbackWithCtx(ctx)
}()
}
wg.Wait()
}
for _, ne := range engines {
ne.Close()
}
return time.Since(start), nil
}
func main() {
var chunks int
var targetsCSV string
var execute bool
flag.IntVar(&chunks, "chunks", 10, "number of simulated chunk engines")
flag.StringVar(&targetsCSV, "targets", "https://scanme.sh,https://honey.scanme.sh", "comma-separated targets")
flag.BoolVar(&execute, "execute", false, "execute scans after loading templates (default: false)")
flag.Parse()
// Ensure templates directory exists
templatesDir := config.DefaultConfig.TemplatesDirectory
if fi, err := os.Stat(templatesDir); err != nil || !fi.IsDir() {
fmt.Printf("templates directory not found: %s\n", templatesDir)
os.Exit(1)
}
targets := []string{}
for _, t := range strings.Split(targetsCSV, ",") {
t = strings.TrimSpace(t)
if t != "" {
targets = append(targets, t)
}
}
fmt.Println("=== Agent-sim: without shared caches ===")
_ = os.Unsetenv("NUCLEI_USE_SHARED_COMPILED")
runtime.GC()
before := memMB()
// create engines to snapshot cache sharing behavior
enginesNS := make([]*nuclei.NucleiEngine, 0, chunks)
for i := 0; i < chunks; i++ {
ne, err := nuclei.NewNucleiEngineCtx(context.Background())
if err != nil {
fmt.Println("error:", err)
os.Exit(1)
}
enginesNS = append(enginesNS, ne)
}
for _, ne := range enginesNS {
if err := ne.LoadAllTemplates(); err != nil {
fmt.Println("error:", err)
os.Exit(1)
}
}
snapshotCaches("cache state (no_shared):", enginesNS)
for _, ne := range enginesNS {
ne.Close()
}
durNoShared, err := runEngines(chunks, targets, execute)
if err != nil {
fmt.Println("error:", err)
os.Exit(1)
}
runtime.GC()
after := memMB()
fmt.Printf("no_shared: duration=%s heap_before=%dMB heap_after=%dMB\n", durNoShared, before, after)
fmt.Println("\n=== Agent-sim: with shared compiled cache ===")
_ = os.Setenv("NUCLEI_USE_SHARED_COMPILED", "1")
runtime.GC()
before = memMB()
enginesSC := make([]*nuclei.NucleiEngine, 0, chunks)
for i := 0; i < chunks; i++ {
ne, err := nuclei.NewNucleiEngineCtx(context.Background())
if err != nil {
fmt.Println("error:", err)
os.Exit(1)
}
enginesSC = append(enginesSC, ne)
}
for _, ne := range enginesSC {
if err := ne.LoadAllTemplates(); err != nil {
fmt.Println("error:", err)
os.Exit(1)
}
}
snapshotCaches("cache state (shared-compiled):", enginesSC)
for _, ne := range enginesSC {
ne.Close()
}
durSharedCompiled, err := runEngines(chunks, targets, execute)
if err != nil {
fmt.Println("error:", err)
os.Exit(1)
}
runtime.GC()
after = memMB()
fmt.Printf("shared_compiled: duration=%s heap_before=%dMB heap_after=%dMB\n", durSharedCompiled, before, after)
fmt.Println("\nDone.")
}