Merge pull request #2130 from projectdiscovery/dev

v2.7.2 Release

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,15 @@
+blank_issues_enabled: false
+
+contact_links:
+
+  - name: Ask an question / advise on using nuclei
+    url: https://github.com/projectdiscovery/nuclei/discussions/categories/q-a
+    about: Ask a question or request support for using nuclei
+
+  - name: Share idea / feature to discuss for nuclei
+    url: https://github.com/projectdiscovery/nuclei/discussions/categories/ideas
+    about: Share idea / feature to discuss for nuclei
+
+  - name: Connect with PD Team (Discord)
+    url: https://discord.gg/projectdiscovery
+    about: Connect with PD Team for direct communication

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,14 +1,21 @@
 ---
 name: Feature request
-about: Suggest an idea for this project
-title: "[feature]"
-labels: ''
+about: Request feature to implement in this project
+title: ""
+labels: 'Type: Enhancement'
 assignees: ''
 
 ---
 
-**Is your feature request related to a problem? Please describe.**
-A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+<!--
+1. Please make sure to provide a detailed description with all the relevant information that might be required to start working on this feature.
+2. In case you are not sure about your request or whether the particular feature is already supported or not, please start a discussion instead.
+3. GitHub Discussion: https://github.com/projectdiscovery/nuclei/discussions/categories/ideas
+4. Join our discord server at https://discord.gg/projectdiscovery to discuss the idea on the #nuclei channel.
+-->
 
-**Describe the solution you'd like**
-A clear and concise description of what you want to happen.
+### Please describe your feature request:
+<!-- A clear and concise description of feature to implement -->
+
+### Describe the use case of this feature:
+<!-- A clear and concise description of the feature request's motivation and the use-cases in which it could be useful. -->

.github/ISSUE_TEMPLATE/issue-report.md

@@ -1,18 +1,36 @@
 ---
 name: Issue report
-about: Create a report to help us improve
-title: "[issue]"
-labels: ''
-assignees: ''
+about: Create a report to help us to improve the project
+labels: 'Type: Bug'
 
 ---
 
-**Describe the bug**
-A clear and concise description of what the bug is.
+<!-- 
+1. Please search to see if an issue already exists for the bug you encountered.
+2. For support requests, FAQs or "How to" questions, please use the GitHub Discussions section instead - https://github.com/projectdiscovery/nuclei/discussions or
+3. Join our discord server at https://discord.gg/projectdiscovery and post the question on the #nuclei channel.
+-->
 
-**Nuclei version**
-Please share the version of the nuclei you are running with `nuclei -version` 
+<!-- ISSUES MISSING IMPORTANT INFORMATION MAY BE CLOSED WITHOUT INVESTIGATION. -->
+
+### Nuclei version:
+<!-- You can find current version of nuclei with "nuclei -version" -->
+<!-- We only accept issues that are reproducible on the latest version of nuclei. -->
+<!-- You can find the latest version of project at https://github.com/projectdiscovery/nuclei/releases/ -->
+
+### Current Behavior:
+<!-- A concise description of what you're experiencing. -->
+
+### Expected Behavior:
+<!-- A concise description of what you expected to happen. -->
+
+### Steps To Reproduce:
+<!--
+Example: steps to reproduce the behavior:
+1. Run 'nuclei -t ... -u ..'
+2. See error...
+-->
 
 
-**Screenshot of the error or bug**
-please add the screenshot showing bug or issue you are facing.
+### Anything else:
+<!-- Links? References? Screenshots? Anything that will give us more context about the issue that you are encountering! -->

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,13 @@
+## Proposed changes
+
+<!-- Describe the overall picture of your modifications to help maintainers understand the pull request. PRs are required to be associated to their related issue tickets or feature request. -->
+
+
+## Checklist
+
+<!-- Put an "x" in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code. -->
+
+- [ ] Pull request is created against the [dev](https://github.com/projectdiscovery/nuclei/tree/dev) branch
+- [ ] All checks passed (lint, unit/integration/regression tests etc.) with my changes
+- [ ] I have added tests that prove my fix is effective or that my feature works
+- [ ] I have added necessary documentation (if appropriate)

.github/dependabot.yml

@@ -15,16 +15,20 @@ updates:
     commit-message:
       prefix: "chore"
       include: "scope"
+    labels:
+      - "Type: Maintenance"
 
   # Maintain dependencies for go modules
   - package-ecosystem: "gomod"
-    directory: "/"
+    directory: "v2/"
     schedule:
-      interval: "weekly"
+      interval: "daily"
     target-branch: "dev"
     commit-message:
       prefix: "chore"
       include: "scope"
+    labels:
+      - "Type: Maintenance"
 
   # Maintain dependencies for docker
   - package-ecosystem: "docker"
@@ -34,4 +38,6 @@ updates:
     target-branch: "dev"
     commit-message:
       prefix: "chore"
-      include: "scope"
+      include: "scope"
+    labels:
+      - "Type: Maintenance"

.github/workflows/build-test.yml

@@ -4,28 +4,40 @@ on:
   pull_request:
   workflow_dispatch:
 
-
-jobs:  
+jobs:
   build:
     name: Test Builds
-    runs-on: ubuntu-latest
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest, windows-latest, macOS-latest]
     steps:
       - name: Set up Go
-        uses: actions/setup-go@v2
+        uses: actions/setup-go@v3
         with:
-          go-version: 1.17
+          go-version: 1.18
 
       - name: Check out code
-        uses: actions/checkout@v2
+        uses: actions/checkout@v3
+
+      - name: Build
+        run: go build .
+        working-directory: v2/cmd/nuclei/
 
       - name: Test
         run: go test ./...
         working-directory: v2/
 
       - name: Integration Tests
-        run: bash run.sh
+        env:
+          GH_ACTION: true
+          GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
+        run: |
+          chmod +x run.sh
+          bash run.sh ${{ matrix.os }}
         working-directory: integration_tests/
 
-      - name: Build
-        run: go build .
-        working-directory: v2/cmd/nuclei/
+      # At the bottom for known issue on OSX - https://github.com/projectdiscovery/nuclei/issues/1309
+      - name: Race Condition Tests
+        run: go build -race .
+        working-directory: v2/cmd/nuclei/

.github/workflows/changelog-indexer.yaml

@@ -0,0 +1,29 @@
+name: 📑 Template-DB Indexer
+
+on:
+  push:
+    tags:
+      - '*'
+  workflow_dispatch:
+
+jobs:
+  index:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/setup-go@v3
+        with:
+          go-version: 1.18
+
+      - name: Intalling Indexer
+        run: |
+          git config --global url."https://${{ secrets.ACCESS_TOKEN }}@github".insteadOf https://github
+          git clone https://github.com/projectdiscovery/nucleish-api.git
+          cd nucleish-api/cmd/generate-index/
+          go install
+
+      - name: Generate Index
+        env:
+          AWS_ACCESS_KEY: ${{ secrets.AWS_ACCESS_KEY }}
+          AWS_SECRET_KEY: ${{ secrets.AWS_SECRET_KEY }}
+        run: |
+          generate-index -mode changelog

.github/workflows/codeql-analysis.yml

@@ -1,8 +1,9 @@
 name: 🚨 CodeQL Analysis
 
 on:
-  workflow_dispatch:
+  push:
   pull_request:
+  workflow_dispatch:
     branches:
       - dev
 
@@ -23,16 +24,16 @@ jobs:
 
     steps:
     - name: Checkout repository
-      uses: actions/checkout@v2
+      uses: actions/checkout@v3
 
     # Initializes the CodeQL tools for scanning.
     - name: Initialize CodeQL
-      uses: github/codeql-action/init@v1
+      uses: github/codeql-action/init@v2
       with:
         languages: ${{ matrix.language }}
 
     - name: Autobuild
-      uses: github/codeql-action/autobuild@v1
+      uses: github/codeql-action/autobuild@v2
 
     - name: Perform CodeQL Analysis
-      uses: github/codeql-action/analyze@v1
+      uses: github/codeql-action/analyze@v2

.github/workflows/dockerhub-push.yml

@@ -1,34 +1,40 @@
 name: 🌥 Docker Push
 
 on:
-  release:
-    types: [published]
+  workflow_run:
+    workflows: ["🎉 Release Binary"]
+    types:
+      - completed
   workflow_dispatch:
 
 jobs:
   docker:
     runs-on: ubuntu-latest
     steps:
-      -
-        name: Checkout
-        uses: actions/checkout@v2
-      -
-        name: Set up QEMU
-        uses: docker/setup-qemu-action@v1
-      -
-        name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v1
-      -
-        name: Login to DockerHub
-        uses: docker/login-action@v1 
+      - name: Git Checkout
+        uses: actions/checkout@v3
+
+      - name: Get Github tag
+        id: meta
+        run: |
+          echo "::set-output name=tag::$(curl --silent "https://api.github.com/repos/projectdiscovery/nuclei/releases/latest" | jq -r .tag_name)"
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v2
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+
+      - name: Login to DockerHub
+        uses: docker/login-action@v2 
         with:
           username: ${{ secrets.DOCKER_USERNAME }}
           password: ${{ secrets.DOCKER_TOKEN }}
-      -
-        name: Build and push
-        uses: docker/build-push-action@v2
+
+      - name: Build and push
+        uses: docker/build-push-action@v3
         with:
           context: .
-          platforms: linux/amd64,linux/arm64,linux/arm
+          platforms: linux/amd64,linux/arm64
           push: true
-          tags: projectdiscovery/nuclei:latest
+          tags: projectdiscovery/nuclei:latest,projectdiscovery/nuclei:${{ steps.meta.outputs.tag }}

.github/workflows/functional-test.yml

@@ -8,18 +8,24 @@ on:
 jobs:  
   functional:
     name: Functional Test
-    runs-on: ubuntu-latest
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest, windows-latest, macOS-latest]
     steps:
       - name: Set up Go
-        uses: actions/setup-go@v2
+        uses: actions/setup-go@v3
         with:
-          go-version: 1.17
+          go-version: 1.18
 
       - name: Check out code
-        uses: actions/checkout@v2
+        uses: actions/checkout@v3
 
       - name: Functional Tests
+        env:
+          GH_ACTION: true
+          GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
         run: |
           chmod +x run.sh
-          bash run.sh
-        working-directory: v2/cmd/functional-test
+          bash run.sh ${{ matrix.os }}
+        working-directory: v2/cmd/functional-test

.github/workflows/lint-test.yml

@@ -1,4 +1,5 @@
 name: 🙏🏻 Lint Test
+
 on:
   push:
   pull_request:
@@ -9,10 +10,14 @@ jobs:
     name: Lint Test
     runs-on: ubuntu-latest
     steps:
+      - name: Set up Go
+        uses: actions/setup-go@v3
+        with:
+          go-version: 1.18
       - name: Checkout code
-        uses: actions/checkout@v2
+        uses: actions/checkout@v3
       - name: Run golangci-lint
-        uses: golangci/golangci-lint-action@v2
+        uses: golangci/golangci-lint-action@v3.2.0
         with:
           version: latest
           args: --timeout 5m

.github/workflows/publish-docs.yaml

@@ -11,16 +11,16 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Check out code
-        uses: actions/checkout@master
+        uses: actions/checkout@v3
         with:
           persist-credentials: false
           fetch-depth: 0
           token: ${{ secrets.GITHUB_TOKEN }}
  
       - name: "Set up Go"
-        uses: actions/setup-go@v2
+        uses: actions/setup-go@v3
         with: 
-          go-version: 1.17
+          go-version: 1.18
 
       - name: Generate YAML Syntax Documentation
         id: generate-docs

.github/workflows/release-binary.yml

@@ -1,30 +1,27 @@
 name: 🎉 Release Binary
 on:
-  create:
+  push:
     tags:
-      - v*
+      - '*'
   workflow_dispatch:
 
 jobs: 
   release: 
     runs-on: ubuntu-latest
+
     steps: 
-      - 
-        name: "Check out code"
-        uses: actions/checkout@v2
+      - uses: actions/checkout@v3
         with: 
           fetch-depth: 0
-      - 
-        name: "Set up Go"
-        uses: actions/setup-go@v2
+
+      - uses: actions/setup-go@v3
         with: 
-          go-version: 1.17
-      - 
-        env: 
-          GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
-        name: "Create release on GitHub"
-        uses: goreleaser/goreleaser-action@v2
+          go-version: 1.18
+
+      - uses: goreleaser/goreleaser-action@v3
         with: 
           args: "release --rm-dist"
           version: latest
-          workdir: v2/
+          workdir: v2/
+        env: 
+          GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"

.github/workflows/sonarcloud.yml

@@ -0,0 +1,40 @@
+name: 👮🏼‍♂️ Sonarcloud
+on:
+  push:
+    branches:
+      - master
+      - dev
+  pull_request:
+    types: [opened, synchronize, reopened]
+  workflow_dispatch:
+
+jobs:
+  sonarcloud:
+    name: SonarCloud
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          fetch-depth: 0  # Shallow clones should be disabled for a better relevancy of analysis
+     
+      - name: "Set up Go"
+        uses: actions/setup-go@v3
+        with: 
+          go-version: 1.18
+      
+      - name: Run unit Tests
+        working-directory: v2/
+        run: |
+          go test -coverprofile=cov.out ./...
+
+      - name: Run Gosec Security Scanner
+        working-directory: v2/
+        run: |
+          go install github.com/securego/gosec/cmd/gosec@latest
+          gosec -no-fail -fmt=sonarqube -out report.json ./...
+     
+      - name: SonarCloud Scan
+        uses: SonarSource/sonarcloud-github-action@master
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}

.github/workflows/template-validate.yml

@@ -0,0 +1,28 @@
+name: 🛠 Template Validate
+
+on: [ push, pull_request ]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-go@v3
+        with: 
+          go-version: 1.18
+
+      - name: Cache Go
+        id: cache-go
+        uses: actions/cache@v3
+        with:
+          path: /home/runner/go
+          key: ${{ runner.os }}-go
+
+      - name: Installing Nuclei
+#        if: steps.cache-go.outputs.cache-hit != 'true'
+        run: |
+          go install github.com/projectdiscovery/nuclei/v2/cmd/nuclei@latest
+      - name: Template Validation
+        run: |
+          nuclei -validate
+          nuclei -validate -w ./workflows

DESIGN.md

@@ -0,0 +1,610 @@
+# Nuclei Architecture Document
+
+A brief overview of Nuclei Engine architecture. This document will be kept updated as the engine progresses.
+
+## pkg/templates
+
+### Template
+
+Template is the basic unit of input to the engine which describes the requests to be made, matching to be done, data to extract, etc.
+
+The template structure is described here. Template level attributes are defined here as well as convenience methods to validate, parse and compile templates creating executers. 
+
+Any attributes etc. required for the template, engine or requests to function are also set here.
+
+Workflows are also compiled, their templates are loaded and compiled as well. Any validations etc. on the paths provided are also done here.
+
+`Parse` function is the main entry point which returns a template for a `filePath` and `executorOptions`. It compiles all the requests for the templates, all the workflows, as well as any self-contained request etc. It also caches the templates in an in-memory cache.
+
+### Preprocessors
+
+Preprocessors are also applied here which can do things at template level. They get data of the template which they can alter at will on runtime. This is used in the engine to do random string generation.
+
+Custom processor can be used if they satisfy the following interface.
+
+```go
+type Preprocessor interface {
+	Process(data []byte) []byte
+}
+```
+
+## pkg/model
+
+Model package implements Information structure for Nuclei Templates. `Info` contains all major metadata information for the template. `Classification` structure can also be used to provide additional context to vulnerability data.
+
+It also specifies a `WorkflowLoader` interface that is used during workflow loading in template compilation stage.
+
+```go
+type WorkflowLoader interface {
+	GetTemplatePathsByTags(tags []string) []string
+	GetTemplatePaths(templatesList []string, noValidate bool) []string
+}
+```
+
+## pkg/protocols
+
+Protocols package implements all the request protocols supported by Nuclei. This includes http, dns, network, headless and file requests as of now. 
+
+### Request
+
+It exposes a `Request` interface that is implemented by all the request protocols supported.
+
+```go
+// Request is an interface implemented any protocol based request generator.
+type Request interface {
+	Compile(options *ExecuterOptions) error
+	Requests() int
+	GetID() string
+	Match(data map[string]interface{}, matcher *matchers.Matcher) (bool, []string)
+	Extract(data map[string]interface{}, matcher *extractors.Extractor) map[string]struct{}
+	ExecuteWithResults(input string, dynamicValues, previous output.InternalEvent, callback OutputEventCallback) error
+	MakeResultEventItem(wrapped *output.InternalWrappedEvent) *output.ResultEvent
+	MakeResultEvent(wrapped *output.InternalWrappedEvent) []*output.ResultEvent
+	GetCompiledOperators() []*operators.Operators
+}
+```
+
+Many of these methods are similar across protocols while some are very protocol specific. 
+
+A brief overview of the methods is provided below -
+
+- **Compile** - Compiles the request with provided options.
+- **Requests** - Returns total requests made.
+- **GetID** - Returns any ID for the request
+- **Match** - Used to perform matching for patterns using matchers
+- **Extract** - Used to perform extraction for patterns using extractors
+- **ExecuteWithResults** - Request execution function for input.
+- **MakeResultEventItem** - Creates a single result event for the intermediate `InternalWrappedEvent` output structure.
+- **MakeResultEvent** - Returns a slice of results based on an `InternalWrappedEvent` internal output event.
+- **GetCompiledOperators** - Returns the compiled operators for the request.
+
+`MakeDefaultResultEvent` function can be used as a default for `MakeResultEvent` function when no protocol-specific features need to be implemented for result generation. 
+
+For reference protocol requests implementations, one can look at the below packages  - 
+
+1. [pkg/protocols/http](./v2/pkg/protocols/http)
+2. [pkg/protocols/dns](./v2/pkg/protocols/dns)
+3. [pkg/protocols/network](./v2/pkg/protocols/network)
+
+### Executer
+
+All these different requests interfaces are converted to an Executer which is also an interface defined in `pkg/protocols` which is used during final execution of the template.
+
+```go
+// Executer is an interface implemented any protocol based request executer.
+type Executer interface {
+	Compile() error
+	Requests() int
+	Execute(input string) (bool, error)
+	ExecuteWithResults(input string, callback OutputEventCallback) error
+}
+```
+
+The `ExecuteWithResults` function accepts a callback, which gets provided with results during execution in form of `*output.InternalWrappedEvent` structure.
+
+The default executer is provided in `pkg/protocols/common/executer` . It takes a list of Requests and relevant `ExecuterOptions` and implements the Executer interface required for template execution. The executer during Template compilation process is created from this package and used as-is.
+
+A different executer is the Clustered Requests executer which implements the Nuclei Request clustering functionality in `pkg/templates`  We have a single HTTP  request in cases where multiple templates can be clustered and multiple operator lists to match/extract. The first HTTP request is executed while all the template matcher/extractor are evaluated separately.
+
+For Workflow execution, a separate RunWorkflow function is used which executes the workflow independently of the template execution.
+
+With this basic premise set, we can now start exploring the current runner implementation which will also walk us through the architecture of nuclei.
+
+## internal/runner
+
+### Template loading
+
+The first process after all CLI specific initialisation is the loading of template/workflow paths that the user wants to run. This is done by the packages described below.
+
+#### pkg/catalog
+
+This package is used to get paths using mixed syntax. It takes a template directory and performs resolving for template paths both from provided template and current user directory.
+
+The syntax is very versatile and can include filenames, glob patterns, directories, absolute paths, and relative-paths.
+
+
+
+Next step is the initialisation of the reporting modules which is handled in `pkg/reporting`.  
+
+#### pkg/reporting
+
+Reporting module contains exporters and trackers as well as a module for deduplication and a module for result formatting. 
+
+Exporters and Trackers are interfaces defined in pkg/reporting.
+
+```go
+// Tracker is an interface implemented by an issue tracker
+type Tracker interface {
+	CreateIssue(event *output.ResultEvent) error
+}
+
+// Exporter is an interface implemented by an issue exporter
+type Exporter interface {
+	Close() error
+	Export(event *output.ResultEvent) error
+}
+```
+
+Exporters include `Elasticsearch`, `markdown`, `sarif` . Trackers include `GitHub` , `Gitlab` and `Jira`.
+
+Each exporter and trackers implement their own configuration in YAML format and are very modular in nature, so adding new ones is easy.
+
+
+
+After reading all the inputs from various sources and initialisation other miscellaneous options, the next bit is the output writing which is done using `pkg/output` module.
+
+#### pkg/output
+
+Output package implements the output writing functionality for Nuclei.
+
+Output Writer implements the Writer interface which is called each time a result is found for nuclei.
+
+```go
+// Writer is an interface which writes output to somewhere for nuclei events.
+type Writer interface {
+	Close()
+	Colorizer() aurora.Aurora
+	Write(*ResultEvent) error
+	Request(templateID, url, requestType string, err error)
+}
+```
+
+ResultEvent structure is passed to the Nuclei Output Writer which contains the entire detail of a found result. Various intermediary types like `InternalWrappedEvent` and `InternalEvent` are used throughout nuclei protocols and matchers to describe results in various stages of execution.
+
+
+
+ Interactsh is also initialised if it is not explicitly disabled. 
+
+#### pkg/protocols/common/interactsh
+
+Interactsh module is used to provide automatic Out-of-Band vulnerability identification in Nuclei. 
+
+It uses two LRU caches, one for storing interactions for request URLs and one for storing requests for interaction URL. These both caches are used to correlated requests received to the Interactsh OOB server and Nuclei Instance. [Interactsh Client](https://github.com/projectdiscovery/interactsh/pkg/client) package does most of the heavy lifting of this module.
+
+Polling for interactions and server registration only starts when a template uses the interactsh module and is executed by nuclei. After that no registration is required for the entire run.
+
+
+
+### RunEnumeration
+
+Next we arrive in the `RunEnumeration` function of the runner.
+
+`HostErrorsCache` is initialised which is used throughout the run of Nuclei enumeration to keep track of errors per host and skip further requests if the errors are greater than the provided threshold. The functionality for the error tracking cache is defined in [hosterrorscache.go](https://github.com/projectdiscovery/nuclei/blob/master/v2/pkg/protocols/common/hosterrorscache/hosterrorscache.go) and is pretty simplistic in nature.
+
+Next the `WorkflowLoader` is initialised which used to load workflows. It exists in `v2/pkg/parsers/workflow_loader.go`
+
+The loader is initialised moving forward which is responsible for Using Catalog, Passed Tags, Filters, Paths, etc. to return compiled `Templates` and `Workflows`. 
+
+#### pkg/catalog/loader
+
+First the input passed by the user as paths is normalised to absolute paths which is done by the `pkg/catalog` module.  Next the path filter module is used to remove the excluded template/workflows paths.
+
+`pkg/parsers` module's `LoadTemplate`,`LoadWorkflow` functions are used to check if the templates pass the validation + are not excluded via tags/severity/etc. filters. If all checks are passed, then the template/workflow is parsed and returned in a compiled form by the `pkg/templates`'s `Parse` function.
+
+`Parse` function performs compilation of all the requests in a template + creates Executers from them returning a runnable Template/Workflow structure.
+
+Clustering module comes in next whose job is to cluster identical HTTP GET requests together (as a lot of the templates perform the same get requests many times, it's a good way to save many requests on large scans with lots of templates). 
+
+### pkg/operators
+
+Operators package implements all the matching and extracting logic of Nuclei. 
+
+```go
+// Operators contain the operators that can be applied on protocols
+type Operators struct {
+	Matchers []*matchers.Matcher
+	Extractors []*extractors.Extractor
+	MatchersCondition string
+}
+```
+
+A protocol only needs to embed the `operators.Operators` type shown above, and it can utilise all the matching/extracting functionality of nuclei.
+
+```go
+// MatchFunc performs matching operation for a matcher on model and returns true or false.
+type MatchFunc func(data map[string]interface{}, matcher *matchers.Matcher) (bool, []string)
+
+// ExtractFunc performs extracting operation for an extractor on model and returns true or false.
+type ExtractFunc func(data map[string]interface{}, matcher *extractors.Extractor) map[string]struct{}
+
+// Execute executes the operators on data and returns a result structure
+func (operators *Operators) Execute(data map[string]interface{}, match MatchFunc, extract ExtractFunc, isDebug bool) (*Result, bool) 
+```
+
+The core of this process is the Execute function which takes an input dictionary as well as a Match and Extract function and return a `Result` structure which is used later during nuclei execution to check for results.
+
+```go
+// Result is a result structure created from operators running on data.
+type Result struct {
+	Matched bool
+	Extracted bool
+	Matches map[string][]string
+	Extracts map[string][]string
+	OutputExtracts []string
+	DynamicValues map[string]interface{}
+	PayloadValues map[string]interface{}
+}
+```
+
+The internal logics for matching and extracting for things like words, regexes, jq, paths, etc. is specified in `pkg/operators/matchers`, `pkg/operators/extractors`. Those packages should be investigated for further look into the topic.
+
+
+### Template Execution
+
+`pkg/core` provides the engine mechanism which runs the templates/workflows on inputs. It exposes an `Execute` function which does the task of execution while also doing template clustering. The clustering can also be disabled optionally by the user.
+ 
+An example of using the core engine is provided below.
+
+```go
+engine := core.New(r.options)
+engine.SetExecuterOptions(executerOpts)
+results := engine.ExecuteWithOpts(finalTemplates, r.hmapInputProvider, true)
+```
+
+### Using Nuclei From Go Code
+
+An example of using Nuclei From Go Code to run templates on targets is provided below.
+
+```go
+package main
+
+import (
+	"fmt"
+	"log"
+	"os"
+	"path"
+
+	"github.com/logrusorgru/aurora"
+	"go.uber.org/ratelimit"
+
+	"github.com/projectdiscovery/goflags"
+	"github.com/projectdiscovery/nuclei/v2/pkg/catalog"
+	"github.com/projectdiscovery/nuclei/v2/pkg/catalog/config"
+	"github.com/projectdiscovery/nuclei/v2/pkg/catalog/loader"
+	"github.com/projectdiscovery/nuclei/v2/pkg/core"
+	"github.com/projectdiscovery/nuclei/v2/pkg/core/inputs"
+	"github.com/projectdiscovery/nuclei/v2/pkg/output"
+	"github.com/projectdiscovery/nuclei/v2/pkg/parsers"
+	"github.com/projectdiscovery/nuclei/v2/pkg/protocols"
+	"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/hosterrorscache"
+	"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/interactsh"
+	"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/protocolinit"
+	"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/protocolstate"
+	"github.com/projectdiscovery/nuclei/v2/pkg/reporting"
+	"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
+	"github.com/projectdiscovery/nuclei/v2/pkg/types"
+)
+
+func main() {
+	cache := hosterrorscache.New(30, hosterrorscache.DefaultMaxHostsCount)
+	defer cache.Close()
+
+	mockProgress := &testutils.MockProgressClient{}
+	reportingClient, _ := reporting.New(&reporting.Options{}, "")
+	defer reportingClient.Close()
+
+	outputWriter := testutils.NewMockOutputWriter()
+	outputWriter.WriteCallback = func(event *output.ResultEvent) {
+		fmt.Printf("Got Result: %v\n", event)
+	}
+
+	defaultOpts := types.DefaultOptions()
+	protocolstate.Init(defaultOpts)
+	protocolinit.Init(defaultOpts)
+
+	defaultOpts.Templates = goflags.FileOriginalNormalizedStringSlice{"dns/cname-service-detection.yaml"}
+	defaultOpts.ExcludeTags = config.ReadIgnoreFile().Tags
+
+	interactOpts := interactsh.NewDefaultOptions(outputWriter, reportingClient, mockProgress)
+	interactClient, err := interactsh.New(interactOpts)
+	if err != nil {
+		log.Fatalf("Could not create interact client: %s\n", err)
+	}
+	defer interactClient.Close()
+
+	home, _ := os.UserHomeDir()
+	catalog := catalog.New(path.Join(home, "nuclei-templates"))
+	executerOpts := protocols.ExecuterOptions{
+		Output:          outputWriter,
+		Options:         defaultOpts,
+		Progress:        mockProgress,
+		Catalog:         catalog,
+		IssuesClient:    reportingClient,
+		RateLimiter:     ratelimit.New(150),
+		Interactsh:      interactClient,
+		HostErrorsCache: cache,
+		Colorizer:       aurora.NewAurora(true),
+		ResumeCfg:       types.NewResumeCfg(),
+	}
+	engine := core.New(defaultOpts)
+	engine.SetExecuterOptions(executerOpts)
+
+	workflowLoader, err := parsers.NewLoader(&executerOpts)
+	if err != nil {
+		log.Fatalf("Could not create workflow loader: %s\n", err)
+	}
+	executerOpts.WorkflowLoader = workflowLoader
+
+	configObject, err := config.ReadConfiguration()
+	if err != nil {
+		log.Fatalf("Could not read config: %s\n", err)
+	}
+	store, err := loader.New(loader.NewConfig(defaultOpts, configObject, catalog, executerOpts))
+	if err != nil {
+		log.Fatalf("Could not create loader client: %s\n", err)
+	}
+	store.Load()
+
+	input := &inputs.SimpleInputProvider{Inputs: []string{"docs.hackerone.com"}}
+	_ = engine.Execute(store.Templates(), input)
+	engine.WorkPool().Wait() // Wait for the scan to finish
+}
+```
+
+### Adding a New Protocol
+
+Protocols form the core of Nuclei Engine. All the request types like `http`, `dns`, etc. are implemented in form of protocol requests.
+
+A protocol must implement the `Protocol` and `Request` interfaces described above in `pkg/protocols`. We'll take the example of an existing protocol implementation - websocket for this short reference around Nuclei internals.
+
+The code for the websocket protocol is contained in `pkg/protocols/others/websocket`. 
+
+Below a high level skeleton of the websocket implementation is provided with all the important parts present.
+
+```go
+package websocket
+
+// Request is a request for the Websocket protocol
+type Request struct {
+	// Operators for the current request go here.
+	operators.Operators `yaml:",inline,omitempty"`
+	CompiledOperators   *operators.Operators `yaml:"-"`
+
+	// description: |
+	//   Address contains address for the request
+	Address string `yaml:"address,omitempty" jsonschema:"title=address for the websocket request,description=Address contains address for the request"`
+
+    // declarations here
+}
+
+// Compile compiles the request generators preparing any requests possible.
+func (r *Request) Compile(options *protocols.ExecuterOptions) error {
+	r.options = options
+
+    // request compilation here as well as client creation
+ 
+	if len(r.Matchers) > 0 || len(r.Extractors) > 0 {
+		compiled := &r.Operators
+		if err := compiled.Compile(); err != nil {
+			return errors.Wrap(err, "could not compile operators")
+		}
+		r.CompiledOperators = compiled
+	}
+	return nil
+}
+
+// Requests returns the total number of requests the rule will perform
+func (r *Request) Requests() int {
+	if r.generator != nil {
+		return r.generator.NewIterator().Total()
+	}
+	return 1
+}
+
+// GetID returns the ID for the request if any.
+func (r *Request) GetID() string {
+	return ""
+}
+
+// ExecuteWithResults executes the protocol requests and returns results instead of writing them.
+func (r *Request) ExecuteWithResults(input string, dynamicValues, previous output.InternalEvent, callback protocols.OutputEventCallback) error {
+    // payloads init here
+	if err := r.executeRequestWithPayloads(input, hostname, value, previous, callback); err != nil {
+		return err
+	}
+	return nil
+}
+
+// ExecuteWithResults executes the protocol requests and returns results instead of writing them.
+func (r *Request) executeRequestWithPayloads(input, hostname string, dynamicValues, previous output.InternalEvent, callback protocols.OutputEventCallback) error {
+	header := http.Header{}
+
+    // make the actual request here after setting all options
+
+	event := eventcreator.CreateEventWithAdditionalOptions(r, data, r.options.Options.Debug || r.options.Options.DebugResponse, func(internalWrappedEvent *output.InternalWrappedEvent) {
+		internalWrappedEvent.OperatorsResult.PayloadValues = payloadValues
+	})
+	if r.options.Options.Debug || r.options.Options.DebugResponse {
+		responseOutput := responseBuilder.String()
+		gologger.Debug().Msgf("[%s] Dumped Websocket response for %s", r.options.TemplateID, input)
+		gologger.Print().Msgf("%s", responsehighlighter.Highlight(event.OperatorsResult, responseOutput, r.options.Options.NoColor))
+	}
+
+	callback(event)
+	return nil
+}
+
+func (r *Request) MakeResultEventItem(wrapped *output.InternalWrappedEvent) *output.ResultEvent {
+	data := &output.ResultEvent{
+		TemplateID:       types.ToString(r.options.TemplateID),
+		TemplatePath:     types.ToString(r.options.TemplatePath),
+		// ... setting more values for result event
+	}
+	return data
+}
+
+// Match performs matching operation for a matcher on model and returns:
+// true and a list of matched snippets if the matcher type is supports it
+// otherwise false and an empty string slice
+func (r *Request) Match(data map[string]interface{}, matcher *matchers.Matcher) (bool, []string) {
+	return protocols.MakeDefaultMatchFunc(data, matcher)
+}
+
+// Extract performs extracting operation for an extractor on model and returns true or false.
+func (r *Request) Extract(data map[string]interface{}, matcher *extractors.Extractor) map[string]struct{} {
+	return protocols.MakeDefaultExtractFunc(data, matcher)
+}
+
+// MakeResultEvent creates a result event from internal wrapped event
+func (r *Request) MakeResultEvent(wrapped *output.InternalWrappedEvent) []*output.ResultEvent {
+	return protocols.MakeDefaultResultEvent(r, wrapped)
+}
+
+// GetCompiledOperators returns a list of the compiled operators
+func (r *Request) GetCompiledOperators() []*operators.Operators {
+	return []*operators.Operators{r.CompiledOperators}
+}
+
+// Type returns the type of the protocol request
+func (r *Request) Type() templateTypes.ProtocolType {
+	return templateTypes.WebsocketProtocol
+}
+```
+
+Almost all of these protocols have boilerplate functions for which default implementations have been provided in the `providers` package. Examples are the implementation of `Match`, `Extract`, `MakeResultEvent`, GetCompiledOperators`, etc. which are almost same throughout Nuclei protocols code. It is enough to copy-paste them unless customization is required.
+
+`eventcreator` package offers `CreateEventWithAdditionalOptions` function which can be used to create result events after doing request execution.
+
+Step by step description of how to add a new protocol to Nuclei - 
+
+1. Add the protocol implementation in `pkg/protocols` directory. If it's a small protocol with fewer options, considering adding it to the `pkg/protocols/others` directory. Add the enum for the new protocol to `v2/pkg/templates/types/types.go`.
+
+2. Add the protocol request structure to the `Template` structure fields. This is done in `pkg/templates/templates.go` with the corresponding import line.
+
+```go
+
+import (
+	...
+	"github.com/projectdiscovery/nuclei/v2/pkg/protocols/others/websocket"
+)
+
+// Template is a YAML input file which defines all the requests and
+// other metadata for a template.
+type Template struct {
+	...
+	// description: |
+	//   Websocket contains the Websocket request to make in the template.
+	RequestsWebsocket []*websocket.Request `yaml:"websocket,omitempty" json:"websocket,omitempty" jsonschema:"title=websocket requests to make,description=Websocket requests to make for the template"`
+	...
+}
+```
+
+Also add the protocol case to the `Type` function as well as the `TemplateTypes` array in the same `templates.go` file.
+
+```go
+// TemplateTypes is a list of accepted template types
+var TemplateTypes = []string{
+	...
+	"websocket",
+}
+
+// Type returns the type of the template
+func (t *Template) Type() templateTypes.ProtocolType {
+	...
+	case len(t.RequestsWebsocket) > 0:
+		return templateTypes.WebsocketProtocol
+	default:
+		return ""
+	}
+}
+```
+
+3. Add the protocol request to the `Requests` function and `compileProtocolRequests` function in the `compile.go` file in same directory.
+
+```go
+
+// Requests return the total request count for the template
+func (template *Template) Requests() int {
+	return len(template.RequestsDNS) +
+		...
+		len(template.RequestsSSL) +
+		len(template.RequestsWebsocket)
+}
+
+
+// compileProtocolRequests compiles all the protocol requests for the template
+func (template *Template) compileProtocolRequests(options protocols.ExecuterOptions) error {
+	...
+
+	case len(template.RequestsWebsocket) > 0:
+		requests = template.convertRequestToProtocolsRequest(template.RequestsWebsocket)
+	}
+	template.Executer = executer.NewExecuter(requests, &options)
+	return nil
+}
+```
+
+That's it, you've added a new protocol to Nuclei. The next good step would be to write integration tests which are described in `integration-tests` and `cmd/integration-tests` directories.
+
+## Project Structure
+
+- [v2/pkg/reporting](./v2/pkg/reporting) - Reporting modules for nuclei.
+- [v2/pkg/reporting/exporters/sarif](./v2/pkg/reporting/exporters/sarif) - Sarif Result Exporter
+- [v2/pkg/reporting/exporters/markdown](./v2/pkg/reporting/exporters/markdown) - Markdown Result Exporter
+- [v2/pkg/reporting/exporters/es](./v2/pkg/reporting/exporters/es) - Elasticsearch Result Exporter
+- [v2/pkg/reporting/dedupe](./v2/pkg/reporting/dedupe) - Dedupe module for Results
+- [v2/pkg/reporting/trackers/gitlab](./v2/pkg/reporting/trackers/gitlab) - Gitlab Issue Tracker Exporter
+- [v2/pkg/reporting/trackers/jira](./v2/pkg/reporting/trackers/jira) - Jira Issue Tracker Exporter
+- [v2/pkg/reporting/trackers/github](./v2/pkg/reporting/trackers/github) - GitHub Issue Tracker Exporter
+- [v2/pkg/reporting/format](./v2/pkg/reporting/format) - Result Formatting Functions
+- [v2/pkg/parsers](./v2/pkg/parsers) - Implements template as well as workflow loader for initial template discovery, validation and - loading.
+- [v2/pkg/types](./v2/pkg/types) - Contains CLI options as well as misc helper functions.
+- [v2/pkg/progress](./v2/pkg/progress) - Progress tracking
+- [v2/pkg/operators](./v2/pkg/operators) - Operators for Nuclei
+- [v2/pkg/operators/common/dsl](./v2/pkg/operators/common/dsl) - DSL functions for Nuclei YAML Syntax
+- [v2/pkg/operators/matchers](./v2/pkg/operators/matchers) - Matchers implementation
+- [v2/pkg/operators/extractors](./v2/pkg/operators/extractors) - Extractors implementation
+- [v2/pkg/catalog](./v2/pkg/catalog) - Template loading from disk helpers
+- [v2/pkg/catalog/config](./v2/pkg/catalog/config) - Internal configuration management
+- [v2/pkg/catalog/loader](./v2/pkg/catalog/loader) - Implements loading and validation of templates and workflows.
+- [v2/pkg/catalog/loader/filter](./v2/pkg/catalog/loader/filter) - Filter filters templates based on tags and paths
+- [v2/pkg/output](./v2/pkg/output) - Output module for nuclei
+- [v2/pkg/workflows](./v2/pkg/workflows) - Workflow execution logic + declarations
+- [v2/pkg/utils](./v2/pkg/utils) - Utility functions
+- [v2/pkg/model](./v2/pkg/model) - Template Info + misc
+- [v2/pkg/templates](./v2/pkg/templates) - Templates core starting point
+- [v2/pkg/templates/cache](./v2/pkg/templates/cache) - Templates cache
+- [v2/pkg/protocols](./v2/pkg/protocols) - Protocol Specification
+- [v2/pkg/protocols/file](./v2/pkg/protocols/file) - File protocol
+- [v2/pkg/protocols/network](./v2/pkg/protocols/network) - Network protocol
+- [v2/pkg/protocols/common/expressions](./v2/pkg/protocols/common/expressions) - Expression evaluation + Templating Support
+- [v2/pkg/protocols/common/interactsh](./v2/pkg/protocols/common/interactsh) - Interactsh integration
+- [v2/pkg/protocols/common/generators](./v2/pkg/protocols/common/generators) - Payload support for Requests (Sniper, etc.)
+- [v2/pkg/protocols/common/executer](./v2/pkg/protocols/common/executer) - Default Template Executer
+- [v2/pkg/protocols/common/replacer](./v2/pkg/protocols/common/replacer) - Template replacement helpers
+- [v2/pkg/protocols/common/helpers/eventcreator](./v2/pkg/protocols/common/helpers/eventcreator) - Result event creator
+- [v2/pkg/protocols/common/helpers/responsehighlighter](./v2/pkg/protocols/common/helpers/responsehighlighter) - Debug response highlighter
+- [v2/pkg/protocols/common/helpers/deserialization](./v2/pkg/protocols/common/helpers/deserialization) - Deserialization helper functions
+- [v2/pkg/protocols/common/hosterrorscache](./v2/pkg/protocols/common/hosterrorscache) - Host errors cache for tracking erroring hosts
+- [v2/pkg/protocols/offlinehttp](./v2/pkg/protocols/offlinehttp) - Offline http protocol
+- [v2/pkg/protocols/http](./v2/pkg/protocols/http) - HTTP protocol
+- [v2/pkg/protocols/http/race](./v2/pkg/protocols/http/race) - HTTP Race Module
+- [v2/pkg/protocols/http/raw](./v2/pkg/protocols/http/raw) - HTTP Raw Request Support
+- [v2/pkg/protocols/headless](./v2/pkg/protocols/headless) - Headless Module
+- [v2/pkg/protocols/headless/engine](./v2/pkg/protocols/headless/engine) - Internal Headless implementation
+- [v2/pkg/protocols/dns](./v2/pkg/protocols/dns) - DNS protocol
+- [v2/pkg/projectfile](./v2/pkg/projectfile) - Project File Implementation
+
+### Notes
+
+1. The matching as well as interim output functionality is a bit complex, we should simplify it a bit as well.

Dockerfile

@@ -1,7 +1,7 @@
-FROM golang:1.17.1-alpine as build-env
-RUN GO111MODULE=on go get -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei
+FROM golang:1.18.3-alpine as build-env
+RUN go install -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei@latest
 
-FROM alpine:3.14
-RUN apk add --no-cache bind-tools ca-certificates
+FROM alpine:3.16.0
+RUN apk add --no-cache bind-tools ca-certificates chromium
 COPY --from=build-env /go/bin/nuclei /usr/local/bin/nuclei
 ENTRYPOINT ["nuclei"]

README.md

@@ -7,12 +7,14 @@
 
 
 <p align="center">
-<a href="https://goreportcard.com/report/github.com/projectdiscovery/nuclei"><img src="https://goreportcard.com/badge/github.com/projectdiscovery/nuclei"></a>
-<a href="https://github.com/projectdiscovery/nuclei/issues"><img src="https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat"></a>
-<a href="https://github.com/projectdiscovery/nuclei/releases"><img src="https://img.shields.io/github/release/projectdiscovery/nuclei"></a>
-<a href="https://twitter.com/pdnuclei"><img src="https://img.shields.io/twitter/follow/pdnuclei.svg?logo=twitter"></a>
+<img src="https://img.shields.io/github/go-mod/go-version/projectdiscovery/nuclei?filename=v2%2Fgo.mod">
+<a href="https://github.com/projectdiscovery/nuclei/releases"><img src="https://img.shields.io/github/downloads/projectdiscovery/nuclei/total">
+<a href="https://github.com/projectdiscovery/nuclei/graphs/contributors"><img src="https://img.shields.io/github/contributors-anon/projectdiscovery/nuclei">
+<a href="https://github.com/projectdiscovery/nuclei/releases/"><img src="https://img.shields.io/github/release/projectdiscovery/nuclei">
+<a href="https://github.com/projectdiscovery/nuclei/issues"><img src="https://img.shields.io/github/issues-raw/projectdiscovery/nuclei">
+<a href="https://github.com/projectdiscovery/nuclei/discussions"><img src="https://img.shields.io/github/discussions/projectdiscovery/nuclei">
 <a href="https://discord.gg/projectdiscovery"><img src="https://img.shields.io/discord/695645237418131507.svg?logo=discord"></a>
-<a href="https://github.com/projectdiscovery/nuclei/actions/workflows/build-test.yml"><img src="https://github.com/projectdiscovery/nuclei/actions/workflows/build-test.yml/badge.svg?branch=master"></a>
+<a href="https://twitter.com/pdnuclei"><img src="https://img.shields.io/twitter/follow/pdnuclei.svg?logo=twitter"></a>
 </p>
       
 <p align="center">
@@ -26,11 +28,16 @@
   <a href="https://discord.gg/projectdiscovery">Join Discord</a>
 </p>
 
+<p align="center">
+  <a href="https://github.com/projectdiscovery/nuclei/blob/master/README.md">English</a> •
+  <a href="https://github.com/projectdiscovery/nuclei/blob/master/README_CN.md">中文</a>
+</p>
+
 ---
 
-Nuclei is used to send requests across targets based on a template leading to zero false positives and providing fast scanning on large number of hosts. Nuclei offers scanning for a variety of protocols including TCP, DNS, HTTP, File, etc. With powerful and flexible templating, all kinds of security checks can be modelled with Nuclei.
+Nuclei is used to send requests across targets based on a template, leading to zero false positives and providing fast scanning on a large number of hosts. Nuclei offers scanning for a variety of protocols, including TCP, DNS, HTTP, SSL, File, Whois, Websocket, Headless etc. With powerful and flexible templating, Nuclei can be used to model all kinds of security checks.
 
-We have a [dedicated repository](https://github.com/projectdiscovery/nuclei-templates) that houses various type of vulnerability templates contributed by **more than 200** security researchers and engineers.
+We have a [dedicated repository](https://github.com/projectdiscovery/nuclei-templates) that houses various type of vulnerability templates contributed by **more than 300** security researchers and engineers.
 
 
 
@@ -44,8 +51,10 @@ We have a [dedicated repository](https://github.com/projectdiscovery/nuclei-temp
 
 # Install Nuclei
 
+Nuclei requires **go1.17** to install successfully. Run the following command to install the latest version -
+
 ```sh
-GO111MODULE=on go get -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei
+go install -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei@latest
 ```
 
 **More installation [methods can be found here](https://nuclei.projectdiscovery.io/nuclei/get-started/).**
@@ -56,7 +65,7 @@ GO111MODULE=on go get -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei
 
 ### Nuclei Templates
 
-Nuclei has had built-in support for automatic template download/update as default since version [v2.5.2](https://github.com/projectdiscovery/nuclei/releases/tag/v2.5.2). [**Nuclei-Templates**](https://github.com/projectdiscovery/nuclei-templates) project provides a community-contributed list of ready-to-use templates that is constantly updated.
+Nuclei has built-in support for automatic template download/update as default since version [v2.5.2](https://github.com/projectdiscovery/nuclei/releases/tag/v2.5.2). [**Nuclei-Templates**](https://github.com/projectdiscovery/nuclei-templates) project provides a community-contributed list of ready-to-use templates that is constantly updated.
 
 You may still use the `update-templates` flag to update the nuclei templates at any time; You can write your own checks for your individual workflow and needs following Nuclei's [templating guide](https://nuclei.projectdiscovery.io/templating-guide/).
 
@@ -75,7 +84,7 @@ nuclei -h
 This will display help for the tool. Here are all the switches it supports.
 
 
-```yaml
+```console
 Nuclei is a fast, template based vulnerability scanner focusing
 on extensive configurability, massive extensibility and ease of use.
 
@@ -86,97 +95,126 @@ Flags:
 TARGET:
    -u, -target string[]  target URLs/hosts to scan
    -l, -list string      path to file containing a list of target URLs/hosts to scan (one per line)
+   -resume string        Resume scan using resume.cfg (clustering will be disabled)
 
 TEMPLATES:
-   -tl                      list all available templates
-   -t, -templates string[]  template or template directory paths to include in the scan
-   -w, -workflows string[]  list of workflows to run
-   -nt, -new-templates      run newly added templates only
-   -validate                validate the passed templates to nuclei
+   -nt, -new-templates          run only new templates added in latest nuclei-templates release
+   -as, -automatic-scan         automatic web scan using wappalyzer technology detection to tags mapping
+   -t, -templates string[]      list of template or template directory to run (comma-separated, file)
+   -tu, -template-url string[]  list of template urls to run (comma-separated, file)
+   -w, -workflows string[]      list of workflow or workflow directory to run (comma-separated, file)
+   -wu, -workflow-url string[]  list of workflow urls to run (comma-separated, file)
+   -validate                    validate the passed templates to nuclei
+   -tl                          list all available templates
 
 FILTERING:
-   -tags string[]                         execute a subset of templates that contain the provided tags
-   -include-tags string[]                 tags from the default deny list that permit executing more intrusive templates
-   -etags, -exclude-tags string[]         exclude templates with the provided tags
-   -include-templates string[]            templates to be executed even if they are excluded either by default or configuration
-   -exclude-templates, -exclude string[]  template or template directory paths to exclude
-   -severity, -impact value[]             Templates to run based on severity. Possible values: info, low, medium, high, critical
-   -author string[]                       execute templates that are (co-)created by the specified authors
+   -a, -author string[]              templates to run based on authors (comma-separated, file)
+   -tags string[]                    templates to run based on tags (comma-separated, file)
+   -etags, -exclude-tags string[]    templates to exclude based on tags (comma-separated, file)
+   -itags, -include-tags string[]    tags to be executed even if they are excluded either by default or configuration
+   -id, -template-id string[]        templates to run based on template ids (comma-separated, file)
+   -eid, -exclude-id string[]        templates to exclude based on template ids (comma-separated, file)
+   -it, -include-templates string[]  templates to be executed even if they are excluded either by default or configuration
+   -et, -exclude-templates string[]  template or template directory to exclude (comma-separated, file)
+   -s, -severity value[]             templates to run based on severity. Possible values: info, low, medium, high, critical, unknown
+   -es, -exclude-severity value[]    templates to exclude based on severity. Possible values: info, low, medium, high, critical, unknown
+   -pt, -type value[]                templates to run based on protocol type. Possible values: dns, file, http, headless, network, workflow, ssl, websocket, whois
+   -ept, -exclude-type value[]       templates to exclude based on protocol type. Possible values: dns, file, http, headless, network, workflow, ssl, websocket, whois
 
 OUTPUT:
    -o, -output string            output file to write found issues/vulnerabilities
+   -sresp, -store-resp           store all request/response passed through nuclei to output directory
+   -srd, -store-resp-dir string  store all request/response passed through nuclei to custom directory (default "output")
    -silent                       display findings only
-   -v, -verbose                  show verbose output
-   -vv                           display extra verbose information
    -nc, -no-color                disable output content coloring (ANSI escape codes)
    -json                         write output in JSONL(ines) format
    -irr, -include-rr             include request/response pairs in the JSONL output (for findings only)
-   -nm, -no-meta                 don't display match metadata in CLI output
-   -nts, -no-timestamp           don't display timestamp metadata in CLI output
-   -rdb, -report-db string       local nuclei reporting database (always use this to persist report data)
+   -nm, -no-meta                 disable printing result metadata in cli output
+   -nts, -no-timestamp           disable printing timestamp in cli output
+   -rdb, -report-db string       nuclei reporting database (always use this to persist report data)
+   -ms, -matcher-status          display match failure status
    -me, -markdown-export string  directory to export results in markdown format
    -se, -sarif-export string     file to export results in SARIF format
 
 CONFIGURATIONS:
    -config string              path to the nuclei configuration file
+   -fr, -follow-redirects      enable following redirects for http templates
+   -mr, -max-redirects int     max number of redirects to follow for http templates (default 10)
+   -dr, -disable-redirects     disable redirects for http templates
    -rc, -report-config string  nuclei reporting module configuration file
-   -H, -header string[]        custom headers in header:value format
-   -V, -var value              custom vars in var=value format
+   -H, -header string[]        custom header/cookie to include in all http request in header:value format (cli, file)
+   -V, -var value              custom vars in key=value format
    -r, -resolvers string       file containing resolver list for nuclei
-   -system-resolvers           use system DNS resolving as error fallback
+   -sr, -system-resolvers      use system DNS resolving as error fallback
    -passive                    enable passive HTTP response processing mode
-   -env-vars                   enable environment variables support
+   -ev, -env-vars              enable environment variables to be used in template
+   -cc, -client-cert string    client certificate file (PEM-encoded) used for authenticating against scanned hosts
+   -ck, -client-key string     client key file (PEM-encoded) used for authenticating against scanned hosts
+   -ca, -client-ca string      client certificate authority file (PEM-encoded) used for authenticating against scanned hosts
+   -sml, -show-match-line      show match lines for file templates, works with extractors only
+   -ztls                       use ztls library with autofallback to standard one for tls13
+   -sni string                 tls sni hostname to use (default: input domain name)
 
 INTERACTSH:
-   -no-interactsh                     disable interactsh server for OOB testing
-   -interactsh-url string             interactsh server url for self-hosted instance (default "https://interact.sh")
-   -interactsh-token string           authentication token for self-hosted interactsh server
-   -interactions-cache-size int       number of requests to keep in the interactions cache (default 5000)
-   -interactions-eviction int         number of seconds to wait before evicting requests from cache (default 60)
-   -interactions-poll-duration int    number of seconds to wait before each interaction poll request (default 5)
-   -interactions-cooldown-period int  extra time for interaction polling before exiting (default 5)
+   -iserver, -interactsh-server string  interactsh server url for self-hosted instance (default: oast.pro,oast.live,oast.site,oast.online,oast.fun,oast.me)
+   -itoken, -interactsh-token string    authentication token for self-hosted interactsh server
+   -interactions-cache-size int         number of requests to keep in the interactions cache (default 5000)
+   -interactions-eviction int           number of seconds to wait before evicting requests from cache (default 60)
+   -interactions-poll-duration int      number of seconds to wait before each interaction poll request (default 5)
+   -interactions-cooldown-period int    extra time for interaction polling before exiting (default 5)
+   -ni, -no-interactsh                  disable interactsh server for OAST testing, exclude OAST based templates
 
 RATE-LIMIT:
-   -rl, -rate-limit int          maximum number of requests to send per second (default 150)
-   -rlm, -rate-limit-minute int  maximum number of requests to send per minute
-   -bs, -bulk-size int           maximum number of hosts to be analyzed in parallel per template (default 25)
-   -c, -concurrency int          maximum number of templates to be executed in parallel (default 10)
+   -rl, -rate-limit int            maximum number of requests to send per second (default 150)
+   -rlm, -rate-limit-minute int    maximum number of requests to send per minute
+   -bs, -bulk-size int             maximum number of hosts to be analyzed in parallel per template (default 25)
+   -c, -concurrency int            maximum number of templates to be executed in parallel (default 25)
+   -hbs, -headless-bulk-size int   maximum number of headless hosts to be analyzed in parallel per template (default 10)
+   -hc, -headless-concurrency int  maximum number of headless templates to be executed in parallel (default 10)
 
 OPTIMIZATIONS:
-   -timeout int               time to wait in seconds before timeout (default 5)
-   -retries int               number of times to retry a failed request (default 1)
-   -max-host-error int        max errors for a host before skipping from scan (default 30)
-   -project                   use a project folder to avoid sending same request multiple times
-   -project-path string       set a specific project path (default "$TMPDIR/")
-   -spm, -stop-at-first-path  stop processing HTTP requests after the first match (may break template/workflow logic)
+   -timeout int                time to wait in seconds before timeout (default 5)
+   -retries int                number of times to retry a failed request (default 1)
+   -ldp, -leave-default-ports  leave default HTTP/HTTPS ports (eg. host:80,host:443
+   -mhe, -max-host-error int   max errors for a host before skipping from scan (default 30)
+   -project                    use a project folder to avoid sending same request multiple times
+   -project-path string        set a specific project path
+   -spm, -stop-at-first-path   stop processing HTTP requests after the first match (may break template/workflow logic)
+   -stream                     stream mode - start elaborating without sorting the input
 
 HEADLESS:
-   -headless          enable templates that require headless browser support
-   -page-timeout int  seconds to wait for each page in headless mode (default 20)
-   -show-browser      show the browser on the screen when running templates with headless mode
+   -headless            enable templates that require headless browser support (root user on linux will disable sandbox)
+   -page-timeout int    seconds to wait for each page in headless mode (default 20)
+   -sb, -show-browser   show the browser on the screen when running templates with headless mode
+   -sc, -system-chrome  Use local installed chrome browser instead of nuclei installed
 
 DEBUG:
-   -debug                     show all requests and responses
-   -debug-req                 show all sent requests
-   -debug-resp                show all received responses
-   -proxy, -proxy-url string  URL of the HTTP proxy server
-   -proxy-socks-url string    URL of the SOCKS proxy server
-   -trace-log string          file to write sent requests trace log
-   -version                   show nuclei version
-   -tv, -templates-version    shows the version of the installed nuclei-templates
+   -debug                    show all requests and responses
+   -dreq, -debug-req         show all sent requests
+   -dresp, -debug-resp       show all received responses
+   -p, -proxy string[]       list of http/socks5 proxy to use (comma separated or file input)
+   -pi, -proxy-internal      proxy all internal requests
+   -tlog, -trace-log string  file to write sent requests trace log
+   -elog, -error-log string  file to write sent requests error log
+   -version                  show nuclei version
+   -hm, -hang-monitor        enable nuclei hang monitoring
+   -v, -verbose              show verbose output
+   -vv                       display templates loaded for scan
+   -ep, -enable-pprof        enable pprof debugging server
+   -tv, -templates-version   shows the version of the installed nuclei-templates
 
 UPDATE:
-   -update                        update nuclei to the latest released version
-   -ut, -update-templates         update the community templates to latest released version
-   -nut, -no-update-templates     do not check for nuclei-templates updates
-   -ud, -update-directory string  overwrite the default nuclei-templates directory (default "$HOME/nuclei-templates")
+   -update                        update nuclei engine to the latest released version
+   -ut, -update-templates         update nuclei-templates to latest released version
+   -ud, -update-directory string  overwrite the default directory to install nuclei-templates
+   -duc, -disable-update-check    disable automatic nuclei/templates update check
 
 STATISTICS:
    -stats                    display statistics about the running scan
-   -stats-json               write statistics data to an output file in JSONL(ines) format
+   -sj, -stats-json          write statistics data to an output file in JSONL(ines) format
    -si, -stats-interval int  number of seconds to wait between showing a statistics update (default 5)
-   -metrics                  expose nuclei metrics on a port
-   -metrics-port int         port to expose nuclei metrics on (default 9092)
+   -m, -metrics              expose nuclei metrics on a port
+   -mp, -metrics-port int    port to expose nuclei metrics on (default 9092)
 ```
 
 ### Running Nuclei
@@ -242,7 +280,7 @@ Please check our other open-source projects that might fit into your bug bounty
 
 Nuclei immensely improve how you approach security assessment by augmenting the manual, repetitive processes. Consultancies are already converting their manual assessment steps with Nuclei, it allows them to run set of their custom assessment approach across thousands of hosts in an automated manner. 
 
-Pen-testers get the full power of our public templates and customization capabilities to speed-up their assessment process, and specifically with the regression cycle where you can easily verify the fix.
+Pen-testers get the full power of our public templates and customization capabilities to speed up their assessment process, and specifically with the regression cycle where you can easily verify the fix.
 
 - Easily create your compliance, standards suite (e.g. OWASP Top 10) checklist.
 - With capabilities like [fuzz](https://nuclei.projectdiscovery.io/templating-guide/#advance-fuzzing) and [workflows](https://nuclei.projectdiscovery.io/templating-guide/#workflows), complex manual steps and repetitive assessment can be easily automated with Nuclei.
@@ -272,13 +310,17 @@ We have [a discussion thread around this](https://github.com/projectdiscovery/nu
 
 ### Resources
 
+- [Finding bugs with Nuclei with PinkDraconian (Robbe Van Roey)](https://www.youtube.com/watch?v=ewP0xVPW-Pk) by **[@PinkDraconian](https://twitter.com/PinkDraconian)** 
+- [Nuclei: Packing a Punch with Vulnerability Scanning](https://bishopfox.com/blog/nuclei-vulnerability-scan) by **Bishopfox**
+- [The WAF efficacy framework](https://www.fastly.com/blog/the-waf-efficacy-framework-measuring-the-effectiveness-of-your-waf) by **Fastly**
+- [Scanning Live Web Applications with Nuclei in CI/CD Pipeline](https://blog.escape.tech/devsecops-part-iii-scanning-live-web-applications/) by **[@TristanKalos](https://twitter.com/TristanKalos)**
 - [Community Powered Scanning with Nuclei](https://blog.projectdiscovery.io/community-powered-scanning-with-nuclei/)
 - [Nuclei Unleashed - Quickly write complex exploits](https://blog.projectdiscovery.io/nuclei-unleashed-quickly-write-complex-exploits/)
 - [Nuclei - Fuzz all the things](https://blog.projectdiscovery.io/nuclei-fuzz-all-the-things/)
 - [Nuclei + Interactsh Integration for Automating OOB Testing](https://blog.projectdiscovery.io/nuclei-interactsh-integration/)
-- [Weaponizes nuclei Workflows to Pwn All the Things](https://medium.com/@dwisiswant0/weaponizes-nuclei-workflows-to-pwn-all-the-things-cd01223feb77) by [@dwisiswant0](https://github.com/dwisiswant0)
-- [How to Scan Continuously with Nuclei?](https://medium.com/@dwisiswant0/how-to-scan-continuously-with-nuclei-fcb7e9d8b8b9) by [@dwisiswant0](https://github.com/dwisiswant0)
-- [Hack with Automation !!!](https://dhiyaneshgeek.github.io/web/security/2021/07/19/hack-with-automation/) by [@DhiyaneshGeek](https://github.com/DhiyaneshGeek)
+- [Weaponizes nuclei Workflows to Pwn All the Things](https://medium.com/@dwisiswant0/weaponizes-nuclei-workflows-to-pwn-all-the-things-cd01223feb77) by **[@dwisiswant0](https://github.com/dwisiswant0)**
+- [How to Scan Continuously with Nuclei?](https://medium.com/@dwisiswant0/how-to-scan-continuously-with-nuclei-fcb7e9d8b8b9) by **[@dwisiswant0](https://github.com/dwisiswant0)**
+- [Hack with Automation !!!](https://dhiyaneshgeek.github.io/web/security/2021/07/19/hack-with-automation/) by **[@DhiyaneshGeek](https://github.com/DhiyaneshGeek)**
 
 ### Credits
 

README_CN.md

@@ -1,253 +1,303 @@
 <h1 align="center">
-  <img src="static/nuclei-logo.png" alt="nuclei" width="200px"></a>
   <br>
+  <a href="https://nuclei.projectdiscovery.io"><img src="static/nuclei-logo.png" width="200px" alt="Nuclei"></a>
 </h1>
 
-[![License](https://img.shields.io/badge/license-MIT-_red.svg)](https://opensource.org/licenses/MIT)
-[![Go Report Card](https://goreportcard.com/badge/github.com/projectdiscovery/nuclei)](https://goreportcard.com/report/github.com/projectdiscovery/nuclei)
-[![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/projectdiscovery/nuclei/issues)
-[![GitHub Release](https://img.shields.io/github/release/projectdiscovery/nuclei)](https://github.com/projectdiscovery/nuclei/releases)
-[![Follow on Twitter](https://img.shields.io/twitter/follow/pdnuclei.svg?logo=twitter)](https://twitter.com/pdnuclei)
-[![Docker Images](https://img.shields.io/docker/pulls/projectdiscovery/nuclei.svg)](https://hub.docker.com/r/projectdiscovery/nuclei)
-[![Chat on Discord](https://img.shields.io/discord/695645237418131507.svg?logo=discord)](https://discord.gg/KECAGdH)
+<h4 align="center">基于YAML语法模板的定制化快速漏洞扫描器</h4>
+
 
 <p align="center">
-<a href="https://nuclei.projectdiscovery.io/templating-guide/" target="_blank"><img src="static/read-the-docs-button.png" height="42px"/></center></a>  <a href="https://github.com/projectdiscovery/nuclei-templates" target="_blank"><img src="static/download-templates-button.png" height="42px"/></a>
+<a href="https://goreportcard.com/report/github.com/projectdiscovery/nuclei"><img src="https://goreportcard.com/badge/github.com/projectdiscovery/nuclei"></a>
+<a href="https://github.com/projectdiscovery/nuclei/issues"><img src="https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat"></a>
+<a href="https://github.com/projectdiscovery/nuclei/releases"><img src="https://img.shields.io/github/release/projectdiscovery/nuclei"></a>
+<a href="https://twitter.com/pdnuclei"><img src="https://img.shields.io/twitter/follow/pdnuclei.svg?logo=twitter"></a>
+<a href="https://discord.gg/projectdiscovery"><img src="https://img.shields.io/discord/695645237418131507.svg?logo=discord"></a>
+<a href="https://github.com/projectdiscovery/nuclei/actions/workflows/build-test.yml"><img src="https://github.com/projectdiscovery/nuclei/actions/workflows/build-test.yml/badge.svg?branch=master"></a>
+</p>
+      
+<p align="center">
+  <a href="#工作流程">工作流程</a> •
+  <a href="#安装Nuclei">安装</a> •
+  <a href="#对于安全工程师">对于安全工程师</a> •
+  <a href="#对于开发者和组织">对于开发者</a> •
+  <a href="https://nuclei.projectdiscovery.io/nuclei/get-started/">文档</a> •
+  <a href="#c致谢">致谢</a> •
+  <a href="https://nuclei.projectdiscovery.io/faq/nuclei/">常见问题</a> •
+  <a href="https://discord.gg/projectdiscovery">加入Discord</a>
 </p>
 
+<p align="center">
+  <a href="https://github.com/projectdiscovery/nuclei/blob/master/README.md">English</a> •
+  <a href="https://github.com/projectdiscovery/nuclei/blob/master/README_CN.md">中文</a>
+</p>
 
-Nuclei是一个基于模板的、可配置攻击目标的扫描快速工具，同时还提供了强大的可扩展性和易用性。
+---
 
-基于模板的nuclei被用来发送请求给目标，有着实现零误报的优点，并且可以对已知的路径进行有效的扫描。nuclei的主要用于在初期的探测阶段快速地对已知的且易于检测的漏洞或者CVE进行扫描。如果存在WAF的话，nuclei使用[retryablehttp-go库](https://github.com/projectdiscovery/retryablehttp-go)来处理各种错误，并且重新尝试攻击，这也是我们自定义功能的核心模块之一。
+Nuclei使用零误报的定制模板向目标发送请求，同时可以对大量主机进行快速扫描。Nuclei提供TCP、DNS、HTTP、FILE等各类协议的扫描，通过强大且灵活的模板，可以使用Nuclei模拟各种安全检查。
 
-我们也维护一个具有各个类型的模板的[开源库](https://github.com/projectdiscovery/nuclei-templates)，我们希望你也能贡献一些模板，贡献的这些模板最好是有效的，并且能允许每个人基于你的模板重新构建。查看[**nuclei.projectdiscovery.io**](https://nuclei.projectdiscovery.io/templating-guide/)这个网站去学习制作模板的入门知识。
+我们的[模板仓库](https://github.com/projectdiscovery/nuclei-templates)包含**超过200**安全研究员和工程师提供的模板。
 
-## 目录
 
-- [目录](#目录)
-- [功能](#功能)
-- [安装](#安装)
-- [Nuclei模板](#nuclei模板)
-- [用法](#用法)
-- [运行Nuclei](#运行nuclei)
-- [排除模板](#排除模板)
-- [致谢](#致谢)
 
-## 功能
+## 工作流程
 
-<h1 align="left">
-  <img src="static/nuclei-run.png" alt="nuclei" width="700px"></a>
-  <br>
-</h1>
 
-- 有着易于开发的、简单的、模块化的代码库
-- 使用了基于模板的引擎，运行速度极快，可以修改所以配置
-- 可以对特殊情况处理、重试、绕过等，可以绕过WAF
-- 智能匹配，零误报
+<h3 align="center">
+  <img src="static/nuclei-flow.jpg" alt="nuclei-flow" width="700px"></a>
+</h3>
 
-## 安装
 
-### 二进制文件安装
+# 安装Nuclei
 
-二进制文件安装很简单，你可以从[Releases](https://github.com/projectdiscovery/nuclei/releases/)页面下载已经构建好的二进制文件压缩包，使用解压工具提取下载的压缩包，并将解压的文件夹移动到$PATH目录，就可以直接使用了。
+Nuclei需要**go1.17**才能安装成功。执行下列命令安装最新版本的Nuclei
 
 ```sh
-Download latest binary from https://github.com/projectdiscovery/nuclei/releases
-
-▶ tar -xzvf nuclei-linux-amd64.tar.gz
-▶ mv nuclei /usr/local/bin/
-▶ nuclei -version
+go install -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei@latest
 ```
 
-### 源码安装
+**更多的安装方式 [请点击此处](https://nuclei.projectdiscovery.io/nuclei/get-started/).**
 
-nuclei需要**go1.14+**才能成功安装，运行以下命令获取repo
+<table>
+<tr>
+<td>  
 
-```sh
-▶ GO111MODULE=on go get -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei
-```
+### Nuclei模板
 
-### GitHub安装
+自从[v2.5.2]((https://github.com/projectdiscovery/nuclei/releases/tag/v2.5.2))起，Nuclei就内置了自动下载和更新模板的功能。[**Nuclei模板**](https://github.com/projectdiscovery/nuclei-templates)仓库随时更新社区中可用的模板列表。
 
-```sh
-▶ git clone https://github.com/projectdiscovery/nuclei.git; cd nuclei/v2/cmd/nuclei/; go build; mv nuclei /usr/local/bin/; nuclei -version
-```
+您仍然可以随时使用`update-templates`命令更新模板，您可以根据[模板指南](https://nuclei.projectdiscovery.io/templating-guide/)编写您自己的模板。
 
-## Nuclei模板
+YAML的语法规范在[这里](SYNTAX-REFERENCE.md)。
 
-你可以使用`update-templates`来下载和更新nuclei模板，该命令会从unclei的[模板库]()中下载最新版本，这个由社区来维护的库是可以随时使用的。
+</td>
+</tr>
+</table>
 
-```sh
-▶ nuclei -update-templates
-```
-
-此外您可以根据自己的工作情况或者需求编写模板，请参阅**nuclei[模板向导](https://nuclei.projectdiscovery.io/templating-guide/)去编写自定义模板**
-
-## 用法
+### 用法
 
 ```sh
 nuclei -h
 ```
 
-这条命令会显示帮助，以下是nuclei支持的所有命令
+这将显示Nuclei的帮助，以下是所有支持的命令
 
-|命令|描述|例子|
-|-----|-----|-----|
-|bulk-size|每个模板最大并行的主机数(默认25)|nuclei -bulk-size 25|
-|burp-collaborator-biid|使用burp-collaborator插件|nuclei -burp-collaborator-biid XXXX|
-|c|并行的最大模板数量(默认10)|nuclei -c 10|
-|l|对URL列表进行测试|nuclei -l urls.txt|
-|target|对目标进行测试|nuclei -target hxxps://example.com -target hxxps://example2.com|
-|t|要检测的模板种类|nuclei -t git-core.yaml -t cves/|
-|no-color|输出不显示颜色|nuclei -no-color|
-|no-meta|不显示匹配的元数据|nuclei -no-meta|
-|json|输出为json格式|nuclei -json|
-|include-rr|json输出格式中包含请求和响应数据|nuclei -json -include-rr|
-|o|输出为文件|nuclei -o output.txt|
-|project|避免发送相同的请求|nuclei -project|
-|stats|使用进度条|nuclei -stats|
-|silent|只输出测试成功的结果|nuclei -silent|
-|retries|失败后的重试次数|nuclei -retries 1|
-|timeout|超时时间(默认为5秒)|nuclei -timeout 5|
-|trace-log|输出日志到log文件|nuclei -trace-log logs|
-|rate-limit|每秒最大请求数(默认150)|nuclei -rate-limit 150|
-|severity|根据严重性选择模板|nuclei  -severity critical,high|
-|stop-at-first-match|第一次匹配不要处理HTTP请求|nuclei -stop-at-frst-match|
-|exclude|排除的模板或文件夹|nuclei -exclude panels -exclude tokens|
-|debug|调试请求或者响应|nuclei -debug|
-|update-templates|下载或者升级模板|nuclei -update-templates|
-|update-directory|选择储存模板的目录(可选)|nuclei -update-directory templates|
-|tl|列出可用的模板|nuclei -tl|
-|templates-version|显示已安装的模板版本|nuclei -templates-version|
-|v|显示发送请求的详细信息|nuclei -v|
-|version|显示nuclei的版本号|nuclei -version|
-|proxy-url|输入代理地址|nuclei -proxy-url hxxp://127.0.0.1:8080|
-|proxy-socks-url|输入socks代理地址|nuclei -proxy-socks-url socks5://127.0.0.1:8080|
-|random-agent|使用随机的UA|nuclei -random-agent|
-|H|自定义请求头|nuclei -H “x-bug-bounty:hacker”|
 
-## 运行Nuclei
+```yaml
+Nuclei是一款注重于可配置性、可扩展性和易用性的基于模板的快速漏洞扫描器。
 
-### 运行单个模板
+用法：
+  nuclei [命令]
 
-这将对`urls.txt`中所有的主机运行`git-core.yaml`并返回结果到`results.txt`
+命令：
+目标：
+   -u, -target string[]                 指定扫描的URL/主机
+   -l, -list string                     指定需要扫描的URL/主机文件（一行一个）
+
+模板：
+   -t, -templates string[]              指定需要扫描的模板或者模板的路径
+   -nt, -new-templates                  只扫描最新版本中添加的模板
+   -w, -workflows string[]              指定扫描中的工作流或者工作流目录
+   -validate                            验证通过的模板
+   -tl                                  列出所有可用的模板
+
+过滤：
+   -tags string[]                       执行有标记的模板子集
+   -etags, -exclude-tags string[]       执行标记为排除的模板
+   -itags, -include-tags string[]       不执行具有攻击性的模板
+   -et, -exclude-templates string[]     要排除的模板或者模板目录
+   -it, -include-templates string[]     执行默认或配置中排除的模板
+   -s, -severity value[]                根据严重程度运行模板，可候选的值有：info,low,medium,high,critical   
+   -es, -exclude-severity value[]       根据严重程度排除模板，可候选的值有：info,low,medium,high,critical   
+   -a, -author string[]                 执行指定作者的模板
+
+输出：
+   -o, -output string                   输出发现的问题到文件
+   -silent                              只显示结果
+   -nc, -no-color                       禁用输出内容着色（ANSI转义码）
+   -json                                输出为jsonL（ines）
+   -irr, -include-rr                    在JSONL中输出对应的请求和相应（仅结果）
+   -nm, -no-meta                        不显示匹配的元数据
+   -nts, -no-timestamp                  不在输出中显示时间戳
+   -rdb, -report-db string              本地的Nuclei结果数据库（始终使用该数据库保存结果）       
+   -me, -markdown-export string         以markdown导出结果
+   -se, -sarif-export string            以SARIF导出结果
+
+配置：
+   -config string                       指定Nuclei的配置文件
+   -rc, -report-config string           指定Nuclei报告模板文件
+   -H, -header string[]                 指定报告中的标题：value格式
+   -V, -var value                       通过var=value指定var值
+   -r, -resolvers string                指定Nuclei的解析文件
+   -sr, -system-resolvers               当DNS错误时使用系统DNS
+   -passive                             启用被动扫描处理HTTP响应
+   -ev, env-vars                        在模板中使用环境变量
+
+交互：
+   -inserver, -ineractsh-server string  使用interactsh反连检测平台（默认为"https://interact.sh"）
+   -itoken, -interactsh-token string    指定反连检测平台的身份凭证
+   -interactions-cache-size int         指定保存在交互缓存中的请求数（默认：5000）
+   -interactions-eviction int           从缓存中删除请求前等待的时间（默认为60秒）
+   -interactions-poll-duration int      每个轮询前等待时间（默认为5秒）
+   -interactions-cooldown-period int    退出轮询前的等待时间（默认为5秒）
+   -ni, -no-interactsh                  禁用反连检测平台，同时排除基于反连检测的模板
+
+限速：
+   -r1, -rate-limit int                 每秒最大请求量（默认：150）
+   -rlm, -rate-limit-minute int         每分钟最大请求量
+   -bs, -bulk-size int                  每个模板最大并行检测数（默认：25）
+   -c, -concurrency int                 并行执行的最大模板数量（默认：25）
+
+
+优化：
+   -timeout int                         超时时间（默认为5秒）
+   -retries int                         重试次数（默认：1）
+   -mhe, -max-host-error int            某主机扫描失败次数，跳过该主机（默认：30）
+   -project                             使用项目文件夹避免多次发送同一请求
+   -project-path string                 设置特定的项目文件夹
+   -spm, -stop-at-first-path            得到一个结果后停止（或许会中断模板和工作流的逻辑）
+   -stream                              流模式 - 在不整理输入的情况下详细描述
+
+无界面浏览器：
+    -headless                           启用需要无界面浏览器的模板
+    -page-timeout int                   在无界面下超时秒数（默认：20）
+    -sb, -show-brower                   在无界面浏览器运行模板时，显示浏览器
+    -sc, -system-chrome                 不使用Nuclei自带的浏览器，使用本地浏览器
+
+调试：
+    -debug                              显示所有请求和响应
+    -debug-req                          显示所有请求
+    -debug-resp                         显示所有响应
+    -proxy, -proxy-url string           使用HTTP代理
+    -proxy-socks-url string             使用SOCK5代理
+    -tlog, -trace-log string            写入请求日志到文件
+    -version                            显示版本信息
+    -v, -verbose                        显示详细信息
+    -vv                                 显示额外的详细信息
+    -tv, -templates-version             显示已安装的模板版本
+
+升级：
+    -update                             更新Nuclei到最新版本
+    -ut, -update-templates              更新Nuclei模板到最新版
+    -ud, -update-directory string       覆盖安装模板
+    -duc, -disable-update-check         禁用更新
+
+统计：
+    -stats                              显示正在扫描的统计信息
+    -sj, -stats-json                    将统计信息以JSONL格式输出到文件
+    -si, -stats-inerval int             显示统计信息更新的间隔秒数（默认：5）
+    -m, -metrics                        显示Nuclei端口信息
+    -mp, -metrics-port int              更改Nuclei默认端口（默认：9092）
+```
+
+### 运行Nuclei
+
+使用[社区提供的模板](https://github.com/projectdiscovery/nuclei-templates)扫描单个目标
 
 ```sh
-▶ nuclei -l urls.txt -t files/git-core.yaml -o results.txt
+nuclei -u https://example.com
 ```
 
-你可以轻松的通过管道使用标准的输入(STDIN)传递URL列表。
+使用[社区提供的模板](https://github.com/projectdiscovery/nuclei-templates)扫描多个目标
 
 ```sh
-▶ cat urls.txt | nuclei -t files/git-core.yaml -o results.txt
+nuclei -list urls.txt
 ```
 
-💡 Nuclei可以接受如下列表的URL作为输入，例如以下URL：
+Example of `urls.txt`:
 
-```
-https://test.some-site.com
-http://vuls-testing.com
-https://test.com
-```
-### 运行多个模板
-
-这将会对`urls.txt`中所有的URL运行`cves`和`files`模板检查，并返回输出到`results.txt`
-
-```sh
-▶ nuclei -l urls.txt -t cves/ -t files/ -o results.txt
+```yaml
+http://example.com
+http://app.example.com
+http://test.example.com
+http://uat.example.com
 ```
 
-### 使用subfinder运行
+**更多关于Nuclei的详细实例可以在[这里](https://nuclei.projectdiscovery.io/nuclei/get-started/#running-nuclei)找到**
 
-```sh
-▶ subfinder -d hackerone.com -silent | httpx -silent | nuclei -t cves/ -o results.txt
-```
+# 对于安全工程师
 
-### 在docker中运行
+Nuclei提供了大量有助于安全工程师在工作流定制相关的功能。通过各种扫描功能（如DNS、HTTP、TCP），安全工程师可以更轻松的使用Nuclei创建一套自定义的检查方式。
 
-你需要使用[nuclei的docker镜像](https://hub.docker.com/r/projectdiscovery/nuclei)来运行
+- 支持多种协议：TCP、DNS、HTTP、FILE等
+- 通过工作流和动态请求实现复杂的漏洞扫描
+- 易于集成到CI/CD，旨在可以轻松的集成到周期扫描中，以主动检测漏洞的修复和重新出现
 
-```sh
-▶ docker pull projectdiscovery/nuclei
-```
+<h1 align="left">
+  <a href="https://nuclei.projectdiscovery.io/nuclei/get-started/"><img src="static/learn-more-button.png" width="170px" alt="Learn More"></a>
+</h1>
 
-下载并构建完成后，运行以下命令：
+<table>
+<tr>
+<td>  
 
-```sh
-▶ docker run -it projectdiscovery/nuclei
-```
+**对于赏金猎人：**
 
-这将会对`urls.txt`中的URL通过docker中的nuclei进行检测，并将结果输出到本机的`results.txt`文件的：
+Nuclei允许您定制自己的测试方法，可以轻松的运行您的程序。此外Nuclei可以更容易的集成到您的漏扫设备中。
 
-```sh
-▶ cat urls.txt | docker run -v /path/to/nuclei-templates:/app/nuclei-templates -v /path/to/nuclei/config:/app/.nuclei-config.json -i projectdiscovery/nuclei -t /app/nuclei-templates/files/git-config.yaml > results.txt
-```
-记住更改的模板路径到本机
+- 可以集成到其他工作流中
+- 可以在几分钟处理上千台主机
+- 使用YAML语法定制自动化测试
 
-### 速率限制
+欢迎查看我们其他的开源项目，可能有适合您的赏金猎人工作流：[github.com/projectdiscovery](http://github.com/projectdiscovery)，我们还使用[Chaos绘制了每日的DNS数据](http://chaos.projectdiscovery.io)。
 
-Nuclei有多种控制速率的方法，包括并行执行多个模板、并行检查多个主机，以及使nuclei限制全局的请求速率，下面就是示例。
+</td>
+</tr>
+</table>
 
-- `-c`参数 => 限制并行的模板数
-- `-bulk-size`参数 => 限制并行的主机数
-- `-rate-limit`参数 => 全局速率限制
+<table>
+<tr>
+<td>
 
-如果你想快速扫描或者控制扫描，请使用这些标志并输入限制数，`速率限制`只保证控制传出的请求，与其他参数无关。
+**对于渗透测试：**
 
-### 排除模板
+Nuclei通过增加手动、自动的过程，极大地改变了安全评估的方式。一些公司已经在用Nuclei升级他们的手动测试步骤，可以使用Nulcei对数千台主机使用同样的流程自动化测试。
 
-[Nuclei模板](https://github.com/projectdiscovery/nuclei-templates)包含多种检查，其中有许多对攻击有用的检查，但并不是都有用的。如果您只希望扫描少数特定的模板或目录，则可以使用如下的参数筛选模板，或将某些模板排除。
+渗透测试员可以使用公共模板或者自定义模板来更快的完成渗透测试，特别是漏洞验证时，可以轻松的验证漏洞是否修复。
 
-#### 排除模板运行
+- 轻松根据您的要求创建标准清单（例如：OWASP TOP 10）
+- 通过[FUZZ](https://nuclei.projectdiscovery.io/templating-guide/#advance-fuzzing)和[工作流](https://nuclei.projectdiscovery.io/templating-guide/#workflows)等功能，可以使用Nuclei完成复杂的手动步骤和重复性渗透测试
+- 只需要重新运行Nuclei即可验证漏洞修复情况
 
-我们不建议同时运行所有的nuclei模板，如果要排除模板，可以使用`exclude`参数来排除特定的目录或模板。
+</td>
+</tr>
+</table>
 
-```sh
-nuclei -l urls.txt -t nuclei-templates -exclude panels/ -exclude technologies -exclude files/wp-xmlrpc.yaml
-```
+# 对于开发和组织
 
-注意：如上述示例中显示的那样，目录和特定模板都将不会扫描
+Nuclei构建很简单，通过数百名安全研究员的社区模板，Nuclei可以随时扫描来了解安全威胁。Nuclei通常用来用于复测，以确定漏洞是否被修复。
 
-#### 基于严重性运行模板
+- **CI/CD：**工程师已经支持了CI/CD，可以使用Nuclei来监控生产环境
+- **周期性扫描：**使用Nuclei创建新发现的漏洞模板，通过Nuclei可以周期性扫描消除漏洞
 
-您可以根据模板的严重性运行模板，扫描时可以选择单个严重性或多个严重性。
+我们有个[讨论组](https://github.com/projectdiscovery/nuclei-templates/discussions/693)，黑客提交自己的模板后可以获得赏金，这可以减少资产的漏洞，并且减少重复。如果你想实行该计划，可以[联系我](mailto:[email protected])。我们非常乐意提供帮助，或者在[讨论组](https://github.com/projectdiscovery/nuclei-templates/discussions/693)中发布相关信息。
 
-```sh
-nuclei -l urls.txt -t cves/ -severity critical,medium
-```
+<h3 align="center">
+  <img src="static/regression-with-nuclei.jpg" alt="regression-cycle-with-nuclei" width="1100px"></a>
+</h3>
 
-上面的例子将运行`cves`目录下所有`严重`和`中等`的模板。
+<h1 align="left">
+  <a href="https://github.com/projectdiscovery/nuclei-action"><img src="static/learn-more-button.png" width="170px" alt="Learn More"></a>
+</h1>
 
-```sh
-nuclei -l urls.txt -t panels/ -t technologies -severity info
-```
+### 资源
+- [使用Nuclei扫描](https://blog.projectdiscovery.io/community-powered-scanning-with-nuclei/)
+- [Nuclei Unleashed - 快速编写复杂漏洞](https://blog.projectdiscovery.io/nuclei-unleashed-quickly-write-complex-exploits/)
+- [Nuclei - FUZZ一切](https://blog.projectdiscovery.io/nuclei-fuzz-all-the-things/)
+- [Nuclei + Interactsh Integration，用于自动化OOB测试](https://blog.projectdiscovery.io/nuclei-interactsh-integration/)
+- [武器化Nuclei](https://medium.com/@dwisiswant0/weaponizes-nuclei-workflows-to-pwn-all-the-things-cd01223feb77) 作者：[@dwisiswant0](https://github.com/dwisiswant0)
+- [如何使用Nuclei连续扫描？](https://medium.com/@dwisiswant0/how-to-scan-continuously-with-nuclei-fcb7e9d8b8b9) 作者：[@dwisiswant0](https://github.com/dwisiswant0)
+- [自动化攻击](https://dhiyaneshgeek.github.io/web/security/2021/07/19/hack-with-automation/) 作者：[@DhiyaneshGeek](https://github.com/DhiyaneshGeek)
 
-上面的例子将运行`panels`和`technologies`目录下严重性标记为`info`的模板
+### 致谢
 
-#### 使用`.nuclei-ignore`文件排除模板
+感谢所有[社区贡献者提供的PR](https://github.com/projectdiscovery/nuclei/graphs/contributors)，另外您可以其他类似的开源项目：
 
-自从nuclei的[v2.1.1版本](https://github.com/projectdiscovery/nuclei/releases/tag/v2.1.1)以来，我们添加了对`.nuclei-ignore`文件的支持，该文件与`update-templates`参数一起使用，在 **.nuclei-ignore** 文件中，您可以定义要从nuclei扫描中排除的所有模板目录或者模板路径，要开始使用此功能，请确保使用`nuclei-update-templates`参数安装nuclei模板，现在可以根据`.nuclei-ignore`的文件来添加、更新、删除模板文件。
+[FFuF](https://github.com/ffuf/ffuf), [Qsfuzz](https://github.com/ameenmaali/qsfuzz), [Inception](https://github.com/proabiral/inception), [Snallygaster](https://github.com/hannob/snallygaster), [Gofingerprint](https://github.com/Static-Flow/gofingerprint), [Sn1per](https://github.com/1N3/Sn1per/tree/master/templates), [Google tsunami](https://github.com/google/tsunami-security-scanner), [Jaeles](https://github.com/jaeles-project/jaeles), [ChopChop](https://github.com/michelin/ChopChop)
 
-```
-nano ~/nuclei-templates/.nuclei-ignore
-```
+### 许可证
 
-默认的**nuclei忽略**列表可以访问[这里]((https://github.com/projectdiscovery/nuclei-templates/blob/master/.nuclei-ignore)，如果不想排除任何内容，只需要删除`.nuclei-ignore`文件。
+Nuclei使用[MIT许可证](https://github.com/projectdiscovery/nuclei/blob/master/LICENSE.md)
 
-* * *
-
-### 📋 笔记
-
-- 进度条是实验性功能，在某些情况下可能无法使用。
-- 进度条不适用于工作流，因为是条件执行，所以不准确。
-
-
-## 致谢
-
-也要看看这些类似的好项目，或许它们也适合你：
-
-[Burp Suite](https://portswigger.net/burp), [FFuF](https://github.com/ffuf/ffuf), [Jaeles](https://github.com/jaeles-project/jaeles), [Qsfuzz](https://github.com/ameenmaali/qsfuzz), [Inception](https://github.com/proabiral/inception), [Snallygaster](https://github.com/hannob/snallygaster), [Gofingerprint](https://github.com/Static-Flow/gofingerprint), [Sn1per](https://github.com/1N3/Sn1per/tree/master/templates), [Google tsunami](https://github.com/google/tsunami-security-scanner), [ChopChop](https://github.com/michelin/ChopChop)
-
---------
-
-Nuclei是由[projectdiscovery](https://projectdiscovery.io)团队用🖤制作的，当然社区也贡献了很多，通过 **[Thanks.md](https://github.com/projectdiscovery/nuclei/blob/master/THANKS.md)**文件以获取更多详细信息。
+<h1 align="left">
+  <a href="https://discord.gg/projectdiscovery"><img src="static/Join-Discord.png" width="380" alt="Join Discord"></a> <a href="https://nuclei.projectdiscovery.io"><img src="static/check-nuclei-documentation.png" width="380" alt="Check Nuclei Documentation"></a>
+</h1>

integration_tests/code/test.yaml

@@ -0,0 +1,17 @@
+id: go-integration-test
+
+info:
+  name: Basic Go Integration Test
+  author: pdteam
+  severity: info
+
+requests:
+  - method: GET
+    path:
+      - "{{BaseURL}}"
+    headers:
+      test: nuclei
+    matchers:
+      - type: word
+        words:
+          - "This is test headers matcher text"

integration_tests/dns/caa.yaml

@@ -0,0 +1,22 @@
+id: caa-fingerprinting
+
+info:
+  name: CAA Fingerprint
+  author: pdteam
+  severity: info
+  tags: dns,caa
+
+dns:
+  - name: "{{FQDN}}"
+    type: CAA
+
+    matchers:
+      - type: word
+        words:
+          - "IN\tCAA"
+
+    extractors:
+      - type: regex
+        group: 1
+        regex:
+          - "IN\tCAA\t(.+)"

integration_tests/dns/cname-fingerprint.yaml

@@ -0,0 +1,22 @@
+id: cname-fingerprint
+
+info:
+  name: CNAME Fingerprint
+  author: pdteam
+  severity: info
+  tags: dns,cname
+
+dns:
+  - name: "{{FQDN}}"
+    type: NS
+
+    matchers:
+      - type: word
+        words:
+          - "IN\tCNAME"
+
+    extractors:
+      - type: regex
+        group: 1
+        regex:
+          - "IN\tCNAME\t(.+)"

integration_tests/dns/ptr.yaml

@@ -0,0 +1,22 @@
+id: ptr-fingerprint
+
+info:
+  name: PTR Fingerprint
+  author: pdteam
+  severity: info
+  tags: dns,ptr
+
+dns:
+  - name: "{{FQDN}}"
+    type: PTR
+
+    matchers:
+      - type: word
+        words:
+          - "IN\tPTR"
+
+    extractors:
+      - type: regex
+        group: 1
+        regex:
+          - "IN\tPTR\t(.+)"

integration_tests/dns/variables.yaml

@@ -0,0 +1,20 @@
+id: variables-example
+
+info:
+  name: Variables Example
+  author: pdteam
+  severity: info
+
+variables:
+  a1: "IN"
+
+dns:
+  - name: "{{FQDN}}"
+    type: A
+    class: inet
+    recursion: true
+    retries: 3
+    matchers:
+      - type: word
+        words:
+          - "{{a1}}"

integration_tests/file/data/test1.txt

@@ -0,0 +1,2 @@
+AAA
+BBB

integration_tests/file/data/test2.txt

@@ -0,0 +1,2 @@
+CCC
+DDD

integration_tests/file/data/test3.txt

@@ -0,0 +1,2 @@
+11 EE 11
+11 FF 11

integration_tests/file/extract.yaml

@@ -0,0 +1,16 @@
+id: file-extract
+
+info:
+  name: File with Extractor
+  author: pdteam
+  severity: info
+  tags: file
+
+file:
+  - extensions:
+      - all
+
+    extractors:
+      - type: regex
+        regex:
+          - "(?m)11\\s(EE|FF)\\s11"

integration_tests/file/matcher-with-and.yaml

@@ -0,0 +1,20 @@
+id: file-matcher-with-or
+
+info:
+  name: File Matcher With AND
+  author: pdteam
+  severity: info
+  tags: file
+
+file:
+  - extensions:
+      - all
+
+    matchers-condition: and
+    matchers:
+      - type: word
+        words:
+          - "CCC"
+      - type: word
+        words:
+          - "DDD"

integration_tests/file/matcher-with-or.yaml

@@ -0,0 +1,19 @@
+id: file-matcher-with-or
+
+info:
+  name: File Matcher With OR
+  author: pdteam
+  severity: info
+  tags: file
+
+file:
+  - extensions:
+      - all
+
+    matchers:
+      - type: word
+        words:
+          - "AA"
+      - type: word
+        words:
+          - "BB"

integration_tests/headless/headless-basic.yaml

@@ -0,0 +1,18 @@
+id: headless-basic
+info:
+  name: Headless Basic
+  author: pdteam
+  severity: info
+  tags: headless
+
+headless:
+  - steps:
+      - action: navigate
+        args:
+          url: "{{BaseURL}}/"
+        
+      - action: waitload
+    matchers:
+      - type: word
+        words:
+          - "<html>"

integration_tests/headless/headless-extract-values.yaml

@@ -0,0 +1,31 @@
+
+id: headless-extract-values
+info:
+  name: Headless Extract Value
+  author: pdteam
+  severity: info
+  tags: headless
+
+headless:
+  - steps:
+      - action: navigate
+        args:
+          url: "{{BaseURL}}"        
+      - action: waitload
+      # From headless/extract-urls.yaml
+      - action: script
+        name: extract
+        args:
+          code: |
+            () => '\n' + [...new Set(Array.from(document.querySelectorAll('[src], [href], [url], [action]')).map(i => i.src || i.href || i.url || i.action))].join('\r\n') + '\n'
+
+    matchers:
+      - type: word
+        words:
+          - "test.html"
+    
+    extractors:
+      - type: kval
+        part: extract
+        kval:
+          - extract

integration_tests/headless/headless-header-action.yaml

@@ -0,0 +1,24 @@
+id: headless-header-action
+info:
+  name: Headless Header Action
+  author: pdteam
+  severity: info
+  tags: headless
+
+headless:
+  - steps:
+    - action: setheader
+      args:
+        part: request
+        key: Test
+        value: test value
+
+    - action: navigate
+      args:
+        url: "{{BaseURL}}/"
+        
+    - action: waitload
+    matchers:
+      - type: word
+        words:
+          - "test value"

integration_tests/headless/headless-payloads.yaml

@@ -0,0 +1,26 @@
+id: headless-payloads
+
+info:
+  name: headless payloads example
+  author: pdteam
+  severity: info
+  tags: headless
+
+headless:
+  - attack: clusterbomb
+    payloads:
+      aa:
+        - aa
+        - bb
+      bb:
+        - cc
+        - dd
+    steps:
+      - args:
+          url: "{{BaseURL}}?aa={{aa}}&bb={{bb}}"
+        action: navigate
+      - action: waitload
+    matchers:
+      - type: word
+        words:
+          - "test"

integration_tests/headless/variables.yaml

@@ -0,0 +1,20 @@
+id: variables-example
+
+info:
+  name: Variables Example
+  author: pdteam
+  severity: info
+
+variables:
+  a1: "{{base64('hello')}}"
+
+headless:
+  - steps:
+      - args:
+          url: "{{BaseURL}}"
+        action: navigate
+      - action: waitload
+    matchers:
+      - type: word
+        words:
+          - "{{a1}}"

integration_tests/http/disable-redirects.yaml

@@ -0,0 +1,17 @@
+id: basic-disable-redirects
+
+info:
+  name: Basic GET Redirects Request
+  author: pdteam
+  severity: info
+
+requests:
+  - method: GET
+    path:
+      - "{{BaseURL}}"
+    redirects: true
+    max-redirects: 2
+    matchers:
+      - type: word
+        words:
+          - "This is test disable redirects matcher text"

integration_tests/http/dsl-functions.yaml

@@ -0,0 +1,75 @@
+id: helper-functions-examples
+
+info:
+  name: RAW Template with Helper Functions
+  author: pdteam
+  severity: info
+
+requests:
+  - raw:
+      - |
+        GET / HTTP/1.1
+        Host: {{Hostname}}
+        01: {{base64("Hello")}}
+        02: {{base64(1234)}}
+        03: {{base64_decode("SGVsbG8=")}}
+        04: {{base64_py("Hello")}}
+        05: {{concat("Hello", "world")}}
+        06: {{contains("Hello", "lo")}}
+        07: {{generate_java_gadget("commons-collections3.1", "wget http://{{interactsh-url}}", "base64")}}
+        08: {{gzip("Hello")}}
+        09: {{hex_decode("6161")}}
+        10: {{hex_encode("aa")}}
+        11: {{html_escape("<body>test</body>")}}
+        12: {{html_unescape("&lt;body&gt;test&lt;/body&gt;")}}
+        13: {{len("Hello")}}
+        14: {{len(5555)}}
+        15: {{md5("Hello")}}
+        16: {{md5(1234)}}
+        17: {{mmh3("Hello")}}
+        18: {{print_debug(1+2, "Hello")}}
+        19: {{rand_base(5, "abc")}}
+        20: {{rand_base(5, "")}}
+        21: {{rand_base(5)}}
+        22: {{rand_char("abc")}}
+        23: {{rand_char("")}}
+        24: {{rand_char()}}
+        25: {{rand_int(1, 10)}}
+        26: {{rand_int(10)}}
+        27: {{rand_int()}}
+        28: {{rand_ip("192.168.0.0/24")}}
+        29: {{rand_ip("2002:c0a8::/24")}}
+        30: {{rand_ip("192.168.0.0/24","10.0.100.0/24")}}
+        31: {{rand_text_alpha(10, "abc")}}
+        32: {{rand_text_alpha(10, "")}}
+        33: {{rand_text_alpha(10)}}
+        34: {{rand_text_alphanumeric(10, "ab12")}}
+        35: {{rand_text_alphanumeric(10)}}
+        36: {{rand_text_numeric(10, 123)}}
+        37: {{rand_text_numeric(10)}}
+        38: {{regex("H([a-z]+)o", "Hello")}}
+        39: {{remove_bad_chars("abcd", "bc")}}
+        40: {{repeat("a", 5)}}
+        41: {{replace("Hello", "He", "Ha")}}
+        42: {{replace_regex("He123llo", "(\\d+)", "")}}
+        43: {{reverse("abc")}}
+        44: {{sha1("Hello")}}
+        45: {{sha256("Hello")}}
+        46: {{to_lower("HELLO")}}
+        47: {{to_upper("hello")}}
+        48: {{trim("aaaHelloddd", "ad")}}
+        49: {{trim_left("aaaHelloddd", "ad")}}
+        50: {{trim_prefix("aaHelloaa", "aa")}}
+        51: {{trim_right("aaaHelloddd", "ad")}}
+        52: {{trim_space("  Hello  ")}}
+        53: {{trim_suffix("aaHelloaa", "aa")}}
+        54: {{unix_time(10)}}
+        55: {{url_decode("https:%2F%2Fprojectdiscovery.io%3Ftest=1")}}
+        56: {{url_encode("https://projectdiscovery.io/test?a=1")}}
+        57: {{wait_for(1)}}
+
+    extractors:
+      - type: regex
+        name: results
+        regex:
+          - '\d+: [^\s]+'

integration_tests/http/dsl-matcher-variable.yaml

@@ -0,0 +1,23 @@
+id: dsl-matcher-variable
+
+info:
+  name: dsl-matcher-variable
+  author: pd-team
+  severity: info
+
+requests: 
+  - 
+    path: 
+      - "{{BaseURL}}"
+    payloads: 
+      VALUES: 
+        - This
+        - is
+        - test
+        - matcher
+        - text
+    matchers: 
+      - 
+        dsl: 
+          - 'contains(body,"{{VALUES}}")'
+        type: dsl

integration_tests/http/get-case-insensitive.yaml

@@ -0,0 +1,16 @@
+id: basic-get-case-insensitive
+
+info:
+  name: Basic GET Request
+  author: pdteam
+  severity: info
+
+requests:
+  - method: GET
+    path:
+      - "{{BaseURL}}"
+    matchers:
+      - type: word
+        case-insensitive: true
+        words:
+          - "ThIS is TEsT MAtcHEr TExT"

integration_tests/http/get-override-sni.yaml

@@ -0,0 +1,18 @@
+id: basic-raw-http-example
+
+info:
+  name: Test RAW GET Template
+  author: pdteam
+  severity: info
+
+requests:
+  - raw:
+      - |
+        @tls-sni:request.host
+        GET / HTTP/1.1
+        Host: test
+
+    matchers:
+      - type: word
+        words:
+          - "test-ok"

integration_tests/http/get-redirects-chain-headers.yaml

@@ -0,0 +1,23 @@
+id: basic-get-redirects-chain-headers
+
+info:
+  name: Basic GET Redirects Request With Chain header
+  author: pdteam
+  severity: info
+
+requests:
+  - method: GET
+    path:
+      - "{{BaseURL}}"
+    redirects: true
+    max-redirects: 3
+    matchers-condition: and
+    matchers:
+      - type: word
+        part: header
+        words:
+          - "TestRedirectHeaderMatch"
+
+      - type: status
+        status:
+          - 302

integration_tests/http/get-sni-unsafe.yaml

@@ -0,0 +1,18 @@
+id: basic-unsafe-get
+
+info:
+  name: Basic Unsafe GET Request with CLI SNI
+  author: pdteam
+  severity: info
+
+requests:
+  - raw:
+      - |+
+        GET / HTTP/1.1
+        Host: {{Hostname}}
+
+    unsafe: true
+    matchers:
+      - type: word
+        words:
+          - "test-ok"

integration_tests/http/get-sni.yaml

@@ -0,0 +1,15 @@
+id: basic-get
+
+info:
+  name: Basic GET Request with CLI SNI
+  author: pdteam
+  severity: info
+
+requests:
+  - method: GET
+    path:
+      - "{{BaseURL}}"
+    matchers:
+      - type: word
+        words:
+          - "test-ok"

integration_tests/http/interactsh-stop-at-first-match.yaml

@@ -0,0 +1,23 @@
+id: interactsh-stop-at-first-match-integration-test
+
+info:
+  name: Interactsh StopAtFirstMatch Integration Test
+  author: pdteam
+  severity: info
+
+requests:
+  - method: GET
+    path:
+      - "{{BaseURL}}"
+      - "{{BaseURL}}"
+      - "{{BaseURL}}"
+    headers:
+      url: 'http://{{interactsh-url}}'
+
+    stop-at-first-match: true
+
+    matchers:
+      - type: word
+        part: interactsh_protocol # Confirms the HTTP Interaction
+        words:
+          - "http"

integration_tests/http/interactsh.yaml

@@ -0,0 +1,19 @@
+id: interactsh-integration-test
+
+info:
+  name: Interactsh Integration Test
+  author: pdteam
+  severity: info
+
+requests:
+  - method: GET
+    path:
+      - "{{BaseURL}}"
+    headers:
+      url: 'http://{{interactsh-url}}'
+
+    matchers:
+      - type: word
+        part: interactsh_protocol # Confirms the HTTP Interaction
+        words:
+          - "http"

integration_tests/http/race-multiple.yaml

@@ -0,0 +1,46 @@
+id: race-condition-testing
+
+info:
+  name: Race condition testing with multiple requests
+  author: pdteam
+  severity: info
+
+requests:
+  - raw:  
+      - |
+        GET / HTTP/1.1
+        Host: {{Hostname}}
+
+        id=1
+
+      - |
+        GET / HTTP/1.1
+        Host: {{Hostname}}
+
+        id=2
+
+      - |
+        GET / HTTP/1.1
+        Host: {{Hostname}}
+
+        id=3
+
+      - |
+        GET / HTTP/1.1
+        Host: {{Hostname}}
+
+        id=4
+
+      - |
+        GET / HTTP/1.1
+        Host: {{Hostname}}
+
+        id=5
+
+    threads: 5
+    race: true
+
+    matchers:
+      - type: status
+        status:
+          - 200

integration_tests/http/race-simple.yaml

@@ -0,0 +1,23 @@
+id: race-condition-testing
+
+info:
+  name: Race Condition testing
+  author: pdteam
+  severity: info
+
+requests:
+  - raw:
+      - |
+        GET / HTTP/1.1
+        Host: {{Hostname}}
+
+        test      
+
+    race: true
+    race_count: 10
+
+    matchers:
+      - type: status
+        part: header
+        status:
+          - 200

integration_tests/http/raw-unsafe-request.yaml

@@ -7,7 +7,7 @@ info:
 
 requests:
   - raw:
-      - |
+      - |+
         GET / HTTP/1.1
         Host: 
         Content-Length: 4

integration_tests/http/redirect-match-url.yaml

@@ -0,0 +1,18 @@
+id: redirect-match-url
+
+info:
+  name: Redirect Match URL
+  author: pdteam
+  severity: info
+
+requests:
+  - method: GET
+    path:
+      - "{{BaseURL}}"
+    stop-at-first-match: true # Confirm stop-at-first-match
+    redirects: true # Confirm redirected URL matched value
+    max-redirects: 3
+    matchers:
+      - type: word
+        words:
+          - "This is test redirects matcher text"

integration_tests/http/self-contained.yaml

@@ -0,0 +1,18 @@
+id: example-self-contained-input
+
+info:
+  name: example-self-contained
+  author: pd-team
+  severity: info
+
+self-contained: true
+requests:
+  - raw:
+      - |
+        GET http://127.0.0.1:5431/ HTTP/1.1
+        Host: {{Hostname}}
+
+    matchers:
+      - type: word
+        words:
+          - This is self-contained response

integration_tests/http/stop-at-first-match-with-extractors.yaml

@@ -0,0 +1,18 @@
+id: stop-at-first-match-with-extractors
+
+info:
+  name: Stop at first match Request with extractors
+  author: pdteam
+  severity: info
+
+requests:
+  - method: GET
+    path:
+      - "{{BaseURL}}?a=1"
+      - "{{BaseURL}}?a=2"
+    stop-at-first-match: true
+    extractors:
+      - type: kval
+        part: header
+        kval:
+          - "date"

integration_tests/http/stop-at-first-match.yaml

@@ -0,0 +1,17 @@
+id: stop-at-first-match
+
+info:
+  name: Stop at first match Request
+  author: pdteam
+  severity: info
+
+requests:
+  - method: GET
+    path:
+      - "{{BaseURL}}?a=1"
+      - "{{BaseURL}}?a=2"
+    matchers:
+      - type: word
+        words:
+          - "This is test"
+    stop-at-first-match: true

integration_tests/http/variables.yaml

@@ -0,0 +1,26 @@
+id: variables-example
+
+info:
+  name: Variables Example
+  author: pdteam
+  severity: info
+
+variables:
+  a1: "value"
+  a2: "{{base64('hello')}}"
+
+requests:
+  - raw:
+      - |
+        GET / HTTP/1.1
+        Host: {{FQDN}}
+        Test: {{a1}}
+        Another: {{a2}}
+
+    stop-at-first-match: true
+    matchers-condition: or
+    matchers:
+      - type: word
+        words: 
+          - "value"
+          - "aGVsbG8="

integration_tests/loader/basic.yaml

@@ -0,0 +1,10 @@
+id: workflow-example
+
+info:
+  name: Test Workflow Template
+  author: pdteam
+  severity: info
+
+workflows:
+  - template: workflow/match-1.yaml
+  - template: workflow/match-2.yaml

integration_tests/loader/condition-matched.yaml

@@ -0,0 +1,11 @@
+id: condition-matched-workflow
+
+info:
+  name: Condition Matched Workflow
+  author: pdteam
+  severity: info
+
+workflows:
+  - template: workflow/match-1.yaml
+    subtemplates:
+      - template: workflow/match-2.yaml

integration_tests/loader/get-headers.yaml

@@ -0,0 +1,17 @@
+id: basic-get-headers
+
+info:
+  name: Basic GET Headers Request
+  author: pdteam
+  severity: info
+
+requests:
+  - method: GET
+    path:
+      - "{{BaseURL}}"
+    headers:
+      test: nuclei
+    matchers:
+      - type: word
+        words:
+          - "This is test headers matcher text"

integration_tests/loader/get.yaml

@@ -0,0 +1,15 @@
+id: basic-get
+
+info:
+  name: Basic GET Request
+  author: pdteam
+  severity: info
+
+requests:
+  - method: GET
+    path:
+      - "{{BaseURL}}"
+    matchers:
+      - type: word
+        words:
+          - "This is test matcher text"

integration_tests/loader/template-list.yaml

@@ -0,0 +1,2 @@
+loader/get.yaml
+loader/get-headers.yaml

integration_tests/loader/workflow-list.yaml

@@ -0,0 +1,2 @@
+loader/basic.yaml
+loader/condition-matched.yaml

integration_tests/network/self-contained.yaml

@@ -0,0 +1,16 @@
+id: example-self-contained-input
+
+info:
+  name: example-self-contained
+  author: pd-team
+  severity: info
+
+self-contained: true
+network:
+  - host:
+      - "127.0.0.1:5431"
+
+    matchers:
+      - type: word
+        words:
+          - "Authentication successful"

integration_tests/network/variables.yaml

@@ -0,0 +1,22 @@
+id: variables-example
+
+info:
+  name: Variables Example
+  author: pdteam
+  severity: info
+
+variables:
+  a1: "PING"
+  a2: "{{base64('hello')}}"
+
+network:
+  - host: 
+      - "{{Hostname}}"
+    inputs:
+      - data: "{{a1}}"
+    read-size: 8
+    matchers:
+      - type: word
+        part: data
+        words:
+          - "{{a2}}"

integration_tests/run.sh

@@ -1,13 +1,23 @@
 #!/bin/bash
 
+echo "::group::Build nuclei"
 rm integration-test nuclei 2>/dev/null
 cd ../v2/cmd/nuclei
 go build
 mv nuclei ../../../integration_tests/nuclei 
+echo "::endgroup::"
+
+echo "::group::Build nuclei integration-test"
 cd ../integration-test
 go build
 mv integration-test ../../../integration_tests/integration-test 
 cd ../../../integration_tests
+echo "::endgroup::"
+
+echo "::group::Installing nuclei templates"
+./nuclei -update-templates
+echo "::endgroup::"
+
 ./integration-test
 if [ $? -eq 0 ]
 then

integration_tests/ssl/basic-ztls.yaml

@@ -0,0 +1,15 @@
+id: basic-ssl-tls
+
+info:
+  name:  Basic SSL Request with ztls
+  author: pdteam
+  severity: info
+
+ssl:
+  - address: "{{Host}}:{{Port}}"
+    min_version: sslv3
+    max_version: tls12
+    matchers:
+      - type: dsl
+        dsl:
+          - "not_after>=0"

integration_tests/ssl/basic.yaml

@@ -0,0 +1,13 @@
+id: expired-ssl
+
+info:
+  name: Basic SSL Request
+  author: pdteam
+  severity: info
+
+ssl:
+  - address: "{{Host}}:{{Port}}"
+    matchers:
+      - type: dsl
+        dsl:
+          - "not_after>=0"

integration_tests/ssl/custom-cipher.yaml

@@ -0,0 +1,16 @@
+id: expired-ssl
+
+info:
+  name: Basic SSL Request
+  author: pdteam
+  severity: info
+
+ssl:
+  - address: "{{Host}}:{{Port}}"
+    cipher_suites: 
+      - TLS_AES_128_GCM_SHA256
+    matchers:
+      - type: word
+        part: response
+        words:
+          - "TLS_AES_128_GCM_SHA256"

integration_tests/ssl/custom-version.yaml

@@ -0,0 +1,16 @@
+id: expired-ssl
+
+info:
+  name: Basic SSL Request
+  author: pdteam
+  severity: info
+
+ssl:
+  - address: "{{Host}}:{{Port}}"
+    min_version: tls12
+    max_version: tls12
+    matchers:
+      - type: word
+        part: response
+        words:
+          - 'TLS12'

integration_tests/test-issue-tracker-config1.yaml

@@ -3,35 +3,35 @@ allow-list:
 deny-list:
   severity: low
 
-# github contains configuration options for github issue tracker
+# GitHub contains configuration options for GitHub issue tracker
 github:
-  # base-url is the optional self-hosted github application url
+  # base-url is the optional self-hosted GitHub application url
   base-url: https://localhost:8443/github
-  # username is the username of the github user
+  # username is the username of the GitHub user
   username: test-username
-  # owner is the owner name of the repository for issues.
+  # owner is the owner name of the repository for issues
   owner: test-owner
-  # token is the token for github account.
+  # token is the token for GitHub account
   token: test-token
-  # project-name is the name of the repository.
+  # project-name is the name of the repository
   project-name: test-project
   # issue-label is the label of the created issue type
   issue-label: bug
 
-# gitlab contains configuration options for gitlab issue tracker
+# GitLab contains configuration options for gitlab issue tracker
 gitlab:
-  # base-url is the optional self-hosted gitlab application url
+  # base-url is the optional self-hosted GitLab application url
   base-url: https://localhost:8443/gitlab
-  # username is the username of the gitlab user
+  # username is the username of the GitLab user
   username: test-username
-  # token is the token for gitlab account.
+  # token is the token for GitLab account
   token: test-token
-  # project-id is the ID of the repository.
-  project-id: 1234
+  # project-name is the name/id of the project(repository)
+  project-name: "1234"
   # issue-label is the label of the created issue type
   issue-label: bug
 
-# jira contains configuration options for jira issue tracker
+# Jira contains configuration options for Jira issue tracker
 jira:
   # cloud is the boolean which tells if Jira instance is running in the cloud or on-prem version is used
   cloud: true
@@ -39,11 +39,11 @@ jira:
   update-existing: false
   # URL is the jira application url
   url: https://localhost/jira
-  # account-id is the account-id of the jira user or username in case of on-prem Jira
+  # account-id is the account-id of the Jira user or username in case of on-prem Jira
   account-id: test-account-id
-  # email is the email of the user for jira instance
+  # email is the email of the user for Jira instance
   email: [email protected]
-  # token is the token for jira instance or password in case of on-prem Jira
+  # token is the token for Jira instance or password in case of on-prem Jira
   token: test-token
   # project-name is the name of the project.
   project-name: test-project-name

integration_tests/test-issue-tracker-config2.yaml

@@ -5,47 +5,47 @@ allow-list:
 deny-list:
   severity: low
 
-# github contains configuration options for github issue tracker
-github:
-  # base-url is the optional self-hosted github application url
-  base-url: https://localhost:8443/github
-  # username is the username of the github user
+# GitHub contains configuration options for GitHub issue tracker
+GitHub:
+  # base-url is the optional self-hosted GitHub application url
+  base-url: https://localhost:8443/GitHub
+  # username is the username of the GitHub user
   username: test-username
   # owner is the owner name of the repository for issues.
   owner: test-owner
-  # token is the token for github account.
+  # token is the token for GitHub account.
   token: test-token
   # project-name is the name of the repository.
   project-name: test-project
   # issue-label is the label of the created issue type
   issue-label: bug
 
-# gitlab contains configuration options for gitlab issue tracker
-gitlab:
-  # base-url is the optional self-hosted gitlab application url
-  base-url: https://localhost:8443/gitlab
-  # username is the username of the gitlab user
+# GitLab contains configuration options for GitLab issue tracker
+GitLab:
+  # base-url is the optional self-hosted GitLab application url
+  base-url: https://localhost:8443/GitLab
+  # username is the username of the GitLab user
   username: test-username
-  # token is the token for gitlab account.
+  # token is the token for GitLab account.
   token: test-token
-  # project-id is the ID of the repository.
-  project-id: 1234
+  # project-name is the name/id of the project(repository).
+  project-name: "1234"
   # issue-label is the label of the created issue type
   issue-label: bug
 
-# jira contains configuration options for jira issue tracker
-jira:
+# Jira contains configuration options for Jira issue tracker
+Jira:
   # cloud is the boolean which tells if Jira instance is running in the cloud or on-prem version is used
   cloud: true
   # update-existing is the boolean which tells if the existing, opened issue should be updated or new one should be created
   update-existing: false
-  # URL is the jira application url
-  url: https://localhost/jira
-  # account-id is the account-id of the jira user or username in case of on-prem Jira
+  # URL is the Jira application url
+  url: https://localhost/Jira
+  # account-id is the account-id of the Jira user or username in case of on-prem Jira
   account-id: test-account-id
-  # email is the email of the user for jira instance
+  # email is the email of the user for Jira instance
   email: [email protected]
-  # token is the token for jira instance or password in case of on-prem Jira
+  # token is the token for Jira instance or password in case of on-prem Jira
   token: test-token
   # project-name is the name of the project.
   project-name: test-project-name

integration_tests/websocket/basic.yaml

@@ -0,0 +1,16 @@
+id: basic-request
+
+info:
+  name: Basic Request
+  author: pdteam
+  severity: info
+
+websocket:
+  - address: '{{Scheme}}://{{Hostname}}'
+    inputs:
+      - data: hello
+    matchers:
+      - type: word
+        words:
+          - world
+        part: response

integration_tests/websocket/cswsh.yaml

@@ -0,0 +1,16 @@
+id: basic-cswsh-request
+
+info:
+  name: Basic cswsh Request
+  author: pdteam
+  severity: info
+
+websocket:
+  - address: '{{Scheme}}://{{Hostname}}'
+    headers: 
+      Origin: 'http://evil.com'
+    matchers:
+      - type: word
+        words:
+          - true
+        part: success

integration_tests/websocket/no-cswsh.yaml

@@ -0,0 +1,16 @@
+id: basic-nocswsh-request
+
+info:
+  name: Basic Non-Vulnerable cswsh Request
+  author: pdteam
+  severity: info
+
+websocket:
+  - address: '{{Scheme}}://{{Hostname}}'
+    headers: 
+      Origin: 'http://evil.com'
+    matchers:
+      - type: word
+        words:
+          - true
+        part: success

integration_tests/websocket/path.yaml

@@ -0,0 +1,16 @@
+id: basic-request-path
+
+info:
+  name: Basic Request Path
+  author: pdteam
+  severity: info
+
+websocket:
+  - address: '{{Scheme}}://{{Hostname}}'
+    inputs:
+      - data: hello
+    matchers:
+      - type: word
+        words:
+          - world
+        part: response

integration_tests/whois/basic.yaml

@@ -0,0 +1,14 @@
+id: basic-whois-example
+
+info:
+  name: test template for WHOIS
+  author: pdteam
+  severity: info
+
+whois:
+  - query: "{{Host}}"
+    extractors:
+      - type: kval
+        kval:
+          - "expiration date"
+          - "registrar"

nuclei-jsonschema.json

@@ -72,7 +72,7 @@
         "metadata": {
           "patternProperties": {
             ".*": {
-              "type": "string"
+              "additionalProperties": true
             }
           },
           "type": "object",
@@ -103,7 +103,8 @@
         "low",
         "medium",
         "high",
-        "critical"
+        "critical",
+        "unknown"
       ],
       "type": "string",
       "title": "severity of the template",
@@ -119,9 +120,20 @@
         }
       ]
     },
+    "userAgent.UserAgentHolder": {
+      "enum": [
+        "off",
+        "default",
+        "custom"
+      ],
+      "type": "string",
+      "title": "userAgent for the headless",
+      "description": "userAgent for the headless http request"
+    },
     "extractors.Extractor": {
       "required": [
-        "type"
+        "type",
+        "DSL"
       ],
       "properties": {
         "name": {
@@ -130,15 +142,8 @@
           "description": "Name of the extractor"
         },
         "type": {
-          "enum": [
-            "regex",
-            "kval",
-            "json",
-            "xpath"
-          ],
-          "type": "string",
-          "title": "type of the extractor",
-          "description": "Type of the extractor"
+          "$schema": "http://json-schema.org/draft-04/schema#",
+          "$ref": "#/definitions/extractors.ExtractorTypeHolder"
         },
         "regex": {
           "items": {
@@ -182,6 +187,12 @@
           "title": "optional attribute to extract from xpath",
           "description": "Optional attribute to extract from response XPath"
         },
+        "DSL": {
+          "items": {
+            "type": "string"
+          },
+          "type": "array"
+        },
         "part": {
           "type": "string",
           "title": "part of response to extract data from",
@@ -191,26 +202,36 @@
           "type": "boolean",
           "title": "mark extracted value for internal variable use",
           "description": "Internal when set to true will allow using the value extracted in the next request for some protocols"
+        },
+        "case-insensitive": {
+          "type": "boolean",
+          "title": "use case insensitive extract",
+          "description": "use case insensitive extract"
         }
       },
       "additionalProperties": false,
       "type": "object"
     },
+    "extractors.ExtractorTypeHolder": {
+      "enum": [
+        "regex",
+        "kval",
+        "xpath",
+        "json",
+        "dsl"
+      ],
+      "type": "string",
+      "title": "type of the extractor",
+      "description": "Type of the extractor"
+    },
     "matchers.Matcher": {
       "required": [
         "type"
       ],
       "properties": {
         "type": {
-          "enum": [
-            "status",
-            "size",
-            "word",
-            "regex",
-            "binary",
-            "dsl"
-          ],
-          "type": "string",
+          "$schema": "http://json-schema.org/draft-04/schema#",
+          "$ref": "#/definitions/matchers.MatcherTypeHolder",
           "title": "type of matcher",
           "description": "Type of the matcher"
         },
@@ -293,11 +314,66 @@
           "type": "string",
           "title": "encoding for word field",
           "description": "Optional encoding for the word fields"
+        },
+        "case-insensitive": {
+          "type": "boolean",
+          "title": "use case insensitive match",
+          "description": "use case insensitive match"
+        },
+        "match-all": {
+          "type": "boolean",
+          "title": "match all values",
+          "description": "match all matcher values ignoring condition"
         }
       },
       "additionalProperties": false,
       "type": "object"
     },
+    "matchers.MatcherTypeHolder": {
+      "enum": [
+        "word",
+        "regex",
+        "binary",
+        "status",
+        "size",
+        "dsl"
+      ],
+      "type": "string",
+      "title": "type of the matcher",
+      "description": "Type of the matcher"
+    },
+    "generators.AttackTypeHolder": {
+      "enum": [
+        "batteringram",
+        "pitchfork",
+        "clusterbomb"
+      ],
+      "type": "string",
+      "title": "type of the attack",
+      "description": "Type of the attack"
+    },
+    "variables.Variable": {
+      "properties": {},
+      "additionalProperties": false,
+      "type": "object"
+    },
+    "dns.DNSRequestTypeHolder": {
+      "enum": [
+        "A",
+        "NS",
+        "DS",
+        "CNAME",
+        "SOA",
+        "PTR",
+        "MX",
+        "TXT",
+        "AAAA",
+        "CAA"
+      ],
+      "type": "string",
+      "title": "type of DNS request to make",
+      "description": "Type is the type of DNS request to make"
+    },
     "dns.Request": {
       "properties": {
         "matchers": {
@@ -336,18 +412,8 @@
           "description": "Name is the Hostname to make DNS request for"
         },
         "type": {
-          "enum": [
-            "A",
-            "NS",
-            "DS",
-            "CNAME",
-            "SOA",
-            "PTR",
-            "MX",
-            "TXT",
-            "AAAA"
-          ],
-          "type": "string",
+          "$schema": "http://json-schema.org/draft-04/schema#",
+          "$ref": "#/definitions/dns.DNSRequestTypeHolder",
           "title": "type of dns request to make",
           "description": "Type is the type of DNS request to make"
         },
@@ -369,16 +435,38 @@
           "title": "retries for dns request",
           "description": "Retries is the number of retries for the DNS request"
         },
+        "trace": {
+          "type": "boolean",
+          "title": "trace operation",
+          "description": "Trace performs a trace operation for the target."
+        },
+        "trace-max-recursion": {
+          "type": "integer",
+          "title": "trace-max-recursion level for dns request",
+          "description": "TraceMaxRecursion is the number of max recursion allowed for trace operations"
+        },
         "recursion": {
           "type": "boolean",
           "title": "recurse all servers",
           "description": "Recursion determines if resolver should recurse all records to get fresh results"
+        },
+        "resolvers": {
+          "items": {
+            "type": "string"
+          },
+          "type": "array",
+          "title": "Resolvers",
+          "description": "Define resolvers to use within the template"
         }
       },
       "additionalProperties": false,
       "type": "object"
     },
     "file.Request": {
+      "required": [
+        "Archive",
+        "MimeType"
+      ],
       "properties": {
         "matchers": {
           "items": {
@@ -418,8 +506,8 @@
             "type": "string"
           },
           "type": "array",
-          "title": "extensions to deny match",
-          "description": "List of file extensions to deny during matching"
+          "title": "denylist",
+          "description": "List of files"
         },
         "id": {
           "type": "string",
@@ -427,10 +515,16 @@
           "description": "ID is the optional ID for the request"
         },
         "max-size": {
-          "type": "integer",
+          "type": "string",
           "title": "max size data to run request on",
           "description": "Maximum size of the file to run request on"
         },
+        "Archive": {
+          "type": "boolean"
+        },
+        "MimeType": {
+          "type": "boolean"
+        },
         "no-recursive": {
           "type": "boolean",
           "title": "do not perform recursion",
@@ -447,6 +541,21 @@
           "title": "id of the request",
           "description": "Optional ID of the headless request"
         },
+        "attack": {
+          "$ref": "#/definitions/generators.AttackTypeHolder",
+          "title": "attack is the payload combination",
+          "description": "Attack is the type of payload combinations to perform"
+        },
+        "payloads": {
+          "patternProperties": {
+            ".*": {
+              "additionalProperties": true
+            }
+          },
+          "type": "object",
+          "title": "payloads for the headless request",
+          "description": "Payloads contains any payloads for the current request"
+        },
         "steps": {
           "items": {
             "$schema": "http://json-schema.org/draft-04/schema#",
@@ -456,6 +565,17 @@
           "title": "list of actions for headless request",
           "description": "List of actions to run for headless request"
         },
+        "user_agent": {
+          "$schema": "http://json-schema.org/draft-04/schema#",
+          "$ref": "#/definitions/userAgent.UserAgentHolder",
+          "title": "user agent for the headless request",
+          "description": "User agent for the headless request"
+        },
+        "custom_user_agent": {
+          "type": "string",
+          "title": "custom user agent for the headless request",
+          "description": "Custom user agent for the headless request"
+        },
         "matchers": {
           "items": {
             "$ref": "#/definitions/matchers.Matcher"
@@ -511,30 +631,8 @@
           "description": "Description of the headless action"
         },
         "action": {
-          "enum": [
-            "navigate",
-            "script",
-            "click",
-            "rightclick",
-            "text",
-            "screenshot",
-            "time",
-            "select",
-            "files",
-            "waitload",
-            "getresource",
-            "extract",
-            "setmethod",
-            "addheader",
-            "setheader",
-            "deleteheader",
-            "setbody",
-            "waitevent",
-            "keyboard",
-            "debug",
-            "sleep"
-          ],
-          "type": "string",
+          "$schema": "http://json-schema.org/draft-04/schema#",
+          "$ref": "#/definitions/engine.ActionTypeHolder",
           "title": "action to perform",
           "description": "Type of actions to perform"
         }
@@ -542,6 +640,53 @@
       "additionalProperties": false,
       "type": "object"
     },
+    "engine.ActionTypeHolder": {
+      "enum": [
+        "navigate",
+        "script",
+        "click",
+        "rightclick",
+        "text",
+        "screenshot",
+        "time",
+        "select",
+        "files",
+        "waitload",
+        "getresource",
+        "extract",
+        "set-method",
+        "addheader",
+        "setheader",
+        "deleteheader",
+        "setbody",
+        "waitevent",
+        "keyboard",
+        "debug",
+        "sleep",
+        "waitvisible"
+      ],
+      "type": "string",
+      "title": "action to perform",
+      "description": "Type of actions to perform"
+    },
+    "http.HTTPMethodTypeHolder": {
+      "enum": [
+        "GET",
+        "HEAD",
+        "POST",
+        "PUT",
+        "DELETE",
+        "CONNECT",
+        "OPTIONS",
+        "TRACE",
+        "PATCH",
+        "PURGE",
+        "DEBUG"
+      ],
+      "type": "string",
+      "title": "method is the HTTP request method",
+      "description": "Method is the HTTP Request Method"
+    },
     "http.Request": {
       "properties": {
         "matchers": {
@@ -597,29 +742,14 @@
           "description": "Optional name for the HTTP Request"
         },
         "attack": {
-          "enum": [
-            "sniper",
-            "pitchfork",
-            "clusterbomb"
-          ],
-          "type": "string",
+          "$schema": "http://json-schema.org/draft-04/schema#",
+          "$ref": "#/definitions/generators.AttackTypeHolder",
           "title": "attack is the payload combination",
           "description": "Attack is the type of payload combinations to perform"
         },
         "method": {
-          "enum": [
-            "GET",
-            "HEAD",
-            "POST",
-            "PUT",
-            "DELETE",
-            "CONNECT",
-            "OPTIONS",
-            "TRACE",
-            "PATCH",
-            "PURGE"
-          ],
-          "type": "string",
+          "$schema": "http://json-schema.org/draft-04/schema#",
+          "$ref": "#/definitions/http.HTTPMethodTypeHolder",
           "title": "method is the http request method",
           "description": "Method is the HTTP Request Method"
         },
@@ -678,11 +808,22 @@
           "title": "maximum http response body size",
           "description": "Maximum size of http response body to read in bytes"
         },
+        "signature": {
+          "$schema": "http://json-schema.org/draft-04/schema#",
+          "$ref": "#/definitions/http.SignatureTypeHolder",
+          "title": "signature is the http request signature method",
+          "description": "Signature is the HTTP Request signature Method"
+        },
         "cookie-reuse": {
           "type": "boolean",
           "title": "optional cookie reuse enable",
           "description": "Optional setting that enables cookie reuse"
         },
+        "read-all": {
+          "type": "boolean",
+          "title": "force read all body",
+          "description": "Enables force reading of entire unsafe http request body"
+        },
         "redirects": {
           "type": "boolean",
           "title": "follow http redirects",
@@ -712,11 +853,39 @@
           "type": "boolean",
           "title": "stop at first match",
           "description": "Stop the execution after a match is found"
+        },
+        "skip-variables-check": {
+          "type": "boolean",
+          "title": "skip variable checks",
+          "description": "Skips the check for unresolved variables in request"
+        },
+        "iterate-all": {
+          "type": "boolean",
+          "title": "iterate all the values",
+          "description": "Iterates all the values extracted from internal extractors"
+        },
+        "digest-username": {
+          "type": "string",
+          "title": "specifies the username for digest authentication",
+          "description": "Optional parameter which specifies the username for digest auth"
+        },
+        "digest-password": {
+          "type": "string",
+          "title": "specifies the password for digest authentication",
+          "description": "Optional parameter which specifies the password for digest auth"
         }
       },
       "additionalProperties": false,
       "type": "object"
     },
+    "http.SignatureTypeHolder": {
+      "enum": [
+        "AWS"
+      ],
+      "type": "string",
+      "title": "type of the signature",
+      "description": "Type of the signature"
+    },
     "network.Input": {
       "properties": {
         "data": {
@@ -725,11 +894,8 @@
           "description": "Data is the data to send as the input"
         },
         "type": {
-          "enum": [
-            "hex",
-            "text"
-          ],
-          "type": "string",
+          "$schema": "http://json-schema.org/draft-04/schema#",
+          "$ref": "#/definitions/network.NetworkInputTypeHolder",
           "title": "type is the type of input data",
           "description": "Type of input specified in data field"
         },
@@ -747,6 +913,15 @@
       "additionalProperties": false,
       "type": "object"
     },
+    "network.NetworkInputTypeHolder": {
+      "enum": [
+        "hex",
+        "text"
+      ],
+      "type": "string",
+      "title": "type is the type of input data",
+      "description": "description=Type of input specified in data field"
+    },
     "network.Request": {
       "properties": {
         "id": {
@@ -763,12 +938,7 @@
           "description": "Host to send network requests to"
         },
         "attack": {
-          "enum": [
-            "sniper",
-            "pitchfork",
-            "clusterbomb"
-          ],
-          "type": "string",
+          "$ref": "#/definitions/generators.AttackTypeHolder",
           "title": "attack is the payload combination",
           "description": "Attack is the type of payload combinations to perform"
         },
@@ -796,6 +966,11 @@
           "title": "size of network response to read",
           "description": "Size of response to read at the end. Default is 1024 bytes"
         },
+        "read-all": {
+          "type": "boolean",
+          "title": "read all response stream",
+          "description": "Read all response stream till the server stops sending"
+        },
         "matchers": {
           "items": {
             "$ref": "#/definitions/matchers.Matcher"
@@ -825,6 +1000,199 @@
       "additionalProperties": false,
       "type": "object"
     },
+    "ssl.Request": {
+      "properties": {
+        "matchers": {
+          "items": {
+            "$ref": "#/definitions/matchers.Matcher"
+          },
+          "type": "array",
+          "title": "matchers to run on response",
+          "description": "Detection mechanism to identify whether the request was successful by doing pattern matching"
+        },
+        "extractors": {
+          "items": {
+            "$ref": "#/definitions/extractors.Extractor"
+          },
+          "type": "array",
+          "title": "extractors to run on response",
+          "description": "Extractors contains the extraction mechanism for the request to identify and extract parts of the response"
+        },
+        "matchers-condition": {
+          "enum": [
+            "and",
+            "or"
+          ],
+          "type": "string",
+          "title": "condition between the matchers",
+          "description": "Conditions between the matchers"
+        },
+        "address": {
+          "type": "string",
+          "title": "address for the ssl request",
+          "description": "Address contains address for the request"
+        },
+        "min_version": {
+          "enum": [
+            "sslv3",
+            "tls10",
+            "tls11",
+            "tls12",
+            "tls13"
+          ],
+          "type": "string",
+          "title": "TLS version",
+          "description": "Minimum tls version - automatic if not specified."
+        },
+        "max_version": {
+          "enum": [
+            "sslv3",
+            "tls10",
+            "tls11",
+            "tls12",
+            "tls13"
+          ],
+          "type": "string",
+          "title": "TLS version",
+          "description": "Max tls version - automatic if not specified."
+        },
+        "cipher_suites": {
+          "items": {
+            "type": "string"
+          },
+          "type": "array"
+        }
+      },
+      "additionalProperties": false,
+      "type": "object"
+    },
+    "websocket.Input": {
+      "properties": {
+        "data": {
+          "type": "string",
+          "title": "data to send as input",
+          "description": "Data is the data to send as the input"
+        },
+        "name": {
+          "type": "string",
+          "title": "optional name for data read",
+          "description": "Optional name of the data read to provide matching on"
+        }
+      },
+      "additionalProperties": false,
+      "type": "object"
+    },
+    "websocket.Request": {
+      "properties": {
+        "matchers": {
+          "items": {
+            "$ref": "#/definitions/matchers.Matcher"
+          },
+          "type": "array",
+          "title": "matchers to run on response",
+          "description": "Detection mechanism to identify whether the request was successful by doing pattern matching"
+        },
+        "extractors": {
+          "items": {
+            "$ref": "#/definitions/extractors.Extractor"
+          },
+          "type": "array",
+          "title": "extractors to run on response",
+          "description": "Extractors contains the extraction mechanism for the request to identify and extract parts of the response"
+        },
+        "matchers-condition": {
+          "enum": [
+            "and",
+            "or"
+          ],
+          "type": "string",
+          "title": "condition between the matchers",
+          "description": "Conditions between the matchers"
+        },
+        "address": {
+          "type": "string",
+          "title": "address for the websocket request",
+          "description": "Address contains address for the request"
+        },
+        "inputs": {
+          "items": {
+            "$schema": "http://json-schema.org/draft-04/schema#",
+            "$ref": "#/definitions/websocket.Input"
+          },
+          "type": "array",
+          "title": "inputs for the websocket request",
+          "description": "Inputs contains any input/output for the current request"
+        },
+        "headers": {
+          "patternProperties": {
+            ".*": {
+              "type": "string"
+            }
+          },
+          "type": "object",
+          "title": "headers contains the request headers",
+          "description": "Headers contains headers for the request"
+        },
+        "attack": {
+          "$ref": "#/definitions/generators.AttackTypeHolder",
+          "title": "attack is the payload combination",
+          "description": "Attack is the type of payload combinations to perform"
+        },
+        "payloads": {
+          "patternProperties": {
+            ".*": {
+              "additionalProperties": true
+            }
+          },
+          "type": "object",
+          "title": "payloads for the webosocket request",
+          "description": "Payloads contains any payloads for the current request"
+        }
+      },
+      "additionalProperties": false,
+      "type": "object"
+    },
+    "whois.Request": {
+      "properties": {
+        "matchers": {
+          "items": {
+            "$ref": "#/definitions/matchers.Matcher"
+          },
+          "type": "array",
+          "title": "matchers to run on response",
+          "description": "Detection mechanism to identify whether the request was successful by doing pattern matching"
+        },
+        "extractors": {
+          "items": {
+            "$ref": "#/definitions/extractors.Extractor"
+          },
+          "type": "array",
+          "title": "extractors to run on response",
+          "description": "Extractors contains the extraction mechanism for the request to identify and extract parts of the response"
+        },
+        "matchers-condition": {
+          "enum": [
+            "and",
+            "or"
+          ],
+          "type": "string",
+          "title": "condition between the matchers",
+          "description": "Conditions between the matchers"
+        },
+        "query": {
+          "type": "string",
+          "title": "query for the WHOIS request",
+          "description": "Query contains query for the request"
+        },
+        "server": {
+          "type": "string",
+          "title": "server url to execute the WHOIS request on",
+          "description": "Server contains the server url to execute the WHOIS request on"
+        }
+      },
+      "additionalProperties": false,
+      "type": "object"
+    },
     "templates.Template": {
       "required": [
         "id",
@@ -832,6 +1200,7 @@
       ],
       "properties": {
         "id": {
+          "pattern": "^([a-zA-Z0-9]+[-_])*[a-zA-Z0-9]+$",
           "type": "string",
           "title": "id of the template",
           "description": "The Unique ID for the template",
@@ -890,6 +1259,33 @@
           "title": "headless requests to make",
           "description": "Headless requests to make for the template"
         },
+        "ssl": {
+          "items": {
+            "$schema": "http://json-schema.org/draft-04/schema#",
+            "$ref": "#/definitions/ssl.Request"
+          },
+          "type": "array",
+          "title": "ssl requests to make",
+          "description": "SSL requests to make for the template"
+        },
+        "websocket": {
+          "items": {
+            "$schema": "http://json-schema.org/draft-04/schema#",
+            "$ref": "#/definitions/websocket.Request"
+          },
+          "type": "array",
+          "title": "websocket requests to make",
+          "description": "Websocket requests to make for the template"
+        },
+        "whois": {
+          "items": {
+            "$schema": "http://json-schema.org/draft-04/schema#",
+            "$ref": "#/definitions/whois.Request"
+          },
+          "type": "array",
+          "title": "whois requests to make",
+          "description": "WHOIS requests to make for the template"
+        },
         "workflows": {
           "items": {
             "$schema": "http://json-schema.org/draft-04/schema#",
@@ -898,6 +1294,27 @@
           "type": "array",
           "title": "list of workflows to execute",
           "description": "List of workflows to execute for template"
+        },
+        "self-contained": {
+          "type": "boolean",
+          "title": "mark requests as self-contained",
+          "description": "Mark Requests for the template as self-contained"
+        },
+        "stop-at-first-match": {
+          "type": "boolean",
+          "title": "stop at first match",
+          "description": "Stop at first match for the template"
+        },
+        "signature": {
+          "$ref": "#/definitions/http.SignatureTypeHolder",
+          "title": "signature is the http request signature method",
+          "description": "Signature is the HTTP Request signature Method"
+        },
+        "variables": {
+          "$schema": "http://json-schema.org/draft-04/schema#",
+          "$ref": "#/definitions/variables.Variable",
+          "title": "variables for the http request",
+          "description": "Variables contains any variables for the current request"
         }
       },
       "additionalProperties": false,

sonar-project.properties

@@ -0,0 +1,17 @@
+sonar.projectKey=projectdiscovery_nuclei
+sonar.organization=projectdiscovery
+
+# This is the name and version displayed in the SonarCloud UI.
+#sonar.projectName=dnsx
+#sonar.projectVersion=1.0
+
+# Path is relative to the sonar-project.properties file. Replace "\" by "/" on Windows.
+sonar.sources=v2/
+sonar.tests=v2/
+sonar.test.inclusions=**/*_test.go
+sonar.exclusions=v2/pkg/protocols/common/helpers/deserialization/testdata/*.java
+sonar.go.coverage.reportPaths=v2/cov.out
+sonar.externalIssuesReportPaths=v2/report.json
+
+# Encoding of the source code. Default is default system encoding
+#sonar.sourceEncoding=UTF-8

v2/.goreleaser.yml

@@ -3,33 +3,48 @@ before:
     - go mod tidy
 
 builds:
-- env:
-  - CGO_ENABLED=0
-  goos:
-    - windows
-    - linux
-    - darwin
-  goarch:
-    - amd64
-    - 386
-    - arm
-    - arm64
 
+- main: cmd/nuclei/main.go
+  binary: nuclei
+  id: nuclei-cli
+
+  env:
+  - CGO_ENABLED=0
+
+  goos: [windows,linux,darwin]
+  goarch: [amd64,386,arm,arm64]
   ignore:
     - goos: darwin
-      goarch: '386'
+      goarch: 386
     - goos: windows
-      goarch: 'arm'
+      goarch: arm
     - goos: windows
-      goarch: 'arm64'
+      goarch: arm64
 
-  binary: '{{ .ProjectName }}'
-  main: cmd/nuclei/main.go
+  flags:
+    - -trimpath
+
+- main: cmd/cve-annotate/main.go
+  binary: cve-annotate
+  id: annotate
+
+  env:
+  - CGO_ENABLED=0
+
+  goos: [linux]
+  goarch: [amd64]
 
 archives:
 - format: zip
+  id: nuclei
+  builds: [nuclei-cli]
   replacements:
       darwin: macOS
 
+- format: zip
+  id: annotate
+  builds: [annotate]
+  name_template: "{{ .Binary }}"
+
 checksum:
   algorithm: sha256

v2/Makefile

@@ -3,11 +3,17 @@ GOCMD=go
 GOBUILD=$(GOCMD) build
 GOMOD=$(GOCMD) mod
 GOTEST=$(GOCMD) test
-GOGET=$(GOCMD) get
+GOFLAGS := -v
+# This should be disabled if the binary uses pprof
+LDFLAGS := -s -w
+
+ifneq ($(shell go env GOOS),darwin)
+LDFLAGS := -extldflags "-static"
+endif
     
 all: build
 build:
-	$(GOBUILD) -v -ldflags="-extldflags=-static" -o "nuclei" cmd/nuclei/main.go
+	$(GOBUILD) $(GOFLAGS) -ldflags '$(LDFLAGS)' -o "nuclei" cmd/nuclei/main.go
 docs:
 	if ! which dstdocgen > /dev/null; then
 		echo -e "Command not found! Install? (y/n) \c"
@@ -17,6 +23,10 @@ docs:
 	$(GOBUILD) -o "cmd/docgen/docgen" cmd/docgen/docgen.go
 	./cmd/docgen/docgen docs.md nuclei-jsonschema.json
 test:
-	$(GOTEST) -v ./...
+	$(GOTEST) $(GOFLAGS) ./...
+integration:
+	bash ../integration_tests/run.sh
+functional:
+	bash cmd/functional-tests/run.sh
 tidy:
 	$(GOMOD) tidy

v2/cmd/cve-annotate/main.go

@@ -1,6 +1,7 @@
 package main
 
 import (
+	"bytes"
 	"flag"
 	"fmt"
 	"io/ioutil"
@@ -9,10 +10,18 @@ import (
 	"regexp"
 	"strings"
 
-	"github.com/Ice3man543/nvd"
+	"github.com/projectdiscovery/nvd"
+	"github.com/projectdiscovery/sliceutil"
+	"github.com/projectdiscovery/stringsutil"
+	"gopkg.in/yaml.v3"
+
 	"github.com/projectdiscovery/nuclei/v2/pkg/catalog"
 )
 
+const (
+	yamlIndentSpaces = 2
+)
+
 var (
 	input       = flag.String("i", "", "Templates to annotate")
 	templateDir = flag.String("d", "", "Custom template directory for update")
@@ -48,7 +57,7 @@ func process() error {
 		return err
 	}
 	for _, path := range paths {
-		data, err := ioutil.ReadFile(path)
+		data, err := os.ReadFile(path)
 		if err != nil {
 			return err
 		}
@@ -62,6 +71,8 @@ var (
 	severityRegex = regexp.MustCompile(`severity: ([a-z]+)`)
 )
 
+const maxReferenceCount = 5
+
 func getCVEData(client *nvd.Client, filePath, data string) {
 	matches := idRegex.FindAllStringSubmatch(data, 1)
 	if len(matches) == 0 {
@@ -70,15 +81,11 @@ func getCVEData(client *nvd.Client, filePath, data string) {
 	cveName := matches[0][1]
 
 	severityMatches := severityRegex.FindAllStringSubmatch(data, 1)
-	if len(matches) == 0 {
+	if len(severityMatches) == 0 {
 		return
 	}
 	severityValue := severityMatches[0][1]
 
-	// Skip if there's classification data already
-	if strings.Contains(data, "classification:") {
-		return
-	}
 	cveItem, err := client.FetchCVE(cveName)
 	if err != nil {
 		log.Printf("Could not fetch cve %s: %s\n", cveName, err)
@@ -97,45 +104,86 @@ func getCVEData(client *nvd.Client, filePath, data string) {
 	infoBlockIndexData := data[strings.Index(data, "info:"):]
 	requestsIndex := strings.Index(infoBlockIndexData, "requests:")
 	networkIndex := strings.Index(infoBlockIndexData, "network:")
-	if requestsIndex == -1 && networkIndex == -1 {
+	variablesIndex := strings.Index(infoBlockIndexData, "variables:")
+	if requestsIndex == -1 && networkIndex == -1 && variablesIndex == -1 {
 		return
 	}
 	if networkIndex != -1 {
 		requestsIndex = networkIndex
 	}
+	if variablesIndex != -1 {
+		requestsIndex = variablesIndex
+	}
 	infoBlockData := infoBlockIndexData[:requestsIndex]
 	infoBlockClean := strings.TrimRight(infoBlockData, "\n")
 
-	newInfoBlock := infoBlockClean
-	var changed bool
+	infoBlock := InfoBlock{}
+	err = yaml.Unmarshal([]byte(data), &infoBlock)
+	if err != nil {
+		log.Printf("Could not unmarshal info block: %s\n", err)
+	}
 
+	var changed bool
 	if newSeverity := isSeverityMatchingCvssScore(severityValue, cvssScore); newSeverity != "" {
 		changed = true
-		newInfoBlock = strings.ReplaceAll(newInfoBlock, severityMatches[0][0], "severity: "+newSeverity)
+		infoBlock.Info.Severity = newSeverity
 		fmt.Printf("Adjusting severity for %s from %s=>%s (%.2f)\n", filePath, severityValue, newSeverity, cvssScore)
 	}
-	if !strings.Contains(infoBlockClean, "classification") && (cvssScore != 0 && cvssMetrics != "") {
+	isCvssEmpty := cvssScore == 0 || cvssMetrics == ""
+	hasCvssChanged := infoBlock.Info.Classification.CvssScore != cvssScore || cvssMetrics != infoBlock.Info.Classification.CvssMetrics
+	if !isCvssEmpty && hasCvssChanged {
 		changed = true
-		newInfoBlock = newInfoBlock + fmt.Sprintf("\n  classification:\n    cvss-metrics: %s\n    cvss-score: %.2f\n    cve-id: %s", cvssMetrics, cvssScore, cveName)
+		infoBlock.Info.Classification.CvssMetrics = cvssMetrics
+		infoBlock.Info.Classification.CvssScore = cvssScore
+		infoBlock.Info.Classification.CveId = cveName
 		if len(cweID) > 0 && (cweID[0] != "NVD-CWE-Other" && cweID[0] != "NVD-CWE-noinfo") {
-			newInfoBlock = newInfoBlock + fmt.Sprintf("\n    cwe-id: %s", strings.Join(cweID, ","))
+			infoBlock.Info.Classification.CweId = strings.Join(cweID, ",")
 		}
 	}
 	// If there is no description field, fill the description from CVE information
-	if !strings.Contains(infoBlockClean, "description:") && len(cveItem.CVE.Description.DescriptionData) > 0 {
+	hasDescriptionData := len(cveItem.CVE.Description.DescriptionData) > 0
+	isDescriptionEmpty := infoBlock.Info.Description == ""
+	if isDescriptionEmpty && hasDescriptionData {
 		changed = true
-		newInfoBlock = newInfoBlock + fmt.Sprintf("\n  description: %s", fmt.Sprintf("%q", cveItem.CVE.Description.DescriptionData[0].Value))
+		// removes all new lines
+		description := stringsutil.ReplaceAny(cveItem.CVE.Description.DescriptionData[0].Value, "", "\n", "\\", "'", "\t")
+		description += "\n"
+		infoBlock.Info.Description = description
 	}
-	if !strings.Contains(infoBlockClean, "reference:") && len(cveItem.CVE.References.ReferenceData) > 0 {
+
+	// we are unmarshaling info block to have valid data
+	var referenceDataURLs []string
+	for _, reference := range cveItem.CVE.References.ReferenceData {
+		referenceDataURLs = append(referenceDataURLs, reference.URL)
+	}
+	hasReferenceData := len(cveItem.CVE.References.ReferenceData) > 0
+	areCveReferencesContained := sliceutil.ContainsItems(infoBlock.Info.Reference, referenceDataURLs)
+	referencesCount := len(infoBlock.Info.Reference)
+	if hasReferenceData && !areCveReferencesContained {
 		changed = true
-		newInfoBlock = newInfoBlock + "\n  reference:"
 		for _, reference := range cveItem.CVE.References.ReferenceData {
-			newInfoBlock = newInfoBlock + fmt.Sprintf("\n    - %s", reference.URL)
+			referencesCount++
+			if referencesCount >= maxReferenceCount {
+				break
+			}
+			infoBlock.Info.Reference = append(infoBlock.Info.Reference, reference.URL)
 		}
+		infoBlock.Info.Reference = sliceutil.PruneEmptyStrings(sliceutil.Dedupe(infoBlock.Info.Reference))
 	}
-	newTemplate := strings.ReplaceAll(data, infoBlockClean, newInfoBlock)
+
+	var newInfoBlock bytes.Buffer
+	yamlEncoder := yaml.NewEncoder(&newInfoBlock)
+	yamlEncoder.SetIndent(yamlIndentSpaces)
+	err = yamlEncoder.Encode(infoBlock)
+	if err != nil {
+		log.Printf("Could not marshal info block: %s\n", err)
+		return
+	}
+	newInfoBlockData := strings.TrimSuffix(newInfoBlock.String(), "\n")
+
+	newTemplate := strings.ReplaceAll(data, infoBlockClean, newInfoBlockData)
 	if changed {
-		_ = ioutil.WriteFile(filePath, []byte(newTemplate), 0777)
+		_ = ioutil.WriteFile(filePath, []byte(newTemplate), 0644)
 		fmt.Printf("Wrote updated template to %s\n", filePath)
 	}
 }
@@ -160,3 +208,27 @@ func isSeverityMatchingCvssScore(severity string, score float64) string {
 	}
 	return ""
 }
+
+// Cloning struct from nuclei as we don't want any validation
+type InfoBlock struct {
+	Info TemplateInfo `yaml:"info"`
+}
+
+type TemplateClassification struct {
+	CvssMetrics string  `yaml:"cvss-metrics,omitempty"`
+	CvssScore   float64 `yaml:"cvss-score,omitempty"`
+	CveId       string  `yaml:"cve-id,omitempty"`
+	CweId       string  `yaml:"cwe-id,omitempty"`
+}
+
+type TemplateInfo struct {
+	Name           string                 `yaml:"name"`
+	Author         string                 `yaml:"author"`
+	Severity       string                 `yaml:"severity"`
+	Description    string                 `yaml:"description,omitempty"`
+	Reference      []string               `yaml:"reference,omitempty"`
+	Remediation    string                 `yaml:"remediation,omitempty"`
+	Classification TemplateClassification `yaml:"classification,omitempty"`
+	Metadata       map[string]string      `yaml:"metadata,omitempty"`
+	Tags           string                 `yaml:"tags,omitempty"`
+}

v2/cmd/docgen/docgen.go

@@ -10,10 +10,11 @@ import (
 	"strings"
 
 	"github.com/alecthomas/jsonschema"
+
 	"github.com/projectdiscovery/nuclei/v2/pkg/templates"
 )
 
-var pathRegex = regexp.MustCompile(`github.com/projectdiscovery/nuclei/v2/(?:internal|pkg)/(?:.*/)?([A-Za-z\.]+)`)
+var pathRegex = regexp.MustCompile(`github\.com/projectdiscovery/nuclei/v2/(?:internal|pkg)/(?:.*/)?([A-Za-z.]+)`)
 
 func main() {
 	// Generate yaml syntax documentation
@@ -21,7 +22,7 @@ func main() {
 	if err != nil {
 		log.Fatalf("Could not encode docs: %s\n", err)
 	}
-	err = ioutil.WriteFile(os.Args[1], data, 0777)
+	err = ioutil.WriteFile(os.Args[1], data, 0644)
 	if err != nil {
 		log.Fatalf("Could not write docs: %s\n", err)
 	}
@@ -43,7 +44,7 @@ func main() {
 	for _, match := range pathRegex.FindAllStringSubmatch(schema, -1) {
 		schema = strings.ReplaceAll(schema, match[0], match[1])
 	}
-	err = ioutil.WriteFile(os.Args[2], []byte(schema), 0777)
+	err = ioutil.WriteFile(os.Args[2], []byte(schema), 0644)
 	if err != nil {
 		log.Fatalf("Could not write jsonschema: %s\n", err)
 	}

v2/cmd/functional-test/main.go

@@ -11,13 +11,13 @@ import (
 	"github.com/logrusorgru/aurora"
 	"github.com/pkg/errors"
 
-	"github.com/projectdiscovery/nuclei/v2/internal/testutils"
+	"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
 )
 
 var (
-	success = aurora.Green("[✓]").String()
-	failed  = aurora.Red("[✘]").String()
-	errored = false
+	success      = aurora.Green("[✓]").String()
+	failed       = aurora.Red("[✘]").String()
+	githubAction = os.Getenv("GH_ACTION") == "true"
 
 	mainNucleiBinary = flag.String("main", "", "Main Branch Nuclei Binary")
 	devNucleiBinary  = flag.String("dev", "", "Dev Branch Nuclei Binary")
@@ -27,49 +27,75 @@ var (
 func main() {
 	flag.Parse()
 
-	if err := runFunctionalTests(); err != nil {
+	debug := os.Getenv("DEBUG") == "true"
+
+	if err, errored := runFunctionalTests(debug); err != nil {
 		log.Fatalf("Could not run functional tests: %s\n", err)
-	}
-	if errored {
+	} else if errored {
 		os.Exit(1)
 	}
 }
 
-func runFunctionalTests() error {
+func runFunctionalTests(debug bool) (error, bool) {
 	file, err := os.Open(*testcases)
 	if err != nil {
-		return errors.Wrap(err, "could not open test cases")
+		return errors.Wrap(err, "could not open test cases"), true
 	}
 	defer file.Close()
 
-	scanner := bufio.NewScanner(file)
-	for scanner.Scan() {
-		text := strings.TrimSpace(scanner.Text())
-		if text == "" {
-			continue
-		}
-		if err := runIndividualTestCase(text); err != nil {
-			errored = true
-			fmt.Fprintf(os.Stderr, "%s Test \"%s\" failed: %s\n", failed, text, err)
-		} else {
-			fmt.Printf("%s Test \"%s\" passed!\n", success, text)
+	errored, failedTestCases := runTestCases(file, debug)
+
+	if githubAction {
+		fmt.Println("::group::Failed tests with debug")
+		for _, failedTestCase := range failedTestCases {
+			_ = runTestCase(failedTestCase, true)
 		}
+		fmt.Println("::endgroup::")
 	}
-	return nil
+
+	return nil, errored
 }
 
-func runIndividualTestCase(testcase string) error {
+func runTestCases(file *os.File, debug bool) (bool, []string) {
+	errored := false
+	var failedTestCases []string
+
+	scanner := bufio.NewScanner(file)
+	for scanner.Scan() {
+		testCase := strings.TrimSpace(scanner.Text())
+		if testCase == "" {
+			continue
+		}
+		if runTestCase(testCase, debug) {
+			errored = true
+			failedTestCases = append(failedTestCases, testCase)
+		}
+	}
+	return errored, failedTestCases
+}
+
+func runTestCase(testCase string, debug bool) bool {
+	if err := runIndividualTestCase(testCase, debug); err != nil {
+		fmt.Fprintf(os.Stderr, "%s Test \"%s\" failed: %s\n", failed, testCase, err)
+		return true
+	} else {
+		fmt.Printf("%s Test \"%s\" passed!\n", success, testCase)
+	}
+	return false
+}
+
+func runIndividualTestCase(testcase string, debug bool) error {
 	parts := strings.Fields(testcase)
 
 	var finalArgs []string
 	if len(parts) > 1 {
 		finalArgs = parts[1:]
 	}
-	mainOutput, err := testutils.RunNucleiBinaryAndGetLoadedTemplates(*mainNucleiBinary, finalArgs)
+	mainOutput, err := testutils.RunNucleiBinaryAndGetLoadedTemplates(*mainNucleiBinary, debug, finalArgs)
 	if err != nil {
 		return errors.Wrap(err, "could not run nuclei main test")
 	}
-	devOutput, err := testutils.RunNucleiBinaryAndGetLoadedTemplates(*devNucleiBinary, finalArgs)
+	devOutput, err := testutils.RunNucleiBinaryAndGetLoadedTemplates(*devNucleiBinary, debug, finalArgs)
 	if err != nil {
 		return errors.Wrap(err, "could not run nuclei dev test")
 	}

v2/cmd/functional-test/run.sh

@@ -1,13 +1,27 @@
 #!/bin/bash
 
-echo 'Building functional-test binary'
-go build
+# reading os type from arguments
+CURRENT_OS=$1
 
-echo 'Building Nuclei binary from current branch'
-go build -o nuclei_dev ../nuclei
+if [ "${CURRENT_OS}" == "windows-latest" ];then
+    extension=.exe
+fi
 
-echo 'Installing latest release of nuclei'
-GO111MODULE=on go get -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei
+echo "::group::Building functional-test binary"
+go build -o functional-test$extension
+echo "::endgroup::"
+
+echo "::group::Building Nuclei binary from current branch"
+go build -o nuclei_dev$extension ../nuclei
+echo "::endgroup::"
+
+echo "::group::Installing nuclei templates"
+./nuclei_dev$extension -update-templates
+echo "::endgroup::"
+
+echo "::group::Building latest release of nuclei"
+go build -o nuclei$extension -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei
+echo "::endgroup::"
 
 echo 'Starting Nuclei functional test'
-./functional-test -main nuclei -dev ./nuclei_dev -testcases testcases.txt
+./functional-test$extension -main ./nuclei$extension -dev ./nuclei_dev$extension -testcases testcases.txt

v2/cmd/integration-test/code.go

@@ -0,0 +1,129 @@
+package main
+
+import (
+	"fmt"
+	"log"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path"
+	"strings"
+
+	"github.com/julienschmidt/httprouter"
+	"github.com/logrusorgru/aurora"
+	"github.com/pkg/errors"
+	"github.com/projectdiscovery/goflags"
+	"github.com/projectdiscovery/nuclei/v2/pkg/catalog"
+	"github.com/projectdiscovery/nuclei/v2/pkg/catalog/config"
+	"github.com/projectdiscovery/nuclei/v2/pkg/catalog/loader"
+	"github.com/projectdiscovery/nuclei/v2/pkg/core"
+	"github.com/projectdiscovery/nuclei/v2/pkg/core/inputs"
+	"github.com/projectdiscovery/nuclei/v2/pkg/output"
+	"github.com/projectdiscovery/nuclei/v2/pkg/parsers"
+	"github.com/projectdiscovery/nuclei/v2/pkg/protocols"
+	"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/hosterrorscache"
+	"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/interactsh"
+	"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/protocolinit"
+	"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/protocolstate"
+	"github.com/projectdiscovery/nuclei/v2/pkg/reporting"
+	"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
+	"github.com/projectdiscovery/nuclei/v2/pkg/types"
+	"go.uber.org/ratelimit"
+)
+
+var codeTestcases = map[string]testutils.TestCase{
+	"code/test.yaml": &goIntegrationTest{},
+}
+
+type goIntegrationTest struct{}
+
+// Execute executes a test case and returns an error if occurred
+//
+// Execute the docs at ../DESIGN.md if the code stops working for integration.
+func (h *goIntegrationTest) Execute(templatePath string) error {
+	router := httprouter.New()
+
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		fmt.Fprintf(w, "This is test matcher text")
+		if strings.EqualFold(r.Header.Get("test"), "nuclei") {
+			fmt.Fprintf(w, "This is test headers matcher text")
+		}
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+
+	results, err := executeNucleiAsCode(templatePath, ts.URL)
+	if err != nil {
+		return err
+	}
+	return expectResultsCount(results, 1)
+}
+
+// executeNucleiAsCode contains an example
+func executeNucleiAsCode(templatePath, templateURL string) ([]string, error) {
+	cache := hosterrorscache.New(30, hosterrorscache.DefaultMaxHostsCount)
+	defer cache.Close()
+
+	mockProgress := &testutils.MockProgressClient{}
+	reportingClient, _ := reporting.New(&reporting.Options{}, "")
+	defer reportingClient.Close()
+
+	outputWriter := testutils.NewMockOutputWriter()
+	var results []string
+	outputWriter.WriteCallback = func(event *output.ResultEvent) {
+		results = append(results, fmt.Sprintf("%v\n", event))
+	}
+
+	defaultOpts := types.DefaultOptions()
+	_ = protocolstate.Init(defaultOpts)
+	_ = protocolinit.Init(defaultOpts)
+
+	defaultOpts.Templates = goflags.FileOriginalNormalizedStringSlice{templatePath}
+	defaultOpts.ExcludeTags = config.ReadIgnoreFile().Tags
+
+	interactOpts := interactsh.NewDefaultOptions(outputWriter, reportingClient, mockProgress)
+	interactClient, err := interactsh.New(interactOpts)
+	if err != nil {
+		return nil, errors.Wrap(err, "could not create interact client")
+	}
+	defer interactClient.Close()
+
+	home, _ := os.UserHomeDir()
+	catalog := catalog.New(path.Join(home, "nuclei-templates"))
+	executerOpts := protocols.ExecuterOptions{
+		Output:          outputWriter,
+		Options:         defaultOpts,
+		Progress:        mockProgress,
+		Catalog:         catalog,
+		IssuesClient:    reportingClient,
+		RateLimiter:     ratelimit.New(150),
+		Interactsh:      interactClient,
+		HostErrorsCache: cache,
+		Colorizer:       aurora.NewAurora(true),
+		ResumeCfg:       types.NewResumeCfg(),
+	}
+	engine := core.New(defaultOpts)
+	engine.SetExecuterOptions(executerOpts)
+
+	workflowLoader, err := parsers.NewLoader(&executerOpts)
+	if err != nil {
+		log.Fatalf("Could not create workflow loader: %s\n", err)
+	}
+	executerOpts.WorkflowLoader = workflowLoader
+
+	configObject, err := config.ReadConfiguration()
+	if err != nil {
+		return nil, errors.Wrap(err, "could not read configuration file")
+	}
+	store, err := loader.New(loader.NewConfig(defaultOpts, configObject, catalog, executerOpts))
+	if err != nil {
+		return nil, errors.Wrap(err, "could not create loader")
+	}
+	store.Load()
+
+	input := &inputs.SimpleInputProvider{Inputs: []string{templateURL}}
+	_ = engine.Execute(store.Templates(), input)
+	engine.WorkPool().Wait() // Wait for the scan to finish
+
+	return results, nil
+}

v2/cmd/integration-test/dns.go

@@ -1,11 +1,14 @@
 package main
 
 import (
-	"github.com/projectdiscovery/nuclei/v2/internal/testutils"
+	"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
 )
 
 var dnsTestCases = map[string]testutils.TestCase{
-	"dns/basic.yaml": &dnsBasic{},
+	"dns/basic.yaml":     &dnsBasic{},
+	"dns/ptr.yaml":       &dnsPtr{},
+	"dns/caa.yaml":       &dnsCAA{},
+	"dns/variables.yaml": &dnsVariables{},
 }
 
 type dnsBasic struct{}
@@ -14,15 +17,60 @@ type dnsBasic struct{}
 func (h *dnsBasic) Execute(filePath string) error {
 	var routerErr error
 
-	results, err := testutils.RunNucleiAndGetResults(filePath, "one.one.one.one", debug)
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "one.one.one.one", debug)
 	if err != nil {
 		return err
 	}
 	if routerErr != nil {
 		return routerErr
 	}
-	if len(results) != 1 {
-		return errIncorrectResultsCount(results)
+	return expectResultsCount(results, 1)
+}
+
+type dnsPtr struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *dnsPtr) Execute(filePath string) error {
+	var routerErr error
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "1.1.1.1", debug)
+	if err != nil {
+		return err
 	}
-	return nil
+	if routerErr != nil {
+		return routerErr
+	}
+	return expectResultsCount(results, 1)
+}
+
+type dnsCAA struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *dnsCAA) Execute(filePath string) error {
+	var routerErr error
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "google.com", debug)
+	if err != nil {
+		return err
+	}
+	if routerErr != nil {
+		return routerErr
+	}
+	return expectResultsCount(results, 1)
+}
+
+type dnsVariables struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *dnsVariables) Execute(filePath string) error {
+	var routerErr error
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "one.one.one.one", debug)
+	if err != nil {
+		return err
+	}
+	if routerErr != nil {
+		return routerErr
+	}
+	return expectResultsCount(results, 1)
 }

v2/cmd/integration-test/file.go

@@ -0,0 +1,47 @@
+package main
+
+import (
+	"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
+)
+
+var fileTestcases = map[string]testutils.TestCase{
+	"file/matcher-with-or.yaml":  &fileWithOrMatcher{},
+	"file/matcher-with-and.yaml": &fileWithAndMatcher{},
+	"file/extract.yaml":          &fileWithExtractor{},
+}
+
+type fileWithOrMatcher struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *fileWithOrMatcher) Execute(filePath string) error {
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "file/data/", debug)
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 1)
+}
+
+type fileWithAndMatcher struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *fileWithAndMatcher) Execute(filePath string) error {
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "file/data/", debug)
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 1)
+}
+
+type fileWithExtractor struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *fileWithExtractor) Execute(filePath string) error {
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "file/data/", debug)
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 1)
+}

v2/cmd/integration-test/headless.go

@@ -0,0 +1,113 @@
+package main
+
+import (
+	"net/http"
+	"net/http/httptest"
+
+	"github.com/julienschmidt/httprouter"
+
+	"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
+)
+
+var headlessTestcases = map[string]testutils.TestCase{
+	"headless/headless-basic.yaml":          &headlessBasic{},
+	"headless/headless-header-action.yaml":  &headlessHeaderActions{},
+	"headless/headless-extract-values.yaml": &headlessExtractValues{},
+	"headless/headless-payloads.yaml":       &headlessPayloads{},
+	"headless/variables.yaml":               &headlessVariables{},
+}
+
+type headlessBasic struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *headlessBasic) Execute(filePath string) error {
+	router := httprouter.New()
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		_, _ = w.Write([]byte("<html><body></body></html>"))
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug, "-headless")
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 1)
+}
+
+type headlessHeaderActions struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *headlessHeaderActions) Execute(filePath string) error {
+	router := httprouter.New()
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		testValue := r.Header.Get("test")
+		if r.Header.Get("test") != "" {
+			_, _ = w.Write([]byte("<html><body>" + testValue + "</body></html>"))
+		}
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug, "-headless")
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 1)
+}
+
+type headlessExtractValues struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *headlessExtractValues) Execute(filePath string) error {
+	router := httprouter.New()
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		_, _ = w.Write([]byte("<html><body><a href='/test.html'>test</a></body></html>"))
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug, "-headless")
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 3)
+}
+
+type headlessPayloads struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *headlessPayloads) Execute(filePath string) error {
+	router := httprouter.New()
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		_, _ = w.Write([]byte("<html><body>test</body></html>"))
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug, "-headless")
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 4)
+}
+
+type headlessVariables struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *headlessVariables) Execute(filePath string) error {
+	router := httprouter.New()
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		_, _ = w.Write([]byte("<html><body>aGVsbG8=</body></html>"))
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug, "-headless")
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 1)
+}

v2/cmd/integration-test/http.go

@@ -8,38 +8,97 @@ import (
 	"net/http"
 	"net/http/httptest"
 	"net/http/httputil"
+	"regexp"
+	"strconv"
 	"strings"
 
 	"github.com/julienschmidt/httprouter"
 
-	"github.com/projectdiscovery/nuclei/v2/internal/testutils"
+	"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
 )
 
 var httpTestcases = map[string]testutils.TestCase{
-	"http/get-headers.yaml":           &httpGetHeaders{},
-	"http/get-query-string.yaml":      &httpGetQueryString{},
-	"http/get-redirects.yaml":         &httpGetRedirects{},
-	"http/get.yaml":                   &httpGet{},
-	"http/post-body.yaml":             &httpPostBody{},
-	"http/post-json-body.yaml":        &httpPostJSONBody{},
-	"http/post-multipart-body.yaml":   &httpPostMultipartBody{},
-	"http/raw-cookie-reuse.yaml":      &httpRawCookieReuse{},
-	"http/raw-dynamic-extractor.yaml": &httpRawDynamicExtractor{},
-	"http/raw-get-query.yaml":         &httpRawGetQuery{},
-	"http/raw-get.yaml":               &httpRawGet{},
-	"http/raw-payload.yaml":           &httpRawPayload{},
-	"http/raw-post-body.yaml":         &httpRawPostBody{},
-	"http/raw-unsafe-request.yaml":    &httpRawUnsafeRequest{},
-	"http/request-condition.yaml":     &httpRequestCondition{},
-	"http/request-condition-new.yaml": &httpRequestCondition{},
+	"http/get-headers.yaml":                         &httpGetHeaders{},
+	"http/get-query-string.yaml":                    &httpGetQueryString{},
+	"http/get-redirects.yaml":                       &httpGetRedirects{},
+	"http/disable-redirects.yaml":                   &httpDisableRedirects{},
+	"http/get.yaml":                                 &httpGet{},
+	"http/post-body.yaml":                           &httpPostBody{},
+	"http/post-json-body.yaml":                      &httpPostJSONBody{},
+	"http/post-multipart-body.yaml":                 &httpPostMultipartBody{},
+	"http/raw-cookie-reuse.yaml":                    &httpRawCookieReuse{},
+	"http/raw-dynamic-extractor.yaml":               &httpRawDynamicExtractor{},
+	"http/raw-get-query.yaml":                       &httpRawGetQuery{},
+	"http/raw-get.yaml":                             &httpRawGet{},
+	"http/raw-payload.yaml":                         &httpRawPayload{},
+	"http/raw-post-body.yaml":                       &httpRawPostBody{},
+	"http/raw-unsafe-request.yaml":                  &httpRawUnsafeRequest{},
+	"http/request-condition.yaml":                   &httpRequestCondition{},
+	"http/request-condition-new.yaml":               &httpRequestCondition{},
+	"http/interactsh.yaml":                          &httpInteractshRequest{},
+	"http/interactsh-stop-at-first-match.yaml":      &httpInteractshStopAtFirstMatchRequest{},
+	"http/self-contained.yaml":                      &httpRequestSelContained{},
+	"http/get-case-insensitive.yaml":                &httpGetCaseInsensitive{},
+	"http/get.yaml,http/get-case-insensitive.yaml":  &httpGetCaseInsensitiveCluster{},
+	"http/get-redirects-chain-headers.yaml":         &httpGetRedirectsChainHeaders{},
+	"http/dsl-matcher-variable.yaml":                &httpDSLVariable{},
+	"http/dsl-functions.yaml":                       &httpDSLFunctions{},
+	"http/race-simple.yaml":                         &httpRaceSimple{},
+	"http/race-multiple.yaml":                       &httpRaceMultiple{},
+	"http/stop-at-first-match.yaml":                 &httpStopAtFirstMatch{},
+	"http/stop-at-first-match-with-extractors.yaml": &httpStopAtFirstMatchWithExtractors{},
+	"http/variables.yaml":                           &httpVariables{},
+	"http/get-override-sni.yaml":                    &httpSniAnnotation{},
+	"http/get-sni.yaml":                             &customCLISNI{},
+	"http/redirect-match-url.yaml":                  &httpRedirectMatchURL{},
+	"http/get-sni-unsafe.yaml":                      &customCLISNIUnsafe{},
 }
 
-func httpDebugRequestDump(r *http.Request) {
-	if debug {
-		if dump, err := httputil.DumpRequest(r, true); err == nil {
-			fmt.Printf("\nRequest dump: \n%s\n\n", string(dump))
+type httpInteractshRequest struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *httpInteractshRequest) Execute(filePath string) error {
+	router := httprouter.New()
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		value := r.Header.Get("url")
+		if value != "" {
+			if resp, _ := http.DefaultClient.Get(value); resp != nil {
+				resp.Body.Close()
+			}
 		}
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
+	if err != nil {
+		return err
 	}
+
+	return expectResultsCount(results, 1)
+}
+
+type httpInteractshStopAtFirstMatchRequest struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *httpInteractshStopAtFirstMatchRequest) Execute(filePath string) error {
+	router := httprouter.New()
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		value := r.Header.Get("url")
+		if value != "" {
+			if resp, _ := http.DefaultClient.Get(value); resp != nil {
+				resp.Body.Close()
+			}
+		}
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
+	if err != nil {
+		return err
+	}
+	return expectResultsCount(results, 1)
 }
 
 type httpGetHeaders struct{}
@@ -47,23 +106,20 @@ type httpGetHeaders struct{}
 // Execute executes a test case and returns an error if occurred
 func (h *httpGetHeaders) Execute(filePath string) error {
 	router := httprouter.New()
-	router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
-		httpDebugRequestDump(r)
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
 		if strings.EqualFold(r.Header.Get("test"), "nuclei") {
 			fmt.Fprintf(w, "This is test headers matcher text")
 		}
-	}))
+	})
 	ts := httptest.NewServer(router)
 	defer ts.Close()
 
-	results, err := testutils.RunNucleiAndGetResults(filePath, ts.URL, debug)
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
 	if err != nil {
 		return err
 	}
-	if len(results) != 1 {
-		return errIncorrectResultsCount(results)
-	}
-	return nil
+
+	return expectResultsCount(results, 1)
 }
 
 type httpGetQueryString struct{}
@@ -71,23 +127,20 @@ type httpGetQueryString struct{}
 // Execute executes a test case and returns an error if occurred
 func (h *httpGetQueryString) Execute(filePath string) error {
 	router := httprouter.New()
-	router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
-		httpDebugRequestDump(r)
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
 		if strings.EqualFold(r.URL.Query().Get("test"), "nuclei") {
 			fmt.Fprintf(w, "This is test querystring matcher text")
 		}
-	}))
+	})
 	ts := httptest.NewServer(router)
 	defer ts.Close()
 
-	results, err := testutils.RunNucleiAndGetResults(filePath, ts.URL, debug)
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
 	if err != nil {
 		return err
 	}
-	if len(results) != 1 {
-		return errIncorrectResultsCount(results)
-	}
-	return nil
+
+	return expectResultsCount(results, 1)
 }
 
 type httpGetRedirects struct{}
@@ -95,25 +148,43 @@ type httpGetRedirects struct{}
 // Execute executes a test case and returns an error if occurred
 func (h *httpGetRedirects) Execute(filePath string) error {
 	router := httprouter.New()
-	router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
-		httpDebugRequestDump(r)
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
 		http.Redirect(w, r, "/redirected", http.StatusFound)
-	}))
-	router.GET("/redirected", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
-		httpDebugRequestDump(r)
+	})
+	router.GET("/redirected", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
 		fmt.Fprintf(w, "This is test redirects matcher text")
-	}))
+	})
 	ts := httptest.NewServer(router)
 	defer ts.Close()
 
-	results, err := testutils.RunNucleiAndGetResults(filePath, ts.URL, debug)
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
 	if err != nil {
 		return err
 	}
-	if len(results) != 1 {
-		return errIncorrectResultsCount(results)
+
+	return expectResultsCount(results, 1)
+}
+
+type httpDisableRedirects struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *httpDisableRedirects) Execute(filePath string) error {
+	router := httprouter.New()
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		http.Redirect(w, r, "/redirected", http.StatusMovedPermanently)
+	})
+	router.GET("/redirected", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		fmt.Fprintf(w, "This is test redirects matcher text")
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug, "-dr")
+	if err != nil {
+		return err
 	}
-	return nil
+
+	return expectResultsCount(results, 0)
 }
 
 type httpGet struct{}
@@ -121,20 +192,89 @@ type httpGet struct{}
 // Execute executes a test case and returns an error if occurred
 func (h *httpGet) Execute(filePath string) error {
 	router := httprouter.New()
-	router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
-		httpDebugRequestDump(r)
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
 		fmt.Fprintf(w, "This is test matcher text")
-	}))
+	})
 	ts := httptest.NewServer(router)
 	defer ts.Close()
 
-	results, err := testutils.RunNucleiAndGetResults(filePath, ts.URL, debug)
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
 	if err != nil {
 		return err
 	}
-	if len(results) != 1 {
-		return errIncorrectResultsCount(results)
+
+	return expectResultsCount(results, 1)
+}
+
+type httpDSLVariable struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *httpDSLVariable) Execute(filePath string) error {
+	router := httprouter.New()
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		fmt.Fprintf(w, "This is test matcher text")
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
+	if err != nil {
+		return err
 	}
+
+	return expectResultsCount(results, 5)
+}
+
+type httpDSLFunctions struct{}
+
+func (h *httpDSLFunctions) Execute(filePath string) error {
+	router := httprouter.New()
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		request, err := httputil.DumpRequest(r, true)
+		if err != nil {
+			_, _ = fmt.Fprint(w, err.Error())
+		} else {
+			_, _ = fmt.Fprint(w, string(request))
+		}
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug, "-nc")
+	if err != nil {
+		return err
+	}
+
+	if err := expectResultsCount(results, 1); err != nil {
+		return err
+	}
+
+	resultPattern := regexp.MustCompile(`\[[^]]+] \[[^]]+] \[[^]]+] [^]]+ \[([^]]+)]`)
+	submatch := resultPattern.FindStringSubmatch(results[0])
+	if len(submatch) != 2 {
+		return errors.New("could not parse the result")
+	}
+
+	totalExtracted := strings.Split(submatch[1], ",")
+	numberOfDslFunctions := 57
+	if len(totalExtracted) != numberOfDslFunctions {
+		return errors.New("incorrect number of results")
+	}
+
+	for _, header := range totalExtracted {
+		parts := strings.Split(header, ": ")
+		index, err := strconv.Atoi(parts[0])
+		if err != nil {
+			return err
+		}
+		if index < 0 || index > numberOfDslFunctions {
+			return fmt.Errorf("incorrect header index found: %d", index)
+		}
+		if strings.TrimSpace(parts[1]) == "" {
+			return fmt.Errorf("the DSL expression with index %d was not evaluated correctly", index)
+		}
+	}
+
 	return nil
 }
 
@@ -145,8 +285,7 @@ func (h *httpPostBody) Execute(filePath string) error {
 	router := httprouter.New()
 	var routerErr error
 
-	router.POST("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
-		httpDebugRequestDump(r)
+	router.POST("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
 		if err := r.ParseForm(); err != nil {
 			routerErr = err
 			return
@@ -154,21 +293,19 @@ func (h *httpPostBody) Execute(filePath string) error {
 		if strings.EqualFold(r.Form.Get("username"), "test") && strings.EqualFold(r.Form.Get("password"), "nuclei") {
 			fmt.Fprintf(w, "This is test post-body matcher text")
 		}
-	}))
+	})
 	ts := httptest.NewServer(router)
 	defer ts.Close()
 
-	results, err := testutils.RunNucleiAndGetResults(filePath, ts.URL, debug)
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
 	if err != nil {
 		return err
 	}
 	if routerErr != nil {
 		return routerErr
 	}
-	if len(results) != 1 {
-		return errIncorrectResultsCount(results)
-	}
-	return nil
+
+	return expectResultsCount(results, 1)
 }
 
 type httpPostJSONBody struct{}
@@ -178,9 +315,7 @@ func (h *httpPostJSONBody) Execute(filePath string) error {
 	router := httprouter.New()
 	var routerErr error
 
-	router.POST("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
-		httpDebugRequestDump(r)
-
+	router.POST("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
 		type doc struct {
 			Username string `json:"username"`
 			Password string `json:"password"`
@@ -193,21 +328,19 @@ func (h *httpPostJSONBody) Execute(filePath string) error {
 		if strings.EqualFold(obj.Username, "test") && strings.EqualFold(obj.Password, "nuclei") {
 			fmt.Fprintf(w, "This is test post-json-body matcher text")
 		}
-	}))
+	})
 	ts := httptest.NewServer(router)
 	defer ts.Close()
 
-	results, err := testutils.RunNucleiAndGetResults(filePath, ts.URL, debug)
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
 	if err != nil {
 		return err
 	}
 	if routerErr != nil {
 		return routerErr
 	}
-	if len(results) != 1 {
-		return errIncorrectResultsCount(results)
-	}
-	return nil
+
+	return expectResultsCount(results, 1)
 }
 
 type httpPostMultipartBody struct{}
@@ -217,8 +350,7 @@ func (h *httpPostMultipartBody) Execute(filePath string) error {
 	router := httprouter.New()
 	var routerErr error
 
-	router.POST("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
-		httpDebugRequestDump(r)
+	router.POST("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
 		if err := r.ParseMultipartForm(1 * 1024); err != nil {
 			routerErr = err
 			return
@@ -236,21 +368,19 @@ func (h *httpPostMultipartBody) Execute(filePath string) error {
 		if strings.EqualFold(password[0], "nuclei") && strings.EqualFold(file[0].Filename, "username") {
 			fmt.Fprintf(w, "This is test post-multipart matcher text")
 		}
-	}))
+	})
 	ts := httptest.NewServer(router)
 	defer ts.Close()
 
-	results, err := testutils.RunNucleiAndGetResults(filePath, ts.URL, debug)
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
 	if err != nil {
 		return err
 	}
 	if routerErr != nil {
 		return routerErr
 	}
-	if len(results) != 1 {
-		return errIncorrectResultsCount(results)
-	}
-	return nil
+
+	return expectResultsCount(results, 1)
 }
 
 type httpRawDynamicExtractor struct{}
@@ -260,8 +390,7 @@ func (h *httpRawDynamicExtractor) Execute(filePath string) error {
 	router := httprouter.New()
 	var routerErr error
 
-	router.POST("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
-		httpDebugRequestDump(r)
+	router.POST("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
 		if err := r.ParseForm(); err != nil {
 			routerErr = err
 			return
@@ -269,27 +398,24 @@ func (h *httpRawDynamicExtractor) Execute(filePath string) error {
 		if strings.EqualFold(r.Form.Get("testing"), "parameter") {
 			fmt.Fprintf(w, "Token: 'nuclei'")
 		}
-	}))
-	router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
-		httpDebugRequestDump(r)
+	})
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
 		if strings.EqualFold(r.URL.Query().Get("username"), "nuclei") {
 			fmt.Fprintf(w, "Test is test-dynamic-extractor-raw matcher text")
 		}
-	}))
+	})
 	ts := httptest.NewServer(router)
 	defer ts.Close()
 
-	results, err := testutils.RunNucleiAndGetResults(filePath, ts.URL, debug)
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
 	if err != nil {
 		return err
 	}
 	if routerErr != nil {
 		return routerErr
 	}
-	if len(results) != 1 {
-		return errIncorrectResultsCount(results)
-	}
-	return nil
+
+	return expectResultsCount(results, 1)
 }
 
 type httpRawGetQuery struct{}
@@ -299,26 +425,23 @@ func (h *httpRawGetQuery) Execute(filePath string) error {
 	router := httprouter.New()
 	var routerErr error
 
-	router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
-		httpDebugRequestDump(r)
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
 		if strings.EqualFold(r.URL.Query().Get("test"), "nuclei") {
 			fmt.Fprintf(w, "Test is test raw-get-query-matcher text")
 		}
-	}))
+	})
 	ts := httptest.NewServer(router)
 	defer ts.Close()
 
-	results, err := testutils.RunNucleiAndGetResults(filePath, ts.URL, debug)
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
 	if err != nil {
 		return err
 	}
 	if routerErr != nil {
 		return routerErr
 	}
-	if len(results) != 1 {
-		return errIncorrectResultsCount(results)
-	}
-	return nil
+
+	return expectResultsCount(results, 1)
 }
 
 type httpRawGet struct{}
@@ -328,25 +451,21 @@ func (h *httpRawGet) Execute(filePath string) error {
 	router := httprouter.New()
 	var routerErr error
 
-	router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
-		httpDebugRequestDump(r)
-
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
 		fmt.Fprintf(w, "Test is test raw-get-matcher text")
-	}))
+	})
 	ts := httptest.NewServer(router)
 	defer ts.Close()
 
-	results, err := testutils.RunNucleiAndGetResults(filePath, ts.URL, debug)
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
 	if err != nil {
 		return err
 	}
 	if routerErr != nil {
 		return routerErr
 	}
-	if len(results) != 1 {
-		return errIncorrectResultsCount(results)
-	}
-	return nil
+
+	return expectResultsCount(results, 1)
 }
 
 type httpRawPayload struct{}
@@ -356,8 +475,7 @@ func (h *httpRawPayload) Execute(filePath string) error {
 	router := httprouter.New()
 	var routerErr error
 
-	router.POST("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
-		httpDebugRequestDump(r)
+	router.POST("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
 		if err := r.ParseForm(); err != nil {
 			routerErr = err
 			return
@@ -368,21 +486,19 @@ func (h *httpRawPayload) Execute(filePath string) error {
 		if strings.EqualFold(r.Form.Get("username"), "test") && (strings.EqualFold(r.Form.Get("password"), "nuclei") || strings.EqualFold(r.Form.Get("password"), "guest")) {
 			fmt.Fprintf(w, "Test is raw-payload matcher text")
 		}
-	}))
+	})
 	ts := httptest.NewServer(router)
 	defer ts.Close()
 
-	results, err := testutils.RunNucleiAndGetResults(filePath, ts.URL, debug)
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
 	if err != nil {
 		return err
 	}
 	if routerErr != nil {
 		return routerErr
 	}
-	if len(results) != 2 {
-		return errIncorrectResultsCount(results)
-	}
-	return nil
+
+	return expectResultsCount(results, 2)
 }
 
 type httpRawPostBody struct{}
@@ -392,8 +508,7 @@ func (h *httpRawPostBody) Execute(filePath string) error {
 	router := httprouter.New()
 	var routerErr error
 
-	router.POST("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
-		httpDebugRequestDump(r)
+	router.POST("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
 		if err := r.ParseForm(); err != nil {
 			routerErr = err
 			return
@@ -401,21 +516,19 @@ func (h *httpRawPostBody) Execute(filePath string) error {
 		if strings.EqualFold(r.Form.Get("username"), "test") && strings.EqualFold(r.Form.Get("password"), "nuclei") {
 			fmt.Fprintf(w, "Test is test raw-post-body-matcher text")
 		}
-	}))
+	})
 	ts := httptest.NewServer(router)
 	defer ts.Close()
 
-	results, err := testutils.RunNucleiAndGetResults(filePath, ts.URL, debug)
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
 	if err != nil {
 		return err
 	}
 	if routerErr != nil {
 		return routerErr
 	}
-	if len(results) != 1 {
-		return errIncorrectResultsCount(results)
-	}
-	return nil
+
+	return expectResultsCount(results, 1)
 }
 
 type httpRawCookieReuse struct{}
@@ -425,8 +538,7 @@ func (h *httpRawCookieReuse) Execute(filePath string) error {
 	router := httprouter.New()
 	var routerErr error
 
-	router.POST("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
-		httpDebugRequestDump(r)
+	router.POST("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
 		if err := r.ParseForm(); err != nil {
 			routerErr = err
 			return
@@ -434,9 +546,8 @@ func (h *httpRawCookieReuse) Execute(filePath string) error {
 		if strings.EqualFold(r.Form.Get("testing"), "parameter") {
 			http.SetCookie(w, &http.Cookie{Name: "nuclei", Value: "test"})
 		}
-	}))
-	router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
-		httpDebugRequestDump(r)
+	})
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
 		if err := r.ParseForm(); err != nil {
 			routerErr = err
 			return
@@ -450,21 +561,19 @@ func (h *httpRawCookieReuse) Execute(filePath string) error {
 		if strings.EqualFold(cookie.Value, "test") {
 			fmt.Fprintf(w, "Test is test-cookie-reuse matcher text")
 		}
-	}))
+	})
 	ts := httptest.NewServer(router)
 	defer ts.Close()
 
-	results, err := testutils.RunNucleiAndGetResults(filePath, ts.URL, debug)
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
 	if err != nil {
 		return err
 	}
 	if routerErr != nil {
 		return routerErr
 	}
-	if len(results) != 1 {
-		return errIncorrectResultsCount(results)
-	}
-	return nil
+
+	return expectResultsCount(results, 1)
 }
 
 type httpRawUnsafeRequest struct{}
@@ -473,23 +582,21 @@ type httpRawUnsafeRequest struct{}
 func (h *httpRawUnsafeRequest) Execute(filePath string) error {
 	var routerErr error
 
-	ts := testutils.NewTCPServer(func(conn net.Conn) {
+	ts := testutils.NewTCPServer(nil, defaultStaticPort, func(conn net.Conn) {
 		defer conn.Close()
 		_, _ = conn.Write([]byte("HTTP/1.1 200 OK\r\nConnection: close\r\nContent-Length: 36\r\nContent-Type: text/plain; charset=utf-8\r\n\r\nThis is test raw-unsafe-matcher test"))
 	})
 	defer ts.Close()
 
-	results, err := testutils.RunNucleiAndGetResults(filePath, "http://"+ts.URL, debug)
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "http://"+ts.URL, debug)
 	if err != nil {
 		return err
 	}
 	if routerErr != nil {
 		return routerErr
 	}
-	if len(results) != 1 {
-		return errIncorrectResultsCount(results)
-	}
-	return nil
+
+	return expectResultsCount(results, 1)
 }
 
 type httpRequestCondition struct{}
@@ -499,26 +606,306 @@ func (h *httpRequestCondition) Execute(filePath string) error {
 	router := httprouter.New()
 	var routerErr error
 
-	router.GET("/200", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
-		httpDebugRequestDump(r)
+	router.GET("/200", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
 		w.WriteHeader(200)
-	}))
-	router.GET("/400", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
-		httpDebugRequestDump(r)
+	})
+	router.GET("/400", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
 		w.WriteHeader(400)
-	}))
+	})
 	ts := httptest.NewServer(router)
 	defer ts.Close()
 
-	results, err := testutils.RunNucleiAndGetResults(filePath, ts.URL, debug)
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
 	if err != nil {
 		return err
 	}
 	if routerErr != nil {
 		return routerErr
 	}
-	if len(results) != 1 {
-		return errIncorrectResultsCount(results)
+
+	return expectResultsCount(results, 1)
+}
+
+type httpRequestSelContained struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *httpRequestSelContained) Execute(filePath string) error {
+	router := httprouter.New()
+	var routerErr error
+
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		_, _ = w.Write([]byte("This is self-contained response"))
+	})
+	server := &http.Server{
+		Addr:    fmt.Sprintf("localhost:%d", defaultStaticPort),
+		Handler: router,
+	}
+	go func() {
+		_ = server.ListenAndServe()
+	}()
+	defer server.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "", debug)
+	if err != nil {
+		return err
+	}
+	if routerErr != nil {
+		return routerErr
+	}
+
+	return expectResultsCount(results, 1)
+}
+
+type httpGetCaseInsensitive struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *httpGetCaseInsensitive) Execute(filePath string) error {
+	router := httprouter.New()
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		fmt.Fprintf(w, "THIS IS TEST MATCHER TEXT")
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 1)
+}
+
+type httpGetCaseInsensitiveCluster struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *httpGetCaseInsensitiveCluster) Execute(filesPath string) error {
+	router := httprouter.New()
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		fmt.Fprintf(w, "This is test matcher text")
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+
+	files := strings.Split(filesPath, ",")
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(files[0], ts.URL, debug, "-t", files[1])
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 2)
+}
+
+type httpGetRedirectsChainHeaders struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *httpGetRedirectsChainHeaders) Execute(filePath string) error {
+	router := httprouter.New()
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		http.Redirect(w, r, "/redirected", http.StatusFound)
+	})
+	router.GET("/redirected", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		w.Header().Set("Secret", "TestRedirectHeaderMatch")
+		http.Redirect(w, r, "/final", http.StatusFound)
+	})
+	router.GET("/final", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		_, _ = w.Write([]byte("ok"))
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 1)
+}
+
+type httpRaceSimple struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *httpRaceSimple) Execute(filePath string) error {
+	router := httprouter.New()
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		w.WriteHeader(http.StatusOK)
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
+	if err != nil {
+		return err
+	}
+	return expectResultsCount(results, 10)
+}
+
+type httpRaceMultiple struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *httpRaceMultiple) Execute(filePath string) error {
+	router := httprouter.New()
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		w.WriteHeader(http.StatusOK)
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
+	if err != nil {
+		return err
+	}
+	return expectResultsCount(results, 5)
+}
+
+type httpStopAtFirstMatch struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *httpStopAtFirstMatch) Execute(filePath string) error {
+	router := httprouter.New()
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		fmt.Fprintf(w, "This is test")
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 1)
+}
+
+type httpStopAtFirstMatchWithExtractors struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *httpStopAtFirstMatchWithExtractors) Execute(filePath string) error {
+	router := httprouter.New()
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		fmt.Fprintf(w, "This is test")
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 2)
+}
+
+type httpVariables struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *httpVariables) Execute(filePath string) error {
+	router := httprouter.New()
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		fmt.Fprintf(w, "%s\n%s", r.Header.Get("Test"), r.Header.Get("Another"))
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 1)
+}
+
+type customCLISNI struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *customCLISNI) Execute(filePath string) error {
+	router := httprouter.New()
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		if r.TLS.ServerName == "test" {
+			_, _ = w.Write([]byte("test-ok"))
+		} else {
+			_, _ = w.Write([]byte("test-ko"))
+		}
+	})
+	ts := httptest.NewTLSServer(router)
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug, "-sni", "test")
+	if err != nil {
+		return err
+	}
+	return expectResultsCount(results, 1)
+}
+
+type httpSniAnnotation struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *httpSniAnnotation) Execute(filePath string) error {
+	router := httprouter.New()
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		if r.TLS.ServerName == "test" {
+			_, _ = w.Write([]byte("test-ok"))
+		} else {
+			_, _ = w.Write([]byte("test-ko"))
+		}
+	})
+	ts := httptest.NewTLSServer(router)
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
+	if err != nil {
+		return err
+	}
+	return expectResultsCount(results, 1)
+}
+
+type httpRedirectMatchURL struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *httpRedirectMatchURL) Execute(filePath string) error {
+	router := httprouter.New()
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		http.Redirect(w, r, "/redirected", http.StatusFound)
+		_, _ = w.Write([]byte("This is test redirects matcher text"))
+	})
+	router.GET("/redirected", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		fmt.Fprintf(w, "This is test redirects matcher text")
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug, "-no-meta")
+	if err != nil {
+		return err
+	}
+
+	if err := expectResultsCount(results, 1); err != nil {
+		return err
+	}
+	if results[0] != fmt.Sprintf("%s/redirected", ts.URL) {
+		return fmt.Errorf("mismatched url found: %s", results[0])
 	}
 	return nil
 }
+
+type customCLISNIUnsafe struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *customCLISNIUnsafe) Execute(filePath string) error {
+	router := httprouter.New()
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		if r.TLS.ServerName == "test" {
+			_, _ = w.Write([]byte("test-ok"))
+		} else {
+			_, _ = w.Write([]byte("test-ko"))
+		}
+	})
+	ts := httptest.NewTLSServer(router)
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug, "-sni", "test")
+	if err != nil {
+		return err
+	}
+	return expectResultsCount(results, 1)
+}

v2/cmd/integration-test/integration-test.go

@@ -6,49 +6,106 @@ import (
 	"strings"
 
 	"github.com/logrusorgru/aurora"
-	"github.com/projectdiscovery/nuclei/v2/internal/testutils"
+
+	"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
 )
 
 var (
-	debug      = os.Getenv("DEBUG") == "true"
-	customTest = os.Getenv("TEST")
-	protocol   = os.Getenv("PROTO")
+	debug        = os.Getenv("DEBUG") == "true"
+	githubAction = os.Getenv("GH_ACTION") == "true"
+	customTests  = os.Getenv("TESTS")
 
-	errored = false
+	success = aurora.Green("[✓]").String()
+	failed  = aurora.Red("[✘]").String()
+
+	protocolTests = map[string]map[string]testutils.TestCase{
+		"http":          httpTestcases,
+		"network":       networkTestcases,
+		"dns":           dnsTestCases,
+		"workflow":      workflowTestcases,
+		"loader":        loaderTestcases,
+		"websocket":     websocketTestCases,
+		"headless":      headlessTestcases,
+		"whois":         whoisTestCases,
+		"ssl":           sslTestcases,
+		"code":          codeTestcases,
+		"templatesPath": templatesPathTestCases,
+		"templatesDir":  templatesDirTestCases,
+		"file":          fileTestcases,
+	}
 )
 
 func main() {
-	success := aurora.Green("[✓]").String()
-	failed := aurora.Red("[✘]").String()
+	failedTestTemplatePaths := runTests(toMap(toSlice(customTests)))
 
-	protocolTests := map[string]map[string]testutils.TestCase{
-		"http":     httpTestcases,
-		"network":  networkTestcases,
-		"dns":      dnsTestCases,
-		"workflow": workflowTestcases,
-	}
-	for proto, tests := range protocolTests {
-		if protocol == "" || protocol == proto {
-			fmt.Printf("Running test cases for \"%s\" protocol\n", aurora.Blue(proto))
-
-			for file, test := range tests {
-				if customTest != "" && !strings.Contains(file, customTest) {
-					continue // only run tests user asked
-				}
-				if err := test.Execute(file); err != nil {
-					fmt.Fprintf(os.Stderr, "%s Test \"%s\" failed: %s\n", failed, file, err)
-					errored = true
-				} else {
-					fmt.Printf("%s Test \"%s\" passed!\n", success, file)
-				}
-			}
+	if len(failedTestTemplatePaths) > 0 {
+		if githubAction {
+			debug = true
+			fmt.Println("::group::Failed integration tests in debug mode")
+			_ = runTests(failedTestTemplatePaths)
+			fmt.Println("::endgroup::")
 		}
-	}
-	if errored {
+
 		os.Exit(1)
 	}
 }
 
-func errIncorrectResultsCount(results []string) error {
-	return fmt.Errorf("incorrect number of results %s", strings.Join(results, "\n\t"))
+func runTests(customTemplatePaths map[string]struct{}) map[string]struct{} {
+	failedTestTemplatePaths := map[string]struct{}{}
+
+	for proto, testCases := range protocolTests {
+		if len(customTemplatePaths) == 0 {
+			fmt.Printf("Running test cases for %q protocol\n", aurora.Blue(proto))
+		}
+
+		for templatePath, testCase := range testCases {
+			if len(customTemplatePaths) == 0 || contains(customTemplatePaths, templatePath) {
+				if err, failedTemplatePath := execute(testCase, templatePath); err != nil {
+					failedTestTemplatePaths[failedTemplatePath] = struct{}{}
+				}
+			}
+		}
+	}
+
+	return failedTestTemplatePaths
+}
+
+func execute(testCase testutils.TestCase, templatePath string) (error, string) {
+	if err := testCase.Execute(templatePath); err != nil {
+		_, _ = fmt.Fprintf(os.Stderr, "%s Test \"%s\" failed: %s\n", failed, templatePath, err)
+		return err, templatePath
+	}
+
+	fmt.Printf("%s Test \"%s\" passed!\n", success, templatePath)
+	return nil, ""
+}
+
+func expectResultsCount(results []string, expectedNumber int) error {
+	if len(results) != expectedNumber {
+		return fmt.Errorf("incorrect number of results: %d (actual) vs %d (expected) \nResults:\n\t%s\n", len(results), expectedNumber, strings.Join(results, "\n\t"))
+	}
+	return nil
+}
+
+func toSlice(value string) []string {
+	if strings.TrimSpace(value) == "" {
+		return []string{}
+	}
+
+	return strings.Split(value, ",")
+}
+
+func toMap(slice []string) map[string]struct{} {
+	result := make(map[string]struct{}, len(slice))
+	for _, value := range slice {
+		if _, ok := result[value]; !ok {
+			result[value] = struct{}{}
+		}
+	}
+	return result
+}
+
+func contains(input map[string]struct{}, value string) bool {
+	_, ok := input[value]
+	return ok
 }

v2/cmd/integration-test/loader.go

@@ -0,0 +1,171 @@
+package main
+
+import (
+	"fmt"
+	"io/ioutil"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"strings"
+
+	"github.com/julienschmidt/httprouter"
+
+	"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
+)
+
+var loaderTestcases = map[string]testutils.TestCase{
+	"loader/template-list.yaml":             &remoteTemplateList{},
+	"loader/workflow-list.yaml":             &remoteWorkflowList{},
+	"loader/nonexistent-template-list.yaml": &nonExistentTemplateList{},
+	"loader/nonexistent-workflow-list.yaml": &nonExistentWorkflowList{},
+	"loader/template-list-not-allowed.yaml": &remoteTemplateListNotAllowed{},
+}
+
+type remoteTemplateList struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *remoteTemplateList) Execute(templateList string) error {
+	router := httprouter.New()
+
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		fmt.Fprintf(w, "This is test matcher text")
+		if strings.EqualFold(r.Header.Get("test"), "nuclei") {
+			fmt.Fprintf(w, "This is test headers matcher text")
+		}
+	})
+
+	router.GET("/template_list", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		file, err := os.ReadFile(templateList)
+		if err != nil {
+			w.WriteHeader(500)
+		}
+		_, err = w.Write(file)
+		if err != nil {
+			w.WriteHeader(500)
+		}
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+
+	configFileData := `remote-template-domain: [ "` + ts.Listener.Addr().String() + `" ]`
+	err := ioutil.WriteFile("test-config.yaml", []byte(configFileData), os.ModePerm)
+	if err != nil {
+		return err
+	}
+	defer os.Remove("test-config.yaml")
+
+	results, err := testutils.RunNucleiBareArgsAndGetResults(debug, "-target", ts.URL, "-tu", ts.URL+"/template_list", "-config", "test-config.yaml")
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 2)
+}
+
+type remoteTemplateListNotAllowed struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *remoteTemplateListNotAllowed) Execute(templateList string) error {
+	router := httprouter.New()
+
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		fmt.Fprintf(w, "This is test matcher text")
+		if strings.EqualFold(r.Header.Get("test"), "nuclei") {
+			fmt.Fprintf(w, "This is test headers matcher text")
+		}
+	})
+
+	router.GET("/template_list", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		file, err := os.ReadFile(templateList)
+		if err != nil {
+			w.WriteHeader(500)
+		}
+		_, err = w.Write(file)
+		if err != nil {
+			w.WriteHeader(500)
+		}
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+
+	_, err := testutils.RunNucleiBareArgsAndGetResults(debug, "-target", ts.URL, "-tu", ts.URL+"/template_list")
+	if err == nil {
+		return fmt.Errorf("expected error for not allowed remote template list url")
+	}
+
+	return nil
+
+}
+
+type remoteWorkflowList struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *remoteWorkflowList) Execute(workflowList string) error {
+	router := httprouter.New()
+
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		fmt.Fprintf(w, "This is test matcher text")
+		if strings.EqualFold(r.Header.Get("test"), "nuclei") {
+			fmt.Fprintf(w, "This is test headers matcher text")
+		}
+	})
+
+	router.GET("/workflow_list", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+		file, err := os.ReadFile(workflowList)
+		if err != nil {
+			w.WriteHeader(500)
+		}
+		_, err = w.Write(file)
+		if err != nil {
+			w.WriteHeader(500)
+		}
+	})
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+
+	configFileData := `remote-template-domain: [ "` + ts.Listener.Addr().String() + `" ]`
+	err := ioutil.WriteFile("test-config.yaml", []byte(configFileData), os.ModePerm)
+	if err != nil {
+		return err
+	}
+	defer os.Remove("test-config.yaml")
+
+	results, err := testutils.RunNucleiBareArgsAndGetResults(debug, "-target", ts.URL, "-wu", ts.URL+"/workflow_list", "-config", "test-config.yaml")
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 3)
+}
+
+type nonExistentTemplateList struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *nonExistentTemplateList) Execute(nonExistingTemplateList string) error {
+	router := httprouter.New()
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+
+	_, err := testutils.RunNucleiBareArgsAndGetResults(debug, "-target", ts.URL, "-tu", ts.URL+"/404")
+	if err == nil {
+		return fmt.Errorf("expected error for nonexisting workflow url")
+	}
+
+	return nil
+}
+
+type nonExistentWorkflowList struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *nonExistentWorkflowList) Execute(nonExistingWorkflowList string) error {
+	router := httprouter.New()
+	ts := httptest.NewServer(router)
+	defer ts.Close()
+
+	_, err := testutils.RunNucleiBareArgsAndGetResults(debug, "-target", ts.URL, "-wu", ts.URL+"/404")
+	if err == nil {
+		return fmt.Errorf("expected error for nonexisting workflow url")
+	}
+
+	return nil
+}

v2/cmd/integration-test/network.go

@@ -3,22 +3,26 @@ package main
 import (
 	"net"
 
-	"github.com/projectdiscovery/nuclei/v2/internal/testutils"
+	"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
 )
 
 var networkTestcases = map[string]testutils.TestCase{
-	"network/basic.yaml":      &networkBasic{},
-	"network/hex.yaml":        &networkBasic{},
-	"network/multi-step.yaml": &networkMultiStep{},
+	"network/basic.yaml":          &networkBasic{},
+	"network/hex.yaml":            &networkBasic{},
+	"network/multi-step.yaml":     &networkMultiStep{},
+	"network/self-contained.yaml": &networkRequestSelContained{},
+	"network/variables.yaml":      &networkVariables{},
 }
 
+const defaultStaticPort = 5431
+
 type networkBasic struct{}
 
 // Execute executes a test case and returns an error if occurred
 func (h *networkBasic) Execute(filePath string) error {
 	var routerErr error
 
-	ts := testutils.NewTCPServer(func(conn net.Conn) {
+	ts := testutils.NewTCPServer(nil, defaultStaticPort, func(conn net.Conn) {
 		defer conn.Close()
 
 		data := make([]byte, 4)
@@ -32,17 +36,15 @@ func (h *networkBasic) Execute(filePath string) error {
 	})
 	defer ts.Close()
 
-	results, err := testutils.RunNucleiAndGetResults(filePath, ts.URL, debug)
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
 	if err != nil {
 		return err
 	}
 	if routerErr != nil {
 		return routerErr
 	}
-	if len(results) != 1 {
-		return errIncorrectResultsCount(results)
-	}
-	return nil
+
+	return expectResultsCount(results, 1)
 }
 
 type networkMultiStep struct{}
@@ -51,7 +53,7 @@ type networkMultiStep struct{}
 func (h *networkMultiStep) Execute(filePath string) error {
 	var routerErr error
 
-	ts := testutils.NewTCPServer(func(conn net.Conn) {
+	ts := testutils.NewTCPServer(nil, defaultStaticPort, func(conn net.Conn) {
 		defer conn.Close()
 
 		data := make([]byte, 5)
@@ -75,15 +77,74 @@ func (h *networkMultiStep) Execute(filePath string) error {
 	})
 	defer ts.Close()
 
-	results, err := testutils.RunNucleiAndGetResults(filePath, ts.URL, debug)
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
 	if err != nil {
 		return err
 	}
 	if routerErr != nil {
 		return routerErr
 	}
-	if len(results) != 1 {
-		return errIncorrectResultsCount(results)
+
+	var expectedResultsSize int
+	if debug {
+		expectedResultsSize = 3
+	} else {
+		expectedResultsSize = 1
 	}
-	return nil
+
+	return expectResultsCount(results, expectedResultsSize)
+}
+
+type networkRequestSelContained struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *networkRequestSelContained) Execute(filePath string) error {
+	var routerErr error
+
+	ts := testutils.NewTCPServer(nil, defaultStaticPort, func(conn net.Conn) {
+		defer conn.Close()
+
+		_, _ = conn.Write([]byte("Authentication successful"))
+	})
+	defer ts.Close()
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "", debug)
+	if err != nil {
+		return err
+	}
+	if routerErr != nil {
+		return routerErr
+	}
+
+	return expectResultsCount(results, 1)
+}
+
+type networkVariables struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *networkVariables) Execute(filePath string) error {
+	var routerErr error
+
+	ts := testutils.NewTCPServer(nil, defaultStaticPort, func(conn net.Conn) {
+		defer conn.Close()
+
+		data := make([]byte, 4)
+		if _, err := conn.Read(data); err != nil {
+			routerErr = err
+			return
+		}
+		if string(data) == "PING" {
+			_, _ = conn.Write([]byte("aGVsbG8="))
+		}
+	})
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
+	if err != nil {
+		return err
+	}
+	if routerErr != nil {
+		return routerErr
+	}
+
+	return expectResultsCount(results, 1)
 }

v2/cmd/integration-test/ssl.go

@@ -0,0 +1,99 @@
+package main
+
+import (
+	"crypto/tls"
+	"net"
+
+	"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
+)
+
+var sslTestcases = map[string]testutils.TestCase{
+	"ssl/basic.yaml":          &sslBasic{},
+	"ssl/basic-ztls.yaml":     &sslBasicZtls{},
+	"ssl/custom-cipher.yaml":  &sslCustomCipher{},
+	"ssl/custom-version.yaml": &sslCustomVersion{},
+}
+
+type sslBasic struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *sslBasic) Execute(filePath string) error {
+	ts := testutils.NewTCPServer(&tls.Config{}, defaultStaticPort, func(conn net.Conn) {
+		defer conn.Close()
+		data := make([]byte, 4)
+		if _, err := conn.Read(data); err != nil {
+			return
+		}
+	})
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 1)
+}
+
+type sslBasicZtls struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *sslBasicZtls) Execute(filePath string) error {
+	ts := testutils.NewTCPServer(&tls.Config{}, defaultStaticPort, func(conn net.Conn) {
+		defer conn.Close()
+		data := make([]byte, 4)
+		if _, err := conn.Read(data); err != nil {
+			return
+		}
+	})
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug, "-ztls")
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 1)
+}
+
+type sslCustomCipher struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *sslCustomCipher) Execute(filePath string) error {
+	ts := testutils.NewTCPServer(&tls.Config{CipherSuites: []uint16{tls.TLS_AES_128_GCM_SHA256}}, defaultStaticPort, func(conn net.Conn) {
+		defer conn.Close()
+		data := make([]byte, 4)
+		if _, err := conn.Read(data); err != nil {
+			return
+		}
+	})
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 1)
+}
+
+type sslCustomVersion struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *sslCustomVersion) Execute(filePath string) error {
+	ts := testutils.NewTCPServer(&tls.Config{MinVersion: tls.VersionTLS12, MaxVersion: tls.VersionTLS12}, defaultStaticPort, func(conn net.Conn) {
+		defer conn.Close()
+		data := make([]byte, 4)
+		if _, err := conn.Read(data); err != nil {
+			return
+		}
+	})
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 1)
+}

v2/cmd/integration-test/template-dir.go

@@ -0,0 +1,34 @@
+package main
+
+import (
+	"os"
+
+	"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
+)
+
+func getTemplatesDir() string {
+	temp := os.TempDir()
+	return temp
+}
+
+var templatesDirTestCases = map[string]testutils.TestCase{
+	"dns/cname-fingerprint.yaml": &templateDirWithTargetTest{},
+}
+
+type templateDirWithTargetTest struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *templateDirWithTargetTest) Execute(filePath string) error {
+	defer os.RemoveAll(getTemplatesDir())
+
+	var routerErr error
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "8x8exch02.8x8.com", debug, "-ud", getTemplatesDir())
+	if err != nil {
+		return err
+	}
+	if routerErr != nil {
+		return routerErr
+	}
+	return expectResultsCount(results, 1)
+}

v2/cmd/integration-test/template-path.go

@@ -0,0 +1,72 @@
+package main
+
+import (
+	"fmt"
+	"strings"
+
+	"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
+	"github.com/projectdiscovery/nuclei/v2/pkg/utils"
+)
+
+func getTemplatePath() string {
+	templatePath, _ := utils.GetDefaultTemplatePath()
+	return templatePath
+}
+
+var templatesPathTestCases = map[string]testutils.TestCase{
+	//template folder path issue
+	"http/get.yaml": &folderPathTemplateTest{},
+	//cwd
+	"./dns/cname-fingerprint.yaml": &cwdTemplateTest{},
+	//relative path
+	"dns/cname-fingerprint.yaml": &relativePathTemplateTest{},
+	//absolute path
+	fmt.Sprintf("%v/dns/cname-fingerprint.yaml", getTemplatePath()): &absolutePathTemplateTest{},
+}
+
+type cwdTemplateTest struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *cwdTemplateTest) Execute(filePath string) error {
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "8x8exch02.8x8.com", debug)
+	if err != nil {
+		return err
+	}
+	return expectResultsCount(results, 1)
+}
+
+type relativePathTemplateTest struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *relativePathTemplateTest) Execute(filePath string) error {
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "8x8exch02.8x8.com", debug)
+	if err != nil {
+		return err
+	}
+	return expectResultsCount(results, 1)
+}
+
+type absolutePathTemplateTest struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *absolutePathTemplateTest) Execute(filePath string) error {
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "8x8exch02.8x8.com", debug)
+	if err != nil {
+		return err
+	}
+	return expectResultsCount(results, 1)
+}
+
+type folderPathTemplateTest struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *folderPathTemplateTest) Execute(filePath string) error {
+	results, err := testutils.RunNucleiBinaryAndGetCombinedOutput(debug, []string{"-t", filePath, "-target", "http://example.com"})
+	if err != nil {
+		return err
+	}
+	if strings.Contains(results, "installing") {
+		return fmt.Errorf("couldn't find template path,re-installing")
+	}
+	return nil
+}

v2/cmd/integration-test/websocket.go

@@ -0,0 +1,107 @@
+package main
+
+import (
+	"net"
+	"strings"
+
+	"github.com/gobwas/ws/wsutil"
+
+	"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
+)
+
+var websocketTestCases = map[string]testutils.TestCase{
+	"websocket/basic.yaml":    &websocketBasic{},
+	"websocket/cswsh.yaml":    &websocketCswsh{},
+	"websocket/no-cswsh.yaml": &websocketNoCswsh{},
+	"websocket/path.yaml":     &websocketWithPath{},
+}
+
+type websocketBasic struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *websocketBasic) Execute(filePath string) error {
+	connHandler := func(conn net.Conn) {
+		for {
+			msg, op, _ := wsutil.ReadClientData(conn)
+			if string(msg) != "hello" {
+				return
+			}
+			_ = wsutil.WriteServerMessage(conn, op, []byte("world"))
+		}
+	}
+	originValidate := func(origin string) bool {
+		return true
+	}
+	ts := testutils.NewWebsocketServer("", connHandler, originValidate)
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, strings.ReplaceAll(ts.URL, "http", "ws"), debug)
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 1)
+}
+
+type websocketCswsh struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *websocketCswsh) Execute(filePath string) error {
+	connHandler := func(conn net.Conn) {
+
+	}
+	originValidate := func(origin string) bool {
+		return true
+	}
+	ts := testutils.NewWebsocketServer("", connHandler, originValidate)
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, strings.ReplaceAll(ts.URL, "http", "ws"), debug)
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 1)
+}
+
+type websocketNoCswsh struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *websocketNoCswsh) Execute(filePath string) error {
+	connHandler := func(conn net.Conn) {
+
+	}
+	originValidate := func(origin string) bool {
+		return origin == "https://google.com"
+	}
+	ts := testutils.NewWebsocketServer("", connHandler, originValidate)
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, strings.ReplaceAll(ts.URL, "http", "ws"), debug)
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 0)
+}
+
+type websocketWithPath struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *websocketWithPath) Execute(filePath string) error {
+	connHandler := func(conn net.Conn) {
+
+	}
+	originValidate := func(origin string) bool {
+		return origin == "https://google.com"
+	}
+	ts := testutils.NewWebsocketServer("/test", connHandler, originValidate)
+	defer ts.Close()
+
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, strings.ReplaceAll(ts.URL, "http", "ws"), debug)
+	if err != nil {
+		return err
+	}
+
+	return expectResultsCount(results, 0)
+}

v2/cmd/integration-test/whois.go

@@ -0,0 +1,20 @@
+package main
+
+import (
+	"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
+)
+
+var whoisTestCases = map[string]testutils.TestCase{
+	"whois/basic.yaml": &whoisBasic{},
+}
+
+type whoisBasic struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *whoisBasic) Execute(filePath string) error {
+	results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "https://example.com", debug)
+	if err != nil {
+		return err
+	}
+	return expectResultsCount(results, 1)
+}

v2/cmd/integration-test/workflow.go

@@ -7,7 +7,7 @@ import (
 
 	"github.com/julienschmidt/httprouter"
 
-	"github.com/projectdiscovery/nuclei/v2/internal/testutils"
+	"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
 )
 
 var workflowTestcases = map[string]testutils.TestCase{
@@ -23,7 +23,6 @@ type workflowBasic struct{}
 func (h *workflowBasic) Execute(filePath string) error {
 	router := httprouter.New()
 	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
-		httpDebugRequestDump(r)
 		fmt.Fprintf(w, "This is test matcher text")
 	})
 	ts := httptest.NewServer(router)
@@ -33,10 +32,8 @@ func (h *workflowBasic) Execute(filePath string) error {
 	if err != nil {
 		return err
 	}
-	if len(results) != 2 {
-		return errIncorrectResultsCount(results)
-	}
-	return nil
+
+	return expectResultsCount(results, 2)
 }
 
 type workflowConditionMatched struct{}
@@ -45,7 +42,6 @@ type workflowConditionMatched struct{}
 func (h *workflowConditionMatched) Execute(filePath string) error {
 	router := httprouter.New()
 	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
-		httpDebugRequestDump(r)
 		fmt.Fprintf(w, "This is test matcher text")
 	})
 	ts := httptest.NewServer(router)
@@ -55,10 +51,8 @@ func (h *workflowConditionMatched) Execute(filePath string) error {
 	if err != nil {
 		return err
 	}
-	if len(results) != 1 {
-		return errIncorrectResultsCount(results)
-	}
-	return nil
+
+	return expectResultsCount(results, 1)
 }
 
 type workflowConditionUnmatch struct{}
@@ -67,7 +61,6 @@ type workflowConditionUnmatch struct{}
 func (h *workflowConditionUnmatch) Execute(filePath string) error {
 	router := httprouter.New()
 	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
-		httpDebugRequestDump(r)
 		fmt.Fprintf(w, "This is test matcher text")
 	})
 	ts := httptest.NewServer(router)
@@ -77,10 +70,8 @@ func (h *workflowConditionUnmatch) Execute(filePath string) error {
 	if err != nil {
 		return err
 	}
-	if len(results) != 0 {
-		return errIncorrectResultsCount(results)
-	}
-	return nil
+
+	return expectResultsCount(results, 0)
 }
 
 type workflowMatcherName struct{}
@@ -89,7 +80,6 @@ type workflowMatcherName struct{}
 func (h *workflowMatcherName) Execute(filePath string) error {
 	router := httprouter.New()
 	router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
-		httpDebugRequestDump(r)
 		fmt.Fprintf(w, "This is test matcher text")
 	})
 	ts := httptest.NewServer(router)
@@ -99,8 +89,6 @@ func (h *workflowMatcherName) Execute(filePath string) error {
 	if err != nil {
 		return err
 	}
-	if len(results) != 1 {
-		return errIncorrectResultsCount(results)
-	}
-	return nil
+
+	return expectResultsCount(results, 1)
 }

v2/cmd/nuclei/issue-tracker-config.yaml

@@ -1,57 +1,57 @@
-# to specify which severities should be reported
 #allow-list:
-#  severity: critical, high
-# to specify which severities should be excluded from reporting
+#  severity: high, critical
 #deny-list:
-#  severity: info, low, medium
-
-# github contains configuration options for github issue tracker
-#github: 
-#  # base-url is the optional self-hosted github application url
-#  base-url: ""
-#  # username is the username of the github user
-#  username: ""
-#  # owner is the owner name of the repository for issues.
-#  owner: ""
-#  # token is the token for github account.
-#  token: ""
-#  # project-name is the name of the repository.
-#  project-name: ""
+#  severity: low
+#
+# GitHub contains configuration options for GitHub issue tracker
+#github:
+#  # base-url is the optional self-hosted GitHub application url
+#  base-url: https://localhost:8443/github
+#  # username is the username of the GitHub user
+#  username: test-username
+#  # owner is the owner name of the repository for issues
+#  owner: test-owner
+#  # token is the token for GitHub account
+#  token: test-token
+#  # project-name is the name of the repository
+#  project-name: test-project
 #  # issue-label is the label of the created issue type
-#  issue-label: ""
-
-# gitlab contains configuration options for gitlab issue tracker
-#gitlab: 
-#  # base-url is the optional self-hosted gitlab application url
-#  base-url: ""
-#  # username is the username of the gitlab user
-#  username: ""
-#  # token is the token for gitlab account.
-#  token: ""
-#  # project-id is the ID of the repository.
-#  project-id: ""
+#  issue-label: bug
+#
+# GitLab contains configuration options for gitlab issue tracker
+#gitlab:
+#  # base-url is the optional self-hosted GitLab application url
+#  base-url: https://localhost:8443/gitlab
+#  # username is the username of the GitLab user
+#  username: test-username
+#  # token is the token for GitLab account
+#  token: test-token
+#  # project-name is the name/id of the project(repository)
+#  project-name: "1234"
 #  # issue-label is the label of the created issue type
-#  issue-label: ""
-
-# jira contains configuration options for jira issue tracker
+#  issue-label: bug
+#
+# Jira contains configuration options for Jira issue tracker
 #jira:
 #  # cloud is the boolean which tells if Jira instance is running in the cloud or on-prem version is used
 #  cloud: true
 #  # update-existing is the boolean which tells if the existing, opened issue should be updated or new one should be created
 #  update-existing: false
 #  # URL is the jira application url
-#  url: ""
-#  # account-id is the account-id of the jira user or username in case of on-prem Jira
-#  account-id: ""
-#  # email is the email of the user for jira instance
-#  email: ""
-#  # token is the token for jira instance or password in case of on-prem Jira
-#  token: ""
+#  url: https://localhost/jira
+#  # account-id is the account-id of the Jira user or username in case of on-prem Jira
+#  account-id: test-account-id
+#  # email is the email of the user for Jira instance
+#  email: [email protected]
+#  # token is the token for Jira instance or password in case of on-prem Jira
+#  token: test-token
 #  # project-name is the name of the project.
-#  project-name: ""
-#  # issue-type is the name of the created issue type
-#  issue-type: ""
-
+#  project-name: test-project-name
+#  # issue-type is the name of the created issue type (case sensitive)
+#  issue-type: Bug
+#  # SeverityAsLabel (optional) sends the severity as the label of the created issue
+#  severity-as-label: true
+#
 # elasticsearch contains configuration options for elasticsearch exporter
 #elasticsearch:
 #  # IP for elasticsearch instance

v2/cmd/nuclei/main.go

@@ -3,13 +3,21 @@ package main
 import (
 	"fmt"
 	"os"
+	"os/signal"
 	"path/filepath"
+	"time"
 
+	"github.com/projectdiscovery/fileutil"
 	"github.com/projectdiscovery/goflags"
 	"github.com/projectdiscovery/gologger"
+	"github.com/projectdiscovery/interactsh/pkg/client"
 	"github.com/projectdiscovery/nuclei/v2/internal/runner"
+	"github.com/projectdiscovery/nuclei/v2/pkg/catalog/config"
 	"github.com/projectdiscovery/nuclei/v2/pkg/model/types/severity"
+	"github.com/projectdiscovery/nuclei/v2/pkg/protocols/http"
+	templateTypes "github.com/projectdiscovery/nuclei/v2/pkg/templates/types"
 	"github.com/projectdiscovery/nuclei/v2/pkg/types"
+	"github.com/projectdiscovery/nuclei/v2/pkg/utils/monitor"
 )
 
 var (
@@ -18,10 +26,19 @@ var (
 )
 
 func main() {
+	if err := runner.ConfigureOptions(); err != nil {
+		gologger.Fatal().Msgf("Could not initialize options: %s\n", err)
+	}
+
 	readConfig()
 
 	runner.ParseOptions(options)
 
+	if options.HangMonitor {
+		cancel := monitor.NewStackMonitor(10 * time.Second)
+		defer cancel()
+	}
+
 	nucleiRunner, err := runner.New(options)
 	if err != nil {
 		gologger.Fatal().Msgf("Could not create runner: %s\n", err)
@@ -29,15 +46,41 @@ func main() {
 	if nucleiRunner == nil {
 		return
 	}
+
+	// Setup graceful exits
+	resumeFileName := types.DefaultResumeFilePath()
+	c := make(chan os.Signal, 1)
+	signal.Notify(c, os.Interrupt)
+	go func() {
+		for range c {
+			gologger.Info().Msgf("CTRL+C pressed: Exiting\n")
+			nucleiRunner.Close()
+			if options.ShouldSaveResume() {
+				gologger.Info().Msgf("Creating resume file: %s\n", resumeFileName)
+				err := nucleiRunner.SaveResumeConfig(resumeFileName)
+				if err != nil {
+					gologger.Error().Msgf("Couldn't create resume file: %s\n", err)
+				}
+			}
+			os.Exit(1)
+		}
+	}()
+
 	if err := nucleiRunner.RunEnumeration(); err != nil {
-		gologger.Fatal().Msgf("Could not run nuclei: %s\n", err)
+		if options.Validate {
+			gologger.Fatal().Msgf("Could not validate templates: %s\n", err)
+		} else {
+			gologger.Fatal().Msgf("Could not run nuclei: %s\n", err)
+		}
 	}
 	nucleiRunner.Close()
+	// on successful execution remove the resume file in case it exists
+	if fileutil.FileExists(resumeFileName) {
+		os.Remove(resumeFileName)
+	}
 }
 
 func readConfig() {
-	home, _ := os.UserHomeDir()
-	templatesDirectory := filepath.Join(home, "nuclei-templates")
 
 	flagSet := goflags.NewFlagSet()
 	flagSet.SetDescription(`Nuclei is a fast, template based vulnerability scanner focusing
@@ -50,136 +93,168 @@ on extensive configurability, massive extensibility and ease of use.`)
 	createGroup(flagSet, "input", "Target",
 		flagSet.StringSliceVarP(&options.Targets, "target", "u", []string{}, "target URLs/hosts to scan"),
 		flagSet.StringVarP(&options.TargetsFilePath, "list", "l", "", "path to file containing a list of target URLs/hosts to scan (one per line)"),
+		flagSet.StringVar(&options.Resume, "resume", "", "Resume scan using resume.cfg (clustering will be disabled)"),
 	)
 
 	createGroup(flagSet, "templates", "Templates",
-		flagSet.BoolVar(&options.TemplateList, "tl", false, "list all available templates"),
-
-		flagSet.StringSliceVarP(&options.Templates, "templates", "t", []string{}, "template or template directory paths to include in the scan"),
-		flagSet.StringSliceVarP(&options.Workflows, "workflows", "w", []string{}, "list of workflows to run"),
-
-		flagSet.BoolVarP(&options.NewTemplates, "new-templates", "nt", false, "run newly added templates only"),
+		flagSet.BoolVarP(&options.NewTemplates, "new-templates", "nt", false, "run only new templates added in latest nuclei-templates release"),
+		flagSet.BoolVarP(&options.AutomaticScan, "automatic-scan", "as", false, "automatic web scan using wappalyzer technology detection to tags mapping"),
+		flagSet.FileNormalizedOriginalStringSliceVarP(&options.Templates, "templates", "t", []string{}, "list of template or template directory to run (comma-separated, file)"),
+		flagSet.FileNormalizedOriginalStringSliceVarP(&options.TemplateURLs, "template-url", "tu", []string{}, "list of template urls to run (comma-separated, file)"),
+		flagSet.FileNormalizedOriginalStringSliceVarP(&options.Workflows, "workflows", "w", []string{}, "list of workflow or workflow directory to run (comma-separated, file)"),
+		flagSet.FileNormalizedOriginalStringSliceVarP(&options.WorkflowURLs, "workflow-url", "wu", []string{}, "list of workflow urls to run (comma-separated, file)"),
 		flagSet.BoolVar(&options.Validate, "validate", false, "validate the passed templates to nuclei"),
+		flagSet.BoolVar(&options.TemplateList, "tl", false, "list all available templates"),
+		flagSet.StringSliceVarConfigOnly(&options.RemoteTemplateDomainList, "remote-template-domain", []string{"api.nuclei.sh"}, "allowed domain list to load remote templates from"),
 	)
 
 	createGroup(flagSet, "filters", "Filtering",
-		flagSet.NormalizedStringSliceVar(&options.Tags, "tags", []string{}, "execute a subset of templates that contain the provided tags"),
-		flagSet.NormalizedStringSliceVar(&options.IncludeTags, "include-tags", []string{}, "tags from the default deny list that permit executing more intrusive templates"), // TODO show default deny list
-		flagSet.NormalizedStringSliceVarP(&options.ExcludeTags, "exclude-tags", "etags", []string{}, "exclude templates with the provided tags"),
-
-		flagSet.StringSliceVar(&options.IncludeTemplates, "include-templates", []string{}, "templates to be executed even if they are excluded either by default or configuration"),
-		flagSet.StringSliceVarP(&options.ExcludedTemplates, "exclude", "exclude-templates", []string{}, "template or template directory paths to exclude"),
-
-		flagSet.VarP(&options.Severities, "impact", "severity", fmt.Sprintf("Templates to run based on severity. Possible values: %s", severity.GetSupportedSeverities().String())),
-		flagSet.NormalizedStringSliceVar(&options.Author, "author", []string{}, "execute templates that are (co-)created by the specified authors"),
+		flagSet.FileNormalizedStringSliceVarP(&options.Authors, "author", "a", []string{}, "templates to run based on authors (comma-separated, file)"),
+		flagSet.FileNormalizedStringSliceVar(&options.Tags, "tags", []string{}, "templates to run based on tags (comma-separated, file)"),
+		flagSet.FileNormalizedStringSliceVarP(&options.ExcludeTags, "exclude-tags", "etags", []string{}, "templates to exclude based on tags (comma-separated, file)"),
+		flagSet.FileNormalizedStringSliceVarP(&options.IncludeTags, "include-tags", "itags", []string{}, "tags to be executed even if they are excluded either by default or configuration"), // TODO show default deny list
+		flagSet.FileNormalizedStringSliceVarP(&options.IncludeIds, "template-id", "id", []string{}, "templates to run based on template ids (comma-separated, file)"),
+		flagSet.FileNormalizedStringSliceVarP(&options.ExcludeIds, "exclude-id", "eid", []string{}, "templates to exclude based on template ids (comma-separated, file)"),
+		flagSet.FileNormalizedOriginalStringSliceVarP(&options.IncludeTemplates, "include-templates", "it", []string{}, "templates to be executed even if they are excluded either by default or configuration"),
+		flagSet.FileNormalizedOriginalStringSliceVarP(&options.ExcludedTemplates, "exclude-templates", "et", []string{}, "template or template directory to exclude (comma-separated, file)"),
+		flagSet.VarP(&options.Severities, "severity", "s", fmt.Sprintf("templates to run based on severity. Possible values: %s", severity.GetSupportedSeverities().String())),
+		flagSet.VarP(&options.ExcludeSeverities, "exclude-severity", "es", fmt.Sprintf("templates to exclude based on severity. Possible values: %s", severity.GetSupportedSeverities().String())),
+		flagSet.VarP(&options.Protocols, "type", "pt", fmt.Sprintf("templates to run based on protocol type. Possible values: %s", templateTypes.GetSupportedProtocolTypes())),
+		flagSet.VarP(&options.ExcludeProtocols, "exclude-type", "ept", fmt.Sprintf("templates to exclude based on protocol type. Possible values: %s", templateTypes.GetSupportedProtocolTypes())),
 	)
 
 	createGroup(flagSet, "output", "Output",
 		flagSet.StringVarP(&options.Output, "output", "o", "", "output file to write found issues/vulnerabilities"),
-
+		flagSet.BoolVarP(&options.StoreResponse, "store-resp", "sresp", false, "store all request/response passed through nuclei to output directory"),
+		flagSet.StringVarP(&options.StoreResponseDir, "store-resp-dir", "srd", runner.DefaultDumpTrafficOutputFolder, "store all request/response passed through nuclei to custom directory"),
 		flagSet.BoolVar(&options.Silent, "silent", false, "display findings only"),
-		flagSet.BoolVarP(&options.Verbose, "verbose", "v", false, "show verbose output"),
-		flagSet.BoolVar(&options.VerboseVerbose, "vv", false, "display extra verbose information"),
 		flagSet.BoolVarP(&options.NoColor, "no-color", "nc", false, "disable output content coloring (ANSI escape codes)"),
-
 		flagSet.BoolVar(&options.JSON, "json", false, "write output in JSONL(ines) format"),
 		flagSet.BoolVarP(&options.JSONRequests, "include-rr", "irr", false, "include request/response pairs in the JSONL output (for findings only)"),
-
-		flagSet.BoolVarP(&options.NoMeta, "no-meta", "nm", false, "don't display match metadata"),
-		flagSet.BoolVarP(&options.NoTimestamp, "no-timestamp", "nts", false, "don't display timestamp metadata in CLI output"),
-		flagSet.StringVarP(&options.ReportingDB, "report-db", "rdb", "", "local nuclei reporting database (always use this to persist report data)"),
-
-		flagSet.StringVarP(&options.DiskExportDirectory, "markdown-export", "me", "", "directory to export results in markdown format"),
+		flagSet.BoolVarP(&options.NoMeta, "no-meta", "nm", false, "disable printing result metadata in cli output"),
+		flagSet.BoolVarP(&options.NoTimestamp, "no-timestamp", "nts", false, "disable printing timestamp in cli output"),
+		flagSet.StringVarP(&options.ReportingDB, "report-db", "rdb", "", "nuclei reporting database (always use this to persist report data)"),
+		flagSet.BoolVarP(&options.MatcherStatus, "matcher-status", "ms", false, "display match failure status"),
+		flagSet.StringVarP(&options.MarkdownExportDirectory, "markdown-export", "me", "", "directory to export results in markdown format"),
 		flagSet.StringVarP(&options.SarifExport, "sarif-export", "se", "", "file to export results in SARIF format"),
 	)
 
 	createGroup(flagSet, "configs", "Configurations",
 		flagSet.StringVar(&cfgFile, "config", "", "path to the nuclei configuration file"),
+		flagSet.BoolVarP(&options.FollowRedirects, "follow-redirects", "fr", false, "enable following redirects for http templates"),
+		flagSet.IntVarP(&options.MaxRedirects, "max-redirects", "mr", 10, "max number of redirects to follow for http templates"),
+		flagSet.BoolVarP(&options.DisableRedirects, "disable-redirects", "dr", false, "disable redirects for http templates"),
 		flagSet.StringVarP(&options.ReportingConfig, "report-config", "rc", "", "nuclei reporting module configuration file"), // TODO merge into the config file or rename to issue-tracking
-
-		flagSet.StringSliceVarP(&options.CustomHeaders, "header", "H", []string{}, "custom headers in header:value format"),
-
-		flagSet.RuntimeMapVarP(&options.Vars, "var", "V", []string{}, "custom vars in var=value format"),
-
+		flagSet.FileStringSliceVarP(&options.CustomHeaders, "header", "H", []string{}, "custom header/cookie to include in all http request in header:value format (cli, file)"),
+		flagSet.RuntimeMapVarP(&options.Vars, "var", "V", []string{}, "custom vars in key=value format"),
 		flagSet.StringVarP(&options.ResolversFile, "resolvers", "r", "", "file containing resolver list for nuclei"),
-		flagSet.BoolVar(&options.SystemResolvers, "system-resolvers", false, "use system DNS resolving as error fallback"),
+		flagSet.BoolVarP(&options.SystemResolvers, "system-resolvers", "sr", false, "use system DNS resolving as error fallback"),
 		flagSet.BoolVar(&options.OfflineHTTP, "passive", false, "enable passive HTTP response processing mode"),
-		flagSet.BoolVar(&options.EnvironmentVariables, "env-vars", false, "enable environment variables support"),
+		flagSet.BoolVarP(&options.EnvironmentVariables, "env-vars", "ev", false, "enable environment variables to be used in template"),
+		flagSet.StringVarP(&options.ClientCertFile, "client-cert", "cc", "", "client certificate file (PEM-encoded) used for authenticating against scanned hosts"),
+		flagSet.StringVarP(&options.ClientKeyFile, "client-key", "ck", "", "client key file (PEM-encoded) used for authenticating against scanned hosts"),
+		flagSet.StringVarP(&options.ClientCAFile, "client-ca", "ca", "", "client certificate authority file (PEM-encoded) used for authenticating against scanned hosts"),
+		flagSet.BoolVarP(&options.ShowMatchLine, "show-match-line", "sml", false, "show match lines for file templates, works with extractors only"),
+		flagSet.BoolVar(&options.ZTLS, "ztls", false, "use ztls library with autofallback to standard one for tls13"),
+		flagSet.StringVar(&options.SNI, "sni", "", "tls sni hostname to use (default: input domain name)"),
 	)
 
 	createGroup(flagSet, "interactsh", "interactsh",
-		flagSet.BoolVar(&options.NoInteractsh, "no-interactsh", false, "disable interactsh server for OOB testing"),
-		flagSet.StringVar(&options.InteractshURL, "interactsh-url", "https://interact.sh", "interactsh server url for self-hosted instance"),
-		flagSet.StringVar(&options.InteractshToken, "interactsh-token", "", "authentication token for self-hosted interactsh server"),
+		flagSet.StringVarP(&options.InteractshURL, "interactsh-server", "iserver", "", fmt.Sprintf("interactsh server url for self-hosted instance (default: %s)", client.DefaultOptions.ServerURL)),
+		flagSet.StringVarP(&options.InteractshToken, "interactsh-token", "itoken", "", "authentication token for self-hosted interactsh server"),
 		flagSet.IntVar(&options.InteractionsCacheSize, "interactions-cache-size", 5000, "number of requests to keep in the interactions cache"),
 		flagSet.IntVar(&options.InteractionsEviction, "interactions-eviction", 60, "number of seconds to wait before evicting requests from cache"),
 		flagSet.IntVar(&options.InteractionsPollDuration, "interactions-poll-duration", 5, "number of seconds to wait before each interaction poll request"),
-		flagSet.IntVar(&options.InteractionsColldownPeriod, "interactions-cooldown-period", 5, "extra time for interaction polling before exiting"),
+		flagSet.IntVar(&options.InteractionsCoolDownPeriod, "interactions-cooldown-period", 5, "extra time for interaction polling before exiting"),
+		flagSet.BoolVarP(&options.NoInteractsh, "no-interactsh", "ni", false, "disable interactsh server for OAST testing, exclude OAST based templates"),
 	)
 
 	createGroup(flagSet, "rate-limit", "Rate-Limit",
 		flagSet.IntVarP(&options.RateLimit, "rate-limit", "rl", 150, "maximum number of requests to send per second"),
 		flagSet.IntVarP(&options.RateLimitMinute, "rate-limit-minute", "rlm", 0, "maximum number of requests to send per minute"),
 		flagSet.IntVarP(&options.BulkSize, "bulk-size", "bs", 25, "maximum number of hosts to be analyzed in parallel per template"),
-		flagSet.IntVarP(&options.TemplateThreads, "concurrency", "c", 10, "maximum number of templates to be executed in parallel"),
+		flagSet.IntVarP(&options.TemplateThreads, "concurrency", "c", 25, "maximum number of templates to be executed in parallel"),
+		flagSet.IntVarP(&options.HeadlessBulkSize, "headless-bulk-size", "hbs", 10, "maximum number of headless hosts to be analyzed in parallel per template"),
+		flagSet.IntVarP(&options.HeadlessTemplateThreads, "headless-concurrency", "hc", 10, "maximum number of headless templates to be executed in parallel"),
 	)
 
 	createGroup(flagSet, "optimization", "Optimizations",
 		flagSet.IntVar(&options.Timeout, "timeout", 5, "time to wait in seconds before timeout"),
 		flagSet.IntVar(&options.Retries, "retries", 1, "number of times to retry a failed request"),
-		flagSet.IntVar(&options.MaxHostError, "max-host-error", 30, "max errors for a host before skipping from scan"),
-
+		flagSet.BoolVarP(&options.LeaveDefaultPorts, "leave-default-ports", "ldp", false, "leave default HTTP/HTTPS ports (eg. host:80,host:443"),
+		flagSet.IntVarP(&options.MaxHostError, "max-host-error", "mhe", 30, "max errors for a host before skipping from scan"),
 		flagSet.BoolVar(&options.Project, "project", false, "use a project folder to avoid sending same request multiple times"),
 		flagSet.StringVar(&options.ProjectPath, "project-path", os.TempDir(), "set a specific project path"),
-
 		flagSet.BoolVarP(&options.StopAtFirstMatch, "stop-at-first-path", "spm", false, "stop processing HTTP requests after the first match (may break template/workflow logic)"),
+		flagSet.BoolVar(&options.Stream, "stream", false, "stream mode - start elaborating without sorting the input"),
 	)
 
 	createGroup(flagSet, "headless", "Headless",
-		flagSet.BoolVar(&options.Headless, "headless", false, "enable templates that require headless browser support"),
+		flagSet.BoolVar(&options.Headless, "headless", false, "enable templates that require headless browser support (root user on linux will disable sandbox)"),
 		flagSet.IntVar(&options.PageTimeout, "page-timeout", 20, "seconds to wait for each page in headless mode"),
-		flagSet.BoolVar(&options.ShowBrowser, "show-browser", false, "show the browser on the screen when running templates with headless mode"),
+		flagSet.BoolVarP(&options.ShowBrowser, "show-browser", "sb", false, "show the browser on the screen when running templates with headless mode"),
+		flagSet.BoolVarP(&options.UseInstalledChrome, "system-chrome", "sc", false, "Use local installed chrome browser instead of nuclei installed"),
 	)
 
 	createGroup(flagSet, "debug", "Debug",
 		flagSet.BoolVar(&options.Debug, "debug", false, "show all requests and responses"),
-		flagSet.BoolVar(&options.DebugRequests, "debug-req", false, "show all sent requests"),
-		flagSet.BoolVar(&options.DebugResponse, "debug-resp", false, "show all received responses"),
-
-		/* TODO why the separation? http://proxy:port vs socks5://proxy:port etc
-		   TODO should auto-set the HTTP_PROXY variable for the process? */
-		flagSet.StringVarP(&options.ProxyURL, "proxy-url", "proxy", "", "URL of the HTTP proxy server"),
-		flagSet.StringVar(&options.ProxySocksURL, "proxy-socks-url", "", "URL of the SOCKS proxy server"),
-
-		flagSet.StringVar(&options.TraceLogFile, "trace-log", "", "file to write sent requests trace log"),
-
+		flagSet.BoolVarP(&options.DebugRequests, "debug-req", "dreq", false, "show all sent requests"),
+		flagSet.BoolVarP(&options.DebugResponse, "debug-resp", "dresp", false, "show all received responses"),
+		flagSet.NormalizedOriginalStringSliceVarP(&options.Proxy, "proxy", "p", []string{}, "list of http/socks5 proxy to use (comma separated or file input)"),
+		flagSet.BoolVarP(&options.ProxyInternal, "proxy-internal", "pi", false, "proxy all internal requests"),
+		flagSet.StringVarP(&options.TraceLogFile, "trace-log", "tlog", "", "file to write sent requests trace log"),
+		flagSet.StringVarP(&options.ErrorLogFile, "error-log", "elog", "", "file to write sent requests error log"),
 		flagSet.BoolVar(&options.Version, "version", false, "show nuclei version"),
+		flagSet.BoolVarP(&options.HangMonitor, "hang-monitor", "hm", false, "enable nuclei hang monitoring"),
+		flagSet.BoolVarP(&options.Verbose, "verbose", "v", false, "show verbose output"),
+		flagSet.BoolVar(&options.VerboseVerbose, "vv", false, "display templates loaded for scan"),
+		flagSet.BoolVarP(&options.EnablePprof, "enable-pprof", "ep", false, "enable pprof debugging server"),
 		flagSet.BoolVarP(&options.TemplatesVersion, "templates-version", "tv", false, "shows the version of the installed nuclei-templates"),
 	)
 
 	createGroup(flagSet, "update", "Update",
-		flagSet.BoolVar(&options.UpdateNuclei, "update", false, "update nuclei to the latest released version"),
-		flagSet.BoolVarP(&options.UpdateTemplates, "update-templates", "ut", false, "update the community templates to latest released version"),
-		flagSet.BoolVarP(&options.NoUpdateTemplates, "no-update-templates", "nut", false, "do not check for nuclei-templates updates"),
-		flagSet.StringVarP(&options.TemplatesDirectory, "update-directory", "ud", templatesDirectory, "overwrite the default nuclei-templates directory"),
+		flagSet.BoolVar(&options.UpdateNuclei, "update", false, "update nuclei engine to the latest released version"),
+		flagSet.BoolVarP(&options.UpdateTemplates, "update-templates", "ut", false, "update nuclei-templates to latest released version"),
+		flagSet.StringVarP(&options.TemplatesDirectory, "update-directory", "ud", "", "overwrite the default directory to install nuclei-templates"),
+		flagSet.BoolVarP(&options.NoUpdateTemplates, "disable-update-check", "duc", false, "disable automatic nuclei/templates update check"),
 	)
 
 	createGroup(flagSet, "stats", "Statistics",
 		flagSet.BoolVar(&options.EnableProgressBar, "stats", false, "display statistics about the running scan"),
-		flagSet.BoolVar(&options.StatsJSON, "stats-json", false, "write statistics data to an output file in JSONL(ines) format"),
+		flagSet.BoolVarP(&options.StatsJSON, "stats-json", "sj", false, "write statistics data to an output file in JSONL(ines) format"),
 		flagSet.IntVarP(&options.StatsInterval, "stats-interval", "si", 5, "number of seconds to wait between showing a statistics update"),
-
-		flagSet.BoolVar(&options.Metrics, "metrics", false, "expose nuclei metrics on a port"),
-		flagSet.IntVar(&options.MetricsPort, "metrics-port", 9092, "port to expose nuclei metrics on"),
+		flagSet.BoolVarP(&options.Metrics, "metrics", "m", false, "expose nuclei metrics on a port"),
+		flagSet.IntVarP(&options.MetricsPort, "metrics-port", "mp", 9092, "port to expose nuclei metrics on"),
 	)
 
 	_ = flagSet.Parse()
 
+	if options.LeaveDefaultPorts {
+		http.LeaveDefaultPorts = true
+	}
+
 	if cfgFile != "" {
 		if err := flagSet.MergeConfigFile(cfgFile); err != nil {
 			gologger.Fatal().Msgf("Could not read config: %s\n", err)
 		}
+		cfgFileFolder := filepath.Dir(cfgFile)
+		if err := config.OverrideIgnoreFilePath(cfgFileFolder); err != nil {
+			gologger.Warning().Msgf("Could not read ignore file from custom path: %s\n", err)
+		}
 	}
+	cleanupOldResumeFiles()
+}
+
+func cleanupOldResumeFiles() {
+	root, err := config.GetConfigDir()
+	if err != nil {
+		return
+	}
+	filter := fileutil.FileFilters{
+		OlderThan: 24 * time.Hour * 10, // cleanup on the 10th day
+		Prefix:    "resume-",
+	}
+	_ = fileutil.DeleteFilesOlderThan(root, filter)
 }
 
 func createGroup(flagSet *goflags.FlagSet, groupName, description string, flags ...*goflags.FlagData) {

v2/go.mod

@@ -3,117 +3,170 @@ module github.com/projectdiscovery/nuclei/v2
 go 1.17
 
 require (
-	github.com/Ice3man543/nvd v1.0.8
 	github.com/Knetic/govaluate v3.0.1-0.20171022003610-9aa49832a739+incompatible
-	github.com/akrylysov/pogreb v0.10.1 // indirect
-	github.com/alecthomas/jsonschema v0.0.0-20210818095345-1014919a589c
-	github.com/andygrunwald/go-jira v1.14.0
-	github.com/antchfx/htmlquery v1.2.3
+	github.com/alecthomas/jsonschema v0.0.0-20211022214203-8b29eab41725
+	github.com/andygrunwald/go-jira v1.15.1
+	github.com/antchfx/htmlquery v1.2.5
 	github.com/apex/log v1.9.0
 	github.com/blang/semver v3.5.1+incompatible
 	github.com/bluele/gcache v0.0.2
-	github.com/c4milo/unpackit v0.1.0 // indirect
-	github.com/corpix/uarand v0.1.1
-	github.com/go-rod/rod v0.101.7
+	github.com/corpix/uarand v0.2.0
+	github.com/go-playground/validator/v10 v10.11.0
+	github.com/go-rod/rod v0.106.8
+	github.com/gobwas/ws v1.1.0
 	github.com/google/go-github v17.0.0+incompatible
-	github.com/gosuri/uilive v0.0.4 // indirect
-	github.com/gosuri/uiprogress v0.0.1 // indirect
-	github.com/itchyny/gojq v0.12.4
+	github.com/itchyny/gojq v0.12.8
 	github.com/json-iterator/go v1.1.12
 	github.com/julienschmidt/httprouter v1.3.0
 	github.com/karlseguin/ccache v2.0.3+incompatible
-	github.com/karrick/godirwalk v1.16.1
+	github.com/karrick/godirwalk v1.17.0 // indirect
 	github.com/logrusorgru/aurora v2.0.3+incompatible
-	github.com/mattn/go-runewidth v0.0.13 // indirect
-	github.com/miekg/dns v1.1.43
+	github.com/miekg/dns v1.1.49
 	github.com/olekukonko/tablewriter v0.0.5
-	github.com/owenrumney/go-sarif v1.0.11
+	github.com/owenrumney/go-sarif/v2 v2.1.1
 	github.com/pkg/errors v0.9.1
 	github.com/projectdiscovery/clistats v0.0.8
-	github.com/projectdiscovery/fastdialer v0.0.13-0.20210917073912-cad93d88e69e
-	github.com/projectdiscovery/goflags v0.0.7
+	github.com/projectdiscovery/cryptoutil v1.0.0
+	github.com/projectdiscovery/fastdialer v0.0.16-0.20220603192502-7d34c304eb65
+	github.com/projectdiscovery/filekv v0.0.0-20210915124239-3467ef45dd08
+	github.com/projectdiscovery/fileutil v0.0.0-20220427234316-40b2541a84b8
+	github.com/projectdiscovery/goflags v0.0.8-0.20220412061559-5119d6086323
 	github.com/projectdiscovery/gologger v1.1.4
 	github.com/projectdiscovery/hmap v0.0.2-0.20210917080408-0fd7bd286bfa
-	github.com/projectdiscovery/interactsh v0.0.4
-	github.com/projectdiscovery/nuclei-updatecheck-api v0.0.0-20210914222811-0a072d262f77
-	github.com/projectdiscovery/rawhttp v0.0.7
-	github.com/projectdiscovery/retryabledns v1.0.13-0.20210916165024-76c5b76fd59a
-	github.com/projectdiscovery/retryablehttp-go v1.0.2
-	github.com/projectdiscovery/stringsutil v0.0.0-20210830151154-f567170afdd9
-	github.com/projectdiscovery/yamldoc-go v1.0.2
+	github.com/projectdiscovery/interactsh v1.0.4
+	github.com/projectdiscovery/nuclei-updatecheck-api v0.0.0-20211006155443-c0a8d610a4df
+	github.com/projectdiscovery/rawhttp v0.0.8-0.20220526170355-03de6bb78f37
+	github.com/projectdiscovery/retryabledns v1.0.13
+	github.com/projectdiscovery/retryablehttp-go v1.0.3-0.20220506110515-811d938bd26d
+	github.com/projectdiscovery/stringsutil v0.0.0-20220422150559-b54fb5dc6833
+	github.com/projectdiscovery/yamldoc-go v1.0.3-0.20211126104922-00d2c6bb43b6
 	github.com/remeh/sizedwaitgroup v1.0.0
-	github.com/rs/xid v1.3.0
+	github.com/rs/xid v1.4.0
 	github.com/segmentio/ksuid v1.0.4
-	github.com/shirou/gopsutil/v3 v3.21.7
+	github.com/shirou/gopsutil/v3 v3.22.5
 	github.com/spaolacci/murmur3 v1.1.0
-	github.com/spf13/cast v1.4.1
-	github.com/stretchr/testify v1.7.0
+	github.com/spf13/cast v1.5.0
 	github.com/syndtr/goleveldb v1.0.0
 	github.com/tj/go-update v2.2.5-0.20200519121640-62b4b798fd68+incompatible
 	github.com/valyala/fasttemplate v1.2.1
-	github.com/xanzy/go-gitlab v0.50.3
-	github.com/ysmood/gson v0.6.4 // indirect
+	github.com/weppos/publicsuffix-go v0.15.1-0.20210928183822-5ee35905bd95
+	github.com/xanzy/go-gitlab v0.68.0
+	github.com/ysmood/gson v0.7.1 // indirect
 	github.com/ysmood/leakless v0.7.0 // indirect
 	go.uber.org/atomic v1.9.0
-	go.uber.org/multierr v1.7.0
+	go.uber.org/multierr v1.8.0
 	go.uber.org/ratelimit v0.2.0
-	golang.org/x/net v0.0.0-20210916014120-12bc252f5db8
-	golang.org/x/oauth2 v0.0.0-20210817223510-7df4dd6e12ab
-	golang.org/x/sys v0.0.0-20210915083310-ed5796bab164 // indirect
+	golang.org/x/net v0.0.0-20220520000938-2e3eb7b945c2
+	golang.org/x/oauth2 v0.0.0-20220411215720-9780585627b5
 	golang.org/x/text v0.3.7
-	google.golang.org/appengine v1.6.7 // indirect
 	gopkg.in/yaml.v2 v2.4.0
+	moul.io/http2curl v1.0.0
+)
+
+require github.com/aws/aws-sdk-go v1.44.27
+
+require github.com/projectdiscovery/folderutil v0.0.0-20220215113126-add60a1e8e08
+
+require (
+	github.com/DataDog/gostackparse v0.5.0
+	github.com/asaskevich/govalidator v0.0.0-20210307081110-f21760c49a8d
+	github.com/docker/go-units v0.4.0
+	github.com/h2non/filetype v1.1.3
+	github.com/hashicorp/go-version v1.5.0
+	github.com/mholt/archiver v3.1.1+incompatible
+	github.com/openrdap/rdap v0.9.1-0.20191017185644-af93e7ef17b7
+	github.com/projectdiscovery/iputil v0.0.0-20210804143329-3a30fcde43f3
+	github.com/projectdiscovery/nvd v1.0.9-0.20220314070650-d4a214c1f87d
+	github.com/projectdiscovery/sliceutil v0.0.0-20220511171050-c7d9bc5cadd9
+	github.com/projectdiscovery/urlutil v0.0.0-20210525140139-b874f06ad921
+	github.com/projectdiscovery/wappalyzergo v0.0.45
+	github.com/stretchr/testify v1.7.1
+	github.com/zmap/zcrypto v0.0.0-20211005224000-2d0ffdec8a9b
+	gopkg.in/yaml.v3 v3.0.1
 )
 
 require (
 	git.mills.io/prologic/smtpd v0.0.0-20210710122116-a525b76c287a // indirect
+	github.com/Mzack9999/go-http-digest-auth-client v0.6.1-0.20220414142836-eb8883508809 // indirect
+	github.com/Mzack9999/ldapserver v1.0.2-0.20211229000134-b44a0d6ad0dd // indirect
 	github.com/PuerkitoBio/goquery v1.6.0 // indirect
-	github.com/StackExchange/wmi v1.2.1 // indirect
+	github.com/akrylysov/pogreb v0.10.1 // indirect
+	github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 // indirect
+	github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 // indirect
 	github.com/andres-erbsen/clock v0.0.0-20160526145045-9e14626cd129 // indirect
 	github.com/andybalholm/cascadia v1.1.0 // indirect
-	github.com/antchfx/xpath v1.1.6 // indirect
+	github.com/antchfx/xpath v1.2.1 // indirect
+	github.com/bits-and-blooms/bitset v1.2.0 // indirect
+	github.com/bits-and-blooms/bloom/v3 v3.0.1 // indirect
+	github.com/c4milo/unpackit v0.1.0 // indirect
+	github.com/caddyserver/certmagic v0.16.1 // indirect
 	github.com/cnf/structhash v0.0.0-20201127153200-e1b16c1ebc08 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
 	github.com/dimchansky/utfbom v1.1.1 // indirect
 	github.com/dsnet/compress v0.0.1 // indirect
-	github.com/eggsampler/acme/v3 v3.2.1 // indirect
 	github.com/fatih/structs v1.1.0 // indirect
-	github.com/go-ole/go-ole v1.2.5 // indirect
-	github.com/golang-jwt/jwt v3.2.1+incompatible // indirect
+	github.com/go-ole/go-ole v1.2.6 // indirect
+	github.com/go-playground/locales v0.14.0 // indirect
+	github.com/go-playground/universal-translator v0.18.0 // indirect
+	github.com/goburrow/cache v0.1.4 // indirect
+	github.com/gobwas/httphead v0.1.0 // indirect
+	github.com/gobwas/pool v0.2.1 // indirect
+	github.com/golang-jwt/jwt/v4 v4.3.0 // indirect
 	github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e // indirect
 	github.com/golang/protobuf v1.5.2 // indirect
 	github.com/golang/snappy v0.0.4 // indirect
-	github.com/google/go-querystring v1.0.0 // indirect
-	github.com/google/uuid v1.2.0 // indirect
-	github.com/hashicorp/go-cleanhttp v0.5.1 // indirect
-	github.com/hashicorp/go-retryablehttp v0.6.8 // indirect
+	github.com/google/go-querystring v1.1.0 // indirect
+	github.com/google/uuid v1.3.0 // indirect
+	github.com/gosuri/uilive v0.0.4 // indirect
+	github.com/gosuri/uiprogress v0.0.1 // indirect
+	github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
+	github.com/hashicorp/go-retryablehttp v0.7.1 // indirect
 	github.com/iancoleman/orderedmap v0.0.0-20190318233801-ac98e3ecb4b0 // indirect
 	github.com/itchyny/timefmt-go v0.1.3 // indirect
-	github.com/jasonlvhit/gocron v0.0.1 // indirect
-	github.com/karlseguin/ccache/v2 v2.0.8 // indirect
-	github.com/klauspost/compress v1.13.4 // indirect
+	github.com/jmespath/go-jmespath v0.4.0 // indirect
+	github.com/klauspost/compress v1.15.2 // indirect
+	github.com/klauspost/cpuid/v2 v2.0.11 // indirect
 	github.com/klauspost/pgzip v1.2.5 // indirect
-	github.com/mattn/go-isatty v0.0.13 // indirect
+	github.com/leodido/go-urn v1.2.1 // indirect
+	github.com/libdns/libdns v0.2.1 // indirect
+	github.com/lor00x/goldap v0.0.0-20180618054307-a546dffdd1a3 // indirect
+	github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
+	github.com/mattn/go-isatty v0.0.14 // indirect
+	github.com/mattn/go-runewidth v0.0.13 // indirect
+	github.com/mholt/acmez v1.0.2 // indirect
+	github.com/mitchellh/go-homedir v1.1.0 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.2 // indirect
+	github.com/nwaples/rardecode v1.1.2 // indirect
+	github.com/pierrec/lz4 v2.6.1+incompatible // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
+	github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
 	github.com/projectdiscovery/blackrock v0.0.0-20210415162320-b38689ae3a2e // indirect
-	github.com/projectdiscovery/cryptoutil v0.0.0-20210805184155-b5d2512f9345 // indirect
-	github.com/projectdiscovery/iputil v0.0.0-20210804143329-3a30fcde43f3 // indirect
 	github.com/projectdiscovery/mapcidr v0.0.8 // indirect
 	github.com/projectdiscovery/networkpolicy v0.0.1 // indirect
 	github.com/rivo/uniseg v0.2.0 // indirect
-	github.com/tklauser/go-sysconf v0.3.7 // indirect
-	github.com/tklauser/numcpus v0.2.3 // indirect
+	github.com/tklauser/go-sysconf v0.3.10 // indirect
+	github.com/tklauser/numcpus v0.4.0 // indirect
 	github.com/trivago/tgo v1.0.7 // indirect
 	github.com/ulikunitz/xz v0.5.10 // indirect
+	github.com/ulule/deepcopier v0.0.0-20200430083143-45decc6639b6 // indirect
 	github.com/valyala/bytebufferpool v1.0.0 // indirect
+	github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 // indirect
 	github.com/yl2chen/cidranger v1.0.2 // indirect
-	github.com/ysmood/goob v0.3.0 // indirect
-	github.com/zclconf/go-cty v1.8.4 // indirect
+	github.com/ysmood/goob v0.4.0 // indirect
+	github.com/yusufpapurcu/wmi v1.2.2 // indirect
+	github.com/zmap/rc2 v0.0.0-20131011165748-24b9757f5521 // indirect
 	go.etcd.io/bbolt v1.3.6 // indirect
-	golang.org/x/time v0.0.0-20191024005414-555d28b269f0 // indirect
-	google.golang.org/protobuf v1.27.1 // indirect
+	go.uber.org/zap v1.21.0 // indirect
+	goftp.io/server/v2 v2.0.0 // indirect
+	golang.org/x/crypto v0.0.0-20220210151621-f4118a5b28e2 // indirect
+	golang.org/x/mod v0.4.2 // indirect
+	golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a // indirect
+	golang.org/x/time v0.0.0-20220411224347-583f2d630306 // indirect
+	golang.org/x/tools v0.1.6-0.20210726203631-07bc1bf47fb2 // indirect
+	golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
+	google.golang.org/appengine v1.6.7 // indirect
+	google.golang.org/protobuf v1.28.0 // indirect
+	gopkg.in/alecthomas/kingpin.v2 v2.2.6 // indirect
 	gopkg.in/corvus-ch/zbase32.v1 v1.0.0 // indirect
-	gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect
 )

v2/go.sum

@@ -36,7 +36,8 @@ git.mills.io/prologic/smtpd v0.0.0-20210710122116-a525b76c287a/go.mod h1:C7hXLmF
 github.com/AndreasBriese/bbloom v0.0.0-20190825152654-46b345b51c96/go.mod h1:bOvUY6CB00SOBii9/FifXqc0awNKxLFCL/+pkDPuyl8=
 github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
 github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
-github.com/Ice3man543/nvd v1.0.8 h1:2CBEgOxyWAkQocnnmEMmRtVPWooPRvcuHFLWj48EM4c=
+github.com/DataDog/gostackparse v0.5.0 h1:jb72P6GFHPHz2W0onsN51cS3FkaMDcjb0QzgxxA4gDk=
+github.com/DataDog/gostackparse v0.5.0/go.mod h1:lTfqcJKqS9KnXQGnyQMCugq3u1FP6UZMfWR0aitKFMM=
 github.com/Ice3man543/nvd v1.0.8/go.mod h1:0DxLJk6revOcJKiZxa2K+rNF/HO1zJO97lqQtXhXfSc=
 github.com/Knetic/govaluate v3.0.0+incompatible/go.mod h1:r7JcOSlj0wfOMncg0iLm8Leh48TZaKVeNIfJntJ2wa0=
 github.com/Knetic/govaluate v3.0.1-0.20171022003610-9aa49832a739+incompatible h1:1G1pk05UrOh0NlF1oeaaix1x8XzrfjIDK47TY0Zehcw=
@@ -44,27 +45,39 @@ github.com/Knetic/govaluate v3.0.1-0.20171022003610-9aa49832a739+incompatible/go
 github.com/Masterminds/glide v0.13.2/go.mod h1:STyF5vcenH/rUqTEv+/hBXlSTo7KYwg2oc2f4tzPWic=
 github.com/Masterminds/semver v1.4.2/go.mod h1:MB6lktGJrhw8PrUyiEoblNEGEQ+RzHPF078ddwwvV3Y=
 github.com/Masterminds/vcs v1.13.0/go.mod h1:N09YCmOQr6RLxC6UNHzuVwAdodYbbnycGHSmwVJjcKA=
+github.com/Mzack9999/go-http-digest-auth-client v0.6.1-0.20220414142836-eb8883508809 h1:ZbFL+BDfBqegi+/Ssh7im5+aQfBRx6it+kHnC7jaDU8=
+github.com/Mzack9999/go-http-digest-auth-client v0.6.1-0.20220414142836-eb8883508809/go.mod h1:upgc3Zs45jBDnBT4tVRgRcgm26ABpaP7MoTSdgysca4=
+github.com/Mzack9999/ldapserver v1.0.2-0.20211229000134-b44a0d6ad0dd h1:RTWs+wEY9efxTKK5aFic5C5KybqQelGcX+JdM69KoTo=
+github.com/Mzack9999/ldapserver v1.0.2-0.20211229000134-b44a0d6ad0dd/go.mod h1:AqtPw7WNT0O69k+AbPKWVGYeW94TqgMW/g+Ppc8AZr4=
 github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
 github.com/OneOfOne/xxhash v1.2.8/go.mod h1:eZbhyaAYD41SGSSsnmcpxVoRiQ/MPUTjUdIIOT9Um7Q=
 github.com/PuerkitoBio/goquery v1.6.0 h1:j7taAbelrdcsOlGeMenZxc2AWXD5fieT1/znArdnx94=
 github.com/PuerkitoBio/goquery v1.6.0/go.mod h1:GsLWisAFVj4WgDibEWF4pvYnkVQBpKBKeU+7zCJoLcc=
-github.com/StackExchange/wmi v1.2.1 h1:VIkavFPXSjcnS+O8yTq7NI32k0R5Aj+v39y29VYDOSA=
 github.com/StackExchange/wmi v1.2.1/go.mod h1:rcmrprowKIVzvc+NUiLncP2uuArMWLCbu9SBzvHz7e8=
 github.com/akrylysov/pogreb v0.10.0/go.mod h1:pNs6QmpQ1UlTJKDezuRWmaqkgUE2TuU0YTWyqJZ7+lI=
 github.com/akrylysov/pogreb v0.10.1 h1:FqlR8VR7uCbJdfUob916tPM+idpKgeESDXOA1K0DK4w=
 github.com/akrylysov/pogreb v0.10.1/go.mod h1:pNs6QmpQ1UlTJKDezuRWmaqkgUE2TuU0YTWyqJZ7+lI=
-github.com/alecthomas/jsonschema v0.0.0-20210818095345-1014919a589c h1:oJsq4z4xKgZWWOhrSZuLZ5KyYfRFytddLL1E5+psfIY=
 github.com/alecthomas/jsonschema v0.0.0-20210818095345-1014919a589c/go.mod h1:/n6+1/DWPltRLWL/VKyUxg6tzsl5kHUCcraimt4vr60=
+github.com/alecthomas/jsonschema v0.0.0-20211022214203-8b29eab41725 h1:NjwIgLQlD46o79bheVG4SCdRnnOz4XtgUN1WABX5DLA=
+github.com/alecthomas/jsonschema v0.0.0-20211022214203-8b29eab41725/go.mod h1:/n6+1/DWPltRLWL/VKyUxg6tzsl5kHUCcraimt4vr60=
+github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 h1:JYp7IbQjafoB+tBA3gMyHYHrpOtNuDiK/uB5uXxq5wM=
+github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
+github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
+github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 h1:s6gZFSlWYmbqAuRjVTiNNhvNRfY2Wxp9nhfyel4rklc=
+github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137/go.mod h1:OMCwj8VM1Kc9e19TLln2VL61YJF0x1XFtfdL4JdbSyE=
 github.com/andres-erbsen/clock v0.0.0-20160526145045-9e14626cd129 h1:MzBOUgng9orim59UnfUTLRjMpd09C5uEVQ6RPGeCaVI=
 github.com/andres-erbsen/clock v0.0.0-20160526145045-9e14626cd129/go.mod h1:rFgpPQZYZ8vdbc+48xibu8ALc3yeyd64IhHS+PU6Yyg=
 github.com/andybalholm/cascadia v1.1.0 h1:BuuO6sSfQNFRu1LppgbD25Hr2vLYW25JvxHs5zzsLTo=
 github.com/andybalholm/cascadia v1.1.0/go.mod h1:GsXiBklL0woXo1j/WYWtSYYC4ouU9PqHO0sqidkEA4Y=
-github.com/andygrunwald/go-jira v1.14.0 h1:7GT/3qhar2dGJ0kq8w0d63liNyHOnxZsUZ9Pe4+AKBI=
 github.com/andygrunwald/go-jira v1.14.0/go.mod h1:KMo2f4DgMZA1C9FdImuLc04x4WQhn5derQpnsuBFgqE=
-github.com/antchfx/htmlquery v1.2.3 h1:sP3NFDneHx2stfNXCKbhHFo8XgNjCACnU/4AO5gWz6M=
+github.com/andygrunwald/go-jira v1.15.1 h1:6J9aYKb9sW8bxv3pBLYBrs0wdsFrmGI5IeTgWSKWKc8=
+github.com/andygrunwald/go-jira v1.15.1/go.mod h1:GIYN1sHOIsENWUZ7B4pDeT/nxEtrZpE8l0987O67ZR8=
 github.com/antchfx/htmlquery v1.2.3/go.mod h1:B0ABL+F5irhhMWg54ymEZinzMSi0Kt3I2if0BLYa3V0=
-github.com/antchfx/xpath v1.1.6 h1:6sVh6hB5T6phw1pFpHRQ+C4bd8sNI+O58flqtg7h0R0=
+github.com/antchfx/htmlquery v1.2.5 h1:1lXnx46/1wtv1E/kzmH8vrfMuUKYgkdDBA9pIdMJnk4=
+github.com/antchfx/htmlquery v1.2.5/go.mod h1:2MCVBzYVafPBmKbrmwB9F5xdd+IEgRY61ci2oOsOQVw=
 github.com/antchfx/xpath v1.1.6/go.mod h1:Yee4kTMuNiPYJ7nSNorELQMr1J33uOpXDMByNYhvtNk=
+github.com/antchfx/xpath v1.2.1 h1:qhp4EW6aCOVr5XIkT+l6LJ9ck/JsUH/yyauNgTQkBF8=
+github.com/antchfx/xpath v1.2.1/go.mod h1:i54GszH55fYfBmoZXapTHN8T8tkcHfRgLyVwwqzXNcs=
 github.com/apex/log v1.9.0 h1:FHtw/xuaM8AgmvDDTI9fiwoAL25Sq2cxojnZICUU8l0=
 github.com/apex/log v1.9.0/go.mod h1:m82fZlWIuiWzWP04XCTXmnX0xRkYYbCdYn8jbJeLBEA=
 github.com/apex/logs v1.0.0/go.mod h1:XzxuLZ5myVHDy9SAmYpamKKRNApGj54PfYLcFrXqDwo=
@@ -72,8 +85,18 @@ github.com/aphistic/golf v0.0.0-20180712155816-02c07f170c5a/go.mod h1:3NqKYiepwy
 github.com/aphistic/sweet v0.2.0/go.mod h1:fWDlIh/isSE9n6EPsRmC0det+whmX6dJid3stzu0Xys=
 github.com/apparentlymart/go-textseg/v13 v13.0.0/go.mod h1:ZK2fH7c4NqDTLtiYLvIkEghdlcqw7yxLeM89kiTRPUo=
 github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
+github.com/asaskevich/govalidator v0.0.0-20210307081110-f21760c49a8d h1:Byv0BzEl3/e6D5CLfI0j/7hiIEtvGVFPCZ7Ei2oq8iQ=
+github.com/asaskevich/govalidator v0.0.0-20210307081110-f21760c49a8d/go.mod h1:WaHUgvxTVq04UNunO+XhnAqY/wQc+bxr74GqbsZ/Jqw=
 github.com/aws/aws-sdk-go v1.20.6/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo=
+github.com/aws/aws-sdk-go v1.44.27 h1:8CMspeZSrewnbvAwgl8qo5R7orDLwQnTGBf/OKPiHxI=
+github.com/aws/aws-sdk-go v1.44.27/go.mod h1:y4AeaBuwd2Lk+GepC1E9v0qOiTws0MIWAX4oIKwKHZo=
 github.com/aybabtme/rgbterm v0.0.0-20170906152045-cc83f3b3ce59/go.mod h1:q/89r3U2H7sSsE2t6Kca0lfwTK8JdoNGS/yzM/4iH5I=
+github.com/benbjohnson/clock v1.1.0 h1:Q92kusRqC1XV2MjkWETPvjJVqKetz1OzxZB7mHJLju8=
+github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA=
+github.com/bits-and-blooms/bitset v1.2.0 h1:Kn4yilvwNtMACtf1eYDlG8H77R07mZSPbMjLyS07ChA=
+github.com/bits-and-blooms/bitset v1.2.0/go.mod h1:gIdJ4wp64HaoK2YrL1Q5/N7Y16edYb8uY+O0FJTyyDA=
+github.com/bits-and-blooms/bloom/v3 v3.0.1 h1:Inlf0YXbgehxVjMPmCGv86iMCKMGPPrPSHtBF5yRHwA=
+github.com/bits-and-blooms/bloom/v3 v3.0.1/go.mod h1:MC8muvBzzPOFsrcdND/A7kU7kMhkqb9KI70JlZCP+C8=
 github.com/blang/semver v3.5.1+incompatible h1:cQNTCjp13qL8KC3Nbxr/y2Bqb63oX6wdnnjpJbkM4JQ=
 github.com/blang/semver v3.5.1+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk=
 github.com/bluele/gcache v0.0.2 h1:WcbfdXICg7G/DGBh1PFfcirkWOQV+v077yF1pSy3DGw=
@@ -82,10 +105,10 @@ github.com/bradfitz/iter v0.0.0-20191230175014-e8f45d346db8 h1:GKTyiRCL6zVf5wWaq
 github.com/bradfitz/iter v0.0.0-20191230175014-e8f45d346db8/go.mod h1:spo1JLcs67NmW1aVLEgtA8Yy1elc+X8y5SRW1sFW4Og=
 github.com/c4milo/unpackit v0.1.0 h1:91pWJ6B3svZ4LOE+p3rnyucRK5fZwBdF/yQ/pcZO31I=
 github.com/c4milo/unpackit v0.1.0/go.mod h1:pvXCMYlSV8zwGFWMaT+PWYkAB/cvDjN2mv9r7ZRSxEo=
+github.com/caddyserver/certmagic v0.16.1 h1:rdSnjcUVJojmL4M0efJ+yHXErrrijS4YYg3FuwRdJkI=
+github.com/caddyserver/certmagic v0.16.1/go.mod h1:jKQ5n+ViHAr6DbPwEGLTSM2vDwTO6EvCKBblBRUvvuQ=
 github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
 github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
-github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
-github.com/cespare/xxhash/v2 v2.1.2/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
 github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
 github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
@@ -97,8 +120,9 @@ github.com/codegangsta/cli v1.20.0/go.mod h1:/qJNoX69yVSKu5o4jLyXAENLRyk1uhi7zkb
 github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
 github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk=
 github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
-github.com/corpix/uarand v0.1.1 h1:RMr1TWc9F4n5jiPDzFHtmaUXLKLNUFK0SgCLo4BhX/U=
 github.com/corpix/uarand v0.1.1/go.mod h1:SFKZvkcRoLqVRFZ4u25xPmp6m9ktANfbpXZ7SJ0/FNU=
+github.com/corpix/uarand v0.2.0 h1:U98xXwud/AVuCpkpgfPF7J5TQgr7R5tqT8VZP5KWbzE=
+github.com/corpix/uarand v0.2.0/go.mod h1:/3Z1QIqWkDIhf6XWn/08/uMHoQ8JUoTIKc2iPchBOmM=
 github.com/cpuguy83/go-md2man v1.0.10/go.mod h1:SmD6nW6nTyfqj6ABTjUi3V3JVMnlJmwcJI5acqYI6dE=
 github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
 github.com/dave/dst v0.26.2/go.mod h1:UMDJuIRPfyUCC78eFuB+SV/WI8oDeyFDvM/JR6NI3IU=
@@ -112,16 +136,16 @@ github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs
 github.com/dgraph-io/badger v1.6.2/go.mod h1:JW2yswe3V058sS0kZ2h/AXeDSqFjxnZcRrVH//y2UQE=
 github.com/dgraph-io/ristretto v0.0.2/go.mod h1:KPxhHT9ZxKefz+PCeOGsrHpl1qZ7i70dGTu2u+Ahh6E=
 github.com/dgraph-io/ristretto v0.0.3/go.mod h1:KPxhHT9ZxKefz+PCeOGsrHpl1qZ7i70dGTu2u+Ahh6E=
-github.com/dgraph-io/ristretto v0.1.0/go.mod h1:fux0lOrBhrVCJd3lcTHsIJhq1T2rokOu6v9Vcb3Q9ug=
 github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw=
 github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw=
 github.com/dimchansky/utfbom v1.1.1 h1:vV6w1AhK4VMnhBno/TPVCoK9U/LP0PkLCS9tbxHdi/U=
 github.com/dimchansky/utfbom v1.1.1/go.mod h1:SxdoEBH5qIqFocHMyGOXVAybYJdr71b1Q/j0mACtrfE=
+github.com/docker/go-units v0.4.0 h1:3uh0PgVws3nIA0Q+MwDC8yjEPf9zjRfZZWXZYDct3Tw=
+github.com/docker/go-units v0.4.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
 github.com/dsnet/compress v0.0.1 h1:PlZu0n3Tuv04TzpfPbrnI0HW/YwodEXDS+oPKahKF0Q=
 github.com/dsnet/compress v0.0.1/go.mod h1:Aw8dCMJ7RioblQeTqt88akK31OvO8Dhf5JflhBbQEHo=
 github.com/dsnet/golib v0.0.0-20171103203638-1ea166775780/go.mod h1:Lj+Z9rebOhdfkVLjJ8T6VcRQv3SXugXy999NBtR9aFY=
 github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
-github.com/eggsampler/acme/v3 v3.2.1 h1:Lfsrg3M2zt00QRnizOFzdpSfsS9oDvPsGrodXS/w1KI=
 github.com/eggsampler/acme/v3 v3.2.1/go.mod h1:/qh0rKC/Dh7Jj+p4So7DbWmFNzC4dpcpK53r226Fhuo=
 github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
 github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
@@ -130,6 +154,8 @@ github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7
 github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
 github.com/fatih/structs v1.1.0 h1:Q7juDM0QtcnhCpeyLGQKyg4TOIghuNXrkL32pHAUMxo=
 github.com/fatih/structs v1.1.0/go.mod h1:9NiDSp5zOcgEDl+j00MP/WkGVPOlPRLejGD8Ga6PJ7M=
+github.com/frankban/quicktest v1.14.3 h1:FJKSZTDHjyhriyC81FLQ0LY93eSai0ZyR/ZIkd3ZUKE=
+github.com/frankban/quicktest v1.14.3/go.mod h1:mgiwOwqx65TmIk1wJ6Q7wvnVMocbUorkibMOrVTHZps=
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
 github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWoS4=
 github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
@@ -137,16 +163,34 @@ github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9
 github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
 github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
 github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
-github.com/go-ole/go-ole v1.2.5 h1:t4MGB5xEDZvXI+0rMjjsfBsD7yAgp/s9ZDkL1JndXwY=
 github.com/go-ole/go-ole v1.2.5/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
+github.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY=
+github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
+github.com/go-playground/assert/v2 v2.0.1 h1:MsBgLAaY856+nPRTKrp3/OZK38U/wa0CcBYNjji3q3A=
+github.com/go-playground/assert/v2 v2.0.1/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
+github.com/go-playground/locales v0.14.0 h1:u50s323jtVGugKlcYeyzC0etD1HifMjqmJqb8WugfUU=
+github.com/go-playground/locales v0.14.0/go.mod h1:sawfccIbzZTqEDETgFXqTho0QybSa7l++s0DH+LDiLs=
+github.com/go-playground/universal-translator v0.18.0 h1:82dyy6p4OuJq4/CByFNOn/jYrnRPArHwAcmLoJZxyho=
+github.com/go-playground/universal-translator v0.18.0/go.mod h1:UvRDBj+xPUEGrFYl+lu/H90nyDXpg0fqeB/AQUGNTVA=
+github.com/go-playground/validator/v10 v10.11.0 h1:0W+xRM511GY47Yy3bZUbJVitCNg2BOGlCyvTqsp/xIw=
+github.com/go-playground/validator/v10 v10.11.0/go.mod h1:i+3WkQ1FvaUjjxh1kSvIA4dMGDBiPU55YFDl0WbKdWU=
 github.com/go-redis/redis v6.15.5+incompatible/go.mod h1:NAIEuMOZ/fxfXJIrKDQDz8wamY7mA7PouImQ2Jvg6kA=
 github.com/go-rod/rod v0.91.1/go.mod h1:/W4lcZiCALPD603MnJGIvhtywP3R6yRB9EDfFfsHiiI=
-github.com/go-rod/rod v0.101.7 h1:kbI5CNvcRhf7feybBln4xDutsM0mbsF0ENNZfKcF6WA=
-github.com/go-rod/rod v0.101.7/go.mod h1:N/zlT53CfSpq74nb6rOR0K8UF0SPUPBmzBnArrms+mY=
+github.com/go-rod/rod v0.106.8 h1:pVMVz0jMtLVyx8FhJEEA6l+EY9Iw/nJTDYT/he4+UJc=
+github.com/go-rod/rod v0.106.8/go.mod h1:xkZOchuKqTOkMOBkrzb7uJpbKZRab1haPCWDvuZkS2U=
+github.com/goburrow/cache v0.1.4 h1:As4KzO3hgmzPlnaMniZU9+VmoNYseUhuELbxy9mRBfw=
+github.com/goburrow/cache v0.1.4/go.mod h1:cDFesZDnIlrHoNlMYqqMpCRawuXulgx+y7mXU8HZ+/c=
+github.com/gobwas/httphead v0.1.0 h1:exrUm0f4YX0L7EBwZHuCF4GDp8aJfVeBrlLQrs6NqWU=
+github.com/gobwas/httphead v0.1.0/go.mod h1:O/RXo79gxV8G+RqlR/otEwx4Q36zl9rqC5u12GKvMCM=
+github.com/gobwas/pool v0.2.1 h1:xfeeEhW7pwmX8nuLVlqbzVc7udMDrwetjEv+TZIz1og=
+github.com/gobwas/pool v0.2.1/go.mod h1:q8bcK0KcYlCgd9e7WYLm9LpyS+YeLd8JVDW6WezmKEw=
+github.com/gobwas/ws v1.1.0 h1:7RFti/xnNkMJnrK7D1yQ/iCIB5OrrY/54/H930kIbHA=
+github.com/gobwas/ws v1.1.0/go.mod h1:nzvNcVha5eUziGrbxFCo6qFIojQHjJV5cLYIbezhfL0=
 github.com/golang-jwt/jwt v3.2.1+incompatible h1:73Z+4BJcrTC+KczS6WvTPvRGOp1WmfEP4Q1lOd9Z/+c=
 github.com/golang-jwt/jwt v3.2.1+incompatible/go.mod h1:8pz2t5EyA70fFQQSrl6XZXzqecmYZeUEB8OUGHkxJ+I=
+github.com/golang-jwt/jwt/v4 v4.3.0 h1:kHL1vqdqWNfATmA0FNMdmZNMyZI1U6O31X4rlIPoBog=
+github.com/golang-jwt/jwt/v4 v4.3.0/go.mod h1:/xlHOz8bRuivTWchD4jCa+NbatV+wEUSzwAxVc6locg=
 github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
-github.com/golang/glog v1.0.0/go.mod h1:EWib/APOK0SL3dFbYqvxE3UYd8E6s1ouQ7iEp/0LWV4=
 github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
 github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
 github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e h1:1r7pUrabqp18hOBcwBwiTsbnFeTZHV9eER/QT5JVZxY=
@@ -188,15 +232,19 @@ github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/
 github.com/google/go-cmp v0.4.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/go-cmp v0.5.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
-github.com/google/go-cmp v0.5.6 h1:BKbKCqvP6I+rmFHt06ZmyQtvB8xAkWdhFyr0ZUNZcxQ=
 github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+github.com/google/go-cmp v0.5.7/go.mod h1:n+brtR0CgQNWTVd5ZUFpTBC8YFBDLK/h/bpaJ8/DtOE=
+github.com/google/go-cmp v0.5.8 h1:e6P7q2lk1O+qJJb4BtCQXlK8vWEO8V1ZeuEdJNOqZyg=
+github.com/google/go-cmp v0.5.8/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
 github.com/google/go-github v17.0.0+incompatible h1:N0LgJ1j65A7kfXrZnUDaYCs/Sf4rEjNlfyDHW9dolSY=
 github.com/google/go-github v17.0.0+incompatible/go.mod h1:zLgOLi98H3fifZn+44m+umXrS52loVEgC2AApnigrVQ=
 github.com/google/go-querystring v0.0.0-20170111101155-53e6ce116135/go.mod h1:odCYkC5MyYFN7vkCjXpyrEuKhc/BUO6wN/zVPAxq5ck=
-github.com/google/go-querystring v1.0.0 h1:Xkwi/a1rcvNg1PPYe5vI8GbeBY/jrVuDX5ASuANWTrk=
 github.com/google/go-querystring v1.0.0/go.mod h1:odCYkC5MyYFN7vkCjXpyrEuKhc/BUO6wN/zVPAxq5ck=
+github.com/google/go-querystring v1.1.0 h1:AnCroh3fv4ZBgVIf1Iwtovgjaw/GiKJo8M8yD/fhyJ8=
+github.com/google/go-querystring v1.1.0/go.mod h1:Kcdr2DB4koayq7X8pmAG4sNG59So17icRSOU623lUBU=
 github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
 github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
 github.com/google/martian/v3 v3.0.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0=
@@ -210,21 +258,30 @@ github.com/google/pprof v0.0.0-20200430221834-fc25d7d30c6d/go.mod h1:ZgVRPoUq/hf
 github.com/google/pprof v0.0.0-20200708004538-1a94d8640e99/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
 github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
 github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
-github.com/google/uuid v1.2.0 h1:qJYtXnJRWmpe7m/3XlyhrsLrEURqHRM2kxzoxXqyUDs=
 github.com/google/uuid v1.2.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
+github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
 github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
+github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8=
+github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
 github.com/gosuri/uilive v0.0.4 h1:hUEBpQDj8D8jXgtCdBu7sWsy5sbW/5GhuO8KBwJ2jyY=
 github.com/gosuri/uilive v0.0.4/go.mod h1:V/epo5LjjlDE5RJUcqx8dbw+zc93y5Ya3yg8tfZ74VI=
 github.com/gosuri/uiprogress v0.0.1 h1:0kpv/XY/qTmFWl/SkaJykZXrBBzwwadmW8fRb7RJSxw=
 github.com/gosuri/uiprogress v0.0.1/go.mod h1:C1RTYn4Sc7iEyf6j8ft5dyoZ4212h8G1ol9QQluh5+0=
-github.com/hashicorp/go-cleanhttp v0.5.1 h1:dH3aiDG9Jvb5r5+bYHsikaOUIpcM0xvgMXVoDkXMzJM=
+github.com/h2non/filetype v1.1.3 h1:FKkx9QbD7HR/zjK1Ia5XiBsq9zdLi5Kf3zGyFTAFkGg=
+github.com/h2non/filetype v1.1.3/go.mod h1:319b3zT68BvV+WRj7cwy856M2ehB3HqNOt6sy1HndBY=
 github.com/hashicorp/go-cleanhttp v0.5.1/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80=
+github.com/hashicorp/go-cleanhttp v0.5.2 h1:035FKYIWjmULyFRBKPs8TBQoi0x6d9G4xc9neXJWAZQ=
+github.com/hashicorp/go-cleanhttp v0.5.2/go.mod h1:kO/YDlP8L1346E6Sodw+PrpBSV4/SoxCXGY6BqNFT48=
 github.com/hashicorp/go-hclog v0.9.2 h1:CG6TE5H9/JXsFWJCfoIVpKFIkFe6ysEuHirp4DxCsHI=
 github.com/hashicorp/go-hclog v0.9.2/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ=
-github.com/hashicorp/go-retryablehttp v0.6.8 h1:92lWxgpa+fF3FozM4B3UZtHZMJX8T5XT+TFdCxsPyWs=
 github.com/hashicorp/go-retryablehttp v0.6.8/go.mod h1:vAew36LZh98gCBJNLH42IQ1ER/9wtLZZ8meHqQvEYWY=
+github.com/hashicorp/go-retryablehttp v0.7.1 h1:sUiuQAnLlbvmExtFQs72iFW/HXeUn8Z1aJLQ4LJJbTQ=
+github.com/hashicorp/go-retryablehttp v0.7.1/go.mod h1:vAew36LZh98gCBJNLH42IQ1ER/9wtLZZ8meHqQvEYWY=
 github.com/hashicorp/go-version v1.3.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA=
+github.com/hashicorp/go-version v1.5.0 h1:O293SZ2Eg+AAYijkVK3jR786Am1bhDEh2GHT0tIVE5E=
+github.com/hashicorp/go-version v1.5.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA=
 github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
 github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
 github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
@@ -236,13 +293,20 @@ github.com/iancoleman/orderedmap v0.0.0-20190318233801-ac98e3ecb4b0/go.mod h1:N0
 github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
 github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
 github.com/itchyny/go-flags v1.5.0/go.mod h1:lenkYuCobuxLBAd/HGFE4LRoW8D3B6iXRQfWYJ+MNbA=
-github.com/itchyny/gojq v0.12.4 h1:8zgOZWMejEWCLjbF/1mWY7hY7QEARm7dtuhC6Bp4R8o=
 github.com/itchyny/gojq v0.12.4/go.mod h1:EQUSKgW/YaOxmXpAwGiowFDO4i2Rmtk5+9dFyeiymAg=
+github.com/itchyny/gojq v0.12.8 h1:Zxcwq8w4IeR8JJYEtoG2MWJZUv0RGY6QqJcO1cqV8+A=
+github.com/itchyny/gojq v0.12.8/go.mod h1:gE2kZ9fVRU0+JAksaTzjIlgnCa2akU+a1V0WXgJQN5c=
 github.com/itchyny/timefmt-go v0.1.3 h1:7M3LGVDsqcd0VZH2U+x393obrzZisp7C0uEe921iRkU=
 github.com/itchyny/timefmt-go v0.1.3/go.mod h1:0osSSCQSASBJMsIZnhAaF1C2fCBTJZXrnj37mG8/c+A=
-github.com/jasonlvhit/gocron v0.0.1 h1:qTt5qF3b3srDjeOIR4Le1LfeyvoYzJlYpqvG7tJX5YU=
+github.com/jarcoal/httpmock v1.0.4 h1:jp+dy/+nonJE4g4xbVtl9QdrUNbn6/3hDT5R4nDIZnA=
+github.com/jarcoal/httpmock v1.0.4/go.mod h1:ATjnClrvW/3tijVmpL/va5Z3aAyGvqU3gCT8nX0Txik=
 github.com/jasonlvhit/gocron v0.0.1/go.mod h1:k9a3TV8VcU73XZxfVHCHWMWF9SOqgoku0/QlY2yvlA4=
+github.com/jlaffaye/ftp v0.0.0-20190624084859-c1312a7102bf/go.mod h1:lli8NYPQOFy3O++YmYbqVgOcQ1JPCwdOy+5zSjKJ9qY=
 github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
+github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
+github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo=
+github.com/jmespath/go-jmespath/internal/testify v1.5.1 h1:shLQSRRSCCPj3f2gpwzGwWFoC7ycTf1rcQZHOlsJ6N8=
+github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U=
 github.com/jpillora/backoff v0.0.0-20180909062703-3050d21c67d7/go.mod h1:2iMrUgbbvHEiQClaW2NsSzMyGHqN+rDFqY705q49KG0=
 github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
 github.com/json-iterator/go v1.1.11/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
@@ -250,6 +314,8 @@ github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnr
 github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
 github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
 github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
+github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
+github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
 github.com/julienschmidt/httprouter v1.3.0 h1:U0609e9tgbseu3rBINet9P48AI/D3oJs4dN7jwJOQ1U=
 github.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM=
 github.com/karlseguin/ccache v2.0.3+incompatible h1:j68C9tWOROiOLWTS/kCGg9IcJG+ACqn5+0+t8Oh83UU=
@@ -258,15 +324,20 @@ github.com/karlseguin/ccache/v2 v2.0.8 h1:lT38cE//uyf6KcFok0rlgXtGFBWxkI6h/qg4tb
 github.com/karlseguin/ccache/v2 v2.0.8/go.mod h1:2BDThcfQMf/c0jnZowt16eW405XIqZPavt+HoYEtcxQ=
 github.com/karlseguin/expect v1.0.2-0.20190806010014-778a5f0c6003 h1:vJ0Snvo+SLMY72r5J4sEfkuE7AFbixEP2qRbEcum/wA=
 github.com/karlseguin/expect v1.0.2-0.20190806010014-778a5f0c6003/go.mod h1:zNBxMY8P21owkeogJELCLeHIt+voOSduHYTFUbwRAV8=
-github.com/karrick/godirwalk v1.16.1 h1:DynhcF+bztK8gooS0+NDJFrdNZjJ3gzVzC545UNA9iw=
 github.com/karrick/godirwalk v1.16.1/go.mod h1:j4mkqPuvaLI8mp1DroR3P6ad7cyYd4c1qeJ3RV7ULlk=
+github.com/karrick/godirwalk v1.17.0 h1:b4kY7nqDdioR/6qnbHQyDvmA17u5G1cZ6J+CZXwSWoI=
+github.com/karrick/godirwalk v1.17.0/go.mod h1:j4mkqPuvaLI8mp1DroR3P6ad7cyYd4c1qeJ3RV7ULlk=
 github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
 github.com/klauspost/compress v1.4.1/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A=
-github.com/klauspost/compress v1.13.4 h1:0zhec2I8zGnjWcKyLl6i3gPqKANCCn5e9xmviEEeX6s=
-github.com/klauspost/compress v1.13.4/go.mod h1:8dP1Hq4DHOhN9w426knH3Rhby4rFm6D8eO+e+Dq5Gzg=
+github.com/klauspost/compress v1.15.2 h1:3WH+AG7s2+T8o3nrM/8u2rdqUEcQhmga7smjrT41nAw=
+github.com/klauspost/compress v1.15.2/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU=
+github.com/klauspost/cpuid v1.2.0 h1:NMpwD2G9JSFOE1/TJjGSo5zG7Yb2bTe7eq1jH+irmeE=
 github.com/klauspost/cpuid v1.2.0/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=
+github.com/klauspost/cpuid/v2 v2.0.11 h1:i2lw1Pm7Yi/4O6XCSyJWqEHI2MDw2FzUK6o/D21xn2A=
+github.com/klauspost/cpuid/v2 v2.0.11/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuObKfj5c0PQa7c=
 github.com/klauspost/pgzip v1.2.5 h1:qnWYvvKqedOF2ulHpMG72XQol4ILEJ8k2wwRl/Km8oE=
 github.com/klauspost/pgzip v1.2.5/go.mod h1:Ch1tH69qFZu15pkjo5kYi6mth2Zzwzt50oCQKQE9RUs=
+github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
 github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
 github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
 github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
@@ -277,24 +348,43 @@ github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
 github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
 github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
 github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
+github.com/leodido/go-urn v1.2.1 h1:BqpAaACuzVSgi/VLzGZIobT2z4v53pjosyNd9Yv6n/w=
+github.com/leodido/go-urn v1.2.1/go.mod h1:zt4jvISO2HfUBqxjfIshjdMTYS56ZS/qv49ictyFfxY=
+github.com/libdns/libdns v0.2.1 h1:Wu59T7wSHRgtA0cfxC+n1c/e+O3upJGWytknkmFEDis=
+github.com/libdns/libdns v0.2.1/go.mod h1:yQCXzk1lEZmmCPa857bnk4TsOiqYasqpyOEeSObbb40=
 github.com/logrusorgru/aurora v0.0.0-20200102142835-e9ef32dff381/go.mod h1:7rIyQOR62GCctdiQpZ/zOJlFyk6y+94wXzv6RNZgaR4=
 github.com/logrusorgru/aurora v2.0.3+incompatible h1:tOpm7WcpBTn4fjmVfgpQq0EfczGlG91VSDkswnjF5A8=
 github.com/logrusorgru/aurora v2.0.3+incompatible/go.mod h1:7rIyQOR62GCctdiQpZ/zOJlFyk6y+94wXzv6RNZgaR4=
+github.com/lor00x/goldap v0.0.0-20180618054307-a546dffdd1a3 h1:wIONC+HMNRqmWBjuMxhatuSzHaljStc4gjDeKycxy0A=
+github.com/lor00x/goldap v0.0.0-20180618054307-a546dffdd1a3/go.mod h1:37YR9jabpiIxsb8X9VCIx8qFOjTDIIrIHHODa8C4gz0=
+github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 h1:6E+4a0GO5zZEnZ81pIr0yLvtUWk2if982qA3F3QD6H4=
+github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0/go.mod h1:zJYVVT2jmtg6P3p1VtQj7WsuWi/y4VnjVBn7F8KPB3I=
 github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
 github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ=
 github.com/mattn/go-colorable v0.1.2/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE=
 github.com/mattn/go-isatty v0.0.5/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
 github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
-github.com/mattn/go-isatty v0.0.13 h1:qdl+GuBjcsKKDco5BsxPJlId98mSWNKqYA+Co0SC1yA=
 github.com/mattn/go-isatty v0.0.13/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU=
+github.com/mattn/go-isatty v0.0.14 h1:yVuAays6BHfxijgZPzw+3Zlu5yQgKGP2/hcQbHb7S9Y=
+github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94=
 github.com/mattn/go-runewidth v0.0.9/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
 github.com/mattn/go-runewidth v0.0.13 h1:lTGmDsbAYt5DmK6OnoV7EuIF1wEIFAcxld6ypU4OSgU=
 github.com/mattn/go-runewidth v0.0.13/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
 github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b/go.mod h1:01TrycV0kFyexm33Z7vhZRXopbI8J3TDReVlkTgMUxE=
+github.com/mholt/acmez v1.0.2 h1:C8wsEBIUVi6e0DYoxqCcFuXtwc4AWXL/jgcDjF7mjVo=
+github.com/mholt/acmez v1.0.2/go.mod h1:8qnn8QA/Ewx8E3ZSsmscqsIjhhpxuy9vqdgbX2ceceM=
+github.com/mholt/archiver v3.1.1+incompatible h1:1dCVxuqs0dJseYEhi5pl7MYPH9zDa1wBi7mF09cbNkU=
+github.com/mholt/archiver v3.1.1+incompatible/go.mod h1:Dh2dOXnSdiLxRiPoVfIr/fI1TwETms9B8CTWfeh7ROU=
 github.com/miekg/dns v1.1.29/go.mod h1:KNUDUusw/aVsxyTYZM1oqvCicbwhgbNgztCETuNZ7xM=
 github.com/miekg/dns v1.1.41/go.mod h1:p6aan82bvRIyn+zDIv9xYNUpwa73JcSh9BKwknJysuI=
-github.com/miekg/dns v1.1.43 h1:JKfpVSCB84vrAmHzyrsxB5NAr5kLoMXZArPSw7Qlgyg=
 github.com/miekg/dns v1.1.43/go.mod h1:+evo5L0630/F6ca/Z9+GAqzhjGyn8/c+TBaOyfEl0V4=
+github.com/miekg/dns v1.1.46/go.mod h1:e3IlAVfNqAllflbibAZEWOXOQ+Ynzk/dDozDxY7XnME=
+github.com/miekg/dns v1.1.48/go.mod h1:e3IlAVfNqAllflbibAZEWOXOQ+Ynzk/dDozDxY7XnME=
+github.com/miekg/dns v1.1.49 h1:qe0mQU3Z/XpFeE+AEBo2rqaS1IPBJ3anmqZ4XiZJVG8=
+github.com/miekg/dns v1.1.49/go.mod h1:e3IlAVfNqAllflbibAZEWOXOQ+Ynzk/dDozDxY7XnME=
+github.com/minio/minio-go/v6 v6.0.46/go.mod h1:qD0lajrGW49lKZLtXKtCB4X/qkMf0a5tBvN2PaZg7Gg=
+github.com/minio/sha256-simd v0.1.1/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM=
+github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
 github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
 github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
 github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
@@ -304,7 +394,10 @@ github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lN
 github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
 github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
 github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
+github.com/mreiferson/go-httpclient v0.0.0-20160630210159-31f0106b4474/go.mod h1:OQA4XLvDbMgS8P0CevmM4m9Q3Jq4phKUzcocxuGJ5m8=
 github.com/ngdinhtoan/glide-cleanup v0.2.0/go.mod h1:UQzsmiDOb8YV3nOsCxK/c9zPpCZVNoHScRE3EO9pVMM=
+github.com/nwaples/rardecode v1.1.2 h1:Cj0yZY6T1Zx1R7AhTbyGSALm44/Mmq+BAPc4B/p/d3M=
+github.com/nwaples/rardecode v1.1.2/go.mod h1:5DzqNKiOdpKKBH87u8VlvAnPZMXcGRhxWkRpHbbfGS0=
 github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A=
 github.com/nxadm/tail v1.4.8 h1:nPr65rt6Y5JFSKQO7qToXr7pePgD6Gwiw05lkbyAQTE=
 github.com/nxadm/tail v1.4.8/go.mod h1:+ncqLTQzXmGhMZNUePPaPqPvBxHAIsmXswZKocGu+AU=
@@ -323,36 +416,58 @@ github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7J
 github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo=
 github.com/onsi/gomega v1.10.5 h1:7n6FEkpFmfCoo2t+YYqXH0evK+a9ICQz0xcAy9dYcaQ=
 github.com/onsi/gomega v1.10.5/go.mod h1:gza4q3jKQJijlu05nKWRCW/GavJumGt8aNRxWg7mt48=
-github.com/owenrumney/go-sarif v1.0.11 h1:7k4TLSi6h3vAozSECjO0arcQoeUNDMgvA7LDac95sJo=
+github.com/op/go-logging v0.0.0-20160315200505-970db520ece7/go.mod h1:HzydrMdWErDVzsI23lYNej1Htcns9BCg93Dk0bBINWk=
+github.com/openrdap/rdap v0.9.1-0.20191017185644-af93e7ef17b7 h1:3Xn/CN6GVY+7mVuGgt5bfp0F9JwcWqnvwfb23Jf8Vxg=
+github.com/openrdap/rdap v0.9.1-0.20191017185644-af93e7ef17b7/go.mod h1:inRbqVxN7ri77yTJY3ZtGtKegIFa3Qnarh7Xp9P7LgY=
 github.com/owenrumney/go-sarif v1.0.11/go.mod h1:hTBFbxU7GuVRUvwMx+eStp9M/Oun4xHCS3vqpPvket8=
+github.com/owenrumney/go-sarif v1.1.1 h1:QNObu6YX1igyFKhdzd7vgzmw7XsWN3/6NMGuDzBgXmE=
+github.com/owenrumney/go-sarif v1.1.1/go.mod h1:dNDiPlF04ESR/6fHlPyq7gHKmrM0sHUvAGjsoh8ZH0U=
+github.com/owenrumney/go-sarif/v2 v2.1.1 h1:JVUO0cEhG8bvEWIxsRmURY4u7wBZUTgdh4zikkkiPM8=
+github.com/owenrumney/go-sarif/v2 v2.1.1/go.mod h1:MSqMMx9WqlBSY7pXoOZWgEsVB4FDNfhcaXDA1j6Sr+w=
 github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
+github.com/pierrec/lz4 v2.6.1+incompatible h1:9UY3+iC23yxF0UfGaYrGplQ+79Rg+h/q9FV9ix19jjM=
+github.com/pierrec/lz4 v2.6.1+incompatible/go.mod h1:pdkljMzZIN41W+lC3N2tnIh5sFi+IEE17M5jbnwPHcY=
 github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
 github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
 github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c h1:ncq/mPwQF4JjgDlrVEn3C11VoGHZN7m8qihwgMEtzYw=
+github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE=
 github.com/projectdiscovery/blackrock v0.0.0-20210415162320-b38689ae3a2e h1:7bwaFH1jvtOo5ndhTQgoA349ozhX+1dc4b6tbaPnBOA=
 github.com/projectdiscovery/blackrock v0.0.0-20210415162320-b38689ae3a2e/go.mod h1:/IsapnEYiWG+yEDPXp0e8NWj3npzB9Ccy9lXEUJwMZs=
 github.com/projectdiscovery/clistats v0.0.8 h1:tjmWb15mqsPf/yrQXVHLe2ThZX/5+mgKSfZBKWWLh20=
 github.com/projectdiscovery/clistats v0.0.8/go.mod h1:lV6jUHAv2bYWqrQstqW8iVIydKJhWlVaLl3Xo9ioVGg=
-github.com/projectdiscovery/cryptoutil v0.0.0-20210805184155-b5d2512f9345 h1:jT6f/cdOpLkp9GAfRrxk57BUjYfIrR8E+AjMv5H5U4U=
 github.com/projectdiscovery/cryptoutil v0.0.0-20210805184155-b5d2512f9345/go.mod h1:clhQmPnt35ziJW1AhJRKyu8aygXCSoyWj6dtmZBRjjc=
+github.com/projectdiscovery/cryptoutil v1.0.0 h1:5rQfnWDthJ5ZFcqze+rmT1N7l1HJQ6EB26MrjaYB7I0=
+github.com/projectdiscovery/cryptoutil v1.0.0/go.mod h1:VJvSNE8f8A1MgpjgAL2GPJSQcJa4jbdaeQJstARFrU4=
 github.com/projectdiscovery/fastdialer v0.0.12/go.mod h1:RkRbxqDCcCFhfNUbkzBIz/ieD4uda2JuUA4WJ+RLee0=
-github.com/projectdiscovery/fastdialer v0.0.13-0.20210917073912-cad93d88e69e h1:xMAFYJgRxopAwKrj7HDwMBKJGCGDbHqopS8f959xges=
-github.com/projectdiscovery/fastdialer v0.0.13-0.20210917073912-cad93d88e69e/go.mod h1:O1l6+vAQy1QRo9FqyuyJ57W3CwpIXXg7oGo14Le6ZYQ=
-github.com/projectdiscovery/goflags v0.0.7 h1:aykmRkrOgDyRwcvGrK3qp+9aqcjGfAMs/+LtRmtyxwk=
+github.com/projectdiscovery/fastdialer v0.0.16-0.20220603192502-7d34c304eb65 h1:6nPONYe5Zas+0toFxP0vvDOxMwQIZo6geajd+FfwzMc=
+github.com/projectdiscovery/fastdialer v0.0.16-0.20220603192502-7d34c304eb65/go.mod h1:k4iAKJMOJJWpqgYA3tDXtNFRkL40H0DXQ0cmyTg5J5k=
+github.com/projectdiscovery/filekv v0.0.0-20210915124239-3467ef45dd08 h1:NwD1R/du1dqrRKN3SJl9kT6tN3K9puuWFXEvYF2ihew=
+github.com/projectdiscovery/filekv v0.0.0-20210915124239-3467ef45dd08/go.mod h1:paLCnwV8sL7ppqIwVQodQrk3F6mnWafwTDwRd7ywZwQ=
+github.com/projectdiscovery/fileutil v0.0.0-20210914153648-31f843feaad4/go.mod h1:U+QCpQnX8o2N2w0VUGyAzjM3yBAe4BKedVElxiImsx0=
+github.com/projectdiscovery/fileutil v0.0.0-20210926202739-6050d0acf73c/go.mod h1:U+QCpQnX8o2N2w0VUGyAzjM3yBAe4BKedVElxiImsx0=
+github.com/projectdiscovery/fileutil v0.0.0-20210928100737-cab279c5d4b5/go.mod h1:U+QCpQnX8o2N2w0VUGyAzjM3yBAe4BKedVElxiImsx0=
+github.com/projectdiscovery/fileutil v0.0.0-20220215113056-ba188a0c8abc/go.mod h1:Pm0f+MWgDFMSSI9NBedNh48LyYPs8gD3Jd8DXGmp4aQ=
+github.com/projectdiscovery/fileutil v0.0.0-20220427234316-40b2541a84b8 h1:2j5ev2I0RESb1+sLpnbLDXzty83Kdz0/bO4OVKWMPfM=
+github.com/projectdiscovery/fileutil v0.0.0-20220427234316-40b2541a84b8/go.mod h1:wjS/oBWbzlayJ/aTK0KW0oOHGO03G8oEYzuN6stI8Ho=
+github.com/projectdiscovery/folderutil v0.0.0-20220215113126-add60a1e8e08 h1:m1pgJisawU7zP9lKGktOEk6KNrNAR7e4Q07Kt3ox0NM=
+github.com/projectdiscovery/folderutil v0.0.0-20220215113126-add60a1e8e08/go.mod h1:BMqXH4jNGByVdE2iLtKvc/6XStaiZRuCIaKv1vw9PnI=
 github.com/projectdiscovery/goflags v0.0.7/go.mod h1:Jjwsf4eEBPXDSQI2Y+6fd3dBumJv/J1U0nmpM+hy2YY=
+github.com/projectdiscovery/goflags v0.0.8-0.20220412061559-5119d6086323 h1:M9DJMe3zMnvHkt5McU1IpoTQbBYc2aB1nsOBLqWlQOs=
+github.com/projectdiscovery/goflags v0.0.8-0.20220412061559-5119d6086323/go.mod h1:uN+pHMLsWQoiZHUg/l0tqf/VdbX3+ecKfYz/H7b/+NA=
 github.com/projectdiscovery/gologger v1.0.1/go.mod h1:Ok+axMqK53bWNwDSU1nTNwITLYMXMdZtRc8/y1c7sWE=
 github.com/projectdiscovery/gologger v1.1.4 h1:qWxGUq7ukHWT849uGPkagPKF3yBPYAsTtMKunQ8O2VI=
 github.com/projectdiscovery/gologger v1.1.4/go.mod h1:Bhb6Bdx2PV1nMaFLoXNBmHIU85iROS9y1tBuv7T5pMY=
 github.com/projectdiscovery/hmap v0.0.1/go.mod h1:VDEfgzkKQdq7iGTKz8Ooul0NuYHQ8qiDs6r8bPD1Sb0=
 github.com/projectdiscovery/hmap v0.0.2-0.20210616215655-7b78e7f33d1f/go.mod h1:FH+MS/WNKTXJQtdRn+/Zg5WlKCiMN0Z1QUedUIuM5n8=
-github.com/projectdiscovery/hmap v0.0.2-0.20210917073634-bfb0e9c03800/go.mod h1:FH+MS/WNKTXJQtdRn+/Zg5WlKCiMN0Z1QUedUIuM5n8=
 github.com/projectdiscovery/hmap v0.0.2-0.20210917080408-0fd7bd286bfa h1:9sZWFUAshIa/ea0RKjGRuuZiS5PzYXAFjTRUnSbezr0=
 github.com/projectdiscovery/hmap v0.0.2-0.20210917080408-0fd7bd286bfa/go.mod h1:lV5f/PNPmCCjCN/dR317/chN9s7VG5h/xcbFfXOz8Fo=
-github.com/projectdiscovery/interactsh v0.0.4 h1:3BtCZrrTovGYiqdFktXJ4NxKAQFvUvzcEI5pJIuShM8=
 github.com/projectdiscovery/interactsh v0.0.4/go.mod h1:PtJrddeBW1/LeOVgTvvnjUl3Hu/17jTkoIi8rXeEODE=
+github.com/projectdiscovery/interactsh v1.0.4 h1:73HHOeqGduU1lkIqeYkuEfzdejXRqxQJFZCEbQ1uvkU=
+github.com/projectdiscovery/interactsh v1.0.4/go.mod h1:57Xl3Q59aTb8VaFNGkCPO8pWTyTYhlM33p+HDhL8Ygw=
 github.com/projectdiscovery/ipranger v0.0.2/go.mod h1:kcAIk/lo5rW+IzUrFkeYyXnFJ+dKwYooEOHGVPP/RWE=
 github.com/projectdiscovery/iputil v0.0.0-20210414194613-4b4d2517acf0/go.mod h1:PQAqn5h5NXsQTF4ZA00ZTYLRzGCjOtcCq8llAqrsd1A=
 github.com/projectdiscovery/iputil v0.0.0-20210429152401-c18a5408ca46/go.mod h1:PQAqn5h5NXsQTF4ZA00ZTYLRzGCjOtcCq8llAqrsd1A=
@@ -365,24 +480,39 @@ github.com/projectdiscovery/mapcidr v0.0.8 h1:16U05F2x3o/jSTsxSCY2hCuCs9xOSwVxjo
 github.com/projectdiscovery/mapcidr v0.0.8/go.mod h1:7CzdUdjuLVI0s33dQ33lWgjg3vPuLFw2rQzZ0RxkT00=
 github.com/projectdiscovery/networkpolicy v0.0.1 h1:RGRuPlxE8WLFF9tdKSjTsYiTIKHNHW20Kl0nGGiRb1I=
 github.com/projectdiscovery/networkpolicy v0.0.1/go.mod h1:asvdg5wMy3LPVMGALatebKeOYH5n5fV5RCTv6DbxpIs=
-github.com/projectdiscovery/nuclei-updatecheck-api v0.0.0-20210914222811-0a072d262f77 h1:SNtAiRRrJtDJJDroaa/bFXt/Tix2LA6+rHRib0ORlJQ=
-github.com/projectdiscovery/nuclei-updatecheck-api v0.0.0-20210914222811-0a072d262f77/go.mod h1:pxWVDgq88t9dWv4+J2AIaWgY+EqOE1AyfHS0Tn23w4M=
+github.com/projectdiscovery/nuclei-updatecheck-api v0.0.0-20211006155443-c0a8d610a4df h1:CvTNAUD5JbLMqpMFoGNgfk2gOcN0NC57ICu0+oK84vs=
+github.com/projectdiscovery/nuclei-updatecheck-api v0.0.0-20211006155443-c0a8d610a4df/go.mod h1:pxWVDgq88t9dWv4+J2AIaWgY+EqOE1AyfHS0Tn23w4M=
 github.com/projectdiscovery/nuclei/v2 v2.5.1/go.mod h1:sU2qcY0MQFS0CqP1BgkR8ZnUyFhqK0BdnY6bvTKNjXY=
-github.com/projectdiscovery/rawhttp v0.0.7 h1:5m4peVgjbl7gqDcRYMTVEuX+Xs/nh76ohTkkvufucLg=
+github.com/projectdiscovery/nvd v1.0.9-0.20220314070650-d4a214c1f87d h1:WK/zpWYGuZhAudzVjbnCYk12L4SqH66jagBzHRm3eCo=
+github.com/projectdiscovery/nvd v1.0.9-0.20220314070650-d4a214c1f87d/go.mod h1:nGHAo7o6G4V4kscZlm488qKp/ZrZYiBoKqAQrn3X4Og=
 github.com/projectdiscovery/rawhttp v0.0.7/go.mod h1:PQERZAhAv7yxI/hR6hdDPgK1WTU56l204BweXrBec+0=
+github.com/projectdiscovery/rawhttp v0.0.8-0.20220526170355-03de6bb78f37 h1:odvvszpfUvNq5UMzUhimete71/ph+XQzzL11C/u3lUc=
+github.com/projectdiscovery/rawhttp v0.0.8-0.20220526170355-03de6bb78f37/go.mod h1:kulxvr2aKZPB6jhq4ZQn/E+ArwRWczs1O6b1ZdcZAxo=
 github.com/projectdiscovery/retryabledns v1.0.11/go.mod h1:4sMC8HZyF01HXukRleSQYwz4870bwgb4+hTSXTMrkf4=
 github.com/projectdiscovery/retryabledns v1.0.12/go.mod h1:4sMC8HZyF01HXukRleSQYwz4870bwgb4+hTSXTMrkf4=
-github.com/projectdiscovery/retryabledns v1.0.13-0.20210916165024-76c5b76fd59a h1:WJQjr9qi/VjWhdNiGyNqcFi0967Gp0W3I769bCpHOJE=
 github.com/projectdiscovery/retryabledns v1.0.13-0.20210916165024-76c5b76fd59a/go.mod h1:tXaLDs4n3pRZHwfa8mdXpUWe/AYDNK3HlWDjldhRbjI=
+github.com/projectdiscovery/retryabledns v1.0.13 h1:Ogfv0fl3Nszb+Nq2S2qQmE+PJDlStVxyLkmTotMohZY=
+github.com/projectdiscovery/retryabledns v1.0.13/go.mod h1:EeqHcAPp0g2GljT4qkxKSAE47Dj0ZrJQ46R9ct3Muhk=
 github.com/projectdiscovery/retryablehttp-go v1.0.1/go.mod h1:SrN6iLZilNG1X4neq1D+SBxoqfAF4nyzvmevkTkWsek=
-github.com/projectdiscovery/retryablehttp-go v1.0.2 h1:LV1/KAQU+yeWhNVlvveaYFsjBYRwXlNEq0PvrezMV0U=
 github.com/projectdiscovery/retryablehttp-go v1.0.2/go.mod h1:dx//aY9V247qHdsRf0vdWHTBZuBQ2vm6Dq5dagxrDYI=
+github.com/projectdiscovery/retryablehttp-go v1.0.3-0.20220506110515-811d938bd26d h1:VR+tDkedzHIp1pGKIDcfPFt7J8KjcjxGsJvBAP6RXFQ=
+github.com/projectdiscovery/retryablehttp-go v1.0.3-0.20220506110515-811d938bd26d/go.mod h1:t4buiLTB0HtI+62iHfGDqQVTv/i+8OhAKwaX93TGsFE=
+github.com/projectdiscovery/sliceutil v0.0.0-20220511171050-c7d9bc5cadd9 h1:dQO4FNxOhkZqZHW5nUh6f1NXIRUyoZ57bXETXjsyfNk=
+github.com/projectdiscovery/sliceutil v0.0.0-20220511171050-c7d9bc5cadd9/go.mod h1:crCchiT/KYrAlEY8JbuMGE6XKym61mhAd1E7REq2VwQ=
+github.com/projectdiscovery/stringsutil v0.0.0-20210524051937-51dabe3b72c0/go.mod h1:TVSdZC0rRQeMIbsNSiGPhbmhyRtxqqtAGA9JiiNp2r4=
 github.com/projectdiscovery/stringsutil v0.0.0-20210804142656-fd3c28dbaafe/go.mod h1:oTRc18WBv9t6BpaN9XBY+QmG28PUpsyDzRht56Qf49I=
 github.com/projectdiscovery/stringsutil v0.0.0-20210823090203-2f5f137e8e1d/go.mod h1:oTRc18WBv9t6BpaN9XBY+QmG28PUpsyDzRht56Qf49I=
-github.com/projectdiscovery/stringsutil v0.0.0-20210830151154-f567170afdd9 h1:xbL1/7h0k6HE3RzPdYk9W/8pUxESrGWewTaZdIB5Pes=
 github.com/projectdiscovery/stringsutil v0.0.0-20210830151154-f567170afdd9/go.mod h1:oTRc18WBv9t6BpaN9XBY+QmG28PUpsyDzRht56Qf49I=
-github.com/projectdiscovery/yamldoc-go v1.0.2 h1:SKb7PHgSOXm27Zci05ba0FxpyQiu6bGEiVMEcjCK1rQ=
+github.com/projectdiscovery/stringsutil v0.0.0-20220208075244-7c05502ca8e9/go.mod h1:oTRc18WBv9t6BpaN9XBY+QmG28PUpsyDzRht56Qf49I=
+github.com/projectdiscovery/stringsutil v0.0.0-20220422150559-b54fb5dc6833 h1:yo7hCL47BOHl8X/aMmPeRQwiqUrH6TZ2WjgqItaSPcc=
+github.com/projectdiscovery/stringsutil v0.0.0-20220422150559-b54fb5dc6833/go.mod h1:oTRc18WBv9t6BpaN9XBY+QmG28PUpsyDzRht56Qf49I=
+github.com/projectdiscovery/urlutil v0.0.0-20210525140139-b874f06ad921 h1:EgaxpJm7+lKppfAHkFHs+S+II0lodp4Gu3leZCCkWlc=
+github.com/projectdiscovery/urlutil v0.0.0-20210525140139-b874f06ad921/go.mod h1:oXLErqOpqEAp/ueQlknysFxHO3CUNoSiDNnkiHG+Jpo=
+github.com/projectdiscovery/wappalyzergo v0.0.45 h1:b2OJ4/FrQg6YBDwuK+xMJj/3+h6u+uHuZYURGYPqw3I=
+github.com/projectdiscovery/wappalyzergo v0.0.45/go.mod h1:vS+npIOANv7eKsEtODsyRQt2n1v8VofCwj2gjmq72EM=
 github.com/projectdiscovery/yamldoc-go v1.0.2/go.mod h1:7uSxfMXaBmzvw8m5EhOEjB6nhz0rK/H9sUjq1ciZu24=
+github.com/projectdiscovery/yamldoc-go v1.0.3-0.20211126104922-00d2c6bb43b6 h1:DvWRQpw7Ib2CRL3ogYm/BWM+X0UGPfz1n9Ix9YKgFM8=
+github.com/projectdiscovery/yamldoc-go v1.0.3-0.20211126104922-00d2c6bb43b6/go.mod h1:8OfZj8p/axkUM/TJoS/O9LDjj/S8u17rxRbqluE9CU4=
 github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
 github.com/remeh/sizedwaitgroup v1.0.0 h1:VNGGFwNo/R5+MJBf6yrsr110p0m4/OX4S3DCy7Kyl5E=
 github.com/remeh/sizedwaitgroup v1.0.0/go.mod h1:3j2R4OIe/SeS6YDhICBy22RWjJC5eNCJ1V+9+NVNYlo=
@@ -394,38 +524,49 @@ github.com/rogpeppe/go-internal v1.6.1/go.mod h1:xXDCJY+GAPziupqXw64V24skbSoqbTE
 github.com/rogpeppe/go-internal v1.6.2/go.mod h1:xXDCJY+GAPziupqXw64V24skbSoqbTEfhy4qGm1nDQc=
 github.com/rogpeppe/go-internal v1.8.0 h1:FCbCCtXNOY3UtUuHUYaghJg4y7Fd14rXifAYUAtL9R8=
 github.com/rogpeppe/go-internal v1.8.0/go.mod h1:WmiCO8CzOY8rg0OYDC4/i/2WRWAB6poM+XZ2dLUbcbE=
-github.com/rs/xid v1.3.0 h1:6NjYksEUlhurdVehpc7S7dk6DAmcKv8V9gG0FsVN2U4=
 github.com/rs/xid v1.3.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg=
+github.com/rs/xid v1.4.0 h1:qd7wPTDkN6KQx2VmMBLrpHkiyQwgFXRnkOLacUiaSNY=
+github.com/rs/xid v1.4.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg=
 github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
 github.com/segmentio/ksuid v1.0.3/go.mod h1:/XUiZBD3kVx5SmUOl55voK5yeAbBNNIed+2O73XgrPE=
 github.com/segmentio/ksuid v1.0.4 h1:sBo2BdShXjmcugAMwjugoGUdUV0pcxY5mW4xKRn3v4c=
 github.com/segmentio/ksuid v1.0.4/go.mod h1:/XUiZBD3kVx5SmUOl55voK5yeAbBNNIed+2O73XgrPE=
 github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
-github.com/shirou/gopsutil/v3 v3.21.7 h1:PnTqQamUjwEDSgn+nBGu0qSDV/CfvyiR/gwTH3i7HTU=
 github.com/shirou/gopsutil/v3 v3.21.7/go.mod h1:RGl11Y7XMTQPmHh8F0ayC6haKNBgH4PXMJuTAcMOlz4=
+github.com/shirou/gopsutil/v3 v3.22.5 h1:atX36I/IXgFiB81687vSiBI5zrMsxcIBkP9cQMJQoJA=
+github.com/shirou/gopsutil/v3 v3.22.5/go.mod h1:so9G9VzeHt/hsd0YwqprnjHnfARAUktauykSbr+y2gA=
+github.com/sirupsen/logrus v1.3.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
+github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
+github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
+github.com/smartystreets/assertions v1.0.0 h1:UVQPSSmc3qtTi+zPPkCXvZX9VvW/xT/NsRvKfwY81a8=
 github.com/smartystreets/assertions v1.0.0/go.mod h1:kHHU4qYBaI3q23Pp3VPrmWhuIUrLW/7eUrw0BU5VaoM=
 github.com/smartystreets/go-aws-auth v0.0.0-20180515143844-0c1422d1fdb9/go.mod h1:SnhjPscd9TpLiy1LpzGSKh3bXCfxxXuqd9xmQJy3slM=
+github.com/smartystreets/goconvey v0.0.0-20190330032615-68dc04aab96a h1:pa8hGb/2YqsZKovtsgrwcDH1RZhVbTKCjLp47XpqCDs=
+github.com/smartystreets/goconvey v0.0.0-20190330032615-68dc04aab96a/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
 github.com/smartystreets/gunit v1.0.0/go.mod h1:qwPWnhz6pn0NnRBP++URONOVyNkPyr4SauJk4cUOwJs=
 github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
 github.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI=
 github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
 github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
 github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
-github.com/spf13/cast v1.4.1 h1:s0hze+J0196ZfEMTs80N7UlFt0BDuQ7Q+JDnHiMWKdA=
 github.com/spf13/cast v1.4.1/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
+github.com/spf13/cast v1.5.0 h1:rj3WzYc11XZaIZMPKmwP96zkFEnnAmV8s6XbB2aY32w=
+github.com/spf13/cast v1.5.0/go.mod h1:SpXXQ5YoyJw6s3/6cMTQuxvgRl3PCJiyaX9p6b155UU=
 github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU=
 github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
 github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
 github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
 github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
 github.com/stretchr/testify v1.3.1-0.20190311161405-34c6fa2dc709/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
 github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
 github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
 github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
-github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
 github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
+github.com/stretchr/testify v1.7.1 h1:5TQK59W5E3v0r2duFAb7P95B6hEeOyEnHRa8MjYSMTY=
+github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/syndtr/goleveldb v1.0.0 h1:fBdIW9lB4Iz0n9khmH8w27SJ3QEJ7+IgjPEwGSZiFdE=
 github.com/syndtr/goleveldb v1.0.0/go.mod h1:ZVVdQEZoIme9iO1Ch2Jdy24qqXrMMOU6lpPAyBWyWuQ=
 github.com/tj/assert v0.0.0-20171129193455-018094318fb0/go.mod h1:mZ9/Rh9oLWpLLDRpvE+3b7gP/C2YyLFYxNmcLnPTMe0=
@@ -437,41 +578,53 @@ github.com/tj/go-kinesis v0.0.0-20171128231115-08b17f58cb1b/go.mod h1:/yhzCV0xPf
 github.com/tj/go-spin v1.1.0/go.mod h1:Mg1mzmePZm4dva8Qz60H2lHwmJ2loum4VIrLgVnKwh4=
 github.com/tj/go-update v2.2.5-0.20200519121640-62b4b798fd68+incompatible h1:guTq1YxwB8XSILkI9q4IrOmrCOS6Hc1L3hmOhi4Swcs=
 github.com/tj/go-update v2.2.5-0.20200519121640-62b4b798fd68+incompatible/go.mod h1:waFwwyiAhGey2e+dNoYQ/iLhIcFqhCW7zL/+vDU1WLo=
-github.com/tklauser/go-sysconf v0.3.7 h1:HT7h4+536gjqeq1ZIJPgOl1rg1XFatQGVZWp7Py53eg=
 github.com/tklauser/go-sysconf v0.3.7/go.mod h1:JZIdXh4RmBvZDBZ41ld2bGxRV3n4daiiqA3skYhAoQ4=
-github.com/tklauser/numcpus v0.2.3 h1:nQ0QYpiritP6ViFhrKYsiv6VVxOpum2Gks5GhnJbS/8=
+github.com/tklauser/go-sysconf v0.3.10 h1:IJ1AZGZRWbY8T5Vfk04D9WOA5WSejdflXxP03OUqALw=
+github.com/tklauser/go-sysconf v0.3.10/go.mod h1:C8XykCvCb+Gn0oNCWPIlcb0RuglQTYaQ2hGm7jmxEFk=
 github.com/tklauser/numcpus v0.2.3/go.mod h1:vpEPS/JC+oZGGQ/My/vJnNsvMDQL6PwOqt8dsCw5j+E=
+github.com/tklauser/numcpus v0.4.0 h1:E53Dm1HjH1/R2/aoCtXtPgzmElmn51aOkhCFSuZq//o=
+github.com/tklauser/numcpus v0.4.0/go.mod h1:1+UI3pD8NW14VMwdgJNJ1ESk2UnwhAnz5hMwiKKqXCQ=
 github.com/trivago/tgo v1.0.7 h1:uaWH/XIy9aWYWpjm2CU3RpcqZXmX2ysQ9/Go+d9gyrM=
 github.com/trivago/tgo v1.0.7/go.mod h1:w4dpD+3tzNIIiIfkWWa85w5/B77tlvdZckQ+6PkFnhc=
 github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0=
 github.com/ulikunitz/xz v0.5.6/go.mod h1:2bypXElzHzzJZwzH67Y6wb67pO62Rzfn7BSiF4ABRW8=
 github.com/ulikunitz/xz v0.5.10 h1:t92gobL9l3HE202wg3rlk19F6X+JOxl9BBrCCMYEYd8=
 github.com/ulikunitz/xz v0.5.10/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14=
+github.com/ulule/deepcopier v0.0.0-20200430083143-45decc6639b6 h1:TtyC78WMafNW8QFfv3TeP3yWNDG+uxNkk9vOrnDu6JA=
+github.com/ulule/deepcopier v0.0.0-20200430083143-45decc6639b6/go.mod h1:h8272+G2omSmi30fBXiZDMkmHuOgonplfKIKjQWzlfs=
 github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
 github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
 github.com/valyala/fasttemplate v1.2.1 h1:TVEnxayobAdVkhQfrfes2IzOB6o+z4roRkPF52WA1u4=
 github.com/valyala/fasttemplate v1.2.1/go.mod h1:KHLXt3tVN2HBp8eijSv/kGJopbvo7S+qRAEEKiv+SiQ=
 github.com/vmihailenco/msgpack/v4 v4.3.12/go.mod h1:gborTTJjAo/GWTqqRjrLCn9pgNN+NXzzngzBKDPIqw4=
 github.com/vmihailenco/tagparser v0.1.1/go.mod h1:OeAg3pn3UbLjkWt+rN9oFYB6u/cQgqMEUPoW2WPyhdI=
+github.com/weppos/publicsuffix-go v0.15.1-0.20210928183822-5ee35905bd95 h1:DyAZOw3JsVd6LJHqhl4MpKQdYQEmat0C6pPPwom39Ow=
+github.com/weppos/publicsuffix-go v0.15.1-0.20210928183822-5ee35905bd95/go.mod h1:HYux0V0Zi04bHNwOHy4cXJVz/TQjYonnF6aoYhj+3QE=
 github.com/wsxiaoys/terminal v0.0.0-20160513160801-0940f3fc43a0 h1:3UeQBvD0TFrlVjOeLOBz+CPAI8dnbqNSVwUwRrkp7vQ=
 github.com/wsxiaoys/terminal v0.0.0-20160513160801-0940f3fc43a0/go.mod h1:IXCdmsXIht47RaVFLEdVnh1t+pgYtTAhQGj73kz+2DM=
-github.com/xanzy/go-gitlab v0.50.3 h1:M7ncgNhCN4jaFNyXxarJhCLa9Qi6fdmCxFFhMTQPZiY=
 github.com/xanzy/go-gitlab v0.50.3/go.mod h1:Q+hQhV508bDPoBijv7YjK/Lvlb4PhVhJdKqXVQrUoAE=
+github.com/xanzy/go-gitlab v0.68.0 h1:b2iMQHgZ1V+NyRqLRJVv6RFfr4xnd/AASeS/PETYL0Y=
+github.com/xanzy/go-gitlab v0.68.0/go.mod h1:o4yExCtdaqlM8YGdDJWuZoBmfxBsmA9TPEjs9mx1UO4=
+github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 h1:nIPpBwaJSVYIxUFsDv3M8ofmx9yWTog9BfvIu0q41lo=
+github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8/go.mod h1:HUYIGzjTL3rfEspMxjDjgmT5uz5wzYJKVo23qUhYTos=
 github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
 github.com/yl2chen/cidranger v1.0.2 h1:lbOWZVCG1tCRX4u24kuM1Tb4nHqWkDxwLdoS+SevawU=
 github.com/yl2chen/cidranger v1.0.2/go.mod h1:9U1yz7WPYDwf0vpNWFaeRh0bjwz5RVgRy/9UEQfHl0g=
-github.com/ysmood/goob v0.3.0 h1:XZ51cZJ4W3WCoCiUktixzMIQF86W7G5VFL4QQ/Q2uS0=
 github.com/ysmood/goob v0.3.0/go.mod h1:S3lq113Y91y1UBf1wj1pFOxeahvfKkCk6mTWTWbDdWs=
+github.com/ysmood/goob v0.4.0 h1:HsxXhyLBeGzWXnqVKtmT9qM7EuVs/XOgkX7T6r1o1AQ=
+github.com/ysmood/goob v0.4.0/go.mod h1:u6yx7ZhS4Exf2MwciFr6nIM8knHQIE22lFpWHnfql18=
 github.com/ysmood/got v0.9.3/go.mod h1:pE1l4LOwOBhQg6A/8IAatkGp7uZjnalzrZolnlhhMgY=
 github.com/ysmood/got v0.14.1/go.mod h1:pE1l4LOwOBhQg6A/8IAatkGp7uZjnalzrZolnlhhMgY=
-github.com/ysmood/got v0.15.1 h1:X5jAbMyBf5yeezuFMp9HaMGXZWMSqIQcUlAHI+kJmUs=
-github.com/ysmood/got v0.15.1/go.mod h1:pE1l4LOwOBhQg6A/8IAatkGp7uZjnalzrZolnlhhMgY=
+github.com/ysmood/got v0.29.1 h1:7TNTm3Bw5kdBGXFp04qnZ9DqlZ1XS1z1JdeobeJY6Mo=
+github.com/ysmood/got v0.29.1/go.mod h1:pE1l4LOwOBhQg6A/8IAatkGp7uZjnalzrZolnlhhMgY=
 github.com/ysmood/gotrace v0.2.0/go.mod h1:TzhIG7nHDry5//eYZDYcTzuJLYQIkykJzCRIo4/dzQM=
-github.com/ysmood/gotrace v0.2.2 h1:006KHGRThSRf8lwh4EyhNmuuq/l+Ygs+JqojkhEG1/E=
 github.com/ysmood/gotrace v0.2.2/go.mod h1:TzhIG7nHDry5//eYZDYcTzuJLYQIkykJzCRIo4/dzQM=
+github.com/ysmood/gotrace v0.6.0 h1:SyI1d4jclswLhg7SWTL6os3L1WOKeNn/ZtzVQF8QmdY=
+github.com/ysmood/gotrace v0.6.0/go.mod h1:TzhIG7nHDry5//eYZDYcTzuJLYQIkykJzCRIo4/dzQM=
 github.com/ysmood/gson v0.6.3/go.mod h1:3Kzs5zDl21g5F/BlLTNcuAGAYLKt2lV5G8D1zF3RNmg=
-github.com/ysmood/gson v0.6.4 h1:Yb6tosv6bk59HqjZu2/7o4BFherpYEMkDkXmlhgryZ4=
 github.com/ysmood/gson v0.6.4/go.mod h1:3Kzs5zDl21g5F/BlLTNcuAGAYLKt2lV5G8D1zF3RNmg=
+github.com/ysmood/gson v0.7.1 h1:zKL2MTGtynxdBdlZjyGsvEOZ7dkxaY5TH6QhAbTgz0Q=
+github.com/ysmood/gson v0.7.1/go.mod h1:3Kzs5zDl21g5F/BlLTNcuAGAYLKt2lV5G8D1zF3RNmg=
 github.com/ysmood/leakless v0.6.12/go.mod h1:R8iAXPRaG97QJwqxs74RdwzcRHT1SWCGTNqY8q0JvMQ=
 github.com/ysmood/leakless v0.7.0 h1:XCGdaPExyoreoQd+H5qgxM3ReNbSPFsEXpSKwbXbwQw=
 github.com/ysmood/leakless v0.7.0/go.mod h1:R8iAXPRaG97QJwqxs74RdwzcRHT1SWCGTNqY8q0JvMQ=
@@ -479,8 +632,16 @@ github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9de
 github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
-github.com/zclconf/go-cty v1.8.4 h1:pwhhz5P+Fjxse7S7UriBrMu6AUJSZM5pKqGem1PjGAs=
+github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
+github.com/yusufpapurcu/wmi v1.2.2 h1:KBNDSne4vP5mbSWnJbO+51IMOXJB67QiYCSBrubbPRg=
+github.com/yusufpapurcu/wmi v1.2.2/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
 github.com/zclconf/go-cty v1.8.4/go.mod h1:vVKLxnk3puL4qRAv72AO+W99LUD4da90g3uUAzyuvAk=
+github.com/zclconf/go-cty v1.10.0/go.mod h1:vVKLxnk3puL4qRAv72AO+W99LUD4da90g3uUAzyuvAk=
+github.com/zmap/rc2 v0.0.0-20131011165748-24b9757f5521 h1:kKCF7VX/wTmdg2ZjEaqlq99Bjsoiz7vH6sFniF/vI4M=
+github.com/zmap/rc2 v0.0.0-20131011165748-24b9757f5521/go.mod h1:3YZ9o3WnatTIZhuOtot4IcUfzoKVjUHqu6WALIyI0nE=
+github.com/zmap/zcertificate v0.0.0-20180516150559-0e3d58b1bac4/go.mod h1:5iU54tB79AMBcySS0R2XIyZBAVmeHranShAFELYx7is=
+github.com/zmap/zcrypto v0.0.0-20211005224000-2d0ffdec8a9b h1:iYQzlljG1dOXBtsJGyzFC/wBK5qUCWs1eLCr/UcJYPA=
+github.com/zmap/zcrypto v0.0.0-20211005224000-2d0ffdec8a9b/go.mod h1:5nID//bFGkx3/+iHcFIFRHQ54EOPJ0iSj0IGKpMElvw=
 go.etcd.io/bbolt v1.3.5/go.mod h1:G5EMThwa9y8QZGBClrRx5EY+Yw9kAhnjy3bSjsnlVTQ=
 go.etcd.io/bbolt v1.3.6 h1:/ecaJf0sk1l4l6V4awd65v2C3ILy7MSj+s/x1ADCIMU=
 go.etcd.io/bbolt v1.3.6/go.mod h1:qXsaaIqmgQH0T+OPdb99Bf+PKfBBQVAdyD6TY9G8XM4=
@@ -489,22 +650,41 @@ go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
 go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
 go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
 go.opencensus.io v0.22.4/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
+go.uber.org/atomic v1.6.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ=
 go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
 go.uber.org/atomic v1.9.0 h1:ECmE8Bn/WFTYwEW/bpKD3M8VtR/zQVbavAoalC1PYyE=
 go.uber.org/atomic v1.9.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
-go.uber.org/multierr v1.7.0 h1:zaiO/rmgFjbmCXdSYJWQcdvOCsthmdaHfr3Gm2Kx4Ec=
+go.uber.org/goleak v1.1.11 h1:wy28qYRKZgnJTxGxvye5/wgWr1EKjmUDGYox5mGlRlI=
+go.uber.org/goleak v1.1.11/go.mod h1:cwTWslyiVhfpKIDGSZEM2HlOvcqm+tG4zioyIeLoqMQ=
+go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU=
+go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU=
 go.uber.org/multierr v1.7.0/go.mod h1:7EAYxJLBy9rStEaz58O2t4Uvip6FSURkq8/ppBp95ak=
+go.uber.org/multierr v1.8.0 h1:dg6GjLku4EH+249NNmoIciG9N/jURbDG+pFlTkhzIC8=
+go.uber.org/multierr v1.8.0/go.mod h1:7EAYxJLBy9rStEaz58O2t4Uvip6FSURkq8/ppBp95ak=
 go.uber.org/ratelimit v0.2.0 h1:UQE2Bgi7p2B85uP5dC2bbRtig0C+OeNRnNEafLjsLPA=
 go.uber.org/ratelimit v0.2.0/go.mod h1:YYBV4e4naJvhpitQrWJu1vCpgB7CboMe0qhltKt6mUg=
+go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA=
+go.uber.org/zap v1.15.0/go.mod h1:Mb2vm2krFEG5DV0W9qcHBYFtp/Wku1cvYaqPsS/WYfc=
+go.uber.org/zap v1.21.0 h1:WefMeulhovoZ2sYXz7st6K0sLj7bBhpiFaud4r4zST8=
+go.uber.org/zap v1.21.0/go.mod h1:wjWOCqI0f2ZZrJF/UufIOkiC8ii6tm1iqIsLo76RfJw=
+goftp.io/server/v2 v2.0.0 h1:FF8JKXXKDxAeO1uXEZz7G+IZwCDhl19dpVIlDtp3QAg=
+goftp.io/server/v2 v2.0.0/go.mod h1:7+H/EIq7tXdfo1Muu5p+l3oQ6rYkDZ8lY7IM5d5kVdQ=
 golang.org/x/arch v0.0.0-20180920145803-b19384d3c130/go.mod h1:cYlCBUl1MsqxdiKgmc4uh7TxZfWSFLOGSRR090WDxt8=
+golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
 golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/crypto v0.0.0-20190513172903-22d7a77e9e5f/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/crypto v0.0.0-20190701094942-4def268fd1a4/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/crypto v0.0.0-20201112155050-0c6587e931a9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
+golang.org/x/crypto v0.0.0-20201124201722-c8d3bf9c5392/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
+golang.org/x/crypto v0.0.0-20211215153901-e495a2d5b3d3/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
+golang.org/x/crypto v0.0.0-20220210151621-f4118a5b28e2 h1:XdAboW3BNMv9ocSCOk/u1MFioZGzCNkiJZ19v9Oe3Ig=
+golang.org/x/crypto v0.0.0-20220210151621-f4118a5b28e2/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@@ -536,6 +716,8 @@ golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzB
 golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
 golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
 golang.org/x/mod v0.4.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
+golang.org/x/mod v0.4.2 h1:Gz96sIWK3OalVv/I/qNygP42zyoKp3xptRVCWRFEBvo=
+golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
 golang.org/x/net v0.0.0-20180218175443-cbe0f9307d01/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -546,6 +728,7 @@ golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn
 golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
 golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
 golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20190628185345-da137c7871d7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
@@ -568,22 +751,30 @@ golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81R
 golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
 golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
 golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
+golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
 golang.org/x/net v0.0.0-20201202161906-c7110b5ffcbb/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
 golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
+golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
 golang.org/x/net v0.0.0-20210414194228-064579744ee0/go.mod h1:9tjilg8BloeKEkVJvy7fQ90B1CfIiPueXVOjqfkSzI8=
 golang.org/x/net v0.0.0-20210428140749-89ef3d95e781/go.mod h1:OJAsFXCWl8Ukc7SiCT/9KSuxbyM7479/AVlXFRxuMCk=
 golang.org/x/net v0.0.0-20210521195947-fe42d452be8f/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
+golang.org/x/net v0.0.0-20210726213435-c6fcb2dbf985/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
 golang.org/x/net v0.0.0-20210813160813-60bc85c4be6d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
-golang.org/x/net v0.0.0-20210916014120-12bc252f5db8 h1:/6y1LfuqNuQdHAm0jjtPtgRcxIxjVZgm5OTu8/QhZvk=
-golang.org/x/net v0.0.0-20210916014120-12bc252f5db8/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
+golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
+golang.org/x/net v0.0.0-20211216030914-fe4d6282115f/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
+golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
+golang.org/x/net v0.0.0-20220425223048-2871e0cb64e4/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
+golang.org/x/net v0.0.0-20220520000938-2e3eb7b945c2 h1:NWy5+hlRbC7HK+PmcXVUmW1IMyFce7to56IUvhUFm7Y=
+golang.org/x/net v0.0.0-20220520000938-2e3eb7b945c2/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20181106182150-f42d05182288/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
-golang.org/x/oauth2 v0.0.0-20210817223510-7df4dd6e12ab h1:llrcWN/wOwO+6gAyfBzxb5hZ+c3mriU/0+KNgYu6adA=
 golang.org/x/oauth2 v0.0.0-20210817223510-7df4dd6e12ab/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
+golang.org/x/oauth2 v0.0.0-20220411215720-9780585627b5 h1:OSnWWcOd/CtWQC2cYSBgbTSJv3ciqd8r54ySIW2y3RE=
+golang.org/x/oauth2 v0.0.0-20220411215720-9780585627b5/go.mod h1:DAh4E804XQdzx2j+YRIaUnCqCV2RuMz24cGBJ5QYIrc=
 golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -597,12 +788,14 @@ golang.org/x/sync v0.0.0-20210220032951-036812b2e83c h1:5KslGYwFpkhGh+Q16bwMP3cO
 golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180903190138-2b024373dcd9/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@@ -614,6 +807,7 @@ golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20190924154521-2837fb4f24fe/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20191001151750-bb3f8db39f24/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@@ -635,6 +829,8 @@ golang.org/x/sys v0.0.0-20200923182605-d9f96fdee20d/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20201113233024-12cec1faf1ba/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20201204225414-ed752295db88/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20201207223542-d4d67f95c62d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210112080510-489259a85091/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210303074136-134d130e1a04/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@@ -642,13 +838,20 @@ golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20210415045647-66c3f260301c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210426230700-d19ff857e887/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20210601080250-7ecdf8ef093b/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20210806184541-e5e7981a1069/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20210823070655-63515b42dcdf/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.0.0-20210915083310-ed5796bab164 h1:7ZDGnxgHAMw7thfC5bEos0RDAccZKxioiWBhfIe+tvw=
-golang.org/x/sys v0.0.0-20210915083310-ed5796bab164/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220128215802-99c3d69c2c27/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a h1:dGzPydgVsqGcTRVwiLJ1jVbufYwmzD3LfVPLKsKg+0k=
+golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
+golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -660,14 +863,16 @@ golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
 golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
 golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
-golang.org/x/time v0.0.0-20191024005414-555d28b269f0 h1:/5xXl8Y5W96D+TtHSlonuFqGHIWVuyCkGJLwGh9JJFs=
 golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
+golang.org/x/time v0.0.0-20220411224347-583f2d630306 h1:+gHMid33q6pen7kv9xvT+JRinntgeXO2AeZVd0AWD3w=
+golang.org/x/time v0.0.0-20220411224347-583f2d630306/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
 golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
 golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
 golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
 golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
 golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
 golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
@@ -677,6 +882,8 @@ golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgw
 golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20191029190741-b9c20aec41a5/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20191113191852-77e3bb0ad9e7/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20191115202509-3a792d9c32b2/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
@@ -707,6 +914,9 @@ golang.org/x/tools v0.0.0-20200825202427-b303f430e36d/go.mod h1:njjCfa9FT2d7l9Bc
 golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
 golang.org/x/tools v0.0.0-20210101214203-2dba1e4ea05c/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
 golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0=
+golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
+golang.org/x/tools v0.1.6-0.20210726203631-07bc1bf47fb2 h1:BonxutuHCTL0rBDnZlKjpGIQFTjyUVTexFOdWkB6Fg0=
+golang.org/x/tools v0.1.6-0.20210726203631-07bc1bf47fb2/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
@@ -790,8 +1000,11 @@ google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGj
 google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c=
 google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
 google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
-google.golang.org/protobuf v1.27.1 h1:SnqbnDw1V7RiZcXPx5MEeqPv2s79L9i7BJUlG/+RurQ=
 google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
+google.golang.org/protobuf v1.28.0 h1:w43yiav+6bVFTBQFZX0r7ipe9JQ1QsbMgHwbBziscLw=
+google.golang.org/protobuf v1.28.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
+gopkg.in/alecthomas/kingpin.v2 v2.2.6 h1:jMFz6MfLP0/4fUyZle81rXUoxOBFi19VUFKVDOQfozc=
+gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
@@ -801,19 +1014,22 @@ gopkg.in/corvus-ch/zbase32.v1 v1.0.0 h1:K4u1NprbDNvKPczKfHLbwdOWHTZ0zfv2ow71H1nR
 gopkg.in/corvus-ch/zbase32.v1 v1.0.0/go.mod h1:T3oKkPOm4AV/bNXCNFUxRmlE9RUyBz/DSo0nK9U+c0Y=
 gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
 gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
+gopkg.in/ini.v1 v1.42.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
 gopkg.in/src-d/go-billy.v4 v4.3.0/go.mod h1:tm33zBoOwxjYHZIE+OV8bxTWFMJLrconzFMd38aARFk=
 gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
 gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
 gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
 gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 gopkg.in/yaml.v3 v3.0.0-20200605160147-a5ece683394c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
-gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b h1:h8qDotaEPuJATrMmW04NCwg7v22aHH28wwpauUhK9Oo=
 gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
 honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
 honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
@@ -821,6 +1037,8 @@ honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWh
 honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg=
 honnef.co/go/tools v0.0.1-2020.1.3/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k=
 honnef.co/go/tools v0.0.1-2020.1.4/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k=
+moul.io/http2curl v1.0.0 h1:6XwpyZOYsgZJrU8exnG87ncVkU1FVCcTRpwzOkTDUi8=
+moul.io/http2curl v1.0.0/go.mod h1:f6cULg+e4Md/oW1cYmwW4IWQOVl2lGbmCNGOHvzX2kE=
 mvdan.cc/gofumpt v0.1.1/go.mod h1:yXG1r1WqZVKWbVRtBWKWX9+CxGYfA51nSomhM0woR48=
 rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
 rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0=